mips: Membar audit. This change should be safe because it doesn't remove or weaken any memory barriers, but does add, clarify, or strengthen barriers. Goals: - Make sure mutex_enter/exit and mutex_spin_enter/exit have acquire/release semantics. - New macros make maintenance easier and purpose clearer: . SYNC_ACQ is for load-before-load/store barrier, and BDSYNC_ACQ for a branch delay slot -- currently defined as plain sync for MP and nothing, or nop, for UP; thus it is no weaker than SYNC and BDSYNC as currently defined, which is syncw on Octeon, plain sync on non-Octeon MP, and nothing/nop on UP. It is not clear to me whether load-then-syncw or ll/sc-then-syncw or even bare load provides load-acquire semantics on Octeon -- if no, this will fix bugs; if yes (like it is on SPARC PSO), we can relax SYNC_ACQ to be syncw or nothing later. . SYNC_REL is for load/store-before-store barrier -- currently defined as plain sync for MP and nothing for UP. It is not clear to me whether syncw-then-store is enough for store-release on Octeon -- if no, we can leave this as is; if yes, we can relax SYNC_REL to be syncw on Octeon. . SYNC_PLUNGER is there to flush clogged Cavium store buffers, and BDSYNC_PLUNGER for a branch delay slot -- syncw on Octeon, nothing or nop on non-Octeon. => This is not necessary (or, as far as I'm aware, sufficient) for acquire semantics -- it serves only to flush store buffers where stores might otherwise linger for hundreds of thousands of cycles, which would, e.g., cause spin locks to be held for unreasonably long durations. Newerish revisions of the MIPS ISA also have finer-grained sync variants that could be plopped in here. Mechanism: Insert these barriers in the right places, replacing only those where the definition is currently equivalent, so this change is safe. - Replace #ifdef _MIPS_ARCH_OCTEONP / syncw / #endif at the end of atomic_cas_* by SYNC_PLUNGER, which is `sync 4' (a.k.a. syncw) if __OCTEON__ and empty otherwise. => From what I can tell, __OCTEON__ is defined in at least as many contexts as _MIPS_ARCH_OCTEONP -- i.e., there are some Octeons with no _MIPS_ARCH_OCTEONP, but I don't know if any of them are relevant to us or ever saw the light of day outside Cavium; we seem to buid with `-march=octeonp' so this is unlikely to make a difference. If it turns out that we do care, well, now there's a central place to make the distinction for sync instructions. - Replace post-ll/sc SYNC by SYNC_ACQ in _atomic_cas_*, which are internal kernel versions used in sys/arch/mips/include/lock.h where it assumes they have load-acquire semantics. Should move this to lock.h later, since we _don't_ define __HAVE_ATOMIC_AS_MEMBAR on MIPS and so the extra barrier might be costly. - Insert SYNC_REL before ll/sc, and replace post-ll/sc SYNC by SYNC_ACQ, in _ucas_*, which is used without any barriers in futex code and doesn't mention barriers in the man page so I have to assume it is required to be a release/acquire barrier. - Change BDSYNC to BDSYNC_ACQ in mutex_enter and mutex_spin_enter. This is necessary to provide load-acquire semantics -- unclear if it was provided already by syncw on Octeon, but it seems more likely that either (a) no sync or syncw is needed at all, or (b) syncw is not enough and sync is needed, since syncw is only a store-before-store ordering barrier. - Insert SYNC_REL before ll/sc in mutex_exit and mutex_spin_exit. This is currently redundant with the SYNC already there, but SYNC_REL more clearly identifies the necessary semantics in case we want to define it differently on different systems, and having a sync in the middle of an ll/sc is a bit weird and possibly not a good idea, so I intend to (carefully) remove the redundant SYNC in a later change. - Change BDSYNC to BDSYNC_PLUNGER at the end of mutex_exit. This has no semantic change right now -- it's syncw on Octeon, sync on non-Octeon MP, nop on UP -- but we can relax it later to nop on non-Cavium MP. - Leave LLSCSYNC in for now -- it is apparently there for a Cavium erratum, but I'm not sure what the erratum is, exactly, and I have no reference for it. I suspect these can be safely removed, but we might have to double up some other syncw instructions -- Linux uses it only in store-release sequences, not at the head of every ll/sc.diff -r1.8 -r1.9 src/common/lib/libc/arch/mips/atomic/atomic_cas.S
(riastradh)
--- src/common/lib/libc/arch/mips/atomic/atomic_cas.S 2020/08/06 10:00:21 1.8
+++ src/common/lib/libc/arch/mips/atomic/atomic_cas.S 2022/02/27 19:21:53 1.9
@@ -1,14 +1,14 @@ | @@ -1,14 +1,14 @@ | |||
1 | /* $NetBSD: atomic_cas.S,v 1.8 2020/08/06 10:00:21 skrll Exp $ */ | 1 | /* $NetBSD: atomic_cas.S,v 1.9 2022/02/27 19:21:53 riastradh Exp $ */ | |
2 | 2 | |||
3 | /*- | 3 | /*- | |
4 | * Copyright (c) 2008 The NetBSD Foundation, Inc. | 4 | * Copyright (c) 2008 The NetBSD Foundation, Inc. | |
5 | * All rights reserved. | 5 | * All rights reserved. | |
6 | * | 6 | * | |
7 | * Redistribution and use in source and binary forms, with or without | 7 | * Redistribution and use in source and binary forms, with or without | |
8 | * modification, are permitted provided that the following conditions | 8 | * modification, are permitted provided that the following conditions | |
9 | * are met: | 9 | * are met: | |
10 | * 1. Redistributions of source code must retain the above copyright | 10 | * 1. Redistributions of source code must retain the above copyright | |
11 | * notice, this list of conditions and the following disclaimer. | 11 | * notice, this list of conditions and the following disclaimer. | |
12 | * 2. Redistributions in binary form must reproduce the above copyright | 12 | * 2. Redistributions in binary form must reproduce the above copyright | |
13 | * notice, this list of conditions and the following disclaimer in the | 13 | * notice, this list of conditions and the following disclaimer in the | |
14 | * documentation and/or other materials provided with the distribution. | 14 | * documentation and/or other materials provided with the distribution. | |
@@ -19,69 +19,65 @@ | @@ -19,69 +19,65 @@ | |||
19 | * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS | 19 | * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS | |
20 | * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR | 20 | * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR | |
21 | * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF | 21 | * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF | |
22 | * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | 22 | * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | |
23 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN | 23 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN | |
24 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) | 24 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) | |
25 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | 25 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | |
26 | * POSSIBILITY OF SUCH DAMAGE. | 26 | * POSSIBILITY OF SUCH DAMAGE. | |
27 | */ | 27 | */ | |
28 | 28 | |||
29 | #include <machine/asm.h> | 29 | #include <machine/asm.h> | |
30 | #include "atomic_op_asm.h" | 30 | #include "atomic_op_asm.h" | |
31 | 31 | |||
32 | RCSID("$NetBSD: atomic_cas.S,v 1.8 2020/08/06 10:00:21 skrll Exp $") | 32 | RCSID("$NetBSD: atomic_cas.S,v 1.9 2022/02/27 19:21:53 riastradh Exp $") | |
33 | 33 | |||
34 | .text | 34 | .text | |
35 | .set noat | 35 | .set noat | |
36 | .set noreorder | 36 | .set noreorder | |
37 | .set nomacro | 37 | .set nomacro | |
38 | 38 | |||
39 | LEAF(_atomic_cas_32) | 39 | LEAF(_atomic_cas_32) | |
40 | LLSCSYNC | 40 | LLSCSYNC | |
41 | 1: INT_LL v0, 0(a0) | 41 | 1: INT_LL v0, 0(a0) | |
42 | nop | 42 | nop | |
43 | bne v0, a1, 2f | 43 | bne v0, a1, 2f | |
44 | nop | 44 | nop | |
45 | move t0, a2 | 45 | move t0, a2 | |
46 | INT_SC t0, 0(a0) | 46 | INT_SC t0, 0(a0) | |
47 | beq t0, zero, 1b | 47 | beq t0, zero, 1b | |
48 | nop | 48 | nop | |
49 | move v0, a1 | 49 | move v0, a1 | |
50 | #ifdef _MIPS_ARCH_OCTEONP | 50 | SYNC_PLUNGER | |
51 | syncw | |||
52 | #endif | |||
53 | 2: | 51 | 2: | |
54 | j ra | 52 | j ra | |
55 | nop | 53 | nop | |
56 | END(_atomic_cas_32) | 54 | END(_atomic_cas_32) | |
57 | ATOMIC_OP_ALIAS(atomic_cas_32, _atomic_cas_32) | 55 | ATOMIC_OP_ALIAS(atomic_cas_32, _atomic_cas_32) | |
58 | ATOMIC_OP_ALIAS(atomic_cas_32_ni, _atomic_cas_32) | 56 | ATOMIC_OP_ALIAS(atomic_cas_32_ni, _atomic_cas_32) | |
59 | 57 | |||
60 | #if !defined(__mips_o32) | 58 | #if !defined(__mips_o32) | |
61 | LEAF(_atomic_cas_64) | 59 | LEAF(_atomic_cas_64) | |
62 | LLSCSYNC | 60 | LLSCSYNC | |
63 | 1: REG_LL v0, 0(a0) | 61 | 1: REG_LL v0, 0(a0) | |
64 | nop | 62 | nop | |
65 | bne v0, a1, 2f | 63 | bne v0, a1, 2f | |
66 | nop | 64 | nop | |
67 | move t0, a2 | 65 | move t0, a2 | |
68 | REG_SC t0, 0(a0) | 66 | REG_SC t0, 0(a0) | |
69 | beq t0, zero, 1b | 67 | beq t0, zero, 1b | |
70 | nop | 68 | nop | |
71 | move v0, a1 | 69 | move v0, a1 | |
72 | #ifdef _MIPS_ARCH_OCTEONP | 70 | SYNC_PLUNGER | |
73 | syncw | |||
74 | #endif | |||
75 | 2: | 71 | 2: | |
76 | j ra | 72 | j ra | |
77 | nop | 73 | nop | |
78 | END(_atomic_cas_64) | 74 | END(_atomic_cas_64) | |
79 | ATOMIC_OP_ALIAS(atomic_cas_64, _atomic_cas_64) | 75 | ATOMIC_OP_ALIAS(atomic_cas_64, _atomic_cas_64) | |
80 | ATOMIC_OP_ALIAS(atomic_cas_64_ni, _atomic_cas_64) | 76 | ATOMIC_OP_ALIAS(atomic_cas_64_ni, _atomic_cas_64) | |
81 | #endif | 77 | #endif | |
82 | 78 | |||
83 | #ifdef _LP64 | 79 | #ifdef _LP64 | |
84 | STRONG_ALIAS(_atomic_cas_ptr, _atomic_cas_64) | 80 | STRONG_ALIAS(_atomic_cas_ptr, _atomic_cas_64) | |
85 | STRONG_ALIAS(_atomic_cas_ptr_ni, _atomic_cas_64) | 81 | STRONG_ALIAS(_atomic_cas_ptr_ni, _atomic_cas_64) | |
86 | STRONG_ALIAS(_atomic_cas_ulong, _atomic_cas_64) | 82 | STRONG_ALIAS(_atomic_cas_ulong, _atomic_cas_64) | |
87 | STRONG_ALIAS(_atomic_cas_ulong_ni, _atomic_cas_64) | 83 | STRONG_ALIAS(_atomic_cas_ulong_ni, _atomic_cas_64) |
--- src/common/lib/libc/arch/mips/atomic/atomic_op_asm.h 2020/08/01 09:26:49 1.4
+++ src/common/lib/libc/arch/mips/atomic/atomic_op_asm.h 2022/02/27 19:21:53 1.5
@@ -1,14 +1,14 @@ | @@ -1,14 +1,14 @@ | |||
1 | /* $NetBSD: atomic_op_asm.h,v 1.4 2020/08/01 09:26:49 skrll Exp $ */ | 1 | /* $NetBSD: atomic_op_asm.h,v 1.5 2022/02/27 19:21:53 riastradh Exp $ */ | |
2 | 2 | |||
3 | /*- | 3 | /*- | |
4 | * Copyright (c) 2007 The NetBSD Foundation, Inc. | 4 | * Copyright (c) 2007 The NetBSD Foundation, Inc. | |
5 | * All rights reserved. | 5 | * All rights reserved. | |
6 | * | 6 | * | |
7 | * This code is derived from software contributed to The NetBSD Foundation | 7 | * This code is derived from software contributed to The NetBSD Foundation | |
8 | * by Jason R. Thorpe. | 8 | * by Jason R. Thorpe. | |
9 | * | 9 | * | |
10 | * Redistribution and use in source and binary forms, with or without | 10 | * Redistribution and use in source and binary forms, with or without | |
11 | * modification, are permitted provided that the following conditions | 11 | * modification, are permitted provided that the following conditions | |
12 | * are met: | 12 | * are met: | |
13 | * 1. Redistributions of source code must retain the above copyright | 13 | * 1. Redistributions of source code must retain the above copyright | |
14 | * notice, this list of conditions and the following disclaimer. | 14 | * notice, this list of conditions and the following disclaimer. | |
@@ -34,20 +34,14 @@ | @@ -34,20 +34,14 @@ | |||
34 | 34 | |||
35 | #include <machine/asm.h> | 35 | #include <machine/asm.h> | |
36 | 36 | |||
37 | #if defined(_KERNEL) | 37 | #if defined(_KERNEL) | |
38 | 38 | |||
39 | #define ATOMIC_OP_ALIAS(a,s) STRONG_ALIAS(a,s) | 39 | #define ATOMIC_OP_ALIAS(a,s) STRONG_ALIAS(a,s) | |
40 | 40 | |||
41 | #else /* _KERNEL */ | 41 | #else /* _KERNEL */ | |
42 | 42 | |||
43 | #define ATOMIC_OP_ALIAS(a,s) WEAK_ALIAS(a,s) | 43 | #define ATOMIC_OP_ALIAS(a,s) WEAK_ALIAS(a,s) | |
44 | 44 | |||
45 | #endif /* _KERNEL */ | 45 | #endif /* _KERNEL */ | |
46 | 46 | |||
47 | #ifdef __OCTEON__ | |||
48 | #define SYNCW syncw | |||
49 | #else | |||
50 | #define SYNCW nop | |||
51 | #endif | |||
52 | ||||
53 | #endif /* _ATOMIC_OP_ASM_H_ */ | 47 | #endif /* _ATOMIC_OP_ASM_H_ */ |
--- src/common/lib/libc/arch/mips/atomic/atomic_swap.S 2020/08/06 10:00:21 1.7
+++ src/common/lib/libc/arch/mips/atomic/atomic_swap.S 2022/02/27 19:21:53 1.8
@@ -1,14 +1,14 @@ | @@ -1,14 +1,14 @@ | |||
1 | /* $NetBSD: atomic_swap.S,v 1.7 2020/08/06 10:00:21 skrll Exp $ */ | 1 | /* $NetBSD: atomic_swap.S,v 1.8 2022/02/27 19:21:53 riastradh Exp $ */ | |
2 | 2 | |||
3 | /*- | 3 | /*- | |
4 | * Copyright (c) 2008 The NetBSD Foundation, Inc. | 4 | * Copyright (c) 2008 The NetBSD Foundation, Inc. | |
5 | * All rights reserved. | 5 | * All rights reserved. | |
6 | * | 6 | * | |
7 | * Redistribution and use in source and binary forms, with or without | 7 | * Redistribution and use in source and binary forms, with or without | |
8 | * modification, are permitted provided that the following conditions | 8 | * modification, are permitted provided that the following conditions | |
9 | * are met: | 9 | * are met: | |
10 | * 1. Redistributions of source code must retain the above copyright | 10 | * 1. Redistributions of source code must retain the above copyright | |
11 | * notice, this list of conditions and the following disclaimer. | 11 | * notice, this list of conditions and the following disclaimer. | |
12 | * 2. Redistributions in binary form must reproduce the above copyright | 12 | * 2. Redistributions in binary form must reproduce the above copyright | |
13 | * notice, this list of conditions and the following disclaimer in the | 13 | * notice, this list of conditions and the following disclaimer in the | |
14 | * documentation and/or other materials provided with the distribution. | 14 | * documentation and/or other materials provided with the distribution. | |
@@ -19,67 +19,67 @@ | @@ -19,67 +19,67 @@ | |||
19 | * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS | 19 | * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS | |
20 | * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR | 20 | * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR | |
21 | * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF | 21 | * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF | |
22 | * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | 22 | * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | |
23 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN | 23 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN | |
24 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) | 24 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) | |
25 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | 25 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | |
26 | * POSSIBILITY OF SUCH DAMAGE. | 26 | * POSSIBILITY OF SUCH DAMAGE. | |
27 | */ | 27 | */ | |
28 | 28 | |||
29 | #include <machine/asm.h> | 29 | #include <machine/asm.h> | |
30 | #include "atomic_op_asm.h" | 30 | #include "atomic_op_asm.h" | |
31 | 31 | |||
32 | RCSID("$NetBSD: atomic_swap.S,v 1.7 2020/08/06 10:00:21 skrll Exp $") | 32 | RCSID("$NetBSD: atomic_swap.S,v 1.8 2022/02/27 19:21:53 riastradh Exp $") | |
33 | 33 | |||
34 | .text | 34 | .text | |
35 | .set noreorder | 35 | .set noreorder | |
36 | #ifdef _KERNEL_OPT | 36 | #ifdef _KERNEL_OPT | |
37 | #include "opt_cputype.h" | 37 | #include "opt_cputype.h" | |
38 | #ifndef MIPS3_LOONGSON2F | 38 | #ifndef MIPS3_LOONGSON2F | |
39 | .set noat | 39 | .set noat | |
40 | .set nomacro | 40 | .set nomacro | |
41 | #endif | 41 | #endif | |
42 | #else /* _KERNEL_OPT */ | 42 | #else /* _KERNEL_OPT */ | |
43 | .set noat | 43 | .set noat | |
44 | .set nomacro | 44 | .set nomacro | |
45 | #endif /* _KERNEL_OPT */ | 45 | #endif /* _KERNEL_OPT */ | |
46 | 46 | |||
47 | LEAF(_atomic_swap_32) | 47 | LEAF(_atomic_swap_32) | |
48 | LLSCSYNC | 48 | LLSCSYNC | |
49 | 1: INT_LL v0, 0(a0) | 49 | 1: INT_LL v0, 0(a0) | |
50 | nop | 50 | nop | |
51 | move t0, a1 | 51 | move t0, a1 | |
52 | INT_SC t0, 0(a0) | 52 | INT_SC t0, 0(a0) | |
53 | beq t0, zero, 1b | 53 | beq t0, zero, 1b | |
54 | nop | 54 | nop | |
55 | 2: | 55 | 2: | |
56 | j ra | 56 | j ra | |
57 | SYNCW | 57 | BDSYNC_PLUNGER | |
58 | END(_atomic_swap_32) | 58 | END(_atomic_swap_32) | |
59 | ATOMIC_OP_ALIAS(atomic_swap_32, _atomic_swap_32) | 59 | ATOMIC_OP_ALIAS(atomic_swap_32, _atomic_swap_32) | |
60 | 60 | |||
61 | #if !defined(__mips_o32) | 61 | #if !defined(__mips_o32) | |
62 | LEAF(_atomic_swap_64) | 62 | LEAF(_atomic_swap_64) | |
63 | LLSCSYNC | 63 | LLSCSYNC | |
64 | 1: REG_LL v0, 0(a0) | 64 | 1: REG_LL v0, 0(a0) | |
65 | nop | 65 | nop | |
66 | move t0, a1 | 66 | move t0, a1 | |
67 | REG_SC t0, 0(a0) | 67 | REG_SC t0, 0(a0) | |
68 | beq t0, zero, 1b | 68 | beq t0, zero, 1b | |
69 | nop | 69 | nop | |
70 | 2: | 70 | 2: | |
71 | j ra | 71 | j ra | |
72 | SYNCW | 72 | BDSYNC_PLUNGER | |
73 | END(_atomic_swap_64) | 73 | END(_atomic_swap_64) | |
74 | ATOMIC_OP_ALIAS(atomic_swap_64, _atomic_swap_64) | 74 | ATOMIC_OP_ALIAS(atomic_swap_64, _atomic_swap_64) | |
75 | #endif | 75 | #endif | |
76 | 76 | |||
77 | #ifdef _LP64 | 77 | #ifdef _LP64 | |
78 | STRONG_ALIAS(_atomic_swap_ptr, _atomic_swap_64) | 78 | STRONG_ALIAS(_atomic_swap_ptr, _atomic_swap_64) | |
79 | STRONG_ALIAS(_atomic_swap_ulong, _atomic_swap_64) | 79 | STRONG_ALIAS(_atomic_swap_ulong, _atomic_swap_64) | |
80 | #else | 80 | #else | |
81 | STRONG_ALIAS(_atomic_swap_ptr, _atomic_swap_32) | 81 | STRONG_ALIAS(_atomic_swap_ptr, _atomic_swap_32) | |
82 | STRONG_ALIAS(_atomic_swap_ulong, _atomic_swap_32) | 82 | STRONG_ALIAS(_atomic_swap_ulong, _atomic_swap_32) | |
83 | #endif | 83 | #endif | |
84 | STRONG_ALIAS(_atomic_swap_uint, _atomic_swap_32) | 84 | STRONG_ALIAS(_atomic_swap_uint, _atomic_swap_32) | |
85 | 85 |
--- src/sys/arch/mips/include/asm.h 2021/02/18 12:28:01 1.65
+++ src/sys/arch/mips/include/asm.h 2022/02/27 19:21:53 1.66
@@ -1,14 +1,14 @@ | @@ -1,14 +1,14 @@ | |||
1 | /* $NetBSD: asm.h,v 1.65 2021/02/18 12:28:01 simonb Exp $ */ | 1 | /* $NetBSD: asm.h,v 1.66 2022/02/27 19:21:53 riastradh Exp $ */ | |
2 | 2 | |||
3 | /* | 3 | /* | |
4 | * Copyright (c) 1992, 1993 | 4 | * Copyright (c) 1992, 1993 | |
5 | * The Regents of the University of California. All rights reserved. | 5 | * The Regents of the University of California. All rights reserved. | |
6 | * | 6 | * | |
7 | * This code is derived from software contributed to Berkeley by | 7 | * This code is derived from software contributed to Berkeley by | |
8 | * Ralph Campbell. | 8 | * Ralph Campbell. | |
9 | * | 9 | * | |
10 | * Redistribution and use in source and binary forms, with or without | 10 | * Redistribution and use in source and binary forms, with or without | |
11 | * modification, are permitted provided that the following conditions | 11 | * modification, are permitted provided that the following conditions | |
12 | * are met: | 12 | * are met: | |
13 | * 1. Redistributions of source code must retain the above copyright | 13 | * 1. Redistributions of source code must retain the above copyright | |
14 | * notice, this list of conditions and the following disclaimer. | 14 | * notice, this list of conditions and the following disclaimer. | |
@@ -566,34 +566,49 @@ _C_LABEL(x): | @@ -566,34 +566,49 @@ _C_LABEL(x): | |||
566 | 566 | |||
567 | #if (MIPS1 + MIPS2) > 0 | 567 | #if (MIPS1 + MIPS2) > 0 | |
568 | #define NOP_L nop | 568 | #define NOP_L nop | |
569 | #else | 569 | #else | |
570 | #define NOP_L /* nothing */ | 570 | #define NOP_L /* nothing */ | |
571 | #endif | 571 | #endif | |
572 | 572 | |||
573 | /* compiler define */ | 573 | /* compiler define */ | |
574 | #if defined(__OCTEON__) | 574 | #if defined(__OCTEON__) | |
575 | /* early cnMIPS have erratum which means 2 */ | 575 | /* early cnMIPS have erratum which means 2 */ | |
576 | #define LLSCSYNC sync 4; sync 4 | 576 | #define LLSCSYNC sync 4; sync 4 | |
577 | #define SYNC sync 4 /* sync 4 == syncw - sync all writes */ | 577 | #define SYNC sync 4 /* sync 4 == syncw - sync all writes */ | |
578 | #define BDSYNC sync 4 /* sync 4 == syncw - sync all writes */ | 578 | #define BDSYNC sync 4 /* sync 4 == syncw - sync all writes */ | |
579 | #define BDSYNC_ACQ sync | |||
580 | #define SYNC_ACQ sync | |||
581 | #define SYNC_REL sync | |||
582 | #define BDSYNC_PLUNGER sync 4 | |||
583 | #define SYNC_PLUNGER sync 4 | |||
579 | #elif __mips >= 3 || !defined(__mips_o32) | 584 | #elif __mips >= 3 || !defined(__mips_o32) | |
580 | #define LLSCSYNC sync | 585 | #define LLSCSYNC sync | |
581 | #define SYNC sync | 586 | #define SYNC sync | |
582 | #define BDSYNC sync | 587 | #define BDSYNC sync | |
588 | #define BDSYNC_ACQ sync | |||
589 | #define SYNC_ACQ sync | |||
590 | #define SYNC_REL sync | |||
591 | #define BDSYNC_PLUNGER nop | |||
592 | #define SYNC_PLUNGER /* nothing */ | |||
583 | #else | 593 | #else | |
584 | #define LLSCSYNC /* nothing */ | 594 | #define LLSCSYNC /* nothing */ | |
585 | #define SYNC /* nothing */ | 595 | #define SYNC /* nothing */ | |
586 | #define BDSYNC nop | 596 | #define BDSYNC nop | |
597 | #define BDSYNC_ACQ nop | |||
598 | #define SYNC_ACQ /* nothing */ | |||
599 | #define SYNC_REL /* nothing */ | |||
600 | #define BDSYNC_PLUNGER nop | |||
601 | #define SYNC_PLUNGER /* nothing */ | |||
587 | #endif | 602 | #endif | |
588 | 603 | |||
589 | /* CPU dependent hook for cp0 load delays */ | 604 | /* CPU dependent hook for cp0 load delays */ | |
590 | #if defined(MIPS1) || defined(MIPS2) || defined(MIPS3) | 605 | #if defined(MIPS1) || defined(MIPS2) || defined(MIPS3) | |
591 | #define MFC0_HAZARD sll $0,$0,1 /* super scalar nop */ | 606 | #define MFC0_HAZARD sll $0,$0,1 /* super scalar nop */ | |
592 | #else | 607 | #else | |
593 | #define MFC0_HAZARD /* nothing */ | 608 | #define MFC0_HAZARD /* nothing */ | |
594 | #endif | 609 | #endif | |
595 | 610 | |||
596 | #if _MIPS_ISA == _MIPS_ISA_MIPS1 || _MIPS_ISA == _MIPS_ISA_MIPS2 || \ | 611 | #if _MIPS_ISA == _MIPS_ISA_MIPS1 || _MIPS_ISA == _MIPS_ISA_MIPS2 || \ | |
597 | _MIPS_ISA == _MIPS_ISA_MIPS32 | 612 | _MIPS_ISA == _MIPS_ISA_MIPS32 | |
598 | #define MFC0 mfc0 | 613 | #define MFC0 mfc0 | |
599 | #define MTC0 mtc0 | 614 | #define MTC0 mtc0 |
--- src/sys/arch/mips/mips/lock_stubs_llsc.S 2022/02/27 19:21:44 1.14
+++ src/sys/arch/mips/mips/lock_stubs_llsc.S 2022/02/27 19:21:53 1.15
@@ -1,14 +1,14 @@ | @@ -1,14 +1,14 @@ | |||
1 | /* $NetBSD: lock_stubs_llsc.S,v 1.14 2022/02/27 19:21:44 riastradh Exp $ */ | 1 | /* $NetBSD: lock_stubs_llsc.S,v 1.15 2022/02/27 19:21:53 riastradh Exp $ */ | |
2 | 2 | |||
3 | /*- | 3 | /*- | |
4 | * Copyright (c) 2007 The NetBSD Foundation, Inc. | 4 | * Copyright (c) 2007 The NetBSD Foundation, Inc. | |
5 | * All rights reserved. | 5 | * All rights reserved. | |
6 | * | 6 | * | |
7 | * This code is derived from software contributed to The NetBSD Foundation | 7 | * This code is derived from software contributed to The NetBSD Foundation | |
8 | * by Andrew Doran. | 8 | * by Andrew Doran. | |
9 | * | 9 | * | |
10 | * Redistribution and use in source and binary forms, with or without | 10 | * Redistribution and use in source and binary forms, with or without | |
11 | * modification, are permitted provided that the following conditions | 11 | * modification, are permitted provided that the following conditions | |
12 | * are met: | 12 | * are met: | |
13 | * 1. Redistributions of source code must retain the above copyright | 13 | * 1. Redistributions of source code must retain the above copyright | |
14 | * notice, this list of conditions and the following disclaimer. | 14 | * notice, this list of conditions and the following disclaimer. | |
@@ -26,143 +26,156 @@ | @@ -26,143 +26,156 @@ | |||
26 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN | 26 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN | |
27 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) | 27 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) | |
28 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | 28 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | |
29 | * POSSIBILITY OF SUCH DAMAGE. | 29 | * POSSIBILITY OF SUCH DAMAGE. | |
30 | */ | 30 | */ | |
31 | 31 | |||
32 | #include "opt_cputype.h" | 32 | #include "opt_cputype.h" | |
33 | #include "opt_lockdebug.h" | 33 | #include "opt_lockdebug.h" | |
34 | 34 | |||
35 | #include <sys/errno.h> | 35 | #include <sys/errno.h> | |
36 | 36 | |||
37 | #include <machine/asm.h> | 37 | #include <machine/asm.h> | |
38 | 38 | |||
39 | RCSID("$NetBSD: lock_stubs_llsc.S,v 1.14 2022/02/27 19:21:44 riastradh Exp $") | 39 | RCSID("$NetBSD: lock_stubs_llsc.S,v 1.15 2022/02/27 19:21:53 riastradh Exp $") | |
40 | 40 | |||
41 | #include "assym.h" | 41 | #include "assym.h" | |
42 | 42 | |||
43 | /* | 43 | /* | |
44 | * Set ISA level for the assembler. | 44 | * Set ISA level for the assembler. | |
45 | * XXX Clean up with a macro? Same code fragment is in mipsX_subr.S too. | 45 | * XXX Clean up with a macro? Same code fragment is in mipsX_subr.S too. | |
46 | * XXX Key off build abi instead of processor type? | 46 | * XXX Key off build abi instead of processor type? | |
47 | */ | 47 | */ | |
48 | #if defined(MIPS3) | 48 | #if defined(MIPS3) | |
49 | .set mips3 | 49 | .set mips3 | |
50 | #endif | 50 | #endif | |
51 | 51 | |||
52 | #if defined(MIPS32) | 52 | #if defined(MIPS32) | |
53 | .set mips32 | 53 | .set mips32 | |
54 | #endif | 54 | #endif | |
55 | 55 | |||
56 | #if defined(MIPS64) | 56 | #if defined(MIPS64) | |
57 | .set mips64 | 57 | .set mips64 | |
58 | #endif | 58 | #endif | |
59 | 59 | |||
60 | .set noreorder | 60 | .set noreorder | |
61 | .set noat | 61 | .set noat | |
62 | 62 | |||
63 | /* | 63 | /* | |
64 | * unsigned long atomic_cas_ulong_llsc(volatile unsigned long *val, | 64 | * unsigned long atomic_cas_ulong_llsc(volatile unsigned long *val, | |
65 | * unsigned long old, unsigned long new); | 65 | * unsigned long old, unsigned long new); | |
66 | * | |||
67 | * For hysterical raisins in sys/arch/mips/include/lock.h, success | |||
68 | * implies load-acquire. The SYNC_ACQ here could be moved there | |||
69 | * instead. | |||
66 | */ | 70 | */ | |
67 | STATIC_LEAF(llsc_atomic_cas_ulong) | 71 | STATIC_LEAF(llsc_atomic_cas_ulong) | |
68 | LLSCSYNC | 72 | LLSCSYNC | |
69 | 1: | 73 | 1: | |
70 | LONG_LL t0, (a0) | 74 | LONG_LL t0, (a0) | |
71 | bne t0, a1, 2f | 75 | bne t0, a1, 2f | |
72 | move t1, a2 | 76 | move t1, a2 | |
73 | LONG_SC t1, (a0) | 77 | LONG_SC t1, (a0) | |
74 | beqz t1, 1b | 78 | beqz t1, 1b | |
75 | nop | 79 | nop | |
76 | SYNC | 80 | SYNC_ACQ | |
77 | j ra | 81 | j ra | |
78 | move v0, a1 | 82 | move v0, a1 | |
79 | 2: | 83 | 2: | |
80 | j ra | 84 | j ra | |
81 | move v0, t0 | 85 | move v0, t0 | |
82 | END(llsc_atomic_cas_ulong) | 86 | END(llsc_atomic_cas_ulong) | |
83 | 87 | |||
84 | /* | 88 | /* | |
85 | * unsigned int _atomic_cas_uint_llsc(volatile unsigned int *val, | 89 | * unsigned int _atomic_cas_uint_llsc(volatile unsigned int *val, | |
86 | * unsigned int old, unsigned int new); | 90 | * unsigned int old, unsigned int new); | |
91 | * | |||
92 | * For hysterical raisins in sys/arch/mips/include/lock.h, success | |||
93 | * implies load-acquire. The SYNC_ACQ here could be moved there | |||
94 | * instead. | |||
87 | */ | 95 | */ | |
88 | STATIC_LEAF(llsc_atomic_cas_uint) | 96 | STATIC_LEAF(llsc_atomic_cas_uint) | |
89 | LLSCSYNC | 97 | LLSCSYNC | |
90 | 1: | 98 | 1: | |
91 | INT_LL t0, (a0) | 99 | INT_LL t0, (a0) | |
92 | bne t0, a1, 2f | 100 | bne t0, a1, 2f | |
93 | move t1, a2 | 101 | move t1, a2 | |
94 | INT_SC t1, (a0) | 102 | INT_SC t1, (a0) | |
95 | beqz t1, 1b | 103 | beqz t1, 1b | |
96 | nop | 104 | nop | |
97 | SYNC | 105 | SYNC_ACQ | |
98 | j ra | 106 | j ra | |
99 | move v0, a1 | 107 | move v0, a1 | |
100 | 2: | 108 | 2: | |
101 | j ra | 109 | j ra | |
102 | move v0, t0 | 110 | move v0, t0 | |
103 | END(llsc_atomic_cas_uint) | 111 | END(llsc_atomic_cas_uint) | |
104 | 112 | |||
105 | /* | 113 | /* | |
106 | * int llsc_ucas_32(volatile uint32_t *ptr, uint32_t old, | 114 | * int llsc_ucas_32(volatile uint32_t *ptr, uint32_t old, | |
107 | * uint32_t new, uint32_t *ret) | 115 | * uint32_t new, uint32_t *ret) | |
116 | * | |||
117 | * Implies release/acquire barriers until someone tells me | |||
118 | * otherwise about _ucas_32/64. | |||
108 | */ | 119 | */ | |
109 | STATIC_LEAF(llsc_ucas_32) | 120 | STATIC_LEAF(llsc_ucas_32) | |
110 | .set at | 121 | .set at | |
111 | PTR_LA v0, _C_LABEL(llsc_ucaserr) | 122 | PTR_LA v0, _C_LABEL(llsc_ucaserr) | |
112 | .set noat | 123 | .set noat | |
113 | PTR_L v1, L_PCB(MIPS_CURLWP) | 124 | PTR_L v1, L_PCB(MIPS_CURLWP) | |
114 | PTR_S v0, PCB_ONFAULT(v1) | 125 | PTR_S v0, PCB_ONFAULT(v1) | |
115 | bltz a0, _C_LABEL(llsc_ucaserr) | 126 | bltz a0, _C_LABEL(llsc_ucaserr) | |
116 | nop | 127 | nop | |
117 | move v0, zero | 128 | move v0, zero | |
129 | SYNC_REL | |||
118 | 130 | |||
119 | LLSCSYNC | 131 | LLSCSYNC | |
120 | 1: ll t0, 0(a0) | 132 | 1: ll t0, 0(a0) | |
121 | bne t0, a1, 2f | 133 | bne t0, a1, 2f | |
122 | move t1, a2 | 134 | move t1, a2 | |
123 | sc t1, 0(a0) | 135 | sc t1, 0(a0) | |
124 | beqz t1, 1b | 136 | beqz t1, 1b | |
125 | nop | 137 | nop | |
126 | SYNC | 138 | SYNC_ACQ | |
127 | 139 | |||
128 | 2: PTR_S zero, PCB_ONFAULT(v1) | 140 | 2: PTR_S zero, PCB_ONFAULT(v1) | |
129 | j ra | 141 | j ra | |
130 | sw t0, 0(a3) | 142 | sw t0, 0(a3) | |
131 | END(llsc_ucas_32) | 143 | END(llsc_ucas_32) | |
132 | 144 | |||
133 | #ifdef _LP64 | 145 | #ifdef _LP64 | |
134 | /* | 146 | /* | |
135 | * int llsc_ucas_64(volatile uint64_t *ptr, uint64_t old, | 147 | * int llsc_ucas_64(volatile uint64_t *ptr, uint64_t old, | |
136 | * uint64_t new, uint64_t *ret) | 148 | * uint64_t new, uint64_t *ret) | |
137 | */ | 149 | */ | |
138 | STATIC_LEAF(llsc_ucas_64) | 150 | STATIC_LEAF(llsc_ucas_64) | |
139 | .set at | 151 | .set at | |
140 | PTR_LA v0, _C_LABEL(llsc_ucaserr) | 152 | PTR_LA v0, _C_LABEL(llsc_ucaserr) | |
141 | .set noat | 153 | .set noat | |
142 | PTR_L v1, L_PCB(MIPS_CURLWP) | 154 | PTR_L v1, L_PCB(MIPS_CURLWP) | |
143 | PTR_S v0, PCB_ONFAULT(v1) | 155 | PTR_S v0, PCB_ONFAULT(v1) | |
144 | bltz a0, _C_LABEL(llsc_ucaserr) | 156 | bltz a0, _C_LABEL(llsc_ucaserr) | |
145 | nop | 157 | nop | |
146 | move v0, zero | 158 | move v0, zero | |
159 | SYNC_REL | |||
147 | 160 | |||
148 | LLSCSYNC | 161 | LLSCSYNC | |
149 | 1: lld t0, 0(a0) | 162 | 1: lld t0, 0(a0) | |
150 | bne t0, a1, 2f | 163 | bne t0, a1, 2f | |
151 | move t1, a2 | 164 | move t1, a2 | |
152 | scd t1, 0(a0) | 165 | scd t1, 0(a0) | |
153 | beqz t1, 1b | 166 | beqz t1, 1b | |
154 | nop | 167 | nop | |
155 | SYNC | 168 | SYNC_ACQ | |
156 | 169 | |||
157 | 2: PTR_S zero, PCB_ONFAULT(v1) | 170 | 2: PTR_S zero, PCB_ONFAULT(v1) | |
158 | j ra | 171 | j ra | |
159 | sd t0, 0(a3) | 172 | sd t0, 0(a3) | |
160 | END(llsc_ucas_64) | 173 | END(llsc_ucas_64) | |
161 | #endif /* _LP64 */ | 174 | #endif /* _LP64 */ | |
162 | 175 | |||
163 | STATIC_LEAF_NOPROFILE(llsc_ucaserr) | 176 | STATIC_LEAF_NOPROFILE(llsc_ucaserr) | |
164 | PTR_S zero, PCB_ONFAULT(v1) # reset fault handler | 177 | PTR_S zero, PCB_ONFAULT(v1) # reset fault handler | |
165 | j ra | 178 | j ra | |
166 | li v0, EFAULT # return EFAULT on error | 179 | li v0, EFAULT # return EFAULT on error | |
167 | END(llsc_ucaserr) | 180 | END(llsc_ucaserr) | |
168 | 181 | |||
@@ -171,47 +184,48 @@ END(llsc_ucaserr) | @@ -171,47 +184,48 @@ END(llsc_ucaserr) | |||
171 | /* | 184 | /* | |
172 | * void mutex_enter(kmutex_t *mtx); | 185 | * void mutex_enter(kmutex_t *mtx); | |
173 | */ | 186 | */ | |
174 | STATIC_LEAF(llsc_mutex_enter) | 187 | STATIC_LEAF(llsc_mutex_enter) | |
175 | LLSCSYNC | 188 | LLSCSYNC | |
176 | PTR_LL t0, MTX_OWNER(a0) | 189 | PTR_LL t0, MTX_OWNER(a0) | |
177 | 1: | 190 | 1: | |
178 | bnez t0, 2f | 191 | bnez t0, 2f | |
179 | move t2, MIPS_CURLWP | 192 | move t2, MIPS_CURLWP | |
180 | PTR_SC t2, MTX_OWNER(a0) | 193 | PTR_SC t2, MTX_OWNER(a0) | |
181 | beqz t2, 1b | 194 | beqz t2, 1b | |
182 | PTR_LL t0, MTX_OWNER(a0) | 195 | PTR_LL t0, MTX_OWNER(a0) | |
183 | j ra | 196 | j ra | |
184 | BDSYNC | 197 | BDSYNC_ACQ | |
185 | 2: | 198 | 2: | |
186 | j _C_LABEL(mutex_vector_enter) | 199 | j _C_LABEL(mutex_vector_enter) | |
187 | nop | 200 | nop | |
188 | END(llsc_mutex_enter) | 201 | END(llsc_mutex_enter) | |
189 | 202 | |||
190 | /* | 203 | /* | |
191 | * void mutex_exit(kmutex_t *mtx); | 204 | * void mutex_exit(kmutex_t *mtx); | |
192 | */ | 205 | */ | |
193 | STATIC_LEAF(llsc_mutex_exit) | 206 | STATIC_LEAF(llsc_mutex_exit) | |
207 | SYNC_REL | |||
194 | LLSCSYNC | 208 | LLSCSYNC | |
195 | PTR_LL t0, MTX_OWNER(a0) | 209 | PTR_LL t0, MTX_OWNER(a0) | |
196 | SYNC | 210 | SYNC | |
197 | 1: | 211 | 1: | |
198 | bne t0, MIPS_CURLWP, 2f | 212 | bne t0, MIPS_CURLWP, 2f | |
199 | move t2, zero | 213 | move t2, zero | |
200 | PTR_SC t2, MTX_OWNER(a0) | 214 | PTR_SC t2, MTX_OWNER(a0) | |
201 | beqz t2, 1b | 215 | beqz t2, 1b | |
202 | PTR_LL t0, MTX_OWNER(a0) | 216 | PTR_LL t0, MTX_OWNER(a0) | |
203 | j ra | 217 | j ra | |
204 | BDSYNC | 218 | BDSYNC_PLUNGER | |
205 | 2: | 219 | 2: | |
206 | j _C_LABEL(mutex_vector_exit) | 220 | j _C_LABEL(mutex_vector_exit) | |
207 | nop | 221 | nop | |
208 | END(llsc_mutex_exit) | 222 | END(llsc_mutex_exit) | |
209 | 223 | |||
210 | /* | 224 | /* | |
211 | * void mutex_spin_enter(kmutex_t *mtx); | 225 | * void mutex_spin_enter(kmutex_t *mtx); | |
212 | */ | 226 | */ | |
213 | STATIC_NESTED(llsc_mutex_spin_enter, CALLFRAME_SIZ, ra) | 227 | STATIC_NESTED(llsc_mutex_spin_enter, CALLFRAME_SIZ, ra) | |
214 | move t0, a0 | 228 | move t0, a0 | |
215 | PTR_L t2, L_CPU(MIPS_CURLWP) | 229 | PTR_L t2, L_CPU(MIPS_CURLWP) | |
216 | INT_L a0, MTX_IPL(t0) | 230 | INT_L a0, MTX_IPL(t0) | |
217 | #ifdef PARANOIA | 231 | #ifdef PARANOIA | |
@@ -254,36 +268,37 @@ STATIC_NESTED(llsc_mutex_spin_enter, CAL | @@ -254,36 +268,37 @@ STATIC_NESTED(llsc_mutex_spin_enter, CAL | |||
254 | 12: bnez v0, 12b # loop forever if any are true | 268 | 12: bnez v0, 12b # loop forever if any are true | |
255 | nop | 269 | nop | |
256 | #endif /* PARANOIA */ | 270 | #endif /* PARANOIA */ | |
257 | 271 | |||
258 | LLSCSYNC | 272 | LLSCSYNC | |
259 | INT_LL t3, MTX_LOCK(t0) | 273 | INT_LL t3, MTX_LOCK(t0) | |
260 | 3: | 274 | 3: | |
261 | bnez t3, 4f | 275 | bnez t3, 4f | |
262 | li t1, 1 | 276 | li t1, 1 | |
263 | INT_SC t1, MTX_LOCK(t0) | 277 | INT_SC t1, MTX_LOCK(t0) | |
264 | beqz t1, 3b | 278 | beqz t1, 3b | |
265 | INT_LL t3, MTX_LOCK(t0) | 279 | INT_LL t3, MTX_LOCK(t0) | |
266 | j ra | 280 | j ra | |
267 | BDSYNC | 281 | BDSYNC_ACQ | |
268 | 4: | 282 | 4: | |
269 | j _C_LABEL(mutex_spin_retry) | 283 | j _C_LABEL(mutex_spin_retry) | |
270 | move a0, t0 | 284 | move a0, t0 | |
271 | END(llsc_mutex_spin_enter) | 285 | END(llsc_mutex_spin_enter) | |
272 | 286 | |||
273 | /* | 287 | /* | |
274 | * void mutex_spin_exit(kmutex_t *mtx); | 288 | * void mutex_spin_exit(kmutex_t *mtx); | |
275 | */ | 289 | */ | |
276 | LEAF(llsc_mutex_spin_exit) | 290 | LEAF(llsc_mutex_spin_exit) | |
291 | SYNC_REL | |||
277 | PTR_L t2, L_CPU(MIPS_CURLWP) | 292 | PTR_L t2, L_CPU(MIPS_CURLWP) | |
278 | #if defined(DIAGNOSTIC) | 293 | #if defined(DIAGNOSTIC) | |
279 | INT_L t0, MTX_LOCK(a0) | 294 | INT_L t0, MTX_LOCK(a0) | |
280 | beqz t0, 2f | 295 | beqz t0, 2f | |
281 | nop | 296 | nop | |
282 | #endif | 297 | #endif | |
283 | INT_S zero, MTX_LOCK(a0) | 298 | INT_S zero, MTX_LOCK(a0) | |
284 | 299 | |||
285 | /* | 300 | /* | |
286 | * We need to grab this before the mutex count is incremented | 301 | * We need to grab this before the mutex count is incremented | |
287 | * because if we get an interrupt, it may see the count as zero | 302 | * because if we get an interrupt, it may see the count as zero | |
288 | * and overwrite the oldspl value with a bogus value. | 303 | * and overwrite the oldspl value with a bogus value. | |
289 | */ | 304 | */ |