From 04e2ed4a9d6056798943183dbb334d502c827a83 Mon Sep 17 00:00:00 2001 From: Kartik Date: Wed, 24 Aug 2022 10:47:03 +0000 Subject: [PATCH] NVIDIA: SAUCE: locking/rtmutex: use cmpxchg in mark_rt_mutex_waiters BugLink: https://bugs.launchpad.net/bugs/2072591 Function `mark_rt_mutex_waiters` is used by lock waiters to intentionally mark that the lock has waiters before trying to acquire the lock. By doing so it serializes all the waiters on `wait_lock`. `wait_lock` is implemented using raw_spinlock, which does not provide any memory barriers. If the system is stressed enough it is possible that `RT_MUTEX_HAS_WAITERS` is updated later in the cache. This makes it possible for the new waiter to acquire the mutex in the fast path, even though there are waitiers in queue. CPU0 CPU1 lock(wait_lock) __try_to_take_rt_mutex { mark_rt_mutex_waiters lock->owner != NULL { return rt_mutex_slowunlock { } } unlock(wait_lock) lock(wait_lock) !rt_mutex_has_waiters(lock) { unlock(mutex_lock) // RT_MUTEX_HAS_WAITERS flag owner->NULL // Updated in cache after this } // check } To fix this use `cmpxchg` instead of `cmpxchg_relaxed` in `mark_rt_mutex_waiters`. `cmpxchg` includes memory barriers to make sure `RT_MUTEX_HAS_WAITERS` is updated before the mutex is release in fast unlock path. http://nvbugs/4015141 Signed-off-by: Kartik Tested-by: Tonny Liang Tested-by: Abhilash G Reviewed-by: Abhilash G Signed-off-by: Laxman Dewangan Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Noah Wager --- kernel/locking/rtmutex.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index 99129c89120a..f355d5dda068 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -241,7 +241,7 @@ static __always_inline void mark_rt_mutex_waiters(struct rt_mutex_base *lock) do { owner = *p; - } while (cmpxchg_relaxed(p, owner, + } while (cmpxchg(p, owner, owner | RT_MUTEX_HAS_WAITERS) != owner); /*