diff --git a/Documentation/dev-tools/context-analysis.rst b/Documentation/dev-tools/context-analysis.rst index 54d9ee28de98..8e71e1e75b5b 100644 --- a/Documentation/dev-tools/context-analysis.rst +++ b/Documentation/dev-tools/context-analysis.rst @@ -17,7 +17,7 @@ features. To enable for Clang, configure the kernel with:: CONFIG_WARN_CONTEXT_ANALYSIS=y -The feature requires Clang 22 or later. +The feature requires Clang 23 or later. The analysis is *opt-in by default*, and requires declaring which modules and subsystems should be analyzed in the respective `Makefile`:: diff --git a/Documentation/locking/robust-futex-ABI.rst b/Documentation/locking/robust-futex-ABI.rst index f24904f1c16f..5e6a0665b8ba 100644 --- a/Documentation/locking/robust-futex-ABI.rst +++ b/Documentation/locking/robust-futex-ABI.rst @@ -153,6 +153,9 @@ On removal: 3) release the futex lock, and 4) clear the 'lock_op_pending' word. +Please note that the removal of a robust futex purely in userspace is +racy. Refer to the next chapter to learn more and how to avoid this. + On exit, the kernel will consider the address stored in 'list_op_pending' and the address of each 'lock word' found by walking the list starting at 'head'. For each such address, if the bottom 30 @@ -182,3 +185,44 @@ any point: When the kernel sees a list entry whose 'lock word' doesn't have the current threads TID in the lower 30 bits, it does nothing with that entry, and goes on to the next entry. + +Robust release is racy +---------------------- + +The removal of a robust futex from the list is racy when doing it solely in +userspace. Quoting Thomas Gleixner for the explanation: + + The robust futex unlock mechanism is racy in respect to the clearing of the + robust_list_head::list_op_pending pointer because unlock and clearing the + pointer are not atomic. The race window is between the unlock and clearing + the pending op pointer. If the task is forced to exit in this window, exit + will access a potentially invalid pending op pointer when cleaning up the + robust list. That happens if another task manages to unmap the object + containing the lock before the cleanup, which results in an UAF. In the + worst case this UAF can lead to memory corruption when unrelated content + has been mapped to the same address by the time the access happens. + +A full in-depth analysis can be read at +https://lore.kernel.org/lkml/20260316162316.356674433@kernel.org/ + +To overcome that, the kernel needs to participate in the lock release operation. +This ensures that the release happens "atomically" with regard to releasing +the lock and removing the address from ``list_op_pending``. If the release is +interrupted by a signal, the kernel will also verify if it interrupted the +release operation. + +For the contended unlock case, where other threads are waiting for the lock +release, there's the ``FUTEX_ROBUST_UNLOCK`` operation feature flag for the +``futex()`` system call, which must be used with one of the following +operations: ``FUTEX_WAKE``, ``FUTEX_WAKE_BITSET`` or ``FUTEX_UNLOCK_PI``. +The kernel will release the lock (set the futex word to zero), clean the +``list_op_pending`` field. Then, it will proceed with the normal wake path. + +For the non-contended path, there's still a race between checking the futex word +and clearing the ``list_op_pending`` field. To solve this without the need of a +complete system call, userspace should call the virtual syscall +``__vdso_futex_robust_listXX_try_unlock()`` (where XX is either 32 or 64, +depending on the size of the pointer). If the vDSO call succeeds, it means that +it released the lock and cleared ``list_op_pending``. If it fails, that means +that there are waiters for this lock and a call to ``futex()`` syscall with +``FUTEX_ROBUST_UNLOCK`` is needed. diff --git a/Documentation/locking/robust-futexes.rst b/Documentation/locking/robust-futexes.rst index 6361fb01c9c1..1423f53ea2f4 100644 --- a/Documentation/locking/robust-futexes.rst +++ b/Documentation/locking/robust-futexes.rst @@ -94,7 +94,7 @@ time, the kernel checks this user-space list: are there any robust futex locks to be cleaned up? In the common case, at do_exit() time, there is no list registered, so -the cost of robust futexes is just a simple current->robust_list != NULL +the cost of robust futexes is just a current->futex.robust_list != NULL comparison. If the thread has registered a list, then normally the list is empty. If the thread/process crashed or terminated in some incorrect way then the list might be non-empty: in this case the kernel carefully @@ -178,9 +178,9 @@ one to query the registered list pointer:: size_t __user *len_ptr); List registration is very fast: the pointer is simply stored in -current->robust_list. [Note that in the future, if robust futexes become -widespread, we could extend sys_clone() to register a robust-list head -for new threads, without the need of another syscall.] +current->futex.robust_list. [Note that in the future, if robust futexes +become widespread, we could extend sys_clone() to register a robust-list +head for new threads, without the need of another syscall.] So there is virtually zero overhead for tasks not using robust futexes, and even for robust futex users, there is only one extra syscall per diff --git a/MAINTAINERS b/MAINTAINERS index e1c67f4d0663..13ffda065439 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -10873,6 +10873,7 @@ L: linux-kernel@vger.kernel.org S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/vdso F: include/asm-generic/vdso/vsyscall.h +F: include/linux/vdso_datastore.h F: include/vdso/ F: kernel/time/namespace_vdso.c F: kernel/time/vsyscall.c @@ -23489,6 +23490,20 @@ S: Maintained T: git https://github.com/Rust-for-Linux/linux.git rust-analyzer-next F: scripts/generate_rust_analyzer.py +RUST [SYNC] +M: Boqun Feng +M: Gary Guo +M: Alice Ryhl +R: Lyude Paul +R: Daniel Almeida +R: "Onur Özkan" +L: linux-kernel@vger.kernel.org +L: rust-for-linux@vger.kernel.org +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/boqun/linux.git rust-sync +F: rust/kernel/sync.rs +F: rust/kernel/sync/ + RXRPC SOCKETS (AF_RXRPC) M: David Howells M: Marc Dionne diff --git a/arch/Kconfig b/arch/Kconfig index 99c2017eb515..fa7507ac8e13 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -403,6 +403,10 @@ config ARCH_32BIT_OFF_T config ARCH_32BIT_USTAT_F_TINODE bool +# Selected by architectures with Total Store Order (TSO) +config ARCH_MEMORY_ORDER_TSO + bool + config HAVE_ASM_MODVERSIONS bool help diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index 7b27ee9482b3..871bd58d2ccc 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -925,7 +925,6 @@ config VDSO depends on AEABI && MMU && CPU_V7 default y if ARM_ARCH_TIMER select HAVE_GENERIC_VDSO - select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY help Place in the process address space an ELF shared object diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 387db357b208..b0f3098c0388 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -140,7 +140,6 @@ config ARM64 select GENERIC_PCI_IOMAP select GENERIC_SCHED_CLOCK select GENERIC_SMP_IDLE_THREAD - select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY select HARDIRQS_SW_RESEND select HAS_IOPORT diff --git a/arch/arm64/kvm/hyp_trace.c b/arch/arm64/kvm/hyp_trace.c index c4b3ee552131..2411b4c32932 100644 --- a/arch/arm64/kvm/hyp_trace.c +++ b/arch/arm64/kvm/hyp_trace.c @@ -51,8 +51,8 @@ static void __hyp_clock_work(struct work_struct *work) hyp_clock = container_of(dwork, struct hyp_trace_clock, work); - ktime_get_snapshot(&snap); - boot = ktime_to_ns(snap.boot); + ktime_get_snapshot_id(CLOCK_BOOTTIME, &snap); + boot = ktime_to_ns(snap.systime); delta_boot = boot - hyp_clock->boot; delta_cycles = snap.cycles - hyp_clock->cycles; @@ -118,9 +118,9 @@ static void hyp_trace_clock_enable(struct hyp_trace_clock *hyp_clock, bool enabl hyp_clock->running = false; } - ktime_get_snapshot(&snap); + ktime_get_snapshot_id(CLOCK_BOOTTIME, &snap); - hyp_clock->boot = ktime_to_ns(snap.boot); + hyp_clock->boot = ktime_to_ns(snap.systime); hyp_clock->cycles = snap.cycles; hyp_clock->mult = 0; diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c index 58c5fe7d7572..b11b8821c9fb 100644 --- a/arch/arm64/kvm/hypercalls.c +++ b/arch/arm64/kvm/hypercalls.c @@ -28,7 +28,7 @@ static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val) * system time and counter value must captured at the same * time to keep consistency and precision. */ - ktime_get_snapshot(&systime_snapshot); + ktime_get_snapshot_id(CLOCK_REALTIME, &systime_snapshot); /* * This is only valid if the current clocksource is the @@ -61,8 +61,8 @@ static void kvm_ptp_get_time(struct kvm_vcpu *vcpu, u64 *val) * in the future (about 292 years from 1970, and at that stage * nobody will give a damn about it). */ - val[0] = upper_32_bits(systime_snapshot.real); - val[1] = lower_32_bits(systime_snapshot.real); + val[0] = upper_32_bits(systime_snapshot.systime); + val[1] = lower_32_bits(systime_snapshot.systime); val[2] = upper_32_bits(cycles); val[3] = lower_32_bits(cycles); } diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 606597da46b8..3f69c5d7e48e 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -110,7 +110,6 @@ config LOONGARCH select GENERIC_PCI_IOMAP select GENERIC_SCHED_CLOCK select GENERIC_SMP_IDLE_THREAD - select GENERIC_TIME_VSYSCALL if GENERIC_GETTIMEOFDAY select GPIOLIB select HAS_IOPORT select HAVE_ALIGNED_STRUCT_PAGE if 64BIT diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index 4364f3dba688..323ca084e79a 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -38,7 +38,6 @@ config MIPS select GENERIC_BUILTIN_DTB if BUILTIN_DTB select GENERIC_CMOS_UPDATE select GENERIC_CPU_AUTOPROBE - select GENERIC_GETTIMEOFDAY select GENERIC_IRQ_PROBE select GENERIC_IRQ_SHOW select GENERIC_ISA_DMA if EISA @@ -51,7 +50,6 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_IDLE_POLL_SETUP - select GENERIC_TIME_VSYSCALL select GUP_GET_PXX_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HAS_IOPORT if !NO_IOPORT_MAP || ISA select HAVE_ARCH_COMPILER_H @@ -76,7 +74,6 @@ config MIPS select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS - select HAVE_GENERIC_VDSO select HAVE_IOREMAP_PROT select HAVE_IRQ_EXIT_ON_IRQ_STACK select HAVE_IRQ_TIME_ACCOUNTING @@ -1136,9 +1133,6 @@ config CSRC_R4K config CSRC_SB1250 bool -config MIPS_CLOCK_VSYSCALL - def_bool CSRC_R4K || CLKSRC_MIPS_GIC - config GPIO_TXX9 select GPIOLIB bool @@ -3170,6 +3164,16 @@ endmenu config MIPS_EXTERNAL_TIMER bool +config MIPS_GENERIC_GETTIMEOFDAY + def_bool y + select GENERIC_GETTIMEOFDAY + select HAVE_GENERIC_VDSO + depends on CSRC_R4K || CLKSRC_MIPS_GIC + # GCC (at least up to version 9.2) appears to emit function calls that make use + # of the GOT when targeting microMIPS, which we can't use in the VDSO due to + # the lack of relocations. As such, we disable the VDSO for microMIPS builds. + depends on !(CPU_MICROMIPS && CC_IS_GCC && GCC_VERSION < 90300) + menu "CPU Power Management" if CPU_SUPPORTS_CPUFREQ && MIPS_EXTERNAL_TIMER @@ -3181,5 +3185,3 @@ source "drivers/cpuidle/Kconfig" endmenu source "arch/mips/kvm/Kconfig" - -source "arch/mips/vdso/Kconfig" diff --git a/arch/mips/kernel/csrc-r4k.c b/arch/mips/kernel/csrc-r4k.c index 59eca397f297..241a934543a8 100644 --- a/arch/mips/kernel/csrc-r4k.c +++ b/arch/mips/kernel/csrc-r4k.c @@ -126,12 +126,14 @@ int __init init_r4k_clocksource(void) clocksource_mips.rating = 200; clocksource_mips.rating += clamp(mips_hpt_frequency / 10000000, 0, 99); +#ifdef CONFIG_GENERIC_GETTIMEOFDAY /* * R2 onwards makes the count accessible to user mode so it can be used * by the VDSO (HWREna is configured by configure_hwrena()). */ if (cpu_has_mips_r2_r6 && rdhwr_count_usable()) clocksource_mips.vdso_clock_mode = VDSO_CLOCKMODE_R4K; +#endif clocksource_register_hz(&clocksource_mips, mips_hpt_frequency); diff --git a/arch/mips/kernel/vdso.c b/arch/mips/kernel/vdso.c index 2fa4df3e46e4..bd1fc17d3975 100644 --- a/arch/mips/kernel/vdso.c +++ b/arch/mips/kernel/vdso.c @@ -129,7 +129,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) * This ensures that when the kernel updates the VDSO data userland * will observe it without requiring cache invalidations. */ - if (cpu_has_dc_aliases) { + if (cpu_has_dc_aliases && IS_ENABLED(CONFIG_HAVE_GENERIC_VDSO)) { base = __ALIGN_MASK(base, shm_align_mask); base += ((unsigned long)vdso_k_time_data - gic_size) & shm_align_mask; } @@ -137,10 +137,12 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) data_addr = base + gic_size; vdso_addr = data_addr + VDSO_NR_PAGES * PAGE_SIZE; - vma = vdso_install_vvar_mapping(mm, data_addr); - if (IS_ERR(vma)) { - ret = PTR_ERR(vma); - goto out; + if (IS_ENABLED(CONFIG_HAVE_GENERIC_VDSO)) { + vma = vdso_install_vvar_mapping(mm, data_addr); + if (IS_ERR(vma)) { + ret = PTR_ERR(vma); + goto out; + } } /* Map GIC user page. */ diff --git a/arch/mips/vdso/Kconfig b/arch/mips/vdso/Kconfig deleted file mode 100644 index 70140248da72..000000000000 --- a/arch/mips/vdso/Kconfig +++ /dev/null @@ -1,6 +0,0 @@ -# GCC (at least up to version 9.2) appears to emit function calls that make use -# of the GOT when targeting microMIPS, which we can't use in the VDSO due to -# the lack of relocations. As such, we disable the VDSO for microMIPS builds. - -config MIPS_DISABLE_VDSO - def_bool CPU_MICROMIPS diff --git a/arch/mips/vdso/Makefile b/arch/mips/vdso/Makefile index 69d4593f64fe..00d3ba2c482a 100644 --- a/arch/mips/vdso/Makefile +++ b/arch/mips/vdso/Makefile @@ -4,7 +4,7 @@ # Include the generic Makefile to check the built vdso. include $(srctree)/lib/vdso/Makefile.include -obj-vdso-y := elf.o vgettimeofday.o sigreturn.o +obj-vdso-y := elf.o sigreturn.o # Common compiler flags between ABIs. ccflags-vdso := \ @@ -36,6 +36,7 @@ aflags-vdso := $(ccflags-vdso) \ -D__ASSEMBLY__ -Wa,-gdwarf-2 ifneq ($(c-gettimeofday-y),) +obj-vdso-y += vgettimeofday.o CFLAGS_vgettimeofday.o = -include $(c-gettimeofday-y) # config-n32-o32-env.c prepares the environment to build a 32bit vDSO @@ -47,10 +48,6 @@ endif CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -ifdef CONFIG_MIPS_DISABLE_VDSO - obj-vdso-y := $(filter-out vgettimeofday.o, $(obj-vdso-y)) -endif - # VDSO linker flags. ldflags-y := -Bsymbolic --no-undefined -soname=linux-vdso.so.1 \ $(filter -E%,$(KBUILD_CFLAGS)) -shared \ diff --git a/arch/mips/vdso/vdso.lds.S b/arch/mips/vdso/vdso.lds.S index 5d08be3a6b85..05badf3ae0ff 100644 --- a/arch/mips/vdso/vdso.lds.S +++ b/arch/mips/vdso/vdso.lds.S @@ -94,12 +94,10 @@ PHDRS VERSION { LINUX_2.6 { -#ifndef CONFIG_MIPS_DISABLE_VDSO +#ifdef CONFIG_GENERIC_GETTIMEOFDAY global: __vdso_clock_gettime; -#ifdef CONFIG_MIPS_CLOCK_VSYSCALL __vdso_gettimeofday; -#endif __vdso_clock_getres; #if _MIPS_SIM != _MIPS_SIM_ABI64 __vdso_clock_gettime64; diff --git a/arch/mips/vdso/vgettimeofday.c b/arch/mips/vdso/vgettimeofday.c index 1d236215e8f6..00f9fcfc327e 100644 --- a/arch/mips/vdso/vgettimeofday.c +++ b/arch/mips/vdso/vgettimeofday.c @@ -18,22 +18,12 @@ int __vdso_clock_gettime(clockid_t clock, return __cvdso_clock_gettime32(clock, ts); } -#ifdef CONFIG_MIPS_CLOCK_VSYSCALL - -/* - * This is behind the ifdef so that we don't provide the symbol when there's no - * possibility of there being a usable clocksource, because there's nothing we - * can do without it. When libc fails the symbol lookup it should fall back on - * the standard syscall path. - */ int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { return __cvdso_gettimeofday(tv, tz); } -#endif /* CONFIG_MIPS_CLOCK_VSYSCALL */ - int __vdso_clock_getres(clockid_t clock_id, struct old_timespec32 *res) { @@ -59,22 +49,12 @@ int __vdso_clock_gettime(clockid_t clock, return __cvdso_clock_gettime(clock, ts); } -#ifdef CONFIG_MIPS_CLOCK_VSYSCALL - -/* - * This is behind the ifdef so that we don't provide the symbol when there's no - * possibility of there being a usable clocksource, because there's nothing we - * can do without it. When libc fails the symbol lookup it should fall back on - * the standard syscall path. - */ int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { return __cvdso_gettimeofday(tv, tz); } -#endif /* CONFIG_MIPS_CLOCK_VSYSCALL */ - int __vdso_clock_getres(clockid_t clock_id, struct __kernel_timespec *res) { diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index e93df95b79e7..c99fd8335ddc 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -213,7 +213,6 @@ config PPC select GENERIC_IRQ_SHOW_LEVEL select GENERIC_PCI_IOMAP if PCI select GENERIC_SMP_IDLE_THREAD - select GENERIC_TIME_VSYSCALL select HAS_IOPORT if PCI select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c index b4472288e0d4..3460d1a5a97c 100644 --- a/arch/powerpc/kernel/time.c +++ b/arch/powerpc/kernel/time.c @@ -376,6 +376,47 @@ void vtime_task_switch(struct task_struct *prev) acct->starttime = acct0->starttime; } } + +#ifdef CONFIG_NO_HZ_COMMON +/** + * vtime_reset - Fast forward vtime entry clocks + * + * Called from dynticks idle IRQ entry to fast-forward the clocks to current time + * so that the IRQ time is still accounted by vtime while nohz cputime is paused. + */ +void vtime_reset(void) +{ + struct cpu_accounting_data *acct = get_accounting(current); + + acct->starttime = mftb(); +#ifdef CONFIG_ARCH_HAS_SCALED_CPUTIME + acct->startspurr = read_spurr(acct->starttime); +#endif +} + +/** + * vtime_dyntick_start - Inform vtime about entry to idle-dynticks + * + * Called when idle enters in dyntick mode. The idle cputime that elapsed so far + * is accumulated and the tick subsystem takes over the idle cputime accounting. + */ +void vtime_dyntick_start(void) +{ + vtime_account_idle(current); +} + +/** + * vtime_dyntick_stop - Inform vtime about exit from idle-dynticks + * + * Called when idle exits from dyntick mode. The vtime entry clocks are + * fast-forward to current time so that idle accounting restarts elapsing from + * now. + */ +void vtime_dyntick_stop(void) +{ + vtime_reset(); +} +#endif /* CONFIG_NO_HZ_COMMON */ #endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ void __no_kcsan __delay(unsigned long loops) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 1a2fadccd4c4..a25830da9640 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -122,7 +122,6 @@ config RISCV select GENERIC_PCI_IOMAP select GENERIC_SCHED_CLOCK select GENERIC_SMP_IDLE_THREAD - select GENERIC_TIME_VSYSCALL if GENERIC_GETTIMEOFDAY select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU select HAVE_ALIGNED_STRUCT_PAGE diff --git a/arch/riscv/include/asm/vdso/gettimeofday.h b/arch/riscv/include/asm/vdso/gettimeofday.h index 9ec08fa04d35..61cb3cbab143 100644 --- a/arch/riscv/include/asm/vdso/gettimeofday.h +++ b/arch/riscv/include/asm/vdso/gettimeofday.h @@ -9,12 +9,6 @@ #include #include -/* - * 32-bit land is lacking generic time vsyscalls as well as the legacy 32-bit - * time syscalls like gettimeofday. Skip these definitions since on 32-bit. - */ -#ifdef CONFIG_GENERIC_TIME_VSYSCALL - #define VDSO_HAS_CLOCK_GETRES 1 static __always_inline @@ -66,8 +60,6 @@ int clock_getres_fallback(clockid_t _clkid, struct __kernel_timespec *_ts) return ret; } -#endif /* CONFIG_GENERIC_TIME_VSYSCALL */ - static __always_inline u64 __arch_get_hw_counter(s32 clock_mode, const struct vdso_time_data *vd) { diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 9921a3772bf7..81185b19e106 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -174,7 +174,6 @@ config S390 select GENERIC_ENTRY select GENERIC_GETTIMEOFDAY select GENERIC_SMP_IDLE_THREAD - select GENERIC_TIME_VSYSCALL select GENERIC_IOREMAP if PCI select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL diff --git a/arch/s390/include/asm/idle.h b/arch/s390/include/asm/idle.h index 32536ee34aa0..e4ad09a22400 100644 --- a/arch/s390/include/asm/idle.h +++ b/arch/s390/include/asm/idle.h @@ -8,10 +8,12 @@ #ifndef _S390_IDLE_H #define _S390_IDLE_H +#include #include #include struct s390_idle_data { + bool idle_dyntick; unsigned long idle_count; unsigned long idle_time; unsigned long clock_idle_enter; diff --git a/arch/s390/kernel/idle.c b/arch/s390/kernel/idle.c index 1f1b06b6b4ef..4685d7c5bc51 100644 --- a/arch/s390/kernel/idle.c +++ b/arch/s390/kernel/idle.c @@ -31,7 +31,10 @@ void account_idle_time_irq(void) /* Account time spent with enabled wait psw loaded as idle time. */ __atomic64_add(idle_time, &idle->idle_time); __atomic64_add_const(1, &idle->idle_count); - account_idle_time(cputime_to_nsecs(idle_time)); + + /* Dyntick idle time accounted by nohz/scheduler */ + if (!idle->idle_dyntick) + account_idle_time(cputime_to_nsecs(idle_time)); } void noinstr arch_cpu_idle(void) diff --git a/arch/s390/kernel/vtime.c b/arch/s390/kernel/vtime.c index bf48744d0912..d1102a6f80bd 100644 --- a/arch/s390/kernel/vtime.c +++ b/arch/s390/kernel/vtime.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include "entry.h" @@ -110,6 +111,16 @@ static void account_system_index_scaled(struct task_struct *p, u64 cputime, account_system_index_time(p, cputime_to_nsecs(cputime), index); } +static inline void vtime_reset_last_update(struct lowcore *lc) +{ + asm volatile( + " stpt %0\n" /* Store current cpu timer value */ + " stckf %1" /* Store current tod clock value */ + : "=Q" (lc->last_update_timer), + "=Q" (lc->last_update_clock) + : : "cc"); +} + /* * Update process times based on virtual cpu times stored by entry.S * to the lowcore fields user_timer, system_timer & steal_clock. @@ -121,17 +132,16 @@ static int do_account_vtime(struct task_struct *tsk) timer = lc->last_update_timer; clock = lc->last_update_clock; - asm volatile( - " stpt %0\n" /* Store current cpu timer value */ - " stckf %1" /* Store current tod clock value */ - : "=Q" (lc->last_update_timer), - "=Q" (lc->last_update_clock) - : : "cc"); + + vtime_reset_last_update(lc); + clock = lc->last_update_clock - clock; timer -= lc->last_update_timer; if (hardirq_count()) lc->hardirq_timer += timer; + else if (in_serving_softirq()) + lc->softirq_timer += timer; else lc->system_timer += timer; @@ -231,14 +241,63 @@ EXPORT_SYMBOL_GPL(vtime_account_kernel); void vtime_account_softirq(struct task_struct *tsk) { - get_lowcore()->softirq_timer += vtime_delta(); + if (!__this_cpu_read(s390_idle.idle_dyntick)) + get_lowcore()->softirq_timer += vtime_delta(); + else + vtime_flush(tsk); } void vtime_account_hardirq(struct task_struct *tsk) { - get_lowcore()->hardirq_timer += vtime_delta(); + if (!__this_cpu_read(s390_idle.idle_dyntick)) { + get_lowcore()->hardirq_timer += vtime_delta(); + } else { + /* + * In dynticks mode, the idle cputime is accounted by the nohz + * subsystem. Therefore the s390 timer/clocks are reset on IRQ + * entry and steal time must be accounted now. + */ + vtime_flush(tsk); + } } +#ifdef CONFIG_NO_HZ_COMMON +/** + * vtime_reset - Fast forward vtime entry clocks + * + * Called from dynticks idle IRQ entry to fast-forward the clocks to current time + * so that the IRQ time is still accounted by vtime while nohz cputime is paused. + */ +void vtime_reset(void) +{ + vtime_reset_last_update(get_lowcore()); +} + +/** + * vtime_dyntick_start - Inform vtime about entry to idle-dynticks + * + * Called when idle enters in dyntick mode. The idle cputime that elapsed so far + * is flushed and the tick subsystem takes over the idle cputime accounting. + */ +void vtime_dyntick_start(void) +{ + __this_cpu_write(s390_idle.idle_dyntick, true); + vtime_flush(current); +} + +/** + * vtime_dyntick_stop - Inform vtime about exit from idle-dynticks + * + * Called when idle exits from dyntick mode. The vtime entry clocks are + * fast-forward to current time and idle accounting resumes. + */ +void vtime_dyntick_stop(void) +{ + vtime_reset_last_update(get_lowcore()); + __this_cpu_write(s390_idle.idle_dyntick, false); +} +#endif /* CONFIG_NO_HZ_COMMON */ + /* * Sorted add to a list. List is linear searched until first bigger * element is found. diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index a6b787efc2c4..f83d5065c3cf 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -103,7 +103,6 @@ config SPARC64 select HAVE_REGS_AND_STACK_ACCESS_API select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS - select GENERIC_TIME_VSYSCALL select ARCH_HAS_PTE_SPECIAL select PCI_DOMAINS if PCI select ARCH_HAS_GIGANTIC_PAGE diff --git a/arch/um/Makefile b/arch/um/Makefile index 721b652ffb65..937639edc295 100644 --- a/arch/um/Makefile +++ b/arch/um/Makefile @@ -71,7 +71,8 @@ USER_CFLAGS = $(patsubst $(KERNEL_DEFINES),,$(patsubst -I%,,$(KBUILD_CFLAGS))) \ -D_FILE_OFFSET_BITS=64 -idirafter $(srctree)/include \ -idirafter $(objtree)/include -D__KERNEL__ -D__UM_HOST__ \ -include $(srctree)/include/linux/compiler-version.h \ - -include $(srctree)/include/linux/kconfig.h + -include $(srctree)/include/linux/kconfig.h \ + -idirafter $(ARCH_DIR)/include/generated #This will adjust *FLAGS accordingly to the platform. include $(srctree)/$(ARCH_DIR)/Makefile-os-Linux diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild index 2a1629ba8140..8fdc0bd9ab6f 100644 --- a/arch/um/include/asm/Kbuild +++ b/arch/um/include/asm/Kbuild @@ -16,6 +16,7 @@ generic-y += module.h generic-y += module.lds.h generic-y += parport.h generic-y += percpu.h +generic-y += percpu_types.h generic-y += preempt.h generic-y += ring_buffer.h generic-y += runtime-const.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b875d2f27e48..8af8e9070ce4 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -113,6 +113,7 @@ config X86 select ARCH_HAS_ZONE_DMA_SET if EXPERT select ARCH_HAVE_NMI_SAFE_CMPXCHG select ARCH_HAVE_EXTRA_ELF_NOTES + select ARCH_MEMORY_ORDER_TSO select ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE select ARCH_MIGHT_HAVE_ACPI_PDC if ACPI select ARCH_MIGHT_HAVE_PC_PARPORT @@ -178,7 +179,6 @@ config X86 select GENERIC_IRQ_SHOW select GENERIC_PENDING_IRQ if SMP select GENERIC_SMP_IDLE_THREAD - select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY select GENERIC_VDSO_OVERFLOW_PROTECT select GUP_GET_PXX_LOW_HIGH if X86_PAE @@ -236,6 +236,7 @@ config X86 select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_EISA if X86_32 select HAVE_EXIT_THREAD + select HAVE_FUTEX_ROBUST_UNLOCK select HAVE_GENERIC_TIF_BITS select HAVE_GUP_FAST select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE @@ -2435,10 +2436,6 @@ config CALL_THUNKS def_bool n select CALL_PADDING -config PREFIX_SYMBOLS - def_bool y - depends on CALL_PADDING && !CFI - menuconfig CPU_MITIGATIONS bool "Mitigations for CPU vulnerabilities" default y diff --git a/arch/x86/entry/vdso/common/vfutex.c b/arch/x86/entry/vdso/common/vfutex.c new file mode 100644 index 000000000000..454f059278e4 --- /dev/null +++ b/arch/x86/entry/vdso/common/vfutex.c @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include + +/* + * Assembly template for the try unlock functions. The basic functionality is: + * + * mov esi, %eax Move the TID into EAX + * xor %ecx, %ecx Clear ECX + * lock_cmpxchgl %ecx, (%rdi) Attempt the TID -> 0 transition + * .Lcs_start: Start of the critical section + * jnz .Lcs_end If cmpxchl failed jump to the end + * .Lcs_success: Start of the success section + * movq %rcx, (%rdx) Set the pending op pointer to 0 + * .Lcs_end: End of the critical section + * + * .Lcs_start and .Lcs_end establish the critical section range. .Lcs_success is + * technically not required, but there for illustration, debugging and testing. + * + * When CONFIG_COMPAT is enabled then the 64-bit VDSO provides two functions. + * One for the regular 64-bit sized pending operation pointer and one for a + * 32-bit sized pointer to support gaming emulators. + * + * The 32-bit VDSO provides only the one for 32-bit sized pointers. + */ +#define __stringify_1(x...) #x +#define __stringify(x...) __stringify_1(x) + +#define LABEL(prefix, which) __stringify(prefix##_try_unlock_cs_##which:) + +#define JNZ_END(prefix) "jnz " __stringify(prefix) "_try_unlock_cs_end\n" + +#define CLEAR_POPQ "movq %[zero], %a[pop]\n" +#define CLEAR_POPL "movl %k[zero], %a[pop]\n" + +#define futex_robust_try_unlock(prefix, clear_pop, __lock, __tid, __pop)\ +({ \ + asm volatile ( \ + " \n" \ + " lock cmpxchgl %k[zero], %a[lock] \n" \ + " \n" \ + LABEL(prefix, start) \ + " \n" \ + JNZ_END(prefix) \ + " \n" \ + LABEL(prefix, success) \ + " \n" \ + clear_pop \ + " \n" \ + LABEL(prefix, end) \ + : [tid] "+&a" (__tid) \ + : [lock] "D" (__lock), \ + [pop] "d" (__pop), \ + [zero] "r" (0UL) \ + : "memory" \ + ); \ + __tid; \ +}) + +#ifdef __x86_64__ +__u32 __vdso_futex_robust_list64_try_unlock(__u32 *lock, __u32 tid, __u64 *pop) +{ + return futex_robust_try_unlock(__futex_list64, CLEAR_POPQ, lock, tid, pop); +} +#endif /* __x86_64__ */ + +#if defined(CONFIG_X86_32) || defined(CONFIG_COMPAT) +__u32 __vdso_futex_robust_list32_try_unlock(__u32 *lock, __u32 tid, __u32 *pop) +{ + return futex_robust_try_unlock(__futex_list32, CLEAR_POPL, lock, tid, pop); +} +#endif /* CONFIG_X86_32 || CONFIG_COMPAT */ diff --git a/arch/x86/entry/vdso/vdso32/Makefile b/arch/x86/entry/vdso/vdso32/Makefile index ded4fc6a48cd..ab4b1f635f66 100644 --- a/arch/x86/entry/vdso/vdso32/Makefile +++ b/arch/x86/entry/vdso/vdso32/Makefile @@ -7,8 +7,9 @@ vdsos-y := 32 # Files to link into the vDSO: -vobjs-y := note.o vclock_gettime.o vgetcpu.o -vobjs-y += system_call.o sigreturn.o +vobjs-y := note.o vclock_gettime.o vgetcpu.o +vobjs-y += system_call.o sigreturn.o +vobjs-$(CONFIG_FUTEX_ROBUST_UNLOCK) += vfutex.o # Compilation flags flags-y := -DBUILD_VDSO32 -m32 -mregparm=0 diff --git a/arch/x86/entry/vdso/vdso32/vdso32.lds.S b/arch/x86/entry/vdso/vdso32/vdso32.lds.S index 55554f80d930..cee8f7f9fe80 100644 --- a/arch/x86/entry/vdso/vdso32/vdso32.lds.S +++ b/arch/x86/entry/vdso/vdso32/vdso32.lds.S @@ -30,6 +30,9 @@ VERSION __vdso_clock_gettime64; __vdso_clock_getres_time64; __vdso_getcpu; +#ifdef CONFIG_FUTEX_ROBUST_UNLOCK + __vdso_futex_robust_list32_try_unlock; +#endif }; LINUX_2.5 { diff --git a/arch/x86/entry/vdso/vdso32/vfutex.c b/arch/x86/entry/vdso/vdso32/vfutex.c new file mode 100644 index 000000000000..940a6ee30026 --- /dev/null +++ b/arch/x86/entry/vdso/vdso32/vfutex.c @@ -0,0 +1 @@ +#include "common/vfutex.c" diff --git a/arch/x86/entry/vdso/vdso64/Makefile b/arch/x86/entry/vdso/vdso64/Makefile index bfffaf1aeecc..7c0790065b5e 100644 --- a/arch/x86/entry/vdso/vdso64/Makefile +++ b/arch/x86/entry/vdso/vdso64/Makefile @@ -8,9 +8,10 @@ vdsos-y := 64 vdsos-$(CONFIG_X86_X32_ABI) += x32 # Files to link into the vDSO: -vobjs-y := note.o vclock_gettime.o vgetcpu.o -vobjs-y += vgetrandom.o vgetrandom-chacha.o -vobjs-$(CONFIG_X86_SGX) += vsgx.o +vobjs-y := note.o vclock_gettime.o vgetcpu.o +vobjs-y += vgetrandom.o vgetrandom-chacha.o +vobjs-$(CONFIG_X86_SGX) += vsgx.o +vobjs-$(CONFIG_FUTEX_ROBUST_UNLOCK) += vfutex.o # Compilation flags flags-y := -DBUILD_VDSO64 -m64 -mcmodel=small diff --git a/arch/x86/entry/vdso/vdso64/vdso64.lds.S b/arch/x86/entry/vdso/vdso64/vdso64.lds.S index 5ce3f2b6373a..4a72122da81b 100644 --- a/arch/x86/entry/vdso/vdso64/vdso64.lds.S +++ b/arch/x86/entry/vdso/vdso64/vdso64.lds.S @@ -32,6 +32,13 @@ VERSION { #endif getrandom; __vdso_getrandom; + +#ifdef CONFIG_FUTEX_ROBUST_UNLOCK + __vdso_futex_robust_list64_try_unlock; +#ifdef CONFIG_COMPAT + __vdso_futex_robust_list32_try_unlock; +#endif +#endif local: *; }; } diff --git a/arch/x86/entry/vdso/vdso64/vdsox32.lds.S b/arch/x86/entry/vdso/vdso64/vdsox32.lds.S index 3dbd20c8dacc..b917dc69f62f 100644 --- a/arch/x86/entry/vdso/vdso64/vdsox32.lds.S +++ b/arch/x86/entry/vdso/vdso64/vdsox32.lds.S @@ -22,6 +22,13 @@ VERSION { __vdso_getcpu; __vdso_time; __vdso_clock_getres; + +#ifdef CONFIG_FUTEX_ROBUST_UNLOCK + __vdso_futex_robust_list64_try_unlock; +#ifdef CONFIG_COMPAT + __vdso_futex_robust_list32_try_unlock; +#endif +#endif local: *; }; } diff --git a/arch/x86/entry/vdso/vdso64/vfutex.c b/arch/x86/entry/vdso/vdso64/vfutex.c new file mode 100644 index 000000000000..940a6ee30026 --- /dev/null +++ b/arch/x86/entry/vdso/vdso64/vfutex.c @@ -0,0 +1 @@ +#include "common/vfutex.c" diff --git a/arch/x86/entry/vdso/vma.c b/arch/x86/entry/vdso/vma.c index d903bce24f15..18dfd80a81ef 100644 --- a/arch/x86/entry/vdso/vma.c +++ b/arch/x86/entry/vdso/vma.c @@ -6,6 +6,7 @@ */ #include #include +#include #include #include #include @@ -73,6 +74,31 @@ static void vdso_fix_landing(const struct vdso_image *image, regs->ip = new_vma->vm_start + ipoffset; } +#ifdef CONFIG_FUTEX_ROBUST_UNLOCK +static void vdso_futex_robust_unlock_update_ips(void) +{ + const struct vdso_image *image = current->mm->context.vdso_image; + unsigned long vdso = (unsigned long) current->mm->context.vdso; + struct futex_mm_data *fd = ¤t->mm->futex; + unsigned int idx = 0; + + futex_reset_cs_ranges(fd); + +#ifdef CONFIG_X86_64 + futex_set_vdso_cs_range(fd, idx, vdso + image->sym___futex_list64_try_unlock_cs_start, + vdso + image->sym___futex_list64_try_unlock_cs_end, false); + idx++; +#endif /* CONFIG_X86_64 */ + +#if defined(CONFIG_X86_32) || defined(CONFIG_COMPAT) + futex_set_vdso_cs_range(fd, idx, vdso + image->sym___futex_list32_try_unlock_cs_start, + vdso + image->sym___futex_list32_try_unlock_cs_end, true); +#endif /* CONFIG_X86_32 || CONFIG_COMPAT */ +} +#else +static inline void vdso_futex_robust_unlock_update_ips(void) { } +#endif + static int vdso_mremap(const struct vm_special_mapping *sm, struct vm_area_struct *new_vma) { @@ -80,6 +106,7 @@ static int vdso_mremap(const struct vm_special_mapping *sm, vdso_fix_landing(image, new_vma); current->mm->context.vdso = (void __user *)new_vma->vm_start; + vdso_futex_robust_unlock_update_ips(); return 0; } @@ -185,6 +212,8 @@ static int map_vdso(const struct vdso_image *image, unsigned long addr) current->mm->context.vdso = (void __user *)text_start; current->mm->context.vdso_image = image; + vdso_futex_robust_unlock_update_ips(); + up_fail: mmap_write_unlock(mm); return ret; diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c index 305774b67995..8ad771f1a0de 100644 --- a/arch/x86/events/amd/core.c +++ b/arch/x86/events/amd/core.c @@ -1412,12 +1412,12 @@ static int __init amd_core_pmu_init(void) u64 even_ctr_mask = 0ULL; int i; - if (!boot_cpu_has(X86_FEATURE_PERFCTR_CORE)) - return 0; - /* Avoid calculating the value each time in the NMI handler */ perf_nmi_window = msecs_to_jiffies(100); + if (!boot_cpu_has(X86_FEATURE_PERFCTR_CORE)) + return 0; + /* * If core performance counter extensions exists, we must use * MSR_F15H_PERF_CTL/MSR_F15H_PERF_CTR msrs. See also diff --git a/arch/x86/events/amd/uncore.c b/arch/x86/events/amd/uncore.c index dd956cfcadef..a0364ca2f917 100644 --- a/arch/x86/events/amd/uncore.c +++ b/arch/x86/events/amd/uncore.c @@ -700,7 +700,7 @@ void amd_uncore_df_ctx_scan(struct amd_uncore *uncore, unsigned int cpu) info.split.aux_data = 0; info.split.num_pmcs = NUM_COUNTERS_NB; info.split.gid = 0; - info.split.cid = topology_logical_package_id(cpu); + info.split.cid = topology_amd_node_id(cpu); if (pmu_version >= 2) { ebx.full = cpuid_ebx(EXT_PERFMON_DEBUG_FEATURES); @@ -999,8 +999,8 @@ void amd_uncore_umc_ctx_scan(struct amd_uncore *uncore, unsigned int cpu) cpuid(EXT_PERFMON_DEBUG_FEATURES, &eax, &ebx.full, &ecx, &edx); info.split.aux_data = ecx; /* stash active mask */ info.split.num_pmcs = ebx.split.num_umc_pmc; - info.split.gid = topology_logical_package_id(cpu); - info.split.cid = topology_logical_package_id(cpu); + info.split.gid = topology_amd_node_id(cpu); + info.split.cid = topology_amd_node_id(cpu); *per_cpu_ptr(uncore->info, cpu) = info; } diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index 1b5ec78076ae..6f4e0e5a2416 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -215,39 +215,47 @@ static struct event_constraint intel_slm_event_constraints[] __read_mostly = static struct event_constraint intel_grt_event_constraints[] __read_mostly = { FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */ + FIXED_EVENT_CONSTRAINT(0x0100, 0), /* pseudo INST_RETIRED.ANY */ FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */ - FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF */ + FIXED_EVENT_CONSTRAINT(0x0200, 1), /* pseudo CPU_CLK_UNHALTED.THREAD */ + FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF_TSC */ FIXED_EVENT_CONSTRAINT(0x013c, 2), /* CPU_CLK_UNHALTED.REF_TSC_P */ EVENT_CONSTRAINT_END }; static struct event_constraint intel_skt_event_constraints[] __read_mostly = { FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */ + FIXED_EVENT_CONSTRAINT(0x0100, 0), /* pseudo INST_RETIRED.ANY */ FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */ - FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF */ + FIXED_EVENT_CONSTRAINT(0x0200, 1), /* pseudo CPU_CLK_UNHALTED.THREAD */ + FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF_TSC */ FIXED_EVENT_CONSTRAINT(0x013c, 2), /* CPU_CLK_UNHALTED.REF_TSC_P */ FIXED_EVENT_CONSTRAINT(0x0073, 4), /* TOPDOWN_BAD_SPECULATION.ALL */ + FIXED_EVENT_CONSTRAINT(0x0500, 4), /* pseudo TOPDOWN_BAD_SPECULATION.ALL */ FIXED_EVENT_CONSTRAINT(0x019c, 5), /* TOPDOWN_FE_BOUND.ALL */ + FIXED_EVENT_CONSTRAINT(0x0600, 5), /* pseudo TOPDOWN_FE_BOUND.ALL */ FIXED_EVENT_CONSTRAINT(0x02c2, 6), /* TOPDOWN_RETIRING.ALL */ + FIXED_EVENT_CONSTRAINT(0x0700, 6), /* pseudo TOPDOWN_RETIRING.ALL */ EVENT_CONSTRAINT_END }; static struct event_constraint intel_arw_event_constraints[] __read_mostly = { FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */ + FIXED_EVENT_CONSTRAINT(0x0100, 0), /* pseudo INST_RETIRED.ANY */ FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */ - FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF */ + FIXED_EVENT_CONSTRAINT(0x0200, 1), /* pseudo CPU_CLK_UNHALTED.THREAD */ + FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF_TSC */ FIXED_EVENT_CONSTRAINT(0x013c, 2), /* CPU_CLK_UNHALTED.REF_TSC_P */ FIXED_EVENT_CONSTRAINT(0x0073, 4), /* TOPDOWN_BAD_SPECULATION.ALL */ + FIXED_EVENT_CONSTRAINT(0x0500, 4), /* pseudo TOPDOWN_BAD_SPECULATION.ALL */ FIXED_EVENT_CONSTRAINT(0x019c, 5), /* TOPDOWN_FE_BOUND.ALL */ + FIXED_EVENT_CONSTRAINT(0x0600, 5), /* pseudo TOPDOWN_FE_BOUND.ALL */ FIXED_EVENT_CONSTRAINT(0x02c2, 6), /* TOPDOWN_RETIRING.ALL */ + FIXED_EVENT_CONSTRAINT(0x0700, 6), /* pseudo TOPDOWN_RETIRING.ALL */ INTEL_UEVENT_CONSTRAINT(0x01b7, 0x1), INTEL_UEVENT_CONSTRAINT(0x02b7, 0x2), INTEL_UEVENT_CONSTRAINT(0x04b7, 0x4), INTEL_UEVENT_CONSTRAINT(0x08b7, 0x8), - INTEL_UEVENT_CONSTRAINT(0x01d4, 0x1), - INTEL_UEVENT_CONSTRAINT(0x02d4, 0x2), - INTEL_UEVENT_CONSTRAINT(0x04d4, 0x4), - INTEL_UEVENT_CONSTRAINT(0x08d4, 0x8), INTEL_UEVENT_CONSTRAINT(0x0175, 0x1), INTEL_UEVENT_CONSTRAINT(0x0275, 0x2), INTEL_UEVENT_CONSTRAINT(0x21d3, 0x1), @@ -310,10 +318,11 @@ static struct extra_reg intel_skl_extra_regs[] __read_mostly = { static struct event_constraint intel_icl_event_constraints[] = { FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */ FIXED_EVENT_CONSTRAINT(0x01c0, 0), /* old INST_RETIRED.PREC_DIST */ - FIXED_EVENT_CONSTRAINT(0x0100, 0), /* INST_RETIRED.PREC_DIST */ + FIXED_EVENT_CONSTRAINT(0x0100, 0), /* pseudo INST_RETIRED.ANY */ FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */ - FIXED_EVENT_CONSTRAINT(0x0300, 2), /* CPU_CLK_UNHALTED.REF */ - FIXED_EVENT_CONSTRAINT(0x0400, 3), /* SLOTS */ + FIXED_EVENT_CONSTRAINT(0x0200, 1), /* pseudo CPU_CLK_UNHALTED.THREAD */ + FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF_TSC */ + FIXED_EVENT_CONSTRAINT(0x0400, 3), /* pseudo TOPDOWN.SLOTS */ METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_RETIRING, 0), METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_BAD_SPEC, 1), METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_FE_BOUND, 2), @@ -355,11 +364,12 @@ static struct extra_reg intel_glc_extra_regs[] __read_mostly = { static struct event_constraint intel_glc_event_constraints[] = { FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */ - FIXED_EVENT_CONSTRAINT(0x0100, 0), /* INST_RETIRED.PREC_DIST */ + FIXED_EVENT_CONSTRAINT(0x0100, 0), /* pseudo INST_RETIRED.ANY */ FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */ - FIXED_EVENT_CONSTRAINT(0x0300, 2), /* CPU_CLK_UNHALTED.REF */ + FIXED_EVENT_CONSTRAINT(0x0200, 1), /* pseudo CPU_CLK_UNHALTED.THREAD */ + FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF_TSC */ FIXED_EVENT_CONSTRAINT(0x013c, 2), /* CPU_CLK_UNHALTED.REF_TSC_P */ - FIXED_EVENT_CONSTRAINT(0x0400, 3), /* SLOTS */ + FIXED_EVENT_CONSTRAINT(0x0400, 3), /* pseudo TOPDOWN.SLOTS */ METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_RETIRING, 0), METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_BAD_SPEC, 1), METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_FE_BOUND, 2), @@ -379,9 +389,13 @@ static struct event_constraint intel_glc_event_constraints[] = { INTEL_UEVENT_CONSTRAINT(0x01a3, 0xf), INTEL_UEVENT_CONSTRAINT(0x02a3, 0xf), + INTEL_UEVENT_CONSTRAINT(0x05a3, 0xf), + INTEL_UEVENT_CONSTRAINT(0x06a3, 0xf), INTEL_UEVENT_CONSTRAINT(0x08a3, 0xf), + INTEL_UEVENT_CONSTRAINT(0x0ca3, 0xf), INTEL_UEVENT_CONSTRAINT(0x04a4, 0x1), INTEL_UEVENT_CONSTRAINT(0x08a4, 0x1), + INTEL_UEVENT_CONSTRAINT(0x01cd, 0xfe), INTEL_UEVENT_CONSTRAINT(0x02cd, 0x1), INTEL_EVENT_CONSTRAINT(0xce, 0x1), INTEL_EVENT_CONSTRAINT_RANGE(0xd0, 0xdf, 0xf), @@ -407,11 +421,12 @@ static struct extra_reg intel_rwc_extra_regs[] __read_mostly = { static struct event_constraint intel_lnc_event_constraints[] = { FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */ - FIXED_EVENT_CONSTRAINT(0x0100, 0), /* INST_RETIRED.PREC_DIST */ + FIXED_EVENT_CONSTRAINT(0x0100, 0), /* pseudo INST_RETIRED.ANY */ FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */ - FIXED_EVENT_CONSTRAINT(0x0300, 2), /* CPU_CLK_UNHALTED.REF */ + FIXED_EVENT_CONSTRAINT(0x0200, 1), /* pseudo CPU_CLK_UNHALTED.THREAD */ + FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF_TSC */ FIXED_EVENT_CONSTRAINT(0x013c, 2), /* CPU_CLK_UNHALTED.REF_TSC_P */ - FIXED_EVENT_CONSTRAINT(0x0400, 3), /* SLOTS */ + FIXED_EVENT_CONSTRAINT(0x0400, 3), /* pseudo TOPDOWN.SLOTS */ METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_RETIRING, 0), METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_BAD_SPEC, 1), METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_FE_BOUND, 2), @@ -423,8 +438,6 @@ static struct event_constraint intel_lnc_event_constraints[] = { INTEL_EVENT_CONSTRAINT(0x20, 0xf), - INTEL_UEVENT_CONSTRAINT(0x012a, 0xf), - INTEL_UEVENT_CONSTRAINT(0x012b, 0xf), INTEL_UEVENT_CONSTRAINT(0x0148, 0x4), INTEL_UEVENT_CONSTRAINT(0x0175, 0x4), @@ -435,15 +448,14 @@ static struct event_constraint intel_lnc_event_constraints[] = { INTEL_UEVENT_CONSTRAINT(0x0ca3, 0x4), INTEL_UEVENT_CONSTRAINT(0x04a4, 0x1), INTEL_UEVENT_CONSTRAINT(0x08a4, 0x1), - INTEL_UEVENT_CONSTRAINT(0x10a4, 0x1), + INTEL_UEVENT_CONSTRAINT(0x10a4, 0x8), INTEL_UEVENT_CONSTRAINT(0x01b1, 0x8), INTEL_UEVENT_CONSTRAINT(0x01cd, 0x3fc), INTEL_UEVENT_CONSTRAINT(0x02cd, 0x3), + INTEL_UEVENT_CONSTRAINT(0x87d0, 0x3ff), INTEL_EVENT_CONSTRAINT_RANGE(0xd0, 0xdf, 0xf), - INTEL_UEVENT_CONSTRAINT(0x00e0, 0xf), - EVENT_CONSTRAINT_END }; @@ -460,11 +472,12 @@ static struct extra_reg intel_lnc_extra_regs[] __read_mostly = { static struct event_constraint intel_pnc_event_constraints[] = { FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */ - FIXED_EVENT_CONSTRAINT(0x0100, 0), /* INST_RETIRED.PREC_DIST */ + FIXED_EVENT_CONSTRAINT(0x0100, 0), /* pseudo INST_RETIRED.ANY */ FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */ - FIXED_EVENT_CONSTRAINT(0x0300, 2), /* CPU_CLK_UNHALTED.REF */ + FIXED_EVENT_CONSTRAINT(0x0200, 1), /* pseudo CPU_CLK_UNHALTED.THREAD */ + FIXED_EVENT_CONSTRAINT(0x0300, 2), /* pseudo CPU_CLK_UNHALTED.REF_TSC */ FIXED_EVENT_CONSTRAINT(0x013c, 2), /* CPU_CLK_UNHALTED.REF_TSC_P */ - FIXED_EVENT_CONSTRAINT(0x0400, 3), /* SLOTS */ + FIXED_EVENT_CONSTRAINT(0x0400, 3), /* pseudo TOPDOWN.SLOTS */ METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_RETIRING, 0), METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_BAD_SPEC, 1), METRIC_EVENT_CONSTRAINT(INTEL_TD_METRIC_FE_BOUND, 2), @@ -706,6 +719,80 @@ static __initconst const u64 glc_hw_cache_event_ids }, }; +/* ADL P-core (Golden cove) specific event code. */ +static __initconst const u64 adl_glc_hw_cache_event_ids + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = +{ + [ C(L1D ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x81d0, + [ C(RESULT_MISS) ] = 0xe124, + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x82d0, + }, + }, + [ C(L1I ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_MISS) ] = 0xe424, + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = -1, + [ C(RESULT_MISS) ] = -1, + }, + }, + [ C(LL ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x12a, + [ C(RESULT_MISS) ] = 0x12a, + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x12a, + [ C(RESULT_MISS) ] = 0x12a, + }, + }, + [ C(DTLB) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x81d0, + [ C(RESULT_MISS) ] = 0xe12, + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x82d0, + [ C(RESULT_MISS) ] = 0xe13, + }, + }, + [ C(ITLB) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = -1, + [ C(RESULT_MISS) ] = 0xe11, + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = -1, + [ C(RESULT_MISS) ] = -1, + }, + [ C(OP_PREFETCH) ] = { + [ C(RESULT_ACCESS) ] = -1, + [ C(RESULT_MISS) ] = -1, + }, + }, + [ C(BPU ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x4c4, + [ C(RESULT_MISS) ] = 0x4c5, + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = -1, + [ C(RESULT_MISS) ] = -1, + }, + [ C(OP_PREFETCH) ] = { + [ C(RESULT_ACCESS) ] = -1, + [ C(RESULT_MISS) ] = -1, + }, + }, +}; + static __initconst const u64 glc_hw_cache_extra_regs [PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_OP_MAX] @@ -713,18 +800,71 @@ static __initconst const u64 glc_hw_cache_extra_regs { [ C(LL ) ] = { [ C(OP_READ) ] = { - [ C(RESULT_ACCESS) ] = 0x10001, - [ C(RESULT_MISS) ] = 0x3fbfc00001, + [ C(RESULT_ACCESS) ] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0x3fbfc00001, /* OCR.DEMAND_DATA_RD.L3_MISS */ }, [ C(OP_WRITE) ] = { - [ C(RESULT_ACCESS) ] = 0x3f3ffc0002, - [ C(RESULT_MISS) ] = 0x3f3fc00002, + [ C(RESULT_ACCESS) ] = 0x3f3ffc0002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0x3f3fc00002, /* OCR.DEMAND_RFO.L3_MISS */ }, }, [ C(NODE) ] = { [ C(OP_READ) ] = { - [ C(RESULT_ACCESS) ] = 0x10c000001, - [ C(RESULT_MISS) ] = 0x3fb3000001, + [ C(RESULT_ACCESS) ] = 0x104000001, /* OCR.DEMAND_DATA_RD.LOCAL_DRAM */ + [ C(RESULT_MISS) ] = 0x730000001, /* OCR.DEMAND_DATA_RD.REMOTE_DRAM */ + }, + }, +}; + +/* ADL P-core (Golden cove) specific extra regs value. */ +static __initconst const u64 adl_glc_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = +{ + [ C(LL ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0x3fbfc00001, /* OCR.DEMAND_DATA_RD.L3_MISS */ + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x10002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0x3fbfc00002, /* OCR.DEMAND_RFO.L3_MISS */ + }, + }, +}; + +static __initconst const u64 lnc_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = +{ + [ C(LL ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0x9E7FA000001, /* OCR.DEMAND_DATA_RD.L3_MISS */ + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x10002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0x9E7FA000002, /* OCR.DEMAND_RFO.L3_MISS */ + }, + }, +}; + +/* ARL specific lioncove hw_cache_extra_regs[] variant. */ +static __initconst const u64 arl_lnc_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = +{ + [ C(LL ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0xFE7F8000001, /* OCR.DEMAND_DATA_RD.L3_MISS */ + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x10002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0xFE7F8000002, /* OCR.DEMAND_RFO.L3_MISS */ }, }, }; @@ -815,12 +955,29 @@ static __initconst const u64 pnc_hw_cache_extra_regs { [ C(LL ) ] = { [ C(OP_READ) ] = { - [ C(RESULT_ACCESS) ] = 0x4000000000000001, - [ C(RESULT_MISS) ] = 0xFFFFF000000001, + [ C(RESULT_ACCESS) ] = 0x4000000000000001, /* OMR.DEMAND_DATA_RD.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0xFFFFF000000001, /* OMR.DEMAND_DATA_RD.L3_MISS */ }, [ C(OP_WRITE) ] = { - [ C(RESULT_ACCESS) ] = 0x4000000000000002, - [ C(RESULT_MISS) ] = 0xFFFFF000000002, + [ C(RESULT_ACCESS) ] = 0x4000000000000002, /* OMR.DEMAND_RFO.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0xFFFFF000000002, /* OMR.DEMAND_RFO.L3_MISS */ + }, + }, +}; + +static __initconst const u64 cyc_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = +{ + [ C(LL ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x4000000000000001, /* OMR.DEMAND_DATA_RD.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0xFF03F000000001, /* OMR.DEMAND_DATA_RD.L3_MISS */ + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x4000000000000002, /* OMR.DEMAND_RFO.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0xFF03F000000002, /* OMR.DEMAND_RFO.L3_MISS */ }, }, }; @@ -1019,6 +1176,41 @@ static __initconst const u64 skl_hw_cache_extra_regs }, }; +static __initconst const u64 snc_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = +{ + [ C(LL ) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0x3FBFC00001, /* OCR.DEMAND_DATA_RD.L3_MISS */ + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x3F3FFC0002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [ C(RESULT_MISS) ] = 0x3F3FC00002, /* OCR.DEMAND_RFO.L3_MISS */ + }, + [ C(OP_PREFETCH) ] = { + [ C(RESULT_ACCESS) ] = 0x0, + [ C(RESULT_MISS) ] = 0x0, + }, + }, + [ C(NODE) ] = { + [ C(OP_READ) ] = { + [ C(RESULT_ACCESS) ] = 0x104000001, /* OCR.DEMAND_DATA_RD.LOCAL_DRAM */ + [ C(RESULT_MISS) ] = 0x730000001, /* OCR.DEMAND_DATA_RD.REMOTE_DRAM */ + }, + [ C(OP_WRITE) ] = { + [ C(RESULT_ACCESS) ] = 0x104000002, /* OCR.DEMAND_RFO.LOCAL_DRAM */ + [ C(RESULT_MISS) ] = 0x730000002, /* OCR.DEMAND_RFO.REMOTE_DRAM */ + }, + [ C(OP_PREFETCH) ] = { + [ C(RESULT_ACCESS) ] = 0x0, + [ C(RESULT_MISS) ] = 0x0, + }, + }, +}; + #define SNB_DMND_DATA_RD (1ULL << 0) #define SNB_DMND_RFO (1ULL << 1) #define SNB_DMND_IFETCH (1ULL << 2) @@ -2342,22 +2534,82 @@ static __initconst const u64 tnt_hw_cache_extra_regs }, }; +static __initconst const u64 grt_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = { + [C(LL)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0x3F84400001, /* OCR.DEMAND_DATA_RD.L3_MISS */ + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = 0x10002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0x3F84400002, /* OCR.DEMAND_RFO.L3_MISS */ + }, + }, +}; + +static __initconst const u64 cmt_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = { + [C(LL)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0x3fbfc00001, /* OCR.DEMAND_DATA_RD.L3_MISS */ + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = 0x10002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0x3fbfc00002, /* OCR.DEMAND_RFO.L3_MISS */ + }, + }, +}; + +static __initconst const u64 skt_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = { + [C(LL)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0x13FBFC00001, /* OCR.DEMAND_DATA_RD.L3_MISS */ + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = 0x10002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0x13FBFC00002, /* OCR.DEMAND_RFO.L3_MISS */ + }, + }, +}; + +static __initconst const u64 dkt_hw_cache_extra_regs + [PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX] = { + [C(LL)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = 0x10001, /* OCR.DEMAND_DATA_RD.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0x33FBFC00001, /* OCR.DEMAND_DATA_RD.L3_MISS */ + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = 0x10002, /* OCR.DEMAND_RFO.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0x33FBFC00002, /* OCR.DEMAND_RFO.L3_MISS */ + }, + }, +}; + static __initconst const u64 arw_hw_cache_extra_regs [PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_OP_MAX] [PERF_COUNT_HW_CACHE_RESULT_MAX] = { [C(LL)] = { [C(OP_READ)] = { - [C(RESULT_ACCESS)] = 0x4000000000000001, - [C(RESULT_MISS)] = 0xFFFFF000000001, + [C(RESULT_ACCESS)] = 0x4000000000000009, /* OMR.DEMAND_DATA_RD.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0xFF03F000000009, /* OMR.DEMAND_DATA_RD.L3_MISS */ }, [C(OP_WRITE)] = { - [C(RESULT_ACCESS)] = 0x4000000000000002, - [C(RESULT_MISS)] = 0xFFFFF000000002, - }, - [C(OP_PREFETCH)] = { - [C(RESULT_ACCESS)] = 0x0, - [C(RESULT_MISS)] = 0x0, + [C(RESULT_ACCESS)] = 0x400000000000000A, /* OMR.DEMAND_RFO.ANY_RESPONSE */ + [C(RESULT_MISS)] = 0xFF03F00000000A, /* OMR.DEMAND_RFO.L3_MISS */ }, }, }; @@ -2392,9 +2644,12 @@ static struct attribute *grt_mem_attrs[] = { }; static struct extra_reg intel_grt_extra_regs[] __read_mostly = { - /* must define OFFCORE_RSP_X first, see intel_fixup_er() */ - INTEL_UEVENT_EXTRA_REG(0x01b7, MSR_OFFCORE_RSP_0, 0x3fffffffffull, RSP_0), - INTEL_UEVENT_EXTRA_REG(0x02b7, MSR_OFFCORE_RSP_1, 0x3fffffffffull, RSP_1), + /* + * Must define OFFCORE_RSP_X first, see intel_fixup_er(). + * Bit 63 only valid on OFFCORE_RSP_0 MSR. + */ + INTEL_UEVENT_EXTRA_REG(0x01b7, MSR_OFFCORE_RSP_0, 0x8003f03fffffffffull, RSP_0), + INTEL_UEVENT_EXTRA_REG(0x02b7, MSR_OFFCORE_RSP_1, 0x3f03fffffffffull, RSP_1), INTEL_UEVENT_PEBS_LDLAT_EXTRA_REG(0x5d0), EVENT_EXTRA_END }; @@ -2426,10 +2681,6 @@ static struct extra_reg intel_arw_extra_regs[] __read_mostly = { INTEL_UEVENT_EXTRA_REG(0x02b7, MSR_OMR_1, 0xc0ffffffffffffffull, OMR_1), INTEL_UEVENT_EXTRA_REG(0x04b7, MSR_OMR_2, 0xc0ffffffffffffffull, OMR_2), INTEL_UEVENT_EXTRA_REG(0x08b7, MSR_OMR_3, 0xc0ffffffffffffffull, OMR_3), - INTEL_UEVENT_EXTRA_REG(0x01d4, MSR_OMR_0, 0xc0ffffffffffffffull, OMR_0), - INTEL_UEVENT_EXTRA_REG(0x02d4, MSR_OMR_1, 0xc0ffffffffffffffull, OMR_1), - INTEL_UEVENT_EXTRA_REG(0x04d4, MSR_OMR_2, 0xc0ffffffffffffffull, OMR_2), - INTEL_UEVENT_EXTRA_REG(0x08d4, MSR_OMR_3, 0xc0ffffffffffffffull, OMR_3), INTEL_UEVENT_PEBS_LDLAT_EXTRA_REG(0x5d0), INTEL_UEVENT_EXTRA_REG(0x0127, MSR_SNOOP_RSP_0, 0xffffffffffffffffull, SNOOP_0), INTEL_UEVENT_EXTRA_REG(0x0227, MSR_SNOOP_RSP_1, 0xffffffffffffffffull, SNOOP_1), @@ -3169,10 +3420,10 @@ static void intel_pmu_config_acr(int idx, u64 mask, u32 reload) wrmsrl(msr_b + msr_offset, mask); cpuc->acr_cfg_b[idx] = mask; } - /* Only need to update the reload value when there is a valid config value. */ - if (mask && cpuc->acr_cfg_c[idx] != reload) { + /* Only update CFG_C reload when ACR is actively enabled (mask != 0) */ + if (mask && ((cpuc->cfg_c_val[idx] & ARCH_PEBS_RELOAD) != reload)) { wrmsrl(msr_c + msr_offset, reload); - cpuc->acr_cfg_c[idx] = reload; + cpuc->cfg_c_val[idx] = reload; } } @@ -3198,14 +3449,15 @@ static void intel_pmu_enable_event_ext(struct perf_event *event) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); struct hw_perf_event *hwc = &event->hw; - union arch_pebs_index old, new; - struct arch_pebs_cap cap; u64 ext = 0; - cap = hybrid(cpuc->pmu, arch_pebs_cap); + if (is_acr_event_group(event)) + ext |= (-hwc->sample_period) & ARCH_PEBS_RELOAD; if (event->attr.precise_ip) { u64 pebs_data_cfg = intel_get_arch_pebs_data_config(event); + struct arch_pebs_cap cap = hybrid(cpuc->pmu, arch_pebs_cap); + union arch_pebs_index old, new; ext |= ARCH_PEBS_EN; if (hwc->flags & PERF_X86_EVENT_AUTO_RELOAD) @@ -7490,6 +7742,15 @@ static __always_inline void intel_pmu_init_glc(struct pmu *pmu) intel_pmu_ref_cycles_ext(); } +static __always_inline void intel_pmu_init_glc_hybrid(struct pmu *pmu) +{ + intel_pmu_init_glc(pmu); + + /* ADL has different extra MSR values from Server for the L3 or node OCR/OMR events. */ + memcpy(hybrid_var(pmu, hw_cache_event_ids), adl_glc_hw_cache_event_ids, sizeof(hw_cache_event_ids)); + memcpy(hybrid_var(pmu, hw_cache_extra_regs), adl_glc_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); +} + static __always_inline void intel_pmu_init_grt(struct pmu *pmu) { x86_pmu.mid_ack = true; @@ -7502,7 +7763,7 @@ static __always_inline void intel_pmu_init_grt(struct pmu *pmu) x86_pmu.flags |= PMU_FL_INSTR_LATENCY; memcpy(hybrid_var(pmu, hw_cache_event_ids), glp_hw_cache_event_ids, sizeof(hw_cache_event_ids)); - memcpy(hybrid_var(pmu, hw_cache_extra_regs), tnt_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); + memcpy(hybrid_var(pmu, hw_cache_extra_regs), grt_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); hybrid_var(pmu, hw_cache_event_ids)[C(ITLB)][C(OP_READ)][C(RESULT_ACCESS)] = -1; hybrid(pmu, event_constraints) = intel_grt_event_constraints; hybrid(pmu, pebs_constraints) = intel_grt_pebs_event_constraints; @@ -7511,12 +7772,24 @@ static __always_inline void intel_pmu_init_grt(struct pmu *pmu) intel_pmu_ref_cycles_ext(); } +static __always_inline void intel_pmu_init_cmt(struct pmu *pmu) +{ + intel_pmu_init_grt(pmu); + memcpy(hybrid_var(pmu, hw_cache_extra_regs), + cmt_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); + hybrid(pmu, pebs_constraints) = intel_cmt_pebs_event_constraints; + hybrid(pmu, extra_regs) = intel_cmt_extra_regs; +} + static __always_inline void intel_pmu_init_lnc(struct pmu *pmu) { intel_pmu_init_glc(pmu); hybrid(pmu, event_constraints) = intel_lnc_event_constraints; hybrid(pmu, pebs_constraints) = intel_lnc_pebs_event_constraints; hybrid(pmu, extra_regs) = intel_lnc_extra_regs; + + memcpy(hybrid_var(pmu, hw_cache_event_ids), adl_glc_hw_cache_event_ids, sizeof(hw_cache_event_ids)); + memcpy(hybrid_var(pmu, hw_cache_extra_regs), lnc_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); } static __always_inline void intel_pmu_init_pnc(struct pmu *pmu) @@ -7534,14 +7807,41 @@ static __always_inline void intel_pmu_init_pnc(struct pmu *pmu) static_call_update(intel_pmu_enable_acr_event, intel_pmu_enable_acr); } +static __always_inline void intel_pmu_init_cyc(struct pmu *pmu) +{ + intel_pmu_init_pnc(pmu); + memcpy(hybrid_var(pmu, hw_cache_extra_regs), + cyc_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); +} + static __always_inline void intel_pmu_init_skt(struct pmu *pmu) { - intel_pmu_init_grt(pmu); + intel_pmu_init_cmt(pmu); hybrid(pmu, event_constraints) = intel_skt_event_constraints; - hybrid(pmu, extra_regs) = intel_cmt_extra_regs; + memcpy(hybrid_var(pmu, hw_cache_extra_regs), + skt_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); static_call_update(intel_pmu_enable_acr_event, intel_pmu_enable_acr); } +/* Hybrid client variant. */ +static __always_inline void intel_pmu_init_dkt_hybrid(struct pmu *pmu) +{ + intel_pmu_init_skt(pmu); + hybrid(pmu, pebs_constraints) = intel_dkt_pebs_event_constraints; +} + +/* + * Darkmont is used by the CWF and PTL E-cores, but their L3 OCR + * events require different extra MSR values. Keep a separate init + * function for the non-hybrid server variant. + */ +static __always_inline void intel_pmu_init_dkt(struct pmu *pmu) +{ + intel_pmu_init_dkt_hybrid(pmu); + memcpy(hybrid_var(pmu, hw_cache_extra_regs), + dkt_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); +} + static __always_inline void intel_pmu_init_arw(struct pmu *pmu) { intel_pmu_init_grt(pmu); @@ -7550,7 +7850,7 @@ static __always_inline void intel_pmu_init_arw(struct pmu *pmu) memcpy(hybrid_var(pmu, hw_cache_extra_regs), arw_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); hybrid(pmu, event_constraints) = intel_arw_event_constraints; - hybrid(pmu, pebs_constraints) = intel_arw_pebs_event_constraints; + hybrid(pmu, pebs_constraints) = intel_dkt_pebs_event_constraints; hybrid(pmu, extra_regs) = intel_arw_extra_regs; static_call_update(intel_pmu_enable_acr_event, intel_pmu_enable_acr); } @@ -7864,8 +8164,7 @@ __init int intel_pmu_init(void) case INTEL_ATOM_CRESTMONT: case INTEL_ATOM_CRESTMONT_X: - intel_pmu_init_grt(NULL); - x86_pmu.extra_regs = intel_cmt_extra_regs; + intel_pmu_init_cmt(NULL); intel_pmu_pebs_data_source_cmt(); x86_pmu.pebs_latency_data = cmt_latency_data; x86_pmu.get_event_constraints = cmt_get_event_constraints; @@ -7877,7 +8176,7 @@ __init int intel_pmu_init(void) break; case INTEL_ATOM_DARKMONT_X: - intel_pmu_init_skt(NULL); + intel_pmu_init_dkt(NULL); intel_pmu_pebs_data_source_cmt(); x86_pmu.pebs_latency_data = cmt_latency_data; x86_pmu.get_event_constraints = cmt_get_event_constraints; @@ -8153,17 +8452,19 @@ __init int intel_pmu_init(void) case INTEL_ICELAKE_X: case INTEL_ICELAKE_D: + memcpy(hw_cache_extra_regs, snc_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); x86_pmu.pebs_ept = 1; pmem = true; - fallthrough; + goto snc_common; case INTEL_ICELAKE_L: case INTEL_ICELAKE: case INTEL_TIGERLAKE_L: case INTEL_TIGERLAKE: case INTEL_ROCKETLAKE: + memcpy(hw_cache_extra_regs, skl_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); + snc_common: x86_pmu.late_ack = true; memcpy(hw_cache_event_ids, skl_hw_cache_event_ids, sizeof(hw_cache_event_ids)); - memcpy(hw_cache_extra_regs, skl_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); hw_cache_event_ids[C(ITLB)][C(OP_READ)][C(RESULT_ACCESS)] = -1; intel_pmu_lbr_init_skl(); @@ -8259,7 +8560,7 @@ __init int intel_pmu_init(void) /* Initialize big core specific PerfMon capabilities.*/ pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX]; - intel_pmu_init_glc(&pmu->pmu); + intel_pmu_init_glc_hybrid(&pmu->pmu); if (cpu_feature_enabled(X86_FEATURE_HYBRID_CPU)) { pmu->cntr_mask64 <<= 2; pmu->cntr_mask64 |= 0x3; @@ -8316,13 +8617,12 @@ __init int intel_pmu_init(void) /* Initialize big core specific PerfMon capabilities.*/ pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX]; - intel_pmu_init_glc(&pmu->pmu); + intel_pmu_init_glc_hybrid(&pmu->pmu); pmu->extra_regs = intel_rwc_extra_regs; /* Initialize Atom core specific PerfMon capabilities.*/ pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX]; - intel_pmu_init_grt(&pmu->pmu); - pmu->extra_regs = intel_cmt_extra_regs; + intel_pmu_init_cmt(&pmu->pmu); intel_pmu_pebs_data_source_mtl(); pr_cont("Meteorlake Hybrid events, "); @@ -8333,16 +8633,50 @@ __init int intel_pmu_init(void) case INTEL_WILDCATLAKE_L: pr_cont("Pantherlake Hybrid events, "); name = "pantherlake_hybrid"; + + intel_pmu_init_hybrid(hybrid_big_small); + + /* Initialize big core specific PerfMon capabilities.*/ + pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX]; + intel_pmu_init_lnc(&pmu->pmu); + /* Initialize Atom core specific PerfMon capabilities.*/ + pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX]; + intel_pmu_init_dkt_hybrid(&pmu->pmu); + + goto lnl_common; + + case INTEL_ARROWLAKE: + pr_cont("Arrowlake Hybrid events, "); + name = "arrowlake_hybrid"; + + intel_pmu_init_hybrid(hybrid_big_small); + + /* Initialize big core specific PerfMon capabilities.*/ + pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX]; + intel_pmu_init_lnc(&pmu->pmu); + memcpy(hybrid_var(&pmu->pmu, hw_cache_extra_regs), + arl_lnc_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); + /* Initialize Atom core specific PerfMon capabilities.*/ + pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX]; + intel_pmu_init_skt(&pmu->pmu); + goto lnl_common; case INTEL_LUNARLAKE_M: - case INTEL_ARROWLAKE: pr_cont("Lunarlake Hybrid events, "); name = "lunarlake_hybrid"; - lnl_common: intel_pmu_init_hybrid(hybrid_big_small); + /* Initialize big core specific PerfMon capabilities.*/ + pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX]; + intel_pmu_init_lnc(&pmu->pmu); + /* Initialize Atom core specific PerfMon capabilities.*/ + pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX]; + intel_pmu_init_skt(&pmu->pmu); + + lnl_common: + x86_pmu.pebs_latency_data = lnl_latency_data; x86_pmu.get_event_constraints = mtl_get_event_constraints; x86_pmu.hw_config = adl_hw_config; @@ -8353,14 +8687,6 @@ __init int intel_pmu_init(void) extra_attr = boot_cpu_has(X86_FEATURE_RTM) ? mtl_hybrid_extra_attr_rtm : mtl_hybrid_extra_attr; - /* Initialize big core specific PerfMon capabilities.*/ - pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX]; - intel_pmu_init_lnc(&pmu->pmu); - - /* Initialize Atom core specific PerfMon capabilities.*/ - pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX]; - intel_pmu_init_skt(&pmu->pmu); - intel_pmu_pebs_data_source_lnl(); break; @@ -8380,6 +8706,8 @@ __init int intel_pmu_init(void) /* Initialize big core specific PerfMon capabilities. */ pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX]; intel_pmu_init_lnc(&pmu->pmu); + memcpy(hybrid_var(&pmu->pmu, hw_cache_extra_regs), + arl_lnc_hw_cache_extra_regs, sizeof(hw_cache_extra_regs)); /* Initialize Atom core specific PerfMon capabilities. */ pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX]; @@ -8387,8 +8715,7 @@ __init int intel_pmu_init(void) /* Initialize Lower Power Atom specific PerfMon capabilities. */ pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_TINY_IDX]; - intel_pmu_init_grt(&pmu->pmu); - pmu->extra_regs = intel_cmt_extra_regs; + intel_pmu_init_cmt(&pmu->pmu); intel_pmu_pebs_data_source_arl_h(); pr_cont("ArrowLake-H Hybrid events, "); @@ -8413,7 +8740,7 @@ __init int intel_pmu_init(void) /* Initialize big core specific PerfMon capabilities.*/ pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_CORE_IDX]; - intel_pmu_init_pnc(&pmu->pmu); + intel_pmu_init_cyc(&pmu->pmu); /* Initialize Atom core specific PerfMon capabilities.*/ pmu = &x86_pmu.hybrid_pmu[X86_HYBRID_PMU_ATOM_IDX]; diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c index 7f0d515c07c5..cb72af9b61ce 100644 --- a/arch/x86/events/intel/ds.c +++ b/arch/x86/events/intel/ds.c @@ -1292,18 +1292,21 @@ struct event_constraint intel_glm_pebs_event_constraints[] = { struct event_constraint intel_grt_pebs_event_constraints[] = { /* Allow all events as PEBS with no flags */ INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0x3), - INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0xf), + INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0x3f), EVENT_CONSTRAINT_END }; -struct event_constraint intel_arw_pebs_event_constraints[] = { +struct event_constraint intel_cmt_pebs_event_constraints[] = { + /* Allow all events as PEBS with no flags */ + INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0x3), + INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0xff), + EVENT_CONSTRAINT_END +}; + +struct event_constraint intel_dkt_pebs_event_constraints[] = { /* Allow all events as PEBS with no flags */ INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0xff), INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0xff), - INTEL_FLAGS_UEVENT_CONSTRAINT(0x01d4, 0x1), - INTEL_FLAGS_UEVENT_CONSTRAINT(0x02d4, 0x2), - INTEL_FLAGS_UEVENT_CONSTRAINT(0x04d4, 0x4), - INTEL_FLAGS_UEVENT_CONSTRAINT(0x08d4, 0x8), EVENT_CONSTRAINT_END }; @@ -1500,6 +1503,13 @@ struct event_constraint intel_lnc_pebs_event_constraints[] = { INTEL_FLAGS_UEVENT_CONSTRAINT(0x100, 0x100000000ULL), /* INST_RETIRED.PREC_DIST */ INTEL_FLAGS_UEVENT_CONSTRAINT(0x0400, 0x800000000ULL), + INTEL_FLAGS_UEVENT_CONSTRAINT(0x012a, 0x1), /* OCR.* events */ + INTEL_FLAGS_UEVENT_CONSTRAINT(0x012b, 0x1), /* OCR.* events */ + + INTEL_FLAGS_UEVENT_CONSTRAINT(0x04a4, 0x1), /* TOPDOWN.BAD_SPEC_SLOTS */ + INTEL_FLAGS_UEVENT_CONSTRAINT(0x08a4, 0x1), /* TOPDOWN.BR_MISPREDICT_SLOTS */ + INTEL_FLAGS_UEVENT_CONSTRAINT(0x10a4, 0x8), /* TOPDOWN.MEMORY_BOUND_SLOTS */ + INTEL_HYBRID_LDLAT_CONSTRAINT(0x1cd, 0x3fc), INTEL_HYBRID_STLAT_CONSTRAINT(0x2cd, 0x3), INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x11d0, 0xf), /* MEM_INST_RETIRED.STLB_MISS_LOADS */ @@ -1509,6 +1519,7 @@ struct event_constraint intel_lnc_pebs_event_constraints[] = { INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x42d0, 0xf), /* MEM_INST_RETIRED.SPLIT_STORES */ INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x81d0, 0xf), /* MEM_INST_RETIRED.ALL_LOADS */ INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x82d0, 0xf), /* MEM_INST_RETIRED.ALL_STORES */ + INTEL_FLAGS_UEVENT_CONSTRAINT(0x87d0, 0x3ff), /* MEM_INST_RETIRED.ANY */ INTEL_FLAGS_EVENT_CONSTRAINT_DATALA_LD_RANGE(0xd1, 0xd4, 0xf), diff --git a/arch/x86/events/intel/uncore.c b/arch/x86/events/intel/uncore.c index e9cc1ba921c5..7857959c6e82 100644 --- a/arch/x86/events/intel/uncore.c +++ b/arch/x86/events/intel/uncore.c @@ -83,6 +83,25 @@ int uncore_device_to_die(struct pci_dev *dev) return -1; } +/* + * Using cpus_read_lock() to ensure cpu is not going down between + * looking at cpu_online_mask. + * + * The lock must be held by the caller. + */ +int uncore_die_to_cpu(int die) +{ + int res = -1, cpu; + + for_each_online_cpu(cpu) { + if (topology_logical_die_id(cpu) == die) { + res = cpu; + break; + } + } + return res; +} + static void uncore_free_pcibus_map(void) { struct pci2phy_map *map, *tmp; @@ -1697,7 +1716,7 @@ err: return ret; } -static int uncore_mmio_global_init(u64 ctl) +static int uncore_mmio_global_init(int die, u64 ctl) { void __iomem *io_addr; @@ -1712,6 +1731,16 @@ static int uncore_mmio_global_init(u64 ctl) return 0; } +static int uncore_msr_global_init(int die, u64 msr) +{ + int cpu = uncore_die_to_cpu(die); + + if (cpu == -1) + return -ENODEV; + + return wrmsrq_on_cpu(cpu, msr, 0); +} + static const struct uncore_plat_init nhm_uncore_init __initconst = { .cpu_init = nhm_uncore_cpu_init, }; @@ -1852,6 +1881,7 @@ static const struct uncore_plat_init gnr_uncore_init __initconst = { .domain[0].base_is_pci = true, .domain[0].discovery_base = UNCORE_DISCOVERY_TABLE_DEVICE, .domain[0].units_ignore = gnr_uncore_units_ignore, + .domain[0].global_init = uncore_msr_global_init, }; static const struct uncore_plat_init dmr_uncore_init __initconst = { diff --git a/arch/x86/events/intel/uncore.h b/arch/x86/events/intel/uncore.h index c35918c01afa..c2e5ccb1d72c 100644 --- a/arch/x86/events/intel/uncore.h +++ b/arch/x86/events/intel/uncore.h @@ -53,7 +53,7 @@ struct uncore_discovery_domain { /* MSR address or PCI device used as the discovery base */ u32 discovery_base; bool base_is_pci; - int (*global_init)(u64 ctl); + int (*global_init)(int die, u64 ctl); /* The units in the discovery table should be ignored. */ int *units_ignore; @@ -235,6 +235,7 @@ struct pci2phy_map *__find_pci2phy_map(int segment); int uncore_pcibus_to_dieid(struct pci_bus *bus); int uncore_die_to_segment(int die); int uncore_device_to_die(struct pci_dev *dev); +int uncore_die_to_cpu(int die); ssize_t uncore_event_show(struct device *dev, struct device_attribute *attr, char *buf); diff --git a/arch/x86/events/intel/uncore_discovery.c b/arch/x86/events/intel/uncore_discovery.c index 583cbd06b9b8..e50776222256 100644 --- a/arch/x86/events/intel/uncore_discovery.c +++ b/arch/x86/events/intel/uncore_discovery.c @@ -287,7 +287,7 @@ static int __parse_discovery_table(struct uncore_discovery_domain *domain, if (!io_addr) return -ENOMEM; - if (domain->global_init && domain->global_init(global.ctl)) { + if (domain->global_init && domain->global_init(die, global.ctl)) { ret = -ENODEV; goto out; } @@ -399,7 +399,6 @@ static bool uncore_discovery_msr(struct uncore_discovery_domain *domain) if (!die_mask) return false; - cpus_read_lock(); for_each_online_cpu(cpu) { die = topology_logical_die_id(cpu); if (__test_and_set_bit(die, die_mask)) @@ -414,8 +413,6 @@ static bool uncore_discovery_msr(struct uncore_discovery_domain *domain) __parse_discovery_table(domain, base, die, &parsed); } - cpus_read_unlock(); - kfree(die_mask); return parsed; } @@ -429,10 +426,14 @@ bool uncore_discovery(struct uncore_plat_init *init) for (i = 0; i < UNCORE_DISCOVERY_DOMAINS; i++) { domain = &init->domain[i]; if (domain->discovery_base) { + cpus_read_lock(); + if (!domain->base_is_pci) ret |= uncore_discovery_msr(domain); else ret |= uncore_discovery_pci(domain); + + cpus_read_unlock(); } } @@ -481,8 +482,8 @@ static u64 intel_generic_uncore_box_ctl(struct intel_uncore_box *box) struct intel_uncore_discovery_unit *unit; unit = intel_uncore_find_discovery_unit(box->pmu->type->boxes, - -1, box->pmu->pmu_idx); - if (WARN_ON_ONCE(!unit)) + box->dieid, box->pmu->pmu_idx); + if (!unit) return 0; return unit->addr; @@ -490,17 +491,28 @@ static u64 intel_generic_uncore_box_ctl(struct intel_uncore_box *box) void intel_generic_uncore_msr_init_box(struct intel_uncore_box *box) { - wrmsrq(intel_generic_uncore_box_ctl(box), GENERIC_PMON_BOX_CTL_INT); + u64 box_ctl = intel_generic_uncore_box_ctl(box); + + if (!box_ctl) + return; + + wrmsrq(box_ctl, GENERIC_PMON_BOX_CTL_INT); } void intel_generic_uncore_msr_disable_box(struct intel_uncore_box *box) { - wrmsrq(intel_generic_uncore_box_ctl(box), GENERIC_PMON_BOX_CTL_FRZ); + u64 box_ctl = intel_generic_uncore_box_ctl(box); + + if (box_ctl) + wrmsrq(box_ctl, GENERIC_PMON_BOX_CTL_FRZ); } void intel_generic_uncore_msr_enable_box(struct intel_uncore_box *box) { - wrmsrq(intel_generic_uncore_box_ctl(box), 0); + u64 box_ctl = intel_generic_uncore_box_ctl(box); + + if (box_ctl) + wrmsrq(box_ctl, 0); } static void intel_generic_uncore_msr_enable_event(struct intel_uncore_box *box, @@ -549,6 +561,7 @@ bool intel_generic_uncore_assign_hw_event(struct perf_event *event, if (box->pci_dev) { box_ctl = UNCORE_DISCOVERY_PCI_BOX_CTRL(box_ctl); + hwc->config_base = box_ctl + uncore_pci_event_ctl(box, hwc->idx); hwc->event_base = box_ctl + uncore_pci_perf_ctr(box, hwc->idx); return true; @@ -567,27 +580,30 @@ static inline int intel_pci_uncore_box_ctl(struct intel_uncore_box *box) void intel_generic_uncore_pci_init_box(struct intel_uncore_box *box) { - struct pci_dev *pdev = box->pci_dev; int box_ctl = intel_pci_uncore_box_ctl(box); + if (!box_ctl) + return; + __set_bit(UNCORE_BOX_FLAG_CTL_OFFS8, &box->flags); - pci_write_config_dword(pdev, box_ctl, GENERIC_PMON_BOX_CTL_INT); + pci_write_config_dword(box->pci_dev, box_ctl, GENERIC_PMON_BOX_CTL_INT); } void intel_generic_uncore_pci_disable_box(struct intel_uncore_box *box) { - struct pci_dev *pdev = box->pci_dev; int box_ctl = intel_pci_uncore_box_ctl(box); - pci_write_config_dword(pdev, box_ctl, GENERIC_PMON_BOX_CTL_FRZ); + if (box_ctl) + pci_write_config_dword(box->pci_dev, box_ctl, + GENERIC_PMON_BOX_CTL_FRZ); } void intel_generic_uncore_pci_enable_box(struct intel_uncore_box *box) { - struct pci_dev *pdev = box->pci_dev; int box_ctl = intel_pci_uncore_box_ctl(box); - pci_write_config_dword(pdev, box_ctl, 0); + if (box_ctl) + pci_write_config_dword(box->pci_dev, box_ctl, 0); } static void intel_generic_uncore_pci_enable_event(struct intel_uncore_box *box, diff --git a/arch/x86/events/intel/uncore_snb.c b/arch/x86/events/intel/uncore_snb.c index 3dbc6bacbd9d..edddd4f9ab5f 100644 --- a/arch/x86/events/intel/uncore_snb.c +++ b/arch/x86/events/intel/uncore_snb.c @@ -563,12 +563,6 @@ void tgl_uncore_cpu_init(void) skl_uncore_msr_ops.init_box = rkl_uncore_msr_init_box; } -static void adl_uncore_msr_init_box(struct intel_uncore_box *box) -{ - if (box->pmu->pmu_idx == 0) - wrmsrq(ADL_UNC_PERF_GLOBAL_CTL, SNB_UNC_GLOBAL_CTL_EN); -} - static void adl_uncore_msr_enable_box(struct intel_uncore_box *box) { wrmsrq(ADL_UNC_PERF_GLOBAL_CTL, SNB_UNC_GLOBAL_CTL_EN); @@ -587,7 +581,6 @@ static void adl_uncore_msr_exit_box(struct intel_uncore_box *box) } static struct intel_uncore_ops adl_uncore_msr_ops = { - .init_box = adl_uncore_msr_init_box, .enable_box = adl_uncore_msr_enable_box, .disable_box = adl_uncore_msr_disable_box, .exit_box = adl_uncore_msr_exit_box, diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index 215d33e260ed..334dc384b5b9 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -3704,25 +3704,6 @@ static int skx_msr_cpu_bus_read(int cpu, u64 *topology) return 0; } -static int die_to_cpu(int die) -{ - int res = 0, cpu, current_die; - /* - * Using cpus_read_lock() to ensure cpu is not going down between - * looking at cpu_online_mask. - */ - cpus_read_lock(); - for_each_online_cpu(cpu) { - current_die = topology_logical_die_id(cpu); - if (current_die == die) { - res = cpu; - break; - } - } - cpus_read_unlock(); - return res; -} - enum { IIO_TOPOLOGY_TYPE, UPI_TOPOLOGY_TYPE, @@ -3791,11 +3772,17 @@ static void pmu_free_topology(struct intel_uncore_type *type) static int skx_pmu_get_topology(struct intel_uncore_type *type, int (*topology_cb)(struct intel_uncore_type*, int, int, u64)) { - int die, ret = -EPERM; + int die, ret = -ENODEV; u64 cpu_bus_msr; + int cpu; + cpus_read_lock(); for (die = 0; die < uncore_max_dies(); die++) { - ret = skx_msr_cpu_bus_read(die_to_cpu(die), &cpu_bus_msr); + cpu = uncore_die_to_cpu(die); + if (cpu == -1) + continue; + + ret = skx_msr_cpu_bus_read(cpu, &cpu_bus_msr); if (ret) break; @@ -3807,6 +3794,7 @@ static int skx_pmu_get_topology(struct intel_uncore_type *type, if (ret) break; } + cpus_read_unlock(); return ret; } @@ -4261,7 +4249,7 @@ err: static int skx_upi_topology_cb(struct intel_uncore_type *type, int segment, int die, u64 cpu_bus_msr) { - int idx, ret; + int idx, ret = 0; struct intel_uncore_topology *upi; unsigned int devfn; struct pci_dev *dev = NULL; @@ -4274,12 +4262,12 @@ static int skx_upi_topology_cb(struct intel_uncore_type *type, int segment, dev = pci_get_domain_bus_and_slot(segment, bus, devfn); if (dev) { ret = upi_fill_topology(dev, upi, idx); + pci_dev_put(dev); if (ret) break; } } - pci_dev_put(dev); return ret; } @@ -5499,6 +5487,7 @@ static int discover_upi_topology(struct intel_uncore_type *type, int ubox_did, i devfn); if (dev) { ret = upi_fill_topology(dev, upi, idx); + pci_dev_put(dev); if (ret) goto err; } @@ -5506,7 +5495,6 @@ static int discover_upi_topology(struct intel_uncore_type *type, int ubox_did, i } err: pci_dev_put(ubox); - pci_dev_put(dev); return ret; } diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h index 524668dcf4cc..eae24bb35dc1 100644 --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -322,10 +322,8 @@ struct cpu_hw_events { u64 fixed_ctrl_val; u64 active_fixed_ctrl_val; - /* Intel ACR configuration */ + /* Intel ACR/arch-PEBS configuration */ u64 acr_cfg_b[X86_PMC_IDX_MAX]; - u64 acr_cfg_c[X86_PMC_IDX_MAX]; - /* Cached CFG_C values */ u64 cfg_c_val[X86_PMC_IDX_MAX]; /* @@ -1712,7 +1710,9 @@ extern struct event_constraint intel_glp_pebs_event_constraints[]; extern struct event_constraint intel_grt_pebs_event_constraints[]; -extern struct event_constraint intel_arw_pebs_event_constraints[]; +extern struct event_constraint intel_cmt_pebs_event_constraints[]; + +extern struct event_constraint intel_dkt_pebs_event_constraints[]; extern struct event_constraint intel_nehalem_pebs_event_constraints[]; diff --git a/arch/x86/include/asm/futex_robust.h b/arch/x86/include/asm/futex_robust.h new file mode 100644 index 000000000000..e87954703ae2 --- /dev/null +++ b/arch/x86/include/asm/futex_robust.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_FUTEX_ROBUST_H +#define _ASM_X86_FUTEX_ROBUST_H + +#include + +static __always_inline void __user *x86_futex_robust_unlock_get_pop(struct pt_regs *regs) +{ + /* + * If ZF is set then the cmpxchg succeeded and the pending op pointer + * needs to be cleared. + */ + return regs->flags & X86_EFLAGS_ZF ? (void __user *)regs->dx : NULL; +} + +#define arch_futex_robust_unlock_get_pop(regs) \ + x86_futex_robust_unlock_get_pop(regs) + +#endif /* _ASM_X86_FUTEX_ROBUST_H */ diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 409981468cba..cef9a4ca9841 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -40,12 +40,10 @@ #endif #define __percpu_prefix -#define __percpu_seg_override CONCATENATE(__seg_, __percpu_seg) #else /* !CONFIG_CC_HAS_NAMED_AS: */ #define __percpu_prefix __force_percpu_prefix -#define __percpu_seg_override #endif /* CONFIG_CC_HAS_NAMED_AS */ @@ -82,7 +80,6 @@ #define __force_percpu_prefix #define __percpu_prefix -#define __percpu_seg_override #define PER_CPU_VAR(var) (var)__percpu_rel @@ -92,8 +89,6 @@ # define __my_cpu_type(var) typeof(var) # define __my_cpu_ptr(ptr) (ptr) # define __my_cpu_var(var) (var) - -# define __percpu_qual __percpu_seg_override #else # define __my_cpu_type(var) typeof(var) __percpu_seg_override # define __my_cpu_ptr(ptr) (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr) diff --git a/arch/x86/include/asm/percpu_types.h b/arch/x86/include/asm/percpu_types.h new file mode 100644 index 000000000000..0aa3e47a3643 --- /dev/null +++ b/arch/x86/include/asm/percpu_types.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_PERCPU_TYPES_H +#define _ASM_X86_PERCPU_TYPES_H + +#if defined(CONFIG_SMP) && defined(CONFIG_CC_HAS_NAMED_AS) +#define __percpu_seg_override CONCATENATE(__seg_, __percpu_seg) +#else /* !CONFIG_CC_HAS_NAMED_AS: */ +#define __percpu_seg_override +#endif + +#if defined(CONFIG_USE_X86_SEG_SUPPORT) && defined(USE_TYPEOF_UNQUAL) +#define __percpu_qual __percpu_seg_override +#endif + +#include + +#endif diff --git a/arch/x86/include/asm/vdso.h b/arch/x86/include/asm/vdso.h index f2d49212ae90..4e735158c1b2 100644 --- a/arch/x86/include/asm/vdso.h +++ b/arch/x86/include/asm/vdso.h @@ -24,6 +24,10 @@ struct vdso_image { long sym_int80_landing_pad; long sym_vdso32_sigreturn_landing_pad; long sym_vdso32_rt_sigreturn_landing_pad; + long sym___futex_list64_try_unlock_cs_start; + long sym___futex_list64_try_unlock_cs_end; + long sym___futex_list32_try_unlock_cs_start; + long sym___futex_list32_try_unlock_cs_end; }; extern const struct vdso_image vdso64_image; diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c index b5991d53fc0e..cb3d0ca1fa22 100644 --- a/arch/x86/kernel/kvmclock.c +++ b/arch/x86/kernel/kvmclock.c @@ -87,6 +87,27 @@ static u64 kvm_clock_get_cycles(struct clocksource *cs) return kvm_clock_read(); } +static u64 kvm_clock_get_cycles_snapshot(struct clocksource *cs, + struct clocksource_hw_snapshot *chs) +{ + struct pvclock_vcpu_time_info *src; + unsigned version; + u64 ret, tsc; + + preempt_disable_notrace(); + src = this_cpu_pvti(); + do { + version = pvclock_read_begin(src); + tsc = rdtsc_ordered(); + ret = __pvclock_read_cycles(src, tsc); + } while (pvclock_read_retry(src, version)); + preempt_enable_notrace(); + + chs->hw_cycles = tsc; + chs->hw_csid = CSID_X86_TSC; + return ret; +} + static noinstr u64 kvm_sched_clock_read(void) { return pvclock_clocksource_read_nowd(this_cpu_pvti()) - kvm_sched_clock_offset; @@ -156,13 +177,14 @@ static int kvm_cs_enable(struct clocksource *cs) } static struct clocksource kvm_clock = { - .name = "kvm-clock", - .read = kvm_clock_get_cycles, - .rating = 400, - .mask = CLOCKSOURCE_MASK(64), - .flags = CLOCK_SOURCE_IS_CONTINUOUS, - .id = CSID_X86_KVM_CLK, - .enable = kvm_cs_enable, + .name = "kvm-clock", + .read = kvm_clock_get_cycles, + .read_snapshot = kvm_clock_get_cycles_snapshot, + .rating = 400, + .mask = CLOCKSOURCE_MASK(64), + .flags = CLOCK_SOURCE_IS_CONTINUOUS, + .id = CSID_X86_KVM_CLK, + .enable = kvm_cs_enable, }; static void kvm_register_clock(char *txt) diff --git a/arch/x86/tools/vdso2c.c b/arch/x86/tools/vdso2c.c index b8a555763f43..64a636b9fc37 100644 --- a/arch/x86/tools/vdso2c.c +++ b/arch/x86/tools/vdso2c.c @@ -75,12 +75,16 @@ struct vdso_sym { }; struct vdso_sym required_syms[] = { - {"__kernel_vsyscall", true}, - {"__kernel_sigreturn", true}, - {"__kernel_rt_sigreturn", true}, - {"int80_landing_pad", true}, - {"vdso32_rt_sigreturn_landing_pad", true}, - {"vdso32_sigreturn_landing_pad", true}, + {"__kernel_vsyscall", true}, + {"__kernel_sigreturn", true}, + {"__kernel_rt_sigreturn", true}, + {"int80_landing_pad", true}, + {"vdso32_rt_sigreturn_landing_pad", true}, + {"vdso32_sigreturn_landing_pad", true}, + {"__futex_list64_try_unlock_cs_start", true}, + {"__futex_list64_try_unlock_cs_end", true}, + {"__futex_list32_try_unlock_cs_start", true}, + {"__futex_list32_try_unlock_cs_end", true}, }; __attribute__((format(printf, 1, 2))) __attribute__((noreturn)) diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c index e9f5034a1bc8..df567795d175 100644 --- a/drivers/clocksource/hyperv_timer.c +++ b/drivers/clocksource/hyperv_timer.c @@ -444,6 +444,22 @@ static u64 notrace read_hv_clock_tsc_cs(struct clocksource *arg) return read_hv_clock_tsc(); } +static u64 notrace read_hv_clock_tsc_cs_snapshot(struct clocksource *arg, + struct clocksource_hw_snapshot *chs) +{ + u64 time; + + if (hv_read_tsc_page_tsc(tsc_page, &chs->hw_cycles, &time)) { + chs->hw_csid = CSID_X86_TSC; + } else { + chs->hw_cycles = 0; + chs->hw_csid = CSID_GENERIC; + time = read_hv_clock_msr(); + } + + return time; +} + static u64 noinstr read_hv_sched_clock_tsc(void) { return (read_hv_clock_tsc() - hv_sched_clock_offset) * @@ -492,18 +508,19 @@ static int hv_cs_enable(struct clocksource *cs) #endif static struct clocksource hyperv_cs_tsc = { - .name = "hyperv_clocksource_tsc_page", - .rating = 500, - .read = read_hv_clock_tsc_cs, - .mask = CLOCKSOURCE_MASK(64), - .flags = CLOCK_SOURCE_IS_CONTINUOUS, - .suspend= suspend_hv_clock_tsc, - .resume = resume_hv_clock_tsc, + .name = "hyperv_clocksource_tsc_page", + .rating = 500, + .read = read_hv_clock_tsc_cs, + .read_snapshot = read_hv_clock_tsc_cs_snapshot, + .mask = CLOCKSOURCE_MASK(64), + .flags = CLOCK_SOURCE_IS_CONTINUOUS, + .suspend = suspend_hv_clock_tsc, + .resume = resume_hv_clock_tsc, #ifdef HAVE_VDSO_CLOCKMODE_HVCLOCK - .enable = hv_cs_enable, - .vdso_clock_mode = VDSO_CLOCKMODE_HVCLOCK, + .enable = hv_cs_enable, + .vdso_clock_mode = VDSO_CLOCKMODE_HVCLOCK, #else - .vdso_clock_mode = VDSO_CLOCKMODE_NONE, + .vdso_clock_mode = VDSO_CLOCKMODE_NONE, #endif }; diff --git a/drivers/clocksource/mips-gic-timer.c b/drivers/clocksource/mips-gic-timer.c index 1501c7db9a8e..a1669266c94d 100644 --- a/drivers/clocksource/mips-gic-timer.c +++ b/drivers/clocksource/mips-gic-timer.c @@ -198,7 +198,9 @@ static struct clocksource gic_clocksource = { .name = "GIC", .read = gic_hpt_read, .flags = CLOCK_SOURCE_IS_CONTINUOUS, +#ifdef CONFIG_GENERIC_GETTIMEOFDAY .vdso_clock_mode = VDSO_CLOCKMODE_GIC, +#endif }; static void gic_clocksource_unstable(char *reason) diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index f59658d40a44..96fc4b60ed9b 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -132,38 +132,11 @@ struct kobject *get_governor_parent_kobj(struct cpufreq_policy *policy) } EXPORT_SYMBOL_GPL(get_governor_parent_kobj); -static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall) -{ - struct kernel_cpustat kcpustat; - u64 cur_wall_time; - u64 idle_time; - u64 busy_time; - - cur_wall_time = jiffies64_to_nsecs(get_jiffies_64()); - - kcpustat_cpu_fetch(&kcpustat, cpu); - - busy_time = kcpustat.cpustat[CPUTIME_USER]; - busy_time += kcpustat.cpustat[CPUTIME_SYSTEM]; - busy_time += kcpustat.cpustat[CPUTIME_IRQ]; - busy_time += kcpustat.cpustat[CPUTIME_SOFTIRQ]; - busy_time += kcpustat.cpustat[CPUTIME_STEAL]; - busy_time += kcpustat.cpustat[CPUTIME_NICE]; - - idle_time = cur_wall_time - busy_time; - if (wall) - *wall = div_u64(cur_wall_time, NSEC_PER_USEC); - - return div_u64(idle_time, NSEC_PER_USEC); -} - u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy) { u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL); - if (idle_time == -1ULL) - return get_cpu_idle_time_jiffy(cpu, wall); - else if (!io_busy) + if (!io_busy) idle_time += get_cpu_iowait_time_us(cpu, wall); return idle_time; diff --git a/drivers/cpufreq/cpufreq_governor.c b/drivers/cpufreq/cpufreq_governor.c index 8a85bd32defe..710d93ec89b5 100644 --- a/drivers/cpufreq/cpufreq_governor.c +++ b/drivers/cpufreq/cpufreq_governor.c @@ -112,7 +112,7 @@ void gov_update_cpu_data(struct dbs_data *dbs_data) j_cdbs->prev_cpu_idle = get_cpu_idle_time(j, &j_cdbs->prev_update_time, dbs_data->io_is_busy); - j_cdbs->prev_cpu_nice = kcpustat_field(&kcpustat_cpu(j), CPUTIME_NICE, j); + j_cdbs->prev_cpu_nice = kcpustat_field(CPUTIME_NICE, j); } mutex_unlock(&policy_dbs->update_mutex); } @@ -180,7 +180,7 @@ unsigned int dbs_update(struct cpufreq_policy *policy) * stale-baseline spike (the delta will be at most one sampling * interval of accumulated nice time, not since boot). */ - cur_nice = kcpustat_field(&kcpustat_cpu(j), CPUTIME_NICE, j); + cur_nice = kcpustat_field(CPUTIME_NICE, j); if (ignore_nice) idle_time += div_u64(cur_nice - j_cdbs->prev_cpu_nice, NSEC_PER_USEC); @@ -552,7 +552,7 @@ int cpufreq_dbs_governor_start(struct cpufreq_policy *policy) * Make the first invocation of dbs_update() compute the load. */ j_cdbs->prev_load = 0; - j_cdbs->prev_cpu_nice = kcpustat_field(&kcpustat_cpu(j), CPUTIME_NICE, j); + j_cdbs->prev_cpu_nice = kcpustat_field(CPUTIME_NICE, j); } mutex_unlock(&policy_dbs->update_mutex); diff --git a/drivers/macintosh/rack-meter.c b/drivers/macintosh/rack-meter.c index 8a1e2c08b096..26cb93191ede 100644 --- a/drivers/macintosh/rack-meter.c +++ b/drivers/macintosh/rack-meter.c @@ -87,7 +87,7 @@ static inline u64 get_cpu_idle_time(unsigned int cpu) kcpustat->cpustat[CPUTIME_IOWAIT]; if (rackmeter_ignore_nice) - retval += kcpustat_field(kcpustat, CPUTIME_NICE, cpu); + retval += kcpustat_field(CPUTIME_NICE, cpu); return retval; } diff --git a/drivers/net/dsa/sja1105/sja1105_main.c b/drivers/net/dsa/sja1105/sja1105_main.c index c72c2bfdcffb..2697073dbf90 100644 --- a/drivers/net/dsa/sja1105/sja1105_main.c +++ b/drivers/net/dsa/sja1105/sja1105_main.c @@ -2310,10 +2310,10 @@ int sja1105_static_config_reload(struct sja1105_private *priv, goto out; } - t1 = timespec64_to_ns(&ptp_sts_before.pre_ts); - t2 = timespec64_to_ns(&ptp_sts_before.post_ts); - t3 = timespec64_to_ns(&ptp_sts_after.pre_ts); - t4 = timespec64_to_ns(&ptp_sts_after.post_ts); + t1 = ktime_to_ns(ptp_sts_before.pre_sts.systime); + t2 = ktime_to_ns(ptp_sts_before.post_sts.systime); + t3 = ktime_to_ns(ptp_sts_after.pre_sts.systime); + t4 = ktime_to_ns(ptp_sts_after.post_sts.systime); /* Mid point, corresponds to pre-reset PTPCLKVAL */ t12 = t1 + (t2 - t1) / 2; /* Mid point, corresponds to post-reset PTPCLKVAL, aka 0 */ diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.c b/drivers/net/ethernet/intel/ice/ice_ptp.c index 36df742c326c..f9e4ec6f7ebb 100644 --- a/drivers/net/ethernet/intel/ice/ice_ptp.c +++ b/drivers/net/ethernet/intel/ice/ice_ptp.c @@ -2065,11 +2065,13 @@ static const struct ice_crosststamp_cfg ice_crosststamp_cfg_e830 = { /** * struct ice_crosststamp_ctx - Device cross timestamp context * @snapshot: snapshot of system clocks for historic interpolation + * @snapshot_clock_id: System clock ID for @snapshot * @pf: pointer to the PF private structure * @cfg: pointer to hardware configuration for cross timestamp */ struct ice_crosststamp_ctx { struct system_time_snapshot snapshot; + clockid_t snapshot_clock_id; struct ice_pf *pf; const struct ice_crosststamp_cfg *cfg; }; @@ -2115,7 +2117,7 @@ static int ice_capture_crosststamp(ktime_t *device, } /* Snapshot system time for historic interpolation */ - ktime_get_snapshot(&ctx->snapshot); + ktime_get_snapshot_id(ctx->snapshot_clock_id, &ctx->snapshot); /* Program cmd to master timer */ ice_ptp_src_cmd(hw, ICE_PTP_READ_TIME); @@ -2176,6 +2178,7 @@ static int ice_ptp_getcrosststamp(struct ptp_clock_info *info, { struct ice_pf *pf = ptp_info_to_pf(info); struct ice_crosststamp_ctx ctx = { + .snapshot_clock_id = cts->clock_id, .pf = pf, }; diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h index 17236813965d..46d625b15f44 100644 --- a/drivers/net/ethernet/intel/igc/igc.h +++ b/drivers/net/ethernet/intel/igc/igc.h @@ -326,6 +326,7 @@ struct igc_adapter { struct timespec64 prev_ptp_time; /* Pre-reset PTP clock */ ktime_t ptp_reset_start; /* Reset time in clock mono */ struct system_time_snapshot snapshot; + clockid_t snapshot_clock_id; struct mutex ptm_lock; /* Only allow one PTM transaction at a time */ char fw_version[32]; diff --git a/drivers/net/ethernet/intel/igc/igc_ptp.c b/drivers/net/ethernet/intel/igc/igc_ptp.c index 3d6b2264164a..b40aba9ab685 100644 --- a/drivers/net/ethernet/intel/igc/igc_ptp.c +++ b/drivers/net/ethernet/intel/igc/igc_ptp.c @@ -1049,7 +1049,7 @@ static int igc_phc_get_syncdevicetime(ktime_t *device, */ do { /* Get a snapshot of system clocks to use as historic value. */ - ktime_get_snapshot(&adapter->snapshot); + ktime_get_snapshot_id(adapter->snapshot_clock_id, &adapter->snapshot); igc_ptm_trigger(hw); @@ -1103,6 +1103,8 @@ static int igc_ptp_getcrosststamp(struct ptp_clock_info *ptp, /* This blocks until any in progress PTM transactions complete */ mutex_lock(&adapter->ptm_lock); + adapter->snapshot_clock_id = cts->clock_id; + ret = get_device_system_crosststamp(igc_phc_get_syncdevicetime, adapter, &adapter->snapshot, cts); mutex_unlock(&adapter->ptm_lock); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c index d785f1b4f2e1..5df786133e4b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/clock.c @@ -340,7 +340,7 @@ static int mlx5_ptp_getcrosststamp(struct ptp_clock_info *ptp, goto unlock; } - ktime_get_snapshot(&history_begin); + ktime_get_snapshot_id(cts->clock_id, &history_begin); err = get_device_system_crosststamp(mlx5_mtctr_syncdevicetime, mdev, &history_begin, cts); @@ -366,7 +366,7 @@ static int mlx5_ptp_getcrosscycles(struct ptp_clock_info *ptp, goto unlock; } - ktime_get_snapshot(&history_begin); + ktime_get_snapshot_id(cts->clock_id, &history_begin); err = get_device_system_crosststamp(mlx5_mtctr_syncdevicecyclestime, mdev, &history_begin, cts); diff --git a/drivers/net/wireless/intel/iwlwifi/mld/ptp.c b/drivers/net/wireless/intel/iwlwifi/mld/ptp.c index c65f4b56a327..f829156d42b3 100644 --- a/drivers/net/wireless/intel/iwlwifi/mld/ptp.c +++ b/drivers/net/wireless/intel/iwlwifi/mld/ptp.c @@ -250,7 +250,8 @@ iwl_mld_phc_get_crosstimestamp(struct ptp_clock_info *ptp, /* System (wall) time */ ktime_t sys_time; - memset(xtstamp, 0, sizeof(struct system_device_crosststamp)); + if (xtstamp->clock_id != CLOCK_REALTIME) + return -ENOTSUPP; ret = iwl_mld_get_crosstimestamp_fw(mld, &gp2, &sys_time); if (ret) { @@ -270,7 +271,7 @@ iwl_mld_phc_get_crosstimestamp(struct ptp_clock_info *ptp, /* System monotonic raw time is not used */ xtstamp->device = ns_to_ktime(gp2_ns); - xtstamp->sys_realtime = sys_time; + xtstamp->sys_systime = sys_time; return ret; } diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ptp.c b/drivers/net/wireless/intel/iwlwifi/mvm/ptp.c index f7b620136c85..bcd6f7cead2a 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/ptp.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/ptp.c @@ -160,13 +160,14 @@ iwl_mvm_phc_get_crosstimestamp(struct ptp_clock_info *ptp, /* System (wall) time */ ktime_t sys_time; - memset(xtstamp, 0, sizeof(struct system_device_crosststamp)); - if (!mvm->ptp_data.ptp_clock) { IWL_ERR(mvm, "No PHC clock registered\n"); return -ENODEV; } + if (xtstamp->clock_id != CLOCK_REALTIME) + return -ENOTSUPP; + mutex_lock(&mvm->mutex); if (fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_SYNCED_TIME)) { ret = iwl_mvm_get_crosstimestamp_fw(mvm, &gp2, &sys_time); @@ -184,7 +185,7 @@ iwl_mvm_phc_get_crosstimestamp(struct ptp_clock_info *ptp, /* System monotonic raw time is not used */ xtstamp->device = (ktime_t)gp2_ns; - xtstamp->sys_realtime = sys_time; + xtstamp->sys_systime = sys_time; out: mutex_unlock(&mvm->mutex); diff --git a/drivers/nvdimm/nd.h b/drivers/nvdimm/nd.h index b199eea3260e..18b64559664b 100644 --- a/drivers/nvdimm/nd.h +++ b/drivers/nvdimm/nd.h @@ -632,8 +632,11 @@ u64 nd_region_interleave_set_cookie(struct nd_region *nd_region, u64 nd_region_interleave_set_altcookie(struct nd_region *nd_region); void nvdimm_bus_lock(struct device *dev); void nvdimm_bus_unlock(struct device *dev); -DEFINE_GUARD(nvdimm_bus, struct device *, - if (_T) nvdimm_bus_lock(_T), if (_T) nvdimm_bus_unlock(_T)); +DEFINE_CLASS(nvdimm_bus, struct device *, + if (_T) nvdimm_bus_unlock(_T), + ({ if (_T) nvdimm_bus_lock(_T); _T; }), + struct device *_T); +DEFINE_CLASS_IS_GUARD(nvdimm_bus); bool is_nvdimm_bus_locked(struct device *dev); void nvdimm_check_and_set_ro(struct gendisk *disk); diff --git a/drivers/pps/generators/pps_gen-dummy.c b/drivers/pps/generators/pps_gen-dummy.c index 547fa7fe29f4..a4395543c4bb 100644 --- a/drivers/pps/generators/pps_gen-dummy.c +++ b/drivers/pps/generators/pps_gen-dummy.c @@ -39,11 +39,7 @@ static void pps_gen_ktimer_event(struct timer_list *unused) static int pps_gen_dummy_get_time(struct pps_gen_device *pps_gen, struct timespec64 *time) { - struct system_time_snapshot snap; - - ktime_get_snapshot(&snap); - *time = ktime_to_timespec64(snap.real); - + ktime_get_real_ts64(time); return 0; } diff --git a/drivers/pps/generators/pps_gen_tio.c b/drivers/pps/generators/pps_gen_tio.c index de00a85bfafa..9483d126ada0 100644 --- a/drivers/pps/generators/pps_gen_tio.c +++ b/drivers/pps/generators/pps_gen_tio.c @@ -189,11 +189,7 @@ static int pps_tio_gen_enable(struct pps_gen_device *pps_gen, bool enable) static int pps_tio_get_time(struct pps_gen_device *pps_gen, struct timespec64 *time) { - struct system_time_snapshot snap; - - ktime_get_snapshot(&snap); - *time = ktime_to_timespec64(snap.real); - + ktime_get_real_ts64(time); return 0; } diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c index c61cf9edac48..dc23cd708cfe 100644 --- a/drivers/ptp/ptp_chardev.c +++ b/drivers/ptp/ptp_chardev.c @@ -317,8 +317,8 @@ typedef int (*ptp_crosststamp_fn)(struct ptp_clock_info *, static long ptp_sys_offset_precise(struct ptp_clock *ptp, void __user *arg, ptp_crosststamp_fn crosststamp_fn) { + struct system_device_crosststamp xtstamp = { .clock_id = CLOCK_REALTIME }; struct ptp_sys_offset_precise precise_offset; - struct system_device_crosststamp xtstamp; struct timespec64 ts; int err; @@ -333,7 +333,7 @@ static long ptp_sys_offset_precise(struct ptp_clock *ptp, void __user *arg, ts = ktime_to_timespec64(xtstamp.device); precise_offset.device.sec = ts.tv_sec; precise_offset.device.nsec = ts.tv_nsec; - ts = ktime_to_timespec64(xtstamp.sys_realtime); + ts = ktime_to_timespec64(xtstamp.sys_systime); precise_offset.sys_realtime.sec = ts.tv_sec; precise_offset.sys_realtime.nsec = ts.tv_nsec; ts = ktime_to_timespec64(xtstamp.sys_monoraw); @@ -386,15 +386,19 @@ static long ptp_sys_offset_extended(struct ptp_clock *ptp, void __user *arg, return err; /* Filter out disabled or unavailable clocks */ - if (sts.pre_ts.tv_sec < 0 || sts.post_ts.tv_sec < 0) + if (!sts.pre_sts.valid || !sts.post_sts.valid) return -EINVAL; - extoff->ts[i][0].sec = sts.pre_ts.tv_sec; - extoff->ts[i][0].nsec = sts.pre_ts.tv_nsec; extoff->ts[i][1].sec = ts.tv_sec; extoff->ts[i][1].nsec = ts.tv_nsec; - extoff->ts[i][2].sec = sts.post_ts.tv_sec; - extoff->ts[i][2].nsec = sts.post_ts.tv_nsec; + + ts = ktime_to_timespec64(sts.pre_sts.systime); + extoff->ts[i][0].sec = ts.tv_sec; + extoff->ts[i][0].nsec = ts.tv_nsec; + + ts = ktime_to_timespec64(sts.post_sts.systime); + extoff->ts[i][2].sec = ts.tv_sec; + extoff->ts[i][2].nsec = ts.tv_nsec; } return copy_to_user(arg, extoff, sizeof(*extoff)) ? -EFAULT : 0; diff --git a/drivers/ptp/ptp_ocp.c b/drivers/ptp/ptp_ocp.c index 735385539b9f..8231c2d7785e 100644 --- a/drivers/ptp/ptp_ocp.c +++ b/drivers/ptp/ptp_ocp.c @@ -1491,11 +1491,8 @@ __ptp_ocp_gettime_locked(struct ptp_ocp *bp, struct timespec64 *ts, } ptp_read_system_postts(sts); - if (sts && bp->ts_window_adjust) { - s64 ns = timespec64_to_ns(&sts->post_ts); - - sts->post_ts = ns_to_timespec64(ns - bp->ts_window_adjust); - } + if (sts && bp->ts_window_adjust) + sts->post_sts.systime -= bp->ts_window_adjust; time_ns = ioread32(&bp->reg->time_ns); time_sec = ioread32(&bp->reg->time_sec); @@ -4600,8 +4597,8 @@ ptp_ocp_summary_show(struct seq_file *s, void *data) struct timespec64 sys_ts; s64 pre_ns, post_ns, ns; - pre_ns = timespec64_to_ns(&sts.pre_ts); - post_ns = timespec64_to_ns(&sts.post_ts); + pre_ns = ktime_to_ns(sts.pre_sts.systime); + post_ns = ktime_to_ns(sts.post_sts.systime); ns = (pre_ns + post_ns) / 2; ns += (s64)bp->utc_tai_offset * NSEC_PER_SEC; sys_ts = ns_to_timespec64(ns); diff --git a/drivers/ptp/ptp_vmclock.c b/drivers/ptp/ptp_vmclock.c index 8b630eb916b5..eebdcd5ebc08 100644 --- a/drivers/ptp/ptp_vmclock.c +++ b/drivers/ptp/ptp_vmclock.c @@ -101,7 +101,6 @@ static int vmclock_get_crosststamp(struct vmclock_state *st, struct timespec64 *tspec) { ktime_t deadline = ktime_add(ktime_get(), VMCLOCK_MAX_WAIT); - struct system_time_snapshot systime_snapshot; uint64_t cycle, delta, seq, frac_sec; #ifdef CONFIG_X86 @@ -132,17 +131,19 @@ static int vmclock_get_crosststamp(struct vmclock_state *st, * will be derived from the *same* counter value. * * If the system isn't using the same counter, then the value - * from ktime_get_snapshot() will still be used as pre_ts, and - * ptp_read_system_postts() is called to populate postts after - * calling get_cycles(). - * - * The conversion to timespec64 happens further down, outside - * the seq_count loop. + * from ptp_read_system_prets() will still be used as pre_ts, + * and ptp_read_system_postts() is called to populate postts + * after calling get_cycles(). */ if (sts) { - ktime_get_snapshot(&systime_snapshot); - if (systime_snapshot.cs_id == st->cs_id) { - cycle = systime_snapshot.cycles; + ptp_read_system_prets(sts); + if (sts->pre_sts.cs_id == st->cs_id) { + cycle = sts->pre_sts.cycles; + sts->post_sts = sts->pre_sts; + } else if (sts->pre_sts.hw_csid == st->cs_id && + sts->pre_sts.hw_cycles) { + cycle = sts->pre_sts.hw_cycles; + sts->post_sts = sts->pre_sts; } else { cycle = get_cycles(); ptp_read_system_postts(sts); @@ -180,12 +181,6 @@ static int vmclock_get_crosststamp(struct vmclock_state *st, system_counter->cs_id = st->cs_id; } - if (sts) { - sts->pre_ts = ktime_to_timespec64(systime_snapshot.real); - if (systime_snapshot.cs_id == st->cs_id) - sts->post_ts = sts->pre_ts; - } - return 0; } @@ -272,7 +267,7 @@ static int ptp_vmclock_getcrosststamp(struct ptp_clock_info *ptp, if (ret == -ENODEV) { struct system_time_snapshot systime_snapshot; - ktime_get_snapshot(&systime_snapshot); + ktime_get_snapshot_id(CLOCK_REALTIME, &systime_snapshot); if (systime_snapshot.cs_id == CSID_X86_TSC || systime_snapshot.cs_id == CSID_X86_KVM_CLK) { diff --git a/drivers/virtio/virtio_rtc_ptp.c b/drivers/virtio/virtio_rtc_ptp.c index f84599950cd4..ff8d834493dc 100644 --- a/drivers/virtio/virtio_rtc_ptp.c +++ b/drivers/virtio/virtio_rtc_ptp.c @@ -139,7 +139,7 @@ static int viortc_ptp_getcrosststamp(struct ptp_clock_info *ptp, if (ret) return ret; - ktime_get_snapshot(&history_begin); + ktime_get_snapshot_id(xtstamp->clock_id, &history_begin); if (history_begin.cs_id != cs_id) return -EOPNOTSUPP; diff --git a/fs/proc/stat.c b/fs/proc/stat.c index 20c3df9a9b80..79b42f2bfd00 100644 --- a/fs/proc/stat.c +++ b/fs/proc/stat.c @@ -19,38 +19,6 @@ #define arch_irq_stat_cpu(cpu) 0 #endif -u64 get_idle_time(struct kernel_cpustat *kcs, int cpu) -{ - u64 idle, idle_usecs = -1ULL; - - if (cpu_online(cpu)) - idle_usecs = get_cpu_idle_time_us(cpu, NULL); - - if (idle_usecs == -1ULL) - /* !NO_HZ or cpu offline so we can rely on cpustat.idle */ - idle = kcs->cpustat[CPUTIME_IDLE]; - else - idle = idle_usecs * NSEC_PER_USEC; - - return idle; -} - -static u64 get_iowait_time(struct kernel_cpustat *kcs, int cpu) -{ - u64 iowait, iowait_usecs = -1ULL; - - if (cpu_online(cpu)) - iowait_usecs = get_cpu_iowait_time_us(cpu, NULL); - - if (iowait_usecs == -1ULL) - /* !NO_HZ or cpu offline so we can rely on cpustat.iowait */ - iowait = kcs->cpustat[CPUTIME_IOWAIT]; - else - iowait = iowait_usecs * NSEC_PER_USEC; - - return iowait; -} - static void show_irq_gap(struct seq_file *p, unsigned int gap) { static const char zeros[] = " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0"; @@ -102,8 +70,8 @@ static int show_stat(struct seq_file *p, void *v) user += cpustat[CPUTIME_USER]; nice += cpustat[CPUTIME_NICE]; system += cpustat[CPUTIME_SYSTEM]; - idle += get_idle_time(&kcpustat, i); - iowait += get_iowait_time(&kcpustat, i); + idle += cpustat[CPUTIME_IDLE]; + iowait += cpustat[CPUTIME_IOWAIT]; irq += cpustat[CPUTIME_IRQ]; softirq += cpustat[CPUTIME_SOFTIRQ]; steal += cpustat[CPUTIME_STEAL]; @@ -142,8 +110,8 @@ static int show_stat(struct seq_file *p, void *v) user = cpustat[CPUTIME_USER]; nice = cpustat[CPUTIME_NICE]; system = cpustat[CPUTIME_SYSTEM]; - idle = get_idle_time(&kcpustat, i); - iowait = get_iowait_time(&kcpustat, i); + idle = cpustat[CPUTIME_IDLE]; + iowait = cpustat[CPUTIME_IOWAIT]; irq = cpustat[CPUTIME_IRQ]; softirq = cpustat[CPUTIME_SOFTIRQ]; steal = cpustat[CPUTIME_STEAL]; diff --git a/fs/proc/uptime.c b/fs/proc/uptime.c index b5343d209381..433aa947cd57 100644 --- a/fs/proc/uptime.c +++ b/fs/proc/uptime.c @@ -18,12 +18,8 @@ static int uptime_proc_show(struct seq_file *m, void *v) int i; idle_nsec = 0; - for_each_possible_cpu(i) { - struct kernel_cpustat kcs; - - kcpustat_cpu_fetch(&kcs, i); - idle_nsec += get_idle_time(&kcs, i); - } + for_each_possible_cpu(i) + idle_nsec += kcpustat_field(CPUTIME_IDLE, i); ktime_get_boottime_ts64(&uptime); timens_add_boottime(&uptime); diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild index 2c53a1e0b760..15df9dcb42a5 100644 --- a/include/asm-generic/Kbuild +++ b/include/asm-generic/Kbuild @@ -44,6 +44,7 @@ mandatory-y += module.lds.h mandatory-y += msi.h mandatory-y += pci.h mandatory-y += percpu.h +mandatory-y += percpu_types.h mandatory-y += pgalloc.h mandatory-y += preempt.h mandatory-y += rqspinlock.h diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index d4f581c1e21d..b99cb57dfccc 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -235,7 +235,7 @@ do { \ /** * smp_cond_load_relaxed() - (Spin) wait for cond with no ordering guarantees * @ptr: pointer to the variable to wait on - * @cond: boolean expression to wait for + * @cond_expr: boolean expression to wait for * * Equivalent to using READ_ONCE() on the condition variable. * @@ -259,7 +259,7 @@ do { \ /** * smp_cond_load_acquire() - (Spin) wait for cond with ACQUIRE ordering * @ptr: pointer to the variable to wait on - * @cond: boolean expression to wait for + * @cond_expr: boolean expression to wait for * * Equivalent to using smp_load_acquire() on the condition variable but employs * the control dependency of the wait to reduce the barrier on many platforms. diff --git a/include/asm-generic/percpu_types.h b/include/asm-generic/percpu_types.h new file mode 100644 index 000000000000..a095cea7fa20 --- /dev/null +++ b/include/asm-generic/percpu_types.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_GENERIC_PERCPU_TYPES_H_ +#define _ASM_GENERIC_PERCPU_TYPES_H_ + +#ifndef __ASSEMBLER__ +/* + * __percpu_qual is the qualifier for the percpu named address space. + * + * Most architectures use generic named address space for percpu variables but + * some architectures define percpu variables in different named address space. + * E.g. on x86, percpu variable may be declared as being relative to the %fs or + * %gs segments using __seg_fs or __seg_gs named address space qualifier. + */ +#ifndef __percpu_qual +# define __percpu_qual +#endif + +#endif /* __ASSEMBLER__ */ +#endif /* _ASM_GENERIC_PERCPU_TYPES_H_ */ diff --git a/include/asm-generic/qspinlock_types.h b/include/asm-generic/qspinlock_types.h index 2fd1fb89ec36..7d17b9bb069c 100644 --- a/include/asm-generic/qspinlock_types.h +++ b/include/asm-generic/qspinlock_types.h @@ -53,8 +53,7 @@ typedef struct qspinlock { * * When NR_CPUS < 16K * 0- 7: locked byte - * 8: pending - * 9-15: not used + * 8-15: pending byte * 16-17: tail index * 18-31: tail cpu (+1) * diff --git a/include/linux/cleanup.h b/include/linux/cleanup.h index ea95ca4bc11c..b1b5698cbf1b 100644 --- a/include/linux/cleanup.h +++ b/include/linux/cleanup.h @@ -397,7 +397,8 @@ static __maybe_unused const bool class_##_name##_is_conditional = _is_cond __DEFINE_GUARD_LOCK_PTR(_name, _T) #define DEFINE_GUARD(_name, _type, _lock, _unlock) \ - DEFINE_CLASS(_name, _type, if (_T) { _unlock; }, ({ _lock; _T; }), _type _T); \ + static __always_inline __nonnull_args(1) _type class_##_name##_constructor(_type _T); \ + DEFINE_CLASS(_name, _type, _unlock, ({ _lock; _T; }), _type _T); \ DEFINE_CLASS_IS_GUARD(_name) #define DEFINE_GUARD_COND_4(_name, _ext, _lock, _cond) \ @@ -491,13 +492,14 @@ typedef struct { \ static __always_inline void class_##_name##_destructor(class_##_name##_t *_T) \ __no_context_analysis \ { \ - if (_T->lock) { _unlock; } \ + _unlock; \ } \ \ __DEFINE_GUARD_LOCK_PTR(_name, &_T->lock) #define __DEFINE_LOCK_GUARD_1(_name, _type, ...) \ -static __always_inline class_##_name##_t class_##_name##_constructor(_type *l) \ +static __always_inline __nonnull_args(1) \ +class_##_name##_t class_##_name##_constructor(_type *l) \ __no_context_analysis \ { \ class_##_name##_t _t = { .lock = l }, *_T = &_t; \ diff --git a/include/linux/clocksource.h b/include/linux/clocksource.h index b12a6d19aa60..283d7297aa79 100644 --- a/include/linux/clocksource.h +++ b/include/linux/clocksource.h @@ -31,6 +31,21 @@ struct module; #include +/** + * struct clocksource_hw_snapshot - Snapshot for the underlying hardware counter of derived + * clocksources like kvmclock or Hyper-V scaled TSC + * @hw_cycles: The hardware counter value + * @hw_csid: Clocksource ID of the hardware counter + * + * Such clocksources must implement the read_snapshot() callback and fill in the + * hardware counter value, the clocksource ID of the hardware counter and derive + * the actual clocksource cycles from @hw_cycles to provide an atomic snapshot + */ +struct clocksource_hw_snapshot { + u64 hw_cycles; + enum clocksource_ids hw_csid; +}; + /** * struct clocksource - hardware abstraction for a free running counter * Provides mostly state-free accessors to the underlying hardware. @@ -72,6 +87,14 @@ struct module; * @flags: Flags describing special properties * @base: Hardware abstraction for clock on which a clocksource * is based + * @read_snapshot: Extended @read() function for clocksources such as + * kvmclock or the Hyper-V scaled TSC where the actual + * clocksource value for timekeeping is calculated from an + * underlying hardware counter. Returns the timekeeping + * relevant cycle value and stores the raw value of the + * underlying counter from which it was calculated + * including the clocksource ID of that counter in the + * clocksource hardware snapshot. * @enable: Optional function to enable the clocksource * @disable: Optional function to disable the clocksource * @suspend: Optional suspend function for the clocksource @@ -113,6 +136,7 @@ struct clocksource { unsigned long flags; struct clocksource_base *base; + u64 (*read_snapshot)(struct clocksource *cs, struct clocksource_hw_snapshot *chs); int (*enable)(struct clocksource *cs); void (*disable)(struct clocksource *cs); void (*suspend)(struct clocksource *cs); diff --git a/include/linux/compiler-context-analysis.h b/include/linux/compiler-context-analysis.h index a9317571e6af..8302ebc2ea8c 100644 --- a/include/linux/compiler-context-analysis.h +++ b/include/linux/compiler-context-analysis.h @@ -39,12 +39,14 @@ # define __assumes_shared_ctx_lock(...) __attribute__((assert_shared_capability(__VA_ARGS__))) /** - * __guarded_by - struct member and globals attribute, declares variable - * only accessible within active context + * __guarded_by() - struct member and globals attribute, declares variable + * only accessible within active context + * @...: context lock instance pointer(s) * * Declares that the struct member or global variable is only accessible within - * the context entered by the given context lock. Read operations on the data - * require shared access, while write operations require exclusive access. + * the context entered by the given context lock(s). Read operations on the data + * require shared access to at least one of the context locks, while write + * operations require exclusive access to all listed context locks. * * .. code-block:: c * @@ -52,17 +54,24 @@ * spinlock_t lock; * long counter __guarded_by(&lock); * }; + * + * struct some_state { + * spinlock_t lock1, lock2; + * long counter __guarded_by(&lock1, &lock2); + * }; */ # define __guarded_by(...) __attribute__((guarded_by(__VA_ARGS__))) /** - * __pt_guarded_by - struct member and globals attribute, declares pointed-to - * data only accessible within active context + * __pt_guarded_by() - struct member and globals attribute, declares pointed-to + * data only accessible within active context + * @...: context lock instance pointer(s) * * Declares that the data pointed to by the struct member pointer or global * pointer is only accessible within the context entered by the given context - * lock. Read operations on the data require shared access, while write - * operations require exclusive access. + * lock(s). Read operations on the data require shared access to at least one + * of the context locks, while write operations require exclusive access to all + * listed context locks. * * .. code-block:: c * @@ -70,6 +79,11 @@ * spinlock_t lock; * long *counter __pt_guarded_by(&lock); * }; + * + * struct some_state { + * spinlock_t lock1, lock2; + * long *counter __pt_guarded_by(&lock1, &lock2); + * }; */ # define __pt_guarded_by(...) __attribute__((pt_guarded_by(__VA_ARGS__))) diff --git a/include/linux/compiler_attributes.h b/include/linux/compiler_attributes.h index 836a50f5917a..476c4c560d17 100644 --- a/include/linux/compiler_attributes.h +++ b/include/linux/compiler_attributes.h @@ -230,6 +230,15 @@ */ #define noinline __attribute__((__noinline__)) +/* + * Note: deliberately not named '__nonnull', to avoid clashing with glibc's + * __nonnull() when kernel and userspace headers are combined. + * + * gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Attributes.html#index-nonnull + * clang: https://clang.llvm.org/docs/AttributeReference.html#nonnull + */ +#define __nonnull_args(x...) __attribute__((__nonnull__(x))) + /* * Optional: only supported since gcc >= 8 * Optional: not supported by clang diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index 369966598a2c..c5921f139007 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -634,6 +634,9 @@ struct ftrace_likely_data { #else #define __unqual_scalar_typeof(x) __typeof_unqual__(x) #endif + +#include + #endif /* !__ASSEMBLY__ */ /* diff --git a/include/linux/futex.h b/include/linux/futex.h index 9e9750f04980..51f4ccdc9092 100644 --- a/include/linux/futex.h +++ b/include/linux/futex.h @@ -64,14 +64,10 @@ enum { static inline void futex_init_task(struct task_struct *tsk) { - tsk->robust_list = NULL; -#ifdef CONFIG_COMPAT - tsk->compat_robust_list = NULL; -#endif - INIT_LIST_HEAD(&tsk->pi_state_list); - tsk->pi_state_cache = NULL; - tsk->futex_state = FUTEX_STATE_OK; - mutex_init(&tsk->futex_exit_mutex); + memset(&tsk->futex, 0, sizeof(tsk->futex)); + INIT_LIST_HEAD(&tsk->futex.pi_state_list); + tsk->futex.state = FUTEX_STATE_OK; + mutex_init(&tsk->futex.exit_mutex); } void futex_exit_recursive(struct task_struct *tsk); @@ -85,22 +81,18 @@ int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4) #ifdef CONFIG_FUTEX_PRIVATE_HASH int futex_hash_allocate_default(void); void futex_hash_free(struct mm_struct *mm); -int futex_mm_init(struct mm_struct *mm); - -#else /* !CONFIG_FUTEX_PRIVATE_HASH */ +#else /* CONFIG_FUTEX_PRIVATE_HASH */ static inline int futex_hash_allocate_default(void) { return 0; } static inline int futex_hash_free(struct mm_struct *mm) { return 0; } -static inline int futex_mm_init(struct mm_struct *mm) { return 0; } -#endif /* CONFIG_FUTEX_PRIVATE_HASH */ +#endif /* !CONFIG_FUTEX_PRIVATE_HASH */ -#else /* !CONFIG_FUTEX */ +#else /* CONFIG_FUTEX */ static inline void futex_init_task(struct task_struct *tsk) { } static inline void futex_exit_recursive(struct task_struct *tsk) { } static inline void futex_exit_release(struct task_struct *tsk) { } static inline void futex_exec_release(struct task_struct *tsk) { } -static inline long do_futex(u32 __user *uaddr, int op, u32 val, - ktime_t *timeout, u32 __user *uaddr2, - u32 val2, u32 val3) +static inline long do_futex(u32 __user *uaddr, int op, u32 val, ktime_t *timeout, + u32 __user *uaddr2, u32 val2, u32 val3) { return -EINVAL; } @@ -108,13 +100,63 @@ static inline int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsig { return -EINVAL; } -static inline int futex_hash_allocate_default(void) -{ - return 0; -} +static inline int futex_hash_allocate_default(void) { return 0; } static inline int futex_hash_free(struct mm_struct *mm) { return 0; } -static inline int futex_mm_init(struct mm_struct *mm) { return 0; } +#endif /* !CONFIG_FUTEX */ +#ifdef CONFIG_FUTEX_ROBUST_UNLOCK +#include + +void futex_reset_cs_ranges(struct futex_mm_data *fd); +void __futex_fixup_robust_unlock(struct pt_regs *regs, struct futex_unlock_cs_range *csr); + +static inline bool futex_within_robust_unlock(struct pt_regs *regs, + struct futex_unlock_cs_range *csr) +{ + unsigned long ip = instruction_pointer(regs); + + return ip >= csr->start_ip && ip < csr->start_ip + csr->len; +} + +static inline void futex_fixup_robust_unlock(struct pt_regs *regs) +{ + struct futex_unlock_cs_range *csr; + + /* + * Avoid dereferencing current->mm if not returning from interrupt. + * current->rseq.event is going to be used subsequently, so bringing the + * cache line in is not a big deal. + */ + if (!current->rseq.event.user_irq) + return; + + csr = current->mm->futex.unlock.cs_ranges; + + /* The loop is optimized out for !COMPAT */ + for (int r = 0; r < FUTEX_ROBUST_MAX_CS_RANGES; r++, csr++) { + if (unlikely(futex_within_robust_unlock(regs, csr))) { + __futex_fixup_robust_unlock(regs, csr); + return; + } + } +} + +static inline void futex_set_vdso_cs_range(struct futex_mm_data *fd, unsigned int idx, + unsigned long start, unsigned long end, bool sz32) +{ + fd->unlock.cs_ranges[idx].start_ip = start; + fd->unlock.cs_ranges[idx].len = end - start; + fd->unlock.cs_ranges[idx].pop_size32 = sz32; +} +#else /* CONFIG_FUTEX_ROBUST_UNLOCK */ +static inline void futex_fixup_robust_unlock(struct pt_regs *regs) { } +#endif /* !CONFIG_FUTEX_ROBUST_UNLOCK */ + + +#if defined(CONFIG_FUTEX_PRIVATE_HASH) || defined(CONFIG_FUTEX_ROBUST_UNLOCK) +void futex_mm_init(struct mm_struct *mm); +#else +static inline void futex_mm_init(struct mm_struct *mm) { } #endif -#endif +#endif /* _LINUX_FUTEX_H */ diff --git a/include/linux/futex_types.h b/include/linux/futex_types.h new file mode 100644 index 000000000000..d320c0571f0c --- /dev/null +++ b/include/linux/futex_types.h @@ -0,0 +1,98 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_FUTEX_TYPES_H +#define _LINUX_FUTEX_TYPES_H + +#ifdef CONFIG_FUTEX +#include +#include +#include + +struct compat_robust_list_head; +struct futex_pi_state; +struct robust_list_head; + +/** + * struct futex_sched_data - Futex related per task data + * @robust_list: User space registered robust list pointer + * @compat_robust_list: User space registered robust list pointer for compat tasks + * @pi_state_list: List head for Priority Inheritance (PI) state management + * @pi_state_cache: Pointer to cache one PI state object per task + * @exit_mutex: Mutex for serializing exit + * @state: Futex handling state to handle exit races correctly + */ +struct futex_sched_data { + struct robust_list_head __user *robust_list; +#ifdef CONFIG_COMPAT + struct compat_robust_list_head __user *compat_robust_list; +#endif + struct list_head pi_state_list; + struct futex_pi_state *pi_state_cache; + struct mutex exit_mutex; + unsigned int state; +}; + +#ifdef CONFIG_FUTEX_PRIVATE_HASH +/** + * struct futex_mm_phash - Futex private hash related per MM data + * @lock: Mutex to protect the private hash operations + * @hash: RCU managed pointer to the private hash + * @hash_new: Pointer to a newly allocated private hash + * @batches: Batch state for RCU synchronization + * @rcu: RCU head for call_rcu() + * @atomic: Aggregate value for @hash_ref + * @ref: Per CPU reference counter for a private hash + */ +struct futex_mm_phash { + struct mutex lock; + struct futex_private_hash __rcu *hash; + struct futex_private_hash *hash_new; + unsigned long batches; + struct rcu_head rcu; + atomic_long_t atomic; + unsigned int __percpu *ref; +}; +#else /* CONFIG_FUTEX_ROBUST_UNLOCK */ +struct futex_mm_phash { }; +#endif /* !CONFIG_FUTEX_ROBUST_UNLOCK */ + +#ifdef CONFIG_FUTEX_ROBUST_UNLOCK +/** + * struct futex_unlock_cs_range - Range for the VDSO unlock critical section + * @start_ip: The start IP of the robust futex unlock critical section (inclusive) + * @len: The length of the robust futex unlock critical section + * @pop_size32: Pending OP pointer size indicator. 0 == 64-bit, 1 == 32-bit + */ +struct futex_unlock_cs_range { + unsigned long start_ip; + unsigned int len; + unsigned int pop_size32; +}; + +#define FUTEX_ROBUST_MAX_CS_RANGES (1 + IS_ENABLED(CONFIG_COMPAT)) + +/** + * struct futex_unlock_cs_ranges - Futex unlock VSDO critical sections + * @cs_ranges: Array of critical section ranges + */ +struct futex_unlock_cs_ranges { + struct futex_unlock_cs_range cs_ranges[FUTEX_ROBUST_MAX_CS_RANGES]; +}; +#else /* CONFIG_FUTEX_ROBUST_UNLOCK */ +struct futex_unlock_cs_ranges { }; +#endif /* !CONFIG_FUTEX_ROBUST_UNLOCK */ + +/** + * struct futex_mm_data - Futex related per MM data + * @phash: Futex private hash related data + * @unlock: Futex unlock VDSO critical sections + */ +struct futex_mm_data { + struct futex_mm_phash phash; + struct futex_unlock_cs_ranges unlock; +}; +#else /* CONFIG_FUTEX */ +struct futex_sched_data { }; +struct futex_mm_data { }; +#endif /* !CONFIG_FUTEX */ + +#endif /* _LINUX_FUTEX_TYPES_H */ diff --git a/include/linux/kernel_stat.h b/include/linux/kernel_stat.h index b97ce2df376f..fce1392e2140 100644 --- a/include/linux/kernel_stat.h +++ b/include/linux/kernel_stat.h @@ -34,7 +34,14 @@ enum cpu_usage_stat { }; struct kernel_cpustat { - u64 cpustat[NR_STATS]; +#ifdef CONFIG_NO_HZ_COMMON + bool idle_dyntick; + bool idle_elapse; + seqcount_t idle_sleeptime_seq; + u64 idle_entrytime; + u64 idle_stealtime[2]; +#endif + u64 cpustat[NR_STATS]; }; struct kernel_stat { @@ -99,23 +106,68 @@ static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu) return kstat_cpu(cpu).irqs_sum; } +#ifdef CONFIG_NO_HZ_COMMON +extern void kcpustat_dyntick_start(u64 now); +extern void kcpustat_dyntick_stop(u64 now); +extern void kcpustat_irq_enter(u64 now); +extern void kcpustat_irq_exit(u64 now); +extern u64 kcpustat_field_idle(int cpu); +extern u64 kcpustat_field_iowait(int cpu); + +static inline bool kcpustat_idle_dyntick(void) +{ + return __this_cpu_read(kernel_cpustat.idle_dyntick); +} +#else +static inline u64 kcpustat_field_idle(int cpu) +{ + return kcpustat_cpu(cpu).cpustat[CPUTIME_IDLE]; +} +static inline u64 kcpustat_field_iowait(int cpu) +{ + return kcpustat_cpu(cpu).cpustat[CPUTIME_IOWAIT]; +} + +static inline bool kcpustat_idle_dyntick(void) +{ + return false; +} +#endif /* CONFIG_NO_HZ_COMMON */ + +extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time); +extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time); + +/* Fetch cputime values when vtime is disabled on a CPU */ +static inline u64 kcpustat_field_default(enum cpu_usage_stat usage, int cpu) +{ + if (usage == CPUTIME_IDLE) + return kcpustat_field_idle(cpu); + if (usage == CPUTIME_IOWAIT) + return kcpustat_field_iowait(cpu); + return kcpustat_cpu(cpu).cpustat[usage]; +} + +static inline void kcpustat_cpu_fetch_default(struct kernel_cpustat *dst, int cpu) +{ + *dst = kcpustat_cpu(cpu); + dst->cpustat[CPUTIME_IDLE] = kcpustat_field_idle(cpu); + dst->cpustat[CPUTIME_IOWAIT] = kcpustat_field_iowait(cpu); +} + #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN -extern u64 kcpustat_field(struct kernel_cpustat *kcpustat, - enum cpu_usage_stat usage, int cpu); +extern u64 kcpustat_field(enum cpu_usage_stat usage, int cpu); extern void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu); #else -static inline u64 kcpustat_field(struct kernel_cpustat *kcpustat, - enum cpu_usage_stat usage, int cpu) +static inline u64 kcpustat_field(enum cpu_usage_stat usage, int cpu) { - return kcpustat->cpustat[usage]; + return kcpustat_field_default(usage, cpu); } static inline void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu) { - *dst = kcpustat_cpu(cpu); + kcpustat_cpu_fetch_default(dst, cpu); } - -#endif +#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */ extern void account_user_time(struct task_struct *, u64); extern void account_guest_time(struct task_struct *, u64); @@ -124,19 +176,17 @@ extern void account_system_index_time(struct task_struct *, u64, enum cpu_usage_stat); extern void account_steal_time(u64); extern void account_idle_time(u64); -extern u64 get_idle_time(struct kernel_cpustat *kcs, int cpu); #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE static inline void account_process_tick(struct task_struct *tsk, int user) { - vtime_flush(tsk); + if (!kcpustat_idle_dyntick()) + vtime_flush(tsk); } #else extern void account_process_tick(struct task_struct *, int user); #endif -extern void account_idle_ticks(unsigned long ticks); - #ifdef CONFIG_SCHED_CORE extern void __account_forceidle_time(struct task_struct *tsk, u64 delta); #endif diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 9588ce3b16df..2fc552b3924a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -1270,16 +1271,7 @@ struct mm_struct { */ seqcount_t mm_lock_seq; #endif -#ifdef CONFIG_FUTEX_PRIVATE_HASH - struct mutex futex_hash_lock; - struct futex_private_hash __rcu *futex_phash; - struct futex_private_hash *futex_phash_new; - /* futex-ref */ - unsigned long futex_batches; - struct rcu_head futex_rcu; - atomic_long_t futex_atomic; - unsigned int __percpu *futex_ref; -#endif + struct futex_mm_data futex; unsigned long hiwater_rss; /* High-watermark of RSS usage */ unsigned long hiwater_vm; /* High-water virtual memory usage */ diff --git a/include/linux/percpu-rwsem.h b/include/linux/percpu-rwsem.h index c8cb010d655e..39d5bf8e6562 100644 --- a/include/linux/percpu-rwsem.h +++ b/include/linux/percpu-rwsem.h @@ -107,6 +107,8 @@ static inline bool percpu_down_read_trylock(struct percpu_rw_semaphore *sem) return ret; } +extern void __percpu_up_read(struct percpu_rw_semaphore *sem); + static inline void percpu_up_read(struct percpu_rw_semaphore *sem) { rwsem_release(&sem->dep_map, _RET_IP_); @@ -118,18 +120,7 @@ static inline void percpu_up_read(struct percpu_rw_semaphore *sem) if (likely(rcu_sync_is_idle(&sem->rss))) { this_cpu_dec(*sem->read_count); } else { - /* - * slowpath; reader will only ever wake a single blocked - * writer. - */ - smp_mb(); /* B matches C */ - /* - * In other words, if they see our decrement (presumably to - * aggregate zero, as that is the only time it matters) they - * will also see our critical section. - */ - this_cpu_dec(*sem->read_count); - rcuwait_wake_up(&sem->writer); + __percpu_up_read(sem); } preempt_enable(); } diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 85bf8dd9f087..2f5a889aa50d 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -3,13 +3,14 @@ #define __LINUX_PERCPU_H #include -#include -#include -#include -#include -#include #include +#include +#include +#include +#include +#include #include +#include #include diff --git a/include/linux/pps_kernel.h b/include/linux/pps_kernel.h index aab0aebb529e..9f088c9023b1 100644 --- a/include/linux/pps_kernel.h +++ b/include/linux/pps_kernel.h @@ -99,12 +99,14 @@ static inline void timespec_to_pps_ktime(struct pps_ktime *kt, static inline void pps_get_ts(struct pps_event_time *ts) { +#ifdef CONFIG_NTP_PPS struct system_time_snapshot snap; - ktime_get_snapshot(&snap); - ts->ts_real = ktime_to_timespec64(snap.real); -#ifdef CONFIG_NTP_PPS - ts->ts_raw = ktime_to_timespec64(snap.raw); + ktime_get_snapshot_id(CLOCK_REALTIME, &snap); + ts->ts_real = ktime_to_timespec64(snap.systime); + ts->ts_raw = ktime_to_timespec64(snap.monoraw); +#else + ktime_get_real_ts64(&ts->ts_real); #endif } diff --git a/include/linux/ptp_clock_kernel.h b/include/linux/ptp_clock_kernel.h index 884364596dd3..36a27a910595 100644 --- a/include/linux/ptp_clock_kernel.h +++ b/include/linux/ptp_clock_kernel.h @@ -12,6 +12,7 @@ #include #include #include +#include #include #define PTP_CLOCK_NAME_LEN 32 @@ -45,13 +46,13 @@ struct system_device_crosststamp; /** * struct ptp_system_timestamp - system time corresponding to a PHC timestamp - * @pre_ts: system timestamp before capturing PHC - * @post_ts: system timestamp after capturing PHC - * @clockid: clock-base used for capturing the system timestamps + * @pre_sts: system time snapshot before capturing PHC + * @post_sts: system time snapshot after capturing PHC + * @clockid: clock-base used for capturing the system timestamps */ struct ptp_system_timestamp { - struct timespec64 pre_ts; - struct timespec64 post_ts; + struct system_time_snapshot pre_sts; + struct system_time_snapshot post_sts; clockid_t clockid; }; @@ -510,13 +511,13 @@ static inline ktime_t ptp_convert_timestamp(const ktime_t *hwtstamp, static inline void ptp_read_system_prets(struct ptp_system_timestamp *sts) { if (sts) - ktime_get_clock_ts64(sts->clockid, &sts->pre_ts); + ktime_get_snapshot_id(sts->clockid, &sts->pre_sts); } static inline void ptp_read_system_postts(struct ptp_system_timestamp *sts) { if (sts) - ktime_get_clock_ts64(sts->clockid, &sts->post_ts); + ktime_get_snapshot_id(sts->clockid, &sts->post_sts); } #endif diff --git a/include/linux/rtmutex.h b/include/linux/rtmutex.h index 78e7e588817c..9e1f012f89db 100644 --- a/include/linux/rtmutex.h +++ b/include/linux/rtmutex.h @@ -56,6 +56,8 @@ static inline struct task_struct *rt_mutex_owner(struct rt_mutex_base *lock) #endif extern void rt_mutex_base_init(struct rt_mutex_base *rtb); +context_lock_struct(rt_mutex); + /** * The rt_mutex structure * @@ -108,8 +110,10 @@ do { \ extern void __rt_mutex_init(struct rt_mutex *lock, const char *name, struct lock_class_key *key); #ifdef CONFIG_DEBUG_LOCK_ALLOC -extern void rt_mutex_lock_nested(struct rt_mutex *lock, unsigned int subclass); -extern void _rt_mutex_lock_nest_lock(struct rt_mutex *lock, struct lockdep_map *nest_lock); +extern void rt_mutex_lock_nested(struct rt_mutex *lock, unsigned int subclass) + __acquires(lock); +extern void _rt_mutex_lock_nest_lock(struct rt_mutex *lock, struct lockdep_map *nest_lock) + __acquires(lock); #define rt_mutex_lock(lock) rt_mutex_lock_nested(lock, 0) #define rt_mutex_lock_nest_lock(lock, nest_lock) \ do { \ @@ -118,15 +122,19 @@ extern void _rt_mutex_lock_nest_lock(struct rt_mutex *lock, struct lockdep_map * } while (0) #else -extern void rt_mutex_lock(struct rt_mutex *lock); +extern void rt_mutex_lock(struct rt_mutex *lock) __acquires(lock); #define rt_mutex_lock_nested(lock, subclass) rt_mutex_lock(lock) #define rt_mutex_lock_nest_lock(lock, nest_lock) rt_mutex_lock(lock) #endif -extern int rt_mutex_lock_interruptible(struct rt_mutex *lock); -extern int rt_mutex_lock_killable(struct rt_mutex *lock); -extern int rt_mutex_trylock(struct rt_mutex *lock); +extern int rt_mutex_lock_interruptible(struct rt_mutex *lock) + __cond_acquires(0, lock); +extern int rt_mutex_lock_killable(struct rt_mutex *lock) + __cond_acquires(0, lock); +extern int rt_mutex_trylock(struct rt_mutex *lock) + __cond_acquires(true, lock); -extern void rt_mutex_unlock(struct rt_mutex *lock); +extern void rt_mutex_unlock(struct rt_mutex *lock) + __releases(lock); #endif diff --git a/include/linux/sched.h b/include/linux/sched.h index d51bb3d75d0e..6d8b20bac518 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -16,6 +16,7 @@ #include #include +#include #include #include #include @@ -65,7 +66,6 @@ struct bpf_net_context; struct capture_control; struct cfs_rq; struct fs_struct; -struct futex_pi_state; struct io_context; struct io_uring_task; struct mempolicy; @@ -77,7 +77,6 @@ struct pid_namespace; struct pipe_inode_info; struct rcu_node; struct reclaim_state; -struct robust_list_head; struct root_domain; struct rq; struct sched_attr; @@ -1341,16 +1340,9 @@ struct task_struct { u32 closid; u32 rmid; #endif -#ifdef CONFIG_FUTEX - struct robust_list_head __user *robust_list; -#ifdef CONFIG_COMPAT - struct compat_robust_list_head __user *compat_robust_list; -#endif - struct list_head pi_state_list; - struct futex_pi_state *pi_state_cache; - struct mutex futex_exit_mutex; - unsigned int futex_state; -#endif + + struct futex_sched_data futex; + #ifdef CONFIG_PERF_EVENTS u8 perf_recursion[PERF_NR_CONTEXTS]; struct perf_event_context *perf_event_ctxp; diff --git a/include/linux/seqlock.h b/include/linux/seqlock.h index 5a40252b8334..f865491c4f2c 100644 --- a/include/linux/seqlock.h +++ b/include/linux/seqlock.h @@ -1259,14 +1259,15 @@ static __always_inline void __scoped_seqlock_cleanup(struct ss_tmp *sst) extern void __scoped_seqlock_invalid_target(void); -#if (defined(CONFIG_CC_IS_GCC) && CONFIG_GCC_VERSION < 90000) || defined(CONFIG_KASAN) +#if (defined(CONFIG_CC_IS_GCC) && CONFIG_GCC_VERSION < 90000) || \ + defined(CONFIG_KASAN) || defined(CONFIG_UBSAN_ALIGNMENT) /* * For some reason some GCC-8 architectures (nios2, alpha) have trouble * determining that the ss_done state is impossible in __scoped_seqlock_next() * below. * - * Similarly KASAN is known to confuse compilers enough to break this. But we - * don't care about code quality for KASAN builds anyway. + * Similarly KASAN and UBSAN_ALIGNMENT are known to confuse compilers enough + * to break this. But we don't care about code quality for such builds anyway. */ static inline void __scoped_seqlock_bug(void) { } #else diff --git a/include/linux/tick.h b/include/linux/tick.h index 738007d6f577..1cf4651f09ad 100644 --- a/include/linux/tick.h +++ b/include/linux/tick.h @@ -139,8 +139,6 @@ extern bool tick_nohz_idle_got_tick(void); extern ktime_t tick_nohz_get_next_hrtimer(void); extern ktime_t tick_nohz_get_sleep_length(ktime_t *delta_next); extern unsigned long tick_nohz_get_idle_calls_cpu(int cpu); -extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time); -extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time); #else /* !CONFIG_NO_HZ_COMMON */ #define tick_nohz_enabled (0) static inline bool tick_nohz_is_active(void) { return false; } @@ -162,8 +160,6 @@ static inline ktime_t tick_nohz_get_sleep_length(ktime_t *delta_next) *delta_next = TICK_NSEC; return *delta_next; } -static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; } -static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; } #endif /* !CONFIG_NO_HZ_COMMON */ /* diff --git a/include/linux/timekeeper_internal.h b/include/linux/timekeeper_internal.h index e36d11e33e0c..4486dfd5d0de 100644 --- a/include/linux/timekeeper_internal.h +++ b/include/linux/timekeeper_internal.h @@ -190,7 +190,7 @@ struct timekeeper { s32 tai_offset; }; -#ifdef CONFIG_GENERIC_TIME_VSYSCALL +#ifdef CONFIG_GENERIC_GETTIMEOFDAY extern void update_vsyscall(struct timekeeper *tk); extern void update_vsyscall_tz(void); diff --git a/include/linux/timekeeping.h b/include/linux/timekeeping.h index aee2c1a46e47..984a866d293b 100644 --- a/include/linux/timekeeping.h +++ b/include/linux/timekeeping.h @@ -276,37 +276,30 @@ static inline bool ktime_get_aux_ts64(clockid_t id, struct timespec64 *kt) { ret #endif /** - * struct system_time_snapshot - simultaneous raw/real time capture with - * counter value - * @cycles: Clocksource counter value to produce the system times - * @real: Realtime system time - * @boot: Boot time - * @raw: Monotonic raw system time - * @cs_id: Clocksource ID + * struct system_time_snapshot - Simultaneous time capture of CLOCK_MONOTONIC_RAW, + * a selected CLOCK_* and the clocksource counter value + * @cycles: Clocksource counter value to produce the system times + * @hw_cycles: For derived clocksources, the hardware counter value from + * which @cycles was derived + * @systime: The system time of the selected CLOCK ID + * @monoraw: Monotonic raw system time + * @cs_id: Clocksource ID + * @hw_csid: Clocksource ID of the underlying hardware counter for derived + * clocksources which implement the read_snapshot() callback. * @clock_was_set_seq: The sequence number of clock-was-set events * @cs_was_changed_seq: The sequence number of clocksource change events + * @valid: True if the snapshot is valid */ struct system_time_snapshot { u64 cycles; - ktime_t real; - ktime_t boot; - ktime_t raw; + u64 hw_cycles; + ktime_t systime; + ktime_t monoraw; enum clocksource_ids cs_id; + enum clocksource_ids hw_csid; unsigned int clock_was_set_seq; u8 cs_was_changed_seq; -}; - -/** - * struct system_device_crosststamp - system/device cross-timestamp - * (synchronized capture) - * @device: Device time - * @sys_realtime: Realtime simultaneous with device time - * @sys_monoraw: Monotonic raw simultaneous with device time - */ -struct system_device_crosststamp { - ktime_t device; - ktime_t sys_realtime; - ktime_t sys_monoraw; + u8 valid; }; /** @@ -325,6 +318,23 @@ struct system_counterval_t { bool use_nsecs; }; +/** + * struct system_device_crosststamp - system/device cross-timestamp + * (synchronized capture) + * @clock_id: System time Clock ID to capture + * @device: Device time + * @sys_counter: Clocksource counter value simultaneous with device time + * @sys_systime: System time for @clock_id + * @sys_monoraw: Monotonic raw simultaneous with device time + */ +struct system_device_crosststamp { + clockid_t clock_id; + ktime_t device; + struct system_counterval_t sys_counter; + ktime_t sys_systime; + ktime_t sys_monoraw; +}; + extern bool ktime_real_to_base_clock(ktime_t treal, enum clocksource_ids base_id, u64 *cycles); extern bool timekeeping_clocksource_has_base(enum clocksource_ids id); @@ -341,9 +351,10 @@ extern int get_device_system_crosststamp( struct system_device_crosststamp *xtstamp); /* - * Simultaneously snapshot realtime and monotonic raw clocks + * Simultaneously snapshot a given clock with MONOTONIC_RAW and the underlying + * clocksource counter value. */ -extern void ktime_get_snapshot(struct system_time_snapshot *systime_snapshot); +extern void ktime_get_snapshot_id(clockid_t clock_id, struct system_time_snapshot *systime_snapshot); /* * Persistent clock related interfaces diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index e4a64976f1c5..8a264662b242 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -712,6 +712,17 @@ static inline void user_access_restore(unsigned long flags) { } #define user_read_access_end user_access_end #endif +#ifndef unsafe_atomic_store_release_user +# define unsafe_atomic_store_release_user(val, uptr, elbl) \ + do { \ + if (!IS_ENABLED(CONFIG_ARCH_MEMORY_ORDER_TSO)) \ + smp_mb(); \ + else \ + barrier(); \ + unsafe_put_user(val, uptr, elbl); \ + } while (0) +#endif + /* Define RW variant so the below _mode macro expansion works */ #define masked_user_rw_access_begin(u) masked_user_access_begin(u) #define user_rw_access_begin(u, s) user_access_begin(u, s) diff --git a/include/linux/vdso_datastore.h b/include/linux/vdso_datastore.h index 0b530428db71..3dfba9502d78 100644 --- a/include/linux/vdso_datastore.h +++ b/include/linux/vdso_datastore.h @@ -2,12 +2,12 @@ #ifndef _LINUX_VDSO_DATASTORE_H #define _LINUX_VDSO_DATASTORE_H -#ifdef CONFIG_HAVE_GENERIC_VDSO #include extern const struct vm_special_mapping vdso_vvar_mapping; struct vm_area_struct *vdso_install_vvar_mapping(struct mm_struct *mm, unsigned long addr); +#ifdef CONFIG_HAVE_GENERIC_VDSO void __init vdso_setup_data_pages(void); #else /* !CONFIG_HAVE_GENERIC_VDSO */ static inline void vdso_setup_data_pages(void) { } diff --git a/include/linux/vtime.h b/include/linux/vtime.h index 29dd5b91dd7d..9dc25b04a119 100644 --- a/include/linux/vtime.h +++ b/include/linux/vtime.h @@ -10,7 +10,6 @@ */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING extern void vtime_account_kernel(struct task_struct *tsk); -extern void vtime_account_idle(struct task_struct *tsk); #endif /* !CONFIG_VIRT_CPU_ACCOUNTING */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN @@ -27,16 +26,33 @@ static inline void vtime_guest_exit(struct task_struct *tsk) { } static inline void vtime_init_idle(struct task_struct *tsk, int cpu) { } #endif +static inline bool vtime_generic_enabled_cpu(int cpu) +{ + return context_tracking_enabled_cpu(cpu); +} + +static inline bool vtime_generic_enabled_this_cpu(void) +{ + return context_tracking_enabled_this_cpu(); +} + #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE +extern void vtime_account_idle(struct task_struct *tsk); extern void vtime_account_irq(struct task_struct *tsk, unsigned int offset); extern void vtime_account_softirq(struct task_struct *tsk); extern void vtime_account_hardirq(struct task_struct *tsk); extern void vtime_flush(struct task_struct *tsk); +extern void vtime_reset(void); +extern void vtime_dyntick_start(void); +extern void vtime_dyntick_stop(void); #else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ static inline void vtime_account_irq(struct task_struct *tsk, unsigned int offset) { } static inline void vtime_account_softirq(struct task_struct *tsk) { } static inline void vtime_account_hardirq(struct task_struct *tsk) { } static inline void vtime_flush(struct task_struct *tsk) { } +static inline void vtime_reset(void) { } +static inline void vtime_dyntick_start(void) { } +static inline void vtime_dyntick_stop(void) { } #endif /* @@ -74,12 +90,12 @@ static inline bool vtime_accounting_enabled(void) static inline bool vtime_accounting_enabled_cpu(int cpu) { - return context_tracking_enabled_cpu(cpu); + return vtime_generic_enabled_cpu(cpu); } static inline bool vtime_accounting_enabled_this_cpu(void) { - return context_tracking_enabled_this_cpu(); + return vtime_generic_enabled_this_cpu(); } extern void vtime_task_switch_generic(struct task_struct *prev); diff --git a/include/trace/events/lock.h b/include/trace/events/lock.h index 8e89baa3775f..1ded869cd619 100644 --- a/include/trace/events/lock.h +++ b/include/trace/events/lock.h @@ -5,7 +5,6 @@ #if !defined(_TRACE_LOCK_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_LOCK_H -#include #include /* flags for lock:contention_begin */ @@ -138,6 +137,23 @@ TRACE_EVENT(contention_end, TP_printk("%p (ret=%d)", __entry->lock_addr, __entry->ret) ); +TRACE_EVENT(contended_release, + + TP_PROTO(void *lock), + + TP_ARGS(lock), + + TP_STRUCT__entry( + __field(void *, lock_addr) + ), + + TP_fast_assign( + __entry->lock_addr = lock; + ), + + TP_printk("%p", __entry->lock_addr) +); + #endif /* _TRACE_LOCK_H */ /* This part must be outside protection */ diff --git a/include/uapi/linux/futex.h b/include/uapi/linux/futex.h index 7e2744ec8933..10a36c551675 100644 --- a/include/uapi/linux/futex.h +++ b/include/uapi/linux/futex.h @@ -25,23 +25,49 @@ #define FUTEX_PRIVATE_FLAG 128 #define FUTEX_CLOCK_REALTIME 256 -#define FUTEX_CMD_MASK ~(FUTEX_PRIVATE_FLAG | FUTEX_CLOCK_REALTIME) +#define FUTEX_ROBUST_UNLOCK 512 +#define FUTEX_ROBUST_LIST32 1024 -#define FUTEX_WAIT_PRIVATE (FUTEX_WAIT | FUTEX_PRIVATE_FLAG) -#define FUTEX_WAKE_PRIVATE (FUTEX_WAKE | FUTEX_PRIVATE_FLAG) -#define FUTEX_REQUEUE_PRIVATE (FUTEX_REQUEUE | FUTEX_PRIVATE_FLAG) -#define FUTEX_CMP_REQUEUE_PRIVATE (FUTEX_CMP_REQUEUE | FUTEX_PRIVATE_FLAG) -#define FUTEX_WAKE_OP_PRIVATE (FUTEX_WAKE_OP | FUTEX_PRIVATE_FLAG) -#define FUTEX_LOCK_PI_PRIVATE (FUTEX_LOCK_PI | FUTEX_PRIVATE_FLAG) -#define FUTEX_LOCK_PI2_PRIVATE (FUTEX_LOCK_PI2 | FUTEX_PRIVATE_FLAG) -#define FUTEX_UNLOCK_PI_PRIVATE (FUTEX_UNLOCK_PI | FUTEX_PRIVATE_FLAG) -#define FUTEX_TRYLOCK_PI_PRIVATE (FUTEX_TRYLOCK_PI | FUTEX_PRIVATE_FLAG) +#define FUTEX_CMD_MASK ~(FUTEX_PRIVATE_FLAG | FUTEX_CLOCK_REALTIME | \ + FUTEX_ROBUST_UNLOCK | FUTEX_ROBUST_LIST32) + +#define FUTEX_WAIT_PRIVATE (FUTEX_WAIT | FUTEX_PRIVATE_FLAG) +#define FUTEX_WAKE_PRIVATE (FUTEX_WAKE | FUTEX_PRIVATE_FLAG) +#define FUTEX_REQUEUE_PRIVATE (FUTEX_REQUEUE | FUTEX_PRIVATE_FLAG) +#define FUTEX_CMP_REQUEUE_PRIVATE (FUTEX_CMP_REQUEUE | FUTEX_PRIVATE_FLAG) +#define FUTEX_WAKE_OP_PRIVATE (FUTEX_WAKE_OP | FUTEX_PRIVATE_FLAG) +#define FUTEX_LOCK_PI_PRIVATE (FUTEX_LOCK_PI | FUTEX_PRIVATE_FLAG) +#define FUTEX_LOCK_PI2_PRIVATE (FUTEX_LOCK_PI2 | FUTEX_PRIVATE_FLAG) +#define FUTEX_UNLOCK_PI_PRIVATE (FUTEX_UNLOCK_PI | FUTEX_PRIVATE_FLAG) +#define FUTEX_TRYLOCK_PI_PRIVATE (FUTEX_TRYLOCK_PI | FUTEX_PRIVATE_FLAG) #define FUTEX_WAIT_BITSET_PRIVATE (FUTEX_WAIT_BITSET | FUTEX_PRIVATE_FLAG) #define FUTEX_WAKE_BITSET_PRIVATE (FUTEX_WAKE_BITSET | FUTEX_PRIVATE_FLAG) -#define FUTEX_WAIT_REQUEUE_PI_PRIVATE (FUTEX_WAIT_REQUEUE_PI | \ - FUTEX_PRIVATE_FLAG) -#define FUTEX_CMP_REQUEUE_PI_PRIVATE (FUTEX_CMP_REQUEUE_PI | \ - FUTEX_PRIVATE_FLAG) +#define FUTEX_WAIT_REQUEUE_PI_PRIVATE (FUTEX_WAIT_REQUEUE_PI | FUTEX_PRIVATE_FLAG) +#define FUTEX_CMP_REQUEUE_PI_PRIVATE (FUTEX_CMP_REQUEUE_PI | FUTEX_PRIVATE_FLAG) + +/* + * Operations to unlock a futex, clear the robust list pending op pointer and + * wake waiters. + */ +#define FUTEX_UNLOCK_PI_LIST64 (FUTEX_UNLOCK_PI | FUTEX_ROBUST_UNLOCK) +#define FUTEX_UNLOCK_PI_LIST64_PRIVATE (FUTEX_UNLOCK_PI_LIST64 | FUTEX_PRIVATE_FLAG) +#define FUTEX_UNLOCK_PI_LIST32 (FUTEX_UNLOCK_PI | FUTEX_ROBUST_UNLOCK | \ + FUTEX_ROBUST_LIST32) +#define FUTEX_UNLOCK_PI_LIST32_PRIVATE (FUTEX_UNLOCK_PI_LIST32 | FUTEX_PRIVATE_FLAG) + +#define FUTEX_UNLOCK_WAKE_LIST64 (FUTEX_WAKE | FUTEX_ROBUST_UNLOCK) +#define FUTEX_UNLOCK_WAKE_LIST64_PRIVATE (FUTEX_UNLOCK_WAKE_LIST64 | FUTEX_PRIVATE_FLAG) + +#define FUTEX_UNLOCK_WAKE_LIST32 (FUTEX_WAKE | FUTEX_ROBUST_UNLOCK | \ + FUTEX_ROBUST_LIST32) +#define FUTEX_UNLOCK_WAKE_LIST32_PRIVATE (FUTEX_UNLOCK_WAKE_LIST32 | FUTEX_PRIVATE_FLAG) + +#define FUTEX_UNLOCK_BITSET_LIST64 (FUTEX_WAKE_BITSET | FUTEX_ROBUST_UNLOCK) +#define FUTEX_UNLOCK_BITSET_LIST64_PRIVATE (FUTEX_UNLOCK_BITSET_LIST64 | FUTEX_PRIVATE_FLAG) + +#define FUTEX_UNLOCK_BITSET_LIST32 (FUTEX_WAKE_BITSET | FUTEX_ROBUST_UNLOCK | \ + FUTEX_ROBUST_LIST32) +#define FUTEX_UNLOCK_BITSET_LIST32_PRIVATE (FUTEX_UNLOCK_BITSET_LIST32 | FUTEX_PRIVATE_FLAG) /* * Flags for futex2 syscalls. @@ -177,6 +203,10 @@ struct robust_list_head { */ #define ROBUST_LIST_LIMIT 2048 +/* Modifiers for robust_list_head::list_op_pending */ +#define FUTEX_ROBUST_MOD_PI (0x1UL) +#define FUTEX_ROBUST_MOD_MASK (FUTEX_ROBUST_MOD_PI) + /* * bitset with all bits set for the FUTEX_xxx_BITSET OPs to request a * match of any bit. diff --git a/include/vdso/futex.h b/include/vdso/futex.h new file mode 100644 index 000000000000..3cd175eefe64 --- /dev/null +++ b/include/vdso/futex.h @@ -0,0 +1,52 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _VDSO_FUTEX_H +#define _VDSO_FUTEX_H + +#include + +/** + * __vdso_futex_robust_list64_try_unlock - Try to unlock an uncontended robust futex + * with a 64-bit pending op pointer + * @lock: Pointer to the futex lock object + * @tid: The TID of the calling task + * @pop: Pointer to the task's robust_list_head::list_pending_op + * + * Return: The content of *@lock. On success this is the same as @tid. + * + * The function implements: + * if (atomic_try_cmpxchg(lock, &tid, 0)) + * *op = NULL; + * return tid; + * + * There is a race between a successful unlock and clearing the pending op + * pointer in the robust list head. If the calling task is interrupted in the + * race window and has to handle a (fatal) signal on return to user space then + * the kernel handles the clearing of @pending_op before attempting to deliver + * the signal. That ensures that a task cannot exit with a potentially invalid + * pending op pointer. + * + * User space uses it in the following way: + * + * if (__vdso_futex_robust_list64_try_unlock(lock, tid, &pending_op) != tid) + * err = sys_futex($OP | FUTEX_ROBUST_UNLOCK,....); + * + * If the unlock attempt fails due to the FUTEX_WAITERS bit set in the lock, + * then the syscall does the unlock, clears the pending op pointer and wakes the + * requested number of waiters. + */ +__u32 __vdso_futex_robust_list64_try_unlock(__u32 *lock, __u32 tid, __u64 *pop); + +/** + * __vdso_futex_robust_list32_try_unlock - Try to unlock an uncontended robust futex + * with a 32-bit pending op pointer + * @lock: Pointer to the futex lock object + * @tid: The TID of the calling task + * @pop: Pointer to the task's robust_list_head::list_pending_op + * + * Return: The content of *@lock. On success this is the same as @tid. + * + * Same as __vdso_futex_robust_list64_try_unlock() just with a 32-bit @pop pointer. + */ +__u32 __vdso_futex_robust_list32_try_unlock(__u32 *lock, __u32 tid, __u32 *pop); + +#endif diff --git a/init/Kconfig b/init/Kconfig index b0d56fbb60c5..35dfbab6abff 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1849,6 +1849,12 @@ config FUTEX_MPOL depends on FUTEX && NUMA default y +config HAVE_FUTEX_ROBUST_UNLOCK + bool + +config FUTEX_ROBUST_UNLOCK + def_bool FUTEX && HAVE_GENERIC_VDSO && GENERIC_IRQ_ENTRY && RSEQ && HAVE_FUTEX_ROBUST_UNLOCK + config EPOLL bool "Enable eventpoll support" if EXPERT default y diff --git a/io_uring/futex.c b/io_uring/futex.c index 9cc1788ef4c6..906701b3c5c6 100644 --- a/io_uring/futex.c +++ b/io_uring/futex.c @@ -327,7 +327,7 @@ int io_futex_wake(struct io_kiocb *req, unsigned int issue_flags) * Strict flags - ensure that waking 0 futexes yields a 0 result. * See commit 43adf8449510 ("futex: FLAGS_STRICT") for details. */ - ret = futex_wake(iof->uaddr, FLAGS_STRICT | iof->futex_flags, + ret = futex_wake(iof->uaddr, FLAGS_STRICT | iof->futex_flags, NULL, iof->futex_val, iof->futex_mask); if (ret < 0) req_set_fail(req); diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 19d2244a9fef..e3d381fd3d25 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -1,11 +1,12 @@ // SPDX-License-Identifier: GPL-2.0 -#include -#include +#include #include +#include #include #include #include +#include #include /* Workaround to allow gradual conversion of architecture code */ @@ -60,8 +61,10 @@ static __always_inline unsigned long __exit_to_user_mode_loop(struct pt_regs *re if (ti_work & _TIF_PATCH_PENDING) klp_update_patch_state(current); - if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) + if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) { + futex_fixup_robust_unlock(regs); arch_do_signal_or_restart(regs); + } if (ti_work & _TIF_NOTIFY_RESUME) resume_user_mode_work(regs); diff --git a/kernel/events/core.c b/kernel/events/core.c index a4c884d4ecfb..25985e9cac78 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -58,6 +58,7 @@ #include #include #include +#include #include "internal.h" @@ -7547,6 +7548,33 @@ static int perf_fasync(int fd, struct file *filp, int on) return 0; } +static void perf_show_fdinfo(struct seq_file *m, struct file *f) +{ + struct perf_event *event = f->private_data; + struct perf_event_context *ctx; + struct mutex *child_mutex; + + ctx = perf_event_ctx_lock(event); + child_mutex = event->parent ? &event->parent->child_mutex : &event->child_mutex; + mutex_lock(child_mutex); + + seq_printf(m, "perf_event_attr.type:\t%u\n", event->orig_type); + if (event->pmu) + seq_printf(m, "pmu_type:\t%u\n", event->pmu->type); + seq_printf(m, "perf_event_attr.config:\t0x%llx\n", (unsigned long long)event->attr.config); + seq_printf(m, "perf_event_attr.config1:\t0x%llx\n", + (unsigned long long)event->attr.config1); + seq_printf(m, "perf_event_attr.config2:\t0x%llx\n", + (unsigned long long)event->attr.config2); + seq_printf(m, "perf_event_attr.config3:\t0x%llx\n", + (unsigned long long)event->attr.config3); + seq_printf(m, "perf_event_attr.config4:\t0x%llx\n", + (unsigned long long)event->attr.config4); + + mutex_unlock(child_mutex); + perf_event_ctx_unlock(event, ctx); +} + static const struct file_operations perf_fops = { .release = perf_release, .read = perf_read, @@ -7555,6 +7583,7 @@ static const struct file_operations perf_fops = { .compat_ioctl = perf_compat_ioctl, .mmap = perf_mmap, .fasync = perf_fasync, + .show_fdinfo = perf_show_fdinfo, }; /* diff --git a/kernel/exit.c b/kernel/exit.c index 9a909993ab1d..63c7c6275ffc 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -988,8 +988,8 @@ void __noreturn do_exit(long code) proc_exit_connector(tsk); mpol_put_task_policy(tsk); #ifdef CONFIG_FUTEX - if (unlikely(current->pi_state_cache)) - kfree(current->pi_state_cache); + if (unlikely(current->futex.pi_state_cache)) + kfree(current->futex.pi_state_cache); #endif /* * Make sure we are holding no locks: diff --git a/kernel/fork.c b/kernel/fork.c index e3a6c9bd366d..c826ed7acec9 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1115,6 +1115,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p) #endif mm_init_uprobes_state(mm); hugetlb_count_init(mm); + futex_mm_init(mm); mm_flags_clear_all(mm); if (current->mm) { @@ -1127,11 +1128,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p) mm->def_flags = 0; } - if (futex_mm_init(mm)) - goto fail_mm_init; - if (mm_alloc_pgd(mm)) - goto fail_nopgd; + goto fail_mm_init; if (mm_alloc_id(mm)) goto fail_noid; @@ -1157,8 +1155,6 @@ fail_nocontext: mm_free_id(mm); fail_noid: mm_free_pgd(mm); -fail_nopgd: - futex_hash_free(mm); fail_mm_init: free_mm(mm); return NULL; diff --git a/kernel/futex/core.c b/kernel/futex/core.c index ff2a4fb2993f..179b26e9c934 100644 --- a/kernel/futex/core.c +++ b/kernel/futex/core.c @@ -32,18 +32,21 @@ * "But they come in a choice of three flavours!" */ #include -#include -#include #include -#include -#include -#include -#include #include -#include -#include +#include +#include +#include #include #include +#include +#include +#include +#include +#include +#include + +#include #include "futex.h" #include "../locking/rtmutex_common.h" @@ -124,7 +127,7 @@ late_initcall(fail_futex_debugfs); #endif /* CONFIG_FAIL_FUTEX */ static struct futex_hash_bucket * -__futex_hash(union futex_key *key, struct futex_private_hash *fph); +__futex_hash(union futex_key *key, struct futex_private_hash *fph, struct futex_private_hash **fph_p); #ifdef CONFIG_FUTEX_PRIVATE_HASH static bool futex_ref_get(struct futex_private_hash *fph); @@ -133,15 +136,6 @@ static bool futex_ref_is_dead(struct futex_private_hash *fph); enum { FR_PERCPU = 0, FR_ATOMIC }; -static inline bool futex_key_is_private(union futex_key *key) -{ - /* - * Relies on get_futex_key() to set either bit for shared - * futexes -- see comment with union futex_key. - */ - return !(key->both.offset & (FUT_OFF_INODE | FUT_OFF_MMSHARED)); -} - static bool futex_private_hash_get(struct futex_private_hash *fph) { return futex_ref_get(fph); @@ -149,51 +143,18 @@ static bool futex_private_hash_get(struct futex_private_hash *fph) void futex_private_hash_put(struct futex_private_hash *fph) { - if (futex_ref_put(fph)) + if (fph && futex_ref_put(fph)) wake_up_var(fph->mm); } -/** - * futex_hash_get - Get an additional reference for the local hash. - * @hb: ptr to the private local hash. - * - * Obtain an additional reference for the already obtained hash bucket. The - * caller must already own an reference. - */ -void futex_hash_get(struct futex_hash_bucket *hb) -{ - struct futex_private_hash *fph = hb->priv; - - if (!fph) - return; - WARN_ON_ONCE(!futex_private_hash_get(fph)); -} - -void futex_hash_put(struct futex_hash_bucket *hb) -{ - struct futex_private_hash *fph = hb->priv; - - if (!fph) - return; - futex_private_hash_put(fph); -} - static struct futex_hash_bucket * __futex_hash_private(union futex_key *key, struct futex_private_hash *fph) { u32 hash; - if (!futex_key_is_private(key)) - return NULL; - - if (!fph) - fph = rcu_dereference(key->private.mm->futex_phash); - if (!fph || !fph->hash_mask) - return NULL; - - hash = jhash2((void *)&key->private.address, - sizeof(key->private.address) / 4, + hash = jhash2((void *)&key->private.address, sizeof(key->private.address) / 4, key->both.offset); + return &fph->queues[hash & fph->hash_mask]; } @@ -211,13 +172,12 @@ static void futex_rehash_private(struct futex_private_hash *old, spin_lock(&hb_old->lock); plist_for_each_entry_safe(this, tmp, &hb_old->chain, list) { - plist_del(&this->list, &hb_old->chain); futex_hb_waiters_dec(hb_old); WARN_ON_ONCE(this->lock_ptr != &hb_old->lock); - hb_new = __futex_hash(&this->key, new); + hb_new = __futex_hash(&this->key, new, NULL); futex_hb_waiters_inc(hb_new); /* * The new pointer isn't published yet but an already @@ -232,18 +192,17 @@ static void futex_rehash_private(struct futex_private_hash *old, } } -static bool __futex_pivot_hash(struct mm_struct *mm, - struct futex_private_hash *new) +static bool __futex_pivot_hash(struct mm_struct *mm, struct futex_private_hash *new) { + struct futex_mm_phash *mmph = &mm->futex.phash; struct futex_private_hash *fph; - WARN_ON_ONCE(mm->futex_phash_new); + WARN_ON_ONCE(mmph->hash_new); - fph = rcu_dereference_protected(mm->futex_phash, - lockdep_is_held(&mm->futex_hash_lock)); + fph = rcu_dereference_protected(mmph->hash, lockdep_is_held(&mmph->lock)); if (fph) { if (!futex_ref_is_dead(fph)) { - mm->futex_phash_new = new; + mmph->hash_new = new; return false; } @@ -251,8 +210,8 @@ static bool __futex_pivot_hash(struct mm_struct *mm, } new->state = FR_PERCPU; scoped_guard(rcu) { - mm->futex_batches = get_state_synchronize_rcu(); - rcu_assign_pointer(mm->futex_phash, new); + mmph->batches = get_state_synchronize_rcu(); + rcu_assign_pointer(mmph->hash, new); } kvfree_rcu(fph, rcu); return true; @@ -260,20 +219,19 @@ static bool __futex_pivot_hash(struct mm_struct *mm, static void futex_pivot_hash(struct mm_struct *mm) { - scoped_guard(mutex, &mm->futex_hash_lock) { + scoped_guard(mutex, &mm->futex.phash.lock) { struct futex_private_hash *fph; - fph = mm->futex_phash_new; + fph = mm->futex.phash.hash_new; if (fph) { - mm->futex_phash_new = NULL; + mm->futex.phash.hash_new = NULL; __futex_pivot_hash(mm, fph); } } } -struct futex_private_hash *futex_private_hash(void) +struct futex_private_hash *futex_private_hash(struct mm_struct *mm) { - struct mm_struct *mm = current->mm; /* * Ideally we don't loop. If there is a replacement in progress * then a new private hash is already prepared and a reference can't be @@ -288,7 +246,7 @@ again: scoped_guard(rcu) { struct futex_private_hash *fph; - fph = rcu_dereference(mm->futex_phash); + fph = rcu_dereference(mm->futex.phash.hash); if (!fph) return NULL; @@ -299,18 +257,17 @@ again: goto again; } -struct futex_hash_bucket *futex_hash(union futex_key *key) +struct futex_bucket_ref futex_hash(union futex_key *key) { - struct futex_private_hash *fph; - struct futex_hash_bucket *hb; - again: scoped_guard(rcu) { - hb = __futex_hash(key, NULL); - fph = hb->priv; + struct futex_private_hash *fph = NULL; + struct futex_hash_bucket *hb; + + hb = __futex_hash(key, NULL, &fph); if (!fph || futex_private_hash_get(fph)) - return hb; + return (struct futex_bucket_ref){ .hb = hb, .fph = fph }; } futex_pivot_hash(key->private.mm); goto again; @@ -318,15 +275,9 @@ again: #else /* !CONFIG_FUTEX_PRIVATE_HASH */ -static struct futex_hash_bucket * -__futex_hash_private(union futex_key *key, struct futex_private_hash *fph) +struct futex_bucket_ref futex_hash(union futex_key *key) { - return NULL; -} - -struct futex_hash_bucket *futex_hash(union futex_key *key) -{ - return __futex_hash(key, NULL); + return (struct futex_bucket_ref){ .hb = __futex_hash(key, NULL, NULL), .fph = NULL }; } #endif /* CONFIG_FUTEX_PRIVATE_HASH */ @@ -404,6 +355,8 @@ static int futex_mpol(struct mm_struct *mm, unsigned long addr) * __futex_hash - Return the hash bucket * @key: Pointer to the futex key for which the hash is calculated * @fph: Pointer to private hash if known + * @fph_p: Pointer to a private hash pointer; output for the private hash + * used when set. * * We hash on the keys returned from get_futex_key (see below) and return the * corresponding hash bucket. @@ -412,21 +365,24 @@ static int futex_mpol(struct mm_struct *mm, unsigned long addr) * global hash is returned. */ static struct futex_hash_bucket * -__futex_hash(union futex_key *key, struct futex_private_hash *fph) +__futex_hash(union futex_key *key, struct futex_private_hash *fph, struct futex_private_hash **fph_p) { int node = key->both.node; u32 hash; - if (node == FUTEX_NO_NODE) { - struct futex_hash_bucket *hb; - - hb = __futex_hash_private(key, fph); - if (hb) - return hb; +#ifdef CONFIG_FUTEX_PRIVATE_HASH + if (node == FUTEX_NO_NODE && futex_key_is_private(key)) { + if (!fph) + fph = rcu_dereference(key->private.mm->futex.phash.hash); + if (fph && fph->hash_mask) { + if (fph_p) + *fph_p = fph; + return __futex_hash_private(key, fph); + } } +#endif - hash = jhash2((u32 *)key, - offsetof(typeof(*key), both.offset) / sizeof(u32), + hash = jhash2((u32 *)key, offsetof(typeof(*key), both.offset) / sizeof(u32), key->both.offset); if (node == FUTEX_NO_NODE) { @@ -441,8 +397,7 @@ __futex_hash(union futex_key *key, struct futex_private_hash *fph) */ node = (hash >> futex_hashshift) % nr_node_ids; if (!node_possible(node)) { - node = find_next_bit_wrap(node_possible_map.bits, - nr_node_ids, node); + node = find_next_bit_wrap(node_possible_map.bits, nr_node_ids, node); } } @@ -459,9 +414,8 @@ __futex_hash(union futex_key *key, struct futex_private_hash *fph) * Return: Initialized hrtimer_sleeper structure or NULL if no timeout * value given */ -struct hrtimer_sleeper * -futex_setup_timer(ktime_t *time, struct hrtimer_sleeper *timeout, - int flags, u64 range_ns) +struct hrtimer_sleeper *futex_setup_timer(ktime_t *time, struct hrtimer_sleeper *timeout, + int flags, u64 range_ns) { if (!time) return NULL; @@ -829,7 +783,7 @@ void wait_for_owner_exiting(int ret, struct task_struct *exiting) if (WARN_ON_ONCE(ret == -EBUSY && !exiting)) return; - mutex_lock(&exiting->futex_exit_mutex); + mutex_lock(&exiting->futex.exit_mutex); /* * No point in doing state checking here. If the waiter got here * while the task was in exec()->exec_futex_release() then it can @@ -838,7 +792,7 @@ void wait_for_owner_exiting(int ret, struct task_struct *exiting) * already. Highly unlikely and not a problem. Just one more round * through the futex maze. */ - mutex_unlock(&exiting->futex_exit_mutex); + mutex_unlock(&exiting->futex.exit_mutex); put_task_struct(exiting); } @@ -1012,8 +966,9 @@ void futex_unqueue_pi(struct futex_q *q) * dying task, and do notification if so: */ static int handle_futex_death(u32 __user *uaddr, struct task_struct *curr, - bool pi, bool pending_op) + unsigned int mod, bool pending_op) { + bool pi = !!(mod & FUTEX_ROBUST_MOD_PI); u32 uval, nval, mval; pid_t owner; int err; @@ -1047,7 +1002,7 @@ retry: * * In both cases the following conditions are met: * - * 1) task->robust_list->list_op_pending != NULL + * 1) task->futex.robust_list->list_op_pending != NULL * @pending_op == true * 2) The owner part of user space futex value == 0 * 3) Regular futex: @pi == false @@ -1065,7 +1020,7 @@ retry: owner = uval & FUTEX_TID_MASK; if (pending_op && !pi && !owner) { - futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, 1, + futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, NULL, 1, FUTEX_BITSET_MATCH_ANY); return 0; } @@ -1119,7 +1074,7 @@ retry: * PI futexes happens in exit_pi_state(): */ if (!pi && (uval & FUTEX_WAITERS)) { - futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, 1, + futex_wake(uaddr, FLAGS_SIZE_32 | FLAGS_SHARED, NULL, 1, FUTEX_BITSET_MATCH_ANY); } @@ -1131,31 +1086,30 @@ retry: */ static inline int fetch_robust_entry(struct robust_list __user **entry, struct robust_list __user * __user *head, - unsigned int *pi) + unsigned int *mod) { unsigned long uentry; if (get_user(uentry, (unsigned long __user *)head)) return -EFAULT; - *entry = (void __user *)(uentry & ~1UL); - *pi = uentry & 1; + *entry = (void __user *)(uentry & ~FUTEX_ROBUST_MOD_MASK); + *mod = uentry & FUTEX_ROBUST_MOD_MASK; return 0; } /* - * Walk curr->robust_list (very carefully, it's a userspace list!) + * Walk curr->futex.robust_list (very carefully, it's a userspace list!) * and mark any locks found there dead, and notify any waiters. * * We silently return on any sign of list-walking problem. */ static void exit_robust_list(struct task_struct *curr) { - struct robust_list_head __user *head = curr->robust_list; + struct robust_list_head __user *head = curr->futex.robust_list; + unsigned int limit = ROBUST_LIST_LIMIT, cur_mod, next_mod, pend_mod; struct robust_list __user *entry, *next_entry, *pending; - unsigned int limit = ROBUST_LIST_LIMIT, pi, pip; - unsigned int next_pi; unsigned long futex_offset; int rc; @@ -1163,7 +1117,7 @@ static void exit_robust_list(struct task_struct *curr) * Fetch the list head (which was registered earlier, via * sys_set_robust_list()): */ - if (fetch_robust_entry(&entry, &head->list.next, &pi)) + if (fetch_robust_entry(&entry, &head->list.next, &cur_mod)) return; /* * Fetch the relative futex offset: @@ -1174,7 +1128,7 @@ static void exit_robust_list(struct task_struct *curr) * Fetch any possibly pending lock-add first, and handle it * if it exists: */ - if (fetch_robust_entry(&pending, &head->list_op_pending, &pip)) + if (fetch_robust_entry(&pending, &head->list_op_pending, &pend_mod)) return; next_entry = NULL; /* avoid warning with gcc */ @@ -1183,20 +1137,20 @@ static void exit_robust_list(struct task_struct *curr) * Fetch the next entry in the list before calling * handle_futex_death: */ - rc = fetch_robust_entry(&next_entry, &entry->next, &next_pi); + rc = fetch_robust_entry(&next_entry, &entry->next, &next_mod); /* * A pending lock might already be on the list, so * don't process it twice: */ if (entry != pending) { if (handle_futex_death((void __user *)entry + futex_offset, - curr, pi, HANDLE_DEATH_LIST)) + curr, cur_mod, HANDLE_DEATH_LIST)) return; } if (rc) return; entry = next_entry; - pi = next_pi; + cur_mod = next_mod; /* * Avoid excessively long or circular lists: */ @@ -1208,10 +1162,31 @@ static void exit_robust_list(struct task_struct *curr) if (pending) { handle_futex_death((void __user *)pending + futex_offset, - curr, pip, HANDLE_DEATH_PENDING); + curr, pend_mod, HANDLE_DEATH_PENDING); } } +static bool robust_list_clear_pending(unsigned long __user *pop) +{ + struct robust_list_head __user *head = current->futex.robust_list; + + if (!put_user(0UL, pop)) + return true; + + /* + * Just give up. The robust list head is usually part of TLS, so the + * chance that this gets resolved is close to zero. + * + * If @pop_addr is the robust_list_head::list_op_pending pointer then + * clear the robust list head pointer to prevent further damage when the + * task exits. Better a few stale futexes than corrupted memory. But + * that's mostly an academic exercise. + */ + if (pop == (unsigned long __user *)&head->list_op_pending) + current->futex.robust_list = NULL; + return false; +} + #ifdef CONFIG_COMPAT static void __user *futex_uaddr(struct robust_list __user *entry, compat_long_t futex_offset) @@ -1227,29 +1202,28 @@ static void __user *futex_uaddr(struct robust_list __user *entry, */ static inline int compat_fetch_robust_entry(compat_uptr_t *uentry, struct robust_list __user **entry, - compat_uptr_t __user *head, unsigned int *pi) + compat_uptr_t __user *head, unsigned int *pflags) { if (get_user(*uentry, head)) return -EFAULT; - *entry = compat_ptr((*uentry) & ~1); - *pi = (unsigned int)(*uentry) & 1; + *entry = compat_ptr((*uentry) & ~FUTEX_ROBUST_MOD_MASK); + *pflags = (unsigned int)(*uentry) & FUTEX_ROBUST_MOD_MASK; return 0; } /* - * Walk curr->robust_list (very carefully, it's a userspace list!) + * Walk curr->futex.robust_list (very carefully, it's a userspace list!) * and mark any locks found there dead, and notify any waiters. * * We silently return on any sign of list-walking problem. */ static void compat_exit_robust_list(struct task_struct *curr) { - struct compat_robust_list_head __user *head = curr->compat_robust_list; + struct compat_robust_list_head __user *head = current->futex.compat_robust_list; + unsigned int limit = ROBUST_LIST_LIMIT, cur_mod, next_mod, pend_mod; struct robust_list __user *entry, *next_entry, *pending; - unsigned int limit = ROBUST_LIST_LIMIT, pi, pip; - unsigned int next_pi; compat_uptr_t uentry, next_uentry, upending; compat_long_t futex_offset; int rc; @@ -1258,7 +1232,7 @@ static void compat_exit_robust_list(struct task_struct *curr) * Fetch the list head (which was registered earlier, via * sys_set_robust_list()): */ - if (compat_fetch_robust_entry(&uentry, &entry, &head->list.next, &pi)) + if (compat_fetch_robust_entry(&uentry, &entry, &head->list.next, &cur_mod)) return; /* * Fetch the relative futex offset: @@ -1269,8 +1243,7 @@ static void compat_exit_robust_list(struct task_struct *curr) * Fetch any possibly pending lock-add first, and handle it * if it exists: */ - if (compat_fetch_robust_entry(&upending, &pending, - &head->list_op_pending, &pip)) + if (compat_fetch_robust_entry(&upending, &pending, &head->list_op_pending, &pend_mod)) return; next_entry = NULL; /* avoid warning with gcc */ @@ -1280,7 +1253,7 @@ static void compat_exit_robust_list(struct task_struct *curr) * handle_futex_death: */ rc = compat_fetch_robust_entry(&next_uentry, &next_entry, - (compat_uptr_t __user *)&entry->next, &next_pi); + (compat_uptr_t __user *)&entry->next, &next_mod); /* * A pending lock might already be on the list, so * dont process it twice: @@ -1288,15 +1261,14 @@ static void compat_exit_robust_list(struct task_struct *curr) if (entry != pending) { void __user *uaddr = futex_uaddr(entry, futex_offset); - if (handle_futex_death(uaddr, curr, pi, - HANDLE_DEATH_LIST)) + if (handle_futex_death(uaddr, curr, cur_mod, HANDLE_DEATH_LIST)) return; } if (rc) return; uentry = next_uentry; entry = next_entry; - pi = next_pi; + cur_mod = next_mod; /* * Avoid excessively long or circular lists: */ @@ -1308,9 +1280,24 @@ static void compat_exit_robust_list(struct task_struct *curr) if (pending) { void __user *uaddr = futex_uaddr(pending, futex_offset); - handle_futex_death(uaddr, curr, pip, HANDLE_DEATH_PENDING); + handle_futex_death(uaddr, curr, pend_mod, HANDLE_DEATH_PENDING); } } + +static bool compat_robust_list_clear_pending(u32 __user *pop) +{ + struct compat_robust_list_head __user *head = current->futex.compat_robust_list; + + if (!put_user(0U, pop)) + return true; + + /* See comment in robust_list_clear_pending(). */ + if (pop == &head->list_op_pending) + current->futex.compat_robust_list = NULL; + return false; +} +#else +static bool compat_robust_list_clear_pending(u32 __user *pop_addr) { return false; } #endif #ifdef CONFIG_FUTEX_PI @@ -1322,7 +1309,7 @@ static void compat_exit_robust_list(struct task_struct *curr) */ static void exit_pi_state_list(struct task_struct *curr) { - struct list_head *next, *head = &curr->pi_state_list; + struct list_head *next, *head = &curr->futex.pi_state_list; struct futex_pi_state *pi_state; union futex_key key = FUTEX_KEY_INIT; @@ -1336,7 +1323,7 @@ static void exit_pi_state_list(struct task_struct *curr) * on the mutex. */ WARN_ON(curr != current); - guard(private_hash)(); + guard(private_hash)(current->mm); /* * We are a ZOMBIE and nobody can enqueue itself on * pi_state_list anymore, but we have to be careful @@ -1348,7 +1335,8 @@ static void exit_pi_state_list(struct task_struct *curr) pi_state = list_entry(next, struct futex_pi_state, list); key = pi_state->key; if (1) { - CLASS(hb, hb)(&key); + CLASS(hbr, hbr)(&key); + auto hb = hbr.hb; /* * We can race against put_pi_state() removing itself from the @@ -1404,21 +1392,50 @@ static void exit_pi_state_list(struct task_struct *curr) static inline void exit_pi_state_list(struct task_struct *curr) { } #endif +bool futex_robust_list_clear_pending(void __user *pop, unsigned int flags) +{ + bool size32bit = !!(flags & FLAGS_ROBUST_LIST32); + + if (!IS_ENABLED(CONFIG_64BIT) && !size32bit) + return false; + + if (IS_ENABLED(CONFIG_64BIT) && size32bit) + return compat_robust_list_clear_pending(pop); + + return robust_list_clear_pending(pop); +} + +#ifdef CONFIG_FUTEX_ROBUST_UNLOCK +void __futex_fixup_robust_unlock(struct pt_regs *regs, struct futex_unlock_cs_range *csr) +{ + /* + * arch_futex_robust_unlock_get_pop() returns the list pending op pointer from + * @regs if the try_cmpxchg() succeeded. + */ + void __user *pop = arch_futex_robust_unlock_get_pop(regs); + + if (!pop) + return; + + futex_robust_list_clear_pending(pop, csr->pop_size32 ? FLAGS_ROBUST_LIST32 : 0); +} +#endif /* CONFIG_FUTEX_ROBUST_UNLOCK */ + static void futex_cleanup(struct task_struct *tsk) { - if (unlikely(tsk->robust_list)) { + if (unlikely(tsk->futex.robust_list)) { exit_robust_list(tsk); - tsk->robust_list = NULL; + tsk->futex.robust_list = NULL; } #ifdef CONFIG_COMPAT - if (unlikely(tsk->compat_robust_list)) { + if (unlikely(tsk->futex.compat_robust_list)) { compat_exit_robust_list(tsk); - tsk->compat_robust_list = NULL; + tsk->futex.compat_robust_list = NULL; } #endif - if (unlikely(!list_empty(&tsk->pi_state_list))) + if (unlikely(!list_empty(&tsk->futex.pi_state_list))) exit_pi_state_list(tsk); } @@ -1442,23 +1459,23 @@ static void futex_cleanup(struct task_struct *tsk) void futex_exit_recursive(struct task_struct *tsk) { /* If the state is FUTEX_STATE_EXITING then futex_exit_mutex is held */ - if (tsk->futex_state == FUTEX_STATE_EXITING) { - __assume_ctx_lock(&tsk->futex_exit_mutex); - mutex_unlock(&tsk->futex_exit_mutex); + if (tsk->futex.state == FUTEX_STATE_EXITING) { + __assume_ctx_lock(&tsk->futex.exit_mutex); + mutex_unlock(&tsk->futex.exit_mutex); } - tsk->futex_state = FUTEX_STATE_DEAD; + tsk->futex.state = FUTEX_STATE_DEAD; } static void futex_cleanup_begin(struct task_struct *tsk) - __acquires(&tsk->futex_exit_mutex) + __acquires(&tsk->futex.exit_mutex) { /* * Prevent various race issues against a concurrent incoming waiter * including live locks by forcing the waiter to block on - * tsk->futex_exit_mutex when it observes FUTEX_STATE_EXITING in + * tsk->futex.exit_mutex when it observes FUTEX_STATE_EXITING in * attach_to_pi_owner(). */ - mutex_lock(&tsk->futex_exit_mutex); + mutex_lock(&tsk->futex.exit_mutex); /* * Switch the state to FUTEX_STATE_EXITING under tsk->pi_lock. @@ -1472,23 +1489,23 @@ static void futex_cleanup_begin(struct task_struct *tsk) * be observed in exit_pi_state_list(). */ raw_spin_lock_irq(&tsk->pi_lock); - tsk->futex_state = FUTEX_STATE_EXITING; + tsk->futex.state = FUTEX_STATE_EXITING; raw_spin_unlock_irq(&tsk->pi_lock); } static void futex_cleanup_end(struct task_struct *tsk, int state) - __releases(&tsk->futex_exit_mutex) + __releases(&tsk->futex.exit_mutex) { /* * Lockless store. The only side effect is that an observer might * take another loop until it becomes visible. */ - tsk->futex_state = state; + tsk->futex.state = state; /* * Drop the exit protection. This unblocks waiters which observed * FUTEX_STATE_EXITING to reevaluate the state. */ - mutex_unlock(&tsk->futex_exit_mutex); + mutex_unlock(&tsk->futex.exit_mutex); } void futex_exec_release(struct task_struct *tsk) @@ -1516,12 +1533,8 @@ void futex_exit_release(struct task_struct *tsk) futex_cleanup_end(tsk, FUTEX_STATE_DEAD); } -static void futex_hash_bucket_init(struct futex_hash_bucket *fhb, - struct futex_private_hash *fph) +static void futex_hash_bucket_init(struct futex_hash_bucket *fhb) { -#ifdef CONFIG_FUTEX_PRIVATE_HASH - fhb->priv = fph; -#endif atomic_set(&fhb->waiters, 0); plist_head_init(&fhb->chain); spin_lock_init(&fhb->lock); @@ -1553,17 +1566,17 @@ static void __futex_ref_atomic_begin(struct futex_private_hash *fph) * otherwise it would be impossible for it to have reported success * from futex_ref_is_dead(). */ - WARN_ON_ONCE(atomic_long_read(&mm->futex_atomic) != 0); + WARN_ON_ONCE(atomic_long_read(&mm->futex.phash.atomic) != 0); /* * Set the atomic to the bias value such that futex_ref_{get,put}() * will never observe 0. Will be fixed up in __futex_ref_atomic_end() * when folding in the percpu count. */ - atomic_long_set(&mm->futex_atomic, LONG_MAX); + atomic_long_set(&mm->futex.phash.atomic, LONG_MAX); smp_store_release(&fph->state, FR_ATOMIC); - call_rcu_hurry(&mm->futex_rcu, futex_ref_rcu); + call_rcu_hurry(&mm->futex.phash.rcu, futex_ref_rcu); } static void __futex_ref_atomic_end(struct futex_private_hash *fph) @@ -1584,7 +1597,7 @@ static void __futex_ref_atomic_end(struct futex_private_hash *fph) * Therefore the per-cpu counter is now stable, sum and reset. */ for_each_possible_cpu(cpu) { - unsigned int *ptr = per_cpu_ptr(mm->futex_ref, cpu); + unsigned int *ptr = per_cpu_ptr(mm->futex.phash.ref, cpu); count += *ptr; *ptr = 0; } @@ -1592,7 +1605,7 @@ static void __futex_ref_atomic_end(struct futex_private_hash *fph) /* * Re-init for the next cycle. */ - this_cpu_inc(*mm->futex_ref); /* 0 -> 1 */ + this_cpu_inc(*mm->futex.phash.ref); /* 0 -> 1 */ /* * Add actual count, subtract bias and initial refcount. @@ -1600,7 +1613,7 @@ static void __futex_ref_atomic_end(struct futex_private_hash *fph) * The moment this atomic operation happens, futex_ref_is_dead() can * become true. */ - ret = atomic_long_add_return(count - LONG_MAX - 1, &mm->futex_atomic); + ret = atomic_long_add_return(count - LONG_MAX - 1, &mm->futex.phash.atomic); if (!ret) wake_up_var(mm); @@ -1610,8 +1623,8 @@ static void __futex_ref_atomic_end(struct futex_private_hash *fph) static void futex_ref_rcu(struct rcu_head *head) { - struct mm_struct *mm = container_of(head, struct mm_struct, futex_rcu); - struct futex_private_hash *fph = rcu_dereference_raw(mm->futex_phash); + struct mm_struct *mm = container_of(head, struct mm_struct, futex.phash.rcu); + struct futex_private_hash *fph = rcu_dereference_raw(mm->futex.phash.hash); if (fph->state == FR_PERCPU) { /* @@ -1640,7 +1653,7 @@ static void futex_ref_drop(struct futex_private_hash *fph) /* * Can only transition the current fph; */ - WARN_ON_ONCE(rcu_dereference_raw(mm->futex_phash) != fph); + WARN_ON_ONCE(rcu_dereference_raw(mm->futex.phash.hash) != fph); /* * We enqueue at least one RCU callback. Ensure mm stays if the task * exits before the transition is completed. @@ -1651,9 +1664,9 @@ static void futex_ref_drop(struct futex_private_hash *fph) * In order to avoid the following scenario: * * futex_hash() __futex_pivot_hash() - * guard(rcu); guard(mm->futex_hash_lock); - * fph = mm->futex_phash; - * rcu_assign_pointer(&mm->futex_phash, new); + * guard(rcu); guard(mm->futex.phash.lock); + * fph = mm->futex.phash.hash; + * rcu_assign_pointer(&mm->futex.phash.hash, new); * futex_hash_allocate() * futex_ref_drop() * fph->state = FR_ATOMIC; @@ -1668,7 +1681,7 @@ static void futex_ref_drop(struct futex_private_hash *fph) * There must be at least one full grace-period between publishing a * new fph and trying to replace it. */ - if (poll_state_synchronize_rcu(mm->futex_batches)) { + if (poll_state_synchronize_rcu(mm->futex.phash.batches)) { /* * There was a grace-period, we can begin now. */ @@ -1676,7 +1689,7 @@ static void futex_ref_drop(struct futex_private_hash *fph) return; } - call_rcu_hurry(&mm->futex_rcu, futex_ref_rcu); + call_rcu_hurry(&mm->futex.phash.rcu, futex_ref_rcu); } static bool futex_ref_get(struct futex_private_hash *fph) @@ -1686,11 +1699,11 @@ static bool futex_ref_get(struct futex_private_hash *fph) guard(preempt)(); if (READ_ONCE(fph->state) == FR_PERCPU) { - __this_cpu_inc(*mm->futex_ref); + __this_cpu_inc(*mm->futex.phash.ref); return true; } - return atomic_long_inc_not_zero(&mm->futex_atomic); + return atomic_long_inc_not_zero(&mm->futex.phash.atomic); } static bool futex_ref_put(struct futex_private_hash *fph) @@ -1700,11 +1713,11 @@ static bool futex_ref_put(struct futex_private_hash *fph) guard(preempt)(); if (READ_ONCE(fph->state) == FR_PERCPU) { - __this_cpu_dec(*mm->futex_ref); + __this_cpu_dec(*mm->futex.phash.ref); return false; } - return atomic_long_dec_and_test(&mm->futex_atomic); + return atomic_long_dec_and_test(&mm->futex.phash.atomic); } static bool futex_ref_is_dead(struct futex_private_hash *fph) @@ -1716,28 +1729,23 @@ static bool futex_ref_is_dead(struct futex_private_hash *fph) if (smp_load_acquire(&fph->state) == FR_PERCPU) return false; - return atomic_long_read(&mm->futex_atomic) == 0; + return atomic_long_read(&mm->futex.phash.atomic) == 0; } -int futex_mm_init(struct mm_struct *mm) +static void futex_hash_init_mm(struct futex_mm_data *fd) { - mutex_init(&mm->futex_hash_lock); - RCU_INIT_POINTER(mm->futex_phash, NULL); - mm->futex_phash_new = NULL; - /* futex-ref */ - mm->futex_ref = NULL; - atomic_long_set(&mm->futex_atomic, 0); - mm->futex_batches = get_state_synchronize_rcu(); - return 0; + memset(&fd->phash, 0, sizeof(fd->phash)); + mutex_init(&fd->phash.lock); + fd->phash.batches = get_state_synchronize_rcu(); } void futex_hash_free(struct mm_struct *mm) { struct futex_private_hash *fph; - free_percpu(mm->futex_ref); - kvfree(mm->futex_phash_new); - fph = rcu_dereference_raw(mm->futex_phash); + free_percpu(mm->futex.phash.ref); + kvfree(mm->futex.phash.hash_new); + fph = rcu_dereference_raw(mm->futex.phash.hash); if (fph) kvfree(fph); } @@ -1748,10 +1756,10 @@ static bool futex_pivot_pending(struct mm_struct *mm) guard(rcu)(); - if (!mm->futex_phash_new) + if (!mm->futex.phash.hash_new) return true; - fph = rcu_dereference(mm->futex_phash); + fph = rcu_dereference(mm->futex.phash.hash); return futex_ref_is_dead(fph); } @@ -1793,7 +1801,7 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) * Once we've disabled the global hash there is no way back. */ scoped_guard(rcu) { - fph = rcu_dereference(mm->futex_phash); + fph = rcu_dereference(mm->futex.phash.hash); if (fph && !fph->hash_mask) { if (custom) return -EBUSY; @@ -1801,15 +1809,15 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) } } - if (!mm->futex_ref) { + if (!mm->futex.phash.ref) { /* * This will always be allocated by the first thread and * therefore requires no locking. */ - mm->futex_ref = alloc_percpu(unsigned int); - if (!mm->futex_ref) + mm->futex.phash.ref = alloc_percpu(unsigned int); + if (!mm->futex.phash.ref) return -ENOMEM; - this_cpu_inc(*mm->futex_ref); /* 0 -> 1 */ + this_cpu_inc(*mm->futex.phash.ref); /* 0 -> 1 */ } fph = kvzalloc(struct_size(fph, queues, hash_slots), @@ -1822,7 +1830,7 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) fph->mm = mm; for (i = 0; i < hash_slots; i++) - futex_hash_bucket_init(&fph->queues[i], fph); + futex_hash_bucket_init(&fph->queues[i]); if (custom) { /* @@ -1832,14 +1840,14 @@ again: wait_var_event(mm, futex_pivot_pending(mm)); } - scoped_guard(mutex, &mm->futex_hash_lock) { + scoped_guard(mutex, &mm->futex.phash.lock) { struct futex_private_hash *free __free(kvfree) = NULL; struct futex_private_hash *cur, *new; - cur = rcu_dereference_protected(mm->futex_phash, - lockdep_is_held(&mm->futex_hash_lock)); - new = mm->futex_phash_new; - mm->futex_phash_new = NULL; + cur = rcu_dereference_protected(mm->futex.phash.hash, + lockdep_is_held(&mm->futex.phash.lock)); + new = mm->futex.phash.hash_new; + mm->futex.phash.hash_new = NULL; if (fph) { if (cur && !cur->hash_mask) { @@ -1849,7 +1857,7 @@ again: * the second one returns here. */ free = fph; - mm->futex_phash_new = new; + mm->futex.phash.hash_new = new; return -EBUSY; } if (cur && !new) { @@ -1879,7 +1887,7 @@ again: if (new) { /* - * Will set mm->futex_phash_new on failure; + * Will set mm->futex.phash.new_hash on failure; * futex_private_hash_get() will try again. */ if (!__futex_pivot_hash(mm, new) && custom) @@ -1898,11 +1906,9 @@ int futex_hash_allocate_default(void) return 0; scoped_guard(rcu) { - threads = min_t(unsigned int, - get_nr_threads(current), - num_online_cpus()); + threads = min_t(unsigned int, get_nr_threads(current), num_online_cpus()); - fph = rcu_dereference(current->mm->futex_phash); + fph = rcu_dereference(current->mm->futex.phash.hash); if (fph) { if (fph->custom) return 0; @@ -1929,24 +1935,52 @@ static int futex_hash_get_slots(void) struct futex_private_hash *fph; guard(rcu)(); - fph = rcu_dereference(current->mm->futex_phash); + fph = rcu_dereference(current->mm->futex.phash.hash); if (fph && fph->hash_mask) return fph->hash_mask + 1; return 0; } +#else /* CONFIG_FUTEX_PRIVATE_HASH */ +static inline int futex_hash_allocate(unsigned int hslots, unsigned int flags) { return -EINVAL; } +static inline int futex_hash_get_slots(void) { return 0; } +static inline void futex_hash_init_mm(struct futex_mm_data *fd) { } +#endif /* !CONFIG_FUTEX_PRIVATE_HASH */ -#else - -static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) +#ifdef CONFIG_FUTEX_ROBUST_UNLOCK +static void futex_invalidate_cs_ranges(struct futex_mm_data *fd) { - return -EINVAL; + /* + * Invalidate start_ip so that the quick check fails for ip >= start_ip + * if VDSO is not mapped or the second slot is not available for compat + * tasks as they use VDSO32 which does not provide the 64-bit pointer + * variant. + */ + for (int i = 0; i < FUTEX_ROBUST_MAX_CS_RANGES; i++) + fd->unlock.cs_ranges[i].start_ip = ~0UL; } -static int futex_hash_get_slots(void) +void futex_reset_cs_ranges(struct futex_mm_data *fd) { - return 0; + memset(fd->unlock.cs_ranges, 0, sizeof(fd->unlock.cs_ranges)); + futex_invalidate_cs_ranges(fd); } +static void futex_robust_unlock_init_mm(struct futex_mm_data *fd) +{ + /* mm_dup() preserves the range, mm_alloc() clears it */ + if (!fd->unlock.cs_ranges[0].start_ip) + futex_invalidate_cs_ranges(fd); +} +#else /* CONFIG_FUTEX_ROBUST_UNLOCK */ +static inline void futex_robust_unlock_init_mm(struct futex_mm_data *fd) { } +#endif /* !CONFIG_FUTEX_ROBUST_UNLOCK */ + +#if defined(CONFIG_FUTEX_PRIVATE_HASH) || defined(CONFIG_FUTEX_ROBUST_UNLOCK) +void futex_mm_init(struct mm_struct *mm) +{ + futex_hash_init_mm(&mm->futex); + futex_robust_unlock_init_mm(&mm->futex); +} #endif int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4) @@ -2001,7 +2035,7 @@ static int __init futex_init(void) BUG_ON(!table); for (i = 0; i < hashsize; i++) - futex_hash_bucket_init(&table[i], NULL); + futex_hash_bucket_init(&table[i]); futex_queues[n] = table; } diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 9f6bf6f585fc..f00f0863ed44 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -40,6 +40,8 @@ #define FLAGS_NUMA 0x0080 #define FLAGS_STRICT 0x0100 #define FLAGS_MPOL 0x0200 +#define FLAGS_ROBUST_UNLOCK 0x0400 +#define FLAGS_ROBUST_LIST32 0x0800 /* FUTEX_ to FLAGS_ */ static inline unsigned int futex_to_flags(unsigned int op) @@ -52,6 +54,12 @@ static inline unsigned int futex_to_flags(unsigned int op) if (op & FUTEX_CLOCK_REALTIME) flags |= FLAGS_CLOCKRT; + if (op & FUTEX_ROBUST_UNLOCK) + flags |= FLAGS_ROBUST_UNLOCK; + + if (op & FUTEX_ROBUST_LIST32) + flags |= FLAGS_ROBUST_LIST32; + return flags; } @@ -126,6 +134,15 @@ static inline bool should_fail_futex(bool fshared) } #endif +static inline bool futex_key_is_private(union futex_key *key) +{ + /* + * Relies on get_futex_key() to set either bit for shared + * futexes -- see comment with union futex_key. + */ + return !(key->both.offset & (FUT_OFF_INODE | FUT_OFF_MMSHARED)); +} + /* * Hash buckets are shared by all the futex_keys that hash to the same * location. Each key may have multiple futex_q structures, one for each task @@ -135,7 +152,6 @@ struct futex_hash_bucket { atomic_t waiters; spinlock_t lock; struct plist_head chain; - struct futex_private_hash *priv; } ____cacheline_aligned_in_smp; /* @@ -175,7 +191,7 @@ typedef void (futex_wake_fn)(struct wake_q_head *wake_q, struct futex_q *q); * @requeue_pi_key: the requeue_pi target futex key * @bitset: bitset for the optional bitmasked wakeup * @requeue_state: State field for futex_requeue_pi() - * @drop_hb_ref: Waiter should drop the extra hash bucket reference if true + * @drop_fph: Waiter should drop the extra private hash reference when set * @requeue_wait: RCU wait for futex_requeue_pi() (RT only) * * We use this hashed waitqueue, instead of a normal wait_queue_entry_t, so @@ -202,7 +218,7 @@ struct futex_q { union futex_key *requeue_pi_key; u32 bitset; atomic_t requeue_state; - bool drop_hb_ref; + struct futex_private_hash *drop_fph; #ifdef CONFIG_PREEMPT_RT struct rcuwait requeue_wait; #endif @@ -222,28 +238,29 @@ extern struct hrtimer_sleeper * futex_setup_timer(ktime_t *time, struct hrtimer_sleeper *timeout, int flags, u64 range_ns); -extern struct futex_hash_bucket *futex_hash(union futex_key *key); -#ifdef CONFIG_FUTEX_PRIVATE_HASH -extern void futex_hash_get(struct futex_hash_bucket *hb); -extern void futex_hash_put(struct futex_hash_bucket *hb); +struct futex_bucket_ref { + struct futex_hash_bucket *hb; + struct futex_private_hash *fph; +}; -extern struct futex_private_hash *futex_private_hash(void); +#ifdef CONFIG_FUTEX_PRIVATE_HASH +extern struct futex_private_hash *futex_private_hash(struct mm_struct *mm); extern void futex_private_hash_put(struct futex_private_hash *fph); #else /* !CONFIG_FUTEX_PRIVATE_HASH */ -static inline void futex_hash_get(struct futex_hash_bucket *hb) { } -static inline void futex_hash_put(struct futex_hash_bucket *hb) { } -static inline struct futex_private_hash *futex_private_hash(void) { return NULL; } +static inline struct futex_private_hash *futex_private_hash(struct mm_struct *mm) { return NULL; } static inline void futex_private_hash_put(struct futex_private_hash *fph) { } #endif -DEFINE_CLASS(hb, struct futex_hash_bucket *, - if (_T) futex_hash_put(_T), +extern struct futex_bucket_ref futex_hash(union futex_key *key); + +DEFINE_CLASS(hbr, struct futex_bucket_ref, + if (_T.fph) futex_private_hash_put(_T.fph), futex_hash(key), union futex_key *key); DEFINE_CLASS(private_hash, struct futex_private_hash *, if (_T) futex_private_hash_put(_T), - futex_private_hash(), void); + futex_private_hash(mm), struct mm_struct *mm); /** * futex_match - Check whether two futex keys are equal @@ -449,13 +466,16 @@ extern int futex_unqueue_multiple(struct futex_vector *v, int count); extern int futex_wait_multiple(struct futex_vector *vs, unsigned int count, struct hrtimer_sleeper *to); -extern int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset); +extern int futex_wake(u32 __user *uaddr, unsigned int flags, void __user *pop, + int nr_wake, u32 bitset); extern int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, int nr_wake, int nr_wake2, int op); -extern int futex_unlock_pi(u32 __user *uaddr, unsigned int flags); +extern int futex_unlock_pi(u32 __user *uaddr, unsigned int flags, void __user *pop); extern int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int trylock); +bool futex_robust_list_clear_pending(void __user *pop, unsigned int flags); + #endif /* _FUTEX_H */ diff --git a/kernel/futex/pi.c b/kernel/futex/pi.c index 643199fdbe62..795011ea1202 100644 --- a/kernel/futex/pi.c +++ b/kernel/futex/pi.c @@ -14,7 +14,7 @@ int refill_pi_state_cache(void) { struct futex_pi_state *pi_state; - if (likely(current->pi_state_cache)) + if (likely(current->futex.pi_state_cache)) return 0; pi_state = kzalloc_obj(*pi_state); @@ -28,17 +28,17 @@ int refill_pi_state_cache(void) refcount_set(&pi_state->refcount, 1); pi_state->key = FUTEX_KEY_INIT; - current->pi_state_cache = pi_state; + current->futex.pi_state_cache = pi_state; return 0; } static struct futex_pi_state *alloc_pi_state(void) { - struct futex_pi_state *pi_state = current->pi_state_cache; + struct futex_pi_state *pi_state = current->futex.pi_state_cache; WARN_ON(!pi_state); - current->pi_state_cache = NULL; + current->futex.pi_state_cache = NULL; return pi_state; } @@ -60,7 +60,7 @@ static void pi_state_update_owner(struct futex_pi_state *pi_state, if (new_owner) { raw_spin_lock(&new_owner->pi_lock); WARN_ON(!list_empty(&pi_state->list)); - list_add(&pi_state->list, &new_owner->pi_state_list); + list_add(&pi_state->list, &new_owner->futex.pi_state_list); pi_state->owner = new_owner; raw_spin_unlock(&new_owner->pi_lock); } @@ -96,7 +96,7 @@ void put_pi_state(struct futex_pi_state *pi_state) raw_spin_unlock_irqrestore(&pi_state->pi_mutex.wait_lock, flags); } - if (current->pi_state_cache) { + if (current->futex.pi_state_cache) { kfree(pi_state); } else { /* @@ -106,7 +106,7 @@ void put_pi_state(struct futex_pi_state *pi_state) */ pi_state->owner = NULL; refcount_set(&pi_state->refcount, 1); - current->pi_state_cache = pi_state; + current->futex.pi_state_cache = pi_state; } } @@ -179,7 +179,7 @@ void put_pi_state(struct futex_pi_state *pi_state) * * p->pi_lock: * - * p->pi_state_list -> pi_state->list, relation + * p->futex.pi_state_list -> pi_state->list, relation * pi_mutex->owner -> pi_state->owner, relation * * pi_state->refcount: @@ -327,7 +327,7 @@ static int handle_exit_race(u32 __user *uaddr, u32 uval, * If the futex exit state is not yet FUTEX_STATE_DEAD, tell the * caller that the alleged owner is busy. */ - if (tsk && tsk->futex_state != FUTEX_STATE_DEAD) + if (tsk && tsk->futex.state != FUTEX_STATE_DEAD) return -EBUSY; /* @@ -346,8 +346,8 @@ static int handle_exit_race(u32 __user *uaddr, u32 uval, * *uaddr = 0xC0000000; tsk = get_task(PID); * } if (!tsk->flags & PF_EXITING) { * ... attach(); - * tsk->futex_state = } else { - * FUTEX_STATE_DEAD; if (tsk->futex_state != + * tsk->futex.state = } else { + * FUTEX_STATE_DEAD; if (tsk->futex.state != * FUTEX_STATE_DEAD) * return -EAGAIN; * return -ESRCH; <--- FAIL @@ -396,7 +396,7 @@ static void __attach_to_pi_owner(struct task_struct *p, union futex_key *key, pi_state->key = *key; WARN_ON(!list_empty(&pi_state->list)); - list_add(&pi_state->list, &p->pi_state_list); + list_add(&pi_state->list, &p->futex.pi_state_list); /* * Assignment without holding pi_state->pi_mutex.wait_lock is safe * because there is no concurrency as the object is not published yet. @@ -440,7 +440,7 @@ static int attach_to_pi_owner(u32 __user *uaddr, u32 uval, union futex_key *key, * in futex_exit_release(), we do this protected by p->pi_lock: */ raw_spin_lock_irq(&p->pi_lock); - if (unlikely(p->futex_state != FUTEX_STATE_OK)) { + if (unlikely(p->futex.state != FUTEX_STATE_OK)) { /* * The task is on the way out. When the futex state is * FUTEX_STATE_DEAD, we know that the task has finished @@ -945,7 +945,8 @@ retry: retry_private: if (1) { - CLASS(hb, hb)(&q.key); + CLASS(hbr, hbr)(&q.key); + auto hb = hbr.hb; futex_q_lock(&q, hb); @@ -1009,7 +1010,7 @@ retry_private: * the thread, performing resize, will block on hb->lock during * the requeue. */ - futex_hash_put(no_free_ptr(hb)); + futex_private_hash_put(no_free_ptr(hbr.fph)); /* * Must be done before we enqueue the waiter, here is unfortunately * under the hb lock, but that *should* work because it does nothing. @@ -1100,11 +1101,9 @@ no_block: __release(&hb->lock); futex_unqueue_pi(&q); spin_unlock(q.lock_ptr); - if (q.drop_hb_ref) { - CLASS(hb, hb)(&q.key); - /* Additional reference from futex_unlock_pi() */ - futex_hash_put(hb); - } + + /* Additional reference from futex_unlock_pi() */ + futex_private_hash_put(q.drop_fph); goto out; out_unlock_put_key: @@ -1139,7 +1138,7 @@ out: * This is the in-kernel slowpath: we look up the PI state (if any), * and do the rt-mutex unlock. */ -int futex_unlock_pi(u32 __user *uaddr, unsigned int flags) +static int __futex_unlock_pi(u32 __user *uaddr, unsigned int flags) { u32 curval, uval, vpid = task_pid_vnr(current); union futex_key key = FUTEX_KEY_INIT; @@ -1148,7 +1147,6 @@ int futex_unlock_pi(u32 __user *uaddr, unsigned int flags) if (!IS_ENABLED(CONFIG_FUTEX_PI)) return -ENOSYS; - retry: if (get_user(uval, uaddr)) return -EFAULT; @@ -1162,7 +1160,8 @@ retry: if (ret) return ret; - CLASS(hb, hb)(&key); + CLASS(hbr, hbr)(&key); + auto hb = hbr.hb; spin_lock(&hb->lock); retry_hb: @@ -1219,8 +1218,9 @@ retry_hb: * Acquire a reference for the leaving waiter to ensure * valid futex_q::lock_ptr. */ - futex_hash_get(hb); - top_waiter->drop_hb_ref = true; + if (futex_key_is_private(&key)) + top_waiter->drop_fph = futex_private_hash(key.private.mm); + __futex_unqueue(top_waiter); raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock); goto retry_hb; @@ -1302,3 +1302,15 @@ pi_faulted: return ret; } +int futex_unlock_pi(u32 __user *uaddr, unsigned int flags, void __user *pop) +{ + int ret = __futex_unlock_pi(uaddr, flags); + + if (ret || !(flags & FLAGS_ROBUST_UNLOCK)) + return ret; + + if (!futex_robust_list_clear_pending(pop, flags)) + return -EFAULT; + + return 0; +} diff --git a/kernel/futex/requeue.c b/kernel/futex/requeue.c index 1d99a84dc9ad..7384672916fb 100644 --- a/kernel/futex/requeue.c +++ b/kernel/futex/requeue.c @@ -241,8 +241,8 @@ void requeue_pi_wake_futex(struct futex_q *q, union futex_key *key, * Acquire a reference for the waiter to ensure valid * futex_q::lock_ptr. */ - futex_hash_get(hb); - q->drop_hb_ref = true; + if (futex_key_is_private(key)) + q->drop_fph = futex_private_hash(key->private.mm); q->lock_ptr = &hb->lock; task = READ_ONCE(q->task); @@ -459,8 +459,10 @@ retry: retry_private: if (1) { - CLASS(hb, hb1)(&key1); - CLASS(hb, hb2)(&key2); + CLASS(hbr, hbr1)(&key1); + CLASS(hbr, hbr2)(&key2); + auto hb1 = hbr1.hb; + auto hb2 = hbr2.hb; futex_hb_waiters_inc(hb2); double_lock_hb(hb1, hb2); @@ -838,7 +840,8 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags, switch (futex_requeue_pi_wakeup_sync(&q)) { case Q_REQUEUE_PI_IGNORE: { - CLASS(hb, hb)(&q.key); + CLASS(hbr, hbr)(&q.key); + auto hb = hbr.hb; /* The waiter is still on uaddr1 */ spin_lock(&hb->lock); ret = handle_early_requeue_pi_wakeup(hb, &q, to); @@ -908,11 +911,8 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags, default: BUG(); } - if (q.drop_hb_ref) { - CLASS(hb, hb)(&q.key); - /* Additional reference from requeue_pi_wake_futex() */ - futex_hash_put(hb); - } + /* Additional reference from requeue_pi_wake_futex() */ + futex_private_hash_put(q.drop_fph); out: if (to) { diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index 77ad9691f6a6..2fa19d9d008d 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -25,17 +25,13 @@ * @head: pointer to the list-head * @len: length of the list-head, as userspace expects */ -SYSCALL_DEFINE2(set_robust_list, struct robust_list_head __user *, head, - size_t, len) +SYSCALL_DEFINE2(set_robust_list, struct robust_list_head __user *, head, size_t, len) { - /* - * The kernel knows only one size for now: - */ + /* The kernel knows only one size for now. */ if (unlikely(len != sizeof(*head))) return -EINVAL; - current->robust_list = head; - + current->futex.robust_list = head; return 0; } @@ -43,9 +39,9 @@ static inline void __user *futex_task_robust_list(struct task_struct *p, bool co { #ifdef CONFIG_COMPAT if (compat) - return p->compat_robust_list; + return p->futex.compat_robust_list; #endif - return p->robust_list; + return p->futex.robust_list; } static void __user *futex_get_robust_list_common(int pid, bool compat) @@ -122,6 +118,13 @@ long do_futex(u32 __user *uaddr, int op, u32 val, ktime_t *timeout, return -ENOSYS; } + if (flags & FLAGS_ROBUST_UNLOCK) { + if (cmd != FUTEX_WAKE && + cmd != FUTEX_WAKE_BITSET && + cmd != FUTEX_UNLOCK_PI) + return -ENOSYS; + } + switch (cmd) { case FUTEX_WAIT: val3 = FUTEX_BITSET_MATCH_ANY; @@ -132,7 +135,7 @@ long do_futex(u32 __user *uaddr, int op, u32 val, ktime_t *timeout, val3 = FUTEX_BITSET_MATCH_ANY; fallthrough; case FUTEX_WAKE_BITSET: - return futex_wake(uaddr, flags, val, val3); + return futex_wake(uaddr, flags, uaddr2, val, val3); case FUTEX_REQUEUE: return futex_requeue(uaddr, flags, uaddr2, flags, val, val2, NULL, 0); case FUTEX_CMP_REQUEUE: @@ -145,7 +148,7 @@ long do_futex(u32 __user *uaddr, int op, u32 val, ktime_t *timeout, case FUTEX_LOCK_PI2: return futex_lock_pi(uaddr, flags, timeout, 0); case FUTEX_UNLOCK_PI: - return futex_unlock_pi(uaddr, flags); + return futex_unlock_pi(uaddr, flags, uaddr2); case FUTEX_TRYLOCK_PI: return futex_lock_pi(uaddr, flags, NULL, 1); case FUTEX_WAIT_REQUEUE_PI: @@ -379,7 +382,7 @@ SYSCALL_DEFINE4(futex_wake, if (!futex_validate_input(flags, mask)) return -EINVAL; - return futex_wake(uaddr, FLAGS_STRICT | flags, nr, mask); + return futex_wake(uaddr, FLAGS_STRICT | flags, NULL, nr, mask); } /* @@ -475,15 +478,13 @@ SYSCALL_DEFINE4(futex_requeue, } #ifdef CONFIG_COMPAT -COMPAT_SYSCALL_DEFINE2(set_robust_list, - struct compat_robust_list_head __user *, head, - compat_size_t, len) +COMPAT_SYSCALL_DEFINE2(set_robust_list, struct compat_robust_list_head __user *, head, + compat_size_t, len) { if (unlikely(len != sizeof(*head))) return -EINVAL; - current->compat_robust_list = head; - + current->futex.compat_robust_list = head; return 0; } @@ -523,4 +524,3 @@ SYSCALL_DEFINE6(futex_time32, u32 __user *, uaddr, int, op, u32, val, return do_futex(uaddr, op, val, tp, uaddr2, (unsigned long)utime, val3); } #endif /* CONFIG_COMPAT_32BIT_TIME */ - diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index ceed9d879059..d4483d15d30a 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -149,13 +149,36 @@ void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q) wake_q_add_safe(wake_q, p); } +/* + * If requested, clear the robust list pending op and unlock the futex + */ +static bool futex_robust_unlock(u32 __user *uaddr, unsigned int flags, void __user *pop) +{ + if (!(flags & FLAGS_ROBUST_UNLOCK)) + return true; + + /* First unlock the futex, which requires release semantics. */ + scoped_user_write_access(uaddr, efault) + unsafe_atomic_store_release_user(0, uaddr, efault); + + /* + * Clear the pending list op now. If that fails, then the task is in + * deeper trouble as the robust list head is usually part of the TLS. + * The chance of survival is close to zero. + */ + return futex_robust_list_clear_pending(pop, flags); + +efault: + return false; +} + /* * Wake up waiters matching bitset queued on this futex (uaddr). */ -int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset) +int futex_wake(u32 __user *uaddr, unsigned int flags, void __user *pop, int nr_wake, u32 bitset) { - struct futex_q *this, *next; union futex_key key = FUTEX_KEY_INIT; + struct futex_q *this, *next; DEFINE_WAKE_Q(wake_q); int ret; @@ -166,10 +189,14 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset) if (unlikely(ret != 0)) return ret; + if (!futex_robust_unlock(uaddr, flags, pop)) + return -EFAULT; + if ((flags & FLAGS_STRICT) && !nr_wake) return 0; - CLASS(hb, hb)(&key); + CLASS(hbr, hbr)(&key); + auto hb = hbr.hb; /* Make sure we really have tasks to wakeup */ if (!futex_hb_waiters_pending(hb)) @@ -266,8 +293,10 @@ retry: retry_private: if (1) { - CLASS(hb, hb1)(&key1); - CLASS(hb, hb2)(&key2); + CLASS(hbr, hbr1)(&key1); + CLASS(hbr, hbr2)(&key2); + auto hb1 = hbr1.hb; + auto hb2 = hbr2.hb; double_lock_hb(hb1, hb2); op_ret = futex_atomic_op_inuser(op, uaddr2); @@ -409,7 +438,7 @@ int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *woken) * Make sure to have a reference on the private_hash such that we * don't block on rehash after changing the task state below. */ - guard(private_hash)(); + guard(private_hash)(current->mm); /* * Enqueuing multiple futexes is tricky, because we need to enqueue @@ -446,7 +475,8 @@ retry: u32 val = vs[i].w.val; if (1) { - CLASS(hb, hb)(&q->key); + CLASS(hbr, hbr)(&q->key); + auto hb = hbr.hb; futex_q_lock(q, hb); ret = futex_get_value_locked(&uval, uaddr); @@ -621,7 +651,8 @@ retry: retry_private: if (1) { - CLASS(hb, hb)(&q->key); + CLASS(hbr, hbr)(&q->key); + auto hb = hbr.hb; futex_q_lock(q, hb); diff --git a/kernel/irq/internals.h b/kernel/irq/internals.h index f9c099d45a64..0ce21dd45404 100644 --- a/kernel/irq/internals.h +++ b/kernel/irq/internals.h @@ -194,7 +194,7 @@ void __irq_put_desc_unlock(struct irq_desc *desc, unsigned long flags, bool bus) __DEFINE_CLASS_IS_CONDITIONAL(irqdesc_lock, true); __DEFINE_UNLOCK_GUARD(irqdesc_lock, struct irq_desc, - __irq_put_desc_unlock(_T->lock, _T->flags, _T->bus), + if (_T->lock) __irq_put_desc_unlock(_T->lock, _T->flags, _T->bus), unsigned long flags; bool bus); static inline class_irqdesc_lock_t class_irqdesc_lock_constructor(unsigned int irq, bool bus, diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 09534628dc01..43b7f7e281a0 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -1023,6 +1023,9 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne wake_q_add(&wake_q, next); } + if (trace_contended_release_enabled() && waiter) + trace_call__contended_release(lock); + if (owner & MUTEX_FLAG_HANDOFF) __mutex_handoff(lock, next); @@ -1220,6 +1223,7 @@ EXPORT_SYMBOL(ww_mutex_lock_interruptible); EXPORT_TRACEPOINT_SYMBOL_GPL(contention_begin); EXPORT_TRACEPOINT_SYMBOL_GPL(contention_end); +EXPORT_TRACEPOINT_SYMBOL_GPL(contended_release); /** * atomic_dec_and_mutex_lock - return holding mutex if we dec to 0 diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index ef234469baac..f7e152c40d6d 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -263,6 +263,9 @@ void percpu_up_write(struct percpu_rw_semaphore *sem) { rwsem_release(&sem->dep_map, _RET_IP_); + if (trace_contended_release_enabled() && wq_has_sleeper(&sem->waiters)) + trace_call__contended_release(sem); + /* * Signal the writer is done, no fast path yet. * @@ -288,3 +291,29 @@ void percpu_up_write(struct percpu_rw_semaphore *sem) rcu_sync_exit(&sem->rss); } EXPORT_SYMBOL_GPL(percpu_up_write); + +void __percpu_up_read(struct percpu_rw_semaphore *sem) +{ + lockdep_assert_preemption_disabled(); + /* + * After percpu_up_write() completes, rcu_sync_is_idle() can still + * return false during the grace period, forcing readers into this + * slowpath. Only trace when a writer is actually waiting for + * readers to drain. + */ + if (trace_contended_release_enabled() && rcuwait_active(&sem->writer)) + trace_call__contended_release(sem); + /* + * slowpath; reader will only ever wake a single blocked + * writer. + */ + smp_mb(); /* B matches C */ + /* + * In other words, if they see our decrement (presumably to + * aggregate zero, as that is the only time it matters) they + * will also see our critical section. + */ + this_cpu_dec(*sem->read_count); + rcuwait_wake_up(&sem->writer); +} +EXPORT_SYMBOL_GPL(__percpu_up_read); diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index daeeeef973e2..4728631ae719 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -484,6 +484,7 @@ static __always_inline bool __waiter_less(struct rb_node *a, const struct rb_nod static __always_inline void rt_mutex_enqueue(struct rt_mutex_base *lock, struct rt_mutex_waiter *waiter) + __must_hold(&lock->wait_lock) { lockdep_assert_held(&lock->wait_lock); @@ -492,6 +493,7 @@ rt_mutex_enqueue(struct rt_mutex_base *lock, struct rt_mutex_waiter *waiter) static __always_inline void rt_mutex_dequeue(struct rt_mutex_base *lock, struct rt_mutex_waiter *waiter) + __must_hold(&lock->wait_lock) { lockdep_assert_held(&lock->wait_lock); @@ -1092,6 +1094,7 @@ static int __sched rt_mutex_adjust_prio_chain(struct task_struct *task, static int __sched try_to_take_rt_mutex(struct rt_mutex_base *lock, struct task_struct *task, struct rt_mutex_waiter *waiter) + __must_hold(&lock->wait_lock) { lockdep_assert_held(&lock->wait_lock); @@ -1319,6 +1322,7 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock, */ static void __sched mark_wakeup_next_waiter(struct rt_wake_q_head *wqh, struct rt_mutex_base *lock) + __must_hold(&lock->wait_lock) { struct rt_mutex_waiter *waiter; @@ -1466,6 +1470,7 @@ static void __sched rt_mutex_slowunlock(struct rt_mutex_base *lock) raw_spin_lock_irqsave(&lock->wait_lock, flags); } + trace_contended_release(lock); /* * The wakeup next waiter path does not suffer from the above * race. See the comments there. diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c index 514fce7a4e0a..5d48d64725b1 100644 --- a/kernel/locking/rtmutex_api.c +++ b/kernel/locking/rtmutex_api.c @@ -41,6 +41,7 @@ static __always_inline int __rt_mutex_lock_common(struct rt_mutex *lock, unsigned int state, struct lockdep_map *nest_lock, unsigned int subclass) + __cond_acquires(0, lock) { int ret; @@ -67,13 +68,27 @@ EXPORT_SYMBOL(rt_mutex_base_init); */ void __sched rt_mutex_lock_nested(struct rt_mutex *lock, unsigned int subclass) { - __rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, NULL, subclass); + if (__rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, NULL, subclass) == 0) + return; + /* + * The code below is never reached because __rt_mutex_lock_common() only + * returns an error code if interrupted by a signal or upon a timeout. + */ + WARN_ON_ONCE(true); + __acquire(lock); } EXPORT_SYMBOL_GPL(rt_mutex_lock_nested); void __sched _rt_mutex_lock_nest_lock(struct rt_mutex *lock, struct lockdep_map *nest_lock) { - __rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, nest_lock, 0); + if (__rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, nest_lock, 0) == 0) + return; + /* + * The code below is never reached because __rt_mutex_lock_common() only + * returns an error code if interrupted by a signal or upon a timeout. + */ + WARN_ON_ONCE(true); + __acquire(lock); } EXPORT_SYMBOL_GPL(_rt_mutex_lock_nest_lock); @@ -86,7 +101,14 @@ EXPORT_SYMBOL_GPL(_rt_mutex_lock_nest_lock); */ void __sched rt_mutex_lock(struct rt_mutex *lock) { - __rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, NULL, 0); + if (__rt_mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, NULL, 0) == 0) + return; + /* + * The code below is never reached because __rt_mutex_lock_common() only + * returns an error code if interrupted by a signal or upon a timeout. + */ + WARN_ON_ONCE(true); + __acquire(lock); } EXPORT_SYMBOL_GPL(rt_mutex_lock); #endif @@ -157,6 +179,7 @@ void __sched rt_mutex_unlock(struct rt_mutex *lock) { mutex_release(&lock->dep_map, _RET_IP_); __rt_mutex_unlock(&lock->rtmutex); + __release(lock); } EXPORT_SYMBOL_GPL(rt_mutex_unlock); @@ -182,6 +205,7 @@ int __sched __rt_mutex_futex_trylock(struct rt_mutex_base *lock) */ bool __sched __rt_mutex_futex_unlock(struct rt_mutex_base *lock, struct rt_wake_q_head *wqh) + __must_hold(&lock->wait_lock) { lockdep_assert_held(&lock->wait_lock); @@ -312,6 +336,7 @@ int __sched __rt_mutex_start_proxy_lock(struct rt_mutex_base *lock, struct rt_mutex_waiter *waiter, struct task_struct *task, struct wake_q_head *wake_q) + __must_hold(&lock->wait_lock) { int ret; diff --git a/kernel/locking/rwbase_rt.c b/kernel/locking/rwbase_rt.c index 82e078c0665a..2835c9ef9b3f 100644 --- a/kernel/locking/rwbase_rt.c +++ b/kernel/locking/rwbase_rt.c @@ -174,6 +174,8 @@ static void __sched __rwbase_read_unlock(struct rwbase_rt *rwb, static __always_inline void rwbase_read_unlock(struct rwbase_rt *rwb, unsigned int state) { + if (trace_contended_release_enabled() && rt_mutex_owner(&rwb->rtmutex)) + trace_call__contended_release(rwb); /* * rwb->readers can only hit 0 when a writer is waiting for the * active readers to leave the critical section. @@ -205,6 +207,8 @@ static inline void rwbase_write_unlock(struct rwbase_rt *rwb) unsigned long flags; raw_spin_lock_irqsave(&rtm->wait_lock, flags); + if (trace_contended_release_enabled() && rt_mutex_has_waiters(rtm)) + trace_call__contended_release(rwb); __rwbase_write_unlock(rwb, WRITER_BIAS, flags); } @@ -214,6 +218,8 @@ static inline void rwbase_write_downgrade(struct rwbase_rt *rwb) unsigned long flags; raw_spin_lock_irqsave(&rtm->wait_lock, flags); + if (trace_contended_release_enabled() && rt_mutex_has_waiters(rtm)) + trace_call__contended_release(rwb); /* Release it and account current as reader */ __rwbase_write_unlock(rwb, WRITER_BIAS - 1, flags); } diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index bf647097369c..b9c180ac1eee 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -1387,6 +1387,8 @@ static inline void __up_read(struct rw_semaphore *sem) rwsem_clear_reader_owned(sem); tmp = atomic_long_add_return_release(-RWSEM_READER_BIAS, &sem->count); DEBUG_RWSEMS_WARN_ON(tmp < 0, sem); + if (trace_contended_release_enabled() && (tmp & RWSEM_FLAG_WAITERS)) + trace_call__contended_release(sem); if (unlikely((tmp & (RWSEM_LOCK_MASK|RWSEM_FLAG_WAITERS)) == RWSEM_FLAG_WAITERS)) { clear_nonspinnable(sem); @@ -1413,8 +1415,10 @@ static inline void __up_write(struct rw_semaphore *sem) preempt_disable(); rwsem_clear_owner(sem); tmp = atomic_long_fetch_add_release(-RWSEM_WRITER_LOCKED, &sem->count); - if (unlikely(tmp & RWSEM_FLAG_WAITERS)) + if (unlikely(tmp & RWSEM_FLAG_WAITERS)) { + trace_contended_release(sem); rwsem_wake(sem); + } preempt_enable(); } @@ -1437,8 +1441,10 @@ static inline void __downgrade_write(struct rw_semaphore *sem) tmp = atomic_long_fetch_add_release( -RWSEM_WRITER_LOCKED+RWSEM_READER_BIAS, &sem->count); rwsem_set_reader_owned(sem); - if (tmp & RWSEM_FLAG_WAITERS) + if (tmp & RWSEM_FLAG_WAITERS) { + trace_contended_release(sem); rwsem_downgrade_wake(sem); + } preempt_enable(); } diff --git a/kernel/locking/semaphore.c b/kernel/locking/semaphore.c index 74d41433ba13..233730c25933 100644 --- a/kernel/locking/semaphore.c +++ b/kernel/locking/semaphore.c @@ -230,6 +230,10 @@ void __sched up(struct semaphore *sem) sem->count++; else __up(sem, &wake_q); + + if (trace_contended_release_enabled() && !wake_q_empty(&wake_q)) + trace_call__contended_release(sem); + raw_spin_unlock_irqrestore(&sem->lock, flags); if (!wake_q_empty(&wake_q)) wake_up_q(&wake_q); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index afb9e7db8f78..03a43d3d2616 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -969,14 +969,11 @@ static int rcu_watching_snap_recheck(struct rcu_data *rdp) if (rcu_cpu_stall_cputime && rdp->snap_record.gp_seq != rdp->gp_seq) { int cpu = rdp->cpu; struct rcu_snap_record *rsrp; - struct kernel_cpustat *kcsp; - - kcsp = &kcpustat_cpu(cpu); rsrp = &rdp->snap_record; - rsrp->cputime_irq = kcpustat_field(kcsp, CPUTIME_IRQ, cpu); - rsrp->cputime_softirq = kcpustat_field(kcsp, CPUTIME_SOFTIRQ, cpu); - rsrp->cputime_system = kcpustat_field(kcsp, CPUTIME_SYSTEM, cpu); + rsrp->cputime_irq = kcpustat_field(CPUTIME_IRQ, cpu); + rsrp->cputime_softirq = kcpustat_field(CPUTIME_SOFTIRQ, cpu); + rsrp->cputime_system = kcpustat_field(CPUTIME_SYSTEM, cpu); rsrp->nr_hardirqs = kstat_cpu_irqs_sum(cpu) + arch_irq_stat_cpu(cpu); rsrp->nr_softirqs = kstat_cpu_softirqs_sum(cpu); rsrp->nr_csw = nr_context_switches_cpu(cpu); diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index b67532cb8770..cf7ae51cba40 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -479,7 +479,6 @@ static void print_cpu_stat_info(int cpu) { struct rcu_snap_record rsr, *rsrp; struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); - struct kernel_cpustat *kcsp = &kcpustat_cpu(cpu); if (!rcu_cpu_stall_cputime) return; @@ -488,9 +487,9 @@ static void print_cpu_stat_info(int cpu) if (rsrp->gp_seq != rdp->gp_seq) return; - rsr.cputime_irq = kcpustat_field(kcsp, CPUTIME_IRQ, cpu); - rsr.cputime_softirq = kcpustat_field(kcsp, CPUTIME_SOFTIRQ, cpu); - rsr.cputime_system = kcpustat_field(kcsp, CPUTIME_SYSTEM, cpu); + rsr.cputime_irq = kcpustat_field(CPUTIME_IRQ, cpu); + rsr.cputime_softirq = kcpustat_field(CPUTIME_SOFTIRQ, cpu); + rsr.cputime_system = kcpustat_field(CPUTIME_SYSTEM, cpu); pr_err("\t hardirqs softirqs csw/system\n"); pr_err("\t number: %8lld %10d %12lld\n", diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ff4d4121c11a..c2307f71a2c3 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5619,7 +5619,11 @@ void sched_exec(void) } DEFINE_PER_CPU(struct kernel_stat, kstat); -DEFINE_PER_CPU(struct kernel_cpustat, kernel_cpustat); +DEFINE_PER_CPU(struct kernel_cpustat, kernel_cpustat) = { +#ifdef CONFIG_NO_HZ_COMMON + .idle_sleeptime_seq = SEQCNT_ZERO(kernel_cpustat.idle_sleeptime_seq) +#endif +}; EXPORT_PER_CPU_SYMBOL(kstat); EXPORT_PER_CPU_SYMBOL(kernel_cpustat); diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c index fe49ad5d4ba1..a8deec35538a 100644 --- a/kernel/sched/cputime.c +++ b/kernel/sched/cputime.c @@ -2,6 +2,7 @@ /* * Simple CPU accounting cgroup controller */ +#include #include #include #include "sched.h" @@ -50,7 +51,8 @@ static void irqtime_account_delta(struct irqtime *irqtime, u64 delta, u64_stats_update_begin(&irqtime->sync); cpustat[idx] += delta; irqtime->total += delta; - irqtime->tick_delta += delta; + if (!kcpustat_idle_dyntick()) + irqtime->tick_delta += delta; u64_stats_update_end(&irqtime->sync); } @@ -430,16 +432,219 @@ static void irqtime_account_process_tick(struct task_struct *p, int user_tick, } } -static void irqtime_account_idle_ticks(int ticks) -{ - irqtime_account_process_tick(current, 0, ticks); -} #else /* !CONFIG_IRQ_TIME_ACCOUNTING: */ -static inline void irqtime_account_idle_ticks(int ticks) { } static inline void irqtime_account_process_tick(struct task_struct *p, int user_tick, int nr_ticks) { } #endif /* !CONFIG_IRQ_TIME_ACCOUNTING */ +#ifdef CONFIG_NO_HZ_COMMON +static void kcpustat_idle_stop(struct kernel_cpustat *kc, u64 now) +{ + u64 *cpustat = kc->cpustat; + u64 delta, steal, steal_delta; + int iowait; + + if (!kc->idle_elapse) + return; + + iowait = nr_iowait_cpu(smp_processor_id()) > 0; + delta = now - kc->idle_entrytime; + steal = steal_account_process_time(delta); + + /* + * Record the idle time after substracting the steal time from + * previous update sequence. Don't substract the steal time from + * the current update sequence to avoid readers moving backward. + */ + write_seqcount_begin(&kc->idle_sleeptime_seq); + steal_delta = min_t(u64, kc->idle_stealtime[iowait], delta); + delta -= steal_delta; + kc->idle_stealtime[iowait] -= steal_delta; + + if (iowait) + cpustat[CPUTIME_IOWAIT] += delta; + else + cpustat[CPUTIME_IDLE] += delta; + + kc->idle_stealtime[iowait] += steal; + kc->idle_entrytime = now; + kc->idle_elapse = false; + write_seqcount_end(&kc->idle_sleeptime_seq); +} + +static void kcpustat_idle_start(struct kernel_cpustat *kc, u64 now) +{ + /* Irqtime accounting might have been enabled in the middle of the IRQ */ + if (kc->idle_elapse) + return; + + write_seqcount_begin(&kc->idle_sleeptime_seq); + kc->idle_entrytime = now; + kc->idle_elapse = true; + write_seqcount_end(&kc->idle_sleeptime_seq); +} + +void kcpustat_dyntick_stop(u64 now) +{ + struct kernel_cpustat *kc = kcpustat_this_cpu; + + if (!vtime_generic_enabled_this_cpu()) { + WARN_ON_ONCE(!kc->idle_dyntick); + kcpustat_idle_stop(kc, now); + kc->idle_dyntick = false; + vtime_dyntick_stop(); + } +} + +void kcpustat_dyntick_start(u64 now) +{ + struct kernel_cpustat *kc = kcpustat_this_cpu; + + if (!vtime_generic_enabled_this_cpu()) { + vtime_dyntick_start(); + kc->idle_dyntick = true; + kcpustat_idle_start(kc, now); + } +} + +void kcpustat_irq_enter(u64 now) +{ + struct kernel_cpustat *kc = kcpustat_this_cpu; + + if (!vtime_generic_enabled_this_cpu() && + (irqtime_enabled() || vtime_accounting_enabled_this_cpu())) + kcpustat_idle_stop(kc, now); +} + +void kcpustat_irq_exit(u64 now) +{ + struct kernel_cpustat *kc = kcpustat_this_cpu; + + /* + * Generic vtime already does its own idle accounting. + * But irqtime accounting or arch vtime which also accounts IRQs + * need to pause nohz accounting. Resume nohz accounting as long + * as the irqtime config is enabled to handle case where irqtime + * accounting got runtime disabled in the middle of an IRQ. + */ + if (!vtime_generic_enabled_this_cpu() && + (IS_ENABLED(CONFIG_IRQ_TIME_ACCOUNTING) || vtime_accounting_enabled_this_cpu())) + kcpustat_idle_start(kc, now); +} + +static u64 kcpustat_field_dyntick(int cpu, enum cpu_usage_stat idx, + bool compute_delta, u64 now) +{ + struct kernel_cpustat *kc = &kcpustat_cpu(cpu); + int iowait = idx == CPUTIME_IOWAIT; + u64 *cpustat = kc->cpustat; + unsigned int seq; + u64 idle; + + do { + seq = read_seqcount_begin(&kc->idle_sleeptime_seq); + + idle = cpustat[idx]; + + if (kc->idle_elapse && compute_delta && now > kc->idle_entrytime) { + u64 delta = now - kc->idle_entrytime; + + delta -= min_t(u64, kc->idle_stealtime[iowait], delta); + idle += delta; + } + } while (read_seqcount_retry(&kc->idle_sleeptime_seq, seq)); + + return idle; +} + +u64 kcpustat_field_idle(int cpu) +{ + return kcpustat_field_dyntick(cpu, CPUTIME_IDLE, + !nr_iowait_cpu(cpu), ktime_get()); +} +EXPORT_SYMBOL_GPL(kcpustat_field_idle); + +u64 kcpustat_field_iowait(int cpu) +{ + return kcpustat_field_dyntick(cpu, CPUTIME_IOWAIT, + nr_iowait_cpu(cpu), ktime_get()); +} +EXPORT_SYMBOL_GPL(kcpustat_field_iowait); +#else +static u64 kcpustat_field_dyntick(int cpu, enum cpu_usage_stat idx, + bool compute_delta, ktime_t now) +{ + return kcpustat_cpu(cpu).cpustat[idx]; +} +#endif /* CONFIG_NO_HZ_COMMON */ + +static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx, + bool compute_delta, u64 *last_update_time) +{ + ktime_t now = ktime_get(); + u64 res; + + if (vtime_generic_enabled_cpu(cpu)) + res = kcpustat_field(idx, cpu); + else + res = kcpustat_field_dyntick(cpu, idx, compute_delta, now); + + do_div(res, NSEC_PER_USEC); + + if (last_update_time) + *last_update_time = ktime_to_us(now); + + return res; +} + +/** + * get_cpu_idle_time_us - get the total idle time of a CPU + * @cpu: CPU number to query + * @last_update_time: variable to store update time in. Do not update + * counters if NULL. + * + * Return the cumulative idle time (since boot) for a given + * CPU, in microseconds. Note that this is partially broken due to + * the counter of iowait tasks that can be remotely updated without + * any synchronization. Therefore it is possible to observe backward + * values within two consecutive reads. + * + * This time is measured via accounting rather than sampling, + * and is as accurate as ktime_get() is. + * + * Return: total idle time of the @cpu + */ +u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) +{ + return get_cpu_sleep_time_us(cpu, CPUTIME_IDLE, + !nr_iowait_cpu(cpu), last_update_time); +} +EXPORT_SYMBOL_GPL(get_cpu_idle_time_us); + +/** + * get_cpu_iowait_time_us - get the total iowait time of a CPU + * @cpu: CPU number to query + * @last_update_time: variable to store update time in. Do not update + * counters if NULL. + * + * Return the cumulative iowait time (since boot) for a given + * CPU, in microseconds. Note this is partially broken due to + * the counter of iowait tasks that can be remotely updated without + * any synchronization. Therefore it is possible to observe backward + * values within two consecutive reads. + * + * This time is measured via accounting rather than sampling, + * and is as accurate as ktime_get() is. + * + * Return: total iowait time of @cpu + */ +u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) +{ + return get_cpu_sleep_time_us(cpu, CPUTIME_IOWAIT, + nr_iowait_cpu(cpu), last_update_time); +} +EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us); + /* * Use precise platform statistics if available: */ @@ -453,11 +658,15 @@ void vtime_account_irq(struct task_struct *tsk, unsigned int offset) vtime_account_hardirq(tsk); } else if (pc & SOFTIRQ_OFFSET) { vtime_account_softirq(tsk); - } else if (!IS_ENABLED(CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE) && - is_idle_task(tsk)) { - vtime_account_idle(tsk); + } else if (!kcpustat_idle_dyntick()) { + if (!IS_ENABLED(CONFIG_HAVE_VIRT_CPU_ACCOUNTING_IDLE) && + is_idle_task(tsk)) { + vtime_account_idle(tsk); + } else { + vtime_account_kernel(tsk); + } } else { - vtime_account_kernel(tsk); + vtime_reset(); } } @@ -500,6 +709,9 @@ void account_process_tick(struct task_struct *p, int user_tick) if (vtime_accounting_enabled_this_cpu()) return; + if (kcpustat_idle_dyntick()) + return; + if (irqtime_enabled()) { irqtime_account_process_tick(p, user_tick, 1); return; @@ -521,29 +733,6 @@ void account_process_tick(struct task_struct *p, int user_tick) account_idle_time(cputime); } -/* - * Account multiple ticks of idle time. - * @ticks: number of stolen ticks - */ -void account_idle_ticks(unsigned long ticks) -{ - u64 cputime, steal; - - if (irqtime_enabled()) { - irqtime_account_idle_ticks(ticks); - return; - } - - cputime = ticks * TICK_NSEC; - steal = steal_account_process_time(ULONG_MAX); - - if (steal >= cputime) - return; - - cputime -= steal; - account_idle_time(cputime); -} - /* * Adjust tick based cputime random precision against scheduler runtime * accounting. @@ -792,9 +981,9 @@ void vtime_guest_exit(struct task_struct *tsk) } EXPORT_SYMBOL_GPL(vtime_guest_exit); -void vtime_account_idle(struct task_struct *tsk) +static void __vtime_account_idle(struct vtime *vtime) { - account_idle_time(get_vtime_delta(&tsk->vtime)); + account_idle_time(get_vtime_delta(vtime)); } void vtime_task_switch_generic(struct task_struct *prev) @@ -803,7 +992,7 @@ void vtime_task_switch_generic(struct task_struct *prev) write_seqcount_begin(&vtime->seqcount); if (vtime->state == VTIME_IDLE) - vtime_account_idle(prev); + __vtime_account_idle(vtime); else __vtime_account_kernel(prev, vtime); vtime->state = VTIME_INACTIVE; @@ -945,6 +1134,7 @@ static int kcpustat_field_vtime(u64 *cpustat, int cpu, u64 *val) { struct vtime *vtime = &tsk->vtime; + struct rq *rq = cpu_rq(cpu); unsigned int seq; do { @@ -986,6 +1176,14 @@ static int kcpustat_field_vtime(u64 *cpustat, if (state == VTIME_GUEST && task_nice(tsk) > 0) *val += vtime->gtime + vtime_delta(vtime); break; + case CPUTIME_IDLE: + if (state == VTIME_IDLE && !atomic_read(&rq->nr_iowait)) + *val += vtime_delta(vtime); + break; + case CPUTIME_IOWAIT: + if (state == VTIME_IDLE && atomic_read(&rq->nr_iowait) > 0) + *val += vtime_delta(vtime); + break; default: break; } @@ -994,16 +1192,15 @@ static int kcpustat_field_vtime(u64 *cpustat, return 0; } -u64 kcpustat_field(struct kernel_cpustat *kcpustat, - enum cpu_usage_stat usage, int cpu) +u64 kcpustat_field(enum cpu_usage_stat usage, int cpu) { - u64 *cpustat = kcpustat->cpustat; + u64 *cpustat = kcpustat_cpu(cpu).cpustat; u64 val = cpustat[usage]; struct rq *rq; int err; - if (!vtime_accounting_enabled_cpu(cpu)) - return val; + if (!vtime_generic_enabled_cpu(cpu)) + return kcpustat_field_default(usage, cpu); rq = cpu_rq(cpu); @@ -1049,8 +1246,8 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst, *dst = *src; cpustat = dst->cpustat; - /* Task is sleeping, dead or idle, nothing to add */ - if (state < VTIME_SYS) + /* Task is sleeping or dead, nothing to add */ + if (state < VTIME_IDLE) continue; delta = vtime_delta(vtime); @@ -1059,15 +1256,17 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst, * Task runs either in user (including guest) or kernel space, * add pending nohz time to the right place. */ - if (state == VTIME_SYS) { + switch (state) { + case VTIME_SYS: cpustat[CPUTIME_SYSTEM] += vtime->stime + delta; - } else if (state == VTIME_USER) { + break; + case VTIME_USER: if (task_nice(tsk) > 0) cpustat[CPUTIME_NICE] += vtime->utime + delta; else cpustat[CPUTIME_USER] += vtime->utime + delta; - } else { - WARN_ON_ONCE(state != VTIME_GUEST); + break; + case VTIME_GUEST: if (task_nice(tsk) > 0) { cpustat[CPUTIME_GUEST_NICE] += vtime->gtime + delta; cpustat[CPUTIME_NICE] += vtime->gtime + delta; @@ -1075,6 +1274,15 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst, cpustat[CPUTIME_GUEST] += vtime->gtime + delta; cpustat[CPUTIME_USER] += vtime->gtime + delta; } + break; + case VTIME_IDLE: + if (atomic_read(&cpu_rq(cpu)->nr_iowait) > 0) + cpustat[CPUTIME_IOWAIT] += delta; + else + cpustat[CPUTIME_IDLE] += delta; + break; + default: + WARN_ON_ONCE(1); } } while (read_seqcount_retry(&vtime->seqcount, seq)); @@ -1087,8 +1295,8 @@ void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu) struct rq *rq; int err; - if (!vtime_accounting_enabled_cpu(cpu)) { - *dst = *src; + if (!vtime_generic_enabled_cpu(cpu)) { + kcpustat_cpu_fetch_default(dst, cpu); return; } @@ -1101,7 +1309,7 @@ void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu) curr = rcu_dereference(rq->curr); if (WARN_ON_ONCE(!curr)) { rcu_read_unlock(); - *dst = *src; + kcpustat_cpu_fetch_default(dst, cpu); return; } diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index a83be0c834dd..aa7e3dc59856 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -280,6 +280,14 @@ static void do_idle(void) int cpu = smp_processor_id(); bool got_tick = false; + if (cpu_is_offline(cpu)) { + local_irq_disable(); + /* All per-CPU kernel threads should be done by now. */ + WARN_ON_ONCE(need_resched()); + cpuhp_report_idle_dead(); + arch_cpu_idle_dead(); + } + /* * Check if we need to update blocked load */ @@ -331,11 +339,6 @@ static void do_idle(void) */ local_irq_disable(); - if (cpu_is_offline(cpu)) { - cpuhp_report_idle_dead(); - arch_cpu_idle_dead(); - } - arch_cpu_idle_enter(); rcu_nocb_flush_deferred_wakeup(); diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c index 226a6329f3e9..cb957b8f1946 100644 --- a/kernel/sched/membarrier.c +++ b/kernel/sched/membarrier.c @@ -164,8 +164,26 @@ | MEMBARRIER_PRIVATE_EXPEDITED_RSEQ_BITMASK \ | MEMBARRIER_CMD_GET_REGISTRATIONS) +/* + * Scoped guard for memory barriers on entry and exit. + * Matches memory barriers before & after rq->curr modification in scheduler. + */ +DEFINE_LOCK_GUARD_0(mb, smp_mb(), smp_mb()) static DEFINE_MUTEX(membarrier_ipi_mutex); +static DEFINE_PER_CPU(struct mutex, membarrier_cpu_mutexes); + #define SERIALIZE_IPI() guard(mutex)(&membarrier_ipi_mutex) +#define SERIALIZE_IPI_CPU(cpu_id) guard(mutex)(&per_cpu(membarrier_cpu_mutexes, cpu_id)) + +static int __init membarrier_init(void) +{ + int i; + + for_each_possible_cpu(i) + mutex_init(&per_cpu(membarrier_cpu_mutexes, i)); + return 0; +} +core_initcall(membarrier_init); static void ipi_mb(void *info) { @@ -258,23 +276,19 @@ void membarrier_update_current_mm(struct mm_struct *next_mm) static int membarrier_global_expedited(void) { + cpumask_var_t __free(free_cpumask_var) tmpmask = CPUMASK_VAR_NULL; int cpu; - cpumask_var_t tmpmask; if (num_online_cpus() == 1) return 0; - /* - * Matches memory barriers after rq->curr modification in - * scheduler. - */ - smp_mb(); /* system call entry is not a mb. */ - if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL)) return -ENOMEM; + guard(mb)(); SERIALIZE_IPI(); - cpus_read_lock(); + guard(cpus_read_lock)(); + rcu_read_lock(); for_each_online_cpu(cpu) { struct task_struct *p; @@ -310,21 +324,11 @@ static int membarrier_global_expedited(void) smp_call_function_many(tmpmask, ipi_mb, NULL, 1); preempt_enable(); - free_cpumask_var(tmpmask); - cpus_read_unlock(); - - /* - * Memory barrier on the caller thread _after_ we finished - * waiting for the last IPI. Matches memory barriers before - * rq->curr modification in scheduler. - */ - smp_mb(); /* exit from system call is not a mb */ return 0; } static int membarrier_private_expedited(int flags, int cpu_id) { - cpumask_var_t tmpmask; struct mm_struct *mm = current->mm; smp_call_func_t ipi_func = ipi_mb; @@ -361,30 +365,45 @@ static int membarrier_private_expedited(int flags, int cpu_id) * On RISC-V, this barrier pairing is also needed for the * SYNC_CORE command when switching between processes, cf. * the inline comments in membarrier_arch_switch_mm(). + * + * Memory barrier on the caller thread _after_ we finished + * waiting for the last IPI. Matches memory barriers before + * rq->curr modification in scheduler. */ - smp_mb(); /* system call entry is not a mb. */ - - if (cpu_id < 0 && !zalloc_cpumask_var(&tmpmask, GFP_KERNEL)) - return -ENOMEM; - - SERIALIZE_IPI(); - cpus_read_lock(); - + guard(mb)(); if (cpu_id >= 0) { + if (cpu_id >= nr_cpu_ids || !cpu_possible(cpu_id)) + return 0; + + SERIALIZE_IPI_CPU(cpu_id); + guard(cpus_read_lock)(); struct task_struct *p; - if (cpu_id >= nr_cpu_ids || !cpu_online(cpu_id)) - goto out; + if (!cpu_online(cpu_id)) + return 0; + rcu_read_lock(); p = rcu_dereference(cpu_rq(cpu_id)->curr); if (!p || p->mm != mm) { rcu_read_unlock(); - goto out; + return 0; } rcu_read_unlock(); + /* + * smp_call_function_single() will call ipi_func() if cpu_id + * is the calling CPU. + */ + smp_call_function_single(cpu_id, ipi_func, NULL, 1); } else { + cpumask_var_t __free(free_cpumask_var) tmpmask = CPUMASK_VAR_NULL; int cpu; + if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL)) + return -ENOMEM; + + SERIALIZE_IPI(); + guard(cpus_read_lock)(); + rcu_read_lock(); for_each_online_cpu(cpu) { struct task_struct *p; @@ -394,15 +413,6 @@ static int membarrier_private_expedited(int flags, int cpu_id) __cpumask_set_cpu(cpu, tmpmask); } rcu_read_unlock(); - } - - if (cpu_id >= 0) { - /* - * smp_call_function_single() will call ipi_func() if cpu_id - * is the calling CPU. - */ - smp_call_function_single(cpu_id, ipi_func, NULL, 1); - } else { /* * For regular membarrier, we can save a few cycles by * skipping the current cpu -- we're about to do smp_mb() @@ -429,18 +439,6 @@ static int membarrier_private_expedited(int flags, int cpu_id) } } -out: - if (cpu_id < 0) - free_cpumask_var(tmpmask); - cpus_read_unlock(); - - /* - * Memory barrier on the caller thread _after_ we finished - * waiting for the last IPI. Matches memory barriers before - * rq->curr modification in scheduler. - */ - smp_mb(); /* exit from system call is not a mb */ - return 0; } diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig index 02aac7c5aa76..d098ac39bde4 100644 --- a/kernel/time/Kconfig +++ b/kernel/time/Kconfig @@ -16,10 +16,6 @@ config ARCH_CLOCKSOURCE_INIT config ARCH_WANTS_CLOCKSOURCE_READ_INLINE bool -# Timekeeping vsyscall support -config GENERIC_TIME_VSYSCALL - bool - # The generic clock events infrastructure config GENERIC_CLOCKEVENTS def_bool !LEGACY_TIMER_TICK diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 660705356036..83670ac5a954 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -288,8 +288,6 @@ static void tick_sched_handle(struct tick_sched *ts, struct pt_regs *regs) if (IS_ENABLED(CONFIG_NO_HZ_COMMON) && tick_sched_flag_test(ts, TS_FLAG_STOPPED)) { touch_softlockup_watchdog_sched(); - if (is_idle_task(current)) - ts->idle_jiffies++; /* * In case the current tick fired too early past its expected * expiration, make sure we don't bypass the next clock reprogramming @@ -754,119 +752,6 @@ static void tick_nohz_update_jiffies(ktime_t now) touch_softlockup_watchdog_sched(); } -static void tick_nohz_stop_idle(struct tick_sched *ts, ktime_t now) -{ - ktime_t delta; - - if (WARN_ON_ONCE(!tick_sched_flag_test(ts, TS_FLAG_IDLE_ACTIVE))) - return; - - delta = ktime_sub(now, ts->idle_entrytime); - - write_seqcount_begin(&ts->idle_sleeptime_seq); - if (nr_iowait_cpu(smp_processor_id()) > 0) - ts->iowait_sleeptime = ktime_add(ts->iowait_sleeptime, delta); - else - ts->idle_sleeptime = ktime_add(ts->idle_sleeptime, delta); - - ts->idle_entrytime = now; - tick_sched_flag_clear(ts, TS_FLAG_IDLE_ACTIVE); - write_seqcount_end(&ts->idle_sleeptime_seq); - - sched_clock_idle_wakeup_event(); -} - -static void tick_nohz_start_idle(struct tick_sched *ts) -{ - write_seqcount_begin(&ts->idle_sleeptime_seq); - ts->idle_entrytime = ktime_get(); - tick_sched_flag_set(ts, TS_FLAG_IDLE_ACTIVE); - write_seqcount_end(&ts->idle_sleeptime_seq); - - sched_clock_idle_sleep_event(); -} - -static u64 get_cpu_sleep_time_us(struct tick_sched *ts, ktime_t *sleeptime, - bool compute_delta, u64 *last_update_time) -{ - ktime_t now, idle; - unsigned int seq; - - if (!tick_nohz_active) - return -1; - - now = ktime_get(); - if (last_update_time) - *last_update_time = ktime_to_us(now); - - do { - seq = read_seqcount_begin(&ts->idle_sleeptime_seq); - - if (tick_sched_flag_test(ts, TS_FLAG_IDLE_ACTIVE) && compute_delta) { - ktime_t delta = ktime_sub(now, ts->idle_entrytime); - - idle = ktime_add(*sleeptime, delta); - } else { - idle = *sleeptime; - } - } while (read_seqcount_retry(&ts->idle_sleeptime_seq, seq)); - - return ktime_to_us(idle); - -} - -/** - * get_cpu_idle_time_us - get the total idle time of a CPU - * @cpu: CPU number to query - * @last_update_time: variable to store update time in. Do not update - * counters if NULL. - * - * Return the cumulative idle time (since boot) for a given - * CPU, in microseconds. Note that this is partially broken due to - * the counter of iowait tasks that can be remotely updated without - * any synchronization. Therefore it is possible to observe backward - * values within two consecutive reads. - * - * This time is measured via accounting rather than sampling, - * and is as accurate as ktime_get() is. - * - * Return: -1 if NOHZ is not enabled, else total idle time of the @cpu - */ -u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) -{ - struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu); - - return get_cpu_sleep_time_us(ts, &ts->idle_sleeptime, - !nr_iowait_cpu(cpu), last_update_time); -} -EXPORT_SYMBOL_GPL(get_cpu_idle_time_us); - -/** - * get_cpu_iowait_time_us - get the total iowait time of a CPU - * @cpu: CPU number to query - * @last_update_time: variable to store update time in. Do not update - * counters if NULL. - * - * Return the cumulative iowait time (since boot) for a given - * CPU, in microseconds. Note this is partially broken due to - * the counter of iowait tasks that can be remotely updated without - * any synchronization. Therefore it is possible to observe backward - * values within two consecutive reads. - * - * This time is measured via accounting rather than sampling, - * and is as accurate as ktime_get() is. - * - * Return: -1 if NOHZ is not enabled, else total iowait time of @cpu - */ -u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) -{ - struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu); - - return get_cpu_sleep_time_us(ts, &ts->iowait_sleeptime, - nr_iowait_cpu(cpu), last_update_time); -} -EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us); - /* Simplified variant of hrtimer_forward_now() */ static ktime_t tick_forward_now(ktime_t expires, ktime_t now) { @@ -1276,7 +1161,7 @@ void tick_nohz_idle_stop_tick(void) ts->idle_expires = expires; if (!was_stopped && tick_sched_flag_test(ts, TS_FLAG_STOPPED)) { - ts->idle_jiffies = ts->last_jiffies; + kcpustat_dyntick_start(ts->idle_entrytime); nohz_balance_enter_idle(cpu); } } else { @@ -1289,6 +1174,20 @@ void tick_nohz_idle_retain_tick(void) tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched)); } +static void tick_nohz_clock_sleep(struct tick_sched *ts) +{ + tick_sched_flag_set(ts, TS_FLAG_IDLE_ACTIVE); + sched_clock_idle_sleep_event(); +} + +static void tick_nohz_clock_wakeup(struct tick_sched *ts) +{ + if (tick_sched_flag_test(ts, TS_FLAG_IDLE_ACTIVE)) { + tick_sched_flag_clear(ts, TS_FLAG_IDLE_ACTIVE); + sched_clock_idle_wakeup_event(); + } +} + /** * tick_nohz_idle_enter - prepare for entering idle on the current CPU * @@ -1303,11 +1202,10 @@ void tick_nohz_idle_enter(void) local_irq_disable(); ts = this_cpu_ptr(&tick_cpu_sched); - WARN_ON_ONCE(ts->timer_expires_base); - tick_sched_flag_set(ts, TS_FLAG_INIDLE); - tick_nohz_start_idle(ts); + ts->idle_entrytime = ktime_get(); + tick_nohz_clock_sleep(ts); local_irq_enable(); } @@ -1335,10 +1233,14 @@ void tick_nohz_irq_exit(void) { struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched); - if (tick_sched_flag_test(ts, TS_FLAG_INIDLE)) - tick_nohz_start_idle(ts); - else + if (tick_sched_flag_test(ts, TS_FLAG_INIDLE)) { + tick_nohz_clock_sleep(ts); + ts->idle_entrytime = ktime_get(); + if (tick_sched_flag_test(ts, TS_FLAG_STOPPED)) + kcpustat_irq_exit(ts->idle_entrytime); + } else { tick_nohz_full_update_tick(ts); + } } /** @@ -1433,36 +1335,20 @@ unsigned long tick_nohz_get_idle_calls_cpu(int cpu) } EXPORT_SYMBOL_GPL(tick_nohz_get_idle_calls_cpu); -static void tick_nohz_account_idle_time(struct tick_sched *ts, - ktime_t now) -{ - unsigned long ticks; - - ts->idle_exittime = now; - - if (vtime_accounting_enabled_this_cpu()) - return; - /* - * We stopped the tick in idle. update_process_times() would miss the - * time we slept, as it does only a 1 tick accounting. - * Enforce that this is accounted to idle ! - */ - ticks = jiffies - ts->idle_jiffies; - /* - * We might be one off. Do not randomly account a huge number of ticks! - */ - if (ticks && ticks < LONG_MAX) - account_idle_ticks(ticks); -} - void tick_nohz_idle_restart_tick(void) { struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched); if (tick_sched_flag_test(ts, TS_FLAG_STOPPED)) { - ktime_t now = ktime_get(); - tick_nohz_restart_sched_tick(ts, now); - tick_nohz_account_idle_time(ts, now); + /* + * Update entrytime here in case the tick restart is due to temporary + * polling on forced broadcast. The tick may be stopped again later within + * the same idle trip. The idle_entrytime was updated recently but make sure + * no tiny amount of idle time is accounted twice. + */ + ts->idle_entrytime = ktime_get(); + kcpustat_dyntick_stop(ts->idle_entrytime); + tick_nohz_restart_sched_tick(ts, ts->idle_entrytime); } } @@ -1472,8 +1358,6 @@ static void tick_nohz_idle_update_tick(struct tick_sched *ts, ktime_t now) __tick_nohz_full_update_tick(ts, now); else tick_nohz_restart_sched_tick(ts, now); - - tick_nohz_account_idle_time(ts, now); } /** @@ -1495,7 +1379,6 @@ static void tick_nohz_idle_update_tick(struct tick_sched *ts, ktime_t now) void tick_nohz_idle_exit(void) { struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched); - bool idle_active, tick_stopped; ktime_t now; local_irq_disable(); @@ -1504,17 +1387,13 @@ void tick_nohz_idle_exit(void) WARN_ON_ONCE(ts->timer_expires_base); tick_sched_flag_clear(ts, TS_FLAG_INIDLE); - idle_active = tick_sched_flag_test(ts, TS_FLAG_IDLE_ACTIVE); - tick_stopped = tick_sched_flag_test(ts, TS_FLAG_STOPPED); + tick_nohz_clock_wakeup(ts); - if (idle_active || tick_stopped) + if (tick_sched_flag_test(ts, TS_FLAG_STOPPED)) { now = ktime_get(); - - if (idle_active) - tick_nohz_stop_idle(ts, now); - - if (tick_stopped) + kcpustat_dyntick_stop(now); tick_nohz_idle_update_tick(ts, now); + } local_irq_enable(); } @@ -1569,11 +1448,14 @@ static inline void tick_nohz_irq_enter(void) struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched); ktime_t now; - if (!tick_sched_flag_test(ts, TS_FLAG_STOPPED | TS_FLAG_IDLE_ACTIVE)) + tick_nohz_clock_wakeup(ts); + + if (!tick_sched_flag_test(ts, TS_FLAG_STOPPED)) return; + now = ktime_get(); - if (tick_sched_flag_test(ts, TS_FLAG_IDLE_ACTIVE)) - tick_nohz_stop_idle(ts, now); + kcpustat_irq_enter(now); + /* * If all CPUs are idle we may need to update a stale jiffies value. * Note nohz_full is a special case: a timekeeper is guaranteed to stay @@ -1581,8 +1463,7 @@ static inline void tick_nohz_irq_enter(void) * rare case (typically stop machine). So we must make sure we have a * last resort. */ - if (tick_sched_flag_test(ts, TS_FLAG_STOPPED)) - tick_nohz_update_jiffies(now); + tick_nohz_update_jiffies(now); } #else @@ -1652,20 +1533,15 @@ void tick_setup_sched_timer(bool hrtimer) void tick_sched_timer_dying(int cpu) { struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu); - ktime_t idle_sleeptime, iowait_sleeptime; unsigned long idle_calls, idle_sleeps; /* This must happen before hrtimers are migrated! */ if (tick_sched_flag_test(ts, TS_FLAG_HIGHRES)) hrtimer_cancel(&ts->sched_timer); - idle_sleeptime = ts->idle_sleeptime; - iowait_sleeptime = ts->iowait_sleeptime; idle_calls = ts->idle_calls; idle_sleeps = ts->idle_sleeps; memset(ts, 0, sizeof(*ts)); - ts->idle_sleeptime = idle_sleeptime; - ts->iowait_sleeptime = iowait_sleeptime; ts->idle_calls = idle_calls; ts->idle_sleeps = idle_sleeps; } diff --git a/kernel/time/tick-sched.h b/kernel/time/tick-sched.h index b4a7822f495d..79b9252047b1 100644 --- a/kernel/time/tick-sched.h +++ b/kernel/time/tick-sched.h @@ -44,9 +44,7 @@ struct tick_device { * to resume the tick timer operation in the timeline * when the CPU returns from nohz sleep. * @next_tick: Next tick to be fired when in dynticks mode. - * @idle_jiffies: jiffies at the entry to idle for idle time accounting * @idle_waketime: Time when the idle was interrupted - * @idle_sleeptime_seq: sequence counter for data consistency * @idle_entrytime: Time when the idle call was entered * @last_jiffies: Base jiffies snapshot when next event was last computed * @timer_expires_base: Base time clock monotonic for @timer_expires @@ -55,9 +53,6 @@ struct tick_device { * @idle_expires: Next tick in idle, for debugging purpose only * @idle_calls: Total number of idle calls * @idle_sleeps: Number of idle calls, where the sched tick was stopped - * @idle_exittime: Time when the idle state was left - * @idle_sleeptime: Sum of the time slept in idle with sched tick stopped - * @iowait_sleeptime: Sum of the time slept in idle with sched tick stopped, with IO outstanding * @tick_dep_mask: Tick dependency mask - is set, if someone needs the tick * @check_clocks: Notification mechanism about clocksource changes */ @@ -73,12 +68,10 @@ struct tick_sched { struct hrtimer sched_timer; ktime_t last_tick; ktime_t next_tick; - unsigned long idle_jiffies; ktime_t idle_waketime; unsigned int got_idle_tick; /* Idle entry */ - seqcount_t idle_sleeptime_seq; ktime_t idle_entrytime; /* Tick stop */ @@ -90,11 +83,6 @@ struct tick_sched { unsigned long idle_calls; unsigned long idle_sleeps; - /* Idle exit */ - ktime_t idle_exittime; - ktime_t idle_sleeptime; - ktime_t iowait_sleeptime; - /* Full dynticks handling */ atomic_t tick_dep_mask; diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index c493a4010305..0d5b67f609bb 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -67,6 +67,7 @@ static inline bool tk_is_aux(const struct timekeeper *tk) { return tk->id >= TIMEKEEPER_AUX_FIRST && tk->id <= TIMEKEEPER_AUX_LAST; } +static inline struct tk_data *aux_get_tk_data(clockid_t id); #else static inline bool tk_get_aux_ts64(unsigned int tkid, struct timespec64 *ts) { @@ -77,6 +78,10 @@ static inline bool tk_is_aux(const struct timekeeper *tk) { return false; } +static inline struct tk_data *aux_get_tk_data(clockid_t id) +{ + return NULL; +} #endif static inline void tk_update_aux_offs(struct timekeeper *tk, ktime_t offs) @@ -315,6 +320,7 @@ static __always_inline u64 tk_clock_read(const struct tk_read_base *tkr) return clock->read(clock); } + static inline void clocksource_disable_inline_read(void) { } static inline void clocksource_enable_inline_read(void) { } #endif @@ -1182,44 +1188,107 @@ noinstr time64_t __ktime_get_real_seconds(void) return tk->xtime_sec; } -/** - * ktime_get_snapshot - snapshots the realtime/monotonic raw clocks with counter - * @systime_snapshot: pointer to struct receiving the system time snapshot - */ -void ktime_get_snapshot(struct system_time_snapshot *systime_snapshot) +static inline u64 tk_clock_read_snapshot(const struct tk_read_base *tkr, + struct clocksource_hw_snapshot *chs) { - struct timekeeper *tk = &tk_core.timekeeper; - unsigned int seq; - ktime_t base_raw; - ktime_t base_real; - ktime_t base_boot; - u64 nsec_raw; - u64 nsec_real; - u64 now; + struct clocksource *clock = READ_ONCE(tkr->clock); - WARN_ON_ONCE(timekeeping_suspended); + if (unlikely(clock->read_snapshot)) + return clock->read_snapshot(clock, chs); + + return clock->read(clock); +} + + +/** + * ktime_get_snapshot_id - Simultaneously snapshot a given clock ID with + * CLOCK_MONOTONIC_RAW and the underlying + * clocksource counter value. + * @clock_id: The clock ID to snapshot + * @systime_snapshot: Pointer to struct receiving the system time snapshot + */ +void ktime_get_snapshot_id(clockid_t clock_id, struct system_time_snapshot *systime_snapshot) +{ + ktime_t base_raw, base_sys, offs_sys, *offs, offs_zero = 0; + u64 nsec_raw, nsec_sys, now; + struct timekeeper *tk; + struct tk_data *tkd; + unsigned int seq; + + /* Invalidate the snapshot for all failure cases */ + systime_snapshot->valid = false; + + if (WARN_ON_ONCE(timekeeping_suspended)) + return; + + switch (clock_id) { + case CLOCK_REALTIME: + tkd = &tk_core; + offs = &tk_core.timekeeper.offs_real; + break; + /* Map RAW to MONOTONIC so the loop below is trivial */ + case CLOCK_MONOTONIC_RAW: + case CLOCK_MONOTONIC: + tkd = &tk_core; + offs = &offs_zero; + break; + case CLOCK_BOOTTIME: + tkd = &tk_core; + offs = &tk_core.timekeeper.offs_boot; + break; + case CLOCK_AUX ... CLOCK_AUX_LAST: + tkd = aux_get_tk_data(clock_id); + if (!tkd) + return; + offs = &tkd->timekeeper.offs_aux; + break; + default: + WARN_ON_ONCE(1); + return; + } + + tk = &tkd->timekeeper; do { - seq = read_seqcount_begin(&tk_core.seq); - now = tk_clock_read(&tk->tkr_mono); + struct clocksource_hw_snapshot chs = { }; + + seq = read_seqcount_begin(&tkd->seq); + + /* Aux clocks can be invalid */ + if (!tk->clock_valid) + return; + + now = tk_clock_read_snapshot(&tk->tkr_mono, &chs); systime_snapshot->cs_id = tk->tkr_mono.clock->id; + + systime_snapshot->hw_cycles = chs.hw_cycles; + systime_snapshot->hw_csid = chs.hw_csid; + systime_snapshot->cs_was_changed_seq = tk->cs_was_changed_seq; systime_snapshot->clock_was_set_seq = tk->clock_was_set_seq; - base_real = ktime_add(tk->tkr_mono.base, - tk_core.timekeeper.offs_real); - base_boot = ktime_add(tk->tkr_mono.base, - tk_core.timekeeper.offs_boot); + + base_sys = tk->tkr_mono.base; + offs_sys = *offs; base_raw = tk->tkr_raw.base; - nsec_real = timekeeping_cycles_to_ns(&tk->tkr_mono, now); - nsec_raw = timekeeping_cycles_to_ns(&tk->tkr_raw, now); - } while (read_seqcount_retry(&tk_core.seq, seq)); + + nsec_sys = timekeeping_cycles_to_ns(&tk->tkr_mono, now); + nsec_raw = timekeeping_cycles_to_ns(&tk->tkr_raw, now); + } while (read_seqcount_retry(&tkd->seq, seq)); systime_snapshot->cycles = now; - systime_snapshot->real = ktime_add_ns(base_real, nsec_real); - systime_snapshot->boot = ktime_add_ns(base_boot, nsec_real); - systime_snapshot->raw = ktime_add_ns(base_raw, nsec_raw); + systime_snapshot->systime = ktime_add_ns(base_sys, offs_sys + nsec_sys); + systime_snapshot->monoraw = ktime_add_ns(base_raw, nsec_raw); + + /* + * Special case for PTP. Just transfer the raw time into sys, + * so the call sites can consistently use snap::systime. + */ + if (clock_id == CLOCK_MONOTONIC_RAW) + systime_snapshot->systime = systime_snapshot->monoraw; + /* Tell the consumer that this snapshot is valid */ + systime_snapshot->valid = true; } -EXPORT_SYMBOL_GPL(ktime_get_snapshot); +EXPORT_SYMBOL_GPL(ktime_get_snapshot_id); /* Scale base by mult/div checking for overflow */ static int scale64_check_overflow(u64 mult, u64 div, u64 *base) @@ -1262,7 +1331,7 @@ static int adjust_historical_crosststamp(struct system_time_snapshot *history, struct system_device_crosststamp *ts) { struct timekeeper *tk = &tk_core.timekeeper; - u64 corr_raw, corr_real; + u64 corr_raw, corr_sys; bool interp_forward; int ret; @@ -1279,8 +1348,7 @@ static int adjust_historical_crosststamp(struct system_time_snapshot *history, * Scale the monotonic raw time delta by: * partial_history_cycles / total_history_cycles */ - corr_raw = (u64)ktime_to_ns( - ktime_sub(ts->sys_monoraw, history->raw)); + corr_raw = (u64)ktime_to_ns(ktime_sub(ts->sys_monoraw, history->monoraw)); ret = scale64_check_overflow(partial_history_cycles, total_history_cycles, &corr_raw); if (ret) @@ -1288,30 +1356,29 @@ static int adjust_historical_crosststamp(struct system_time_snapshot *history, /* * If there is a discontinuity in the history, scale monotonic raw - * correction by: - * mult(real)/mult(raw) yielding the realtime correction - * Otherwise, calculate the realtime correction similar to monotonic - * raw calculation + * correction by: + * mult(sys)/mult(raw) yielding the system time correction + * + * Otherwise, calculate the system time correction similar to monotonic + * raw calculation */ if (discontinuity) { - corr_real = mul_u64_u32_div - (corr_raw, tk->tkr_mono.mult, tk->tkr_raw.mult); + corr_sys = mul_u64_u32_div(corr_raw, tk->tkr_mono.mult, tk->tkr_raw.mult); } else { - corr_real = (u64)ktime_to_ns( - ktime_sub(ts->sys_realtime, history->real)); - ret = scale64_check_overflow(partial_history_cycles, - total_history_cycles, &corr_real); + corr_sys = (u64)ktime_to_ns(ktime_sub(ts->sys_systime, history->systime)); + ret = scale64_check_overflow(partial_history_cycles, total_history_cycles, + &corr_sys); if (ret) return ret; } - /* Fixup monotonic raw and real time time values */ + /* Fixup monotonic raw and system time time values */ if (interp_forward) { - ts->sys_monoraw = ktime_add_ns(history->raw, corr_raw); - ts->sys_realtime = ktime_add_ns(history->real, corr_real); + ts->sys_monoraw = ktime_add_ns(history->monoraw, corr_raw); + ts->sys_systime = ktime_add_ns(history->systime, corr_sys); } else { ts->sys_monoraw = ktime_sub_ns(ts->sys_monoraw, corr_raw); - ts->sys_realtime = ktime_sub_ns(ts->sys_realtime, corr_real); + ts->sys_systime = ktime_sub_ns(ts->sys_systime, corr_sys); } return 0; @@ -1368,6 +1435,8 @@ static bool convert_base_to_cs(struct system_counterval_t *scv) return false; scv->cycles += base->offset; + /* Set the clocksource ID as scv::cycles is now clocksource based */ + scv->cs_id = cs->id; return true; } @@ -1435,11 +1504,11 @@ EXPORT_SYMBOL_GPL(ktime_real_to_base_clock); /** * get_device_system_crosststamp - Synchronously capture system/device timestamp - * @get_time_fn: Callback to get simultaneous device time and - * system counter from the device driver + * @get_time_fn: Callback to get simultaneous device time and system counter + * from the device driver * @ctx: Context passed to get_time_fn() - * @history_begin: Historical reference point used to interpolate system - * time when counter provided by the driver is before the current interval + * @history_begin: Historical reference point used to interpolate system time when + * the counter value provided by the driver is before the current interval * @xtstamp: Receives simultaneously captured system and device time * * Reads a timestamp from a device and correlates it to system time @@ -1452,36 +1521,54 @@ int get_device_system_crosststamp(int (*get_time_fn) struct system_time_snapshot *history_begin, struct system_device_crosststamp *xtstamp) { - struct system_counterval_t system_counterval = {}; - struct timekeeper *tk = &tk_core.timekeeper; - u64 cycles, now, interval_start; - unsigned int clock_was_set_seq = 0; - ktime_t base_real, base_raw; - u64 nsec_real, nsec_raw; + u64 syscnt_cycles, cycles, now, interval_start; + unsigned int seq, clock_was_set_seq = 0; + ktime_t base_sys, base_raw, *offs; + u64 nsec_sys, nsec_raw; u8 cs_was_changed_seq; - unsigned int seq; bool do_interp; + struct timekeeper *tk; + struct tk_data *tkd; int ret; + switch (xtstamp->clock_id) { + case CLOCK_REALTIME: + tkd = &tk_core; + offs = &tk_core.timekeeper.offs_real; + break; + case CLOCK_AUX ... CLOCK_AUX_LAST: + tkd = aux_get_tk_data(xtstamp->clock_id); + if (!tkd) + return -ENODEV; + offs = &tkd->timekeeper.offs_aux; + break; + default: + WARN_ON_ONCE(1); + return -ENODEV; + } + + tk = &tkd->timekeeper; + do { - seq = read_seqcount_begin(&tk_core.seq); + seq = read_seqcount_begin(&tkd->seq); /* * Try to synchronously capture device time and a system * counter value calling back into the device driver */ - ret = get_time_fn(&xtstamp->device, &system_counterval, ctx); + ret = get_time_fn(&xtstamp->device, &xtstamp->sys_counter, ctx); if (ret) return ret; /* * Verify that the clocksource ID associated with the captured * system counter value is the same as for the currently - * installed timekeeper clocksource + * installed timekeeper clocksource and convert to it. */ - if (system_counterval.cs_id == CSID_GENERIC || - !convert_base_to_cs(&system_counterval)) + if (xtstamp->sys_counter.cs_id == CSID_GENERIC || + !convert_base_to_cs(&xtstamp->sys_counter)) return -ENODEV; - cycles = system_counterval.cycles; + + cycles = syscnt_cycles = xtstamp->sys_counter.cycles; /* * Check whether the system counter value provided by the @@ -1498,15 +1585,14 @@ int get_device_system_crosststamp(int (*get_time_fn) do_interp = false; } - base_real = ktime_add(tk->tkr_mono.base, - tk_core.timekeeper.offs_real); + base_sys = ktime_add(tk->tkr_mono.base, *offs); base_raw = tk->tkr_raw.base; - nsec_real = timekeeping_cycles_to_ns(&tk->tkr_mono, cycles); + nsec_sys = timekeeping_cycles_to_ns(&tk->tkr_mono, cycles); nsec_raw = timekeeping_cycles_to_ns(&tk->tkr_raw, cycles); - } while (read_seqcount_retry(&tk_core.seq, seq)); + } while (read_seqcount_retry(&tkd->seq, seq)); - xtstamp->sys_realtime = ktime_add_ns(base_real, nsec_real); + xtstamp->sys_systime = ktime_add_ns(base_sys, nsec_sys); xtstamp->sys_monoraw = ktime_add_ns(base_raw, nsec_raw); /* @@ -1523,24 +1609,19 @@ int get_device_system_crosststamp(int (*get_time_fn) * clocksource change */ if (!history_begin || - !timestamp_in_interval(history_begin->cycles, - cycles, system_counterval.cycles) || + !timestamp_in_interval(history_begin->cycles, cycles, syscnt_cycles) || history_begin->cs_was_changed_seq != cs_was_changed_seq) return -EINVAL; - partial_history_cycles = cycles - system_counterval.cycles; - total_history_cycles = cycles - history_begin->cycles; - discontinuity = - history_begin->clock_was_set_seq != clock_was_set_seq; - ret = adjust_historical_crosststamp(history_begin, - partial_history_cycles, - total_history_cycles, - discontinuity, xtstamp); - if (ret) - return ret; + partial_history_cycles = cycles - syscnt_cycles; + total_history_cycles = cycles - history_begin->cycles; + discontinuity = history_begin->clock_was_set_seq != clock_was_set_seq; + + ret = adjust_historical_crosststamp(history_begin, partial_history_cycles, + total_history_cycles, discontinuity, xtstamp); } - return 0; + return ret; } EXPORT_SYMBOL_GPL(get_device_system_crosststamp); diff --git a/kernel/time/timer_list.c b/kernel/time/timer_list.c index 427d7ddea3af..514802def1e0 100644 --- a/kernel/time/timer_list.c +++ b/kernel/time/timer_list.c @@ -152,14 +152,10 @@ static void print_cpu(struct seq_file *m, int cpu, u64 now) P_flag(highres, TS_FLAG_HIGHRES); P_ns(last_tick); P_flag(tick_stopped, TS_FLAG_STOPPED); - P(idle_jiffies); P(idle_calls); P(idle_sleeps); P_ns(idle_entrytime); P_ns(idle_waketime); - P_ns(idle_exittime); - P_ns(idle_sleeptime); - P_ns(iowait_sleeptime); P(last_jiffies); P(next_timer); P_ns(idle_expires); @@ -256,7 +252,7 @@ static void timer_list_show_tickdevices_header(struct seq_file *m) static inline void timer_list_header(struct seq_file *m, u64 now) { - SEQ_printf(m, "Timer List Version: v0.10\n"); + SEQ_printf(m, "Timer List Version: v0.11\n"); SEQ_printf(m, "HRTIMER_MAX_CLOCK_BASES: %d\n", HRTIMER_MAX_CLOCK_BASES); SEQ_printf(m, "now at %Ld nsecs\n", (unsigned long long)now); SEQ_printf(m, "\n"); diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 22745485aa53..fcbb144baef9 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -636,7 +636,7 @@ config DEBUG_FORCE_WEAK_PER_CPU config WARN_CONTEXT_ANALYSIS bool "Compiler context-analysis warnings" - depends on CC_IS_CLANG && CLANG_VERSION >= 220100 + depends on CC_IS_CLANG && CLANG_VERSION >= 230000 # Branch profiling re-defines "if", which messes with the compiler's # ability to analyze __cond_acquires(..), resulting in false positives. depends on !TRACE_BRANCH_PROFILING @@ -647,7 +647,7 @@ config WARN_CONTEXT_ANALYSIS and releasing user-definable "context locks". Clang's name of the feature is "Thread Safety Analysis". Requires - Clang 22.1.0 or later. + Clang 23 or later. Produces warnings by default. Select CONFIG_WERROR if you wish to turn these warnings into errors. @@ -3076,7 +3076,7 @@ config FORTIFY_KUNIT_TEST config LONGEST_SYM_KUNIT_TEST tristate "Test the longest symbol possible" if !KUNIT_ALL_TESTS depends on KUNIT && KPROBES - depends on !PREFIX_SYMBOLS && !CFI && !GCOV_KERNEL + depends on !CALL_PADDING && !CFI && !GCOV_KERNEL default KUNIT_ALL_TESTS help Tests the longest symbol possible diff --git a/lib/locking-selftest.c b/lib/locking-selftest.c index d939403331b5..bfafe1204c7b 100644 --- a/lib/locking-selftest.c +++ b/lib/locking-selftest.c @@ -1431,10 +1431,9 @@ static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask) { int saved_preempt_count = preempt_count(); #ifdef CONFIG_PREEMPT_RT -#ifdef CONFIG_SMP int saved_mgd_count = current->migration_disabled; -#endif int saved_rcu_count = current->rcu_read_lock_nesting; + int saved_sched_rt_mutex = current->sched_rt_mutex; #endif WARN_ON(irqs_disabled()); @@ -1471,10 +1470,10 @@ static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask) preempt_count_set(saved_preempt_count); #ifdef CONFIG_PREEMPT_RT -#ifdef CONFIG_SMP + current->sched_rt_mutex = saved_sched_rt_mutex; + while (current->migration_disabled > saved_mgd_count) migrate_enable(); -#endif while (current->rcu_read_lock_nesting > saved_rcu_count) rcu_read_unlock(); diff --git a/lib/test_context-analysis.c b/lib/test_context-analysis.c index 06b4a6a028e0..316f4dfcda65 100644 --- a/lib/test_context-analysis.c +++ b/lib/test_context-analysis.c @@ -159,6 +159,10 @@ TEST_SPINLOCK_COMMON(read_lock, struct test_mutex_data { struct mutex mtx; int counter __guarded_by(&mtx); + + struct mutex mtx2; + int anyread __guarded_by(&mtx, &mtx2); + int *anyptr __pt_guarded_by(&mtx, &mtx2); }; static void __used test_mutex_init(struct test_mutex_data *d) @@ -219,6 +223,26 @@ static void __used test_mutex_cond_guard(struct test_mutex_data *d) } } +static void __used test_mutex_multiguard(struct test_mutex_data *d) +{ + mutex_lock(&d->mtx); + (void)d->anyread; + (void)*d->anyptr; + mutex_unlock(&d->mtx); + + mutex_lock(&d->mtx2); + (void)d->anyread; + (void)*d->anyptr; + mutex_unlock(&d->mtx2); + + mutex_lock(&d->mtx); + mutex_lock(&d->mtx2); + d->anyread++; + (*d->anyptr)++; + mutex_unlock(&d->mtx2); + mutex_unlock(&d->mtx); +} + struct test_seqlock_data { seqlock_t sl; int counter __guarded_by(&sl); diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index da224011fafd..e0f289d3d110 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -126,7 +126,7 @@ bool vdso_get_timestamp(const struct vdso_time_data *vd, const struct vdso_clock } static __always_inline -const struct vdso_time_data *__arch_get_vdso_u_timens_data(const struct vdso_time_data *vd) +const struct vdso_time_data *vdso_timens_data(const struct vdso_time_data *vd) { return (void *)vd + PAGE_SIZE; } @@ -135,7 +135,7 @@ static __always_inline bool do_hres_timens(const struct vdso_time_data *vdns, const struct vdso_clock *vcns, clockid_t clk, struct __kernel_timespec *ts) { - const struct vdso_time_data *vd = __arch_get_vdso_u_timens_data(vdns); + const struct vdso_time_data *vd = vdso_timens_data(vdns); const struct timens_offset *offs = &vcns->offset[clk]; const struct vdso_clock *vc = vd->clock_data; u32 seq; @@ -191,7 +191,7 @@ static __always_inline bool do_coarse_timens(const struct vdso_time_data *vdns, const struct vdso_clock *vcns, clockid_t clk, struct __kernel_timespec *ts) { - const struct vdso_time_data *vd = __arch_get_vdso_u_timens_data(vdns); + const struct vdso_time_data *vd = vdso_timens_data(vdns); const struct timens_offset *offs = &vcns->offset[clk]; const struct vdso_clock *vc = vd->clock_data; const struct vdso_timestamp *vdso_ts; @@ -250,7 +250,7 @@ bool do_aux(const struct vdso_time_data *vd, clockid_t clock, struct __kernel_ti do { while (vdso_read_begin_timens(vc, &seq)) { /* Re-read from the real time data page, reload seq by looping */ - vd = __arch_get_vdso_u_timens_data(vd); + vd = vdso_timens_data(vd); vc = &vd->aux_clock_data[idx]; } @@ -360,7 +360,7 @@ __cvdso_gettimeofday_data(const struct vdso_time_data *vd, if (unlikely(tz != NULL)) { if (vdso_is_timens_clock(vc)) - vd = __arch_get_vdso_u_timens_data(vd); + vd = vdso_timens_data(vd); tz->tz_minuteswest = vd[CS_HRES_COARSE].tz_minuteswest; tz->tz_dsttime = vd[CS_HRES_COARSE].tz_dsttime; @@ -383,7 +383,7 @@ __cvdso_time_data(const struct vdso_time_data *vd, __kernel_old_time_t *time) __kernel_old_time_t t; if (vdso_is_timens_clock(vc)) { - vd = __arch_get_vdso_u_timens_data(vd); + vd = vdso_timens_data(vd); vc = vd->clock_data; } @@ -414,7 +414,7 @@ bool __cvdso_clock_getres_common(const struct vdso_time_data *vd, clockid_t cloc return false; if (vdso_is_timens_clock(vc)) - vd = __arch_get_vdso_u_timens_data(vd); + vd = vdso_timens_data(vd); /* * Convert the clockid to a bitmask and use it to check which diff --git a/rust/kernel/sync/completion.rs b/rust/kernel/sync/completion.rs index c50012a940a3..35ff049ff078 100644 --- a/rust/kernel/sync/completion.rs +++ b/rust/kernel/sync/completion.rs @@ -94,6 +94,7 @@ impl Completion { /// /// This method wakes up all tasks waiting on this completion; after this operation the /// completion is permanently done, i.e. signals all current and future waiters. + #[inline] pub fn complete_all(&self) { // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`. unsafe { bindings::complete_all(self.as_raw()) }; @@ -105,6 +106,7 @@ impl Completion { /// timeout. /// /// See also [`Completion::complete_all`]. + #[inline] pub fn wait_for_completion(&self) { // SAFETY: `self.as_raw()` is a pointer to a valid `struct completion`. unsafe { bindings::wait_for_completion(self.as_raw()) }; diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index afd0e411e48e..9fc2c7a0a45e 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -190,7 +190,11 @@ objtool-args-$(CONFIG_HAVE_JUMP_LABEL_HACK) += --hacks=jump_label objtool-args-$(CONFIG_HAVE_NOINSTR_HACK) += --hacks=noinstr objtool-args-$(CONFIG_MITIGATION_CALL_DEPTH_TRACKING) += --hacks=skylake objtool-args-$(CONFIG_X86_KERNEL_IBT) += --ibt -objtool-args-$(CONFIG_FINEIBT) += --cfi +objtool-args-$(CONFIG_CALL_PADDING) += --prefix=$(CONFIG_FUNCTION_PADDING_BYTES) +ifdef CONFIG_CALL_PADDING +objtool-args-$(CONFIG_CFI) += --cfi +objtool-args-$(CONFIG_FINEIBT) += --fineibt +endif objtool-args-$(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL) += --mcount ifdef CONFIG_FTRACE_MCOUNT_USE_OBJTOOL objtool-args-$(CONFIG_HAVE_OBJTOOL_NOP_MCOUNT) += --mnop @@ -203,7 +207,6 @@ objtool-args-$(CONFIG_STACK_VALIDATION) += --stackval objtool-args-$(CONFIG_HAVE_STATIC_CALL_INLINE) += --static-call objtool-args-$(CONFIG_HAVE_UACCESS_VALIDATION) += --uaccess objtool-args-$(or $(CONFIG_GCOV_KERNEL),$(CONFIG_KCOV)) += --no-unreachable -objtool-args-$(CONFIG_PREFIX_SYMBOLS) += --prefix=$(CONFIG_FUNCTION_PADDING_BYTES) objtool-args-$(CONFIG_OBJTOOL_WERROR) += --werror objtool-args = $(objtool-args-y) \ diff --git a/scripts/gdb/linux/timerlist.py b/scripts/gdb/linux/timerlist.py index 9fb3436a217c..744b032e4d38 100644 --- a/scripts/gdb/linux/timerlist.py +++ b/scripts/gdb/linux/timerlist.py @@ -90,14 +90,10 @@ def print_cpu(hrtimer_bases, cpu, max_clock_bases): text += f" .{'nohz':15s}: {int(bool(ts['flags'] & TS_FLAG_NOHZ))}\n" text += f" .{'last_tick':15s}: {ts['last_tick']}\n" text += f" .{'tick_stopped':15s}: {int(bool(ts['flags'] & TS_FLAG_STOPPED))}\n" - text += f" .{'idle_jiffies':15s}: {ts['idle_jiffies']}\n" text += f" .{'idle_calls':15s}: {ts['idle_calls']}\n" text += f" .{'idle_sleeps':15s}: {ts['idle_sleeps']}\n" text += f" .{'idle_entrytime':15s}: {ts['idle_entrytime']} nsecs\n" text += f" .{'idle_waketime':15s}: {ts['idle_waketime']} nsecs\n" - text += f" .{'idle_exittime':15s}: {ts['idle_exittime']} nsecs\n" - text += f" .{'idle_sleeptime':15s}: {ts['idle_sleeptime']} nsecs\n" - text += f" .{'iowait_sleeptime':15s}: {ts['iowait_sleeptime']} nsecs\n" text += f" .{'last_jiffies':15s}: {ts['last_jiffies']}\n" text += f" .{'next_timer':15s}: {ts['next_timer']}\n" text += f" .{'idle_expires':15s}: {ts['idle_expires']} nsecs\n" diff --git a/scripts/livepatch/klp-build b/scripts/livepatch/klp-build index 0ad7e6631314..c4a7acf8edc3 100755 --- a/scripts/livepatch/klp-build +++ b/scripts/livepatch/klp-build @@ -3,7 +3,7 @@ # # Build a livepatch module -# shellcheck disable=SC1090,SC2155 +# shellcheck disable=SC1090,SC2155,SC2164 if (( BASH_VERSINFO[0] < 4 || \ (BASH_VERSINFO[0] == 4 && BASH_VERSINFO[1] < 4) )); then @@ -11,21 +11,19 @@ if (( BASH_VERSINFO[0] < 4 || \ exit 1 fi -set -o errexit set -o errtrace set -o pipefail set -o nounset # Allow doing 'cmd | mapfile -t array' instead of 'mapfile -t array < <(cmd)'. -# This helps keep execution in pipes so pipefail+errexit can catch errors. +# This helps keep execution in pipes so pipefail+ERR trap can catch errors. shopt -s lastpipe -unset DEBUG_CLONE DIFF_CHECKSUM SKIP_CLEANUP XTRACE +unset DEBUG_CLONE DIFF_CHECKSUM SKIP_CLEANUP VERBOSE XTRACE REPLACE=1 SHORT_CIRCUIT=0 JOBS="$(getconf _NPROCESSORS_ONLN)" -VERBOSE="-s" shopt -o xtrace | grep -q 'on' && XTRACE=1 # Avoid removing the previous $TMP_DIR until args have been fully processed. @@ -35,16 +33,16 @@ SCRIPT="$(basename "$0")" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FIX_PATCH_LINES="$SCRIPT_DIR/fix-patch-lines" -SRC="$(pwd)" -OBJ="$(pwd)" +OBJTOOL="$PWD/tools/objtool/objtool" +CONFIG="$PWD/.config" +TMP_DIR="$PWD/klp-tmp" -CONFIG="$OBJ/.config" -TMP_DIR="$OBJ/klp-tmp" - -ORIG_DIR="$TMP_DIR/orig" -PATCHED_DIR="$TMP_DIR/patched" -DIFF_DIR="$TMP_DIR/diff" -KMOD_DIR="$TMP_DIR/kmod" +ORIG_DIR="$TMP_DIR/1-orig" +PATCHED_DIR="$TMP_DIR/2-patched" +ORIG_CSUM_DIR="$TMP_DIR/3-checksum-orig" +PATCHED_CSUM_DIR="$TMP_DIR/3-checksum-patched" +DIFF_DIR="$TMP_DIR/4-diff" +KMOD_DIR="$TMP_DIR/5-kmod" STASH_DIR="$TMP_DIR/stash" TIMESTAMP="$TMP_DIR/timestamp" @@ -90,7 +88,7 @@ declare -a STASHED_FILES stash_file() { local file="$1" - local rel_file="${file#"$SRC"/}" + local rel_file="${file#"$PWD"/}" [[ ! -e "$file" ]] && die "no file to stash: $file" @@ -104,7 +102,7 @@ restore_files() { local file for file in "${STASHED_FILES[@]}"; do - mv -f "$STASH_DIR/$file" "$SRC/$file" || warn "can't restore file: $file" + mv -f "$STASH_DIR/$file" "$PWD/$file" || warn "can't restore file: $file" done STASHED_FILES=() @@ -140,10 +138,11 @@ Options: Advanced Options: -d, --debug Show symbol/reloc cloning decisions -S, --short-circuit=STEP Start at build step (requires prior --keep-tmp) - 1|orig Build original kernel (default) - 2|patched Build patched kernel - 3|diff Diff objects - 4|kmod Build patch module + 1|orig Build original kernel (default) + 2|patched Build patched kernel + 3|checksum Generate checksums + 4|diff Diff objects + 5|kmod Build patch module -T, --keep-tmp Preserve tmp dir on exit EOF @@ -158,6 +157,7 @@ process_args() { local short local long local args + local patch short="hfj:o:vdS:T" long="help,show-first-changed,jobs:,output:,no-replace,verbose,debug,short-circuit:,keep-tmp" @@ -194,7 +194,7 @@ process_args() { shift ;; -v | --verbose) - VERBOSE="V=1" + VERBOSE=1 shift ;; -d | --debug) @@ -206,10 +206,11 @@ process_args() { [[ ! -d "$TMP_DIR" ]] && die "--short-circuit requires preserved klp-tmp dir" keep_tmp=1 case "$2" in - 1 | orig) SHORT_CIRCUIT=1; ;; - 2 | patched) SHORT_CIRCUIT=2; ;; - 3 | diff) SHORT_CIRCUIT=3; ;; - 4 | mod) SHORT_CIRCUIT=4; ;; + 1 | orig) SHORT_CIRCUIT=1; ;; + 2 | patched) SHORT_CIRCUIT=2; ;; + 3 | checksum) SHORT_CIRCUIT=3; ;; + 4 | diff) SHORT_CIRCUIT=4; ;; + 5 | kmod) SHORT_CIRCUIT=5; ;; *) die "invalid short-circuit step '$2'" ;; esac shift 2 @@ -236,6 +237,10 @@ process_args() { KEEP_TMP="$keep_tmp" PATCHES=("$@") + + for patch in "${PATCHES[@]}"; do + [[ -f "$patch" ]] || die "$patch doesn't exist" + done } # temporarily disable xtrace for especially verbose code @@ -270,6 +275,9 @@ validate_config() { [[ "$CONFIG_AS_VERSION" -lt 200000 ]] && \ die "Clang assembler version < 20 not supported" + [[ -x "$OBJTOOL" ]] && "$OBJTOOL" klp 2>&1 | command grep -q "not implemented" && \ + die "objtool not built with KLP support; install xxhash-devel/libxxhash-dev (version >= 0.8) and recompile" + return 0 } @@ -301,12 +309,17 @@ set_module_name() { # Hardcode the value printed by the localversion script to prevent patch # application from appending it with '+' due to a dirty working tree. set_kernelversion() { - local file="$SRC/scripts/setlocalversion" + local file="$PWD/scripts/setlocalversion" local kernelrelease stash_file "$file" - kernelrelease="$(cd "$SRC" && make syncconfig &>/dev/null && make -s kernelrelease)" + if [[ -n "$(make -s listnewconfig 2>/dev/null)" ]]; then + die ".config mismatch, check your .config or run 'make olddefconfig'" + fi + make syncconfig &>/dev/null || die "make syncconfig failed" + + kernelrelease="$(make -s kernelrelease)" [[ -z "$kernelrelease" ]] && die "failed to get kernel version" sed -i "2i echo $kernelrelease; exit 0" scripts/setlocalversion @@ -349,7 +362,7 @@ check_unsupported_patches() { for file in "${files[@]}"; do case "$file" in - lib/*|*.S) + lib/*|*/vdso/*|*/realmode/rm/*|*.S) die "${patch}: unsupported patch to $file" ;; esac @@ -367,24 +380,24 @@ apply_patch() { [[ ! -f "$patch" ]] && die "$patch doesn't exist" status=0 - output=$(patch -d "$SRC" -p1 --dry-run --no-backup-if-mismatch -r /dev/null "${extra_args[@]}" < "$patch" 2>&1) || status=$? + output=$(patch -p1 --dry-run --no-backup-if-mismatch -r /dev/null "${extra_args[@]}" < "$patch" 2>&1) || status=$? if [[ "$status" -ne 0 ]]; then echo "$output" >&2 die "$patch did not apply" elif [[ "$output" =~ $drift_regex ]]; then - echo "$output" >&2 + [[ -v VERBOSE ]] && echo "$output" >&2 warn "${patch} applied with fuzz" fi - patch -d "$SRC" -p1 --no-backup-if-mismatch -r /dev/null "${extra_args[@]}" --silent < "$patch" APPLIED_PATCHES+=("$patch") + patch -p1 --no-backup-if-mismatch -r /dev/null "${extra_args[@]}" --silent < "$patch" } revert_patch() { local patch="$1" local tmp=() - patch -d "$SRC" -p1 -R --silent --no-backup-if-mismatch -r /dev/null < "$patch" + patch -p1 -R --force --no-backup-if-mismatch -r /dev/null &> /dev/null < "$patch" || true for p in "${APPLIED_PATCHES[@]}"; do [[ "$p" == "$patch" ]] && continue @@ -422,8 +435,21 @@ validate_patches() { do_init() { # We're not yet smart enough to handle anything other than in-tree # builds in pwd. - [[ ! "$SRC" -ef "$SCRIPT_DIR/../.." ]] && die "please run from the kernel root directory" - [[ ! "$OBJ" -ef "$SCRIPT_DIR/../.." ]] && die "please run from the kernel root directory" + [[ ! "$PWD" -ef "$SCRIPT_DIR/../.." ]] && die "please run from the kernel root directory" + + if (( SHORT_CIRCUIT >= 2 )); then + [[ -f "$ORIG_DIR/.complete" ]] || die "-S $SHORT_CIRCUIT requires completed $ORIG_DIR" + fi + if (( SHORT_CIRCUIT >= 3 )); then + [[ -f "$PATCHED_DIR/.complete" ]] || die "-S $SHORT_CIRCUIT requires completed $PATCHED_DIR" + fi + if (( SHORT_CIRCUIT >= 4 )); then + [[ -f "$ORIG_CSUM_DIR/.complete" ]] || die "-S $SHORT_CIRCUIT requires completed $ORIG_CSUM_DIR" + [[ -f "$PATCHED_CSUM_DIR/.complete" ]] || die "-S $SHORT_CIRCUIT requires completed $PATCHED_CSUM_DIR" + fi + if (( SHORT_CIRCUIT >= 5 )); then + [[ -f "$DIFF_DIR/.complete" ]] || die "-S $SHORT_CIRCUIT requires completed $DIFF_DIR" + fi (( SHORT_CIRCUIT <= 1 )) && rm -rf "$TMP_DIR" mkdir -p "$TMP_DIR" @@ -454,11 +480,11 @@ refresh_patch() { get_patch_output_files "$patch" | mapfile -t output_files # Copy orig source files to 'a' - ( cd "$SRC" && echo "${input_files[@]}" | xargs cp --parents --target-directory="$tmpdir/a" ) + echo "${input_files[@]}" | xargs cp --parents --target-directory="$tmpdir/a" # Copy patched source files to 'b' apply_patch "$patch" "--silent" - ( cd "$SRC" && echo "${output_files[@]}" | xargs cp --parents --target-directory="$tmpdir/b" ) + echo "${output_files[@]}" | xargs cp --parents --target-directory="$tmpdir/b" revert_patch "$patch" # Diff 'a' and 'b' to make a clean patch @@ -502,20 +528,14 @@ clean_kernel() { cmd+=("-j$JOBS") cmd+=("clean") - ( - cd "$SRC" - "${cmd[@]}" - ) + "${cmd[@]}" } build_kernel() { local build="$1" local log="$TMP_DIR/build.log" - local objtool_args=() local cmd=() - objtool_args=("--checksum") - cmd=("make") # When a patch to a kernel module references a newly created unexported @@ -535,19 +555,20 @@ build_kernel() { # cmd+=("KBUILD_MODPOST_WARN=1") - cmd+=("$VERBOSE") + if [[ -v VERBOSE ]]; then + cmd+=("V=1") + else + cmd+=("-s") + fi cmd+=("-j$JOBS") cmd+=("KCFLAGS=-ffunction-sections -fdata-sections") - cmd+=("OBJTOOL_ARGS=${objtool_args[*]}") cmd+=("vmlinux") cmd+=("modules") - ( - cd "$SRC" - "${cmd[@]}" \ - 1> >(tee -a "$log") \ - 2> >(tee -a "$log" | grep0 -v "modpost.*undefined!" >&2) - ) || die "$build kernel build failed" + "${cmd[@]}" \ + 1> >(tee -a "$log") \ + 2> >(tee -a "$log" | grep0 -v "modpost.*undefined!" >&2) \ + || die "$build kernel build failed" } find_objects() { @@ -555,9 +576,9 @@ find_objects() { # Find root-level vmlinux.o and non-root-level .ko files, # excluding klp-tmp/ and .git/ - find "$OBJ" \( -path "$TMP_DIR" -o -path "$OBJ/.git" -o -regex "$OBJ/[^/][^/]*\.ko" \) -prune -o \ + find "$PWD" \( -path "$TMP_DIR" -o -path "$PWD/.git" -o -regex "$PWD/[^/][^/]*\.ko" \) -prune -o \ -type f "${opts[@]}" \ - \( -name "*.ko" -o -path "$OBJ/vmlinux.o" \) \ + \( -name "*.ko" -o -path "$PWD/vmlinux.o" \) \ -printf '%P\n' } @@ -570,10 +591,10 @@ copy_orig_objects() { find_objects | mapfile -t files - xtrace_save "copying orig objects" + xtrace_save "copying original objects" for _file in "${files[@]}"; do local rel_file="${_file/.ko/.o}" - local file="$OBJ/$rel_file" + local file="$PWD/$rel_file" local orig_file="$ORIG_DIR/$rel_file" local orig_dir="$(dirname "$orig_file")" @@ -586,6 +607,7 @@ copy_orig_objects() { mv -f "$TMP_DIR/build.log" "$ORIG_DIR" touch "$TIMESTAMP" + touch "$ORIG_DIR/.complete" } # Copy all changed objects to $PATCHED_DIR @@ -606,7 +628,7 @@ copy_patched_objects() { xtrace_save "copying changed objects" for _file in "${files[@]}"; do local rel_file="${_file/.ko/.o}" - local file="$OBJ/$rel_file" + local file="$PWD/$rel_file" local orig_file="$ORIG_DIR/$rel_file" local patched_file="$PATCHED_DIR/$rel_file" local patched_dir="$(dirname "$patched_file")" @@ -624,6 +646,36 @@ copy_patched_objects() { (( found == 0 )) && die "no changes detected" mv -f "$TMP_DIR/build.log" "$PATCHED_DIR" + touch "$PATCHED_DIR/.complete" +} + +# Copy .o files to a separate directory and run "objtool klp checksum" on each +# copy. The checksums are written to a .discard.sym_checksum section. +# +# If match_dir is given, only process files which also exist there. +generate_checksums() { + local src_dir="$1" + local dest_dir="$2" + local match_dir="${3:-}" + local files=() + local file + + rm -rf "$dest_dir" + mkdir -p "$dest_dir" + + find "$src_dir" -type f -name "*.o" | mapfile -t files + for file in "${files[@]}"; do + local rel="${file#"$src_dir"/}" + local dest="$dest_dir/$rel" + + [[ -n "$match_dir" && ! -f "$match_dir/$rel" ]] && continue + + mkdir -p "$(dirname "$dest")" + cp -f "$file" "$dest" + "$OBJTOOL" klp checksum "$dest" + done + + touch "$dest_dir/.complete" } # Diff changed objects, writing output object to $DIFF_DIR @@ -635,23 +687,23 @@ diff_objects() { rm -rf "$DIFF_DIR" mkdir -p "$DIFF_DIR" - find "$PATCHED_DIR" -type f -name "*.o" | mapfile -t files + find "$PATCHED_CSUM_DIR" -type f -name "*.o" | mapfile -t files [[ ${#files[@]} -eq 0 ]] && die "no changes detected" [[ -v DEBUG_CLONE ]] && opts=("--debug") # Diff all changed objects for file in "${files[@]}"; do - local rel_file="${file#"$PATCHED_DIR"/}" + local rel_file="${file#"$PATCHED_CSUM_DIR"/}" local orig_file="$rel_file" - local patched_file="$PATCHED_DIR/$rel_file" + local patched_file="$PATCHED_CSUM_DIR/$rel_file" local out_file="$DIFF_DIR/$rel_file" local filter=() local cmd=() mkdir -p "$(dirname "$out_file")" - cmd=("$SRC/tools/objtool/objtool") + cmd=("$OBJTOOL") cmd+=("klp") cmd+=("diff") (( ${#opts[@]} > 0 )) && cmd+=("${opts[@]}") @@ -668,18 +720,21 @@ diff_objects() { fi ( - cd "$ORIG_DIR" + cd "$ORIG_CSUM_DIR" + [[ -v VERBOSE ]] && echo "cd $ORIG_CSUM_DIR && ${cmd[*]}" "${cmd[@]}" \ 1> >(tee -a "$log") \ 2> >(tee -a "$log" | "${filter[@]}" >&2) || \ die "objtool klp diff failed" ) done + + touch "$DIFF_DIR/.complete" } -# For each changed object, run objtool with --debug-checksum to get the -# per-instruction checksums, and then diff those to find the first changed -# instruction for each function. +# For each changed object, run "objtool klp checksum" with --debug-checksum to +# get the per-instruction checksums, and then diff those to find the first +# changed instruction for each function. diff_checksums() { local orig_log="$ORIG_DIR/checksum.log" local patched_log="$PATCHED_DIR/checksum.log" @@ -703,9 +758,8 @@ diff_checksums() { fi done - cmd=("$SRC/tools/objtool/objtool") - cmd+=("--checksum") - cmd+=("--link") + cmd=("$OBJTOOL") + cmd+=("klp" "checksum") cmd+=("--dry-run") for file in "${!funcs[@]}"; do @@ -714,21 +768,37 @@ diff_checksums() { ( cd "$ORIG_DIR" "${cmd[@]}" "$opt" "$file" &> "$orig_log" || \ - ( cat "$orig_log" >&2; die "objtool --debug-checksum failed" ) + ( cat "$orig_log" >&2; die "objtool klp checksum failed" ) cd "$PATCHED_DIR" "${cmd[@]}" "$opt" "$file" &> "$patched_log" || \ - ( cat "$patched_log" >&2; die "objtool --debug-checksum failed" ) + ( cat "$patched_log" >&2; die "objtool klp checksum failed" ) ) for func in ${funcs[$file]}; do - diff <( grep0 -E "^DEBUG: .*checksum: $func " "$orig_log" | sed "s|$ORIG_DIR/||") \ - <( grep0 -E "^DEBUG: .*checksum: $func " "$patched_log" | sed "s|$PATCHED_DIR/||") \ - | gawk '/^< DEBUG: / { - gsub(/:/, "") - printf "%s: %s: %s\n", $3, $5, $6 - exit - }' || true + local -a orig patched + paste <(grep0 -E "^DEBUG: .*checksum: $func " "$orig_log") \ + <(grep0 -E "^DEBUG: .*checksum: $func " "$patched_log") | + while IFS= read -r line; do + read -ra orig <<< "${line%%$'\t'*}" + read -ra patched <<< "${line#*$'\t'}" + + if [[ ${#patched[@]} -eq 0 ]]; then + printf "%s: %s: %s (removed)\n" "${orig[1]%:}" "${orig[3]}" "${orig[-2]}" + break + elif [[ ${#orig[@]} -eq 0 ]]; then + printf "%s: %s: %s (added)\n" "${patched[1]%:}" "${patched[3]}" "${patched[-2]}" + break + fi + + [[ "${orig[-1]}" == "${patched[-1]}" ]] && continue + + printf "%s: %s: %s" "${orig[1]%:}" "${orig[3]}" "${orig[-2]}" + [[ "${orig[-2]}" != "${patched[-2]}" ]] && \ + printf " (patched: %s)" "${patched[-2]}" + printf "\n" + break + done || true done done } @@ -745,7 +815,7 @@ build_patch_module() { rm -rf "$KMOD_DIR" mkdir -p "$KMOD_DIR" - cp -f "$SRC/scripts/livepatch/init.c" "$KMOD_DIR" + cp -f "$SCRIPT_DIR/init.c" "$KMOD_DIR" echo "obj-m := $NAME.o" > "$makefile" echo -n "$NAME-y := init.o" >> "$makefile" @@ -780,19 +850,20 @@ build_patch_module() { [[ $REPLACE -eq 0 ]] && cflags+=("-DKLP_NO_REPLACE") cmd=("make") - cmd+=("$VERBOSE") + if [[ -v VERBOSE ]]; then + cmd+=("V=1") + else + cmd+=("-s") + fi cmd+=("-j$JOBS") cmd+=("--directory=.") cmd+=("M=$KMOD_DIR") cmd+=("KCFLAGS=${cflags[*]}") # Build a "normal" kernel module with init.c and the diffed objects - ( - cd "$SRC" - "${cmd[@]}" \ - 1> >(tee -a "$log") \ - 2> >(tee -a "$log" >&2) - ) + "${cmd[@]}" \ + 1> >(tee -a "$log") \ + 2> >(tee -a "$log" >&2) kmod_file="$KMOD_DIR/$NAME.ko" @@ -803,7 +874,7 @@ build_patch_module() { objcopy --remove-section=.BTF "$kmod_file" # Fix (and work around) linker wreckage for klp syms / relocs - "$SRC/tools/objtool/objtool" klp post-link "$kmod_file" || die "objtool klp post-link failed" + "$OBJTOOL" klp post-link "$kmod_file" || die "objtool klp post-link failed" cp -f "$kmod_file" "$OUTFILE" } @@ -839,6 +910,13 @@ if (( SHORT_CIRCUIT <= 2 )); then fi if (( SHORT_CIRCUIT <= 3 )); then + status "Generating original checksums" + generate_checksums "$ORIG_DIR" "$ORIG_CSUM_DIR" "$PATCHED_DIR" + status "Generating patched checksums" + generate_checksums "$PATCHED_DIR" "$PATCHED_CSUM_DIR" +fi + +if (( SHORT_CIRCUIT <= 4 )); then status "Diffing objects" diff_objects if [[ -v DIFF_CHECKSUM ]]; then @@ -847,7 +925,7 @@ if (( SHORT_CIRCUIT <= 3 )); then fi fi -if (( SHORT_CIRCUIT <= 4 )); then +if (( SHORT_CIRCUIT <= 5 )); then status "Building patch module: $OUTFILE" build_patch_module fi diff --git a/sound/hda/common/controller.c b/sound/hda/common/controller.c index a847546753db..afec5c5546ec 100644 --- a/sound/hda/common/controller.c +++ b/sound/hda/common/controller.c @@ -491,9 +491,9 @@ static int azx_get_time_info(struct snd_pcm_substream *substream, struct snd_pcm_audio_tstamp_config *audio_tstamp_config, struct snd_pcm_audio_tstamp_report *audio_tstamp_report) { + struct system_device_crosststamp xtstamp = { .clock_id = CLOCK_REALTIME }; struct azx_dev *azx_dev = get_azx_dev(substream); struct snd_pcm_runtime *runtime = substream->runtime; - struct system_device_crosststamp xtstamp; int ret; u64 nsec; @@ -527,7 +527,7 @@ static int azx_get_time_info(struct snd_pcm_substream *substream, break; default: - *system_ts = ktime_to_timespec64(xtstamp.sys_realtime); + *system_ts = ktime_to_timespec64(xtstamp.sys_systime); break; } diff --git a/tools/objtool/Build b/tools/objtool/Build index 600da051af12..93a37b0dfd31 100644 --- a/tools/objtool/Build +++ b/tools/objtool/Build @@ -12,7 +12,7 @@ objtool-$(BUILD_DISAS) += disas.o objtool-$(BUILD_DISAS) += trace.o objtool-$(BUILD_ORC) += orc_gen.o orc_dump.o -objtool-$(BUILD_KLP) += builtin-klp.o klp-diff.o klp-post-link.o +objtool-$(BUILD_KLP) += builtin-klp.o klp-checksum.o klp-diff.o klp-post-link.o objtool-y += libstring.o objtool-y += libctype.o diff --git a/tools/objtool/Makefile b/tools/objtool/Makefile index 1b0544785ba6..0fc9f316199b 100644 --- a/tools/objtool/Makefile +++ b/tools/objtool/Makefile @@ -58,7 +58,7 @@ INCLUDES := -I$(srctree)/tools/include \ -I$(srctree)/tools/arch/$(SRCARCH)/include \ -I$(srctree)/tools/objtool/include \ -I$(srctree)/tools/objtool/arch/$(SRCARCH)/include \ - -I$(LIBSUBCMD_OUTPUT)/include + -I$(srctree)/tools/lib OBJTOOL_CFLAGS := -std=gnu11 -fomit-frame-pointer -O2 -g $(WARNINGS) \ $(INCLUDES) $(LIBELF_FLAGS) $(LIBXXHASH_CFLAGS) $(HOSTCFLAGS) @@ -135,7 +135,7 @@ $(LIBSUBCMD): fixdep $(LIBSUBCMD_OUTPUT) FORCE $(Q)$(MAKE) -C $(LIBSUBCMD_DIR) O=$(LIBSUBCMD_OUTPUT) \ DESTDIR=$(LIBSUBCMD_OUTPUT) prefix= subdir= \ $(HOST_OVERRIDES) EXTRA_CFLAGS="$(OBJTOOL_CFLAGS)" \ - $@ install_headers + $@ $(LIBSUBCMD)-clean: $(call QUIET_CLEAN, libsubcmd) diff --git a/tools/objtool/arch/x86/decode.c b/tools/objtool/arch/x86/decode.c index 350b8ee6e776..1b387d5a195b 100644 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -805,14 +805,27 @@ int arch_decode_instruction(struct objtool_file *file, const struct section *sec break; } - if (ins.immediate.nbytes) + if (ins.immediate.nbytes) { insn->immediate = ins.immediate.value; - else if (ins.displacement.nbytes) + insn->immediate_len = ins.immediate.nbytes; + } else if (ins.displacement.nbytes) { insn->immediate = ins.displacement.value; + insn->immediate_len = ins.displacement.nbytes; + } return 0; } +size_t arch_jump_opcode_bytes(struct objtool_file *file, struct instruction *insn, + unsigned char *buf) +{ + size_t len; + + len = insn->len - insn->immediate_len; + memcpy(buf, insn->sec->data->d_buf + insn->offset, len); + return len; +} + void arch_initial_func_cfi_state(struct cfi_init_state *state) { int i; diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c index b780df513715..118c3de2f293 100644 --- a/tools/objtool/builtin-check.c +++ b/tools/objtool/builtin-check.c @@ -73,8 +73,6 @@ static int parse_hacks(const struct option *opt, const char *str, int unset) static const struct option check_options[] = { OPT_GROUP("Actions:"), - OPT_BOOLEAN(0, "checksum", &opts.checksum, "generate per-function checksums"), - OPT_BOOLEAN(0, "cfi", &opts.cfi, "annotate kernel control flow integrity (kCFI) function preambles"), OPT_STRING_OPTARG('d', "disas", &opts.disas, "function-pattern", "disassemble functions", "*"), OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake", "patch toolchain bugs/limitations", parse_hacks), OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"), @@ -85,7 +83,7 @@ static const struct option check_options[] = { OPT_BOOLEAN('r', "retpoline", &opts.retpoline, "validate and annotate retpoline usage"), OPT_BOOLEAN(0, "rethunk", &opts.rethunk, "validate and annotate rethunk usage"), OPT_BOOLEAN(0, "unret", &opts.unret, "validate entry unret placement"), - OPT_INTEGER(0, "prefix", &opts.prefix, "generate prefix symbols"), + OPT_INTEGER(0, "prefix", &opts.prefix, "generate or grow prefix symbols for N-byte function padding"), OPT_BOOLEAN('l', "sls", &opts.sls, "validate straight-line-speculation mitigations"), OPT_BOOLEAN('s', "stackval", &opts.stackval, "validate frame pointer rules"), OPT_BOOLEAN('t', "static-call", &opts.static_call, "annotate static calls"), @@ -93,9 +91,10 @@ static const struct option check_options[] = { OPT_CALLBACK_OPTARG(0, "dump", NULL, NULL, "orc", "dump metadata", parse_dump), OPT_GROUP("Options:"), + OPT_BOOLEAN(0, "cfi", &opts.cfi, "grow kCFI preamble symbols (use with --prefix)"), + OPT_BOOLEAN(0, "fineibt", &opts.fineibt, "create .cfi_sites section for FineIBT"), OPT_BOOLEAN(0, "backtrace", &opts.backtrace, "unwind on error"), OPT_BOOLEAN(0, "backup", &opts.backup, "create backup (.orig) file on warning/error"), - OPT_STRING(0, "debug-checksum", &opts.debug_checksum, "funcs", "enable checksum debug output"), OPT_BOOLEAN(0, "dry-run", &opts.dryrun, "don't write modifications"), OPT_BOOLEAN(0, "link", &opts.link, "object is a linked object"), OPT_BOOLEAN(0, "module", &opts.module, "object is part of a kernel module"), @@ -165,20 +164,17 @@ static bool opts_valid(void) return false; } -#ifndef BUILD_KLP - if (opts.checksum) { - ERROR("--checksum not supported; install xxhash-devel/libxxhash-dev (version >= 0.8) and recompile"); - return false; - } -#endif - - if (opts.debug_checksum && !opts.checksum) { - ERROR("--debug-checksum requires --checksum"); + if (opts.cfi && !opts.prefix) { + ERROR("--cfi requires --prefix"); return false; } - if (opts.checksum || - opts.disas || + if (opts.fineibt && !opts.cfi) { + ERROR("--fineibt requires --cfi"); + return false; + } + + if (opts.disas || opts.hack_jump_label || opts.hack_noinstr || opts.ibt || diff --git a/tools/objtool/builtin-klp.c b/tools/objtool/builtin-klp.c index 56d5a5b92f72..58c3b9bda3eb 100644 --- a/tools/objtool/builtin-klp.c +++ b/tools/objtool/builtin-klp.c @@ -13,6 +13,7 @@ struct subcmd { }; static struct subcmd subcmds[] = { + { "checksum", "Generate per-function checksums", cmd_klp_checksum, }, { "diff", "Generate binary diff of two object files", cmd_klp_diff, }, { "post-link", "Finalize klp symbols/relocs after module linking", cmd_klp_post_link, }, }; diff --git a/tools/objtool/check.c b/tools/objtool/check.c index 9b11cf3193b9..10b18cf9c360 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -18,7 +18,6 @@ #include #include #include -#include #include #include @@ -64,8 +63,8 @@ struct instruction *next_insn_same_sec(struct objtool_file *file, return insn; } -static struct instruction *next_insn_same_func(struct objtool_file *file, - struct instruction *insn) +struct instruction *next_insn_same_func(struct objtool_file *file, + struct instruction *insn) { struct instruction *next = next_insn_same_sec(file, insn); struct symbol *func = insn_func(insn); @@ -113,10 +112,6 @@ static struct instruction *prev_insn_same_sym(struct objtool_file *file, for_each_sec(file->elf, __sec) \ sec_for_each_insn(file, __sec, insn) -#define func_for_each_insn(file, func, insn) \ - for (insn = find_insn(file, func->sec, func->offset); \ - insn; \ - insn = next_insn_same_func(file, insn)) #define sym_for_each_insn(file, sym, insn) \ for (insn = find_insn(file, sym->sec, sym->offset); \ @@ -491,7 +486,7 @@ static int decode_instructions(struct objtool_file *file) return -1; } - if (func->embedded_insn || func->alias != func) + if (func->embedded_insn || is_alias_sym(func)) continue; if (!find_insn(file, sec, func->offset)) { @@ -500,7 +495,7 @@ static int decode_instructions(struct objtool_file *file) } sym_for_each_insn(file, func, insn) { - insn->sym = func; + insn->_sym = func; if (is_func_sym(func) && insn->type == INSN_ENDBR && list_empty(&insn->call_node)) { @@ -864,15 +859,14 @@ static int create_ibt_endbr_seal_sections(struct objtool_file *file) list_for_each_entry(insn, &file->endbr_list, call_node) { int *site = (int *)sec->data->d_buf + idx; - struct symbol *sym = insn->sym; + struct symbol *func = insn_func(insn); *site = 0; - if (opts.module && sym && is_func_sym(sym) && - insn->offset == sym->offset && - (!strcmp(sym->name, "init_module") || - !strcmp(sym->name, "cleanup_module"))) { + if (opts.module && func && insn->offset == func->offset && + (!strcmp(func->name, "init_module") || + !strcmp(func->name, "cleanup_module"))) { ERROR("%s(): Magic init_module() function name is deprecated, use module_init(fn) instead", - sym->name); + func->name); return -1; } @@ -887,6 +881,31 @@ static int create_ibt_endbr_seal_sections(struct objtool_file *file) return 0; } +/* +* Grow __cfi_ symbols to fill the NOP gap between the 'mov , %rax' and +* the start of the function. +*/ +static int grow_cfi_symbols(struct objtool_file *file) +{ + struct symbol *sym; + + for_each_sym(file->elf, sym) { + if (!is_func_sym(sym) || !strstarts(sym->name, "__cfi_") || + sym->len != 5) + continue; + + if (!find_func_by_offset(sym->sec, sym->offset + sym->len + opts.prefix)) + continue; + + sym->len += opts.prefix; + sym->sym.st_size = sym->len; + if (elf_write_symbol(file->elf, sym)) + return -1; + } + + return 0; +} + static int create_cfi_sections(struct objtool_file *file) { struct section *sec; @@ -1023,59 +1042,6 @@ static int create_direct_call_sections(struct objtool_file *file) return 0; } -#ifdef BUILD_KLP -static int create_sym_checksum_section(struct objtool_file *file) -{ - struct section *sec; - struct symbol *sym; - unsigned int idx = 0; - struct sym_checksum *checksum; - size_t entsize = sizeof(struct sym_checksum); - - sec = find_section_by_name(file->elf, ".discard.sym_checksum"); - if (sec) { - if (!opts.dryrun) - WARN("file already has .discard.sym_checksum section, skipping"); - - return 0; - } - - for_each_sym(file->elf, sym) - if (sym->csum.checksum) - idx++; - - if (!idx) - return 0; - - sec = elf_create_section_pair(file->elf, ".discard.sym_checksum", entsize, - idx, idx); - if (!sec) - return -1; - - idx = 0; - for_each_sym(file->elf, sym) { - if (!sym->csum.checksum) - continue; - - if (!elf_init_reloc(file->elf, sec->rsec, idx, idx * entsize, - sym, 0, R_TEXT64)) - return -1; - - checksum = (struct sym_checksum *)sec->data->d_buf + idx; - checksum->addr = 0; /* reloc */ - checksum->checksum = sym->csum.checksum; - - mark_sec_changed(file->elf, sec, true); - - idx++; - } - - return 0; -} -#else -static int create_sym_checksum_section(struct objtool_file *file) { return -EINVAL; } -#endif - /* * Warnings shouldn't be reported for ignored functions. */ @@ -1349,14 +1315,11 @@ __weak bool arch_is_embedded_insn(struct symbol *sym) return false; } -static struct reloc *insn_reloc(struct objtool_file *file, struct instruction *insn) +struct reloc *insn_reloc(struct objtool_file *file, struct instruction *insn) { struct reloc *reloc; - if (insn->no_reloc) - return NULL; - - if (!file) + if (!file || insn->no_reloc || insn->fake) return NULL; reloc = find_reloc_by_dest_range(file->elf, insn->sec, @@ -1642,7 +1605,7 @@ static int add_jump_destinations(struct objtool_file *file) } if (!dest_sym || is_sec_sym(dest_sym)) { - dest_sym = dest_insn->sym; + dest_sym = insn_sym(dest_insn); if (!dest_sym) goto set_jump_dest; } @@ -1658,7 +1621,7 @@ static int add_jump_destinations(struct objtool_file *file) continue; } - if (!insn->sym || insn->sym->pfunc == dest_sym->pfunc) + if (!insn_sym(insn) || insn_sym(insn)->pfunc == dest_sym->pfunc) goto set_jump_dest; /* @@ -1831,7 +1794,6 @@ static int handle_group_alt(struct objtool_file *file, nop->offset = special_alt->new_off + special_alt->new_len; nop->len = special_alt->orig_len - special_alt->new_len; nop->type = INSN_NOP; - nop->sym = orig_insn->sym; nop->alt_group = new_alt_group; nop->fake = 1; } @@ -1850,7 +1812,6 @@ static int handle_group_alt(struct objtool_file *file, last_new_insn = insn; - insn->sym = orig_insn->sym; insn->alt_group = new_alt_group; /* @@ -2232,7 +2193,7 @@ static int add_jump_table_alts(struct objtool_file *file) return 0; for_each_sym(file->elf, func) { - if (!is_func_sym(func) || func->alias != func) + if (!is_func_sym(func) || is_alias_sym(func)) continue; mark_func_jump_tables(file, func); @@ -2493,12 +2454,12 @@ static int __annotate_late(struct objtool_file *file, int type, struct instructi break; case ANNOTYPE_NOCFI: - sym = insn->sym; + sym = insn_sym(insn); if (!sym) { ERROR_INSN(insn, "dodgy NOCFI annotation"); return -1; } - insn->sym->nocfi = 1; + sym->nocfi = 1; break; default: @@ -2566,7 +2527,6 @@ static int classify_symbols(struct objtool_file *file) static void mark_rodata(struct objtool_file *file) { struct section *sec; - bool found = false; /* * Search for the following rodata sections, each of which can @@ -2579,15 +2539,11 @@ static void mark_rodata(struct objtool_file *file) * .rodata.str1.* sections are ignored; they don't contain jump tables. */ for_each_sec(file->elf, sec) { - if ((!strncmp(sec->name, ".rodata", 7) && - !strstr(sec->name, ".str1.")) || - !strncmp(sec->name, ".data.rel.ro", 12)) { - sec->rodata = true; - found = true; + if (is_rodata_sec(sec)) { + file->rodata = true; + return; } } - - file->rodata = found; } static void mark_holes(struct objtool_file *file) @@ -2604,7 +2560,7 @@ static void mark_holes(struct objtool_file *file) * favour of a regular symbol, but leaves the code in place. */ for_each_insn(file, insn) { - if (insn->sym || !find_symbol_hole_containing(insn->sec, insn->offset)) { + if (insn_sym(insn) || !find_symbol_hole_containing(insn->sec, insn->offset)) { in_hole = false; continue; } @@ -2622,7 +2578,7 @@ static void mark_holes(struct objtool_file *file) if (insn->jump_dest) { struct symbol *dest_func = insn_func(insn->jump_dest); - if (dest_func && dest_func->cold) + if (dest_func && is_cold_func(dest_func)) dest_func->ignore = true; } } @@ -2630,14 +2586,35 @@ static void mark_holes(struct objtool_file *file) static bool validate_branch_enabled(void) { - return opts.stackval || - opts.orc || - opts.uaccess || + return opts.stackval || + opts.orc || + opts.uaccess; +} + +static bool alts_needed(void) +{ + return validate_branch_enabled() || + opts.noinstr || + opts.hack_jump_label || + opts.disas || opts.checksum; } -static int decode_sections(struct objtool_file *file) +int decode_file(struct objtool_file *file) { + arch_initial_func_cfi_state(&initial_func_cfi); + init_cfi_state(&init_cfi); + init_cfi_state(&func_cfi); + set_func_state(&func_cfi); + init_cfi_state(&force_undefined_cfi); + force_undefined_cfi.force_undefined = true; + + if (!cfi_hash_alloc(1UL << (file->elf->symbol_bits - 3))) + return -1; + + cfi_hash_add(&init_cfi); + cfi_hash_add(&func_cfi); + file->klp = is_livepatch_module(file); mark_rodata(file); @@ -2666,7 +2643,7 @@ static int decode_sections(struct objtool_file *file) * Must be before add_jump_destinations(), which depends on 'func' * being set for alternatives, to enable proper sibling call detection. */ - if (validate_branch_enabled() || opts.noinstr || opts.hack_jump_label || opts.disas) { + if (alts_needed()) { if (add_special_section_alts(file)) return -1; } @@ -3027,7 +3004,7 @@ static int update_cfi_state(struct instruction *insn, } if (op->dest.reg == CFI_BP && op->src.reg == CFI_SP && - insn->sym->frame_pointer) { + insn_sym(insn)->frame_pointer) { /* addi.d fp,sp,imm on LoongArch */ if (cfa->base == CFI_SP && cfa->offset == op->src.offset) { cfa->base = CFI_BP; @@ -3039,7 +3016,7 @@ static int update_cfi_state(struct instruction *insn, if (op->dest.reg == CFI_SP && op->src.reg == CFI_BP) { /* addi.d sp,fp,imm on LoongArch */ if (cfa->base == CFI_BP && cfa->offset == 0) { - if (insn->sym->frame_pointer) { + if (insn_sym(insn)->frame_pointer) { cfa->base = CFI_SP; cfa->offset = -op->src.offset; } @@ -3662,88 +3639,6 @@ static bool skip_alt_group(struct instruction *insn) return alt_insn->type == INSN_CLAC || alt_insn->type == INSN_STAC; } -static int checksum_debug_init(struct objtool_file *file) -{ - char *dup, *s; - - if (!opts.debug_checksum) - return 0; - - dup = strdup(opts.debug_checksum); - if (!dup) { - ERROR_GLIBC("strdup"); - return -1; - } - - s = dup; - while (*s) { - struct symbol *func; - char *comma; - - comma = strchr(s, ','); - if (comma) - *comma = '\0'; - - func = find_symbol_by_name(file->elf, s); - if (!func || !is_func_sym(func)) - WARN("--debug-checksum: can't find '%s'", s); - else - func->debug_checksum = 1; - - if (!comma) - break; - - s = comma + 1; - } - - free(dup); - return 0; -} - -static void checksum_update_insn(struct objtool_file *file, struct symbol *func, - struct instruction *insn) -{ - struct reloc *reloc = insn_reloc(file, insn); - unsigned long offset; - struct symbol *sym; - - if (insn->fake) - return; - - checksum_update(func, insn, insn->sec->data->d_buf + insn->offset, insn->len); - - if (!reloc) { - struct symbol *call_dest = insn_call_dest(insn); - - if (call_dest) - checksum_update(func, insn, call_dest->demangled_name, - strlen(call_dest->demangled_name)); - return; - } - - sym = reloc->sym; - offset = arch_insn_adjusted_addend(insn, reloc); - - if (is_string_sec(sym->sec)) { - char *str; - - str = sym->sec->data->d_buf + sym->offset + offset; - checksum_update(func, insn, str, strlen(str)); - return; - } - - if (is_sec_sym(sym)) { - sym = find_symbol_containing(reloc->sym->sec, offset); - if (!sym) - return; - - offset -= sym->offset; - } - - checksum_update(func, insn, sym->demangled_name, strlen(sym->demangled_name)); - checksum_update(func, insn, &offset, sizeof(offset)); -} - static int validate_branch(struct objtool_file *file, struct symbol *func, struct instruction *insn, struct insn_state state); static int do_validate_branch(struct objtool_file *file, struct symbol *func, @@ -4025,9 +3920,6 @@ static int do_validate_branch(struct objtool_file *file, struct symbol *func, insn->trace = 0; next_insn = next_insn_to_validate(file, insn); - if (opts.checksum && func && insn->sec) - checksum_update_insn(file, func, insn); - if (func && insn_func(insn) && func != insn_func(insn)->pfunc) { /* Ignore KCFI type preambles, which always fall through */ if (is_prefix_func(func)) @@ -4093,9 +3985,6 @@ static int validate_unwind_hint(struct objtool_file *file, struct symbol *func = insn_func(insn); int ret; - if (opts.checksum) - checksum_init(func); - ret = validate_branch(file, func, insn, *state); if (ret) BT_INSN(insn, "<=== (hint)"); @@ -4304,7 +4193,7 @@ static int validate_retpoline(struct objtool_file *file) * broken. */ list_for_each_entry(insn, &file->retpoline_call_list, call_node) { - struct symbol *sym = insn->sym; + struct symbol *sym = insn_sym(insn); if (sym && (is_notype_sym(sym) || is_func_sym(sym)) && !sym->nocfi) { @@ -4407,17 +4296,6 @@ static bool ignore_unreachable_insn(struct objtool_file *file, struct instructio * For FineIBT or kCFI, a certain number of bytes preceding the function may be * NOPs. Those NOPs may be rewritten at runtime and executed, so give them a * proper function name: __pfx_. - * - * The NOPs may not exist for the following cases: - * - * - compiler cloned functions (*.cold, *.part0, etc) - * - asm functions created with inline asm or without SYM_FUNC_START() - * - * Also, the function may already have a prefix from a previous objtool run - * (livepatch extracted functions, or manually running objtool multiple times). - * - * So return 0 if the NOPs are missing or the function already has a prefix - * symbol. */ static int create_prefix_symbol(struct objtool_file *file, struct symbol *func) { @@ -4425,10 +4303,6 @@ static int create_prefix_symbol(struct objtool_file *file, struct symbol *func) char name[SYM_NAME_LEN]; struct cfi_state *cfi; - if (!is_func_sym(func) || is_prefix_func(func) || - func->cold || func->static_call_tramp) - return 0; - if ((strlen(func->name) + sizeof("__pfx_") > SYM_NAME_LEN)) { WARN("%s: symbol name too long, can't create __pfx_ symbol", func->name); @@ -4438,59 +4312,21 @@ static int create_prefix_symbol(struct objtool_file *file, struct symbol *func) if (snprintf_check(name, SYM_NAME_LEN, "__pfx_%s", func->name)) return -1; - if (file->klp) { - struct symbol *pfx; - - pfx = find_symbol_by_offset(func->sec, func->offset - opts.prefix); - if (pfx && is_prefix_func(pfx) && !strcmp(pfx->name, name)) - return 0; - } - - insn = find_insn(file, func->sec, func->offset); - if (!insn) { - WARN("%s: can't find starting instruction", func->name); + if (!elf_create_symbol(file->elf, name, func->sec, + GELF_ST_BIND(func->sym.st_info), + GELF_ST_TYPE(func->sym.st_info), + func->offset - opts.prefix, opts.prefix)) return -1; - } - - for (prev = prev_insn_same_sec(file, insn); - prev; - prev = prev_insn_same_sec(file, prev)) { - u64 offset; - - if (prev->type != INSN_NOP) - return 0; - - offset = func->offset - prev->offset; - - if (offset > opts.prefix) - return 0; - - if (offset < opts.prefix) - continue; - - if (!elf_create_symbol(file->elf, name, func->sec, - GELF_ST_BIND(func->sym.st_info), - GELF_ST_TYPE(func->sym.st_info), - prev->offset, opts.prefix)) - return -1; - - break; - } - - if (!prev) - return 0; - - if (!insn->cfi) { - /* - * This can happen if stack validation isn't enabled or the - * function is annotated with STACK_FRAME_NON_STANDARD. - */ - return 0; - } /* Propagate insn->cfi to the prefix code */ + insn = find_insn(file, func->sec, func->offset); + if (!insn || !insn->cfi) + return 0; + cfi = cfi_hash_find_or_add(insn->cfi); - for (; prev != insn; prev = next_insn_same_sec(file, prev)) + for (prev = find_insn(file, func->sec, func->offset - opts.prefix); + prev && prev != insn; + prev = next_insn_same_sec(file, prev)) prev->cfi = cfi; return 0; @@ -4498,15 +4334,20 @@ static int create_prefix_symbol(struct objtool_file *file, struct symbol *func) static int create_prefix_symbols(struct objtool_file *file) { - struct section *sec; + struct section *pfe_sec; struct symbol *func; + struct reloc *reloc; - for_each_sec(file->elf, sec) { - if (!is_text_sec(sec)) + for_each_sec(file->elf, pfe_sec) { + if (strcmp(pfe_sec->name, "__patchable_function_entries")) + continue; + if (!pfe_sec->rsec) continue; - sec_for_each_sym(sec, func) { - if (create_prefix_symbol(file, func)) + for_each_reloc(pfe_sec->rsec, reloc) { + func = find_func_by_offset(reloc->sym->sec, + reloc->sym->offset + reloc_addend(reloc) + opts.prefix); + if (func && create_prefix_symbol(file, func)) return -1; } } @@ -4526,7 +4367,7 @@ static int validate_symbol(struct objtool_file *file, struct section *sec, return 1; } - if (sym->pfunc != sym || sym->alias != sym) + if (sym->pfunc != sym || is_alias_sym(sym)) return 0; insn = find_insn(file, sec, sym->offset); @@ -4538,9 +4379,6 @@ static int validate_symbol(struct objtool_file *file, struct section *sec, func = insn_func(insn); - if (opts.checksum) - checksum_init(func); - if (opts.trace && !fnmatch(opts.trace, sym->name, 0)) { trace_enable(); TRACE("%s: validation begin\n", sym->name); @@ -4553,9 +4391,6 @@ static int validate_symbol(struct objtool_file *file, struct section *sec, TRACE("%s: validation %s\n\n", sym->name, ret ? "failed" : "end"); trace_disable(); - if (opts.checksum) - checksum_finish(func); - return ret; } @@ -4948,7 +4783,7 @@ struct insn_chunk { * which can trigger more allocations for .debug_* sections whose data hasn't * been read yet. */ -static void free_insns(struct objtool_file *file) +void free_insns(struct objtool_file *file) { struct instruction *insn; struct insn_chunk *chunks = NULL, *chunk; @@ -4995,26 +4830,7 @@ int check(struct objtool_file *file) objtool_disas_ctx = disas_ctx; } - arch_initial_func_cfi_state(&initial_func_cfi); - init_cfi_state(&init_cfi); - init_cfi_state(&func_cfi); - set_func_state(&func_cfi); - init_cfi_state(&force_undefined_cfi); - force_undefined_cfi.force_undefined = true; - - if (!cfi_hash_alloc(1UL << (file->elf->symbol_bits - 3))) { - ret = -1; - goto out; - } - - cfi_hash_add(&init_cfi); - cfi_hash_add(&func_cfi); - - ret = checksum_debug_init(file); - if (ret) - goto out; - - ret = decode_sections(file); + ret = decode_file(file); if (ret) goto out; @@ -5064,12 +4880,6 @@ int check(struct objtool_file *file) goto out; } - if (opts.cfi) { - ret = create_cfi_sections(file); - if (ret) - goto out; - } - if (opts.rethunk) { ret = create_return_sites_sections(file); if (ret) @@ -5089,9 +4899,21 @@ int check(struct objtool_file *file) } if (opts.prefix) { - ret = create_prefix_symbols(file); - if (ret) - goto out; + if (!opts.cfi) { + ret = create_prefix_symbols(file); + if (ret) + goto out; + } else { + ret = grow_cfi_symbols(file); + if (ret) + goto out; + + if (opts.fineibt) { + ret = create_cfi_sections(file); + if (ret) + goto out; + } + } } if (opts.ibt) { @@ -5103,12 +4925,6 @@ int check(struct objtool_file *file) if (opts.noabs) warnings += check_abs_references(file); - if (opts.checksum) { - ret = create_sym_checksum_section(file); - if (ret) - goto out; - } - if (opts.orc && nr_insns) { ret = orc_create(file); if (ret) diff --git a/tools/objtool/disas.c b/tools/objtool/disas.c index 59090234af19..e6a54a83605c 100644 --- a/tools/objtool/disas.c +++ b/tools/objtool/disas.c @@ -210,7 +210,7 @@ static bool disas_print_addr_alt(bfd_vma addr, struct disassemble_info *dinfo) offset = addr - alt_group->first_insn->offset; addr = orig_first_insn->offset + offset; - sym = orig_first_insn->sym; + sym = insn_sym(orig_first_insn); disas_print_addr_sym(orig_first_insn->sec, sym, addr, dinfo); @@ -222,15 +222,13 @@ static void disas_print_addr_noreloc(bfd_vma addr, { struct disas_context *dctx = dinfo->application_data; struct instruction *insn = dctx->insn; - struct symbol *sym = NULL; + struct symbol *sym = insn_sym(insn); if (disas_print_addr_alt(addr, dinfo)) return; - if (insn->sym && addr >= insn->sym->offset && - addr < insn->sym->offset + insn->sym->len) { - sym = insn->sym; - } + if (sym && (addr < sym->offset || addr >= sym->offset + sym->len)) + sym = NULL; disas_print_addr_sym(insn->sec, sym, addr, dinfo); } @@ -291,9 +289,9 @@ static void disas_print_address(bfd_vma addr, struct disassemble_info *dinfo) * up. So check it first. */ jump_dest = insn->jump_dest; - if (jump_dest && jump_dest->sym && jump_dest->offset == addr) { + if (jump_dest && insn_sym(jump_dest) && jump_dest->offset == addr) { if (!disas_print_addr_alt(addr, dinfo)) - disas_print_addr_sym(jump_dest->sec, jump_dest->sym, + disas_print_addr_sym(jump_dest->sec, insn_sym(jump_dest), addr, dinfo); return; } @@ -768,8 +766,8 @@ static int disas_alt_jump(struct disas_alt *dalt) if (orig_insn->len == 5) suffix[0] = 'q'; str = strfmt("jmp%-3s %lx <%s+0x%lx>", suffix, - dest_insn->offset, dest_insn->sym->name, - dest_insn->offset - dest_insn->sym->offset); + dest_insn->offset, insn_sym(dest_insn)->name, + dest_insn->offset - insn_sym(dest_insn)->offset); nops = 0; } else { str = strfmt("nop%d", orig_insn->len); @@ -794,8 +792,8 @@ static int disas_alt_extable(struct disas_alt *dalt) alt_insn = dalt->alt->insn; str = strfmt("resume at 0x%lx <%s+0x%lx>", - alt_insn->offset, alt_insn->sym->name, - alt_insn->offset - alt_insn->sym->offset); + alt_insn->offset, insn_sym(alt_insn)->name, + alt_insn->offset - insn_sym(alt_insn)->offset); if (!str) return -1; diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c index f3df2bde119f..33c95a74a51b 100644 --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -27,27 +27,16 @@ static ssize_t demangled_name_len(const char *name); -static inline u32 str_hash(const char *str) -{ - return jhash(str, strlen(str), 0); -} - -static inline u32 str_hash_demangled(const char *str) +u32 str_hash_demangled(const char *str) { return jhash(str, demangled_name_len(str), 0); } -#define __elf_table(name) (elf->name##_hash) -#define __elf_bits(name) (elf->name##_bits) - -#define __elf_table_entry(name, key) \ - __elf_table(name)[hash_min(key, __elf_bits(name))] - #define elf_hash_add(name, node, key) \ ({ \ struct elf_hash_node *__node = node; \ - __node->next = __elf_table_entry(name, key); \ - __elf_table_entry(name, key) = __node; \ + __node->next = __elf_table_entry(elf, name, key); \ + __elf_table_entry(elf, name, key) = __node; \ }) static inline void __elf_hash_del(struct elf_hash_node *node, @@ -69,30 +58,20 @@ static inline void __elf_hash_del(struct elf_hash_node *node, } #define elf_hash_del(name, node, key) \ - __elf_hash_del(node, &__elf_table_entry(name, key)) - -#define elf_list_entry(ptr, type, member) \ -({ \ - typeof(ptr) __ptr = (ptr); \ - __ptr ? container_of(__ptr, type, member) : NULL; \ -}) - -#define elf_hash_for_each_possible(name, obj, member, key) \ - for (obj = elf_list_entry(__elf_table_entry(name, key), typeof(*obj), member); \ - obj; \ - obj = elf_list_entry(obj->member.next, typeof(*(obj)), member)) + __elf_hash_del(node, &__elf_table_entry(elf, name, key)) #define elf_alloc_hash(name, size) \ ({ \ - __elf_bits(name) = max(10, ilog2(size)); \ - __elf_table(name) = mmap(NULL, sizeof(struct elf_hash_node *) << __elf_bits(name), \ + __elf_bits(elf, name) = max(10, ilog2(size)); \ + __elf_table(elf, name) = mmap(NULL, \ + sizeof(struct elf_hash_node *) << __elf_bits(elf, name), \ PROT_READ|PROT_WRITE, \ MAP_PRIVATE|MAP_ANON, -1, 0); \ - if (__elf_table(name) == (void *)-1L) { \ + if (__elf_table(elf, name) == (void *)-1L) { \ ERROR_GLIBC("mmap fail " #name); \ - __elf_table(name) = NULL; \ + __elf_table(elf, name) = NULL; \ } \ - __elf_table(name); \ + __elf_table(elf, name); \ }) static inline unsigned long __sym_start(struct symbol *s) @@ -141,7 +120,7 @@ struct section *find_section_by_name(const struct elf *elf, const char *name) { struct section *sec; - elf_hash_for_each_possible(section_name, sec, name_hash, str_hash(name)) { + elf_hash_for_each_possible(elf, section_name, sec, name_hash, str_hash(name)) { if (!strcmp(sec->name, name)) return sec; } @@ -154,7 +133,7 @@ static struct section *find_section_by_index(struct elf *elf, { struct section *sec; - elf_hash_for_each_possible(section, sec, hash, idx) { + elf_hash_for_each_possible(elf, section, sec, hash, idx) { if (sec->idx == idx) return sec; } @@ -166,7 +145,7 @@ static struct symbol *find_symbol_by_index(struct elf *elf, unsigned int idx) { struct symbol *sym; - elf_hash_for_each_possible(symbol, sym, hash, idx) { + elf_hash_for_each_possible(elf, symbol, sym, hash, idx) { if (sym->idx == idx) return sym; } @@ -228,6 +207,20 @@ struct symbol *find_symbol_containing(const struct section *sec, unsigned long o return sym ? sym->alias : NULL; } +/* + * Also match the symbol end address which can be used for a bounds comparison. + */ +struct symbol *find_symbol_containing_inclusive(const struct section *sec, + unsigned long offset) +{ + struct symbol *sym = find_symbol_containing(sec, offset); + + if (!sym && offset) + sym = find_symbol_containing(sec, offset - 1); + + return sym; +} + /* * Returns size of hole starting at @offset. */ @@ -285,7 +278,7 @@ struct symbol *find_symbol_by_name(const struct elf *elf, const char *name) { struct symbol *sym; - elf_hash_for_each_possible(symbol_name, sym, name_hash, str_hash(name)) { + elf_hash_for_each_possible(elf, symbol_name, sym, name_hash, str_hash(name)) { if (!strcmp(sym->name, name)) return sym; } @@ -300,7 +293,7 @@ static struct symbol *find_local_symbol_by_file_and_name(const struct elf *elf, { struct symbol *sym; - elf_hash_for_each_possible(symbol_name, sym, name_hash, str_hash_demangled(name)) { + elf_hash_for_each_possible(elf, symbol_name, sym, name_hash, str_hash_demangled(name)) { if (sym->bind == STB_LOCAL && sym->file == file && !strcmp(sym->name, name)) { return sym; @@ -314,7 +307,7 @@ struct symbol *find_global_symbol_by_name(const struct elf *elf, const char *nam { struct symbol *sym; - elf_hash_for_each_possible(symbol_name, sym, name_hash, str_hash_demangled(name)) { + elf_hash_for_each_possible(elf, symbol_name, sym, name_hash, str_hash_demangled(name)) { if (!strcmp(sym->name, name) && !is_local_sym(sym)) return sym; } @@ -322,21 +315,9 @@ struct symbol *find_global_symbol_by_name(const struct elf *elf, const char *nam return NULL; } -void iterate_global_symbol_by_demangled_name(const struct elf *elf, - const char *demangled_name, - void (*process)(struct symbol *sym, void *data), - void *data) -{ - struct symbol *sym; - - elf_hash_for_each_possible(symbol_name, sym, name_hash, str_hash(demangled_name)) { - if (!strcmp(sym->demangled_name, demangled_name) && !is_local_sym(sym)) - process(sym, data); - } -} - +/* If there are multiple matches, return the first one in the range */ struct reloc *find_reloc_by_dest_range(const struct elf *elf, struct section *sec, - unsigned long offset, unsigned int len) + unsigned long offset, unsigned int len) { struct reloc *reloc, *r = NULL; struct section *rsec; @@ -347,7 +328,7 @@ struct reloc *find_reloc_by_dest_range(const struct elf *elf, struct section *se return NULL; for_offset_range(o, offset, offset + len) { - elf_hash_for_each_possible(reloc, reloc, hash, + elf_hash_for_each_possible(elf, reloc, reloc, hash, sec_offset_hash(rsec, o)) { if (reloc->sec != rsec) continue; @@ -358,11 +339,11 @@ struct reloc *find_reloc_by_dest_range(const struct elf *elf, struct section *se r = reloc; } } - if (r) + if (r && (reloc_offset(r) & OFFSET_STRIDE_MASK) == o) return r; } - return NULL; + return r; } struct reloc *find_reloc_by_dest(const struct elf *elf, struct section *sec, unsigned long offset) @@ -668,7 +649,7 @@ static int read_symbols(struct elf *elf) if (is_file_sym(sym)) file = sym; - else if (sym->bind == STB_LOCAL) + else if (sym->bind == STB_LOCAL && !is_sec_sym(sym)) sym->file = file; } @@ -1016,6 +997,26 @@ non_local: return sym; } +int elf_write_symbol(struct elf *elf, struct symbol *sym) +{ + struct section *symtab, *symtab_shndx; + + symtab = find_section_by_name(elf, ".symtab"); + if (!symtab) { + ERROR("no .symtab"); + return -1; + } + + symtab_shndx = find_section_by_name(elf, ".symtab_shndx"); + + if (elf_update_symbol(elf, symtab, symtab_shndx, sym)) + return -1; + + mark_sec_changed(elf, symtab, true); + + return 0; +} + struct symbol *elf_create_section_symbol(struct elf *elf, struct section *sec) { struct symbol *sym = calloc(1, sizeof(*sym)); @@ -1172,6 +1173,17 @@ static int read_relocs(struct elf *elf) return 0; } +static void mark_rodata(struct elf *elf) +{ + struct section *sec; + + for_each_sec(elf, sec) { + if ((strstarts(sec->name, ".rodata") && !strstr(sec->name, ".str1.")) || + strstarts(sec->name, ".data.rel.ro")) + sec->rodata = true; + } +} + struct elf *elf_open_read(const char *name, int flags) { struct elf *elf; @@ -1222,6 +1234,8 @@ struct elf *elf_open_read(const char *name, int flags) if (read_sections(elf)) goto err; + mark_rodata(elf); + if (read_symbols(elf)) goto err; diff --git a/tools/objtool/include/objtool/arch.h b/tools/objtool/include/objtool/arch.h index 8866158975fc..96d828a8401f 100644 --- a/tools/objtool/include/objtool/arch.h +++ b/tools/objtool/include/objtool/arch.h @@ -79,6 +79,9 @@ int arch_decode_instruction(struct objtool_file *file, const struct section *sec unsigned long offset, unsigned int maxlen, struct instruction *insn); +size_t arch_jump_opcode_bytes(struct objtool_file *file, struct instruction *insn, + unsigned char *buf); + bool arch_callee_saved_reg(unsigned char reg); unsigned long arch_jump_destination(struct instruction *insn); diff --git a/tools/objtool/include/objtool/builtin.h b/tools/objtool/include/objtool/builtin.h index b9e229ed4dc0..e844e9c82b7b 100644 --- a/tools/objtool/include/objtool/builtin.h +++ b/tools/objtool/include/objtool/builtin.h @@ -9,8 +9,8 @@ struct opts { /* actions: */ - bool cfi; bool checksum; + const char *disas; bool dump_orc; bool hack_jump_label; bool hack_noinstr; @@ -20,6 +20,7 @@ struct opts { bool noabs; bool noinstr; bool orc; + int prefix; bool retpoline; bool rethunk; bool unret; @@ -27,14 +28,14 @@ struct opts { bool stackval; bool static_call; bool uaccess; - int prefix; - const char *disas; /* options: */ bool backtrace; bool backup; + bool cfi; const char *debug_checksum; bool dryrun; + bool fineibt; bool link; bool mnop; bool module; diff --git a/tools/objtool/include/objtool/check.h b/tools/objtool/include/objtool/check.h index 5f2f77bd9b41..063f5985fecd 100644 --- a/tools/objtool/include/objtool/check.h +++ b/tools/objtool/include/objtool/check.h @@ -68,6 +68,7 @@ struct instruction { s8 instr; u32 idx : INSN_CHUNK_BITS, + immediate_len : 4, dead_end : 1, ignore_alts : 1, hint : 1, @@ -81,7 +82,7 @@ struct instruction { hole : 1, fake : 1, trace : 1; - /* 9 bit hole */ + /* 4 bit hole */ struct alt_group *alt_group; struct instruction *jump_dest; @@ -94,14 +95,30 @@ struct instruction { }; }; struct alternative *alts; - struct symbol *sym; + struct symbol *_sym; struct stack_op *stack_ops; struct cfi_state *cfi; }; +/* + * Return the symbol associated with an instruction. For alternative + * replacements, return the symbol of the original code being replaced rather + * than NULL. insn->_sym reflects the actual location in the ELF file. + */ +static inline struct symbol *insn_sym(struct instruction *insn) +{ + struct symbol *sym = insn->_sym; + + if ((!sym || !is_func_sym(sym)) && + insn->alt_group && insn->alt_group->orig_group) + sym = insn->alt_group->orig_group->first_insn->_sym; + + return sym; +} + static inline struct symbol *insn_func(struct instruction *insn) { - struct symbol *sym = insn->sym; + struct symbol *sym = insn_sym(insn); if (sym && sym->type != STT_FUNC) sym = NULL; @@ -144,6 +161,12 @@ struct instruction *find_insn(struct objtool_file *file, struct section *sec, unsigned long offset); struct instruction *next_insn_same_sec(struct objtool_file *file, struct instruction *insn); +struct instruction *next_insn_same_func(struct objtool_file *file, struct instruction *insn); + +#define func_for_each_insn(file, func, insn) \ + for (insn = find_insn(file, func->sec, func->offset); \ + insn; \ + insn = next_insn_same_func(file, insn)) #define sec_for_each_insn(file, _sec, insn) \ for (insn = find_insn(file, _sec, 0); \ @@ -155,6 +178,11 @@ struct instruction *next_insn_same_sec(struct objtool_file *file, struct instruc insn && insn->offset < sym->offset + sym->len; \ insn = next_insn_same_sec(file, insn)) +struct reloc *insn_reloc(struct objtool_file *file, struct instruction *insn); + +int decode_file(struct objtool_file *file); +void free_insns(struct objtool_file *file); + const char *objtool_disas_insn(struct instruction *insn); extern size_t sym_name_max_len; diff --git a/tools/objtool/include/objtool/checksum.h b/tools/objtool/include/objtool/checksum.h index 7fe21608722a..d46293f54716 100644 --- a/tools/objtool/include/objtool/checksum.h +++ b/tools/objtool/include/objtool/checksum.h @@ -6,37 +6,54 @@ #ifdef BUILD_KLP -static inline void checksum_init(struct symbol *func) +static inline void checksum_init(struct symbol *sym) { - if (func && !func->csum.state) { - func->csum.state = XXH3_createState(); - XXH3_64bits_reset(func->csum.state); + if (sym && !sym->csum.state) { + sym->csum.state = XXH3_createState(); + XXH3_64bits_reset(sym->csum.state); } } -static inline void checksum_update(struct symbol *func, - struct instruction *insn, - const void *data, size_t size) +static inline void __checksum_update(struct symbol *sym, const void *data, + size_t size) { - XXH3_64bits_update(func->csum.state, data, size); - dbg_checksum(func, insn, XXH3_64bits_digest(func->csum.state)); + XXH3_64bits_update(sym->csum.state, data, size); } -static inline void checksum_finish(struct symbol *func) +static inline void __checksum_update_insn(struct symbol *sym, + struct instruction *insn, + const void *data, size_t size) { - if (func && func->csum.state) { - func->csum.checksum = XXH3_64bits_digest(func->csum.state); - func->csum.state = NULL; + __checksum_update(sym, data, size); + dbg_checksum_insn(sym, insn, XXH3_64bits_digest(sym->csum.state)); +} + +static inline void __checksum_update_object(struct symbol *sym, + unsigned long offset, + const char *what, const void *data, + size_t size) +{ + __checksum_update(sym, &offset, sizeof(offset)); + __checksum_update(sym, data, size); + dbg_checksum_object(sym, offset, what, XXH3_64bits_digest(sym->csum.state)); +} + +static inline void checksum_finish(struct symbol *sym) +{ + if (sym && sym->csum.state) { + sym->csum.checksum = XXH3_64bits_digest(sym->csum.state); + XXH3_freeState(sym->csum.state); + sym->csum.state = NULL; } } +int calculate_checksums(struct objtool_file *file); +int create_sym_checksum_section(struct objtool_file *file); + #else /* !BUILD_KLP */ -static inline void checksum_init(struct symbol *func) {} -static inline void checksum_update(struct symbol *func, - struct instruction *insn, - const void *data, size_t size) {} -static inline void checksum_finish(struct symbol *func) {} +static inline int calculate_checksums(struct objtool_file *file) { return -ENOSYS; } +static inline int create_sym_checksum_section(struct objtool_file *file) { return -EINVAL; } #endif /* !BUILD_KLP */ diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h index 25573e5af76e..d9c44df9cc76 100644 --- a/tools/objtool/include/objtool/elf.h +++ b/tools/objtool/include/objtool/elf.h @@ -21,6 +21,13 @@ #define SEC_NAME_LEN 1024 #define SYM_NAME_LEN 512 +static inline u32 str_hash(const char *str) +{ + return jhash(str, strlen(str), 0); +} + +u32 str_hash_demangled(const char *str); + #define bswap_if_needed(elf, val) __bswap_if_needed(&elf->ehdr, val) #ifdef LIBELF_USE_DEPRECATED @@ -89,6 +96,7 @@ struct symbol { u8 changed : 1; u8 included : 1; u8 klp : 1; + u8 dont_correlate : 1; struct list_head pv_target; struct reloc *relocs; struct section *group_sec; @@ -130,6 +138,23 @@ struct elf { struct symbol *symbol_data; }; +#define __elf_table(elf, name) ((elf)->name##_hash) +#define __elf_bits(elf, name) ((elf)->name##_bits) + +#define __elf_table_entry(elf, name, key) \ + __elf_table(elf, name)[hash_min(key, __elf_bits(elf, name))] + +#define elf_list_entry(ptr, type, member) \ +({ \ + typeof(ptr) __ptr = (ptr); \ + __ptr ? container_of(__ptr, type, member) : NULL; \ +}) + +#define elf_hash_for_each_possible(elf, name, obj, member, key) \ + for (obj = elf_list_entry(__elf_table_entry(elf, name, key), typeof(*obj), member); \ + obj; \ + obj = elf_list_entry(obj->member.next, typeof(*(obj)), member)) + struct elf *elf_open_read(const char *name, int flags); struct elf *elf_create_file(GElf_Ehdr *ehdr, const char *name); @@ -175,6 +200,7 @@ struct reloc *elf_init_reloc_data_sym(struct elf *elf, struct section *sec, struct symbol *sym, s64 addend); +int elf_write_symbol(struct elf *elf, struct symbol *sym); int elf_write_insn(struct elf *elf, struct section *sec, unsigned long offset, unsigned int len, const char *insn); @@ -186,10 +212,8 @@ struct symbol *find_func_by_offset(struct section *sec, unsigned long offset); struct symbol *find_symbol_by_offset(struct section *sec, unsigned long offset); struct symbol *find_symbol_by_name(const struct elf *elf, const char *name); struct symbol *find_global_symbol_by_name(const struct elf *elf, const char *name); -void iterate_global_symbol_by_demangled_name(const struct elf *elf, const char *demangled_name, - void (*process)(struct symbol *sym, void *data), - void *data); struct symbol *find_symbol_containing(const struct section *sec, unsigned long offset); +struct symbol *find_symbol_containing_inclusive(const struct section *sec, unsigned long offset); int find_symbol_hole_containing(const struct section *sec, unsigned long offset); struct reloc *find_reloc_by_dest(const struct elf *elf, struct section *sec, unsigned long offset); struct reloc *find_reloc_by_dest_range(const struct elf *elf, struct section *sec, @@ -276,11 +300,21 @@ static inline bool is_local_sym(struct symbol *sym) return sym->bind == STB_LOCAL; } +static inline bool is_alias_sym(struct symbol *sym) +{ + return sym->alias != sym; +} + static inline bool is_prefix_func(struct symbol *sym) { return sym->prefix; } +static inline bool is_cold_func(struct symbol *sym) +{ + return sym->cold; +} + static inline bool is_reloc_sec(struct section *sec) { return sec->sh.sh_type == SHT_RELA || sec->sh.sh_type == SHT_REL; @@ -296,6 +330,11 @@ static inline bool is_text_sec(struct section *sec) return sec->sh.sh_flags & SHF_EXECINSTR; } +static inline bool is_rodata_sec(struct section *sec) +{ + return sec->rodata; +} + static inline bool sec_changed(struct section *sec) { return sec->_changed; @@ -468,6 +507,16 @@ static inline void set_sym_next_reloc(struct reloc *reloc, struct reloc *next) #define for_each_sym_continue(elf, sym) \ list_for_each_entry_continue(sym, &elf->symbols, global_list) +#define for_each_sym_by_name(elf, _name, sym) \ + elf_hash_for_each_possible(elf, symbol_name, sym, name_hash, \ + str_hash_demangled(_name)) \ + if (strcmp(sym->name, _name)) {} else + +#define for_each_sym_by_demangled_name(elf, name, sym) \ + elf_hash_for_each_possible(elf, symbol_name, sym, name_hash, \ + str_hash(name)) \ + if (strcmp(sym->demangled_name, name)) {} else + #define rsec_next_reloc(rsec, reloc) \ reloc_idx(reloc) < sec_num_entries(rsec) - 1 ? reloc + 1 : NULL @@ -491,10 +540,10 @@ static inline struct symbol *get_func_prefix(struct symbol *func) { struct symbol *prev; - if (!is_func_sym(func)) + if (!is_func_sym(func) || !func->offset) return NULL; - prev = sec_prev_sym(func); + prev = find_func_containing(func->sec, func->offset - 1); if (prev && is_prefix_func(prev)) return prev; diff --git a/tools/objtool/include/objtool/klp.h b/tools/objtool/include/objtool/klp.h index e32e5e8bc631..6f60cf05db86 100644 --- a/tools/objtool/include/objtool/klp.h +++ b/tools/objtool/include/objtool/klp.h @@ -29,6 +29,7 @@ struct klp_reloc { u32 type; }; +int cmd_klp_checksum(int argc, const char **argv); int cmd_klp_diff(int argc, const char **argv); int cmd_klp_post_link(int argc, const char **argv); diff --git a/tools/objtool/include/objtool/warn.h b/tools/objtool/include/objtool/warn.h index fa8b7d292e83..870e147f3a56 100644 --- a/tools/objtool/include/objtool/warn.h +++ b/tools/objtool/include/objtool/warn.h @@ -77,13 +77,13 @@ static inline char *offstr(struct section *sec, unsigned long offset) #define WARN_INSN(insn, format, ...) \ ({ \ struct instruction *_insn = (insn); \ - if (!_insn->sym || !_insn->sym->warned) { \ + if (!insn_sym(_insn) || !insn_sym(_insn)->warned) { \ WARN_FUNC(_insn->sec, _insn->offset, format, \ ##__VA_ARGS__); \ BT_INSN(_insn, ""); \ } \ - if (_insn->sym) \ - _insn->sym->warned = 1; \ + if (insn_sym(_insn)) \ + insn_sym(_insn)->warned = 1; \ }) #define BT_INSN(insn, format, ...) \ @@ -109,7 +109,7 @@ static inline char *offstr(struct section *sec, unsigned long offset) #define ERROR_FUNC(sec, offset, format, ...) __WARN_FUNC(ERROR_STR, sec, offset, format, ##__VA_ARGS__) #define ERROR_INSN(insn, format, ...) ERROR_FUNC(insn->sec, insn->offset, format, ##__VA_ARGS__) -extern bool debug; +extern bool debug, debug_correlate, debug_clone; extern int indent; static inline void unindent(int *unused) { indent--; } @@ -130,32 +130,39 @@ static inline void unindent(int *unused) { indent--; } objname ? ": " : "", \ ##__VA_ARGS__) -#define dbg(args...) \ +#define dbg_checksum_insn(func, insn, checksum) \ ({ \ - if (unlikely(debug)) \ - __dbg(args); \ -}) - -#define __dbg_indent(format, ...) \ -({ \ - if (unlikely(debug)) \ - __dbg("%*s" format, indent * 8, "", ##__VA_ARGS__); \ -}) - -#define dbg_indent(args...) \ - int __cleanup(unindent) __dummy_##__COUNTER__; \ - __dbg_indent(args); \ - indent++ - -#define dbg_checksum(func, insn, checksum) \ -({ \ - if (unlikely(insn->sym && insn->sym->pfunc && \ - insn->sym->pfunc->debug_checksum)) { \ + if (unlikely(func->debug_checksum)) { \ char *insn_off = offstr(insn->sec, insn->offset); \ - __dbg("checksum: %s %s %016llx", \ + __dbg("checksum: %s(): %s %016llx", \ func->name, insn_off, (unsigned long long)checksum);\ free(insn_off); \ } \ }) +#define dbg_checksum_object(sym, offset, what, checksum) \ +({ \ + if (unlikely(sym->debug_checksum)) \ + __dbg("checksum: %s+0x%lx: %s %016llx", \ + sym->name, offset, what, \ + (unsigned long long)checksum); \ +}) + +#define dbg_correlate(args...) \ +({ \ + if (unlikely(debug_correlate)) \ + __dbg(args); \ +}) + +#define __dbg_clone(format, ...) \ +({ \ + if (unlikely(debug_clone)) \ + __dbg("%*s" format, indent * 8, "", ##__VA_ARGS__); \ +}) + +#define dbg_clone(args...) \ + int __cleanup(unindent) __dummy_##__COUNTER__; \ + __dbg_clone(args); \ + indent++ + #endif /* _WARN_H */ diff --git a/tools/objtool/klp-checksum.c b/tools/objtool/klp-checksum.c new file mode 100644 index 000000000000..b8e47f28997e --- /dev/null +++ b/tools/objtool/klp-checksum.c @@ -0,0 +1,347 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +static int checksum_debug_init(struct objtool_file *file) +{ + char *dup, *s; + + if (!opts.debug_checksum) + return 0; + + dup = strdup(opts.debug_checksum); + if (!dup) { + ERROR_GLIBC("strdup"); + return -1; + } + + s = dup; + while (*s) { + bool found = false; + struct symbol *sym; + char *comma; + + comma = strchr(s, ','); + if (comma) + *comma = '\0'; + + for_each_sym_by_name(file->elf, s, sym) { + if (!is_func_sym(sym) && !is_object_sym(sym)) + continue; + sym->debug_checksum = 1; + found = true; + } + + if (!found) + WARN("--debug-checksum: can't find '%s'", s); + + if (!comma) + break; + + s = comma + 1; + } + + free(dup); + return 0; +} + +static void checksum_update_insn(struct objtool_file *file, struct symbol *func, + struct instruction *insn) +{ + struct reloc *reloc = insn_reloc(file, insn); + struct alternative *alt; + unsigned long offset; + struct symbol *sym; + static bool in_alt; + + if (insn->fake) + return; + + if (!reloc) { + struct symbol *call_dest = insn_call_dest(insn); + struct instruction *jump_dest = insn->jump_dest; + + /* + * For a jump/call non-relocated dest offset embedded in the + * instruction, the offset may vary due to changes in + * surrounding code. Just hash the opcode and a + * position-independent representation of the destination. + */ + + if (call_dest || jump_dest) { + unsigned char buf[16]; + size_t len; + + len = arch_jump_opcode_bytes(file, insn, buf); + __checksum_update_insn(func, insn, buf, len); + + if (call_dest) { + __checksum_update_insn(func, insn, call_dest->demangled_name, + strlen(call_dest->demangled_name)); + + } else if (jump_dest) { + struct symbol *dest_sym; + unsigned long offset; + + /* + * use insn->_sym instead of insn_sym() here. + * For alternative replacements, the latter + * would give the function of the code being + * replaced. + */ + dest_sym = jump_dest->_sym; + if (!dest_sym) + goto alts; + + __checksum_update_insn(func, insn, dest_sym->demangled_name, + strlen(dest_sym->demangled_name)); + + offset = jump_dest->offset - dest_sym->offset; + __checksum_update_insn(func, insn, &offset, sizeof(offset)); + } + + goto alts; + } + } + + __checksum_update_insn(func, insn, insn->sec->data->d_buf + insn->offset, insn->len); + + if (!reloc) + goto alts; + + sym = reloc->sym; + offset = arch_insn_adjusted_addend(insn, reloc); + + if (is_string_sec(sym->sec)) { + char *str; + + str = sym->sec->data->d_buf + sym->offset + offset; + __checksum_update_insn(func, insn, str, strlen(str)); + goto alts; + } + + if (is_sec_sym(sym)) { + sym = find_symbol_containing(reloc->sym->sec, offset); + if (!sym) + goto alts; + + offset -= sym->offset; + } + + __checksum_update_insn(func, insn, sym->demangled_name, + strlen(sym->demangled_name)); + __checksum_update_insn(func, insn, &offset, sizeof(offset)); + +alts: + for (alt = insn->alts; alt; alt = alt->next) { + struct alt_group *alt_group = alt->insn->alt_group; + + /* Prevent __ex_table recursion, e.g. LOAD_SEGMENT() */ + if (in_alt) + break; + in_alt = true; + + __checksum_update_insn(func, insn, &alt->type, + sizeof(alt->type)); + + if (alt_group && alt_group->orig_group) { + struct instruction *alt_insn; + + __checksum_update_insn(func, insn, &alt_group->feature,sizeof(alt_group->feature)); + + for (alt_insn = alt->insn; alt_insn; alt_insn = next_insn_same_sec(file, alt_insn)) { + checksum_update_insn(file, func, alt_insn); + if (!alt_group->last_insn || alt_insn == alt_group->last_insn) + break; + } + } else { + checksum_update_insn(file, func, alt->insn); + } + + in_alt = false; + } +} + +static void checksum_update_object(struct objtool_file *file, struct symbol *sym) +{ + struct reloc *reloc; + + __checksum_update_object(sym, 0, "len", &sym->len, sizeof(sym->len)); + + if (sym->sec->data->d_buf) + __checksum_update_object(sym, 0, "data", + sym->sec->data->d_buf + sym->offset, + sym->len); + + sym_for_each_reloc(file->elf, sym, reloc) { + unsigned long sym_offset = reloc_offset(reloc) - sym->offset; + struct symbol *target = reloc->sym; + s64 offset; + + offset = reloc_addend(reloc); + + if (is_string_sec(target->sec)) { + char *str; + + str = target->sec->data->d_buf + target->offset + offset; + __checksum_update_object(sym, sym_offset, + "reloc string", str, strlen(str)); + continue; + } + + if (is_sec_sym(target)) { + target = find_symbol_containing(reloc->sym->sec, offset); + if (!target) + continue; + + offset -= target->offset; + } + + __checksum_update_object(sym, sym_offset, "reloc name", + target->demangled_name, + strlen(target->demangled_name)); + __checksum_update_object(sym, sym_offset, "reloc addend", + &offset, sizeof(offset)); + } +} + +int calculate_checksums(struct objtool_file *file) +{ + struct instruction *insn; + struct symbol *sym; + + if (checksum_debug_init(file)) + return -1; + + for_each_sym(file->elf, sym) { + + /* + * Skip cold subfunctions and aliases: they share the + * parent's checksum via func_for_each_insn() which + * follows func->cfunc into the cold subfunction. + */ + if (is_cold_func(sym) || is_alias_sym(sym) || !sym->len || + !sym->sec || !sym->sec->data) + continue; + + if (is_func_sym(sym)) { + checksum_init(sym); + func_for_each_insn(file, sym, insn) + checksum_update_insn(file, sym, insn); + checksum_finish(sym); + + } else if (is_object_sym(sym)) { + checksum_init(sym); + checksum_update_object(file, sym); + checksum_finish(sym); + } + + } + + return 0; +} + +int create_sym_checksum_section(struct objtool_file *file) +{ + struct section *sec; + struct symbol *sym; + unsigned int idx = 0; + struct sym_checksum *checksum; + size_t entsize = sizeof(struct sym_checksum); + + sec = find_section_by_name(file->elf, ".discard.sym_checksum"); + if (sec) { + if (!opts.dryrun) + WARN("file already has .discard.sym_checksum section, skipping"); + + return 0; + } + + for_each_sym(file->elf, sym) + if (sym->csum.checksum) + idx++; + + sec = elf_create_section_pair(file->elf, ".discard.sym_checksum", entsize, + idx, idx); + if (!sec) + return -1; + + idx = 0; + for_each_sym(file->elf, sym) { + if (!sym->csum.checksum) + continue; + + if (!elf_init_reloc(file->elf, sec->rsec, idx, idx * entsize, + sym, 0, R_TEXT64)) + return -1; + + checksum = (struct sym_checksum *)sec->data->d_buf + idx; + checksum->addr = 0; /* reloc */ + checksum->checksum = sym->csum.checksum; + + mark_sec_changed(file->elf, sec, true); + + idx++; + } + + return 0; +} + +static const char * const klp_checksum_usage[] = { + "objtool klp checksum [] file.o", + NULL, +}; + +int cmd_klp_checksum(int argc, const char **argv) +{ + struct objtool_file *file; + int ret; + + const struct option options[] = { + OPT_STRING(0, "debug-checksum", &opts.debug_checksum, "syms", "enable checksum debug output"), + OPT_BOOLEAN(0, "dry-run", &opts.dryrun, "don't write modifications"), + OPT_END(), + }; + + argc = parse_options(argc, argv, options, klp_checksum_usage, 0); + if (argc != 1) + usage_with_options(klp_checksum_usage, options); + + opts.checksum = true; + + objname = argv[0]; + + file = objtool_open_read(objname); + if (!file) + return 1; + + ret = decode_file(file); + if (ret) + goto out; + + ret = calculate_checksums(file); + if (ret) + goto out; + + ret = create_sym_checksum_section(file); + +out: + free_insns(file); + + if (ret) + return ret; + + if (!opts.dryrun && file->elf->changed && elf_write(file->elf)) + return 1; + + return elf_close(file->elf); +} diff --git a/tools/objtool/klp-diff.c b/tools/objtool/klp-diff.c index 0b0d1503851f..f8787d7d1454 100644 --- a/tools/objtool/klp-diff.c +++ b/tools/objtool/klp-diff.c @@ -33,6 +33,9 @@ struct export { char *mod, *sym; }; +bool debug, debug_correlate, debug_clone; +int indent; + static const char * const klp_diff_usage[] = { "objtool klp diff [] ", NULL, @@ -40,17 +43,14 @@ static const char * const klp_diff_usage[] = { static const struct option klp_diff_options[] = { OPT_GROUP("Options:"), - OPT_BOOLEAN('d', "debug", &debug, "enable debug output"), + OPT_BOOLEAN('d', "debug", &debug, "enable all debug output"), + OPT_BOOLEAN(0, "debug-correlate", &debug_correlate, "enable correlation debug output"), + OPT_BOOLEAN(0, "debug-clone", &debug_clone, "enable cloning debug output"), OPT_END(), }; static DEFINE_HASHTABLE(exports, 15); -static inline u32 str_hash(const char *str) -{ - return jhash(str, strlen(str), 0); -} - static char *escape_str(const char *orig) { size_t len = 0; @@ -171,7 +171,7 @@ static int read_sym_checksums(struct elf *elf) sec = find_section_by_name(elf, ".discard.sym_checksum"); if (!sec) { - ERROR("'%s' missing .discard.sym_checksum section, file not processed by 'objtool --checksum'?", + ERROR("'%s' missing .discard.sym_checksum section, file not processed by 'objtool klp checksum'?", elf->name); return -1; } @@ -206,7 +206,7 @@ static int read_sym_checksums(struct elf *elf) return -1; } - if (is_func_sym(sym)) + if (is_func_sym(sym) || is_object_sym(sym)) sym->csum.checksum = sym_checksum->checksum; } @@ -242,25 +242,39 @@ static struct symbol *next_file_symbol(struct elf *elf, struct symbol *sym) static bool is_uncorrelated_static_local(struct symbol *sym) { static const char * const vars[] = { - "__already_done.", - "__func__.", - "__key.", - "__warned.", - "_entry.", - "_entry_ptr.", - "_rs.", - "descriptor.", - "CSWTCH.", + "__already_done", + "__func__", + "__key", + "__warned", + "_entry", + "_entry_ptr", + "_rs", + "descriptor", + "CSWTCH", }; + const char *dot; if (!is_object_sym(sym) || !is_local_sym(sym)) return false; - if (!strcmp(sym->sec->name, ".data.once")) + /* WARN_ONCE, etc */ + if (!strcmp(sym->sec->name, ".data..once")) return true; + dot = strchr(sym->name, '.'); + if (!dot) + return false; + for (int i = 0; i < ARRAY_SIZE(vars); i++) { - if (strstarts(sym->name, vars[i])) + size_t len = strlen(vars[i]); + + /* GCC: . */ + if (strstarts(sym->name, vars[i]) && (sym->name[len] == '.')) + return true; + + /* Clang: .[.] */ + if (strstarts(dot + 1, vars[i]) && + (dot[1 + len] == '.' || dot[1 + len] == '\0')) return true; } @@ -268,20 +282,21 @@ static bool is_uncorrelated_static_local(struct symbol *sym) } /* - * Clang emits several useless .Ltmp_* code labels. + * .L symbols are assembler-local labels not present in kallsyms. They must + * never become KLP relocations; instead their data is cloned into the patch + * module. This covers .Ltmp* (Clang temp labels), .L__const.* (Clang local + * constants), and any other assembler-local pattern. */ -static bool is_clang_tmp_label(struct symbol *sym) +static bool is_local_label(struct symbol *sym) { - return is_notype_sym(sym) && - is_text_sec(sym->sec) && - strstarts(sym->name, ".Ltmp") && - isdigit(sym->name[5]); + return strstarts(sym->name, ".L"); } static bool is_special_section(struct section *sec) { static const char * const specials[] = { ".altinstructions", + ".kcfi_traps", ".smp_locks", "__bug_table", "__ex_table", @@ -338,6 +353,45 @@ static bool is_special_section_aux(struct section *sec) return false; } +/* + * Symbols created by ___ADDRESSABLE() are only used to convince the toolchain + * not to optimize out the referenced symbol. + */ +static bool is_addressable_sym(struct symbol *sym) +{ + return !strcmp(sym->sec->name, ".discard.addressable"); +} + +/* + * ABS symbols are typically assembly .set/.equ constants which are never + * referenced by relocations. (Exclude FILE symbols which are also SHN_ABS.) + */ +static bool is_abs_sym(struct symbol *sym) +{ + return sym->sym.st_shndx == SHN_ABS && !is_file_sym(sym); +} + +static bool is_initcall_sym(struct symbol *sym) +{ + return strstarts(sym->name, "__initcall__") || + strstarts(sym->name, "__initstub__"); +} + +/* + * Some .rodata is anonymous and can't be correlated due to there being no + * symbol names. + * + * The .rodata.cst* sections aren't technically anonymous, they're SHF_MERGE + * constant pool sections containing small fixed-size data (lookup tables, + * bitmasks) which are only read by value, so pointer equivalence isn't needed. + * They are typically referenced by UBSAN data sections. + */ +static bool is_anonymous_rodata(struct symbol *sym) +{ + return is_rodata_sec(sym->sec) && + (!is_object_sym(sym) || strstarts(sym->sec->name, ".rodata.cst")); +} + /* * These symbols should never be correlated, so their local patched versions * are used instead of linking to the originals. @@ -347,96 +401,391 @@ static bool dont_correlate(struct symbol *sym) return is_file_sym(sym) || is_null_sym(sym) || is_sec_sym(sym) || + is_abs_sym(sym) || is_prefix_func(sym) || is_uncorrelated_static_local(sym) || - is_clang_tmp_label(sym) || + is_local_label(sym) || is_string_sec(sym->sec) || + is_anonymous_rodata(sym) || + is_initcall_sym(sym) || + is_addressable_sym(sym) || is_special_section(sym->sec) || - is_special_section_aux(sym->sec) || - strstarts(sym->name, "__initcall__"); + is_special_section_aux(sym->sec); } -struct process_demangled_name_data { - struct symbol *ret; - int count; -}; - -static void process_demangled_name(struct symbol *sym, void *d) +static const char *llvm_suffix(const char *name) { - struct process_demangled_name_data *data = d; + return strstr(name, ".llvm."); +} - if (sym->twin) - return; - - data->count++; - data->ret = sym; +static bool is_llvm_sym(struct symbol *sym) +{ + return llvm_suffix(sym->name); } /* - * When there is no full name match, try match demangled_name. This would - * match original foo.llvm.123 to patched foo.llvm.456. + * Determine if two symbols have compatible source file origins: * - * Note that, in very rare cases, it is possible to have multiple - * foo.llvm. in the same kernel. When this happens, report error and - * fail the diff. + * - If both symbols are local, only return true if they belong to the same + * ELF file symbol. + * + * - If both symbols are global, always return true, as globals don't have + * file associations. + * + * - If they have different scopes, also return true, as the patch might have + * changed the symbol's scope. + * + * Works for both same-ELF (direct pointer compare) and cross-ELF + * (compare via file->twin) cases. */ -static int find_global_symbol_by_demangled_name(struct elf *elf, struct symbol *sym, - struct symbol **out_sym) +static bool maybe_same_file(struct symbol *sym1, struct symbol *sym2) { - struct process_demangled_name_data data = {}; + if (!sym1->file || !sym2->file) + return true; + if (sym1->file == sym2->file) + return true; + return sym1->file->twin == sym2->file; +} - iterate_global_symbol_by_demangled_name(elf, sym->demangled_name, - process_demangled_name, - &data); - if (data.count > 1) { - ERROR("Multiple (%d) correlation candidates for %s", data.count, sym->name); - return -1; +/* + * Similar to maybe_same_file(), but strict: no scope changes allowed. + * + * Works for both same-ELF (direct pointer compare) and cross-ELF + * (compare via file->twin) cases. + */ +static bool same_file(struct symbol *sym1, struct symbol *sym2) +{ + if (llvm_suffix(sym1->name) && llvm_suffix(sym2->name)) + return true; + if (!sym1->file && !sym2->file) + return true; + if (!sym1->file || !sym2->file) + return false; + if (sym1->file == sym2->file) + return true; + return sym1->file->twin == sym2->file; +} + +/* + * Is it a local symbol, or at least was it local in the translation unit + * before LLVM promoted it? + */ +static bool is_tu_local_sym(struct symbol *sym) +{ + return is_local_sym(sym) || is_llvm_sym(sym); +} + +/* + * Try to find sym1's twin in patched using deterministic matching. + * + * Multiple symbols can share a demangled name (e.g., static functions in + * different TUs). This function counts same-named candidates through a + * funnel of progressively tighter filters. Each level is a strict subset + * of the previous one. + * + * The widest level that yields a 1:1 match wins. Narrower levels are only + * needed when the wider level is ambiguous (count > 1). + * + * Candidates are pre-filtered by maybe_same_file(), which narrows most + * local symbols to their own TU. For example, 19 different static + * type_show() functions across vmlinux.o each see only one candidate after + * pre-filtering, so they match immediately at Level 1. + * + * Level 1 (name): Works when the demangled name is unique after + * pre-filtering. Handles most symbols: unique globals like copy_signal(), + * or per-TU locals like pcspkr_probe(). + * + * Level 2 (scope): Filters by local-vs-global (TU-local-vs-not). Example: + * parse_header() exists as both a static and a global function. Level 1 + * sees both (same demangled name), but Level 2 separates them by scope. + * + * Level 3 (file): Strict file matching via same_file(), which rejects scope + * changes. Example: LLVM-promoted foo.llvm.12345 (global, no FILE symbol) + * vs genuine local foo (has FILE symbol). Both are TU-local so Level 2 + * can't distinguish them, but same_file() rejects the pair because one has + * a file association and the other doesn't. + * + * Level 4 (checksum): Distinguishes by function checksum. Example: + * usb_devnode.llvm.AAA and usb_devnode.llvm.BBB are two LLVM-promoted + * functions from different TUs with the same demangled name. After a TU + * change, the .llvm. hashes change but the functions themselves may be + * unchanged. Level 4 matches each to the patched candidate with the + * same checksum. + */ +static struct symbol *find_twin(struct elfs *e, struct symbol *sym1) +{ + struct symbol *name_last = NULL, *scope_last = NULL, + *file_last = NULL, *csum_last = NULL; + unsigned int name_orig = 0, name_patched = 0; + unsigned int scope_orig = 0, scope_patched = 0; + unsigned int file_orig = 0, file_patched = 0; + unsigned int csum_orig = 0, csum_patched = 0; + struct symbol *sym2, *match = NULL; + + /* Count orig candidates */ + for_each_sym_by_demangled_name(e->orig, sym1->demangled_name, sym2) { + if (sym2->twin || sym1->type != sym2->type || sym2->dont_correlate || + (!maybe_same_file(sym1, sym2))) + continue; + + /* Level 1: name match (widest filter) */ + name_orig++; + + /* Level 2: scope (scope changes allowed) */ + if (is_tu_local_sym(sym1) != is_tu_local_sym(sym2)) + continue; + scope_orig++; + + /* Level 3: file (scope changes disallowed) */ + if (!same_file(sym1, sym2)) + continue; + file_orig++; + + /* Level 4: checksum (unchanged symbols) */ + if (sym1->len != sym2->len || !sym1->csum.checksum || + sym1->csum.checksum != sym2->csum.checksum) + continue; + csum_orig++; } - *out_sym = data.ret; + + /* Count patched candidates */ + for_each_sym_by_demangled_name(e->patched, sym1->demangled_name, sym2) { + if (sym2->twin || sym1->type != sym2->type || sym2->dont_correlate || + !maybe_same_file(sym1, sym2)) + continue; + + /* Level 1 */ + name_patched++; + name_last = sym2; + + /* Level 2 */ + if (is_tu_local_sym(sym1) != is_tu_local_sym(sym2)) + continue; + scope_patched++; + scope_last = sym2; + + /* Level 3 */ + if (!same_file(sym1, sym2)) + continue; + file_patched++; + file_last = sym2; + + /* Level 4 */ + if (sym1->len != sym2->len || !sym1->csum.checksum || + sym1->csum.checksum != sym2->csum.checksum) + continue; + csum_patched++; + csum_last = sym2; + } + + /* Return the widest level that yields a unique (1:1) match */ + if (name_orig == 1 && name_patched == 1) + match = name_last; + else if (scope_orig == 1 && scope_patched == 1) + match = scope_last; + else if (file_orig == 1 && file_patched == 1) + match = file_last; + else if (csum_orig == 1 && csum_patched == 1) + match = csum_last; + + if (!match) + return NULL; + + if (name_orig != 1 || name_patched != 1) + dbg_correlate("find_twin(): %s%s -> %s%s", + sym1->name, is_func_sym(sym1) ? "()" : "", + match->name, is_func_sym(match) ? "()" : ""); + + return match; +} + +struct llvm_suffix_pair { + struct hlist_node hash; + const char *orig; + const char *patched; +}; + +static DECLARE_HASHTABLE(suffix_map, 7); + +/* + * Build a mapping of known orig-to-patched LLVM suffixes based on + * already-correlated symbol pairs. All promoted symbols from the same TU + * share the same .llvm. suffix, so one correlated pair seeds the map + * for the entire TU. + */ +static int update_suffix_map(struct elf *elf) +{ + struct llvm_suffix_pair *entry; + struct symbol *sym; + + for_each_sym(elf, sym) { + const char *s1, *s2; + bool found; + + if (!sym->twin) + continue; + + s1 = llvm_suffix(sym->name); + s2 = llvm_suffix(sym->twin->name); + + if (!s1 || !s2) + continue; + + found = false; + hash_for_each_possible(suffix_map, entry, hash, str_hash(s1)) { + if (!strcmp(entry->orig, s1)) { + found = true; + break; + } + } + if (found) + continue; + + entry = calloc(1, sizeof(*entry)); + if (!entry) { + ERROR_GLIBC("calloc"); + return -1; + } + + entry->orig = s1; + entry->patched = s2; + hash_add(suffix_map, &entry->hash, str_hash(s1)); + } + return 0; } /* - * For each symbol in the original kernel, find its corresponding "twin" in the - * patched kernel. + * Match by translating the symbol's .llvm. suffix through the suffix + * map to find the corresponding hash suffix for the patched object. + * + * Example: In the original kernel, TU drivers/base/core.c contains + * foo.llvm.12345 and bar.llvm.12345 (same TU, same hash). After patching, + * they become foo.llvm.67890 and bar.llvm.67890. If foo was already + * correlated by find_twin() (e.g., unique by name), the suffix map records + * .llvm.12345 -> .llvm.67890. When processing bar.llvm.12345, this + * function looks up .llvm.12345, gets .llvm.67890, constructs the name + * bar.llvm.67890, and finds the match. + */ +static struct symbol *find_twin_suffixed(struct elf *elf, struct symbol *sym1) +{ + const char *suffix, *patched_suffix = NULL; + struct symbol *sym2, *match = NULL; + char name[SYM_NAME_LEN]; + struct llvm_suffix_pair *entry; + int count = 0; + + suffix = llvm_suffix(sym1->name); + if (!suffix) + return NULL; + + hash_for_each_possible(suffix_map, entry, hash, str_hash(suffix)) { + if (!strcmp(entry->orig, suffix)) { + patched_suffix = entry->patched; + break; + } + } + if (!patched_suffix) + return NULL; + + if (snprintf_check(name, SYM_NAME_LEN, "%s%s", + sym1->demangled_name, patched_suffix)) + return NULL; + + for_each_sym_by_name(elf, name, sym2) { + if (sym2->twin || sym1->type != sym2->type || sym2->dont_correlate) + continue; + count++; + match = sym2; + } + + if (count != 1) + return NULL; + + dbg_correlate("find_suffixed_twin(): %s%s -> %s%s", + sym1->name, is_func_sym(sym1) ? "()" : "", + match->name, is_func_sym(match) ? "()" : ""); + + return match; +} + +/* + * Last-resort positional matching. + * + * Finds a symbol with the same position in the symbol table among + * same-demangled-name candidates, similar to livepatch sympos. Note that + * LLVM-promoted symbols are globals, which come after locals in the symbol + * table, so we have to be careful not to compare different scopes. + * + * Example: arch/x86/events/intel/core.c defines many __quirk variables via + * X86_MATCH_*() macros. In the symbol table they appear as __quirk.90, + * __quirk.97, __quirk.101, etc., all with demangled name __quirk, same + * scope, and same FILE symbol. No deterministic filter can distinguish + * them, so they're matched by position: the 1st __quirk in orig matches the + * 1st in patched, the 2nd matches the 2nd, etc. + * + * This is less deterministic than the other strategies, so it's done last. + */ +static struct symbol *find_twin_positional(struct elfs *e, struct symbol *sym1) +{ + unsigned int idx_orig = 0, idx_patched = 0; + unsigned int sym1_pos = 0; + struct symbol *sym2, *match = NULL; + + for_each_sym_by_demangled_name(e->orig, sym1->demangled_name, sym2) { + if (sym2->twin || sym1->type != sym2->type || sym2->dont_correlate || + !maybe_same_file(sym1, sym2)) + continue; + if (is_tu_local_sym(sym1) != is_tu_local_sym(sym2) || + is_llvm_sym(sym1) != is_llvm_sym(sym2)) + continue; + if (sym1 == sym2) + sym1_pos = idx_orig; + idx_orig++; + } + + for_each_sym_by_demangled_name(e->patched, sym1->demangled_name, sym2) { + if (sym2->twin || sym1->type != sym2->type || sym2->dont_correlate || + !maybe_same_file(sym1, sym2)) + continue; + if (is_tu_local_sym(sym1) != is_tu_local_sym(sym2) || + is_llvm_sym(sym1) != is_llvm_sym(sym2)) + continue; + if (idx_patched == sym1_pos) + match = sym2; + idx_patched++; + } + + if (idx_orig != idx_patched) + return NULL; + + dbg_correlate("find_twin_positional(): %s%s -> %s%s", + sym1->name, is_func_sym(sym1) ? "()" : "", + match->name, is_func_sym(match) ? "()" : ""); + + return match; +} + +/* + * Correlate symbols between the orig and patched objects. This is a + * prerequisite for detecting changed functions, as well as for properly + * translating relocations so they point to the correct symbol. */ static int correlate_symbols(struct elfs *e) { struct symbol *file1_sym, *file2_sym; struct symbol *sym1, *sym2; + bool progress; + for_each_sym(e->orig, sym1) + sym1->dont_correlate = dont_correlate(sym1); + for_each_sym(e->patched, sym2) + sym2->dont_correlate = dont_correlate(sym2); + + /* Correlate FILE symbols */ file1_sym = first_file_symbol(e->orig); file2_sym = first_file_symbol(e->patched); - /* - * Correlate any locals before the first FILE symbol. This has been - * seen when LTO inexplicably strips the initramfs_data.o FILE symbol - * due to the file only containing data and no code. - */ - for_each_sym(e->orig, sym1) { - if (sym1 == file1_sym || !is_local_sym(sym1)) - break; - - if (dont_correlate(sym1)) - continue; - - for_each_sym(e->patched, sym2) { - if (sym2 == file2_sym || !is_local_sym(sym2)) - break; - - if (sym2->twin || dont_correlate(sym2)) - continue; - - if (strcmp(sym1->demangled_name, sym2->demangled_name)) - continue; - - sym1->twin = sym2; - sym2->twin = sym1; - break; - } - } - - /* Correlate locals after the first FILE symbol */ for (; ; file1_sym = next_file_symbol(e->orig, file1_sym), file2_sym = next_file_symbol(e->patched, file2_sym)) { @@ -460,96 +809,56 @@ static int correlate_symbols(struct elfs *e) file1_sym->twin = file2_sym; file2_sym->twin = file1_sym; - - sym1 = file1_sym; - - for_each_sym_continue(e->orig, sym1) { - if (is_file_sym(sym1) || !is_local_sym(sym1)) - break; - - if (dont_correlate(sym1)) - continue; - - sym2 = file2_sym; - for_each_sym_continue(e->patched, sym2) { - if (is_file_sym(sym2) || !is_local_sym(sym2)) - break; - - if (sym2->twin || dont_correlate(sym2)) - continue; - - if (strcmp(sym1->demangled_name, sym2->demangled_name)) - continue; - - sym1->twin = sym2; - sym2->twin = sym1; - break; - } - } } - /* Correlate globals */ - for_each_sym(e->orig, sym1) { - if (sym1->bind == STB_LOCAL) - continue; - - sym2 = find_global_symbol_by_name(e->patched, sym1->name); - if (sym2 && !sym2->twin) { - sym1->twin = sym2; - sym2->twin = sym1; - } - } /* - * Correlate globals with demangled_name. - * A separate loop is needed because we want to finish all the - * full name correlations first. + * Correlate in two phases: loop deterministic levels until no more + * progress, then use positional fallback for the rest. This prevents + * the nondeterministic positional matching from stealing symbols that + * have deterministic matches. */ + hash_init(suffix_map); + do { + progress = false; + for_each_sym(e->orig, sym1) { + if (sym1->twin || sym1->dont_correlate) + continue; + sym2 = find_twin(e, sym1); + if (!sym2) + continue; + sym1->twin = sym2; + sym2->twin = sym1; + progress = true; + } + + if (update_suffix_map(e->orig)) + return -1; + + for_each_sym(e->orig, sym1) { + if (sym1->twin || sym1->dont_correlate) + continue; + sym2 = find_twin_suffixed(e->patched, sym1); + if (!sym2) + continue; + sym1->twin = sym2; + sym2->twin = sym1; + progress = true; + } + } while (progress); + for_each_sym(e->orig, sym1) { - if (sym1->bind == STB_LOCAL || sym1->twin) + if (sym1->twin || sym1->dont_correlate) continue; - - if (find_global_symbol_by_demangled_name(e->patched, sym1, &sym2)) - return -1; - - if (sym2 && !sym2->twin) { - sym1->twin = sym2; - sym2->twin = sym1; - } - } - - /* Correlate original locals with patched globals */ - for_each_sym(e->orig, sym1) { - if (sym1->twin || dont_correlate(sym1) || !is_local_sym(sym1)) + sym2 = find_twin_positional(e, sym1); + if (!sym2) continue; - - sym2 = find_global_symbol_by_name(e->patched, sym1->name); - if (!sym2 && find_global_symbol_by_demangled_name(e->patched, sym1, &sym2)) - return -1; - - if (sym2 && !sym2->twin) { - sym1->twin = sym2; - sym2->twin = sym1; - } - } - - /* Correlate original globals with patched locals */ - for_each_sym(e->patched, sym2) { - if (sym2->twin || dont_correlate(sym2) || !is_local_sym(sym2)) - continue; - - sym1 = find_global_symbol_by_name(e->orig, sym2->name); - if (!sym1 && find_global_symbol_by_demangled_name(e->orig, sym2, &sym1)) - return -1; - - if (sym1 && !sym1->twin) { - sym2->twin = sym1; - sym1->twin = sym2; - } + sym1->twin = sym2; + sym2->twin = sym1; } for_each_sym(e->orig, sym1) { - if (sym1->twin || dont_correlate(sym1)) + if (sym1->twin || sym1->dont_correlate) continue; WARN("no correlation: %s", sym1->name); } @@ -655,7 +964,7 @@ static struct symbol *__clone_symbol(struct elf *elf, struct symbol *patched_sym size_t size; /* bss doesn't have data */ - if (patched_sym->sec->data->d_buf) + if (patched_sym->sec->data && patched_sym->sec->data->d_buf) data = patched_sym->sec->data->d_buf + patched_sym->offset; if (is_sec_sym(patched_sym)) @@ -715,7 +1024,7 @@ static struct symbol *clone_symbol(struct elfs *e, struct symbol *patched_sym, if (patched_sym->clone) return patched_sym->clone; - dbg_indent("%s%s", patched_sym->name, data_too ? " [+DATA]" : ""); + dbg_clone("%s%s", patched_sym->name, data_too ? " [+DATA]" : ""); /* Make sure the prefix gets cloned first */ if (is_func_sym(patched_sym) && data_too) { @@ -757,19 +1066,24 @@ static void mark_included_function(struct symbol *func) */ static int mark_changed_functions(struct elfs *e) { - struct symbol *sym_orig, *patched_sym; + struct symbol *orig_sym, *patched_sym; bool changed = false; /* Find changed functions */ - for_each_sym(e->orig, sym_orig) { - if (!is_func_sym(sym_orig) || is_prefix_func(sym_orig)) + for_each_sym(e->orig, orig_sym) { + if (orig_sym->dont_correlate) continue; - patched_sym = sym_orig->twin; + patched_sym = orig_sym->twin; if (!patched_sym) continue; - if (sym_orig->csum.checksum != patched_sym->csum.checksum) { + if (orig_sym->csum.checksum != patched_sym->csum.checksum) { + if (!is_func_sym(orig_sym)) { + ERROR("changed data: %s", orig_sym->name); + return -1; + } + patched_sym->changed = 1; mark_included_function(patched_sym); changed = true; @@ -778,7 +1092,7 @@ static int mark_changed_functions(struct elfs *e) /* Find added functions and print them */ for_each_sym(e->patched, patched_sym) { - if (!is_func_sym(patched_sym) || is_prefix_func(patched_sym)) + if (!is_func_sym(patched_sym) || patched_sym->dont_correlate) continue; if (!patched_sym->twin) { @@ -794,7 +1108,7 @@ static int mark_changed_functions(struct elfs *e) printf("%s: changed function: %s\n", objname, patched_sym->name); } - return !changed ? -1 : 0; + return !changed ? 1 : 0; } static int clone_included_functions(struct elfs *e) @@ -811,39 +1125,6 @@ static int clone_included_functions(struct elfs *e) return 0; } -/* - * Determine whether a relocation should reference the section rather than the - * underlying symbol. - */ -static bool section_reference_needed(struct section *sec) -{ - /* - * String symbols are zero-length and uncorrelated. It's easier to - * deal with them as section symbols. - */ - if (is_string_sec(sec)) - return true; - - /* - * .rodata has mostly anonymous data so there's no way to determine the - * length of a needed reference. just copy the whole section if needed. - */ - if (strstarts(sec->name, ".rodata")) - return true; - - /* UBSAN anonymous data */ - if (strstarts(sec->name, ".data..Lubsan") || /* GCC */ - strstarts(sec->name, ".data..L__unnamed_")) /* Clang */ - return true; - - return false; -} - -static bool is_reloc_allowed(struct reloc *reloc) -{ - return section_reference_needed(reloc->sym->sec) == is_sec_sym(reloc->sym); -} - static struct export *find_export(struct symbol *sym) { struct export *export; @@ -917,7 +1198,7 @@ static bool klp_reloc_needed(struct reloc *patched_reloc) struct export *export; /* no external symbol to reference */ - if (dont_correlate(patched_sym)) + if (patched_sym->dont_correlate) return false; /* For included functions, a regular reloc will do. */ @@ -952,33 +1233,59 @@ static bool klp_reloc_needed(struct reloc *patched_reloc) return true; } +/* Return -1 error, 0 success, 1 skip */ static int convert_reloc_sym_to_secsym(struct elf *elf, struct reloc *reloc) { struct symbol *sym = reloc->sym; struct section *sec = sym->sec; + if (is_sec_sym(sym)) + return 0; + if (!sec->sym && !elf_create_section_symbol(elf, sec)) return -1; reloc->sym = sec->sym; - set_reloc_sym(elf, reloc, sym->idx); + set_reloc_sym(elf, reloc, sec->sym->idx); set_reloc_addend(elf, reloc, sym->offset + reloc_addend(reloc)); return 0; } +/* Return -1 error, 0 success, 1 skip */ static int convert_reloc_secsym_to_sym(struct elf *elf, struct reloc *reloc) { struct symbol *sym = reloc->sym; struct section *sec = sym->sec; + if (!is_sec_sym(sym)) + return 0; + /* If the symbol has a dedicated section, it's easy to find */ sym = find_symbol_by_offset(sec, 0); if (sym && sym->len == sec_size(sec)) goto found_sym; /* No dedicated section; find the symbol manually */ - sym = find_symbol_containing(sec, arch_adjusted_addend(reloc)); + sym = find_symbol_containing_inclusive(sec, arch_adjusted_addend(reloc)); if (!sym) { + /* + * This is presumably an .altinstr_replacement section which is + * empty due to it only having zero-length replacement(s). + */ + if (!sec_size(sec)) + return 1; + + /* + * .rodata is a mixed bag of named objects and anonymous data. + * + * Convert section symbol references to named object symbols + * when possible, to preserve pointer identity for const + * structs like file_operations. Otherwise a section symbol is + * fine. + */ + if (is_rodata_sec(sec)) + return 0; + /* * This can happen for special section references to weak code * whose symbol has been stripped by the linker. @@ -993,19 +1300,34 @@ found_sym: return 0; } +/* + * Sections with anonymous or uncorrelated data (strings, UBSAN data, Clang + * anonymous constants) need section symbol references. + */ +static bool is_uncorrelated_section(struct section *sec) +{ + return is_string_sec(sec) || + strstarts(sec->name, ".data..Lubsan") || /* GCC */ + strstarts(sec->name, ".data..L__unnamed_") || /* Clang */ + strstarts(sec->name, ".data..Lanon."); /* Clang */ +} + /* * Convert a relocation symbol reference to the needed format: either a section - * symbol or the underlying symbol itself. + * symbol or the underlying symbol itself. Return -1 error, 0 success, 1 skip. */ static int convert_reloc_sym(struct elf *elf, struct reloc *reloc) { - if (is_reloc_allowed(reloc)) - return 0; + struct section *sec = reloc->sym->sec; - if (section_reference_needed(reloc->sym->sec)) + if (reloc_type(reloc) == R_NONE) + return 1; + + if (is_uncorrelated_section(sec)) return convert_reloc_sym_to_secsym(elf, reloc); - else - return convert_reloc_secsym_to_sym(elf, reloc); + + /* Everything else: references should use named symbols. */ + return convert_reloc_secsym_to_sym(elf, reloc); } /* @@ -1079,7 +1401,7 @@ static int clone_reloc_klp(struct elfs *e, struct reloc *patched_reloc, klp_sym = find_symbol_by_name(e->out, sym_name); if (!klp_sym) { - __dbg_indent("%s", sym_name); + __dbg_clone("%s", sym_name); /* STB_WEAK: avoid modpost undefined symbol warnings */ klp_sym = elf_create_symbol(e->out, sym_name, NULL, @@ -1130,7 +1452,7 @@ static int clone_reloc_klp(struct elfs *e, struct reloc *patched_reloc, } #define dbg_clone_reloc(sec, offset, patched_sym, addend, export, klp) \ - dbg_indent("%s+0x%lx: %s%s0x%lx [%s%s%s%s%s%s]", \ + dbg_clone("%s+0x%lx: %s%s0x%lx [%s%s%s%s%s%s]", \ sec->name, offset, patched_sym->name, \ addend >= 0 ? "+" : "-", labs(addend), \ sym_type(patched_sym), \ @@ -1150,13 +1472,6 @@ static int clone_reloc(struct elfs *e, struct reloc *patched_reloc, struct symbol *out_sym; bool klp; - if (!is_reloc_allowed(patched_reloc)) { - ERROR_FUNC(patched_reloc->sec->base, reloc_offset(patched_reloc), - "missing symbol for reference to %s+%ld", - patched_sym->name, addend); - return -1; - } - klp = klp_reloc_needed(patched_reloc); dbg_clone_reloc(sec, offset, patched_sym, addend, export, klp); @@ -1186,13 +1501,13 @@ static int clone_reloc(struct elfs *e, struct reloc *patched_reloc, /* * For strings, all references use section symbols, thanks to - * section_reference_needed(). clone_symbol() has cloned an empty + * convert_reloc_sym(). clone_symbol() has cloned an empty * version of the string section. Now copy the string itself. */ if (is_string_sec(patched_sym->sec)) { const char *str = patched_sym->sec->data->d_buf + addend; - __dbg_indent("\"%s\"", escape_str(str)); + __dbg_clone("\"%s\"", escape_str(str)); addend = elf_add_string(e->out, out_sym->sec, str); if (addend == -1) @@ -1239,6 +1554,7 @@ static int clone_sym_relocs(struct elfs *e, struct symbol *patched_sym) for_each_reloc(patched_rsec, patched_reloc) { unsigned long offset; + int ret; if (reloc_offset(patched_reloc) < start || reloc_offset(patched_reloc) >= end) @@ -1252,12 +1568,15 @@ static int clone_sym_relocs(struct elfs *e, struct symbol *patched_sym) !strcmp(patched_reloc->sym->sec->name, ".altinstr_aux")) continue; - if (convert_reloc_sym(e->patched, patched_reloc)) { + ret = convert_reloc_sym(e->patched, patched_reloc); + if (ret < 0) { ERROR_FUNC(patched_rsec->base, reloc_offset(patched_reloc), "failed to convert reloc sym '%s' to its proper format", patched_reloc->sym->name); return -1; } + if (ret > 0) + continue; offset = out_sym->offset + (reloc_offset(patched_reloc) - patched_sym->offset); @@ -1334,7 +1653,7 @@ static int create_fake_symbols(struct elf *elf) sec = find_section_by_name(elf, ".discard.annotate_data"); if (!sec || !sec->rsec) - return 0; + goto entsize; for_each_reloc(sec->rsec, reloc) { unsigned long offset, size; @@ -1366,7 +1685,7 @@ static int create_fake_symbols(struct elf *elf) /* * 2) Make symbols for sh_entsize, and simple arrays of pointers: */ - +entsize: for_each_sec(elf, sec) { unsigned int entry_size; unsigned long offset; @@ -1400,6 +1719,7 @@ static int create_fake_symbols(struct elf *elf) /* Keep a special section entry if it references an included function */ static bool should_keep_special_sym(struct elf *elf, struct symbol *sym) { + bool annotate_insn = !strcmp(sym->sec->name, ".discard.annotate_insn"); struct reloc *reloc; if (is_sec_sym(sym) || !sym->sec->rsec) @@ -1409,7 +1729,16 @@ static bool should_keep_special_sym(struct elf *elf, struct symbol *sym) if (convert_reloc_sym(elf, reloc)) continue; - if (is_func_sym(reloc->sym) && reloc->sym->included) + if (!reloc->sym->clone || is_undef_sym(reloc->sym->clone)) + continue; + + /* + * Keep special section references to cloned functions. + * In some cases annotate_insn can also reference cloned alt + * replacement fake symbols; keep those references as well. + */ + if (is_func_sym(reloc->sym) || + (annotate_insn && is_notype_sym(reloc->sym))) return true; } @@ -1553,15 +1882,28 @@ static int clone_special_section(struct elfs *e, struct section *patched_sec) /* Extract only the needed bits from special sections */ static int clone_special_sections(struct elfs *e) { - struct section *patched_sec; + struct section *sec, *annotate_insn = NULL; - for_each_sec(e->patched, patched_sec) { - if (is_special_section(patched_sec)) { - if (clone_special_section(e, patched_sec)) + for_each_sec(e->patched, sec) { + if (is_special_section(sec)) { + if (!strcmp(sec->name, ".discard.annotate_insn")) { + annotate_insn = sec; + continue; + } + if (clone_special_section(e, sec)) return -1; } } + /* + * Do .discard.annotate_insn last, it can reference other special + * sections (alt replacements) so they need to be cloned first. + */ + if (annotate_insn) { + if (clone_special_section(e, annotate_insn)) + return -1; + } + return 0; } @@ -1638,7 +1980,8 @@ static int create_klp_sections(struct elfs *e) unsigned long sympos; void *func_data; - if (!is_func_sym(sym) || sym->cold || !sym->clone || !sym->clone->changed) + if (!is_func_sym(sym) || is_cold_func(sym) || + !sym->clone || !sym->clone->changed) continue; /* allocate klp_func_ext */ @@ -1798,11 +2141,17 @@ static int copy_import_ns(struct elfs *e) int cmd_klp_diff(int argc, const char **argv) { struct elfs e = {0}; + int ret; argc = parse_options(argc, argv, klp_diff_options, klp_diff_usage, 0); if (argc != 3) usage_with_options(klp_diff_usage, klp_diff_options); + if (debug) { + debug_correlate = true; + debug_clone = true; + } + objname = argv[0]; e.orig = elf_open_read(argv[0], O_RDONLY); @@ -1824,7 +2173,10 @@ int cmd_klp_diff(int argc, const char **argv) if (correlate_symbols(&e)) return -1; - if (mark_changed_functions(&e)) + ret = mark_changed_functions(&e); + if (ret < 0) + return -1; + if (ret > 0) return 0; e.out = elf_create_file(&e.orig->ehdr, argv[2]); diff --git a/tools/objtool/objtool.c b/tools/objtool/objtool.c index 1c3622117c33..a4e139dee7e9 100644 --- a/tools/objtool/objtool.c +++ b/tools/objtool/objtool.c @@ -16,9 +16,6 @@ #include #include -bool debug; -int indent; - static struct objtool_file file; struct objtool_file *objtool_open_read(const char *filename) diff --git a/tools/objtool/trace.c b/tools/objtool/trace.c index 5dec44dab781..61c6aa302bc3 100644 --- a/tools/objtool/trace.c +++ b/tools/objtool/trace.c @@ -169,8 +169,8 @@ void trace_alt_begin(struct instruction *orig_insn, struct alternative *alt, */ TRACE_ALT_INFO_NOADDR(orig_insn, "/ ", "%s for instruction at 0x%lx <%s+0x%lx>", alt_name, - orig_insn->offset, orig_insn->sym->name, - orig_insn->offset - orig_insn->sym->offset); + orig_insn->offset, insn_sym(orig_insn)->name, + orig_insn->offset - insn_sym(orig_insn)->offset); } else { TRACE_ALT_INFO_NOADDR(orig_insn, "/ ", "%s", alt_name); } @@ -185,8 +185,8 @@ void trace_alt_begin(struct instruction *orig_insn, struct alternative *alt, if (orig_insn->type == INSN_NOP) { suffix[0] = (orig_insn->len == 5) ? 'q' : '\0'; TRACE_ADDR(orig_insn, "jmp%-3s %lx <%s+0x%lx>", suffix, - alt_insn->offset, alt_insn->sym->name, - alt_insn->offset - alt_insn->sym->offset); + alt_insn->offset, insn_sym(alt_insn)->name, + alt_insn->offset - insn_sym(alt_insn)->offset); } else { TRACE_ADDR(orig_insn, "nop%d", orig_insn->len); trace_depth--; diff --git a/tools/testing/selftests/futex/functional/robust_list.c b/tools/testing/selftests/futex/functional/robust_list.c index e7d1254e18ca..b3fab60181d5 100644 --- a/tools/testing/selftests/futex/functional/robust_list.c +++ b/tools/testing/selftests/futex/functional/robust_list.c @@ -27,12 +27,15 @@ #include "futextest.h" #include "../../kselftest_harness.h" +#include #include #include #include +#include #include #include #include +#include #include #include @@ -42,6 +45,10 @@ #define SLEEP_US 100 +#if __SIZEOF_LONG__ == 8 +# define BUILD_64 +#endif + static pthread_barrier_t barrier, barrier2; static int set_robust_list(struct robust_list_head *head, size_t len) @@ -54,6 +61,12 @@ static int get_robust_list(int pid, struct robust_list_head **head, size_t *len_ return syscall(SYS_get_robust_list, pid, head, len_ptr); } +static int sys_futex_robust_unlock(_Atomic(uint32_t) *uaddr, unsigned int op, int val, + void *list_op_pending, unsigned int val3) +{ + return syscall(SYS_futex, uaddr, op, val, NULL, list_op_pending, val3, 0); +} + /* * Basic lock struct, contains just the futex word and the robust list element * Real implementations have also a *prev to easily walk in the list @@ -549,4 +562,230 @@ TEST(test_circular_list) ksft_test_result_pass("%s\n", __func__); } +/* + * Below are tests for the fix of robust release race condition. Please read the following + * thread to learn more about the issue in the first place and why the following functions fix it: + * https://lore.kernel.org/lkml/20260316162316.356674433@kernel.org/ + */ + +/* + * Auxiliary code for binding the vDSO functions + */ +static void *get_vdso_func_addr(const char *function) +{ + const char *vdso_names[] = { + "linux-vdso.so.1", "linux-gate.so.1", "linux-vdso32.so.1", "linux-vdso64.so.1", + }; + + for (int i = 0; i < ARRAY_SIZE(vdso_names); i++) { + void *vdso = dlopen(vdso_names[i], RTLD_LAZY | RTLD_LOCAL | RTLD_NOLOAD); + + if (vdso) + return dlsym(vdso, function); + } + return NULL; +} + +/* + * These are the real vDSO function signatures: + * + * __vdso_futex_robust_list64_try_unlock(__u32 *lock, __u32 tid, __u64 *pop) + * __vdso_futex_robust_list32_try_unlock(__u32 *lock, __u32 tid, __u32 *pop) + * + * So for the generic entry point we need to use a void pointer as the last argument + */ +FIXTURE(vdso_unlock) +{ + uint32_t (*vdso)(_Atomic(uint32_t) *lock, uint32_t tid, void *pop); +}; + +FIXTURE_VARIANT(vdso_unlock) +{ + bool is_32; + char func_name[]; +}; + +FIXTURE_SETUP(vdso_unlock) +{ + self->vdso = get_vdso_func_addr(variant->func_name); +} + +FIXTURE_TEARDOWN(vdso_unlock) {} + +FIXTURE_VARIANT_ADD(vdso_unlock, 32) +{ + .func_name = "__vdso_futex_robust_list32_try_unlock", + .is_32 = true, +}; + +FIXTURE_VARIANT_ADD(vdso_unlock, 64) +{ + .func_name = "__vdso_futex_robust_list64_try_unlock", + .is_32 = false, +}; + +/* + * Test the vDSO robust_listXX_try_unlock() for the uncontended case. The virtual syscall should + * return the thread ID of the lock owner, the lock word must be 0 and the list_op_pending should + * be NULL. + */ +TEST_F(vdso_unlock, test_robust_try_unlock_uncontended) +{ + struct lock_struct lock = { .futex = 0 }; + _Atomic(unsigned int) *futex = &lock.futex; + struct robust_list_head head; + uintptr_t exp = (uintptr_t) NULL; + pid_t tid = gettid(); + int ret; + + if (!self->vdso) { + ksft_test_result_skip("%s not found\n", variant->func_name); + return; + } + + *futex = tid; + + ret = set_list(&head); + if (ret) + ksft_test_result_fail("set_robust_list error\n"); + + head.list_op_pending = &lock.list; + + ret = self->vdso(futex, tid, &head.list_op_pending); + + ASSERT_EQ(ret, tid); + ASSERT_EQ(*futex, 0); + + /* Check only the lower 32 bits for the 32-bit entry point */ + if (variant->is_32) { + exp = (uintptr_t)(unsigned long)&lock.list; + exp &= ~0xFFFFFFFFULL; + } + + ASSERT_EQ((uintptr_t)(unsigned long)head.list_op_pending, exp); +} + +/* + * If the lock is contended, the operation fails. The return value is the value found at the + * futex word (tid | FUTEX_WAITERS), the futex word is not modified and the list_op_pending is_32 + * not cleared. + */ +TEST_F(vdso_unlock, test_robust_try_unlock_contended) +{ + struct lock_struct lock = { .futex = 0 }; + _Atomic(unsigned int) *futex = &lock.futex; + struct robust_list_head head; + pid_t tid = gettid(); + int ret; + + if (!self->vdso) { + ksft_test_result_skip("%s not found\n", variant->func_name); + return; + } + + *futex = tid | FUTEX_WAITERS; + + ret = set_list(&head); + if (ret) + ksft_test_result_fail("set_robust_list error\n"); + + head.list_op_pending = &lock.list; + + ret = self->vdso(futex, tid, &head.list_op_pending); + + ASSERT_EQ(ret, tid | FUTEX_WAITERS); + ASSERT_EQ(*futex, tid | FUTEX_WAITERS); + ASSERT_EQ(head.list_op_pending, &lock.list); +} + +FIXTURE(futex_op) {}; + +FIXTURE_VARIANT(futex_op) +{ + unsigned int op; + unsigned int val3; +}; + +FIXTURE_SETUP(futex_op) {} + +FIXTURE_TEARDOWN(futex_op) {} + +FIXTURE_VARIANT_ADD(futex_op, wake) +{ + .op = FUTEX_WAKE, + .val3 = 0, +}; + +FIXTURE_VARIANT_ADD(futex_op, wake_bitset) +{ + .op = FUTEX_WAKE_BITSET, + .val3 = FUTEX_BITSET_MATCH_ANY, +}; + +FIXTURE_VARIANT_ADD(futex_op, unlock_pi) +{ + .op = FUTEX_UNLOCK_PI, + .val3 = 0, +}; + +FIXTURE_VARIANT_ADD(futex_op, wake32) +{ + .op = FUTEX_WAKE | FUTEX_ROBUST_LIST32, + .val3 = 0, +}; + +FIXTURE_VARIANT_ADD(futex_op, wake_bitset32) +{ + .op = FUTEX_WAKE_BITSET | FUTEX_ROBUST_LIST32, + .val3 = FUTEX_BITSET_MATCH_ANY, +}; + +FIXTURE_VARIANT_ADD(futex_op, unlock_pi32) +{ + .op = FUTEX_UNLOCK_PI | FUTEX_ROBUST_LIST32, + .val3 = 0, +}; + +/* + * The syscall should return the number of tasks waken (for this test, 0), clear the futex word and + * clear list_op_pending + */ +TEST_F(futex_op, test_futex_robust_unlock) +{ + struct lock_struct lock = { .futex = 0 }; + _Atomic(unsigned int) *futex = &lock.futex; + uintptr_t exp = (uintptr_t) NULL; + struct robust_list_head head; + pid_t tid = gettid(); + int ret; + +#ifndef BUILD_64 + if (!(variant->op & FUTEX_ROBUST_LIST32)) { + ksft_test_result_skip("Not supported for 32 bit build\n"); + return; + } +#endif + + *futex = tid | FUTEX_WAITERS; + + ret = set_list(&head); + if (ret) + ksft_test_result_fail("set_robust_list error\n"); + + head.list_op_pending = &lock.list; + + ret = sys_futex_robust_unlock(futex, FUTEX_ROBUST_UNLOCK | variant->op, tid, + &head.list_op_pending, variant->val3); + + ASSERT_EQ(ret, 0); + ASSERT_EQ(*futex, 0); + + if (variant->op & FUTEX_ROBUST_LIST32) { + exp = (uint64_t)(unsigned long)&lock.list; + exp &= ~0xFFFFFFFFULL; + } + + ASSERT_EQ((uintptr_t)(unsigned long)head.list_op_pending, exp); +} + TEST_HARNESS_MAIN diff --git a/tools/testing/selftests/futex/include/futextest.h b/tools/testing/selftests/futex/include/futextest.h index 3d48e9789d9f..df33f31d6994 100644 --- a/tools/testing/selftests/futex/include/futextest.h +++ b/tools/testing/selftests/futex/include/futextest.h @@ -38,6 +38,12 @@ typedef volatile u_int32_t futex_t; #ifndef FUTEX_CMP_REQUEUE_PI #define FUTEX_CMP_REQUEUE_PI 12 #endif +#ifndef FUTEX_ROBUST_UNLOCK +#define FUTEX_ROBUST_UNLOCK 512 +#endif +#ifndef FUTEX_ROBUST_LIST32 +#define FUTEX_ROBUST_LIST32 1024 +#endif #ifndef FUTEX_WAIT_REQUEUE_PI_PRIVATE #define FUTEX_WAIT_REQUEUE_PI_PRIVATE (FUTEX_WAIT_REQUEUE_PI | \ FUTEX_PRIVATE_FLAG) diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/selftests/membarrier/Makefile index fc840e06ff56..829f95c83515 100644 --- a/tools/testing/selftests/membarrier/Makefile +++ b/tools/testing/selftests/membarrier/Makefile @@ -1,8 +1,9 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -g $(KHDR_INCLUDES) +CFLAGS += -g $(KHDR_INCLUDES) -pthread -I../../../../tools/include LDLIBS += -lpthread TEST_GEN_PROGS := membarrier_test_single_thread \ - membarrier_test_multi_thread + membarrier_test_multi_thread \ + membarrier_rseq_stress include ../lib.mk diff --git a/tools/testing/selftests/membarrier/membarrier_rseq_stress.c b/tools/testing/selftests/membarrier/membarrier_rseq_stress.c new file mode 100644 index 000000000000..c188d7498610 --- /dev/null +++ b/tools/testing/selftests/membarrier/membarrier_rseq_stress.c @@ -0,0 +1,951 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Membarrier stress test for CFS throttle interactions. + * + * Reproducer for the interaction between CFS throttle and expedited membarrier. + */ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +/* -- Architecture-specific rseq signature -- */ +#if defined(__x86_64__) || defined(__i386__) +# define RSEQ_SIG 0x53053053U +#elif defined(__aarch64__) +# define RSEQ_SIG 0xd428bc00U +#elif defined(__powerpc__) || defined(__powerpc64__) +# define RSEQ_SIG 0x0f000000U +#elif defined(__s390__) || defined(__s390x__) +# define RSEQ_SIG 0x0c000000U +#else +# define RSEQ_SIG 0 +# define UNSUPPORTED_ARCH 1 +#endif + +/* -- rseq ABI (kernel uapi; define locally for portability) -- */ +#define RSEQ_CPU_ID_UNINITIALIZED ((__u32)-1) + +#include + +struct rseq_abi { + __u32 cpu_id_start; + __u32 cpu_id; + __u64 rseq_cs; + __u32 flags; + __u32 node_id; + __u32 mm_cid; + char end[0]; +} __aligned(32); + +/* -- membarrier constants (not in all distro headers) -- */ +#ifndef MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ +# define MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ (1 << 7) +#endif +#ifndef MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ +# define MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ (1 << 8) +#endif +#ifndef MEMBARRIER_CMD_FLAG_CPU +# define MEMBARRIER_CMD_FLAG_CPU (1 << 0) +#endif + +/* -- Test parameters -- */ +#define N_SIBLINGS 2000 +#define NEST_DEPTH 5 +static char g_cgroup_path[4096]; +static int use_cgroup_v2; + +#define CFS_QUOTA_US 1000 +#define CFS_PERIOD_US 5000 +#define N_HAMMER_PER_CPU 25 +#define N_BURNER_PER_CPU 50 +#define MAX_STRESS_CPUS 1024 +#define TEST_DURATION_SEC 20 + +/* Latency thresholds for the sentinel */ +#define LATENCY_WARN_MS 50 +#define LATENCY_CRITICAL_MS 200 + +/* Sentinel sampling interval */ +#define SENTINEL_INTERVAL_US 500 + +/* -- Shared globals -- */ +static atomic_int g_stop; +static atomic_int g_stop_sentinel; +static atomic_long g_max_latency_us; +static atomic_long g_interval_max_latency_us; +static atomic_long g_mb_ok; +static atomic_long g_mb_err; +static int g_ncpus_stress; +static int *g_stress_cpus; + +static atomic_int g_test_ready; + +/* Per-thread rseq ABI block registered with the kernel */ +static __thread struct rseq_abi tls_rseq + __attribute__((tls_model("initial-exec"))) __aligned(32) = { + .cpu_id = RSEQ_CPU_ID_UNINITIALIZED, +}; + +/* -- Utility -- */ +static int write_file(const char *path, const char *val) +{ + int fd = open(path, O_WRONLY | O_CLOEXEC); + + if (fd < 0) + return -errno; + + size_t len = strlen(val); + ssize_t r = write(fd, val, len); + + close(fd); + if (r < 0) + return -errno; + if ((size_t)r != len) + return -EIO; + return 0; +} + +static uint64_t monotonic_us(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000ULL + ts.tv_nsec / 1000ULL; +} + +static void update_max_latency(long lat) +{ + long old = atomic_load_explicit(&g_max_latency_us, memory_order_relaxed); + + while (lat > old) { + if (atomic_compare_exchange_weak_explicit(&g_max_latency_us, &old, lat, + memory_order_relaxed, memory_order_relaxed)) + break; + } + + old = atomic_load_explicit(&g_interval_max_latency_us, memory_order_relaxed); + while (lat > old) { + if (atomic_compare_exchange_weak_explicit(&g_interval_max_latency_us, &old, lat, + memory_order_relaxed, memory_order_relaxed)) + break; + } +} + +static void init_stress_cpus(void) +{ + cpu_set_t set; + int capacity = MAX_STRESS_CPUS; + + g_stress_cpus = malloc(capacity * sizeof(int)); + if (!g_stress_cpus) + ksft_exit_fail_msg("malloc failed for g_stress_cpus\n"); + + if (sched_getaffinity(0, sizeof(set), &set) < 0) + ksft_exit_fail_msg("sched_getaffinity failed\n"); + + for (int i = 0; i < CPU_SETSIZE && g_ncpus_stress < capacity; i++) { + if (CPU_ISSET(i, &set)) + g_stress_cpus[g_ncpus_stress++] = i; + } + + if (g_ncpus_stress == 0) + ksft_exit_skip("No CPUs available for stress test\n"); + + ksft_print_msg("Stressing %d CPUs discovered via affinity\n", g_ncpus_stress); +} + +/* -- rseq / membarrier helpers -- */ +static int rseq_register_thread(void) +{ + int r = syscall(SYS_rseq, &tls_rseq, sizeof(tls_rseq), 0, RSEQ_SIG); + + return (r == 0 || errno == EBUSY || errno == EINVAL) ? 0 : -1; +} + +static int rseq_register_thread_at(struct rseq_abi *rseq) +{ + int r = syscall(SYS_rseq, rseq, sizeof(*rseq), 0, RSEQ_SIG); + + return (r == 0 || errno == EBUSY || errno == EINVAL) ? 0 : -1; +} + +static int membarrier_register_rseq_mm(void) +{ + return syscall(SYS_membarrier, + MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ, 0, 0); +} + +/* -- cgroup helpers -- */ +static void rm_cgroup_recursive(const char *path) +{ + DIR *dir = opendir(path); + + if (!dir) + return; + struct dirent *entry; + + while ((entry = readdir(dir)) != NULL) { + if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) + continue; + if (entry->d_type == DT_DIR) { + char sub_path[4096]; + + snprintf(sub_path, sizeof(sub_path), "%s/%s", path, entry->d_name); + rm_cgroup_recursive(sub_path); + } + } + closedir(dir); + rmdir(path); +} + +static void cgroup_teardown(void); + +static int cgroup_setup(void) +{ + struct stat st; + + if (stat("/sys/fs/cgroup/cpu", &st) == 0) { + use_cgroup_v2 = 0; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/sys/fs/cgroup/cpu/membarrier_stress_test"); + } else if (stat("/dev/cgroup/cpu", &st) == 0) { + use_cgroup_v2 = 0; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/dev/cgroup/cpu/membarrier_stress_test"); + } else if (stat("/cgroup/cpu", &st) == 0) { + use_cgroup_v2 = 0; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/cgroup/cpu/membarrier_stress_test"); + } else if (stat("/sys/fs/cgroup/cgroup.controllers", &st) == 0) { + use_cgroup_v2 = 1; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/sys/fs/cgroup/membarrier_stress_test"); + } else { + ksft_print_msg("WARN: cgroup mount not found. Using v2 at /sys/fs/cgroup\n"); + use_cgroup_v2 = 1; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/sys/fs/cgroup/membarrier_stress_test"); + } + + /* Robust cleanup before setup */ + cgroup_teardown(); + + if (use_cgroup_v2) { + /* Enable cpu controller in root cgroup */ + if (write_file("/sys/fs/cgroup/cgroup.subtree_control", "+cpu") < 0) + ksft_print_msg("WARN: failed to enable cpu controller in /sys/fs/cgroup\n"); + } + + if (mkdir(g_cgroup_path, 0755) < 0 && errno != EEXIST) { + ksft_print_msg("mkdir base %s failed: %s\n", g_cgroup_path, strerror(errno)); + return -1; + } + + if (use_cgroup_v2) { + char ctrl_path[4096]; + + snprintf(ctrl_path, sizeof(ctrl_path), "%s/cgroup.subtree_control", g_cgroup_path); + if (write_file(ctrl_path, "+cpu") < 0) + ksft_print_msg("WARN: failed to enable cpu controller in %s\n", + g_cgroup_path); + } + + for (int i = 0; i < N_SIBLINGS; i++) { + char sibling_path[4096]; + + snprintf(sibling_path, sizeof(sibling_path), "%s/n%d", g_cgroup_path, i); + if (mkdir(sibling_path, 0755) < 0 && errno != EEXIST) { + ksft_print_msg("mkdir wide %s failed: %s\n", sibling_path, strerror(errno)); + return -1; + } + + if (use_cgroup_v2) { + char ctrl_path[4096]; + + snprintf(ctrl_path, sizeof(ctrl_path), + "%s/cgroup.subtree_control", sibling_path); + if (write_file(ctrl_path, "+cpu") < 0) + ksft_print_msg("WARN: failed to enable cpu controller in %s\n", + sibling_path); + } + + char current_path[4096]; + + snprintf(current_path, sizeof(current_path), "%s", sibling_path); + for (int j = 0; j < NEST_DEPTH; j++) { + snprintf(current_path + strlen(current_path), + sizeof(current_path) - strlen(current_path), "/d%d", j); + if (mkdir(current_path, 0755) < 0 && errno != EEXIST) { + ksft_print_msg("mkdir deep %s failed: %s\n", + current_path, strerror(errno)); + return -1; + } + + /* Enable for all but the leaf */ + if (use_cgroup_v2 && j < NEST_DEPTH - 1) { + char ctrl_path[4096]; + + snprintf(ctrl_path, sizeof(ctrl_path), "%s/cgroup.subtree_control", + current_path); + if (write_file(ctrl_path, "+cpu") < 0) + ksft_print_msg("WARN: cannot enable cpu controller in %s\n", + current_path); + } + } + } + + char quota[64], period[64], max_str[128]; + + snprintf(quota, sizeof(quota), "%d", CFS_QUOTA_US); + snprintf(period, sizeof(period), "%d", CFS_PERIOD_US); + snprintf(max_str, sizeof(max_str), "%d %d", CFS_QUOTA_US, CFS_PERIOD_US); + + if (use_cgroup_v2) { + char max_path[4096]; + + snprintf(max_path, sizeof(max_path), "%s/cpu.max", g_cgroup_path); + if (write_file(max_path, max_str) < 0) { + ksft_print_msg("ERROR: cannot write cpu.max at %s\n", max_path); + return -1; + } + ksft_print_msg("cgroup (v2) %s: cpu.max=%s\n", g_cgroup_path, max_str); + } else { + char quota_path[4096], period_path[4096]; + + snprintf(quota_path, sizeof(quota_path), "%s/cpu.cfs_quota_us", g_cgroup_path); + snprintf(period_path, sizeof(period_path), "%s/cpu.cfs_period_us", g_cgroup_path); + + if (write_file(period_path, period) < 0) { + ksft_print_msg("ERROR: cannot write cpu.cfs_period_us at %s\n", + period_path); + return -1; + } + if (write_file(quota_path, quota) < 0) { + ksft_print_msg("ERROR: cannot write cpu.cfs_quota_us at %s\n", quota_path); + return -1; + } + ksft_print_msg("cgroup (v1) %s: cpu.cfs_quota_us=%d cpu.cfs_period_us=%d\n", + g_cgroup_path, CFS_QUOTA_US, CFS_PERIOD_US); + } + + return 0; +} + +static int cgroup_add_pid_to_path(pid_t pid, const char *path) +{ + char buf[32], file_path[4096]; + + snprintf(buf, sizeof(buf), "%d", (int)pid); + if (use_cgroup_v2) { + snprintf(file_path, sizeof(file_path), "%s/cgroup.procs", path); + return write_file(file_path, buf); + } + /* In v1, try tasks first, fallback to cgroup.procs */ + snprintf(file_path, sizeof(file_path), "%s/tasks", path); + int r = write_file(file_path, buf); + + if (r < 0) { + snprintf(file_path, sizeof(file_path), "%s/cgroup.procs", path); + r = write_file(file_path, buf); + } + return r; +} + +static void cgroup_teardown(void) +{ + rm_cgroup_recursive(g_cgroup_path); +} + +static void cgroup_unthrottle(void) +{ + if (use_cgroup_v2) { + char max_path[4096]; + + snprintf(max_path, sizeof(max_path), "%s/cpu.max", g_cgroup_path); + write_file(max_path, "max"); + } else { + char quota_path[4096]; + + snprintf(quota_path, sizeof(quota_path), "%s/cpu.cfs_quota_us", g_cgroup_path); + write_file(quota_path, "-1"); + } +} + +/* -- CPU burner (inside throttled child process) -- */ +static void *burner_thread_fn(void *arg) +{ + struct rseq_abi my_rseq; + int cpu = (int)(uintptr_t)arg; + + memset(&my_rseq, 0, sizeof(my_rseq)); + my_rseq.cpu_id = RSEQ_CPU_ID_UNINITIALIZED; + + if (rseq_register_thread_at(&my_rseq) < 0) { + perror("rseq_register (burner)"); + return NULL; + } + + cpu_set_t set; + + CPU_ZERO(&set); + CPU_SET(cpu, &set); + if (sched_setaffinity(0, sizeof(set), &set) < 0) + perror("sched_setaffinity (burner)"); + + unsigned long sink = 0; + + while (!atomic_load_explicit(&g_stop, memory_order_relaxed)) { + sink++; + /* Prevent compiler from optimizing the loop away */ + asm volatile("" : "+g"(sink)); + } + + return NULL; +} + +static int burner_thread_fn_wrapper(void *arg) +{ + burner_thread_fn(arg); + return 0; +} + +static int leaf_child_fn(void *arg) +{ + int i = (int)(uintptr_t)arg; + int total_burners = g_ncpus_stress * N_BURNER_PER_CPU; + int n_threads_per_leaf = total_burners / N_SIBLINGS; + + if (i < (total_burners % N_SIBLINGS)) + n_threads_per_leaf++; + + prctl(PR_SET_PDEATHSIG, SIGTERM); + if (getppid() == 1) + _exit(1); + + char leaf_path[4096]; + + snprintf(leaf_path, sizeof(leaf_path), "%s/n%d", g_cgroup_path, i); + for (int j = 0; j < NEST_DEPTH; j++) + snprintf(leaf_path + strlen(leaf_path), + sizeof(leaf_path) - strlen(leaf_path), "/d%d", j); + + int r = cgroup_add_pid_to_path(getpid(), leaf_path); + + if (r < 0) { + char buf[512]; + int len = snprintf(buf, sizeof(buf), + "[leaf child %d] failed to join cgroup %s: err %d\n", + i, leaf_path, -r); + (void)!write(2, buf, len); + _exit(1); + } + + for (int j = 0; j < n_threads_per_leaf; j++) { + int cpu = g_stress_cpus[(i * n_threads_per_leaf + j) % g_ncpus_stress]; + + /* Allocate stack via mmap (bypasses heap) */ + size_t stack_size = 64 * 1024; + void *stack = mmap(NULL, stack_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (stack == MAP_FAILED) { + const char *msg = "mmap stack failed\n"; + (void)!write(2, msg, strlen(msg)); + _exit(1); + } + + /* Use raw clone to create a thread sharing the VM and thread group */ + pid_t pid = clone(burner_thread_fn_wrapper, stack + stack_size, + CLONE_VM | CLONE_THREAD | CLONE_SIGHAND, + (void *)(uintptr_t)cpu); + if (pid < 0) { + const char *msg = "clone burner failed\n"; + (void)!write(2, msg, strlen(msg)); + _exit(1); + } + } + + // Wait for SIGTERM + sigset_t mask; + + sigemptyset(&mask); + sigaddset(&mask, SIGTERM); + int sig; + + sigwait(&mask, &sig); + + _exit(0); +} + +struct leaf_info { + pid_t pid; + void *stack; +}; + +static int run_throttle_child(void *arg) +{ + (void)arg; + prctl(PR_SET_PDEATHSIG, SIGTERM); + if (getppid() == 1) + _exit(1); + + int n_leafs = N_SIBLINGS; + + /* Block signals before spawning to avoid missing early failures */ + sigset_t mask; + + sigemptyset(&mask); + sigaddset(&mask, SIGTERM); + sigaddset(&mask, SIGCHLD); + sigprocmask(SIG_BLOCK, &mask, NULL); + + /* Use mmap for tracking structures to avoid glibc heap usage */ + struct leaf_info *leaves = mmap(NULL, n_leafs * sizeof(struct leaf_info), + PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (leaves == MAP_FAILED) { + const char *msg = "mmap leaves array failed\n"; + (void)!write(2, msg, strlen(msg)); + _exit(1); + } + + for (int i = 0; i < n_leafs; i++) { + size_t stack_size = 64 * 1024; + void *stack = mmap(NULL, stack_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (stack == MAP_FAILED) { + const char *msg = "mmap leaf stack failed\n"; + (void)!write(2, msg, strlen(msg)); + _exit(1); + } + + leaves[i].stack = stack; + + pid_t pid = clone(leaf_child_fn, stack + stack_size, + CLONE_VM | SIGCHLD, (void *)(uintptr_t)i); + + if (pid < 0) { + const char *msg = "clone (leaf child) failed\n"; + (void)!write(2, msg, strlen(msg)); + + /* Clean up successfully spawned children */ + for (int j = 0; j < i; j++) { + kill(leaves[j].pid, SIGTERM); + waitpid(leaves[j].pid, NULL, 0); + munmap(leaves[j].stack, stack_size); + } + munmap(leaves, n_leafs * sizeof(struct leaf_info)); + + if (errno == EAGAIN) + _exit(4); + else + _exit(1); + } + leaves[i].pid = pid; + } + + int failed = 0; + + while (1) { + int sig; + + sigwait(&mask, &sig); + + if (sig == SIGTERM) { + break; + } else if (sig == SIGCHLD) { + int status; + pid_t pid; + + // Reap all dead children + while ((pid = waitpid(-1, &status, WNOHANG)) > 0) { + for (int i = 0; i < n_leafs; i++) { + if (leaves[i].pid == pid) { + leaves[i].pid = 0; + break; + } + } + if ((WIFEXITED(status) && WEXITSTATUS(status) != 0) || + WIFSIGNALED(status)) { + char buf[128]; + int len = snprintf(buf, sizeof(buf), + "[manager] child %d died unexpectedly (status %d)\n", + pid, WEXITSTATUS(status)); + (void)!write(2, buf, len); + failed = 1; + } + } + if (failed) + break; + } + } + + // Terminate all leaf kids + for (int i = 0; i < n_leafs; i++) { + if (leaves[i].pid > 0) + kill(leaves[i].pid, SIGTERM); + } + + for (int i = 0; i < n_leafs; i++) { + if (leaves[i].pid > 0) + waitpid(leaves[i].pid, NULL, 0); + munmap(leaves[i].stack, 64 * 1024); + } + + munmap(leaves, n_leafs * sizeof(struct leaf_info)); + + _exit(failed ? 1 : 0); +} + +/* -- Membarrier hammer thread -- */ +static void *hammer_thread_fn(void *arg) +{ + int target_cpu = *(int *)arg; + long local_ok = 0; + long local_err = 0; + int count = 0; + const int batch_size = 1024; + + if (rseq_register_thread() < 0) { + ksft_print_msg("[hammer] rseq_register failed: %s\n", strerror(errno)); + return NULL; + } + + membarrier_register_rseq_mm(); + + while (!atomic_load_explicit(&g_stop, memory_order_relaxed)) { + int r = syscall(SYS_membarrier, + MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ, + MEMBARRIER_CMD_FLAG_CPU, + target_cpu); + if (__builtin_expect(r == 0, 1)) + local_ok++; + else + local_err++; + + count++; + if (__builtin_expect(count >= batch_size, 0)) { + atomic_fetch_add_explicit(&g_mb_ok, local_ok, memory_order_relaxed); + atomic_fetch_add_explicit(&g_mb_err, local_err, memory_order_relaxed); + local_ok = 0; + local_err = 0; + count = 0; + } + } + + /* Flush any remaining counts on exit */ + if (local_ok > 0) + atomic_fetch_add_explicit(&g_mb_ok, local_ok, memory_order_relaxed); + if (local_err > 0) + atomic_fetch_add_explicit(&g_mb_err, local_err, memory_order_relaxed); + + return NULL; +} + +/* -- Latency sentinel -- */ +static void *sentinel_thread_fn(void *arg) +{ + (void)arg; + struct sched_param sp = { .sched_priority = 20 }; + + if (sched_setscheduler(0, SCHED_FIFO, &sp) < 0) + ksft_print_msg("WARN: no SCHED_FIFO for sentinel (less precise)\n"); + + while (!atomic_load_explicit(&g_test_ready, memory_order_relaxed) && + !atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) { + struct timespec ts = {0, 1000 * 1000}; /* 1ms */ + + clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL); + } + + uint64_t prev = monotonic_us(); + + while (!atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) { + struct timespec ts = { + .tv_sec = 0, + .tv_nsec = SENTINEL_INTERVAL_US * 1000L, + }; + clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL); + + uint64_t now = monotonic_us(); + long latency_us = (long)(now - prev) - SENTINEL_INTERVAL_US; + + prev = now; + + if (latency_us <= 0) + continue; + + update_max_latency(latency_us); + + if (latency_us > LATENCY_CRITICAL_MS * 1000L) { + ksft_print_msg("\n[SENTINEL] CRITICAL: %ld ms delay (lockup precursor!)\n", + latency_us / 1000); + } else if (latency_us > LATENCY_WARN_MS * 1000L) { + ksft_print_msg("\n[SENTINEL] WARN: %ld ms latency spike\n", + latency_us / 1000); + } + } + return NULL; +} + +/* -- Progress reporter -- */ +static void *reporter_thread_fn(void *arg) +{ + (void)arg; + int elapsed = 0; + + while (!atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) { + for (int i = 0; i < 5; i++) { + sleep(1); + if (atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) + break; + } + if (atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) + break; + elapsed += 5; + long interval_max = atomic_exchange_explicit(&g_interval_max_latency_us, + 0, memory_order_relaxed); + + ksft_print_msg("[%3ds] mb: ok=%-10ld err=%-8ld | max_lat=%ld us\n", + elapsed, + atomic_load(&g_mb_ok), + atomic_load(&g_mb_err), + interval_max); + } + return NULL; +} + +/* -- Main -- */ +int main(void) +{ + ksft_print_header(); +#ifdef UNSUPPORTED_ARCH + ksft_exit_skip("Unsupported architecture\n"); +#endif + ksft_set_plan(1); + + if (geteuid() != 0) + ksft_exit_skip("Must run as root (cgroup + SCHED_FIFO)\n"); + + init_stress_cpus(); + + ksft_print_msg("=== membarrier rseq + CFS unthrottle stress ===\n"); + ksft_print_msg("Stressing CPUs: %d\n", g_ncpus_stress); + ksft_print_msg("Quota: %d/%d us (~%d unthrottles/sec/CPU)\n", + CFS_QUOTA_US, CFS_PERIOD_US, + 1000000 / CFS_PERIOD_US); + ksft_print_msg("Hammer threads: %d per CPU (%d total)\n", + N_HAMMER_PER_CPU, g_ncpus_stress * N_HAMMER_PER_CPU); + ksft_print_msg("Duration: %d seconds\n\n", TEST_DURATION_SEC); + + if (cgroup_setup() < 0) { + cgroup_teardown(); + ksft_exit_skip("cgroup_setup failed (missing permissions or v2 ctrls?)\n"); + } + + if (rseq_register_thread() < 0) { + ksft_print_msg("rseq_register (%s) failed: %s\n", __func__, strerror(errno)); + cgroup_teardown(); + ksft_exit_skip("rseq syscall failed or not available\n"); + } + if (membarrier_register_rseq_mm() < 0) { + ksft_print_msg("MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ: %s\n" + "Kernel >= 5.10 with CONFIG_RSEQ required.\n", + strerror(errno)); + cgroup_teardown(); + ksft_exit_skip("membarrier register failed\n"); + } + ksft_print_msg("rseq membarrier registered OK\n"); + + sigset_t sigmask; + + sigemptyset(&sigmask); + sigaddset(&sigmask, SIGTERM); + sigprocmask(SIG_BLOCK, &sigmask, NULL); + + void *stack = malloc(1024 * 1024); + + if (!stack) { + perror("malloc stack"); + cgroup_teardown(); + ksft_exit_fail_msg("Malloc stack failed\n"); + } + pid_t child = clone(run_throttle_child, stack + 1024 * 1024, CLONE_VM | SIGCHLD, NULL); + + if (child < 0) { + perror("clone"); + cgroup_teardown(); + ksft_exit_fail_msg("Clone failed\n"); + } + + sigprocmask(SIG_UNBLOCK, &sigmask, NULL); + ksft_print_msg("Throttle child PID %d started\n", child); + + int n_threads = g_ncpus_stress * N_HAMMER_PER_CPU + 2; + pthread_t *threads = (pthread_t *)calloc(n_threads, sizeof(pthread_t)); + int *cpuargs = (int *)calloc(g_ncpus_stress * N_HAMMER_PER_CPU, sizeof(int)); + + if (!threads || !cpuargs) { + perror("calloc"); + kill(child, SIGTERM); + waitpid(child, NULL, 0); + cgroup_teardown(); + ksft_exit_fail_msg("Thread allocation failed\n"); + } + + int ti = 0, ai = 0; + int r; + + ksft_print_msg("Creating sentinel thread...\n"); + r = pthread_create(&threads[ti], NULL, sentinel_thread_fn, NULL); + if (r != 0) { + kill(child, SIGTERM); + waitpid(child, NULL, 0); + cgroup_teardown(); + free(threads); + free(cpuargs); + free(g_stress_cpus); + ksft_exit_fail_msg("pthread_create (sentinel) failed: %s\n", strerror(r)); + } + ti++; + + ksft_print_msg("Creating reporter thread...\n"); + r = pthread_create(&threads[ti], NULL, reporter_thread_fn, NULL); + if (r != 0) { + atomic_store(&g_stop_sentinel, 1); + pthread_join(threads[0], NULL); + kill(child, SIGTERM); + waitpid(child, NULL, 0); + cgroup_teardown(); + free(threads); + free(cpuargs); + free(g_stress_cpus); + ksft_exit_fail_msg("pthread_create (reporter) failed: %s\n", strerror(r)); + } + ti++; + + ksft_print_msg("Creating %d hammer threads...\n", g_ncpus_stress * N_HAMMER_PER_CPU); + for (int i = 0; i < g_ncpus_stress; i++) { + int cpu = g_stress_cpus[i]; + + for (int j = 0; j < N_HAMMER_PER_CPU; j++) { + cpuargs[ai] = cpu; + r = pthread_create(&threads[ti], NULL, hammer_thread_fn, &cpuargs[ai]); + if (r != 0) { + ksft_print_msg("pthread_create failed at thread %d: %s\n", + ti, strerror(r)); + + atomic_store(&g_stop_sentinel, 1); + pthread_join(threads[0], NULL); + pthread_join(threads[1], NULL); + + atomic_store(&g_stop, 1); + for (int k = 2; k < ti; k++) + pthread_join(threads[k], NULL); + + kill(child, SIGTERM); + waitpid(child, NULL, 0); + cgroup_teardown(); + + free(threads); + free(cpuargs); + free(g_stress_cpus); + + if (r == EAGAIN) + ksft_exit_skip("Resource limits prevent threads\n"); + else + ksft_exit_fail_msg("Failed to create hammer thread\n"); + } + ti++; + ai++; + } + } + + ksft_print_msg("All threads running. Tip: monitor dmesg for lockups\n\n"); + + atomic_store_explicit(&g_test_ready, 1, memory_order_relaxed); + int child_failed = 0; + int child_status = 0; + + for (int i = 0; i < TEST_DURATION_SEC; i++) { + sleep(1); + int r = waitpid(child, &child_status, WNOHANG); + + if (r == child) { + child_failed = 1; + break; + } + } + + atomic_store(&g_stop_sentinel, 1); + pthread_join(threads[0], NULL); + pthread_join(threads[1], NULL); + + atomic_store(&g_stop, 1); + + /* Unthrottle to allow children to exit quickly */ + cgroup_unthrottle(); + + if (!child_failed) { + kill(child, SIGTERM); + waitpid(child, NULL, 0); + } + for (int i = 2; i < ti; i++) + pthread_join(threads[i], NULL); + + long max_lat = atomic_load(&g_max_latency_us); + long total_ok = atomic_load(&g_mb_ok); + long total_err = atomic_load(&g_mb_err); + + ksft_print_msg("\n=== RESULTS ===\n"); + ksft_print_msg("membarrier syscalls : %ld ok %ld errors\n", total_ok, total_err); + ksft_print_msg("Max scheduler latency: %ld us (%ld ms)\n", max_lat, max_lat / 1000); + cgroup_teardown(); + free(threads); + free(cpuargs); + free(g_stress_cpus); + + if (child_failed) { + if (WIFEXITED(child_status) && WEXITSTATUS(child_status) == 4) + ksft_exit_skip("Manager child skipped (resource limits?)\n"); + ksft_test_result_fail("membarrier_rseq_stress: Manager child died early\n"); + ksft_exit_fail(); + } else if (total_ok == 0) { + ksft_test_result_fail("membarrier_rseq_stress: No successful membarrier calls\n"); + ksft_exit_fail(); + } else if (total_err > 0) { + ksft_test_result_fail("membarrier_rseq_stress: syscall errors\n"); + ksft_exit_fail(); + } else if (max_lat > LATENCY_CRITICAL_MS * 1000L) { + ksft_test_result_fail("membarrier_rseq_stress: LOCKUP PRECURSOR\n"); + ksft_exit_fail(); + } else if (max_lat > LATENCY_WARN_MS * 1000L) { + ksft_test_result_fail("membarrier_rseq_stress: significant latency spike\n"); + ksft_exit_fail(); + } else { + ksft_test_result_pass("membarrier_rseq_stress\n"); + ksft_exit_pass(); + } + + return 0; +}