diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst index 49387d823619..f3b605285a87 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2071,41 +2071,7 @@ call. Because RCU avoids interrupting idle CPUs, it is illegal to execute an RCU read-side critical section on an idle CPU. (Kernels built with -``CONFIG_PROVE_RCU=y`` will splat if you try it.) The RCU_NONIDLE() -macro and ``_rcuidle`` event tracing is provided to work around this -restriction. In addition, rcu_is_watching() may be used to test -whether or not it is currently legal to run RCU read-side critical -sections on this CPU. I learned of the need for diagnostics on the one -hand and RCU_NONIDLE() on the other while inspecting idle-loop code. -Steven Rostedt supplied ``_rcuidle`` event tracing, which is used quite -heavily in the idle loop. However, there are some restrictions on the -code placed within RCU_NONIDLE(): - -#. Blocking is prohibited. In practice, this is not a serious - restriction given that idle tasks are prohibited from blocking to - begin with. -#. Although nesting RCU_NONIDLE() is permitted, they cannot nest - indefinitely deeply. However, given that they can be nested on the - order of a million deep, even on 32-bit systems, this should not be a - serious restriction. This nesting limit would probably be reached - long after the compiler OOMed or the stack overflowed. -#. Any code path that enters RCU_NONIDLE() must sequence out of that - same RCU_NONIDLE(). For example, the following is grossly - illegal: - - :: - - 1 RCU_NONIDLE({ - 2 do_something(); - 3 goto bad_idea; /* BUG!!! */ - 4 do_something_else();}); - 5 bad_idea: - - - It is just as illegal to transfer control into the middle of - RCU_NONIDLE()'s argument. Yes, in theory, you could transfer in - as long as you also transferred out, but in practice you could also - expect to get sharply worded review comments. +``CONFIG_PROVE_RCU=y`` will splat if you try it.) It is similarly socially unacceptable to interrupt an ``nohz_full`` CPU running in userspace. RCU must therefore track ``nohz_full`` userspace diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst index 8eddef28d3a1..e488c8e557a9 100644 --- a/Documentation/RCU/whatisRCU.rst +++ b/Documentation/RCU/whatisRCU.rst @@ -1117,7 +1117,6 @@ All: lockdep-checked RCU utility APIs:: RCU_LOCKDEP_WARN rcu_sleep_check - RCU_NONIDLE All: Unchecked RCU-protected pointer access:: diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 5eb8a581caf8..3bdf89def04e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -429,6 +429,9 @@ arm64.nosme [ARM64] Unconditionally disable Scalable Matrix Extension support + arm64.nomops [ARM64] Unconditionally disable Memory Copy and Memory + Set instructions support + ataflop= [HW,M68k] atarimouse= [HW,MOUSE] Atari Mouse @@ -2113,6 +2116,16 @@ disable Do not enable intel_pstate as the default scaling driver for the supported processors + active + Use intel_pstate driver to bypass the scaling + governors layer of cpufreq and provides it own + algorithms for p-state selection. There are two + P-state selection algorithms provided by + intel_pstate in the active mode: powersave and + performance. The way they both operate depends + on whether or not the hardware managed P-states + (HWP) feature has been enabled in the processor + and possibly on the processor model. passive Use intel_pstate as a scaling driver, but configure it to work with generic cpufreq governors (instead of @@ -2547,12 +2560,13 @@ If the value is 0 (the default), KVM will pick a period based on the ratio, such that a page is zapped after 1 hour on average. - kvm-amd.nested= [KVM,AMD] Allow nested virtualization in KVM/SVM. - Default is 1 (enabled) + kvm-amd.nested= [KVM,AMD] Control nested virtualization feature in + KVM/SVM. Default is 1 (enabled). - kvm-amd.npt= [KVM,AMD] Disable nested paging (virtualized MMU) - for all guests. - Default is 1 (enabled) if in 64-bit or 32-bit PAE mode. + kvm-amd.npt= [KVM,AMD] Control KVM's use of Nested Page Tables, + a.k.a. Two-Dimensional Page Tables. Default is 1 + (enabled). Disable by KVM if hardware lacks support + for NPT. kvm-arm.mode= [KVM,ARM] Select one of KVM/arm64's modes of operation. @@ -2598,30 +2612,33 @@ Format: Default: 5 - kvm-intel.ept= [KVM,Intel] Disable extended page tables - (virtualized MMU) support on capable Intel chips. - Default is 1 (enabled) + kvm-intel.ept= [KVM,Intel] Control KVM's use of Extended Page Tables, + a.k.a. Two-Dimensional Page Tables. Default is 1 + (enabled). Disable by KVM if hardware lacks support + for EPT. kvm-intel.emulate_invalid_guest_state= - [KVM,Intel] Disable emulation of invalid guest state. - Ignored if kvm-intel.enable_unrestricted_guest=1, as - guest state is never invalid for unrestricted guests. - This param doesn't apply to nested guests (L2), as KVM - never emulates invalid L2 guest state. - Default is 1 (enabled) + [KVM,Intel] Control whether to emulate invalid guest + state. Ignored if kvm-intel.enable_unrestricted_guest=1, + as guest state is never invalid for unrestricted + guests. This param doesn't apply to nested guests (L2), + as KVM never emulates invalid L2 guest state. + Default is 1 (enabled). kvm-intel.flexpriority= - [KVM,Intel] Disable FlexPriority feature (TPR shadow). - Default is 1 (enabled) + [KVM,Intel] Control KVM's use of FlexPriority feature + (TPR shadow). Default is 1 (enabled). Disalbe by KVM if + hardware lacks support for it. kvm-intel.nested= - [KVM,Intel] Enable VMX nesting (nVMX). - Default is 0 (disabled) + [KVM,Intel] Control nested virtualization feature in + KVM/VMX. Default is 1 (enabled). kvm-intel.unrestricted_guest= - [KVM,Intel] Disable unrestricted guest feature - (virtualized real and unpaged mode) on capable - Intel chips. Default is 1 (enabled) + [KVM,Intel] Control KVM's use of unrestricted guest + feature (virtualized real and unpaged mode). Default + is 1 (enabled). Disable by KVM if EPT is disabled or + hardware lacks support for it. kvm-intel.vmentry_l1d_flush=[KVM,Intel] Mitigation for L1 Terminal Fault CVE-2018-3620. @@ -2635,9 +2652,10 @@ Default is cond (do L1 cache flush in specific instances) - kvm-intel.vpid= [KVM,Intel] Disable Virtual Processor Identification - feature (tagged TLBs) on capable Intel chips. - Default is 1 (enabled) + kvm-intel.vpid= [KVM,Intel] Control KVM's use of Virtual Processor + Identification feature (tagged TLBs). Default is 1 + (enabled). Disable by KVM if hardware lacks support + for it. l1d_flush= [X86,INTEL] Control mitigation for L1D based snooping vulnerability. @@ -3419,6 +3437,10 @@ [HW] Make the MicroTouch USB driver use raw coordinates ('y', default) or cooked coordinates ('n') + mtrr=debug [X86] + Enable printing debug information related to MTRR + registers at boot time. + mtrr_chunk_size=nn[KMG] [X86] used for mtrr cleanup. It is largest continuous chunk that could hold holes aka. UC entries. @@ -4732,43 +4754,6 @@ the propagation of recent CPU-hotplug changes up the rcu_node combining tree. - rcutree.use_softirq= [KNL] - If set to zero, move all RCU_SOFTIRQ processing to - per-CPU rcuc kthreads. Defaults to a non-zero - value, meaning that RCU_SOFTIRQ is used by default. - Specify rcutree.use_softirq=0 to use rcuc kthreads. - - But note that CONFIG_PREEMPT_RT=y kernels disable - this kernel boot parameter, forcibly setting it - to zero. - - rcutree.rcu_fanout_exact= [KNL] - Disable autobalancing of the rcu_node combining - tree. This is used by rcutorture, and might - possibly be useful for architectures having high - cache-to-cache transfer latencies. - - rcutree.rcu_fanout_leaf= [KNL] - Change the number of CPUs assigned to each - leaf rcu_node structure. Useful for very - large systems, which will choose the value 64, - and for NUMA systems with large remote-access - latencies, which will choose a value aligned - with the appropriate hardware boundaries. - - rcutree.rcu_min_cached_objs= [KNL] - Minimum number of objects which are cached and - maintained per one CPU. Object size is equal - to PAGE_SIZE. The cache allows to reduce the - pressure to page allocator, also it makes the - whole algorithm to behave better in low memory - condition. - - rcutree.rcu_delay_page_cache_fill_msec= [KNL] - Set the page-cache refill delay (in milliseconds) - in response to low-memory conditions. The range - of permitted values is in the range 0:100000. - rcutree.jiffies_till_first_fqs= [KNL] Set delay from grace-period initialization to first attempt to force quiescent states. @@ -4807,21 +4792,6 @@ When RCU_NOCB_CPU is set, also adjust the priority of NOCB callback kthreads. - rcutree.rcu_divisor= [KNL] - Set the shift-right count to use to compute - the callback-invocation batch limit bl from - the number of callbacks queued on this CPU. - The result will be bounded below by the value of - the rcutree.blimit kernel parameter. Every bl - callbacks, the softirq handler will exit in - order to allow the CPU to do other work. - - Please note that this callback-invocation batch - limit applies only to non-offloaded callback - invocation. Offloaded callbacks are instead - invoked in the context of an rcuoc kthread, which - scheduler will preempt as it does any other task. - rcutree.nocb_nobypass_lim_per_jiffy= [KNL] On callback-offloaded (rcu_nocbs) CPUs, RCU reduces the lock contention that would @@ -4835,14 +4805,6 @@ the ->nocb_bypass queue. The definition of "too many" is supplied by this kernel boot parameter. - rcutree.rcu_nocb_gp_stride= [KNL] - Set the number of NOCB callback kthreads in - each group, which defaults to the square root - of the number of CPUs. Larger numbers reduce - the wakeup overhead on the global grace-period - kthread, but increases that same overhead on - each group's NOCB grace-period kthread. - rcutree.qhimark= [KNL] Set threshold of queued RCU callbacks beyond which batch limiting is disabled. @@ -4860,6 +4822,56 @@ on rcutree.qhimark at boot time and to zero to disable more aggressive help enlistment. + rcutree.rcu_delay_page_cache_fill_msec= [KNL] + Set the page-cache refill delay (in milliseconds) + in response to low-memory conditions. The range + of permitted values is in the range 0:100000. + + rcutree.rcu_divisor= [KNL] + Set the shift-right count to use to compute + the callback-invocation batch limit bl from + the number of callbacks queued on this CPU. + The result will be bounded below by the value of + the rcutree.blimit kernel parameter. Every bl + callbacks, the softirq handler will exit in + order to allow the CPU to do other work. + + Please note that this callback-invocation batch + limit applies only to non-offloaded callback + invocation. Offloaded callbacks are instead + invoked in the context of an rcuoc kthread, which + scheduler will preempt as it does any other task. + + rcutree.rcu_fanout_exact= [KNL] + Disable autobalancing of the rcu_node combining + tree. This is used by rcutorture, and might + possibly be useful for architectures having high + cache-to-cache transfer latencies. + + rcutree.rcu_fanout_leaf= [KNL] + Change the number of CPUs assigned to each + leaf rcu_node structure. Useful for very + large systems, which will choose the value 64, + and for NUMA systems with large remote-access + latencies, which will choose a value aligned + with the appropriate hardware boundaries. + + rcutree.rcu_min_cached_objs= [KNL] + Minimum number of objects which are cached and + maintained per one CPU. Object size is equal + to PAGE_SIZE. The cache allows to reduce the + pressure to page allocator, also it makes the + whole algorithm to behave better in low memory + condition. + + rcutree.rcu_nocb_gp_stride= [KNL] + Set the number of NOCB callback kthreads in + each group, which defaults to the square root + of the number of CPUs. Larger numbers reduce + the wakeup overhead on the global grace-period + kthread, but increases that same overhead on + each group's NOCB grace-period kthread. + rcutree.rcu_kick_kthreads= [KNL] Cause the grace-period kthread to get an extra wake_up() if it sleeps three times longer than @@ -4867,6 +4879,13 @@ This wake_up() will be accompanied by a WARN_ONCE() splat and an ftrace_dump(). + rcutree.rcu_resched_ns= [KNL] + Limit the time spend invoking a batch of RCU + callbacks to the specified number of nanoseconds. + By default, this limit is checked only once + every 32 callbacks in order to limit the pain + inflicted by local_clock() overhead. + rcutree.rcu_unlock_delay= [KNL] In CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels, this specifies an rcu_read_unlock()-time delay @@ -4881,6 +4900,16 @@ rcu_node tree with an eye towards determining why a new grace period has not yet started. + rcutree.use_softirq= [KNL] + If set to zero, move all RCU_SOFTIRQ processing to + per-CPU rcuc kthreads. Defaults to a non-zero + value, meaning that RCU_SOFTIRQ is used by default. + Specify rcutree.use_softirq=0 to use rcuc kthreads. + + But note that CONFIG_PREEMPT_RT=y kernels disable + this kernel boot parameter, forcibly setting it + to zero. + rcuscale.gp_async= [KNL] Measure performance of asynchronous grace-period primitives such as call_rcu(). @@ -5083,8 +5112,17 @@ rcutorture.stall_cpu_block= [KNL] Sleep while stalling if set. This will result - in warnings from preemptible RCU in addition - to any other stall-related activity. + in warnings from preemptible RCU in addition to + any other stall-related activity. Note that + in kernels built with CONFIG_PREEMPTION=n and + CONFIG_PREEMPT_COUNT=y, this parameter will + cause the CPU to pass through a quiescent state. + Given CONFIG_PREEMPTION=n, this will suppress + RCU CPU stall warnings, but will instead result + in scheduling-while-atomic splats. + + Use of this module parameter results in splats. + rcutorture.stall_cpu_holdoff= [KNL] Time to wait (s) after boot before inducing stall. diff --git a/Documentation/admin-guide/perf/hisi-pmu.rst b/Documentation/admin-guide/perf/hisi-pmu.rst index 546979360513..e0174d20809a 100644 --- a/Documentation/admin-guide/perf/hisi-pmu.rst +++ b/Documentation/admin-guide/perf/hisi-pmu.rst @@ -56,14 +56,14 @@ Example usage of perf:: For HiSilicon uncore PMU v2 whose identifier is 0x30, the topology is the same as PMU v1, but some new functions are added to the hardware. -(a) L3C PMU supports filtering by core/thread within the cluster which can be +1. L3C PMU supports filtering by core/thread within the cluster which can be specified as a bitmap:: $# perf stat -a -e hisi_sccl3_l3c0/config=0x02,tt_core=0x3/ sleep 5 This will only count the operations from core/thread 0 and 1 in this cluster. -(b) Tracetag allow the user to chose to count only read, write or atomic +2. Tracetag allow the user to chose to count only read, write or atomic operations via the tt_req parameeter in perf. The default value counts all operations. tt_req is 3bits, 3'b100 represents read operations, 3'b101 represents write operations, 3'b110 represents atomic store operations and @@ -73,14 +73,16 @@ represents write operations, 3'b110 represents atomic store operations and This will only count the read operations in this cluster. -(c) Datasrc allows the user to check where the data comes from. It is 5 bits. +3. Datasrc allows the user to check where the data comes from. It is 5 bits. Some important codes are as follows: -5'b00001: comes from L3C in this die; -5'b01000: comes from L3C in the cross-die; -5'b01001: comes from L3C which is in another socket; -5'b01110: comes from the local DDR; -5'b01111: comes from the cross-die DDR; -5'b10000: comes from cross-socket DDR; + +- 5'b00001: comes from L3C in this die; +- 5'b01000: comes from L3C in the cross-die; +- 5'b01001: comes from L3C which is in another socket; +- 5'b01110: comes from the local DDR; +- 5'b01111: comes from the cross-die DDR; +- 5'b10000: comes from cross-socket DDR; + etc, it is mainly helpful to find that the data source is nearest from the CPU cores. If datasrc_cfg is used in the multi-chips, the datasrc_skt shall be configured in perf command:: @@ -88,15 +90,25 @@ configured in perf command:: $# perf stat -a -e hisi_sccl3_l3c0/config=0xb9,datasrc_cfg=0xE/, hisi_sccl3_l3c0/config=0xb9,datasrc_cfg=0xF/ sleep 5 -(d)Some HiSilicon SoCs encapsulate multiple CPU and IO dies. Each CPU die +4. Some HiSilicon SoCs encapsulate multiple CPU and IO dies. Each CPU die contains several Compute Clusters (CCLs). The I/O dies are called Super I/O clusters (SICL) containing multiple I/O clusters (ICLs). Each CCL/ICL in the SoC has a unique ID. Each ID is 11bits, include a 6-bit SCCL-ID and 5-bit CCL/ICL-ID. For I/O die, the ICL-ID is followed by: -5'b00000: I/O_MGMT_ICL; -5'b00001: Network_ICL; -5'b00011: HAC_ICL; -5'b10000: PCIe_ICL; + +- 5'b00000: I/O_MGMT_ICL; +- 5'b00001: Network_ICL; +- 5'b00011: HAC_ICL; +- 5'b10000: PCIe_ICL; + +5. uring_channel: UC PMU events 0x47~0x59 supports filtering by tx request +uring channel. It is 2 bits. Some important codes are as follows: + +- 2'b11: count the events which sent to the uring_ext (MATA) channel; +- 2'b01: is the same as 2'b11; +- 2'b10: count the events which sent to the uring (non-MATA) channel; +- 2'b00: default value, count the events which sent to the both uring and + uring_ext channel; Users could configure IDs to count data come from specific CCL/ICL, by setting srcid_cmd & srcid_msk, and data desitined for specific CCL/ICL by setting diff --git a/Documentation/arm/arm.rst b/Documentation/arch/arm/arm.rst similarity index 100% rename from Documentation/arm/arm.rst rename to Documentation/arch/arm/arm.rst diff --git a/Documentation/arm/booting.rst b/Documentation/arch/arm/booting.rst similarity index 100% rename from Documentation/arm/booting.rst rename to Documentation/arch/arm/booting.rst diff --git a/Documentation/arm/cluster-pm-race-avoidance.rst b/Documentation/arch/arm/cluster-pm-race-avoidance.rst similarity index 100% rename from Documentation/arm/cluster-pm-race-avoidance.rst rename to Documentation/arch/arm/cluster-pm-race-avoidance.rst diff --git a/Documentation/arm/features.rst b/Documentation/arch/arm/features.rst similarity index 100% rename from Documentation/arm/features.rst rename to Documentation/arch/arm/features.rst diff --git a/Documentation/arm/firmware.rst b/Documentation/arch/arm/firmware.rst similarity index 100% rename from Documentation/arm/firmware.rst rename to Documentation/arch/arm/firmware.rst diff --git a/Documentation/arm/google/chromebook-boot-flow.rst b/Documentation/arch/arm/google/chromebook-boot-flow.rst similarity index 100% rename from Documentation/arm/google/chromebook-boot-flow.rst rename to Documentation/arch/arm/google/chromebook-boot-flow.rst diff --git a/Documentation/arm/index.rst b/Documentation/arch/arm/index.rst similarity index 100% rename from Documentation/arm/index.rst rename to Documentation/arch/arm/index.rst diff --git a/Documentation/arm/interrupts.rst b/Documentation/arch/arm/interrupts.rst similarity index 100% rename from Documentation/arm/interrupts.rst rename to Documentation/arch/arm/interrupts.rst diff --git a/Documentation/arm/ixp4xx.rst b/Documentation/arch/arm/ixp4xx.rst similarity index 100% rename from Documentation/arm/ixp4xx.rst rename to Documentation/arch/arm/ixp4xx.rst diff --git a/Documentation/arm/kernel_mode_neon.rst b/Documentation/arch/arm/kernel_mode_neon.rst similarity index 100% rename from Documentation/arm/kernel_mode_neon.rst rename to Documentation/arch/arm/kernel_mode_neon.rst diff --git a/Documentation/arm/kernel_user_helpers.rst b/Documentation/arch/arm/kernel_user_helpers.rst similarity index 100% rename from Documentation/arm/kernel_user_helpers.rst rename to Documentation/arch/arm/kernel_user_helpers.rst diff --git a/Documentation/arm/keystone/knav-qmss.rst b/Documentation/arch/arm/keystone/knav-qmss.rst similarity index 100% rename from Documentation/arm/keystone/knav-qmss.rst rename to Documentation/arch/arm/keystone/knav-qmss.rst diff --git a/Documentation/arm/keystone/overview.rst b/Documentation/arch/arm/keystone/overview.rst similarity index 100% rename from Documentation/arm/keystone/overview.rst rename to Documentation/arch/arm/keystone/overview.rst diff --git a/Documentation/arm/marvell.rst b/Documentation/arch/arm/marvell.rst similarity index 100% rename from Documentation/arm/marvell.rst rename to Documentation/arch/arm/marvell.rst diff --git a/Documentation/arm/mem_alignment.rst b/Documentation/arch/arm/mem_alignment.rst similarity index 100% rename from Documentation/arm/mem_alignment.rst rename to Documentation/arch/arm/mem_alignment.rst diff --git a/Documentation/arm/memory.rst b/Documentation/arch/arm/memory.rst similarity index 100% rename from Documentation/arm/memory.rst rename to Documentation/arch/arm/memory.rst diff --git a/Documentation/arm/microchip.rst b/Documentation/arch/arm/microchip.rst similarity index 100% rename from Documentation/arm/microchip.rst rename to Documentation/arch/arm/microchip.rst diff --git a/Documentation/arm/netwinder.rst b/Documentation/arch/arm/netwinder.rst similarity index 100% rename from Documentation/arm/netwinder.rst rename to Documentation/arch/arm/netwinder.rst diff --git a/Documentation/arm/nwfpe/index.rst b/Documentation/arch/arm/nwfpe/index.rst similarity index 100% rename from Documentation/arm/nwfpe/index.rst rename to Documentation/arch/arm/nwfpe/index.rst diff --git a/Documentation/arm/nwfpe/netwinder-fpe.rst b/Documentation/arch/arm/nwfpe/netwinder-fpe.rst similarity index 100% rename from Documentation/arm/nwfpe/netwinder-fpe.rst rename to Documentation/arch/arm/nwfpe/netwinder-fpe.rst diff --git a/Documentation/arm/nwfpe/notes.rst b/Documentation/arch/arm/nwfpe/notes.rst similarity index 100% rename from Documentation/arm/nwfpe/notes.rst rename to Documentation/arch/arm/nwfpe/notes.rst diff --git a/Documentation/arm/nwfpe/nwfpe.rst b/Documentation/arch/arm/nwfpe/nwfpe.rst similarity index 100% rename from Documentation/arm/nwfpe/nwfpe.rst rename to Documentation/arch/arm/nwfpe/nwfpe.rst diff --git a/Documentation/arm/nwfpe/todo.rst b/Documentation/arch/arm/nwfpe/todo.rst similarity index 100% rename from Documentation/arm/nwfpe/todo.rst rename to Documentation/arch/arm/nwfpe/todo.rst diff --git a/Documentation/arm/omap/dss.rst b/Documentation/arch/arm/omap/dss.rst similarity index 100% rename from Documentation/arm/omap/dss.rst rename to Documentation/arch/arm/omap/dss.rst diff --git a/Documentation/arm/omap/index.rst b/Documentation/arch/arm/omap/index.rst similarity index 100% rename from Documentation/arm/omap/index.rst rename to Documentation/arch/arm/omap/index.rst diff --git a/Documentation/arm/omap/omap.rst b/Documentation/arch/arm/omap/omap.rst similarity index 100% rename from Documentation/arm/omap/omap.rst rename to Documentation/arch/arm/omap/omap.rst diff --git a/Documentation/arm/omap/omap_pm.rst b/Documentation/arch/arm/omap/omap_pm.rst similarity index 100% rename from Documentation/arm/omap/omap_pm.rst rename to Documentation/arch/arm/omap/omap_pm.rst diff --git a/Documentation/arm/porting.rst b/Documentation/arch/arm/porting.rst similarity index 100% rename from Documentation/arm/porting.rst rename to Documentation/arch/arm/porting.rst diff --git a/Documentation/arm/pxa/mfp.rst b/Documentation/arch/arm/pxa/mfp.rst similarity index 100% rename from Documentation/arm/pxa/mfp.rst rename to Documentation/arch/arm/pxa/mfp.rst diff --git a/Documentation/arm/sa1100/assabet.rst b/Documentation/arch/arm/sa1100/assabet.rst similarity index 100% rename from Documentation/arm/sa1100/assabet.rst rename to Documentation/arch/arm/sa1100/assabet.rst diff --git a/Documentation/arm/sa1100/cerf.rst b/Documentation/arch/arm/sa1100/cerf.rst similarity index 100% rename from Documentation/arm/sa1100/cerf.rst rename to Documentation/arch/arm/sa1100/cerf.rst diff --git a/Documentation/arm/sa1100/index.rst b/Documentation/arch/arm/sa1100/index.rst similarity index 100% rename from Documentation/arm/sa1100/index.rst rename to Documentation/arch/arm/sa1100/index.rst diff --git a/Documentation/arm/sa1100/lart.rst b/Documentation/arch/arm/sa1100/lart.rst similarity index 100% rename from Documentation/arm/sa1100/lart.rst rename to Documentation/arch/arm/sa1100/lart.rst diff --git a/Documentation/arm/sa1100/serial_uart.rst b/Documentation/arch/arm/sa1100/serial_uart.rst similarity index 100% rename from Documentation/arm/sa1100/serial_uart.rst rename to Documentation/arch/arm/sa1100/serial_uart.rst diff --git a/Documentation/arm/samsung/bootloader-interface.rst b/Documentation/arch/arm/samsung/bootloader-interface.rst similarity index 100% rename from Documentation/arm/samsung/bootloader-interface.rst rename to Documentation/arch/arm/samsung/bootloader-interface.rst diff --git a/Documentation/arm/samsung/clksrc-change-registers.awk b/Documentation/arch/arm/samsung/clksrc-change-registers.awk similarity index 100% rename from Documentation/arm/samsung/clksrc-change-registers.awk rename to Documentation/arch/arm/samsung/clksrc-change-registers.awk diff --git a/Documentation/arm/samsung/gpio.rst b/Documentation/arch/arm/samsung/gpio.rst similarity index 100% rename from Documentation/arm/samsung/gpio.rst rename to Documentation/arch/arm/samsung/gpio.rst diff --git a/Documentation/arm/samsung/index.rst b/Documentation/arch/arm/samsung/index.rst similarity index 100% rename from Documentation/arm/samsung/index.rst rename to Documentation/arch/arm/samsung/index.rst diff --git a/Documentation/arm/samsung/overview.rst b/Documentation/arch/arm/samsung/overview.rst similarity index 100% rename from Documentation/arm/samsung/overview.rst rename to Documentation/arch/arm/samsung/overview.rst diff --git a/Documentation/arm/setup.rst b/Documentation/arch/arm/setup.rst similarity index 100% rename from Documentation/arm/setup.rst rename to Documentation/arch/arm/setup.rst diff --git a/Documentation/arm/spear/overview.rst b/Documentation/arch/arm/spear/overview.rst similarity index 100% rename from Documentation/arm/spear/overview.rst rename to Documentation/arch/arm/spear/overview.rst diff --git a/Documentation/arm/sti/overview.rst b/Documentation/arch/arm/sti/overview.rst similarity index 100% rename from Documentation/arm/sti/overview.rst rename to Documentation/arch/arm/sti/overview.rst diff --git a/Documentation/arm/sti/stih407-overview.rst b/Documentation/arch/arm/sti/stih407-overview.rst similarity index 100% rename from Documentation/arm/sti/stih407-overview.rst rename to Documentation/arch/arm/sti/stih407-overview.rst diff --git a/Documentation/arm/sti/stih418-overview.rst b/Documentation/arch/arm/sti/stih418-overview.rst similarity index 100% rename from Documentation/arm/sti/stih418-overview.rst rename to Documentation/arch/arm/sti/stih418-overview.rst diff --git a/Documentation/arm/stm32/overview.rst b/Documentation/arch/arm/stm32/overview.rst similarity index 100% rename from Documentation/arm/stm32/overview.rst rename to Documentation/arch/arm/stm32/overview.rst diff --git a/Documentation/arm/stm32/stm32-dma-mdma-chaining.rst b/Documentation/arch/arm/stm32/stm32-dma-mdma-chaining.rst similarity index 100% rename from Documentation/arm/stm32/stm32-dma-mdma-chaining.rst rename to Documentation/arch/arm/stm32/stm32-dma-mdma-chaining.rst diff --git a/Documentation/arm/stm32/stm32f429-overview.rst b/Documentation/arch/arm/stm32/stm32f429-overview.rst similarity index 100% rename from Documentation/arm/stm32/stm32f429-overview.rst rename to Documentation/arch/arm/stm32/stm32f429-overview.rst diff --git a/Documentation/arm/stm32/stm32f746-overview.rst b/Documentation/arch/arm/stm32/stm32f746-overview.rst similarity index 100% rename from Documentation/arm/stm32/stm32f746-overview.rst rename to Documentation/arch/arm/stm32/stm32f746-overview.rst diff --git a/Documentation/arm/stm32/stm32f769-overview.rst b/Documentation/arch/arm/stm32/stm32f769-overview.rst similarity index 100% rename from Documentation/arm/stm32/stm32f769-overview.rst rename to Documentation/arch/arm/stm32/stm32f769-overview.rst diff --git a/Documentation/arm/stm32/stm32h743-overview.rst b/Documentation/arch/arm/stm32/stm32h743-overview.rst similarity index 100% rename from Documentation/arm/stm32/stm32h743-overview.rst rename to Documentation/arch/arm/stm32/stm32h743-overview.rst diff --git a/Documentation/arm/stm32/stm32h750-overview.rst b/Documentation/arch/arm/stm32/stm32h750-overview.rst similarity index 100% rename from Documentation/arm/stm32/stm32h750-overview.rst rename to Documentation/arch/arm/stm32/stm32h750-overview.rst diff --git a/Documentation/arm/stm32/stm32mp13-overview.rst b/Documentation/arch/arm/stm32/stm32mp13-overview.rst similarity index 100% rename from Documentation/arm/stm32/stm32mp13-overview.rst rename to Documentation/arch/arm/stm32/stm32mp13-overview.rst diff --git a/Documentation/arm/stm32/stm32mp151-overview.rst b/Documentation/arch/arm/stm32/stm32mp151-overview.rst similarity index 100% rename from Documentation/arm/stm32/stm32mp151-overview.rst rename to Documentation/arch/arm/stm32/stm32mp151-overview.rst diff --git a/Documentation/arm/stm32/stm32mp157-overview.rst b/Documentation/arch/arm/stm32/stm32mp157-overview.rst similarity index 100% rename from Documentation/arm/stm32/stm32mp157-overview.rst rename to Documentation/arch/arm/stm32/stm32mp157-overview.rst diff --git a/Documentation/arm/sunxi.rst b/Documentation/arch/arm/sunxi.rst similarity index 100% rename from Documentation/arm/sunxi.rst rename to Documentation/arch/arm/sunxi.rst diff --git a/Documentation/arm/sunxi/clocks.rst b/Documentation/arch/arm/sunxi/clocks.rst similarity index 100% rename from Documentation/arm/sunxi/clocks.rst rename to Documentation/arch/arm/sunxi/clocks.rst diff --git a/Documentation/arm/swp_emulation.rst b/Documentation/arch/arm/swp_emulation.rst similarity index 100% rename from Documentation/arm/swp_emulation.rst rename to Documentation/arch/arm/swp_emulation.rst diff --git a/Documentation/arm/tcm.rst b/Documentation/arch/arm/tcm.rst similarity index 100% rename from Documentation/arm/tcm.rst rename to Documentation/arch/arm/tcm.rst diff --git a/Documentation/arm/uefi.rst b/Documentation/arch/arm/uefi.rst similarity index 100% rename from Documentation/arm/uefi.rst rename to Documentation/arch/arm/uefi.rst diff --git a/Documentation/arm/vfp/release-notes.rst b/Documentation/arch/arm/vfp/release-notes.rst similarity index 100% rename from Documentation/arm/vfp/release-notes.rst rename to Documentation/arch/arm/vfp/release-notes.rst diff --git a/Documentation/arm/vlocks.rst b/Documentation/arch/arm/vlocks.rst similarity index 100% rename from Documentation/arm/vlocks.rst rename to Documentation/arch/arm/vlocks.rst diff --git a/Documentation/arch/index.rst b/Documentation/arch/index.rst index 80ee31016584..21e3d0b61004 100644 --- a/Documentation/arch/index.rst +++ b/Documentation/arch/index.rst @@ -10,7 +10,7 @@ implementation. :maxdepth: 2 arc/index - ../arm/index + arm/index ../arm64/index ia64/index ../loongarch/index diff --git a/Documentation/arm64/acpi_object_usage.rst b/Documentation/arm64/acpi_object_usage.rst index 484ef9676653..1da22200fdf8 100644 --- a/Documentation/arm64/acpi_object_usage.rst +++ b/Documentation/arm64/acpi_object_usage.rst @@ -17,16 +17,37 @@ For ACPI on arm64, tables also fall into the following categories: - Recommended: BERT, EINJ, ERST, HEST, PCCT, SSDT - - Optional: BGRT, CPEP, CSRT, DBG2, DRTM, ECDT, FACS, FPDT, IBFT, - IORT, MCHI, MPST, MSCT, NFIT, PMTT, RASF, SBST, SLIT, SPMI, SRAT, - STAO, TCPA, TPM2, UEFI, XENV + - Optional: AGDI, BGRT, CEDT, CPEP, CSRT, DBG2, DRTM, ECDT, FACS, FPDT, + HMAT, IBFT, IORT, MCHI, MPAM, MPST, MSCT, NFIT, PMTT, PPTT, RASF, SBST, + SDEI, SLIT, SPMI, SRAT, STAO, TCPA, TPM2, UEFI, XENV - - Not supported: BOOT, DBGP, DMAR, ETDT, HPET, IVRS, LPIT, MSDM, OEMx, - PSDT, RSDT, SLIC, WAET, WDAT, WDRT, WPBT + - Not supported: AEST, APMT, BOOT, DBGP, DMAR, ETDT, HPET, IVRS, LPIT, + MSDM, OEMx, PDTT, PSDT, RAS2, RSDT, SLIC, WAET, WDAT, WDRT, WPBT ====== ======================================================================== Table Usage for ARMv8 Linux ====== ======================================================================== +AEST Signature Reserved (signature == "AEST") + + **Arm Error Source Table** + + This table informs the OS of any error nodes in the system that are + compliant with the Arm RAS architecture. + +AGDI Signature Reserved (signature == "AGDI") + + **Arm Generic diagnostic Dump and Reset Device Interface Table** + + This table describes a non-maskable event, that is used by the platform + firmware, to request the OS to generate a diagnostic dump and reset the device. + +APMT Signature Reserved (signature == "APMT") + + **Arm Performance Monitoring Table** + + This table describes the properties of PMU support implmented by + components in the system. + BERT Section 18.3 (signature == "BERT") **Boot Error Record Table** @@ -47,6 +68,13 @@ BGRT Section 5.2.22 (signature == "BGRT") Optional, not currently supported, with no real use-case for an ARM server. +CEDT Signature Reserved (signature == "CEDT") + + **CXL Early Discovery Table** + + This table allows the OS to discover any CXL Host Bridges and the Host + Bridge registers. + CPEP Section 5.2.18 (signature == "CPEP") **Corrected Platform Error Polling table** @@ -184,6 +212,15 @@ HEST Section 18.3.2 (signature == "HEST") Must be supplied if RAS support is provided by the platform. It is recommended this table be supplied. +HMAT Section 5.2.28 (signature == "HMAT") + + **Heterogeneous Memory Attribute Table** + + This table describes the memory attributes, such as memory side cache + attributes and bandwidth and latency details, related to Memory Proximity + Domains. The OS uses this information to optimize the system memory + configuration. + HPET Signature Reserved (signature == "HPET") **High Precision Event timer Table** @@ -241,6 +278,13 @@ MCHI Signature Reserved (signature == "MCHI") Optional, not currently supported. +MPAM Signature Reserved (signature == "MPAM") + + **Memory Partitioning And Monitoring table** + + This table allows the OS to discover the MPAM controls implemented by + the subsystems. + MPST Section 5.2.21 (signature == "MPST") **Memory Power State Table** @@ -281,18 +325,39 @@ PCCT Section 14.1 (signature == "PCCT) Recommend for use on arm64; use of PCC is recommended when using CPPC to control performance and power for platform processors. +PDTT Section 5.2.29 (signature == "PDTT") + + **Platform Debug Trigger Table** + + This table describes PCC channels used to gather debug logs of + non-architectural features. + + PMTT Section 5.2.21.12 (signature == "PMTT") **Platform Memory Topology Table** Optional, not currently supported. +PPTT Section 5.2.30 (signature == "PPTT") + + **Processor Properties Topology Table** + + This table provides the processor and cache topology. + PSDT Section 5.2.11.3 (signature == "PSDT") **Persistent System Description Table** Obsolete table, will not be supported. +RAS2 Section 5.2.21 (signature == "RAS2") + + **RAS Features 2 table** + + This table provides interfaces for the RAS capabilities implemented in + the platform. + RASF Section 5.2.20 (signature == "RASF") **RAS Feature table** @@ -318,6 +383,12 @@ SBST Section 5.2.14 (signature == "SBST") Optional, not currently supported. +SDEI Signature Reserved (signature == "SDEI") + + **Software Delegated Exception Interface table** + + This table advertises the presence of the SDEI interface. + SLIC Signature Reserved (signature == "SLIC") **Software LIcensing table** diff --git a/Documentation/arm64/arm-acpi.rst b/Documentation/arm64/arm-acpi.rst index 47ecb9930dde..37ec5e9b1575 100644 --- a/Documentation/arm64/arm-acpi.rst +++ b/Documentation/arm64/arm-acpi.rst @@ -1,40 +1,41 @@ -===================== -ACPI on ARMv8 Servers -===================== +=================== +ACPI on Arm systems +=================== -ACPI can be used for ARMv8 general purpose servers designed to follow -the ARM SBSA (Server Base System Architecture) [0] and SBBR (Server -Base Boot Requirements) [1] specifications. Please note that the SBBR -can be retrieved simply by visiting [1], but the SBSA is currently only -available to those with an ARM login due to ARM IP licensing concerns. +ACPI can be used for Armv8 and Armv9 systems designed to follow +the BSA (Arm Base System Architecture) [0] and BBR (Arm +Base Boot Requirements) [1] specifications. Both BSA and BBR are publicly +accessible documents. +Arm Servers, in addition to being BSA compliant, comply with a set +of rules defined in SBSA (Server Base System Architecture) [2]. -The ARMv8 kernel implements the reduced hardware model of ACPI version +The Arm kernel implements the reduced hardware model of ACPI version 5.1 or later. Links to the specification and all external documents it refers to are managed by the UEFI Forum. The specification is available at http://www.uefi.org/specifications and documents referenced by the specification can be found via http://www.uefi.org/acpi. -If an ARMv8 system does not meet the requirements of the SBSA and SBBR, +If an Arm system does not meet the requirements of the BSA and BBR, or cannot be described using the mechanisms defined in the required ACPI specifications, then ACPI may not be a good fit for the hardware. While the documents mentioned above set out the requirements for building -industry-standard ARMv8 servers, they also apply to more than one operating +industry-standard Arm systems, they also apply to more than one operating system. The purpose of this document is to describe the interaction between -ACPI and Linux only, on an ARMv8 system -- that is, what Linux expects of +ACPI and Linux only, on an Arm system -- that is, what Linux expects of ACPI and what ACPI can expect of Linux. -Why ACPI on ARM? +Why ACPI on Arm? ---------------- Before examining the details of the interface between ACPI and Linux, it is useful to understand why ACPI is being used. Several technologies already exist in Linux for describing non-enumerable hardware, after all. In this -section we summarize a blog post [2] from Grant Likely that outlines the -reasoning behind ACPI on ARMv8 servers. Actually, we snitch a good portion +section we summarize a blog post [3] from Grant Likely that outlines the +reasoning behind ACPI on Arm systems. Actually, we snitch a good portion of the summary text almost directly, to be honest. -The short form of the rationale for ACPI on ARM is: +The short form of the rationale for ACPI on Arm is: - ACPI’s byte code (AML) allows the platform to encode hardware behavior, while DT explicitly does not support this. For hardware vendors, being @@ -47,7 +48,7 @@ The short form of the rationale for ACPI on ARM is: - In the enterprise server environment, ACPI has established bindings (such as for RAS) which are currently used in production systems. DT does not. - Such bindings could be defined in DT at some point, but doing so means ARM + Such bindings could be defined in DT at some point, but doing so means Arm and x86 would end up using completely different code paths in both firmware and the kernel. @@ -108,7 +109,7 @@ recent version of the kernel. Relationship with Device Tree ----------------------------- -ACPI support in drivers and subsystems for ARMv8 should never be mutually +ACPI support in drivers and subsystems for Arm should never be mutually exclusive with DT support at compile time. At boot time the kernel will only use one description method depending on @@ -121,11 +122,11 @@ time). Booting using ACPI tables ------------------------- -The only defined method for passing ACPI tables to the kernel on ARMv8 +The only defined method for passing ACPI tables to the kernel on Arm is via the UEFI system configuration table. Just so it is explicit, this means that ACPI is only supported on platforms that boot via UEFI. -When an ARMv8 system boots, it can either have DT information, ACPI tables, +When an Arm system boots, it can either have DT information, ACPI tables, or in some very unusual cases, both. If no command line parameters are used, the kernel will try to use DT for device enumeration; if there is no DT present, the kernel will try to use ACPI tables, but only if they are present. @@ -169,7 +170,7 @@ hardware reduced mode must be set to zero. For the ACPI core to operate properly, and in turn provide the information the kernel needs to configure devices, it expects to find the following -tables (all section numbers refer to the ACPI 6.1 specification): +tables (all section numbers refer to the ACPI 6.5 specification): - RSDP (Root System Description Pointer), section 5.2.5 @@ -184,20 +185,76 @@ tables (all section numbers refer to the ACPI 6.1 specification): - GTDT (Generic Timer Description Table), section 5.2.24 + - PPTT (Processor Properties Topology Table), section 5.2.30 + + - DBG2 (DeBuG port table 2), section 5.2.6, specifically Table 5-6. + + - APMT (Arm Performance Monitoring unit Table), section 5.2.6, specifically Table 5-6. + + - AGDI (Arm Generic diagnostic Dump and Reset Device Interface Table), section 5.2.6, specifically Table 5-6. + - If PCI is supported, the MCFG (Memory mapped ConFiGuration - Table), section 5.2.6, specifically Table 5-31. + Table), section 5.2.6, specifically Table 5-6. - If booting without a console= kernel parameter is supported, the SPCR (Serial Port Console Redirection table), - section 5.2.6, specifically Table 5-31. + section 5.2.6, specifically Table 5-6. - If necessary to describe the I/O topology, SMMUs and GIC ITSs, the IORT (Input Output Remapping Table, section 5.2.6, specifically - Table 5-31). + Table 5-6). + + - If NUMA is supported, the following tables are required: + + - SRAT (System Resource Affinity Table), section 5.2.16 + + - SLIT (System Locality distance Information Table), section 5.2.17 + + - If NUMA is supported, and the system contains heterogeneous memory, + the HMAT (Heterogeneous Memory Attribute Table), section 5.2.28. + + - If the ACPI Platform Error Interfaces are required, the following + tables are conditionally required: + + - BERT (Boot Error Record Table, section 18.3.1) + + - EINJ (Error INJection table, section 18.6.1) + + - ERST (Error Record Serialization Table, section 18.5) + + - HEST (Hardware Error Source Table, section 18.3.2) + + - SDEI (Software Delegated Exception Interface table, section 5.2.6, + specifically Table 5-6) + + - AEST (Arm Error Source Table, section 5.2.6, + specifically Table 5-6) + + - RAS2 (ACPI RAS2 feature table, section 5.2.21) + + - If the system contains controllers using PCC channel, the + PCCT (Platform Communications Channel Table), section 14.1 + + - If the system contains a controller to capture board-level system state, + and communicates with the host via PCC, the PDTT (Platform Debug Trigger + Table), section 5.2.29. + + - If NVDIMM is supported, the NFIT (NVDIMM Firmware Interface Table), section 5.2.26 + + - If video framebuffer is present, the BGRT (Boot Graphics Resource Table), section 5.2.23 + + - If IPMI is implemented, the SPMI (Server Platform Management Interface), + section 5.2.6, specifically Table 5-6. + + - If the system contains a CXL Host Bridge, the CEDT (CXL Early Discovery + Table), section 5.2.6, specifically Table 5-6. + + - If the system supports MPAM, the MPAM (Memory Partitioning And Monitoring table), section 5.2.6, + specifically Table 5-6. + + - If the system lacks persistent storage, the IBFT (ISCSI Boot Firmware + Table), section 5.2.6, specifically Table 5-6. - - If NUMA is supported, the SRAT (System Resource Affinity Table) - and SLIT (System Locality distance Information Table), sections - 5.2.16 and 5.2.17, respectively. If the above tables are not all present, the kernel may or may not be able to boot properly since it may not be able to configure all of the @@ -269,16 +326,14 @@ Drivers should look for device properties in the _DSD object ONLY; the _DSD object is described in the ACPI specification section 6.2.5, but this only describes how to define the structure of an object returned via _DSD, and how specific data structures are defined by specific UUIDs. Linux should -only use the _DSD Device Properties UUID [5]: +only use the _DSD Device Properties UUID [4]: - UUID: daffd814-6eba-4d8c-8a91-bc9bbf4aa301 - - https://www.uefi.org/sites/default/files/resources/_DSD-device-properties-UUID.pdf - -The UEFI Forum provides a mechanism for registering device properties [4] -so that they may be used across all operating systems supporting ACPI. -Device properties that have not been registered with the UEFI Forum should -not be used. +Common device properties can be registered by creating a pull request to [4] so +that they may be used across all operating systems supporting ACPI. +Device properties that have not been registered with the UEFI Forum can be used +but not as "uefi-" common properties. Before creating new device properties, check to be sure that they have not been defined before and either registered in the Linux kernel documentation @@ -306,7 +361,7 @@ process. Once registration and review have been completed, the kernel provides an interface for looking up device properties in a manner independent of -whether DT or ACPI is being used. This API should be used [6]; it can +whether DT or ACPI is being used. This API should be used [5]; it can eliminate some duplication of code paths in driver probing functions and discourage divergence between DT bindings and ACPI device properties. @@ -448,15 +503,15 @@ ASWG ---- The ACPI specification changes regularly. During the year 2014, for instance, version 5.1 was released and version 6.0 substantially completed, with most of -the changes being driven by ARM-specific requirements. Proposed changes are +the changes being driven by Arm-specific requirements. Proposed changes are presented and discussed in the ASWG (ACPI Specification Working Group) which is a part of the UEFI Forum. The current version of the ACPI specification -is 6.1 release in January 2016. +is 6.5 release in August 2022. Participation in this group is open to all UEFI members. Please see http://www.uefi.org/workinggroup for details on group membership. -It is the intent of the ARMv8 ACPI kernel code to follow the ACPI specification +It is the intent of the Arm ACPI kernel code to follow the ACPI specification as closely as possible, and to only implement functionality that complies with the released standards from UEFI ASWG. As a practical matter, there will be vendors that provide bad ACPI tables or violate the standards in some way. @@ -470,12 +525,12 @@ likely be willing to assist in submitting ECRs. Linux Code ---------- -Individual items specific to Linux on ARM, contained in the Linux +Individual items specific to Linux on Arm, contained in the Linux source code, are in the list that follows: ACPI_OS_NAME This macro defines the string to be returned when - an ACPI method invokes the _OS method. On ARM64 + an ACPI method invokes the _OS method. On Arm systems, this macro will be "Linux" by default. The command line parameter acpi_os= can be used to set it to some other value. The @@ -490,31 +545,23 @@ Documentation/arm64/acpi_object_usage.rst. References ---------- -[0] http://silver.arm.com - document ARM-DEN-0029, or newer: - "Server Base System Architecture", version 2.3, dated 27 Mar 2014 +[0] https://developer.arm.com/documentation/den0094/latest + document Arm-DEN-0094: "Arm Base System Architecture", version 1.0C, dated 6 Oct 2022 -[1] http://infocenter.arm.com/help/topic/com.arm.doc.den0044a/Server_Base_Boot_Requirements.pdf - Document ARM-DEN-0044A, or newer: "Server Base Boot Requirements, System - Software on ARM Platforms", dated 16 Aug 2014 +[1] https://developer.arm.com/documentation/den0044/latest + Document Arm-DEN-0044: "Arm Base Boot Requirements", version 2.0G, dated 15 Apr 2022 -[2] http://www.secretlab.ca/archives/151, +[2] https://developer.arm.com/documentation/den0029/latest + Document Arm-DEN-0029: "Arm Server Base System Architecture", version 7.1, dated 06 Oct 2022 + +[3] http://www.secretlab.ca/archives/151, 10 Jan 2015, Copyright (c) 2015, Linaro Ltd., written by Grant Likely. -[3] AMD ACPI for Seattle platform documentation - http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/Seattle_ACPI_Guide.pdf +[4] _DSD (Device Specific Data) Implementation Guide + https://github.com/UEFI/DSD-Guide/blob/main/dsd-guide.pdf - -[4] http://www.uefi.org/acpi - please see the link for the "ACPI _DSD Device - Property Registry Instructions" - -[5] http://www.uefi.org/acpi - please see the link for the "_DSD (Device - Specific Data) Implementation Guide" - -[6] Kernel code for the unified device +[5] Kernel code for the unified device property interface can be found in include/linux/property.h and drivers/base/property.c. diff --git a/Documentation/arm64/booting.rst b/Documentation/arm64/booting.rst index ffeccdd6bdac..b57776a68f15 100644 --- a/Documentation/arm64/booting.rst +++ b/Documentation/arm64/booting.rst @@ -379,6 +379,38 @@ Before jumping into the kernel, the following conditions must be met: - SMCR_EL2.EZT0 (bit 30) must be initialised to 0b1. + For CPUs with Memory Copy and Memory Set instructions (FEAT_MOPS): + + - If the kernel is entered at EL1 and EL2 is present: + + - HCRX_EL2.MSCEn (bit 11) must be initialised to 0b1. + + For CPUs with the Extended Translation Control Register feature (FEAT_TCR2): + + - If EL3 is present: + + - SCR_EL3.TCR2En (bit 43) must be initialised to 0b1. + + - If the kernel is entered at EL1 and EL2 is present: + + - HCRX_EL2.TCR2En (bit 14) must be initialised to 0b1. + + For CPUs with the Stage 1 Permission Indirection Extension feature (FEAT_S1PIE): + + - If EL3 is present: + + - SCR_EL3.PIEn (bit 45) must be initialised to 0b1. + + - If the kernel is entered at EL1 and EL2 is present: + + - HFGRTR_EL2.nPIR_EL1 (bit 58) must be initialised to 0b1. + + - HFGWTR_EL2.nPIR_EL1 (bit 58) must be initialised to 0b1. + + - HFGRTR_EL2.nPIRE0_EL1 (bit 57) must be initialised to 0b1. + + - HFGRWR_EL2.nPIRE0_EL1 (bit 57) must be initialised to 0b1. + The requirements described above for CPU mode, caches, MMUs, architected timers, coherency and system registers apply to all CPUs. All CPUs must enter the kernel in the same exception level. Where the values documented diff --git a/Documentation/arm64/cpu-feature-registers.rst b/Documentation/arm64/cpu-feature-registers.rst index c7adc7897df6..4e4625f2455f 100644 --- a/Documentation/arm64/cpu-feature-registers.rst +++ b/Documentation/arm64/cpu-feature-registers.rst @@ -288,6 +288,8 @@ infrastructure: +------------------------------+---------+---------+ | Name | bits | visible | +------------------------------+---------+---------+ + | MOPS | [19-16] | y | + +------------------------------+---------+---------+ | RPRES | [7-4] | y | +------------------------------+---------+---------+ | WFXT | [3-0] | y | diff --git a/Documentation/arm64/elf_hwcaps.rst b/Documentation/arm64/elf_hwcaps.rst index 83e57e4d38e2..8f847d0dcf57 100644 --- a/Documentation/arm64/elf_hwcaps.rst +++ b/Documentation/arm64/elf_hwcaps.rst @@ -302,6 +302,9 @@ HWCAP2_SMEB16B16 HWCAP2_SMEF16F16 Functionality implied by ID_AA64SMFR0_EL1.F16F16 == 0b1 +HWCAP2_MOPS + Functionality implied by ID_AA64ISAR2_EL1.MOPS == 0b0001. + 4. Unused AT_HWCAP bits ----------------------- diff --git a/Documentation/arm64/index.rst b/Documentation/arm64/index.rst index ae21f8118830..d08e924204bf 100644 --- a/Documentation/arm64/index.rst +++ b/Documentation/arm64/index.rst @@ -15,11 +15,13 @@ ARM64 Architecture cpu-feature-registers elf_hwcaps hugetlbpage + kdump legacy_instructions memory memory-tagging-extension perf pointer-authentication + ptdump silicon-errata sme sve diff --git a/Documentation/arm64/kdump.rst b/Documentation/arm64/kdump.rst new file mode 100644 index 000000000000..56a89f45df28 --- /dev/null +++ b/Documentation/arm64/kdump.rst @@ -0,0 +1,92 @@ +======================================= +crashkernel memory reservation on arm64 +======================================= + +Author: Baoquan He + +Kdump mechanism is used to capture a corrupted kernel vmcore so that +it can be subsequently analyzed. In order to do this, a preliminarily +reserved memory is needed to pre-load the kdump kernel and boot such +kernel if corruption happens. + +That reserved memory for kdump is adapted to be able to minimally +accommodate the kdump kernel and the user space programs needed for the +vmcore collection. + +Kernel parameter +================ + +Through the kernel parameters below, memory can be reserved accordingly +during the early stage of the first kernel booting so that a continuous +large chunk of memomy can be found. The low memory reservation needs to +be considered if the crashkernel is reserved from the high memory area. + +- crashkernel=size@offset +- crashkernel=size +- crashkernel=size,high crashkernel=size,low + +Low memory and high memory +========================== + +For kdump reservations, low memory is the memory area under a specific +limit, usually decided by the accessible address bits of the DMA-capable +devices needed by the kdump kernel to run. Those devices not related to +vmcore dumping can be ignored. On arm64, the low memory upper bound is +not fixed: it is 1G on the RPi4 platform but 4G on most other systems. +On special kernels built with CONFIG_ZONE_(DMA|DMA32) disabled, the +whole system RAM is low memory. Outside of the low memory described +above, the rest of system RAM is considered high memory. + +Implementation +============== + +1) crashkernel=size@offset +-------------------------- + +The crashkernel memory must be reserved at the user-specified region or +fail if already occupied. + + +2) crashkernel=size +------------------- + +The crashkernel memory region will be reserved in any available position +according to the search order: + +Firstly, the kernel searches the low memory area for an available region +with the specified size. + +If searching for low memory fails, the kernel falls back to searching +the high memory area for an available region of the specified size. If +the reservation in high memory succeeds, a default size reservation in +the low memory will be done. Currently the default size is 128M, +sufficient for the low memory needs of the kdump kernel. + +Note: crashkernel=size is the recommended option for crashkernel kernel +reservations. The user would not need to know the system memory layout +for a specific platform. + +3) crashkernel=size,high crashkernel=size,low +--------------------------------------------- + +crashkernel=size,(high|low) are an important supplement to +crashkernel=size. They allows the user to specify how much memory needs +to be allocated from the high memory and low memory respectively. On +many systems the low memory is precious and crashkernel reservations +from this area should be kept to a minimum. + +To reserve memory for crashkernel=size,high, searching is first +attempted from the high memory region. If the reservation succeeds, the +low memory reservation will be done subsequently. + +If reservation from the high memory failed, the kernel falls back to +searching the low memory with the specified size in crashkernel=,high. +If it succeeds, no further reservation for low memory is needed. + +Notes: + +- If crashkernel=,low is not specified, the default low memory + reservation will be done automatically. + +- if crashkernel=0,low is specified, it means that the low memory + reservation is omitted intentionally. diff --git a/Documentation/arm64/memory.rst b/Documentation/arm64/memory.rst index 2a641ba7be3b..55a55f30eed8 100644 --- a/Documentation/arm64/memory.rst +++ b/Documentation/arm64/memory.rst @@ -33,8 +33,8 @@ AArch64 Linux memory layout with 4KB pages + 4 levels (48-bit):: 0000000000000000 0000ffffffffffff 256TB user ffff000000000000 ffff7fffffffffff 128TB kernel logical memory map [ffff600000000000 ffff7fffffffffff] 32TB [kasan shadow region] - ffff800000000000 ffff800007ffffff 128MB modules - ffff800008000000 fffffbffefffffff 124TB vmalloc + ffff800000000000 ffff80007fffffff 2GB modules + ffff800080000000 fffffbffefffffff 124TB vmalloc fffffbfff0000000 fffffbfffdffffff 224MB fixed mappings (top down) fffffbfffe000000 fffffbfffe7fffff 8MB [guard region] fffffbfffe800000 fffffbffff7fffff 16MB PCI I/O space @@ -50,8 +50,8 @@ AArch64 Linux memory layout with 64KB pages + 3 levels (52-bit with HW support): 0000000000000000 000fffffffffffff 4PB user fff0000000000000 ffff7fffffffffff ~4PB kernel logical memory map [fffd800000000000 ffff7fffffffffff] 512TB [kasan shadow region] - ffff800000000000 ffff800007ffffff 128MB modules - ffff800008000000 fffffbffefffffff 124TB vmalloc + ffff800000000000 ffff80007fffffff 2GB modules + ffff800080000000 fffffbffefffffff 124TB vmalloc fffffbfff0000000 fffffbfffdffffff 224MB fixed mappings (top down) fffffbfffe000000 fffffbfffe7fffff 8MB [guard region] fffffbfffe800000 fffffbffff7fffff 16MB PCI I/O space diff --git a/Documentation/arm64/ptdump.rst b/Documentation/arm64/ptdump.rst new file mode 100644 index 000000000000..5dcfc5d7cddf --- /dev/null +++ b/Documentation/arm64/ptdump.rst @@ -0,0 +1,96 @@ +====================== +Kernel page table dump +====================== + +ptdump is a debugfs interface that provides a detailed dump of the +kernel page tables. It offers a comprehensive overview of the kernel +virtual memory layout as well as the attributes associated with the +various regions in a human-readable format. It is useful to dump the +kernel page tables to verify permissions and memory types. Examining the +page table entries and permissions helps identify potential security +vulnerabilities such as mappings with overly permissive access rights or +improper memory protections. + +Memory hotplug allows dynamic expansion or contraction of available +memory without requiring a system reboot. To maintain the consistency +and integrity of the memory management data structures, arm64 makes use +of the ``mem_hotplug_lock`` semaphore in write mode. Additionally, in +read mode, ``mem_hotplug_lock`` supports an efficient implementation of +``get_online_mems()`` and ``put_online_mems()``. These protect the +offlining of memory being accessed by the ptdump code. + +In order to dump the kernel page tables, enable the following +configurations and mount debugfs:: + + CONFIG_GENERIC_PTDUMP=y + CONFIG_PTDUMP_CORE=y + CONFIG_PTDUMP_DEBUGFS=y + + mount -t debugfs nodev /sys/kernel/debug + cat /sys/kernel/debug/kernel_page_tables + +On analysing the output of ``cat /sys/kernel/debug/kernel_page_tables`` +one can derive information about the virtual address range of the entry, +followed by size of the memory region covered by this entry, the +hierarchical structure of the page tables and finally the attributes +associated with each page. The page attributes provide information about +access permissions, execution capability, type of mapping such as leaf +level PTE or block level PGD, PMD and PUD, and access status of a page +within the kernel memory. Assessing these attributes can assist in +understanding the memory layout, access patterns and security +characteristics of the kernel pages. + +Kernel virtual memory layout example:: + + start address end address size attributes + +---------------------------------------------------------------------------------------+ + | ---[ Linear Mapping start ]---------------------------------------------------------- | + | .................. | + | 0xfff0000000000000-0xfff0000000210000 2112K PTE RW NX SHD AF UXN MEM/NORMAL-TAGGED | + | 0xfff0000000210000-0xfff0000001c00000 26560K PTE ro NX SHD AF UXN MEM/NORMAL | + | .................. | + | ---[ Linear Mapping end ]------------------------------------------------------------ | + +---------------------------------------------------------------------------------------+ + | ---[ Modules start ]----------------------------------------------------------------- | + | .................. | + | 0xffff800000000000-0xffff800008000000 128M PTE | + | .................. | + | ---[ Modules end ]------------------------------------------------------------------- | + +---------------------------------------------------------------------------------------+ + | ---[ vmalloc() area ]---------------------------------------------------------------- | + | .................. | + | 0xffff800008010000-0xffff800008200000 1984K PTE ro x SHD AF UXN MEM/NORMAL | + | 0xffff800008200000-0xffff800008e00000 12M PTE ro x SHD AF CON UXN MEM/NORMAL | + | .................. | + | ---[ vmalloc() end ]----------------------------------------------------------------- | + +---------------------------------------------------------------------------------------+ + | ---[ Fixmap start ]------------------------------------------------------------------ | + | .................. | + | 0xfffffbfffdb80000-0xfffffbfffdb90000 64K PTE ro x SHD AF UXN MEM/NORMAL | + | 0xfffffbfffdb90000-0xfffffbfffdba0000 64K PTE ro NX SHD AF UXN MEM/NORMAL | + | .................. | + | ---[ Fixmap end ]-------------------------------------------------------------------- | + +---------------------------------------------------------------------------------------+ + | ---[ PCI I/O start ]----------------------------------------------------------------- | + | .................. | + | 0xfffffbfffe800000-0xfffffbffff800000 16M PTE | + | .................. | + | ---[ PCI I/O end ]------------------------------------------------------------------- | + +---------------------------------------------------------------------------------------+ + | ---[ vmemmap start ]----------------------------------------------------------------- | + | .................. | + | 0xfffffc0002000000-0xfffffc0002200000 2M PTE RW NX SHD AF UXN MEM/NORMAL | + | 0xfffffc0002200000-0xfffffc0020000000 478M PTE | + | .................. | + | ---[ vmemmap end ]------------------------------------------------------------------- | + +---------------------------------------------------------------------------------------+ + +``cat /sys/kernel/debug/kernel_page_tables`` output:: + + 0xfff0000001c00000-0xfff0000080000000 2020M PTE RW NX SHD AF UXN MEM/NORMAL-TAGGED + 0xfff0000080000000-0xfff0000800000000 30G PMD + 0xfff0000800000000-0xfff0000800700000 7M PTE RW NX SHD AF UXN MEM/NORMAL-TAGGED + 0xfff0000800700000-0xfff0000800710000 64K PTE ro NX SHD AF UXN MEM/NORMAL-TAGGED + 0xfff0000800710000-0xfff0000880000000 2089920K PTE RW NX SHD AF UXN MEM/NORMAL-TAGGED + 0xfff0000880000000-0xfff0040000000000 4062G PMD + 0xfff0040000000000-0xffff800000000000 3964T PGD diff --git a/Documentation/conf.py b/Documentation/conf.py index 37314afd1ac8..d4fdf6a3875a 100644 --- a/Documentation/conf.py +++ b/Documentation/conf.py @@ -74,6 +74,7 @@ if major >= 3: "__percpu", "__rcu", "__user", + "__force", # include/linux/compiler_attributes.h: "__alias", diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst index 9b3f3e5f5a95..f2bcc5a7ea43 100644 --- a/Documentation/core-api/kernel-api.rst +++ b/Documentation/core-api/kernel-api.rst @@ -96,6 +96,12 @@ Command-line Parsing .. kernel-doc:: lib/cmdline.c :export: +Error Pointers +-------------- + +.. kernel-doc:: include/linux/err.h + :internal: + Sorting ------- @@ -412,3 +418,15 @@ Read-Copy Update (RCU) .. kernel-doc:: include/linux/rcu_sync.h .. kernel-doc:: kernel/rcu/sync.c + +.. kernel-doc:: kernel/rcu/tasks.h + +.. kernel-doc:: kernel/rcu/tree_stall.h + +.. kernel-doc:: include/linux/rcupdate_trace.h + +.. kernel-doc:: include/linux/rcupdate_wait.h + +.. kernel-doc:: include/linux/rcuref.h + +.. kernel-doc:: include/linux/rcutree.h diff --git a/Documentation/crypto/async-tx-api.rst b/Documentation/crypto/async-tx-api.rst index bfc773991bdc..27c146b54d71 100644 --- a/Documentation/crypto/async-tx-api.rst +++ b/Documentation/crypto/async-tx-api.rst @@ -66,7 +66,7 @@ features surfaced as a result: :: struct dma_async_tx_descriptor * - async_(, struct async_submit ctl *submit) + async_(, struct async_submit_ctl *submit) 3.2 Supported operations ------------------------ diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 12b575b76b20..dd214af7b7ff 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times:: For other features see the script usage output, seen with the `-h` option. +Timeout for selftests +===================== + +Selftests are designed to be quick and so a default timeout is used of 45 +seconds for each test. Tests can override the default timeout by adding +a settings file in their directory and set a timeout variable there to the +configured a desired upper timeout for the test. Only a few tests override +the timeout with a value higher than 45 seconds, selftests strives to keep +it that way. Timeouts in selftests are not considered fatal because the +system under which a test runs may change and this can also modify the +expected time it takes to run a test. If you have control over the systems +which will run the tests you can configure a test runner on those systems to +use a greater or lower timeout on the command line as with the `-o` or +the `--override-timeout` argument. For example to use 165 seconds instead +one would use: + + $ ./run_kselftest.sh --override-timeout 165 + +You can look at the TAP output to see if you ran into the timeout. Test +runners which know a test must run under a specific time can then optionally +treat these timeouts then as fatal. + Packaging selftests =================== diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentation/dev-tools/kunit/architecture.rst index e95ab05342bb..f335f883f8f6 100644 --- a/Documentation/dev-tools/kunit/architecture.rst +++ b/Documentation/dev-tools/kunit/architecture.rst @@ -119,9 +119,9 @@ All expectations/assertions are formatted as: terminated immediately. - Assertions call the function: - ``void __noreturn kunit_abort(struct kunit *)``. + ``void __noreturn __kunit_abort(struct kunit *)``. - - ``kunit_abort`` calls the function: + - ``__kunit_abort`` calls the function: ``void __noreturn kunit_try_catch_throw(struct kunit_try_catch *try_catch)``. - ``kunit_try_catch_throw`` calls the function: diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-tools/kunit/start.rst index c736613c9b19..a98235326bab 100644 --- a/Documentation/dev-tools/kunit/start.rst +++ b/Documentation/dev-tools/kunit/start.rst @@ -250,15 +250,20 @@ Now we are ready to write the test cases. }; kunit_test_suite(misc_example_test_suite); + MODULE_LICENSE("GPL"); + 2. Add the following lines to ``drivers/misc/Kconfig``: .. code-block:: kconfig config MISC_EXAMPLE_TEST tristate "Test for my example" if !KUNIT_ALL_TESTS - depends on MISC_EXAMPLE && KUNIT=y + depends on MISC_EXAMPLE && KUNIT default KUNIT_ALL_TESTS +Note: If your test does not support being built as a loadable module (which is +discouraged), replace tristate by bool, and depend on KUNIT=y instead of KUNIT. + 3. Add the following lines to ``drivers/misc/Makefile``: .. code-block:: make diff --git a/Documentation/dev-tools/kunit/usage.rst b/Documentation/dev-tools/kunit/usage.rst index 9faf2b4153fc..c27e1646ecd9 100644 --- a/Documentation/dev-tools/kunit/usage.rst +++ b/Documentation/dev-tools/kunit/usage.rst @@ -121,6 +121,12 @@ there's an allocation error. ``return`` so they only work from the test function. In KUnit, we stop the current kthread on failure, so you can call them from anywhere. +.. note:: + Warning: There is an exception to the above rule. You shouldn't use assertions + in the suite's exit() function, or in the free function for a resource. These + run when a test is shutting down, and an assertion here prevents further + cleanup code from running, potentially leading to a memory leak. + Customizing error messages -------------------------- @@ -160,7 +166,12 @@ many similar tests. In order to reduce duplication in these closely related tests, most unit testing frameworks (including KUnit) provide the concept of a *test suite*. A test suite is a collection of test cases for a unit of code with optional setup and teardown functions that run before/after the whole -suite and/or every test case. For example: +suite and/or every test case. + +.. note:: + A test case will only run if it is associated with a test suite. + +For example: .. code-block:: c @@ -190,7 +201,10 @@ after everything else. ``kunit_test_suite(example_test_suite)`` registers the test suite with the KUnit test framework. .. note:: - A test case will only run if it is associated with a test suite. + The ``exit`` and ``suite_exit`` functions will run even if ``init`` or + ``suite_init`` fail. Make sure that they can handle any inconsistent + state which may result from ``init`` or ``suite_init`` encountering errors + or exiting early. ``kunit_test_suite(...)`` is a macro which tells the linker to put the specified test suite in a special linker section so that it can be run by KUnit @@ -601,6 +615,57 @@ For example: KUNIT_ASSERT_STREQ(test, buffer, ""); } +Registering Cleanup Actions +--------------------------- + +If you need to perform some cleanup beyond simple use of ``kunit_kzalloc``, +you can register a custom "deferred action", which is a cleanup function +run when the test exits (whether cleanly, or via a failed assertion). + +Actions are simple functions with no return value, and a single ``void*`` +context argument, and fulfill the same role as "cleanup" functions in Python +and Go tests, "defer" statements in languages which support them, and +(in some cases) destructors in RAII languages. + +These are very useful for unregistering things from global lists, closing +files or other resources, or freeing resources. + +For example: + +.. code-block:: C + + static void cleanup_device(void *ctx) + { + struct device *dev = (struct device *)ctx; + + device_unregister(dev); + } + + void example_device_test(struct kunit *test) + { + struct my_device dev; + + device_register(&dev); + + kunit_add_action(test, &cleanup_device, &dev); + } + +Note that, for functions like device_unregister which only accept a single +pointer-sized argument, it's possible to directly cast that function to +a ``kunit_action_t`` rather than writing a wrapper function, for example: + +.. code-block:: C + + kunit_add_action(test, (kunit_action_t *)&device_unregister, &dev); + +``kunit_add_action`` can fail if, for example, the system is out of memory. +You can use ``kunit_add_action_or_reset`` instead which runs the action +immediately if it cannot be deferred. + +If you need more control over when the cleanup function is called, you +can trigger it early using ``kunit_release_action``, or cancel it entirely +with ``kunit_remove_action``. + Testing Static Functions ------------------------ diff --git a/Documentation/devicetree/bindings/arm/xen.txt b/Documentation/devicetree/bindings/arm/xen.txt index 61d77acbeb5e..f925290d4641 100644 --- a/Documentation/devicetree/bindings/arm/xen.txt +++ b/Documentation/devicetree/bindings/arm/xen.txt @@ -56,7 +56,7 @@ hypervisor { }; The format and meaning of the "xen,uefi-*" parameters are similar to those in -Documentation/arm/uefi.rst, which are provided by the regular UEFI stub. However +Documentation/arch/arm/uefi.rst, which are provided by the regular UEFI stub. However they differ because they are provided by the Xen hypervisor, together with a set of UEFI runtime services implemented via hypercalls, see http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,platform.h.html. diff --git a/Documentation/devicetree/bindings/perf/fsl-imx-ddr.yaml b/Documentation/devicetree/bindings/perf/fsl-imx-ddr.yaml index 80a92385367e..e9fad4b3de68 100644 --- a/Documentation/devicetree/bindings/perf/fsl-imx-ddr.yaml +++ b/Documentation/devicetree/bindings/perf/fsl-imx-ddr.yaml @@ -4,7 +4,7 @@ $id: http://devicetree.org/schemas/perf/fsl-imx-ddr.yaml# $schema: http://devicetree.org/meta-schemas/core.yaml# -title: Freescale(NXP) IMX8 DDR performance monitor +title: Freescale(NXP) IMX8/9 DDR performance monitor maintainers: - Frank Li @@ -19,6 +19,7 @@ properties: - fsl,imx8mm-ddr-pmu - fsl,imx8mn-ddr-pmu - fsl,imx8mp-ddr-pmu + - fsl,imx93-ddr-pmu - items: - enum: - fsl,imx8mm-ddr-pmu diff --git a/Documentation/devicetree/bindings/thermal/armada-thermal.txt b/Documentation/devicetree/bindings/thermal/armada-thermal.txt index b0bee7e42038..ab8b8fccc7af 100644 --- a/Documentation/devicetree/bindings/thermal/armada-thermal.txt +++ b/Documentation/devicetree/bindings/thermal/armada-thermal.txt @@ -8,6 +8,7 @@ Required properties: * marvell,armada380-thermal * marvell,armadaxp-thermal * marvell,armada-ap806-thermal + * marvell,armada-ap807-thermal * marvell,armada-cp110-thermal Note: these bindings are deprecated for AP806/CP110 and should instead diff --git a/Documentation/devicetree/bindings/thermal/brcm,bcm2835-thermal.txt b/Documentation/devicetree/bindings/thermal/brcm,bcm2835-thermal.txt deleted file mode 100644 index a3e9ec5dc7ac..000000000000 --- a/Documentation/devicetree/bindings/thermal/brcm,bcm2835-thermal.txt +++ /dev/null @@ -1,41 +0,0 @@ -Binding for Thermal Sensor driver for BCM2835 SoCs. - -Required parameters: -------------------- - -compatible: should be one of: "brcm,bcm2835-thermal", - "brcm,bcm2836-thermal" or "brcm,bcm2837-thermal" -reg: Address range of the thermal registers. -clocks: Phandle of the clock used by the thermal sensor. -#thermal-sensor-cells: should be 0 (see Documentation/devicetree/bindings/thermal/thermal-sensor.yaml) - -Example: - -thermal-zones { - cpu_thermal: cpu-thermal { - polling-delay-passive = <0>; - polling-delay = <1000>; - - thermal-sensors = <&thermal>; - - trips { - cpu-crit { - temperature = <80000>; - hysteresis = <0>; - type = "critical"; - }; - }; - - coefficients = <(-538) 407000>; - - cooling-maps { - }; - }; -}; - -thermal: thermal@7e212000 { - compatible = "brcm,bcm2835-thermal"; - reg = <0x7e212000 0x8>; - clocks = <&clocks BCM2835_CLOCK_TSENS>; - #thermal-sensor-cells = <0>; -}; diff --git a/Documentation/devicetree/bindings/thermal/brcm,bcm2835-thermal.yaml b/Documentation/devicetree/bindings/thermal/brcm,bcm2835-thermal.yaml new file mode 100644 index 000000000000..2b6026d9fbcf --- /dev/null +++ b/Documentation/devicetree/bindings/thermal/brcm,bcm2835-thermal.yaml @@ -0,0 +1,48 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/thermal/brcm,bcm2835-thermal.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Broadcom BCM2835 thermal sensor + +maintainers: + - Stefan Wahren + +allOf: + - $ref: thermal-sensor.yaml# + +properties: + compatible: + enum: + - brcm,bcm2835-thermal + - brcm,bcm2836-thermal + - brcm,bcm2837-thermal + + reg: + maxItems: 1 + + clocks: + maxItems: 1 + + "#thermal-sensor-cells": + const: 0 + +unevaluatedProperties: false + +required: + - compatible + - reg + - clocks + - '#thermal-sensor-cells' + +examples: + - | + #include + + thermal@7e212000 { + compatible = "brcm,bcm2835-thermal"; + reg = <0x7e212000 0x8>; + clocks = <&clocks BCM2835_CLOCK_TSENS>; + #thermal-sensor-cells = <0>; + }; diff --git a/Documentation/devicetree/bindings/thermal/qcom-tsens.yaml b/Documentation/devicetree/bindings/thermal/qcom-tsens.yaml index d1ec963a6834..27e9e16e6455 100644 --- a/Documentation/devicetree/bindings/thermal/qcom-tsens.yaml +++ b/Documentation/devicetree/bindings/thermal/qcom-tsens.yaml @@ -29,6 +29,8 @@ properties: items: - enum: - qcom,mdm9607-tsens + - qcom,msm8226-tsens + - qcom,msm8909-tsens - qcom,msm8916-tsens - qcom,msm8939-tsens - qcom,msm8974-tsens @@ -48,6 +50,7 @@ properties: - qcom,msm8953-tsens - qcom,msm8996-tsens - qcom,msm8998-tsens + - qcom,qcm2290-tsens - qcom,sc7180-tsens - qcom,sc7280-tsens - qcom,sc8180x-tsens @@ -56,6 +59,7 @@ properties: - qcom,sdm845-tsens - qcom,sm6115-tsens - qcom,sm6350-tsens + - qcom,sm6375-tsens - qcom,sm8150-tsens - qcom,sm8250-tsens - qcom,sm8350-tsens @@ -67,6 +71,12 @@ properties: enum: - qcom,ipq8074-tsens + - description: v2 of TSENS with combined interrupt + items: + - enum: + - qcom,ipq9574-tsens + - const: qcom,ipq8074-tsens + reg: items: - description: TM registers @@ -223,12 +233,7 @@ allOf: contains: enum: - qcom,ipq8064-tsens - - qcom,mdm9607-tsens - - qcom,msm8916-tsens - qcom,msm8960-tsens - - qcom,msm8974-tsens - - qcom,msm8976-tsens - - qcom,qcs404-tsens - qcom,tsens-v0_1 - qcom,tsens-v1 then: @@ -244,22 +249,7 @@ allOf: properties: compatible: contains: - enum: - - qcom,msm8953-tsens - - qcom,msm8996-tsens - - qcom,msm8998-tsens - - qcom,sc7180-tsens - - qcom,sc7280-tsens - - qcom,sc8180x-tsens - - qcom,sc8280xp-tsens - - qcom,sdm630-tsens - - qcom,sdm845-tsens - - qcom,sm6350-tsens - - qcom,sm8150-tsens - - qcom,sm8250-tsens - - qcom,sm8350-tsens - - qcom,sm8450-tsens - - qcom,tsens-v2 + const: qcom,tsens-v2 then: properties: interrupts: diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst index 23edb427e76f..cd8ad7904491 100644 --- a/Documentation/doc-guide/sphinx.rst +++ b/Documentation/doc-guide/sphinx.rst @@ -313,9 +313,18 @@ the documentation build system will automatically turn a reference to function name exists. If you see ``c:func:`` use in a kernel document, please feel free to remove it. +Tables +------ + +ReStructuredText provides several options for table syntax. Kernel style for +tables is to prefer *simple table* syntax or *grid table* syntax. See the +`reStructuredText user reference for table syntax`_ for more details. + +.. _reStructuredText user reference for table syntax: + https://docutils.sourceforge.io/docs/user/rst/quickref.html#tables list tables ------------ +~~~~~~~~~~~ The list-table formats can be useful for tables that are not easily laid out in the usual Sphinx ASCII-art formats. These formats are nearly diff --git a/Documentation/maintainer/configure-git.rst b/Documentation/maintainer/configure-git.rst index 80ae5030a590..ec0ddfb9cdd3 100644 --- a/Documentation/maintainer/configure-git.rst +++ b/Documentation/maintainer/configure-git.rst @@ -56,7 +56,7 @@ by adding the following hook into your git: $ cat >.git/hooks/applypatch-msg <<'EOF' #!/bin/sh . git-sh-setup - perl -pi -e 's|^Message-Id:\s*]+)>?$|Link: https://lore.kernel.org/r/$1|g;' "$1" + perl -pi -e 's|^Message-I[dD]:\s*]+)>?$|Link: https://lore.kernel.org/r/$1|g;' "$1" test -x "$GIT_DIR/hooks/commit-msg" && exec "$GIT_DIR/hooks/commit-msg" ${1+"$@"} : diff --git a/Documentation/mm/page_tables.rst b/Documentation/mm/page_tables.rst index 96939571d7bc..7840c1891751 100644 --- a/Documentation/mm/page_tables.rst +++ b/Documentation/mm/page_tables.rst @@ -3,3 +3,152 @@ =========== Page Tables =========== + +Paged virtual memory was invented along with virtual memory as a concept in +1962 on the Ferranti Atlas Computer which was the first computer with paged +virtual memory. The feature migrated to newer computers and became a de facto +feature of all Unix-like systems as time went by. In 1985 the feature was +included in the Intel 80386, which was the CPU Linux 1.0 was developed on. + +Page tables map virtual addresses as seen by the CPU into physical addresses +as seen on the external memory bus. + +Linux defines page tables as a hierarchy which is currently five levels in +height. The architecture code for each supported architecture will then +map this to the restrictions of the hardware. + +The physical address corresponding to the virtual address is often referenced +by the underlying physical page frame. The **page frame number** or **pfn** +is the physical address of the page (as seen on the external memory bus) +divided by `PAGE_SIZE`. + +Physical memory address 0 will be *pfn 0* and the highest pfn will be +the last page of physical memory the external address bus of the CPU can +address. + +With a page granularity of 4KB and a address range of 32 bits, pfn 0 is at +address 0x00000000, pfn 1 is at address 0x00001000, pfn 2 is at 0x00002000 +and so on until we reach pfn 0xfffff at 0xfffff000. With 16KB pages pfs are +at 0x00004000, 0x00008000 ... 0xffffc000 and pfn goes from 0 to 0x3fffff. + +As you can see, with 4KB pages the page base address uses bits 12-31 of the +address, and this is why `PAGE_SHIFT` in this case is defined as 12 and +`PAGE_SIZE` is usually defined in terms of the page shift as `(1 << PAGE_SHIFT)` + +Over time a deeper hierarchy has been developed in response to increasing memory +sizes. When Linux was created, 4KB pages and a single page table called +`swapper_pg_dir` with 1024 entries was used, covering 4MB which coincided with +the fact that Torvald's first computer had 4MB of physical memory. Entries in +this single table were referred to as *PTE*:s - page table entries. + +The software page table hierarchy reflects the fact that page table hardware has +become hierarchical and that in turn is done to save page table memory and +speed up mapping. + +One could of course imagine a single, linear page table with enormous amounts +of entries, breaking down the whole memory into single pages. Such a page table +would be very sparse, because large portions of the virtual memory usually +remains unused. By using hierarchical page tables large holes in the virtual +address space does not waste valuable page table memory, because it will suffice +to mark large areas as unmapped at a higher level in the page table hierarchy. + +Additionally, on modern CPUs, a higher level page table entry can point directly +to a physical memory range, which allows mapping a contiguous range of several +megabytes or even gigabytes in a single high-level page table entry, taking +shortcuts in mapping virtual memory to physical memory: there is no need to +traverse deeper in the hierarchy when you find a large mapped range like this. + +The page table hierarchy has now developed into this:: + + +-----+ + | PGD | + +-----+ + | + | +-----+ + +-->| P4D | + +-----+ + | + | +-----+ + +-->| PUD | + +-----+ + | + | +-----+ + +-->| PMD | + +-----+ + | + | +-----+ + +-->| PTE | + +-----+ + + +Symbols on the different levels of the page table hierarchy have the following +meaning beginning from the bottom: + +- **pte**, `pte_t`, `pteval_t` = **Page Table Entry** - mentioned earlier. + The *pte* is an array of `PTRS_PER_PTE` elements of the `pteval_t` type, each + mapping a single page of virtual memory to a single page of physical memory. + The architecture defines the size and contents of `pteval_t`. + + A typical example is that the `pteval_t` is a 32- or 64-bit value with the + upper bits being a **pfn** (page frame number), and the lower bits being some + architecture-specific bits such as memory protection. + + The **entry** part of the name is a bit confusing because while in Linux 1.0 + this did refer to a single page table entry in the single top level page + table, it was retrofitted to be an array of mapping elements when two-level + page tables were first introduced, so the *pte* is the lowermost page + *table*, not a page table *entry*. + +- **pmd**, `pmd_t`, `pmdval_t` = **Page Middle Directory**, the hierarchy right + above the *pte*, with `PTRS_PER_PMD` references to the *pte*:s. + +- **pud**, `pud_t`, `pudval_t` = **Page Upper Directory** was introduced after + the other levels to handle 4-level page tables. It is potentially unused, + or *folded* as we will discuss later. + +- **p4d**, `p4d_t`, `p4dval_t` = **Page Level 4 Directory** was introduced to + handle 5-level page tables after the *pud* was introduced. Now it was clear + that we needed to replace *pgd*, *pmd*, *pud* etc with a figure indicating the + directory level and that we cannot go on with ad hoc names any more. This + is only used on systems which actually have 5 levels of page tables, otherwise + it is folded. + +- **pgd**, `pgd_t`, `pgdval_t` = **Page Global Directory** - the Linux kernel + main page table handling the PGD for the kernel memory is still found in + `swapper_pg_dir`, but each userspace process in the system also has its own + memory context and thus its own *pgd*, found in `struct mm_struct` which + in turn is referenced to in each `struct task_struct`. So tasks have memory + context in the form of a `struct mm_struct` and this in turn has a + `struct pgt_t *pgd` pointer to the corresponding page global directory. + +To repeat: each level in the page table hierarchy is a *array of pointers*, so +the **pgd** contains `PTRS_PER_PGD` pointers to the next level below, **p4d** +contains `PTRS_PER_P4D` pointers to **pud** items and so on. The number of +pointers on each level is architecture-defined.:: + + PMD + --> +-----+ PTE + | ptr |-------> +-----+ + | ptr |- | ptr |-------> PAGE + | ptr | \ | ptr | + | ptr | \ ... + | ... | \ + | ptr | \ PTE + +-----+ +----> +-----+ + | ptr |-------> PAGE + | ptr | + ... + + +Page Table Folding +================== + +If the architecture does not use all the page table levels, they can be *folded* +which means skipped, and all operations performed on page tables will be +compile-time augmented to just skip a level when accessing the next lower +level. + +Page table handling code that wishes to be architecture-neutral, such as the +virtual memory manager, will need to be written so that it traverses all of the +currently five levels. This style should also be preferred for +architecture-specific code, so as to be robust to future changes. diff --git a/Documentation/process/2.Process.rst b/Documentation/process/2.Process.rst index 6a919cffcbfd..613a01da4717 100644 --- a/Documentation/process/2.Process.rst +++ b/Documentation/process/2.Process.rst @@ -434,9 +434,10 @@ There are a few hints which can help with linux-kernel survival: questions. Some developers can get impatient with people who clearly have not done their homework. -- Avoid top-posting (the practice of putting your answer above the quoted - text you are responding to). It makes your response harder to read and - makes a poor impression. +- Use interleaved ("inline") replies, which makes your response easier to + read. (i.e. avoid top-posting -- the practice of putting your answer above + the quoted text you are responding to.) For more details, see + :ref:`Documentation/process/submitting-patches.rst `. - Ask on the correct mailing list. Linux-kernel may be the general meeting point, but it is not the best place to find developers from all diff --git a/Documentation/process/handling-regressions.rst b/Documentation/process/handling-regressions.rst index abb741b1aeee..5d3c3de3f4ec 100644 --- a/Documentation/process/handling-regressions.rst +++ b/Documentation/process/handling-regressions.rst @@ -129,88 +129,132 @@ tools and scripts used by other kernel developers or Linux distributions; one of these tools is regzbot, which heavily relies on the "Link:" tags to associate reports for regression with changes resolving them. -Prioritize work on fixing regressions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Expectations and best practices for fixing regressions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You should fix any reported regression as quickly as possible, to provide -affected users with a solution in a timely manner and prevent more users from -running into the issue; nevertheless developers need to take enough time and -care to ensure regression fixes do not cause additional damage. +As a Linux kernel developer, you are expected to give your best to prevent +situations where a regression caused by a recent change of yours leaves users +only these options: -In the end though, developers should give their best to prevent users from -running into situations where a regression leaves them only three options: "run -a kernel with a regression that seriously impacts usage", "continue running an -outdated and thus potentially insecure kernel version for more than two weeks -after a regression's culprit was identified", and "downgrade to a still -supported kernel series that lack required features". + * Run a kernel with a regression that impacts usage. -How to realize this depends a lot on the situation. Here are a few rules of -thumb for you, in order or importance: + * Switch to an older or newer kernel series. - * Prioritize work on handling regression reports and fixing regression over all - other Linux kernel work, unless the latter concerns acute security issues or - bugs causing data loss or damage. + * Continue running an outdated and thus potentially insecure kernel for more + than three weeks after the regression's culprit was identified. Ideally it + should be less than two. And it ought to be just a few days, if the issue is + severe or affects many users -- either in general or in prevalent + environments. - * Always consider reverting the culprit commits and reapplying them later - together with necessary fixes, as this might be the least dangerous and - quickest way to fix a regression. +How to realize that in practice depends on various factors. Use the following +rules of thumb as a guide. - * Developers should handle regressions in all supported kernel series, but are - free to delegate the work to the stable team, if the issue probably at no - point in time occurred with mainline. +In general: - * Try to resolve any regressions introduced in the current development before - its end. If you fear a fix might be too risky to apply only days before a new - mainline release, let Linus decide: submit the fix separately to him as soon - as possible with the explanation of the situation. He then can make a call - and postpone the release if necessary, for example if multiple such changes - show up in his inbox. + * Prioritize work on regressions over all other Linux kernel work, unless the + latter concerns a severe issue (e.g. acute security vulnerability, data loss, + bricked hardware, ...). - * Address regressions in stable, longterm, or proper mainline releases with - more urgency than regressions in mainline pre-releases. That changes after - the release of the fifth pre-release, aka "-rc5": mainline then becomes as - important, to ensure all the improvements and fixes are ideally tested - together for at least one week before Linus releases a new mainline version. + * Expedite fixing mainline regressions that recently made it into a proper + mainline, stable, or longterm release (either directly or via backport). - * Fix regressions within two or three days, if they are critical for some - reason -- for example, if the issue is likely to affect many users of the - kernel series in question on all or certain architectures. Note, this - includes mainline, as issues like compile errors otherwise might prevent many - testers or continuous integration systems from testing the series. + * Do not consider regressions from the current cycle as something that can wait + till the end of the cycle, as the issue might discourage or prevent users and + CI systems from testing mainline now or generally. - * Aim to fix regressions within one week after the culprit was identified, if - the issue was introduced in either: + * Work with the required care to avoid additional or bigger damage, even if + resolving an issue then might take longer than outlined below. - * a recent stable/longterm release +On timing once the culprit of a regression is known: - * the development cycle of the latest proper mainline release + * Aim to mainline a fix within two or three days, if the issue is severe or + bothering many users -- either in general or in prevalent conditions like a + particular hardware environment, distribution, or stable/longterm series. - In the latter case (say Linux v5.14), try to address regressions even - quicker, if the stable series for the predecessor (v5.13) will be abandoned - soon or already was stamped "End-of-Life" (EOL) -- this usually happens about - three to four weeks after a new mainline release. + * Aim to mainline a fix by Sunday after the next, if the culprit made it + into a recent mainline, stable, or longterm release (either directly or via + backport); if the culprit became known early during a week and is simple to + resolve, try to mainline the fix within the same week. - * Try to fix all other regressions within two weeks after the culprit was - found. Two or three additional weeks are acceptable for performance - regressions and other issues which are annoying, but don't prevent anyone - from running Linux (unless it's an issue in the current development cycle, - as those should ideally be addressed before the release). A few weeks in - total are acceptable if a regression can only be fixed with a risky change - and at the same time is affecting only a few users; as much time is - also okay if the regression is already present in the second newest longterm - kernel series. + * For other regressions, aim to mainline fixes before the hindmost Sunday + within the next three weeks. One or two Sundays later are acceptable, if the + regression is something people can live with easily for a while -- like a + mild performance regression. -Note: The aforementioned time frames for resolving regressions are meant to -include getting the fix tested, reviewed, and merged into mainline, ideally with -the fix being in linux-next at least briefly. This leads to delays you need to -account for. + * It's strongly discouraged to delay mainlining regression fixes till the next + merge window, except when the fix is extraordinarily risky or when the + culprit was mainlined more than a year ago. -Subsystem maintainers are expected to assist in reaching those periods by doing -timely reviews and quick handling of accepted patches. They thus might have to -send git-pull requests earlier or more often than usual; depending on the fix, -it might even be acceptable to skip testing in linux-next. Especially fixes for -regressions in stable and longterm kernels need to be handled quickly, as fixes -need to be merged in mainline before they can be backported to older series. +On procedure: + + * Always consider reverting the culprit, as it's often the quickest and least + dangerous way to fix a regression. Don't worry about mainlining a fixed + variant later: that should be straight-forward, as most of the code went + through review once already. + + * Try to resolve any regressions introduced in mainline during the past + twelve months before the current development cycle ends: Linus wants such + regressions to be handled like those from the current cycle, unless fixing + bears unusual risks. + + * Consider CCing Linus on discussions or patch review, if a regression seems + tangly. Do the same in precarious or urgent cases -- especially if the + subsystem maintainer might be unavailable. Also CC the stable team, when you + know such a regression made it into a mainline, stable, or longterm release. + + * For urgent regressions, consider asking Linus to pick up the fix straight + from the mailing list: he is totally fine with that for uncontroversial + fixes. Ideally though such requests should happen in accordance with the + subsystem maintainers or come directly from them. + + * In case you are unsure if a fix is worth the risk applying just days before + a new mainline release, send Linus a mail with the usual lists and people in + CC; in it, summarize the situation while asking him to consider picking up + the fix straight from the list. He then himself can make the call and when + needed even postpone the release. Such requests again should ideally happen + in accordance with the subsystem maintainers or come directly from them. + +Regarding stable and longterm kernels: + + * You are free to leave regressions to the stable team, if they at no point in + time occurred with mainline or were fixed there already. + + * If a regression made it into a proper mainline release during the past + twelve months, ensure to tag the fix with "Cc: stable@vger.kernel.org", as a + "Fixes:" tag alone does not guarantee a backport. Please add the same tag, + in case you know the culprit was backported to stable or longterm kernels. + + * When receiving reports about regressions in recent stable or longterm kernel + series, please evaluate at least briefly if the issue might happen in current + mainline as well -- and if that seems likely, take hold of the report. If in + doubt, ask the reporter to check mainline. + + * Whenever you want to swiftly resolve a regression that recently also made it + into a proper mainline, stable, or longterm release, fix it quickly in + mainline; when appropriate thus involve Linus to fast-track the fix (see + above). That's because the stable team normally does neither revert nor fix + any changes that cause the same problems in mainline. + + * In case of urgent regression fixes you might want to ensure prompt + backporting by dropping the stable team a note once the fix was mainlined; + this is especially advisable during merge windows and shortly thereafter, as + the fix otherwise might land at the end of a huge patch queue. + +On patch flow: + + * Developers, when trying to reach the time periods mentioned above, remember + to account for the time it takes to get fixes tested, reviewed, and merged by + Linus, ideally with them being in linux-next at least briefly. Hence, if a + fix is urgent, make it obvious to ensure others handle it appropriately. + + * Reviewers, you are kindly asked to assist developers in reaching the time + periods mentioned above by reviewing regression fixes in a timely manner. + + * Subsystem maintainers, you likewise are encouraged to expedite the handling + of regression fixes. Thus evaluate if skipping linux-next is an option for + the particular fix. Also consider sending git pull requests more often than + usual when needed. And try to avoid holding onto regression fixes over + weekends -- especially when the fix is marked for backporting. More aspects regarding regressions developers should be aware of diff --git a/Documentation/process/submitting-patches.rst b/Documentation/process/submitting-patches.rst index 486875fd73c0..efac910e2659 100644 --- a/Documentation/process/submitting-patches.rst +++ b/Documentation/process/submitting-patches.rst @@ -331,6 +331,31 @@ explaining difference against previous submission (see See Documentation/process/email-clients.rst for recommendations on email clients and mailing list etiquette. +.. _interleaved_replies: + +Use trimmed interleaved replies in email discussions +---------------------------------------------------- +Top-posting is strongly discouraged in Linux kernel development +discussions. Interleaved (or "inline") replies make conversations much +easier to follow. For more details see: +https://en.wikipedia.org/wiki/Posting_style#Interleaved_style + +As is frequently quoted on the mailing list:: + + A: http://en.wikipedia.org/wiki/Top_post + Q: Were do I find info about this thing called top-posting? + A: Because it messes up the order in which people normally read text. + Q: Why is top-posting such a bad thing? + A: Top-posting. + Q: What is the most annoying thing in e-mail? + +Similarly, please trim all unneeded quotations that aren't relevant +to your reply. This makes responses easier to find, and saves time and +space. For more details see: http://daringfireball.net/2007/07/on_top :: + + A: No. + Q: Should I include quotations after my reply? + .. _resend_reminders: Don't get discouraged - or impatient diff --git a/Documentation/subsystem-apis.rst b/Documentation/subsystem-apis.rst index b51f38527e14..02d6dc3a49c8 100644 --- a/Documentation/subsystem-apis.rst +++ b/Documentation/subsystem-apis.rst @@ -10,6 +10,30 @@ is taken directly from the kernel source, with supplemental material added as needed (or at least as we managed to add it — probably *not* all that is needed). +Human interfaces +---------------- + +.. toctree:: + :maxdepth: 1 + + input/index + hid/index + sound/index + gpu/index + fb/index + +Storage interfaces +------------------ + +.. toctree:: + :maxdepth: 1 + + filesystems/index + block/index + cdrom/index + scsi/index + target/index + **Fixme**: much more organizational work is needed here. .. toctree:: @@ -19,12 +43,8 @@ needed). core-api/index locking/index accounting/index - block/index - cdrom/index cpu-freq/index - fb/index fpga/index - hid/index i2c/index iio/index isdn/index @@ -34,25 +54,19 @@ needed). networking/index pcmcia/index power/index - target/index timers/index spi/index w1/index watchdog/index virt/index - input/index hwmon/index - gpu/index accel/index security/index - sound/index crypto/index - filesystems/index mm/index bpf/index usb/index PCI/index - scsi/index misc-devices/index scheduler/index mhi/index diff --git a/Documentation/translations/zh_CN/arm/Booting b/Documentation/translations/zh_CN/arch/arm/Booting similarity index 98% rename from Documentation/translations/zh_CN/arm/Booting rename to Documentation/translations/zh_CN/arch/arm/Booting index 5ecea0767893..f18585156b67 100644 --- a/Documentation/translations/zh_CN/arm/Booting +++ b/Documentation/translations/zh_CN/arch/arm/Booting @@ -1,4 +1,4 @@ -Chinese translated version of Documentation/arm/booting.rst +Chinese translated version of Documentation/arch/arm/booting.rst If you have any comment or update to the content, please contact the original document maintainer directly. However, if you have a problem @@ -9,7 +9,7 @@ or if there is a problem with the translation. Maintainer: Russell King Chinese maintainer: Fu Wei --------------------------------------------------------------------- -Documentation/arm/booting.rst 的中文翻译 +Documentation/arch/arm/booting.rst 的中文翻译 如果想评论或更新本文的内容，请直接联系原文档的维护者。如果你使用英文 交流有困难的话，也可以向中文版维护者求助。如果本翻译更新不及时或者翻 diff --git a/Documentation/translations/zh_CN/arm/kernel_user_helpers.txt b/Documentation/translations/zh_CN/arch/arm/kernel_user_helpers.txt similarity index 98% rename from Documentation/translations/zh_CN/arm/kernel_user_helpers.txt rename to Documentation/translations/zh_CN/arch/arm/kernel_user_helpers.txt index 99af4363984d..018eb7d54233 100644 --- a/Documentation/translations/zh_CN/arm/kernel_user_helpers.txt +++ b/Documentation/translations/zh_CN/arch/arm/kernel_user_helpers.txt @@ -1,4 +1,4 @@ -Chinese translated version of Documentation/arm/kernel_user_helpers.rst +Chinese translated version of Documentation/arch/arm/kernel_user_helpers.rst If you have any comment or update to the content, please contact the original document maintainer directly. However, if you have a problem @@ -10,7 +10,7 @@ Maintainer: Nicolas Pitre Dave Martin Chinese maintainer: Fu Wei --------------------------------------------------------------------- -Documentation/arm/kernel_user_helpers.rst 的中文翻译 +Documentation/arch/arm/kernel_user_helpers.rst 的中文翻译 如果想评论或更新本文的内容，请直接联系原文档的维护者。如果你使用英文 交流有困难的话，也可以向中文版维护者求助。如果本翻译更新不及时或者翻 diff --git a/Documentation/virt/guest-halt-polling.rst b/Documentation/virt/guest-halt-polling.rst index b4e747942417..922291ddc40c 100644 --- a/Documentation/virt/guest-halt-polling.rst +++ b/Documentation/virt/guest-halt-polling.rst @@ -72,7 +72,7 @@ high once achieves global guest_halt_poll_ns value). Default: Y -The module parameters can be set from the debugfs files in:: +The module parameters can be set from the sysfs files in:: /sys/module/haltpoll/parameters/ diff --git a/Documentation/virt/kvm/halt-polling.rst b/Documentation/virt/kvm/halt-polling.rst index 3fae39b1a5ba..4f1a1b23d99c 100644 --- a/Documentation/virt/kvm/halt-polling.rst +++ b/Documentation/virt/kvm/halt-polling.rst @@ -112,11 +112,11 @@ powerpc kvm-hv case. | | function. | | +-----------------------+---------------------------+-------------------------+ -These module parameters can be set from the debugfs files in: +These module parameters can be set from the sysfs files in: /sys/module/kvm/parameters/ -Note: that these module parameters are system wide values and are not able to +Note: these module parameters are system-wide values and are not able to be tuned on a per vm basis. Any changes to these parameters will be picked up by new and existing vCPUs the @@ -142,12 +142,12 @@ Further Notes global max polling interval (halt_poll_ns) then the host will always poll for the entire block time and thus cpu utilisation will go to 100%. -- Halt polling essentially presents a trade off between power usage and latency and +- Halt polling essentially presents a trade-off between power usage and latency and the module parameters should be used to tune the affinity for this. Idle cpu time is essentially converted to host kernel time with the aim of decreasing latency when entering the guest. - Halt polling will only be conducted by the host when no other tasks are runnable on that cpu, otherwise the polling will cease immediately and schedule will be invoked to - allow that other task to run. Thus this doesn't allow a guest to denial of service the - cpu. + allow that other task to run. Thus this doesn't allow a guest to cause denial of service + of the cpu. diff --git a/Documentation/virt/kvm/locking.rst b/Documentation/virt/kvm/locking.rst index 8c77554e4896..3a034db5e55f 100644 --- a/Documentation/virt/kvm/locking.rst +++ b/Documentation/virt/kvm/locking.rst @@ -67,7 +67,7 @@ following two cases: 2. Write-Protection: The SPTE is present and the fault is caused by write-protect. That means we just need to change the W bit of the spte. -What we use to avoid all the race is the Host-writable bit and MMU-writable bit +What we use to avoid all the races is the Host-writable bit and MMU-writable bit on the spte: - Host-writable means the gfn is writable in the host kernel page tables and in @@ -130,7 +130,7 @@ to gfn. For indirect sp, we disabled fast page fault for simplicity. A solution for indirect sp could be to pin the gfn, for example via kvm_vcpu_gfn_to_pfn_atomic, before the cmpxchg. After the pinning: -- We have held the refcount of pfn that means the pfn can not be freed and +- We have held the refcount of pfn; that means the pfn can not be freed and be reused for another gfn. - The pfn is writable and therefore it cannot be shared between different gfns by KSM. @@ -186,22 +186,22 @@ writable between reading spte and updating spte. Like below case: The Dirty bit is lost in this case. In order to avoid this kind of issue, we always treat the spte as "volatile" -if it can be updated out of mmu-lock, see spte_has_volatile_bits(), it means, +if it can be updated out of mmu-lock [see spte_has_volatile_bits()]; it means the spte is always atomically updated in this case. 3) flush tlbs due to spte updated -If the spte is updated from writable to readonly, we should flush all TLBs, +If the spte is updated from writable to read-only, we should flush all TLBs, otherwise rmap_write_protect will find a read-only spte, even though the writable spte might be cached on a CPU's TLB. As mentioned before, the spte can be updated to writable out of mmu-lock on -fast page fault path, in order to easily audit the path, we see if TLBs need -be flushed caused by this reason in mmu_spte_update() since this is a common +fast page fault path. In order to easily audit the path, we see if TLBs needing +to be flushed caused this reason in mmu_spte_update() since this is a common function to update spte (present -> present). Since the spte is "volatile" if it can be updated out of mmu-lock, we always -atomically update the spte, the race caused by fast page fault can be avoided, +atomically update the spte and the race caused by fast page fault can be avoided. See the comments in spte_has_volatile_bits() and mmu_spte_update(). Lockless Access Tracking: @@ -283,9 +283,9 @@ time it will be set using the Dirty tracking mechanism described above. :Arch: x86 :Protects: wakeup_vcpus_on_cpu :Comment: This is a per-CPU lock and it is used for VT-d posted-interrupts. - When VT-d posted-interrupts is supported and the VM has assigned + When VT-d posted-interrupts are supported and the VM has assigned devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu - protected by blocked_vcpu_on_cpu_lock, when VT-d hardware issues + protected by blocked_vcpu_on_cpu_lock. When VT-d hardware issues wakeup notification event since external interrupts from the assigned devices happens, we will find the vCPU on the list to wakeup. diff --git a/Documentation/virt/kvm/ppc-pv.rst b/Documentation/virt/kvm/ppc-pv.rst index 5fdb907670be..740d03d25300 100644 --- a/Documentation/virt/kvm/ppc-pv.rst +++ b/Documentation/virt/kvm/ppc-pv.rst @@ -89,7 +89,7 @@ also define a new hypercall feature to indicate that the host can give you more registers. Only if the host supports the additional features, make use of them. The magic page layout is described by struct kvm_vcpu_arch_shared -in arch/powerpc/include/asm/kvm_para.h. +in arch/powerpc/include/uapi/asm/kvm_para.h. Magic page features =================== @@ -112,7 +112,7 @@ Magic page flags ================ In addition to features that indicate whether a host is capable of a particular -feature we also have a channel for a guest to tell the guest whether it's capable +feature we also have a channel for a guest to tell the host whether it's capable of something. This is what we call "flags". Flags are passed to the host in the low 12 bits of the Effective Address. @@ -139,7 +139,7 @@ Patched instructions ==================== The "ld" and "std" instructions are transformed to "lwz" and "stw" instructions -respectively on 32 bit systems with an added offset of 4 to accommodate for big +respectively on 32-bit systems with an added offset of 4 to accommodate for big endianness. The following is a list of mapping the Linux kernel performs when running as @@ -210,7 +210,7 @@ available on all targets. 2) PAPR hypercalls PAPR hypercalls are needed to run server PowerPC PAPR guests (-M pseries in QEMU). -These are the same hypercalls that pHyp, the POWER hypervisor implements. Some of +These are the same hypercalls that pHyp, the POWER hypervisor, implements. Some of them are handled in the kernel, some are handled in user space. This is only available on book3s_64. diff --git a/Documentation/virt/kvm/vcpu-requests.rst b/Documentation/virt/kvm/vcpu-requests.rst index 87f04c1fa53d..06718b9bc959 100644 --- a/Documentation/virt/kvm/vcpu-requests.rst +++ b/Documentation/virt/kvm/vcpu-requests.rst @@ -101,7 +101,7 @@ also be used, e.g. :: However, VCPU request users should refrain from doing so, as it would break the abstraction. The first 8 bits are reserved for architecture -independent requests, all additional bits are available for architecture +independent requests; all additional bits are available for architecture dependent requests. Architecture Independent Requests @@ -151,8 +151,8 @@ KVM_REQUEST_NO_WAKEUP This flag is applied to requests that only need immediate attention from VCPUs running in guest mode. That is, sleeping VCPUs do not need - to be awaken for these requests. Sleeping VCPUs will handle the - requests when they are awaken later for some other reason. + to be awakened for these requests. Sleeping VCPUs will handle the + requests when they are awakened later for some other reason. KVM_REQUEST_WAIT diff --git a/MAINTAINERS b/MAINTAINERS index 9878aa0ee5c3..43fc15a281d1 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2710,7 +2710,7 @@ Q: https://patchwork.kernel.org/project/linux-samsung-soc/list/ B: mailto:linux-samsung-soc@vger.kernel.org C: irc://irc.libera.chat/linux-exynos T: git git://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux.git -F: Documentation/arm/samsung/ +F: Documentation/arch/arm/samsung/ F: Documentation/devicetree/bindings/arm/samsung/ F: Documentation/devicetree/bindings/hwinfo/samsung,* F: Documentation/devicetree/bindings/power/pd-samsung.yaml @@ -6239,6 +6239,12 @@ X: Documentation/power/ X: Documentation/spi/ X: Documentation/userspace-api/media/ +DOCUMENTATION PROCESS +M: Jonathan Corbet +S: Maintained +F: Documentation/process/ +L: workflows@vger.kernel.org + DOCUMENTATION REPORTING ISSUES M: Thorsten Leemhuis L: linux-doc@vger.kernel.org @@ -11363,6 +11369,8 @@ L: linux-kselftest@vger.kernel.org L: kunit-dev@googlegroups.com S: Maintained W: https://google.github.io/kunit-docs/third_party/kernel/docs/ +T: git git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git kunit +T: git git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git kunit-fixes F: Documentation/dev-tools/kunit/ F: include/kunit/ F: lib/kunit/ @@ -15330,7 +15338,7 @@ OMAP DISPLAY SUBSYSTEM and FRAMEBUFFER SUPPORT (DSS2) L: linux-omap@vger.kernel.org L: linux-fbdev@vger.kernel.org S: Orphan -F: Documentation/arm/omap/dss.rst +F: Documentation/arch/arm/omap/dss.rst F: drivers/video/fbdev/omap2/ OMAP FRAMEBUFFER SUPPORT @@ -17829,7 +17837,7 @@ M: Boqun Feng R: Steven Rostedt R: Mathieu Desnoyers R: Lai Jiangshan -R: Zqiang +R: Zqiang L: rcu@vger.kernel.org S: Supported W: http://www.rdrop.com/users/paulmck/RCU/ diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 1b158bbab29a..cef741bb03b3 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1782,7 +1782,7 @@ config VFP Say Y to include VFP support code in the kernel. This is needed if your hardware includes a VFP unit. - Please see for + Please see for release notes and additional status information. Say N if your target does not have VFP hardware. diff --git a/arch/arm/boot/compressed/atags_to_fdt.c b/arch/arm/boot/compressed/atags_to_fdt.c index 1feb6b0f7a1f..627752f18661 100644 --- a/arch/arm/boot/compressed/atags_to_fdt.c +++ b/arch/arm/boot/compressed/atags_to_fdt.c @@ -2,6 +2,7 @@ #include #include #include +#include "misc.h" #if defined(CONFIG_ARM_ATAG_DTB_COMPAT_CMDLINE_EXTEND) #define do_extend_cmdline 1 diff --git a/arch/arm/boot/compressed/fdt_check_mem_start.c b/arch/arm/boot/compressed/fdt_check_mem_start.c index 9291a2661bdf..aa856567fd33 100644 --- a/arch/arm/boot/compressed/fdt_check_mem_start.c +++ b/arch/arm/boot/compressed/fdt_check_mem_start.c @@ -3,6 +3,7 @@ #include #include #include +#include "misc.h" static const void *get_prop(const void *fdt, const char *node_path, const char *property, int minlen) diff --git a/arch/arm/boot/compressed/misc.c b/arch/arm/boot/compressed/misc.c index abfed1aa2baa..6b4baa6a9a50 100644 --- a/arch/arm/boot/compressed/misc.c +++ b/arch/arm/boot/compressed/misc.c @@ -103,9 +103,6 @@ static void putstr(const char *ptr) /* * gzip declarations */ -extern char input_data[]; -extern char input_data_end[]; - unsigned char *output_data; unsigned long free_mem_ptr; @@ -131,9 +128,6 @@ asmlinkage void __div0(void) error("Attempting division by 0!"); } -extern int do_decompress(u8 *input, int len, u8 *output, void (*error)(char *x)); - - void decompress_kernel(unsigned long output_start, unsigned long free_mem_ptr_p, unsigned long free_mem_ptr_end_p, diff --git a/arch/arm/boot/compressed/misc.h b/arch/arm/boot/compressed/misc.h index c958dccd1d97..6da00a26ac08 100644 --- a/arch/arm/boot/compressed/misc.h +++ b/arch/arm/boot/compressed/misc.h @@ -6,5 +6,16 @@ void error(char *x) __noreturn; extern unsigned long free_mem_ptr; extern unsigned long free_mem_end_ptr; +void __div0(void); +void +decompress_kernel(unsigned long output_start, unsigned long free_mem_ptr_p, + unsigned long free_mem_ptr_end_p, int arch_id); +void fortify_panic(const char *name); +int atags_to_fdt(void *atag_list, void *fdt, int total_space); +uint32_t fdt_check_mem_start(uint32_t mem_start, const void *fdt); +int do_decompress(u8 *input, int len, u8 *output, void (*error)(char *x)); + +extern char input_data[]; +extern char input_data_end[]; #endif diff --git a/arch/arm/common/mcpm_entry.c b/arch/arm/common/mcpm_entry.c index 8a9aeeb504dd..e013ff1168d3 100644 --- a/arch/arm/common/mcpm_entry.c +++ b/arch/arm/common/mcpm_entry.c @@ -21,7 +21,7 @@ /* * The public API for this code is documented in arch/arm/include/asm/mcpm.h. * For a comprehensive description of the main algorithm used here, please - * see Documentation/arm/cluster-pm-race-avoidance.rst. + * see Documentation/arch/arm/cluster-pm-race-avoidance.rst. */ struct sync_struct mcpm_sync; diff --git a/arch/arm/common/mcpm_head.S b/arch/arm/common/mcpm_head.S index 299495c43dfd..f590e803ca11 100644 --- a/arch/arm/common/mcpm_head.S +++ b/arch/arm/common/mcpm_head.S @@ -5,7 +5,7 @@ * Created by: Nicolas Pitre, March 2012 * Copyright: (C) 2012-2013 Linaro Limited * - * Refer to Documentation/arm/cluster-pm-race-avoidance.rst + * Refer to Documentation/arch/arm/cluster-pm-race-avoidance.rst * for details of the synchronisation algorithms used here. */ diff --git a/arch/arm/common/vlock.S b/arch/arm/common/vlock.S index 1fa09c4697ed..c5eaed5a76f0 100644 --- a/arch/arm/common/vlock.S +++ b/arch/arm/common/vlock.S @@ -6,7 +6,7 @@ * Copyright: (C) 2012-2013 Linaro Limited * * This algorithm is described in more detail in - * Documentation/arm/vlocks.rst. + * Documentation/arch/arm/vlocks.rst. */ #include diff --git a/arch/arm/include/asm/ftrace.h b/arch/arm/include/asm/ftrace.h index 7e9251ca29fe..5be3ddc96a50 100644 --- a/arch/arm/include/asm/ftrace.h +++ b/arch/arm/include/asm/ftrace.h @@ -75,6 +75,10 @@ static inline bool arch_syscall_match_sym_name(const char *sym, return !strcasecmp(sym, name); } +void prepare_ftrace_return(unsigned long *parent, unsigned long self, + unsigned long frame_pointer, + unsigned long stack_pointer); + #endif /* ifndef __ASSEMBLY__ */ #endif /* _ASM_ARM_FTRACE */ diff --git a/arch/arm/include/asm/mach/arch.h b/arch/arm/include/asm/mach/arch.h index 9349e7a82c9c..2b18a258204d 100644 --- a/arch/arm/include/asm/mach/arch.h +++ b/arch/arm/include/asm/mach/arch.h @@ -56,7 +56,6 @@ struct machine_desc { void (*init_time)(void); void (*init_machine)(void); void (*init_late)(void); - void (*handle_irq)(struct pt_regs *); void (*restart)(enum reboot_mode, const char *); }; diff --git a/arch/arm/include/asm/page.h b/arch/arm/include/asm/page.h index 74bb5947b387..28c63d172a96 100644 --- a/arch/arm/include/asm/page.h +++ b/arch/arm/include/asm/page.h @@ -113,6 +113,28 @@ struct cpu_user_fns { unsigned long vaddr, struct vm_area_struct *vma); }; +void fa_copy_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma); +void fa_clear_user_highpage(struct page *page, unsigned long vaddr); +void feroceon_copy_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma); +void feroceon_clear_user_highpage(struct page *page, unsigned long vaddr); +void v4_mc_copy_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma); +void v4_mc_clear_user_highpage(struct page *page, unsigned long vaddr); +void v4wb_copy_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma); +void v4wb_clear_user_highpage(struct page *page, unsigned long vaddr); +void v4wt_copy_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma); +void v4wt_clear_user_highpage(struct page *page, unsigned long vaddr); +void xsc3_mc_copy_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma); +void xsc3_mc_clear_user_highpage(struct page *page, unsigned long vaddr); +void xscale_mc_copy_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma); +void xscale_mc_clear_user_highpage(struct page *page, unsigned long vaddr); + #ifdef MULTI_USER extern struct cpu_user_fns cpu_user; diff --git a/arch/arm/include/asm/ptrace.h b/arch/arm/include/asm/ptrace.h index 483b8ddfcb82..7f44e88d1f25 100644 --- a/arch/arm/include/asm/ptrace.h +++ b/arch/arm/include/asm/ptrace.h @@ -193,5 +193,8 @@ static inline unsigned long it_advance(unsigned long cpsr) return cpsr; } +int syscall_trace_enter(struct pt_regs *regs); +void syscall_trace_exit(struct pt_regs *regs); + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm/include/asm/setup.h b/arch/arm/include/asm/setup.h index ba0872a8dcda..546af8b1e3f6 100644 --- a/arch/arm/include/asm/setup.h +++ b/arch/arm/include/asm/setup.h @@ -5,7 +5,7 @@ * Copyright (C) 1997-1999 Russell King * * Structure passed to kernel to tell it about the - * hardware it's running on. See Documentation/arm/setup.rst + * hardware it's running on. See Documentation/arch/arm/setup.rst * for more info. */ #ifndef __ASMARM_SETUP_H @@ -28,4 +28,11 @@ extern void save_atags(const struct tag *tags); static inline void save_atags(const struct tag *tags) { } #endif +struct machine_desc; +void init_default_cache_policy(unsigned long); +void paging_init(const struct machine_desc *desc); +void early_mm_init(const struct machine_desc *); +void adjust_lowmem_bounds(void); +void setup_dma_zone(const struct machine_desc *desc); + #endif diff --git a/arch/arm/include/asm/signal.h b/arch/arm/include/asm/signal.h index 430be7774402..8b84092d1518 100644 --- a/arch/arm/include/asm/signal.h +++ b/arch/arm/include/asm/signal.h @@ -22,4 +22,9 @@ typedef struct { #define __ARCH_HAS_SA_RESTORER #include + +void do_rseq_syscall(struct pt_regs *regs); +int do_work_pending(struct pt_regs *regs, unsigned int thread_flags, + int syscall); + #endif diff --git a/arch/arm/include/asm/spectre.h b/arch/arm/include/asm/spectre.h index 85f9e538fb32..d9c28b3b6b62 100644 --- a/arch/arm/include/asm/spectre.h +++ b/arch/arm/include/asm/spectre.h @@ -35,4 +35,8 @@ static inline void spectre_v2_update_state(unsigned int state, int spectre_bhb_update_vectors(unsigned int method); +void cpu_v7_ca8_ibe(void); +void cpu_v7_ca15_ibe(void); +void cpu_v7_bugs_init(void); + #endif diff --git a/arch/arm/include/asm/suspend.h b/arch/arm/include/asm/suspend.h index 506314265c6f..be81b9ca2ea1 100644 --- a/arch/arm/include/asm/suspend.h +++ b/arch/arm/include/asm/suspend.h @@ -13,5 +13,6 @@ extern void cpu_resume(void); extern void cpu_resume_no_hyp(void); extern void cpu_resume_arm(void); extern int cpu_suspend(unsigned long, int (*)(unsigned long)); +extern void __cpu_suspend_save(u32 *ptr, u32 ptrsz, u32 sp, u32 *save_ptr); #endif diff --git a/arch/arm/include/asm/syscalls.h b/arch/arm/include/asm/syscalls.h new file mode 100644 index 000000000000..5912e7cffa6a --- /dev/null +++ b/arch/arm/include/asm/syscalls.h @@ -0,0 +1,51 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_SYSCALLS_H +#define __ASM_SYSCALLS_H + +#include +#include + +struct pt_regs; +asmlinkage int sys_sigreturn(struct pt_regs *regs); +asmlinkage int sys_rt_sigreturn(struct pt_regs *regs); +asmlinkage long sys_arm_fadvise64_64(int fd, int advice, + loff_t offset, loff_t len); + +struct oldabi_stat64; +asmlinkage long sys_oabi_stat64(const char __user * filename, + struct oldabi_stat64 __user * statbuf); +asmlinkage long sys_oabi_lstat64(const char __user * filename, + struct oldabi_stat64 __user * statbuf); +asmlinkage long sys_oabi_fstat64(unsigned long fd, + struct oldabi_stat64 __user * statbuf); +asmlinkage long sys_oabi_fstatat64(int dfd, + const char __user *filename, + struct oldabi_stat64 __user *statbuf, + int flag); +asmlinkage long sys_oabi_fcntl64(unsigned int fd, unsigned int cmd, + unsigned long arg); +struct oabi_epoll_event; +asmlinkage long sys_oabi_epoll_ctl(int epfd, int op, int fd, + struct oabi_epoll_event __user *event); +struct oabi_sembuf; +struct old_timespec32; +asmlinkage long sys_oabi_semtimedop(int semid, + struct oabi_sembuf __user *tsops, + unsigned nsops, + const struct old_timespec32 __user *timeout); +asmlinkage long sys_oabi_semop(int semid, struct oabi_sembuf __user *tsops, + unsigned nsops); +asmlinkage int sys_oabi_ipc(uint call, int first, int second, int third, + void __user *ptr, long fifth); +struct sockaddr; +asmlinkage long sys_oabi_bind(int fd, struct sockaddr __user *addr, int addrlen); +asmlinkage long sys_oabi_connect(int fd, struct sockaddr __user *addr, int addrlen); +asmlinkage long sys_oabi_sendto(int fd, void __user *buff, + size_t len, unsigned flags, + struct sockaddr __user *addr, + int addrlen); +struct user_msghdr; +asmlinkage long sys_oabi_sendmsg(int fd, struct user_msghdr __user *msg, unsigned flags); +asmlinkage long sys_oabi_socketcall(int call, unsigned long __user *args); + +#endif diff --git a/arch/arm/include/asm/tcm.h b/arch/arm/include/asm/tcm.h index d8bd8a4b0ede..e1f7dca86a22 100644 --- a/arch/arm/include/asm/tcm.h +++ b/arch/arm/include/asm/tcm.h @@ -9,9 +9,7 @@ #ifndef __ASMARM_TCM_H #define __ASMARM_TCM_H -#ifndef CONFIG_HAVE_TCM -#error "You should not be including tcm.h unless you have a TCM!" -#endif +#ifdef CONFIG_HAVE_TCM #include @@ -29,4 +27,11 @@ void tcm_free(void *addr, size_t len); bool tcm_dtcm_present(void); bool tcm_itcm_present(void); +void __init tcm_init(void); +#else +/* No TCM support, just blank inlines to be optimized out */ +static inline void tcm_init(void) +{ +} +#endif #endif diff --git a/arch/arm/include/asm/traps.h b/arch/arm/include/asm/traps.h index 987fefb0a4db..0aaefe3e1700 100644 --- a/arch/arm/include/asm/traps.h +++ b/arch/arm/include/asm/traps.h @@ -35,4 +35,13 @@ extern void ptrace_break(struct pt_regs *regs); extern void *vectors_page; +asmlinkage void dump_backtrace_stm(u32 *stack, u32 instruction, const char *loglvl); +asmlinkage void do_undefinstr(struct pt_regs *regs); +asmlinkage void handle_fiq_as_nmi(struct pt_regs *regs); +asmlinkage void bad_mode(struct pt_regs *regs, int reason); +asmlinkage int arm_syscall(int no, struct pt_regs *regs); +asmlinkage void baddataabort(int code, unsigned long instr, struct pt_regs *regs); +asmlinkage void __div0(void); +asmlinkage void handle_bad_stack(struct pt_regs *regs); + #endif diff --git a/arch/arm/include/asm/unwind.h b/arch/arm/include/asm/unwind.h index b51f85417f58..d60b09a5acfc 100644 --- a/arch/arm/include/asm/unwind.h +++ b/arch/arm/include/asm/unwind.h @@ -40,6 +40,10 @@ extern void unwind_table_del(struct unwind_table *tab); extern void unwind_backtrace(struct pt_regs *regs, struct task_struct *tsk, const char *loglvl); +void __aeabi_unwind_cpp_pr0(void); +void __aeabi_unwind_cpp_pr1(void); +void __aeabi_unwind_cpp_pr2(void); + #endif /* !__ASSEMBLY__ */ #ifdef CONFIG_ARM_UNWIND diff --git a/arch/arm/include/asm/vdso.h b/arch/arm/include/asm/vdso.h index 5b85889f82ee..422c3afa806a 100644 --- a/arch/arm/include/asm/vdso.h +++ b/arch/arm/include/asm/vdso.h @@ -24,6 +24,11 @@ static inline void arm_install_vdso(struct mm_struct *mm, unsigned long addr) #endif /* CONFIG_VDSO */ +int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts); +int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts); +int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz); +int __vdso_clock_getres(clockid_t clock_id, struct old_timespec32 *res); + #endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */ diff --git a/arch/arm/include/asm/vfp.h b/arch/arm/include/asm/vfp.h index 157ea3426158..5b57b8768bac 100644 --- a/arch/arm/include/asm/vfp.h +++ b/arch/arm/include/asm/vfp.h @@ -102,6 +102,7 @@ #ifndef __ASSEMBLY__ void vfp_disable(void); +void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs); #endif #endif /* __ASM_VFP_H */ diff --git a/arch/arm/include/uapi/asm/setup.h b/arch/arm/include/uapi/asm/setup.h index 25ceda63b284..8e50e034fec7 100644 --- a/arch/arm/include/uapi/asm/setup.h +++ b/arch/arm/include/uapi/asm/setup.h @@ -9,7 +9,7 @@ * published by the Free Software Foundation. * * Structure passed to kernel to tell it about the - * hardware it's running on. See Documentation/arm/setup.rst + * hardware it's running on. See Documentation/arch/arm/setup.rst * for more info. */ #ifndef _UAPI__ASMARM_SETUP_H diff --git a/arch/arm/kernel/atags_parse.c b/arch/arm/kernel/atags_parse.c index 373b61f9a4f0..33f6eb5213a5 100644 --- a/arch/arm/kernel/atags_parse.c +++ b/arch/arm/kernel/atags_parse.c @@ -127,7 +127,7 @@ static int __init parse_tag_cmdline(const struct tag *tag) #elif defined(CONFIG_CMDLINE_FORCE) pr_warn("Ignoring tag cmdline (using the default kernel command line)\n"); #else - strlcpy(default_command_line, tag->u.cmdline.cmdline, + strscpy(default_command_line, tag->u.cmdline.cmdline, COMMAND_LINE_SIZE); #endif return 0; @@ -224,7 +224,7 @@ setup_machine_tags(void *atags_vaddr, unsigned int machine_nr) } /* parse_early_param needs a boot_command_line */ - strlcpy(boot_command_line, from, COMMAND_LINE_SIZE); + strscpy(boot_command_line, from, COMMAND_LINE_SIZE); return mdesc; } diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index c39303e5c234..291dc48d6bed 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -875,7 +875,7 @@ ENDPROC(__bad_stack) * existing ones. This mechanism should be used only for things that are * really small and justified, and not be abused freely. * - * See Documentation/arm/kernel_user_helpers.rst for formal definitions. + * See Documentation/arch/arm/kernel_user_helpers.rst for formal definitions. */ THUMB( .arm ) diff --git a/arch/arm/kernel/fiq.c b/arch/arm/kernel/fiq.c index 98ca3e3fa847..d2c8e5313539 100644 --- a/arch/arm/kernel/fiq.c +++ b/arch/arm/kernel/fiq.c @@ -45,6 +45,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm/kernel/head-inflate-data.c b/arch/arm/kernel/head-inflate-data.c index 89a52104d32a..225c0699a12c 100644 --- a/arch/arm/kernel/head-inflate-data.c +++ b/arch/arm/kernel/head-inflate-data.c @@ -8,16 +8,13 @@ #include #include +#include "head.h" /* for struct inflate_state */ #include "../../../lib/zlib_inflate/inftrees.h" #include "../../../lib/zlib_inflate/inflate.h" #include "../../../lib/zlib_inflate/infutil.h" -extern char __data_loc[]; -extern char _edata_loc[]; -extern char _sdata[]; - /* * This code is called very early during the boot process to decompress * the .data segment stored compressed in ROM. Therefore none of the global diff --git a/arch/arm/kernel/head.h b/arch/arm/kernel/head.h new file mode 100644 index 000000000000..0eb5accf7141 --- /dev/null +++ b/arch/arm/kernel/head.h @@ -0,0 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0-only + +extern char __data_loc[]; +extern char _edata_loc[]; +extern char _sdata[]; + +int __init __inflate_kernel_data(void); diff --git a/arch/arm/kernel/module.c b/arch/arm/kernel/module.c index d59c36dc0494..e74d84f58b77 100644 --- a/arch/arm/kernel/module.c +++ b/arch/arm/kernel/module.c @@ -169,8 +169,7 @@ apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, unsigned int symindex, offset = __mem_to_opcode_arm(*(u32 *)loc); offset = (offset & 0x00ffffff) << 2; - if (offset & 0x02000000) - offset -= 0x04000000; + offset = sign_extend32(offset, 25); offset += sym->st_value - loc; @@ -236,7 +235,7 @@ apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, unsigned int symindex, case R_ARM_MOVT_PREL: offset = tmp = __mem_to_opcode_arm(*(u32 *)loc); offset = ((offset & 0xf0000) >> 4) | (offset & 0xfff); - offset = (offset ^ 0x8000) - 0x8000; + offset = sign_extend32(offset, 15); offset += sym->st_value; if (ELF32_R_TYPE(rel->r_info) == R_ARM_MOVT_PREL || @@ -344,8 +343,7 @@ apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, unsigned int symindex, ((~(j2 ^ sign) & 1) << 22) | ((upper & 0x03ff) << 12) | ((lower & 0x07ff) << 1); - if (offset & 0x01000000) - offset -= 0x02000000; + offset = sign_extend32(offset, 24); offset += sym->st_value - loc; /* @@ -401,7 +399,7 @@ apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, unsigned int symindex, offset = ((upper & 0x000f) << 12) | ((upper & 0x0400) << 1) | ((lower & 0x7000) >> 4) | (lower & 0x00ff); - offset = (offset ^ 0x8000) - 0x8000; + offset = sign_extend32(offset, 15); offset += sym->st_value; if (ELF32_R_TYPE(rel->r_info) == R_ARM_THM_MOVT_PREL || diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index 75cd4699e7b3..c66b560562b3 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -76,13 +76,6 @@ static int __init fpe_setup(char *line) __setup("fpe=", fpe_setup); #endif -extern void init_default_cache_policy(unsigned long); -extern void paging_init(const struct machine_desc *desc); -extern void early_mm_init(const struct machine_desc *); -extern void adjust_lowmem_bounds(void); -extern enum reboot_mode reboot_mode; -extern void setup_dma_zone(const struct machine_desc *desc); - unsigned int processor_id; EXPORT_SYMBOL(processor_id); unsigned int __machine_arch_type __read_mostly; @@ -1142,7 +1135,7 @@ void __init setup_arch(char **cmdline_p) setup_initial_init_mm(_text, _etext, _edata, _end); /* populate cmd_line too for later use, preserving boot_command_line */ - strlcpy(cmd_line, boot_command_line, COMMAND_LINE_SIZE); + strscpy(cmd_line, boot_command_line, COMMAND_LINE_SIZE); *cmdline_p = cmd_line; early_fixmap_init(); @@ -1198,10 +1191,6 @@ void __init setup_arch(char **cmdline_p) reserve_crashkernel(); -#ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER - handle_arch_irq = mdesc->handle_irq; -#endif - #ifdef CONFIG_VT #if defined(CONFIG_VGA_CONSOLE) conswitchp = &vga_con; diff --git a/arch/arm/kernel/signal.c b/arch/arm/kernel/signal.c index e07f359254c3..8d0afa11bed5 100644 --- a/arch/arm/kernel/signal.c +++ b/arch/arm/kernel/signal.c @@ -18,6 +18,7 @@ #include #include #include +#include #include "signal.h" diff --git a/arch/arm/kernel/sys_arm.c b/arch/arm/kernel/sys_arm.c index a5f183cfecb1..0141e9bb02e8 100644 --- a/arch/arm/kernel/sys_arm.c +++ b/arch/arm/kernel/sys_arm.c @@ -24,6 +24,7 @@ #include #include #include +#include /* * Since loff_t is a 64 bit type we avoid a lot of ABI hassle diff --git a/arch/arm/kernel/sys_oabi-compat.c b/arch/arm/kernel/sys_oabi-compat.c index 006163195d67..d00f4040a9f5 100644 --- a/arch/arm/kernel/sys_oabi-compat.c +++ b/arch/arm/kernel/sys_oabi-compat.c @@ -10,6 +10,8 @@ * Copyright: MontaVista Software, Inc. */ +#include + /* * The legacy ABI and the new ARM EABI have different rules making some * syscalls incompatible especially with structure arguments. diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 40c7c807d67f..3bad79db5d6e 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -756,6 +756,7 @@ void __readwrite_bug(const char *fn) } EXPORT_SYMBOL(__readwrite_bug); +#ifdef CONFIG_MMU void __pte_error(const char *file, int line, pte_t pte) { pr_err("%s:%d: bad pte %08llx.\n", file, line, (long long)pte_val(pte)); @@ -770,6 +771,7 @@ void __pgd_error(const char *file, int line, pgd_t pgd) { pr_err("%s:%d: bad pgd %08llx.\n", file, line, (long long)pgd_val(pgd)); } +#endif asmlinkage void __div0(void) { diff --git a/arch/arm/kernel/vdso.c b/arch/arm/kernel/vdso.c index 3408269d19c7..f297d66a8a76 100644 --- a/arch/arm/kernel/vdso.c +++ b/arch/arm/kernel/vdso.c @@ -135,7 +135,7 @@ static Elf32_Sym * __init find_symbol(struct elfinfo *lib, const char *symname) if (lib->dynsym[i].st_name == 0) continue; - strlcpy(name, lib->dynstr + lib->dynsym[i].st_name, + strscpy(name, lib->dynstr + lib->dynsym[i].st_name, MAX_SYMNAME); c = strchr(name, '@'); if (c) diff --git a/arch/arm/mach-exynos/common.h b/arch/arm/mach-exynos/common.h index 29eb075b24a4..b5287ff1c542 100644 --- a/arch/arm/mach-exynos/common.h +++ b/arch/arm/mach-exynos/common.h @@ -106,7 +106,7 @@ void exynos_firmware_init(void); #define C2_STATE (1 << 3) /* * Magic values for bootloader indicating chosen low power mode. - * See also Documentation/arm/samsung/bootloader-interface.rst + * See also Documentation/arch/arm/samsung/bootloader-interface.rst */ #define EXYNOS_SLEEP_MAGIC 0x00000bad #define EXYNOS_AFTR_MAGIC 0xfcba0d10 diff --git a/arch/arm/mach-mxs/mach-mxs.c b/arch/arm/mach-mxs/mach-mxs.c index 51e47053c816..3faf9a1e3e36 100644 --- a/arch/arm/mach-mxs/mach-mxs.c +++ b/arch/arm/mach-mxs/mach-mxs.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include #include @@ -472,7 +471,6 @@ static const char *const mxs_dt_compat[] __initconst = { }; DT_MACHINE_START(MXS, "Freescale MXS (Device Tree)") - .handle_irq = icoll_handle_irq, .init_machine = mxs_machine_init, .init_late = mxs_pm_init, .dt_compat = mxs_dt_compat, diff --git a/arch/arm/mach-omap1/board-ams-delta.c b/arch/arm/mach-omap1/board-ams-delta.c index 9108c871d129..88139200449e 100644 --- a/arch/arm/mach-omap1/board-ams-delta.c +++ b/arch/arm/mach-omap1/board-ams-delta.c @@ -877,7 +877,6 @@ MACHINE_START(AMS_DELTA, "Amstrad E3 (Delta)") .map_io = ams_delta_map_io, .init_early = omap1_init_early, .init_irq = omap1_init_irq, - .handle_irq = omap1_handle_irq, .init_machine = ams_delta_init, .init_late = ams_delta_init_late, .init_time = omap1_timer_init, diff --git a/arch/arm/mach-omap1/board-nokia770.c b/arch/arm/mach-omap1/board-nokia770.c index a501a473ffd6..b56cea9f9d2f 100644 --- a/arch/arm/mach-omap1/board-nokia770.c +++ b/arch/arm/mach-omap1/board-nokia770.c @@ -291,7 +291,6 @@ MACHINE_START(NOKIA770, "Nokia 770") .map_io = omap1_map_io, .init_early = omap1_init_early, .init_irq = omap1_init_irq, - .handle_irq = omap1_handle_irq, .init_machine = omap_nokia770_init, .init_late = omap1_init_late, .init_time = omap1_timer_init, diff --git a/arch/arm/mach-omap1/board-osk.c b/arch/arm/mach-omap1/board-osk.c index df758c1f9237..46eda4ff4797 100644 --- a/arch/arm/mach-omap1/board-osk.c +++ b/arch/arm/mach-omap1/board-osk.c @@ -389,7 +389,6 @@ MACHINE_START(OMAP_OSK, "TI-OSK") .map_io = omap1_map_io, .init_early = omap1_init_early, .init_irq = omap1_init_irq, - .handle_irq = omap1_handle_irq, .init_machine = osk_init, .init_late = omap1_init_late, .init_time = omap1_timer_init, diff --git a/arch/arm/mach-omap1/board-palmte.c b/arch/arm/mach-omap1/board-palmte.c index f79c497f04d5..91df3dc365af 100644 --- a/arch/arm/mach-omap1/board-palmte.c +++ b/arch/arm/mach-omap1/board-palmte.c @@ -259,7 +259,6 @@ MACHINE_START(OMAP_PALMTE, "OMAP310 based Palm Tungsten E") .map_io = omap1_map_io, .init_early = omap1_init_early, .init_irq = omap1_init_irq, - .handle_irq = omap1_handle_irq, .init_machine = omap_palmte_init, .init_late = omap1_init_late, .init_time = omap1_timer_init, diff --git a/arch/arm/mach-omap1/board-sx1.c b/arch/arm/mach-omap1/board-sx1.c index 0c0cdd5e77c7..3ae295af96fd 100644 --- a/arch/arm/mach-omap1/board-sx1.c +++ b/arch/arm/mach-omap1/board-sx1.c @@ -338,7 +338,6 @@ MACHINE_START(SX1, "OMAP310 based Siemens SX1") .map_io = omap1_map_io, .init_early = omap1_init_early, .init_irq = omap1_init_irq, - .handle_irq = omap1_handle_irq, .init_machine = omap_sx1_init, .init_late = omap1_init_late, .init_time = omap1_timer_init, diff --git a/arch/arm/mach-omap1/irq.c b/arch/arm/mach-omap1/irq.c index bfc7ab010ae2..3d9e72e1eddc 100644 --- a/arch/arm/mach-omap1/irq.c +++ b/arch/arm/mach-omap1/irq.c @@ -37,6 +37,7 @@ */ #include #include +#include #include #include #include @@ -254,4 +255,6 @@ void __init omap1_init_irq(void) ct = irq_data_get_chip_type(d); ct->chip.irq_unmask(d); } + + set_handle_irq(omap1_handle_irq); } diff --git a/arch/arm/mach-pxa/gumstix.c b/arch/arm/mach-pxa/gumstix.c index 72b08a9bf0fd..6b7197ae3c72 100644 --- a/arch/arm/mach-pxa/gumstix.c +++ b/arch/arm/mach-pxa/gumstix.c @@ -233,7 +233,6 @@ MACHINE_START(GUMSTIX, "Gumstix") .map_io = pxa25x_map_io, .nr_irqs = PXA_NR_IRQS, .init_irq = pxa25x_init_irq, - .handle_irq = pxa25x_handle_irq, .init_time = pxa_timer_init, .init_machine = gumstix_init, .restart = pxa_restart, diff --git a/arch/arm/mach-pxa/pxa25x.c b/arch/arm/mach-pxa/pxa25x.c index 1b83be181bab..032dc897fe94 100644 --- a/arch/arm/mach-pxa/pxa25x.c +++ b/arch/arm/mach-pxa/pxa25x.c @@ -143,6 +143,7 @@ set_pwer: void __init pxa25x_init_irq(void) { pxa_init_irq(32, pxa25x_set_wake); + set_handle_irq(pxa25x_handle_irq); } static int __init __init diff --git a/arch/arm/mach-pxa/pxa27x.c b/arch/arm/mach-pxa/pxa27x.c index 4135ba2877c4..c9b56424b653 100644 --- a/arch/arm/mach-pxa/pxa27x.c +++ b/arch/arm/mach-pxa/pxa27x.c @@ -228,6 +228,7 @@ static int pxa27x_set_wake(struct irq_data *d, unsigned int on) void __init pxa27x_init_irq(void) { pxa_init_irq(34, pxa27x_set_wake); + set_handle_irq(pxa27x_handle_irq); } static int __init diff --git a/arch/arm/mach-pxa/spitz.c b/arch/arm/mach-pxa/spitz.c index 4325bdc2b9ff..042922a0a9d6 100644 --- a/arch/arm/mach-pxa/spitz.c +++ b/arch/arm/mach-pxa/spitz.c @@ -1043,7 +1043,6 @@ MACHINE_START(SPITZ, "SHARP Spitz") .map_io = pxa27x_map_io, .nr_irqs = PXA_NR_IRQS, .init_irq = pxa27x_init_irq, - .handle_irq = pxa27x_handle_irq, .init_machine = spitz_init, .init_time = pxa_timer_init, .restart = spitz_restart, @@ -1056,7 +1055,6 @@ MACHINE_START(BORZOI, "SHARP Borzoi") .map_io = pxa27x_map_io, .nr_irqs = PXA_NR_IRQS, .init_irq = pxa27x_init_irq, - .handle_irq = pxa27x_handle_irq, .init_machine = spitz_init, .init_time = pxa_timer_init, .restart = spitz_restart, @@ -1069,7 +1067,6 @@ MACHINE_START(AKITA, "SHARP Akita") .map_io = pxa27x_map_io, .nr_irqs = PXA_NR_IRQS, .init_irq = pxa27x_init_irq, - .handle_irq = pxa27x_handle_irq, .init_machine = spitz_init, .init_time = pxa_timer_init, .restart = spitz_restart, diff --git a/arch/arm/mach-sti/Kconfig b/arch/arm/mach-sti/Kconfig index b2d45cf10a3c..b3842c971d31 100644 --- a/arch/arm/mach-sti/Kconfig +++ b/arch/arm/mach-sti/Kconfig @@ -21,7 +21,7 @@ menuconfig ARCH_STI help Include support for STMicroelectronics' STiH415/416, STiH407/10 and STiH418 family SoCs using the Device Tree for discovery. More - information can be found in Documentation/arm/sti/ and + information can be found in Documentation/arch/arm/sti/ and Documentation/devicetree. if ARCH_STI diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index be183ed1232d..c164cde50243 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -712,7 +712,7 @@ config ARM_VIRT_EXT assistance. A compliant bootloader is required in order to make maximum - use of this feature. Refer to Documentation/arm/booting.rst for + use of this feature. Refer to Documentation/arch/arm/booting.rst for details. config SWP_EMULATE @@ -904,7 +904,7 @@ config KUSER_HELPERS the CPU type fitted to the system. This permits binaries to be run on ARMv4 through to ARMv7 without modification. - See Documentation/arm/kernel_user_helpers.rst for details. + See Documentation/arch/arm/kernel_user_helpers.rst for details. However, the fixed address nature of these helpers can be used by ROP (return orientated programming) authors when creating diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index b4a33358d2e9..bc4ed5ce3e00 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -258,12 +258,14 @@ static struct dma_contig_early_reserve dma_mmu_remap[MAX_CMA_AREAS] __initdata; static int dma_mmu_remap_num __initdata; +#ifdef CONFIG_DMA_CMA void __init dma_contiguous_early_fixup(phys_addr_t base, unsigned long size) { dma_mmu_remap[dma_mmu_remap_num].base = base; dma_mmu_remap[dma_mmu_remap_num].size = size; dma_mmu_remap_num++; } +#endif void __init dma_contiguous_remap(void) { diff --git a/arch/arm/mm/fault.h b/arch/arm/mm/fault.h index 54927ba1fa6e..e8f8c1902544 100644 --- a/arch/arm/mm/fault.h +++ b/arch/arm/mm/fault.h @@ -37,5 +37,9 @@ static inline int fsr_fs(unsigned int fsr) void do_bad_area(unsigned long addr, unsigned int fsr, struct pt_regs *regs); void early_abt_enable(void); +asmlinkage void do_DataAbort(unsigned long addr, unsigned int fsr, + struct pt_regs *regs); +asmlinkage void do_PrefetchAbort(unsigned long addr, unsigned int ifsr, + struct pt_regs *regs); #endif /* __ARCH_ARM_FAULT_H */ diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c index 7ff9feea13a6..2508be91b7a0 100644 --- a/arch/arm/mm/flush.c +++ b/arch/arm/mm/flush.c @@ -354,6 +354,7 @@ EXPORT_SYMBOL(flush_dcache_page); * memcpy() to/from page * if written to page, flush_dcache_page() */ +void __flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned long vmaddr); void __flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned long vmaddr) { unsigned long pfn; diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c index 463fc2a8448f..f3a52c08a200 100644 --- a/arch/arm/mm/mmu.c +++ b/arch/arm/mm/mmu.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -37,7 +38,6 @@ #include "fault.h" #include "mm.h" -#include "tcm.h" extern unsigned long __atags_pointer; diff --git a/arch/arm/mm/nommu.c b/arch/arm/mm/nommu.c index 53f2d8774fdb..43cfd06bbeba 100644 --- a/arch/arm/mm/nommu.c +++ b/arch/arm/mm/nommu.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "mm.h" diff --git a/arch/arm/mm/tcm.h b/arch/arm/mm/tcm.h deleted file mode 100644 index 6b80a760d875..000000000000 --- a/arch/arm/mm/tcm.h +++ /dev/null @@ -1,17 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * Copyright (C) 2008-2009 ST-Ericsson AB - * TCM memory handling for ARM systems - * - * Author: Linus Walleij - * Author: Rickard Andersson - */ - -#ifdef CONFIG_HAVE_TCM -void __init tcm_init(void); -#else -/* No TCM support, just blank inlines to be optimized out */ -static inline void tcm_init(void) -{ -} -#endif diff --git a/arch/arm/probes/kprobes/checkers-common.c b/arch/arm/probes/kprobes/checkers-common.c index 4d720990cf2a..eba7ac4725c0 100644 --- a/arch/arm/probes/kprobes/checkers-common.c +++ b/arch/arm/probes/kprobes/checkers-common.c @@ -40,7 +40,7 @@ enum probes_insn checker_stack_use_imm_0xx(probes_opcode_t insn, * Different from other insn uses imm8, the real addressing offset of * STRD in T32 encoding should be imm8 * 4. See ARMARM description. */ -enum probes_insn checker_stack_use_t32strd(probes_opcode_t insn, +static enum probes_insn checker_stack_use_t32strd(probes_opcode_t insn, struct arch_probes_insn *asi, const struct decode_header *h) { diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c index 9090c3a74dcc..d8238da095df 100644 --- a/arch/arm/probes/kprobes/core.c +++ b/arch/arm/probes/kprobes/core.c @@ -233,7 +233,7 @@ singlestep(struct kprobe *p, struct pt_regs *regs, struct kprobe_ctlblk *kcb) * kprobe, and that level is reserved for user kprobe handlers, so we can't * risk encountering a new kprobe in an interrupt handler. */ -void __kprobes kprobe_handler(struct pt_regs *regs) +static void __kprobes kprobe_handler(struct pt_regs *regs) { struct kprobe *p, *cur; struct kprobe_ctlblk *kcb; diff --git a/arch/arm/probes/kprobes/opt-arm.c b/arch/arm/probes/kprobes/opt-arm.c index dbef34ed933f..7f65048380ca 100644 --- a/arch/arm/probes/kprobes/opt-arm.c +++ b/arch/arm/probes/kprobes/opt-arm.c @@ -145,8 +145,6 @@ __arch_remove_optimized_kprobe(struct optimized_kprobe *op, int dirty) } } -extern void kprobe_handler(struct pt_regs *regs); - static void optimized_callback(struct optimized_kprobe *op, struct pt_regs *regs) { diff --git a/arch/arm/probes/kprobes/test-core.c b/arch/arm/probes/kprobes/test-core.c index c562832b8627..171c7076b89f 100644 --- a/arch/arm/probes/kprobes/test-core.c +++ b/arch/arm/probes/kprobes/test-core.c @@ -720,7 +720,7 @@ static const char coverage_register_lookup[16] = { [REG_TYPE_NOSPPCX] = COVERAGE_ANY_REG | COVERAGE_SP, }; -unsigned coverage_start_registers(const struct decode_header *h) +static unsigned coverage_start_registers(const struct decode_header *h) { unsigned regs = 0; int i; diff --git a/arch/arm/probes/kprobes/test-core.h b/arch/arm/probes/kprobes/test-core.h index 56ad3c0aaeea..c7297037c162 100644 --- a/arch/arm/probes/kprobes/test-core.h +++ b/arch/arm/probes/kprobes/test-core.h @@ -454,3 +454,7 @@ void kprobe_thumb32_test_cases(void); #else void kprobe_arm_test_cases(void); #endif + +void __kprobes_test_case_start(void); +void __kprobes_test_case_end_16(void); +void __kprobes_test_case_end_32(void); diff --git a/arch/arm/tools/mach-types b/arch/arm/tools/mach-types index 9e74c7ff6b04..97e2bfa01f4b 100644 --- a/arch/arm/tools/mach-types +++ b/arch/arm/tools/mach-types @@ -7,7 +7,7 @@ # http://www.arm.linux.org.uk/developer/machines/download.php # # Please do not send patches to this file; it is automatically generated! -# To add an entry into this database, please see Documentation/arm/arm.rst, +# To add an entry into this database, please see Documentation/arch/arm/arm.rst, # or visit: # # http://www.arm.linux.org.uk/developer/machines/?action=new diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c index 1976c6f325a4..a003beacac76 100644 --- a/arch/arm/vdso/vgettimeofday.c +++ b/arch/arm/vdso/vgettimeofday.c @@ -6,6 +6,8 @@ */ #include #include +#include +#include int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c index 349dcb944a93..1ba5078c1025 100644 --- a/arch/arm/vfp/vfpmodule.c +++ b/arch/arm/vfp/vfpmodule.c @@ -25,6 +25,7 @@ #include #include #include +#include #include "vfpinstr.h" #include "vfp.h" diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 44911bce5389..a8d0bd4136db 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -207,6 +207,7 @@ config ARM64 select HAVE_IOREMAP_PROT select HAVE_IRQ_TIME_ACCOUNTING select HAVE_KVM + select HAVE_MOD_ARCH_SPECIFIC select HAVE_NMI select HAVE_PERF_EVENTS select HAVE_PERF_REGS @@ -578,7 +579,6 @@ config ARM64_ERRATUM_845719 config ARM64_ERRATUM_843419 bool "Cortex-A53: 843419: A load or store might access an incorrect address" default y - select ARM64_MODULE_PLTS if MODULES help This option links the kernel with '--fix-cortex-a53-843419' and enables PLT support to replace certain ADRP instructions, which can @@ -1620,7 +1620,7 @@ config KUSER_HELPERS the system. This permits binaries to be run on ARMv4 through to ARMv8 without modification. - See Documentation/arm/kernel_user_helpers.rst for details. + See Documentation/arch/arm/kernel_user_helpers.rst for details. However, the fixed address nature of these helpers can be used by ROP (return orientated programming) authors when creating @@ -2108,26 +2108,6 @@ config ARM64_SME register state capable of holding two dimensional matrix tiles to enable various matrix operations. -config ARM64_MODULE_PLTS - bool "Use PLTs to allow module memory to spill over into vmalloc area" - depends on MODULES - select HAVE_MOD_ARCH_SPECIFIC - help - Allocate PLTs when loading modules so that jumps and calls whose - targets are too far away for their relative offsets to be encoded - in the instructions themselves can be bounced via veneers in the - module's PLT. This allows modules to be allocated in the generic - vmalloc area after the dedicated module memory area has been - exhausted. - - When running with address space randomization (KASLR), the module - region itself may be too far away for ordinary relative jumps and - calls, and so in that case, module PLTs are required and cannot be - disabled. - - Specific errata workaround(s) might also force module PLTs to be - enabled (ARM64_ERRATUM_843419). - config ARM64_PSEUDO_NMI bool "Support for NMI-like interrupts" select ARM_GIC_V3 @@ -2168,7 +2148,6 @@ config RELOCATABLE config RANDOMIZE_BASE bool "Randomize the address of the kernel image" - select ARM64_MODULE_PLTS if MODULES select RELOCATABLE help Randomizes the virtual address at which the kernel image is @@ -2199,9 +2178,8 @@ config RANDOMIZE_MODULE_REGION_FULL When this option is not set, the module region will be randomized over a limited range that contains the [_stext, _etext] interval of the core kernel, so branch relocations are almost always in range unless - ARM64_MODULE_PLTS is enabled and the region is exhausted. In this - particular case of region exhaustion, modules might be able to fall - back to a larger 2GB area. + the region is exhausted. In this particular case of region + exhaustion, modules might be able to fall back to a larger 2GB area. config CC_HAVE_STACKPROTECTOR_SYSREG def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0) diff --git a/arch/arm64/include/asm/alternative-macros.h b/arch/arm64/include/asm/alternative-macros.h index bdf1f6bcd010..94b486192e1f 100644 --- a/arch/arm64/include/asm/alternative-macros.h +++ b/arch/arm64/include/asm/alternative-macros.h @@ -23,17 +23,17 @@ #include -#define ALTINSTR_ENTRY(feature) \ +#define ALTINSTR_ENTRY(cpucap) \ " .word 661b - .\n" /* label */ \ " .word 663f - .\n" /* new instruction */ \ - " .hword " __stringify(feature) "\n" /* feature bit */ \ + " .hword " __stringify(cpucap) "\n" /* cpucap */ \ " .byte 662b-661b\n" /* source len */ \ " .byte 664f-663f\n" /* replacement len */ -#define ALTINSTR_ENTRY_CB(feature, cb) \ +#define ALTINSTR_ENTRY_CB(cpucap, cb) \ " .word 661b - .\n" /* label */ \ - " .word " __stringify(cb) "- .\n" /* callback */ \ - " .hword " __stringify(feature) "\n" /* feature bit */ \ + " .word " __stringify(cb) "- .\n" /* callback */ \ + " .hword " __stringify(cpucap) "\n" /* cpucap */ \ " .byte 662b-661b\n" /* source len */ \ " .byte 664f-663f\n" /* replacement len */ @@ -53,13 +53,13 @@ * * Alternatives with callbacks do not generate replacement instructions. */ -#define __ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg_enabled) \ +#define __ALTERNATIVE_CFG(oldinstr, newinstr, cpucap, cfg_enabled) \ ".if "__stringify(cfg_enabled)" == 1\n" \ "661:\n\t" \ oldinstr "\n" \ "662:\n" \ ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(feature) \ + ALTINSTR_ENTRY(cpucap) \ ".popsection\n" \ ".subsection 1\n" \ "663:\n\t" \ @@ -70,31 +70,31 @@ ".previous\n" \ ".endif\n" -#define __ALTERNATIVE_CFG_CB(oldinstr, feature, cfg_enabled, cb) \ +#define __ALTERNATIVE_CFG_CB(oldinstr, cpucap, cfg_enabled, cb) \ ".if "__stringify(cfg_enabled)" == 1\n" \ "661:\n\t" \ oldinstr "\n" \ "662:\n" \ ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY_CB(feature, cb) \ + ALTINSTR_ENTRY_CB(cpucap, cb) \ ".popsection\n" \ "663:\n\t" \ "664:\n\t" \ ".endif\n" -#define _ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg, ...) \ - __ALTERNATIVE_CFG(oldinstr, newinstr, feature, IS_ENABLED(cfg)) +#define _ALTERNATIVE_CFG(oldinstr, newinstr, cpucap, cfg, ...) \ + __ALTERNATIVE_CFG(oldinstr, newinstr, cpucap, IS_ENABLED(cfg)) -#define ALTERNATIVE_CB(oldinstr, feature, cb) \ - __ALTERNATIVE_CFG_CB(oldinstr, (1 << ARM64_CB_SHIFT) | (feature), 1, cb) +#define ALTERNATIVE_CB(oldinstr, cpucap, cb) \ + __ALTERNATIVE_CFG_CB(oldinstr, (1 << ARM64_CB_SHIFT) | (cpucap), 1, cb) #else #include -.macro altinstruction_entry orig_offset alt_offset feature orig_len alt_len +.macro altinstruction_entry orig_offset alt_offset cpucap orig_len alt_len .word \orig_offset - . .word \alt_offset - . - .hword (\feature) + .hword (\cpucap) .byte \orig_len .byte \alt_len .endm @@ -210,9 +210,9 @@ alternative_endif #endif /* __ASSEMBLY__ */ /* - * Usage: asm(ALTERNATIVE(oldinstr, newinstr, feature)); + * Usage: asm(ALTERNATIVE(oldinstr, newinstr, cpucap)); * - * Usage: asm(ALTERNATIVE(oldinstr, newinstr, feature, CONFIG_FOO)); + * Usage: asm(ALTERNATIVE(oldinstr, newinstr, cpucap, CONFIG_FOO)); * N.B. If CONFIG_FOO is specified, but not selected, the whole block * will be omitted, including oldinstr. */ @@ -224,15 +224,15 @@ alternative_endif #include static __always_inline bool -alternative_has_feature_likely(const unsigned long feature) +alternative_has_cap_likely(const unsigned long cpucap) { - compiletime_assert(feature < ARM64_NCAPS, - "feature must be < ARM64_NCAPS"); + compiletime_assert(cpucap < ARM64_NCAPS, + "cpucap must be < ARM64_NCAPS"); asm_volatile_goto( - ALTERNATIVE_CB("b %l[l_no]", %[feature], alt_cb_patch_nops) + ALTERNATIVE_CB("b %l[l_no]", %[cpucap], alt_cb_patch_nops) : - : [feature] "i" (feature) + : [cpucap] "i" (cpucap) : : l_no); @@ -242,15 +242,15 @@ l_no: } static __always_inline bool -alternative_has_feature_unlikely(const unsigned long feature) +alternative_has_cap_unlikely(const unsigned long cpucap) { - compiletime_assert(feature < ARM64_NCAPS, - "feature must be < ARM64_NCAPS"); + compiletime_assert(cpucap < ARM64_NCAPS, + "cpucap must be < ARM64_NCAPS"); asm_volatile_goto( - ALTERNATIVE("nop", "b %l[l_yes]", %[feature]) + ALTERNATIVE("nop", "b %l[l_yes]", %[cpucap]) : - : [feature] "i" (feature) + : [cpucap] "i" (cpucap) : : l_yes); diff --git a/arch/arm64/include/asm/alternative.h b/arch/arm64/include/asm/alternative.h index a38b92e11811..00d97b8a757f 100644 --- a/arch/arm64/include/asm/alternative.h +++ b/arch/arm64/include/asm/alternative.h @@ -13,7 +13,7 @@ struct alt_instr { s32 orig_offset; /* offset to original instruction */ s32 alt_offset; /* offset to replacement instruction */ - u16 cpufeature; /* cpufeature bit set for replacement */ + u16 cpucap; /* cpucap bit set for replacement */ u8 orig_len; /* size of original instruction(s) */ u8 alt_len; /* size of new instruction(s), <= orig_len */ }; @@ -23,7 +23,7 @@ typedef void (*alternative_cb_t)(struct alt_instr *alt, void __init apply_boot_alternatives(void); void __init apply_alternatives_all(void); -bool alternative_is_applied(u16 cpufeature); +bool alternative_is_applied(u16 cpucap); #ifdef CONFIG_MODULES void apply_alternatives_module(void *start, size_t length); @@ -31,5 +31,8 @@ void apply_alternatives_module(void *start, size_t length); static inline void apply_alternatives_module(void *start, size_t length) { } #endif +void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, + __le32 *updptr, int nr_inst); + #endif /* __ASSEMBLY__ */ #endif /* __ASM_ALTERNATIVE_H */ diff --git a/arch/arm64/include/asm/archrandom.h b/arch/arm64/include/asm/archrandom.h index 2f5f3da34782..b0abc64f86b0 100644 --- a/arch/arm64/include/asm/archrandom.h +++ b/arch/arm64/include/asm/archrandom.h @@ -129,4 +129,6 @@ static inline bool __init __early_cpu_has_rndr(void) return (ftr >> ID_AA64ISAR0_EL1_RNDR_SHIFT) & 0xf; } +u64 kaslr_early_init(void *fdt); + #endif /* _ASM_ARCHRANDOM_H */ diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/asm-uaccess.h index 75b211c98dea..5b6efe8abeeb 100644 --- a/arch/arm64/include/asm/asm-uaccess.h +++ b/arch/arm64/include/asm/asm-uaccess.h @@ -18,7 +18,6 @@ bic \tmp1, \tmp1, #TTBR_ASID_MASK sub \tmp1, \tmp1, #RESERVED_SWAPPER_OFFSET // reserved_pg_dir msr ttbr0_el1, \tmp1 // set reserved TTBR0_EL1 - isb add \tmp1, \tmp1, #RESERVED_SWAPPER_OFFSET msr ttbr1_el1, \tmp1 // set reserved ASID isb @@ -31,7 +30,6 @@ extr \tmp2, \tmp2, \tmp1, #48 ror \tmp2, \tmp2, #16 msr ttbr1_el1, \tmp2 // set the active ASID - isb msr ttbr0_el1, \tmp1 // set the non-PAN TTBR0_EL1 isb .endm diff --git a/arch/arm64/include/asm/compat.h b/arch/arm64/include/asm/compat.h index 74575c3d6987..ae904a1ad529 100644 --- a/arch/arm64/include/asm/compat.h +++ b/arch/arm64/include/asm/compat.h @@ -96,6 +96,8 @@ static inline int is_compat_thread(struct thread_info *thread) return test_ti_thread_flag(thread, TIF_32BIT); } +long compat_arm_syscall(struct pt_regs *regs, int scno); + #else /* !CONFIG_COMPAT */ static inline int is_compat_thread(struct thread_info *thread) diff --git a/arch/arm64/include/asm/cpu.h b/arch/arm64/include/asm/cpu.h index fd7a92219eea..e749838b9c5d 100644 --- a/arch/arm64/include/asm/cpu.h +++ b/arch/arm64/include/asm/cpu.h @@ -56,6 +56,7 @@ struct cpuinfo_arm64 { u64 reg_id_aa64mmfr0; u64 reg_id_aa64mmfr1; u64 reg_id_aa64mmfr2; + u64 reg_id_aa64mmfr3; u64 reg_id_aa64pfr0; u64 reg_id_aa64pfr1; u64 reg_id_aa64zfr0; diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 6bf013fb110d..7a95c324e52a 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -107,7 +107,7 @@ extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0; * CPU capabilities: * * We use arm64_cpu_capabilities to represent system features, errata work - * arounds (both used internally by kernel and tracked in cpu_hwcaps) and + * arounds (both used internally by kernel and tracked in system_cpucaps) and * ELF HWCAPs (which are exposed to user). * * To support systems with heterogeneous CPUs, we need to make sure that we @@ -419,12 +419,12 @@ static __always_inline bool is_hyp_code(void) return is_vhe_hyp_code() || is_nvhe_hyp_code(); } -extern DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); +extern DECLARE_BITMAP(system_cpucaps, ARM64_NCAPS); -extern DECLARE_BITMAP(boot_capabilities, ARM64_NCAPS); +extern DECLARE_BITMAP(boot_cpucaps, ARM64_NCAPS); #define for_each_available_cap(cap) \ - for_each_set_bit(cap, cpu_hwcaps, ARM64_NCAPS) + for_each_set_bit(cap, system_cpucaps, ARM64_NCAPS) bool this_cpu_has_cap(unsigned int cap); void cpu_set_feature(unsigned int num); @@ -437,7 +437,7 @@ unsigned long cpu_get_elf_hwcap2(void); static __always_inline bool system_capabilities_finalized(void) { - return alternative_has_feature_likely(ARM64_ALWAYS_SYSTEM); + return alternative_has_cap_likely(ARM64_ALWAYS_SYSTEM); } /* @@ -449,7 +449,7 @@ static __always_inline bool cpus_have_cap(unsigned int num) { if (num >= ARM64_NCAPS) return false; - return arch_test_bit(num, cpu_hwcaps); + return arch_test_bit(num, system_cpucaps); } /* @@ -464,7 +464,7 @@ static __always_inline bool __cpus_have_const_cap(int num) { if (num >= ARM64_NCAPS) return false; - return alternative_has_feature_unlikely(num); + return alternative_has_cap_unlikely(num); } /* @@ -504,16 +504,6 @@ static __always_inline bool cpus_have_const_cap(int num) return cpus_have_cap(num); } -static inline void cpus_set_cap(unsigned int num) -{ - if (num >= ARM64_NCAPS) { - pr_warn("Attempt to set an illegal CPU capability (%d >= %d)\n", - num, ARM64_NCAPS); - } else { - __set_bit(num, cpu_hwcaps); - } -} - static inline int __attribute_const__ cpuid_feature_extract_signed_field_width(u64 features, int field, int width) { diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index f86b157a5da3..ef46f2daca62 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -166,4 +166,6 @@ static inline void efi_capsule_flush_cache_range(void *addr, int size) dcache_clean_inval_poc((unsigned long)addr, (unsigned long)addr + size); } +efi_status_t efi_handle_corrupted_x18(efi_status_t s, const char *f); + #endif /* _ASM_EFI_H */ diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h index 037724b19c5c..f4c3d30bf746 100644 --- a/arch/arm64/include/asm/el2_setup.h +++ b/arch/arm64/include/asm/el2_setup.h @@ -22,6 +22,15 @@ isb .endm +.macro __init_el2_hcrx + mrs x0, id_aa64mmfr1_el1 + ubfx x0, x0, #ID_AA64MMFR1_EL1_HCX_SHIFT, #4 + cbz x0, .Lskip_hcrx_\@ + mov_q x0, HCRX_HOST_FLAGS + msr_s SYS_HCRX_EL2, x0 +.Lskip_hcrx_\@: +.endm + /* * Allow Non-secure EL1 and EL0 to access physical timer and counter. * This is not necessary for VHE, since the host kernel runs in EL2, @@ -69,7 +78,7 @@ cbz x0, .Lskip_trace_\@ // Skip if TraceBuffer is not present mrs_s x0, SYS_TRBIDR_EL1 - and x0, x0, TRBIDR_PROG + and x0, x0, TRBIDR_EL1_P cbnz x0, .Lskip_trace_\@ // If TRBE is available at EL2 mov x0, #(MDCR_EL2_E2TB_MASK << MDCR_EL2_E2TB_SHIFT) @@ -150,12 +159,21 @@ mov x0, xzr mrs x1, id_aa64pfr1_el1 ubfx x1, x1, #ID_AA64PFR1_EL1_SME_SHIFT, #4 - cbz x1, .Lset_fgt_\@ + cbz x1, .Lset_pie_fgt_\@ /* Disable nVHE traps of TPIDR2 and SMPRI */ orr x0, x0, #HFGxTR_EL2_nSMPRI_EL1_MASK orr x0, x0, #HFGxTR_EL2_nTPIDR2_EL0_MASK +.Lset_pie_fgt_\@: + mrs_s x1, SYS_ID_AA64MMFR3_EL1 + ubfx x1, x1, #ID_AA64MMFR3_EL1_S1PIE_SHIFT, #4 + cbz x1, .Lset_fgt_\@ + + /* Disable trapping of PIR_EL1 / PIRE0_EL1 */ + orr x0, x0, #HFGxTR_EL2_nPIR_EL1 + orr x0, x0, #HFGxTR_EL2_nPIRE0_EL1 + .Lset_fgt_\@: msr_s SYS_HFGRTR_EL2, x0 msr_s SYS_HFGWTR_EL2, x0 @@ -184,6 +202,7 @@ */ .macro init_el2_state __init_el2_sctlr + __init_el2_hcrx __init_el2_timers __init_el2_debug __init_el2_lor @@ -284,14 +303,6 @@ cbz x1, .Lskip_sme_\@ msr_s SYS_SMPRIMAP_EL2, xzr // Make all priorities equal - - mrs x1, id_aa64mmfr1_el1 // HCRX_EL2 present? - ubfx x1, x1, #ID_AA64MMFR1_EL1_HCX_SHIFT, #4 - cbz x1, .Lskip_sme_\@ - - mrs_s x1, SYS_HCRX_EL2 - orr x1, x1, #HCRX_EL2_SMPME_MASK // Enable priority mapping - msr_s SYS_HCRX_EL2, x1 .Lskip_sme_\@: .endm diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 8487aec9b658..ae35939f395b 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -47,7 +47,7 @@ #define ESR_ELx_EC_DABT_LOW (0x24) #define ESR_ELx_EC_DABT_CUR (0x25) #define ESR_ELx_EC_SP_ALIGN (0x26) -/* Unallocated EC: 0x27 */ +#define ESR_ELx_EC_MOPS (0x27) #define ESR_ELx_EC_FP_EXC32 (0x28) /* Unallocated EC: 0x29 - 0x2B */ #define ESR_ELx_EC_FP_EXC64 (0x2C) @@ -75,8 +75,11 @@ #define ESR_ELx_IL_SHIFT (25) #define ESR_ELx_IL (UL(1) << ESR_ELx_IL_SHIFT) -#define ESR_ELx_ISS_MASK (ESR_ELx_IL - 1) +#define ESR_ELx_ISS_MASK (GENMASK(24, 0)) #define ESR_ELx_ISS(esr) ((esr) & ESR_ELx_ISS_MASK) +#define ESR_ELx_ISS2_SHIFT (32) +#define ESR_ELx_ISS2_MASK (GENMASK_ULL(55, 32)) +#define ESR_ELx_ISS2(esr) (((esr) & ESR_ELx_ISS2_MASK) >> ESR_ELx_ISS2_SHIFT) /* ISS field definitions shared by different classes */ #define ESR_ELx_WNR_SHIFT (6) @@ -140,6 +143,20 @@ #define ESR_ELx_CM_SHIFT (8) #define ESR_ELx_CM (UL(1) << ESR_ELx_CM_SHIFT) +/* ISS2 field definitions for Data Aborts */ +#define ESR_ELx_TnD_SHIFT (10) +#define ESR_ELx_TnD (UL(1) << ESR_ELx_TnD_SHIFT) +#define ESR_ELx_TagAccess_SHIFT (9) +#define ESR_ELx_TagAccess (UL(1) << ESR_ELx_TagAccess_SHIFT) +#define ESR_ELx_GCS_SHIFT (8) +#define ESR_ELx_GCS (UL(1) << ESR_ELx_GCS_SHIFT) +#define ESR_ELx_Overlay_SHIFT (6) +#define ESR_ELx_Overlay (UL(1) << ESR_ELx_Overlay_SHIFT) +#define ESR_ELx_DirtyBit_SHIFT (5) +#define ESR_ELx_DirtyBit (UL(1) << ESR_ELx_DirtyBit_SHIFT) +#define ESR_ELx_Xs_SHIFT (0) +#define ESR_ELx_Xs_MASK (GENMASK_ULL(4, 0)) + /* ISS field definitions for exceptions taken in to Hyp */ #define ESR_ELx_CV (UL(1) << 24) #define ESR_ELx_COND_SHIFT (20) @@ -356,6 +373,15 @@ #define ESR_ELx_SME_ISS_ZA_DISABLED 3 #define ESR_ELx_SME_ISS_ZT_DISABLED 4 +/* ISS field definitions for MOPS exceptions */ +#define ESR_ELx_MOPS_ISS_MEM_INST (UL(1) << 24) +#define ESR_ELx_MOPS_ISS_FROM_EPILOGUE (UL(1) << 18) +#define ESR_ELx_MOPS_ISS_WRONG_OPTION (UL(1) << 17) +#define ESR_ELx_MOPS_ISS_OPTION_A (UL(1) << 16) +#define ESR_ELx_MOPS_ISS_DESTREG(esr) (((esr) & (UL(0x1f) << 10)) >> 10) +#define ESR_ELx_MOPS_ISS_SRCREG(esr) (((esr) & (UL(0x1f) << 5)) >> 5) +#define ESR_ELx_MOPS_ISS_SIZEREG(esr) (((esr) & (UL(0x1f) << 0)) >> 0) + #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/include/asm/exception.h b/arch/arm64/include/asm/exception.h index e73af709cb7a..ad688e157c9b 100644 --- a/arch/arm64/include/asm/exception.h +++ b/arch/arm64/include/asm/exception.h @@ -8,16 +8,11 @@ #define __ASM_EXCEPTION_H #include -#include #include #include -#ifdef CONFIG_FUNCTION_GRAPH_TRACER #define __exception_irq_entry __irq_entry -#else -#define __exception_irq_entry __kprobes -#endif static inline unsigned long disr_to_esr(u64 disr) { @@ -77,6 +72,7 @@ void do_el0_svc(struct pt_regs *regs); void do_el0_svc_compat(struct pt_regs *regs); void do_el0_fpac(struct pt_regs *regs, unsigned long esr); void do_el1_fpac(struct pt_regs *regs, unsigned long esr); +void do_el0_mops(struct pt_regs *regs, unsigned long esr); void do_serror(struct pt_regs *regs, unsigned long esr); void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags); diff --git a/arch/arm64/include/asm/hw_breakpoint.h b/arch/arm64/include/asm/hw_breakpoint.h index fa4c6ff3aa9b..84055329cd8b 100644 --- a/arch/arm64/include/asm/hw_breakpoint.h +++ b/arch/arm64/include/asm/hw_breakpoint.h @@ -154,4 +154,12 @@ static inline int get_num_wrps(void) ID_AA64DFR0_EL1_WRPs_SHIFT); } +#ifdef CONFIG_CPU_PM +extern void cpu_suspend_set_dbg_restorer(int (*hw_bp_restore)(unsigned int)); +#else +static inline void cpu_suspend_set_dbg_restorer(int (*hw_bp_restore)(unsigned int)) +{ +} +#endif + #endif /* __ASM_BREAKPOINT_H */ diff --git a/arch/arm64/include/asm/hwcap.h b/arch/arm64/include/asm/hwcap.h index 5d45f19fda7f..692b1ec663b2 100644 --- a/arch/arm64/include/asm/hwcap.h +++ b/arch/arm64/include/asm/hwcap.h @@ -137,6 +137,7 @@ #define KERNEL_HWCAP_SME_BI32I32 __khwcap2_feature(SME_BI32I32) #define KERNEL_HWCAP_SME_B16B16 __khwcap2_feature(SME_B16B16) #define KERNEL_HWCAP_SME_F16F16 __khwcap2_feature(SME_F16F16) +#define KERNEL_HWCAP_MOPS __khwcap2_feature(MOPS) /* * This yields a mask that user programs can use to figure out what diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h index e0f5f6b73edd..1f31ec146d16 100644 --- a/arch/arm64/include/asm/irqflags.h +++ b/arch/arm64/include/asm/irqflags.h @@ -24,7 +24,7 @@ static __always_inline bool __irqflags_uses_pmr(void) { return IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && - alternative_has_feature_unlikely(ARM64_HAS_GIC_PRIO_MASKING); + alternative_has_cap_unlikely(ARM64_HAS_GIC_PRIO_MASKING); } static __always_inline void __daif_local_irq_enable(void) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 186dd7f85b14..577773870b66 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -107,14 +107,14 @@ /* * Initial memory map attributes. */ -#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) +#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED | PTE_UXN) +#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S | PTE_UXN) #ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS) +#define SWAPPER_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS | PTE_WRITE) #define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PMD_SECT_RDONLY) #else -#define SWAPPER_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS) +#define SWAPPER_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) #define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PTE_RDONLY) #endif diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index baef29fcbeee..c6e12e8f2751 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -9,6 +9,7 @@ #include #include +#include #include /* Hyp Configuration Register (HCR) bits */ @@ -92,6 +93,9 @@ #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC) #define HCR_HOST_VHE_FLAGS (HCR_RW | HCR_TGE | HCR_E2H) +#define HCRX_GUEST_FLAGS (HCRX_EL2_SMPME | HCRX_EL2_TCR2En) +#define HCRX_HOST_FLAGS (HCRX_EL2_MSCEn | HCRX_EL2_TCR2En) + /* TCR_EL2 Registers bits */ #define TCR_EL2_RES1 ((1U << 31) | (1 << 23)) #define TCR_EL2_TBI (1 << 20) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 43c3bc0f9544..86042afa86c3 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -267,6 +267,24 @@ extern u64 __kvm_get_mdcr_el2(void); __kvm_at_err; \ } ) +void __noreturn hyp_panic(void); +asmlinkage void kvm_unexpected_el2_exception(void); +asmlinkage void __noreturn hyp_panic(void); +asmlinkage void __noreturn hyp_panic_bad_stack(void); +asmlinkage void kvm_unexpected_el2_exception(void); +struct kvm_cpu_context; +void handle_trap(struct kvm_cpu_context *host_ctxt); +asmlinkage void __noreturn kvm_host_psci_cpu_entry(bool is_cpu_on); +void __noreturn __pkvm_init_finalise(void); +void kvm_nvhe_prepare_backtrace(unsigned long fp, unsigned long pc); +void kvm_patch_vector_branch(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst); +void kvm_get_kimage_voffset(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst); +void kvm_compute_final_ctr_el0(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst); +void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, + u64 elr_phys, u64 par, uintptr_t vcpu, u64 far, u64 hpfar); #else /* __ASSEMBLY__ */ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 9787503ff43f..d48609d95423 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -279,6 +279,7 @@ enum vcpu_sysreg { TTBR0_EL1, /* Translation Table Base Register 0 */ TTBR1_EL1, /* Translation Table Base Register 1 */ TCR_EL1, /* Translation Control Register */ + TCR2_EL1, /* Extended Translation Control Register */ ESR_EL1, /* Exception Syndrome Register */ AFSR0_EL1, /* Auxiliary Fault Status Register 0 */ AFSR1_EL1, /* Auxiliary Fault Status Register 1 */ @@ -339,6 +340,10 @@ enum vcpu_sysreg { TFSR_EL1, /* Tag Fault Status Register (EL1) */ TFSRE0_EL1, /* Tag Fault Status Register (EL0) */ + /* Permission Indirection Extension registers */ + PIR_EL1, /* Permission Indirection Register 1 (EL1) */ + PIRE0_EL1, /* Permission Indirection Register 0 (EL1) */ + /* 32bit specific registers. */ DACR32_EL2, /* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */ @@ -1033,7 +1038,7 @@ void kvm_arm_clear_debug(struct kvm_vcpu *vcpu); void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu); #define kvm_vcpu_os_lock_enabled(vcpu) \ - (!!(__vcpu_sys_reg(vcpu, OSLSR_EL1) & SYS_OSLSR_OSLK)) + (!!(__vcpu_sys_reg(vcpu, OSLSR_EL1) & OSLSR_EL1_OSLK)) int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr); diff --git a/arch/arm64/include/asm/lse.h b/arch/arm64/include/asm/lse.h index f99d74826a7e..cbbcdc35c4cd 100644 --- a/arch/arm64/include/asm/lse.h +++ b/arch/arm64/include/asm/lse.h @@ -18,7 +18,7 @@ static __always_inline bool system_uses_lse_atomics(void) { - return alternative_has_feature_likely(ARM64_HAS_LSE_ATOMICS); + return alternative_has_cap_likely(ARM64_HAS_LSE_ATOMICS); } #define __lse_ll_sc_body(op, ...) \ diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index c735afdf639b..6e0e5722f229 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -46,7 +46,7 @@ #define KIMAGE_VADDR (MODULES_END) #define MODULES_END (MODULES_VADDR + MODULES_VSIZE) #define MODULES_VADDR (_PAGE_END(VA_BITS_MIN)) -#define MODULES_VSIZE (SZ_128M) +#define MODULES_VSIZE (SZ_2G) #define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) #define PCI_IO_END (VMEMMAP_START - SZ_8M) @@ -204,15 +204,17 @@ static inline unsigned long kaslr_offset(void) return kimage_vaddr - KIMAGE_VADDR; } +#ifdef CONFIG_RANDOMIZE_BASE +void kaslr_init(void); static inline bool kaslr_enabled(void) { - /* - * The KASLR offset modulo MIN_KIMG_ALIGN is taken from the physical - * placement of the image rather than from the seed, so a displacement - * of less than MIN_KIMG_ALIGN means that no seed was provided. - */ - return kaslr_offset() >= MIN_KIMG_ALIGN; + extern bool __kaslr_is_enabled; + return __kaslr_is_enabled; } +#else +static inline void kaslr_init(void) { } +static inline bool kaslr_enabled(void) { return false; } +#endif /* * Allow all memory at the discovery stage. We will clip it later. diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 56911691bef0..a6fb325424e7 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -39,11 +39,16 @@ static inline void contextidr_thread_switch(struct task_struct *next) /* * Set TTBR0 to reserved_pg_dir. No translations will be possible via TTBR0. */ -static inline void cpu_set_reserved_ttbr0(void) +static inline void cpu_set_reserved_ttbr0_nosync(void) { unsigned long ttbr = phys_to_ttbr(__pa_symbol(reserved_pg_dir)); write_sysreg(ttbr, ttbr0_el1); +} + +static inline void cpu_set_reserved_ttbr0(void) +{ + cpu_set_reserved_ttbr0_nosync(); isb(); } @@ -52,7 +57,6 @@ void cpu_do_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm); static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) { BUG_ON(pgd == swapper_pg_dir); - cpu_set_reserved_ttbr0(); cpu_do_switch_mm(virt_to_phys(pgd),mm); } @@ -164,7 +168,7 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) * up (i.e. cpufeature framework is not up yet) and * latter only when we enable CNP via cpufeature's * enable() callback. - * Also we rely on the cpu_hwcap bit being set before + * Also we rely on the system_cpucaps bit being set before * calling the enable() function. */ ttbr1 |= TTBR_CNP_BIT; diff --git a/arch/arm64/include/asm/module.h b/arch/arm64/include/asm/module.h index 18734fed3bdd..bfa6638b4c93 100644 --- a/arch/arm64/include/asm/module.h +++ b/arch/arm64/include/asm/module.h @@ -7,7 +7,6 @@ #include -#ifdef CONFIG_ARM64_MODULE_PLTS struct mod_plt_sec { int plt_shndx; int plt_num_entries; @@ -21,7 +20,6 @@ struct mod_arch_specific { /* for CONFIG_DYNAMIC_FTRACE */ struct plt_entry *ftrace_trampolines; }; -#endif u64 module_emit_plt_entry(struct module *mod, Elf64_Shdr *sechdrs, void *loc, const Elf64_Rela *rela, @@ -30,12 +28,6 @@ u64 module_emit_plt_entry(struct module *mod, Elf64_Shdr *sechdrs, u64 module_emit_veneer_for_adrp(struct module *mod, Elf64_Shdr *sechdrs, void *loc, u64 val); -#ifdef CONFIG_RANDOMIZE_BASE -extern u64 module_alloc_base; -#else -#define module_alloc_base ((u64)_etext - MODULES_VSIZE) -#endif - struct plt_entry { /* * A program that conforms to the AArch64 Procedure Call Standard diff --git a/arch/arm64/include/asm/module.lds.h b/arch/arm64/include/asm/module.lds.h index dbba4b7559aa..b9ae8349e35d 100644 --- a/arch/arm64/include/asm/module.lds.h +++ b/arch/arm64/include/asm/module.lds.h @@ -1,9 +1,7 @@ SECTIONS { -#ifdef CONFIG_ARM64_MODULE_PLTS .plt 0 : { BYTE(0) } .init.plt 0 : { BYTE(0) } .text.ftrace_trampoline 0 : { BYTE(0) } -#endif #ifdef CONFIG_KASAN_SW_TAGS /* diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index f658aafc47df..e4944d517c99 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -170,6 +170,14 @@ #define PTE_ATTRINDX(t) (_AT(pteval_t, (t)) << 2) #define PTE_ATTRINDX_MASK (_AT(pteval_t, 7) << 2) +/* + * PIIndex[3:0] encoding (Permission Indirection Extension) + */ +#define PTE_PI_IDX_0 6 /* AP[1], USER */ +#define PTE_PI_IDX_1 51 /* DBM */ +#define PTE_PI_IDX_2 53 /* PXN */ +#define PTE_PI_IDX_3 54 /* UXN */ + /* * Memory Attribute override for Stage-2 (MemAttr[3:0]) */ diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 9b165117a454..eed814b00a38 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -27,6 +27,40 @@ */ #define PMD_PRESENT_INVALID (_AT(pteval_t, 1) << 59) /* only when !PMD_SECT_VALID */ +#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) +#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) + +#define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG) +#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG) + +#define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE)) +#define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE)) +#define PROT_NORMAL_NC (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_NC)) +#define PROT_NORMAL (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL)) +#define PROT_NORMAL_TAGGED (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_TAGGED)) + +#define PROT_SECT_DEVICE_nGnRE (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_DEVICE_nGnRE)) +#define PROT_SECT_NORMAL (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PTE_WRITE | PMD_ATTRINDX(MT_NORMAL)) +#define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL)) + +#define _PAGE_DEFAULT (_PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL)) + +#define _PAGE_KERNEL (PROT_NORMAL) +#define _PAGE_KERNEL_RO ((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY) +#define _PAGE_KERNEL_ROX ((PROT_NORMAL & ~(PTE_WRITE | PTE_PXN)) | PTE_RDONLY) +#define _PAGE_KERNEL_EXEC (PROT_NORMAL & ~PTE_PXN) +#define _PAGE_KERNEL_EXEC_CONT ((PROT_NORMAL & ~PTE_PXN) | PTE_CONT) + +#define _PAGE_SHARED (_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE) +#define _PAGE_SHARED_EXEC (_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_WRITE) +#define _PAGE_READONLY (_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) +#define _PAGE_READONLY_EXEC (_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN) +#define _PAGE_EXECONLY (_PAGE_DEFAULT | PTE_RDONLY | PTE_NG | PTE_PXN) + +#ifdef __ASSEMBLY__ +#define PTE_MAYBE_NG 0 +#endif + #ifndef __ASSEMBLY__ #include @@ -34,9 +68,6 @@ extern bool arm64_use_ng_mappings; -#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) - #define PTE_MAYBE_NG (arm64_use_ng_mappings ? PTE_NG : 0) #define PMD_MAYBE_NG (arm64_use_ng_mappings ? PMD_SECT_NG : 0) @@ -50,26 +81,11 @@ extern bool arm64_use_ng_mappings; #define PTE_MAYBE_GP 0 #endif -#define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG) -#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG) - -#define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE)) -#define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE)) -#define PROT_NORMAL_NC (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_NC)) -#define PROT_NORMAL (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL)) -#define PROT_NORMAL_TAGGED (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_NORMAL_TAGGED)) - -#define PROT_SECT_DEVICE_nGnRE (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_DEVICE_nGnRE)) -#define PROT_SECT_NORMAL (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL)) -#define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL)) - -#define _PAGE_DEFAULT (_PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL)) - -#define PAGE_KERNEL __pgprot(PROT_NORMAL) -#define PAGE_KERNEL_RO __pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY) -#define PAGE_KERNEL_ROX __pgprot((PROT_NORMAL & ~(PTE_WRITE | PTE_PXN)) | PTE_RDONLY) -#define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN) -#define PAGE_KERNEL_EXEC_CONT __pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT) +#define PAGE_KERNEL __pgprot(_PAGE_KERNEL) +#define PAGE_KERNEL_RO __pgprot(_PAGE_KERNEL_RO) +#define PAGE_KERNEL_ROX __pgprot(_PAGE_KERNEL_ROX) +#define PAGE_KERNEL_EXEC __pgprot(_PAGE_KERNEL_EXEC) +#define PAGE_KERNEL_EXEC_CONT __pgprot(_PAGE_KERNEL_EXEC_CONT) #define PAGE_S2_MEMATTR(attr, has_fwb) \ ({ \ @@ -83,12 +99,62 @@ extern bool arm64_use_ng_mappings; #define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) /* shared+writable pages are clean by default, hence PTE_RDONLY|PTE_WRITE */ -#define PAGE_SHARED __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE) -#define PAGE_SHARED_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_WRITE) -#define PAGE_READONLY __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) -#define PAGE_READONLY_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN) -#define PAGE_EXECONLY __pgprot(_PAGE_DEFAULT | PTE_RDONLY | PTE_NG | PTE_PXN) +#define PAGE_SHARED __pgprot(_PAGE_SHARED) +#define PAGE_SHARED_EXEC __pgprot(_PAGE_SHARED_EXEC) +#define PAGE_READONLY __pgprot(_PAGE_READONLY) +#define PAGE_READONLY_EXEC __pgprot(_PAGE_READONLY_EXEC) +#define PAGE_EXECONLY __pgprot(_PAGE_EXECONLY) #endif /* __ASSEMBLY__ */ +#define pte_pi_index(pte) ( \ + ((pte & BIT(PTE_PI_IDX_3)) >> (PTE_PI_IDX_3 - 3)) | \ + ((pte & BIT(PTE_PI_IDX_2)) >> (PTE_PI_IDX_2 - 2)) | \ + ((pte & BIT(PTE_PI_IDX_1)) >> (PTE_PI_IDX_1 - 1)) | \ + ((pte & BIT(PTE_PI_IDX_0)) >> (PTE_PI_IDX_0 - 0))) + +/* + * Page types used via Permission Indirection Extension (PIE). PIE uses + * the USER, DBM, PXN and UXN bits to to generate an index which is used + * to look up the actual permission in PIR_ELx and PIRE0_EL1. We define + * combinations we use on non-PIE systems with the same encoding, for + * convenience these are listed here as comments as are the unallocated + * encodings. + */ + +/* 0: PAGE_DEFAULT */ +/* 1: PTE_USER */ +/* 2: PTE_WRITE */ +/* 3: PTE_WRITE | PTE_USER */ +/* 4: PAGE_EXECONLY PTE_PXN */ +/* 5: PAGE_READONLY_EXEC PTE_PXN | PTE_USER */ +/* 6: PTE_PXN | PTE_WRITE */ +/* 7: PAGE_SHARED_EXEC PTE_PXN | PTE_WRITE | PTE_USER */ +/* 8: PAGE_KERNEL_ROX PTE_UXN */ +/* 9: PTE_UXN | PTE_USER */ +/* a: PAGE_KERNEL_EXEC PTE_UXN | PTE_WRITE */ +/* b: PTE_UXN | PTE_WRITE | PTE_USER */ +/* c: PAGE_KERNEL_RO PTE_UXN | PTE_PXN */ +/* d: PAGE_READONLY PTE_UXN | PTE_PXN | PTE_USER */ +/* e: PAGE_KERNEL PTE_UXN | PTE_PXN | PTE_WRITE */ +/* f: PAGE_SHARED PTE_UXN | PTE_PXN | PTE_WRITE | PTE_USER */ + +#define PIE_E0 ( \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_EXECONLY), PIE_X_O) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_READONLY_EXEC), PIE_RX) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_SHARED_EXEC), PIE_RWX) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_READONLY), PIE_R) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_SHARED), PIE_RW)) + +#define PIE_E1 ( \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_EXECONLY), PIE_NONE_O) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_READONLY_EXEC), PIE_R) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_SHARED_EXEC), PIE_RW) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_READONLY), PIE_R) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_SHARED), PIE_RW) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_KERNEL_RO), PIE_R) | \ + PIRx_ELx_PERM(pte_pi_index(_PAGE_KERNEL), PIE_RW)) + #endif /* __ASM_PGTABLE_PROT_H */ diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 13df982a0808..3fdae5fe3142 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -73,6 +73,7 @@ static inline void dynamic_scs_init(void) {} #endif int scs_patch(const u8 eh_frame[], int size); +asmlinkage void scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/include/asm/spectre.h b/arch/arm64/include/asm/spectre.h index db7b371b367c..9cc501450486 100644 --- a/arch/arm64/include/asm/spectre.h +++ b/arch/arm64/include/asm/spectre.h @@ -100,5 +100,21 @@ bool is_spectre_bhb_affected(const struct arm64_cpu_capabilities *entry, int sco u8 spectre_bhb_loop_affected(int scope); void spectre_bhb_enable_mitigation(const struct arm64_cpu_capabilities *__unused); bool try_emulate_el1_ssbs(struct pt_regs *regs, u32 instr); + +void spectre_v4_patch_fw_mitigation_enable(struct alt_instr *alt, __le32 *origptr, + __le32 *updptr, int nr_inst); +void smccc_patch_fw_mitigation_conduit(struct alt_instr *alt, __le32 *origptr, + __le32 *updptr, int nr_inst); +void spectre_bhb_patch_loop_mitigation_enable(struct alt_instr *alt, __le32 *origptr, + __le32 *updptr, int nr_inst); +void spectre_bhb_patch_fw_mitigation_enabled(struct alt_instr *alt, __le32 *origptr, + __le32 *updptr, int nr_inst); +void spectre_bhb_patch_loop_iter(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst); +void spectre_bhb_patch_wa3(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst); +void spectre_bhb_patch_clearbhb(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst); + #endif /* __ASSEMBLY__ */ #endif /* __ASM_SPECTRE_H */ diff --git a/arch/arm64/include/asm/syscall_wrapper.h b/arch/arm64/include/asm/syscall_wrapper.h index d30217c21eff..17f687510c48 100644 --- a/arch/arm64/include/asm/syscall_wrapper.h +++ b/arch/arm64/include/asm/syscall_wrapper.h @@ -38,6 +38,7 @@ asmlinkage long __arm64_compat_sys_##sname(const struct pt_regs *__unused) #define COND_SYSCALL_COMPAT(name) \ + asmlinkage long __arm64_compat_sys_##name(const struct pt_regs *regs); \ asmlinkage long __weak __arm64_compat_sys_##name(const struct pt_regs *regs) \ { \ return sys_ni_syscall(); \ @@ -53,6 +54,7 @@ ALLOW_ERROR_INJECTION(__arm64_sys##name, ERRNO); \ static long __se_sys##name(__MAP(x,__SC_LONG,__VA_ARGS__)); \ static inline long __do_sys##name(__MAP(x,__SC_DECL,__VA_ARGS__)); \ + asmlinkage long __arm64_sys##name(const struct pt_regs *regs); \ asmlinkage long __arm64_sys##name(const struct pt_regs *regs) \ { \ return __se_sys##name(SC_ARM64_REGS_TO_ARGS(x,__VA_ARGS__)); \ @@ -73,11 +75,13 @@ asmlinkage long __arm64_sys_##sname(const struct pt_regs *__unused) #define COND_SYSCALL(name) \ + asmlinkage long __arm64_sys_##name(const struct pt_regs *regs); \ asmlinkage long __weak __arm64_sys_##name(const struct pt_regs *regs) \ { \ return sys_ni_syscall(); \ } +asmlinkage long __arm64_sys_ni_syscall(const struct pt_regs *__unused); #define SYS_NI(name) SYSCALL_ALIAS(__arm64_sys_##name, sys_ni_posix_timers); #endif /* __ASM_SYSCALL_WRAPPER_H */ diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index eefd712f2430..7a1e62631814 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -140,25 +140,17 @@ #define SYS_SVCR_SMSTART_SM_EL0 sys_reg(0, 3, 4, 3, 3) #define SYS_SVCR_SMSTOP_SMZA_EL0 sys_reg(0, 3, 4, 6, 3) -#define SYS_OSDTRRX_EL1 sys_reg(2, 0, 0, 0, 2) -#define SYS_MDCCINT_EL1 sys_reg(2, 0, 0, 2, 0) -#define SYS_MDSCR_EL1 sys_reg(2, 0, 0, 2, 2) -#define SYS_OSDTRTX_EL1 sys_reg(2, 0, 0, 3, 2) -#define SYS_OSECCR_EL1 sys_reg(2, 0, 0, 6, 2) #define SYS_DBGBVRn_EL1(n) sys_reg(2, 0, 0, n, 4) #define SYS_DBGBCRn_EL1(n) sys_reg(2, 0, 0, n, 5) #define SYS_DBGWVRn_EL1(n) sys_reg(2, 0, 0, n, 6) #define SYS_DBGWCRn_EL1(n) sys_reg(2, 0, 0, n, 7) #define SYS_MDRAR_EL1 sys_reg(2, 0, 1, 0, 0) -#define SYS_OSLAR_EL1 sys_reg(2, 0, 1, 0, 4) -#define SYS_OSLAR_OSLK BIT(0) - #define SYS_OSLSR_EL1 sys_reg(2, 0, 1, 1, 4) -#define SYS_OSLSR_OSLM_MASK (BIT(3) | BIT(0)) -#define SYS_OSLSR_OSLM_NI 0 -#define SYS_OSLSR_OSLM_IMPLEMENTED BIT(3) -#define SYS_OSLSR_OSLK BIT(1) +#define OSLSR_EL1_OSLM_MASK (BIT(3) | BIT(0)) +#define OSLSR_EL1_OSLM_NI 0 +#define OSLSR_EL1_OSLM_IMPLEMENTED BIT(3) +#define OSLSR_EL1_OSLK BIT(1) #define SYS_OSDLR_EL1 sys_reg(2, 0, 1, 3, 4) #define SYS_DBGPRCR_EL1 sys_reg(2, 0, 1, 4, 4) @@ -241,54 +233,8 @@ /*** End of Statistical Profiling Extension ***/ -/* - * TRBE Registers - */ -#define SYS_TRBLIMITR_EL1 sys_reg(3, 0, 9, 11, 0) -#define SYS_TRBPTR_EL1 sys_reg(3, 0, 9, 11, 1) -#define SYS_TRBBASER_EL1 sys_reg(3, 0, 9, 11, 2) -#define SYS_TRBSR_EL1 sys_reg(3, 0, 9, 11, 3) -#define SYS_TRBMAR_EL1 sys_reg(3, 0, 9, 11, 4) -#define SYS_TRBTRG_EL1 sys_reg(3, 0, 9, 11, 6) -#define SYS_TRBIDR_EL1 sys_reg(3, 0, 9, 11, 7) - -#define TRBLIMITR_LIMIT_MASK GENMASK_ULL(51, 0) -#define TRBLIMITR_LIMIT_SHIFT 12 -#define TRBLIMITR_NVM BIT(5) -#define TRBLIMITR_TRIG_MODE_MASK GENMASK(1, 0) -#define TRBLIMITR_TRIG_MODE_SHIFT 3 -#define TRBLIMITR_FILL_MODE_MASK GENMASK(1, 0) -#define TRBLIMITR_FILL_MODE_SHIFT 1 -#define TRBLIMITR_ENABLE BIT(0) -#define TRBPTR_PTR_MASK GENMASK_ULL(63, 0) -#define TRBPTR_PTR_SHIFT 0 -#define TRBBASER_BASE_MASK GENMASK_ULL(51, 0) -#define TRBBASER_BASE_SHIFT 12 -#define TRBSR_EC_MASK GENMASK(5, 0) -#define TRBSR_EC_SHIFT 26 -#define TRBSR_IRQ BIT(22) -#define TRBSR_TRG BIT(21) -#define TRBSR_WRAP BIT(20) -#define TRBSR_ABORT BIT(18) -#define TRBSR_STOP BIT(17) -#define TRBSR_MSS_MASK GENMASK(15, 0) -#define TRBSR_MSS_SHIFT 0 -#define TRBSR_BSC_MASK GENMASK(5, 0) -#define TRBSR_BSC_SHIFT 0 -#define TRBSR_FSC_MASK GENMASK(5, 0) -#define TRBSR_FSC_SHIFT 0 -#define TRBMAR_SHARE_MASK GENMASK(1, 0) -#define TRBMAR_SHARE_SHIFT 8 -#define TRBMAR_OUTER_MASK GENMASK(3, 0) -#define TRBMAR_OUTER_SHIFT 4 -#define TRBMAR_INNER_MASK GENMASK(3, 0) -#define TRBMAR_INNER_SHIFT 0 -#define TRBTRG_TRG_MASK GENMASK(31, 0) -#define TRBTRG_TRG_SHIFT 0 -#define TRBIDR_FLAG BIT(5) -#define TRBIDR_PROG BIT(4) -#define TRBIDR_ALIGN_MASK GENMASK(3, 0) -#define TRBIDR_ALIGN_SHIFT 0 +#define TRBSR_EL1_BSC_MASK GENMASK(5, 0) +#define TRBSR_EL1_BSC_SHIFT 0 #define SYS_PMINTENSET_EL1 sys_reg(3, 0, 9, 14, 1) #define SYS_PMINTENCLR_EL1 sys_reg(3, 0, 9, 14, 2) @@ -764,6 +710,25 @@ #define ICH_VTR_TDS_SHIFT 19 #define ICH_VTR_TDS_MASK (1 << ICH_VTR_TDS_SHIFT) +/* + * Permission Indirection Extension (PIE) permission encodings. + * Encodings with the _O suffix, have overlays applied (Permission Overlay Extension). + */ +#define PIE_NONE_O 0x0 +#define PIE_R_O 0x1 +#define PIE_X_O 0x2 +#define PIE_RX_O 0x3 +#define PIE_RW_O 0x5 +#define PIE_RWnX_O 0x6 +#define PIE_RWX_O 0x7 +#define PIE_R 0x8 +#define PIE_GCS 0x9 +#define PIE_RX 0xa +#define PIE_RW 0xc +#define PIE_RWX 0xe + +#define PIRx_ELx_PERM(idx, perm) ((perm) << ((idx) * 4)) + #define ARM64_FEATURE_FIELD_BITS 4 /* Defined for compatibility only, do not add new users. */ diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h index 1f361e2da516..d66dfb3a72dd 100644 --- a/arch/arm64/include/asm/traps.h +++ b/arch/arm64/include/asm/traps.h @@ -29,6 +29,8 @@ void arm64_force_sig_fault(int signo, int code, unsigned long far, const char *s void arm64_force_sig_mceerr(int code, unsigned long far, short lsb, const char *str); void arm64_force_sig_ptrace_errno_trap(int errno, unsigned long far, const char *str); +int early_brk64(unsigned long addr, unsigned long esr, struct pt_regs *regs); + /* * Move regs->pc to next instruction and do necessary setup before it * is executed. diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index 05f4fc265428..14be5000c5a0 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -65,7 +65,6 @@ static inline void __uaccess_ttbr0_disable(void) ttbr &= ~TTBR_ASID_MASK; /* reserved_pg_dir placed before swapper_pg_dir */ write_sysreg(ttbr - RESERVED_SWAPPER_OFFSET, ttbr0_el1); - isb(); /* Set reserved ASID */ write_sysreg(ttbr, ttbr1_el1); isb(); @@ -89,7 +88,6 @@ static inline void __uaccess_ttbr0_enable(void) ttbr1 &= ~TTBR_ASID_MASK; /* safety measure */ ttbr1 |= ttbr0 & TTBR_ASID_MASK; write_sysreg(ttbr1, ttbr1_el1); - isb(); /* Restore user page table */ write_sysreg(ttbr0, ttbr0_el1); diff --git a/arch/arm64/include/uapi/asm/hwcap.h b/arch/arm64/include/uapi/asm/hwcap.h index 69a4fb749c65..a2cac4305b1e 100644 --- a/arch/arm64/include/uapi/asm/hwcap.h +++ b/arch/arm64/include/uapi/asm/hwcap.h @@ -102,5 +102,6 @@ #define HWCAP2_SME_BI32I32 (1UL << 40) #define HWCAP2_SME_B16B16 (1UL << 41) #define HWCAP2_SME_F16F16 (1UL << 42) +#define HWCAP2_MOPS (1UL << 43) #endif /* _UAPI__ASM_HWCAP_H */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 7c2bb4e72476..3864a64e2b2b 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -42,8 +42,7 @@ obj-$(CONFIG_COMPAT) += sigreturn32.o obj-$(CONFIG_COMPAT_ALIGNMENT_FIXUPS) += compat_alignment.o obj-$(CONFIG_KUSER_HELPERS) += kuser32.o obj-$(CONFIG_FUNCTION_TRACER) += ftrace.o entry-ftrace.o -obj-$(CONFIG_MODULES) += module.o -obj-$(CONFIG_ARM64_MODULE_PLTS) += module-plts.o +obj-$(CONFIG_MODULES) += module.o module-plts.o obj-$(CONFIG_PERF_EVENTS) += perf_regs.o perf_callchain.o obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o obj-$(CONFIG_CPU_PM) += sleep.o suspend.o diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index d32d4ed5519b..8ff6610af496 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -24,8 +24,8 @@ #define ALT_ORIG_PTR(a) __ALT_PTR(a, orig_offset) #define ALT_REPL_PTR(a) __ALT_PTR(a, alt_offset) -#define ALT_CAP(a) ((a)->cpufeature & ~ARM64_CB_BIT) -#define ALT_HAS_CB(a) ((a)->cpufeature & ARM64_CB_BIT) +#define ALT_CAP(a) ((a)->cpucap & ~ARM64_CB_BIT) +#define ALT_HAS_CB(a) ((a)->cpucap & ARM64_CB_BIT) /* Volatile, as we may be patching the guts of READ_ONCE() */ static volatile int all_alternatives_applied; @@ -37,12 +37,12 @@ struct alt_region { struct alt_instr *end; }; -bool alternative_is_applied(u16 cpufeature) +bool alternative_is_applied(u16 cpucap) { - if (WARN_ON(cpufeature >= ARM64_NCAPS)) + if (WARN_ON(cpucap >= ARM64_NCAPS)) return false; - return test_bit(cpufeature, applied_alternatives); + return test_bit(cpucap, applied_alternatives); } /* @@ -121,11 +121,11 @@ static noinstr void patch_alternative(struct alt_instr *alt, * accidentally call into the cache.S code, which is patched by us at * runtime. */ -static void clean_dcache_range_nopatch(u64 start, u64 end) +static noinstr void clean_dcache_range_nopatch(u64 start, u64 end) { u64 cur, d_size, ctr_el0; - ctr_el0 = read_sanitised_ftr_reg(SYS_CTR_EL0); + ctr_el0 = arm64_ftr_reg_ctrel0.sys_val; d_size = 4 << cpuid_feature_extract_unsigned_field(ctr_el0, CTR_EL0_DminLine_SHIFT); cur = start & ~(d_size - 1); @@ -141,7 +141,7 @@ static void clean_dcache_range_nopatch(u64 start, u64 end) static void __apply_alternatives(const struct alt_region *region, bool is_module, - unsigned long *feature_mask) + unsigned long *cpucap_mask) { struct alt_instr *alt; __le32 *origptr, *updptr; @@ -151,7 +151,7 @@ static void __apply_alternatives(const struct alt_region *region, int nr_inst; int cap = ALT_CAP(alt); - if (!test_bit(cap, feature_mask)) + if (!test_bit(cap, cpucap_mask)) continue; if (!cpus_have_cap(cap)) @@ -188,11 +188,10 @@ static void __apply_alternatives(const struct alt_region *region, icache_inval_all_pou(); isb(); - /* Ignore ARM64_CB bit from feature mask */ bitmap_or(applied_alternatives, applied_alternatives, - feature_mask, ARM64_NCAPS); + cpucap_mask, ARM64_NCAPS); bitmap_and(applied_alternatives, applied_alternatives, - cpu_hwcaps, ARM64_NCAPS); + system_cpucaps, ARM64_NCAPS); } } @@ -239,7 +238,7 @@ static int __init __apply_alternatives_multi_stop(void *unused) } else { DECLARE_BITMAP(remaining_capabilities, ARM64_NCAPS); - bitmap_complement(remaining_capabilities, boot_capabilities, + bitmap_complement(remaining_capabilities, boot_cpucaps, ARM64_NCAPS); BUG_ON(all_alternatives_applied); @@ -274,7 +273,7 @@ void __init apply_boot_alternatives(void) pr_info("applying boot alternatives\n"); __apply_alternatives(&kernel_alternatives, false, - &boot_capabilities[0]); + &boot_cpucaps[0]); } #ifdef CONFIG_MODULES diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index f90f34fcf98d..8cd21c8bf98e 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -105,11 +105,11 @@ unsigned int compat_elf_hwcap __read_mostly = COMPAT_ELF_HWCAP_DEFAULT; unsigned int compat_elf_hwcap2 __read_mostly; #endif -DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); -EXPORT_SYMBOL(cpu_hwcaps); -static struct arm64_cpu_capabilities const __ro_after_init *cpu_hwcaps_ptrs[ARM64_NCAPS]; +DECLARE_BITMAP(system_cpucaps, ARM64_NCAPS); +EXPORT_SYMBOL(system_cpucaps); +static struct arm64_cpu_capabilities const __ro_after_init *cpucap_ptrs[ARM64_NCAPS]; -DECLARE_BITMAP(boot_capabilities, ARM64_NCAPS); +DECLARE_BITMAP(boot_cpucaps, ARM64_NCAPS); bool arm64_use_ng_mappings = false; EXPORT_SYMBOL(arm64_use_ng_mappings); @@ -137,7 +137,7 @@ static cpumask_var_t cpu_32bit_el0_mask __cpumask_var_read_mostly; void dump_cpu_features(void) { /* file-wide pr_fmt adds "CPU features: " prefix */ - pr_emerg("0x%*pb\n", ARM64_NCAPS, &cpu_hwcaps); + pr_emerg("0x%*pb\n", ARM64_NCAPS, &system_cpucaps); } #define ARM64_CPUID_FIELDS(reg, field, min_value) \ @@ -223,6 +223,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar2[] = { ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_CSSC_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_RPRFM_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_HIGHER_SAFE, ID_AA64ISAR2_EL1_BC_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR2_EL1_MOPS_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_PTR_AUTH), FTR_STRICT, FTR_EXACT, ID_AA64ISAR2_EL1_APA3_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_PTR_AUTH), @@ -364,6 +365,7 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr0[] = { static const struct arm64_ftr_bits ftr_id_aa64mmfr1[] = { ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR1_EL1_TIDCP1_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR1_EL1_AFP_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR1_EL1_HCX_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR1_EL1_ETS_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR1_EL1_TWED_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR1_EL1_XNX_SHIFT, 4, 0), @@ -396,6 +398,12 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr2[] = { ARM64_FTR_END, }; +static const struct arm64_ftr_bits ftr_id_aa64mmfr3[] = { + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL1_S1PIE_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR3_EL1_TCRX_SHIFT, 4, 0), + ARM64_FTR_END, +}; + static const struct arm64_ftr_bits ftr_ctr[] = { ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RES1 */ ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, CTR_EL0_DIC_SHIFT, 1, 1), @@ -722,6 +730,7 @@ static const struct __ftr_reg_entry { ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1, &id_aa64mmfr1_override), ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2), + ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3), /* Op1 = 0, CRn = 1, CRm = 2 */ ARM64_FTR_REG(SYS_ZCR_EL1, ftr_zcr), @@ -954,24 +963,24 @@ extern const struct arm64_cpu_capabilities arm64_errata[]; static const struct arm64_cpu_capabilities arm64_features[]; static void __init -init_cpu_hwcaps_indirect_list_from_array(const struct arm64_cpu_capabilities *caps) +init_cpucap_indirect_list_from_array(const struct arm64_cpu_capabilities *caps) { for (; caps->matches; caps++) { if (WARN(caps->capability >= ARM64_NCAPS, "Invalid capability %d\n", caps->capability)) continue; - if (WARN(cpu_hwcaps_ptrs[caps->capability], + if (WARN(cpucap_ptrs[caps->capability], "Duplicate entry for capability %d\n", caps->capability)) continue; - cpu_hwcaps_ptrs[caps->capability] = caps; + cpucap_ptrs[caps->capability] = caps; } } -static void __init init_cpu_hwcaps_indirect_list(void) +static void __init init_cpucap_indirect_list(void) { - init_cpu_hwcaps_indirect_list_from_array(arm64_features); - init_cpu_hwcaps_indirect_list_from_array(arm64_errata); + init_cpucap_indirect_list_from_array(arm64_features); + init_cpucap_indirect_list_from_array(arm64_errata); } static void __init setup_boot_cpu_capabilities(void); @@ -1017,6 +1026,7 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info) init_cpu_ftr_reg(SYS_ID_AA64MMFR0_EL1, info->reg_id_aa64mmfr0); init_cpu_ftr_reg(SYS_ID_AA64MMFR1_EL1, info->reg_id_aa64mmfr1); init_cpu_ftr_reg(SYS_ID_AA64MMFR2_EL1, info->reg_id_aa64mmfr2); + init_cpu_ftr_reg(SYS_ID_AA64MMFR3_EL1, info->reg_id_aa64mmfr3); init_cpu_ftr_reg(SYS_ID_AA64PFR0_EL1, info->reg_id_aa64pfr0); init_cpu_ftr_reg(SYS_ID_AA64PFR1_EL1, info->reg_id_aa64pfr1); init_cpu_ftr_reg(SYS_ID_AA64ZFR0_EL1, info->reg_id_aa64zfr0); @@ -1049,10 +1059,10 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info) init_cpu_ftr_reg(SYS_GMID_EL1, info->reg_gmid); /* - * Initialize the indirect array of CPU hwcaps capabilities pointers - * before we handle the boot CPU below. + * Initialize the indirect array of CPU capabilities pointers before we + * handle the boot CPU below. */ - init_cpu_hwcaps_indirect_list(); + init_cpucap_indirect_list(); /* * Detect and enable early CPU capabilities based on the boot CPU, @@ -1262,6 +1272,8 @@ void update_cpu_features(int cpu, info->reg_id_aa64mmfr1, boot->reg_id_aa64mmfr1); taint |= check_update_ftr_reg(SYS_ID_AA64MMFR2_EL1, cpu, info->reg_id_aa64mmfr2, boot->reg_id_aa64mmfr2); + taint |= check_update_ftr_reg(SYS_ID_AA64MMFR3_EL1, cpu, + info->reg_id_aa64mmfr3, boot->reg_id_aa64mmfr3); taint |= check_update_ftr_reg(SYS_ID_AA64PFR0_EL1, cpu, info->reg_id_aa64pfr0, boot->reg_id_aa64pfr0); @@ -1391,6 +1403,7 @@ u64 __read_sysreg_by_encoding(u32 sys_id) read_sysreg_case(SYS_ID_AA64MMFR0_EL1); read_sysreg_case(SYS_ID_AA64MMFR1_EL1); read_sysreg_case(SYS_ID_AA64MMFR2_EL1); + read_sysreg_case(SYS_ID_AA64MMFR3_EL1); read_sysreg_case(SYS_ID_AA64ISAR0_EL1); read_sysreg_case(SYS_ID_AA64ISAR1_EL1); read_sysreg_case(SYS_ID_AA64ISAR2_EL1); @@ -2049,9 +2062,9 @@ static bool has_address_auth_cpucap(const struct arm64_cpu_capabilities *entry, static bool has_address_auth_metacap(const struct arm64_cpu_capabilities *entry, int scope) { - bool api = has_address_auth_cpucap(cpu_hwcaps_ptrs[ARM64_HAS_ADDRESS_AUTH_IMP_DEF], scope); - bool apa = has_address_auth_cpucap(cpu_hwcaps_ptrs[ARM64_HAS_ADDRESS_AUTH_ARCH_QARMA5], scope); - bool apa3 = has_address_auth_cpucap(cpu_hwcaps_ptrs[ARM64_HAS_ADDRESS_AUTH_ARCH_QARMA3], scope); + bool api = has_address_auth_cpucap(cpucap_ptrs[ARM64_HAS_ADDRESS_AUTH_IMP_DEF], scope); + bool apa = has_address_auth_cpucap(cpucap_ptrs[ARM64_HAS_ADDRESS_AUTH_ARCH_QARMA5], scope); + bool apa3 = has_address_auth_cpucap(cpucap_ptrs[ARM64_HAS_ADDRESS_AUTH_ARCH_QARMA3], scope); return apa || apa3 || api; } @@ -2187,6 +2200,11 @@ static void cpu_enable_dit(const struct arm64_cpu_capabilities *__unused) set_pstate_dit(1); } +static void cpu_enable_mops(const struct arm64_cpu_capabilities *__unused) +{ + sysreg_clear_set(sctlr_el1, 0, SCTLR_EL1_MSCEn); +} + /* Internal helper functions to match cpu capability type */ static bool cpucap_late_cpu_optional(const struct arm64_cpu_capabilities *cap) @@ -2236,11 +2254,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .capability = ARM64_HAS_ECV_CNTPOFF, .type = ARM64_CPUCAP_SYSTEM_FEATURE, .matches = has_cpuid_feature, - .sys_reg = SYS_ID_AA64MMFR0_EL1, - .field_pos = ID_AA64MMFR0_EL1_ECV_SHIFT, - .field_width = 4, - .sign = FTR_UNSIGNED, - .min_field_value = ID_AA64MMFR0_EL1_ECV_CNTPOFF, + ARM64_CPUID_FIELDS(ID_AA64MMFR0_EL1, ECV, CNTPOFF) }, #ifdef CONFIG_ARM64_PAN { @@ -2310,6 +2324,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .type = ARM64_CPUCAP_SYSTEM_FEATURE, .matches = is_kvm_protected_mode, }, + { + .desc = "HCRX_EL2 register", + .capability = ARM64_HAS_HCX, + .type = ARM64_CPUCAP_STRICT_BOOT_CPU_FEATURE, + .matches = has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HCX, IMP) + }, #endif { .desc = "Kernel page table isolation (KPTI)", @@ -2642,6 +2663,27 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .cpu_enable = cpu_enable_dit, ARM64_CPUID_FIELDS(ID_AA64PFR0_EL1, DIT, IMP) }, + { + .desc = "Memory Copy and Memory Set instructions", + .capability = ARM64_HAS_MOPS, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_cpuid_feature, + .cpu_enable = cpu_enable_mops, + ARM64_CPUID_FIELDS(ID_AA64ISAR2_EL1, MOPS, IMP) + }, + { + .capability = ARM64_HAS_TCR2, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR3_EL1, TCRX, IMP) + }, + { + .desc = "Stage-1 Permission Indirection Extension (S1PIE)", + .capability = ARM64_HAS_S1PIE, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .matches = has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR3_EL1, S1PIE, IMP) + }, {}, }; @@ -2770,6 +2812,7 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = { HWCAP_CAP(ID_AA64ISAR2_EL1, RPRFM, IMP, CAP_HWCAP, KERNEL_HWCAP_RPRFM), HWCAP_CAP(ID_AA64ISAR2_EL1, RPRES, IMP, CAP_HWCAP, KERNEL_HWCAP_RPRES), HWCAP_CAP(ID_AA64ISAR2_EL1, WFxT, IMP, CAP_HWCAP, KERNEL_HWCAP_WFXT), + HWCAP_CAP(ID_AA64ISAR2_EL1, MOPS, IMP, CAP_HWCAP, KERNEL_HWCAP_MOPS), #ifdef CONFIG_ARM64_SME HWCAP_CAP(ID_AA64PFR1_EL1, SME, IMP, CAP_HWCAP, KERNEL_HWCAP_SME), HWCAP_CAP(ID_AA64SMFR0_EL1, FA64, IMP, CAP_HWCAP, KERNEL_HWCAP_SME_FA64), @@ -2896,7 +2939,7 @@ static void update_cpu_capabilities(u16 scope_mask) scope_mask &= ARM64_CPUCAP_SCOPE_MASK; for (i = 0; i < ARM64_NCAPS; i++) { - caps = cpu_hwcaps_ptrs[i]; + caps = cpucap_ptrs[i]; if (!caps || !(caps->type & scope_mask) || cpus_have_cap(caps->capability) || !caps->matches(caps, cpucap_default_scope(caps))) @@ -2904,10 +2947,11 @@ static void update_cpu_capabilities(u16 scope_mask) if (caps->desc) pr_info("detected: %s\n", caps->desc); - cpus_set_cap(caps->capability); + + __set_bit(caps->capability, system_cpucaps); if ((scope_mask & SCOPE_BOOT_CPU) && (caps->type & SCOPE_BOOT_CPU)) - set_bit(caps->capability, boot_capabilities); + set_bit(caps->capability, boot_cpucaps); } } @@ -2921,7 +2965,7 @@ static int cpu_enable_non_boot_scope_capabilities(void *__unused) u16 non_boot_scope = SCOPE_ALL & ~SCOPE_BOOT_CPU; for_each_available_cap(i) { - const struct arm64_cpu_capabilities *cap = cpu_hwcaps_ptrs[i]; + const struct arm64_cpu_capabilities *cap = cpucap_ptrs[i]; if (WARN_ON(!cap)) continue; @@ -2951,7 +2995,7 @@ static void __init enable_cpu_capabilities(u16 scope_mask) for (i = 0; i < ARM64_NCAPS; i++) { unsigned int num; - caps = cpu_hwcaps_ptrs[i]; + caps = cpucap_ptrs[i]; if (!caps || !(caps->type & scope_mask)) continue; num = caps->capability; @@ -2996,7 +3040,7 @@ static void verify_local_cpu_caps(u16 scope_mask) scope_mask &= ARM64_CPUCAP_SCOPE_MASK; for (i = 0; i < ARM64_NCAPS; i++) { - caps = cpu_hwcaps_ptrs[i]; + caps = cpucap_ptrs[i]; if (!caps || !(caps->type & scope_mask)) continue; @@ -3195,7 +3239,7 @@ static void __init setup_boot_cpu_capabilities(void) bool this_cpu_has_cap(unsigned int n) { if (!WARN_ON(preemptible()) && n < ARM64_NCAPS) { - const struct arm64_cpu_capabilities *cap = cpu_hwcaps_ptrs[n]; + const struct arm64_cpu_capabilities *cap = cpucap_ptrs[n]; if (cap) return cap->matches(cap, SCOPE_LOCAL_CPU); @@ -3208,13 +3252,13 @@ EXPORT_SYMBOL_GPL(this_cpu_has_cap); /* * This helper function is used in a narrow window when, * - The system wide safe registers are set with all the SMP CPUs and, - * - The SYSTEM_FEATURE cpu_hwcaps may not have been set. + * - The SYSTEM_FEATURE system_cpucaps may not have been set. * In all other cases cpus_have_{const_}cap() should be used. */ static bool __maybe_unused __system_matches_cap(unsigned int n) { if (n < ARM64_NCAPS) { - const struct arm64_cpu_capabilities *cap = cpu_hwcaps_ptrs[n]; + const struct arm64_cpu_capabilities *cap = cpucap_ptrs[n]; if (cap) return cap->matches(cap, SCOPE_SYSTEM); diff --git a/arch/arm64/kernel/cpuidle.c b/arch/arm64/kernel/cpuidle.c index 42e19fff40ee..d1f68599c29f 100644 --- a/arch/arm64/kernel/cpuidle.c +++ b/arch/arm64/kernel/cpuidle.c @@ -13,7 +13,7 @@ #include #include -#ifdef CONFIG_ACPI +#ifdef CONFIG_ACPI_PROCESSOR_IDLE #include diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c index eb4378c23b3c..58622dc85917 100644 --- a/arch/arm64/kernel/cpuinfo.c +++ b/arch/arm64/kernel/cpuinfo.c @@ -125,6 +125,7 @@ static const char *const hwcap_str[] = { [KERNEL_HWCAP_SME_BI32I32] = "smebi32i32", [KERNEL_HWCAP_SME_B16B16] = "smeb16b16", [KERNEL_HWCAP_SME_F16F16] = "smef16f16", + [KERNEL_HWCAP_MOPS] = "mops", }; #ifdef CONFIG_COMPAT @@ -446,6 +447,7 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) info->reg_id_aa64mmfr0 = read_cpuid(ID_AA64MMFR0_EL1); info->reg_id_aa64mmfr1 = read_cpuid(ID_AA64MMFR1_EL1); info->reg_id_aa64mmfr2 = read_cpuid(ID_AA64MMFR2_EL1); + info->reg_id_aa64mmfr3 = read_cpuid(ID_AA64MMFR3_EL1); info->reg_id_aa64pfr0 = read_cpuid(ID_AA64PFR0_EL1); info->reg_id_aa64pfr1 = read_cpuid(ID_AA64PFR1_EL1); info->reg_id_aa64zfr0 = read_cpuid(ID_AA64ZFR0_EL1); diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 3af3c01c93a6..6b2e0c367702 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -126,7 +126,7 @@ static __always_inline void __exit_to_user_mode(void) lockdep_hardirqs_on(CALLER_ADDR0); } -static __always_inline void prepare_exit_to_user_mode(struct pt_regs *regs) +static __always_inline void exit_to_user_mode_prepare(struct pt_regs *regs) { unsigned long flags; @@ -135,11 +135,13 @@ static __always_inline void prepare_exit_to_user_mode(struct pt_regs *regs) flags = read_thread_flags(); if (unlikely(flags & _TIF_WORK_MASK)) do_notify_resume(regs, flags); + + lockdep_sys_exit(); } static __always_inline void exit_to_user_mode(struct pt_regs *regs) { - prepare_exit_to_user_mode(regs); + exit_to_user_mode_prepare(regs); mte_check_tfsr_exit(); __exit_to_user_mode(); } @@ -611,6 +613,14 @@ static void noinstr el0_bti(struct pt_regs *regs) exit_to_user_mode(regs); } +static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr) +{ + enter_from_user_mode(regs); + local_daif_restore(DAIF_PROCCTX); + do_el0_mops(regs, esr); + exit_to_user_mode(regs); +} + static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr) { enter_from_user_mode(regs); @@ -688,6 +698,9 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs) case ESR_ELx_EC_BTI: el0_bti(regs); break; + case ESR_ELx_EC_MOPS: + el0_mops(regs, esr); + break; case ESR_ELx_EC_BREAKPT_LOW: case ESR_ELx_EC_SOFTSTP_LOW: case ESR_ELx_EC_WATCHPT_LOW: diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ab2a6e33c052..a40e5e50fa55 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -101,12 +101,11 @@ .org .Lventry_start\@ + 128 // Did we overflow the ventry slot? .endm - .macro tramp_alias, dst, sym, tmp - mov_q \dst, TRAMP_VALIAS - adr_l \tmp, \sym - add \dst, \dst, \tmp - adr_l \tmp, .entry.tramp.text - sub \dst, \dst, \tmp + .macro tramp_alias, dst, sym + .set .Lalias\@, TRAMP_VALIAS + \sym - .entry.tramp.text + movz \dst, :abs_g2_s:.Lalias\@ + movk \dst, :abs_g1_nc:.Lalias\@ + movk \dst, :abs_g0_nc:.Lalias\@ .endm /* @@ -435,13 +434,14 @@ alternative_if_not ARM64_UNMAP_KERNEL_AT_EL0 eret alternative_else_nop_endif #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 - bne 4f msr far_el1, x29 - tramp_alias x30, tramp_exit_native, x29 - br x30 -4: - tramp_alias x30, tramp_exit_compat, x29 - br x30 + + ldr_this_cpu x30, this_cpu_vector, x29 + tramp_alias x29, tramp_exit + msr vbar_el1, x30 // install vector table + ldr lr, [sp, #S_LR] // restore x30 + add sp, sp, #PT_REGS_SIZE // restore sp + br x29 #endif .else ldr lr, [sp, #S_LR] @@ -732,22 +732,6 @@ alternative_else_nop_endif .org 1b + 128 // Did we overflow the ventry slot? .endm - .macro tramp_exit, regsize = 64 - tramp_data_read_var x30, this_cpu_vector - get_this_cpu_offset x29 - ldr x30, [x30, x29] - - msr vbar_el1, x30 - ldr lr, [sp, #S_LR] - tramp_unmap_kernel x29 - .if \regsize == 64 - mrs x29, far_el1 - .endif - add sp, sp, #PT_REGS_SIZE // restore sp - eret - sb - .endm - .macro generate_tramp_vector, kpti, bhb .Lvector_start\@: .space 0x400 @@ -768,7 +752,7 @@ alternative_else_nop_endif */ .pushsection ".entry.tramp.text", "ax" .align 11 -SYM_CODE_START_NOALIGN(tramp_vectors) +SYM_CODE_START_LOCAL_NOALIGN(tramp_vectors) #ifdef CONFIG_MITIGATE_SPECTRE_BRANCH_HISTORY generate_tramp_vector kpti=1, bhb=BHB_MITIGATION_LOOP generate_tramp_vector kpti=1, bhb=BHB_MITIGATION_FW @@ -777,13 +761,12 @@ SYM_CODE_START_NOALIGN(tramp_vectors) generate_tramp_vector kpti=1, bhb=BHB_MITIGATION_NONE SYM_CODE_END(tramp_vectors) -SYM_CODE_START(tramp_exit_native) - tramp_exit -SYM_CODE_END(tramp_exit_native) - -SYM_CODE_START(tramp_exit_compat) - tramp_exit 32 -SYM_CODE_END(tramp_exit_compat) +SYM_CODE_START_LOCAL(tramp_exit) + tramp_unmap_kernel x29 + mrs x29, far_el1 // restore x29 + eret + sb +SYM_CODE_END(tramp_exit) .popsection // .entry.tramp.text #endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ @@ -1077,7 +1060,7 @@ alternative_if_not ARM64_UNMAP_KERNEL_AT_EL0 alternative_else_nop_endif #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 - tramp_alias dst=x5, sym=__sdei_asm_exit_trampoline, tmp=x3 + tramp_alias dst=x5, sym=__sdei_asm_exit_trampoline br x5 #endif SYM_CODE_END(__sdei_asm_handler) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 2fbafa5cc7ac..7a1aeb95d7c3 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1649,6 +1649,7 @@ void fpsimd_flush_thread(void) fpsimd_flush_thread_vl(ARM64_VEC_SME); current->thread.svcr = 0; + sme_smstop(); } current->thread.fp_type = FP_STATE_FPSIMD; diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index 432626c866a8..a650f5e11fc5 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -197,7 +197,7 @@ int ftrace_update_ftrace_func(ftrace_func_t func) static struct plt_entry *get_ftrace_plt(struct module *mod) { -#ifdef CONFIG_ARM64_MODULE_PLTS +#ifdef CONFIG_MODULES struct plt_entry *plt = mod->arch.ftrace_trampolines; return &plt[FTRACE_PLT_IDX]; @@ -249,7 +249,7 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec, * must use a PLT to reach it. We can only place PLTs for modules, and * only when module PLT support is built-in. */ - if (!IS_ENABLED(CONFIG_ARM64_MODULE_PLTS)) + if (!IS_ENABLED(CONFIG_MODULES)) return false; /* @@ -431,10 +431,8 @@ int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, * * Note: 'mod' is only set at module load time. */ - if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_ARGS) && - IS_ENABLED(CONFIG_ARM64_MODULE_PLTS) && mod) { + if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_ARGS) && mod) return aarch64_insn_patch_text_nosync((void *)pc, new); - } if (!ftrace_find_callable_addr(rec, mod, &addr)) return -EINVAL; diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e92caebff46a..0f5a30f109d9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -382,7 +382,7 @@ SYM_FUNC_START_LOCAL(create_idmap) adrp x0, init_idmap_pg_dir adrp x3, _text adrp x6, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - mov x7, SWAPPER_RX_MMUFLAGS + mov_q x7, SWAPPER_RX_MMUFLAGS map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT @@ -391,7 +391,7 @@ SYM_FUNC_START_LOCAL(create_idmap) adrp x2, init_pg_dir adrp x3, init_pg_end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 - mov x5, SWAPPER_RW_MMUFLAGS + mov_q x5, SWAPPER_RW_MMUFLAGS mov x6, #SWAPPER_BLOCK_SHIFT bl remap_region @@ -402,7 +402,7 @@ SYM_FUNC_START_LOCAL(create_idmap) bfi x22, x21, #0, #SWAPPER_BLOCK_SHIFT // remapped FDT address add x3, x2, #MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE bic x4, x21, #SWAPPER_BLOCK_SIZE - 1 - mov x5, SWAPPER_RW_MMUFLAGS + mov_q x5, SWAPPER_RW_MMUFLAGS mov x6, #SWAPPER_BLOCK_SHIFT bl remap_region @@ -430,7 +430,7 @@ SYM_FUNC_START_LOCAL(create_kernel_mapping) adrp x3, _text // runtime __pa(_text) sub x6, x6, x3 // _end - _text add x6, x6, x5 // runtime __va(_end) - mov x7, SWAPPER_RW_MMUFLAGS + mov_q x7, SWAPPER_RW_MMUFLAGS map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 788597a6b6a2..02870beb271e 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -99,7 +99,6 @@ int pfn_is_nosave(unsigned long pfn) void notrace save_processor_state(void) { - WARN_ON(num_online_cpus() != 1); } void notrace restore_processor_state(void) diff --git a/arch/arm64/kernel/hw_breakpoint.c b/arch/arm64/kernel/hw_breakpoint.c index b29a311bb055..db2a1861bb97 100644 --- a/arch/arm64/kernel/hw_breakpoint.c +++ b/arch/arm64/kernel/hw_breakpoint.c @@ -973,14 +973,6 @@ static int hw_breakpoint_reset(unsigned int cpu) return 0; } -#ifdef CONFIG_CPU_PM -extern void cpu_suspend_set_dbg_restorer(int (*hw_bp_restore)(unsigned int)); -#else -static inline void cpu_suspend_set_dbg_restorer(int (*hw_bp_restore)(unsigned int)) -{ -} -#endif - /* * One-time initialisation. */ diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S index 9439240c3fcf..d63de1973ddb 100644 --- a/arch/arm64/kernel/hyp-stub.S +++ b/arch/arm64/kernel/hyp-stub.S @@ -119,6 +119,24 @@ SYM_CODE_START_LOCAL(__finalise_el2) msr ttbr1_el1, x0 mrs_s x0, SYS_MAIR_EL12 msr mair_el1, x0 + mrs x1, REG_ID_AA64MMFR3_EL1 + ubfx x1, x1, #ID_AA64MMFR3_EL1_TCRX_SHIFT, #4 + cbz x1, .Lskip_tcr2 + mrs x0, REG_TCR2_EL12 + msr REG_TCR2_EL1, x0 + + // Transfer permission indirection state + mrs x1, REG_ID_AA64MMFR3_EL1 + ubfx x1, x1, #ID_AA64MMFR3_EL1_S1PIE_SHIFT, #4 + cbz x1, .Lskip_indirection + mrs x0, REG_PIRE0_EL12 + msr REG_PIRE0_EL1, x0 + mrs x0, REG_PIR_EL12 + msr REG_PIR_EL1, x0 + +.Lskip_indirection: +.Lskip_tcr2: + isb // Hack the exception return to stay at EL2 diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 370ab84fd06e..8439248c21d3 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -123,6 +123,7 @@ static const struct ftr_set_desc isar2 __initconst = { .fields = { FIELD("gpa3", ID_AA64ISAR2_EL1_GPA3_SHIFT, NULL), FIELD("apa3", ID_AA64ISAR2_EL1_APA3_SHIFT, NULL), + FIELD("mops", ID_AA64ISAR2_EL1_MOPS_SHIFT, NULL), {} }, }; @@ -174,6 +175,7 @@ static const struct { "id_aa64isar1.gpi=0 id_aa64isar1.gpa=0 " "id_aa64isar1.api=0 id_aa64isar1.apa=0 " "id_aa64isar2.gpa3=0 id_aa64isar2.apa3=0" }, + { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "kaslr.disabled=1" }, }; diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index e7477f21a4c9..17f96a19781d 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -4,90 +4,35 @@ */ #include -#include #include -#include -#include -#include -#include -#include -#include +#include -#include -#include +#include #include -#include -#include -#include -u64 __ro_after_init module_alloc_base; u16 __initdata memstart_offset_seed; struct arm64_ftr_override kaslr_feature_override __initdata; -static int __init kaslr_init(void) +bool __ro_after_init __kaslr_is_enabled = false; + +void __init kaslr_init(void) { - u64 module_range; - u32 seed; - - /* - * Set a reasonable default for module_alloc_base in case - * we end up running with module randomization disabled. - */ - module_alloc_base = (u64)_etext - MODULES_VSIZE; - if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { pr_info("KASLR disabled on command line\n"); - return 0; + return; } - if (!kaslr_enabled()) { + /* + * The KASLR offset modulo MIN_KIMG_ALIGN is taken from the physical + * placement of the image rather than from the seed, so a displacement + * of less than MIN_KIMG_ALIGN means that no seed was provided. + */ + if (kaslr_offset() < MIN_KIMG_ALIGN) { pr_warn("KASLR disabled due to lack of seed\n"); - return 0; + return; } pr_info("KASLR enabled\n"); - - /* - * KASAN without KASAN_VMALLOC does not expect the module region to - * intersect the vmalloc region, since shadow memory is allocated for - * each module at load time, whereas the vmalloc region will already be - * shadowed by KASAN zero pages. - */ - BUILD_BUG_ON((IS_ENABLED(CONFIG_KASAN_GENERIC) || - IS_ENABLED(CONFIG_KASAN_SW_TAGS)) && - !IS_ENABLED(CONFIG_KASAN_VMALLOC)); - - seed = get_random_u32(); - - if (IS_ENABLED(CONFIG_RANDOMIZE_MODULE_REGION_FULL)) { - /* - * Randomize the module region over a 2 GB window covering the - * kernel. This reduces the risk of modules leaking information - * about the address of the kernel itself, but results in - * branches between modules and the core kernel that are - * resolved via PLTs. (Branches between modules will be - * resolved normally.) - */ - module_range = SZ_2G - (u64)(_end - _stext); - module_alloc_base = max((u64)_end - SZ_2G, (u64)MODULES_VADDR); - } else { - /* - * Randomize the module region by setting module_alloc_base to - * a PAGE_SIZE multiple in the range [_etext - MODULES_VSIZE, - * _stext) . This guarantees that the resulting region still - * covers [_stext, _etext], and that all relative branches can - * be resolved without veneers unless this region is exhausted - * and we fall back to a larger 2GB window in module_alloc() - * when ARM64_MODULE_PLTS is enabled. - */ - module_range = MODULES_VSIZE - (u64)(_etext - _stext); - } - - /* use the lower 21 bits to randomize the base of the module region */ - module_alloc_base += (module_range * (seed & ((1 << 21) - 1))) >> 21; - module_alloc_base &= PAGE_MASK; - - return 0; + __kaslr_is_enabled = true; } -subsys_initcall(kaslr_init) diff --git a/arch/arm64/kernel/kuser32.S b/arch/arm64/kernel/kuser32.S index 692e9d2e31e5..af046ceac22d 100644 --- a/arch/arm64/kernel/kuser32.S +++ b/arch/arm64/kernel/kuser32.S @@ -10,7 +10,7 @@ * aarch32_setup_additional_pages() and are provided for compatibility * reasons with 32 bit (aarch32) applications that need them. * - * See Documentation/arm/kernel_user_helpers.rst for formal definitions. + * See Documentation/arch/arm/kernel_user_helpers.rst for formal definitions. */ #include diff --git a/arch/arm64/kernel/module-plts.c b/arch/arm64/kernel/module-plts.c index 543493bf924d..ad02058756b5 100644 --- a/arch/arm64/kernel/module-plts.c +++ b/arch/arm64/kernel/module-plts.c @@ -7,6 +7,7 @@ #include #include #include +#include #include static struct plt_entry __get_adrp_add_pair(u64 dst, u64 pc, diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index 5af4975caeb5..dd851297596e 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -7,6 +7,8 @@ * Author: Will Deacon */ +#define pr_fmt(fmt) "Modules: " fmt + #include #include #include @@ -15,52 +17,131 @@ #include #include #include +#include #include #include + #include #include #include #include +static u64 module_direct_base __ro_after_init = 0; +static u64 module_plt_base __ro_after_init = 0; + +/* + * Choose a random page-aligned base address for a window of 'size' bytes which + * entirely contains the interval [start, end - 1]. + */ +static u64 __init random_bounding_box(u64 size, u64 start, u64 end) +{ + u64 max_pgoff, pgoff; + + if ((end - start) >= size) + return 0; + + max_pgoff = (size - (end - start)) / PAGE_SIZE; + pgoff = get_random_u32_inclusive(0, max_pgoff); + + return start - pgoff * PAGE_SIZE; +} + +/* + * Modules may directly reference data and text anywhere within the kernel + * image and other modules. References using PREL32 relocations have a +/-2G + * range, and so we need to ensure that the entire kernel image and all modules + * fall within a 2G window such that these are always within range. + * + * Modules may directly branch to functions and code within the kernel text, + * and to functions and code within other modules. These branches will use + * CALL26/JUMP26 relocations with a +/-128M range. Without PLTs, we must ensure + * that the entire kernel text and all module text falls within a 128M window + * such that these are always within range. With PLTs, we can expand this to a + * 2G window. + * + * We chose the 128M region to surround the entire kernel image (rather than + * just the text) as using the same bounds for the 128M and 2G regions ensures + * by construction that we never select a 128M region that is not a subset of + * the 2G region. For very large and unusual kernel configurations this means + * we may fall back to PLTs where they could have been avoided, but this keeps + * the logic significantly simpler. + */ +static int __init module_init_limits(void) +{ + u64 kernel_end = (u64)_end; + u64 kernel_start = (u64)_text; + u64 kernel_size = kernel_end - kernel_start; + + /* + * The default modules region is placed immediately below the kernel + * image, and is large enough to use the full 2G relocation range. + */ + BUILD_BUG_ON(KIMAGE_VADDR != MODULES_END); + BUILD_BUG_ON(MODULES_VSIZE < SZ_2G); + + if (!kaslr_enabled()) { + if (kernel_size < SZ_128M) + module_direct_base = kernel_end - SZ_128M; + if (kernel_size < SZ_2G) + module_plt_base = kernel_end - SZ_2G; + } else { + u64 min = kernel_start; + u64 max = kernel_end; + + if (IS_ENABLED(CONFIG_RANDOMIZE_MODULE_REGION_FULL)) { + pr_info("2G module region forced by RANDOMIZE_MODULE_REGION_FULL\n"); + } else { + module_direct_base = random_bounding_box(SZ_128M, min, max); + if (module_direct_base) { + min = module_direct_base; + max = module_direct_base + SZ_128M; + } + } + + module_plt_base = random_bounding_box(SZ_2G, min, max); + } + + pr_info("%llu pages in range for non-PLT usage", + module_direct_base ? (SZ_128M - kernel_size) / PAGE_SIZE : 0); + pr_info("%llu pages in range for PLT usage", + module_plt_base ? (SZ_2G - kernel_size) / PAGE_SIZE : 0); + + return 0; +} +subsys_initcall(module_init_limits); + void *module_alloc(unsigned long size) { - u64 module_alloc_end = module_alloc_base + MODULES_VSIZE; - gfp_t gfp_mask = GFP_KERNEL; - void *p; + void *p = NULL; - /* Silence the initial allocation */ - if (IS_ENABLED(CONFIG_ARM64_MODULE_PLTS)) - gfp_mask |= __GFP_NOWARN; + /* + * Where possible, prefer to allocate within direct branch range of the + * kernel such that no PLTs are necessary. + */ + if (module_direct_base) { + p = __vmalloc_node_range(size, MODULE_ALIGN, + module_direct_base, + module_direct_base + SZ_128M, + GFP_KERNEL | __GFP_NOWARN, + PAGE_KERNEL, 0, NUMA_NO_NODE, + __builtin_return_address(0)); + } - if (IS_ENABLED(CONFIG_KASAN_GENERIC) || - IS_ENABLED(CONFIG_KASAN_SW_TAGS)) - /* don't exceed the static module region - see below */ - module_alloc_end = MODULES_END; + if (!p && module_plt_base) { + p = __vmalloc_node_range(size, MODULE_ALIGN, + module_plt_base, + module_plt_base + SZ_2G, + GFP_KERNEL | __GFP_NOWARN, + PAGE_KERNEL, 0, NUMA_NO_NODE, + __builtin_return_address(0)); + } - p = __vmalloc_node_range(size, MODULE_ALIGN, module_alloc_base, - module_alloc_end, gfp_mask, PAGE_KERNEL, VM_DEFER_KMEMLEAK, - NUMA_NO_NODE, __builtin_return_address(0)); + if (!p) { + pr_warn_ratelimited("%s: unable to allocate memory\n", + __func__); + } - if (!p && IS_ENABLED(CONFIG_ARM64_MODULE_PLTS) && - (IS_ENABLED(CONFIG_KASAN_VMALLOC) || - (!IS_ENABLED(CONFIG_KASAN_GENERIC) && - !IS_ENABLED(CONFIG_KASAN_SW_TAGS)))) - /* - * KASAN without KASAN_VMALLOC can only deal with module - * allocations being served from the reserved module region, - * since the remainder of the vmalloc region is already - * backed by zero shadow pages, and punching holes into it - * is non-trivial. Since the module region is not randomized - * when KASAN is enabled without KASAN_VMALLOC, it is even - * less likely that the module region gets exhausted, so we - * can simply omit this fallback in that case. - */ - p = __vmalloc_node_range(size, MODULE_ALIGN, module_alloc_base, - module_alloc_base + SZ_2G, GFP_KERNEL, - PAGE_KERNEL, 0, NUMA_NO_NODE, - __builtin_return_address(0)); - - if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) { + if (p && (kasan_alloc_module_shadow(p, size, GFP_KERNEL) < 0)) { vfree(p); return NULL; } @@ -448,9 +529,7 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, case R_AARCH64_CALL26: ovf = reloc_insn_imm(RELOC_OP_PREL, loc, val, 2, 26, AARCH64_INSN_IMM_26); - - if (IS_ENABLED(CONFIG_ARM64_MODULE_PLTS) && - ovf == -ERANGE) { + if (ovf == -ERANGE) { val = module_emit_plt_entry(me, sechdrs, loc, &rel[i], sym); if (!val) return -ENOEXEC; @@ -487,7 +566,7 @@ static int module_init_ftrace_plt(const Elf_Ehdr *hdr, const Elf_Shdr *sechdrs, struct module *mod) { -#if defined(CONFIG_ARM64_MODULE_PLTS) && defined(CONFIG_DYNAMIC_FTRACE) +#if defined(CONFIG_DYNAMIC_FTRACE) const Elf_Shdr *s; struct plt_entry *plts; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index b8ec7b3ac9cb..417a8a86b2db 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -296,6 +296,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) *cmdline_p = boot_command_line; + kaslr_init(); + /* * If know now we are going to need KPTI then use non-global * mappings from the start, avoiding the cost of rewriting diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index 2cfc810d0a5b..e304f7ebec2a 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -398,7 +399,7 @@ static int restore_tpidr2_context(struct user_ctxs *user) __get_user_error(tpidr2_el0, &user->tpidr2->tpidr2, err); if (!err) - current->thread.tpidr2_el0 = tpidr2_el0; + write_sysreg_s(tpidr2_el0, SYS_TPIDR2_EL0); return err; } diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index da84cf855c44..5a668d7f3c1f 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -147,11 +147,9 @@ static void el0_svc_common(struct pt_regs *regs, int scno, int sc_nr, * exit regardless, as the old entry assembly did. */ if (!has_syscall_work(flags) && !IS_ENABLED(CONFIG_DEBUG_RSEQ)) { - local_daif_mask(); flags = read_thread_flags(); if (!has_syscall_work(flags) && !(flags & _TIF_SINGLESTEP)) return; - local_daif_restore(DAIF_PROCCTX); } trace_exit: diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 4bb1b8f47298..794a2dd3659a 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -514,6 +514,63 @@ void do_el1_fpac(struct pt_regs *regs, unsigned long esr) die("Oops - FPAC", regs, esr); } +void do_el0_mops(struct pt_regs *regs, unsigned long esr) +{ + bool wrong_option = esr & ESR_ELx_MOPS_ISS_WRONG_OPTION; + bool option_a = esr & ESR_ELx_MOPS_ISS_OPTION_A; + int dstreg = ESR_ELx_MOPS_ISS_DESTREG(esr); + int srcreg = ESR_ELx_MOPS_ISS_SRCREG(esr); + int sizereg = ESR_ELx_MOPS_ISS_SIZEREG(esr); + unsigned long dst, src, size; + + dst = pt_regs_read_reg(regs, dstreg); + src = pt_regs_read_reg(regs, srcreg); + size = pt_regs_read_reg(regs, sizereg); + + /* + * Put the registers back in the original format suitable for a + * prologue instruction, using the generic return routine from the + * Arm ARM (DDI 0487I.a) rules CNTMJ and MWFQH. + */ + if (esr & ESR_ELx_MOPS_ISS_MEM_INST) { + /* SET* instruction */ + if (option_a ^ wrong_option) { + /* Format is from Option A; forward set */ + pt_regs_write_reg(regs, dstreg, dst + size); + pt_regs_write_reg(regs, sizereg, -size); + } + } else { + /* CPY* instruction */ + if (!(option_a ^ wrong_option)) { + /* Format is from Option B */ + if (regs->pstate & PSR_N_BIT) { + /* Backward copy */ + pt_regs_write_reg(regs, dstreg, dst - size); + pt_regs_write_reg(regs, srcreg, src - size); + } + } else { + /* Format is from Option A */ + if (size & BIT(63)) { + /* Forward copy */ + pt_regs_write_reg(regs, dstreg, dst + size); + pt_regs_write_reg(regs, srcreg, src + size); + pt_regs_write_reg(regs, sizereg, -size); + } + } + } + + if (esr & ESR_ELx_MOPS_ISS_FROM_EPILOGUE) + regs->pc -= 8; + else + regs->pc -= 4; + + /* + * If single stepping then finish the step before executing the + * prologue instruction. + */ + user_fastforward_single_step(current); +} + #define __user_cache_maint(insn, address, res) \ if (address >= TASK_SIZE_MAX) { \ res = -EFAULT; \ @@ -824,6 +881,7 @@ static const char *esr_class_str[] = { [ESR_ELx_EC_DABT_LOW] = "DABT (lower EL)", [ESR_ELx_EC_DABT_CUR] = "DABT (current EL)", [ESR_ELx_EC_SP_ALIGN] = "SP Alignment", + [ESR_ELx_EC_MOPS] = "MOPS", [ESR_ELx_EC_FP_EXC32] = "FP (AArch32)", [ESR_ELx_EC_FP_EXC64] = "FP (AArch64)", [ESR_ELx_EC_SERROR] = "SError", @@ -947,7 +1005,7 @@ void do_serror(struct pt_regs *regs, unsigned long esr) } /* GENERIC_BUG traps */ - +#ifdef CONFIG_GENERIC_BUG int is_valid_bugaddr(unsigned long addr) { /* @@ -959,6 +1017,7 @@ int is_valid_bugaddr(unsigned long addr) */ return 1; } +#endif static int bug_handler(struct pt_regs *regs, unsigned long esr) { diff --git a/arch/arm64/kvm/debug.c b/arch/arm64/kvm/debug.c index 55f80fb93925..8725291cb00a 100644 --- a/arch/arm64/kvm/debug.c +++ b/arch/arm64/kvm/debug.c @@ -333,7 +333,7 @@ void kvm_arch_vcpu_load_debug_state_flags(struct kvm_vcpu *vcpu) /* Check if we have TRBE implemented and available at the host */ if (cpuid_feature_extract_unsigned_field(dfr0, ID_AA64DFR0_EL1_TraceBuffer_SHIFT) && - !(read_sysreg_s(SYS_TRBIDR_EL1) & TRBIDR_PROG)) + !(read_sysreg_s(SYS_TRBIDR_EL1) & TRBIDR_EL1_P)) vcpu_set_flag(vcpu, DEBUG_STATE_SAVE_TRBE); } diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 4fe217efa218..2f6e0b3e4a75 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -141,6 +141,9 @@ static inline void ___activate_traps(struct kvm_vcpu *vcpu) if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN) && (hcr & HCR_VSE)) write_sysreg_s(vcpu->arch.vsesr_el2, SYS_VSESR_EL2); + + if (cpus_have_final_cap(ARM64_HAS_HCX)) + write_sysreg_s(HCRX_GUEST_FLAGS, SYS_HCRX_EL2); } static inline void ___deactivate_traps(struct kvm_vcpu *vcpu) @@ -155,6 +158,9 @@ static inline void ___deactivate_traps(struct kvm_vcpu *vcpu) vcpu->arch.hcr_el2 &= ~HCR_VSE; vcpu->arch.hcr_el2 |= read_sysreg(hcr_el2) & HCR_VSE; } + + if (cpus_have_final_cap(ARM64_HAS_HCX)) + write_sysreg_s(HCRX_HOST_FLAGS, SYS_HCRX_EL2); } static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index 699ea1f8d409..bb6b571ec627 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -44,6 +44,8 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, TTBR0_EL1) = read_sysreg_el1(SYS_TTBR0); ctxt_sys_reg(ctxt, TTBR1_EL1) = read_sysreg_el1(SYS_TTBR1); ctxt_sys_reg(ctxt, TCR_EL1) = read_sysreg_el1(SYS_TCR); + if (cpus_have_final_cap(ARM64_HAS_TCR2)) + ctxt_sys_reg(ctxt, TCR2_EL1) = read_sysreg_el1(SYS_TCR2); ctxt_sys_reg(ctxt, ESR_EL1) = read_sysreg_el1(SYS_ESR); ctxt_sys_reg(ctxt, AFSR0_EL1) = read_sysreg_el1(SYS_AFSR0); ctxt_sys_reg(ctxt, AFSR1_EL1) = read_sysreg_el1(SYS_AFSR1); @@ -53,6 +55,10 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, CONTEXTIDR_EL1) = read_sysreg_el1(SYS_CONTEXTIDR); ctxt_sys_reg(ctxt, AMAIR_EL1) = read_sysreg_el1(SYS_AMAIR); ctxt_sys_reg(ctxt, CNTKCTL_EL1) = read_sysreg_el1(SYS_CNTKCTL); + if (cpus_have_final_cap(ARM64_HAS_S1PIE)) { + ctxt_sys_reg(ctxt, PIR_EL1) = read_sysreg_el1(SYS_PIR); + ctxt_sys_reg(ctxt, PIRE0_EL1) = read_sysreg_el1(SYS_PIRE0); + } ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg_par(); ctxt_sys_reg(ctxt, TPIDR_EL1) = read_sysreg(tpidr_el1); @@ -114,6 +120,8 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) write_sysreg_el1(ctxt_sys_reg(ctxt, CPACR_EL1), SYS_CPACR); write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL1), SYS_TTBR0); write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL1), SYS_TTBR1); + if (cpus_have_final_cap(ARM64_HAS_TCR2)) + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR2_EL1), SYS_TCR2); write_sysreg_el1(ctxt_sys_reg(ctxt, ESR_EL1), SYS_ESR); write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR0_EL1), SYS_AFSR0); write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR1_EL1), SYS_AFSR1); @@ -123,6 +131,10 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) write_sysreg_el1(ctxt_sys_reg(ctxt, CONTEXTIDR_EL1), SYS_CONTEXTIDR); write_sysreg_el1(ctxt_sys_reg(ctxt, AMAIR_EL1), SYS_AMAIR); write_sysreg_el1(ctxt_sys_reg(ctxt, CNTKCTL_EL1), SYS_CNTKCTL); + if (cpus_have_final_cap(ARM64_HAS_S1PIE)) { + write_sysreg_el1(ctxt_sys_reg(ctxt, PIR_EL1), SYS_PIR); + write_sysreg_el1(ctxt_sys_reg(ctxt, PIRE0_EL1), SYS_PIRE0); + } write_sysreg(ctxt_sys_reg(ctxt, PAR_EL1), par_el1); write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL1), tpidr_el1); diff --git a/arch/arm64/kvm/hyp/nvhe/debug-sr.c b/arch/arm64/kvm/hyp/nvhe/debug-sr.c index d756b939f296..4558c02eb352 100644 --- a/arch/arm64/kvm/hyp/nvhe/debug-sr.c +++ b/arch/arm64/kvm/hyp/nvhe/debug-sr.c @@ -56,7 +56,7 @@ static void __debug_save_trace(u64 *trfcr_el1) *trfcr_el1 = 0; /* Check if the TRBE is enabled */ - if (!(read_sysreg_s(SYS_TRBLIMITR_EL1) & TRBLIMITR_ENABLE)) + if (!(read_sysreg_s(SYS_TRBLIMITR_EL1) & TRBLIMITR_EL1_E)) return; /* * Prohibit trace generation while we are in guest. diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 753aa7418149..5b5d5e5449dc 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -401,9 +401,9 @@ static bool trap_oslar_el1(struct kvm_vcpu *vcpu, return read_from_write_only(vcpu, p, r); /* Forward the OSLK bit to OSLSR */ - oslsr = __vcpu_sys_reg(vcpu, OSLSR_EL1) & ~SYS_OSLSR_OSLK; - if (p->regval & SYS_OSLAR_OSLK) - oslsr |= SYS_OSLSR_OSLK; + oslsr = __vcpu_sys_reg(vcpu, OSLSR_EL1) & ~OSLSR_EL1_OSLK; + if (p->regval & OSLAR_EL1_OSLK) + oslsr |= OSLSR_EL1_OSLK; __vcpu_sys_reg(vcpu, OSLSR_EL1) = oslsr; return true; @@ -427,7 +427,7 @@ static int set_oslsr_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, * The only modifiable bit is the OSLK bit. Refuse the write if * userspace attempts to change any other bit in the register. */ - if ((val ^ rd->val) & ~SYS_OSLSR_OSLK) + if ((val ^ rd->val) & ~OSLSR_EL1_OSLK) return -EINVAL; __vcpu_sys_reg(vcpu, rd->reg) = val; @@ -1265,6 +1265,7 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu, struct sys_reg_desc const *r ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_GPA3)); if (!cpus_have_final_cap(ARM64_HAS_WFXT)) val &= ~ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_WFxT); + val &= ~ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_MOPS); break; case SYS_ID_AA64DFR0_EL1: /* Limit debug to ARMv8.0 */ @@ -1800,7 +1801,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_MDRAR_EL1), trap_raz_wi }, { SYS_DESC(SYS_OSLAR_EL1), trap_oslar_el1 }, { SYS_DESC(SYS_OSLSR_EL1), trap_oslsr_el1, reset_val, OSLSR_EL1, - SYS_OSLSR_OSLM_IMPLEMENTED, .set_user = set_oslsr_el1, }, + OSLSR_EL1_OSLM_IMPLEMENTED, .set_user = set_oslsr_el1, }, { SYS_DESC(SYS_OSDLR_EL1), trap_raz_wi }, { SYS_DESC(SYS_DBGPRCR_EL1), trap_raz_wi }, { SYS_DESC(SYS_DBGCLAIMSET_EL1), trap_raz_wi }, @@ -1891,7 +1892,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { ID_SANITISED(ID_AA64MMFR0_EL1), ID_SANITISED(ID_AA64MMFR1_EL1), ID_SANITISED(ID_AA64MMFR2_EL1), - ID_UNALLOCATED(7,3), + ID_SANITISED(ID_AA64MMFR3_EL1), ID_UNALLOCATED(7,4), ID_UNALLOCATED(7,5), ID_UNALLOCATED(7,6), @@ -1911,6 +1912,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_TTBR0_EL1), access_vm_reg, reset_unknown, TTBR0_EL1 }, { SYS_DESC(SYS_TTBR1_EL1), access_vm_reg, reset_unknown, TTBR1_EL1 }, { SYS_DESC(SYS_TCR_EL1), access_vm_reg, reset_val, TCR_EL1, 0 }, + { SYS_DESC(SYS_TCR2_EL1), access_vm_reg, reset_val, TCR2_EL1, 0 }, PTRAUTH_KEY(APIA), PTRAUTH_KEY(APIB), @@ -1960,6 +1962,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_PMMIR_EL1), trap_raz_wi }, { SYS_DESC(SYS_MAIR_EL1), access_vm_reg, reset_unknown, MAIR_EL1 }, + { SYS_DESC(SYS_PIRE0_EL1), access_vm_reg, reset_unknown, PIRE0_EL1 }, + { SYS_DESC(SYS_PIR_EL1), access_vm_reg, reset_unknown, PIR_EL1 }, { SYS_DESC(SYS_AMAIR_EL1), access_vm_reg, reset_amair_el1, AMAIR_EL1 }, { SYS_DESC(SYS_LORSA_EL1), trap_loregion }, diff --git a/arch/arm64/lib/xor-neon.c b/arch/arm64/lib/xor-neon.c index 96b171995d19..f9a53b7f9842 100644 --- a/arch/arm64/lib/xor-neon.c +++ b/arch/arm64/lib/xor-neon.c @@ -10,7 +10,7 @@ #include #include -void xor_arm64_neon_2(unsigned long bytes, unsigned long * __restrict p1, +static void xor_arm64_neon_2(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2) { uint64_t *dp1 = (uint64_t *)p1; @@ -37,7 +37,7 @@ void xor_arm64_neon_2(unsigned long bytes, unsigned long * __restrict p1, } while (--lines > 0); } -void xor_arm64_neon_3(unsigned long bytes, unsigned long * __restrict p1, +static void xor_arm64_neon_3(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2, const unsigned long * __restrict p3) { @@ -73,7 +73,7 @@ void xor_arm64_neon_3(unsigned long bytes, unsigned long * __restrict p1, } while (--lines > 0); } -void xor_arm64_neon_4(unsigned long bytes, unsigned long * __restrict p1, +static void xor_arm64_neon_4(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2, const unsigned long * __restrict p3, const unsigned long * __restrict p4) @@ -118,7 +118,7 @@ void xor_arm64_neon_4(unsigned long bytes, unsigned long * __restrict p1, } while (--lines > 0); } -void xor_arm64_neon_5(unsigned long bytes, unsigned long * __restrict p1, +static void xor_arm64_neon_5(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2, const unsigned long * __restrict p3, const unsigned long * __restrict p4, diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index e1e0dca01839..188197590fc9 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -364,8 +364,8 @@ void cpu_do_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm) ttbr1 &= ~TTBR_ASID_MASK; ttbr1 |= FIELD_PREP(TTBR_ASID_MASK, asid); + cpu_set_reserved_ttbr0_nosync(); write_sysreg(ttbr1, ttbr1_el1); - isb(); write_sysreg(ttbr0, ttbr0_el1); isb(); post_ttbr_update_workaround(); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 6045a5117ac1..c85b6d70b222 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -66,6 +66,8 @@ static inline const struct fault_info *esr_to_debug_fault_info(unsigned long esr static void data_abort_decode(unsigned long esr) { + unsigned long iss2 = ESR_ELx_ISS2(esr); + pr_alert("Data abort info:\n"); if (esr & ESR_ELx_ISV) { @@ -78,12 +80,21 @@ static void data_abort_decode(unsigned long esr) (esr & ESR_ELx_SF) >> ESR_ELx_SF_SHIFT, (esr & ESR_ELx_AR) >> ESR_ELx_AR_SHIFT); } else { - pr_alert(" ISV = 0, ISS = 0x%08lx\n", esr & ESR_ELx_ISS_MASK); + pr_alert(" ISV = 0, ISS = 0x%08lx, ISS2 = 0x%08lx\n", + esr & ESR_ELx_ISS_MASK, iss2); } - pr_alert(" CM = %lu, WnR = %lu\n", + pr_alert(" CM = %lu, WnR = %lu, TnD = %lu, TagAccess = %lu\n", (esr & ESR_ELx_CM) >> ESR_ELx_CM_SHIFT, - (esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT); + (esr & ESR_ELx_WNR) >> ESR_ELx_WNR_SHIFT, + (iss2 & ESR_ELx_TnD) >> ESR_ELx_TnD_SHIFT, + (iss2 & ESR_ELx_TagAccess) >> ESR_ELx_TagAccess_SHIFT); + + pr_alert(" GCS = %ld, Overlay = %lu, DirtyBit = %lu, Xs = %llu\n", + (iss2 & ESR_ELx_GCS) >> ESR_ELx_GCS_SHIFT, + (iss2 & ESR_ELx_Overlay) >> ESR_ELx_Overlay_SHIFT, + (iss2 & ESR_ELx_DirtyBit) >> ESR_ELx_DirtyBit_SHIFT, + (iss2 & ESR_ELx_Xs_MASK) >> ESR_ELx_Xs_SHIFT); } static void mem_abort_decode(unsigned long esr) @@ -885,9 +896,6 @@ void do_sp_pc_abort(unsigned long addr, unsigned long esr, struct pt_regs *regs) } NOKPROBE_SYMBOL(do_sp_pc_abort); -int __init early_brk64(unsigned long addr, unsigned long esr, - struct pt_regs *regs); - /* * __refdata because early_brk64 is __init, but the reference to it is * clobbered at arch_initcall time. diff --git a/arch/arm64/mm/flush.c b/arch/arm64/mm/flush.c index 5f9379b3c8c8..4e6476094952 100644 --- a/arch/arm64/mm/flush.c +++ b/arch/arm64/mm/flush.c @@ -8,6 +8,7 @@ #include #include +#include #include #include diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 66e70ca47680..c28c2c8483cc 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -69,6 +69,7 @@ phys_addr_t __ro_after_init arm64_dma_phys_limit; #define CRASH_ADDR_LOW_MAX arm64_dma_phys_limit #define CRASH_ADDR_HIGH_MAX (PHYS_MASK + 1) +#define CRASH_HIGH_SEARCH_BASE SZ_4G #define DEFAULT_CRASH_KERNEL_LOW_SIZE (128UL << 20) @@ -101,12 +102,13 @@ static int __init reserve_crashkernel_low(unsigned long long low_size) */ static void __init reserve_crashkernel(void) { - unsigned long long crash_base, crash_size; - unsigned long long crash_low_size = 0; + unsigned long long crash_low_size = 0, search_base = 0; unsigned long long crash_max = CRASH_ADDR_LOW_MAX; + unsigned long long crash_base, crash_size; char *cmdline = boot_command_line; - int ret; bool fixed_base = false; + bool high = false; + int ret; if (!IS_ENABLED(CONFIG_KEXEC_CORE)) return; @@ -129,7 +131,9 @@ static void __init reserve_crashkernel(void) else if (ret) return; + search_base = CRASH_HIGH_SEARCH_BASE; crash_max = CRASH_ADDR_HIGH_MAX; + high = true; } else if (ret || !crash_size) { /* The specified value is invalid */ return; @@ -140,31 +144,51 @@ static void __init reserve_crashkernel(void) /* User specifies base address explicitly. */ if (crash_base) { fixed_base = true; + search_base = crash_base; crash_max = crash_base + crash_size; } retry: crash_base = memblock_phys_alloc_range(crash_size, CRASH_ALIGN, - crash_base, crash_max); + search_base, crash_max); if (!crash_base) { /* - * If the first attempt was for low memory, fall back to - * high memory, the minimum required low memory will be - * reserved later. + * For crashkernel=size[KMG]@offset[KMG], print out failure + * message if can't reserve the specified region. */ - if (!fixed_base && (crash_max == CRASH_ADDR_LOW_MAX)) { + if (fixed_base) { + pr_warn("crashkernel reservation failed - memory is in use.\n"); + return; + } + + /* + * For crashkernel=size[KMG], if the first attempt was for + * low memory, fall back to high memory, the minimum required + * low memory will be reserved later. + */ + if (!high && crash_max == CRASH_ADDR_LOW_MAX) { crash_max = CRASH_ADDR_HIGH_MAX; + search_base = CRASH_ADDR_LOW_MAX; crash_low_size = DEFAULT_CRASH_KERNEL_LOW_SIZE; goto retry; } + /* + * For crashkernel=size[KMG],high, if the first attempt was + * for high memory, fall back to low memory. + */ + if (high && crash_max == CRASH_ADDR_HIGH_MAX) { + crash_max = CRASH_ADDR_LOW_MAX; + search_base = 0; + goto retry; + } pr_warn("cannot allocate crashkernel (size:0x%llx)\n", crash_size); return; } - if ((crash_base > CRASH_ADDR_LOW_MAX - crash_low_size) && - crash_low_size && reserve_crashkernel_low(crash_low_size)) { + if ((crash_base >= CRASH_ADDR_LOW_MAX) && crash_low_size && + reserve_crashkernel_low(crash_low_size)) { memblock_phys_free(crash_base, crash_size); return; } diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index e969e68de005..f17d066e85eb 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -214,7 +214,7 @@ static void __init clear_pgds(unsigned long start, static void __init kasan_init_shadow(void) { u64 kimg_shadow_start, kimg_shadow_end; - u64 mod_shadow_start, mod_shadow_end; + u64 mod_shadow_start; u64 vmalloc_shadow_end; phys_addr_t pa_start, pa_end; u64 i; @@ -223,7 +223,6 @@ static void __init kasan_init_shadow(void) kimg_shadow_end = PAGE_ALIGN((u64)kasan_mem_to_shadow(KERNEL_END)); mod_shadow_start = (u64)kasan_mem_to_shadow((void *)MODULES_VADDR); - mod_shadow_end = (u64)kasan_mem_to_shadow((void *)MODULES_END); vmalloc_shadow_end = (u64)kasan_mem_to_shadow((void *)VMALLOC_END); @@ -246,17 +245,9 @@ static void __init kasan_init_shadow(void) kasan_populate_early_shadow(kasan_mem_to_shadow((void *)PAGE_END), (void *)mod_shadow_start); - if (IS_ENABLED(CONFIG_KASAN_VMALLOC)) { - BUILD_BUG_ON(VMALLOC_START != MODULES_END); - kasan_populate_early_shadow((void *)vmalloc_shadow_end, - (void *)KASAN_SHADOW_END); - } else { - kasan_populate_early_shadow((void *)kimg_shadow_end, - (void *)KASAN_SHADOW_END); - if (kimg_shadow_start > mod_shadow_end) - kasan_populate_early_shadow((void *)mod_shadow_end, - (void *)kimg_shadow_start); - } + BUILD_BUG_ON(VMALLOC_START != MODULES_END); + kasan_populate_early_shadow((void *)vmalloc_shadow_end, + (void *)KASAN_SHADOW_END); for_each_mem_range(i, &pa_start, &pa_end) { void *start = (void *)__phys_to_virt(pa_start); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index af6bc8403ee4..95d360805f8a 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -451,7 +451,7 @@ static phys_addr_t pgd_pgtable_alloc(int shift) void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot) { - if ((virt >= PAGE_END) && (virt < VMALLOC_START)) { + if (virt < PAGE_OFFSET) { pr_warn("BUG: not creating mapping for %pa at 0x%016lx - outside kernel range\n", &phys, virt); return; @@ -478,7 +478,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, static void update_mapping_prot(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot) { - if ((virt >= PAGE_END) && (virt < VMALLOC_START)) { + if (virt < PAGE_OFFSET) { pr_warn("BUG: not updating mapping for %pa at 0x%016lx - outside kernel range\n", &phys, virt); return; @@ -663,12 +663,17 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, vm_area_add_early(vma); } +static pgprot_t kernel_exec_prot(void) +{ + return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; +} + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static int __init map_entry_trampoline(void) { int i; - pgprot_t prot = rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; + pgprot_t prot = kernel_exec_prot(); phys_addr_t pa_start = __pa_symbol(__entry_tramp_text_start); /* The trampoline is always mapped and can therefore be global */ @@ -723,7 +728,7 @@ static void __init map_kernel(pgd_t *pgdp) * mapping to install SW breakpoints. Allow this (only) when * explicitly requested with rodata=off. */ - pgprot_t text_prot = rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; + pgprot_t text_prot = kernel_exec_prot(); /* * If we have a CPU that supports BTI and a kernel built for diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index c2cb437821ca..2baeec419f62 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -199,7 +199,7 @@ SYM_FUNC_END(idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS) +#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) .pushsection ".idmap.text", "a" @@ -290,7 +290,7 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) isb mov temp_pte, x5 - mov pte_flags, #KPTI_NG_PTE_FLAGS + mov_q pte_flags, KPTI_NG_PTE_FLAGS /* Everybody is enjoying the idmap, so we can rewrite swapper. */ /* PGD */ @@ -454,6 +454,21 @@ SYM_FUNC_START(__cpu_setup) #endif /* CONFIG_ARM64_HW_AFDBM */ msr mair_el1, mair msr tcr_el1, tcr + + mrs_s x1, SYS_ID_AA64MMFR3_EL1 + ubfx x1, x1, #ID_AA64MMFR3_EL1_S1PIE_SHIFT, #4 + cbz x1, .Lskip_indirection + + mov_q x0, PIE_E0 + msr REG_PIRE0_EL1, x0 + mov_q x0, PIE_E1 + msr REG_PIR_EL1, x0 + + mov x0, TCR2_EL1x_PIE + msr REG_TCR2_EL1, x0 + +.Lskip_indirection: + /* * Prepare SCTLR */ diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 40ba95472594..19c23c4fa2da 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -32,16 +32,20 @@ HAS_GENERIC_AUTH_IMP_DEF HAS_GIC_CPUIF_SYSREGS HAS_GIC_PRIO_MASKING HAS_GIC_PRIO_RELAXED_SYNC +HAS_HCX HAS_LDAPR HAS_LSE_ATOMICS +HAS_MOPS HAS_NESTED_VIRT HAS_NO_FPSIMD HAS_NO_HW_PREFETCH HAS_PAN +HAS_S1PIE HAS_RAS_EXTN HAS_RNG HAS_SB HAS_STAGE2_FWB +HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE HAS_VIRT_HOST_EXTN diff --git a/arch/arm64/tools/gen-cpucaps.awk b/arch/arm64/tools/gen-cpucaps.awk index 00c9e72a200a..8525980379d7 100755 --- a/arch/arm64/tools/gen-cpucaps.awk +++ b/arch/arm64/tools/gen-cpucaps.awk @@ -24,12 +24,12 @@ BEGIN { } /^[vA-Z0-9_]+$/ { - printf("#define ARM64_%-30s\t%d\n", $0, cap_num++) + printf("#define ARM64_%-40s\t%d\n", $0, cap_num++) next } END { - printf("#define ARM64_NCAPS\t\t\t\t%d\n", cap_num) + printf("#define ARM64_NCAPS\t\t\t\t\t%d\n", cap_num) print "" print "#endif /* __ASM_CPUCAPS_H */" } diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index c9a0d1fa3209..1ea4a3dc68f8 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -48,6 +48,61 @@ # feature that introduces them (eg, FEAT_LS64_ACCDATA introduces enumeration # item ACCDATA) though it may be more taseful to do something else. +Sysreg OSDTRRX_EL1 2 0 0 0 2 +Res0 63:32 +Field 31:0 DTRRX +EndSysreg + +Sysreg MDCCINT_EL1 2 0 0 2 0 +Res0 63:31 +Field 30 RX +Field 29 TX +Res0 28:0 +EndSysreg + +Sysreg MDSCR_EL1 2 0 0 2 2 +Res0 63:36 +Field 35 EHBWE +Field 34 EnSPM +Field 33 TTA +Field 32 EMBWE +Field 31 TFO +Field 30 RXfull +Field 29 TXfull +Res0 28 +Field 27 RXO +Field 26 TXU +Res0 25:24 +Field 23:22 INTdis +Field 21 TDA +Res0 20 +Field 19 SC2 +Res0 18:16 +Field 15 MDE +Field 14 HDE +Field 13 KDE +Field 12 TDCC +Res0 11:7 +Field 6 ERR +Res0 5:1 +Field 0 SS +EndSysreg + +Sysreg OSDTRTX_EL1 2 0 0 3 2 +Res0 63:32 +Field 31:0 DTRTX +EndSysreg + +Sysreg OSECCR_EL1 2 0 0 6 2 +Res0 63:32 +Field 31:0 EDECCR +EndSysreg + +Sysreg OSLAR_EL1 2 0 1 0 4 +Res0 63:1 +Field 0 OSLK +EndSysreg + Sysreg ID_PFR0_EL1 3 0 0 1 0 Res0 63:32 UnsignedEnum 31:28 RAS @@ -1538,6 +1593,78 @@ UnsignedEnum 3:0 CnP EndEnum EndSysreg +Sysreg ID_AA64MMFR3_EL1 3 0 0 7 3 +UnsignedEnum 63:60 Spec_FPACC + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 59:56 ADERR + 0b0000 NI + 0b0001 DEV_ASYNC + 0b0010 FEAT_ADERR + 0b0011 FEAT_ADERR_IND +EndEnum +UnsignedEnum 55:52 SDERR + 0b0000 NI + 0b0001 DEV_SYNC + 0b0010 FEAT_ADERR + 0b0011 FEAT_ADERR_IND +EndEnum +Res0 51:48 +UnsignedEnum 47:44 ANERR + 0b0000 NI + 0b0001 ASYNC + 0b0010 FEAT_ANERR + 0b0011 FEAT_ANERR_IND +EndEnum +UnsignedEnum 43:40 SNERR + 0b0000 NI + 0b0001 SYNC + 0b0010 FEAT_ANERR + 0b0011 FEAT_ANERR_IND +EndEnum +UnsignedEnum 39:36 D128_2 + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 35:32 D128 + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 31:28 MEC + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 27:24 AIE + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 23:20 S2POE + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 19:16 S1POE + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 15:12 S2PIE + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 11:8 S1PIE + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 7:4 SCTLRX + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 3:0 TCRX + 0b0000 NI + 0b0001 IMP +EndEnum +EndSysreg + Sysreg SCTLR_EL1 3 0 1 0 0 Field 63 TIDCP Field 62 SPINTMASK @@ -2034,7 +2161,17 @@ Fields ZCR_ELx EndSysreg Sysreg HCRX_EL2 3 4 1 2 2 -Res0 63:12 +Res0 63:23 +Field 22 GCSEn +Field 21 EnIDCP128 +Field 20 EnSDERR +Field 19 TMEA +Field 18 EnSNERR +Field 17 D128En +Field 16 PTTWI +Field 15 SCTLR2En +Field 14 TCR2En +Res0 13:12 Field 11 MSCEn Field 10 MCE2 Field 9 CMOW @@ -2153,6 +2290,87 @@ Sysreg TTBR1_EL1 3 0 2 0 1 Fields TTBRx_EL1 EndSysreg +SysregFields TCR2_EL1x +Res0 63:16 +Field 15 DisCH1 +Field 14 DisCH0 +Res0 13:12 +Field 11 HAFT +Field 10 PTTWI +Res0 9:6 +Field 5 D128 +Field 4 AIE +Field 3 POE +Field 2 E0POE +Field 1 PIE +Field 0 PnCH +EndSysregFields + +Sysreg TCR2_EL1 3 0 2 0 3 +Fields TCR2_EL1x +EndSysreg + +Sysreg TCR2_EL12 3 5 2 0 3 +Fields TCR2_EL1x +EndSysreg + +Sysreg TCR2_EL2 3 4 2 0 3 +Res0 63:16 +Field 15 DisCH1 +Field 14 DisCH0 +Field 13 AMEC1 +Field 12 AMEC0 +Field 11 HAFT +Field 10 PTTWI +Field 9:8 SKL1 +Field 7:6 SKL0 +Field 5 D128 +Field 4 AIE +Field 3 POE +Field 2 E0POE +Field 1 PIE +Field 0 PnCH +EndSysreg + +SysregFields PIRx_ELx +Field 63:60 Perm15 +Field 59:56 Perm14 +Field 55:52 Perm13 +Field 51:48 Perm12 +Field 47:44 Perm11 +Field 43:40 Perm10 +Field 39:36 Perm9 +Field 35:32 Perm8 +Field 31:28 Perm7 +Field 27:24 Perm6 +Field 23:20 Perm5 +Field 19:16 Perm4 +Field 15:12 Perm3 +Field 11:8 Perm2 +Field 7:4 Perm1 +Field 3:0 Perm0 +EndSysregFields + +Sysreg PIRE0_EL1 3 0 10 2 2 +Fields PIRx_ELx +EndSysreg + +Sysreg PIRE0_EL12 3 5 10 2 2 +Fields PIRx_ELx +EndSysreg + +Sysreg PIR_EL1 3 0 10 2 3 +Fields PIRx_ELx +EndSysreg + +Sysreg PIR_EL12 3 5 10 2 3 +Fields PIRx_ELx +EndSysreg + +Sysreg PIR_EL2 3 4 10 2 3 +Fields PIRx_ELx +EndSysreg + Sysreg LORSA_EL1 3 0 10 4 0 Res0 63:52 Field 51:16 SA @@ -2200,3 +2418,80 @@ Sysreg ICC_NMIAR1_EL1 3 0 12 9 5 Res0 63:24 Field 23:0 INTID EndSysreg + +Sysreg TRBLIMITR_EL1 3 0 9 11 0 +Field 63:12 LIMIT +Res0 11:7 +Field 6 XE +Field 5 nVM +Enum 4:3 TM + 0b00 STOP + 0b01 IRQ + 0b11 IGNR +EndEnum +Enum 2:1 FM + 0b00 FILL + 0b01 WRAP + 0b11 CBUF +EndEnum +Field 0 E +EndSysreg + +Sysreg TRBPTR_EL1 3 0 9 11 1 +Field 63:0 PTR +EndSysreg + +Sysreg TRBBASER_EL1 3 0 9 11 2 +Field 63:12 BASE +Res0 11:0 +EndSysreg + +Sysreg TRBSR_EL1 3 0 9 11 3 +Res0 63:56 +Field 55:32 MSS2 +Field 31:26 EC +Res0 25:24 +Field 23 DAT +Field 22 IRQ +Field 21 TRG +Field 20 WRAP +Res0 19 +Field 18 EA +Field 17 S +Res0 16 +Field 15:0 MSS +EndSysreg + +Sysreg TRBMAR_EL1 3 0 9 11 4 +Res0 63:12 +Enum 11:10 PAS + 0b00 SECURE + 0b01 NON_SECURE + 0b10 ROOT + 0b11 REALM +EndEnum +Enum 9:8 SH + 0b00 NON_SHAREABLE + 0b10 OUTER_SHAREABLE + 0b11 INNER_SHAREABLE +EndEnum +Field 7:0 Attr +EndSysreg + +Sysreg TRBTRG_EL1 3 0 9 11 6 +Res0 63:32 +Field 31:0 TRG +EndSysreg + +Sysreg TRBIDR_EL1 3 0 9 11 7 +Res0 63:12 +Enum 11:8 EA + 0b0000 NON_DESC + 0b0001 IGNORE + 0b0010 SERROR +EndEnum +Res0 7:6 +Field 5 F +Field 4 P +Field 3:0 Align +EndSysreg diff --git a/arch/mips/bmips/setup.c b/arch/mips/bmips/setup.c index 549a6392a3d2..053805cb741c 100644 --- a/arch/mips/bmips/setup.c +++ b/arch/mips/bmips/setup.c @@ -178,7 +178,10 @@ void __init plat_mem_setup(void) ioport_resource.start = 0; ioport_resource.end = ~0; - /* intended to somewhat resemble ARM; see Documentation/arm/booting.rst */ + /* + * intended to somewhat resemble ARM; see + * Documentation/arch/arm/booting.rst + */ if (fw_arg0 == 0 && fw_arg1 == 0xffffffff) dtb = phys_to_virt(fw_arg2); else diff --git a/arch/x86/coco/core.c b/arch/x86/coco/core.c index 73f83233d25d..eeec9986570e 100644 --- a/arch/x86/coco/core.c +++ b/arch/x86/coco/core.c @@ -13,10 +13,10 @@ #include #include -enum cc_vendor cc_vendor __ro_after_init; +enum cc_vendor cc_vendor __ro_after_init = CC_VENDOR_NONE; static u64 cc_mask __ro_after_init; -static bool intel_cc_platform_has(enum cc_attr attr) +static bool noinstr intel_cc_platform_has(enum cc_attr attr) { switch (attr) { case CC_ATTR_GUEST_UNROLL_STRING_IO: @@ -34,7 +34,7 @@ static bool intel_cc_platform_has(enum cc_attr attr) * the other levels of SME/SEV functionality, including C-bit * based SEV-SNP, are not enabled. */ -static __maybe_unused bool amd_cc_platform_vtom(enum cc_attr attr) +static __maybe_unused __always_inline bool amd_cc_platform_vtom(enum cc_attr attr) { switch (attr) { case CC_ATTR_GUEST_MEM_ENCRYPT: @@ -58,7 +58,7 @@ static __maybe_unused bool amd_cc_platform_vtom(enum cc_attr attr) * the trampoline area must be encrypted. */ -static bool amd_cc_platform_has(enum cc_attr attr) +static bool noinstr amd_cc_platform_has(enum cc_attr attr) { #ifdef CONFIG_AMD_MEM_ENCRYPT @@ -97,7 +97,7 @@ static bool amd_cc_platform_has(enum cc_attr attr) #endif } -bool cc_platform_has(enum cc_attr attr) +bool noinstr cc_platform_has(enum cc_attr attr) { switch (cc_vendor) { case CC_VENDOR_AMD: diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index 0e2a16e7d685..1d6b863c42b0 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -769,7 +769,7 @@ void __init tdx_early_init(void) setup_force_cpu_cap(X86_FEATURE_TDX_GUEST); - cc_set_vendor(CC_VENDOR_INTEL); + cc_vendor = CC_VENDOR_INTEL; tdx_parse_tdinfo(&cc_mask); cc_set_mask(cc_mask); diff --git a/arch/x86/hyperv/ivm.c b/arch/x86/hyperv/ivm.c index cc92388b7a99..14f46ad2ca64 100644 --- a/arch/x86/hyperv/ivm.c +++ b/arch/x86/hyperv/ivm.c @@ -17,6 +17,7 @@ #include #include #include +#include #ifdef CONFIG_AMD_MEM_ENCRYPT @@ -364,7 +365,7 @@ void __init hv_vtom_init(void) * Set it here to indicate a vTOM VM. */ sev_status = MSR_AMD64_SNP_VTOM; - cc_set_vendor(CC_VENDOR_AMD); + cc_vendor = CC_VENDOR_AMD; cc_set_mask(ms_hyperv.shared_gpa_boundary); physical_mask &= ms_hyperv.shared_gpa_boundary - 1; @@ -372,6 +373,9 @@ void __init hv_vtom_init(void) x86_platform.guest.enc_cache_flush_required = hv_vtom_cache_flush_required; x86_platform.guest.enc_tlb_flush_required = hv_vtom_tlb_flush_required; x86_platform.guest.enc_status_change_finish = hv_vtom_set_host_visibility; + + /* Set WB as the default cache mode. */ + mtrr_overwrite_state(NULL, 0, MTRR_TYPE_WRBACK); } #endif /* CONFIG_AMD_MEM_ENCRYPT */ diff --git a/arch/x86/include/asm/coco.h b/arch/x86/include/asm/coco.h index eb08796002f3..6ae2d16a7613 100644 --- a/arch/x86/include/asm/coco.h +++ b/arch/x86/include/asm/coco.h @@ -10,30 +10,13 @@ enum cc_vendor { CC_VENDOR_INTEL, }; -#ifdef CONFIG_ARCH_HAS_CC_PLATFORM extern enum cc_vendor cc_vendor; -static inline enum cc_vendor cc_get_vendor(void) -{ - return cc_vendor; -} - -static inline void cc_set_vendor(enum cc_vendor vendor) -{ - cc_vendor = vendor; -} - +#ifdef CONFIG_ARCH_HAS_CC_PLATFORM void cc_set_mask(u64 mask); u64 cc_mkenc(u64 val); u64 cc_mkdec(u64 val); #else -static inline enum cc_vendor cc_get_vendor(void) -{ - return CC_VENDOR_NONE; -} - -static inline void cc_set_vendor(enum cc_vendor vendor) { } - static inline u64 cc_mkenc(u64 val) { return val; diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h index f0eeaf6e5f5f..090d658a85a6 100644 --- a/arch/x86/include/asm/mtrr.h +++ b/arch/x86/include/asm/mtrr.h @@ -23,14 +23,43 @@ #ifndef _ASM_X86_MTRR_H #define _ASM_X86_MTRR_H +#include #include +/* Defines for hardware MTRR registers. */ +#define MTRR_CAP_VCNT GENMASK(7, 0) +#define MTRR_CAP_FIX BIT_MASK(8) +#define MTRR_CAP_WC BIT_MASK(10) + +#define MTRR_DEF_TYPE_TYPE GENMASK(7, 0) +#define MTRR_DEF_TYPE_FE BIT_MASK(10) +#define MTRR_DEF_TYPE_E BIT_MASK(11) + +#define MTRR_DEF_TYPE_ENABLE (MTRR_DEF_TYPE_FE | MTRR_DEF_TYPE_E) +#define MTRR_DEF_TYPE_DISABLE ~(MTRR_DEF_TYPE_TYPE | MTRR_DEF_TYPE_ENABLE) + +#define MTRR_PHYSBASE_TYPE GENMASK(7, 0) +#define MTRR_PHYSBASE_RSVD GENMASK(11, 8) + +#define MTRR_PHYSMASK_RSVD GENMASK(10, 0) +#define MTRR_PHYSMASK_V BIT_MASK(11) + +struct mtrr_state_type { + struct mtrr_var_range var_ranges[MTRR_MAX_VAR_RANGES]; + mtrr_type fixed_ranges[MTRR_NUM_FIXED_RANGES]; + unsigned char enabled; + bool have_fixed; + mtrr_type def_type; +}; + /* * The following functions are for use by other drivers that cannot use * arch_phys_wc_add and arch_phys_wc_del. */ # ifdef CONFIG_MTRR void mtrr_bp_init(void); +void mtrr_overwrite_state(struct mtrr_var_range *var, unsigned int num_var, + mtrr_type def_type); extern u8 mtrr_type_lookup(u64 addr, u64 end, u8 *uniform); extern void mtrr_save_fixed_ranges(void *); extern void mtrr_save_state(void); @@ -40,7 +69,6 @@ extern int mtrr_add_page(unsigned long base, unsigned long size, unsigned int type, bool increment); extern int mtrr_del(int reg, unsigned long base, unsigned long size); extern int mtrr_del_page(int reg, unsigned long base, unsigned long size); -extern void mtrr_centaur_report_mcr(int mcr, u32 lo, u32 hi); extern void mtrr_bp_restore(void); extern int mtrr_trim_uncached_memory(unsigned long end_pfn); extern int amd_special_default_mtrr(void); @@ -48,12 +76,21 @@ void mtrr_disable(void); void mtrr_enable(void); void mtrr_generic_set_state(void); # else +static inline void mtrr_overwrite_state(struct mtrr_var_range *var, + unsigned int num_var, + mtrr_type def_type) +{ +} + static inline u8 mtrr_type_lookup(u64 addr, u64 end, u8 *uniform) { /* - * Return no-MTRRs: + * Return the default MTRR type, without any known other types in + * that range. */ - return MTRR_TYPE_INVALID; + *uniform = 1; + + return MTRR_TYPE_UNCACHABLE; } #define mtrr_save_fixed_ranges(arg) do {} while (0) #define mtrr_save_state() do {} while (0) @@ -79,9 +116,6 @@ static inline int mtrr_trim_uncached_memory(unsigned long end_pfn) { return 0; } -static inline void mtrr_centaur_report_mcr(int mcr, u32 lo, u32 hi) -{ -} #define mtrr_bp_init() do {} while (0) #define mtrr_bp_restore() do {} while (0) #define mtrr_disable() do {} while (0) @@ -121,7 +155,8 @@ struct mtrr_gentry32 { #endif /* CONFIG_COMPAT */ /* Bit fields for enabled in struct mtrr_state_type */ -#define MTRR_STATE_MTRR_FIXED_ENABLED 0x01 -#define MTRR_STATE_MTRR_ENABLED 0x02 +#define MTRR_STATE_SHIFT 10 +#define MTRR_STATE_MTRR_FIXED_ENABLED (MTRR_DEF_TYPE_FE >> MTRR_STATE_SHIFT) +#define MTRR_STATE_MTRR_ENABLED (MTRR_DEF_TYPE_E >> MTRR_STATE_SHIFT) #endif /* _ASM_X86_MTRR_H */ diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index 86e1296e87f5..66c806784c52 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -14,6 +14,7 @@ #include #include #include +#include #define GHCB_PROTOCOL_MIN 1ULL #define GHCB_PROTOCOL_MAX 2ULL @@ -140,24 +141,26 @@ struct snp_secrets_page_layout { } __packed; #ifdef CONFIG_AMD_MEM_ENCRYPT -extern struct static_key_false sev_es_enable_key; extern void __sev_es_ist_enter(struct pt_regs *regs); extern void __sev_es_ist_exit(void); static __always_inline void sev_es_ist_enter(struct pt_regs *regs) { - if (static_branch_unlikely(&sev_es_enable_key)) + if (cc_vendor == CC_VENDOR_AMD && + cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) __sev_es_ist_enter(regs); } static __always_inline void sev_es_ist_exit(void) { - if (static_branch_unlikely(&sev_es_enable_key)) + if (cc_vendor == CC_VENDOR_AMD && + cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) __sev_es_ist_exit(); } extern int sev_es_setup_ap_jump_table(struct real_mode_header *rmh); extern void __sev_es_nmi_complete(void); static __always_inline void sev_es_nmi_complete(void) { - if (static_branch_unlikely(&sev_es_enable_key)) + if (cc_vendor == CC_VENDOR_AMD && + cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) __sev_es_nmi_complete(); } extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd); diff --git a/arch/x86/include/uapi/asm/mtrr.h b/arch/x86/include/uapi/asm/mtrr.h index 376563f2bac1..3a8a8eb8ac3a 100644 --- a/arch/x86/include/uapi/asm/mtrr.h +++ b/arch/x86/include/uapi/asm/mtrr.h @@ -81,14 +81,6 @@ typedef __u8 mtrr_type; #define MTRR_NUM_FIXED_RANGES 88 #define MTRR_MAX_VAR_RANGES 256 -struct mtrr_state_type { - struct mtrr_var_range var_ranges[MTRR_MAX_VAR_RANGES]; - mtrr_type fixed_ranges[MTRR_NUM_FIXED_RANGES]; - unsigned char enabled; - unsigned char have_fixed; - mtrr_type def_type; -}; - #define MTRRphysBase_MSR(reg) (0x200 + 2 * (reg)) #define MTRRphysMask_MSR(reg) (0x200 + 2 * (reg) + 1) @@ -115,9 +107,9 @@ struct mtrr_state_type { #define MTRR_NUM_TYPES 7 /* - * Invalid MTRR memory type. mtrr_type_lookup() returns this value when - * MTRRs are disabled. Note, this value is allocated from the reserved - * values (0x7-0xff) of the MTRR memory types. + * Invalid MTRR memory type. No longer used outside of MTRR code. + * Note, this value is allocated from the reserved values (0x7-0xff) of + * the MTRR memory types. */ #define MTRR_TYPE_INVALID 0xff diff --git a/arch/x86/kernel/cpu/microcode/amd.c b/arch/x86/kernel/cpu/microcode/amd.c index f5fdeb1e3606..87208e46f7ed 100644 --- a/arch/x86/kernel/cpu/microcode/amd.c +++ b/arch/x86/kernel/cpu/microcode/amd.c @@ -78,8 +78,6 @@ static u16 find_equiv_id(struct equiv_cpu_table *et, u32 sig) if (sig == e->installed_cpu) return e->equiv_cpu; - - e++; } return 0; } @@ -596,11 +594,6 @@ void reload_ucode_amd(unsigned int cpu) } } } -static u16 __find_equiv_id(unsigned int cpu) -{ - struct ucode_cpu_info *uci = ucode_cpu_info + cpu; - return find_equiv_id(&equiv_table, uci->cpu_sig.sig); -} /* * a small, trivial cache of per-family ucode patches @@ -651,9 +644,11 @@ static void free_cache(void) static struct ucode_patch *find_patch(unsigned int cpu) { + struct ucode_cpu_info *uci = ucode_cpu_info + cpu; u16 equiv_id; - equiv_id = __find_equiv_id(cpu); + + equiv_id = find_equiv_id(&equiv_table, uci->cpu_sig.sig); if (!equiv_id) return NULL; @@ -705,7 +700,7 @@ static enum ucode_state apply_microcode_amd(int cpu) rdmsr(MSR_AMD64_PATCH_LEVEL, rev, dummy); /* need to apply patch? */ - if (rev >= mc_amd->hdr.patch_id) { + if (rev > mc_amd->hdr.patch_id) { ret = UCODE_OK; goto out; } diff --git a/arch/x86/kernel/cpu/mtrr/Makefile b/arch/x86/kernel/cpu/mtrr/Makefile index cc4f9f1cb94c..aee4bc5ad496 100644 --- a/arch/x86/kernel/cpu/mtrr/Makefile +++ b/arch/x86/kernel/cpu/mtrr/Makefile @@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only obj-y := mtrr.o if.o generic.o cleanup.o -obj-$(CONFIG_X86_32) += amd.o cyrix.o centaur.o +obj-$(CONFIG_X86_32) += amd.o cyrix.o centaur.o legacy.o diff --git a/arch/x86/kernel/cpu/mtrr/amd.c b/arch/x86/kernel/cpu/mtrr/amd.c index eff6ac62c0ff..ef3e8e42b782 100644 --- a/arch/x86/kernel/cpu/mtrr/amd.c +++ b/arch/x86/kernel/cpu/mtrr/amd.c @@ -110,7 +110,7 @@ amd_validate_add_page(unsigned long base, unsigned long size, unsigned int type) } const struct mtrr_ops amd_mtrr_ops = { - .vendor = X86_VENDOR_AMD, + .var_regs = 2, .set = amd_set_mtrr, .get = amd_get_mtrr, .get_free_region = generic_get_free_region, diff --git a/arch/x86/kernel/cpu/mtrr/centaur.c b/arch/x86/kernel/cpu/mtrr/centaur.c index b8a74eddde83..6f6c3ae92943 100644 --- a/arch/x86/kernel/cpu/mtrr/centaur.c +++ b/arch/x86/kernel/cpu/mtrr/centaur.c @@ -45,15 +45,6 @@ centaur_get_free_region(unsigned long base, unsigned long size, int replace_reg) return -ENOSPC; } -/* - * Report boot time MCR setups - */ -void mtrr_centaur_report_mcr(int mcr, u32 lo, u32 hi) -{ - centaur_mcr[mcr].low = lo; - centaur_mcr[mcr].high = hi; -} - static void centaur_get_mcr(unsigned int reg, unsigned long *base, unsigned long *size, mtrr_type * type) @@ -112,7 +103,7 @@ centaur_validate_add_page(unsigned long base, unsigned long size, unsigned int t } const struct mtrr_ops centaur_mtrr_ops = { - .vendor = X86_VENDOR_CENTAUR, + .var_regs = 8, .set = centaur_set_mcr, .get = centaur_get_mcr, .get_free_region = centaur_get_free_region, diff --git a/arch/x86/kernel/cpu/mtrr/cleanup.c b/arch/x86/kernel/cpu/mtrr/cleanup.c index b5f43049fa5f..18cf79d6e2c5 100644 --- a/arch/x86/kernel/cpu/mtrr/cleanup.c +++ b/arch/x86/kernel/cpu/mtrr/cleanup.c @@ -55,9 +55,6 @@ static int __initdata nr_range; static struct var_mtrr_range_state __initdata range_state[RANGE_NUM]; -static int __initdata debug_print; -#define Dprintk(x...) do { if (debug_print) pr_debug(x); } while (0) - #define BIOS_BUG_MSG \ "WARNING: BIOS bug: VAR MTRR %d contains strange UC entry under 1M, check with your system vendor!\n" @@ -79,12 +76,11 @@ x86_get_mtrr_mem_range(struct range *range, int nr_range, nr_range = add_range_with_merge(range, RANGE_NUM, nr_range, base, base + size); } - if (debug_print) { - pr_debug("After WB checking\n"); - for (i = 0; i < nr_range; i++) - pr_debug("MTRR MAP PFN: %016llx - %016llx\n", - range[i].start, range[i].end); - } + + Dprintk("After WB checking\n"); + for (i = 0; i < nr_range; i++) + Dprintk("MTRR MAP PFN: %016llx - %016llx\n", + range[i].start, range[i].end); /* Take out UC ranges: */ for (i = 0; i < num_var_ranges; i++) { @@ -112,24 +108,22 @@ x86_get_mtrr_mem_range(struct range *range, int nr_range, subtract_range(range, RANGE_NUM, extra_remove_base, extra_remove_base + extra_remove_size); - if (debug_print) { - pr_debug("After UC checking\n"); - for (i = 0; i < RANGE_NUM; i++) { - if (!range[i].end) - continue; - pr_debug("MTRR MAP PFN: %016llx - %016llx\n", - range[i].start, range[i].end); - } + Dprintk("After UC checking\n"); + for (i = 0; i < RANGE_NUM; i++) { + if (!range[i].end) + continue; + + Dprintk("MTRR MAP PFN: %016llx - %016llx\n", + range[i].start, range[i].end); } /* sort the ranges */ nr_range = clean_sort_range(range, RANGE_NUM); - if (debug_print) { - pr_debug("After sorting\n"); - for (i = 0; i < nr_range; i++) - pr_debug("MTRR MAP PFN: %016llx - %016llx\n", - range[i].start, range[i].end); - } + + Dprintk("After sorting\n"); + for (i = 0; i < nr_range; i++) + Dprintk("MTRR MAP PFN: %016llx - %016llx\n", + range[i].start, range[i].end); return nr_range; } @@ -164,16 +158,9 @@ static int __init enable_mtrr_cleanup_setup(char *str) } early_param("enable_mtrr_cleanup", enable_mtrr_cleanup_setup); -static int __init mtrr_cleanup_debug_setup(char *str) -{ - debug_print = 1; - return 0; -} -early_param("mtrr_cleanup_debug", mtrr_cleanup_debug_setup); - static void __init set_var_mtrr(unsigned int reg, unsigned long basek, unsigned long sizek, - unsigned char type, unsigned int address_bits) + unsigned char type) { u32 base_lo, base_hi, mask_lo, mask_hi; u64 base, mask; @@ -183,7 +170,7 @@ set_var_mtrr(unsigned int reg, unsigned long basek, unsigned long sizek, return; } - mask = (1ULL << address_bits) - 1; + mask = (1ULL << boot_cpu_data.x86_phys_bits) - 1; mask &= ~((((u64)sizek) << 10) - 1); base = ((u64)basek) << 10; @@ -209,7 +196,7 @@ save_var_mtrr(unsigned int reg, unsigned long basek, unsigned long sizek, range_state[reg].type = type; } -static void __init set_var_mtrr_all(unsigned int address_bits) +static void __init set_var_mtrr_all(void) { unsigned long basek, sizek; unsigned char type; @@ -220,7 +207,7 @@ static void __init set_var_mtrr_all(unsigned int address_bits) sizek = range_state[reg].size_pfn << (PAGE_SHIFT - 10); type = range_state[reg].type; - set_var_mtrr(reg, basek, sizek, type, address_bits); + set_var_mtrr(reg, basek, sizek, type); } } @@ -267,7 +254,7 @@ range_to_mtrr(unsigned int reg, unsigned long range_startk, align = max_align; sizek = 1UL << align; - if (debug_print) { + if (mtrr_debug) { char start_factor = 'K', size_factor = 'K'; unsigned long start_base, size_base; @@ -542,7 +529,7 @@ static void __init print_out_mtrr_range_state(void) start_base = to_size_factor(start_base, &start_factor); type = range_state[i].type; - pr_debug("reg %d, base: %ld%cB, range: %ld%cB, type %s\n", + Dprintk("reg %d, base: %ld%cB, range: %ld%cB, type %s\n", i, start_base, start_factor, size_base, size_factor, (type == MTRR_TYPE_UNCACHABLE) ? "UC" : @@ -680,7 +667,7 @@ static int __init mtrr_search_optimal_index(void) return index_good; } -int __init mtrr_cleanup(unsigned address_bits) +int __init mtrr_cleanup(void) { unsigned long x_remove_base, x_remove_size; unsigned long base, size, def, dummy; @@ -689,7 +676,10 @@ int __init mtrr_cleanup(unsigned address_bits) int index_good; int i; - if (!is_cpu(INTEL) || enable_mtrr_cleanup < 1) + if (!mtrr_enabled()) + return 0; + + if (!cpu_feature_enabled(X86_FEATURE_MTRR) || enable_mtrr_cleanup < 1) return 0; rdmsr(MSR_MTRRdefType, def, dummy); @@ -711,7 +701,7 @@ int __init mtrr_cleanup(unsigned address_bits) return 0; /* Print original var MTRRs at first, for debugging: */ - pr_debug("original variable MTRRs\n"); + Dprintk("original variable MTRRs\n"); print_out_mtrr_range_state(); memset(range, 0, sizeof(range)); @@ -742,8 +732,8 @@ int __init mtrr_cleanup(unsigned address_bits) mtrr_print_out_one_result(i); if (!result[i].bad) { - set_var_mtrr_all(address_bits); - pr_debug("New variable MTRRs\n"); + set_var_mtrr_all(); + Dprintk("New variable MTRRs\n"); print_out_mtrr_range_state(); return 1; } @@ -763,7 +753,7 @@ int __init mtrr_cleanup(unsigned address_bits) mtrr_calc_range_state(chunk_size, gran_size, x_remove_base, x_remove_size, i); - if (debug_print) { + if (mtrr_debug) { mtrr_print_out_one_result(i); pr_info("\n"); } @@ -786,8 +776,8 @@ int __init mtrr_cleanup(unsigned address_bits) gran_size = result[i].gran_sizek; gran_size <<= 10; x86_setup_var_mtrrs(range, nr_range, chunk_size, gran_size); - set_var_mtrr_all(address_bits); - pr_debug("New variable MTRRs\n"); + set_var_mtrr_all(); + Dprintk("New variable MTRRs\n"); print_out_mtrr_range_state(); return 1; } else { @@ -802,7 +792,7 @@ int __init mtrr_cleanup(unsigned address_bits) return 0; } #else -int __init mtrr_cleanup(unsigned address_bits) +int __init mtrr_cleanup(void) { return 0; } @@ -882,15 +872,18 @@ int __init mtrr_trim_uncached_memory(unsigned long end_pfn) /* extra one for all 0 */ int num[MTRR_NUM_TYPES + 1]; + if (!mtrr_enabled()) + return 0; + /* * Make sure we only trim uncachable memory on machines that * support the Intel MTRR architecture: */ - if (!is_cpu(INTEL) || disable_mtrr_trim) + if (!cpu_feature_enabled(X86_FEATURE_MTRR) || disable_mtrr_trim) return 0; rdmsr(MSR_MTRRdefType, def, dummy); - def &= 0xff; + def &= MTRR_DEF_TYPE_TYPE; if (def != MTRR_TYPE_UNCACHABLE) return 0; diff --git a/arch/x86/kernel/cpu/mtrr/cyrix.c b/arch/x86/kernel/cpu/mtrr/cyrix.c index 173b9e01e623..238dad57d4d6 100644 --- a/arch/x86/kernel/cpu/mtrr/cyrix.c +++ b/arch/x86/kernel/cpu/mtrr/cyrix.c @@ -235,7 +235,7 @@ static void cyrix_set_arr(unsigned int reg, unsigned long base, } const struct mtrr_ops cyrix_mtrr_ops = { - .vendor = X86_VENDOR_CYRIX, + .var_regs = 8, .set = cyrix_set_arr, .get = cyrix_get_arr, .get_free_region = cyrix_get_free_region, diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/generic.c index ee09d359e08f..2d6aa5d2e3d7 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -8,10 +8,12 @@ #include #include #include - +#include #include #include #include +#include +#include #include #include #include @@ -31,6 +33,55 @@ static struct fixed_range_block fixed_range_blocks[] = { {} }; +struct cache_map { + u64 start; + u64 end; + u64 flags; + u64 type:8; + u64 fixed:1; +}; + +bool mtrr_debug; + +static int __init mtrr_param_setup(char *str) +{ + int rc = 0; + + if (!str) + return -EINVAL; + if (!strcmp(str, "debug")) + mtrr_debug = true; + else + rc = -EINVAL; + + return rc; +} +early_param("mtrr", mtrr_param_setup); + +/* + * CACHE_MAP_MAX is the maximum number of memory ranges in cache_map, where + * no 2 adjacent ranges have the same cache mode (those would be merged). + * The number is based on the worst case: + * - no two adjacent fixed MTRRs share the same cache mode + * - one variable MTRR is spanning a huge area with mode WB + * - 255 variable MTRRs with mode UC all overlap with the WB MTRR, creating 2 + * additional ranges each (result like "ababababa...aba" with a = WB, b = UC), + * accounting for MTRR_MAX_VAR_RANGES * 2 - 1 range entries + * - a TOP_MEM2 area (even with overlapping an UC MTRR can't add 2 range entries + * to the possible maximum, as it always starts at 4GB, thus it can't be in + * the middle of that MTRR, unless that MTRR starts at 0, which would remove + * the initial "a" from the "abababa" pattern above) + * The map won't contain ranges with no matching MTRR (those fall back to the + * default cache mode). + */ +#define CACHE_MAP_MAX (MTRR_NUM_FIXED_RANGES + MTRR_MAX_VAR_RANGES * 2) + +static struct cache_map init_cache_map[CACHE_MAP_MAX] __initdata; +static struct cache_map *cache_map __refdata = init_cache_map; +static unsigned int cache_map_size = CACHE_MAP_MAX; +static unsigned int cache_map_n; +static unsigned int cache_map_fixed; + static unsigned long smp_changes_mask; static int mtrr_state_set; u64 mtrr_tom2; @@ -38,6 +89,9 @@ u64 mtrr_tom2; struct mtrr_state_type mtrr_state; EXPORT_SYMBOL_GPL(mtrr_state); +/* Reserved bits in the high portion of the MTRRphysBaseN MSR. */ +u32 phys_hi_rsvd; + /* * BIOS is expected to clear MtrrFixDramModEn bit, see for example * "BIOS and Kernel Developer's Guide for the AMD Athlon 64 and AMD @@ -69,175 +123,370 @@ static u64 get_mtrr_size(u64 mask) { u64 size; - mask >>= PAGE_SHIFT; - mask |= size_or_mask; + mask |= (u64)phys_hi_rsvd << 32; size = -mask; - size <<= PAGE_SHIFT; + return size; } -/* - * Check and return the effective type for MTRR-MTRR type overlap. - * Returns 1 if the effective type is UNCACHEABLE, else returns 0 - */ -static int check_type_overlap(u8 *prev, u8 *curr) +static u8 get_var_mtrr_state(unsigned int reg, u64 *start, u64 *size) { - if (*prev == MTRR_TYPE_UNCACHABLE || *curr == MTRR_TYPE_UNCACHABLE) { - *prev = MTRR_TYPE_UNCACHABLE; - *curr = MTRR_TYPE_UNCACHABLE; + struct mtrr_var_range *mtrr = mtrr_state.var_ranges + reg; + + if (!(mtrr->mask_lo & MTRR_PHYSMASK_V)) + return MTRR_TYPE_INVALID; + + *start = (((u64)mtrr->base_hi) << 32) + (mtrr->base_lo & PAGE_MASK); + *size = get_mtrr_size((((u64)mtrr->mask_hi) << 32) + + (mtrr->mask_lo & PAGE_MASK)); + + return mtrr->base_lo & MTRR_PHYSBASE_TYPE; +} + +static u8 get_effective_type(u8 type1, u8 type2) +{ + if (type1 == MTRR_TYPE_UNCACHABLE || type2 == MTRR_TYPE_UNCACHABLE) + return MTRR_TYPE_UNCACHABLE; + + if ((type1 == MTRR_TYPE_WRBACK && type2 == MTRR_TYPE_WRTHROUGH) || + (type1 == MTRR_TYPE_WRTHROUGH && type2 == MTRR_TYPE_WRBACK)) + return MTRR_TYPE_WRTHROUGH; + + if (type1 != type2) + return MTRR_TYPE_UNCACHABLE; + + return type1; +} + +static void rm_map_entry_at(int idx) +{ + cache_map_n--; + if (cache_map_n > idx) { + memmove(cache_map + idx, cache_map + idx + 1, + sizeof(*cache_map) * (cache_map_n - idx)); + } +} + +/* + * Add an entry into cache_map at a specific index. Merges adjacent entries if + * appropriate. Return the number of merges for correcting the scan index + * (this is needed as merging will reduce the number of entries, which will + * result in skipping entries in future iterations if the scan index isn't + * corrected). + * Note that the corrected index can never go below -1 (resulting in being 0 in + * the next scan iteration), as "2" is returned only if the current index is + * larger than zero. + */ +static int add_map_entry_at(u64 start, u64 end, u8 type, int idx) +{ + bool merge_prev = false, merge_next = false; + + if (start >= end) + return 0; + + if (idx > 0) { + struct cache_map *prev = cache_map + idx - 1; + + if (!prev->fixed && start == prev->end && type == prev->type) + merge_prev = true; + } + + if (idx < cache_map_n) { + struct cache_map *next = cache_map + idx; + + if (!next->fixed && end == next->start && type == next->type) + merge_next = true; + } + + if (merge_prev && merge_next) { + cache_map[idx - 1].end = cache_map[idx].end; + rm_map_entry_at(idx); + return 2; + } + if (merge_prev) { + cache_map[idx - 1].end = end; + return 1; + } + if (merge_next) { + cache_map[idx].start = start; return 1; } - if ((*prev == MTRR_TYPE_WRBACK && *curr == MTRR_TYPE_WRTHROUGH) || - (*prev == MTRR_TYPE_WRTHROUGH && *curr == MTRR_TYPE_WRBACK)) { - *prev = MTRR_TYPE_WRTHROUGH; - *curr = MTRR_TYPE_WRTHROUGH; + /* Sanity check: the array should NEVER be too small! */ + if (cache_map_n == cache_map_size) { + WARN(1, "MTRR cache mode memory map exhausted!\n"); + cache_map_n = cache_map_fixed; + return 0; } - if (*prev != *curr) { - *prev = MTRR_TYPE_UNCACHABLE; - *curr = MTRR_TYPE_UNCACHABLE; - return 1; + if (cache_map_n > idx) { + memmove(cache_map + idx + 1, cache_map + idx, + sizeof(*cache_map) * (cache_map_n - idx)); } + cache_map[idx].start = start; + cache_map[idx].end = end; + cache_map[idx].type = type; + cache_map[idx].fixed = 0; + cache_map_n++; + return 0; } -/** - * mtrr_type_lookup_fixed - look up memory type in MTRR fixed entries - * - * Return the MTRR fixed memory type of 'start'. - * - * MTRR fixed entries are divided into the following ways: - * 0x00000 - 0x7FFFF : This range is divided into eight 64KB sub-ranges - * 0x80000 - 0xBFFFF : This range is divided into sixteen 16KB sub-ranges - * 0xC0000 - 0xFFFFF : This range is divided into sixty-four 4KB sub-ranges - * - * Return Values: - * MTRR_TYPE_(type) - Matched memory type - * MTRR_TYPE_INVALID - Unmatched - */ -static u8 mtrr_type_lookup_fixed(u64 start, u64 end) +/* Clear a part of an entry. Return 1 if start of entry is still valid. */ +static int clr_map_range_at(u64 start, u64 end, int idx) { - int idx; + int ret = start != cache_map[idx].start; + u64 tmp; - if (start >= 0x100000) - return MTRR_TYPE_INVALID; - - /* 0x0 - 0x7FFFF */ - if (start < 0x80000) { - idx = 0; - idx += (start >> 16); - return mtrr_state.fixed_ranges[idx]; - /* 0x80000 - 0xBFFFF */ - } else if (start < 0xC0000) { - idx = 1 * 8; - idx += ((start - 0x80000) >> 14); - return mtrr_state.fixed_ranges[idx]; + if (start == cache_map[idx].start && end == cache_map[idx].end) { + rm_map_entry_at(idx); + } else if (start == cache_map[idx].start) { + cache_map[idx].start = end; + } else if (end == cache_map[idx].end) { + cache_map[idx].end = start; + } else { + tmp = cache_map[idx].end; + cache_map[idx].end = start; + add_map_entry_at(end, tmp, cache_map[idx].type, idx + 1); } - /* 0xC0000 - 0xFFFFF */ - idx = 3 * 8; - idx += ((start - 0xC0000) >> 12); - return mtrr_state.fixed_ranges[idx]; + return ret; +} + +/* + * Add MTRR to the map. The current map is scanned and each part of the MTRR + * either overlapping with an existing entry or with a hole in the map is + * handled separately. + */ +static void add_map_entry(u64 start, u64 end, u8 type) +{ + u8 new_type, old_type; + u64 tmp; + int i; + + for (i = 0; i < cache_map_n && start < end; i++) { + if (start >= cache_map[i].end) + continue; + + if (start < cache_map[i].start) { + /* Region start has no overlap. */ + tmp = min(end, cache_map[i].start); + i -= add_map_entry_at(start, tmp, type, i); + start = tmp; + continue; + } + + new_type = get_effective_type(type, cache_map[i].type); + old_type = cache_map[i].type; + + if (cache_map[i].fixed || new_type == old_type) { + /* Cut off start of new entry. */ + start = cache_map[i].end; + continue; + } + + /* Handle only overlapping part of region. */ + tmp = min(end, cache_map[i].end); + i += clr_map_range_at(start, tmp, i); + i -= add_map_entry_at(start, tmp, new_type, i); + start = tmp; + } + + /* Add rest of region after last map entry (rest might be empty). */ + add_map_entry_at(start, end, type, i); +} + +/* Add variable MTRRs to cache map. */ +static void map_add_var(void) +{ + u64 start, size; + unsigned int i; + u8 type; + + /* + * Add AMD TOP_MEM2 area. Can't be added in mtrr_build_map(), as it + * needs to be added again when rebuilding the map due to potentially + * having moved as a result of variable MTRRs for memory below 4GB. + */ + if (mtrr_tom2) { + add_map_entry(BIT_ULL(32), mtrr_tom2, MTRR_TYPE_WRBACK); + cache_map[cache_map_n - 1].fixed = 1; + } + + for (i = 0; i < num_var_ranges; i++) { + type = get_var_mtrr_state(i, &start, &size); + if (type != MTRR_TYPE_INVALID) + add_map_entry(start, start + size, type); + } +} + +/* + * Rebuild map by replacing variable entries. Needs to be called when MTRR + * registers are being changed after boot, as such changes could include + * removals of registers, which are complicated to handle without rebuild of + * the map. + */ +void generic_rebuild_map(void) +{ + if (mtrr_if != &generic_mtrr_ops) + return; + + cache_map_n = cache_map_fixed; + + map_add_var(); +} + +static unsigned int __init get_cache_map_size(void) +{ + return cache_map_fixed + 2 * num_var_ranges + (mtrr_tom2 != 0); +} + +/* Build the cache_map containing the cache modes per memory range. */ +void __init mtrr_build_map(void) +{ + u64 start, end, size; + unsigned int i; + u8 type; + + /* Add fixed MTRRs, optimize for adjacent entries with same type. */ + if (mtrr_state.enabled & MTRR_STATE_MTRR_FIXED_ENABLED) { + /* + * Start with 64k size fixed entries, preset 1st one (hence the + * loop below is starting with index 1). + */ + start = 0; + end = size = 0x10000; + type = mtrr_state.fixed_ranges[0]; + + for (i = 1; i < MTRR_NUM_FIXED_RANGES; i++) { + /* 8 64k entries, then 16 16k ones, rest 4k. */ + if (i == 8 || i == 24) + size >>= 2; + + if (mtrr_state.fixed_ranges[i] != type) { + add_map_entry(start, end, type); + start = end; + type = mtrr_state.fixed_ranges[i]; + } + end += size; + } + add_map_entry(start, end, type); + } + + /* Mark fixed, they take precedence. */ + for (i = 0; i < cache_map_n; i++) + cache_map[i].fixed = 1; + cache_map_fixed = cache_map_n; + + map_add_var(); + + pr_info("MTRR map: %u entries (%u fixed + %u variable; max %u), built from %u variable MTRRs\n", + cache_map_n, cache_map_fixed, cache_map_n - cache_map_fixed, + get_cache_map_size(), num_var_ranges + (mtrr_tom2 != 0)); + + if (mtrr_debug) { + for (i = 0; i < cache_map_n; i++) { + pr_info("%3u: %016llx-%016llx %s\n", i, + cache_map[i].start, cache_map[i].end - 1, + mtrr_attrib_to_str(cache_map[i].type)); + } + } +} + +/* Copy the cache_map from __initdata memory to dynamically allocated one. */ +void __init mtrr_copy_map(void) +{ + unsigned int new_size = get_cache_map_size(); + + if (!mtrr_state.enabled || !new_size) { + cache_map = NULL; + return; + } + + mutex_lock(&mtrr_mutex); + + cache_map = kcalloc(new_size, sizeof(*cache_map), GFP_KERNEL); + if (cache_map) { + memmove(cache_map, init_cache_map, + cache_map_n * sizeof(*cache_map)); + cache_map_size = new_size; + } else { + mtrr_state.enabled = 0; + pr_err("MTRRs disabled due to allocation failure for lookup map.\n"); + } + + mutex_unlock(&mtrr_mutex); } /** - * mtrr_type_lookup_variable - look up memory type in MTRR variable entries + * mtrr_overwrite_state - set static MTRR state * - * Return Value: - * MTRR_TYPE_(type) - Matched memory type or default memory type (unmatched) - * - * Output Arguments: - * repeat - Set to 1 when [start:end] spanned across MTRR range and type - * returned corresponds only to [start:*partial_end]. Caller has - * to lookup again for [*partial_end:end]. - * - * uniform - Set to 1 when an MTRR covers the region uniformly, i.e. the - * region is fully covered by a single MTRR entry or the default - * type. + * Used to set MTRR state via different means (e.g. with data obtained from + * a hypervisor). + * Is allowed only for special cases when running virtualized. Must be called + * from the x86_init.hyper.init_platform() hook. It can be called only once. + * The MTRR state can't be changed afterwards. To ensure that, X86_FEATURE_MTRR + * is cleared. */ -static u8 mtrr_type_lookup_variable(u64 start, u64 end, u64 *partial_end, - int *repeat, u8 *uniform) +void mtrr_overwrite_state(struct mtrr_var_range *var, unsigned int num_var, + mtrr_type def_type) { - int i; - u64 base, mask; - u8 prev_match, curr_match; + unsigned int i; - *repeat = 0; - *uniform = 1; + /* Only allowed to be called once before mtrr_bp_init(). */ + if (WARN_ON_ONCE(mtrr_state_set)) + return; - prev_match = MTRR_TYPE_INVALID; - for (i = 0; i < num_var_ranges; ++i) { - unsigned short start_state, end_state, inclusive; + /* Only allowed when running virtualized. */ + if (!cpu_feature_enabled(X86_FEATURE_HYPERVISOR)) + return; - if (!(mtrr_state.var_ranges[i].mask_lo & (1 << 11))) - continue; + /* + * Only allowed for special virtualization cases: + * - when running as Hyper-V, SEV-SNP guest using vTOM + * - when running as Xen PV guest + * - when running as SEV-SNP or TDX guest to avoid unnecessary + * VMM communication/Virtualization exceptions (#VC, #VE) + */ + if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP) && + !hv_is_isolation_supported() && + !cpu_feature_enabled(X86_FEATURE_XENPV) && + !cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) + return; - base = (((u64)mtrr_state.var_ranges[i].base_hi) << 32) + - (mtrr_state.var_ranges[i].base_lo & PAGE_MASK); - mask = (((u64)mtrr_state.var_ranges[i].mask_hi) << 32) + - (mtrr_state.var_ranges[i].mask_lo & PAGE_MASK); + /* Disable MTRR in order to disable MTRR modifications. */ + setup_clear_cpu_cap(X86_FEATURE_MTRR); - start_state = ((start & mask) == (base & mask)); - end_state = ((end & mask) == (base & mask)); - inclusive = ((start < base) && (end > base)); - - if ((start_state != end_state) || inclusive) { - /* - * We have start:end spanning across an MTRR. - * We split the region into either - * - * - start_state:1 - * (start:mtrr_end)(mtrr_end:end) - * - end_state:1 - * (start:mtrr_start)(mtrr_start:end) - * - inclusive:1 - * (start:mtrr_start)(mtrr_start:mtrr_end)(mtrr_end:end) - * - * depending on kind of overlap. - * - * Return the type of the first region and a pointer - * to the start of next region so that caller will be - * advised to lookup again after having adjusted start - * and end. - * - * Note: This way we handle overlaps with multiple - * entries and the default type properly. - */ - if (start_state) - *partial_end = base + get_mtrr_size(mask); - else - *partial_end = base; - - if (unlikely(*partial_end <= start)) { - WARN_ON(1); - *partial_end = start + PAGE_SIZE; - } - - end = *partial_end - 1; /* end is inclusive */ - *repeat = 1; - *uniform = 0; + if (var) { + if (num_var > MTRR_MAX_VAR_RANGES) { + pr_warn("Trying to overwrite MTRR state with %u variable entries\n", + num_var); + num_var = MTRR_MAX_VAR_RANGES; } - - if ((start & mask) != (base & mask)) - continue; - - curr_match = mtrr_state.var_ranges[i].base_lo & 0xff; - if (prev_match == MTRR_TYPE_INVALID) { - prev_match = curr_match; - continue; - } - - *uniform = 0; - if (check_type_overlap(&prev_match, &curr_match)) - return curr_match; + for (i = 0; i < num_var; i++) + mtrr_state.var_ranges[i] = var[i]; + num_var_ranges = num_var; } - if (prev_match != MTRR_TYPE_INVALID) - return prev_match; + mtrr_state.def_type = def_type; + mtrr_state.enabled |= MTRR_STATE_MTRR_ENABLED; - return mtrr_state.def_type; + mtrr_state_set = 1; +} + +static u8 type_merge(u8 type, u8 new_type, u8 *uniform) +{ + u8 effective_type; + + if (type == MTRR_TYPE_INVALID) + return new_type; + + effective_type = get_effective_type(type, new_type); + if (type != effective_type) + *uniform = 0; + + return effective_type; } /** @@ -248,66 +497,49 @@ static u8 mtrr_type_lookup_variable(u64 start, u64 end, u64 *partial_end, * MTRR_TYPE_INVALID - MTRR is disabled * * Output Argument: - * uniform - Set to 1 when an MTRR covers the region uniformly, i.e. the - * region is fully covered by a single MTRR entry or the default - * type. + * uniform - Set to 1 when the returned MTRR type is valid for the whole + * region, set to 0 else. */ u8 mtrr_type_lookup(u64 start, u64 end, u8 *uniform) { - u8 type, prev_type, is_uniform = 1, dummy; - int repeat; - u64 partial_end; + u8 type = MTRR_TYPE_INVALID; + unsigned int i; - /* Make end inclusive instead of exclusive */ - end--; + if (!mtrr_state_set) { + /* Uniformity is unknown. */ + *uniform = 0; + return MTRR_TYPE_UNCACHABLE; + } - if (!mtrr_state_set) - return MTRR_TYPE_INVALID; + *uniform = 1; if (!(mtrr_state.enabled & MTRR_STATE_MTRR_ENABLED)) - return MTRR_TYPE_INVALID; + return MTRR_TYPE_UNCACHABLE; - /* - * Look up the fixed ranges first, which take priority over - * the variable ranges. - */ - if ((start < 0x100000) && - (mtrr_state.have_fixed) && - (mtrr_state.enabled & MTRR_STATE_MTRR_FIXED_ENABLED)) { - is_uniform = 0; - type = mtrr_type_lookup_fixed(start, end); - goto out; + for (i = 0; i < cache_map_n && start < end; i++) { + /* Region after current map entry? -> continue with next one. */ + if (start >= cache_map[i].end) + continue; + + /* Start of region not covered by current map entry? */ + if (start < cache_map[i].start) { + /* At least some part of region has default type. */ + type = type_merge(type, mtrr_state.def_type, uniform); + /* End of region not covered, too? -> lookup done. */ + if (end <= cache_map[i].start) + return type; + } + + /* At least part of region covered by map entry. */ + type = type_merge(type, cache_map[i].type, uniform); + + start = cache_map[i].end; } - /* - * Look up the variable ranges. Look of multiple ranges matching - * this address and pick type as per MTRR precedence. - */ - type = mtrr_type_lookup_variable(start, end, &partial_end, - &repeat, &is_uniform); + /* End of region past last entry in map? -> use default type. */ + if (start < end) + type = type_merge(type, mtrr_state.def_type, uniform); - /* - * Common path is with repeat = 0. - * However, we can have cases where [start:end] spans across some - * MTRR ranges and/or the default type. Do repeated lookups for - * that case here. - */ - while (repeat) { - prev_type = type; - start = partial_end; - is_uniform = 0; - type = mtrr_type_lookup_variable(start, end, &partial_end, - &repeat, &dummy); - - if (check_type_overlap(&prev_type, &type)) - goto out; - } - - if (mtrr_tom2 && (start >= (1ULL<<32)) && (end < mtrr_tom2)) - type = MTRR_TYPE_WRBACK; - -out: - *uniform = is_uniform; return type; } @@ -363,8 +595,8 @@ static void __init print_fixed_last(void) if (!last_fixed_end) return; - pr_debug(" %05X-%05X %s\n", last_fixed_start, - last_fixed_end - 1, mtrr_attrib_to_str(last_fixed_type)); + pr_info(" %05X-%05X %s\n", last_fixed_start, + last_fixed_end - 1, mtrr_attrib_to_str(last_fixed_type)); last_fixed_end = 0; } @@ -402,10 +634,10 @@ static void __init print_mtrr_state(void) unsigned int i; int high_width; - pr_debug("MTRR default type: %s\n", - mtrr_attrib_to_str(mtrr_state.def_type)); + pr_info("MTRR default type: %s\n", + mtrr_attrib_to_str(mtrr_state.def_type)); if (mtrr_state.have_fixed) { - pr_debug("MTRR fixed ranges %sabled:\n", + pr_info("MTRR fixed ranges %sabled:\n", ((mtrr_state.enabled & MTRR_STATE_MTRR_ENABLED) && (mtrr_state.enabled & MTRR_STATE_MTRR_FIXED_ENABLED)) ? "en" : "dis"); @@ -420,26 +652,27 @@ static void __init print_mtrr_state(void) /* tail */ print_fixed_last(); } - pr_debug("MTRR variable ranges %sabled:\n", - mtrr_state.enabled & MTRR_STATE_MTRR_ENABLED ? "en" : "dis"); - high_width = (__ffs64(size_or_mask) - (32 - PAGE_SHIFT) + 3) / 4; + pr_info("MTRR variable ranges %sabled:\n", + mtrr_state.enabled & MTRR_STATE_MTRR_ENABLED ? "en" : "dis"); + high_width = (boot_cpu_data.x86_phys_bits - (32 - PAGE_SHIFT) + 3) / 4; for (i = 0; i < num_var_ranges; ++i) { - if (mtrr_state.var_ranges[i].mask_lo & (1 << 11)) - pr_debug(" %u base %0*X%05X000 mask %0*X%05X000 %s\n", - i, - high_width, - mtrr_state.var_ranges[i].base_hi, - mtrr_state.var_ranges[i].base_lo >> 12, - high_width, - mtrr_state.var_ranges[i].mask_hi, - mtrr_state.var_ranges[i].mask_lo >> 12, - mtrr_attrib_to_str(mtrr_state.var_ranges[i].base_lo & 0xff)); + if (mtrr_state.var_ranges[i].mask_lo & MTRR_PHYSMASK_V) + pr_info(" %u base %0*X%05X000 mask %0*X%05X000 %s\n", + i, + high_width, + mtrr_state.var_ranges[i].base_hi, + mtrr_state.var_ranges[i].base_lo >> 12, + high_width, + mtrr_state.var_ranges[i].mask_hi, + mtrr_state.var_ranges[i].mask_lo >> 12, + mtrr_attrib_to_str(mtrr_state.var_ranges[i].base_lo & + MTRR_PHYSBASE_TYPE)); else - pr_debug(" %u disabled\n", i); + pr_info(" %u disabled\n", i); } if (mtrr_tom2) - pr_debug("TOM2: %016llx aka %lldM\n", mtrr_tom2, mtrr_tom2>>20); + pr_info("TOM2: %016llx aka %lldM\n", mtrr_tom2, mtrr_tom2>>20); } /* Grab all of the MTRR state for this CPU into *state */ @@ -452,7 +685,7 @@ bool __init get_mtrr_state(void) vrs = mtrr_state.var_ranges; rdmsr(MSR_MTRRcap, lo, dummy); - mtrr_state.have_fixed = (lo >> 8) & 1; + mtrr_state.have_fixed = lo & MTRR_CAP_FIX; for (i = 0; i < num_var_ranges; i++) get_mtrr_var_range(i, &vrs[i]); @@ -460,8 +693,8 @@ bool __init get_mtrr_state(void) get_fixed_ranges(mtrr_state.fixed_ranges); rdmsr(MSR_MTRRdefType, lo, dummy); - mtrr_state.def_type = (lo & 0xff); - mtrr_state.enabled = (lo & 0xc00) >> 10; + mtrr_state.def_type = lo & MTRR_DEF_TYPE_TYPE; + mtrr_state.enabled = (lo & MTRR_DEF_TYPE_ENABLE) >> MTRR_STATE_SHIFT; if (amd_special_default_mtrr()) { unsigned low, high; @@ -474,7 +707,8 @@ bool __init get_mtrr_state(void) mtrr_tom2 &= 0xffffff800000ULL; } - print_mtrr_state(); + if (mtrr_debug) + print_mtrr_state(); mtrr_state_set = 1; @@ -574,7 +808,7 @@ static void generic_get_mtrr(unsigned int reg, unsigned long *base, rdmsr(MTRRphysMask_MSR(reg), mask_lo, mask_hi); - if ((mask_lo & 0x800) == 0) { + if (!(mask_lo & MTRR_PHYSMASK_V)) { /* Invalid (i.e. free) range */ *base = 0; *size = 0; @@ -585,8 +819,8 @@ static void generic_get_mtrr(unsigned int reg, unsigned long *base, rdmsr(MTRRphysBase_MSR(reg), base_lo, base_hi); /* Work out the shifted address mask: */ - tmp = (u64)mask_hi << (32 - PAGE_SHIFT) | mask_lo >> PAGE_SHIFT; - mask = size_or_mask | tmp; + tmp = (u64)mask_hi << 32 | (mask_lo & PAGE_MASK); + mask = (u64)phys_hi_rsvd << 32 | tmp; /* Expand tmp with high bits to all 1s: */ hi = fls64(tmp); @@ -604,9 +838,9 @@ static void generic_get_mtrr(unsigned int reg, unsigned long *base, * This works correctly if size is a power of two, i.e. a * contiguous range: */ - *size = -mask; + *size = -mask >> PAGE_SHIFT; *base = (u64)base_hi << (32 - PAGE_SHIFT) | base_lo >> PAGE_SHIFT; - *type = base_lo & 0xff; + *type = base_lo & MTRR_PHYSBASE_TYPE; out_put_cpu: put_cpu(); @@ -644,9 +878,8 @@ static bool set_mtrr_var_ranges(unsigned int index, struct mtrr_var_range *vr) bool changed = false; rdmsr(MTRRphysBase_MSR(index), lo, hi); - if ((vr->base_lo & 0xfffff0ffUL) != (lo & 0xfffff0ffUL) - || (vr->base_hi & (size_and_mask >> (32 - PAGE_SHIFT))) != - (hi & (size_and_mask >> (32 - PAGE_SHIFT)))) { + if ((vr->base_lo & ~MTRR_PHYSBASE_RSVD) != (lo & ~MTRR_PHYSBASE_RSVD) + || (vr->base_hi & ~phys_hi_rsvd) != (hi & ~phys_hi_rsvd)) { mtrr_wrmsr(MTRRphysBase_MSR(index), vr->base_lo, vr->base_hi); changed = true; @@ -654,9 +887,8 @@ static bool set_mtrr_var_ranges(unsigned int index, struct mtrr_var_range *vr) rdmsr(MTRRphysMask_MSR(index), lo, hi); - if ((vr->mask_lo & 0xfffff800UL) != (lo & 0xfffff800UL) - || (vr->mask_hi & (size_and_mask >> (32 - PAGE_SHIFT))) != - (hi & (size_and_mask >> (32 - PAGE_SHIFT)))) { + if ((vr->mask_lo & ~MTRR_PHYSMASK_RSVD) != (lo & ~MTRR_PHYSMASK_RSVD) + || (vr->mask_hi & ~phys_hi_rsvd) != (hi & ~phys_hi_rsvd)) { mtrr_wrmsr(MTRRphysMask_MSR(index), vr->mask_lo, vr->mask_hi); changed = true; } @@ -691,11 +923,12 @@ static unsigned long set_mtrr_state(void) * Set_mtrr_restore restores the old value of MTRRdefType, * so to set it we fiddle with the saved value: */ - if ((deftype_lo & 0xff) != mtrr_state.def_type - || ((deftype_lo & 0xc00) >> 10) != mtrr_state.enabled) { + if ((deftype_lo & MTRR_DEF_TYPE_TYPE) != mtrr_state.def_type || + ((deftype_lo & MTRR_DEF_TYPE_ENABLE) >> MTRR_STATE_SHIFT) != mtrr_state.enabled) { - deftype_lo = (deftype_lo & ~0xcff) | mtrr_state.def_type | - (mtrr_state.enabled << 10); + deftype_lo = (deftype_lo & MTRR_DEF_TYPE_DISABLE) | + mtrr_state.def_type | + (mtrr_state.enabled << MTRR_STATE_SHIFT); change_mask |= MTRR_CHANGE_MASK_DEFTYPE; } @@ -708,7 +941,7 @@ void mtrr_disable(void) rdmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); /* Disable MTRRs, and set the default type to uncached */ - mtrr_wrmsr(MSR_MTRRdefType, deftype_lo & ~0xcff, deftype_hi); + mtrr_wrmsr(MSR_MTRRdefType, deftype_lo & MTRR_DEF_TYPE_DISABLE, deftype_hi); } void mtrr_enable(void) @@ -762,9 +995,9 @@ static void generic_set_mtrr(unsigned int reg, unsigned long base, memset(vr, 0, sizeof(struct mtrr_var_range)); } else { vr->base_lo = base << PAGE_SHIFT | type; - vr->base_hi = (base & size_and_mask) >> (32 - PAGE_SHIFT); - vr->mask_lo = -size << PAGE_SHIFT | 0x800; - vr->mask_hi = (-size & size_and_mask) >> (32 - PAGE_SHIFT); + vr->base_hi = (base >> (32 - PAGE_SHIFT)) & ~phys_hi_rsvd; + vr->mask_lo = -size << PAGE_SHIFT | MTRR_PHYSMASK_V; + vr->mask_hi = (-size >> (32 - PAGE_SHIFT)) & ~phys_hi_rsvd; mtrr_wrmsr(MTRRphysBase_MSR(reg), vr->base_lo, vr->base_hi); mtrr_wrmsr(MTRRphysMask_MSR(reg), vr->mask_lo, vr->mask_hi); @@ -783,7 +1016,7 @@ int generic_validate_add_page(unsigned long base, unsigned long size, * For Intel PPro stepping <= 7 * must be 4 MiB aligned and not touch 0x70000000 -> 0x7003FFFF */ - if (is_cpu(INTEL) && boot_cpu_data.x86 == 6 && + if (mtrr_if == &generic_mtrr_ops && boot_cpu_data.x86 == 6 && boot_cpu_data.x86_model == 1 && boot_cpu_data.x86_stepping <= 7) { if (base & ((1 << (22 - PAGE_SHIFT)) - 1)) { @@ -817,7 +1050,7 @@ static int generic_have_wrcomb(void) { unsigned long config, dummy; rdmsr(MSR_MTRRcap, config, dummy); - return config & (1 << 10); + return config & MTRR_CAP_WC; } int positive_have_wrcomb(void) diff --git a/arch/x86/kernel/cpu/mtrr/legacy.c b/arch/x86/kernel/cpu/mtrr/legacy.c new file mode 100644 index 000000000000..d25882fcf181 --- /dev/null +++ b/arch/x86/kernel/cpu/mtrr/legacy.c @@ -0,0 +1,90 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include +#include +#include +#include "mtrr.h" + +void mtrr_set_if(void) +{ + switch (boot_cpu_data.x86_vendor) { + case X86_VENDOR_AMD: + /* Pre-Athlon (K6) AMD CPU MTRRs */ + if (cpu_feature_enabled(X86_FEATURE_K6_MTRR)) + mtrr_if = &amd_mtrr_ops; + break; + case X86_VENDOR_CENTAUR: + if (cpu_feature_enabled(X86_FEATURE_CENTAUR_MCR)) + mtrr_if = ¢aur_mtrr_ops; + break; + case X86_VENDOR_CYRIX: + if (cpu_feature_enabled(X86_FEATURE_CYRIX_ARR)) + mtrr_if = &cyrix_mtrr_ops; + break; + default: + break; + } +} + +/* + * The suspend/resume methods are only for CPUs without MTRR. CPUs using generic + * MTRR driver don't require this. + */ +struct mtrr_value { + mtrr_type ltype; + unsigned long lbase; + unsigned long lsize; +}; + +static struct mtrr_value *mtrr_value; + +static int mtrr_save(void) +{ + int i; + + if (!mtrr_value) + return -ENOMEM; + + for (i = 0; i < num_var_ranges; i++) { + mtrr_if->get(i, &mtrr_value[i].lbase, + &mtrr_value[i].lsize, + &mtrr_value[i].ltype); + } + return 0; +} + +static void mtrr_restore(void) +{ + int i; + + for (i = 0; i < num_var_ranges; i++) { + if (mtrr_value[i].lsize) { + mtrr_if->set(i, mtrr_value[i].lbase, + mtrr_value[i].lsize, + mtrr_value[i].ltype); + } + } +} + +static struct syscore_ops mtrr_syscore_ops = { + .suspend = mtrr_save, + .resume = mtrr_restore, +}; + +void mtrr_register_syscore(void) +{ + mtrr_value = kcalloc(num_var_ranges, sizeof(*mtrr_value), GFP_KERNEL); + + /* + * The CPU has no MTRR and seems to not support SMP. They have + * specific drivers, we use a tricky method to support + * suspend/resume for them. + * + * TBD: is there any system with such CPU which supports + * suspend/resume? If no, we should remove the code. + */ + register_syscore_ops(&mtrr_syscore_ops); +} diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtrr.c index 783f3210d582..767bf1c71aad 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -59,15 +59,9 @@ #define MTRR_TO_PHYS_WC_OFFSET 1000 u32 num_var_ranges; -static bool mtrr_enabled(void) -{ - return !!mtrr_if; -} unsigned int mtrr_usage_table[MTRR_MAX_VAR_RANGES]; -static DEFINE_MUTEX(mtrr_mutex); - -u64 size_or_mask, size_and_mask; +DEFINE_MUTEX(mtrr_mutex); const struct mtrr_ops *mtrr_if; @@ -105,21 +99,6 @@ static int have_wrcomb(void) return mtrr_if->have_wrcomb ? mtrr_if->have_wrcomb() : 0; } -/* This function returns the number of variable MTRRs */ -static void __init set_num_var_ranges(bool use_generic) -{ - unsigned long config = 0, dummy; - - if (use_generic) - rdmsr(MSR_MTRRcap, config, dummy); - else if (is_cpu(AMD) || is_cpu(HYGON)) - config = 2; - else if (is_cpu(CYRIX) || is_cpu(CENTAUR)) - config = 8; - - num_var_ranges = config & 0xff; -} - static void __init init_table(void) { int i, max; @@ -194,20 +173,8 @@ static inline int types_compatible(mtrr_type type1, mtrr_type type2) * Note that the mechanism is the same for UP systems, too; all the SMP stuff * becomes nops. */ -static void -set_mtrr(unsigned int reg, unsigned long base, unsigned long size, mtrr_type type) -{ - struct set_mtrr_data data = { .smp_reg = reg, - .smp_base = base, - .smp_size = size, - .smp_type = type - }; - - stop_machine(mtrr_rendezvous_handler, &data, cpu_online_mask); -} - -static void set_mtrr_cpuslocked(unsigned int reg, unsigned long base, - unsigned long size, mtrr_type type) +static void set_mtrr(unsigned int reg, unsigned long base, unsigned long size, + mtrr_type type) { struct set_mtrr_data data = { .smp_reg = reg, .smp_base = base, @@ -216,6 +183,8 @@ static void set_mtrr_cpuslocked(unsigned int reg, unsigned long base, }; stop_machine_cpuslocked(mtrr_rendezvous_handler, &data, cpu_online_mask); + + generic_rebuild_map(); } /** @@ -337,7 +306,7 @@ int mtrr_add_page(unsigned long base, unsigned long size, /* Search for an empty MTRR */ i = mtrr_if->get_free_region(base, size, replace); if (i >= 0) { - set_mtrr_cpuslocked(i, base, size, type); + set_mtrr(i, base, size, type); if (likely(replace < 0)) { mtrr_usage_table[i] = 1; } else { @@ -345,7 +314,7 @@ int mtrr_add_page(unsigned long base, unsigned long size, if (increment) mtrr_usage_table[i]++; if (unlikely(replace != i)) { - set_mtrr_cpuslocked(replace, 0, 0, 0); + set_mtrr(replace, 0, 0, 0); mtrr_usage_table[replace] = 0; } } @@ -363,7 +332,7 @@ static int mtrr_check(unsigned long base, unsigned long size) { if ((base & (PAGE_SIZE - 1)) || (size & (PAGE_SIZE - 1))) { pr_warn("size and base must be multiples of 4 kiB\n"); - pr_debug("size: 0x%lx base: 0x%lx\n", size, base); + Dprintk("size: 0x%lx base: 0x%lx\n", size, base); dump_stack(); return -1; } @@ -454,8 +423,7 @@ int mtrr_del_page(int reg, unsigned long base, unsigned long size) } } if (reg < 0) { - pr_debug("no MTRR for %lx000,%lx000 found\n", - base, size); + Dprintk("no MTRR for %lx000,%lx000 found\n", base, size); goto out; } } @@ -473,7 +441,7 @@ int mtrr_del_page(int reg, unsigned long base, unsigned long size) goto out; } if (--mtrr_usage_table[reg] < 1) - set_mtrr_cpuslocked(reg, 0, 0, 0); + set_mtrr(reg, 0, 0, 0); error = reg; out: mutex_unlock(&mtrr_mutex); @@ -574,136 +542,54 @@ int arch_phys_wc_index(int handle) } EXPORT_SYMBOL_GPL(arch_phys_wc_index); -/* The suspend/resume methods are only for CPU without MTRR. CPU using generic - * MTRR driver doesn't require this - */ -struct mtrr_value { - mtrr_type ltype; - unsigned long lbase; - unsigned long lsize; -}; - -static struct mtrr_value mtrr_value[MTRR_MAX_VAR_RANGES]; - -static int mtrr_save(void) -{ - int i; - - for (i = 0; i < num_var_ranges; i++) { - mtrr_if->get(i, &mtrr_value[i].lbase, - &mtrr_value[i].lsize, - &mtrr_value[i].ltype); - } - return 0; -} - -static void mtrr_restore(void) -{ - int i; - - for (i = 0; i < num_var_ranges; i++) { - if (mtrr_value[i].lsize) { - set_mtrr(i, mtrr_value[i].lbase, - mtrr_value[i].lsize, - mtrr_value[i].ltype); - } - } -} - - - -static struct syscore_ops mtrr_syscore_ops = { - .suspend = mtrr_save, - .resume = mtrr_restore, -}; - int __initdata changed_by_mtrr_cleanup; -#define SIZE_OR_MASK_BITS(n) (~((1ULL << ((n) - PAGE_SHIFT)) - 1)) /** - * mtrr_bp_init - initialize mtrrs on the boot CPU + * mtrr_bp_init - initialize MTRRs on the boot CPU * * This needs to be called early; before any of the other CPUs are * initialized (i.e. before smp_init()). - * */ void __init mtrr_bp_init(void) { + bool generic_mtrrs = cpu_feature_enabled(X86_FEATURE_MTRR); const char *why = "(not available)"; - u32 phys_addr; + unsigned long config, dummy; - phys_addr = 32; - - if (boot_cpu_has(X86_FEATURE_MTRR)) { - mtrr_if = &generic_mtrr_ops; - size_or_mask = SIZE_OR_MASK_BITS(36); - size_and_mask = 0x00f00000; - phys_addr = 36; + phys_hi_rsvd = GENMASK(31, boot_cpu_data.x86_phys_bits - 32); + if (!generic_mtrrs && mtrr_state.enabled) { /* - * This is an AMD specific MSR, but we assume(hope?) that - * Intel will implement it too when they extend the address - * bus of the Xeon. + * Software overwrite of MTRR state, only for generic case. + * Note that X86_FEATURE_MTRR has been reset in this case. */ - if (cpuid_eax(0x80000000) >= 0x80000008) { - phys_addr = cpuid_eax(0x80000008) & 0xff; - /* CPUID workaround for Intel 0F33/0F34 CPU */ - if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && - boot_cpu_data.x86 == 0xF && - boot_cpu_data.x86_model == 0x3 && - (boot_cpu_data.x86_stepping == 0x3 || - boot_cpu_data.x86_stepping == 0x4)) - phys_addr = 36; + init_table(); + mtrr_build_map(); + pr_info("MTRRs set to read-only\n"); - size_or_mask = SIZE_OR_MASK_BITS(phys_addr); - size_and_mask = ~size_or_mask & 0xfffff00000ULL; - } else if (boot_cpu_data.x86_vendor == X86_VENDOR_CENTAUR && - boot_cpu_data.x86 == 6) { - /* - * VIA C* family have Intel style MTRRs, - * but don't support PAE - */ - size_or_mask = SIZE_OR_MASK_BITS(32); - size_and_mask = 0; - phys_addr = 32; - } - } else { - switch (boot_cpu_data.x86_vendor) { - case X86_VENDOR_AMD: - if (cpu_feature_enabled(X86_FEATURE_K6_MTRR)) { - /* Pre-Athlon (K6) AMD CPU MTRRs */ - mtrr_if = &amd_mtrr_ops; - size_or_mask = SIZE_OR_MASK_BITS(32); - size_and_mask = 0; - } - break; - case X86_VENDOR_CENTAUR: - if (cpu_feature_enabled(X86_FEATURE_CENTAUR_MCR)) { - mtrr_if = ¢aur_mtrr_ops; - size_or_mask = SIZE_OR_MASK_BITS(32); - size_and_mask = 0; - } - break; - case X86_VENDOR_CYRIX: - if (cpu_feature_enabled(X86_FEATURE_CYRIX_ARR)) { - mtrr_if = &cyrix_mtrr_ops; - size_or_mask = SIZE_OR_MASK_BITS(32); - size_and_mask = 0; - } - break; - default: - break; - } + return; } + if (generic_mtrrs) + mtrr_if = &generic_mtrr_ops; + else + mtrr_set_if(); + if (mtrr_enabled()) { - set_num_var_ranges(mtrr_if == &generic_mtrr_ops); + /* Get the number of variable MTRR ranges. */ + if (mtrr_if == &generic_mtrr_ops) + rdmsr(MSR_MTRRcap, config, dummy); + else + config = mtrr_if->var_regs; + num_var_ranges = config & MTRR_CAP_VCNT; + init_table(); if (mtrr_if == &generic_mtrr_ops) { /* BIOS may override */ if (get_mtrr_state()) { memory_caching_control |= CACHE_MTRR; - changed_by_mtrr_cleanup = mtrr_cleanup(phys_addr); + changed_by_mtrr_cleanup = mtrr_cleanup(); + mtrr_build_map(); } else { mtrr_if = NULL; why = "by BIOS"; @@ -730,8 +616,14 @@ void mtrr_save_state(void) smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1); } -static int __init mtrr_init_finialize(void) +static int __init mtrr_init_finalize(void) { + /* + * Map might exist if mtrr_overwrite_state() has been called or if + * mtrr_enabled() returns true. + */ + mtrr_copy_map(); + if (!mtrr_enabled()) return 0; @@ -741,16 +633,8 @@ static int __init mtrr_init_finialize(void) return 0; } - /* - * The CPU has no MTRR and seems to not support SMP. They have - * specific drivers, we use a tricky method to support - * suspend/resume for them. - * - * TBD: is there any system with such CPU which supports - * suspend/resume? If no, we should remove the code. - */ - register_syscore_ops(&mtrr_syscore_ops); + mtrr_register_syscore(); return 0; } -subsys_initcall(mtrr_init_finialize); +subsys_initcall(mtrr_init_finalize); diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.h b/arch/x86/kernel/cpu/mtrr/mtrr.h index 02eb5871492d..5655f253d929 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.h +++ b/arch/x86/kernel/cpu/mtrr/mtrr.h @@ -10,10 +10,13 @@ #define MTRR_CHANGE_MASK_VARIABLE 0x02 #define MTRR_CHANGE_MASK_DEFTYPE 0x04 +extern bool mtrr_debug; +#define Dprintk(x...) do { if (mtrr_debug) pr_info(x); } while (0) + extern unsigned int mtrr_usage_table[MTRR_MAX_VAR_RANGES]; struct mtrr_ops { - u32 vendor; + u32 var_regs; void (*set)(unsigned int reg, unsigned long base, unsigned long size, mtrr_type type); void (*get)(unsigned int reg, unsigned long *base, @@ -51,18 +54,26 @@ void fill_mtrr_var_range(unsigned int index, u32 base_lo, u32 base_hi, u32 mask_lo, u32 mask_hi); bool get_mtrr_state(void); -extern u64 size_or_mask, size_and_mask; extern const struct mtrr_ops *mtrr_if; - -#define is_cpu(vnd) (mtrr_if && mtrr_if->vendor == X86_VENDOR_##vnd) +extern struct mutex mtrr_mutex; extern unsigned int num_var_ranges; extern u64 mtrr_tom2; extern struct mtrr_state_type mtrr_state; +extern u32 phys_hi_rsvd; void mtrr_state_warn(void); const char *mtrr_attrib_to_str(int x); void mtrr_wrmsr(unsigned, unsigned, unsigned); +#ifdef CONFIG_X86_32 +void mtrr_set_if(void); +void mtrr_register_syscore(void); +#else +static inline void mtrr_set_if(void) { } +static inline void mtrr_register_syscore(void) { } +#endif +void mtrr_build_map(void); +void mtrr_copy_map(void); /* CPU specific mtrr_ops vectors. */ extern const struct mtrr_ops amd_mtrr_ops; @@ -70,4 +81,14 @@ extern const struct mtrr_ops cyrix_mtrr_ops; extern const struct mtrr_ops centaur_mtrr_ops; extern int changed_by_mtrr_cleanup; -extern int mtrr_cleanup(unsigned address_bits); +extern int mtrr_cleanup(void); + +/* + * Must be used by code which uses mtrr_if to call platform-specific + * MTRR manipulation functions. + */ +static inline bool mtrr_enabled(void) +{ + return !!mtrr_if; +} +void generic_rebuild_map(void); diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c index 2a0e90fe2abc..91fa70e51004 100644 --- a/arch/x86/kernel/cpu/sgx/encl.c +++ b/arch/x86/kernel/cpu/sgx/encl.c @@ -755,6 +755,7 @@ static void sgx_mmu_notifier_release(struct mmu_notifier *mn, { struct sgx_encl_mm *encl_mm = container_of(mn, struct sgx_encl_mm, mmu_notifier); struct sgx_encl_mm *tmp = NULL; + bool found = false; /* * The enclave itself can remove encl_mm. Note, objects can't be moved @@ -764,12 +765,13 @@ static void sgx_mmu_notifier_release(struct mmu_notifier *mn, list_for_each_entry(tmp, &encl_mm->encl->mm_list, list) { if (tmp == encl_mm) { list_del_rcu(&encl_mm->list); + found = true; break; } } spin_unlock(&encl_mm->encl->mm_lock); - if (tmp == encl_mm) { + if (found) { synchronize_srcu(&encl_mm->encl->srcu); mmu_notifier_put(mn); } diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 16babff771bd..0cccfeb67c3a 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -1037,6 +1037,8 @@ void __init setup_arch(char **cmdline_p) /* * VMware detection requires dmi to be available, so this * needs to be done after dmi_setup(), for the boot CPU. + * For some guest types (Xen PV, SEV-SNP, TDX) it is required to be + * called before cache_bp_init() for setting up MTRR state. */ init_hypervisor_platform(); diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index a0af3908ed49..1ee7bed453de 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -113,8 +113,6 @@ struct ghcb_state { }; static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data); -DEFINE_STATIC_KEY_FALSE(sev_es_enable_key); - static DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa); struct sev_config { @@ -1355,9 +1353,6 @@ void __init sev_es_init_vc_handling(void) sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); } - /* Enable SEV-ES special handling */ - static_branch_enable(&sev_es_enable_key); - /* Initialize per-cpu GHCB pages */ for_each_possible_cpu(cpu) { alloc_runtime_data(cpu); diff --git a/arch/x86/lib/csum-partial_64.c b/arch/x86/lib/csum-partial_64.c index 50734a23034c..cea25ca8b8cf 100644 --- a/arch/x86/lib/csum-partial_64.c +++ b/arch/x86/lib/csum-partial_64.c @@ -5,22 +5,34 @@ * This file contains network checksum routines that are better done * in an architecture-specific manner due to speed. */ - + #include #include #include #include -static inline unsigned short from32to16(unsigned a) +static inline unsigned short from32to16(unsigned a) { - unsigned short b = a >> 16; + unsigned short b = a >> 16; asm("addw %w2,%w0\n\t" - "adcw $0,%w0\n" + "adcw $0,%w0\n" : "=r" (b) : "0" (b), "r" (a)); return b; } +static inline __wsum csum_tail(u64 temp64, int odd) +{ + unsigned int result; + + result = add32_with_carry(temp64 >> 32, temp64 & 0xffffffff); + if (unlikely(odd)) { + result = from32to16(result); + result = ((result >> 8) & 0xff) | ((result & 0xff) << 8); + } + return (__force __wsum)result; +} + /* * Do a checksum on an arbitrary memory area. * Returns a 32bit checksum. @@ -35,7 +47,7 @@ static inline unsigned short from32to16(unsigned a) __wsum csum_partial(const void *buff, int len, __wsum sum) { u64 temp64 = (__force u64)sum; - unsigned odd, result; + unsigned odd; odd = 1 & (unsigned long) buff; if (unlikely(odd)) { @@ -47,21 +59,52 @@ __wsum csum_partial(const void *buff, int len, __wsum sum) buff++; } - while (unlikely(len >= 64)) { + /* + * len == 40 is the hot case due to IPv6 headers, but annotating it likely() + * has noticeable negative affect on codegen for all other cases with + * minimal performance benefit here. + */ + if (len == 40) { asm("addq 0*8(%[src]),%[res]\n\t" "adcq 1*8(%[src]),%[res]\n\t" "adcq 2*8(%[src]),%[res]\n\t" "adcq 3*8(%[src]),%[res]\n\t" "adcq 4*8(%[src]),%[res]\n\t" - "adcq 5*8(%[src]),%[res]\n\t" - "adcq 6*8(%[src]),%[res]\n\t" - "adcq 7*8(%[src]),%[res]\n\t" "adcq $0,%[res]" - : [res] "+r" (temp64) - : [src] "r" (buff) - : "memory"); - buff += 64; - len -= 64; + : [res] "+r"(temp64) + : [src] "r"(buff), "m"(*(const char(*)[40])buff)); + return csum_tail(temp64, odd); + } + if (unlikely(len >= 64)) { + /* + * Extra accumulators for better ILP in the loop. + */ + u64 tmp_accum, tmp_carries; + + asm("xorl %k[tmp_accum],%k[tmp_accum]\n\t" + "xorl %k[tmp_carries],%k[tmp_carries]\n\t" + "subl $64, %[len]\n\t" + "1:\n\t" + "addq 0*8(%[src]),%[res]\n\t" + "adcq 1*8(%[src]),%[res]\n\t" + "adcq 2*8(%[src]),%[res]\n\t" + "adcq 3*8(%[src]),%[res]\n\t" + "adcl $0,%k[tmp_carries]\n\t" + "addq 4*8(%[src]),%[tmp_accum]\n\t" + "adcq 5*8(%[src]),%[tmp_accum]\n\t" + "adcq 6*8(%[src]),%[tmp_accum]\n\t" + "adcq 7*8(%[src]),%[tmp_accum]\n\t" + "adcl $0,%k[tmp_carries]\n\t" + "addq $64, %[src]\n\t" + "subl $64, %[len]\n\t" + "jge 1b\n\t" + "addq %[tmp_accum],%[res]\n\t" + "adcq %[tmp_carries],%[res]\n\t" + "adcq $0,%[res]" + : [tmp_accum] "=&r"(tmp_accum), + [tmp_carries] "=&r"(tmp_carries), [res] "+r"(temp64), + [len] "+r"(len), [src] "+r"(buff) + : "m"(*(const char *)buff)); } if (len & 32) { @@ -70,45 +113,37 @@ __wsum csum_partial(const void *buff, int len, __wsum sum) "adcq 2*8(%[src]),%[res]\n\t" "adcq 3*8(%[src]),%[res]\n\t" "adcq $0,%[res]" - : [res] "+r" (temp64) - : [src] "r" (buff) - : "memory"); + : [res] "+r"(temp64) + : [src] "r"(buff), "m"(*(const char(*)[32])buff)); buff += 32; } if (len & 16) { asm("addq 0*8(%[src]),%[res]\n\t" "adcq 1*8(%[src]),%[res]\n\t" "adcq $0,%[res]" - : [res] "+r" (temp64) - : [src] "r" (buff) - : "memory"); + : [res] "+r"(temp64) + : [src] "r"(buff), "m"(*(const char(*)[16])buff)); buff += 16; } if (len & 8) { asm("addq 0*8(%[src]),%[res]\n\t" "adcq $0,%[res]" - : [res] "+r" (temp64) - : [src] "r" (buff) - : "memory"); + : [res] "+r"(temp64) + : [src] "r"(buff), "m"(*(const char(*)[8])buff)); buff += 8; } if (len & 7) { - unsigned int shift = (8 - (len & 7)) * 8; + unsigned int shift = (-len << 3) & 63; unsigned long trail; trail = (load_unaligned_zeropad(buff) << shift) >> shift; asm("addq %[trail],%[res]\n\t" "adcq $0,%[res]" - : [res] "+r" (temp64) - : [trail] "r" (trail)); + : [res] "+r"(temp64) + : [trail] "r"(trail)); } - result = add32_with_carry(temp64 >> 32, temp64 & 0xffffffff); - if (unlikely(odd)) { - result = from32to16(result); - result = ((result >> 8) & 0xff) | ((result & 0xff) << 8); - } - return (__force __wsum)result; + return csum_tail(temp64, odd); } EXPORT_SYMBOL(csum_partial); @@ -118,6 +153,6 @@ EXPORT_SYMBOL(csum_partial); */ __sum16 ip_compute_csum(const void *buff, int len) { - return csum_fold(csum_partial(buff,len,0)); + return csum_fold(csum_partial(buff, len, 0)); } EXPORT_SYMBOL(ip_compute_csum); diff --git a/arch/x86/lib/getuser.S b/arch/x86/lib/getuser.S index b64a2bd1a1ef..9c63713477bb 100644 --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -143,43 +143,43 @@ SYM_FUNC_END(__get_user_nocheck_8) EXPORT_SYMBOL(__get_user_nocheck_8) -SYM_CODE_START_LOCAL(.Lbad_get_user_clac) +SYM_CODE_START_LOCAL(__get_user_handle_exception) ASM_CLAC .Lbad_get_user: xor %edx,%edx mov $(-EFAULT),%_ASM_AX RET -SYM_CODE_END(.Lbad_get_user_clac) +SYM_CODE_END(__get_user_handle_exception) #ifdef CONFIG_X86_32 -SYM_CODE_START_LOCAL(.Lbad_get_user_8_clac) +SYM_CODE_START_LOCAL(__get_user_8_handle_exception) ASM_CLAC bad_get_user_8: xor %edx,%edx xor %ecx,%ecx mov $(-EFAULT),%_ASM_AX RET -SYM_CODE_END(.Lbad_get_user_8_clac) +SYM_CODE_END(__get_user_8_handle_exception) #endif /* get_user */ - _ASM_EXTABLE(1b, .Lbad_get_user_clac) - _ASM_EXTABLE(2b, .Lbad_get_user_clac) - _ASM_EXTABLE(3b, .Lbad_get_user_clac) + _ASM_EXTABLE(1b, __get_user_handle_exception) + _ASM_EXTABLE(2b, __get_user_handle_exception) + _ASM_EXTABLE(3b, __get_user_handle_exception) #ifdef CONFIG_X86_64 - _ASM_EXTABLE(4b, .Lbad_get_user_clac) + _ASM_EXTABLE(4b, __get_user_handle_exception) #else - _ASM_EXTABLE(4b, .Lbad_get_user_8_clac) - _ASM_EXTABLE(5b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(4b, __get_user_8_handle_exception) + _ASM_EXTABLE(5b, __get_user_8_handle_exception) #endif /* __get_user */ - _ASM_EXTABLE(6b, .Lbad_get_user_clac) - _ASM_EXTABLE(7b, .Lbad_get_user_clac) - _ASM_EXTABLE(8b, .Lbad_get_user_clac) + _ASM_EXTABLE(6b, __get_user_handle_exception) + _ASM_EXTABLE(7b, __get_user_handle_exception) + _ASM_EXTABLE(8b, __get_user_handle_exception) #ifdef CONFIG_X86_64 - _ASM_EXTABLE(9b, .Lbad_get_user_clac) + _ASM_EXTABLE(9b, __get_user_handle_exception) #else - _ASM_EXTABLE(9b, .Lbad_get_user_8_clac) - _ASM_EXTABLE(10b, .Lbad_get_user_8_clac) + _ASM_EXTABLE(9b, __get_user_8_handle_exception) + _ASM_EXTABLE(10b, __get_user_8_handle_exception) #endif diff --git a/arch/x86/lib/putuser.S b/arch/x86/lib/putuser.S index 3062d09a776d..1451e0c4ae22 100644 --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -131,22 +131,22 @@ SYM_FUNC_START(__put_user_nocheck_8) SYM_FUNC_END(__put_user_nocheck_8) EXPORT_SYMBOL(__put_user_nocheck_8) -SYM_CODE_START_LOCAL(.Lbad_put_user_clac) +SYM_CODE_START_LOCAL(__put_user_handle_exception) ASM_CLAC .Lbad_put_user: movl $-EFAULT,%ecx RET -SYM_CODE_END(.Lbad_put_user_clac) +SYM_CODE_END(__put_user_handle_exception) - _ASM_EXTABLE(1b, .Lbad_put_user_clac) - _ASM_EXTABLE(2b, .Lbad_put_user_clac) - _ASM_EXTABLE(3b, .Lbad_put_user_clac) - _ASM_EXTABLE(4b, .Lbad_put_user_clac) - _ASM_EXTABLE(5b, .Lbad_put_user_clac) - _ASM_EXTABLE(6b, .Lbad_put_user_clac) - _ASM_EXTABLE(7b, .Lbad_put_user_clac) - _ASM_EXTABLE(9b, .Lbad_put_user_clac) + _ASM_EXTABLE(1b, __put_user_handle_exception) + _ASM_EXTABLE(2b, __put_user_handle_exception) + _ASM_EXTABLE(3b, __put_user_handle_exception) + _ASM_EXTABLE(4b, __put_user_handle_exception) + _ASM_EXTABLE(5b, __put_user_handle_exception) + _ASM_EXTABLE(6b, __put_user_handle_exception) + _ASM_EXTABLE(7b, __put_user_handle_exception) + _ASM_EXTABLE(9b, __put_user_handle_exception) #ifdef CONFIG_X86_32 - _ASM_EXTABLE(8b, .Lbad_put_user_clac) - _ASM_EXTABLE(10b, .Lbad_put_user_clac) + _ASM_EXTABLE(8b, __put_user_handle_exception) + _ASM_EXTABLE(10b, __put_user_handle_exception) #endif diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index c6efcf559d88..bfe22fd5a1d7 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -612,7 +612,7 @@ void __init sme_enable(struct boot_params *bp) out: if (sme_me_mask) { physical_mask &= ~sme_me_mask; - cc_set_vendor(CC_VENDOR_AMD); + cc_vendor = CC_VENDOR_AMD; cc_set_mask(sme_me_mask); } } diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index e4f499eb0f29..15a8009a4480 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -702,14 +702,8 @@ void p4d_clear_huge(p4d_t *p4d) * pud_set_huge - setup kernel PUD mapping * * MTRRs can override PAT memory types with 4KiB granularity. Therefore, this - * function sets up a huge page only if any of the following conditions are met: - * - * - MTRRs are disabled, or - * - * - MTRRs are enabled and the range is completely covered by a single MTRR, or - * - * - MTRRs are enabled and the corresponding MTRR memory type is WB, which - * has no effect on the requested PAT memory type. + * function sets up a huge page only if the complete range has the same MTRR + * caching mode. * * Callers should try to decrease page size (1GB -> 2MB -> 4K) if the bigger * page mapping attempt fails. @@ -718,11 +712,10 @@ void p4d_clear_huge(p4d_t *p4d) */ int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) { - u8 mtrr, uniform; + u8 uniform; - mtrr = mtrr_type_lookup(addr, addr + PUD_SIZE, &uniform); - if ((mtrr != MTRR_TYPE_INVALID) && (!uniform) && - (mtrr != MTRR_TYPE_WRBACK)) + mtrr_type_lookup(addr, addr + PUD_SIZE, &uniform); + if (!uniform) return 0; /* Bail out if we are we on a populated non-leaf entry: */ @@ -745,11 +738,10 @@ int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) */ int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot) { - u8 mtrr, uniform; + u8 uniform; - mtrr = mtrr_type_lookup(addr, addr + PMD_SIZE, &uniform); - if ((mtrr != MTRR_TYPE_INVALID) && (!uniform) && - (mtrr != MTRR_TYPE_WRBACK)) { + mtrr_type_lookup(addr, addr + PMD_SIZE, &uniform); + if (!uniform) { pr_warn_once("%s: Cannot satisfy [mem %#010llx-%#010llx] with a huge-page mapping due to MTRR override.\n", __func__, addr, addr + PMD_SIZE); return 0; diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 093b78c8bbec..93b658248d01 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -68,6 +68,7 @@ #include #include #include +#include #include #include #include @@ -119,6 +120,54 @@ static int __init parse_xen_msr_safe(char *str) } early_param("xen_msr_safe", parse_xen_msr_safe); +/* Get MTRR settings from Xen and put them into mtrr_state. */ +static void __init xen_set_mtrr_data(void) +{ +#ifdef CONFIG_MTRR + struct xen_platform_op op = { + .cmd = XENPF_read_memtype, + .interface_version = XENPF_INTERFACE_VERSION, + }; + unsigned int reg; + unsigned long mask; + uint32_t eax, width; + static struct mtrr_var_range var[MTRR_MAX_VAR_RANGES] __initdata; + + /* Get physical address width (only 64-bit cpus supported). */ + width = 36; + eax = cpuid_eax(0x80000000); + if ((eax >> 16) == 0x8000 && eax >= 0x80000008) { + eax = cpuid_eax(0x80000008); + width = eax & 0xff; + } + + for (reg = 0; reg < MTRR_MAX_VAR_RANGES; reg++) { + op.u.read_memtype.reg = reg; + if (HYPERVISOR_platform_op(&op)) + break; + + /* + * Only called in dom0, which has all RAM PFNs mapped at + * RAM MFNs, and all PCI space etc. is identity mapped. + * This means we can treat MFN == PFN regarding MTRR settings. + */ + var[reg].base_lo = op.u.read_memtype.type; + var[reg].base_lo |= op.u.read_memtype.mfn << PAGE_SHIFT; + var[reg].base_hi = op.u.read_memtype.mfn >> (32 - PAGE_SHIFT); + mask = ~((op.u.read_memtype.nr_mfns << PAGE_SHIFT) - 1); + mask &= (1UL << width) - 1; + if (mask) + mask |= MTRR_PHYSMASK_V; + var[reg].mask_lo = mask; + var[reg].mask_hi = mask >> 32; + } + + /* Only overwrite MTRR state if any MTRR could be got from Xen. */ + if (reg) + mtrr_overwrite_state(var, reg, MTRR_TYPE_UNCACHABLE); +#endif +} + static void __init xen_pv_init_platform(void) { /* PV guests can't operate virtio devices without grants. */ @@ -135,6 +184,11 @@ static void __init xen_pv_init_platform(void) /* pvclock is in shared info area */ xen_init_time_ops(); + + if (xen_initial_domain()) + xen_set_mtrr_data(); + else + mtrr_overwrite_state(NULL, 0, MTRR_TYPE_WRBACK); } static void __init xen_pv_guest_late_init(void) diff --git a/drivers/acpi/acpi_ffh.c b/drivers/acpi/acpi_ffh.c index 19aff808bbb8..8d5126963dc7 100644 --- a/drivers/acpi/acpi_ffh.c +++ b/drivers/acpi/acpi_ffh.c @@ -9,8 +9,6 @@ #include #include -#include - static struct acpi_ffh_info ffh_ctx; int __weak acpi_ffh_address_space_arch_setup(void *handler_ctxt, diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c index 77186f084d3a..539e700de4d2 100644 --- a/drivers/acpi/acpi_lpss.c +++ b/drivers/acpi/acpi_lpss.c @@ -201,11 +201,19 @@ static void byt_i2c_setup(struct lpss_private_data *pdata) writel(0, pdata->mmio_base + LPSS_I2C_ENABLE); } -/* BSW PWM used for backlight control by the i915 driver */ +/* + * BSW PWM1 is used for backlight control by the i915 driver + * BSW PWM2 is used for backlight control for fixed (etched into the glass) + * touch controls on some models. These touch-controls have specialized + * drivers which know they need the "pwm_soc_lpss_2" con-id. + */ static struct pwm_lookup bsw_pwm_lookup[] = { PWM_LOOKUP_WITH_MODULE("80862288:00", 0, "0000:00:02.0", "pwm_soc_backlight", 0, PWM_POLARITY_NORMAL, "pwm-lpss-platform"), + PWM_LOOKUP_WITH_MODULE("80862289:00", 0, NULL, + "pwm_soc_lpss_2", 0, PWM_POLARITY_NORMAL, + "pwm-lpss-platform"), }; static void bsw_pwm_setup(struct lpss_private_data *pdata) diff --git a/drivers/acpi/acpi_pad.c b/drivers/acpi/acpi_pad.c index 02f1a1b1143c..7a453c5ff303 100644 --- a/drivers/acpi/acpi_pad.c +++ b/drivers/acpi/acpi_pad.c @@ -66,6 +66,7 @@ static void power_saving_mwait_init(void) case X86_VENDOR_AMD: case X86_VENDOR_INTEL: case X86_VENDOR_ZHAOXIN: + case X86_VENDOR_CENTAUR: /* * AMD Fam10h TSC will tick in all * C/P/S0/S1 states when this bit is set. diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c index 7514e38d5640..5427e49e646b 100644 --- a/drivers/acpi/apei/bert.c +++ b/drivers/acpi/apei/bert.c @@ -34,7 +34,7 @@ #define ACPI_BERT_PRINT_MAX_RECORDS 5 #define ACPI_BERT_PRINT_MAX_LEN 1024 -static int bert_disable; +static int bert_disable __initdata; /* * Print "all" the error records in the BERT table, but avoid huge spam to diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c index 34ad071a64e9..ef59d6ea16da 100644 --- a/drivers/acpi/apei/ghes.c +++ b/drivers/acpi/apei/ghes.c @@ -152,7 +152,6 @@ struct ghes_vendor_record_entry { }; static struct gen_pool *ghes_estatus_pool; -static unsigned long ghes_estatus_pool_size_request; static struct ghes_estatus_cache __rcu *ghes_estatus_caches[GHES_ESTATUS_CACHES_SIZE]; static atomic_t ghes_estatus_cache_alloced; @@ -191,7 +190,6 @@ int ghes_estatus_pool_init(unsigned int num_ghes) len = GHES_ESTATUS_CACHE_AVG_SIZE * GHES_ESTATUS_CACHE_ALLOCED_MAX; len += (num_ghes * GHES_ESOURCE_PREALLOC_MAX_SIZE); - ghes_estatus_pool_size_request = PAGE_ALIGN(len); addr = (unsigned long)vmalloc(PAGE_ALIGN(len)); if (!addr) goto err_pool_alloc; @@ -1544,6 +1542,8 @@ struct list_head *ghes_get_devices(void) pr_warn_once("Force-loading ghes_edac on an unsupported platform. You're on your own!\n"); } + } else if (list_empty(&ghes_devs)) { + return NULL; } return &ghes_devs; diff --git a/drivers/acpi/arm64/Makefile b/drivers/acpi/arm64/Makefile index e21a9e84e394..f81fe24894b2 100644 --- a/drivers/acpi/arm64/Makefile +++ b/drivers/acpi/arm64/Makefile @@ -3,4 +3,4 @@ obj-$(CONFIG_ACPI_AGDI) += agdi.o obj-$(CONFIG_ACPI_IORT) += iort.o obj-$(CONFIG_ACPI_GTDT) += gtdt.o obj-$(CONFIG_ACPI_APMT) += apmt.o -obj-y += dma.o +obj-y += dma.o init.o diff --git a/drivers/acpi/arm64/agdi.c b/drivers/acpi/arm64/agdi.c index f605302395c3..8b3c7d42b41b 100644 --- a/drivers/acpi/arm64/agdi.c +++ b/drivers/acpi/arm64/agdi.c @@ -9,11 +9,11 @@ #define pr_fmt(fmt) "ACPI: AGDI: " fmt #include -#include #include #include #include #include +#include "init.h" struct agdi_data { int sdei_event; diff --git a/drivers/acpi/arm64/apmt.c b/drivers/acpi/arm64/apmt.c index 8cab69fa5d59..bb010f6164e5 100644 --- a/drivers/acpi/arm64/apmt.c +++ b/drivers/acpi/arm64/apmt.c @@ -10,10 +10,10 @@ #define pr_fmt(fmt) "ACPI: APMT: " fmt #include -#include #include #include #include +#include "init.h" #define DEV_NAME "arm-cs-arch-pmu" @@ -35,11 +35,13 @@ static int __init apmt_init_resources(struct resource *res, num_res++; - res[num_res].start = node->base_address1; - res[num_res].end = node->base_address1 + SZ_4K - 1; - res[num_res].flags = IORESOURCE_MEM; + if (node->flags & ACPI_APMT_FLAGS_DUAL_PAGE) { + res[num_res].start = node->base_address1; + res[num_res].end = node->base_address1 + SZ_4K - 1; + res[num_res].flags = IORESOURCE_MEM; - num_res++; + num_res++; + } if (node->ovflw_irq != 0) { trigger = (node->ovflw_irq_flags & ACPI_APMT_OVFLW_IRQ_FLAGS_MODE); diff --git a/drivers/acpi/arm64/init.c b/drivers/acpi/arm64/init.c new file mode 100644 index 000000000000..d3ce53dda122 --- /dev/null +++ b/drivers/acpi/arm64/init.c @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include "init.h" + +void __init acpi_arm_init(void) +{ + if (IS_ENABLED(CONFIG_ACPI_AGDI)) + acpi_agdi_init(); + if (IS_ENABLED(CONFIG_ACPI_APMT)) + acpi_apmt_init(); + if (IS_ENABLED(CONFIG_ACPI_IORT)) + acpi_iort_init(); +} diff --git a/drivers/acpi/arm64/init.h b/drivers/acpi/arm64/init.h new file mode 100644 index 000000000000..a1715a2a34e9 --- /dev/null +++ b/drivers/acpi/arm64/init.h @@ -0,0 +1,6 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#include + +void __init acpi_agdi_init(void); +void __init acpi_apmt_init(void); +void __init acpi_iort_init(void); diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c index 38fb84974f35..3631230a61c8 100644 --- a/drivers/acpi/arm64/iort.c +++ b/drivers/acpi/arm64/iort.c @@ -19,6 +19,7 @@ #include #include #include +#include "init.h" #define IORT_TYPE_MASK(type) (1 << (type)) #define IORT_MSI_TYPE (1 << ACPI_IORT_NODE_ITS_GROUP) diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c index d161ff707de4..e3e0bd0c5a50 100644 --- a/drivers/acpi/bus.c +++ b/drivers/acpi/bus.c @@ -26,9 +26,6 @@ #include #include #endif -#include -#include -#include #include #include #include @@ -530,65 +527,30 @@ static void acpi_notify_device(acpi_handle handle, u32 event, void *data) acpi_drv->ops.notify(device, event); } -static void acpi_notify_device_fixed(void *data) -{ - struct acpi_device *device = data; - - /* Fixed hardware devices have no handles */ - acpi_notify_device(NULL, ACPI_FIXED_HARDWARE_EVENT, device); -} - -static u32 acpi_device_fixed_event(void *data) -{ - acpi_os_execute(OSL_NOTIFY_HANDLER, acpi_notify_device_fixed, data); - return ACPI_INTERRUPT_HANDLED; -} - static int acpi_device_install_notify_handler(struct acpi_device *device, struct acpi_driver *acpi_drv) { + u32 type = acpi_drv->flags & ACPI_DRIVER_ALL_NOTIFY_EVENTS ? + ACPI_ALL_NOTIFY : ACPI_DEVICE_NOTIFY; acpi_status status; - if (device->device_type == ACPI_BUS_TYPE_POWER_BUTTON) { - status = - acpi_install_fixed_event_handler(ACPI_EVENT_POWER_BUTTON, - acpi_device_fixed_event, - device); - } else if (device->device_type == ACPI_BUS_TYPE_SLEEP_BUTTON) { - status = - acpi_install_fixed_event_handler(ACPI_EVENT_SLEEP_BUTTON, - acpi_device_fixed_event, - device); - } else { - u32 type = acpi_drv->flags & ACPI_DRIVER_ALL_NOTIFY_EVENTS ? - ACPI_ALL_NOTIFY : ACPI_DEVICE_NOTIFY; - - status = acpi_install_notify_handler(device->handle, type, - acpi_notify_device, - device); - } - + status = acpi_install_notify_handler(device->handle, type, + acpi_notify_device, device); if (ACPI_FAILURE(status)) return -EINVAL; + return 0; } static void acpi_device_remove_notify_handler(struct acpi_device *device, struct acpi_driver *acpi_drv) { - if (device->device_type == ACPI_BUS_TYPE_POWER_BUTTON) { - acpi_remove_fixed_event_handler(ACPI_EVENT_POWER_BUTTON, - acpi_device_fixed_event); - } else if (device->device_type == ACPI_BUS_TYPE_SLEEP_BUTTON) { - acpi_remove_fixed_event_handler(ACPI_EVENT_SLEEP_BUTTON, - acpi_device_fixed_event); - } else { - u32 type = acpi_drv->flags & ACPI_DRIVER_ALL_NOTIFY_EVENTS ? + u32 type = acpi_drv->flags & ACPI_DRIVER_ALL_NOTIFY_EVENTS ? ACPI_ALL_NOTIFY : ACPI_DEVICE_NOTIFY; - acpi_remove_notify_handler(device->handle, type, - acpi_notify_device); - } + acpi_remove_notify_handler(device->handle, type, + acpi_notify_device); + acpi_os_wait_events_complete(); } @@ -1408,7 +1370,7 @@ static int __init acpi_init(void) acpi_init_ffh(); pci_mmcfg_late_init(); - acpi_iort_init(); + acpi_arm_init(); acpi_viot_early_init(); acpi_hest_init(); acpi_ghes_init(); @@ -1420,8 +1382,6 @@ static int __init acpi_init(void) acpi_debugger_init(); acpi_setup_sb_notify_handler(); acpi_viot_init(); - acpi_agdi_init(); - acpi_apmt_init(); return 0; } diff --git a/drivers/acpi/button.c b/drivers/acpi/button.c index 475e1eddfa3b..1e76a64cce0a 100644 --- a/drivers/acpi/button.c +++ b/drivers/acpi/button.c @@ -77,6 +77,15 @@ static const struct dmi_system_id dmi_lid_quirks[] = { }, .driver_data = (void *)(long)ACPI_BUTTON_LID_INIT_DISABLED, }, + { + /* Nextbook Ares 8A tablet, _LID device always reports lid closed */ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "Insyde"), + DMI_MATCH(DMI_PRODUCT_NAME, "CherryTrail"), + DMI_MATCH(DMI_BIOS_VERSION, "M882"), + }, + .driver_data = (void *)(long)ACPI_BUTTON_LID_INIT_DISABLED, + }, { /* * Lenovo Yoga 9 14ITL5, initial notification of the LID device @@ -126,7 +135,6 @@ static const struct dmi_system_id dmi_lid_quirks[] = { static int acpi_button_add(struct acpi_device *device); static void acpi_button_remove(struct acpi_device *device); -static void acpi_button_notify(struct acpi_device *device, u32 event); #ifdef CONFIG_PM_SLEEP static int acpi_button_suspend(struct device *dev); @@ -144,7 +152,6 @@ static struct acpi_driver acpi_button_driver = { .ops = { .add = acpi_button_add, .remove = acpi_button_remove, - .notify = acpi_button_notify, }, .drv.pm = &acpi_button_pm, }; @@ -400,45 +407,65 @@ static void acpi_lid_initialize_state(struct acpi_device *device) button->lid_state_initialized = true; } -static void acpi_button_notify(struct acpi_device *device, u32 event) +static void acpi_lid_notify(acpi_handle handle, u32 event, void *data) { - struct acpi_button *button = acpi_driver_data(device); - struct input_dev *input; + struct acpi_device *device = data; + struct acpi_button *button; - switch (event) { - case ACPI_FIXED_HARDWARE_EVENT: - event = ACPI_BUTTON_NOTIFY_STATUS; - fallthrough; - case ACPI_BUTTON_NOTIFY_STATUS: - input = button->input; - if (button->type == ACPI_BUTTON_TYPE_LID) { - if (button->lid_state_initialized) - acpi_lid_update_state(device, true); - } else { - int keycode; - - acpi_pm_wakeup_event(&device->dev); - if (button->suspended) - break; - - keycode = test_bit(KEY_SLEEP, input->keybit) ? - KEY_SLEEP : KEY_POWER; - input_report_key(input, keycode, 1); - input_sync(input); - input_report_key(input, keycode, 0); - input_sync(input); - - acpi_bus_generate_netlink_event( - device->pnp.device_class, - dev_name(&device->dev), - event, ++button->pushed); - } - break; - default: + if (event != ACPI_BUTTON_NOTIFY_STATUS) { acpi_handle_debug(device->handle, "Unsupported event [0x%x]\n", event); - break; + return; } + + button = acpi_driver_data(device); + if (!button->lid_state_initialized) + return; + + acpi_lid_update_state(device, true); +} + +static void acpi_button_notify(acpi_handle handle, u32 event, void *data) +{ + struct acpi_device *device = data; + struct acpi_button *button; + struct input_dev *input; + int keycode; + + if (event != ACPI_BUTTON_NOTIFY_STATUS) { + acpi_handle_debug(device->handle, "Unsupported event [0x%x]\n", + event); + return; + } + + acpi_pm_wakeup_event(&device->dev); + + button = acpi_driver_data(device); + if (button->suspended) + return; + + input = button->input; + keycode = test_bit(KEY_SLEEP, input->keybit) ? KEY_SLEEP : KEY_POWER; + + input_report_key(input, keycode, 1); + input_sync(input); + input_report_key(input, keycode, 0); + input_sync(input); + + acpi_bus_generate_netlink_event(device->pnp.device_class, + dev_name(&device->dev), + event, ++button->pushed); +} + +static void acpi_button_notify_run(void *data) +{ + acpi_button_notify(NULL, ACPI_BUTTON_NOTIFY_STATUS, data); +} + +static u32 acpi_button_event(void *data) +{ + acpi_os_execute(OSL_NOTIFY_HANDLER, acpi_button_notify_run, data); + return ACPI_INTERRUPT_HANDLED; } #ifdef CONFIG_PM_SLEEP @@ -480,11 +507,13 @@ static int acpi_lid_input_open(struct input_dev *input) static int acpi_button_add(struct acpi_device *device) { + acpi_notify_handler handler; struct acpi_button *button; struct input_dev *input; const char *hid = acpi_device_hid(device); + acpi_status status; char *name, *class; - int error; + int error = 0; if (!strcmp(hid, ACPI_BUTTON_HID_LID) && lid_init_state == ACPI_BUTTON_LID_INIT_DISABLED) @@ -508,17 +537,20 @@ static int acpi_button_add(struct acpi_device *device) if (!strcmp(hid, ACPI_BUTTON_HID_POWER) || !strcmp(hid, ACPI_BUTTON_HID_POWERF)) { button->type = ACPI_BUTTON_TYPE_POWER; + handler = acpi_button_notify; strcpy(name, ACPI_BUTTON_DEVICE_NAME_POWER); sprintf(class, "%s/%s", ACPI_BUTTON_CLASS, ACPI_BUTTON_SUBCLASS_POWER); } else if (!strcmp(hid, ACPI_BUTTON_HID_SLEEP) || !strcmp(hid, ACPI_BUTTON_HID_SLEEPF)) { button->type = ACPI_BUTTON_TYPE_SLEEP; + handler = acpi_button_notify; strcpy(name, ACPI_BUTTON_DEVICE_NAME_SLEEP); sprintf(class, "%s/%s", ACPI_BUTTON_CLASS, ACPI_BUTTON_SUBCLASS_SLEEP); } else if (!strcmp(hid, ACPI_BUTTON_HID_LID)) { button->type = ACPI_BUTTON_TYPE_LID; + handler = acpi_lid_notify; strcpy(name, ACPI_BUTTON_DEVICE_NAME_LID); sprintf(class, "%s/%s", ACPI_BUTTON_CLASS, ACPI_BUTTON_SUBCLASS_LID); @@ -526,12 +558,15 @@ static int acpi_button_add(struct acpi_device *device) } else { pr_info("Unsupported hid [%s]\n", hid); error = -ENODEV; - goto err_free_input; } - error = acpi_button_add_fs(device); - if (error) - goto err_free_input; + if (!error) + error = acpi_button_add_fs(device); + + if (error) { + input_free_device(input); + goto err_free_button; + } snprintf(button->phys, sizeof(button->phys), "%s/button/input0", hid); @@ -559,6 +594,29 @@ static int acpi_button_add(struct acpi_device *device) error = input_register_device(input); if (error) goto err_remove_fs; + + switch (device->device_type) { + case ACPI_BUS_TYPE_POWER_BUTTON: + status = acpi_install_fixed_event_handler(ACPI_EVENT_POWER_BUTTON, + acpi_button_event, + device); + break; + case ACPI_BUS_TYPE_SLEEP_BUTTON: + status = acpi_install_fixed_event_handler(ACPI_EVENT_SLEEP_BUTTON, + acpi_button_event, + device); + break; + default: + status = acpi_install_notify_handler(device->handle, + ACPI_DEVICE_NOTIFY, handler, + device); + break; + } + if (ACPI_FAILURE(status)) { + error = -ENODEV; + goto err_input_unregister; + } + if (button->type == ACPI_BUTTON_TYPE_LID) { /* * This assumes there's only one lid device, or if there are @@ -571,11 +629,11 @@ static int acpi_button_add(struct acpi_device *device) pr_info("%s [%s]\n", name, acpi_device_bid(device)); return 0; - err_remove_fs: +err_input_unregister: + input_unregister_device(input); +err_remove_fs: acpi_button_remove_fs(device); - err_free_input: - input_free_device(input); - err_free_button: +err_free_button: kfree(button); return error; } @@ -584,6 +642,24 @@ static void acpi_button_remove(struct acpi_device *device) { struct acpi_button *button = acpi_driver_data(device); + switch (device->device_type) { + case ACPI_BUS_TYPE_POWER_BUTTON: + acpi_remove_fixed_event_handler(ACPI_EVENT_POWER_BUTTON, + acpi_button_event); + break; + case ACPI_BUS_TYPE_SLEEP_BUTTON: + acpi_remove_fixed_event_handler(ACPI_EVENT_SLEEP_BUTTON, + acpi_button_event); + break; + default: + acpi_remove_notify_handler(device->handle, ACPI_DEVICE_NOTIFY, + button->type == ACPI_BUTTON_TYPE_LID ? + acpi_lid_notify : + acpi_button_notify); + break; + } + acpi_os_wait_events_complete(); + acpi_button_remove_fs(device); input_unregister_device(button->input); kfree(button); diff --git a/drivers/acpi/ec.c b/drivers/acpi/ec.c index 928899ab9502..8569f55e55b6 100644 --- a/drivers/acpi/ec.c +++ b/drivers/acpi/ec.c @@ -662,21 +662,6 @@ static void advance_transaction(struct acpi_ec *ec, bool interrupt) ec_dbg_stm("%s (%d)", interrupt ? "IRQ" : "TASK", smp_processor_id()); - /* - * Clear GPE_STS upfront to allow subsequent hardware GPE_STS 0->1 - * changes to always trigger a GPE interrupt. - * - * GPE STS is a W1C register, which means: - * - * 1. Software can clear it without worrying about clearing the other - * GPEs' STS bits when the hardware sets them in parallel. - * - * 2. As long as software can ensure only clearing it when it is set, - * hardware won't set it in parallel. - */ - if (ec->gpe >= 0 && acpi_ec_gpe_status_set(ec)) - acpi_clear_gpe(NULL, ec->gpe); - status = acpi_ec_read_status(ec); /* @@ -1287,6 +1272,22 @@ static void acpi_ec_handle_interrupt(struct acpi_ec *ec) unsigned long flags; spin_lock_irqsave(&ec->lock, flags); + + /* + * Clear GPE_STS upfront to allow subsequent hardware GPE_STS 0->1 + * changes to always trigger a GPE interrupt. + * + * GPE STS is a W1C register, which means: + * + * 1. Software can clear it without worrying about clearing the other + * GPEs' STS bits when the hardware sets them in parallel. + * + * 2. As long as software can ensure only clearing it when it is set, + * hardware won't set it in parallel. + */ + if (ec->gpe >= 0 && acpi_ec_gpe_status_set(ec)) + acpi_clear_gpe(NULL, ec->gpe); + advance_transaction(ec, true); spin_unlock_irqrestore(&ec->lock, flags); } diff --git a/drivers/acpi/nfit/nfit.h b/drivers/acpi/nfit/nfit.h index 6023ad61831a..573bc0de2990 100644 --- a/drivers/acpi/nfit/nfit.h +++ b/drivers/acpi/nfit/nfit.h @@ -347,4 +347,6 @@ int acpi_nfit_ctl(struct nvdimm_bus_descriptor *nd_desc, struct nvdimm *nvdimm, void acpi_nfit_desc_init(struct acpi_nfit_desc *acpi_desc, struct device *dev); bool intel_fwa_supported(struct nvdimm_bus *nvdimm_bus); extern struct device_attribute dev_attr_firmware_activate_noidle; +void nfit_intel_shutdown_status(struct nfit_mem *nfit_mem); + #endif /* __NFIT_H__ */ diff --git a/drivers/acpi/resource.c b/drivers/acpi/resource.c index 0800a9d77558..1dd8d5aebf67 100644 --- a/drivers/acpi/resource.c +++ b/drivers/acpi/resource.c @@ -470,52 +470,6 @@ static const struct dmi_system_id asus_laptop[] = { { } }; -static const struct dmi_system_id lenovo_laptop[] = { - { - .ident = "LENOVO IdeaPad Flex 5 14ALC7", - .matches = { - DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), - DMI_MATCH(DMI_PRODUCT_NAME, "82R9"), - }, - }, - { - .ident = "LENOVO IdeaPad Flex 5 16ALC7", - .matches = { - DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), - DMI_MATCH(DMI_PRODUCT_NAME, "82RA"), - }, - }, - { } -}; - -static const struct dmi_system_id tongfang_gm_rg[] = { - { - .ident = "TongFang GMxRGxx/XMG CORE 15 (M22)/TUXEDO Stellaris 15 Gen4 AMD", - .matches = { - DMI_MATCH(DMI_BOARD_NAME, "GMxRGxx"), - }, - }, - { } -}; - -static const struct dmi_system_id maingear_laptop[] = { - { - .ident = "MAINGEAR Vector Pro 2 15", - .matches = { - DMI_MATCH(DMI_SYS_VENDOR, "Micro Electronics Inc"), - DMI_MATCH(DMI_PRODUCT_NAME, "MG-VCP2-15A3070T"), - } - }, - { - .ident = "MAINGEAR Vector Pro 2 17", - .matches = { - DMI_MATCH(DMI_SYS_VENDOR, "Micro Electronics Inc"), - DMI_MATCH(DMI_PRODUCT_NAME, "MG-VCP2-17A3070T"), - }, - }, - { } -}; - static const struct dmi_system_id lg_laptop[] = { { .ident = "LG Electronics 17U70P", @@ -539,10 +493,6 @@ struct irq_override_cmp { static const struct irq_override_cmp override_table[] = { { medion_laptop, 1, ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_LOW, 0, false }, { asus_laptop, 1, ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_LOW, 0, false }, - { lenovo_laptop, 6, ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_LOW, 0, true }, - { lenovo_laptop, 10, ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_LOW, 0, true }, - { tongfang_gm_rg, 1, ACPI_EDGE_SENSITIVE, ACPI_ACTIVE_LOW, 1, true }, - { maingear_laptop, 1, ACPI_EDGE_SENSITIVE, ACPI_ACTIVE_LOW, 1, true }, { lg_laptop, 1, ACPI_LEVEL_SENSITIVE, ACPI_ACTIVE_LOW, 0, false }, }; @@ -562,16 +512,6 @@ static bool acpi_dev_irq_override(u32 gsi, u8 triggering, u8 polarity, return entry->override; } -#ifdef CONFIG_X86 - /* - * IRQ override isn't needed on modern AMD Zen systems and - * this override breaks active low IRQs on AMD Ryzen 6000 and - * newer systems. Skip it. - */ - if (boot_cpu_has(X86_FEATURE_ZEN)) - return false; -#endif - return true; } diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c index 0c6f06abe3f4..1c3e1e2bb0b5 100644 --- a/drivers/acpi/scan.c +++ b/drivers/acpi/scan.c @@ -2029,8 +2029,6 @@ static u32 acpi_scan_check_dep(acpi_handle handle, bool check_dep) return count; } -static bool acpi_bus_scan_second_pass; - static acpi_status acpi_bus_check_add(acpi_handle handle, bool check_dep, struct acpi_device **adev_p) { @@ -2050,10 +2048,8 @@ static acpi_status acpi_bus_check_add(acpi_handle handle, bool check_dep, return AE_OK; /* Bail out if there are dependencies. */ - if (acpi_scan_check_dep(handle, check_dep) > 0) { - acpi_bus_scan_second_pass = true; + if (acpi_scan_check_dep(handle, check_dep) > 0) return AE_CTRL_DEPTH; - } fallthrough; case ACPI_TYPE_ANY: /* for ACPI_ROOT_OBJECT */ @@ -2301,6 +2297,12 @@ static bool acpi_scan_clear_dep_queue(struct acpi_device *adev) return true; } +static void acpi_scan_delete_dep_data(struct acpi_dep_data *dep) +{ + list_del(&dep->node); + kfree(dep); +} + static int acpi_scan_clear_dep(struct acpi_dep_data *dep, void *data) { struct acpi_device *adev = acpi_get_acpi_dev(dep->consumer); @@ -2311,8 +2313,10 @@ static int acpi_scan_clear_dep(struct acpi_dep_data *dep, void *data) acpi_dev_put(adev); } - list_del(&dep->node); - kfree(dep); + if (dep->free_when_met) + acpi_scan_delete_dep_data(dep); + else + dep->met = true; return 0; } @@ -2406,6 +2410,55 @@ struct acpi_device *acpi_dev_get_next_consumer_dev(struct acpi_device *supplier, } EXPORT_SYMBOL_GPL(acpi_dev_get_next_consumer_dev); +static void acpi_scan_postponed_branch(acpi_handle handle) +{ + struct acpi_device *adev = NULL; + + if (ACPI_FAILURE(acpi_bus_check_add(handle, false, &adev))) + return; + + acpi_walk_namespace(ACPI_TYPE_ANY, handle, ACPI_UINT32_MAX, + acpi_bus_check_add_2, NULL, NULL, (void **)&adev); + acpi_bus_attach(adev, NULL); +} + +static void acpi_scan_postponed(void) +{ + struct acpi_dep_data *dep, *tmp; + + mutex_lock(&acpi_dep_list_lock); + + list_for_each_entry_safe(dep, tmp, &acpi_dep_list, node) { + acpi_handle handle = dep->consumer; + + /* + * In case there are multiple acpi_dep_list entries with the + * same consumer, skip the current entry if the consumer device + * object corresponding to it is present already. + */ + if (!acpi_fetch_acpi_dev(handle)) { + /* + * Even though the lock is released here, tmp is + * guaranteed to be valid, because none of the list + * entries following dep is marked as "free when met" + * and so they cannot be deleted. + */ + mutex_unlock(&acpi_dep_list_lock); + + acpi_scan_postponed_branch(handle); + + mutex_lock(&acpi_dep_list_lock); + } + + if (dep->met) + acpi_scan_delete_dep_data(dep); + else + dep->free_when_met = true; + } + + mutex_unlock(&acpi_dep_list_lock); +} + /** * acpi_bus_scan - Add ACPI device node objects in a given namespace scope. * @handle: Root of the namespace scope to scan. @@ -2424,8 +2477,6 @@ int acpi_bus_scan(acpi_handle handle) { struct acpi_device *device = NULL; - acpi_bus_scan_second_pass = false; - /* Pass 1: Avoid enumerating devices with missing dependencies. */ if (ACPI_SUCCESS(acpi_bus_check_add(handle, true, &device))) @@ -2438,19 +2489,9 @@ int acpi_bus_scan(acpi_handle handle) acpi_bus_attach(device, (void *)true); - if (!acpi_bus_scan_second_pass) - return 0; - /* Pass 2: Enumerate all of the remaining devices. */ - device = NULL; - - if (ACPI_SUCCESS(acpi_bus_check_add(handle, false, &device))) - acpi_walk_namespace(ACPI_TYPE_ANY, handle, ACPI_UINT32_MAX, - acpi_bus_check_add_2, NULL, NULL, - (void **)&device); - - acpi_bus_attach(device, NULL); + acpi_scan_postponed(); return 0; } diff --git a/drivers/acpi/sleep.c b/drivers/acpi/sleep.c index f32570f72b90..808484d11209 100644 --- a/drivers/acpi/sleep.c +++ b/drivers/acpi/sleep.c @@ -848,7 +848,7 @@ void __weak acpi_s2idle_setup(void) s2idle_set_ops(&acpi_s2idle_ops); } -static void acpi_sleep_suspend_setup(void) +static void __init acpi_sleep_suspend_setup(void) { bool suspend_ops_needed = false; int i; diff --git a/drivers/acpi/thermal.c b/drivers/acpi/thermal.c index 4720a3649a61..f9f6ebb08fdb 100644 --- a/drivers/acpi/thermal.c +++ b/drivers/acpi/thermal.c @@ -40,12 +40,35 @@ #define ACPI_THERMAL_NOTIFY_HOT 0xF1 #define ACPI_THERMAL_MODE_ACTIVE 0x00 -#define ACPI_THERMAL_MAX_ACTIVE 10 -#define ACPI_THERMAL_MAX_LIMIT_STR_LEN 65 +#define ACPI_THERMAL_MAX_ACTIVE 10 +#define ACPI_THERMAL_MAX_LIMIT_STR_LEN 65 -MODULE_AUTHOR("Paul Diefenbaugh"); -MODULE_DESCRIPTION("ACPI Thermal Zone Driver"); -MODULE_LICENSE("GPL"); +#define ACPI_TRIPS_CRITICAL BIT(0) +#define ACPI_TRIPS_HOT BIT(1) +#define ACPI_TRIPS_PASSIVE BIT(2) +#define ACPI_TRIPS_ACTIVE BIT(3) +#define ACPI_TRIPS_DEVICES BIT(4) + +#define ACPI_TRIPS_THRESHOLDS (ACPI_TRIPS_PASSIVE | ACPI_TRIPS_ACTIVE) + +#define ACPI_TRIPS_INIT (ACPI_TRIPS_CRITICAL | ACPI_TRIPS_HOT | \ + ACPI_TRIPS_PASSIVE | ACPI_TRIPS_ACTIVE | \ + ACPI_TRIPS_DEVICES) + +/* + * This exception is thrown out in two cases: + * 1.An invalid trip point becomes invalid or a valid trip point becomes invalid + * when re-evaluating the AML code. + * 2.TODO: Devices listed in _PSL, _ALx, _TZD may change. + * We need to re-bind the cooling devices of a thermal zone when this occurs. + */ +#define ACPI_THERMAL_TRIPS_EXCEPTION(flags, tz, str) \ +do { \ + if (flags != ACPI_TRIPS_INIT) \ + acpi_handle_info(tz->device->handle, \ + "ACPI thermal trip point %s changed\n" \ + "Please report to linux-acpi@vger.kernel.org\n", str); \ +} while (0) static int act; module_param(act, int, 0644); @@ -73,75 +96,30 @@ MODULE_PARM_DESC(psv, "Disable or override all passive trip points."); static struct workqueue_struct *acpi_thermal_pm_queue; -static int acpi_thermal_add(struct acpi_device *device); -static void acpi_thermal_remove(struct acpi_device *device); -static void acpi_thermal_notify(struct acpi_device *device, u32 event); - -static const struct acpi_device_id thermal_device_ids[] = { - {ACPI_THERMAL_HID, 0}, - {"", 0}, -}; -MODULE_DEVICE_TABLE(acpi, thermal_device_ids); - -#ifdef CONFIG_PM_SLEEP -static int acpi_thermal_suspend(struct device *dev); -static int acpi_thermal_resume(struct device *dev); -#else -#define acpi_thermal_suspend NULL -#define acpi_thermal_resume NULL -#endif -static SIMPLE_DEV_PM_OPS(acpi_thermal_pm, acpi_thermal_suspend, acpi_thermal_resume); - -static struct acpi_driver acpi_thermal_driver = { - .name = "thermal", - .class = ACPI_THERMAL_CLASS, - .ids = thermal_device_ids, - .ops = { - .add = acpi_thermal_add, - .remove = acpi_thermal_remove, - .notify = acpi_thermal_notify, - }, - .drv.pm = &acpi_thermal_pm, -}; - -struct acpi_thermal_state { - u8 critical:1; - u8 hot:1; - u8 passive:1; - u8 active:1; - u8 reserved:4; - int active_index; -}; - -struct acpi_thermal_state_flags { - u8 valid:1; - u8 enabled:1; - u8 reserved:6; -}; - struct acpi_thermal_critical { - struct acpi_thermal_state_flags flags; unsigned long temperature; + bool valid; }; struct acpi_thermal_hot { - struct acpi_thermal_state_flags flags; unsigned long temperature; + bool valid; }; struct acpi_thermal_passive { - struct acpi_thermal_state_flags flags; + struct acpi_handle_list devices; unsigned long temperature; unsigned long tc1; unsigned long tc2; unsigned long tsp; - struct acpi_handle_list devices; + bool valid; }; struct acpi_thermal_active { - struct acpi_thermal_state_flags flags; - unsigned long temperature; struct acpi_handle_list devices; + unsigned long temperature; + bool valid; + bool enabled; }; struct acpi_thermal_trips { @@ -151,12 +129,6 @@ struct acpi_thermal_trips { struct acpi_thermal_active active[ACPI_THERMAL_MAX_ACTIVE]; }; -struct acpi_thermal_flags { - u8 cooling_mode:1; /* _SCP */ - u8 devices:1; /* _TZD */ - u8 reserved:6; -}; - struct acpi_thermal { struct acpi_device *device; acpi_bus_id name; @@ -164,8 +136,6 @@ struct acpi_thermal { unsigned long last_temperature; unsigned long polling_frequency; volatile u8 zombie; - struct acpi_thermal_flags flags; - struct acpi_thermal_state state; struct acpi_thermal_trips trips; struct acpi_handle_list devices; struct thermal_zone_device *thermal_zone; @@ -220,52 +190,12 @@ static int acpi_thermal_get_polling_frequency(struct acpi_thermal *tz) return 0; } -static int acpi_thermal_set_cooling_mode(struct acpi_thermal *tz, int mode) -{ - if (!tz) - return -EINVAL; - - if (ACPI_FAILURE(acpi_execute_simple_method(tz->device->handle, - "_SCP", mode))) - return -ENODEV; - - return 0; -} - -#define ACPI_TRIPS_CRITICAL 0x01 -#define ACPI_TRIPS_HOT 0x02 -#define ACPI_TRIPS_PASSIVE 0x04 -#define ACPI_TRIPS_ACTIVE 0x08 -#define ACPI_TRIPS_DEVICES 0x10 - -#define ACPI_TRIPS_REFRESH_THRESHOLDS (ACPI_TRIPS_PASSIVE | ACPI_TRIPS_ACTIVE) -#define ACPI_TRIPS_REFRESH_DEVICES ACPI_TRIPS_DEVICES - -#define ACPI_TRIPS_INIT (ACPI_TRIPS_CRITICAL | ACPI_TRIPS_HOT | \ - ACPI_TRIPS_PASSIVE | ACPI_TRIPS_ACTIVE | \ - ACPI_TRIPS_DEVICES) - -/* - * This exception is thrown out in two cases: - * 1.An invalid trip point becomes invalid or a valid trip point becomes invalid - * when re-evaluating the AML code. - * 2.TODO: Devices listed in _PSL, _ALx, _TZD may change. - * We need to re-bind the cooling devices of a thermal zone when this occurs. - */ -#define ACPI_THERMAL_TRIPS_EXCEPTION(flags, tz, str) \ -do { \ - if (flags != ACPI_TRIPS_INIT) \ - acpi_handle_info(tz->device->handle, \ - "ACPI thermal trip point %s changed\n" \ - "Please report to linux-acpi@vger.kernel.org\n", str); \ -} while (0) - static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) { acpi_status status; unsigned long long tmp; struct acpi_handle_list devices; - int valid = 0; + bool valid = false; int i; /* Critical Shutdown */ @@ -279,21 +209,21 @@ static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) * ... so lets discard those as invalid. */ if (ACPI_FAILURE(status)) { - tz->trips.critical.flags.valid = 0; + tz->trips.critical.valid = false; acpi_handle_debug(tz->device->handle, "No critical threshold\n"); } else if (tmp <= 2732) { pr_info(FW_BUG "Invalid critical threshold (%llu)\n", tmp); - tz->trips.critical.flags.valid = 0; + tz->trips.critical.valid = false; } else { - tz->trips.critical.flags.valid = 1; + tz->trips.critical.valid = true; acpi_handle_debug(tz->device->handle, "Found critical threshold [%lu]\n", tz->trips.critical.temperature); } - if (tz->trips.critical.flags.valid) { + if (tz->trips.critical.valid) { if (crt == -1) { - tz->trips.critical.flags.valid = 0; + tz->trips.critical.valid = false; } else if (crt > 0) { unsigned long crt_k = celsius_to_deci_kelvin(crt); @@ -312,12 +242,12 @@ static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) if (flag & ACPI_TRIPS_HOT) { status = acpi_evaluate_integer(tz->device->handle, "_HOT", NULL, &tmp); if (ACPI_FAILURE(status)) { - tz->trips.hot.flags.valid = 0; + tz->trips.hot.valid = false; acpi_handle_debug(tz->device->handle, "No hot threshold\n"); } else { tz->trips.hot.temperature = tmp; - tz->trips.hot.flags.valid = 1; + tz->trips.hot.valid = true; acpi_handle_debug(tz->device->handle, "Found hot threshold [%lu]\n", tz->trips.hot.temperature); @@ -325,9 +255,9 @@ static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) } /* Passive (optional) */ - if (((flag & ACPI_TRIPS_PASSIVE) && tz->trips.passive.flags.valid) || + if (((flag & ACPI_TRIPS_PASSIVE) && tz->trips.passive.valid) || flag == ACPI_TRIPS_INIT) { - valid = tz->trips.passive.flags.valid; + valid = tz->trips.passive.valid; if (psv == -1) { status = AE_SUPPORT; } else if (psv > 0) { @@ -339,44 +269,44 @@ static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) } if (ACPI_FAILURE(status)) { - tz->trips.passive.flags.valid = 0; + tz->trips.passive.valid = false; } else { tz->trips.passive.temperature = tmp; - tz->trips.passive.flags.valid = 1; + tz->trips.passive.valid = true; if (flag == ACPI_TRIPS_INIT) { status = acpi_evaluate_integer(tz->device->handle, "_TC1", NULL, &tmp); if (ACPI_FAILURE(status)) - tz->trips.passive.flags.valid = 0; + tz->trips.passive.valid = false; else tz->trips.passive.tc1 = tmp; status = acpi_evaluate_integer(tz->device->handle, "_TC2", NULL, &tmp); if (ACPI_FAILURE(status)) - tz->trips.passive.flags.valid = 0; + tz->trips.passive.valid = false; else tz->trips.passive.tc2 = tmp; status = acpi_evaluate_integer(tz->device->handle, "_TSP", NULL, &tmp); if (ACPI_FAILURE(status)) - tz->trips.passive.flags.valid = 0; + tz->trips.passive.valid = false; else tz->trips.passive.tsp = tmp; } } } - if ((flag & ACPI_TRIPS_DEVICES) && tz->trips.passive.flags.valid) { + if ((flag & ACPI_TRIPS_DEVICES) && tz->trips.passive.valid) { memset(&devices, 0, sizeof(struct acpi_handle_list)); status = acpi_evaluate_reference(tz->device->handle, "_PSL", NULL, &devices); if (ACPI_FAILURE(status)) { acpi_handle_info(tz->device->handle, "Invalid passive threshold\n"); - tz->trips.passive.flags.valid = 0; + tz->trips.passive.valid = false; } else { - tz->trips.passive.flags.valid = 1; + tz->trips.passive.valid = true; } if (memcmp(&tz->trips.passive.devices, &devices, @@ -387,24 +317,24 @@ static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) } } if ((flag & ACPI_TRIPS_PASSIVE) || (flag & ACPI_TRIPS_DEVICES)) { - if (valid != tz->trips.passive.flags.valid) + if (valid != tz->trips.passive.valid) ACPI_THERMAL_TRIPS_EXCEPTION(flag, tz, "state"); } /* Active (optional) */ for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE; i++) { char name[5] = { '_', 'A', 'C', ('0' + i), '\0' }; - valid = tz->trips.active[i].flags.valid; + valid = tz->trips.active[i].valid; if (act == -1) break; /* disable all active trip points */ if (flag == ACPI_TRIPS_INIT || ((flag & ACPI_TRIPS_ACTIVE) && - tz->trips.active[i].flags.valid)) { + tz->trips.active[i].valid)) { status = acpi_evaluate_integer(tz->device->handle, name, NULL, &tmp); if (ACPI_FAILURE(status)) { - tz->trips.active[i].flags.valid = 0; + tz->trips.active[i].valid = false; if (i == 0) break; @@ -426,21 +356,21 @@ static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) break; } else { tz->trips.active[i].temperature = tmp; - tz->trips.active[i].flags.valid = 1; + tz->trips.active[i].valid = true; } } name[2] = 'L'; - if ((flag & ACPI_TRIPS_DEVICES) && tz->trips.active[i].flags.valid) { + if ((flag & ACPI_TRIPS_DEVICES) && tz->trips.active[i].valid) { memset(&devices, 0, sizeof(struct acpi_handle_list)); status = acpi_evaluate_reference(tz->device->handle, name, NULL, &devices); if (ACPI_FAILURE(status)) { acpi_handle_info(tz->device->handle, "Invalid active%d threshold\n", i); - tz->trips.active[i].flags.valid = 0; + tz->trips.active[i].valid = false; } else { - tz->trips.active[i].flags.valid = 1; + tz->trips.active[i].valid = true; } if (memcmp(&tz->trips.active[i].devices, &devices, @@ -451,10 +381,10 @@ static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) } } if ((flag & ACPI_TRIPS_ACTIVE) || (flag & ACPI_TRIPS_DEVICES)) - if (valid != tz->trips.active[i].flags.valid) + if (valid != tz->trips.active[i].valid) ACPI_THERMAL_TRIPS_EXCEPTION(flag, tz, "state"); - if (!tz->trips.active[i].flags.valid) + if (!tz->trips.active[i].valid) break; } @@ -474,17 +404,18 @@ static int acpi_thermal_trips_update(struct acpi_thermal *tz, int flag) static int acpi_thermal_get_trip_points(struct acpi_thermal *tz) { - int i, valid, ret = acpi_thermal_trips_update(tz, ACPI_TRIPS_INIT); + int i, ret = acpi_thermal_trips_update(tz, ACPI_TRIPS_INIT); + bool valid; if (ret) return ret; - valid = tz->trips.critical.flags.valid | - tz->trips.hot.flags.valid | - tz->trips.passive.flags.valid; + valid = tz->trips.critical.valid | + tz->trips.hot.valid | + tz->trips.passive.valid; for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE; i++) - valid |= tz->trips.active[i].flags.valid; + valid = valid || tz->trips.active[i].valid; if (!valid) { pr_warn(FW_BUG "No valid trip found\n"); @@ -521,7 +452,7 @@ static int thermal_get_trip_type(struct thermal_zone_device *thermal, if (!tz || trip < 0) return -EINVAL; - if (tz->trips.critical.flags.valid) { + if (tz->trips.critical.valid) { if (!trip) { *type = THERMAL_TRIP_CRITICAL; return 0; @@ -529,7 +460,7 @@ static int thermal_get_trip_type(struct thermal_zone_device *thermal, trip--; } - if (tz->trips.hot.flags.valid) { + if (tz->trips.hot.valid) { if (!trip) { *type = THERMAL_TRIP_HOT; return 0; @@ -537,7 +468,7 @@ static int thermal_get_trip_type(struct thermal_zone_device *thermal, trip--; } - if (tz->trips.passive.flags.valid) { + if (tz->trips.passive.valid) { if (!trip) { *type = THERMAL_TRIP_PASSIVE; return 0; @@ -545,7 +476,7 @@ static int thermal_get_trip_type(struct thermal_zone_device *thermal, trip--; } - for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE && tz->trips.active[i].flags.valid; i++) { + for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE && tz->trips.active[i].valid; i++) { if (!trip) { *type = THERMAL_TRIP_ACTIVE; return 0; @@ -565,7 +496,7 @@ static int thermal_get_trip_temp(struct thermal_zone_device *thermal, if (!tz || trip < 0) return -EINVAL; - if (tz->trips.critical.flags.valid) { + if (tz->trips.critical.valid) { if (!trip) { *temp = deci_kelvin_to_millicelsius_with_offset( tz->trips.critical.temperature, @@ -575,7 +506,7 @@ static int thermal_get_trip_temp(struct thermal_zone_device *thermal, trip--; } - if (tz->trips.hot.flags.valid) { + if (tz->trips.hot.valid) { if (!trip) { *temp = deci_kelvin_to_millicelsius_with_offset( tz->trips.hot.temperature, @@ -585,7 +516,7 @@ static int thermal_get_trip_temp(struct thermal_zone_device *thermal, trip--; } - if (tz->trips.passive.flags.valid) { + if (tz->trips.passive.valid) { if (!trip) { *temp = deci_kelvin_to_millicelsius_with_offset( tz->trips.passive.temperature, @@ -596,7 +527,7 @@ static int thermal_get_trip_temp(struct thermal_zone_device *thermal, } for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE && - tz->trips.active[i].flags.valid; i++) { + tz->trips.active[i].valid; i++) { if (!trip) { *temp = deci_kelvin_to_millicelsius_with_offset( tz->trips.active[i].temperature, @@ -614,7 +545,7 @@ static int thermal_get_crit_temp(struct thermal_zone_device *thermal, { struct acpi_thermal *tz = thermal_zone_device_priv(thermal); - if (tz->trips.critical.flags.valid) { + if (tz->trips.critical.valid) { *temperature = deci_kelvin_to_millicelsius_with_offset( tz->trips.critical.temperature, tz->kelvin_offset); @@ -700,13 +631,13 @@ static int acpi_thermal_cooling_device_cb(struct thermal_zone_device *thermal, int trip = -1; int result = 0; - if (tz->trips.critical.flags.valid) + if (tz->trips.critical.valid) trip++; - if (tz->trips.hot.flags.valid) + if (tz->trips.hot.valid) trip++; - if (tz->trips.passive.flags.valid) { + if (tz->trips.passive.valid) { trip++; for (i = 0; i < tz->trips.passive.devices.count; i++) { handle = tz->trips.passive.devices.handles[i]; @@ -731,7 +662,7 @@ static int acpi_thermal_cooling_device_cb(struct thermal_zone_device *thermal, } for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE; i++) { - if (!tz->trips.active[i].flags.valid) + if (!tz->trips.active[i].valid) break; trip++; @@ -819,19 +750,19 @@ static int acpi_thermal_register_thermal_zone(struct acpi_thermal *tz) acpi_status status; int i; - if (tz->trips.critical.flags.valid) + if (tz->trips.critical.valid) trips++; - if (tz->trips.hot.flags.valid) + if (tz->trips.hot.valid) trips++; - if (tz->trips.passive.flags.valid) + if (tz->trips.passive.valid) trips++; - for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE && tz->trips.active[i].flags.valid; + for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE && tz->trips.active[i].valid; i++, trips++); - if (tz->trips.passive.flags.valid) + if (tz->trips.passive.valid) tz->thermal_zone = thermal_zone_device_register("acpitz", trips, 0, tz, &acpi_thermal_zone_ops, NULL, tz->trips.passive.tsp * 100, @@ -906,13 +837,13 @@ static void acpi_thermal_notify(struct acpi_device *device, u32 event) acpi_queue_thermal_check(tz); break; case ACPI_THERMAL_NOTIFY_THRESHOLDS: - acpi_thermal_trips_update(tz, ACPI_TRIPS_REFRESH_THRESHOLDS); + acpi_thermal_trips_update(tz, ACPI_TRIPS_THRESHOLDS); acpi_queue_thermal_check(tz); acpi_bus_generate_netlink_event(device->pnp.device_class, dev_name(&device->dev), event, 0); break; case ACPI_THERMAL_NOTIFY_DEVICES: - acpi_thermal_trips_update(tz, ACPI_TRIPS_REFRESH_DEVICES); + acpi_thermal_trips_update(tz, ACPI_TRIPS_DEVICES); acpi_queue_thermal_check(tz); acpi_bus_generate_netlink_event(device->pnp.device_class, dev_name(&device->dev), event, 0); @@ -976,9 +907,8 @@ static int acpi_thermal_get_info(struct acpi_thermal *tz) return result; /* Set the cooling mode [_SCP] to active cooling (default) */ - result = acpi_thermal_set_cooling_mode(tz, ACPI_THERMAL_MODE_ACTIVE); - if (!result) - tz->flags.cooling_mode = 1; + acpi_execute_simple_method(tz->device->handle, "_SCP", + ACPI_THERMAL_MODE_ACTIVE); /* Get default polling frequency [_TZP] (optional) */ if (tzp) @@ -1001,7 +931,7 @@ static int acpi_thermal_get_info(struct acpi_thermal *tz) */ static void acpi_thermal_guess_offset(struct acpi_thermal *tz) { - if (tz->trips.critical.flags.valid && + if (tz->trips.critical.valid && (tz->trips.critical.temperature % 5) == 1) tz->kelvin_offset = 273100; else @@ -1110,27 +1040,48 @@ static int acpi_thermal_resume(struct device *dev) return -EINVAL; for (i = 0; i < ACPI_THERMAL_MAX_ACTIVE; i++) { - if (!tz->trips.active[i].flags.valid) + if (!tz->trips.active[i].valid) break; - tz->trips.active[i].flags.enabled = 1; + tz->trips.active[i].enabled = true; for (j = 0; j < tz->trips.active[i].devices.count; j++) { result = acpi_bus_update_power( tz->trips.active[i].devices.handles[j], &power_state); if (result || (power_state != ACPI_STATE_D0)) { - tz->trips.active[i].flags.enabled = 0; + tz->trips.active[i].enabled = false; break; } } - tz->state.active |= tz->trips.active[i].flags.enabled; } acpi_queue_thermal_check(tz); return AE_OK; } +#else +#define acpi_thermal_suspend NULL +#define acpi_thermal_resume NULL #endif +static SIMPLE_DEV_PM_OPS(acpi_thermal_pm, acpi_thermal_suspend, acpi_thermal_resume); + +static const struct acpi_device_id thermal_device_ids[] = { + {ACPI_THERMAL_HID, 0}, + {"", 0}, +}; +MODULE_DEVICE_TABLE(acpi, thermal_device_ids); + +static struct acpi_driver acpi_thermal_driver = { + .name = "thermal", + .class = ACPI_THERMAL_CLASS, + .ids = thermal_device_ids, + .ops = { + .add = acpi_thermal_add, + .remove = acpi_thermal_remove, + .notify = acpi_thermal_notify, + }, + .drv.pm = &acpi_thermal_pm, +}; static int thermal_act(const struct dmi_system_id *d) { if (act == 0) { @@ -1236,3 +1187,7 @@ static void __exit acpi_thermal_exit(void) module_init(acpi_thermal_init); module_exit(acpi_thermal_exit); + +MODULE_AUTHOR("Paul Diefenbaugh"); +MODULE_DESCRIPTION("ACPI Thermal Zone Driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/acpi/tiny-power-button.c b/drivers/acpi/tiny-power-button.c index 598f548b21f3..6353be6fec69 100644 --- a/drivers/acpi/tiny-power-button.c +++ b/drivers/acpi/tiny-power-button.c @@ -19,18 +19,52 @@ static const struct acpi_device_id tiny_power_button_device_ids[] = { }; MODULE_DEVICE_TABLE(acpi, tiny_power_button_device_ids); -static int acpi_noop_add(struct acpi_device *device) +static void acpi_tiny_power_button_notify(acpi_handle handle, u32 event, void *data) { + kill_cad_pid(power_signal, 1); +} + +static void acpi_tiny_power_button_notify_run(void *not_used) +{ + acpi_tiny_power_button_notify(NULL, ACPI_FIXED_HARDWARE_EVENT, NULL); +} + +static u32 acpi_tiny_power_button_event(void *not_used) +{ + acpi_os_execute(OSL_NOTIFY_HANDLER, acpi_tiny_power_button_notify_run, NULL); + return ACPI_INTERRUPT_HANDLED; +} + +static int acpi_tiny_power_button_add(struct acpi_device *device) +{ + acpi_status status; + + if (device->device_type == ACPI_BUS_TYPE_POWER_BUTTON) { + status = acpi_install_fixed_event_handler(ACPI_EVENT_POWER_BUTTON, + acpi_tiny_power_button_event, + NULL); + } else { + status = acpi_install_notify_handler(device->handle, + ACPI_DEVICE_NOTIFY, + acpi_tiny_power_button_notify, + NULL); + } + if (ACPI_FAILURE(status)) + return -ENODEV; + return 0; } -static void acpi_noop_remove(struct acpi_device *device) +static void acpi_tiny_power_button_remove(struct acpi_device *device) { -} - -static void acpi_tiny_power_button_notify(struct acpi_device *device, u32 event) -{ - kill_cad_pid(power_signal, 1); + if (device->device_type == ACPI_BUS_TYPE_POWER_BUTTON) { + acpi_remove_fixed_event_handler(ACPI_EVENT_POWER_BUTTON, + acpi_tiny_power_button_event); + } else { + acpi_remove_notify_handler(device->handle, ACPI_DEVICE_NOTIFY, + acpi_tiny_power_button_notify); + } + acpi_os_wait_events_complete(); } static struct acpi_driver acpi_tiny_power_button_driver = { @@ -38,9 +72,8 @@ static struct acpi_driver acpi_tiny_power_button_driver = { .class = "tiny-power-button", .ids = tiny_power_button_device_ids, .ops = { - .add = acpi_noop_add, - .remove = acpi_noop_remove, - .notify = acpi_tiny_power_button_notify, + .add = acpi_tiny_power_button_add, + .remove = acpi_tiny_power_button_remove, }, }; diff --git a/drivers/acpi/video_detect.c b/drivers/acpi/video_detect.c index bcc25d457581..18cc08c858cf 100644 --- a/drivers/acpi/video_detect.c +++ b/drivers/acpi/video_detect.c @@ -470,6 +470,22 @@ static const struct dmi_system_id video_detect_dmi_table[] = { DMI_MATCH(DMI_PRODUCT_NAME, "82BK"), }, }, + { + .callback = video_detect_force_native, + /* Lenovo ThinkPad X131e (3371 AMD version) */ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), + DMI_MATCH(DMI_PRODUCT_NAME, "3371"), + }, + }, + { + .callback = video_detect_force_native, + /* Apple iMac11,3 */ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "Apple Inc."), + DMI_MATCH(DMI_PRODUCT_NAME, "iMac11,3"), + }, + }, { /* https://bugzilla.redhat.com/show_bug.cgi?id=1217249 */ .callback = video_detect_force_native, @@ -512,6 +528,14 @@ static const struct dmi_system_id video_detect_dmi_table[] = { DMI_MATCH(DMI_PRODUCT_NAME, "Precision 7510"), }, }, + { + .callback = video_detect_force_native, + /* Dell Studio 1569 */ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."), + DMI_MATCH(DMI_PRODUCT_NAME, "Studio 1569"), + }, + }, { .callback = video_detect_force_native, /* Acer Aspire 3830TG */ @@ -828,6 +852,27 @@ enum acpi_backlight_type __acpi_video_get_backlight_type(bool native, bool *auto if (native_available) return acpi_backlight_native; + /* + * The vendor specific BIOS interfaces are only necessary for + * laptops from before ~2008. + * + * For laptops from ~2008 till ~2023 this point is never reached + * because on those (video_caps & ACPI_VIDEO_BACKLIGHT) above is true. + * + * Laptops from after ~2023 no longer support ACPI_VIDEO_BACKLIGHT, + * if this point is reached on those, this likely means that + * the GPU kms driver which sets native_available has not loaded yet. + * + * Returning acpi_backlight_vendor in this case is known to sometimes + * cause a non working vendor specific /sys/class/backlight device to + * get registered. + * + * Return acpi_backlight_none on laptops with ACPI tables written + * for Windows 8 (laptops from after ~2012) to avoid this problem. + */ + if (acpi_osi_is_win8()) + return acpi_backlight_none; + /* No ACPI video/native (old hw), use vendor specific fw methods. */ return acpi_backlight_vendor; } diff --git a/drivers/acpi/x86/s2idle.c b/drivers/acpi/x86/s2idle.c index e499c60c4579..ce62e61a9605 100644 --- a/drivers/acpi/x86/s2idle.c +++ b/drivers/acpi/x86/s2idle.c @@ -59,6 +59,7 @@ static int lps0_dsm_func_mask; static guid_t lps0_dsm_guid_microsoft; static int lps0_dsm_func_mask_microsoft; +static int lps0_dsm_state; /* Device constraint entry structure */ struct lpi_device_info { @@ -320,6 +321,44 @@ static void lpi_check_constraints(void) } } +static bool acpi_s2idle_vendor_amd(void) +{ + return boot_cpu_data.x86_vendor == X86_VENDOR_AMD; +} + +static const char *acpi_sleep_dsm_state_to_str(unsigned int state) +{ + if (lps0_dsm_func_mask_microsoft || !acpi_s2idle_vendor_amd()) { + switch (state) { + case ACPI_LPS0_SCREEN_OFF: + return "screen off"; + case ACPI_LPS0_SCREEN_ON: + return "screen on"; + case ACPI_LPS0_ENTRY: + return "lps0 entry"; + case ACPI_LPS0_EXIT: + return "lps0 exit"; + case ACPI_LPS0_MS_ENTRY: + return "lps0 ms entry"; + case ACPI_LPS0_MS_EXIT: + return "lps0 ms exit"; + } + } else { + switch (state) { + case ACPI_LPS0_SCREEN_ON_AMD: + return "screen on"; + case ACPI_LPS0_SCREEN_OFF_AMD: + return "screen off"; + case ACPI_LPS0_ENTRY_AMD: + return "lps0 entry"; + case ACPI_LPS0_EXIT_AMD: + return "lps0 exit"; + } + } + + return "unknown"; +} + static void acpi_sleep_run_lps0_dsm(unsigned int func, unsigned int func_mask, guid_t dsm_guid) { union acpi_object *out_obj; @@ -331,14 +370,15 @@ static void acpi_sleep_run_lps0_dsm(unsigned int func, unsigned int func_mask, g rev_id, func, NULL); ACPI_FREE(out_obj); - acpi_handle_debug(lps0_device_handle, "_DSM function %u evaluation %s\n", - func, out_obj ? "successful" : "failed"); + lps0_dsm_state = func; + if (pm_debug_messages_on) { + acpi_handle_info(lps0_device_handle, + "%s transitioned to state %s\n", + out_obj ? "Successfully" : "Failed to", + acpi_sleep_dsm_state_to_str(lps0_dsm_state)); + } } -static bool acpi_s2idle_vendor_amd(void) -{ - return boot_cpu_data.x86_vendor == X86_VENDOR_AMD; -} static int validate_dsm(acpi_handle handle, const char *uuid, int rev, guid_t *dsm_guid) { @@ -485,11 +525,11 @@ int acpi_s2idle_prepare_late(void) ACPI_LPS0_ENTRY, lps0_dsm_func_mask, lps0_dsm_guid); if (lps0_dsm_func_mask_microsoft > 0) { - acpi_sleep_run_lps0_dsm(ACPI_LPS0_ENTRY, - lps0_dsm_func_mask_microsoft, lps0_dsm_guid_microsoft); /* modern standby entry */ acpi_sleep_run_lps0_dsm(ACPI_LPS0_MS_ENTRY, lps0_dsm_func_mask_microsoft, lps0_dsm_guid_microsoft); + acpi_sleep_run_lps0_dsm(ACPI_LPS0_ENTRY, + lps0_dsm_func_mask_microsoft, lps0_dsm_guid_microsoft); } list_for_each_entry(handler, &lps0_s2idle_devops_head, list_node) { @@ -524,11 +564,6 @@ void acpi_s2idle_restore_early(void) if (handler->restore) handler->restore(); - /* Modern standby exit */ - if (lps0_dsm_func_mask_microsoft > 0) - acpi_sleep_run_lps0_dsm(ACPI_LPS0_MS_EXIT, - lps0_dsm_func_mask_microsoft, lps0_dsm_guid_microsoft); - /* LPS0 exit */ if (lps0_dsm_func_mask > 0) acpi_sleep_run_lps0_dsm(acpi_s2idle_vendor_amd() ? @@ -539,6 +574,11 @@ void acpi_s2idle_restore_early(void) acpi_sleep_run_lps0_dsm(ACPI_LPS0_EXIT, lps0_dsm_func_mask_microsoft, lps0_dsm_guid_microsoft); + /* Modern standby exit */ + if (lps0_dsm_func_mask_microsoft > 0) + acpi_sleep_run_lps0_dsm(ACPI_LPS0_MS_EXIT, + lps0_dsm_func_mask_microsoft, lps0_dsm_guid_microsoft); + /* Screen on */ if (lps0_dsm_func_mask_microsoft > 0) acpi_sleep_run_lps0_dsm(ACPI_LPS0_SCREEN_ON, diff --git a/drivers/acpi/x86/utils.c b/drivers/acpi/x86/utils.c index 9c2d6f35f88a..c2b925f8cd4e 100644 --- a/drivers/acpi/x86/utils.c +++ b/drivers/acpi/x86/utils.c @@ -259,10 +259,11 @@ bool force_storage_d3(void) * drivers/platform/x86/x86-android-tablets.c kernel module. */ #define ACPI_QUIRK_SKIP_I2C_CLIENTS BIT(0) -#define ACPI_QUIRK_UART1_TTY_UART2_SKIP BIT(1) -#define ACPI_QUIRK_SKIP_ACPI_AC_AND_BATTERY BIT(2) -#define ACPI_QUIRK_USE_ACPI_AC_AND_BATTERY BIT(3) -#define ACPI_QUIRK_SKIP_GPIO_EVENT_HANDLERS BIT(4) +#define ACPI_QUIRK_UART1_SKIP BIT(1) +#define ACPI_QUIRK_UART1_TTY_UART2_SKIP BIT(2) +#define ACPI_QUIRK_SKIP_ACPI_AC_AND_BATTERY BIT(3) +#define ACPI_QUIRK_USE_ACPI_AC_AND_BATTERY BIT(4) +#define ACPI_QUIRK_SKIP_GPIO_EVENT_HANDLERS BIT(5) static const struct dmi_system_id acpi_quirk_skip_dmi_ids[] = { /* @@ -319,6 +320,7 @@ static const struct dmi_system_id acpi_quirk_skip_dmi_ids[] = { DMI_EXACT_MATCH(DMI_PRODUCT_VERSION, "YETI-11"), }, .driver_data = (void *)(ACPI_QUIRK_SKIP_I2C_CLIENTS | + ACPI_QUIRK_UART1_SKIP | ACPI_QUIRK_SKIP_ACPI_AC_AND_BATTERY | ACPI_QUIRK_SKIP_GPIO_EVENT_HANDLERS), }, @@ -365,7 +367,7 @@ static const struct dmi_system_id acpi_quirk_skip_dmi_ids[] = { ACPI_QUIRK_SKIP_ACPI_AC_AND_BATTERY), }, { - /* Nextbook Ares 8 */ + /* Nextbook Ares 8 (BYT version)*/ .matches = { DMI_MATCH(DMI_SYS_VENDOR, "Insyde"), DMI_MATCH(DMI_PRODUCT_NAME, "M890BAP"), @@ -374,6 +376,16 @@ static const struct dmi_system_id acpi_quirk_skip_dmi_ids[] = { ACPI_QUIRK_SKIP_ACPI_AC_AND_BATTERY | ACPI_QUIRK_SKIP_GPIO_EVENT_HANDLERS), }, + { + /* Nextbook Ares 8A (CHT version)*/ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "Insyde"), + DMI_MATCH(DMI_PRODUCT_NAME, "CherryTrail"), + DMI_MATCH(DMI_BIOS_VERSION, "M882"), + }, + .driver_data = (void *)(ACPI_QUIRK_SKIP_I2C_CLIENTS | + ACPI_QUIRK_SKIP_ACPI_AC_AND_BATTERY), + }, { /* Whitelabel (sold as various brands) TM800A550L */ .matches = { @@ -392,6 +404,7 @@ static const struct dmi_system_id acpi_quirk_skip_dmi_ids[] = { #if IS_ENABLED(CONFIG_X86_ANDROID_TABLETS) static const struct acpi_device_id i2c_acpi_known_good_ids[] = { { "10EC5640", 0 }, /* RealTek ALC5640 audio codec */ + { "10EC5651", 0 }, /* RealTek ALC5651 audio codec */ { "INT33F4", 0 }, /* X-Powers AXP288 PMIC */ { "INT33FD", 0 }, /* Intel Crystal Cove PMIC */ { "INT34D3", 0 }, /* Intel Whiskey Cove PMIC */ @@ -438,6 +451,9 @@ int acpi_quirk_skip_serdev_enumeration(struct device *controller_parent, bool *s if (dmi_id) quirks = (unsigned long)dmi_id->driver_data; + if ((quirks & ACPI_QUIRK_UART1_SKIP) && uid == 1) + *skip = true; + if (quirks & ACPI_QUIRK_UART1_TTY_UART2_SKIP) { if (uid == 1) return -ENODEV; /* Create tty cdev instead of serdev */ diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c index 32084e38b73d..5cb2023581d4 100644 --- a/drivers/base/power/domain.c +++ b/drivers/base/power/domain.c @@ -1632,9 +1632,6 @@ static int genpd_add_device(struct generic_pm_domain *genpd, struct device *dev, dev_dbg(dev, "%s()\n", __func__); - if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev)) - return -EINVAL; - gpd_data = genpd_alloc_dev_data(dev, gd); if (IS_ERR(gpd_data)) return PTR_ERR(gpd_data); @@ -1676,6 +1673,9 @@ int pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev) { int ret; + if (!genpd || !dev) + return -EINVAL; + mutex_lock(&gpd_list_lock); ret = genpd_add_device(genpd, dev, dev); mutex_unlock(&gpd_list_lock); @@ -2523,6 +2523,9 @@ int of_genpd_add_device(struct of_phandle_args *genpdspec, struct device *dev) struct generic_pm_domain *genpd; int ret; + if (!dev) + return -EINVAL; + mutex_lock(&gpd_list_lock); genpd = genpd_get_from_provider(genpdspec); @@ -2939,10 +2942,10 @@ static int genpd_parse_state(struct genpd_power_state *genpd_state, err = of_property_read_u32(state_node, "min-residency-us", &residency); if (!err) - genpd_state->residency_ns = 1000 * residency; + genpd_state->residency_ns = 1000LL * residency; - genpd_state->power_on_latency_ns = 1000 * exit_latency; - genpd_state->power_off_latency_ns = 1000 * entry_latency; + genpd_state->power_on_latency_ns = 1000LL * exit_latency; + genpd_state->power_off_latency_ns = 1000LL * entry_latency; genpd_state->fwnode = &state_node->fwnode; return 0; diff --git a/drivers/base/power/wakeup.c b/drivers/base/power/wakeup.c index 441092f988e8..addbd42930bf 100644 --- a/drivers/base/power/wakeup.c +++ b/drivers/base/power/wakeup.c @@ -22,11 +22,6 @@ #include "power.h" -#ifndef CONFIG_SUSPEND -suspend_state_t pm_suspend_target_state; -#define pm_suspend_target_state (PM_SUSPEND_ON) -#endif - #define list_for_each_entry_rcu_locked(pos, head, member) \ list_for_each_entry_rcu(pos, head, member, \ srcu_read_lock_held(&wakeup_srcu)) diff --git a/drivers/cpufreq/Kconfig b/drivers/cpufreq/Kconfig index 83e84294ea62..abd5778a6f95 100644 --- a/drivers/cpufreq/Kconfig +++ b/drivers/cpufreq/Kconfig @@ -45,7 +45,7 @@ choice prompt "Default CPUFreq governor" default CPU_FREQ_DEFAULT_GOV_USERSPACE if ARM_SA1110_CPUFREQ default CPU_FREQ_DEFAULT_GOV_SCHEDUTIL if ARM64 || ARM - default CPU_FREQ_DEFAULT_GOV_SCHEDUTIL if X86_INTEL_PSTATE && SMP + default CPU_FREQ_DEFAULT_GOV_SCHEDUTIL if (X86_INTEL_PSTATE || X86_AMD_PSTATE) && SMP default CPU_FREQ_DEFAULT_GOV_PERFORMANCE help This option sets which CPUFreq governor shall be loaded at diff --git a/drivers/cpufreq/Kconfig.x86 b/drivers/cpufreq/Kconfig.x86 index 00476e94db90..438c9e75a04d 100644 --- a/drivers/cpufreq/Kconfig.x86 +++ b/drivers/cpufreq/Kconfig.x86 @@ -51,6 +51,23 @@ config X86_AMD_PSTATE If in doubt, say N. +config X86_AMD_PSTATE_DEFAULT_MODE + int "AMD Processor P-State default mode" + depends on X86_AMD_PSTATE + default 3 if X86_AMD_PSTATE + range 1 4 + help + Select the default mode the amd-pstate driver will use on + supported hardware. + The value set has the following meanings: + 1 -> Disabled + 2 -> Passive + 3 -> Active (EPP) + 4 -> Guided + + For details, take a look at: + . + config X86_AMD_PSTATE_UT tristate "selftest for AMD Processor P-State driver" depends on X86 && ACPI_PROCESSOR diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index ddd346a239e0..81fba0dcbee9 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -62,7 +62,8 @@ static struct cpufreq_driver *current_pstate_driver; static struct cpufreq_driver amd_pstate_driver; static struct cpufreq_driver amd_pstate_epp_driver; -static int cppc_state = AMD_PSTATE_DISABLE; +static int cppc_state = AMD_PSTATE_UNDEFINED; +static bool cppc_enabled; /* * AMD Energy Preference Performance (EPP) @@ -228,7 +229,28 @@ static int amd_pstate_set_energy_pref_index(struct amd_cpudata *cpudata, static inline int pstate_enable(bool enable) { - return wrmsrl_safe(MSR_AMD_CPPC_ENABLE, enable); + int ret, cpu; + unsigned long logical_proc_id_mask = 0; + + if (enable == cppc_enabled) + return 0; + + for_each_present_cpu(cpu) { + unsigned long logical_id = topology_logical_die_id(cpu); + + if (test_bit(logical_id, &logical_proc_id_mask)) + continue; + + set_bit(logical_id, &logical_proc_id_mask); + + ret = wrmsrl_safe_on_cpu(cpu, MSR_AMD_CPPC_ENABLE, + enable); + if (ret) + return ret; + } + + cppc_enabled = enable; + return 0; } static int cppc_enable(bool enable) @@ -236,6 +258,9 @@ static int cppc_enable(bool enable) int cpu, ret = 0; struct cppc_perf_ctrls perf_ctrls; + if (enable == cppc_enabled) + return 0; + for_each_present_cpu(cpu) { ret = cppc_set_enable(cpu, enable); if (ret) @@ -251,6 +276,7 @@ static int cppc_enable(bool enable) } } + cppc_enabled = enable; return ret; } @@ -1045,6 +1071,26 @@ static const struct attribute_group amd_pstate_global_attr_group = { .attrs = pstate_global_attributes, }; +static bool amd_pstate_acpi_pm_profile_server(void) +{ + switch (acpi_gbl_FADT.preferred_profile) { + case PM_ENTERPRISE_SERVER: + case PM_SOHO_SERVER: + case PM_PERFORMANCE_SERVER: + return true; + } + return false; +} + +static bool amd_pstate_acpi_pm_profile_undefined(void) +{ + if (acpi_gbl_FADT.preferred_profile == PM_UNSPECIFIED) + return true; + if (acpi_gbl_FADT.preferred_profile >= NR_PM_PROFILES) + return true; + return false; +} + static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) { int min_freq, max_freq, nominal_freq, lowest_nonlinear_freq, ret; @@ -1102,10 +1148,14 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) policy->max = policy->cpuinfo.max_freq; /* - * Set the policy to powersave to provide a valid fallback value in case + * Set the policy to provide a valid fallback value in case * the default cpufreq governor is neither powersave nor performance. */ - policy->policy = CPUFREQ_POLICY_POWERSAVE; + if (amd_pstate_acpi_pm_profile_server() || + amd_pstate_acpi_pm_profile_undefined()) + policy->policy = CPUFREQ_POLICY_PERFORMANCE; + else + policy->policy = CPUFREQ_POLICY_POWERSAVE; if (boot_cpu_has(X86_FEATURE_CPPC)) { ret = rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, &value); @@ -1356,10 +1406,29 @@ static struct cpufreq_driver amd_pstate_epp_driver = { .online = amd_pstate_epp_cpu_online, .suspend = amd_pstate_epp_suspend, .resume = amd_pstate_epp_resume, - .name = "amd_pstate_epp", + .name = "amd-pstate-epp", .attr = amd_pstate_epp_attr, }; +static int __init amd_pstate_set_driver(int mode_idx) +{ + if (mode_idx >= AMD_PSTATE_DISABLE && mode_idx < AMD_PSTATE_MAX) { + cppc_state = mode_idx; + if (cppc_state == AMD_PSTATE_DISABLE) + pr_info("driver is explicitly disabled\n"); + + if (cppc_state == AMD_PSTATE_ACTIVE) + current_pstate_driver = &amd_pstate_epp_driver; + + if (cppc_state == AMD_PSTATE_PASSIVE || cppc_state == AMD_PSTATE_GUIDED) + current_pstate_driver = &amd_pstate_driver; + + return 0; + } + + return -EINVAL; +} + static int __init amd_pstate_init(void) { struct device *dev_root; @@ -1367,15 +1436,6 @@ static int __init amd_pstate_init(void) if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD) return -ENODEV; - /* - * by default the pstate driver is disabled to load - * enable the amd_pstate passive mode driver explicitly - * with amd_pstate=passive or other modes in kernel command line - */ - if (cppc_state == AMD_PSTATE_DISABLE) { - pr_info("driver load is disabled, boot with specific mode to enable this\n"); - return -ENODEV; - } if (!acpi_cpc_valid()) { pr_warn_once("the _CPC object is not present in SBIOS or ACPI disabled\n"); @@ -1386,6 +1446,33 @@ static int __init amd_pstate_init(void) if (cpufreq_get_current_driver()) return -EEXIST; + switch (cppc_state) { + case AMD_PSTATE_UNDEFINED: + /* Disable on the following configs by default: + * 1. Undefined platforms + * 2. Server platforms + * 3. Shared memory designs + */ + if (amd_pstate_acpi_pm_profile_undefined() || + amd_pstate_acpi_pm_profile_server() || + !boot_cpu_has(X86_FEATURE_CPPC)) { + pr_info("driver load is disabled, boot with specific mode to enable this\n"); + return -ENODEV; + } + ret = amd_pstate_set_driver(CONFIG_X86_AMD_PSTATE_DEFAULT_MODE); + if (ret) + return ret; + break; + case AMD_PSTATE_DISABLE: + return -ENODEV; + case AMD_PSTATE_PASSIVE: + case AMD_PSTATE_ACTIVE: + case AMD_PSTATE_GUIDED: + break; + default: + return -EINVAL; + } + /* capability check */ if (boot_cpu_has(X86_FEATURE_CPPC)) { pr_debug("AMD CPPC MSR based functionality is supported\n"); @@ -1438,21 +1525,7 @@ static int __init amd_pstate_param(char *str) size = strlen(str); mode_idx = get_mode_idx_from_str(str, size); - if (mode_idx >= AMD_PSTATE_DISABLE && mode_idx < AMD_PSTATE_MAX) { - cppc_state = mode_idx; - if (cppc_state == AMD_PSTATE_DISABLE) - pr_info("driver is explicitly disabled\n"); - - if (cppc_state == AMD_PSTATE_ACTIVE) - current_pstate_driver = &amd_pstate_epp_driver; - - if (cppc_state == AMD_PSTATE_PASSIVE || cppc_state == AMD_PSTATE_GUIDED) - current_pstate_driver = &amd_pstate_driver; - - return 0; - } - - return -EINVAL; + return amd_pstate_set_driver(mode_idx); } early_param("amd_pstate", amd_pstate_param); diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 70462cf8e5e9..1c2f520b9791 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -2839,7 +2839,8 @@ int cpufreq_register_driver(struct cpufreq_driver *driver_data) (driver_data->setpolicy && (driver_data->target_index || driver_data->target)) || (!driver_data->get_intermediate != !driver_data->target_intermediate) || - (!driver_data->online != !driver_data->offline)) + (!driver_data->online != !driver_data->offline) || + (driver_data->adjust_perf && !driver_data->fast_switch)) return -EINVAL; pr_debug("trying to register driver %s\n", driver_data->name); diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c index 2548ec92faa2..f29182512b98 100644 --- a/drivers/cpufreq/intel_pstate.c +++ b/drivers/cpufreq/intel_pstate.c @@ -824,6 +824,8 @@ static ssize_t store_energy_performance_preference( err = cpufreq_start_governor(policy); if (!ret) ret = err; + } else { + ret = 0; } } diff --git a/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-cipher.c b/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-cipher.c index 10fe9f73a5fb..f2dd66706c10 100644 --- a/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-cipher.c +++ b/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-cipher.c @@ -8,7 +8,7 @@ * keysize in CBC and ECB mode. * Add support also for DES and 3DES in CBC and ECB mode. * - * You could find the datasheet in Documentation/arm/sunxi.rst + * You could find the datasheet in Documentation/arch/arm/sunxi.rst */ #include "sun4i-ss.h" diff --git a/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-core.c b/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-core.c index 006e40133c28..51a3a7b5b985 100644 --- a/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-core.c +++ b/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-core.c @@ -6,7 +6,7 @@ * * Core file which registers crypto algorithms supported by the SS. * - * You could find a link for the datasheet in Documentation/arm/sunxi.rst + * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst */ #include #include diff --git a/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-hash.c b/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-hash.c index d28292762b32..f7893e4ac59d 100644 --- a/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-hash.c +++ b/drivers/crypto/allwinner/sun4i-ss/sun4i-ss-hash.c @@ -6,7 +6,7 @@ * * This file add support for MD5 and SHA1. * - * You could find the datasheet in Documentation/arm/sunxi.rst + * You could find the datasheet in Documentation/arch/arm/sunxi.rst */ #include "sun4i-ss.h" #include diff --git a/drivers/crypto/allwinner/sun4i-ss/sun4i-ss.h b/drivers/crypto/allwinner/sun4i-ss/sun4i-ss.h index ba59c7a48825..6c5d4aa6453c 100644 --- a/drivers/crypto/allwinner/sun4i-ss/sun4i-ss.h +++ b/drivers/crypto/allwinner/sun4i-ss/sun4i-ss.h @@ -8,7 +8,7 @@ * Support MD5 and SHA1 hash algorithms. * Support DES and 3DES * - * You could find the datasheet in Documentation/arm/sunxi.rst + * You could find the datasheet in Documentation/arch/arm/sunxi.rst */ #include diff --git a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-cipher.c b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-cipher.c index 74b4e910a38d..c13550090785 100644 --- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-cipher.c +++ b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-cipher.c @@ -8,7 +8,7 @@ * This file add support for AES cipher with 128,192,256 bits keysize in * CBC and ECB mode. * - * You could find a link for the datasheet in Documentation/arm/sunxi.rst + * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst */ #include diff --git a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c index a6865ff4d400..07ea0cc82b16 100644 --- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c +++ b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-core.c @@ -7,7 +7,7 @@ * * Core file which registers crypto algorithms supported by the CryptoEngine. * - * You could find a link for the datasheet in Documentation/arm/sunxi.rst + * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst */ #include #include diff --git a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-hash.c b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-hash.c index 8b5b9b9d04c3..930ad157004c 100644 --- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-hash.c +++ b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-hash.c @@ -7,7 +7,7 @@ * * This file add support for MD5 and SHA1/SHA224/SHA256/SHA384/SHA512. * - * You could find the datasheet in Documentation/arm/sunxi.rst + * You could find the datasheet in Documentation/arch/arm/sunxi.rst */ #include #include diff --git a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c index b3cc43ea6c8a..80815379f6fc 100644 --- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c +++ b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-prng.c @@ -7,7 +7,7 @@ * * This file handle the PRNG * - * You could find a link for the datasheet in Documentation/arm/sunxi.rst + * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst */ #include "sun8i-ce.h" #include diff --git a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-trng.c b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-trng.c index e2b9b9104694..9c35f2a83eda 100644 --- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-trng.c +++ b/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-trng.c @@ -7,7 +7,7 @@ * * This file handle the TRNG * - * You could find a link for the datasheet in Documentation/arm/sunxi.rst + * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst */ #include "sun8i-ce.h" #include diff --git a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-cipher.c b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-cipher.c index 16966cc94e24..381a90fbeaff 100644 --- a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-cipher.c +++ b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-cipher.c @@ -8,7 +8,7 @@ * This file add support for AES cipher with 128,192,256 bits keysize in * CBC and ECB mode. * - * You could find a link for the datasheet in Documentation/arm/sunxi.rst + * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst */ #include diff --git a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c index c9dc06f97857..3dd844b40ff7 100644 --- a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c +++ b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-core.c @@ -7,7 +7,7 @@ * * Core file which registers crypto algorithms supported by the SecuritySystem * - * You could find a link for the datasheet in Documentation/arm/sunxi.rst + * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst */ #include #include diff --git a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-hash.c b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-hash.c index 577bf636f7fb..a4b67d130d11 100644 --- a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-hash.c +++ b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-hash.c @@ -7,7 +7,7 @@ * * This file add support for MD5 and SHA1/SHA224/SHA256. * - * You could find the datasheet in Documentation/arm/sunxi.rst + * You could find the datasheet in Documentation/arch/arm/sunxi.rst */ #include #include diff --git a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c index 70c7b5d571b8..a923cfc6553f 100644 --- a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c +++ b/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-prng.c @@ -7,7 +7,7 @@ * * This file handle the PRNG found in the SS * - * You could find a link for the datasheet in Documentation/arm/sunxi.rst + * You could find a link for the datasheet in Documentation/arch/arm/sunxi.rst */ #include "sun8i-ss.h" #include diff --git a/drivers/devfreq/exynos-bus.c b/drivers/devfreq/exynos-bus.c index 88414445adf3..245898f1a88e 100644 --- a/drivers/devfreq/exynos-bus.c +++ b/drivers/devfreq/exynos-bus.c @@ -518,6 +518,7 @@ static struct platform_driver exynos_bus_platdrv = { }; module_platform_driver(exynos_bus_platdrv); +MODULE_SOFTDEP("pre: exynos_ppmu"); MODULE_DESCRIPTION("Generic Exynos Bus frequency driver"); MODULE_AUTHOR("Chanwoo Choi "); MODULE_LICENSE("GPL v2"); diff --git a/drivers/devfreq/mtk-cci-devfreq.c b/drivers/devfreq/mtk-cci-devfreq.c index e5458ada5197..6354622eda65 100644 --- a/drivers/devfreq/mtk-cci-devfreq.c +++ b/drivers/devfreq/mtk-cci-devfreq.c @@ -127,7 +127,7 @@ static int mtk_ccifreq_target(struct device *dev, unsigned long *freq, u32 flags) { struct mtk_ccifreq_drv *drv = dev_get_drvdata(dev); - struct clk *cci_pll = clk_get_parent(drv->cci_clk); + struct clk *cci_pll; struct dev_pm_opp *opp; unsigned long opp_rate; int voltage, pre_voltage, inter_voltage, target_voltage, ret; @@ -139,6 +139,7 @@ static int mtk_ccifreq_target(struct device *dev, unsigned long *freq, return 0; inter_voltage = drv->inter_voltage; + cci_pll = clk_get_parent(drv->cci_clk); opp_rate = *freq; opp = devfreq_recommended_opp(dev, &opp_rate, 1); diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 1fc4fd79a1c6..1bab91ce8e95 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -218,7 +218,7 @@ static inline void set_trbe_enabled(struct trbe_cpudata *cpudata, u64 trblimitr) * Enable the TRBE without clearing LIMITPTR which * might be required for fetching the buffer limits. */ - trblimitr |= TRBLIMITR_ENABLE; + trblimitr |= TRBLIMITR_EL1_E; write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); /* Synchronize the TRBE enable event */ @@ -236,7 +236,7 @@ static inline void set_trbe_disabled(struct trbe_cpudata *cpudata) * Disable the TRBE without clearing LIMITPTR which * might be required for fetching the buffer limits. */ - trblimitr &= ~TRBLIMITR_ENABLE; + trblimitr &= ~TRBLIMITR_EL1_E; write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); if (trbe_needs_drain_after_disable(cpudata)) @@ -582,12 +582,12 @@ static void clr_trbe_status(void) u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); WARN_ON(is_trbe_enabled()); - trbsr &= ~TRBSR_IRQ; - trbsr &= ~TRBSR_TRG; - trbsr &= ~TRBSR_WRAP; - trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); - trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); - trbsr &= ~TRBSR_STOP; + trbsr &= ~TRBSR_EL1_IRQ; + trbsr &= ~TRBSR_EL1_TRG; + trbsr &= ~TRBSR_EL1_WRAP; + trbsr &= ~TRBSR_EL1_EC_MASK; + trbsr &= ~TRBSR_EL1_BSC_MASK; + trbsr &= ~TRBSR_EL1_S; write_sysreg_s(trbsr, SYS_TRBSR_EL1); } @@ -596,13 +596,13 @@ static void set_trbe_limit_pointer_enabled(struct trbe_buf *buf) u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); unsigned long addr = buf->trbe_limit; - WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_EL1_LIMIT_SHIFT))); WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); - trblimitr &= ~TRBLIMITR_NVM; - trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); - trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); - trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); + trblimitr &= ~TRBLIMITR_EL1_nVM; + trblimitr &= ~TRBLIMITR_EL1_FM_MASK; + trblimitr &= ~TRBLIMITR_EL1_TM_MASK; + trblimitr &= ~TRBLIMITR_EL1_LIMIT_MASK; /* * Fill trace buffer mode is used here while configuring the @@ -613,14 +613,15 @@ static void set_trbe_limit_pointer_enabled(struct trbe_buf *buf) * trace data in the interrupt handler, before reconfiguring * the TRBE. */ - trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; + trblimitr |= (TRBLIMITR_EL1_FM_FILL << TRBLIMITR_EL1_FM_SHIFT) & + TRBLIMITR_EL1_FM_MASK; /* * Trigger mode is not used here while configuring the TRBE for * the trace capture. Hence just keep this in the ignore mode. */ - trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << - TRBLIMITR_TRIG_MODE_SHIFT; + trblimitr |= (TRBLIMITR_EL1_TM_IGNR << TRBLIMITR_EL1_TM_SHIFT) & + TRBLIMITR_EL1_TM_MASK; trblimitr |= (addr & PAGE_MASK); set_trbe_enabled(buf->cpudata, trblimitr); } diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h index 98ff1b17ad07..77cbb5c63878 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.h +++ b/drivers/hwtracing/coresight/coresight-trbe.h @@ -30,7 +30,7 @@ static inline bool is_trbe_enabled(void) { u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); - return trblimitr & TRBLIMITR_ENABLE; + return trblimitr & TRBLIMITR_EL1_E; } #define TRBE_EC_OTHERS 0 @@ -39,7 +39,7 @@ static inline bool is_trbe_enabled(void) static inline int get_trbe_ec(u64 trbsr) { - return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; + return (trbsr & TRBSR_EL1_EC_MASK) >> TRBSR_EL1_EC_SHIFT; } #define TRBE_BSC_NOT_STOPPED 0 @@ -48,63 +48,55 @@ static inline int get_trbe_ec(u64 trbsr) static inline int get_trbe_bsc(u64 trbsr) { - return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; + return (trbsr & TRBSR_EL1_BSC_MASK) >> TRBSR_EL1_BSC_SHIFT; } static inline void clr_trbe_irq(void) { u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); - trbsr &= ~TRBSR_IRQ; + trbsr &= ~TRBSR_EL1_IRQ; write_sysreg_s(trbsr, SYS_TRBSR_EL1); } static inline bool is_trbe_irq(u64 trbsr) { - return trbsr & TRBSR_IRQ; + return trbsr & TRBSR_EL1_IRQ; } static inline bool is_trbe_trg(u64 trbsr) { - return trbsr & TRBSR_TRG; + return trbsr & TRBSR_EL1_TRG; } static inline bool is_trbe_wrap(u64 trbsr) { - return trbsr & TRBSR_WRAP; + return trbsr & TRBSR_EL1_WRAP; } static inline bool is_trbe_abort(u64 trbsr) { - return trbsr & TRBSR_ABORT; + return trbsr & TRBSR_EL1_EA; } static inline bool is_trbe_running(u64 trbsr) { - return !(trbsr & TRBSR_STOP); + return !(trbsr & TRBSR_EL1_S); } -#define TRBE_TRIG_MODE_STOP 0 -#define TRBE_TRIG_MODE_IRQ 1 -#define TRBE_TRIG_MODE_IGNORE 3 - -#define TRBE_FILL_MODE_FILL 0 -#define TRBE_FILL_MODE_WRAP 1 -#define TRBE_FILL_MODE_CIRCULAR_BUFFER 3 - static inline bool get_trbe_flag_update(u64 trbidr) { - return trbidr & TRBIDR_FLAG; + return trbidr & TRBIDR_EL1_F; } static inline bool is_trbe_programmable(u64 trbidr) { - return !(trbidr & TRBIDR_PROG); + return !(trbidr & TRBIDR_EL1_P); } static inline int get_trbe_address_align(u64 trbidr) { - return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; + return (trbidr & TRBIDR_EL1_Align_MASK) >> TRBIDR_EL1_Align_SHIFT; } static inline unsigned long get_trbe_write_pointer(void) @@ -121,7 +113,7 @@ static inline void set_trbe_write_pointer(unsigned long addr) static inline unsigned long get_trbe_limit_pointer(void) { u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); - unsigned long addr = trblimitr & (TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); + unsigned long addr = trblimitr & TRBLIMITR_EL1_LIMIT_MASK; WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); return addr; @@ -130,7 +122,7 @@ static inline unsigned long get_trbe_limit_pointer(void) static inline unsigned long get_trbe_base_pointer(void) { u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); - unsigned long addr = trbbaser & (TRBBASER_BASE_MASK << TRBBASER_BASE_SHIFT); + unsigned long addr = trbbaser & TRBBASER_EL1_BASE_MASK; WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); return addr; @@ -139,7 +131,7 @@ static inline unsigned long get_trbe_base_pointer(void) static inline void set_trbe_base_pointer(unsigned long addr) { WARN_ON(is_trbe_enabled()); - WARN_ON(!IS_ALIGNED(addr, (1UL << TRBBASER_BASE_SHIFT))); + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBBASER_EL1_BASE_SHIFT))); WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); write_sysreg_s(addr, SYS_TRBBASER_EL1); } diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c index aa2d19db2b1d..34201d7ef33e 100644 --- a/drivers/idle/intel_idle.c +++ b/drivers/idle/intel_idle.c @@ -199,6 +199,43 @@ static __cpuidle int intel_idle_xstate(struct cpuidle_device *dev, return __intel_idle(dev, drv, index); } +static __always_inline int __intel_idle_hlt(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index) +{ + raw_safe_halt(); + raw_local_irq_disable(); + return index; +} + +/** + * intel_idle_hlt - Ask the processor to enter the given idle state using hlt. + * @dev: cpuidle device of the target CPU. + * @drv: cpuidle driver (assumed to point to intel_idle_driver). + * @index: Target idle state index. + * + * Use the HLT instruction to notify the processor that the CPU represented by + * @dev is idle and it can try to enter the idle state corresponding to @index. + * + * Must be called under local_irq_disable(). + */ +static __cpuidle int intel_idle_hlt(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index) +{ + return __intel_idle_hlt(dev, drv, index); +} + +static __cpuidle int intel_idle_hlt_irq_on(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index) +{ + int ret; + + raw_local_irq_enable(); + ret = __intel_idle_hlt(dev, drv, index); + raw_local_irq_disable(); + + return ret; +} + /** * intel_idle_s2idle - Ask the processor to enter the given idle state. * @dev: cpuidle device of the target CPU. @@ -1242,6 +1279,25 @@ static struct cpuidle_state snr_cstates[] __initdata = { .enter = NULL } }; +static struct cpuidle_state vmguest_cstates[] __initdata = { + { + .name = "C1", + .desc = "HLT", + .flags = MWAIT2flg(0x00) | CPUIDLE_FLAG_IRQ_ENABLE, + .exit_latency = 5, + .target_residency = 10, + .enter = &intel_idle_hlt, }, + { + .name = "C1L", + .desc = "Long HLT", + .flags = MWAIT2flg(0x00) | CPUIDLE_FLAG_TLB_FLUSHED, + .exit_latency = 5, + .target_residency = 200, + .enter = &intel_idle_hlt, }, + { + .enter = NULL } +}; + static const struct idle_cpu idle_cpu_nehalem __initconst = { .state_table = nehalem_cstates, .auto_demotion_disable_flags = NHM_C1_AUTO_DEMOTE | NHM_C3_AUTO_DEMOTE, @@ -1839,6 +1895,66 @@ static bool __init intel_idle_verify_cstate(unsigned int mwait_hint) return true; } +static void state_update_enter_method(struct cpuidle_state *state, int cstate) +{ + if (state->enter == intel_idle_hlt) { + if (force_irq_on) { + pr_info("forced intel_idle_irq for state %d\n", cstate); + state->enter = intel_idle_hlt_irq_on; + } + return; + } + if (state->enter == intel_idle_hlt_irq_on) + return; /* no update scenarios */ + + if (state->flags & CPUIDLE_FLAG_INIT_XSTATE) { + /* + * Combining with XSTATE with IBRS or IRQ_ENABLE flags + * is not currently supported but this driver. + */ + WARN_ON_ONCE(state->flags & CPUIDLE_FLAG_IBRS); + WARN_ON_ONCE(state->flags & CPUIDLE_FLAG_IRQ_ENABLE); + state->enter = intel_idle_xstate; + return; + } + + if (cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS) && + state->flags & CPUIDLE_FLAG_IBRS) { + /* + * IBRS mitigation requires that C-states are entered + * with interrupts disabled. + */ + WARN_ON_ONCE(state->flags & CPUIDLE_FLAG_IRQ_ENABLE); + state->enter = intel_idle_ibrs; + return; + } + + if (state->flags & CPUIDLE_FLAG_IRQ_ENABLE) { + state->enter = intel_idle_irq; + return; + } + + if (force_irq_on) { + pr_info("forced intel_idle_irq for state %d\n", cstate); + state->enter = intel_idle_irq; + } +} + +/* + * For mwait based states, we want to verify the cpuid data to see if the state + * is actually supported by this specific CPU. + * For non-mwait based states, this check should be skipped. + */ +static bool should_verify_mwait(struct cpuidle_state *state) +{ + if (state->enter == intel_idle_hlt) + return false; + if (state->enter == intel_idle_hlt_irq_on) + return false; + + return true; +} + static void __init intel_idle_init_cstates_icpu(struct cpuidle_driver *drv) { int cstate; @@ -1887,35 +2003,15 @@ static void __init intel_idle_init_cstates_icpu(struct cpuidle_driver *drv) } mwait_hint = flg2MWAIT(cpuidle_state_table[cstate].flags); - if (!intel_idle_verify_cstate(mwait_hint)) + if (should_verify_mwait(&cpuidle_state_table[cstate]) && !intel_idle_verify_cstate(mwait_hint)) continue; /* Structure copy. */ drv->states[drv->state_count] = cpuidle_state_table[cstate]; state = &drv->states[drv->state_count]; - if (state->flags & CPUIDLE_FLAG_INIT_XSTATE) { - /* - * Combining with XSTATE with IBRS or IRQ_ENABLE flags - * is not currently supported but this driver. - */ - WARN_ON_ONCE(state->flags & CPUIDLE_FLAG_IBRS); - WARN_ON_ONCE(state->flags & CPUIDLE_FLAG_IRQ_ENABLE); - state->enter = intel_idle_xstate; - } else if (cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS) && - state->flags & CPUIDLE_FLAG_IBRS) { - /* - * IBRS mitigation requires that C-states are entered - * with interrupts disabled. - */ - WARN_ON_ONCE(state->flags & CPUIDLE_FLAG_IRQ_ENABLE); - state->enter = intel_idle_ibrs; - } else if (state->flags & CPUIDLE_FLAG_IRQ_ENABLE) { - state->enter = intel_idle_irq; - } else if (force_irq_on) { - pr_info("forced intel_idle_irq for state %d\n", cstate); - state->enter = intel_idle_irq; - } + state_update_enter_method(state, cstate); + if ((disabled_states_mask & BIT(drv->state_count)) || ((icpu->use_acpi || force_use_acpi) && @@ -2041,6 +2137,93 @@ static void __init intel_idle_cpuidle_devices_uninit(void) cpuidle_unregister_device(per_cpu_ptr(intel_idle_cpuidle_devices, i)); } +/* + * Match up the latency and break even point of the bare metal (cpu based) + * states with the deepest VM available state. + * + * We only want to do this for the deepest state, the ones that has + * the TLB_FLUSHED flag set on the . + * + * All our short idle states are dominated by vmexit/vmenter latencies, + * not the underlying hardware latencies so we keep our values for these. + */ +static void matchup_vm_state_with_baremetal(void) +{ + int cstate; + + for (cstate = 0; cstate < CPUIDLE_STATE_MAX; ++cstate) { + int matching_cstate; + + if (intel_idle_max_cstate_reached(cstate)) + break; + + if (!cpuidle_state_table[cstate].enter) + break; + + if (!(cpuidle_state_table[cstate].flags & CPUIDLE_FLAG_TLB_FLUSHED)) + continue; + + for (matching_cstate = 0; matching_cstate < CPUIDLE_STATE_MAX; ++matching_cstate) { + if (!icpu->state_table[matching_cstate].enter) + break; + if (icpu->state_table[matching_cstate].exit_latency > cpuidle_state_table[cstate].exit_latency) { + cpuidle_state_table[cstate].exit_latency = icpu->state_table[matching_cstate].exit_latency; + cpuidle_state_table[cstate].target_residency = icpu->state_table[matching_cstate].target_residency; + } + } + + } +} + + +static int __init intel_idle_vminit(const struct x86_cpu_id *id) +{ + int retval; + + cpuidle_state_table = vmguest_cstates; + + icpu = (const struct idle_cpu *)id->driver_data; + + pr_debug("v" INTEL_IDLE_VERSION " model 0x%X\n", + boot_cpu_data.x86_model); + + intel_idle_cpuidle_devices = alloc_percpu(struct cpuidle_device); + if (!intel_idle_cpuidle_devices) + return -ENOMEM; + + /* + * We don't know exactly what the host will do when we go idle, but as a worst estimate + * we can assume that the exit latency of the deepest host state will be hit for our + * deep (long duration) guest idle state. + * The same logic applies to the break even point for the long duration guest idle state. + * So lets copy these two properties from the table we found for the host CPU type. + */ + matchup_vm_state_with_baremetal(); + + intel_idle_cpuidle_driver_init(&intel_idle_driver); + + retval = cpuidle_register_driver(&intel_idle_driver); + if (retval) { + struct cpuidle_driver *drv = cpuidle_get_driver(); + printk(KERN_DEBUG pr_fmt("intel_idle yielding to %s\n"), + drv ? drv->name : "none"); + goto init_driver_fail; + } + + retval = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "idle/intel:online", + intel_idle_cpu_online, NULL); + if (retval < 0) + goto hp_setup_fail; + + return 0; +hp_setup_fail: + intel_idle_cpuidle_devices_uninit(); + cpuidle_unregister_driver(&intel_idle_driver); +init_driver_fail: + free_percpu(intel_idle_cpuidle_devices); + return retval; +} + static int __init intel_idle_init(void) { const struct x86_cpu_id *id; @@ -2059,6 +2242,8 @@ static int __init intel_idle_init(void) id = x86_match_cpu(intel_idle_ids); if (id) { if (!boot_cpu_has(X86_FEATURE_MWAIT)) { + if (boot_cpu_has(X86_FEATURE_HYPERVISOR)) + return intel_idle_vminit(id); pr_debug("Please enable MWAIT in BIOS SETUP\n"); return -ENODEV; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 4d8f6b8051ff..83093e16b6c6 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1357,7 +1357,7 @@ static int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) if (cleanup_err) rxe_err_mr(mr, "cleanup failed, err = %d", cleanup_err); - kfree_rcu(mr); + kfree_rcu_mightsleep(mr); return 0; err_out: diff --git a/drivers/input/touchscreen/sun4i-ts.c b/drivers/input/touchscreen/sun4i-ts.c index 577c75c83e25..bb3c6072fc82 100644 --- a/drivers/input/touchscreen/sun4i-ts.c +++ b/drivers/input/touchscreen/sun4i-ts.c @@ -22,7 +22,7 @@ * in the kernel). So this driver offers straight forward, reliable single * touch functionality only. * - * s.a. A20 User Manual "1.15 TP" (Documentation/arm/sunxi.rst) + * s.a. A20 User Manual "1.15 TP" (Documentation/arch/arm/sunxi.rst) * (looks like the description in the A20 User Manual v1.3 is better * than the one in the A10 User Manual v.1.5) */ diff --git a/drivers/irqchip/irq-mxs.c b/drivers/irqchip/irq-mxs.c index 55cb6b5a686e..be9680645545 100644 --- a/drivers/irqchip/irq-mxs.c +++ b/drivers/irqchip/irq-mxs.c @@ -201,6 +201,7 @@ static int __init icoll_of_init(struct device_node *np, stmp_reset_block(icoll_priv.ctrl); icoll_add_domain(np, ICOLL_NUM_IRQS); + set_handle_irq(icoll_handle_irq); return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/thermal.c b/drivers/net/ethernet/mellanox/mlx5/core/thermal.c index e47fa6fb836f..20bb5eb266c1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/thermal.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/thermal.c @@ -45,7 +45,7 @@ static int mlx5_thermal_get_mtmp_temp(struct mlx5_core_dev *mdev, u32 id, int *p static int mlx5_thermal_get_temp(struct thermal_zone_device *tzdev, int *p_temp) { - struct mlx5_thermal *thermal = tzdev->devdata; + struct mlx5_thermal *thermal = thermal_zone_device_priv(tzdev); struct mlx5_core_dev *mdev = thermal->mdev; int err; @@ -81,12 +81,13 @@ int mlx5_thermal_init(struct mlx5_core_dev *mdev) return -ENOMEM; thermal->mdev = mdev; - thermal->tzdev = thermal_zone_device_register(data, - MLX5_THERMAL_NUM_TRIPS, - MLX5_THERMAL_TRIP_MASK, - thermal, - &mlx5_thermal_ops, - NULL, 0, MLX5_THERMAL_POLL_INT_MSEC); + thermal->tzdev = thermal_zone_device_register_with_trips(data, + NULL, + MLX5_THERMAL_NUM_TRIPS, + MLX5_THERMAL_TRIP_MASK, + thermal, + &mlx5_thermal_ops, + NULL, 0, MLX5_THERMAL_POLL_INT_MSEC); if (IS_ERR(thermal->tzdev)) { dev_err(mdev->device, "Failed to register thermal zone device (%s) %ld\n", data, PTR_ERR(thermal->tzdev)); diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 711f82400086..4c07d71cab0b 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -127,6 +127,14 @@ config FSL_IMX8_DDR_PMU can give information about memory throughput and other related events. +config FSL_IMX9_DDR_PMU + tristate "Freescale i.MX9 DDR perf monitor" + depends on ARCH_MXC + help + Provides support for the DDR performance monitor in i.MX9, which + can give information about memory throughput and other related + events. + config QCOM_L2_PMU bool "Qualcomm Technologies L2-cache PMU" depends on ARCH_QCOM && ARM64 && ACPI diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index dabc859540ce..5cfe8954c250 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -8,6 +8,7 @@ obj-$(CONFIG_ARM_PMU_ACPI) += arm_pmu_acpi.o obj-$(CONFIG_ARM_PMUV3) += arm_pmuv3.o obj-$(CONFIG_ARM_SMMU_V3_PMU) += arm_smmuv3_pmu.o obj-$(CONFIG_FSL_IMX8_DDR_PMU) += fsl_imx8_ddr_perf.o +obj-$(CONFIG_FSL_IMX9_DDR_PMU) += fsl_imx9_ddr_perf.o obj-$(CONFIG_HISI_PMU) += hisilicon/ obj-$(CONFIG_QCOM_L2_PMU) += qcom_l2_pmu.o obj-$(CONFIG_QCOM_L3_PMU) += qcom_l3_pmu.o diff --git a/drivers/perf/apple_m1_cpu_pmu.c b/drivers/perf/apple_m1_cpu_pmu.c index 8574c6e58c83..cd2de44b61b9 100644 --- a/drivers/perf/apple_m1_cpu_pmu.c +++ b/drivers/perf/apple_m1_cpu_pmu.c @@ -493,6 +493,17 @@ static int m1_pmu_map_event(struct perf_event *event) return armpmu_map_event(event, &m1_pmu_perf_map, NULL, M1_PMU_CFG_EVENT); } +static int m2_pmu_map_event(struct perf_event *event) +{ + /* + * Same deal as the above, except that M2 has 64bit counters. + * Which, as far as we're concerned, actually means 63 bits. + * Yes, this is getting awkward. + */ + event->hw.flags |= ARMPMU_EVT_63BIT; + return armpmu_map_event(event, &m1_pmu_perf_map, NULL, M1_PMU_CFG_EVENT); +} + static void m1_pmu_reset(void *info) { int i; @@ -525,7 +536,7 @@ static int m1_pmu_set_event_filter(struct hw_perf_event *event, return 0; } -static int m1_pmu_init(struct arm_pmu *cpu_pmu) +static int m1_pmu_init(struct arm_pmu *cpu_pmu, u32 flags) { cpu_pmu->handle_irq = m1_pmu_handle_irq; cpu_pmu->enable = m1_pmu_enable_event; @@ -536,7 +547,14 @@ static int m1_pmu_init(struct arm_pmu *cpu_pmu) cpu_pmu->clear_event_idx = m1_pmu_clear_event_idx; cpu_pmu->start = m1_pmu_start; cpu_pmu->stop = m1_pmu_stop; - cpu_pmu->map_event = m1_pmu_map_event; + + if (flags & ARMPMU_EVT_47BIT) + cpu_pmu->map_event = m1_pmu_map_event; + else if (flags & ARMPMU_EVT_63BIT) + cpu_pmu->map_event = m2_pmu_map_event; + else + return WARN_ON(-EINVAL); + cpu_pmu->reset = m1_pmu_reset; cpu_pmu->set_event_filter = m1_pmu_set_event_filter; @@ -550,25 +568,25 @@ static int m1_pmu_init(struct arm_pmu *cpu_pmu) static int m1_pmu_ice_init(struct arm_pmu *cpu_pmu) { cpu_pmu->name = "apple_icestorm_pmu"; - return m1_pmu_init(cpu_pmu); + return m1_pmu_init(cpu_pmu, ARMPMU_EVT_47BIT); } static int m1_pmu_fire_init(struct arm_pmu *cpu_pmu) { cpu_pmu->name = "apple_firestorm_pmu"; - return m1_pmu_init(cpu_pmu); + return m1_pmu_init(cpu_pmu, ARMPMU_EVT_47BIT); } static int m2_pmu_avalanche_init(struct arm_pmu *cpu_pmu) { cpu_pmu->name = "apple_avalanche_pmu"; - return m1_pmu_init(cpu_pmu); + return m1_pmu_init(cpu_pmu, ARMPMU_EVT_63BIT); } static int m2_pmu_blizzard_init(struct arm_pmu *cpu_pmu) { cpu_pmu->name = "apple_blizzard_pmu"; - return m1_pmu_init(cpu_pmu); + return m1_pmu_init(cpu_pmu, ARMPMU_EVT_63BIT); } static const struct of_device_id m1_pmu_of_device_ids[] = { diff --git a/drivers/perf/arm-cci.c b/drivers/perf/arm-cci.c index 03b1309875ae..998259f1d973 100644 --- a/drivers/perf/arm-cci.c +++ b/drivers/perf/arm-cci.c @@ -645,7 +645,7 @@ static void cci_pmu_sync_counters(struct cci_pmu *cci_pmu) struct cci_pmu_hw_events *cci_hw = &cci_pmu->hw_events; DECLARE_BITMAP(mask, HW_CNTRS_MAX); - bitmap_zero(mask, cci_pmu->num_cntrs); + bitmap_zero(mask, HW_CNTRS_MAX); for_each_set_bit(i, cci_pmu->hw_events.used_mask, cci_pmu->num_cntrs) { struct perf_event *event = cci_hw->events[i]; @@ -656,7 +656,7 @@ static void cci_pmu_sync_counters(struct cci_pmu *cci_pmu) if (event->hw.state & PERF_HES_STOPPED) continue; if (event->hw.state & PERF_HES_ARCH) { - set_bit(i, mask); + __set_bit(i, mask); event->hw.state &= ~PERF_HES_ARCH; } } diff --git a/drivers/perf/arm-cmn.c b/drivers/perf/arm-cmn.c index 47d359f72957..b8c15878bc86 100644 --- a/drivers/perf/arm-cmn.c +++ b/drivers/perf/arm-cmn.c @@ -44,8 +44,11 @@ #define CMN_MAX_DTMS (CMN_MAX_XPS + (CMN_MAX_DIMENSION - 1) * 4) /* The CFG node has various info besides the discovery tree */ -#define CMN_CFGM_PERIPH_ID_2 0x0010 -#define CMN_CFGM_PID2_REVISION GENMASK(7, 4) +#define CMN_CFGM_PERIPH_ID_01 0x0008 +#define CMN_CFGM_PID0_PART_0 GENMASK_ULL(7, 0) +#define CMN_CFGM_PID1_PART_1 GENMASK_ULL(35, 32) +#define CMN_CFGM_PERIPH_ID_23 0x0010 +#define CMN_CFGM_PID2_REVISION GENMASK_ULL(7, 4) #define CMN_CFGM_INFO_GLOBAL 0x900 #define CMN_INFO_MULTIPLE_DTM_EN BIT_ULL(63) @@ -186,6 +189,7 @@ #define CMN_WP_DOWN 2 +/* Internal values for encoding event support */ enum cmn_model { CMN600 = 1, CMN650 = 2, @@ -197,26 +201,34 @@ enum cmn_model { CMN_650ON = CMN650 | CMN700, }; +/* Actual part numbers and revision IDs defined by the hardware */ +enum cmn_part { + PART_CMN600 = 0x434, + PART_CMN650 = 0x436, + PART_CMN700 = 0x43c, + PART_CI700 = 0x43a, +}; + /* CMN-600 r0px shouldn't exist in silicon, thankfully */ enum cmn_revision { - CMN600_R1P0, - CMN600_R1P1, - CMN600_R1P2, - CMN600_R1P3, - CMN600_R2P0, - CMN600_R3P0, - CMN600_R3P1, - CMN650_R0P0 = 0, - CMN650_R1P0, - CMN650_R1P1, - CMN650_R2P0, - CMN650_R1P2, - CMN700_R0P0 = 0, - CMN700_R1P0, - CMN700_R2P0, - CI700_R0P0 = 0, - CI700_R1P0, - CI700_R2P0, + REV_CMN600_R1P0, + REV_CMN600_R1P1, + REV_CMN600_R1P2, + REV_CMN600_R1P3, + REV_CMN600_R2P0, + REV_CMN600_R3P0, + REV_CMN600_R3P1, + REV_CMN650_R0P0 = 0, + REV_CMN650_R1P0, + REV_CMN650_R1P1, + REV_CMN650_R2P0, + REV_CMN650_R1P2, + REV_CMN700_R0P0 = 0, + REV_CMN700_R1P0, + REV_CMN700_R2P0, + REV_CI700_R0P0 = 0, + REV_CI700_R1P0, + REV_CI700_R2P0, }; enum cmn_node_type { @@ -306,7 +318,7 @@ struct arm_cmn { unsigned int state; enum cmn_revision rev; - enum cmn_model model; + enum cmn_part part; u8 mesh_x; u8 mesh_y; u16 num_xps; @@ -394,19 +406,35 @@ static struct arm_cmn_node *arm_cmn_node(const struct arm_cmn *cmn, return NULL; } +static enum cmn_model arm_cmn_model(const struct arm_cmn *cmn) +{ + switch (cmn->part) { + case PART_CMN600: + return CMN600; + case PART_CMN650: + return CMN650; + case PART_CMN700: + return CMN700; + case PART_CI700: + return CI700; + default: + return 0; + }; +} + static u32 arm_cmn_device_connect_info(const struct arm_cmn *cmn, const struct arm_cmn_node *xp, int port) { int offset = CMN_MXP__CONNECT_INFO(port); if (port >= 2) { - if (cmn->model & (CMN600 | CMN650)) + if (cmn->part == PART_CMN600 || cmn->part == PART_CMN650) return 0; /* * CI-700 may have extra ports, but still has the * mesh_port_connect_info registers in the way. */ - if (cmn->model == CI700) + if (cmn->part == PART_CI700) offset += CI700_CONNECT_INFO_P2_5_OFFSET; } @@ -640,7 +668,7 @@ static umode_t arm_cmn_event_attr_is_visible(struct kobject *kobj, eattr = container_of(attr, typeof(*eattr), attr.attr); - if (!(eattr->model & cmn->model)) + if (!(eattr->model & arm_cmn_model(cmn))) return 0; type = eattr->type; @@ -658,7 +686,7 @@ static umode_t arm_cmn_event_attr_is_visible(struct kobject *kobj, if ((intf & 4) && !(cmn->ports_used & BIT(intf & 3))) return 0; - if (chan == 4 && cmn->model == CMN600) + if (chan == 4 && cmn->part == PART_CMN600) return 0; if ((chan == 5 && cmn->rsp_vc_num < 2) || @@ -669,19 +697,19 @@ static umode_t arm_cmn_event_attr_is_visible(struct kobject *kobj, } /* Revision-specific differences */ - if (cmn->model == CMN600) { - if (cmn->rev < CMN600_R1P3) { + if (cmn->part == PART_CMN600) { + if (cmn->rev < REV_CMN600_R1P3) { if (type == CMN_TYPE_CXRA && eventid > 0x10) return 0; } - if (cmn->rev < CMN600_R1P2) { + if (cmn->rev < REV_CMN600_R1P2) { if (type == CMN_TYPE_HNF && eventid == 0x1b) return 0; if (type == CMN_TYPE_CXRA || type == CMN_TYPE_CXHA) return 0; } - } else if (cmn->model == CMN650) { - if (cmn->rev < CMN650_R2P0 || cmn->rev == CMN650_R1P2) { + } else if (cmn->part == PART_CMN650) { + if (cmn->rev < REV_CMN650_R2P0 || cmn->rev == REV_CMN650_R1P2) { if (type == CMN_TYPE_HNF && eventid > 0x22) return 0; if (type == CMN_TYPE_SBSX && eventid == 0x17) @@ -689,8 +717,8 @@ static umode_t arm_cmn_event_attr_is_visible(struct kobject *kobj, if (type == CMN_TYPE_RNI && eventid > 0x10) return 0; } - } else if (cmn->model == CMN700) { - if (cmn->rev < CMN700_R2P0) { + } else if (cmn->part == PART_CMN700) { + if (cmn->rev < REV_CMN700_R2P0) { if (type == CMN_TYPE_HNF && eventid > 0x2c) return 0; if (type == CMN_TYPE_CCHA && eventid > 0x74) @@ -698,7 +726,7 @@ static umode_t arm_cmn_event_attr_is_visible(struct kobject *kobj, if (type == CMN_TYPE_CCLA && eventid > 0x27) return 0; } - if (cmn->rev < CMN700_R1P0) { + if (cmn->rev < REV_CMN700_R1P0) { if (type == CMN_TYPE_HNF && eventid > 0x2b) return 0; } @@ -1171,19 +1199,31 @@ static ssize_t arm_cmn_cpumask_show(struct device *dev, static struct device_attribute arm_cmn_cpumask_attr = __ATTR(cpumask, 0444, arm_cmn_cpumask_show, NULL); -static struct attribute *arm_cmn_cpumask_attrs[] = { +static ssize_t arm_cmn_identifier_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct arm_cmn *cmn = to_cmn(dev_get_drvdata(dev)); + + return sysfs_emit(buf, "%03x%02x\n", cmn->part, cmn->rev); +} + +static struct device_attribute arm_cmn_identifier_attr = + __ATTR(identifier, 0444, arm_cmn_identifier_show, NULL); + +static struct attribute *arm_cmn_other_attrs[] = { &arm_cmn_cpumask_attr.attr, + &arm_cmn_identifier_attr.attr, NULL, }; -static const struct attribute_group arm_cmn_cpumask_attr_group = { - .attrs = arm_cmn_cpumask_attrs, +static const struct attribute_group arm_cmn_other_attrs_group = { + .attrs = arm_cmn_other_attrs, }; static const struct attribute_group *arm_cmn_attr_groups[] = { &arm_cmn_event_attrs_group, &arm_cmn_format_attrs_group, - &arm_cmn_cpumask_attr_group, + &arm_cmn_other_attrs_group, NULL }; @@ -1200,7 +1240,7 @@ static u32 arm_cmn_wp_config(struct perf_event *event) u32 grp = CMN_EVENT_WP_GRP(event); u32 exc = CMN_EVENT_WP_EXCLUSIVE(event); u32 combine = CMN_EVENT_WP_COMBINE(event); - bool is_cmn600 = to_cmn(event->pmu)->model == CMN600; + bool is_cmn600 = to_cmn(event->pmu)->part == PART_CMN600; config = FIELD_PREP(CMN_DTM_WPn_CONFIG_WP_DEV_SEL, dev) | FIELD_PREP(CMN_DTM_WPn_CONFIG_WP_CHN_SEL, chn) | @@ -1520,14 +1560,14 @@ done: return ret; } -static enum cmn_filter_select arm_cmn_filter_sel(enum cmn_model model, +static enum cmn_filter_select arm_cmn_filter_sel(const struct arm_cmn *cmn, enum cmn_node_type type, unsigned int eventid) { struct arm_cmn_event_attr *e; - int i; + enum cmn_model model = arm_cmn_model(cmn); - for (i = 0; i < ARRAY_SIZE(arm_cmn_event_attrs) - 1; i++) { + for (int i = 0; i < ARRAY_SIZE(arm_cmn_event_attrs) - 1; i++) { e = container_of(arm_cmn_event_attrs[i], typeof(*e), attr.attr); if (e->model & model && e->type == type && e->eventid == eventid) return e->fsel; @@ -1570,12 +1610,12 @@ static int arm_cmn_event_init(struct perf_event *event) /* ...but the DTM may depend on which port we're watching */ if (cmn->multi_dtm) hw->dtm_offset = CMN_EVENT_WP_DEV_SEL(event) / 2; - } else if (type == CMN_TYPE_XP && cmn->model == CMN700) { + } else if (type == CMN_TYPE_XP && cmn->part == PART_CMN700) { hw->wide_sel = true; } /* This is sufficiently annoying to recalculate, so cache it */ - hw->filter_sel = arm_cmn_filter_sel(cmn->model, type, eventid); + hw->filter_sel = arm_cmn_filter_sel(cmn, type, eventid); bynodeid = CMN_EVENT_BYNODEID(event); nodeid = CMN_EVENT_NODEID(event); @@ -1899,9 +1939,10 @@ static int arm_cmn_init_dtc(struct arm_cmn *cmn, struct arm_cmn_node *dn, int id if (dtc->irq < 0) return dtc->irq; - writel_relaxed(0, dtc->base + CMN_DT_PMCR); + writel_relaxed(CMN_DT_DTC_CTL_DT_EN, dtc->base + CMN_DT_DTC_CTL); + writel_relaxed(CMN_DT_PMCR_PMU_EN | CMN_DT_PMCR_OVFL_INTR_EN, dtc->base + CMN_DT_PMCR); + writeq_relaxed(0, dtc->base + CMN_DT_PMCCNTR); writel_relaxed(0x1ff, dtc->base + CMN_DT_PMOVSR_CLR); - writel_relaxed(CMN_DT_PMCR_OVFL_INTR_EN, dtc->base + CMN_DT_PMCR); return 0; } @@ -1961,7 +2002,7 @@ static int arm_cmn_init_dtcs(struct arm_cmn *cmn) dn->type = CMN_TYPE_CCLA; } - writel_relaxed(CMN_DT_DTC_CTL_DT_EN, cmn->dtc[0].base + CMN_DT_DTC_CTL); + arm_cmn_set_state(cmn, CMN_STATE_DISABLED); return 0; } @@ -2006,6 +2047,7 @@ static int arm_cmn_discover(struct arm_cmn *cmn, unsigned int rgn_offset) void __iomem *cfg_region; struct arm_cmn_node cfg, *dn; struct arm_cmn_dtm *dtm; + enum cmn_part part; u16 child_count, child_poff; u32 xp_offset[CMN_MAX_XPS]; u64 reg; @@ -2017,7 +2059,19 @@ static int arm_cmn_discover(struct arm_cmn *cmn, unsigned int rgn_offset) return -ENODEV; cfg_region = cmn->base + rgn_offset; - reg = readl_relaxed(cfg_region + CMN_CFGM_PERIPH_ID_2); + + reg = readq_relaxed(cfg_region + CMN_CFGM_PERIPH_ID_01); + part = FIELD_GET(CMN_CFGM_PID0_PART_0, reg); + part |= FIELD_GET(CMN_CFGM_PID1_PART_1, reg) << 8; + if (cmn->part && cmn->part != part) + dev_warn(cmn->dev, + "Firmware binding mismatch: expected part number 0x%x, found 0x%x\n", + cmn->part, part); + cmn->part = part; + if (!arm_cmn_model(cmn)) + dev_warn(cmn->dev, "Unknown part number: 0x%x\n", part); + + reg = readl_relaxed(cfg_region + CMN_CFGM_PERIPH_ID_23); cmn->rev = FIELD_GET(CMN_CFGM_PID2_REVISION, reg); reg = readq_relaxed(cfg_region + CMN_CFGM_INFO_GLOBAL); @@ -2081,7 +2135,7 @@ static int arm_cmn_discover(struct arm_cmn *cmn, unsigned int rgn_offset) if (xp->id == (1 << 3)) cmn->mesh_x = xp->logid; - if (cmn->model == CMN600) + if (cmn->part == PART_CMN600) xp->dtc = 0xf; else xp->dtc = 1 << readl_relaxed(xp_region + CMN_DTM_UNIT_INFO); @@ -2201,7 +2255,7 @@ static int arm_cmn_discover(struct arm_cmn *cmn, unsigned int rgn_offset) if (cmn->num_xps == 1) dev_warn(cmn->dev, "1x1 config not fully supported, translate XP events manually\n"); - dev_dbg(cmn->dev, "model %d, periph_id_2 revision %d\n", cmn->model, cmn->rev); + dev_dbg(cmn->dev, "periph_id part 0x%03x revision %d\n", cmn->part, cmn->rev); reg = cmn->ports_used; dev_dbg(cmn->dev, "mesh %dx%d, ID width %d, ports %6pbl%s\n", cmn->mesh_x, cmn->mesh_y, arm_cmn_xyidbits(cmn), ®, @@ -2256,17 +2310,17 @@ static int arm_cmn_probe(struct platform_device *pdev) return -ENOMEM; cmn->dev = &pdev->dev; - cmn->model = (unsigned long)device_get_match_data(cmn->dev); + cmn->part = (unsigned long)device_get_match_data(cmn->dev); platform_set_drvdata(pdev, cmn); - if (cmn->model == CMN600 && has_acpi_companion(cmn->dev)) { + if (cmn->part == PART_CMN600 && has_acpi_companion(cmn->dev)) { rootnode = arm_cmn600_acpi_probe(pdev, cmn); } else { rootnode = 0; cmn->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(cmn->base)) return PTR_ERR(cmn->base); - if (cmn->model == CMN600) + if (cmn->part == PART_CMN600) rootnode = arm_cmn600_of_probe(pdev->dev.of_node); } if (rootnode < 0) @@ -2335,10 +2389,10 @@ static int arm_cmn_remove(struct platform_device *pdev) #ifdef CONFIG_OF static const struct of_device_id arm_cmn_of_match[] = { - { .compatible = "arm,cmn-600", .data = (void *)CMN600 }, - { .compatible = "arm,cmn-650", .data = (void *)CMN650 }, - { .compatible = "arm,cmn-700", .data = (void *)CMN700 }, - { .compatible = "arm,ci-700", .data = (void *)CI700 }, + { .compatible = "arm,cmn-600", .data = (void *)PART_CMN600 }, + { .compatible = "arm,cmn-650" }, + { .compatible = "arm,cmn-700" }, + { .compatible = "arm,ci-700" }, {} }; MODULE_DEVICE_TABLE(of, arm_cmn_of_match); @@ -2346,9 +2400,9 @@ MODULE_DEVICE_TABLE(of, arm_cmn_of_match); #ifdef CONFIG_ACPI static const struct acpi_device_id arm_cmn_acpi_match[] = { - { "ARMHC600", CMN600 }, - { "ARMHC650", CMN650 }, - { "ARMHC700", CMN700 }, + { "ARMHC600", PART_CMN600 }, + { "ARMHC650" }, + { "ARMHC700" }, {} }; MODULE_DEVICE_TABLE(acpi, arm_cmn_acpi_match); diff --git a/drivers/perf/arm_cspmu/Kconfig b/drivers/perf/arm_cspmu/Kconfig index 0b316fe69a45..25d25ded0983 100644 --- a/drivers/perf/arm_cspmu/Kconfig +++ b/drivers/perf/arm_cspmu/Kconfig @@ -4,8 +4,7 @@ config ARM_CORESIGHT_PMU_ARCH_SYSTEM_PMU tristate "ARM Coresight Architecture PMU" - depends on ARM64 && ACPI - depends on ACPI_APMT || COMPILE_TEST + depends on ARM64 || COMPILE_TEST help Provides support for performance monitoring unit (PMU) devices based on ARM CoreSight PMU architecture. Note that this PMU diff --git a/drivers/perf/arm_cspmu/arm_cspmu.c b/drivers/perf/arm_cspmu/arm_cspmu.c index a3f1c410b417..e2b7827c4563 100644 --- a/drivers/perf/arm_cspmu/arm_cspmu.c +++ b/drivers/perf/arm_cspmu/arm_cspmu.c @@ -28,7 +28,6 @@ #include #include #include -#include #include "arm_cspmu.h" #include "nvidia_cspmu.h" @@ -101,10 +100,6 @@ #define ARM_CSPMU_ACTIVE_CPU_MASK 0x0 #define ARM_CSPMU_ASSOCIATED_CPU_MASK 0x1 -/* Check if field f in flags is set with value v */ -#define CHECK_APMT_FLAG(flags, f, v) \ - ((flags & (ACPI_APMT_FLAGS_ ## f)) == (ACPI_APMT_FLAGS_ ## f ## _ ## v)) - /* Check and use default if implementer doesn't provide attribute callback */ #define CHECK_DEFAULT_IMPL_OPS(ops, callback) \ do { \ @@ -122,6 +117,11 @@ static unsigned long arm_cspmu_cpuhp_state; +static struct acpi_apmt_node *arm_cspmu_apmt_node(struct device *dev) +{ + return *(struct acpi_apmt_node **)dev_get_platdata(dev); +} + /* * In CoreSight PMU architecture, all of the MMIO registers are 32-bit except * counter register. The counter register can be implemented as 32-bit or 64-bit @@ -156,12 +156,6 @@ static u64 read_reg64_hilohi(const void __iomem *addr, u32 max_poll_count) return val; } -/* Check if PMU supports 64-bit single copy atomic. */ -static inline bool supports_64bit_atomics(const struct arm_cspmu *cspmu) -{ - return CHECK_APMT_FLAG(cspmu->apmt_node->flags, ATOMIC, SUPP); -} - /* Check if cycle counter is supported. */ static inline bool supports_cycle_counter(const struct arm_cspmu *cspmu) { @@ -189,10 +183,10 @@ static inline bool use_64b_counter_reg(const struct arm_cspmu *cspmu) ssize_t arm_cspmu_sysfs_event_show(struct device *dev, struct device_attribute *attr, char *buf) { - struct dev_ext_attribute *eattr = - container_of(attr, struct dev_ext_attribute, attr); - return sysfs_emit(buf, "event=0x%llx\n", - (unsigned long long)eattr->var); + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, typeof(*pmu_attr), attr); + return sysfs_emit(buf, "event=0x%llx\n", pmu_attr->id); } EXPORT_SYMBOL_GPL(arm_cspmu_sysfs_event_show); @@ -320,7 +314,7 @@ static const char *arm_cspmu_get_name(const struct arm_cspmu *cspmu) static atomic_t pmu_idx[ACPI_APMT_NODE_TYPE_COUNT] = { 0 }; dev = cspmu->dev; - apmt_node = cspmu->apmt_node; + apmt_node = arm_cspmu_apmt_node(dev); pmu_type = apmt_node->type; if (pmu_type >= ACPI_APMT_NODE_TYPE_COUNT) { @@ -397,8 +391,8 @@ static const struct impl_match impl_match[] = { static int arm_cspmu_init_impl_ops(struct arm_cspmu *cspmu) { int ret; - struct acpi_apmt_node *apmt_node = cspmu->apmt_node; struct arm_cspmu_impl_ops *impl_ops = &cspmu->impl.ops; + struct acpi_apmt_node *apmt_node = arm_cspmu_apmt_node(cspmu->dev); const struct impl_match *match = impl_match; /* @@ -720,7 +714,7 @@ static u64 arm_cspmu_read_counter(struct perf_event *event) offset = counter_offset(sizeof(u64), event->hw.idx); counter_addr = cspmu->base1 + offset; - return supports_64bit_atomics(cspmu) ? + return cspmu->has_atomic_dword ? readq(counter_addr) : read_reg64_hilohi(counter_addr, HILOHI_MAX_POLL); } @@ -911,24 +905,18 @@ static struct arm_cspmu *arm_cspmu_alloc(struct platform_device *pdev) { struct acpi_apmt_node *apmt_node; struct arm_cspmu *cspmu; - struct device *dev; - - dev = &pdev->dev; - apmt_node = *(struct acpi_apmt_node **)dev_get_platdata(dev); - if (!apmt_node) { - dev_err(dev, "failed to get APMT node\n"); - return NULL; - } + struct device *dev = &pdev->dev; cspmu = devm_kzalloc(dev, sizeof(*cspmu), GFP_KERNEL); if (!cspmu) return NULL; cspmu->dev = dev; - cspmu->apmt_node = apmt_node; - platform_set_drvdata(pdev, cspmu); + apmt_node = arm_cspmu_apmt_node(dev); + cspmu->has_atomic_dword = apmt_node->flags & ACPI_APMT_FLAGS_ATOMIC; + return cspmu; } @@ -936,11 +924,9 @@ static int arm_cspmu_init_mmio(struct arm_cspmu *cspmu) { struct device *dev; struct platform_device *pdev; - struct acpi_apmt_node *apmt_node; dev = cspmu->dev; pdev = to_platform_device(dev); - apmt_node = cspmu->apmt_node; /* Base address for page 0. */ cspmu->base0 = devm_platform_ioremap_resource(pdev, 0); @@ -951,7 +937,7 @@ static int arm_cspmu_init_mmio(struct arm_cspmu *cspmu) /* Base address for page 1 if supported. Otherwise point to page 0. */ cspmu->base1 = cspmu->base0; - if (CHECK_APMT_FLAG(apmt_node->flags, DUAL_PAGE, SUPP)) { + if (platform_get_resource(pdev, IORESOURCE_MEM, 1)) { cspmu->base1 = devm_platform_ioremap_resource(pdev, 1); if (IS_ERR(cspmu->base1)) { dev_err(dev, "ioremap failed for page-1 resource\n"); @@ -1048,19 +1034,14 @@ static int arm_cspmu_request_irq(struct arm_cspmu *cspmu) int irq, ret; struct device *dev; struct platform_device *pdev; - struct acpi_apmt_node *apmt_node; dev = cspmu->dev; pdev = to_platform_device(dev); - apmt_node = cspmu->apmt_node; /* Skip IRQ request if the PMU does not support overflow interrupt. */ - if (apmt_node->ovflw_irq == 0) - return 0; - - irq = platform_get_irq(pdev, 0); + irq = platform_get_irq_optional(pdev, 0); if (irq < 0) - return irq; + return irq == -ENXIO ? 0 : irq; ret = devm_request_irq(dev, irq, arm_cspmu_handle_irq, IRQF_NOBALANCING | IRQF_NO_THREAD, dev_name(dev), @@ -1075,6 +1056,9 @@ static int arm_cspmu_request_irq(struct arm_cspmu *cspmu) return 0; } +#if defined(CONFIG_ACPI) && defined(CONFIG_ARM64) +#include + static inline int arm_cspmu_find_cpu_container(int cpu, u32 container_uid) { u32 acpi_uid; @@ -1099,15 +1083,13 @@ static inline int arm_cspmu_find_cpu_container(int cpu, u32 container_uid) return -ENODEV; } -static int arm_cspmu_get_cpus(struct arm_cspmu *cspmu) +static int arm_cspmu_acpi_get_cpus(struct arm_cspmu *cspmu) { - struct device *dev; struct acpi_apmt_node *apmt_node; int affinity_flag; int cpu; - dev = cspmu->pmu.dev; - apmt_node = cspmu->apmt_node; + apmt_node = arm_cspmu_apmt_node(cspmu->dev); affinity_flag = apmt_node->flags & ACPI_APMT_FLAGS_AFFINITY; if (affinity_flag == ACPI_APMT_FLAGS_AFFINITY_PROC) { @@ -1129,12 +1111,23 @@ static int arm_cspmu_get_cpus(struct arm_cspmu *cspmu) } if (cpumask_empty(&cspmu->associated_cpus)) { - dev_dbg(dev, "No cpu associated with the PMU\n"); + dev_dbg(cspmu->dev, "No cpu associated with the PMU\n"); return -ENODEV; } return 0; } +#else +static int arm_cspmu_acpi_get_cpus(struct arm_cspmu *cspmu) +{ + return -ENODEV; +} +#endif + +static int arm_cspmu_get_cpus(struct arm_cspmu *cspmu) +{ + return arm_cspmu_acpi_get_cpus(cspmu); +} static int arm_cspmu_register_pmu(struct arm_cspmu *cspmu) { @@ -1220,6 +1213,12 @@ static int arm_cspmu_device_remove(struct platform_device *pdev) return 0; } +static const struct platform_device_id arm_cspmu_id[] = { + {DRVNAME, 0}, + { }, +}; +MODULE_DEVICE_TABLE(platform, arm_cspmu_id); + static struct platform_driver arm_cspmu_driver = { .driver = { .name = DRVNAME, @@ -1227,12 +1226,14 @@ static struct platform_driver arm_cspmu_driver = { }, .probe = arm_cspmu_device_probe, .remove = arm_cspmu_device_remove, + .id_table = arm_cspmu_id, }; static void arm_cspmu_set_active_cpu(int cpu, struct arm_cspmu *cspmu) { cpumask_set_cpu(cpu, &cspmu->active_cpu); - WARN_ON(irq_set_affinity(cspmu->irq, &cspmu->active_cpu)); + if (cspmu->irq) + WARN_ON(irq_set_affinity(cspmu->irq, &cspmu->active_cpu)); } static int arm_cspmu_cpu_online(unsigned int cpu, struct hlist_node *node) diff --git a/drivers/perf/arm_cspmu/arm_cspmu.h b/drivers/perf/arm_cspmu/arm_cspmu.h index 51323b175a4a..83df53d1c132 100644 --- a/drivers/perf/arm_cspmu/arm_cspmu.h +++ b/drivers/perf/arm_cspmu/arm_cspmu.h @@ -8,7 +8,6 @@ #ifndef __ARM_CSPMU_H__ #define __ARM_CSPMU_H__ -#include #include #include #include @@ -118,16 +117,16 @@ struct arm_cspmu_impl { struct arm_cspmu { struct pmu pmu; struct device *dev; - struct acpi_apmt_node *apmt_node; const char *name; const char *identifier; void __iomem *base0; void __iomem *base1; - int irq; cpumask_t associated_cpus; cpumask_t active_cpu; struct hlist_node cpuhp_node; + int irq; + bool has_atomic_dword; u32 pmcfgr; u32 num_logical_ctrs; u32 num_set_clr_reg; diff --git a/drivers/perf/arm_dmc620_pmu.c b/drivers/perf/arm_dmc620_pmu.c index 5de06f9a4dd3..9d0f01c4455a 100644 --- a/drivers/perf/arm_dmc620_pmu.c +++ b/drivers/perf/arm_dmc620_pmu.c @@ -227,9 +227,31 @@ static const struct attribute_group dmc620_pmu_format_attr_group = { .attrs = dmc620_pmu_formats_attrs, }; +static ssize_t dmc620_pmu_cpumask_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct dmc620_pmu *dmc620_pmu = to_dmc620_pmu(dev_get_drvdata(dev)); + + return cpumap_print_to_pagebuf(true, buf, + cpumask_of(dmc620_pmu->irq->cpu)); +} + +static struct device_attribute dmc620_pmu_cpumask_attr = + __ATTR(cpumask, 0444, dmc620_pmu_cpumask_show, NULL); + +static struct attribute *dmc620_pmu_cpumask_attrs[] = { + &dmc620_pmu_cpumask_attr.attr, + NULL, +}; + +static const struct attribute_group dmc620_pmu_cpumask_attr_group = { + .attrs = dmc620_pmu_cpumask_attrs, +}; + static const struct attribute_group *dmc620_pmu_attr_groups[] = { &dmc620_pmu_events_attr_group, &dmc620_pmu_format_attr_group, + &dmc620_pmu_cpumask_attr_group, NULL, }; diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c index 15bd1e34a88e..277e29fbd504 100644 --- a/drivers/perf/arm_pmu.c +++ b/drivers/perf/arm_pmu.c @@ -109,6 +109,8 @@ static inline u64 arm_pmu_event_max_period(struct perf_event *event) { if (event->hw.flags & ARMPMU_EVT_64BIT) return GENMASK_ULL(63, 0); + else if (event->hw.flags & ARMPMU_EVT_63BIT) + return GENMASK_ULL(62, 0); else if (event->hw.flags & ARMPMU_EVT_47BIT) return GENMASK_ULL(46, 0); else diff --git a/drivers/perf/fsl_imx9_ddr_perf.c b/drivers/perf/fsl_imx9_ddr_perf.c new file mode 100644 index 000000000000..71d5b07e3aff --- /dev/null +++ b/drivers/perf/fsl_imx9_ddr_perf.c @@ -0,0 +1,711 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright 2023 NXP + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* Performance monitor configuration */ +#define PMCFG1 0x00 +#define PMCFG1_RD_TRANS_FILT_EN BIT(31) +#define PMCFG1_WR_TRANS_FILT_EN BIT(30) +#define PMCFG1_RD_BT_FILT_EN BIT(29) +#define PMCFG1_ID_MASK GENMASK(17, 0) + +#define PMCFG2 0x04 +#define PMCFG2_ID GENMASK(17, 0) + +/* Global control register affects all counters and takes priority over local control registers */ +#define PMGC0 0x40 +/* Global control register bits */ +#define PMGC0_FAC BIT(31) +#define PMGC0_PMIE BIT(30) +#define PMGC0_FCECE BIT(29) + +/* + * 64bit counter0 exclusively dedicated to counting cycles + * 32bit counters monitor counter-specific events in addition to counting reference events + */ +#define PMLCA(n) (0x40 + 0x10 + (0x10 * n)) +#define PMLCB(n) (0x40 + 0x14 + (0x10 * n)) +#define PMC(n) (0x40 + 0x18 + (0x10 * n)) +/* Local control register bits */ +#define PMLCA_FC BIT(31) +#define PMLCA_CE BIT(26) +#define PMLCA_EVENT GENMASK(22, 16) + +#define NUM_COUNTERS 11 +#define CYCLES_COUNTER 0 + +#define to_ddr_pmu(p) container_of(p, struct ddr_pmu, pmu) + +#define DDR_PERF_DEV_NAME "imx9_ddr" +#define DDR_CPUHP_CB_NAME DDR_PERF_DEV_NAME "_perf_pmu" + +static DEFINE_IDA(ddr_ida); + +struct imx_ddr_devtype_data { + const char *identifier; /* system PMU identifier for userspace */ +}; + +struct ddr_pmu { + struct pmu pmu; + void __iomem *base; + unsigned int cpu; + struct hlist_node node; + struct device *dev; + struct perf_event *events[NUM_COUNTERS]; + int active_events; + enum cpuhp_state cpuhp_state; + const struct imx_ddr_devtype_data *devtype_data; + int irq; + int id; +}; + +static const struct imx_ddr_devtype_data imx93_devtype_data = { + .identifier = "imx93", +}; + +static const struct of_device_id imx_ddr_pmu_dt_ids[] = { + {.compatible = "fsl,imx93-ddr-pmu", .data = &imx93_devtype_data}, + { /* sentinel */ } +}; +MODULE_DEVICE_TABLE(of, imx_ddr_pmu_dt_ids); + +static ssize_t ddr_perf_identifier_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct ddr_pmu *pmu = dev_get_drvdata(dev); + + return sysfs_emit(page, "%s\n", pmu->devtype_data->identifier); +} + +static struct device_attribute ddr_perf_identifier_attr = + __ATTR(identifier, 0444, ddr_perf_identifier_show, NULL); + +static struct attribute *ddr_perf_identifier_attrs[] = { + &ddr_perf_identifier_attr.attr, + NULL, +}; + +static struct attribute_group ddr_perf_identifier_attr_group = { + .attrs = ddr_perf_identifier_attrs, +}; + +static ssize_t ddr_perf_cpumask_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct ddr_pmu *pmu = dev_get_drvdata(dev); + + return cpumap_print_to_pagebuf(true, buf, cpumask_of(pmu->cpu)); +} + +static struct device_attribute ddr_perf_cpumask_attr = + __ATTR(cpumask, 0444, ddr_perf_cpumask_show, NULL); + +static struct attribute *ddr_perf_cpumask_attrs[] = { + &ddr_perf_cpumask_attr.attr, + NULL, +}; + +static const struct attribute_group ddr_perf_cpumask_attr_group = { + .attrs = ddr_perf_cpumask_attrs, +}; + +static ssize_t ddr_pmu_event_show(struct device *dev, + struct device_attribute *attr, char *page) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr); + return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id); +} + +#define IMX9_DDR_PMU_EVENT_ATTR(_name, _id) \ + (&((struct perf_pmu_events_attr[]) { \ + { .attr = __ATTR(_name, 0444, ddr_pmu_event_show, NULL),\ + .id = _id, } \ + })[0].attr.attr) + +static struct attribute *ddr_perf_events_attrs[] = { + /* counter0 cycles event */ + IMX9_DDR_PMU_EVENT_ATTR(cycles, 0), + + /* reference events for all normal counters, need assert DEBUG19[21] bit */ + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ddrc1_rmw_for_ecc, 12), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_rreorder, 13), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_wreorder, 14), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_0, 15), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_1, 16), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_2, 17), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_3, 18), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_4, 19), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_5, 22), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_6, 23), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_7, 24), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_8, 25), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_9, 26), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_10, 27), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_11, 28), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_12, 31), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_13, 59), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_15, 61), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_pm_29, 63), + + /* counter1 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_riq_0, 64), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_riq_1, 65), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_riq_2, 66), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_riq_3, 67), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_riq_4, 68), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_riq_5, 69), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_riq_6, 70), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_riq_7, 71), + + /* counter2 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_wiq_0, 64), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_wiq_1, 65), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_wiq_2, 66), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_wiq_3, 67), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_wiq_4, 68), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_wiq_5, 69), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_wiq_6, 70), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_ld_wiq_7, 71), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_empty, 72), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pm_rd_trans_filt, 73), + + /* counter3 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_collision_0, 64), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_collision_1, 65), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_collision_2, 66), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_collision_3, 67), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_collision_4, 68), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_collision_5, 69), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_collision_6, 70), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_collision_7, 71), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_full, 72), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pm_wr_trans_filt, 73), + + /* counter4 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_open_0, 64), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_open_1, 65), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_open_2, 66), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_open_3, 67), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_open_4, 68), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_open_5, 69), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_open_6, 70), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_row_open_7, 71), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_ld_rdq2_rmw, 72), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pm_rd_beat_filt, 73), + + /* counter5 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_start_0, 64), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_start_1, 65), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_start_2, 66), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_start_3, 67), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_start_4, 68), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_start_5, 69), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_start_6, 70), + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_start_7, 71), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_ld_rdq1, 72), + + /* counter6 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(ddrc_qx_valid_end_0, 64), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_ld_rdq2, 72), + + /* counter7 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_1_2_full, 64), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_ld_wrq0, 65), + + /* counter8 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_bias_switched, 64), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_1_4_full, 65), + + /* counter9 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_ld_wrq1, 65), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_3_4_full, 66), + + /* counter10 specific events */ + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_misc_mrk, 65), + IMX9_DDR_PMU_EVENT_ATTR(eddrtq_pmon_ld_rdq0, 66), + NULL, +}; + +static const struct attribute_group ddr_perf_events_attr_group = { + .name = "events", + .attrs = ddr_perf_events_attrs, +}; + +PMU_FORMAT_ATTR(event, "config:0-7"); +PMU_FORMAT_ATTR(counter, "config:8-15"); +PMU_FORMAT_ATTR(axi_id, "config1:0-17"); +PMU_FORMAT_ATTR(axi_mask, "config2:0-17"); + +static struct attribute *ddr_perf_format_attrs[] = { + &format_attr_event.attr, + &format_attr_counter.attr, + &format_attr_axi_id.attr, + &format_attr_axi_mask.attr, + NULL, +}; + +static const struct attribute_group ddr_perf_format_attr_group = { + .name = "format", + .attrs = ddr_perf_format_attrs, +}; + +static const struct attribute_group *attr_groups[] = { + &ddr_perf_identifier_attr_group, + &ddr_perf_cpumask_attr_group, + &ddr_perf_events_attr_group, + &ddr_perf_format_attr_group, + NULL, +}; + +static void ddr_perf_clear_counter(struct ddr_pmu *pmu, int counter) +{ + if (counter == CYCLES_COUNTER) { + writel(0, pmu->base + PMC(counter) + 0x4); + writel(0, pmu->base + PMC(counter)); + } else { + writel(0, pmu->base + PMC(counter)); + } +} + +static u64 ddr_perf_read_counter(struct ddr_pmu *pmu, int counter) +{ + u32 val_lower, val_upper; + u64 val; + + if (counter != CYCLES_COUNTER) { + val = readl_relaxed(pmu->base + PMC(counter)); + goto out; + } + + /* special handling for reading 64bit cycle counter */ + do { + val_upper = readl_relaxed(pmu->base + PMC(counter) + 0x4); + val_lower = readl_relaxed(pmu->base + PMC(counter)); + } while (val_upper != readl_relaxed(pmu->base + PMC(counter) + 0x4)); + + val = val_upper; + val = (val << 32); + val |= val_lower; +out: + return val; +} + +static void ddr_perf_counter_global_config(struct ddr_pmu *pmu, bool enable) +{ + u32 ctrl; + + ctrl = readl_relaxed(pmu->base + PMGC0); + + if (enable) { + /* + * The performance monitor must be reset before event counting + * sequences. The performance monitor can be reset by first freezing + * one or more counters and then clearing the freeze condition to + * allow the counters to count according to the settings in the + * performance monitor registers. Counters can be frozen individually + * by setting PMLCAn[FC] bits, or simultaneously by setting PMGC0[FAC]. + * Simply clearing these freeze bits will then allow the performance + * monitor to begin counting based on the register settings. + */ + ctrl |= PMGC0_FAC; + writel(ctrl, pmu->base + PMGC0); + + /* + * Freeze all counters disabled, interrupt enabled, and freeze + * counters on condition enabled. + */ + ctrl &= ~PMGC0_FAC; + ctrl |= PMGC0_PMIE | PMGC0_FCECE; + writel(ctrl, pmu->base + PMGC0); + } else { + ctrl |= PMGC0_FAC; + ctrl &= ~(PMGC0_PMIE | PMGC0_FCECE); + writel(ctrl, pmu->base + PMGC0); + } +} + +static void ddr_perf_counter_local_config(struct ddr_pmu *pmu, int config, + int counter, bool enable) +{ + u32 ctrl_a; + + ctrl_a = readl_relaxed(pmu->base + PMLCA(counter)); + + if (enable) { + ctrl_a |= PMLCA_FC; + writel(ctrl_a, pmu->base + PMLCA(counter)); + + ddr_perf_clear_counter(pmu, counter); + + /* Freeze counter disabled, condition enabled, and program event.*/ + ctrl_a &= ~PMLCA_FC; + ctrl_a |= PMLCA_CE; + ctrl_a &= ~FIELD_PREP(PMLCA_EVENT, 0x7F); + ctrl_a |= FIELD_PREP(PMLCA_EVENT, (config & 0x000000FF)); + writel(ctrl_a, pmu->base + PMLCA(counter)); + } else { + /* Freeze counter. */ + ctrl_a |= PMLCA_FC; + writel(ctrl_a, pmu->base + PMLCA(counter)); + } +} + +static void ddr_perf_monitor_config(struct ddr_pmu *pmu, int cfg, int cfg1, int cfg2) +{ + u32 pmcfg1, pmcfg2; + int event, counter; + + event = cfg & 0x000000FF; + counter = (cfg & 0x0000FF00) >> 8; + + pmcfg1 = readl_relaxed(pmu->base + PMCFG1); + + if (counter == 2 && event == 73) + pmcfg1 |= PMCFG1_RD_TRANS_FILT_EN; + else if (counter == 2 && event != 73) + pmcfg1 &= ~PMCFG1_RD_TRANS_FILT_EN; + + if (counter == 3 && event == 73) + pmcfg1 |= PMCFG1_WR_TRANS_FILT_EN; + else if (counter == 3 && event != 73) + pmcfg1 &= ~PMCFG1_WR_TRANS_FILT_EN; + + if (counter == 4 && event == 73) + pmcfg1 |= PMCFG1_RD_BT_FILT_EN; + else if (counter == 4 && event != 73) + pmcfg1 &= ~PMCFG1_RD_BT_FILT_EN; + + pmcfg1 &= ~FIELD_PREP(PMCFG1_ID_MASK, 0x3FFFF); + pmcfg1 |= FIELD_PREP(PMCFG1_ID_MASK, cfg2); + writel(pmcfg1, pmu->base + PMCFG1); + + pmcfg2 = readl_relaxed(pmu->base + PMCFG2); + pmcfg2 &= ~FIELD_PREP(PMCFG2_ID, 0x3FFFF); + pmcfg2 |= FIELD_PREP(PMCFG2_ID, cfg1); + writel(pmcfg2, pmu->base + PMCFG2); +} + +static void ddr_perf_event_update(struct perf_event *event) +{ + struct ddr_pmu *pmu = to_ddr_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + int counter = hwc->idx; + u64 new_raw_count; + + new_raw_count = ddr_perf_read_counter(pmu, counter); + local64_add(new_raw_count, &event->count); + + /* clear counter's value every time */ + ddr_perf_clear_counter(pmu, counter); +} + +static int ddr_perf_event_init(struct perf_event *event) +{ + struct ddr_pmu *pmu = to_ddr_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + struct perf_event *sibling; + + if (event->attr.type != event->pmu->type) + return -ENOENT; + + if (is_sampling_event(event) || event->attach_state & PERF_ATTACH_TASK) + return -EOPNOTSUPP; + + if (event->cpu < 0) { + dev_warn(pmu->dev, "Can't provide per-task data!\n"); + return -EOPNOTSUPP; + } + + /* + * We must NOT create groups containing mixed PMUs, although software + * events are acceptable (for example to create a CCN group + * periodically read when a hrtimer aka cpu-clock leader triggers). + */ + if (event->group_leader->pmu != event->pmu && + !is_software_event(event->group_leader)) + return -EINVAL; + + for_each_sibling_event(sibling, event->group_leader) { + if (sibling->pmu != event->pmu && + !is_software_event(sibling)) + return -EINVAL; + } + + event->cpu = pmu->cpu; + hwc->idx = -1; + + return 0; +} + +static void ddr_perf_event_start(struct perf_event *event, int flags) +{ + struct ddr_pmu *pmu = to_ddr_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + int counter = hwc->idx; + + local64_set(&hwc->prev_count, 0); + + ddr_perf_counter_local_config(pmu, event->attr.config, counter, true); + hwc->state = 0; +} + +static int ddr_perf_event_add(struct perf_event *event, int flags) +{ + struct ddr_pmu *pmu = to_ddr_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + int cfg = event->attr.config; + int cfg1 = event->attr.config1; + int cfg2 = event->attr.config2; + int counter; + + counter = (cfg & 0x0000FF00) >> 8; + + pmu->events[counter] = event; + pmu->active_events++; + hwc->idx = counter; + hwc->state |= PERF_HES_STOPPED; + + if (flags & PERF_EF_START) + ddr_perf_event_start(event, flags); + + /* read trans, write trans, read beat */ + ddr_perf_monitor_config(pmu, cfg, cfg1, cfg2); + + return 0; +} + +static void ddr_perf_event_stop(struct perf_event *event, int flags) +{ + struct ddr_pmu *pmu = to_ddr_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + int counter = hwc->idx; + + ddr_perf_counter_local_config(pmu, event->attr.config, counter, false); + ddr_perf_event_update(event); + + hwc->state |= PERF_HES_STOPPED; +} + +static void ddr_perf_event_del(struct perf_event *event, int flags) +{ + struct ddr_pmu *pmu = to_ddr_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + + ddr_perf_event_stop(event, PERF_EF_UPDATE); + + pmu->active_events--; + hwc->idx = -1; +} + +static void ddr_perf_pmu_enable(struct pmu *pmu) +{ + struct ddr_pmu *ddr_pmu = to_ddr_pmu(pmu); + + ddr_perf_counter_global_config(ddr_pmu, true); +} + +static void ddr_perf_pmu_disable(struct pmu *pmu) +{ + struct ddr_pmu *ddr_pmu = to_ddr_pmu(pmu); + + ddr_perf_counter_global_config(ddr_pmu, false); +} + +static void ddr_perf_init(struct ddr_pmu *pmu, void __iomem *base, + struct device *dev) +{ + *pmu = (struct ddr_pmu) { + .pmu = (struct pmu) { + .module = THIS_MODULE, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE, + .task_ctx_nr = perf_invalid_context, + .attr_groups = attr_groups, + .event_init = ddr_perf_event_init, + .add = ddr_perf_event_add, + .del = ddr_perf_event_del, + .start = ddr_perf_event_start, + .stop = ddr_perf_event_stop, + .read = ddr_perf_event_update, + .pmu_enable = ddr_perf_pmu_enable, + .pmu_disable = ddr_perf_pmu_disable, + }, + .base = base, + .dev = dev, + }; +} + +static irqreturn_t ddr_perf_irq_handler(int irq, void *p) +{ + struct ddr_pmu *pmu = (struct ddr_pmu *)p; + struct perf_event *event; + int i; + + /* + * Counters can generate an interrupt on an overflow when msb of a + * counter changes from 0 to 1. For the interrupt to be signalled, + * below condition mush be satisfied: + * PMGC0[PMIE] = 1, PMGC0[FCECE] = 1, PMLCAn[CE] = 1 + * When an interrupt is signalled, PMGC0[FAC] is set by hardware and + * all of the registers are frozen. + * Software can clear the interrupt condition by resetting the performance + * monitor and clearing the most significant bit of the counter that + * generate the overflow. + */ + for (i = 0; i < NUM_COUNTERS; i++) { + if (!pmu->events[i]) + continue; + + event = pmu->events[i]; + + ddr_perf_event_update(event); + } + + ddr_perf_counter_global_config(pmu, true); + + return IRQ_HANDLED; +} + +static int ddr_perf_offline_cpu(unsigned int cpu, struct hlist_node *node) +{ + struct ddr_pmu *pmu = hlist_entry_safe(node, struct ddr_pmu, node); + int target; + + if (cpu != pmu->cpu) + return 0; + + target = cpumask_any_but(cpu_online_mask, cpu); + if (target >= nr_cpu_ids) + return 0; + + perf_pmu_migrate_context(&pmu->pmu, cpu, target); + pmu->cpu = target; + + WARN_ON(irq_set_affinity(pmu->irq, cpumask_of(pmu->cpu))); + + return 0; +} + +static int ddr_perf_probe(struct platform_device *pdev) +{ + struct ddr_pmu *pmu; + void __iomem *base; + int ret, irq; + char *name; + + base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(base)) + return PTR_ERR(base); + + pmu = devm_kzalloc(&pdev->dev, sizeof(*pmu), GFP_KERNEL); + if (!pmu) + return -ENOMEM; + + ddr_perf_init(pmu, base, &pdev->dev); + + pmu->devtype_data = of_device_get_match_data(&pdev->dev); + + platform_set_drvdata(pdev, pmu); + + pmu->id = ida_simple_get(&ddr_ida, 0, 0, GFP_KERNEL); + name = devm_kasprintf(&pdev->dev, GFP_KERNEL, DDR_PERF_DEV_NAME "%d", pmu->id); + if (!name) { + ret = -ENOMEM; + goto format_string_err; + } + + pmu->cpu = raw_smp_processor_id(); + ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DDR_CPUHP_CB_NAME, + NULL, ddr_perf_offline_cpu); + if (ret < 0) { + dev_err(&pdev->dev, "Failed to add callbacks for multi state\n"); + goto cpuhp_state_err; + } + pmu->cpuhp_state = ret; + + /* Register the pmu instance for cpu hotplug */ + ret = cpuhp_state_add_instance_nocalls(pmu->cpuhp_state, &pmu->node); + if (ret) { + dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); + goto cpuhp_instance_err; + } + + /* Request irq */ + irq = platform_get_irq(pdev, 0); + if (irq < 0) { + ret = irq; + goto ddr_perf_err; + } + + ret = devm_request_irq(&pdev->dev, irq, ddr_perf_irq_handler, + IRQF_NOBALANCING | IRQF_NO_THREAD, + DDR_CPUHP_CB_NAME, pmu); + if (ret < 0) { + dev_err(&pdev->dev, "Request irq failed: %d", ret); + goto ddr_perf_err; + } + + pmu->irq = irq; + ret = irq_set_affinity(pmu->irq, cpumask_of(pmu->cpu)); + if (ret) { + dev_err(pmu->dev, "Failed to set interrupt affinity\n"); + goto ddr_perf_err; + } + + ret = perf_pmu_register(&pmu->pmu, name, -1); + if (ret) + goto ddr_perf_err; + + return 0; + +ddr_perf_err: + cpuhp_state_remove_instance_nocalls(pmu->cpuhp_state, &pmu->node); +cpuhp_instance_err: + cpuhp_remove_multi_state(pmu->cpuhp_state); +cpuhp_state_err: +format_string_err: + ida_simple_remove(&ddr_ida, pmu->id); + dev_warn(&pdev->dev, "i.MX9 DDR Perf PMU failed (%d), disabled\n", ret); + return ret; +} + +static int ddr_perf_remove(struct platform_device *pdev) +{ + struct ddr_pmu *pmu = platform_get_drvdata(pdev); + + cpuhp_state_remove_instance_nocalls(pmu->cpuhp_state, &pmu->node); + cpuhp_remove_multi_state(pmu->cpuhp_state); + + perf_pmu_unregister(&pmu->pmu); + + ida_simple_remove(&ddr_ida, pmu->id); + + return 0; +} + +static struct platform_driver imx_ddr_pmu_driver = { + .driver = { + .name = "imx9-ddr-pmu", + .of_match_table = imx_ddr_pmu_dt_ids, + .suppress_bind_attrs = true, + }, + .probe = ddr_perf_probe, + .remove = ddr_perf_remove, +}; +module_platform_driver(imx_ddr_pmu_driver); + +MODULE_AUTHOR("Xu Yang "); +MODULE_LICENSE("GPL v2"); +MODULE_DESCRIPTION("DDRC PerfMon for i.MX9 SoCs"); diff --git a/drivers/perf/hisilicon/Makefile b/drivers/perf/hisilicon/Makefile index 4d2c9abe3372..48dcc8381ea7 100644 --- a/drivers/perf/hisilicon/Makefile +++ b/drivers/perf/hisilicon/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_HISI_PMU) += hisi_uncore_pmu.o hisi_uncore_l3c_pmu.o \ hisi_uncore_hha_pmu.o hisi_uncore_ddrc_pmu.o hisi_uncore_sllc_pmu.o \ - hisi_uncore_pa_pmu.o hisi_uncore_cpa_pmu.o + hisi_uncore_pa_pmu.o hisi_uncore_cpa_pmu.o hisi_uncore_uc_pmu.o obj-$(CONFIG_HISI_PCIE_PMU) += hisi_pcie_pmu.o obj-$(CONFIG_HNS3_PMU) += hns3_pmu.o diff --git a/drivers/perf/hisilicon/hisi_pcie_pmu.c b/drivers/perf/hisilicon/hisi_pcie_pmu.c index 6fee0b6e163b..e10fc7cb9493 100644 --- a/drivers/perf/hisilicon/hisi_pcie_pmu.c +++ b/drivers/perf/hisilicon/hisi_pcie_pmu.c @@ -683,7 +683,7 @@ static int hisi_pcie_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node) pcie_pmu->on_cpu = -1; /* Choose a new CPU from all online cpus. */ - target = cpumask_first(cpu_online_mask); + target = cpumask_any_but(cpu_online_mask, cpu); if (target >= nr_cpu_ids) { pci_err(pcie_pmu->pdev, "There is no CPU to set\n"); return 0; diff --git a/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c b/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c index 71b6687d6696..d941e746b424 100644 --- a/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_pa_pmu.c @@ -22,9 +22,15 @@ #define PA_TT_CTRL 0x1c08 #define PA_TGTID_CTRL 0x1c14 #define PA_SRCID_CTRL 0x1c18 + +/* H32 PA interrupt registers */ #define PA_INT_MASK 0x1c70 #define PA_INT_STATUS 0x1c78 #define PA_INT_CLEAR 0x1c7c + +#define H60PA_INT_STATUS 0x1c70 +#define H60PA_INT_MASK 0x1c74 + #define PA_EVENT_TYPE0 0x1c80 #define PA_PMU_VERSION 0x1cf0 #define PA_EVENT_CNT0_L 0x1d00 @@ -46,6 +52,12 @@ HISI_PMU_EVENT_ATTR_EXTRACTOR(srcid_cmd, config1, 32, 22); HISI_PMU_EVENT_ATTR_EXTRACTOR(srcid_msk, config1, 43, 33); HISI_PMU_EVENT_ATTR_EXTRACTOR(tracetag_en, config1, 44, 44); +struct hisi_pa_pmu_int_regs { + u32 mask_offset; + u32 clear_offset; + u32 status_offset; +}; + static void hisi_pa_pmu_enable_tracetag(struct perf_event *event) { struct hisi_pmu *pa_pmu = to_hisi_pmu(event->pmu); @@ -219,40 +231,40 @@ static void hisi_pa_pmu_disable_counter(struct hisi_pmu *pa_pmu, static void hisi_pa_pmu_enable_counter_int(struct hisi_pmu *pa_pmu, struct hw_perf_event *hwc) { + struct hisi_pa_pmu_int_regs *regs = pa_pmu->dev_info->private; u32 val; /* Write 0 to enable interrupt */ - val = readl(pa_pmu->base + PA_INT_MASK); + val = readl(pa_pmu->base + regs->mask_offset); val &= ~(1 << hwc->idx); - writel(val, pa_pmu->base + PA_INT_MASK); + writel(val, pa_pmu->base + regs->mask_offset); } static void hisi_pa_pmu_disable_counter_int(struct hisi_pmu *pa_pmu, struct hw_perf_event *hwc) { + struct hisi_pa_pmu_int_regs *regs = pa_pmu->dev_info->private; u32 val; /* Write 1 to mask interrupt */ - val = readl(pa_pmu->base + PA_INT_MASK); + val = readl(pa_pmu->base + regs->mask_offset); val |= 1 << hwc->idx; - writel(val, pa_pmu->base + PA_INT_MASK); + writel(val, pa_pmu->base + regs->mask_offset); } static u32 hisi_pa_pmu_get_int_status(struct hisi_pmu *pa_pmu) { - return readl(pa_pmu->base + PA_INT_STATUS); + struct hisi_pa_pmu_int_regs *regs = pa_pmu->dev_info->private; + + return readl(pa_pmu->base + regs->status_offset); } static void hisi_pa_pmu_clear_int_status(struct hisi_pmu *pa_pmu, int idx) { - writel(1 << idx, pa_pmu->base + PA_INT_CLEAR); -} + struct hisi_pa_pmu_int_regs *regs = pa_pmu->dev_info->private; -static const struct acpi_device_id hisi_pa_pmu_acpi_match[] = { - { "HISI0273", }, - {} -}; -MODULE_DEVICE_TABLE(acpi, hisi_pa_pmu_acpi_match); + writel(1 << idx, pa_pmu->base + regs->clear_offset); +} static int hisi_pa_pmu_init_data(struct platform_device *pdev, struct hisi_pmu *pa_pmu) @@ -276,6 +288,10 @@ static int hisi_pa_pmu_init_data(struct platform_device *pdev, pa_pmu->ccl_id = -1; pa_pmu->sccl_id = -1; + pa_pmu->dev_info = device_get_match_data(&pdev->dev); + if (!pa_pmu->dev_info) + return -ENODEV; + pa_pmu->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(pa_pmu->base)) { dev_err(&pdev->dev, "ioremap failed for pa_pmu resource.\n"); @@ -314,6 +330,32 @@ static const struct attribute_group hisi_pa_pmu_v2_events_group = { .attrs = hisi_pa_pmu_v2_events_attr, }; +static struct attribute *hisi_pa_pmu_v3_events_attr[] = { + HISI_PMU_EVENT_ATTR(tx_req, 0x0), + HISI_PMU_EVENT_ATTR(tx_dat, 0x1), + HISI_PMU_EVENT_ATTR(tx_snp, 0x2), + HISI_PMU_EVENT_ATTR(rx_req, 0x7), + HISI_PMU_EVENT_ATTR(rx_dat, 0x8), + HISI_PMU_EVENT_ATTR(rx_snp, 0x9), + NULL +}; + +static const struct attribute_group hisi_pa_pmu_v3_events_group = { + .name = "events", + .attrs = hisi_pa_pmu_v3_events_attr, +}; + +static struct attribute *hisi_h60pa_pmu_events_attr[] = { + HISI_PMU_EVENT_ATTR(rx_flit, 0x50), + HISI_PMU_EVENT_ATTR(tx_flit, 0x65), + NULL +}; + +static const struct attribute_group hisi_h60pa_pmu_events_group = { + .name = "events", + .attrs = hisi_h60pa_pmu_events_attr, +}; + static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL); static struct attribute *hisi_pa_pmu_cpumask_attrs[] = { @@ -337,6 +379,12 @@ static const struct attribute_group hisi_pa_pmu_identifier_group = { .attrs = hisi_pa_pmu_identifier_attrs, }; +static struct hisi_pa_pmu_int_regs hisi_pa_pmu_regs = { + .mask_offset = PA_INT_MASK, + .clear_offset = PA_INT_CLEAR, + .status_offset = PA_INT_STATUS, +}; + static const struct attribute_group *hisi_pa_pmu_v2_attr_groups[] = { &hisi_pa_pmu_v2_format_group, &hisi_pa_pmu_v2_events_group, @@ -345,6 +393,46 @@ static const struct attribute_group *hisi_pa_pmu_v2_attr_groups[] = { NULL }; +static const struct hisi_pmu_dev_info hisi_h32pa_v2 = { + .name = "pa", + .attr_groups = hisi_pa_pmu_v2_attr_groups, + .private = &hisi_pa_pmu_regs, +}; + +static const struct attribute_group *hisi_pa_pmu_v3_attr_groups[] = { + &hisi_pa_pmu_v2_format_group, + &hisi_pa_pmu_v3_events_group, + &hisi_pa_pmu_cpumask_attr_group, + &hisi_pa_pmu_identifier_group, + NULL +}; + +static const struct hisi_pmu_dev_info hisi_h32pa_v3 = { + .name = "pa", + .attr_groups = hisi_pa_pmu_v3_attr_groups, + .private = &hisi_pa_pmu_regs, +}; + +static struct hisi_pa_pmu_int_regs hisi_h60pa_pmu_regs = { + .mask_offset = H60PA_INT_MASK, + .clear_offset = H60PA_INT_STATUS, /* Clear on write */ + .status_offset = H60PA_INT_STATUS, +}; + +static const struct attribute_group *hisi_h60pa_pmu_attr_groups[] = { + &hisi_pa_pmu_v2_format_group, + &hisi_h60pa_pmu_events_group, + &hisi_pa_pmu_cpumask_attr_group, + &hisi_pa_pmu_identifier_group, + NULL +}; + +static const struct hisi_pmu_dev_info hisi_h60pa = { + .name = "h60pa", + .attr_groups = hisi_h60pa_pmu_attr_groups, + .private = &hisi_h60pa_pmu_regs, +}; + static const struct hisi_uncore_ops hisi_uncore_pa_ops = { .write_evtype = hisi_pa_pmu_write_evtype, .get_event_idx = hisi_uncore_pmu_get_event_idx, @@ -375,7 +463,7 @@ static int hisi_pa_pmu_dev_probe(struct platform_device *pdev, if (ret) return ret; - pa_pmu->pmu_events.attr_groups = hisi_pa_pmu_v2_attr_groups; + pa_pmu->pmu_events.attr_groups = pa_pmu->dev_info->attr_groups; pa_pmu->num_counters = PA_NR_COUNTERS; pa_pmu->ops = &hisi_uncore_pa_ops; pa_pmu->check_event = 0xB0; @@ -400,8 +488,9 @@ static int hisi_pa_pmu_probe(struct platform_device *pdev) if (ret) return ret; - name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sicl%u_pa%u", - pa_pmu->sicl_id, pa_pmu->index_id); + name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sicl%d_%s%u", + pa_pmu->sicl_id, pa_pmu->dev_info->name, + pa_pmu->index_id); if (!name) return -ENOMEM; @@ -435,6 +524,14 @@ static int hisi_pa_pmu_remove(struct platform_device *pdev) return 0; } +static const struct acpi_device_id hisi_pa_pmu_acpi_match[] = { + { "HISI0273", (kernel_ulong_t)&hisi_h32pa_v2 }, + { "HISI0275", (kernel_ulong_t)&hisi_h32pa_v3 }, + { "HISI0274", (kernel_ulong_t)&hisi_h60pa }, + {} +}; +MODULE_DEVICE_TABLE(acpi, hisi_pa_pmu_acpi_match); + static struct platform_driver hisi_pa_pmu_driver = { .driver = { .name = "hisi_pa_pmu", diff --git a/drivers/perf/hisilicon/hisi_uncore_pmu.c b/drivers/perf/hisilicon/hisi_uncore_pmu.c index 2823f381930d..04031450d5fe 100644 --- a/drivers/perf/hisilicon/hisi_uncore_pmu.c +++ b/drivers/perf/hisilicon/hisi_uncore_pmu.c @@ -20,7 +20,6 @@ #include "hisi_uncore_pmu.h" -#define HISI_GET_EVENTID(ev) (ev->hw.config_base & 0xff) #define HISI_MAX_PERIOD(nr) (GENMASK_ULL((nr) - 1, 0)) /* @@ -226,6 +225,9 @@ int hisi_uncore_pmu_event_init(struct perf_event *event) hwc->idx = -1; hwc->config_base = event->attr.config; + if (hisi_pmu->ops->check_filter && hisi_pmu->ops->check_filter(event)) + return -EINVAL; + /* Enforce to use the same CPU for all events in this PMU */ event->cpu = hisi_pmu->on_cpu; diff --git a/drivers/perf/hisilicon/hisi_uncore_pmu.h b/drivers/perf/hisilicon/hisi_uncore_pmu.h index 07890a8e96ca..92402aa69d70 100644 --- a/drivers/perf/hisilicon/hisi_uncore_pmu.h +++ b/drivers/perf/hisilicon/hisi_uncore_pmu.h @@ -43,9 +43,15 @@ return FIELD_GET(GENMASK_ULL(hi, lo), event->attr.config); \ } +#define HISI_GET_EVENTID(ev) (ev->hw.config_base & 0xff) + +#define HISI_PMU_EVTYPE_BITS 8 +#define HISI_PMU_EVTYPE_SHIFT(idx) ((idx) % 4 * HISI_PMU_EVTYPE_BITS) + struct hisi_pmu; struct hisi_uncore_ops { + int (*check_filter)(struct perf_event *event); void (*write_evtype)(struct hisi_pmu *, int, u32); int (*get_event_idx)(struct perf_event *); u64 (*read_counter)(struct hisi_pmu *, struct hw_perf_event *); @@ -62,6 +68,13 @@ struct hisi_uncore_ops { void (*disable_filter)(struct perf_event *event); }; +/* Describes the HISI PMU chip features information */ +struct hisi_pmu_dev_info { + const char *name; + const struct attribute_group **attr_groups; + void *private; +}; + struct hisi_pmu_hwevents { struct perf_event *hw_events[HISI_MAX_COUNTERS]; DECLARE_BITMAP(used_mask, HISI_MAX_COUNTERS); @@ -72,6 +85,7 @@ struct hisi_pmu_hwevents { struct hisi_pmu { struct pmu pmu; const struct hisi_uncore_ops *ops; + const struct hisi_pmu_dev_info *dev_info; struct hisi_pmu_hwevents pmu_events; /* associated_cpus: All CPUs associated with the PMU */ cpumask_t associated_cpus; diff --git a/drivers/perf/hisilicon/hisi_uncore_uc_pmu.c b/drivers/perf/hisilicon/hisi_uncore_uc_pmu.c new file mode 100644 index 000000000000..63da05e5831c --- /dev/null +++ b/drivers/perf/hisilicon/hisi_uncore_uc_pmu.c @@ -0,0 +1,578 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * HiSilicon SoC UC (unified cache) uncore Hardware event counters support + * + * Copyright (C) 2023 HiSilicon Limited + * + * This code is based on the uncore PMUs like hisi_uncore_l3c_pmu. + */ +#include +#include +#include +#include +#include +#include + +#include "hisi_uncore_pmu.h" + +/* Dynamic CPU hotplug state used by UC PMU */ +static enum cpuhp_state hisi_uc_pmu_online; + +/* UC register definition */ +#define HISI_UC_INT_MASK_REG 0x0800 +#define HISI_UC_INT_STS_REG 0x0808 +#define HISI_UC_INT_CLEAR_REG 0x080c +#define HISI_UC_TRACETAG_CTRL_REG 0x1b2c +#define HISI_UC_TRACETAG_REQ_MSK GENMASK(9, 7) +#define HISI_UC_TRACETAG_MARK_EN BIT(0) +#define HISI_UC_TRACETAG_REQ_EN (HISI_UC_TRACETAG_MARK_EN | BIT(2)) +#define HISI_UC_TRACETAG_SRCID_EN BIT(3) +#define HISI_UC_SRCID_CTRL_REG 0x1b40 +#define HISI_UC_SRCID_MSK GENMASK(14, 1) +#define HISI_UC_EVENT_CTRL_REG 0x1c00 +#define HISI_UC_EVENT_TRACETAG_EN BIT(29) +#define HISI_UC_EVENT_URING_MSK GENMASK(28, 27) +#define HISI_UC_EVENT_GLB_EN BIT(26) +#define HISI_UC_VERSION_REG 0x1cf0 +#define HISI_UC_EVTYPE_REGn(n) (0x1d00 + (n) * 4) +#define HISI_UC_EVTYPE_MASK GENMASK(7, 0) +#define HISI_UC_CNTR_REGn(n) (0x1e00 + (n) * 8) + +#define HISI_UC_NR_COUNTERS 0x8 +#define HISI_UC_V2_NR_EVENTS 0xFF +#define HISI_UC_CNTR_REG_BITS 64 + +#define HISI_UC_RD_REQ_TRACETAG 0x4 +#define HISI_UC_URING_EVENT_MIN 0x47 +#define HISI_UC_URING_EVENT_MAX 0x59 + +HISI_PMU_EVENT_ATTR_EXTRACTOR(rd_req_en, config1, 0, 0); +HISI_PMU_EVENT_ATTR_EXTRACTOR(uring_channel, config1, 5, 4); +HISI_PMU_EVENT_ATTR_EXTRACTOR(srcid, config1, 19, 6); +HISI_PMU_EVENT_ATTR_EXTRACTOR(srcid_en, config1, 20, 20); + +static int hisi_uc_pmu_check_filter(struct perf_event *event) +{ + struct hisi_pmu *uc_pmu = to_hisi_pmu(event->pmu); + + if (hisi_get_srcid_en(event) && !hisi_get_rd_req_en(event)) { + dev_err(uc_pmu->dev, + "rcid_en depends on rd_req_en being enabled!\n"); + return -EINVAL; + } + + if (!hisi_get_uring_channel(event)) + return 0; + + if ((HISI_GET_EVENTID(event) < HISI_UC_URING_EVENT_MIN) || + (HISI_GET_EVENTID(event) > HISI_UC_URING_EVENT_MAX)) + dev_warn(uc_pmu->dev, + "Only events: [%#x ~ %#x] support channel filtering!", + HISI_UC_URING_EVENT_MIN, HISI_UC_URING_EVENT_MAX); + + return 0; +} + +static void hisi_uc_pmu_config_req_tracetag(struct perf_event *event) +{ + struct hisi_pmu *uc_pmu = to_hisi_pmu(event->pmu); + u32 val; + + if (!hisi_get_rd_req_en(event)) + return; + + val = readl(uc_pmu->base + HISI_UC_TRACETAG_CTRL_REG); + + /* The request-type has been configured */ + if (FIELD_GET(HISI_UC_TRACETAG_REQ_MSK, val) == HISI_UC_RD_REQ_TRACETAG) + return; + + /* Set request-type for tracetag, only read request is supported! */ + val &= ~HISI_UC_TRACETAG_REQ_MSK; + val |= FIELD_PREP(HISI_UC_TRACETAG_REQ_MSK, HISI_UC_RD_REQ_TRACETAG); + val |= HISI_UC_TRACETAG_REQ_EN; + writel(val, uc_pmu->base + HISI_UC_TRACETAG_CTRL_REG); +} + +static void hisi_uc_pmu_clear_req_tracetag(struct perf_event *event) +{ + struct hisi_pmu *uc_pmu = to_hisi_pmu(event->pmu); + u32 val; + + if (!hisi_get_rd_req_en(event)) + return; + + val = readl(uc_pmu->base + HISI_UC_TRACETAG_CTRL_REG); + + /* Do nothing, the request-type tracetag has been cleaned up */ + if (FIELD_GET(HISI_UC_TRACETAG_REQ_MSK, val) == 0) + return; + + /* Clear request-type */ + val &= ~HISI_UC_TRACETAG_REQ_MSK; + val &= ~HISI_UC_TRACETAG_REQ_EN; + writel(val, uc_pmu->base + HISI_UC_TRACETAG_CTRL_REG); +} + +static void hisi_uc_pmu_config_srcid_tracetag(struct perf_event *event) +{ + struct hisi_pmu *uc_pmu = to_hisi_pmu(event->pmu); + u32 val; + + if (!hisi_get_srcid_en(event)) + return; + + val = readl(uc_pmu->base + HISI_UC_TRACETAG_CTRL_REG); + + /* Do nothing, the source id has been configured */ + if (FIELD_GET(HISI_UC_TRACETAG_SRCID_EN, val)) + return; + + /* Enable source id tracetag */ + val |= HISI_UC_TRACETAG_SRCID_EN; + writel(val, uc_pmu->base + HISI_UC_TRACETAG_CTRL_REG); + + val = readl(uc_pmu->base + HISI_UC_SRCID_CTRL_REG); + val &= ~HISI_UC_SRCID_MSK; + val |= FIELD_PREP(HISI_UC_SRCID_MSK, hisi_get_srcid(event)); + writel(val, uc_pmu->base + HISI_UC_SRCID_CTRL_REG); + + /* Depend on request-type tracetag enabled */ + hisi_uc_pmu_config_req_tracetag(event); +} + +static void hisi_uc_pmu_clear_srcid_tracetag(struct perf_event *event) +{ + struct hisi_pmu *uc_pmu = to_hisi_pmu(event->pmu); + u32 val; + + if (!hisi_get_srcid_en(event)) + return; + + val = readl(uc_pmu->base + HISI_UC_TRACETAG_CTRL_REG); + + /* Do nothing, the source id has been cleaned up */ + if (FIELD_GET(HISI_UC_TRACETAG_SRCID_EN, val) == 0) + return; + + hisi_uc_pmu_clear_req_tracetag(event); + + /* Disable source id tracetag */ + val &= ~HISI_UC_TRACETAG_SRCID_EN; + writel(val, uc_pmu->base + HISI_UC_TRACETAG_CTRL_REG); + + val = readl(uc_pmu->base + HISI_UC_SRCID_CTRL_REG); + val &= ~HISI_UC_SRCID_MSK; + writel(val, uc_pmu->base + HISI_UC_SRCID_CTRL_REG); +} + +static void hisi_uc_pmu_config_uring_channel(struct perf_event *event) +{ + struct hisi_pmu *uc_pmu = to_hisi_pmu(event->pmu); + u32 uring_channel = hisi_get_uring_channel(event); + u32 val; + + /* Do nothing if not being set or is set explicitly to zero (default) */ + if (uring_channel == 0) + return; + + val = readl(uc_pmu->base + HISI_UC_EVENT_CTRL_REG); + + /* Do nothing, the uring_channel has been configured */ + if (uring_channel == FIELD_GET(HISI_UC_EVENT_URING_MSK, val)) + return; + + val &= ~HISI_UC_EVENT_URING_MSK; + val |= FIELD_PREP(HISI_UC_EVENT_URING_MSK, uring_channel); + writel(val, uc_pmu->base + HISI_UC_EVENT_CTRL_REG); +} + +static void hisi_uc_pmu_clear_uring_channel(struct perf_event *event) +{ + struct hisi_pmu *uc_pmu = to_hisi_pmu(event->pmu); + u32 val; + + /* Do nothing if not being set or is set explicitly to zero (default) */ + if (hisi_get_uring_channel(event) == 0) + return; + + val = readl(uc_pmu->base + HISI_UC_EVENT_CTRL_REG); + + /* Do nothing, the uring_channel has been cleaned up */ + if (FIELD_GET(HISI_UC_EVENT_URING_MSK, val) == 0) + return; + + val &= ~HISI_UC_EVENT_URING_MSK; + writel(val, uc_pmu->base + HISI_UC_EVENT_CTRL_REG); +} + +static void hisi_uc_pmu_enable_filter(struct perf_event *event) +{ + if (event->attr.config1 == 0) + return; + + hisi_uc_pmu_config_uring_channel(event); + hisi_uc_pmu_config_req_tracetag(event); + hisi_uc_pmu_config_srcid_tracetag(event); +} + +static void hisi_uc_pmu_disable_filter(struct perf_event *event) +{ + if (event->attr.config1 == 0) + return; + + hisi_uc_pmu_clear_srcid_tracetag(event); + hisi_uc_pmu_clear_req_tracetag(event); + hisi_uc_pmu_clear_uring_channel(event); +} + +static void hisi_uc_pmu_write_evtype(struct hisi_pmu *uc_pmu, int idx, u32 type) +{ + u32 val; + + /* + * Select the appropriate event select register. + * There are 2 32-bit event select registers for the + * 8 hardware counters, each event code is 8-bit wide. + */ + val = readl(uc_pmu->base + HISI_UC_EVTYPE_REGn(idx / 4)); + val &= ~(HISI_UC_EVTYPE_MASK << HISI_PMU_EVTYPE_SHIFT(idx)); + val |= (type << HISI_PMU_EVTYPE_SHIFT(idx)); + writel(val, uc_pmu->base + HISI_UC_EVTYPE_REGn(idx / 4)); +} + +static void hisi_uc_pmu_start_counters(struct hisi_pmu *uc_pmu) +{ + u32 val; + + val = readl(uc_pmu->base + HISI_UC_EVENT_CTRL_REG); + val |= HISI_UC_EVENT_GLB_EN; + writel(val, uc_pmu->base + HISI_UC_EVENT_CTRL_REG); +} + +static void hisi_uc_pmu_stop_counters(struct hisi_pmu *uc_pmu) +{ + u32 val; + + val = readl(uc_pmu->base + HISI_UC_EVENT_CTRL_REG); + val &= ~HISI_UC_EVENT_GLB_EN; + writel(val, uc_pmu->base + HISI_UC_EVENT_CTRL_REG); +} + +static void hisi_uc_pmu_enable_counter(struct hisi_pmu *uc_pmu, + struct hw_perf_event *hwc) +{ + u32 val; + + /* Enable counter index */ + val = readl(uc_pmu->base + HISI_UC_EVENT_CTRL_REG); + val |= (1 << hwc->idx); + writel(val, uc_pmu->base + HISI_UC_EVENT_CTRL_REG); +} + +static void hisi_uc_pmu_disable_counter(struct hisi_pmu *uc_pmu, + struct hw_perf_event *hwc) +{ + u32 val; + + /* Clear counter index */ + val = readl(uc_pmu->base + HISI_UC_EVENT_CTRL_REG); + val &= ~(1 << hwc->idx); + writel(val, uc_pmu->base + HISI_UC_EVENT_CTRL_REG); +} + +static u64 hisi_uc_pmu_read_counter(struct hisi_pmu *uc_pmu, + struct hw_perf_event *hwc) +{ + return readq(uc_pmu->base + HISI_UC_CNTR_REGn(hwc->idx)); +} + +static void hisi_uc_pmu_write_counter(struct hisi_pmu *uc_pmu, + struct hw_perf_event *hwc, u64 val) +{ + writeq(val, uc_pmu->base + HISI_UC_CNTR_REGn(hwc->idx)); +} + +static void hisi_uc_pmu_enable_counter_int(struct hisi_pmu *uc_pmu, + struct hw_perf_event *hwc) +{ + u32 val; + + val = readl(uc_pmu->base + HISI_UC_INT_MASK_REG); + val &= ~(1 << hwc->idx); + writel(val, uc_pmu->base + HISI_UC_INT_MASK_REG); +} + +static void hisi_uc_pmu_disable_counter_int(struct hisi_pmu *uc_pmu, + struct hw_perf_event *hwc) +{ + u32 val; + + val = readl(uc_pmu->base + HISI_UC_INT_MASK_REG); + val |= (1 << hwc->idx); + writel(val, uc_pmu->base + HISI_UC_INT_MASK_REG); +} + +static u32 hisi_uc_pmu_get_int_status(struct hisi_pmu *uc_pmu) +{ + return readl(uc_pmu->base + HISI_UC_INT_STS_REG); +} + +static void hisi_uc_pmu_clear_int_status(struct hisi_pmu *uc_pmu, int idx) +{ + writel(1 << idx, uc_pmu->base + HISI_UC_INT_CLEAR_REG); +} + +static int hisi_uc_pmu_init_data(struct platform_device *pdev, + struct hisi_pmu *uc_pmu) +{ + /* + * Use SCCL (Super CPU Cluster) ID and CCL (CPU Cluster) ID to + * identify the topology information of UC PMU devices in the chip. + * They have some CCLs per SCCL and then 4 UC PMU per CCL. + */ + if (device_property_read_u32(&pdev->dev, "hisilicon,scl-id", + &uc_pmu->sccl_id)) { + dev_err(&pdev->dev, "Can not read uc sccl-id!\n"); + return -EINVAL; + } + + if (device_property_read_u32(&pdev->dev, "hisilicon,ccl-id", + &uc_pmu->ccl_id)) { + dev_err(&pdev->dev, "Can not read uc ccl-id!\n"); + return -EINVAL; + } + + if (device_property_read_u32(&pdev->dev, "hisilicon,sub-id", + &uc_pmu->sub_id)) { + dev_err(&pdev->dev, "Can not read sub-id!\n"); + return -EINVAL; + } + + uc_pmu->base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(uc_pmu->base)) { + dev_err(&pdev->dev, "ioremap failed for uc_pmu resource\n"); + return PTR_ERR(uc_pmu->base); + } + + uc_pmu->identifier = readl(uc_pmu->base + HISI_UC_VERSION_REG); + + return 0; +} + +static struct attribute *hisi_uc_pmu_format_attr[] = { + HISI_PMU_FORMAT_ATTR(event, "config:0-7"), + HISI_PMU_FORMAT_ATTR(rd_req_en, "config1:0-0"), + HISI_PMU_FORMAT_ATTR(uring_channel, "config1:4-5"), + HISI_PMU_FORMAT_ATTR(srcid, "config1:6-19"), + HISI_PMU_FORMAT_ATTR(srcid_en, "config1:20-20"), + NULL +}; + +static const struct attribute_group hisi_uc_pmu_format_group = { + .name = "format", + .attrs = hisi_uc_pmu_format_attr, +}; + +static struct attribute *hisi_uc_pmu_events_attr[] = { + HISI_PMU_EVENT_ATTR(sq_time, 0x00), + HISI_PMU_EVENT_ATTR(pq_time, 0x01), + HISI_PMU_EVENT_ATTR(hbm_time, 0x02), + HISI_PMU_EVENT_ATTR(iq_comp_time_cring, 0x03), + HISI_PMU_EVENT_ATTR(iq_comp_time_uring, 0x05), + HISI_PMU_EVENT_ATTR(cpu_rd, 0x10), + HISI_PMU_EVENT_ATTR(cpu_rd64, 0x17), + HISI_PMU_EVENT_ATTR(cpu_rs64, 0x19), + HISI_PMU_EVENT_ATTR(cpu_mru, 0x1a), + HISI_PMU_EVENT_ATTR(cycles, 0x9c), + HISI_PMU_EVENT_ATTR(spipe_hit, 0xb3), + HISI_PMU_EVENT_ATTR(hpipe_hit, 0xdb), + HISI_PMU_EVENT_ATTR(cring_rxdat_cnt, 0xfa), + HISI_PMU_EVENT_ATTR(cring_txdat_cnt, 0xfb), + HISI_PMU_EVENT_ATTR(uring_rxdat_cnt, 0xfc), + HISI_PMU_EVENT_ATTR(uring_txdat_cnt, 0xfd), + NULL +}; + +static const struct attribute_group hisi_uc_pmu_events_group = { + .name = "events", + .attrs = hisi_uc_pmu_events_attr, +}; + +static DEVICE_ATTR(cpumask, 0444, hisi_cpumask_sysfs_show, NULL); + +static struct attribute *hisi_uc_pmu_cpumask_attrs[] = { + &dev_attr_cpumask.attr, + NULL, +}; + +static const struct attribute_group hisi_uc_pmu_cpumask_attr_group = { + .attrs = hisi_uc_pmu_cpumask_attrs, +}; + +static struct device_attribute hisi_uc_pmu_identifier_attr = + __ATTR(identifier, 0444, hisi_uncore_pmu_identifier_attr_show, NULL); + +static struct attribute *hisi_uc_pmu_identifier_attrs[] = { + &hisi_uc_pmu_identifier_attr.attr, + NULL +}; + +static const struct attribute_group hisi_uc_pmu_identifier_group = { + .attrs = hisi_uc_pmu_identifier_attrs, +}; + +static const struct attribute_group *hisi_uc_pmu_attr_groups[] = { + &hisi_uc_pmu_format_group, + &hisi_uc_pmu_events_group, + &hisi_uc_pmu_cpumask_attr_group, + &hisi_uc_pmu_identifier_group, + NULL +}; + +static const struct hisi_uncore_ops hisi_uncore_uc_pmu_ops = { + .check_filter = hisi_uc_pmu_check_filter, + .write_evtype = hisi_uc_pmu_write_evtype, + .get_event_idx = hisi_uncore_pmu_get_event_idx, + .start_counters = hisi_uc_pmu_start_counters, + .stop_counters = hisi_uc_pmu_stop_counters, + .enable_counter = hisi_uc_pmu_enable_counter, + .disable_counter = hisi_uc_pmu_disable_counter, + .enable_counter_int = hisi_uc_pmu_enable_counter_int, + .disable_counter_int = hisi_uc_pmu_disable_counter_int, + .write_counter = hisi_uc_pmu_write_counter, + .read_counter = hisi_uc_pmu_read_counter, + .get_int_status = hisi_uc_pmu_get_int_status, + .clear_int_status = hisi_uc_pmu_clear_int_status, + .enable_filter = hisi_uc_pmu_enable_filter, + .disable_filter = hisi_uc_pmu_disable_filter, +}; + +static int hisi_uc_pmu_dev_probe(struct platform_device *pdev, + struct hisi_pmu *uc_pmu) +{ + int ret; + + ret = hisi_uc_pmu_init_data(pdev, uc_pmu); + if (ret) + return ret; + + ret = hisi_uncore_pmu_init_irq(uc_pmu, pdev); + if (ret) + return ret; + + uc_pmu->pmu_events.attr_groups = hisi_uc_pmu_attr_groups; + uc_pmu->check_event = HISI_UC_EVTYPE_MASK; + uc_pmu->ops = &hisi_uncore_uc_pmu_ops; + uc_pmu->counter_bits = HISI_UC_CNTR_REG_BITS; + uc_pmu->num_counters = HISI_UC_NR_COUNTERS; + uc_pmu->dev = &pdev->dev; + uc_pmu->on_cpu = -1; + + return 0; +} + +static void hisi_uc_pmu_remove_cpuhp_instance(void *hotplug_node) +{ + cpuhp_state_remove_instance_nocalls(hisi_uc_pmu_online, hotplug_node); +} + +static void hisi_uc_pmu_unregister_pmu(void *pmu) +{ + perf_pmu_unregister(pmu); +} + +static int hisi_uc_pmu_probe(struct platform_device *pdev) +{ + struct hisi_pmu *uc_pmu; + char *name; + int ret; + + uc_pmu = devm_kzalloc(&pdev->dev, sizeof(*uc_pmu), GFP_KERNEL); + if (!uc_pmu) + return -ENOMEM; + + platform_set_drvdata(pdev, uc_pmu); + + ret = hisi_uc_pmu_dev_probe(pdev, uc_pmu); + if (ret) + return ret; + + name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "hisi_sccl%d_uc%d_%u", + uc_pmu->sccl_id, uc_pmu->ccl_id, uc_pmu->sub_id); + if (!name) + return -ENOMEM; + + ret = cpuhp_state_add_instance(hisi_uc_pmu_online, &uc_pmu->node); + if (ret) + return dev_err_probe(&pdev->dev, ret, "Error registering hotplug\n"); + + ret = devm_add_action_or_reset(&pdev->dev, + hisi_uc_pmu_remove_cpuhp_instance, + &uc_pmu->node); + if (ret) + return ret; + + hisi_pmu_init(uc_pmu, THIS_MODULE); + + ret = perf_pmu_register(&uc_pmu->pmu, name, -1); + if (ret) + return ret; + + return devm_add_action_or_reset(&pdev->dev, + hisi_uc_pmu_unregister_pmu, + &uc_pmu->pmu); +} + +static const struct acpi_device_id hisi_uc_pmu_acpi_match[] = { + { "HISI0291", }, + {} +}; +MODULE_DEVICE_TABLE(acpi, hisi_uc_pmu_acpi_match); + +static struct platform_driver hisi_uc_pmu_driver = { + .driver = { + .name = "hisi_uc_pmu", + .acpi_match_table = hisi_uc_pmu_acpi_match, + /* + * We have not worked out a safe bind/unbind process, + * Forcefully unbinding during sampling will lead to a + * kernel panic, so this is not supported yet. + */ + .suppress_bind_attrs = true, + }, + .probe = hisi_uc_pmu_probe, +}; + +static int __init hisi_uc_pmu_module_init(void) +{ + int ret; + + ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, + "perf/hisi/uc:online", + hisi_uncore_pmu_online_cpu, + hisi_uncore_pmu_offline_cpu); + if (ret < 0) { + pr_err("UC PMU: Error setup hotplug, ret = %d\n", ret); + return ret; + } + hisi_uc_pmu_online = ret; + + ret = platform_driver_register(&hisi_uc_pmu_driver); + if (ret) + cpuhp_remove_multi_state(hisi_uc_pmu_online); + + return ret; +} +module_init(hisi_uc_pmu_module_init); + +static void __exit hisi_uc_pmu_module_exit(void) +{ + platform_driver_unregister(&hisi_uc_pmu_driver); + cpuhp_remove_multi_state(hisi_uc_pmu_online); +} +module_exit(hisi_uc_pmu_module_exit); + +MODULE_DESCRIPTION("HiSilicon SoC UC uncore PMU driver"); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Junhao He "); diff --git a/drivers/perf/qcom_l2_pmu.c b/drivers/perf/qcom_l2_pmu.c index aaca6db7d8f6..3f9a98c17a89 100644 --- a/drivers/perf/qcom_l2_pmu.c +++ b/drivers/perf/qcom_l2_pmu.c @@ -857,7 +857,6 @@ static int l2_cache_pmu_probe_cluster(struct device *dev, void *data) return -ENOMEM; INIT_LIST_HEAD(&cluster->next); - list_add(&cluster->next, &l2cache_pmu->clusters); cluster->cluster_id = fw_cluster_id; irq = platform_get_irq(sdev, 0); @@ -883,6 +882,7 @@ static int l2_cache_pmu_probe_cluster(struct device *dev, void *data) spin_lock_init(&cluster->pmu_lock); + list_add(&cluster->next, &l2cache_pmu->clusters); l2cache_pmu->num_pmus++; return 0; diff --git a/drivers/pinctrl/pinctrl-amd.c b/drivers/pinctrl/pinctrl-amd.c index f279b360c20d..43d3530bab48 100644 --- a/drivers/pinctrl/pinctrl-amd.c +++ b/drivers/pinctrl/pinctrl-amd.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "core.h" #include "pinctrl-utils.h" @@ -636,9 +637,8 @@ static bool do_amd_gpio_irq_handler(int irq, void *dev_id) regval = readl(regs + i); if (regval & PIN_IRQ_PENDING) - dev_dbg(&gpio_dev->pdev->dev, - "GPIO %d is active: 0x%x", - irqnr + i, regval); + pm_pr_dbg("GPIO %d is active: 0x%x", + irqnr + i, regval); /* caused wake on resume context for shared IRQ */ if (irq < 0 && (regval & BIT(WAKE_STS_OFF))) diff --git a/drivers/platform/chrome/cros_ec_i2c.c b/drivers/platform/chrome/cros_ec_i2c.c index dbe698f33128..e29c51cbfd71 100644 --- a/drivers/platform/chrome/cros_ec_i2c.c +++ b/drivers/platform/chrome/cros_ec_i2c.c @@ -372,7 +372,7 @@ static struct i2c_driver cros_ec_driver = { .of_match_table = of_match_ptr(cros_ec_i2c_of_match), .pm = &cros_ec_i2c_pm_ops, }, - .probe_new = cros_ec_i2c_probe, + .probe = cros_ec_i2c_probe, .remove = cros_ec_i2c_remove, .id_table = cros_ec_i2c_id, }; diff --git a/drivers/platform/chrome/cros_ec_lpc.c b/drivers/platform/chrome/cros_ec_lpc.c index 68bba0fcafab..500a61b093e4 100644 --- a/drivers/platform/chrome/cros_ec_lpc.c +++ b/drivers/platform/chrome/cros_ec_lpc.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -315,6 +316,7 @@ static int cros_ec_lpc_readmem(struct cros_ec_device *ec, unsigned int offset, static void cros_ec_lpc_acpi_notify(acpi_handle device, u32 value, void *data) { + static const char *env[] = { "ERROR=PANIC", NULL }; struct cros_ec_device *ec_dev = data; bool ec_has_more_events; int ret; @@ -324,6 +326,7 @@ static void cros_ec_lpc_acpi_notify(acpi_handle device, u32 value, void *data) if (value == ACPI_NOTIFY_CROS_EC_PANIC) { dev_emerg(ec_dev->dev, "CrOS EC Panic Reported. Shutdown is imminent!"); blocking_notifier_call_chain(&ec_dev->panic_notifier, 0, ec_dev); + kobject_uevent_env(&ec_dev->dev->kobj, KOBJ_CHANGE, (char **)env); /* Begin orderly shutdown. Force shutdown after 1 second. */ hw_protection_shutdown("CrOS EC Panic", 1000); /* Do not query for other events after a panic is reported */ @@ -543,23 +546,25 @@ static const struct dmi_system_id cros_ec_lpc_dmi_table[] __initconst = { MODULE_DEVICE_TABLE(dmi, cros_ec_lpc_dmi_table); #ifdef CONFIG_PM_SLEEP -static int cros_ec_lpc_suspend(struct device *dev) +static int cros_ec_lpc_prepare(struct device *dev) { struct cros_ec_device *ec_dev = dev_get_drvdata(dev); return cros_ec_suspend(ec_dev); } -static int cros_ec_lpc_resume(struct device *dev) +static void cros_ec_lpc_complete(struct device *dev) { struct cros_ec_device *ec_dev = dev_get_drvdata(dev); - - return cros_ec_resume(ec_dev); + cros_ec_resume(ec_dev); } #endif static const struct dev_pm_ops cros_ec_lpc_pm_ops = { - SET_LATE_SYSTEM_SLEEP_PM_OPS(cros_ec_lpc_suspend, cros_ec_lpc_resume) +#ifdef CONFIG_PM_SLEEP + .prepare = cros_ec_lpc_prepare, + .complete = cros_ec_lpc_complete +#endif }; static struct platform_driver cros_ec_lpc_driver = { diff --git a/drivers/platform/chrome/cros_ec_spi.c b/drivers/platform/chrome/cros_ec_spi.c index 21143dba8970..3e88cc92e819 100644 --- a/drivers/platform/chrome/cros_ec_spi.c +++ b/drivers/platform/chrome/cros_ec_spi.c @@ -104,13 +104,7 @@ static void debug_packet(struct device *dev, const char *name, u8 *ptr, int len) { #ifdef DEBUG - int i; - - dev_dbg(dev, "%s: ", name); - for (i = 0; i < len; i++) - pr_cont(" %02x", ptr[i]); - - pr_cont("\n"); + dev_dbg(dev, "%s: %*ph\n", name, len, ptr); #endif } diff --git a/drivers/platform/chrome/cros_hps_i2c.c b/drivers/platform/chrome/cros_hps_i2c.c index 62ccb1acb5de..b31313080332 100644 --- a/drivers/platform/chrome/cros_hps_i2c.c +++ b/drivers/platform/chrome/cros_hps_i2c.c @@ -143,7 +143,7 @@ MODULE_DEVICE_TABLE(acpi, hps_acpi_id); #endif /* CONFIG_ACPI */ static struct i2c_driver hps_i2c_driver = { - .probe_new = hps_i2c_probe, + .probe = hps_i2c_probe, .remove = hps_i2c_remove, .id_table = hps_i2c_id, .driver = { diff --git a/drivers/platform/chrome/cros_typec_switch.c b/drivers/platform/chrome/cros_typec_switch.c index 752720483753..0eefdcf14d63 100644 --- a/drivers/platform/chrome/cros_typec_switch.c +++ b/drivers/platform/chrome/cros_typec_switch.c @@ -51,13 +51,18 @@ static int cros_typec_cmd_mux_set(struct cros_typec_switch_data *sdata, int port static int cros_typec_get_mux_state(unsigned long mode, struct typec_altmode *alt) { int ret = -EOPNOTSUPP; + u8 pin_assign; - if (mode == TYPEC_STATE_SAFE) + if (mode == TYPEC_STATE_SAFE) { ret = USB_PD_MUX_SAFE_MODE; - else if (mode == TYPEC_STATE_USB) + } else if (mode == TYPEC_STATE_USB) { ret = USB_PD_MUX_USB_ENABLED; - else if (alt && alt->svid == USB_TYPEC_DP_SID) + } else if (alt && alt->svid == USB_TYPEC_DP_SID) { ret = USB_PD_MUX_DP_ENABLED; + pin_assign = mode - TYPEC_STATE_MODAL; + if (pin_assign & DP_PIN_ASSIGN_D) + ret |= USB_PD_MUX_USB_ENABLED; + } return ret; } diff --git a/drivers/platform/x86/amd/pmc.c b/drivers/platform/x86/amd/pmc.c index 427905714f79..1304cd6f13f6 100644 --- a/drivers/platform/x86/amd/pmc.c +++ b/drivers/platform/x86/amd/pmc.c @@ -543,7 +543,7 @@ static int amd_pmc_idlemask_read(struct amd_pmc_dev *pdev, struct device *dev, } if (dev) - dev_dbg(pdev->dev, "SMU idlemask s0i3: 0x%x\n", val); + pm_pr_dbg("SMU idlemask s0i3: 0x%x\n", val); if (s) seq_printf(s, "SMU idlemask : 0x%x\n", val); @@ -769,7 +769,7 @@ static int amd_pmc_verify_czn_rtc(struct amd_pmc_dev *pdev, u32 *arg) *arg |= (duration << 16); rc = rtc_alarm_irq_enable(rtc_device, 0); - dev_dbg(pdev->dev, "wakeup timer programmed for %lld seconds\n", duration); + pm_pr_dbg("wakeup timer programmed for %lld seconds\n", duration); return rc; } diff --git a/drivers/powercap/Kconfig b/drivers/powercap/Kconfig index 90d33cd1b670..69ef8d081c98 100644 --- a/drivers/powercap/Kconfig +++ b/drivers/powercap/Kconfig @@ -18,10 +18,12 @@ if POWERCAP # Client driver configurations go here. config INTEL_RAPL_CORE tristate + depends on PCI + select IOSF_MBI config INTEL_RAPL tristate "Intel RAPL Support via MSR Interface" - depends on X86 && IOSF_MBI + depends on X86 && PCI select INTEL_RAPL_CORE help This enables support for the Intel Running Average Power Limit (RAPL) @@ -33,6 +35,20 @@ config INTEL_RAPL controller, CPU core (Power Plane 0), graphics uncore (Power Plane 1), etc. +config INTEL_RAPL_TPMI + tristate "Intel RAPL Support via TPMI Interface" + depends on X86 + depends on INTEL_TPMI + select INTEL_RAPL_CORE + help + This enables support for the Intel Running Average Power Limit (RAPL) + technology via TPMI interface, which allows power limits to be enforced + and monitored. + + In RAPL, the platform level settings are divided into domains for + fine grained control. These domains include processor package, DRAM + controller, platform, etc. + config IDLE_INJECT bool "Idle injection framework" depends on CPU_IDLE diff --git a/drivers/powercap/Makefile b/drivers/powercap/Makefile index 4474201b4aa7..5ab0dce565b9 100644 --- a/drivers/powercap/Makefile +++ b/drivers/powercap/Makefile @@ -5,5 +5,6 @@ obj-$(CONFIG_DTPM_DEVFREQ) += dtpm_devfreq.o obj-$(CONFIG_POWERCAP) += powercap_sys.o obj-$(CONFIG_INTEL_RAPL_CORE) += intel_rapl_common.o obj-$(CONFIG_INTEL_RAPL) += intel_rapl_msr.o +obj-$(CONFIG_INTEL_RAPL_TPMI) += intel_rapl_tpmi.o obj-$(CONFIG_IDLE_INJECT) += idle_inject.o obj-$(CONFIG_ARM_SCMI_POWERCAP) += arm_scmi_powercap.o diff --git a/drivers/powercap/intel_rapl_common.c b/drivers/powercap/intel_rapl_common.c index 8970c7b80884..4e646e5e48f6 100644 --- a/drivers/powercap/intel_rapl_common.c +++ b/drivers/powercap/intel_rapl_common.c @@ -75,6 +75,15 @@ #define PSYS_TIME_WINDOW1_MASK (0x7FULL<<19) #define PSYS_TIME_WINDOW2_MASK (0x7FULL<<51) +/* bitmasks for RAPL TPMI, used by primitive access functions */ +#define TPMI_POWER_LIMIT_MASK 0x3FFFF +#define TPMI_POWER_LIMIT_ENABLE BIT_ULL(62) +#define TPMI_TIME_WINDOW_MASK (0x7FULL<<18) +#define TPMI_INFO_SPEC_MASK 0x3FFFF +#define TPMI_INFO_MIN_MASK (0x3FFFFULL << 18) +#define TPMI_INFO_MAX_MASK (0x3FFFFULL << 36) +#define TPMI_INFO_MAX_TIME_WIN_MASK (0x7FULL << 54) + /* Non HW constants */ #define RAPL_PRIMITIVE_DERIVED BIT(1) /* not from raw data */ #define RAPL_PRIMITIVE_DUMMY BIT(2) @@ -94,26 +103,120 @@ enum unit_type { #define DOMAIN_STATE_INACTIVE BIT(0) #define DOMAIN_STATE_POWER_LIMIT_SET BIT(1) -#define DOMAIN_STATE_BIOS_LOCKED BIT(2) -static const char pl1_name[] = "long_term"; -static const char pl2_name[] = "short_term"; -static const char pl4_name[] = "peak_power"; +static const char *pl_names[NR_POWER_LIMITS] = { + [POWER_LIMIT1] = "long_term", + [POWER_LIMIT2] = "short_term", + [POWER_LIMIT4] = "peak_power", +}; + +enum pl_prims { + PL_ENABLE, + PL_CLAMP, + PL_LIMIT, + PL_TIME_WINDOW, + PL_MAX_POWER, + PL_LOCK, +}; + +static bool is_pl_valid(struct rapl_domain *rd, int pl) +{ + if (pl < POWER_LIMIT1 || pl > POWER_LIMIT4) + return false; + return rd->rpl[pl].name ? true : false; +} + +static int get_pl_lock_prim(struct rapl_domain *rd, int pl) +{ + if (rd->rp->priv->type == RAPL_IF_TPMI) { + if (pl == POWER_LIMIT1) + return PL1_LOCK; + if (pl == POWER_LIMIT2) + return PL2_LOCK; + if (pl == POWER_LIMIT4) + return PL4_LOCK; + } + + /* MSR/MMIO Interface doesn't have Lock bit for PL4 */ + if (pl == POWER_LIMIT4) + return -EINVAL; + + /* + * Power Limit register that supports two power limits has a different + * bit position for the Lock bit. + */ + if (rd->rp->priv->limits[rd->id] & BIT(POWER_LIMIT2)) + return FW_HIGH_LOCK; + return FW_LOCK; +} + +static int get_pl_prim(struct rapl_domain *rd, int pl, enum pl_prims prim) +{ + switch (pl) { + case POWER_LIMIT1: + if (prim == PL_ENABLE) + return PL1_ENABLE; + if (prim == PL_CLAMP && rd->rp->priv->type != RAPL_IF_TPMI) + return PL1_CLAMP; + if (prim == PL_LIMIT) + return POWER_LIMIT1; + if (prim == PL_TIME_WINDOW) + return TIME_WINDOW1; + if (prim == PL_MAX_POWER) + return THERMAL_SPEC_POWER; + if (prim == PL_LOCK) + return get_pl_lock_prim(rd, pl); + return -EINVAL; + case POWER_LIMIT2: + if (prim == PL_ENABLE) + return PL2_ENABLE; + if (prim == PL_CLAMP && rd->rp->priv->type != RAPL_IF_TPMI) + return PL2_CLAMP; + if (prim == PL_LIMIT) + return POWER_LIMIT2; + if (prim == PL_TIME_WINDOW) + return TIME_WINDOW2; + if (prim == PL_MAX_POWER) + return MAX_POWER; + if (prim == PL_LOCK) + return get_pl_lock_prim(rd, pl); + return -EINVAL; + case POWER_LIMIT4: + if (prim == PL_LIMIT) + return POWER_LIMIT4; + if (prim == PL_ENABLE) + return PL4_ENABLE; + /* PL4 would be around two times PL2, use same prim as PL2. */ + if (prim == PL_MAX_POWER) + return MAX_POWER; + if (prim == PL_LOCK) + return get_pl_lock_prim(rd, pl); + return -EINVAL; + default: + return -EINVAL; + } +} #define power_zone_to_rapl_domain(_zone) \ container_of(_zone, struct rapl_domain, power_zone) struct rapl_defaults { u8 floor_freq_reg_addr; - int (*check_unit)(struct rapl_package *rp, int cpu); + int (*check_unit)(struct rapl_domain *rd); void (*set_floor_freq)(struct rapl_domain *rd, bool mode); - u64 (*compute_time_window)(struct rapl_package *rp, u64 val, + u64 (*compute_time_window)(struct rapl_domain *rd, u64 val, bool to_raw); unsigned int dram_domain_energy_unit; unsigned int psys_domain_energy_unit; bool spr_psys_bits; }; -static struct rapl_defaults *rapl_defaults; +static struct rapl_defaults *defaults_msr; +static const struct rapl_defaults defaults_tpmi; + +static struct rapl_defaults *get_defaults(struct rapl_package *rp) +{ + return rp->priv->defaults; +} /* Sideband MBI registers */ #define IOSF_CPU_POWER_BUDGET_CTL_BYT (0x2) @@ -150,6 +253,12 @@ static int rapl_read_data_raw(struct rapl_domain *rd, static int rapl_write_data_raw(struct rapl_domain *rd, enum rapl_primitives prim, unsigned long long value); +static int rapl_read_pl_data(struct rapl_domain *rd, int pl, + enum pl_prims pl_prim, + bool xlate, u64 *data); +static int rapl_write_pl_data(struct rapl_domain *rd, int pl, + enum pl_prims pl_prim, + unsigned long long value); static u64 rapl_unit_xlate(struct rapl_domain *rd, enum unit_type type, u64 value, int to_raw); static void package_power_limit_irq_save(struct rapl_package *rp); @@ -217,7 +326,7 @@ static int find_nr_power_limit(struct rapl_domain *rd) int i, nr_pl = 0; for (i = 0; i < NR_POWER_LIMITS; i++) { - if (rd->rpl[i].name) + if (is_pl_valid(rd, i)) nr_pl++; } @@ -227,37 +336,35 @@ static int find_nr_power_limit(struct rapl_domain *rd) static int set_domain_enable(struct powercap_zone *power_zone, bool mode) { struct rapl_domain *rd = power_zone_to_rapl_domain(power_zone); - - if (rd->state & DOMAIN_STATE_BIOS_LOCKED) - return -EACCES; + struct rapl_defaults *defaults = get_defaults(rd->rp); + int ret; cpus_read_lock(); - rapl_write_data_raw(rd, PL1_ENABLE, mode); - if (rapl_defaults->set_floor_freq) - rapl_defaults->set_floor_freq(rd, mode); + ret = rapl_write_pl_data(rd, POWER_LIMIT1, PL_ENABLE, mode); + if (!ret && defaults->set_floor_freq) + defaults->set_floor_freq(rd, mode); cpus_read_unlock(); - return 0; + return ret; } static int get_domain_enable(struct powercap_zone *power_zone, bool *mode) { struct rapl_domain *rd = power_zone_to_rapl_domain(power_zone); u64 val; + int ret; - if (rd->state & DOMAIN_STATE_BIOS_LOCKED) { + if (rd->rpl[POWER_LIMIT1].locked) { *mode = false; return 0; } cpus_read_lock(); - if (rapl_read_data_raw(rd, PL1_ENABLE, true, &val)) { - cpus_read_unlock(); - return -EIO; - } - *mode = val; + ret = rapl_read_pl_data(rd, POWER_LIMIT1, PL_ENABLE, true, &val); + if (!ret) + *mode = val; cpus_read_unlock(); - return 0; + return ret; } /* per RAPL domain ops, in the order of rapl_domain_type */ @@ -313,8 +420,8 @@ static int contraint_to_pl(struct rapl_domain *rd, int cid) { int i, j; - for (i = 0, j = 0; i < NR_POWER_LIMITS; i++) { - if ((rd->rpl[i].name) && j++ == cid) { + for (i = POWER_LIMIT1, j = 0; i < NR_POWER_LIMITS; i++) { + if (is_pl_valid(rd, i) && j++ == cid) { pr_debug("%s: index %d\n", __func__, i); return i; } @@ -335,36 +442,11 @@ static int set_power_limit(struct powercap_zone *power_zone, int cid, cpus_read_lock(); rd = power_zone_to_rapl_domain(power_zone); id = contraint_to_pl(rd, cid); - if (id < 0) { - ret = id; - goto set_exit; - } - rp = rd->rp; - if (rd->state & DOMAIN_STATE_BIOS_LOCKED) { - dev_warn(&power_zone->dev, - "%s locked by BIOS, monitoring only\n", rd->name); - ret = -EACCES; - goto set_exit; - } - - switch (rd->rpl[id].prim_id) { - case PL1_ENABLE: - rapl_write_data_raw(rd, POWER_LIMIT1, power_limit); - break; - case PL2_ENABLE: - rapl_write_data_raw(rd, POWER_LIMIT2, power_limit); - break; - case PL4_ENABLE: - rapl_write_data_raw(rd, POWER_LIMIT4, power_limit); - break; - default: - ret = -EINVAL; - } + ret = rapl_write_pl_data(rd, id, PL_LIMIT, power_limit); if (!ret) package_power_limit_irq_save(rp); -set_exit: cpus_read_unlock(); return ret; } @@ -374,38 +456,17 @@ static int get_current_power_limit(struct powercap_zone *power_zone, int cid, { struct rapl_domain *rd; u64 val; - int prim; int ret = 0; int id; cpus_read_lock(); rd = power_zone_to_rapl_domain(power_zone); id = contraint_to_pl(rd, cid); - if (id < 0) { - ret = id; - goto get_exit; - } - switch (rd->rpl[id].prim_id) { - case PL1_ENABLE: - prim = POWER_LIMIT1; - break; - case PL2_ENABLE: - prim = POWER_LIMIT2; - break; - case PL4_ENABLE: - prim = POWER_LIMIT4; - break; - default: - cpus_read_unlock(); - return -EINVAL; - } - if (rapl_read_data_raw(rd, prim, true, &val)) - ret = -EIO; - else + ret = rapl_read_pl_data(rd, id, PL_LIMIT, true, &val); + if (!ret) *data = val; -get_exit: cpus_read_unlock(); return ret; @@ -421,23 +482,9 @@ static int set_time_window(struct powercap_zone *power_zone, int cid, cpus_read_lock(); rd = power_zone_to_rapl_domain(power_zone); id = contraint_to_pl(rd, cid); - if (id < 0) { - ret = id; - goto set_time_exit; - } - switch (rd->rpl[id].prim_id) { - case PL1_ENABLE: - rapl_write_data_raw(rd, TIME_WINDOW1, window); - break; - case PL2_ENABLE: - rapl_write_data_raw(rd, TIME_WINDOW2, window); - break; - default: - ret = -EINVAL; - } + ret = rapl_write_pl_data(rd, id, PL_TIME_WINDOW, window); -set_time_exit: cpus_read_unlock(); return ret; } @@ -453,33 +500,11 @@ static int get_time_window(struct powercap_zone *power_zone, int cid, cpus_read_lock(); rd = power_zone_to_rapl_domain(power_zone); id = contraint_to_pl(rd, cid); - if (id < 0) { - ret = id; - goto get_time_exit; - } - switch (rd->rpl[id].prim_id) { - case PL1_ENABLE: - ret = rapl_read_data_raw(rd, TIME_WINDOW1, true, &val); - break; - case PL2_ENABLE: - ret = rapl_read_data_raw(rd, TIME_WINDOW2, true, &val); - break; - case PL4_ENABLE: - /* - * Time window parameter is not applicable for PL4 entry - * so assigining '0' as default value. - */ - val = 0; - break; - default: - cpus_read_unlock(); - return -EINVAL; - } + ret = rapl_read_pl_data(rd, id, PL_TIME_WINDOW, true, &val); if (!ret) *data = val; -get_time_exit: cpus_read_unlock(); return ret; @@ -499,36 +524,23 @@ static const char *get_constraint_name(struct powercap_zone *power_zone, return NULL; } -static int get_max_power(struct powercap_zone *power_zone, int id, u64 *data) +static int get_max_power(struct powercap_zone *power_zone, int cid, u64 *data) { struct rapl_domain *rd; u64 val; - int prim; int ret = 0; + int id; cpus_read_lock(); rd = power_zone_to_rapl_domain(power_zone); - switch (rd->rpl[id].prim_id) { - case PL1_ENABLE: - prim = THERMAL_SPEC_POWER; - break; - case PL2_ENABLE: - prim = MAX_POWER; - break; - case PL4_ENABLE: - prim = MAX_POWER; - break; - default: - cpus_read_unlock(); - return -EINVAL; - } - if (rapl_read_data_raw(rd, prim, true, &val)) - ret = -EIO; - else + id = contraint_to_pl(rd, cid); + + ret = rapl_read_pl_data(rd, id, PL_MAX_POWER, true, &val); + if (!ret) *data = val; /* As a generalization rule, PL4 would be around two times PL2. */ - if (rd->rpl[id].prim_id == PL4_ENABLE) + if (id == POWER_LIMIT4) *data = *data * 2; cpus_read_unlock(); @@ -545,6 +557,12 @@ static const struct powercap_zone_constraint_ops constraint_ops = { .get_name = get_constraint_name, }; +/* Return the id used for read_raw/write_raw callback */ +static int get_rid(struct rapl_package *rp) +{ + return rp->lead_cpu >= 0 ? rp->lead_cpu : rp->id; +} + /* called after domain detection and package level data are set */ static void rapl_init_domains(struct rapl_package *rp) { @@ -554,6 +572,7 @@ static void rapl_init_domains(struct rapl_package *rp) for (i = 0; i < RAPL_DOMAIN_MAX; i++) { unsigned int mask = rp->domain_map & (1 << i); + int t; if (!mask) continue; @@ -562,51 +581,26 @@ static void rapl_init_domains(struct rapl_package *rp) if (i == RAPL_DOMAIN_PLATFORM && rp->id > 0) { snprintf(rd->name, RAPL_DOMAIN_NAME_LENGTH, "psys-%d", - topology_physical_package_id(rp->lead_cpu)); - } else + rp->lead_cpu >= 0 ? topology_physical_package_id(rp->lead_cpu) : + rp->id); + } else { snprintf(rd->name, RAPL_DOMAIN_NAME_LENGTH, "%s", rapl_domain_names[i]); - - rd->id = i; - rd->rpl[0].prim_id = PL1_ENABLE; - rd->rpl[0].name = pl1_name; - - /* - * The PL2 power domain is applicable for limits two - * and limits three - */ - if (rp->priv->limits[i] >= 2) { - rd->rpl[1].prim_id = PL2_ENABLE; - rd->rpl[1].name = pl2_name; } - /* Enable PL4 domain if the total power limits are three */ - if (rp->priv->limits[i] == 3) { - rd->rpl[2].prim_id = PL4_ENABLE; - rd->rpl[2].name = pl4_name; + rd->id = i; + + /* PL1 is supported by default */ + rp->priv->limits[i] |= BIT(POWER_LIMIT1); + + for (t = POWER_LIMIT1; t < NR_POWER_LIMITS; t++) { + if (rp->priv->limits[i] & BIT(t)) + rd->rpl[t].name = pl_names[t]; } for (j = 0; j < RAPL_DOMAIN_REG_MAX; j++) rd->regs[j] = rp->priv->regs[i][j]; - switch (i) { - case RAPL_DOMAIN_DRAM: - rd->domain_energy_unit = - rapl_defaults->dram_domain_energy_unit; - if (rd->domain_energy_unit) - pr_info("DRAM domain energy unit %dpj\n", - rd->domain_energy_unit); - break; - case RAPL_DOMAIN_PLATFORM: - rd->domain_energy_unit = - rapl_defaults->psys_domain_energy_unit; - if (rd->domain_energy_unit) - pr_info("Platform domain energy unit %dpj\n", - rd->domain_energy_unit); - break; - default: - break; - } rd++; } } @@ -615,23 +609,19 @@ static u64 rapl_unit_xlate(struct rapl_domain *rd, enum unit_type type, u64 value, int to_raw) { u64 units = 1; - struct rapl_package *rp = rd->rp; + struct rapl_defaults *defaults = get_defaults(rd->rp); u64 scale = 1; switch (type) { case POWER_UNIT: - units = rp->power_unit; + units = rd->power_unit; break; case ENERGY_UNIT: scale = ENERGY_UNIT_SCALE; - /* per domain unit takes precedence */ - if (rd->domain_energy_unit) - units = rd->domain_energy_unit; - else - units = rp->energy_unit; + units = rd->energy_unit; break; case TIME_UNIT: - return rapl_defaults->compute_time_window(rp, value, to_raw); + return defaults->compute_time_window(rd, value, to_raw); case ARBITRARY_UNIT: default: return value; @@ -645,67 +635,141 @@ static u64 rapl_unit_xlate(struct rapl_domain *rd, enum unit_type type, return div64_u64(value, scale); } -/* in the order of enum rapl_primitives */ -static struct rapl_primitive_info rpi[] = { +/* RAPL primitives for MSR and MMIO I/F */ +static struct rapl_primitive_info rpi_msr[NR_RAPL_PRIMITIVES] = { /* name, mask, shift, msr index, unit divisor */ - PRIMITIVE_INFO_INIT(ENERGY_COUNTER, ENERGY_STATUS_MASK, 0, - RAPL_DOMAIN_REG_STATUS, ENERGY_UNIT, 0), - PRIMITIVE_INFO_INIT(POWER_LIMIT1, POWER_LIMIT1_MASK, 0, + [POWER_LIMIT1] = PRIMITIVE_INFO_INIT(POWER_LIMIT1, POWER_LIMIT1_MASK, 0, RAPL_DOMAIN_REG_LIMIT, POWER_UNIT, 0), - PRIMITIVE_INFO_INIT(POWER_LIMIT2, POWER_LIMIT2_MASK, 32, + [POWER_LIMIT2] = PRIMITIVE_INFO_INIT(POWER_LIMIT2, POWER_LIMIT2_MASK, 32, RAPL_DOMAIN_REG_LIMIT, POWER_UNIT, 0), - PRIMITIVE_INFO_INIT(POWER_LIMIT4, POWER_LIMIT4_MASK, 0, + [POWER_LIMIT4] = PRIMITIVE_INFO_INIT(POWER_LIMIT4, POWER_LIMIT4_MASK, 0, RAPL_DOMAIN_REG_PL4, POWER_UNIT, 0), - PRIMITIVE_INFO_INIT(FW_LOCK, POWER_LOW_LOCK, 31, + [ENERGY_COUNTER] = PRIMITIVE_INFO_INIT(ENERGY_COUNTER, ENERGY_STATUS_MASK, 0, + RAPL_DOMAIN_REG_STATUS, ENERGY_UNIT, 0), + [FW_LOCK] = PRIMITIVE_INFO_INIT(FW_LOCK, POWER_LOW_LOCK, 31, RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(PL1_ENABLE, POWER_LIMIT1_ENABLE, 15, + [FW_HIGH_LOCK] = PRIMITIVE_INFO_INIT(FW_LOCK, POWER_HIGH_LOCK, 63, RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(PL1_CLAMP, POWER_LIMIT1_CLAMP, 16, + [PL1_ENABLE] = PRIMITIVE_INFO_INIT(PL1_ENABLE, POWER_LIMIT1_ENABLE, 15, RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(PL2_ENABLE, POWER_LIMIT2_ENABLE, 47, + [PL1_CLAMP] = PRIMITIVE_INFO_INIT(PL1_CLAMP, POWER_LIMIT1_CLAMP, 16, RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(PL2_CLAMP, POWER_LIMIT2_CLAMP, 48, + [PL2_ENABLE] = PRIMITIVE_INFO_INIT(PL2_ENABLE, POWER_LIMIT2_ENABLE, 47, RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(PL4_ENABLE, POWER_LIMIT4_MASK, 0, + [PL2_CLAMP] = PRIMITIVE_INFO_INIT(PL2_CLAMP, POWER_LIMIT2_CLAMP, 48, + RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), + [PL4_ENABLE] = PRIMITIVE_INFO_INIT(PL4_ENABLE, POWER_LIMIT4_MASK, 0, RAPL_DOMAIN_REG_PL4, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(TIME_WINDOW1, TIME_WINDOW1_MASK, 17, + [TIME_WINDOW1] = PRIMITIVE_INFO_INIT(TIME_WINDOW1, TIME_WINDOW1_MASK, 17, RAPL_DOMAIN_REG_LIMIT, TIME_UNIT, 0), - PRIMITIVE_INFO_INIT(TIME_WINDOW2, TIME_WINDOW2_MASK, 49, + [TIME_WINDOW2] = PRIMITIVE_INFO_INIT(TIME_WINDOW2, TIME_WINDOW2_MASK, 49, RAPL_DOMAIN_REG_LIMIT, TIME_UNIT, 0), - PRIMITIVE_INFO_INIT(THERMAL_SPEC_POWER, POWER_INFO_THERMAL_SPEC_MASK, + [THERMAL_SPEC_POWER] = PRIMITIVE_INFO_INIT(THERMAL_SPEC_POWER, POWER_INFO_THERMAL_SPEC_MASK, 0, RAPL_DOMAIN_REG_INFO, POWER_UNIT, 0), - PRIMITIVE_INFO_INIT(MAX_POWER, POWER_INFO_MAX_MASK, 32, + [MAX_POWER] = PRIMITIVE_INFO_INIT(MAX_POWER, POWER_INFO_MAX_MASK, 32, RAPL_DOMAIN_REG_INFO, POWER_UNIT, 0), - PRIMITIVE_INFO_INIT(MIN_POWER, POWER_INFO_MIN_MASK, 16, + [MIN_POWER] = PRIMITIVE_INFO_INIT(MIN_POWER, POWER_INFO_MIN_MASK, 16, RAPL_DOMAIN_REG_INFO, POWER_UNIT, 0), - PRIMITIVE_INFO_INIT(MAX_TIME_WINDOW, POWER_INFO_MAX_TIME_WIN_MASK, 48, + [MAX_TIME_WINDOW] = PRIMITIVE_INFO_INIT(MAX_TIME_WINDOW, POWER_INFO_MAX_TIME_WIN_MASK, 48, RAPL_DOMAIN_REG_INFO, TIME_UNIT, 0), - PRIMITIVE_INFO_INIT(THROTTLED_TIME, PERF_STATUS_THROTTLE_TIME_MASK, 0, + [THROTTLED_TIME] = PRIMITIVE_INFO_INIT(THROTTLED_TIME, PERF_STATUS_THROTTLE_TIME_MASK, 0, RAPL_DOMAIN_REG_PERF, TIME_UNIT, 0), - PRIMITIVE_INFO_INIT(PRIORITY_LEVEL, PP_POLICY_MASK, 0, + [PRIORITY_LEVEL] = PRIMITIVE_INFO_INIT(PRIORITY_LEVEL, PP_POLICY_MASK, 0, RAPL_DOMAIN_REG_POLICY, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(PSYS_POWER_LIMIT1, PSYS_POWER_LIMIT1_MASK, 0, + [PSYS_POWER_LIMIT1] = PRIMITIVE_INFO_INIT(PSYS_POWER_LIMIT1, PSYS_POWER_LIMIT1_MASK, 0, RAPL_DOMAIN_REG_LIMIT, POWER_UNIT, 0), - PRIMITIVE_INFO_INIT(PSYS_POWER_LIMIT2, PSYS_POWER_LIMIT2_MASK, 32, + [PSYS_POWER_LIMIT2] = PRIMITIVE_INFO_INIT(PSYS_POWER_LIMIT2, PSYS_POWER_LIMIT2_MASK, 32, RAPL_DOMAIN_REG_LIMIT, POWER_UNIT, 0), - PRIMITIVE_INFO_INIT(PSYS_PL1_ENABLE, PSYS_POWER_LIMIT1_ENABLE, 17, + [PSYS_PL1_ENABLE] = PRIMITIVE_INFO_INIT(PSYS_PL1_ENABLE, PSYS_POWER_LIMIT1_ENABLE, 17, RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(PSYS_PL2_ENABLE, PSYS_POWER_LIMIT2_ENABLE, 49, + [PSYS_PL2_ENABLE] = PRIMITIVE_INFO_INIT(PSYS_PL2_ENABLE, PSYS_POWER_LIMIT2_ENABLE, 49, RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), - PRIMITIVE_INFO_INIT(PSYS_TIME_WINDOW1, PSYS_TIME_WINDOW1_MASK, 19, + [PSYS_TIME_WINDOW1] = PRIMITIVE_INFO_INIT(PSYS_TIME_WINDOW1, PSYS_TIME_WINDOW1_MASK, 19, RAPL_DOMAIN_REG_LIMIT, TIME_UNIT, 0), - PRIMITIVE_INFO_INIT(PSYS_TIME_WINDOW2, PSYS_TIME_WINDOW2_MASK, 51, + [PSYS_TIME_WINDOW2] = PRIMITIVE_INFO_INIT(PSYS_TIME_WINDOW2, PSYS_TIME_WINDOW2_MASK, 51, RAPL_DOMAIN_REG_LIMIT, TIME_UNIT, 0), /* non-hardware */ - PRIMITIVE_INFO_INIT(AVERAGE_POWER, 0, 0, 0, POWER_UNIT, + [AVERAGE_POWER] = PRIMITIVE_INFO_INIT(AVERAGE_POWER, 0, 0, 0, POWER_UNIT, RAPL_PRIMITIVE_DERIVED), - {NULL, 0, 0, 0}, }; +/* RAPL primitives for TPMI I/F */ +static struct rapl_primitive_info rpi_tpmi[NR_RAPL_PRIMITIVES] = { + /* name, mask, shift, msr index, unit divisor */ + [POWER_LIMIT1] = PRIMITIVE_INFO_INIT(POWER_LIMIT1, TPMI_POWER_LIMIT_MASK, 0, + RAPL_DOMAIN_REG_LIMIT, POWER_UNIT, 0), + [POWER_LIMIT2] = PRIMITIVE_INFO_INIT(POWER_LIMIT2, TPMI_POWER_LIMIT_MASK, 0, + RAPL_DOMAIN_REG_PL2, POWER_UNIT, 0), + [POWER_LIMIT4] = PRIMITIVE_INFO_INIT(POWER_LIMIT4, TPMI_POWER_LIMIT_MASK, 0, + RAPL_DOMAIN_REG_PL4, POWER_UNIT, 0), + [ENERGY_COUNTER] = PRIMITIVE_INFO_INIT(ENERGY_COUNTER, ENERGY_STATUS_MASK, 0, + RAPL_DOMAIN_REG_STATUS, ENERGY_UNIT, 0), + [PL1_LOCK] = PRIMITIVE_INFO_INIT(PL1_LOCK, POWER_HIGH_LOCK, 63, + RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), + [PL2_LOCK] = PRIMITIVE_INFO_INIT(PL2_LOCK, POWER_HIGH_LOCK, 63, + RAPL_DOMAIN_REG_PL2, ARBITRARY_UNIT, 0), + [PL4_LOCK] = PRIMITIVE_INFO_INIT(PL4_LOCK, POWER_HIGH_LOCK, 63, + RAPL_DOMAIN_REG_PL4, ARBITRARY_UNIT, 0), + [PL1_ENABLE] = PRIMITIVE_INFO_INIT(PL1_ENABLE, TPMI_POWER_LIMIT_ENABLE, 62, + RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), + [PL2_ENABLE] = PRIMITIVE_INFO_INIT(PL2_ENABLE, TPMI_POWER_LIMIT_ENABLE, 62, + RAPL_DOMAIN_REG_PL2, ARBITRARY_UNIT, 0), + [PL4_ENABLE] = PRIMITIVE_INFO_INIT(PL4_ENABLE, TPMI_POWER_LIMIT_ENABLE, 62, + RAPL_DOMAIN_REG_PL4, ARBITRARY_UNIT, 0), + [TIME_WINDOW1] = PRIMITIVE_INFO_INIT(TIME_WINDOW1, TPMI_TIME_WINDOW_MASK, 18, + RAPL_DOMAIN_REG_LIMIT, TIME_UNIT, 0), + [TIME_WINDOW2] = PRIMITIVE_INFO_INIT(TIME_WINDOW2, TPMI_TIME_WINDOW_MASK, 18, + RAPL_DOMAIN_REG_PL2, TIME_UNIT, 0), + [THERMAL_SPEC_POWER] = PRIMITIVE_INFO_INIT(THERMAL_SPEC_POWER, TPMI_INFO_SPEC_MASK, 0, + RAPL_DOMAIN_REG_INFO, POWER_UNIT, 0), + [MAX_POWER] = PRIMITIVE_INFO_INIT(MAX_POWER, TPMI_INFO_MAX_MASK, 36, + RAPL_DOMAIN_REG_INFO, POWER_UNIT, 0), + [MIN_POWER] = PRIMITIVE_INFO_INIT(MIN_POWER, TPMI_INFO_MIN_MASK, 18, + RAPL_DOMAIN_REG_INFO, POWER_UNIT, 0), + [MAX_TIME_WINDOW] = PRIMITIVE_INFO_INIT(MAX_TIME_WINDOW, TPMI_INFO_MAX_TIME_WIN_MASK, 54, + RAPL_DOMAIN_REG_INFO, TIME_UNIT, 0), + [THROTTLED_TIME] = PRIMITIVE_INFO_INIT(THROTTLED_TIME, PERF_STATUS_THROTTLE_TIME_MASK, 0, + RAPL_DOMAIN_REG_PERF, TIME_UNIT, 0), + /* non-hardware */ + [AVERAGE_POWER] = PRIMITIVE_INFO_INIT(AVERAGE_POWER, 0, 0, 0, + POWER_UNIT, RAPL_PRIMITIVE_DERIVED), +}; + +static struct rapl_primitive_info *get_rpi(struct rapl_package *rp, int prim) +{ + struct rapl_primitive_info *rpi = rp->priv->rpi; + + if (prim < 0 || prim > NR_RAPL_PRIMITIVES || !rpi) + return NULL; + + return &rpi[prim]; +} + +static int rapl_config(struct rapl_package *rp) +{ + switch (rp->priv->type) { + /* MMIO I/F shares the same register layout as MSR registers */ + case RAPL_IF_MMIO: + case RAPL_IF_MSR: + rp->priv->defaults = (void *)defaults_msr; + rp->priv->rpi = (void *)rpi_msr; + break; + case RAPL_IF_TPMI: + rp->priv->defaults = (void *)&defaults_tpmi; + rp->priv->rpi = (void *)rpi_tpmi; + break; + default: + return -EINVAL; + } + return 0; +} + static enum rapl_primitives prim_fixups(struct rapl_domain *rd, enum rapl_primitives prim) { - if (!rapl_defaults->spr_psys_bits) + struct rapl_defaults *defaults = get_defaults(rd->rp); + + if (!defaults->spr_psys_bits) return prim; if (rd->id != RAPL_DOMAIN_PLATFORM) @@ -747,41 +811,33 @@ static int rapl_read_data_raw(struct rapl_domain *rd, { u64 value; enum rapl_primitives prim_fixed = prim_fixups(rd, prim); - struct rapl_primitive_info *rp = &rpi[prim_fixed]; + struct rapl_primitive_info *rpi = get_rpi(rd->rp, prim_fixed); struct reg_action ra; - int cpu; - if (!rp->name || rp->flag & RAPL_PRIMITIVE_DUMMY) + if (!rpi || !rpi->name || rpi->flag & RAPL_PRIMITIVE_DUMMY) return -EINVAL; - ra.reg = rd->regs[rp->id]; + ra.reg = rd->regs[rpi->id]; if (!ra.reg) return -EINVAL; - cpu = rd->rp->lead_cpu; - - /* domain with 2 limits has different bit */ - if (prim == FW_LOCK && rd->rp->priv->limits[rd->id] == 2) { - rp->mask = POWER_HIGH_LOCK; - rp->shift = 63; - } /* non-hardware data are collected by the polling thread */ - if (rp->flag & RAPL_PRIMITIVE_DERIVED) { + if (rpi->flag & RAPL_PRIMITIVE_DERIVED) { *data = rd->rdd.primitives[prim]; return 0; } - ra.mask = rp->mask; + ra.mask = rpi->mask; - if (rd->rp->priv->read_raw(cpu, &ra)) { - pr_debug("failed to read reg 0x%llx on cpu %d\n", ra.reg, cpu); + if (rd->rp->priv->read_raw(get_rid(rd->rp), &ra)) { + pr_debug("failed to read reg 0x%llx for %s:%s\n", ra.reg, rd->rp->name, rd->name); return -EIO; } - value = ra.value >> rp->shift; + value = ra.value >> rpi->shift; if (xlate) - *data = rapl_unit_xlate(rd, rp->unit, value, 0); + *data = rapl_unit_xlate(rd, rpi->unit, value, 0); else *data = value; @@ -794,28 +850,56 @@ static int rapl_write_data_raw(struct rapl_domain *rd, unsigned long long value) { enum rapl_primitives prim_fixed = prim_fixups(rd, prim); - struct rapl_primitive_info *rp = &rpi[prim_fixed]; - int cpu; + struct rapl_primitive_info *rpi = get_rpi(rd->rp, prim_fixed); u64 bits; struct reg_action ra; int ret; - cpu = rd->rp->lead_cpu; - bits = rapl_unit_xlate(rd, rp->unit, value, 1); - bits <<= rp->shift; - bits &= rp->mask; + if (!rpi || !rpi->name || rpi->flag & RAPL_PRIMITIVE_DUMMY) + return -EINVAL; + + bits = rapl_unit_xlate(rd, rpi->unit, value, 1); + bits <<= rpi->shift; + bits &= rpi->mask; memset(&ra, 0, sizeof(ra)); - ra.reg = rd->regs[rp->id]; - ra.mask = rp->mask; + ra.reg = rd->regs[rpi->id]; + ra.mask = rpi->mask; ra.value = bits; - ret = rd->rp->priv->write_raw(cpu, &ra); + ret = rd->rp->priv->write_raw(get_rid(rd->rp), &ra); return ret; } +static int rapl_read_pl_data(struct rapl_domain *rd, int pl, + enum pl_prims pl_prim, bool xlate, u64 *data) +{ + enum rapl_primitives prim = get_pl_prim(rd, pl, pl_prim); + + if (!is_pl_valid(rd, pl)) + return -EINVAL; + + return rapl_read_data_raw(rd, prim, xlate, data); +} + +static int rapl_write_pl_data(struct rapl_domain *rd, int pl, + enum pl_prims pl_prim, + unsigned long long value) +{ + enum rapl_primitives prim = get_pl_prim(rd, pl, pl_prim); + + if (!is_pl_valid(rd, pl)) + return -EINVAL; + + if (rd->rpl[pl].locked) { + pr_warn("%s:%s:%s locked by BIOS\n", rd->rp->name, rd->name, pl_names[pl]); + return -EACCES; + } + + return rapl_write_data_raw(rd, prim, value); +} /* * Raw RAPL data stored in MSRs are in certain scales. We need to * convert them into standard units based on the units reported in @@ -827,58 +911,58 @@ static int rapl_write_data_raw(struct rapl_domain *rd, * power unit : microWatts : Represented in milliWatts by default * time unit : microseconds: Represented in seconds by default */ -static int rapl_check_unit_core(struct rapl_package *rp, int cpu) +static int rapl_check_unit_core(struct rapl_domain *rd) { struct reg_action ra; u32 value; - ra.reg = rp->priv->reg_unit; + ra.reg = rd->regs[RAPL_DOMAIN_REG_UNIT]; ra.mask = ~0; - if (rp->priv->read_raw(cpu, &ra)) { - pr_err("Failed to read power unit REG 0x%llx on CPU %d, exit.\n", - rp->priv->reg_unit, cpu); + if (rd->rp->priv->read_raw(get_rid(rd->rp), &ra)) { + pr_err("Failed to read power unit REG 0x%llx on %s:%s, exit.\n", + ra.reg, rd->rp->name, rd->name); return -ENODEV; } value = (ra.value & ENERGY_UNIT_MASK) >> ENERGY_UNIT_OFFSET; - rp->energy_unit = ENERGY_UNIT_SCALE * 1000000 / (1 << value); + rd->energy_unit = ENERGY_UNIT_SCALE * 1000000 / (1 << value); value = (ra.value & POWER_UNIT_MASK) >> POWER_UNIT_OFFSET; - rp->power_unit = 1000000 / (1 << value); + rd->power_unit = 1000000 / (1 << value); value = (ra.value & TIME_UNIT_MASK) >> TIME_UNIT_OFFSET; - rp->time_unit = 1000000 / (1 << value); + rd->time_unit = 1000000 / (1 << value); - pr_debug("Core CPU %s energy=%dpJ, time=%dus, power=%duW\n", - rp->name, rp->energy_unit, rp->time_unit, rp->power_unit); + pr_debug("Core CPU %s:%s energy=%dpJ, time=%dus, power=%duW\n", + rd->rp->name, rd->name, rd->energy_unit, rd->time_unit, rd->power_unit); return 0; } -static int rapl_check_unit_atom(struct rapl_package *rp, int cpu) +static int rapl_check_unit_atom(struct rapl_domain *rd) { struct reg_action ra; u32 value; - ra.reg = rp->priv->reg_unit; + ra.reg = rd->regs[RAPL_DOMAIN_REG_UNIT]; ra.mask = ~0; - if (rp->priv->read_raw(cpu, &ra)) { - pr_err("Failed to read power unit REG 0x%llx on CPU %d, exit.\n", - rp->priv->reg_unit, cpu); + if (rd->rp->priv->read_raw(get_rid(rd->rp), &ra)) { + pr_err("Failed to read power unit REG 0x%llx on %s:%s, exit.\n", + ra.reg, rd->rp->name, rd->name); return -ENODEV; } value = (ra.value & ENERGY_UNIT_MASK) >> ENERGY_UNIT_OFFSET; - rp->energy_unit = ENERGY_UNIT_SCALE * 1 << value; + rd->energy_unit = ENERGY_UNIT_SCALE * 1 << value; value = (ra.value & POWER_UNIT_MASK) >> POWER_UNIT_OFFSET; - rp->power_unit = (1 << value) * 1000; + rd->power_unit = (1 << value) * 1000; value = (ra.value & TIME_UNIT_MASK) >> TIME_UNIT_OFFSET; - rp->time_unit = 1000000 / (1 << value); + rd->time_unit = 1000000 / (1 << value); - pr_debug("Atom %s energy=%dpJ, time=%dus, power=%duW\n", - rp->name, rp->energy_unit, rp->time_unit, rp->power_unit); + pr_debug("Atom %s:%s energy=%dpJ, time=%dus, power=%duW\n", + rd->rp->name, rd->name, rd->energy_unit, rd->time_unit, rd->power_unit); return 0; } @@ -910,6 +994,9 @@ static void power_limit_irq_save_cpu(void *info) static void package_power_limit_irq_save(struct rapl_package *rp) { + if (rp->lead_cpu < 0) + return; + if (!boot_cpu_has(X86_FEATURE_PTS) || !boot_cpu_has(X86_FEATURE_PLN)) return; @@ -924,6 +1011,9 @@ static void package_power_limit_irq_restore(struct rapl_package *rp) { u32 l, h; + if (rp->lead_cpu < 0) + return; + if (!boot_cpu_has(X86_FEATURE_PTS) || !boot_cpu_has(X86_FEATURE_PLN)) return; @@ -943,33 +1033,33 @@ static void package_power_limit_irq_restore(struct rapl_package *rp) static void set_floor_freq_default(struct rapl_domain *rd, bool mode) { - int nr_powerlimit = find_nr_power_limit(rd); + int i; /* always enable clamp such that p-state can go below OS requested * range. power capping priority over guranteed frequency. */ - rapl_write_data_raw(rd, PL1_CLAMP, mode); + rapl_write_pl_data(rd, POWER_LIMIT1, PL_CLAMP, mode); - /* some domains have pl2 */ - if (nr_powerlimit > 1) { - rapl_write_data_raw(rd, PL2_ENABLE, mode); - rapl_write_data_raw(rd, PL2_CLAMP, mode); + for (i = POWER_LIMIT2; i < NR_POWER_LIMITS; i++) { + rapl_write_pl_data(rd, i, PL_ENABLE, mode); + rapl_write_pl_data(rd, i, PL_CLAMP, mode); } } static void set_floor_freq_atom(struct rapl_domain *rd, bool enable) { static u32 power_ctrl_orig_val; + struct rapl_defaults *defaults = get_defaults(rd->rp); u32 mdata; - if (!rapl_defaults->floor_freq_reg_addr) { + if (!defaults->floor_freq_reg_addr) { pr_err("Invalid floor frequency config register\n"); return; } if (!power_ctrl_orig_val) iosf_mbi_read(BT_MBI_UNIT_PMC, MBI_CR_READ, - rapl_defaults->floor_freq_reg_addr, + defaults->floor_freq_reg_addr, &power_ctrl_orig_val); mdata = power_ctrl_orig_val; if (enable) { @@ -977,10 +1067,10 @@ static void set_floor_freq_atom(struct rapl_domain *rd, bool enable) mdata |= 1 << 8; } iosf_mbi_write(BT_MBI_UNIT_PMC, MBI_CR_WRITE, - rapl_defaults->floor_freq_reg_addr, mdata); + defaults->floor_freq_reg_addr, mdata); } -static u64 rapl_compute_time_window_core(struct rapl_package *rp, u64 value, +static u64 rapl_compute_time_window_core(struct rapl_domain *rd, u64 value, bool to_raw) { u64 f, y; /* fraction and exp. used for time unit */ @@ -992,12 +1082,12 @@ static u64 rapl_compute_time_window_core(struct rapl_package *rp, u64 value, if (!to_raw) { f = (value & 0x60) >> 5; y = value & 0x1f; - value = (1 << y) * (4 + f) * rp->time_unit / 4; + value = (1 << y) * (4 + f) * rd->time_unit / 4; } else { - if (value < rp->time_unit) + if (value < rd->time_unit) return 0; - do_div(value, rp->time_unit); + do_div(value, rd->time_unit); y = ilog2(value); /* @@ -1013,7 +1103,7 @@ static u64 rapl_compute_time_window_core(struct rapl_package *rp, u64 value, return value; } -static u64 rapl_compute_time_window_atom(struct rapl_package *rp, u64 value, +static u64 rapl_compute_time_window_atom(struct rapl_domain *rd, u64 value, bool to_raw) { /* @@ -1021,13 +1111,56 @@ static u64 rapl_compute_time_window_atom(struct rapl_package *rp, u64 value, * where time_unit is default to 1 sec. Never 0. */ if (!to_raw) - return (value) ? value * rp->time_unit : rp->time_unit; + return (value) ? value * rd->time_unit : rd->time_unit; - value = div64_u64(value, rp->time_unit); + value = div64_u64(value, rd->time_unit); return value; } +/* TPMI Unit register has different layout */ +#define TPMI_POWER_UNIT_OFFSET POWER_UNIT_OFFSET +#define TPMI_POWER_UNIT_MASK POWER_UNIT_MASK +#define TPMI_ENERGY_UNIT_OFFSET 0x06 +#define TPMI_ENERGY_UNIT_MASK 0x7C0 +#define TPMI_TIME_UNIT_OFFSET 0x0C +#define TPMI_TIME_UNIT_MASK 0xF000 + +static int rapl_check_unit_tpmi(struct rapl_domain *rd) +{ + struct reg_action ra; + u32 value; + + ra.reg = rd->regs[RAPL_DOMAIN_REG_UNIT]; + ra.mask = ~0; + if (rd->rp->priv->read_raw(get_rid(rd->rp), &ra)) { + pr_err("Failed to read power unit REG 0x%llx on %s:%s, exit.\n", + ra.reg, rd->rp->name, rd->name); + return -ENODEV; + } + + value = (ra.value & TPMI_ENERGY_UNIT_MASK) >> TPMI_ENERGY_UNIT_OFFSET; + rd->energy_unit = ENERGY_UNIT_SCALE * 1000000 / (1 << value); + + value = (ra.value & TPMI_POWER_UNIT_MASK) >> TPMI_POWER_UNIT_OFFSET; + rd->power_unit = 1000000 / (1 << value); + + value = (ra.value & TPMI_TIME_UNIT_MASK) >> TPMI_TIME_UNIT_OFFSET; + rd->time_unit = 1000000 / (1 << value); + + pr_debug("Core CPU %s:%s energy=%dpJ, time=%dus, power=%duW\n", + rd->rp->name, rd->name, rd->energy_unit, rd->time_unit, rd->power_unit); + + return 0; +} + +static const struct rapl_defaults defaults_tpmi = { + .check_unit = rapl_check_unit_tpmi, + /* Reuse existing logic, ignore the PL_CLAMP failures and enable all Power Limits */ + .set_floor_freq = set_floor_freq_default, + .compute_time_window = rapl_compute_time_window_core, +}; + static const struct rapl_defaults rapl_defaults_core = { .floor_freq_reg_addr = 0, .check_unit = rapl_check_unit_core, @@ -1159,8 +1292,10 @@ static void rapl_update_domain_data(struct rapl_package *rp) rp->domains[dmn].name); /* exclude non-raw primitives */ for (prim = 0; prim < NR_RAW_PRIMITIVES; prim++) { + struct rapl_primitive_info *rpi = get_rpi(rp, prim); + if (!rapl_read_data_raw(&rp->domains[dmn], prim, - rpi[prim].unit, &val)) + rpi->unit, &val)) rp->domains[dmn].rdd.primitives[prim] = val; } } @@ -1239,7 +1374,7 @@ err_cleanup: return ret; } -static int rapl_check_domain(int cpu, int domain, struct rapl_package *rp) +static int rapl_check_domain(int domain, struct rapl_package *rp) { struct reg_action ra; @@ -1260,12 +1395,46 @@ static int rapl_check_domain(int cpu, int domain, struct rapl_package *rp) */ ra.mask = ENERGY_STATUS_MASK; - if (rp->priv->read_raw(cpu, &ra) || !ra.value) + if (rp->priv->read_raw(get_rid(rp), &ra) || !ra.value) return -ENODEV; return 0; } +/* + * Get per domain energy/power/time unit. + * RAPL Interfaces without per domain unit register will use the package + * scope unit register to set per domain units. + */ +static int rapl_get_domain_unit(struct rapl_domain *rd) +{ + struct rapl_defaults *defaults = get_defaults(rd->rp); + int ret; + + if (!rd->regs[RAPL_DOMAIN_REG_UNIT]) { + if (!rd->rp->priv->reg_unit) { + pr_err("No valid Unit register found\n"); + return -ENODEV; + } + rd->regs[RAPL_DOMAIN_REG_UNIT] = rd->rp->priv->reg_unit; + } + + if (!defaults->check_unit) { + pr_err("missing .check_unit() callback\n"); + return -ENODEV; + } + + ret = defaults->check_unit(rd); + if (ret) + return ret; + + if (rd->id == RAPL_DOMAIN_DRAM && defaults->dram_domain_energy_unit) + rd->energy_unit = defaults->dram_domain_energy_unit; + if (rd->id == RAPL_DOMAIN_PLATFORM && defaults->psys_domain_energy_unit) + rd->energy_unit = defaults->psys_domain_energy_unit; + return 0; +} + /* * Check if power limits are available. Two cases when they are not available: * 1. Locked by BIOS, in this case we still provide read-only access so that @@ -1280,19 +1449,16 @@ static void rapl_detect_powerlimit(struct rapl_domain *rd) u64 val64; int i; - /* check if the domain is locked by BIOS, ignore if MSR doesn't exist */ - if (!rapl_read_data_raw(rd, FW_LOCK, false, &val64)) { - if (val64) { - pr_info("RAPL %s domain %s locked by BIOS\n", - rd->rp->name, rd->name); - rd->state |= DOMAIN_STATE_BIOS_LOCKED; + for (i = POWER_LIMIT1; i < NR_POWER_LIMITS; i++) { + if (!rapl_read_pl_data(rd, i, PL_LOCK, false, &val64)) { + if (val64) { + rd->rpl[i].locked = true; + pr_info("%s:%s:%s locked by BIOS\n", + rd->rp->name, rd->name, pl_names[i]); + } } - } - /* check if power limit MSR exists, otherwise domain is monitoring only */ - for (i = 0; i < NR_POWER_LIMITS; i++) { - int prim = rd->rpl[i].prim_id; - if (rapl_read_data_raw(rd, prim, false, &val64)) + if (rapl_read_pl_data(rd, i, PL_ENABLE, false, &val64)) rd->rpl[i].name = NULL; } } @@ -1300,14 +1466,14 @@ static void rapl_detect_powerlimit(struct rapl_domain *rd) /* Detect active and valid domains for the given CPU, caller must * ensure the CPU belongs to the targeted package and CPU hotlug is disabled. */ -static int rapl_detect_domains(struct rapl_package *rp, int cpu) +static int rapl_detect_domains(struct rapl_package *rp) { struct rapl_domain *rd; int i; for (i = 0; i < RAPL_DOMAIN_MAX; i++) { /* use physical package id to read counters */ - if (!rapl_check_domain(cpu, i, rp)) { + if (!rapl_check_domain(i, rp)) { rp->domain_map |= 1 << i; pr_info("Found RAPL domain %s\n", rapl_domain_names[i]); } @@ -1326,8 +1492,10 @@ static int rapl_detect_domains(struct rapl_package *rp, int cpu) rapl_init_domains(rp); - for (rd = rp->domains; rd < rp->domains + rp->nr_domains; rd++) + for (rd = rp->domains; rd < rp->domains + rp->nr_domains; rd++) { + rapl_get_domain_unit(rd); rapl_detect_powerlimit(rd); + } return 0; } @@ -1340,13 +1508,13 @@ void rapl_remove_package(struct rapl_package *rp) package_power_limit_irq_restore(rp); for (rd = rp->domains; rd < rp->domains + rp->nr_domains; rd++) { - rapl_write_data_raw(rd, PL1_ENABLE, 0); - rapl_write_data_raw(rd, PL1_CLAMP, 0); - if (find_nr_power_limit(rd) > 1) { - rapl_write_data_raw(rd, PL2_ENABLE, 0); - rapl_write_data_raw(rd, PL2_CLAMP, 0); - rapl_write_data_raw(rd, PL4_ENABLE, 0); + int i; + + for (i = POWER_LIMIT1; i < NR_POWER_LIMITS; i++) { + rapl_write_pl_data(rd, i, PL_ENABLE, 0); + rapl_write_pl_data(rd, i, PL_CLAMP, 0); } + if (rd->id == RAPL_DOMAIN_PACKAGE) { rd_package = rd; continue; @@ -1365,13 +1533,18 @@ void rapl_remove_package(struct rapl_package *rp) EXPORT_SYMBOL_GPL(rapl_remove_package); /* caller to ensure CPU hotplug lock is held */ -struct rapl_package *rapl_find_package_domain(int cpu, struct rapl_if_priv *priv) +struct rapl_package *rapl_find_package_domain(int id, struct rapl_if_priv *priv, bool id_is_cpu) { - int id = topology_logical_die_id(cpu); struct rapl_package *rp; + int uid; + + if (id_is_cpu) + uid = topology_logical_die_id(id); + else + uid = id; list_for_each_entry(rp, &rapl_packages, plist) { - if (rp->id == id + if (rp->id == uid && rp->priv->control_type == priv->control_type) return rp; } @@ -1381,34 +1554,37 @@ struct rapl_package *rapl_find_package_domain(int cpu, struct rapl_if_priv *priv EXPORT_SYMBOL_GPL(rapl_find_package_domain); /* called from CPU hotplug notifier, hotplug lock held */ -struct rapl_package *rapl_add_package(int cpu, struct rapl_if_priv *priv) +struct rapl_package *rapl_add_package(int id, struct rapl_if_priv *priv, bool id_is_cpu) { - int id = topology_logical_die_id(cpu); struct rapl_package *rp; int ret; - if (!rapl_defaults) - return ERR_PTR(-ENODEV); - rp = kzalloc(sizeof(struct rapl_package), GFP_KERNEL); if (!rp) return ERR_PTR(-ENOMEM); - /* add the new package to the list */ - rp->id = id; - rp->lead_cpu = cpu; - rp->priv = priv; + if (id_is_cpu) { + rp->id = topology_logical_die_id(id); + rp->lead_cpu = id; + if (topology_max_die_per_package() > 1) + snprintf(rp->name, PACKAGE_DOMAIN_NAME_LENGTH, "package-%d-die-%d", + topology_physical_package_id(id), topology_die_id(id)); + else + snprintf(rp->name, PACKAGE_DOMAIN_NAME_LENGTH, "package-%d", + topology_physical_package_id(id)); + } else { + rp->id = id; + rp->lead_cpu = -1; + snprintf(rp->name, PACKAGE_DOMAIN_NAME_LENGTH, "package-%d", id); + } - if (topology_max_die_per_package() > 1) - snprintf(rp->name, PACKAGE_DOMAIN_NAME_LENGTH, - "package-%d-die-%d", - topology_physical_package_id(cpu), topology_die_id(cpu)); - else - snprintf(rp->name, PACKAGE_DOMAIN_NAME_LENGTH, "package-%d", - topology_physical_package_id(cpu)); + rp->priv = priv; + ret = rapl_config(rp); + if (ret) + goto err_free_package; /* check if the package contains valid domains */ - if (rapl_detect_domains(rp, cpu) || rapl_defaults->check_unit(rp, cpu)) { + if (rapl_detect_domains(rp)) { ret = -ENODEV; goto err_free_package; } @@ -1430,38 +1606,18 @@ static void power_limit_state_save(void) { struct rapl_package *rp; struct rapl_domain *rd; - int nr_pl, ret, i; + int ret, i; cpus_read_lock(); list_for_each_entry(rp, &rapl_packages, plist) { if (!rp->power_zone) continue; rd = power_zone_to_rapl_domain(rp->power_zone); - nr_pl = find_nr_power_limit(rd); - for (i = 0; i < nr_pl; i++) { - switch (rd->rpl[i].prim_id) { - case PL1_ENABLE: - ret = rapl_read_data_raw(rd, - POWER_LIMIT1, true, + for (i = POWER_LIMIT1; i < NR_POWER_LIMITS; i++) { + ret = rapl_read_pl_data(rd, i, PL_LIMIT, true, &rd->rpl[i].last_power_limit); - if (ret) - rd->rpl[i].last_power_limit = 0; - break; - case PL2_ENABLE: - ret = rapl_read_data_raw(rd, - POWER_LIMIT2, true, - &rd->rpl[i].last_power_limit); - if (ret) - rd->rpl[i].last_power_limit = 0; - break; - case PL4_ENABLE: - ret = rapl_read_data_raw(rd, - POWER_LIMIT4, true, - &rd->rpl[i].last_power_limit); - if (ret) - rd->rpl[i].last_power_limit = 0; - break; - } + if (ret) + rd->rpl[i].last_power_limit = 0; } } cpus_read_unlock(); @@ -1471,33 +1627,17 @@ static void power_limit_state_restore(void) { struct rapl_package *rp; struct rapl_domain *rd; - int nr_pl, i; + int i; cpus_read_lock(); list_for_each_entry(rp, &rapl_packages, plist) { if (!rp->power_zone) continue; rd = power_zone_to_rapl_domain(rp->power_zone); - nr_pl = find_nr_power_limit(rd); - for (i = 0; i < nr_pl; i++) { - switch (rd->rpl[i].prim_id) { - case PL1_ENABLE: - if (rd->rpl[i].last_power_limit) - rapl_write_data_raw(rd, POWER_LIMIT1, - rd->rpl[i].last_power_limit); - break; - case PL2_ENABLE: - if (rd->rpl[i].last_power_limit) - rapl_write_data_raw(rd, POWER_LIMIT2, - rd->rpl[i].last_power_limit); - break; - case PL4_ENABLE: - if (rd->rpl[i].last_power_limit) - rapl_write_data_raw(rd, POWER_LIMIT4, - rd->rpl[i].last_power_limit); - break; - } - } + for (i = POWER_LIMIT1; i < NR_POWER_LIMITS; i++) + if (rd->rpl[i].last_power_limit) + rapl_write_pl_data(rd, i, PL_LIMIT, + rd->rpl[i].last_power_limit); } cpus_read_unlock(); } @@ -1528,32 +1668,25 @@ static int __init rapl_init(void) int ret; id = x86_match_cpu(rapl_ids); - if (!id) { - pr_err("driver does not support CPU family %d model %d\n", - boot_cpu_data.x86, boot_cpu_data.x86_model); + if (id) { + defaults_msr = (struct rapl_defaults *)id->driver_data; - return -ENODEV; + rapl_msr_platdev = platform_device_alloc("intel_rapl_msr", 0); + if (!rapl_msr_platdev) + return -ENOMEM; + + ret = platform_device_add(rapl_msr_platdev); + if (ret) { + platform_device_put(rapl_msr_platdev); + return ret; + } } - rapl_defaults = (struct rapl_defaults *)id->driver_data; - ret = register_pm_notifier(&rapl_pm_notifier); - if (ret) - return ret; - - rapl_msr_platdev = platform_device_alloc("intel_rapl_msr", 0); - if (!rapl_msr_platdev) { - ret = -ENOMEM; - goto end; - } - - ret = platform_device_add(rapl_msr_platdev); - if (ret) + if (ret && rapl_msr_platdev) { + platform_device_del(rapl_msr_platdev); platform_device_put(rapl_msr_platdev); - -end: - if (ret) - unregister_pm_notifier(&rapl_pm_notifier); + } return ret; } diff --git a/drivers/powercap/intel_rapl_msr.c b/drivers/powercap/intel_rapl_msr.c index a27673706c3d..569e25eab1e1 100644 --- a/drivers/powercap/intel_rapl_msr.c +++ b/drivers/powercap/intel_rapl_msr.c @@ -22,7 +22,6 @@ #include #include -#include #include #include @@ -34,6 +33,7 @@ static struct rapl_if_priv *rapl_msr_priv; static struct rapl_if_priv rapl_msr_priv_intel = { + .type = RAPL_IF_MSR, .reg_unit = MSR_RAPL_POWER_UNIT, .regs[RAPL_DOMAIN_PACKAGE] = { MSR_PKG_POWER_LIMIT, MSR_PKG_ENERGY_STATUS, MSR_PKG_PERF_STATUS, 0, MSR_PKG_POWER_INFO }, @@ -45,11 +45,12 @@ static struct rapl_if_priv rapl_msr_priv_intel = { MSR_DRAM_POWER_LIMIT, MSR_DRAM_ENERGY_STATUS, MSR_DRAM_PERF_STATUS, 0, MSR_DRAM_POWER_INFO }, .regs[RAPL_DOMAIN_PLATFORM] = { MSR_PLATFORM_POWER_LIMIT, MSR_PLATFORM_ENERGY_STATUS, 0, 0, 0}, - .limits[RAPL_DOMAIN_PACKAGE] = 2, - .limits[RAPL_DOMAIN_PLATFORM] = 2, + .limits[RAPL_DOMAIN_PACKAGE] = BIT(POWER_LIMIT2), + .limits[RAPL_DOMAIN_PLATFORM] = BIT(POWER_LIMIT2), }; static struct rapl_if_priv rapl_msr_priv_amd = { + .type = RAPL_IF_MSR, .reg_unit = MSR_AMD_RAPL_POWER_UNIT, .regs[RAPL_DOMAIN_PACKAGE] = { 0, MSR_AMD_PKG_ENERGY_STATUS, 0, 0, 0 }, @@ -68,9 +69,9 @@ static int rapl_cpu_online(unsigned int cpu) { struct rapl_package *rp; - rp = rapl_find_package_domain(cpu, rapl_msr_priv); + rp = rapl_find_package_domain(cpu, rapl_msr_priv, true); if (!rp) { - rp = rapl_add_package(cpu, rapl_msr_priv); + rp = rapl_add_package(cpu, rapl_msr_priv, true); if (IS_ERR(rp)) return PTR_ERR(rp); } @@ -83,7 +84,7 @@ static int rapl_cpu_down_prep(unsigned int cpu) struct rapl_package *rp; int lead_cpu; - rp = rapl_find_package_domain(cpu, rapl_msr_priv); + rp = rapl_find_package_domain(cpu, rapl_msr_priv, true); if (!rp) return 0; @@ -137,14 +138,14 @@ static int rapl_msr_write_raw(int cpu, struct reg_action *ra) /* List of verified CPUs. */ static const struct x86_cpu_id pl4_support_ids[] = { - { X86_VENDOR_INTEL, 6, INTEL_FAM6_TIGERLAKE_L, X86_FEATURE_ANY }, - { X86_VENDOR_INTEL, 6, INTEL_FAM6_ALDERLAKE, X86_FEATURE_ANY }, - { X86_VENDOR_INTEL, 6, INTEL_FAM6_ALDERLAKE_L, X86_FEATURE_ANY }, - { X86_VENDOR_INTEL, 6, INTEL_FAM6_ALDERLAKE_N, X86_FEATURE_ANY }, - { X86_VENDOR_INTEL, 6, INTEL_FAM6_RAPTORLAKE, X86_FEATURE_ANY }, - { X86_VENDOR_INTEL, 6, INTEL_FAM6_RAPTORLAKE_P, X86_FEATURE_ANY }, - { X86_VENDOR_INTEL, 6, INTEL_FAM6_METEORLAKE, X86_FEATURE_ANY }, - { X86_VENDOR_INTEL, 6, INTEL_FAM6_METEORLAKE_L, X86_FEATURE_ANY }, + X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE_L, NULL), + X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE, NULL), + X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE_L, NULL), + X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE_N, NULL), + X86_MATCH_INTEL_FAM6_MODEL(RAPTORLAKE, NULL), + X86_MATCH_INTEL_FAM6_MODEL(RAPTORLAKE_P, NULL), + X86_MATCH_INTEL_FAM6_MODEL(METEORLAKE, NULL), + X86_MATCH_INTEL_FAM6_MODEL(METEORLAKE_L, NULL), {} }; @@ -169,7 +170,7 @@ static int rapl_msr_probe(struct platform_device *pdev) rapl_msr_priv->write_raw = rapl_msr_write_raw; if (id) { - rapl_msr_priv->limits[RAPL_DOMAIN_PACKAGE] = 3; + rapl_msr_priv->limits[RAPL_DOMAIN_PACKAGE] |= BIT(POWER_LIMIT4); rapl_msr_priv->regs[RAPL_DOMAIN_PACKAGE][RAPL_DOMAIN_REG_PL4] = MSR_VR_CURRENT_CONFIG; pr_info("PL4 support detected.\n"); diff --git a/drivers/powercap/intel_rapl_tpmi.c b/drivers/powercap/intel_rapl_tpmi.c new file mode 100644 index 000000000000..4f4f13ded225 --- /dev/null +++ b/drivers/powercap/intel_rapl_tpmi.c @@ -0,0 +1,325 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * intel_rapl_tpmi: Intel RAPL driver via TPMI interface + * + * Copyright (c) 2023, Intel Corporation. + * All Rights Reserved. + * + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include + +#define TPMI_RAPL_VERSION 1 + +/* 1 header + 10 registers + 5 reserved. 8 bytes for each. */ +#define TPMI_RAPL_DOMAIN_SIZE 128 + +enum tpmi_rapl_domain_type { + TPMI_RAPL_DOMAIN_INVALID, + TPMI_RAPL_DOMAIN_SYSTEM, + TPMI_RAPL_DOMAIN_PACKAGE, + TPMI_RAPL_DOMAIN_RESERVED, + TPMI_RAPL_DOMAIN_MEMORY, + TPMI_RAPL_DOMAIN_MAX, +}; + +enum tpmi_rapl_register { + TPMI_RAPL_REG_HEADER, + TPMI_RAPL_REG_UNIT, + TPMI_RAPL_REG_PL1, + TPMI_RAPL_REG_PL2, + TPMI_RAPL_REG_PL3, + TPMI_RAPL_REG_PL4, + TPMI_RAPL_REG_RESERVED, + TPMI_RAPL_REG_ENERGY_STATUS, + TPMI_RAPL_REG_PERF_STATUS, + TPMI_RAPL_REG_POWER_INFO, + TPMI_RAPL_REG_INTERRUPT, + TPMI_RAPL_REG_MAX = 15, +}; + +struct tpmi_rapl_package { + struct rapl_if_priv priv; + struct intel_tpmi_plat_info *tpmi_info; + struct rapl_package *rp; + void __iomem *base; + struct list_head node; +}; + +static LIST_HEAD(tpmi_rapl_packages); +static DEFINE_MUTEX(tpmi_rapl_lock); + +static struct powercap_control_type *tpmi_control_type; + +static int tpmi_rapl_read_raw(int id, struct reg_action *ra) +{ + if (!ra->reg) + return -EINVAL; + + ra->value = readq((void __iomem *)ra->reg); + + ra->value &= ra->mask; + return 0; +} + +static int tpmi_rapl_write_raw(int id, struct reg_action *ra) +{ + u64 val; + + if (!ra->reg) + return -EINVAL; + + val = readq((void __iomem *)ra->reg); + + val &= ~ra->mask; + val |= ra->value; + + writeq(val, (void __iomem *)ra->reg); + return 0; +} + +static struct tpmi_rapl_package *trp_alloc(int pkg_id) +{ + struct tpmi_rapl_package *trp; + int ret; + + mutex_lock(&tpmi_rapl_lock); + + if (list_empty(&tpmi_rapl_packages)) { + tpmi_control_type = powercap_register_control_type(NULL, "intel-rapl", NULL); + if (IS_ERR(tpmi_control_type)) { + ret = PTR_ERR(tpmi_control_type); + goto err_unlock; + } + } + + trp = kzalloc(sizeof(*trp), GFP_KERNEL); + if (!trp) { + ret = -ENOMEM; + goto err_del_powercap; + } + + list_add(&trp->node, &tpmi_rapl_packages); + + mutex_unlock(&tpmi_rapl_lock); + return trp; + +err_del_powercap: + if (list_empty(&tpmi_rapl_packages)) + powercap_unregister_control_type(tpmi_control_type); +err_unlock: + mutex_unlock(&tpmi_rapl_lock); + return ERR_PTR(ret); +} + +static void trp_release(struct tpmi_rapl_package *trp) +{ + mutex_lock(&tpmi_rapl_lock); + list_del(&trp->node); + + if (list_empty(&tpmi_rapl_packages)) + powercap_unregister_control_type(tpmi_control_type); + + kfree(trp); + mutex_unlock(&tpmi_rapl_lock); +} + +static int parse_one_domain(struct tpmi_rapl_package *trp, u32 offset) +{ + u8 tpmi_domain_version; + enum rapl_domain_type domain_type; + enum tpmi_rapl_domain_type tpmi_domain_type; + enum tpmi_rapl_register reg_index; + enum rapl_domain_reg_id reg_id; + int tpmi_domain_size, tpmi_domain_flags; + u64 *tpmi_rapl_regs = trp->base + offset; + u64 tpmi_domain_header = readq((void __iomem *)tpmi_rapl_regs); + + /* Domain Parent bits are ignored for now */ + tpmi_domain_version = tpmi_domain_header & 0xff; + tpmi_domain_type = tpmi_domain_header >> 8 & 0xff; + tpmi_domain_size = tpmi_domain_header >> 16 & 0xff; + tpmi_domain_flags = tpmi_domain_header >> 32 & 0xffff; + + if (tpmi_domain_version != TPMI_RAPL_VERSION) { + pr_warn(FW_BUG "Unsupported version:%d\n", tpmi_domain_version); + return -ENODEV; + } + + /* Domain size: in unit of 128 Bytes */ + if (tpmi_domain_size != 1) { + pr_warn(FW_BUG "Invalid Domain size %d\n", tpmi_domain_size); + return -EINVAL; + } + + /* Unit register and Energy Status register are mandatory for each domain */ + if (!(tpmi_domain_flags & BIT(TPMI_RAPL_REG_UNIT)) || + !(tpmi_domain_flags & BIT(TPMI_RAPL_REG_ENERGY_STATUS))) { + pr_warn(FW_BUG "Invalid Domain flag 0x%x\n", tpmi_domain_flags); + return -EINVAL; + } + + switch (tpmi_domain_type) { + case TPMI_RAPL_DOMAIN_PACKAGE: + domain_type = RAPL_DOMAIN_PACKAGE; + break; + case TPMI_RAPL_DOMAIN_SYSTEM: + domain_type = RAPL_DOMAIN_PLATFORM; + break; + case TPMI_RAPL_DOMAIN_MEMORY: + domain_type = RAPL_DOMAIN_DRAM; + break; + default: + pr_warn(FW_BUG "Unsupported Domain type %d\n", tpmi_domain_type); + return -EINVAL; + } + + if (trp->priv.regs[domain_type][RAPL_DOMAIN_REG_UNIT]) { + pr_warn(FW_BUG "Duplicate Domain type %d\n", tpmi_domain_type); + return -EINVAL; + } + + reg_index = TPMI_RAPL_REG_HEADER; + while (++reg_index != TPMI_RAPL_REG_MAX) { + if (!(tpmi_domain_flags & BIT(reg_index))) + continue; + + switch (reg_index) { + case TPMI_RAPL_REG_UNIT: + reg_id = RAPL_DOMAIN_REG_UNIT; + break; + case TPMI_RAPL_REG_PL1: + reg_id = RAPL_DOMAIN_REG_LIMIT; + trp->priv.limits[domain_type] |= BIT(POWER_LIMIT1); + break; + case TPMI_RAPL_REG_PL2: + reg_id = RAPL_DOMAIN_REG_PL2; + trp->priv.limits[domain_type] |= BIT(POWER_LIMIT2); + break; + case TPMI_RAPL_REG_PL4: + reg_id = RAPL_DOMAIN_REG_PL4; + trp->priv.limits[domain_type] |= BIT(POWER_LIMIT4); + break; + case TPMI_RAPL_REG_ENERGY_STATUS: + reg_id = RAPL_DOMAIN_REG_STATUS; + break; + case TPMI_RAPL_REG_PERF_STATUS: + reg_id = RAPL_DOMAIN_REG_PERF; + break; + case TPMI_RAPL_REG_POWER_INFO: + reg_id = RAPL_DOMAIN_REG_INFO; + break; + default: + continue; + } + trp->priv.regs[domain_type][reg_id] = (u64)&tpmi_rapl_regs[reg_index]; + } + + return 0; +} + +static int intel_rapl_tpmi_probe(struct auxiliary_device *auxdev, + const struct auxiliary_device_id *id) +{ + struct tpmi_rapl_package *trp; + struct intel_tpmi_plat_info *info; + struct resource *res; + u32 offset; + int ret; + + info = tpmi_get_platform_data(auxdev); + if (!info) + return -ENODEV; + + trp = trp_alloc(info->package_id); + if (IS_ERR(trp)) + return PTR_ERR(trp); + + if (tpmi_get_resource_count(auxdev) > 1) { + dev_err(&auxdev->dev, "does not support multiple resources\n"); + ret = -EINVAL; + goto err; + } + + res = tpmi_get_resource_at_index(auxdev, 0); + if (!res) { + dev_err(&auxdev->dev, "can't fetch device resource info\n"); + ret = -EIO; + goto err; + } + + trp->base = devm_ioremap_resource(&auxdev->dev, res); + if (IS_ERR(trp->base)) { + ret = PTR_ERR(trp->base); + goto err; + } + + for (offset = 0; offset < resource_size(res); offset += TPMI_RAPL_DOMAIN_SIZE) { + ret = parse_one_domain(trp, offset); + if (ret) + goto err; + } + + trp->tpmi_info = info; + trp->priv.type = RAPL_IF_TPMI; + trp->priv.read_raw = tpmi_rapl_read_raw; + trp->priv.write_raw = tpmi_rapl_write_raw; + trp->priv.control_type = tpmi_control_type; + + /* RAPL TPMI I/F is per physical package */ + trp->rp = rapl_find_package_domain(info->package_id, &trp->priv, false); + if (trp->rp) { + dev_err(&auxdev->dev, "Domain for Package%d already exists\n", info->package_id); + ret = -EEXIST; + goto err; + } + + trp->rp = rapl_add_package(info->package_id, &trp->priv, false); + if (IS_ERR(trp->rp)) { + dev_err(&auxdev->dev, "Failed to add RAPL Domain for Package%d, %ld\n", + info->package_id, PTR_ERR(trp->rp)); + ret = PTR_ERR(trp->rp); + goto err; + } + + auxiliary_set_drvdata(auxdev, trp); + + return 0; +err: + trp_release(trp); + return ret; +} + +static void intel_rapl_tpmi_remove(struct auxiliary_device *auxdev) +{ + struct tpmi_rapl_package *trp = auxiliary_get_drvdata(auxdev); + + rapl_remove_package(trp->rp); + trp_release(trp); +} + +static const struct auxiliary_device_id intel_rapl_tpmi_ids[] = { + {.name = "intel_vsec.tpmi-rapl" }, + { } +}; + +MODULE_DEVICE_TABLE(auxiliary, intel_rapl_tpmi_ids); + +static struct auxiliary_driver intel_rapl_tpmi_driver = { + .probe = intel_rapl_tpmi_probe, + .remove = intel_rapl_tpmi_remove, + .id_table = intel_rapl_tpmi_ids, +}; + +module_auxiliary_driver(intel_rapl_tpmi_driver) + +MODULE_IMPORT_NS(INTEL_TPMI); + +MODULE_DESCRIPTION("Intel RAPL TPMI Driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/pwm/pwm-atmel.c b/drivers/pwm/pwm-atmel.c index 0c567d9623cd..5f7d286871cf 100644 --- a/drivers/pwm/pwm-atmel.c +++ b/drivers/pwm/pwm-atmel.c @@ -6,7 +6,7 @@ * Bo Shen * * Links to reference manuals for the supported PWM chips can be found in - * Documentation/arm/microchip.rst. + * Documentation/arch/arm/microchip.rst. * * Limitations: * - Periods start with the inactive level. diff --git a/drivers/pwm/pwm-pxa.c b/drivers/pwm/pwm-pxa.c index 46ed668bd141..762429d5647f 100644 --- a/drivers/pwm/pwm-pxa.c +++ b/drivers/pwm/pwm-pxa.c @@ -8,7 +8,7 @@ * eric miao * * Links to reference manuals for some of the supported PWM chips can be found - * in Documentation/arm/marvell.rst. + * in Documentation/arch/arm/marvell.rst. * * Limitations: * - When PWM is stopped, the current PWM period stops abruptly at the next diff --git a/drivers/thermal/Kconfig b/drivers/thermal/Kconfig index 4cd7ab707315..19a4b33cb564 100644 --- a/drivers/thermal/Kconfig +++ b/drivers/thermal/Kconfig @@ -130,6 +130,14 @@ config THERMAL_DEFAULT_GOV_POWER_ALLOCATOR system and device power allocation. This governor can only operate on cooling devices that implement the power API. +config THERMAL_DEFAULT_GOV_BANG_BANG + bool "bang_bang" + depends on THERMAL_GOV_BANG_BANG + help + Use the bang_bang governor as default. This throttles the + devices one step at the time, taking into account the trip + point hysteresis. + endchoice config THERMAL_GOV_FAIR_SHARE diff --git a/drivers/thermal/amlogic_thermal.c b/drivers/thermal/amlogic_thermal.c index 3abc2dcef408..756b218880a7 100644 --- a/drivers/thermal/amlogic_thermal.c +++ b/drivers/thermal/amlogic_thermal.c @@ -282,8 +282,7 @@ static int amlogic_thermal_probe(struct platform_device *pdev) return ret; } - if (devm_thermal_add_hwmon_sysfs(&pdev->dev, pdata->tzd)) - dev_warn(&pdev->dev, "Failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(&pdev->dev, pdata->tzd); ret = amlogic_thermal_initialize(pdata); if (ret) diff --git a/drivers/thermal/armada_thermal.c b/drivers/thermal/armada_thermal.c index 0e8dfa6a7757..9f6dc4fc9112 100644 --- a/drivers/thermal/armada_thermal.c +++ b/drivers/thermal/armada_thermal.c @@ -231,7 +231,7 @@ static void armada380_init(struct platform_device *pdev, regmap_write(priv->syscon, data->syscon_control0_off, reg); } -static void armada_ap806_init(struct platform_device *pdev, +static void armada_ap80x_init(struct platform_device *pdev, struct armada_thermal_priv *priv) { struct armada_thermal_data *data = priv->data; @@ -614,7 +614,7 @@ static const struct armada_thermal_data armada380_data = { }; static const struct armada_thermal_data armada_ap806_data = { - .init = armada_ap806_init, + .init = armada_ap80x_init, .is_valid_bit = BIT(16), .temp_shift = 0, .temp_mask = 0x3ff, @@ -637,6 +637,30 @@ static const struct armada_thermal_data armada_ap806_data = { .cpu_nr = 4, }; +static const struct armada_thermal_data armada_ap807_data = { + .init = armada_ap80x_init, + .is_valid_bit = BIT(16), + .temp_shift = 0, + .temp_mask = 0x3ff, + .thresh_shift = 3, + .hyst_shift = 19, + .hyst_mask = 0x3, + .coef_b = -128900LL, + .coef_m = 394ULL, + .coef_div = 1, + .inverted = true, + .signed_sample = true, + .syscon_control0_off = 0x84, + .syscon_control1_off = 0x88, + .syscon_status_off = 0x8C, + .dfx_irq_cause_off = 0x108, + .dfx_irq_mask_off = 0x10C, + .dfx_overheat_irq = BIT(22), + .dfx_server_irq_mask_off = 0x104, + .dfx_server_irq_en = BIT(1), + .cpu_nr = 4, +}; + static const struct armada_thermal_data armada_cp110_data = { .init = armada_cp110_init, .is_valid_bit = BIT(10), @@ -680,6 +704,10 @@ static const struct of_device_id armada_thermal_id_table[] = { .compatible = "marvell,armada-ap806-thermal", .data = &armada_ap806_data, }, + { + .compatible = "marvell,armada-ap807-thermal", + .data = &armada_ap807_data, + }, { .compatible = "marvell,armada-cp110-thermal", .data = &armada_cp110_data, diff --git a/drivers/thermal/imx8mm_thermal.c b/drivers/thermal/imx8mm_thermal.c index d8005e9ec992..d4b40869c7d7 100644 --- a/drivers/thermal/imx8mm_thermal.c +++ b/drivers/thermal/imx8mm_thermal.c @@ -343,8 +343,7 @@ static int imx8mm_tmu_probe(struct platform_device *pdev) } tmu->sensors[i].hw_id = i; - if (devm_thermal_add_hwmon_sysfs(&pdev->dev, tmu->sensors[i].tzd)) - dev_warn(&pdev->dev, "failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(&pdev->dev, tmu->sensors[i].tzd); } platform_set_drvdata(pdev, tmu); diff --git a/drivers/thermal/imx_sc_thermal.c b/drivers/thermal/imx_sc_thermal.c index 839bb9958f60..8d6b4ef23746 100644 --- a/drivers/thermal/imx_sc_thermal.c +++ b/drivers/thermal/imx_sc_thermal.c @@ -116,8 +116,7 @@ static int imx_sc_thermal_probe(struct platform_device *pdev) return ret; } - if (devm_thermal_add_hwmon_sysfs(&pdev->dev, sensor->tzd)) - dev_warn(&pdev->dev, "failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(&pdev->dev, sensor->tzd); } return 0; diff --git a/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.c b/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.c index 01b80331eab6..dc519a665c18 100644 --- a/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.c +++ b/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.c @@ -203,6 +203,151 @@ end: } EXPORT_SYMBOL(acpi_parse_art); +/* + * acpi_parse_psvt - Passive Table (PSVT) for passive cooling + * + * @handle: ACPI handle of the device which contains PSVT + * @psvt_count: the number of valid entries resulted from parsing PSVT + * @psvtp: pointer to array of psvt entries + * + */ +static int acpi_parse_psvt(acpi_handle handle, int *psvt_count, struct psvt **psvtp) +{ + struct acpi_buffer buffer = { ACPI_ALLOCATE_BUFFER, NULL }; + int nr_bad_entries = 0, revision = 0; + union acpi_object *p; + acpi_status status; + int i, result = 0; + struct psvt *psvts; + + if (!acpi_has_method(handle, "PSVT")) + return -ENODEV; + + status = acpi_evaluate_object(handle, "PSVT", NULL, &buffer); + if (ACPI_FAILURE(status)) + return -ENODEV; + + p = buffer.pointer; + if (!p || (p->type != ACPI_TYPE_PACKAGE)) { + result = -EFAULT; + goto end; + } + + /* first package is the revision number */ + if (p->package.count > 0) { + union acpi_object *prev = &(p->package.elements[0]); + + if (prev->type == ACPI_TYPE_INTEGER) + revision = (int)prev->integer.value; + } else { + result = -EFAULT; + goto end; + } + + /* Support only version 2 */ + if (revision != 2) { + result = -EFAULT; + goto end; + } + + *psvt_count = p->package.count - 1; + if (!*psvt_count) { + result = -EFAULT; + goto end; + } + + psvts = kcalloc(*psvt_count, sizeof(*psvts), GFP_KERNEL); + if (!psvts) { + result = -ENOMEM; + goto end; + } + + /* Start index is 1 because the first package is the revision number */ + for (i = 1; i < p->package.count; i++) { + struct acpi_buffer psvt_int_format = { sizeof("RRNNNNNNNNNN"), "RRNNNNNNNNNN" }; + struct acpi_buffer psvt_str_format = { sizeof("RRNNNNNSNNNN"), "RRNNNNNSNNNN" }; + union acpi_object *package = &(p->package.elements[i]); + struct psvt *psvt = &psvts[i - 1 - nr_bad_entries]; + struct acpi_buffer *psvt_format = &psvt_int_format; + struct acpi_buffer element = { 0, NULL }; + union acpi_object *knob; + struct acpi_device *res; + struct psvt *psvt_ptr; + + element.length = ACPI_ALLOCATE_BUFFER; + element.pointer = NULL; + + if (package->package.count >= ACPI_NR_PSVT_ELEMENTS) { + knob = &(package->package.elements[ACPI_PSVT_CONTROL_KNOB]); + } else { + nr_bad_entries++; + pr_info("PSVT package %d is invalid, ignored\n", i); + continue; + } + + if (knob->type == ACPI_TYPE_STRING) { + psvt_format = &psvt_str_format; + if (knob->string.length > ACPI_LIMIT_STR_MAX_LEN - 1) { + pr_info("PSVT package %d limit string len exceeds max\n", i); + knob->string.length = ACPI_LIMIT_STR_MAX_LEN - 1; + } + } + + status = acpi_extract_package(&(p->package.elements[i]), psvt_format, &element); + if (ACPI_FAILURE(status)) { + nr_bad_entries++; + pr_info("PSVT package %d is invalid, ignored\n", i); + continue; + } + + psvt_ptr = (struct psvt *)element.pointer; + + memcpy(psvt, psvt_ptr, sizeof(*psvt)); + + /* The limit element can be string or U64 */ + psvt->control_knob_type = (u64)knob->type; + + if (knob->type == ACPI_TYPE_STRING) { + memset(&psvt->limit, 0, sizeof(u64)); + strncpy(psvt->limit.string, psvt_ptr->limit.str_ptr, knob->string.length); + } else { + psvt->limit.integer = psvt_ptr->limit.integer; + } + + kfree(element.pointer); + + res = acpi_fetch_acpi_dev(psvt->source); + if (!res) { + nr_bad_entries++; + pr_info("Failed to get source ACPI device\n"); + continue; + } + + res = acpi_fetch_acpi_dev(psvt->target); + if (!res) { + nr_bad_entries++; + pr_info("Failed to get target ACPI device\n"); + continue; + } + } + + /* don't count bad entries */ + *psvt_count -= nr_bad_entries; + + if (!*psvt_count) { + result = -EFAULT; + kfree(psvts); + goto end; + } + + *psvtp = psvts; + + return 0; + +end: + kfree(buffer.pointer); + return result; +} /* get device name from acpi handle */ static void get_single_name(acpi_handle handle, char *name) @@ -289,6 +434,57 @@ free_trt: return ret; } +static int fill_psvt(char __user *ubuf) +{ + int i, ret, count, psvt_len; + union psvt_object *psvt_user; + struct psvt *psvts; + + ret = acpi_parse_psvt(acpi_thermal_rel_handle, &count, &psvts); + if (ret) + return ret; + + psvt_len = count * sizeof(*psvt_user); + + psvt_user = kzalloc(psvt_len, GFP_KERNEL); + if (!psvt_user) { + ret = -ENOMEM; + goto free_psvt; + } + + /* now fill in user psvt data */ + for (i = 0; i < count; i++) { + /* userspace psvt needs device name instead of acpi reference */ + get_single_name(psvts[i].source, psvt_user[i].source_device); + get_single_name(psvts[i].target, psvt_user[i].target_device); + + psvt_user[i].priority = psvts[i].priority; + psvt_user[i].sample_period = psvts[i].sample_period; + psvt_user[i].passive_temp = psvts[i].passive_temp; + psvt_user[i].source_domain = psvts[i].source_domain; + psvt_user[i].control_knob = psvts[i].control_knob; + psvt_user[i].step_size = psvts[i].step_size; + psvt_user[i].limit_coeff = psvts[i].limit_coeff; + psvt_user[i].unlimit_coeff = psvts[i].unlimit_coeff; + psvt_user[i].control_knob_type = psvts[i].control_knob_type; + if (psvt_user[i].control_knob_type == ACPI_TYPE_STRING) + strncpy(psvt_user[i].limit.string, psvts[i].limit.string, + ACPI_LIMIT_STR_MAX_LEN); + else + psvt_user[i].limit.integer = psvts[i].limit.integer; + + } + + if (copy_to_user(ubuf, psvt_user, psvt_len)) + ret = -EFAULT; + + kfree(psvt_user); + +free_psvt: + kfree(psvts); + return ret; +} + static long acpi_thermal_rel_ioctl(struct file *f, unsigned int cmd, unsigned long __arg) { @@ -298,6 +494,7 @@ static long acpi_thermal_rel_ioctl(struct file *f, unsigned int cmd, char __user *arg = (void __user *)__arg; struct trt *trts = NULL; struct art *arts = NULL; + struct psvt *psvts; switch (cmd) { case ACPI_THERMAL_GET_TRT_COUNT: @@ -336,6 +533,27 @@ static long acpi_thermal_rel_ioctl(struct file *f, unsigned int cmd, case ACPI_THERMAL_GET_ART: return fill_art(arg); + case ACPI_THERMAL_GET_PSVT_COUNT: + ret = acpi_parse_psvt(acpi_thermal_rel_handle, &count, &psvts); + if (!ret) { + kfree(psvts); + return put_user(count, (unsigned long __user *)__arg); + } + return ret; + + case ACPI_THERMAL_GET_PSVT_LEN: + /* total length of the data retrieved (count * PSVT entry size) */ + ret = acpi_parse_psvt(acpi_thermal_rel_handle, &count, &psvts); + length = count * sizeof(union psvt_object); + if (!ret) { + kfree(psvts); + return put_user(length, (unsigned long __user *)__arg); + } + return ret; + + case ACPI_THERMAL_GET_PSVT: + return fill_psvt(arg); + default: return -ENOTTY; } diff --git a/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.h b/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.h index 78d942477035..ac376d8f9ee4 100644 --- a/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.h +++ b/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.h @@ -14,6 +14,16 @@ #define ACPI_THERMAL_GET_TRT _IOR(ACPI_THERMAL_MAGIC, 5, unsigned long) #define ACPI_THERMAL_GET_ART _IOR(ACPI_THERMAL_MAGIC, 6, unsigned long) +/* + * ACPI_THERMAL_GET_PSVT_COUNT = Number of PSVT entries + * ACPI_THERMAL_GET_PSVT_LEN = Total return data size (PSVT count x each + * PSVT entry size) + * ACPI_THERMAL_GET_PSVT = Get the data as an array of psvt_objects + */ +#define ACPI_THERMAL_GET_PSVT_LEN _IOR(ACPI_THERMAL_MAGIC, 7, unsigned long) +#define ACPI_THERMAL_GET_PSVT_COUNT _IOR(ACPI_THERMAL_MAGIC, 8, unsigned long) +#define ACPI_THERMAL_GET_PSVT _IOR(ACPI_THERMAL_MAGIC, 9, unsigned long) + struct art { acpi_handle source; acpi_handle target; @@ -43,6 +53,32 @@ struct trt { u64 reserved4; } __packed; +#define ACPI_NR_PSVT_ELEMENTS 12 +#define ACPI_PSVT_CONTROL_KNOB 7 +#define ACPI_LIMIT_STR_MAX_LEN 8 + +struct psvt { + acpi_handle source; + acpi_handle target; + u64 priority; + u64 sample_period; + u64 passive_temp; + u64 source_domain; + u64 control_knob; + union { + /* For limit_type = ACPI_TYPE_INTEGER */ + u64 integer; + /* For limit_type = ACPI_TYPE_STRING */ + char string[ACPI_LIMIT_STR_MAX_LEN]; + char *str_ptr; + } limit; + u64 step_size; + u64 limit_coeff; + u64 unlimit_coeff; + /* Spec calls this field reserved, so we borrow it for type info */ + u64 control_knob_type; /* ACPI_TYPE_STRING or ACPI_TYPE_INTEGER */ +} __packed; + #define ACPI_NR_ART_ELEMENTS 13 /* for usrspace */ union art_object { @@ -77,6 +113,27 @@ union trt_object { u64 __data[8]; }; +union psvt_object { + struct { + char source_device[8]; + char target_device[8]; + u64 priority; + u64 sample_period; + u64 passive_temp; + u64 source_domain; + u64 control_knob; + union { + u64 integer; + char string[ACPI_LIMIT_STR_MAX_LEN]; + } limit; + u64 step_size; + u64 limit_coeff; + u64 unlimit_coeff; + u64 control_knob_type; + }; + u64 __data[ACPI_NR_PSVT_ELEMENTS]; +}; + #ifdef __KERNEL__ int acpi_thermal_rel_misc_device_add(acpi_handle handle); int acpi_thermal_rel_misc_device_remove(acpi_handle handle); diff --git a/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.c b/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.c index a205221ec8df..013f1633f082 100644 --- a/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.c +++ b/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.c @@ -15,8 +15,8 @@ static const struct rapl_mmio_regs rapl_mmio_default = { .reg_unit = 0x5938, .regs[RAPL_DOMAIN_PACKAGE] = { 0x59a0, 0x593c, 0x58f0, 0, 0x5930}, .regs[RAPL_DOMAIN_DRAM] = { 0x58e0, 0x58e8, 0x58ec, 0, 0}, - .limits[RAPL_DOMAIN_PACKAGE] = 2, - .limits[RAPL_DOMAIN_DRAM] = 2, + .limits[RAPL_DOMAIN_PACKAGE] = BIT(POWER_LIMIT2), + .limits[RAPL_DOMAIN_DRAM] = BIT(POWER_LIMIT2), }; static int rapl_mmio_cpu_online(unsigned int cpu) @@ -27,9 +27,9 @@ static int rapl_mmio_cpu_online(unsigned int cpu) if (topology_physical_package_id(cpu)) return 0; - rp = rapl_find_package_domain(cpu, &rapl_mmio_priv); + rp = rapl_find_package_domain(cpu, &rapl_mmio_priv, true); if (!rp) { - rp = rapl_add_package(cpu, &rapl_mmio_priv); + rp = rapl_add_package(cpu, &rapl_mmio_priv, true); if (IS_ERR(rp)) return PTR_ERR(rp); } @@ -42,7 +42,7 @@ static int rapl_mmio_cpu_down_prep(unsigned int cpu) struct rapl_package *rp; int lead_cpu; - rp = rapl_find_package_domain(cpu, &rapl_mmio_priv); + rp = rapl_find_package_domain(cpu, &rapl_mmio_priv, true); if (!rp) return 0; @@ -97,6 +97,7 @@ int proc_thermal_rapl_add(struct pci_dev *pdev, struct proc_thermal_device *proc rapl_regs->regs[domain][reg]; rapl_mmio_priv.limits[domain] = rapl_regs->limits[domain]; } + rapl_mmio_priv.type = RAPL_IF_MMIO; rapl_mmio_priv.reg_unit = (u64)proc_priv->mmio_base + rapl_regs->reg_unit; rapl_mmio_priv.read_raw = rapl_mmio_read_raw; diff --git a/drivers/thermal/k3_bandgap.c b/drivers/thermal/k3_bandgap.c index 791210458606..1c3e590157ec 100644 --- a/drivers/thermal/k3_bandgap.c +++ b/drivers/thermal/k3_bandgap.c @@ -222,8 +222,7 @@ static int k3_bandgap_probe(struct platform_device *pdev) goto err_alloc; } - if (devm_thermal_add_hwmon_sysfs(dev, data[id].tzd)) - dev_warn(dev, "Failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(dev, data[id].tzd); } platform_set_drvdata(pdev, bgp); diff --git a/drivers/thermal/mediatek/auxadc_thermal.c b/drivers/thermal/mediatek/auxadc_thermal.c index 0b5528804bbd..f59d36de20a0 100644 --- a/drivers/thermal/mediatek/auxadc_thermal.c +++ b/drivers/thermal/mediatek/auxadc_thermal.c @@ -1222,12 +1222,7 @@ static int mtk_thermal_probe(struct platform_device *pdev) return -ENODEV; } - auxadc_base = devm_of_iomap(&pdev->dev, auxadc, 0, NULL); - if (IS_ERR(auxadc_base)) { - of_node_put(auxadc); - return PTR_ERR(auxadc_base); - } - + auxadc_base = of_iomap(auxadc, 0); auxadc_phys_base = of_get_phys_base(auxadc); of_node_put(auxadc); @@ -1243,12 +1238,7 @@ static int mtk_thermal_probe(struct platform_device *pdev) return -ENODEV; } - apmixed_base = devm_of_iomap(&pdev->dev, apmixedsys, 0, NULL); - if (IS_ERR(apmixed_base)) { - of_node_put(apmixedsys); - return PTR_ERR(apmixed_base); - } - + apmixed_base = of_iomap(apmixedsys, 0); apmixed_phys_base = of_get_phys_base(apmixedsys); of_node_put(apmixedsys); diff --git a/drivers/thermal/mediatek/lvts_thermal.c b/drivers/thermal/mediatek/lvts_thermal.c index d0a3f95b7884..b693fac2d677 100644 --- a/drivers/thermal/mediatek/lvts_thermal.c +++ b/drivers/thermal/mediatek/lvts_thermal.c @@ -19,6 +19,8 @@ #include #include +#include "../thermal_hwmon.h" + #define LVTS_MONCTL0(__base) (__base + 0x0000) #define LVTS_MONCTL1(__base) (__base + 0x0004) #define LVTS_MONCTL2(__base) (__base + 0x0008) @@ -996,6 +998,8 @@ static int lvts_ctrl_start(struct device *dev, struct lvts_ctrl *lvts_ctrl) return PTR_ERR(tz); } + devm_thermal_add_hwmon_sysfs(dev, tz); + /* * The thermal zone pointer will be needed in the * interrupt handler, we store it in the sensor diff --git a/drivers/thermal/qcom/qcom-spmi-adc-tm5.c b/drivers/thermal/qcom/qcom-spmi-adc-tm5.c index 5749149ae2e4..5ddc39b2be32 100644 --- a/drivers/thermal/qcom/qcom-spmi-adc-tm5.c +++ b/drivers/thermal/qcom/qcom-spmi-adc-tm5.c @@ -689,9 +689,7 @@ static int adc_tm5_register_tzd(struct adc_tm5_chip *adc_tm) return PTR_ERR(tzd); } adc_tm->channels[i].tzd = tzd; - if (devm_thermal_add_hwmon_sysfs(adc_tm->dev, tzd)) - dev_warn(adc_tm->dev, - "Failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(adc_tm->dev, tzd); } return 0; diff --git a/drivers/thermal/qcom/qcom-spmi-temp-alarm.c b/drivers/thermal/qcom/qcom-spmi-temp-alarm.c index 0f88e98428ac..0e8ebfcd84c5 100644 --- a/drivers/thermal/qcom/qcom-spmi-temp-alarm.c +++ b/drivers/thermal/qcom/qcom-spmi-temp-alarm.c @@ -411,22 +411,19 @@ static int qpnp_tm_probe(struct platform_device *pdev) chip->base = res; ret = qpnp_tm_read(chip, QPNP_TM_REG_TYPE, &type); - if (ret < 0) { - dev_err(&pdev->dev, "could not read type\n"); - return ret; - } + if (ret < 0) + return dev_err_probe(&pdev->dev, ret, + "could not read type\n"); ret = qpnp_tm_read(chip, QPNP_TM_REG_SUBTYPE, &subtype); - if (ret < 0) { - dev_err(&pdev->dev, "could not read subtype\n"); - return ret; - } + if (ret < 0) + return dev_err_probe(&pdev->dev, ret, + "could not read subtype\n"); ret = qpnp_tm_read(chip, QPNP_TM_REG_DIG_MAJOR, &dig_major); - if (ret < 0) { - dev_err(&pdev->dev, "could not read dig_major\n"); - return ret; - } + if (ret < 0) + return dev_err_probe(&pdev->dev, ret, + "could not read dig_major\n"); if (type != QPNP_TM_TYPE || (subtype != QPNP_TM_SUBTYPE_GEN1 && subtype != QPNP_TM_SUBTYPE_GEN2)) { @@ -448,20 +445,15 @@ static int qpnp_tm_probe(struct platform_device *pdev) */ chip->tz_dev = devm_thermal_of_zone_register( &pdev->dev, 0, chip, &qpnp_tm_sensor_ops); - if (IS_ERR(chip->tz_dev)) { - dev_err(&pdev->dev, "failed to register sensor\n"); - return PTR_ERR(chip->tz_dev); - } + if (IS_ERR(chip->tz_dev)) + return dev_err_probe(&pdev->dev, PTR_ERR(chip->tz_dev), + "failed to register sensor\n"); ret = qpnp_tm_init(chip); - if (ret < 0) { - dev_err(&pdev->dev, "init failed\n"); - return ret; - } + if (ret < 0) + return dev_err_probe(&pdev->dev, ret, "init failed\n"); - if (devm_thermal_add_hwmon_sysfs(&pdev->dev, chip->tz_dev)) - dev_warn(&pdev->dev, - "Failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(&pdev->dev, chip->tz_dev); ret = devm_request_threaded_irq(&pdev->dev, irq, NULL, qpnp_tm_isr, IRQF_ONESHOT, node->name, chip); diff --git a/drivers/thermal/qcom/tsens-v0_1.c b/drivers/thermal/qcom/tsens-v0_1.c index e89c6f39a3ae..a941b4241b0a 100644 --- a/drivers/thermal/qcom/tsens-v0_1.c +++ b/drivers/thermal/qcom/tsens-v0_1.c @@ -39,26 +39,6 @@ struct tsens_legacy_calibration_format tsens_8916_nvmem = { }, }; -struct tsens_legacy_calibration_format tsens_8939_nvmem = { - .base_len = 8, - .base_shift = 2, - .sp_len = 6, - .mode = { 12, 0 }, - .invalid = { 12, 2 }, - .base = { { 0, 0 }, { 1, 24 } }, - .sp = { - { { 12, 3 }, { 12, 9 } }, - { { 12, 15 }, { 12, 21 } }, - { { 12, 27 }, { 13, 1 } }, - { { 13, 7 }, { 13, 13 } }, - { { 13, 19 }, { 13, 25 } }, - { { 0, 8 }, { 0, 14 } }, - { { 0, 20 }, { 0, 26 } }, - { { 1, 0 }, { 1, 6 } }, - { { 1, 12 }, { 1, 18 } }, - }, -}; - struct tsens_legacy_calibration_format tsens_8974_nvmem = { .base_len = 8, .base_shift = 2, @@ -103,22 +83,6 @@ struct tsens_legacy_calibration_format tsens_8974_backup_nvmem = { }, }; -struct tsens_legacy_calibration_format tsens_9607_nvmem = { - .base_len = 8, - .base_shift = 2, - .sp_len = 6, - .mode = { 2, 20 }, - .invalid = { 2, 22 }, - .base = { { 0, 0 }, { 2, 12 } }, - .sp = { - { { 0, 8 }, { 0, 14 } }, - { { 0, 20 }, { 0, 26 } }, - { { 1, 0 }, { 1, 6 } }, - { { 1, 12 }, { 1, 18 } }, - { { 2, 0 }, { 2, 6 } }, - }, -}; - static int calibrate_8916(struct tsens_priv *priv) { u32 p1[5], p2[5]; @@ -243,6 +207,39 @@ static int calibrate_8974(struct tsens_priv *priv) return 0; } +static int __init init_8226(struct tsens_priv *priv) +{ + priv->sensor[0].slope = 2901; + priv->sensor[1].slope = 2846; + priv->sensor[2].slope = 3038; + priv->sensor[3].slope = 2955; + priv->sensor[4].slope = 2901; + priv->sensor[5].slope = 2846; + + return init_common(priv); +} + +static int __init init_8909(struct tsens_priv *priv) +{ + int i; + + for (i = 0; i < priv->num_sensors; ++i) + priv->sensor[i].slope = 3000; + + priv->sensor[0].p1_calib_offset = 0; + priv->sensor[0].p2_calib_offset = 0; + priv->sensor[1].p1_calib_offset = -10; + priv->sensor[1].p2_calib_offset = -6; + priv->sensor[2].p1_calib_offset = 0; + priv->sensor[2].p2_calib_offset = 0; + priv->sensor[3].p1_calib_offset = -9; + priv->sensor[3].p2_calib_offset = -9; + priv->sensor[4].p1_calib_offset = -8; + priv->sensor[4].p2_calib_offset = -10; + + return init_common(priv); +} + static int __init init_8939(struct tsens_priv *priv) { priv->sensor[0].slope = 2911; priv->sensor[1].slope = 2789; @@ -258,7 +255,28 @@ static int __init init_8939(struct tsens_priv *priv) { return init_common(priv); } -/* v0.1: 8916, 8939, 8974, 9607 */ +static int __init init_9607(struct tsens_priv *priv) +{ + int i; + + for (i = 0; i < priv->num_sensors; ++i) + priv->sensor[i].slope = 3000; + + priv->sensor[0].p1_calib_offset = 1; + priv->sensor[0].p2_calib_offset = 1; + priv->sensor[1].p1_calib_offset = -4; + priv->sensor[1].p2_calib_offset = -2; + priv->sensor[2].p1_calib_offset = 4; + priv->sensor[2].p2_calib_offset = 8; + priv->sensor[3].p1_calib_offset = -3; + priv->sensor[3].p2_calib_offset = -5; + priv->sensor[4].p1_calib_offset = -4; + priv->sensor[4].p2_calib_offset = -4; + + return init_common(priv); +} + +/* v0.1: 8226, 8909, 8916, 8939, 8974, 9607 */ static struct tsens_features tsens_v0_1_feat = { .ver_major = VER_0_1, @@ -313,6 +331,32 @@ static const struct tsens_ops ops_v0_1 = { .get_temp = get_temp_common, }; +static const struct tsens_ops ops_8226 = { + .init = init_8226, + .calibrate = tsens_calibrate_common, + .get_temp = get_temp_common, +}; + +struct tsens_plat_data data_8226 = { + .num_sensors = 6, + .ops = &ops_8226, + .feat = &tsens_v0_1_feat, + .fields = tsens_v0_1_regfields, +}; + +static const struct tsens_ops ops_8909 = { + .init = init_8909, + .calibrate = tsens_calibrate_common, + .get_temp = get_temp_common, +}; + +struct tsens_plat_data data_8909 = { + .num_sensors = 5, + .ops = &ops_8909, + .feat = &tsens_v0_1_feat, + .fields = tsens_v0_1_regfields, +}; + static const struct tsens_ops ops_8916 = { .init = init_common, .calibrate = calibrate_8916, @@ -356,9 +400,15 @@ struct tsens_plat_data data_8974 = { .fields = tsens_v0_1_regfields, }; +static const struct tsens_ops ops_9607 = { + .init = init_9607, + .calibrate = tsens_calibrate_common, + .get_temp = get_temp_common, +}; + struct tsens_plat_data data_9607 = { .num_sensors = 5, - .ops = &ops_v0_1, + .ops = &ops_9607, .feat = &tsens_v0_1_feat, .fields = tsens_v0_1_regfields, }; diff --git a/drivers/thermal/qcom/tsens-v1.c b/drivers/thermal/qcom/tsens-v1.c index b822a426066d..51322430f1fe 100644 --- a/drivers/thermal/qcom/tsens-v1.c +++ b/drivers/thermal/qcom/tsens-v1.c @@ -42,28 +42,6 @@ struct tsens_legacy_calibration_format tsens_qcs404_nvmem = { }, }; -struct tsens_legacy_calibration_format tsens_8976_nvmem = { - .base_len = 8, - .base_shift = 2, - .sp_len = 6, - .mode = { 4, 0 }, - .invalid = { 4, 2 }, - .base = { { 0, 0 }, { 2, 8 } }, - .sp = { - { { 0, 8 }, { 0, 14 } }, - { { 0, 20 }, { 0, 26 } }, - { { 1, 0 }, { 1, 6 } }, - { { 1, 12 }, { 1, 18 } }, - { { 2, 8 }, { 2, 14 } }, - { { 2, 20 }, { 2, 26 } }, - { { 3, 0 }, { 3, 6 } }, - { { 3, 12 }, { 3, 18 } }, - { { 4, 2 }, { 4, 9 } }, - { { 4, 14 }, { 4, 21 } }, - { { 4, 26 }, { 5, 1 } }, - }, -}; - static int calibrate_v1(struct tsens_priv *priv) { u32 p1[10], p2[10]; diff --git a/drivers/thermal/qcom/tsens.c b/drivers/thermal/qcom/tsens.c index d3218127e617..98c356acfe98 100644 --- a/drivers/thermal/qcom/tsens.c +++ b/drivers/thermal/qcom/tsens.c @@ -134,10 +134,12 @@ int tsens_read_calibration(struct tsens_priv *priv, int shift, u32 *p1, u32 *p2, p1[i] = p1[i] + (base1 << shift); break; case TWO_PT_CALIB: + case TWO_PT_CALIB_NO_OFFSET: for (i = 0; i < priv->num_sensors; i++) p2[i] = (p2[i] + base2) << shift; fallthrough; case ONE_PT_CALIB2: + case ONE_PT_CALIB2_NO_OFFSET: for (i = 0; i < priv->num_sensors; i++) p1[i] = (p1[i] + base1) << shift; break; @@ -149,6 +151,18 @@ int tsens_read_calibration(struct tsens_priv *priv, int shift, u32 *p1, u32 *p2, } } + /* Apply calibration offset workaround except for _NO_OFFSET modes */ + switch (mode) { + case TWO_PT_CALIB: + for (i = 0; i < priv->num_sensors; i++) + p2[i] += priv->sensor[i].p2_calib_offset; + fallthrough; + case ONE_PT_CALIB2: + for (i = 0; i < priv->num_sensors; i++) + p1[i] += priv->sensor[i].p1_calib_offset; + break; + } + return mode; } @@ -254,7 +268,7 @@ void compute_intercept_slope(struct tsens_priv *priv, u32 *p1, if (!priv->sensor[i].slope) priv->sensor[i].slope = SLOPE_DEFAULT; - if (mode == TWO_PT_CALIB) { + if (mode == TWO_PT_CALIB || mode == TWO_PT_CALIB_NO_OFFSET) { /* * slope (m) = adc_code2 - adc_code1 (y2 - y1)/ * temp_120_degc - temp_30_degc (x2 - x1) @@ -1095,6 +1109,12 @@ static const struct of_device_id tsens_table[] = { }, { .compatible = "qcom,mdm9607-tsens", .data = &data_9607, + }, { + .compatible = "qcom,msm8226-tsens", + .data = &data_8226, + }, { + .compatible = "qcom,msm8909-tsens", + .data = &data_8909, }, { .compatible = "qcom,msm8916-tsens", .data = &data_8916, @@ -1189,9 +1209,7 @@ static int tsens_register(struct tsens_priv *priv) if (priv->ops->enable) priv->ops->enable(priv, i); - if (devm_thermal_add_hwmon_sysfs(priv->dev, tzd)) - dev_warn(priv->dev, - "Failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(priv->dev, tzd); } /* VER_0 require to set MIN and MAX THRESH diff --git a/drivers/thermal/qcom/tsens.h b/drivers/thermal/qcom/tsens.h index dba9cd38f637..2805de1c6827 100644 --- a/drivers/thermal/qcom/tsens.h +++ b/drivers/thermal/qcom/tsens.h @@ -10,6 +10,8 @@ #define ONE_PT_CALIB 0x1 #define ONE_PT_CALIB2 0x2 #define TWO_PT_CALIB 0x3 +#define ONE_PT_CALIB2_NO_OFFSET 0x6 +#define TWO_PT_CALIB_NO_OFFSET 0x7 #define CAL_DEGC_PT1 30 #define CAL_DEGC_PT2 120 #define SLOPE_FACTOR 1000 @@ -57,6 +59,8 @@ struct tsens_sensor { unsigned int hw_id; int slope; u32 status; + int p1_calib_offset; + int p2_calib_offset; }; /** @@ -635,7 +639,7 @@ int get_temp_common(const struct tsens_sensor *s, int *temp); extern struct tsens_plat_data data_8960; /* TSENS v0.1 targets */ -extern struct tsens_plat_data data_8916, data_8939, data_8974, data_9607; +extern struct tsens_plat_data data_8226, data_8909, data_8916, data_8939, data_8974, data_9607; /* TSENS v1 targets */ extern struct tsens_plat_data data_tsens_v1, data_8976, data_8956; diff --git a/drivers/thermal/qoriq_thermal.c b/drivers/thermal/qoriq_thermal.c index e58756323457..ccc2eea7f9f5 100644 --- a/drivers/thermal/qoriq_thermal.c +++ b/drivers/thermal/qoriq_thermal.c @@ -31,7 +31,6 @@ #define TMR_DISABLE 0x0 #define TMR_ME 0x80000000 #define TMR_ALPF 0x0c000000 -#define TMR_MSITE_ALL GENMASK(15, 0) #define REGS_TMTMIR 0x008 /* Temperature measurement interval Register */ #define TMTMIR_DEFAULT 0x0000000f @@ -51,6 +50,7 @@ * Site Register */ #define TRITSR_V BIT(31) +#define TRITSR_TP5 BIT(9) #define REGS_V2_TMSAR(n) (0x304 + 16 * (n)) /* TMU monitoring * site adjustment register */ @@ -105,6 +105,11 @@ static int tmu_get_temp(struct thermal_zone_device *tz, int *temp) * within sensor range. TEMP is an 9 bit value representing * temperature in KelVin. */ + + regmap_read(qdata->regmap, REGS_TMR, &val); + if (!(val & TMR_ME)) + return -EAGAIN; + if (regmap_read_poll_timeout(qdata->regmap, REGS_TRITSR(qsensor->id), val, @@ -113,10 +118,15 @@ static int tmu_get_temp(struct thermal_zone_device *tz, int *temp) 10 * USEC_PER_MSEC)) return -ENODATA; - if (qdata->ver == TMU_VER1) + if (qdata->ver == TMU_VER1) { *temp = (val & GENMASK(7, 0)) * MILLIDEGREE_PER_DEGREE; - else - *temp = kelvin_to_millicelsius(val & GENMASK(8, 0)); + } else { + if (val & TRITSR_TP5) + *temp = milli_kelvin_to_millicelsius((val & GENMASK(8, 0)) * + MILLIDEGREE_PER_DEGREE + 500); + else + *temp = kelvin_to_millicelsius(val & GENMASK(8, 0)); + } return 0; } @@ -128,15 +138,7 @@ static const struct thermal_zone_device_ops tmu_tz_ops = { static int qoriq_tmu_register_tmu_zone(struct device *dev, struct qoriq_tmu_data *qdata) { - int id; - - if (qdata->ver == TMU_VER1) { - regmap_write(qdata->regmap, REGS_TMR, - TMR_MSITE_ALL | TMR_ME | TMR_ALPF); - } else { - regmap_write(qdata->regmap, REGS_V2_TMSR, TMR_MSITE_ALL); - regmap_write(qdata->regmap, REGS_TMR, TMR_ME | TMR_ALPF_V2); - } + int id, sites = 0; for (id = 0; id < SITES_MAX; id++) { struct thermal_zone_device *tzd; @@ -153,14 +155,24 @@ static int qoriq_tmu_register_tmu_zone(struct device *dev, if (ret == -ENODEV) continue; - regmap_write(qdata->regmap, REGS_TMR, TMR_DISABLE); return ret; } - if (devm_thermal_add_hwmon_sysfs(dev, tzd)) - dev_warn(dev, - "Failed to add hwmon sysfs attributes\n"); + if (qdata->ver == TMU_VER1) + sites |= 0x1 << (15 - id); + else + sites |= 0x1 << id; + devm_thermal_add_hwmon_sysfs(dev, tzd); + } + + if (sites) { + if (qdata->ver == TMU_VER1) { + regmap_write(qdata->regmap, REGS_TMR, TMR_ME | TMR_ALPF | sites); + } else { + regmap_write(qdata->regmap, REGS_V2_TMSR, sites); + regmap_write(qdata->regmap, REGS_TMR, TMR_ME | TMR_ALPF_V2); + } } return 0; @@ -208,8 +220,6 @@ static int qoriq_tmu_calibration(struct device *dev, static void qoriq_tmu_init_device(struct qoriq_tmu_data *data) { - int i; - /* Disable interrupt, using polling instead */ regmap_write(data->regmap, REGS_TIER, TIER_DISABLE); @@ -220,8 +230,6 @@ static void qoriq_tmu_init_device(struct qoriq_tmu_data *data) } else { regmap_write(data->regmap, REGS_V2_TMTMIR, TMTMIR_DEFAULT); regmap_write(data->regmap, REGS_V2_TEUMR(0), TEUMR0_V2); - for (i = 0; i < SITES_MAX; i++) - regmap_write(data->regmap, REGS_V2_TMSAR(i), TMSARA_V2); } /* Disable monitoring */ @@ -230,7 +238,7 @@ static void qoriq_tmu_init_device(struct qoriq_tmu_data *data) static const struct regmap_range qoriq_yes_ranges[] = { regmap_reg_range(REGS_TMR, REGS_TSCFGR), - regmap_reg_range(REGS_TTRnCR(0), REGS_TTRnCR(3)), + regmap_reg_range(REGS_TTRnCR(0), REGS_TTRnCR(15)), regmap_reg_range(REGS_V2_TEUMR(0), REGS_V2_TEUMR(2)), regmap_reg_range(REGS_V2_TMSAR(0), REGS_V2_TMSAR(15)), regmap_reg_range(REGS_IPBRR(0), REGS_IPBRR(1)), diff --git a/drivers/thermal/rcar_gen3_thermal.c b/drivers/thermal/rcar_gen3_thermal.c index 42a4724d3920..9029d01e029b 100644 --- a/drivers/thermal/rcar_gen3_thermal.c +++ b/drivers/thermal/rcar_gen3_thermal.c @@ -35,6 +35,12 @@ #define REG_GEN3_PTAT2 0x60 #define REG_GEN3_PTAT3 0x64 #define REG_GEN3_THSCP 0x68 +#define REG_GEN4_THSFMON00 0x180 +#define REG_GEN4_THSFMON01 0x184 +#define REG_GEN4_THSFMON02 0x188 +#define REG_GEN4_THSFMON15 0x1BC +#define REG_GEN4_THSFMON16 0x1C0 +#define REG_GEN4_THSFMON17 0x1C4 /* IRQ{STR,MSK,EN} bits */ #define IRQ_TEMP1 BIT(0) @@ -55,6 +61,7 @@ #define MCELSIUS(temp) ((temp) * 1000) #define GEN3_FUSE_MASK 0xFFF +#define GEN4_FUSE_MASK 0xFFF #define TSC_MAX_NUM 5 @@ -66,6 +73,13 @@ struct equation_coefs { int b2; }; +struct rcar_gen3_thermal_priv; + +struct rcar_thermal_info { + int ths_tj_1; + void (*read_fuses)(struct rcar_gen3_thermal_priv *priv); +}; + struct rcar_gen3_thermal_tsc { void __iomem *base; struct thermal_zone_device *zone; @@ -79,6 +93,7 @@ struct rcar_gen3_thermal_priv { struct thermal_zone_device_ops ops; unsigned int num_tscs; int ptat[3]; + const struct rcar_thermal_info *info; }; static inline u32 rcar_gen3_thermal_read(struct rcar_gen3_thermal_tsc *tsc, @@ -236,6 +251,62 @@ static irqreturn_t rcar_gen3_thermal_irq(int irq, void *data) return IRQ_HANDLED; } +static void rcar_gen3_thermal_read_fuses_gen3(struct rcar_gen3_thermal_priv *priv) +{ + unsigned int i; + + /* + * Set the pseudo calibration points with fused values. + * PTAT is shared between all TSCs but only fused for the first + * TSC while THCODEs are fused for each TSC. + */ + priv->ptat[0] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN3_PTAT1) & + GEN3_FUSE_MASK; + priv->ptat[1] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN3_PTAT2) & + GEN3_FUSE_MASK; + priv->ptat[2] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN3_PTAT3) & + GEN3_FUSE_MASK; + + for (i = 0; i < priv->num_tscs; i++) { + struct rcar_gen3_thermal_tsc *tsc = priv->tscs[i]; + + tsc->thcode[0] = rcar_gen3_thermal_read(tsc, REG_GEN3_THCODE1) & + GEN3_FUSE_MASK; + tsc->thcode[1] = rcar_gen3_thermal_read(tsc, REG_GEN3_THCODE2) & + GEN3_FUSE_MASK; + tsc->thcode[2] = rcar_gen3_thermal_read(tsc, REG_GEN3_THCODE3) & + GEN3_FUSE_MASK; + } +} + +static void rcar_gen3_thermal_read_fuses_gen4(struct rcar_gen3_thermal_priv *priv) +{ + unsigned int i; + + /* + * Set the pseudo calibration points with fused values. + * PTAT is shared between all TSCs but only fused for the first + * TSC while THCODEs are fused for each TSC. + */ + priv->ptat[0] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN4_THSFMON16) & + GEN4_FUSE_MASK; + priv->ptat[1] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN4_THSFMON17) & + GEN4_FUSE_MASK; + priv->ptat[2] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN4_THSFMON15) & + GEN4_FUSE_MASK; + + for (i = 0; i < priv->num_tscs; i++) { + struct rcar_gen3_thermal_tsc *tsc = priv->tscs[i]; + + tsc->thcode[0] = rcar_gen3_thermal_read(tsc, REG_GEN4_THSFMON01) & + GEN4_FUSE_MASK; + tsc->thcode[1] = rcar_gen3_thermal_read(tsc, REG_GEN4_THSFMON02) & + GEN4_FUSE_MASK; + tsc->thcode[2] = rcar_gen3_thermal_read(tsc, REG_GEN4_THSFMON00) & + GEN4_FUSE_MASK; + } +} + static bool rcar_gen3_thermal_read_fuses(struct rcar_gen3_thermal_priv *priv) { unsigned int i; @@ -243,7 +314,8 @@ static bool rcar_gen3_thermal_read_fuses(struct rcar_gen3_thermal_priv *priv) /* If fuses are not set, fallback to pseudo values. */ thscp = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN3_THSCP); - if ((thscp & THSCP_COR_PARA_VLD) != THSCP_COR_PARA_VLD) { + if (!priv->info->read_fuses || + (thscp & THSCP_COR_PARA_VLD) != THSCP_COR_PARA_VLD) { /* Default THCODE values in case FUSEs are not set. */ static const int thcodes[TSC_MAX_NUM][3] = { { 3397, 2800, 2221 }, @@ -268,29 +340,7 @@ static bool rcar_gen3_thermal_read_fuses(struct rcar_gen3_thermal_priv *priv) return false; } - /* - * Set the pseudo calibration points with fused values. - * PTAT is shared between all TSCs but only fused for the first - * TSC while THCODEs are fused for each TSC. - */ - priv->ptat[0] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN3_PTAT1) & - GEN3_FUSE_MASK; - priv->ptat[1] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN3_PTAT2) & - GEN3_FUSE_MASK; - priv->ptat[2] = rcar_gen3_thermal_read(priv->tscs[0], REG_GEN3_PTAT3) & - GEN3_FUSE_MASK; - - for (i = 0; i < priv->num_tscs; i++) { - struct rcar_gen3_thermal_tsc *tsc = priv->tscs[i]; - - tsc->thcode[0] = rcar_gen3_thermal_read(tsc, REG_GEN3_THCODE1) & - GEN3_FUSE_MASK; - tsc->thcode[1] = rcar_gen3_thermal_read(tsc, REG_GEN3_THCODE2) & - GEN3_FUSE_MASK; - tsc->thcode[2] = rcar_gen3_thermal_read(tsc, REG_GEN3_THCODE3) & - GEN3_FUSE_MASK; - } - + priv->info->read_fuses(priv); return true; } @@ -318,52 +368,65 @@ static void rcar_gen3_thermal_init(struct rcar_gen3_thermal_priv *priv, usleep_range(1000, 2000); } -static const int rcar_gen3_ths_tj_1 = 126; -static const int rcar_gen3_ths_tj_1_m3_w = 116; +static const struct rcar_thermal_info rcar_m3w_thermal_info = { + .ths_tj_1 = 116, + .read_fuses = rcar_gen3_thermal_read_fuses_gen3, +}; + +static const struct rcar_thermal_info rcar_gen3_thermal_info = { + .ths_tj_1 = 126, + .read_fuses = rcar_gen3_thermal_read_fuses_gen3, +}; + +static const struct rcar_thermal_info rcar_gen4_thermal_info = { + .ths_tj_1 = 126, + .read_fuses = rcar_gen3_thermal_read_fuses_gen4, +}; + static const struct of_device_id rcar_gen3_thermal_dt_ids[] = { { .compatible = "renesas,r8a774a1-thermal", - .data = &rcar_gen3_ths_tj_1_m3_w, + .data = &rcar_m3w_thermal_info, }, { .compatible = "renesas,r8a774b1-thermal", - .data = &rcar_gen3_ths_tj_1, + .data = &rcar_gen3_thermal_info, }, { .compatible = "renesas,r8a774e1-thermal", - .data = &rcar_gen3_ths_tj_1, + .data = &rcar_gen3_thermal_info, }, { .compatible = "renesas,r8a7795-thermal", - .data = &rcar_gen3_ths_tj_1, + .data = &rcar_gen3_thermal_info, }, { .compatible = "renesas,r8a7796-thermal", - .data = &rcar_gen3_ths_tj_1_m3_w, + .data = &rcar_m3w_thermal_info, }, { .compatible = "renesas,r8a77961-thermal", - .data = &rcar_gen3_ths_tj_1_m3_w, + .data = &rcar_m3w_thermal_info, }, { .compatible = "renesas,r8a77965-thermal", - .data = &rcar_gen3_ths_tj_1, + .data = &rcar_gen3_thermal_info, }, { .compatible = "renesas,r8a77980-thermal", - .data = &rcar_gen3_ths_tj_1, + .data = &rcar_gen3_thermal_info, }, { .compatible = "renesas,r8a779a0-thermal", - .data = &rcar_gen3_ths_tj_1, + .data = &rcar_gen3_thermal_info, }, { .compatible = "renesas,r8a779f0-thermal", - .data = &rcar_gen3_ths_tj_1, + .data = &rcar_gen4_thermal_info, }, { .compatible = "renesas,r8a779g0-thermal", - .data = &rcar_gen3_ths_tj_1, + .data = &rcar_gen4_thermal_info, }, {}, }; @@ -418,7 +481,6 @@ static int rcar_gen3_thermal_probe(struct platform_device *pdev) { struct rcar_gen3_thermal_priv *priv; struct device *dev = &pdev->dev; - const int *ths_tj_1 = of_device_get_match_data(dev); struct resource *res; struct thermal_zone_device *zone; unsigned int i; @@ -430,6 +492,7 @@ static int rcar_gen3_thermal_probe(struct platform_device *pdev) priv->ops = rcar_gen3_tz_of_ops; + priv->info = of_device_get_match_data(dev); platform_set_drvdata(pdev, priv); if (rcar_gen3_thermal_request_irqs(priv, pdev)) @@ -469,7 +532,7 @@ static int rcar_gen3_thermal_probe(struct platform_device *pdev) struct rcar_gen3_thermal_tsc *tsc = priv->tscs[i]; rcar_gen3_thermal_init(priv, tsc); - rcar_gen3_thermal_calc_coefs(priv, tsc, *ths_tj_1); + rcar_gen3_thermal_calc_coefs(priv, tsc, priv->info->ths_tj_1); zone = devm_thermal_of_zone_register(dev, i, tsc, &priv->ops); if (IS_ERR(zone)) { diff --git a/drivers/thermal/st/st_thermal.c b/drivers/thermal/st/st_thermal.c index 2d3042098445..0d6249b36609 100644 --- a/drivers/thermal/st/st_thermal.c +++ b/drivers/thermal/st/st_thermal.c @@ -227,14 +227,12 @@ sensor_off: } EXPORT_SYMBOL_GPL(st_thermal_register); -int st_thermal_unregister(struct platform_device *pdev) +void st_thermal_unregister(struct platform_device *pdev) { struct st_thermal_sensor *sensor = platform_get_drvdata(pdev); st_thermal_sensor_off(sensor); thermal_zone_device_unregister(sensor->thermal_dev); - - return 0; } EXPORT_SYMBOL_GPL(st_thermal_unregister); diff --git a/drivers/thermal/st/st_thermal.h b/drivers/thermal/st/st_thermal.h index d661b2f2ef29..75a84e6ec6a7 100644 --- a/drivers/thermal/st/st_thermal.h +++ b/drivers/thermal/st/st_thermal.h @@ -94,7 +94,7 @@ struct st_thermal_sensor { extern int st_thermal_register(struct platform_device *pdev, const struct of_device_id *st_thermal_of_match); -extern int st_thermal_unregister(struct platform_device *pdev); +extern void st_thermal_unregister(struct platform_device *pdev); extern const struct dev_pm_ops st_thermal_pm_ops; #endif /* __STI_RESET_SYSCFG_H */ diff --git a/drivers/thermal/st/st_thermal_memmap.c b/drivers/thermal/st/st_thermal_memmap.c index d68596c40be9..e8cfa83b724a 100644 --- a/drivers/thermal/st/st_thermal_memmap.c +++ b/drivers/thermal/st/st_thermal_memmap.c @@ -172,9 +172,9 @@ static int st_mmap_probe(struct platform_device *pdev) return st_thermal_register(pdev, st_mmap_thermal_of_match); } -static int st_mmap_remove(struct platform_device *pdev) +static void st_mmap_remove(struct platform_device *pdev) { - return st_thermal_unregister(pdev); + st_thermal_unregister(pdev); } static struct platform_driver st_mmap_thermal_driver = { @@ -184,7 +184,7 @@ static struct platform_driver st_mmap_thermal_driver = { .of_match_table = st_mmap_thermal_of_match, }, .probe = st_mmap_probe, - .remove = st_mmap_remove, + .remove_new = st_mmap_remove, }; module_platform_driver(st_mmap_thermal_driver); diff --git a/drivers/thermal/sun8i_thermal.c b/drivers/thermal/sun8i_thermal.c index 793ddce72132..195f3c5d0b38 100644 --- a/drivers/thermal/sun8i_thermal.c +++ b/drivers/thermal/sun8i_thermal.c @@ -319,6 +319,11 @@ out: return ret; } +static void sun8i_ths_reset_control_assert(void *data) +{ + reset_control_assert(data); +} + static int sun8i_ths_resource_init(struct ths_device *tmdev) { struct device *dev = tmdev->dev; @@ -339,47 +344,35 @@ static int sun8i_ths_resource_init(struct ths_device *tmdev) if (IS_ERR(tmdev->reset)) return PTR_ERR(tmdev->reset); - tmdev->bus_clk = devm_clk_get(&pdev->dev, "bus"); + ret = reset_control_deassert(tmdev->reset); + if (ret) + return ret; + + ret = devm_add_action_or_reset(dev, sun8i_ths_reset_control_assert, + tmdev->reset); + if (ret) + return ret; + + tmdev->bus_clk = devm_clk_get_enabled(&pdev->dev, "bus"); if (IS_ERR(tmdev->bus_clk)) return PTR_ERR(tmdev->bus_clk); } if (tmdev->chip->has_mod_clk) { - tmdev->mod_clk = devm_clk_get(&pdev->dev, "mod"); + tmdev->mod_clk = devm_clk_get_enabled(&pdev->dev, "mod"); if (IS_ERR(tmdev->mod_clk)) return PTR_ERR(tmdev->mod_clk); } - ret = reset_control_deassert(tmdev->reset); + ret = clk_set_rate(tmdev->mod_clk, 24000000); if (ret) return ret; - ret = clk_prepare_enable(tmdev->bus_clk); - if (ret) - goto assert_reset; - - ret = clk_set_rate(tmdev->mod_clk, 24000000); - if (ret) - goto bus_disable; - - ret = clk_prepare_enable(tmdev->mod_clk); - if (ret) - goto bus_disable; - ret = sun8i_ths_calibrate(tmdev); if (ret) - goto mod_disable; + return ret; return 0; - -mod_disable: - clk_disable_unprepare(tmdev->mod_clk); -bus_disable: - clk_disable_unprepare(tmdev->bus_clk); -assert_reset: - reset_control_assert(tmdev->reset); - - return ret; } static int sun8i_h3_thermal_init(struct ths_device *tmdev) @@ -475,9 +468,7 @@ static int sun8i_ths_register(struct ths_device *tmdev) if (IS_ERR(tmdev->sensor[i].tzd)) return PTR_ERR(tmdev->sensor[i].tzd); - if (devm_thermal_add_hwmon_sysfs(tmdev->dev, tmdev->sensor[i].tzd)) - dev_warn(tmdev->dev, - "Failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(tmdev->dev, tmdev->sensor[i].tzd); } return 0; @@ -530,17 +521,6 @@ static int sun8i_ths_probe(struct platform_device *pdev) return 0; } -static int sun8i_ths_remove(struct platform_device *pdev) -{ - struct ths_device *tmdev = platform_get_drvdata(pdev); - - clk_disable_unprepare(tmdev->mod_clk); - clk_disable_unprepare(tmdev->bus_clk); - reset_control_assert(tmdev->reset); - - return 0; -} - static const struct ths_thermal_chip sun8i_a83t_ths = { .sensor_num = 3, .scale = 705, @@ -642,7 +622,6 @@ MODULE_DEVICE_TABLE(of, of_ths_match); static struct platform_driver ths_driver = { .probe = sun8i_ths_probe, - .remove = sun8i_ths_remove, .driver = { .name = "sun8i-thermal", .of_match_table = of_ths_match, diff --git a/drivers/thermal/tegra/tegra30-tsensor.c b/drivers/thermal/tegra/tegra30-tsensor.c index cb584a5735ed..c243e9d76d3c 100644 --- a/drivers/thermal/tegra/tegra30-tsensor.c +++ b/drivers/thermal/tegra/tegra30-tsensor.c @@ -523,8 +523,7 @@ static int tegra_tsensor_register_channel(struct tegra_tsensor *ts, return 0; } - if (devm_thermal_add_hwmon_sysfs(ts->dev, tsc->tzd)) - dev_warn(ts->dev, "failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(ts->dev, tsc->tzd); return 0; } diff --git a/drivers/thermal/thermal-generic-adc.c b/drivers/thermal/thermal-generic-adc.c index 017b0ce52122..f4f1a04f8c0f 100644 --- a/drivers/thermal/thermal-generic-adc.c +++ b/drivers/thermal/thermal-generic-adc.c @@ -13,6 +13,8 @@ #include #include +#include "thermal_hwmon.h" + struct gadc_thermal_info { struct device *dev; struct thermal_zone_device *tz_dev; @@ -153,6 +155,8 @@ static int gadc_thermal_probe(struct platform_device *pdev) return ret; } + devm_thermal_add_hwmon_sysfs(&pdev->dev, gti->tz_dev); + return 0; } diff --git a/drivers/thermal/thermal_core.h b/drivers/thermal/thermal_core.h index 3d4a787c6b28..17c1bbed734d 100644 --- a/drivers/thermal/thermal_core.h +++ b/drivers/thermal/thermal_core.h @@ -23,6 +23,8 @@ #define DEFAULT_THERMAL_GOVERNOR "user_space" #elif defined(CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR) #define DEFAULT_THERMAL_GOVERNOR "power_allocator" +#elif defined(CONFIG_THERMAL_DEFAULT_GOV_BANG_BANG) +#define DEFAULT_THERMAL_GOVERNOR "bang_bang" #endif /* Initial state of a cooling device during binding */ diff --git a/drivers/thermal/thermal_hwmon.c b/drivers/thermal/thermal_hwmon.c index fbe55509e307..c3ae44659b81 100644 --- a/drivers/thermal/thermal_hwmon.c +++ b/drivers/thermal/thermal_hwmon.c @@ -271,11 +271,14 @@ int devm_thermal_add_hwmon_sysfs(struct device *dev, struct thermal_zone_device ptr = devres_alloc(devm_thermal_hwmon_release, sizeof(*ptr), GFP_KERNEL); - if (!ptr) + if (!ptr) { + dev_warn(dev, "Failed to allocate device resource data\n"); return -ENOMEM; + } ret = thermal_add_hwmon_sysfs(tz); if (ret) { + dev_warn(dev, "Failed to add hwmon sysfs attributes\n"); devres_free(ptr); return ret; } diff --git a/drivers/thermal/ti-soc-thermal/ti-thermal-common.c b/drivers/thermal/ti-soc-thermal/ti-thermal-common.c index 6a5335931f4d..d414a4b7a94a 100644 --- a/drivers/thermal/ti-soc-thermal/ti-thermal-common.c +++ b/drivers/thermal/ti-soc-thermal/ti-thermal-common.c @@ -182,8 +182,7 @@ int ti_thermal_expose_sensor(struct ti_bandgap *bgp, int id, ti_bandgap_write_update_interval(bgp, data->sensor_id, TI_BANDGAP_UPDATE_INTERVAL_MS); - if (devm_thermal_add_hwmon_sysfs(bgp->dev, data->ti_thermal)) - dev_warn(bgp->dev, "failed to add hwmon sysfs attributes\n"); + devm_thermal_add_hwmon_sysfs(bgp->dev, data->ti_thermal); return 0; } diff --git a/drivers/tty/serial/Kconfig b/drivers/tty/serial/Kconfig index d945e532571a..f56e3567acb1 100644 --- a/drivers/tty/serial/Kconfig +++ b/drivers/tty/serial/Kconfig @@ -449,8 +449,8 @@ config SERIAL_SA1100 help If you have a machine based on a SA1100/SA1110 StrongARM(R) CPU you can enable its onboard serial port by enabling this option. - Please read for further - info. + Please read for + further info. config SERIAL_SA1100_CONSOLE bool "Console on SA1100 serial port" diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h index a6affc0550b0..c941d99162c0 100644 --- a/include/acpi/acpi_bus.h +++ b/include/acpi/acpi_bus.h @@ -289,6 +289,8 @@ struct acpi_dep_data { acpi_handle supplier; acpi_handle consumer; bool honor_dep; + bool met; + bool free_when_met; }; /* Performance Management */ diff --git a/include/acpi/actbl.h b/include/acpi/actbl.h index e5dfb6f4de52..451f6276da49 100644 --- a/include/acpi/actbl.h +++ b/include/acpi/actbl.h @@ -307,7 +307,8 @@ enum acpi_preferred_pm_profiles { PM_SOHO_SERVER = 5, PM_APPLIANCE_PC = 6, PM_PERFORMANCE_SERVER = 7, - PM_TABLET = 8 + PM_TABLET = 8, + NR_PM_PROFILES = 9 }; /* Values for sleep_status and sleep_control registers (V5+ FADT) */ diff --git a/include/kunit/resource.h b/include/kunit/resource.h index c0d88b318e90..c7383e90f5c9 100644 --- a/include/kunit/resource.h +++ b/include/kunit/resource.h @@ -387,4 +387,96 @@ static inline int kunit_destroy_named_resource(struct kunit *test, */ void kunit_remove_resource(struct kunit *test, struct kunit_resource *res); +/* A 'deferred action' function to be used with kunit_add_action. */ +typedef void (kunit_action_t)(void *); + +/** + * kunit_add_action() - Call a function when the test ends. + * @test: Test case to associate the action with. + * @action: The function to run on test exit + * @ctx: Data passed into @func + * + * Defer the execution of a function until the test exits, either normally or + * due to a failure. @ctx is passed as additional context. All functions + * registered with kunit_add_action() will execute in the opposite order to that + * they were registered in. + * + * This is useful for cleaning up allocated memory and resources, as these + * functions are called even if the test aborts early due to, e.g., a failed + * assertion. + * + * See also: devm_add_action() for the devres equivalent. + * + * Returns: + * 0 on success, an error if the action could not be deferred. + */ +int kunit_add_action(struct kunit *test, kunit_action_t *action, void *ctx); + +/** + * kunit_add_action_or_reset() - Call a function when the test ends. + * @test: Test case to associate the action with. + * @action: The function to run on test exit + * @ctx: Data passed into @func + * + * Defer the execution of a function until the test exits, either normally or + * due to a failure. @ctx is passed as additional context. All functions + * registered with kunit_add_action() will execute in the opposite order to that + * they were registered in. + * + * This is useful for cleaning up allocated memory and resources, as these + * functions are called even if the test aborts early due to, e.g., a failed + * assertion. + * + * If the action cannot be created (e.g., due to the system being out of memory), + * then action(ctx) will be called immediately, and an error will be returned. + * + * See also: devm_add_action_or_reset() for the devres equivalent. + * + * Returns: + * 0 on success, an error if the action could not be deferred. + */ +int kunit_add_action_or_reset(struct kunit *test, kunit_action_t *action, + void *ctx); + +/** + * kunit_remove_action() - Cancel a matching deferred action. + * @test: Test case the action is associated with. + * @action: The deferred function to cancel. + * @ctx: The context passed to the deferred function to trigger. + * + * Prevent an action deferred via kunit_add_action() from executing when the + * test terminates. + * + * If the function/context pair was deferred multiple times, only the most + * recent one will be cancelled. + * + * See also: devm_remove_action() for the devres equivalent. + */ +void kunit_remove_action(struct kunit *test, + kunit_action_t *action, + void *ctx); + +/** + * kunit_release_action() - Run a matching action call immediately. + * @test: Test case the action is associated with. + * @action: The deferred function to trigger. + * @ctx: The context passed to the deferred function to trigger. + * + * Execute a function deferred via kunit_add_action()) immediately, rather than + * when the test ends. + * + * If the function/context pair was deferred multiple times, it will only be + * executed once here. The most recent deferral will no longer execute when + * the test ends. + * + * kunit_release_action(test, func, ctx); + * is equivalent to + * func(ctx); + * kunit_remove_action(test, func, ctx); + * + * See also: devm_release_action() for the devres equivalent. + */ +void kunit_release_action(struct kunit *test, + kunit_action_t *action, + void *ctx); #endif /* _KUNIT_RESOURCE_H */ diff --git a/include/kunit/test.h b/include/kunit/test.h index 57b309c6ca27..23120d50499e 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -47,6 +47,7 @@ struct kunit; * sub-subtest. See the "Subtests" section in * https://node-tap.org/tap-protocol/ */ +#define KUNIT_INDENT_LEN 4 #define KUNIT_SUBTEST_INDENT " " #define KUNIT_SUBSUBTEST_INDENT " " @@ -168,6 +169,9 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) * test case, similar to the notion of a *test fixture* or a *test class* * in other unit testing frameworks like JUnit or Googletest. * + * Note that @exit and @suite_exit will run even if @init or @suite_init + * fail: make sure they can handle any inconsistent state which may result. + * * Every &struct kunit_case must be associated with a kunit_suite for KUnit * to run it. */ @@ -321,8 +325,11 @@ enum kunit_status kunit_suite_has_succeeded(struct kunit_suite *suite); * @gfp: flags passed to underlying kmalloc(). * * Just like `kmalloc_array(...)`, except the allocation is managed by the test case - * and is automatically cleaned up after the test case concludes. See &struct - * kunit_resource for more information. + * and is automatically cleaned up after the test case concludes. See kunit_add_action() + * for more information. + * + * Note that some internal context data is also allocated with GFP_KERNEL, + * regardless of the gfp passed in. */ void *kunit_kmalloc_array(struct kunit *test, size_t n, size_t size, gfp_t gfp); @@ -333,6 +340,9 @@ void *kunit_kmalloc_array(struct kunit *test, size_t n, size_t size, gfp_t gfp); * @gfp: flags passed to underlying kmalloc(). * * See kmalloc() and kunit_kmalloc_array() for more information. + * + * Note that some internal context data is also allocated with GFP_KERNEL, + * regardless of the gfp passed in. */ static inline void *kunit_kmalloc(struct kunit *test, size_t size, gfp_t gfp) { @@ -472,7 +482,9 @@ void __printf(2, 3) kunit_log_append(char *log, const char *fmt, ...); */ #define KUNIT_SUCCEED(test) do {} while (0) -void kunit_do_failed_assertion(struct kunit *test, +void __noreturn __kunit_abort(struct kunit *test); + +void __kunit_do_failed_assertion(struct kunit *test, const struct kunit_loc *loc, enum kunit_assert_type type, const struct kunit_assert *assert, @@ -482,13 +494,15 @@ void kunit_do_failed_assertion(struct kunit *test, #define _KUNIT_FAILED(test, assert_type, assert_class, assert_format, INITIALIZER, fmt, ...) do { \ static const struct kunit_loc __loc = KUNIT_CURRENT_LOC; \ const struct assert_class __assertion = INITIALIZER; \ - kunit_do_failed_assertion(test, \ - &__loc, \ - assert_type, \ - &__assertion.assert, \ - assert_format, \ - fmt, \ - ##__VA_ARGS__); \ + __kunit_do_failed_assertion(test, \ + &__loc, \ + assert_type, \ + &__assertion.assert, \ + assert_format, \ + fmt, \ + ##__VA_ARGS__); \ + if (assert_type == KUNIT_ASSERTION) \ + __kunit_abort(test); \ } while (0) diff --git a/include/linux/acpi.h b/include/linux/acpi.h index 7b71dd74baeb..5ef126a0a50f 100644 --- a/include/linux/acpi.h +++ b/include/linux/acpi.h @@ -1507,6 +1507,12 @@ static inline int find_acpi_cpu_topology_hetero_id(unsigned int cpu) } #endif +#ifdef CONFIG_ARM64 +void acpi_arm_init(void); +#else +static inline void acpi_arm_init(void) { } +#endif + #ifdef CONFIG_ACPI_PCC void acpi_init_pcc(void); #else diff --git a/include/linux/acpi_agdi.h b/include/linux/acpi_agdi.h deleted file mode 100644 index f477f0b452fa..000000000000 --- a/include/linux/acpi_agdi.h +++ /dev/null @@ -1,13 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ - -#ifndef __ACPI_AGDI_H__ -#define __ACPI_AGDI_H__ - -#include - -#ifdef CONFIG_ACPI_AGDI -void __init acpi_agdi_init(void); -#else -static inline void acpi_agdi_init(void) {} -#endif -#endif /* __ACPI_AGDI_H__ */ diff --git a/include/linux/acpi_apmt.h b/include/linux/acpi_apmt.h deleted file mode 100644 index 40bd634d082f..000000000000 --- a/include/linux/acpi_apmt.h +++ /dev/null @@ -1,19 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 - * - * ARM CoreSight PMU driver. - * Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. - * - */ - -#ifndef __ACPI_APMT_H__ -#define __ACPI_APMT_H__ - -#include - -#ifdef CONFIG_ACPI_APMT -void acpi_apmt_init(void); -#else -static inline void acpi_apmt_init(void) { } -#endif /* CONFIG_ACPI_APMT */ - -#endif /* __ACPI_APMT_H__ */ diff --git a/include/linux/acpi_iort.h b/include/linux/acpi_iort.h index 6b70d02bc5f9..ee7cb6aaff71 100644 --- a/include/linux/acpi_iort.h +++ b/include/linux/acpi_iort.h @@ -29,7 +29,6 @@ struct fwnode_handle *iort_find_domain_token(int trans_id); int iort_pmsi_get_dev_id(struct device *dev, u32 *dev_id); #ifdef CONFIG_ACPI_IORT -void acpi_iort_init(void); u32 iort_msi_map_id(struct device *dev, u32 id); struct irq_domain *iort_get_device_domain(struct device *dev, u32 id, enum irq_domain_bus_token bus_token); @@ -44,7 +43,6 @@ int iort_iommu_configure_id(struct device *dev, const u32 *id_in); void iort_iommu_get_resv_regions(struct device *dev, struct list_head *head); phys_addr_t acpi_iort_dma_get_max_cpu_address(void); #else -static inline void acpi_iort_init(void) { } static inline u32 iort_msi_map_id(struct device *dev, u32 id) { return id; } static inline struct irq_domain *iort_get_device_domain( diff --git a/include/linux/amd-pstate.h b/include/linux/amd-pstate.h index c10ebf8c42e6..446394f84606 100644 --- a/include/linux/amd-pstate.h +++ b/include/linux/amd-pstate.h @@ -94,7 +94,8 @@ struct amd_cpudata { * enum amd_pstate_mode - driver working mode of amd pstate */ enum amd_pstate_mode { - AMD_PSTATE_DISABLE = 0, + AMD_PSTATE_UNDEFINED = 0, + AMD_PSTATE_DISABLE, AMD_PSTATE_PASSIVE, AMD_PSTATE_ACTIVE, AMD_PSTATE_GUIDED, @@ -102,6 +103,7 @@ enum amd_pstate_mode { }; static const char * const amd_pstate_mode_string[] = { + [AMD_PSTATE_UNDEFINED] = "undefined", [AMD_PSTATE_DISABLE] = "disable", [AMD_PSTATE_PASSIVE] = "passive", [AMD_PSTATE_ACTIVE] = "active", diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 44fb4c5421e4..0468db478e05 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -340,7 +340,10 @@ struct cpufreq_driver { /* * ->fast_switch() replacement for drivers that use an internal * representation of performance levels and can pass hints other than - * the target performance level to the hardware. + * the target performance level to the hardware. This can only be set + * if ->fast_switch is set too, because in those cases (under specific + * conditions) scale invariance can be disabled, which causes the + * schedutil governor to fall back to the latter. */ void (*adjust_perf)(unsigned int cpu, unsigned long min_perf, diff --git a/include/linux/devfreq.h b/include/linux/devfreq.h index 7fd704bb8f3d..d312ffbac4dd 100644 --- a/include/linux/devfreq.h +++ b/include/linux/devfreq.h @@ -108,7 +108,6 @@ struct devfreq_dev_profile { unsigned long initial_freq; unsigned int polling_ms; enum devfreq_timer timer; - bool is_cooling_device; int (*target)(struct device *dev, unsigned long *freq, u32 flags); int (*get_dev_status)(struct device *dev, @@ -118,6 +117,8 @@ struct devfreq_dev_profile { unsigned long *freq_table; unsigned int max_state; + + bool is_cooling_device; }; /** diff --git a/include/linux/err.h b/include/linux/err.h index a139c64aef2a..b5d9bb2a2349 100644 --- a/include/linux/err.h +++ b/include/linux/err.h @@ -19,23 +19,54 @@ #ifndef __ASSEMBLY__ +/** + * IS_ERR_VALUE - Detect an error pointer. + * @x: The pointer to check. + * + * Like IS_ERR(), but does not generate a compiler warning if result is unused. + */ #define IS_ERR_VALUE(x) unlikely((unsigned long)(void *)(x) >= (unsigned long)-MAX_ERRNO) +/** + * ERR_PTR - Create an error pointer. + * @error: A negative error code. + * + * Encodes @error into a pointer value. Users should consider the result + * opaque and not assume anything about how the error is encoded. + * + * Return: A pointer with @error encoded within its value. + */ static inline void * __must_check ERR_PTR(long error) { return (void *) error; } +/** + * PTR_ERR - Extract the error code from an error pointer. + * @ptr: An error pointer. + * Return: The error code within @ptr. + */ static inline long __must_check PTR_ERR(__force const void *ptr) { return (long) ptr; } +/** + * IS_ERR - Detect an error pointer. + * @ptr: The pointer to check. + * Return: true if @ptr is an error pointer, false otherwise. + */ static inline bool __must_check IS_ERR(__force const void *ptr) { return IS_ERR_VALUE((unsigned long)ptr); } +/** + * IS_ERR_OR_NULL - Detect an error pointer or a null pointer. + * @ptr: The pointer to check. + * + * Like IS_ERR(), but also returns true for a null pointer. + */ static inline bool __must_check IS_ERR_OR_NULL(__force const void *ptr) { return unlikely(!ptr) || IS_ERR_VALUE((unsigned long)ptr); @@ -54,6 +85,23 @@ static inline void * __must_check ERR_CAST(__force const void *ptr) return (void *) ptr; } +/** + * PTR_ERR_OR_ZERO - Extract the error code from a pointer if it has one. + * @ptr: A potential error pointer. + * + * Convenience function that can be used inside a function that returns + * an error code to propagate errors received as error pointers. + * For example, ``return PTR_ERR_OR_ZERO(ptr);`` replaces: + * + * .. code-block:: c + * + * if (IS_ERR(ptr)) + * return PTR_ERR(ptr); + * else + * return 0; + * + * Return: The error code within @ptr if it is an error pointer; 0 otherwise. + */ static inline int __must_check PTR_ERR_OR_ZERO(__force const void *ptr) { if (IS_ERR(ptr)) diff --git a/include/linux/intel_rapl.h b/include/linux/intel_rapl.h index 9f4b6f5b822f..e6936cb25047 100644 --- a/include/linux/intel_rapl.h +++ b/include/linux/intel_rapl.h @@ -14,6 +14,12 @@ #include #include +enum rapl_if_type { + RAPL_IF_MSR, /* RAPL I/F using MSR registers */ + RAPL_IF_MMIO, /* RAPL I/F using MMIO registers */ + RAPL_IF_TPMI, /* RAPL I/F using TPMI registers */ +}; + enum rapl_domain_type { RAPL_DOMAIN_PACKAGE, /* entire package/socket */ RAPL_DOMAIN_PP0, /* core power plane */ @@ -30,17 +36,23 @@ enum rapl_domain_reg_id { RAPL_DOMAIN_REG_POLICY, RAPL_DOMAIN_REG_INFO, RAPL_DOMAIN_REG_PL4, + RAPL_DOMAIN_REG_UNIT, + RAPL_DOMAIN_REG_PL2, RAPL_DOMAIN_REG_MAX, }; struct rapl_domain; enum rapl_primitives { - ENERGY_COUNTER, POWER_LIMIT1, POWER_LIMIT2, POWER_LIMIT4, + ENERGY_COUNTER, FW_LOCK, + FW_HIGH_LOCK, + PL1_LOCK, + PL2_LOCK, + PL4_LOCK, PL1_ENABLE, /* power limit 1, aka long term */ PL1_CLAMP, /* allow frequency to go below OS request */ @@ -74,12 +86,13 @@ struct rapl_domain_data { unsigned long timestamp; }; -#define NR_POWER_LIMITS (3) +#define NR_POWER_LIMITS (POWER_LIMIT4 + 1) + struct rapl_power_limit { struct powercap_zone_constraint *constraint; - int prim_id; /* primitive ID used to enable */ struct rapl_domain *domain; const char *name; + bool locked; u64 last_power_limit; }; @@ -96,7 +109,9 @@ struct rapl_domain { struct rapl_power_limit rpl[NR_POWER_LIMITS]; u64 attr_map; /* track capabilities */ unsigned int state; - unsigned int domain_energy_unit; + unsigned int power_unit; + unsigned int energy_unit; + unsigned int time_unit; struct rapl_package *rp; }; @@ -121,16 +136,20 @@ struct reg_action { * registers. * @write_raw: Callback for writing RAPL interface specific * registers. + * @defaults: internal pointer to interface default settings + * @rpi: internal pointer to interface primitive info */ struct rapl_if_priv { + enum rapl_if_type type; struct powercap_control_type *control_type; - struct rapl_domain *platform_rapl_domain; enum cpuhp_state pcap_rapl_online; u64 reg_unit; u64 regs[RAPL_DOMAIN_MAX][RAPL_DOMAIN_REG_MAX]; int limits[RAPL_DOMAIN_MAX]; - int (*read_raw)(int cpu, struct reg_action *ra); - int (*write_raw)(int cpu, struct reg_action *ra); + int (*read_raw)(int id, struct reg_action *ra); + int (*write_raw)(int id, struct reg_action *ra); + void *defaults; + void *rpi; }; /* maximum rapl package domain name: package-%d-die-%d */ @@ -140,9 +159,6 @@ struct rapl_package { unsigned int id; /* logical die id, equals physical 1-die systems */ unsigned int nr_domains; unsigned long domain_map; /* bit map of active domains */ - unsigned int power_unit; - unsigned int energy_unit; - unsigned int time_unit; struct rapl_domain *domains; /* array of domains, sized at runtime */ struct powercap_zone *power_zone; /* keep track of parent zone */ unsigned long power_limit_irq; /* keep track of package power limit @@ -156,8 +172,8 @@ struct rapl_package { struct rapl_if_priv *priv; }; -struct rapl_package *rapl_find_package_domain(int cpu, struct rapl_if_priv *priv); -struct rapl_package *rapl_add_package(int cpu, struct rapl_if_priv *priv); +struct rapl_package *rapl_find_package_domain(int id, struct rapl_if_priv *priv, bool id_is_cpu); +struct rapl_package *rapl_add_package(int id, struct rapl_if_priv *priv, bool id_is_cpu); void rapl_remove_package(struct rapl_package *rp); #endif /* __INTEL_RAPL_H__ */ diff --git a/include/linux/irqchip/mxs.h b/include/linux/irqchip/mxs.h deleted file mode 100644 index 4f447e3f0f3a..000000000000 --- a/include/linux/irqchip/mxs.h +++ /dev/null @@ -1,11 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * Copyright (C) 2013 Freescale Semiconductor, Inc. - */ - -#ifndef __LINUX_IRQCHIP_MXS_H -#define __LINUX_IRQCHIP_MXS_H - -extern void icoll_handle_irq(struct pt_regs *); - -#endif diff --git a/include/linux/perf/arm_pmu.h b/include/linux/perf/arm_pmu.h index 525b5d64e394..c0e4baf940dc 100644 --- a/include/linux/perf/arm_pmu.h +++ b/include/linux/perf/arm_pmu.h @@ -26,9 +26,11 @@ */ #define ARMPMU_EVT_64BIT 0x00001 /* Event uses a 64bit counter */ #define ARMPMU_EVT_47BIT 0x00002 /* Event uses a 47bit counter */ +#define ARMPMU_EVT_63BIT 0x00004 /* Event uses a 63bit counter */ static_assert((PERF_EVENT_FLAG_ARCH & ARMPMU_EVT_64BIT) == ARMPMU_EVT_64BIT); static_assert((PERF_EVENT_FLAG_ARCH & ARMPMU_EVT_47BIT) == ARMPMU_EVT_47BIT); +static_assert((PERF_EVENT_FLAG_ARCH & ARMPMU_EVT_63BIT) == ARMPMU_EVT_63BIT); #define HW_OP_UNSUPPORTED 0xFFFF #define C(_x) PERF_COUNT_HW_CACHE_##_x diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index dcd2cf1e8326..7d9c2a63b7cd 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -156,31 +156,6 @@ static inline int rcu_nocb_cpu_deoffload(int cpu) { return 0; } static inline void rcu_nocb_flush_deferred_wakeup(void) { } #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ -/** - * RCU_NONIDLE - Indicate idle-loop code that needs RCU readers - * @a: Code that RCU needs to pay attention to. - * - * RCU read-side critical sections are forbidden in the inner idle loop, - * that is, between the ct_idle_enter() and the ct_idle_exit() -- RCU - * will happily ignore any such read-side critical sections. However, - * things like powertop need tracepoints in the inner idle loop. - * - * This macro provides the way out: RCU_NONIDLE(do_something_with_RCU()) - * will tell RCU that it needs to pay attention, invoke its argument - * (in this example, calling the do_something_with_RCU() function), - * and then tell RCU to go back to ignoring this CPU. It is permissible - * to nest RCU_NONIDLE() wrappers, but not indefinitely (but the limit is - * on the order of a million or so, even on 32-bit systems). It is - * not legal to block within RCU_NONIDLE(), nor is it permissible to - * transfer control either into or out of RCU_NONIDLE()'s statement. - */ -#define RCU_NONIDLE(a) \ - do { \ - ct_irq_enter_irqson(); \ - do { a; } while (0); \ - ct_irq_exit_irqson(); \ - } while (0) - /* * Note a quasi-voluntary context switch for RCU-tasks's benefit. * This is a macro rather than an inline function to avoid #include hell. @@ -957,9 +932,8 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) /** * kfree_rcu() - kfree an object after a grace period. - * @ptr: pointer to kfree for both single- and double-argument invocations. - * @rhf: the name of the struct rcu_head within the type of @ptr, - * but only for double-argument invocations. + * @ptr: pointer to kfree for double-argument invocations. + * @rhf: the name of the struct rcu_head within the type of @ptr. * * Many rcu callbacks functions just call kfree() on the base structure. * These functions are trivial, but their size adds up, and furthermore @@ -984,26 +958,18 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * The BUILD_BUG_ON check must not involve any function calls, hence the * checks are done in macros here. */ -#define kfree_rcu(ptr, rhf...) kvfree_rcu(ptr, ## rhf) +#define kfree_rcu(ptr, rhf) kvfree_rcu_arg_2(ptr, rhf) +#define kvfree_rcu(ptr, rhf) kvfree_rcu_arg_2(ptr, rhf) /** - * kvfree_rcu() - kvfree an object after a grace period. - * - * This macro consists of one or two arguments and it is - * based on whether an object is head-less or not. If it - * has a head then a semantic stays the same as it used - * to be before: - * - * kvfree_rcu(ptr, rhf); - * - * where @ptr is a pointer to kvfree(), @rhf is the name - * of the rcu_head structure within the type of @ptr. + * kfree_rcu_mightsleep() - kfree an object after a grace period. + * @ptr: pointer to kfree for single-argument invocations. * * When it comes to head-less variant, only one argument * is passed and that is just a pointer which has to be * freed after a grace period. Therefore the semantic is * - * kvfree_rcu(ptr); + * kfree_rcu_mightsleep(ptr); * * where @ptr is the pointer to be freed by kvfree(). * @@ -1012,13 +978,9 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * annotation. Otherwise, please switch and embed the * rcu_head structure within the type of @ptr. */ -#define kvfree_rcu(...) KVFREE_GET_MACRO(__VA_ARGS__, \ - kvfree_rcu_arg_2, kvfree_rcu_arg_1)(__VA_ARGS__) - +#define kfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) #define kvfree_rcu_mightsleep(ptr) kvfree_rcu_arg_1(ptr) -#define kfree_rcu_mightsleep(ptr) kvfree_rcu_mightsleep(ptr) -#define KVFREE_GET_MACRO(_1, _2, NAME, ...) NAME #define kvfree_rcu_arg_2(ptr, rhf) \ do { \ typeof (ptr) ___p = (ptr); \ diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 41c4b26fb1c1..eb92a50a4599 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -212,7 +212,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) srcu_check_nmi_safety(ssp, false); retval = __srcu_read_lock(ssp); - srcu_lock_acquire(&(ssp)->dep_map); + srcu_lock_acquire(&ssp->dep_map); return retval; } @@ -229,7 +229,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp srcu_check_nmi_safety(ssp, true); retval = __srcu_read_lock_nmisafe(ssp); - rcu_lock_acquire(&(ssp)->dep_map); + rcu_lock_acquire(&ssp->dep_map); return retval; } @@ -284,7 +284,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) { WARN_ON_ONCE(idx & ~0x1); srcu_check_nmi_safety(ssp, false); - srcu_lock_release(&(ssp)->dep_map); + srcu_lock_release(&ssp->dep_map); __srcu_read_unlock(ssp, idx); } @@ -300,7 +300,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) { WARN_ON_ONCE(idx & ~0x1); srcu_check_nmi_safety(ssp, true); - rcu_lock_release(&(ssp)->dep_map); + rcu_lock_release(&ssp->dep_map); __srcu_read_unlock_nmisafe(ssp, idx); } diff --git a/include/linux/suspend.h b/include/linux/suspend.h index 622838660951..6fe4c8cc32d9 100644 --- a/include/linux/suspend.h +++ b/include/linux/suspend.h @@ -202,6 +202,7 @@ struct platform_s2idle_ops { }; #ifdef CONFIG_SUSPEND +extern suspend_state_t pm_suspend_target_state; extern suspend_state_t mem_sleep_current; extern suspend_state_t mem_sleep_default; @@ -337,6 +338,8 @@ extern bool sync_on_suspend_enabled; #else /* !CONFIG_SUSPEND */ #define suspend_valid_only_mem NULL +#define pm_suspend_target_state (PM_SUSPEND_ON) + static inline void pm_suspend_clear_flags(void) {} static inline void pm_set_suspend_via_firmware(void) {} static inline void pm_set_resume_via_firmware(void) {} @@ -472,6 +475,8 @@ static inline int hibernate_quiet_exec(int (*func)(void *data), void *data) { } #endif /* CONFIG_HIBERNATION */ +int arch_resume_nosmt(void); + #ifdef CONFIG_HIBERNATION_SNAPSHOT_DEV int is_hibernate_resume_dev(dev_t dev); #else @@ -507,7 +512,6 @@ extern void pm_report_max_hw_sleep(u64 t); /* drivers/base/power/wakeup.c */ extern bool events_check_enabled; -extern suspend_state_t pm_suspend_target_state; extern bool pm_wakeup_pending(void); extern void pm_system_wakeup(void); @@ -556,6 +560,7 @@ static inline void unlock_system_sleep(unsigned int flags) {} #ifdef CONFIG_PM_SLEEP_DEBUG extern bool pm_print_times_enabled; extern bool pm_debug_messages_on; +extern bool pm_debug_messages_should_print(void); static inline int pm_dyn_debug_messages_on(void) { #ifdef CONFIG_DYNAMIC_DEBUG @@ -569,14 +574,14 @@ static inline int pm_dyn_debug_messages_on(void) #endif #define __pm_pr_dbg(fmt, ...) \ do { \ - if (pm_debug_messages_on) \ + if (pm_debug_messages_should_print()) \ printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \ else if (pm_dyn_debug_messages_on()) \ pr_debug(fmt, ##__VA_ARGS__); \ } while (0) #define __pm_deferred_pr_dbg(fmt, ...) \ do { \ - if (pm_debug_messages_on) \ + if (pm_debug_messages_should_print()) \ printk_deferred(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \ } while (0) #else @@ -594,7 +599,8 @@ static inline int pm_dyn_debug_messages_on(void) /** * pm_pr_dbg - print pm sleep debug messages * - * If pm_debug_messages_on is enabled, print message. + * If pm_debug_messages_on is enabled and the system is entering/leaving + * suspend, print message. * If pm_debug_messages_on is disabled and CONFIG_DYNAMIC_DEBUG is enabled, * print message only from instances explicitly enabled on dynamic debug's * control. diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c index 153ddc4c47ef..949d3deae506 100644 --- a/kernel/locking/locktorture.c +++ b/kernel/locking/locktorture.c @@ -33,24 +33,19 @@ MODULE_LICENSE("GPL"); MODULE_AUTHOR("Paul E. McKenney "); -torture_param(int, nwriters_stress, -1, - "Number of write-locking stress-test threads"); -torture_param(int, nreaders_stress, -1, - "Number of read-locking stress-test threads"); +torture_param(int, nwriters_stress, -1, "Number of write-locking stress-test threads"); +torture_param(int, nreaders_stress, -1, "Number of read-locking stress-test threads"); +torture_param(int, long_hold, 100, "Do occasional long hold of lock (ms), 0=disable"); torture_param(int, onoff_holdoff, 0, "Time after boot before CPU hotplugs (s)"); -torture_param(int, onoff_interval, 0, - "Time between CPU hotplugs (s), 0=disable"); -torture_param(int, shuffle_interval, 3, - "Number of jiffies between shuffles, 0=disable"); +torture_param(int, onoff_interval, 0, "Time between CPU hotplugs (s), 0=disable"); +torture_param(int, shuffle_interval, 3, "Number of jiffies between shuffles, 0=disable"); torture_param(int, shutdown_secs, 0, "Shutdown time (j), <= zero to disable."); -torture_param(int, stat_interval, 60, - "Number of seconds between stats printk()s"); +torture_param(int, stat_interval, 60, "Number of seconds between stats printk()s"); torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable"); torture_param(int, rt_boost, 2, - "Do periodic rt-boost. 0=Disable, 1=Only for rt_mutex, 2=For all lock types."); + "Do periodic rt-boost. 0=Disable, 1=Only for rt_mutex, 2=For all lock types."); torture_param(int, rt_boost_factor, 50, "A factor determining how often rt-boost happens."); -torture_param(int, verbose, 1, - "Enable verbose debugging printk()s"); +torture_param(int, verbose, 1, "Enable verbose debugging printk()s"); torture_param(int, nested_locks, 0, "Number of nested locks (max = 8)"); /* Going much higher trips "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!" errors */ #define MAX_NESTED_LOCKS 8 @@ -120,7 +115,7 @@ static int torture_lock_busted_write_lock(int tid __maybe_unused) static void torture_lock_busted_write_delay(struct torture_random_state *trsp) { - const unsigned long longdelay_ms = 100; + const unsigned long longdelay_ms = long_hold ? long_hold : ULONG_MAX; /* We want a long delay occasionally to force massive contention. */ if (!(torture_random(trsp) % @@ -198,16 +193,18 @@ __acquires(torture_spinlock) static void torture_spin_lock_write_delay(struct torture_random_state *trsp) { const unsigned long shortdelay_us = 2; - const unsigned long longdelay_ms = 100; + const unsigned long longdelay_ms = long_hold ? long_hold : ULONG_MAX; + unsigned long j; /* We want a short delay mostly to emulate likely code, and * we want a long delay occasionally to force massive contention. */ - if (!(torture_random(trsp) % - (cxt.nrealwriters_stress * 2000 * longdelay_ms))) + if (!(torture_random(trsp) % (cxt.nrealwriters_stress * 2000 * longdelay_ms))) { + j = jiffies; mdelay(longdelay_ms); - if (!(torture_random(trsp) % - (cxt.nrealwriters_stress * 2 * shortdelay_us))) + pr_alert("%s: delay = %lu jiffies.\n", __func__, jiffies - j); + } + if (!(torture_random(trsp) % (cxt.nrealwriters_stress * 200 * shortdelay_us))) udelay(shortdelay_us); if (!(torture_random(trsp) % (cxt.nrealwriters_stress * 20000))) torture_preempt_schedule(); /* Allow test to be preempted. */ @@ -322,7 +319,7 @@ __acquires(torture_rwlock) static void torture_rwlock_write_delay(struct torture_random_state *trsp) { const unsigned long shortdelay_us = 2; - const unsigned long longdelay_ms = 100; + const unsigned long longdelay_ms = long_hold ? long_hold : ULONG_MAX; /* We want a short delay mostly to emulate likely code, and * we want a long delay occasionally to force massive contention. @@ -455,14 +452,12 @@ __acquires(torture_mutex) static void torture_mutex_delay(struct torture_random_state *trsp) { - const unsigned long longdelay_ms = 100; + const unsigned long longdelay_ms = long_hold ? long_hold : ULONG_MAX; /* We want a long delay occasionally to force massive contention. */ if (!(torture_random(trsp) % (cxt.nrealwriters_stress * 2000 * longdelay_ms))) mdelay(longdelay_ms * 5); - else - mdelay(longdelay_ms / 5); if (!(torture_random(trsp) % (cxt.nrealwriters_stress * 20000))) torture_preempt_schedule(); /* Allow test to be preempted. */ } @@ -630,7 +625,7 @@ __acquires(torture_rtmutex) static void torture_rtmutex_delay(struct torture_random_state *trsp) { const unsigned long shortdelay_us = 2; - const unsigned long longdelay_ms = 100; + const unsigned long longdelay_ms = long_hold ? long_hold : ULONG_MAX; /* * We want a short delay mostly to emulate likely code, and @@ -640,7 +635,7 @@ static void torture_rtmutex_delay(struct torture_random_state *trsp) (cxt.nrealwriters_stress * 2000 * longdelay_ms))) mdelay(longdelay_ms); if (!(torture_random(trsp) % - (cxt.nrealwriters_stress * 2 * shortdelay_us))) + (cxt.nrealwriters_stress * 200 * shortdelay_us))) udelay(shortdelay_us); if (!(torture_random(trsp) % (cxt.nrealwriters_stress * 20000))) torture_preempt_schedule(); /* Allow test to be preempted. */ @@ -695,14 +690,12 @@ __acquires(torture_rwsem) static void torture_rwsem_write_delay(struct torture_random_state *trsp) { - const unsigned long longdelay_ms = 100; + const unsigned long longdelay_ms = long_hold ? long_hold : ULONG_MAX; /* We want a long delay occasionally to force massive contention. */ if (!(torture_random(trsp) % (cxt.nrealwriters_stress * 2000 * longdelay_ms))) mdelay(longdelay_ms * 10); - else - mdelay(longdelay_ms / 10); if (!(torture_random(trsp) % (cxt.nrealwriters_stress * 20000))) torture_preempt_schedule(); /* Allow test to be preempted. */ } @@ -848,8 +841,8 @@ static int lock_torture_writer(void *arg) lwsp->n_lock_acquired++; } - cxt.cur_ops->write_delay(&rand); if (!skip_main_lock) { + cxt.cur_ops->write_delay(&rand); lock_is_write_held = false; WRITE_ONCE(last_lock_release, jiffies); cxt.cur_ops->writeunlock(tid); diff --git a/kernel/power/main.c b/kernel/power/main.c index 3113ec2f1db4..daa535012e51 100644 --- a/kernel/power/main.c +++ b/kernel/power/main.c @@ -556,6 +556,12 @@ power_attr_ro(pm_wakeup_irq); bool pm_debug_messages_on __read_mostly; +bool pm_debug_messages_should_print(void) +{ + return pm_debug_messages_on && pm_suspend_target_state != PM_SUSPEND_ON; +} +EXPORT_SYMBOL_GPL(pm_debug_messages_should_print); + static ssize_t pm_debug_messages_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index cd8b7b35f1e8..b27affb7503f 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -398,7 +398,7 @@ struct mem_zone_bm_rtree { unsigned int blocks; /* Number of Bitmap Blocks */ }; -/* strcut bm_position is used for browsing memory bitmaps */ +/* struct bm_position is used for browsing memory bitmaps */ struct bm_position { struct mem_zone_bm_rtree *zone; diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 9071182b1284..bdd7eadb33d8 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -314,4 +314,22 @@ config RCU_LAZY To save power, batch RCU callbacks and flush after delay, memory pressure, or callback list growing too big. +config RCU_DOUBLE_CHECK_CB_TIME + bool "RCU callback-batch backup time check" + depends on RCU_EXPERT + default n + help + Use this option to provide more precise enforcement of the + rcutree.rcu_resched_ns module parameter in situations where + a single RCU callback might run for hundreds of microseconds, + thus defeating the 32-callback batching used to amortize the + cost of the fine-grained but expensive local_clock() function. + + This option rounds rcutree.rcu_resched_ns up to the next + jiffy, and overrides the 32-callback batching if this limit + is exceeded. + + Say Y here if you need tighter callback-limit enforcement. + Say N here if you are unsure. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 4a1b9622598b..98c1544cf572 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -642,4 +642,10 @@ void show_rcu_tasks_trace_gp_kthread(void); static inline void show_rcu_tasks_trace_gp_kthread(void) {} #endif +#ifdef CONFIG_TINY_RCU +static inline bool rcu_cpu_beenfullyonline(int cpu) { return true; } +#else +bool rcu_cpu_beenfullyonline(int cpu); +#endif + #endif /* __LINUX_RCU_H */ diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index e82ec9f9a5d8..d1221731c7cf 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -522,89 +522,6 @@ rcu_scale_print_module_parms(struct rcu_scale_ops *cur_ops, const char *tag) scale_type, tag, nrealreaders, nrealwriters, verbose, shutdown); } -static void -rcu_scale_cleanup(void) -{ - int i; - int j; - int ngps = 0; - u64 *wdp; - u64 *wdpp; - - /* - * Would like warning at start, but everything is expedited - * during the mid-boot phase, so have to wait till the end. - */ - if (rcu_gp_is_expedited() && !rcu_gp_is_normal() && !gp_exp) - SCALEOUT_ERRSTRING("All grace periods expedited, no normal ones to measure!"); - if (rcu_gp_is_normal() && gp_exp) - SCALEOUT_ERRSTRING("All grace periods normal, no expedited ones to measure!"); - if (gp_exp && gp_async) - SCALEOUT_ERRSTRING("No expedited async GPs, so went with async!"); - - if (torture_cleanup_begin()) - return; - if (!cur_ops) { - torture_cleanup_end(); - return; - } - - if (reader_tasks) { - for (i = 0; i < nrealreaders; i++) - torture_stop_kthread(rcu_scale_reader, - reader_tasks[i]); - kfree(reader_tasks); - } - - if (writer_tasks) { - for (i = 0; i < nrealwriters; i++) { - torture_stop_kthread(rcu_scale_writer, - writer_tasks[i]); - if (!writer_n_durations) - continue; - j = writer_n_durations[i]; - pr_alert("%s%s writer %d gps: %d\n", - scale_type, SCALE_FLAG, i, j); - ngps += j; - } - pr_alert("%s%s start: %llu end: %llu duration: %llu gps: %d batches: %ld\n", - scale_type, SCALE_FLAG, - t_rcu_scale_writer_started, t_rcu_scale_writer_finished, - t_rcu_scale_writer_finished - - t_rcu_scale_writer_started, - ngps, - rcuscale_seq_diff(b_rcu_gp_test_finished, - b_rcu_gp_test_started)); - for (i = 0; i < nrealwriters; i++) { - if (!writer_durations) - break; - if (!writer_n_durations) - continue; - wdpp = writer_durations[i]; - if (!wdpp) - continue; - for (j = 0; j < writer_n_durations[i]; j++) { - wdp = &wdpp[j]; - pr_alert("%s%s %4d writer-duration: %5d %llu\n", - scale_type, SCALE_FLAG, - i, j, *wdp); - if (j % 100 == 0) - schedule_timeout_uninterruptible(1); - } - kfree(writer_durations[i]); - } - kfree(writer_tasks); - kfree(writer_durations); - kfree(writer_n_durations); - } - - /* Do torture-type-specific cleanup operations. */ - if (cur_ops->cleanup != NULL) - cur_ops->cleanup(); - - torture_cleanup_end(); -} - /* * Return the number if non-negative. If -1, the number of CPUs. * If less than -1, that much less than the number of CPUs, but @@ -624,20 +541,6 @@ static int compute_real(int n) return nr; } -/* - * RCU scalability shutdown kthread. Just waits to be awakened, then shuts - * down system. - */ -static int -rcu_scale_shutdown(void *arg) -{ - wait_event_idle(shutdown_wq, atomic_read(&n_rcu_scale_writer_finished) >= nrealwriters); - smp_mb(); /* Wake before output. */ - rcu_scale_cleanup(); - kernel_power_off(); - return -EINVAL; -} - /* * kfree_rcu() scalability tests: Start a kfree_rcu() loop on all CPUs for number * of iterations and measure total time and number of GP for all iterations to complete. @@ -874,6 +777,108 @@ unwind: return firsterr; } +static void +rcu_scale_cleanup(void) +{ + int i; + int j; + int ngps = 0; + u64 *wdp; + u64 *wdpp; + + /* + * Would like warning at start, but everything is expedited + * during the mid-boot phase, so have to wait till the end. + */ + if (rcu_gp_is_expedited() && !rcu_gp_is_normal() && !gp_exp) + SCALEOUT_ERRSTRING("All grace periods expedited, no normal ones to measure!"); + if (rcu_gp_is_normal() && gp_exp) + SCALEOUT_ERRSTRING("All grace periods normal, no expedited ones to measure!"); + if (gp_exp && gp_async) + SCALEOUT_ERRSTRING("No expedited async GPs, so went with async!"); + + if (kfree_rcu_test) { + kfree_scale_cleanup(); + return; + } + + if (torture_cleanup_begin()) + return; + if (!cur_ops) { + torture_cleanup_end(); + return; + } + + if (reader_tasks) { + for (i = 0; i < nrealreaders; i++) + torture_stop_kthread(rcu_scale_reader, + reader_tasks[i]); + kfree(reader_tasks); + } + + if (writer_tasks) { + for (i = 0; i < nrealwriters; i++) { + torture_stop_kthread(rcu_scale_writer, + writer_tasks[i]); + if (!writer_n_durations) + continue; + j = writer_n_durations[i]; + pr_alert("%s%s writer %d gps: %d\n", + scale_type, SCALE_FLAG, i, j); + ngps += j; + } + pr_alert("%s%s start: %llu end: %llu duration: %llu gps: %d batches: %ld\n", + scale_type, SCALE_FLAG, + t_rcu_scale_writer_started, t_rcu_scale_writer_finished, + t_rcu_scale_writer_finished - + t_rcu_scale_writer_started, + ngps, + rcuscale_seq_diff(b_rcu_gp_test_finished, + b_rcu_gp_test_started)); + for (i = 0; i < nrealwriters; i++) { + if (!writer_durations) + break; + if (!writer_n_durations) + continue; + wdpp = writer_durations[i]; + if (!wdpp) + continue; + for (j = 0; j < writer_n_durations[i]; j++) { + wdp = &wdpp[j]; + pr_alert("%s%s %4d writer-duration: %5d %llu\n", + scale_type, SCALE_FLAG, + i, j, *wdp); + if (j % 100 == 0) + schedule_timeout_uninterruptible(1); + } + kfree(writer_durations[i]); + } + kfree(writer_tasks); + kfree(writer_durations); + kfree(writer_n_durations); + } + + /* Do torture-type-specific cleanup operations. */ + if (cur_ops->cleanup != NULL) + cur_ops->cleanup(); + + torture_cleanup_end(); +} + +/* + * RCU scalability shutdown kthread. Just waits to be awakened, then shuts + * down system. + */ +static int +rcu_scale_shutdown(void *arg) +{ + wait_event_idle(shutdown_wq, atomic_read(&n_rcu_scale_writer_finished) >= nrealwriters); + smp_mb(); /* Wake before output. */ + rcu_scale_cleanup(); + kernel_power_off(); + return -EINVAL; +} + static int __init rcu_scale_init(void) { diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 5f4fc8184dd0..b770add3f843 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -241,7 +241,6 @@ static void cblist_init_generic(struct rcu_tasks *rtp) if (rcu_task_enqueue_lim < 0) { rcu_task_enqueue_lim = 1; rcu_task_cb_adjust = true; - pr_info("%s: Setting adjustable number of callback queues.\n", __func__); } else if (rcu_task_enqueue_lim == 0) { rcu_task_enqueue_lim = 1; } @@ -272,7 +271,9 @@ static void cblist_init_generic(struct rcu_tasks *rtp) raw_spin_unlock_rcu_node(rtpcp); // irqs remain disabled. } raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags); - pr_info("%s: Setting shift to %d and lim to %d.\n", __func__, data_race(rtp->percpu_enqueue_shift), data_race(rtp->percpu_enqueue_lim)); + + pr_info("%s: Setting shift to %d and lim to %d rcu_task_cb_adjust=%d.\n", rtp->name, + data_race(rtp->percpu_enqueue_shift), data_race(rtp->percpu_enqueue_lim), rcu_task_cb_adjust); } // IRQ-work handler that does deferred wakeup for call_rcu_tasks_generic(). @@ -463,6 +464,7 @@ static void rcu_tasks_invoke_cbs(struct rcu_tasks *rtp, struct rcu_tasks_percpu { int cpu; int cpunext; + int cpuwq; unsigned long flags; int len; struct rcu_head *rhp; @@ -473,11 +475,13 @@ static void rcu_tasks_invoke_cbs(struct rcu_tasks *rtp, struct rcu_tasks_percpu cpunext = cpu * 2 + 1; if (cpunext < smp_load_acquire(&rtp->percpu_dequeue_lim)) { rtpcp_next = per_cpu_ptr(rtp->rtpcpu, cpunext); - queue_work_on(cpunext, system_wq, &rtpcp_next->rtp_work); + cpuwq = rcu_cpu_beenfullyonline(cpunext) ? cpunext : WORK_CPU_UNBOUND; + queue_work_on(cpuwq, system_wq, &rtpcp_next->rtp_work); cpunext++; if (cpunext < smp_load_acquire(&rtp->percpu_dequeue_lim)) { rtpcp_next = per_cpu_ptr(rtp->rtpcpu, cpunext); - queue_work_on(cpunext, system_wq, &rtpcp_next->rtp_work); + cpuwq = rcu_cpu_beenfullyonline(cpunext) ? cpunext : WORK_CPU_UNBOUND; + queue_work_on(cpuwq, system_wq, &rtpcp_next->rtp_work); } } diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f52ff7241041..1449cb69a0e0 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2046,19 +2046,35 @@ rcu_check_quiescent_state(struct rcu_data *rdp) rcu_report_qs_rdp(rdp); } +/* Return true if callback-invocation time limit exceeded. */ +static bool rcu_do_batch_check_time(long count, long tlimit, + bool jlimit_check, unsigned long jlimit) +{ + // Invoke local_clock() only once per 32 consecutive callbacks. + return unlikely(tlimit) && + (!likely(count & 31) || + (IS_ENABLED(CONFIG_RCU_DOUBLE_CHECK_CB_TIME) && + jlimit_check && time_after(jiffies, jlimit))) && + local_clock() >= tlimit; +} + /* * Invoke any RCU callbacks that have made it to the end of their grace * period. Throttle as specified by rdp->blimit. */ static void rcu_do_batch(struct rcu_data *rdp) { + long bl; + long count = 0; int div; bool __maybe_unused empty; unsigned long flags; - struct rcu_head *rhp; + unsigned long jlimit; + bool jlimit_check = false; + long pending; struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl); - long bl, count = 0; - long pending, tlimit = 0; + struct rcu_head *rhp; + long tlimit = 0; /* If no callbacks are ready, just return. */ if (!rcu_segcblist_ready_cbs(&rdp->cblist)) { @@ -2082,11 +2098,15 @@ static void rcu_do_batch(struct rcu_data *rdp) div = READ_ONCE(rcu_divisor); div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div; bl = max(rdp->blimit, pending >> div); - if (in_serving_softirq() && unlikely(bl > 100)) { + if ((in_serving_softirq() || rdp->rcu_cpu_kthread_status == RCU_KTHREAD_RUNNING) && + (IS_ENABLED(CONFIG_RCU_DOUBLE_CHECK_CB_TIME) || unlikely(bl > 100))) { + const long npj = NSEC_PER_SEC / HZ; long rrn = READ_ONCE(rcu_resched_ns); rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn; tlimit = local_clock() + rrn; + jlimit = jiffies + (rrn + npj + 1) / npj; + jlimit_check = true; } trace_rcu_batch_start(rcu_state.name, rcu_segcblist_n_cbs(&rdp->cblist), bl); @@ -2126,21 +2146,23 @@ static void rcu_do_batch(struct rcu_data *rdp) * Make sure we don't spend too much time here and deprive other * softirq vectors of CPU cycles. */ - if (unlikely(tlimit)) { - /* only call local_clock() every 32 callbacks */ - if (likely((count & 31) || local_clock() < tlimit)) - continue; - /* Exceeded the time limit, so leave. */ + if (rcu_do_batch_check_time(count, tlimit, jlimit_check, jlimit)) break; - } } else { - // In rcuoc context, so no worries about depriving - // other softirq vectors of CPU cycles. + // In rcuc/rcuoc context, so no worries about + // depriving other softirq vectors of CPU cycles. local_bh_enable(); lockdep_assert_irqs_enabled(); cond_resched_tasks_rcu_qs(); lockdep_assert_irqs_enabled(); local_bh_disable(); + // But rcuc kthreads can delay quiescent-state + // reporting, so check time limits for them. + if (rdp->rcu_cpu_kthread_status == RCU_KTHREAD_RUNNING && + rcu_do_batch_check_time(count, tlimit, jlimit_check, jlimit)) { + rdp->rcu_cpu_has_work = 1; + break; + } } } @@ -2459,12 +2481,12 @@ static void rcu_cpu_kthread(unsigned int cpu) *statusp = RCU_KTHREAD_RUNNING; local_irq_disable(); work = *workp; - *workp = 0; + WRITE_ONCE(*workp, 0); local_irq_enable(); if (work) rcu_core(); local_bh_enable(); - if (*workp == 0) { + if (!READ_ONCE(*workp)) { trace_rcu_utilization(TPS("End CPU kthread@rcu_wait")); *statusp = RCU_KTHREAD_WAITING; return; @@ -2756,7 +2778,7 @@ EXPORT_SYMBOL_GPL(call_rcu); */ struct kvfree_rcu_bulk_data { struct list_head list; - unsigned long gp_snap; + struct rcu_gp_oldstate gp_snap; unsigned long nr_records; void *records[]; }; @@ -2773,6 +2795,7 @@ struct kvfree_rcu_bulk_data { * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period * @head_free: List of kfree_rcu() objects waiting for a grace period + * @head_free_gp_snap: Grace-period snapshot to check for attempted premature frees. * @bulk_head_free: Bulk-List of kvfree_rcu() objects waiting for a grace period * @krcp: Pointer to @kfree_rcu_cpu structure */ @@ -2780,6 +2803,7 @@ struct kvfree_rcu_bulk_data { struct kfree_rcu_cpu_work { struct rcu_work rcu_work; struct rcu_head *head_free; + struct rcu_gp_oldstate head_free_gp_snap; struct list_head bulk_head_free[FREE_N_CHANNELS]; struct kfree_rcu_cpu *krcp; }; @@ -2900,6 +2924,9 @@ drain_page_cache(struct kfree_rcu_cpu *krcp) struct llist_node *page_list, *pos, *n; int freed = 0; + if (!rcu_min_cached_objs) + return 0; + raw_spin_lock_irqsave(&krcp->lock, flags); page_list = llist_del_all(&krcp->bkvcache); WRITE_ONCE(krcp->nr_bkv_objs, 0); @@ -2920,24 +2947,25 @@ kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, unsigned long flags; int i; - debug_rcu_bhead_unqueue(bnode); + if (!WARN_ON_ONCE(!poll_state_synchronize_rcu_full(&bnode->gp_snap))) { + debug_rcu_bhead_unqueue(bnode); + rcu_lock_acquire(&rcu_callback_map); + if (idx == 0) { // kmalloc() / kfree(). + trace_rcu_invoke_kfree_bulk_callback( + rcu_state.name, bnode->nr_records, + bnode->records); - rcu_lock_acquire(&rcu_callback_map); - if (idx == 0) { // kmalloc() / kfree(). - trace_rcu_invoke_kfree_bulk_callback( - rcu_state.name, bnode->nr_records, - bnode->records); + kfree_bulk(bnode->nr_records, bnode->records); + } else { // vmalloc() / vfree(). + for (i = 0; i < bnode->nr_records; i++) { + trace_rcu_invoke_kvfree_callback( + rcu_state.name, bnode->records[i], 0); - kfree_bulk(bnode->nr_records, bnode->records); - } else { // vmalloc() / vfree(). - for (i = 0; i < bnode->nr_records; i++) { - trace_rcu_invoke_kvfree_callback( - rcu_state.name, bnode->records[i], 0); - - vfree(bnode->records[i]); + vfree(bnode->records[i]); + } } + rcu_lock_release(&rcu_callback_map); } - rcu_lock_release(&rcu_callback_map); raw_spin_lock_irqsave(&krcp->lock, flags); if (put_cached_bnode(krcp, bnode)) @@ -2984,6 +3012,7 @@ static void kfree_rcu_work(struct work_struct *work) struct rcu_head *head; struct kfree_rcu_cpu *krcp; struct kfree_rcu_cpu_work *krwp; + struct rcu_gp_oldstate head_gp_snap; int i; krwp = container_of(to_rcu_work(work), @@ -2998,6 +3027,7 @@ static void kfree_rcu_work(struct work_struct *work) // Channel 3. head = krwp->head_free; krwp->head_free = NULL; + head_gp_snap = krwp->head_free_gp_snap; raw_spin_unlock_irqrestore(&krcp->lock, flags); // Handle the first two channels. @@ -3014,7 +3044,8 @@ static void kfree_rcu_work(struct work_struct *work) * queued on a linked list through their rcu_head structures. * This list is named "Channel 3". */ - kvfree_rcu_list(head); + if (head && !WARN_ON_ONCE(!poll_state_synchronize_rcu_full(&head_gp_snap))) + kvfree_rcu_list(head); } static bool @@ -3081,7 +3112,7 @@ kvfree_rcu_drain_ready(struct kfree_rcu_cpu *krcp) INIT_LIST_HEAD(&bulk_ready[i]); list_for_each_entry_safe_reverse(bnode, n, &krcp->bulk_head[i], list) { - if (!poll_state_synchronize_rcu(bnode->gp_snap)) + if (!poll_state_synchronize_rcu_full(&bnode->gp_snap)) break; atomic_sub(bnode->nr_records, &krcp->bulk_count[i]); @@ -3146,6 +3177,7 @@ static void kfree_rcu_monitor(struct work_struct *work) // objects queued on the linked list. if (!krwp->head_free) { krwp->head_free = krcp->head; + get_state_synchronize_rcu_full(&krwp->head_free_gp_snap); atomic_set(&krcp->head_count, 0); WRITE_ONCE(krcp->head, NULL); } @@ -3194,7 +3226,7 @@ static void fill_page_cache_func(struct work_struct *work) nr_pages = atomic_read(&krcp->backoff_page_cache_fill) ? 1 : rcu_min_cached_objs; - for (i = 0; i < nr_pages; i++) { + for (i = READ_ONCE(krcp->nr_bkv_objs); i < nr_pages; i++) { bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_KERNEL | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN); @@ -3218,6 +3250,10 @@ static void fill_page_cache_func(struct work_struct *work) static void run_page_cache_worker(struct kfree_rcu_cpu *krcp) { + // If cache disabled, bail out. + if (!rcu_min_cached_objs) + return; + if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING && !atomic_xchg(&krcp->work_in_progress, 1)) { if (atomic_read(&krcp->backoff_page_cache_fill)) { @@ -3272,7 +3308,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, // scenarios. bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_KERNEL | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN); - *krcp = krc_this_cpu_lock(flags); + raw_spin_lock_irqsave(&(*krcp)->lock, *flags); } if (!bnode) @@ -3285,7 +3321,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, // Finally insert and update the GP for this page. bnode->records[bnode->nr_records++] = ptr; - bnode->gp_snap = get_state_synchronize_rcu(); + get_state_synchronize_rcu_full(&bnode->gp_snap); atomic_inc(&(*krcp)->bulk_count[idx]); return true; @@ -4283,7 +4319,6 @@ int rcutree_prepare_cpu(unsigned int cpu) */ rnp = rdp->mynode; raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */ - rdp->beenonline = true; /* We have now been online. */ rdp->gp_seq = READ_ONCE(rnp->gp_seq); rdp->gp_seq_needed = rdp->gp_seq; rdp->cpu_no_qs.b.norm = true; @@ -4310,6 +4345,16 @@ static void rcutree_affinity_setting(unsigned int cpu, int outgoing) rcu_boost_kthread_setaffinity(rdp->mynode, outgoing); } +/* + * Has the specified (known valid) CPU ever been fully online? + */ +bool rcu_cpu_beenfullyonline(int cpu) +{ + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + return smp_load_acquire(&rdp->beenonline); +} + /* * Near the end of the CPU-online process. Pretty much all services * enabled, and the CPU is now very much alive. @@ -4368,15 +4413,16 @@ int rcutree_offline_cpu(unsigned int cpu) * Note that this function is special in that it is invoked directly * from the incoming CPU rather than from the cpuhp_step mechanism. * This is because this function must be invoked at a precise location. + * This incoming CPU must not have enabled interrupts yet. */ void rcu_cpu_starting(unsigned int cpu) { - unsigned long flags; unsigned long mask; struct rcu_data *rdp; struct rcu_node *rnp; bool newcpu; + lockdep_assert_irqs_disabled(); rdp = per_cpu_ptr(&rcu_data, cpu); if (rdp->cpu_started) return; @@ -4384,7 +4430,6 @@ void rcu_cpu_starting(unsigned int cpu) rnp = rdp->mynode; mask = rdp->grpmask; - local_irq_save(flags); arch_spin_lock(&rcu_state.ofl_lock); rcu_dynticks_eqs_online(); raw_spin_lock(&rcu_state.barrier_lock); @@ -4403,17 +4448,17 @@ void rcu_cpu_starting(unsigned int cpu) /* An incoming CPU should never be blocking a grace period. */ if (WARN_ON_ONCE(rnp->qsmask & mask)) { /* RCU waiting on incoming CPU? */ /* rcu_report_qs_rnp() *really* wants some flags to restore */ - unsigned long flags2; + unsigned long flags; - local_irq_save(flags2); + local_irq_save(flags); rcu_disable_urgency_upon_qs(rdp); /* Report QS -after- changing ->qsmaskinitnext! */ - rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags2); + rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags); } else { raw_spin_unlock_rcu_node(rnp); } arch_spin_unlock(&rcu_state.ofl_lock); - local_irq_restore(flags); + smp_store_release(&rdp->beenonline, true); smp_mb(); /* Ensure RCU read-side usage follows above initialization. */ } diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 3b7abb58157d..8239b39d945b 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -643,7 +643,7 @@ static void synchronize_rcu_expedited_wait(void) "O."[!!cpu_online(cpu)], "o."[!!(rdp->grpmask & rnp->expmaskinit)], "N."[!!(rdp->grpmask & rnp->expmaskinitnext)], - "D."[!!(rdp->cpu_no_qs.b.exp)]); + "D."[!!data_race(rdp->cpu_no_qs.b.exp)]); } } pr_cont(" } %lu jiffies s: %lu root: %#lx/%c\n", diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f2280616f9d5..43229d2b0c44 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1319,13 +1319,22 @@ lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) int cpu; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ + if (!mutex_trylock(&rcu_state.barrier_mutex)) + return 0; + /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); count += READ_ONCE(rdp->lazy_len); } + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_EMPTY; } @@ -1336,15 +1345,45 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; - /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { - struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); - int _count = READ_ONCE(rdp->lazy_len); + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + /* + * Protect against concurrent (de-)offloading. Otherwise nocb locking + * may be ignored or imbalanced. + */ + if (!mutex_trylock(&rcu_state.barrier_mutex)) { + /* + * But really don't insist if barrier_mutex is contended since we + * can't guarantee that it will never engage in a dependency + * chain involving memory allocation. The lock is seldom contended + * anyway. + */ + return 0; + } - if (_count == 0) + /* Snapshot count of all CPUs */ + for_each_cpu(cpu, rcu_nocb_mask) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count; + + if (WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp))) continue; + + if (!READ_ONCE(rdp->lazy_len)) + continue; + rcu_nocb_lock_irqsave(rdp, flags); - WRITE_ONCE(rdp->lazy_len, 0); + /* + * Recheck under the nocb lock. Since we are not holding the bypass + * lock we may still race with increments from the enqueuer but still + * we know for sure if there is at least one lazy callback. + */ + _count = READ_ONCE(rdp->lazy_len); + if (!_count) { + rcu_nocb_unlock_irqrestore(rdp, flags); + continue; + } + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); sc->nr_to_scan -= _count; @@ -1352,6 +1391,9 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (sc->nr_to_scan <= 0) break; } + + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_STOP; } diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 7b0fe741a088..41021080ad25 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -257,6 +257,8 @@ static void rcu_preempt_ctxt_queue(struct rcu_node *rnp, struct rcu_data *rdp) * GP should not be able to end until we report, so there should be * no need to check for a subsequent expedited GP. (Though we are * still in a quiescent state in any case.) + * + * Interrupts are disabled, so ->cpu_no_qs.b.exp cannot change. */ if (blkd_state & RCU_EXP_BLKD && rdp->cpu_no_qs.b.exp) rcu_report_exp_rdp(rdp); @@ -941,7 +943,7 @@ notrace void rcu_preempt_deferred_qs(struct task_struct *t) { struct rcu_data *rdp = this_cpu_ptr(&rcu_data); - if (rdp->cpu_no_qs.b.exp) + if (READ_ONCE(rdp->cpu_no_qs.b.exp)) rcu_report_exp_rdp(rdp); } diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ce51d4dc6803..f2d6e9dd8021 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2453,6 +2453,23 @@ config BITFIELD_KUNIT If unsure, say N. +config CHECKSUM_KUNIT + tristate "KUnit test checksum functions at runtime" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Enable this option to test the checksum functions at boot. + + KUnit tests run during boot and output the results to the debug log + in TAP format (http://testanything.org/). Only useful for kernel devs + running the KUnit test harness, and not intended for inclusion into a + production build. + + For more information on KUnit and unit tests in general please refer + to the KUnit documentation in Documentation/dev-tools/kunit/. + + If unsure, say N. + config HASH_KUNIT_TEST tristate "KUnit Test for integer hash functions" if !KUNIT_ALL_TESTS depends on KUNIT diff --git a/lib/Makefile b/lib/Makefile index 876fcdeae34e..cd37ec1d9a2f 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -377,6 +377,7 @@ obj-$(CONFIG_PLDMFW) += pldmfw/ # KUnit tests CFLAGS_bitfield_kunit.o := $(DISABLE_STRUCTLEAK_PLUGIN) obj-$(CONFIG_BITFIELD_KUNIT) += bitfield_kunit.o +obj-$(CONFIG_CHECKSUM_KUNIT) += checksum_kunit.o obj-$(CONFIG_LIST_KUNIT_TEST) += list-test.o obj-$(CONFIG_HASHTABLE_KUNIT_TEST) += hashtable_test.o obj-$(CONFIG_LINEAR_RANGES_TEST) += test_linear_ranges.o diff --git a/lib/checksum_kunit.c b/lib/checksum_kunit.c new file mode 100644 index 000000000000..ace3c4799fe1 --- /dev/null +++ b/lib/checksum_kunit.c @@ -0,0 +1,334 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Test cases csum_partial and csum_fold + */ + +#include +#include + +#define MAX_LEN 512 +#define MAX_ALIGN 64 +#define TEST_BUFLEN (MAX_LEN + MAX_ALIGN) + +static const __wsum random_init_sum = 0x2847aab; +static const u8 random_buf[] = { + 0xac, 0xd7, 0x76, 0x69, 0x6e, 0xf2, 0x93, 0x2c, 0x1f, 0xe0, 0xde, 0x86, + 0x8f, 0x54, 0x33, 0x90, 0x95, 0xbf, 0xff, 0xb9, 0xea, 0x62, 0x6e, 0xb5, + 0xd3, 0x4f, 0xf5, 0x60, 0x50, 0x5c, 0xc7, 0xfa, 0x6d, 0x1a, 0xc7, 0xf0, + 0xd2, 0x2c, 0x12, 0x3d, 0x88, 0xe3, 0x14, 0x21, 0xb1, 0x5e, 0x45, 0x31, + 0xa2, 0x85, 0x36, 0x76, 0xba, 0xd8, 0xad, 0xbb, 0x9e, 0x49, 0x8f, 0xf7, + 0xce, 0xea, 0xef, 0xca, 0x2c, 0x29, 0xf7, 0x15, 0x5c, 0x1d, 0x4d, 0x09, + 0x1f, 0xe2, 0x14, 0x31, 0x8c, 0x07, 0x57, 0x23, 0x1f, 0x6f, 0x03, 0xe1, + 0x93, 0x19, 0x53, 0x03, 0x45, 0x49, 0x9a, 0x3b, 0x8e, 0x0c, 0x12, 0x5d, + 0x8a, 0xb8, 0x9b, 0x8c, 0x9a, 0x03, 0xe5, 0xa2, 0x43, 0xd2, 0x3b, 0x4e, + 0x7e, 0x30, 0x3c, 0x22, 0x2d, 0xc5, 0xfc, 0x9e, 0xdb, 0xc6, 0xf9, 0x69, + 0x12, 0x39, 0x1f, 0xa0, 0x11, 0x0c, 0x3f, 0xf5, 0x53, 0xc9, 0x30, 0xfb, + 0xb0, 0xdd, 0x21, 0x1d, 0x34, 0xe2, 0x65, 0x30, 0xf1, 0xe8, 0x1b, 0xe7, + 0x55, 0x0d, 0xeb, 0xbd, 0xcc, 0x9d, 0x24, 0xa4, 0xad, 0xa7, 0x93, 0x47, + 0x19, 0x2e, 0xc4, 0x5c, 0x3b, 0xc7, 0x6d, 0x95, 0x0c, 0x47, 0x60, 0xaf, + 0x5b, 0x47, 0xee, 0xdc, 0x31, 0x31, 0x14, 0x12, 0x7e, 0x9e, 0x45, 0xb1, + 0xc1, 0x69, 0x4b, 0x84, 0xfc, 0x88, 0xc1, 0x9e, 0x46, 0xb4, 0xc2, 0x25, + 0xc5, 0x6c, 0x4c, 0x22, 0x58, 0x5c, 0xbe, 0xff, 0xea, 0x88, 0x88, 0x7a, + 0xcb, 0x1c, 0x5d, 0x63, 0xa1, 0xf2, 0x33, 0x0c, 0xa2, 0x16, 0x0b, 0x6e, + 0x2b, 0x79, 0x58, 0xf7, 0xac, 0xd3, 0x6a, 0x3f, 0x81, 0x57, 0x48, 0x45, + 0xe3, 0x7c, 0xdc, 0xd6, 0x34, 0x7e, 0xe6, 0x73, 0xfa, 0xcb, 0x31, 0x18, + 0xa9, 0x0b, 0xee, 0x6b, 0x99, 0xb9, 0x2d, 0xde, 0x22, 0x0e, 0x71, 0x57, + 0x0e, 0x9b, 0x11, 0xd1, 0x15, 0x41, 0xd0, 0x6b, 0x50, 0x8a, 0x23, 0x64, + 0xe3, 0x9c, 0xb3, 0x55, 0x09, 0xe9, 0x32, 0x67, 0xf9, 0xe0, 0x73, 0xf1, + 0x60, 0x66, 0x0b, 0x88, 0x79, 0x8d, 0x4b, 0x52, 0x83, 0x20, 0x26, 0x78, + 0x49, 0x27, 0xe7, 0x3e, 0x29, 0xa8, 0x18, 0x82, 0x41, 0xdd, 0x1e, 0xcc, + 0x3b, 0xc4, 0x65, 0xd1, 0x21, 0x40, 0x72, 0xb2, 0x87, 0x5e, 0x16, 0x10, + 0x80, 0x3f, 0x4b, 0x58, 0x1c, 0xc2, 0x79, 0x20, 0xf0, 0xe0, 0x80, 0xd3, + 0x52, 0xa5, 0x19, 0x6e, 0x47, 0x90, 0x08, 0xf5, 0x50, 0xe2, 0xd6, 0xae, + 0xe9, 0x2e, 0xdc, 0xd5, 0xb4, 0x90, 0x1f, 0x79, 0x49, 0x82, 0x21, 0x84, + 0xa0, 0xb5, 0x2f, 0xff, 0x30, 0x71, 0xed, 0x80, 0x68, 0xb1, 0x6d, 0xef, + 0xf6, 0xcf, 0xb8, 0x41, 0x79, 0xf5, 0x01, 0xbc, 0x0c, 0x9b, 0x0e, 0x06, + 0xf3, 0xb0, 0xbb, 0x97, 0xb8, 0xb1, 0xfd, 0x51, 0x4e, 0xef, 0x0a, 0x3d, + 0x7a, 0x3d, 0xbd, 0x61, 0x00, 0xa2, 0xb3, 0xf0, 0x1d, 0x77, 0x7b, 0x6c, + 0x01, 0x61, 0xa5, 0xa3, 0xdb, 0xd5, 0xd5, 0xf4, 0xb5, 0x28, 0x9f, 0x0a, + 0xa3, 0x82, 0x5f, 0x4b, 0x40, 0x0f, 0x05, 0x0e, 0x78, 0xed, 0xbf, 0x17, + 0xf6, 0x5a, 0x8a, 0x7d, 0xf9, 0x45, 0xc1, 0xd7, 0x1b, 0x9d, 0x6c, 0x07, + 0x88, 0xf3, 0xbc, 0xf1, 0xea, 0x28, 0x1f, 0xb8, 0x7a, 0x60, 0x3c, 0xce, + 0x3e, 0x50, 0xb2, 0x0b, 0xcf, 0xe5, 0x08, 0x1f, 0x48, 0x04, 0xf9, 0x35, + 0x29, 0x15, 0xbe, 0x82, 0x96, 0xc2, 0x55, 0x04, 0x6c, 0x19, 0x45, 0x29, + 0x0b, 0xb6, 0x49, 0x12, 0xfb, 0x8d, 0x1b, 0x75, 0x8b, 0xd9, 0x6a, 0x5c, + 0xbe, 0x46, 0x2b, 0x41, 0xfe, 0x21, 0xad, 0x1f, 0x75, 0xe7, 0x90, 0x3d, + 0xe1, 0xdf, 0x4b, 0xe1, 0x81, 0xe2, 0x17, 0x02, 0x7b, 0x58, 0x8b, 0x92, + 0x1a, 0xac, 0x46, 0xdd, 0x2e, 0xce, 0x40, 0x09 +}; +static const __sum16 expected_results[] = { + 0x82d0, 0x8224, 0xab23, 0xaaad, 0x41ad, 0x413f, 0x4f3e, 0x4eab, 0x22ab, + 0x228c, 0x428b, 0x41ad, 0xbbac, 0xbb1d, 0x671d, 0x66ea, 0xd6e9, 0xd654, + 0x1754, 0x1655, 0x5d54, 0x5c6a, 0xfa69, 0xf9fb, 0x44fb, 0x4428, 0xf527, + 0xf432, 0x9432, 0x93e2, 0x37e2, 0x371b, 0x3d1a, 0x3cad, 0x22ad, 0x21e6, + 0x31e5, 0x3113, 0x0513, 0x0501, 0xc800, 0xc778, 0xe477, 0xe463, 0xc363, + 0xc2b2, 0x64b2, 0x646d, 0x336d, 0x32cb, 0xadca, 0xad94, 0x3794, 0x36da, + 0x5ed9, 0x5e2c, 0xa32b, 0xa28d, 0x598d, 0x58fe, 0x61fd, 0x612f, 0x772e, + 0x763f, 0xac3e, 0xac12, 0x8312, 0x821b, 0x6d1b, 0x6cbf, 0x4fbf, 0x4f72, + 0x4672, 0x4653, 0x6452, 0x643e, 0x333e, 0x32b2, 0x2bb2, 0x2b5b, 0x085b, + 0x083c, 0x993b, 0x9938, 0xb837, 0xb7a4, 0x9ea4, 0x9e51, 0x9b51, 0x9b0c, + 0x520c, 0x5172, 0x1672, 0x15e4, 0x09e4, 0x09d2, 0xacd1, 0xac47, 0xf446, + 0xf3ab, 0x67ab, 0x6711, 0x6411, 0x632c, 0xc12b, 0xc0e8, 0xeee7, 0xeeac, + 0xa0ac, 0xa02e, 0x702e, 0x6ff2, 0x4df2, 0x4dc5, 0x88c4, 0x87c8, 0xe9c7, + 0xe8ec, 0x22ec, 0x21f3, 0xb8f2, 0xb8e0, 0x7fe0, 0x7fc1, 0xdfc0, 0xdfaf, + 0xd3af, 0xd370, 0xde6f, 0xde1c, 0x151c, 0x14ec, 0x19eb, 0x193b, 0x3c3a, + 0x3c19, 0x1f19, 0x1ee5, 0x3ce4, 0x3c7f, 0x0c7f, 0x0b8e, 0x238d, 0x2372, + 0x3c71, 0x3c1c, 0x2f1c, 0x2e31, 0x7130, 0x7064, 0xd363, 0xd33f, 0x2f3f, + 0x2e92, 0x8791, 0x86fe, 0x3ffe, 0x3fe5, 0x11e5, 0x1121, 0xb520, 0xb4e5, + 0xede4, 0xed77, 0x5877, 0x586b, 0x116b, 0x110b, 0x620a, 0x61af, 0x1aaf, + 0x19c1, 0x3dc0, 0x3d8f, 0x0c8f, 0x0c7b, 0xfa7a, 0xf9fc, 0x5bfc, 0x5bb7, + 0xaab6, 0xa9f5, 0x40f5, 0x40aa, 0xbca9, 0xbbad, 0x33ad, 0x32ec, 0x94eb, + 0x94a5, 0xe0a4, 0xdfe2, 0xbae2, 0xba1d, 0x4e1d, 0x4dd1, 0x2bd1, 0x2b79, + 0xcf78, 0xceba, 0xcfb9, 0xcecf, 0x46cf, 0x4647, 0xcc46, 0xcb7b, 0xaf7b, + 0xaf1e, 0x4c1e, 0x4b7d, 0x597c, 0x5949, 0x4d49, 0x4ca7, 0x36a7, 0x369c, + 0xc89b, 0xc870, 0x4f70, 0x4f18, 0x5817, 0x576b, 0x846a, 0x8400, 0x4500, + 0x447f, 0xed7e, 0xed36, 0xa836, 0xa753, 0x2b53, 0x2a77, 0x5476, 0x5442, + 0xd641, 0xd55b, 0x625b, 0x6161, 0x9660, 0x962f, 0x7e2f, 0x7d86, 0x7286, + 0x7198, 0x0698, 0x05ff, 0x4cfe, 0x4cd1, 0x6ed0, 0x6eae, 0x60ae, 0x603d, + 0x093d, 0x092f, 0x6e2e, 0x6e1d, 0x9d1c, 0x9d07, 0x5c07, 0x5b37, 0xf036, + 0xefe6, 0x65e6, 0x65c3, 0x01c3, 0x00e0, 0x64df, 0x642c, 0x0f2c, 0x0f23, + 0x2622, 0x25f0, 0xbeef, 0xbdf6, 0xddf5, 0xdd82, 0xec81, 0xec21, 0x8621, + 0x8616, 0xfe15, 0xfd9c, 0x709c, 0x7051, 0x1e51, 0x1dce, 0xfdcd, 0xfda7, + 0x85a7, 0x855e, 0x5e5e, 0x5d77, 0x1f77, 0x1f4e, 0x774d, 0x7735, 0xf534, + 0xf4f3, 0x17f3, 0x17d5, 0x4bd4, 0x4b99, 0x8798, 0x8733, 0xb632, 0xb611, + 0x7611, 0x759f, 0xc39e, 0xc317, 0x6517, 0x6501, 0x5501, 0x5481, 0x1581, + 0x1536, 0xbd35, 0xbd19, 0xfb18, 0xfa9f, 0xda9f, 0xd9af, 0xf9ae, 0xf92e, + 0x262e, 0x25dc, 0x80db, 0x80c2, 0x12c2, 0x127b, 0x827a, 0x8272, 0x8d71, + 0x8d21, 0xab20, 0xaa4a, 0xfc49, 0xfb60, 0xcd60, 0xcc84, 0xf783, 0xf6cf, + 0x66cf, 0x66b0, 0xedaf, 0xed66, 0x6b66, 0x6b45, 0xe744, 0xe6a4, 0x31a4, + 0x3175, 0x3274, 0x3244, 0xc143, 0xc056, 0x4056, 0x3fee, 0x8eed, 0x8e80, + 0x9f7f, 0x9e89, 0xcf88, 0xced0, 0x8dd0, 0x8d57, 0x9856, 0x9855, 0xdc54, + 0xdc48, 0x4148, 0x413a, 0x3b3a, 0x3a47, 0x8a46, 0x898b, 0xf28a, 0xf1d2, + 0x40d2, 0x3fd5, 0xeed4, 0xee86, 0xff85, 0xff7b, 0xc27b, 0xc201, 0x8501, + 0x8444, 0x2344, 0x2344, 0x8143, 0x8090, 0x908f, 0x9072, 0x1972, 0x18f7, + 0xacf6, 0xacf5, 0x4bf5, 0x4b50, 0xa84f, 0xa774, 0xd273, 0xd19e, 0xdd9d, + 0xdce8, 0xb4e8, 0xb449, 0xaa49, 0xa9a6, 0x27a6, 0x2747, 0xdc46, 0xdc06, + 0xcd06, 0xcd01, 0xbf01, 0xbe89, 0xd188, 0xd0c9, 0xb9c9, 0xb8d3, 0x5ed3, + 0x5e49, 0xe148, 0xe04f, 0x9b4f, 0x9a8e, 0xc38d, 0xc372, 0x2672, 0x2606, + 0x1f06, 0x1e7e, 0x2b7d, 0x2ac1, 0x39c0, 0x38d6, 0x10d6, 0x10b7, 0x58b6, + 0x583c, 0xf83b, 0xf7ff, 0x29ff, 0x29c1, 0xd9c0, 0xd90e, 0xce0e, 0xcd3f, + 0xe83e, 0xe836, 0xc936, 0xc8ee, 0xc4ee, 0xc3f5, 0x8ef5, 0x8ecc, 0x79cc, + 0x790e, 0xf70d, 0xf677, 0x3477, 0x3422, 0x3022, 0x2fb6, 0x16b6, 0x1671, + 0xed70, 0xed65, 0x3765, 0x371c, 0x251c, 0x2421, 0x9720, 0x9705, 0x2205, + 0x217a, 0x4879, 0x480f, 0xec0e, 0xeb50, 0xa550, 0xa525, 0x6425, 0x6327, + 0x4227, 0x417a, 0x227a, 0x2205, 0x3b04, 0x3a74, 0xfd73, 0xfc92, 0x1d92, + 0x1d47, 0x3c46, 0x3bc5, 0x59c4, 0x59ad, 0x57ad, 0x5732, 0xff31, 0xfea6, + 0x6ca6, 0x6c8c, 0xc08b, 0xc045, 0xe344, 0xe316, 0x1516, 0x14d6, +}; +static const __wsum init_sums_no_overflow[] = { + 0xffffffff, 0xfffffffb, 0xfffffbfb, 0xfffffbf7, 0xfffff7f7, 0xfffff7f3, + 0xfffff3f3, 0xfffff3ef, 0xffffefef, 0xffffefeb, 0xffffebeb, 0xffffebe7, + 0xffffe7e7, 0xffffe7e3, 0xffffe3e3, 0xffffe3df, 0xffffdfdf, 0xffffdfdb, + 0xffffdbdb, 0xffffdbd7, 0xffffd7d7, 0xffffd7d3, 0xffffd3d3, 0xffffd3cf, + 0xffffcfcf, 0xffffcfcb, 0xffffcbcb, 0xffffcbc7, 0xffffc7c7, 0xffffc7c3, + 0xffffc3c3, 0xffffc3bf, 0xffffbfbf, 0xffffbfbb, 0xffffbbbb, 0xffffbbb7, + 0xffffb7b7, 0xffffb7b3, 0xffffb3b3, 0xffffb3af, 0xffffafaf, 0xffffafab, + 0xffffabab, 0xffffaba7, 0xffffa7a7, 0xffffa7a3, 0xffffa3a3, 0xffffa39f, + 0xffff9f9f, 0xffff9f9b, 0xffff9b9b, 0xffff9b97, 0xffff9797, 0xffff9793, + 0xffff9393, 0xffff938f, 0xffff8f8f, 0xffff8f8b, 0xffff8b8b, 0xffff8b87, + 0xffff8787, 0xffff8783, 0xffff8383, 0xffff837f, 0xffff7f7f, 0xffff7f7b, + 0xffff7b7b, 0xffff7b77, 0xffff7777, 0xffff7773, 0xffff7373, 0xffff736f, + 0xffff6f6f, 0xffff6f6b, 0xffff6b6b, 0xffff6b67, 0xffff6767, 0xffff6763, + 0xffff6363, 0xffff635f, 0xffff5f5f, 0xffff5f5b, 0xffff5b5b, 0xffff5b57, + 0xffff5757, 0xffff5753, 0xffff5353, 0xffff534f, 0xffff4f4f, 0xffff4f4b, + 0xffff4b4b, 0xffff4b47, 0xffff4747, 0xffff4743, 0xffff4343, 0xffff433f, + 0xffff3f3f, 0xffff3f3b, 0xffff3b3b, 0xffff3b37, 0xffff3737, 0xffff3733, + 0xffff3333, 0xffff332f, 0xffff2f2f, 0xffff2f2b, 0xffff2b2b, 0xffff2b27, + 0xffff2727, 0xffff2723, 0xffff2323, 0xffff231f, 0xffff1f1f, 0xffff1f1b, + 0xffff1b1b, 0xffff1b17, 0xffff1717, 0xffff1713, 0xffff1313, 0xffff130f, + 0xffff0f0f, 0xffff0f0b, 0xffff0b0b, 0xffff0b07, 0xffff0707, 0xffff0703, + 0xffff0303, 0xffff02ff, 0xfffffefe, 0xfffffefa, 0xfffffafa, 0xfffffaf6, + 0xfffff6f6, 0xfffff6f2, 0xfffff2f2, 0xfffff2ee, 0xffffeeee, 0xffffeeea, + 0xffffeaea, 0xffffeae6, 0xffffe6e6, 0xffffe6e2, 0xffffe2e2, 0xffffe2de, + 0xffffdede, 0xffffdeda, 0xffffdada, 0xffffdad6, 0xffffd6d6, 0xffffd6d2, + 0xffffd2d2, 0xffffd2ce, 0xffffcece, 0xffffceca, 0xffffcaca, 0xffffcac6, + 0xffffc6c6, 0xffffc6c2, 0xffffc2c2, 0xffffc2be, 0xffffbebe, 0xffffbeba, + 0xffffbaba, 0xffffbab6, 0xffffb6b6, 0xffffb6b2, 0xffffb2b2, 0xffffb2ae, + 0xffffaeae, 0xffffaeaa, 0xffffaaaa, 0xffffaaa6, 0xffffa6a6, 0xffffa6a2, + 0xffffa2a2, 0xffffa29e, 0xffff9e9e, 0xffff9e9a, 0xffff9a9a, 0xffff9a96, + 0xffff9696, 0xffff9692, 0xffff9292, 0xffff928e, 0xffff8e8e, 0xffff8e8a, + 0xffff8a8a, 0xffff8a86, 0xffff8686, 0xffff8682, 0xffff8282, 0xffff827e, + 0xffff7e7e, 0xffff7e7a, 0xffff7a7a, 0xffff7a76, 0xffff7676, 0xffff7672, + 0xffff7272, 0xffff726e, 0xffff6e6e, 0xffff6e6a, 0xffff6a6a, 0xffff6a66, + 0xffff6666, 0xffff6662, 0xffff6262, 0xffff625e, 0xffff5e5e, 0xffff5e5a, + 0xffff5a5a, 0xffff5a56, 0xffff5656, 0xffff5652, 0xffff5252, 0xffff524e, + 0xffff4e4e, 0xffff4e4a, 0xffff4a4a, 0xffff4a46, 0xffff4646, 0xffff4642, + 0xffff4242, 0xffff423e, 0xffff3e3e, 0xffff3e3a, 0xffff3a3a, 0xffff3a36, + 0xffff3636, 0xffff3632, 0xffff3232, 0xffff322e, 0xffff2e2e, 0xffff2e2a, + 0xffff2a2a, 0xffff2a26, 0xffff2626, 0xffff2622, 0xffff2222, 0xffff221e, + 0xffff1e1e, 0xffff1e1a, 0xffff1a1a, 0xffff1a16, 0xffff1616, 0xffff1612, + 0xffff1212, 0xffff120e, 0xffff0e0e, 0xffff0e0a, 0xffff0a0a, 0xffff0a06, + 0xffff0606, 0xffff0602, 0xffff0202, 0xffff01fe, 0xfffffdfd, 0xfffffdf9, + 0xfffff9f9, 0xfffff9f5, 0xfffff5f5, 0xfffff5f1, 0xfffff1f1, 0xfffff1ed, + 0xffffeded, 0xffffede9, 0xffffe9e9, 0xffffe9e5, 0xffffe5e5, 0xffffe5e1, + 0xffffe1e1, 0xffffe1dd, 0xffffdddd, 0xffffddd9, 0xffffd9d9, 0xffffd9d5, + 0xffffd5d5, 0xffffd5d1, 0xffffd1d1, 0xffffd1cd, 0xffffcdcd, 0xffffcdc9, + 0xffffc9c9, 0xffffc9c5, 0xffffc5c5, 0xffffc5c1, 0xffffc1c1, 0xffffc1bd, + 0xffffbdbd, 0xffffbdb9, 0xffffb9b9, 0xffffb9b5, 0xffffb5b5, 0xffffb5b1, + 0xffffb1b1, 0xffffb1ad, 0xffffadad, 0xffffada9, 0xffffa9a9, 0xffffa9a5, + 0xffffa5a5, 0xffffa5a1, 0xffffa1a1, 0xffffa19d, 0xffff9d9d, 0xffff9d99, + 0xffff9999, 0xffff9995, 0xffff9595, 0xffff9591, 0xffff9191, 0xffff918d, + 0xffff8d8d, 0xffff8d89, 0xffff8989, 0xffff8985, 0xffff8585, 0xffff8581, + 0xffff8181, 0xffff817d, 0xffff7d7d, 0xffff7d79, 0xffff7979, 0xffff7975, + 0xffff7575, 0xffff7571, 0xffff7171, 0xffff716d, 0xffff6d6d, 0xffff6d69, + 0xffff6969, 0xffff6965, 0xffff6565, 0xffff6561, 0xffff6161, 0xffff615d, + 0xffff5d5d, 0xffff5d59, 0xffff5959, 0xffff5955, 0xffff5555, 0xffff5551, + 0xffff5151, 0xffff514d, 0xffff4d4d, 0xffff4d49, 0xffff4949, 0xffff4945, + 0xffff4545, 0xffff4541, 0xffff4141, 0xffff413d, 0xffff3d3d, 0xffff3d39, + 0xffff3939, 0xffff3935, 0xffff3535, 0xffff3531, 0xffff3131, 0xffff312d, + 0xffff2d2d, 0xffff2d29, 0xffff2929, 0xffff2925, 0xffff2525, 0xffff2521, + 0xffff2121, 0xffff211d, 0xffff1d1d, 0xffff1d19, 0xffff1919, 0xffff1915, + 0xffff1515, 0xffff1511, 0xffff1111, 0xffff110d, 0xffff0d0d, 0xffff0d09, + 0xffff0909, 0xffff0905, 0xffff0505, 0xffff0501, 0xffff0101, 0xffff00fd, + 0xfffffcfc, 0xfffffcf8, 0xfffff8f8, 0xfffff8f4, 0xfffff4f4, 0xfffff4f0, + 0xfffff0f0, 0xfffff0ec, 0xffffecec, 0xffffece8, 0xffffe8e8, 0xffffe8e4, + 0xffffe4e4, 0xffffe4e0, 0xffffe0e0, 0xffffe0dc, 0xffffdcdc, 0xffffdcd8, + 0xffffd8d8, 0xffffd8d4, 0xffffd4d4, 0xffffd4d0, 0xffffd0d0, 0xffffd0cc, + 0xffffcccc, 0xffffccc8, 0xffffc8c8, 0xffffc8c4, 0xffffc4c4, 0xffffc4c0, + 0xffffc0c0, 0xffffc0bc, 0xffffbcbc, 0xffffbcb8, 0xffffb8b8, 0xffffb8b4, + 0xffffb4b4, 0xffffb4b0, 0xffffb0b0, 0xffffb0ac, 0xffffacac, 0xffffaca8, + 0xffffa8a8, 0xffffa8a4, 0xffffa4a4, 0xffffa4a0, 0xffffa0a0, 0xffffa09c, + 0xffff9c9c, 0xffff9c98, 0xffff9898, 0xffff9894, 0xffff9494, 0xffff9490, + 0xffff9090, 0xffff908c, 0xffff8c8c, 0xffff8c88, 0xffff8888, 0xffff8884, + 0xffff8484, 0xffff8480, 0xffff8080, 0xffff807c, 0xffff7c7c, 0xffff7c78, + 0xffff7878, 0xffff7874, 0xffff7474, 0xffff7470, 0xffff7070, 0xffff706c, + 0xffff6c6c, 0xffff6c68, 0xffff6868, 0xffff6864, 0xffff6464, 0xffff6460, + 0xffff6060, 0xffff605c, 0xffff5c5c, 0xffff5c58, 0xffff5858, 0xffff5854, + 0xffff5454, 0xffff5450, 0xffff5050, 0xffff504c, 0xffff4c4c, 0xffff4c48, + 0xffff4848, 0xffff4844, 0xffff4444, 0xffff4440, 0xffff4040, 0xffff403c, + 0xffff3c3c, 0xffff3c38, 0xffff3838, 0xffff3834, 0xffff3434, 0xffff3430, + 0xffff3030, 0xffff302c, 0xffff2c2c, 0xffff2c28, 0xffff2828, 0xffff2824, + 0xffff2424, 0xffff2420, 0xffff2020, 0xffff201c, 0xffff1c1c, 0xffff1c18, + 0xffff1818, 0xffff1814, 0xffff1414, 0xffff1410, 0xffff1010, 0xffff100c, + 0xffff0c0c, 0xffff0c08, 0xffff0808, 0xffff0804, 0xffff0404, 0xffff0400, + 0xffff0000, 0xfffffffb, +}; + +static u8 tmp_buf[TEST_BUFLEN]; + +#define full_csum(buff, len, sum) csum_fold(csum_partial(buff, len, sum)) + +#define CHECK_EQ(lhs, rhs) KUNIT_ASSERT_EQ(test, lhs, rhs) + +static void assert_setup_correct(struct kunit *test) +{ + CHECK_EQ(sizeof(random_buf) / sizeof(random_buf[0]), MAX_LEN); + CHECK_EQ(sizeof(expected_results) / sizeof(expected_results[0]), + MAX_LEN); + CHECK_EQ(sizeof(init_sums_no_overflow) / + sizeof(init_sums_no_overflow[0]), + MAX_LEN); +} + +/* + * Test with randomized input (pre determined random with known results). + */ +static void test_csum_fixed_random_inputs(struct kunit *test) +{ + int len, align; + __wsum result, expec, sum; + + assert_setup_correct(test); + for (align = 0; align < TEST_BUFLEN; ++align) { + memcpy(&tmp_buf[align], random_buf, + min(MAX_LEN, TEST_BUFLEN - align)); + for (len = 0; len < MAX_LEN && (align + len) < TEST_BUFLEN; + ++len) { + /* + * Test the precomputed random input. + */ + sum = random_init_sum; + result = full_csum(&tmp_buf[align], len, sum); + expec = expected_results[len]; + CHECK_EQ(result, expec); + } + } +} + +/* + * All ones input test. If there are any missing carry operations, it fails. + */ +static void test_csum_all_carry_inputs(struct kunit *test) +{ + int len, align; + __wsum result, expec, sum; + + assert_setup_correct(test); + memset(tmp_buf, 0xff, TEST_BUFLEN); + for (align = 0; align < TEST_BUFLEN; ++align) { + for (len = 0; len < MAX_LEN && (align + len) < TEST_BUFLEN; + ++len) { + /* + * All carries from input and initial sum. + */ + sum = 0xffffffff; + result = full_csum(&tmp_buf[align], len, sum); + expec = (len & 1) ? 0xff00 : 0; + CHECK_EQ(result, expec); + + /* + * All carries from input. + */ + sum = 0; + result = full_csum(&tmp_buf[align], len, sum); + if (len & 1) + expec = 0xff00; + else if (len) + expec = 0; + else + expec = 0xffff; + CHECK_EQ(result, expec); + } + } +} + +/* + * Test with input that alone doesn't cause any carries. By selecting the + * maximum initial sum, this allows us to test that there are no carries + * where there shouldn't be. + */ +static void test_csum_no_carry_inputs(struct kunit *test) +{ + int len, align; + __wsum result, expec, sum; + + assert_setup_correct(test); + memset(tmp_buf, 0x4, TEST_BUFLEN); + for (align = 0; align < TEST_BUFLEN; ++align) { + for (len = 0; len < MAX_LEN && (align + len) < TEST_BUFLEN; + ++len) { + /* + * Expect no carries. + */ + sum = init_sums_no_overflow[len]; + result = full_csum(&tmp_buf[align], len, sum); + expec = 0; + CHECK_EQ(result, expec); + + /* + * Expect one carry. + */ + sum = init_sums_no_overflow[len] + 1; + result = full_csum(&tmp_buf[align], len, sum); + expec = len ? 0xfffe : 0xffff; + CHECK_EQ(result, expec); + } + } +} + +static struct kunit_case __refdata checksum_test_cases[] = { + KUNIT_CASE(test_csum_fixed_random_inputs), + KUNIT_CASE(test_csum_all_carry_inputs), + KUNIT_CASE(test_csum_no_carry_inputs), + {} +}; + +static struct kunit_suite checksum_test_suite = { + .name = "checksum", + .test_cases = checksum_test_cases, +}; + +kunit_test_suites(&checksum_test_suite); + +MODULE_AUTHOR("Noah Goldstein "); +MODULE_LICENSE("GPL"); diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c index 0cea31c27b23..ce6749af374d 100644 --- a/lib/kunit/executor_test.c +++ b/lib/kunit/executor_test.c @@ -125,11 +125,6 @@ kunit_test_suites(&executor_test_suite); /* Test helpers */ -static void kfree_res_free(struct kunit_resource *res) -{ - kfree(res->data); -} - /* Use the resource API to register a call to kfree(to_free). * Since we never actually use the resource, it's safe to use on const data. */ @@ -138,8 +133,10 @@ static void kfree_at_end(struct kunit *test, const void *to_free) /* kfree() handles NULL already, but avoid allocating a no-op cleanup. */ if (IS_ERR_OR_NULL(to_free)) return; - kunit_alloc_resource(test, NULL, kfree_res_free, GFP_KERNEL, - (void *)to_free); + + kunit_add_action(test, + (kunit_action_t *)kfree, + (void *)to_free); } static struct kunit_suite *alloc_fake_suite(struct kunit *test, diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c index cd8b7e51d02b..b69b689ea850 100644 --- a/lib/kunit/kunit-example-test.c +++ b/lib/kunit/kunit-example-test.c @@ -41,6 +41,16 @@ static int example_test_init(struct kunit *test) return 0; } +/* + * This is run once after each test case, see the comment on + * example_test_suite for more information. + */ +static void example_test_exit(struct kunit *test) +{ + kunit_info(test, "cleaning up\n"); +} + + /* * This is run once before all test cases in the suite. * See the comment on example_test_suite for more information. @@ -52,6 +62,16 @@ static int example_test_init_suite(struct kunit_suite *suite) return 0; } +/* + * This is run once after all test cases in the suite. + * See the comment on example_test_suite for more information. + */ +static void example_test_exit_suite(struct kunit_suite *suite) +{ + kunit_info(suite, "exiting suite\n"); +} + + /* * This test should always be skipped. */ @@ -167,6 +187,39 @@ static void example_static_stub_test(struct kunit *test) KUNIT_EXPECT_EQ(test, add_one(1), 2); } +static const struct example_param { + int value; +} example_params_array[] = { + { .value = 2, }, + { .value = 1, }, + { .value = 0, }, +}; + +static void example_param_get_desc(const struct example_param *p, char *desc) +{ + snprintf(desc, KUNIT_PARAM_DESC_SIZE, "example value %d", p->value); +} + +KUNIT_ARRAY_PARAM(example, example_params_array, example_param_get_desc); + +/* + * This test shows the use of params. + */ +static void example_params_test(struct kunit *test) +{ + const struct example_param *param = test->param_value; + + /* By design, param pointer will not be NULL */ + KUNIT_ASSERT_NOT_NULL(test, param); + + /* Test can be skipped on unsupported param values */ + if (!param->value) + kunit_skip(test, "unsupported param value"); + + /* You can use param values for parameterized testing */ + KUNIT_EXPECT_EQ(test, param->value % param->value, 0); +} + /* * Here we make a list of all the test cases we want to add to the test suite * below. @@ -183,6 +236,7 @@ static struct kunit_case example_test_cases[] = { KUNIT_CASE(example_mark_skipped_test), KUNIT_CASE(example_all_expect_macros_test), KUNIT_CASE(example_static_stub_test), + KUNIT_CASE_PARAM(example_params_test, example_gen_params), {} }; @@ -211,7 +265,9 @@ static struct kunit_case example_test_cases[] = { static struct kunit_suite example_test_suite = { .name = "example", .init = example_test_init, + .exit = example_test_exit, .suite_init = example_test_init_suite, + .suite_exit = example_test_exit_suite, .test_cases = example_test_cases, }; diff --git a/lib/kunit/kunit-test.c b/lib/kunit/kunit-test.c index 42e44caa1bdd..83d8e90ca7a2 100644 --- a/lib/kunit/kunit-test.c +++ b/lib/kunit/kunit-test.c @@ -112,7 +112,7 @@ struct kunit_test_resource_context { struct kunit test; bool is_resource_initialized; int allocate_order[2]; - int free_order[2]; + int free_order[4]; }; static int fake_resource_init(struct kunit_resource *res, void *context) @@ -403,6 +403,88 @@ static void kunit_resource_test_named(struct kunit *test) KUNIT_EXPECT_TRUE(test, list_empty(&test->resources)); } +static void increment_int(void *ctx) +{ + int *i = (int *)ctx; + (*i)++; +} + +static void kunit_resource_test_action(struct kunit *test) +{ + int num_actions = 0; + + kunit_add_action(test, increment_int, &num_actions); + KUNIT_EXPECT_EQ(test, num_actions, 0); + kunit_cleanup(test); + KUNIT_EXPECT_EQ(test, num_actions, 1); + + /* Once we've cleaned up, the action queue is empty. */ + kunit_cleanup(test); + KUNIT_EXPECT_EQ(test, num_actions, 1); + + /* Check the same function can be deferred multiple times. */ + kunit_add_action(test, increment_int, &num_actions); + kunit_add_action(test, increment_int, &num_actions); + kunit_cleanup(test); + KUNIT_EXPECT_EQ(test, num_actions, 3); +} +static void kunit_resource_test_remove_action(struct kunit *test) +{ + int num_actions = 0; + + kunit_add_action(test, increment_int, &num_actions); + KUNIT_EXPECT_EQ(test, num_actions, 0); + + kunit_remove_action(test, increment_int, &num_actions); + kunit_cleanup(test); + KUNIT_EXPECT_EQ(test, num_actions, 0); +} +static void kunit_resource_test_release_action(struct kunit *test) +{ + int num_actions = 0; + + kunit_add_action(test, increment_int, &num_actions); + KUNIT_EXPECT_EQ(test, num_actions, 0); + /* Runs immediately on trigger. */ + kunit_release_action(test, increment_int, &num_actions); + KUNIT_EXPECT_EQ(test, num_actions, 1); + + /* Doesn't run again on test exit. */ + kunit_cleanup(test); + KUNIT_EXPECT_EQ(test, num_actions, 1); +} +static void action_order_1(void *ctx) +{ + struct kunit_test_resource_context *res_ctx = (struct kunit_test_resource_context *)ctx; + + KUNIT_RESOURCE_TEST_MARK_ORDER(res_ctx, free_order, 1); + kunit_log(KERN_INFO, current->kunit_test, "action_order_1"); +} +static void action_order_2(void *ctx) +{ + struct kunit_test_resource_context *res_ctx = (struct kunit_test_resource_context *)ctx; + + KUNIT_RESOURCE_TEST_MARK_ORDER(res_ctx, free_order, 2); + kunit_log(KERN_INFO, current->kunit_test, "action_order_2"); +} +static void kunit_resource_test_action_ordering(struct kunit *test) +{ + struct kunit_test_resource_context *ctx = test->priv; + + kunit_add_action(test, action_order_1, ctx); + kunit_add_action(test, action_order_2, ctx); + kunit_add_action(test, action_order_1, ctx); + kunit_add_action(test, action_order_2, ctx); + kunit_remove_action(test, action_order_1, ctx); + kunit_release_action(test, action_order_2, ctx); + kunit_cleanup(test); + + /* [2 is triggered] [2], [(1 is cancelled)] [1] */ + KUNIT_EXPECT_EQ(test, ctx->free_order[0], 2); + KUNIT_EXPECT_EQ(test, ctx->free_order[1], 2); + KUNIT_EXPECT_EQ(test, ctx->free_order[2], 1); +} + static int kunit_resource_test_init(struct kunit *test) { struct kunit_test_resource_context *ctx = @@ -434,6 +516,10 @@ static struct kunit_case kunit_resource_test_cases[] = { KUNIT_CASE(kunit_resource_test_proper_free_ordering), KUNIT_CASE(kunit_resource_test_static), KUNIT_CASE(kunit_resource_test_named), + KUNIT_CASE(kunit_resource_test_action), + KUNIT_CASE(kunit_resource_test_remove_action), + KUNIT_CASE(kunit_resource_test_release_action), + KUNIT_CASE(kunit_resource_test_action_ordering), {} }; diff --git a/lib/kunit/resource.c b/lib/kunit/resource.c index c414df922f34..f0209252b179 100644 --- a/lib/kunit/resource.c +++ b/lib/kunit/resource.c @@ -77,3 +77,102 @@ int kunit_destroy_resource(struct kunit *test, kunit_resource_match_t match, return 0; } EXPORT_SYMBOL_GPL(kunit_destroy_resource); + +struct kunit_action_ctx { + struct kunit_resource res; + kunit_action_t *func; + void *ctx; +}; + +static void __kunit_action_free(struct kunit_resource *res) +{ + struct kunit_action_ctx *action_ctx = container_of(res, struct kunit_action_ctx, res); + + action_ctx->func(action_ctx->ctx); +} + + +int kunit_add_action(struct kunit *test, void (*action)(void *), void *ctx) +{ + struct kunit_action_ctx *action_ctx; + + KUNIT_ASSERT_NOT_NULL_MSG(test, action, "Tried to action a NULL function!"); + + action_ctx = kzalloc(sizeof(*action_ctx), GFP_KERNEL); + if (!action_ctx) + return -ENOMEM; + + action_ctx->func = action; + action_ctx->ctx = ctx; + + action_ctx->res.should_kfree = true; + /* As init is NULL, this cannot fail. */ + __kunit_add_resource(test, NULL, __kunit_action_free, &action_ctx->res, action_ctx); + + return 0; +} +EXPORT_SYMBOL_GPL(kunit_add_action); + +int kunit_add_action_or_reset(struct kunit *test, void (*action)(void *), + void *ctx) +{ + int res = kunit_add_action(test, action, ctx); + + if (res) + action(ctx); + return res; +} +EXPORT_SYMBOL_GPL(kunit_add_action_or_reset); + +static bool __kunit_action_match(struct kunit *test, + struct kunit_resource *res, void *match_data) +{ + struct kunit_action_ctx *match_ctx = (struct kunit_action_ctx *)match_data; + struct kunit_action_ctx *res_ctx = container_of(res, struct kunit_action_ctx, res); + + /* Make sure this is a free function. */ + if (res->free != __kunit_action_free) + return false; + + /* Both the function and context data should match. */ + return (match_ctx->func == res_ctx->func) && (match_ctx->ctx == res_ctx->ctx); +} + +void kunit_remove_action(struct kunit *test, + kunit_action_t *action, + void *ctx) +{ + struct kunit_action_ctx match_ctx; + struct kunit_resource *res; + + match_ctx.func = action; + match_ctx.ctx = ctx; + + res = kunit_find_resource(test, __kunit_action_match, &match_ctx); + if (res) { + /* Remove the free function so we don't run the action. */ + res->free = NULL; + kunit_remove_resource(test, res); + kunit_put_resource(res); + } +} +EXPORT_SYMBOL_GPL(kunit_remove_action); + +void kunit_release_action(struct kunit *test, + kunit_action_t *action, + void *ctx) +{ + struct kunit_action_ctx match_ctx; + struct kunit_resource *res; + + match_ctx.func = action; + match_ctx.ctx = ctx; + + res = kunit_find_resource(test, __kunit_action_match, &match_ctx); + if (res) { + kunit_remove_resource(test, res); + /* We have to put() this here, else free won't be called. */ + kunit_put_resource(res); + } +} +EXPORT_SYMBOL_GPL(kunit_release_action); diff --git a/lib/kunit/test.c b/lib/kunit/test.c index e2910b261112..84e4666555c9 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -185,16 +185,28 @@ static void kunit_print_suite_start(struct kunit_suite *suite) kunit_suite_num_test_cases(suite)); } -static void kunit_print_ok_not_ok(void *test_or_suite, - bool is_test, +/* Currently supported test levels */ +enum { + KUNIT_LEVEL_SUITE = 0, + KUNIT_LEVEL_CASE, + KUNIT_LEVEL_CASE_PARAM, +}; + +static void kunit_print_ok_not_ok(struct kunit *test, + unsigned int test_level, enum kunit_status status, size_t test_number, const char *description, const char *directive) { - struct kunit_suite *suite = is_test ? NULL : test_or_suite; - struct kunit *test = is_test ? test_or_suite : NULL; const char *directive_header = (status == KUNIT_SKIPPED) ? " # SKIP " : ""; + const char *directive_body = (status == KUNIT_SKIPPED) ? directive : ""; + + /* + * When test is NULL assume that results are from the suite + * and today suite results are expected at level 0 only. + */ + WARN(!test && test_level, "suite test level can't be %u!\n", test_level); /* * We do not log the test suite results as doing so would @@ -203,17 +215,18 @@ static void kunit_print_ok_not_ok(void *test_or_suite, * separately seq_printf() the suite results for the debugfs * representation. */ - if (suite) + if (!test) pr_info("%s %zd %s%s%s\n", kunit_status_to_ok_not_ok(status), test_number, description, directive_header, - (status == KUNIT_SKIPPED) ? directive : ""); + directive_body); else kunit_log(KERN_INFO, test, - KUNIT_SUBTEST_INDENT "%s %zd %s%s%s", + "%*s%s %zd %s%s%s", + KUNIT_INDENT_LEN * test_level, "", kunit_status_to_ok_not_ok(status), test_number, description, directive_header, - (status == KUNIT_SKIPPED) ? directive : ""); + directive_body); } enum kunit_status kunit_suite_has_succeeded(struct kunit_suite *suite) @@ -239,7 +252,7 @@ static size_t kunit_suite_counter = 1; static void kunit_print_suite_end(struct kunit_suite *suite) { - kunit_print_ok_not_ok((void *)suite, false, + kunit_print_ok_not_ok(NULL, KUNIT_LEVEL_SUITE, kunit_suite_has_succeeded(suite), kunit_suite_counter++, suite->name, @@ -310,7 +323,7 @@ static void kunit_fail(struct kunit *test, const struct kunit_loc *loc, string_stream_destroy(stream); } -static void __noreturn kunit_abort(struct kunit *test) +void __noreturn __kunit_abort(struct kunit *test) { kunit_try_catch_throw(&test->try_catch); /* Does not return. */ @@ -322,8 +335,9 @@ static void __noreturn kunit_abort(struct kunit *test) */ WARN_ONCE(true, "Throw could not abort from test!\n"); } +EXPORT_SYMBOL_GPL(__kunit_abort); -void kunit_do_failed_assertion(struct kunit *test, +void __kunit_do_failed_assertion(struct kunit *test, const struct kunit_loc *loc, enum kunit_assert_type type, const struct kunit_assert *assert, @@ -340,11 +354,8 @@ void kunit_do_failed_assertion(struct kunit *test, kunit_fail(test, loc, type, assert, assert_format, &message); va_end(args); - - if (type == KUNIT_ASSERTION) - kunit_abort(test); } -EXPORT_SYMBOL_GPL(kunit_do_failed_assertion); +EXPORT_SYMBOL_GPL(__kunit_do_failed_assertion); void kunit_init_test(struct kunit *test, const char *name, char *log) { @@ -419,15 +430,54 @@ static void kunit_try_run_case(void *data) * thread will resume control and handle any necessary clean up. */ kunit_run_case_internal(test, suite, test_case); - /* This line may never be reached. */ +} + +static void kunit_try_run_case_cleanup(void *data) +{ + struct kunit_try_catch_context *ctx = data; + struct kunit *test = ctx->test; + struct kunit_suite *suite = ctx->suite; + + current->kunit_test = test; + kunit_run_case_cleanup(test, suite); } +static void kunit_catch_run_case_cleanup(void *data) +{ + struct kunit_try_catch_context *ctx = data; + struct kunit *test = ctx->test; + int try_exit_code = kunit_try_catch_get_result(&test->try_catch); + + /* It is always a failure if cleanup aborts. */ + kunit_set_failure(test); + + if (try_exit_code) { + /* + * Test case could not finish, we have no idea what state it is + * in, so don't do clean up. + */ + if (try_exit_code == -ETIMEDOUT) { + kunit_err(test, "test case cleanup timed out\n"); + /* + * Unknown internal error occurred preventing test case from + * running, so there is nothing to clean up. + */ + } else { + kunit_err(test, "internal error occurred during test case cleanup: %d\n", + try_exit_code); + } + return; + } + + kunit_err(test, "test aborted during cleanup. continuing without cleaning up\n"); +} + + static void kunit_catch_run_case(void *data) { struct kunit_try_catch_context *ctx = data; struct kunit *test = ctx->test; - struct kunit_suite *suite = ctx->suite; int try_exit_code = kunit_try_catch_get_result(&test->try_catch); if (try_exit_code) { @@ -448,12 +498,6 @@ static void kunit_catch_run_case(void *data) } return; } - - /* - * Test case was run, but aborted. It is the test case's business as to - * whether it failed or not, we just need to clean up. - */ - kunit_run_case_cleanup(test, suite); } /* @@ -478,6 +522,13 @@ static void kunit_run_case_catch_errors(struct kunit_suite *suite, context.test_case = test_case; kunit_try_catch_run(try_catch, &context); + /* Now run the cleanup */ + kunit_try_catch_init(try_catch, + test, + kunit_try_run_case_cleanup, + kunit_catch_run_case_cleanup); + kunit_try_catch_run(try_catch, &context); + /* Propagate the parameter result to the test case. */ if (test->status == KUNIT_FAILURE) test_case->status = KUNIT_FAILURE; @@ -585,11 +636,11 @@ int kunit_run_tests(struct kunit_suite *suite) "param-%d", test.param_index); } - kunit_log(KERN_INFO, &test, - KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT - "%s %d %s", - kunit_status_to_ok_not_ok(test.status), - test.param_index + 1, param_desc); + kunit_print_ok_not_ok(&test, KUNIT_LEVEL_CASE_PARAM, + test.status, + test.param_index + 1, + param_desc, + test.status_comment); /* Get next param. */ param_desc[0] = '\0'; @@ -603,7 +654,7 @@ int kunit_run_tests(struct kunit_suite *suite) kunit_print_test_stats(&test, param_stats); - kunit_print_ok_not_ok(&test, true, test_case->status, + kunit_print_ok_not_ok(&test, KUNIT_LEVEL_CASE, test_case->status, kunit_test_case_num(suite, test_case), test_case->name, test.status_comment); @@ -712,58 +763,28 @@ static struct notifier_block kunit_mod_nb = { }; #endif -struct kunit_kmalloc_array_params { - size_t n; - size_t size; - gfp_t gfp; -}; - -static int kunit_kmalloc_array_init(struct kunit_resource *res, void *context) -{ - struct kunit_kmalloc_array_params *params = context; - - res->data = kmalloc_array(params->n, params->size, params->gfp); - if (!res->data) - return -ENOMEM; - - return 0; -} - -static void kunit_kmalloc_array_free(struct kunit_resource *res) -{ - kfree(res->data); -} - void *kunit_kmalloc_array(struct kunit *test, size_t n, size_t size, gfp_t gfp) { - struct kunit_kmalloc_array_params params = { - .size = size, - .n = n, - .gfp = gfp - }; + void *data; - return kunit_alloc_resource(test, - kunit_kmalloc_array_init, - kunit_kmalloc_array_free, - gfp, - ¶ms); + data = kmalloc_array(n, size, gfp); + + if (!data) + return NULL; + + if (kunit_add_action_or_reset(test, (kunit_action_t *)kfree, data) != 0) + return NULL; + + return data; } EXPORT_SYMBOL_GPL(kunit_kmalloc_array); -static inline bool kunit_kfree_match(struct kunit *test, - struct kunit_resource *res, void *match_data) -{ - /* Only match resources allocated with kunit_kmalloc() and friends. */ - return res->free == kunit_kmalloc_array_free && res->data == match_data; -} - void kunit_kfree(struct kunit *test, const void *ptr) { if (!ptr) return; - if (kunit_destroy_resource(test, kunit_kfree_match, (void *)ptr)) - KUNIT_FAIL(test, "kunit_kfree: %px already freed or not allocated by kunit", ptr); + kunit_release_action(test, (kunit_action_t *)kfree, (void *)ptr); } EXPORT_SYMBOL_GPL(kunit_kfree); diff --git a/tools/arch/x86/kcpuid/.gitignore b/tools/arch/x86/kcpuid/.gitignore new file mode 100644 index 000000000000..1b8541bc8dd0 --- /dev/null +++ b/tools/arch/x86/kcpuid/.gitignore @@ -0,0 +1 @@ +kcpuid diff --git a/tools/arch/x86/kcpuid/kcpuid.c b/tools/arch/x86/kcpuid/kcpuid.c index 416f5b35dd8f..24b7d017ec2c 100644 --- a/tools/arch/x86/kcpuid/kcpuid.c +++ b/tools/arch/x86/kcpuid/kcpuid.c @@ -517,15 +517,16 @@ static void show_range(struct cpuid_range *range) static inline struct cpuid_func *index_to_func(u32 index) { struct cpuid_range *range; + u32 func_idx; range = (index & 0x80000000) ? leafs_ext : leafs_basic; - index &= 0x7FFFFFFF; + func_idx = index & 0xffff; - if (((index & 0xFFFF) + 1) > (u32)range->nr) { + if ((func_idx + 1) > (u32)range->nr) { printf("ERR: invalid input index (0x%x)\n", index); return NULL; } - return &range->funcs[index]; + return &range->funcs[func_idx]; } static void show_info(void) diff --git a/tools/include/nolibc/Makefile b/tools/include/nolibc/Makefile index 9839feafd38a..64d67b080744 100644 --- a/tools/include/nolibc/Makefile +++ b/tools/include/nolibc/Makefile @@ -25,8 +25,23 @@ endif nolibc_arch := $(patsubst arm64,aarch64,$(ARCH)) arch_file := arch-$(nolibc_arch).h -all_files := ctype.h errno.h nolibc.h signal.h stackprotector.h std.h stdint.h \ - stdio.h stdlib.h string.h sys.h time.h types.h unistd.h +all_files := \ + compiler.h \ + ctype.h \ + errno.h \ + nolibc.h \ + signal.h \ + stackprotector.h \ + std.h \ + stdint.h \ + stdlib.h \ + string.h \ + sys.h \ + time.h \ + types.h \ + unistd.h \ + stdio.h \ + # install all headers needed to support a bare-metal compiler all: headers diff --git a/tools/include/nolibc/arch-aarch64.h b/tools/include/nolibc/arch-aarch64.h index 383baddef701..11f294a406b7 100644 --- a/tools/include/nolibc/arch-aarch64.h +++ b/tools/include/nolibc/arch-aarch64.h @@ -7,6 +7,8 @@ #ifndef _NOLIBC_ARCH_AARCH64_H #define _NOLIBC_ARCH_AARCH64_H +#include "compiler.h" + /* The struct returned by the newfstatat() syscall. Differs slightly from the * x86_64's stat one by field ordering, so be careful. */ @@ -173,27 +175,30 @@ char **environ __attribute__((weak)); const unsigned long *_auxv __attribute__((weak)); /* startup code */ -void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) _start(void) +void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __no_stack_protector _start(void) { __asm__ volatile ( - "ldr x0, [sp]\n" // argc (x0) was in the stack - "add x1, sp, 8\n" // argv (x1) = sp - "lsl x2, x0, 3\n" // envp (x2) = 8*argc ... - "add x2, x2, 8\n" // + 8 (skip null) - "add x2, x2, x1\n" // + argv - "adrp x3, environ\n" // x3 = &environ (high bits) - "str x2, [x3, #:lo12:environ]\n" // store envp into environ - "mov x4, x2\n" // search for auxv (follows NULL after last env) +#ifdef _NOLIBC_STACKPROTECTOR + "bl __stack_chk_init\n" /* initialize stack protector */ +#endif + "ldr x0, [sp]\n" /* argc (x0) was in the stack */ + "add x1, sp, 8\n" /* argv (x1) = sp */ + "lsl x2, x0, 3\n" /* envp (x2) = 8*argc ... */ + "add x2, x2, 8\n" /* + 8 (skip null) */ + "add x2, x2, x1\n" /* + argv */ + "adrp x3, environ\n" /* x3 = &environ (high bits) */ + "str x2, [x3, #:lo12:environ]\n" /* store envp into environ */ + "mov x4, x2\n" /* search for auxv (follows NULL after last env) */ "0:\n" - "ldr x5, [x4], 8\n" // x5 = *x4; x4 += 8 - "cbnz x5, 0b\n" // and stop at NULL after last env - "adrp x3, _auxv\n" // x3 = &_auxv (high bits) - "str x4, [x3, #:lo12:_auxv]\n" // store x4 into _auxv - "and sp, x1, -16\n" // sp must be 16-byte aligned in the callee - "bl main\n" // main() returns the status code, we'll exit with it. - "mov x8, 93\n" // NR_exit == 93 + "ldr x5, [x4], 8\n" /* x5 = *x4; x4 += 8 */ + "cbnz x5, 0b\n" /* and stop at NULL after last env */ + "adrp x3, _auxv\n" /* x3 = &_auxv (high bits) */ + "str x4, [x3, #:lo12:_auxv]\n" /* store x4 into _auxv */ + "and sp, x1, -16\n" /* sp must be 16-byte aligned in the callee */ + "bl main\n" /* main() returns the status code, we'll exit with it. */ + "mov x8, 93\n" /* NR_exit == 93 */ "svc #0\n" ); __builtin_unreachable(); } -#endif // _NOLIBC_ARCH_AARCH64_H +#endif /* _NOLIBC_ARCH_AARCH64_H */ diff --git a/tools/include/nolibc/arch-arm.h b/tools/include/nolibc/arch-arm.h index 42499f23e73c..ca4c66987497 100644 --- a/tools/include/nolibc/arch-arm.h +++ b/tools/include/nolibc/arch-arm.h @@ -7,6 +7,8 @@ #ifndef _NOLIBC_ARCH_ARM_H #define _NOLIBC_ARCH_ARM_H +#include "compiler.h" + /* The struct returned by the stat() syscall, 32-bit only, the syscall returns * exactly 56 bytes (stops before the unused array). In big endian, the format * differs as devices are returned as short only. @@ -196,41 +198,67 @@ struct sys_stat_struct { _arg1; \ }) +#define my_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6) \ +({ \ + register long _num __asm__(_NOLIBC_SYSCALL_REG) = (num); \ + register long _arg1 __asm__ ("r0") = (long)(arg1); \ + register long _arg2 __asm__ ("r1") = (long)(arg2); \ + register long _arg3 __asm__ ("r2") = (long)(arg3); \ + register long _arg4 __asm__ ("r3") = (long)(arg4); \ + register long _arg5 __asm__ ("r4") = (long)(arg5); \ + register long _arg6 __asm__ ("r5") = (long)(arg6); \ + \ + __asm__ volatile ( \ + _NOLIBC_THUMB_SET_R7 \ + "svc #0\n" \ + _NOLIBC_THUMB_RESTORE_R7 \ + : "=r"(_arg1), "=r" (_num) \ + : "r"(_arg1), "r"(_arg2), "r"(_arg3), "r"(_arg4), "r"(_arg5), \ + "r"(_arg6), "r"(_num) \ + : "memory", "cc", "lr" \ + ); \ + _arg1; \ +}) + + char **environ __attribute__((weak)); const unsigned long *_auxv __attribute__((weak)); /* startup code */ -void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) _start(void) +void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __no_stack_protector _start(void) { __asm__ volatile ( - "pop {%r0}\n" // argc was in the stack - "mov %r1, %sp\n" // argv = sp +#ifdef _NOLIBC_STACKPROTECTOR + "bl __stack_chk_init\n" /* initialize stack protector */ +#endif + "pop {%r0}\n" /* argc was in the stack */ + "mov %r1, %sp\n" /* argv = sp */ - "add %r2, %r0, $1\n" // envp = (argc + 1) ... - "lsl %r2, %r2, $2\n" // * 4 ... - "add %r2, %r2, %r1\n" // + argv - "ldr %r3, 1f\n" // r3 = &environ (see below) - "str %r2, [r3]\n" // store envp into environ + "add %r2, %r0, $1\n" /* envp = (argc + 1) ... */ + "lsl %r2, %r2, $2\n" /* * 4 ... */ + "add %r2, %r2, %r1\n" /* + argv */ + "ldr %r3, 1f\n" /* r3 = &environ (see below) */ + "str %r2, [r3]\n" /* store envp into environ */ - "mov r4, r2\n" // search for auxv (follows NULL after last env) + "mov r4, r2\n" /* search for auxv (follows NULL after last env) */ "0:\n" - "mov r5, r4\n" // r5 = r4 - "add r4, r4, #4\n" // r4 += 4 - "ldr r5,[r5]\n" // r5 = *r5 = *(r4-4) - "cmp r5, #0\n" // and stop at NULL after last env + "mov r5, r4\n" /* r5 = r4 */ + "add r4, r4, #4\n" /* r4 += 4 */ + "ldr r5,[r5]\n" /* r5 = *r5 = *(r4-4) */ + "cmp r5, #0\n" /* and stop at NULL after last env */ "bne 0b\n" - "ldr %r3, 2f\n" // r3 = &_auxv (low bits) - "str r4, [r3]\n" // store r4 into _auxv + "ldr %r3, 2f\n" /* r3 = &_auxv (low bits) */ + "str r4, [r3]\n" /* store r4 into _auxv */ - "mov %r3, $8\n" // AAPCS : sp must be 8-byte aligned in the - "neg %r3, %r3\n" // callee, and bl doesn't push (lr=pc) - "and %r3, %r3, %r1\n" // so we do sp = r1(=sp) & r3(=-8); - "mov %sp, %r3\n" // + "mov %r3, $8\n" /* AAPCS : sp must be 8-byte aligned in the */ + "neg %r3, %r3\n" /* callee, and bl doesn't push (lr=pc) */ + "and %r3, %r3, %r1\n" /* so we do sp = r1(=sp) & r3(=-8); */ + "mov %sp, %r3\n" - "bl main\n" // main() returns the status code, we'll exit with it. - "movs r7, $1\n" // NR_exit == 1 + "bl main\n" /* main() returns the status code, we'll exit with it. */ + "movs r7, $1\n" /* NR_exit == 1 */ "svc $0x00\n" - ".align 2\n" // below are the pointers to a few variables + ".align 2\n" /* below are the pointers to a few variables */ "1:\n" ".word environ\n" "2:\n" @@ -239,4 +267,4 @@ void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) _start(void) __builtin_unreachable(); } -#endif // _NOLIBC_ARCH_ARM_H +#endif /* _NOLIBC_ARCH_ARM_H */ diff --git a/tools/include/nolibc/arch-i386.h b/tools/include/nolibc/arch-i386.h index 2d98d78fd3f3..3d672d925e9e 100644 --- a/tools/include/nolibc/arch-i386.h +++ b/tools/include/nolibc/arch-i386.h @@ -7,6 +7,8 @@ #ifndef _NOLIBC_ARCH_I386_H #define _NOLIBC_ARCH_I386_H +#include "compiler.h" + /* The struct returned by the stat() syscall, 32-bit only, the syscall returns * exactly 56 bytes (stops before the unused array). */ @@ -181,8 +183,6 @@ struct sys_stat_struct { char **environ __attribute__((weak)); const unsigned long *_auxv __attribute__((weak)); -#define __ARCH_SUPPORTS_STACK_PROTECTOR - /* startup code */ /* * i386 System V ABI mandates: @@ -190,35 +190,35 @@ const unsigned long *_auxv __attribute__((weak)); * 2) The deepest stack frame should be set to zero * */ -void __attribute__((weak,noreturn,optimize("omit-frame-pointer"),no_stack_protector)) _start(void) +void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __no_stack_protector _start(void) { __asm__ volatile ( -#ifdef NOLIBC_STACKPROTECTOR - "call __stack_chk_init\n" // initialize stack protector +#ifdef _NOLIBC_STACKPROTECTOR + "call __stack_chk_init\n" /* initialize stack protector */ #endif - "pop %eax\n" // argc (first arg, %eax) - "mov %esp, %ebx\n" // argv[] (second arg, %ebx) - "lea 4(%ebx,%eax,4),%ecx\n" // then a NULL then envp (third arg, %ecx) - "mov %ecx, environ\n" // save environ - "xor %ebp, %ebp\n" // zero the stack frame - "mov %ecx, %edx\n" // search for auxv (follows NULL after last env) + "pop %eax\n" /* argc (first arg, %eax) */ + "mov %esp, %ebx\n" /* argv[] (second arg, %ebx) */ + "lea 4(%ebx,%eax,4),%ecx\n" /* then a NULL then envp (third arg, %ecx) */ + "mov %ecx, environ\n" /* save environ */ + "xor %ebp, %ebp\n" /* zero the stack frame */ + "mov %ecx, %edx\n" /* search for auxv (follows NULL after last env) */ "0:\n" - "add $4, %edx\n" // search for auxv using edx, it follows the - "cmp -4(%edx), %ebp\n" // ... NULL after last env (ebp is zero here) + "add $4, %edx\n" /* search for auxv using edx, it follows the */ + "cmp -4(%edx), %ebp\n" /* ... NULL after last env (ebp is zero here) */ "jnz 0b\n" - "mov %edx, _auxv\n" // save it into _auxv - "and $-16, %esp\n" // x86 ABI : esp must be 16-byte aligned before - "sub $4, %esp\n" // the call instruction (args are aligned) - "push %ecx\n" // push all registers on the stack so that we - "push %ebx\n" // support both regparm and plain stack modes + "mov %edx, _auxv\n" /* save it into _auxv */ + "and $-16, %esp\n" /* x86 ABI : esp must be 16-byte aligned before */ + "sub $4, %esp\n" /* the call instruction (args are aligned) */ + "push %ecx\n" /* push all registers on the stack so that we */ + "push %ebx\n" /* support both regparm and plain stack modes */ "push %eax\n" - "call main\n" // main() returns the status code in %eax - "mov %eax, %ebx\n" // retrieve exit code (32-bit int) - "movl $1, %eax\n" // NR_exit == 1 - "int $0x80\n" // exit now - "hlt\n" // ensure it does not + "call main\n" /* main() returns the status code in %eax */ + "mov %eax, %ebx\n" /* retrieve exit code (32-bit int) */ + "movl $1, %eax\n" /* NR_exit == 1 */ + "int $0x80\n" /* exit now */ + "hlt\n" /* ensure it does not */ ); __builtin_unreachable(); } -#endif // _NOLIBC_ARCH_I386_H +#endif /* _NOLIBC_ARCH_I386_H */ diff --git a/tools/include/nolibc/arch-loongarch.h b/tools/include/nolibc/arch-loongarch.h index 029ee3cd6baf..ad3f266e7093 100644 --- a/tools/include/nolibc/arch-loongarch.h +++ b/tools/include/nolibc/arch-loongarch.h @@ -7,6 +7,8 @@ #ifndef _NOLIBC_ARCH_LOONGARCH_H #define _NOLIBC_ARCH_LOONGARCH_H +#include "compiler.h" + /* Syscalls for LoongArch : * - stack is 16-byte aligned * - syscall number is passed in a7 @@ -158,7 +160,7 @@ const unsigned long *_auxv __attribute__((weak)); #define LONG_ADDI "addi.w" #define LONG_SLL "slli.w" #define LONG_BSTRINS "bstrins.w" -#else // __loongarch_grlen == 64 +#else /* __loongarch_grlen == 64 */ #define LONGLOG "3" #define SZREG "8" #define REG_L "ld.d" @@ -170,31 +172,34 @@ const unsigned long *_auxv __attribute__((weak)); #endif /* startup code */ -void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) _start(void) +void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __no_stack_protector _start(void) { __asm__ volatile ( - REG_L " $a0, $sp, 0\n" // argc (a0) was in the stack - LONG_ADDI " $a1, $sp, "SZREG"\n" // argv (a1) = sp + SZREG - LONG_SLL " $a2, $a0, "LONGLOG"\n" // envp (a2) = SZREG*argc ... - LONG_ADDI " $a2, $a2, "SZREG"\n" // + SZREG (skip null) - LONG_ADD " $a2, $a2, $a1\n" // + argv +#ifdef _NOLIBC_STACKPROTECTOR + "bl __stack_chk_init\n" /* initialize stack protector */ +#endif + REG_L " $a0, $sp, 0\n" /* argc (a0) was in the stack */ + LONG_ADDI " $a1, $sp, "SZREG"\n" /* argv (a1) = sp + SZREG */ + LONG_SLL " $a2, $a0, "LONGLOG"\n" /* envp (a2) = SZREG*argc ... */ + LONG_ADDI " $a2, $a2, "SZREG"\n" /* + SZREG (skip null) */ + LONG_ADD " $a2, $a2, $a1\n" /* + argv */ - "move $a3, $a2\n" // iterate a3 over envp to find auxv (after NULL) - "0:\n" // do { - REG_L " $a4, $a3, 0\n" // a4 = *a3; - LONG_ADDI " $a3, $a3, "SZREG"\n" // a3 += sizeof(void*); - "bne $a4, $zero, 0b\n" // } while (a4); - "la.pcrel $a4, _auxv\n" // a4 = &_auxv - LONG_S " $a3, $a4, 0\n" // store a3 into _auxv + "move $a3, $a2\n" /* iterate a3 over envp to find auxv (after NULL) */ + "0:\n" /* do { */ + REG_L " $a4, $a3, 0\n" /* a4 = *a3; */ + LONG_ADDI " $a3, $a3, "SZREG"\n" /* a3 += sizeof(void*); */ + "bne $a4, $zero, 0b\n" /* } while (a4); */ + "la.pcrel $a4, _auxv\n" /* a4 = &_auxv */ + LONG_S " $a3, $a4, 0\n" /* store a3 into _auxv */ - "la.pcrel $a3, environ\n" // a3 = &environ - LONG_S " $a2, $a3, 0\n" // store envp(a2) into environ - LONG_BSTRINS " $sp, $zero, 3, 0\n" // sp must be 16-byte aligned - "bl main\n" // main() returns the status code, we'll exit with it. - "li.w $a7, 93\n" // NR_exit == 93 + "la.pcrel $a3, environ\n" /* a3 = &environ */ + LONG_S " $a2, $a3, 0\n" /* store envp(a2) into environ */ + LONG_BSTRINS " $sp, $zero, 3, 0\n" /* sp must be 16-byte aligned */ + "bl main\n" /* main() returns the status code, we'll exit with it. */ + "li.w $a7, 93\n" /* NR_exit == 93 */ "syscall 0\n" ); __builtin_unreachable(); } -#endif // _NOLIBC_ARCH_LOONGARCH_H +#endif /* _NOLIBC_ARCH_LOONGARCH_H */ diff --git a/tools/include/nolibc/arch-mips.h b/tools/include/nolibc/arch-mips.h index bf83432d23ed..db24e0837a39 100644 --- a/tools/include/nolibc/arch-mips.h +++ b/tools/include/nolibc/arch-mips.h @@ -7,6 +7,8 @@ #ifndef _NOLIBC_ARCH_MIPS_H #define _NOLIBC_ARCH_MIPS_H +#include "compiler.h" + /* The struct returned by the stat() syscall. 88 bytes are returned by the * syscall. */ @@ -180,45 +182,49 @@ char **environ __attribute__((weak)); const unsigned long *_auxv __attribute__((weak)); /* startup code, note that it's called __start on MIPS */ -void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __start(void) +void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __no_stack_protector __start(void) { __asm__ volatile ( - //".set nomips16\n" + /*".set nomips16\n"*/ ".set push\n" ".set noreorder\n" ".option pic0\n" - //".ent __start\n" - //"__start:\n" - "lw $a0,($sp)\n" // argc was in the stack - "addiu $a1, $sp, 4\n" // argv = sp + 4 - "sll $a2, $a0, 2\n" // a2 = argc * 4 - "add $a2, $a2, $a1\n" // envp = argv + 4*argc ... - "addiu $a2, $a2, 4\n" // ... + 4 - "lui $a3, %hi(environ)\n" // load environ into a3 (hi) - "addiu $a3, %lo(environ)\n" // load environ into a3 (lo) - "sw $a2,($a3)\n" // store envp(a2) into environ +#ifdef _NOLIBC_STACKPROTECTOR + "jal __stack_chk_init\n" /* initialize stack protector */ + "nop\n" /* delayed slot */ +#endif + /*".ent __start\n"*/ + /*"__start:\n"*/ + "lw $a0,($sp)\n" /* argc was in the stack */ + "addiu $a1, $sp, 4\n" /* argv = sp + 4 */ + "sll $a2, $a0, 2\n" /* a2 = argc * 4 */ + "add $a2, $a2, $a1\n" /* envp = argv + 4*argc ... */ + "addiu $a2, $a2, 4\n" /* ... + 4 */ + "lui $a3, %hi(environ)\n" /* load environ into a3 (hi) */ + "addiu $a3, %lo(environ)\n" /* load environ into a3 (lo) */ + "sw $a2,($a3)\n" /* store envp(a2) into environ */ - "move $t0, $a2\n" // iterate t0 over envp, look for NULL - "0:" // do { - "lw $a3, ($t0)\n" // a3=*(t0); - "bne $a3, $0, 0b\n" // } while (a3); - "addiu $t0, $t0, 4\n" // delayed slot: t0+=4; - "lui $a3, %hi(_auxv)\n" // load _auxv into a3 (hi) - "addiu $a3, %lo(_auxv)\n" // load _auxv into a3 (lo) - "sw $t0, ($a3)\n" // store t0 into _auxv + "move $t0, $a2\n" /* iterate t0 over envp, look for NULL */ + "0:" /* do { */ + "lw $a3, ($t0)\n" /* a3=*(t0); */ + "bne $a3, $0, 0b\n" /* } while (a3); */ + "addiu $t0, $t0, 4\n" /* delayed slot: t0+=4; */ + "lui $a3, %hi(_auxv)\n" /* load _auxv into a3 (hi) */ + "addiu $a3, %lo(_auxv)\n" /* load _auxv into a3 (lo) */ + "sw $t0, ($a3)\n" /* store t0 into _auxv */ "li $t0, -8\n" - "and $sp, $sp, $t0\n" // sp must be 8-byte aligned - "addiu $sp,$sp,-16\n" // the callee expects to save a0..a3 there! - "jal main\n" // main() returns the status code, we'll exit with it. - "nop\n" // delayed slot - "move $a0, $v0\n" // retrieve 32-bit exit code from v0 - "li $v0, 4001\n" // NR_exit == 4001 + "and $sp, $sp, $t0\n" /* sp must be 8-byte aligned */ + "addiu $sp,$sp,-16\n" /* the callee expects to save a0..a3 there! */ + "jal main\n" /* main() returns the status code, we'll exit with it. */ + "nop\n" /* delayed slot */ + "move $a0, $v0\n" /* retrieve 32-bit exit code from v0 */ + "li $v0, 4001\n" /* NR_exit == 4001 */ "syscall\n" - //".end __start\n" + /*".end __start\n"*/ ".set pop\n" ); __builtin_unreachable(); } -#endif // _NOLIBC_ARCH_MIPS_H +#endif /* _NOLIBC_ARCH_MIPS_H */ diff --git a/tools/include/nolibc/arch-riscv.h b/tools/include/nolibc/arch-riscv.h index e197fcb10ac0..a2e8564e66d6 100644 --- a/tools/include/nolibc/arch-riscv.h +++ b/tools/include/nolibc/arch-riscv.h @@ -7,6 +7,8 @@ #ifndef _NOLIBC_ARCH_RISCV_H #define _NOLIBC_ARCH_RISCV_H +#include "compiler.h" + struct sys_stat_struct { unsigned long st_dev; /* Device. */ unsigned long st_ino; /* File serial number. */ @@ -33,9 +35,13 @@ struct sys_stat_struct { #if __riscv_xlen == 64 #define PTRLOG "3" #define SZREG "8" +#define REG_L "ld" +#define REG_S "sd" #elif __riscv_xlen == 32 #define PTRLOG "2" #define SZREG "4" +#define REG_L "lw" +#define REG_S "sw" #endif /* Syscalls for RISCV : @@ -174,35 +180,38 @@ char **environ __attribute__((weak)); const unsigned long *_auxv __attribute__((weak)); /* startup code */ -void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) _start(void) +void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __no_stack_protector _start(void) { __asm__ volatile ( ".option push\n" ".option norelax\n" "lla gp, __global_pointer$\n" ".option pop\n" - "lw a0, 0(sp)\n" // argc (a0) was in the stack - "add a1, sp, "SZREG"\n" // argv (a1) = sp - "slli a2, a0, "PTRLOG"\n" // envp (a2) = SZREG*argc ... - "add a2, a2, "SZREG"\n" // + SZREG (skip null) - "add a2,a2,a1\n" // + argv +#ifdef _NOLIBC_STACKPROTECTOR + "call __stack_chk_init\n" /* initialize stack protector */ +#endif + REG_L" a0, 0(sp)\n" /* argc (a0) was in the stack */ + "add a1, sp, "SZREG"\n" /* argv (a1) = sp */ + "slli a2, a0, "PTRLOG"\n" /* envp (a2) = SZREG*argc ... */ + "add a2, a2, "SZREG"\n" /* + SZREG (skip null) */ + "add a2,a2,a1\n" /* + argv */ - "add a3, a2, zero\n" // iterate a3 over envp to find auxv (after NULL) - "0:\n" // do { - "ld a4, 0(a3)\n" // a4 = *a3; - "add a3, a3, "SZREG"\n" // a3 += sizeof(void*); - "bne a4, zero, 0b\n" // } while (a4); - "lui a4, %hi(_auxv)\n" // a4 = &_auxv (high bits) - "sd a3, %lo(_auxv)(a4)\n" // store a3 into _auxv + "add a3, a2, zero\n" /* iterate a3 over envp to find auxv (after NULL) */ + "0:\n" /* do { */ + REG_L" a4, 0(a3)\n" /* a4 = *a3; */ + "add a3, a3, "SZREG"\n" /* a3 += sizeof(void*); */ + "bne a4, zero, 0b\n" /* } while (a4); */ + "lui a4, %hi(_auxv)\n" /* a4 = &_auxv (high bits) */ + REG_S" a3, %lo(_auxv)(a4)\n" /* store a3 into _auxv */ - "lui a3, %hi(environ)\n" // a3 = &environ (high bits) - "sd a2,%lo(environ)(a3)\n" // store envp(a2) into environ - "andi sp,a1,-16\n" // sp must be 16-byte aligned - "call main\n" // main() returns the status code, we'll exit with it. - "li a7, 93\n" // NR_exit == 93 + "lui a3, %hi(environ)\n" /* a3 = &environ (high bits) */ + REG_S" a2,%lo(environ)(a3)\n"/* store envp(a2) into environ */ + "andi sp,a1,-16\n" /* sp must be 16-byte aligned */ + "call main\n" /* main() returns the status code, we'll exit with it. */ + "li a7, 93\n" /* NR_exit == 93 */ "ecall\n" ); __builtin_unreachable(); } -#endif // _NOLIBC_ARCH_RISCV_H +#endif /* _NOLIBC_ARCH_RISCV_H */ diff --git a/tools/include/nolibc/arch-s390.h b/tools/include/nolibc/arch-s390.h index 6b0e54ed543d..516dff5bff8b 100644 --- a/tools/include/nolibc/arch-s390.h +++ b/tools/include/nolibc/arch-s390.h @@ -5,8 +5,11 @@ #ifndef _NOLIBC_ARCH_S390_H #define _NOLIBC_ARCH_S390_H +#include #include +#include "compiler.h" + /* The struct returned by the stat() syscall, equivalent to stat64(). The * syscall returns 116 bytes and stops in the middle of __unused. */ @@ -163,7 +166,7 @@ char **environ __attribute__((weak)); const unsigned long *_auxv __attribute__((weak)); /* startup code */ -void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) _start(void) +void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __no_stack_protector _start(void) { __asm__ volatile ( "lg %r2,0(%r15)\n" /* argument count */ @@ -223,4 +226,12 @@ void *sys_mmap(void *addr, size_t length, int prot, int flags, int fd, return (void *)my_syscall1(__NR_mmap, &args); } #define sys_mmap sys_mmap -#endif // _NOLIBC_ARCH_S390_H + +static __attribute__((unused)) +pid_t sys_fork(void) +{ + return my_syscall5(__NR_clone, 0, SIGCHLD, 0, 0, 0); +} +#define sys_fork sys_fork + +#endif /* _NOLIBC_ARCH_S390_H */ diff --git a/tools/include/nolibc/arch-x86_64.h b/tools/include/nolibc/arch-x86_64.h index f7f2a11d4c3b..6fc4d8392742 100644 --- a/tools/include/nolibc/arch-x86_64.h +++ b/tools/include/nolibc/arch-x86_64.h @@ -7,6 +7,8 @@ #ifndef _NOLIBC_ARCH_X86_64_H #define _NOLIBC_ARCH_X86_64_H +#include "compiler.h" + /* The struct returned by the stat() syscall, equivalent to stat64(). The * syscall returns 116 bytes and stops in the middle of __unused. */ @@ -181,8 +183,6 @@ struct sys_stat_struct { char **environ __attribute__((weak)); const unsigned long *_auxv __attribute__((weak)); -#define __ARCH_SUPPORTS_STACK_PROTECTOR - /* startup code */ /* * x86-64 System V ABI mandates: @@ -190,31 +190,31 @@ const unsigned long *_auxv __attribute__((weak)); * 2) The deepest stack frame should be zero (the %rbp). * */ -void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) _start(void) +void __attribute__((weak,noreturn,optimize("omit-frame-pointer"))) __no_stack_protector _start(void) { __asm__ volatile ( -#ifdef NOLIBC_STACKPROTECTOR - "call __stack_chk_init\n" // initialize stack protector +#ifdef _NOLIBC_STACKPROTECTOR + "call __stack_chk_init\n" /* initialize stack protector */ #endif - "pop %rdi\n" // argc (first arg, %rdi) - "mov %rsp, %rsi\n" // argv[] (second arg, %rsi) - "lea 8(%rsi,%rdi,8),%rdx\n" // then a NULL then envp (third arg, %rdx) - "mov %rdx, environ\n" // save environ - "xor %ebp, %ebp\n" // zero the stack frame - "mov %rdx, %rax\n" // search for auxv (follows NULL after last env) + "pop %rdi\n" /* argc (first arg, %rdi) */ + "mov %rsp, %rsi\n" /* argv[] (second arg, %rsi) */ + "lea 8(%rsi,%rdi,8),%rdx\n" /* then a NULL then envp (third arg, %rdx) */ + "mov %rdx, environ\n" /* save environ */ + "xor %ebp, %ebp\n" /* zero the stack frame */ + "mov %rdx, %rax\n" /* search for auxv (follows NULL after last env) */ "0:\n" - "add $8, %rax\n" // search for auxv using rax, it follows the - "cmp -8(%rax), %rbp\n" // ... NULL after last env (rbp is zero here) + "add $8, %rax\n" /* search for auxv using rax, it follows the */ + "cmp -8(%rax), %rbp\n" /* ... NULL after last env (rbp is zero here) */ "jnz 0b\n" - "mov %rax, _auxv\n" // save it into _auxv - "and $-16, %rsp\n" // x86 ABI : esp must be 16-byte aligned before call - "call main\n" // main() returns the status code, we'll exit with it. - "mov %eax, %edi\n" // retrieve exit code (32 bit) - "mov $60, %eax\n" // NR_exit == 60 - "syscall\n" // really exit - "hlt\n" // ensure it does not return + "mov %rax, _auxv\n" /* save it into _auxv */ + "and $-16, %rsp\n" /* x86 ABI : esp must be 16-byte aligned before call */ + "call main\n" /* main() returns the status code, we'll exit with it. */ + "mov %eax, %edi\n" /* retrieve exit code (32 bit) */ + "mov $60, %eax\n" /* NR_exit == 60 */ + "syscall\n" /* really exit */ + "hlt\n" /* ensure it does not return */ ); __builtin_unreachable(); } -#endif // _NOLIBC_ARCH_X86_64_H +#endif /* _NOLIBC_ARCH_X86_64_H */ diff --git a/tools/include/nolibc/arch.h b/tools/include/nolibc/arch.h index 2d5386a8d6aa..82b43935650f 100644 --- a/tools/include/nolibc/arch.h +++ b/tools/include/nolibc/arch.h @@ -7,7 +7,7 @@ * the syscall declarations and the _start code definition. This is the only * global part. On all architectures the kernel puts everything in the stack * before jumping to _start just above us, without any return address (_start - * is not a function but an entry pint). So at the stack pointer we find argc. + * is not a function but an entry point). So at the stack pointer we find argc. * Then argv[] begins, and ends at the first NULL. Then we have envp which * starts and ends with a NULL as well. So envp=argv+argc+1. */ diff --git a/tools/include/nolibc/compiler.h b/tools/include/nolibc/compiler.h new file mode 100644 index 000000000000..beddc3665d69 --- /dev/null +++ b/tools/include/nolibc/compiler.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * NOLIBC compiler support header + * Copyright (C) 2023 Thomas Weißschuh + */ +#ifndef _NOLIBC_COMPILER_H +#define _NOLIBC_COMPILER_H + +#if defined(__SSP__) || defined(__SSP_STRONG__) || defined(__SSP_ALL__) || defined(__SSP_EXPLICIT__) + +#define _NOLIBC_STACKPROTECTOR + +#endif /* defined(__SSP__) ... */ + +#if defined(__has_attribute) +# if __has_attribute(no_stack_protector) +# define __no_stack_protector __attribute__((no_stack_protector)) +# else +# define __no_stack_protector __attribute__((__optimize__("-fno-stack-protector"))) +# endif +#else +# define __no_stack_protector __attribute__((__optimize__("-fno-stack-protector"))) +#endif /* defined(__has_attribute) */ + +#endif /* _NOLIBC_COMPILER_H */ diff --git a/tools/include/nolibc/nolibc.h b/tools/include/nolibc/nolibc.h index 04739a6293c4..05a228a6ee78 100644 --- a/tools/include/nolibc/nolibc.h +++ b/tools/include/nolibc/nolibc.h @@ -99,11 +99,11 @@ #include "sys.h" #include "ctype.h" #include "signal.h" +#include "unistd.h" #include "stdio.h" #include "stdlib.h" #include "string.h" #include "time.h" -#include "unistd.h" #include "stackprotector.h" /* Used by programs to avoid std includes */ diff --git a/tools/include/nolibc/stackprotector.h b/tools/include/nolibc/stackprotector.h index d119cbbbc256..88f7b2d098ff 100644 --- a/tools/include/nolibc/stackprotector.h +++ b/tools/include/nolibc/stackprotector.h @@ -7,13 +7,9 @@ #ifndef _NOLIBC_STACKPROTECTOR_H #define _NOLIBC_STACKPROTECTOR_H -#include "arch.h" +#include "compiler.h" -#if defined(NOLIBC_STACKPROTECTOR) - -#if !defined(__ARCH_SUPPORTS_STACK_PROTECTOR) -#error "nolibc does not support stack protectors on this arch" -#endif +#if defined(_NOLIBC_STACKPROTECTOR) #include "sys.h" #include "stdlib.h" @@ -41,13 +37,14 @@ void __stack_chk_fail_local(void) __attribute__((weak,section(".data.nolibc_stack_chk"))) uintptr_t __stack_chk_guard; -__attribute__((weak,no_stack_protector,section(".text.nolibc_stack_chk"))) +__attribute__((weak,section(".text.nolibc_stack_chk"))) __no_stack_protector void __stack_chk_init(void) { my_syscall3(__NR_getrandom, &__stack_chk_guard, sizeof(__stack_chk_guard), 0); - /* a bit more randomness in case getrandom() fails */ - __stack_chk_guard ^= (uintptr_t) &__stack_chk_guard; + /* a bit more randomness in case getrandom() fails, ensure the guard is never 0 */ + if (__stack_chk_guard != (uintptr_t) &__stack_chk_guard) + __stack_chk_guard ^= (uintptr_t) &__stack_chk_guard; } -#endif // defined(NOLIBC_STACKPROTECTOR) +#endif /* defined(_NOLIBC_STACKPROTECTOR) */ -#endif // _NOLIBC_STACKPROTECTOR_H +#endif /* _NOLIBC_STACKPROTECTOR_H */ diff --git a/tools/include/nolibc/stdint.h b/tools/include/nolibc/stdint.h index c1ce4f5e0603..4b282435a59a 100644 --- a/tools/include/nolibc/stdint.h +++ b/tools/include/nolibc/stdint.h @@ -36,8 +36,8 @@ typedef ssize_t int_fast16_t; typedef size_t uint_fast16_t; typedef ssize_t int_fast32_t; typedef size_t uint_fast32_t; -typedef ssize_t int_fast64_t; -typedef size_t uint_fast64_t; +typedef int64_t int_fast64_t; +typedef uint64_t uint_fast64_t; typedef int64_t intmax_t; typedef uint64_t uintmax_t; @@ -84,16 +84,30 @@ typedef uint64_t uintmax_t; #define INT_FAST8_MIN INT8_MIN #define INT_FAST16_MIN INTPTR_MIN #define INT_FAST32_MIN INTPTR_MIN -#define INT_FAST64_MIN INTPTR_MIN +#define INT_FAST64_MIN INT64_MIN #define INT_FAST8_MAX INT8_MAX #define INT_FAST16_MAX INTPTR_MAX #define INT_FAST32_MAX INTPTR_MAX -#define INT_FAST64_MAX INTPTR_MAX +#define INT_FAST64_MAX INT64_MAX #define UINT_FAST8_MAX UINT8_MAX #define UINT_FAST16_MAX SIZE_MAX #define UINT_FAST32_MAX SIZE_MAX -#define UINT_FAST64_MAX SIZE_MAX +#define UINT_FAST64_MAX UINT64_MAX + +#ifndef INT_MIN +#define INT_MIN (-__INT_MAX__ - 1) +#endif +#ifndef INT_MAX +#define INT_MAX __INT_MAX__ +#endif + +#ifndef LONG_MIN +#define LONG_MIN (-__LONG_MAX__ - 1) +#endif +#ifndef LONG_MAX +#define LONG_MAX __LONG_MAX__ +#endif #endif /* _NOLIBC_STDINT_H */ diff --git a/tools/include/nolibc/stdio.h b/tools/include/nolibc/stdio.h index 6cbbb52836a0..0eef91daf289 100644 --- a/tools/include/nolibc/stdio.h +++ b/tools/include/nolibc/stdio.h @@ -21,17 +21,75 @@ #define EOF (-1) #endif -/* just define FILE as a non-empty type */ +/* just define FILE as a non-empty type. The value of the pointer gives + * the FD: FILE=~fd for fd>=0 or NULL for fd<0. This way positive FILE + * are immediately identified as abnormal entries (i.e. possible copies + * of valid pointers to something else). + */ typedef struct FILE { char dummy[1]; } FILE; -/* We define the 3 common stdio files as constant invalid pointers that - * are easily recognized. - */ -static __attribute__((unused)) FILE* const stdin = (FILE*)-3; -static __attribute__((unused)) FILE* const stdout = (FILE*)-2; -static __attribute__((unused)) FILE* const stderr = (FILE*)-1; +static __attribute__((unused)) FILE* const stdin = (FILE*)(intptr_t)~STDIN_FILENO; +static __attribute__((unused)) FILE* const stdout = (FILE*)(intptr_t)~STDOUT_FILENO; +static __attribute__((unused)) FILE* const stderr = (FILE*)(intptr_t)~STDERR_FILENO; + +/* provides a FILE* equivalent of fd. The mode is ignored. */ +static __attribute__((unused)) +FILE *fdopen(int fd, const char *mode __attribute__((unused))) +{ + if (fd < 0) { + SET_ERRNO(EBADF); + return NULL; + } + return (FILE*)(intptr_t)~fd; +} + +/* provides the fd of stream. */ +static __attribute__((unused)) +int fileno(FILE *stream) +{ + intptr_t i = (intptr_t)stream; + + if (i >= 0) { + SET_ERRNO(EBADF); + return -1; + } + return ~i; +} + +/* flush a stream. */ +static __attribute__((unused)) +int fflush(FILE *stream) +{ + intptr_t i = (intptr_t)stream; + + /* NULL is valid here. */ + if (i > 0) { + SET_ERRNO(EBADF); + return -1; + } + + /* Don't do anything, nolibc does not support buffering. */ + return 0; +} + +/* flush a stream. */ +static __attribute__((unused)) +int fclose(FILE *stream) +{ + intptr_t i = (intptr_t)stream; + + if (i >= 0) { + SET_ERRNO(EBADF); + return -1; + } + + if (close(~i)) + return EOF; + + return 0; +} /* getc(), fgetc(), getchar() */ @@ -41,14 +99,8 @@ static __attribute__((unused)) int fgetc(FILE* stream) { unsigned char ch; - int fd; - if (stream < stdin || stream > stderr) - return EOF; - - fd = 3 + (long)stream; - - if (read(fd, &ch, 1) <= 0) + if (read(fileno(stream), &ch, 1) <= 0) return EOF; return ch; } @@ -68,14 +120,8 @@ static __attribute__((unused)) int fputc(int c, FILE* stream) { unsigned char ch = c; - int fd; - if (stream < stdin || stream > stderr) - return EOF; - - fd = 3 + (long)stream; - - if (write(fd, &ch, 1) <= 0) + if (write(fileno(stream), &ch, 1) <= 0) return EOF; return ch; } @@ -96,12 +142,7 @@ static __attribute__((unused)) int _fwrite(const void *buf, size_t size, FILE *stream) { ssize_t ret; - int fd; - - if (stream < stdin || stream > stderr) - return EOF; - - fd = 3 + (long)stream; + int fd = fileno(stream); while (size) { ret = write(fd, buf, size); diff --git a/tools/include/nolibc/stdlib.h b/tools/include/nolibc/stdlib.h index 894c955d027e..902162f80337 100644 --- a/tools/include/nolibc/stdlib.h +++ b/tools/include/nolibc/stdlib.h @@ -102,7 +102,7 @@ char *_getenv(const char *name, char **environ) return NULL; } -static inline __attribute__((unused,always_inline)) +static __inline__ __attribute__((unused,always_inline)) char *getenv(const char *name) { extern char **environ; @@ -231,7 +231,7 @@ int utoh_r(unsigned long in, char *buffer) /* converts unsigned long to an hex string using the static itoa_buffer * and returns the pointer to that string. */ -static inline __attribute__((unused)) +static __inline__ __attribute__((unused)) char *utoh(unsigned long in) { utoh_r(in, itoa_buffer); @@ -293,7 +293,7 @@ int itoa_r(long in, char *buffer) /* for historical compatibility, same as above but returns the pointer to the * buffer. */ -static inline __attribute__((unused)) +static __inline__ __attribute__((unused)) char *ltoa_r(long in, char *buffer) { itoa_r(in, buffer); @@ -303,7 +303,7 @@ char *ltoa_r(long in, char *buffer) /* converts long integer to a string using the static itoa_buffer and * returns the pointer to that string. */ -static inline __attribute__((unused)) +static __inline__ __attribute__((unused)) char *itoa(long in) { itoa_r(in, itoa_buffer); @@ -313,7 +313,7 @@ char *itoa(long in) /* converts long integer to a string using the static itoa_buffer and * returns the pointer to that string. Same as above, for compatibility. */ -static inline __attribute__((unused)) +static __inline__ __attribute__((unused)) char *ltoa(long in) { itoa_r(in, itoa_buffer); @@ -323,7 +323,7 @@ char *ltoa(long in) /* converts unsigned long integer to a string using the static itoa_buffer * and returns the pointer to that string. */ -static inline __attribute__((unused)) +static __inline__ __attribute__((unused)) char *utoa(unsigned long in) { utoa_r(in, itoa_buffer); @@ -367,7 +367,7 @@ int u64toh_r(uint64_t in, char *buffer) /* converts uint64_t to an hex string using the static itoa_buffer and * returns the pointer to that string. */ -static inline __attribute__((unused)) +static __inline__ __attribute__((unused)) char *u64toh(uint64_t in) { u64toh_r(in, itoa_buffer); @@ -429,7 +429,7 @@ int i64toa_r(int64_t in, char *buffer) /* converts int64_t to a string using the static itoa_buffer and returns * the pointer to that string. */ -static inline __attribute__((unused)) +static __inline__ __attribute__((unused)) char *i64toa(int64_t in) { i64toa_r(in, itoa_buffer); @@ -439,7 +439,7 @@ char *i64toa(int64_t in) /* converts uint64_t to a string using the static itoa_buffer and returns * the pointer to that string. */ -static inline __attribute__((unused)) +static __inline__ __attribute__((unused)) char *u64toa(uint64_t in) { u64toa_r(in, itoa_buffer); diff --git a/tools/include/nolibc/string.h b/tools/include/nolibc/string.h index fffdaf6ff467..0c2e06c7c477 100644 --- a/tools/include/nolibc/string.h +++ b/tools/include/nolibc/string.h @@ -90,7 +90,7 @@ void *memset(void *dst, int b, size_t len) while (len--) { /* prevent gcc from recognizing memset() here */ - asm volatile(""); + __asm__ volatile(""); *(p++) = b; } return dst; @@ -139,7 +139,7 @@ size_t strlen(const char *str) size_t len; for (len = 0; str[len]; len++) - asm(""); + __asm__(""); return len; } diff --git a/tools/include/nolibc/sys.h b/tools/include/nolibc/sys.h index 5d624dc63a42..856249a11890 100644 --- a/tools/include/nolibc/sys.h +++ b/tools/include/nolibc/sys.h @@ -12,15 +12,17 @@ /* system includes */ #include -#include // for SIGCHLD +#include /* for SIGCHLD */ #include #include #include #include #include #include -#include // for O_* and AT_* -#include // for statx() +#include /* for O_* and AT_* */ +#include /* for statx() */ +#include /* for LINUX_REBOOT_* */ +#include #include "arch.h" #include "errno.h" @@ -322,7 +324,7 @@ static __attribute__((noreturn,unused)) void sys_exit(int status) { my_syscall1(__NR_exit, status & 255); - while(1); // shut the "noreturn" warnings. + while(1); /* shut the "noreturn" warnings. */ } static __attribute__((noreturn,unused)) @@ -336,6 +338,7 @@ void exit(int status) * pid_t fork(void); */ +#ifndef sys_fork static __attribute__((unused)) pid_t sys_fork(void) { @@ -351,6 +354,7 @@ pid_t sys_fork(void) #error Neither __NR_clone nor __NR_fork defined, cannot implement sys_fork() #endif } +#endif static __attribute__((unused)) pid_t fork(void) @@ -858,7 +862,7 @@ int open(const char *path, int flags, ...) va_list args; va_start(args, flags); - mode = va_arg(args, mode_t); + mode = va_arg(args, int); va_end(args); } @@ -872,6 +876,32 @@ int open(const char *path, int flags, ...) } +/* + * int prctl(int option, unsigned long arg2, unsigned long arg3, + * unsigned long arg4, unsigned long arg5); + */ + +static __attribute__((unused)) +int sys_prctl(int option, unsigned long arg2, unsigned long arg3, + unsigned long arg4, unsigned long arg5) +{ + return my_syscall5(__NR_prctl, option, arg2, arg3, arg4, arg5); +} + +static __attribute__((unused)) +int prctl(int option, unsigned long arg2, unsigned long arg3, + unsigned long arg4, unsigned long arg5) +{ + int ret = sys_prctl(option, arg2, arg3, arg4, arg5); + + if (ret < 0) { + SET_ERRNO(-ret); + ret = -1; + } + return ret; +} + + /* * int pivot_root(const char *new, const char *old); */ @@ -909,7 +939,7 @@ int sys_poll(struct pollfd *fds, int nfds, int timeout) t.tv_sec = timeout / 1000; t.tv_nsec = (timeout % 1000) * 1000000; } - return my_syscall4(__NR_ppoll, fds, nfds, (timeout >= 0) ? &t : NULL, NULL); + return my_syscall5(__NR_ppoll, fds, nfds, (timeout >= 0) ? &t : NULL, NULL, 0); #elif defined(__NR_poll) return my_syscall3(__NR_poll, fds, nfds, timeout); #else @@ -1131,23 +1161,26 @@ int sys_stat(const char *path, struct stat *buf) long ret; ret = sys_statx(AT_FDCWD, path, AT_NO_AUTOMOUNT, STATX_BASIC_STATS, &statx); - buf->st_dev = ((statx.stx_dev_minor & 0xff) - | (statx.stx_dev_major << 8) - | ((statx.stx_dev_minor & ~0xff) << 12)); - buf->st_ino = statx.stx_ino; - buf->st_mode = statx.stx_mode; - buf->st_nlink = statx.stx_nlink; - buf->st_uid = statx.stx_uid; - buf->st_gid = statx.stx_gid; - buf->st_rdev = ((statx.stx_rdev_minor & 0xff) - | (statx.stx_rdev_major << 8) - | ((statx.stx_rdev_minor & ~0xff) << 12)); - buf->st_size = statx.stx_size; - buf->st_blksize = statx.stx_blksize; - buf->st_blocks = statx.stx_blocks; - buf->st_atime = statx.stx_atime.tv_sec; - buf->st_mtime = statx.stx_mtime.tv_sec; - buf->st_ctime = statx.stx_ctime.tv_sec; + buf->st_dev = ((statx.stx_dev_minor & 0xff) + | (statx.stx_dev_major << 8) + | ((statx.stx_dev_minor & ~0xff) << 12)); + buf->st_ino = statx.stx_ino; + buf->st_mode = statx.stx_mode; + buf->st_nlink = statx.stx_nlink; + buf->st_uid = statx.stx_uid; + buf->st_gid = statx.stx_gid; + buf->st_rdev = ((statx.stx_rdev_minor & 0xff) + | (statx.stx_rdev_major << 8) + | ((statx.stx_rdev_minor & ~0xff) << 12)); + buf->st_size = statx.stx_size; + buf->st_blksize = statx.stx_blksize; + buf->st_blocks = statx.stx_blocks; + buf->st_atim.tv_sec = statx.stx_atime.tv_sec; + buf->st_atim.tv_nsec = statx.stx_atime.tv_nsec; + buf->st_mtim.tv_sec = statx.stx_mtime.tv_sec; + buf->st_mtim.tv_nsec = statx.stx_mtime.tv_nsec; + buf->st_ctim.tv_sec = statx.stx_ctime.tv_sec; + buf->st_ctim.tv_nsec = statx.stx_ctime.tv_nsec; return ret; } #else @@ -1165,19 +1198,22 @@ int sys_stat(const char *path, struct stat *buf) #else #error Neither __NR_newfstatat nor __NR_stat defined, cannot implement sys_stat() #endif - buf->st_dev = stat.st_dev; - buf->st_ino = stat.st_ino; - buf->st_mode = stat.st_mode; - buf->st_nlink = stat.st_nlink; - buf->st_uid = stat.st_uid; - buf->st_gid = stat.st_gid; - buf->st_rdev = stat.st_rdev; - buf->st_size = stat.st_size; - buf->st_blksize = stat.st_blksize; - buf->st_blocks = stat.st_blocks; - buf->st_atime = stat.st_atime; - buf->st_mtime = stat.st_mtime; - buf->st_ctime = stat.st_ctime; + buf->st_dev = stat.st_dev; + buf->st_ino = stat.st_ino; + buf->st_mode = stat.st_mode; + buf->st_nlink = stat.st_nlink; + buf->st_uid = stat.st_uid; + buf->st_gid = stat.st_gid; + buf->st_rdev = stat.st_rdev; + buf->st_size = stat.st_size; + buf->st_blksize = stat.st_blksize; + buf->st_blocks = stat.st_blocks; + buf->st_atim.tv_sec = stat.st_atime; + buf->st_atim.tv_nsec = stat.st_atime_nsec; + buf->st_mtim.tv_sec = stat.st_mtime; + buf->st_mtim.tv_nsec = stat.st_mtime_nsec; + buf->st_ctim.tv_sec = stat.st_ctime; + buf->st_ctim.tv_nsec = stat.st_ctime_nsec; return ret; } #endif @@ -1365,6 +1401,29 @@ ssize_t write(int fd, const void *buf, size_t count) return ret; } + +/* + * int memfd_create(const char *name, unsigned int flags); + */ + +static __attribute__((unused)) +int sys_memfd_create(const char *name, unsigned int flags) +{ + return my_syscall2(__NR_memfd_create, name, flags); +} + +static __attribute__((unused)) +int memfd_create(const char *name, unsigned int flags) +{ + ssize_t ret = sys_memfd_create(name, flags); + + if (ret < 0) { + SET_ERRNO(-ret); + ret = -1; + } + return ret; +} + /* make sure to include all global symbols */ #include "nolibc.h" diff --git a/tools/include/nolibc/types.h b/tools/include/nolibc/types.h index aedd7d9e3f64..f96e28bff4ba 100644 --- a/tools/include/nolibc/types.h +++ b/tools/include/nolibc/types.h @@ -86,14 +86,6 @@ #define SEEK_CUR 1 #define SEEK_END 2 -/* cmd for reboot() */ -#define LINUX_REBOOT_MAGIC1 0xfee1dead -#define LINUX_REBOOT_MAGIC2 0x28121969 -#define LINUX_REBOOT_CMD_HALT 0xcdef0123 -#define LINUX_REBOOT_CMD_POWER_OFF 0x4321fedc -#define LINUX_REBOOT_CMD_RESTART 0x01234567 -#define LINUX_REBOOT_CMD_SW_SUSPEND 0xd000fce2 - /* Macros used on waitpid()'s return status */ #define WEXITSTATUS(status) (((status) & 0xff00) >> 8) #define WIFEXITED(status) (((status) & 0x7f) == 0) @@ -206,9 +198,9 @@ struct stat { off_t st_size; /* total size, in bytes */ blksize_t st_blksize; /* blocksize for file system I/O */ blkcnt_t st_blocks; /* number of 512B blocks allocated */ - time_t st_atime; /* time of last access */ - time_t st_mtime; /* time of last modification */ - time_t st_ctime; /* time of last status change */ + union { time_t st_atime; struct timespec st_atim; }; /* time of last access */ + union { time_t st_mtime; struct timespec st_mtim; }; /* time of last modification */ + union { time_t st_ctime; struct timespec st_ctim; }; /* time of last status change */ }; /* WARNING, it only deals with the 4096 first majors and 256 first minors */ diff --git a/tools/include/nolibc/unistd.h b/tools/include/nolibc/unistd.h index ac7d53d986cd..0e832e10a0b2 100644 --- a/tools/include/nolibc/unistd.h +++ b/tools/include/nolibc/unistd.h @@ -56,6 +56,21 @@ int tcsetpgrp(int fd, pid_t pid) return ioctl(fd, TIOCSPGRP, &pid); } +#define _syscall(N, ...) \ +({ \ + long _ret = my_syscall##N(__VA_ARGS__); \ + if (_ret < 0) { \ + SET_ERRNO(-_ret); \ + _ret = -1; \ + } \ + _ret; \ +}) + +#define _syscall_narg(...) __syscall_narg(__VA_ARGS__, 6, 5, 4, 3, 2, 1, 0) +#define __syscall_narg(_0, _1, _2, _3, _4, _5, _6, N, ...) N +#define _syscall_n(N, ...) _syscall(N, __VA_ARGS__) +#define syscall(...) _syscall_n(_syscall_narg(__VA_ARGS__), ##__VA_ARGS__) + /* make sure to include all global symbols */ #include "nolibc.h" diff --git a/tools/objtool/check.c b/tools/objtool/check.c index c19b1103e4ec..9e3cc53a7f4c 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -198,6 +198,7 @@ static bool __dead_end_function(struct objtool_file *file, struct symbol *func, */ static const char * const global_noreturns[] = { "__invalid_creds", + "__kunit_abort", "__module_put_and_kthread_exit", "__reiserfs_panic", "__stack_chk_fail", diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index f01f94106129..7f648802caf6 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -92,7 +92,7 @@ class LinuxSourceTreeOperations: if stderr: # likely only due to build warnings print(stderr.decode()) - def start(self, params: List[str], build_dir: str) -> subprocess.Popen[str]: + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: raise RuntimeError('not implemented!') @@ -113,7 +113,7 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): kconfig.merge_in_entries(base_kunitconfig) return kconfig - def start(self, params: List[str], build_dir: str) -> subprocess.Popen[str]: + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: kernel_path = os.path.join(build_dir, self._kernel_path) qemu_command = ['qemu-system-' + self._qemu_arch, '-nodefaults', @@ -142,7 +142,7 @@ class LinuxSourceTreeOperationsUml(LinuxSourceTreeOperations): kconfig.merge_in_entries(base_kunitconfig) return kconfig - def start(self, params: List[str], build_dir: str) -> subprocess.Popen[str]: + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: """Runs the Linux UML binary. Must be named 'linux'.""" linux_bin = os.path.join(build_dir, 'linux') params.extend(['mem=1G', 'console=tty', 'kunit_shutdown=halt']) diff --git a/tools/testing/kunit/mypy.ini b/tools/testing/kunit/mypy.ini new file mode 100644 index 000000000000..ddd288309efa --- /dev/null +++ b/tools/testing/kunit/mypy.ini @@ -0,0 +1,6 @@ +[mypy] +strict = True + +# E.g. we can't write subprocess.Popen[str] until Python 3.9+. +# But kunit.py tries to support Python 3.7+, so let's disable it. +disable_error_code = type-arg diff --git a/tools/testing/kunit/run_checks.py b/tools/testing/kunit/run_checks.py index 8208c3b3135e..c6d494ea3373 100755 --- a/tools/testing/kunit/run_checks.py +++ b/tools/testing/kunit/run_checks.py @@ -23,7 +23,7 @@ commands: Dict[str, Sequence[str]] = { 'kunit_tool_test.py': ['./kunit_tool_test.py'], 'kunit smoke test': ['./kunit.py', 'run', '--kunitconfig=lib/kunit', '--build_dir=kunit_run_checks'], 'pytype': ['/bin/sh', '-c', 'pytype *.py'], - 'mypy': ['mypy', '--strict', '--exclude', '_test.py$', '--exclude', 'qemu_configs/', '.'], + 'mypy': ['mypy', '--config-file', 'mypy.ini', '--exclude', '_test.py$', '--exclude', 'qemu_configs/', '.'], } # The user might not have mypy or pytype installed, skip them if so. diff --git a/tools/testing/selftests/arm64/abi/hwcap.c b/tools/testing/selftests/arm64/abi/hwcap.c index 93333a90bf3a..d4ad813fed10 100644 --- a/tools/testing/selftests/arm64/abi/hwcap.c +++ b/tools/testing/selftests/arm64/abi/hwcap.c @@ -39,6 +39,20 @@ static void cssc_sigill(void) asm volatile(".inst 0xdac01c00" : : : "x0"); } +static void mops_sigill(void) +{ + char dst[1], src[1]; + register char *dstp asm ("x0") = dst; + register char *srcp asm ("x1") = src; + register long size asm ("x2") = 1; + + /* CPYP [x0]!, [x1]!, x2! */ + asm volatile(".inst 0x1d010440" + : "+r" (dstp), "+r" (srcp), "+r" (size) + : + : "cc", "memory"); +} + static void rng_sigill(void) { asm volatile("mrs x0, S3_3_C2_C4_0" : : : "x0"); @@ -209,6 +223,14 @@ static const struct hwcap_data { .cpuinfo = "cssc", .sigill_fn = cssc_sigill, }, + { + .name = "MOPS", + .at_hwcap = AT_HWCAP2, + .hwcap_bit = HWCAP2_MOPS, + .cpuinfo = "mops", + .sigill_fn = mops_sigill, + .sigill_reliable = true, + }, { .name = "RNG", .at_hwcap = AT_HWCAP2, diff --git a/tools/testing/selftests/arm64/abi/ptrace.c b/tools/testing/selftests/arm64/abi/ptrace.c index be952511af22..abe4d58d731d 100644 --- a/tools/testing/selftests/arm64/abi/ptrace.c +++ b/tools/testing/selftests/arm64/abi/ptrace.c @@ -20,7 +20,7 @@ #include "../../kselftest.h" -#define EXPECTED_TESTS 7 +#define EXPECTED_TESTS 11 #define MAX_TPIDRS 2 @@ -132,6 +132,34 @@ static void test_tpidr(pid_t child) } } +static void test_hw_debug(pid_t child, int type, const char *type_name) +{ + struct user_hwdebug_state state; + struct iovec iov; + int slots, arch, ret; + + iov.iov_len = sizeof(state); + iov.iov_base = &state; + + /* Should be able to read the values */ + ret = ptrace(PTRACE_GETREGSET, child, type, &iov); + ksft_test_result(ret == 0, "read_%s\n", type_name); + + if (ret == 0) { + /* Low 8 bits is the number of slots, next 4 bits the arch */ + slots = state.dbg_info & 0xff; + arch = (state.dbg_info >> 8) & 0xf; + + ksft_print_msg("%s version %d with %d slots\n", type_name, + arch, slots); + + /* Zero is not currently architecturally valid */ + ksft_test_result(arch, "%s_arch_set\n", type_name); + } else { + ksft_test_result_skip("%s_arch_set\n"); + } +} + static int do_child(void) { if (ptrace(PTRACE_TRACEME, -1, NULL, NULL)) @@ -207,6 +235,8 @@ static int do_parent(pid_t child) ksft_print_msg("Parent is %d, child is %d\n", getpid(), child); test_tpidr(child); + test_hw_debug(child, NT_ARM_HW_WATCH, "NT_ARM_HW_WATCH"); + test_hw_debug(child, NT_ARM_HW_BREAK, "NT_ARM_HW_BREAK"); ret = EXIT_SUCCESS; diff --git a/tools/testing/selftests/arm64/signal/.gitignore b/tools/testing/selftests/arm64/signal/.gitignore index 8ab4c86837fd..839e3a252629 100644 --- a/tools/testing/selftests/arm64/signal/.gitignore +++ b/tools/testing/selftests/arm64/signal/.gitignore @@ -4,7 +4,7 @@ fake_sigreturn_* sme_* ssve_* sve_* -tpidr2_siginfo +tpidr2_* za_* zt_* !*.[ch] diff --git a/tools/testing/selftests/arm64/signal/test_signals_utils.c b/tools/testing/selftests/arm64/signal/test_signals_utils.c index 40be8443949d..0dc948db3a4a 100644 --- a/tools/testing/selftests/arm64/signal/test_signals_utils.c +++ b/tools/testing/selftests/arm64/signal/test_signals_utils.c @@ -249,7 +249,8 @@ static void default_handler(int signum, siginfo_t *si, void *uc) fprintf(stderr, "-- Timeout !\n"); } else { fprintf(stderr, - "-- RX UNEXPECTED SIGNAL: %d\n", signum); + "-- RX UNEXPECTED SIGNAL: %d code %d address %p\n", + signum, si->si_code, si->si_addr); } default_result(current, 1); } diff --git a/tools/testing/selftests/arm64/signal/testcases/tpidr2_restore.c b/tools/testing/selftests/arm64/signal/testcases/tpidr2_restore.c new file mode 100644 index 000000000000..f9a86c00c28c --- /dev/null +++ b/tools/testing/selftests/arm64/signal/testcases/tpidr2_restore.c @@ -0,0 +1,86 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2023 ARM Limited + * + * Verify that the TPIDR2 register context in signal frames is restored. + */ + +#include +#include +#include +#include +#include +#include + +#include "test_signals_utils.h" +#include "testcases.h" + +#define SYS_TPIDR2 "S3_3_C13_C0_5" + +static uint64_t get_tpidr2(void) +{ + uint64_t val; + + asm volatile ( + "mrs %0, " SYS_TPIDR2 "\n" + : "=r"(val) + : + : "cc"); + + return val; +} + +static void set_tpidr2(uint64_t val) +{ + asm volatile ( + "msr " SYS_TPIDR2 ", %0\n" + : + : "r"(val) + : "cc"); +} + + +static uint64_t initial_tpidr2; + +static bool save_tpidr2(struct tdescr *td) +{ + initial_tpidr2 = get_tpidr2(); + fprintf(stderr, "Initial TPIDR2: %lx\n", initial_tpidr2); + + return true; +} + +static int modify_tpidr2(struct tdescr *td, siginfo_t *si, ucontext_t *uc) +{ + uint64_t my_tpidr2 = get_tpidr2(); + + my_tpidr2++; + fprintf(stderr, "Setting TPIDR2 to %lx\n", my_tpidr2); + set_tpidr2(my_tpidr2); + + return 0; +} + +static void check_tpidr2(struct tdescr *td) +{ + uint64_t tpidr2 = get_tpidr2(); + + td->pass = tpidr2 == initial_tpidr2; + + if (td->pass) + fprintf(stderr, "TPIDR2 restored\n"); + else + fprintf(stderr, "TPIDR2 was %lx but is now %lx\n", + initial_tpidr2, tpidr2); +} + +struct tdescr tde = { + .name = "TPIDR2 restore", + .descr = "Validate that TPIDR2 is restored from the sigframe", + .feats_required = FEAT_SME, + .timeout = 3, + .sig_trig = SIGUSR1, + .init = save_tpidr2, + .run = modify_tpidr2, + .check_result = check_tpidr2, +}; diff --git a/tools/testing/selftests/clone3/clone3.c b/tools/testing/selftests/clone3/clone3.c index e495f895a2cd..e60cf4da8fb0 100644 --- a/tools/testing/selftests/clone3/clone3.c +++ b/tools/testing/selftests/clone3/clone3.c @@ -129,7 +129,7 @@ int main(int argc, char *argv[]) uid_t uid = getuid(); ksft_print_header(); - ksft_set_plan(18); + ksft_set_plan(19); test_clone3_supported(); /* Just a simple clone3() should return 0.*/ @@ -198,5 +198,8 @@ int main(int argc, char *argv[]) /* Do a clone3() in a new time namespace */ test_clone3(CLONE_NEWTIME, 0, 0, CLONE3_ARGS_NO_TEST); + /* Do a clone3() with exit signal (SIGCHLD) in flags */ + test_clone3(SIGCHLD, 0, -EINVAL, CLONE3_ARGS_NO_TEST); + ksft_finished(); } diff --git a/tools/testing/selftests/cpufreq/config b/tools/testing/selftests/cpufreq/config index 75e900793e8a..ce5068f5a6a2 100644 --- a/tools/testing/selftests/cpufreq/config +++ b/tools/testing/selftests/cpufreq/config @@ -5,11 +5,3 @@ CONFIG_CPU_FREQ_GOV_USERSPACE=y CONFIG_CPU_FREQ_GOV_ONDEMAND=y CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y -CONFIG_DEBUG_RT_MUTEXES=y -CONFIG_DEBUG_PLIST=y -CONFIG_DEBUG_SPINLOCK=y -CONFIG_DEBUG_MUTEXES=y -CONFIG_DEBUG_LOCK_ALLOC=y -CONFIG_PROVE_LOCKING=y -CONFIG_LOCKDEP=y -CONFIG_DEBUG_ATOMIC_SLEEP=y diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest index 2506621e75df..cb5f18c06593 100755 --- a/tools/testing/selftests/ftrace/ftracetest +++ b/tools/testing/selftests/ftrace/ftracetest @@ -301,7 +301,7 @@ ktaptest() { # result comment comment="# $comment" fi - echo $CASENO $result $INSTANCE$CASENAME $comment + echo $result $CASENO $INSTANCE$CASENAME $comment } eval_result() { # sigval diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_opt_types.tc b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_opt_types.tc new file mode 100644 index 000000000000..9f5d99328086 --- /dev/null +++ b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_opt_types.tc @@ -0,0 +1,34 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (C) 2023 Akanksha J N, IBM corporation +# description: Register/unregister optimized probe +# requires: kprobe_events + +case `uname -m` in +x86_64) +;; +arm*) +;; +ppc*) +;; +*) + echo "Please implement other architecture here" + exit_unsupported +esac + +DEFAULT=$(cat /proc/sys/debug/kprobes-optimization) +echo 1 > /proc/sys/debug/kprobes-optimization +for i in `seq 0 255`; do + echo "p:testprobe $FUNCTION_FORK+${i}" > kprobe_events || continue + echo 1 > events/kprobes/enable || continue + (echo "forked") + PROBE=$(grep $FUNCTION_FORK /sys/kernel/debug/kprobes/list) + echo 0 > events/kprobes/enable + echo > kprobe_events + if echo $PROBE | grep -q OPTIMIZED; then + echo "$DEFAULT" > /proc/sys/debug/kprobes-optimization + exit_pass + fi +done +echo "$DEFAULT" > /proc/sys/debug/kprobes-optimization +exit_unresolved diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 294619ade49f..1c952d1401d4 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -8,7 +8,8 @@ export logfile=/dev/stdout export per_test_logging= # Defaults for "settings" file fields: -# "timeout" how many seconds to let each test run before failing. +# "timeout" how many seconds to let each test run before running +# over our soft timeout limit. export kselftest_default_timeout=45 # There isn't a shell-agnostic way to find the path of a sourced file, @@ -90,6 +91,14 @@ run_one() done < "$settings" fi + # Command line timeout overrides the settings file + if [ -n "$kselftest_override_timeout" ]; then + kselftest_timeout="$kselftest_override_timeout" + echo "# overriding timeout to $kselftest_timeout" >> "$logfile" + else + echo "# timeout set to $kselftest_timeout" >> "$logfile" + fi + TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" echo "# $TEST_HDR_MSG" if [ ! -e "$TEST" ]; then diff --git a/tools/testing/selftests/kvm/aarch64/get-reg-list.c b/tools/testing/selftests/kvm/aarch64/get-reg-list.c index d4e1f4af29d6..4f10055af2aa 100644 --- a/tools/testing/selftests/kvm/aarch64/get-reg-list.c +++ b/tools/testing/selftests/kvm/aarch64/get-reg-list.c @@ -48,6 +48,34 @@ struct reg_sublist { __u64 rejects_set_n; }; +struct feature_id_reg { + __u64 reg; + __u64 id_reg; + __u64 feat_shift; + __u64 feat_min; +}; + +static struct feature_id_reg feat_id_regs[] = { + { + ARM64_SYS_REG(3, 0, 2, 0, 3), /* TCR2_EL1 */ + ARM64_SYS_REG(3, 0, 0, 7, 3), /* ID_AA64MMFR3_EL1 */ + 0, + 1 + }, + { + ARM64_SYS_REG(3, 0, 10, 2, 2), /* PIRE0_EL1 */ + ARM64_SYS_REG(3, 0, 0, 7, 3), /* ID_AA64MMFR3_EL1 */ + 4, + 1 + }, + { + ARM64_SYS_REG(3, 0, 10, 2, 3), /* PIR_EL1 */ + ARM64_SYS_REG(3, 0, 0, 7, 3), /* ID_AA64MMFR3_EL1 */ + 4, + 1 + } +}; + struct vcpu_config { char *name; struct reg_sublist sublists[]; @@ -68,7 +96,8 @@ static int vcpu_configs_n; #define for_each_missing_reg(i) \ for ((i) = 0; (i) < blessed_n; ++(i)) \ - if (!find_reg(reg_list->reg, reg_list->n, blessed_reg[i])) + if (!find_reg(reg_list->reg, reg_list->n, blessed_reg[i])) \ + if (check_supported_feat_reg(vcpu, blessed_reg[i])) #define for_each_new_reg(i) \ for_each_reg_filtered(i) \ @@ -132,6 +161,25 @@ static bool find_reg(__u64 regs[], __u64 nr_regs, __u64 reg) return false; } +static bool check_supported_feat_reg(struct kvm_vcpu *vcpu, __u64 reg) +{ + int i, ret; + __u64 data, feat_val; + + for (i = 0; i < ARRAY_SIZE(feat_id_regs); i++) { + if (feat_id_regs[i].reg == reg) { + ret = __vcpu_get_reg(vcpu, feat_id_regs[i].id_reg, &data); + if (ret < 0) + return false; + + feat_val = ((data >> feat_id_regs[i].feat_shift) & 0xf); + return feat_val >= feat_id_regs[i].feat_min; + } + } + + return true; +} + static const char *str_with_index(const char *template, __u64 index) { char *str, *p; @@ -843,12 +891,15 @@ static __u64 base_regs[] = { ARM64_SYS_REG(3, 0, 2, 0, 0), /* TTBR0_EL1 */ ARM64_SYS_REG(3, 0, 2, 0, 1), /* TTBR1_EL1 */ ARM64_SYS_REG(3, 0, 2, 0, 2), /* TCR_EL1 */ + ARM64_SYS_REG(3, 0, 2, 0, 3), /* TCR2_EL1 */ ARM64_SYS_REG(3, 0, 5, 1, 0), /* AFSR0_EL1 */ ARM64_SYS_REG(3, 0, 5, 1, 1), /* AFSR1_EL1 */ ARM64_SYS_REG(3, 0, 5, 2, 0), /* ESR_EL1 */ ARM64_SYS_REG(3, 0, 6, 0, 0), /* FAR_EL1 */ ARM64_SYS_REG(3, 0, 7, 4, 0), /* PAR_EL1 */ ARM64_SYS_REG(3, 0, 10, 2, 0), /* MAIR_EL1 */ + ARM64_SYS_REG(3, 0, 10, 2, 2), /* PIRE0_EL1 */ + ARM64_SYS_REG(3, 0, 10, 2, 3), /* PIR_EL1 */ ARM64_SYS_REG(3, 0, 10, 3, 0), /* AMAIR_EL1 */ ARM64_SYS_REG(3, 0, 12, 0, 0), /* VBAR_EL1 */ ARM64_SYS_REG(3, 0, 12, 1, 1), /* DISR_EL1 */ diff --git a/tools/testing/selftests/media_tests/video_device_test.c b/tools/testing/selftests/media_tests/video_device_test.c index 0f6aef2e2593..2c44e115f2f0 100644 --- a/tools/testing/selftests/media_tests/video_device_test.c +++ b/tools/testing/selftests/media_tests/video_device_test.c @@ -37,45 +37,58 @@ #include #include -int main(int argc, char **argv) +#define PRIORITY_MAX 4 + +int priority_test(int fd) +{ + /* This test will try to update the priority associated with a file descriptor */ + + enum v4l2_priority old_priority, new_priority, priority_to_compare; + int ret; + int result = 0; + + ret = ioctl(fd, VIDIOC_G_PRIORITY, &old_priority); + if (ret < 0) { + printf("Failed to get priority: %s\n", strerror(errno)); + return -1; + } + new_priority = (old_priority + 1) % PRIORITY_MAX; + ret = ioctl(fd, VIDIOC_S_PRIORITY, &new_priority); + if (ret < 0) { + printf("Failed to set priority: %s\n", strerror(errno)); + return -1; + } + ret = ioctl(fd, VIDIOC_G_PRIORITY, &priority_to_compare); + if (ret < 0) { + printf("Failed to get new priority: %s\n", strerror(errno)); + result = -1; + goto cleanup; + } + if (priority_to_compare != new_priority) { + printf("Priority wasn't set - test failed\n"); + result = -1; + } + +cleanup: + ret = ioctl(fd, VIDIOC_S_PRIORITY, &old_priority); + if (ret < 0) { + printf("Failed to restore priority: %s\n", strerror(errno)); + return -1; + } + return result; +} + +int loop_test(int fd) { - int opt; - char video_dev[256]; int count; struct v4l2_tuner vtuner; struct v4l2_capability vcap; int ret; - int fd; - - if (argc < 2) { - printf("Usage: %s [-d ]\n", argv[0]); - exit(-1); - } - - /* Process arguments */ - while ((opt = getopt(argc, argv, "d:")) != -1) { - switch (opt) { - case 'd': - strncpy(video_dev, optarg, sizeof(video_dev) - 1); - video_dev[sizeof(video_dev)-1] = '\0'; - break; - default: - printf("Usage: %s [-d ]\n", argv[0]); - exit(-1); - } - } /* Generate random number of interations */ srand((unsigned int) time(NULL)); count = rand(); - /* Open Video device and keep it open */ - fd = open(video_dev, O_RDWR); - if (fd == -1) { - printf("Video Device open errno %s\n", strerror(errno)); - exit(-1); - } - printf("\nNote:\n" "While test is running, remove the device or unbind\n" "driver and ensure there are no use after free errors\n" @@ -98,4 +111,46 @@ int main(int argc, char **argv) sleep(10); count--; } + return 0; +} + +int main(int argc, char **argv) +{ + int opt; + char video_dev[256]; + int fd; + int test_result; + + if (argc < 2) { + printf("Usage: %s [-d ]\n", argv[0]); + exit(-1); + } + + /* Process arguments */ + while ((opt = getopt(argc, argv, "d:")) != -1) { + switch (opt) { + case 'd': + strncpy(video_dev, optarg, sizeof(video_dev) - 1); + video_dev[sizeof(video_dev)-1] = '\0'; + break; + default: + printf("Usage: %s [-d ]\n", argv[0]); + exit(-1); + } + } + + /* Open Video device and keep it open */ + fd = open(video_dev, O_RDWR); + if (fd == -1) { + printf("Video Device open errno %s\n", strerror(errno)); + exit(-1); + } + + test_result = priority_test(fd); + if (!test_result) + printf("Priority test - PASSED\n"); + else + printf("Priority test - FAILED\n"); + + loop_test(fd); } diff --git a/tools/testing/selftests/nolibc/.gitignore b/tools/testing/selftests/nolibc/.gitignore index 4696df589d68..52f613cdad54 100644 --- a/tools/testing/selftests/nolibc/.gitignore +++ b/tools/testing/selftests/nolibc/.gitignore @@ -1,4 +1,5 @@ /initramfs/ +/libc-test /nolibc-test /run.out /sysroot/ diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile index bbce57420465..1b7b3c82f8ad 100644 --- a/tools/testing/selftests/nolibc/Makefile +++ b/tools/testing/selftests/nolibc/Makefile @@ -64,7 +64,7 @@ QEMU_ARGS_mips = -M malta -append "panic=-1 $(TEST:%=NOLIBC_TEST=%)" QEMU_ARGS_riscv = -M virt -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)" QEMU_ARGS_s390 = -M s390-ccw-virtio -m 1G -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)" QEMU_ARGS_loongarch = -M virt -append "console=ttyS0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)" -QEMU_ARGS = $(QEMU_ARGS_$(ARCH)) +QEMU_ARGS = $(QEMU_ARGS_$(ARCH)) $(QEMU_ARGS_EXTRA) # OUTPUT is only set when run from the main makefile, otherwise # it defaults to this nolibc directory. @@ -76,16 +76,12 @@ else Q=@ endif -CFLAGS_STACKPROTECTOR = -DNOLIBC_STACKPROTECTOR \ - $(call cc-option,-mstack-protector-guard=global) \ - $(call cc-option,-fstack-protector-all) -CFLAGS_STKP_i386 = $(CFLAGS_STACKPROTECTOR) -CFLAGS_STKP_x86_64 = $(CFLAGS_STACKPROTECTOR) -CFLAGS_STKP_x86 = $(CFLAGS_STACKPROTECTOR) CFLAGS_s390 = -m64 -CFLAGS ?= -Os -fno-ident -fno-asynchronous-unwind-tables \ +CFLAGS_mips = -EL +CFLAGS_STACKPROTECTOR ?= $(call cc-option,-mstack-protector-guard=global $(call cc-option,-fstack-protector-all)) +CFLAGS ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 \ $(call cc-option,-fno-stack-protector) \ - $(CFLAGS_STKP_$(ARCH)) $(CFLAGS_$(ARCH)) + $(CFLAGS_$(ARCH)) $(CFLAGS_STACKPROTECTOR) LDFLAGS := -s help: @@ -94,6 +90,7 @@ help: @echo " help this help" @echo " sysroot create the nolibc sysroot here (uses \$$ARCH)" @echo " nolibc-test build the executable (uses \$$CC and \$$CROSS_COMPILE)" + @echo " libc-test build an executable using the compiler's default libc instead" @echo " run-user runs the executable under QEMU (uses \$$ARCH, \$$TEST)" @echo " initramfs prepare the initramfs with nolibc-test" @echo " defconfig create a fresh new default config (uses \$$ARCH)" @@ -128,10 +125,16 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include $(QUIET_CC)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ \ -nostdlib -static -Isysroot/$(ARCH)/include $< -lgcc +libc-test: nolibc-test.c + $(QUIET_CC)$(CC) -o $@ $< + # qemu user-land test run-user: nolibc-test $(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || : - $(Q)grep -w FAIL "$(CURDIR)/run.out" && echo "See all results in $(CURDIR)/run.out" || echo "$$(grep -c ^[0-9].*OK $(CURDIR)/run.out) test(s) passed." + $(Q)awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \ + END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ + if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ + $(CURDIR)/run.out initramfs: nolibc-test $(QUIET_MKDIR)mkdir -p initramfs @@ -147,18 +150,26 @@ kernel: initramfs # run the tests after building the kernel run: kernel $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" - $(Q)grep -w FAIL "$(CURDIR)/run.out" && echo "See all results in $(CURDIR)/run.out" || echo "$$(grep -c ^[0-9].*OK $(CURDIR)/run.out) test(s) passed." + $(Q)awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \ + END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ + if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ + $(CURDIR)/run.out # re-run the tests from an existing kernel rerun: $(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out" - $(Q)grep -w FAIL "$(CURDIR)/run.out" && echo "See all results in $(CURDIR)/run.out" || echo "$$(grep -c ^[0-9].*OK $(CURDIR)/run.out) test(s) passed." + $(Q)awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \ + END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \ + if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \ + $(CURDIR)/run.out clean: $(call QUIET_CLEAN, sysroot) $(Q)rm -rf sysroot $(call QUIET_CLEAN, nolibc-test) $(Q)rm -f nolibc-test + $(call QUIET_CLEAN, libc-test) + $(Q)rm -f libc-test $(call QUIET_CLEAN, initramfs) $(Q)rm -rf initramfs $(call QUIET_CLEAN, run.out) diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c index 21bacc928bf7..486334981e60 100644 --- a/tools/testing/selftests/nolibc/nolibc-test.c +++ b/tools/testing/selftests/nolibc/nolibc-test.c @@ -1,10 +1,7 @@ -// SPDX-License-Identifier: GPL-2.0 +/* SPDX-License-Identifier: GPL-2.0 */ #define _GNU_SOURCE -/* platform-specific include files coming from the compiler */ -#include - /* libc-specific include files * The program may be built in 3 ways: * $(CC) -nostdlib -include /path/to/nolibc.h => NOLIBC already defined @@ -20,7 +17,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -34,7 +33,10 @@ #include #include #include +#include +#include #include +#include #endif #endif @@ -43,8 +45,8 @@ char **environ; /* definition of a series of tests */ struct test { - const char *name; // test name - int (*func)(int min, int max); // handler + const char *name; /* test name */ + int (*func)(int min, int max); /* handler */ }; #ifndef _NOLIBC_STDLIB_H @@ -103,24 +105,32 @@ const char *errorname(int err) CASE_ERR(EDOM); CASE_ERR(ERANGE); CASE_ERR(ENOSYS); + CASE_ERR(EOVERFLOW); default: return itoa(err); } } +static void putcharn(char c, size_t n) +{ + char buf[64]; + + memset(buf, c, n); + buf[n] = '\0'; + fputs(buf, stdout); +} + static int pad_spc(int llen, int cnt, const char *fmt, ...) { va_list args; - int len; int ret; - for (len = 0; len < cnt - llen; len++) - putchar(' '); + putcharn(' ', cnt - llen); va_start(args, fmt); ret = vfprintf(stdout, fmt, args); va_end(args); - return ret < 0 ? ret : ret + len; + return ret < 0 ? ret : ret + cnt - llen; } /* The tests below are intended to be used by the macroes, which evaluate @@ -162,7 +172,7 @@ static int expect_eq(uint64_t expr, int llen, uint64_t val) { int ret = !(expr == val); - llen += printf(" = %lld ", expr); + llen += printf(" = %lld ", (long long)expr); pad_spc(llen, 64, ret ? "[FAIL]\n" : " [OK]\n"); return ret; } @@ -290,18 +300,24 @@ static int expect_sysne(int expr, int llen, int val) } -#define EXPECT_SYSER(cond, expr, expret, experr) \ - do { if (!cond) pad_spc(llen, 64, "[SKIPPED]\n"); else ret += expect_syserr(expr, expret, experr, llen); } while (0) +#define EXPECT_SYSER2(cond, expr, expret, experr1, experr2) \ + do { if (!cond) pad_spc(llen, 64, "[SKIPPED]\n"); else ret += expect_syserr2(expr, expret, experr1, experr2, llen); } while (0) -static int expect_syserr(int expr, int expret, int experr, int llen) +#define EXPECT_SYSER(cond, expr, expret, experr) \ + EXPECT_SYSER2(cond, expr, expret, experr, 0) + +static int expect_syserr2(int expr, int expret, int experr1, int experr2, int llen) { int ret = 0; int _errno = errno; llen += printf(" = %d %s ", expr, errorname(_errno)); - if (expr != expret || _errno != experr) { + if (expr != expret || (_errno != experr1 && _errno != experr2)) { ret = 1; - llen += printf(" != (%d %s) ", expret, errorname(experr)); + if (experr2 == 0) + llen += printf(" != (%d %s) ", expret, errorname(experr1)); + else + llen += printf(" != (%d %s %s) ", expret, errorname(experr1), errorname(experr2)); llen += pad_spc(llen, 64, "[FAIL]\n"); } else { llen += pad_spc(llen, 64, " [OK]\n"); @@ -471,11 +487,60 @@ static int test_getpagesize(void) return !c; } +static int test_fork(void) +{ + int status; + pid_t pid; + + /* flush the printf buffer to avoid child flush it */ + fflush(stdout); + fflush(stderr); + + pid = fork(); + + switch (pid) { + case -1: + return 1; + + case 0: + exit(123); + + default: + pid = waitpid(pid, &status, 0); + + return pid == -1 || !WIFEXITED(status) || WEXITSTATUS(status) != 123; + } +} + +static int test_stat_timestamps(void) +{ + struct stat st; + + if (sizeof(st.st_atim.tv_sec) != sizeof(st.st_atime)) + return 1; + + if (stat("/proc/self/", &st)) + return 1; + + if (st.st_atim.tv_sec != st.st_atime || st.st_atim.tv_nsec > 1000000000) + return 1; + + if (st.st_mtim.tv_sec != st.st_mtime || st.st_mtim.tv_nsec > 1000000000) + return 1; + + if (st.st_ctim.tv_sec != st.st_ctime || st.st_ctim.tv_nsec > 1000000000) + return 1; + + return 0; +} + /* Run syscall tests between IDs and . * Return 0 on success, non-zero on failure. */ int run_syscall(int min, int max) { + struct timeval tv; + struct timezone tz; struct stat stat_buf; int euid0; int proc; @@ -491,7 +556,7 @@ int run_syscall(int min, int max) euid0 = geteuid() == 0; for (test = min; test >= 0 && test <= max; test++) { - int llen = 0; // line length + int llen = 0; /* line length */ /* avoid leaving empty lines below, this will insert holes into * test numbers. @@ -527,14 +592,11 @@ int run_syscall(int min, int max) CASE_TEST(dup3_0); tmp = dup3(0, 100, 0); EXPECT_SYSNE(1, tmp, -1); close(tmp); break; CASE_TEST(dup3_m1); tmp = dup3(-1, 100, 0); EXPECT_SYSER(1, tmp, -1, EBADF); if (tmp != -1) close(tmp); break; CASE_TEST(execve_root); EXPECT_SYSER(1, execve("/", (char*[]){ [0] = "/", [1] = NULL }, NULL), -1, EACCES); break; + CASE_TEST(fork); EXPECT_SYSZR(1, test_fork()); break; CASE_TEST(getdents64_root); EXPECT_SYSNE(1, test_getdents64("/"), -1); break; CASE_TEST(getdents64_null); EXPECT_SYSER(1, test_getdents64("/dev/null"), -1, ENOTDIR); break; - CASE_TEST(gettimeofday_null); EXPECT_SYSZR(1, gettimeofday(NULL, NULL)); break; -#ifdef NOLIBC - CASE_TEST(gettimeofday_bad1); EXPECT_SYSER(1, gettimeofday((void *)1, NULL), -1, EFAULT); break; - CASE_TEST(gettimeofday_bad2); EXPECT_SYSER(1, gettimeofday(NULL, (void *)1), -1, EFAULT); break; - CASE_TEST(gettimeofday_bad2); EXPECT_SYSER(1, gettimeofday(NULL, (void *)1), -1, EFAULT); break; -#endif + CASE_TEST(gettimeofday_tv); EXPECT_SYSZR(1, gettimeofday(&tv, NULL)); break; + CASE_TEST(gettimeofday_tv_tz);EXPECT_SYSZR(1, gettimeofday(&tv, &tz)); break; CASE_TEST(getpagesize); EXPECT_SYSZR(1, test_getpagesize()); break; CASE_TEST(ioctl_tiocinq); EXPECT_SYSZR(1, ioctl(0, TIOCINQ, &tmp)); break; CASE_TEST(ioctl_tiocinq); EXPECT_SYSZR(1, ioctl(0, TIOCINQ, &tmp)); break; @@ -550,6 +612,7 @@ int run_syscall(int min, int max) CASE_TEST(poll_null); EXPECT_SYSZR(1, poll(NULL, 0, 0)); break; CASE_TEST(poll_stdout); EXPECT_SYSNE(1, ({ struct pollfd fds = { 1, POLLOUT, 0}; poll(&fds, 1, 0); }), -1); break; CASE_TEST(poll_fault); EXPECT_SYSER(1, poll((void *)1, 1, 0), -1, EFAULT); break; + CASE_TEST(prctl); EXPECT_SYSER(1, prctl(PR_SET_NAME, (unsigned long)NULL, 0, 0, 0), -1, EFAULT); break; CASE_TEST(read_badf); EXPECT_SYSER(1, read(-1, &tmp, 1), -1, EBADF); break; CASE_TEST(sched_yield); EXPECT_SYSZR(1, sched_yield()); break; CASE_TEST(select_null); EXPECT_SYSZR(1, ({ struct timeval tv = { 0 }; select(0, NULL, NULL, NULL, &tv); })); break; @@ -557,6 +620,7 @@ int run_syscall(int min, int max) CASE_TEST(select_fault); EXPECT_SYSER(1, select(1, (void *)1, NULL, NULL, 0), -1, EFAULT); break; CASE_TEST(stat_blah); EXPECT_SYSER(1, stat("/proc/self/blah", &stat_buf), -1, ENOENT); break; CASE_TEST(stat_fault); EXPECT_SYSER(1, stat(NULL, &stat_buf), -1, EFAULT); break; + CASE_TEST(stat_timestamps); EXPECT_SYSZR(1, test_stat_timestamps()); break; CASE_TEST(symlink_root); EXPECT_SYSER(1, symlink("/", "/"), -1, EEXIST); break; CASE_TEST(unlink_root); EXPECT_SYSER(1, unlink("/"), -1, EISDIR); break; CASE_TEST(unlink_blah); EXPECT_SYSER(1, unlink("/proc/self/blah"), -1, ENOENT); break; @@ -565,6 +629,8 @@ int run_syscall(int min, int max) CASE_TEST(waitpid_child); EXPECT_SYSER(1, waitpid(getpid(), &tmp, WNOHANG), -1, ECHILD); break; CASE_TEST(write_badf); EXPECT_SYSER(1, write(-1, &tmp, 1), -1, EBADF); break; CASE_TEST(write_zero); EXPECT_SYSZR(1, write(1, &tmp, 0)); break; + CASE_TEST(syscall_noargs); EXPECT_SYSEQ(1, syscall(__NR_getpid), getpid()); break; + CASE_TEST(syscall_args); EXPECT_SYSER(1, syscall(__NR_statx, 0, NULL, 0, 0, NULL), -1, EFAULT); break; case __LINE__: return ret; /* must be last */ /* note: do not set any defaults so as to permit holes above */ @@ -581,7 +647,7 @@ int run_stdlib(int min, int max) void *p1, *p2; for (test = min; test >= 0 && test <= max; test++) { - int llen = 0; // line length + int llen = 0; /* line length */ /* avoid leaving empty lines below, this will insert holes into * test numbers. @@ -639,9 +705,9 @@ int run_stdlib(int min, int max) CASE_TEST(limit_int_fast32_min); EXPECT_EQ(1, INT_FAST32_MIN, (int_fast32_t) INTPTR_MIN); break; CASE_TEST(limit_int_fast32_max); EXPECT_EQ(1, INT_FAST32_MAX, (int_fast32_t) INTPTR_MAX); break; CASE_TEST(limit_uint_fast32_max); EXPECT_EQ(1, UINT_FAST32_MAX, (uint_fast32_t) UINTPTR_MAX); break; - CASE_TEST(limit_int_fast64_min); EXPECT_EQ(1, INT_FAST64_MIN, (int_fast64_t) INTPTR_MIN); break; - CASE_TEST(limit_int_fast64_max); EXPECT_EQ(1, INT_FAST64_MAX, (int_fast64_t) INTPTR_MAX); break; - CASE_TEST(limit_uint_fast64_max); EXPECT_EQ(1, UINT_FAST64_MAX, (uint_fast64_t) UINTPTR_MAX); break; + CASE_TEST(limit_int_fast64_min); EXPECT_EQ(1, INT_FAST64_MIN, (int_fast64_t) INT64_MIN); break; + CASE_TEST(limit_int_fast64_max); EXPECT_EQ(1, INT_FAST64_MAX, (int_fast64_t) INT64_MAX); break; + CASE_TEST(limit_uint_fast64_max); EXPECT_EQ(1, UINT_FAST64_MAX, (uint_fast64_t) UINT64_MAX); break; #if __SIZEOF_LONG__ == 8 CASE_TEST(limit_intptr_min); EXPECT_EQ(1, INTPTR_MIN, (intptr_t) 0x8000000000000000LL); break; CASE_TEST(limit_intptr_max); EXPECT_EQ(1, INTPTR_MAX, (intptr_t) 0x7fffffffffffffffLL); break; @@ -667,17 +733,98 @@ int run_stdlib(int min, int max) return ret; } -#if defined(__clang__) -__attribute__((optnone)) -#elif defined(__GNUC__) -__attribute__((optimize("O0"))) -#endif +#define EXPECT_VFPRINTF(c, expected, fmt, ...) \ + ret += expect_vfprintf(llen, c, expected, fmt, ##__VA_ARGS__) + +static int expect_vfprintf(int llen, size_t c, const char *expected, const char *fmt, ...) +{ + int ret, fd, w, r; + char buf[100]; + FILE *memfile; + va_list args; + + fd = memfd_create("vfprintf", 0); + if (fd == -1) { + pad_spc(llen, 64, "[FAIL]\n"); + return 1; + } + + memfile = fdopen(fd, "w+"); + if (!memfile) { + pad_spc(llen, 64, "[FAIL]\n"); + return 1; + } + + va_start(args, fmt); + w = vfprintf(memfile, fmt, args); + va_end(args); + + if (w != c) { + llen += printf(" written(%d) != %d", w, (int) c); + pad_spc(llen, 64, "[FAIL]\n"); + return 1; + } + + fflush(memfile); + lseek(fd, 0, SEEK_SET); + + r = read(fd, buf, sizeof(buf) - 1); + buf[r] = '\0'; + + fclose(memfile); + + if (r != w) { + llen += printf(" written(%d) != read(%d)", w, r); + pad_spc(llen, 64, "[FAIL]\n"); + return 1; + } + + llen += printf(" \"%s\" = \"%s\"", expected, buf); + ret = strncmp(expected, buf, c); + + pad_spc(llen, 64, ret ? "[FAIL]\n" : " [OK]\n"); + return ret; +} + +static int run_vfprintf(int min, int max) +{ + int test; + int tmp; + int ret = 0; + void *p1, *p2; + + for (test = min; test >= 0 && test <= max; test++) { + int llen = 0; /* line length */ + + /* avoid leaving empty lines below, this will insert holes into + * test numbers. + */ + switch (test + __LINE__ + 1) { + CASE_TEST(empty); EXPECT_VFPRINTF(0, "", ""); break; + CASE_TEST(simple); EXPECT_VFPRINTF(3, "foo", "foo"); break; + CASE_TEST(string); EXPECT_VFPRINTF(3, "foo", "%s", "foo"); break; + CASE_TEST(number); EXPECT_VFPRINTF(4, "1234", "%d", 1234); break; + CASE_TEST(negnumber); EXPECT_VFPRINTF(5, "-1234", "%d", -1234); break; + CASE_TEST(unsigned); EXPECT_VFPRINTF(5, "12345", "%u", 12345); break; + CASE_TEST(char); EXPECT_VFPRINTF(1, "c", "%c", 'c'); break; + CASE_TEST(hex); EXPECT_VFPRINTF(1, "f", "%x", 0xf); break; + CASE_TEST(pointer); EXPECT_VFPRINTF(3, "0x1", "%p", (void *) 0x1); break; + case __LINE__: + return ret; /* must be last */ + /* note: do not set any defaults so as to permit holes above */ + } + } + return ret; +} + static int smash_stack(void) { char buf[100]; + volatile char *ptr = buf; + size_t i; - for (size_t i = 0; i < 200; i++) - buf[i] = 'P'; + for (i = 0; i < 200; i++) + ptr[i] = 'P'; return 1; } @@ -689,12 +836,20 @@ static int run_protection(int min, int max) llen += printf("0 -fstackprotector "); -#if !defined(NOLIBC_STACKPROTECTOR) +#if !defined(_NOLIBC_STACKPROTECTOR) llen += printf("not supported"); pad_spc(llen, 64, "[SKIPPED]\n"); return 0; #endif +#if defined(_NOLIBC_STACKPROTECTOR) + if (!__stack_chk_guard) { + llen += printf("__stack_chk_guard not initialized"); + pad_spc(llen, 64, "[FAIL]\n"); + return 1; + } +#endif + pid = -1; pid = fork(); @@ -708,6 +863,7 @@ static int run_protection(int min, int max) close(STDOUT_FILENO); close(STDERR_FILENO); + prctl(PR_SET_DUMPABLE, 0, 0, 0, 0); smash_stack(); return 1; @@ -778,6 +934,7 @@ static const struct test test_names[] = { /* add new tests here */ { .name = "syscall", .func = run_syscall }, { .name = "stdlib", .func = run_stdlib }, + { .name = "vfprintf", .func = run_vfprintf }, { .name = "protection", .func = run_protection }, { 0 } }; @@ -785,7 +942,7 @@ static const struct test test_names[] = { int main(int argc, char **argv, char **envp) { int min = 0; - int max = __INT_MAX__; + int max = INT_MAX; int ret = 0; int err; int idx; @@ -833,7 +990,7 @@ int main(int argc, char **argv, char **envp) * here, which defaults to the full range. */ do { - min = 0; max = __INT_MAX__; + min = 0; max = INT_MAX; value = colon; if (value && *value) { colon = strchr(value, ':'); @@ -899,7 +1056,7 @@ int main(int argc, char **argv, char **envp) #else else if (ioperm(0x501, 1, 1) == 0) #endif - asm volatile ("outb %%al, %%dx" :: "d"(0x501), "a"(0)); + __asm__ volatile ("outb %%al, %%dx" :: "d"(0x501), "a"(0)); /* if it does nothing, fall back to the regular panic */ #endif } diff --git a/tools/testing/selftests/pidfd/pidfd.h b/tools/testing/selftests/pidfd/pidfd.h index 6922d6417e1c..88d6830ee004 100644 --- a/tools/testing/selftests/pidfd/pidfd.h +++ b/tools/testing/selftests/pidfd/pidfd.h @@ -90,7 +90,6 @@ again: } ret = WEXITSTATUS(status); - ksft_print_msg("waitpid WEXITSTATUS=%d\n", ret); return ret; } diff --git a/tools/testing/selftests/pidfd/pidfd_fdinfo_test.c b/tools/testing/selftests/pidfd/pidfd_fdinfo_test.c index 3fd8e903118f..4e86f927880c 100644 --- a/tools/testing/selftests/pidfd/pidfd_fdinfo_test.c +++ b/tools/testing/selftests/pidfd/pidfd_fdinfo_test.c @@ -143,6 +143,7 @@ static inline int child_join(struct child *child, struct error *err) r = -1; } + ksft_print_msg("waitpid WEXITSTATUS=%d\n", r); return r; } diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c index e2dd4ed84984..00a07e7c571c 100644 --- a/tools/testing/selftests/pidfd/pidfd_test.c +++ b/tools/testing/selftests/pidfd/pidfd_test.c @@ -115,7 +115,8 @@ static int test_pidfd_send_signal_exited_fail(void) pidfd = open(buf, O_DIRECTORY | O_CLOEXEC); - (void)wait_for_pid(pid); + ret = wait_for_pid(pid); + ksft_print_msg("waitpid WEXITSTATUS=%d\n", ret); if (pidfd < 0) ksft_exit_fail_msg( diff --git a/tools/testing/selftests/prctl/set-anon-vma-name-test.c b/tools/testing/selftests/prctl/set-anon-vma-name-test.c index 26d853c5a0c1..4275cb256dce 100644 --- a/tools/testing/selftests/prctl/set-anon-vma-name-test.c +++ b/tools/testing/selftests/prctl/set-anon-vma-name-test.c @@ -97,7 +97,7 @@ TEST_F(vma, renaming) { TH_LOG("Try to pass invalid name (with non-printable character \\1) to rename the VMA"); EXPECT_EQ(rename_vma((unsigned long)self->ptr_anon, AREA_SIZE, BAD_NAME), -EINVAL); - TH_LOG("Try to rename non-anonynous VMA"); + TH_LOG("Try to rename non-anonymous VMA"); EXPECT_EQ(rename_vma((unsigned long) self->ptr_not_anon, AREA_SIZE, GOOD_NAME), -EINVAL); } diff --git a/tools/testing/selftests/rcutorture/bin/functions.sh b/tools/testing/selftests/rcutorture/bin/functions.sh index b52d5069563c..48b9147e8c91 100644 --- a/tools/testing/selftests/rcutorture/bin/functions.sh +++ b/tools/testing/selftests/rcutorture/bin/functions.sh @@ -250,7 +250,7 @@ identify_qemu_args () { echo -machine virt,gic-version=host -cpu host ;; qemu-system-ppc64) - echo -enable-kvm -M pseries -nodefaults + echo -M pseries -nodefaults echo -device spapr-vscsi if test -n "$TORTURE_QEMU_INTERACTIVE" -a -n "$TORTURE_QEMU_MAC" then diff --git a/tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot b/tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot index f57720c52c0f..84f6bb98ce99 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot +++ b/tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot @@ -5,4 +5,4 @@ rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 rcutree.kthread_prio=2 threadirqs -tree.use_softirq=0 +rcutree.use_softirq=0 diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE03.boot b/tools/testing/selftests/rcutorture/configs/rcu/TREE03.boot index 64f864f1f361..8e50bfd4b710 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/TREE03.boot +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE03.boot @@ -4,4 +4,4 @@ rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 rcutree.kthread_prio=2 threadirqs -tree.use_softirq=0 +rcutree.use_softirq=0 diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh index 97165a83df63..92743980e553 100755 --- a/tools/testing/selftests/run_kselftest.sh +++ b/tools/testing/selftests/run_kselftest.sh @@ -26,6 +26,7 @@ Usage: $0 [OPTIONS] -l | --list List the available collection:test entries -d | --dry-run Don't actually run any tests -h | --help Show this usage info + -o | --override-timeout Number of seconds after which we timeout EOF exit $1 } @@ -33,6 +34,7 @@ EOF COLLECTIONS="" TESTS="" dryrun="" +kselftest_override_timeout="" while true; do case "$1" in -s | --summary) @@ -51,6 +53,9 @@ while true; do -d | --dry-run) dryrun="echo" shift ;; + -o | --override-timeout) + kselftest_override_timeout="$2" + shift 2 ;; -h | --help) usage 0 ;; "") @@ -85,7 +90,7 @@ if [ -n "$TESTS" ]; then available="$(echo "$valid" | sed -e 's/ /\n/g')" fi -collections=$(echo "$available" | cut -d: -f1 | uniq) +collections=$(echo "$available" | cut -d: -f1 | sort | uniq) for collection in $collections ; do [ -w /dev/kmsg ] && echo "kselftest: Running tests in $collection" >> /dev/kmsg tests=$(echo "$available" | grep "^$collection:" | cut -d: -f2) diff --git a/tools/testing/selftests/vDSO/vdso_test_clock_getres.c b/tools/testing/selftests/vDSO/vdso_test_clock_getres.c index 15dcee16ff72..38d46a8bf7cb 100644 --- a/tools/testing/selftests/vDSO/vdso_test_clock_getres.c +++ b/tools/testing/selftests/vDSO/vdso_test_clock_getres.c @@ -84,12 +84,12 @@ static inline int vdso_test_clock(unsigned int clock_id) int main(int argc, char **argv) { - int ret; + int ret = 0; #if _POSIX_TIMERS > 0 #ifdef CLOCK_REALTIME - ret = vdso_test_clock(CLOCK_REALTIME); + ret += vdso_test_clock(CLOCK_REALTIME); #endif #ifdef CLOCK_BOOTTIME