diff --git a/Documentation/admin-guide/perf/alibaba_pmu.rst b/Documentation/admin-guide/perf/alibaba_pmu.rst index 11de998bb480..7d840023903f 100644 --- a/Documentation/admin-guide/perf/alibaba_pmu.rst +++ b/Documentation/admin-guide/perf/alibaba_pmu.rst @@ -88,6 +88,11 @@ data bandwidth:: -e ali_drw_27080/hif_rmw/ \ -e ali_drw_27080/cycle/ -- sleep 10 +Example usage of counting all memory read/write bandwidth by metric:: + + perf stat -M ddr_read_bandwidth.all -- sleep 10 + perf stat -M ddr_write_bandwidth.all -- sleep 10 + The average DRAM bandwidth can be calculated as follows: - Read Bandwidth = perf_hif_rd * DDRC_WIDTH * DDRC_Freq / DDRC_Cycle diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index 8019103aac10..cf33de56da27 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -450,6 +450,35 @@ this allows system administrators to override the ``IA64_THREAD_UAC_NOPRINT`` ``prctl`` and avoid logs being flooded. +io_uring_disabled +================= + +Prevents all processes from creating new io_uring instances. Enabling this +shrinks the kernel's attack surface. + += ====================================================================== +0 All processes can create io_uring instances as normal. This is the + default setting. +1 io_uring creation is disabled (io_uring_setup() will fail with + -EPERM) for unprivileged processes not in the io_uring_group group. + Existing io_uring instances can still be used. See the + documentation for io_uring_group for more information. +2 io_uring creation is disabled for all processes. io_uring_setup() + always fails with -EPERM. Existing io_uring instances can still be + used. += ====================================================================== + + +io_uring_group +============== + +When io_uring_disabled is set to 1, a process must either be +privileged (CAP_SYS_ADMIN) or be in the io_uring_group group in order +to create an io_uring instance. If io_uring_group is set to -1 (the +default), only processes with the CAP_SYS_ADMIN capability may create +io_uring instances. + + kexec_load_disabled =================== diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst index f32db1f44ae9..e43c2fdafcd7 100644 --- a/Documentation/bpf/btf.rst +++ b/Documentation/bpf/btf.rst @@ -726,8 +726,8 @@ same as the one describe in :ref:`BTF_Type_String`. 4.2 .BTF.ext section -------------------- -The .BTF.ext section encodes func_info and line_info which needs loader -manipulation before loading into the kernel. +The .BTF.ext section encodes func_info, line_info and CO-RE relocations +which needs loader manipulation before loading into the kernel. The specification for .BTF.ext section is defined at ``tools/lib/bpf/btf.h`` and ``tools/lib/bpf/btf.c``. @@ -745,15 +745,20 @@ The current header of .BTF.ext section:: __u32 func_info_len; __u32 line_info_off; __u32 line_info_len; + + /* optional part of .BTF.ext header */ + __u32 core_relo_off; + __u32 core_relo_len; }; It is very similar to .BTF section. Instead of type/string section, it -contains func_info and line_info section. See :ref:`BPF_Prog_Load` for details -about func_info and line_info record format. +contains func_info, line_info and core_relo sub-sections. +See :ref:`BPF_Prog_Load` for details about func_info and line_info +record format. The func_info is organized as below.:: - func_info_rec_size + func_info_rec_size /* __u32 value */ btf_ext_info_sec for section #1 /* func_info for section #1 */ btf_ext_info_sec for section #2 /* func_info for section #2 */ ... @@ -773,7 +778,7 @@ Here, num_info must be greater than 0. The line_info is organized as below.:: - line_info_rec_size + line_info_rec_size /* __u32 value */ btf_ext_info_sec for section #1 /* line_info for section #1 */ btf_ext_info_sec for section #2 /* line_info for section #2 */ ... @@ -787,6 +792,20 @@ kernel API, the ``insn_off`` is the instruction offset in the unit of ``struct bpf_insn``. For ELF API, the ``insn_off`` is the byte offset from the beginning of section (``btf_ext_info_sec->sec_name_off``). +The core_relo is organized as below.:: + + core_relo_rec_size /* __u32 value */ + btf_ext_info_sec for section #1 /* core_relo for section #1 */ + btf_ext_info_sec for section #2 /* core_relo for section #2 */ + +``core_relo_rec_size`` specifies the size of ``bpf_core_relo`` +structure when .BTF.ext is generated. All ``bpf_core_relo`` structures +within a single ``btf_ext_info_sec`` describe relocations applied to +section named by ``btf_ext_info_sec->sec_name_off``. + +See :ref:`Documentation/bpf/llvm_reloc.rst ` +for more information on CO-RE relocations. + 4.2 .BTF_ids section -------------------- diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst index 1ff177b89d66..aeaeb35e6d4a 100644 --- a/Documentation/bpf/index.rst +++ b/Documentation/bpf/index.rst @@ -29,6 +29,7 @@ that goes into great technical depth about the BPF Architecture. bpf_licensing test_debug clang-notes + linux-notes other redirect diff --git a/Documentation/bpf/standardization/linux-notes.rst b/Documentation/bpf/linux-notes.rst similarity index 100% rename from Documentation/bpf/standardization/linux-notes.rst rename to Documentation/bpf/linux-notes.rst diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst index 450e6403fe3d..44188e219d32 100644 --- a/Documentation/bpf/llvm_reloc.rst +++ b/Documentation/bpf/llvm_reloc.rst @@ -240,3 +240,307 @@ The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations:: Offset Info Type Symbol's Value Symbol's Name 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text + +.. _btf-co-re-relocations: + +================= +CO-RE Relocations +================= + +From object file point of view CO-RE mechanism is implemented as a set +of CO-RE specific relocation records. These relocation records are not +related to ELF relocations and are encoded in .BTF.ext section. +See :ref:`Documentation/bpf/btf.rst ` for more +information on .BTF.ext structure. + +CO-RE relocations are applied to BPF instructions to update immediate +or offset fields of the instruction at load time with information +relevant for target kernel. + +Field to patch is selected basing on the instruction class: + +* For BPF_ALU, BPF_ALU64, BPF_LD `immediate` field is patched; +* For BPF_LDX, BPF_STX, BPF_ST `offset` field is patched; +* BPF_JMP, BPF_JMP32 instructions **should not** be patched. + +Relocation kinds +================ + +There are several kinds of CO-RE relocations that could be split in +three groups: + +* Field-based - patch instruction with field related information, e.g. + change offset field of the BPF_LDX instruction to reflect offset + of a specific structure field in the target kernel. + +* Type-based - patch instruction with type related information, e.g. + change immediate field of the BPF_ALU move instruction to 0 or 1 to + reflect if specific type is present in the target kernel. + +* Enum-based - patch instruction with enum related information, e.g. + change immediate field of the BPF_LD_IMM64 instruction to reflect + value of a specific enum literal in the target kernel. + +The complete list of relocation kinds is represented by the following enum: + +.. code-block:: c + + enum bpf_core_relo_kind { + BPF_CORE_FIELD_BYTE_OFFSET = 0, /* field byte offset */ + BPF_CORE_FIELD_BYTE_SIZE = 1, /* field size in bytes */ + BPF_CORE_FIELD_EXISTS = 2, /* field existence in target kernel */ + BPF_CORE_FIELD_SIGNED = 3, /* field signedness (0 - unsigned, 1 - signed) */ + BPF_CORE_FIELD_LSHIFT_U64 = 4, /* bitfield-specific left bitshift */ + BPF_CORE_FIELD_RSHIFT_U64 = 5, /* bitfield-specific right bitshift */ + BPF_CORE_TYPE_ID_LOCAL = 6, /* type ID in local BPF object */ + BPF_CORE_TYPE_ID_TARGET = 7, /* type ID in target kernel */ + BPF_CORE_TYPE_EXISTS = 8, /* type existence in target kernel */ + BPF_CORE_TYPE_SIZE = 9, /* type size in bytes */ + BPF_CORE_ENUMVAL_EXISTS = 10, /* enum value existence in target kernel */ + BPF_CORE_ENUMVAL_VALUE = 11, /* enum value integer value */ + BPF_CORE_TYPE_MATCHES = 12, /* type match in target kernel */ + }; + +Notes: + +* ``BPF_CORE_FIELD_LSHIFT_U64`` and ``BPF_CORE_FIELD_RSHIFT_U64`` are + supposed to be used to read bitfield values using the following + algorithm: + + .. code-block:: c + + // To read bitfield ``f`` from ``struct s`` + is_signed = relo(s->f, BPF_CORE_FIELD_SIGNED) + off = relo(s->f, BPF_CORE_FIELD_BYTE_OFFSET) + sz = relo(s->f, BPF_CORE_FIELD_BYTE_SIZE) + l = relo(s->f, BPF_CORE_FIELD_LSHIFT_U64) + r = relo(s->f, BPF_CORE_FIELD_RSHIFT_U64) + // define ``v`` as signed or unsigned integer of size ``sz`` + v = *({s|u} *)((void *)s + off) + v <<= l + v >>= r + +* The ``BPF_CORE_TYPE_MATCHES`` queries matching relation, defined as + follows: + + * for integers: types match if size and signedness match; + * for arrays & pointers: target types are recursively matched; + * for structs & unions: + + * local members need to exist in target with the same name; + + * for each member we recursively check match unless it is already behind a + pointer, in which case we only check matching names and compatible kind; + + * for enums: + + * local variants have to have a match in target by symbolic name (but not + numeric value); + + * size has to match (but enum may match enum64 and vice versa); + + * for function pointers: + + * number and position of arguments in local type has to match target; + * for each argument and the return value we recursively check match. + +CO-RE Relocation Record +======================= + +Relocation record is encoded as the following structure: + +.. code-block:: c + + struct bpf_core_relo { + __u32 insn_off; + __u32 type_id; + __u32 access_str_off; + enum bpf_core_relo_kind kind; + }; + +* ``insn_off`` - instruction offset (in bytes) within a code section + associated with this relocation; + +* ``type_id`` - BTF type ID of the "root" (containing) entity of a + relocatable type or field; + +* ``access_str_off`` - offset into corresponding .BTF string section. + String interpretation depends on specific relocation kind: + + * for field-based relocations, string encodes an accessed field using + a sequence of field and array indices, separated by colon (:). It's + conceptually very close to LLVM's `getelementptr `_ instruction's + arguments for identifying offset to a field. For example, consider the + following C code: + + .. code-block:: c + + struct sample { + int a; + int b; + struct { int c[10]; }; + } __attribute__((preserve_access_index)); + struct sample *s; + + * Access to ``s[0].a`` would be encoded as ``0:0``: + + * ``0``: first element of ``s`` (as if ``s`` is an array); + * ``0``: index of field ``a`` in ``struct sample``. + + * Access to ``s->a`` would be encoded as ``0:0`` as well. + * Access to ``s->b`` would be encoded as ``0:1``: + + * ``0``: first element of ``s``; + * ``1``: index of field ``b`` in ``struct sample``. + + * Access to ``s[1].c[5]`` would be encoded as ``1:2:0:5``: + + * ``1``: second element of ``s``; + * ``2``: index of anonymous structure field in ``struct sample``; + * ``0``: index of field ``c`` in anonymous structure; + * ``5``: access to array element #5. + + * for type-based relocations, string is expected to be just "0"; + + * for enum value-based relocations, string contains an index of enum + value within its enum type; + +* ``kind`` - one of ``enum bpf_core_relo_kind``. + +.. _GEP: https://llvm.org/docs/LangRef.html#getelementptr-instruction + +.. _btf_co_re_relocation_examples: + +CO-RE Relocation Examples +========================= + +For the following C code: + +.. code-block:: c + + struct foo { + int a; + int b; + unsigned c:15; + } __attribute__((preserve_access_index)); + + enum bar { U, V }; + +With the following BTF definitions: + +.. code-block:: + + ... + [2] STRUCT 'foo' size=8 vlen=2 + 'a' type_id=3 bits_offset=0 + 'b' type_id=3 bits_offset=32 + 'c' type_id=4 bits_offset=64 bitfield_size=15 + [3] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED + [4] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none) + ... + [16] ENUM 'bar' encoding=UNSIGNED size=4 vlen=2 + 'U' val=0 + 'V' val=1 + +Field offset relocations are generated automatically when +``__attribute__((preserve_access_index))`` is used, for example: + +.. code-block:: c + + void alpha(struct foo *s, volatile unsigned long *g) { + *g = s->a; + s->a = 1; + } + + 00 : + 0: r3 = *(s32 *)(r1 + 0x0) + 00: CO-RE [2] struct foo::a (0:0) + 1: *(u64 *)(r2 + 0x0) = r3 + 2: *(u32 *)(r1 + 0x0) = 0x1 + 10: CO-RE [2] struct foo::a (0:0) + 3: exit + + +All relocation kinds could be requested via built-in functions. +E.g. field-based relocations: + +.. code-block:: c + + void bravo(struct foo *s, volatile unsigned long *g) { + *g = __builtin_preserve_field_info(s->b, 0 /* field byte offset */); + *g = __builtin_preserve_field_info(s->b, 1 /* field byte size */); + *g = __builtin_preserve_field_info(s->b, 2 /* field existence */); + *g = __builtin_preserve_field_info(s->b, 3 /* field signedness */); + *g = __builtin_preserve_field_info(s->c, 4 /* bitfield left shift */); + *g = __builtin_preserve_field_info(s->c, 5 /* bitfield right shift */); + } + + 20 : + 4: r1 = 0x4 + 20: CO-RE [2] struct foo::b (0:1) + 5: *(u64 *)(r2 + 0x0) = r1 + 6: r1 = 0x4 + 30: CO-RE [2] struct foo::b (0:1) + 7: *(u64 *)(r2 + 0x0) = r1 + 8: r1 = 0x1 + 40: CO-RE [2] struct foo::b (0:1) + 9: *(u64 *)(r2 + 0x0) = r1 + 10: r1 = 0x1 + 50: CO-RE [2] struct foo::b (0:1) + 11: *(u64 *)(r2 + 0x0) = r1 + 12: r1 = 0x31 + 60: CO-RE [2] struct foo::c (0:2) + 13: *(u64 *)(r2 + 0x0) = r1 + 14: r1 = 0x31 + 70: CO-RE [2] struct foo::c (0:2) + 15: *(u64 *)(r2 + 0x0) = r1 + 16: exit + + +Type-based relocations: + +.. code-block:: c + + void charlie(struct foo *s, volatile unsigned long *g) { + *g = __builtin_preserve_type_info(*s, 0 /* type existence */); + *g = __builtin_preserve_type_info(*s, 1 /* type size */); + *g = __builtin_preserve_type_info(*s, 2 /* type matches */); + *g = __builtin_btf_type_id(*s, 0 /* type id in this object file */); + *g = __builtin_btf_type_id(*s, 1 /* type id in target kernel */); + } + + 88 : + 17: r1 = 0x1 + 88: CO-RE [2] struct foo + 18: *(u64 *)(r2 + 0x0) = r1 + 19: r1 = 0xc + 98: CO-RE [2] struct foo + 20: *(u64 *)(r2 + 0x0) = r1 + 21: r1 = 0x1 + a8: CO-RE [2] struct foo + 22: *(u64 *)(r2 + 0x0) = r1 + 23: r1 = 0x2 ll + b8: CO-RE [2] struct foo + 25: *(u64 *)(r2 + 0x0) = r1 + 26: r1 = 0x2 ll + d0: CO-RE [2] struct foo + 28: *(u64 *)(r2 + 0x0) = r1 + 29: exit + +Enum-based relocations: + +.. code-block:: c + + void delta(struct foo *s, volatile unsigned long *g) { + *g = __builtin_preserve_enum_value(*(enum bar *)U, 0 /* enum literal existence */); + *g = __builtin_preserve_enum_value(*(enum bar *)V, 1 /* enum literal value */); + } + + f0 : + 30: r1 = 0x1 ll + f0: CO-RE [16] enum bar::U = 0 + 32: *(u64 *)(r2 + 0x0) = r1 + 33: r1 = 0x1 ll + 108: CO-RE [16] enum bar::V = 1 + 35: *(u64 *)(r2 + 0x0) = r1 + 36: exit diff --git a/Documentation/bpf/standardization/abi.rst b/Documentation/bpf/standardization/abi.rst new file mode 100644 index 000000000000..0c2e10eeb89a --- /dev/null +++ b/Documentation/bpf/standardization/abi.rst @@ -0,0 +1,25 @@ +.. contents:: +.. sectnum:: + +=================================================== +BPF ABI Recommended Conventions and Guidelines v1.0 +=================================================== + +This is version 1.0 of an informational document containing recommended +conventions and guidelines for producing portable BPF program binaries. + +Registers and calling convention +================================ + +BPF has 10 general purpose registers and a read-only frame pointer register, +all of which are 64-bits wide. + +The BPF calling convention is defined as: + +* R0: return value from function calls, and exit value for BPF programs +* R1 - R5: arguments for function calls +* R6 - R9: callee saved registers that function calls will preserve +* R10: read-only frame pointer to access stack + +R0 - R5 are scratch registers and BPF programs needs to spill/fill them if +necessary across calls. diff --git a/Documentation/bpf/standardization/index.rst b/Documentation/bpf/standardization/index.rst index 09c6ba055fd7..a50c3baf6345 100644 --- a/Documentation/bpf/standardization/index.rst +++ b/Documentation/bpf/standardization/index.rst @@ -12,7 +12,7 @@ for the working group charter, documents, and more. :maxdepth: 1 instruction-set - linux-notes + abi .. Links: .. _IETF BPF Working Group: https://datatracker.ietf.org/wg/bpf/about/ diff --git a/Documentation/bpf/standardization/instruction-set.rst b/Documentation/bpf/standardization/instruction-set.rst index 4f73e9dc8d9e..c5d53a6e8c79 100644 --- a/Documentation/bpf/standardization/instruction-set.rst +++ b/Documentation/bpf/standardization/instruction-set.rst @@ -1,11 +1,11 @@ .. contents:: .. sectnum:: -======================================== -eBPF Instruction Set Specification, v1.0 -======================================== +======================================= +BPF Instruction Set Specification, v1.0 +======================================= -This document specifies version 1.0 of the eBPF instruction set. +This document specifies version 1.0 of the BPF instruction set. Documentation conventions ========================= @@ -97,26 +97,10 @@ Definitions A: 10000110 B: 11111111 10000110 -Registers and calling convention -================================ - -eBPF has 10 general purpose registers and a read-only frame pointer register, -all of which are 64-bits wide. - -The eBPF calling convention is defined as: - -* R0: return value from function calls, and exit value for eBPF programs -* R1 - R5: arguments for function calls -* R6 - R9: callee saved registers that function calls will preserve -* R10: read-only frame pointer to access stack - -R0 - R5 are scratch registers and eBPF programs needs to spill/fill them if -necessary across calls. - Instruction encoding ==================== -eBPF has two instruction encodings: +BPF has two instruction encodings: * the basic instruction encoding, which uses 64 bits to encode an instruction * the wide instruction encoding, which appends a second 64-bit immediate (i.e., @@ -260,7 +244,7 @@ BPF_END 0xd0 0 byte swap operations (see `Byte swap instructions`_ b ========= ===== ======= ========================================================== Underflow and overflow are allowed during arithmetic operations, meaning -the 64-bit or 32-bit value will wrap. If eBPF program execution would +the 64-bit or 32-bit value will wrap. If BPF program execution would result in division by zero, the destination register is instead set to zero. If execution would result in modulo by zero, for ``BPF_ALU64`` the value of the destination register is unchanged whereas for ``BPF_ALU`` the upper @@ -373,7 +357,7 @@ BPF_JNE 0x5 any PC += offset if dst != src BPF_JSGT 0x6 any PC += offset if dst > src signed BPF_JSGE 0x7 any PC += offset if dst >= src signed BPF_CALL 0x8 0x0 call helper function by address see `Helper functions`_ -BPF_CALL 0x8 0x1 call PC += offset see `Program-local functions`_ +BPF_CALL 0x8 0x1 call PC += imm see `Program-local functions`_ BPF_CALL 0x8 0x2 call helper function by BTF ID see `Helper functions`_ BPF_EXIT 0x9 0x0 return BPF_JMP only BPF_JLT 0xa any PC += offset if dst < src unsigned @@ -382,7 +366,7 @@ BPF_JSLT 0xc any PC += offset if dst < src signed BPF_JSLE 0xd any PC += offset if dst <= src signed ======== ===== === =========================================== ========================================= -The eBPF program needs to store the return value into register R0 before doing a +The BPF program needs to store the return value into register R0 before doing a ``BPF_EXIT``. Example: @@ -424,8 +408,8 @@ Program-local functions ~~~~~~~~~~~~~~~~~~~~~~~ Program-local functions are functions exposed by the same BPF program as the caller, and are referenced by offset from the call instruction, similar to -``BPF_JA``. A ``BPF_EXIT`` within the program-local function will return to -the caller. +``BPF_JA``. The offset is encoded in the imm field of the call instruction. +A ``BPF_EXIT`` within the program-local function will return to the caller. Load and store instructions =========================== @@ -502,9 +486,9 @@ Atomic operations Atomic operations are operations that operate on memory and can not be interrupted or corrupted by other access to the same memory region -by other eBPF programs or means outside of this specification. +by other BPF programs or means outside of this specification. -All atomic operations supported by eBPF are encoded as store operations +All atomic operations supported by BPF are encoded as store operations that use the ``BPF_ATOMIC`` mode modifier as follows: * ``BPF_ATOMIC | BPF_W | BPF_STX`` for 32-bit operations @@ -594,7 +578,7 @@ where Maps ~~~~ -Maps are shared memory regions accessible by eBPF programs on some platforms. +Maps are shared memory regions accessible by BPF programs on some platforms. A map can have various semantics as defined in a separate document, and may or may not have a single contiguous memory region, but the 'map_val(map)' is currently only defined for maps that do have a single contiguous memory region. @@ -616,6 +600,6 @@ identified by the given id. Legacy BPF Packet access instructions ------------------------------------- -eBPF previously introduced special instructions for access to packet data that were +BPF previously introduced special instructions for access to packet data that were carried over from classic BPF. However, these instructions are deprecated and should no longer be used. diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst index f4acf9c2e90f..382818a7197a 100644 --- a/Documentation/dev-tools/kasan.rst +++ b/Documentation/dev-tools/kasan.rst @@ -41,8 +41,8 @@ Support Architectures ~~~~~~~~~~~~~ -Generic KASAN is supported on x86_64, arm, arm64, powerpc, riscv, s390, and -xtensa, and the tag-based KASAN modes are supported only on arm64. +Generic KASAN is supported on x86_64, arm, arm64, powerpc, riscv, s390, xtensa, +and loongarch, and the tag-based KASAN modes are supported only on arm64. Compilers ~~~~~~~~~ diff --git a/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml new file mode 100644 index 000000000000..9ab5f0c435d4 --- /dev/null +++ b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright (C) 2023 Renesas Electronics Corp. +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/cache/andestech,ax45mp-cache.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Andestech AX45MP L2 Cache Controller + +maintainers: + - Lad Prabhakar + +description: + A level-2 cache (L2C) is used to improve the system performance by providing + a large amount of cache line entries and reasonable access delays. The L2C + is shared between cores, and a non-inclusive non-exclusive policy is used. + +select: + properties: + compatible: + contains: + enum: + - andestech,ax45mp-cache + + required: + - compatible + +properties: + compatible: + items: + - const: andestech,ax45mp-cache + - const: cache + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + cache-line-size: + const: 64 + + cache-level: + const: 2 + + cache-sets: + const: 1024 + + cache-size: + enum: [131072, 262144, 524288, 1048576, 2097152] + + cache-unified: true + + next-level-cache: true + +additionalProperties: false + +required: + - compatible + - reg + - interrupts + - cache-line-size + - cache-level + - cache-sets + - cache-size + - cache-unified + +examples: + - | + #include + + cache-controller@2010000 { + compatible = "andestech,ax45mp-cache", "cache"; + reg = <0x13400000 0x100000>; + interrupts = <508 IRQ_TYPE_LEVEL_HIGH>; + cache-line-size = <64>; + cache-level = <2>; + cache-sets = <1024>; + cache-size = <262144>; + cache-unified; + }; diff --git a/Documentation/devicetree/bindings/i2c/qcom,i2c-cci.yaml b/Documentation/devicetree/bindings/i2c/qcom,i2c-cci.yaml index ec79b7270437..042d4dc636ee 100644 --- a/Documentation/devicetree/bindings/i2c/qcom,i2c-cci.yaml +++ b/Documentation/devicetree/bindings/i2c/qcom,i2c-cci.yaml @@ -269,6 +269,7 @@ examples: port { ov7251_ep: endpoint { data-lanes = <0 1>; + link-frequencies = /bits/ 64 <240000000 319200000>; remote-endpoint = <&csiphy3_ep>; }; }; diff --git a/Documentation/devicetree/bindings/i3c/i3c.yaml b/Documentation/devicetree/bindings/i3c/i3c.yaml index fdb4212149e7..ab69f4115de4 100644 --- a/Documentation/devicetree/bindings/i3c/i3c.yaml +++ b/Documentation/devicetree/bindings/i3c/i3c.yaml @@ -135,9 +135,10 @@ patternProperties: minimum: 0x1 maximum: 0xff description: | - Dynamic address to be assigned to this device. This property is only - valid if the I3C device has a static address (first cell of the reg - property != 0). + Dynamic address to be assigned to this device. In case static address is + present (first cell of the reg property != 0), this address is assigned + through SETDASA. If static address is not present, this address is assigned + through SETNEWDA after assigning a temporary address via ENTDAA. required: - reg @@ -163,12 +164,18 @@ examples: pagesize = <0x8>; }; - /* I3C device with a static I2C address. */ + /* I3C device with a static I2C address and assigned address. */ thermal_sensor: sensor@68,39200144004 { reg = <0x68 0x392 0x144004>; assigned-address = <0xa>; }; + /* I3C device with only assigned address. */ + pressure_sensor: sensor@0,39200124004 { + reg = <0x0 0x392 0x124000>; + assigned-address = <0xc>; + }; + /* * I3C device without a static I2C address but requiring * resources described in the DT. diff --git a/Documentation/devicetree/bindings/input/azoteq,iqs7222.yaml b/Documentation/devicetree/bindings/input/azoteq,iqs7222.yaml index 9ddba7f2e7aa..5b1769c19b17 100644 --- a/Documentation/devicetree/bindings/input/azoteq,iqs7222.yaml +++ b/Documentation/devicetree/bindings/input/azoteq,iqs7222.yaml @@ -4,14 +4,14 @@ $id: http://devicetree.org/schemas/input/azoteq,iqs7222.yaml# $schema: http://devicetree.org/meta-schemas/core.yaml# -title: Azoteq IQS7222A/B/C Capacitive Touch Controller +title: Azoteq IQS7222A/B/C/D Capacitive Touch Controller maintainers: - Jeff LaBundy description: | - The Azoteq IQS7222A, IQS7222B and IQS7222C are multichannel capacitive touch - controllers that feature additional sensing capabilities. + The Azoteq IQS7222A, IQS7222B, IQS7222C and IQS7222D are multichannel + capacitive touch controllers that feature additional sensing capabilities. Link to datasheets: https://www.azoteq.com/ @@ -21,6 +21,7 @@ properties: - azoteq,iqs7222a - azoteq,iqs7222b - azoteq,iqs7222c + - azoteq,iqs7222d reg: maxItems: 1 @@ -173,6 +174,152 @@ properties: maximum: 3000 description: Specifies the report rate (in ms) during ultra-low-power mode. + touchscreen-size-x: true + touchscreen-size-y: true + touchscreen-inverted-x: true + touchscreen-inverted-y: true + touchscreen-swapped-x-y: true + + trackpad: + type: object + description: Represents all channels associated with the trackpad. + + properties: + azoteq,channel-select: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 12 + items: + minimum: 0 + maximum: 13 + description: + Specifies the order of the channels that participate in the trackpad. + Specify 255 to omit a given channel for the purpose of mapping a non- + rectangular trackpad. + + azoteq,num-rows: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 1 + maximum: 12 + description: Specifies the number of rows that comprise the trackpad. + + azoteq,num-cols: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 1 + maximum: 12 + description: Specifies the number of columns that comprise the trackpad. + + azoteq,top-speed: + $ref: /schemas/types.yaml#/definitions/uint32 + multipleOf: 4 + minimum: 0 + maximum: 1020 + description: + Specifies the speed (in coordinates traveled per conversion) after + which coordinate filtering is no longer applied. + + azoteq,bottom-speed: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: + Specifies the speed (in coordinates traveled per conversion) after + which coordinate filtering is linearly reduced. + + azoteq,use-prox: + type: boolean + description: + Directs the trackpad to respond to the proximity states of the + selected channels instead of their corresponding touch states. + Note the trackpad cannot report granular coordinates during a + state of proximity. + + patternProperties: + "^azoteq,lower-cal-(x|y)$": + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the trackpad's lower starting points. + + "^azoteq,upper-cal-(x|y)$": + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the trackpad's upper starting points. + + "^event-(press|tap|(swipe|flick)-(x|y)-(pos|neg))$": + type: object + $ref: input.yaml# + description: + Represents a press or gesture event reported by the trackpad. Specify + 'linux,code' under the press event to report absolute coordinates. + + properties: + linux,code: true + + azoteq,gesture-angle-tighten: + type: boolean + description: + Limits the tangent of the gesture angle to 0.5 (axial gestures + only). If specified in one direction, the effect is applied in + either direction. + + azoteq,gesture-max-ms: + multipleOf: 16 + minimum: 0 + maximum: 4080 + description: + Specifies the length of time (in ms) within which a tap, swipe + or flick gesture must be completed in order to be acknowledged + by the device. The number specified for any one swipe or flick + gesture applies to all other swipe or flick gestures. + + azoteq,gesture-min-ms: + multipleOf: 16 + minimum: 0 + maximum: 4080 + description: + Specifies the length of time (in ms) for which a tap gesture must + be held in order to be acknowledged by the device. + + azoteq,gesture-dist: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 65535 + description: + Specifies the distance (in coordinates) across which a swipe or + flick gesture must travel in order to be acknowledged by the + device. The number specified for any one swipe or flick gesture + applies to all remaining swipe or flick gestures. + + For tap gestures, this property specifies the distance from the + original point of contact across which the contact is permitted + to travel before the gesture is rejected by the device. + + azoteq,gpio-select: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 3 + items: + minimum: 0 + maximum: 2 + description: | + Specifies one or more GPIO mapped to the event as follows: + 0: GPIO0 + 1: GPIO3 + 2: GPIO4 + + Note that although multiple events can be mapped to a single + GPIO, they must all be of the same type (proximity, touch or + trackpad gesture). + + additionalProperties: false + + required: + - azoteq,channel-select + + additionalProperties: false + patternProperties: "^cycle-[0-9]$": type: object @@ -288,6 +435,10 @@ patternProperties: Activates the reference channel in response to proximity events instead of touch events. + azoteq,counts-filt-enable: + type: boolean + description: Applies counts filtering to the channel. + azoteq,ati-band: $ref: /schemas/types.yaml#/definitions/uint32 enum: [0, 1, 2, 3] @@ -432,12 +583,12 @@ patternProperties: description: | Specifies one or more GPIO mapped to the event as follows: 0: GPIO0 - 1: GPIO3 (IQS7222C only) - 2: GPIO4 (IQS7222C only) + 1: GPIO3 + 2: GPIO4 Note that although multiple events can be mapped to a single GPIO, they must all be of the same type (proximity, touch or - slider gesture). + slider/trackpad gesture). azoteq,thresh: $ref: /schemas/types.yaml#/definitions/uint32 @@ -521,16 +672,16 @@ patternProperties: minimum: 0 maximum: 65535 description: - Specifies the speed of movement after which coordinate filtering is - no longer applied. + Specifies the speed (in coordinates traveled per conversion) after + which coordinate filtering is no longer applied. azoteq,bottom-speed: $ref: /schemas/types.yaml#/definitions/uint32 minimum: 0 maximum: 255 description: - Specifies the speed of movement after which coordinate filtering is - linearly reduced. + Specifies the speed (in coordinates traveled per conversion) after + which coordinate filtering is linearly reduced. azoteq,bottom-beta: $ref: /schemas/types.yaml#/definitions/uint32 @@ -595,10 +746,10 @@ patternProperties: minimum: 0 maximum: 4080 description: - Specifies the distance across which a swipe or flick gesture must - travel in order to be acknowledged by the device. The number spec- - ified for any one swipe or flick gesture applies to all remaining - swipe or flick gestures. + Specifies the distance (in coordinates) across which a swipe or + flick gesture must travel in order to be acknowledged by the + device. The number specified for any one swipe or flick gesture + applies to all remaining swipe or flick gestures. azoteq,gpio-select: $ref: /schemas/types.yaml#/definitions/uint32-array @@ -610,8 +761,8 @@ patternProperties: description: | Specifies one or more GPIO mapped to the event as follows: 0: GPIO0 - 1: GPIO3 (IQS7222C only) - 2: GPIO4 (IQS7222C only) + 1: GPIO3 + 2: GPIO4 Note that although multiple events can be mapped to a single GPIO, they must all be of the same type (proximity, touch or @@ -629,8 +780,8 @@ patternProperties: description: | Represents a GPIO mapped to one or more events as follows: gpio-0: GPIO0 - gpio-1: GPIO3 (IQS7222C only) - gpio-2: GPIO4 (IQS7222C only) + gpio-1: GPIO3 + gpio-2: GPIO4 allOf: - $ref: ../pinctrl/pincfg-node.yaml# @@ -641,11 +792,53 @@ patternProperties: additionalProperties: false allOf: + - $ref: touchscreen/touchscreen.yaml# + - if: properties: compatible: contains: - const: azoteq,iqs7222b + enum: + - azoteq,iqs7222a + - azoteq,iqs7222b + - azoteq,iqs7222c + + then: + properties: + touchscreen-size-x: false + touchscreen-size-y: false + touchscreen-inverted-x: false + touchscreen-inverted-y: false + touchscreen-swapped-x-y: false + + trackpad: false + + patternProperties: + "^channel-([0-9]|1[0-9])$": + properties: + azoteq,counts-filt-enable: false + + - if: + properties: + compatible: + contains: + enum: + - azoteq,iqs7222b + - azoteq,iqs7222c + + then: + patternProperties: + "^channel-([0-9]|1[0-9])$": + properties: + azoteq,ulp-allow: false + + - if: + properties: + compatible: + contains: + enum: + - azoteq,iqs7222b + - azoteq,iqs7222d then: patternProperties: @@ -657,13 +850,22 @@ allOf: properties: azoteq,ref-select: false + "^slider-[0-1]$": false + + - if: + properties: + compatible: + contains: + const: azoteq,iqs7222b + + then: + patternProperties: + "^channel-([0-9]|1[0-9])$": patternProperties: "^event-(prox|touch)$": properties: azoteq,gpio-select: false - "^slider-[0-1]$": false - "^gpio-[0-2]$": false - if: @@ -704,10 +906,6 @@ allOf: else: patternProperties: - "^channel-([0-9]|1[0-9])$": - properties: - azoteq,ulp-allow: false - "^slider-[0-1]$": patternProperties: "^event-(press|tap|(swipe|flick)-(pos|neg))$": diff --git a/Documentation/devicetree/bindings/input/touchscreen/azoteq,iqs7211.yaml b/Documentation/devicetree/bindings/input/touchscreen/azoteq,iqs7211.yaml new file mode 100644 index 000000000000..8cf371b99f19 --- /dev/null +++ b/Documentation/devicetree/bindings/input/touchscreen/azoteq,iqs7211.yaml @@ -0,0 +1,769 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/input/touchscreen/azoteq,iqs7211.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Azoteq IQS7210A/7211A/E Trackpad/Touchscreen Controller + +maintainers: + - Jeff LaBundy + +description: | + The Azoteq IQS7210A, IQS7211A and IQS7211E trackpad and touchscreen control- + lers employ projected-capacitance sensing and can track two contacts. + + Link to datasheets: https://www.azoteq.com/ + +properties: + compatible: + enum: + - azoteq,iqs7210a + - azoteq,iqs7211a + - azoteq,iqs7211e + + reg: + maxItems: 1 + + irq-gpios: + maxItems: 1 + description: + Specifies the GPIO connected to the device's active-low RDY output. The + pin doubles as the IQS7211E's active-low MCLR input, in which case this + GPIO must be configured as open-drain. + + reset-gpios: + maxItems: 1 + description: + Specifies the GPIO connected to the device's active-low MCLR input. The + device is temporarily held in hardware reset prior to initialization if + this property is present. + + azoteq,forced-comms: + type: boolean + description: + Enables forced communication; to be used with host adapters that cannot + tolerate clock stretching. + + azoteq,forced-comms-default: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1] + description: + Indicates if the device's OTP memory enables (1) or disables (0) forced + communication by default. Specifying this property can expedite startup + time if the default value is known. + + If this property is not specified, communication is not initiated until + the device asserts its RDY pin shortly after exiting hardware reset. At + that point, forced communication is either enabled or disabled based on + the presence or absence of the 'azoteq,forced-comms' property. + + azoteq,rate-active-ms: + minimum: 0 + maximum: 65535 + description: Specifies the report rate (in ms) during active mode. + + azoteq,rate-touch-ms: + minimum: 0 + maximum: 65535 + description: Specifies the report rate (in ms) during idle-touch mode. + + azoteq,rate-idle-ms: + minimum: 0 + maximum: 65535 + description: Specifies the report rate (in ms) during idle mode. + + azoteq,rate-lp1-ms: + minimum: 0 + maximum: 65535 + description: Specifies the report rate (in ms) during low-power mode 1. + + azoteq,rate-lp2-ms: + minimum: 0 + maximum: 65535 + description: Specifies the report rate (in ms) during low-power mode 2. + + azoteq,timeout-active-ms: + multipleOf: 1000 + minimum: 0 + maximum: 65535000 + description: + Specifies the length of time (in ms) to wait for an event before moving + from active mode to idle or idle-touch modes. + + azoteq,timeout-touch-ms: + multipleOf: 1000 + minimum: 0 + maximum: 65535000 + description: + Specifies the length of time (in ms) to wait for an event before moving + from idle-touch mode to idle mode. + + azoteq,timeout-idle-ms: + multipleOf: 1000 + minimum: 0 + maximum: 65535000 + description: + Specifies the length of time (in ms) to wait for an event before moving + from idle mode to low-power mode 1. + + azoteq,timeout-lp1-ms: + multipleOf: 1000 + minimum: 0 + maximum: 65535000 + description: + Specifies the length of time (in ms) to wait for an event before moving + from low-power mode 1 to low-power mode 2. + + azoteq,timeout-lp2-ms: + multipleOf: 1000 + minimum: 0 + maximum: 60000 + description: + Specifies the rate (in ms) at which the trackpad reference values + are updated during low-power modes 1 and 2. + + azoteq,timeout-ati-ms: + multipleOf: 1000 + minimum: 0 + maximum: 60000 + description: + Specifies the delay (in ms) before the automatic tuning implementation + (ATI) is retried in the event it fails to complete. + + azoteq,timeout-comms-ms: + minimum: 0 + maximum: 65535 + description: + Specifies the delay (in ms) before a communication window is closed. + + azoteq,timeout-press-ms: + multipleOf: 1000 + minimum: 0 + maximum: 60000 + description: + Specifies the length of time (in ms) to wait before automatically + releasing a press event. Specify zero to allow the press state to + persist indefinitely. + + azoteq,fosc-freq: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1] + description: | + Specifies the device's core clock frequency as follows: + 0: 14 MHz + 1: 18 MHz + + azoteq,fosc-trim: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 15 + description: Specifies the device's core clock frequency trim. + + azoteq,num-contacts: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 2 + default: 0 + description: Specifies the number of contacts reported by the device. + + azoteq,contact-split: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the contact (finger) split factor. + + azoteq,trim-x: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the horizontal trim width. + + azoteq,trim-y: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the vertical trim height. + + trackpad: + type: object + description: Represents all channels associated with the trackpad. + + properties: + azoteq,rx-enable: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 8 + items: + minimum: 0 + maximum: 7 + description: + Specifies the order of the CRx pin(s) associated with the trackpad. + + azoteq,tx-enable: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 12 + items: + minimum: 0 + maximum: 11 + description: + Specifies the order of the CTx pin(s) associated with the trackpad. + + azoteq,channel-select: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 36 + items: + minimum: 0 + maximum: 255 + description: | + Specifies the channels mapped to each cycle in the following order: + Cycle 0, slot 0 + Cycle 0, slot 1 + Cycle 1, slot 0 + Cycle 1, slot 1 + ...and so on. Specify 255 to disable a given slot. + + azoteq,ati-frac-div-fine: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the trackpad's ATI fine fractional divider. + + azoteq,ati-frac-mult-coarse: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 15 + description: Specifies the trackpad's ATI coarse fractional multiplier. + + azoteq,ati-frac-div-coarse: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the trackpad's ATI coarse fractional divider. + + azoteq,ati-comp-div: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the trackpad's ATI compensation divider. + + azoteq,ati-target: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 65535 + description: Specifies the trackpad's ATI target. + + azoteq,touch-enter: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the trackpad's touch entrance factor. + + azoteq,touch-exit: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the trackpad's touch exit factor. + + azoteq,thresh: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the trackpad's stationary touch threshold. + + azoteq,conv-period: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the trackpad's conversion period. + + azoteq,conv-frac: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the trackpad's conversion frequency fraction. + + patternProperties: + "^event-(tap(-double|-triple)?|hold|palm|swipe-(x|y)-(pos|neg)(-hold)?)$": + type: object + $ref: ../input.yaml# + description: + Represents a gesture event reported by the trackpad. In the case of + axial gestures, the duration or distance specified in one direction + applies to both directions along the same axis. + + properties: + linux,code: true + + azoteq,gesture-max-ms: + minimum: 0 + maximum: 65535 + description: Specifies the maximum duration of tap/swipe gestures. + + azoteq,gesture-mid-ms: + minimum: 0 + maximum: 65535 + description: + Specifies the maximum duration between subsequent tap gestures + (IQS7211E only). + + azoteq,gesture-min-ms: + minimum: 0 + maximum: 65535 + description: Specifies the minimum duration of hold gestures. + + azoteq,gesture-dist: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 65535 + description: + Specifies the minimum (swipe) or maximum (tap and hold) distance + a finger may travel to be considered a gesture. + + azoteq,gesture-dist-rep: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 65535 + description: + Specifies the minimum distance a finger must travel to elicit a + repeated swipe gesture (IQS7211E only). + + azoteq,gesture-angle: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 75 + description: + Specifies the maximum angle (in degrees) a finger may travel to + be considered a swipe gesture. + + azoteq,thresh: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 42 + description: Specifies the palm gesture threshold (IQS7211E only). + + additionalProperties: false + + dependencies: + azoteq,rx-enable: ["azoteq,tx-enable"] + azoteq,tx-enable: ["azoteq,rx-enable"] + azoteq,channel-select: ["azoteq,rx-enable"] + + additionalProperties: false + + alp: + type: object + $ref: ../input.yaml# + description: Represents the alternate low-power channel (ALP). + + properties: + azoteq,rx-enable: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 8 + items: + minimum: 0 + maximum: 7 + description: + Specifies the CRx pin(s) associated with the ALP in no particular + order. + + azoteq,tx-enable: + $ref: /schemas/types.yaml#/definitions/uint32-array + minItems: 1 + maxItems: 12 + items: + minimum: 0 + maximum: 11 + description: + Specifies the CTx pin(s) associated with the ALP in no particular + order. + + azoteq,ati-frac-div-fine: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the ALP's ATI fine fractional divider. + + azoteq,ati-frac-mult-coarse: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 15 + description: Specifies the ALP's ATI coarse fractional multiplier. + + azoteq,ati-frac-div-coarse: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the ALP's ATI coarse fractional divider. + + azoteq,ati-comp-div: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the ALP's ATI compensation divider. + + azoteq,ati-target: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 65535 + description: Specifies the ALP's ATI target. + + azoteq,ati-base: + $ref: /schemas/types.yaml#/definitions/uint32 + multipleOf: 8 + minimum: 0 + maximum: 255 + description: Specifies the ALP's ATI base. + + azoteq,ati-mode: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1] + description: | + Specifies the ALP's ATI mode as follows: + 0: Partial + 1: Full + + azoteq,sense-mode: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1] + description: | + Specifies the ALP's sensing mode as follows: + 0: Self capacitive + 1: Mutual capacitive + + azoteq,debounce-enter: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the ALP's debounce entrance factor. + + azoteq,debounce-exit: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the ALP's debounce exit factor. + + azoteq,thresh: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 65535 + description: Specifies the ALP's proximity or touch threshold. + + azoteq,conv-period: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the ALP's conversion period. + + azoteq,conv-frac: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the ALP's conversion frequency fraction. + + linux,code: true + + additionalProperties: false + + button: + type: object + description: Represents the inductive or capacitive button. + + properties: + azoteq,ati-frac-div-fine: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the button's ATI fine fractional divider. + + azoteq,ati-frac-mult-coarse: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 15 + description: Specifies the button's ATI coarse fractional multiplier. + + azoteq,ati-frac-div-coarse: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the button's ATI coarse fractional divider. + + azoteq,ati-comp-div: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 31 + description: Specifies the button's ATI compensation divider. + + azoteq,ati-target: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 65535 + description: Specifies the button's ATI target. + + azoteq,ati-base: + $ref: /schemas/types.yaml#/definitions/uint32 + multipleOf: 8 + minimum: 0 + maximum: 255 + description: Specifies the button's ATI base. + + azoteq,ati-mode: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1] + description: | + Specifies the button's ATI mode as follows: + 0: Partial + 1: Full + + azoteq,sense-mode: + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [0, 1, 2] + description: | + Specifies the button's sensing mode as follows: + 0: Self capacitive + 1: Mutual capacitive + 2: Inductive + + azoteq,touch-enter: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the button's touch entrance factor. + + azoteq,touch-exit: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the button's touch exit factor. + + azoteq,debounce-enter: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the button's debounce entrance factor. + + azoteq,debounce-exit: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the button's debounce exit factor. + + azoteq,thresh: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 65535 + description: Specifies the button's proximity threshold. + + azoteq,conv-period: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the button's conversion period. + + azoteq,conv-frac: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + description: Specifies the button's conversion frequency fraction. + + patternProperties: + "^event-(prox|touch)$": + type: object + $ref: ../input.yaml# + description: + Represents a proximity or touch event reported by the button. + + properties: + linux,code: true + + additionalProperties: false + + additionalProperties: false + + wakeup-source: true + + touchscreen-size-x: true + touchscreen-size-y: true + touchscreen-inverted-x: true + touchscreen-inverted-y: true + touchscreen-swapped-x-y: true + +dependencies: + touchscreen-size-x: ["azoteq,num-contacts"] + touchscreen-size-y: ["azoteq,num-contacts"] + touchscreen-inverted-x: ["azoteq,num-contacts"] + touchscreen-inverted-y: ["azoteq,num-contacts"] + touchscreen-swapped-x-y: ["azoteq,num-contacts"] + +required: + - compatible + - reg + - irq-gpios + +additionalProperties: false + +allOf: + - $ref: touchscreen.yaml# + + - if: + properties: + compatible: + contains: + const: azoteq,iqs7210a + + then: + properties: + alp: + properties: + azoteq,rx-enable: + maxItems: 4 + items: + minimum: 4 + + else: + properties: + azoteq,timeout-press-ms: false + + alp: + properties: + azoteq,ati-mode: false + + button: false + + - if: + properties: + compatible: + contains: + const: azoteq,iqs7211e + + then: + properties: + reset-gpios: false + + trackpad: + properties: + azoteq,tx-enable: + maxItems: 13 + items: + maximum: 12 + + alp: + properties: + azoteq,tx-enable: + maxItems: 13 + items: + maximum: 12 + +examples: + - | + #include + #include + + i2c { + #address-cells = <1>; + #size-cells = <0>; + + touch@56 { + compatible = "azoteq,iqs7210a"; + reg = <0x56>; + irq-gpios = <&gpio 4 GPIO_ACTIVE_LOW>; + reset-gpios = <&gpio 17 (GPIO_ACTIVE_LOW | + GPIO_PUSH_PULL)>; + azoteq,num-contacts = <2>; + + trackpad { + azoteq,rx-enable = <6>, <5>, <4>, <3>, <2>; + azoteq,tx-enable = <1>, <7>, <8>, <9>, <10>; + }; + + button { + azoteq,sense-mode = <2>; + azoteq,touch-enter = <40>; + azoteq,touch-exit = <36>; + + event-touch { + linux,code = ; + }; + }; + + alp { + azoteq,sense-mode = <1>; + linux,code = ; + }; + }; + }; + + - | + #include + #include + + i2c { + #address-cells = <1>; + #size-cells = <0>; + + touch@56 { + compatible = "azoteq,iqs7211e"; + reg = <0x56>; + irq-gpios = <&gpio 4 (GPIO_ACTIVE_LOW | + GPIO_OPEN_DRAIN)>; + + trackpad { + event-tap { + linux,code = ; + }; + + event-tap-double { + linux,code = ; + }; + + event-tap-triple { + linux,code = ; + }; + + event-hold { + linux,code = ; + }; + + event-palm { + linux,code = ; + }; + + event-swipe-x-pos { + linux,code = ; + }; + + event-swipe-x-pos-hold { + linux,code = ; + }; + + event-swipe-x-neg { + linux,code = ; + }; + + event-swipe-x-neg-hold { + linux,code = ; + }; + + event-swipe-y-pos { + linux,code = ; + }; + + event-swipe-y-pos-hold { + linux,code = ; + }; + + event-swipe-y-neg { + linux,code = ; + }; + + event-swipe-y-neg-hold { + linux,code = ; + }; + }; + }; + }; + +... diff --git a/Documentation/devicetree/bindings/input/touchscreen/edt-ft5x06.yaml b/Documentation/devicetree/bindings/input/touchscreen/edt-ft5x06.yaml index ef4c841387bd..f2808cb4d99d 100644 --- a/Documentation/devicetree/bindings/input/touchscreen/edt-ft5x06.yaml +++ b/Documentation/devicetree/bindings/input/touchscreen/edt-ft5x06.yaml @@ -93,6 +93,12 @@ properties: minimum: 1 maximum: 255 + threshold: + description: Allows setting the "click"-threshold in the range from 0 to 255. + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 255 + touchscreen-size-x: true touchscreen-size-y: true touchscreen-fuzz-x: true diff --git a/Documentation/devicetree/bindings/input/touchscreen/eeti,exc3000.yaml b/Documentation/devicetree/bindings/input/touchscreen/eeti,exc3000.yaml index 007adbc89c14..9dc25d30a0a8 100644 --- a/Documentation/devicetree/bindings/input/touchscreen/eeti,exc3000.yaml +++ b/Documentation/devicetree/bindings/input/touchscreen/eeti,exc3000.yaml @@ -24,6 +24,8 @@ properties: maxItems: 1 reset-gpios: maxItems: 1 + vdd-supply: + description: Power supply regulator for the chip touchscreen-size-x: true touchscreen-size-y: true touchscreen-inverted-x: true diff --git a/Documentation/devicetree/bindings/input/touchscreen/melfas,mms114.yaml b/Documentation/devicetree/bindings/input/touchscreen/melfas,mms114.yaml index fdd02898e249..07f9dd6b1c9c 100644 --- a/Documentation/devicetree/bindings/input/touchscreen/melfas,mms114.yaml +++ b/Documentation/devicetree/bindings/input/touchscreen/melfas,mms114.yaml @@ -52,6 +52,11 @@ properties: touchscreen-swapped-x-y: true touchscreen-max-pressure: true + linux,keycodes: + description: Keycodes for the touch keys + minItems: 1 + maxItems: 15 + additionalProperties: false required: diff --git a/Documentation/devicetree/bindings/interrupt-controller/st,sti-irq-syscfg.txt b/Documentation/devicetree/bindings/interrupt-controller/st,sti-irq-syscfg.txt deleted file mode 100644 index 977d7ed3670e..000000000000 --- a/Documentation/devicetree/bindings/interrupt-controller/st,sti-irq-syscfg.txt +++ /dev/null @@ -1,30 +0,0 @@ -STMicroelectronics STi System Configuration Controlled IRQs ------------------------------------------------------------ - -On STi based systems; External, CTI (Core Sight), PMU (Performance Management), -and PL310 L2 Cache IRQs are controlled using System Configuration registers. -This driver is used to unmask them prior to use. - -Required properties: -- compatible : Should be "st,stih407-irq-syscfg" -- st,syscfg : Phandle to Cortex-A9 IRQ system config registers -- st,irq-device : Array of IRQs to enable - should be 2 in length -- st,fiq-device : Array of FIQs to enable - should be 2 in length - -Optional properties: -- st,invert-ext : External IRQs can be inverted at will. This property inverts - these IRQs using bitwise logic. A number of defines have been - provided for convenience: - ST_IRQ_SYSCFG_EXT_1_INV - ST_IRQ_SYSCFG_EXT_2_INV - ST_IRQ_SYSCFG_EXT_3_INV -Example: - -irq-syscfg { - compatible = "st,stih407-irq-syscfg"; - st,syscfg = <&syscfg_cpu>; - st,irq-device = , - ; - st,fiq-device = , - ; -}; diff --git a/Documentation/devicetree/bindings/interrupt-controller/st,stih407-irq-syscfg.yaml b/Documentation/devicetree/bindings/interrupt-controller/st,stih407-irq-syscfg.yaml new file mode 100644 index 000000000000..2b153d7c5421 --- /dev/null +++ b/Documentation/devicetree/bindings/interrupt-controller/st,stih407-irq-syscfg.yaml @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/interrupt-controller/st,stih407-irq-syscfg.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: STMicroelectronics STi System Configuration Controlled IRQs + +maintainers: + - Patrice Chotard + +description: + On STi based systems; External, CTI (Core Sight), PMU (Performance + Management), and PL310 L2 Cache IRQs are controlled using System + Configuration registers. This device is used to unmask them prior to use. + +properties: + compatible: + const: st,stih407-irq-syscfg + + st,syscfg: + description: Phandle to Cortex-A9 IRQ system config registers + $ref: /schemas/types.yaml#/definitions/phandle + + st,irq-device: + description: Array of IRQs to enable. + $ref: /schemas/types.yaml#/definitions/uint32-array + items: + - description: Enable the IRQ of the channel one. + - description: Enable the IRQ of the channel two. + + st,fiq-device: + description: Array of FIQs to enable. + $ref: /schemas/types.yaml#/definitions/uint32-array + items: + - description: Enable the IRQ of the channel one. + - description: Enable the IRQ of the channel two. + + st,invert-ext: + description: External IRQs can be inverted at will. This property inverts + these three IRQs using bitwise logic, each one being encoded respectively + on the first, second and fourth bit. + $ref: /schemas/types.yaml#/definitions/uint32 + enum: [ 1, 2, 3, 4, 5, 6 ] + +required: + - compatible + - st,syscfg + - st,irq-device + - st,fiq-device + +additionalProperties: false + +examples: + - | + #include + irq-syscfg { + compatible = "st,stih407-irq-syscfg"; + st,syscfg = <&syscfg_cpu>; + st,irq-device = , + ; + st,fiq-device = , + ; + }; +... diff --git a/Documentation/devicetree/bindings/media/i2c/ov5695.txt b/Documentation/devicetree/bindings/media/i2c/ov5695.txt deleted file mode 100644 index 640a63717d96..000000000000 --- a/Documentation/devicetree/bindings/media/i2c/ov5695.txt +++ /dev/null @@ -1,41 +0,0 @@ -* Omnivision OV5695 MIPI CSI-2 sensor - -Required Properties: -- compatible: shall be "ovti,ov5695" -- clocks: reference to the xvclk input clock -- clock-names: shall be "xvclk" -- avdd-supply: Analog voltage supply, 2.8 volts -- dovdd-supply: Digital I/O voltage supply, 1.8 volts -- dvdd-supply: Digital core voltage supply, 1.2 volts -- reset-gpios: Low active reset gpio - -The device node shall contain one 'port' child node with an -'endpoint' subnode for its digital output video port, -in accordance with the video interface bindings defined in -Documentation/devicetree/bindings/media/video-interfaces.txt. -The endpoint optional property 'data-lanes' shall be "<1 2>". - -Example: -&i2c7 { - ov5695: camera-sensor@36 { - compatible = "ovti,ov5695"; - reg = <0x36>; - pinctrl-names = "default"; - pinctrl-0 = <&clk_24m_cam>; - - clocks = <&cru SCLK_TESTCLKOUT1>; - clock-names = "xvclk"; - - avdd-supply = <&pp2800_cam>; - dovdd-supply = <&pp1800>; - dvdd-supply = <&pp1250_cam>; - reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; - - port { - wcam_out: endpoint { - remote-endpoint = <&mipi_in_wcam>; - data-lanes = <1 2>; - }; - }; - }; -}; diff --git a/Documentation/devicetree/bindings/media/i2c/ov7251.txt b/Documentation/devicetree/bindings/media/i2c/ov7251.txt deleted file mode 100644 index 8281151f7493..000000000000 --- a/Documentation/devicetree/bindings/media/i2c/ov7251.txt +++ /dev/null @@ -1,52 +0,0 @@ -* Omnivision 1/7.5-Inch B&W VGA CMOS Digital Image Sensor - -The Omnivision OV7251 is a 1/7.5-Inch CMOS active pixel digital image sensor -with an active array size of 640H x 480V. It is programmable through a serial -I2C interface. - -Required Properties: -- compatible: Value should be "ovti,ov7251". -- clocks: Reference to the xclk clock. -- clock-names: Should be "xclk". -- clock-frequency: Frequency of the xclk clock. -- enable-gpios: Chip enable GPIO. Polarity is GPIO_ACTIVE_HIGH. This corresponds - to the hardware pin XSHUTDOWN which is physically active low. -- vdddo-supply: Chip digital IO regulator. -- vdda-supply: Chip analog regulator. -- vddd-supply: Chip digital core regulator. - -The device node shall contain one 'port' child node with a single 'endpoint' -subnode for its digital output video port, in accordance with the video -interface bindings defined in -Documentation/devicetree/bindings/media/video-interfaces.txt. - -Example: - - &i2c1 { - ... - - ov7251: camera-sensor@60 { - compatible = "ovti,ov7251"; - reg = <0x60>; - - enable-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>; - pinctrl-names = "default"; - pinctrl-0 = <&camera_bw_default>; - - clocks = <&clks 200>; - clock-names = "xclk"; - clock-frequency = <24000000>; - - vdddo-supply = <&camera_dovdd_1v8>; - vdda-supply = <&camera_avdd_2v8>; - vddd-supply = <&camera_dvdd_1v2>; - - port { - ov7251_ep: endpoint { - clock-lanes = <1>; - data-lanes = <0>; - remote-endpoint = <&csi0_ep>; - }; - }; - }; - }; diff --git a/Documentation/devicetree/bindings/media/i2c/ovti,ov5693.yaml b/Documentation/devicetree/bindings/media/i2c/ovti,ov5693.yaml index 359dc08440a8..6829a4aadd22 100644 --- a/Documentation/devicetree/bindings/media/i2c/ovti,ov5693.yaml +++ b/Documentation/devicetree/bindings/media/i2c/ovti,ov5693.yaml @@ -5,26 +5,41 @@ $id: http://devicetree.org/schemas/media/i2c/ovti,ov5693.yaml# $schema: http://devicetree.org/meta-schemas/core.yaml# -title: Omnivision OV5693 CMOS Sensor +title: Omnivision OV5693/OV5695 CMOS Sensors maintainers: - Tommaso Merciai description: | - The Omnivision OV5693 is a high performance, 1/4-inch, 5 megapixel, CMOS - image sensor that delivers 2592x1944 at 30fps. It provides full-frame, + The Omnivision OV5693/OV5695 are high performance, 1/4-inch, 5 megapixel, CMOS + image sensors that deliver 2592x1944 at 30fps. It provides full-frame, sub-sampled, and windowed 10-bit MIPI images in various formats via the Serial Camera Control Bus (SCCB) interface. - OV5693 is controlled via I2C and two-wire Serial Camera Control Bus (SCCB). - The sensor output is available via CSI-2 serial data output (up to 2-lane). + OV5693/OV5695 are controlled via I2C and two-wire Serial Camera Control Bus + (SCCB). The sensor output is available via CSI-2 serial data output (up to + 2-lane). allOf: - $ref: /schemas/media/video-interface-devices.yaml# + - if: + properties: + compatible: + contains: + const: ovti,ov5693 + then: + properties: + port: + properties: + endpoint: + required: + - link-frequencies properties: compatible: - const: ovti,ov5693 + enum: + - ovti,ov5693 + - ovti,ov5695 reg: maxItems: 1 @@ -34,6 +49,9 @@ properties: System input clock (aka XVCLK). From 6 to 27 MHz. maxItems: 1 + clock-names: + const: xvclk + dovdd-supply: description: Digital I/O voltage supply, 1.8V. @@ -72,7 +90,6 @@ properties: required: - data-lanes - - link-frequencies required: - compatible diff --git a/Documentation/devicetree/bindings/media/i2c/ovti,ov7251.yaml b/Documentation/devicetree/bindings/media/i2c/ovti,ov7251.yaml new file mode 100644 index 000000000000..2e5187acbbb8 --- /dev/null +++ b/Documentation/devicetree/bindings/media/i2c/ovti,ov7251.yaml @@ -0,0 +1,109 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/media/i2c/ovti,ov7251.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: OmniVision OV7251 Image Sensor + +description: + The Omnivision OV7251 is a 1/7.5-Inch CMOS active pixel digital image sensor + with an active array size of 640H x 480V. It is programmable through a serial + I2C interface. + +maintainers: + - Todor Tomov + +properties: + compatible: + const: ovti,ov7251 + + reg: + maxItems: 1 + + clocks: + description: XCLK Input Clock + + clock-names: + const: xclk + + clock-frequency: + description: Frequency of the xclk clock in Hz. + + vdda-supply: + description: Analog voltage supply, 2.8 volts + + vddd-supply: + description: Digital core voltage supply, 1.2 volts + + vdddo-supply: + description: Digital I/O voltage supply, 1.8 volts + + enable-gpios: + maxItems: 1 + description: + Reference to the GPIO connected to the XSHUTDOWN pin, if any. Polarity + is GPIO_ACTIVE_HIGH. + + port: + description: Digital Output Port + $ref: /schemas/graph.yaml#/$defs/port-base + additionalProperties: false + + properties: + endpoint: + $ref: /schemas/media/video-interfaces.yaml# + unevaluatedProperties: false + + properties: + clock-lanes: + maximum: 1 + + data-lanes: + maxItems: 1 + + link-frequencies: true + + required: + - data-lanes + - link-frequencies + +required: + - compatible + - reg + - clocks + - vdddo-supply + - vdda-supply + - port + +additionalProperties: false + +examples: + - | + #include + + i2c { + #address-cells = <1>; + #size-cells = <0>; + + camera@3c { + compatible = "ovti,ov7251"; + reg = <0x3c>; + clocks = <&clks 1>; + clock-frequency = <24000000>; + vdddo-supply = <&ov7251_vdddo_1v8>; + vdda-supply = <&ov7251_vdda_2v8>; + vddd-supply = <&ov7251_vddd_1v5>; + enable-gpios = <&gpio1 19 GPIO_ACTIVE_HIGH>; + + port { + ov7251_ep: endpoint { + remote-endpoint = <&csi0_ep>; + clock-lanes = <1>; + data-lanes = <0>; + link-frequencies = /bits/ 64 <240000000 319200000>; + }; + }; + }; + }; +... diff --git a/Documentation/devicetree/bindings/media/rockchip-isp1.yaml b/Documentation/devicetree/bindings/media/rockchip-isp1.yaml index 0bad7e640148..e466dff8286d 100644 --- a/Documentation/devicetree/bindings/media/rockchip-isp1.yaml +++ b/Documentation/devicetree/bindings/media/rockchip-isp1.yaml @@ -199,6 +199,7 @@ examples: wcam: camera@36 { compatible = "ovti,ov5695"; reg = <0x36>; + clocks = <&cru SCLK_TESTCLKOUT1>; port { wcam_out: endpoint { diff --git a/Documentation/devicetree/bindings/pwm/brcm,kona-pwm.txt b/Documentation/devicetree/bindings/pwm/brcm,kona-pwm.txt deleted file mode 100644 index c42eecfc81ed..000000000000 --- a/Documentation/devicetree/bindings/pwm/brcm,kona-pwm.txt +++ /dev/null @@ -1,21 +0,0 @@ -Broadcom Kona PWM controller device tree bindings - -This controller has 6 channels. - -Required Properties : -- compatible: should contain "brcm,kona-pwm" -- reg: physical base address and length of the controller's registers -- clocks: phandle + clock specifier pair for the external clock -- #pwm-cells: Should be 3. See pwm.yaml in this directory for a - description of the cells format. - -Refer to clocks/clock-bindings.txt for generic clock consumer properties. - -Example: - -pwm: pwm@3e01a000 { - compatible = "brcm,bcm11351-pwm", "brcm,kona-pwm"; - reg = <0x3e01a000 0xc4>; - clocks = <&pwm_clk>; - #pwm-cells = <3>; -}; diff --git a/Documentation/devicetree/bindings/pwm/brcm,kona-pwm.yaml b/Documentation/devicetree/bindings/pwm/brcm,kona-pwm.yaml new file mode 100644 index 000000000000..e86c8053b366 --- /dev/null +++ b/Documentation/devicetree/bindings/pwm/brcm,kona-pwm.yaml @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/pwm/brcm,kona-pwm.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Broadcom Kona family PWM controller + +description: + This controller has 6 channels. + +maintainers: + - Florian Fainelli + +allOf: + - $ref: pwm.yaml# + +properties: + compatible: + items: + - enum: + - brcm,bcm11351-pwm + - const: brcm,kona-pwm + + reg: + maxItems: 1 + + clocks: + maxItems: 1 + + '#pwm-cells': + const: 3 + +required: + - compatible + - reg + - clocks + +unevaluatedProperties: false + +examples: + - | + #include + + pwm@3e01a000 { + compatible = "brcm,bcm11351-pwm", "brcm,kona-pwm"; + reg = <0x3e01a000 0xcc>; + clocks = <&slave_ccu BCM281XX_SLAVE_CCU_PWM>; + #pwm-cells = <3>; + }; +... diff --git a/Documentation/devicetree/bindings/rtc/atmel,at91rm9200-rtc.yaml b/Documentation/devicetree/bindings/rtc/atmel,at91rm9200-rtc.yaml index 4d2bef15fb7a..c8bb2eef442d 100644 --- a/Documentation/devicetree/bindings/rtc/atmel,at91rm9200-rtc.yaml +++ b/Documentation/devicetree/bindings/rtc/atmel,at91rm9200-rtc.yaml @@ -14,13 +14,17 @@ maintainers: properties: compatible: - enum: - - atmel,at91rm9200-rtc - - atmel,at91sam9x5-rtc - - atmel,sama5d4-rtc - - atmel,sama5d2-rtc - - microchip,sam9x60-rtc - - microchip,sama7g5-rtc + oneOf: + - enum: + - atmel,at91rm9200-rtc + - atmel,at91sam9x5-rtc + - atmel,sama5d4-rtc + - atmel,sama5d2-rtc + - microchip,sam9x60-rtc + - microchip,sama7g5-rtc + - items: + - const: microchip,sam9x7-rtc + - const: microchip,sam9x60-rtc reg: maxItems: 1 diff --git a/Documentation/devicetree/bindings/rtc/intersil,isl12022.yaml b/Documentation/devicetree/bindings/rtc/intersil,isl12022.yaml new file mode 100644 index 000000000000..c2d1441ef273 --- /dev/null +++ b/Documentation/devicetree/bindings/rtc/intersil,isl12022.yaml @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/rtc/intersil,isl12022.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Intersil ISL12022 Real-time Clock + +maintainers: + - Alexandre Belloni + +properties: + compatible: + const: isil,isl12022 + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + '#clock-cells': + const: 0 + + isil,battery-trip-levels-microvolt: + description: + The battery voltages at which the first alarm and second alarm + should trigger (normally ~85% and ~75% of nominal V_BAT). + items: + - enum: [2125000, 2295000, 2550000, 2805000, 3060000, 4250000, 4675000] + - enum: [1875000, 2025000, 2250000, 2475000, 2700000, 3750000, 4125000] + +required: + - compatible + - reg + +allOf: + - $ref: rtc.yaml# + # If #clock-cells is present, interrupts must not be present + - if: + required: + - '#clock-cells' + then: + properties: + interrupts: false + +unevaluatedProperties: false + +examples: + - | + #include + i2c { + #address-cells = <1>; + #size-cells = <0>; + + rtc@6f { + compatible = "isil,isl12022"; + reg = <0x6f>; + interrupts-extended = <&gpio1 5 IRQ_TYPE_LEVEL_LOW>; + isil,battery-trip-levels-microvolt = <2550000>, <2250000>; + }; + }; + +... diff --git a/Documentation/devicetree/bindings/rtc/maxim,ds3231.txt b/Documentation/devicetree/bindings/rtc/maxim,ds3231.txt deleted file mode 100644 index 85be53a42180..000000000000 --- a/Documentation/devicetree/bindings/rtc/maxim,ds3231.txt +++ /dev/null @@ -1,38 +0,0 @@ -* Maxim DS3231 Real Time Clock - -Required properties: -- compatible: Should contain "maxim,ds3231". -- reg: I2C address for chip. - -Optional property: -- #clock-cells: Should be 1. -- clock-output-names: - overwrite the default clock names "ds3231_clk_sqw" and "ds3231_clk_32khz". - -Each clock is assigned an identifier and client nodes can use this identifier -to specify the clock which they consume. Following indices are allowed: - - 0: square-wave output on the SQW pin - - 1: square-wave output on the 32kHz pin - -- interrupts: rtc alarm/event interrupt. When this property is selected, - clock on the SQW pin cannot be used. - -Example: - -ds3231: ds3231@51 { - compatible = "maxim,ds3231"; - reg = <0x68>; - #clock-cells = <1>; -}; - -device1 { -... - clocks = <&ds3231 0>; -... -}; - -device2 { -... - clocks = <&ds3231 1>; -... -}; diff --git a/Documentation/devicetree/bindings/rtc/nxp,pcf2127.yaml b/Documentation/devicetree/bindings/rtc/nxp,pcf2127.yaml index bcb230027622..2d9fe5a75b06 100644 --- a/Documentation/devicetree/bindings/rtc/nxp,pcf2127.yaml +++ b/Documentation/devicetree/bindings/rtc/nxp,pcf2127.yaml @@ -18,6 +18,7 @@ properties: - nxp,pca2129 - nxp,pcf2127 - nxp,pcf2129 + - nxp,pcf2131 reg: maxItems: 1 diff --git a/Documentation/devicetree/bindings/rtc/st,m48t86.yaml b/Documentation/devicetree/bindings/rtc/st,m48t86.yaml new file mode 100644 index 000000000000..e3e12fa23380 --- /dev/null +++ b/Documentation/devicetree/bindings/rtc/st,m48t86.yaml @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/rtc/st,m48t86.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: ST M48T86 / Dallas DS12887 RTC with SRAM + +maintainers: + - Alexandre Belloni + +allOf: + - $ref: rtc.yaml + +properties: + compatible: + enum: + - st,m48t86 + + reg: + items: + - description: index register + - description: data register + +required: + - compatible + - reg + +unevaluatedProperties: false + +examples: + - | + rtc@10800000 { + compatible = "st,m48t86"; + reg = <0x10800000 0x1>, <0x11700000 0x1>; + }; + +... diff --git a/Documentation/devicetree/bindings/rtc/trivial-rtc.yaml b/Documentation/devicetree/bindings/rtc/trivial-rtc.yaml index 9af77f21bb7f..2a65f31ac5a0 100644 --- a/Documentation/devicetree/bindings/rtc/trivial-rtc.yaml +++ b/Documentation/devicetree/bindings/rtc/trivial-rtc.yaml @@ -45,8 +45,6 @@ properties: - isil,isl1208 # Intersil ISL1218 Low Power RTC with Battery Backed SRAM - isil,isl1218 - # Intersil ISL12022 Real-time Clock - - isil,isl12022 # Real Time Clock Module with I2C-Bus - microcrystal,rv3029 # Real Time Clock diff --git a/Documentation/devicetree/bindings/sound/fsl,easrc.yaml b/Documentation/devicetree/bindings/sound/fsl,easrc.yaml index bdde68a1059c..a680d7aff237 100644 --- a/Documentation/devicetree/bindings/sound/fsl,easrc.yaml +++ b/Documentation/devicetree/bindings/sound/fsl,easrc.yaml @@ -14,7 +14,13 @@ properties: pattern: "^easrc@.*" compatible: - const: fsl,imx8mn-easrc + oneOf: + - enum: + - fsl,imx8mn-easrc + - items: + - enum: + - fsl,imx8mp-easrc + - const: fsl,imx8mn-easrc reg: maxItems: 1 diff --git a/Documentation/devicetree/bindings/watchdog/amlogic,meson-gxbb-wdt.yaml b/Documentation/devicetree/bindings/watchdog/amlogic,meson-gxbb-wdt.yaml index f5cc7aa1b93b..443e2e7ab467 100644 --- a/Documentation/devicetree/bindings/watchdog/amlogic,meson-gxbb-wdt.yaml +++ b/Documentation/devicetree/bindings/watchdog/amlogic,meson-gxbb-wdt.yaml @@ -17,6 +17,7 @@ properties: compatible: enum: - amlogic,meson-gxbb-wdt + - amlogic,t7-wdt reg: maxItems: 1 diff --git a/Documentation/devicetree/bindings/watchdog/marvell,cn10624-wdt.yaml b/Documentation/devicetree/bindings/watchdog/marvell,cn10624-wdt.yaml new file mode 100644 index 000000000000..1b583f232e53 --- /dev/null +++ b/Documentation/devicetree/bindings/watchdog/marvell,cn10624-wdt.yaml @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/watchdog/marvell,cn10624-wdt.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Marvell Global Timer (GTI) system watchdog + +maintainers: + - Bharat Bhushan + +allOf: + - $ref: watchdog.yaml# + +properties: + compatible: + oneOf: + - enum: + - marvell,cn9670-wdt + - marvell,cn10624-wdt + + - items: + - enum: + - marvell,cn9880-wdt + - marvell,cnf9535-wdt + - const: marvell,cn9670-wdt + + - items: + - enum: + - marvell,cn10308-wdt + - marvell,cnf10518-wdt + - const: marvell,cn10624-wdt + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + clocks: + maxItems: 1 + + clock-names: + items: + - const: refclk + + marvell,wdt-timer-index: + $ref: /schemas/types.yaml#/definitions/uint32 + minimum: 0 + maximum: 63 + description: + An SoC have many timers (up to 64), firmware can reserve one or more timer + for some other use case and configures one of the global timer as watchdog + timer. Firmware will update this field with the timer number configured + as watchdog timer. + +required: + - compatible + - reg + - interrupts + - clocks + - clock-names + +unevaluatedProperties: false + +examples: + - | + #include + soc { + #address-cells = <2>; + #size-cells = <2>; + + watchdog@802000040000 { + compatible = "marvell,cn9670-wdt"; + reg = <0x00008020 0x00040000 0x00000000 0x00020000>; + interrupts = ; + clocks = <&sclk>; + clock-names = "refclk"; + marvell,wdt-timer-index = <63>; + }; + }; + +... diff --git a/Documentation/devicetree/bindings/watchdog/qcom-wdt.yaml b/Documentation/devicetree/bindings/watchdog/qcom-wdt.yaml index 6d0fe6abd06a..5046dfa55f13 100644 --- a/Documentation/devicetree/bindings/watchdog/qcom-wdt.yaml +++ b/Documentation/devicetree/bindings/watchdog/qcom-wdt.yaml @@ -18,6 +18,7 @@ properties: - items: - enum: - qcom,kpss-wdt-ipq4019 + - qcom,apss-wdt-ipq5018 - qcom,apss-wdt-ipq5332 - qcom,apss-wdt-ipq9574 - qcom,apss-wdt-msm8994 diff --git a/Documentation/devicetree/bindings/watchdog/ti,rti-wdt.yaml b/Documentation/devicetree/bindings/watchdog/ti,rti-wdt.yaml index fc553211e42d..62ddc284a524 100644 --- a/Documentation/devicetree/bindings/watchdog/ti,rti-wdt.yaml +++ b/Documentation/devicetree/bindings/watchdog/ti,rti-wdt.yaml @@ -34,6 +34,20 @@ properties: power-domains: maxItems: 1 + memory-region: + maxItems: 1 + description: + Contains the watchdog reserved memory. It is optional. + In the reserved memory, the specified values, which are + PON_REASON_SOF_NUM(0xBBBBCCCC), PON_REASON_MAGIC_NUM(0xDDDDDDDD), + and PON_REASON_EOF_NUM(0xCCCCBBBB), are pre-stored at the first + 3 * 4 bytes to tell that last boot was caused by watchdog reset. + Once the PON reason is captured by driver(rti_wdt.c), the driver + is supposed to wipe the whole memory region. Surely, if this + property is set, at least 12 bytes reserved memory starting from + specific memory address(0xa220000) should be set. More please + refer to example. + required: - compatible - reg @@ -47,7 +61,18 @@ examples: /* * RTI WDT in main domain on J721e SoC. Assigned clocks are used to * select the source clock for the watchdog, forcing it to tick with - * a 32kHz clock in this case. + * a 32kHz clock in this case. Add a reserved memory(optional) to keep + * the watchdog reset cause persistent, which was be written in 12 bytes + * starting from 0xa2200000 by RTI Watchdog Firmware, then make it + * possible to get watchdog reset cause in driver. + * + * Reserved memory should be defined as follows: + * reserved-memory { + * wdt_reset_memory_region: wdt-memory@a2200000 { + * reg = <0x00 0xa2200000 0x00 0x1000>; + * no-map; + * }; + * } */ #include @@ -58,4 +83,5 @@ examples: power-domains = <&k3_pds 252 TI_SCI_PD_EXCLUSIVE>; assigned-clocks = <&k3_clks 252 1>; assigned-clock-parents = <&k3_clks 252 5>; + memory-region = <&wdt_reset_memory_region>; }; diff --git a/Documentation/features/debug/KASAN/arch-support.txt b/Documentation/features/debug/KASAN/arch-support.txt index bf0124fae643..c4581c2edb28 100644 --- a/Documentation/features/debug/KASAN/arch-support.txt +++ b/Documentation/features/debug/KASAN/arch-support.txt @@ -13,7 +13,7 @@ | csky: | TODO | | hexagon: | TODO | | ia64: | TODO | - | loongarch: | TODO | + | loongarch: | ok | | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | diff --git a/Documentation/features/debug/kcov/arch-support.txt b/Documentation/features/debug/kcov/arch-support.txt index ffcc9f2b1d74..de84cefbcdd3 100644 --- a/Documentation/features/debug/kcov/arch-support.txt +++ b/Documentation/features/debug/kcov/arch-support.txt @@ -13,7 +13,7 @@ | csky: | TODO | | hexagon: | TODO | | ia64: | TODO | - | loongarch: | TODO | + | loongarch: | ok | | m68k: | TODO | | microblaze: | TODO | | mips: | ok | diff --git a/Documentation/features/debug/kgdb/arch-support.txt b/Documentation/features/debug/kgdb/arch-support.txt index 958498f9f2a4..5e91ec78c80b 100644 --- a/Documentation/features/debug/kgdb/arch-support.txt +++ b/Documentation/features/debug/kgdb/arch-support.txt @@ -13,7 +13,7 @@ | csky: | TODO | | hexagon: | ok | | ia64: | TODO | - | loongarch: | TODO | + | loongarch: | ok | | m68k: | TODO | | microblaze: | ok | | mips: | ok | diff --git a/Documentation/filesystems/ceph.rst b/Documentation/filesystems/ceph.rst index 76ce938e7024..085f309ece60 100644 --- a/Documentation/filesystems/ceph.rst +++ b/Documentation/filesystems/ceph.rst @@ -57,6 +57,16 @@ a snapshot on any subdirectory (and its nested contents) in the system. Snapshot creation and deletion are as simple as 'mkdir .snap/foo' and 'rmdir .snap/foo'. +Snapshot names have two limitations: + +* They can not start with an underscore ('_'), as these names are reserved + for internal usage by the MDS. +* They can not exceed 240 characters in size. This is because the MDS makes + use of long snapshot names internally, which follow the format: + `__`. Since filenames in general can't have + more than 255 characters, and `` takes 13 characters, the long + snapshot names can take as much as 255 - 1 - 1 - 13 = 240. + Ceph also provides some recursive accounting on directories for nested files and bytes. That is, a 'getfattr -d foo' on any directory in the system will reveal the total number of nested regular files and diff --git a/Documentation/filesystems/gfs2-glocks.rst b/Documentation/filesystems/gfs2-glocks.rst index bec25c8b3e4b..8a5842929b60 100644 --- a/Documentation/filesystems/gfs2-glocks.rst +++ b/Documentation/filesystems/gfs2-glocks.rst @@ -20,8 +20,7 @@ The gl_holders list contains all the queued lock requests (not just the holders) associated with the glock. If there are any held locks, then they will be contiguous entries at the head of the list. Locks are granted in strictly the order that they -are queued, except for those marked LM_FLAG_PRIORITY which are -used only during recovery, and even then only for journal locks. +are queued. There are three lock states that users of the glock layer can request, namely shared (SH), deferred (DF) and exclusive (EX). Those translate diff --git a/Documentation/gpu/amdgpu/driver-misc.rst b/Documentation/gpu/amdgpu/driver-misc.rst index be131e963d87..4321c38fef21 100644 --- a/Documentation/gpu/amdgpu/driver-misc.rst +++ b/Documentation/gpu/amdgpu/driver-misc.rst @@ -11,19 +11,19 @@ via sysfs product_name ------------ -.. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +.. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.c :doc: product_name product_number -------------- -.. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_device.c - :doc: product_name +.. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.c + :doc: product_number serial_number ------------- -.. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +.. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.c :doc: serial_number unique_id diff --git a/Documentation/gpu/automated_testing.rst b/Documentation/gpu/automated_testing.rst new file mode 100644 index 000000000000..469b6fb65c30 --- /dev/null +++ b/Documentation/gpu/automated_testing.rst @@ -0,0 +1,144 @@ +.. SPDX-License-Identifier: GPL-2.0+ + +========================================= +Automated testing of the DRM subsystem +========================================= + +Introduction +============ + +Making sure that changes to the core or drivers don't introduce regressions can +be very time-consuming when lots of different hardware configurations need to +be tested. Moreover, it isn't practical for each person interested in this +testing to have to acquire and maintain what can be a considerable amount of +hardware. + +Also, it is desirable for developers to check for regressions in their code by +themselves, instead of relying on the maintainers to find them and then +reporting back. + +There are facilities in gitlab.freedesktop.org to automatically test Mesa that +can be used as well for testing the DRM subsystem. This document explains how +people interested in testing it can use this shared infrastructure to save +quite some time and effort. + + +Relevant files +============== + +drivers/gpu/drm/ci/gitlab-ci.yml +-------------------------------- + +This is the root configuration file for GitLab CI. Among other less interesting +bits, it specifies the specific version of the scripts to be used. There are +some variables that can be modified to change the behavior of the pipeline: + +DRM_CI_PROJECT_PATH + Repository that contains the Mesa software infrastructure for CI + +DRM_CI_COMMIT_SHA + A particular revision to use from that repository + +UPSTREAM_REPO + URL to git repository containing the target branch + +TARGET_BRANCH + Branch to which this branch is to be merged into + +IGT_VERSION + Revision of igt-gpu-tools being used, from + https://gitlab.freedesktop.org/drm/igt-gpu-tools + +drivers/gpu/drm/ci/testlist.txt +------------------------------- + +IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt +file, see below). + +drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt +---------------------------------------------------------- + +Lists the known failures for a given driver on a specific hardware revision. + +drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt +----------------------------------------------------------- + +Lists the tests that for a given driver on a specific hardware revision are +known to behave unreliably. These tests won't cause a job to fail regardless of +the result. They will still be run. + +drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt +----------------------------------------------------------- + +Lists the tests that won't be run for a given driver on a specific hardware +revision. These are usually tests that interfere with the running of the test +list due to hanging the machine, causing OOM, taking too long, etc. + + +How to enable automated testing on your tree +============================================ + +1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one +yet + +2. In your kernel repo's configuration (eg. +https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the +CI/CD configuration file from .gitlab-ci.yml to +drivers/gpu/drm/ci/gitlab-ci.yml. + +3. Next time you push to this repository, you will see a CI pipeline being +created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines) + +4. The various jobs will be run and when the pipeline is finished, all jobs +should be green unless a regression has been found. + + +How to update test expectations +=============================== + +If your changes to the code fix any tests, you will have to remove one or more +lines from one or more of the files in +drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms +affected by the change. + + +How to expand coverage +====================== + +If your code changes make it possible to run more tests (by solving reliability +issues, for example), you can remove tests from the flakes and/or skips lists, +and then the expected results if there are any known failures. + +If there is a need for updating the version of IGT being used (maybe you have +added more tests to it), update the IGT_VERSION variable at the top of the +gitlab-ci.yml file. + + +How to test your changes to the scripts +======================================= + +For testing changes to the scripts in the drm-ci repo, change the +DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in +drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg. +janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/. + + +How to incorporate external fixes in your testing +================================================= + +Often, regressions in other trees will prevent testing changes local to the +tree under test. These fixes will be automatically merged in during the build +jobs from a branch in the target tree that is named as +${TARGET_BRANCH}-external-fixes. + +If the pipeline is not in a merge request and a branch with the same name +exists in the local tree, commits from that branch will be merged in as well. + + +How to deal with automated testing labs that may be down +======================================================== + +If a hardware farm is down and thus causing pipelines to fail that would +otherwise pass, one can disable all jobs that would be submitted to that farm +by editing the file at +https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml. diff --git a/Documentation/gpu/index.rst b/Documentation/gpu/index.rst index eee5996acf2c..e45ff0915246 100644 --- a/Documentation/gpu/index.rst +++ b/Documentation/gpu/index.rst @@ -17,6 +17,7 @@ GPU Driver Developer's Guide backlight vga-switcheroo vgaarbiter + automated_testing todo rfc/index diff --git a/Documentation/process/maintainer-netdev.rst b/Documentation/process/maintainer-netdev.rst index c1c732e9748b..09dcf6377c27 100644 --- a/Documentation/process/maintainer-netdev.rst +++ b/Documentation/process/maintainer-netdev.rst @@ -98,7 +98,7 @@ If you aren't subscribed to netdev and/or are simply unsure if repository link above for any new networking-related commits. You may also check the following website for the current status: - https://patchwork.hopto.org/net-next.html + https://netdev.bots.linux.dev/net-next.html The ``net`` tree continues to collect fixes for the vX.Y content, and is fed back to Linus at regular (~weekly) intervals. Meaning that the @@ -120,7 +120,37 @@ queue for netdev: https://patchwork.kernel.org/project/netdevbpf/list/ The "State" field will tell you exactly where things are at with your -patch. Patches are indexed by the ``Message-ID`` header of the emails +patch: + +================== ============================================================= +Patch state Description +================== ============================================================= +New, Under review pending review, patch is in the maintainer’s queue for + review; the two states are used interchangeably (depending on + the exact co-maintainer handling patchwork at the time) +Accepted patch was applied to the appropriate networking tree, this is + usually set automatically by the pw-bot +Needs ACK waiting for an ack from an area expert or testing +Changes requested patch has not passed the review, new revision is expected + with appropriate code and commit message changes +Rejected patch has been rejected and new revision is not expected +Not applicable patch is expected to be applied outside of the networking + subsystem +Awaiting upstream patch should be reviewed and handled by appropriate + sub-maintainer, who will send it on to the networking trees; + patches set to ``Awaiting upstream`` in netdev's patchwork + will usually remain in this state, whether the sub-maintainer + requested changes, accepted or rejected the patch +Deferred patch needs to be reposted later, usually due to dependency + or because it was posted for a closed tree +Superseded new version of the patch was posted, usually set by the + pw-bot +RFC not to be applied, usually not in maintainer’s review queue, + pw-bot can automatically set patches to this state based + on subject tags +================== ============================================================= + +Patches are indexed by the ``Message-ID`` header of the emails which carried them so if you have trouble finding your patch append the value of ``Message-ID`` to the URL above. @@ -155,7 +185,7 @@ must match the MAINTAINERS entry) and a handful of senior reviewers. Bot records its activity here: - https://patchwork.hopto.org/pw-bot.html + https://netdev.bots.linux.dev/pw-bot.html Review timelines ~~~~~~~~~~~~~~~~ diff --git a/Documentation/riscv/hwprobe.rst b/Documentation/riscv/hwprobe.rst index 20eff9650da9..a52996b22f75 100644 --- a/Documentation/riscv/hwprobe.rst +++ b/Documentation/riscv/hwprobe.rst @@ -87,13 +87,12 @@ The following keys are defined: emulated via software, either in or below the kernel. These accesses are always extremely slow. - * :c:macro:`RISCV_HWPROBE_MISALIGNED_SLOW`: Misaligned accesses are supported - in hardware, but are slower than the corresponding aligned accesses - sequences. + * :c:macro:`RISCV_HWPROBE_MISALIGNED_SLOW`: Misaligned accesses are slower + than equivalent byte accesses. Misaligned accesses may be supported + directly in hardware, or trapped and emulated by software. - * :c:macro:`RISCV_HWPROBE_MISALIGNED_FAST`: Misaligned accesses are supported - in hardware and are faster than the corresponding aligned accesses - sequences. + * :c:macro:`RISCV_HWPROBE_MISALIGNED_FAST`: Misaligned accesses are faster + than equivalent byte accesses. * :c:macro:`RISCV_HWPROBE_MISALIGNED_UNSUPPORTED`: Misaligned accesses are not supported at all and will generate a misaligned address fault. diff --git a/Documentation/translations/zh_CN/dev-tools/kasan.rst b/Documentation/translations/zh_CN/dev-tools/kasan.rst index 05ef904dbcfb..8fdb20c9665b 100644 --- a/Documentation/translations/zh_CN/dev-tools/kasan.rst +++ b/Documentation/translations/zh_CN/dev-tools/kasan.rst @@ -42,7 +42,7 @@ KASAN有三种模式: 体系架构 ~~~~~~~~ -在x86_64、arm、arm64、powerpc、riscv、s390和xtensa上支持通用KASAN， +在x86_64、arm、arm64、powerpc、riscv、s390、xtensa和loongarch上支持通用KASAN， 而基于标签的KASAN模式只在arm64上支持。 编译器 diff --git a/Documentation/userspace-api/netlink/intro.rst b/Documentation/userspace-api/netlink/intro.rst index af94e71761ec..7b1d401210ef 100644 --- a/Documentation/userspace-api/netlink/intro.rst +++ b/Documentation/userspace-api/netlink/intro.rst @@ -528,6 +528,8 @@ families may, however, require a larger buffer. 32kB buffer is recommended for most efficient handling of dumps (larger buffer fits more dumped objects and therefore fewer recvmsg() calls are needed). +.. _classic_netlink: + Classic Netlink =============== diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 73db30cb60fb..21a7578142a1 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -2259,6 +2259,8 @@ Errors: EINVAL invalid register ID, or no such register or used with VMs in protected virtualization mode on s390 EPERM (arm64) register access not allowed before vcpu finalization + EBUSY (riscv) changing register value not allowed after the vcpu + has run at least once ====== ============================================================ (These error codes are indicative only: do not rely on a specific error @@ -3499,7 +3501,7 @@ VCPU matching underlying host. --------------------- :Capability: basic -:Architectures: arm64, mips +:Architectures: arm64, mips, riscv :Type: vcpu ioctl :Parameters: struct kvm_reg_list (in/out) :Returns: 0 on success; -1 on error diff --git a/MAINTAINERS b/MAINTAINERS index a653a7b46258..70319e9694c1 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -6118,7 +6118,7 @@ F: include/video/udlfb.h DISTRIBUTED LOCK MANAGER (DLM) M: Christine Caulfield M: David Teigland -L: cluster-devel@redhat.com +L: gfs2@lists.linux.dev S: Supported W: http://sources.redhat.com/cluster/ T: git git://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm.git @@ -7165,6 +7165,14 @@ T: git git://anongit.freedesktop.org/drm/drm-misc F: drivers/gpu/drm/ttm/ F: include/drm/ttm/ +DRM AUTOMATED TESTING +M: Helen Koike +L: dri-devel@lists.freedesktop.org +S: Maintained +T: git git://anongit.freedesktop.org/drm/drm-misc +F: Documentation/gpu/automated_testing.rst +F: drivers/gpu/drm/ci/ + DSBR100 USB FM RADIO DRIVER M: Alexey Klimov L: linux-media@vger.kernel.org @@ -8774,7 +8782,7 @@ F: scripts/get_maintainer.pl GFS2 FILE SYSTEM M: Bob Peterson M: Andreas Gruenbacher -L: cluster-devel@redhat.com +L: gfs2@lists.linux.dev S: Supported B: https://bugzilla.kernel.org/enter_bug.cgi?product=File%20System&component=gfs2 T: git git://git.kernel.org/pub/scm/linux/kernel/git/gfs2/linux-gfs2.git @@ -11596,6 +11604,8 @@ F: arch/x86/include/uapi/asm/svm.h F: arch/x86/include/uapi/asm/vmx.h F: arch/x86/kvm/ F: arch/x86/kvm/*/ +F: tools/testing/selftests/kvm/*/x86_64/ +F: tools/testing/selftests/kvm/x86_64/ KERNFS M: Greg Kroah-Hartman @@ -16768,6 +16778,8 @@ L: linux-kernel@vger.kernel.org S: Supported W: https://perf.wiki.kernel.org/ T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/core +T: git git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools.git perf-tools +T: git git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git perf-tools-next F: arch/*/events/* F: arch/*/events/*/* F: arch/*/include/asm/perf_event.h @@ -18089,7 +18101,6 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/abelloni/linux.git F: Documentation/admin-guide/rtc.rst F: Documentation/devicetree/bindings/rtc/ F: drivers/rtc/ -F: include/linux/platform_data/rtc-* F: include/linux/rtc.h F: include/linux/rtc/ F: include/uapi/linux/rtc.h @@ -20412,6 +20423,13 @@ S: Supported T: git git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git F: drivers/staging/ +STANDALONE CACHE CONTROLLER DRIVERS +M: Conor Dooley +L: linux-riscv@lists.infradead.org +S: Maintained +T: git https://git.kernel.org/pub/scm/linux/kernel/git/conor/linux.git/ +F: drivers/cache + STARFIRE/DURALAN NETWORK DRIVER M: Ion Badulescu S: Odd Fixes @@ -21250,7 +21268,7 @@ F: sound/soc/ti/ TEXAS INSTRUMENTS AUDIO (ASoC/HDA) DRIVERS M: Shenghao Ding M: Kevin Lu -M: Baojun Xu +M: Baojun Xu L: alsa-devel@alsa-project.org (moderated for non-subscribers) S: Maintained F: Documentation/devicetree/bindings/sound/tas2552.txt diff --git a/arch/arm/include/asm/arm_pmuv3.h b/arch/arm/include/asm/arm_pmuv3.h index f3cd04ff022d..72529f5e2bed 100644 --- a/arch/arm/include/asm/arm_pmuv3.h +++ b/arch/arm/include/asm/arm_pmuv3.h @@ -227,6 +227,8 @@ static inline bool kvm_set_pmuserenr(u64 val) return false; } +static inline void kvm_vcpu_pmu_resync_el0(void) {} + /* PMU Version in DFR Register */ #define ARMV8_PMU_DFR_VER_NI 0 #define ARMV8_PMU_DFR_VER_V3P4 0x5 diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index f482b994c608..bcd5622aa096 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -156,4 +156,6 @@ static inline void efi_capsule_flush_cache_range(void *addr, int size) efi_status_t efi_handle_corrupted_x18(efi_status_t s, const char *f); +void efi_icache_sync(unsigned long start, unsigned long end); + #endif /* _ASM_EFI_H */ diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 58e5eb27da68..5882b2415596 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -18,10 +18,19 @@ #define HCR_DCT (UL(1) << 57) #define HCR_ATA_SHIFT 56 #define HCR_ATA (UL(1) << HCR_ATA_SHIFT) +#define HCR_TTLBOS (UL(1) << 55) +#define HCR_TTLBIS (UL(1) << 54) +#define HCR_ENSCXT (UL(1) << 53) +#define HCR_TOCU (UL(1) << 52) #define HCR_AMVOFFEN (UL(1) << 51) +#define HCR_TICAB (UL(1) << 50) #define HCR_TID4 (UL(1) << 49) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_NV2 (UL(1) << 45) +#define HCR_AT (UL(1) << 44) +#define HCR_NV1 (UL(1) << 43) +#define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) #define HCR_APK (UL(1) << 40) #define HCR_TEA (UL(1) << 37) @@ -89,7 +98,6 @@ HCR_BSU_IS | HCR_FB | HCR_TACR | \ HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \ HCR_FMO | HCR_IMO | HCR_PTW | HCR_TID3) -#define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF) #define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA) #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC) #define HCR_HOST_VHE_FLAGS (HCR_RW | HCR_TGE | HCR_E2H) @@ -324,6 +332,47 @@ BIT(18) | \ GENMASK(16, 15)) +/* + * FGT register definitions + * + * RES0 and polarity masks as of DDI0487J.a, to be updated as needed. + * We're not using the generated masks as they are usually ahead of + * the published ARM ARM, which we use as a reference. + * + * Once we get to a point where the two describe the same thing, we'll + * merge the definitions. One day. + */ +#define __HFGRTR_EL2_RES0 (GENMASK(63, 56) | GENMASK(53, 51)) +#define __HFGRTR_EL2_MASK GENMASK(49, 0) +#define __HFGRTR_EL2_nMASK (GENMASK(55, 54) | BIT(50)) + +#define __HFGWTR_EL2_RES0 (GENMASK(63, 56) | GENMASK(53, 51) | \ + BIT(46) | BIT(42) | BIT(40) | BIT(28) | \ + GENMASK(26, 25) | BIT(21) | BIT(18) | \ + GENMASK(15, 14) | GENMASK(10, 9) | BIT(2)) +#define __HFGWTR_EL2_MASK GENMASK(49, 0) +#define __HFGWTR_EL2_nMASK (GENMASK(55, 54) | BIT(50)) + +#define __HFGITR_EL2_RES0 GENMASK(63, 57) +#define __HFGITR_EL2_MASK GENMASK(54, 0) +#define __HFGITR_EL2_nMASK GENMASK(56, 55) + +#define __HDFGRTR_EL2_RES0 (BIT(49) | BIT(42) | GENMASK(39, 38) | \ + GENMASK(21, 20) | BIT(8)) +#define __HDFGRTR_EL2_MASK ~__HDFGRTR_EL2_nMASK +#define __HDFGRTR_EL2_nMASK GENMASK(62, 59) + +#define __HDFGWTR_EL2_RES0 (BIT(63) | GENMASK(59, 58) | BIT(51) | BIT(47) | \ + BIT(43) | GENMASK(40, 38) | BIT(34) | BIT(30) | \ + BIT(22) | BIT(9) | BIT(6)) +#define __HDFGWTR_EL2_MASK ~__HDFGWTR_EL2_nMASK +#define __HDFGWTR_EL2_nMASK GENMASK(62, 60) + +/* Similar definitions for HCRX_EL2 */ +#define __HCRX_EL2_RES0 (GENMASK(63, 16) | GENMASK(13, 12)) +#define __HCRX_EL2_MASK (0) +#define __HCRX_EL2_nMASK (GENMASK(15, 14) | GENMASK(4, 0)) + /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */ #define HPFAR_MASK (~UL(0xf)) /* diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 24e28bb2d95b..24b5e6b23417 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -70,6 +70,7 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa, __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_ipa_nsh, __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid, + __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range, __KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context, __KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff, __KVM_HOST_SMCCC_FUNC___vgic_v3_read_vmcr, @@ -229,6 +230,8 @@ extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, extern void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level); +extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, unsigned long pages); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_timer_set_cntvoff(u64 cntvoff); diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index d3dd05bbfe23..af06ccb7ee34 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -49,6 +49,7 @@ #define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) #define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5) #define KVM_REQ_SUSPEND KVM_ARCH_REQ(6) +#define KVM_REQ_RESYNC_PMU_EL0 KVM_ARCH_REQ(7) #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET) @@ -380,6 +381,7 @@ enum vcpu_sysreg { CPTR_EL2, /* Architectural Feature Trap Register (EL2) */ HSTR_EL2, /* Hypervisor System Trap Register */ HACR_EL2, /* Hypervisor Auxiliary Control Register */ + HCRX_EL2, /* Extended Hypervisor Configuration Register */ TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */ TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */ TCR_EL2, /* Translation Control Register (EL2) */ @@ -400,6 +402,11 @@ enum vcpu_sysreg { TPIDR_EL2, /* EL2 Software Thread ID Register */ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ SP_EL2, /* EL2 Stack Pointer */ + HFGRTR_EL2, + HFGWTR_EL2, + HFGITR_EL2, + HDFGRTR_EL2, + HDFGWTR_EL2, CNTHP_CTL_EL2, CNTHP_CVAL_EL2, CNTHV_CTL_EL2, @@ -567,8 +574,7 @@ struct kvm_vcpu_arch { /* Cache some mmu pages needed inside spinlock regions */ struct kvm_mmu_memory_cache mmu_page_cache; - /* Target CPU and feature flags */ - int target; + /* feature flags */ DECLARE_BITMAP(features, KVM_VCPU_MAX_FEATURES); /* Virtual SError ESR to restore when HCR_EL2.VSE is set */ @@ -669,6 +675,8 @@ struct kvm_vcpu_arch { #define VCPU_SVE_FINALIZED __vcpu_single_flag(cflags, BIT(1)) /* PTRAUTH exposed to guest */ #define GUEST_HAS_PTRAUTH __vcpu_single_flag(cflags, BIT(2)) +/* KVM_ARM_VCPU_INIT completed */ +#define VCPU_INITIALIZED __vcpu_single_flag(cflags, BIT(3)) /* Exception pending */ #define PENDING_EXCEPTION __vcpu_single_flag(iflags, BIT(0)) @@ -899,7 +907,6 @@ struct kvm_vcpu_stat { u64 exits; }; -void kvm_vcpu_preferred_target(struct kvm_vcpu_init *init); unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu); int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices); int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg); @@ -967,8 +974,6 @@ void kvm_arm_resume_guest(struct kvm *kvm); #define kvm_call_hyp_nvhe(f, ...) f(__VA_ARGS__) #endif /* __KVM_NVHE_HYPERVISOR__ */ -void force_vm_exit(const cpumask_t *mask); - int handle_exit(struct kvm_vcpu *vcpu, int exception_index); void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index); @@ -983,6 +988,7 @@ int kvm_handle_cp10_id(struct kvm_vcpu *vcpu); void kvm_reset_sys_regs(struct kvm_vcpu *vcpu); int __init kvm_sys_reg_table_init(void); +int __init populate_nv_trap_config(void); bool lock_all_vcpus(struct kvm *kvm); void unlock_all_vcpus(struct kvm *kvm); @@ -1049,8 +1055,6 @@ static inline bool kvm_system_needs_idmapped_vectors(void) return cpus_have_const_cap(ARM64_SPECTRE_V3A); } -void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu); - static inline void kvm_arch_sync_events(struct kvm *kvm) {} static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {} @@ -1113,13 +1117,15 @@ int __init kvm_set_ipa_limit(void); #define __KVM_HAVE_ARCH_VM_ALLOC struct kvm *kvm_arch_alloc_vm(void); +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS + +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE + static inline bool kvm_vm_is_protected(struct kvm *kvm) { return false; } -void kvm_init_protected_traps(struct kvm_vcpu *vcpu); - int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature); bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 0e1e1ab17b4d..96a80e8f6226 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -168,6 +168,7 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, void __iomem **haddr); int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr); +int create_hyp_stack(phys_addr_t phys_addr, unsigned long *haddr); void __init free_hyp_pgds(void); void stage2_unmap_vm(struct kvm *kvm); diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 8fb67f032fd1..fa23cc9c2adc 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -11,6 +11,8 @@ static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); } +extern bool __check_nv_sr_forward(struct kvm_vcpu *vcpu); + struct sys_reg_params; struct sys_reg_desc; diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 929d355eae0a..d3e354bb8351 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -746,4 +746,14 @@ enum kvm_pgtable_prot kvm_pgtable_stage2_pte_prot(kvm_pte_t pte); * kvm_pgtable_prot format. */ enum kvm_pgtable_prot kvm_pgtable_hyp_pte_prot(kvm_pte_t pte); + +/** + * kvm_tlb_flush_vmid_range() - Invalidate/flush a range of TLB entries + * + * @mmu: Stage-2 KVM MMU struct + * @addr: The base Intermediate physical address from which to invalidate + * @size: Size of the range from the base to invalidate + */ +void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, size_t size); #endif /* __ARM64_KVM_PGTABLE_H__ */ diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 16464bf9a8aa..38296579a4fd 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -124,6 +124,37 @@ #define SYS_DC_CIGSW sys_insn(1, 0, 7, 14, 4) #define SYS_DC_CIGDSW sys_insn(1, 0, 7, 14, 6) +#define SYS_IC_IALLUIS sys_insn(1, 0, 7, 1, 0) +#define SYS_IC_IALLU sys_insn(1, 0, 7, 5, 0) +#define SYS_IC_IVAU sys_insn(1, 3, 7, 5, 1) + +#define SYS_DC_IVAC sys_insn(1, 0, 7, 6, 1) +#define SYS_DC_IGVAC sys_insn(1, 0, 7, 6, 3) +#define SYS_DC_IGDVAC sys_insn(1, 0, 7, 6, 5) + +#define SYS_DC_CVAC sys_insn(1, 3, 7, 10, 1) +#define SYS_DC_CGVAC sys_insn(1, 3, 7, 10, 3) +#define SYS_DC_CGDVAC sys_insn(1, 3, 7, 10, 5) + +#define SYS_DC_CVAU sys_insn(1, 3, 7, 11, 1) + +#define SYS_DC_CVAP sys_insn(1, 3, 7, 12, 1) +#define SYS_DC_CGVAP sys_insn(1, 3, 7, 12, 3) +#define SYS_DC_CGDVAP sys_insn(1, 3, 7, 12, 5) + +#define SYS_DC_CVADP sys_insn(1, 3, 7, 13, 1) +#define SYS_DC_CGVADP sys_insn(1, 3, 7, 13, 3) +#define SYS_DC_CGDVADP sys_insn(1, 3, 7, 13, 5) + +#define SYS_DC_CIVAC sys_insn(1, 3, 7, 14, 1) +#define SYS_DC_CIGVAC sys_insn(1, 3, 7, 14, 3) +#define SYS_DC_CIGDVAC sys_insn(1, 3, 7, 14, 5) + +/* Data cache zero operations */ +#define SYS_DC_ZVA sys_insn(1, 3, 7, 4, 1) +#define SYS_DC_GVA sys_insn(1, 3, 7, 4, 3) +#define SYS_DC_GZVA sys_insn(1, 3, 7, 4, 4) + /* * Automatically generated definitions for system registers, the * manual encodings below are in the process of being converted to @@ -163,6 +194,82 @@ #define SYS_DBGDTRTX_EL0 sys_reg(2, 3, 0, 5, 0) #define SYS_DBGVCR32_EL2 sys_reg(2, 4, 0, 7, 0) +#define SYS_BRBINF_EL1(n) sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 0)) +#define SYS_BRBINFINJ_EL1 sys_reg(2, 1, 9, 1, 0) +#define SYS_BRBSRC_EL1(n) sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 1)) +#define SYS_BRBSRCINJ_EL1 sys_reg(2, 1, 9, 1, 1) +#define SYS_BRBTGT_EL1(n) sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 2)) +#define SYS_BRBTGTINJ_EL1 sys_reg(2, 1, 9, 1, 2) +#define SYS_BRBTS_EL1 sys_reg(2, 1, 9, 0, 2) + +#define SYS_BRBCR_EL1 sys_reg(2, 1, 9, 0, 0) +#define SYS_BRBFCR_EL1 sys_reg(2, 1, 9, 0, 1) +#define SYS_BRBIDR0_EL1 sys_reg(2, 1, 9, 2, 0) + +#define SYS_TRCITECR_EL1 sys_reg(3, 0, 1, 2, 3) +#define SYS_TRCACATR(m) sys_reg(2, 1, 2, ((m & 7) << 1), (2 | (m >> 3))) +#define SYS_TRCACVR(m) sys_reg(2, 1, 2, ((m & 7) << 1), (0 | (m >> 3))) +#define SYS_TRCAUTHSTATUS sys_reg(2, 1, 7, 14, 6) +#define SYS_TRCAUXCTLR sys_reg(2, 1, 0, 6, 0) +#define SYS_TRCBBCTLR sys_reg(2, 1, 0, 15, 0) +#define SYS_TRCCCCTLR sys_reg(2, 1, 0, 14, 0) +#define SYS_TRCCIDCCTLR0 sys_reg(2, 1, 3, 0, 2) +#define SYS_TRCCIDCCTLR1 sys_reg(2, 1, 3, 1, 2) +#define SYS_TRCCIDCVR(m) sys_reg(2, 1, 3, ((m & 7) << 1), 0) +#define SYS_TRCCLAIMCLR sys_reg(2, 1, 7, 9, 6) +#define SYS_TRCCLAIMSET sys_reg(2, 1, 7, 8, 6) +#define SYS_TRCCNTCTLR(m) sys_reg(2, 1, 0, (4 | (m & 3)), 5) +#define SYS_TRCCNTRLDVR(m) sys_reg(2, 1, 0, (0 | (m & 3)), 5) +#define SYS_TRCCNTVR(m) sys_reg(2, 1, 0, (8 | (m & 3)), 5) +#define SYS_TRCCONFIGR sys_reg(2, 1, 0, 4, 0) +#define SYS_TRCDEVARCH sys_reg(2, 1, 7, 15, 6) +#define SYS_TRCDEVID sys_reg(2, 1, 7, 2, 7) +#define SYS_TRCEVENTCTL0R sys_reg(2, 1, 0, 8, 0) +#define SYS_TRCEVENTCTL1R sys_reg(2, 1, 0, 9, 0) +#define SYS_TRCEXTINSELR(m) sys_reg(2, 1, 0, (8 | (m & 3)), 4) +#define SYS_TRCIDR0 sys_reg(2, 1, 0, 8, 7) +#define SYS_TRCIDR10 sys_reg(2, 1, 0, 2, 6) +#define SYS_TRCIDR11 sys_reg(2, 1, 0, 3, 6) +#define SYS_TRCIDR12 sys_reg(2, 1, 0, 4, 6) +#define SYS_TRCIDR13 sys_reg(2, 1, 0, 5, 6) +#define SYS_TRCIDR1 sys_reg(2, 1, 0, 9, 7) +#define SYS_TRCIDR2 sys_reg(2, 1, 0, 10, 7) +#define SYS_TRCIDR3 sys_reg(2, 1, 0, 11, 7) +#define SYS_TRCIDR4 sys_reg(2, 1, 0, 12, 7) +#define SYS_TRCIDR5 sys_reg(2, 1, 0, 13, 7) +#define SYS_TRCIDR6 sys_reg(2, 1, 0, 14, 7) +#define SYS_TRCIDR7 sys_reg(2, 1, 0, 15, 7) +#define SYS_TRCIDR8 sys_reg(2, 1, 0, 0, 6) +#define SYS_TRCIDR9 sys_reg(2, 1, 0, 1, 6) +#define SYS_TRCIMSPEC(m) sys_reg(2, 1, 0, (m & 7), 7) +#define SYS_TRCITEEDCR sys_reg(2, 1, 0, 2, 1) +#define SYS_TRCOSLSR sys_reg(2, 1, 1, 1, 4) +#define SYS_TRCPRGCTLR sys_reg(2, 1, 0, 1, 0) +#define SYS_TRCQCTLR sys_reg(2, 1, 0, 1, 1) +#define SYS_TRCRSCTLR(m) sys_reg(2, 1, 1, (m & 15), (0 | (m >> 4))) +#define SYS_TRCRSR sys_reg(2, 1, 0, 10, 0) +#define SYS_TRCSEQEVR(m) sys_reg(2, 1, 0, (m & 3), 4) +#define SYS_TRCSEQRSTEVR sys_reg(2, 1, 0, 6, 4) +#define SYS_TRCSEQSTR sys_reg(2, 1, 0, 7, 4) +#define SYS_TRCSSCCR(m) sys_reg(2, 1, 1, (m & 7), 2) +#define SYS_TRCSSCSR(m) sys_reg(2, 1, 1, (8 | (m & 7)), 2) +#define SYS_TRCSSPCICR(m) sys_reg(2, 1, 1, (m & 7), 3) +#define SYS_TRCSTALLCTLR sys_reg(2, 1, 0, 11, 0) +#define SYS_TRCSTATR sys_reg(2, 1, 0, 3, 0) +#define SYS_TRCSYNCPR sys_reg(2, 1, 0, 13, 0) +#define SYS_TRCTRACEIDR sys_reg(2, 1, 0, 0, 1) +#define SYS_TRCTSCTLR sys_reg(2, 1, 0, 12, 0) +#define SYS_TRCVICTLR sys_reg(2, 1, 0, 0, 2) +#define SYS_TRCVIIECTLR sys_reg(2, 1, 0, 1, 2) +#define SYS_TRCVIPCSSCTLR sys_reg(2, 1, 0, 3, 2) +#define SYS_TRCVISSCTLR sys_reg(2, 1, 0, 2, 2) +#define SYS_TRCVMIDCCTLR0 sys_reg(2, 1, 3, 2, 2) +#define SYS_TRCVMIDCCTLR1 sys_reg(2, 1, 3, 3, 2) +#define SYS_TRCVMIDCVR(m) sys_reg(2, 1, 3, ((m & 7) << 1), 1) + +/* ETM */ +#define SYS_TRCOSLAR sys_reg(2, 1, 1, 0, 4) + #define SYS_MIDR_EL1 sys_reg(3, 0, 0, 0, 0) #define SYS_MPIDR_EL1 sys_reg(3, 0, 0, 0, 5) #define SYS_REVIDR_EL1 sys_reg(3, 0, 0, 0, 6) @@ -203,8 +310,13 @@ #define SYS_ERXCTLR_EL1 sys_reg(3, 0, 5, 4, 1) #define SYS_ERXSTATUS_EL1 sys_reg(3, 0, 5, 4, 2) #define SYS_ERXADDR_EL1 sys_reg(3, 0, 5, 4, 3) +#define SYS_ERXPFGF_EL1 sys_reg(3, 0, 5, 4, 4) +#define SYS_ERXPFGCTL_EL1 sys_reg(3, 0, 5, 4, 5) +#define SYS_ERXPFGCDN_EL1 sys_reg(3, 0, 5, 4, 6) #define SYS_ERXMISC0_EL1 sys_reg(3, 0, 5, 5, 0) #define SYS_ERXMISC1_EL1 sys_reg(3, 0, 5, 5, 1) +#define SYS_ERXMISC2_EL1 sys_reg(3, 0, 5, 5, 2) +#define SYS_ERXMISC3_EL1 sys_reg(3, 0, 5, 5, 3) #define SYS_TFSR_EL1 sys_reg(3, 0, 5, 6, 0) #define SYS_TFSRE0_EL1 sys_reg(3, 0, 5, 6, 1) @@ -275,6 +387,8 @@ #define SYS_ICC_IGRPEN0_EL1 sys_reg(3, 0, 12, 12, 6) #define SYS_ICC_IGRPEN1_EL1 sys_reg(3, 0, 12, 12, 7) +#define SYS_ACCDATA_EL1 sys_reg(3, 0, 13, 0, 5) + #define SYS_CNTKCTL_EL1 sys_reg(3, 0, 14, 1, 0) #define SYS_AIDR_EL1 sys_reg(3, 1, 0, 0, 7) @@ -383,8 +497,6 @@ #define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) #define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1) -#define SYS_HDFGRTR_EL2 sys_reg(3, 4, 3, 1, 4) -#define SYS_HDFGWTR_EL2 sys_reg(3, 4, 3, 1, 5) #define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6) #define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0) #define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1) @@ -478,6 +590,158 @@ #define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) +/* AT instructions */ +#define AT_Op0 1 +#define AT_CRn 7 + +#define OP_AT_S1E1R sys_insn(AT_Op0, 0, AT_CRn, 8, 0) +#define OP_AT_S1E1W sys_insn(AT_Op0, 0, AT_CRn, 8, 1) +#define OP_AT_S1E0R sys_insn(AT_Op0, 0, AT_CRn, 8, 2) +#define OP_AT_S1E0W sys_insn(AT_Op0, 0, AT_CRn, 8, 3) +#define OP_AT_S1E1RP sys_insn(AT_Op0, 0, AT_CRn, 9, 0) +#define OP_AT_S1E1WP sys_insn(AT_Op0, 0, AT_CRn, 9, 1) +#define OP_AT_S1E2R sys_insn(AT_Op0, 4, AT_CRn, 8, 0) +#define OP_AT_S1E2W sys_insn(AT_Op0, 4, AT_CRn, 8, 1) +#define OP_AT_S12E1R sys_insn(AT_Op0, 4, AT_CRn, 8, 4) +#define OP_AT_S12E1W sys_insn(AT_Op0, 4, AT_CRn, 8, 5) +#define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6) +#define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) + +/* TLBI instructions */ +#define OP_TLBI_VMALLE1OS sys_insn(1, 0, 8, 1, 0) +#define OP_TLBI_VAE1OS sys_insn(1, 0, 8, 1, 1) +#define OP_TLBI_ASIDE1OS sys_insn(1, 0, 8, 1, 2) +#define OP_TLBI_VAAE1OS sys_insn(1, 0, 8, 1, 3) +#define OP_TLBI_VALE1OS sys_insn(1, 0, 8, 1, 5) +#define OP_TLBI_VAALE1OS sys_insn(1, 0, 8, 1, 7) +#define OP_TLBI_RVAE1IS sys_insn(1, 0, 8, 2, 1) +#define OP_TLBI_RVAAE1IS sys_insn(1, 0, 8, 2, 3) +#define OP_TLBI_RVALE1IS sys_insn(1, 0, 8, 2, 5) +#define OP_TLBI_RVAALE1IS sys_insn(1, 0, 8, 2, 7) +#define OP_TLBI_VMALLE1IS sys_insn(1, 0, 8, 3, 0) +#define OP_TLBI_VAE1IS sys_insn(1, 0, 8, 3, 1) +#define OP_TLBI_ASIDE1IS sys_insn(1, 0, 8, 3, 2) +#define OP_TLBI_VAAE1IS sys_insn(1, 0, 8, 3, 3) +#define OP_TLBI_VALE1IS sys_insn(1, 0, 8, 3, 5) +#define OP_TLBI_VAALE1IS sys_insn(1, 0, 8, 3, 7) +#define OP_TLBI_RVAE1OS sys_insn(1, 0, 8, 5, 1) +#define OP_TLBI_RVAAE1OS sys_insn(1, 0, 8, 5, 3) +#define OP_TLBI_RVALE1OS sys_insn(1, 0, 8, 5, 5) +#define OP_TLBI_RVAALE1OS sys_insn(1, 0, 8, 5, 7) +#define OP_TLBI_RVAE1 sys_insn(1, 0, 8, 6, 1) +#define OP_TLBI_RVAAE1 sys_insn(1, 0, 8, 6, 3) +#define OP_TLBI_RVALE1 sys_insn(1, 0, 8, 6, 5) +#define OP_TLBI_RVAALE1 sys_insn(1, 0, 8, 6, 7) +#define OP_TLBI_VMALLE1 sys_insn(1, 0, 8, 7, 0) +#define OP_TLBI_VAE1 sys_insn(1, 0, 8, 7, 1) +#define OP_TLBI_ASIDE1 sys_insn(1, 0, 8, 7, 2) +#define OP_TLBI_VAAE1 sys_insn(1, 0, 8, 7, 3) +#define OP_TLBI_VALE1 sys_insn(1, 0, 8, 7, 5) +#define OP_TLBI_VAALE1 sys_insn(1, 0, 8, 7, 7) +#define OP_TLBI_VMALLE1OSNXS sys_insn(1, 0, 9, 1, 0) +#define OP_TLBI_VAE1OSNXS sys_insn(1, 0, 9, 1, 1) +#define OP_TLBI_ASIDE1OSNXS sys_insn(1, 0, 9, 1, 2) +#define OP_TLBI_VAAE1OSNXS sys_insn(1, 0, 9, 1, 3) +#define OP_TLBI_VALE1OSNXS sys_insn(1, 0, 9, 1, 5) +#define OP_TLBI_VAALE1OSNXS sys_insn(1, 0, 9, 1, 7) +#define OP_TLBI_RVAE1ISNXS sys_insn(1, 0, 9, 2, 1) +#define OP_TLBI_RVAAE1ISNXS sys_insn(1, 0, 9, 2, 3) +#define OP_TLBI_RVALE1ISNXS sys_insn(1, 0, 9, 2, 5) +#define OP_TLBI_RVAALE1ISNXS sys_insn(1, 0, 9, 2, 7) +#define OP_TLBI_VMALLE1ISNXS sys_insn(1, 0, 9, 3, 0) +#define OP_TLBI_VAE1ISNXS sys_insn(1, 0, 9, 3, 1) +#define OP_TLBI_ASIDE1ISNXS sys_insn(1, 0, 9, 3, 2) +#define OP_TLBI_VAAE1ISNXS sys_insn(1, 0, 9, 3, 3) +#define OP_TLBI_VALE1ISNXS sys_insn(1, 0, 9, 3, 5) +#define OP_TLBI_VAALE1ISNXS sys_insn(1, 0, 9, 3, 7) +#define OP_TLBI_RVAE1OSNXS sys_insn(1, 0, 9, 5, 1) +#define OP_TLBI_RVAAE1OSNXS sys_insn(1, 0, 9, 5, 3) +#define OP_TLBI_RVALE1OSNXS sys_insn(1, 0, 9, 5, 5) +#define OP_TLBI_RVAALE1OSNXS sys_insn(1, 0, 9, 5, 7) +#define OP_TLBI_RVAE1NXS sys_insn(1, 0, 9, 6, 1) +#define OP_TLBI_RVAAE1NXS sys_insn(1, 0, 9, 6, 3) +#define OP_TLBI_RVALE1NXS sys_insn(1, 0, 9, 6, 5) +#define OP_TLBI_RVAALE1NXS sys_insn(1, 0, 9, 6, 7) +#define OP_TLBI_VMALLE1NXS sys_insn(1, 0, 9, 7, 0) +#define OP_TLBI_VAE1NXS sys_insn(1, 0, 9, 7, 1) +#define OP_TLBI_ASIDE1NXS sys_insn(1, 0, 9, 7, 2) +#define OP_TLBI_VAAE1NXS sys_insn(1, 0, 9, 7, 3) +#define OP_TLBI_VALE1NXS sys_insn(1, 0, 9, 7, 5) +#define OP_TLBI_VAALE1NXS sys_insn(1, 0, 9, 7, 7) +#define OP_TLBI_IPAS2E1IS sys_insn(1, 4, 8, 0, 1) +#define OP_TLBI_RIPAS2E1IS sys_insn(1, 4, 8, 0, 2) +#define OP_TLBI_IPAS2LE1IS sys_insn(1, 4, 8, 0, 5) +#define OP_TLBI_RIPAS2LE1IS sys_insn(1, 4, 8, 0, 6) +#define OP_TLBI_ALLE2OS sys_insn(1, 4, 8, 1, 0) +#define OP_TLBI_VAE2OS sys_insn(1, 4, 8, 1, 1) +#define OP_TLBI_ALLE1OS sys_insn(1, 4, 8, 1, 4) +#define OP_TLBI_VALE2OS sys_insn(1, 4, 8, 1, 5) +#define OP_TLBI_VMALLS12E1OS sys_insn(1, 4, 8, 1, 6) +#define OP_TLBI_RVAE2IS sys_insn(1, 4, 8, 2, 1) +#define OP_TLBI_RVALE2IS sys_insn(1, 4, 8, 2, 5) +#define OP_TLBI_ALLE2IS sys_insn(1, 4, 8, 3, 0) +#define OP_TLBI_VAE2IS sys_insn(1, 4, 8, 3, 1) +#define OP_TLBI_ALLE1IS sys_insn(1, 4, 8, 3, 4) +#define OP_TLBI_VALE2IS sys_insn(1, 4, 8, 3, 5) +#define OP_TLBI_VMALLS12E1IS sys_insn(1, 4, 8, 3, 6) +#define OP_TLBI_IPAS2E1OS sys_insn(1, 4, 8, 4, 0) +#define OP_TLBI_IPAS2E1 sys_insn(1, 4, 8, 4, 1) +#define OP_TLBI_RIPAS2E1 sys_insn(1, 4, 8, 4, 2) +#define OP_TLBI_RIPAS2E1OS sys_insn(1, 4, 8, 4, 3) +#define OP_TLBI_IPAS2LE1OS sys_insn(1, 4, 8, 4, 4) +#define OP_TLBI_IPAS2LE1 sys_insn(1, 4, 8, 4, 5) +#define OP_TLBI_RIPAS2LE1 sys_insn(1, 4, 8, 4, 6) +#define OP_TLBI_RIPAS2LE1OS sys_insn(1, 4, 8, 4, 7) +#define OP_TLBI_RVAE2OS sys_insn(1, 4, 8, 5, 1) +#define OP_TLBI_RVALE2OS sys_insn(1, 4, 8, 5, 5) +#define OP_TLBI_RVAE2 sys_insn(1, 4, 8, 6, 1) +#define OP_TLBI_RVALE2 sys_insn(1, 4, 8, 6, 5) +#define OP_TLBI_ALLE2 sys_insn(1, 4, 8, 7, 0) +#define OP_TLBI_VAE2 sys_insn(1, 4, 8, 7, 1) +#define OP_TLBI_ALLE1 sys_insn(1, 4, 8, 7, 4) +#define OP_TLBI_VALE2 sys_insn(1, 4, 8, 7, 5) +#define OP_TLBI_VMALLS12E1 sys_insn(1, 4, 8, 7, 6) +#define OP_TLBI_IPAS2E1ISNXS sys_insn(1, 4, 9, 0, 1) +#define OP_TLBI_RIPAS2E1ISNXS sys_insn(1, 4, 9, 0, 2) +#define OP_TLBI_IPAS2LE1ISNXS sys_insn(1, 4, 9, 0, 5) +#define OP_TLBI_RIPAS2LE1ISNXS sys_insn(1, 4, 9, 0, 6) +#define OP_TLBI_ALLE2OSNXS sys_insn(1, 4, 9, 1, 0) +#define OP_TLBI_VAE2OSNXS sys_insn(1, 4, 9, 1, 1) +#define OP_TLBI_ALLE1OSNXS sys_insn(1, 4, 9, 1, 4) +#define OP_TLBI_VALE2OSNXS sys_insn(1, 4, 9, 1, 5) +#define OP_TLBI_VMALLS12E1OSNXS sys_insn(1, 4, 9, 1, 6) +#define OP_TLBI_RVAE2ISNXS sys_insn(1, 4, 9, 2, 1) +#define OP_TLBI_RVALE2ISNXS sys_insn(1, 4, 9, 2, 5) +#define OP_TLBI_ALLE2ISNXS sys_insn(1, 4, 9, 3, 0) +#define OP_TLBI_VAE2ISNXS sys_insn(1, 4, 9, 3, 1) +#define OP_TLBI_ALLE1ISNXS sys_insn(1, 4, 9, 3, 4) +#define OP_TLBI_VALE2ISNXS sys_insn(1, 4, 9, 3, 5) +#define OP_TLBI_VMALLS12E1ISNXS sys_insn(1, 4, 9, 3, 6) +#define OP_TLBI_IPAS2E1OSNXS sys_insn(1, 4, 9, 4, 0) +#define OP_TLBI_IPAS2E1NXS sys_insn(1, 4, 9, 4, 1) +#define OP_TLBI_RIPAS2E1NXS sys_insn(1, 4, 9, 4, 2) +#define OP_TLBI_RIPAS2E1OSNXS sys_insn(1, 4, 9, 4, 3) +#define OP_TLBI_IPAS2LE1OSNXS sys_insn(1, 4, 9, 4, 4) +#define OP_TLBI_IPAS2LE1NXS sys_insn(1, 4, 9, 4, 5) +#define OP_TLBI_RIPAS2LE1NXS sys_insn(1, 4, 9, 4, 6) +#define OP_TLBI_RIPAS2LE1OSNXS sys_insn(1, 4, 9, 4, 7) +#define OP_TLBI_RVAE2OSNXS sys_insn(1, 4, 9, 5, 1) +#define OP_TLBI_RVALE2OSNXS sys_insn(1, 4, 9, 5, 5) +#define OP_TLBI_RVAE2NXS sys_insn(1, 4, 9, 6, 1) +#define OP_TLBI_RVALE2NXS sys_insn(1, 4, 9, 6, 5) +#define OP_TLBI_ALLE2NXS sys_insn(1, 4, 9, 7, 0) +#define OP_TLBI_VAE2NXS sys_insn(1, 4, 9, 7, 1) +#define OP_TLBI_ALLE1NXS sys_insn(1, 4, 9, 7, 4) +#define OP_TLBI_VALE2NXS sys_insn(1, 4, 9, 7, 5) +#define OP_TLBI_VMALLS12E1NXS sys_insn(1, 4, 9, 7, 6) + +/* Misc instructions */ +#define OP_BRB_IALL sys_insn(1, 1, 7, 2, 4) +#define OP_BRB_INJ sys_insn(1, 1, 7, 2, 5) +#define OP_CFP_RCTX sys_insn(1, 3, 7, 3, 4) +#define OP_DVP_RCTX sys_insn(1, 3, 7, 3, 5) +#define OP_CPP_RCTX sys_insn(1, 3, 7, 3, 7) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_ENTP2 (BIT(60)) #define SCTLR_ELx_DSSBS (BIT(44)) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 55b50e1d4a84..b149cf9f91bc 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -335,14 +335,77 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) */ #define MAX_TLBI_OPS PTRS_PER_PTE +/* + * __flush_tlb_range_op - Perform TLBI operation upon a range + * + * @op: TLBI instruction that operates on a range (has 'r' prefix) + * @start: The start address of the range + * @pages: Range as the number of pages from 'start' + * @stride: Flush granularity + * @asid: The ASID of the task (0 for IPA instructions) + * @tlb_level: Translation Table level hint, if known + * @tlbi_user: If 'true', call an additional __tlbi_user() + * (typically for user ASIDs). 'flase' for IPA instructions + * + * When the CPU does not support TLB range operations, flush the TLB + * entries one by one at the granularity of 'stride'. If the TLB + * range ops are supported, then: + * + * 1. If 'pages' is odd, flush the first page through non-range + * operations; + * + * 2. For remaining pages: the minimum range granularity is decided + * by 'scale', so multiple range TLBI operations may be required. + * Start from scale = 0, flush the corresponding number of pages + * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it + * until no pages left. + * + * Note that certain ranges can be represented by either num = 31 and + * scale or num = 0 and scale + 1. The loop below favours the latter + * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. + */ +#define __flush_tlb_range_op(op, start, pages, stride, \ + asid, tlb_level, tlbi_user) \ +do { \ + int num = 0; \ + int scale = 0; \ + unsigned long addr; \ + \ + while (pages > 0) { \ + if (!system_supports_tlb_range() || \ + pages % 2 == 1) { \ + addr = __TLBI_VADDR(start, asid); \ + __tlbi_level(op, addr, tlb_level); \ + if (tlbi_user) \ + __tlbi_user_level(op, addr, tlb_level); \ + start += stride; \ + pages -= stride >> PAGE_SHIFT; \ + continue; \ + } \ + \ + num = __TLBI_RANGE_NUM(pages, scale); \ + if (num >= 0) { \ + addr = __TLBI_VADDR_RANGE(start, asid, scale, \ + num, tlb_level); \ + __tlbi(r##op, addr); \ + if (tlbi_user) \ + __tlbi_user(r##op, addr); \ + start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; \ + pages -= __TLBI_RANGE_PAGES(num, scale); \ + } \ + scale++; \ + } \ +} while (0) + +#define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \ + __flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false) + static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) { - int num = 0; - int scale = 0; - unsigned long asid, addr, pages; + unsigned long asid, pages; start = round_down(start, stride); end = round_up(end, stride); @@ -364,56 +427,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); asid = ASID(vma->vm_mm); - /* - * When the CPU does not support TLB range operations, flush the TLB - * entries one by one at the granularity of 'stride'. If the TLB - * range ops are supported, then: - * - * 1. If 'pages' is odd, flush the first page through non-range - * operations; - * - * 2. For remaining pages: the minimum range granularity is decided - * by 'scale', so multiple range TLBI operations may be required. - * Start from scale = 0, flush the corresponding number of pages - * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it - * until no pages left. - * - * Note that certain ranges can be represented by either num = 31 and - * scale or num = 0 and scale + 1. The loop below favours the latter - * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. - */ - while (pages > 0) { - if (!system_supports_tlb_range() || - pages % 2 == 1) { - addr = __TLBI_VADDR(start, asid); - if (last_level) { - __tlbi_level(vale1is, addr, tlb_level); - __tlbi_user_level(vale1is, addr, tlb_level); - } else { - __tlbi_level(vae1is, addr, tlb_level); - __tlbi_user_level(vae1is, addr, tlb_level); - } - start += stride; - pages -= stride >> PAGE_SHIFT; - continue; - } + if (last_level) + __flush_tlb_range_op(vale1is, start, pages, stride, asid, tlb_level, true); + else + __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true); - num = __TLBI_RANGE_NUM(pages, scale); - if (num >= 0) { - addr = __TLBI_VADDR_RANGE(start, asid, scale, - num, tlb_level); - if (last_level) { - __tlbi(rvale1is, addr); - __tlbi_user(rvale1is, addr); - } else { - __tlbi(rvae1is, addr); - __tlbi_user(rvae1is, addr); - } - start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; - pages -= __TLBI_RANGE_PAGES(num, scale); - } - scale++; - } dsb(ish); mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 01631af7d156..a47b29cc1beb 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2628,6 +2628,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64ISAR1_EL1, LRCPC, IMP) }, + { + .desc = "Fine Grained Traps", + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .capability = ARM64_HAS_FGT, + .matches = has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR0_EL1, FGT, IMP) + }, #ifdef CONFIG_ARM64_SME { .desc = "Scalable Matrix Extension", diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index aee12c75b738..3addc09f8746 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -262,9 +262,9 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) if (!len) return; - len = strscpy(buf, cmdline, ARRAY_SIZE(buf)); - if (len == -E2BIG) - len = ARRAY_SIZE(buf) - 1; + len = min(len, ARRAY_SIZE(buf) - 1); + memcpy(buf, cmdline, len); + buf[len] = '\0'; if (strcmp(buf, "--") == 0) return; diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index f531da6b362e..83c1e09be42e 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -25,7 +25,6 @@ menuconfig KVM select MMU_NOTIFIER select PREEMPT_NOTIFIERS select HAVE_KVM_CPU_RELAX_INTERCEPT - select HAVE_KVM_ARCH_TLB_FLUSH_ALL select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT select KVM_XFER_TO_GUEST_WORK @@ -43,6 +42,7 @@ menuconfig KVM select SCHED_INFO select GUEST_PERF_EVENTS if PERF_EVENTS select INTERVAL_TREE + select XARRAY_MULTI help Support hosting virtualized guest machines. diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index d1cb298a58a0..4866b3f7b4ea 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -365,7 +366,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) #endif /* Force users to call KVM_ARM_VCPU_INIT */ - vcpu->arch.target = -1; + vcpu_clear_flag(vcpu, VCPU_INITIALIZED); bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES); vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO; @@ -462,7 +463,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) vcpu_ptrauth_disable(vcpu); kvm_arch_vcpu_load_debug_state_flags(vcpu); - if (!cpumask_test_cpu(smp_processor_id(), vcpu->kvm->arch.supported_cpus)) + if (!cpumask_test_cpu(cpu, vcpu->kvm->arch.supported_cpus)) vcpu_set_on_unsupported_cpu(vcpu); } @@ -574,7 +575,7 @@ unsigned long kvm_arch_vcpu_get_ip(struct kvm_vcpu *vcpu) static int kvm_vcpu_initialized(struct kvm_vcpu *vcpu) { - return vcpu->arch.target >= 0; + return vcpu_get_flag(vcpu, VCPU_INITIALIZED); } /* @@ -803,6 +804,9 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu) kvm_pmu_handle_pmcr(vcpu, __vcpu_sys_reg(vcpu, PMCR_EL0)); + if (kvm_check_request(KVM_REQ_RESYNC_PMU_EL0, vcpu)) + kvm_vcpu_pmu_restore_guest(vcpu); + if (kvm_check_request(KVM_REQ_SUSPEND, vcpu)) return kvm_vcpu_suspend(vcpu); @@ -818,6 +822,9 @@ static bool vcpu_mode_is_bad_32bit(struct kvm_vcpu *vcpu) if (likely(!vcpu_mode_is_32bit(vcpu))) return false; + if (vcpu_has_nv(vcpu)) + return true; + return !kvm_supports_32bit_el0(); } @@ -1058,7 +1065,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) * invalid. The VMM can try and fix it by issuing a * KVM_ARM_VCPU_INIT if it really wants to. */ - vcpu->arch.target = -1; + vcpu_clear_flag(vcpu, VCPU_INITIALIZED); ret = ARM_EXCEPTION_IL; } @@ -1219,8 +1226,7 @@ static bool kvm_vcpu_init_changed(struct kvm_vcpu *vcpu, { unsigned long features = init->features[0]; - return !bitmap_equal(vcpu->arch.features, &features, KVM_VCPU_MAX_FEATURES) || - vcpu->arch.target != init->target; + return !bitmap_equal(vcpu->arch.features, &features, KVM_VCPU_MAX_FEATURES); } static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, @@ -1236,20 +1242,18 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu, !bitmap_equal(kvm->arch.vcpu_features, &features, KVM_VCPU_MAX_FEATURES)) goto out_unlock; - vcpu->arch.target = init->target; bitmap_copy(vcpu->arch.features, &features, KVM_VCPU_MAX_FEATURES); /* Now we know what it is, we can reset it. */ ret = kvm_reset_vcpu(vcpu); if (ret) { - vcpu->arch.target = -1; bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES); goto out_unlock; } bitmap_copy(kvm->arch.vcpu_features, &features, KVM_VCPU_MAX_FEATURES); set_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags); - + vcpu_set_flag(vcpu, VCPU_INITIALIZED); out_unlock: mutex_unlock(&kvm->arch.config_lock); return ret; @@ -1260,14 +1264,15 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu, { int ret; - if (init->target != kvm_target_cpu()) + if (init->target != KVM_ARM_TARGET_GENERIC_V8 && + init->target != kvm_target_cpu()) return -EINVAL; ret = kvm_vcpu_init_check_features(vcpu, init); if (ret) return ret; - if (vcpu->arch.target == -1) + if (!kvm_vcpu_initialized(vcpu)) return __kvm_vcpu_set_target(vcpu, init); if (kvm_vcpu_init_changed(vcpu, init)) @@ -1532,12 +1537,6 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) } -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot) -{ - kvm_flush_remote_tlbs(kvm); -} - static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm, struct kvm_arm_device_addr *dev_addr) { @@ -1595,9 +1594,9 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) return kvm_vm_ioctl_set_device_addr(kvm, &dev_addr); } case KVM_ARM_PREFERRED_TARGET: { - struct kvm_vcpu_init init; - - kvm_vcpu_preferred_target(&init); + struct kvm_vcpu_init init = { + .target = KVM_ARM_TARGET_GENERIC_V8, + }; if (copy_to_user(argp, &init, sizeof(init))) return -EFAULT; @@ -2276,30 +2275,8 @@ static int __init init_hyp_mode(void) for_each_possible_cpu(cpu) { struct kvm_nvhe_init_params *params = per_cpu_ptr_nvhe_sym(kvm_init_params, cpu); char *stack_page = (char *)per_cpu(kvm_arm_hyp_stack_page, cpu); - unsigned long hyp_addr; - /* - * Allocate a contiguous HYP private VA range for the stack - * and guard page. The allocation is also aligned based on - * the order of its size. - */ - err = hyp_alloc_private_va_range(PAGE_SIZE * 2, &hyp_addr); - if (err) { - kvm_err("Cannot allocate hyp stack guard page\n"); - goto out_err; - } - - /* - * Since the stack grows downwards, map the stack to the page - * at the higher address and leave the lower guard page - * unbacked. - * - * Any valid stack address now has the PAGE_SHIFT bit as 1 - * and addresses corresponding to the guard page have the - * PAGE_SHIFT bit as 0 - this is used for overflow detection. - */ - err = __create_hyp_mappings(hyp_addr + PAGE_SIZE, PAGE_SIZE, - __pa(stack_page), PAGE_HYP); + err = create_hyp_stack(__pa(stack_page), ¶ms->stack_hyp_va); if (err) { kvm_err("Cannot map hyp stack\n"); goto out_err; @@ -2312,8 +2289,6 @@ static int __init init_hyp_mode(void) * has been mapped in the flexible private VA space. */ params->stack_pa = __pa(stack_page); - - params->stack_hyp_va = hyp_addr + (2 * PAGE_SIZE); } for_each_possible_cpu(cpu) { diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index b96662029fb1..9ced1bf0c2b7 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -14,6 +14,1858 @@ #include "trace.h" +enum trap_behaviour { + BEHAVE_HANDLE_LOCALLY = 0, + BEHAVE_FORWARD_READ = BIT(0), + BEHAVE_FORWARD_WRITE = BIT(1), + BEHAVE_FORWARD_ANY = BEHAVE_FORWARD_READ | BEHAVE_FORWARD_WRITE, +}; + +struct trap_bits { + const enum vcpu_sysreg index; + const enum trap_behaviour behaviour; + const u64 value; + const u64 mask; +}; + +/* Coarse Grained Trap definitions */ +enum cgt_group_id { + /* Indicates no coarse trap control */ + __RESERVED__, + + /* + * The first batch of IDs denote coarse trapping that are used + * on their own instead of being part of a combination of + * trap controls. + */ + CGT_HCR_TID1, + CGT_HCR_TID2, + CGT_HCR_TID3, + CGT_HCR_IMO, + CGT_HCR_FMO, + CGT_HCR_TIDCP, + CGT_HCR_TACR, + CGT_HCR_TSW, + CGT_HCR_TPC, + CGT_HCR_TPU, + CGT_HCR_TTLB, + CGT_HCR_TVM, + CGT_HCR_TDZ, + CGT_HCR_TRVM, + CGT_HCR_TLOR, + CGT_HCR_TERR, + CGT_HCR_APK, + CGT_HCR_NV, + CGT_HCR_NV_nNV2, + CGT_HCR_NV1_nNV2, + CGT_HCR_AT, + CGT_HCR_nFIEN, + CGT_HCR_TID4, + CGT_HCR_TICAB, + CGT_HCR_TOCU, + CGT_HCR_ENSCXT, + CGT_HCR_TTLBIS, + CGT_HCR_TTLBOS, + + CGT_MDCR_TPMCR, + CGT_MDCR_TPM, + CGT_MDCR_TDE, + CGT_MDCR_TDA, + CGT_MDCR_TDOSA, + CGT_MDCR_TDRA, + CGT_MDCR_E2PB, + CGT_MDCR_TPMS, + CGT_MDCR_TTRF, + CGT_MDCR_E2TB, + CGT_MDCR_TDCC, + + /* + * Anything after this point is a combination of coarse trap + * controls, which must all be evaluated to decide what to do. + */ + __MULTIPLE_CONTROL_BITS__, + CGT_HCR_IMO_FMO = __MULTIPLE_CONTROL_BITS__, + CGT_HCR_TID2_TID4, + CGT_HCR_TTLB_TTLBIS, + CGT_HCR_TTLB_TTLBOS, + CGT_HCR_TVM_TRVM, + CGT_HCR_TPU_TICAB, + CGT_HCR_TPU_TOCU, + CGT_HCR_NV1_nNV2_ENSCXT, + CGT_MDCR_TPM_TPMCR, + CGT_MDCR_TDE_TDA, + CGT_MDCR_TDE_TDOSA, + CGT_MDCR_TDE_TDRA, + CGT_MDCR_TDCC_TDE_TDA, + + /* + * Anything after this point requires a callback evaluating a + * complex trap condition. Ugly stuff. + */ + __COMPLEX_CONDITIONS__, + CGT_CNTHCTL_EL1PCTEN = __COMPLEX_CONDITIONS__, + CGT_CNTHCTL_EL1PTEN, + + /* Must be last */ + __NR_CGT_GROUP_IDS__ +}; + +static const struct trap_bits coarse_trap_bits[] = { + [CGT_HCR_TID1] = { + .index = HCR_EL2, + .value = HCR_TID1, + .mask = HCR_TID1, + .behaviour = BEHAVE_FORWARD_READ, + }, + [CGT_HCR_TID2] = { + .index = HCR_EL2, + .value = HCR_TID2, + .mask = HCR_TID2, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TID3] = { + .index = HCR_EL2, + .value = HCR_TID3, + .mask = HCR_TID3, + .behaviour = BEHAVE_FORWARD_READ, + }, + [CGT_HCR_IMO] = { + .index = HCR_EL2, + .value = HCR_IMO, + .mask = HCR_IMO, + .behaviour = BEHAVE_FORWARD_WRITE, + }, + [CGT_HCR_FMO] = { + .index = HCR_EL2, + .value = HCR_FMO, + .mask = HCR_FMO, + .behaviour = BEHAVE_FORWARD_WRITE, + }, + [CGT_HCR_TIDCP] = { + .index = HCR_EL2, + .value = HCR_TIDCP, + .mask = HCR_TIDCP, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TACR] = { + .index = HCR_EL2, + .value = HCR_TACR, + .mask = HCR_TACR, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TSW] = { + .index = HCR_EL2, + .value = HCR_TSW, + .mask = HCR_TSW, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TPC] = { /* Also called TCPC when FEAT_DPB is implemented */ + .index = HCR_EL2, + .value = HCR_TPC, + .mask = HCR_TPC, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TPU] = { + .index = HCR_EL2, + .value = HCR_TPU, + .mask = HCR_TPU, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TTLB] = { + .index = HCR_EL2, + .value = HCR_TTLB, + .mask = HCR_TTLB, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TVM] = { + .index = HCR_EL2, + .value = HCR_TVM, + .mask = HCR_TVM, + .behaviour = BEHAVE_FORWARD_WRITE, + }, + [CGT_HCR_TDZ] = { + .index = HCR_EL2, + .value = HCR_TDZ, + .mask = HCR_TDZ, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TRVM] = { + .index = HCR_EL2, + .value = HCR_TRVM, + .mask = HCR_TRVM, + .behaviour = BEHAVE_FORWARD_READ, + }, + [CGT_HCR_TLOR] = { + .index = HCR_EL2, + .value = HCR_TLOR, + .mask = HCR_TLOR, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TERR] = { + .index = HCR_EL2, + .value = HCR_TERR, + .mask = HCR_TERR, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_APK] = { + .index = HCR_EL2, + .value = 0, + .mask = HCR_APK, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_NV] = { + .index = HCR_EL2, + .value = HCR_NV, + .mask = HCR_NV, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_NV_nNV2] = { + .index = HCR_EL2, + .value = HCR_NV, + .mask = HCR_NV | HCR_NV2, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_NV1_nNV2] = { + .index = HCR_EL2, + .value = HCR_NV | HCR_NV1, + .mask = HCR_NV | HCR_NV1 | HCR_NV2, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_AT] = { + .index = HCR_EL2, + .value = HCR_AT, + .mask = HCR_AT, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_nFIEN] = { + .index = HCR_EL2, + .value = 0, + .mask = HCR_FIEN, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TID4] = { + .index = HCR_EL2, + .value = HCR_TID4, + .mask = HCR_TID4, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TICAB] = { + .index = HCR_EL2, + .value = HCR_TICAB, + .mask = HCR_TICAB, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TOCU] = { + .index = HCR_EL2, + .value = HCR_TOCU, + .mask = HCR_TOCU, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_ENSCXT] = { + .index = HCR_EL2, + .value = 0, + .mask = HCR_ENSCXT, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TTLBIS] = { + .index = HCR_EL2, + .value = HCR_TTLBIS, + .mask = HCR_TTLBIS, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_HCR_TTLBOS] = { + .index = HCR_EL2, + .value = HCR_TTLBOS, + .mask = HCR_TTLBOS, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TPMCR] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TPMCR, + .mask = MDCR_EL2_TPMCR, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TPM] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TPM, + .mask = MDCR_EL2_TPM, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TDE] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TDE, + .mask = MDCR_EL2_TDE, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TDA] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TDA, + .mask = MDCR_EL2_TDA, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TDOSA] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TDOSA, + .mask = MDCR_EL2_TDOSA, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TDRA] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TDRA, + .mask = MDCR_EL2_TDRA, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_E2PB] = { + .index = MDCR_EL2, + .value = 0, + .mask = BIT(MDCR_EL2_E2PB_SHIFT), + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TPMS] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TPMS, + .mask = MDCR_EL2_TPMS, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TTRF] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TTRF, + .mask = MDCR_EL2_TTRF, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_E2TB] = { + .index = MDCR_EL2, + .value = 0, + .mask = BIT(MDCR_EL2_E2TB_SHIFT), + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_MDCR_TDCC] = { + .index = MDCR_EL2, + .value = MDCR_EL2_TDCC, + .mask = MDCR_EL2_TDCC, + .behaviour = BEHAVE_FORWARD_ANY, + }, +}; + +#define MCB(id, ...) \ + [id - __MULTIPLE_CONTROL_BITS__] = \ + (const enum cgt_group_id[]){ \ + __VA_ARGS__, __RESERVED__ \ + } + +static const enum cgt_group_id *coarse_control_combo[] = { + MCB(CGT_HCR_IMO_FMO, CGT_HCR_IMO, CGT_HCR_FMO), + MCB(CGT_HCR_TID2_TID4, CGT_HCR_TID2, CGT_HCR_TID4), + MCB(CGT_HCR_TTLB_TTLBIS, CGT_HCR_TTLB, CGT_HCR_TTLBIS), + MCB(CGT_HCR_TTLB_TTLBOS, CGT_HCR_TTLB, CGT_HCR_TTLBOS), + MCB(CGT_HCR_TVM_TRVM, CGT_HCR_TVM, CGT_HCR_TRVM), + MCB(CGT_HCR_TPU_TICAB, CGT_HCR_TPU, CGT_HCR_TICAB), + MCB(CGT_HCR_TPU_TOCU, CGT_HCR_TPU, CGT_HCR_TOCU), + MCB(CGT_HCR_NV1_nNV2_ENSCXT, CGT_HCR_NV1_nNV2, CGT_HCR_ENSCXT), + MCB(CGT_MDCR_TPM_TPMCR, CGT_MDCR_TPM, CGT_MDCR_TPMCR), + MCB(CGT_MDCR_TDE_TDA, CGT_MDCR_TDE, CGT_MDCR_TDA), + MCB(CGT_MDCR_TDE_TDOSA, CGT_MDCR_TDE, CGT_MDCR_TDOSA), + MCB(CGT_MDCR_TDE_TDRA, CGT_MDCR_TDE, CGT_MDCR_TDRA), + MCB(CGT_MDCR_TDCC_TDE_TDA, CGT_MDCR_TDCC, CGT_MDCR_TDE, CGT_MDCR_TDA), +}; + +typedef enum trap_behaviour (*complex_condition_check)(struct kvm_vcpu *); + +/* + * Warning, maximum confusion ahead. + * + * When E2H=0, CNTHCTL_EL2[1:0] are defined as EL1PCEN:EL1PCTEN + * When E2H=1, CNTHCTL_EL2[11:10] are defined as EL1PTEN:EL1PCTEN + * + * Note the single letter difference? Yet, the bits have the same + * function despite a different layout and a different name. + * + * We don't try to reconcile this mess. We just use the E2H=0 bits + * to generate something that is in the E2H=1 format, and live with + * it. You're welcome. + */ +static u64 get_sanitized_cnthctl(struct kvm_vcpu *vcpu) +{ + u64 val = __vcpu_sys_reg(vcpu, CNTHCTL_EL2); + + if (!vcpu_el2_e2h_is_set(vcpu)) + val = (val & (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN)) << 10; + + return val & ((CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN) << 10); +} + +static enum trap_behaviour check_cnthctl_el1pcten(struct kvm_vcpu *vcpu) +{ + if (get_sanitized_cnthctl(vcpu) & (CNTHCTL_EL1PCTEN << 10)) + return BEHAVE_HANDLE_LOCALLY; + + return BEHAVE_FORWARD_ANY; +} + +static enum trap_behaviour check_cnthctl_el1pten(struct kvm_vcpu *vcpu) +{ + if (get_sanitized_cnthctl(vcpu) & (CNTHCTL_EL1PCEN << 10)) + return BEHAVE_HANDLE_LOCALLY; + + return BEHAVE_FORWARD_ANY; +} + +#define CCC(id, fn) \ + [id - __COMPLEX_CONDITIONS__] = fn + +static const complex_condition_check ccc[] = { + CCC(CGT_CNTHCTL_EL1PCTEN, check_cnthctl_el1pcten), + CCC(CGT_CNTHCTL_EL1PTEN, check_cnthctl_el1pten), +}; + +/* + * Bit assignment for the trap controls. We use a 64bit word with the + * following layout for each trapped sysreg: + * + * [9:0] enum cgt_group_id (10 bits) + * [13:10] enum fgt_group_id (4 bits) + * [19:14] bit number in the FGT register (6 bits) + * [20] trap polarity (1 bit) + * [25:21] FG filter (5 bits) + * [62:26] Unused (37 bits) + * [63] RES0 - Must be zero, as lost on insertion in the xarray + */ +#define TC_CGT_BITS 10 +#define TC_FGT_BITS 4 +#define TC_FGF_BITS 5 + +union trap_config { + u64 val; + struct { + unsigned long cgt:TC_CGT_BITS; /* Coarse Grained Trap id */ + unsigned long fgt:TC_FGT_BITS; /* Fine Grained Trap id */ + unsigned long bit:6; /* Bit number */ + unsigned long pol:1; /* Polarity */ + unsigned long fgf:TC_FGF_BITS; /* Fine Grained Filter */ + unsigned long unused:37; /* Unused, should be zero */ + unsigned long mbz:1; /* Must Be Zero */ + }; +}; + +struct encoding_to_trap_config { + const u32 encoding; + const u32 end; + const union trap_config tc; + const unsigned int line; +}; + +#define SR_RANGE_TRAP(sr_start, sr_end, trap_id) \ + { \ + .encoding = sr_start, \ + .end = sr_end, \ + .tc = { \ + .cgt = trap_id, \ + }, \ + .line = __LINE__, \ + } + +#define SR_TRAP(sr, trap_id) SR_RANGE_TRAP(sr, sr, trap_id) + +/* + * Map encoding to trap bits for exception reported with EC=0x18. + * These must only be evaluated when running a nested hypervisor, but + * that the current context is not a hypervisor context. When the + * trapped access matches one of the trap controls, the exception is + * re-injected in the nested hypervisor. + */ +static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = { + SR_TRAP(SYS_REVIDR_EL1, CGT_HCR_TID1), + SR_TRAP(SYS_AIDR_EL1, CGT_HCR_TID1), + SR_TRAP(SYS_SMIDR_EL1, CGT_HCR_TID1), + SR_TRAP(SYS_CTR_EL0, CGT_HCR_TID2), + SR_TRAP(SYS_CCSIDR_EL1, CGT_HCR_TID2_TID4), + SR_TRAP(SYS_CCSIDR2_EL1, CGT_HCR_TID2_TID4), + SR_TRAP(SYS_CLIDR_EL1, CGT_HCR_TID2_TID4), + SR_TRAP(SYS_CSSELR_EL1, CGT_HCR_TID2_TID4), + SR_RANGE_TRAP(SYS_ID_PFR0_EL1, + sys_reg(3, 0, 0, 7, 7), CGT_HCR_TID3), + SR_TRAP(SYS_ICC_SGI0R_EL1, CGT_HCR_IMO_FMO), + SR_TRAP(SYS_ICC_ASGI1R_EL1, CGT_HCR_IMO_FMO), + SR_TRAP(SYS_ICC_SGI1R_EL1, CGT_HCR_IMO_FMO), + SR_RANGE_TRAP(sys_reg(3, 0, 11, 0, 0), + sys_reg(3, 0, 11, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 1, 11, 0, 0), + sys_reg(3, 1, 11, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 2, 11, 0, 0), + sys_reg(3, 2, 11, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 3, 11, 0, 0), + sys_reg(3, 3, 11, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 4, 11, 0, 0), + sys_reg(3, 4, 11, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 5, 11, 0, 0), + sys_reg(3, 5, 11, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 6, 11, 0, 0), + sys_reg(3, 6, 11, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 7, 11, 0, 0), + sys_reg(3, 7, 11, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 0, 15, 0, 0), + sys_reg(3, 0, 15, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 1, 15, 0, 0), + sys_reg(3, 1, 15, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 2, 15, 0, 0), + sys_reg(3, 2, 15, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 3, 15, 0, 0), + sys_reg(3, 3, 15, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 4, 15, 0, 0), + sys_reg(3, 4, 15, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 5, 15, 0, 0), + sys_reg(3, 5, 15, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 6, 15, 0, 0), + sys_reg(3, 6, 15, 15, 7), CGT_HCR_TIDCP), + SR_RANGE_TRAP(sys_reg(3, 7, 15, 0, 0), + sys_reg(3, 7, 15, 15, 7), CGT_HCR_TIDCP), + SR_TRAP(SYS_ACTLR_EL1, CGT_HCR_TACR), + SR_TRAP(SYS_DC_ISW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_CSW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_CISW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_IGSW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_IGDSW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_CGSW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_CGDSW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_CIGSW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_CIGDSW, CGT_HCR_TSW), + SR_TRAP(SYS_DC_CIVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CVAP, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CVADP, CGT_HCR_TPC), + SR_TRAP(SYS_DC_IVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CIGVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CIGDVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_IGVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_IGDVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CGVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CGDVAC, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CGVAP, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CGDVAP, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CGVADP, CGT_HCR_TPC), + SR_TRAP(SYS_DC_CGDVADP, CGT_HCR_TPC), + SR_TRAP(SYS_IC_IVAU, CGT_HCR_TPU_TOCU), + SR_TRAP(SYS_IC_IALLU, CGT_HCR_TPU_TOCU), + SR_TRAP(SYS_IC_IALLUIS, CGT_HCR_TPU_TICAB), + SR_TRAP(SYS_DC_CVAU, CGT_HCR_TPU_TOCU), + SR_TRAP(OP_TLBI_RVAE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_RVAAE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_RVALE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_RVAALE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VMALLE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VAE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_ASIDE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VAAE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VALE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VAALE1, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_RVAE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_RVAAE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_RVALE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_RVAALE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VMALLE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VAE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_ASIDE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VAAE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VALE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_VAALE1NXS, CGT_HCR_TTLB), + SR_TRAP(OP_TLBI_RVAE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_RVAAE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_RVALE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_RVAALE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VMALLE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VAE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_ASIDE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VAAE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VALE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VAALE1IS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_RVAE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_RVAAE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_RVALE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_RVAALE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VMALLE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VAE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_ASIDE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VAAE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VALE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VAALE1ISNXS, CGT_HCR_TTLB_TTLBIS), + SR_TRAP(OP_TLBI_VMALLE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VAE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_ASIDE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VAAE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VALE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VAALE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_RVAE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_RVAAE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_RVALE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_RVAALE1OS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VMALLE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VAE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_ASIDE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VAAE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VALE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_VAALE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_RVAE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_RVAAE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_RVALE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(OP_TLBI_RVAALE1OSNXS, CGT_HCR_TTLB_TTLBOS), + SR_TRAP(SYS_SCTLR_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_TTBR0_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_TTBR1_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_TCR_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_ESR_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_FAR_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_AFSR0_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_AFSR1_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_MAIR_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_AMAIR_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_CONTEXTIDR_EL1, CGT_HCR_TVM_TRVM), + SR_TRAP(SYS_DC_ZVA, CGT_HCR_TDZ), + SR_TRAP(SYS_DC_GVA, CGT_HCR_TDZ), + SR_TRAP(SYS_DC_GZVA, CGT_HCR_TDZ), + SR_TRAP(SYS_LORSA_EL1, CGT_HCR_TLOR), + SR_TRAP(SYS_LOREA_EL1, CGT_HCR_TLOR), + SR_TRAP(SYS_LORN_EL1, CGT_HCR_TLOR), + SR_TRAP(SYS_LORC_EL1, CGT_HCR_TLOR), + SR_TRAP(SYS_LORID_EL1, CGT_HCR_TLOR), + SR_TRAP(SYS_ERRIDR_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERRSELR_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERXADDR_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERXCTLR_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERXFR_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERXMISC0_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERXMISC1_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERXMISC2_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERXMISC3_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_ERXSTATUS_EL1, CGT_HCR_TERR), + SR_TRAP(SYS_APIAKEYLO_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APIAKEYHI_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APIBKEYLO_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APIBKEYHI_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APDAKEYLO_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APDAKEYHI_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APDBKEYLO_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APDBKEYHI_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APGAKEYLO_EL1, CGT_HCR_APK), + SR_TRAP(SYS_APGAKEYHI_EL1, CGT_HCR_APK), + /* All _EL2 registers */ + SR_RANGE_TRAP(sys_reg(3, 4, 0, 0, 0), + sys_reg(3, 4, 3, 15, 7), CGT_HCR_NV), + /* Skip the SP_EL1 encoding... */ + SR_TRAP(SYS_SPSR_EL2, CGT_HCR_NV), + SR_TRAP(SYS_ELR_EL2, CGT_HCR_NV), + SR_RANGE_TRAP(sys_reg(3, 4, 4, 1, 1), + sys_reg(3, 4, 10, 15, 7), CGT_HCR_NV), + SR_RANGE_TRAP(sys_reg(3, 4, 12, 0, 0), + sys_reg(3, 4, 14, 15, 7), CGT_HCR_NV), + /* All _EL02, _EL12 registers */ + SR_RANGE_TRAP(sys_reg(3, 5, 0, 0, 0), + sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV), + SR_RANGE_TRAP(sys_reg(3, 5, 12, 0, 0), + sys_reg(3, 5, 14, 15, 7), CGT_HCR_NV), + SR_TRAP(OP_AT_S1E2R, CGT_HCR_NV), + SR_TRAP(OP_AT_S1E2W, CGT_HCR_NV), + SR_TRAP(OP_AT_S12E1R, CGT_HCR_NV), + SR_TRAP(OP_AT_S12E1W, CGT_HCR_NV), + SR_TRAP(OP_AT_S12E0R, CGT_HCR_NV), + SR_TRAP(OP_AT_S12E0W, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2E1, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2E1, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2LE1, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2LE1, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVAE2, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVALE2, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE2, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VAE2, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE1, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VALE2, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VMALLS12E1, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2E1NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2E1NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2LE1NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2LE1NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVAE2NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVALE2NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE2NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VAE2NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE1NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VALE2NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VMALLS12E1NXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2E1IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2E1IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2LE1IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2LE1IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVAE2IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVALE2IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE2IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VAE2IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE1IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VALE2IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VMALLS12E1IS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2E1ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2E1ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2LE1ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2LE1ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVAE2ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVALE2ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE2ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VAE2ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE1ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VALE2ISNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VMALLS12E1ISNXS,CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE2OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VAE2OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE1OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VALE2OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VMALLS12E1OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2E1OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2E1OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2LE1OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2LE1OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVAE2OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVALE2OS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE2OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VAE2OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_ALLE1OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VALE2OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_VMALLS12E1OSNXS,CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2E1OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2E1OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_IPAS2LE1OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RIPAS2LE1OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVAE2OSNXS, CGT_HCR_NV), + SR_TRAP(OP_TLBI_RVALE2OSNXS, CGT_HCR_NV), + SR_TRAP(OP_CPP_RCTX, CGT_HCR_NV), + SR_TRAP(OP_DVP_RCTX, CGT_HCR_NV), + SR_TRAP(OP_CFP_RCTX, CGT_HCR_NV), + SR_TRAP(SYS_SP_EL1, CGT_HCR_NV_nNV2), + SR_TRAP(SYS_VBAR_EL1, CGT_HCR_NV1_nNV2), + SR_TRAP(SYS_ELR_EL1, CGT_HCR_NV1_nNV2), + SR_TRAP(SYS_SPSR_EL1, CGT_HCR_NV1_nNV2), + SR_TRAP(SYS_SCXTNUM_EL1, CGT_HCR_NV1_nNV2_ENSCXT), + SR_TRAP(SYS_SCXTNUM_EL0, CGT_HCR_ENSCXT), + SR_TRAP(OP_AT_S1E1R, CGT_HCR_AT), + SR_TRAP(OP_AT_S1E1W, CGT_HCR_AT), + SR_TRAP(OP_AT_S1E0R, CGT_HCR_AT), + SR_TRAP(OP_AT_S1E0W, CGT_HCR_AT), + SR_TRAP(OP_AT_S1E1RP, CGT_HCR_AT), + SR_TRAP(OP_AT_S1E1WP, CGT_HCR_AT), + SR_TRAP(SYS_ERXPFGF_EL1, CGT_HCR_nFIEN), + SR_TRAP(SYS_ERXPFGCTL_EL1, CGT_HCR_nFIEN), + SR_TRAP(SYS_ERXPFGCDN_EL1, CGT_HCR_nFIEN), + SR_TRAP(SYS_PMCR_EL0, CGT_MDCR_TPM_TPMCR), + SR_TRAP(SYS_PMCNTENSET_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMCNTENCLR_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMOVSSET_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMOVSCLR_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMCEID0_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMCEID1_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMXEVTYPER_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMSWINC_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMSELR_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMXEVCNTR_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMCCNTR_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMUSERENR_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_PMINTENSET_EL1, CGT_MDCR_TPM), + SR_TRAP(SYS_PMINTENCLR_EL1, CGT_MDCR_TPM), + SR_TRAP(SYS_PMMIR_EL1, CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(0), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(1), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(2), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(3), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(4), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(5), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(6), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(7), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(8), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(9), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(10), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(11), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(12), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(13), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(14), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(15), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(16), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(17), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(18), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(19), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(20), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(21), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(22), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(23), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(24), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(25), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(26), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(27), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(28), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(29), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVCNTRn_EL0(30), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(0), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(1), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(2), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(3), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(4), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(5), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(6), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(7), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(8), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(9), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(10), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(11), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(12), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(13), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(14), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(15), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(16), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(17), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(18), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(19), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(20), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(21), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(22), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(23), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(24), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(25), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(26), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(27), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(28), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(29), CGT_MDCR_TPM), + SR_TRAP(SYS_PMEVTYPERn_EL0(30), CGT_MDCR_TPM), + SR_TRAP(SYS_PMCCFILTR_EL0, CGT_MDCR_TPM), + SR_TRAP(SYS_MDCCSR_EL0, CGT_MDCR_TDCC_TDE_TDA), + SR_TRAP(SYS_MDCCINT_EL1, CGT_MDCR_TDCC_TDE_TDA), + SR_TRAP(SYS_OSDTRRX_EL1, CGT_MDCR_TDCC_TDE_TDA), + SR_TRAP(SYS_OSDTRTX_EL1, CGT_MDCR_TDCC_TDE_TDA), + SR_TRAP(SYS_DBGDTR_EL0, CGT_MDCR_TDCC_TDE_TDA), + /* + * Also covers DBGDTRRX_EL0, which has the same encoding as + * SYS_DBGDTRTX_EL0... + */ + SR_TRAP(SYS_DBGDTRTX_EL0, CGT_MDCR_TDCC_TDE_TDA), + SR_TRAP(SYS_MDSCR_EL1, CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_OSECCR_EL1, CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(0), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(1), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(2), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(3), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(4), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(5), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(6), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(7), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(8), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(9), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(10), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(11), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(12), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(13), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(14), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBVRn_EL1(15), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(0), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(1), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(2), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(3), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(4), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(5), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(6), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(7), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(8), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(9), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(10), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(11), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(12), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(13), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(14), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGBCRn_EL1(15), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(0), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(1), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(2), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(3), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(4), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(5), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(6), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(7), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(8), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(9), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(10), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(11), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(12), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(13), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(14), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWVRn_EL1(15), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(0), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(1), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(2), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(3), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(4), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(5), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(6), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(7), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(8), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(9), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(10), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(11), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(12), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(13), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGWCRn_EL1(14), CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGCLAIMSET_EL1, CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGCLAIMCLR_EL1, CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_DBGAUTHSTATUS_EL1, CGT_MDCR_TDE_TDA), + SR_TRAP(SYS_OSLAR_EL1, CGT_MDCR_TDE_TDOSA), + SR_TRAP(SYS_OSLSR_EL1, CGT_MDCR_TDE_TDOSA), + SR_TRAP(SYS_OSDLR_EL1, CGT_MDCR_TDE_TDOSA), + SR_TRAP(SYS_DBGPRCR_EL1, CGT_MDCR_TDE_TDOSA), + SR_TRAP(SYS_MDRAR_EL1, CGT_MDCR_TDE_TDRA), + SR_TRAP(SYS_PMBLIMITR_EL1, CGT_MDCR_E2PB), + SR_TRAP(SYS_PMBPTR_EL1, CGT_MDCR_E2PB), + SR_TRAP(SYS_PMBSR_EL1, CGT_MDCR_E2PB), + SR_TRAP(SYS_PMSCR_EL1, CGT_MDCR_TPMS), + SR_TRAP(SYS_PMSEVFR_EL1, CGT_MDCR_TPMS), + SR_TRAP(SYS_PMSFCR_EL1, CGT_MDCR_TPMS), + SR_TRAP(SYS_PMSICR_EL1, CGT_MDCR_TPMS), + SR_TRAP(SYS_PMSIDR_EL1, CGT_MDCR_TPMS), + SR_TRAP(SYS_PMSIRR_EL1, CGT_MDCR_TPMS), + SR_TRAP(SYS_PMSLATFR_EL1, CGT_MDCR_TPMS), + SR_TRAP(SYS_PMSNEVFR_EL1, CGT_MDCR_TPMS), + SR_TRAP(SYS_TRFCR_EL1, CGT_MDCR_TTRF), + SR_TRAP(SYS_TRBBASER_EL1, CGT_MDCR_E2TB), + SR_TRAP(SYS_TRBLIMITR_EL1, CGT_MDCR_E2TB), + SR_TRAP(SYS_TRBMAR_EL1, CGT_MDCR_E2TB), + SR_TRAP(SYS_TRBPTR_EL1, CGT_MDCR_E2TB), + SR_TRAP(SYS_TRBSR_EL1, CGT_MDCR_E2TB), + SR_TRAP(SYS_TRBTRG_EL1, CGT_MDCR_E2TB), + SR_TRAP(SYS_CNTP_TVAL_EL0, CGT_CNTHCTL_EL1PTEN), + SR_TRAP(SYS_CNTP_CVAL_EL0, CGT_CNTHCTL_EL1PTEN), + SR_TRAP(SYS_CNTP_CTL_EL0, CGT_CNTHCTL_EL1PTEN), + SR_TRAP(SYS_CNTPCT_EL0, CGT_CNTHCTL_EL1PCTEN), + SR_TRAP(SYS_CNTPCTSS_EL0, CGT_CNTHCTL_EL1PCTEN), +}; + +static DEFINE_XARRAY(sr_forward_xa); + +enum fgt_group_id { + __NO_FGT_GROUP__, + HFGxTR_GROUP, + HDFGRTR_GROUP, + HDFGWTR_GROUP, + HFGITR_GROUP, + + /* Must be last */ + __NR_FGT_GROUP_IDS__ +}; + +enum fg_filter_id { + __NO_FGF__, + HCRX_FGTnXS, + + /* Must be last */ + __NR_FG_FILTER_IDS__ +}; + +#define SR_FGF(sr, g, b, p, f) \ + { \ + .encoding = sr, \ + .end = sr, \ + .tc = { \ + .fgt = g ## _GROUP, \ + .bit = g ## _EL2_ ## b ## _SHIFT, \ + .pol = p, \ + .fgf = f, \ + }, \ + .line = __LINE__, \ + } + +#define SR_FGT(sr, g, b, p) SR_FGF(sr, g, b, p, __NO_FGF__) + +static const struct encoding_to_trap_config encoding_to_fgt[] __initconst = { + /* HFGRTR_EL2, HFGWTR_EL2 */ + SR_FGT(SYS_TPIDR2_EL0, HFGxTR, nTPIDR2_EL0, 0), + SR_FGT(SYS_SMPRI_EL1, HFGxTR, nSMPRI_EL1, 0), + SR_FGT(SYS_ACCDATA_EL1, HFGxTR, nACCDATA_EL1, 0), + SR_FGT(SYS_ERXADDR_EL1, HFGxTR, ERXADDR_EL1, 1), + SR_FGT(SYS_ERXPFGCDN_EL1, HFGxTR, ERXPFGCDN_EL1, 1), + SR_FGT(SYS_ERXPFGCTL_EL1, HFGxTR, ERXPFGCTL_EL1, 1), + SR_FGT(SYS_ERXPFGF_EL1, HFGxTR, ERXPFGF_EL1, 1), + SR_FGT(SYS_ERXMISC0_EL1, HFGxTR, ERXMISCn_EL1, 1), + SR_FGT(SYS_ERXMISC1_EL1, HFGxTR, ERXMISCn_EL1, 1), + SR_FGT(SYS_ERXMISC2_EL1, HFGxTR, ERXMISCn_EL1, 1), + SR_FGT(SYS_ERXMISC3_EL1, HFGxTR, ERXMISCn_EL1, 1), + SR_FGT(SYS_ERXSTATUS_EL1, HFGxTR, ERXSTATUS_EL1, 1), + SR_FGT(SYS_ERXCTLR_EL1, HFGxTR, ERXCTLR_EL1, 1), + SR_FGT(SYS_ERXFR_EL1, HFGxTR, ERXFR_EL1, 1), + SR_FGT(SYS_ERRSELR_EL1, HFGxTR, ERRSELR_EL1, 1), + SR_FGT(SYS_ERRIDR_EL1, HFGxTR, ERRIDR_EL1, 1), + SR_FGT(SYS_ICC_IGRPEN0_EL1, HFGxTR, ICC_IGRPENn_EL1, 1), + SR_FGT(SYS_ICC_IGRPEN1_EL1, HFGxTR, ICC_IGRPENn_EL1, 1), + SR_FGT(SYS_VBAR_EL1, HFGxTR, VBAR_EL1, 1), + SR_FGT(SYS_TTBR1_EL1, HFGxTR, TTBR1_EL1, 1), + SR_FGT(SYS_TTBR0_EL1, HFGxTR, TTBR0_EL1, 1), + SR_FGT(SYS_TPIDR_EL0, HFGxTR, TPIDR_EL0, 1), + SR_FGT(SYS_TPIDRRO_EL0, HFGxTR, TPIDRRO_EL0, 1), + SR_FGT(SYS_TPIDR_EL1, HFGxTR, TPIDR_EL1, 1), + SR_FGT(SYS_TCR_EL1, HFGxTR, TCR_EL1, 1), + SR_FGT(SYS_SCXTNUM_EL0, HFGxTR, SCXTNUM_EL0, 1), + SR_FGT(SYS_SCXTNUM_EL1, HFGxTR, SCXTNUM_EL1, 1), + SR_FGT(SYS_SCTLR_EL1, HFGxTR, SCTLR_EL1, 1), + SR_FGT(SYS_REVIDR_EL1, HFGxTR, REVIDR_EL1, 1), + SR_FGT(SYS_PAR_EL1, HFGxTR, PAR_EL1, 1), + SR_FGT(SYS_MPIDR_EL1, HFGxTR, MPIDR_EL1, 1), + SR_FGT(SYS_MIDR_EL1, HFGxTR, MIDR_EL1, 1), + SR_FGT(SYS_MAIR_EL1, HFGxTR, MAIR_EL1, 1), + SR_FGT(SYS_LORSA_EL1, HFGxTR, LORSA_EL1, 1), + SR_FGT(SYS_LORN_EL1, HFGxTR, LORN_EL1, 1), + SR_FGT(SYS_LORID_EL1, HFGxTR, LORID_EL1, 1), + SR_FGT(SYS_LOREA_EL1, HFGxTR, LOREA_EL1, 1), + SR_FGT(SYS_LORC_EL1, HFGxTR, LORC_EL1, 1), + SR_FGT(SYS_ISR_EL1, HFGxTR, ISR_EL1, 1), + SR_FGT(SYS_FAR_EL1, HFGxTR, FAR_EL1, 1), + SR_FGT(SYS_ESR_EL1, HFGxTR, ESR_EL1, 1), + SR_FGT(SYS_DCZID_EL0, HFGxTR, DCZID_EL0, 1), + SR_FGT(SYS_CTR_EL0, HFGxTR, CTR_EL0, 1), + SR_FGT(SYS_CSSELR_EL1, HFGxTR, CSSELR_EL1, 1), + SR_FGT(SYS_CPACR_EL1, HFGxTR, CPACR_EL1, 1), + SR_FGT(SYS_CONTEXTIDR_EL1, HFGxTR, CONTEXTIDR_EL1, 1), + SR_FGT(SYS_CLIDR_EL1, HFGxTR, CLIDR_EL1, 1), + SR_FGT(SYS_CCSIDR_EL1, HFGxTR, CCSIDR_EL1, 1), + SR_FGT(SYS_APIBKEYLO_EL1, HFGxTR, APIBKey, 1), + SR_FGT(SYS_APIBKEYHI_EL1, HFGxTR, APIBKey, 1), + SR_FGT(SYS_APIAKEYLO_EL1, HFGxTR, APIAKey, 1), + SR_FGT(SYS_APIAKEYHI_EL1, HFGxTR, APIAKey, 1), + SR_FGT(SYS_APGAKEYLO_EL1, HFGxTR, APGAKey, 1), + SR_FGT(SYS_APGAKEYHI_EL1, HFGxTR, APGAKey, 1), + SR_FGT(SYS_APDBKEYLO_EL1, HFGxTR, APDBKey, 1), + SR_FGT(SYS_APDBKEYHI_EL1, HFGxTR, APDBKey, 1), + SR_FGT(SYS_APDAKEYLO_EL1, HFGxTR, APDAKey, 1), + SR_FGT(SYS_APDAKEYHI_EL1, HFGxTR, APDAKey, 1), + SR_FGT(SYS_AMAIR_EL1, HFGxTR, AMAIR_EL1, 1), + SR_FGT(SYS_AIDR_EL1, HFGxTR, AIDR_EL1, 1), + SR_FGT(SYS_AFSR1_EL1, HFGxTR, AFSR1_EL1, 1), + SR_FGT(SYS_AFSR0_EL1, HFGxTR, AFSR0_EL1, 1), + /* HFGITR_EL2 */ + SR_FGT(OP_BRB_IALL, HFGITR, nBRBIALL, 0), + SR_FGT(OP_BRB_INJ, HFGITR, nBRBINJ, 0), + SR_FGT(SYS_DC_CVAC, HFGITR, DCCVAC, 1), + SR_FGT(SYS_DC_CGVAC, HFGITR, DCCVAC, 1), + SR_FGT(SYS_DC_CGDVAC, HFGITR, DCCVAC, 1), + SR_FGT(OP_CPP_RCTX, HFGITR, CPPRCTX, 1), + SR_FGT(OP_DVP_RCTX, HFGITR, DVPRCTX, 1), + SR_FGT(OP_CFP_RCTX, HFGITR, CFPRCTX, 1), + SR_FGT(OP_TLBI_VAALE1, HFGITR, TLBIVAALE1, 1), + SR_FGT(OP_TLBI_VALE1, HFGITR, TLBIVALE1, 1), + SR_FGT(OP_TLBI_VAAE1, HFGITR, TLBIVAAE1, 1), + SR_FGT(OP_TLBI_ASIDE1, HFGITR, TLBIASIDE1, 1), + SR_FGT(OP_TLBI_VAE1, HFGITR, TLBIVAE1, 1), + SR_FGT(OP_TLBI_VMALLE1, HFGITR, TLBIVMALLE1, 1), + SR_FGT(OP_TLBI_RVAALE1, HFGITR, TLBIRVAALE1, 1), + SR_FGT(OP_TLBI_RVALE1, HFGITR, TLBIRVALE1, 1), + SR_FGT(OP_TLBI_RVAAE1, HFGITR, TLBIRVAAE1, 1), + SR_FGT(OP_TLBI_RVAE1, HFGITR, TLBIRVAE1, 1), + SR_FGT(OP_TLBI_RVAALE1IS, HFGITR, TLBIRVAALE1IS, 1), + SR_FGT(OP_TLBI_RVALE1IS, HFGITR, TLBIRVALE1IS, 1), + SR_FGT(OP_TLBI_RVAAE1IS, HFGITR, TLBIRVAAE1IS, 1), + SR_FGT(OP_TLBI_RVAE1IS, HFGITR, TLBIRVAE1IS, 1), + SR_FGT(OP_TLBI_VAALE1IS, HFGITR, TLBIVAALE1IS, 1), + SR_FGT(OP_TLBI_VALE1IS, HFGITR, TLBIVALE1IS, 1), + SR_FGT(OP_TLBI_VAAE1IS, HFGITR, TLBIVAAE1IS, 1), + SR_FGT(OP_TLBI_ASIDE1IS, HFGITR, TLBIASIDE1IS, 1), + SR_FGT(OP_TLBI_VAE1IS, HFGITR, TLBIVAE1IS, 1), + SR_FGT(OP_TLBI_VMALLE1IS, HFGITR, TLBIVMALLE1IS, 1), + SR_FGT(OP_TLBI_RVAALE1OS, HFGITR, TLBIRVAALE1OS, 1), + SR_FGT(OP_TLBI_RVALE1OS, HFGITR, TLBIRVALE1OS, 1), + SR_FGT(OP_TLBI_RVAAE1OS, HFGITR, TLBIRVAAE1OS, 1), + SR_FGT(OP_TLBI_RVAE1OS, HFGITR, TLBIRVAE1OS, 1), + SR_FGT(OP_TLBI_VAALE1OS, HFGITR, TLBIVAALE1OS, 1), + SR_FGT(OP_TLBI_VALE1OS, HFGITR, TLBIVALE1OS, 1), + SR_FGT(OP_TLBI_VAAE1OS, HFGITR, TLBIVAAE1OS, 1), + SR_FGT(OP_TLBI_ASIDE1OS, HFGITR, TLBIASIDE1OS, 1), + SR_FGT(OP_TLBI_VAE1OS, HFGITR, TLBIVAE1OS, 1), + SR_FGT(OP_TLBI_VMALLE1OS, HFGITR, TLBIVMALLE1OS, 1), + /* nXS variants must be checked against HCRX_EL2.FGTnXS */ + SR_FGF(OP_TLBI_VAALE1NXS, HFGITR, TLBIVAALE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VALE1NXS, HFGITR, TLBIVALE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VAAE1NXS, HFGITR, TLBIVAAE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_ASIDE1NXS, HFGITR, TLBIASIDE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VAE1NXS, HFGITR, TLBIVAE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VMALLE1NXS, HFGITR, TLBIVMALLE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAALE1NXS, HFGITR, TLBIRVAALE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVALE1NXS, HFGITR, TLBIRVALE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAAE1NXS, HFGITR, TLBIRVAAE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAE1NXS, HFGITR, TLBIRVAE1, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAALE1ISNXS, HFGITR, TLBIRVAALE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVALE1ISNXS, HFGITR, TLBIRVALE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAAE1ISNXS, HFGITR, TLBIRVAAE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAE1ISNXS, HFGITR, TLBIRVAE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VAALE1ISNXS, HFGITR, TLBIVAALE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VALE1ISNXS, HFGITR, TLBIVALE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VAAE1ISNXS, HFGITR, TLBIVAAE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_ASIDE1ISNXS, HFGITR, TLBIASIDE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VAE1ISNXS, HFGITR, TLBIVAE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VMALLE1ISNXS, HFGITR, TLBIVMALLE1IS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAALE1OSNXS, HFGITR, TLBIRVAALE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVALE1OSNXS, HFGITR, TLBIRVALE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAAE1OSNXS, HFGITR, TLBIRVAAE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_RVAE1OSNXS, HFGITR, TLBIRVAE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VAALE1OSNXS, HFGITR, TLBIVAALE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VALE1OSNXS, HFGITR, TLBIVALE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VAAE1OSNXS, HFGITR, TLBIVAAE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_ASIDE1OSNXS, HFGITR, TLBIASIDE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VAE1OSNXS, HFGITR, TLBIVAE1OS, 1, HCRX_FGTnXS), + SR_FGF(OP_TLBI_VMALLE1OSNXS, HFGITR, TLBIVMALLE1OS, 1, HCRX_FGTnXS), + SR_FGT(OP_AT_S1E1WP, HFGITR, ATS1E1WP, 1), + SR_FGT(OP_AT_S1E1RP, HFGITR, ATS1E1RP, 1), + SR_FGT(OP_AT_S1E0W, HFGITR, ATS1E0W, 1), + SR_FGT(OP_AT_S1E0R, HFGITR, ATS1E0R, 1), + SR_FGT(OP_AT_S1E1W, HFGITR, ATS1E1W, 1), + SR_FGT(OP_AT_S1E1R, HFGITR, ATS1E1R, 1), + SR_FGT(SYS_DC_ZVA, HFGITR, DCZVA, 1), + SR_FGT(SYS_DC_GVA, HFGITR, DCZVA, 1), + SR_FGT(SYS_DC_GZVA, HFGITR, DCZVA, 1), + SR_FGT(SYS_DC_CIVAC, HFGITR, DCCIVAC, 1), + SR_FGT(SYS_DC_CIGVAC, HFGITR, DCCIVAC, 1), + SR_FGT(SYS_DC_CIGDVAC, HFGITR, DCCIVAC, 1), + SR_FGT(SYS_DC_CVADP, HFGITR, DCCVADP, 1), + SR_FGT(SYS_DC_CGVADP, HFGITR, DCCVADP, 1), + SR_FGT(SYS_DC_CGDVADP, HFGITR, DCCVADP, 1), + SR_FGT(SYS_DC_CVAP, HFGITR, DCCVAP, 1), + SR_FGT(SYS_DC_CGVAP, HFGITR, DCCVAP, 1), + SR_FGT(SYS_DC_CGDVAP, HFGITR, DCCVAP, 1), + SR_FGT(SYS_DC_CVAU, HFGITR, DCCVAU, 1), + SR_FGT(SYS_DC_CISW, HFGITR, DCCISW, 1), + SR_FGT(SYS_DC_CIGSW, HFGITR, DCCISW, 1), + SR_FGT(SYS_DC_CIGDSW, HFGITR, DCCISW, 1), + SR_FGT(SYS_DC_CSW, HFGITR, DCCSW, 1), + SR_FGT(SYS_DC_CGSW, HFGITR, DCCSW, 1), + SR_FGT(SYS_DC_CGDSW, HFGITR, DCCSW, 1), + SR_FGT(SYS_DC_ISW, HFGITR, DCISW, 1), + SR_FGT(SYS_DC_IGSW, HFGITR, DCISW, 1), + SR_FGT(SYS_DC_IGDSW, HFGITR, DCISW, 1), + SR_FGT(SYS_DC_IVAC, HFGITR, DCIVAC, 1), + SR_FGT(SYS_DC_IGVAC, HFGITR, DCIVAC, 1), + SR_FGT(SYS_DC_IGDVAC, HFGITR, DCIVAC, 1), + SR_FGT(SYS_IC_IVAU, HFGITR, ICIVAU, 1), + SR_FGT(SYS_IC_IALLU, HFGITR, ICIALLU, 1), + SR_FGT(SYS_IC_IALLUIS, HFGITR, ICIALLUIS, 1), + /* HDFGRTR_EL2 */ + SR_FGT(SYS_PMBIDR_EL1, HDFGRTR, PMBIDR_EL1, 1), + SR_FGT(SYS_PMSNEVFR_EL1, HDFGRTR, nPMSNEVFR_EL1, 0), + SR_FGT(SYS_BRBINF_EL1(0), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(1), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(2), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(3), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(4), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(5), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(6), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(7), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(8), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(9), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(10), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(11), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(12), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(13), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(14), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(15), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(16), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(17), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(18), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(19), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(20), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(21), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(22), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(23), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(24), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(25), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(26), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(27), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(28), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(29), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(30), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINF_EL1(31), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBINFINJ_EL1, HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(0), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(1), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(2), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(3), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(4), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(5), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(6), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(7), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(8), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(9), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(10), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(11), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(12), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(13), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(14), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(15), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(16), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(17), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(18), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(19), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(20), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(21), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(22), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(23), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(24), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(25), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(26), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(27), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(28), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(29), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(30), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRC_EL1(31), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBSRCINJ_EL1, HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(0), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(1), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(2), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(3), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(4), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(5), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(6), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(7), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(8), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(9), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(10), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(11), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(12), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(13), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(14), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(15), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(16), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(17), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(18), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(19), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(20), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(21), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(22), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(23), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(24), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(25), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(26), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(27), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(28), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(29), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(30), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGT_EL1(31), HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTGTINJ_EL1, HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBTS_EL1, HDFGRTR, nBRBDATA, 0), + SR_FGT(SYS_BRBCR_EL1, HDFGRTR, nBRBCTL, 0), + SR_FGT(SYS_BRBFCR_EL1, HDFGRTR, nBRBCTL, 0), + SR_FGT(SYS_BRBIDR0_EL1, HDFGRTR, nBRBIDR, 0), + SR_FGT(SYS_PMCEID0_EL0, HDFGRTR, PMCEIDn_EL0, 1), + SR_FGT(SYS_PMCEID1_EL0, HDFGRTR, PMCEIDn_EL0, 1), + SR_FGT(SYS_PMUSERENR_EL0, HDFGRTR, PMUSERENR_EL0, 1), + SR_FGT(SYS_TRBTRG_EL1, HDFGRTR, TRBTRG_EL1, 1), + SR_FGT(SYS_TRBSR_EL1, HDFGRTR, TRBSR_EL1, 1), + SR_FGT(SYS_TRBPTR_EL1, HDFGRTR, TRBPTR_EL1, 1), + SR_FGT(SYS_TRBMAR_EL1, HDFGRTR, TRBMAR_EL1, 1), + SR_FGT(SYS_TRBLIMITR_EL1, HDFGRTR, TRBLIMITR_EL1, 1), + SR_FGT(SYS_TRBIDR_EL1, HDFGRTR, TRBIDR_EL1, 1), + SR_FGT(SYS_TRBBASER_EL1, HDFGRTR, TRBBASER_EL1, 1), + SR_FGT(SYS_TRCVICTLR, HDFGRTR, TRCVICTLR, 1), + SR_FGT(SYS_TRCSTATR, HDFGRTR, TRCSTATR, 1), + SR_FGT(SYS_TRCSSCSR(0), HDFGRTR, TRCSSCSRn, 1), + SR_FGT(SYS_TRCSSCSR(1), HDFGRTR, TRCSSCSRn, 1), + SR_FGT(SYS_TRCSSCSR(2), HDFGRTR, TRCSSCSRn, 1), + SR_FGT(SYS_TRCSSCSR(3), HDFGRTR, TRCSSCSRn, 1), + SR_FGT(SYS_TRCSSCSR(4), HDFGRTR, TRCSSCSRn, 1), + SR_FGT(SYS_TRCSSCSR(5), HDFGRTR, TRCSSCSRn, 1), + SR_FGT(SYS_TRCSSCSR(6), HDFGRTR, TRCSSCSRn, 1), + SR_FGT(SYS_TRCSSCSR(7), HDFGRTR, TRCSSCSRn, 1), + SR_FGT(SYS_TRCSEQSTR, HDFGRTR, TRCSEQSTR, 1), + SR_FGT(SYS_TRCPRGCTLR, HDFGRTR, TRCPRGCTLR, 1), + SR_FGT(SYS_TRCOSLSR, HDFGRTR, TRCOSLSR, 1), + SR_FGT(SYS_TRCIMSPEC(0), HDFGRTR, TRCIMSPECn, 1), + SR_FGT(SYS_TRCIMSPEC(1), HDFGRTR, TRCIMSPECn, 1), + SR_FGT(SYS_TRCIMSPEC(2), HDFGRTR, TRCIMSPECn, 1), + SR_FGT(SYS_TRCIMSPEC(3), HDFGRTR, TRCIMSPECn, 1), + SR_FGT(SYS_TRCIMSPEC(4), HDFGRTR, TRCIMSPECn, 1), + SR_FGT(SYS_TRCIMSPEC(5), HDFGRTR, TRCIMSPECn, 1), + SR_FGT(SYS_TRCIMSPEC(6), HDFGRTR, TRCIMSPECn, 1), + SR_FGT(SYS_TRCIMSPEC(7), HDFGRTR, TRCIMSPECn, 1), + SR_FGT(SYS_TRCDEVARCH, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCDEVID, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR0, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR1, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR2, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR3, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR4, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR5, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR6, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR7, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR8, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR9, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR10, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR11, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR12, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCIDR13, HDFGRTR, TRCID, 1), + SR_FGT(SYS_TRCCNTVR(0), HDFGRTR, TRCCNTVRn, 1), + SR_FGT(SYS_TRCCNTVR(1), HDFGRTR, TRCCNTVRn, 1), + SR_FGT(SYS_TRCCNTVR(2), HDFGRTR, TRCCNTVRn, 1), + SR_FGT(SYS_TRCCNTVR(3), HDFGRTR, TRCCNTVRn, 1), + SR_FGT(SYS_TRCCLAIMCLR, HDFGRTR, TRCCLAIM, 1), + SR_FGT(SYS_TRCCLAIMSET, HDFGRTR, TRCCLAIM, 1), + SR_FGT(SYS_TRCAUXCTLR, HDFGRTR, TRCAUXCTLR, 1), + SR_FGT(SYS_TRCAUTHSTATUS, HDFGRTR, TRCAUTHSTATUS, 1), + SR_FGT(SYS_TRCACATR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(4), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(5), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(6), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(7), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(8), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(9), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(10), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(11), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(12), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(13), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(14), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACATR(15), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(4), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(5), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(6), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(7), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(8), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(9), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(10), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(11), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(12), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(13), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(14), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCACVR(15), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCBBCTLR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCCCTLR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCCTLR0, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCCTLR1, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCVR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCVR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCVR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCVR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCVR(4), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCVR(5), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCVR(6), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCIDCVR(7), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCNTCTLR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCNTCTLR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCNTCTLR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCNTCTLR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCNTRLDVR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCNTRLDVR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCNTRLDVR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCNTRLDVR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCCONFIGR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCEVENTCTL0R, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCEVENTCTL1R, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCEXTINSELR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCEXTINSELR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCEXTINSELR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCEXTINSELR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCQCTLR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(4), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(5), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(6), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(7), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(8), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(9), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(10), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(11), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(12), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(13), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(14), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(15), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(16), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(17), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(18), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(19), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(20), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(21), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(22), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(23), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(24), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(25), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(26), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(27), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(28), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(29), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(30), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSCTLR(31), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCRSR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSEQEVR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSEQEVR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSEQEVR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSEQRSTEVR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSCCR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSCCR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSCCR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSCCR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSCCR(4), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSCCR(5), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSCCR(6), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSCCR(7), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSPCICR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSPCICR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSPCICR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSPCICR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSPCICR(4), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSPCICR(5), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSPCICR(6), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSSPCICR(7), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSTALLCTLR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCSYNCPR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCTRACEIDR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCTSCTLR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVIIECTLR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVIPCSSCTLR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVISSCTLR, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCCTLR0, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCCTLR1, HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCVR(0), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCVR(1), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCVR(2), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCVR(3), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCVR(4), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCVR(5), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCVR(6), HDFGRTR, TRC, 1), + SR_FGT(SYS_TRCVMIDCVR(7), HDFGRTR, TRC, 1), + SR_FGT(SYS_PMSLATFR_EL1, HDFGRTR, PMSLATFR_EL1, 1), + SR_FGT(SYS_PMSIRR_EL1, HDFGRTR, PMSIRR_EL1, 1), + SR_FGT(SYS_PMSIDR_EL1, HDFGRTR, PMSIDR_EL1, 1), + SR_FGT(SYS_PMSICR_EL1, HDFGRTR, PMSICR_EL1, 1), + SR_FGT(SYS_PMSFCR_EL1, HDFGRTR, PMSFCR_EL1, 1), + SR_FGT(SYS_PMSEVFR_EL1, HDFGRTR, PMSEVFR_EL1, 1), + SR_FGT(SYS_PMSCR_EL1, HDFGRTR, PMSCR_EL1, 1), + SR_FGT(SYS_PMBSR_EL1, HDFGRTR, PMBSR_EL1, 1), + SR_FGT(SYS_PMBPTR_EL1, HDFGRTR, PMBPTR_EL1, 1), + SR_FGT(SYS_PMBLIMITR_EL1, HDFGRTR, PMBLIMITR_EL1, 1), + SR_FGT(SYS_PMMIR_EL1, HDFGRTR, PMMIR_EL1, 1), + SR_FGT(SYS_PMSELR_EL0, HDFGRTR, PMSELR_EL0, 1), + SR_FGT(SYS_PMOVSCLR_EL0, HDFGRTR, PMOVS, 1), + SR_FGT(SYS_PMOVSSET_EL0, HDFGRTR, PMOVS, 1), + SR_FGT(SYS_PMINTENCLR_EL1, HDFGRTR, PMINTEN, 1), + SR_FGT(SYS_PMINTENSET_EL1, HDFGRTR, PMINTEN, 1), + SR_FGT(SYS_PMCNTENCLR_EL0, HDFGRTR, PMCNTEN, 1), + SR_FGT(SYS_PMCNTENSET_EL0, HDFGRTR, PMCNTEN, 1), + SR_FGT(SYS_PMCCNTR_EL0, HDFGRTR, PMCCNTR_EL0, 1), + SR_FGT(SYS_PMCCFILTR_EL0, HDFGRTR, PMCCFILTR_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(0), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(1), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(2), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(3), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(4), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(5), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(6), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(7), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(8), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(9), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(10), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(11), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(12), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(13), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(14), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(15), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(16), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(17), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(18), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(19), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(20), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(21), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(22), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(23), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(24), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(25), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(26), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(27), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(28), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(29), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVTYPERn_EL0(30), HDFGRTR, PMEVTYPERn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(0), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(1), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(2), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(3), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(4), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(5), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(6), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(7), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(8), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(9), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(10), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(11), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(12), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(13), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(14), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(15), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(16), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(17), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(18), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(19), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(20), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(21), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(22), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(23), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(24), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(25), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(26), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(27), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(28), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(29), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_PMEVCNTRn_EL0(30), HDFGRTR, PMEVCNTRn_EL0, 1), + SR_FGT(SYS_OSDLR_EL1, HDFGRTR, OSDLR_EL1, 1), + SR_FGT(SYS_OSECCR_EL1, HDFGRTR, OSECCR_EL1, 1), + SR_FGT(SYS_OSLSR_EL1, HDFGRTR, OSLSR_EL1, 1), + SR_FGT(SYS_DBGPRCR_EL1, HDFGRTR, DBGPRCR_EL1, 1), + SR_FGT(SYS_DBGAUTHSTATUS_EL1, HDFGRTR, DBGAUTHSTATUS_EL1, 1), + SR_FGT(SYS_DBGCLAIMSET_EL1, HDFGRTR, DBGCLAIM, 1), + SR_FGT(SYS_DBGCLAIMCLR_EL1, HDFGRTR, DBGCLAIM, 1), + SR_FGT(SYS_MDSCR_EL1, HDFGRTR, MDSCR_EL1, 1), + /* + * The trap bits capture *64* debug registers per bit, but the + * ARM ARM only describes the encoding for the first 16, and + * we don't really support more than that anyway. + */ + SR_FGT(SYS_DBGWVRn_EL1(0), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(1), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(2), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(3), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(4), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(5), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(6), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(7), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(8), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(9), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(10), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(11), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(12), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(13), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(14), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWVRn_EL1(15), HDFGRTR, DBGWVRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(0), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(1), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(2), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(3), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(4), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(5), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(6), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(7), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(8), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(9), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(10), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(11), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(12), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(13), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(14), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGWCRn_EL1(15), HDFGRTR, DBGWCRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(0), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(1), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(2), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(3), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(4), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(5), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(6), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(7), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(8), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(9), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(10), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(11), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(12), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(13), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(14), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBVRn_EL1(15), HDFGRTR, DBGBVRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(0), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(1), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(2), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(3), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(4), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(5), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(6), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(7), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(8), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(9), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(10), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(11), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(12), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(13), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(14), HDFGRTR, DBGBCRn_EL1, 1), + SR_FGT(SYS_DBGBCRn_EL1(15), HDFGRTR, DBGBCRn_EL1, 1), + /* + * HDFGWTR_EL2 + * + * Although HDFGRTR_EL2 and HDFGWTR_EL2 registers largely + * overlap in their bit assignment, there are a number of bits + * that are RES0 on one side, and an actual trap bit on the + * other. The policy chosen here is to describe all the + * read-side mappings, and only the write-side mappings that + * differ from the read side, and the trap handler will pick + * the correct shadow register based on the access type. + */ + SR_FGT(SYS_TRFCR_EL1, HDFGWTR, TRFCR_EL1, 1), + SR_FGT(SYS_TRCOSLAR, HDFGWTR, TRCOSLAR, 1), + SR_FGT(SYS_PMCR_EL0, HDFGWTR, PMCR_EL0, 1), + SR_FGT(SYS_PMSWINC_EL0, HDFGWTR, PMSWINC_EL0, 1), + SR_FGT(SYS_OSLAR_EL1, HDFGWTR, OSLAR_EL1, 1), +}; + +static union trap_config get_trap_config(u32 sysreg) +{ + return (union trap_config) { + .val = xa_to_value(xa_load(&sr_forward_xa, sysreg)), + }; +} + +static __init void print_nv_trap_error(const struct encoding_to_trap_config *tc, + const char *type, int err) +{ + kvm_err("%s line %d encoding range " + "(%d, %d, %d, %d, %d) - (%d, %d, %d, %d, %d) (err=%d)\n", + type, tc->line, + sys_reg_Op0(tc->encoding), sys_reg_Op1(tc->encoding), + sys_reg_CRn(tc->encoding), sys_reg_CRm(tc->encoding), + sys_reg_Op2(tc->encoding), + sys_reg_Op0(tc->end), sys_reg_Op1(tc->end), + sys_reg_CRn(tc->end), sys_reg_CRm(tc->end), + sys_reg_Op2(tc->end), + err); +} + +int __init populate_nv_trap_config(void) +{ + int ret = 0; + + BUILD_BUG_ON(sizeof(union trap_config) != sizeof(void *)); + BUILD_BUG_ON(__NR_CGT_GROUP_IDS__ > BIT(TC_CGT_BITS)); + BUILD_BUG_ON(__NR_FGT_GROUP_IDS__ > BIT(TC_FGT_BITS)); + BUILD_BUG_ON(__NR_FG_FILTER_IDS__ > BIT(TC_FGF_BITS)); + + for (int i = 0; i < ARRAY_SIZE(encoding_to_cgt); i++) { + const struct encoding_to_trap_config *cgt = &encoding_to_cgt[i]; + void *prev; + + if (cgt->tc.val & BIT(63)) { + kvm_err("CGT[%d] has MBZ bit set\n", i); + ret = -EINVAL; + } + + if (cgt->encoding != cgt->end) { + prev = xa_store_range(&sr_forward_xa, + cgt->encoding, cgt->end, + xa_mk_value(cgt->tc.val), + GFP_KERNEL); + } else { + prev = xa_store(&sr_forward_xa, cgt->encoding, + xa_mk_value(cgt->tc.val), GFP_KERNEL); + if (prev && !xa_is_err(prev)) { + ret = -EINVAL; + print_nv_trap_error(cgt, "Duplicate CGT", ret); + } + } + + if (xa_is_err(prev)) { + ret = xa_err(prev); + print_nv_trap_error(cgt, "Failed CGT insertion", ret); + } + } + + kvm_info("nv: %ld coarse grained trap handlers\n", + ARRAY_SIZE(encoding_to_cgt)); + + if (!cpus_have_final_cap(ARM64_HAS_FGT)) + goto check_mcb; + + for (int i = 0; i < ARRAY_SIZE(encoding_to_fgt); i++) { + const struct encoding_to_trap_config *fgt = &encoding_to_fgt[i]; + union trap_config tc; + + if (fgt->tc.fgt >= __NR_FGT_GROUP_IDS__) { + ret = -EINVAL; + print_nv_trap_error(fgt, "Invalid FGT", ret); + } + + tc = get_trap_config(fgt->encoding); + + if (tc.fgt) { + ret = -EINVAL; + print_nv_trap_error(fgt, "Duplicate FGT", ret); + } + + tc.val |= fgt->tc.val; + xa_store(&sr_forward_xa, fgt->encoding, + xa_mk_value(tc.val), GFP_KERNEL); + } + + kvm_info("nv: %ld fine grained trap handlers\n", + ARRAY_SIZE(encoding_to_fgt)); + +check_mcb: + for (int id = __MULTIPLE_CONTROL_BITS__; id < __COMPLEX_CONDITIONS__; id++) { + const enum cgt_group_id *cgids; + + cgids = coarse_control_combo[id - __MULTIPLE_CONTROL_BITS__]; + + for (int i = 0; cgids[i] != __RESERVED__; i++) { + if (cgids[i] >= __MULTIPLE_CONTROL_BITS__) { + kvm_err("Recursive MCB %d/%d\n", id, cgids[i]); + ret = -EINVAL; + } + } + } + + if (ret) + xa_destroy(&sr_forward_xa); + + return ret; +} + +static enum trap_behaviour get_behaviour(struct kvm_vcpu *vcpu, + const struct trap_bits *tb) +{ + enum trap_behaviour b = BEHAVE_HANDLE_LOCALLY; + u64 val; + + val = __vcpu_sys_reg(vcpu, tb->index); + if ((val & tb->mask) == tb->value) + b |= tb->behaviour; + + return b; +} + +static enum trap_behaviour __compute_trap_behaviour(struct kvm_vcpu *vcpu, + const enum cgt_group_id id, + enum trap_behaviour b) +{ + switch (id) { + const enum cgt_group_id *cgids; + + case __RESERVED__ ... __MULTIPLE_CONTROL_BITS__ - 1: + if (likely(id != __RESERVED__)) + b |= get_behaviour(vcpu, &coarse_trap_bits[id]); + break; + case __MULTIPLE_CONTROL_BITS__ ... __COMPLEX_CONDITIONS__ - 1: + /* Yes, this is recursive. Don't do anything stupid. */ + cgids = coarse_control_combo[id - __MULTIPLE_CONTROL_BITS__]; + for (int i = 0; cgids[i] != __RESERVED__; i++) + b |= __compute_trap_behaviour(vcpu, cgids[i], b); + break; + default: + if (ARRAY_SIZE(ccc)) + b |= ccc[id - __COMPLEX_CONDITIONS__](vcpu); + break; + } + + return b; +} + +static enum trap_behaviour compute_trap_behaviour(struct kvm_vcpu *vcpu, + const union trap_config tc) +{ + enum trap_behaviour b = BEHAVE_HANDLE_LOCALLY; + + return __compute_trap_behaviour(vcpu, tc.cgt, b); +} + +static bool check_fgt_bit(u64 val, const union trap_config tc) +{ + return ((val >> tc.bit) & 1) == tc.pol; +} + +#define sanitised_sys_reg(vcpu, reg) \ + ({ \ + u64 __val; \ + __val = __vcpu_sys_reg(vcpu, reg); \ + __val &= ~__ ## reg ## _RES0; \ + (__val); \ + }) + +bool __check_nv_sr_forward(struct kvm_vcpu *vcpu) +{ + union trap_config tc; + enum trap_behaviour b; + bool is_read; + u32 sysreg; + u64 esr, val; + + if (!vcpu_has_nv(vcpu) || is_hyp_ctxt(vcpu)) + return false; + + esr = kvm_vcpu_get_esr(vcpu); + sysreg = esr_sys64_to_sysreg(esr); + is_read = (esr & ESR_ELx_SYS64_ISS_DIR_MASK) == ESR_ELx_SYS64_ISS_DIR_READ; + + tc = get_trap_config(sysreg); + + /* + * A value of 0 for the whole entry means that we know nothing + * for this sysreg, and that it cannot be re-injected into the + * nested hypervisor. In this situation, let's cut it short. + * + * Note that ultimately, we could also make use of the xarray + * to store the index of the sysreg in the local descriptor + * array, avoiding another search... Hint, hint... + */ + if (!tc.val) + return false; + + switch ((enum fgt_group_id)tc.fgt) { + case __NO_FGT_GROUP__: + break; + + case HFGxTR_GROUP: + if (is_read) + val = sanitised_sys_reg(vcpu, HFGRTR_EL2); + else + val = sanitised_sys_reg(vcpu, HFGWTR_EL2); + break; + + case HDFGRTR_GROUP: + case HDFGWTR_GROUP: + if (is_read) + val = sanitised_sys_reg(vcpu, HDFGRTR_EL2); + else + val = sanitised_sys_reg(vcpu, HDFGWTR_EL2); + break; + + case HFGITR_GROUP: + val = sanitised_sys_reg(vcpu, HFGITR_EL2); + switch (tc.fgf) { + u64 tmp; + + case __NO_FGF__: + break; + + case HCRX_FGTnXS: + tmp = sanitised_sys_reg(vcpu, HCRX_EL2); + if (tmp & HCRX_EL2_FGTnXS) + tc.fgt = __NO_FGT_GROUP__; + } + break; + + case __NR_FGT_GROUP_IDS__: + /* Something is really wrong, bail out */ + WARN_ONCE(1, "__NR_FGT_GROUP_IDS__"); + return false; + } + + if (tc.fgt != __NO_FGT_GROUP__ && check_fgt_bit(val, tc)) + goto inject; + + b = compute_trap_behaviour(vcpu, tc); + + if (((b & BEHAVE_FORWARD_READ) && is_read) || + ((b & BEHAVE_FORWARD_WRITE) && !is_read)) + goto inject; + + return false; + +inject: + trace_kvm_forward_sysreg_trap(vcpu, sysreg, is_read); + + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + return true; +} + static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr) { u64 mode = spsr & PSR_MODE_MASK; diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 20280a5233f6..95f6945c4432 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -884,21 +884,6 @@ u32 __attribute_const__ kvm_target_cpu(void) return KVM_ARM_TARGET_GENERIC_V8; } -void kvm_vcpu_preferred_target(struct kvm_vcpu_init *init) -{ - u32 target = kvm_target_cpu(); - - memset(init, 0, sizeof(*init)); - - /* - * For now, we don't return any features. - * In future, we might use features to return target - * specific features available for the preferred - * target type. - */ - init->target = (__u32)target; -} - int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) { return -EINVAL; diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 6dcd6604b6bc..617ae6dea5d5 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -222,7 +222,33 @@ static int kvm_handle_eret(struct kvm_vcpu *vcpu) if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERET) return kvm_handle_ptrauth(vcpu); - kvm_emulate_nested_eret(vcpu); + /* + * If we got here, two possibilities: + * + * - the guest is in EL2, and we need to fully emulate ERET + * + * - the guest is in EL1, and we need to reinject the + * exception into the L1 hypervisor. + * + * If KVM ever traps ERET for its own use, we'll have to + * revisit this. + */ + if (is_hyp_ctxt(vcpu)) + kvm_emulate_nested_eret(vcpu); + else + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + + return 1; +} + +static int handle_svc(struct kvm_vcpu *vcpu) +{ + /* + * So far, SVC traps only for NV via HFGITR_EL2. A SVC from a + * 32bit guest would be caught by vpcu_mode_is_bad_32bit(), so + * we should only have to deal with a 64 bit exception. + */ + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); return 1; } @@ -239,6 +265,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SMC32] = handle_smc, [ESR_ELx_EC_HVC64] = handle_hvc, [ESR_ELx_EC_SMC64] = handle_smc, + [ESR_ELx_EC_SVC64] = handle_svc, [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, [ESR_ELx_EC_SVE] = handle_sve, [ESR_ELx_EC_ERET] = kvm_handle_eret, diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 34f222af6165..9cfe6bd1dbe4 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -70,20 +70,26 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu) } } -static inline bool __hfgxtr_traps_required(void) -{ - if (cpus_have_final_cap(ARM64_SME)) - return true; - - if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38)) - return true; - - return false; -} - -static inline void __activate_traps_hfgxtr(void) +#define compute_clr_set(vcpu, reg, clr, set) \ + do { \ + u64 hfg; \ + hfg = __vcpu_sys_reg(vcpu, reg) & ~__ ## reg ## _RES0; \ + set |= hfg & __ ## reg ## _MASK; \ + clr |= ~hfg & __ ## reg ## _nMASK; \ + } while(0) + + +static inline void __activate_traps_hfgxtr(struct kvm_vcpu *vcpu) { + struct kvm_cpu_context *hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp; + u64 r_val, w_val; + + if (!cpus_have_final_cap(ARM64_HAS_FGT)) + return; + + ctxt_sys_reg(hctxt, HFGRTR_EL2) = read_sysreg_s(SYS_HFGRTR_EL2); + ctxt_sys_reg(hctxt, HFGWTR_EL2) = read_sysreg_s(SYS_HFGWTR_EL2); if (cpus_have_final_cap(ARM64_SME)) { tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK; @@ -98,26 +104,72 @@ static inline void __activate_traps_hfgxtr(void) if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38)) w_set |= HFGxTR_EL2_TCR_EL1_MASK; - sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set); - sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set); -} - -static inline void __deactivate_traps_hfgxtr(void) -{ - u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp; - - if (cpus_have_final_cap(ARM64_SME)) { - tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK; - - r_set |= tmp; - w_set |= tmp; + if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) { + compute_clr_set(vcpu, HFGRTR_EL2, r_clr, r_set); + compute_clr_set(vcpu, HFGWTR_EL2, w_clr, w_set); } - if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38)) - w_clr |= HFGxTR_EL2_TCR_EL1_MASK; + /* The default is not to trap anything but ACCDATA_EL1 */ + r_val = __HFGRTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1; + r_val |= r_set; + r_val &= ~r_clr; - sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set); - sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set); + w_val = __HFGWTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1; + w_val |= w_set; + w_val &= ~w_clr; + + write_sysreg_s(r_val, SYS_HFGRTR_EL2); + write_sysreg_s(w_val, SYS_HFGWTR_EL2); + + if (!vcpu_has_nv(vcpu) || is_hyp_ctxt(vcpu)) + return; + + ctxt_sys_reg(hctxt, HFGITR_EL2) = read_sysreg_s(SYS_HFGITR_EL2); + + r_set = r_clr = 0; + compute_clr_set(vcpu, HFGITR_EL2, r_clr, r_set); + r_val = __HFGITR_EL2_nMASK; + r_val |= r_set; + r_val &= ~r_clr; + + write_sysreg_s(r_val, SYS_HFGITR_EL2); + + ctxt_sys_reg(hctxt, HDFGRTR_EL2) = read_sysreg_s(SYS_HDFGRTR_EL2); + ctxt_sys_reg(hctxt, HDFGWTR_EL2) = read_sysreg_s(SYS_HDFGWTR_EL2); + + r_clr = r_set = w_clr = w_set = 0; + + compute_clr_set(vcpu, HDFGRTR_EL2, r_clr, r_set); + compute_clr_set(vcpu, HDFGWTR_EL2, w_clr, w_set); + + r_val = __HDFGRTR_EL2_nMASK; + r_val |= r_set; + r_val &= ~r_clr; + + w_val = __HDFGWTR_EL2_nMASK; + w_val |= w_set; + w_val &= ~w_clr; + + write_sysreg_s(r_val, SYS_HDFGRTR_EL2); + write_sysreg_s(w_val, SYS_HDFGWTR_EL2); +} + +static inline void __deactivate_traps_hfgxtr(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; + + if (!cpus_have_final_cap(ARM64_HAS_FGT)) + return; + + write_sysreg_s(ctxt_sys_reg(hctxt, HFGRTR_EL2), SYS_HFGRTR_EL2); + write_sysreg_s(ctxt_sys_reg(hctxt, HFGWTR_EL2), SYS_HFGWTR_EL2); + + if (!vcpu_has_nv(vcpu) || is_hyp_ctxt(vcpu)) + return; + + write_sysreg_s(ctxt_sys_reg(hctxt, HFGITR_EL2), SYS_HFGITR_EL2); + write_sysreg_s(ctxt_sys_reg(hctxt, HDFGRTR_EL2), SYS_HDFGRTR_EL2); + write_sysreg_s(ctxt_sys_reg(hctxt, HDFGWTR_EL2), SYS_HDFGWTR_EL2); } static inline void __activate_traps_common(struct kvm_vcpu *vcpu) @@ -145,8 +197,21 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu) vcpu->arch.mdcr_el2_host = read_sysreg(mdcr_el2); write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2); - if (__hfgxtr_traps_required()) - __activate_traps_hfgxtr(); + if (cpus_have_final_cap(ARM64_HAS_HCX)) { + u64 hcrx = HCRX_GUEST_FLAGS; + if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) { + u64 clr = 0, set = 0; + + compute_clr_set(vcpu, HCRX_EL2, clr, set); + + hcrx |= set; + hcrx &= ~clr; + } + + write_sysreg_s(hcrx, SYS_HCRX_EL2); + } + + __activate_traps_hfgxtr(vcpu); } static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu) @@ -162,8 +227,10 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu) vcpu_clear_flag(vcpu, PMUSERENR_ON_CPU); } - if (__hfgxtr_traps_required()) - __deactivate_traps_hfgxtr(); + if (cpus_have_final_cap(ARM64_HAS_HCX)) + write_sysreg_s(HCRX_HOST_FLAGS, SYS_HCRX_EL2); + + __deactivate_traps_hfgxtr(vcpu); } static inline void ___activate_traps(struct kvm_vcpu *vcpu) @@ -177,9 +244,6 @@ static inline void ___activate_traps(struct kvm_vcpu *vcpu) if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN) && (hcr & HCR_VSE)) write_sysreg_s(vcpu->arch.vsesr_el2, SYS_VSESR_EL2); - - if (cpus_have_final_cap(ARM64_HAS_HCX)) - write_sysreg_s(HCRX_GUEST_FLAGS, SYS_HCRX_EL2); } static inline void ___deactivate_traps(struct kvm_vcpu *vcpu) @@ -194,9 +258,6 @@ static inline void ___deactivate_traps(struct kvm_vcpu *vcpu) vcpu->arch.hcr_el2 &= ~HCR_VSE; vcpu->arch.hcr_el2 |= read_sysreg(hcr_el2) & HCR_VSE; } - - if (cpus_have_final_cap(ARM64_HAS_HCX)) - write_sysreg_s(HCRX_HOST_FLAGS, SYS_HCRX_EL2); } static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mm.h b/arch/arm64/kvm/hyp/include/nvhe/mm.h index d5ec972b5c1e..230e4f2527de 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mm.h @@ -26,6 +26,7 @@ int pkvm_create_mappings_locked(void *from, void *to, enum kvm_pgtable_prot prot int __pkvm_create_private_mapping(phys_addr_t phys, size_t size, enum kvm_pgtable_prot prot, unsigned long *haddr); +int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr); int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr); #endif /* __KVM_HYP_MM_H */ diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index a169c619db60..857d9bc04fd4 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -135,6 +135,16 @@ static void handle___kvm_tlb_flush_vmid_ipa_nsh(struct kvm_cpu_context *host_ctx __kvm_tlb_flush_vmid_ipa_nsh(kern_hyp_va(mmu), ipa, level); } +static void +handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); + DECLARE_REG(phys_addr_t, start, host_ctxt, 2); + DECLARE_REG(unsigned long, pages, host_ctxt, 3); + + __kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, pages); +} + static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); @@ -327,6 +337,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa), HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa_nsh), HANDLE_FUNC(__kvm_tlb_flush_vmid), + HANDLE_FUNC(__kvm_tlb_flush_vmid_range), HANDLE_FUNC(__kvm_flush_cpu_context), HANDLE_FUNC(__kvm_timer_set_cntvoff), HANDLE_FUNC(__vgic_v3_read_vmcr), diff --git a/arch/arm64/kvm/hyp/nvhe/mm.c b/arch/arm64/kvm/hyp/nvhe/mm.c index 318298eb3d6b..65a7a186d7b2 100644 --- a/arch/arm64/kvm/hyp/nvhe/mm.c +++ b/arch/arm64/kvm/hyp/nvhe/mm.c @@ -44,6 +44,27 @@ static int __pkvm_create_mappings(unsigned long start, unsigned long size, return err; } +static int __pkvm_alloc_private_va_range(unsigned long start, size_t size) +{ + unsigned long cur; + + hyp_assert_lock_held(&pkvm_pgd_lock); + + if (!start || start < __io_map_base) + return -EINVAL; + + /* The allocated size is always a multiple of PAGE_SIZE */ + cur = start + PAGE_ALIGN(size); + + /* Are we overflowing on the vmemmap ? */ + if (cur > __hyp_vmemmap) + return -ENOMEM; + + __io_map_base = cur; + + return 0; +} + /** * pkvm_alloc_private_va_range - Allocates a private VA range. * @size: The size of the VA range to reserve. @@ -56,27 +77,16 @@ static int __pkvm_create_mappings(unsigned long start, unsigned long size, */ int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr) { - unsigned long base, addr; - int ret = 0; + unsigned long addr; + int ret; hyp_spin_lock(&pkvm_pgd_lock); - - /* Align the allocation based on the order of its size */ - addr = ALIGN(__io_map_base, PAGE_SIZE << get_order(size)); - - /* The allocated size is always a multiple of PAGE_SIZE */ - base = addr + PAGE_ALIGN(size); - - /* Are we overflowing on the vmemmap ? */ - if (!addr || base > __hyp_vmemmap) - ret = -ENOMEM; - else { - __io_map_base = base; - *haddr = addr; - } - + addr = __io_map_base; + ret = __pkvm_alloc_private_va_range(addr, size); hyp_spin_unlock(&pkvm_pgd_lock); + *haddr = addr; + return ret; } @@ -340,6 +350,45 @@ int hyp_create_idmap(u32 hyp_va_bits) return __pkvm_create_mappings(start, end - start, start, PAGE_HYP_EXEC); } +int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr) +{ + unsigned long addr, prev_base; + size_t size; + int ret; + + hyp_spin_lock(&pkvm_pgd_lock); + + prev_base = __io_map_base; + /* + * Efficient stack verification using the PAGE_SHIFT bit implies + * an alignment of our allocation on the order of the size. + */ + size = PAGE_SIZE * 2; + addr = ALIGN(__io_map_base, size); + + ret = __pkvm_alloc_private_va_range(addr, size); + if (!ret) { + /* + * Since the stack grows downwards, map the stack to the page + * at the higher address and leave the lower guard page + * unbacked. + * + * Any valid stack address now has the PAGE_SHIFT bit as 1 + * and addresses corresponding to the guard page have the + * PAGE_SHIFT bit as 0 - this is used for overflow detection. + */ + ret = kvm_pgtable_hyp_map(&pkvm_pgtable, addr + PAGE_SIZE, + PAGE_SIZE, phys, PAGE_HYP); + if (ret) + __io_map_base = prev_base; + } + hyp_spin_unlock(&pkvm_pgd_lock); + + *haddr = addr + size; + + return ret; +} + static void *admit_host_page(void *arg) { struct kvm_hyp_memcache *host_mc = arg; diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index bb98630dfeaf..0d5e0a89ddce 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -113,7 +113,6 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, for (i = 0; i < hyp_nr_cpus; i++) { struct kvm_nvhe_init_params *params = per_cpu_ptr(&kvm_init_params, i); - unsigned long hyp_addr; start = (void *)kern_hyp_va(per_cpu_base[i]); end = start + PAGE_ALIGN(hyp_percpu_size); @@ -121,33 +120,9 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, if (ret) return ret; - /* - * Allocate a contiguous HYP private VA range for the stack - * and guard page. The allocation is also aligned based on - * the order of its size. - */ - ret = pkvm_alloc_private_va_range(PAGE_SIZE * 2, &hyp_addr); + ret = pkvm_create_stack(params->stack_pa, ¶ms->stack_hyp_va); if (ret) return ret; - - /* - * Since the stack grows downwards, map the stack to the page - * at the higher address and leave the lower guard page - * unbacked. - * - * Any valid stack address now has the PAGE_SHIFT bit as 1 - * and addresses corresponding to the guard page have the - * PAGE_SHIFT bit as 0 - this is used for overflow detection. - */ - hyp_spin_lock(&pkvm_pgd_lock); - ret = kvm_pgtable_hyp_map(&pkvm_pgtable, hyp_addr + PAGE_SIZE, - PAGE_SIZE, params->stack_pa, PAGE_HYP); - hyp_spin_unlock(&pkvm_pgd_lock); - if (ret) - return ret; - - /* Update stack_hyp_va to end of the stack's private VA range */ - params->stack_hyp_va = hyp_addr + (2 * PAGE_SIZE); } /* diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index e89a23153e85..c353a06ee7e6 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -236,7 +236,7 @@ static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) * KVM_ARM_VCPU_INIT, however, this is likely not possible for * protected VMs. */ - vcpu->arch.target = -1; + vcpu_clear_flag(vcpu, VCPU_INITIALIZED); *exit_code &= BIT(ARM_EXIT_WITH_SERROR_BIT); *exit_code |= ARM_EXCEPTION_IL; } diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index b9991bbd8e3f..1b265713d6be 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -182,6 +182,36 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu, __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, unsigned long pages) +{ + struct tlb_inv_context cxt; + unsigned long stride; + + /* + * Since the range of addresses may not be mapped at + * the same level, assume the worst case as PAGE_SIZE + */ + stride = PAGE_SIZE; + start = round_down(start, stride); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt, false); + + __flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0); + + dsb(ish); + __tlbi(vmalle1is); + dsb(ish); + isb(); + + /* See the comment in __kvm_tlb_flush_vmid_ipa() */ + if (icache_is_vpipt()) + icache_inval_all_pou(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index f7a93ef29250..f155b8c9e98c 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -670,6 +670,26 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt) return !(pgt->flags & KVM_PGTABLE_S2_NOFWB); } +void kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, size_t size) +{ + unsigned long pages, inval_pages; + + if (!system_supports_tlb_range()) { + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + return; + } + + pages = size >> PAGE_SHIFT; + while (pages > 0) { + inval_pages = min(pages, MAX_TLBI_RANGE_PAGES); + kvm_call_hyp(__kvm_tlb_flush_vmid_range, mmu, addr, inval_pages); + + addr += inval_pages << PAGE_SHIFT; + pages -= inval_pages; + } +} + #define KVM_S2_MEMATTR(pgt, attr) PAGE_S2_MEMATTR(attr, stage2_has_fwb(pgt)) static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot prot, @@ -786,7 +806,8 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, * evicted pte value (if any). */ if (kvm_pte_table(ctx->old, ctx->level)) - kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + kvm_tlb_flush_vmid_range(mmu, ctx->addr, + kvm_granule_size(ctx->level)); else if (kvm_pte_valid(ctx->old)) kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); @@ -810,16 +831,36 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n smp_store_release(ctx->ptep, new); } -static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu, - struct kvm_pgtable_mm_ops *mm_ops) +static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt) { /* - * Clear the existing PTE, and perform break-before-make with - * TLB maintenance if it was valid. + * If FEAT_TLBIRANGE is implemented, defer the individual + * TLB invalidations until the entire walk is finished, and + * then use the range-based TLBI instructions to do the + * invalidations. Condition deferred TLB invalidation on the + * system supporting FWB as the optimization is entirely + * pointless when the unmap walker needs to perform CMOs. + */ + return system_supports_tlb_range() && stage2_has_fwb(pgt); +} + +static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx, + struct kvm_s2_mmu *mmu, + struct kvm_pgtable_mm_ops *mm_ops) +{ + struct kvm_pgtable *pgt = ctx->arg; + + /* + * Clear the existing PTE, and perform break-before-make if it was + * valid. Depending on the system support, defer the TLB maintenance + * for the same until the entire unmap walk is completed. */ if (kvm_pte_valid(ctx->old)) { kvm_clear_pte(ctx->ptep); - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); + + if (!stage2_unmap_defer_tlb_flush(pgt)) + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, + ctx->addr, ctx->level); } mm_ops->put_page(ctx->ptep); @@ -1077,7 +1118,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, * block entry and rely on the remaining portions being faulted * back lazily. */ - stage2_put_pte(ctx, mmu, mm_ops); + stage2_unmap_put_pte(ctx, mmu, mm_ops); if (need_flush && mm_ops->dcache_clean_inval_poc) mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops), @@ -1091,13 +1132,19 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) { + int ret; struct kvm_pgtable_walker walker = { .cb = stage2_unmap_walker, .arg = pgt, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; - return kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker); + if (stage2_unmap_defer_tlb_flush(pgt)) + /* Perform the deferred TLB invalidations */ + kvm_tlb_flush_vmid_range(pgt->mmu, addr, size); + + return ret; } struct stage2_attr_data { diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index e69da550cdc5..46bd43f61d76 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -143,6 +143,34 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu, __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, unsigned long pages) +{ + struct tlb_inv_context cxt; + unsigned long stride; + + /* + * Since the range of addresses may not be mapped at + * the same level, assume the worst case as PAGE_SIZE + */ + stride = PAGE_SIZE; + start = round_down(start, stride); + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + __flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0); + + dsb(ish); + __tlbi(vmalle1is); + dsb(ish); + isb(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d3b4feed460c..587a104f66c3 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -161,15 +161,23 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) } /** - * kvm_flush_remote_tlbs() - flush all VM TLB entries for v7/8 + * kvm_arch_flush_remote_tlbs() - flush all VM TLB entries for v7/8 * @kvm: pointer to kvm structure. * * Interface to HYP function to flush all VM TLB entries */ -void kvm_flush_remote_tlbs(struct kvm *kvm) +int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { - ++kvm->stat.generic.remote_tlb_flush_requests; kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + return 0; +} + +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, + gfn_t gfn, u64 nr_pages) +{ + kvm_tlb_flush_vmid_range(&kvm->arch.mmu, + gfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT); + return 0; } static bool kvm_is_device_pfn(unsigned long pfn) @@ -592,6 +600,25 @@ int create_hyp_mappings(void *from, void *to, enum kvm_pgtable_prot prot) return 0; } +static int __hyp_alloc_private_va_range(unsigned long base) +{ + lockdep_assert_held(&kvm_hyp_pgd_mutex); + + if (!PAGE_ALIGNED(base)) + return -EINVAL; + + /* + * Verify that BIT(VA_BITS - 1) hasn't been flipped by + * allocating the new area, as it would indicate we've + * overflowed the idmap/IO address range. + */ + if ((base ^ io_map_base) & BIT(VA_BITS - 1)) + return -ENOMEM; + + io_map_base = base; + + return 0; +} /** * hyp_alloc_private_va_range - Allocates a private VA range. @@ -612,26 +639,16 @@ int hyp_alloc_private_va_range(size_t size, unsigned long *haddr) /* * This assumes that we have enough space below the idmap - * page to allocate our VAs. If not, the check below will - * kick. A potential alternative would be to detect that - * overflow and switch to an allocation above the idmap. + * page to allocate our VAs. If not, the check in + * __hyp_alloc_private_va_range() will kick. A potential + * alternative would be to detect that overflow and switch + * to an allocation above the idmap. * * The allocated size is always a multiple of PAGE_SIZE. */ - base = io_map_base - PAGE_ALIGN(size); - - /* Align the allocation based on the order of its size */ - base = ALIGN_DOWN(base, PAGE_SIZE << get_order(size)); - - /* - * Verify that BIT(VA_BITS - 1) hasn't been flipped by - * allocating the new area, as it would indicate we've - * overflowed the idmap/IO address range. - */ - if ((base ^ io_map_base) & BIT(VA_BITS - 1)) - ret = -ENOMEM; - else - *haddr = io_map_base = base; + size = PAGE_ALIGN(size); + base = io_map_base - size; + ret = __hyp_alloc_private_va_range(base); mutex_unlock(&kvm_hyp_pgd_mutex); @@ -668,6 +685,48 @@ static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size, return ret; } +int create_hyp_stack(phys_addr_t phys_addr, unsigned long *haddr) +{ + unsigned long base; + size_t size; + int ret; + + mutex_lock(&kvm_hyp_pgd_mutex); + /* + * Efficient stack verification using the PAGE_SHIFT bit implies + * an alignment of our allocation on the order of the size. + */ + size = PAGE_SIZE * 2; + base = ALIGN_DOWN(io_map_base - size, size); + + ret = __hyp_alloc_private_va_range(base); + + mutex_unlock(&kvm_hyp_pgd_mutex); + + if (ret) { + kvm_err("Cannot allocate hyp stack guard page\n"); + return ret; + } + + /* + * Since the stack grows downwards, map the stack to the page + * at the higher address and leave the lower guard page + * unbacked. + * + * Any valid stack address now has the PAGE_SHIFT bit as 1 + * and addresses corresponding to the guard page have the + * PAGE_SHIFT bit as 0 - this is used for overflow detection. + */ + ret = __create_hyp_mappings(base + PAGE_SIZE, PAGE_SIZE, phys_addr, + PAGE_HYP); + if (ret) + kvm_err("Cannot map hyp stack\n"); + + *haddr = base + size; + + return ret; +} + /** * create_hyp_io_mappings - Map IO into both kernel and HYP * @phys_addr: The physical start address which gets mapped @@ -1075,7 +1134,7 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) write_lock(&kvm->mmu_lock); stage2_wp_range(&kvm->arch.mmu, start, end); write_unlock(&kvm->mmu_lock); - kvm_flush_remote_tlbs(kvm); + kvm_flush_remote_tlbs_memslot(kvm, memslot); } /** @@ -1541,7 +1600,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, out_unlock: read_unlock(&kvm->mmu_lock); - kvm_set_pfn_accessed(pfn); kvm_release_pfn_clean(pfn); return ret != -EAGAIN ? ret : 0; } @@ -1721,7 +1779,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - kvm_pfn_t pfn = pte_pfn(range->pte); + kvm_pfn_t pfn = pte_pfn(range->arg.pte); if (!kvm->arch.mmu.pgt) return false; diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 315354d27978..042695a210ce 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -71,8 +71,9 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, break; case SYS_ID_AA64MMFR0_EL1: - /* Hide ECV, FGT, ExS, Secure Memory */ - val &= ~(GENMASK_ULL(63, 43) | + /* Hide ECV, ExS, Secure Memory */ + val &= ~(NV_FTR(MMFR0, ECV) | + NV_FTR(MMFR0, EXS) | NV_FTR(MMFR0, TGRAN4_2) | NV_FTR(MMFR0, TGRAN16_2) | NV_FTR(MMFR0, TGRAN64_2) | @@ -116,7 +117,8 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, break; case SYS_ID_AA64MMFR1_EL1: - val &= (NV_FTR(MMFR1, PAN) | + val &= (NV_FTR(MMFR1, HCX) | + NV_FTR(MMFR1, PAN) | NV_FTR(MMFR1, LO) | NV_FTR(MMFR1, HPDS) | NV_FTR(MMFR1, VH) | @@ -124,8 +126,7 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, break; case SYS_ID_AA64MMFR2_EL1: - val &= ~(NV_FTR(MMFR2, EVT) | - NV_FTR(MMFR2, BBM) | + val &= ~(NV_FTR(MMFR2, BBM) | NV_FTR(MMFR2, TTL) | GENMASK_ULL(47, 44) | NV_FTR(MMFR2, ST) | diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c index 560650972478..6b066e04dc5d 100644 --- a/arch/arm64/kvm/pmu-emul.c +++ b/arch/arm64/kvm/pmu-emul.c @@ -14,6 +14,7 @@ #include #include #include +#include #define PERF_ATTR_CFG1_COUNTER_64BIT BIT(0) @@ -35,12 +36,8 @@ static struct kvm_pmc *kvm_vcpu_idx_to_pmc(struct kvm_vcpu *vcpu, int cnt_idx) return &vcpu->arch.pmu.pmc[cnt_idx]; } -static u32 kvm_pmu_event_mask(struct kvm *kvm) +static u32 __kvm_pmu_event_mask(unsigned int pmuver) { - unsigned int pmuver; - - pmuver = kvm->arch.arm_pmu->pmuver; - switch (pmuver) { case ID_AA64DFR0_EL1_PMUVer_IMP: return GENMASK(9, 0); @@ -55,6 +52,14 @@ static u32 kvm_pmu_event_mask(struct kvm *kvm) } } +static u32 kvm_pmu_event_mask(struct kvm *kvm) +{ + u64 dfr0 = IDREG(kvm, SYS_ID_AA64DFR0_EL1); + u8 pmuver = SYS_FIELD_GET(ID_AA64DFR0_EL1, PMUVer, dfr0); + + return __kvm_pmu_event_mask(pmuver); +} + /** * kvm_pmc_is_64bit - determine if counter is 64bit * @pmc: counter context @@ -672,8 +677,11 @@ void kvm_host_pmu_init(struct arm_pmu *pmu) { struct arm_pmu_entry *entry; - if (pmu->pmuver == ID_AA64DFR0_EL1_PMUVer_NI || - pmu->pmuver == ID_AA64DFR0_EL1_PMUVer_IMP_DEF) + /* + * Check the sanitised PMU version for the system, as KVM does not + * support implementations where PMUv3 exists on a subset of CPUs. + */ + if (!pmuv3_implemented(kvm_arm_pmu_get_pmuver_limit())) return; mutex_lock(&arm_pmus_lock); @@ -750,11 +758,12 @@ u64 kvm_pmu_get_pmceid(struct kvm_vcpu *vcpu, bool pmceid1) } else { val = read_sysreg(pmceid1_el0); /* - * Don't advertise STALL_SLOT, as PMMIR_EL0 is handled + * Don't advertise STALL_SLOT*, as PMMIR_EL0 is handled * as RAZ */ - if (vcpu->kvm->arch.arm_pmu->pmuver >= ID_AA64DFR0_EL1_PMUVer_V3P4) - val &= ~BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT - 32); + val &= ~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT - 32) | + BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND - 32) | + BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND - 32)); base = 32; } @@ -950,11 +959,17 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) return 0; } case KVM_ARM_VCPU_PMU_V3_FILTER: { + u8 pmuver = kvm_arm_pmu_get_pmuver_limit(); struct kvm_pmu_event_filter __user *uaddr; struct kvm_pmu_event_filter filter; int nr_events; - nr_events = kvm_pmu_event_mask(kvm) + 1; + /* + * Allow userspace to specify an event filter for the entire + * event range supported by PMUVer of the hardware, rather + * than the guest's PMUVer for KVM backward compatibility. + */ + nr_events = __kvm_pmu_event_mask(pmuver) + 1; uaddr = (struct kvm_pmu_event_filter __user *)(long)attr->addr; diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c index 121f1a14c829..0eea225fd09a 100644 --- a/arch/arm64/kvm/pmu.c +++ b/arch/arm64/kvm/pmu.c @@ -236,3 +236,21 @@ bool kvm_set_pmuserenr(u64 val) ctxt_sys_reg(hctxt, PMUSERENR_EL0) = val; return true; } + +/* + * If we interrupted the guest to update the host PMU context, make + * sure we re-apply the guest EL0 state. + */ +void kvm_vcpu_pmu_resync_el0(void) +{ + struct kvm_vcpu *vcpu; + + if (!has_vhe() || !in_interrupt()) + return; + + vcpu = kvm_get_running_vcpu(); + if (!vcpu) + return; + + kvm_make_request(KVM_REQ_RESYNC_PMU_EL0, vcpu); +} diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index bc8556b6f459..7a65a35ee4ac 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -248,21 +248,16 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) } } - switch (vcpu->arch.target) { - default: - if (vcpu_el1_is_32bit(vcpu)) { - pstate = VCPU_RESET_PSTATE_SVC; - } else if (vcpu_has_nv(vcpu)) { - pstate = VCPU_RESET_PSTATE_EL2; - } else { - pstate = VCPU_RESET_PSTATE_EL1; - } + if (vcpu_el1_is_32bit(vcpu)) + pstate = VCPU_RESET_PSTATE_SVC; + else if (vcpu_has_nv(vcpu)) + pstate = VCPU_RESET_PSTATE_EL2; + else + pstate = VCPU_RESET_PSTATE_EL1; - if (kvm_vcpu_has_pmu(vcpu) && !kvm_arm_support_pmu_v3()) { - ret = -EINVAL; - goto out; - } - break; + if (kvm_vcpu_has_pmu(vcpu) && !kvm_arm_support_pmu_v3()) { + ret = -EINVAL; + goto out; } /* Reset core registers */ diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 2ca2973abe66..e92ec810d449 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2151,6 +2151,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CONTEXTIDR_EL1), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 }, { SYS_DESC(SYS_TPIDR_EL1), NULL, reset_unknown, TPIDR_EL1 }, + { SYS_DESC(SYS_ACCDATA_EL1), undef_access }, + { SYS_DESC(SYS_SCXTNUM_EL1), undef_access }, { SYS_DESC(SYS_CNTKCTL_EL1), NULL, reset_val, CNTKCTL_EL1, 0}, @@ -2365,8 +2367,13 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(MDCR_EL2, access_rw, reset_val, 0), EL2_REG(CPTR_EL2, access_rw, reset_val, CPTR_NVHE_EL2_RES1), EL2_REG(HSTR_EL2, access_rw, reset_val, 0), + EL2_REG(HFGRTR_EL2, access_rw, reset_val, 0), + EL2_REG(HFGWTR_EL2, access_rw, reset_val, 0), + EL2_REG(HFGITR_EL2, access_rw, reset_val, 0), EL2_REG(HACR_EL2, access_rw, reset_val, 0), + EL2_REG(HCRX_EL2, access_rw, reset_val, 0), + EL2_REG(TTBR0_EL2, access_rw, reset_val, 0), EL2_REG(TTBR1_EL2, access_rw, reset_val, 0), EL2_REG(TCR_EL2, access_rw, reset_val, TCR_EL2_RES1), @@ -2374,6 +2381,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(VTCR_EL2, access_rw, reset_val, 0), { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, + EL2_REG(HDFGRTR_EL2, access_rw, reset_val, 0), + EL2_REG(HDFGWTR_EL2, access_rw, reset_val, 0), EL2_REG(SPSR_EL2, access_rw, reset_val, 0), EL2_REG(ELR_EL2, access_rw, reset_val, 0), { SYS_DESC(SYS_SP_EL1), access_sp_el1}, @@ -3170,6 +3179,9 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu) trace_kvm_handle_sys_reg(esr); + if (__check_nv_sr_forward(vcpu)) + return 1; + params = esr_sys64_to_params(esr); params.regval = vcpu_get_reg(vcpu, Rt); @@ -3587,5 +3599,8 @@ int __init kvm_sys_reg_table_init(void) if (!first_idreg) return -EINVAL; + if (kvm_get_mode() == KVM_MODE_NV) + return populate_nv_trap_config(); + return 0; } diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h index 6ce5c025218d..8ad53104934d 100644 --- a/arch/arm64/kvm/trace_arm.h +++ b/arch/arm64/kvm/trace_arm.h @@ -364,6 +364,32 @@ TRACE_EVENT(kvm_inject_nested_exception, __entry->hcr_el2) ); +TRACE_EVENT(kvm_forward_sysreg_trap, + TP_PROTO(struct kvm_vcpu *vcpu, u32 sysreg, bool is_read), + TP_ARGS(vcpu, sysreg, is_read), + + TP_STRUCT__entry( + __field(u64, pc) + __field(u32, sysreg) + __field(bool, is_read) + ), + + TP_fast_assign( + __entry->pc = *vcpu_pc(vcpu); + __entry->sysreg = sysreg; + __entry->is_read = is_read; + ), + + TP_printk("%llx %c (%d,%d,%d,%d,%d)", + __entry->pc, + __entry->is_read ? 'R' : 'W', + sys_reg_Op0(__entry->sysreg), + sys_reg_Op1(__entry->sysreg), + sys_reg_CRn(__entry->sysreg), + sys_reg_CRm(__entry->sysreg), + sys_reg_Op2(__entry->sysreg)) +); + #endif /* _TRACE_ARM_ARM64_KVM_H */ #undef TRACE_INCLUDE_PATH diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h index f9923beedd27..0ab09b0d4440 100644 --- a/arch/arm64/kvm/vgic/vgic.h +++ b/arch/arm64/kvm/vgic/vgic.h @@ -199,7 +199,6 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu); void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr); void vgic_v2_clear_lr(struct kvm_vcpu *vcpu, int lr); void vgic_v2_set_underflow(struct kvm_vcpu *vcpu); -void vgic_v2_set_npie(struct kvm_vcpu *vcpu); int vgic_v2_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr *attr); int vgic_v2_dist_uaccess(struct kvm_vcpu *vcpu, bool is_write, int offset, u32 *val); @@ -233,7 +232,6 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu); void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr); void vgic_v3_clear_lr(struct kvm_vcpu *vcpu, int lr); void vgic_v3_set_underflow(struct kvm_vcpu *vcpu); -void vgic_v3_set_npie(struct kvm_vcpu *vcpu); void vgic_v3_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr); void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr); void vgic_v3_enable(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/lib/csum.c b/arch/arm64/lib/csum.c index 78b87a64ca0a..2432683e48a6 100644 --- a/arch/arm64/lib/csum.c +++ b/arch/arm64/lib/csum.c @@ -24,7 +24,7 @@ unsigned int __no_sanitize_address do_csum(const unsigned char *buff, int len) const u64 *ptr; u64 data, sum64 = 0; - if (unlikely(len == 0)) + if (unlikely(len <= 0)) return 0; offset = (unsigned long)buff & 7; diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index c80ed4f3cbce..c3f06fdef609 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -26,6 +26,7 @@ HAS_ECV HAS_ECV_CNTPOFF HAS_EPAN HAS_EVT +HAS_FGT HAS_GENERIC_AUTH HAS_GENERIC_AUTH_ARCH_QARMA3 HAS_GENERIC_AUTH_ARCH_QARMA5 diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index 65866bf819c3..2517ef7c21cf 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -2156,6 +2156,135 @@ Field 1 ICIALLU Field 0 ICIALLUIS EndSysreg +Sysreg HDFGRTR_EL2 3 4 3 1 4 +Field 63 PMBIDR_EL1 +Field 62 nPMSNEVFR_EL1 +Field 61 nBRBDATA +Field 60 nBRBCTL +Field 59 nBRBIDR +Field 58 PMCEIDn_EL0 +Field 57 PMUSERENR_EL0 +Field 56 TRBTRG_EL1 +Field 55 TRBSR_EL1 +Field 54 TRBPTR_EL1 +Field 53 TRBMAR_EL1 +Field 52 TRBLIMITR_EL1 +Field 51 TRBIDR_EL1 +Field 50 TRBBASER_EL1 +Res0 49 +Field 48 TRCVICTLR +Field 47 TRCSTATR +Field 46 TRCSSCSRn +Field 45 TRCSEQSTR +Field 44 TRCPRGCTLR +Field 43 TRCOSLSR +Res0 42 +Field 41 TRCIMSPECn +Field 40 TRCID +Res0 39:38 +Field 37 TRCCNTVRn +Field 36 TRCCLAIM +Field 35 TRCAUXCTLR +Field 34 TRCAUTHSTATUS +Field 33 TRC +Field 32 PMSLATFR_EL1 +Field 31 PMSIRR_EL1 +Field 30 PMSIDR_EL1 +Field 29 PMSICR_EL1 +Field 28 PMSFCR_EL1 +Field 27 PMSEVFR_EL1 +Field 26 PMSCR_EL1 +Field 25 PMBSR_EL1 +Field 24 PMBPTR_EL1 +Field 23 PMBLIMITR_EL1 +Field 22 PMMIR_EL1 +Res0 21:20 +Field 19 PMSELR_EL0 +Field 18 PMOVS +Field 17 PMINTEN +Field 16 PMCNTEN +Field 15 PMCCNTR_EL0 +Field 14 PMCCFILTR_EL0 +Field 13 PMEVTYPERn_EL0 +Field 12 PMEVCNTRn_EL0 +Field 11 OSDLR_EL1 +Field 10 OSECCR_EL1 +Field 9 OSLSR_EL1 +Res0 8 +Field 7 DBGPRCR_EL1 +Field 6 DBGAUTHSTATUS_EL1 +Field 5 DBGCLAIM +Field 4 MDSCR_EL1 +Field 3 DBGWVRn_EL1 +Field 2 DBGWCRn_EL1 +Field 1 DBGBVRn_EL1 +Field 0 DBGBCRn_EL1 +EndSysreg + +Sysreg HDFGWTR_EL2 3 4 3 1 5 +Res0 63 +Field 62 nPMSNEVFR_EL1 +Field 61 nBRBDATA +Field 60 nBRBCTL +Res0 59:58 +Field 57 PMUSERENR_EL0 +Field 56 TRBTRG_EL1 +Field 55 TRBSR_EL1 +Field 54 TRBPTR_EL1 +Field 53 TRBMAR_EL1 +Field 52 TRBLIMITR_EL1 +Res0 51 +Field 50 TRBBASER_EL1 +Field 49 TRFCR_EL1 +Field 48 TRCVICTLR +Res0 47 +Field 46 TRCSSCSRn +Field 45 TRCSEQSTR +Field 44 TRCPRGCTLR +Res0 43 +Field 42 TRCOSLAR +Field 41 TRCIMSPECn +Res0 40:38 +Field 37 TRCCNTVRn +Field 36 TRCCLAIM +Field 35 TRCAUXCTLR +Res0 34 +Field 33 TRC +Field 32 PMSLATFR_EL1 +Field 31 PMSIRR_EL1 +Res0 30 +Field 29 PMSICR_EL1 +Field 28 PMSFCR_EL1 +Field 27 PMSEVFR_EL1 +Field 26 PMSCR_EL1 +Field 25 PMBSR_EL1 +Field 24 PMBPTR_EL1 +Field 23 PMBLIMITR_EL1 +Res0 22 +Field 21 PMCR_EL0 +Field 20 PMSWINC_EL0 +Field 19 PMSELR_EL0 +Field 18 PMOVS +Field 17 PMINTEN +Field 16 PMCNTEN +Field 15 PMCCNTR_EL0 +Field 14 PMCCFILTR_EL0 +Field 13 PMEVTYPERn_EL0 +Field 12 PMEVCNTRn_EL0 +Field 11 OSDLR_EL1 +Field 10 OSECCR_EL1 +Res0 9 +Field 8 OSLAR_EL1 +Field 7 DBGPRCR_EL1 +Res0 6 +Field 5 DBGCLAIM +Field 4 MDSCR_EL1 +Field 3 DBGWVRn_EL1 +Field 2 DBGWCRn_EL1 +Field 1 DBGBVRn_EL1 +Field 0 DBGBCRn_EL1 +EndSysreg + Sysreg ZCR_EL2 3 4 1 2 0 Fields ZCR_ELx EndSysreg diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index ecf282dee513..e14396a2ddcb 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -8,11 +8,13 @@ config LOONGARCH select ACPI_PPTT if ACPI select ACPI_SYSTEM_POWER_STATES_SUPPORT if ACPI select ARCH_BINFMT_ELF_STATE + select ARCH_DISABLE_KASAN_INLINE select ARCH_ENABLE_MEMORY_HOTPLUG select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_FORTIFY_SOURCE + select ARCH_HAS_KCOV select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PTE_SPECIAL @@ -91,6 +93,9 @@ config LOONGARCH select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE + select HAVE_ARCH_KASAN + select HAVE_ARCH_KFENCE + select HAVE_ARCH_KGDB if PERF_EVENTS select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK @@ -115,6 +120,7 @@ config LOONGARCH select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER + select HAVE_GCC_PLUGINS select HAVE_GENERIC_VDSO select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IOREMAP_PROT @@ -254,6 +260,9 @@ config AS_HAS_LSX_EXTENSION config AS_HAS_LASX_EXTENSION def_bool $(as-instr,xvld \$xr0$(comma)\$a0$(comma)0) +config AS_HAS_LBT_EXTENSION + def_bool $(as-instr,movscr2gr \$a0$(comma)\$scr0) + menu "Kernel type and options" source "kernel/Kconfig.hz" @@ -534,6 +543,18 @@ config CPU_HAS_LASX If unsure, say Y. +config CPU_HAS_LBT + bool "Support for the Loongson Binary Translation Extension" + depends on AS_HAS_LBT_EXTENSION + help + Loongson Binary Translation (LBT) introduces 4 scratch registers (SCR0 + to SCR3), x86/ARM eflags (eflags) and x87 fpu stack pointer (ftop). + Enabling this option allows the kernel to allocate and switch registers + specific to LBT. + + If you want to use this feature, such as the Loongson Architecture + Translator (LAT), say Y. + config CPU_HAS_PREFETCH bool default y @@ -638,6 +659,11 @@ config ARCH_MMAP_RND_BITS_MAX config ARCH_SUPPORTS_UPROBES def_bool y +config KASAN_SHADOW_OFFSET + hex + default 0x0 + depends on KASAN + menu "Power management options" config ARCH_SUSPEND_POSSIBLE diff --git a/arch/loongarch/Makefile b/arch/loongarch/Makefile index ef87bab46754..fb0fada43197 100644 --- a/arch/loongarch/Makefile +++ b/arch/loongarch/Makefile @@ -84,7 +84,10 @@ LDFLAGS_vmlinux += -static -pie --no-dynamic-linker -z notext endif cflags-y += $(call cc-option, -mno-check-zero-division) + +ifndef CONFIG_KASAN cflags-y += -fno-builtin-memcpy -fno-builtin-memmove -fno-builtin-memset +endif load-y = 0x9000000000200000 bootvars-y = VMLINUX_LOAD_ADDRESS=$(load-y) diff --git a/arch/loongarch/configs/loongson3_defconfig b/arch/loongarch/configs/loongson3_defconfig index d64849b4cba1..a3b52aaa83b3 100644 --- a/arch/loongarch/configs/loongson3_defconfig +++ b/arch/loongarch/configs/loongson3_defconfig @@ -30,7 +30,6 @@ CONFIG_NAMESPACES=y CONFIG_USER_NS=y CONFIG_CHECKPOINT_RESTORE=y CONFIG_SCHED_AUTOGROUP=y -CONFIG_SYSFS_DEPRECATED=y CONFIG_RELAY=y CONFIG_BLK_DEV_INITRD=y CONFIG_EXPERT=y @@ -47,8 +46,12 @@ CONFIG_SMP=y CONFIG_HOTPLUG_CPU=y CONFIG_NR_CPUS=64 CONFIG_NUMA=y +CONFIG_CPU_HAS_FPU=y +CONFIG_CPU_HAS_LSX=y +CONFIG_CPU_HAS_LASX=y CONFIG_KEXEC=y CONFIG_CRASH_DUMP=y +CONFIG_RANDOMIZE_BASE=y CONFIG_SUSPEND=y CONFIG_HIBERNATION=y CONFIG_ACPI=y @@ -63,6 +66,7 @@ CONFIG_EFI_ZBOOT=y CONFIG_EFI_GENERIC_STUB_INITRD_CMDLINE_LOADER=y CONFIG_EFI_CAPSULE_LOADER=m CONFIG_EFI_TEST=m +CONFIG_JUMP_LABEL=y CONFIG_MODULES=y CONFIG_MODULE_FORCE_LOAD=y CONFIG_MODULE_UNLOAD=y @@ -108,7 +112,12 @@ CONFIG_IP_PNP_BOOTP=y CONFIG_IP_PNP_RARP=y CONFIG_NET_IPIP=m CONFIG_NET_IPGRE_DEMUX=m +CONFIG_NET_IPGRE=m +CONFIG_NET_IPGRE_BROADCAST=y CONFIG_IP_MROUTE=y +CONFIG_IP_MROUTE_MULTIPLE_TABLES=y +CONFIG_IP_PIMSM_V1=y +CONFIG_IP_PIMSM_V2=y CONFIG_INET_ESP=m CONFIG_INET_UDP_DIAG=y CONFIG_TCP_CONG_ADVANCED=y @@ -137,7 +146,6 @@ CONFIG_NFT_MASQ=m CONFIG_NFT_REDIR=m CONFIG_NFT_NAT=m CONFIG_NFT_TUNNEL=m -CONFIG_NFT_OBJREF=m CONFIG_NFT_QUEUE=m CONFIG_NFT_QUOTA=m CONFIG_NFT_REJECT=m @@ -208,7 +216,11 @@ CONFIG_IP_VS=m CONFIG_IP_VS_IPV6=y CONFIG_IP_VS_PROTO_TCP=y CONFIG_IP_VS_PROTO_UDP=y +CONFIG_IP_VS_PROTO_ESP=y +CONFIG_IP_VS_PROTO_AH=y +CONFIG_IP_VS_PROTO_SCTP=y CONFIG_IP_VS_RR=m +CONFIG_IP_VS_WRR=m CONFIG_IP_VS_NFCT=y CONFIG_NF_TABLES_IPV4=y CONFIG_NFT_DUP_IPV4=m @@ -227,7 +239,6 @@ CONFIG_IP_NF_TARGET_MASQUERADE=m CONFIG_IP_NF_TARGET_NETMAP=m CONFIG_IP_NF_TARGET_REDIRECT=m CONFIG_IP_NF_MANGLE=m -CONFIG_IP_NF_TARGET_CLUSTERIP=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_RAW=m @@ -363,6 +374,8 @@ CONFIG_MTD_CFI_AMDSTD=m CONFIG_MTD_CFI_STAA=m CONFIG_MTD_RAM=m CONFIG_MTD_ROM=m +CONFIG_MTD_UBI=m +CONFIG_MTD_UBI_BLOCK=y CONFIG_PARPORT=y CONFIG_PARPORT_PC=y CONFIG_PARPORT_SERIAL=y @@ -370,6 +383,7 @@ CONFIG_PARPORT_PC_FIFO=y CONFIG_ZRAM=m CONFIG_ZRAM_DEF_COMP_ZSTD=y CONFIG_BLK_DEV_LOOP=y +CONFIG_BLK_DEV_DRBD=m CONFIG_BLK_DEV_NBD=m CONFIG_BLK_DEV_RAM=y CONFIG_BLK_DEV_RAM_SIZE=8192 @@ -516,6 +530,8 @@ CONFIG_STMMAC_ETH=y # CONFIG_NET_VENDOR_TEHUTI is not set # CONFIG_NET_VENDOR_TI is not set # CONFIG_NET_VENDOR_VIA is not set +CONFIG_NGBE=y +CONFIG_TXGBE=y # CONFIG_NET_VENDOR_WIZNET is not set # CONFIG_NET_VENDOR_XILINX is not set CONFIG_PPP=m @@ -602,9 +618,15 @@ CONFIG_HW_RANDOM_VIRTIO=m CONFIG_I2C_CHARDEV=y CONFIG_I2C_PIIX4=y CONFIG_I2C_GPIO=y +CONFIG_I2C_LS2X=y CONFIG_SPI=y +CONFIG_SPI_LOONGSON_PCI=m +CONFIG_SPI_LOONGSON_PLATFORM=m +CONFIG_PINCTRL=y +CONFIG_PINCTRL_LOONGSON2=y CONFIG_GPIO_SYSFS=y CONFIG_GPIO_LOONGSON=y +CONFIG_GPIO_LOONGSON_64BIT=y CONFIG_POWER_RESET=y CONFIG_POWER_RESET_RESTART=y CONFIG_POWER_RESET_SYSCON=y @@ -614,6 +636,7 @@ CONFIG_SENSORS_LM75=m CONFIG_SENSORS_LM93=m CONFIG_SENSORS_W83795=m CONFIG_SENSORS_W83627HF=m +CONFIG_LOONGSON2_THERMAL=m CONFIG_RC_CORE=m CONFIG_LIRC=y CONFIG_RC_DECODERS=y @@ -643,6 +666,7 @@ CONFIG_DRM_AMDGPU_USERPTR=y CONFIG_DRM_AST=y CONFIG_DRM_QXL=m CONFIG_DRM_VIRTIO_GPU=m +CONFIG_DRM_LOONGSON=y CONFIG_FB=y CONFIG_FB_EFI=y CONFIG_FB_RADEON=y @@ -712,6 +736,7 @@ CONFIG_UCSI_ACPI=m CONFIG_INFINIBAND=m CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_EFI=y +CONFIG_RTC_DRV_LOONGSON=y CONFIG_DMADEVICES=y CONFIG_UIO=m CONFIG_UIO_PDRV_GENIRQ=m @@ -745,7 +770,9 @@ CONFIG_COMEDI_NI_LABPC_PCI=m CONFIG_COMEDI_NI_PCIDIO=m CONFIG_COMEDI_NI_PCIMIO=m CONFIG_STAGING=y -CONFIG_R8188EU=m +CONFIG_COMMON_CLK_LOONGSON2=y +CONFIG_LOONGSON2_GUTS=y +CONFIG_LOONGSON2_PM=y CONFIG_PM_DEVFREQ=y CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y CONFIG_DEVFREQ_GOV_PERFORMANCE=y @@ -759,10 +786,17 @@ CONFIG_EXT2_FS_SECURITY=y CONFIG_EXT3_FS=y CONFIG_EXT3_FS_POSIX_ACL=y CONFIG_EXT3_FS_SECURITY=y +CONFIG_JFS_FS=m +CONFIG_JFS_POSIX_ACL=y +CONFIG_JFS_SECURITY=y CONFIG_XFS_FS=y CONFIG_XFS_QUOTA=y CONFIG_XFS_POSIX_ACL=y +CONFIG_GFS2_FS=m +CONFIG_GFS2_FS_LOCKING_DLM=y +CONFIG_OCFS2_FS=m CONFIG_BTRFS_FS=y +CONFIG_BTRFS_FS_POSIX_ACL=y CONFIG_FANOTIFY=y CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y CONFIG_QUOTA=y @@ -771,11 +805,14 @@ CONFIG_QFMT_V1=m CONFIG_QFMT_V2=m CONFIG_AUTOFS_FS=y CONFIG_FUSE_FS=m +CONFIG_CUSE=m +CONFIG_VIRTIO_FS=m CONFIG_OVERLAY_FS=y CONFIG_OVERLAY_FS_INDEX=y CONFIG_OVERLAY_FS_XINO_AUTO=y CONFIG_OVERLAY_FS_METACOPY=y CONFIG_FSCACHE=y +CONFIG_CACHEFILES=m CONFIG_ISO9660_FS=y CONFIG_JOLIET=y CONFIG_ZISOFS=y @@ -784,19 +821,42 @@ CONFIG_MSDOS_FS=m CONFIG_VFAT_FS=m CONFIG_FAT_DEFAULT_CODEPAGE=936 CONFIG_FAT_DEFAULT_IOCHARSET="gb2312" +CONFIG_EXFAT_FS=m +CONFIG_NTFS3_FS=m +CONFIG_NTFS3_64BIT_CLUSTER=y +CONFIG_NTFS3_LZX_XPRESS=y CONFIG_PROC_KCORE=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_HUGETLBFS=y CONFIG_CONFIGFS_FS=y +CONFIG_ORANGEFS_FS=m +CONFIG_ECRYPT_FS=m +CONFIG_ECRYPT_FS_MESSAGING=y CONFIG_HFS_FS=m CONFIG_HFSPLUS_FS=m +CONFIG_UBIFS_FS=m +CONFIG_UBIFS_FS_ADVANCED_COMPR=y CONFIG_CRAMFS=m CONFIG_SQUASHFS=y CONFIG_SQUASHFS_XATTR=y CONFIG_SQUASHFS_LZ4=y CONFIG_SQUASHFS_LZO=y CONFIG_SQUASHFS_XZ=y +CONFIG_MINIX_FS=m +CONFIG_ROMFS_FS=m +CONFIG_PSTORE=m +CONFIG_PSTORE_LZO_COMPRESS=m +CONFIG_PSTORE_LZ4_COMPRESS=m +CONFIG_PSTORE_LZ4HC_COMPRESS=m +CONFIG_PSTORE_842_COMPRESS=y +CONFIG_PSTORE_ZSTD_COMPRESS=y +CONFIG_PSTORE_ZSTD_COMPRESS_DEFAULT=y +CONFIG_SYSV_FS=m +CONFIG_UFS_FS=m +CONFIG_EROFS_FS=m +CONFIG_EROFS_FS_ZIP_LZMA=y +CONFIG_EROFS_FS_PCPU_KTHREAD=y CONFIG_NFS_FS=y CONFIG_NFS_V3_ACL=y CONFIG_NFS_V4=y @@ -807,6 +867,10 @@ CONFIG_NFSD=y CONFIG_NFSD_V3_ACL=y CONFIG_NFSD_V4=y CONFIG_NFSD_BLOCKLAYOUT=y +CONFIG_CEPH_FS=m +CONFIG_CEPH_FSCACHE=y +CONFIG_CEPH_FS_POSIX_ACL=y +CONFIG_CEPH_FS_SECURITY_LABEL=y CONFIG_CIFS=m # CONFIG_CIFS_DEBUG is not set CONFIG_9P_FS=y @@ -814,6 +878,7 @@ CONFIG_NLS_CODEPAGE_437=y CONFIG_NLS_CODEPAGE_936=y CONFIG_NLS_ASCII=y CONFIG_NLS_UTF8=y +CONFIG_DLM=m CONFIG_KEY_DH_OPERATIONS=y CONFIG_SECURITY=y CONFIG_SECURITY_SELINUX=y @@ -847,6 +912,7 @@ CONFIG_CRYPTO_USER_API_HASH=m CONFIG_CRYPTO_USER_API_SKCIPHER=m CONFIG_CRYPTO_USER_API_RNG=m CONFIG_CRYPTO_USER_API_AEAD=m +CONFIG_CRYPTO_CRC32_LOONGARCH=m CONFIG_CRYPTO_DEV_VIRTIO=m CONFIG_PRINTK_TIME=y CONFIG_STRIP_ASM_SYMS=y diff --git a/arch/loongarch/include/asm/asm-prototypes.h b/arch/loongarch/include/asm/asm-prototypes.h index ed06d3997420..cf8e1a4e7c19 100644 --- a/arch/loongarch/include/asm/asm-prototypes.h +++ b/arch/loongarch/include/asm/asm-prototypes.h @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include #include +#include #include #include #include diff --git a/arch/loongarch/include/asm/asmmacro.h b/arch/loongarch/include/asm/asmmacro.h index 79e1d53fea89..c9544f358c33 100644 --- a/arch/loongarch/include/asm/asmmacro.h +++ b/arch/loongarch/include/asm/asmmacro.h @@ -10,113 +10,6 @@ #include #include - .macro parse_v var val - \var = \val - .endm - - .macro parse_r var r - \var = -1 - .ifc \r, $r0 - \var = 0 - .endif - .ifc \r, $r1 - \var = 1 - .endif - .ifc \r, $r2 - \var = 2 - .endif - .ifc \r, $r3 - \var = 3 - .endif - .ifc \r, $r4 - \var = 4 - .endif - .ifc \r, $r5 - \var = 5 - .endif - .ifc \r, $r6 - \var = 6 - .endif - .ifc \r, $r7 - \var = 7 - .endif - .ifc \r, $r8 - \var = 8 - .endif - .ifc \r, $r9 - \var = 9 - .endif - .ifc \r, $r10 - \var = 10 - .endif - .ifc \r, $r11 - \var = 11 - .endif - .ifc \r, $r12 - \var = 12 - .endif - .ifc \r, $r13 - \var = 13 - .endif - .ifc \r, $r14 - \var = 14 - .endif - .ifc \r, $r15 - \var = 15 - .endif - .ifc \r, $r16 - \var = 16 - .endif - .ifc \r, $r17 - \var = 17 - .endif - .ifc \r, $r18 - \var = 18 - .endif - .ifc \r, $r19 - \var = 19 - .endif - .ifc \r, $r20 - \var = 20 - .endif - .ifc \r, $r21 - \var = 21 - .endif - .ifc \r, $r22 - \var = 22 - .endif - .ifc \r, $r23 - \var = 23 - .endif - .ifc \r, $r24 - \var = 24 - .endif - .ifc \r, $r25 - \var = 25 - .endif - .ifc \r, $r26 - \var = 26 - .endif - .ifc \r, $r27 - \var = 27 - .endif - .ifc \r, $r28 - \var = 28 - .endif - .ifc \r, $r29 - \var = 29 - .endif - .ifc \r, $r30 - \var = 30 - .endif - .ifc \r, $r31 - \var = 31 - .endif - .iflt \var - .error "Unable to parse register name \r" - .endif - .endm - .macro cpu_save_nonscratch thread stptr.d s0, \thread, THREAD_REG23 stptr.d s1, \thread, THREAD_REG24 @@ -148,12 +41,51 @@ .macro fpu_save_csr thread tmp movfcsr2gr \tmp, fcsr0 - stptr.w \tmp, \thread, THREAD_FCSR + stptr.w \tmp, \thread, THREAD_FCSR +#ifdef CONFIG_CPU_HAS_LBT + /* TM bit is always 0 if LBT not supported */ + andi \tmp, \tmp, FPU_CSR_TM + beqz \tmp, 1f + /* Save FTOP */ + x86mftop \tmp + stptr.w \tmp, \thread, THREAD_FTOP + /* Turn off TM to ensure the order of FPR in memory independent of TM */ + x86clrtm +1: +#endif .endm - .macro fpu_restore_csr thread tmp - ldptr.w \tmp, \thread, THREAD_FCSR - movgr2fcsr fcsr0, \tmp + .macro fpu_restore_csr thread tmp0 tmp1 + ldptr.w \tmp0, \thread, THREAD_FCSR + movgr2fcsr fcsr0, \tmp0 +#ifdef CONFIG_CPU_HAS_LBT + /* TM bit is always 0 if LBT not supported */ + andi \tmp0, \tmp0, FPU_CSR_TM + beqz \tmp0, 2f + /* Restore FTOP */ + ldptr.w \tmp0, \thread, THREAD_FTOP + andi \tmp0, \tmp0, 0x7 + la.pcrel \tmp1, 1f + alsl.d \tmp1, \tmp0, \tmp1, 3 + jr \tmp1 +1: + x86mttop 0 + b 2f + x86mttop 1 + b 2f + x86mttop 2 + b 2f + x86mttop 3 + b 2f + x86mttop 4 + b 2f + x86mttop 5 + b 2f + x86mttop 6 + b 2f + x86mttop 7 +2: +#endif .endm .macro fpu_save_cc thread tmp0 tmp1 @@ -353,7 +285,7 @@ .macro lsx_restore_all thread tmp0 tmp1 lsx_restore_data \thread, \tmp0 fpu_restore_cc \thread, \tmp0, \tmp1 - fpu_restore_csr \thread, \tmp0 + fpu_restore_csr \thread, \tmp0, \tmp1 .endm .macro lsx_save_upper vd base tmp off @@ -563,7 +495,7 @@ .macro lasx_restore_all thread tmp0 tmp1 lasx_restore_data \thread, \tmp0 fpu_restore_cc \thread, \tmp0, \tmp1 - fpu_restore_csr \thread, \tmp0 + fpu_restore_csr \thread, \tmp0, \tmp1 .endm .macro lasx_save_upper xd base tmp off diff --git a/arch/loongarch/include/asm/kasan.h b/arch/loongarch/include/asm/kasan.h new file mode 100644 index 000000000000..deeff8158f45 --- /dev/null +++ b/arch/loongarch/include/asm/kasan.h @@ -0,0 +1,126 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_KASAN_H +#define __ASM_KASAN_H + +#ifndef __ASSEMBLY__ + +#include +#include +#include +#include +#include + +#define __HAVE_ARCH_SHADOW_MAP + +#define KASAN_SHADOW_SCALE_SHIFT 3 +#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) + +#define XRANGE_SHIFT (48) + +/* Valid address length */ +#define XRANGE_SHADOW_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) +/* Used for taking out the valid address */ +#define XRANGE_SHADOW_MASK GENMASK_ULL(XRANGE_SHADOW_SHIFT - 1, 0) +/* One segment whole address space size */ +#define XRANGE_SIZE (XRANGE_SHADOW_MASK + 1) + +/* 64-bit segment value. */ +#define XKPRANGE_UC_SEG (0x8000) +#define XKPRANGE_CC_SEG (0x9000) +#define XKVRANGE_VC_SEG (0xffff) + +/* Cached */ +#define XKPRANGE_CC_START CACHE_BASE +#define XKPRANGE_CC_SIZE XRANGE_SIZE +#define XKPRANGE_CC_KASAN_OFFSET (0) +#define XKPRANGE_CC_SHADOW_SIZE (XKPRANGE_CC_SIZE >> KASAN_SHADOW_SCALE_SHIFT) +#define XKPRANGE_CC_SHADOW_END (XKPRANGE_CC_KASAN_OFFSET + XKPRANGE_CC_SHADOW_SIZE) + +/* UnCached */ +#define XKPRANGE_UC_START UNCACHE_BASE +#define XKPRANGE_UC_SIZE XRANGE_SIZE +#define XKPRANGE_UC_KASAN_OFFSET XKPRANGE_CC_SHADOW_END +#define XKPRANGE_UC_SHADOW_SIZE (XKPRANGE_UC_SIZE >> KASAN_SHADOW_SCALE_SHIFT) +#define XKPRANGE_UC_SHADOW_END (XKPRANGE_UC_KASAN_OFFSET + XKPRANGE_UC_SHADOW_SIZE) + +/* VMALLOC (Cached or UnCached) */ +#define XKVRANGE_VC_START MODULES_VADDR +#define XKVRANGE_VC_SIZE round_up(KFENCE_AREA_END - MODULES_VADDR + 1, PGDIR_SIZE) +#define XKVRANGE_VC_KASAN_OFFSET XKPRANGE_UC_SHADOW_END +#define XKVRANGE_VC_SHADOW_SIZE (XKVRANGE_VC_SIZE >> KASAN_SHADOW_SCALE_SHIFT) +#define XKVRANGE_VC_SHADOW_END (XKVRANGE_VC_KASAN_OFFSET + XKVRANGE_VC_SHADOW_SIZE) + +/* KAsan shadow memory start right after vmalloc. */ +#define KASAN_SHADOW_START round_up(KFENCE_AREA_END, PGDIR_SIZE) +#define KASAN_SHADOW_SIZE (XKVRANGE_VC_SHADOW_END - XKPRANGE_CC_KASAN_OFFSET) +#define KASAN_SHADOW_END round_up(KASAN_SHADOW_START + KASAN_SHADOW_SIZE, PGDIR_SIZE) + +#define XKPRANGE_CC_SHADOW_OFFSET (KASAN_SHADOW_START + XKPRANGE_CC_KASAN_OFFSET) +#define XKPRANGE_UC_SHADOW_OFFSET (KASAN_SHADOW_START + XKPRANGE_UC_KASAN_OFFSET) +#define XKVRANGE_VC_SHADOW_OFFSET (KASAN_SHADOW_START + XKVRANGE_VC_KASAN_OFFSET) + +extern bool kasan_early_stage; +extern unsigned char kasan_early_shadow_page[PAGE_SIZE]; + +#define kasan_arch_is_ready kasan_arch_is_ready +static __always_inline bool kasan_arch_is_ready(void) +{ + return !kasan_early_stage; +} + +static inline void *kasan_mem_to_shadow(const void *addr) +{ + if (!kasan_arch_is_ready()) { + return (void *)(kasan_early_shadow_page); + } else { + unsigned long maddr = (unsigned long)addr; + unsigned long xrange = (maddr >> XRANGE_SHIFT) & 0xffff; + unsigned long offset = 0; + + maddr &= XRANGE_SHADOW_MASK; + switch (xrange) { + case XKPRANGE_CC_SEG: + offset = XKPRANGE_CC_SHADOW_OFFSET; + break; + case XKPRANGE_UC_SEG: + offset = XKPRANGE_UC_SHADOW_OFFSET; + break; + case XKVRANGE_VC_SEG: + offset = XKVRANGE_VC_SHADOW_OFFSET; + break; + default: + WARN_ON(1); + return NULL; + } + + return (void *)((maddr >> KASAN_SHADOW_SCALE_SHIFT) + offset); + } +} + +static inline const void *kasan_shadow_to_mem(const void *shadow_addr) +{ + unsigned long addr = (unsigned long)shadow_addr; + + if (unlikely(addr > KASAN_SHADOW_END) || + unlikely(addr < KASAN_SHADOW_START)) { + WARN_ON(1); + return NULL; + } + + if (addr >= XKVRANGE_VC_SHADOW_OFFSET) + return (void *)(((addr - XKVRANGE_VC_SHADOW_OFFSET) << KASAN_SHADOW_SCALE_SHIFT) + XKVRANGE_VC_START); + else if (addr >= XKPRANGE_UC_SHADOW_OFFSET) + return (void *)(((addr - XKPRANGE_UC_SHADOW_OFFSET) << KASAN_SHADOW_SCALE_SHIFT) + XKPRANGE_UC_START); + else if (addr >= XKPRANGE_CC_SHADOW_OFFSET) + return (void *)(((addr - XKPRANGE_CC_SHADOW_OFFSET) << KASAN_SHADOW_SCALE_SHIFT) + XKPRANGE_CC_START); + else { + WARN_ON(1); + return NULL; + } +} + +void kasan_init(void); +asmlinkage void kasan_early_init(void); + +#endif +#endif diff --git a/arch/loongarch/include/asm/kfence.h b/arch/loongarch/include/asm/kfence.h new file mode 100644 index 000000000000..6c82aea1c993 --- /dev/null +++ b/arch/loongarch/include/asm/kfence.h @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * KFENCE support for LoongArch. + * + * Author: Enze Li + * Copyright (C) 2022-2023 KylinSoft Corporation. + */ + +#ifndef _ASM_LOONGARCH_KFENCE_H +#define _ASM_LOONGARCH_KFENCE_H + +#include +#include +#include + +static inline bool arch_kfence_init_pool(void) +{ + int err; + char *kfence_pool = __kfence_pool; + struct vm_struct *area; + + area = __get_vm_area_caller(KFENCE_POOL_SIZE, VM_IOREMAP, + KFENCE_AREA_START, KFENCE_AREA_END, + __builtin_return_address(0)); + if (!area) + return false; + + __kfence_pool = (char *)area->addr; + err = ioremap_page_range((unsigned long)__kfence_pool, + (unsigned long)__kfence_pool + KFENCE_POOL_SIZE, + virt_to_phys((void *)kfence_pool), PAGE_KERNEL); + if (err) { + free_vm_area(area); + __kfence_pool = kfence_pool; + return false; + } + + return true; +} + +/* Protect the given page and flush TLB. */ +static inline bool kfence_protect_page(unsigned long addr, bool protect) +{ + pte_t *pte = virt_to_kpte(addr); + + if (WARN_ON(!pte) || pte_none(*pte)) + return false; + + if (protect) + set_pte(pte, __pte(pte_val(*pte) & ~(_PAGE_VALID | _PAGE_PRESENT))); + else + set_pte(pte, __pte(pte_val(*pte) | (_PAGE_VALID | _PAGE_PRESENT))); + + preempt_disable(); + local_flush_tlb_one(addr); + preempt_enable(); + + return true; +} + +#endif /* _ASM_LOONGARCH_KFENCE_H */ diff --git a/arch/loongarch/include/asm/kgdb.h b/arch/loongarch/include/asm/kgdb.h new file mode 100644 index 000000000000..2041ae58b161 --- /dev/null +++ b/arch/loongarch/include/asm/kgdb.h @@ -0,0 +1,97 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023 Loongson Technology Corporation Limited + */ + +#ifndef _ASM_LOONGARCH_KGDB_H +#define _ASM_LOONGARCH_KGDB_H + +#define GDB_SIZEOF_REG sizeof(u64) + +/* gdb remote procotol expects the following register layout. */ + +/* + * General purpose registers: + * r0-r31: 64 bit + * orig_a0: 64 bit + * pc : 64 bit + * csr_badvaddr: 64 bit + */ +#define DBG_PT_REGS_BASE 0 +#define DBG_PT_REGS_NUM 35 +#define DBG_PT_REGS_END (DBG_PT_REGS_BASE + DBG_PT_REGS_NUM - 1) + +/* + * Floating point registers: + * f0-f31: 64 bit + */ +#define DBG_FPR_BASE (DBG_PT_REGS_END + 1) +#define DBG_FPR_NUM 32 +#define DBG_FPR_END (DBG_FPR_BASE + DBG_FPR_NUM - 1) + +/* + * Condition Flag registers: + * fcc0-fcc8: 8 bit + */ +#define DBG_FCC_BASE (DBG_FPR_END + 1) +#define DBG_FCC_NUM 8 +#define DBG_FCC_END (DBG_FCC_BASE + DBG_FCC_NUM - 1) + +/* + * Floating-point Control and Status registers: + * fcsr: 32 bit + */ +#define DBG_FCSR_NUM 1 +#define DBG_FCSR (DBG_FCC_END + 1) + +#define DBG_MAX_REG_NUM (DBG_FCSR + 1) + +/* + * Size of I/O buffer for gdb packet. + * considering to hold all register contents, size is set + */ +#define BUFMAX 2048 + +/* + * Number of bytes required for gdb_regs buffer. + * PT_REGS and FPR: 8 bytes; FCSR: 4 bytes; FCC: 1 bytes. + * GDB fails to connect for size beyond this with error + * "'g' packet reply is too long" + */ +#define NUMREGBYTES ((DBG_PT_REGS_NUM + DBG_FPR_NUM) * GDB_SIZEOF_REG + DBG_FCC_NUM * 1 + DBG_FCSR_NUM * 4) + +#define BREAK_INSTR_SIZE 4 +#define CACHE_FLUSH_IS_SAFE 0 + +/* Register numbers of various important registers. */ +enum dbg_loongarch_regnum { + DBG_LOONGARCH_ZERO = 0, + DBG_LOONGARCH_RA, + DBG_LOONGARCH_TP, + DBG_LOONGARCH_SP, + DBG_LOONGARCH_A0, + DBG_LOONGARCH_FP = 22, + DBG_LOONGARCH_S0, + DBG_LOONGARCH_S1, + DBG_LOONGARCH_S2, + DBG_LOONGARCH_S3, + DBG_LOONGARCH_S4, + DBG_LOONGARCH_S5, + DBG_LOONGARCH_S6, + DBG_LOONGARCH_S7, + DBG_LOONGARCH_S8, + DBG_LOONGARCH_ORIG_A0, + DBG_LOONGARCH_PC, + DBG_LOONGARCH_BADV +}; + +void kgdb_breakinst(void); +void arch_kgdb_breakpoint(void); + +#ifdef CONFIG_KGDB +bool kgdb_breakpoint_handler(struct pt_regs *regs); +#else /* !CONFIG_KGDB */ +static inline bool kgdb_breakpoint_handler(struct pt_regs *regs) { return false; } +#endif /* CONFIG_KGDB */ + +#endif /* __ASM_KGDB_H_ */ diff --git a/arch/loongarch/include/asm/lbt.h b/arch/loongarch/include/asm/lbt.h new file mode 100644 index 000000000000..e671978bf552 --- /dev/null +++ b/arch/loongarch/include/asm/lbt.h @@ -0,0 +1,109 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Author: Qi Hu + * Huacai Chen + * Copyright (C) 2020-2023 Loongson Technology Corporation Limited + */ +#ifndef _ASM_LBT_H +#define _ASM_LBT_H + +#include +#include +#include +#include + +extern void _init_lbt(void); +extern void _save_lbt(struct loongarch_lbt *); +extern void _restore_lbt(struct loongarch_lbt *); + +static inline int is_lbt_enabled(void) +{ + if (!cpu_has_lbt) + return 0; + + return (csr_read32(LOONGARCH_CSR_EUEN) & CSR_EUEN_LBTEN) ? + 1 : 0; +} + +static inline int is_lbt_owner(void) +{ + return test_thread_flag(TIF_USEDLBT); +} + +#ifdef CONFIG_CPU_HAS_LBT + +static inline void enable_lbt(void) +{ + if (cpu_has_lbt) + csr_xchg32(CSR_EUEN_LBTEN, CSR_EUEN_LBTEN, LOONGARCH_CSR_EUEN); +} + +static inline void disable_lbt(void) +{ + if (cpu_has_lbt) + csr_xchg32(0, CSR_EUEN_LBTEN, LOONGARCH_CSR_EUEN); +} + +static inline void __own_lbt(void) +{ + enable_lbt(); + set_thread_flag(TIF_USEDLBT); + KSTK_EUEN(current) |= CSR_EUEN_LBTEN; +} + +static inline void own_lbt_inatomic(int restore) +{ + if (cpu_has_lbt && !is_lbt_owner()) { + __own_lbt(); + if (restore) + _restore_lbt(¤t->thread.lbt); + } +} + +static inline void own_lbt(int restore) +{ + preempt_disable(); + own_lbt_inatomic(restore); + preempt_enable(); +} + +static inline void lose_lbt_inatomic(int save, struct task_struct *tsk) +{ + if (cpu_has_lbt && is_lbt_owner()) { + if (save) + _save_lbt(&tsk->thread.lbt); + + disable_lbt(); + clear_tsk_thread_flag(tsk, TIF_USEDLBT); + } + KSTK_EUEN(tsk) &= ~(CSR_EUEN_LBTEN); +} + +static inline void lose_lbt(int save) +{ + preempt_disable(); + lose_lbt_inatomic(save, current); + preempt_enable(); +} + +static inline void init_lbt(void) +{ + __own_lbt(); + _init_lbt(); +} +#else +static inline void own_lbt_inatomic(int restore) {} +static inline void lose_lbt_inatomic(int save, struct task_struct *tsk) {} +static inline void init_lbt(void) {} +static inline void lose_lbt(int save) {} +#endif + +static inline int thread_lbt_context_live(void) +{ + if (!cpu_has_lbt) + return 0; + + return test_thread_flag(TIF_LBT_CTX_LIVE); +} + +#endif /* _ASM_LBT_H */ diff --git a/arch/loongarch/include/asm/loongarch.h b/arch/loongarch/include/asm/loongarch.h index 10748a20a2ab..33531d432b49 100644 --- a/arch/loongarch/include/asm/loongarch.h +++ b/arch/loongarch/include/asm/loongarch.h @@ -12,49 +12,6 @@ #ifndef __ASSEMBLY__ #include -/* - * parse_r var, r - Helper assembler macro for parsing register names. - * - * This converts the register name in $n form provided in \r to the - * corresponding register number, which is assigned to the variable \var. It is - * needed to allow explicit encoding of instructions in inline assembly where - * registers are chosen by the compiler in $n form, allowing us to avoid using - * fixed register numbers. - * - * It also allows newer instructions (not implemented by the assembler) to be - * transparently implemented using assembler macros, instead of needing separate - * cases depending on toolchain support. - * - * Simple usage example: - * __asm__ __volatile__("parse_r addr, %0\n\t" - * "#invtlb op, 0, %0\n\t" - * ".word ((0x6498000) | (addr << 10) | (0 << 5) | op)" - * : "=r" (status); - */ - -/* Match an individual register number and assign to \var */ -#define _IFC_REG(n) \ - ".ifc \\r, $r" #n "\n\t" \ - "\\var = " #n "\n\t" \ - ".endif\n\t" - -__asm__(".macro parse_r var r\n\t" - "\\var = -1\n\t" - _IFC_REG(0) _IFC_REG(1) _IFC_REG(2) _IFC_REG(3) - _IFC_REG(4) _IFC_REG(5) _IFC_REG(6) _IFC_REG(7) - _IFC_REG(8) _IFC_REG(9) _IFC_REG(10) _IFC_REG(11) - _IFC_REG(12) _IFC_REG(13) _IFC_REG(14) _IFC_REG(15) - _IFC_REG(16) _IFC_REG(17) _IFC_REG(18) _IFC_REG(19) - _IFC_REG(20) _IFC_REG(21) _IFC_REG(22) _IFC_REG(23) - _IFC_REG(24) _IFC_REG(25) _IFC_REG(26) _IFC_REG(27) - _IFC_REG(28) _IFC_REG(29) _IFC_REG(30) _IFC_REG(31) - ".iflt \\var\n\t" - ".error \"Unable to parse register name \\r\"\n\t" - ".endif\n\t" - ".endm"); - -#undef _IFC_REG - /* CPUCFG */ #define read_cpucfg(reg) __cpucfg(reg) @@ -1453,6 +1410,10 @@ __BUILD_CSR_OP(tlbidx) #define FPU_CSR_RU 0x200 /* towards +Infinity */ #define FPU_CSR_RD 0x300 /* towards -Infinity */ +/* Bit 6 of FPU Status Register specify the LBT TOP simulation mode */ +#define FPU_CSR_TM_SHIFT 0x6 +#define FPU_CSR_TM (_ULCAST_(1) << FPU_CSR_TM_SHIFT) + #define read_fcsr(source) \ ({ \ unsigned int __res; \ diff --git a/arch/loongarch/include/asm/mmzone.h b/arch/loongarch/include/asm/mmzone.h index fe67d0b4b33d..2b9a90727e19 100644 --- a/arch/loongarch/include/asm/mmzone.h +++ b/arch/loongarch/include/asm/mmzone.h @@ -13,6 +13,4 @@ extern struct pglist_data *node_data[]; #define NODE_DATA(nid) (node_data[(nid)]) -extern void setup_zero_pages(void); - #endif /* _ASM_MMZONE_H_ */ diff --git a/arch/loongarch/include/asm/page.h b/arch/loongarch/include/asm/page.h index 26e8dccb6619..63f137ce82a4 100644 --- a/arch/loongarch/include/asm/page.h +++ b/arch/loongarch/include/asm/page.h @@ -84,7 +84,12 @@ typedef struct { unsigned long pgprot; } pgprot_t; #define sym_to_pfn(x) __phys_to_pfn(__pa_symbol(x)) #define virt_to_pfn(kaddr) PFN_DOWN(PHYSADDR(kaddr)) -#define virt_to_page(kaddr) pfn_to_page(virt_to_pfn(kaddr)) + +#define virt_to_page(kaddr) \ +({ \ + (likely((unsigned long)kaddr < vm_map_base)) ? \ + dmw_virt_to_page((unsigned long)kaddr) : tlb_virt_to_page((unsigned long)kaddr);\ +}) extern int __virt_addr_valid(volatile void *kaddr); #define virt_addr_valid(kaddr) __virt_addr_valid((volatile void *)(kaddr)) diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index 23f5b1107246..79470f0b4f1d 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -94,4 +94,5 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) #endif /* __PAGETABLE_PUD_FOLDED */ +extern pte_t * __init populate_kernel_pte(unsigned long addr); #endif /* _ASM_PGALLOC_H */ diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h index 06963a172319..29d9b12298bc 100644 --- a/arch/loongarch/include/asm/pgtable.h +++ b/arch/loongarch/include/asm/pgtable.h @@ -70,12 +70,9 @@ struct vm_area_struct; * for zero-mapped memory areas etc.. */ -extern unsigned long empty_zero_page; -extern unsigned long zero_page_mask; +extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; -#define ZERO_PAGE(vaddr) \ - (virt_to_page((void *)(empty_zero_page + (((unsigned long)(vaddr)) & zero_page_mask)))) -#define __HAVE_COLOR_ZERO_PAGE +#define ZERO_PAGE(vaddr) virt_to_page(empty_zero_page) /* * TLB refill handlers may also map the vmalloc area into xkvrange. @@ -85,14 +82,30 @@ extern unsigned long zero_page_mask; #define MODULES_VADDR (vm_map_base + PCI_IOSIZE + (2 * PAGE_SIZE)) #define MODULES_END (MODULES_VADDR + SZ_256M) +#ifdef CONFIG_KFENCE +#define KFENCE_AREA_SIZE (((CONFIG_KFENCE_NUM_OBJECTS + 1) * 2 + 2) * PAGE_SIZE) +#else +#define KFENCE_AREA_SIZE 0 +#endif + #define VMALLOC_START MODULES_END + +#ifndef CONFIG_KASAN #define VMALLOC_END \ (vm_map_base + \ - min(PTRS_PER_PGD * PTRS_PER_PUD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE, (1UL << cpu_vabits)) - PMD_SIZE - VMEMMAP_SIZE) + min(PTRS_PER_PGD * PTRS_PER_PUD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE, (1UL << cpu_vabits)) - PMD_SIZE - VMEMMAP_SIZE - KFENCE_AREA_SIZE) +#else +#define VMALLOC_END \ + (vm_map_base + \ + min(PTRS_PER_PGD * PTRS_PER_PUD * PTRS_PER_PMD * PTRS_PER_PTE * PAGE_SIZE, (1UL << cpu_vabits) / 2) - PMD_SIZE - VMEMMAP_SIZE - KFENCE_AREA_SIZE) +#endif #define vmemmap ((struct page *)((VMALLOC_END + PMD_SIZE) & PMD_MASK)) #define VMEMMAP_END ((unsigned long)vmemmap + VMEMMAP_SIZE - 1) +#define KFENCE_AREA_START (VMEMMAP_END + 1) +#define KFENCE_AREA_END (KFENCE_AREA_START + KFENCE_AREA_SIZE - 1) + #define pte_ERROR(e) \ pr_err("%s:%d: bad pte %016lx.\n", __FILE__, __LINE__, pte_val(e)) #ifndef __PAGETABLE_PMD_FOLDED @@ -350,6 +363,9 @@ static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *pt extern pgd_t swapper_pg_dir[]; extern pgd_t invalid_pg_dir[]; +struct page *dmw_virt_to_page(unsigned long kaddr); +struct page *tlb_virt_to_page(unsigned long kaddr); + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -596,6 +612,9 @@ static inline long pmd_protnone(pmd_t pmd) } #endif /* CONFIG_NUMA_BALANCING */ +#define pmd_leaf(pmd) ((pmd_val(pmd) & _PAGE_HUGE) != 0) +#define pud_leaf(pud) ((pud_val(pud) & _PAGE_HUGE) != 0) + /* * We provide our own get_unmapped area to cope with the virtual aliasing * constraints placed on us by the cache architecture. diff --git a/arch/loongarch/include/asm/processor.h b/arch/loongarch/include/asm/processor.h index 636e1c66398c..c3bc44b5f5b3 100644 --- a/arch/loongarch/include/asm/processor.h +++ b/arch/loongarch/include/asm/processor.h @@ -80,11 +80,22 @@ BUILD_FPR_ACCESS(32) BUILD_FPR_ACCESS(64) struct loongarch_fpu { - unsigned int fcsr; uint64_t fcc; /* 8x8 */ + uint32_t fcsr; + uint32_t ftop; union fpureg fpr[NUM_FPU_REGS]; }; +struct loongarch_lbt { + /* Scratch registers */ + unsigned long scr0; + unsigned long scr1; + unsigned long scr2; + unsigned long scr3; + /* Eflags register */ + unsigned long eflags; +}; + #define INIT_CPUMASK { \ {0,} \ } @@ -113,15 +124,6 @@ struct thread_struct { unsigned long csr_ecfg; unsigned long csr_badvaddr; /* Last user fault */ - /* Scratch registers */ - unsigned long scr0; - unsigned long scr1; - unsigned long scr2; - unsigned long scr3; - - /* Eflags register */ - unsigned long eflags; - /* Other stuff associated with the thread. */ unsigned long trap_nr; unsigned long error_code; @@ -133,6 +135,7 @@ struct thread_struct { * context because they are conditionally copied at fork(). */ struct loongarch_fpu fpu FPU_ALIGN; + struct loongarch_lbt lbt; /* Also conditionally copied */ /* Hardware breakpoints pinned to this task. */ struct perf_event *hbp_break[LOONGARCH_MAX_BRP]; @@ -174,8 +177,9 @@ struct thread_struct { * FPU & vector registers \ */ \ .fpu = { \ - .fcsr = 0, \ .fcc = 0, \ + .fcsr = 0, \ + .ftop = 0, \ .fpr = {{{0,},},}, \ }, \ .hbp_break = {0}, \ diff --git a/arch/loongarch/include/asm/setup.h b/arch/loongarch/include/asm/setup.h index be05c0e706a2..a0bc159ce8bd 100644 --- a/arch/loongarch/include/asm/setup.h +++ b/arch/loongarch/include/asm/setup.h @@ -7,6 +7,7 @@ #define _LOONGARCH_SETUP_H #include +#include #include #define VECSIZE 0x200 @@ -33,8 +34,13 @@ extern long __la_abs_end; extern long __rela_dyn_begin; extern long __rela_dyn_end; -extern void * __init relocate_kernel(void); +extern unsigned long __init relocate_kernel(void); #endif +static inline unsigned long kaslr_offset(void) +{ + return (unsigned long)&_text - VMLINUX_LOAD_ADDRESS; +} + #endif /* __SETUP_H */ diff --git a/arch/loongarch/include/asm/stackframe.h b/arch/loongarch/include/asm/stackframe.h index 7df80e6ae9d2..4fb1e6408b98 100644 --- a/arch/loongarch/include/asm/stackframe.h +++ b/arch/loongarch/include/asm/stackframe.h @@ -158,6 +158,10 @@ cfi_st u0, PT_R21, \docfi csrrd u0, PERCPU_BASE_KS 9: +#ifdef CONFIG_KGDB + li.w t0, CSR_CRMD_WE + csrxchg t0, t0, LOONGARCH_CSR_CRMD +#endif .endm .macro SAVE_ALL docfi=0 diff --git a/arch/loongarch/include/asm/string.h b/arch/loongarch/include/asm/string.h index 7b29cc9c70aa..5bb5a90d2681 100644 --- a/arch/loongarch/include/asm/string.h +++ b/arch/loongarch/include/asm/string.h @@ -7,11 +7,31 @@ #define __HAVE_ARCH_MEMSET extern void *memset(void *__s, int __c, size_t __count); +extern void *__memset(void *__s, int __c, size_t __count); #define __HAVE_ARCH_MEMCPY extern void *memcpy(void *__to, __const__ void *__from, size_t __n); +extern void *__memcpy(void *__to, __const__ void *__from, size_t __n); #define __HAVE_ARCH_MEMMOVE extern void *memmove(void *__dest, __const__ void *__src, size_t __n); +extern void *__memmove(void *__dest, __const__ void *__src, size_t __n); + +#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__) + +/* + * For files that are not instrumented (e.g. mm/slub.c) we + * should use not instrumented version of mem* functions. + */ + +#define memset(s, c, n) __memset(s, c, n) +#define memcpy(dst, src, len) __memcpy(dst, src, len) +#define memmove(dst, src, len) __memmove(dst, src, len) + +#ifndef __NO_FORTIFY +#define __NO_FORTIFY /* FORTIFY_SOURCE uses __builtin_memcpy, etc. */ +#endif + +#endif #endif /* _ASM_STRING_H */ diff --git a/arch/loongarch/include/asm/switch_to.h b/arch/loongarch/include/asm/switch_to.h index 24e3094bebab..5b225aff3ba2 100644 --- a/arch/loongarch/include/asm/switch_to.h +++ b/arch/loongarch/include/asm/switch_to.h @@ -7,6 +7,7 @@ #include #include +#include struct task_struct; @@ -34,6 +35,7 @@ extern asmlinkage struct task_struct *__switch_to(struct task_struct *prev, #define switch_to(prev, next, last) \ do { \ lose_fpu_inatomic(1, prev); \ + lose_lbt_inatomic(1, prev); \ hw_breakpoint_thread_switch(next); \ (last) = __switch_to(prev, next, task_thread_info(next), \ __builtin_return_address(0), __builtin_frame_address(0)); \ diff --git a/arch/loongarch/include/asm/thread_info.h b/arch/loongarch/include/asm/thread_info.h index 1a3354ca056e..8cb653d49a54 100644 --- a/arch/loongarch/include/asm/thread_info.h +++ b/arch/loongarch/include/asm/thread_info.h @@ -84,6 +84,8 @@ register unsigned long current_stack_pointer __asm__("$sp"); #define TIF_SINGLESTEP 16 /* Single Step */ #define TIF_LSX_CTX_LIVE 17 /* LSX context must be preserved */ #define TIF_LASX_CTX_LIVE 18 /* LASX context must be preserved */ +#define TIF_USEDLBT 19 /* LBT was used by this task this quantum (SMP) */ +#define TIF_LBT_CTX_LIVE 20 /* LBT context must be preserved */ #define _TIF_SIGPENDING (1< + */ +#ifndef _ASM_LOONGARCH_XOR_H +#define _ASM_LOONGARCH_XOR_H + +#include +#include + +#ifdef CONFIG_CPU_HAS_LSX +static struct xor_block_template xor_block_lsx = { + .name = "lsx", + .do_2 = xor_lsx_2, + .do_3 = xor_lsx_3, + .do_4 = xor_lsx_4, + .do_5 = xor_lsx_5, +}; + +#define XOR_SPEED_LSX() \ + do { \ + if (cpu_has_lsx) \ + xor_speed(&xor_block_lsx); \ + } while (0) +#else /* CONFIG_CPU_HAS_LSX */ +#define XOR_SPEED_LSX() +#endif /* CONFIG_CPU_HAS_LSX */ + +#ifdef CONFIG_CPU_HAS_LASX +static struct xor_block_template xor_block_lasx = { + .name = "lasx", + .do_2 = xor_lasx_2, + .do_3 = xor_lasx_3, + .do_4 = xor_lasx_4, + .do_5 = xor_lasx_5, +}; + +#define XOR_SPEED_LASX() \ + do { \ + if (cpu_has_lasx) \ + xor_speed(&xor_block_lasx); \ + } while (0) +#else /* CONFIG_CPU_HAS_LASX */ +#define XOR_SPEED_LASX() +#endif /* CONFIG_CPU_HAS_LASX */ + +/* + * For grins, also test the generic routines. + * + * More importantly: it cannot be ruled out at this point of time, that some + * future (maybe reduced) models could run the vector algorithms slower than + * the scalar ones, maybe for errata or micro-op reasons. It may be + * appropriate to revisit this after one or two more uarch generations. + */ +#include + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ +do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_8regs_p); \ + xor_speed(&xor_block_32regs); \ + xor_speed(&xor_block_32regs_p); \ + XOR_SPEED_LSX(); \ + XOR_SPEED_LASX(); \ +} while (0) + +#endif /* _ASM_LOONGARCH_XOR_H */ diff --git a/arch/loongarch/include/asm/xor_simd.h b/arch/loongarch/include/asm/xor_simd.h new file mode 100644 index 000000000000..471b96332f38 --- /dev/null +++ b/arch/loongarch/include/asm/xor_simd.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2023 WANG Xuerui + */ +#ifndef _ASM_LOONGARCH_XOR_SIMD_H +#define _ASM_LOONGARCH_XOR_SIMD_H + +#ifdef CONFIG_CPU_HAS_LSX +void xor_lsx_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2); +void xor_lsx_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3); +void xor_lsx_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3, + const unsigned long * __restrict p4); +void xor_lsx_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3, + const unsigned long * __restrict p4, const unsigned long * __restrict p5); +#endif /* CONFIG_CPU_HAS_LSX */ + +#ifdef CONFIG_CPU_HAS_LASX +void xor_lasx_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2); +void xor_lasx_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3); +void xor_lasx_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3, + const unsigned long * __restrict p4); +void xor_lasx_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3, + const unsigned long * __restrict p4, const unsigned long * __restrict p5); +#endif /* CONFIG_CPU_HAS_LASX */ + +#endif /* _ASM_LOONGARCH_XOR_SIMD_H */ diff --git a/arch/loongarch/include/uapi/asm/ptrace.h b/arch/loongarch/include/uapi/asm/ptrace.h index 06e3be52cb04..ac915f841650 100644 --- a/arch/loongarch/include/uapi/asm/ptrace.h +++ b/arch/loongarch/include/uapi/asm/ptrace.h @@ -56,6 +56,12 @@ struct user_lasx_state { uint64_t vregs[32*4]; }; +struct user_lbt_state { + uint64_t scr[4]; + uint32_t eflags; + uint32_t ftop; +}; + struct user_watch_state { uint64_t dbg_info; struct { diff --git a/arch/loongarch/include/uapi/asm/sigcontext.h b/arch/loongarch/include/uapi/asm/sigcontext.h index 4cd7d16f7037..6c22f616b8f1 100644 --- a/arch/loongarch/include/uapi/asm/sigcontext.h +++ b/arch/loongarch/include/uapi/asm/sigcontext.h @@ -59,4 +59,14 @@ struct lasx_context { __u32 fcsr; }; +/* LBT context */ +#define LBT_CTX_MAGIC 0x42540001 +#define LBT_CTX_ALIGN 8 +struct lbt_context { + __u64 regs[4]; + __u32 eflags; + __u32 ftop; +}; + + #endif /* _UAPI_ASM_SIGCONTEXT_H */ diff --git a/arch/loongarch/kernel/Makefile b/arch/loongarch/kernel/Makefile index 8e279f04f9e7..c56ea0b75448 100644 --- a/arch/loongarch/kernel/Makefile +++ b/arch/loongarch/kernel/Makefile @@ -15,6 +15,8 @@ obj-$(CONFIG_EFI) += efi.o obj-$(CONFIG_CPU_HAS_FPU) += fpu.o kfpu.o +obj-$(CONFIG_CPU_HAS_LBT) += lbt.o + obj-$(CONFIG_ARCH_STRICT_ALIGN) += unaligned.o ifdef CONFIG_FUNCTION_TRACER @@ -32,6 +34,12 @@ ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_rethook_trampoline.o = $(CC_FLAGS_FTRACE) endif +KASAN_SANITIZE_efi.o := n +KASAN_SANITIZE_cpu-probe.o := n +KASAN_SANITIZE_traps.o := n +KASAN_SANITIZE_smp.o := n +KASAN_SANITIZE_vdso.o := n + obj-$(CONFIG_MODULES) += module.o module-sections.o obj-$(CONFIG_STACKTRACE) += stacktrace.o @@ -54,6 +62,7 @@ obj-$(CONFIG_UNWINDER_PROLOGUE) += unwind_prologue.o obj-$(CONFIG_PERF_EVENTS) += perf_event.o perf_regs.o obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o +obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_KPROBES) += kprobes.o obj-$(CONFIG_RETHOOK) += rethook.o rethook_trampoline.o obj-$(CONFIG_UPROBES) += uprobes.o diff --git a/arch/loongarch/kernel/asm-offsets.c b/arch/loongarch/kernel/asm-offsets.c index 505e4bf59603..8da0726777ed 100644 --- a/arch/loongarch/kernel/asm-offsets.c +++ b/arch/loongarch/kernel/asm-offsets.c @@ -118,13 +118,6 @@ void output_thread_defines(void) OFFSET(THREAD_CSRECFG, task_struct, thread.csr_ecfg); - OFFSET(THREAD_SCR0, task_struct, thread.scr0); - OFFSET(THREAD_SCR1, task_struct, thread.scr1); - OFFSET(THREAD_SCR2, task_struct, thread.scr2); - OFFSET(THREAD_SCR3, task_struct, thread.scr3); - - OFFSET(THREAD_EFLAGS, task_struct, thread.eflags); - OFFSET(THREAD_FPU, task_struct, thread.fpu); OFFSET(THREAD_BVADDR, task_struct, \ @@ -172,6 +165,17 @@ void output_thread_fpu_defines(void) OFFSET(THREAD_FCSR, loongarch_fpu, fcsr); OFFSET(THREAD_FCC, loongarch_fpu, fcc); + OFFSET(THREAD_FTOP, loongarch_fpu, ftop); + BLANK(); +} + +void output_thread_lbt_defines(void) +{ + OFFSET(THREAD_SCR0, loongarch_lbt, scr0); + OFFSET(THREAD_SCR1, loongarch_lbt, scr1); + OFFSET(THREAD_SCR2, loongarch_lbt, scr2); + OFFSET(THREAD_SCR3, loongarch_lbt, scr3); + OFFSET(THREAD_EFLAGS, loongarch_lbt, eflags); BLANK(); } diff --git a/arch/loongarch/kernel/cpu-probe.c b/arch/loongarch/kernel/cpu-probe.c index e925579c7a71..55320813ee08 100644 --- a/arch/loongarch/kernel/cpu-probe.c +++ b/arch/loongarch/kernel/cpu-probe.c @@ -144,6 +144,20 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |= LOONGARCH_CPU_LVZ; elf_hwcap |= HWCAP_LOONGARCH_LVZ; } +#ifdef CONFIG_CPU_HAS_LBT + if (config & CPUCFG2_X86BT) { + c->options |= LOONGARCH_CPU_LBT_X86; + elf_hwcap |= HWCAP_LOONGARCH_LBT_X86; + } + if (config & CPUCFG2_ARMBT) { + c->options |= LOONGARCH_CPU_LBT_ARM; + elf_hwcap |= HWCAP_LOONGARCH_LBT_ARM; + } + if (config & CPUCFG2_MIPSBT) { + c->options |= LOONGARCH_CPU_LBT_MIPS; + elf_hwcap |= HWCAP_LOONGARCH_LBT_MIPS; + } +#endif config = read_cpucfg(LOONGARCH_CPUCFG6); if (config & CPUCFG6_PMP) diff --git a/arch/loongarch/kernel/entry.S b/arch/loongarch/kernel/entry.S index d737e3cf42d3..65518bb8f472 100644 --- a/arch/loongarch/kernel/entry.S +++ b/arch/loongarch/kernel/entry.S @@ -58,6 +58,11 @@ SYM_FUNC_START(handle_syscall) SAVE_STATIC +#ifdef CONFIG_KGDB + li.w t1, CSR_CRMD_WE + csrxchg t1, t1, LOONGARCH_CSR_CRMD +#endif + move u0, t0 li.d tp, ~_THREAD_MASK and tp, tp, sp diff --git a/arch/loongarch/kernel/fpu.S b/arch/loongarch/kernel/fpu.S index 501094a09f5d..d53ab10f4644 100644 --- a/arch/loongarch/kernel/fpu.S +++ b/arch/loongarch/kernel/fpu.S @@ -22,7 +22,7 @@ .macro EX insn, reg, src, offs .ex\@: \insn \reg, \src, \offs - _asm_extable .ex\@, fault + _asm_extable .ex\@, .L_fpu_fault .endm .macro sc_save_fp base @@ -138,6 +138,13 @@ .macro sc_save_fcsr base, tmp0 movfcsr2gr \tmp0, fcsr0 EX st.w \tmp0, \base, 0 +#if defined(CONFIG_CPU_HAS_LBT) + /* TM bit is always 0 if LBT not supported */ + andi \tmp0, \tmp0, FPU_CSR_TM + beqz \tmp0, 1f + x86clrtm +1: +#endif .endm .macro sc_restore_fcsr base, tmp0 @@ -309,7 +316,7 @@ EXPORT_SYMBOL(_save_fp) */ SYM_FUNC_START(_restore_fp) fpu_restore_double a0 t1 # clobbers t1 - fpu_restore_csr a0 t1 + fpu_restore_csr a0 t1 t2 fpu_restore_cc a0 t1 t2 # clobbers t1, t2 jr ra SYM_FUNC_END(_restore_fp) @@ -514,7 +521,6 @@ SYM_FUNC_START(_restore_lasx_context) jr ra SYM_FUNC_END(_restore_lasx_context) -SYM_FUNC_START(fault) +.L_fpu_fault: li.w a0, -EFAULT # failure jr ra -SYM_FUNC_END(fault) diff --git a/arch/loongarch/kernel/head.S b/arch/loongarch/kernel/head.S index 5e828a8bc0a0..53b883db0786 100644 --- a/arch/loongarch/kernel/head.S +++ b/arch/loongarch/kernel/head.S @@ -95,12 +95,17 @@ SYM_CODE_START(kernel_entry) # kernel entry point PTR_LI sp, (_THREAD_SIZE - PT_SIZE) PTR_ADD sp, sp, tp set_saved_sp sp, t0, t1 -#endif - /* relocate_kernel() returns the new kernel entry point */ - jr a0 - ASM_BUG() + /* Jump to the new kernel: new_pc = current_pc + random_offset */ + pcaddi t0, 0 + add.d t0, t0, a0 + jirl zero, t0, 0xc +#endif /* CONFIG_RANDOMIZE_BASE */ +#endif /* CONFIG_RELOCATABLE */ + +#ifdef CONFIG_KASAN + bl kasan_early_init #endif bl start_kernel diff --git a/arch/loongarch/kernel/kfpu.c b/arch/loongarch/kernel/kfpu.c index 5c46ae8c6cac..ec5b28e570c9 100644 --- a/arch/loongarch/kernel/kfpu.c +++ b/arch/loongarch/kernel/kfpu.c @@ -8,19 +8,40 @@ #include #include +static unsigned int euen_mask = CSR_EUEN_FPEN; + +/* + * The critical section between kernel_fpu_begin() and kernel_fpu_end() + * is non-reentrant. It is the caller's responsibility to avoid reentrance. + * See drivers/gpu/drm/amd/display/amdgpu_dm/dc_fpu.c as an example. + */ static DEFINE_PER_CPU(bool, in_kernel_fpu); +static DEFINE_PER_CPU(unsigned int, euen_current); void kernel_fpu_begin(void) { + unsigned int *euen_curr; + preempt_disable(); WARN_ON(this_cpu_read(in_kernel_fpu)); this_cpu_write(in_kernel_fpu, true); + euen_curr = this_cpu_ptr(&euen_current); - if (!is_fpu_owner()) - enable_fpu(); + *euen_curr = csr_xchg32(euen_mask, euen_mask, LOONGARCH_CSR_EUEN); + +#ifdef CONFIG_CPU_HAS_LASX + if (*euen_curr & CSR_EUEN_LASXEN) + _save_lasx(¤t->thread.fpu); else +#endif +#ifdef CONFIG_CPU_HAS_LSX + if (*euen_curr & CSR_EUEN_LSXEN) + _save_lsx(¤t->thread.fpu); + else +#endif + if (*euen_curr & CSR_EUEN_FPEN) _save_fp(¤t->thread.fpu); write_fcsr(LOONGARCH_FCSR0, 0); @@ -29,15 +50,41 @@ EXPORT_SYMBOL_GPL(kernel_fpu_begin); void kernel_fpu_end(void) { + unsigned int *euen_curr; + WARN_ON(!this_cpu_read(in_kernel_fpu)); - if (!is_fpu_owner()) - disable_fpu(); + euen_curr = this_cpu_ptr(&euen_current); + +#ifdef CONFIG_CPU_HAS_LASX + if (*euen_curr & CSR_EUEN_LASXEN) + _restore_lasx(¤t->thread.fpu); else +#endif +#ifdef CONFIG_CPU_HAS_LSX + if (*euen_curr & CSR_EUEN_LSXEN) + _restore_lsx(¤t->thread.fpu); + else +#endif + if (*euen_curr & CSR_EUEN_FPEN) _restore_fp(¤t->thread.fpu); + *euen_curr = csr_xchg32(*euen_curr, euen_mask, LOONGARCH_CSR_EUEN); + this_cpu_write(in_kernel_fpu, false); preempt_enable(); } EXPORT_SYMBOL_GPL(kernel_fpu_end); + +static int __init init_euen_mask(void) +{ + if (cpu_has_lsx) + euen_mask |= CSR_EUEN_LSXEN; + + if (cpu_has_lasx) + euen_mask |= CSR_EUEN_LASXEN; + + return 0; +} +arch_initcall(init_euen_mask); diff --git a/arch/loongarch/kernel/kgdb.c b/arch/loongarch/kernel/kgdb.c new file mode 100644 index 000000000000..445c452d72a7 --- /dev/null +++ b/arch/loongarch/kernel/kgdb.c @@ -0,0 +1,727 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * LoongArch KGDB support + * + * Copyright (C) 2023 Loongson Technology Corporation Limited + */ + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +int kgdb_watch_activated; +static unsigned int stepped_opcode; +static unsigned long stepped_address; + +struct dbg_reg_def_t dbg_reg_def[DBG_MAX_REG_NUM] = { + { "r0", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[0]) }, + { "r1", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[1]) }, + { "r2", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[2]) }, + { "r3", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[3]) }, + { "r4", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[4]) }, + { "r5", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[5]) }, + { "r6", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[6]) }, + { "r7", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[7]) }, + { "r8", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[8]) }, + { "r9", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[9]) }, + { "r10", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[10]) }, + { "r11", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[11]) }, + { "r12", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[12]) }, + { "r13", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[13]) }, + { "r14", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[14]) }, + { "r15", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[15]) }, + { "r16", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[16]) }, + { "r17", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[17]) }, + { "r18", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[18]) }, + { "r19", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[19]) }, + { "r20", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[20]) }, + { "r21", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[21]) }, + { "r22", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[22]) }, + { "r23", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[23]) }, + { "r24", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[24]) }, + { "r25", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[25]) }, + { "r26", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[26]) }, + { "r27", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[27]) }, + { "r28", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[28]) }, + { "r29", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[29]) }, + { "r30", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[30]) }, + { "r31", GDB_SIZEOF_REG, offsetof(struct pt_regs, regs[31]) }, + { "orig_a0", GDB_SIZEOF_REG, offsetof(struct pt_regs, orig_a0) }, + { "pc", GDB_SIZEOF_REG, offsetof(struct pt_regs, csr_era) }, + { "badv", GDB_SIZEOF_REG, offsetof(struct pt_regs, csr_badvaddr) }, + { "f0", GDB_SIZEOF_REG, 0 }, + { "f1", GDB_SIZEOF_REG, 1 }, + { "f2", GDB_SIZEOF_REG, 2 }, + { "f3", GDB_SIZEOF_REG, 3 }, + { "f4", GDB_SIZEOF_REG, 4 }, + { "f5", GDB_SIZEOF_REG, 5 }, + { "f6", GDB_SIZEOF_REG, 6 }, + { "f7", GDB_SIZEOF_REG, 7 }, + { "f8", GDB_SIZEOF_REG, 8 }, + { "f9", GDB_SIZEOF_REG, 9 }, + { "f10", GDB_SIZEOF_REG, 10 }, + { "f11", GDB_SIZEOF_REG, 11 }, + { "f12", GDB_SIZEOF_REG, 12 }, + { "f13", GDB_SIZEOF_REG, 13 }, + { "f14", GDB_SIZEOF_REG, 14 }, + { "f15", GDB_SIZEOF_REG, 15 }, + { "f16", GDB_SIZEOF_REG, 16 }, + { "f17", GDB_SIZEOF_REG, 17 }, + { "f18", GDB_SIZEOF_REG, 18 }, + { "f19", GDB_SIZEOF_REG, 19 }, + { "f20", GDB_SIZEOF_REG, 20 }, + { "f21", GDB_SIZEOF_REG, 21 }, + { "f22", GDB_SIZEOF_REG, 22 }, + { "f23", GDB_SIZEOF_REG, 23 }, + { "f24", GDB_SIZEOF_REG, 24 }, + { "f25", GDB_SIZEOF_REG, 25 }, + { "f26", GDB_SIZEOF_REG, 26 }, + { "f27", GDB_SIZEOF_REG, 27 }, + { "f28", GDB_SIZEOF_REG, 28 }, + { "f29", GDB_SIZEOF_REG, 29 }, + { "f30", GDB_SIZEOF_REG, 30 }, + { "f31", GDB_SIZEOF_REG, 31 }, + { "fcc0", 1, 0 }, + { "fcc1", 1, 1 }, + { "fcc2", 1, 2 }, + { "fcc3", 1, 3 }, + { "fcc4", 1, 4 }, + { "fcc5", 1, 5 }, + { "fcc6", 1, 6 }, + { "fcc7", 1, 7 }, + { "fcsr", 4, 0 }, +}; + +char *dbg_get_reg(int regno, void *mem, struct pt_regs *regs) +{ + int reg_offset, reg_size; + + if (regno < 0 || regno >= DBG_MAX_REG_NUM) + return NULL; + + reg_offset = dbg_reg_def[regno].offset; + reg_size = dbg_reg_def[regno].size; + + if (reg_offset == -1) + goto out; + + /* Handle general-purpose/orig_a0/pc/badv registers */ + if (regno <= DBG_PT_REGS_END) { + memcpy(mem, (void *)regs + reg_offset, reg_size); + goto out; + } + + if (!(regs->csr_euen & CSR_EUEN_FPEN)) + goto out; + + save_fp(current); + + /* Handle FP registers */ + switch (regno) { + case DBG_FCSR: /* Process the fcsr */ + memcpy(mem, (void *)¤t->thread.fpu.fcsr, reg_size); + break; + case DBG_FCC_BASE ... DBG_FCC_END: /* Process the fcc */ + memcpy(mem, (void *)¤t->thread.fpu.fcc + reg_offset, reg_size); + break; + case DBG_FPR_BASE ... DBG_FPR_END: /* Process the fpr */ + memcpy(mem, (void *)¤t->thread.fpu.fpr[reg_offset], reg_size); + break; + default: + break; + } + +out: + return dbg_reg_def[regno].name; +} + +int dbg_set_reg(int regno, void *mem, struct pt_regs *regs) +{ + int reg_offset, reg_size; + + if (regno < 0 || regno >= DBG_MAX_REG_NUM) + return -EINVAL; + + reg_offset = dbg_reg_def[regno].offset; + reg_size = dbg_reg_def[regno].size; + + if (reg_offset == -1) + return 0; + + /* Handle general-purpose/orig_a0/pc/badv registers */ + if (regno <= DBG_PT_REGS_END) { + memcpy((void *)regs + reg_offset, mem, reg_size); + return 0; + } + + if (!(regs->csr_euen & CSR_EUEN_FPEN)) + return 0; + + /* Handle FP registers */ + switch (regno) { + case DBG_FCSR: /* Process the fcsr */ + memcpy((void *)¤t->thread.fpu.fcsr, mem, reg_size); + break; + case DBG_FCC_BASE ... DBG_FCC_END: /* Process the fcc */ + memcpy((void *)¤t->thread.fpu.fcc + reg_offset, mem, reg_size); + break; + case DBG_FPR_BASE ... DBG_FPR_END: /* Process the fpr */ + memcpy((void *)¤t->thread.fpu.fpr[reg_offset], mem, reg_size); + break; + default: + break; + } + + restore_fp(current); + + return 0; +} + +/* + * Similar to regs_to_gdb_regs() except that process is sleeping and so + * we may not be able to get all the info. + */ +void sleeping_thread_to_gdb_regs(unsigned long *gdb_regs, struct task_struct *p) +{ + /* Initialize to zero */ + memset((char *)gdb_regs, 0, NUMREGBYTES); + + gdb_regs[DBG_LOONGARCH_RA] = p->thread.reg01; + gdb_regs[DBG_LOONGARCH_TP] = (long)p; + gdb_regs[DBG_LOONGARCH_SP] = p->thread.reg03; + + /* S0 - S8 */ + gdb_regs[DBG_LOONGARCH_S0] = p->thread.reg23; + gdb_regs[DBG_LOONGARCH_S1] = p->thread.reg24; + gdb_regs[DBG_LOONGARCH_S2] = p->thread.reg25; + gdb_regs[DBG_LOONGARCH_S3] = p->thread.reg26; + gdb_regs[DBG_LOONGARCH_S4] = p->thread.reg27; + gdb_regs[DBG_LOONGARCH_S5] = p->thread.reg28; + gdb_regs[DBG_LOONGARCH_S6] = p->thread.reg29; + gdb_regs[DBG_LOONGARCH_S7] = p->thread.reg30; + gdb_regs[DBG_LOONGARCH_S8] = p->thread.reg31; + + /* + * PC use return address (RA), i.e. the moment after return from __switch_to() + */ + gdb_regs[DBG_LOONGARCH_PC] = p->thread.reg01; +} + +void kgdb_arch_set_pc(struct pt_regs *regs, unsigned long pc) +{ + regs->csr_era = pc; +} + +void arch_kgdb_breakpoint(void) +{ + __asm__ __volatile__ ( \ + ".globl kgdb_breakinst\n\t" \ + "nop\n" \ + "kgdb_breakinst:\tbreak 2\n\t"); /* BRK_KDB = 2 */ +} + +/* + * Calls linux_debug_hook before the kernel dies. If KGDB is enabled, + * then try to fall into the debugger + */ +static int kgdb_loongarch_notify(struct notifier_block *self, unsigned long cmd, void *ptr) +{ + struct die_args *args = (struct die_args *)ptr; + struct pt_regs *regs = args->regs; + + /* Userspace events, ignore. */ + if (user_mode(regs)) + return NOTIFY_DONE; + + if (!kgdb_io_module_registered) + return NOTIFY_DONE; + + if (atomic_read(&kgdb_active) != -1) + kgdb_nmicallback(smp_processor_id(), regs); + + if (kgdb_handle_exception(args->trapnr, args->signr, cmd, regs)) + return NOTIFY_DONE; + + if (atomic_read(&kgdb_setting_breakpoint)) + if (regs->csr_era == (unsigned long)&kgdb_breakinst) + regs->csr_era += LOONGARCH_INSN_SIZE; + + return NOTIFY_STOP; +} + +bool kgdb_breakpoint_handler(struct pt_regs *regs) +{ + struct die_args args = { + .regs = regs, + .str = "Break", + .err = BRK_KDB, + .trapnr = read_csr_excode(), + .signr = SIGTRAP, + + }; + + return (kgdb_loongarch_notify(NULL, DIE_TRAP, &args) == NOTIFY_STOP) ? true : false; +} + +static struct notifier_block kgdb_notifier = { + .notifier_call = kgdb_loongarch_notify, +}; + +static inline void kgdb_arch_update_addr(struct pt_regs *regs, + char *remcom_in_buffer) +{ + unsigned long addr; + char *ptr; + + ptr = &remcom_in_buffer[1]; + if (kgdb_hex2long(&ptr, &addr)) + regs->csr_era = addr; +} + +/* Calculate the new address for after a step */ +static int get_step_address(struct pt_regs *regs, unsigned long *next_addr) +{ + char cj_val; + unsigned int si, si_l, si_h, rd, rj, cj; + unsigned long pc = instruction_pointer(regs); + union loongarch_instruction *ip = (union loongarch_instruction *)pc; + + if (pc & 3) { + pr_warn("%s: invalid pc 0x%lx\n", __func__, pc); + return -EINVAL; + } + + *next_addr = pc + LOONGARCH_INSN_SIZE; + + si_h = ip->reg0i26_format.immediate_h; + si_l = ip->reg0i26_format.immediate_l; + switch (ip->reg0i26_format.opcode) { + case b_op: + *next_addr = pc + sign_extend64((si_h << 16 | si_l) << 2, 27); + return 0; + case bl_op: + *next_addr = pc + sign_extend64((si_h << 16 | si_l) << 2, 27); + regs->regs[1] = pc + LOONGARCH_INSN_SIZE; + return 0; + } + + rj = ip->reg1i21_format.rj; + cj = (rj & 0x07) + DBG_FCC_BASE; + si_l = ip->reg1i21_format.immediate_l; + si_h = ip->reg1i21_format.immediate_h; + dbg_get_reg(cj, &cj_val, regs); + switch (ip->reg1i21_format.opcode) { + case beqz_op: + if (regs->regs[rj] == 0) + *next_addr = pc + sign_extend64((si_h << 16 | si_l) << 2, 22); + return 0; + case bnez_op: + if (regs->regs[rj] != 0) + *next_addr = pc + sign_extend64((si_h << 16 | si_l) << 2, 22); + return 0; + case bceqz_op: /* bceqz_op = bcnez_op */ + if (((rj & 0x18) == 0x00) && !cj_val) /* bceqz */ + *next_addr = pc + sign_extend64((si_h << 16 | si_l) << 2, 22); + if (((rj & 0x18) == 0x08) && cj_val) /* bcnez */ + *next_addr = pc + sign_extend64((si_h << 16 | si_l) << 2, 22); + return 0; + } + + rj = ip->reg2i16_format.rj; + rd = ip->reg2i16_format.rd; + si = ip->reg2i16_format.immediate; + switch (ip->reg2i16_format.opcode) { + case beq_op: + if (regs->regs[rj] == regs->regs[rd]) + *next_addr = pc + sign_extend64(si << 2, 17); + return 0; + case bne_op: + if (regs->regs[rj] != regs->regs[rd]) + *next_addr = pc + sign_extend64(si << 2, 17); + return 0; + case blt_op: + if ((long)regs->regs[rj] < (long)regs->regs[rd]) + *next_addr = pc + sign_extend64(si << 2, 17); + return 0; + case bge_op: + if ((long)regs->regs[rj] >= (long)regs->regs[rd]) + *next_addr = pc + sign_extend64(si << 2, 17); + return 0; + case bltu_op: + if (regs->regs[rj] < regs->regs[rd]) + *next_addr = pc + sign_extend64(si << 2, 17); + return 0; + case bgeu_op: + if (regs->regs[rj] >= regs->regs[rd]) + *next_addr = pc + sign_extend64(si << 2, 17); + return 0; + case jirl_op: + regs->regs[rd] = pc + LOONGARCH_INSN_SIZE; + *next_addr = regs->regs[rj] + sign_extend64(si << 2, 17); + return 0; + } + + return 0; +} + +static int do_single_step(struct pt_regs *regs) +{ + int error = 0; + unsigned long addr = 0; /* Determine where the target instruction will send us to */ + + error = get_step_address(regs, &addr); + if (error) + return error; + + /* Store the opcode in the stepped address */ + error = get_kernel_nofault(stepped_opcode, (void *)addr); + if (error) + return error; + + stepped_address = addr; + + /* Replace the opcode with the break instruction */ + error = copy_to_kernel_nofault((void *)stepped_address, + arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE); + flush_icache_range(addr, addr + BREAK_INSTR_SIZE); + + if (error) { + stepped_opcode = 0; + stepped_address = 0; + } else { + kgdb_single_step = 1; + atomic_set(&kgdb_cpu_doing_single_step, raw_smp_processor_id()); + } + + return error; +} + +/* Undo a single step */ +static void undo_single_step(struct pt_regs *regs) +{ + if (stepped_opcode) { + copy_to_kernel_nofault((void *)stepped_address, + (void *)&stepped_opcode, BREAK_INSTR_SIZE); + flush_icache_range(stepped_address, stepped_address + BREAK_INSTR_SIZE); + } + + stepped_opcode = 0; + stepped_address = 0; + kgdb_single_step = 0; + atomic_set(&kgdb_cpu_doing_single_step, -1); +} + +int kgdb_arch_handle_exception(int vector, int signo, int err_code, + char *remcom_in_buffer, char *remcom_out_buffer, + struct pt_regs *regs) +{ + int ret = 0; + + undo_single_step(regs); + regs->csr_prmd |= CSR_PRMD_PWE; + + switch (remcom_in_buffer[0]) { + case 'D': + case 'k': + regs->csr_prmd &= ~CSR_PRMD_PWE; + fallthrough; + case 'c': + kgdb_arch_update_addr(regs, remcom_in_buffer); + break; + case 's': + kgdb_arch_update_addr(regs, remcom_in_buffer); + ret = do_single_step(regs); + break; + default: + ret = -1; + } + + return ret; +} + +static struct hw_breakpoint { + unsigned int enabled; + unsigned long addr; + int len; + int type; + struct perf_event * __percpu *pev; +} breakinfo[LOONGARCH_MAX_BRP]; + +static int hw_break_reserve_slot(int breakno) +{ + int cpu, cnt = 0; + struct perf_event **pevent; + + for_each_online_cpu(cpu) { + cnt++; + pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu); + if (dbg_reserve_bp_slot(*pevent)) + goto fail; + } + + return 0; + +fail: + for_each_online_cpu(cpu) { + cnt--; + if (!cnt) + break; + pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu); + dbg_release_bp_slot(*pevent); + } + + return -1; +} + +static int hw_break_release_slot(int breakno) +{ + int cpu; + struct perf_event **pevent; + + if (dbg_is_early) + return 0; + + for_each_online_cpu(cpu) { + pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu); + if (dbg_release_bp_slot(*pevent)) + /* + * The debugger is responsible for handing the retry on + * remove failure. + */ + return -1; + } + + return 0; +} + +static int kgdb_set_hw_break(unsigned long addr, int len, enum kgdb_bptype bptype) +{ + int i; + + for (i = 0; i < LOONGARCH_MAX_BRP; i++) + if (!breakinfo[i].enabled) + break; + + if (i == LOONGARCH_MAX_BRP) + return -1; + + switch (bptype) { + case BP_HARDWARE_BREAKPOINT: + breakinfo[i].type = HW_BREAKPOINT_X; + break; + case BP_READ_WATCHPOINT: + breakinfo[i].type = HW_BREAKPOINT_R; + break; + case BP_WRITE_WATCHPOINT: + breakinfo[i].type = HW_BREAKPOINT_W; + break; + case BP_ACCESS_WATCHPOINT: + breakinfo[i].type = HW_BREAKPOINT_RW; + break; + default: + return -1; + } + + switch (len) { + case 1: + breakinfo[i].len = HW_BREAKPOINT_LEN_1; + break; + case 2: + breakinfo[i].len = HW_BREAKPOINT_LEN_2; + break; + case 4: + breakinfo[i].len = HW_BREAKPOINT_LEN_4; + break; + case 8: + breakinfo[i].len = HW_BREAKPOINT_LEN_8; + break; + default: + return -1; + } + + breakinfo[i].addr = addr; + if (hw_break_reserve_slot(i)) { + breakinfo[i].addr = 0; + return -1; + } + breakinfo[i].enabled = 1; + + return 0; +} + +static int kgdb_remove_hw_break(unsigned long addr, int len, enum kgdb_bptype bptype) +{ + int i; + + for (i = 0; i < LOONGARCH_MAX_BRP; i++) + if (breakinfo[i].addr == addr && breakinfo[i].enabled) + break; + + if (i == LOONGARCH_MAX_BRP) + return -1; + + if (hw_break_release_slot(i)) { + pr_err("Cannot remove hw breakpoint at %lx\n", addr); + return -1; + } + breakinfo[i].enabled = 0; + + return 0; +} + +static void kgdb_disable_hw_break(struct pt_regs *regs) +{ + int i; + int cpu = raw_smp_processor_id(); + struct perf_event *bp; + + for (i = 0; i < LOONGARCH_MAX_BRP; i++) { + if (!breakinfo[i].enabled) + continue; + + bp = *per_cpu_ptr(breakinfo[i].pev, cpu); + if (bp->attr.disabled == 1) + continue; + + arch_uninstall_hw_breakpoint(bp); + bp->attr.disabled = 1; + } + + /* Disable hardware debugging while we are in kgdb */ + csr_xchg32(0, CSR_CRMD_WE, LOONGARCH_CSR_CRMD); +} + +static void kgdb_remove_all_hw_break(void) +{ + int i; + int cpu = raw_smp_processor_id(); + struct perf_event *bp; + + for (i = 0; i < LOONGARCH_MAX_BRP; i++) { + if (!breakinfo[i].enabled) + continue; + + bp = *per_cpu_ptr(breakinfo[i].pev, cpu); + if (!bp->attr.disabled) { + arch_uninstall_hw_breakpoint(bp); + bp->attr.disabled = 1; + continue; + } + + if (hw_break_release_slot(i)) + pr_err("KGDB: hw bpt remove failed %lx\n", breakinfo[i].addr); + breakinfo[i].enabled = 0; + } + + csr_xchg32(0, CSR_CRMD_WE, LOONGARCH_CSR_CRMD); + kgdb_watch_activated = 0; +} + +static void kgdb_correct_hw_break(void) +{ + int i, activated = 0; + + for (i = 0; i < LOONGARCH_MAX_BRP; i++) { + struct perf_event *bp; + int val; + int cpu = raw_smp_processor_id(); + + if (!breakinfo[i].enabled) + continue; + + bp = *per_cpu_ptr(breakinfo[i].pev, cpu); + if (bp->attr.disabled != 1) + continue; + + bp->attr.bp_addr = breakinfo[i].addr; + bp->attr.bp_len = breakinfo[i].len; + bp->attr.bp_type = breakinfo[i].type; + + val = hw_breakpoint_arch_parse(bp, &bp->attr, counter_arch_bp(bp)); + if (val) + return; + + val = arch_install_hw_breakpoint(bp); + if (!val) + bp->attr.disabled = 0; + activated = 1; + } + + csr_xchg32(activated ? CSR_CRMD_WE : 0, CSR_CRMD_WE, LOONGARCH_CSR_CRMD); + kgdb_watch_activated = activated; +} + +const struct kgdb_arch arch_kgdb_ops = { + .gdb_bpt_instr = {0x02, 0x00, break_op >> 1, 0x00}, /* BRK_KDB = 2 */ + .flags = KGDB_HW_BREAKPOINT, + .set_hw_breakpoint = kgdb_set_hw_break, + .remove_hw_breakpoint = kgdb_remove_hw_break, + .disable_hw_break = kgdb_disable_hw_break, + .remove_all_hw_break = kgdb_remove_all_hw_break, + .correct_hw_break = kgdb_correct_hw_break, +}; + +int kgdb_arch_init(void) +{ + return register_die_notifier(&kgdb_notifier); +} + +void kgdb_arch_late(void) +{ + int i, cpu; + struct perf_event_attr attr; + struct perf_event **pevent; + + hw_breakpoint_init(&attr); + + attr.bp_addr = (unsigned long)kgdb_arch_init; + attr.bp_len = HW_BREAKPOINT_LEN_4; + attr.bp_type = HW_BREAKPOINT_W; + attr.disabled = 1; + + for (i = 0; i < LOONGARCH_MAX_BRP; i++) { + if (breakinfo[i].pev) + continue; + + breakinfo[i].pev = register_wide_hw_breakpoint(&attr, NULL, NULL); + if (IS_ERR((void * __force)breakinfo[i].pev)) { + pr_err("kgdb: Could not allocate hw breakpoints.\n"); + breakinfo[i].pev = NULL; + return; + } + + for_each_online_cpu(cpu) { + pevent = per_cpu_ptr(breakinfo[i].pev, cpu); + if (pevent[0]->destroy) { + pevent[0]->destroy = NULL; + release_bp_slot(*pevent); + } + } + } +} + +void kgdb_arch_exit(void) +{ + int i; + + for (i = 0; i < LOONGARCH_MAX_BRP; i++) { + if (breakinfo[i].pev) { + unregister_wide_hw_breakpoint(breakinfo[i].pev); + breakinfo[i].pev = NULL; + } + } + + unregister_die_notifier(&kgdb_notifier); +} diff --git a/arch/loongarch/kernel/lbt.S b/arch/loongarch/kernel/lbt.S new file mode 100644 index 000000000000..9c75120a26d8 --- /dev/null +++ b/arch/loongarch/kernel/lbt.S @@ -0,0 +1,155 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Author: Qi Hu + * Huacai Chen + * + * Copyright (C) 2020-2023 Loongson Technology Corporation Limited + */ +#include +#include +#include +#include +#include +#include + +#define SCR_REG_WIDTH 8 + + .macro EX insn, reg, src, offs +.ex\@: \insn \reg, \src, \offs + _asm_extable .ex\@, .L_lbt_fault + .endm + +/* + * Save a thread's lbt context. + */ +SYM_FUNC_START(_save_lbt) + movscr2gr t1, $scr0 # save scr + stptr.d t1, a0, THREAD_SCR0 + movscr2gr t1, $scr1 + stptr.d t1, a0, THREAD_SCR1 + movscr2gr t1, $scr2 + stptr.d t1, a0, THREAD_SCR2 + movscr2gr t1, $scr3 + stptr.d t1, a0, THREAD_SCR3 + + x86mfflag t1, 0x3f # save eflags + stptr.d t1, a0, THREAD_EFLAGS + jr ra +SYM_FUNC_END(_save_lbt) +EXPORT_SYMBOL(_save_lbt) + +/* + * Restore a thread's lbt context. + */ +SYM_FUNC_START(_restore_lbt) + ldptr.d t1, a0, THREAD_SCR0 # restore scr + movgr2scr $scr0, t1 + ldptr.d t1, a0, THREAD_SCR1 + movgr2scr $scr1, t1 + ldptr.d t1, a0, THREAD_SCR2 + movgr2scr $scr2, t1 + ldptr.d t1, a0, THREAD_SCR3 + movgr2scr $scr3, t1 + + ldptr.d t1, a0, THREAD_EFLAGS # restore eflags + x86mtflag t1, 0x3f + jr ra +SYM_FUNC_END(_restore_lbt) +EXPORT_SYMBOL(_restore_lbt) + +/* + * Load scr/eflag with zero. + */ +SYM_FUNC_START(_init_lbt) + movgr2scr $scr0, zero + movgr2scr $scr1, zero + movgr2scr $scr2, zero + movgr2scr $scr3, zero + + x86mtflag zero, 0x3f + jr ra +SYM_FUNC_END(_init_lbt) + +/* + * a0: scr + * a1: eflag + */ +SYM_FUNC_START(_save_lbt_context) + movscr2gr t1, $scr0 # save scr + EX st.d t1, a0, (0 * SCR_REG_WIDTH) + movscr2gr t1, $scr1 + EX st.d t1, a0, (1 * SCR_REG_WIDTH) + movscr2gr t1, $scr2 + EX st.d t1, a0, (2 * SCR_REG_WIDTH) + movscr2gr t1, $scr3 + EX st.d t1, a0, (3 * SCR_REG_WIDTH) + + x86mfflag t1, 0x3f # save eflags + EX st.w t1, a1, 0 + li.w a0, 0 # success + jr ra +SYM_FUNC_END(_save_lbt_context) + +/* + * a0: scr + * a1: eflag + */ +SYM_FUNC_START(_restore_lbt_context) + EX ld.d t1, a0, (0 * SCR_REG_WIDTH) # restore scr + movgr2scr $scr0, t1 + EX ld.d t1, a0, (1 * SCR_REG_WIDTH) + movgr2scr $scr1, t1 + EX ld.d t1, a0, (2 * SCR_REG_WIDTH) + movgr2scr $scr2, t1 + EX ld.d t1, a0, (3 * SCR_REG_WIDTH) + movgr2scr $scr3, t1 + + EX ld.w t1, a1, 0 # restore eflags + x86mtflag t1, 0x3f + li.w a0, 0 # success + jr ra +SYM_FUNC_END(_restore_lbt_context) + +/* + * a0: ftop + */ +SYM_FUNC_START(_save_ftop_context) + x86mftop t1 + st.w t1, a0, 0 + li.w a0, 0 # success + jr ra +SYM_FUNC_END(_save_ftop_context) + +/* + * a0: ftop + */ +SYM_FUNC_START(_restore_ftop_context) + ld.w t1, a0, 0 + andi t1, t1, 0x7 + la.pcrel a0, 1f + alsl.d a0, t1, a0, 3 + jr a0 +1: + x86mttop 0 + b 2f + x86mttop 1 + b 2f + x86mttop 2 + b 2f + x86mttop 3 + b 2f + x86mttop 4 + b 2f + x86mttop 5 + b 2f + x86mttop 6 + b 2f + x86mttop 7 +2: + li.w a0, 0 # success + jr ra +SYM_FUNC_END(_restore_ftop_context) + +.L_lbt_fault: + li.w a0, -EFAULT # failure + jr ra diff --git a/arch/loongarch/kernel/numa.c b/arch/loongarch/kernel/numa.c index 708665895b47..c7d33c489e04 100644 --- a/arch/loongarch/kernel/numa.c +++ b/arch/loongarch/kernel/numa.c @@ -67,39 +67,7 @@ static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) void __init pcpu_populate_pte(unsigned long addr) { - pgd_t *pgd = pgd_offset_k(addr); - p4d_t *p4d = p4d_offset(pgd, addr); - pud_t *pud; - pmd_t *pmd; - - if (p4d_none(*p4d)) { - pud_t *new; - - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); - pgd_populate(&init_mm, pgd, new); -#ifndef __PAGETABLE_PUD_FOLDED - pud_init(new); -#endif - } - - pud = pud_offset(p4d, addr); - if (pud_none(*pud)) { - pmd_t *new; - - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); - pud_populate(&init_mm, pud, new); -#ifndef __PAGETABLE_PMD_FOLDED - pmd_init(new); -#endif - } - - pmd = pmd_offset(pud, addr); - if (!pmd_present(*pmd)) { - pte_t *new; - - new = memblock_alloc(PAGE_SIZE, PAGE_SIZE); - pmd_populate_kernel(&init_mm, pmd, new); - } + populate_kernel_pte(addr); } void __init setup_per_cpu_areas(void) @@ -470,7 +438,6 @@ void __init mem_init(void) { high_memory = (void *) __va(get_num_physpages() << PAGE_SHIFT); memblock_free_all(); - setup_zero_pages(); /* This comes from node 0 */ } int pcibus_to_node(struct pci_bus *bus) diff --git a/arch/loongarch/kernel/process.c b/arch/loongarch/kernel/process.c index ba457e43f5be..3cb082e0c992 100644 --- a/arch/loongarch/kernel/process.c +++ b/arch/loongarch/kernel/process.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include #include @@ -82,9 +83,11 @@ void start_thread(struct pt_regs *regs, unsigned long pc, unsigned long sp) euen = regs->csr_euen & ~(CSR_EUEN_FPEN); regs->csr_euen = euen; lose_fpu(0); + lose_lbt(0); clear_thread_flag(TIF_LSX_CTX_LIVE); clear_thread_flag(TIF_LASX_CTX_LIVE); + clear_thread_flag(TIF_LBT_CTX_LIVE); clear_used_math(); regs->csr_era = pc; regs->regs[3] = sp; @@ -121,10 +124,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) preempt_enable(); - if (used_math()) - memcpy(dst, src, sizeof(struct task_struct)); - else + if (!used_math()) memcpy(dst, src, offsetof(struct task_struct, thread.fpu.fpr)); + else + memcpy(dst, src, offsetof(struct task_struct, thread.lbt.scr0)); + +#ifdef CONFIG_CPU_HAS_LBT + memcpy(&dst->thread.lbt, &src->thread.lbt, sizeof(struct loongarch_lbt)); +#endif return 0; } @@ -189,8 +196,10 @@ out: ptrace_hw_copy_thread(p); clear_tsk_thread_flag(p, TIF_USEDFPU); clear_tsk_thread_flag(p, TIF_USEDSIMD); + clear_tsk_thread_flag(p, TIF_USEDLBT); clear_tsk_thread_flag(p, TIF_LSX_CTX_LIVE); clear_tsk_thread_flag(p, TIF_LASX_CTX_LIVE); + clear_tsk_thread_flag(p, TIF_LBT_CTX_LIVE); return 0; } diff --git a/arch/loongarch/kernel/ptrace.c b/arch/loongarch/kernel/ptrace.c index f72adbf530c6..c114c5ef1332 100644 --- a/arch/loongarch/kernel/ptrace.c +++ b/arch/loongarch/kernel/ptrace.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include #include @@ -338,6 +339,46 @@ static int simd_set(struct task_struct *target, #endif /* CONFIG_CPU_HAS_LSX */ +#ifdef CONFIG_CPU_HAS_LBT +static int lbt_get(struct task_struct *target, + const struct user_regset *regset, + struct membuf to) +{ + int r; + + r = membuf_write(&to, &target->thread.lbt.scr0, sizeof(target->thread.lbt.scr0)); + r = membuf_write(&to, &target->thread.lbt.scr1, sizeof(target->thread.lbt.scr1)); + r = membuf_write(&to, &target->thread.lbt.scr2, sizeof(target->thread.lbt.scr2)); + r = membuf_write(&to, &target->thread.lbt.scr3, sizeof(target->thread.lbt.scr3)); + r = membuf_write(&to, &target->thread.lbt.eflags, sizeof(u32)); + r = membuf_write(&to, &target->thread.fpu.ftop, sizeof(u32)); + + return r; +} + +static int lbt_set(struct task_struct *target, + const struct user_regset *regset, + unsigned int pos, unsigned int count, + const void *kbuf, const void __user *ubuf) +{ + int err = 0; + const int eflags_start = 4 * sizeof(target->thread.lbt.scr0); + const int ftop_start = eflags_start + sizeof(u32); + + err |= user_regset_copyin(&pos, &count, &kbuf, &ubuf, + &target->thread.lbt.scr0, + 0, 4 * sizeof(target->thread.lbt.scr0)); + err |= user_regset_copyin(&pos, &count, &kbuf, &ubuf, + &target->thread.lbt.eflags, + eflags_start, ftop_start); + err |= user_regset_copyin(&pos, &count, &kbuf, &ubuf, + &target->thread.fpu.ftop, + ftop_start, ftop_start + sizeof(u32)); + + return err; +} +#endif /* CONFIG_CPU_HAS_LBT */ + #ifdef CONFIG_HAVE_HW_BREAKPOINT /* @@ -802,6 +843,9 @@ enum loongarch_regset { #ifdef CONFIG_CPU_HAS_LASX REGSET_LASX, #endif +#ifdef CONFIG_CPU_HAS_LBT + REGSET_LBT, +#endif #ifdef CONFIG_HAVE_HW_BREAKPOINT REGSET_HW_BREAK, REGSET_HW_WATCH, @@ -853,6 +897,16 @@ static const struct user_regset loongarch64_regsets[] = { .set = simd_set, }, #endif +#ifdef CONFIG_CPU_HAS_LBT + [REGSET_LBT] = { + .core_note_type = NT_LOONGARCH_LBT, + .n = 5, + .size = sizeof(u64), + .align = sizeof(u64), + .regset_get = lbt_get, + .set = lbt_set, + }, +#endif #ifdef CONFIG_HAVE_HW_BREAKPOINT [REGSET_HW_BREAK] = { .core_note_type = NT_LOONGARCH_HW_BREAK, diff --git a/arch/loongarch/kernel/relocate.c b/arch/loongarch/kernel/relocate.c index 01f94d1e3edf..6c3eff9af9fb 100644 --- a/arch/loongarch/kernel/relocate.c +++ b/arch/loongarch/kernel/relocate.c @@ -157,12 +157,11 @@ static inline void __init update_reloc_offset(unsigned long *addr, long random_o *new_addr = (unsigned long)reloc_offset; } -void * __init relocate_kernel(void) +unsigned long __init relocate_kernel(void) { unsigned long kernel_length; unsigned long random_offset = 0; void *location_new = _text; /* Default to original kernel start */ - void *kernel_entry = start_kernel; /* Default to original kernel entry point */ char *cmdline = early_ioremap(fw_arg1, COMMAND_LINE_SIZE); /* Boot command line is passed in fw_arg1 */ strscpy(boot_command_line, cmdline, COMMAND_LINE_SIZE); @@ -190,9 +189,6 @@ void * __init relocate_kernel(void) reloc_offset += random_offset; - /* Return the new kernel's entry point */ - kernel_entry = RELOCATED_KASLR(start_kernel); - /* The current thread is now within the relocated kernel */ __current_thread_info = RELOCATED_KASLR(__current_thread_info); @@ -204,7 +200,7 @@ void * __init relocate_kernel(void) relocate_absolute(random_offset); - return kernel_entry; + return random_offset; } /* diff --git a/arch/loongarch/kernel/setup.c b/arch/loongarch/kernel/setup.c index 9d830ab4e302..7783f0a3d742 100644 --- a/arch/loongarch/kernel/setup.c +++ b/arch/loongarch/kernel/setup.c @@ -626,4 +626,8 @@ void __init setup_arch(char **cmdline_p) #endif paging_init(); + +#ifdef CONFIG_KASAN + kasan_init(); +#endif } diff --git a/arch/loongarch/kernel/signal.c b/arch/loongarch/kernel/signal.c index ceb899366c0a..504fdfe85203 100644 --- a/arch/loongarch/kernel/signal.c +++ b/arch/loongarch/kernel/signal.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -44,6 +45,9 @@ /* Make sure we will not lose FPU ownership */ #define lock_fpu_owner() ({ preempt_disable(); pagefault_disable(); }) #define unlock_fpu_owner() ({ pagefault_enable(); preempt_enable(); }) +/* Make sure we will not lose LBT ownership */ +#define lock_lbt_owner() ({ preempt_disable(); pagefault_disable(); }) +#define unlock_lbt_owner() ({ pagefault_enable(); preempt_enable(); }) /* Assembly functions to move context to/from the FPU */ extern asmlinkage int @@ -59,6 +63,13 @@ _save_lasx_context(void __user *fpregs, void __user *fcc, void __user *fcsr); extern asmlinkage int _restore_lasx_context(void __user *fpregs, void __user *fcc, void __user *fcsr); +#ifdef CONFIG_CPU_HAS_LBT +extern asmlinkage int _save_lbt_context(void __user *regs, void __user *eflags); +extern asmlinkage int _restore_lbt_context(void __user *regs, void __user *eflags); +extern asmlinkage int _save_ftop_context(void __user *ftop); +extern asmlinkage int _restore_ftop_context(void __user *ftop); +#endif + struct rt_sigframe { struct siginfo rs_info; struct ucontext rs_uctx; @@ -75,6 +86,7 @@ struct extctx_layout { struct _ctx_layout fpu; struct _ctx_layout lsx; struct _ctx_layout lasx; + struct _ctx_layout lbt; struct _ctx_layout end; }; @@ -215,6 +227,52 @@ static int copy_lasx_from_sigcontext(struct lasx_context __user *ctx) return err; } +#ifdef CONFIG_CPU_HAS_LBT +static int copy_lbt_to_sigcontext(struct lbt_context __user *ctx) +{ + int err = 0; + uint64_t __user *regs = (uint64_t *)&ctx->regs; + uint32_t __user *eflags = (uint32_t *)&ctx->eflags; + + err |= __put_user(current->thread.lbt.scr0, ®s[0]); + err |= __put_user(current->thread.lbt.scr1, ®s[1]); + err |= __put_user(current->thread.lbt.scr2, ®s[2]); + err |= __put_user(current->thread.lbt.scr3, ®s[3]); + err |= __put_user(current->thread.lbt.eflags, eflags); + + return err; +} + +static int copy_lbt_from_sigcontext(struct lbt_context __user *ctx) +{ + int err = 0; + uint64_t __user *regs = (uint64_t *)&ctx->regs; + uint32_t __user *eflags = (uint32_t *)&ctx->eflags; + + err |= __get_user(current->thread.lbt.scr0, ®s[0]); + err |= __get_user(current->thread.lbt.scr1, ®s[1]); + err |= __get_user(current->thread.lbt.scr2, ®s[2]); + err |= __get_user(current->thread.lbt.scr3, ®s[3]); + err |= __get_user(current->thread.lbt.eflags, eflags); + + return err; +} + +static int copy_ftop_to_sigcontext(struct lbt_context __user *ctx) +{ + uint32_t __user *ftop = &ctx->ftop; + + return __put_user(current->thread.fpu.ftop, ftop); +} + +static int copy_ftop_from_sigcontext(struct lbt_context __user *ctx) +{ + uint32_t __user *ftop = &ctx->ftop; + + return __get_user(current->thread.fpu.ftop, ftop); +} +#endif + /* * Wrappers for the assembly _{save,restore}_fp_context functions. */ @@ -272,6 +330,41 @@ static int restore_hw_lasx_context(struct lasx_context __user *ctx) return _restore_lasx_context(regs, fcc, fcsr); } +/* + * Wrappers for the assembly _{save,restore}_lbt_context functions. + */ +#ifdef CONFIG_CPU_HAS_LBT +static int save_hw_lbt_context(struct lbt_context __user *ctx) +{ + uint64_t __user *regs = (uint64_t *)&ctx->regs; + uint32_t __user *eflags = (uint32_t *)&ctx->eflags; + + return _save_lbt_context(regs, eflags); +} + +static int restore_hw_lbt_context(struct lbt_context __user *ctx) +{ + uint64_t __user *regs = (uint64_t *)&ctx->regs; + uint32_t __user *eflags = (uint32_t *)&ctx->eflags; + + return _restore_lbt_context(regs, eflags); +} + +static int save_hw_ftop_context(struct lbt_context __user *ctx) +{ + uint32_t __user *ftop = &ctx->ftop; + + return _save_ftop_context(ftop); +} + +static int restore_hw_ftop_context(struct lbt_context __user *ctx) +{ + uint32_t __user *ftop = &ctx->ftop; + + return _restore_ftop_context(ftop); +} +#endif + static int fcsr_pending(unsigned int __user *fcsr) { int err, sig = 0; @@ -519,6 +612,77 @@ static int protected_restore_lasx_context(struct extctx_layout *extctx) return err ?: sig; } +#ifdef CONFIG_CPU_HAS_LBT +static int protected_save_lbt_context(struct extctx_layout *extctx) +{ + int err = 0; + struct sctx_info __user *info = extctx->lbt.addr; + struct lbt_context __user *lbt_ctx = + (struct lbt_context *)get_ctx_through_ctxinfo(info); + uint64_t __user *regs = (uint64_t *)&lbt_ctx->regs; + uint32_t __user *eflags = (uint32_t *)&lbt_ctx->eflags; + + while (1) { + lock_lbt_owner(); + if (is_lbt_owner()) + err |= save_hw_lbt_context(lbt_ctx); + else + err |= copy_lbt_to_sigcontext(lbt_ctx); + if (is_fpu_owner()) + err |= save_hw_ftop_context(lbt_ctx); + else + err |= copy_ftop_to_sigcontext(lbt_ctx); + unlock_lbt_owner(); + + err |= __put_user(LBT_CTX_MAGIC, &info->magic); + err |= __put_user(extctx->lbt.size, &info->size); + + if (likely(!err)) + break; + /* Touch the LBT context and try again */ + err = __put_user(0, ®s[0]) | __put_user(0, eflags); + + if (err) + return err; + } + + return err; +} + +static int protected_restore_lbt_context(struct extctx_layout *extctx) +{ + int err = 0, tmp __maybe_unused; + struct sctx_info __user *info = extctx->lbt.addr; + struct lbt_context __user *lbt_ctx = + (struct lbt_context *)get_ctx_through_ctxinfo(info); + uint64_t __user *regs = (uint64_t *)&lbt_ctx->regs; + uint32_t __user *eflags = (uint32_t *)&lbt_ctx->eflags; + + while (1) { + lock_lbt_owner(); + if (is_lbt_owner()) + err |= restore_hw_lbt_context(lbt_ctx); + else + err |= copy_lbt_from_sigcontext(lbt_ctx); + if (is_fpu_owner()) + err |= restore_hw_ftop_context(lbt_ctx); + else + err |= copy_ftop_from_sigcontext(lbt_ctx); + unlock_lbt_owner(); + + if (likely(!err)) + break; + /* Touch the LBT context and try again */ + err = __get_user(tmp, ®s[0]) | __get_user(tmp, eflags); + + if (err) + return err; + } + + return err; +} +#endif + static int setup_sigcontext(struct pt_regs *regs, struct sigcontext __user *sc, struct extctx_layout *extctx) { @@ -539,6 +703,11 @@ static int setup_sigcontext(struct pt_regs *regs, struct sigcontext __user *sc, else if (extctx->fpu.addr) err |= protected_save_fpu_context(extctx); +#ifdef CONFIG_CPU_HAS_LBT + if (extctx->lbt.addr) + err |= protected_save_lbt_context(extctx); +#endif + /* Set the "end" magic */ info = (struct sctx_info *)extctx->end.addr; err |= __put_user(0, &info->magic); @@ -584,6 +753,13 @@ static int parse_extcontext(struct sigcontext __user *sc, struct extctx_layout * extctx->lasx.addr = info; break; + case LBT_CTX_MAGIC: + if (size < (sizeof(struct sctx_info) + + sizeof(struct lbt_context))) + goto invalid; + extctx->lbt.addr = info; + break; + default: goto invalid; } @@ -636,6 +812,11 @@ static int restore_sigcontext(struct pt_regs *regs, struct sigcontext __user *sc else if (extctx.fpu.addr) err |= protected_restore_fpu_context(&extctx); +#ifdef CONFIG_CPU_HAS_LBT + if (extctx.lbt.addr) + err |= protected_restore_lbt_context(&extctx); +#endif + bad: return err; } @@ -700,6 +881,13 @@ static unsigned long setup_extcontext(struct extctx_layout *extctx, unsigned lon sizeof(struct fpu_context), FPU_CTX_ALIGN, new_sp); } +#ifdef CONFIG_CPU_HAS_LBT + if (cpu_has_lbt && thread_lbt_context_live()) { + new_sp = extframe_alloc(extctx, &extctx->lbt, + sizeof(struct lbt_context), LBT_CTX_ALIGN, new_sp); + } +#endif + return new_sp; } diff --git a/arch/loongarch/kernel/stacktrace.c b/arch/loongarch/kernel/stacktrace.c index 2463d2fea21f..92270f14db94 100644 --- a/arch/loongarch/kernel/stacktrace.c +++ b/arch/loongarch/kernel/stacktrace.c @@ -18,17 +18,19 @@ void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie, struct pt_regs dummyregs; struct unwind_state state; - regs = &dummyregs; + if (!regs) { + regs = &dummyregs; - if (task == current) { - regs->regs[3] = (unsigned long)__builtin_frame_address(0); - regs->csr_era = (unsigned long)__builtin_return_address(0); - } else { - regs->regs[3] = thread_saved_fp(task); - regs->csr_era = thread_saved_ra(task); + if (task == current) { + regs->regs[3] = (unsigned long)__builtin_frame_address(0); + regs->csr_era = (unsigned long)__builtin_return_address(0); + } else { + regs->regs[3] = thread_saved_fp(task); + regs->csr_era = thread_saved_ra(task); + } + regs->regs[1] = 0; } - regs->regs[1] = 0; for (unwind_start(&state, task, regs); !unwind_done(&state) && !unwind_error(&state); unwind_next_frame(&state)) { addr = unwind_get_return_address(&state); diff --git a/arch/loongarch/kernel/traps.c b/arch/loongarch/kernel/traps.c index 89699db45cec..65214774ef7c 100644 --- a/arch/loongarch/kernel/traps.c +++ b/arch/loongarch/kernel/traps.c @@ -36,7 +36,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -702,6 +704,11 @@ asmlinkage void noinstr do_bp(struct pt_regs *regs) * pertain to them. */ switch (bcode) { + case BRK_KDB: + if (kgdb_breakpoint_handler(regs)) + goto out; + else + break; case BRK_KPROBE_BP: if (kprobe_breakpoint_handler(regs)) goto out; @@ -768,6 +775,9 @@ asmlinkage void noinstr do_watch(struct pt_regs *regs) #ifndef CONFIG_HAVE_HW_BREAKPOINT pr_warn("Hardware watch point handler not implemented!\n"); #else + if (kgdb_breakpoint_handler(regs)) + goto out; + if (test_tsk_thread_flag(current, TIF_SINGLESTEP)) { int llbit = (csr_read32(LOONGARCH_CSR_LLBCTL) & 0x1); unsigned long pc = instruction_pointer(regs); @@ -966,13 +976,47 @@ out: irqentry_exit(regs, state); } +static void init_restore_lbt(void) +{ + if (!thread_lbt_context_live()) { + /* First time LBT context user */ + init_lbt(); + set_thread_flag(TIF_LBT_CTX_LIVE); + } else { + if (!is_lbt_owner()) + own_lbt_inatomic(1); + } + + BUG_ON(!is_lbt_enabled()); +} + asmlinkage void noinstr do_lbt(struct pt_regs *regs) { irqentry_state_t state = irqentry_enter(regs); - local_irq_enable(); - force_sig(SIGILL); - local_irq_disable(); + /* + * BTD (Binary Translation Disable exception) can be triggered + * during FP save/restore if TM (Top Mode) is on, which may + * cause irq_enable during 'switch_to'. To avoid this situation + * (including the user using 'MOVGR2GCSR' to turn on TM, which + * will not trigger the BTE), we need to check PRMD first. + */ + if (regs->csr_prmd & CSR_PRMD_PIE) + local_irq_enable(); + + if (!cpu_has_lbt) { + force_sig(SIGILL); + goto out; + } + BUG_ON(is_lbt_enabled()); + + preempt_disable(); + init_restore_lbt(); + preempt_enable(); + +out: + if (regs->csr_prmd & CSR_PRMD_PIE) + local_irq_disable(); irqentry_exit(regs, state); } diff --git a/arch/loongarch/lib/Makefile b/arch/loongarch/lib/Makefile index d60d4e096cfa..a77bf160bfc4 100644 --- a/arch/loongarch/lib/Makefile +++ b/arch/loongarch/lib/Makefile @@ -6,4 +6,6 @@ lib-y += delay.o memset.o memcpy.o memmove.o \ clear_user.o copy_user.o csum.o dump_tlb.o unaligned.o +obj-$(CONFIG_CPU_HAS_LSX) += xor_simd.o xor_simd_glue.o + obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o diff --git a/arch/loongarch/lib/clear_user.S b/arch/loongarch/lib/clear_user.S index 0790eadce166..be741544e62b 100644 --- a/arch/loongarch/lib/clear_user.S +++ b/arch/loongarch/lib/clear_user.S @@ -11,19 +11,6 @@ #include #include -.irp to, 0, 1, 2, 3, 4, 5, 6, 7 -.L_fixup_handle_\to\(): - sub.d a0, a2, a0 - addi.d a0, a0, (\to) * (-8) - jr ra -.endr - -.irp to, 0, 2, 4 -.L_fixup_handle_s\to\(): - addi.d a0, a1, -\to - jr ra -.endr - SYM_FUNC_START(__clear_user) /* * Some CPUs support hardware unaligned access @@ -51,7 +38,7 @@ SYM_FUNC_START(__clear_user_generic) 2: move a0, a1 jr ra - _asm_extable 1b, .L_fixup_handle_s0 + _asm_extable 1b, 2b SYM_FUNC_END(__clear_user_generic) /* @@ -173,33 +160,47 @@ SYM_FUNC_START(__clear_user_fast) jr ra /* fixup and ex_table */ - _asm_extable 0b, .L_fixup_handle_0 - _asm_extable 1b, .L_fixup_handle_0 - _asm_extable 2b, .L_fixup_handle_1 - _asm_extable 3b, .L_fixup_handle_2 - _asm_extable 4b, .L_fixup_handle_3 - _asm_extable 5b, .L_fixup_handle_4 - _asm_extable 6b, .L_fixup_handle_5 - _asm_extable 7b, .L_fixup_handle_6 - _asm_extable 8b, .L_fixup_handle_7 - _asm_extable 9b, .L_fixup_handle_0 - _asm_extable 10b, .L_fixup_handle_1 - _asm_extable 11b, .L_fixup_handle_2 - _asm_extable 12b, .L_fixup_handle_3 - _asm_extable 13b, .L_fixup_handle_0 - _asm_extable 14b, .L_fixup_handle_1 - _asm_extable 15b, .L_fixup_handle_0 - _asm_extable 16b, .L_fixup_handle_0 - _asm_extable 17b, .L_fixup_handle_s0 - _asm_extable 18b, .L_fixup_handle_s0 - _asm_extable 19b, .L_fixup_handle_s0 - _asm_extable 20b, .L_fixup_handle_s2 - _asm_extable 21b, .L_fixup_handle_s0 - _asm_extable 22b, .L_fixup_handle_s0 - _asm_extable 23b, .L_fixup_handle_s4 - _asm_extable 24b, .L_fixup_handle_s0 - _asm_extable 25b, .L_fixup_handle_s4 - _asm_extable 26b, .L_fixup_handle_s0 - _asm_extable 27b, .L_fixup_handle_s4 - _asm_extable 28b, .L_fixup_handle_s0 +.Llarge_fixup: + sub.d a1, a2, a0 + +.Lsmall_fixup: +29: st.b zero, a0, 0 + addi.d a0, a0, 1 + addi.d a1, a1, -1 + bgt a1, zero, 29b + +.Lexit: + move a0, a1 + jr ra + + _asm_extable 0b, .Lsmall_fixup + _asm_extable 1b, .Llarge_fixup + _asm_extable 2b, .Llarge_fixup + _asm_extable 3b, .Llarge_fixup + _asm_extable 4b, .Llarge_fixup + _asm_extable 5b, .Llarge_fixup + _asm_extable 6b, .Llarge_fixup + _asm_extable 7b, .Llarge_fixup + _asm_extable 8b, .Llarge_fixup + _asm_extable 9b, .Llarge_fixup + _asm_extable 10b, .Llarge_fixup + _asm_extable 11b, .Llarge_fixup + _asm_extable 12b, .Llarge_fixup + _asm_extable 13b, .Llarge_fixup + _asm_extable 14b, .Llarge_fixup + _asm_extable 15b, .Llarge_fixup + _asm_extable 16b, .Llarge_fixup + _asm_extable 17b, .Lexit + _asm_extable 18b, .Lsmall_fixup + _asm_extable 19b, .Lsmall_fixup + _asm_extable 20b, .Lsmall_fixup + _asm_extable 21b, .Lsmall_fixup + _asm_extable 22b, .Lsmall_fixup + _asm_extable 23b, .Lsmall_fixup + _asm_extable 24b, .Lsmall_fixup + _asm_extable 25b, .Lsmall_fixup + _asm_extable 26b, .Lsmall_fixup + _asm_extable 27b, .Lsmall_fixup + _asm_extable 28b, .Lsmall_fixup + _asm_extable 29b, .Lexit SYM_FUNC_END(__clear_user_fast) diff --git a/arch/loongarch/lib/copy_user.S b/arch/loongarch/lib/copy_user.S index bfe3d2793d00..feec3d362803 100644 --- a/arch/loongarch/lib/copy_user.S +++ b/arch/loongarch/lib/copy_user.S @@ -11,19 +11,6 @@ #include #include -.irp to, 0, 1, 2, 3, 4, 5, 6, 7 -.L_fixup_handle_\to\(): - sub.d a0, a2, a0 - addi.d a0, a0, (\to) * (-8) - jr ra -.endr - -.irp to, 0, 2, 4 -.L_fixup_handle_s\to\(): - addi.d a0, a2, -\to - jr ra -.endr - SYM_FUNC_START(__copy_user) /* * Some CPUs support hardware unaligned access @@ -54,8 +41,8 @@ SYM_FUNC_START(__copy_user_generic) 3: move a0, a2 jr ra - _asm_extable 1b, .L_fixup_handle_s0 - _asm_extable 2b, .L_fixup_handle_s0 + _asm_extable 1b, 3b + _asm_extable 2b, 3b SYM_FUNC_END(__copy_user_generic) /* @@ -69,10 +56,10 @@ SYM_FUNC_START(__copy_user_fast) sltui t0, a2, 9 bnez t0, .Lsmall - add.d a3, a1, a2 - add.d a2, a0, a2 0: ld.d t0, a1, 0 1: st.d t0, a0, 0 + add.d a3, a1, a2 + add.d a2, a0, a2 /* align up destination address */ andi t1, a0, 7 @@ -94,7 +81,6 @@ SYM_FUNC_START(__copy_user_fast) 7: ld.d t5, a1, 40 8: ld.d t6, a1, 48 9: ld.d t7, a1, 56 - addi.d a1, a1, 64 10: st.d t0, a0, 0 11: st.d t1, a0, 8 12: st.d t2, a0, 16 @@ -103,6 +89,7 @@ SYM_FUNC_START(__copy_user_fast) 15: st.d t5, a0, 40 16: st.d t6, a0, 48 17: st.d t7, a0, 56 + addi.d a1, a1, 64 addi.d a0, a0, 64 bltu a1, a4, .Lloop64 @@ -114,11 +101,11 @@ SYM_FUNC_START(__copy_user_fast) 19: ld.d t1, a1, 8 20: ld.d t2, a1, 16 21: ld.d t3, a1, 24 - addi.d a1, a1, 32 22: st.d t0, a0, 0 23: st.d t1, a0, 8 24: st.d t2, a0, 16 25: st.d t3, a0, 24 + addi.d a1, a1, 32 addi.d a0, a0, 32 .Llt32: @@ -126,9 +113,9 @@ SYM_FUNC_START(__copy_user_fast) bgeu a1, a4, .Llt16 26: ld.d t0, a1, 0 27: ld.d t1, a1, 8 - addi.d a1, a1, 16 28: st.d t0, a0, 0 29: st.d t1, a0, 8 + addi.d a1, a1, 16 addi.d a0, a0, 16 .Llt16: @@ -136,6 +123,7 @@ SYM_FUNC_START(__copy_user_fast) bgeu a1, a4, .Llt8 30: ld.d t0, a1, 0 31: st.d t0, a0, 0 + addi.d a1, a1, 8 addi.d a0, a0, 8 .Llt8: @@ -214,62 +202,79 @@ SYM_FUNC_START(__copy_user_fast) jr ra /* fixup and ex_table */ - _asm_extable 0b, .L_fixup_handle_0 - _asm_extable 1b, .L_fixup_handle_0 - _asm_extable 2b, .L_fixup_handle_0 - _asm_extable 3b, .L_fixup_handle_0 - _asm_extable 4b, .L_fixup_handle_0 - _asm_extable 5b, .L_fixup_handle_0 - _asm_extable 6b, .L_fixup_handle_0 - _asm_extable 7b, .L_fixup_handle_0 - _asm_extable 8b, .L_fixup_handle_0 - _asm_extable 9b, .L_fixup_handle_0 - _asm_extable 10b, .L_fixup_handle_0 - _asm_extable 11b, .L_fixup_handle_1 - _asm_extable 12b, .L_fixup_handle_2 - _asm_extable 13b, .L_fixup_handle_3 - _asm_extable 14b, .L_fixup_handle_4 - _asm_extable 15b, .L_fixup_handle_5 - _asm_extable 16b, .L_fixup_handle_6 - _asm_extable 17b, .L_fixup_handle_7 - _asm_extable 18b, .L_fixup_handle_0 - _asm_extable 19b, .L_fixup_handle_0 - _asm_extable 20b, .L_fixup_handle_0 - _asm_extable 21b, .L_fixup_handle_0 - _asm_extable 22b, .L_fixup_handle_0 - _asm_extable 23b, .L_fixup_handle_1 - _asm_extable 24b, .L_fixup_handle_2 - _asm_extable 25b, .L_fixup_handle_3 - _asm_extable 26b, .L_fixup_handle_0 - _asm_extable 27b, .L_fixup_handle_0 - _asm_extable 28b, .L_fixup_handle_0 - _asm_extable 29b, .L_fixup_handle_1 - _asm_extable 30b, .L_fixup_handle_0 - _asm_extable 31b, .L_fixup_handle_0 - _asm_extable 32b, .L_fixup_handle_0 - _asm_extable 33b, .L_fixup_handle_0 - _asm_extable 34b, .L_fixup_handle_s0 - _asm_extable 35b, .L_fixup_handle_s0 - _asm_extable 36b, .L_fixup_handle_s0 - _asm_extable 37b, .L_fixup_handle_s0 - _asm_extable 38b, .L_fixup_handle_s0 - _asm_extable 39b, .L_fixup_handle_s0 - _asm_extable 40b, .L_fixup_handle_s0 - _asm_extable 41b, .L_fixup_handle_s2 - _asm_extable 42b, .L_fixup_handle_s0 - _asm_extable 43b, .L_fixup_handle_s0 - _asm_extable 44b, .L_fixup_handle_s0 - _asm_extable 45b, .L_fixup_handle_s0 - _asm_extable 46b, .L_fixup_handle_s0 - _asm_extable 47b, .L_fixup_handle_s4 - _asm_extable 48b, .L_fixup_handle_s0 - _asm_extable 49b, .L_fixup_handle_s0 - _asm_extable 50b, .L_fixup_handle_s0 - _asm_extable 51b, .L_fixup_handle_s4 - _asm_extable 52b, .L_fixup_handle_s0 - _asm_extable 53b, .L_fixup_handle_s0 - _asm_extable 54b, .L_fixup_handle_s0 - _asm_extable 55b, .L_fixup_handle_s4 - _asm_extable 56b, .L_fixup_handle_s0 - _asm_extable 57b, .L_fixup_handle_s0 +.Llarge_fixup: + sub.d a2, a2, a0 + +.Lsmall_fixup: +58: ld.b t0, a1, 0 +59: st.b t0, a0, 0 + addi.d a0, a0, 1 + addi.d a1, a1, 1 + addi.d a2, a2, -1 + bgt a2, zero, 58b + +.Lexit: + move a0, a2 + jr ra + + _asm_extable 0b, .Lsmall_fixup + _asm_extable 1b, .Lsmall_fixup + _asm_extable 2b, .Llarge_fixup + _asm_extable 3b, .Llarge_fixup + _asm_extable 4b, .Llarge_fixup + _asm_extable 5b, .Llarge_fixup + _asm_extable 6b, .Llarge_fixup + _asm_extable 7b, .Llarge_fixup + _asm_extable 8b, .Llarge_fixup + _asm_extable 9b, .Llarge_fixup + _asm_extable 10b, .Llarge_fixup + _asm_extable 11b, .Llarge_fixup + _asm_extable 12b, .Llarge_fixup + _asm_extable 13b, .Llarge_fixup + _asm_extable 14b, .Llarge_fixup + _asm_extable 15b, .Llarge_fixup + _asm_extable 16b, .Llarge_fixup + _asm_extable 17b, .Llarge_fixup + _asm_extable 18b, .Llarge_fixup + _asm_extable 19b, .Llarge_fixup + _asm_extable 20b, .Llarge_fixup + _asm_extable 21b, .Llarge_fixup + _asm_extable 22b, .Llarge_fixup + _asm_extable 23b, .Llarge_fixup + _asm_extable 24b, .Llarge_fixup + _asm_extable 25b, .Llarge_fixup + _asm_extable 26b, .Llarge_fixup + _asm_extable 27b, .Llarge_fixup + _asm_extable 28b, .Llarge_fixup + _asm_extable 29b, .Llarge_fixup + _asm_extable 30b, .Llarge_fixup + _asm_extable 31b, .Llarge_fixup + _asm_extable 32b, .Llarge_fixup + _asm_extable 33b, .Llarge_fixup + _asm_extable 34b, .Lexit + _asm_extable 35b, .Lexit + _asm_extable 36b, .Lsmall_fixup + _asm_extable 37b, .Lsmall_fixup + _asm_extable 38b, .Lsmall_fixup + _asm_extable 39b, .Lsmall_fixup + _asm_extable 40b, .Lsmall_fixup + _asm_extable 41b, .Lsmall_fixup + _asm_extable 42b, .Lsmall_fixup + _asm_extable 43b, .Lsmall_fixup + _asm_extable 44b, .Lsmall_fixup + _asm_extable 45b, .Lsmall_fixup + _asm_extable 46b, .Lsmall_fixup + _asm_extable 47b, .Lsmall_fixup + _asm_extable 48b, .Lsmall_fixup + _asm_extable 49b, .Lsmall_fixup + _asm_extable 50b, .Lsmall_fixup + _asm_extable 51b, .Lsmall_fixup + _asm_extable 52b, .Lsmall_fixup + _asm_extable 53b, .Lsmall_fixup + _asm_extable 54b, .Lsmall_fixup + _asm_extable 55b, .Lsmall_fixup + _asm_extable 56b, .Lsmall_fixup + _asm_extable 57b, .Lsmall_fixup + _asm_extable 58b, .Lexit + _asm_extable 59b, .Lexit SYM_FUNC_END(__copy_user_fast) diff --git a/arch/loongarch/lib/memcpy.S b/arch/loongarch/lib/memcpy.S index cc30b3b6252f..fa1148878d2b 100644 --- a/arch/loongarch/lib/memcpy.S +++ b/arch/loongarch/lib/memcpy.S @@ -10,6 +10,8 @@ #include #include +.section .noinstr.text, "ax" + SYM_FUNC_START(memcpy) /* * Some CPUs support hardware unaligned access @@ -17,9 +19,13 @@ SYM_FUNC_START(memcpy) ALTERNATIVE "b __memcpy_generic", \ "b __memcpy_fast", CPU_FEATURE_UAL SYM_FUNC_END(memcpy) -_ASM_NOKPROBE(memcpy) +SYM_FUNC_ALIAS(__memcpy, memcpy) EXPORT_SYMBOL(memcpy) +EXPORT_SYMBOL(__memcpy) + +_ASM_NOKPROBE(memcpy) +_ASM_NOKPROBE(__memcpy) /* * void *__memcpy_generic(void *dst, const void *src, size_t n) diff --git a/arch/loongarch/lib/memmove.S b/arch/loongarch/lib/memmove.S index 7dc76d1484b6..82dae062fec8 100644 --- a/arch/loongarch/lib/memmove.S +++ b/arch/loongarch/lib/memmove.S @@ -10,23 +10,29 @@ #include #include +.section .noinstr.text, "ax" + SYM_FUNC_START(memmove) - blt a0, a1, memcpy /* dst < src, memcpy */ - blt a1, a0, rmemcpy /* src < dst, rmemcpy */ - jr ra /* dst == src, return */ + blt a0, a1, __memcpy /* dst < src, memcpy */ + blt a1, a0, __rmemcpy /* src < dst, rmemcpy */ + jr ra /* dst == src, return */ SYM_FUNC_END(memmove) -_ASM_NOKPROBE(memmove) +SYM_FUNC_ALIAS(__memmove, memmove) EXPORT_SYMBOL(memmove) +EXPORT_SYMBOL(__memmove) -SYM_FUNC_START(rmemcpy) +_ASM_NOKPROBE(memmove) +_ASM_NOKPROBE(__memmove) + +SYM_FUNC_START(__rmemcpy) /* * Some CPUs support hardware unaligned access */ ALTERNATIVE "b __rmemcpy_generic", \ "b __rmemcpy_fast", CPU_FEATURE_UAL -SYM_FUNC_END(rmemcpy) -_ASM_NOKPROBE(rmemcpy) +SYM_FUNC_END(__rmemcpy) +_ASM_NOKPROBE(__rmemcpy) /* * void *__rmemcpy_generic(void *dst, const void *src, size_t n) diff --git a/arch/loongarch/lib/memset.S b/arch/loongarch/lib/memset.S index 3f20f7996e8e..06d3ca54cbfe 100644 --- a/arch/loongarch/lib/memset.S +++ b/arch/loongarch/lib/memset.S @@ -16,6 +16,8 @@ bstrins.d \r0, \r0, 63, 32 .endm +.section .noinstr.text, "ax" + SYM_FUNC_START(memset) /* * Some CPUs support hardware unaligned access @@ -23,9 +25,13 @@ SYM_FUNC_START(memset) ALTERNATIVE "b __memset_generic", \ "b __memset_fast", CPU_FEATURE_UAL SYM_FUNC_END(memset) -_ASM_NOKPROBE(memset) +SYM_FUNC_ALIAS(__memset, memset) EXPORT_SYMBOL(memset) +EXPORT_SYMBOL(__memset) + +_ASM_NOKPROBE(memset) +_ASM_NOKPROBE(__memset) /* * void *__memset_generic(void *s, int c, size_t n) diff --git a/arch/loongarch/lib/xor_simd.c b/arch/loongarch/lib/xor_simd.c new file mode 100644 index 000000000000..84cd24b728c4 --- /dev/null +++ b/arch/loongarch/lib/xor_simd.c @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * LoongArch SIMD XOR operations + * + * Copyright (C) 2023 WANG Xuerui + */ + +#include "xor_simd.h" + +/* + * Process one cache line (64 bytes) per loop. This is assuming all future + * popular LoongArch cores are similar performance-characteristics-wise to the + * current models. + */ +#define LINE_WIDTH 64 + +#ifdef CONFIG_CPU_HAS_LSX + +#define LD(reg, base, offset) \ + "vld $vr" #reg ", %[" #base "], " #offset "\n\t" +#define ST(reg, base, offset) \ + "vst $vr" #reg ", %[" #base "], " #offset "\n\t" +#define XOR(dj, k) "vxor.v $vr" #dj ", $vr" #dj ", $vr" #k "\n\t" + +#define LD_INOUT_LINE(base) \ + LD(0, base, 0) \ + LD(1, base, 16) \ + LD(2, base, 32) \ + LD(3, base, 48) + +#define LD_AND_XOR_LINE(base) \ + LD(4, base, 0) \ + LD(5, base, 16) \ + LD(6, base, 32) \ + LD(7, base, 48) \ + XOR(0, 4) \ + XOR(1, 5) \ + XOR(2, 6) \ + XOR(3, 7) + +#define ST_LINE(base) \ + ST(0, base, 0) \ + ST(1, base, 16) \ + ST(2, base, 32) \ + ST(3, base, 48) + +#define XOR_FUNC_NAME(nr) __xor_lsx_##nr +#include "xor_template.c" + +#undef LD +#undef ST +#undef XOR +#undef LD_INOUT_LINE +#undef LD_AND_XOR_LINE +#undef ST_LINE +#undef XOR_FUNC_NAME + +#endif /* CONFIG_CPU_HAS_LSX */ + +#ifdef CONFIG_CPU_HAS_LASX + +#define LD(reg, base, offset) \ + "xvld $xr" #reg ", %[" #base "], " #offset "\n\t" +#define ST(reg, base, offset) \ + "xvst $xr" #reg ", %[" #base "], " #offset "\n\t" +#define XOR(dj, k) "xvxor.v $xr" #dj ", $xr" #dj ", $xr" #k "\n\t" + +#define LD_INOUT_LINE(base) \ + LD(0, base, 0) \ + LD(1, base, 32) + +#define LD_AND_XOR_LINE(base) \ + LD(2, base, 0) \ + LD(3, base, 32) \ + XOR(0, 2) \ + XOR(1, 3) + +#define ST_LINE(base) \ + ST(0, base, 0) \ + ST(1, base, 32) + +#define XOR_FUNC_NAME(nr) __xor_lasx_##nr +#include "xor_template.c" + +#undef LD +#undef ST +#undef XOR +#undef LD_INOUT_LINE +#undef LD_AND_XOR_LINE +#undef ST_LINE +#undef XOR_FUNC_NAME + +#endif /* CONFIG_CPU_HAS_LASX */ diff --git a/arch/loongarch/lib/xor_simd.h b/arch/loongarch/lib/xor_simd.h new file mode 100644 index 000000000000..f50f32514d80 --- /dev/null +++ b/arch/loongarch/lib/xor_simd.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Simple interface to link xor_simd.c and xor_simd_glue.c + * + * Separating these files ensures that no SIMD instructions are run outside of + * the kfpu critical section. + */ + +#ifndef __LOONGARCH_LIB_XOR_SIMD_H +#define __LOONGARCH_LIB_XOR_SIMD_H + +#ifdef CONFIG_CPU_HAS_LSX +void __xor_lsx_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2); +void __xor_lsx_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3); +void __xor_lsx_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3, + const unsigned long * __restrict p4); +void __xor_lsx_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3, + const unsigned long * __restrict p4, const unsigned long * __restrict p5); +#endif /* CONFIG_CPU_HAS_LSX */ + +#ifdef CONFIG_CPU_HAS_LASX +void __xor_lasx_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2); +void __xor_lasx_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3); +void __xor_lasx_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3, + const unsigned long * __restrict p4); +void __xor_lasx_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, const unsigned long * __restrict p3, + const unsigned long * __restrict p4, const unsigned long * __restrict p5); +#endif /* CONFIG_CPU_HAS_LASX */ + +#endif /* __LOONGARCH_LIB_XOR_SIMD_H */ diff --git a/arch/loongarch/lib/xor_simd_glue.c b/arch/loongarch/lib/xor_simd_glue.c new file mode 100644 index 000000000000..393f689dbcf6 --- /dev/null +++ b/arch/loongarch/lib/xor_simd_glue.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * LoongArch SIMD XOR operations + * + * Copyright (C) 2023 WANG Xuerui + */ + +#include +#include +#include +#include +#include "xor_simd.h" + +#define MAKE_XOR_GLUE_2(flavor) \ +void xor_##flavor##_2(unsigned long bytes, unsigned long * __restrict p1, \ + const unsigned long * __restrict p2) \ +{ \ + kernel_fpu_begin(); \ + __xor_##flavor##_2(bytes, p1, p2); \ + kernel_fpu_end(); \ +} \ +EXPORT_SYMBOL_GPL(xor_##flavor##_2) + +#define MAKE_XOR_GLUE_3(flavor) \ +void xor_##flavor##_3(unsigned long bytes, unsigned long * __restrict p1, \ + const unsigned long * __restrict p2, \ + const unsigned long * __restrict p3) \ +{ \ + kernel_fpu_begin(); \ + __xor_##flavor##_3(bytes, p1, p2, p3); \ + kernel_fpu_end(); \ +} \ +EXPORT_SYMBOL_GPL(xor_##flavor##_3) + +#define MAKE_XOR_GLUE_4(flavor) \ +void xor_##flavor##_4(unsigned long bytes, unsigned long * __restrict p1, \ + const unsigned long * __restrict p2, \ + const unsigned long * __restrict p3, \ + const unsigned long * __restrict p4) \ +{ \ + kernel_fpu_begin(); \ + __xor_##flavor##_4(bytes, p1, p2, p3, p4); \ + kernel_fpu_end(); \ +} \ +EXPORT_SYMBOL_GPL(xor_##flavor##_4) + +#define MAKE_XOR_GLUE_5(flavor) \ +void xor_##flavor##_5(unsigned long bytes, unsigned long * __restrict p1, \ + const unsigned long * __restrict p2, \ + const unsigned long * __restrict p3, \ + const unsigned long * __restrict p4, \ + const unsigned long * __restrict p5) \ +{ \ + kernel_fpu_begin(); \ + __xor_##flavor##_5(bytes, p1, p2, p3, p4, p5); \ + kernel_fpu_end(); \ +} \ +EXPORT_SYMBOL_GPL(xor_##flavor##_5) + +#define MAKE_XOR_GLUES(flavor) \ + MAKE_XOR_GLUE_2(flavor); \ + MAKE_XOR_GLUE_3(flavor); \ + MAKE_XOR_GLUE_4(flavor); \ + MAKE_XOR_GLUE_5(flavor) + +#ifdef CONFIG_CPU_HAS_LSX +MAKE_XOR_GLUES(lsx); +#endif + +#ifdef CONFIG_CPU_HAS_LASX +MAKE_XOR_GLUES(lasx); +#endif diff --git a/arch/loongarch/lib/xor_template.c b/arch/loongarch/lib/xor_template.c new file mode 100644 index 000000000000..0358ced7fe33 --- /dev/null +++ b/arch/loongarch/lib/xor_template.c @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2023 WANG Xuerui + * + * Template for XOR operations, instantiated in xor_simd.c. + * + * Expected preprocessor definitions: + * + * - LINE_WIDTH + * - XOR_FUNC_NAME(nr) + * - LD_INOUT_LINE(buf) + * - LD_AND_XOR_LINE(buf) + * - ST_LINE(buf) + */ + +void XOR_FUNC_NAME(2)(unsigned long bytes, + unsigned long * __restrict v1, + const unsigned long * __restrict v2) +{ + unsigned long lines = bytes / LINE_WIDTH; + + do { + __asm__ __volatile__ ( + LD_INOUT_LINE(v1) + LD_AND_XOR_LINE(v2) + ST_LINE(v1) + : : [v1] "r"(v1), [v2] "r"(v2) : "memory" + ); + + v1 += LINE_WIDTH / sizeof(unsigned long); + v2 += LINE_WIDTH / sizeof(unsigned long); + } while (--lines > 0); +} + +void XOR_FUNC_NAME(3)(unsigned long bytes, + unsigned long * __restrict v1, + const unsigned long * __restrict v2, + const unsigned long * __restrict v3) +{ + unsigned long lines = bytes / LINE_WIDTH; + + do { + __asm__ __volatile__ ( + LD_INOUT_LINE(v1) + LD_AND_XOR_LINE(v2) + LD_AND_XOR_LINE(v3) + ST_LINE(v1) + : : [v1] "r"(v1), [v2] "r"(v2), [v3] "r"(v3) : "memory" + ); + + v1 += LINE_WIDTH / sizeof(unsigned long); + v2 += LINE_WIDTH / sizeof(unsigned long); + v3 += LINE_WIDTH / sizeof(unsigned long); + } while (--lines > 0); +} + +void XOR_FUNC_NAME(4)(unsigned long bytes, + unsigned long * __restrict v1, + const unsigned long * __restrict v2, + const unsigned long * __restrict v3, + const unsigned long * __restrict v4) +{ + unsigned long lines = bytes / LINE_WIDTH; + + do { + __asm__ __volatile__ ( + LD_INOUT_LINE(v1) + LD_AND_XOR_LINE(v2) + LD_AND_XOR_LINE(v3) + LD_AND_XOR_LINE(v4) + ST_LINE(v1) + : : [v1] "r"(v1), [v2] "r"(v2), [v3] "r"(v3), [v4] "r"(v4) + : "memory" + ); + + v1 += LINE_WIDTH / sizeof(unsigned long); + v2 += LINE_WIDTH / sizeof(unsigned long); + v3 += LINE_WIDTH / sizeof(unsigned long); + v4 += LINE_WIDTH / sizeof(unsigned long); + } while (--lines > 0); +} + +void XOR_FUNC_NAME(5)(unsigned long bytes, + unsigned long * __restrict v1, + const unsigned long * __restrict v2, + const unsigned long * __restrict v3, + const unsigned long * __restrict v4, + const unsigned long * __restrict v5) +{ + unsigned long lines = bytes / LINE_WIDTH; + + do { + __asm__ __volatile__ ( + LD_INOUT_LINE(v1) + LD_AND_XOR_LINE(v2) + LD_AND_XOR_LINE(v3) + LD_AND_XOR_LINE(v4) + LD_AND_XOR_LINE(v5) + ST_LINE(v1) + : : [v1] "r"(v1), [v2] "r"(v2), [v3] "r"(v3), [v4] "r"(v4), + [v5] "r"(v5) : "memory" + ); + + v1 += LINE_WIDTH / sizeof(unsigned long); + v2 += LINE_WIDTH / sizeof(unsigned long); + v3 += LINE_WIDTH / sizeof(unsigned long); + v4 += LINE_WIDTH / sizeof(unsigned long); + v5 += LINE_WIDTH / sizeof(unsigned long); + } while (--lines > 0); +} diff --git a/arch/loongarch/mm/Makefile b/arch/loongarch/mm/Makefile index 8ffc6383f836..e4d1e581dbae 100644 --- a/arch/loongarch/mm/Makefile +++ b/arch/loongarch/mm/Makefile @@ -7,3 +7,6 @@ obj-y += init.o cache.o tlb.o tlbex.o extable.o \ fault.o ioremap.o maccess.o mmap.o pgtable.o page.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o +obj-$(CONFIG_KASAN) += kasan_init.o + +KASAN_SANITIZE_kasan_init.o := n diff --git a/arch/loongarch/mm/cache.c b/arch/loongarch/mm/cache.c index 72685a48eaf0..6be04d36ca07 100644 --- a/arch/loongarch/mm/cache.c +++ b/arch/loongarch/mm/cache.c @@ -156,7 +156,6 @@ void cpu_cache_init(void) current_cpu_data.cache_leaves_present = leaf; current_cpu_data.options |= LOONGARCH_CPU_PREFETCH; - shm_align_mask = PAGE_SIZE - 1; } static const pgprot_t protection_map[16] = { diff --git a/arch/loongarch/mm/fault.c b/arch/loongarch/mm/fault.c index da5b6d518cdb..e6376e3dce86 100644 --- a/arch/loongarch/mm/fault.c +++ b/arch/loongarch/mm/fault.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include @@ -30,7 +31,8 @@ int show_unhandled_signals = 1; -static void __kprobes no_context(struct pt_regs *regs, unsigned long address) +static void __kprobes no_context(struct pt_regs *regs, + unsigned long write, unsigned long address) { const int field = sizeof(unsigned long) * 2; @@ -38,6 +40,9 @@ static void __kprobes no_context(struct pt_regs *regs, unsigned long address) if (fixup_exception(regs)) return; + if (kfence_handle_page_fault(address, write, regs)) + return; + /* * Oops. The kernel tried to access some bad page. We'll have to * terminate things with extreme prejudice. @@ -51,14 +56,15 @@ static void __kprobes no_context(struct pt_regs *regs, unsigned long address) die("Oops", regs); } -static void __kprobes do_out_of_memory(struct pt_regs *regs, unsigned long address) +static void __kprobes do_out_of_memory(struct pt_regs *regs, + unsigned long write, unsigned long address) { /* * We ran out of memory, call the OOM killer, and return the userspace * (which will retry the fault, or kill us if we got oom-killed). */ if (!user_mode(regs)) { - no_context(regs, address); + no_context(regs, write, address); return; } pagefault_out_of_memory(); @@ -69,7 +75,7 @@ static void __kprobes do_sigbus(struct pt_regs *regs, { /* Kernel mode? Handle exceptions or die */ if (!user_mode(regs)) { - no_context(regs, address); + no_context(regs, write, address); return; } @@ -90,7 +96,7 @@ static void __kprobes do_sigsegv(struct pt_regs *regs, /* Kernel mode? Handle exceptions or die */ if (!user_mode(regs)) { - no_context(regs, address); + no_context(regs, write, address); return; } @@ -149,7 +155,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, */ if (address & __UA_LIMIT) { if (!user_mode(regs)) - no_context(regs, address); + no_context(regs, write, address); else do_sigsegv(regs, write, address, si_code); return; @@ -211,7 +217,7 @@ good_area: if (fault_signal_pending(fault, regs)) { if (!user_mode(regs)) - no_context(regs, address); + no_context(regs, write, address); return; } @@ -232,7 +238,7 @@ good_area: if (unlikely(fault & VM_FAULT_ERROR)) { mmap_read_unlock(mm); if (fault & VM_FAULT_OOM) { - do_out_of_memory(regs, address); + do_out_of_memory(regs, write, address); return; } else if (fault & VM_FAULT_SIGSEGV) { do_sigsegv(regs, write, address, si_code); diff --git a/arch/loongarch/mm/init.c b/arch/loongarch/mm/init.c index 3b7d8129570b..f3fe8c06ba4d 100644 --- a/arch/loongarch/mm/init.c +++ b/arch/loongarch/mm/init.c @@ -35,33 +35,8 @@ #include #include -/* - * We have up to 8 empty zeroed pages so we can map one of the right colour - * when needed. Since page is never written to after the initialization we - * don't have to care about aliases on other CPUs. - */ -unsigned long empty_zero_page, zero_page_mask; +unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)] __page_aligned_bss; EXPORT_SYMBOL(empty_zero_page); -EXPORT_SYMBOL(zero_page_mask); - -void setup_zero_pages(void) -{ - unsigned int order, i; - struct page *page; - - order = 0; - - empty_zero_page = __get_free_pages(GFP_KERNEL | __GFP_ZERO, order); - if (!empty_zero_page) - panic("Oh boy, that early out of memory?"); - - page = virt_to_page((void *)empty_zero_page); - split_page(page, order); - for (i = 0; i < (1 << order); i++, page++) - mark_page_reserved(page); - - zero_page_mask = ((PAGE_SIZE << order) - 1) & PAGE_MASK; -} void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr, struct vm_area_struct *vma) @@ -106,7 +81,6 @@ void __init mem_init(void) high_memory = (void *) __va(max_low_pfn << PAGE_SHIFT); memblock_free_all(); - setup_zero_pages(); /* Setup zeroed pages. */ } #endif /* !CONFIG_NUMA */ @@ -191,43 +165,42 @@ void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *al #endif #endif -static pte_t *fixmap_pte(unsigned long addr) +pte_t * __init populate_kernel_pte(unsigned long addr) { - pgd_t *pgd; - p4d_t *p4d; + pgd_t *pgd = pgd_offset_k(addr); + p4d_t *p4d = p4d_offset(pgd, addr); pud_t *pud; pmd_t *pmd; - pgd = pgd_offset_k(addr); - p4d = p4d_offset(pgd, addr); - - if (pgd_none(*pgd)) { - pud_t *new __maybe_unused; - - new = memblock_alloc_low(PAGE_SIZE, PAGE_SIZE); - pgd_populate(&init_mm, pgd, new); + if (p4d_none(*p4d)) { + pud = memblock_alloc(PAGE_SIZE, PAGE_SIZE); + if (!pud) + panic("%s: Failed to allocate memory\n", __func__); + p4d_populate(&init_mm, p4d, pud); #ifndef __PAGETABLE_PUD_FOLDED - pud_init(new); + pud_init(pud); #endif } pud = pud_offset(p4d, addr); if (pud_none(*pud)) { - pmd_t *new __maybe_unused; - - new = memblock_alloc_low(PAGE_SIZE, PAGE_SIZE); - pud_populate(&init_mm, pud, new); + pmd = memblock_alloc(PAGE_SIZE, PAGE_SIZE); + if (!pmd) + panic("%s: Failed to allocate memory\n", __func__); + pud_populate(&init_mm, pud, pmd); #ifndef __PAGETABLE_PMD_FOLDED - pmd_init(new); + pmd_init(pmd); #endif } pmd = pmd_offset(pud, addr); - if (pmd_none(*pmd)) { - pte_t *new __maybe_unused; + if (!pmd_present(*pmd)) { + pte_t *pte; - new = memblock_alloc_low(PAGE_SIZE, PAGE_SIZE); - pmd_populate_kernel(&init_mm, pmd, new); + pte = memblock_alloc(PAGE_SIZE, PAGE_SIZE); + if (!pte) + panic("%s: Failed to allocate memory\n", __func__); + pmd_populate_kernel(&init_mm, pmd, pte); } return pte_offset_kernel(pmd, addr); @@ -241,7 +214,7 @@ void __init __set_fixmap(enum fixed_addresses idx, BUG_ON(idx <= FIX_HOLE || idx >= __end_of_fixed_addresses); - ptep = fixmap_pte(addr); + ptep = populate_kernel_pte(addr); if (!pte_none(*ptep)) { pte_ERROR(*ptep); return; diff --git a/arch/loongarch/mm/kasan_init.c b/arch/loongarch/mm/kasan_init.c new file mode 100644 index 000000000000..da68bc1a4643 --- /dev/null +++ b/arch/loongarch/mm/kasan_init.c @@ -0,0 +1,243 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 Loongson Technology Corporation Limited + */ +#define pr_fmt(fmt) "kasan: " fmt +#include +#include +#include + +#include +#include +#include + +static pgd_t kasan_pg_dir[PTRS_PER_PGD] __initdata __aligned(PAGE_SIZE); + +#ifdef __PAGETABLE_PUD_FOLDED +#define __p4d_none(early, p4d) (0) +#else +#define __p4d_none(early, p4d) (early ? (p4d_val(p4d) == 0) : \ +(__pa(p4d_val(p4d)) == (unsigned long)__pa(kasan_early_shadow_pud))) +#endif + +#ifdef __PAGETABLE_PMD_FOLDED +#define __pud_none(early, pud) (0) +#else +#define __pud_none(early, pud) (early ? (pud_val(pud) == 0) : \ +(__pa(pud_val(pud)) == (unsigned long)__pa(kasan_early_shadow_pmd))) +#endif + +#define __pmd_none(early, pmd) (early ? (pmd_val(pmd) == 0) : \ +(__pa(pmd_val(pmd)) == (unsigned long)__pa(kasan_early_shadow_pte))) + +#define __pte_none(early, pte) (early ? pte_none(pte) : \ +((pte_val(pte) & _PFN_MASK) == (unsigned long)__pa(kasan_early_shadow_page))) + +bool kasan_early_stage = true; + +/* + * Alloc memory for shadow memory page table. + */ +static phys_addr_t __init kasan_alloc_zeroed_page(int node) +{ + void *p = memblock_alloc_try_nid(PAGE_SIZE, PAGE_SIZE, + __pa(MAX_DMA_ADDRESS), MEMBLOCK_ALLOC_ACCESSIBLE, node); + if (!p) + panic("%s: Failed to allocate %lu bytes align=0x%lx nid=%d from=%llx\n", + __func__, PAGE_SIZE, PAGE_SIZE, node, __pa(MAX_DMA_ADDRESS)); + + return __pa(p); +} + +static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node, bool early) +{ + if (__pmd_none(early, READ_ONCE(*pmdp))) { + phys_addr_t pte_phys = early ? + __pa_symbol(kasan_early_shadow_pte) : kasan_alloc_zeroed_page(node); + if (!early) + memcpy(__va(pte_phys), kasan_early_shadow_pte, sizeof(kasan_early_shadow_pte)); + pmd_populate_kernel(NULL, pmdp, (pte_t *)__va(pte_phys)); + } + + return pte_offset_kernel(pmdp, addr); +} + +static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node, bool early) +{ + if (__pud_none(early, READ_ONCE(*pudp))) { + phys_addr_t pmd_phys = early ? + __pa_symbol(kasan_early_shadow_pmd) : kasan_alloc_zeroed_page(node); + if (!early) + memcpy(__va(pmd_phys), kasan_early_shadow_pmd, sizeof(kasan_early_shadow_pmd)); + pud_populate(&init_mm, pudp, (pmd_t *)__va(pmd_phys)); + } + + return pmd_offset(pudp, addr); +} + +static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node, bool early) +{ + if (__p4d_none(early, READ_ONCE(*p4dp))) { + phys_addr_t pud_phys = early ? + __pa_symbol(kasan_early_shadow_pud) : kasan_alloc_zeroed_page(node); + if (!early) + memcpy(__va(pud_phys), kasan_early_shadow_pud, sizeof(kasan_early_shadow_pud)); + p4d_populate(&init_mm, p4dp, (pud_t *)__va(pud_phys)); + } + + return pud_offset(p4dp, addr); +} + +static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, + unsigned long end, int node, bool early) +{ + unsigned long next; + pte_t *ptep = kasan_pte_offset(pmdp, addr, node, early); + + do { + phys_addr_t page_phys = early ? + __pa_symbol(kasan_early_shadow_page) + : kasan_alloc_zeroed_page(node); + next = addr + PAGE_SIZE; + set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); + } while (ptep++, addr = next, addr != end && __pte_none(early, READ_ONCE(*ptep))); +} + +static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, + unsigned long end, int node, bool early) +{ + unsigned long next; + pmd_t *pmdp = kasan_pmd_offset(pudp, addr, node, early); + + do { + next = pmd_addr_end(addr, end); + kasan_pte_populate(pmdp, addr, next, node, early); + } while (pmdp++, addr = next, addr != end && __pmd_none(early, READ_ONCE(*pmdp))); +} + +static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr, + unsigned long end, int node, bool early) +{ + unsigned long next; + pud_t *pudp = kasan_pud_offset(p4dp, addr, node, early); + + do { + next = pud_addr_end(addr, end); + kasan_pmd_populate(pudp, addr, next, node, early); + } while (pudp++, addr = next, addr != end); +} + +static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, + unsigned long end, int node, bool early) +{ + unsigned long next; + p4d_t *p4dp = p4d_offset(pgdp, addr); + + do { + next = p4d_addr_end(addr, end); + kasan_pud_populate(p4dp, addr, next, node, early); + } while (p4dp++, addr = next, addr != end); +} + +static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, + int node, bool early) +{ + unsigned long next; + pgd_t *pgdp; + + pgdp = pgd_offset_k(addr); + + do { + next = pgd_addr_end(addr, end); + kasan_p4d_populate(pgdp, addr, next, node, early); + } while (pgdp++, addr = next, addr != end); + +} + +/* Set up full kasan mappings, ensuring that the mapped pages are zeroed */ +static void __init kasan_map_populate(unsigned long start, unsigned long end, + int node) +{ + kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); +} + +asmlinkage void __init kasan_early_init(void) +{ + BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_START, PGDIR_SIZE)); + BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, PGDIR_SIZE)); +} + +static inline void kasan_set_pgd(pgd_t *pgdp, pgd_t pgdval) +{ + WRITE_ONCE(*pgdp, pgdval); +} + +static void __init clear_pgds(unsigned long start, unsigned long end) +{ + /* + * Remove references to kasan page tables from + * swapper_pg_dir. pgd_clear() can't be used + * here because it's nop on 2,3-level pagetable setups + */ + for (; start < end; start += PGDIR_SIZE) + kasan_set_pgd((pgd_t *)pgd_offset_k(start), __pgd(0)); +} + +void __init kasan_init(void) +{ + u64 i; + phys_addr_t pa_start, pa_end; + + /* + * PGD was populated as invalid_pmd_table or invalid_pud_table + * in pagetable_init() which depends on how many levels of page + * table you are using, but we had to clean the gpd of kasan + * shadow memory, as the pgd value is none-zero. + * The assertion pgd_none is going to be false and the formal populate + * afterwards is not going to create any new pgd at all. + */ + memcpy(kasan_pg_dir, swapper_pg_dir, sizeof(kasan_pg_dir)); + csr_write64(__pa_symbol(kasan_pg_dir), LOONGARCH_CSR_PGDH); + local_flush_tlb_all(); + + clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + + /* Maps everything to a single page of zeroes */ + kasan_pgd_populate(KASAN_SHADOW_START, KASAN_SHADOW_END, NUMA_NO_NODE, true); + + kasan_populate_early_shadow(kasan_mem_to_shadow((void *)VMALLOC_START), + kasan_mem_to_shadow((void *)KFENCE_AREA_END)); + + kasan_early_stage = false; + + /* Populate the linear mapping */ + for_each_mem_range(i, &pa_start, &pa_end) { + void *start = (void *)phys_to_virt(pa_start); + void *end = (void *)phys_to_virt(pa_end); + + if (start >= end) + break; + + kasan_map_populate((unsigned long)kasan_mem_to_shadow(start), + (unsigned long)kasan_mem_to_shadow(end), NUMA_NO_NODE); + } + + /* Populate modules mapping */ + kasan_map_populate((unsigned long)kasan_mem_to_shadow((void *)MODULES_VADDR), + (unsigned long)kasan_mem_to_shadow((void *)MODULES_END), NUMA_NO_NODE); + /* + * KAsan may reuse the contents of kasan_early_shadow_pte directly, so we + * should make sure that it maps the zero page read-only. + */ + for (i = 0; i < PTRS_PER_PTE; i++) + set_pte(&kasan_early_shadow_pte[i], + pfn_pte(__phys_to_pfn(__pa_symbol(kasan_early_shadow_page)), PAGE_KERNEL_RO)); + + memset(kasan_early_shadow_page, 0, PAGE_SIZE); + csr_write64(__pa_symbol(swapper_pg_dir), LOONGARCH_CSR_PGDH); + local_flush_tlb_all(); + + /* At this point kasan is fully initialized. Enable error messages */ + init_task.kasan_depth = 0; + pr_info("KernelAddressSanitizer initialized.\n"); +} diff --git a/arch/loongarch/mm/mmap.c b/arch/loongarch/mm/mmap.c index fbe1a4856fc4..a9630a81b38a 100644 --- a/arch/loongarch/mm/mmap.c +++ b/arch/loongarch/mm/mmap.c @@ -8,12 +8,11 @@ #include #include -unsigned long shm_align_mask = PAGE_SIZE - 1; /* Sane caches */ -EXPORT_SYMBOL(shm_align_mask); +#define SHM_ALIGN_MASK (SHMLBA - 1) -#define COLOUR_ALIGN(addr, pgoff) \ - ((((addr) + shm_align_mask) & ~shm_align_mask) + \ - (((pgoff) << PAGE_SHIFT) & shm_align_mask)) +#define COLOUR_ALIGN(addr, pgoff) \ + ((((addr) + SHM_ALIGN_MASK) & ~SHM_ALIGN_MASK) \ + + (((pgoff) << PAGE_SHIFT) & SHM_ALIGN_MASK)) enum mmap_allocation_direction {UP, DOWN}; @@ -40,7 +39,7 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp, * cache aliasing constraints. */ if ((flags & MAP_SHARED) && - ((addr - (pgoff << PAGE_SHIFT)) & shm_align_mask)) + ((addr - (pgoff << PAGE_SHIFT)) & SHM_ALIGN_MASK)) return -EINVAL; return addr; } @@ -63,7 +62,7 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp, } info.length = len; - info.align_mask = do_color_align ? (PAGE_MASK & shm_align_mask) : 0; + info.align_mask = do_color_align ? (PAGE_MASK & SHM_ALIGN_MASK) : 0; info.align_offset = pgoff << PAGE_SHIFT; if (dir == DOWN) { diff --git a/arch/loongarch/mm/pgtable.c b/arch/loongarch/mm/pgtable.c index b14343e211b6..71d0539e2d0b 100644 --- a/arch/loongarch/mm/pgtable.c +++ b/arch/loongarch/mm/pgtable.c @@ -9,6 +9,18 @@ #include #include +struct page *dmw_virt_to_page(unsigned long kaddr) +{ + return pfn_to_page(virt_to_pfn(kaddr)); +} +EXPORT_SYMBOL_GPL(dmw_virt_to_page); + +struct page *tlb_virt_to_page(unsigned long kaddr) +{ + return pfn_to_page(pte_pfn(*virt_to_kpte(kaddr))); +} +EXPORT_SYMBOL_GPL(tlb_virt_to_page); + pgd_t *pgd_alloc(struct mm_struct *mm) { pgd_t *init, *ret = NULL; diff --git a/arch/loongarch/vdso/Makefile b/arch/loongarch/vdso/Makefile index a50308b6fc25..5c97d1463328 100644 --- a/arch/loongarch/vdso/Makefile +++ b/arch/loongarch/vdso/Makefile @@ -1,6 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 # Objects to go into the VDSO. +KASAN_SANITIZE := n +KCOV_INSTRUMENT := n + # Include the generic Makefile to check the built vdso. include $(srctree)/lib/vdso/Makefile diff --git a/arch/mips/Makefile b/arch/mips/Makefile index a47593d72f6f..f49807e1f19b 100644 --- a/arch/mips/Makefile +++ b/arch/mips/Makefile @@ -181,12 +181,16 @@ endif cflags-$(CONFIG_CAVIUM_CN63XXP1) += -Wa,-mfix-cn63xxp1 cflags-$(CONFIG_CPU_BMIPS) += -march=mips32 -Wa,-mips32 -Wa,--trap -cflags-$(CONFIG_CPU_LOONGSON2E) += $(call cc-option,-march=loongson2e) -Wa,--trap -cflags-$(CONFIG_CPU_LOONGSON2F) += $(call cc-option,-march=loongson2f) -Wa,--trap -cflags-$(CONFIG_CPU_LOONGSON64) += $(call cc-option,-march=loongson3a,-march=mips64r2) -Wa,--trap +cflags-$(CONFIG_CPU_LOONGSON2E) += -march=loongson2e -Wa,--trap +cflags-$(CONFIG_CPU_LOONGSON2F) += -march=loongson2f -Wa,--trap # Some -march= flags enable MMI instructions, and GCC complains about that # support being enabled alongside -msoft-float. Thus explicitly disable MMI. cflags-$(CONFIG_CPU_LOONGSON2EF) += $(call cc-option,-mno-loongson-mmi) +ifdef CONFIG_CPU_LOONGSON64 +cflags-$(CONFIG_CPU_LOONGSON64) += -Wa,--trap +cflags-$(CONFIG_CC_IS_GCC) += -march=loongson3a +cflags-$(CONFIG_CC_IS_CLANG) += -march=mips64r2 +endif cflags-$(CONFIG_CPU_LOONGSON64) += $(call cc-option,-mno-loongson-mmi) cflags-$(CONFIG_CPU_R4000_WORKAROUNDS) += $(call cc-option,-mfix-r4000,) @@ -299,8 +303,8 @@ ifdef CONFIG_64BIT endif endif - ifeq ($(KBUILD_SYM32)$(call cc-option-yn,-msym32), yy) - cflags-y += -msym32 -DKBUILD_64BIT_SYM32 + ifeq ($(KBUILD_SYM32), y) + cflags-$(KBUILD_SYM32) += -msym32 -DKBUILD_64BIT_SYM32 else ifeq ($(CONFIG_CPU_DADDI_WORKAROUNDS), y) $(error CONFIG_CPU_DADDI_WORKAROUNDS unsupported without -msym32) @@ -341,7 +345,7 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables KBUILD_LDFLAGS += -m $(ld-emul) -ifdef CONFIG_MIPS +ifdef need-compiler CHECKFLAGS += $(shell $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) -dM -E -x c /dev/null | \ grep -E -vw '__GNUC_(MINOR_|PATCHLEVEL_)?_' | \ sed -e "s/^\#define /-D'/" -e "s/ /'='/" -e "s/$$/'/" -e 's/\$$/&&/g') diff --git a/arch/mips/bmips/setup.c b/arch/mips/bmips/setup.c index 053805cb741c..ec180ab92eaa 100644 --- a/arch/mips/bmips/setup.c +++ b/arch/mips/bmips/setup.c @@ -16,7 +16,6 @@ #include #include #include -#include #include #include #include diff --git a/arch/mips/cavium-octeon/flash_setup.c b/arch/mips/cavium-octeon/flash_setup.c index c8a8c6d359b9..3395acde4d60 100644 --- a/arch/mips/cavium-octeon/flash_setup.c +++ b/arch/mips/cavium-octeon/flash_setup.c @@ -12,7 +12,8 @@ #include #include #include -#include +#include +#include #include #include diff --git a/arch/mips/cavium-octeon/octeon-memcpy.S b/arch/mips/cavium-octeon/octeon-memcpy.S index 25860fba6218..fef0c6de3fa1 100644 --- a/arch/mips/cavium-octeon/octeon-memcpy.S +++ b/arch/mips/cavium-octeon/octeon-memcpy.S @@ -13,9 +13,9 @@ * Mnemonic names for arguments to memcpy/__copy_user */ +#include #include #include -#include #include #define dst a0 diff --git a/arch/mips/cavium-octeon/octeon-platform.c b/arch/mips/cavium-octeon/octeon-platform.c index 235c77ce7b18..f76783c24338 100644 --- a/arch/mips/cavium-octeon/octeon-platform.c +++ b/arch/mips/cavium-octeon/octeon-platform.c @@ -8,8 +8,10 @@ */ #include +#include #include #include +#include #include #include diff --git a/arch/mips/configs/ip22_defconfig b/arch/mips/configs/ip22_defconfig index 44821f497261..dc49b09d492b 100644 --- a/arch/mips/configs/ip22_defconfig +++ b/arch/mips/configs/ip22_defconfig @@ -127,7 +127,6 @@ CONFIG_IP_NF_MATCH_TTL=m CONFIG_IP_NF_FILTER=m CONFIG_IP_NF_TARGET_REJECT=m CONFIG_IP_NF_MANGLE=m -CONFIG_IP_NF_TARGET_CLUSTERIP=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_RAW=m diff --git a/arch/mips/configs/loongson3_defconfig b/arch/mips/configs/loongson3_defconfig index 2b4133176930..07839a4b397e 100644 --- a/arch/mips/configs/loongson3_defconfig +++ b/arch/mips/configs/loongson3_defconfig @@ -283,6 +283,7 @@ CONFIG_DRM_AMDGPU_USERPTR=y CONFIG_DRM_AMD_ACP=y CONFIG_DRM_AMD_DC=y CONFIG_DRM_AMD_DC_SI=y +CONFIG_DRM_AST=m CONFIG_DRM_RADEON=m CONFIG_DRM_QXL=y CONFIG_DRM_VIRTIO_GPU=y diff --git a/arch/mips/configs/malta_defconfig b/arch/mips/configs/malta_defconfig index 743209047792..ae1a7793e810 100644 --- a/arch/mips/configs/malta_defconfig +++ b/arch/mips/configs/malta_defconfig @@ -127,7 +127,6 @@ CONFIG_IP_NF_MATCH_TTL=m CONFIG_IP_NF_FILTER=m CONFIG_IP_NF_TARGET_REJECT=m CONFIG_IP_NF_MANGLE=m -CONFIG_IP_NF_TARGET_CLUSTERIP=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_RAW=m diff --git a/arch/mips/configs/malta_kvm_defconfig b/arch/mips/configs/malta_kvm_defconfig index dd2b9c181f32..c07e30f63d8b 100644 --- a/arch/mips/configs/malta_kvm_defconfig +++ b/arch/mips/configs/malta_kvm_defconfig @@ -131,7 +131,6 @@ CONFIG_IP_NF_MATCH_TTL=m CONFIG_IP_NF_FILTER=m CONFIG_IP_NF_TARGET_REJECT=m CONFIG_IP_NF_MANGLE=m -CONFIG_IP_NF_TARGET_CLUSTERIP=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_RAW=m diff --git a/arch/mips/configs/maltaup_xpa_defconfig b/arch/mips/configs/maltaup_xpa_defconfig index 97c2d7f530b3..0a5701020d3f 100644 --- a/arch/mips/configs/maltaup_xpa_defconfig +++ b/arch/mips/configs/maltaup_xpa_defconfig @@ -128,7 +128,6 @@ CONFIG_IP_NF_MATCH_TTL=m CONFIG_IP_NF_FILTER=m CONFIG_IP_NF_TARGET_REJECT=m CONFIG_IP_NF_MANGLE=m -CONFIG_IP_NF_TARGET_CLUSTERIP=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_RAW=m diff --git a/arch/mips/configs/rm200_defconfig b/arch/mips/configs/rm200_defconfig index e0e312dd968a..5c5e2186210c 100644 --- a/arch/mips/configs/rm200_defconfig +++ b/arch/mips/configs/rm200_defconfig @@ -90,7 +90,6 @@ CONFIG_IP_NF_MATCH_TTL=m CONFIG_IP_NF_FILTER=m CONFIG_IP_NF_TARGET_REJECT=m CONFIG_IP_NF_MANGLE=m -CONFIG_IP_NF_TARGET_CLUSTERIP=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_RAW=m diff --git a/arch/mips/include/asm/Kbuild b/arch/mips/include/asm/Kbuild index dee172716581..7ba67a0d6c97 100644 --- a/arch/mips/include/asm/Kbuild +++ b/arch/mips/include/asm/Kbuild @@ -7,7 +7,6 @@ generated-y += unistd_nr_n32.h generated-y += unistd_nr_n64.h generated-y += unistd_nr_o32.h -generic-y += export.h generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += parport.h diff --git a/arch/mips/include/asm/kvm_host.h b/arch/mips/include/asm/kvm_host.h index 04cedf9f8811..54a85f1d4f2c 100644 --- a/arch/mips/include/asm/kvm_host.h +++ b/arch/mips/include/asm/kvm_host.h @@ -896,7 +896,6 @@ static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {} static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {} static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} -#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB -int kvm_arch_flush_remote_tlb(struct kvm *kvm); +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS #endif /* __MIPS_KVM_HOST_H__ */ diff --git a/arch/mips/include/asm/mach-loongson32/loongson1.h b/arch/mips/include/asm/mach-loongson32/loongson1.h index 7971272345d3..84f45461c832 100644 --- a/arch/mips/include/asm/mach-loongson32/loongson1.h +++ b/arch/mips/include/asm/mach-loongson32/loongson1.h @@ -45,8 +45,6 @@ #define LS1X_NAND_BASE 0x1fe78000 #define LS1X_CLK_BASE 0x1fe78030 -#include #include -#include #endif /* __ASM_MACH_LOONGSON32_LOONGSON1_H */ diff --git a/arch/mips/include/asm/mach-loongson32/regs-clk.h b/arch/mips/include/asm/mach-loongson32/regs-clk.h deleted file mode 100644 index 98136fa8bee1..000000000000 --- a/arch/mips/include/asm/mach-loongson32/regs-clk.h +++ /dev/null @@ -1,81 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * Copyright (c) 2011 Zhang, Keguang - * - * Loongson 1 Clock Register Definitions. - */ - -#ifndef __ASM_MACH_LOONGSON32_REGS_CLK_H -#define __ASM_MACH_LOONGSON32_REGS_CLK_H - -#define LS1X_CLK_REG(x) \ - ((void __iomem *)KSEG1ADDR(LS1X_CLK_BASE + (x))) - -#define LS1X_CLK_PLL_FREQ LS1X_CLK_REG(0x0) -#define LS1X_CLK_PLL_DIV LS1X_CLK_REG(0x4) - -#if defined(CONFIG_LOONGSON1_LS1B) -/* Clock PLL Divisor Register Bits */ -#define DIV_DC_EN BIT(31) -#define DIV_DC_RST BIT(30) -#define DIV_CPU_EN BIT(25) -#define DIV_CPU_RST BIT(24) -#define DIV_DDR_EN BIT(19) -#define DIV_DDR_RST BIT(18) -#define RST_DC_EN BIT(5) -#define RST_DC BIT(4) -#define RST_DDR_EN BIT(3) -#define RST_DDR BIT(2) -#define RST_CPU_EN BIT(1) -#define RST_CPU BIT(0) - -#define DIV_DC_SHIFT 26 -#define DIV_CPU_SHIFT 20 -#define DIV_DDR_SHIFT 14 - -#define DIV_DC_WIDTH 4 -#define DIV_CPU_WIDTH 4 -#define DIV_DDR_WIDTH 4 - -#define BYPASS_DC_SHIFT 12 -#define BYPASS_DDR_SHIFT 10 -#define BYPASS_CPU_SHIFT 8 - -#define BYPASS_DC_WIDTH 1 -#define BYPASS_DDR_WIDTH 1 -#define BYPASS_CPU_WIDTH 1 - -#elif defined(CONFIG_LOONGSON1_LS1C) -/* PLL/SDRAM Frequency configuration register Bits */ -#define PLL_VALID BIT(31) -#define FRAC_N GENMASK(23, 16) -#define RST_TIME GENMASK(3, 2) -#define SDRAM_DIV GENMASK(1, 0) - -/* CPU/CAMERA/DC Frequency configuration register Bits */ -#define DIV_DC_EN BIT(31) -#define DIV_DC GENMASK(30, 24) -#define DIV_CAM_EN BIT(23) -#define DIV_CAM GENMASK(22, 16) -#define DIV_CPU_EN BIT(15) -#define DIV_CPU GENMASK(14, 8) -#define DIV_DC_SEL_EN BIT(5) -#define DIV_DC_SEL BIT(4) -#define DIV_CAM_SEL_EN BIT(3) -#define DIV_CAM_SEL BIT(2) -#define DIV_CPU_SEL_EN BIT(1) -#define DIV_CPU_SEL BIT(0) - -#define DIV_DC_SHIFT 24 -#define DIV_CAM_SHIFT 16 -#define DIV_CPU_SHIFT 8 -#define DIV_DDR_SHIFT 0 - -#define DIV_DC_WIDTH 7 -#define DIV_CAM_WIDTH 7 -#define DIV_CPU_WIDTH 7 -#define DIV_DDR_WIDTH 2 - -#endif - -#endif /* __ASM_MACH_LOONGSON32_REGS_CLK_H */ diff --git a/arch/mips/include/asm/mach-loongson32/regs-rtc.h b/arch/mips/include/asm/mach-loongson32/regs-rtc.h deleted file mode 100644 index a3d096be1607..000000000000 --- a/arch/mips/include/asm/mach-loongson32/regs-rtc.h +++ /dev/null @@ -1,19 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * Copyright (c) 2016 Yang Ling - * - * Loongson 1 RTC timer Register Definitions. - */ - -#ifndef __ASM_MACH_LOONGSON32_REGS_RTC_H -#define __ASM_MACH_LOONGSON32_REGS_RTC_H - -#define LS1X_RTC_REG(x) \ - ((void __iomem *)KSEG1ADDR(LS1X_RTC_BASE + (x))) - -#define LS1X_RTC_CTRL LS1X_RTC_REG(0x40) - -#define RTC_EXTCLK_OK (BIT(5) | BIT(8)) -#define RTC_EXTCLK_EN BIT(8) - -#endif /* __ASM_MACH_LOONGSON32_REGS_RTC_H */ diff --git a/arch/mips/kernel/mcount.S b/arch/mips/kernel/mcount.S index cff52b283e03..fcec579f64e9 100644 --- a/arch/mips/kernel/mcount.S +++ b/arch/mips/kernel/mcount.S @@ -10,7 +10,7 @@ * Author: Wu Zhangjin */ -#include +#include #include #include #include diff --git a/arch/mips/kernel/octeon_switch.S b/arch/mips/kernel/octeon_switch.S index 9b7c8ab6f08c..447a3ea14aa1 100644 --- a/arch/mips/kernel/octeon_switch.S +++ b/arch/mips/kernel/octeon_switch.S @@ -11,7 +11,6 @@ * written by Carsten Langgaard, carstenl@mips.com */ #include -#include #include #include #include diff --git a/arch/mips/kernel/r2300_fpu.S b/arch/mips/kernel/r2300_fpu.S index 6c745aa9e825..c000b22e3fcd 100644 --- a/arch/mips/kernel/r2300_fpu.S +++ b/arch/mips/kernel/r2300_fpu.S @@ -11,10 +11,10 @@ * Further modifications to make this work: * Copyright (c) 1998 Harald Koerfgen */ +#include #include #include #include -#include #include #include #include diff --git a/arch/mips/kernel/r2300_switch.S b/arch/mips/kernel/r2300_switch.S index 71b1aafae1bb..48e63943e6f7 100644 --- a/arch/mips/kernel/r2300_switch.S +++ b/arch/mips/kernel/r2300_switch.S @@ -13,7 +13,6 @@ */ #include #include -#include #include #include #include diff --git a/arch/mips/kernel/r4k_fpu.S b/arch/mips/kernel/r4k_fpu.S index 4e8c98517d9d..4bb97ee89904 100644 --- a/arch/mips/kernel/r4k_fpu.S +++ b/arch/mips/kernel/r4k_fpu.S @@ -12,10 +12,10 @@ * Copyright (C) 2000 MIPS Technologies, Inc. * Copyright (C) 1999, 2001 Silicon Graphics, Inc. */ +#include #include #include #include -#include #include #include #include diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index aa5583a7b05b..231ac052b506 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -199,7 +199,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, /* Flush slot from GPA */ kvm_mips_flush_gpa_pt(kvm, slot->base_gfn, slot->base_gfn + slot->npages - 1); - kvm_arch_flush_remote_tlbs_memslot(kvm, slot); + kvm_flush_remote_tlbs_memslot(kvm, slot); spin_unlock(&kvm->mmu_lock); } @@ -235,7 +235,7 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, needs_flush = kvm_mips_mkclean_gpa_pt(kvm, new->base_gfn, new->base_gfn + new->npages - 1); if (needs_flush) - kvm_arch_flush_remote_tlbs_memslot(kvm, new); + kvm_flush_remote_tlbs_memslot(kvm, new); spin_unlock(&kvm->mmu_lock); } } @@ -981,18 +981,12 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) } -int kvm_arch_flush_remote_tlb(struct kvm *kvm) +int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { kvm_mips_callbacks->prepare_flush_shadow(kvm); return 1; } -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot) -{ - kvm_flush_remote_tlbs(kvm); -} - int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { int r; diff --git a/arch/mips/kvm/mmu.c b/arch/mips/kvm/mmu.c index e8c08988ed37..7b2ac1319d70 100644 --- a/arch/mips/kvm/mmu.c +++ b/arch/mips/kvm/mmu.c @@ -447,7 +447,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { gpa_t gpa = range->start << PAGE_SHIFT; - pte_t hva_pte = range->pte; + pte_t hva_pte = range->arg.pte; pte_t *gpa_pte = kvm_mips_pte_for_gpa(kvm, NULL, gpa); pte_t old_pte; diff --git a/arch/mips/lantiq/irq.c b/arch/mips/lantiq/irq.c index 20622bf0a9b3..8f208007b8e8 100644 --- a/arch/mips/lantiq/irq.c +++ b/arch/mips/lantiq/irq.c @@ -10,7 +10,7 @@ #include #include #include -#include +#include #include #include diff --git a/arch/mips/lantiq/xway/dcdc.c b/arch/mips/lantiq/xway/dcdc.c index 96199966a350..4a808f8c5beb 100644 --- a/arch/mips/lantiq/xway/dcdc.c +++ b/arch/mips/lantiq/xway/dcdc.c @@ -6,7 +6,8 @@ */ #include -#include +#include +#include #include diff --git a/arch/mips/lantiq/xway/gptu.c b/arch/mips/lantiq/xway/gptu.c index a492b1eb1925..8d52001301de 100644 --- a/arch/mips/lantiq/xway/gptu.c +++ b/arch/mips/lantiq/xway/gptu.c @@ -8,8 +8,9 @@ #include #include #include -#include +#include #include +#include #include #include "../clk.h" diff --git a/arch/mips/lantiq/xway/sysctrl.c b/arch/mips/lantiq/xway/sysctrl.c index d444a1b98a72..3ed078225222 100644 --- a/arch/mips/lantiq/xway/sysctrl.c +++ b/arch/mips/lantiq/xway/sysctrl.c @@ -10,7 +10,6 @@ #include #include #include -#include #include #include diff --git a/arch/mips/lantiq/xway/vmmc.c b/arch/mips/lantiq/xway/vmmc.c index 2796e87dfcae..37c133052ef7 100644 --- a/arch/mips/lantiq/xway/vmmc.c +++ b/arch/mips/lantiq/xway/vmmc.c @@ -7,7 +7,8 @@ #include #include #include -#include +#include +#include #include #include diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S index 7767137c3e49..3d2ff4118d79 100644 --- a/arch/mips/lib/csum_partial.S +++ b/arch/mips/lib/csum_partial.S @@ -11,9 +11,9 @@ * Copyright (C) 2014 Imagination Technologies Ltd. */ #include +#include #include #include -#include #include #ifdef CONFIG_64BIT diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S index 18a43f2e29c8..a4b4e805ff13 100644 --- a/arch/mips/lib/memcpy.S +++ b/arch/mips/lib/memcpy.S @@ -32,9 +32,9 @@ #undef CONFIG_CPU_HAS_PREFETCH #endif +#include #include #include -#include #include #define dst a0 diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S index 0b342bae9a98..79405c32cc85 100644 --- a/arch/mips/lib/memset.S +++ b/arch/mips/lib/memset.S @@ -8,9 +8,9 @@ * Copyright (C) 2007 by Maciej W. Rozycki * Copyright (C) 2011, 2012 MIPS Technologies, Inc. */ +#include #include #include -#include #include #if LONGSIZE == 4 diff --git a/arch/mips/lib/strncpy_user.S b/arch/mips/lib/strncpy_user.S index 13aaa9927ad1..94f4203563c1 100644 --- a/arch/mips/lib/strncpy_user.S +++ b/arch/mips/lib/strncpy_user.S @@ -7,9 +7,9 @@ * Copyright (C) 2011 MIPS Technologies, Inc. */ #include +#include #include #include -#include #include #define EX(insn,reg,addr,handler) \ diff --git a/arch/mips/lib/strnlen_user.S b/arch/mips/lib/strnlen_user.S index 6de31b616f9c..c192a6f6cd84 100644 --- a/arch/mips/lib/strnlen_user.S +++ b/arch/mips/lib/strnlen_user.S @@ -6,9 +6,9 @@ * Copyright (c) 1996, 1998, 1999, 2004 by Ralf Baechle * Copyright (c) 1999 Silicon Graphics, Inc. */ +#include #include #include -#include #include #define EX(insn,reg,addr,handler) \ diff --git a/arch/mips/loongson32/common/platform.c b/arch/mips/loongson32/common/platform.c index 64d7979394e6..8075590a9f83 100644 --- a/arch/mips/loongson32/common/platform.c +++ b/arch/mips/loongson32/common/platform.c @@ -265,14 +265,6 @@ struct platform_device ls1x_ehci_pdev = { }; /* Real Time Clock */ -void __init ls1x_rtc_set_extclk(struct platform_device *pdev) -{ - u32 val = __raw_readl(LS1X_RTC_CTRL); - - if (!(val & RTC_EXTCLK_OK)) - __raw_writel(val | RTC_EXTCLK_EN, LS1X_RTC_CTRL); -} - struct platform_device ls1x_rtc_pdev = { .name = "ls1x-rtc", .id = -1, diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c index cdecd7af11a6..e015a26a40f7 100644 --- a/arch/mips/loongson64/smp.c +++ b/arch/mips/loongson64/smp.c @@ -187,181 +187,181 @@ static void csr_ipi_probe(void) static void ipi_set0_regs_init(void) { - ipi_set0_regs[0] = (void *) + ipi_set0_regs[0] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE0_OFFSET + SET0); - ipi_set0_regs[1] = (void *) + ipi_set0_regs[1] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE1_OFFSET + SET0); - ipi_set0_regs[2] = (void *) + ipi_set0_regs[2] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE2_OFFSET + SET0); - ipi_set0_regs[3] = (void *) + ipi_set0_regs[3] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE3_OFFSET + SET0); - ipi_set0_regs[4] = (void *) + ipi_set0_regs[4] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE0_OFFSET + SET0); - ipi_set0_regs[5] = (void *) + ipi_set0_regs[5] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE1_OFFSET + SET0); - ipi_set0_regs[6] = (void *) + ipi_set0_regs[6] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE2_OFFSET + SET0); - ipi_set0_regs[7] = (void *) + ipi_set0_regs[7] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE3_OFFSET + SET0); - ipi_set0_regs[8] = (void *) + ipi_set0_regs[8] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE0_OFFSET + SET0); - ipi_set0_regs[9] = (void *) + ipi_set0_regs[9] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE1_OFFSET + SET0); - ipi_set0_regs[10] = (void *) + ipi_set0_regs[10] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE2_OFFSET + SET0); - ipi_set0_regs[11] = (void *) + ipi_set0_regs[11] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE3_OFFSET + SET0); - ipi_set0_regs[12] = (void *) + ipi_set0_regs[12] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE0_OFFSET + SET0); - ipi_set0_regs[13] = (void *) + ipi_set0_regs[13] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE1_OFFSET + SET0); - ipi_set0_regs[14] = (void *) + ipi_set0_regs[14] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE2_OFFSET + SET0); - ipi_set0_regs[15] = (void *) + ipi_set0_regs[15] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE3_OFFSET + SET0); } static void ipi_clear0_regs_init(void) { - ipi_clear0_regs[0] = (void *) + ipi_clear0_regs[0] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE0_OFFSET + CLEAR0); - ipi_clear0_regs[1] = (void *) + ipi_clear0_regs[1] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE1_OFFSET + CLEAR0); - ipi_clear0_regs[2] = (void *) + ipi_clear0_regs[2] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE2_OFFSET + CLEAR0); - ipi_clear0_regs[3] = (void *) + ipi_clear0_regs[3] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE3_OFFSET + CLEAR0); - ipi_clear0_regs[4] = (void *) + ipi_clear0_regs[4] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE0_OFFSET + CLEAR0); - ipi_clear0_regs[5] = (void *) + ipi_clear0_regs[5] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE1_OFFSET + CLEAR0); - ipi_clear0_regs[6] = (void *) + ipi_clear0_regs[6] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE2_OFFSET + CLEAR0); - ipi_clear0_regs[7] = (void *) + ipi_clear0_regs[7] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE3_OFFSET + CLEAR0); - ipi_clear0_regs[8] = (void *) + ipi_clear0_regs[8] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE0_OFFSET + CLEAR0); - ipi_clear0_regs[9] = (void *) + ipi_clear0_regs[9] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE1_OFFSET + CLEAR0); - ipi_clear0_regs[10] = (void *) + ipi_clear0_regs[10] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE2_OFFSET + CLEAR0); - ipi_clear0_regs[11] = (void *) + ipi_clear0_regs[11] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE3_OFFSET + CLEAR0); - ipi_clear0_regs[12] = (void *) + ipi_clear0_regs[12] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE0_OFFSET + CLEAR0); - ipi_clear0_regs[13] = (void *) + ipi_clear0_regs[13] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE1_OFFSET + CLEAR0); - ipi_clear0_regs[14] = (void *) + ipi_clear0_regs[14] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE2_OFFSET + CLEAR0); - ipi_clear0_regs[15] = (void *) + ipi_clear0_regs[15] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE3_OFFSET + CLEAR0); } static void ipi_status0_regs_init(void) { - ipi_status0_regs[0] = (void *) + ipi_status0_regs[0] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE0_OFFSET + STATUS0); - ipi_status0_regs[1] = (void *) + ipi_status0_regs[1] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE1_OFFSET + STATUS0); - ipi_status0_regs[2] = (void *) + ipi_status0_regs[2] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE2_OFFSET + STATUS0); - ipi_status0_regs[3] = (void *) + ipi_status0_regs[3] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE3_OFFSET + STATUS0); - ipi_status0_regs[4] = (void *) + ipi_status0_regs[4] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE0_OFFSET + STATUS0); - ipi_status0_regs[5] = (void *) + ipi_status0_regs[5] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE1_OFFSET + STATUS0); - ipi_status0_regs[6] = (void *) + ipi_status0_regs[6] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE2_OFFSET + STATUS0); - ipi_status0_regs[7] = (void *) + ipi_status0_regs[7] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE3_OFFSET + STATUS0); - ipi_status0_regs[8] = (void *) + ipi_status0_regs[8] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE0_OFFSET + STATUS0); - ipi_status0_regs[9] = (void *) + ipi_status0_regs[9] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE1_OFFSET + STATUS0); - ipi_status0_regs[10] = (void *) + ipi_status0_regs[10] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE2_OFFSET + STATUS0); - ipi_status0_regs[11] = (void *) + ipi_status0_regs[11] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE3_OFFSET + STATUS0); - ipi_status0_regs[12] = (void *) + ipi_status0_regs[12] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE0_OFFSET + STATUS0); - ipi_status0_regs[13] = (void *) + ipi_status0_regs[13] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE1_OFFSET + STATUS0); - ipi_status0_regs[14] = (void *) + ipi_status0_regs[14] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE2_OFFSET + STATUS0); - ipi_status0_regs[15] = (void *) + ipi_status0_regs[15] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE3_OFFSET + STATUS0); } static void ipi_en0_regs_init(void) { - ipi_en0_regs[0] = (void *) + ipi_en0_regs[0] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE0_OFFSET + EN0); - ipi_en0_regs[1] = (void *) + ipi_en0_regs[1] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE1_OFFSET + EN0); - ipi_en0_regs[2] = (void *) + ipi_en0_regs[2] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE2_OFFSET + EN0); - ipi_en0_regs[3] = (void *) + ipi_en0_regs[3] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE3_OFFSET + EN0); - ipi_en0_regs[4] = (void *) + ipi_en0_regs[4] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE0_OFFSET + EN0); - ipi_en0_regs[5] = (void *) + ipi_en0_regs[5] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE1_OFFSET + EN0); - ipi_en0_regs[6] = (void *) + ipi_en0_regs[6] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE2_OFFSET + EN0); - ipi_en0_regs[7] = (void *) + ipi_en0_regs[7] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE3_OFFSET + EN0); - ipi_en0_regs[8] = (void *) + ipi_en0_regs[8] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE0_OFFSET + EN0); - ipi_en0_regs[9] = (void *) + ipi_en0_regs[9] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE1_OFFSET + EN0); - ipi_en0_regs[10] = (void *) + ipi_en0_regs[10] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE2_OFFSET + EN0); - ipi_en0_regs[11] = (void *) + ipi_en0_regs[11] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE3_OFFSET + EN0); - ipi_en0_regs[12] = (void *) + ipi_en0_regs[12] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE0_OFFSET + EN0); - ipi_en0_regs[13] = (void *) + ipi_en0_regs[13] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE1_OFFSET + EN0); - ipi_en0_regs[14] = (void *) + ipi_en0_regs[14] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE2_OFFSET + EN0); - ipi_en0_regs[15] = (void *) + ipi_en0_regs[15] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE3_OFFSET + EN0); } static void ipi_mailbox_buf_init(void) { - ipi_mailbox_buf[0] = (void *) + ipi_mailbox_buf[0] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE0_OFFSET + BUF); - ipi_mailbox_buf[1] = (void *) + ipi_mailbox_buf[1] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE1_OFFSET + BUF); - ipi_mailbox_buf[2] = (void *) + ipi_mailbox_buf[2] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE2_OFFSET + BUF); - ipi_mailbox_buf[3] = (void *) + ipi_mailbox_buf[3] = (void __iomem *) (SMP_CORE_GROUP0_BASE + SMP_CORE3_OFFSET + BUF); - ipi_mailbox_buf[4] = (void *) + ipi_mailbox_buf[4] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE0_OFFSET + BUF); - ipi_mailbox_buf[5] = (void *) + ipi_mailbox_buf[5] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE1_OFFSET + BUF); - ipi_mailbox_buf[6] = (void *) + ipi_mailbox_buf[6] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE2_OFFSET + BUF); - ipi_mailbox_buf[7] = (void *) + ipi_mailbox_buf[7] = (void __iomem *) (SMP_CORE_GROUP1_BASE + SMP_CORE3_OFFSET + BUF); - ipi_mailbox_buf[8] = (void *) + ipi_mailbox_buf[8] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE0_OFFSET + BUF); - ipi_mailbox_buf[9] = (void *) + ipi_mailbox_buf[9] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE1_OFFSET + BUF); - ipi_mailbox_buf[10] = (void *) + ipi_mailbox_buf[10] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE2_OFFSET + BUF); - ipi_mailbox_buf[11] = (void *) + ipi_mailbox_buf[11] = (void __iomem *) (SMP_CORE_GROUP2_BASE + SMP_CORE3_OFFSET + BUF); - ipi_mailbox_buf[12] = (void *) + ipi_mailbox_buf[12] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE0_OFFSET + BUF); - ipi_mailbox_buf[13] = (void *) + ipi_mailbox_buf[13] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE1_OFFSET + BUF); - ipi_mailbox_buf[14] = (void *) + ipi_mailbox_buf[14] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE2_OFFSET + BUF); - ipi_mailbox_buf[15] = (void *) + ipi_mailbox_buf[15] = (void __iomem *) (SMP_CORE_GROUP3_BASE + SMP_CORE3_OFFSET + BUF); } diff --git a/arch/mips/mm/page-funcs.S b/arch/mips/mm/page-funcs.S index 43181ac0a1af..42d0516ca18a 100644 --- a/arch/mips/mm/page-funcs.S +++ b/arch/mips/mm/page-funcs.S @@ -8,8 +8,8 @@ * Copyright (C) 2012 MIPS Technologies, Inc. * Copyright (C) 2012 Ralf Baechle */ +#include #include -#include #include #ifdef CONFIG_SIBYTE_DMA_PAGEOPS diff --git a/arch/mips/mm/tlb-funcs.S b/arch/mips/mm/tlb-funcs.S index 00fef578c8cd..2705d7dcb33e 100644 --- a/arch/mips/mm/tlb-funcs.S +++ b/arch/mips/mm/tlb-funcs.S @@ -11,8 +11,8 @@ * Copyright (C) 2012 MIPS Technologies, Inc. * Copyright (C) 2012 Ralf Baechle */ +#include #include -#include #include #define FASTPATH_SIZE 128 diff --git a/arch/mips/pci/pci-lantiq.c b/arch/mips/pci/pci-lantiq.c index 79e29bf42a24..80f7293166bb 100644 --- a/arch/mips/pci/pci-lantiq.c +++ b/arch/mips/pci/pci-lantiq.c @@ -13,9 +13,9 @@ #include #include #include -#include -#include +#include #include +#include #include diff --git a/arch/mips/pci/pci-rt2880.c b/arch/mips/pci/pci-rt2880.c index e9dd01431f21..1cada09fa5db 100644 --- a/arch/mips/pci/pci-rt2880.c +++ b/arch/mips/pci/pci-rt2880.c @@ -13,9 +13,8 @@ #include #include #include -#include -#include -#include +#include +#include #include diff --git a/arch/mips/pic32/pic32mzda/config.c b/arch/mips/pic32/pic32mzda/config.c index f69532007717..73be5689e0df 100644 --- a/arch/mips/pic32/pic32mzda/config.c +++ b/arch/mips/pic32/pic32mzda/config.c @@ -5,7 +5,7 @@ */ #include #include -#include +#include #include diff --git a/arch/mips/ralink/ill_acc.c b/arch/mips/ralink/ill_acc.c index f395ae218470..25341b2319d0 100644 --- a/arch/mips/ralink/ill_acc.c +++ b/arch/mips/ralink/ill_acc.c @@ -5,8 +5,10 @@ */ #include +#include #include #include +#include #include diff --git a/arch/mips/ralink/irq.c b/arch/mips/ralink/irq.c index fa353bc13947..46aef0a1b22a 100644 --- a/arch/mips/ralink/irq.c +++ b/arch/mips/ralink/irq.c @@ -7,7 +7,7 @@ #include #include -#include +#include #include #include #include diff --git a/arch/mips/ralink/of.c b/arch/mips/ralink/of.c index 45d60c094496..7f90068c68f2 100644 --- a/arch/mips/ralink/of.c +++ b/arch/mips/ralink/of.c @@ -14,7 +14,7 @@ #include #include #include -#include +#include #include #include diff --git a/arch/mips/ralink/prom.c b/arch/mips/ralink/prom.c index aaac1e6ec7d9..c3b96861844c 100644 --- a/arch/mips/ralink/prom.c +++ b/arch/mips/ralink/prom.c @@ -7,8 +7,6 @@ */ #include -#include -#include #include #include diff --git a/arch/mips/txx9/generic/pci.c b/arch/mips/txx9/generic/pci.c index e98845543b77..5ae30b78d38d 100644 --- a/arch/mips/txx9/generic/pci.c +++ b/arch/mips/txx9/generic/pci.c @@ -51,6 +51,7 @@ int __init txx9_pci66_check(struct pci_controller *hose, int top_bus, unsigned short vid; int cap66 = -1; u16 stat; + int ret; /* It seems SLC90E66 needs some time after PCI reset... */ mdelay(80); @@ -60,9 +61,9 @@ int __init txx9_pci66_check(struct pci_controller *hose, int top_bus, for (pci_devfn = 0; pci_devfn < 0xff; pci_devfn++) { if (PCI_FUNC(pci_devfn)) continue; - if (early_read_config_word(hose, top_bus, current_bus, - pci_devfn, PCI_VENDOR_ID, &vid) != - PCIBIOS_SUCCESSFUL) + ret = early_read_config_word(hose, top_bus, current_bus, + pci_devfn, PCI_VENDOR_ID, &vid); + if (ret != PCIBIOS_SUCCESSFUL) continue; if (vid == 0xffff) continue; @@ -343,26 +344,28 @@ static void tc35815_fixup(struct pci_dev *dev) static void final_fixup(struct pci_dev *dev) { + unsigned long timeout; unsigned char bist; + int ret; /* Do build-in self test */ - if (pci_read_config_byte(dev, PCI_BIST, &bist) == PCIBIOS_SUCCESSFUL && - (bist & PCI_BIST_CAPABLE)) { - unsigned long timeout; - pci_set_power_state(dev, PCI_D0); - pr_info("PCI: %s BIST...", pci_name(dev)); - pci_write_config_byte(dev, PCI_BIST, PCI_BIST_START); - timeout = jiffies + HZ * 2; /* timeout after 2 sec */ - do { - pci_read_config_byte(dev, PCI_BIST, &bist); - if (time_after(jiffies, timeout)) - break; - } while (bist & PCI_BIST_START); - if (bist & (PCI_BIST_CODE_MASK | PCI_BIST_START)) - pr_cont("failed. (0x%x)\n", bist); - else - pr_cont("OK.\n"); - } + ret = pci_read_config_byte(dev, PCI_BIST, &bist); + if ((ret != PCIBIOS_SUCCESSFUL) || !(bist & PCI_BIST_CAPABLE)) + return; + + pci_set_power_state(dev, PCI_D0); + pr_info("PCI: %s BIST...", pci_name(dev)); + pci_write_config_byte(dev, PCI_BIST, PCI_BIST_START); + timeout = jiffies + HZ * 2; /* timeout after 2 sec */ + do { + pci_read_config_byte(dev, PCI_BIST, &bist); + if (time_after(jiffies, timeout)) + break; + } while (bist & PCI_BIST_START); + if (bist & (PCI_BIST_CODE_MASK | PCI_BIST_START)) + pr_cont("failed. (0x%x)\n", bist); + else + pr_cont("OK.\n"); } #ifdef CONFIG_TOSHIBA_FPCIB0 diff --git a/arch/mips/vdso/vdso.lds.S b/arch/mips/vdso/vdso.lds.S index d90b65724d78..836465e3bcb8 100644 --- a/arch/mips/vdso/vdso.lds.S +++ b/arch/mips/vdso/vdso.lds.S @@ -94,7 +94,9 @@ VERSION #ifndef CONFIG_MIPS_DISABLE_VDSO global: __vdso_clock_gettime; +#ifdef CONFIG_MIPS_CLOCK_VSYSCALL __vdso_gettimeofday; +#endif __vdso_clock_getres; #if _MIPS_SIM != _MIPS_SIM_ABI64 __vdso_clock_gettime64; diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index c9151577a342..497c419dcce3 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -274,7 +274,14 @@ config RISCV_DMA_NONCOHERENT select ARCH_HAS_SYNC_DMA_FOR_CPU select ARCH_HAS_SYNC_DMA_FOR_DEVICE select DMA_BOUNCE_UNALIGNED_KMALLOC if SWIOTLB - select DMA_DIRECT_REMAP + select DMA_DIRECT_REMAP if MMU + +config RISCV_NONSTANDARD_CACHE_OPS + bool + depends on RISCV_DMA_NONCOHERENT + help + This enables function pointer support for non-standard noncoherent + systems to handle cache management. config AS_HAS_INSN def_bool $(as-instr,.insn r 51$(comma) 0$(comma) 0$(comma) t0$(comma) t0$(comma) zero) @@ -714,6 +721,25 @@ config RELOCATABLE If unsure, say N. +config RANDOMIZE_BASE + bool "Randomize the address of the kernel image" + select RELOCATABLE + depends on MMU && 64BIT && !XIP_KERNEL + help + Randomizes the virtual address at which the kernel image is + loaded, as a security feature that deters exploit attempts + relying on knowledge of the location of kernel internals. + + It is the bootloader's job to provide entropy, by passing a + random u64 value in /chosen/kaslr-seed at kernel entry. + + When booting via the UEFI stub, it will invoke the firmware's + EFI_RNG_PROTOCOL implementation (if available) to supply entropy + to the kernel proper. In addition, it will randomise the physical + location of the kernel Image as well. + + If unsure, say N. + endmenu # "Kernel features" menu "Boot options" diff --git a/arch/riscv/Kconfig.errata b/arch/riscv/Kconfig.errata index 0c8f4652cd82..566bcefeab50 100644 --- a/arch/riscv/Kconfig.errata +++ b/arch/riscv/Kconfig.errata @@ -1,5 +1,26 @@ menu "CPU errata selection" +config ERRATA_ANDES + bool "Andes AX45MP errata" + depends on RISCV_ALTERNATIVE && RISCV_SBI + help + All Andes errata Kconfig depend on this Kconfig. Disabling + this Kconfig will disable all Andes errata. Please say "Y" + here if your platform uses Andes CPU cores. + + Otherwise, please say "N" here to avoid unnecessary overhead. + +config ERRATA_ANDES_CMO + bool "Apply Andes cache management errata" + depends on ERRATA_ANDES && ARCH_R9A07G043 + select RISCV_DMA_NONCOHERENT + default y + help + This will apply the cache management errata to handle the + non-standard handling on non-coherent operations on Andes cores. + + If you don't know what to do here, say "Y". + config ERRATA_SIFIVE bool "SiFive errata" depends on RISCV_ALTERNATIVE diff --git a/arch/riscv/errata/Makefile b/arch/riscv/errata/Makefile index 7b2637c8c332..8a2739485123 100644 --- a/arch/riscv/errata/Makefile +++ b/arch/riscv/errata/Makefile @@ -2,5 +2,6 @@ ifdef CONFIG_RELOCATABLE KBUILD_CFLAGS += -fno-pie endif +obj-$(CONFIG_ERRATA_ANDES) += andes/ obj-$(CONFIG_ERRATA_SIFIVE) += sifive/ obj-$(CONFIG_ERRATA_THEAD) += thead/ diff --git a/arch/riscv/errata/andes/Makefile b/arch/riscv/errata/andes/Makefile new file mode 100644 index 000000000000..2d644e19caef --- /dev/null +++ b/arch/riscv/errata/andes/Makefile @@ -0,0 +1 @@ +obj-y += errata.o diff --git a/arch/riscv/errata/andes/errata.c b/arch/riscv/errata/andes/errata.c new file mode 100644 index 000000000000..197db68cc8da --- /dev/null +++ b/arch/riscv/errata/andes/errata.c @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Erratas to be applied for Andes CPU cores + * + * Copyright (C) 2023 Renesas Electronics Corporation. + * + * Author: Lad Prabhakar + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#define ANDESTECH_AX45MP_MARCHID 0x8000000000008a45UL +#define ANDESTECH_AX45MP_MIMPID 0x500UL +#define ANDESTECH_SBI_EXT_ANDES 0x0900031E + +#define ANDES_SBI_EXT_IOCP_SW_WORKAROUND 1 + +static long ax45mp_iocp_sw_workaround(void) +{ + struct sbiret ret; + + /* + * ANDES_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP is missing and + * cache is controllable only then CMO will be applied to the platform. + */ + ret = sbi_ecall(ANDESTECH_SBI_EXT_ANDES, ANDES_SBI_EXT_IOCP_SW_WORKAROUND, + 0, 0, 0, 0, 0, 0); + + return ret.error ? 0 : ret.value; +} + +static bool errata_probe_iocp(unsigned int stage, unsigned long arch_id, unsigned long impid) +{ + if (!IS_ENABLED(CONFIG_ERRATA_ANDES_CMO)) + return false; + + if (arch_id != ANDESTECH_AX45MP_MARCHID || impid != ANDESTECH_AX45MP_MIMPID) + return false; + + if (!ax45mp_iocp_sw_workaround()) + return false; + + /* Set this just to make core cbo code happy */ + riscv_cbom_block_size = 1; + riscv_noncoherent_supported(); + + return true; +} + +void __init_or_module andes_errata_patch_func(struct alt_entry *begin, struct alt_entry *end, + unsigned long archid, unsigned long impid, + unsigned int stage) +{ + errata_probe_iocp(stage, archid, impid); + + /* we have nothing to patch here ATM so just return back */ +} diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c index be84b14f0118..0554ed4bf087 100644 --- a/arch/riscv/errata/thead/errata.c +++ b/arch/riscv/errata/thead/errata.c @@ -120,11 +120,3 @@ void thead_errata_patch_func(struct alt_entry *begin, struct alt_entry *end, if (stage == RISCV_ALTERNATIVES_EARLY_BOOT) local_flush_icache_all(); } - -void thead_feature_probe_func(unsigned int cpu, - unsigned long archid, - unsigned long impid) -{ - if ((archid == 0) && (impid == 0)) - per_cpu(misaligned_access_speed, cpu) = RISCV_HWPROBE_MISALIGNED_FAST; -} diff --git a/arch/riscv/include/asm/alternative.h b/arch/riscv/include/asm/alternative.h index 6a41537826a7..3c2b59b25017 100644 --- a/arch/riscv/include/asm/alternative.h +++ b/arch/riscv/include/asm/alternative.h @@ -30,7 +30,6 @@ #define ALT_OLD_PTR(a) __ALT_PTR(a, old_offset) #define ALT_ALT_PTR(a) __ALT_PTR(a, alt_offset) -void probe_vendor_features(unsigned int cpu); void __init apply_boot_alternatives(void); void __init apply_early_boot_alternatives(void); void apply_module_alternatives(void *start, size_t length); @@ -46,6 +45,9 @@ struct alt_entry { u32 patch_id; /* The patch ID (erratum ID or cpufeature ID) */ }; +void andes_errata_patch_func(struct alt_entry *begin, struct alt_entry *end, + unsigned long archid, unsigned long impid, + unsigned int stage); void sifive_errata_patch_func(struct alt_entry *begin, struct alt_entry *end, unsigned long archid, unsigned long impid, unsigned int stage); @@ -53,15 +55,11 @@ void thead_errata_patch_func(struct alt_entry *begin, struct alt_entry *end, unsigned long archid, unsigned long impid, unsigned int stage); -void thead_feature_probe_func(unsigned int cpu, unsigned long archid, - unsigned long impid); - void riscv_cpufeature_patch_func(struct alt_entry *begin, struct alt_entry *end, unsigned int stage); #else /* CONFIG_RISCV_ALTERNATIVE */ -static inline void probe_vendor_features(unsigned int cpu) { } static inline void apply_boot_alternatives(void) { } static inline void apply_early_boot_alternatives(void) { } static inline void apply_module_alternatives(void *start, size_t length) { } diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 23fed53b8815..d0345bd659c9 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -30,4 +30,6 @@ DECLARE_PER_CPU(long, misaligned_access_speed); /* Per-cpu ISA extensions. */ extern struct riscv_isainfo hart_isa[NR_CPUS]; +void check_unaligned_access(int cpu); + #endif diff --git a/arch/riscv/include/asm/csr.h b/arch/riscv/include/asm/csr.h index 7bac43a3176e..777cb8299551 100644 --- a/arch/riscv/include/asm/csr.h +++ b/arch/riscv/include/asm/csr.h @@ -54,6 +54,7 @@ #ifndef CONFIG_64BIT #define SATP_PPN _AC(0x003FFFFF, UL) #define SATP_MODE_32 _AC(0x80000000, UL) +#define SATP_MODE_SHIFT 31 #define SATP_ASID_BITS 9 #define SATP_ASID_SHIFT 22 #define SATP_ASID_MASK _AC(0x1FF, UL) @@ -62,6 +63,7 @@ #define SATP_MODE_39 _AC(0x8000000000000000, UL) #define SATP_MODE_48 _AC(0x9000000000000000, UL) #define SATP_MODE_57 _AC(0xa000000000000000, UL) +#define SATP_MODE_SHIFT 60 #define SATP_ASID_BITS 16 #define SATP_ASID_SHIFT 44 #define SATP_ASID_MASK _AC(0xFFFF, UL) diff --git a/arch/riscv/include/asm/dma-noncoherent.h b/arch/riscv/include/asm/dma-noncoherent.h new file mode 100644 index 000000000000..312cfa0858fb --- /dev/null +++ b/arch/riscv/include/asm/dma-noncoherent.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2023 Renesas Electronics Corp. + */ + +#ifndef __ASM_DMA_NONCOHERENT_H +#define __ASM_DMA_NONCOHERENT_H + +#include + +/* + * struct riscv_nonstd_cache_ops - Structure for non-standard CMO function pointers + * + * @wback: Function pointer for cache writeback + * @inv: Function pointer for invalidating cache + * @wback_inv: Function pointer for flushing the cache (writeback + invalidating) + */ +struct riscv_nonstd_cache_ops { + void (*wback)(phys_addr_t paddr, size_t size); + void (*inv)(phys_addr_t paddr, size_t size); + void (*wback_inv)(phys_addr_t paddr, size_t size); +}; + +extern struct riscv_nonstd_cache_ops noncoherent_cache_ops; + +void riscv_noncoherent_register_cache_ops(const struct riscv_nonstd_cache_ops *ops); + +#endif /* __ASM_DMA_NONCOHERENT_H */ diff --git a/arch/riscv/include/asm/efi.h b/arch/riscv/include/asm/efi.h index 8a6a128ec57f..46a355913b27 100644 --- a/arch/riscv/include/asm/efi.h +++ b/arch/riscv/include/asm/efi.h @@ -45,4 +45,6 @@ void arch_efi_call_virt_teardown(void); unsigned long stext_offset(void); +void efi_icache_sync(unsigned long start, unsigned long end); + #endif /* _ASM_EFI_H */ diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index fb1a810f3d8c..e2ecd01bfac7 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -11,6 +11,11 @@ #include #include +#ifdef CONFIG_ERRATA_ANDES +#define ERRATA_ANDESTECH_NO_IOCP 0 +#define ERRATA_ANDESTECH_NUMBER 1 +#endif + #ifdef CONFIG_ERRATA_SIFIVE #define ERRATA_SIFIVE_CIP_453 0 #define ERRATA_SIFIVE_CIP_1200 1 diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h index 2d8ee53b66c7..1ebf20dfbaa6 100644 --- a/arch/riscv/include/asm/kvm_host.h +++ b/arch/riscv/include/asm/kvm_host.h @@ -337,6 +337,15 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run, void __kvm_riscv_switch_to(struct kvm_vcpu_arch *vcpu_arch); +void kvm_riscv_vcpu_setup_isa(struct kvm_vcpu *vcpu); +unsigned long kvm_riscv_vcpu_num_regs(struct kvm_vcpu *vcpu); +int kvm_riscv_vcpu_copy_reg_indices(struct kvm_vcpu *vcpu, + u64 __user *uindices); +int kvm_riscv_vcpu_get_reg(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg); +int kvm_riscv_vcpu_set_reg(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg); + int kvm_riscv_vcpu_set_interrupt(struct kvm_vcpu *vcpu, unsigned int irq); int kvm_riscv_vcpu_unset_interrupt(struct kvm_vcpu *vcpu, unsigned int irq); void kvm_riscv_vcpu_flush_interrupts(struct kvm_vcpu *vcpu); diff --git a/arch/riscv/include/asm/kvm_vcpu_vector.h b/arch/riscv/include/asm/kvm_vcpu_vector.h index ff994fdd6d0d..27f5bccdd8b0 100644 --- a/arch/riscv/include/asm/kvm_vcpu_vector.h +++ b/arch/riscv/include/asm/kvm_vcpu_vector.h @@ -74,9 +74,7 @@ static inline void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu) #endif int kvm_riscv_vcpu_get_reg_vector(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg, - unsigned long rtype); + const struct kvm_one_reg *reg); int kvm_riscv_vcpu_set_reg_vector(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg, - unsigned long rtype); + const struct kvm_one_reg *reg); #endif diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index b55ba20903ec..5488ecc337b6 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -106,6 +106,7 @@ typedef struct page *pgtable_t; struct kernel_mapping { unsigned long page_offset; unsigned long virt_addr; + unsigned long virt_offset; uintptr_t phys_addr; uintptr_t size; /* Offset between linear mapping virtual address and kernel load address */ @@ -185,6 +186,8 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x); #define sym_to_pfn(x) __phys_to_pfn(__pa_symbol(x)) +unsigned long kaslr_offset(void); + #endif /* __ASSEMBLY__ */ #define virt_addr_valid(vaddr) ({ \ diff --git a/arch/riscv/include/asm/patch.h b/arch/riscv/include/asm/patch.h index 63c98833d510..e88b52d39eac 100644 --- a/arch/riscv/include/asm/patch.h +++ b/arch/riscv/include/asm/patch.h @@ -7,6 +7,7 @@ #define _ASM_RISCV_PATCH_H int patch_text_nosync(void *addr, const void *insns, size_t len); +int patch_text_set_nosync(void *addr, u8 c, size_t len); int patch_text(void *addr, u32 *insns, int ninsns); extern int riscv_patch_in_stop_machine; diff --git a/arch/riscv/include/asm/vendorid_list.h b/arch/riscv/include/asm/vendorid_list.h index cb89af3f0704..e55407ace0c3 100644 --- a/arch/riscv/include/asm/vendorid_list.h +++ b/arch/riscv/include/asm/vendorid_list.h @@ -5,6 +5,7 @@ #ifndef ASM_VENDOR_LIST_H #define ASM_VENDOR_LIST_H +#define ANDESTECH_VENDOR_ID 0x31e #define SIFIVE_VENDOR_ID 0x489 #define THEAD_VENDOR_ID 0x5b7 diff --git a/arch/riscv/include/uapi/asm/kvm.h b/arch/riscv/include/uapi/asm/kvm.h index 930fdc4101cd..992c5e407104 100644 --- a/arch/riscv/include/uapi/asm/kvm.h +++ b/arch/riscv/include/uapi/asm/kvm.h @@ -55,6 +55,7 @@ struct kvm_riscv_config { unsigned long marchid; unsigned long mimpid; unsigned long zicboz_block_size; + unsigned long satp_mode; }; /* CORE registers for KVM_GET_ONE_REG and KVM_SET_ONE_REG */ @@ -124,6 +125,12 @@ enum KVM_RISCV_ISA_EXT_ID { KVM_RISCV_ISA_EXT_SSAIA, KVM_RISCV_ISA_EXT_V, KVM_RISCV_ISA_EXT_SVNAPOT, + KVM_RISCV_ISA_EXT_ZBA, + KVM_RISCV_ISA_EXT_ZBS, + KVM_RISCV_ISA_EXT_ZICNTR, + KVM_RISCV_ISA_EXT_ZICSR, + KVM_RISCV_ISA_EXT_ZIFENCEI, + KVM_RISCV_ISA_EXT_ZIHPM, KVM_RISCV_ISA_EXT_MAX, }; @@ -193,6 +200,15 @@ enum KVM_RISCV_SBI_EXT_ID { /* ISA Extension registers are mapped as type 7 */ #define KVM_REG_RISCV_ISA_EXT (0x07 << KVM_REG_RISCV_TYPE_SHIFT) +#define KVM_REG_RISCV_ISA_SINGLE (0x0 << KVM_REG_RISCV_SUBTYPE_SHIFT) +#define KVM_REG_RISCV_ISA_MULTI_EN (0x1 << KVM_REG_RISCV_SUBTYPE_SHIFT) +#define KVM_REG_RISCV_ISA_MULTI_DIS (0x2 << KVM_REG_RISCV_SUBTYPE_SHIFT) +#define KVM_REG_RISCV_ISA_MULTI_REG(__ext_id) \ + ((__ext_id) / __BITS_PER_LONG) +#define KVM_REG_RISCV_ISA_MULTI_MASK(__ext_id) \ + (1UL << ((__ext_id) % __BITS_PER_LONG)) +#define KVM_REG_RISCV_ISA_MULTI_REG_LAST \ + KVM_REG_RISCV_ISA_MULTI_REG(KVM_RISCV_ISA_EXT_MAX - 1) /* SBI extension registers are mapped as type 8 */ #define KVM_REG_RISCV_SBI_EXT (0x08 << KVM_REG_RISCV_TYPE_SHIFT) diff --git a/arch/riscv/include/uapi/asm/ptrace.h b/arch/riscv/include/uapi/asm/ptrace.h index 6d2d9afaabea..a38268b19c3d 100644 --- a/arch/riscv/include/uapi/asm/ptrace.h +++ b/arch/riscv/include/uapi/asm/ptrace.h @@ -108,13 +108,18 @@ struct __riscv_v_ext_state { * In signal handler, datap will be set a correct user stack offset * and vector registers will be copied to the address of datap * pointer. - * - * In ptrace syscall, datap will be set to zero and the vector - * registers will be copied to the address right after this - * structure. */ }; +struct __riscv_v_regset_state { + unsigned long vstart; + unsigned long vl; + unsigned long vtype; + unsigned long vcsr; + unsigned long vlenb; + char vreg[]; +}; + /* * According to spec: The number of bits in a single vector register, * VLEN >= ELEN, which must be a power of 2, and must be no greater than diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 6ac56af42f4a..95cf25d48405 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -38,6 +38,7 @@ extra-y += vmlinux.lds obj-y += head.o obj-y += soc.o obj-$(CONFIG_RISCV_ALTERNATIVE) += alternative.o +obj-y += copy-unaligned.o obj-y += cpu.o obj-y += cpufeature.o obj-y += entry.o diff --git a/arch/riscv/kernel/alternative.c b/arch/riscv/kernel/alternative.c index 6b75788c18e6..319a1da0358b 100644 --- a/arch/riscv/kernel/alternative.c +++ b/arch/riscv/kernel/alternative.c @@ -27,8 +27,6 @@ struct cpu_manufacturer_info_t { void (*patch_func)(struct alt_entry *begin, struct alt_entry *end, unsigned long archid, unsigned long impid, unsigned int stage); - void (*feature_probe_func)(unsigned int cpu, unsigned long archid, - unsigned long impid); }; static void riscv_fill_cpu_mfr_info(struct cpu_manufacturer_info_t *cpu_mfr_info) @@ -43,8 +41,12 @@ static void riscv_fill_cpu_mfr_info(struct cpu_manufacturer_info_t *cpu_mfr_info cpu_mfr_info->imp_id = sbi_get_mimpid(); #endif - cpu_mfr_info->feature_probe_func = NULL; switch (cpu_mfr_info->vendor_id) { +#ifdef CONFIG_ERRATA_ANDES + case ANDESTECH_VENDOR_ID: + cpu_mfr_info->patch_func = andes_errata_patch_func; + break; +#endif #ifdef CONFIG_ERRATA_SIFIVE case SIFIVE_VENDOR_ID: cpu_mfr_info->patch_func = sifive_errata_patch_func; @@ -53,7 +55,6 @@ static void riscv_fill_cpu_mfr_info(struct cpu_manufacturer_info_t *cpu_mfr_info #ifdef CONFIG_ERRATA_THEAD case THEAD_VENDOR_ID: cpu_mfr_info->patch_func = thead_errata_patch_func; - cpu_mfr_info->feature_probe_func = thead_feature_probe_func; break; #endif default: @@ -143,20 +144,6 @@ void riscv_alternative_fix_offsets(void *alt_ptr, unsigned int len, } } -/* Called on each CPU as it starts */ -void probe_vendor_features(unsigned int cpu) -{ - struct cpu_manufacturer_info_t cpu_mfr_info; - - riscv_fill_cpu_mfr_info(&cpu_mfr_info); - if (!cpu_mfr_info.feature_probe_func) - return; - - cpu_mfr_info.feature_probe_func(cpu, - cpu_mfr_info.arch_id, - cpu_mfr_info.imp_id); -} - /* * This is called very early in the boot process (directly after we run * a feature detect on the boot CPU). No need to worry about other CPUs @@ -211,7 +198,6 @@ void __init apply_boot_alternatives(void) /* If called on non-boot cpu things could go wrong */ WARN_ON(smp_processor_id() != 0); - probe_vendor_features(0); _apply_alternatives((struct alt_entry *)__alt_start, (struct alt_entry *)__alt_end, RISCV_ALTERNATIVES_BOOT); diff --git a/arch/riscv/kernel/copy-unaligned.S b/arch/riscv/kernel/copy-unaligned.S new file mode 100644 index 000000000000..cfdecfbaad62 --- /dev/null +++ b/arch/riscv/kernel/copy-unaligned.S @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (C) 2023 Rivos Inc. */ + +#include +#include + + .text + +/* void __riscv_copy_words_unaligned(void *, const void *, size_t) */ +/* Performs a memcpy without aligning buffers, using word loads and stores. */ +/* Note: The size is truncated to a multiple of 8 * SZREG */ +ENTRY(__riscv_copy_words_unaligned) + andi a4, a2, ~((8*SZREG)-1) + beqz a4, 2f + add a3, a1, a4 +1: + REG_L a4, 0(a1) + REG_L a5, SZREG(a1) + REG_L a6, 2*SZREG(a1) + REG_L a7, 3*SZREG(a1) + REG_L t0, 4*SZREG(a1) + REG_L t1, 5*SZREG(a1) + REG_L t2, 6*SZREG(a1) + REG_L t3, 7*SZREG(a1) + REG_S a4, 0(a0) + REG_S a5, SZREG(a0) + REG_S a6, 2*SZREG(a0) + REG_S a7, 3*SZREG(a0) + REG_S t0, 4*SZREG(a0) + REG_S t1, 5*SZREG(a0) + REG_S t2, 6*SZREG(a0) + REG_S t3, 7*SZREG(a0) + addi a0, a0, 8*SZREG + addi a1, a1, 8*SZREG + bltu a1, a3, 1b + +2: + ret +END(__riscv_copy_words_unaligned) + +/* void __riscv_copy_bytes_unaligned(void *, const void *, size_t) */ +/* Performs a memcpy without aligning buffers, using only byte accesses. */ +/* Note: The size is truncated to a multiple of 8 */ +ENTRY(__riscv_copy_bytes_unaligned) + andi a4, a2, ~(8-1) + beqz a4, 2f + add a3, a1, a4 +1: + lb a4, 0(a1) + lb a5, 1(a1) + lb a6, 2(a1) + lb a7, 3(a1) + lb t0, 4(a1) + lb t1, 5(a1) + lb t2, 6(a1) + lb t3, 7(a1) + sb a4, 0(a0) + sb a5, 1(a0) + sb a6, 2(a0) + sb a7, 3(a0) + sb t0, 4(a0) + sb t1, 5(a0) + sb t2, 6(a0) + sb t3, 7(a0) + addi a0, a0, 8 + addi a1, a1, 8 + bltu a1, a3, 1b + +2: + ret +END(__riscv_copy_bytes_unaligned) diff --git a/arch/riscv/kernel/copy-unaligned.h b/arch/riscv/kernel/copy-unaligned.h new file mode 100644 index 000000000000..e3d70d35b708 --- /dev/null +++ b/arch/riscv/kernel/copy-unaligned.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023 Rivos, Inc. + */ +#ifndef __RISCV_KERNEL_COPY_UNALIGNED_H +#define __RISCV_KERNEL_COPY_UNALIGNED_H + +#include + +void __riscv_copy_words_unaligned(void *dst, const void *src, size_t size); +void __riscv_copy_bytes_unaligned(void *dst, const void *src, size_t size); + +#endif /* __RISCV_KERNEL_COPY_UNALIGNED_H */ diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index ef7b4fd9e876..1cfbba65d11a 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -18,12 +18,19 @@ #include #include #include +#include #include #include #include +#include "copy-unaligned.h" + #define NUM_ALPHA_EXTS ('z' - 'a' + 1) +#define MISALIGNED_ACCESS_JIFFIES_LG2 1 +#define MISALIGNED_BUFFER_SIZE 0x4000 +#define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) + unsigned long elf_hwcap __read_mostly; /* Host ISA bitmap */ @@ -549,6 +556,103 @@ unsigned long riscv_get_elf_hwcap(void) return hwcap; } +void check_unaligned_access(int cpu) +{ + u64 start_cycles, end_cycles; + u64 word_cycles; + u64 byte_cycles; + int ratio; + unsigned long start_jiffies, now; + struct page *page; + void *dst; + void *src; + long speed = RISCV_HWPROBE_MISALIGNED_SLOW; + + page = alloc_pages(GFP_NOWAIT, get_order(MISALIGNED_BUFFER_SIZE)); + if (!page) { + pr_warn("Can't alloc pages to measure memcpy performance"); + return; + } + + /* Make an unaligned destination buffer. */ + dst = (void *)((unsigned long)page_address(page) | 0x1); + /* Unalign src as well, but differently (off by 1 + 2 = 3). */ + src = dst + (MISALIGNED_BUFFER_SIZE / 2); + src += 2; + word_cycles = -1ULL; + /* Do a warmup. */ + __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); + preempt_disable(); + start_jiffies = jiffies; + while ((now = jiffies) == start_jiffies) + cpu_relax(); + + /* + * For a fixed amount of time, repeatedly try the function, and take + * the best time in cycles as the measurement. + */ + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { + start_cycles = get_cycles64(); + /* Ensure the CSR read can't reorder WRT to the copy. */ + mb(); + __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); + /* Ensure the copy ends before the end time is snapped. */ + mb(); + end_cycles = get_cycles64(); + if ((end_cycles - start_cycles) < word_cycles) + word_cycles = end_cycles - start_cycles; + } + + byte_cycles = -1ULL; + __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); + start_jiffies = jiffies; + while ((now = jiffies) == start_jiffies) + cpu_relax(); + + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { + start_cycles = get_cycles64(); + mb(); + __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); + mb(); + end_cycles = get_cycles64(); + if ((end_cycles - start_cycles) < byte_cycles) + byte_cycles = end_cycles - start_cycles; + } + + preempt_enable(); + + /* Don't divide by zero. */ + if (!word_cycles || !byte_cycles) { + pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned access speed\n", + cpu); + + goto out; + } + + if (word_cycles < byte_cycles) + speed = RISCV_HWPROBE_MISALIGNED_FAST; + + ratio = div_u64((byte_cycles * 100), word_cycles); + pr_info("cpu%d: Ratio of byte access time to unaligned word access is %d.%02d, unaligned accesses are %s\n", + cpu, + ratio / 100, + ratio % 100, + (speed == RISCV_HWPROBE_MISALIGNED_FAST) ? "fast" : "slow"); + + per_cpu(misaligned_access_speed, cpu) = speed; + +out: + __free_pages(page, get_order(MISALIGNED_BUFFER_SIZE)); +} + +static int check_unaligned_access_boot_cpu(void) +{ + check_unaligned_access(0); + return 0; +} + +arch_initcall(check_unaligned_access_boot_cpu); + #ifdef CONFIG_RISCV_ALTERNATIVE /* * Alternative patch sites consider 48 bits when determining when to patch diff --git a/arch/riscv/kernel/image-vars.h b/arch/riscv/kernel/image-vars.h index 15616155008c..ea1a10355ce9 100644 --- a/arch/riscv/kernel/image-vars.h +++ b/arch/riscv/kernel/image-vars.h @@ -27,6 +27,7 @@ __efistub__start = _start; __efistub__start_kernel = _start_kernel; __efistub__end = _end; __efistub__edata = _edata; +__efistub___init_text_end = __init_text_end; __efistub_screen_info = screen_info; #endif diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c index 575e71d6c8ae..13ee7bf589a1 100644 --- a/arch/riscv/kernel/patch.c +++ b/arch/riscv/kernel/patch.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -53,12 +54,51 @@ static void patch_unmap(int fixmap) } NOKPROBE_SYMBOL(patch_unmap); -static int patch_insn_write(void *addr, const void *insn, size_t len) +static int __patch_insn_set(void *addr, u8 c, size_t len) +{ + void *waddr = addr; + bool across_pages = (((uintptr_t)addr & ~PAGE_MASK) + len) > PAGE_SIZE; + + /* + * Only two pages can be mapped at a time for writing. + */ + if (len + offset_in_page(addr) > 2 * PAGE_SIZE) + return -EINVAL; + /* + * Before reaching here, it was expected to lock the text_mutex + * already, so we don't need to give another lock here and could + * ensure that it was safe between each cores. + */ + lockdep_assert_held(&text_mutex); + + if (across_pages) + patch_map(addr + PAGE_SIZE, FIX_TEXT_POKE1); + + waddr = patch_map(addr, FIX_TEXT_POKE0); + + memset(waddr, c, len); + + patch_unmap(FIX_TEXT_POKE0); + + if (across_pages) + patch_unmap(FIX_TEXT_POKE1); + + return 0; +} +NOKPROBE_SYMBOL(__patch_insn_set); + +static int __patch_insn_write(void *addr, const void *insn, size_t len) { void *waddr = addr; bool across_pages = (((uintptr_t) addr & ~PAGE_MASK) + len) > PAGE_SIZE; int ret; + /* + * Only two pages can be mapped at a time for writing. + */ + if (len + offset_in_page(addr) > 2 * PAGE_SIZE) + return -EINVAL; + /* * Before reaching here, it was expected to lock the text_mutex * already, so we don't need to give another lock here and could @@ -74,7 +114,7 @@ static int patch_insn_write(void *addr, const void *insn, size_t len) lockdep_assert_held(&text_mutex); if (across_pages) - patch_map(addr + len, FIX_TEXT_POKE1); + patch_map(addr + PAGE_SIZE, FIX_TEXT_POKE1); waddr = patch_map(addr, FIX_TEXT_POKE0); @@ -87,15 +127,79 @@ static int patch_insn_write(void *addr, const void *insn, size_t len) return ret; } -NOKPROBE_SYMBOL(patch_insn_write); +NOKPROBE_SYMBOL(__patch_insn_write); #else -static int patch_insn_write(void *addr, const void *insn, size_t len) +static int __patch_insn_set(void *addr, u8 c, size_t len) +{ + memset(addr, c, len); + + return 0; +} +NOKPROBE_SYMBOL(__patch_insn_set); + +static int __patch_insn_write(void *addr, const void *insn, size_t len) { return copy_to_kernel_nofault(addr, insn, len); } -NOKPROBE_SYMBOL(patch_insn_write); +NOKPROBE_SYMBOL(__patch_insn_write); #endif /* CONFIG_MMU */ +static int patch_insn_set(void *addr, u8 c, size_t len) +{ + size_t patched = 0; + size_t size; + int ret = 0; + + /* + * __patch_insn_set() can only work on 2 pages at a time so call it in a + * loop with len <= 2 * PAGE_SIZE. + */ + while (patched < len && !ret) { + size = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(addr + patched), len - patched); + ret = __patch_insn_set(addr + patched, c, size); + + patched += size; + } + + return ret; +} +NOKPROBE_SYMBOL(patch_insn_set); + +int patch_text_set_nosync(void *addr, u8 c, size_t len) +{ + u32 *tp = addr; + int ret; + + ret = patch_insn_set(tp, c, len); + + if (!ret) + flush_icache_range((uintptr_t)tp, (uintptr_t)tp + len); + + return ret; +} +NOKPROBE_SYMBOL(patch_text_set_nosync); + +static int patch_insn_write(void *addr, const void *insn, size_t len) +{ + size_t patched = 0; + size_t size; + int ret = 0; + + /* + * Copy the instructions to the destination address, two pages at a time + * because __patch_insn_write() can only handle len <= 2 * PAGE_SIZE. + */ + while (patched < len && !ret) { + size = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(addr + patched), len - patched); + ret = __patch_insn_write(addr + patched, insn + patched, size); + + patched += size; + } + + return ret; +} +NOKPROBE_SYMBOL(patch_insn_write); + int patch_text_nosync(void *addr, const void *insns, size_t len) { u32 *tp = addr; diff --git a/arch/riscv/kernel/pi/Makefile b/arch/riscv/kernel/pi/Makefile index 7b593d44c712..07915dc9279e 100644 --- a/arch/riscv/kernel/pi/Makefile +++ b/arch/riscv/kernel/pi/Makefile @@ -35,5 +35,5 @@ $(obj)/string.o: $(srctree)/lib/string.c FORCE $(obj)/ctype.o: $(srctree)/lib/ctype.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := cmdline_early.pi.o string.pi.o ctype.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-y := cmdline_early.pi.o fdt_early.pi.o string.pi.o ctype.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/riscv/kernel/pi/cmdline_early.c b/arch/riscv/kernel/pi/cmdline_early.c index 05652d13c746..68e786c84c94 100644 --- a/arch/riscv/kernel/pi/cmdline_early.c +++ b/arch/riscv/kernel/pi/cmdline_early.c @@ -14,6 +14,7 @@ static char early_cmdline[COMMAND_LINE_SIZE]; * LLVM complain because the function is actually unused in this file). */ u64 set_satp_mode_from_cmdline(uintptr_t dtb_pa); +bool set_nokaslr_from_cmdline(uintptr_t dtb_pa); static char *get_early_cmdline(uintptr_t dtb_pa) { @@ -60,3 +61,15 @@ u64 set_satp_mode_from_cmdline(uintptr_t dtb_pa) return match_noXlvl(cmdline); } + +static bool match_nokaslr(char *cmdline) +{ + return strstr(cmdline, "nokaslr"); +} + +bool set_nokaslr_from_cmdline(uintptr_t dtb_pa) +{ + char *cmdline = get_early_cmdline(dtb_pa); + + return match_nokaslr(cmdline); +} diff --git a/arch/riscv/kernel/pi/fdt_early.c b/arch/riscv/kernel/pi/fdt_early.c new file mode 100644 index 000000000000..899610e042ab --- /dev/null +++ b/arch/riscv/kernel/pi/fdt_early.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include + +/* + * Declare the functions that are exported (but prefixed) here so that LLVM + * does not complain it lacks the 'static' keyword (which, if added, makes + * LLVM complain because the function is actually unused in this file). + */ +u64 get_kaslr_seed(uintptr_t dtb_pa); + +u64 get_kaslr_seed(uintptr_t dtb_pa) +{ + int node, len; + fdt64_t *prop; + u64 ret; + + node = fdt_path_offset((void *)dtb_pa, "/chosen"); + if (node < 0) + return 0; + + prop = fdt_getprop_w((void *)dtb_pa, node, "kaslr-seed", &len); + if (!prop || len != sizeof(u64)) + return 0; + + ret = fdt64_to_cpu(*prop); + *prop = 0; + return ret; +} diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 487303e3ef22..2afe460de16a 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -25,6 +25,9 @@ enum riscv_regset { #ifdef CONFIG_FPU REGSET_F, #endif +#ifdef CONFIG_RISCV_ISA_V + REGSET_V, +#endif }; static int riscv_gpr_get(struct task_struct *target, @@ -81,6 +84,71 @@ static int riscv_fpr_set(struct task_struct *target, } #endif +#ifdef CONFIG_RISCV_ISA_V +static int riscv_vr_get(struct task_struct *target, + const struct user_regset *regset, + struct membuf to) +{ + struct __riscv_v_ext_state *vstate = &target->thread.vstate; + struct __riscv_v_regset_state ptrace_vstate; + + if (!riscv_v_vstate_query(task_pt_regs(target))) + return -EINVAL; + + /* + * Ensure the vector registers have been saved to the memory before + * copying them to membuf. + */ + if (target == current) + riscv_v_vstate_save(current, task_pt_regs(current)); + + ptrace_vstate.vstart = vstate->vstart; + ptrace_vstate.vl = vstate->vl; + ptrace_vstate.vtype = vstate->vtype; + ptrace_vstate.vcsr = vstate->vcsr; + ptrace_vstate.vlenb = vstate->vlenb; + + /* Copy vector header from vstate. */ + membuf_write(&to, &ptrace_vstate, sizeof(struct __riscv_v_regset_state)); + + /* Copy all the vector registers from vstate. */ + return membuf_write(&to, vstate->datap, riscv_v_vsize); +} + +static int riscv_vr_set(struct task_struct *target, + const struct user_regset *regset, + unsigned int pos, unsigned int count, + const void *kbuf, const void __user *ubuf) +{ + int ret; + struct __riscv_v_ext_state *vstate = &target->thread.vstate; + struct __riscv_v_regset_state ptrace_vstate; + + if (!riscv_v_vstate_query(task_pt_regs(target))) + return -EINVAL; + + /* Copy rest of the vstate except datap */ + ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &ptrace_vstate, 0, + sizeof(struct __riscv_v_regset_state)); + if (unlikely(ret)) + return ret; + + if (vstate->vlenb != ptrace_vstate.vlenb) + return -EINVAL; + + vstate->vstart = ptrace_vstate.vstart; + vstate->vl = ptrace_vstate.vl; + vstate->vtype = ptrace_vstate.vtype; + vstate->vcsr = ptrace_vstate.vcsr; + + /* Copy all the vector registers. */ + pos = 0; + ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, vstate->datap, + 0, riscv_v_vsize); + return ret; +} +#endif + static const struct user_regset riscv_user_regset[] = { [REGSET_X] = { .core_note_type = NT_PRSTATUS, @@ -100,6 +168,17 @@ static const struct user_regset riscv_user_regset[] = { .set = riscv_fpr_set, }, #endif +#ifdef CONFIG_RISCV_ISA_V + [REGSET_V] = { + .core_note_type = NT_RISCV_VECTOR, + .align = 16, + .n = ((32 * RISCV_MAX_VLENB) + + sizeof(struct __riscv_v_regset_state)) / sizeof(__u32), + .size = sizeof(__u32), + .regset_get = riscv_vr_get, + .set = riscv_vr_set, + }, +#endif }; static const struct user_regset_view riscv_user_native_view = { diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 32c2e1eb71bd..e600aab116a4 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include @@ -347,3 +348,27 @@ void free_initmem(void) free_initmem_default(POISON_FREE_INITMEM); } + +static int dump_kernel_offset(struct notifier_block *self, + unsigned long v, void *p) +{ + pr_emerg("Kernel Offset: 0x%lx from 0x%lx\n", + kernel_map.virt_offset, + KERNEL_LINK_ADDR); + + return 0; +} + +static struct notifier_block kernel_offset_notifier = { + .notifier_call = dump_kernel_offset +}; + +static int __init register_kernel_offset_dumper(void) +{ + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) + atomic_notifier_chain_register(&panic_notifier_list, + &kernel_offset_notifier); + + return 0; +} +device_initcall(register_kernel_offset_dumper); diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index f4d6acb38dd0..1b8da4e40a4d 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -245,7 +246,7 @@ asmlinkage __visible void smp_callin(void) numa_add_cpu(curr_cpuid); set_cpu_online(curr_cpuid, 1); - probe_vendor_features(curr_cpuid); + check_unaligned_access(curr_cpuid); if (has_vector()) { if (riscv_v_setup_vsize()) diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile index fee0671e2dc1..4c2067fc59fc 100644 --- a/arch/riscv/kvm/Makefile +++ b/arch/riscv/kvm/Makefile @@ -19,6 +19,7 @@ kvm-y += vcpu_exit.o kvm-y += vcpu_fp.o kvm-y += vcpu_vector.o kvm-y += vcpu_insn.o +kvm-y += vcpu_onereg.o kvm-y += vcpu_switch.o kvm-y += vcpu_sbi.o kvm-$(CONFIG_RISCV_SBI_V01) += vcpu_sbi_v01.o diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c index 585a3b42c52c..74bb27440527 100644 --- a/arch/riscv/kvm/aia.c +++ b/arch/riscv/kvm/aia.c @@ -176,7 +176,7 @@ int kvm_riscv_vcpu_aia_get_csr(struct kvm_vcpu *vcpu, struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr; if (reg_num >= sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long)) - return -EINVAL; + return -ENOENT; *out_val = 0; if (kvm_riscv_aia_available()) @@ -192,7 +192,7 @@ int kvm_riscv_vcpu_aia_set_csr(struct kvm_vcpu *vcpu, struct kvm_vcpu_aia_csr *csr = &vcpu->arch.aia_context.guest_csr; if (reg_num >= sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long)) - return -EINVAL; + return -ENOENT; if (kvm_riscv_aia_available()) { ((unsigned long *)csr)[reg_num] = val; diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index f2eb47925806..068c74593871 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -406,12 +406,6 @@ void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) { } -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot) -{ - kvm_flush_remote_tlbs(kvm); -} - void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *free) { } @@ -559,7 +553,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { int ret; - kvm_pfn_t pfn = pte_pfn(range->pte); + kvm_pfn_t pfn = pte_pfn(range->arg.pte); if (!kvm->arch.pgd) return false; diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index d12ef99901fc..82229db1ce73 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -13,16 +13,12 @@ #include #include #include -#include #include #include #include #include #include #include -#include -#include -#include #include const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { @@ -46,79 +42,6 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { sizeof(kvm_vcpu_stats_desc), }; -#define KVM_RISCV_BASE_ISA_MASK GENMASK(25, 0) - -#define KVM_ISA_EXT_ARR(ext) [KVM_RISCV_ISA_EXT_##ext] = RISCV_ISA_EXT_##ext - -/* Mapping between KVM ISA Extension ID & Host ISA extension ID */ -static const unsigned long kvm_isa_ext_arr[] = { - [KVM_RISCV_ISA_EXT_A] = RISCV_ISA_EXT_a, - [KVM_RISCV_ISA_EXT_C] = RISCV_ISA_EXT_c, - [KVM_RISCV_ISA_EXT_D] = RISCV_ISA_EXT_d, - [KVM_RISCV_ISA_EXT_F] = RISCV_ISA_EXT_f, - [KVM_RISCV_ISA_EXT_H] = RISCV_ISA_EXT_h, - [KVM_RISCV_ISA_EXT_I] = RISCV_ISA_EXT_i, - [KVM_RISCV_ISA_EXT_M] = RISCV_ISA_EXT_m, - [KVM_RISCV_ISA_EXT_V] = RISCV_ISA_EXT_v, - - KVM_ISA_EXT_ARR(SSAIA), - KVM_ISA_EXT_ARR(SSTC), - KVM_ISA_EXT_ARR(SVINVAL), - KVM_ISA_EXT_ARR(SVNAPOT), - KVM_ISA_EXT_ARR(SVPBMT), - KVM_ISA_EXT_ARR(ZBB), - KVM_ISA_EXT_ARR(ZIHINTPAUSE), - KVM_ISA_EXT_ARR(ZICBOM), - KVM_ISA_EXT_ARR(ZICBOZ), -}; - -static unsigned long kvm_riscv_vcpu_base2isa_ext(unsigned long base_ext) -{ - unsigned long i; - - for (i = 0; i < KVM_RISCV_ISA_EXT_MAX; i++) { - if (kvm_isa_ext_arr[i] == base_ext) - return i; - } - - return KVM_RISCV_ISA_EXT_MAX; -} - -static bool kvm_riscv_vcpu_isa_enable_allowed(unsigned long ext) -{ - switch (ext) { - case KVM_RISCV_ISA_EXT_H: - return false; - case KVM_RISCV_ISA_EXT_V: - return riscv_v_vstate_ctrl_user_allowed(); - default: - break; - } - - return true; -} - -static bool kvm_riscv_vcpu_isa_disable_allowed(unsigned long ext) -{ - switch (ext) { - case KVM_RISCV_ISA_EXT_A: - case KVM_RISCV_ISA_EXT_C: - case KVM_RISCV_ISA_EXT_I: - case KVM_RISCV_ISA_EXT_M: - case KVM_RISCV_ISA_EXT_SSAIA: - case KVM_RISCV_ISA_EXT_SSTC: - case KVM_RISCV_ISA_EXT_SVINVAL: - case KVM_RISCV_ISA_EXT_SVNAPOT: - case KVM_RISCV_ISA_EXT_ZIHINTPAUSE: - case KVM_RISCV_ISA_EXT_ZBB: - return false; - default: - break; - } - - return true; -} - static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) { struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; @@ -176,7 +99,6 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) int rc; struct kvm_cpu_context *cntx; struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr; - unsigned long host_isa, i; /* Mark this VCPU never ran */ vcpu->arch.ran_atleast_once = false; @@ -184,12 +106,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) bitmap_zero(vcpu->arch.isa, RISCV_ISA_EXT_MAX); /* Setup ISA features available to VCPU */ - for (i = 0; i < ARRAY_SIZE(kvm_isa_ext_arr); i++) { - host_isa = kvm_isa_ext_arr[i]; - if (__riscv_isa_extension_available(NULL, host_isa) && - kvm_riscv_vcpu_isa_enable_allowed(i)) - set_bit(host_isa, vcpu->arch.isa); - } + kvm_riscv_vcpu_setup_isa(vcpu); /* Setup vendor, arch, and implementation details */ vcpu->arch.mvendorid = sbi_get_mvendorid(); @@ -294,450 +211,6 @@ vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) return VM_FAULT_SIGBUS; } -static int kvm_riscv_vcpu_get_reg_config(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - unsigned long __user *uaddr = - (unsigned long __user *)(unsigned long)reg->addr; - unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | - KVM_REG_SIZE_MASK | - KVM_REG_RISCV_CONFIG); - unsigned long reg_val; - - if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) - return -EINVAL; - - switch (reg_num) { - case KVM_REG_RISCV_CONFIG_REG(isa): - reg_val = vcpu->arch.isa[0] & KVM_RISCV_BASE_ISA_MASK; - break; - case KVM_REG_RISCV_CONFIG_REG(zicbom_block_size): - if (!riscv_isa_extension_available(vcpu->arch.isa, ZICBOM)) - return -EINVAL; - reg_val = riscv_cbom_block_size; - break; - case KVM_REG_RISCV_CONFIG_REG(zicboz_block_size): - if (!riscv_isa_extension_available(vcpu->arch.isa, ZICBOZ)) - return -EINVAL; - reg_val = riscv_cboz_block_size; - break; - case KVM_REG_RISCV_CONFIG_REG(mvendorid): - reg_val = vcpu->arch.mvendorid; - break; - case KVM_REG_RISCV_CONFIG_REG(marchid): - reg_val = vcpu->arch.marchid; - break; - case KVM_REG_RISCV_CONFIG_REG(mimpid): - reg_val = vcpu->arch.mimpid; - break; - default: - return -EINVAL; - } - - if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) - return -EFAULT; - - return 0; -} - -static int kvm_riscv_vcpu_set_reg_config(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - unsigned long __user *uaddr = - (unsigned long __user *)(unsigned long)reg->addr; - unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | - KVM_REG_SIZE_MASK | - KVM_REG_RISCV_CONFIG); - unsigned long i, isa_ext, reg_val; - - if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) - return -EINVAL; - - if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) - return -EFAULT; - - switch (reg_num) { - case KVM_REG_RISCV_CONFIG_REG(isa): - /* - * This ONE REG interface is only defined for - * single letter extensions. - */ - if (fls(reg_val) >= RISCV_ISA_EXT_BASE) - return -EINVAL; - - if (!vcpu->arch.ran_atleast_once) { - /* Ignore the enable/disable request for certain extensions */ - for (i = 0; i < RISCV_ISA_EXT_BASE; i++) { - isa_ext = kvm_riscv_vcpu_base2isa_ext(i); - if (isa_ext >= KVM_RISCV_ISA_EXT_MAX) { - reg_val &= ~BIT(i); - continue; - } - if (!kvm_riscv_vcpu_isa_enable_allowed(isa_ext)) - if (reg_val & BIT(i)) - reg_val &= ~BIT(i); - if (!kvm_riscv_vcpu_isa_disable_allowed(isa_ext)) - if (!(reg_val & BIT(i))) - reg_val |= BIT(i); - } - reg_val &= riscv_isa_extension_base(NULL); - /* Do not modify anything beyond single letter extensions */ - reg_val = (vcpu->arch.isa[0] & ~KVM_RISCV_BASE_ISA_MASK) | - (reg_val & KVM_RISCV_BASE_ISA_MASK); - vcpu->arch.isa[0] = reg_val; - kvm_riscv_vcpu_fp_reset(vcpu); - } else { - return -EOPNOTSUPP; - } - break; - case KVM_REG_RISCV_CONFIG_REG(zicbom_block_size): - return -EOPNOTSUPP; - case KVM_REG_RISCV_CONFIG_REG(zicboz_block_size): - return -EOPNOTSUPP; - case KVM_REG_RISCV_CONFIG_REG(mvendorid): - if (!vcpu->arch.ran_atleast_once) - vcpu->arch.mvendorid = reg_val; - else - return -EBUSY; - break; - case KVM_REG_RISCV_CONFIG_REG(marchid): - if (!vcpu->arch.ran_atleast_once) - vcpu->arch.marchid = reg_val; - else - return -EBUSY; - break; - case KVM_REG_RISCV_CONFIG_REG(mimpid): - if (!vcpu->arch.ran_atleast_once) - vcpu->arch.mimpid = reg_val; - else - return -EBUSY; - break; - default: - return -EINVAL; - } - - return 0; -} - -static int kvm_riscv_vcpu_get_reg_core(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; - unsigned long __user *uaddr = - (unsigned long __user *)(unsigned long)reg->addr; - unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | - KVM_REG_SIZE_MASK | - KVM_REG_RISCV_CORE); - unsigned long reg_val; - - if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) - return -EINVAL; - if (reg_num >= sizeof(struct kvm_riscv_core) / sizeof(unsigned long)) - return -EINVAL; - - if (reg_num == KVM_REG_RISCV_CORE_REG(regs.pc)) - reg_val = cntx->sepc; - else if (KVM_REG_RISCV_CORE_REG(regs.pc) < reg_num && - reg_num <= KVM_REG_RISCV_CORE_REG(regs.t6)) - reg_val = ((unsigned long *)cntx)[reg_num]; - else if (reg_num == KVM_REG_RISCV_CORE_REG(mode)) - reg_val = (cntx->sstatus & SR_SPP) ? - KVM_RISCV_MODE_S : KVM_RISCV_MODE_U; - else - return -EINVAL; - - if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) - return -EFAULT; - - return 0; -} - -static int kvm_riscv_vcpu_set_reg_core(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; - unsigned long __user *uaddr = - (unsigned long __user *)(unsigned long)reg->addr; - unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | - KVM_REG_SIZE_MASK | - KVM_REG_RISCV_CORE); - unsigned long reg_val; - - if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) - return -EINVAL; - if (reg_num >= sizeof(struct kvm_riscv_core) / sizeof(unsigned long)) - return -EINVAL; - - if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) - return -EFAULT; - - if (reg_num == KVM_REG_RISCV_CORE_REG(regs.pc)) - cntx->sepc = reg_val; - else if (KVM_REG_RISCV_CORE_REG(regs.pc) < reg_num && - reg_num <= KVM_REG_RISCV_CORE_REG(regs.t6)) - ((unsigned long *)cntx)[reg_num] = reg_val; - else if (reg_num == KVM_REG_RISCV_CORE_REG(mode)) { - if (reg_val == KVM_RISCV_MODE_S) - cntx->sstatus |= SR_SPP; - else - cntx->sstatus &= ~SR_SPP; - } else - return -EINVAL; - - return 0; -} - -static int kvm_riscv_vcpu_general_get_csr(struct kvm_vcpu *vcpu, - unsigned long reg_num, - unsigned long *out_val) -{ - struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; - - if (reg_num >= sizeof(struct kvm_riscv_csr) / sizeof(unsigned long)) - return -EINVAL; - - if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { - kvm_riscv_vcpu_flush_interrupts(vcpu); - *out_val = (csr->hvip >> VSIP_TO_HVIP_SHIFT) & VSIP_VALID_MASK; - *out_val |= csr->hvip & ~IRQ_LOCAL_MASK; - } else - *out_val = ((unsigned long *)csr)[reg_num]; - - return 0; -} - -static inline int kvm_riscv_vcpu_general_set_csr(struct kvm_vcpu *vcpu, - unsigned long reg_num, - unsigned long reg_val) -{ - struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; - - if (reg_num >= sizeof(struct kvm_riscv_csr) / sizeof(unsigned long)) - return -EINVAL; - - if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { - reg_val &= VSIP_VALID_MASK; - reg_val <<= VSIP_TO_HVIP_SHIFT; - } - - ((unsigned long *)csr)[reg_num] = reg_val; - - if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) - WRITE_ONCE(vcpu->arch.irqs_pending_mask[0], 0); - - return 0; -} - -static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - int rc; - unsigned long __user *uaddr = - (unsigned long __user *)(unsigned long)reg->addr; - unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | - KVM_REG_SIZE_MASK | - KVM_REG_RISCV_CSR); - unsigned long reg_val, reg_subtype; - - if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) - return -EINVAL; - - reg_subtype = reg_num & KVM_REG_RISCV_SUBTYPE_MASK; - reg_num &= ~KVM_REG_RISCV_SUBTYPE_MASK; - switch (reg_subtype) { - case KVM_REG_RISCV_CSR_GENERAL: - rc = kvm_riscv_vcpu_general_get_csr(vcpu, reg_num, ®_val); - break; - case KVM_REG_RISCV_CSR_AIA: - rc = kvm_riscv_vcpu_aia_get_csr(vcpu, reg_num, ®_val); - break; - default: - rc = -EINVAL; - break; - } - if (rc) - return rc; - - if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) - return -EFAULT; - - return 0; -} - -static int kvm_riscv_vcpu_set_reg_csr(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - int rc; - unsigned long __user *uaddr = - (unsigned long __user *)(unsigned long)reg->addr; - unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | - KVM_REG_SIZE_MASK | - KVM_REG_RISCV_CSR); - unsigned long reg_val, reg_subtype; - - if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) - return -EINVAL; - - if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) - return -EFAULT; - - reg_subtype = reg_num & KVM_REG_RISCV_SUBTYPE_MASK; - reg_num &= ~KVM_REG_RISCV_SUBTYPE_MASK; - switch (reg_subtype) { - case KVM_REG_RISCV_CSR_GENERAL: - rc = kvm_riscv_vcpu_general_set_csr(vcpu, reg_num, reg_val); - break; - case KVM_REG_RISCV_CSR_AIA: - rc = kvm_riscv_vcpu_aia_set_csr(vcpu, reg_num, reg_val); - break; - default: - rc = -EINVAL; - break; - } - if (rc) - return rc; - - return 0; -} - -static int kvm_riscv_vcpu_get_reg_isa_ext(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - unsigned long __user *uaddr = - (unsigned long __user *)(unsigned long)reg->addr; - unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | - KVM_REG_SIZE_MASK | - KVM_REG_RISCV_ISA_EXT); - unsigned long reg_val = 0; - unsigned long host_isa_ext; - - if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) - return -EINVAL; - - if (reg_num >= KVM_RISCV_ISA_EXT_MAX || - reg_num >= ARRAY_SIZE(kvm_isa_ext_arr)) - return -EINVAL; - - host_isa_ext = kvm_isa_ext_arr[reg_num]; - if (__riscv_isa_extension_available(vcpu->arch.isa, host_isa_ext)) - reg_val = 1; /* Mark the given extension as available */ - - if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) - return -EFAULT; - - return 0; -} - -static int kvm_riscv_vcpu_set_reg_isa_ext(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - unsigned long __user *uaddr = - (unsigned long __user *)(unsigned long)reg->addr; - unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | - KVM_REG_SIZE_MASK | - KVM_REG_RISCV_ISA_EXT); - unsigned long reg_val; - unsigned long host_isa_ext; - - if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) - return -EINVAL; - - if (reg_num >= KVM_RISCV_ISA_EXT_MAX || - reg_num >= ARRAY_SIZE(kvm_isa_ext_arr)) - return -EINVAL; - - if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) - return -EFAULT; - - host_isa_ext = kvm_isa_ext_arr[reg_num]; - if (!__riscv_isa_extension_available(NULL, host_isa_ext)) - return -EOPNOTSUPP; - - if (!vcpu->arch.ran_atleast_once) { - /* - * All multi-letter extension and a few single letter - * extension can be disabled - */ - if (reg_val == 1 && - kvm_riscv_vcpu_isa_enable_allowed(reg_num)) - set_bit(host_isa_ext, vcpu->arch.isa); - else if (!reg_val && - kvm_riscv_vcpu_isa_disable_allowed(reg_num)) - clear_bit(host_isa_ext, vcpu->arch.isa); - else - return -EINVAL; - kvm_riscv_vcpu_fp_reset(vcpu); - } else { - return -EOPNOTSUPP; - } - - return 0; -} - -static int kvm_riscv_vcpu_set_reg(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - switch (reg->id & KVM_REG_RISCV_TYPE_MASK) { - case KVM_REG_RISCV_CONFIG: - return kvm_riscv_vcpu_set_reg_config(vcpu, reg); - case KVM_REG_RISCV_CORE: - return kvm_riscv_vcpu_set_reg_core(vcpu, reg); - case KVM_REG_RISCV_CSR: - return kvm_riscv_vcpu_set_reg_csr(vcpu, reg); - case KVM_REG_RISCV_TIMER: - return kvm_riscv_vcpu_set_reg_timer(vcpu, reg); - case KVM_REG_RISCV_FP_F: - return kvm_riscv_vcpu_set_reg_fp(vcpu, reg, - KVM_REG_RISCV_FP_F); - case KVM_REG_RISCV_FP_D: - return kvm_riscv_vcpu_set_reg_fp(vcpu, reg, - KVM_REG_RISCV_FP_D); - case KVM_REG_RISCV_ISA_EXT: - return kvm_riscv_vcpu_set_reg_isa_ext(vcpu, reg); - case KVM_REG_RISCV_SBI_EXT: - return kvm_riscv_vcpu_set_reg_sbi_ext(vcpu, reg); - case KVM_REG_RISCV_VECTOR: - return kvm_riscv_vcpu_set_reg_vector(vcpu, reg, - KVM_REG_RISCV_VECTOR); - default: - break; - } - - return -EINVAL; -} - -static int kvm_riscv_vcpu_get_reg(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg) -{ - switch (reg->id & KVM_REG_RISCV_TYPE_MASK) { - case KVM_REG_RISCV_CONFIG: - return kvm_riscv_vcpu_get_reg_config(vcpu, reg); - case KVM_REG_RISCV_CORE: - return kvm_riscv_vcpu_get_reg_core(vcpu, reg); - case KVM_REG_RISCV_CSR: - return kvm_riscv_vcpu_get_reg_csr(vcpu, reg); - case KVM_REG_RISCV_TIMER: - return kvm_riscv_vcpu_get_reg_timer(vcpu, reg); - case KVM_REG_RISCV_FP_F: - return kvm_riscv_vcpu_get_reg_fp(vcpu, reg, - KVM_REG_RISCV_FP_F); - case KVM_REG_RISCV_FP_D: - return kvm_riscv_vcpu_get_reg_fp(vcpu, reg, - KVM_REG_RISCV_FP_D); - case KVM_REG_RISCV_ISA_EXT: - return kvm_riscv_vcpu_get_reg_isa_ext(vcpu, reg); - case KVM_REG_RISCV_SBI_EXT: - return kvm_riscv_vcpu_get_reg_sbi_ext(vcpu, reg); - case KVM_REG_RISCV_VECTOR: - return kvm_riscv_vcpu_get_reg_vector(vcpu, reg, - KVM_REG_RISCV_VECTOR); - default: - break; - } - - return -EINVAL; -} - long kvm_arch_vcpu_async_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -781,6 +254,24 @@ long kvm_arch_vcpu_ioctl(struct file *filp, r = kvm_riscv_vcpu_get_reg(vcpu, ®); break; } + case KVM_GET_REG_LIST: { + struct kvm_reg_list __user *user_list = argp; + struct kvm_reg_list reg_list; + unsigned int n; + + r = -EFAULT; + if (copy_from_user(®_list, user_list, sizeof(reg_list))) + break; + n = reg_list.n; + reg_list.n = kvm_riscv_vcpu_num_regs(vcpu); + if (copy_to_user(user_list, ®_list, sizeof(reg_list))) + break; + r = -E2BIG; + if (n < reg_list.n) + break; + r = kvm_riscv_vcpu_copy_reg_indices(vcpu, user_list->reg); + break; + } default: break; } diff --git a/arch/riscv/kvm/vcpu_fp.c b/arch/riscv/kvm/vcpu_fp.c index 9d8cbc42057a..08ba48a395aa 100644 --- a/arch/riscv/kvm/vcpu_fp.c +++ b/arch/riscv/kvm/vcpu_fp.c @@ -96,7 +96,7 @@ int kvm_riscv_vcpu_get_reg_fp(struct kvm_vcpu *vcpu, reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) reg_val = &cntx->fp.f.f[reg_num]; else - return -EINVAL; + return -ENOENT; } else if ((rtype == KVM_REG_RISCV_FP_D) && riscv_isa_extension_available(vcpu->arch.isa, d)) { if (reg_num == KVM_REG_RISCV_FP_D_REG(fcsr)) { @@ -109,9 +109,9 @@ int kvm_riscv_vcpu_get_reg_fp(struct kvm_vcpu *vcpu, return -EINVAL; reg_val = &cntx->fp.d.f[reg_num]; } else - return -EINVAL; + return -ENOENT; } else - return -EINVAL; + return -ENOENT; if (copy_to_user(uaddr, reg_val, KVM_REG_SIZE(reg->id))) return -EFAULT; @@ -141,7 +141,7 @@ int kvm_riscv_vcpu_set_reg_fp(struct kvm_vcpu *vcpu, reg_num <= KVM_REG_RISCV_FP_F_REG(f[31])) reg_val = &cntx->fp.f.f[reg_num]; else - return -EINVAL; + return -ENOENT; } else if ((rtype == KVM_REG_RISCV_FP_D) && riscv_isa_extension_available(vcpu->arch.isa, d)) { if (reg_num == KVM_REG_RISCV_FP_D_REG(fcsr)) { @@ -154,9 +154,9 @@ int kvm_riscv_vcpu_set_reg_fp(struct kvm_vcpu *vcpu, return -EINVAL; reg_val = &cntx->fp.d.f[reg_num]; } else - return -EINVAL; + return -ENOENT; } else - return -EINVAL; + return -ENOENT; if (copy_from_user(reg_val, uaddr, KVM_REG_SIZE(reg->id))) return -EFAULT; diff --git a/arch/riscv/kvm/vcpu_onereg.c b/arch/riscv/kvm/vcpu_onereg.c new file mode 100644 index 000000000000..1b7e9fa265cb --- /dev/null +++ b/arch/riscv/kvm/vcpu_onereg.c @@ -0,0 +1,1051 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2019 Western Digital Corporation or its affiliates. + * Copyright (C) 2023 Ventana Micro Systems Inc. + * + * Authors: + * Anup Patel + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define KVM_RISCV_BASE_ISA_MASK GENMASK(25, 0) + +#define KVM_ISA_EXT_ARR(ext) \ +[KVM_RISCV_ISA_EXT_##ext] = RISCV_ISA_EXT_##ext + +/* Mapping between KVM ISA Extension ID & Host ISA extension ID */ +static const unsigned long kvm_isa_ext_arr[] = { + /* Single letter extensions (alphabetically sorted) */ + [KVM_RISCV_ISA_EXT_A] = RISCV_ISA_EXT_a, + [KVM_RISCV_ISA_EXT_C] = RISCV_ISA_EXT_c, + [KVM_RISCV_ISA_EXT_D] = RISCV_ISA_EXT_d, + [KVM_RISCV_ISA_EXT_F] = RISCV_ISA_EXT_f, + [KVM_RISCV_ISA_EXT_H] = RISCV_ISA_EXT_h, + [KVM_RISCV_ISA_EXT_I] = RISCV_ISA_EXT_i, + [KVM_RISCV_ISA_EXT_M] = RISCV_ISA_EXT_m, + [KVM_RISCV_ISA_EXT_V] = RISCV_ISA_EXT_v, + /* Multi letter extensions (alphabetically sorted) */ + KVM_ISA_EXT_ARR(SSAIA), + KVM_ISA_EXT_ARR(SSTC), + KVM_ISA_EXT_ARR(SVINVAL), + KVM_ISA_EXT_ARR(SVNAPOT), + KVM_ISA_EXT_ARR(SVPBMT), + KVM_ISA_EXT_ARR(ZBA), + KVM_ISA_EXT_ARR(ZBB), + KVM_ISA_EXT_ARR(ZBS), + KVM_ISA_EXT_ARR(ZICBOM), + KVM_ISA_EXT_ARR(ZICBOZ), + KVM_ISA_EXT_ARR(ZICNTR), + KVM_ISA_EXT_ARR(ZICSR), + KVM_ISA_EXT_ARR(ZIFENCEI), + KVM_ISA_EXT_ARR(ZIHINTPAUSE), + KVM_ISA_EXT_ARR(ZIHPM), +}; + +static unsigned long kvm_riscv_vcpu_base2isa_ext(unsigned long base_ext) +{ + unsigned long i; + + for (i = 0; i < KVM_RISCV_ISA_EXT_MAX; i++) { + if (kvm_isa_ext_arr[i] == base_ext) + return i; + } + + return KVM_RISCV_ISA_EXT_MAX; +} + +static bool kvm_riscv_vcpu_isa_enable_allowed(unsigned long ext) +{ + switch (ext) { + case KVM_RISCV_ISA_EXT_H: + return false; + case KVM_RISCV_ISA_EXT_V: + return riscv_v_vstate_ctrl_user_allowed(); + default: + break; + } + + return true; +} + +static bool kvm_riscv_vcpu_isa_disable_allowed(unsigned long ext) +{ + switch (ext) { + case KVM_RISCV_ISA_EXT_A: + case KVM_RISCV_ISA_EXT_C: + case KVM_RISCV_ISA_EXT_I: + case KVM_RISCV_ISA_EXT_M: + case KVM_RISCV_ISA_EXT_SSAIA: + case KVM_RISCV_ISA_EXT_SSTC: + case KVM_RISCV_ISA_EXT_SVINVAL: + case KVM_RISCV_ISA_EXT_SVNAPOT: + case KVM_RISCV_ISA_EXT_ZBA: + case KVM_RISCV_ISA_EXT_ZBB: + case KVM_RISCV_ISA_EXT_ZBS: + case KVM_RISCV_ISA_EXT_ZICNTR: + case KVM_RISCV_ISA_EXT_ZICSR: + case KVM_RISCV_ISA_EXT_ZIFENCEI: + case KVM_RISCV_ISA_EXT_ZIHINTPAUSE: + case KVM_RISCV_ISA_EXT_ZIHPM: + return false; + default: + break; + } + + return true; +} + +void kvm_riscv_vcpu_setup_isa(struct kvm_vcpu *vcpu) +{ + unsigned long host_isa, i; + + for (i = 0; i < ARRAY_SIZE(kvm_isa_ext_arr); i++) { + host_isa = kvm_isa_ext_arr[i]; + if (__riscv_isa_extension_available(NULL, host_isa) && + kvm_riscv_vcpu_isa_enable_allowed(i)) + set_bit(host_isa, vcpu->arch.isa); + } +} + +static int kvm_riscv_vcpu_get_reg_config(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + unsigned long __user *uaddr = + (unsigned long __user *)(unsigned long)reg->addr; + unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | + KVM_REG_SIZE_MASK | + KVM_REG_RISCV_CONFIG); + unsigned long reg_val; + + if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) + return -EINVAL; + + switch (reg_num) { + case KVM_REG_RISCV_CONFIG_REG(isa): + reg_val = vcpu->arch.isa[0] & KVM_RISCV_BASE_ISA_MASK; + break; + case KVM_REG_RISCV_CONFIG_REG(zicbom_block_size): + if (!riscv_isa_extension_available(vcpu->arch.isa, ZICBOM)) + return -ENOENT; + reg_val = riscv_cbom_block_size; + break; + case KVM_REG_RISCV_CONFIG_REG(zicboz_block_size): + if (!riscv_isa_extension_available(vcpu->arch.isa, ZICBOZ)) + return -ENOENT; + reg_val = riscv_cboz_block_size; + break; + case KVM_REG_RISCV_CONFIG_REG(mvendorid): + reg_val = vcpu->arch.mvendorid; + break; + case KVM_REG_RISCV_CONFIG_REG(marchid): + reg_val = vcpu->arch.marchid; + break; + case KVM_REG_RISCV_CONFIG_REG(mimpid): + reg_val = vcpu->arch.mimpid; + break; + case KVM_REG_RISCV_CONFIG_REG(satp_mode): + reg_val = satp_mode >> SATP_MODE_SHIFT; + break; + default: + return -ENOENT; + } + + if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) + return -EFAULT; + + return 0; +} + +static int kvm_riscv_vcpu_set_reg_config(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + unsigned long __user *uaddr = + (unsigned long __user *)(unsigned long)reg->addr; + unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | + KVM_REG_SIZE_MASK | + KVM_REG_RISCV_CONFIG); + unsigned long i, isa_ext, reg_val; + + if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) + return -EINVAL; + + if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) + return -EFAULT; + + switch (reg_num) { + case KVM_REG_RISCV_CONFIG_REG(isa): + /* + * This ONE REG interface is only defined for + * single letter extensions. + */ + if (fls(reg_val) >= RISCV_ISA_EXT_BASE) + return -EINVAL; + + /* + * Return early (i.e. do nothing) if reg_val is the same + * value retrievable via kvm_riscv_vcpu_get_reg_config(). + */ + if (reg_val == (vcpu->arch.isa[0] & KVM_RISCV_BASE_ISA_MASK)) + break; + + if (!vcpu->arch.ran_atleast_once) { + /* Ignore the enable/disable request for certain extensions */ + for (i = 0; i < RISCV_ISA_EXT_BASE; i++) { + isa_ext = kvm_riscv_vcpu_base2isa_ext(i); + if (isa_ext >= KVM_RISCV_ISA_EXT_MAX) { + reg_val &= ~BIT(i); + continue; + } + if (!kvm_riscv_vcpu_isa_enable_allowed(isa_ext)) + if (reg_val & BIT(i)) + reg_val &= ~BIT(i); + if (!kvm_riscv_vcpu_isa_disable_allowed(isa_ext)) + if (!(reg_val & BIT(i))) + reg_val |= BIT(i); + } + reg_val &= riscv_isa_extension_base(NULL); + /* Do not modify anything beyond single letter extensions */ + reg_val = (vcpu->arch.isa[0] & ~KVM_RISCV_BASE_ISA_MASK) | + (reg_val & KVM_RISCV_BASE_ISA_MASK); + vcpu->arch.isa[0] = reg_val; + kvm_riscv_vcpu_fp_reset(vcpu); + } else { + return -EBUSY; + } + break; + case KVM_REG_RISCV_CONFIG_REG(zicbom_block_size): + if (!riscv_isa_extension_available(vcpu->arch.isa, ZICBOM)) + return -ENOENT; + if (reg_val != riscv_cbom_block_size) + return -EINVAL; + break; + case KVM_REG_RISCV_CONFIG_REG(zicboz_block_size): + if (!riscv_isa_extension_available(vcpu->arch.isa, ZICBOZ)) + return -ENOENT; + if (reg_val != riscv_cboz_block_size) + return -EINVAL; + break; + case KVM_REG_RISCV_CONFIG_REG(mvendorid): + if (reg_val == vcpu->arch.mvendorid) + break; + if (!vcpu->arch.ran_atleast_once) + vcpu->arch.mvendorid = reg_val; + else + return -EBUSY; + break; + case KVM_REG_RISCV_CONFIG_REG(marchid): + if (reg_val == vcpu->arch.marchid) + break; + if (!vcpu->arch.ran_atleast_once) + vcpu->arch.marchid = reg_val; + else + return -EBUSY; + break; + case KVM_REG_RISCV_CONFIG_REG(mimpid): + if (reg_val == vcpu->arch.mimpid) + break; + if (!vcpu->arch.ran_atleast_once) + vcpu->arch.mimpid = reg_val; + else + return -EBUSY; + break; + case KVM_REG_RISCV_CONFIG_REG(satp_mode): + if (reg_val != (satp_mode >> SATP_MODE_SHIFT)) + return -EINVAL; + break; + default: + return -ENOENT; + } + + return 0; +} + +static int kvm_riscv_vcpu_get_reg_core(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; + unsigned long __user *uaddr = + (unsigned long __user *)(unsigned long)reg->addr; + unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | + KVM_REG_SIZE_MASK | + KVM_REG_RISCV_CORE); + unsigned long reg_val; + + if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) + return -EINVAL; + if (reg_num >= sizeof(struct kvm_riscv_core) / sizeof(unsigned long)) + return -ENOENT; + + if (reg_num == KVM_REG_RISCV_CORE_REG(regs.pc)) + reg_val = cntx->sepc; + else if (KVM_REG_RISCV_CORE_REG(regs.pc) < reg_num && + reg_num <= KVM_REG_RISCV_CORE_REG(regs.t6)) + reg_val = ((unsigned long *)cntx)[reg_num]; + else if (reg_num == KVM_REG_RISCV_CORE_REG(mode)) + reg_val = (cntx->sstatus & SR_SPP) ? + KVM_RISCV_MODE_S : KVM_RISCV_MODE_U; + else + return -ENOENT; + + if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) + return -EFAULT; + + return 0; +} + +static int kvm_riscv_vcpu_set_reg_core(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; + unsigned long __user *uaddr = + (unsigned long __user *)(unsigned long)reg->addr; + unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | + KVM_REG_SIZE_MASK | + KVM_REG_RISCV_CORE); + unsigned long reg_val; + + if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) + return -EINVAL; + if (reg_num >= sizeof(struct kvm_riscv_core) / sizeof(unsigned long)) + return -ENOENT; + + if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) + return -EFAULT; + + if (reg_num == KVM_REG_RISCV_CORE_REG(regs.pc)) + cntx->sepc = reg_val; + else if (KVM_REG_RISCV_CORE_REG(regs.pc) < reg_num && + reg_num <= KVM_REG_RISCV_CORE_REG(regs.t6)) + ((unsigned long *)cntx)[reg_num] = reg_val; + else if (reg_num == KVM_REG_RISCV_CORE_REG(mode)) { + if (reg_val == KVM_RISCV_MODE_S) + cntx->sstatus |= SR_SPP; + else + cntx->sstatus &= ~SR_SPP; + } else + return -ENOENT; + + return 0; +} + +static int kvm_riscv_vcpu_general_get_csr(struct kvm_vcpu *vcpu, + unsigned long reg_num, + unsigned long *out_val) +{ + struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; + + if (reg_num >= sizeof(struct kvm_riscv_csr) / sizeof(unsigned long)) + return -ENOENT; + + if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { + kvm_riscv_vcpu_flush_interrupts(vcpu); + *out_val = (csr->hvip >> VSIP_TO_HVIP_SHIFT) & VSIP_VALID_MASK; + *out_val |= csr->hvip & ~IRQ_LOCAL_MASK; + } else + *out_val = ((unsigned long *)csr)[reg_num]; + + return 0; +} + +static int kvm_riscv_vcpu_general_set_csr(struct kvm_vcpu *vcpu, + unsigned long reg_num, + unsigned long reg_val) +{ + struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; + + if (reg_num >= sizeof(struct kvm_riscv_csr) / sizeof(unsigned long)) + return -ENOENT; + + if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) { + reg_val &= VSIP_VALID_MASK; + reg_val <<= VSIP_TO_HVIP_SHIFT; + } + + ((unsigned long *)csr)[reg_num] = reg_val; + + if (reg_num == KVM_REG_RISCV_CSR_REG(sip)) + WRITE_ONCE(vcpu->arch.irqs_pending_mask[0], 0); + + return 0; +} + +static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + int rc; + unsigned long __user *uaddr = + (unsigned long __user *)(unsigned long)reg->addr; + unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | + KVM_REG_SIZE_MASK | + KVM_REG_RISCV_CSR); + unsigned long reg_val, reg_subtype; + + if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) + return -EINVAL; + + reg_subtype = reg_num & KVM_REG_RISCV_SUBTYPE_MASK; + reg_num &= ~KVM_REG_RISCV_SUBTYPE_MASK; + switch (reg_subtype) { + case KVM_REG_RISCV_CSR_GENERAL: + rc = kvm_riscv_vcpu_general_get_csr(vcpu, reg_num, ®_val); + break; + case KVM_REG_RISCV_CSR_AIA: + rc = kvm_riscv_vcpu_aia_get_csr(vcpu, reg_num, ®_val); + break; + default: + rc = -ENOENT; + break; + } + if (rc) + return rc; + + if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) + return -EFAULT; + + return 0; +} + +static int kvm_riscv_vcpu_set_reg_csr(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + int rc; + unsigned long __user *uaddr = + (unsigned long __user *)(unsigned long)reg->addr; + unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | + KVM_REG_SIZE_MASK | + KVM_REG_RISCV_CSR); + unsigned long reg_val, reg_subtype; + + if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) + return -EINVAL; + + if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) + return -EFAULT; + + reg_subtype = reg_num & KVM_REG_RISCV_SUBTYPE_MASK; + reg_num &= ~KVM_REG_RISCV_SUBTYPE_MASK; + switch (reg_subtype) { + case KVM_REG_RISCV_CSR_GENERAL: + rc = kvm_riscv_vcpu_general_set_csr(vcpu, reg_num, reg_val); + break; + case KVM_REG_RISCV_CSR_AIA: + rc = kvm_riscv_vcpu_aia_set_csr(vcpu, reg_num, reg_val); + break; + default: + rc = -ENOENT; + break; + } + if (rc) + return rc; + + return 0; +} + +static int riscv_vcpu_get_isa_ext_single(struct kvm_vcpu *vcpu, + unsigned long reg_num, + unsigned long *reg_val) +{ + unsigned long host_isa_ext; + + if (reg_num >= KVM_RISCV_ISA_EXT_MAX || + reg_num >= ARRAY_SIZE(kvm_isa_ext_arr)) + return -ENOENT; + + *reg_val = 0; + host_isa_ext = kvm_isa_ext_arr[reg_num]; + if (__riscv_isa_extension_available(vcpu->arch.isa, host_isa_ext)) + *reg_val = 1; /* Mark the given extension as available */ + + return 0; +} + +static int riscv_vcpu_set_isa_ext_single(struct kvm_vcpu *vcpu, + unsigned long reg_num, + unsigned long reg_val) +{ + unsigned long host_isa_ext; + + if (reg_num >= KVM_RISCV_ISA_EXT_MAX || + reg_num >= ARRAY_SIZE(kvm_isa_ext_arr)) + return -ENOENT; + + host_isa_ext = kvm_isa_ext_arr[reg_num]; + if (!__riscv_isa_extension_available(NULL, host_isa_ext)) + return -ENOENT; + + if (reg_val == test_bit(host_isa_ext, vcpu->arch.isa)) + return 0; + + if (!vcpu->arch.ran_atleast_once) { + /* + * All multi-letter extension and a few single letter + * extension can be disabled + */ + if (reg_val == 1 && + kvm_riscv_vcpu_isa_enable_allowed(reg_num)) + set_bit(host_isa_ext, vcpu->arch.isa); + else if (!reg_val && + kvm_riscv_vcpu_isa_disable_allowed(reg_num)) + clear_bit(host_isa_ext, vcpu->arch.isa); + else + return -EINVAL; + kvm_riscv_vcpu_fp_reset(vcpu); + } else { + return -EBUSY; + } + + return 0; +} + +static int riscv_vcpu_get_isa_ext_multi(struct kvm_vcpu *vcpu, + unsigned long reg_num, + unsigned long *reg_val) +{ + unsigned long i, ext_id, ext_val; + + if (reg_num > KVM_REG_RISCV_ISA_MULTI_REG_LAST) + return -ENOENT; + + for (i = 0; i < BITS_PER_LONG; i++) { + ext_id = i + reg_num * BITS_PER_LONG; + if (ext_id >= KVM_RISCV_ISA_EXT_MAX) + break; + + ext_val = 0; + riscv_vcpu_get_isa_ext_single(vcpu, ext_id, &ext_val); + if (ext_val) + *reg_val |= KVM_REG_RISCV_ISA_MULTI_MASK(ext_id); + } + + return 0; +} + +static int riscv_vcpu_set_isa_ext_multi(struct kvm_vcpu *vcpu, + unsigned long reg_num, + unsigned long reg_val, bool enable) +{ + unsigned long i, ext_id; + + if (reg_num > KVM_REG_RISCV_ISA_MULTI_REG_LAST) + return -ENOENT; + + for_each_set_bit(i, ®_val, BITS_PER_LONG) { + ext_id = i + reg_num * BITS_PER_LONG; + if (ext_id >= KVM_RISCV_ISA_EXT_MAX) + break; + + riscv_vcpu_set_isa_ext_single(vcpu, ext_id, enable); + } + + return 0; +} + +static int kvm_riscv_vcpu_get_reg_isa_ext(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + int rc; + unsigned long __user *uaddr = + (unsigned long __user *)(unsigned long)reg->addr; + unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | + KVM_REG_SIZE_MASK | + KVM_REG_RISCV_ISA_EXT); + unsigned long reg_val, reg_subtype; + + if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) + return -EINVAL; + + reg_subtype = reg_num & KVM_REG_RISCV_SUBTYPE_MASK; + reg_num &= ~KVM_REG_RISCV_SUBTYPE_MASK; + + reg_val = 0; + switch (reg_subtype) { + case KVM_REG_RISCV_ISA_SINGLE: + rc = riscv_vcpu_get_isa_ext_single(vcpu, reg_num, ®_val); + break; + case KVM_REG_RISCV_ISA_MULTI_EN: + case KVM_REG_RISCV_ISA_MULTI_DIS: + rc = riscv_vcpu_get_isa_ext_multi(vcpu, reg_num, ®_val); + if (!rc && reg_subtype == KVM_REG_RISCV_ISA_MULTI_DIS) + reg_val = ~reg_val; + break; + default: + rc = -ENOENT; + } + if (rc) + return rc; + + if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) + return -EFAULT; + + return 0; +} + +static int kvm_riscv_vcpu_set_reg_isa_ext(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + unsigned long __user *uaddr = + (unsigned long __user *)(unsigned long)reg->addr; + unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | + KVM_REG_SIZE_MASK | + KVM_REG_RISCV_ISA_EXT); + unsigned long reg_val, reg_subtype; + + if (KVM_REG_SIZE(reg->id) != sizeof(unsigned long)) + return -EINVAL; + + reg_subtype = reg_num & KVM_REG_RISCV_SUBTYPE_MASK; + reg_num &= ~KVM_REG_RISCV_SUBTYPE_MASK; + + if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) + return -EFAULT; + + switch (reg_subtype) { + case KVM_REG_RISCV_ISA_SINGLE: + return riscv_vcpu_set_isa_ext_single(vcpu, reg_num, reg_val); + case KVM_REG_RISCV_SBI_MULTI_EN: + return riscv_vcpu_set_isa_ext_multi(vcpu, reg_num, reg_val, true); + case KVM_REG_RISCV_SBI_MULTI_DIS: + return riscv_vcpu_set_isa_ext_multi(vcpu, reg_num, reg_val, false); + default: + return -ENOENT; + } + + return 0; +} + +static int copy_config_reg_indices(const struct kvm_vcpu *vcpu, + u64 __user *uindices) +{ + int n = 0; + + for (int i = 0; i < sizeof(struct kvm_riscv_config)/sizeof(unsigned long); + i++) { + u64 size; + u64 reg; + + /* + * Avoid reporting config reg if the corresponding extension + * was not available. + */ + if (i == KVM_REG_RISCV_CONFIG_REG(zicbom_block_size) && + !riscv_isa_extension_available(vcpu->arch.isa, ZICBOM)) + continue; + else if (i == KVM_REG_RISCV_CONFIG_REG(zicboz_block_size) && + !riscv_isa_extension_available(vcpu->arch.isa, ZICBOZ)) + continue; + + size = IS_ENABLED(CONFIG_32BIT) ? KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64; + reg = KVM_REG_RISCV | size | KVM_REG_RISCV_CONFIG | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + + n++; + } + + return n; +} + +static unsigned long num_config_regs(const struct kvm_vcpu *vcpu) +{ + return copy_config_reg_indices(vcpu, NULL); +} + +static inline unsigned long num_core_regs(void) +{ + return sizeof(struct kvm_riscv_core) / sizeof(unsigned long); +} + +static int copy_core_reg_indices(u64 __user *uindices) +{ + int n = num_core_regs(); + + for (int i = 0; i < n; i++) { + u64 size = IS_ENABLED(CONFIG_32BIT) ? + KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64; + u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_CORE | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + + return n; +} + +static inline unsigned long num_csr_regs(const struct kvm_vcpu *vcpu) +{ + unsigned long n = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long); + + if (riscv_isa_extension_available(vcpu->arch.isa, SSAIA)) + n += sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long); + + return n; +} + +static int copy_csr_reg_indices(const struct kvm_vcpu *vcpu, + u64 __user *uindices) +{ + int n1 = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long); + int n2 = 0; + + /* copy general csr regs */ + for (int i = 0; i < n1; i++) { + u64 size = IS_ENABLED(CONFIG_32BIT) ? + KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64; + u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_CSR | + KVM_REG_RISCV_CSR_GENERAL | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + + /* copy AIA csr regs */ + if (riscv_isa_extension_available(vcpu->arch.isa, SSAIA)) { + n2 = sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long); + + for (int i = 0; i < n2; i++) { + u64 size = IS_ENABLED(CONFIG_32BIT) ? + KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64; + u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_CSR | + KVM_REG_RISCV_CSR_AIA | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + } + + return n1 + n2; +} + +static inline unsigned long num_timer_regs(void) +{ + return sizeof(struct kvm_riscv_timer) / sizeof(u64); +} + +static int copy_timer_reg_indices(u64 __user *uindices) +{ + int n = num_timer_regs(); + + for (int i = 0; i < n; i++) { + u64 reg = KVM_REG_RISCV | KVM_REG_SIZE_U64 | + KVM_REG_RISCV_TIMER | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + + return n; +} + +static inline unsigned long num_fp_f_regs(const struct kvm_vcpu *vcpu) +{ + const struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; + + if (riscv_isa_extension_available(vcpu->arch.isa, f)) + return sizeof(cntx->fp.f) / sizeof(u32); + else + return 0; +} + +static int copy_fp_f_reg_indices(const struct kvm_vcpu *vcpu, + u64 __user *uindices) +{ + int n = num_fp_f_regs(vcpu); + + for (int i = 0; i < n; i++) { + u64 reg = KVM_REG_RISCV | KVM_REG_SIZE_U32 | + KVM_REG_RISCV_FP_F | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + + return n; +} + +static inline unsigned long num_fp_d_regs(const struct kvm_vcpu *vcpu) +{ + const struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; + + if (riscv_isa_extension_available(vcpu->arch.isa, d)) + return sizeof(cntx->fp.d.f) / sizeof(u64) + 1; + else + return 0; +} + +static int copy_fp_d_reg_indices(const struct kvm_vcpu *vcpu, + u64 __user *uindices) +{ + int i; + int n = num_fp_d_regs(vcpu); + u64 reg; + + /* copy fp.d.f indices */ + for (i = 0; i < n-1; i++) { + reg = KVM_REG_RISCV | KVM_REG_SIZE_U64 | + KVM_REG_RISCV_FP_D | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + + /* copy fp.d.fcsr indices */ + reg = KVM_REG_RISCV | KVM_REG_SIZE_U32 | KVM_REG_RISCV_FP_D | i; + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + + return n; +} + +static int copy_isa_ext_reg_indices(const struct kvm_vcpu *vcpu, + u64 __user *uindices) +{ + unsigned int n = 0; + unsigned long isa_ext; + + for (int i = 0; i < KVM_RISCV_ISA_EXT_MAX; i++) { + u64 size = IS_ENABLED(CONFIG_32BIT) ? + KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64; + u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_ISA_EXT | i; + + isa_ext = kvm_isa_ext_arr[i]; + if (!__riscv_isa_extension_available(vcpu->arch.isa, isa_ext)) + continue; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + + n++; + } + + return n; +} + +static inline unsigned long num_isa_ext_regs(const struct kvm_vcpu *vcpu) +{ + return copy_isa_ext_reg_indices(vcpu, NULL);; +} + +static inline unsigned long num_sbi_ext_regs(void) +{ + /* + * number of KVM_REG_RISCV_SBI_SINGLE + + * 2 x (number of KVM_REG_RISCV_SBI_MULTI) + */ + return KVM_RISCV_SBI_EXT_MAX + 2*(KVM_REG_RISCV_SBI_MULTI_REG_LAST+1); +} + +static int copy_sbi_ext_reg_indices(u64 __user *uindices) +{ + int n; + + /* copy KVM_REG_RISCV_SBI_SINGLE */ + n = KVM_RISCV_SBI_EXT_MAX; + for (int i = 0; i < n; i++) { + u64 size = IS_ENABLED(CONFIG_32BIT) ? + KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64; + u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_SBI_EXT | + KVM_REG_RISCV_SBI_SINGLE | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + + /* copy KVM_REG_RISCV_SBI_MULTI */ + n = KVM_REG_RISCV_SBI_MULTI_REG_LAST + 1; + for (int i = 0; i < n; i++) { + u64 size = IS_ENABLED(CONFIG_32BIT) ? + KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64; + u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_SBI_EXT | + KVM_REG_RISCV_SBI_MULTI_EN | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + + reg = KVM_REG_RISCV | size | KVM_REG_RISCV_SBI_EXT | + KVM_REG_RISCV_SBI_MULTI_DIS | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + + return num_sbi_ext_regs(); +} + +/* + * kvm_riscv_vcpu_num_regs - how many registers do we present via KVM_GET/SET_ONE_REG + * + * This is for all registers. + */ +unsigned long kvm_riscv_vcpu_num_regs(struct kvm_vcpu *vcpu) +{ + unsigned long res = 0; + + res += num_config_regs(vcpu); + res += num_core_regs(); + res += num_csr_regs(vcpu); + res += num_timer_regs(); + res += num_fp_f_regs(vcpu); + res += num_fp_d_regs(vcpu); + res += num_isa_ext_regs(vcpu); + res += num_sbi_ext_regs(); + + return res; +} + +/* + * kvm_riscv_vcpu_copy_reg_indices - get indices of all registers. + */ +int kvm_riscv_vcpu_copy_reg_indices(struct kvm_vcpu *vcpu, + u64 __user *uindices) +{ + int ret; + + ret = copy_config_reg_indices(vcpu, uindices); + if (ret < 0) + return ret; + uindices += ret; + + ret = copy_core_reg_indices(uindices); + if (ret < 0) + return ret; + uindices += ret; + + ret = copy_csr_reg_indices(vcpu, uindices); + if (ret < 0) + return ret; + uindices += ret; + + ret = copy_timer_reg_indices(uindices); + if (ret < 0) + return ret; + uindices += ret; + + ret = copy_fp_f_reg_indices(vcpu, uindices); + if (ret < 0) + return ret; + uindices += ret; + + ret = copy_fp_d_reg_indices(vcpu, uindices); + if (ret < 0) + return ret; + uindices += ret; + + ret = copy_isa_ext_reg_indices(vcpu, uindices); + if (ret < 0) + return ret; + uindices += ret; + + ret = copy_sbi_ext_reg_indices(uindices); + if (ret < 0) + return ret; + + return 0; +} + +int kvm_riscv_vcpu_set_reg(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + switch (reg->id & KVM_REG_RISCV_TYPE_MASK) { + case KVM_REG_RISCV_CONFIG: + return kvm_riscv_vcpu_set_reg_config(vcpu, reg); + case KVM_REG_RISCV_CORE: + return kvm_riscv_vcpu_set_reg_core(vcpu, reg); + case KVM_REG_RISCV_CSR: + return kvm_riscv_vcpu_set_reg_csr(vcpu, reg); + case KVM_REG_RISCV_TIMER: + return kvm_riscv_vcpu_set_reg_timer(vcpu, reg); + case KVM_REG_RISCV_FP_F: + return kvm_riscv_vcpu_set_reg_fp(vcpu, reg, + KVM_REG_RISCV_FP_F); + case KVM_REG_RISCV_FP_D: + return kvm_riscv_vcpu_set_reg_fp(vcpu, reg, + KVM_REG_RISCV_FP_D); + case KVM_REG_RISCV_ISA_EXT: + return kvm_riscv_vcpu_set_reg_isa_ext(vcpu, reg); + case KVM_REG_RISCV_SBI_EXT: + return kvm_riscv_vcpu_set_reg_sbi_ext(vcpu, reg); + case KVM_REG_RISCV_VECTOR: + return kvm_riscv_vcpu_set_reg_vector(vcpu, reg); + default: + break; + } + + return -ENOENT; +} + +int kvm_riscv_vcpu_get_reg(struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + switch (reg->id & KVM_REG_RISCV_TYPE_MASK) { + case KVM_REG_RISCV_CONFIG: + return kvm_riscv_vcpu_get_reg_config(vcpu, reg); + case KVM_REG_RISCV_CORE: + return kvm_riscv_vcpu_get_reg_core(vcpu, reg); + case KVM_REG_RISCV_CSR: + return kvm_riscv_vcpu_get_reg_csr(vcpu, reg); + case KVM_REG_RISCV_TIMER: + return kvm_riscv_vcpu_get_reg_timer(vcpu, reg); + case KVM_REG_RISCV_FP_F: + return kvm_riscv_vcpu_get_reg_fp(vcpu, reg, + KVM_REG_RISCV_FP_F); + case KVM_REG_RISCV_FP_D: + return kvm_riscv_vcpu_get_reg_fp(vcpu, reg, + KVM_REG_RISCV_FP_D); + case KVM_REG_RISCV_ISA_EXT: + return kvm_riscv_vcpu_get_reg_isa_ext(vcpu, reg); + case KVM_REG_RISCV_SBI_EXT: + return kvm_riscv_vcpu_get_reg_sbi_ext(vcpu, reg); + case KVM_REG_RISCV_VECTOR: + return kvm_riscv_vcpu_get_reg_vector(vcpu, reg); + default: + break; + } + + return -ENOENT; +} diff --git a/arch/riscv/kvm/vcpu_sbi.c b/arch/riscv/kvm/vcpu_sbi.c index 7b46e04fb667..9cd97091c723 100644 --- a/arch/riscv/kvm/vcpu_sbi.c +++ b/arch/riscv/kvm/vcpu_sbi.c @@ -140,8 +140,10 @@ static int riscv_vcpu_set_sbi_ext_single(struct kvm_vcpu *vcpu, const struct kvm_riscv_sbi_extension_entry *sext = NULL; struct kvm_vcpu_sbi_context *scontext = &vcpu->arch.sbi_context; - if (reg_num >= KVM_RISCV_SBI_EXT_MAX || - (reg_val != 1 && reg_val != 0)) + if (reg_num >= KVM_RISCV_SBI_EXT_MAX) + return -ENOENT; + + if (reg_val != 1 && reg_val != 0) return -EINVAL; for (i = 0; i < ARRAY_SIZE(sbi_ext); i++) { @@ -175,7 +177,7 @@ static int riscv_vcpu_get_sbi_ext_single(struct kvm_vcpu *vcpu, struct kvm_vcpu_sbi_context *scontext = &vcpu->arch.sbi_context; if (reg_num >= KVM_RISCV_SBI_EXT_MAX) - return -EINVAL; + return -ENOENT; for (i = 0; i < ARRAY_SIZE(sbi_ext); i++) { if (sbi_ext[i].ext_idx == reg_num) { @@ -206,7 +208,7 @@ static int riscv_vcpu_set_sbi_ext_multi(struct kvm_vcpu *vcpu, unsigned long i, ext_id; if (reg_num > KVM_REG_RISCV_SBI_MULTI_REG_LAST) - return -EINVAL; + return -ENOENT; for_each_set_bit(i, ®_val, BITS_PER_LONG) { ext_id = i + reg_num * BITS_PER_LONG; @@ -226,7 +228,7 @@ static int riscv_vcpu_get_sbi_ext_multi(struct kvm_vcpu *vcpu, unsigned long i, ext_id, ext_val; if (reg_num > KVM_REG_RISCV_SBI_MULTI_REG_LAST) - return -EINVAL; + return -ENOENT; for (i = 0; i < BITS_PER_LONG; i++) { ext_id = i + reg_num * BITS_PER_LONG; @@ -272,7 +274,7 @@ int kvm_riscv_vcpu_set_reg_sbi_ext(struct kvm_vcpu *vcpu, case KVM_REG_RISCV_SBI_MULTI_DIS: return riscv_vcpu_set_sbi_ext_multi(vcpu, reg_num, reg_val, false); default: - return -EINVAL; + return -ENOENT; } return 0; @@ -307,7 +309,7 @@ int kvm_riscv_vcpu_get_reg_sbi_ext(struct kvm_vcpu *vcpu, reg_val = ~reg_val; break; default: - rc = -EINVAL; + rc = -ENOENT; } if (rc) return rc; diff --git a/arch/riscv/kvm/vcpu_timer.c b/arch/riscv/kvm/vcpu_timer.c index 3ac2ff6a65da..75486b25ac45 100644 --- a/arch/riscv/kvm/vcpu_timer.c +++ b/arch/riscv/kvm/vcpu_timer.c @@ -170,7 +170,7 @@ int kvm_riscv_vcpu_get_reg_timer(struct kvm_vcpu *vcpu, if (KVM_REG_SIZE(reg->id) != sizeof(u64)) return -EINVAL; if (reg_num >= sizeof(struct kvm_riscv_timer) / sizeof(u64)) - return -EINVAL; + return -ENOENT; switch (reg_num) { case KVM_REG_RISCV_TIMER_REG(frequency): @@ -187,7 +187,7 @@ int kvm_riscv_vcpu_get_reg_timer(struct kvm_vcpu *vcpu, KVM_RISCV_TIMER_STATE_OFF; break; default: - return -EINVAL; + return -ENOENT; } if (copy_to_user(uaddr, ®_val, KVM_REG_SIZE(reg->id))) @@ -211,14 +211,15 @@ int kvm_riscv_vcpu_set_reg_timer(struct kvm_vcpu *vcpu, if (KVM_REG_SIZE(reg->id) != sizeof(u64)) return -EINVAL; if (reg_num >= sizeof(struct kvm_riscv_timer) / sizeof(u64)) - return -EINVAL; + return -ENOENT; if (copy_from_user(®_val, uaddr, KVM_REG_SIZE(reg->id))) return -EFAULT; switch (reg_num) { case KVM_REG_RISCV_TIMER_REG(frequency): - ret = -EOPNOTSUPP; + if (reg_val != riscv_timebase) + return -EINVAL; break; case KVM_REG_RISCV_TIMER_REG(time): gt->time_delta = reg_val - get_cycles64(); @@ -233,7 +234,7 @@ int kvm_riscv_vcpu_set_reg_timer(struct kvm_vcpu *vcpu, ret = kvm_riscv_vcpu_timer_cancel(t); break; default: - ret = -EINVAL; + ret = -ENOENT; break; } diff --git a/arch/riscv/kvm/vcpu_vector.c b/arch/riscv/kvm/vcpu_vector.c index edd2eecbddc2..b430cbb69521 100644 --- a/arch/riscv/kvm/vcpu_vector.c +++ b/arch/riscv/kvm/vcpu_vector.c @@ -91,95 +91,93 @@ void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu) } #endif -static void *kvm_riscv_vcpu_vreg_addr(struct kvm_vcpu *vcpu, - unsigned long reg_num, - size_t reg_size) +static int kvm_riscv_vcpu_vreg_addr(struct kvm_vcpu *vcpu, + unsigned long reg_num, + size_t reg_size, + void **reg_addr) { struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; - void *reg_val; size_t vlenb = riscv_v_vsize / 32; if (reg_num < KVM_REG_RISCV_VECTOR_REG(0)) { if (reg_size != sizeof(unsigned long)) - return NULL; + return -EINVAL; switch (reg_num) { case KVM_REG_RISCV_VECTOR_CSR_REG(vstart): - reg_val = &cntx->vector.vstart; + *reg_addr = &cntx->vector.vstart; break; case KVM_REG_RISCV_VECTOR_CSR_REG(vl): - reg_val = &cntx->vector.vl; + *reg_addr = &cntx->vector.vl; break; case KVM_REG_RISCV_VECTOR_CSR_REG(vtype): - reg_val = &cntx->vector.vtype; + *reg_addr = &cntx->vector.vtype; break; case KVM_REG_RISCV_VECTOR_CSR_REG(vcsr): - reg_val = &cntx->vector.vcsr; + *reg_addr = &cntx->vector.vcsr; break; case KVM_REG_RISCV_VECTOR_CSR_REG(datap): default: - return NULL; + return -ENOENT; } } else if (reg_num <= KVM_REG_RISCV_VECTOR_REG(31)) { if (reg_size != vlenb) - return NULL; - reg_val = cntx->vector.datap - + (reg_num - KVM_REG_RISCV_VECTOR_REG(0)) * vlenb; + return -EINVAL; + *reg_addr = cntx->vector.datap + + (reg_num - KVM_REG_RISCV_VECTOR_REG(0)) * vlenb; } else { - return NULL; + return -ENOENT; } - return reg_val; + return 0; } int kvm_riscv_vcpu_get_reg_vector(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg, - unsigned long rtype) + const struct kvm_one_reg *reg) { unsigned long *isa = vcpu->arch.isa; unsigned long __user *uaddr = (unsigned long __user *)(unsigned long)reg->addr; unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | KVM_REG_SIZE_MASK | - rtype); - void *reg_val = NULL; + KVM_REG_RISCV_VECTOR); size_t reg_size = KVM_REG_SIZE(reg->id); + void *reg_addr; + int rc; - if (rtype == KVM_REG_RISCV_VECTOR && - riscv_isa_extension_available(isa, v)) { - reg_val = kvm_riscv_vcpu_vreg_addr(vcpu, reg_num, reg_size); - } + if (!riscv_isa_extension_available(isa, v)) + return -ENOENT; - if (!reg_val) - return -EINVAL; + rc = kvm_riscv_vcpu_vreg_addr(vcpu, reg_num, reg_size, ®_addr); + if (rc) + return rc; - if (copy_to_user(uaddr, reg_val, reg_size)) + if (copy_to_user(uaddr, reg_addr, reg_size)) return -EFAULT; return 0; } int kvm_riscv_vcpu_set_reg_vector(struct kvm_vcpu *vcpu, - const struct kvm_one_reg *reg, - unsigned long rtype) + const struct kvm_one_reg *reg) { unsigned long *isa = vcpu->arch.isa; unsigned long __user *uaddr = (unsigned long __user *)(unsigned long)reg->addr; unsigned long reg_num = reg->id & ~(KVM_REG_ARCH_MASK | KVM_REG_SIZE_MASK | - rtype); - void *reg_val = NULL; + KVM_REG_RISCV_VECTOR); size_t reg_size = KVM_REG_SIZE(reg->id); + void *reg_addr; + int rc; - if (rtype == KVM_REG_RISCV_VECTOR && - riscv_isa_extension_available(isa, v)) { - reg_val = kvm_riscv_vcpu_vreg_addr(vcpu, reg_num, reg_size); - } + if (!riscv_isa_extension_available(isa, v)) + return -ENOENT; - if (!reg_val) - return -EINVAL; + rc = kvm_riscv_vcpu_vreg_addr(vcpu, reg_num, reg_size, ®_addr); + if (rc) + return rc; - if (copy_from_user(reg_val, uaddr, reg_size)) + if (copy_from_user(reg_addr, uaddr, reg_size)) return -EFAULT; return 0; diff --git a/arch/riscv/mm/dma-noncoherent.c b/arch/riscv/mm/dma-noncoherent.c index 7270b4d8c05b..b76e7e192eb1 100644 --- a/arch/riscv/mm/dma-noncoherent.c +++ b/arch/riscv/mm/dma-noncoherent.c @@ -9,26 +9,93 @@ #include #include #include +#include static bool noncoherent_supported __ro_after_init; int dma_cache_alignment __ro_after_init = ARCH_DMA_MINALIGN; EXPORT_SYMBOL_GPL(dma_cache_alignment); -void arch_sync_dma_for_device(phys_addr_t paddr, size_t size, - enum dma_data_direction dir) +struct riscv_nonstd_cache_ops noncoherent_cache_ops __ro_after_init = { + .wback = NULL, + .inv = NULL, + .wback_inv = NULL, +}; + +static inline void arch_dma_cache_wback(phys_addr_t paddr, size_t size) { void *vaddr = phys_to_virt(paddr); +#ifdef CONFIG_RISCV_NONSTANDARD_CACHE_OPS + if (unlikely(noncoherent_cache_ops.wback)) { + noncoherent_cache_ops.wback(paddr, size); + return; + } +#endif + ALT_CMO_OP(clean, vaddr, size, riscv_cbom_block_size); +} + +static inline void arch_dma_cache_inv(phys_addr_t paddr, size_t size) +{ + void *vaddr = phys_to_virt(paddr); + +#ifdef CONFIG_RISCV_NONSTANDARD_CACHE_OPS + if (unlikely(noncoherent_cache_ops.inv)) { + noncoherent_cache_ops.inv(paddr, size); + return; + } +#endif + + ALT_CMO_OP(inval, vaddr, size, riscv_cbom_block_size); +} + +static inline void arch_dma_cache_wback_inv(phys_addr_t paddr, size_t size) +{ + void *vaddr = phys_to_virt(paddr); + +#ifdef CONFIG_RISCV_NONSTANDARD_CACHE_OPS + if (unlikely(noncoherent_cache_ops.wback_inv)) { + noncoherent_cache_ops.wback_inv(paddr, size); + return; + } +#endif + + ALT_CMO_OP(flush, vaddr, size, riscv_cbom_block_size); +} + +static inline bool arch_sync_dma_clean_before_fromdevice(void) +{ + return true; +} + +static inline bool arch_sync_dma_cpu_needs_post_dma_flush(void) +{ + return true; +} + +void arch_sync_dma_for_device(phys_addr_t paddr, size_t size, + enum dma_data_direction dir) +{ switch (dir) { case DMA_TO_DEVICE: - ALT_CMO_OP(clean, vaddr, size, riscv_cbom_block_size); + arch_dma_cache_wback(paddr, size); break; + case DMA_FROM_DEVICE: - ALT_CMO_OP(clean, vaddr, size, riscv_cbom_block_size); - break; + if (!arch_sync_dma_clean_before_fromdevice()) { + arch_dma_cache_inv(paddr, size); + break; + } + fallthrough; + case DMA_BIDIRECTIONAL: - ALT_CMO_OP(flush, vaddr, size, riscv_cbom_block_size); + /* Skip the invalidate here if it's done later */ + if (IS_ENABLED(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) && + arch_sync_dma_cpu_needs_post_dma_flush()) + arch_dma_cache_wback(paddr, size); + else + arch_dma_cache_wback_inv(paddr, size); break; + default: break; } @@ -37,15 +104,17 @@ void arch_sync_dma_for_device(phys_addr_t paddr, size_t size, void arch_sync_dma_for_cpu(phys_addr_t paddr, size_t size, enum dma_data_direction dir) { - void *vaddr = phys_to_virt(paddr); - switch (dir) { case DMA_TO_DEVICE: break; + case DMA_FROM_DEVICE: case DMA_BIDIRECTIONAL: - ALT_CMO_OP(flush, vaddr, size, riscv_cbom_block_size); + /* FROM_DEVICE invalidate needed if speculative CPU prefetch only */ + if (arch_sync_dma_cpu_needs_post_dma_flush()) + arch_dma_cache_inv(paddr, size); break; + default: break; } @@ -55,6 +124,13 @@ void arch_dma_prep_coherent(struct page *page, size_t size) { void *flush_addr = page_address(page); +#ifdef CONFIG_RISCV_NONSTANDARD_CACHE_OPS + if (unlikely(noncoherent_cache_ops.wback_inv)) { + noncoherent_cache_ops.wback_inv(page_to_phys(page), size); + return; + } +#endif + ALT_CMO_OP(flush, flush_addr, size, riscv_cbom_block_size); } @@ -86,3 +162,12 @@ void __init riscv_set_dma_cache_alignment(void) if (!noncoherent_supported) dma_cache_alignment = 1; } + +void riscv_noncoherent_register_cache_ops(const struct riscv_nonstd_cache_ops *ops) +{ + if (!ops) + return; + + noncoherent_cache_ops = *ops; +} +EXPORT_SYMBOL_GPL(riscv_noncoherent_register_cache_ops); diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 943c18d6ef4d..0798bd861dcb 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1014,11 +1014,45 @@ static void __init pt_ops_set_late(void) #endif } +#ifdef CONFIG_RANDOMIZE_BASE +extern bool __init __pi_set_nokaslr_from_cmdline(uintptr_t dtb_pa); +extern u64 __init __pi_get_kaslr_seed(uintptr_t dtb_pa); + +static int __init print_nokaslr(char *p) +{ + pr_info("Disabled KASLR"); + return 0; +} +early_param("nokaslr", print_nokaslr); + +unsigned long kaslr_offset(void) +{ + return kernel_map.virt_offset; +} +#endif + asmlinkage void __init setup_vm(uintptr_t dtb_pa) { pmd_t __maybe_unused fix_bmap_spmd, fix_bmap_epmd; - kernel_map.virt_addr = KERNEL_LINK_ADDR; +#ifdef CONFIG_RANDOMIZE_BASE + if (!__pi_set_nokaslr_from_cmdline(dtb_pa)) { + u64 kaslr_seed = __pi_get_kaslr_seed(dtb_pa); + u32 kernel_size = (uintptr_t)(&_end) - (uintptr_t)(&_start); + u32 nr_pos; + + /* + * Compute the number of positions available: we are limited + * by the early page table that only has one PUD and we must + * be aligned on PMD_SIZE. + */ + nr_pos = (PUD_SIZE - kernel_size) / PMD_SIZE; + + kernel_map.virt_offset = (kaslr_seed % nr_pos) * PMD_SIZE; + } +#endif + + kernel_map.virt_addr = KERNEL_LINK_ADDR + kernel_map.virt_offset; kernel_map.page_offset = _AC(CONFIG_PAGE_OFFSET, UL); #ifdef CONFIG_XIP_KERNEL diff --git a/arch/riscv/mm/pmem.c b/arch/riscv/mm/pmem.c index 089df92ae876..c5fc5ec96f6d 100644 --- a/arch/riscv/mm/pmem.c +++ b/arch/riscv/mm/pmem.c @@ -7,15 +7,28 @@ #include #include +#include void arch_wb_cache_pmem(void *addr, size_t size) { +#ifdef CONFIG_RISCV_NONSTANDARD_CACHE_OPS + if (unlikely(noncoherent_cache_ops.wback)) { + noncoherent_cache_ops.wback(virt_to_phys(addr), size); + return; + } +#endif ALT_CMO_OP(clean, addr, size, riscv_cbom_block_size); } EXPORT_SYMBOL_GPL(arch_wb_cache_pmem); void arch_invalidate_pmem(void *addr, size_t size) { +#ifdef CONFIG_RISCV_NONSTANDARD_CACHE_OPS + if (unlikely(noncoherent_cache_ops.inv)) { + noncoherent_cache_ops.inv(virt_to_phys(addr), size); + return; + } +#endif ALT_CMO_OP(inval, addr, size, riscv_cbom_block_size); } EXPORT_SYMBOL_GPL(arch_invalidate_pmem); diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h index d21c6c92a683..a5ce1ab76ece 100644 --- a/arch/riscv/net/bpf_jit.h +++ b/arch/riscv/net/bpf_jit.h @@ -68,6 +68,7 @@ static inline bool is_creg(u8 reg) struct rv_jit_context { struct bpf_prog *prog; u16 *insns; /* RV insns */ + u16 *ro_insns; int ninsns; int prologue_len; int epilogue_offset; @@ -85,7 +86,9 @@ static inline int ninsns_rvoff(int ninsns) struct rv_jit_data { struct bpf_binary_header *header; + struct bpf_binary_header *ro_header; u8 *image; + u8 *ro_image; struct rv_jit_context ctx; }; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 8423f4ddf8f5..ecd3ae6f4116 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -144,7 +144,11 @@ static bool in_auipc_jalr_range(s64 val) /* Emit fixed-length instructions for address */ static int emit_addr(u8 rd, u64 addr, bool extra_pass, struct rv_jit_context *ctx) { - u64 ip = (u64)(ctx->insns + ctx->ninsns); + /* + * Use the ro_insns(RX) to calculate the offset as the BPF program will + * finally run from this memory region. + */ + u64 ip = (u64)(ctx->ro_insns + ctx->ninsns); s64 off = addr - ip; s64 upper = (off + (1 << 11)) >> 12; s64 lower = off & 0xfff; @@ -464,8 +468,12 @@ static int emit_call(u64 addr, bool fixed_addr, struct rv_jit_context *ctx) s64 off = 0; u64 ip; - if (addr && ctx->insns) { - ip = (u64)(long)(ctx->insns + ctx->ninsns); + if (addr && ctx->insns && ctx->ro_insns) { + /* + * Use the ro_insns(RX) to calculate the offset as the BPF + * program will finally run from this memory region. + */ + ip = (u64)(long)(ctx->ro_insns + ctx->ninsns); off = addr - ip; } @@ -578,9 +586,10 @@ static int add_exception_handler(const struct bpf_insn *insn, { struct exception_table_entry *ex; unsigned long pc; - off_t offset; + off_t ins_offset; + off_t fixup_offset; - if (!ctx->insns || !ctx->prog->aux->extable || + if (!ctx->insns || !ctx->ro_insns || !ctx->prog->aux->extable || (BPF_MODE(insn->code) != BPF_PROBE_MEM && BPF_MODE(insn->code) != BPF_PROBE_MEMSX)) return 0; @@ -594,12 +603,17 @@ static int add_exception_handler(const struct bpf_insn *insn, return -EINVAL; ex = &ctx->prog->aux->extable[ctx->nexentries]; - pc = (unsigned long)&ctx->insns[ctx->ninsns - insn_len]; + pc = (unsigned long)&ctx->ro_insns[ctx->ninsns - insn_len]; - offset = pc - (long)&ex->insn; - if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN)) + /* + * This is the relative offset of the instruction that may fault from + * the exception table itself. This will be written to the exception + * table and if this instruction faults, the destination register will + * be set to '0' and the execution will jump to the next instruction. + */ + ins_offset = pc - (long)&ex->insn; + if (WARN_ON_ONCE(ins_offset >= 0 || ins_offset < INT_MIN)) return -ERANGE; - ex->insn = offset; /* * Since the extable follows the program, the fixup offset is always @@ -608,12 +622,25 @@ static int add_exception_handler(const struct bpf_insn *insn, * bits. We don't need to worry about buildtime or runtime sort * modifying the upper bits because the table is already sorted, and * isn't part of the main exception table. + * + * The fixup_offset is set to the next instruction from the instruction + * that may fault. The execution will jump to this after handling the + * fault. */ - offset = (long)&ex->fixup - (pc + insn_len * sizeof(u16)); - if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, offset)) + fixup_offset = (long)&ex->fixup - (pc + insn_len * sizeof(u16)); + if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, fixup_offset)) return -ERANGE; - ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, offset) | + /* + * The offsets above have been calculated using the RO buffer but we + * need to use the R/W buffer for writes. + * switch ex to rw buffer for writing. + */ + ex = (void *)ctx->insns + ((void *)ex - (void *)ctx->ro_insns); + + ex->insn = ins_offset; + + ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, fixup_offset) | FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg); ex->type = EX_TYPE_BPF; @@ -1007,6 +1034,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, ctx.ninsns = 0; ctx.insns = NULL; + ctx.ro_insns = NULL; ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); if (ret < 0) return ret; @@ -1015,7 +1043,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, return -EFBIG; ctx.ninsns = 0; + /* + * The bpf_int_jit_compile() uses a RW buffer (ctx.insns) to write the + * JITed instructions and later copies it to a RX region (ctx.ro_insns). + * It also uses ctx.ro_insns to calculate offsets for jumps etc. As the + * trampoline image uses the same memory area for writing and execution, + * both ctx.insns and ctx.ro_insns can be set to image. + */ ctx.insns = image; + ctx.ro_insns = image; ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); if (ret < 0) return ret; diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index 7a26a3e1c73c..7b70ccb7fec3 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -8,6 +8,8 @@ #include #include +#include +#include #include "bpf_jit.h" /* Number of iterations to try until offsets converge. */ @@ -117,16 +119,24 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) sizeof(struct exception_table_entry); prog_size = sizeof(*ctx->insns) * ctx->ninsns; - jit_data->header = - bpf_jit_binary_alloc(prog_size + extable_size, - &jit_data->image, - sizeof(u32), - bpf_fill_ill_insns); - if (!jit_data->header) { + jit_data->ro_header = + bpf_jit_binary_pack_alloc(prog_size + extable_size, + &jit_data->ro_image, sizeof(u32), + &jit_data->header, &jit_data->image, + bpf_fill_ill_insns); + if (!jit_data->ro_header) { prog = orig_prog; goto out_offset; } + /* + * Use the image(RW) for writing the JITed instructions. But also save + * the ro_image(RX) for calculating the offsets in the image. The RW + * image will be later copied to the RX image from where the program + * will run. The bpf_jit_binary_pack_finalize() will do this copy in the + * final step. + */ + ctx->ro_insns = (u16 *)jit_data->ro_image; ctx->insns = (u16 *)jit_data->image; /* * Now, when the image is allocated, the image can @@ -138,14 +148,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) if (i == NR_JIT_ITERATIONS) { pr_err("bpf-jit: image did not converge in <%d passes!\n", i); - if (jit_data->header) - bpf_jit_binary_free(jit_data->header); prog = orig_prog; - goto out_offset; + goto out_free_hdr; } if (extable_size) - prog->aux->extable = (void *)ctx->insns + prog_size; + prog->aux->extable = (void *)ctx->ro_insns + prog_size; skip_init_ctx: pass++; @@ -154,23 +162,33 @@ skip_init_ctx: bpf_jit_build_prologue(ctx); if (build_body(ctx, extra_pass, NULL)) { - bpf_jit_binary_free(jit_data->header); prog = orig_prog; - goto out_offset; + goto out_free_hdr; } bpf_jit_build_epilogue(ctx); if (bpf_jit_enable > 1) bpf_jit_dump(prog->len, prog_size, pass, ctx->insns); - prog->bpf_func = (void *)ctx->insns; + prog->bpf_func = (void *)ctx->ro_insns; prog->jited = 1; prog->jited_len = prog_size; - bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); - if (!prog->is_func || extra_pass) { - bpf_jit_binary_lock_ro(jit_data->header); + if (WARN_ON(bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, + jit_data->header))) { + /* ro_header has been freed */ + jit_data->ro_header = NULL; + prog = orig_prog; + goto out_offset; + } + /* + * The instructions have now been copied to the ROX region from + * where they will execute. + * Write any modified data cache blocks out to memory and + * invalidate the corresponding blocks in the instruction cache. + */ + bpf_flush_icache(jit_data->ro_header, ctx->ro_insns + ctx->ninsns); for (i = 0; i < prog->len; i++) ctx->offset[i] = ninsns_rvoff(ctx->offset[i]); bpf_prog_fill_jited_linfo(prog, ctx->offset); @@ -185,6 +203,14 @@ out: bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog); return prog; + +out_free_hdr: + if (jit_data->header) { + bpf_arch_text_copy(&jit_data->ro_header->size, &jit_data->header->size, + sizeof(jit_data->header->size)); + bpf_jit_binary_pack_free(jit_data->ro_header, jit_data->header); + } + goto out_offset; } u64 bpf_jit_alloc_exec_limit(void) @@ -204,3 +230,51 @@ void bpf_jit_free_exec(void *addr) { return vfree(addr); } + +void *bpf_arch_text_copy(void *dst, void *src, size_t len) +{ + int ret; + + mutex_lock(&text_mutex); + ret = patch_text_nosync(dst, src, len); + mutex_unlock(&text_mutex); + + if (ret) + return ERR_PTR(-EINVAL); + + return dst; +} + +int bpf_arch_text_invalidate(void *dst, size_t len) +{ + int ret; + + mutex_lock(&text_mutex); + ret = patch_text_set_nosync(dst, 0, len); + mutex_unlock(&text_mutex); + + return ret; +} + +void bpf_jit_free(struct bpf_prog *prog) +{ + if (prog->jited) { + struct rv_jit_data *jit_data = prog->aux->jit_data; + struct bpf_binary_header *hdr; + + /* + * If we fail the final pass of JIT (from jit_subprogs), + * the program may not be finalized yet. Call finalize here + * before freeing it. + */ + if (jit_data) { + bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, jit_data->header); + kfree(jit_data); + } + hdr = bpf_jit_binary_pack_hdr(prog); + bpf_jit_binary_pack_free(hdr, NULL); + WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog)); + } + + bpf_prog_unlock_free(prog); +} diff --git a/arch/s390/boot/ipl_parm.c b/arch/s390/boot/ipl_parm.c index 8753cb0339e5..7b7521762633 100644 --- a/arch/s390/boot/ipl_parm.c +++ b/arch/s390/boot/ipl_parm.c @@ -19,7 +19,6 @@ struct parmarea parmarea __section(".parmarea") = { }; char __bootdata(early_command_line)[COMMAND_LINE_SIZE]; -int __bootdata(noexec_disabled); unsigned int __bootdata_preserved(zlib_dfltcc_support) = ZLIB_DFLTCC_FULL; struct ipl_parameter_block __bootdata_preserved(ipl_block); @@ -290,12 +289,6 @@ void parse_boot_command_line(void) zlib_dfltcc_support = ZLIB_DFLTCC_FULL_DEBUG; } - if (!strcmp(param, "noexec")) { - rc = kstrtobool(val, &enabled); - if (!rc && !enabled) - noexec_disabled = 1; - } - if (!strcmp(param, "facilities") && val) modify_fac_list(val); diff --git a/arch/s390/boot/startup.c b/arch/s390/boot/startup.c index b9681cb22753..d3e48bd9c394 100644 --- a/arch/s390/boot/startup.c +++ b/arch/s390/boot/startup.c @@ -53,10 +53,8 @@ static void detect_facilities(void) } if (test_facility(78)) machine.has_edat2 = 1; - if (!noexec_disabled && test_facility(130)) { + if (test_facility(130)) machine.has_nx = 1; - __ctl_set_bit(0, 20); - } } static void setup_lpp(void) diff --git a/arch/s390/boot/vmem.c b/arch/s390/boot/vmem.c index c67f59db7a51..01257ce3b89c 100644 --- a/arch/s390/boot/vmem.c +++ b/arch/s390/boot/vmem.c @@ -287,7 +287,9 @@ static void pgtable_pte_populate(pmd_t *pmd, unsigned long addr, unsigned long e if (kasan_pte_populate_zero_shadow(pte, mode)) continue; entry = __pte(_pa(addr, PAGE_SIZE, mode)); - entry = set_pte_bit(entry, PAGE_KERNEL_EXEC); + entry = set_pte_bit(entry, PAGE_KERNEL); + if (!machine.has_nx) + entry = clear_pte_bit(entry, __pgprot(_PAGE_NOEXEC)); set_pte(pte, entry); pages++; } @@ -311,7 +313,9 @@ static void pgtable_pmd_populate(pud_t *pud, unsigned long addr, unsigned long e continue; if (can_large_pmd(pmd, addr, next)) { entry = __pmd(_pa(addr, _SEGMENT_SIZE, mode)); - entry = set_pmd_bit(entry, SEGMENT_KERNEL_EXEC); + entry = set_pmd_bit(entry, SEGMENT_KERNEL); + if (!machine.has_nx) + entry = clear_pmd_bit(entry, __pgprot(_SEGMENT_ENTRY_NOEXEC)); set_pmd(pmd, entry); pages++; continue; @@ -342,7 +346,9 @@ static void pgtable_pud_populate(p4d_t *p4d, unsigned long addr, unsigned long e continue; if (can_large_pud(pud, addr, next)) { entry = __pud(_pa(addr, _REGION3_SIZE, mode)); - entry = set_pud_bit(entry, REGION3_KERNEL_EXEC); + entry = set_pud_bit(entry, REGION3_KERNEL); + if (!machine.has_nx) + entry = clear_pud_bit(entry, __pgprot(_REGION_ENTRY_NOEXEC)); set_pud(pud, entry); pages++; continue; diff --git a/arch/s390/include/asm/airq.h b/arch/s390/include/asm/airq.h index e82e5626e139..c4c28c2609a5 100644 --- a/arch/s390/include/asm/airq.h +++ b/arch/s390/include/asm/airq.h @@ -18,7 +18,6 @@ struct airq_struct { struct hlist_node list; /* Handler queueing. */ void (*handler)(struct airq_struct *airq, struct tpi_info *tpi_info); u8 *lsi_ptr; /* Local-Summary-Indicator pointer */ - u8 lsi_mask; /* Local-Summary-Indicator mask */ u8 isc; /* Interrupt-subclass */ u8 flags; }; diff --git a/arch/s390/include/asm/dma.h b/arch/s390/include/asm/dma.h index c260adb25997..7fe3e31956d7 100644 --- a/arch/s390/include/asm/dma.h +++ b/arch/s390/include/asm/dma.h @@ -9,6 +9,6 @@ * to DMA. It _is_ used for the s390 memory zone split at 2GB caused * by the 31 bit heritage. */ -#define MAX_DMA_ADDRESS 0x80000000 +#define MAX_DMA_ADDRESS __va(0x80000000) #endif /* _ASM_S390_DMA_H */ diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h index 91bfecb91321..427f9528a7b6 100644 --- a/arch/s390/include/asm/kvm_host.h +++ b/arch/s390/include/asm/kvm_host.h @@ -817,6 +817,8 @@ struct kvm_s390_cpu_model { __u64 *fac_list; u64 cpuid; unsigned short ibc; + /* subset of available UV-features for pv-guests enabled by user space */ + struct kvm_s390_vm_cpu_uv_feat uv_feat_guest; }; typedef int (*crypto_hook)(struct kvm_vcpu *vcpu); diff --git a/arch/s390/include/asm/sections.h b/arch/s390/include/asm/sections.h index 3fecaa4e8b74..0486e6ef62bf 100644 --- a/arch/s390/include/asm/sections.h +++ b/arch/s390/include/asm/sections.h @@ -23,7 +23,7 @@ */ #define __bootdata_preserved(var) __section(".boot.preserved.data." #var) var -extern unsigned long __samode31, __eamode31; -extern unsigned long __stext_amode31, __etext_amode31; +extern char *__samode31, *__eamode31; +extern char *__stext_amode31, *__etext_amode31; #endif diff --git a/arch/s390/include/asm/set_memory.h b/arch/s390/include/asm/set_memory.h index 7a3eefd7a242..06fbabe2f66c 100644 --- a/arch/s390/include/asm/set_memory.h +++ b/arch/s390/include/asm/set_memory.h @@ -24,43 +24,41 @@ enum { #define SET_MEMORY_INV BIT(_SET_MEMORY_INV_BIT) #define SET_MEMORY_DEF BIT(_SET_MEMORY_DEF_BIT) -int __set_memory(unsigned long addr, int numpages, unsigned long flags); - -static inline int set_memory_ro(unsigned long addr, int numpages) -{ - return __set_memory(addr, numpages, SET_MEMORY_RO); -} - -static inline int set_memory_rw(unsigned long addr, int numpages) -{ - return __set_memory(addr, numpages, SET_MEMORY_RW); -} - -static inline int set_memory_nx(unsigned long addr, int numpages) -{ - return __set_memory(addr, numpages, SET_MEMORY_NX); -} - -static inline int set_memory_x(unsigned long addr, int numpages) -{ - return __set_memory(addr, numpages, SET_MEMORY_X); -} +int __set_memory(unsigned long addr, unsigned long numpages, unsigned long flags); #define set_memory_rox set_memory_rox -static inline int set_memory_rox(unsigned long addr, int numpages) -{ - return __set_memory(addr, numpages, SET_MEMORY_RO | SET_MEMORY_X); + +/* + * Generate two variants of each set_memory() function: + * + * set_memory_yy(unsigned long addr, int numpages); + * __set_memory_yy(void *start, void *end); + * + * The second variant exists for both convenience to avoid the usual + * (unsigned long) casts, but unlike the first variant it can also be used + * for areas larger than 8TB, which may happen at memory initialization. + */ +#define __SET_MEMORY_FUNC(fname, flags) \ +static inline int fname(unsigned long addr, int numpages) \ +{ \ + return __set_memory(addr, numpages, (flags)); \ +} \ + \ +static inline int __##fname(void *start, void *end) \ +{ \ + unsigned long numpages; \ + \ + numpages = (end - start) >> PAGE_SHIFT; \ + return __set_memory((unsigned long)start, numpages, (flags)); \ } -static inline int set_memory_rwnx(unsigned long addr, int numpages) -{ - return __set_memory(addr, numpages, SET_MEMORY_RW | SET_MEMORY_NX); -} - -static inline int set_memory_4k(unsigned long addr, int numpages) -{ - return __set_memory(addr, numpages, SET_MEMORY_4K); -} +__SET_MEMORY_FUNC(set_memory_ro, SET_MEMORY_RO) +__SET_MEMORY_FUNC(set_memory_rw, SET_MEMORY_RW) +__SET_MEMORY_FUNC(set_memory_nx, SET_MEMORY_NX) +__SET_MEMORY_FUNC(set_memory_x, SET_MEMORY_X) +__SET_MEMORY_FUNC(set_memory_rox, SET_MEMORY_RO | SET_MEMORY_X) +__SET_MEMORY_FUNC(set_memory_rwnx, SET_MEMORY_RW | SET_MEMORY_NX) +__SET_MEMORY_FUNC(set_memory_4k, SET_MEMORY_4K) int set_direct_map_invalid_noflush(struct page *page); int set_direct_map_default_noflush(struct page *page); diff --git a/arch/s390/include/asm/setup.h b/arch/s390/include/asm/setup.h index b30fe91166e3..25cadc2b9cff 100644 --- a/arch/s390/include/asm/setup.h +++ b/arch/s390/include/asm/setup.h @@ -72,7 +72,6 @@ extern unsigned int zlib_dfltcc_support; #define ZLIB_DFLTCC_INFLATE_ONLY 3 #define ZLIB_DFLTCC_FULL_DEBUG 4 -extern int noexec_disabled; extern unsigned long ident_map_size; extern unsigned long max_mappable; diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h index d2cd42bb2c26..0e7bd3873907 100644 --- a/arch/s390/include/asm/uv.h +++ b/arch/s390/include/asm/uv.h @@ -99,6 +99,8 @@ enum uv_cmds_inst { enum uv_feat_ind { BIT_UV_FEAT_MISC = 0, BIT_UV_FEAT_AIV = 1, + BIT_UV_FEAT_AP = 4, + BIT_UV_FEAT_AP_INTR = 5, }; struct uv_cb_header { @@ -159,7 +161,15 @@ struct uv_cb_cgc { u64 guest_handle; u64 conf_base_stor_origin; u64 conf_virt_stor_origin; - u64 reserved30; + u8 reserved30[6]; + union { + struct { + u16 : 14; + u16 ap_instr_intr : 1; + u16 ap_allow_instr : 1; + }; + u16 raw; + } flags; u64 guest_stor_origin; u64 guest_stor_len; u64 guest_sca; @@ -397,6 +407,13 @@ struct uv_info { extern struct uv_info uv_info; +static inline bool uv_has_feature(u8 feature_bit) +{ + if (feature_bit >= sizeof(uv_info.uv_feature_indications) * 8) + return false; + return test_bit_inv(feature_bit, &uv_info.uv_feature_indications); +} + #ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST extern int prot_virt_guest; diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h index a73cf01a1606..abe926d43cbe 100644 --- a/arch/s390/include/uapi/asm/kvm.h +++ b/arch/s390/include/uapi/asm/kvm.h @@ -159,6 +159,22 @@ struct kvm_s390_vm_cpu_subfunc { __u8 reserved[1728]; }; +#define KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST 6 +#define KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST 7 + +#define KVM_S390_VM_CPU_UV_FEAT_NR_BITS 64 +struct kvm_s390_vm_cpu_uv_feat { + union { + struct { + __u64 : 4; + __u64 ap : 1; /* bit 4 */ + __u64 ap_intr : 1; /* bit 5 */ + __u64 : 58; + }; + __u64 feat; + }; +}; + /* kvm attributes for crypto */ #define KVM_S390_VM_CRYPTO_ENABLE_AES_KW 0 #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1 diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c index 2dd5976a55ac..442ce0489e1a 100644 --- a/arch/s390/kernel/early.c +++ b/arch/s390/kernel/early.c @@ -44,7 +44,6 @@ early_param(#param, ignore_decompressor_param_##param) decompressor_handled_param(mem); decompressor_handled_param(vmalloc); decompressor_handled_param(dfltcc); -decompressor_handled_param(noexec); decompressor_handled_param(facilities); decompressor_handled_param(nokaslr); #if IS_ENABLED(CONFIG_KVM) @@ -233,10 +232,8 @@ static __init void detect_machine_facilities(void) S390_lowcore.machine_flags |= MACHINE_FLAG_VX; __ctl_set_bit(0, 17); } - if (test_facility(130) && !noexec_disabled) { + if (test_facility(130)) S390_lowcore.machine_flags |= MACHINE_FLAG_NX; - __ctl_set_bit(0, 20); - } if (test_facility(133)) S390_lowcore.machine_flags |= MACHINE_FLAG_GS; if (test_facility(139) && (tod_clock_base.tod >> 63)) { diff --git a/arch/s390/kernel/machine_kexec.c b/arch/s390/kernel/machine_kexec.c index 12a2bd4fc88c..ce65fc01671f 100644 --- a/arch/s390/kernel/machine_kexec.c +++ b/arch/s390/kernel/machine_kexec.c @@ -216,8 +216,8 @@ void arch_crash_save_vmcoreinfo(void) VMCOREINFO_SYMBOL(lowcore_ptr); VMCOREINFO_SYMBOL(high_memory); VMCOREINFO_LENGTH(lowcore_ptr, NR_CPUS); - vmcoreinfo_append_str("SAMODE31=%lx\n", __samode31); - vmcoreinfo_append_str("EAMODE31=%lx\n", __eamode31); + vmcoreinfo_append_str("SAMODE31=%lx\n", (unsigned long)__samode31); + vmcoreinfo_append_str("EAMODE31=%lx\n", (unsigned long)__eamode31); vmcoreinfo_append_str("KERNELOFFSET=%lx\n", kaslr_offset()); abs_lc = get_abs_lowcore(); abs_lc->vmcore_info = paddr_vmcoreinfo_note(); diff --git a/arch/s390/kernel/setup.c b/arch/s390/kernel/setup.c index c744104e4a9c..de6ad0fb2328 100644 --- a/arch/s390/kernel/setup.c +++ b/arch/s390/kernel/setup.c @@ -97,10 +97,10 @@ EXPORT_SYMBOL(console_irq); * relocated above 2 GB, because it has to use 31 bit addresses. * Such code and data is part of the .amode31 section. */ -unsigned long __amode31_ref __samode31 = (unsigned long)&_samode31; -unsigned long __amode31_ref __eamode31 = (unsigned long)&_eamode31; -unsigned long __amode31_ref __stext_amode31 = (unsigned long)&_stext_amode31; -unsigned long __amode31_ref __etext_amode31 = (unsigned long)&_etext_amode31; +char __amode31_ref *__samode31 = _samode31; +char __amode31_ref *__eamode31 = _eamode31; +char __amode31_ref *__stext_amode31 = _stext_amode31; +char __amode31_ref *__etext_amode31 = _etext_amode31; struct exception_table_entry __amode31_ref *__start_amode31_ex_table = _start_amode31_ex_table; struct exception_table_entry __amode31_ref *__stop_amode31_ex_table = _stop_amode31_ex_table; @@ -145,7 +145,6 @@ static u32 __amode31_ref *__ctl_duald = __ctl_duald_amode31; static u32 __amode31_ref *__ctl_linkage_stack = __ctl_linkage_stack_amode31; static u32 __amode31_ref *__ctl_duct = __ctl_duct_amode31; -int __bootdata(noexec_disabled); unsigned long __bootdata_preserved(max_mappable); unsigned long __bootdata(ident_map_size); struct physmem_info __bootdata(physmem_info); @@ -771,15 +770,15 @@ static void __init setup_memory(void) static void __init relocate_amode31_section(void) { unsigned long amode31_size = __eamode31 - __samode31; - long amode31_offset = physmem_info.reserved[RR_AMODE31].start - __samode31; - long *ptr; + long amode31_offset, *ptr; + amode31_offset = physmem_info.reserved[RR_AMODE31].start - (unsigned long)__samode31; pr_info("Relocating AMODE31 section of size 0x%08lx\n", amode31_size); /* Move original AMODE31 section to the new one */ - memmove((void *)physmem_info.reserved[RR_AMODE31].start, (void *)__samode31, amode31_size); + memmove((void *)physmem_info.reserved[RR_AMODE31].start, __samode31, amode31_size); /* Zero out the old AMODE31 section to catch invalid accesses within it */ - memset((void *)__samode31, 0, amode31_size); + memset(__samode31, 0, amode31_size); /* Update all AMODE31 region references */ for (ptr = _start_amode31_refs; ptr != _end_amode31_refs; ptr++) diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c index b771f1b4cdd1..fc07bc39e698 100644 --- a/arch/s390/kernel/uv.c +++ b/arch/s390/kernel/uv.c @@ -258,7 +258,7 @@ static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_str * shared page from a different protected VM will automatically also * transfer its ownership. */ - if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications)) + if (uv_has_feature(BIT_UV_FEAT_MISC)) return false; if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED) return false; diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c index 341abafb96e4..b16352083ff9 100644 --- a/arch/s390/kvm/intercept.c +++ b/arch/s390/kvm/intercept.c @@ -228,6 +228,21 @@ static int handle_itdb(struct kvm_vcpu *vcpu) #define per_event(vcpu) (vcpu->arch.sie_block->iprcc & PGM_PER) +static bool should_handle_per_event(const struct kvm_vcpu *vcpu) +{ + if (!guestdbg_enabled(vcpu) || !per_event(vcpu)) + return false; + if (guestdbg_sstep_enabled(vcpu) && + vcpu->arch.sie_block->iprcc != PGM_PER) { + /* + * __vcpu_run() will exit after delivering the concurrently + * indicated condition. + */ + return false; + } + return true; +} + static int handle_prog(struct kvm_vcpu *vcpu) { psw_t psw; @@ -242,7 +257,7 @@ static int handle_prog(struct kvm_vcpu *vcpu) if (kvm_s390_pv_cpu_is_protected(vcpu)) return -EOPNOTSUPP; - if (guestdbg_enabled(vcpu) && per_event(vcpu)) { + if (should_handle_per_event(vcpu)) { rc = kvm_s390_handle_per_event(vcpu); if (rc) return rc; @@ -571,6 +586,19 @@ static int handle_pv_notification(struct kvm_vcpu *vcpu) return handle_instruction(vcpu); } +static bool should_handle_per_ifetch(const struct kvm_vcpu *vcpu, int rc) +{ + /* Process PER, also if the instruction is processed in user space. */ + if (!(vcpu->arch.sie_block->icptstatus & 0x02)) + return false; + if (rc != 0 && rc != -EOPNOTSUPP) + return false; + if (guestdbg_sstep_enabled(vcpu) && vcpu->arch.local_int.pending_irqs) + /* __vcpu_run() will exit after delivering the interrupt. */ + return false; + return true; +} + int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu) { int rc, per_rc = 0; @@ -605,8 +633,8 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu) rc = handle_partial_execution(vcpu); break; case ICPT_KSS: - rc = kvm_s390_skey_check_enable(vcpu); - break; + /* Instruction will be redriven, skip the PER check. */ + return kvm_s390_skey_check_enable(vcpu); case ICPT_MCHKREQ: case ICPT_INT_ENABLE: /* @@ -633,9 +661,7 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu) return -EOPNOTSUPP; } - /* process PER, also if the instruction is processed in user space */ - if (vcpu->arch.sie_block->icptstatus & 0x02 && - (!rc || rc == -EOPNOTSUPP)) + if (should_handle_per_ifetch(vcpu, rc)) per_rc = kvm_s390_handle_per_ifetch_icpt(vcpu); return per_rc ? per_rc : rc; } diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c index 9bd0a873f3b1..c1b47d608a2b 100644 --- a/arch/s390/kvm/interrupt.c +++ b/arch/s390/kvm/interrupt.c @@ -1392,6 +1392,7 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) { struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int; int rc = 0; + bool delivered = false; unsigned long irq_type; unsigned long irqs; @@ -1465,6 +1466,19 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) WARN_ONCE(1, "Unknown pending irq type %ld", irq_type); clear_bit(irq_type, &li->pending_irqs); } + delivered |= !rc; + } + + /* + * We delivered at least one interrupt and modified the PC. Force a + * singlestep event now. + */ + if (delivered && guestdbg_sstep_enabled(vcpu)) { + struct kvm_debug_exit_arch *debug_exit = &vcpu->run->debug.arch; + + debug_exit->addr = vcpu->arch.sie_block->gpsw.addr; + debug_exit->type = KVM_SINGLESTEP; + vcpu->guest_debug |= KVM_GUESTDBG_EXIT_PENDING; } set_intercept_indicators(vcpu); @@ -3398,7 +3412,6 @@ static void gib_alert_irq_handler(struct airq_struct *airq, static struct airq_struct gib_alert_irq = { .handler = gib_alert_irq_handler, - .lsi_ptr = &gib_alert_irq.lsi_mask, }; void kvm_s390_gib_destroy(void) @@ -3438,6 +3451,8 @@ int __init kvm_s390_gib_init(u8 nisc) rc = -EIO; goto out_free_gib; } + /* adapter interrupts used for AP (applicable here) don't use the LSI */ + *gib_alert_irq.lsi_ptr = 0xff; gib->nisc = nisc; gib_origin = virt_to_phys(gib); diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index d1e768bcfe1d..b3f17e014cab 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -1531,6 +1531,39 @@ static int kvm_s390_set_processor_subfunc(struct kvm *kvm, return 0; } +#define KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK \ +( \ + ((struct kvm_s390_vm_cpu_uv_feat){ \ + .ap = 1, \ + .ap_intr = 1, \ + }) \ + .feat \ +) + +static int kvm_s390_set_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr) +{ + struct kvm_s390_vm_cpu_uv_feat __user *ptr = (void __user *)attr->addr; + unsigned long data, filter; + + filter = uv_info.uv_feature_indications & KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK; + if (get_user(data, &ptr->feat)) + return -EFAULT; + if (!bitmap_subset(&data, &filter, KVM_S390_VM_CPU_UV_FEAT_NR_BITS)) + return -EINVAL; + + mutex_lock(&kvm->lock); + if (kvm->created_vcpus) { + mutex_unlock(&kvm->lock); + return -EBUSY; + } + kvm->arch.model.uv_feat_guest.feat = data; + mutex_unlock(&kvm->lock); + + VM_EVENT(kvm, 3, "SET: guest UV-feat: 0x%16.16lx", data); + + return 0; +} + static int kvm_s390_set_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr) { int ret = -ENXIO; @@ -1545,6 +1578,9 @@ static int kvm_s390_set_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr) case KVM_S390_VM_CPU_PROCESSOR_SUBFUNC: ret = kvm_s390_set_processor_subfunc(kvm, attr); break; + case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST: + ret = kvm_s390_set_uv_feat(kvm, attr); + break; } return ret; } @@ -1777,6 +1813,33 @@ static int kvm_s390_get_machine_subfunc(struct kvm *kvm, return 0; } +static int kvm_s390_get_processor_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr) +{ + struct kvm_s390_vm_cpu_uv_feat __user *dst = (void __user *)attr->addr; + unsigned long feat = kvm->arch.model.uv_feat_guest.feat; + + if (put_user(feat, &dst->feat)) + return -EFAULT; + VM_EVENT(kvm, 3, "GET: guest UV-feat: 0x%16.16lx", feat); + + return 0; +} + +static int kvm_s390_get_machine_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr) +{ + struct kvm_s390_vm_cpu_uv_feat __user *dst = (void __user *)attr->addr; + unsigned long feat; + + BUILD_BUG_ON(sizeof(*dst) != sizeof(uv_info.uv_feature_indications)); + + feat = uv_info.uv_feature_indications & KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK; + if (put_user(feat, &dst->feat)) + return -EFAULT; + VM_EVENT(kvm, 3, "GET: guest UV-feat: 0x%16.16lx", feat); + + return 0; +} + static int kvm_s390_get_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr) { int ret = -ENXIO; @@ -1800,6 +1863,12 @@ static int kvm_s390_get_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr) case KVM_S390_VM_CPU_MACHINE_SUBFUNC: ret = kvm_s390_get_machine_subfunc(kvm, attr); break; + case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST: + ret = kvm_s390_get_processor_uv_feat(kvm, attr); + break; + case KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST: + ret = kvm_s390_get_machine_uv_feat(kvm, attr); + break; } return ret; } @@ -1952,6 +2021,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr) case KVM_S390_VM_CPU_MACHINE_FEAT: case KVM_S390_VM_CPU_MACHINE_SUBFUNC: case KVM_S390_VM_CPU_PROCESSOR_SUBFUNC: + case KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST: + case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST: ret = 0; break; default: @@ -2406,7 +2477,7 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc) struct kvm_vcpu *vcpu; /* Disable the GISA if the ultravisor does not support AIV. */ - if (!test_bit_inv(BIT_UV_FEAT_AIV, &uv_info.uv_feature_indications)) + if (!uv_has_feature(BIT_UV_FEAT_AIV)) kvm_s390_gisa_disable(kvm); kvm_for_each_vcpu(i, vcpu, kvm) { @@ -3296,6 +3367,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm->arch.model.cpuid = kvm_s390_get_initial_cpuid(); kvm->arch.model.ibc = sclp.ibc & 0x0fff; + kvm->arch.model.uv_feat_guest.feat = 0; + kvm_s390_crypto_init(kvm); if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) { @@ -4611,7 +4684,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu) if (!kvm_is_ucontrol(vcpu->kvm)) { rc = kvm_s390_deliver_pending_interrupts(vcpu); - if (rc) + if (rc || guestdbg_exit_pending(vcpu)) return rc; } @@ -4738,7 +4811,7 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) do { rc = vcpu_pre_run(vcpu); - if (rc) + if (rc || guestdbg_exit_pending(vcpu)) break; kvm_vcpu_srcu_read_unlock(vcpu); @@ -5383,6 +5456,7 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp, { struct kvm_vcpu *vcpu = filp->private_data; void __user *argp = (void __user *)arg; + int rc; switch (ioctl) { case KVM_S390_IRQ: { @@ -5390,7 +5464,8 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp, if (copy_from_user(&s390irq, argp, sizeof(s390irq))) return -EFAULT; - return kvm_s390_inject_vcpu(vcpu, &s390irq); + rc = kvm_s390_inject_vcpu(vcpu, &s390irq); + break; } case KVM_S390_INTERRUPT: { struct kvm_s390_interrupt s390int; @@ -5400,10 +5475,25 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp, return -EFAULT; if (s390int_to_s390irq(&s390int, &s390irq)) return -EINVAL; - return kvm_s390_inject_vcpu(vcpu, &s390irq); + rc = kvm_s390_inject_vcpu(vcpu, &s390irq); + break; } + default: + rc = -ENOIOCTLCMD; + break; } - return -ENOIOCTLCMD; + + /* + * To simplify single stepping of userspace-emulated instructions, + * KVM_EXIT_S390_SIEIC exit sets KVM_GUESTDBG_EXIT_PENDING (see + * should_handle_per_ifetch()). However, if userspace emulation injects + * an interrupt, it needs to be cleared, so that KVM_EXIT_DEBUG happens + * after (and not before) the interrupt delivery. + */ + if (!rc) + vcpu->guest_debug &= ~KVM_GUESTDBG_EXIT_PENDING; + + return rc; } static int kvm_s390_handle_pv_vcpu_dump(struct kvm_vcpu *vcpu, diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c index 8d3f39a8a11e..75e81ba26d04 100644 --- a/arch/s390/kvm/pv.c +++ b/arch/s390/kvm/pv.c @@ -285,7 +285,8 @@ static int kvm_s390_pv_deinit_vm_fast(struct kvm *kvm, u16 *rc, u16 *rrc) WRITE_ONCE(kvm->arch.gmap->guest_handle, 0); KVM_UV_EVENT(kvm, 3, "PROTVIRT DESTROY VM FAST: rc %x rrc %x", uvcb.header.rc, uvcb.header.rrc); - WARN_ONCE(cc, "protvirt destroy vm fast failed handle %llx rc %x rrc %x", + WARN_ONCE(cc && uvcb.header.rc != 0x104, + "protvirt destroy vm fast failed handle %llx rc %x rrc %x", kvm_s390_pv_get_handle(kvm), uvcb.header.rc, uvcb.header.rrc); /* Intended memory leak on "impossible" error */ if (!cc) @@ -575,12 +576,14 @@ int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc) uvcb.conf_base_stor_origin = virt_to_phys((void *)kvm->arch.pv.stor_base); uvcb.conf_virt_stor_origin = (u64)kvm->arch.pv.stor_var; + uvcb.flags.ap_allow_instr = kvm->arch.model.uv_feat_guest.ap; + uvcb.flags.ap_instr_intr = kvm->arch.model.uv_feat_guest.ap_intr; cc = uv_call_sched(0, (u64)&uvcb); *rc = uvcb.header.rc; *rrc = uvcb.header.rrc; - KVM_UV_EVENT(kvm, 3, "PROTVIRT CREATE VM: handle %llx len %llx rc %x rrc %x", - uvcb.guest_handle, uvcb.guest_stor_len, *rc, *rrc); + KVM_UV_EVENT(kvm, 3, "PROTVIRT CREATE VM: handle %llx len %llx rc %x rrc %x flags %04x", + uvcb.guest_handle, uvcb.guest_stor_len, *rc, *rrc, uvcb.flags.raw); /* Outputs */ kvm->arch.pv.handle = uvcb.guest_handle; diff --git a/arch/s390/mm/dump_pagetables.c b/arch/s390/mm/dump_pagetables.c index afa5db750d92..b51666967aa1 100644 --- a/arch/s390/mm/dump_pagetables.c +++ b/arch/s390/mm/dump_pagetables.c @@ -290,8 +290,8 @@ static int pt_dump_init(void) max_addr = (S390_lowcore.kernel_asce & _REGION_ENTRY_TYPE_MASK) >> 2; max_addr = 1UL << (max_addr * 11 + 31); address_markers[IDENTITY_AFTER_END_NR].start_address = ident_map_size; - address_markers[AMODE31_START_NR].start_address = __samode31; - address_markers[AMODE31_END_NR].start_address = __eamode31; + address_markers[AMODE31_START_NR].start_address = (unsigned long)__samode31; + address_markers[AMODE31_END_NR].start_address = (unsigned long)__eamode31; address_markers[MODULES_NR].start_address = MODULES_VADDR; address_markers[MODULES_END_NR].start_address = MODULES_END; address_markers[ABS_LOWCORE_NR].start_address = __abs_lowcore; diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index 099c4824dd8a..b678295931c3 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -598,7 +598,7 @@ void do_secure_storage_access(struct pt_regs *regs) * reliable without the misc UV feature so we need to check * for that as well. */ - if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications) && + if (uv_has_feature(BIT_UV_FEAT_MISC) && !test_bit_inv(61, ®s->int_parm_long)) { /* * When this happens, userspace did something that it diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index 8d94e29adcdb..8b94d2212d33 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -98,7 +98,7 @@ void __init paging_init(void) sparse_init(); zone_dma_bits = 31; memset(max_zone_pfns, 0, sizeof(max_zone_pfns)); - max_zone_pfns[ZONE_DMA] = PFN_DOWN(MAX_DMA_ADDRESS); + max_zone_pfns[ZONE_DMA] = virt_to_pfn(MAX_DMA_ADDRESS); max_zone_pfns[ZONE_NORMAL] = max_low_pfn; free_area_init(max_zone_pfns); } @@ -107,7 +107,7 @@ void mark_rodata_ro(void) { unsigned long size = __end_ro_after_init - __start_ro_after_init; - set_memory_ro((unsigned long)__start_ro_after_init, size >> PAGE_SHIFT); + __set_memory_ro(__start_ro_after_init, __end_ro_after_init); pr_info("Write protected read-only-after-init data: %luk\n", size >> 10); debug_checkwx(); } diff --git a/arch/s390/mm/pageattr.c b/arch/s390/mm/pageattr.c index e5ec76271b16..b87e96c64b61 100644 --- a/arch/s390/mm/pageattr.c +++ b/arch/s390/mm/pageattr.c @@ -373,7 +373,7 @@ static int change_page_attr_alias(unsigned long addr, unsigned long end, return rc; } -int __set_memory(unsigned long addr, int numpages, unsigned long flags) +int __set_memory(unsigned long addr, unsigned long numpages, unsigned long flags) { unsigned long end; int rc; diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c index e44243b9c0a4..6957d2ed97bf 100644 --- a/arch/s390/mm/vmem.c +++ b/arch/s390/mm/vmem.c @@ -5,7 +5,6 @@ #include #include -#include #include #include #include @@ -291,14 +290,9 @@ out: static void try_free_pmd_table(pud_t *pud, unsigned long start) { - const unsigned long end = start + PUD_SIZE; pmd_t *pmd; int i; - /* Don't mess with any tables not fully in 1:1 mapping & vmemmap area */ - if (end > VMALLOC_START) - return; - pmd = pmd_offset(pud, start); for (i = 0; i < PTRS_PER_PMD; i++, pmd++) if (!pmd_none(*pmd)) @@ -363,14 +357,9 @@ out: static void try_free_pud_table(p4d_t *p4d, unsigned long start) { - const unsigned long end = start + P4D_SIZE; pud_t *pud; int i; - /* Don't mess with any tables not fully in 1:1 mapping & vmemmap area */ - if (end > VMALLOC_START) - return; - pud = pud_offset(p4d, start); for (i = 0; i < PTRS_PER_PUD; i++, pud++) { if (!pud_none(*pud)) @@ -413,14 +402,9 @@ out: static void try_free_p4d_table(pgd_t *pgd, unsigned long start) { - const unsigned long end = start + PGDIR_SIZE; p4d_t *p4d; int i; - /* Don't mess with any tables not fully in 1:1 mapping & vmemmap area */ - if (end > VMALLOC_START) - return; - p4d = p4d_offset(pgd, start); for (i = 0; i < PTRS_PER_P4D; i++, p4d++) { if (!p4d_none(*p4d)) @@ -440,6 +424,9 @@ static int modify_pagetable(unsigned long start, unsigned long end, bool add, if (WARN_ON_ONCE(!PAGE_ALIGNED(start | end))) return -EINVAL; + /* Don't mess with any tables not fully in 1:1 mapping & vmemmap area */ + if (WARN_ON_ONCE(end > VMALLOC_START)) + return -EINVAL; for (addr = start; addr < end; addr = next) { next = pgd_addr_end(addr, end); pgd = pgd_offset_k(addr); @@ -650,122 +637,29 @@ void vmem_unmap_4k_page(unsigned long addr) mutex_unlock(&vmem_mutex); } -static int __init memblock_region_cmp(const void *a, const void *b) -{ - const struct memblock_region *r1 = a; - const struct memblock_region *r2 = b; - - if (r1->base < r2->base) - return -1; - if (r1->base > r2->base) - return 1; - return 0; -} - -static void __init memblock_region_swap(void *a, void *b, int size) -{ - swap(*(struct memblock_region *)a, *(struct memblock_region *)b); -} - -#ifdef CONFIG_KASAN -#define __sha(x) ((unsigned long)kasan_mem_to_shadow((void *)x)) - -static inline int set_memory_kasan(unsigned long start, unsigned long end) -{ - start = PAGE_ALIGN_DOWN(__sha(start)); - end = PAGE_ALIGN(__sha(end)); - return set_memory_rwnx(start, (end - start) >> PAGE_SHIFT); -} -#endif - -/* - * map whole physical memory to virtual memory (identity mapping) - * we reserve enough space in the vmalloc area for vmemmap to hotplug - * additional memory segments. - */ void __init vmem_map_init(void) { - struct memblock_region memory_rwx_regions[] = { - { - .base = 0, - .size = sizeof(struct lowcore), - .flags = MEMBLOCK_NONE, -#ifdef CONFIG_NUMA - .nid = NUMA_NO_NODE, -#endif - }, - { - .base = __pa(_stext), - .size = _etext - _stext, - .flags = MEMBLOCK_NONE, -#ifdef CONFIG_NUMA - .nid = NUMA_NO_NODE, -#endif - }, - { - .base = __pa(_sinittext), - .size = _einittext - _sinittext, - .flags = MEMBLOCK_NONE, -#ifdef CONFIG_NUMA - .nid = NUMA_NO_NODE, -#endif - }, - { - .base = __stext_amode31, - .size = __etext_amode31 - __stext_amode31, - .flags = MEMBLOCK_NONE, -#ifdef CONFIG_NUMA - .nid = NUMA_NO_NODE, -#endif - }, - }; - struct memblock_type memory_rwx = { - .regions = memory_rwx_regions, - .cnt = ARRAY_SIZE(memory_rwx_regions), - .max = ARRAY_SIZE(memory_rwx_regions), - }; - phys_addr_t base, end; - u64 i; - + __set_memory_rox(_stext, _etext); + __set_memory_ro(_etext, __end_rodata); + __set_memory_rox(_sinittext, _einittext); + __set_memory_rox(__stext_amode31, __etext_amode31); /* - * Set RW+NX attribute on all memory, except regions enumerated with - * memory_rwx exclude type. These regions need different attributes, - * which are enforced afterwards. - * - * __for_each_mem_range() iterate and exclude types should be sorted. - * The relative location of _stext and _sinittext is hardcoded in the - * linker script. However a location of __stext_amode31 and the kernel - * image itself are chosen dynamically. Thus, sort the exclude type. + * If the BEAR-enhancement facility is not installed the first + * prefix page is used to return to the previous context with + * an LPSWE instruction and therefore must be executable. */ - sort(&memory_rwx_regions, - ARRAY_SIZE(memory_rwx_regions), sizeof(memory_rwx_regions[0]), - memblock_region_cmp, memblock_region_swap); - __for_each_mem_range(i, &memblock.memory, &memory_rwx, - NUMA_NO_NODE, MEMBLOCK_NONE, &base, &end, NULL) { - set_memory_rwnx((unsigned long)__va(base), - (end - base) >> PAGE_SHIFT); + if (!static_key_enabled(&cpu_has_bear)) + set_memory_x(0, 1); + if (debug_pagealloc_enabled()) { + /* + * Use RELOC_HIDE() as long as __va(0) translates to NULL, + * since performing pointer arithmetic on a NULL pointer + * has undefined behavior and generates compiler warnings. + */ + __set_memory_4k(__va(0), RELOC_HIDE(__va(0), ident_map_size)); } - -#ifdef CONFIG_KASAN - for_each_mem_range(i, &base, &end) - set_memory_kasan(base, end); -#endif - set_memory_rox((unsigned long)_stext, - (unsigned long)(_etext - _stext) >> PAGE_SHIFT); - set_memory_ro((unsigned long)_etext, - (unsigned long)(__end_rodata - _etext) >> PAGE_SHIFT); - set_memory_rox((unsigned long)_sinittext, - (unsigned long)(_einittext - _sinittext) >> PAGE_SHIFT); - set_memory_rox(__stext_amode31, - (__etext_amode31 - __stext_amode31) >> PAGE_SHIFT); - - /* lowcore must be executable for LPSWE */ - if (static_key_enabled(&cpu_has_bear)) - set_memory_nx(0, 1); - set_memory_nx(PAGE_SIZE, 1); - if (debug_pagealloc_enabled()) - set_memory_4k(0, ident_map_size >> PAGE_SHIFT); - + if (MACHINE_HAS_NX) + ctl_set_bit(0, 20); pr_info("Write protected kernel read-only data: %luk\n", (unsigned long)(__end_rodata - _stext) >> 10); } diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index 5e9371fbf3d5..de2fb12120d2 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -2088,6 +2088,7 @@ struct bpf_tramp_jit { */ int r14_off; /* Offset of saved %r14 */ int run_ctx_off; /* Offset of struct bpf_tramp_run_ctx */ + int tccnt_off; /* Offset of saved tailcall counter */ int do_fexit; /* do_fexit: label */ }; @@ -2258,12 +2259,16 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, tjit->r14_off = alloc_stack(tjit, sizeof(u64)); tjit->run_ctx_off = alloc_stack(tjit, sizeof(struct bpf_tramp_run_ctx)); + tjit->tccnt_off = alloc_stack(tjit, sizeof(u64)); /* The caller has already reserved STACK_FRAME_OVERHEAD bytes. */ tjit->stack_size -= STACK_FRAME_OVERHEAD; tjit->orig_stack_args_off = tjit->stack_size + STACK_FRAME_OVERHEAD; /* aghi %r15,-stack_size */ EMIT4_IMM(0xa70b0000, REG_15, -tjit->stack_size); + /* mvc tccnt_off(4,%r15),stack_size+STK_OFF_TCCNT(%r15) */ + _EMIT6(0xd203f000 | tjit->tccnt_off, + 0xf000 | (tjit->stack_size + STK_OFF_TCCNT)); /* stmg %r2,%rN,fwd_reg_args_off(%r15) */ if (nr_reg_args) EMIT6_DISP_LH(0xeb000000, 0x0024, REG_2, @@ -2400,6 +2405,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, (nr_stack_args * sizeof(u64) - 1) << 16 | tjit->stack_args_off, 0xf000 | tjit->orig_stack_args_off); + /* mvc STK_OFF_TCCNT(4,%r15),tccnt_off(%r15) */ + _EMIT6(0xd203f000 | STK_OFF_TCCNT, 0xf000 | tjit->tccnt_off); /* lgr %r1,%r8 */ EMIT4(0xb9040000, REG_1, REG_8); /* %r1() */ @@ -2456,6 +2463,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, if (flags & (BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_RET_FENTRY_RET)) EMIT6_DISP_LH(0xe3000000, 0x0004, REG_2, REG_0, REG_15, tjit->retval_off); + /* mvc stack_size+STK_OFF_TCCNT(4,%r15),tccnt_off(%r15) */ + _EMIT6(0xd203f000 | (tjit->stack_size + STK_OFF_TCCNT), + 0xf000 | tjit->tccnt_off); /* aghi %r15,stack_size */ EMIT4_IMM(0xa70b0000, REG_15, tjit->stack_size); /* Emit an expoline for the following indirect jump. */ diff --git a/arch/sh/boards/mach-ap325rxa/setup.c b/arch/sh/boards/mach-ap325rxa/setup.c index 151792162152..645cccf3da88 100644 --- a/arch/sh/boards/mach-ap325rxa/setup.c +++ b/arch/sh/boards/mach-ap325rxa/setup.c @@ -531,7 +531,7 @@ static int __init ap325rxa_devices_setup(void) device_initialize(&ap325rxa_ceu_device.dev); dma_declare_coherent_memory(&ap325rxa_ceu_device.dev, ceu_dma_membase, ceu_dma_membase, - ceu_dma_membase + CEU_BUFFER_MEMORY_SIZE - 1); + CEU_BUFFER_MEMORY_SIZE); platform_device_add(&ap325rxa_ceu_device); diff --git a/arch/sh/boards/mach-ecovec24/setup.c b/arch/sh/boards/mach-ecovec24/setup.c index 310513646c9b..3be293335de5 100644 --- a/arch/sh/boards/mach-ecovec24/setup.c +++ b/arch/sh/boards/mach-ecovec24/setup.c @@ -1454,15 +1454,13 @@ static int __init arch_setup(void) device_initialize(&ecovec_ceu_devices[0]->dev); dma_declare_coherent_memory(&ecovec_ceu_devices[0]->dev, ceu0_dma_membase, ceu0_dma_membase, - ceu0_dma_membase + - CEU_BUFFER_MEMORY_SIZE - 1); + CEU_BUFFER_MEMORY_SIZE); platform_device_add(ecovec_ceu_devices[0]); device_initialize(&ecovec_ceu_devices[1]->dev); dma_declare_coherent_memory(&ecovec_ceu_devices[1]->dev, ceu1_dma_membase, ceu1_dma_membase, - ceu1_dma_membase + - CEU_BUFFER_MEMORY_SIZE - 1); + CEU_BUFFER_MEMORY_SIZE); platform_device_add(ecovec_ceu_devices[1]); gpiod_add_lookup_table(&cn12_power_gpiod_table); diff --git a/arch/sh/boards/mach-kfr2r09/setup.c b/arch/sh/boards/mach-kfr2r09/setup.c index a18e80394aed..6b775eae85c0 100644 --- a/arch/sh/boards/mach-kfr2r09/setup.c +++ b/arch/sh/boards/mach-kfr2r09/setup.c @@ -603,7 +603,7 @@ static int __init kfr2r09_devices_setup(void) device_initialize(&kfr2r09_ceu_device.dev); dma_declare_coherent_memory(&kfr2r09_ceu_device.dev, ceu_dma_membase, ceu_dma_membase, - ceu_dma_membase + CEU_BUFFER_MEMORY_SIZE - 1); + CEU_BUFFER_MEMORY_SIZE); platform_device_add(&kfr2r09_ceu_device); diff --git a/arch/sh/boards/mach-migor/setup.c b/arch/sh/boards/mach-migor/setup.c index f60061283c48..773ee767d0c4 100644 --- a/arch/sh/boards/mach-migor/setup.c +++ b/arch/sh/boards/mach-migor/setup.c @@ -604,7 +604,7 @@ static int __init migor_devices_setup(void) device_initialize(&migor_ceu_device.dev); dma_declare_coherent_memory(&migor_ceu_device.dev, ceu_dma_membase, ceu_dma_membase, - ceu_dma_membase + CEU_BUFFER_MEMORY_SIZE - 1); + CEU_BUFFER_MEMORY_SIZE); platform_device_add(&migor_ceu_device); diff --git a/arch/sh/boards/mach-se/7724/setup.c b/arch/sh/boards/mach-se/7724/setup.c index b60a2626e18b..6495f9354065 100644 --- a/arch/sh/boards/mach-se/7724/setup.c +++ b/arch/sh/boards/mach-se/7724/setup.c @@ -940,15 +940,13 @@ static int __init devices_setup(void) device_initialize(&ms7724se_ceu_devices[0]->dev); dma_declare_coherent_memory(&ms7724se_ceu_devices[0]->dev, ceu0_dma_membase, ceu0_dma_membase, - ceu0_dma_membase + - CEU_BUFFER_MEMORY_SIZE - 1); + CEU_BUFFER_MEMORY_SIZE); platform_device_add(ms7724se_ceu_devices[0]); device_initialize(&ms7724se_ceu_devices[1]->dev); dma_declare_coherent_memory(&ms7724se_ceu_devices[1]->dev, ceu1_dma_membase, ceu1_dma_membase, - ceu1_dma_membase + - CEU_BUFFER_MEMORY_SIZE - 1); + CEU_BUFFER_MEMORY_SIZE); platform_device_add(ms7724se_ceu_devices[1]); return platform_add_devices(ms7724se_devices, diff --git a/arch/sh/drivers/push-switch.c b/arch/sh/drivers/push-switch.c index c95f48ff3f6f..6ecba5f521eb 100644 --- a/arch/sh/drivers/push-switch.c +++ b/arch/sh/drivers/push-switch.c @@ -101,8 +101,8 @@ static int switch_drv_remove(struct platform_device *pdev) device_remove_file(&pdev->dev, &dev_attr_switch); platform_set_drvdata(pdev, NULL); - flush_work(&psw->work); timer_shutdown_sync(&psw->debounce); + flush_work(&psw->work); free_irq(irq, pdev); kfree(psw); diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index 4d349986f76a..8250f0f59c2b 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -6474,8 +6474,18 @@ void spr_uncore_cpu_init(void) type = uncore_find_type_by_id(uncore_msr_uncores, UNCORE_SPR_CHA); if (type) { + /* + * The value from the discovery table (stored in the type->num_boxes + * of UNCORE_SPR_CHA) is incorrect on some SPR variants because of a + * firmware bug. Using the value from SPR_MSR_UNC_CBO_CONFIG to replace it. + */ rdmsrl(SPR_MSR_UNC_CBO_CONFIG, num_cbo); - type->num_boxes = num_cbo; + /* + * The MSR doesn't work on the EMR XCC, but the firmware bug doesn't impact + * the EMR XCC. Don't let the value from the MSR replace the existing value. + */ + if (num_cbo) + type->num_boxes = num_cbo; } spr_uncore_iio_free_running.num_boxes = uncore_type_max_boxes(uncore_msr_uncores, UNCORE_SPR_IIO); } diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 2061ed1c398f..58cb9495e40f 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -439,6 +439,7 @@ #define X86_FEATURE_SEV_ES (19*32+ 3) /* AMD Secure Encrypted Virtualization - Encrypted State */ #define X86_FEATURE_V_TSC_AUX (19*32+ 9) /* "" Virtual TSC_AUX */ #define X86_FEATURE_SME_COHERENT (19*32+10) /* "" AMD hardware-enforced cache coherency */ +#define X86_FEATURE_DEBUG_SWAP (19*32+14) /* AMD SEV-ES full debug state swap support */ /* AMD-defined Extended Feature 2 EAX, CPUID level 0x80000021 (EAX), word 20 */ #define X86_FEATURE_NO_NESTED_DATA_BP (20*32+ 0) /* "" No Nested Data Breakpoints */ diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h index 3be6a98751f0..c9f6a6c5de3c 100644 --- a/arch/x86/include/asm/kexec.h +++ b/arch/x86/include/asm/kexec.h @@ -205,8 +205,6 @@ int arch_kimage_file_post_load_cleanup(struct kimage *image); #endif #endif -typedef void crash_vmclear_fn(void); -extern crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss; extern void kdump_nmi_shootdown_cpus(void); #ifdef CONFIG_CRASH_HOTPLUG diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 3bc146dfd38d..1a4def36d5bb 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -288,13 +288,13 @@ struct kvm_kernel_irq_routing_entry; * kvm_mmu_page_role tracks the properties of a shadow page (where shadow page * also includes TDP pages) to determine whether or not a page can be used in * the given MMU context. This is a subset of the overall kvm_cpu_role to - * minimize the size of kvm_memory_slot.arch.gfn_track, i.e. allows allocating - * 2 bytes per gfn instead of 4 bytes per gfn. + * minimize the size of kvm_memory_slot.arch.gfn_write_track, i.e. allows + * allocating 2 bytes per gfn instead of 4 bytes per gfn. * * Upper-level shadow pages having gptes are tracked for write-protection via - * gfn_track. As above, gfn_track is a 16 bit counter, so KVM must not create - * more than 2^16-1 upper-level shadow pages at a single gfn, otherwise - * gfn_track will overflow and explosions will ensure. + * gfn_write_track. As above, gfn_write_track is a 16 bit counter, so KVM must + * not create more than 2^16-1 upper-level shadow pages at a single gfn, + * otherwise gfn_write_track will overflow and explosions will ensue. * * A unique shadow page (SP) for a gfn is created if and only if an existing SP * cannot be reused. The ability to reuse a SP is tracked by its role, which @@ -746,7 +746,6 @@ struct kvm_vcpu_arch { u64 smi_count; bool at_instruction_boundary; bool tpr_access_reporting; - bool xsaves_enabled; bool xfd_no_write_intercept; u64 ia32_xss; u64 microcode_version; @@ -831,6 +830,25 @@ struct kvm_vcpu_arch { struct kvm_cpuid_entry2 *cpuid_entries; struct kvm_hypervisor_cpuid kvm_cpuid; + /* + * FIXME: Drop this macro and use KVM_NR_GOVERNED_FEATURES directly + * when "struct kvm_vcpu_arch" is no longer defined in an + * arch/x86/include/asm header. The max is mostly arbitrary, i.e. + * can be increased as necessary. + */ +#define KVM_MAX_NR_GOVERNED_FEATURES BITS_PER_LONG + + /* + * Track whether or not the guest is allowed to use features that are + * governed by KVM, where "governed" means KVM needs to manage state + * and/or explicitly enable the feature in hardware. Typically, but + * not always, governed features can be used by the guest if and only + * if both KVM and userspace want to expose the feature to the guest. + */ + struct { + DECLARE_BITMAP(enabled, KVM_MAX_NR_GOVERNED_FEATURES); + } governed_features; + u64 reserved_gpa_bits; int maxphyaddr; @@ -1005,7 +1023,7 @@ struct kvm_lpage_info { struct kvm_arch_memory_slot { struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES]; struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES - 1]; - unsigned short *gfn_track[KVM_PAGE_TRACK_MAX]; + unsigned short *gfn_write_track; }; /* @@ -1247,8 +1265,9 @@ struct kvm_arch { * create an NX huge page (without hanging the guest). */ struct list_head possible_nx_huge_pages; - struct kvm_page_track_notifier_node mmu_sp_tracker; +#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING struct kvm_page_track_notifier_head track_notifier_head; +#endif /* * Protects marking pages unsync during page faults, as TDP MMU page * faults only take mmu_lock for read. For simplicity, the unsync @@ -1655,8 +1674,8 @@ struct kvm_x86_ops { u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu); u64 (*get_l2_tsc_multiplier)(struct kvm_vcpu *vcpu); - void (*write_tsc_offset)(struct kvm_vcpu *vcpu, u64 offset); - void (*write_tsc_multiplier)(struct kvm_vcpu *vcpu, u64 multiplier); + void (*write_tsc_offset)(struct kvm_vcpu *vcpu); + void (*write_tsc_multiplier)(struct kvm_vcpu *vcpu); /* * Retrieve somewhat arbitrary exit information. Intended to @@ -1795,8 +1814,8 @@ static inline struct kvm *kvm_arch_alloc_vm(void) #define __KVM_HAVE_ARCH_VM_FREE void kvm_arch_free_vm(struct kvm *kvm); -#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB -static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm) +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS +static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { if (kvm_x86_ops.flush_remote_tlbs && !static_call(kvm_x86_flush_remote_tlbs)(kvm)) @@ -1805,6 +1824,8 @@ static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm) return -ENOTSUPP; } +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE + #define kvm_arch_pmi_in_guest(vcpu) \ ((vcpu) && (vcpu)->arch.handling_intr_from_guest) @@ -1833,7 +1854,6 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *memslot); void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, const struct kvm_memory_slot *memslot); -void kvm_mmu_zap_all(struct kvm *kvm); void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen); void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages); diff --git a/arch/x86/include/asm/kvm_page_track.h b/arch/x86/include/asm/kvm_page_track.h index eb186bc57f6a..3d040741044b 100644 --- a/arch/x86/include/asm/kvm_page_track.h +++ b/arch/x86/include/asm/kvm_page_track.h @@ -2,11 +2,9 @@ #ifndef _ASM_X86_KVM_PAGE_TRACK_H #define _ASM_X86_KVM_PAGE_TRACK_H -enum kvm_page_track_mode { - KVM_PAGE_TRACK_WRITE, - KVM_PAGE_TRACK_MAX, -}; +#include +#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING /* * The notifier represented by @kvm_page_track_notifier_node is linked into * the head which will be notified when guest is triggering the track event. @@ -26,54 +24,39 @@ struct kvm_page_track_notifier_node { * It is called when guest is writing the write-tracked page * and write emulation is finished at that time. * - * @vcpu: the vcpu where the write access happened. * @gpa: the physical address written by guest. * @new: the data was written to the address. * @bytes: the written length. * @node: this node */ - void (*track_write)(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, - int bytes, struct kvm_page_track_notifier_node *node); - /* - * It is called when memory slot is being moved or removed - * users can drop write-protection for the pages in that memory slot - * - * @kvm: the kvm where memory slot being moved or removed - * @slot: the memory slot being moved or removed - * @node: this node - */ - void (*track_flush_slot)(struct kvm *kvm, struct kvm_memory_slot *slot, + void (*track_write)(gpa_t gpa, const u8 *new, int bytes, struct kvm_page_track_notifier_node *node); + + /* + * Invoked when a memory region is removed from the guest. Or in KVM + * terms, when a memslot is deleted. + * + * @gfn: base gfn of the region being removed + * @nr_pages: number of pages in the to-be-removed region + * @node: this node + */ + void (*track_remove_region)(gfn_t gfn, unsigned long nr_pages, + struct kvm_page_track_notifier_node *node); }; -int kvm_page_track_init(struct kvm *kvm); -void kvm_page_track_cleanup(struct kvm *kvm); +int kvm_page_track_register_notifier(struct kvm *kvm, + struct kvm_page_track_notifier_node *n); +void kvm_page_track_unregister_notifier(struct kvm *kvm, + struct kvm_page_track_notifier_node *n); -bool kvm_page_track_write_tracking_enabled(struct kvm *kvm); -int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot); +int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn); +int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn); +#else +/* + * Allow defining a node in a structure even if page tracking is disabled, e.g. + * to play nice with testing headers via direct inclusion from the command line. + */ +struct kvm_page_track_notifier_node {}; +#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */ -void kvm_page_track_free_memslot(struct kvm_memory_slot *slot); -int kvm_page_track_create_memslot(struct kvm *kvm, - struct kvm_memory_slot *slot, - unsigned long npages); - -void kvm_slot_page_track_add_page(struct kvm *kvm, - struct kvm_memory_slot *slot, gfn_t gfn, - enum kvm_page_track_mode mode); -void kvm_slot_page_track_remove_page(struct kvm *kvm, - struct kvm_memory_slot *slot, gfn_t gfn, - enum kvm_page_track_mode mode); -bool kvm_slot_page_track_is_active(struct kvm *kvm, - const struct kvm_memory_slot *slot, - gfn_t gfn, enum kvm_page_track_mode mode); - -void -kvm_page_track_register_notifier(struct kvm *kvm, - struct kvm_page_track_notifier_node *n); -void -kvm_page_track_unregister_notifier(struct kvm *kvm, - struct kvm_page_track_notifier_node *n); -void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, - int bytes); -void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot); #endif diff --git a/arch/x86/include/asm/mman.h b/arch/x86/include/asm/mman.h new file mode 100644 index 000000000000..12b820259b9f --- /dev/null +++ b/arch/x86/include/asm/mman.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_MMAN_H__ +#define __ASM_MMAN_H__ + +#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS +#define arch_calc_vm_prot_bits(prot, key) ( \ + ((key) & 0x1 ? VM_PKEY_BIT0 : 0) | \ + ((key) & 0x2 ? VM_PKEY_BIT1 : 0) | \ + ((key) & 0x4 ? VM_PKEY_BIT2 : 0) | \ + ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) +#endif + +#include + +#endif /* __ASM_MMAN_H__ */ diff --git a/arch/x86/include/asm/reboot.h b/arch/x86/include/asm/reboot.h index 9177b4354c3f..6536873f8fc0 100644 --- a/arch/x86/include/asm/reboot.h +++ b/arch/x86/include/asm/reboot.h @@ -25,7 +25,14 @@ void __noreturn machine_real_restart(unsigned int type); #define MRR_BIOS 0 #define MRR_APM 1 +#if IS_ENABLED(CONFIG_KVM_INTEL) || IS_ENABLED(CONFIG_KVM_AMD) +typedef void (cpu_emergency_virt_cb)(void); +void cpu_emergency_register_virt_callback(cpu_emergency_virt_cb *callback); +void cpu_emergency_unregister_virt_callback(cpu_emergency_virt_cb *callback); void cpu_emergency_disable_virtualization(void); +#else +static inline void cpu_emergency_disable_virtualization(void) {} +#endif /* CONFIG_KVM_INTEL || CONFIG_KVM_AMD */ typedef void (*nmi_shootdown_cb)(int, struct pt_regs*); void nmi_shootdown_cpus(nmi_shootdown_cb callback); diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h index e7c7379d6ac7..19bf955b67e0 100644 --- a/arch/x86/include/asm/svm.h +++ b/arch/x86/include/asm/svm.h @@ -288,6 +288,7 @@ static_assert((X2AVIC_MAX_PHYSICAL_ID & AVIC_PHYSICAL_MAX_INDEX_MASK) == X2AVIC_ #define AVIC_HPA_MASK ~((0xFFFULL << 52) | 0xFFF) +#define SVM_SEV_FEAT_DEBUG_SWAP BIT(5) struct vmcb_seg { u16 selector; @@ -345,7 +346,7 @@ struct vmcb_save_area { u64 last_excp_from; u64 last_excp_to; u8 reserved_0x298[72]; - u32 spec_ctrl; /* Guest version of SPEC_CTRL at 0x2E0 */ + u64 spec_ctrl; /* Guest version of SPEC_CTRL at 0x2E0 */ } __packed; /* Save area definition for SEV-ES and SEV-SNP guests */ @@ -512,7 +513,7 @@ struct ghcb { } __packed; -#define EXPECTED_VMCB_SAVE_AREA_SIZE 740 +#define EXPECTED_VMCB_SAVE_AREA_SIZE 744 #define EXPECTED_GHCB_SAVE_AREA_SIZE 1032 #define EXPECTED_SEV_ES_SAVE_AREA_SIZE 1648 #define EXPECTED_VMCB_CONTROL_AREA_SIZE 1024 diff --git a/arch/x86/include/asm/virtext.h b/arch/x86/include/asm/virtext.h deleted file mode 100644 index 3b12e6b99412..000000000000 --- a/arch/x86/include/asm/virtext.h +++ /dev/null @@ -1,154 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* CPU virtualization extensions handling - * - * This should carry the code for handling CPU virtualization extensions - * that needs to live in the kernel core. - * - * Author: Eduardo Habkost - * - * Copyright (C) 2008, Red Hat Inc. - * - * Contains code from KVM, Copyright (C) 2006 Qumranet, Inc. - */ -#ifndef _ASM_X86_VIRTEX_H -#define _ASM_X86_VIRTEX_H - -#include - -#include -#include -#include - -/* - * VMX functions: - */ - -static inline int cpu_has_vmx(void) -{ - unsigned long ecx = cpuid_ecx(1); - return test_bit(5, &ecx); /* CPUID.1:ECX.VMX[bit 5] -> VT */ -} - - -/** - * cpu_vmxoff() - Disable VMX on the current CPU - * - * Disable VMX and clear CR4.VMXE (even if VMXOFF faults) - * - * Note, VMXOFF causes a #UD if the CPU is !post-VMXON, but it's impossible to - * atomically track post-VMXON state, e.g. this may be called in NMI context. - * Eat all faults as all other faults on VMXOFF faults are mode related, i.e. - * faults are guaranteed to be due to the !post-VMXON check unless the CPU is - * magically in RM, VM86, compat mode, or at CPL>0. - */ -static inline int cpu_vmxoff(void) -{ - asm_volatile_goto("1: vmxoff\n\t" - _ASM_EXTABLE(1b, %l[fault]) - ::: "cc", "memory" : fault); - - cr4_clear_bits(X86_CR4_VMXE); - return 0; - -fault: - cr4_clear_bits(X86_CR4_VMXE); - return -EIO; -} - -static inline int cpu_vmx_enabled(void) -{ - return __read_cr4() & X86_CR4_VMXE; -} - -/** Disable VMX if it is enabled on the current CPU - * - * You shouldn't call this if cpu_has_vmx() returns 0. - */ -static inline void __cpu_emergency_vmxoff(void) -{ - if (cpu_vmx_enabled()) - cpu_vmxoff(); -} - -/** Disable VMX if it is supported and enabled on the current CPU - */ -static inline void cpu_emergency_vmxoff(void) -{ - if (cpu_has_vmx()) - __cpu_emergency_vmxoff(); -} - - - - -/* - * SVM functions: - */ - -/** Check if the CPU has SVM support - * - * You can use the 'msg' arg to get a message describing the problem, - * if the function returns zero. Simply pass NULL if you are not interested - * on the messages; gcc should take care of not generating code for - * the messages on this case. - */ -static inline int cpu_has_svm(const char **msg) -{ - if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD && - boot_cpu_data.x86_vendor != X86_VENDOR_HYGON) { - if (msg) - *msg = "not amd or hygon"; - return 0; - } - - if (boot_cpu_data.extended_cpuid_level < SVM_CPUID_FUNC) { - if (msg) - *msg = "can't execute cpuid_8000000a"; - return 0; - } - - if (!boot_cpu_has(X86_FEATURE_SVM)) { - if (msg) - *msg = "svm not available"; - return 0; - } - return 1; -} - - -/** Disable SVM on the current CPU - * - * You should call this only if cpu_has_svm() returned true. - */ -static inline void cpu_svm_disable(void) -{ - uint64_t efer; - - wrmsrl(MSR_VM_HSAVE_PA, 0); - rdmsrl(MSR_EFER, efer); - if (efer & EFER_SVME) { - /* - * Force GIF=1 prior to disabling SVM to ensure INIT and NMI - * aren't blocked, e.g. if a fatal error occurred between CLGI - * and STGI. Note, STGI may #UD if SVM is disabled from NMI - * context between reading EFER and executing STGI. In that - * case, GIF must already be set, otherwise the NMI would have - * been blocked, so just eat the fault. - */ - asm_volatile_goto("1: stgi\n\t" - _ASM_EXTABLE(1b, %l[fault]) - ::: "memory" : fault); -fault: - wrmsrl(MSR_EFER, efer & ~EFER_SVME); - } -} - -/** Makes sure SVM is disabled, if it is supported on the CPU - */ -static inline void cpu_emergency_svm_disable(void) -{ - if (cpu_has_svm(NULL)) - cpu_svm_disable(); -} - -#endif /* _ASM_X86_VIRTEX_H */ diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index 0d02c4aafa6f..0e73616b82f3 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -71,7 +71,7 @@ #define SECONDARY_EXEC_RDSEED_EXITING VMCS_CONTROL_BIT(RDSEED_EXITING) #define SECONDARY_EXEC_ENABLE_PML VMCS_CONTROL_BIT(PAGE_MOD_LOGGING) #define SECONDARY_EXEC_PT_CONCEAL_VMX VMCS_CONTROL_BIT(PT_CONCEAL_VMX) -#define SECONDARY_EXEC_XSAVES VMCS_CONTROL_BIT(XSAVES) +#define SECONDARY_EXEC_ENABLE_XSAVES VMCS_CONTROL_BIT(XSAVES) #define SECONDARY_EXEC_MODE_BASED_EPT_EXEC VMCS_CONTROL_BIT(MODE_BASED_EPT_EXEC) #define SECONDARY_EXEC_PT_USE_GPA VMCS_CONTROL_BIT(PT_USE_GPA) #define SECONDARY_EXEC_TSC_SCALING VMCS_CONTROL_BIT(TSC_SCALING) diff --git a/arch/x86/include/uapi/asm/mman.h b/arch/x86/include/uapi/asm/mman.h index 8148bdddbd2c..46cdc941f958 100644 --- a/arch/x86/include/uapi/asm/mman.h +++ b/arch/x86/include/uapi/asm/mman.h @@ -5,14 +5,6 @@ #define MAP_32BIT 0x40 /* only give out 32bit addresses */ #define MAP_ABOVE4G 0x80 /* only map above 4GB */ -#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS -#define arch_calc_vm_prot_bits(prot, key) ( \ - ((key) & 0x1 ? VM_PKEY_BIT0 : 0) | \ - ((key) & 0x2 ? VM_PKEY_BIT1 : 0) | \ - ((key) & 0x4 ? VM_PKEY_BIT2 : 0) | \ - ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) -#endif - /* Flags for map_shadow_stack(2) */ #define SHADOW_STACK_SET_TOKEN (1ULL << 0) /* Set up a restore token in the shadow stack */ diff --git a/arch/x86/kernel/cpu/sgx/virt.c b/arch/x86/kernel/cpu/sgx/virt.c index c3e37eaec8ec..7aaa3652e31d 100644 --- a/arch/x86/kernel/cpu/sgx/virt.c +++ b/arch/x86/kernel/cpu/sgx/virt.c @@ -204,6 +204,7 @@ static int sgx_vepc_release(struct inode *inode, struct file *file) continue; xa_erase(&vepc->page_array, index); + cond_resched(); } /* @@ -222,6 +223,7 @@ static int sgx_vepc_release(struct inode *inode, struct file *file) list_add_tail(&epc_page->list, &secs_pages); xa_erase(&vepc->page_array, index); + cond_resched(); } /* @@ -243,6 +245,7 @@ static int sgx_vepc_release(struct inode *inode, struct file *file) if (sgx_vepc_free_page(epc_page)) list_add_tail(&epc_page->list, &secs_pages); + cond_resched(); } if (!list_empty(&secs_pages)) diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c index 587c7743fd21..c92d88680dbf 100644 --- a/arch/x86/kernel/crash.c +++ b/arch/x86/kernel/crash.c @@ -48,38 +48,12 @@ struct crash_memmap_data { unsigned int type; }; -/* - * This is used to VMCLEAR all VMCSs loaded on the - * processor. And when loading kvm_intel module, the - * callback function pointer will be assigned. - * - * protected by rcu. - */ -crash_vmclear_fn __rcu *crash_vmclear_loaded_vmcss = NULL; -EXPORT_SYMBOL_GPL(crash_vmclear_loaded_vmcss); - -static inline void cpu_crash_vmclear_loaded_vmcss(void) -{ - crash_vmclear_fn *do_vmclear_operation = NULL; - - rcu_read_lock(); - do_vmclear_operation = rcu_dereference(crash_vmclear_loaded_vmcss); - if (do_vmclear_operation) - do_vmclear_operation(); - rcu_read_unlock(); -} - #if defined(CONFIG_SMP) && defined(CONFIG_X86_LOCAL_APIC) static void kdump_nmi_callback(int cpu, struct pt_regs *regs) { crash_save_cpu(regs, cpu); - /* - * VMCLEAR VMCSs loaded on all cpus if needed. - */ - cpu_crash_vmclear_loaded_vmcss(); - /* * Disable Intel PT to stop its logging */ @@ -133,11 +107,6 @@ void native_machine_crash_shutdown(struct pt_regs *regs) crash_smp_send_stop(); - /* - * VMCLEAR VMCSs loaded on this cpu if needed. - */ - cpu_crash_vmclear_loaded_vmcss(); - cpu_emergency_disable_virtualization(); /* diff --git a/arch/x86/kernel/reboot.c b/arch/x86/kernel/reboot.c index 3adbe97015c1..830425e6d38e 100644 --- a/arch/x86/kernel/reboot.c +++ b/arch/x86/kernel/reboot.c @@ -22,7 +22,6 @@ #include #include #include -#include #include #include #include @@ -530,9 +529,54 @@ static inline void kb_wait(void) static inline void nmi_shootdown_cpus_on_restart(void); +#if IS_ENABLED(CONFIG_KVM_INTEL) || IS_ENABLED(CONFIG_KVM_AMD) +/* RCU-protected callback to disable virtualization prior to reboot. */ +static cpu_emergency_virt_cb __rcu *cpu_emergency_virt_callback; + +void cpu_emergency_register_virt_callback(cpu_emergency_virt_cb *callback) +{ + if (WARN_ON_ONCE(rcu_access_pointer(cpu_emergency_virt_callback))) + return; + + rcu_assign_pointer(cpu_emergency_virt_callback, callback); +} +EXPORT_SYMBOL_GPL(cpu_emergency_register_virt_callback); + +void cpu_emergency_unregister_virt_callback(cpu_emergency_virt_cb *callback) +{ + if (WARN_ON_ONCE(rcu_access_pointer(cpu_emergency_virt_callback) != callback)) + return; + + rcu_assign_pointer(cpu_emergency_virt_callback, NULL); + synchronize_rcu(); +} +EXPORT_SYMBOL_GPL(cpu_emergency_unregister_virt_callback); + +/* + * Disable virtualization, i.e. VMX or SVM, to ensure INIT is recognized during + * reboot. VMX blocks INIT if the CPU is post-VMXON, and SVM blocks INIT if + * GIF=0, i.e. if the crash occurred between CLGI and STGI. + */ +void cpu_emergency_disable_virtualization(void) +{ + cpu_emergency_virt_cb *callback; + + /* + * IRQs must be disabled as KVM enables virtualization in hardware via + * function call IPIs, i.e. IRQs need to be disabled to guarantee + * virtualization stays disabled. + */ + lockdep_assert_irqs_disabled(); + + rcu_read_lock(); + callback = rcu_dereference(cpu_emergency_virt_callback); + if (callback) + callback(); + rcu_read_unlock(); +} + static void emergency_reboot_disable_virtualization(void) { - /* Just make sure we won't change CPUs while doing this */ local_irq_disable(); /* @@ -545,7 +589,7 @@ static void emergency_reboot_disable_virtualization(void) * Do the NMI shootdown even if virtualization is off on _this_ CPU, as * other CPUs may have virtualization enabled. */ - if (cpu_has_vmx() || cpu_has_svm(NULL)) { + if (rcu_access_pointer(cpu_emergency_virt_callback)) { /* Safely force _this_ CPU out of VMX/SVM operation. */ cpu_emergency_disable_virtualization(); @@ -553,7 +597,9 @@ static void emergency_reboot_disable_virtualization(void) nmi_shootdown_cpus_on_restart(); } } - +#else +static void emergency_reboot_disable_virtualization(void) { } +#endif /* CONFIG_KVM_INTEL || CONFIG_KVM_AMD */ void __attribute__((weak)) mach_reboot_fixups(void) { @@ -787,21 +833,9 @@ void machine_crash_shutdown(struct pt_regs *regs) } #endif - /* This is the CPU performing the emergency shutdown work. */ int crashing_cpu = -1; -/* - * Disable virtualization, i.e. VMX or SVM, to ensure INIT is recognized during - * reboot. VMX blocks INIT if the CPU is post-VMXON, and SVM blocks INIT if - * GIF=0, i.e. if the crash occurred between CLGI and STGI. - */ -void cpu_emergency_disable_virtualization(void) -{ - cpu_emergency_vmxoff(); - cpu_emergency_svm_disable(); -} - #if defined(CONFIG_SMP) static nmi_shootdown_cb shootdown_callback; diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index d7667a29acf3..4e45ff44aa07 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1250,7 +1250,7 @@ bool smp_park_other_cpus_in_init(void) if (this_cpu) return false; - for_each_present_cpu(cpu) { + for_each_cpu_and(cpu, &cpus_booted_once_mask, cpu_present_mask) { if (cpu == this_cpu) continue; apicid = apic->cpu_present_to_apicid(cpu); diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 83d41c2601d7..f15fb71f280e 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -156,7 +156,7 @@ SECTIONS ALIGN_ENTRY_TEXT_END *(.gnu.warning) - } :text =0xcccc + } :text = 0xcccccccc /* End of text section, which should occupy whole number of pages */ _etext = .; diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index 89ca7f4c1464..ed90f148140d 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -101,7 +101,7 @@ config X86_SGX_KVM config KVM_AMD tristate "KVM for AMD processors support" - depends on KVM + depends on KVM && (CPU_SUP_AMD || CPU_SUP_HYGON) help Provides support for KVM on AMD processors equipped with the AMD-V (SVM) extensions. @@ -138,6 +138,19 @@ config KVM_XEN If in doubt, say "N". +config KVM_PROVE_MMU + bool "Prove KVM MMU correctness" + depends on DEBUG_KERNEL + depends on KVM + depends on EXPERT + help + Enables runtime assertions in KVM's MMU that are too costly to enable + in anything remotely resembling a production environment, e.g. this + gates code that verifies a to-be-freed page table doesn't have any + present SPTEs. + + If in doubt, say "N". + config KVM_EXTERNAL_WRITE_TRACKING bool diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index d3432687c9e6..0544e30b4946 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -11,6 +11,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include +#include "linux/lockdep.h" #include #include #include @@ -84,6 +85,18 @@ static inline struct kvm_cpuid_entry2 *cpuid_entry2_find( struct kvm_cpuid_entry2 *e; int i; + /* + * KVM has a semi-arbitrary rule that querying the guest's CPUID model + * with IRQs disabled is disallowed. The CPUID model can legitimately + * have over one hundred entries, i.e. the lookup is slow, and IRQs are + * typically disabled in KVM only when KVM is in a performance critical + * path, e.g. the core VM-Enter/VM-Exit run loop. Nothing will break + * if this rule is violated, this assertion is purely to flag potential + * performance issues. If this fires, consider moving the lookup out + * of the hotpath, e.g. by caching information during CPUID updates. + */ + lockdep_assert_irqs_enabled(); + for (i = 0; i < nent; i++) { e = &entries[i]; @@ -312,6 +325,27 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) { struct kvm_lapic *apic = vcpu->arch.apic; struct kvm_cpuid_entry2 *best; + bool allow_gbpages; + + BUILD_BUG_ON(KVM_NR_GOVERNED_FEATURES > KVM_MAX_NR_GOVERNED_FEATURES); + bitmap_zero(vcpu->arch.governed_features.enabled, + KVM_MAX_NR_GOVERNED_FEATURES); + + /* + * If TDP is enabled, let the guest use GBPAGES if they're supported in + * hardware. The hardware page walker doesn't let KVM disable GBPAGES, + * i.e. won't treat them as reserved, and KVM doesn't redo the GVA->GPA + * walk for performance and complexity reasons. Not to mention KVM + * _can't_ solve the problem because GVA->GPA walks aren't visible to + * KVM once a TDP translation is installed. Mimic hardware behavior so + * that KVM's is at least consistent, i.e. doesn't randomly inject #PF. + * If TDP is disabled, honor *only* guest CPUID as KVM has full control + * and can install smaller shadow pages if the host lacks 1GiB support. + */ + allow_gbpages = tdp_enabled ? boot_cpu_has(X86_FEATURE_GBPAGES) : + guest_cpuid_has(vcpu, X86_FEATURE_GBPAGES); + if (allow_gbpages) + kvm_governed_feature_set(vcpu, X86_FEATURE_GBPAGES); best = kvm_find_cpuid_entry(vcpu, 1); if (best && apic) { @@ -647,7 +681,8 @@ void kvm_set_cpu_caps(void) ); kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX, - F(AVX_VNNI_INT8) | F(AVX_NE_CONVERT) | F(PREFETCHITI) + F(AVX_VNNI_INT8) | F(AVX_NE_CONVERT) | F(PREFETCHITI) | + F(AMX_COMPLEX) ); kvm_cpu_cap_mask(CPUID_D_1_EAX, @@ -1154,6 +1189,9 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) cpuid_entry_override(entry, CPUID_8000_0001_EDX); cpuid_entry_override(entry, CPUID_8000_0001_ECX); break; + case 0x80000005: + /* Pass host L1 cache and TLB info. */ + break; case 0x80000006: /* Drop reserved bits, pass host L2 cache and TLB info. */ entry->edx &= ~GENMASK(17, 16); diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h index b1658c0de847..284fa4704553 100644 --- a/arch/x86/kvm/cpuid.h +++ b/arch/x86/kvm/cpuid.h @@ -232,4 +232,50 @@ static __always_inline bool guest_pv_has(struct kvm_vcpu *vcpu, return vcpu->arch.pv_cpuid.features & (1u << kvm_feature); } +enum kvm_governed_features { +#define KVM_GOVERNED_FEATURE(x) KVM_GOVERNED_##x, +#include "governed_features.h" + KVM_NR_GOVERNED_FEATURES +}; + +static __always_inline int kvm_governed_feature_index(unsigned int x86_feature) +{ + switch (x86_feature) { +#define KVM_GOVERNED_FEATURE(x) case x: return KVM_GOVERNED_##x; +#include "governed_features.h" + default: + return -1; + } +} + +static __always_inline bool kvm_is_governed_feature(unsigned int x86_feature) +{ + return kvm_governed_feature_index(x86_feature) >= 0; +} + +static __always_inline void kvm_governed_feature_set(struct kvm_vcpu *vcpu, + unsigned int x86_feature) +{ + BUILD_BUG_ON(!kvm_is_governed_feature(x86_feature)); + + __set_bit(kvm_governed_feature_index(x86_feature), + vcpu->arch.governed_features.enabled); +} + +static __always_inline void kvm_governed_feature_check_and_set(struct kvm_vcpu *vcpu, + unsigned int x86_feature) +{ + if (kvm_cpu_cap_has(x86_feature) && guest_cpuid_has(vcpu, x86_feature)) + kvm_governed_feature_set(vcpu, x86_feature); +} + +static __always_inline bool guest_can_use(struct kvm_vcpu *vcpu, + unsigned int x86_feature) +{ + BUILD_BUG_ON(!kvm_is_governed_feature(x86_feature)); + + return test_bit(kvm_governed_feature_index(x86_feature), + vcpu->arch.governed_features.enabled); +} + #endif diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index 936a397a08cd..2673cd5c46cb 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -1799,13 +1799,11 @@ static int writeback(struct x86_emulate_ctxt *ctxt, struct operand *op) op->addr.mem, &op->val, op->bytes); - break; case OP_MEM_STR: return segmented_write(ctxt, op->addr.mem, op->data, op->bytes * op->count); - break; case OP_XMM: kvm_write_sse_reg(op->addr.xmm, &op->vec_val); break; diff --git a/arch/x86/kvm/governed_features.h b/arch/x86/kvm/governed_features.h new file mode 100644 index 000000000000..423a73395c10 --- /dev/null +++ b/arch/x86/kvm/governed_features.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(KVM_GOVERNED_FEATURE) || defined(KVM_GOVERNED_X86_FEATURE) +BUILD_BUG() +#endif + +#define KVM_GOVERNED_X86_FEATURE(x) KVM_GOVERNED_FEATURE(X86_FEATURE_##x) + +KVM_GOVERNED_X86_FEATURE(GBPAGES) +KVM_GOVERNED_X86_FEATURE(XSAVES) +KVM_GOVERNED_X86_FEATURE(VMX) +KVM_GOVERNED_X86_FEATURE(NRIPS) +KVM_GOVERNED_X86_FEATURE(TSCRATEMSR) +KVM_GOVERNED_X86_FEATURE(V_VMSAVE_VMLOAD) +KVM_GOVERNED_X86_FEATURE(LBRV) +KVM_GOVERNED_X86_FEATURE(PAUSEFILTER) +KVM_GOVERNED_X86_FEATURE(PFTHRESHOLD) +KVM_GOVERNED_X86_FEATURE(VGIF) +KVM_GOVERNED_X86_FEATURE(VNMI) + +#undef KVM_GOVERNED_X86_FEATURE +#undef KVM_GOVERNED_FEATURE diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index b28fd020066f..7c2dac6824e2 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -1293,7 +1293,6 @@ static bool hv_check_msr_access(struct kvm_vcpu_hv *hv_vcpu, u32 msr) case HV_X64_MSR_VP_ASSIST_PAGE: return hv_vcpu->cpuid_cache.features_eax & HV_MSR_APIC_ACCESS_AVAILABLE; - break; case HV_X64_MSR_TSC_FREQUENCY: case HV_X64_MSR_APIC_FREQUENCY: return hv_vcpu->cpuid_cache.features_eax & diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h index ab65f3a47dfd..be7aeb9b8ea3 100644 --- a/arch/x86/kvm/kvm_emulate.h +++ b/arch/x86/kvm/kvm_emulate.h @@ -213,7 +213,6 @@ struct x86_emulate_ops { bool (*get_cpuid)(struct x86_emulate_ctxt *ctxt, u32 *eax, u32 *ebx, u32 *ecx, u32 *edx, bool exact_only); - bool (*guest_has_long_mode)(struct x86_emulate_ctxt *ctxt); bool (*guest_has_movbe)(struct x86_emulate_ctxt *ctxt); bool (*guest_has_fxsr)(struct x86_emulate_ctxt *ctxt); bool (*guest_has_rdpid)(struct x86_emulate_ctxt *ctxt); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index a983a16163b1..dcd60b39e794 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -376,7 +376,8 @@ void kvm_recalculate_apic_map(struct kvm *kvm) struct kvm_vcpu *vcpu; unsigned long i; u32 max_id = 255; /* enough space for any xAPIC ID */ - bool xapic_id_mismatch = false; + bool xapic_id_mismatch; + int r; /* Read kvm->arch.apic_map_dirty before kvm->arch.apic_map. */ if (atomic_read_acquire(&kvm->arch.apic_map_dirty) == CLEAN) @@ -386,9 +387,14 @@ void kvm_recalculate_apic_map(struct kvm *kvm) "Dirty APIC map without an in-kernel local APIC"); mutex_lock(&kvm->arch.apic_map_lock); + +retry: /* - * Read kvm->arch.apic_map_dirty before kvm->arch.apic_map - * (if clean) or the APIC registers (if dirty). + * Read kvm->arch.apic_map_dirty before kvm->arch.apic_map (if clean) + * or the APIC registers (if dirty). Note, on retry the map may have + * not yet been marked dirty by whatever task changed a vCPU's x2APIC + * ID, i.e. the map may still show up as in-progress. In that case + * this task still needs to retry and complete its calculation. */ if (atomic_cmpxchg_acquire(&kvm->arch.apic_map_dirty, DIRTY, UPDATE_IN_PROGRESS) == CLEAN) { @@ -397,6 +403,15 @@ void kvm_recalculate_apic_map(struct kvm *kvm) return; } + /* + * Reset the mismatch flag between attempts so that KVM does the right + * thing if a vCPU changes its xAPIC ID, but do NOT reset max_id, i.e. + * keep max_id strictly increasing. Disallowing max_id from shrinking + * ensures KVM won't get stuck in an infinite loop, e.g. if the vCPU + * with the highest x2APIC ID is toggling its APIC on and off. + */ + xapic_id_mismatch = false; + kvm_for_each_vcpu(i, vcpu, kvm) if (kvm_apic_present(vcpu)) max_id = max(max_id, kvm_x2apic_id(vcpu->arch.apic)); @@ -415,9 +430,15 @@ void kvm_recalculate_apic_map(struct kvm *kvm) if (!kvm_apic_present(vcpu)) continue; - if (kvm_recalculate_phys_map(new, vcpu, &xapic_id_mismatch)) { + r = kvm_recalculate_phys_map(new, vcpu, &xapic_id_mismatch); + if (r) { kvfree(new); new = NULL; + if (r == -E2BIG) { + cond_resched(); + goto retry; + } + goto out; } diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 92d5a1924fc1..253fb2093d5d 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -121,6 +121,8 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu); void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu); void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu); void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu); +void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, + int bytes); static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ec169f5c7dce..e1d011c67cc6 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -25,6 +25,7 @@ #include "kvm_cache_regs.h" #include "smm.h" #include "kvm_emulate.h" +#include "page_track.h" #include "cpuid.h" #include "spte.h" @@ -53,7 +54,7 @@ #include #include #include -#include + #include "trace.h" extern bool itlb_multihit_kvm_mitigation; @@ -115,11 +116,6 @@ static int max_huge_page_level __read_mostly; static int tdp_root_level __read_mostly; static int max_tdp_level __read_mostly; -#ifdef MMU_DEBUG -bool dbg = 0; -module_param(dbg, bool, 0644); -#endif - #define PTE_PREFETCH_NUM 8 #include @@ -278,16 +274,12 @@ static inline bool kvm_available_flush_remote_tlbs_range(void) return kvm_x86_ops.flush_remote_tlbs_range; } -void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, - gfn_t nr_pages) +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages) { - int ret = -EOPNOTSUPP; + if (!kvm_x86_ops.flush_remote_tlbs_range) + return -EOPNOTSUPP; - if (kvm_x86_ops.flush_remote_tlbs_range) - ret = static_call(kvm_x86_flush_remote_tlbs_range)(kvm, start_gfn, - nr_pages); - if (ret) - kvm_flush_remote_tlbs(kvm); + return static_call(kvm_x86_flush_remote_tlbs_range)(kvm, gfn, nr_pages); } static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index); @@ -490,7 +482,7 @@ retry: */ static void mmu_spte_set(u64 *sptep, u64 new_spte) { - WARN_ON(is_shadow_present_pte(*sptep)); + WARN_ON_ONCE(is_shadow_present_pte(*sptep)); __set_spte(sptep, new_spte); } @@ -502,7 +494,7 @@ static u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte) { u64 old_spte = *sptep; - WARN_ON(!is_shadow_present_pte(new_spte)); + WARN_ON_ONCE(!is_shadow_present_pte(new_spte)); check_spte_writable_invariants(new_spte); if (!is_shadow_present_pte(old_spte)) { @@ -515,7 +507,7 @@ static u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte) else old_spte = __update_clear_spte_slow(sptep, new_spte); - WARN_ON(spte_to_pfn(old_spte) != spte_to_pfn(new_spte)); + WARN_ON_ONCE(spte_to_pfn(old_spte) != spte_to_pfn(new_spte)); return old_spte; } @@ -597,7 +589,7 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep) * by a refcounted page, the refcount is elevated. */ page = kvm_pfn_to_refcounted_page(pfn); - WARN_ON(page && !page_count(page)); + WARN_ON_ONCE(page && !page_count(page)); if (is_accessed_spte(old_spte)) kvm_set_pfn_accessed(pfn); @@ -812,7 +804,7 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot, for (i = PG_LEVEL_2M; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) { linfo = lpage_info_slot(gfn, slot, i); linfo->disallow_lpage += count; - WARN_ON(linfo->disallow_lpage < 0); + WARN_ON_ONCE(linfo->disallow_lpage < 0); } } @@ -839,8 +831,7 @@ static void account_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp) /* the non-leaf shadow pages are keeping readonly. */ if (sp->role.level > PG_LEVEL_4K) - return kvm_slot_page_track_add_page(kvm, slot, gfn, - KVM_PAGE_TRACK_WRITE); + return __kvm_write_track_add_gfn(kvm, slot, gfn); kvm_mmu_gfn_disallow_lpage(slot, gfn); @@ -886,8 +877,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp) slots = kvm_memslots_for_spte_role(kvm, sp->role); slot = __gfn_to_memslot(slots, gfn); if (sp->role.level > PG_LEVEL_4K) - return kvm_slot_page_track_remove_page(kvm, slot, gfn, - KVM_PAGE_TRACK_WRITE); + return __kvm_write_track_remove_gfn(kvm, slot, gfn); kvm_mmu_gfn_allow_lpage(slot, gfn); } @@ -941,10 +931,8 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, int count = 0; if (!rmap_head->val) { - rmap_printk("%p %llx 0->1\n", spte, *spte); rmap_head->val = (unsigned long)spte; } else if (!(rmap_head->val & 1)) { - rmap_printk("%p %llx 1->many\n", spte, *spte); desc = kvm_mmu_memory_cache_alloc(cache); desc->sptes[0] = (u64 *)rmap_head->val; desc->sptes[1] = spte; @@ -953,7 +941,6 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, rmap_head->val = (unsigned long)desc | 1; ++count; } else { - rmap_printk("%p %llx many->many\n", spte, *spte); desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); count = desc->tail_count + desc->spte_count; @@ -973,7 +960,8 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, return count; } -static void pte_list_desc_remove_entry(struct kvm_rmap_head *rmap_head, +static void pte_list_desc_remove_entry(struct kvm *kvm, + struct kvm_rmap_head *rmap_head, struct pte_list_desc *desc, int i) { struct pte_list_desc *head_desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); @@ -984,7 +972,7 @@ static void pte_list_desc_remove_entry(struct kvm_rmap_head *rmap_head, * when adding an entry and the previous head is full, and heads are * removed (this flow) when they become empty. */ - BUG_ON(j < 0); + KVM_BUG_ON_DATA_CORRUPTION(j < 0, kvm); /* * Replace the to-be-freed SPTE with the last valid entry from the head @@ -1009,35 +997,34 @@ static void pte_list_desc_remove_entry(struct kvm_rmap_head *rmap_head, mmu_free_pte_list_desc(head_desc); } -static void pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head) +static void pte_list_remove(struct kvm *kvm, u64 *spte, + struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc; int i; - if (!rmap_head->val) { - pr_err("%s: %p 0->BUG\n", __func__, spte); - BUG(); - } else if (!(rmap_head->val & 1)) { - rmap_printk("%p 1->0\n", spte); - if ((u64 *)rmap_head->val != spte) { - pr_err("%s: %p 1->BUG\n", __func__, spte); - BUG(); - } + if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_head->val, kvm)) + return; + + if (!(rmap_head->val & 1)) { + if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_head->val != spte, kvm)) + return; + rmap_head->val = 0; } else { - rmap_printk("%p many->many\n", spte); desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); while (desc) { for (i = 0; i < desc->spte_count; ++i) { if (desc->sptes[i] == spte) { - pte_list_desc_remove_entry(rmap_head, desc, i); + pte_list_desc_remove_entry(kvm, rmap_head, + desc, i); return; } } desc = desc->more; } - pr_err("%s: %p many->many\n", __func__, spte); - BUG(); + + KVM_BUG_ON_DATA_CORRUPTION(true, kvm); } } @@ -1045,7 +1032,7 @@ static void kvm_zap_one_rmap_spte(struct kvm *kvm, struct kvm_rmap_head *rmap_head, u64 *sptep) { mmu_spte_clear_track_bits(kvm, sptep); - pte_list_remove(sptep, rmap_head); + pte_list_remove(kvm, sptep, rmap_head); } /* Return true if at least one SPTE was zapped, false otherwise */ @@ -1120,7 +1107,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte) slot = __gfn_to_memslot(slots, gfn); rmap_head = gfn_to_rmap(gfn, sp->role.level, slot); - pte_list_remove(spte, rmap_head); + pte_list_remove(kvm, spte, rmap_head); } /* @@ -1212,7 +1199,7 @@ static void drop_large_spte(struct kvm *kvm, u64 *sptep, bool flush) struct kvm_mmu_page *sp; sp = sptep_to_sp(sptep); - WARN_ON(sp->role.level == PG_LEVEL_4K); + WARN_ON_ONCE(sp->role.level == PG_LEVEL_4K); drop_spte(kvm, sptep); @@ -1241,8 +1228,6 @@ static bool spte_write_protect(u64 *sptep, bool pt_protect) !(pt_protect && is_mmu_writable_spte(spte))) return false; - rmap_printk("spte %p %llx\n", sptep, *sptep); - if (pt_protect) spte &= ~shadow_mmu_writable_mask; spte = spte & ~PT_WRITABLE_MASK; @@ -1267,9 +1252,7 @@ static bool spte_clear_dirty(u64 *sptep) { u64 spte = *sptep; - rmap_printk("spte %p %llx\n", sptep, *sptep); - - MMU_WARN_ON(!spte_ad_enabled(spte)); + KVM_MMU_WARN_ON(!spte_ad_enabled(spte)); spte &= ~shadow_dirty_mask; return mmu_spte_update(sptep, spte); } @@ -1475,14 +1458,11 @@ static bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, u64 new_spte; kvm_pfn_t new_pfn; - WARN_ON(pte_huge(pte)); + WARN_ON_ONCE(pte_huge(pte)); new_pfn = pte_pfn(pte); restart: for_each_rmap_spte(rmap_head, &iter, sptep) { - rmap_printk("spte %p %llx gfn %llx (%d)\n", - sptep, *sptep, gfn, level); - need_flush = true; if (pte_write(pte)) { @@ -1588,7 +1568,7 @@ static __always_inline bool kvm_handle_gfn_range(struct kvm *kvm, for_each_slot_rmap_range(range->slot, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, range->start, range->end - 1, &iterator) ret |= handler(kvm, iterator.rmap, range->slot, iterator.gfn, - iterator.level, range->pte); + iterator.level, range->arg.pte); return ret; } @@ -1710,21 +1690,19 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return young; } -#ifdef MMU_DEBUG -static int is_empty_shadow_page(u64 *spt) +static void kvm_mmu_check_sptes_at_free(struct kvm_mmu_page *sp) { - u64 *pos; - u64 *end; +#ifdef CONFIG_KVM_PROVE_MMU + int i; - for (pos = spt, end = pos + SPTE_ENT_PER_PAGE; pos != end; pos++) - if (is_shadow_present_pte(*pos)) { - printk(KERN_ERR "%s: %p %llx\n", __func__, - pos, *pos); - return 0; - } - return 1; -} + for (i = 0; i < SPTE_ENT_PER_PAGE; i++) { + if (KVM_MMU_WARN_ON(is_shadow_present_pte(sp->spt[i]))) + pr_err_ratelimited("SPTE %llx (@ %p) for gfn %llx shadow-present at free", + sp->spt[i], &sp->spt[i], + kvm_mmu_page_get_gfn(sp, i)); + } #endif +} /* * This value is the sum of all of the kvm instances's @@ -1752,7 +1730,8 @@ static void kvm_unaccount_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) static void kvm_mmu_free_shadow_page(struct kvm_mmu_page *sp) { - MMU_WARN_ON(!is_empty_shadow_page(sp->spt)); + kvm_mmu_check_sptes_at_free(sp); + hlist_del(&sp->hash_link); list_del(&sp->link); free_page((unsigned long)sp->spt); @@ -1775,16 +1754,16 @@ static void mmu_page_add_parent_pte(struct kvm_mmu_memory_cache *cache, pte_list_add(cache, parent_pte, &sp->parent_ptes); } -static void mmu_page_remove_parent_pte(struct kvm_mmu_page *sp, +static void mmu_page_remove_parent_pte(struct kvm *kvm, struct kvm_mmu_page *sp, u64 *parent_pte) { - pte_list_remove(parent_pte, &sp->parent_ptes); + pte_list_remove(kvm, parent_pte, &sp->parent_ptes); } -static void drop_parent_pte(struct kvm_mmu_page *sp, +static void drop_parent_pte(struct kvm *kvm, struct kvm_mmu_page *sp, u64 *parent_pte) { - mmu_page_remove_parent_pte(sp, parent_pte); + mmu_page_remove_parent_pte(kvm, sp, parent_pte); mmu_spte_clear_no_track(parent_pte); } @@ -1840,7 +1819,7 @@ static int mmu_pages_add(struct kvm_mmu_pages *pvec, struct kvm_mmu_page *sp, static inline void clear_unsync_child_bit(struct kvm_mmu_page *sp, int idx) { --sp->unsync_children; - WARN_ON((int)sp->unsync_children < 0); + WARN_ON_ONCE((int)sp->unsync_children < 0); __clear_bit(idx, sp->unsync_child_bitmap); } @@ -1898,7 +1877,7 @@ static int mmu_unsync_walk(struct kvm_mmu_page *sp, static void kvm_unlink_unsync_page(struct kvm *kvm, struct kvm_mmu_page *sp) { - WARN_ON(!sp->unsync); + WARN_ON_ONCE(!sp->unsync); trace_kvm_mmu_sync_page(sp); sp->unsync = 0; --kvm->stat.mmu_unsync; @@ -2073,11 +2052,11 @@ static int mmu_pages_first(struct kvm_mmu_pages *pvec, if (pvec->nr == 0) return 0; - WARN_ON(pvec->page[0].idx != INVALID_INDEX); + WARN_ON_ONCE(pvec->page[0].idx != INVALID_INDEX); sp = pvec->page[0].sp; level = sp->role.level; - WARN_ON(level == PG_LEVEL_4K); + WARN_ON_ONCE(level == PG_LEVEL_4K); parents->parent[level-2] = sp; @@ -2099,7 +2078,7 @@ static void mmu_pages_clear_parents(struct mmu_page_path *parents) if (!sp) return; - WARN_ON(idx == INVALID_INDEX); + WARN_ON_ONCE(idx == INVALID_INDEX); clear_unsync_child_bit(sp, idx); level++; } while (!sp->unsync_children); @@ -2220,7 +2199,7 @@ static struct kvm_mmu_page *kvm_mmu_find_shadow_page(struct kvm *kvm, if (ret < 0) break; - WARN_ON(!list_empty(&invalid_list)); + WARN_ON_ONCE(!list_empty(&invalid_list)); if (ret > 0) kvm_flush_remote_tlbs(kvm); } @@ -2499,7 +2478,7 @@ static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, if (child->role.access == direct_access) return; - drop_parent_pte(child, sptep); + drop_parent_pte(vcpu->kvm, child, sptep); kvm_flush_remote_tlbs_sptep(vcpu->kvm, sptep); } } @@ -2517,7 +2496,7 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, drop_spte(kvm, spte); } else { child = spte_to_child_sp(pte); - drop_parent_pte(child, spte); + drop_parent_pte(kvm, child, spte); /* * Recursively zap nested TDP SPs, parentless SPs are @@ -2548,13 +2527,13 @@ static int kvm_mmu_page_unlink_children(struct kvm *kvm, return zapped; } -static void kvm_mmu_unlink_parents(struct kvm_mmu_page *sp) +static void kvm_mmu_unlink_parents(struct kvm *kvm, struct kvm_mmu_page *sp) { u64 *sptep; struct rmap_iterator iter; while ((sptep = rmap_get_first(&sp->parent_ptes, &iter))) - drop_parent_pte(sp, sptep); + drop_parent_pte(kvm, sp, sptep); } static int mmu_zap_unsync_children(struct kvm *kvm, @@ -2593,7 +2572,7 @@ static bool __kvm_mmu_prepare_zap_page(struct kvm *kvm, ++kvm->stat.mmu_shadow_zapped; *nr_zapped = mmu_zap_unsync_children(kvm, sp, invalid_list); *nr_zapped += kvm_mmu_page_unlink_children(kvm, sp, invalid_list); - kvm_mmu_unlink_parents(sp); + kvm_mmu_unlink_parents(kvm, sp); /* Zapping children means active_mmu_pages has become unstable. */ list_unstable = *nr_zapped; @@ -2675,7 +2654,7 @@ static void kvm_mmu_commit_zap_page(struct kvm *kvm, kvm_flush_remote_tlbs(kvm); list_for_each_entry_safe(sp, nsp, invalid_list, link) { - WARN_ON(!sp->role.invalid || sp->root_count); + WARN_ON_ONCE(!sp->role.invalid || sp->root_count); kvm_mmu_free_shadow_page(sp); } } @@ -2775,12 +2754,9 @@ int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn) LIST_HEAD(invalid_list); int r; - pgprintk("%s: looking for gfn %llx\n", __func__, gfn); r = 0; write_lock(&kvm->mmu_lock); for_each_gfn_valid_sp_with_gptes(kvm, sp, gfn) { - pgprintk("%s: gfn %llx role %x\n", __func__, gfn, - sp->role.word); r = 1; kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list); } @@ -2831,7 +2807,7 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, * track machinery is used to write-protect upper-level shadow pages, * i.e. this guards the role.level == 4K assertion below! */ - if (kvm_slot_page_track_is_active(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE)) + if (kvm_gfn_is_write_tracked(kvm, slot, gfn)) return -EPERM; /* @@ -2873,7 +2849,7 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, continue; } - WARN_ON(sp->role.level != PG_LEVEL_4K); + WARN_ON_ONCE(sp->role.level != PG_LEVEL_4K); kvm_unsync_page(kvm, sp); } if (locked) @@ -2938,9 +2914,6 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, bool prefetch = !fault || fault->prefetch; bool write_fault = fault && fault->write; - pgprintk("%s: spte %llx write_fault %d gfn %llx\n", __func__, - *sptep, write_fault, gfn); - if (unlikely(is_noslot_pfn(pfn))) { vcpu->stat.pf_mmio_spte_created++; mark_mmio_spte(vcpu, sptep, gfn, pte_access); @@ -2957,11 +2930,9 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, u64 pte = *sptep; child = spte_to_child_sp(pte); - drop_parent_pte(child, sptep); + drop_parent_pte(vcpu->kvm, child, sptep); flush = true; } else if (pfn != spte_to_pfn(*sptep)) { - pgprintk("hfn old %llx new %llx\n", - spte_to_pfn(*sptep), pfn); drop_spte(vcpu->kvm, sptep); flush = true; } else @@ -2986,8 +2957,6 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, if (flush) kvm_flush_remote_tlbs_gfn(vcpu->kvm, gfn, level); - pgprintk("%s: setting spte %llx\n", __func__, *sptep); - if (!was_rmapped) { WARN_ON_ONCE(ret == RET_PF_SPURIOUS); rmap_add(vcpu, slot, sptep, gfn, pte_access); @@ -3033,7 +3002,7 @@ static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *spte, *start = NULL; int i; - WARN_ON(!sp->role.direct); + WARN_ON_ONCE(!sp->role.direct); i = spte_index(sptep) & ~(PTE_PREFETCH_NUM - 1); spte = sp->spt + i; @@ -3574,12 +3543,8 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, if (!VALID_PAGE(*root_hpa)) return; - /* - * The "root" may be a special root, e.g. a PAE entry, treat it as a - * SPTE to ensure any non-PA bits are dropped. - */ - sp = spte_to_child_sp(*root_hpa); - if (WARN_ON(!sp)) + sp = root_to_sp(*root_hpa); + if (WARN_ON_ONCE(!sp)) return; if (is_tdp_mmu_page(sp)) @@ -3624,7 +3589,9 @@ void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, &invalid_list); if (free_active_root) { - if (to_shadow_page(mmu->root.hpa)) { + if (kvm_mmu_is_dummy_root(mmu->root.hpa)) { + /* Nothing to cleanup for dummy roots. */ + } else if (root_to_sp(mmu->root.hpa)) { mmu_free_root_page(kvm, &mmu->root.hpa, &invalid_list); } else if (mmu->pae_root) { for (i = 0; i < 4; ++i) { @@ -3648,6 +3615,7 @@ EXPORT_SYMBOL_GPL(kvm_mmu_free_roots); void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) { unsigned long roots_to_free = 0; + struct kvm_mmu_page *sp; hpa_t root_hpa; int i; @@ -3662,8 +3630,8 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) if (!VALID_PAGE(root_hpa)) continue; - if (!to_shadow_page(root_hpa) || - to_shadow_page(root_hpa)->role.guest_mode) + sp = root_to_sp(root_hpa); + if (!sp || sp->role.guest_mode) roots_to_free |= KVM_MMU_ROOT_PREVIOUS(i); } @@ -3671,19 +3639,6 @@ void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu) } EXPORT_SYMBOL_GPL(kvm_mmu_free_guest_mode_roots); - -static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_t root_gfn) -{ - int ret = 0; - - if (!kvm_vcpu_is_visible_gfn(vcpu, root_gfn)) { - kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu); - ret = 1; - } - - return ret; -} - static hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, u8 level) { @@ -3821,8 +3776,10 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) root_pgd = kvm_mmu_get_guest_pgd(vcpu, mmu); root_gfn = root_pgd >> PAGE_SHIFT; - if (mmu_check_root(vcpu, root_gfn)) - return 1; + if (!kvm_vcpu_is_visible_gfn(vcpu, root_gfn)) { + mmu->root.hpa = kvm_mmu_get_dummy_root(); + return 0; + } /* * On SVM, reading PDPTRs might access guest memory, which might fault @@ -3834,8 +3791,8 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) if (!(pdptrs[i] & PT_PRESENT_MASK)) continue; - if (mmu_check_root(vcpu, pdptrs[i] >> PAGE_SHIFT)) - return 1; + if (!kvm_vcpu_is_visible_gfn(vcpu, pdptrs[i] >> PAGE_SHIFT)) + pdptrs[i] = 0; } } @@ -4002,7 +3959,7 @@ static bool is_unsync_root(hpa_t root) { struct kvm_mmu_page *sp; - if (!VALID_PAGE(root)) + if (!VALID_PAGE(root) || kvm_mmu_is_dummy_root(root)) return false; /* @@ -4018,7 +3975,7 @@ static bool is_unsync_root(hpa_t root) * requirement isn't satisfied. */ smp_rmb(); - sp = to_shadow_page(root); + sp = root_to_sp(root); /* * PAE roots (somewhat arbitrarily) aren't backed by shadow pages, the @@ -4048,11 +4005,12 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) if (vcpu->arch.mmu->cpu_role.base.level >= PT64_ROOT_4LEVEL) { hpa_t root = vcpu->arch.mmu->root.hpa; - sp = to_shadow_page(root); if (!is_unsync_root(root)) return; + sp = root_to_sp(root); + write_lock(&vcpu->kvm->mmu_lock); mmu_sync_children(vcpu, sp, true); write_unlock(&vcpu->kvm->mmu_lock); @@ -4194,7 +4152,7 @@ static int handle_mmio_page_fault(struct kvm_vcpu *vcpu, u64 addr, bool direct) return RET_PF_EMULATE; reserved = get_mmio_spte(vcpu, addr, &spte); - if (WARN_ON(reserved)) + if (WARN_ON_ONCE(reserved)) return -EINVAL; if (is_mmio_spte(spte)) { @@ -4232,7 +4190,7 @@ static bool page_fault_handle_page_track(struct kvm_vcpu *vcpu, * guest is writing the page which is write tracked which can * not be fixed by page fault handler. */ - if (kvm_slot_page_track_is_active(vcpu->kvm, fault->slot, fault->gfn, KVM_PAGE_TRACK_WRITE)) + if (kvm_gfn_is_write_tracked(vcpu->kvm, fault->slot, fault->gfn)) return true; return false; @@ -4382,7 +4340,7 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, static bool is_page_fault_stale(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { - struct kvm_mmu_page *sp = to_shadow_page(vcpu->arch.mmu->root.hpa); + struct kvm_mmu_page *sp = root_to_sp(vcpu->arch.mmu->root.hpa); /* Special roots, e.g. pae_root, are not backed by shadow pages. */ if (sp && is_obsolete_sp(vcpu->kvm, sp)) @@ -4407,6 +4365,10 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault { int r; + /* Dummy roots are used only for shadowing bad guest roots. */ + if (WARN_ON_ONCE(kvm_mmu_is_dummy_root(vcpu->arch.mmu->root.hpa))) + return RET_PF_RETRY; + if (page_fault_handle_page_track(vcpu, fault)) return RET_PF_EMULATE; @@ -4443,8 +4405,6 @@ out_unlock: static int nonpaging_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { - pgprintk("%s: gva %lx error %x\n", __func__, fault->addr, fault->error_code); - /* This path builds a PAE pagetable, we can map 2mb pages at maximum. */ fault->max_level = PG_LEVEL_2M; return direct_page_fault(vcpu, fault); @@ -4562,9 +4522,19 @@ static void nonpaging_init_context(struct kvm_mmu *context) static inline bool is_root_usable(struct kvm_mmu_root_info *root, gpa_t pgd, union kvm_mmu_page_role role) { - return (role.direct || pgd == root->pgd) && - VALID_PAGE(root->hpa) && - role.word == to_shadow_page(root->hpa)->role.word; + struct kvm_mmu_page *sp; + + if (!VALID_PAGE(root->hpa)) + return false; + + if (!role.direct && pgd != root->pgd) + return false; + + sp = root_to_sp(root->hpa); + if (WARN_ON_ONCE(!sp)) + return false; + + return role.word == sp->role.word; } /* @@ -4634,11 +4604,10 @@ static bool fast_pgd_switch(struct kvm *kvm, struct kvm_mmu *mmu, gpa_t new_pgd, union kvm_mmu_page_role new_role) { /* - * For now, limit the caching to 64-bit hosts+VMs in order to avoid - * having to deal with PDPTEs. We may add support for 32-bit hosts/VMs - * later if necessary. + * Limit reuse to 64-bit hosts+VMs without "special" roots in order to + * avoid having to deal with PDPTEs and other complexities. */ - if (VALID_PAGE(mmu->root.hpa) && !to_shadow_page(mmu->root.hpa)) + if (VALID_PAGE(mmu->root.hpa) && !root_to_sp(mmu->root.hpa)) kvm_mmu_free_roots(kvm, mmu, KVM_MMU_ROOT_CURRENT); if (VALID_PAGE(mmu->root.hpa)) @@ -4684,9 +4653,12 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) * If this is a direct root page, it doesn't have a write flooding * count. Otherwise, clear the write flooding count. */ - if (!new_role.direct) - __clear_sp_write_flooding_count( - to_shadow_page(vcpu->arch.mmu->root.hpa)); + if (!new_role.direct) { + struct kvm_mmu_page *sp = root_to_sp(vcpu->arch.mmu->root.hpa); + + if (!WARN_ON_ONCE(!sp)) + __clear_sp_write_flooding_count(sp); + } } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); @@ -4808,28 +4780,13 @@ static void __reset_rsvds_bits_mask(struct rsvd_bits_validate *rsvd_check, } } -static bool guest_can_use_gbpages(struct kvm_vcpu *vcpu) -{ - /* - * If TDP is enabled, let the guest use GBPAGES if they're supported in - * hardware. The hardware page walker doesn't let KVM disable GBPAGES, - * i.e. won't treat them as reserved, and KVM doesn't redo the GVA->GPA - * walk for performance and complexity reasons. Not to mention KVM - * _can't_ solve the problem because GVA->GPA walks aren't visible to - * KVM once a TDP translation is installed. Mimic hardware behavior so - * that KVM's is at least consistent, i.e. doesn't randomly inject #PF. - */ - return tdp_enabled ? boot_cpu_has(X86_FEATURE_GBPAGES) : - guest_cpuid_has(vcpu, X86_FEATURE_GBPAGES); -} - static void reset_guest_rsvds_bits_mask(struct kvm_vcpu *vcpu, struct kvm_mmu *context) { __reset_rsvds_bits_mask(&context->guest_rsvd_check, vcpu->arch.reserved_gpa_bits, context->cpu_role.base.level, is_efer_nx(context), - guest_can_use_gbpages(vcpu), + guest_can_use(vcpu, X86_FEATURE_GBPAGES), is_cr4_pse(context), guest_cpuid_is_amd_or_hygon(vcpu)); } @@ -4906,7 +4863,8 @@ static void reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, __reset_rsvds_bits_mask(shadow_zero_check, reserved_hpa_bits(), context->root_role.level, context->root_role.efer_nx, - guest_can_use_gbpages(vcpu), is_pse, is_amd); + guest_can_use(vcpu, X86_FEATURE_GBPAGES), + is_pse, is_amd); if (!shadow_me_mask) return; @@ -5467,8 +5425,8 @@ void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu) * physical address properties) in a single VM would require tracking * all relevant CPUID information in kvm_mmu_page_role. That is very * undesirable as it would increase the memory requirements for - * gfn_track (see struct kvm_mmu_page_role comments). For now that - * problem is swept under the rug; KVM's CPUID API is horrific and + * gfn_write_track (see struct kvm_mmu_page_role comments). For now + * that problem is swept under the rug; KVM's CPUID API is horrific and * it's all but impossible to solve it without introducing a new API. */ vcpu->arch.root_mmu.root_role.word = 0; @@ -5531,9 +5489,9 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu) struct kvm *kvm = vcpu->kvm; kvm_mmu_free_roots(kvm, &vcpu->arch.root_mmu, KVM_MMU_ROOTS_ALL); - WARN_ON(VALID_PAGE(vcpu->arch.root_mmu.root.hpa)); + WARN_ON_ONCE(VALID_PAGE(vcpu->arch.root_mmu.root.hpa)); kvm_mmu_free_roots(kvm, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); - WARN_ON(VALID_PAGE(vcpu->arch.guest_mmu.root.hpa)); + WARN_ON_ONCE(VALID_PAGE(vcpu->arch.guest_mmu.root.hpa)); vcpu_clear_mmio_info(vcpu, MMIO_GVA_ANY); } @@ -5546,16 +5504,21 @@ static bool is_obsolete_root(struct kvm *kvm, hpa_t root_hpa) /* * When freeing obsolete roots, treat roots as obsolete if they don't - * have an associated shadow page. This does mean KVM will get false + * have an associated shadow page, as it's impossible to determine if + * such roots are fresh or stale. This does mean KVM will get false * positives and free roots that don't strictly need to be freed, but * such false positives are relatively rare: * - * (a) only PAE paging and nested NPT has roots without shadow pages + * (a) only PAE paging and nested NPT have roots without shadow pages + * (or any shadow paging flavor with a dummy root, see note below) * (b) remote reloads due to a memslot update obsoletes _all_ roots * (c) KVM doesn't track previous roots for PAE paging, and the guest * is unlikely to zap an in-use PGD. + * + * Note! Dummy roots are unique in that they are obsoleted by memslot + * _creation_! See also FNAME(fetch). */ - sp = to_shadow_page(root_hpa); + sp = root_to_sp(root_hpa); return !sp || is_obsolete_sp(kvm, sp); } @@ -5634,9 +5597,6 @@ static bool detect_write_misaligned(struct kvm_mmu_page *sp, gpa_t gpa, { unsigned offset, pte_size, misaligned; - pgprintk("misaligned: gpa %llx bytes %d role %x\n", - gpa, bytes, sp->role.word); - offset = offset_in_page(gpa); pte_size = sp->role.has_4_byte_gpte ? 4 : 8; @@ -5684,9 +5644,8 @@ static u64 *get_written_sptes(struct kvm_mmu_page *sp, gpa_t gpa, int *nspte) return spte; } -static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, - const u8 *new, int bytes, - struct kvm_page_track_notifier_node *node) +void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, + int bytes) { gfn_t gfn = gpa >> PAGE_SHIFT; struct kvm_mmu_page *sp; @@ -5702,8 +5661,6 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, if (!READ_ONCE(vcpu->kvm->arch.indirect_shadow_pages)) return; - pgprintk("%s: gpa %llx bytes %d\n", __func__, gpa, bytes); - write_lock(&vcpu->kvm->mmu_lock); gentry = mmu_pte_write_fetch_gpte(vcpu, &gpa, &bytes); @@ -5742,7 +5699,18 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err int r, emulation_type = EMULTYPE_PF; bool direct = vcpu->arch.mmu->root_role.direct; - if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa))) + /* + * IMPLICIT_ACCESS is a KVM-defined flag used to correctly perform SMAP + * checks when emulating instructions that triggers implicit access. + * WARN if hardware generates a fault with an error code that collides + * with the KVM-defined value. Clear the flag and continue on, i.e. + * don't terminate the VM, as KVM can't possibly be relying on a flag + * that KVM doesn't know about. + */ + if (WARN_ON_ONCE(error_code & PFERR_IMPLICIT_ACCESS)) + error_code &= ~PFERR_IMPLICIT_ACCESS; + + if (WARN_ON_ONCE(!VALID_PAGE(vcpu->arch.mmu->root.hpa))) return RET_PF_RETRY; r = RET_PF_INVALID; @@ -6099,7 +6067,7 @@ restart: * pages. Skip the bogus page, otherwise we'll get stuck in an * infinite loop if the page gets put back on the list (again). */ - if (WARN_ON(sp->role.invalid)) + if (WARN_ON_ONCE(sp->role.invalid)) continue; /* @@ -6199,16 +6167,8 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm) return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages)); } -static void kvm_mmu_invalidate_zap_pages_in_memslot(struct kvm *kvm, - struct kvm_memory_slot *slot, - struct kvm_page_track_notifier_node *node) -{ - kvm_mmu_zap_all_fast(kvm); -} - int kvm_mmu_init_vm(struct kvm *kvm) { - struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker; int r; INIT_LIST_HEAD(&kvm->arch.active_mmu_pages); @@ -6222,10 +6182,6 @@ int kvm_mmu_init_vm(struct kvm *kvm) return r; } - node->track_write = kvm_mmu_pte_write; - node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot; - kvm_page_track_register_notifier(kvm, node); - kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache; kvm->arch.split_page_header_cache.gfp_zero = __GFP_ZERO; @@ -6246,10 +6202,6 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm) void kvm_mmu_uninit_vm(struct kvm *kvm) { - struct kvm_page_track_notifier_node *node = &kvm->arch.mmu_sp_tracker; - - kvm_page_track_unregister_notifier(kvm, node); - if (tdp_mmu_enabled) kvm_mmu_uninit_tdp_mmu(kvm); @@ -6670,7 +6622,7 @@ static void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm, */ if (walk_slot_rmaps(kvm, slot, kvm_mmu_zap_collapsible_spte, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL - 1, true)) - kvm_arch_flush_remote_tlbs_memslot(kvm, slot); + kvm_flush_remote_tlbs_memslot(kvm, slot); } void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, @@ -6689,20 +6641,6 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, } } -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot) -{ - /* - * All current use cases for flushing the TLBs for a specific memslot - * related to dirty logging, and many do the TLB flush out of mmu_lock. - * The interaction between the various operations on memslot must be - * serialized by slots_locks to ensure the TLB flush from one operation - * is observed by any other operation on the same memslot. - */ - lockdep_assert_held(&kvm->slots_lock); - kvm_flush_remote_tlbs_range(kvm, memslot->base_gfn, memslot->npages); -} - void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, const struct kvm_memory_slot *memslot) { @@ -6732,7 +6670,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, */ } -void kvm_mmu_zap_all(struct kvm *kvm) +static void kvm_mmu_zap_all(struct kvm *kvm) { struct kvm_mmu_page *sp, *node; LIST_HEAD(invalid_list); @@ -6741,7 +6679,7 @@ void kvm_mmu_zap_all(struct kvm *kvm) write_lock(&kvm->mmu_lock); restart: list_for_each_entry_safe(sp, node, &kvm->arch.active_mmu_pages, link) { - if (WARN_ON(sp->role.invalid)) + if (WARN_ON_ONCE(sp->role.invalid)) continue; if (__kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list, &ign)) goto restart; @@ -6757,9 +6695,20 @@ restart: write_unlock(&kvm->mmu_lock); } +void kvm_arch_flush_shadow_all(struct kvm *kvm) +{ + kvm_mmu_zap_all(kvm); +} + +void kvm_arch_flush_shadow_memslot(struct kvm *kvm, + struct kvm_memory_slot *slot) +{ + kvm_mmu_zap_all_fast(kvm); +} + void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen) { - WARN_ON(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS); + WARN_ON_ONCE(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS); gen &= MMIO_SPTE_GEN_MASK; @@ -6862,7 +6811,7 @@ static void mmu_destroy_caches(void) static int get_nx_huge_pages(char *buffer, const struct kernel_param *kp) { if (nx_hugepage_mitigation_hard_disabled) - return sprintf(buffer, "never\n"); + return sysfs_emit(buffer, "never\n"); return param_get_bool(buffer, kp); } diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index d39af5639ce9..b102014e2c60 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -6,18 +6,10 @@ #include #include -#undef MMU_DEBUG - -#ifdef MMU_DEBUG -extern bool dbg; - -#define pgprintk(x...) do { if (dbg) printk(x); } while (0) -#define rmap_printk(fmt, args...) do { if (dbg) printk("%s: " fmt, __func__, ## args); } while (0) -#define MMU_WARN_ON(x) WARN_ON(x) +#ifdef CONFIG_KVM_PROVE_MMU +#define KVM_MMU_WARN_ON(x) WARN_ON_ONCE(x) #else -#define pgprintk(x...) do { } while (0) -#define rmap_printk(x...) do { } while (0) -#define MMU_WARN_ON(x) do { } while (0) +#define KVM_MMU_WARN_ON(x) BUILD_BUG_ON_INVALID(x) #endif /* Page table builder macros common to shadow (host) PTEs and guest PTEs. */ @@ -44,6 +36,16 @@ extern bool dbg; #define INVALID_PAE_ROOT 0 #define IS_VALID_PAE_ROOT(x) (!!(x)) +static inline hpa_t kvm_mmu_get_dummy_root(void) +{ + return my_zero_pfn(0) << PAGE_SHIFT; +} + +static inline bool kvm_mmu_is_dummy_root(hpa_t shadow_page) +{ + return is_zero_pfn(shadow_page >> PAGE_SHIFT); +} + typedef u64 __rcu *tdp_ptep_t; struct kvm_mmu_page { @@ -170,9 +172,6 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, struct kvm_memory_slot *slot, u64 gfn, int min_level); -void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, - gfn_t nr_pages); - /* Flush the given page (huge or not) of guest memory. */ static inline void kvm_flush_remote_tlbs_gfn(struct kvm *kvm, gfn_t gfn, int level) { diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c index 0a2ac438d647..c87da11f3a04 100644 --- a/arch/x86/kvm/mmu/page_track.c +++ b/arch/x86/kvm/mmu/page_track.c @@ -12,13 +12,13 @@ */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include #include #include -#include - #include "mmu.h" #include "mmu_internal.h" +#include "page_track.h" bool kvm_page_track_write_tracking_enabled(struct kvm *kvm) { @@ -28,103 +28,64 @@ bool kvm_page_track_write_tracking_enabled(struct kvm *kvm) void kvm_page_track_free_memslot(struct kvm_memory_slot *slot) { - int i; + kvfree(slot->arch.gfn_write_track); + slot->arch.gfn_write_track = NULL; +} - for (i = 0; i < KVM_PAGE_TRACK_MAX; i++) { - kvfree(slot->arch.gfn_track[i]); - slot->arch.gfn_track[i] = NULL; - } +static int __kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot, + unsigned long npages) +{ + const size_t size = sizeof(*slot->arch.gfn_write_track); + + if (!slot->arch.gfn_write_track) + slot->arch.gfn_write_track = __vcalloc(npages, size, + GFP_KERNEL_ACCOUNT); + + return slot->arch.gfn_write_track ? 0 : -ENOMEM; } int kvm_page_track_create_memslot(struct kvm *kvm, struct kvm_memory_slot *slot, unsigned long npages) { - int i; + if (!kvm_page_track_write_tracking_enabled(kvm)) + return 0; - for (i = 0; i < KVM_PAGE_TRACK_MAX; i++) { - if (i == KVM_PAGE_TRACK_WRITE && - !kvm_page_track_write_tracking_enabled(kvm)) - continue; - - slot->arch.gfn_track[i] = - __vcalloc(npages, sizeof(*slot->arch.gfn_track[i]), - GFP_KERNEL_ACCOUNT); - if (!slot->arch.gfn_track[i]) - goto track_free; - } - - return 0; - -track_free: - kvm_page_track_free_memslot(slot); - return -ENOMEM; -} - -static inline bool page_track_mode_is_valid(enum kvm_page_track_mode mode) -{ - if (mode < 0 || mode >= KVM_PAGE_TRACK_MAX) - return false; - - return true; + return __kvm_page_track_write_tracking_alloc(slot, npages); } int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot) { - unsigned short *gfn_track; - - if (slot->arch.gfn_track[KVM_PAGE_TRACK_WRITE]) - return 0; - - gfn_track = __vcalloc(slot->npages, sizeof(*gfn_track), - GFP_KERNEL_ACCOUNT); - if (gfn_track == NULL) - return -ENOMEM; - - slot->arch.gfn_track[KVM_PAGE_TRACK_WRITE] = gfn_track; - return 0; + return __kvm_page_track_write_tracking_alloc(slot, slot->npages); } -static void update_gfn_track(struct kvm_memory_slot *slot, gfn_t gfn, - enum kvm_page_track_mode mode, short count) +static void update_gfn_write_track(struct kvm_memory_slot *slot, gfn_t gfn, + short count) { int index, val; index = gfn_to_index(gfn, slot->base_gfn, PG_LEVEL_4K); - val = slot->arch.gfn_track[mode][index]; + val = slot->arch.gfn_write_track[index]; - if (WARN_ON(val + count < 0 || val + count > USHRT_MAX)) + if (WARN_ON_ONCE(val + count < 0 || val + count > USHRT_MAX)) return; - slot->arch.gfn_track[mode][index] += count; + slot->arch.gfn_write_track[index] += count; } -/* - * add guest page to the tracking pool so that corresponding access on that - * page will be intercepted. - * - * It should be called under the protection both of mmu-lock and kvm->srcu - * or kvm->slots_lock. - * - * @kvm: the guest instance we are interested in. - * @slot: the @gfn belongs to. - * @gfn: the guest page. - * @mode: tracking mode, currently only write track is supported. - */ -void kvm_slot_page_track_add_page(struct kvm *kvm, - struct kvm_memory_slot *slot, gfn_t gfn, - enum kvm_page_track_mode mode) +void __kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, + gfn_t gfn) { + lockdep_assert_held_write(&kvm->mmu_lock); - if (WARN_ON(!page_track_mode_is_valid(mode))) + lockdep_assert_once(lockdep_is_held(&kvm->slots_lock) || + srcu_read_lock_held(&kvm->srcu)); + + if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm)) return; - if (WARN_ON(mode == KVM_PAGE_TRACK_WRITE && - !kvm_page_track_write_tracking_enabled(kvm))) - return; - - update_gfn_track(slot, gfn, mode, 1); + update_gfn_write_track(slot, gfn, 1); /* * new track stops large page mapping for the @@ -132,37 +93,22 @@ void kvm_slot_page_track_add_page(struct kvm *kvm, */ kvm_mmu_gfn_disallow_lpage(slot, gfn); - if (mode == KVM_PAGE_TRACK_WRITE) - if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K)) - kvm_flush_remote_tlbs(kvm); + if (kvm_mmu_slot_gfn_write_protect(kvm, slot, gfn, PG_LEVEL_4K)) + kvm_flush_remote_tlbs(kvm); } -EXPORT_SYMBOL_GPL(kvm_slot_page_track_add_page); -/* - * remove the guest page from the tracking pool which stops the interception - * of corresponding access on that page. It is the opposed operation of - * kvm_slot_page_track_add_page(). - * - * It should be called under the protection both of mmu-lock and kvm->srcu - * or kvm->slots_lock. - * - * @kvm: the guest instance we are interested in. - * @slot: the @gfn belongs to. - * @gfn: the guest page. - * @mode: tracking mode, currently only write track is supported. - */ -void kvm_slot_page_track_remove_page(struct kvm *kvm, - struct kvm_memory_slot *slot, gfn_t gfn, - enum kvm_page_track_mode mode) +void __kvm_write_track_remove_gfn(struct kvm *kvm, + struct kvm_memory_slot *slot, gfn_t gfn) { - if (WARN_ON(!page_track_mode_is_valid(mode))) + lockdep_assert_held_write(&kvm->mmu_lock); + + lockdep_assert_once(lockdep_is_held(&kvm->slots_lock) || + srcu_read_lock_held(&kvm->srcu)); + + if (KVM_BUG_ON(!kvm_page_track_write_tracking_enabled(kvm), kvm)) return; - if (WARN_ON(mode == KVM_PAGE_TRACK_WRITE && - !kvm_page_track_write_tracking_enabled(kvm))) - return; - - update_gfn_track(slot, gfn, mode, -1); + update_gfn_write_track(slot, gfn, -1); /* * allow large page mapping for the tracked page @@ -170,31 +116,26 @@ void kvm_slot_page_track_remove_page(struct kvm *kvm, */ kvm_mmu_gfn_allow_lpage(slot, gfn); } -EXPORT_SYMBOL_GPL(kvm_slot_page_track_remove_page); /* * check if the corresponding access on the specified guest page is tracked. */ -bool kvm_slot_page_track_is_active(struct kvm *kvm, - const struct kvm_memory_slot *slot, - gfn_t gfn, enum kvm_page_track_mode mode) +bool kvm_gfn_is_write_tracked(struct kvm *kvm, + const struct kvm_memory_slot *slot, gfn_t gfn) { int index; - if (WARN_ON(!page_track_mode_is_valid(mode))) - return false; - if (!slot) return false; - if (mode == KVM_PAGE_TRACK_WRITE && - !kvm_page_track_write_tracking_enabled(kvm)) + if (!kvm_page_track_write_tracking_enabled(kvm)) return false; index = gfn_to_index(gfn, slot->base_gfn, PG_LEVEL_4K); - return !!READ_ONCE(slot->arch.gfn_track[mode][index]); + return !!READ_ONCE(slot->arch.gfn_write_track[index]); } +#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING void kvm_page_track_cleanup(struct kvm *kvm) { struct kvm_page_track_notifier_head *head; @@ -216,17 +157,22 @@ int kvm_page_track_init(struct kvm *kvm) * register the notifier so that event interception for the tracked guest * pages can be received. */ -void -kvm_page_track_register_notifier(struct kvm *kvm, - struct kvm_page_track_notifier_node *n) +int kvm_page_track_register_notifier(struct kvm *kvm, + struct kvm_page_track_notifier_node *n) { struct kvm_page_track_notifier_head *head; + if (!kvm || kvm->mm != current->mm) + return -ESRCH; + + kvm_get_kvm(kvm); + head = &kvm->arch.track_notifier_head; write_lock(&kvm->mmu_lock); hlist_add_head_rcu(&n->node, &head->track_notifier_list); write_unlock(&kvm->mmu_lock); + return 0; } EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier); @@ -234,9 +180,8 @@ EXPORT_SYMBOL_GPL(kvm_page_track_register_notifier); * stop receiving the event interception. It is the opposed operation of * kvm_page_track_register_notifier(). */ -void -kvm_page_track_unregister_notifier(struct kvm *kvm, - struct kvm_page_track_notifier_node *n) +void kvm_page_track_unregister_notifier(struct kvm *kvm, + struct kvm_page_track_notifier_node *n) { struct kvm_page_track_notifier_head *head; @@ -246,6 +191,8 @@ kvm_page_track_unregister_notifier(struct kvm *kvm, hlist_del_rcu(&n->node); write_unlock(&kvm->mmu_lock); synchronize_srcu(&head->track_srcu); + + kvm_put_kvm(kvm); } EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier); @@ -256,34 +203,7 @@ EXPORT_SYMBOL_GPL(kvm_page_track_unregister_notifier); * The node should figure out if the written page is the one that node is * interested in by itself. */ -void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, - int bytes) -{ - struct kvm_page_track_notifier_head *head; - struct kvm_page_track_notifier_node *n; - int idx; - - head = &vcpu->kvm->arch.track_notifier_head; - - if (hlist_empty(&head->track_notifier_list)) - return; - - idx = srcu_read_lock(&head->track_srcu); - hlist_for_each_entry_srcu(n, &head->track_notifier_list, node, - srcu_read_lock_held(&head->track_srcu)) - if (n->track_write) - n->track_write(vcpu, gpa, new, bytes, n); - srcu_read_unlock(&head->track_srcu, idx); -} - -/* - * Notify the node that memory slot is being removed or moved so that it can - * drop write-protection for the pages in the memory slot. - * - * The node should figure out it has any write-protected pages in this slot - * by itself. - */ -void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) +void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int bytes) { struct kvm_page_track_notifier_head *head; struct kvm_page_track_notifier_node *n; @@ -296,8 +216,92 @@ void kvm_page_track_flush_slot(struct kvm *kvm, struct kvm_memory_slot *slot) idx = srcu_read_lock(&head->track_srcu); hlist_for_each_entry_srcu(n, &head->track_notifier_list, node, - srcu_read_lock_held(&head->track_srcu)) - if (n->track_flush_slot) - n->track_flush_slot(kvm, slot, n); + srcu_read_lock_held(&head->track_srcu)) + if (n->track_write) + n->track_write(gpa, new, bytes, n); srcu_read_unlock(&head->track_srcu, idx); } + +/* + * Notify external page track nodes that a memory region is being removed from + * the VM, e.g. so that users can free any associated metadata. + */ +void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot) +{ + struct kvm_page_track_notifier_head *head; + struct kvm_page_track_notifier_node *n; + int idx; + + head = &kvm->arch.track_notifier_head; + + if (hlist_empty(&head->track_notifier_list)) + return; + + idx = srcu_read_lock(&head->track_srcu); + hlist_for_each_entry_srcu(n, &head->track_notifier_list, node, + srcu_read_lock_held(&head->track_srcu)) + if (n->track_remove_region) + n->track_remove_region(slot->base_gfn, slot->npages, n); + srcu_read_unlock(&head->track_srcu, idx); +} + +/* + * add guest page to the tracking pool so that corresponding access on that + * page will be intercepted. + * + * @kvm: the guest instance we are interested in. + * @gfn: the guest page. + */ +int kvm_write_track_add_gfn(struct kvm *kvm, gfn_t gfn) +{ + struct kvm_memory_slot *slot; + int idx; + + idx = srcu_read_lock(&kvm->srcu); + + slot = gfn_to_memslot(kvm, gfn); + if (!slot) { + srcu_read_unlock(&kvm->srcu, idx); + return -EINVAL; + } + + write_lock(&kvm->mmu_lock); + __kvm_write_track_add_gfn(kvm, slot, gfn); + write_unlock(&kvm->mmu_lock); + + srcu_read_unlock(&kvm->srcu, idx); + + return 0; +} +EXPORT_SYMBOL_GPL(kvm_write_track_add_gfn); + +/* + * remove the guest page from the tracking pool which stops the interception + * of corresponding access on that page. + * + * @kvm: the guest instance we are interested in. + * @gfn: the guest page. + */ +int kvm_write_track_remove_gfn(struct kvm *kvm, gfn_t gfn) +{ + struct kvm_memory_slot *slot; + int idx; + + idx = srcu_read_lock(&kvm->srcu); + + slot = gfn_to_memslot(kvm, gfn); + if (!slot) { + srcu_read_unlock(&kvm->srcu, idx); + return -EINVAL; + } + + write_lock(&kvm->mmu_lock); + __kvm_write_track_remove_gfn(kvm, slot, gfn); + write_unlock(&kvm->mmu_lock); + + srcu_read_unlock(&kvm->srcu, idx); + + return 0; +} +EXPORT_SYMBOL_GPL(kvm_write_track_remove_gfn); +#endif diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h new file mode 100644 index 000000000000..d4d72ed999b1 --- /dev/null +++ b/arch/x86/kvm/mmu/page_track.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_PAGE_TRACK_H +#define __KVM_X86_PAGE_TRACK_H + +#include + +#include + + +bool kvm_page_track_write_tracking_enabled(struct kvm *kvm); +int kvm_page_track_write_tracking_alloc(struct kvm_memory_slot *slot); + +void kvm_page_track_free_memslot(struct kvm_memory_slot *slot); +int kvm_page_track_create_memslot(struct kvm *kvm, + struct kvm_memory_slot *slot, + unsigned long npages); + +void __kvm_write_track_add_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, + gfn_t gfn); +void __kvm_write_track_remove_gfn(struct kvm *kvm, + struct kvm_memory_slot *slot, gfn_t gfn); + +bool kvm_gfn_is_write_tracked(struct kvm *kvm, + const struct kvm_memory_slot *slot, gfn_t gfn); + +#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING +int kvm_page_track_init(struct kvm *kvm); +void kvm_page_track_cleanup(struct kvm *kvm); + +void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, const u8 *new, int bytes); +void kvm_page_track_delete_slot(struct kvm *kvm, struct kvm_memory_slot *slot); + +static inline bool kvm_page_track_has_external_user(struct kvm *kvm) +{ + return !hlist_empty(&kvm->arch.track_notifier_head.track_notifier_list); +} +#else +static inline int kvm_page_track_init(struct kvm *kvm) { return 0; } +static inline void kvm_page_track_cleanup(struct kvm *kvm) { } + +static inline void __kvm_page_track_write(struct kvm *kvm, gpa_t gpa, + const u8 *new, int bytes) { } +static inline void kvm_page_track_delete_slot(struct kvm *kvm, + struct kvm_memory_slot *slot) { } + +static inline bool kvm_page_track_has_external_user(struct kvm *kvm) { return false; } + +#endif /* CONFIG_KVM_EXTERNAL_WRITE_TRACKING */ + +static inline void kvm_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, + const u8 *new, int bytes) +{ + __kvm_page_track_write(vcpu->kvm, gpa, new, bytes); + + kvm_mmu_track_write(vcpu, gpa, new, bytes); +} + +#endif /* __KVM_X86_PAGE_TRACK_H */ diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 0662e0278e70..c85255073f67 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -338,7 +338,6 @@ retry_walk: } #endif walker->max_level = walker->level; - ASSERT(!(is_long_mode(vcpu) && !is_pae(vcpu))); /* * FIXME: on Intel processors, loads of the PDPTE registers for PAE paging @@ -348,9 +347,21 @@ retry_walk: nested_access = (have_ad ? PFERR_WRITE_MASK : 0) | PFERR_USER_MASK; pte_access = ~0; + + /* + * Queue a page fault for injection if this assertion fails, as callers + * assume that walker.fault contains sane info on a walk failure. I.e. + * avoid making the situation worse by inducing even worse badness + * between when the assertion fails and when KVM kicks the vCPU out to + * userspace (because the VM is bugged). + */ + if (KVM_BUG_ON(is_long_mode(vcpu) && !is_pae(vcpu), vcpu->kvm)) + goto error; + ++walker->level; do { + struct kvm_memory_slot *slot; unsigned long host_addr; pt_access = pte_access; @@ -381,7 +392,11 @@ retry_walk: if (unlikely(real_gpa == INVALID_GPA)) return 0; - host_addr = kvm_vcpu_gfn_to_hva_prot(vcpu, gpa_to_gfn(real_gpa), + slot = kvm_vcpu_gfn_to_memslot(vcpu, gpa_to_gfn(real_gpa)); + if (!kvm_is_visible_memslot(slot)) + goto error; + + host_addr = gfn_to_hva_memslot_prot(slot, gpa_to_gfn(real_gpa), &walker->pte_writable[walker->level - 1]); if (unlikely(kvm_is_error_hva(host_addr))) goto error; @@ -456,9 +471,6 @@ retry_walk: goto retry_walk; } - pgprintk("%s: pte %llx pte_access %x pt_access %x\n", - __func__, (u64)pte, walker->pte_access, - walker->pt_access[walker->level - 1]); return 1; error: @@ -529,8 +541,6 @@ FNAME(prefetch_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, if (FNAME(prefetch_invalid_gpte)(vcpu, sp, spte, gpte)) return false; - pgprintk("%s: gpte %llx spte %p\n", __func__, (u64)gpte, spte); - gfn = gpte_to_gfn(gpte); pte_access = sp->role.access & FNAME(gpte_access)(gpte); FNAME(protect_clean_gpte)(vcpu->arch.mmu, &pte_access, gpte); @@ -638,9 +648,20 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, if (FNAME(gpte_changed)(vcpu, gw, top_level)) goto out_gpte_changed; - if (WARN_ON(!VALID_PAGE(vcpu->arch.mmu->root.hpa))) + if (WARN_ON_ONCE(!VALID_PAGE(vcpu->arch.mmu->root.hpa))) goto out_gpte_changed; + /* + * Load a new root and retry the faulting instruction in the extremely + * unlikely scenario that the guest root gfn became visible between + * loading a dummy root and handling the resulting page fault, e.g. if + * userspace create a memslot in the interim. + */ + if (unlikely(kvm_mmu_is_dummy_root(vcpu->arch.mmu->root.hpa))) { + kvm_make_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu); + goto out_gpte_changed; + } + for_each_shadow_entry(vcpu, fault->addr, it) { gfn_t table_gfn; @@ -758,7 +779,6 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault struct guest_walker walker; int r; - pgprintk("%s: addr %lx err %x\n", __func__, fault->addr, fault->error_code); WARN_ON_ONCE(fault->is_tdp); /* @@ -773,7 +793,6 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault * The page is not mapped by the guest. Let the guest handle it. */ if (!r) { - pgprintk("%s: guest page fault\n", __func__); if (!fault->prefetch) kvm_inject_emulated_page_fault(vcpu, &walker.fault); @@ -837,7 +856,7 @@ static gpa_t FNAME(get_level1_sp_gpa)(struct kvm_mmu_page *sp) { int offset = 0; - WARN_ON(sp->role.level != PG_LEVEL_4K); + WARN_ON_ONCE(sp->role.level != PG_LEVEL_4K); if (PTTYPE == 32) offset = sp->role.quadrant << SPTE_LEVEL_BITS; diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index cf2c6426a6fc..4a599130e9c9 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -61,7 +61,7 @@ static u64 generation_mmio_spte_mask(u64 gen) { u64 mask; - WARN_ON(gen & ~MMIO_SPTE_GEN_MASK); + WARN_ON_ONCE(gen & ~MMIO_SPTE_GEN_MASK); mask = (gen << MMIO_SPTE_GEN_LOW_SHIFT) & MMIO_SPTE_GEN_LOW_MASK; mask |= (gen << MMIO_SPTE_GEN_HIGH_SHIFT) & MMIO_SPTE_GEN_HIGH_MASK; @@ -221,8 +221,6 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, * shadow pages and unsync'ing pages is not allowed. */ if (mmu_try_to_unsync_pages(vcpu->kvm, slot, gfn, can_unsync, prefetch)) { - pgprintk("%s: found shadow page for %llx, marking ro\n", - __func__, gfn); wrprot = true; pte_access &= ~ACC_WRITE_MASK; spte &= ~(PT_WRITABLE_MASK | shadow_mmu_writable_mask); @@ -242,7 +240,7 @@ out: if ((spte & PT_WRITABLE_MASK) && kvm_slot_dirty_track_enabled(slot)) { /* Enforced by kvm_mmu_hugepage_adjust. */ - WARN_ON(level > PG_LEVEL_4K); + WARN_ON_ONCE(level > PG_LEVEL_4K); mark_page_dirty_in_slot(vcpu->kvm, slot, gfn); } diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 1279db2eab44..a129951c9a88 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -3,6 +3,7 @@ #ifndef KVM_X86_MMU_SPTE_H #define KVM_X86_MMU_SPTE_H +#include "mmu.h" #include "mmu_internal.h" /* @@ -236,6 +237,18 @@ static inline struct kvm_mmu_page *sptep_to_sp(u64 *sptep) return to_shadow_page(__pa(sptep)); } +static inline struct kvm_mmu_page *root_to_sp(hpa_t root) +{ + if (kvm_mmu_is_dummy_root(root)) + return NULL; + + /* + * The "root" may be a special root, e.g. a PAE entry, treat it as a + * SPTE to ensure any non-PA bits are dropped. + */ + return spte_to_child_sp(root); +} + static inline bool is_mmio_spte(u64 spte) { return (spte & shadow_mmio_mask) == shadow_mmio_value && @@ -265,13 +278,13 @@ static inline bool sp_ad_disabled(struct kvm_mmu_page *sp) static inline bool spte_ad_enabled(u64 spte) { - MMU_WARN_ON(!is_shadow_present_pte(spte)); + KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); return (spte & SPTE_TDP_AD_MASK) != SPTE_TDP_AD_DISABLED; } static inline bool spte_ad_need_write_protect(u64 spte) { - MMU_WARN_ON(!is_shadow_present_pte(spte)); + KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); /* * This is benign for non-TDP SPTEs as SPTE_TDP_AD_ENABLED is '0', * and non-TDP SPTEs will never set these bits. Optimize for 64-bit @@ -282,13 +295,13 @@ static inline bool spte_ad_need_write_protect(u64 spte) static inline u64 spte_shadow_accessed_mask(u64 spte) { - MMU_WARN_ON(!is_shadow_present_pte(spte)); + KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); return spte_ad_enabled(spte) ? shadow_accessed_mask : 0; } static inline u64 spte_shadow_dirty_mask(u64 spte) { - MMU_WARN_ON(!is_shadow_present_pte(spte)); + KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); return spte_ad_enabled(spte) ? shadow_dirty_mask : 0; } diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c index d2eb0d4f8710..bd30ebfb2f2c 100644 --- a/arch/x86/kvm/mmu/tdp_iter.c +++ b/arch/x86/kvm/mmu/tdp_iter.c @@ -39,13 +39,14 @@ void tdp_iter_restart(struct tdp_iter *iter) void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn) { - int root_level = root->role.level; - - WARN_ON(root_level < 1); - WARN_ON(root_level > PT64_ROOT_MAX_LEVEL); + if (WARN_ON_ONCE(!root || (root->role.level < 1) || + (root->role.level > PT64_ROOT_MAX_LEVEL))) { + iter->valid = false; + return; + } iter->next_last_level_gfn = next_last_level_gfn; - iter->root_level = root_level; + iter->root_level = root->role.level; iter->min_level = min_level; iter->pt_path[iter->root_level - 1] = (tdp_ptep_t)root->spt; iter->as_id = kvm_mmu_page_as_id(root); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 512163d52194..6c63f2d1675f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -475,9 +475,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, bool is_leaf = is_present && is_last_spte(new_spte, level); bool pfn_changed = spte_to_pfn(old_spte) != spte_to_pfn(new_spte); - WARN_ON(level > PT64_ROOT_MAX_LEVEL); - WARN_ON(level < PG_LEVEL_4K); - WARN_ON(gfn & (KVM_PAGES_PER_HPAGE(level) - 1)); + WARN_ON_ONCE(level > PT64_ROOT_MAX_LEVEL); + WARN_ON_ONCE(level < PG_LEVEL_4K); + WARN_ON_ONCE(gfn & (KVM_PAGES_PER_HPAGE(level) - 1)); /* * If this warning were to trigger it would indicate that there was a @@ -522,9 +522,9 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, * impact the guest since both the former and current SPTEs * are nonpresent. */ - if (WARN_ON(!is_mmio_spte(old_spte) && - !is_mmio_spte(new_spte) && - !is_removed_spte(new_spte))) + if (WARN_ON_ONCE(!is_mmio_spte(old_spte) && + !is_mmio_spte(new_spte) && + !is_removed_spte(new_spte))) pr_err("Unexpected SPTE change! Nonpresent SPTEs\n" "should not be replaced with another,\n" "different nonpresent SPTE, unless one or both\n" @@ -661,7 +661,7 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, * should be used. If operating under the MMU lock in write mode, the * use of the removed SPTE should not be necessary. */ - WARN_ON(is_removed_spte(old_spte) || is_removed_spte(new_spte)); + WARN_ON_ONCE(is_removed_spte(old_spte) || is_removed_spte(new_spte)); old_spte = kvm_tdp_mmu_write_spte(sptep, old_spte, new_spte, level); @@ -689,7 +689,7 @@ static inline void tdp_mmu_iter_set_spte(struct kvm *kvm, struct tdp_iter *iter, else #define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end) \ - for_each_tdp_pte(_iter, to_shadow_page(_mmu->root.hpa), _start, _end) + for_each_tdp_pte(_iter, root_to_sp(_mmu->root.hpa), _start, _end) /* * Yield if the MMU lock is contended or this thread needs to return control @@ -709,7 +709,7 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm, struct tdp_iter *iter, bool flush, bool shared) { - WARN_ON(iter->yielded); + WARN_ON_ONCE(iter->yielded); /* Ensure forward progress has been made before yielding. */ if (iter->next_last_level_gfn == iter->yielded_gfn) @@ -728,7 +728,7 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm, rcu_read_lock(); - WARN_ON(iter->gfn > iter->next_last_level_gfn); + WARN_ON_ONCE(iter->gfn > iter->next_last_level_gfn); iter->yielded = true; } @@ -1241,7 +1241,7 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte; /* Huge pages aren't expected to be modified without first being zapped. */ - WARN_ON(pte_huge(range->pte) || range->start + 1 != range->end); + WARN_ON_ONCE(pte_huge(range->arg.pte) || range->start + 1 != range->end); if (iter->level != PG_LEVEL_4K || !is_shadow_present_pte(iter->old_spte)) @@ -1255,9 +1255,9 @@ static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter, */ tdp_mmu_iter_set_spte(kvm, iter, 0); - if (!pte_write(range->pte)) { + if (!pte_write(range->arg.pte)) { new_spte = kvm_mmu_changed_pte_notifier_make_spte(iter->old_spte, - pte_pfn(range->pte)); + pte_pfn(range->arg.pte)); tdp_mmu_iter_set_spte(kvm, iter, new_spte); } @@ -1548,8 +1548,8 @@ retry: if (!is_shadow_present_pte(iter.old_spte)) continue; - MMU_WARN_ON(kvm_ad_enabled() && - spte_ad_need_write_protect(iter.old_spte)); + KVM_MMU_WARN_ON(kvm_ad_enabled() && + spte_ad_need_write_protect(iter.old_spte)); if (!(iter.old_spte & dbit)) continue; @@ -1600,6 +1600,8 @@ static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root, shadow_dirty_mask; struct tdp_iter iter; + lockdep_assert_held_write(&kvm->mmu_lock); + rcu_read_lock(); tdp_root_for_each_leaf_pte(iter, root, gfn + __ffs(mask), @@ -1607,8 +1609,8 @@ static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root, if (!mask) break; - MMU_WARN_ON(kvm_ad_enabled() && - spte_ad_need_write_protect(iter.old_spte)); + KVM_MMU_WARN_ON(kvm_ad_enabled() && + spte_ad_need_write_protect(iter.old_spte)); if (iter.level > PG_LEVEL_4K || !(mask & (1UL << (iter.gfn - gfn)))) @@ -1646,7 +1648,6 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm, { struct kvm_mmu_page *root; - lockdep_assert_held_write(&kvm->mmu_lock); for_each_tdp_mmu_root(kvm, root, slot->as_id) clear_dirty_pt_masked(kvm, root, gfn, mask, wrprot); } diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index bf653df86112..edb89b51b383 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -382,9 +382,6 @@ static bool check_pmu_event_filter(struct kvm_pmc *pmc) struct kvm_x86_pmu_event_filter *filter; struct kvm *kvm = pmc->vcpu->kvm; - if (!static_call(kvm_x86_pmu_hw_event_available)(pmc)) - return false; - filter = srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu); if (!filter) return true; @@ -398,6 +395,7 @@ static bool check_pmu_event_filter(struct kvm_pmc *pmc) static bool pmc_event_is_allowed(struct kvm_pmc *pmc) { return pmc_is_globally_enabled(pmc) && pmc_speculative_in_use(pmc) && + static_call(kvm_x86_pmu_hw_event_available)(pmc) && check_pmu_event_filter(pmc); } diff --git a/arch/x86/kvm/reverse_cpuid.h b/arch/x86/kvm/reverse_cpuid.h index 56cbdb24400a..b81650678375 100644 --- a/arch/x86/kvm/reverse_cpuid.h +++ b/arch/x86/kvm/reverse_cpuid.h @@ -43,6 +43,7 @@ enum kvm_only_cpuid_leafs { /* Intel-defined sub-features, CPUID level 0x00000007:1 (EDX) */ #define X86_FEATURE_AVX_VNNI_INT8 KVM_X86_FEATURE(CPUID_7_1_EDX, 4) #define X86_FEATURE_AVX_NE_CONVERT KVM_X86_FEATURE(CPUID_7_1_EDX, 5) +#define X86_FEATURE_AMX_COMPLEX KVM_X86_FEATURE(CPUID_7_1_EDX, 8) #define X86_FEATURE_PREFETCHITI KVM_X86_FEATURE(CPUID_7_1_EDX, 14) /* CPUID level 0x80000007 (EDX). */ diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c index cfc8ab773025..2092db892d7d 100644 --- a/arch/x86/kvm/svm/avic.c +++ b/arch/x86/kvm/svm/avic.c @@ -791,6 +791,7 @@ static int svm_ir_list_add(struct vcpu_svm *svm, struct amd_iommu_pi_data *pi) int ret = 0; unsigned long flags; struct amd_svm_iommu_ir *ir; + u64 entry; /** * In some cases, the existing irte is updated and re-set, @@ -824,6 +825,18 @@ static int svm_ir_list_add(struct vcpu_svm *svm, struct amd_iommu_pi_data *pi) ir->data = pi->ir_data; spin_lock_irqsave(&svm->ir_list_lock, flags); + + /* + * Update the target pCPU for IOMMU doorbells if the vCPU is running. + * If the vCPU is NOT running, i.e. is blocking or scheduled out, KVM + * will update the pCPU info when the vCPU awkened and/or scheduled in. + * See also avic_vcpu_load(). + */ + entry = READ_ONCE(*(svm->avic_physical_id_cache)); + if (entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK) + amd_iommu_update_ga(entry & AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK, + true, pi->ir_data); + list_add(&ir->node, &svm->ir_list); spin_unlock_irqrestore(&svm->ir_list_lock, flags); out: @@ -986,10 +999,11 @@ static inline int avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r) { int ret = 0; - unsigned long flags; struct amd_svm_iommu_ir *ir; struct vcpu_svm *svm = to_svm(vcpu); + lockdep_assert_held(&svm->ir_list_lock); + if (!kvm_arch_has_assigned_device(vcpu->kvm)) return 0; @@ -997,19 +1011,15 @@ avic_update_iommu_vcpu_affinity(struct kvm_vcpu *vcpu, int cpu, bool r) * Here, we go through the per-vcpu ir_list to update all existing * interrupt remapping table entry targeting this vcpu. */ - spin_lock_irqsave(&svm->ir_list_lock, flags); - if (list_empty(&svm->ir_list)) - goto out; + return 0; list_for_each_entry(ir, &svm->ir_list, node) { ret = amd_iommu_update_ga(cpu, r, ir->data); if (ret) - break; + return ret; } -out: - spin_unlock_irqrestore(&svm->ir_list_lock, flags); - return ret; + return 0; } void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu) @@ -1017,6 +1027,7 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu) u64 entry; int h_physical_id = kvm_cpu_get_apicid(cpu); struct vcpu_svm *svm = to_svm(vcpu); + unsigned long flags; lockdep_assert_preemption_disabled(); @@ -1033,6 +1044,15 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu) if (kvm_vcpu_is_blocking(vcpu)) return; + /* + * Grab the per-vCPU interrupt remapping lock even if the VM doesn't + * _currently_ have assigned devices, as that can change. Holding + * ir_list_lock ensures that either svm_ir_list_add() will consume + * up-to-date entry information, or that this task will wait until + * svm_ir_list_add() completes to set the new target pCPU. + */ + spin_lock_irqsave(&svm->ir_list_lock, flags); + entry = READ_ONCE(*(svm->avic_physical_id_cache)); WARN_ON_ONCE(entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK); @@ -1042,25 +1062,48 @@ void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu) WRITE_ONCE(*(svm->avic_physical_id_cache), entry); avic_update_iommu_vcpu_affinity(vcpu, h_physical_id, true); + + spin_unlock_irqrestore(&svm->ir_list_lock, flags); } void avic_vcpu_put(struct kvm_vcpu *vcpu) { u64 entry; struct vcpu_svm *svm = to_svm(vcpu); + unsigned long flags; lockdep_assert_preemption_disabled(); + /* + * Note, reading the Physical ID entry outside of ir_list_lock is safe + * as only the pCPU that has loaded (or is loading) the vCPU is allowed + * to modify the entry, and preemption is disabled. I.e. the vCPU + * can't be scheduled out and thus avic_vcpu_{put,load}() can't run + * recursively. + */ entry = READ_ONCE(*(svm->avic_physical_id_cache)); /* Nothing to do if IsRunning == '0' due to vCPU blocking. */ if (!(entry & AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK)) return; + /* + * Take and hold the per-vCPU interrupt remapping lock while updating + * the Physical ID entry even though the lock doesn't protect against + * multiple writers (see above). Holding ir_list_lock ensures that + * either svm_ir_list_add() will consume up-to-date entry information, + * or that this task will wait until svm_ir_list_add() completes to + * mark the vCPU as not running. + */ + spin_lock_irqsave(&svm->ir_list_lock, flags); + avic_update_iommu_vcpu_affinity(vcpu, -1, 0); entry &= ~AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK; WRITE_ONCE(*(svm->avic_physical_id_cache), entry); + + spin_unlock_irqrestore(&svm->ir_list_lock, flags); + } void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 96936ddf1b3c..dd496c9e5f91 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -107,7 +107,7 @@ static void nested_svm_uninit_mmu_context(struct kvm_vcpu *vcpu) static bool nested_vmcb_needs_vls_intercept(struct vcpu_svm *svm) { - if (!svm->v_vmload_vmsave_enabled) + if (!guest_can_use(&svm->vcpu, X86_FEATURE_V_VMSAVE_VMLOAD)) return true; if (!nested_npt_enabled(svm)) @@ -552,6 +552,7 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm, struct vmcb *vmcb12 bool new_vmcb12 = false; struct vmcb *vmcb01 = svm->vmcb01.ptr; struct vmcb *vmcb02 = svm->nested.vmcb02.ptr; + struct kvm_vcpu *vcpu = &svm->vcpu; nested_vmcb02_compute_g_pat(svm); @@ -577,18 +578,18 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm, struct vmcb *vmcb12 vmcb_mark_dirty(vmcb02, VMCB_DT); } - kvm_set_rflags(&svm->vcpu, vmcb12->save.rflags | X86_EFLAGS_FIXED); + kvm_set_rflags(vcpu, vmcb12->save.rflags | X86_EFLAGS_FIXED); - svm_set_efer(&svm->vcpu, svm->nested.save.efer); + svm_set_efer(vcpu, svm->nested.save.efer); - svm_set_cr0(&svm->vcpu, svm->nested.save.cr0); - svm_set_cr4(&svm->vcpu, svm->nested.save.cr4); + svm_set_cr0(vcpu, svm->nested.save.cr0); + svm_set_cr4(vcpu, svm->nested.save.cr4); svm->vcpu.arch.cr2 = vmcb12->save.cr2; - kvm_rax_write(&svm->vcpu, vmcb12->save.rax); - kvm_rsp_write(&svm->vcpu, vmcb12->save.rsp); - kvm_rip_write(&svm->vcpu, vmcb12->save.rip); + kvm_rax_write(vcpu, vmcb12->save.rax); + kvm_rsp_write(vcpu, vmcb12->save.rsp); + kvm_rip_write(vcpu, vmcb12->save.rip); /* In case we don't even reach vcpu_run, the fields are not updated */ vmcb02->save.rax = vmcb12->save.rax; @@ -602,7 +603,8 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm, struct vmcb *vmcb12 vmcb_mark_dirty(vmcb02, VMCB_DR); } - if (unlikely(svm->lbrv_enabled && (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK))) { + if (unlikely(guest_can_use(vcpu, X86_FEATURE_LBRV) && + (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK))) { /* * Reserved bits of DEBUGCTL are ignored. Be consistent with * svm_set_msr's definition of reserved bits. @@ -658,7 +660,8 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, * exit_int_info, exit_int_info_err, next_rip, insn_len, insn_bytes. */ - if (svm->vgif_enabled && (svm->nested.ctl.int_ctl & V_GIF_ENABLE_MASK)) + if (guest_can_use(vcpu, X86_FEATURE_VGIF) && + (svm->nested.ctl.int_ctl & V_GIF_ENABLE_MASK)) int_ctl_vmcb12_bits |= (V_GIF_MASK | V_GIF_ENABLE_MASK); else int_ctl_vmcb01_bits |= (V_GIF_MASK | V_GIF_ENABLE_MASK); @@ -695,10 +698,9 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, vmcb02->control.tsc_offset = vcpu->arch.tsc_offset; - if (svm->tsc_ratio_msr != kvm_caps.default_tsc_scaling_ratio) { - WARN_ON(!svm->tsc_scaling_enabled); + if (guest_can_use(vcpu, X86_FEATURE_TSCRATEMSR) && + svm->tsc_ratio_msr != kvm_caps.default_tsc_scaling_ratio) nested_svm_update_tsc_ratio_msr(vcpu); - } vmcb02->control.int_ctl = (svm->nested.ctl.int_ctl & int_ctl_vmcb12_bits) | @@ -717,7 +719,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, * what a nrips=0 CPU would do (L1 is responsible for advancing RIP * prior to injecting the event). */ - if (svm->nrips_enabled) + if (guest_can_use(vcpu, X86_FEATURE_NRIPS)) vmcb02->control.next_rip = svm->nested.ctl.next_rip; else if (boot_cpu_has(X86_FEATURE_NRIPS)) vmcb02->control.next_rip = vmcb12_rip; @@ -727,7 +729,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, svm->soft_int_injected = true; svm->soft_int_csbase = vmcb12_csbase; svm->soft_int_old_rip = vmcb12_rip; - if (svm->nrips_enabled) + if (guest_can_use(vcpu, X86_FEATURE_NRIPS)) svm->soft_int_next_rip = svm->nested.ctl.next_rip; else svm->soft_int_next_rip = vmcb12_rip; @@ -735,15 +737,21 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, vmcb02->control.virt_ext = vmcb01->control.virt_ext & LBR_CTL_ENABLE_MASK; - if (svm->lbrv_enabled) + if (guest_can_use(vcpu, X86_FEATURE_LBRV)) vmcb02->control.virt_ext |= (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK); if (!nested_vmcb_needs_vls_intercept(svm)) vmcb02->control.virt_ext |= VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK; - pause_count12 = svm->pause_filter_enabled ? svm->nested.ctl.pause_filter_count : 0; - pause_thresh12 = svm->pause_threshold_enabled ? svm->nested.ctl.pause_filter_thresh : 0; + if (guest_can_use(vcpu, X86_FEATURE_PAUSEFILTER)) + pause_count12 = svm->nested.ctl.pause_filter_count; + else + pause_count12 = 0; + if (guest_can_use(vcpu, X86_FEATURE_PFTHRESHOLD)) + pause_thresh12 = svm->nested.ctl.pause_filter_thresh; + else + pause_thresh12 = 0; if (kvm_pause_in_guest(svm->vcpu.kvm)) { /* use guest values since host doesn't intercept PAUSE */ vmcb02->control.pause_filter_count = pause_count12; @@ -1027,7 +1035,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm) if (vmcb12->control.exit_code != SVM_EXIT_ERR) nested_save_pending_event_to_vmcb12(svm, vmcb12); - if (svm->nrips_enabled) + if (guest_can_use(vcpu, X86_FEATURE_NRIPS)) vmcb12->control.next_rip = vmcb02->control.next_rip; vmcb12->control.int_ctl = svm->nested.ctl.int_ctl; @@ -1066,7 +1074,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm) if (!nested_exit_on_intr(svm)) kvm_make_request(KVM_REQ_EVENT, &svm->vcpu); - if (unlikely(svm->lbrv_enabled && (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK))) { + if (unlikely(guest_can_use(vcpu, X86_FEATURE_LBRV) && + (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK))) { svm_copy_lbrs(vmcb12, vmcb02); svm_update_lbrv(vcpu); } else if (unlikely(vmcb01->control.virt_ext & LBR_CTL_ENABLE_MASK)) { @@ -1101,10 +1110,10 @@ int nested_svm_vmexit(struct vcpu_svm *svm) vmcb_mark_dirty(vmcb01, VMCB_INTERCEPTS); } - if (svm->tsc_ratio_msr != kvm_caps.default_tsc_scaling_ratio) { - WARN_ON(!svm->tsc_scaling_enabled); + if (kvm_caps.has_tsc_control && + vcpu->arch.tsc_scaling_ratio != vcpu->arch.l1_tsc_scaling_ratio) { vcpu->arch.tsc_scaling_ratio = vcpu->arch.l1_tsc_scaling_ratio; - __svm_write_tsc_multiplier(vcpu->arch.tsc_scaling_ratio); + svm_write_tsc_multiplier(vcpu); } svm->nested.ctl.nested_cr3 = 0; @@ -1537,7 +1546,7 @@ void nested_svm_update_tsc_ratio_msr(struct kvm_vcpu *vcpu) vcpu->arch.tsc_scaling_ratio = kvm_calc_nested_tsc_multiplier(vcpu->arch.l1_tsc_scaling_ratio, svm->tsc_ratio_msr); - __svm_write_tsc_multiplier(vcpu->arch.tsc_scaling_ratio); + svm_write_tsc_multiplier(vcpu); } /* Inverse operation of nested_copy_vmcb_control_to_cache(). asid is copied too. */ diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index d3aec1f2cad2..b9a0a939d59f 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "mmu.h" #include "x86.h" @@ -54,9 +55,14 @@ module_param_named(sev, sev_enabled, bool, 0444); /* enable/disable SEV-ES support */ static bool sev_es_enabled = true; module_param_named(sev_es, sev_es_enabled, bool, 0444); + +/* enable/disable SEV-ES DebugSwap support */ +static bool sev_es_debug_swap_enabled = true; +module_param_named(debug_swap, sev_es_debug_swap_enabled, bool, 0444); #else #define sev_enabled false #define sev_es_enabled false +#define sev_es_debug_swap_enabled false #endif /* CONFIG_KVM_AMD_SEV */ static u8 sev_enc_bit; @@ -606,6 +612,9 @@ static int sev_es_sync_vmsa(struct vcpu_svm *svm) save->xss = svm->vcpu.arch.ia32_xss; save->dr6 = svm->vcpu.arch.dr6; + if (sev_es_debug_swap_enabled) + save->sev_features |= SVM_SEV_FEAT_DEBUG_SWAP; + pr_debug("Virtual Machine Save Area (VMSA):\n"); print_hex_dump_debug("", DUMP_PREFIX_NONE, 16, 1, save, sizeof(*save), false); @@ -619,6 +628,11 @@ static int __sev_launch_update_vmsa(struct kvm *kvm, struct kvm_vcpu *vcpu, struct vcpu_svm *svm = to_svm(vcpu); int ret; + if (vcpu->guest_debug) { + pr_warn_once("KVM_SET_GUEST_DEBUG for SEV-ES guest is not supported"); + return -EINVAL; + } + /* Perform some pre-encryption checks against the VMSA */ ret = sev_es_sync_vmsa(svm); if (ret) @@ -1725,7 +1739,7 @@ static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm) * Note, the source is not required to have the same number of * vCPUs as the destination when migrating a vanilla SEV VM. */ - src_vcpu = kvm_get_vcpu(dst_kvm, i); + src_vcpu = kvm_get_vcpu(src_kvm, i); src_svm = to_svm(src_vcpu); /* @@ -2171,7 +2185,7 @@ void __init sev_hardware_setup(void) bool sev_es_supported = false; bool sev_supported = false; - if (!sev_enabled || !npt_enabled) + if (!sev_enabled || !npt_enabled || !nrips) goto out; /* @@ -2256,6 +2270,9 @@ out: sev_enabled = sev_supported; sev_es_enabled = sev_es_supported; + if (!sev_es_enabled || !cpu_feature_enabled(X86_FEATURE_DEBUG_SWAP) || + !cpu_feature_enabled(X86_FEATURE_NO_NESTED_DATA_BP)) + sev_es_debug_swap_enabled = false; #endif } @@ -2881,7 +2898,10 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu) svm->sev_es.ghcb_sa); break; case SVM_VMGEXIT_NMI_COMPLETE: - ret = svm_invoke_exit_handler(vcpu, SVM_EXIT_IRET); + ++vcpu->stat.nmi_window_exits; + svm->nmi_masked = false; + kvm_make_request(KVM_REQ_EVENT, vcpu); + ret = 1; break; case SVM_VMGEXIT_AP_HLT_LOOP: ret = kvm_emulate_ap_reset_hold(vcpu); @@ -2944,6 +2964,7 @@ int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in) static void sev_es_init_vmcb(struct vcpu_svm *svm) { + struct vmcb *vmcb = svm->vmcb01.ptr; struct kvm_vcpu *vcpu = &svm->vcpu; svm->vmcb->control.nested_ctl |= SVM_NESTED_CTL_SEV_ES_ENABLE; @@ -2952,9 +2973,12 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm) /* * An SEV-ES guest requires a VMSA area that is a separate from the * VMCB page. Do not include the encryption mask on the VMSA physical - * address since hardware will access it using the guest key. + * address since hardware will access it using the guest key. Note, + * the VMSA will be NULL if this vCPU is the destination for intrahost + * migration, and will be copied later. */ - svm->vmcb->control.vmsa_pa = __pa(svm->sev_es.vmsa); + if (svm->sev_es.vmsa) + svm->vmcb->control.vmsa_pa = __pa(svm->sev_es.vmsa); /* Can't intercept CR register access, HV can't modify CR registers */ svm_clr_intercept(svm, INTERCEPT_CR0_READ); @@ -2972,8 +2996,23 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm) svm_set_intercept(svm, TRAP_CR4_WRITE); svm_set_intercept(svm, TRAP_CR8_WRITE); - /* No support for enable_vmware_backdoor */ - clr_exception_intercept(svm, GP_VECTOR); + vmcb->control.intercepts[INTERCEPT_DR] = 0; + if (!sev_es_debug_swap_enabled) { + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_WRITE); + recalc_intercepts(svm); + } else { + /* + * Disable #DB intercept iff DebugSwap is enabled. KVM doesn't + * allow debugging SEV-ES guests, and enables DebugSwap iff + * NO_NESTED_DATA_BP is supported, so there's no reason to + * intercept #DB when DebugSwap is enabled. For simplicity + * with respect to guest debug, intercept #DB for other VMs + * even if NO_NESTED_DATA_BP is supported, i.e. even if the + * guest can't DoS the CPU with infinite #DB vectoring. + */ + clr_exception_intercept(svm, DB_VECTOR); + } /* Can't intercept XSETBV, HV can't modify XCR0 directly */ svm_clr_intercept(svm, INTERCEPT_XSETBV); @@ -3000,6 +3039,12 @@ void sev_init_vmcb(struct vcpu_svm *svm) svm->vmcb->control.nested_ctl |= SVM_NESTED_CTL_SEV_ENABLE; clr_exception_intercept(svm, UD_VECTOR); + /* + * Don't intercept #GP for SEV guests, e.g. for the VMware backdoor, as + * KVM can't decrypt guest memory to decode the faulting instruction. + */ + clr_exception_intercept(svm, GP_VECTOR); + if (sev_es_guest(svm->vcpu.kvm)) sev_es_init_vmcb(svm); } @@ -3018,20 +3063,41 @@ void sev_es_vcpu_reset(struct vcpu_svm *svm) void sev_es_prepare_switch_to_guest(struct sev_es_save_area *hostsa) { /* - * As an SEV-ES guest, hardware will restore the host state on VMEXIT, - * of which one step is to perform a VMLOAD. KVM performs the - * corresponding VMSAVE in svm_prepare_guest_switch for both - * traditional and SEV-ES guests. + * All host state for SEV-ES guests is categorized into three swap types + * based on how it is handled by hardware during a world switch: + * + * A: VMRUN: Host state saved in host save area + * VMEXIT: Host state loaded from host save area + * + * B: VMRUN: Host state _NOT_ saved in host save area + * VMEXIT: Host state loaded from host save area + * + * C: VMRUN: Host state _NOT_ saved in host save area + * VMEXIT: Host state initialized to default(reset) values + * + * Manually save type-B state, i.e. state that is loaded by VMEXIT but + * isn't saved by VMRUN, that isn't already saved by VMSAVE (performed + * by common SVM code). */ - - /* XCR0 is restored on VMEXIT, save the current host value */ hostsa->xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK); - - /* PKRU is restored on VMEXIT, save the current host value */ hostsa->pkru = read_pkru(); - - /* MSR_IA32_XSS is restored on VMEXIT, save the currnet host value */ hostsa->xss = host_xss; + + /* + * If DebugSwap is enabled, debug registers are loaded but NOT saved by + * the CPU (Type-B). If DebugSwap is disabled/unsupported, the CPU both + * saves and loads debug registers (Type-A). + */ + if (sev_es_debug_swap_enabled) { + hostsa->dr0 = native_get_debugreg(0); + hostsa->dr1 = native_get_debugreg(1); + hostsa->dr2 = native_get_debugreg(2); + hostsa->dr3 = native_get_debugreg(3); + hostsa->dr0_addr_mask = amd_get_dr_addr_mask(0); + hostsa->dr1_addr_mask = amd_get_dr_addr_mask(1); + hostsa->dr2_addr_mask = amd_get_dr_addr_mask(2); + hostsa->dr3_addr_mask = amd_get_dr_addr_mask(3); + } } void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index d4bfdc607fe7..f283eb47f6ac 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -39,10 +39,9 @@ #include #include #include +#include #include -#include - #include #include "trace.h" @@ -203,7 +202,7 @@ static int nested = true; module_param(nested, int, S_IRUGO); /* enable/disable Next RIP Save */ -static int nrips = true; +int nrips = true; module_param(nrips, int, 0444); /* enable/disable Virtual VMLOAD VMSAVE */ @@ -365,6 +364,8 @@ static void svm_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask) svm->vmcb->control.int_state |= SVM_INTERRUPT_SHADOW_MASK; } +static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type, + void *insn, int insn_len); static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu, bool commit_side_effects) @@ -385,6 +386,14 @@ static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu, } if (!svm->next_rip) { + /* + * FIXME: Drop this when kvm_emulate_instruction() does the + * right thing and treats "can't emulate" as outright failure + * for EMULTYPE_SKIP. + */ + if (!svm_can_emulate_instruction(vcpu, EMULTYPE_SKIP, NULL, 0)) + return 0; + if (unlikely(!commit_side_effects)) old_rflags = svm->vmcb->save.rflags; @@ -517,14 +526,21 @@ static void svm_init_osvw(struct kvm_vcpu *vcpu) vcpu->arch.osvw.status |= 1; } -static bool kvm_is_svm_supported(void) +static bool __kvm_is_svm_supported(void) { - int cpu = raw_smp_processor_id(); - const char *msg; + int cpu = smp_processor_id(); + struct cpuinfo_x86 *c = &cpu_data(cpu); + u64 vm_cr; - if (!cpu_has_svm(&msg)) { - pr_err("SVM not supported by CPU %d, %s\n", cpu, msg); + if (c->x86_vendor != X86_VENDOR_AMD && + c->x86_vendor != X86_VENDOR_HYGON) { + pr_err("CPU %d isn't AMD or Hygon\n", cpu); + return false; + } + + if (!cpu_has(c, X86_FEATURE_SVM)) { + pr_err("SVM not supported by CPU %d\n", cpu); return false; } @@ -542,25 +558,55 @@ static bool kvm_is_svm_supported(void) return true; } +static bool kvm_is_svm_supported(void) +{ + bool supported; + + migrate_disable(); + supported = __kvm_is_svm_supported(); + migrate_enable(); + + return supported; +} + static int svm_check_processor_compat(void) { - if (!kvm_is_svm_supported()) + if (!__kvm_is_svm_supported()) return -EIO; return 0; } -void __svm_write_tsc_multiplier(u64 multiplier) +static void __svm_write_tsc_multiplier(u64 multiplier) { - preempt_disable(); - if (multiplier == __this_cpu_read(current_tsc_ratio)) - goto out; + return; wrmsrl(MSR_AMD64_TSC_RATIO, multiplier); __this_cpu_write(current_tsc_ratio, multiplier); -out: - preempt_enable(); +} + +static inline void kvm_cpu_svm_disable(void) +{ + uint64_t efer; + + wrmsrl(MSR_VM_HSAVE_PA, 0); + rdmsrl(MSR_EFER, efer); + if (efer & EFER_SVME) { + /* + * Force GIF=1 prior to disabling SVM, e.g. to ensure INIT and + * NMI aren't blocked. + */ + stgi(); + wrmsrl(MSR_EFER, efer & ~EFER_SVME); + } +} + +static void svm_emergency_disable(void) +{ + kvm_rebooting = true; + + kvm_cpu_svm_disable(); } static void svm_hardware_disable(void) @@ -569,7 +615,7 @@ static void svm_hardware_disable(void) if (tsc_scaling) __svm_write_tsc_multiplier(SVM_TSC_RATIO_DEFAULT); - cpu_svm_disable(); + kvm_cpu_svm_disable(); amd_pmu_disable_virt(); } @@ -677,6 +723,39 @@ free_save_area: } +static void set_dr_intercepts(struct vcpu_svm *svm) +{ + struct vmcb *vmcb = svm->vmcb01.ptr; + + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR0_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR1_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR2_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR3_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR4_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR5_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR6_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR0_WRITE); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR1_WRITE); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR2_WRITE); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR3_WRITE); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR4_WRITE); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR5_WRITE); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR6_WRITE); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_READ); + vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_WRITE); + + recalc_intercepts(svm); +} + +static void clr_dr_intercepts(struct vcpu_svm *svm) +{ + struct vmcb *vmcb = svm->vmcb01.ptr; + + vmcb->control.intercepts[INTERCEPT_DR] = 0; + + recalc_intercepts(svm); +} + static int direct_access_msr_slot(u32 msr) { u32 i; @@ -947,50 +1026,24 @@ static void svm_disable_lbrv(struct kvm_vcpu *vcpu) svm_copy_lbrs(svm->vmcb01.ptr, svm->vmcb); } -static int svm_get_lbr_msr(struct vcpu_svm *svm, u32 index) +static struct vmcb *svm_get_lbr_vmcb(struct vcpu_svm *svm) { /* - * If the LBR virtualization is disabled, the LBR msrs are always - * kept in the vmcb01 to avoid copying them on nested guest entries. - * - * If nested, and the LBR virtualization is enabled/disabled, the msrs - * are moved between the vmcb01 and vmcb02 as needed. + * If LBR virtualization is disabled, the LBR MSRs are always kept in + * vmcb01. If LBR virtualization is enabled and L1 is running VMs of + * its own, the MSRs are moved between vmcb01 and vmcb02 as needed. */ - struct vmcb *vmcb = - (svm->vmcb->control.virt_ext & LBR_CTL_ENABLE_MASK) ? - svm->vmcb : svm->vmcb01.ptr; - - switch (index) { - case MSR_IA32_DEBUGCTLMSR: - return vmcb->save.dbgctl; - case MSR_IA32_LASTBRANCHFROMIP: - return vmcb->save.br_from; - case MSR_IA32_LASTBRANCHTOIP: - return vmcb->save.br_to; - case MSR_IA32_LASTINTFROMIP: - return vmcb->save.last_excp_from; - case MSR_IA32_LASTINTTOIP: - return vmcb->save.last_excp_to; - default: - KVM_BUG(false, svm->vcpu.kvm, - "%s: Unknown MSR 0x%x", __func__, index); - return 0; - } + return svm->vmcb->control.virt_ext & LBR_CTL_ENABLE_MASK ? svm->vmcb : + svm->vmcb01.ptr; } void svm_update_lbrv(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); - - bool enable_lbrv = svm_get_lbr_msr(svm, MSR_IA32_DEBUGCTLMSR) & - DEBUGCTLMSR_LBR; - - bool current_enable_lbrv = !!(svm->vmcb->control.virt_ext & - LBR_CTL_ENABLE_MASK); - - if (unlikely(is_guest_mode(vcpu) && svm->lbrv_enabled)) - if (unlikely(svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK)) - enable_lbrv = true; + bool current_enable_lbrv = svm->vmcb->control.virt_ext & LBR_CTL_ENABLE_MASK; + bool enable_lbrv = (svm_get_lbr_vmcb(svm)->save.dbgctl & DEBUGCTLMSR_LBR) || + (is_guest_mode(vcpu) && guest_can_use(vcpu, X86_FEATURE_LBRV) && + (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK)); if (enable_lbrv == current_enable_lbrv) return; @@ -1101,21 +1154,23 @@ static u64 svm_get_l2_tsc_multiplier(struct kvm_vcpu *vcpu) return svm->tsc_ratio_msr; } -static void svm_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset) +static void svm_write_tsc_offset(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); svm->vmcb01.ptr->control.tsc_offset = vcpu->arch.l1_tsc_offset; - svm->vmcb->control.tsc_offset = offset; + svm->vmcb->control.tsc_offset = vcpu->arch.tsc_offset; vmcb_mark_dirty(svm->vmcb, VMCB_INTERCEPTS); } -static void svm_write_tsc_multiplier(struct kvm_vcpu *vcpu, u64 multiplier) +void svm_write_tsc_multiplier(struct kvm_vcpu *vcpu) { - __svm_write_tsc_multiplier(multiplier); + preempt_disable(); + if (to_svm(vcpu)->guest_state_loaded) + __svm_write_tsc_multiplier(vcpu->arch.tsc_scaling_ratio); + preempt_enable(); } - /* Evaluate instruction intercepts that depend on guest CPUID features. */ static void svm_recalc_instruction_intercepts(struct kvm_vcpu *vcpu, struct vcpu_svm *svm) @@ -1156,8 +1211,6 @@ static inline void init_vmcb_after_set_cpuid(struct kvm_vcpu *vcpu) set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_EIP, 0, 0); set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_ESP, 0, 0); - - svm->v_vmload_vmsave_enabled = false; } else { /* * If hardware supports Virtual VMLOAD VMSAVE then enable it @@ -1201,10 +1254,9 @@ static void init_vmcb(struct kvm_vcpu *vcpu) * Guest access to VMware backdoor ports could legitimately * trigger #GP because of TSS I/O permission bitmap. * We intercept those #GP and allow access to them anyway - * as VMware does. Don't intercept #GP for SEV guests as KVM can't - * decrypt guest memory to decode the faulting instruction. + * as VMware does. */ - if (enable_vmware_backdoor && !sev_guest(vcpu->kvm)) + if (enable_vmware_backdoor) set_exception_intercept(svm, GP_VECTOR); svm_set_intercept(svm, INTERCEPT_INTR); @@ -1949,7 +2001,7 @@ static void svm_sync_dirty_debug_regs(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); - if (vcpu->arch.guest_state_protected) + if (WARN_ON_ONCE(sev_es_guest(vcpu->kvm))) return; get_debugreg(vcpu->arch.db[0], 0); @@ -2510,12 +2562,13 @@ static int iret_interception(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); + WARN_ON_ONCE(sev_es_guest(vcpu->kvm)); + ++vcpu->stat.nmi_window_exits; svm->awaiting_iret_completion = true; svm_clr_iret_intercept(svm); - if (!sev_es_guest(vcpu->kvm)) - svm->nmi_iret_rip = kvm_rip_read(vcpu); + svm->nmi_iret_rip = kvm_rip_read(vcpu); kvm_make_request(KVM_REQ_EVENT, vcpu); return 1; @@ -2680,6 +2733,13 @@ static int dr_interception(struct kvm_vcpu *vcpu) unsigned long val; int err = 0; + /* + * SEV-ES intercepts DR7 only to disable guest debugging and the guest issues a VMGEXIT + * for DR7 write only. KVM cannot change DR7 (always swapped as type 'A') so return early. + */ + if (sev_es_guest(vcpu->kvm)) + return 1; + if (vcpu->guest_debug == 0) { /* * No more DR vmexits; force a reload of the debug registers @@ -2764,7 +2824,8 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) switch (msr_info->index) { case MSR_AMD64_TSC_RATIO: - if (!msr_info->host_initiated && !svm->tsc_scaling_enabled) + if (!msr_info->host_initiated && + !guest_can_use(vcpu, X86_FEATURE_TSCRATEMSR)) return 1; msr_info->data = svm->tsc_ratio_msr; break; @@ -2802,11 +2863,19 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) msr_info->data = svm->tsc_aux; break; case MSR_IA32_DEBUGCTLMSR: + msr_info->data = svm_get_lbr_vmcb(svm)->save.dbgctl; + break; case MSR_IA32_LASTBRANCHFROMIP: + msr_info->data = svm_get_lbr_vmcb(svm)->save.br_from; + break; case MSR_IA32_LASTBRANCHTOIP: + msr_info->data = svm_get_lbr_vmcb(svm)->save.br_to; + break; case MSR_IA32_LASTINTFROMIP: + msr_info->data = svm_get_lbr_vmcb(svm)->save.last_excp_from; + break; case MSR_IA32_LASTINTTOIP: - msr_info->data = svm_get_lbr_msr(svm, msr_info->index); + msr_info->data = svm_get_lbr_vmcb(svm)->save.last_excp_to; break; case MSR_VM_HSAVE_PA: msr_info->data = svm->nested.hsave_msr; @@ -2906,7 +2975,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) switch (ecx) { case MSR_AMD64_TSC_RATIO: - if (!svm->tsc_scaling_enabled) { + if (!guest_can_use(vcpu, X86_FEATURE_TSCRATEMSR)) { if (!msr->host_initiated) return 1; @@ -2928,7 +2997,8 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) svm->tsc_ratio_msr = data; - if (svm->tsc_scaling_enabled && is_guest_mode(vcpu)) + if (guest_can_use(vcpu, X86_FEATURE_TSCRATEMSR) && + is_guest_mode(vcpu)) nested_svm_update_tsc_ratio_msr(vcpu); break; @@ -3037,13 +3107,8 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) if (data & DEBUGCTL_RESERVED_BITS) return 1; - if (svm->vmcb->control.virt_ext & LBR_CTL_ENABLE_MASK) - svm->vmcb->save.dbgctl = data; - else - svm->vmcb01.ptr->save.dbgctl = data; - + svm_get_lbr_vmcb(svm)->save.dbgctl = data; svm_update_lbrv(vcpu); - break; case MSR_VM_HSAVE_PA: /* @@ -3769,6 +3834,19 @@ static void svm_enable_nmi_window(struct kvm_vcpu *vcpu) if (svm_get_nmi_mask(vcpu) && !svm->awaiting_iret_completion) return; /* IRET will cause a vm exit */ + /* + * SEV-ES guests are responsible for signaling when a vCPU is ready to + * receive a new NMI, as SEV-ES guests can't be single-stepped, i.e. + * KVM can't intercept and single-step IRET to detect when NMIs are + * unblocked (architecturally speaking). See SVM_VMGEXIT_NMI_COMPLETE. + * + * Note, GIF is guaranteed to be '1' for SEV-ES guests as hardware + * ignores SEV-ES guest writes to EFER.SVME *and* CLGI/STGI are not + * supported NAEs in the GHCB protocol. + */ + if (sev_es_guest(vcpu->kvm)) + return; + if (!gif_set(svm)) { if (vgif) svm_set_intercept(svm, INTERCEPT_STGI); @@ -3918,12 +3996,11 @@ static void svm_complete_interrupts(struct kvm_vcpu *vcpu) svm->soft_int_injected = false; /* - * If we've made progress since setting HF_IRET_MASK, we've + * If we've made progress since setting awaiting_iret_completion, we've * executed an IRET and can allow NMI injection. */ if (svm->awaiting_iret_completion && - (sev_es_guest(vcpu->kvm) || - kvm_rip_read(vcpu) != svm->nmi_iret_rip)) { + kvm_rip_read(vcpu) != svm->nmi_iret_rip) { svm->awaiting_iret_completion = false; svm->nmi_masked = false; kvm_make_request(KVM_REQ_EVENT, vcpu); @@ -4209,28 +4286,37 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) struct vcpu_svm *svm = to_svm(vcpu); struct kvm_cpuid_entry2 *best; - vcpu->arch.xsaves_enabled = guest_cpuid_has(vcpu, X86_FEATURE_XSAVE) && - boot_cpu_has(X86_FEATURE_XSAVE) && - boot_cpu_has(X86_FEATURE_XSAVES); + /* + * SVM doesn't provide a way to disable just XSAVES in the guest, KVM + * can only disable all variants of by disallowing CR4.OSXSAVE from + * being set. As a result, if the host has XSAVE and XSAVES, and the + * guest has XSAVE enabled, the guest can execute XSAVES without + * faulting. Treat XSAVES as enabled in this case regardless of + * whether it's advertised to the guest so that KVM context switches + * XSS on VM-Enter/VM-Exit. Failure to do so would effectively give + * the guest read/write access to the host's XSS. + */ + if (boot_cpu_has(X86_FEATURE_XSAVE) && + boot_cpu_has(X86_FEATURE_XSAVES) && + guest_cpuid_has(vcpu, X86_FEATURE_XSAVE)) + kvm_governed_feature_set(vcpu, X86_FEATURE_XSAVES); - /* Update nrips enabled cache */ - svm->nrips_enabled = kvm_cpu_cap_has(X86_FEATURE_NRIPS) && - guest_cpuid_has(vcpu, X86_FEATURE_NRIPS); + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_NRIPS); + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_TSCRATEMSR); + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_LBRV); - svm->tsc_scaling_enabled = tsc_scaling && guest_cpuid_has(vcpu, X86_FEATURE_TSCRATEMSR); - svm->lbrv_enabled = lbrv && guest_cpuid_has(vcpu, X86_FEATURE_LBRV); + /* + * Intercept VMLOAD if the vCPU mode is Intel in order to emulate that + * VMLOAD drops bits 63:32 of SYSENTER (ignoring the fact that exposing + * SVM on Intel is bonkers and extremely unlikely to work). + */ + if (!guest_cpuid_is_intel(vcpu)) + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_V_VMSAVE_VMLOAD); - svm->v_vmload_vmsave_enabled = vls && guest_cpuid_has(vcpu, X86_FEATURE_V_VMSAVE_VMLOAD); - - svm->pause_filter_enabled = kvm_cpu_cap_has(X86_FEATURE_PAUSEFILTER) && - guest_cpuid_has(vcpu, X86_FEATURE_PAUSEFILTER); - - svm->pause_threshold_enabled = kvm_cpu_cap_has(X86_FEATURE_PFTHRESHOLD) && - guest_cpuid_has(vcpu, X86_FEATURE_PFTHRESHOLD); - - svm->vgif_enabled = vgif && guest_cpuid_has(vcpu, X86_FEATURE_VGIF); - - svm->vnmi_enabled = vnmi && guest_cpuid_has(vcpu, X86_FEATURE_VNMI); + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_PAUSEFILTER); + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_PFTHRESHOLD); + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_VGIF); + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_VNMI); svm_recalc_instruction_intercepts(vcpu, svm); @@ -4651,16 +4737,25 @@ static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type, * and cannot be decrypted by KVM, i.e. KVM would read cyphertext and * decode garbage. * - * Inject #UD if KVM reached this point without an instruction buffer. - * In practice, this path should never be hit by a well-behaved guest, - * e.g. KVM doesn't intercept #UD or #GP for SEV guests, but this path - * is still theoretically reachable, e.g. via unaccelerated fault-like - * AVIC access, and needs to be handled by KVM to avoid putting the - * guest into an infinite loop. Injecting #UD is somewhat arbitrary, - * but its the least awful option given lack of insight into the guest. + * If KVM is NOT trying to simply skip an instruction, inject #UD if + * KVM reached this point without an instruction buffer. In practice, + * this path should never be hit by a well-behaved guest, e.g. KVM + * doesn't intercept #UD or #GP for SEV guests, but this path is still + * theoretically reachable, e.g. via unaccelerated fault-like AVIC + * access, and needs to be handled by KVM to avoid putting the guest + * into an infinite loop. Injecting #UD is somewhat arbitrary, but + * its the least awful option given lack of insight into the guest. + * + * If KVM is trying to skip an instruction, simply resume the guest. + * If a #NPF occurs while the guest is vectoring an INT3/INTO, then KVM + * will attempt to re-inject the INT3/INTO and skip the instruction. + * In that scenario, retrying the INT3/INTO and hoping the guest will + * make forward progress is the only option that has a chance of + * success (and in practice it will work the vast majority of the time). */ if (unlikely(!insn)) { - kvm_queue_exception(vcpu, UD_VECTOR); + if (!(emul_type & EMULTYPE_SKIP)) + kvm_queue_exception(vcpu, UD_VECTOR); return false; } @@ -5112,9 +5207,11 @@ static __init int svm_hardware_setup(void) svm_adjust_mmio_mask(); + nrips = nrips && boot_cpu_has(X86_FEATURE_NRIPS); + /* * Note, SEV setup consumes npt_enabled and enable_mmio_caching (which - * may be modified by svm_adjust_mmio_mask()). + * may be modified by svm_adjust_mmio_mask()), as well as nrips. */ sev_hardware_setup(); @@ -5126,11 +5223,6 @@ static __init int svm_hardware_setup(void) goto err; } - if (nrips) { - if (!boot_cpu_has(X86_FEATURE_NRIPS)) - nrips = false; - } - enable_apicv = avic = avic && avic_hardware_setup(); if (!enable_apicv) { @@ -5213,6 +5305,13 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = { .pmu_ops = &amd_pmu_ops, }; +static void __svm_exit(void) +{ + kvm_x86_vendor_exit(); + + cpu_emergency_unregister_virt_callback(svm_emergency_disable); +} + static int __init svm_init(void) { int r; @@ -5226,6 +5325,8 @@ static int __init svm_init(void) if (r) return r; + cpu_emergency_register_virt_callback(svm_emergency_disable); + /* * Common KVM initialization _must_ come last, after this, /dev/kvm is * exposed to userspace! @@ -5238,14 +5339,14 @@ static int __init svm_init(void) return 0; err_kvm_init: - kvm_x86_vendor_exit(); + __svm_exit(); return r; } static void __exit svm_exit(void) { kvm_exit(); - kvm_x86_vendor_exit(); + __svm_exit(); } module_init(svm_init) diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 8239c8de45ac..f41253958357 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -22,6 +22,7 @@ #include #include +#include "cpuid.h" #include "kvm_cache_regs.h" #define __sme_page_pa(x) __sme_set(page_to_pfn(x) << PAGE_SHIFT) @@ -33,6 +34,7 @@ #define MSRPM_OFFSETS 32 extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly; extern bool npt_enabled; +extern int nrips; extern int vgif; extern bool intercept_smi; extern bool x2avic_enabled; @@ -260,16 +262,6 @@ struct vcpu_svm { unsigned long soft_int_next_rip; bool soft_int_injected; - /* optional nested SVM features that are enabled for this guest */ - bool nrips_enabled : 1; - bool tsc_scaling_enabled : 1; - bool v_vmload_vmsave_enabled : 1; - bool lbrv_enabled : 1; - bool pause_filter_enabled : 1; - bool pause_threshold_enabled : 1; - bool vgif_enabled : 1; - bool vnmi_enabled : 1; - u32 ldr_reg; u32 dfr_reg; struct page *avic_backing_page; @@ -406,48 +398,6 @@ static inline bool vmcb12_is_intercept(struct vmcb_ctrl_area_cached *control, u3 return test_bit(bit, (unsigned long *)&control->intercepts); } -static inline void set_dr_intercepts(struct vcpu_svm *svm) -{ - struct vmcb *vmcb = svm->vmcb01.ptr; - - if (!sev_es_guest(svm->vcpu.kvm)) { - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR0_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR1_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR2_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR3_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR4_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR5_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR6_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR0_WRITE); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR1_WRITE); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR2_WRITE); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR3_WRITE); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR4_WRITE); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR5_WRITE); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR6_WRITE); - } - - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_WRITE); - - recalc_intercepts(svm); -} - -static inline void clr_dr_intercepts(struct vcpu_svm *svm) -{ - struct vmcb *vmcb = svm->vmcb01.ptr; - - vmcb->control.intercepts[INTERCEPT_DR] = 0; - - /* DR7 access must remain intercepted for an SEV-ES guest */ - if (sev_es_guest(svm->vcpu.kvm)) { - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_READ); - vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_WRITE); - } - - recalc_intercepts(svm); -} - static inline void set_exception_intercept(struct vcpu_svm *svm, u32 bit) { struct vmcb *vmcb = svm->vmcb01.ptr; @@ -493,7 +443,8 @@ static inline bool svm_is_intercept(struct vcpu_svm *svm, int bit) static inline bool nested_vgif_enabled(struct vcpu_svm *svm) { - return svm->vgif_enabled && (svm->nested.ctl.int_ctl & V_GIF_ENABLE_MASK); + return guest_can_use(&svm->vcpu, X86_FEATURE_VGIF) && + (svm->nested.ctl.int_ctl & V_GIF_ENABLE_MASK); } static inline struct vmcb *get_vgif_vmcb(struct vcpu_svm *svm) @@ -544,7 +495,7 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm) static inline bool nested_vnmi_enabled(struct vcpu_svm *svm) { - return svm->vnmi_enabled && + return guest_can_use(&svm->vcpu, X86_FEATURE_VNMI) && (svm->nested.ctl.int_ctl & V_NMI_ENABLE_MASK); } @@ -660,7 +611,7 @@ int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr, bool has_error_code, u32 error_code); int nested_svm_exit_special(struct vcpu_svm *svm); void nested_svm_update_tsc_ratio_msr(struct kvm_vcpu *vcpu); -void __svm_write_tsc_multiplier(u64 multiplier); +void svm_write_tsc_multiplier(struct kvm_vcpu *vcpu); void nested_copy_vmcb_control_to_cache(struct vcpu_svm *svm, struct vmcb_control_area *control); void nested_copy_vmcb_save_to_cache(struct vcpu_svm *svm, diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h index d0abee35d7ba..41a4533f9989 100644 --- a/arch/x86/kvm/vmx/capabilities.h +++ b/arch/x86/kvm/vmx/capabilities.h @@ -252,7 +252,7 @@ static inline bool cpu_has_vmx_pml(void) static inline bool cpu_has_vmx_xsaves(void) { return vmcs_config.cpu_based_2nd_exec_ctrl & - SECONDARY_EXEC_XSAVES; + SECONDARY_EXEC_ENABLE_XSAVES; } static inline bool cpu_has_vmx_waitpkg(void) diff --git a/arch/x86/kvm/vmx/hyperv.c b/arch/x86/kvm/vmx/hyperv.c index 79450e1ed7cf..313b8bb5b8a7 100644 --- a/arch/x86/kvm/vmx/hyperv.c +++ b/arch/x86/kvm/vmx/hyperv.c @@ -78,7 +78,7 @@ SECONDARY_EXEC_DESC | \ SECONDARY_EXEC_ENABLE_RDTSCP | \ SECONDARY_EXEC_ENABLE_INVPCID | \ - SECONDARY_EXEC_XSAVES | \ + SECONDARY_EXEC_ENABLE_XSAVES | \ SECONDARY_EXEC_RDSEED_EXITING | \ SECONDARY_EXEC_RDRAND_EXITING | \ SECONDARY_EXEC_TSC_SCALING | \ diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 516391cc0d64..c5ec0ef51ff7 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -2307,7 +2307,7 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0 SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE | SECONDARY_EXEC_ENABLE_INVPCID | SECONDARY_EXEC_ENABLE_RDTSCP | - SECONDARY_EXEC_XSAVES | + SECONDARY_EXEC_ENABLE_XSAVES | SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE | SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | SECONDARY_EXEC_APIC_REGISTER_VIRT | @@ -6331,7 +6331,7 @@ static bool nested_vmx_l1_wants_exit(struct kvm_vcpu *vcpu, * If if it were, XSS would have to be checked against * the XSS exit bitmap in vmcs12. */ - return nested_cpu_has2(vmcs12, SECONDARY_EXEC_XSAVES); + return nested_cpu_has2(vmcs12, SECONDARY_EXEC_ENABLE_XSAVES); case EXIT_REASON_UMWAIT: case EXIT_REASON_TPAUSE: return nested_cpu_has2(vmcs12, @@ -6426,7 +6426,7 @@ static int vmx_get_nested_state(struct kvm_vcpu *vcpu, vmx = to_vmx(vcpu); vmcs12 = get_vmcs12(vcpu); - if (nested_vmx_allowed(vcpu) && + if (guest_can_use(vcpu, X86_FEATURE_VMX) && (vmx->nested.vmxon || vmx->nested.smm.vmxon)) { kvm_state.hdr.vmx.vmxon_pa = vmx->nested.vmxon_ptr; kvm_state.hdr.vmx.vmcs12_pa = vmx->nested.current_vmptr; @@ -6567,7 +6567,7 @@ static int vmx_set_nested_state(struct kvm_vcpu *vcpu, if (kvm_state->flags & ~KVM_STATE_NESTED_EVMCS) return -EINVAL; } else { - if (!nested_vmx_allowed(vcpu)) + if (!guest_can_use(vcpu, X86_FEATURE_VMX)) return -EINVAL; if (!page_address_valid(vcpu, kvm_state->hdr.vmx.vmxon_pa)) @@ -6601,7 +6601,8 @@ static int vmx_set_nested_state(struct kvm_vcpu *vcpu, return -EINVAL; if ((kvm_state->flags & KVM_STATE_NESTED_EVMCS) && - (!nested_vmx_allowed(vcpu) || !vmx->nested.enlightened_vmcs_enabled)) + (!guest_can_use(vcpu, X86_FEATURE_VMX) || + !vmx->nested.enlightened_vmcs_enabled)) return -EINVAL; vmx_leave_nested(vcpu); @@ -6874,7 +6875,7 @@ static void nested_vmx_setup_secondary_ctls(u32 ept_caps, SECONDARY_EXEC_ENABLE_INVPCID | SECONDARY_EXEC_ENABLE_VMFUNC | SECONDARY_EXEC_RDSEED_EXITING | - SECONDARY_EXEC_XSAVES | + SECONDARY_EXEC_ENABLE_XSAVES | SECONDARY_EXEC_TSC_SCALING | SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE; diff --git a/arch/x86/kvm/vmx/nested.h b/arch/x86/kvm/vmx/nested.h index 96952263b029..b4b9d51438c6 100644 --- a/arch/x86/kvm/vmx/nested.h +++ b/arch/x86/kvm/vmx/nested.h @@ -168,7 +168,7 @@ static inline int nested_cpu_has_ept(struct vmcs12 *vmcs12) static inline bool nested_cpu_has_xsaves(struct vmcs12 *vmcs12) { - return nested_cpu_has2(vmcs12, SECONDARY_EXEC_XSAVES); + return nested_cpu_has2(vmcs12, SECONDARY_EXEC_ENABLE_XSAVES); } static inline bool nested_cpu_has_pml(struct vmcs12 *vmcs12) diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c index 80c769c58a87..f2efa0bf7ae8 100644 --- a/arch/x86/kvm/vmx/pmu_intel.c +++ b/arch/x86/kvm/vmx/pmu_intel.c @@ -22,23 +22,51 @@ #define MSR_PMC_FULL_WIDTH_BIT (MSR_IA32_PMC0 - MSR_IA32_PERFCTR0) +enum intel_pmu_architectural_events { + /* + * The order of the architectural events matters as support for each + * event is enumerated via CPUID using the index of the event. + */ + INTEL_ARCH_CPU_CYCLES, + INTEL_ARCH_INSTRUCTIONS_RETIRED, + INTEL_ARCH_REFERENCE_CYCLES, + INTEL_ARCH_LLC_REFERENCES, + INTEL_ARCH_LLC_MISSES, + INTEL_ARCH_BRANCHES_RETIRED, + INTEL_ARCH_BRANCHES_MISPREDICTED, + + NR_REAL_INTEL_ARCH_EVENTS, + + /* + * Pseudo-architectural event used to implement IA32_FIXED_CTR2, a.k.a. + * TSC reference cycles. The architectural reference cycles event may + * or may not actually use the TSC as the reference, e.g. might use the + * core crystal clock or the bus clock (yeah, "architectural"). + */ + PSEUDO_ARCH_REFERENCE_CYCLES = NR_REAL_INTEL_ARCH_EVENTS, + NR_INTEL_ARCH_EVENTS, +}; + static struct { u8 eventsel; u8 unit_mask; } const intel_arch_events[] = { - [0] = { 0x3c, 0x00 }, - [1] = { 0xc0, 0x00 }, - [2] = { 0x3c, 0x01 }, - [3] = { 0x2e, 0x4f }, - [4] = { 0x2e, 0x41 }, - [5] = { 0xc4, 0x00 }, - [6] = { 0xc5, 0x00 }, - /* The above index must match CPUID 0x0A.EBX bit vector */ - [7] = { 0x00, 0x03 }, + [INTEL_ARCH_CPU_CYCLES] = { 0x3c, 0x00 }, + [INTEL_ARCH_INSTRUCTIONS_RETIRED] = { 0xc0, 0x00 }, + [INTEL_ARCH_REFERENCE_CYCLES] = { 0x3c, 0x01 }, + [INTEL_ARCH_LLC_REFERENCES] = { 0x2e, 0x4f }, + [INTEL_ARCH_LLC_MISSES] = { 0x2e, 0x41 }, + [INTEL_ARCH_BRANCHES_RETIRED] = { 0xc4, 0x00 }, + [INTEL_ARCH_BRANCHES_MISPREDICTED] = { 0xc5, 0x00 }, + [PSEUDO_ARCH_REFERENCE_CYCLES] = { 0x00, 0x03 }, }; /* mapping between fixed pmc index and intel_arch_events array */ -static int fixed_pmc_events[] = {1, 0, 7}; +static int fixed_pmc_events[] = { + [0] = INTEL_ARCH_INSTRUCTIONS_RETIRED, + [1] = INTEL_ARCH_CPU_CYCLES, + [2] = PSEUDO_ARCH_REFERENCE_CYCLES, +}; static void reprogram_fixed_counters(struct kvm_pmu *pmu, u64 data) { @@ -80,16 +108,18 @@ static bool intel_hw_event_available(struct kvm_pmc *pmc) u8 unit_mask = (pmc->eventsel & ARCH_PERFMON_EVENTSEL_UMASK) >> 8; int i; - for (i = 0; i < ARRAY_SIZE(intel_arch_events); i++) { + BUILD_BUG_ON(ARRAY_SIZE(intel_arch_events) != NR_INTEL_ARCH_EVENTS); + + /* + * Disallow events reported as unavailable in guest CPUID. Note, this + * doesn't apply to pseudo-architectural events. + */ + for (i = 0; i < NR_REAL_INTEL_ARCH_EVENTS; i++) { if (intel_arch_events[i].eventsel != event_select || intel_arch_events[i].unit_mask != unit_mask) continue; - /* disable event that reported as not present by cpuid */ - if ((i < 7) && !(pmu->available_event_types & (1 << i))) - return false; - - break; + return pmu->available_event_types & BIT(i); } return true; @@ -438,16 +468,17 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) static void setup_fixed_pmc_eventsel(struct kvm_pmu *pmu) { - size_t size = ARRAY_SIZE(fixed_pmc_events); - struct kvm_pmc *pmc; - u32 event; int i; + BUILD_BUG_ON(ARRAY_SIZE(fixed_pmc_events) != KVM_PMC_MAX_FIXED); + for (i = 0; i < pmu->nr_arch_fixed_counters; i++) { - pmc = &pmu->fixed_counters[i]; - event = fixed_pmc_events[array_index_nospec(i, size)]; + int index = array_index_nospec(i, KVM_PMC_MAX_FIXED); + struct kvm_pmc *pmc = &pmu->fixed_counters[index]; + u32 event = fixed_pmc_events[index]; + pmc->eventsel = (intel_arch_events[event].unit_mask << 8) | - intel_arch_events[event].eventsel; + intel_arch_events[event].eventsel; } } @@ -508,10 +539,8 @@ static void intel_pmu_refresh(struct kvm_vcpu *vcpu) if (pmu->version == 1) { pmu->nr_arch_fixed_counters = 0; } else { - pmu->nr_arch_fixed_counters = - min3(ARRAY_SIZE(fixed_pmc_events), - (size_t) edx.split.num_counters_fixed, - (size_t)kvm_pmu_cap.num_counters_fixed); + pmu->nr_arch_fixed_counters = min_t(int, edx.split.num_counters_fixed, + kvm_pmu_cap.num_counters_fixed); edx.split.bit_width_fixed = min_t(int, edx.split.bit_width_fixed, kvm_pmu_cap.bit_width_fixed); pmu->counter_bitmask[KVM_PMC_FIXED] = diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index b483a8baaacf..72e3943f3693 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -41,13 +41,12 @@ #include #include #include -#include +#include #include #include #include #include #include -#include #include #include "capabilities.h" @@ -237,9 +236,6 @@ static const struct { #define L1D_CACHE_ORDER 4 static void *vmx_l1d_flush_pages; -/* Control for disabling CPU Fill buffer clear */ -static bool __read_mostly vmx_fb_clear_ctrl_available; - static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) { struct page *page; @@ -255,14 +251,9 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) return 0; } - if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES)) { - u64 msr; - - rdmsrl(MSR_IA32_ARCH_CAPABILITIES, msr); - if (msr & ARCH_CAP_SKIP_VMENTRY_L1DFLUSH) { - l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_NOT_REQUIRED; - return 0; - } + if (host_arch_capabilities & ARCH_CAP_SKIP_VMENTRY_L1DFLUSH) { + l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_NOT_REQUIRED; + return 0; } /* If set to auto use the default l1tf mitigation method */ @@ -366,22 +357,9 @@ static int vmentry_l1d_flush_set(const char *s, const struct kernel_param *kp) static int vmentry_l1d_flush_get(char *s, const struct kernel_param *kp) { if (WARN_ON_ONCE(l1tf_vmx_mitigation >= ARRAY_SIZE(vmentry_l1d_param))) - return sprintf(s, "???\n"); + return sysfs_emit(s, "???\n"); - return sprintf(s, "%s\n", vmentry_l1d_param[l1tf_vmx_mitigation].option); -} - -static void vmx_setup_fb_clear_ctrl(void) -{ - u64 msr; - - if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES) && - !boot_cpu_has_bug(X86_BUG_MDS) && - !boot_cpu_has_bug(X86_BUG_TAA)) { - rdmsrl(MSR_IA32_ARCH_CAPABILITIES, msr); - if (msr & ARCH_CAP_FB_CLEAR_CTRL) - vmx_fb_clear_ctrl_available = true; - } + return sysfs_emit(s, "%s\n", vmentry_l1d_param[l1tf_vmx_mitigation].option); } static __always_inline void vmx_disable_fb_clear(struct vcpu_vmx *vmx) @@ -409,7 +387,9 @@ static __always_inline void vmx_enable_fb_clear(struct vcpu_vmx *vmx) static void vmx_update_fb_clear_dis(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx) { - vmx->disable_fb_clear = vmx_fb_clear_ctrl_available; + vmx->disable_fb_clear = (host_arch_capabilities & ARCH_CAP_FB_CLEAR_CTRL) && + !boot_cpu_has_bug(X86_BUG_MDS) && + !boot_cpu_has_bug(X86_BUG_TAA); /* * If guest will not execute VERW, there is no need to set FB_CLEAR_DIS @@ -754,17 +734,51 @@ static int vmx_set_guest_uret_msr(struct vcpu_vmx *vmx, return ret; } -#ifdef CONFIG_KEXEC_CORE -static void crash_vmclear_local_loaded_vmcss(void) +/* + * Disable VMX and clear CR4.VMXE (even if VMXOFF faults) + * + * Note, VMXOFF causes a #UD if the CPU is !post-VMXON, but it's impossible to + * atomically track post-VMXON state, e.g. this may be called in NMI context. + * Eat all faults as all other faults on VMXOFF faults are mode related, i.e. + * faults are guaranteed to be due to the !post-VMXON check unless the CPU is + * magically in RM, VM86, compat mode, or at CPL>0. + */ +static int kvm_cpu_vmxoff(void) +{ + asm_volatile_goto("1: vmxoff\n\t" + _ASM_EXTABLE(1b, %l[fault]) + ::: "cc", "memory" : fault); + + cr4_clear_bits(X86_CR4_VMXE); + return 0; + +fault: + cr4_clear_bits(X86_CR4_VMXE); + return -EIO; +} + +static void vmx_emergency_disable(void) { int cpu = raw_smp_processor_id(); struct loaded_vmcs *v; + kvm_rebooting = true; + + /* + * Note, CR4.VMXE can be _cleared_ in NMI context, but it can only be + * set in task context. If this races with VMX is disabled by an NMI, + * VMCLEAR and VMXOFF may #UD, but KVM will eat those faults due to + * kvm_rebooting set. + */ + if (!(__read_cr4() & X86_CR4_VMXE)) + return; + list_for_each_entry(v, &per_cpu(loaded_vmcss_on_cpu, cpu), loaded_vmcss_on_cpu_link) vmcs_clear(v->vmcs); + + kvm_cpu_vmxoff(); } -#endif /* CONFIG_KEXEC_CORE */ static void __loaded_vmcs_clear(void *arg) { @@ -1899,25 +1913,14 @@ u64 vmx_get_l2_tsc_multiplier(struct kvm_vcpu *vcpu) return kvm_caps.default_tsc_scaling_ratio; } -static void vmx_write_tsc_offset(struct kvm_vcpu *vcpu, u64 offset) +static void vmx_write_tsc_offset(struct kvm_vcpu *vcpu) { - vmcs_write64(TSC_OFFSET, offset); + vmcs_write64(TSC_OFFSET, vcpu->arch.tsc_offset); } -static void vmx_write_tsc_multiplier(struct kvm_vcpu *vcpu, u64 multiplier) +static void vmx_write_tsc_multiplier(struct kvm_vcpu *vcpu) { - vmcs_write64(TSC_MULTIPLIER, multiplier); -} - -/* - * nested_vmx_allowed() checks whether a guest should be allowed to use VMX - * instructions and MSRs (i.e., nested VMX). Nested VMX is disabled for - * all guests if the "nested" module option is off, and can also be disabled - * for a single guest by disabling its VMX cpuid bit. - */ -bool nested_vmx_allowed(struct kvm_vcpu *vcpu) -{ - return nested && guest_cpuid_has(vcpu, X86_FEATURE_VMX); + vmcs_write64(TSC_MULTIPLIER, vcpu->arch.tsc_scaling_ratio); } /* @@ -2047,7 +2050,7 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) [msr_info->index - MSR_IA32_SGXLEPUBKEYHASH0]; break; case KVM_FIRST_EMULATED_VMX_MSR ... KVM_LAST_EMULATED_VMX_MSR: - if (!nested_vmx_allowed(vcpu)) + if (!guest_can_use(vcpu, X86_FEATURE_VMX)) return 1; if (vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index, &msr_info->data)) @@ -2355,7 +2358,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) case KVM_FIRST_EMULATED_VMX_MSR ... KVM_LAST_EMULATED_VMX_MSR: if (!msr_info->host_initiated) return 1; /* they are read-only */ - if (!nested_vmx_allowed(vcpu)) + if (!guest_can_use(vcpu, X86_FEATURE_VMX)) return 1; return vmx_set_vmx_msr(vcpu, msr_index, data); case MSR_IA32_RTIT_CTL: @@ -2729,11 +2732,11 @@ static int setup_vmcs_config(struct vmcs_config *vmcs_conf, return 0; } -static bool kvm_is_vmx_supported(void) +static bool __kvm_is_vmx_supported(void) { - int cpu = raw_smp_processor_id(); + int cpu = smp_processor_id(); - if (!cpu_has_vmx()) { + if (!(cpuid_ecx(1) & feature_bit(VMX))) { pr_err("VMX not supported by CPU %d\n", cpu); return false; } @@ -2747,13 +2750,24 @@ static bool kvm_is_vmx_supported(void) return true; } +static bool kvm_is_vmx_supported(void) +{ + bool supported; + + migrate_disable(); + supported = __kvm_is_vmx_supported(); + migrate_enable(); + + return supported; +} + static int vmx_check_processor_compat(void) { int cpu = raw_smp_processor_id(); struct vmcs_config vmcs_conf; struct vmx_capability vmx_cap; - if (!kvm_is_vmx_supported()) + if (!__kvm_is_vmx_supported()) return -EIO; if (setup_vmcs_config(&vmcs_conf, &vmx_cap) < 0) { @@ -2833,7 +2847,7 @@ static void vmx_hardware_disable(void) { vmclear_local_loaded_vmcss(); - if (cpu_vmxoff()) + if (kvm_cpu_vmxoff()) kvm_spurious_fault(); hv_reset_evmcs(); @@ -3071,13 +3085,6 @@ static void enter_rmode(struct kvm_vcpu *vcpu) vmx->rmode.vm86_active = 1; - /* - * Very old userspace does not call KVM_SET_TSS_ADDR before entering - * vcpu. Warn the user that an update is overdue. - */ - if (!kvm_vmx->tss_addr) - pr_warn_once("KVM_SET_TSS_ADDR needs to be called before running vCPU\n"); - vmx_segment_cache_clear(vmx); vmcs_writel(GUEST_TR_BASE, kvm_vmx->tss_addr); @@ -3350,7 +3357,7 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) vmx->emulation_required = vmx_emulation_required(vcpu); } -static int vmx_get_max_tdp_level(void) +static int vmx_get_max_ept_level(void) { if (cpu_has_vmx_ept_5levels()) return 5; @@ -4553,16 +4560,19 @@ vmx_adjust_secondary_exec_control(struct vcpu_vmx *vmx, u32 *exec_control, * based on a single guest CPUID bit, with a dedicated feature bit. This also * verifies that the control is actually supported by KVM and hardware. */ -#define vmx_adjust_sec_exec_control(vmx, exec_control, name, feat_name, ctrl_name, exiting) \ -({ \ - bool __enabled; \ - \ - if (cpu_has_vmx_##name()) { \ - __enabled = guest_cpuid_has(&(vmx)->vcpu, \ - X86_FEATURE_##feat_name); \ - vmx_adjust_secondary_exec_control(vmx, exec_control, \ - SECONDARY_EXEC_##ctrl_name, __enabled, exiting); \ - } \ +#define vmx_adjust_sec_exec_control(vmx, exec_control, name, feat_name, ctrl_name, exiting) \ +({ \ + struct kvm_vcpu *__vcpu = &(vmx)->vcpu; \ + bool __enabled; \ + \ + if (cpu_has_vmx_##name()) { \ + if (kvm_is_governed_feature(X86_FEATURE_##feat_name)) \ + __enabled = guest_can_use(__vcpu, X86_FEATURE_##feat_name); \ + else \ + __enabled = guest_cpuid_has(__vcpu, X86_FEATURE_##feat_name); \ + vmx_adjust_secondary_exec_control(vmx, exec_control, SECONDARY_EXEC_##ctrl_name,\ + __enabled, exiting); \ + } \ }) /* More macro magic for ENABLE_/opt-in versus _EXITING/opt-out controls. */ @@ -4622,19 +4632,7 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx *vmx) if (!enable_pml || !atomic_read(&vcpu->kvm->nr_memslots_dirty_logging)) exec_control &= ~SECONDARY_EXEC_ENABLE_PML; - if (cpu_has_vmx_xsaves()) { - /* Exposing XSAVES only when XSAVE is exposed */ - bool xsaves_enabled = - boot_cpu_has(X86_FEATURE_XSAVE) && - guest_cpuid_has(vcpu, X86_FEATURE_XSAVE) && - guest_cpuid_has(vcpu, X86_FEATURE_XSAVES); - - vcpu->arch.xsaves_enabled = xsaves_enabled; - - vmx_adjust_secondary_exec_control(vmx, &exec_control, - SECONDARY_EXEC_XSAVES, - xsaves_enabled, false); - } + vmx_adjust_sec_exec_feature(vmx, &exec_control, xsaves, XSAVES); /* * RDPID is also gated by ENABLE_RDTSCP, turn on the control if either @@ -4653,6 +4651,7 @@ static u32 vmx_secondary_exec_control(struct vcpu_vmx *vmx) SECONDARY_EXEC_ENABLE_RDTSCP, rdpid_or_rdtscp_enabled, false); } + vmx_adjust_sec_exec_feature(vmx, &exec_control, invpcid, INVPCID); vmx_adjust_sec_exec_exiting(vmx, &exec_control, rdrand, RDRAND); @@ -6796,8 +6795,10 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu) vmcs_write64(APIC_ACCESS_ADDR, pfn_to_hpa(pfn)); read_unlock(&vcpu->kvm->mmu_lock); - vmx_flush_tlb_current(vcpu); - + /* + * No need for a manual TLB flush at this point, KVM has already done a + * flush if there were SPTEs pointing at the previous page. + */ out: /* * Do not pin apic access page in memory, the MMU notifier @@ -7243,13 +7244,20 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, flags); vcpu->arch.cr2 = native_read_cr2(); + vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET; + + vmx->idt_vectoring_info = 0; vmx_enable_fb_clear(vmx); - if (unlikely(vmx->fail)) + if (unlikely(vmx->fail)) { vmx->exit_reason.full = 0xdead; - else - vmx->exit_reason.full = vmcs_read32(VM_EXIT_REASON); + goto out; + } + + vmx->exit_reason.full = vmcs_read32(VM_EXIT_REASON); + if (likely(!vmx->exit_reason.failed_vmentry)) + vmx->idt_vectoring_info = vmcs_read32(IDT_VECTORING_INFO_FIELD); if ((u16)vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI && is_nmi(vmx_get_intr_info(vcpu))) { @@ -7258,6 +7266,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, kvm_after_interrupt(vcpu); } +out: guest_state_exit_irqoff(); } @@ -7379,8 +7388,6 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) loadsegment(es, __USER_DS); #endif - vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET; - pt_guest_exit(vmx); kvm_load_host_xsave_state(vcpu); @@ -7397,17 +7404,12 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) vmx->nested.nested_run_pending = 0; } - vmx->idt_vectoring_info = 0; - if (unlikely(vmx->fail)) return EXIT_FASTPATH_NONE; if (unlikely((u16)vmx->exit_reason.basic == EXIT_REASON_MCE_DURING_VMENTRY)) kvm_machine_check(); - if (likely(!vmx->exit_reason.failed_vmentry)) - vmx->idt_vectoring_info = vmcs_read32(IDT_VECTORING_INFO_FIELD); - trace_kvm_exit(vcpu, KVM_ISA_VMX); if (unlikely(vmx->exit_reason.failed_vmentry)) @@ -7751,8 +7753,16 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); - /* xsaves_enabled is recomputed in vmx_compute_secondary_exec_control(). */ - vcpu->arch.xsaves_enabled = false; + /* + * XSAVES is effectively enabled if and only if XSAVE is also exposed + * to the guest. XSAVES depends on CR4.OSXSAVE, and CR4.OSXSAVE can be + * set if and only if XSAVE is supported. + */ + if (boot_cpu_has(X86_FEATURE_XSAVE) && + guest_cpuid_has(vcpu, X86_FEATURE_XSAVE)) + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_XSAVES); + + kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_VMX); vmx_setup_uret_msrs(vmx); @@ -7760,7 +7770,7 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vmcs_set_secondary_exec_control(vmx, vmx_secondary_exec_control(vmx)); - if (nested_vmx_allowed(vcpu)) + if (guest_can_use(vcpu, X86_FEATURE_VMX)) vmx->msr_ia32_feature_control_valid_bits |= FEAT_CTL_VMX_ENABLED_INSIDE_SMX | FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX; @@ -7769,7 +7779,7 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) ~(FEAT_CTL_VMX_ENABLED_INSIDE_SMX | FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX); - if (nested_vmx_allowed(vcpu)) + if (guest_can_use(vcpu, X86_FEATURE_VMX)) nested_vmx_cr_fixed1_bits_update(vcpu); if (boot_cpu_has(X86_FEATURE_INTEL_PT) && @@ -8526,7 +8536,7 @@ static __init int hardware_setup(void) */ vmx_setup_me_spte_mask(); - kvm_configure_mmu(enable_ept, 0, vmx_get_max_tdp_level(), + kvm_configure_mmu(enable_ept, 0, vmx_get_max_ept_level(), ept_caps_to_lpage_level(vmx_capability.ept)); /* @@ -8622,10 +8632,8 @@ static void __vmx_exit(void) { allow_smaller_maxphyaddr = false; -#ifdef CONFIG_KEXEC_CORE - RCU_INIT_POINTER(crash_vmclear_loaded_vmcss, NULL); - synchronize_rcu(); -#endif + cpu_emergency_unregister_virt_callback(vmx_emergency_disable); + vmx_cleanup_l1d_flush(); } @@ -8666,18 +8674,14 @@ static int __init vmx_init(void) if (r) goto err_l1d_flush; - vmx_setup_fb_clear_ctrl(); - for_each_possible_cpu(cpu) { INIT_LIST_HEAD(&per_cpu(loaded_vmcss_on_cpu, cpu)); pi_init_cpu(cpu); } -#ifdef CONFIG_KEXEC_CORE - rcu_assign_pointer(crash_vmclear_loaded_vmcss, - crash_vmclear_local_loaded_vmcss); -#endif + cpu_emergency_register_virt_callback(vmx_emergency_disable); + vmx_check_vmcs12_offsets(); /* diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 32384ba38499..c2130d2c8e24 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -374,7 +374,6 @@ struct kvm_vmx { u64 *pid_table; }; -bool nested_vmx_allowed(struct kvm_vcpu *vcpu); void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu, struct loaded_vmcs *buddy); int allocate_vpid(void); @@ -562,7 +561,7 @@ static inline u8 vmx_get_rvi(void) SECONDARY_EXEC_APIC_REGISTER_VIRT | \ SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | \ SECONDARY_EXEC_SHADOW_VMCS | \ - SECONDARY_EXEC_XSAVES | \ + SECONDARY_EXEC_ENABLE_XSAVES | \ SECONDARY_EXEC_RDSEED_EXITING | \ SECONDARY_EXEC_RDRAND_EXITING | \ SECONDARY_EXEC_ENABLE_PML | \ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index c381770bcbf1..6c9c81e82e65 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -25,6 +25,7 @@ #include "tss.h" #include "kvm_cache_regs.h" #include "kvm_emulate.h" +#include "mmu/page_track.h" #include "x86.h" #include "cpuid.h" #include "pmu.h" @@ -237,6 +238,9 @@ EXPORT_SYMBOL_GPL(enable_apicv); u64 __read_mostly host_xss; EXPORT_SYMBOL_GPL(host_xss); +u64 __read_mostly host_arch_capabilities; +EXPORT_SYMBOL_GPL(host_arch_capabilities); + const struct _kvm_stats_desc kvm_vm_stats_desc[] = { KVM_GENERIC_VM_STATS(), STATS_DESC_COUNTER(VM, mmu_shadow_zapped), @@ -1021,7 +1025,7 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) if (vcpu->arch.xcr0 != host_xcr0) xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0); - if (vcpu->arch.xsaves_enabled && + if (guest_can_use(vcpu, X86_FEATURE_XSAVES) && vcpu->arch.ia32_xss != host_xss) wrmsrl(MSR_IA32_XSS, vcpu->arch.ia32_xss); } @@ -1052,7 +1056,7 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu) if (vcpu->arch.xcr0 != host_xcr0) xsetbv(XCR_XFEATURE_ENABLED_MASK, host_xcr0); - if (vcpu->arch.xsaves_enabled && + if (guest_can_use(vcpu, X86_FEATURE_XSAVES) && vcpu->arch.ia32_xss != host_xss) wrmsrl(MSR_IA32_XSS, host_xss); } @@ -1620,12 +1624,7 @@ static bool kvm_is_immutable_feature_msr(u32 msr) static u64 kvm_get_arch_capabilities(void) { - u64 data = 0; - - if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES)) { - rdmsrl(MSR_IA32_ARCH_CAPABILITIES, data); - data &= KVM_SUPPORTED_ARCH_CAP; - } + u64 data = host_arch_capabilities & KVM_SUPPORTED_ARCH_CAP; /* * If nx_huge_pages is enabled, KVM's shadow paging will ensure that @@ -2631,7 +2630,7 @@ static void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset) else vcpu->arch.tsc_offset = l1_offset; - static_call(kvm_x86_write_tsc_offset)(vcpu, vcpu->arch.tsc_offset); + static_call(kvm_x86_write_tsc_offset)(vcpu); } static void kvm_vcpu_write_tsc_multiplier(struct kvm_vcpu *vcpu, u64 l1_multiplier) @@ -2647,8 +2646,7 @@ static void kvm_vcpu_write_tsc_multiplier(struct kvm_vcpu *vcpu, u64 l1_multipli vcpu->arch.tsc_scaling_ratio = l1_multiplier; if (kvm_caps.has_tsc_control) - static_call(kvm_x86_write_tsc_multiplier)( - vcpu, vcpu->arch.tsc_scaling_ratio); + static_call(kvm_x86_write_tsc_multiplier)(vcpu); } static inline bool kvm_check_tsc_unstable(void) @@ -4665,7 +4663,6 @@ static int kvm_x86_dev_get_attr(struct kvm_device_attr *attr) return 0; default: return -ENXIO; - break; } } @@ -6532,7 +6529,7 @@ static void kvm_free_msr_filter(struct kvm_x86_msr_filter *msr_filter) static int kvm_add_msr_filter(struct kvm_x86_msr_filter *msr_filter, struct kvm_msr_filter_range *user_range) { - unsigned long *bitmap = NULL; + unsigned long *bitmap; size_t bitmap_size; if (!user_range->nmsrs) @@ -8245,11 +8242,6 @@ static bool emulator_get_cpuid(struct x86_emulate_ctxt *ctxt, return kvm_cpuid(emul_to_vcpu(ctxt), eax, ebx, ecx, edx, exact_only); } -static bool emulator_guest_has_long_mode(struct x86_emulate_ctxt *ctxt) -{ - return guest_cpuid_has(emul_to_vcpu(ctxt), X86_FEATURE_LM); -} - static bool emulator_guest_has_movbe(struct x86_emulate_ctxt *ctxt) { return guest_cpuid_has(emul_to_vcpu(ctxt), X86_FEATURE_MOVBE); @@ -8351,7 +8343,6 @@ static const struct x86_emulate_ops emulate_ops = { .fix_hypercall = emulator_fix_hypercall, .intercept = emulator_intercept, .get_cpuid = emulator_get_cpuid, - .guest_has_long_mode = emulator_guest_has_long_mode, .guest_has_movbe = emulator_guest_has_movbe, .guest_has_fxsr = emulator_guest_has_fxsr, .guest_has_rdpid = emulator_guest_has_rdpid, @@ -9172,7 +9163,7 @@ static int kvmclock_cpu_down_prep(unsigned int cpu) static void tsc_khz_changed(void *data) { struct cpufreq_freqs *freq = data; - unsigned long khz = 0; + unsigned long khz; WARN_ON_ONCE(boot_cpu_has(X86_FEATURE_CONSTANT_TSC)); @@ -9512,6 +9503,9 @@ static int __kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) kvm_init_pmu_capability(ops->pmu_ops); + if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES)) + rdmsrl(MSR_IA32_ARCH_CAPABILITIES, host_arch_capabilities); + r = ops->hardware_setup(); if (r != 0) goto out_mmu_exit; @@ -11111,12 +11105,17 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) r = -EINTR; goto out; } - /* - * It should be impossible for the hypervisor timer to be in - * use before KVM has ever run the vCPU. - */ - WARN_ON_ONCE(kvm_lapic_hv_timer_in_use(vcpu)); + /* + * Don't bother switching APIC timer emulation from the + * hypervisor timer to the software timer, the only way for the + * APIC timer to be active is if userspace stuffed vCPU state, + * i.e. put the vCPU into a nonsensical state. Only an INIT + * will transition the vCPU out of UNINITIALIZED (without more + * state stuffing from userspace), which will reset the local + * APIC and thus cancel the timer or drop the IRQ (if the timer + * already expired). + */ kvm_vcpu_srcu_read_unlock(vcpu); kvm_vcpu_block(vcpu); kvm_vcpu_srcu_read_lock(vcpu); @@ -11798,15 +11797,22 @@ static int sync_regs(struct kvm_vcpu *vcpu) __set_regs(vcpu, &vcpu->run->s.regs.regs); vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_REGS; } + if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) { - if (__set_sregs(vcpu, &vcpu->run->s.regs.sregs)) + struct kvm_sregs sregs = vcpu->run->s.regs.sregs; + + if (__set_sregs(vcpu, &sregs)) return -EINVAL; + vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_SREGS; } + if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_EVENTS) { - if (kvm_vcpu_ioctl_x86_set_vcpu_events( - vcpu, &vcpu->run->s.regs.events)) + struct kvm_vcpu_events events = vcpu->run->s.regs.events; + + if (kvm_vcpu_ioctl_x86_set_vcpu_events(vcpu, &events)) return -EINVAL; + vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_EVENTS; } @@ -12627,6 +12633,13 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *new, enum kvm_mr_change change) { + /* + * KVM doesn't support moving memslots when there are external page + * trackers attached to the VM, i.e. if KVMGT is in use. + */ + if (change == KVM_MR_MOVE && kvm_page_track_has_external_user(kvm)) + return -EINVAL; + if (change == KVM_MR_CREATE || change == KVM_MR_MOVE) { if ((new->base_gfn + new->npages - 1) > kvm_mmu_max_gfn()) return -EINVAL; @@ -12772,7 +12785,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, * See is_writable_pte() for more details (the case involving * access-tracked SPTEs is particularly relevant). */ - kvm_arch_flush_remote_tlbs_memslot(kvm, new); + kvm_flush_remote_tlbs_memslot(kvm, new); } } @@ -12781,6 +12794,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_memory_slot *new, enum kvm_mr_change change) { + if (change == KVM_MR_DELETE) + kvm_page_track_delete_slot(kvm, old); + if (!kvm->arch.n_requested_mmu_pages && (change == KVM_MR_CREATE || change == KVM_MR_DELETE)) { unsigned long nr_mmu_pages; @@ -12797,17 +12813,6 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, kvm_arch_free_memslot(kvm, old); } -void kvm_arch_flush_shadow_all(struct kvm *kvm) -{ - kvm_mmu_zap_all(kvm); -} - -void kvm_arch_flush_shadow_memslot(struct kvm *kvm, - struct kvm_memory_slot *slot) -{ - kvm_page_track_flush_slot(kvm, slot); -} - static inline bool kvm_guest_apic_has_interrupt(struct kvm_vcpu *vcpu) { return (is_guest_mode(vcpu) && diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 82e3dafc5453..1e7be1f6ab29 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -323,6 +323,7 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu); extern u64 host_xcr0; extern u64 host_xss; +extern u64 host_arch_capabilities; extern struct kvm_caps kvm_caps; diff --git a/arch/xtensa/Kconfig b/arch/xtensa/Kconfig index a5488cc40f58..7d792077e5fd 100644 --- a/arch/xtensa/Kconfig +++ b/arch/xtensa/Kconfig @@ -71,6 +71,9 @@ config ARCH_HAS_ILOG2_U32 config ARCH_HAS_ILOG2_U64 def_bool n +config ARCH_MTD_XIP + def_bool y + config NO_IOPORT_MAP def_bool n diff --git a/arch/xtensa/include/asm/core.h b/arch/xtensa/include/asm/core.h index 0e1bb6f019d6..3f5ffae89b58 100644 --- a/arch/xtensa/include/asm/core.h +++ b/arch/xtensa/include/asm/core.h @@ -52,4 +52,13 @@ #define XTENSA_STACK_ALIGNMENT 16 #endif +#ifndef XCHAL_HW_MIN_VERSION +#if defined(XCHAL_HW_MIN_VERSION_MAJOR) && defined(XCHAL_HW_MIN_VERSION_MINOR) +#define XCHAL_HW_MIN_VERSION (XCHAL_HW_MIN_VERSION_MAJOR * 100 + \ + XCHAL_HW_MIN_VERSION_MINOR) +#else +#define XCHAL_HW_MIN_VERSION 0 +#endif +#endif + #endif diff --git a/arch/xtensa/include/asm/mtd-xip.h b/arch/xtensa/include/asm/mtd-xip.h new file mode 100644 index 000000000000..514325155cf8 --- /dev/null +++ b/arch/xtensa/include/asm/mtd-xip.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef _ASM_MTD_XIP_H +#define _ASM_MTD_XIP_H + +#include + +#define xip_irqpending() (xtensa_get_sr(interrupt) & xtensa_get_sr(intenable)) +#define xip_currtime() (xtensa_get_sr(ccount)) +#define xip_elapsed_since(x) ((xtensa_get_sr(ccount) - (x)) / 1000) /* should work up to 1GHz */ +#define xip_cpu_idle() do { asm volatile ("waiti 0"); } while (0) + +#endif /* _ASM_MTD_XIP_H */ + diff --git a/arch/xtensa/include/asm/sections.h b/arch/xtensa/include/asm/sections.h index 3bc6b9afa993..e5da6d7092be 100644 --- a/arch/xtensa/include/asm/sections.h +++ b/arch/xtensa/include/asm/sections.h @@ -34,6 +34,10 @@ extern char _SecondaryResetVector_text_start[]; extern char _SecondaryResetVector_text_end[]; #endif #ifdef CONFIG_XIP_KERNEL +#ifdef CONFIG_VECTORS_ADDR +extern char _xip_text_start[]; +extern char _xip_text_end[]; +#endif extern char _xip_start[]; extern char _xip_end[]; #endif diff --git a/arch/xtensa/kernel/perf_event.c b/arch/xtensa/kernel/perf_event.c index a0d05c8598d0..183618090d05 100644 --- a/arch/xtensa/kernel/perf_event.c +++ b/arch/xtensa/kernel/perf_event.c @@ -13,17 +13,26 @@ #include #include +#include #include #include +#define XTENSA_HWVERSION_RG_2015_0 260000 + +#if XCHAL_HW_MIN_VERSION >= XTENSA_HWVERSION_RG_2015_0 +#define XTENSA_PMU_ERI_BASE 0x00101000 +#else +#define XTENSA_PMU_ERI_BASE 0x00001000 +#endif + /* Global control/status for all perf counters */ -#define XTENSA_PMU_PMG 0x1000 +#define XTENSA_PMU_PMG XTENSA_PMU_ERI_BASE /* Perf counter values */ -#define XTENSA_PMU_PM(i) (0x1080 + (i) * 4) +#define XTENSA_PMU_PM(i) (XTENSA_PMU_ERI_BASE + 0x80 + (i) * 4) /* Perf counter control registers */ -#define XTENSA_PMU_PMCTRL(i) (0x1100 + (i) * 4) +#define XTENSA_PMU_PMCTRL(i) (XTENSA_PMU_ERI_BASE + 0x100 + (i) * 4) /* Perf counter status registers */ -#define XTENSA_PMU_PMSTAT(i) (0x1180 + (i) * 4) +#define XTENSA_PMU_PMSTAT(i) (XTENSA_PMU_ERI_BASE + 0x180 + (i) * 4) #define XTENSA_PMU_PMG_PMEN 0x1 diff --git a/arch/xtensa/kernel/setup.c b/arch/xtensa/kernel/setup.c index aba3ff4e60d8..52d6e4870a04 100644 --- a/arch/xtensa/kernel/setup.c +++ b/arch/xtensa/kernel/setup.c @@ -311,6 +311,9 @@ void __init setup_arch(char **cmdline_p) mem_reserve(__pa(_stext), __pa(_end)); #ifdef CONFIG_XIP_KERNEL +#ifdef CONFIG_VECTORS_ADDR + mem_reserve(__pa(_xip_text_start), __pa(_xip_text_end)); +#endif mem_reserve(__pa(_xip_start), __pa(_xip_end)); #endif diff --git a/arch/xtensa/kernel/vmlinux.lds.S b/arch/xtensa/kernel/vmlinux.lds.S index c14fd96f459d..f47e9bbbd291 100644 --- a/arch/xtensa/kernel/vmlinux.lds.S +++ b/arch/xtensa/kernel/vmlinux.lds.S @@ -118,6 +118,7 @@ SECTIONS SECTION_VECTOR2 (.DoubleExceptionVector.text, DOUBLEEXC_VECTOR_VADDR) *(.exception.text) + *(.xiptext) #endif IRQENTRY_TEXT @@ -201,6 +202,9 @@ SECTIONS .DebugInterruptVector.text); RELOCATE_ENTRY(_exception_text, .exception.text); +#ifdef CONFIG_XIP_KERNEL + RELOCATE_ENTRY(_xip_text, .xiptext); +#endif #endif #ifdef CONFIG_XIP_KERNEL RELOCATE_ENTRY(_xip_data, .data); @@ -319,7 +323,12 @@ SECTIONS LAST) #undef LAST #define LAST .exception.text - + SECTION_VECTOR4 (_xip_text, + .xiptext, + , + LAST) +#undef LAST +#define LAST .xiptext #endif . = (LOADADDR(LAST) + SIZEOF(LAST) + 3) & ~ 3; diff --git a/block/blk-map.c b/block/blk-map.c index 44d74a30ddac..8584babf3ea0 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -315,12 +315,11 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, n = bytes; if (!bio_add_hw_page(rq->q, bio, page, n, offs, - max_sectors, &same_page)) { - if (same_page) - bio_release_page(bio, page); + max_sectors, &same_page)) break; - } + if (same_page) + bio_release_page(bio, page); bytes -= n; offs = 0; } diff --git a/block/blk-throttle.c b/block/blk-throttle.c index 7397ff199d66..38a881cf97d0 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -697,66 +697,6 @@ static bool throtl_slice_used(struct throtl_grp *tg, bool rw) return true; } -/* Trim the used slices and adjust slice start accordingly */ -static inline void throtl_trim_slice(struct throtl_grp *tg, bool rw) -{ - unsigned long nr_slices, time_elapsed, io_trim; - u64 bytes_trim, tmp; - - BUG_ON(time_before(tg->slice_end[rw], tg->slice_start[rw])); - - /* - * If bps are unlimited (-1), then time slice don't get - * renewed. Don't try to trim the slice if slice is used. A new - * slice will start when appropriate. - */ - if (throtl_slice_used(tg, rw)) - return; - - /* - * A bio has been dispatched. Also adjust slice_end. It might happen - * that initially cgroup limit was very low resulting in high - * slice_end, but later limit was bumped up and bio was dispatched - * sooner, then we need to reduce slice_end. A high bogus slice_end - * is bad because it does not allow new slice to start. - */ - - throtl_set_slice_end(tg, rw, jiffies + tg->td->throtl_slice); - - time_elapsed = jiffies - tg->slice_start[rw]; - - nr_slices = time_elapsed / tg->td->throtl_slice; - - if (!nr_slices) - return; - tmp = tg_bps_limit(tg, rw) * tg->td->throtl_slice * nr_slices; - do_div(tmp, HZ); - bytes_trim = tmp; - - io_trim = (tg_iops_limit(tg, rw) * tg->td->throtl_slice * nr_slices) / - HZ; - - if (!bytes_trim && !io_trim) - return; - - if (tg->bytes_disp[rw] >= bytes_trim) - tg->bytes_disp[rw] -= bytes_trim; - else - tg->bytes_disp[rw] = 0; - - if (tg->io_disp[rw] >= io_trim) - tg->io_disp[rw] -= io_trim; - else - tg->io_disp[rw] = 0; - - tg->slice_start[rw] += nr_slices * tg->td->throtl_slice; - - throtl_log(&tg->service_queue, - "[%c] trim slice nr=%lu bytes=%llu io=%lu start=%lu end=%lu jiffies=%lu", - rw == READ ? 'R' : 'W', nr_slices, bytes_trim, io_trim, - tg->slice_start[rw], tg->slice_end[rw], jiffies); -} - static unsigned int calculate_io_allowed(u32 iops_limit, unsigned long jiffy_elapsed) { @@ -786,6 +726,67 @@ static u64 calculate_bytes_allowed(u64 bps_limit, unsigned long jiffy_elapsed) return mul_u64_u64_div_u64(bps_limit, (u64)jiffy_elapsed, (u64)HZ); } +/* Trim the used slices and adjust slice start accordingly */ +static inline void throtl_trim_slice(struct throtl_grp *tg, bool rw) +{ + unsigned long time_elapsed; + long long bytes_trim; + int io_trim; + + BUG_ON(time_before(tg->slice_end[rw], tg->slice_start[rw])); + + /* + * If bps are unlimited (-1), then time slice don't get + * renewed. Don't try to trim the slice if slice is used. A new + * slice will start when appropriate. + */ + if (throtl_slice_used(tg, rw)) + return; + + /* + * A bio has been dispatched. Also adjust slice_end. It might happen + * that initially cgroup limit was very low resulting in high + * slice_end, but later limit was bumped up and bio was dispatched + * sooner, then we need to reduce slice_end. A high bogus slice_end + * is bad because it does not allow new slice to start. + */ + + throtl_set_slice_end(tg, rw, jiffies + tg->td->throtl_slice); + + time_elapsed = rounddown(jiffies - tg->slice_start[rw], + tg->td->throtl_slice); + if (!time_elapsed) + return; + + bytes_trim = calculate_bytes_allowed(tg_bps_limit(tg, rw), + time_elapsed) + + tg->carryover_bytes[rw]; + io_trim = calculate_io_allowed(tg_iops_limit(tg, rw), time_elapsed) + + tg->carryover_ios[rw]; + if (bytes_trim <= 0 && io_trim <= 0) + return; + + tg->carryover_bytes[rw] = 0; + if ((long long)tg->bytes_disp[rw] >= bytes_trim) + tg->bytes_disp[rw] -= bytes_trim; + else + tg->bytes_disp[rw] = 0; + + tg->carryover_ios[rw] = 0; + if ((int)tg->io_disp[rw] >= io_trim) + tg->io_disp[rw] -= io_trim; + else + tg->io_disp[rw] = 0; + + tg->slice_start[rw] += time_elapsed; + + throtl_log(&tg->service_queue, + "[%c] trim slice nr=%lu bytes=%lld io=%d start=%lu end=%lu jiffies=%lu", + rw == READ ? 'R' : 'W', time_elapsed / tg->td->throtl_slice, + bytes_trim, io_trim, tg->slice_start[rw], tg->slice_end[rw], + jiffies); +} + static void __tg_update_carryover(struct throtl_grp *tg, bool rw) { unsigned long jiffy_elapsed = jiffies - tg->slice_start[rw]; @@ -816,7 +817,7 @@ static void tg_update_carryover(struct throtl_grp *tg) __tg_update_carryover(tg, WRITE); /* see comments in struct throtl_grp for meaning of these fields. */ - throtl_log(&tg->service_queue, "%s: %llu %llu %u %u\n", __func__, + throtl_log(&tg->service_queue, "%s: %lld %lld %d %d\n", __func__, tg->carryover_bytes[READ], tg->carryover_bytes[WRITE], tg->carryover_ios[READ], tg->carryover_ios[WRITE]); } @@ -825,7 +826,7 @@ static unsigned long tg_within_iops_limit(struct throtl_grp *tg, struct bio *bio u32 iops_limit) { bool rw = bio_data_dir(bio); - unsigned int io_allowed; + int io_allowed; unsigned long jiffy_elapsed, jiffy_wait, jiffy_elapsed_rnd; if (iops_limit == UINT_MAX) { @@ -838,9 +839,8 @@ static unsigned long tg_within_iops_limit(struct throtl_grp *tg, struct bio *bio jiffy_elapsed_rnd = roundup(jiffy_elapsed + 1, tg->td->throtl_slice); io_allowed = calculate_io_allowed(iops_limit, jiffy_elapsed_rnd) + tg->carryover_ios[rw]; - if (tg->io_disp[rw] + 1 <= io_allowed) { + if (io_allowed > 0 && tg->io_disp[rw] + 1 <= io_allowed) return 0; - } /* Calc approx time to dispatch */ jiffy_wait = jiffy_elapsed_rnd - jiffy_elapsed; @@ -851,7 +851,8 @@ static unsigned long tg_within_bps_limit(struct throtl_grp *tg, struct bio *bio, u64 bps_limit) { bool rw = bio_data_dir(bio); - u64 bytes_allowed, extra_bytes; + long long bytes_allowed; + u64 extra_bytes; unsigned long jiffy_elapsed, jiffy_wait, jiffy_elapsed_rnd; unsigned int bio_size = throtl_bio_data_size(bio); @@ -869,9 +870,8 @@ static unsigned long tg_within_bps_limit(struct throtl_grp *tg, struct bio *bio, jiffy_elapsed_rnd = roundup(jiffy_elapsed_rnd, tg->td->throtl_slice); bytes_allowed = calculate_bytes_allowed(bps_limit, jiffy_elapsed_rnd) + tg->carryover_bytes[rw]; - if (tg->bytes_disp[rw] + bio_size <= bytes_allowed) { + if (bytes_allowed > 0 && tg->bytes_disp[rw] + bio_size <= bytes_allowed) return 0; - } /* Calc approx time to dispatch */ extra_bytes = tg->bytes_disp[rw] + bio_size - bytes_allowed; diff --git a/block/blk-throttle.h b/block/blk-throttle.h index d1ccbfe9f797..bffbc9cfc8ab 100644 --- a/block/blk-throttle.h +++ b/block/blk-throttle.h @@ -127,8 +127,8 @@ struct throtl_grp { * bytes/ios are waited already in previous configuration, and they will * be used to calculate wait time under new configuration. */ - uint64_t carryover_bytes[2]; - unsigned int carryover_ios[2]; + long long carryover_bytes[2]; + int carryover_ios[2]; unsigned long last_check_time; diff --git a/block/fops.c b/block/fops.c index a24a624d3bf7..acff3d5d22d4 100644 --- a/block/fops.c +++ b/block/fops.c @@ -671,10 +671,6 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, struct iov_iter *from) iov_iter_truncate(from, size); } - ret = file_remove_privs(file); - if (ret) - return ret; - ret = file_update_time(file); if (ret) return ret; diff --git a/block/ioctl.c b/block/ioctl.c index 648670ddb164..d5f5cd61efd7 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -20,6 +20,8 @@ static int blkpg_do_ioctl(struct block_device *bdev, struct blkpg_partition p; long long start, length; + if (disk->flags & GENHD_FL_NO_PART) + return -EINVAL; if (!capable(CAP_SYS_ADMIN)) return -EACCES; if (copy_from_user(&p, upart, sizeof(struct blkpg_partition))) diff --git a/drivers/Kconfig b/drivers/Kconfig index 496ca02ee18f..efb66e25fa2d 100644 --- a/drivers/Kconfig +++ b/drivers/Kconfig @@ -15,6 +15,8 @@ source "drivers/base/Kconfig" source "drivers/bus/Kconfig" +source "drivers/cache/Kconfig" + source "drivers/connector/Kconfig" source "drivers/firmware/Kconfig" diff --git a/drivers/Makefile b/drivers/Makefile index 0957f63ecb42..cb0afca2e4a0 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -11,6 +11,7 @@ ifdef building_out_of_srctree MAKEFLAGS += --include-dir=$(srctree) endif +obj-y += cache/ obj-y += irqchip/ obj-y += bus/ diff --git a/drivers/accel/ivpu/ivpu_jsm_msg.c b/drivers/accel/ivpu/ivpu_jsm_msg.c index 831bfd2b2d39..bdddef2c59ee 100644 --- a/drivers/accel/ivpu/ivpu_jsm_msg.c +++ b/drivers/accel/ivpu/ivpu_jsm_msg.c @@ -118,8 +118,7 @@ int ivpu_jsm_dyndbg_control(struct ivpu_device *vdev, char *command, size_t size struct vpu_jsm_msg resp; int ret; - if (!strncpy(req.payload.dyndbg_control.dyndbg_cmd, command, VPU_DYNDBG_CMD_MAX_LEN - 1)) - return -ENOMEM; + strscpy(req.payload.dyndbg_control.dyndbg_cmd, command, VPU_DYNDBG_CMD_MAX_LEN); ret = ivpu_ipc_send_receive(vdev, &req, VPU_JSM_MSG_DYNDBG_CONTROL_RSP, &resp, VPU_IPC_CHAN_ASYNC_CMD, vdev->timeout.jsm); diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c index e4e4175e3e83..d3f28b82c97b 100644 --- a/drivers/ata/libata-scsi.c +++ b/drivers/ata/libata-scsi.c @@ -4752,7 +4752,7 @@ void ata_scsi_dev_rescan(struct work_struct *work) } spin_unlock_irqrestore(ap->lock, flags); - scsi_rescan_device(&(sdev->sdev_gendev)); + scsi_rescan_device(sdev); scsi_device_put(sdev); spin_lock_irqsave(ap->lock, flags); } diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 79ab532aabaf..6bc86106c7b2 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1557,7 +1557,7 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa do { int sent; - bvec_set_page(&bvec, page, offset, len); + bvec_set_page(&bvec, page, len, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); sent = sock_sendmsg(socket, &msg); diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index 864013019d6b..968090935eb2 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -1643,9 +1643,12 @@ static int null_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob) struct nullb_queue *nq = hctx->driver_data; LIST_HEAD(list); int nr = 0; + struct request *rq; spin_lock(&nq->poll_lock); list_splice_init(&nq->poll_list, &list); + list_for_each_entry(rq, &list, queuelist) + blk_mq_set_request_complete(rq); spin_unlock(&nq->poll_lock); while (!list_empty(&list)) { @@ -1671,16 +1674,21 @@ static enum blk_eh_timer_return null_timeout_rq(struct request *rq) struct blk_mq_hw_ctx *hctx = rq->mq_hctx; struct nullb_cmd *cmd = blk_mq_rq_to_pdu(rq); - pr_info("rq %p timed out\n", rq); - if (hctx->type == HCTX_TYPE_POLL) { struct nullb_queue *nq = hctx->driver_data; spin_lock(&nq->poll_lock); + /* The request may have completed meanwhile. */ + if (blk_mq_request_completed(rq)) { + spin_unlock(&nq->poll_lock); + return BLK_EH_DONE; + } list_del_init(&rq->queuelist); spin_unlock(&nq->poll_lock); } + pr_info("rq %p timed out\n", rq); + /* * If the device is marked as blocking (i.e. memory backed or zoned * device), the submission path may be blocked waiting for resources diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 2328cc05be36..3de11f077144 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -7199,7 +7199,6 @@ static void rbd_dev_remove_parent(struct rbd_device *rbd_dev) static ssize_t do_rbd_remove(const char *buf, size_t count) { struct rbd_device *rbd_dev = NULL; - struct list_head *tmp; int dev_id; char opt_buf[6]; bool force = false; @@ -7226,8 +7225,7 @@ static ssize_t do_rbd_remove(const char *buf, size_t count) ret = -ENOENT; spin_lock(&rbd_dev_list_lock); - list_for_each(tmp, &rbd_dev_list) { - rbd_dev = list_entry(tmp, struct rbd_device, node); + list_for_each_entry(rbd_dev, &rbd_dev_list, node) { if (rbd_dev->dev_id == dev_id) { ret = 0; break; diff --git a/drivers/cache/Kconfig b/drivers/cache/Kconfig new file mode 100644 index 000000000000..a57677f908f3 --- /dev/null +++ b/drivers/cache/Kconfig @@ -0,0 +1,11 @@ +# SPDX-License-Identifier: GPL-2.0 +menu "Cache Drivers" + +config AX45MP_L2_CACHE + bool "Andes Technology AX45MP L2 Cache controller" + depends on RISCV_DMA_NONCOHERENT + select RISCV_NONSTANDARD_CACHE_OPS + help + Support for the L2 cache controller on Andes Technology AX45MP platforms. + +endmenu diff --git a/drivers/cache/Makefile b/drivers/cache/Makefile new file mode 100644 index 000000000000..2012e7fb978d --- /dev/null +++ b/drivers/cache/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_AX45MP_L2_CACHE) += ax45mp_cache.o diff --git a/drivers/cache/ax45mp_cache.c b/drivers/cache/ax45mp_cache.c new file mode 100644 index 000000000000..57186c58dc84 --- /dev/null +++ b/drivers/cache/ax45mp_cache.c @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * non-coherent cache functions for Andes AX45MP + * + * Copyright (C) 2023 Renesas Electronics Corp. + */ + +#include +#include +#include +#include +#include + +#include + +/* L2 cache registers */ +#define AX45MP_L2C_REG_CTL_OFFSET 0x8 + +#define AX45MP_L2C_REG_C0_CMD_OFFSET 0x40 +#define AX45MP_L2C_REG_C0_ACC_OFFSET 0x48 +#define AX45MP_L2C_REG_STATUS_OFFSET 0x80 + +/* D-cache operation */ +#define AX45MP_CCTL_L1D_VA_INVAL 0 /* Invalidate an L1 cache entry */ +#define AX45MP_CCTL_L1D_VA_WB 1 /* Write-back an L1 cache entry */ + +/* L2 CCTL status */ +#define AX45MP_CCTL_L2_STATUS_IDLE 0 + +/* L2 CCTL status cores mask */ +#define AX45MP_CCTL_L2_STATUS_C0_MASK 0xf + +/* L2 cache operation */ +#define AX45MP_CCTL_L2_PA_INVAL 0x8 /* Invalidate an L2 cache entry */ +#define AX45MP_CCTL_L2_PA_WB 0x9 /* Write-back an L2 cache entry */ + +#define AX45MP_L2C_REG_PER_CORE_OFFSET 0x10 +#define AX45MP_CCTL_L2_STATUS_PER_CORE_OFFSET 4 + +#define AX45MP_L2C_REG_CN_CMD_OFFSET(n) \ + (AX45MP_L2C_REG_C0_CMD_OFFSET + ((n) * AX45MP_L2C_REG_PER_CORE_OFFSET)) +#define AX45MP_L2C_REG_CN_ACC_OFFSET(n) \ + (AX45MP_L2C_REG_C0_ACC_OFFSET + ((n) * AX45MP_L2C_REG_PER_CORE_OFFSET)) +#define AX45MP_CCTL_L2_STATUS_CN_MASK(n) \ + (AX45MP_CCTL_L2_STATUS_C0_MASK << ((n) * AX45MP_CCTL_L2_STATUS_PER_CORE_OFFSET)) + +#define AX45MP_CCTL_REG_UCCTLBEGINADDR_NUM 0x80b +#define AX45MP_CCTL_REG_UCCTLCOMMAND_NUM 0x80c + +#define AX45MP_CACHE_LINE_SIZE 64 + +struct ax45mp_priv { + void __iomem *l2c_base; + u32 ax45mp_cache_line_size; +}; + +static struct ax45mp_priv ax45mp_priv; + +/* L2 Cache operations */ +static inline uint32_t ax45mp_cpu_l2c_get_cctl_status(void) +{ + return readl(ax45mp_priv.l2c_base + AX45MP_L2C_REG_STATUS_OFFSET); +} + +static void ax45mp_cpu_cache_operation(unsigned long start, unsigned long end, + unsigned int l1_op, unsigned int l2_op) +{ + unsigned long line_size = ax45mp_priv.ax45mp_cache_line_size; + void __iomem *base = ax45mp_priv.l2c_base; + int mhartid = smp_processor_id(); + unsigned long pa; + + while (end > start) { + csr_write(AX45MP_CCTL_REG_UCCTLBEGINADDR_NUM, start); + csr_write(AX45MP_CCTL_REG_UCCTLCOMMAND_NUM, l1_op); + + pa = virt_to_phys((void *)start); + writel(pa, base + AX45MP_L2C_REG_CN_ACC_OFFSET(mhartid)); + writel(l2_op, base + AX45MP_L2C_REG_CN_CMD_OFFSET(mhartid)); + while ((ax45mp_cpu_l2c_get_cctl_status() & + AX45MP_CCTL_L2_STATUS_CN_MASK(mhartid)) != + AX45MP_CCTL_L2_STATUS_IDLE) + ; + + start += line_size; + } +} + +/* Write-back L1 and L2 cache entry */ +static inline void ax45mp_cpu_dcache_wb_range(unsigned long start, unsigned long end) +{ + ax45mp_cpu_cache_operation(start, end, AX45MP_CCTL_L1D_VA_WB, + AX45MP_CCTL_L2_PA_WB); +} + +/* Invalidate the L1 and L2 cache entry */ +static inline void ax45mp_cpu_dcache_inval_range(unsigned long start, unsigned long end) +{ + ax45mp_cpu_cache_operation(start, end, AX45MP_CCTL_L1D_VA_INVAL, + AX45MP_CCTL_L2_PA_INVAL); +} + +static void ax45mp_dma_cache_inv(phys_addr_t paddr, size_t size) +{ + unsigned long start = (unsigned long)phys_to_virt(paddr); + unsigned long end = start + size; + unsigned long line_size; + unsigned long flags; + + if (unlikely(start == end)) + return; + + line_size = ax45mp_priv.ax45mp_cache_line_size; + + start = start & (~(line_size - 1)); + end = ((end + line_size - 1) & (~(line_size - 1))); + + local_irq_save(flags); + + ax45mp_cpu_dcache_inval_range(start, end); + + local_irq_restore(flags); +} + +static void ax45mp_dma_cache_wback(phys_addr_t paddr, size_t size) +{ + unsigned long start = (unsigned long)phys_to_virt(paddr); + unsigned long end = start + size; + unsigned long line_size; + unsigned long flags; + + line_size = ax45mp_priv.ax45mp_cache_line_size; + start = start & (~(line_size - 1)); + local_irq_save(flags); + ax45mp_cpu_dcache_wb_range(start, end); + local_irq_restore(flags); +} + +static void ax45mp_dma_cache_wback_inv(phys_addr_t paddr, size_t size) +{ + ax45mp_dma_cache_wback(paddr, size); + ax45mp_dma_cache_inv(paddr, size); +} + +static int ax45mp_get_l2_line_size(struct device_node *np) +{ + int ret; + + ret = of_property_read_u32(np, "cache-line-size", &ax45mp_priv.ax45mp_cache_line_size); + if (ret) { + pr_err("Failed to get cache-line-size, defaulting to 64 bytes\n"); + return ret; + } + + if (ax45mp_priv.ax45mp_cache_line_size != AX45MP_CACHE_LINE_SIZE) { + pr_err("Expected cache-line-size to be 64 bytes (found:%u)\n", + ax45mp_priv.ax45mp_cache_line_size); + return -EINVAL; + } + + return 0; +} + +static const struct riscv_nonstd_cache_ops ax45mp_cmo_ops __initdata = { + .wback = &ax45mp_dma_cache_wback, + .inv = &ax45mp_dma_cache_inv, + .wback_inv = &ax45mp_dma_cache_wback_inv, +}; + +static const struct of_device_id ax45mp_cache_ids[] = { + { .compatible = "andestech,ax45mp-cache" }, + { /* sentinel */ } +}; + +static int __init ax45mp_cache_init(void) +{ + struct device_node *np; + struct resource res; + int ret; + + np = of_find_matching_node(NULL, ax45mp_cache_ids); + if (!of_device_is_available(np)) + return -ENODEV; + + ret = of_address_to_resource(np, 0, &res); + if (ret) + return ret; + + /* + * If IOCP is present on the Andes AX45MP core riscv_cbom_block_size + * will be 0 for sure, so we can definitely rely on it. If + * riscv_cbom_block_size = 0 we don't need to handle CMO using SW any + * more so we just return success here and only if its being set we + * continue further in the probe path. + */ + if (!riscv_cbom_block_size) + return 0; + + ax45mp_priv.l2c_base = ioremap(res.start, resource_size(&res)); + if (!ax45mp_priv.l2c_base) + return -ENOMEM; + + ret = ax45mp_get_l2_line_size(np); + if (ret) { + iounmap(ax45mp_priv.l2c_base); + return ret; + } + + riscv_noncoherent_register_cache_ops(&ax45mp_cmo_ops); + + return 0; +} +early_initcall(ax45mp_cache_init); diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index 92389a5481ff..a1157c2a7170 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -86,10 +86,10 @@ lib-$(CONFIG_EFI_GENERIC_STUB) += efi-stub.o string.o intrinsics.o systable.o \ screen_info.o efi-stub-entry.o lib-$(CONFIG_ARM) += arm32-stub.o -lib-$(CONFIG_ARM64) += arm64.o arm64-stub.o smbios.o +lib-$(CONFIG_ARM64) += kaslr.o arm64.o arm64-stub.o smbios.o lib-$(CONFIG_X86) += x86-stub.o lib-$(CONFIG_X86_64) += x86-5lvl.o -lib-$(CONFIG_RISCV) += riscv.o riscv-stub.o +lib-$(CONFIG_RISCV) += kaslr.o riscv.o riscv-stub.o lib-$(CONFIG_LOONGARCH) += loongarch.o loongarch-stub.o CFLAGS_arm32-stub.o := -DTEXT_OFFSET=$(TEXT_OFFSET) diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index 8c40fc89f5f9..452b7ccd330e 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -14,42 +14,6 @@ #include "efistub.h" -/* - * Distro versions of GRUB may ignore the BSS allocation entirely (i.e., fail - * to provide space, and fail to zero it). Check for this condition by double - * checking that the first and the last byte of the image are covered by the - * same EFI memory map entry. - */ -static bool check_image_region(u64 base, u64 size) -{ - struct efi_boot_memmap *map; - efi_status_t status; - bool ret = false; - int map_offset; - - status = efi_get_memory_map(&map, false); - if (status != EFI_SUCCESS) - return false; - - for (map_offset = 0; map_offset < map->map_size; map_offset += map->desc_size) { - efi_memory_desc_t *md = (void *)map->map + map_offset; - u64 end = md->phys_addr + md->num_pages * EFI_PAGE_SIZE; - - /* - * Find the region that covers base, and return whether - * it covers base+size bytes. - */ - if (base >= md->phys_addr && base < end) { - ret = (base + size) <= end; - break; - } - } - - efi_bs_call(free_pool, map); - - return ret; -} - efi_status_t handle_kernel_image(unsigned long *image_addr, unsigned long *image_size, unsigned long *reserve_addr, @@ -59,31 +23,6 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, { efi_status_t status; unsigned long kernel_size, kernel_codesize, kernel_memsize; - u32 phys_seed = 0; - u64 min_kimg_align = efi_get_kimg_min_align(); - - if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { - efi_guid_t li_fixed_proto = LINUX_EFI_LOADED_IMAGE_FIXED_GUID; - void *p; - - if (efi_nokaslr) { - efi_info("KASLR disabled on kernel command line\n"); - } else if (efi_bs_call(handle_protocol, image_handle, - &li_fixed_proto, &p) == EFI_SUCCESS) { - efi_info("Image placement fixed by loader\n"); - } else { - status = efi_get_random_bytes(sizeof(phys_seed), - (u8 *)&phys_seed); - if (status == EFI_NOT_FOUND) { - efi_info("EFI_RNG_PROTOCOL unavailable\n"); - efi_nokaslr = true; - } else if (status != EFI_SUCCESS) { - efi_err("efi_get_random_bytes() failed (0x%lx)\n", - status); - efi_nokaslr = true; - } - } - } if (image->image_base != _text) { efi_err("FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value\n"); @@ -98,50 +37,15 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, kernel_codesize = __inittext_end - _text; kernel_memsize = kernel_size + (_end - _edata); *reserve_size = kernel_memsize; + *image_addr = (unsigned long)_text; - if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && phys_seed != 0) { - /* - * If KASLR is enabled, and we have some randomness available, - * locate the kernel at a randomized offset in physical memory. - */ - status = efi_random_alloc(*reserve_size, min_kimg_align, - reserve_addr, phys_seed, - EFI_LOADER_CODE, EFI_ALLOC_LIMIT); - if (status != EFI_SUCCESS) - efi_warn("efi_random_alloc() failed: 0x%lx\n", status); - } else { - status = EFI_OUT_OF_RESOURCES; - } - - if (status != EFI_SUCCESS) { - if (!check_image_region((u64)_text, kernel_memsize)) { - efi_err("FIRMWARE BUG: Image BSS overlaps adjacent EFI memory region\n"); - } else if (IS_ALIGNED((u64)_text, min_kimg_align) && - (u64)_end < EFI_ALLOC_LIMIT) { - /* - * Just execute from wherever we were loaded by the - * UEFI PE/COFF loader if the placement is suitable. - */ - *image_addr = (u64)_text; - *reserve_size = 0; - return EFI_SUCCESS; - } - - status = efi_allocate_pages_aligned(*reserve_size, reserve_addr, - ULONG_MAX, min_kimg_align, - EFI_LOADER_CODE); - - if (status != EFI_SUCCESS) { - efi_err("Failed to relocate kernel\n"); - *reserve_size = 0; - return status; - } - } - - *image_addr = *reserve_addr; - memcpy((void *)*image_addr, _text, kernel_size); - caches_clean_inval_pou(*image_addr, *image_addr + kernel_codesize); - efi_remap_image(*image_addr, *reserve_size, kernel_codesize); + status = efi_kaslr_relocate_kernel(image_addr, + reserve_addr, reserve_size, + kernel_size, kernel_codesize, + kernel_memsize, + efi_kaslr_get_phys_seed(image_handle)); + if (status != EFI_SUCCESS) + return status; return EFI_SUCCESS; } @@ -159,3 +63,8 @@ unsigned long primary_entry_offset(void) */ return (char *)primary_entry - _text; } + +void efi_icache_sync(unsigned long start, unsigned long end) +{ + caches_clean_inval_pou(start, end); +} diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index 9823f6fb3e01..212687c30d79 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -1133,6 +1133,14 @@ const u8 *__efi_get_smbios_string(const struct efi_smbios_record *record, void efi_remap_image(unsigned long image_base, unsigned alloc_size, unsigned long code_size); +efi_status_t efi_kaslr_relocate_kernel(unsigned long *image_addr, + unsigned long *reserve_addr, + unsigned long *reserve_size, + unsigned long kernel_size, + unsigned long kernel_codesize, + unsigned long kernel_memsize, + u32 phys_seed); +u32 efi_kaslr_get_phys_seed(efi_handle_t image_handle); asmlinkage efi_status_t __efiapi efi_zboot_entry(efi_handle_t handle, efi_system_table_t *systab); diff --git a/drivers/firmware/efi/libstub/kaslr.c b/drivers/firmware/efi/libstub/kaslr.c new file mode 100644 index 000000000000..62d63f7a2645 --- /dev/null +++ b/drivers/firmware/efi/libstub/kaslr.c @@ -0,0 +1,159 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Helper functions used by the EFI stub on multiple + * architectures to deal with physical address space randomization. + */ +#include + +#include "efistub.h" + +/** + * efi_kaslr_get_phys_seed() - Get random seed for physical kernel KASLR + * @image_handle: Handle to the image + * + * If KASLR is not disabled, obtain a random seed using EFI_RNG_PROTOCOL + * that will be used to move the kernel physical mapping. + * + * Return: the random seed + */ +u32 efi_kaslr_get_phys_seed(efi_handle_t image_handle) +{ + efi_status_t status; + u32 phys_seed; + efi_guid_t li_fixed_proto = LINUX_EFI_LOADED_IMAGE_FIXED_GUID; + void *p; + + if (!IS_ENABLED(CONFIG_RANDOMIZE_BASE)) + return 0; + + if (efi_nokaslr) { + efi_info("KASLR disabled on kernel command line\n"); + } else if (efi_bs_call(handle_protocol, image_handle, + &li_fixed_proto, &p) == EFI_SUCCESS) { + efi_info("Image placement fixed by loader\n"); + } else { + status = efi_get_random_bytes(sizeof(phys_seed), + (u8 *)&phys_seed); + if (status == EFI_SUCCESS) { + return phys_seed; + } else if (status == EFI_NOT_FOUND) { + efi_info("EFI_RNG_PROTOCOL unavailable\n"); + efi_nokaslr = true; + } else if (status != EFI_SUCCESS) { + efi_err("efi_get_random_bytes() failed (0x%lx)\n", + status); + efi_nokaslr = true; + } + } + + return 0; +} + +/* + * Distro versions of GRUB may ignore the BSS allocation entirely (i.e., fail + * to provide space, and fail to zero it). Check for this condition by double + * checking that the first and the last byte of the image are covered by the + * same EFI memory map entry. + */ +static bool check_image_region(u64 base, u64 size) +{ + struct efi_boot_memmap *map; + efi_status_t status; + bool ret = false; + int map_offset; + + status = efi_get_memory_map(&map, false); + if (status != EFI_SUCCESS) + return false; + + for (map_offset = 0; map_offset < map->map_size; map_offset += map->desc_size) { + efi_memory_desc_t *md = (void *)map->map + map_offset; + u64 end = md->phys_addr + md->num_pages * EFI_PAGE_SIZE; + + /* + * Find the region that covers base, and return whether + * it covers base+size bytes. + */ + if (base >= md->phys_addr && base < end) { + ret = (base + size) <= end; + break; + } + } + + efi_bs_call(free_pool, map); + + return ret; +} + +/** + * efi_kaslr_relocate_kernel() - Relocate the kernel (random if KASLR enabled) + * @image_addr: Pointer to the current kernel location + * @reserve_addr: Pointer to the relocated kernel location + * @reserve_size: Size of the relocated kernel + * @kernel_size: Size of the text + data + * @kernel_codesize: Size of the text + * @kernel_memsize: Size of the text + data + bss + * @phys_seed: Random seed used for the relocation + * + * If KASLR is not enabled, this function relocates the kernel to a fixed + * address (or leave it as its current location). If KASLR is enabled, the + * kernel physical location is randomized using the seed in parameter. + * + * Return: status code, EFI_SUCCESS if relocation is successful + */ +efi_status_t efi_kaslr_relocate_kernel(unsigned long *image_addr, + unsigned long *reserve_addr, + unsigned long *reserve_size, + unsigned long kernel_size, + unsigned long kernel_codesize, + unsigned long kernel_memsize, + u32 phys_seed) +{ + efi_status_t status; + u64 min_kimg_align = efi_get_kimg_min_align(); + + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && phys_seed != 0) { + /* + * If KASLR is enabled, and we have some randomness available, + * locate the kernel at a randomized offset in physical memory. + */ + status = efi_random_alloc(*reserve_size, min_kimg_align, + reserve_addr, phys_seed, + EFI_LOADER_CODE, EFI_ALLOC_LIMIT); + if (status != EFI_SUCCESS) + efi_warn("efi_random_alloc() failed: 0x%lx\n", status); + } else { + status = EFI_OUT_OF_RESOURCES; + } + + if (status != EFI_SUCCESS) { + if (!check_image_region(*image_addr, kernel_memsize)) { + efi_err("FIRMWARE BUG: Image BSS overlaps adjacent EFI memory region\n"); + } else if (IS_ALIGNED(*image_addr, min_kimg_align) && + (unsigned long)_end < EFI_ALLOC_LIMIT) { + /* + * Just execute from wherever we were loaded by the + * UEFI PE/COFF loader if the placement is suitable. + */ + *reserve_size = 0; + return EFI_SUCCESS; + } + + status = efi_allocate_pages_aligned(*reserve_size, reserve_addr, + ULONG_MAX, min_kimg_align, + EFI_LOADER_CODE); + + if (status != EFI_SUCCESS) { + efi_err("Failed to relocate kernel\n"); + *reserve_size = 0; + return status; + } + } + + memcpy((void *)*reserve_addr, (void *)*image_addr, kernel_size); + *image_addr = *reserve_addr; + efi_icache_sync(*image_addr, *image_addr + kernel_codesize); + efi_remap_image(*image_addr, *reserve_size, kernel_codesize); + + return status; +} diff --git a/drivers/firmware/efi/libstub/riscv-stub.c b/drivers/firmware/efi/libstub/riscv-stub.c index 145c9f0ba217..c96d6dcee86c 100644 --- a/drivers/firmware/efi/libstub/riscv-stub.c +++ b/drivers/firmware/efi/libstub/riscv-stub.c @@ -30,32 +30,29 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, efi_loaded_image_t *image, efi_handle_t image_handle) { - unsigned long kernel_size = 0; - unsigned long preferred_addr; + unsigned long kernel_size, kernel_codesize, kernel_memsize; efi_status_t status; kernel_size = _edata - _start; + kernel_codesize = __init_text_end - _start; + kernel_memsize = kernel_size + (_end - _edata); *image_addr = (unsigned long)_start; - *image_size = kernel_size + (_end - _edata); - - /* - * RISC-V kernel maps PAGE_OFFSET virtual address to the same physical - * address where kernel is booted. That's why kernel should boot from - * as low as possible to avoid wastage of memory. Currently, dram_base - * is occupied by the firmware. So the preferred address for kernel to - * boot is next aligned address. If preferred address is not available, - * relocate_kernel will fall back to efi_low_alloc_above to allocate - * lowest possible memory region as long as the address and size meets - * the alignment constraints. - */ - preferred_addr = EFI_KIMG_PREFERRED_ADDRESS; - status = efi_relocate_kernel(image_addr, kernel_size, *image_size, - preferred_addr, efi_get_kimg_min_align(), - 0x0); + *image_size = kernel_memsize; + *reserve_size = *image_size; + status = efi_kaslr_relocate_kernel(image_addr, + reserve_addr, reserve_size, + kernel_size, kernel_codesize, kernel_memsize, + efi_kaslr_get_phys_seed(image_handle)); if (status != EFI_SUCCESS) { efi_err("Failed to relocate kernel\n"); *image_size = 0; } + return status; } + +void efi_icache_sync(unsigned long start, unsigned long end) +{ + asm volatile ("fence.i" ::: "memory"); +} diff --git a/drivers/gpio/gpio-zynq.c b/drivers/gpio/gpio-zynq.c index 0a7264aabe48..324e942c0650 100644 --- a/drivers/gpio/gpio-zynq.c +++ b/drivers/gpio/gpio-zynq.c @@ -575,6 +575,26 @@ static int zynq_gpio_set_wake(struct irq_data *data, unsigned int on) return 0; } +static int zynq_gpio_irq_reqres(struct irq_data *d) +{ + struct gpio_chip *chip = irq_data_get_irq_chip_data(d); + int ret; + + ret = pm_runtime_resume_and_get(chip->parent); + if (ret < 0) + return ret; + + return gpiochip_reqres_irq(chip, d->hwirq); +} + +static void zynq_gpio_irq_relres(struct irq_data *d) +{ + struct gpio_chip *chip = irq_data_get_irq_chip_data(d); + + gpiochip_relres_irq(chip, d->hwirq); + pm_runtime_put(chip->parent); +} + /* irq chip descriptor */ static const struct irq_chip zynq_gpio_level_irqchip = { .name = DRIVER_NAME, @@ -584,9 +604,10 @@ static const struct irq_chip zynq_gpio_level_irqchip = { .irq_unmask = zynq_gpio_irq_unmask, .irq_set_type = zynq_gpio_set_irq_type, .irq_set_wake = zynq_gpio_set_wake, + .irq_request_resources = zynq_gpio_irq_reqres, + .irq_release_resources = zynq_gpio_irq_relres, .flags = IRQCHIP_EOI_THREADED | IRQCHIP_EOI_IF_HANDLED | IRQCHIP_MASK_ON_SUSPEND | IRQCHIP_IMMUTABLE, - GPIOCHIP_IRQ_RESOURCE_HELPERS, }; static const struct irq_chip zynq_gpio_edge_irqchip = { @@ -597,8 +618,9 @@ static const struct irq_chip zynq_gpio_edge_irqchip = { .irq_unmask = zynq_gpio_irq_unmask, .irq_set_type = zynq_gpio_set_irq_type, .irq_set_wake = zynq_gpio_set_wake, + .irq_request_resources = zynq_gpio_irq_reqres, + .irq_release_resources = zynq_gpio_irq_relres, .flags = IRQCHIP_MASK_ON_SUSPEND | IRQCHIP_IMMUTABLE, - GPIOCHIP_IRQ_RESOURCE_HELPERS, }; static void zynq_gpio_handle_bank_irq(struct zynq_gpio *gpio, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c index df633e9ce920..cdf6087706aa 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c @@ -442,9 +442,7 @@ void amdgpu_amdkfd_get_local_mem_info(struct amdgpu_device *adev, mem_info->local_mem_size_public, mem_info->local_mem_size_private); - if (amdgpu_sriov_vf(adev)) - mem_info->mem_clk_max = adev->clock.default_mclk / 100; - else if (adev->pm.dpm_enabled) { + if (adev->pm.dpm_enabled) { if (amdgpu_emu_mode == 1) mem_info->mem_clk_max = 0; else @@ -463,9 +461,7 @@ uint64_t amdgpu_amdkfd_get_gpu_clock_counter(struct amdgpu_device *adev) uint32_t amdgpu_amdkfd_get_max_engine_clock_in_mhz(struct amdgpu_device *adev) { /* the sclk is in quantas of 10kHz */ - if (amdgpu_sriov_vf(adev)) - return adev->clock.default_sclk / 100; - else if (adev->pm.dpm_enabled) + if (adev->pm.dpm_enabled) return amdgpu_dpm_get_sclk(adev, false) / 100; else return 100; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.c index 835980e94b9e..fb2681dd6b33 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.c @@ -217,6 +217,7 @@ union umc_info { struct atom_umc_info_v3_1 v31; struct atom_umc_info_v3_2 v32; struct atom_umc_info_v3_3 v33; + struct atom_umc_info_v4_0 v40; }; union vram_info { @@ -508,9 +509,8 @@ bool amdgpu_atomfirmware_mem_ecc_supported(struct amdgpu_device *adev) if (amdgpu_atom_parse_data_header(mode_info->atom_context, index, &size, &frev, &crev, &data_offset)) { + umc_info = (union umc_info *)(mode_info->atom_context->bios + data_offset); if (frev == 3) { - umc_info = (union umc_info *) - (mode_info->atom_context->bios + data_offset); switch (crev) { case 1: umc_config = le32_to_cpu(umc_info->v31.umc_config); @@ -533,6 +533,20 @@ bool amdgpu_atomfirmware_mem_ecc_supported(struct amdgpu_device *adev) /* unsupported crev */ return false; } + } else if (frev == 4) { + switch (crev) { + case 0: + umc_config1 = le32_to_cpu(umc_info->v40.umc_config1); + ecc_default_enabled = + (umc_config1 & UMC_CONFIG1__ENABLE_ECC_CAPABLE) ? true : false; + break; + default: + /* unsupported crev */ + return false; + } + } else { + /* unsupported frev */ + return false; } } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c index 49dd9aa8da70..efdb1c48f431 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c @@ -127,7 +127,6 @@ static int amdgpu_cs_p1_user_fence(struct amdgpu_cs_parser *p, { struct drm_gem_object *gobj; unsigned long size; - int r; gobj = drm_gem_object_lookup(p->filp, data->handle); if (gobj == NULL) @@ -137,23 +136,14 @@ static int amdgpu_cs_p1_user_fence(struct amdgpu_cs_parser *p, drm_gem_object_put(gobj); size = amdgpu_bo_size(p->uf_bo); - if (size != PAGE_SIZE || (data->offset + 8) > size) { - r = -EINVAL; - goto error_unref; - } + if (size != PAGE_SIZE || data->offset > (size - 8)) + return -EINVAL; - if (amdgpu_ttm_tt_get_usermm(p->uf_bo->tbo.ttm)) { - r = -EINVAL; - goto error_unref; - } + if (amdgpu_ttm_tt_get_usermm(p->uf_bo->tbo.ttm)) + return -EINVAL; *offset = data->offset; - return 0; - -error_unref: - amdgpu_bo_unref(&p->uf_bo); - return r; } static int amdgpu_cs_p1_bo_handles(struct amdgpu_cs_parser *p, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index e77f048c99d8..3f001a50b34a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -885,13 +885,20 @@ static void amdgpu_block_invalid_wreg(struct amdgpu_device *adev, */ static int amdgpu_device_asic_init(struct amdgpu_device *adev) { + int ret; + amdgpu_asic_pre_asic_init(adev); if (adev->ip_versions[GC_HWIP][0] == IP_VERSION(9, 4, 3) || - adev->ip_versions[GC_HWIP][0] >= IP_VERSION(11, 0, 0)) - return amdgpu_atomfirmware_asic_init(adev, true); - else + adev->ip_versions[GC_HWIP][0] >= IP_VERSION(11, 0, 0)) { + amdgpu_psp_wait_for_bootloader(adev); + ret = amdgpu_atomfirmware_asic_init(adev, true); + return ret; + } else { return amdgpu_atom_asic_init(adev->mode_info.atom_context); + } + + return 0; } /** @@ -4694,9 +4701,12 @@ int amdgpu_device_mode1_reset(struct amdgpu_device *adev) } if (ret) - dev_err(adev->dev, "GPU mode1 reset failed\n"); + goto mode1_reset_failed; amdgpu_device_load_pci_state(adev->pdev); + ret = amdgpu_psp_wait_for_bootloader(adev); + if (ret) + goto mode1_reset_failed; /* wait for asic to come out of reset */ for (i = 0; i < adev->usec_timeout; i++) { @@ -4707,7 +4717,17 @@ int amdgpu_device_mode1_reset(struct amdgpu_device *adev) udelay(1); } + if (i >= adev->usec_timeout) { + ret = -ETIMEDOUT; + goto mode1_reset_failed; + } + amdgpu_atombios_scratch_regs_engine_hung(adev, false); + + return 0; + +mode1_reset_failed: + dev_err(adev->dev, "GPU mode1 reset failed\n"); return ret; } @@ -4849,7 +4869,7 @@ static void amdgpu_reset_capture_coredumpm(struct amdgpu_device *adev) struct drm_device *dev = adev_to_drm(adev); ktime_get_ts64(&adev->reset_time); - dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_KERNEL, + dev_coredumpm(dev->dev, THIS_MODULE, adev, 0, GFP_NOWAIT, amdgpu_devcoredump_read, amdgpu_devcoredump_free); } #endif diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c index 74ffe6581c85..7d5e7ad28ba8 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c @@ -1390,6 +1390,7 @@ union gc_info { struct gc_info_v1_1 v1_1; struct gc_info_v1_2 v1_2; struct gc_info_v2_0 v2; + struct gc_info_v2_1 v2_1; }; static int amdgpu_discovery_get_gfx_info(struct amdgpu_device *adev) @@ -1465,6 +1466,15 @@ static int amdgpu_discovery_get_gfx_info(struct amdgpu_device *adev) adev->gfx.config.num_sc_per_sh = le32_to_cpu(gc_info->v2.gc_num_sc_per_se) / le32_to_cpu(gc_info->v2.gc_num_sh_per_se); adev->gfx.config.num_packer_per_sc = le32_to_cpu(gc_info->v2.gc_num_packer_per_sc); + if (gc_info->v2.header.version_minor == 1) { + adev->gfx.config.gc_num_tcp_per_sa = le32_to_cpu(gc_info->v2_1.gc_num_tcp_per_sh); + adev->gfx.config.gc_tcp_size_per_cu = le32_to_cpu(gc_info->v2_1.gc_tcp_size_per_cu); + adev->gfx.config.gc_num_sdp_interface = le32_to_cpu(gc_info->v2_1.gc_num_sdp_interface); /* per XCD */ + adev->gfx.config.gc_num_cu_per_sqc = le32_to_cpu(gc_info->v2_1.gc_num_cu_per_sqc); + adev->gfx.config.gc_l1_instruction_cache_size_per_sqc = le32_to_cpu(gc_info->v2_1.gc_instruction_cache_size_per_sqc); + adev->gfx.config.gc_l1_data_cache_size_per_sqc = le32_to_cpu(gc_info->v2_1.gc_scalar_data_cache_size_per_sqc); + adev->gfx.config.gc_tcc_size = le32_to_cpu(gc_info->v2_1.gc_tcc_size); /* per XCD */ + } break; default: dev_err(adev->dev, @@ -1478,6 +1488,7 @@ static int amdgpu_discovery_get_gfx_info(struct amdgpu_device *adev) union mall_info { struct mall_info_v1_0 v1; + struct mall_info_v2_0 v2; }; static int amdgpu_discovery_get_mall_info(struct amdgpu_device *adev) @@ -1518,6 +1529,10 @@ static int amdgpu_discovery_get_mall_info(struct amdgpu_device *adev) adev->gmc.mall_size = mall_size; adev->gmc.m_half_use = half_use; break; + case 2: + mall_size_per_umc = le32_to_cpu(mall_info->v2.mall_size_per_umc); + adev->gmc.mall_size = mall_size_per_umc * adev->gmc.num_umc; + break; default: dev_err(adev->dev, "Unhandled MALL info table %d.%d\n", diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_display.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_display.c index d20dd3f852fc..363e6a2cad8c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_display.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_display.c @@ -38,6 +38,8 @@ #include #include #include +#include +#include #include #include #include @@ -532,11 +534,29 @@ bool amdgpu_display_ddc_probe(struct amdgpu_connector *amdgpu_connector, return true; } +static int amdgpu_dirtyfb(struct drm_framebuffer *fb, struct drm_file *file, + unsigned int flags, unsigned int color, + struct drm_clip_rect *clips, unsigned int num_clips) +{ + + if (file) + return -ENOSYS; + + return drm_atomic_helper_dirtyfb(fb, file, flags, color, clips, + num_clips); +} + static const struct drm_framebuffer_funcs amdgpu_fb_funcs = { .destroy = drm_gem_fb_destroy, .create_handle = drm_gem_fb_create_handle, }; +static const struct drm_framebuffer_funcs amdgpu_fb_funcs_atomic = { + .destroy = drm_gem_fb_destroy, + .create_handle = drm_gem_fb_create_handle, + .dirty = amdgpu_dirtyfb +}; + uint32_t amdgpu_display_supported_domains(struct amdgpu_device *adev, uint64_t bo_flags) { @@ -1139,7 +1159,11 @@ static int amdgpu_display_gem_fb_verify_and_init(struct drm_device *dev, if (ret) goto err; - ret = drm_framebuffer_init(dev, &rfb->base, &amdgpu_fb_funcs); + if (drm_drv_uses_atomic_modeset(dev)) + ret = drm_framebuffer_init(dev, &rfb->base, + &amdgpu_fb_funcs_atomic); + else + ret = drm_framebuffer_init(dev, &rfb->base, &amdgpu_fb_funcs); if (ret) goto err; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h index a4ff515ce896..395c1768b9fc 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h @@ -241,6 +241,9 @@ struct amdgpu_gfx_config { uint32_t gc_gl1c_per_sa; uint32_t gc_gl1c_size_per_instance; uint32_t gc_gl2c_per_gpu; + uint32_t gc_tcp_size_per_cu; + uint32_t gc_num_cu_per_sqc; + uint32_t gc_tcc_size; }; struct amdgpu_cu_info { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c index 8fdca54bb8a1..429ef212c1f2 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c @@ -2078,6 +2078,17 @@ int psp_securedisplay_invoke(struct psp_context *psp, uint32_t ta_cmd_id) } /* SECUREDISPLAY end */ +int amdgpu_psp_wait_for_bootloader(struct amdgpu_device *adev) +{ + struct psp_context *psp = &adev->psp; + int ret = 0; + + if (!amdgpu_sriov_vf(adev) && psp->funcs && psp->funcs->wait_for_bootloader != NULL) + ret = psp->funcs->wait_for_bootloader(psp); + + return ret; +} + static int psp_hw_start(struct psp_context *psp) { struct amdgpu_device *adev = psp->adev; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h index 3384eb94fde0..3e67ed63e638 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h @@ -109,6 +109,7 @@ enum psp_reg_prog_id { struct psp_funcs { int (*init_microcode)(struct psp_context *psp); + int (*wait_for_bootloader)(struct psp_context *psp); int (*bootloader_load_kdb)(struct psp_context *psp); int (*bootloader_load_spl)(struct psp_context *psp); int (*bootloader_load_sysdrv)(struct psp_context *psp); @@ -533,4 +534,6 @@ int psp_spatial_partition(struct psp_context *psp, int mode); int is_psp_fw_valid(struct psp_bin_desc bin); +int amdgpu_psp_wait_for_bootloader(struct amdgpu_device *adev); + #endif diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c index 7689395e44fd..3c4600e15b86 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c @@ -764,7 +764,7 @@ int amdgpu_ras_feature_enable(struct amdgpu_device *adev, { struct amdgpu_ras *con = amdgpu_ras_get_context(adev); union ta_ras_cmd_input *info; - int ret = 0; + int ret; if (!con) return -EINVAL; @@ -773,7 +773,7 @@ int amdgpu_ras_feature_enable(struct amdgpu_device *adev, if (enable && head->block != AMDGPU_RAS_BLOCK__GFX && !amdgpu_ras_is_feature_allowed(adev, head)) - goto out; + return 0; /* Only enable gfx ras feature from host side */ if (head->block == AMDGPU_RAS_BLOCK__GFX && @@ -801,16 +801,16 @@ int amdgpu_ras_feature_enable(struct amdgpu_device *adev, enable ? "enable":"disable", get_ras_block_str(head), amdgpu_ras_is_poison_mode_supported(adev), ret); - goto out; + return ret; } + + kfree(info); } /* setup the obj */ __amdgpu_ras_feature_enable(adev, head, enable); -out: - if (head->block == AMDGPU_RAS_BLOCK__GFX) - kfree(info); - return ret; + + return 0; } /* Only used in device probe stage and called only once. */ @@ -2399,6 +2399,7 @@ static bool amdgpu_ras_asic_supported(struct amdgpu_device *adev) if (amdgpu_sriov_vf(adev)) { switch (adev->ip_versions[MP0_HWIP][0]) { case IP_VERSION(13, 0, 2): + case IP_VERSION(13, 0, 6): return true; default: return false; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c index 4764d2171f92..595d5e535aca 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c @@ -158,9 +158,10 @@ static bool __is_ras_eeprom_supported(struct amdgpu_device *adev) case IP_VERSION(11, 0, 7): /* Sienna cichlid */ case IP_VERSION(13, 0, 0): case IP_VERSION(13, 0, 2): /* Aldebaran */ - case IP_VERSION(13, 0, 6): case IP_VERSION(13, 0, 10): return true; + case IP_VERSION(13, 0, 6): + return (adev->gmc.is_app_apu) ? false : true; default: return false; } diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c index 57ed4e5c294c..0a26a00074a6 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c @@ -203,6 +203,9 @@ static void gfx_v9_4_3_init_golden_registers(struct amdgpu_device *adev) if (adev->rev_id == 0) { WREG32_FIELD15_PREREG(GC, dev_inst, TCP_UTCL1_CNTL1, REDUCE_FIFO_DEPTH_BY_2, 2); + } else { + WREG32_FIELD15_PREREG(GC, dev_inst, TCP_UTCL1_CNTL2, + SPARE, 0x1); } } } @@ -860,11 +863,15 @@ static int gfx_v9_4_3_sw_init(void *handle) if (r) return r; - r = amdgpu_gfx_sysfs_init(adev); + r = amdgpu_gfx_ras_sw_init(adev); if (r) return r; - return amdgpu_gfx_ras_sw_init(adev); + + if (!amdgpu_sriov_vf(adev)) + r = amdgpu_gfx_sysfs_init(adev); + + return r; } static int gfx_v9_4_3_sw_fini(void *handle) @@ -885,7 +892,8 @@ static int gfx_v9_4_3_sw_fini(void *handle) gfx_v9_4_3_mec_fini(adev); amdgpu_bo_unref(&adev->gfx.rlc.clear_state_obj); gfx_v9_4_3_free_microcode(adev); - amdgpu_gfx_sysfs_fini(adev); + if (!amdgpu_sriov_vf(adev)) + amdgpu_gfx_sysfs_fini(adev); return 0; } @@ -2219,15 +2227,6 @@ static void gfx_v9_4_3_xcc_update_sram_fgcg(struct amdgpu_device *adev, WREG32_SOC15(GC, GET_INST(GC, xcc_id), regRLC_CGTT_MGCG_OVERRIDE, data); - def = data = RREG32_SOC15(GC, GET_INST(GC, xcc_id), regRLC_CLK_CNTL); - - if (enable) - data &= ~RLC_CLK_CNTL__RLC_SRAM_CLK_GATER_OVERRIDE_MASK; - else - data |= RLC_CLK_CNTL__RLC_SRAM_CLK_GATER_OVERRIDE_MASK; - - if (def != data) - WREG32_SOC15(GC, GET_INST(GC, xcc_id), regRLC_CLK_CNTL, data); } static void gfx_v9_4_3_xcc_update_repeater_fgcg(struct amdgpu_device *adev, @@ -4048,7 +4047,8 @@ static void gfx_v9_4_3_inst_enable_watchdog_timer(struct amdgpu_device *adev, uint32_t i; uint32_t data; - data = REG_SET_FIELD(0, SQ_TIMEOUT_CONFIG, TIMEOUT_FATAL_DISABLE, + data = RREG32_SOC15(GC, GET_INST(GC, 0), regSQ_TIMEOUT_CONFIG); + data = REG_SET_FIELD(data, SQ_TIMEOUT_CONFIG, TIMEOUT_FATAL_DISABLE, amdgpu_watchdog_timer.timeout_fatal_disable ? 1 : 0); if (amdgpu_watchdog_timer.timeout_fatal_disable && diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c index 15612915bb6c..1de79d660285 100644 --- a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c +++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c @@ -360,8 +360,10 @@ static int jpeg_v4_0_3_hw_fini(void *handle) cancel_delayed_work_sync(&adev->jpeg.idle_work); - if (adev->jpeg.cur_state != AMD_PG_STATE_GATE) - ret = jpeg_v4_0_3_set_powergating_state(adev, AMD_PG_STATE_GATE); + if (!amdgpu_sriov_vf(adev)) { + if (adev->jpeg.cur_state != AMD_PG_STATE_GATE) + ret = jpeg_v4_0_3_set_powergating_state(adev, AMD_PG_STATE_GATE); + } return ret; } diff --git a/drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c b/drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c index 9ea072374cb7..f85eec05d218 100644 --- a/drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c +++ b/drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c @@ -437,6 +437,24 @@ static void nbio_v7_9_init_registers(struct amdgpu_device *adev) XCC_DOORBELL_FENCE__SHUB_SLV_MODE_MASK); } + + if (!amdgpu_sriov_vf(adev)) { + u32 baco_cntl; + for_each_inst(i, adev->aid_mask) { + baco_cntl = RREG32_SOC15(NBIO, i, regBIF_BX0_BACO_CNTL); + if (baco_cntl & (BIF_BX0_BACO_CNTL__BACO_DUMMY_EN_MASK | + BIF_BX0_BACO_CNTL__BACO_EN_MASK)) { + baco_cntl &= ~( + BIF_BX0_BACO_CNTL__BACO_DUMMY_EN_MASK | + BIF_BX0_BACO_CNTL__BACO_EN_MASK); + dev_dbg(adev->dev, + "Unsetting baco dummy mode %x", + baco_cntl); + WREG32_SOC15(NBIO, i, regBIF_BX0_BACO_CNTL, + baco_cntl); + } + } + } } static u64 nbio_v7_9_get_pcie_replay_count(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c b/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c index 10b17bd5aebe..469eed084976 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c +++ b/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c @@ -133,12 +133,32 @@ static bool psp_v13_0_is_sos_alive(struct psp_context *psp) return sol_reg != 0x0; } +static int psp_v13_0_wait_for_vmbx_ready(struct psp_context *psp) +{ + struct amdgpu_device *adev = psp->adev; + int retry_loop, ret; + + for (retry_loop = 0; retry_loop < 70; retry_loop++) { + /* Wait for bootloader to signify that is + ready having bit 31 of C2PMSG_33 set to 1 */ + ret = psp_wait_for( + psp, SOC15_REG_OFFSET(MP0, 0, regMP0_SMN_C2PMSG_33), + 0x80000000, 0xffffffff, false); + + if (ret == 0) + break; + } + + if (ret) + dev_warn(adev->dev, "Bootloader wait timed out"); + + return ret; +} + static int psp_v13_0_wait_for_bootloader(struct psp_context *psp) { struct amdgpu_device *adev = psp->adev; - - int ret; - int retry_loop; + int retry_loop, ret; /* Wait for bootloader to signify that it is ready having bit 31 of * C2PMSG_35 set to 1. All other bits are expected to be cleared. @@ -157,6 +177,19 @@ static int psp_v13_0_wait_for_bootloader(struct psp_context *psp) return ret; } +static int psp_v13_0_wait_for_bootloader_steady_state(struct psp_context *psp) +{ + struct amdgpu_device *adev = psp->adev; + + if (adev->ip_versions[MP0_HWIP][0] == IP_VERSION(13, 0, 6)) { + psp_v13_0_wait_for_vmbx_ready(psp); + + return psp_v13_0_wait_for_bootloader(psp); + } + + return 0; +} + static int psp_v13_0_bootloader_load_component(struct psp_context *psp, struct psp_bin_desc *bin_desc, enum psp_bootloader_cmd bl_cmd) @@ -714,6 +747,7 @@ static int psp_v13_0_fatal_error_recovery_quirk(struct psp_context *psp) static const struct psp_funcs psp_v13_0_funcs = { .init_microcode = psp_v13_0_init_microcode, + .wait_for_bootloader = psp_v13_0_wait_for_bootloader_steady_state, .bootloader_load_kdb = psp_v13_0_bootloader_load_kdb, .bootloader_load_spl = psp_v13_0_bootloader_load_spl, .bootloader_load_sysdrv = psp_v13_0_bootloader_load_sysdrv, diff --git a/drivers/gpu/drm/amd/amdgpu/soc15.c b/drivers/gpu/drm/amd/amdgpu/soc15.c index c45721ca916e..f5be40d7ba36 100644 --- a/drivers/gpu/drm/amd/amdgpu/soc15.c +++ b/drivers/gpu/drm/amd/amdgpu/soc15.c @@ -559,8 +559,10 @@ soc15_asic_reset_method(struct amdgpu_device *adev) */ if (amdgpu_gpu_recovery == 4 || amdgpu_gpu_recovery == 5) return AMD_RESET_METHOD_MODE2; + else if (!(adev->flags & AMD_IS_APU)) + return AMD_RESET_METHOD_MODE1; else - return AMD_RESET_METHOD_NONE; + return AMD_RESET_METHOD_MODE2; default: break; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c b/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c index f0731a6a5306..830396b1c3b1 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c @@ -384,7 +384,7 @@ static void event_interrupt_wq_v9(struct kfd_node *dev, default: break; } - kfd_signal_event_interrupt(pasid, context_id0 & 0xffffff, 24); + kfd_signal_event_interrupt(pasid, sq_int_data, 24); } else if (source_id == SOC15_INTSRC_CP_BAD_OPCODE) { kfd_set_dbg_ev_from_interrupt(dev, pasid, KFD_DEBUG_DOORBELL_ID(context_id0), diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v11.c b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v11.c index 2319467d2d95..0bbf0edbabd4 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v11.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v11.c @@ -457,6 +457,7 @@ struct mqd_manager *mqd_manager_init_v11(enum KFD_MQD_TYPE type, mqd->is_occupied = kfd_is_occupied_cp; mqd->mqd_size = sizeof(struct v11_compute_mqd); mqd->get_wave_state = get_wave_state; + mqd->mqd_stride = kfd_mqd_stride; #if defined(CONFIG_DEBUG_FS) mqd->debugfs_show_mqd = debugfs_show_mqd; #endif @@ -472,6 +473,7 @@ struct mqd_manager *mqd_manager_init_v11(enum KFD_MQD_TYPE type, mqd->destroy_mqd = destroy_hiq_mqd; mqd->is_occupied = kfd_is_occupied_cp; mqd->mqd_size = sizeof(struct v11_compute_mqd); + mqd->mqd_stride = kfd_mqd_stride; #if defined(CONFIG_DEBUG_FS) mqd->debugfs_show_mqd = debugfs_show_mqd; #endif @@ -501,6 +503,7 @@ struct mqd_manager *mqd_manager_init_v11(enum KFD_MQD_TYPE type, mqd->destroy_mqd = kfd_destroy_mqd_sdma; mqd->is_occupied = kfd_is_occupied_sdma; mqd->mqd_size = sizeof(struct v11_sdma_mqd); + mqd->mqd_stride = kfd_mqd_stride; #if defined(CONFIG_DEBUG_FS) mqd->debugfs_show_mqd = debugfs_show_mqd_sdma; #endif diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 011561605983..bb16b795d1bc 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1686,6 +1686,8 @@ static int svm_range_validate_and_map(struct mm_struct *mm, WRITE_ONCE(p->svms.faulting_task, NULL); if (r) { pr_debug("failed %d to get svm range pages\n", r); + if (r == -EBUSY) + r = -EAGAIN; goto unreserve_out; } diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c index 268cb99a4c4b..88ba8b66de1f 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c @@ -65,6 +65,7 @@ #include "amdgpu_dm_debugfs.h" #endif #include "amdgpu_dm_psr.h" +#include "amdgpu_dm_replay.h" #include "ivsrcid/ivsrcid_vislands30.h" @@ -4265,6 +4266,7 @@ static int amdgpu_dm_initialize_drm_device(struct amdgpu_device *adev) enum dc_connection_type new_connection_type = dc_connection_none; const struct dc_plane_cap *plane; bool psr_feature_enabled = false; + bool replay_feature_enabled = false; int max_overlay = dm->dc->caps.max_slave_planes; dm->display_indexes_num = dm->dc->caps.max_streams; @@ -4374,6 +4376,20 @@ static int amdgpu_dm_initialize_drm_device(struct amdgpu_device *adev) } } + if (!(amdgpu_dc_debug_mask & DC_DISABLE_REPLAY)) { + switch (adev->ip_versions[DCE_HWIP][0]) { + case IP_VERSION(3, 1, 4): + case IP_VERSION(3, 1, 5): + case IP_VERSION(3, 1, 6): + case IP_VERSION(3, 2, 0): + case IP_VERSION(3, 2, 1): + replay_feature_enabled = true; + break; + default: + replay_feature_enabled = amdgpu_dc_feature_mask & DC_REPLAY_MASK; + break; + } + } /* loops over all connectors on the board */ for (i = 0; i < link_cnt; i++) { struct dc_link *link = NULL; @@ -4422,6 +4438,12 @@ static int amdgpu_dm_initialize_drm_device(struct amdgpu_device *adev) amdgpu_dm_update_connector_after_detect(aconnector); setup_backlight_device(dm, aconnector); + /* + * Disable psr if replay can be enabled + */ + if (replay_feature_enabled && amdgpu_dm_setup_replay(link, aconnector)) + psr_feature_enabled = false; + if (psr_feature_enabled) amdgpu_dm_set_psr_caps(link); @@ -6004,7 +6026,7 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, if (recalculate_timing) drm_mode_set_crtcinfo(&saved_mode, 0); - else + else if (!old_stream) drm_mode_set_crtcinfo(&mode, 0); /* diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c index 30d4c6fd95f5..97b7a0b8a1c2 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c @@ -29,6 +29,7 @@ #include "dc.h" #include "amdgpu.h" #include "amdgpu_dm_psr.h" +#include "amdgpu_dm_replay.h" #include "amdgpu_dm_crtc.h" #include "amdgpu_dm_plane.h" #include "amdgpu_dm_trace.h" @@ -123,7 +124,12 @@ static void vblank_control_worker(struct work_struct *work) * fill_dc_dirty_rects(). */ if (vblank_work->stream && vblank_work->stream->link) { - if (vblank_work->enable) { + /* + * Prioritize replay, instead of psr + */ + if (vblank_work->stream->link->replay_settings.replay_feature_enabled) + amdgpu_dm_replay_enable(vblank_work->stream, false); + else if (vblank_work->enable) { if (vblank_work->stream->link->psr_settings.psr_version < DC_PSR_VERSION_SU_1 && vblank_work->stream->link->psr_settings.psr_allow_active) amdgpu_dm_psr_disable(vblank_work->stream); @@ -132,6 +138,7 @@ static void vblank_control_worker(struct work_struct *work) #ifdef CONFIG_DRM_AMD_SECURE_DISPLAY !amdgpu_dm_crc_window_is_activated(&vblank_work->acrtc->base) && #endif + vblank_work->stream->link->panel_config.psr.disallow_replay && vblank_work->acrtc->dm_irq_params.allow_psr_entry) { amdgpu_dm_psr_enable(vblank_work->stream); } diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c index 8eeca160d434..cc74dd69acf2 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1269,6 +1269,13 @@ void amdgpu_dm_plane_handle_cursor_update(struct drm_plane *plane, attributes.rotation_angle = 0; attributes.attribute_flags.value = 0; + /* Enable cursor degamma ROM on DCN3+ for implicit sRGB degamma in DRM + * legacy gamma setup. + */ + if (crtc_state->cm_is_degamma_srgb && + adev->dm.dc->caps.color.dpp.gamma_corr) + attributes.attribute_flags.bits.ENABLE_CURSOR_DEGAMMA = 1; + attributes.pitch = afb->base.pitches[0] / afb->base.format->cpp[0]; if (crtc_state->stream) { @@ -1468,6 +1475,15 @@ int amdgpu_dm_plane_init(struct amdgpu_display_manager *dm, drm_plane_create_blend_mode_property(plane, blend_caps); } + if (plane->type == DRM_PLANE_TYPE_PRIMARY) { + drm_plane_create_zpos_immutable_property(plane, 0); + } else if (plane->type == DRM_PLANE_TYPE_OVERLAY) { + unsigned int zpos = 1 + drm_plane_index(plane); + drm_plane_create_zpos_property(plane, zpos, 1, 254); + } else if (plane->type == DRM_PLANE_TYPE_CURSOR) { + drm_plane_create_zpos_immutable_property(plane, 255); + } + if (plane->type == DRM_PLANE_TYPE_PRIMARY && plane_cap && (plane_cap->pixel_format_support.nv12 || diff --git a/drivers/gpu/drm/amd/display/dc/Makefile b/drivers/gpu/drm/amd/display/dc/Makefile index 69ffd4424dc7..1b8c2aef4633 100644 --- a/drivers/gpu/drm/amd/display/dc/Makefile +++ b/drivers/gpu/drm/amd/display/dc/Makefile @@ -78,3 +78,4 @@ DC_EDID += dc_edid_parser.o AMD_DISPLAY_DMUB = $(addprefix $(AMDDALPATH)/dc/,$(DC_DMUB)) AMD_DISPLAY_EDID = $(addprefix $(AMDDALPATH)/dc/,$(DC_EDID)) AMD_DISPLAY_FILES += $(AMD_DISPLAY_DMUB) $(AMD_DISPLAY_EDID) + diff --git a/drivers/gpu/drm/amd/display/dc/clk_mgr/dcn315/dcn315_smu.c b/drivers/gpu/drm/amd/display/dc/clk_mgr/dcn315/dcn315_smu.c index 3e0da873cf4c..1042cf1a3ab0 100644 --- a/drivers/gpu/drm/amd/display/dc/clk_mgr/dcn315/dcn315_smu.c +++ b/drivers/gpu/drm/amd/display/dc/clk_mgr/dcn315/dcn315_smu.c @@ -32,6 +32,7 @@ #define MAX_INSTANCE 6 #define MAX_SEGMENT 6 +#define SMU_REGISTER_WRITE_RETRY_COUNT 5 struct IP_BASE_INSTANCE { unsigned int segment[MAX_SEGMENT]; @@ -132,6 +133,8 @@ static int dcn315_smu_send_msg_with_param( unsigned int msg_id, unsigned int param) { uint32_t result; + uint32_t i = 0; + uint32_t read_back_data; result = dcn315_smu_wait_for_response(clk_mgr, 10, 200000); @@ -148,10 +151,19 @@ static int dcn315_smu_send_msg_with_param( /* Set the parameter register for the SMU message, unit is Mhz */ REG_WRITE(MP1_SMN_C2PMSG_37, param); - /* Trigger the message transaction by writing the message ID */ - generic_write_indirect_reg(CTX, - REG_NBIO(RSMU_INDEX), REG_NBIO(RSMU_DATA), - mmMP1_C2PMSG_3, msg_id); + for (i = 0; i < SMU_REGISTER_WRITE_RETRY_COUNT; i++) { + /* Trigger the message transaction by writing the message ID */ + generic_write_indirect_reg(CTX, + REG_NBIO(RSMU_INDEX), REG_NBIO(RSMU_DATA), + mmMP1_C2PMSG_3, msg_id); + read_back_data = generic_read_indirect_reg(CTX, + REG_NBIO(RSMU_INDEX), REG_NBIO(RSMU_DATA), + mmMP1_C2PMSG_3); + if (read_back_data == msg_id) + break; + udelay(2); + smu_print("SMU msg id write fail %x times. \n", i + 1); + } result = dcn315_smu_wait_for_response(clk_mgr, 10, 200000); diff --git a/drivers/gpu/drm/amd/display/dc/core/dc.c b/drivers/gpu/drm/amd/display/dc/core/dc.c index 566d7045b2de..3a9077b60029 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc.c @@ -2073,12 +2073,12 @@ enum dc_status dc_commit_streams(struct dc *dc, } } - /* Check for case where we are going from odm 2:1 to max - * pipe scenario. For these cases, we will call - * commit_minimal_transition_state() to exit out of odm 2:1 - * first before processing new streams + /* ODM Combine 2:1 power optimization is only applied for single stream + * scenario, it uses extra pipes than needed to reduce power consumption + * We need to switch off this feature to make room for new streams. */ - if (stream_count == dc->res_pool->pipe_count) { + if (stream_count > dc->current_state->stream_count && + dc->current_state->stream_count == 1) { for (i = 0; i < dc->res_pool->pipe_count; i++) { pipe = &dc->current_state->res_ctx.pipe_ctx[i]; if (pipe->next_odm_pipe) @@ -3501,6 +3501,45 @@ static void commit_planes_for_stream_fast(struct dc *dc, top_pipe_to_program->stream->update_flags.raw = 0; } +static void wait_for_outstanding_hw_updates(struct dc *dc, const struct dc_state *dc_context) +{ +/* + * This function calls HWSS to wait for any potentially double buffered + * operations to complete. It should be invoked as a pre-amble prior + * to full update programming before asserting any HW locks. + */ + int pipe_idx; + int opp_inst; + int opp_count = dc->res_pool->pipe_count; + struct hubp *hubp; + int mpcc_inst; + const struct pipe_ctx *pipe_ctx; + + for (pipe_idx = 0; pipe_idx < dc->res_pool->pipe_count; pipe_idx++) { + pipe_ctx = &dc_context->res_ctx.pipe_ctx[pipe_idx]; + + if (!pipe_ctx->stream) + continue; + + if (pipe_ctx->stream_res.tg->funcs->wait_drr_doublebuffer_pending_clear) + pipe_ctx->stream_res.tg->funcs->wait_drr_doublebuffer_pending_clear(pipe_ctx->stream_res.tg); + + hubp = pipe_ctx->plane_res.hubp; + if (!hubp) + continue; + + mpcc_inst = hubp->inst; + // MPCC inst is equal to pipe index in practice + for (opp_inst = 0; opp_inst < opp_count; opp_inst++) { + if (dc->res_pool->opps[opp_inst]->mpcc_disconnect_pending[mpcc_inst]) { + dc->res_pool->mpc->funcs->wait_for_idle(dc->res_pool->mpc, mpcc_inst); + dc->res_pool->opps[opp_inst]->mpcc_disconnect_pending[mpcc_inst] = false; + break; + } + } + } +} + static void commit_planes_for_stream(struct dc *dc, struct dc_surface_update *srf_updates, int surface_count, @@ -3519,24 +3558,9 @@ static void commit_planes_for_stream(struct dc *dc, // dc->current_state anymore, so we have to cache it before we apply // the new SubVP context subvp_prev_use = false; - - dc_z10_restore(dc); - - if (update_type == UPDATE_TYPE_FULL) { - /* wait for all double-buffer activity to clear on all pipes */ - int pipe_idx; - - for (pipe_idx = 0; pipe_idx < dc->res_pool->pipe_count; pipe_idx++) { - struct pipe_ctx *pipe_ctx = &context->res_ctx.pipe_ctx[pipe_idx]; - - if (!pipe_ctx->stream) - continue; - - if (pipe_ctx->stream_res.tg->funcs->wait_drr_doublebuffer_pending_clear) - pipe_ctx->stream_res.tg->funcs->wait_drr_doublebuffer_pending_clear(pipe_ctx->stream_res.tg); - } - } + if (update_type == UPDATE_TYPE_FULL) + wait_for_outstanding_hw_updates(dc, context); if (update_type == UPDATE_TYPE_FULL) { dc_allow_idle_optimizations(dc, false); diff --git a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c index 65fa9e21ad9c..e72f15ac0048 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c +++ b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c @@ -1106,29 +1106,6 @@ void dcn20_blank_pixel_data( v_active, offset); - if (!blank && dc->debug.enable_single_display_2to1_odm_policy) { - /* when exiting dynamic ODM need to reinit DPG state for unused pipes */ - struct pipe_ctx *old_odm_pipe = dc->current_state->res_ctx.pipe_ctx[pipe_ctx->pipe_idx].next_odm_pipe; - - odm_pipe = pipe_ctx->next_odm_pipe; - - while (old_odm_pipe) { - if (!odm_pipe || old_odm_pipe->pipe_idx != odm_pipe->pipe_idx) - dc->hwss.set_disp_pattern_generator(dc, - old_odm_pipe, - CONTROLLER_DP_TEST_PATTERN_VIDEOMODE, - CONTROLLER_DP_COLOR_SPACE_UDEFINED, - COLOR_DEPTH_888, - NULL, - 0, - 0, - 0); - old_odm_pipe = old_odm_pipe->next_odm_pipe; - if (odm_pipe) - odm_pipe = odm_pipe->next_odm_pipe; - } - } - if (!blank) if (stream_res->abm) { dc->hwss.set_pipe(pipe_ctx); @@ -1584,17 +1561,6 @@ static void dcn20_update_dchubp_dpp( || plane_state->update_flags.bits.global_alpha_change || plane_state->update_flags.bits.per_pixel_alpha_change) { // MPCC inst is equal to pipe index in practice - int mpcc_inst = hubp->inst; - int opp_inst; - int opp_count = dc->res_pool->pipe_count; - - for (opp_inst = 0; opp_inst < opp_count; opp_inst++) { - if (dc->res_pool->opps[opp_inst]->mpcc_disconnect_pending[mpcc_inst]) { - dc->res_pool->mpc->funcs->wait_for_idle(dc->res_pool->mpc, mpcc_inst); - dc->res_pool->opps[opp_inst]->mpcc_disconnect_pending[mpcc_inst] = false; - break; - } - } hws->funcs.update_mpcc(dc, pipe_ctx); } @@ -1722,11 +1688,16 @@ static void dcn20_program_pipe( struct dc_state *context) { struct dce_hwseq *hws = dc->hwseq; - /* Only need to unblank on top pipe */ - if ((pipe_ctx->update_flags.bits.enable || pipe_ctx->stream->update_flags.bits.abm_level) - && !pipe_ctx->top_pipe && !pipe_ctx->prev_odm_pipe) - hws->funcs.blank_pixel_data(dc, pipe_ctx, !pipe_ctx->plane_state->visible); + /* Only need to unblank on top pipe */ + if (resource_is_pipe_type(pipe_ctx, OTG_MASTER)) { + if (pipe_ctx->update_flags.bits.enable || + pipe_ctx->update_flags.bits.odm || + pipe_ctx->stream->update_flags.bits.abm_level) + hws->funcs.blank_pixel_data(dc, pipe_ctx, + !pipe_ctx->plane_state || + !pipe_ctx->plane_state->visible); + } /* Only update TG on top pipe */ if (pipe_ctx->update_flags.bits.global_sync && !pipe_ctx->top_pipe diff --git a/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_hwseq.c index 6cef62d7a2e5..255713ec29bb 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_hwseq.c +++ b/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_hwseq.c @@ -987,3 +987,20 @@ void dcn30_prepare_bandwidth(struct dc *dc, } } +void dcn30_set_static_screen_control(struct pipe_ctx **pipe_ctx, + int num_pipes, const struct dc_static_screen_params *params) +{ + unsigned int i; + unsigned int triggers = 0; + + if (params->triggers.surface_update) + triggers |= 0x100; + if (params->triggers.cursor_update) + triggers |= 0x8; + if (params->triggers.force_trigger) + triggers |= 0x1; + + for (i = 0; i < num_pipes; i++) + pipe_ctx[i]->stream_res.tg->funcs->set_static_screen_control(pipe_ctx[i]->stream_res.tg, + triggers, params->num_frames); +} diff --git a/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_hwseq.h b/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_hwseq.h index a24a8e33a3d2..ce19c54097f8 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_hwseq.h +++ b/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_hwseq.h @@ -87,5 +87,7 @@ void dcn30_set_hubp_blank(const struct dc *dc, void dcn30_prepare_bandwidth(struct dc *dc, struct dc_state *context); +void dcn30_set_static_screen_control(struct pipe_ctx **pipe_ctx, + int num_pipes, const struct dc_static_screen_params *params); #endif /* __DC_HWSS_DCN30_H__ */ diff --git a/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_init.c b/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_init.c index 3d19acaa12f3..0de8b2783cf6 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_init.c +++ b/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_init.c @@ -64,7 +64,7 @@ static const struct hw_sequencer_funcs dcn30_funcs = { .update_bandwidth = dcn20_update_bandwidth, .set_drr = dcn10_set_drr, .get_position = dcn10_get_position, - .set_static_screen_control = dcn10_set_static_screen_control, + .set_static_screen_control = dcn30_set_static_screen_control, .setup_stereo = dcn10_setup_stereo, .set_avmute = dcn30_set_avmute, .log_hw_state = dcn10_log_hw_state, diff --git a/drivers/gpu/drm/amd/display/dc/dcn301/dcn301_init.c b/drivers/gpu/drm/amd/display/dc/dcn301/dcn301_init.c index 257df8660b4c..61205cdbe2d5 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn301/dcn301_init.c +++ b/drivers/gpu/drm/amd/display/dc/dcn301/dcn301_init.c @@ -75,6 +75,7 @@ static const struct hw_sequencer_funcs dcn301_funcs = { .get_hw_state = dcn10_get_hw_state, .clear_status_bits = dcn10_clear_status_bits, .wait_for_mpcc_disconnect = dcn10_wait_for_mpcc_disconnect, + .edp_backlight_control = dce110_edp_backlight_control, .edp_power_control = dce110_edp_power_control, .edp_wait_for_hpd_ready = dce110_edp_wait_for_hpd_ready, .set_cursor_position = dcn10_set_cursor_position, diff --git a/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_init.c b/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_init.c index fc25cc300a17..1d7bc1e39afe 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_init.c +++ b/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_init.c @@ -67,7 +67,7 @@ static const struct hw_sequencer_funcs dcn31_funcs = { .update_bandwidth = dcn20_update_bandwidth, .set_drr = dcn10_set_drr, .get_position = dcn10_get_position, - .set_static_screen_control = dcn10_set_static_screen_control, + .set_static_screen_control = dcn30_set_static_screen_control, .setup_stereo = dcn10_setup_stereo, .set_avmute = dcn30_set_avmute, .log_hw_state = dcn10_log_hw_state, diff --git a/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_init.c b/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_init.c index ca8fe55c33b8..4ef85c3a0688 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_init.c +++ b/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_init.c @@ -69,7 +69,7 @@ static const struct hw_sequencer_funcs dcn314_funcs = { .update_bandwidth = dcn20_update_bandwidth, .set_drr = dcn10_set_drr, .get_position = dcn10_get_position, - .set_static_screen_control = dcn10_set_static_screen_control, + .set_static_screen_control = dcn30_set_static_screen_control, .setup_stereo = dcn10_setup_stereo, .set_avmute = dcn30_set_avmute, .log_hw_state = dcn10_log_hw_state, diff --git a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_init.c b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_init.c index 777b2fac20c4..c7417147dff1 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_init.c +++ b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_init.c @@ -65,7 +65,7 @@ static const struct hw_sequencer_funcs dcn32_funcs = { .update_bandwidth = dcn20_update_bandwidth, .set_drr = dcn10_set_drr, .get_position = dcn10_get_position, - .set_static_screen_control = dcn10_set_static_screen_control, + .set_static_screen_control = dcn30_set_static_screen_control, .setup_stereo = dcn10_setup_stereo, .set_avmute = dcn30_set_avmute, .log_hw_state = dcn10_log_hw_state, diff --git a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c index 935cd23e6a01..f9d601c8c721 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c @@ -2564,18 +2564,128 @@ static int find_optimal_free_pipe_as_secondary_dpp_pipe( return free_pipe_idx; } +static struct pipe_ctx *find_idle_secondary_pipe_check_mpo( + struct resource_context *res_ctx, + const struct resource_pool *pool, + const struct pipe_ctx *primary_pipe) +{ + int i; + struct pipe_ctx *secondary_pipe = NULL; + struct pipe_ctx *next_odm_mpo_pipe = NULL; + int primary_index, preferred_pipe_idx; + struct pipe_ctx *old_primary_pipe = NULL; + + /* + * Modified from find_idle_secondary_pipe + * With windowed MPO and ODM, we want to avoid the case where we want a + * free pipe for the left side but the free pipe is being used on the + * right side. + * Add check on current_state if the primary_pipe is the left side, + * to check the right side ( primary_pipe->next_odm_pipe ) to see if + * it is using a pipe for MPO ( primary_pipe->next_odm_pipe->bottom_pipe ) + * - If so, then don't use this pipe + * EXCEPTION - 3 plane ( 2 MPO plane ) case + * - in this case, the primary pipe has already gotten a free pipe for the + * MPO window in the left + * - when it tries to get a free pipe for the MPO window on the right, + * it will see that it is already assigned to the right side + * ( primary_pipe->next_odm_pipe ). But in this case, we want this + * free pipe, since it will be for the right side. So add an + * additional condition, that skipping the free pipe on the right only + * applies if the primary pipe has no bottom pipe currently assigned + */ + if (primary_pipe) { + primary_index = primary_pipe->pipe_idx; + old_primary_pipe = &primary_pipe->stream->ctx->dc->current_state->res_ctx.pipe_ctx[primary_index]; + if ((old_primary_pipe->next_odm_pipe) && (old_primary_pipe->next_odm_pipe->bottom_pipe) + && (!primary_pipe->bottom_pipe)) + next_odm_mpo_pipe = old_primary_pipe->next_odm_pipe->bottom_pipe; + + preferred_pipe_idx = (pool->pipe_count - 1) - primary_pipe->pipe_idx; + if ((res_ctx->pipe_ctx[preferred_pipe_idx].stream == NULL) && + !(next_odm_mpo_pipe && next_odm_mpo_pipe->pipe_idx == preferred_pipe_idx)) { + secondary_pipe = &res_ctx->pipe_ctx[preferred_pipe_idx]; + secondary_pipe->pipe_idx = preferred_pipe_idx; + } + } + + /* + * search backwards for the second pipe to keep pipe + * assignment more consistent + */ + if (!secondary_pipe) + for (i = pool->pipe_count - 1; i >= 0; i--) { + if ((res_ctx->pipe_ctx[i].stream == NULL) && + !(next_odm_mpo_pipe && next_odm_mpo_pipe->pipe_idx == i)) { + secondary_pipe = &res_ctx->pipe_ctx[i]; + secondary_pipe->pipe_idx = i; + break; + } + } + + return secondary_pipe; +} + +static struct pipe_ctx *dcn32_acquire_idle_pipe_for_head_pipe_in_layer( + struct dc_state *state, + const struct resource_pool *pool, + struct dc_stream_state *stream, + const struct pipe_ctx *head_pipe) +{ + struct resource_context *res_ctx = &state->res_ctx; + struct pipe_ctx *idle_pipe, *pipe; + struct resource_context *old_ctx = &stream->ctx->dc->current_state->res_ctx; + int head_index; + + if (!head_pipe) + ASSERT(0); + + /* + * Modified from dcn20_acquire_idle_pipe_for_layer + * Check if head_pipe in old_context already has bottom_pipe allocated. + * - If so, check if that pipe is available in the current context. + * -- If so, reuse pipe from old_context + */ + head_index = head_pipe->pipe_idx; + pipe = &old_ctx->pipe_ctx[head_index]; + if (pipe->bottom_pipe && res_ctx->pipe_ctx[pipe->bottom_pipe->pipe_idx].stream == NULL) { + idle_pipe = &res_ctx->pipe_ctx[pipe->bottom_pipe->pipe_idx]; + idle_pipe->pipe_idx = pipe->bottom_pipe->pipe_idx; + } else { + idle_pipe = find_idle_secondary_pipe_check_mpo(res_ctx, pool, head_pipe); + if (!idle_pipe) + return NULL; + } + + idle_pipe->stream = head_pipe->stream; + idle_pipe->stream_res.tg = head_pipe->stream_res.tg; + idle_pipe->stream_res.opp = head_pipe->stream_res.opp; + + idle_pipe->plane_res.hubp = pool->hubps[idle_pipe->pipe_idx]; + idle_pipe->plane_res.ipp = pool->ipps[idle_pipe->pipe_idx]; + idle_pipe->plane_res.dpp = pool->dpps[idle_pipe->pipe_idx]; + idle_pipe->plane_res.mpcc_inst = pool->dpps[idle_pipe->pipe_idx]->inst; + + return idle_pipe; +} + struct pipe_ctx *dcn32_acquire_free_pipe_as_secondary_dpp_pipe( const struct dc_state *cur_ctx, struct dc_state *new_ctx, const struct resource_pool *pool, const struct pipe_ctx *opp_head_pipe) { - int free_pipe_idx = - find_optimal_free_pipe_as_secondary_dpp_pipe( - &cur_ctx->res_ctx, &new_ctx->res_ctx, - pool, opp_head_pipe); + + int free_pipe_idx; struct pipe_ctx *free_pipe; + if (!opp_head_pipe->stream->ctx->dc->config.enable_windowed_mpo_odm) + return dcn32_acquire_idle_pipe_for_head_pipe_in_layer( + new_ctx, pool, opp_head_pipe->stream, opp_head_pipe); + + free_pipe_idx = find_optimal_free_pipe_as_secondary_dpp_pipe( + &cur_ctx->res_ctx, &new_ctx->res_ctx, + pool, opp_head_pipe); if (free_pipe_idx >= 0) { free_pipe = &new_ctx->res_ctx.pipe_ctx[free_pipe_idx]; free_pipe->pipe_idx = free_pipe_idx; diff --git a/drivers/gpu/drm/amd/display/dc/dml/dcn20/dcn20_fpu.c b/drivers/gpu/drm/amd/display/dc/dml/dcn20/dcn20_fpu.c index 8afda5ecc0cd..5805fb02af14 100644 --- a/drivers/gpu/drm/amd/display/dc/dml/dcn20/dcn20_fpu.c +++ b/drivers/gpu/drm/amd/display/dc/dml/dcn20/dcn20_fpu.c @@ -1099,6 +1099,11 @@ void dcn20_calculate_dlg_params(struct dc *dc, context->res_ctx.pipe_ctx[i].plane_res.bw.dppclk_khz = pipes[pipe_idx].clks_cfg.dppclk_mhz * 1000; context->res_ctx.pipe_ctx[i].pipe_dlg_param = pipes[pipe_idx].pipe.dest; + if (dc->ctx->dce_version < DCN_VERSION_3_1 && + context->res_ctx.pipe_ctx[i].stream->adaptive_sync_infopacket.valid) + dcn20_adjust_freesync_v_startup( + &context->res_ctx.pipe_ctx[i].stream->timing, + &context->res_ctx.pipe_ctx[i].pipe_dlg_param.vstartup_start); pipe_idx++; } @@ -1927,7 +1932,6 @@ static bool dcn20_validate_bandwidth_internal(struct dc *dc, struct dc_state *co int vlevel = 0; int pipe_split_from[MAX_PIPES]; int pipe_cnt = 0; - int i = 0; display_e2e_pipe_params_st *pipes = kzalloc(dc->res_pool->pipe_count * sizeof(display_e2e_pipe_params_st), GFP_ATOMIC); DC_LOGGER_INIT(dc->ctx->logger); @@ -1951,15 +1955,6 @@ static bool dcn20_validate_bandwidth_internal(struct dc *dc, struct dc_state *co dcn20_calculate_wm(dc, context, pipes, &pipe_cnt, pipe_split_from, vlevel, fast_validate); dcn20_calculate_dlg_params(dc, context, pipes, pipe_cnt, vlevel); - for (i = 0; i < dc->res_pool->pipe_count; i++) { - if (!context->res_ctx.pipe_ctx[i].stream) - continue; - if (context->res_ctx.pipe_ctx[i].stream->adaptive_sync_infopacket.valid) - dcn20_adjust_freesync_v_startup( - &context->res_ctx.pipe_ctx[i].stream->timing, - &context->res_ctx.pipe_ctx[i].pipe_dlg_param.vstartup_start); - } - BW_VAL_TRACE_END_WATERMARKS(); goto validate_out; @@ -2232,7 +2227,6 @@ bool dcn21_validate_bandwidth_fp(struct dc *dc, int vlevel = 0; int pipe_split_from[MAX_PIPES]; int pipe_cnt = 0; - int i = 0; display_e2e_pipe_params_st *pipes = kzalloc(dc->res_pool->pipe_count * sizeof(display_e2e_pipe_params_st), GFP_ATOMIC); DC_LOGGER_INIT(dc->ctx->logger); @@ -2261,15 +2255,6 @@ bool dcn21_validate_bandwidth_fp(struct dc *dc, dcn21_calculate_wm(dc, context, pipes, &pipe_cnt, pipe_split_from, vlevel, fast_validate); dcn20_calculate_dlg_params(dc, context, pipes, pipe_cnt, vlevel); - for (i = 0; i < dc->res_pool->pipe_count; i++) { - if (!context->res_ctx.pipe_ctx[i].stream) - continue; - if (context->res_ctx.pipe_ctx[i].stream->adaptive_sync_infopacket.valid) - dcn20_adjust_freesync_v_startup( - &context->res_ctx.pipe_ctx[i].stream->timing, - &context->res_ctx.pipe_ctx[i].pipe_dlg_param.vstartup_start); - } - BW_VAL_TRACE_END_WATERMARKS(); goto validate_out; diff --git a/drivers/gpu/drm/amd/display/dc/dml/dcn314/dcn314_fpu.c b/drivers/gpu/drm/amd/display/dc/dml/dcn314/dcn314_fpu.c index 07adb614366e..fb21572750e8 100644 --- a/drivers/gpu/drm/amd/display/dc/dml/dcn314/dcn314_fpu.c +++ b/drivers/gpu/drm/amd/display/dc/dml/dcn314/dcn314_fpu.c @@ -293,6 +293,17 @@ static unsigned int micro_sec_to_vert_lines(unsigned int num_us, struct dc_crtc_ return num_lines; } +static unsigned int get_vertical_back_porch(struct dc_crtc_timing *timing) +{ + unsigned int v_active = 0, v_blank = 0, v_back_porch = 0; + + v_active = timing->v_border_top + timing->v_addressable + timing->v_border_bottom; + v_blank = timing->v_total - v_active; + v_back_porch = v_blank - timing->v_front_porch - timing->v_sync_width; + + return v_back_porch; +} + int dcn314_populate_dml_pipes_from_context_fpu(struct dc *dc, struct dc_state *context, display_e2e_pipe_params_st *pipes, bool fast_validate) @@ -310,6 +321,7 @@ int dcn314_populate_dml_pipes_from_context_fpu(struct dc *dc, struct dc_state *c for (i = 0, pipe_cnt = 0; i < dc->res_pool->pipe_count; i++) { struct dc_crtc_timing *timing; unsigned int num_lines = 0; + unsigned int v_back_porch = 0; if (!res_ctx->pipe_ctx[i].stream) continue; @@ -323,9 +335,16 @@ int dcn314_populate_dml_pipes_from_context_fpu(struct dc *dc, struct dc_state *c else pipes[pipe_cnt].pipe.dest.vtotal = timing->v_total; + v_back_porch = get_vertical_back_porch(timing); + pipes[pipe_cnt].pipe.dest.vblank_nom = timing->v_total - pipes[pipe_cnt].pipe.dest.vactive; pipes[pipe_cnt].pipe.dest.vblank_nom = min(pipes[pipe_cnt].pipe.dest.vblank_nom, num_lines); - pipes[pipe_cnt].pipe.dest.vblank_nom = max(pipes[pipe_cnt].pipe.dest.vblank_nom, timing->v_sync_width); + // vblank_nom should not smaller than (VSync (timing->v_sync_width + v_back_porch) + 2) + // + 2 is because + // 1 -> VStartup_start should be 1 line before VSync + // 1 -> always reserve 1 line between start of vblank to vstartup signal + pipes[pipe_cnt].pipe.dest.vblank_nom = + max(pipes[pipe_cnt].pipe.dest.vblank_nom, timing->v_sync_width + v_back_porch + 2); pipes[pipe_cnt].pipe.dest.vblank_nom = min(pipes[pipe_cnt].pipe.dest.vblank_nom, max_allowed_vblank_nom); if (pipe->plane_state && diff --git a/drivers/gpu/drm/amd/display/modules/freesync/freesync.c b/drivers/gpu/drm/amd/display/modules/freesync/freesync.c index dbd60811f95d..ef3a67409021 100644 --- a/drivers/gpu/drm/amd/display/modules/freesync/freesync.c +++ b/drivers/gpu/drm/amd/display/modules/freesync/freesync.c @@ -338,7 +338,9 @@ static void apply_below_the_range(struct core_freesync *core_freesync, * - Delta for CEIL: delta_from_mid_point_in_us_1 * - Delta for FLOOR: delta_from_mid_point_in_us_2 */ - if ((last_render_time_in_us / mid_point_frames_ceil) < in_out_vrr->min_duration_in_us) { + if (mid_point_frames_ceil && + (last_render_time_in_us / mid_point_frames_ceil) < + in_out_vrr->min_duration_in_us) { /* Check for out of range. * If using CEIL produces a value that is out of range, * then we are forced to use FLOOR. @@ -385,8 +387,9 @@ static void apply_below_the_range(struct core_freesync *core_freesync, /* Either we've calculated the number of frames to insert, * or we need to insert min duration frames */ - if (last_render_time_in_us / frames_to_insert < - in_out_vrr->min_duration_in_us){ + if (frames_to_insert && + (last_render_time_in_us / frames_to_insert) < + in_out_vrr->min_duration_in_us){ frames_to_insert -= (frames_to_insert > 1) ? 1 : 0; } diff --git a/drivers/gpu/drm/amd/include/amd_shared.h b/drivers/gpu/drm/amd/include/amd_shared.h index abe829bbd54a..67d7b7ee8a2a 100644 --- a/drivers/gpu/drm/amd/include/amd_shared.h +++ b/drivers/gpu/drm/amd/include/amd_shared.h @@ -240,6 +240,7 @@ enum DC_FEATURE_MASK { DC_DISABLE_LTTPR_DP2_0 = (1 << 6), //0x40, disabled by default DC_PSR_ALLOW_SMU_OPT = (1 << 7), //0x80, disabled by default DC_PSR_ALLOW_MULTI_DISP_OPT = (1 << 8), //0x100, disabled by default + DC_REPLAY_MASK = (1 << 9), //0x200, disabled by default for dcn < 3.1.4 }; enum DC_DEBUG_MASK { @@ -250,6 +251,7 @@ enum DC_DEBUG_MASK { DC_DISABLE_PSR = 0x10, DC_FORCE_SUBVP_MCLK_SWITCH = 0x20, DC_DISABLE_MPO = 0x40, + DC_DISABLE_REPLAY = 0x50, DC_ENABLE_DPIA_TRACE = 0x80, }; diff --git a/drivers/gpu/drm/amd/include/atomfirmware.h b/drivers/gpu/drm/amd/include/atomfirmware.h index e68c1e280322..fa7d6ced786f 100644 --- a/drivers/gpu/drm/amd/include/atomfirmware.h +++ b/drivers/gpu/drm/amd/include/atomfirmware.h @@ -3117,6 +3117,24 @@ enum atom_umc_config1_def { UMC_CONFIG1__ENABLE_ECC_CAPABLE = 0x00010000, }; +struct atom_umc_info_v4_0 { + struct atom_common_table_header table_header; + uint32_t ucode_reserved[5]; + uint8_t umcip_min_ver; + uint8_t umcip_max_ver; + uint8_t vram_type; + uint8_t umc_config; + uint32_t mem_refclk_10khz; + uint32_t clk_reserved[4]; + uint32_t golden_reserved; + uint32_t umc_config1; + uint32_t reserved[2]; + uint8_t channel_num; + uint8_t channel_width; + uint8_t channel_reserve[2]; + uint8_t umc_info_reserved[16]; +}; + /* *************************************************************************** Data Table vram_info structure diff --git a/drivers/gpu/drm/amd/include/discovery.h b/drivers/gpu/drm/amd/include/discovery.h index f43e29722ef7..7a9d473d0917 100644 --- a/drivers/gpu/drm/amd/include/discovery.h +++ b/drivers/gpu/drm/amd/include/discovery.h @@ -30,7 +30,7 @@ #define GC_TABLE_ID 0x4347 #define HARVEST_TABLE_SIGNATURE 0x56524148 #define VCN_INFO_TABLE_ID 0x004E4356 -#define MALL_INFO_TABLE_ID 0x4D414C4C +#define MALL_INFO_TABLE_ID 0x4C4C414D typedef enum { @@ -280,6 +280,36 @@ struct gc_info_v2_0 { uint32_t gc_num_packer_per_sc; }; +struct gc_info_v2_1 { + struct gpu_info_header header; + + uint32_t gc_num_se; + uint32_t gc_num_cu_per_sh; + uint32_t gc_num_sh_per_se; + uint32_t gc_num_rb_per_se; + uint32_t gc_num_tccs; + uint32_t gc_num_gprs; + uint32_t gc_num_max_gs_thds; + uint32_t gc_gs_table_depth; + uint32_t gc_gsprim_buff_depth; + uint32_t gc_parameter_cache_depth; + uint32_t gc_double_offchip_lds_buffer; + uint32_t gc_wave_size; + uint32_t gc_max_waves_per_simd; + uint32_t gc_max_scratch_slots_per_cu; + uint32_t gc_lds_size; + uint32_t gc_num_sc_per_se; + uint32_t gc_num_packer_per_sc; + /* new for v2_1 */ + uint32_t gc_num_tcp_per_sh; + uint32_t gc_tcp_size_per_cu; + uint32_t gc_num_sdp_interface; + uint32_t gc_num_cu_per_sqc; + uint32_t gc_instruction_cache_size_per_sqc; + uint32_t gc_scalar_data_cache_size_per_sqc; + uint32_t gc_tcc_size; +}; + typedef struct harvest_info_header { uint32_t signature; /* Table Signature */ uint32_t version; /* Table Version */ @@ -312,6 +342,12 @@ struct mall_info_v1_0 { uint32_t reserved[5]; }; +struct mall_info_v2_0 { + struct mall_info_header header; + uint32_t mall_size_per_umc; + uint32_t reserved[8]; +}; + #define VCN_INFO_TABLE_MAX_NUM_INSTANCES 4 struct vcn_info_header { diff --git a/drivers/gpu/drm/amd/pm/amdgpu_pm.c b/drivers/gpu/drm/amd/pm/amdgpu_pm.c index 5b1d73b00ef7..41147da54458 100644 --- a/drivers/gpu/drm/amd/pm/amdgpu_pm.c +++ b/drivers/gpu/drm/amd/pm/amdgpu_pm.c @@ -3311,8 +3311,10 @@ static umode_t hwmon_attributes_visible(struct kobject *kobj, (gc_ver != IP_VERSION(9, 4, 3)) && (attr == &sensor_dev_attr_temp2_input.dev_attr.attr || attr == &sensor_dev_attr_temp2_label.dev_attr.attr || + attr == &sensor_dev_attr_temp2_crit.dev_attr.attr || attr == &sensor_dev_attr_temp3_input.dev_attr.attr || - attr == &sensor_dev_attr_temp3_label.dev_attr.attr)) + attr == &sensor_dev_attr_temp3_label.dev_attr.attr || + attr == &sensor_dev_attr_temp3_crit.dev_attr.attr)) return 0; /* hotspot temperature for gc 9,4,3*/ @@ -3324,9 +3326,7 @@ static umode_t hwmon_attributes_visible(struct kobject *kobj, /* only SOC15 dGPUs support hotspot and mem temperatures */ if (((adev->flags & AMD_IS_APU) || gc_ver < IP_VERSION(9, 0, 0) || (gc_ver == IP_VERSION(9, 4, 3))) && - (attr == &sensor_dev_attr_temp2_crit.dev_attr.attr || - attr == &sensor_dev_attr_temp2_crit_hyst.dev_attr.attr || - attr == &sensor_dev_attr_temp3_crit.dev_attr.attr || + (attr == &sensor_dev_attr_temp2_crit_hyst.dev_attr.attr || attr == &sensor_dev_attr_temp3_crit_hyst.dev_attr.attr || attr == &sensor_dev_attr_temp1_emergency.dev_attr.attr || attr == &sensor_dev_attr_temp2_emergency.dev_attr.attr || @@ -3471,6 +3471,9 @@ static int amdgpu_debugfs_pm_info_pp(struct seq_file *m, struct amdgpu_device *a size = sizeof(uint32_t); if (!amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_AVG_POWER, (void *)&query, &size)) seq_printf(m, "\t%u.%u W (average GPU)\n", query >> 8, query & 0xff); + size = sizeof(uint32_t); + if (!amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_INPUT_POWER, (void *)&query, &size)) + seq_printf(m, "\t%u.%u W (current GPU)\n", query >> 8, query & 0xff); size = sizeof(value); seq_printf(m, "\n"); diff --git a/drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h b/drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h index 95eb8a5eb54f..5a52098bcf16 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h +++ b/drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h @@ -1031,10 +1031,7 @@ struct pptable_funcs { enum smu_feature_mask mask); /** - * @notify_display_change: Enable fast memory clock switching. - * - * Allows for fine grained memory clock switching but has more stringent - * timing requirements. + * @notify_display_change: General interface call to let SMU know about DC change */ int (*notify_display_change)(struct smu_context *smu); diff --git a/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_0_ppsmc.h b/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_0_ppsmc.h index 10cff75b44d5..e2ee855c7748 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_0_ppsmc.h +++ b/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_0_ppsmc.h @@ -138,7 +138,10 @@ #define PPSMC_MSG_SetBadMemoryPagesRetiredFlagsPerChannel 0x4A #define PPSMC_MSG_SetPriorityDeltaGain 0x4B #define PPSMC_MSG_AllowIHHostInterrupt 0x4C -#define PPSMC_Message_Count 0x4D + +#define PPSMC_MSG_DALNotPresent 0x4E + +#define PPSMC_Message_Count 0x4F //Debug Dump Message #define DEBUGSMC_MSG_TestMessage 0x1 diff --git a/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_pmfw.h b/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_pmfw.h index 252aef190c5c..9be4051c0865 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_pmfw.h +++ b/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_pmfw.h @@ -123,7 +123,7 @@ typedef enum { VOLTAGE_GUARDBAND_COUNT } GFX_GUARDBAND_e; -#define SMU_METRICS_TABLE_VERSION 0x5 +#define SMU_METRICS_TABLE_VERSION 0x7 typedef struct __attribute__((packed, aligned(4))) { uint32_t AccumulationCounter; @@ -198,7 +198,7 @@ typedef struct __attribute__((packed, aligned(4))) { uint32_t SocketThmResidencyAcc; uint32_t VrThmResidencyAcc; uint32_t HbmThmResidencyAcc; - uint32_t spare; + uint32_t GfxLockXCDMak; // New Items at end to maintain driver compatibility uint32_t GfxclkFrequency[8]; diff --git a/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_ppsmc.h b/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_ppsmc.h index ae4f44c4b877..70a4a717fd3f 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_ppsmc.h +++ b/drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_ppsmc.h @@ -83,13 +83,27 @@ #define PPSMC_MSG_GetMinGfxDpmFreq 0x32 #define PPSMC_MSG_GetMaxGfxDpmFreq 0x33 #define PPSMC_MSG_PrepareForDriverUnload 0x34 -#define PPSMC_Message_Count 0x35 +#define PPSMC_MSG_ReadThrottlerLimit 0x35 +#define PPSMC_MSG_QueryValidMcaCount 0x36 +#define PPSMC_MSG_McaBankDumpDW 0x37 +#define PPSMC_MSG_GetCTFLimit 0x38 +#define PPSMC_Message_Count 0x39 //PPSMC Reset Types for driver msg argument #define PPSMC_RESET_TYPE_DRIVER_MODE_1_RESET 0x1 #define PPSMC_RESET_TYPE_DRIVER_MODE_2_RESET 0x2 #define PPSMC_RESET_TYPE_DRIVER_MODE_3_RESET 0x3 +//PPSMC Reset Types for driver msg argument +#define PPSMC_THROTTLING_LIMIT_TYPE_SOCKET 0x1 +#define PPSMC_THROTTLING_LIMIT_TYPE_HBM 0x2 + +//CTF/Throttle Limit types +#define PPSMC_AID_THM_TYPE 0x1 +#define PPSMC_CCD_THM_TYPE 0x2 +#define PPSMC_XCD_THM_TYPE 0x3 +#define PPSMC_HBM_THM_TYPE 0x4 + typedef uint32_t PPSMC_Result; typedef uint32_t PPSMC_MSG; diff --git a/drivers/gpu/drm/amd/pm/swsmu/inc/smu_types.h b/drivers/gpu/drm/amd/pm/swsmu/inc/smu_types.h index 297b70b9388f..e57265cf637c 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/inc/smu_types.h +++ b/drivers/gpu/drm/amd/pm/swsmu/inc/smu_types.h @@ -84,6 +84,7 @@ __SMU_DUMMY_MAP(SetTjMax), \ __SMU_DUMMY_MAP(SetFanTemperatureTarget), \ __SMU_DUMMY_MAP(PrepareMp1ForUnload), \ + __SMU_DUMMY_MAP(GetCTFLimit), \ __SMU_DUMMY_MAP(DramLogSetDramAddrHigh), \ __SMU_DUMMY_MAP(DramLogSetDramAddrLow), \ __SMU_DUMMY_MAP(DramLogSetDramSize), \ @@ -245,7 +246,8 @@ __SMU_DUMMY_MAP(AllowGpo), \ __SMU_DUMMY_MAP(Mode2Reset), \ __SMU_DUMMY_MAP(RequestI2cTransaction), \ - __SMU_DUMMY_MAP(GetMetricsTable), + __SMU_DUMMY_MAP(GetMetricsTable), \ + __SMU_DUMMY_MAP(DALNotPresent), #undef __SMU_DUMMY_MAP #define __SMU_DUMMY_MAP(type) SMU_MSG_##type diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c index f1282fc4b90a..0232adb95df3 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c +++ b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c @@ -837,12 +837,8 @@ int smu_v13_0_notify_display_change(struct smu_context *smu) { int ret = 0; - if (!smu->pm_enabled) - return ret; - - if (smu_cmn_feature_is_enabled(smu, SMU_FEATURE_DPM_UCLK_BIT) && - smu->adev->gmc.vram_type == AMDGPU_VRAM_TYPE_HBM) - ret = smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_SetUclkFastSwitch, 1, NULL); + if (!amdgpu_device_has_dc_support(smu->adev)) + ret = smu_cmn_send_smc_msg(smu, SMU_MSG_DALNotPresent, NULL); return ret; } diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c index 8b7403ba89d7..3903a47669e4 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c +++ b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c @@ -162,6 +162,7 @@ static struct cmn2asic_msg_mapping smu_v13_0_0_message_map[SMU_MSG_MAX_COUNT] = MSG_MAP(AllowGpo, PPSMC_MSG_SetGpoAllow, 0), MSG_MAP(AllowIHHostInterrupt, PPSMC_MSG_AllowIHHostInterrupt, 0), MSG_MAP(ReenableAcDcInterrupt, PPSMC_MSG_ReenableAcDcInterrupt, 0), + MSG_MAP(DALNotPresent, PPSMC_MSG_DALNotPresent, 0), }; static struct cmn2asic_mapping smu_v13_0_0_clk_map[SMU_CLK_COUNT] = { @@ -2687,6 +2688,7 @@ static const struct pptable_funcs smu_v13_0_0_ppt_funcs = { .send_hbm_bad_channel_flag = smu_v13_0_0_send_bad_mem_channel_flag, .gpo_control = smu_v13_0_gpo_control, .get_ecc_info = smu_v13_0_0_get_ecc_info, + .notify_display_change = smu_v13_0_notify_display_change, }; void smu_v13_0_0_set_ppt_funcs(struct smu_context *smu) diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c index 6ed9cd0a1e4e..199a673b8120 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c +++ b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c @@ -132,6 +132,7 @@ static const struct cmn2asic_msg_mapping smu_v13_0_6_message_map[SMU_MSG_MAX_COU MSG_MAP(SetSoftMinGfxclk, PPSMC_MSG_SetSoftMinGfxClk, 0), MSG_MAP(SetSoftMaxGfxClk, PPSMC_MSG_SetSoftMaxGfxClk, 0), MSG_MAP(PrepareMp1ForUnload, PPSMC_MSG_PrepareForDriverUnload, 0), + MSG_MAP(GetCTFLimit, PPSMC_MSG_GetCTFLimit, 0), }; static const struct cmn2asic_mapping smu_v13_0_6_clk_map[SMU_CLK_COUNT] = { @@ -2081,6 +2082,55 @@ out: return ret; } +static int smu_v13_0_6_get_thermal_temperature_range(struct smu_context *smu, + struct smu_temperature_range *range) +{ + struct amdgpu_device *adev = smu->adev; + u32 aid_temp, xcd_temp, mem_temp; + uint32_t smu_version; + u32 ccd_temp = 0; + int ret; + + if (amdgpu_sriov_vf(smu->adev)) + return 0; + + if (!range) + return -EINVAL; + + /*Check smu version, GetCtfLimit message only supported for smu version 85.69 or higher */ + smu_cmn_get_smc_version(smu, NULL, &smu_version); + if (smu_version < 0x554500) + return 0; + + ret = smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_GetCTFLimit, + PPSMC_AID_THM_TYPE, &aid_temp); + if (ret) + goto failed; + + if (adev->flags & AMD_IS_APU) { + ret = smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_GetCTFLimit, + PPSMC_CCD_THM_TYPE, &ccd_temp); + if (ret) + goto failed; + } + + ret = smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_GetCTFLimit, + PPSMC_XCD_THM_TYPE, &xcd_temp); + if (ret) + goto failed; + + range->hotspot_crit_max = max3(aid_temp, xcd_temp, ccd_temp) * + SMU_TEMPERATURE_UNITS_PER_CENTIGRADES; + ret = smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_GetCTFLimit, + PPSMC_HBM_THM_TYPE, &mem_temp); + if (ret) + goto failed; + + range->mem_crit_max = mem_temp * SMU_TEMPERATURE_UNITS_PER_CENTIGRADES; +failed: + return ret; +} + static int smu_v13_0_6_mode1_reset(struct smu_context *smu) { struct amdgpu_device *adev = smu->adev; @@ -2108,8 +2158,7 @@ static int smu_v13_0_6_mode1_reset(struct smu_context *smu) static bool smu_v13_0_6_is_mode1_reset_supported(struct smu_context *smu) { - /* TODO: Enable this when FW support is added */ - return false; + return true; } static bool smu_v13_0_6_is_mode2_reset_supported(struct smu_context *smu) @@ -2177,6 +2226,7 @@ static const struct pptable_funcs smu_v13_0_6_ppt_funcs = { .get_pp_feature_mask = smu_cmn_get_pp_feature_mask, .set_pp_feature_mask = smu_cmn_set_pp_feature_mask, .get_gpu_metrics = smu_v13_0_6_get_gpu_metrics, + .get_thermal_temperature_range = smu_v13_0_6_get_thermal_temperature_range, .mode1_reset_is_support = smu_v13_0_6_is_mode1_reset_supported, .mode2_reset_is_support = smu_v13_0_6_is_mode2_reset_supported, .mode1_reset = smu_v13_0_6_mode1_reset, diff --git a/drivers/gpu/drm/ci/arm.config b/drivers/gpu/drm/ci/arm.config new file mode 100644 index 000000000000..871f4de063ad --- /dev/null +++ b/drivers/gpu/drm/ci/arm.config @@ -0,0 +1,69 @@ +CONFIG_LOCALVERSION_AUTO=y +CONFIG_DEBUG_KERNEL=y + +CONFIG_CRYPTO_ZSTD=y +CONFIG_ZRAM_MEMORY_TRACKING=y +CONFIG_ZRAM_WRITEBACK=y +CONFIG_ZRAM=y +CONFIG_ZSMALLOC_STAT=y + +# abootimg with a 'dummy' rootfs fails with root=/dev/nfs +CONFIG_BLK_DEV_INITRD=n + +CONFIG_DEVFREQ_GOV_PERFORMANCE=y +CONFIG_DEVFREQ_GOV_POWERSAVE=y +CONFIG_DEVFREQ_GOV_USERSPACE=y +CONFIG_DEVFREQ_GOV_PASSIVE=y +CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y + +CONFIG_DRM=y +CONFIG_DRM_ETNAVIV=y +CONFIG_DRM_ROCKCHIP=y +CONFIG_DRM_PANFROST=y +CONFIG_DRM_LIMA=y +CONFIG_DRM_PANEL_SIMPLE=y +CONFIG_PWM_CROS_EC=y +CONFIG_BACKLIGHT_PWM=y + +CONFIG_ROCKCHIP_CDN_DP=n + +CONFIG_SPI_ROCKCHIP=y +CONFIG_PWM_ROCKCHIP=y +CONFIG_PHY_ROCKCHIP_DP=y +CONFIG_DWMAC_ROCKCHIP=y + +CONFIG_MFD_RK808=y +CONFIG_REGULATOR_RK808=y +CONFIG_RTC_DRV_RK808=y +CONFIG_COMMON_CLK_RK808=y + +CONFIG_REGULATOR_FAN53555=y +CONFIG_REGULATOR=y + +CONFIG_REGULATOR_VCTRL=y + +CONFIG_KASAN=n +CONFIG_KASAN_INLINE=n +CONFIG_STACKTRACE=n + +CONFIG_TMPFS=y + +CONFIG_PROVE_LOCKING=n +CONFIG_DEBUG_LOCKDEP=n +CONFIG_SOFTLOCKUP_DETECTOR=n +CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=n + +CONFIG_FW_LOADER_COMPRESS=y + +CONFIG_USB_USBNET=y +CONFIG_NETDEVICES=y +CONFIG_USB_NET_DRIVERS=y +CONFIG_USB_RTL8152=y +CONFIG_USB_NET_AX8817X=y +CONFIG_USB_NET_SMSC95XX=y + +# TK1 +CONFIG_ARM_TEGRA_DEVFREQ=y + +# 32-bit build failure +CONFIG_DRM_MSM=n diff --git a/drivers/gpu/drm/ci/arm64.config b/drivers/gpu/drm/ci/arm64.config new file mode 100644 index 000000000000..817e18ddfd4f --- /dev/null +++ b/drivers/gpu/drm/ci/arm64.config @@ -0,0 +1,199 @@ +CONFIG_LOCALVERSION_AUTO=y +CONFIG_DEBUG_KERNEL=y + +CONFIG_CRYPTO_ZSTD=y +CONFIG_ZRAM_MEMORY_TRACKING=y +CONFIG_ZRAM_WRITEBACK=y +CONFIG_ZRAM=y +CONFIG_ZSMALLOC_STAT=y + +# abootimg with a 'dummy' rootfs fails with root=/dev/nfs +CONFIG_BLK_DEV_INITRD=n + +CONFIG_DEVFREQ_GOV_PERFORMANCE=y +CONFIG_DEVFREQ_GOV_POWERSAVE=y +CONFIG_DEVFREQ_GOV_USERSPACE=y +CONFIG_DEVFREQ_GOV_PASSIVE=y + +CONFIG_DRM=y +CONFIG_DRM_ROCKCHIP=y +CONFIG_DRM_PANFROST=y +CONFIG_DRM_LIMA=y +CONFIG_DRM_PANEL_SIMPLE=y +CONFIG_DRM_PANEL_EDP=y +CONFIG_DRM_MSM=y +CONFIG_DRM_ETNAVIV=y +CONFIG_DRM_I2C_ADV7511=y +CONFIG_PWM_CROS_EC=y +CONFIG_BACKLIGHT_PWM=y + +CONFIG_ROCKCHIP_CDN_DP=n + +CONFIG_SPI_ROCKCHIP=y +CONFIG_PWM_ROCKCHIP=y +CONFIG_PHY_ROCKCHIP_DP=y +CONFIG_DWMAC_ROCKCHIP=y +CONFIG_STMMAC_ETH=y +CONFIG_TYPEC_FUSB302=y +CONFIG_TYPEC=y +CONFIG_TYPEC_TCPM=y + +# MSM platform bits + +# For CONFIG_QCOM_LMH +CONFIG_OF=y + +CONFIG_ARM_SMMU_QCOM=y +CONFIG_QCOM_COMMAND_DB=y +CONFIG_QCOM_RPMHPD=y +CONFIG_QCOM_RPMPD=y +CONFIG_QCOM_OCMEM=y +CONFIG_SDM_GPUCC_845=y +CONFIG_SDM_VIDEOCC_845=y +CONFIG_SDM_DISPCC_845=y +CONFIG_SDM_LPASSCC_845=y +CONFIG_SDM_CAMCC_845=y +CONFIG_RESET_QCOM_PDC=y +CONFIG_DRM_TI_SN65DSI86=y +CONFIG_I2C_QCOM_GENI=y +CONFIG_SPI_QCOM_GENI=y +CONFIG_PHY_QCOM_QUSB2=y +CONFIG_PHY_QCOM_QMP=y +CONFIG_MSM_GCC_8996=y +CONFIG_QCOM_CLK_APCC_MSM8996=y +CONFIG_QCOM_LLCC=y +CONFIG_QCOM_LMH=y +CONFIG_QCOM_SPMI_TEMP_ALARM=y +CONFIG_QCOM_WDT=y +CONFIG_POWER_RESET_QCOM_PON=y +CONFIG_RTC_DRV_PM8XXX=y +CONFIG_INTERCONNECT=y +CONFIG_INTERCONNECT_QCOM=y +CONFIG_INTERCONNECT_QCOM_MSM8996=y +CONFIG_INTERCONNECT_QCOM_SDM845=y +CONFIG_INTERCONNECT_QCOM_MSM8916=y +CONFIG_INTERCONNECT_QCOM_MSM8996=y +CONFIG_INTERCONNECT_QCOM_OSM_L3=y +CONFIG_INTERCONNECT_QCOM_SC7180=y +CONFIG_INTERCONNECT_QCOM_SM8350=y +CONFIG_CRYPTO_DEV_QCOM_RNG=y +CONFIG_SC_DISPCC_7180=y +CONFIG_SC_GPUCC_7180=y +CONFIG_SM_GPUCC_8350=y +CONFIG_QCOM_SPMI_ADC5=y +CONFIG_DRM_PARADE_PS8640=y +CONFIG_DRM_LONTIUM_LT9611UXC=y +CONFIG_PHY_QCOM_USB_HS=y +CONFIG_QCOM_GPI_DMA=y +CONFIG_USB_ONBOARD_HUB=y +CONFIG_NVMEM_QCOM_QFPROM=y +CONFIG_PHY_QCOM_USB_SNPS_FEMTO_V2=y + + +# db410c ethernet +CONFIG_USB_RTL8152=y +# db820c ethernet +CONFIG_ATL1C=y +# Chromebooks ethernet +CONFIG_USB_ONBOARD_HUB=y +# 888 HDK ethernet +CONFIG_USB_LAN78XX=y + +CONFIG_ARCH_ALPINE=n +CONFIG_ARCH_BCM2835=y +CONFIG_ARCH_BCM_IPROC=n +CONFIG_ARCH_BERLIN=n +CONFIG_ARCH_BRCMSTB=n +CONFIG_ARCH_EXYNOS=n +CONFIG_ARCH_K3=n +CONFIG_ARCH_LAYERSCAPE=n +CONFIG_ARCH_LG1K=n +CONFIG_ARCH_HISI=n +CONFIG_ARCH_MVEBU=n +CONFIG_ARCH_SEATTLE=n +CONFIG_ARCH_SYNQUACER=n +CONFIG_ARCH_RENESAS=n +CONFIG_ARCH_R8A774A1=n +CONFIG_ARCH_R8A774C0=n +CONFIG_ARCH_R8A7795=n +CONFIG_ARCH_R8A7796=n +CONFIG_ARCH_R8A77965=n +CONFIG_ARCH_R8A77970=n +CONFIG_ARCH_R8A77980=n +CONFIG_ARCH_R8A77990=n +CONFIG_ARCH_R8A77995=n +CONFIG_ARCH_STRATIX10=n +CONFIG_ARCH_TEGRA=n +CONFIG_ARCH_SPRD=n +CONFIG_ARCH_THUNDER=n +CONFIG_ARCH_THUNDER2=n +CONFIG_ARCH_UNIPHIER=n +CONFIG_ARCH_VEXPRESS=n +CONFIG_ARCH_XGENE=n +CONFIG_ARCH_ZX=n +CONFIG_ARCH_ZYNQMP=n + +# Strip out some stuff we don't need for graphics testing, to reduce +# the build. +CONFIG_CAN=n +CONFIG_WIRELESS=n +CONFIG_RFKILL=n +CONFIG_WLAN=n + +CONFIG_REGULATOR_FAN53555=y +CONFIG_REGULATOR=y + +CONFIG_REGULATOR_VCTRL=y + +CONFIG_KASAN=n +CONFIG_KASAN_INLINE=n +CONFIG_STACKTRACE=n + +CONFIG_TMPFS=y + +CONFIG_PROVE_LOCKING=n +CONFIG_DEBUG_LOCKDEP=n +CONFIG_SOFTLOCKUP_DETECTOR=y +CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y + +CONFIG_DETECT_HUNG_TASK=y + +CONFIG_FW_LOADER_COMPRESS=y +CONFIG_FW_LOADER_USER_HELPER=n + +CONFIG_USB_USBNET=y +CONFIG_NETDEVICES=y +CONFIG_USB_NET_DRIVERS=y +CONFIG_USB_RTL8152=y +CONFIG_USB_NET_AX8817X=y +CONFIG_USB_NET_SMSC95XX=y + +# For amlogic +CONFIG_MESON_GXL_PHY=y +CONFIG_MDIO_BUS_MUX_MESON_G12A=y +CONFIG_DRM_MESON=y + +# For Mediatek +CONFIG_DRM_MEDIATEK=y +CONFIG_PWM_MEDIATEK=y +CONFIG_DRM_MEDIATEK_HDMI=y +CONFIG_GNSS=y +CONFIG_GNSS_MTK_SERIAL=y +CONFIG_HW_RANDOM=y +CONFIG_HW_RANDOM_MTK=y +CONFIG_MTK_DEVAPC=y +CONFIG_PWM_MTK_DISP=y +CONFIG_MTK_CMDQ=y + +# For nouveau. Note that DRM must be a module so that it's loaded after NFS is up to provide the firmware. +CONFIG_ARCH_TEGRA=y +CONFIG_DRM_NOUVEAU=m +CONFIG_DRM_TEGRA=m +CONFIG_R8169=y +CONFIG_STAGING=y +CONFIG_DRM_TEGRA_STAGING=y +CONFIG_TEGRA_HOST1X=y +CONFIG_ARM_TEGRA_DEVFREQ=y +CONFIG_TEGRA_SOCTHERM=y +CONFIG_DRM_TEGRA_DEBUG=y +CONFIG_PWM_TEGRA=y diff --git a/drivers/gpu/drm/ci/build-igt.sh b/drivers/gpu/drm/ci/build-igt.sh new file mode 100644 index 000000000000..500fa4f5c30a --- /dev/null +++ b/drivers/gpu/drm/ci/build-igt.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# SPDX-License-Identifier: MIT + +set -ex + +git clone https://gitlab.freedesktop.org/drm/igt-gpu-tools.git --single-branch --no-checkout +cd igt-gpu-tools +git checkout $IGT_VERSION + +if [[ "$KERNEL_ARCH" = "arm" ]]; then + . ../.gitlab-ci/container/create-cross-file.sh armhf + EXTRA_MESON_ARGS="--cross-file /cross_file-armhf.txt" +fi + +MESON_OPTIONS="-Doverlay=disabled \ + -Dchamelium=disabled \ + -Dvalgrind=disabled \ + -Dman=enabled \ + -Dtests=enabled \ + -Drunner=enabled \ + -Dlibunwind=enabled \ + -Dprefix=/igt" + +mkdir -p /igt +meson build $MESON_OPTIONS $EXTRA_MESON_ARGS +ninja -C build -j${FDO_CI_CONCURRENT:-4} || ninja -C build -j 1 +ninja -C build install + +mkdir -p artifacts/ +tar -cf artifacts/igt.tar /igt + +# Pass needed files to the test stage +S3_ARTIFACT_NAME="igt.tar.gz" +gzip -c artifacts/igt.tar > ${S3_ARTIFACT_NAME} +ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${KERNEL_ARCH}/${S3_ARTIFACT_NAME} diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh new file mode 100644 index 000000000000..7b014287a041 --- /dev/null +++ b/drivers/gpu/drm/ci/build.sh @@ -0,0 +1,157 @@ +#!/bin/bash +# SPDX-License-Identifier: MIT + +set -ex + +# Clean up stale rebases that GitLab might not have removed when reusing a checkout dir +rm -rf .git/rebase-apply + +. .gitlab-ci/container/container_pre_build.sh + +# libssl-dev was uninstalled because it was considered an ephemeral package +apt-get update +apt-get install -y libssl-dev + +if [[ "$KERNEL_ARCH" = "arm64" ]]; then + GCC_ARCH="aarch64-linux-gnu" + DEBIAN_ARCH="arm64" + DEVICE_TREES="arch/arm64/boot/dts/rockchip/rk3399-gru-kevin.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-gxl-s805x-libretech-ac.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/allwinner/sun50i-h6-pine-h64.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-gxm-khadas-vim2.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/qcom/apq8016-sbc.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/qcom/apq8096-db820c.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-g12b-a311d-khadas-vim3.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8173-elm-hana.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8183-kukui-jacuzzi-juniper-sku16.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8192-asurada-spherion-r0.dtb" + DEVICE_TREES+=" arch/arm64/boot/dts/qcom/sc7180-trogdor-lazor-limozeen-nots-r5.dtb" +elif [[ "$KERNEL_ARCH" = "arm" ]]; then + GCC_ARCH="arm-linux-gnueabihf" + DEBIAN_ARCH="armhf" + DEVICE_TREES="arch/arm/boot/dts/rockchip/rk3288-veyron-jaq.dtb" + DEVICE_TREES+=" arch/arm/boot/dts/allwinner/sun8i-h3-libretech-all-h3-cc.dtb" + DEVICE_TREES+=" arch/arm/boot/dts/nxp/imx/imx6q-cubox-i.dtb" + apt-get install -y libssl-dev:armhf +else + GCC_ARCH="x86_64-linux-gnu" + DEBIAN_ARCH="x86_64" + DEVICE_TREES="" +fi + +export ARCH=${KERNEL_ARCH} +export CROSS_COMPILE="${GCC_ARCH}-" + +# The kernel doesn't like the gold linker (or the old lld in our debians). +# Sneak in some override symlinks during kernel build until we can update +# debian. +mkdir -p ld-links +for i in /usr/bin/*-ld /usr/bin/ld; do + i=$(basename $i) + ln -sf /usr/bin/$i.bfd ld-links/$i +done + +NEWPATH=$(pwd)/ld-links +export PATH=$NEWPATH:$PATH + +git config --global user.email "fdo@example.com" +git config --global user.name "freedesktop.org CI" +git config --global pull.rebase true + +# Try to merge fixes from target repo +if [ "$(git ls-remote --exit-code --heads ${UPSTREAM_REPO} ${TARGET_BRANCH}-external-fixes)" ]; then + git pull ${UPSTREAM_REPO} ${TARGET_BRANCH}-external-fixes +fi + +# Try to merge fixes from local repo if this isn't a merge request +if [ -z "$CI_MERGE_REQUEST_PROJECT_PATH" ]; then + if [ "$(git ls-remote --exit-code --heads origin ${TARGET_BRANCH}-external-fixes)" ]; then + git pull origin ${TARGET_BRANCH}-external-fixes + fi +fi + +for opt in $ENABLE_KCONFIGS; do + echo CONFIG_$opt=y >> drivers/gpu/drm/ci/${KERNEL_ARCH}.config +done +for opt in $DISABLE_KCONFIGS; do + echo CONFIG_$opt=n >> drivers/gpu/drm/ci/${KERNEL_ARCH}.config +done + +if [[ -n "${MERGE_FRAGMENT}" ]]; then + ./scripts/kconfig/merge_config.sh ${DEFCONFIG} drivers/gpu/drm/ci/${MERGE_FRAGMENT} +else + make `basename ${DEFCONFIG}` +fi + +make ${KERNEL_IMAGE_NAME} + +mkdir -p /lava-files/ +for image in ${KERNEL_IMAGE_NAME}; do + cp arch/${KERNEL_ARCH}/boot/${image} /lava-files/. +done + +if [[ -n ${DEVICE_TREES} ]]; then + make dtbs + cp ${DEVICE_TREES} /lava-files/. +fi + +make modules +mkdir -p install/modules/ +INSTALL_MOD_PATH=install/modules/ make modules_install + +if [[ ${DEBIAN_ARCH} = "arm64" ]]; then + make Image.lzma + mkimage \ + -f auto \ + -A arm \ + -O linux \ + -d arch/arm64/boot/Image.lzma \ + -C lzma\ + -b arch/arm64/boot/dts/qcom/sdm845-cheza-r3.dtb \ + /lava-files/cheza-kernel + KERNEL_IMAGE_NAME+=" cheza-kernel" + + # Make a gzipped copy of the Image for db410c. + gzip -k /lava-files/Image + KERNEL_IMAGE_NAME+=" Image.gz" +fi + +# Pass needed files to the test stage +mkdir -p install +cp -rfv .gitlab-ci/* install/. +cp -rfv install/common install/ci-common +cp -rfv drivers/gpu/drm/ci/* install/. + +. .gitlab-ci/container/container_post_build.sh + +if [[ "$UPLOAD_TO_MINIO" = "1" ]]; then + xz -7 -c -T${FDO_CI_CONCURRENT:-4} vmlinux > /lava-files/vmlinux.xz + FILES_TO_UPLOAD="$KERNEL_IMAGE_NAME vmlinux.xz" + + if [[ -n $DEVICE_TREES ]]; then + FILES_TO_UPLOAD="$FILES_TO_UPLOAD $(basename -a $DEVICE_TREES)" + fi + + for f in $FILES_TO_UPLOAD; do + ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" /lava-files/$f \ + https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/$f + done + + S3_ARTIFACT_NAME="kernel-files.tar.zst" + tar --zstd -cf $S3_ARTIFACT_NAME install + ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/${S3_ARTIFACT_NAME} + + echo "Download vmlinux.xz from https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/vmlinux.xz" +fi + +mkdir -p artifacts/install/lib +mv install/* artifacts/install/. +rm -rf artifacts/install/modules +ln -s common artifacts/install/ci-common + +for image in ${KERNEL_IMAGE_NAME}; do + cp /lava-files/$image artifacts/install/. +done + +tar -C artifacts -cf artifacts/install.tar install +rm -rf artifacts/install diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml new file mode 100644 index 000000000000..e6503f1c5927 --- /dev/null +++ b/drivers/gpu/drm/ci/build.yml @@ -0,0 +1,110 @@ +.build: + extends: + - .build-rules + stage: build + artifacts: + paths: + - artifacts + script: + - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build.sh + +.build:arm32: + extends: + - .build + - .use-debian/arm64_build + tags: + - aarch64 + variables: + DEFCONFIG: "arch/arm/configs/multi_v7_defconfig" + KERNEL_IMAGE_NAME: "zImage" + KERNEL_ARCH: "arm" + +.build:arm64: + extends: + - .build + - .use-debian/arm64_build + tags: + - aarch64 + variables: + DEFCONFIG: "arch/arm64/configs/defconfig" + KERNEL_IMAGE_NAME: "Image" + KERNEL_ARCH: "arm64" + +.build:x86_64: + extends: + - .build + - .use-debian/x86_64_build + variables: + DEFCONFIG: "arch/x86/configs/x86_64_defconfig" + KERNEL_IMAGE_NAME: "bzImage" + KERNEL_ARCH: "x86_64" + + +# Build IGT for testing on devices + +igt:arm32: + extends: .build:arm32 + script: + - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh + +igt:arm64: + extends: .build:arm64 + script: + - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh + +igt:x86_64: + extends: .build:x86_64 + script: + - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh + +# Build kernels for testing on devices + +testing:arm32: + extends: .build:arm32 + variables: + # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of + # PROVE_LOCKING and KASAN as of 5.17. + # + # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel + # becoming too big for their bootloaders. + ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT" + UPLOAD_TO_MINIO: 1 + MERGE_FRAGMENT: arm.config + +testing:arm64: + extends: .build:arm64 + variables: + # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of + # PROVE_LOCKING and KASAN as of 5.17. + # + # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel + # becoming too big for their bootloaders. + ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT" + UPLOAD_TO_MINIO: 1 + MERGE_FRAGMENT: arm64.config + +testing:x86_64: + extends: .build:x86_64 + variables: + # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of + # PROVE_LOCKING and KASAN as of 5.17. + # + # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel + # becoming too big for their bootloaders. + ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT" + UPLOAD_TO_MINIO: 1 + MERGE_FRAGMENT: x86_64.config + + +# Jobs for build-testing different configurations + +build:arm32: + extends: .build:arm32 + +build-nodebugfs:arm64: + extends: .build:arm64 + variables: + DISABLE_KCONFIGS: "DEBUG_FS" + +build:x86_64: + extends: .build:x86_64 diff --git a/drivers/gpu/drm/ci/check-patch.py b/drivers/gpu/drm/ci/check-patch.py new file mode 100755 index 000000000000..a5f399a20e25 --- /dev/null +++ b/drivers/gpu/drm/ci/check-patch.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# +# check-patch.py: run checkpatch.pl across all commits in a branch +# +# Based on qemu/.gitlab-ci.d/check-patch.py +# +# Copyright (C) 2020 Red Hat, Inc. +# Copyright (C) 2022 Collabora Ltd. + +import os +import os.path +import sys +import subprocess + +repourl = "https://gitlab.freedesktop.org/%s.git" % os.environ["CI_MERGE_REQUEST_PROJECT_PATH"] + +# GitLab CI environment does not give us any direct info about the +# base for the user's branch. We thus need to figure out a common +# ancestor between the user's branch and current git master. +os.environ["GIT_DEPTH"] = "1000" +subprocess.call(["git", "remote", "remove", "check-patch"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) +subprocess.check_call(["git", "remote", "add", "check-patch", repourl]) +subprocess.check_call(["git", "fetch", "check-patch", os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"]], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL) + +ancestor = subprocess.check_output(["git", "merge-base", + "check-patch/%s" % os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"], "HEAD"], + universal_newlines=True) + +ancestor = ancestor.strip() + +log = subprocess.check_output(["git", "log", "--format=%H %s", + ancestor + "..."], + universal_newlines=True) + +subprocess.check_call(["git", "remote", "rm", "check-patch"]) + +if log == "": + print("\nNo commits since %s, skipping checks\n" % ancestor) + sys.exit(0) + +errors = False + +print("\nChecking all commits since %s...\n" % ancestor, flush=True) + +ret = subprocess.run(["scripts/checkpatch.pl", + "--terse", + "--types", os.environ["CHECKPATCH_TYPES"], + "--git", ancestor + "..."]) + +if ret.returncode != 0: + print(" ❌ FAIL one or more commits failed scripts/checkpatch.pl") + sys.exit(1) + +sys.exit(0) diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml new file mode 100644 index 000000000000..9764e7921a4f --- /dev/null +++ b/drivers/gpu/drm/ci/container.yml @@ -0,0 +1,65 @@ +.container: + variables: + CI_REPOSITORY_URL: ${DRM_CI_PROJECT_URL}.git # So ci-templates clones drm-ci instead of the repo to test + CI_COMMIT_SHA: ${DRM_CI_COMMIT_SHA} + +debian/x86_64_build-base: + variables: + EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libkmod2 libkmod-dev libpciaccess-dev libproc2-dev libudev-dev libunwind-dev python3-docutils bc python3-ply libssl-dev bc" + +debian/x86_64_test-gl: + variables: + EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5 libkmod-dev libkmod2 libgles2 libproc2-dev" + +debian/arm64_build: + variables: + EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libproc2-dev libkmod2 libkmod-dev libpciaccess-dev libudev-dev libunwind-dev python3-docutils libssl-dev crossbuild-essential-armhf libkmod-dev:armhf libproc2-dev:armhf libunwind-dev:armhf libdw-dev:armhf libpixman-1-dev:armhf libcairo-dev:armhf libudev-dev:armhf libjson-c-dev:armhf" + +.kernel+rootfs: + variables: + EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5" + +# Disable container jobs that we won't use +alpine/x86_64_build: + rules: + - when: never + +debian/x86_64_test-vk: + rules: + - when: never + +fedora/x86_64_build: + rules: + - when: never + +debian/android_build: + rules: + - when: never + +debian/x86_64_test-android: + rules: + - when: never + +windows_build_vs2019: + rules: + - when: never + +windows_test_vs2019: + rules: + - when: never + +.debian/x86_64_build-mingw: + rules: + - when: never + +rustfmt: + rules: + - when: never + +windows_vs2019: + rules: + - when: never + +clang-format: + rules: + - when: never \ No newline at end of file diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml new file mode 100644 index 000000000000..2c4df53f5dfe --- /dev/null +++ b/drivers/gpu/drm/ci/gitlab-ci.yml @@ -0,0 +1,251 @@ +variables: + DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa + DRM_CI_COMMIT_SHA: &drm-ci-commit-sha 0dc961645c4f0241f8512cb0ec3ad59635842072 + + UPSTREAM_REPO: git://anongit.freedesktop.org/drm/drm + TARGET_BRANCH: drm-next + + IGT_VERSION: 471bfababd070e1dac0ebb87470ac4f2ae85e663 + + DEQP_RUNNER_GIT_URL: https://gitlab.freedesktop.org/anholt/deqp-runner.git + DEQP_RUNNER_GIT_TAG: v0.15.0 + + FDO_UPSTREAM_REPO: helen.fornazier/linux # The repo where the git-archive daily runs + MESA_TEMPLATES_COMMIT: &ci-templates-commit d5aa3941aa03c2f716595116354fb81eb8012acb + DRM_CI_PROJECT_URL: https://gitlab.freedesktop.org/${DRM_CI_PROJECT_PATH} + CI_PRE_CLONE_SCRIPT: |- + set -o xtrace + curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s ${DRM_CI_PROJECT_URL}/-/raw/${DRM_CI_COMMIT_SHA}/.gitlab-ci/download-git-cache.sh -o download-git-cache.sh + bash download-git-cache.sh + rm download-git-cache.sh + set +o xtrace + S3_HOST: s3.freedesktop.org + # per-pipeline artifact storage on MinIO + PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/artifacts/${CI_PROJECT_PATH}/${CI_PIPELINE_ID} + # per-job artifact storage on MinIO + JOB_ARTIFACTS_BASE: ${PIPELINE_ARTIFACTS_BASE}/${CI_JOB_ID} + + LAVA_JOB_PRIORITY: 30 + +default: + before_script: + - export SCRIPTS_DIR=$(mktemp -d) + - curl -L -s --retry 4 -f --retry-all-errors --retry-delay 60 -O --output-dir "${SCRIPTS_DIR}" "${DRM_CI_PROJECT_URL}/-/raw/${DRM_CI_COMMIT_SHA}/.gitlab-ci/setup-test-env.sh" + - source ${SCRIPTS_DIR}/setup-test-env.sh + - echo -e "\e[0Ksection_start:$(date +%s):unset_env_vars_section[collapsed=true]\r\e[0KUnsetting vulnerable environment variables" + - export CI_JOB_JWT_FILE="${CI_JOB_JWT_FILE:-$(mktemp)}" + - echo -n "${CI_JOB_JWT}" > "${CI_JOB_JWT_FILE}" + - unset CI_JOB_JWT + - echo -e "\e[0Ksection_end:$(date +%s):unset_env_vars_section\r\e[0K" + + - echo -e "\e[0Ksection_start:$(date +%s):drm_ci_download_section[collapsed=true]\r\e[0KDownloading mesa from $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz" + - cd $CI_PROJECT_DIR + - curl --output - $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz | tar -xz + - mv mesa-$DRM_CI_COMMIT_SHA/.gitlab-ci* . + - rm -rf mesa-$DRM_CI_COMMIT_SHA/ + - echo -e "\e[0Ksection_end:$(date +%s):drm_ci_download_section\r\e[0K" + + after_script: + - > + set +x + + test -e "${CI_JOB_JWT_FILE}" && + export CI_JOB_JWT="$(<${CI_JOB_JWT_FILE})" && + rm "${CI_JOB_JWT_FILE}" + + # Retry when job fails. + retry: + max: 1 + # Ignore runner_unsupported, stale_schedule, archived_failure, or + # unmet_prerequisites + when: + - api_failure + - runner_system_failure + - script_failure + - job_execution_timeout + - scheduler_failure + - data_integrity_failure + - unknown_failure + +include: + - project: 'freedesktop/ci-templates' + ref: 16bc29078de5e0a067ff84a1a199a3760d3b3811 + file: + - '/templates/ci-fairy.yml' + - project: 'freedesktop/ci-templates' + ref: *ci-templates-commit + file: + - '/templates/alpine.yml' + - '/templates/debian.yml' + - '/templates/fedora.yml' + - project: *drm-ci-project-path + ref: *drm-ci-commit-sha + file: + - '/.gitlab-ci/farm-rules.yml' + - '/.gitlab-ci/test-source-dep.yml' + - '/.gitlab-ci/container/gitlab-ci.yml' + - '/.gitlab-ci/test/gitlab-ci.yml' + - '/.gitlab-ci/lava/lava-gitlab-ci.yml' + - drivers/gpu/drm/ci/image-tags.yml + - drivers/gpu/drm/ci/container.yml + - drivers/gpu/drm/ci/static-checks.yml + - drivers/gpu/drm/ci/build.yml + - drivers/gpu/drm/ci/test.yml + - 'https://gitlab.freedesktop.org/gfx-ci/lab-status/-/raw/main/lab-status.yml' + + +stages: + - sanity + - container + - git-archive + - build + - amdgpu + - i915 + - mediatek + - meson + - msm + - rockchip + - virtio-gpu + - lint + +# YAML anchors for rule conditions +# -------------------------------- +.rules-anchors: + rules: + # Pipeline for forked project branch + - if: &is-forked-branch '$CI_COMMIT_BRANCH && $CI_PROJECT_NAMESPACE != "mesa"' + when: manual + # Forked project branch / pre-merge pipeline not for Marge bot + - if: &is-forked-branch-or-pre-merge-not-for-marge '$CI_PROJECT_NAMESPACE != "mesa" || ($GITLAB_USER_LOGIN != "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event")' + when: manual + # Pipeline runs for the main branch of the upstream Mesa project + - if: &is-mesa-main '$CI_PROJECT_NAMESPACE == "mesa" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH && $CI_COMMIT_BRANCH' + when: always + # Post-merge pipeline + - if: &is-post-merge '$CI_PROJECT_NAMESPACE == "mesa" && $CI_COMMIT_BRANCH' + when: on_success + # Post-merge pipeline, not for Marge Bot + - if: &is-post-merge-not-for-marge '$CI_PROJECT_NAMESPACE == "mesa" && $GITLAB_USER_LOGIN != "marge-bot" && $CI_COMMIT_BRANCH' + when: on_success + # Pre-merge pipeline + - if: &is-pre-merge '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: on_success + # Pre-merge pipeline for Marge Bot + - if: &is-pre-merge-for-marge '$GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event"' + when: on_success + +# Rule to filter for only scheduled pipelines. +.scheduled_pipeline-rules: + rules: + - if: &is-scheduled-pipeline '$CI_PIPELINE_SOURCE == "schedule"' + when: on_success + +# Generic rule to not run the job during scheduled pipelines. Jobs that aren't +# something like a nightly run should include this rule. +.no_scheduled_pipelines-rules: + rules: + - if: *is-scheduled-pipeline + when: never + +# When to automatically run the CI for build jobs +.build-rules: + rules: + - !reference [.no_scheduled_pipelines-rules, rules] + # Run automatically once all dependency jobs have passed + - when: on_success + + +.ci-deqp-artifacts: + artifacts: + name: "mesa_${CI_JOB_NAME}" + when: always + untracked: false + paths: + # Watch out! Artifacts are relative to the build dir. + # https://gitlab.com/gitlab-org/gitlab-ce/commit/8788fb925706cad594adf6917a6c5f6587dd1521 + - artifacts + - _build/meson-logs/*.txt + - _build/meson-logs/strace + + +.container-rules: + rules: + - !reference [.no_scheduled_pipelines-rules, rules] + # Run pipeline by default in the main project if any CI pipeline + # configuration files were changed, to ensure docker images are up to date + - if: *is-post-merge + changes: + - drivers/gpu/drm/ci/**/* + when: on_success + # Run pipeline by default if it was triggered by Marge Bot, is for a + # merge request, and any files affecting the pipeline were changed + - if: *is-pre-merge-for-marge + when: on_success + # Run pipeline by default in the main project if it was not triggered by + # Marge Bot, and any files affecting the pipeline were changed + - if: *is-post-merge-not-for-marge + when: on_success + # Allow triggering jobs manually in other cases + - when: manual + + + +# Git archive + +make git archive: + extends: + - .fdo.ci-fairy + stage: git-archive + rules: + - !reference [.scheduled_pipeline-rules, rules] + # ensure we are running on packet + tags: + - packet.net + script: + # Remove drm-ci files we just added + - rm -rf .gitlab-ci.* + + # Compactify the .git directory + - git gc --aggressive + # compress the current folder + - tar -cvzf ../$CI_PROJECT_NAME.tar.gz . + + # login with the JWT token file + - ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ../$CI_PROJECT_NAME.tar.gz https://$S3_HOST/git-cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$CI_PROJECT_NAME.tar.gz + + +# Sanity checks of MR settings and commit logs +sanity: + extends: + - .fdo.ci-fairy + stage: sanity + rules: + - if: *is-pre-merge + when: on_success + # Other cases default to never + variables: + GIT_STRATEGY: none + script: + # ci-fairy check-commits --junit-xml=check-commits.xml + - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml + artifacts: + when: on_failure + reports: + junit: check-*.xml + +# Rules for tests that should not block merging, but should be available to +# optionally run with the "play" button in the UI in pre-merge non-marge +# pipelines. This should appear in "extends:" after any includes of +# test-source-dep.yml rules, so that these rules replace those. +.test-manual-mr: + rules: + - !reference [.no_scheduled_pipelines-rules, rules] + - if: *is-forked-branch-or-pre-merge-not-for-marge + when: manual + variables: + JOB_TIMEOUT: 80 + + +# Jobs that need to pass before spending hardware resources on further testing +.required-for-hardware-jobs: + needs: [] \ No newline at end of file diff --git a/drivers/gpu/drm/ci/igt_runner.sh b/drivers/gpu/drm/ci/igt_runner.sh new file mode 100755 index 000000000000..2bb759165063 --- /dev/null +++ b/drivers/gpu/drm/ci/igt_runner.sh @@ -0,0 +1,77 @@ +#!/bin/sh +# SPDX-License-Identifier: MIT + +set -ex + +export IGT_FORCE_DRIVER=${DRIVER_NAME} +export PATH=$PATH:/igt/bin/ +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/igt/lib/aarch64-linux-gnu/:/igt/lib/x86_64-linux-gnu:/igt/lib:/igt/lib64 + +# Uncomment the below to debug problems with driver probing +: ' +ls -l /dev/dri/ +cat /sys/kernel/debug/devices_deferred +cat /sys/kernel/debug/device_component/* +' + +# Dump drm state to confirm that kernel was able to find a connected display: +# TODO this path might not exist for all drivers.. maybe run modetest instead? +set +e +cat /sys/kernel/debug/dri/*/state +set -e + +# Cannot use HWCI_KERNEL_MODULES as at that point we don't have the module in /lib +if [ "$IGT_FORCE_DRIVER" = "amdgpu" ]; then + mv /install/modules/lib/modules/* /lib/modules/. + modprobe amdgpu +fi + +if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-skips.txt" ]; then + IGT_SKIPS="--skips /install/xfails/$DRIVER_NAME-$GPU_VERSION-skips.txt" +fi + +if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-flakes.txt" ]; then + IGT_FLAKES="--flakes /install/xfails/$DRIVER_NAME-$GPU_VERSION-flakes.txt" +fi + +if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-fails.txt" ]; then + IGT_FAILS="--baseline /install/xfails/$DRIVER_NAME-$GPU_VERSION-fails.txt" +fi + +if [ "`uname -m`" = "aarch64" ]; then + ARCH="arm64" +elif [ "`uname -m`" = "armv7l" ]; then + ARCH="arm" +else + ARCH="x86_64" +fi + +curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s ${FDO_HTTP_CACHE_URI:-}$PIPELINE_ARTIFACTS_BASE/$ARCH/igt.tar.gz | tar --zstd -v -x -C / + +set +e +igt-runner \ + run \ + --igt-folder /igt/libexec/igt-gpu-tools \ + --caselist /install/testlist.txt \ + --output /results \ + $IGT_SKIPS \ + $IGT_FLAKES \ + $IGT_FAILS \ + --fraction-start $CI_NODE_INDEX \ + --fraction $CI_NODE_TOTAL \ + --jobs 1 +ret=$? +set -e + +deqp-runner junit \ + --testsuite IGT \ + --results /results/failures.csv \ + --output /results/junit.xml \ + --limit 50 \ + --template "See https://$CI_PROJECT_ROOT_NAMESPACE.pages.freedesktop.org/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/results/{{testcase}}.xml" + +# Store the results also in the simpler format used by the runner in ChromeOS CI +#sed -r 's/(dmesg-warn|pass)/success/g' /results/results.txt > /results/results_simple.txt + +cd $oldpath +exit $ret diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml new file mode 100644 index 000000000000..f051b6c547c5 --- /dev/null +++ b/drivers/gpu/drm/ci/image-tags.yml @@ -0,0 +1,15 @@ +variables: + CONTAINER_TAG: "2023-08-10-mesa-uprev" + DEBIAN_X86_64_BUILD_BASE_IMAGE: "debian/x86_64_build-base" + DEBIAN_BASE_TAG: "${CONTAINER_TAG}" + + DEBIAN_X86_64_BUILD_IMAGE_PATH: "debian/x86_64_build" + DEBIAN_BUILD_TAG: "${CONTAINER_TAG}" + + KERNEL_ROOTFS_TAG: "${CONTAINER_TAG}" + + DEBIAN_X86_64_TEST_BASE_IMAGE: "debian/x86_64_test-base" + DEBIAN_X86_64_TEST_IMAGE_GL_PATH: "debian/x86_64_test-gl" + DEBIAN_X86_64_TEST_GL_TAG: "${CONTAINER_TAG}" + + ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}" \ No newline at end of file diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh new file mode 100755 index 000000000000..0c4456b21b0f --- /dev/null +++ b/drivers/gpu/drm/ci/lava-submit.sh @@ -0,0 +1,57 @@ +#!/bin/bash +# SPDX-License-Identifier: MIT + +set -e +set -x + +# Try to use the kernel and rootfs built in mainline first, so we're more +# likely to hit cache +if curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s "https://${BASE_SYSTEM_MAINLINE_HOST_PATH}/done"; then + BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_MAINLINE_HOST_PATH}" +else + BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_FORK_HOST_PATH}" +fi + +rm -rf results +mkdir -p results/job-rootfs-overlay/ + +cp artifacts/ci-common/capture-devcoredump.sh results/job-rootfs-overlay/ +cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/ +cp artifacts/ci-common/intel-gpu-freq.sh results/job-rootfs-overlay/ +cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/ + +# Prepare env vars for upload. +section_start variables "Variables passed through:" +KERNEL_IMAGE_BASE_URL="https://${BASE_SYSTEM_HOST_PATH}" \ + artifacts/ci-common/generate-env.sh | tee results/job-rootfs-overlay/set-job-env-vars.sh +section_end variables + +tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ . +ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}" + +touch results/lava.log +tail -f results/lava.log & + +PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \ + submit \ + --dump-yaml \ + --pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \ + --rootfs-url-prefix "https://${BASE_SYSTEM_HOST_PATH}" \ + --kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${ARCH}" \ + --build-url "${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${ARCH}/kernel-files.tar.zst" \ + --job-rootfs-overlay-url "${FDO_HTTP_CACHE_URI:-}https://${JOB_ROOTFS_OVERLAY_PATH}" \ + --job-timeout-min ${JOB_TIMEOUT:-80} \ + --first-stage-init artifacts/ci-common/init-stage1.sh \ + --ci-project-dir "${CI_PROJECT_DIR}" \ + --device-type "${DEVICE_TYPE}" \ + --dtb-filename "${DTB}" \ + --jwt-file "${CI_JOB_JWT_FILE}" \ + --kernel-image-name "${KERNEL_IMAGE_NAME}" \ + --kernel-image-type "${KERNEL_IMAGE_TYPE}" \ + --boot-method "${BOOT_METHOD}" \ + --visibility-group "${VISIBILITY_GROUP}" \ + --lava-tags "${LAVA_TAGS}" \ + --mesa-job-name "$CI_JOB_NAME" \ + --structured-log-file "results/lava_job_detail.json" \ + --ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \ + >> results/lava.log diff --git a/drivers/gpu/drm/ci/static-checks.yml b/drivers/gpu/drm/ci/static-checks.yml new file mode 100644 index 000000000000..13ffa827b7fa --- /dev/null +++ b/drivers/gpu/drm/ci/static-checks.yml @@ -0,0 +1,12 @@ +check-patch: + extends: + - .build + - .use-debian/x86_64_build + script: + - drivers/gpu/drm/ci/check-patch.py + variables: + CHECKPATCH_TYPES: "BAD_SIGN_OFF,BAD_STABLE_ADDRESS_STYLE,COMMIT_COMMENT_SYMBOL,COMMIT_MESSAGE,EMAIL_SUBJECT,FROM_SIGN_OFF_MISMATCH,MISSING_SIGN_OFF,NO_AUTHOR_SIGN_OFF,DIFF_IN_COMMIT_MSG,GERRIT_CHANGE_ID,GIT_COMMIT_ID,UNKNOWN_COMMIT_ID,CODE_INDENT,BIT_MACRO,DOS_LINE_ENDINGS" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: on_success + # Other cases default to never diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml new file mode 100644 index 000000000000..6473cddaa7a9 --- /dev/null +++ b/drivers/gpu/drm/ci/test.yml @@ -0,0 +1,335 @@ +.test-rules: + rules: + - if: '$FD_FARM == "offline" && $RUNNER_TAG =~ /^google-freedreno-/' + when: never + - if: '$COLLABORA_FARM == "offline" && $RUNNER_TAG =~ /^mesa-ci-x86-64-lava-/' + when: never + - !reference [.no_scheduled_pipelines-rules, rules] + - when: on_success + +.lava-test: + extends: + - .test-rules + script: + # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY + - rm -rf install + - tar -xf artifacts/install.tar + - mv install/* artifacts/. + # Override it with our lava-submit.sh script + - ./artifacts/lava-submit.sh + +.lava-igt:arm32: + extends: + - .lava-test:arm32 + variables: + HWCI_TEST_SCRIPT: "/install/igt_runner.sh" + ARCH: "armhf" + dependencies: + - testing:arm32 + needs: + - alpine/x86_64_lava_ssh_client + - kernel+rootfs_arm32 + - debian/x86_64_build + - testing:arm32 + - igt:arm32 + +.lava-igt:arm64: + extends: + - .lava-test:arm64 + variables: + HWCI_TEST_SCRIPT: "/install/igt_runner.sh" + ARCH: "arm64" + dependencies: + - testing:arm64 + needs: + - alpine/x86_64_lava_ssh_client + - kernel+rootfs_arm64 + - debian/x86_64_build + - testing:arm64 + - igt:arm64 + +.lava-igt:x86_64: + extends: + - .lava-test:x86_64 + variables: + HWCI_TEST_SCRIPT: "/install/igt_runner.sh" + ARCH: "x86_64" + dependencies: + - testing:x86_64 + needs: + - alpine/x86_64_lava_ssh_client + - kernel+rootfs_x86_64 + - debian/x86_64_build + - testing:x86_64 + - igt:x86_64 + +.baremetal-igt-arm64: + extends: + - .baremetal-test-arm64 + - .use-debian/arm64_test + - .test-rules + variables: + FDO_CI_CONCURRENT: 10 + HWCI_TEST_SCRIPT: "/install/igt_runner.sh" + S3_ARTIFACT_NAME: "arm64/kernel-files" + BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/Image.gz + BM_CMDLINE: "ip=dhcp console=ttyMSM0,115200n8 $BM_KERNEL_EXTRA_ARGS root=/dev/nfs rw nfsrootdebug nfsroot=,tcp,nfsvers=4.2 init=/init $BM_KERNELARGS" + needs: + - debian/arm64_test + - job: testing:arm64 + artifacts: false + - igt:arm64 + tags: + - $RUNNER_TAG + +msm:sc7180: + extends: + - .lava-igt:arm64 + stage: msm + parallel: 2 + variables: + DRIVER_NAME: msm + DEVICE_TYPE: sc7180-trogdor-lazor-limozeen + DTB: sc7180-trogdor-lazor-limozeen-nots-r5 + BOOT_METHOD: depthcharge + KERNEL_IMAGE_TYPE: "" + GPU_VERSION: sc7180 + RUNNER_TAG: mesa-ci-x86-64-lava-sc7180-trogdor-lazor-limozeen + +msm:apq8016: + extends: + - .baremetal-igt-arm64 + stage: msm + variables: + DRIVER_NAME: msm + BM_DTB: https://${PIPELINE_ARTIFACTS_BASE}/arm64/apq8016-sbc.dtb + GPU_VERSION: apq8016 + BM_CMDLINE: "ip=dhcp console=ttyMSM0,115200n8 $BM_KERNEL_EXTRA_ARGS root=/dev/nfs rw nfsrootdebug nfsroot=,tcp,nfsvers=4.2 init=/init $BM_KERNELARGS" + RUNNER_TAG: google-freedreno-db410c + script: + - ./install/bare-metal/fastboot.sh + rules: + # TODO: current issue: it is not fiding the NFS root. Fix and remove this rule. + - when: never + +msm:apq8096: + extends: + - .baremetal-igt-arm64 + stage: msm + variables: + DRIVER_NAME: msm + BM_KERNEL_EXTRA_ARGS: maxcpus=2 + BM_DTB: https://${PIPELINE_ARTIFACTS_BASE}/arm64/apq8096-db820c.dtb + GPU_VERSION: apq8096 + RUNNER_TAG: google-freedreno-db820c + script: + - ./install/bare-metal/fastboot.sh + +msm:sdm845: + extends: + - .baremetal-igt-arm64 + stage: msm + parallel: 6 + variables: + DRIVER_NAME: msm + BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/cheza-kernel + GPU_VERSION: sdm845 + RUNNER_TAG: google-freedreno-cheza + script: + - ./install/bare-metal/cros-servo.sh + +rockchip:rk3288: + extends: + - .lava-igt:arm32 + stage: rockchip + variables: + DRIVER_NAME: rockchip + DEVICE_TYPE: rk3288-veyron-jaq + DTB: ${DEVICE_TYPE} + BOOT_METHOD: depthcharge + KERNEL_IMAGE_TYPE: "zimage" + GPU_VERSION: rk3288 + RUNNER_TAG: mesa-ci-x86-64-lava-rk3288-veyron-jaq + +rockchip:rk3399: + extends: + - .lava-igt:arm64 + stage: rockchip + parallel: 3 + variables: + DRIVER_NAME: rockchip + DEVICE_TYPE: rk3399-gru-kevin + DTB: ${DEVICE_TYPE} + BOOT_METHOD: depthcharge + KERNEL_IMAGE_TYPE: "" + GPU_VERSION: rk3399 + RUNNER_TAG: mesa-ci-x86-64-lava-rk3399-gru-kevin + +.i915: + extends: + - .lava-igt:x86_64 + stage: i915 + variables: + DRIVER_NAME: i915 + DTB: "" + BOOT_METHOD: depthcharge + KERNEL_IMAGE_TYPE: "" + +i915:apl: + extends: + - .i915 + parallel: 12 + variables: + DEVICE_TYPE: asus-C523NA-A20057-coral + GPU_VERSION: apl + RUNNER_TAG: mesa-ci-x86-64-lava-asus-C523NA-A20057-coral + +i915:glk: + extends: + - .i915 + parallel: 5 + variables: + DEVICE_TYPE: hp-x360-12b-ca0010nr-n4020-octopus + GPU_VERSION: glk + RUNNER_TAG: mesa-ci-x86-64-lava-hp-x360-12b-ca0010nr-n4020-octopus + +i915:amly: + extends: + - .i915 + parallel: 8 + variables: + DEVICE_TYPE: asus-C433TA-AJ0005-rammus + GPU_VERSION: amly + RUNNER_TAG: mesa-ci-x86-64-lava-asus-C433TA-AJ0005-rammus + +i915:kbl: + extends: + - .i915 + parallel: 5 + variables: + DEVICE_TYPE: hp-x360-14-G1-sona + GPU_VERSION: kbl + RUNNER_TAG: mesa-ci-x86-64-lava-hp-x360-14-G1-sona + +i915:whl: + extends: + - .i915 + parallel: 8 + variables: + DEVICE_TYPE: dell-latitude-5400-8665U-sarien + GPU_VERSION: whl + RUNNER_TAG: mesa-ci-x86-64-lava-dell-latitude-5400-8665U-sarien + +i915:cml: + extends: + - .i915 + parallel: 6 + variables: + DEVICE_TYPE: asus-C436FA-Flip-hatch + GPU_VERSION: cml + RUNNER_TAG: mesa-ci-x86-64-lava-asus-C436FA-flip-hatch + +i915:tgl: + extends: + - .i915 + parallel: 6 + variables: + DEVICE_TYPE: asus-cx9400-volteer + GPU_VERSION: tgl + RUNNER_TAG: mesa-ci-x86-64-lava-asus-cx9400-volteer + +.amdgpu: + extends: + - .lava-igt:x86_64 + stage: amdgpu + variables: + DRIVER_NAME: amdgpu + DTB: "" + BOOT_METHOD: depthcharge + KERNEL_IMAGE_TYPE: "" + +amdgpu:stoney: + extends: + - .amdgpu + variables: + DEVICE_TYPE: hp-11A-G6-EE-grunt + GPU_VERSION: stoney + RUNNER_TAG: mesa-ci-x86-64-lava-hp-11A-G6-EE-grunt + +.mediatek: + extends: + - .lava-igt:arm64 + stage: mediatek + variables: + DRIVER_NAME: mediatek + DTB: ${DEVICE_TYPE} + BOOT_METHOD: depthcharge + KERNEL_IMAGE_TYPE: "" + +mediatek:mt8173: + extends: + - .mediatek + variables: + DEVICE_TYPE: mt8173-elm-hana + GPU_VERSION: mt8173 + RUNNER_TAG: mesa-ci-x86-64-lava-mt8173-elm-hana + rules: + # TODO: current issue: device is hanging. Fix and remove this rule. + - when: never + +mediatek:mt8183: + extends: + - .mediatek + variables: + DEVICE_TYPE: mt8183-kukui-jacuzzi-juniper-sku16 + GPU_VERSION: mt8183 + RUNNER_TAG: mesa-ci-x86-64-lava-mt8183-kukui-jacuzzi-juniper-sku16 + +# drm-mtk doesn't even probe yet in mainline for mt8192 +.mediatek:mt8192: + extends: + - .mediatek + variables: + DEVICE_TYPE: mt8192-asurada-spherion-r0 + GPU_VERSION: mt8192 + RUNNER_TAG: mesa-ci-x86-64-lava-mt8192-asurada-spherion-r0 + +.meson: + extends: + - .lava-igt:arm64 + stage: meson + variables: + DRIVER_NAME: meson + DTB: ${DEVICE_TYPE} + BOOT_METHOD: u-boot + KERNEL_IMAGE_TYPE: "image" + +meson:g12b: + extends: + - .meson + variables: + DEVICE_TYPE: meson-g12b-a311d-khadas-vim3 + GPU_VERSION: g12b + RUNNER_TAG: mesa-ci-x86-64-lava-meson-g12b-a311d-khadas-vim3 + +virtio_gpu:none: + stage: virtio-gpu + variables: + CROSVM_GALLIUM_DRIVER: llvmpipe + DRIVER_NAME: virtio_gpu + GPU_VERSION: none + extends: + - .test-gl + tags: + - kvm + script: + - ln -sf $CI_PROJECT_DIR/install /install + - mv install/bzImage /lava-files/bzImage + - install/crosvm-runner.sh install/igt_runner.sh + needs: + - debian/x86_64_test-gl + - testing:x86_64 + - igt:x86_64 + rules: + # TODO: current issue: malloc(): corrupted top size. Fix and remove this rule. + - when: never \ No newline at end of file diff --git a/drivers/gpu/drm/ci/testlist.txt b/drivers/gpu/drm/ci/testlist.txt new file mode 100644 index 000000000000..f82cd90372f4 --- /dev/null +++ b/drivers/gpu/drm/ci/testlist.txt @@ -0,0 +1,2912 @@ +core_auth@getclient-simple +core_auth@getclient-master-drop +core_auth@basic-auth +core_auth@many-magics +core_getclient +core_getstats +core_getversion +core_setmaster_vs_auth +drm_read@invalid-buffer +drm_read@fault-buffer +drm_read@empty-block +drm_read@empty-nonblock +drm_read@short-buffer-block +drm_read@short-buffer-nonblock +drm_read@short-buffer-wakeup +gem_eio@throttle +gem_eio@create +gem_eio@create-ext +gem_eio@context-create +gem_eio@execbuf +gem_eio@banned +gem_eio@suspend +gem_eio@hibernate +gem_eio@in-flight-external +gem_eio@in-flight-suspend +gem_eio@reset-stress +gem_eio@unwedge-stress +gem_eio@wait-immediate +gem_eio@wait-wedge-immediate +gem_eio@in-flight-immediate +gem_eio@in-flight-contexts-immediate +gem_eio@in-flight-internal-immediate +gem_eio@wait-1us +gem_eio@wait-wedge-1us +gem_eio@in-flight-1us +gem_eio@in-flight-contexts-1us +gem_eio@in-flight-internal-1us +gem_eio@wait-10ms +gem_eio@wait-wedge-10ms +gem_eio@in-flight-10ms +gem_eio@in-flight-contexts-10ms +gem_eio@in-flight-internal-10ms +gem_eio@kms +kms_3d +kms_addfb_basic@unused-handle +kms_addfb_basic@unused-pitches +kms_addfb_basic@unused-offsets +kms_addfb_basic@unused-modifier +kms_addfb_basic@clobberred-modifier +kms_addfb_basic@invalid-smem-bo-on-discrete +kms_addfb_basic@legacy-format +kms_addfb_basic@no-handle +kms_addfb_basic@basic +kms_addfb_basic@bad-pitch-0 +kms_addfb_basic@bad-pitch-32 +kms_addfb_basic@bad-pitch-63 +kms_addfb_basic@bad-pitch-128 +kms_addfb_basic@bad-pitch-256 +kms_addfb_basic@bad-pitch-1024 +kms_addfb_basic@bad-pitch-999 +kms_addfb_basic@bad-pitch-65536 +kms_addfb_basic@invalid-get-prop-any +kms_addfb_basic@invalid-get-prop +kms_addfb_basic@invalid-set-prop-any +kms_addfb_basic@invalid-set-prop +kms_addfb_basic@master-rmfb +kms_addfb_basic@addfb25-modifier-no-flag +kms_addfb_basic@addfb25-bad-modifier +kms_addfb_basic@addfb25-x-tiled-mismatch-legacy +kms_addfb_basic@addfb25-x-tiled-legacy +kms_addfb_basic@addfb25-framebuffer-vs-set-tiling +kms_addfb_basic@basic-x-tiled-legacy +kms_addfb_basic@framebuffer-vs-set-tiling +kms_addfb_basic@tile-pitch-mismatch +kms_addfb_basic@basic-y-tiled-legacy +kms_addfb_basic@size-max +kms_addfb_basic@too-wide +kms_addfb_basic@too-high +kms_addfb_basic@bo-too-small +kms_addfb_basic@small-bo +kms_addfb_basic@bo-too-small-due-to-tiling +kms_addfb_basic@addfb25-y-tiled-legacy +kms_addfb_basic@addfb25-yf-tiled-legacy +kms_addfb_basic@addfb25-y-tiled-small-legacy +kms_addfb_basic@addfb25-4-tiled +kms_async_flips@async-flip-with-page-flip-events +kms_async_flips@alternate-sync-async-flip +kms_async_flips@test-time-stamp +kms_async_flips@test-cursor +kms_async_flips@invalid-async-flip +kms_async_flips@crc +kms_atomic@plane-overlay-legacy +kms_atomic@plane-primary-legacy +kms_atomic@plane-primary-overlay-mutable-zpos +kms_atomic@plane-immutable-zpos +kms_atomic@test-only +kms_atomic@plane-cursor-legacy +kms_atomic@plane-invalid-params +kms_atomic@plane-invalid-params-fence +kms_atomic@crtc-invalid-params +kms_atomic@crtc-invalid-params-fence +kms_atomic@atomic-invalid-params +kms_atomic@atomic_plane_damage +kms_atomic_interruptible@legacy-setmode +kms_atomic_interruptible@atomic-setmode +kms_atomic_interruptible@legacy-dpms +kms_atomic_interruptible@legacy-pageflip +kms_atomic_interruptible@legacy-cursor +kms_atomic_interruptible@universal-setplane-primary +kms_atomic_interruptible@universal-setplane-cursor +kms_atomic_transition@plane-primary-toggle-with-vblank-wait +kms_atomic_transition@plane-all-transition +kms_atomic_transition@plane-all-transition-fencing +kms_atomic_transition@plane-all-transition-nonblocking +kms_atomic_transition@plane-all-transition-nonblocking-fencing +kms_atomic_transition@plane-use-after-nonblocking-unbind +kms_atomic_transition@plane-use-after-nonblocking-unbind-fencing +kms_atomic_transition@plane-all-modeset-transition +kms_atomic_transition@plane-all-modeset-transition-fencing +kms_atomic_transition@plane-all-modeset-transition-internal-panels +kms_atomic_transition@plane-all-modeset-transition-fencing-internal-panels +kms_atomic_transition@plane-toggle-modeset-transition +kms_atomic_transition@modeset-transition +kms_atomic_transition@modeset-transition-fencing +kms_atomic_transition@modeset-transition-nonblocking +kms_atomic_transition@modeset-transition-nonblocking-fencing +kms_big_fb@x-tiled-addfb-size-overflow +kms_big_fb@y-tiled-addfb-size-overflow +kms_big_fb@yf-tiled-addfb-size-overflow +kms_big_fb@4-tiled-addfb-size-overflow +kms_big_fb@x-tiled-addfb-size-offset-overflow +kms_big_fb@y-tiled-addfb-size-offset-overflow +kms_big_fb@yf-tiled-addfb-size-offset-overflow +kms_big_fb@4-tiled-addfb-size-offset-overflow +kms_big_fb@linear-addfb +kms_big_fb@x-tiled-addfb +kms_big_fb@y-tiled-addfb +kms_big_fb@yf-tiled-addfb +kms_big_fb@4-tiled-addfb +kms_big_fb@linear-8bpp-rotate-0 +kms_big_fb@linear-8bpp-rotate-90 +kms_big_fb@linear-8bpp-rotate-180 +kms_big_fb@linear-8bpp-rotate-270 +kms_big_fb@linear-16bpp-rotate-0 +kms_big_fb@linear-16bpp-rotate-90 +kms_big_fb@linear-16bpp-rotate-180 +kms_big_fb@linear-16bpp-rotate-270 +kms_big_fb@linear-32bpp-rotate-0 +kms_big_fb@linear-32bpp-rotate-90 +kms_big_fb@linear-32bpp-rotate-180 +kms_big_fb@linear-32bpp-rotate-270 +kms_big_fb@linear-64bpp-rotate-0 +kms_big_fb@linear-64bpp-rotate-90 +kms_big_fb@linear-64bpp-rotate-180 +kms_big_fb@linear-64bpp-rotate-270 +kms_big_fb@x-tiled-8bpp-rotate-0 +kms_big_fb@x-tiled-8bpp-rotate-90 +kms_big_fb@x-tiled-8bpp-rotate-180 +kms_big_fb@x-tiled-8bpp-rotate-270 +kms_big_fb@x-tiled-16bpp-rotate-0 +kms_big_fb@x-tiled-16bpp-rotate-90 +kms_big_fb@x-tiled-16bpp-rotate-180 +kms_big_fb@x-tiled-16bpp-rotate-270 +kms_big_fb@x-tiled-32bpp-rotate-0 +kms_big_fb@x-tiled-32bpp-rotate-90 +kms_big_fb@x-tiled-32bpp-rotate-180 +kms_big_fb@x-tiled-32bpp-rotate-270 +kms_big_fb@x-tiled-64bpp-rotate-0 +kms_big_fb@x-tiled-64bpp-rotate-90 +kms_big_fb@x-tiled-64bpp-rotate-180 +kms_big_fb@x-tiled-64bpp-rotate-270 +kms_big_fb@y-tiled-8bpp-rotate-0 +kms_big_fb@y-tiled-8bpp-rotate-90 +kms_big_fb@y-tiled-8bpp-rotate-180 +kms_big_fb@y-tiled-8bpp-rotate-270 +kms_big_fb@y-tiled-16bpp-rotate-0 +kms_big_fb@y-tiled-16bpp-rotate-90 +kms_big_fb@y-tiled-16bpp-rotate-180 +kms_big_fb@y-tiled-16bpp-rotate-270 +kms_big_fb@y-tiled-32bpp-rotate-0 +kms_big_fb@y-tiled-32bpp-rotate-90 +kms_big_fb@y-tiled-32bpp-rotate-180 +kms_big_fb@y-tiled-32bpp-rotate-270 +kms_big_fb@y-tiled-64bpp-rotate-0 +kms_big_fb@y-tiled-64bpp-rotate-90 +kms_big_fb@y-tiled-64bpp-rotate-180 +kms_big_fb@y-tiled-64bpp-rotate-270 +kms_big_fb@yf-tiled-8bpp-rotate-0 +kms_big_fb@yf-tiled-8bpp-rotate-90 +kms_big_fb@yf-tiled-8bpp-rotate-180 +kms_big_fb@yf-tiled-8bpp-rotate-270 +kms_big_fb@yf-tiled-16bpp-rotate-0 +kms_big_fb@yf-tiled-16bpp-rotate-90 +kms_big_fb@yf-tiled-16bpp-rotate-180 +kms_big_fb@yf-tiled-16bpp-rotate-270 +kms_big_fb@yf-tiled-32bpp-rotate-0 +kms_big_fb@yf-tiled-32bpp-rotate-90 +kms_big_fb@yf-tiled-32bpp-rotate-180 +kms_big_fb@yf-tiled-32bpp-rotate-270 +kms_big_fb@yf-tiled-64bpp-rotate-0 +kms_big_fb@yf-tiled-64bpp-rotate-90 +kms_big_fb@yf-tiled-64bpp-rotate-180 +kms_big_fb@yf-tiled-64bpp-rotate-270 +kms_big_fb@4-tiled-8bpp-rotate-0 +kms_big_fb@4-tiled-8bpp-rotate-90 +kms_big_fb@4-tiled-8bpp-rotate-180 +kms_big_fb@4-tiled-8bpp-rotate-270 +kms_big_fb@4-tiled-16bpp-rotate-0 +kms_big_fb@4-tiled-16bpp-rotate-90 +kms_big_fb@4-tiled-16bpp-rotate-180 +kms_big_fb@4-tiled-16bpp-rotate-270 +kms_big_fb@4-tiled-32bpp-rotate-0 +kms_big_fb@4-tiled-32bpp-rotate-90 +kms_big_fb@4-tiled-32bpp-rotate-180 +kms_big_fb@4-tiled-32bpp-rotate-270 +kms_big_fb@4-tiled-64bpp-rotate-0 +kms_big_fb@4-tiled-64bpp-rotate-90 +kms_big_fb@4-tiled-64bpp-rotate-180 +kms_big_fb@4-tiled-64bpp-rotate-270 +kms_big_fb@linear-max-hw-stride-32bpp-rotate-0 +kms_big_fb@linear-max-hw-stride-32bpp-rotate-180 +kms_big_fb@linear-max-hw-stride-64bpp-rotate-0 +kms_big_fb@linear-max-hw-stride-64bpp-rotate-180 +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0 +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-async-flip +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180 +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-async-flip +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0 +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-async-flip +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180 +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-async-flip +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-hflip +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-hflip +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-hflip +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-hflip +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0 +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-async-flip +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180 +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-async-flip +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0 +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-async-flip +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180 +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-async-flip +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-hflip +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-hflip +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-hflip +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-hflip +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0 +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-async-flip +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180 +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-async-flip +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0 +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-async-flip +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180 +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-async-flip +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-hflip +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-hflip +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-hflip +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-hflip +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0 +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-async-flip +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180 +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-async-flip +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0 +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-async-flip +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180 +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-async-flip +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-hflip +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-hflip +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-hflip +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-hflip +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip +kms_big_joiner@basic +kms_big_joiner@invalid-modeset +kms_big_joiner@2x-modeset +kms_busy@basic +kms_busy@basic-hang +kms_busy@extended-pageflip-modeset-hang-oldfb +kms_busy@extended-pageflip-hang-oldfb +kms_busy@extended-pageflip-hang-newfb +kms_busy@extended-modeset-hang-oldfb +kms_busy@extended-modeset-hang-newfb +kms_busy@extended-modeset-hang-oldfb-with-reset +kms_busy@extended-modeset-hang-newfb-with-reset +kms_bw@linear-tiling-1-displays-1920x1080p +kms_bw@linear-tiling-1-displays-2560x1440p +kms_bw@linear-tiling-1-displays-3840x2160p +kms_bw@linear-tiling-2-displays-1920x1080p +kms_bw@linear-tiling-2-displays-2560x1440p +kms_bw@linear-tiling-2-displays-3840x2160p +kms_bw@linear-tiling-3-displays-1920x1080p +kms_bw@linear-tiling-3-displays-2560x1440p +kms_bw@linear-tiling-3-displays-3840x2160p +kms_bw@linear-tiling-4-displays-1920x1080p +kms_bw@linear-tiling-4-displays-2560x1440p +kms_bw@linear-tiling-4-displays-3840x2160p +kms_bw@linear-tiling-5-displays-1920x1080p +kms_bw@linear-tiling-5-displays-2560x1440p +kms_bw@linear-tiling-5-displays-3840x2160p +kms_bw@linear-tiling-6-displays-1920x1080p +kms_bw@linear-tiling-6-displays-2560x1440p +kms_bw@linear-tiling-6-displays-3840x2160p +kms_bw@linear-tiling-7-displays-1920x1080p +kms_bw@linear-tiling-7-displays-2560x1440p +kms_bw@linear-tiling-7-displays-3840x2160p +kms_bw@linear-tiling-8-displays-1920x1080p +kms_bw@linear-tiling-8-displays-2560x1440p +kms_bw@linear-tiling-8-displays-3840x2160p +kms_ccs@pipe-A-bad-pixel-format-y_tiled_ccs +kms_ccs@pipe-A-bad-pixel-format-yf_tiled_ccs +kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_rc_ccs +kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_mc_ccs +kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-A-bad-rotation-90-y_tiled_ccs +kms_ccs@pipe-A-bad-rotation-90-yf_tiled_ccs +kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_rc_ccs +kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_mc_ccs +kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-A-crc-primary-basic-y_tiled_ccs +kms_ccs@pipe-A-crc-primary-basic-yf_tiled_ccs +kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_ccs +kms_ccs@pipe-A-crc-primary-rotation-180-yf_tiled_ccs +kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_rc_ccs +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_mc_ccs +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-A-random-ccs-data-y_tiled_ccs +kms_ccs@pipe-A-random-ccs-data-yf_tiled_ccs +kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_rc_ccs +kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_mc_ccs +kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_ccs +kms_ccs@pipe-A-missing-ccs-buffer-yf_tiled_ccs +kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_ccs +kms_ccs@pipe-A-ccs-on-another-bo-yf_tiled_ccs +kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-A-bad-aux-stride-y_tiled_ccs +kms_ccs@pipe-A-bad-aux-stride-yf_tiled_ccs +kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_ccs +kms_ccs@pipe-A-crc-sprite-planes-basic-yf_tiled_ccs +kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-bad-pixel-format-y_tiled_ccs +kms_ccs@pipe-B-bad-pixel-format-yf_tiled_ccs +kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_rc_ccs +kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_mc_ccs +kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-bad-rotation-90-y_tiled_ccs +kms_ccs@pipe-B-bad-rotation-90-yf_tiled_ccs +kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_rc_ccs +kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_mc_ccs +kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-crc-primary-basic-y_tiled_ccs +kms_ccs@pipe-B-crc-primary-basic-yf_tiled_ccs +kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_ccs +kms_ccs@pipe-B-crc-primary-rotation-180-yf_tiled_ccs +kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_rc_ccs +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_mc_ccs +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-random-ccs-data-y_tiled_ccs +kms_ccs@pipe-B-random-ccs-data-yf_tiled_ccs +kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_rc_ccs +kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_mc_ccs +kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_ccs +kms_ccs@pipe-B-missing-ccs-buffer-yf_tiled_ccs +kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_ccs +kms_ccs@pipe-B-ccs-on-another-bo-yf_tiled_ccs +kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-bad-aux-stride-y_tiled_ccs +kms_ccs@pipe-B-bad-aux-stride-yf_tiled_ccs +kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_ccs +kms_ccs@pipe-B-crc-sprite-planes-basic-yf_tiled_ccs +kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-bad-pixel-format-y_tiled_ccs +kms_ccs@pipe-C-bad-pixel-format-yf_tiled_ccs +kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_rc_ccs +kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_mc_ccs +kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-bad-rotation-90-y_tiled_ccs +kms_ccs@pipe-C-bad-rotation-90-yf_tiled_ccs +kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_rc_ccs +kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_mc_ccs +kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-crc-primary-basic-y_tiled_ccs +kms_ccs@pipe-C-crc-primary-basic-yf_tiled_ccs +kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_ccs +kms_ccs@pipe-C-crc-primary-rotation-180-yf_tiled_ccs +kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_rc_ccs +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_mc_ccs +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-random-ccs-data-y_tiled_ccs +kms_ccs@pipe-C-random-ccs-data-yf_tiled_ccs +kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_rc_ccs +kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_mc_ccs +kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_ccs +kms_ccs@pipe-C-missing-ccs-buffer-yf_tiled_ccs +kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_ccs +kms_ccs@pipe-C-ccs-on-another-bo-yf_tiled_ccs +kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-bad-aux-stride-y_tiled_ccs +kms_ccs@pipe-C-bad-aux-stride-yf_tiled_ccs +kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_ccs +kms_ccs@pipe-C-crc-sprite-planes-basic-yf_tiled_ccs +kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-bad-pixel-format-y_tiled_ccs +kms_ccs@pipe-D-bad-pixel-format-yf_tiled_ccs +kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_rc_ccs +kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_mc_ccs +kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-bad-rotation-90-y_tiled_ccs +kms_ccs@pipe-D-bad-rotation-90-yf_tiled_ccs +kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_rc_ccs +kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_mc_ccs +kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-crc-primary-basic-y_tiled_ccs +kms_ccs@pipe-D-crc-primary-basic-yf_tiled_ccs +kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_ccs +kms_ccs@pipe-D-crc-primary-rotation-180-yf_tiled_ccs +kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_rc_ccs +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_mc_ccs +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-random-ccs-data-y_tiled_ccs +kms_ccs@pipe-D-random-ccs-data-yf_tiled_ccs +kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_rc_ccs +kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_mc_ccs +kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_ccs +kms_ccs@pipe-D-missing-ccs-buffer-yf_tiled_ccs +kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_ccs +kms_ccs@pipe-D-ccs-on-another-bo-yf_tiled_ccs +kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-bad-aux-stride-y_tiled_ccs +kms_ccs@pipe-D-bad-aux-stride-yf_tiled_ccs +kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_ccs +kms_ccs@pipe-D-crc-sprite-planes-basic-yf_tiled_ccs +kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-bad-pixel-format-y_tiled_ccs +kms_ccs@pipe-E-bad-pixel-format-yf_tiled_ccs +kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_rc_ccs +kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_mc_ccs +kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-bad-rotation-90-y_tiled_ccs +kms_ccs@pipe-E-bad-rotation-90-yf_tiled_ccs +kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_rc_ccs +kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_mc_ccs +kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-crc-primary-basic-y_tiled_ccs +kms_ccs@pipe-E-crc-primary-basic-yf_tiled_ccs +kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_ccs +kms_ccs@pipe-E-crc-primary-rotation-180-yf_tiled_ccs +kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_rc_ccs +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_mc_ccs +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-random-ccs-data-y_tiled_ccs +kms_ccs@pipe-E-random-ccs-data-yf_tiled_ccs +kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_rc_ccs +kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_mc_ccs +kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_ccs +kms_ccs@pipe-E-missing-ccs-buffer-yf_tiled_ccs +kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_ccs +kms_ccs@pipe-E-ccs-on-another-bo-yf_tiled_ccs +kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-bad-aux-stride-y_tiled_ccs +kms_ccs@pipe-E-bad-aux-stride-yf_tiled_ccs +kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_ccs +kms_ccs@pipe-E-crc-sprite-planes-basic-yf_tiled_ccs +kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-bad-pixel-format-y_tiled_ccs +kms_ccs@pipe-F-bad-pixel-format-yf_tiled_ccs +kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_rc_ccs +kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_mc_ccs +kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-bad-rotation-90-y_tiled_ccs +kms_ccs@pipe-F-bad-rotation-90-yf_tiled_ccs +kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_rc_ccs +kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_mc_ccs +kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-crc-primary-basic-y_tiled_ccs +kms_ccs@pipe-F-crc-primary-basic-yf_tiled_ccs +kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_ccs +kms_ccs@pipe-F-crc-primary-rotation-180-yf_tiled_ccs +kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_rc_ccs +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_mc_ccs +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-random-ccs-data-y_tiled_ccs +kms_ccs@pipe-F-random-ccs-data-yf_tiled_ccs +kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_rc_ccs +kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_mc_ccs +kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_ccs +kms_ccs@pipe-F-missing-ccs-buffer-yf_tiled_ccs +kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_ccs +kms_ccs@pipe-F-ccs-on-another-bo-yf_tiled_ccs +kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-bad-aux-stride-y_tiled_ccs +kms_ccs@pipe-F-bad-aux-stride-yf_tiled_ccs +kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_ccs +kms_ccs@pipe-F-crc-sprite-planes-basic-yf_tiled_ccs +kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-bad-pixel-format-y_tiled_ccs +kms_ccs@pipe-G-bad-pixel-format-yf_tiled_ccs +kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_rc_ccs +kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_mc_ccs +kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-bad-rotation-90-y_tiled_ccs +kms_ccs@pipe-G-bad-rotation-90-yf_tiled_ccs +kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_rc_ccs +kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_mc_ccs +kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-crc-primary-basic-y_tiled_ccs +kms_ccs@pipe-G-crc-primary-basic-yf_tiled_ccs +kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_ccs +kms_ccs@pipe-G-crc-primary-rotation-180-yf_tiled_ccs +kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_rc_ccs +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_mc_ccs +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-random-ccs-data-y_tiled_ccs +kms_ccs@pipe-G-random-ccs-data-yf_tiled_ccs +kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_rc_ccs +kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_mc_ccs +kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_ccs +kms_ccs@pipe-G-missing-ccs-buffer-yf_tiled_ccs +kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_ccs +kms_ccs@pipe-G-ccs-on-another-bo-yf_tiled_ccs +kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-bad-aux-stride-y_tiled_ccs +kms_ccs@pipe-G-bad-aux-stride-yf_tiled_ccs +kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_ccs +kms_ccs@pipe-G-crc-sprite-planes-basic-yf_tiled_ccs +kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-bad-pixel-format-y_tiled_ccs +kms_ccs@pipe-H-bad-pixel-format-yf_tiled_ccs +kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_rc_ccs +kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_mc_ccs +kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-bad-rotation-90-y_tiled_ccs +kms_ccs@pipe-H-bad-rotation-90-yf_tiled_ccs +kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_rc_ccs +kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_mc_ccs +kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-crc-primary-basic-y_tiled_ccs +kms_ccs@pipe-H-crc-primary-basic-yf_tiled_ccs +kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_ccs +kms_ccs@pipe-H-crc-primary-rotation-180-yf_tiled_ccs +kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_rc_ccs +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_mc_ccs +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-random-ccs-data-y_tiled_ccs +kms_ccs@pipe-H-random-ccs-data-yf_tiled_ccs +kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_rc_ccs +kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_mc_ccs +kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_ccs +kms_ccs@pipe-H-missing-ccs-buffer-yf_tiled_ccs +kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_ccs +kms_ccs@pipe-H-ccs-on-another-bo-yf_tiled_ccs +kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-bad-aux-stride-y_tiled_ccs +kms_ccs@pipe-H-bad-aux-stride-yf_tiled_ccs +kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_rc_ccs_cc +kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_ccs +kms_ccs@pipe-H-crc-sprite-planes-basic-yf_tiled_ccs +kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs +kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc +kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc +kms_cdclk@plane-scaling +kms_cdclk@mode-transition +kms_cdclk@mode-transition-all-outputs +kms_color@degamma +kms_color@gamma +kms_color@legacy-gamma +kms_color@legacy-gamma-reset +kms_color@ctm-red-to-blue +kms_color@ctm-green-to-red +kms_color@ctm-blue-to-red +kms_color@ctm-max +kms_color@ctm-negative +kms_color@ctm-0-25 +kms_color@ctm-0-50 +kms_color@ctm-0-75 +kms_color@ctm-signed +kms_color@deep-color +kms_color@invalid-gamma-lut-sizes +kms_color@invalid-degamma-lut-sizes +kms_color@invalid-ctm-matrix-sizes +kms_concurrent@pipe-A +kms_concurrent@pipe-B +kms_concurrent@pipe-C +kms_concurrent@pipe-D +kms_concurrent@pipe-E +kms_concurrent@pipe-F +kms_concurrent@pipe-G +kms_concurrent@pipe-H +kms_content_protection@legacy +kms_content_protection@atomic +kms_content_protection@atomic-dpms +kms_content_protection@LIC +kms_content_protection@type1 +kms_content_protection@mei_interface +kms_content_protection@content_type_change +kms_content_protection@uevent +kms_content_protection@srm +kms_content_protection@dp-mst-type-0 +kms_content_protection@dp-mst-lic-type-0 +kms_content_protection@dp-mst-type-1 +kms_content_protection@dp-mst-lic-type-1 +kms_cursor_crc@cursor-size-change +kms_cursor_crc@cursor-alpha-opaque +kms_cursor_crc@cursor-alpha-transparent +kms_cursor_crc@cursor-dpms +kms_cursor_crc@cursor-suspend +kms_cursor_crc@cursor-onscreen-32x32 +kms_cursor_crc@cursor-offscreen-32x32 +kms_cursor_crc@cursor-sliding-32x32 +kms_cursor_crc@cursor-random-32x32 +kms_cursor_crc@cursor-rapid-movement-32x32 +kms_cursor_crc@cursor-onscreen-32x10 +kms_cursor_crc@cursor-offscreen-32x10 +kms_cursor_crc@cursor-sliding-32x10 +kms_cursor_crc@cursor-random-32x10 +kms_cursor_crc@cursor-rapid-movement-32x10 +kms_cursor_crc@cursor-onscreen-64x64 +kms_cursor_crc@cursor-offscreen-64x64 +kms_cursor_crc@cursor-sliding-64x64 +kms_cursor_crc@cursor-random-64x64 +kms_cursor_crc@cursor-rapid-movement-64x64 +kms_cursor_crc@cursor-onscreen-64x21 +kms_cursor_crc@cursor-offscreen-64x21 +kms_cursor_crc@cursor-sliding-64x21 +kms_cursor_crc@cursor-random-64x21 +kms_cursor_crc@cursor-rapid-movement-64x21 +kms_cursor_crc@cursor-onscreen-128x128 +kms_cursor_crc@cursor-offscreen-128x128 +kms_cursor_crc@cursor-sliding-128x128 +kms_cursor_crc@cursor-random-128x128 +kms_cursor_crc@cursor-rapid-movement-128x128 +kms_cursor_crc@cursor-onscreen-128x42 +kms_cursor_crc@cursor-offscreen-128x42 +kms_cursor_crc@cursor-sliding-128x42 +kms_cursor_crc@cursor-random-128x42 +kms_cursor_crc@cursor-rapid-movement-128x42 +kms_cursor_crc@cursor-onscreen-256x256 +kms_cursor_crc@cursor-offscreen-256x256 +kms_cursor_crc@cursor-sliding-256x256 +kms_cursor_crc@cursor-random-256x256 +kms_cursor_crc@cursor-rapid-movement-256x256 +kms_cursor_crc@cursor-onscreen-256x85 +kms_cursor_crc@cursor-offscreen-256x85 +kms_cursor_crc@cursor-sliding-256x85 +kms_cursor_crc@cursor-random-256x85 +kms_cursor_crc@cursor-rapid-movement-256x85 +kms_cursor_crc@cursor-onscreen-512x512 +kms_cursor_crc@cursor-offscreen-512x512 +kms_cursor_crc@cursor-sliding-512x512 +kms_cursor_crc@cursor-random-512x512 +kms_cursor_crc@cursor-rapid-movement-512x512 +kms_cursor_crc@cursor-onscreen-512x170 +kms_cursor_crc@cursor-offscreen-512x170 +kms_cursor_crc@cursor-sliding-512x170 +kms_cursor_crc@cursor-random-512x170 +kms_cursor_crc@cursor-rapid-movement-512x170 +kms_cursor_crc@cursor-onscreen-max-size +kms_cursor_crc@cursor-offscreen-max-size +kms_cursor_crc@cursor-sliding-max-size +kms_cursor_crc@cursor-random-max-size +kms_cursor_crc@cursor-rapid-movement-max-size +kms_cursor_legacy@single-bo +kms_cursor_legacy@single-move +kms_cursor_legacy@forked-bo +kms_cursor_legacy@forked-move +kms_cursor_legacy@torture-bo +kms_cursor_legacy@torture-move +kms_cursor_legacy@nonblocking-modeset-vs-cursor-atomic +kms_cursor_legacy@long-nonblocking-modeset-vs-cursor-atomic +kms_cursor_legacy@2x-flip-vs-cursor-legacy +kms_cursor_legacy@2x-flip-vs-cursor-atomic +kms_cursor_legacy@2x-long-flip-vs-cursor-legacy +kms_cursor_legacy@2x-long-flip-vs-cursor-atomic +kms_cursor_legacy@2x-nonblocking-modeset-vs-cursor-atomic +kms_cursor_legacy@2x-long-nonblocking-modeset-vs-cursor-atomic +kms_cursor_legacy@2x-cursor-vs-flip-legacy +kms_cursor_legacy@2x-long-cursor-vs-flip-legacy +kms_cursor_legacy@2x-cursor-vs-flip-atomic +kms_cursor_legacy@2x-long-cursor-vs-flip-atomic +kms_cursor_legacy@flip-vs-cursor-crc-legacy +kms_cursor_legacy@flip-vs-cursor-crc-atomic +kms_cursor_legacy@flip-vs-cursor-busy-crc-legacy +kms_cursor_legacy@flip-vs-cursor-busy-crc-atomic +kms_cursor_legacy@basic-flip-before-cursor-legacy +kms_cursor_legacy@basic-busy-flip-before-cursor-legacy +kms_cursor_legacy@basic-flip-after-cursor-legacy +kms_cursor_legacy@basic-flip-before-cursor-varying-size +kms_cursor_legacy@basic-busy-flip-before-cursor-varying-size +kms_cursor_legacy@basic-flip-after-cursor-varying-size +kms_cursor_legacy@short-flip-before-cursor-toggle +kms_cursor_legacy@short-busy-flip-before-cursor-toggle +kms_cursor_legacy@short-flip-after-cursor-toggle +kms_cursor_legacy@basic-flip-before-cursor-atomic +kms_cursor_legacy@basic-busy-flip-before-cursor-atomic +kms_cursor_legacy@basic-flip-after-cursor-atomic +kms_cursor_legacy@short-flip-before-cursor-atomic-transitions +kms_cursor_legacy@short-busy-flip-before-cursor-atomic-transitions +kms_cursor_legacy@short-flip-after-cursor-atomic-transitions +kms_cursor_legacy@short-flip-before-cursor-atomic-transitions-varying-size +kms_cursor_legacy@short-busy-flip-before-cursor-atomic-transitions-varying-size +kms_cursor_legacy@short-flip-after-cursor-atomic-transitions-varying-size +kms_cursor_legacy@cursor-vs-flip-legacy +kms_cursor_legacy@flip-vs-cursor-legacy +kms_cursor_legacy@cursorA-vs-flipA-legacy +kms_cursor_legacy@cursorA-vs-flipB-legacy +kms_cursor_legacy@cursorB-vs-flipA-legacy +kms_cursor_legacy@cursorB-vs-flipB-legacy +kms_cursor_legacy@cursor-vs-flip-varying-size +kms_cursor_legacy@flip-vs-cursor-varying-size +kms_cursor_legacy@cursorA-vs-flipA-varying-size +kms_cursor_legacy@cursorA-vs-flipB-varying-size +kms_cursor_legacy@cursorB-vs-flipA-varying-size +kms_cursor_legacy@cursorB-vs-flipB-varying-size +kms_cursor_legacy@cursor-vs-flip-toggle +kms_cursor_legacy@flip-vs-cursor-toggle +kms_cursor_legacy@cursorA-vs-flipA-toggle +kms_cursor_legacy@cursorA-vs-flipB-toggle +kms_cursor_legacy@cursorB-vs-flipA-toggle +kms_cursor_legacy@cursorB-vs-flipB-toggle +kms_cursor_legacy@cursor-vs-flip-atomic +kms_cursor_legacy@flip-vs-cursor-atomic +kms_cursor_legacy@cursorA-vs-flipA-atomic +kms_cursor_legacy@cursorA-vs-flipB-atomic +kms_cursor_legacy@cursorB-vs-flipA-atomic +kms_cursor_legacy@cursorB-vs-flipB-atomic +kms_cursor_legacy@cursor-vs-flip-atomic-transitions +kms_cursor_legacy@flip-vs-cursor-atomic-transitions +kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions +kms_cursor_legacy@cursorA-vs-flipB-atomic-transitions +kms_cursor_legacy@cursorB-vs-flipA-atomic-transitions +kms_cursor_legacy@cursorB-vs-flipB-atomic-transitions +kms_cursor_legacy@cursor-vs-flip-atomic-transitions-varying-size +kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size +kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions-varying-size +kms_cursor_legacy@cursorA-vs-flipB-atomic-transitions-varying-size +kms_cursor_legacy@cursorB-vs-flipA-atomic-transitions-varying-size +kms_cursor_legacy@cursorB-vs-flipB-atomic-transitions-varying-size +kms_dither@FB-8BPC-Vs-Panel-6BPC +kms_dither@FB-8BPC-Vs-Panel-8BPC +kms_dp_aux_dev +kms_tiled_display@basic-test-pattern +kms_tiled_display@basic-test-pattern-with-chamelium +kms_draw_crc@draw-method-mmap-cpu +kms_draw_crc@draw-method-mmap-gtt +kms_draw_crc@draw-method-mmap-wc +kms_draw_crc@draw-method-pwrite +kms_draw_crc@draw-method-blt +kms_draw_crc@draw-method-render +kms_draw_crc@fill-fb +kms_dsc@dsc-basic +kms_dsc@dsc-with-formats +kms_dsc@dsc-with-bpc +kms_dsc@dsc-with-bpc-formats +kms_dsc@dsc-with-output-formats +kms_fbcon_fbt@fbc +kms_fbcon_fbt@psr +kms_fbcon_fbt@fbc-suspend +kms_fbcon_fbt@psr-suspend +kms_fence_pin_leak +kms_flip@nonblocking-read +kms_flip@wf_vblank-ts-check +kms_flip@2x-wf_vblank-ts-check +kms_flip@blocking-wf_vblank +kms_flip@2x-blocking-wf_vblank +kms_flip@absolute-wf_vblank +kms_flip@2x-absolute-wf_vblank +kms_flip@blocking-absolute-wf_vblank +kms_flip@2x-blocking-absolute-wf_vblank +kms_flip@basic-plain-flip +kms_flip@2x-plain-flip +kms_flip@busy-flip +kms_flip@2x-busy-flip +kms_flip@flip-vs-fences +kms_flip@2x-flip-vs-fences +kms_flip@plain-flip-ts-check +kms_flip@2x-plain-flip-ts-check +kms_flip@plain-flip-fb-recreate +kms_flip@2x-plain-flip-fb-recreate +kms_flip@flip-vs-rmfb +kms_flip@2x-flip-vs-rmfb +kms_flip@basic-flip-vs-dpms +kms_flip@2x-flip-vs-dpms +kms_flip@flip-vs-panning +kms_flip@2x-flip-vs-panning +kms_flip@basic-flip-vs-modeset +kms_flip@2x-flip-vs-modeset +kms_flip@flip-vs-expired-vblank +kms_flip@2x-flip-vs-expired-vblank +kms_flip@flip-vs-absolute-wf_vblank +kms_flip@2x-flip-vs-absolute-wf_vblank +kms_flip@basic-flip-vs-wf_vblank +kms_flip@2x-flip-vs-wf_vblank +kms_flip@flip-vs-blocking-wf-vblank +kms_flip@2x-flip-vs-blocking-wf-vblank +kms_flip@flip-vs-modeset-vs-hang +kms_flip@2x-flip-vs-modeset-vs-hang +kms_flip@flip-vs-panning-vs-hang +kms_flip@2x-flip-vs-panning-vs-hang +kms_flip@flip-vs-dpms-off-vs-modeset +kms_flip@2x-flip-vs-dpms-off-vs-modeset +kms_flip@single-buffer-flip-vs-dpms-off-vs-modeset +kms_flip@2x-single-buffer-flip-vs-dpms-off-vs-modeset +kms_flip@dpms-off-confusion +kms_flip@nonexisting-fb +kms_flip@2x-nonexisting-fb +kms_flip@dpms-vs-vblank-race +kms_flip@2x-dpms-vs-vblank-race +kms_flip@modeset-vs-vblank-race +kms_flip@2x-modeset-vs-vblank-race +kms_flip@bo-too-big +kms_flip@flip-vs-suspend +kms_flip@2x-flip-vs-suspend +kms_flip@wf_vblank-ts-check-interruptible +kms_flip@2x-wf_vblank-ts-check-interruptible +kms_flip@absolute-wf_vblank-interruptible +kms_flip@2x-absolute-wf_vblank-interruptible +kms_flip@blocking-absolute-wf_vblank-interruptible +kms_flip@2x-blocking-absolute-wf_vblank-interruptible +kms_flip@plain-flip-interruptible +kms_flip@2x-plain-flip-interruptible +kms_flip@flip-vs-fences-interruptible +kms_flip@2x-flip-vs-fences-interruptible +kms_flip@plain-flip-ts-check-interruptible +kms_flip@2x-plain-flip-ts-check-interruptible +kms_flip@plain-flip-fb-recreate-interruptible +kms_flip@2x-plain-flip-fb-recreate-interruptible +kms_flip@flip-vs-rmfb-interruptible +kms_flip@2x-flip-vs-rmfb-interruptible +kms_flip@flip-vs-panning-interruptible +kms_flip@2x-flip-vs-panning-interruptible +kms_flip@flip-vs-expired-vblank-interruptible +kms_flip@2x-flip-vs-expired-vblank-interruptible +kms_flip@flip-vs-absolute-wf_vblank-interruptible +kms_flip@2x-flip-vs-absolute-wf_vblank-interruptible +kms_flip@flip-vs-wf_vblank-interruptible +kms_flip@2x-flip-vs-wf_vblank-interruptible +kms_flip@flip-vs-dpms-off-vs-modeset-interruptible +kms_flip@2x-flip-vs-dpms-off-vs-modeset-interruptible +kms_flip@single-buffer-flip-vs-dpms-off-vs-modeset-interruptible +kms_flip@2x-single-buffer-flip-vs-dpms-off-vs-modeset-interruptible +kms_flip@dpms-off-confusion-interruptible +kms_flip@nonexisting-fb-interruptible +kms_flip@2x-nonexisting-fb-interruptible +kms_flip@dpms-vs-vblank-race-interruptible +kms_flip@2x-dpms-vs-vblank-race-interruptible +kms_flip@modeset-vs-vblank-race-interruptible +kms_flip@2x-modeset-vs-vblank-race-interruptible +kms_flip@bo-too-big-interruptible +kms_flip@flip-vs-suspend-interruptible +kms_flip@2x-flip-vs-suspend-interruptible +kms_flip_event_leak@basic +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-yftile-to-64bpp-yftile-downscaling +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling +kms_flip_scaled_crc@flip-32bpp-4tile-to-64bpp-4tile-downscaling +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-yftile-to-32bpp-yftile-downscaling +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling +kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-downscaling +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-downscaling +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling +kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-downscaling +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs-downscaling +kms_flip_scaled_crc@flip-32bpp-4tile-to-32bpp-4tiledg2rcccs-downscaling +kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytileccs-downscaling +kms_flip_scaled_crc@flip-32bpp-yftile-to-32bpp-yftileccs-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling +kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tiledg2rcccs-downscaling +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-32bpp-yftile-to-64bpp-yftile-upscaling +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling +kms_flip_scaled_crc@flip-32bpp-4tile-to-64bpp-4tile-upscaling +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-yftile-to-32bpp-yftile-upscaling +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling +kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-upscaling +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling +kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-upscaling +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-upscaling +kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling +kms_flip_scaled_crc@flip-32bpp-4tile-to-32bpp-4tiledg2rcccs-upscaling +kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytileccs-upscaling +kms_flip_scaled_crc@flip-32bpp-yftile-to-32bpp-yftileccs-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling +kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tiledg2rcccs-upscaling +kms_force_connector_basic@force-load-detect +kms_force_connector_basic@force-connector-state +kms_force_connector_basic@force-edid +kms_force_connector_basic@prune-stale-modes +kms_frontbuffer_tracking@fbc-1p-rte +kms_frontbuffer_tracking@fbc-2p-rte +kms_frontbuffer_tracking@psr-1p-rte +kms_frontbuffer_tracking@psr-2p-rte +kms_frontbuffer_tracking@fbcpsr-1p-rte +kms_frontbuffer_tracking@fbcpsr-2p-rte +kms_frontbuffer_tracking@drrs-1p-rte +kms_frontbuffer_tracking@drrs-2p-rte +kms_frontbuffer_tracking@fbcdrrs-1p-rte +kms_frontbuffer_tracking@fbcdrrs-2p-rte +kms_frontbuffer_tracking@psrdrrs-1p-rte +kms_frontbuffer_tracking@psrdrrs-2p-rte +kms_frontbuffer_tracking@fbcpsrdrrs-1p-rte +kms_frontbuffer_tracking@fbcpsrdrrs-2p-rte +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-render +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-blt +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-render +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-blt +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-render +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-blt +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-render +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-blt +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-render +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-render +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-render +kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@psr-1p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@psr-1p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@psr-1p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@psr-2p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@psr-2p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@psr-2p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-msflip-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-plflip-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-msflip-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-plflip-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-msflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-plflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-msflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-plflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-pgflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-msflip-blt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-plflip-blt +kms_frontbuffer_tracking@fbc-1p-indfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbc-1p-shrfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-indfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbc-2p-shrfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-1p-indfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-1p-shrfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-2p-shrfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-1p-indfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-1p-shrfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-indfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-2p-shrfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-indfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-1p-shrfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-indfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-2p-shrfb-fliptrack-mmap-gtt +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-move +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-move +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-onoff +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-move +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-onoff +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-move +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-onoff +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-move +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-onoff +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-move +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-onoff +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-move +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-onoff +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-move +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-move +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-onoff +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-move +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-onoff +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-fullscreen +kms_frontbuffer_tracking@fbc-1p-pri-indfb-multidraw +kms_frontbuffer_tracking@fbc-2p-pri-indfb-multidraw +kms_frontbuffer_tracking@psr-1p-pri-indfb-multidraw +kms_frontbuffer_tracking@psr-2p-pri-indfb-multidraw +kms_frontbuffer_tracking@fbcpsr-1p-pri-indfb-multidraw +kms_frontbuffer_tracking@fbcpsr-2p-pri-indfb-multidraw +kms_frontbuffer_tracking@drrs-1p-pri-indfb-multidraw +kms_frontbuffer_tracking@drrs-2p-pri-indfb-multidraw +kms_frontbuffer_tracking@fbcdrrs-1p-pri-indfb-multidraw +kms_frontbuffer_tracking@fbcdrrs-2p-pri-indfb-multidraw +kms_frontbuffer_tracking@psrdrrs-1p-pri-indfb-multidraw +kms_frontbuffer_tracking@psrdrrs-2p-pri-indfb-multidraw +kms_frontbuffer_tracking@fbcpsrdrrs-1p-pri-indfb-multidraw +kms_frontbuffer_tracking@fbcpsrdrrs-2p-pri-indfb-multidraw +kms_frontbuffer_tracking@fbc-farfromfence-mmap-gtt +kms_frontbuffer_tracking@psr-farfromfence-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-farfromfence-mmap-gtt +kms_frontbuffer_tracking@drrs-farfromfence-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-farfromfence-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-farfromfence-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-farfromfence-mmap-gtt +kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-cpu +kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-gtt +kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-wc +kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-wc +kms_frontbuffer_tracking@fbc-rgb565-draw-pwrite +kms_frontbuffer_tracking@fbc-rgb101010-draw-pwrite +kms_frontbuffer_tracking@fbc-rgb565-draw-blt +kms_frontbuffer_tracking@fbc-rgb101010-draw-blt +kms_frontbuffer_tracking@fbc-rgb565-draw-render +kms_frontbuffer_tracking@fbc-rgb101010-draw-render +kms_frontbuffer_tracking@psr-rgb565-draw-mmap-cpu +kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-cpu +kms_frontbuffer_tracking@psr-rgb565-draw-mmap-gtt +kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-gtt +kms_frontbuffer_tracking@psr-rgb565-draw-mmap-wc +kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-wc +kms_frontbuffer_tracking@psr-rgb565-draw-pwrite +kms_frontbuffer_tracking@psr-rgb101010-draw-pwrite +kms_frontbuffer_tracking@psr-rgb565-draw-blt +kms_frontbuffer_tracking@psr-rgb101010-draw-blt +kms_frontbuffer_tracking@psr-rgb565-draw-render +kms_frontbuffer_tracking@psr-rgb101010-draw-render +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-pwrite +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-blt +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-blt +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-render +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-render +kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-cpu +kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-gtt +kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-wc +kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-wc +kms_frontbuffer_tracking@drrs-rgb565-draw-pwrite +kms_frontbuffer_tracking@drrs-rgb101010-draw-pwrite +kms_frontbuffer_tracking@drrs-rgb565-draw-blt +kms_frontbuffer_tracking@drrs-rgb101010-draw-blt +kms_frontbuffer_tracking@drrs-rgb565-draw-render +kms_frontbuffer_tracking@drrs-rgb101010-draw-render +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-cpu +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-gtt +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-wc +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-pwrite +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-blt +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-blt +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-render +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-render +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-cpu +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-gtt +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-wc +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-pwrite +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-blt +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-blt +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-render +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-cpu +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-gtt +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-wc +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-pwrite +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-blt +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-render +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-render +kms_frontbuffer_tracking@fbc-indfb-scaledprimary +kms_frontbuffer_tracking@fbc-shrfb-scaledprimary +kms_frontbuffer_tracking@psr-indfb-scaledprimary +kms_frontbuffer_tracking@psr-shrfb-scaledprimary +kms_frontbuffer_tracking@fbcpsr-indfb-scaledprimary +kms_frontbuffer_tracking@fbcpsr-shrfb-scaledprimary +kms_frontbuffer_tracking@drrs-indfb-scaledprimary +kms_frontbuffer_tracking@drrs-shrfb-scaledprimary +kms_frontbuffer_tracking@fbcdrrs-indfb-scaledprimary +kms_frontbuffer_tracking@fbcdrrs-shrfb-scaledprimary +kms_frontbuffer_tracking@psrdrrs-indfb-scaledprimary +kms_frontbuffer_tracking@psrdrrs-shrfb-scaledprimary +kms_frontbuffer_tracking@fbcpsrdrrs-indfb-scaledprimary +kms_frontbuffer_tracking@fbcpsrdrrs-shrfb-scaledprimary +kms_frontbuffer_tracking@fbc-modesetfrombusy +kms_frontbuffer_tracking@fbc-badstride +kms_frontbuffer_tracking@fbc-stridechange +kms_frontbuffer_tracking@fbc-tiling-linear +kms_frontbuffer_tracking@fbc-tiling-y +kms_frontbuffer_tracking@fbc-tiling-4 +kms_frontbuffer_tracking@fbc-suspend +kms_frontbuffer_tracking@psr-modesetfrombusy +kms_frontbuffer_tracking@psr-slowdraw +kms_frontbuffer_tracking@psr-suspend +kms_frontbuffer_tracking@fbcpsr-modesetfrombusy +kms_frontbuffer_tracking@fbcpsr-badstride +kms_frontbuffer_tracking@fbcpsr-stridechange +kms_frontbuffer_tracking@fbcpsr-tiling-linear +kms_frontbuffer_tracking@fbcpsr-tiling-y +kms_frontbuffer_tracking@fbcpsr-tiling-4 +kms_frontbuffer_tracking@fbcpsr-slowdraw +kms_frontbuffer_tracking@fbcpsr-suspend +kms_frontbuffer_tracking@drrs-modesetfrombusy +kms_frontbuffer_tracking@drrs-slowdraw +kms_frontbuffer_tracking@drrs-suspend +kms_frontbuffer_tracking@fbcdrrs-modesetfrombusy +kms_frontbuffer_tracking@fbcdrrs-badstride +kms_frontbuffer_tracking@fbcdrrs-stridechange +kms_frontbuffer_tracking@fbcdrrs-tiling-linear +kms_frontbuffer_tracking@fbcdrrs-tiling-y +kms_frontbuffer_tracking@fbcdrrs-tiling-4 +kms_frontbuffer_tracking@fbcdrrs-slowdraw +kms_frontbuffer_tracking@fbcdrrs-suspend +kms_frontbuffer_tracking@psrdrrs-modesetfrombusy +kms_frontbuffer_tracking@psrdrrs-slowdraw +kms_frontbuffer_tracking@psrdrrs-suspend +kms_frontbuffer_tracking@fbcpsrdrrs-modesetfrombusy +kms_frontbuffer_tracking@fbcpsrdrrs-badstride +kms_frontbuffer_tracking@fbcpsrdrrs-stridechange +kms_frontbuffer_tracking@fbcpsrdrrs-tiling-linear +kms_frontbuffer_tracking@fbcpsrdrrs-tiling-y +kms_frontbuffer_tracking@fbcpsrdrrs-tiling-4 +kms_frontbuffer_tracking@fbcpsrdrrs-slowdraw +kms_frontbuffer_tracking@fbcpsrdrrs-suspend +kms_frontbuffer_tracking@basic +kms_getfb@getfb-handle-zero +kms_getfb@getfb-handle-valid +kms_getfb@getfb-handle-closed +kms_getfb@getfb-handle-not-fb +kms_getfb@getfb-addfb-different-handles +kms_getfb@getfb-repeated-different-handles +kms_getfb@getfb-reject-ccs +kms_getfb@getfb2-handle-zero +kms_getfb@getfb2-handle-closed +kms_getfb@getfb2-handle-not-fb +kms_getfb@getfb2-accept-ccs +kms_getfb@getfb2-into-addfb2 +kms_getfb@getfb-handle-protection +kms_getfb@getfb2-handle-protection +kms_hdmi_inject@inject-4k +kms_hdmi_inject@inject-audio +kms_hdr@bpc-switch +kms_hdr@bpc-switch-dpms +kms_hdr@bpc-switch-suspend +kms_hdr@static-toggle +kms_hdr@static-toggle-dpms +kms_hdr@static-toggle-suspend +kms_hdr@static-swap +kms_hdr@invalid-metadata-sizes +kms_hdr@invalid-hdr +kms_invalid_mode@clock-too-high +kms_invalid_mode@zero-clock +kms_invalid_mode@int-max-clock +kms_invalid_mode@uint-max-clock +kms_invalid_mode@zero-hdisplay +kms_invalid_mode@zero-vdisplay +kms_invalid_mode@bad-hsync-start +kms_invalid_mode@bad-vsync-start +kms_invalid_mode@bad-hsync-end +kms_invalid_mode@bad-vsync-end +kms_invalid_mode@bad-htotal +kms_invalid_mode@bad-vtotal +kms_legacy_colorkey@basic +kms_legacy_colorkey@invalid-plane +kms_multipipe_modeset@basic-max-pipe-crc-check +kms_panel_fitting@legacy +kms_panel_fitting@atomic-fastset +kms_pipe_b_c_ivb@pipe-B-dpms-off-modeset-pipe-C +kms_pipe_b_c_ivb@pipe-B-double-modeset-then-modeset-pipe-C +kms_pipe_b_c_ivb@disable-pipe-B-enable-pipe-C +kms_pipe_b_c_ivb@from-pipe-C-to-B-with-3-lanes +kms_pipe_b_c_ivb@enable-pipe-C-while-B-has-3-lanes +kms_pipe_crc_basic@bad-source +kms_pipe_crc_basic@read-crc +kms_pipe_crc_basic@read-crc-frame-sequence +kms_pipe_crc_basic@nonblocking-crc +kms_pipe_crc_basic@nonblocking-crc-frame-sequence +kms_pipe_crc_basic@suspend-read-crc +kms_pipe_crc_basic@hang-read-crc +kms_pipe_crc_basic@disable-crc-after-crtc +kms_pipe_crc_basic@compare-crc-sanitycheck-xr24 +kms_pipe_crc_basic@compare-crc-sanitycheck-nv12 +kms_plane@pixel-format +kms_plane@pixel-format-source-clamping +kms_plane@plane-position-covered +kms_plane@plane-position-hole +kms_plane@plane-position-hole-dpms +kms_plane@plane-panning-top-left +kms_plane@plane-panning-bottom-right +kms_plane@plane-panning-bottom-right-suspend +kms_plane@invalid-pixel-format-settings +kms_plane_alpha_blend@alpha-basic +kms_plane_alpha_blend@alpha-7efc +kms_plane_alpha_blend@coverage-7efc +kms_plane_alpha_blend@coverage-vs-premult-vs-constant +kms_plane_alpha_blend@alpha-transparent-fb +kms_plane_alpha_blend@alpha-opaque-fb +kms_plane_alpha_blend@constant-alpha-min +kms_plane_alpha_blend@constant-alpha-mid +kms_plane_alpha_blend@constant-alpha-max +kms_plane_cursor@primary +kms_plane_cursor@overlay +kms_plane_cursor@viewport +kms_plane_lowres@tiling-none +kms_plane_lowres@tiling-x +kms_plane_lowres@tiling-y +kms_plane_lowres@tiling-yf +kms_plane_lowres@tiling-4 +kms_plane_multiple@tiling-none +kms_plane_multiple@tiling-x +kms_plane_multiple@tiling-y +kms_plane_multiple@tiling-yf +kms_plane_multiple@tiling-4 +kms_plane_scaling@plane-upscale-with-pixel-format-20x20 +kms_plane_scaling@plane-upscale-with-pixel-format-factor-0-25 +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-25 +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-5 +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-75 +kms_plane_scaling@plane-scaler-with-pixel-format-unity-scaling +kms_plane_scaling@plane-upscale-with-rotation-20x20 +kms_plane_scaling@plane-upscale-with-rotation-factor-0-25 +kms_plane_scaling@plane-downscale-with-rotation-factor-0-25 +kms_plane_scaling@plane-downscale-with-rotation-factor-0-5 +kms_plane_scaling@plane-downscale-with-rotation-factor-0-75 +kms_plane_scaling@plane-scaler-with-rotation-unity-scaling +kms_plane_scaling@plane-upscale-with-modifiers-20x20 +kms_plane_scaling@plane-upscale-with-modifiers-factor-0-25 +kms_plane_scaling@plane-downscale-with-modifiers-factor-0-25 +kms_plane_scaling@plane-downscale-with-modifiers-factor-0-5 +kms_plane_scaling@plane-downscale-with-modifiers-factor-0-75 +kms_plane_scaling@plane-scaler-with-modifiers-unity-scaling +kms_plane_scaling@plane-scaler-with-clipping-clamping-pixel-formats +kms_plane_scaling@plane-scaler-with-clipping-clamping-rotation +kms_plane_scaling@plane-scaler-with-clipping-clamping-modifiers +kms_plane_scaling@planes-upscale-20x20 +kms_plane_scaling@planes-upscale-factor-0-25 +kms_plane_scaling@planes-scaler-unity-scaling +kms_plane_scaling@planes-downscale-factor-0-25 +kms_plane_scaling@planes-downscale-factor-0-5 +kms_plane_scaling@planes-downscale-factor-0-75 +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25 +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5 +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75 +kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-25 +kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-5 +kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-75 +kms_plane_scaling@planes-unity-scaling-downscale-factor-0-25 +kms_plane_scaling@planes-unity-scaling-downscale-factor-0-5 +kms_plane_scaling@planes-unity-scaling-downscale-factor-0-75 +kms_plane_scaling@planes-downscale-factor-0-25-upscale-20x20 +kms_plane_scaling@planes-downscale-factor-0-25-upscale-factor-0-25 +kms_plane_scaling@planes-downscale-factor-0-25-unity-scaling +kms_plane_scaling@planes-downscale-factor-0-5-upscale-20x20 +kms_plane_scaling@planes-downscale-factor-0-5-upscale-factor-0-25 +kms_plane_scaling@planes-downscale-factor-0-5-unity-scaling +kms_plane_scaling@planes-downscale-factor-0-75-upscale-20x20 +kms_plane_scaling@planes-downscale-factor-0-75-upscale-factor-0-25 +kms_plane_scaling@planes-downscale-factor-0-75-unity-scaling +kms_plane_scaling@intel-max-src-size +kms_plane_scaling@invalid-num-scalers +kms_plane_scaling@invalid-parameters +kms_plane_scaling@2x-scaler-multi-pipe +kms_prime@basic-crc-hybrid +kms_prime@basic-modeset-hybrid +kms_prime@D3hot +kms_prime@basic-crc-vgem +kms_prop_blob@basic +kms_prop_blob@blob-prop-core +kms_prop_blob@blob-prop-validate +kms_prop_blob@blob-prop-lifetime +kms_prop_blob@blob-multiple +kms_prop_blob@invalid-get-prop-any +kms_prop_blob@invalid-get-prop +kms_prop_blob@invalid-set-prop-any +kms_prop_blob@invalid-set-prop +kms_properties@plane-properties-legacy +kms_properties@plane-properties-atomic +kms_properties@crtc-properties-legacy +kms_properties@crtc-properties-atomic +kms_properties@connector-properties-legacy +kms_properties@connector-properties-atomic +kms_properties@invalid-properties-legacy +kms_properties@invalid-properties-atomic +kms_properties@get_properties-sanity-atomic +kms_properties@get_properties-sanity-non-atomic +kms_psr@basic +kms_psr@no_drrs +kms_psr@primary_page_flip +kms_psr@primary_mmap_gtt +kms_psr@primary_mmap_cpu +kms_psr@primary_blt +kms_psr@primary_render +kms_psr@sprite_mmap_gtt +kms_psr@cursor_mmap_gtt +kms_psr@sprite_mmap_cpu +kms_psr@cursor_mmap_cpu +kms_psr@sprite_blt +kms_psr@cursor_blt +kms_psr@sprite_render +kms_psr@cursor_render +kms_psr@sprite_plane_move +kms_psr@cursor_plane_move +kms_psr@sprite_plane_onoff +kms_psr@cursor_plane_onoff +kms_psr@dpms +kms_psr@suspend +kms_psr@psr2_basic +kms_psr@psr2_no_drrs +kms_psr@psr2_primary_page_flip +kms_psr@psr2_primary_mmap_gtt +kms_psr@psr2_primary_mmap_cpu +kms_psr@psr2_primary_blt +kms_psr@psr2_primary_render +kms_psr@psr2_sprite_mmap_gtt +kms_psr@psr2_cursor_mmap_gtt +kms_psr@psr2_sprite_mmap_cpu +kms_psr@psr2_cursor_mmap_cpu +kms_psr@psr2_sprite_blt +kms_psr@psr2_cursor_blt +kms_psr@psr2_sprite_render +kms_psr@psr2_cursor_render +kms_psr@psr2_sprite_plane_move +kms_psr@psr2_cursor_plane_move +kms_psr@psr2_sprite_plane_onoff +kms_psr@psr2_cursor_plane_onoff +kms_psr@psr2_dpms +kms_psr@psr2_suspend +kms_psr2_sf@primary-plane-update-sf-dmg-area +kms_psr2_sf@primary-plane-update-sf-dmg-area-big-fb +kms_psr2_sf@overlay-plane-update-sf-dmg-area +kms_psr2_sf@cursor-plane-update-sf +kms_psr2_sf@cursor-plane-move-continuous-sf +kms_psr2_sf@cursor-plane-move-continuous-exceed-sf +kms_psr2_sf@cursor-plane-move-continuous-exceed-fully-sf +kms_psr2_sf@plane-move-sf-dmg-area +kms_psr2_sf@overlay-plane-move-continuous-sf +kms_psr2_sf@overlay-plane-move-continuous-exceed-sf +kms_psr2_sf@overlay-plane-move-continuous-exceed-fully-sf +kms_psr2_sf@overlay-primary-update-sf-dmg-area +kms_psr2_sf@overlay-plane-update-continuous-sf +kms_psr2_su@page_flip-XRGB8888 +kms_psr2_su@page_flip-NV12 +kms_psr2_su@page_flip-P010 +kms_psr2_su@frontbuffer-XRGB8888 +kms_pwrite_crc +kms_rmfb@rmfb-ioctl +kms_rmfb@close-fd +kms_rotation_crc@primary-rotation-90 +kms_rotation_crc@primary-rotation-180 +kms_rotation_crc@primary-rotation-270 +kms_rotation_crc@sprite-rotation-90 +kms_rotation_crc@sprite-rotation-180 +kms_rotation_crc@sprite-rotation-270 +kms_rotation_crc@cursor-rotation-180 +kms_rotation_crc@sprite-rotation-90-pos-100-0 +kms_rotation_crc@bad-pixel-format +kms_rotation_crc@bad-tiling +kms_rotation_crc@primary-x-tiled-reflect-x-0 +kms_rotation_crc@primary-x-tiled-reflect-x-180 +kms_rotation_crc@primary-y-tiled-reflect-x-0 +kms_rotation_crc@primary-y-tiled-reflect-x-90 +kms_rotation_crc@primary-y-tiled-reflect-x-180 +kms_rotation_crc@primary-y-tiled-reflect-x-270 +kms_rotation_crc@primary-yf-tiled-reflect-x-0 +kms_rotation_crc@primary-yf-tiled-reflect-x-90 +kms_rotation_crc@primary-yf-tiled-reflect-x-180 +kms_rotation_crc@primary-yf-tiled-reflect-x-270 +kms_rotation_crc@primary-4-tiled-reflect-x-0 +kms_rotation_crc@primary-4-tiled-reflect-x-180 +kms_rotation_crc@multiplane-rotation +kms_rotation_crc@multiplane-rotation-cropping-top +kms_rotation_crc@multiplane-rotation-cropping-bottom +kms_rotation_crc@exhaust-fences +kms_scaling_modes@scaling-mode-full +kms_scaling_modes@scaling-mode-center +kms_scaling_modes@scaling-mode-full-aspect +kms_scaling_modes@scaling-mode-none +kms_selftest@drm_cmdline +kms_selftest@drm_damage +kms_selftest@drm_dp_mst +kms_selftest@drm_format_helper +kms_selftest@drm_format +kms_selftest@framebuffer +kms_selftest@drm_plane +kms_setmode@basic +kms_setmode@basic-clone-single-crtc +kms_setmode@invalid-clone-single-crtc +kms_setmode@invalid-clone-exclusive-crtc +kms_setmode@clone-exclusive-crtc +kms_setmode@invalid-clone-single-crtc-stealing +kms_sysfs_edid_timing +kms_tv_load_detect@load-detect +kms_universal_plane@universal-plane-pipe-A-functional +kms_universal_plane@universal-plane-pipe-A-sanity +kms_universal_plane@disable-primary-vs-flip-pipe-A +kms_universal_plane@cursor-fb-leak-pipe-A +kms_universal_plane@universal-plane-pageflip-windowed-pipe-A +kms_universal_plane@universal-plane-pipe-B-functional +kms_universal_plane@universal-plane-pipe-B-sanity +kms_universal_plane@disable-primary-vs-flip-pipe-B +kms_universal_plane@cursor-fb-leak-pipe-B +kms_universal_plane@universal-plane-pageflip-windowed-pipe-B +kms_universal_plane@universal-plane-pipe-C-functional +kms_universal_plane@universal-plane-pipe-C-sanity +kms_universal_plane@disable-primary-vs-flip-pipe-C +kms_universal_plane@cursor-fb-leak-pipe-C +kms_universal_plane@universal-plane-pageflip-windowed-pipe-C +kms_universal_plane@universal-plane-pipe-D-functional +kms_universal_plane@universal-plane-pipe-D-sanity +kms_universal_plane@disable-primary-vs-flip-pipe-D +kms_universal_plane@cursor-fb-leak-pipe-D +kms_universal_plane@universal-plane-pageflip-windowed-pipe-D +kms_universal_plane@universal-plane-pipe-E-functional +kms_universal_plane@universal-plane-pipe-E-sanity +kms_universal_plane@disable-primary-vs-flip-pipe-E +kms_universal_plane@cursor-fb-leak-pipe-E +kms_universal_plane@universal-plane-pageflip-windowed-pipe-E +kms_universal_plane@universal-plane-pipe-F-functional +kms_universal_plane@universal-plane-pipe-F-sanity +kms_universal_plane@disable-primary-vs-flip-pipe-F +kms_universal_plane@cursor-fb-leak-pipe-F +kms_universal_plane@universal-plane-pageflip-windowed-pipe-F +kms_universal_plane@universal-plane-pipe-G-functional +kms_universal_plane@universal-plane-pipe-G-sanity +kms_universal_plane@disable-primary-vs-flip-pipe-G +kms_universal_plane@cursor-fb-leak-pipe-G +kms_universal_plane@universal-plane-pageflip-windowed-pipe-G +kms_universal_plane@universal-plane-pipe-H-functional +kms_universal_plane@universal-plane-pipe-H-sanity +kms_universal_plane@disable-primary-vs-flip-pipe-H +kms_universal_plane@cursor-fb-leak-pipe-H +kms_universal_plane@universal-plane-pageflip-windowed-pipe-H +kms_vblank@invalid +kms_vblank@crtc-id +kms_vblank@pipe-A-accuracy-idle +kms_vblank@pipe-A-query-idle +kms_vblank@pipe-A-query-idle-hang +kms_vblank@pipe-A-query-forked +kms_vblank@pipe-A-query-forked-hang +kms_vblank@pipe-A-query-busy +kms_vblank@pipe-A-query-busy-hang +kms_vblank@pipe-A-query-forked-busy +kms_vblank@pipe-A-query-forked-busy-hang +kms_vblank@pipe-A-wait-idle +kms_vblank@pipe-A-wait-idle-hang +kms_vblank@pipe-A-wait-forked +kms_vblank@pipe-A-wait-forked-hang +kms_vblank@pipe-A-wait-busy +kms_vblank@pipe-A-wait-busy-hang +kms_vblank@pipe-A-wait-forked-busy +kms_vblank@pipe-A-wait-forked-busy-hang +kms_vblank@pipe-A-ts-continuation-idle +kms_vblank@pipe-A-ts-continuation-idle-hang +kms_vblank@pipe-A-ts-continuation-dpms-rpm +kms_vblank@pipe-A-ts-continuation-dpms-suspend +kms_vblank@pipe-A-ts-continuation-suspend +kms_vblank@pipe-A-ts-continuation-modeset +kms_vblank@pipe-A-ts-continuation-modeset-hang +kms_vblank@pipe-A-ts-continuation-modeset-rpm +kms_vblank@pipe-B-accuracy-idle +kms_vblank@pipe-B-query-idle +kms_vblank@pipe-B-query-idle-hang +kms_vblank@pipe-B-query-forked +kms_vblank@pipe-B-query-forked-hang +kms_vblank@pipe-B-query-busy +kms_vblank@pipe-B-query-busy-hang +kms_vblank@pipe-B-query-forked-busy +kms_vblank@pipe-B-query-forked-busy-hang +kms_vblank@pipe-B-wait-idle +kms_vblank@pipe-B-wait-idle-hang +kms_vblank@pipe-B-wait-forked +kms_vblank@pipe-B-wait-forked-hang +kms_vblank@pipe-B-wait-busy +kms_vblank@pipe-B-wait-busy-hang +kms_vblank@pipe-B-wait-forked-busy +kms_vblank@pipe-B-wait-forked-busy-hang +kms_vblank@pipe-B-ts-continuation-idle +kms_vblank@pipe-B-ts-continuation-idle-hang +kms_vblank@pipe-B-ts-continuation-dpms-rpm +kms_vblank@pipe-B-ts-continuation-dpms-suspend +kms_vblank@pipe-B-ts-continuation-suspend +kms_vblank@pipe-B-ts-continuation-modeset +kms_vblank@pipe-B-ts-continuation-modeset-hang +kms_vblank@pipe-B-ts-continuation-modeset-rpm +kms_vblank@pipe-C-accuracy-idle +kms_vblank@pipe-C-query-idle +kms_vblank@pipe-C-query-idle-hang +kms_vblank@pipe-C-query-forked +kms_vblank@pipe-C-query-forked-hang +kms_vblank@pipe-C-query-busy +kms_vblank@pipe-C-query-busy-hang +kms_vblank@pipe-C-query-forked-busy +kms_vblank@pipe-C-query-forked-busy-hang +kms_vblank@pipe-C-wait-idle +kms_vblank@pipe-C-wait-idle-hang +kms_vblank@pipe-C-wait-forked +kms_vblank@pipe-C-wait-forked-hang +kms_vblank@pipe-C-wait-busy +kms_vblank@pipe-C-wait-busy-hang +kms_vblank@pipe-C-wait-forked-busy +kms_vblank@pipe-C-wait-forked-busy-hang +kms_vblank@pipe-C-ts-continuation-idle +kms_vblank@pipe-C-ts-continuation-idle-hang +kms_vblank@pipe-C-ts-continuation-dpms-rpm +kms_vblank@pipe-C-ts-continuation-dpms-suspend +kms_vblank@pipe-C-ts-continuation-suspend +kms_vblank@pipe-C-ts-continuation-modeset +kms_vblank@pipe-C-ts-continuation-modeset-hang +kms_vblank@pipe-C-ts-continuation-modeset-rpm +kms_vblank@pipe-D-accuracy-idle +kms_vblank@pipe-D-query-idle +kms_vblank@pipe-D-query-idle-hang +kms_vblank@pipe-D-query-forked +kms_vblank@pipe-D-query-forked-hang +kms_vblank@pipe-D-query-busy +kms_vblank@pipe-D-query-busy-hang +kms_vblank@pipe-D-query-forked-busy +kms_vblank@pipe-D-query-forked-busy-hang +kms_vblank@pipe-D-wait-idle +kms_vblank@pipe-D-wait-idle-hang +kms_vblank@pipe-D-wait-forked +kms_vblank@pipe-D-wait-forked-hang +kms_vblank@pipe-D-wait-busy +kms_vblank@pipe-D-wait-busy-hang +kms_vblank@pipe-D-wait-forked-busy +kms_vblank@pipe-D-wait-forked-busy-hang +kms_vblank@pipe-D-ts-continuation-idle +kms_vblank@pipe-D-ts-continuation-idle-hang +kms_vblank@pipe-D-ts-continuation-dpms-rpm +kms_vblank@pipe-D-ts-continuation-dpms-suspend +kms_vblank@pipe-D-ts-continuation-suspend +kms_vblank@pipe-D-ts-continuation-modeset +kms_vblank@pipe-D-ts-continuation-modeset-hang +kms_vblank@pipe-D-ts-continuation-modeset-rpm +kms_vblank@pipe-E-accuracy-idle +kms_vblank@pipe-E-query-idle +kms_vblank@pipe-E-query-idle-hang +kms_vblank@pipe-E-query-forked +kms_vblank@pipe-E-query-forked-hang +kms_vblank@pipe-E-query-busy +kms_vblank@pipe-E-query-busy-hang +kms_vblank@pipe-E-query-forked-busy +kms_vblank@pipe-E-query-forked-busy-hang +kms_vblank@pipe-E-wait-idle +kms_vblank@pipe-E-wait-idle-hang +kms_vblank@pipe-E-wait-forked +kms_vblank@pipe-E-wait-forked-hang +kms_vblank@pipe-E-wait-busy +kms_vblank@pipe-E-wait-busy-hang +kms_vblank@pipe-E-wait-forked-busy +kms_vblank@pipe-E-wait-forked-busy-hang +kms_vblank@pipe-E-ts-continuation-idle +kms_vblank@pipe-E-ts-continuation-idle-hang +kms_vblank@pipe-E-ts-continuation-dpms-rpm +kms_vblank@pipe-E-ts-continuation-dpms-suspend +kms_vblank@pipe-E-ts-continuation-suspend +kms_vblank@pipe-E-ts-continuation-modeset +kms_vblank@pipe-E-ts-continuation-modeset-hang +kms_vblank@pipe-E-ts-continuation-modeset-rpm +kms_vblank@pipe-F-accuracy-idle +kms_vblank@pipe-F-query-idle +kms_vblank@pipe-F-query-idle-hang +kms_vblank@pipe-F-query-forked +kms_vblank@pipe-F-query-forked-hang +kms_vblank@pipe-F-query-busy +kms_vblank@pipe-F-query-busy-hang +kms_vblank@pipe-F-query-forked-busy +kms_vblank@pipe-F-query-forked-busy-hang +kms_vblank@pipe-F-wait-idle +kms_vblank@pipe-F-wait-idle-hang +kms_vblank@pipe-F-wait-forked +kms_vblank@pipe-F-wait-forked-hang +kms_vblank@pipe-F-wait-busy +kms_vblank@pipe-F-wait-busy-hang +kms_vblank@pipe-F-wait-forked-busy +kms_vblank@pipe-F-wait-forked-busy-hang +kms_vblank@pipe-F-ts-continuation-idle +kms_vblank@pipe-F-ts-continuation-idle-hang +kms_vblank@pipe-F-ts-continuation-dpms-rpm +kms_vblank@pipe-F-ts-continuation-dpms-suspend +kms_vblank@pipe-F-ts-continuation-suspend +kms_vblank@pipe-F-ts-continuation-modeset +kms_vblank@pipe-F-ts-continuation-modeset-hang +kms_vblank@pipe-F-ts-continuation-modeset-rpm +kms_vblank@pipe-G-accuracy-idle +kms_vblank@pipe-G-query-idle +kms_vblank@pipe-G-query-idle-hang +kms_vblank@pipe-G-query-forked +kms_vblank@pipe-G-query-forked-hang +kms_vblank@pipe-G-query-busy +kms_vblank@pipe-G-query-busy-hang +kms_vblank@pipe-G-query-forked-busy +kms_vblank@pipe-G-query-forked-busy-hang +kms_vblank@pipe-G-wait-idle +kms_vblank@pipe-G-wait-idle-hang +kms_vblank@pipe-G-wait-forked +kms_vblank@pipe-G-wait-forked-hang +kms_vblank@pipe-G-wait-busy +kms_vblank@pipe-G-wait-busy-hang +kms_vblank@pipe-G-wait-forked-busy +kms_vblank@pipe-G-wait-forked-busy-hang +kms_vblank@pipe-G-ts-continuation-idle +kms_vblank@pipe-G-ts-continuation-idle-hang +kms_vblank@pipe-G-ts-continuation-dpms-rpm +kms_vblank@pipe-G-ts-continuation-dpms-suspend +kms_vblank@pipe-G-ts-continuation-suspend +kms_vblank@pipe-G-ts-continuation-modeset +kms_vblank@pipe-G-ts-continuation-modeset-hang +kms_vblank@pipe-G-ts-continuation-modeset-rpm +kms_vblank@pipe-H-accuracy-idle +kms_vblank@pipe-H-query-idle +kms_vblank@pipe-H-query-idle-hang +kms_vblank@pipe-H-query-forked +kms_vblank@pipe-H-query-forked-hang +kms_vblank@pipe-H-query-busy +kms_vblank@pipe-H-query-busy-hang +kms_vblank@pipe-H-query-forked-busy +kms_vblank@pipe-H-query-forked-busy-hang +kms_vblank@pipe-H-wait-idle +kms_vblank@pipe-H-wait-idle-hang +kms_vblank@pipe-H-wait-forked +kms_vblank@pipe-H-wait-forked-hang +kms_vblank@pipe-H-wait-busy +kms_vblank@pipe-H-wait-busy-hang +kms_vblank@pipe-H-wait-forked-busy +kms_vblank@pipe-H-wait-forked-busy-hang +kms_vblank@pipe-H-ts-continuation-idle +kms_vblank@pipe-H-ts-continuation-idle-hang +kms_vblank@pipe-H-ts-continuation-dpms-rpm +kms_vblank@pipe-H-ts-continuation-dpms-suspend +kms_vblank@pipe-H-ts-continuation-suspend +kms_vblank@pipe-H-ts-continuation-modeset +kms_vblank@pipe-H-ts-continuation-modeset-hang +kms_vblank@pipe-H-ts-continuation-modeset-rpm +kms_vrr@flip-basic +kms_vrr@flip-dpms +kms_vrr@flip-suspend +kms_vrr@flipline +kms_vrr@negative-basic +kms_writeback@writeback-pixel-formats +kms_writeback@writeback-invalid-parameters +kms_writeback@writeback-fb-id +kms_writeback@writeback-check-output +prime_mmap_kms@buffer-sharing diff --git a/drivers/gpu/drm/ci/x86_64.config b/drivers/gpu/drm/ci/x86_64.config new file mode 100644 index 000000000000..1cbd49a5b23a --- /dev/null +++ b/drivers/gpu/drm/ci/x86_64.config @@ -0,0 +1,111 @@ +CONFIG_LOCALVERSION_AUTO=y +CONFIG_DEBUG_KERNEL=y + +CONFIG_CRYPTO_ZSTD=y +CONFIG_ZRAM_MEMORY_TRACKING=y +CONFIG_ZRAM_WRITEBACK=y +CONFIG_ZRAM=y +CONFIG_ZSMALLOC_STAT=y + +CONFIG_PWM=y +CONFIG_PM_DEVFREQ=y +CONFIG_OF=y +CONFIG_CROS_EC=y + +# abootimg with a 'dummy' rootfs fails with root=/dev/nfs +CONFIG_BLK_DEV_INITRD=n + +CONFIG_DEVFREQ_GOV_PERFORMANCE=y +CONFIG_DEVFREQ_GOV_POWERSAVE=y +CONFIG_DEVFREQ_GOV_USERSPACE=y +CONFIG_DEVFREQ_GOV_PASSIVE=y + +CONFIG_DRM=y +CONFIG_DRM_PANEL_SIMPLE=y +CONFIG_PWM_CROS_EC=y +CONFIG_BACKLIGHT_PWM=y + +# Strip out some stuff we don't need for graphics testing, to reduce +# the build. +CONFIG_CAN=n +CONFIG_WIRELESS=n +CONFIG_RFKILL=n +CONFIG_WLAN=n + +CONFIG_REGULATOR_FAN53555=y +CONFIG_REGULATOR=y + +CONFIG_REGULATOR_VCTRL=y + +CONFIG_KASAN=n +CONFIG_KASAN_INLINE=n +CONFIG_STACKTRACE=n + +CONFIG_TMPFS=y + +CONFIG_PROVE_LOCKING=n +CONFIG_DEBUG_LOCKDEP=n +CONFIG_SOFTLOCKUP_DETECTOR=y +CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y + +CONFIG_DETECT_HUNG_TASK=y + +CONFIG_USB_USBNET=y +CONFIG_NETDEVICES=y +CONFIG_USB_NET_DRIVERS=y +CONFIG_USB_RTL8152=y +CONFIG_USB_NET_AX8817X=y +CONFIG_USB_NET_SMSC95XX=y +CONFIG_USB_GADGET=y +CONFIG_USB_ETH=y + +CONFIG_FW_LOADER_COMPRESS=y + +# options for AMD devices +CONFIG_X86_AMD_PLATFORM_DEVICE=y +CONFIG_ACPI_VIDEO=y +CONFIG_X86_AMD_FREQ_SENSITIVITY=y +CONFIG_PINCTRL=y +CONFIG_PINCTRL_AMD=y +CONFIG_DRM_AMDGPU=m +CONFIG_DRM_AMDGPU_SI=y +CONFIG_DRM_AMDGPU_USERPTR=y +CONFIG_DRM_AMD_ACP=n +CONFIG_ACPI_WMI=y +CONFIG_MXM_WMI=y +CONFIG_PARPORT=y +CONFIG_PARPORT_PC=y +CONFIG_PARPORT_SERIAL=y +CONFIG_SERIAL_8250_DW=y +CONFIG_CHROME_PLATFORMS=y +CONFIG_KVM_AMD=m + +#options for Intel devices +CONFIG_MFD_INTEL_LPSS_PCI=y +CONFIG_KVM_INTEL=m + +#options for KVM guests +CONFIG_FUSE_FS=y +CONFIG_HYPERVISOR_GUEST=y +CONFIG_KVM=y +CONFIG_KVM_GUEST=y +CONFIG_VIRT_DRIVERS=y +CONFIG_VIRTIO_FS=y +CONFIG_DRM_VIRTIO_GPU=y +CONFIG_SERIAL_8250_CONSOLE=y +CONFIG_VIRTIO_NET=y +CONFIG_VIRTIO_CONSOLE=y +CONFIG_PARAVIRT=y +CONFIG_VIRTIO_BLK=y +CONFIG_VIRTUALIZATION=y +CONFIG_VIRTIO=y +CONFIG_VIRTIO_PCI=y +CONFIG_VIRTIO_MMIO=y +CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y +CONFIG_CRYPTO_DEV_VIRTIO=y +CONFIG_HW_RANDOM_VIRTIO=y +CONFIG_BLK_MQ_VIRTIO=y +CONFIG_TUN=y +CONFIG_VSOCKETS=y +CONFIG_VIRTIO_VSOCKETS=y +CONFIG_VHOST_VSOCK=m diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt new file mode 100644 index 000000000000..bd9392536e7c --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt @@ -0,0 +1,19 @@ +kms_addfb_basic@bad-pitch-65536,Fail +kms_addfb_basic@bo-too-small,Fail +kms_async_flips@invalid-async-flip,Fail +kms_atomic@plane-immutable-zpos,Fail +kms_atomic_transition@plane-toggle-modeset-transition,Fail +kms_bw@linear-tiling-1-displays-2560x1440p,Fail +kms_bw@linear-tiling-1-displays-3840x2160p,Fail +kms_bw@linear-tiling-2-displays-3840x2160p,Fail +kms_bw@linear-tiling-3-displays-1920x1080p,Fail +kms_color@degamma,Fail +kms_cursor_crc@cursor-size-change,Fail +kms_cursor_crc@pipe-A-cursor-size-change,Fail +kms_cursor_crc@pipe-B-cursor-size-change,Fail +kms_cursor_legacy@forked-move,Fail +kms_hdr@bpc-switch,Fail +kms_hdr@bpc-switch-dpms,Fail +kms_plane_multiple@atomic-pipe-A-tiling-none,Fail +kms_rmfb@close-fd,Fail +kms_rotation_crc@primary-rotation-180,Fail diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt new file mode 100644 index 000000000000..f8defa0f9e67 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt @@ -0,0 +1,21 @@ +kms_addfb_basic@too-high +kms_async_flips@alternate-sync-async-flip +kms_async_flips@async-flip-with-page-flip-events +kms_async_flips@crc +kms_async_flips@test-cursor +kms_async_flips@test-time-stamp +kms_atomic_transition@plane-all-modeset-transition-internal-panels +kms_atomic_transition@plane-all-transition +kms_atomic_transition@plane-use-after-nonblocking-unbind +kms_bw@linear-tiling-1-displays-1920x1080p +kms_bw@linear-tiling-2-displays-1920x1080p +kms_bw@linear-tiling-2-displays-2560x1440p +kms_bw@linear-tiling-3-displays-2560x1440p +kms_bw@linear-tiling-3-displays-3840x2160p +kms_cursor_crc@pipe-A-cursor-alpha-opaque +kms_cursor_crc@pipe-B-cursor-alpha-opaque +kms_plane@pixel-format +kms_plane_multiple@atomic-pipe-B-tiling-none +kms_plane_scaling@downscale-with-rotation-factor-0-5 +kms_universal_plane@disable-primary-vs-flip-pipe-A +kms_universal_plane@disable-primary-vs-flip-pipe-B diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt new file mode 100644 index 000000000000..e2c538a0f954 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt @@ -0,0 +1,2 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt b/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt new file mode 100644 index 000000000000..5f513c638beb --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt @@ -0,0 +1,17 @@ +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail +kms_plane_alpha_blend@alpha-basic,Fail +kms_plane_alpha_blend@alpha-opaque-fb,Fail +kms_plane_alpha_blend@alpha-transparent-fb,Fail +kms_plane_alpha_blend@constant-alpha-max,Fail diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt new file mode 100644 index 000000000000..d5000515a315 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt @@ -0,0 +1,32 @@ +kms_bw@linear-tiling-2-displays-1920x1080p +kms_bw@linear-tiling-2-displays-2560x1440p +kms_bw@linear-tiling-2-displays-3840x2160p +kms_bw@linear-tiling-3-displays-1920x1080p +kms_bw@linear-tiling-3-displays-2560x1440p +kms_bw@linear-tiling-3-displays-3840x2160p +kms_bw@linear-tiling-4-displays-1920x1080p +kms_bw@linear-tiling-4-displays-2560x1440p +kms_bw@linear-tiling-4-displays-3840x2160p +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling +kms_plane_alpha_blend@pipe-A-alpha-basic +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb +kms_plane_alpha_blend@pipe-A-constant-alpha-max +kms_plane_alpha_blend@pipe-B-alpha-basic +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb +kms_plane_alpha_blend@pipe-B-constant-alpha-max +kms_plane_alpha_blend@pipe-C-alpha-basic +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb +kms_plane_alpha_blend@pipe-C-constant-alpha-max +kms_sysfs_edid_timing diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt b/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt new file mode 100644 index 000000000000..fe55540a3f9a --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt @@ -0,0 +1,4 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* +# This is generating kernel oops with divide error +kms_plane_scaling@invalid-parameters \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt new file mode 100644 index 000000000000..46397ce38d5a --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt @@ -0,0 +1,58 @@ +kms_3d,Timeout +kms_bw@linear-tiling-2-displays-1920x1080p,Fail +kms_bw@linear-tiling-2-displays-2560x1440p,Fail +kms_bw@linear-tiling-2-displays-3840x2160p,Fail +kms_bw@linear-tiling-3-displays-1920x1080p,Fail +kms_bw@linear-tiling-3-displays-2560x1440p,Fail +kms_bw@linear-tiling-3-displays-3840x2160p,Fail +kms_bw@linear-tiling-4-displays-1920x1080p,Fail +kms_bw@linear-tiling-4-displays-2560x1440p,Fail +kms_bw@linear-tiling-4-displays-3840x2160p,Fail +kms_color@ctm-0-25,Fail +kms_color@ctm-0-50,Fail +kms_color@ctm-0-75,Fail +kms_color@ctm-max,Fail +kms_color@ctm-negative,Fail +kms_color@ctm-red-to-blue,Fail +kms_color@ctm-signed,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail +kms_hdmi_inject@inject-4k,Timeout +kms_plane@plane-position-hole,Timeout +kms_plane_alpha_blend@alpha-basic,Fail +kms_plane_alpha_blend@alpha-opaque-fb,Fail +kms_plane_alpha_blend@alpha-transparent-fb,Fail +kms_plane_alpha_blend@constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb,Fail +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb,Fail +kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb,Fail +kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb,Fail +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb,Fail +kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail +kms_plane_multiple@tiling-y,Timeout +kms_pwrite_crc,Timeout +kms_sysfs_edid_timing,Fail diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt new file mode 100644 index 000000000000..331c5841bb41 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt @@ -0,0 +1 @@ +kms_frontbuffer_tracking@fbc-tiling-linear diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt new file mode 100644 index 000000000000..3430b215c06e --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt @@ -0,0 +1,6 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* +# This is generating kernel oops with divide error +kms_plane_scaling@invalid-parameters +# This is cascading issues +kms_3d \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt b/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt new file mode 100644 index 000000000000..6139b410e767 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt @@ -0,0 +1,18 @@ +kms_color@ctm-0-25,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail +kms_plane_alpha_blend@alpha-basic,Fail +kms_plane_alpha_blend@alpha-opaque-fb,Fail +kms_plane_alpha_blend@alpha-transparent-fb,Fail +kms_plane_alpha_blend@constant-alpha-max,Fail diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt new file mode 100644 index 000000000000..0514a7b3fdb0 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt @@ -0,0 +1,38 @@ +kms_bw@linear-tiling-2-displays-1920x1080p +kms_bw@linear-tiling-2-displays-2560x1440p +kms_bw@linear-tiling-2-displays-3840x2160p +kms_bw@linear-tiling-3-displays-1920x1080p +kms_bw@linear-tiling-3-displays-2560x1440p +kms_bw@linear-tiling-3-displays-3840x2160p +kms_bw@linear-tiling-4-displays-1920x1080p +kms_bw@linear-tiling-4-displays-2560x1440p +kms_bw@linear-tiling-4-displays-3840x2160p +kms_draw_crc@draw-method-xrgb8888-render-xtiled +kms_flip@flip-vs-suspend +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling +kms_hdr@bpc-switch-suspend +kms_plane_alpha_blend@constant-alpha-min +kms_plane_alpha_blend@pipe-A-alpha-basic +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb +kms_plane_alpha_blend@pipe-A-constant-alpha-max +kms_plane_alpha_blend@pipe-B-alpha-basic +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb +kms_plane_alpha_blend@pipe-B-constant-alpha-max +kms_plane_alpha_blend@pipe-C-alpha-basic +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb +kms_plane_alpha_blend@pipe-C-constant-alpha-max +kms_psr2_su@page_flip-NV12 +kms_psr2_su@page_flip-P010 +kms_setmode@basic diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt b/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt new file mode 100644 index 000000000000..6d3d7ddc377f --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt @@ -0,0 +1,2 @@ +# This is generating kernel oops with divide error +kms_plane_scaling@invalid-parameters diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt b/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt new file mode 100644 index 000000000000..5bd432e78129 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt @@ -0,0 +1,19 @@ +kms_fbcon_fbt@fbc,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail +kms_frontbuffer_tracking@fbcdrrs-tiling-linear,Fail +kms_plane_alpha_blend@alpha-basic,Fail +kms_plane_alpha_blend@alpha-opaque-fb,Fail +kms_plane_alpha_blend@alpha-transparent-fb,Fail +kms_plane_alpha_blend@constant-alpha-max,Fail diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt new file mode 100644 index 000000000000..fc41d13a2d56 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt @@ -0,0 +1,41 @@ +kms_bw@linear-tiling-1-displays-3840x2160p +kms_bw@linear-tiling-2-displays-1920x1080p +kms_bw@linear-tiling-2-displays-2560x1440p +kms_bw@linear-tiling-2-displays-3840x2160p +kms_bw@linear-tiling-3-displays-1920x1080p +kms_bw@linear-tiling-3-displays-2560x1440p +kms_bw@linear-tiling-3-displays-3840x2160p +kms_bw@linear-tiling-4-displays-1920x1080p +kms_bw@linear-tiling-4-displays-2560x1440p +kms_bw@linear-tiling-4-displays-3840x2160p +kms_flip@blocking-wf_vblank +kms_flip@wf_vblank-ts-check +kms_flip@wf_vblank-ts-check-interruptible +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling +kms_frontbuffer_tracking@fbc-tiling-linear +kms_plane_alpha_blend@pipe-A-alpha-basic +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb +kms_plane_alpha_blend@pipe-A-constant-alpha-max +kms_plane_alpha_blend@pipe-B-alpha-basic +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb +kms_plane_alpha_blend@pipe-B-constant-alpha-max +kms_plane_alpha_blend@pipe-C-alpha-basic +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb +kms_plane_alpha_blend@pipe-C-constant-alpha-max +kms_prop_blob@invalid-set-prop-any +kms_rotation_crc@multiplane-rotation +kms_rotation_crc@multiplane-rotation-cropping-bottom +kms_rotation_crc@multiplane-rotation-cropping-top +kms_setmode@basic diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt b/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt new file mode 100644 index 000000000000..4c7d00ce14bc --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt @@ -0,0 +1,5 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* + +# This is generating kernel oops with divide error +kms_plane_scaling@invalid-parameters \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt new file mode 100644 index 000000000000..56ec021a7679 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt @@ -0,0 +1,25 @@ +kms_bw@linear-tiling-2-displays-2560x1440p,Fail +kms_bw@linear-tiling-4-displays-2560x1440p,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail +kms_plane_alpha_blend@alpha-basic,Fail +kms_plane_alpha_blend@alpha-opaque-fb,Fail +kms_plane_alpha_blend@alpha-transparent-fb,Fail +kms_plane_alpha_blend@constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail +kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt new file mode 100644 index 000000000000..f3ba1c4c5d46 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt @@ -0,0 +1,26 @@ +kms_async_flips@crc +kms_bw@linear-tiling-2-displays-1920x1080p +kms_bw@linear-tiling-2-displays-3840x2160p +kms_bw@linear-tiling-3-displays-1920x1080p +kms_bw@linear-tiling-3-displays-2560x1440p +kms_bw@linear-tiling-3-displays-3840x2160p +kms_bw@linear-tiling-4-displays-1920x1080p +kms_bw@linear-tiling-4-displays-3840x2160p +kms_color@ctm-0-25 +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling +kms_plane_alpha_blend@pipe-A-alpha-basic +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb +kms_plane_alpha_blend@pipe-B-alpha-basic +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb +kms_plane_alpha_blend@pipe-B-constant-alpha-max +kms_plane_alpha_blend@pipe-C-alpha-basic +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb +kms_sysfs_edid_timing diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt new file mode 100644 index 000000000000..4c7d00ce14bc --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt @@ -0,0 +1,5 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* + +# This is generating kernel oops with divide error +kms_plane_scaling@invalid-parameters \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt new file mode 100644 index 000000000000..a6da5544e198 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt @@ -0,0 +1,37 @@ +kms_bw@linear-tiling-2-displays-3840x2160p,Fail +kms_bw@linear-tiling-3-displays-1920x1080p,Fail +kms_bw@linear-tiling-3-displays-2560x1440p,Fail +kms_bw@linear-tiling-3-displays-3840x2160p,Fail +kms_bw@linear-tiling-4-displays-1920x1080p,Fail +kms_bw@linear-tiling-4-displays-2560x1440p,Fail +kms_bw@linear-tiling-4-displays-3840x2160p,Fail +kms_bw@linear-tiling-5-displays-1920x1080p,Fail +kms_bw@linear-tiling-5-displays-2560x1440p,Fail +kms_bw@linear-tiling-5-displays-3840x2160p,Fail +kms_color@ctm-0-25,Fail +kms_flip@flip-vs-panning-vs-hang,Timeout +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail +kms_rotation_crc@bad-pixel-format,Fail +kms_rotation_crc@multiplane-rotation,Fail +kms_rotation_crc@multiplane-rotation-cropping-bottom,Fail +kms_rotation_crc@multiplane-rotation-cropping-top,Fail diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt new file mode 100644 index 000000000000..1cd910ee06df --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt @@ -0,0 +1,5 @@ +kms_draw_crc@.* +kms_flip@blocking-absolute-wf_vblank +kms_flip@bo-too-big-interruptible +kms_flip@busy-flip +kms_flip@flip-vs-rmfb-interruptible diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt new file mode 100644 index 000000000000..1d0621750b14 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt @@ -0,0 +1,11 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* + +# GPU hangs, then the whole machine +gem_eio.* + +# Whole machine hangs +kms_flip@absolute-wf_vblank@a-edp1 + +# This is generating kernel oops with divide error +kms_plane_scaling@invalid-parameters \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt new file mode 100644 index 000000000000..967327ddc1ac --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt @@ -0,0 +1,48 @@ +kms_bw@linear-tiling-2-displays-1920x1080p,Fail +kms_bw@linear-tiling-2-displays-2560x1440p,Fail +kms_bw@linear-tiling-2-displays-3840x2160p,Fail +kms_bw@linear-tiling-3-displays-1920x1080p,Fail +kms_bw@linear-tiling-3-displays-2560x1440p,Fail +kms_bw@linear-tiling-3-displays-3840x2160p,Fail +kms_bw@linear-tiling-4-displays-1920x1080p,Fail +kms_bw@linear-tiling-4-displays-2560x1440p,Fail +kms_bw@linear-tiling-4-displays-3840x2160p,Fail +kms_fbcon_fbt@fbc,Fail +kms_fbcon_fbt@fbc-suspend,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail +kms_frontbuffer_tracking@fbc-tiling-linear,Fail +kms_plane_alpha_blend@alpha-basic,Fail +kms_plane_alpha_blend@alpha-opaque-fb,Fail +kms_plane_alpha_blend@alpha-transparent-fb,Fail +kms_plane_alpha_blend@constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb,Fail +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb,Fail +kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb,Fail +kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb,Fail +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb,Fail +kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt new file mode 100644 index 000000000000..c33202e7e2a1 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt @@ -0,0 +1 @@ +kms_flip@flip-vs-suspend diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt new file mode 100644 index 000000000000..f3be0888a214 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt @@ -0,0 +1,2 @@ +# This is generating kernel oops with divide error +kms_plane_scaling@invalid-parameters \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt new file mode 100644 index 000000000000..671916067dba --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt @@ -0,0 +1,29 @@ +kms_3d,Fail +kms_addfb_basic@addfb25-bad-modifier,Fail +kms_bw@linear-tiling-1-displays-1920x1080p,Fail +kms_bw@linear-tiling-1-displays-2560x1440p,Fail +kms_bw@linear-tiling-1-displays-3840x2160p,Fail +kms_bw@linear-tiling-2-displays-1920x1080p,Fail +kms_bw@linear-tiling-2-displays-2560x1440p,Fail +kms_bw@linear-tiling-2-displays-3840x2160p,Fail +kms_bw@linear-tiling-3-displays-1920x1080p,Fail +kms_bw@linear-tiling-3-displays-2560x1440p,Fail +kms_bw@linear-tiling-3-displays-3840x2160p,Fail +kms_color@pipe-A-invalid-gamma-lut-sizes,Fail +kms_color@pipe-B-invalid-gamma-lut-sizes,Fail +kms_force_connector_basic@force-connector-state,Fail +kms_force_connector_basic@force-edid,Fail +kms_force_connector_basic@force-load-detect,Fail +kms_force_connector_basic@prune-stale-modes,Fail +kms_invalid_mode@int-max-clock,Fail +kms_plane_scaling@planes-upscale-20x20,Fail +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25,Fail +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5,Fail +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75,Fail +kms_plane_scaling@upscale-with-modifier-20x20,Fail +kms_plane_scaling@upscale-with-pixel-format-20x20,Fail +kms_plane_scaling@upscale-with-rotation-20x20,Fail +kms_properties@get_properties-sanity-atomic,Fail +kms_properties@plane-properties-atomic,Fail +kms_properties@plane-properties-legacy,Fail +kms_rmfb@close-fd,Fail diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt new file mode 100644 index 000000000000..6ff81d00e84e --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt @@ -0,0 +1,10 @@ +kms_addfb_basic@addfb25-bad-modifier,Fail +kms_bw@linear-tiling-1-displays-2560x1440p,Fail +kms_bw@linear-tiling-2-displays-1920x1080p,Fail +kms_bw@linear-tiling-2-displays-2560x1440p,Fail +kms_bw@linear-tiling-2-displays-3840x2160p,Fail +kms_bw@linear-tiling-3-displays-2560x1440p,Fail +kms_bw@linear-tiling-3-displays-3840x2160p,Fail +kms_color@pipe-A-invalid-gamma-lut-sizes,Fail +kms_plane_scaling@upscale-with-rotation-20x20,Fail +kms_rmfb@close-fd,Fail \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt new file mode 100644 index 000000000000..208890b79eb0 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt @@ -0,0 +1,14 @@ +core_setmaster_vs_auth +kms_bw@linear-tiling-1-displays-1920x1080p +kms_bw@linear-tiling-1-displays-3840x2160p +kms_bw@linear-tiling-3-displays-1920x1080p +kms_cursor_legacy@cursor-vs-flip-atomic +kms_plane_scaling@invalid-num-scalers +kms_plane_scaling@planes-upscale-20x20 +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5 +kms_plane_scaling@upscale-with-modifier-20x20 +kms_plane_scaling@upscale-with-pixel-format-20x20 +kms_prop_blob@invalid-set-prop-any +kms_properties@get_properties-sanity-atomic +kms_properties@plane-properties-atomic +kms_properties@plane-properties-legacy \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt b/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt new file mode 100644 index 000000000000..860e702091e2 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt @@ -0,0 +1,12 @@ +kms_3d,Fail +kms_properties@connector-properties-atomic,Fail +kms_properties@get_properties-sanity-atomic,Fail +kms_properties@get_properties-sanity-non-atomic,Fail +kms_properties@connector-properties-legacy,Fail +kms_cursor_legacy@forked-bo,Fail +kms_cursor_legacy@forked-move,Fail +kms_cursor_legacy@single-bo,Fail +kms_cursor_legacy@single-move,Fail +kms_cursor_legacy@torture-bo,Fail +kms_cursor_legacy@torture-move,Fail +kms_hdmi_inject@inject-4k,Fail \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt b/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt new file mode 100644 index 000000000000..b63329d06767 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt @@ -0,0 +1,4 @@ +kms_force_connector_basic@force-connector-state +kms_force_connector_basic@force-edid +kms_force_connector_basic@force-load-detect +kms_force_connector_basic@prune-stale-modes \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt new file mode 100644 index 000000000000..9981682feab2 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt @@ -0,0 +1,15 @@ +kms_3d,Fail +kms_addfb_basic@addfb25-bad-modifier,Fail +kms_cursor_legacy@all-pipes-forked-bo,Fail +kms_cursor_legacy@all-pipes-forked-move,Fail +kms_cursor_legacy@all-pipes-single-bo,Fail +kms_cursor_legacy@all-pipes-single-move,Fail +kms_cursor_legacy@all-pipes-torture-bo,Fail +kms_cursor_legacy@all-pipes-torture-move,Fail +kms_cursor_legacy@pipe-A-forked-bo,Fail +kms_cursor_legacy@pipe-A-forked-move,Fail +kms_cursor_legacy@pipe-A-single-bo,Fail +kms_cursor_legacy@pipe-A-single-move,Fail +kms_cursor_legacy@pipe-A-torture-bo,Fail +kms_cursor_legacy@pipe-A-torture-move,Fail +kms_hdmi_inject@inject-4k,Fail diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt new file mode 100644 index 000000000000..0e3b60d3fade --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt @@ -0,0 +1,4 @@ +kms_force_connector_basic@force-connector-state +kms_force_connector_basic@force-edid +kms_force_connector_basic@force-load-detect +kms_force_connector_basic@prune-stale-modes diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt new file mode 100644 index 000000000000..88a1fc0a3b0d --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt @@ -0,0 +1,2 @@ +kms_3d,Fail +kms_addfb_basic@addfb25-bad-modifier,Fail diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt new file mode 100644 index 000000000000..0e3b60d3fade --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt @@ -0,0 +1,4 @@ +kms_force_connector_basic@force-connector-state +kms_force_connector_basic@force-edid +kms_force_connector_basic@force-load-detect +kms_force_connector_basic@prune-stale-modes diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt new file mode 100644 index 000000000000..cd49c8ce2059 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt @@ -0,0 +1,2 @@ +# Whole machine hangs +kms_cursor_legacy@all-pipes-torture-move \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt new file mode 100644 index 000000000000..14adeba3b62d --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt @@ -0,0 +1,25 @@ +kms_cursor_legacy@cursor-vs-flip-toggle,Fail +kms_cursor_legacy@cursor-vs-flip-varying-size,Fail +kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions,Crash +kms_pipe_crc_basic@compare-crc-sanitycheck-nv12,Fail +kms_plane@pixel-format,Fail +kms_plane@pixel-format-source-clamping,Fail +kms_plane@plane-position-covered,Fail +kms_plane@plane-position-hole,Fail +kms_plane@plane-position-hole-dpms,Fail +kms_plane_alpha_blend@alpha-7efc,Fail +kms_plane_alpha_blend@coverage-7efc,Fail +kms_plane_alpha_blend@coverage-vs-premult-vs-constant,Fail +kms_plane_alpha_blend@pipe-A-alpha-7efc,Fail +kms_plane_alpha_blend@pipe-A-coverage-7efc,Fail +kms_plane_alpha_blend@pipe-A-coverage-vs-premult-vs-constant,Fail +kms_plane_alpha_blend@pipe-B-alpha-7efc,Fail +kms_plane_alpha_blend@pipe-B-alpha-basic,Fail +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail +kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail +kms_plane_alpha_blend@pipe-B-constant-alpha-mid,Fail +kms_plane_alpha_blend@pipe-B-coverage-7efc,Fail +kms_plane_alpha_blend@pipe-B-coverage-vs-premult-vs-constant,Fail +kms_rmfb@close-fd,Fail +kms_universal_plane@disable-primary-vs-flip-pipe-b,Fail +kms_universal_plane@universal-plane-pipe-B-sanity,Fail diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt new file mode 100644 index 000000000000..636563d3e59a --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt @@ -0,0 +1,7 @@ + +# Test ends up reading CRC from frame before cursor update +# bug +# sometimes.. tbd if this is a kernel CRC bug or a test +kms_cursor_crc@.* +kms_plane_multiple@atomic-pipe-A-tiling-none +kms_atomic_transition@modeset-transition-nonblocking-fencing,Fail \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt new file mode 100644 index 000000000000..410e0eeb3161 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt @@ -0,0 +1,23 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* + +# Test incorrectly assumes that CTM support implies gamma/degamma +# LUT support. None of the subtests handle the case of only having +# CTM support +kms_color.* + +# 4k@60 is not supported on this hw, but driver doesn't handle it +# too gracefully.. https://gitlab.freedesktop.org/drm/msm/-/issues/15 +kms_bw@linear-tiling-.*-displays-3840x2160p + +# Until igt fix lands: https://patchwork.freedesktop.org/patch/493175/ +kms_bw@linear-tiling-2.* +kms_bw@linear-tiling-3.* +kms_bw@linear-tiling-4.* +kms_bw@linear-tiling-5.* +kms_bw@linear-tiling-6.* + +# igt fix posted: https://patchwork.freedesktop.org/patch/499926/ +# failure mode is flakey due to randomization but fails frequently +# enough to be detected as a Crash or occasionally UnexpectedPass. +kms_plane_multiple@atomic-pipe-A-tiling-none diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt new file mode 100644 index 000000000000..09c0c623cd75 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt @@ -0,0 +1,68 @@ +kms_color@ctm-0-25,Fail +kms_color@ctm-0-50,Fail +kms_color@ctm-0-75,Fail +kms_color@ctm-blue-to-red,Fail +kms_color@ctm-green-to-red,Fail +kms_color@ctm-negative,Fail +kms_color@ctm-red-to-blue,Fail +kms_color@ctm-signed,Fail +kms_color@pipe-A-ctm-0-25,Fail +kms_color@pipe-A-ctm-0-5,Fail +kms_color@pipe-A-ctm-0-75,Fail +kms_color@pipe-A-ctm-blue-to-red,Fail +kms_color@pipe-A-ctm-green-to-red,Fail +kms_color@pipe-A-ctm-max,Fail +kms_color@pipe-A-ctm-negative,Fail +kms_color@pipe-A-ctm-red-to-blue,Fail +kms_color@pipe-A-legacy-gamma,Fail +kms_cursor_legacy@basic-flip-after-cursor-atomic,Fail +kms_cursor_legacy@basic-flip-after-cursor-legacy,Fail +kms_cursor_legacy@basic-flip-after-cursor-varying-size,Fail +kms_cursor_legacy@basic-flip-before-cursor-atomic,Fail +kms_cursor_legacy@basic-flip-before-cursor-legacy,Fail +kms_cursor_legacy@basic-flip-before-cursor-varying-size,Fail +kms_cursor_legacy@cursor-vs-flip-atomic,Fail +kms_cursor_legacy@cursor-vs-flip-atomic-transitions,Fail +kms_cursor_legacy@cursor-vs-flip-atomic-transitions-varying-size,Fail +kms_cursor_legacy@cursor-vs-flip-legacy,Fail +kms_cursor_legacy@cursor-vs-flip-toggle,Fail +kms_cursor_legacy@cursor-vs-flip-varying-size,Fail +kms_cursor_legacy@cursorA-vs-flipA-toggle,Fail +kms_cursor_legacy@flip-vs-cursor-atomic,Fail +kms_cursor_legacy@flip-vs-cursor-crc-atomic,Fail +kms_cursor_legacy@flip-vs-cursor-crc-legacy,Fail +kms_cursor_legacy@flip-vs-cursor-legacy,Fail +kms_cursor_legacy@short-flip-after-cursor-atomic-transitions,Fail +kms_cursor_legacy@short-flip-after-cursor-atomic-transitions-varying-size,Fail +kms_cursor_legacy@short-flip-after-cursor-toggle,Fail +kms_cursor_legacy@short-flip-before-cursor-atomic-transitions,Fail +kms_cursor_legacy@short-flip-before-cursor-atomic-transitions-varying-size,Fail +kms_pipe_crc_basic@compare-crc-sanitycheck-nv12,Fail +kms_plane@pixel-format,Fail +kms_plane@pixel-format-source-clamping,Fail +kms_plane_alpha_blend@alpha-7efc,Fail +kms_plane_alpha_blend@coverage-7efc,Fail +kms_plane_alpha_blend@coverage-vs-premult-vs-constant,Fail +kms_plane_alpha_blend@pipe-A-alpha-7efc,Fail +kms_plane_alpha_blend@pipe-A-coverage-7efc,Fail +kms_plane_alpha_blend@pipe-A-coverage-vs-premult-vs-constant,Fail +kms_plane_cursor@overlay,Fail +kms_plane_cursor@pipe-A-overlay-size-128,Fail +kms_plane_cursor@pipe-A-overlay-size-256,Fail +kms_plane_cursor@pipe-A-overlay-size-64,Fail +kms_plane_cursor@pipe-A-viewport-size-128,Fail +kms_plane_cursor@pipe-A-viewport-size-256,Fail +kms_plane_cursor@pipe-A-viewport-size-64,Fail +kms_plane_cursor@viewport,Fail +kms_plane_scaling@downscale-with-pixel-format-factor-0-25,Timeout +kms_plane_scaling@downscale-with-pixel-format-factor-0-5,Timeout +kms_plane_scaling@downscale-with-pixel-format-factor-0-75,Timeout +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-25,Timeout +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-5,Timeout +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-75,Timeout +kms_plane_scaling@plane-scaler-with-clipping-clamping-pixel-formats,Timeout +kms_plane_scaling@plane-scaler-with-pixel-format-unity-scaling,Timeout +kms_plane_scaling@planes-downscale-factor-0-25,Fail +kms_plane_scaling@scaler-with-clipping-clamping,Timeout +kms_plane_scaling@scaler-with-pixel-format-unity-scaling,Timeout +kms_rmfb@close-fd,Fail diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt new file mode 100644 index 000000000000..5b3aaab7ac3f --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt @@ -0,0 +1,11 @@ + + +# Test ends up reading CRC from frame before cursor update +# bug +# sometimes.. tbd if this is a kernel CRC bug or a test +kms_cursor_crc@.* +kms_cursor_legacy@flip-vs-cursor-toggle +kms_cursor_legacy@pipe-A-forked-bo +kms_cursor_legacy@pipe-A-forked-move +kms_cursor_legacy@short-flip-before-cursor-toggle +kms_flip@dpms-vs-vblank-race-interruptible diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt new file mode 100644 index 000000000000..42675f1c6d76 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt @@ -0,0 +1,2 @@ +# Hangs machine +kms_bw.* \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt new file mode 100644 index 000000000000..2a1baa948e12 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt @@ -0,0 +1,48 @@ +kms_3d,Crash +kms_bw@linear-tiling-2-displays-1920x1080p,Fail +kms_bw@linear-tiling-2-displays-2560x1440p,Fail +kms_bw@linear-tiling-2-displays-3840x2160p,Fail +kms_bw@linear-tiling-3-displays-1920x1080p,Fail +kms_bw@linear-tiling-3-displays-2560x1440p,Fail +kms_bw@linear-tiling-3-displays-3840x2160p,Fail +kms_force_connector_basic@force-load-detect,Fail +kms_invalid_mode@int-max-clock,Crash +kms_plane@pixel-format,Crash +kms_plane@pixel-format-source-clamping,Crash +kms_plane@plane-position-hole,Crash +kms_plane@plane-position-hole-dpms,Crash +kms_plane_cursor@overlay,Crash +kms_plane_cursor@pipe-A-overlay-size-128,Fail +kms_plane_cursor@pipe-A-overlay-size-256,Fail +kms_plane_cursor@pipe-A-overlay-size-64,Fail +kms_plane_cursor@pipe-A-primary-size-128,Fail +kms_plane_cursor@pipe-A-primary-size-256,Fail +kms_plane_cursor@pipe-A-primary-size-64,Fail +kms_plane_cursor@pipe-A-viewport-size-128,Fail +kms_plane_cursor@pipe-A-viewport-size-256,Fail +kms_plane_cursor@pipe-A-viewport-size-64,Fail +kms_plane_cursor@pipe-B-overlay-size-128,Fail +kms_plane_cursor@pipe-B-overlay-size-256,Fail +kms_plane_cursor@pipe-B-overlay-size-64,Fail +kms_plane_cursor@pipe-B-primary-size-128,Fail +kms_plane_cursor@pipe-B-primary-size-256,Fail +kms_plane_cursor@pipe-B-primary-size-64,Fail +kms_plane_cursor@pipe-B-viewport-size-128,Fail +kms_plane_cursor@pipe-B-viewport-size-256,Fail +kms_plane_cursor@pipe-B-viewport-size-64,Fail +kms_plane_cursor@primary,Crash +kms_plane_cursor@viewport,Crash +kms_plane_lowres@tiling-none,Fail +kms_plane_scaling@downscale-with-modifier-factor-0-25,Fail +kms_plane_scaling@downscale-with-rotation-factor-0-25,Fail +kms_plane_scaling@upscale-with-modifier-20x20,Fail +kms_plane_scaling@upscale-with-modifier-factor-0-25,Fail +kms_plane_scaling@upscale-with-pixel-format-20x20,Fail +kms_plane_scaling@upscale-with-pixel-format-factor-0-25,Fail +kms_plane_scaling@upscale-with-rotation-20x20,Fail +kms_prime@basic-crc,Fail +kms_properties@connector-properties-atomic,Crash +kms_properties@connector-properties-legacy,Crash +kms_properties@get_properties-sanity-atomic,Crash +kms_properties@get_properties-sanity-non-atomic,Crash +kms_setmode@invalid-clone-single-crtc,Crash diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt new file mode 100644 index 000000000000..45c54c75c899 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt @@ -0,0 +1,9 @@ +kms_addfb_basic@addfb25-bad-modifier +kms_cursor_crc@.* +kms_flip@basic-flip-vs-wf_vblank +kms_invalid_mode@int-max-clock,Crash +kms_pipe_crc_basic@.* +kms_properties@connector-properties-atomic,Crash +kms_properties@get_properties-sanity-atomic,Crash +kms_properties@get_properties-sanity-non-atomic,Crash +kms_rmfb@close-fd diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt new file mode 100644 index 000000000000..f20c3574b75a --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt @@ -0,0 +1,52 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* + +# Too unstable, machine ends up hanging after lots of Oopses +kms_cursor_legacy.* + +# Started hanging the machine on Linux 5.19-rc2: +# +# [IGT] kms_plane_lowres: executing +# [IGT] kms_plane_lowres: starting subtest pipe-F-tiling-y +# [IGT] kms_plane_lowres: exiting, ret=77 +# Console: switching to colour frame buffer device 170x48 +# rockchip-drm display-subsystem: [drm] *ERROR* flip_done timed out +# rockchip-drm display-subsystem: [drm] *ERROR* [CRTC:35:crtc-0] commit wait timed out +# BUG: spinlock bad magic on CPU#3, kms_plane_lowre/482 +# 8<--- cut here --- +# Unable to handle kernel paging request at virtual address 7812078e +# [7812078e] *pgd=00000000 +# Internal error: Oops: 5 [#1] SMP ARM +# Modules linked in: +# CPU: 3 PID: 482 Comm: kms_plane_lowre Tainted: G W 5.19.0-rc2-323596-g00535de92171 #1 +# Hardware name: Rockchip (Device Tree) +# Process kms_plane_lowre (pid: 482, stack limit = 0x1193ac2b) +# spin_dump from do_raw_spin_lock+0xa4/0xe8 +# do_raw_spin_lock from wait_for_completion_timeout+0x2c/0x120 +# wait_for_completion_timeout from drm_crtc_commit_wait+0x18/0x7c +# drm_crtc_commit_wait from drm_atomic_helper_wait_for_dependencies+0x44/0x168 +# drm_atomic_helper_wait_for_dependencies from commit_tail+0x34/0x180 +# commit_tail from drm_atomic_helper_commit+0x164/0x18c +# drm_atomic_helper_commit from drm_atomic_commit+0xac/0xe4 +# drm_atomic_commit from drm_client_modeset_commit_atomic+0x23c/0x284 +# drm_client_modeset_commit_atomic from drm_client_modeset_commit_locked+0x60/0x1c8 +# drm_client_modeset_commit_locked from drm_client_modeset_commit+0x24/0x40 +# drm_client_modeset_commit from drm_fbdev_client_restore+0x58/0x94 +# drm_fbdev_client_restore from drm_client_dev_restore+0x70/0xbc +# drm_client_dev_restore from drm_release+0xf4/0x114 +# drm_release from __fput+0x74/0x240 +# __fput from task_work_run+0x84/0xb4 +# task_work_run from do_exit+0x34c/0xa20 +# do_exit from do_group_exit+0x34/0x98 +# do_group_exit from __wake_up_parent+0x0/0x18 +# Code: e595c008 12843d19 03e00000 03093168 (15940508) +# ---[ end trace 0000000000000000 ]--- +# note: kms_plane_lowre[482] exited with preempt_count 1 +# Fixing recursive fault but reboot is needed! +kms_plane_lowres@pipe-F-tiling-y + +# Take too long, we have only two machines, and these are very flaky +kms_cursor_crc.* + +# Machine is hanging in this test, so skip it +kms_pipe_crc_basic@disable-crc-after-crtc \ No newline at end of file diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt new file mode 100644 index 000000000000..6db08ba6b008 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt @@ -0,0 +1,37 @@ +kms_color@legacy-gamma,Fail +kms_color@pipe-A-legacy-gamma,Fail +kms_color@pipe-B-legacy-gamma,Fail +kms_flip@basic-flip-vs-wf_vblank,Fail +kms_flip@blocking-wf_vblank,Fail +kms_flip@dpms-vs-vblank-race,Fail +kms_flip@flip-vs-absolute-wf_vblank,Fail +kms_flip@flip-vs-absolute-wf_vblank-interruptible,Fail +kms_flip@flip-vs-blocking-wf-vblank,Fail +kms_flip@flip-vs-panning,Fail +kms_flip@flip-vs-panning-interruptible,Fail +kms_flip@flip-vs-wf_vblank-interruptible,Fail +kms_flip@plain-flip-fb-recreate,Fail +kms_flip@plain-flip-fb-recreate-interruptible,Fail +kms_flip@plain-flip-ts-check,Fail +kms_flip@plain-flip-ts-check-interruptible,Fail +kms_flip@wf_vblank-ts-check,Fail +kms_flip@wf_vblank-ts-check-interruptible,Fail +kms_invalid_mode@int-max-clock,Fail +kms_plane@pixel-format,Fail +kms_plane@pixel-format-source-clamping,Fail +kms_plane@plane-panning-bottom-right,Fail +kms_plane@plane-panning-top-left,Fail +kms_plane@plane-position-covered,Fail +kms_plane_cursor@pipe-B-overlay-size-128,Fail +kms_plane_cursor@pipe-B-overlay-size-256,Fail +kms_plane_cursor@pipe-B-overlay-size-64,Fail +kms_plane_cursor@pipe-B-primary-size-128,Fail +kms_plane_cursor@pipe-B-primary-size-256,Fail +kms_plane_cursor@pipe-B-primary-size-64,Fail +kms_plane_cursor@pipe-B-viewport-size-128,Fail +kms_plane_cursor@pipe-B-viewport-size-256,Fail +kms_plane_cursor@pipe-B-viewport-size-64,Fail +kms_plane_multiple@atomic-pipe-B-tiling-none,Fail +kms_plane_multiple@tiling-none,Fail +kms_prime@basic-crc,Fail +kms_rmfb@close-fd,Fail diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt new file mode 100644 index 000000000000..4c0539b4beaf --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt @@ -0,0 +1,23 @@ + +kms_cursor_crc@.* +kms_flip@dpms-vs-vblank-race-interruptible +kms_flip@flip-vs-expired-vblank +kms_flip@modeset-vs-vblank-race-interruptible +kms_pipe_crc_basic@.* +kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-A +kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-B +kms_plane@plane-position-hole +kms_plane_multiple@atomic-pipe-A-tiling-none +kms_plane_multiple@atomic-pipe-B-tiling-none +kms_sequence@get-forked +kms_sequence@get-forked-busy +kms_setmode@basic +kms_universal_plane@universal-plane-pipe-B-functional,UnexpectedPass +kms_vblank@pipe-A-accuracy-idle +kms_vblank@pipe-A-query-busy +kms_vblank@pipe-A-query-forked-busy +kms_vblank@pipe-A-wait-idle +kms_vblank@pipe-B-accuracy-idle +kms_vblank@pipe-B-query-busy +kms_vblank@pipe-B-query-forked-busy +kms_vblank@pipe-B-wait-idle diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt new file mode 100644 index 000000000000..10c3d81a919a --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt @@ -0,0 +1,5 @@ +# Suspend to RAM seems to be broken on this machine +.*suspend.* + +# Too unstable, machine ends up hanging after lots of Oopses +kms_cursor_legacy.* diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt new file mode 100644 index 000000000000..9586b2339f6f --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt @@ -0,0 +1,38 @@ +kms_addfb_basic@addfb25-bad-modifier,Fail +kms_addfb_basic@bad-pitch-65536,Fail +kms_addfb_basic@bo-too-small,Fail +kms_addfb_basic@size-max,Fail +kms_addfb_basic@too-high,Fail +kms_atomic_transition@plane-primary-toggle-with-vblank-wait,Fail +kms_bw@linear-tiling-1-displays-1920x1080p,Fail +kms_bw@linear-tiling-1-displays-2560x1440p,Fail +kms_bw@linear-tiling-1-displays-3840x2160p,Fail +kms_bw@linear-tiling-2-displays-1920x1080p,Fail +kms_bw@linear-tiling-2-displays-2560x1440p,Fail +kms_bw@linear-tiling-2-displays-3840x2160p,Fail +kms_invalid_mode@int-max-clock,Fail +kms_plane_scaling@downscale-with-modifier-factor-0-25,Fail +kms_plane_scaling@downscale-with-rotation-factor-0-25,Fail +kms_plane_scaling@planes-upscale-20x20,Fail +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25,Fail +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5,Fail +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75,Fail +kms_plane_scaling@upscale-with-modifier-20x20,Fail +kms_plane_scaling@upscale-with-modifier-factor-0-25,Fail +kms_plane_scaling@upscale-with-pixel-format-20x20,Fail +kms_plane_scaling@upscale-with-pixel-format-factor-0-25,Fail +kms_plane_scaling@upscale-with-rotation-20x20,Fail +kms_vblank@crtc-id,Fail +kms_vblank@invalid,Fail +kms_vblank@pipe-A-accuracy-idle,Fail +kms_vblank@pipe-A-query-busy,Fail +kms_vblank@pipe-A-query-forked,Fail +kms_vblank@pipe-A-query-forked-busy,Fail +kms_vblank@pipe-A-query-idle,Fail +kms_vblank@pipe-A-ts-continuation-idle,Fail +kms_vblank@pipe-A-ts-continuation-modeset,Fail +kms_vblank@pipe-A-ts-continuation-suspend,Fail +kms_vblank@pipe-A-wait-busy,Fail +kms_vblank@pipe-A-wait-forked,Fail +kms_vblank@pipe-A-wait-forked-busy,Fail +kms_vblank@pipe-A-wait-idle,Fail diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt new file mode 100644 index 000000000000..78be18174012 --- /dev/null +++ b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt @@ -0,0 +1,6 @@ +# Hits a "refcount_t: underflow; use-after-free" in virtio_gpu_fence_event_process +# When run in a particular order with other tests +kms_cursor_legacy.* + +# Job just hangs without any output +kms_flip@flip-vs-suspend.* \ No newline at end of file diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h index e99a6fa03d45..a7e677598004 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h @@ -58,6 +58,7 @@ struct i915_perf_group; typedef u32 intel_engine_mask_t; #define ALL_ENGINES ((intel_engine_mask_t)~0ul) +#define VIRTUAL_ENGINES BIT(BITS_PER_TYPE(intel_engine_mask_t) - 1) struct intel_hw_status_page { struct list_head timelines; diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c index a0e3ef1c65d2..b5b7f2fe8c78 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c @@ -5470,6 +5470,9 @@ guc_create_virtual(struct intel_engine_cs **siblings, unsigned int count, ve->base.flags = I915_ENGINE_IS_VIRTUAL; + BUILD_BUG_ON(ilog2(VIRTUAL_ENGINES) < I915_NUM_ENGINES); + ve->base.mask = VIRTUAL_ENGINES; + intel_context_init(&ve->context, &ve->base); for (n = 0; n < count; n++) { diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 4ec85308379a..094fca9b0e73 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -49,22 +49,6 @@ static bool enable_out_of_sync = false; static int preallocated_oos_pages = 8192; -static bool intel_gvt_is_valid_gfn(struct intel_vgpu *vgpu, unsigned long gfn) -{ - struct kvm *kvm = vgpu->vfio_device.kvm; - int idx; - bool ret; - - if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status)) - return false; - - idx = srcu_read_lock(&kvm->srcu); - ret = kvm_is_visible_gfn(kvm, gfn); - srcu_read_unlock(&kvm->srcu, idx); - - return ret; -} - /* * validate a gm address and related range size, * translate it to host gm address @@ -1161,31 +1145,6 @@ static inline void ppgtt_generate_shadow_entry(struct intel_gvt_gtt_entry *se, ops->set_pfn(se, s->shadow_page.mfn); } -/* - * Check if can do 2M page - * @vgpu: target vgpu - * @entry: target pfn's gtt entry - * - * Return 1 if 2MB huge gtt shadowing is possible, 0 if miscondition, - * negative if found err. - */ -static int is_2MB_gtt_possible(struct intel_vgpu *vgpu, - struct intel_gvt_gtt_entry *entry) -{ - const struct intel_gvt_gtt_pte_ops *ops = vgpu->gvt->gtt.pte_ops; - kvm_pfn_t pfn; - - if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M)) - return 0; - - if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status)) - return -EINVAL; - pfn = gfn_to_pfn(vgpu->vfio_device.kvm, ops->get_pfn(entry)); - if (is_error_noslot_pfn(pfn)) - return -EINVAL; - return PageTransHuge(pfn_to_page(pfn)); -} - static int split_2MB_gtt_entry(struct intel_vgpu *vgpu, struct intel_vgpu_ppgtt_spt *spt, unsigned long index, struct intel_gvt_gtt_entry *se) @@ -1279,7 +1238,7 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu, { const struct intel_gvt_gtt_pte_ops *pte_ops = vgpu->gvt->gtt.pte_ops; struct intel_gvt_gtt_entry se = *ge; - unsigned long gfn, page_size = PAGE_SIZE; + unsigned long gfn; dma_addr_t dma_addr; int ret; @@ -1291,6 +1250,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu, switch (ge->type) { case GTT_TYPE_PPGTT_PTE_4K_ENTRY: gvt_vdbg_mm("shadow 4K gtt entry\n"); + ret = intel_gvt_dma_map_guest_page(vgpu, gfn, PAGE_SIZE, &dma_addr); + if (ret) + return -ENXIO; break; case GTT_TYPE_PPGTT_PTE_64K_ENTRY: gvt_vdbg_mm("shadow 64K gtt entry\n"); @@ -1302,25 +1264,20 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu, return split_64KB_gtt_entry(vgpu, spt, index, &se); case GTT_TYPE_PPGTT_PTE_2M_ENTRY: gvt_vdbg_mm("shadow 2M gtt entry\n"); - ret = is_2MB_gtt_possible(vgpu, ge); - if (ret == 0) + if (!HAS_PAGE_SIZES(vgpu->gvt->gt->i915, I915_GTT_PAGE_SIZE_2M) || + intel_gvt_dma_map_guest_page(vgpu, gfn, + I915_GTT_PAGE_SIZE_2M, &dma_addr)) return split_2MB_gtt_entry(vgpu, spt, index, &se); - else if (ret < 0) - return ret; - page_size = I915_GTT_PAGE_SIZE_2M; break; case GTT_TYPE_PPGTT_PTE_1G_ENTRY: gvt_vgpu_err("GVT doesn't support 1GB entry\n"); return -EINVAL; default: GEM_BUG_ON(1); + return -EINVAL; } - /* direct shadow */ - ret = intel_gvt_dma_map_guest_page(vgpu, gfn, page_size, &dma_addr); - if (ret) - return -ENXIO; - + /* Successfully shadowed a 4K or 2M page (without splitting). */ pte_ops->set_pfn(&se, dma_addr >> PAGE_SHIFT); ppgtt_set_shadow_entry(spt, &se, index); return 0; @@ -1329,11 +1286,9 @@ static int ppgtt_populate_shadow_entry(struct intel_vgpu *vgpu, static int ppgtt_populate_spt(struct intel_vgpu_ppgtt_spt *spt) { struct intel_vgpu *vgpu = spt->vgpu; - struct intel_gvt *gvt = vgpu->gvt; - const struct intel_gvt_gtt_pte_ops *ops = gvt->gtt.pte_ops; struct intel_vgpu_ppgtt_spt *s; struct intel_gvt_gtt_entry se, ge; - unsigned long gfn, i; + unsigned long i; int ret; trace_spt_change(spt->vgpu->id, "born", spt, @@ -1350,13 +1305,6 @@ static int ppgtt_populate_spt(struct intel_vgpu_ppgtt_spt *spt) ppgtt_generate_shadow_entry(&se, s, &ge); ppgtt_set_shadow_entry(spt, &se, i); } else { - gfn = ops->get_pfn(&ge); - if (!intel_gvt_is_valid_gfn(vgpu, gfn)) { - ops->set_pfn(&se, gvt->gtt.scratch_mfn); - ppgtt_set_shadow_entry(spt, &se, i); - continue; - } - ret = ppgtt_populate_shadow_entry(vgpu, spt, i, &ge); if (ret) goto fail; @@ -1845,6 +1793,9 @@ static int shadow_ppgtt_mm(struct intel_vgpu_mm *mm) if (mm->ppgtt_mm.shadowed) return 0; + if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status)) + return -EINVAL; + mm->ppgtt_mm.shadowed = true; for (index = 0; index < ARRAY_SIZE(mm->ppgtt_mm.guest_pdps); index++) { @@ -2331,14 +2282,6 @@ static int emulate_ggtt_mmio_write(struct intel_vgpu *vgpu, unsigned int off, m.val64 = e.val64; m.type = e.type; - /* one PTE update may be issued in multiple writes and the - * first write may not construct a valid gfn - */ - if (!intel_gvt_is_valid_gfn(vgpu, gfn)) { - ops->set_pfn(&m, gvt->gtt.scratch_mfn); - goto out; - } - ret = intel_gvt_dma_map_guest_page(vgpu, gfn, PAGE_SIZE, &dma_addr); if (ret) { @@ -2355,7 +2298,6 @@ static int emulate_ggtt_mmio_write(struct intel_vgpu *vgpu, unsigned int off, ops->clear_present(&m); } -out: ggtt_set_guest_entry(ggtt_mm, &e, g_gtt_index); ggtt_get_host_entry(ggtt_mm, &e, g_gtt_index); @@ -2875,24 +2817,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool invalidate_old) ggtt_invalidate(gvt->gt); } -/** - * intel_vgpu_reset_gtt - reset the all GTT related status - * @vgpu: a vGPU - * - * This function is called from vfio core to reset reset all - * GTT related status, including GGTT, PPGTT, scratch page. - * - */ -void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu) -{ - /* Shadow pages are only created when there is no page - * table tracking data, so remove page tracking data after - * removing the shadow pages. - */ - intel_vgpu_destroy_all_ppgtt_mm(vgpu); - intel_vgpu_reset_ggtt(vgpu, true); -} - /** * intel_gvt_restore_ggtt - restore all vGPU's ggtt entries * @gvt: intel gvt device diff --git a/drivers/gpu/drm/i915/gvt/gtt.h b/drivers/gpu/drm/i915/gvt/gtt.h index a3b0f59ec8bd..4cb183e06e95 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.h +++ b/drivers/gpu/drm/i915/gvt/gtt.h @@ -224,7 +224,6 @@ void intel_vgpu_reset_ggtt(struct intel_vgpu *vgpu, bool invalidate_old); void intel_vgpu_invalidate_ppgtt(struct intel_vgpu *vgpu); int intel_gvt_init_gtt(struct intel_gvt *gvt); -void intel_vgpu_reset_gtt(struct intel_vgpu *vgpu); void intel_gvt_clean_gtt(struct intel_gvt *gvt); struct intel_vgpu_mm *intel_gvt_find_ppgtt_mm(struct intel_vgpu *vgpu, diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h index 2d65800d8e93..53a0a42a50db 100644 --- a/drivers/gpu/drm/i915/gvt/gvt.h +++ b/drivers/gpu/drm/i915/gvt/gvt.h @@ -34,10 +34,11 @@ #define _GVT_H_ #include -#include #include #include +#include + #include "i915_drv.h" #include "intel_gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c index 9cd9e9da60dd..42ce20e72db7 100644 --- a/drivers/gpu/drm/i915/gvt/kvmgt.c +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c @@ -106,12 +106,10 @@ struct gvt_dma { #define vfio_dev_to_vgpu(vfio_dev) \ container_of((vfio_dev), struct intel_vgpu, vfio_device) -static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, - const u8 *val, int len, - struct kvm_page_track_notifier_node *node); -static void kvmgt_page_track_flush_slot(struct kvm *kvm, - struct kvm_memory_slot *slot, - struct kvm_page_track_notifier_node *node); +static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len, + struct kvm_page_track_notifier_node *node); +static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages, + struct kvm_page_track_notifier_node *node); static ssize_t intel_vgpu_show_description(struct mdev_type *mtype, char *buf) { @@ -161,8 +159,7 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, unsigned long gfn, if (npage == 0) base_page = cur_page; - else if (base_page + npage != cur_page) { - gvt_vgpu_err("The pages are not continuous\n"); + else if (page_to_pfn(base_page) + npage != page_to_pfn(cur_page)) { ret = -EINVAL; npage++; goto err; @@ -172,7 +169,8 @@ static int gvt_pin_guest_page(struct intel_vgpu *vgpu, unsigned long gfn, *page = base_page; return 0; err: - gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE); + if (npage) + gvt_unpin_guest_page(vgpu, gfn, npage * PAGE_SIZE); return ret; } @@ -352,6 +350,8 @@ __kvmgt_protect_table_find(struct intel_vgpu *info, gfn_t gfn) { struct kvmgt_pgfn *p, *res = NULL; + lockdep_assert_held(&info->vgpu_lock); + hash_for_each_possible(info->ptable, p, hnode, gfn) { if (gfn == p->gfn) { res = p; @@ -654,21 +654,19 @@ out: static int intel_vgpu_open_device(struct vfio_device *vfio_dev) { struct intel_vgpu *vgpu = vfio_dev_to_vgpu(vfio_dev); - - if (!vgpu->vfio_device.kvm || - vgpu->vfio_device.kvm->mm != current->mm) { - gvt_vgpu_err("KVM is required to use Intel vGPU\n"); - return -ESRCH; - } + int ret; if (__kvmgt_vgpu_exist(vgpu)) return -EEXIST; vgpu->track_node.track_write = kvmgt_page_track_write; - vgpu->track_node.track_flush_slot = kvmgt_page_track_flush_slot; - kvm_get_kvm(vgpu->vfio_device.kvm); - kvm_page_track_register_notifier(vgpu->vfio_device.kvm, - &vgpu->track_node); + vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region; + ret = kvm_page_track_register_notifier(vgpu->vfio_device.kvm, + &vgpu->track_node); + if (ret) { + gvt_vgpu_err("KVM is required to use Intel vGPU\n"); + return ret; + } set_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status); @@ -703,7 +701,6 @@ static void intel_vgpu_close_device(struct vfio_device *vfio_dev) kvm_page_track_unregister_notifier(vgpu->vfio_device.kvm, &vgpu->track_node); - kvm_put_kvm(vgpu->vfio_device.kvm); kvmgt_protect_table_destroy(vgpu); gvt_cache_destroy(vgpu); @@ -1547,95 +1544,70 @@ static struct mdev_driver intel_vgpu_mdev_driver = { int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn) { - struct kvm *kvm = info->vfio_device.kvm; - struct kvm_memory_slot *slot; - int idx; + int r; if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status)) return -ESRCH; - idx = srcu_read_lock(&kvm->srcu); - slot = gfn_to_memslot(kvm, gfn); - if (!slot) { - srcu_read_unlock(&kvm->srcu, idx); - return -EINVAL; - } - - write_lock(&kvm->mmu_lock); - if (kvmgt_gfn_is_write_protected(info, gfn)) - goto out; + return 0; + + r = kvm_write_track_add_gfn(info->vfio_device.kvm, gfn); + if (r) + return r; - kvm_slot_page_track_add_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE); kvmgt_protect_table_add(info, gfn); - -out: - write_unlock(&kvm->mmu_lock); - srcu_read_unlock(&kvm->srcu, idx); return 0; } int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn) { - struct kvm *kvm = info->vfio_device.kvm; - struct kvm_memory_slot *slot; - int idx; + int r; if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, info->status)) return -ESRCH; - idx = srcu_read_lock(&kvm->srcu); - slot = gfn_to_memslot(kvm, gfn); - if (!slot) { - srcu_read_unlock(&kvm->srcu, idx); - return -EINVAL; - } - - write_lock(&kvm->mmu_lock); - if (!kvmgt_gfn_is_write_protected(info, gfn)) - goto out; + return 0; + + r = kvm_write_track_remove_gfn(info->vfio_device.kvm, gfn); + if (r) + return r; - kvm_slot_page_track_remove_page(kvm, slot, gfn, KVM_PAGE_TRACK_WRITE); kvmgt_protect_table_del(info, gfn); - -out: - write_unlock(&kvm->mmu_lock); - srcu_read_unlock(&kvm->srcu, idx); return 0; } -static void kvmgt_page_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, - const u8 *val, int len, - struct kvm_page_track_notifier_node *node) +static void kvmgt_page_track_write(gpa_t gpa, const u8 *val, int len, + struct kvm_page_track_notifier_node *node) { struct intel_vgpu *info = container_of(node, struct intel_vgpu, track_node); - if (kvmgt_gfn_is_write_protected(info, gpa_to_gfn(gpa))) + mutex_lock(&info->vgpu_lock); + + if (kvmgt_gfn_is_write_protected(info, gpa >> PAGE_SHIFT)) intel_vgpu_page_track_handler(info, gpa, (void *)val, len); + + mutex_unlock(&info->vgpu_lock); } -static void kvmgt_page_track_flush_slot(struct kvm *kvm, - struct kvm_memory_slot *slot, - struct kvm_page_track_notifier_node *node) +static void kvmgt_page_track_remove_region(gfn_t gfn, unsigned long nr_pages, + struct kvm_page_track_notifier_node *node) { - int i; - gfn_t gfn; + unsigned long i; struct intel_vgpu *info = container_of(node, struct intel_vgpu, track_node); - write_lock(&kvm->mmu_lock); - for (i = 0; i < slot->npages; i++) { - gfn = slot->base_gfn + i; - if (kvmgt_gfn_is_write_protected(info, gfn)) { - kvm_slot_page_track_remove_page(kvm, slot, gfn, - KVM_PAGE_TRACK_WRITE); - kvmgt_protect_table_del(info, gfn); - } + mutex_lock(&info->vgpu_lock); + + for (i = 0; i < nr_pages; i++) { + if (kvmgt_gfn_is_write_protected(info, gfn + i)) + kvmgt_protect_table_del(info, gfn + i); } - write_unlock(&kvm->mmu_lock); + + mutex_unlock(&info->vgpu_lock); } void intel_vgpu_detach_regions(struct intel_vgpu *vgpu) diff --git a/drivers/gpu/drm/i915/gvt/page_track.c b/drivers/gpu/drm/i915/gvt/page_track.c index df34e73cba41..60a65435556d 100644 --- a/drivers/gpu/drm/i915/gvt/page_track.c +++ b/drivers/gpu/drm/i915/gvt/page_track.c @@ -162,13 +162,9 @@ int intel_vgpu_page_track_handler(struct intel_vgpu *vgpu, u64 gpa, struct intel_vgpu_page_track *page_track; int ret = 0; - mutex_lock(&vgpu->vgpu_lock); - page_track = intel_vgpu_find_page_track(vgpu, gpa >> PAGE_SHIFT); - if (!page_track) { - ret = -ENXIO; - goto out; - } + if (!page_track) + return -ENXIO; if (unlikely(vgpu->failsafe)) { /* Remove write protection to prevent furture traps. */ @@ -179,7 +175,5 @@ int intel_vgpu_page_track_handler(struct intel_vgpu *vgpu, u64 gpa, gvt_err("guest page write error, gpa %llx\n", gpa); } -out: - mutex_unlock(&vgpu->vgpu_lock); return ret; } diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c index 7c7da284990d..f59081066a19 100644 --- a/drivers/gpu/drm/i915/i915_request.c +++ b/drivers/gpu/drm/i915/i915_request.c @@ -134,9 +134,7 @@ static void i915_fence_release(struct dma_fence *fence) i915_sw_fence_fini(&rq->semaphore); /* - * Keep one request on each engine for reserved use under mempressure - * do not use with virtual engines as this really is only needed for - * kernel contexts. + * Keep one request on each engine for reserved use under mempressure. * * We do not hold a reference to the engine here and so have to be * very careful in what rq->engine we poke. The virtual engine is @@ -166,8 +164,7 @@ static void i915_fence_release(struct dma_fence *fence) * know that if the rq->execution_mask is a single bit, rq->engine * can be a physical engine with the exact corresponding mask. */ - if (!intel_engine_is_virtual(rq->engine) && - is_power_of_2(rq->execution_mask) && + if (is_power_of_2(rq->execution_mask) && !cmpxchg(&rq->engine->request_pool, NULL, rq)) return; diff --git a/drivers/gpu/drm/nouveau/dispnv04/crtc.c b/drivers/gpu/drm/nouveau/dispnv04/crtc.c index a34924523133..a34917b048f9 100644 --- a/drivers/gpu/drm/nouveau/dispnv04/crtc.c +++ b/drivers/gpu/drm/nouveau/dispnv04/crtc.c @@ -1122,18 +1122,11 @@ nv04_page_flip_emit(struct nouveau_channel *chan, PUSH_NVSQ(push, NV_SW, NV_SW_PAGE_FLIP, 0x00000000); PUSH_KICK(push); - ret = nouveau_fence_new(pfence); + ret = nouveau_fence_new(pfence, chan); if (ret) goto fail; - ret = nouveau_fence_emit(*pfence, chan); - if (ret) - goto fail_fence_unref; - return 0; - -fail_fence_unref: - nouveau_fence_unref(pfence); fail: spin_lock_irqsave(&dev->event_lock, flags); list_del(&s->head); diff --git a/drivers/gpu/drm/nouveau/nouveau_bo.c b/drivers/gpu/drm/nouveau/nouveau_bo.c index 19cab37ac69c..0f3bd187ede6 100644 --- a/drivers/gpu/drm/nouveau/nouveau_bo.c +++ b/drivers/gpu/drm/nouveau/nouveau_bo.c @@ -875,16 +875,10 @@ nouveau_bo_move_m2mf(struct ttm_buffer_object *bo, int evict, if (ret) goto out_unlock; - ret = nouveau_fence_new(&fence); + ret = nouveau_fence_new(&fence, chan); if (ret) goto out_unlock; - ret = nouveau_fence_emit(fence, chan); - if (ret) { - nouveau_fence_unref(&fence); - goto out_unlock; - } - /* TODO: figure out a better solution here * * wait on the fence here explicitly as going through diff --git a/drivers/gpu/drm/nouveau/nouveau_chan.c b/drivers/gpu/drm/nouveau/nouveau_chan.c index 1fd5ccf41128..bb3d6e5c122f 100644 --- a/drivers/gpu/drm/nouveau/nouveau_chan.c +++ b/drivers/gpu/drm/nouveau/nouveau_chan.c @@ -70,11 +70,9 @@ nouveau_channel_idle(struct nouveau_channel *chan) struct nouveau_fence *fence = NULL; int ret; - ret = nouveau_fence_new(&fence); + ret = nouveau_fence_new(&fence, chan); if (!ret) { - ret = nouveau_fence_emit(fence, chan); - if (!ret) - ret = nouveau_fence_wait(fence, false, false); + ret = nouveau_fence_wait(fence, false, false); nouveau_fence_unref(&fence); } diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c index 61e84562094a..12feecf71e75 100644 --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c @@ -209,8 +209,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) goto done; } - if (!nouveau_fence_new(&fence)) - nouveau_fence_emit(fence, dmem->migrate.chan); + nouveau_fence_new(&fence, dmem->migrate.chan); migrate_vma_pages(&args); nouveau_dmem_fence_done(&fence); dma_unmap_page(drm->dev->dev, dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL); @@ -403,8 +402,7 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk) } } - if (!nouveau_fence_new(&fence)) - nouveau_fence_emit(fence, chunk->drm->dmem->migrate.chan); + nouveau_fence_new(&fence, chunk->drm->dmem->migrate.chan); migrate_device_pages(src_pfns, dst_pfns, npages); nouveau_dmem_fence_done(&fence); migrate_device_finalize(src_pfns, dst_pfns, npages); @@ -677,8 +675,7 @@ static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm, addr += PAGE_SIZE; } - if (!nouveau_fence_new(&fence)) - nouveau_fence_emit(fence, drm->dmem->migrate.chan); + nouveau_fence_new(&fence, drm->dmem->migrate.chan); migrate_vma_pages(args); nouveau_dmem_fence_done(&fence); nouveau_pfns_map(svmm, args->vma->vm_mm, args->start, pfns, i); diff --git a/drivers/gpu/drm/nouveau/nouveau_exec.c b/drivers/gpu/drm/nouveau/nouveau_exec.c index a90c4cd8cbb2..19024ce21fbb 100644 --- a/drivers/gpu/drm/nouveau/nouveau_exec.c +++ b/drivers/gpu/drm/nouveau/nouveau_exec.c @@ -96,7 +96,8 @@ nouveau_exec_job_submit(struct nouveau_job *job) unsigned long index; int ret; - ret = nouveau_fence_new(&exec_job->fence); + /* Create a new fence, but do not emit yet. */ + ret = nouveau_fence_create(&exec_job->fence, exec_job->chan); if (ret) return ret; @@ -170,13 +171,17 @@ nouveau_exec_job_run(struct nouveau_job *job) nv50_dma_push(chan, p->va, p->va_len, no_prefetch); } - ret = nouveau_fence_emit(fence, chan); + ret = nouveau_fence_emit(fence); if (ret) { + nouveau_fence_unref(&exec_job->fence); NV_PRINTK(err, job->cli, "error fencing pushbuf: %d\n", ret); WIND_RING(chan); return ERR_PTR(ret); } + /* The fence was emitted successfully, set the job's fence pointer to + * NULL in order to avoid freeing it up when the job is cleaned up. + */ exec_job->fence = NULL; return &fence->base; @@ -189,7 +194,7 @@ nouveau_exec_job_free(struct nouveau_job *job) nouveau_job_free(job); - nouveau_fence_unref(&exec_job->fence); + kfree(exec_job->fence); kfree(exec_job->push.s); kfree(exec_job); } diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c b/drivers/gpu/drm/nouveau/nouveau_fence.c index 77c739a55b19..61d9e70da9fd 100644 --- a/drivers/gpu/drm/nouveau/nouveau_fence.c +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c @@ -205,16 +205,13 @@ nouveau_fence_context_new(struct nouveau_channel *chan, struct nouveau_fence_cha } int -nouveau_fence_emit(struct nouveau_fence *fence, struct nouveau_channel *chan) +nouveau_fence_emit(struct nouveau_fence *fence) { + struct nouveau_channel *chan = fence->channel; struct nouveau_fence_chan *fctx = chan->fence; struct nouveau_fence_priv *priv = (void*)chan->drm->fence; int ret; - if (unlikely(!chan->fence)) - return -ENODEV; - - fence->channel = chan; fence->timeout = jiffies + (15 * HZ); if (priv->uevent) @@ -406,18 +403,41 @@ nouveau_fence_unref(struct nouveau_fence **pfence) } int -nouveau_fence_new(struct nouveau_fence **pfence) +nouveau_fence_create(struct nouveau_fence **pfence, + struct nouveau_channel *chan) { struct nouveau_fence *fence; + if (unlikely(!chan->fence)) + return -ENODEV; + fence = kzalloc(sizeof(*fence), GFP_KERNEL); if (!fence) return -ENOMEM; + fence->channel = chan; + *pfence = fence; return 0; } +int +nouveau_fence_new(struct nouveau_fence **pfence, + struct nouveau_channel *chan) +{ + int ret = 0; + + ret = nouveau_fence_create(pfence, chan); + if (ret) + return ret; + + ret = nouveau_fence_emit(*pfence); + if (ret) + nouveau_fence_unref(pfence); + + return ret; +} + static const char *nouveau_fence_get_get_driver_name(struct dma_fence *fence) { return "nouveau"; diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.h b/drivers/gpu/drm/nouveau/nouveau_fence.h index 2c72d96ef17d..64d33ae7f356 100644 --- a/drivers/gpu/drm/nouveau/nouveau_fence.h +++ b/drivers/gpu/drm/nouveau/nouveau_fence.h @@ -17,10 +17,11 @@ struct nouveau_fence { unsigned long timeout; }; -int nouveau_fence_new(struct nouveau_fence **); +int nouveau_fence_create(struct nouveau_fence **, struct nouveau_channel *); +int nouveau_fence_new(struct nouveau_fence **, struct nouveau_channel *); void nouveau_fence_unref(struct nouveau_fence **); -int nouveau_fence_emit(struct nouveau_fence *, struct nouveau_channel *); +int nouveau_fence_emit(struct nouveau_fence *); bool nouveau_fence_done(struct nouveau_fence *); int nouveau_fence_wait(struct nouveau_fence *, bool lazy, bool intr); int nouveau_fence_sync(struct nouveau_bo *, struct nouveau_channel *, bool exclusive, bool intr); diff --git a/drivers/gpu/drm/nouveau/nouveau_gem.c b/drivers/gpu/drm/nouveau/nouveau_gem.c index c0b10d8d3d03..a0d303e5ce3d 100644 --- a/drivers/gpu/drm/nouveau/nouveau_gem.c +++ b/drivers/gpu/drm/nouveau/nouveau_gem.c @@ -914,11 +914,8 @@ revalidate: } } - ret = nouveau_fence_new(&fence); - if (!ret) - ret = nouveau_fence_emit(fence, chan); + ret = nouveau_fence_new(&fence, chan); if (ret) { - nouveau_fence_unref(&fence); NV_PRINTK(err, cli, "error fencing pushbuf: %d\n", ret); WIND_RING(chan); goto out; diff --git a/drivers/i3c/master.c b/drivers/i3c/master.c index 08aeb69a7800..87283e4a4607 100644 --- a/drivers/i3c/master.c +++ b/drivers/i3c/master.c @@ -1308,7 +1308,11 @@ static int i3c_master_get_i3c_addrs(struct i3c_dev_desc *dev) if (dev->info.static_addr) { status = i3c_bus_get_addr_slot_status(&master->bus, dev->info.static_addr); - if (status != I3C_ADDR_SLOT_FREE) + /* Since static address and assigned dynamic address can be + * equal, allow this case to pass. + */ + if (status != I3C_ADDR_SLOT_FREE && + dev->info.static_addr != dev->boardinfo->init_dyn_addr) return -EBUSY; i3c_bus_set_addr_slot_status(&master->bus, diff --git a/drivers/i3c/master/ast2600-i3c-master.c b/drivers/i3c/master/ast2600-i3c-master.c index 09ed19d489e9..01a47d3dd499 100644 --- a/drivers/i3c/master/ast2600-i3c-master.c +++ b/drivers/i3c/master/ast2600-i3c-master.c @@ -8,7 +8,6 @@ #include #include #include -#include #include #include diff --git a/drivers/i3c/master/i3c-master-cdns.c b/drivers/i3c/master/i3c-master-cdns.c index 01610fa5b0cc..49551db71bc9 100644 --- a/drivers/i3c/master/i3c-master-cdns.c +++ b/drivers/i3c/master/i3c-master-cdns.c @@ -22,7 +22,6 @@ #include #include #include -#include #define DEV_ID 0x0 #define DEV_ID_I3C_MASTER 0x5034 diff --git a/drivers/i3c/master/mipi-i3c-hci/cmd_v1.c b/drivers/i3c/master/mipi-i3c-hci/cmd_v1.c index d97c3175e0e2..6a781f89b0e4 100644 --- a/drivers/i3c/master/mipi-i3c-hci/cmd_v1.c +++ b/drivers/i3c/master/mipi-i3c-hci/cmd_v1.c @@ -339,7 +339,7 @@ static int hci_cmd_v1_daa(struct i3c_hci *hci) break; } if (RESP_STATUS(xfer[0].response) == RESP_ERR_NACK && - RESP_STATUS(xfer[0].response) == 1) { + RESP_DATA_LENGTH(xfer->response) == 1) { ret = 0; /* no more devices to be assigned */ break; } diff --git a/drivers/i3c/master/svc-i3c-master.c b/drivers/i3c/master/svc-i3c-master.c index 0d63b732ef0c..8f8295acdadb 100644 --- a/drivers/i3c/master/svc-i3c-master.c +++ b/drivers/i3c/master/svc-i3c-master.c @@ -156,6 +156,7 @@ struct svc_i3c_regs_save { * @base: I3C master controller * @dev: Corresponding device * @regs: Memory mapping + * @saved_regs: Volatile values for PM operations * @free_slots: Bit array of available slots * @addrs: Array containing the dynamic addresses of each attached device * @descs: Array of descriptors, one per attached device @@ -789,6 +790,10 @@ static int svc_i3c_master_do_daa_locked(struct svc_i3c_master *master, */ break; } else if (SVC_I3C_MSTATUS_NACKED(reg)) { + /* No I3C devices attached */ + if (dev_nb == 0) + break; + /* * A slave device nacked the address, this is * allowed only once, DAA will be stopped and @@ -1263,11 +1268,17 @@ static int svc_i3c_master_send_ccc_cmd(struct i3c_master_controller *m, { struct svc_i3c_master *master = to_svc_i3c_master(m); bool broadcast = cmd->id < 0x80; + int ret; if (broadcast) - return svc_i3c_master_send_bdcast_ccc_cmd(master, cmd); + ret = svc_i3c_master_send_bdcast_ccc_cmd(master, cmd); else - return svc_i3c_master_send_direct_ccc_cmd(master, cmd); + ret = svc_i3c_master_send_direct_ccc_cmd(master, cmd); + + if (ret) + cmd->err = I3C_ERROR_M2; + + return ret; } static int svc_i3c_master_priv_xfers(struct i3c_dev_desc *dev, @@ -1518,8 +1529,8 @@ static int svc_i3c_master_probe(struct platform_device *pdev) return PTR_ERR(master->sclk); master->irq = platform_get_irq(pdev, 0); - if (master->irq <= 0) - return -ENOENT; + if (master->irq < 0) + return master->irq; master->dev = dev; diff --git a/drivers/input/gameport/Kconfig b/drivers/input/gameport/Kconfig index 5a2c2fb3217d..fe73b26e647a 100644 --- a/drivers/input/gameport/Kconfig +++ b/drivers/input/gameport/Kconfig @@ -25,6 +25,7 @@ if GAMEPORT config GAMEPORT_NS558 tristate "Classic ISA and PnP gameport support" + depends on ISA help Say Y here if you have an ISA or PnP gameport. @@ -35,6 +36,7 @@ config GAMEPORT_NS558 config GAMEPORT_L4 tristate "PDPI Lightning 4 gamecard support" + depends on ISA help Say Y here if you have a PDPI Lightning 4 gamecard. @@ -53,7 +55,7 @@ config GAMEPORT_EMU10K1 config GAMEPORT_FM801 tristate "ForteMedia FM801 gameport support" - depends on PCI + depends on PCI && HAS_IOPORT help Say Y here if you have ForteMedia FM801 PCI audio controller (Abit AU10, Genius Sound Maker, HP Workstation zx2000, diff --git a/drivers/input/gameport/gameport.c b/drivers/input/gameport/gameport.c index a1443320b419..34f416a3ebcb 100644 --- a/drivers/input/gameport/gameport.c +++ b/drivers/input/gameport/gameport.c @@ -519,12 +519,32 @@ EXPORT_SYMBOL(gameport_set_phys); static void gameport_default_trigger(struct gameport *gameport) { +#ifdef CONFIG_HAS_IOPORT outb(0xff, gameport->io); +#endif } static unsigned char gameport_default_read(struct gameport *gameport) { +#ifdef CONFIG_HAS_IOPORT return inb(gameport->io); +#else + return 0xff; +#endif +} + +static void gameport_setup_default_handlers(struct gameport *gameport) +{ + if ((!gameport->trigger || !gameport->read) && + !IS_ENABLED(CONFIG_HAS_IOPORT)) + dev_err(&gameport->dev, + "I/O port access is required for %s (%s) but is not available\n", + gameport->phys, gameport->name); + + if (!gameport->trigger) + gameport->trigger = gameport_default_trigger; + if (!gameport->read) + gameport->read = gameport_default_read; } /* @@ -545,11 +565,7 @@ static void gameport_init_port(struct gameport *gameport) if (gameport->parent) gameport->dev.parent = &gameport->parent->dev; - if (!gameport->trigger) - gameport->trigger = gameport_default_trigger; - if (!gameport->read) - gameport->read = gameport_default_read; - + gameport_setup_default_handlers(gameport); INIT_LIST_HEAD(&gameport->node); spin_lock_init(&gameport->timer_lock); timer_setup(&gameport->poll_timer, gameport_run_poll_handler, 0); diff --git a/drivers/input/joystick/xpad.c b/drivers/input/joystick/xpad.c index cdb193317c3b..ede380551e55 100644 --- a/drivers/input/joystick/xpad.c +++ b/drivers/input/joystick/xpad.c @@ -264,6 +264,7 @@ static const struct xpad_device { { 0x0f0d, 0x0067, "HORIPAD ONE", 0, XTYPE_XBOXONE }, { 0x0f0d, 0x0078, "Hori Real Arcade Pro V Kai Xbox One", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOXONE }, { 0x0f0d, 0x00c5, "Hori Fighting Commander ONE", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOXONE }, + { 0x0f0d, 0x00dc, "HORIPAD FPS for Nintendo Switch", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 }, { 0x0f30, 0x010b, "Philips Recoil", 0, XTYPE_XBOX }, { 0x0f30, 0x0202, "Joytech Advanced Controller", 0, XTYPE_XBOX }, { 0x0f30, 0x8888, "BigBen XBMiniPad Controller", 0, XTYPE_XBOX }, @@ -365,6 +366,7 @@ static const struct xpad_device { { 0x31e3, 0x1300, "Wooting 60HE (AVR)", 0, XTYPE_XBOX360 }, { 0x31e3, 0x1310, "Wooting 60HE (ARM)", 0, XTYPE_XBOX360 }, { 0x3285, 0x0607, "Nacon GC-100", 0, XTYPE_XBOX360 }, + { 0x3537, 0x1004, "GameSir T4 Kaleid", 0, XTYPE_XBOX360 }, { 0x3767, 0x0101, "Fanatec Speedster 3 Forceshock Wheel", 0, XTYPE_XBOX }, { 0xffff, 0xffff, "Chinese-made Xbox Controller", 0, XTYPE_XBOX }, { 0x0000, 0x0000, "Generic X-Box pad", 0, XTYPE_UNKNOWN } @@ -499,6 +501,8 @@ static const struct usb_device_id xpad_table[] = { XPAD_XBOX360_VENDOR(0x2f24), /* GameSir controllers */ XPAD_XBOX360_VENDOR(0x31e3), /* Wooting Keyboards */ XPAD_XBOX360_VENDOR(0x3285), /* Nacon GC-100 */ + XPAD_XBOX360_VENDOR(0x3537), /* GameSir Controllers */ + XPAD_XBOXONE_VENDOR(0x3537), /* GameSir Controllers */ { } }; @@ -1720,6 +1724,27 @@ static int xpad_start_input(struct usb_xpad *xpad) return error; } } + if (xpad->xtype == XTYPE_XBOX360) { + /* + * Some third-party controllers Xbox 360-style controllers + * require this message to finish initialization. + */ + u8 dummy[20]; + + error = usb_control_msg_recv(xpad->udev, 0, + /* bRequest */ 0x01, + /* bmRequestType */ + USB_TYPE_VENDOR | USB_DIR_IN | + USB_RECIP_INTERFACE, + /* wValue */ 0x100, + /* wIndex */ 0x00, + dummy, sizeof(dummy), + 25, GFP_KERNEL); + if (error) + dev_warn(&xpad->dev->dev, + "unable to receive magic message: %d\n", + error); + } return 0; } diff --git a/drivers/input/keyboard/adp5588-keys.c b/drivers/input/keyboard/adp5588-keys.c index 896a5a989ddc..61e8e43e9c2b 100644 --- a/drivers/input/keyboard/adp5588-keys.c +++ b/drivers/input/keyboard/adp5588-keys.c @@ -713,17 +713,11 @@ static int adp5588_fw_parse(struct adp5588_kpad *kpad) return 0; } -static void adp5588_disable_regulator(void *reg) -{ - regulator_disable(reg); -} - static int adp5588_probe(struct i2c_client *client) { struct adp5588_kpad *kpad; struct input_dev *input; struct gpio_desc *gpio; - struct regulator *vcc; unsigned int revid; int ret; int error; @@ -749,16 +743,7 @@ static int adp5588_probe(struct i2c_client *client) if (error) return error; - vcc = devm_regulator_get(&client->dev, "vcc"); - if (IS_ERR(vcc)) - return PTR_ERR(vcc); - - error = regulator_enable(vcc); - if (error) - return error; - - error = devm_add_action_or_reset(&client->dev, - adp5588_disable_regulator, vcc); + error = devm_regulator_get_enable(&client->dev, "vcc"); if (error) return error; diff --git a/drivers/input/keyboard/amikbd.c b/drivers/input/keyboard/amikbd.c index a20a4e186639..e305c44cd0aa 100644 --- a/drivers/input/keyboard/amikbd.c +++ b/drivers/input/keyboard/amikbd.c @@ -196,7 +196,7 @@ static int __init amikbd_probe(struct platform_device *pdev) struct input_dev *dev; int i, err; - dev = input_allocate_device(); + dev = devm_input_allocate_device(&pdev->dev); if (!dev) { dev_err(&pdev->dev, "Not enough memory for input device\n"); return -ENOMEM; @@ -208,7 +208,6 @@ static int __init amikbd_probe(struct platform_device *pdev) dev->id.vendor = 0x0001; dev->id.product = 0x0001; dev->id.version = 0x0100; - dev->dev.parent = &pdev->dev; dev->evbit[0] = BIT_MASK(EV_KEY) | BIT_MASK(EV_REP); @@ -218,35 +217,21 @@ static int __init amikbd_probe(struct platform_device *pdev) amikbd_init_console_keymaps(); ciaa.cra &= ~0x41; /* serial data in, turn off TA */ - err = request_irq(IRQ_AMIGA_CIAA_SP, amikbd_interrupt, 0, "amikbd", - dev); + err = devm_request_irq(&pdev->dev, IRQ_AMIGA_CIAA_SP, amikbd_interrupt, + 0, "amikbd", dev); if (err) - goto fail2; + return err; err = input_register_device(dev); if (err) - goto fail3; + return err; platform_set_drvdata(pdev, dev); return 0; - - fail3: free_irq(IRQ_AMIGA_CIAA_SP, dev); - fail2: input_free_device(dev); - return err; -} - -static int __exit amikbd_remove(struct platform_device *pdev) -{ - struct input_dev *dev = platform_get_drvdata(pdev); - - free_irq(IRQ_AMIGA_CIAA_SP, dev); - input_unregister_device(dev); - return 0; } static struct platform_driver amikbd_driver = { - .remove = __exit_p(amikbd_remove), .driver = { .name = "amiga-keyboard", }, diff --git a/drivers/input/keyboard/bcm-keypad.c b/drivers/input/keyboard/bcm-keypad.c index 56a919ec23b5..f3c3746acd4c 100644 --- a/drivers/input/keyboard/bcm-keypad.c +++ b/drivers/input/keyboard/bcm-keypad.c @@ -307,7 +307,6 @@ static int bcm_kp_probe(struct platform_device *pdev) { struct bcm_kp *kp; struct input_dev *input_dev; - struct resource *res; int error; kp = devm_kzalloc(&pdev->dev, sizeof(*kp), GFP_KERNEL); @@ -353,29 +352,16 @@ static int bcm_kp_probe(struct platform_device *pdev) return error; } - /* Get the KEYPAD base address */ - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (!res) { - dev_err(&pdev->dev, "Missing keypad base address resource\n"); - return -ENODEV; - } - - kp->base = devm_ioremap_resource(&pdev->dev, res); + kp->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(kp->base)) return PTR_ERR(kp->base); /* Enable clock */ - kp->clk = devm_clk_get(&pdev->dev, "peri_clk"); + kp->clk = devm_clk_get_optional(&pdev->dev, "peri_clk"); if (IS_ERR(kp->clk)) { - error = PTR_ERR(kp->clk); - if (error != -ENOENT) { - if (error != -EPROBE_DEFER) - dev_err(&pdev->dev, "Failed to get clock\n"); - return error; - } - dev_dbg(&pdev->dev, - "No clock specified. Assuming it's enabled\n"); - kp->clk = NULL; + return dev_err_probe(&pdev->dev, PTR_ERR(kp->clk), "Failed to get clock\n"); + } else if (!kp->clk) { + dev_dbg(&pdev->dev, "No clock specified. Assuming it's enabled\n"); } else { unsigned int desired_rate; long actual_rate; diff --git a/drivers/input/keyboard/gpio_keys.c b/drivers/input/keyboard/gpio_keys.c index c928829a8b0c..2e7c2c046e67 100644 --- a/drivers/input/keyboard/gpio_keys.c +++ b/drivers/input/keyboard/gpio_keys.c @@ -523,18 +523,15 @@ static int gpio_keys_setup_key(struct platform_device *pdev, NULL, GPIOD_IN, desc); if (IS_ERR(bdata->gpiod)) { error = PTR_ERR(bdata->gpiod); - if (error == -ENOENT) { - /* - * GPIO is optional, we may be dealing with - * purely interrupt-driven setup. - */ - bdata->gpiod = NULL; - } else { - if (error != -EPROBE_DEFER) - dev_err(dev, "failed to get gpio: %d\n", - error); - return error; - } + if (error != -ENOENT) + return dev_err_probe(dev, error, + "failed to get gpio\n"); + + /* + * GPIO is optional, we may be dealing with + * purely interrupt-driven setup. + */ + bdata->gpiod = NULL; } } else if (gpio_is_valid(button->gpio)) { /* diff --git a/drivers/input/keyboard/gpio_keys_polled.c b/drivers/input/keyboard/gpio_keys_polled.c index c3937d2fc744..ba00ecfbd343 100644 --- a/drivers/input/keyboard/gpio_keys_polled.c +++ b/drivers/input/keyboard/gpio_keys_polled.c @@ -299,13 +299,9 @@ static int gpio_keys_polled_probe(struct platform_device *pdev) NULL, GPIOD_IN, button->desc); if (IS_ERR(bdata->gpiod)) { - error = PTR_ERR(bdata->gpiod); - if (error != -EPROBE_DEFER) - dev_err(dev, - "failed to get gpio: %d\n", - error); fwnode_handle_put(child); - return error; + return dev_err_probe(dev, PTR_ERR(bdata->gpiod), + "failed to get gpio\n"); } } else if (gpio_is_valid(button->gpio)) { /* diff --git a/drivers/input/keyboard/lm8323.c b/drivers/input/keyboard/lm8323.c index 3964f6e0f6af..7bee93e9b0f5 100644 --- a/drivers/input/keyboard/lm8323.c +++ b/drivers/input/keyboard/lm8323.c @@ -556,6 +556,7 @@ static int init_pwm(struct lm8323_chip *lm, int id, struct device *dev, const char *name) { struct lm8323_pwm *pwm; + int err; BUG_ON(id > 3); @@ -575,9 +576,11 @@ static int init_pwm(struct lm8323_chip *lm, int id, struct device *dev, pwm->cdev.name = name; pwm->cdev.brightness_set = lm8323_pwm_set_brightness; pwm->cdev.groups = lm8323_pwm_groups; - if (led_classdev_register(dev, &pwm->cdev) < 0) { - dev_err(dev, "couldn't register PWM %d\n", id); - return -1; + + err = devm_led_classdev_register(dev, &pwm->cdev); + if (err) { + dev_err(dev, "couldn't register PWM %d: %d\n", id, err); + return err; } pwm->enabled = true; } @@ -585,8 +588,6 @@ static int init_pwm(struct lm8323_chip *lm, int id, struct device *dev, return 0; } -static struct i2c_driver lm8323_i2c_driver; - static ssize_t lm8323_show_disable(struct device *dev, struct device_attribute *attr, char *buf) { @@ -615,6 +616,12 @@ static ssize_t lm8323_set_disable(struct device *dev, } static DEVICE_ATTR(disable_kp, 0644, lm8323_show_disable, lm8323_set_disable); +static struct attribute *lm8323_attrs[] = { + &dev_attr_disable_kp.attr, + NULL, +}; +ATTRIBUTE_GROUPS(lm8323); + static int lm8323_probe(struct i2c_client *client) { struct lm8323_platform_data *pdata = dev_get_platdata(&client->dev); @@ -642,12 +649,13 @@ static int lm8323_probe(struct i2c_client *client) return -EINVAL; } - lm = kzalloc(sizeof *lm, GFP_KERNEL); - idev = input_allocate_device(); - if (!lm || !idev) { - err = -ENOMEM; - goto fail1; - } + lm = devm_kzalloc(&client->dev, sizeof(*lm), GFP_KERNEL); + if (!lm) + return -ENOMEM; + + idev = devm_input_allocate_device(&client->dev); + if (!idev) + return -ENOMEM; lm->client = client; lm->idev = idev; @@ -663,8 +671,10 @@ static int lm8323_probe(struct i2c_client *client) lm8323_reset(lm); - /* Nothing's set up to service the IRQ yet, so just spin for max. - * 100ms until we can configure. */ + /* + * Nothing's set up to service the IRQ yet, so just spin for max. + * 100ms until we can configure. + */ tmo = jiffies + msecs_to_jiffies(100); while (lm8323_read(lm, LM8323_CMD_READ_INT, data, 1) == 1) { if (data[0] & INT_NOINIT) @@ -684,21 +694,17 @@ static int lm8323_probe(struct i2c_client *client) /* If a true probe check the device */ if (lm8323_read_id(lm, data) != 0) { dev_err(&client->dev, "device not found\n"); - err = -ENODEV; - goto fail1; + return -ENODEV; } for (pwm = 0; pwm < LM8323_NUM_PWMS; pwm++) { err = init_pwm(lm, pwm + 1, &client->dev, pdata->pwm_names[pwm]); - if (err < 0) - goto fail2; + if (err) + return err; } lm->kp_enabled = true; - err = device_create_file(&client->dev, &dev_attr_disable_kp); - if (err < 0) - goto fail2; idev->name = pdata->name ? : "LM8323 keypad"; snprintf(lm->phys, sizeof(lm->phys), @@ -719,14 +725,16 @@ static int lm8323_probe(struct i2c_client *client) err = input_register_device(idev); if (err) { dev_dbg(&client->dev, "error registering input device\n"); - goto fail3; + return err; } - err = request_threaded_irq(client->irq, NULL, lm8323_irq, - IRQF_TRIGGER_LOW|IRQF_ONESHOT, "lm8323", lm); + err = devm_request_threaded_irq(&client->dev, client->irq, + NULL, lm8323_irq, + IRQF_TRIGGER_LOW | IRQF_ONESHOT, + "lm8323", lm); if (err) { dev_err(&client->dev, "could not get IRQ %d\n", client->irq); - goto fail4; + return err; } i2c_set_clientdata(client, lm); @@ -735,39 +743,6 @@ static int lm8323_probe(struct i2c_client *client) enable_irq_wake(client->irq); return 0; - -fail4: - input_unregister_device(idev); - idev = NULL; -fail3: - device_remove_file(&client->dev, &dev_attr_disable_kp); -fail2: - while (--pwm >= 0) - if (lm->pwm[pwm].enabled) - led_classdev_unregister(&lm->pwm[pwm].cdev); -fail1: - input_free_device(idev); - kfree(lm); - return err; -} - -static void lm8323_remove(struct i2c_client *client) -{ - struct lm8323_chip *lm = i2c_get_clientdata(client); - int i; - - disable_irq_wake(client->irq); - free_irq(client->irq, lm); - - input_unregister_device(lm->idev); - - device_remove_file(&lm->client->dev, &dev_attr_disable_kp); - - for (i = 0; i < 3; i++) - if (lm->pwm[i].enabled) - led_classdev_unregister(&lm->pwm[i].cdev); - - kfree(lm); } /* @@ -823,11 +798,11 @@ static const struct i2c_device_id lm8323_id[] = { static struct i2c_driver lm8323_i2c_driver = { .driver = { - .name = "lm8323", - .pm = pm_sleep_ptr(&lm8323_pm_ops), + .name = "lm8323", + .pm = pm_sleep_ptr(&lm8323_pm_ops), + .dev_groups = lm8323_groups, }, .probe = lm8323_probe, - .remove = lm8323_remove, .id_table = lm8323_id, }; MODULE_DEVICE_TABLE(i2c, lm8323_id); diff --git a/drivers/input/keyboard/lm8333.c b/drivers/input/keyboard/lm8333.c index c9f05764e36d..1c070c499c85 100644 --- a/drivers/input/keyboard/lm8333.c +++ b/drivers/input/keyboard/lm8333.c @@ -142,18 +142,18 @@ static int lm8333_probe(struct i2c_client *client) return -EINVAL; } - lm8333 = kzalloc(sizeof(*lm8333), GFP_KERNEL); - input = input_allocate_device(); - if (!lm8333 || !input) { - err = -ENOMEM; - goto free_mem; - } + lm8333 = devm_kzalloc(&client->dev, sizeof(*lm8333), GFP_KERNEL); + if (!lm8333) + return -ENOMEM; + + input = devm_input_allocate_device(&client->dev); + if (!input) + return -ENOMEM; lm8333->client = client; lm8333->input = input; input->name = client->name; - input->dev.parent = &client->dev; input->id.bustype = BUS_I2C; input_set_capability(input, EV_MSC, MSC_SCAN); @@ -162,7 +162,7 @@ static int lm8333_probe(struct i2c_client *client) LM8333_NUM_ROWS, LM8333_NUM_COLS, lm8333->keycodes, input); if (err) - goto free_mem; + return err; if (pdata->debounce_time) { err = lm8333_write8(lm8333, LM8333_DEBOUNCE, @@ -178,34 +178,19 @@ static int lm8333_probe(struct i2c_client *client) dev_warn(&client->dev, "Unable to set active time\n"); } - err = request_threaded_irq(client->irq, NULL, lm8333_irq_thread, - IRQF_TRIGGER_FALLING | IRQF_ONESHOT, - "lm8333", lm8333); + err = devm_request_threaded_irq(&client->dev, client->irq, + NULL, lm8333_irq_thread, + IRQF_TRIGGER_FALLING | IRQF_ONESHOT, + "lm8333", lm8333); if (err) - goto free_mem; + return err; err = input_register_device(input); if (err) - goto free_irq; + return err; i2c_set_clientdata(client, lm8333); return 0; - - free_irq: - free_irq(client->irq, lm8333); - free_mem: - input_free_device(input); - kfree(lm8333); - return err; -} - -static void lm8333_remove(struct i2c_client *client) -{ - struct lm8333 *lm8333 = i2c_get_clientdata(client); - - free_irq(client->irq, lm8333); - input_unregister_device(lm8333->input); - kfree(lm8333); } static const struct i2c_device_id lm8333_id[] = { @@ -219,7 +204,6 @@ static struct i2c_driver lm8333_driver = { .name = "lm8333", }, .probe = lm8333_probe, - .remove = lm8333_remove, .id_table = lm8333_id, }; module_i2c_driver(lm8333_driver); diff --git a/drivers/input/keyboard/lpc32xx-keys.c b/drivers/input/keyboard/lpc32xx-keys.c index 911e1181cd6f..322a87807159 100644 --- a/drivers/input/keyboard/lpc32xx-keys.c +++ b/drivers/input/keyboard/lpc32xx-keys.c @@ -160,17 +160,10 @@ static int lpc32xx_kscan_probe(struct platform_device *pdev) { struct lpc32xx_kscan_drv *kscandat; struct input_dev *input; - struct resource *res; size_t keymap_size; int error; int irq; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (!res) { - dev_err(&pdev->dev, "failed to get platform I/O memory\n"); - return -EINVAL; - } - irq = platform_get_irq(pdev, 0); if (irq < 0) return -EINVAL; @@ -221,7 +214,7 @@ static int lpc32xx_kscan_probe(struct platform_device *pdev) input_set_drvdata(kscandat->input, kscandat); - kscandat->kscan_base = devm_ioremap_resource(&pdev->dev, res); + kscandat->kscan_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(kscandat->kscan_base)) return PTR_ERR(kscandat->kscan_base); diff --git a/drivers/input/keyboard/mcs_touchkey.c b/drivers/input/keyboard/mcs_touchkey.c index de312d8eb974..2410f676c7f9 100644 --- a/drivers/input/keyboard/mcs_touchkey.c +++ b/drivers/input/keyboard/mcs_touchkey.c @@ -92,6 +92,13 @@ static irqreturn_t mcs_touchkey_interrupt(int irq, void *dev_id) return IRQ_HANDLED; } +static void mcs_touchkey_poweroff(void *data) +{ + struct mcs_touchkey_data *touchkey = data; + + touchkey->poweron(false); +} + static int mcs_touchkey_probe(struct i2c_client *client) { const struct i2c_device_id *id = i2c_client_get_device_id(client); @@ -109,13 +116,16 @@ static int mcs_touchkey_probe(struct i2c_client *client) return -EINVAL; } - data = kzalloc(struct_size(data, keycodes, pdata->key_maxval + 1), - GFP_KERNEL); - input_dev = input_allocate_device(); - if (!data || !input_dev) { - dev_err(&client->dev, "Failed to allocate memory\n"); - error = -ENOMEM; - goto err_free_mem; + data = devm_kzalloc(&client->dev, + struct_size(data, keycodes, pdata->key_maxval + 1), + GFP_KERNEL); + if (!data) + return -ENOMEM; + + input_dev = devm_input_allocate_device(&client->dev); + if (!input_dev) { + dev_err(&client->dev, "Failed to allocate input device\n"); + return -ENOMEM; } data->client = client; @@ -136,15 +146,13 @@ static int mcs_touchkey_probe(struct i2c_client *client) fw_ver = i2c_smbus_read_byte_data(client, fw_reg); if (fw_ver < 0) { - error = fw_ver; - dev_err(&client->dev, "i2c read error[%d]\n", error); - goto err_free_mem; + dev_err(&client->dev, "i2c read error[%d]\n", fw_ver); + return fw_ver; } dev_info(&client->dev, "Firmware version: %d\n", fw_ver); input_dev->name = "MELFAS MCS Touchkey"; input_dev->id.bustype = BUS_I2C; - input_dev->dev.parent = &client->dev; input_dev->evbit[0] = BIT_MASK(EV_KEY); if (!pdata->no_autorepeat) input_dev->evbit[0] |= BIT_MASK(EV_REP); @@ -169,40 +177,28 @@ static int mcs_touchkey_probe(struct i2c_client *client) if (pdata->poweron) { data->poweron = pdata->poweron; data->poweron(true); + + error = devm_add_action_or_reset(&client->dev, + mcs_touchkey_poweroff, data); + if (error) + return error; } - error = request_threaded_irq(client->irq, NULL, mcs_touchkey_interrupt, - IRQF_TRIGGER_FALLING | IRQF_ONESHOT, - client->dev.driver->name, data); + error = devm_request_threaded_irq(&client->dev, client->irq, + NULL, mcs_touchkey_interrupt, + IRQF_TRIGGER_FALLING | IRQF_ONESHOT, + client->dev.driver->name, data); if (error) { dev_err(&client->dev, "Failed to register interrupt\n"); - goto err_free_mem; + return error; } error = input_register_device(input_dev); if (error) - goto err_free_irq; + return error; i2c_set_clientdata(client, data); return 0; - -err_free_irq: - free_irq(client->irq, data); -err_free_mem: - input_free_device(input_dev); - kfree(data); - return error; -} - -static void mcs_touchkey_remove(struct i2c_client *client) -{ - struct mcs_touchkey_data *data = i2c_get_clientdata(client); - - free_irq(client->irq, data); - if (data->poweron) - data->poweron(false); - input_unregister_device(data->input_dev); - kfree(data); } static void mcs_touchkey_shutdown(struct i2c_client *client) @@ -259,7 +255,6 @@ static struct i2c_driver mcs_touchkey_driver = { .pm = pm_sleep_ptr(&mcs_touchkey_pm_ops), }, .probe = mcs_touchkey_probe, - .remove = mcs_touchkey_remove, .shutdown = mcs_touchkey_shutdown, .id_table = mcs_touchkey_id, }; diff --git a/drivers/input/keyboard/nomadik-ske-keypad.c b/drivers/input/keyboard/nomadik-ske-keypad.c index 970f2a671c2e..b3ccc97f61e1 100644 --- a/drivers/input/keyboard/nomadik-ske-keypad.c +++ b/drivers/input/keyboard/nomadik-ske-keypad.c @@ -221,13 +221,20 @@ static irqreturn_t ske_keypad_irq(int irq, void *dev_id) return IRQ_HANDLED; } +static void ske_keypad_board_exit(void *data) +{ + struct ske_keypad *keypad = data; + + keypad->board->exit(); +} + static int __init ske_keypad_probe(struct platform_device *pdev) { const struct ske_keypad_platform_data *plat = dev_get_platdata(&pdev->dev); + struct device *dev = &pdev->dev; struct ske_keypad *keypad; struct input_dev *input; - struct resource *res; int irq; int error; @@ -238,20 +245,14 @@ static int __init ske_keypad_probe(struct platform_device *pdev) irq = platform_get_irq(pdev, 0); if (irq < 0) - return -EINVAL; + return irq; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (!res) { - dev_err(&pdev->dev, "missing platform resources\n"); - return -EINVAL; - } - - keypad = kzalloc(sizeof(struct ske_keypad), GFP_KERNEL); - input = input_allocate_device(); + keypad = devm_kzalloc(dev, sizeof(struct ske_keypad), + GFP_KERNEL); + input = devm_input_allocate_device(dev); if (!keypad || !input) { dev_err(&pdev->dev, "failed to allocate keypad memory\n"); - error = -ENOMEM; - goto err_free_mem; + return -ENOMEM; } keypad->irq = irq; @@ -259,31 +260,20 @@ static int __init ske_keypad_probe(struct platform_device *pdev) keypad->input = input; spin_lock_init(&keypad->ske_keypad_lock); - if (!request_mem_region(res->start, resource_size(res), pdev->name)) { - dev_err(&pdev->dev, "failed to request I/O memory\n"); - error = -EBUSY; - goto err_free_mem; - } + keypad->reg_base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(keypad->reg_base)) + return PTR_ERR(keypad->reg_base); - keypad->reg_base = ioremap(res->start, resource_size(res)); - if (!keypad->reg_base) { - dev_err(&pdev->dev, "failed to remap I/O memory\n"); - error = -ENXIO; - goto err_free_mem_region; - } - - keypad->pclk = clk_get(&pdev->dev, "apb_pclk"); + keypad->pclk = devm_clk_get_enabled(dev, "apb_pclk"); if (IS_ERR(keypad->pclk)) { dev_err(&pdev->dev, "failed to get pclk\n"); - error = PTR_ERR(keypad->pclk); - goto err_iounmap; + return PTR_ERR(keypad->pclk); } - keypad->clk = clk_get(&pdev->dev, NULL); + keypad->clk = devm_clk_get_enabled(dev, NULL); if (IS_ERR(keypad->clk)) { dev_err(&pdev->dev, "failed to get clk\n"); - error = PTR_ERR(keypad->clk); - goto err_pclk; + return PTR_ERR(keypad->clk); } input->id.bustype = BUS_HOST; @@ -295,48 +285,43 @@ static int __init ske_keypad_probe(struct platform_device *pdev) keypad->keymap, input); if (error) { dev_err(&pdev->dev, "Failed to build keymap\n"); - goto err_clk; + return error; } input_set_capability(input, EV_MSC, MSC_SCAN); if (!plat->no_autorepeat) __set_bit(EV_REP, input->evbit); - error = clk_prepare_enable(keypad->pclk); - if (error) { - dev_err(&pdev->dev, "Failed to prepare/enable pclk\n"); - goto err_clk; - } - - error = clk_prepare_enable(keypad->clk); - if (error) { - dev_err(&pdev->dev, "Failed to prepare/enable clk\n"); - goto err_pclk_disable; - } - - /* go through board initialization helpers */ if (keypad->board->init) keypad->board->init(); + if (keypad->board->exit) { + error = devm_add_action_or_reset(dev, ske_keypad_board_exit, + keypad); + if (error) + return error; + } + error = ske_keypad_chip_init(keypad); if (error) { dev_err(&pdev->dev, "unable to init keypad hardware\n"); - goto err_clk_disable; + return error; } - error = request_threaded_irq(keypad->irq, NULL, ske_keypad_irq, - IRQF_ONESHOT, "ske-keypad", keypad); + error = devm_request_threaded_irq(dev, keypad->irq, + NULL, ske_keypad_irq, + IRQF_ONESHOT, "ske-keypad", keypad); if (error) { dev_err(&pdev->dev, "allocate irq %d failed\n", keypad->irq); - goto err_clk_disable; + return error; } error = input_register_device(input); if (error) { dev_err(&pdev->dev, - "unable to register input device: %d\n", error); - goto err_free_irq; + "unable to register input device: %d\n", error); + return error; } if (plat->wakeup_enable) @@ -344,47 +329,6 @@ static int __init ske_keypad_probe(struct platform_device *pdev) platform_set_drvdata(pdev, keypad); - return 0; - -err_free_irq: - free_irq(keypad->irq, keypad); -err_clk_disable: - clk_disable_unprepare(keypad->clk); -err_pclk_disable: - clk_disable_unprepare(keypad->pclk); -err_clk: - clk_put(keypad->clk); -err_pclk: - clk_put(keypad->pclk); -err_iounmap: - iounmap(keypad->reg_base); -err_free_mem_region: - release_mem_region(res->start, resource_size(res)); -err_free_mem: - input_free_device(input); - kfree(keypad); - return error; -} - -static int ske_keypad_remove(struct platform_device *pdev) -{ - struct ske_keypad *keypad = platform_get_drvdata(pdev); - struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - free_irq(keypad->irq, keypad); - - input_unregister_device(keypad->input); - - clk_disable_unprepare(keypad->clk); - clk_put(keypad->clk); - - if (keypad->board->exit) - keypad->board->exit(); - - iounmap(keypad->reg_base); - release_mem_region(res->start, resource_size(res)); - kfree(keypad); - return 0; } @@ -424,7 +368,6 @@ static struct platform_driver ske_keypad_driver = { .name = "nmk-ske-keypad", .pm = pm_sleep_ptr(&ske_keypad_dev_pm_ops), }, - .remove = ske_keypad_remove, }; module_platform_driver_probe(ske_keypad_driver, ske_keypad_probe); diff --git a/drivers/input/keyboard/nspire-keypad.c b/drivers/input/keyboard/nspire-keypad.c index e9fa1423f136..096c18d7bca1 100644 --- a/drivers/input/keyboard/nspire-keypad.c +++ b/drivers/input/keyboard/nspire-keypad.c @@ -186,8 +186,7 @@ static int nspire_keypad_probe(struct platform_device *pdev) return PTR_ERR(keypad->clk); } - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - keypad->reg_base = devm_ioremap_resource(&pdev->dev, res); + keypad->reg_base = devm_platform_get_and_ioremap_resource(pdev, 0, &res); if (IS_ERR(keypad->reg_base)) return PTR_ERR(keypad->reg_base); diff --git a/drivers/input/keyboard/omap4-keypad.c b/drivers/input/keyboard/omap4-keypad.c index 9f085d5679db..773e55eed88b 100644 --- a/drivers/input/keyboard/omap4-keypad.c +++ b/drivers/input/keyboard/omap4-keypad.c @@ -341,17 +341,10 @@ static int omap4_keypad_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct omap4_keypad *keypad_data; struct input_dev *input_dev; - struct resource *res; unsigned int max_keys; int irq; int error; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (!res) { - dev_err(&pdev->dev, "no base address specified\n"); - return -EINVAL; - } - irq = platform_get_irq(pdev, 0); if (irq < 0) return irq; @@ -370,7 +363,7 @@ static int omap4_keypad_probe(struct platform_device *pdev) if (error) return error; - keypad_data->base = devm_ioremap_resource(dev, res); + keypad_data->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(keypad_data->base)) return PTR_ERR(keypad_data->base); diff --git a/drivers/input/keyboard/opencores-kbd.c b/drivers/input/keyboard/opencores-kbd.c index b0ea387414c1..7ffe1a70c856 100644 --- a/drivers/input/keyboard/opencores-kbd.c +++ b/drivers/input/keyboard/opencores-kbd.c @@ -39,15 +39,8 @@ static int opencores_kbd_probe(struct platform_device *pdev) { struct input_dev *input; struct opencores_kbd *opencores_kbd; - struct resource *res; int irq, i, error; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (!res) { - dev_err(&pdev->dev, "missing board memory resource\n"); - return -EINVAL; - } - irq = platform_get_irq(pdev, 0); if (irq < 0) return -EINVAL; @@ -65,7 +58,7 @@ static int opencores_kbd_probe(struct platform_device *pdev) opencores_kbd->input = input; - opencores_kbd->addr = devm_ioremap_resource(&pdev->dev, res); + opencores_kbd->addr = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(opencores_kbd->addr)) return PTR_ERR(opencores_kbd->addr); diff --git a/drivers/input/keyboard/pinephone-keyboard.c b/drivers/input/keyboard/pinephone-keyboard.c index 038ff3549a7a..147b1f288a33 100644 --- a/drivers/input/keyboard/pinephone-keyboard.c +++ b/drivers/input/keyboard/pinephone-keyboard.c @@ -318,40 +318,22 @@ static void ppkb_close(struct input_dev *input) ppkb_set_scan(client, false); } -static void ppkb_regulator_disable(void *regulator) -{ - regulator_disable(regulator); -} - static int ppkb_probe(struct i2c_client *client) { struct device *dev = &client->dev; unsigned int phys_rows, phys_cols; struct pinephone_keyboard *ppkb; - struct regulator *vbat_supply; u8 info[PPKB_MATRIX_SIZE + 1]; struct device_node *i2c_bus; int ret; int error; - vbat_supply = devm_regulator_get(dev, "vbat"); - error = PTR_ERR_OR_ZERO(vbat_supply); + error = devm_regulator_get_enable(dev, "vbat"); if (error) { dev_err(dev, "Failed to get VBAT supply: %d\n", error); return error; } - error = regulator_enable(vbat_supply); - if (error) { - dev_err(dev, "Failed to enable VBAT: %d\n", error); - return error; - } - - error = devm_add_action_or_reset(dev, ppkb_regulator_disable, - vbat_supply); - if (error) - return error; - ret = i2c_smbus_read_i2c_block_data(client, 0, sizeof(info), info); if (ret != sizeof(info)) { error = ret < 0 ? ret : -EIO; diff --git a/drivers/input/keyboard/pxa27x_keypad.c b/drivers/input/keyboard/pxa27x_keypad.c index 871f858d0ba7..3724363d140e 100644 --- a/drivers/input/keyboard/pxa27x_keypad.c +++ b/drivers/input/keyboard/pxa27x_keypad.c @@ -717,7 +717,6 @@ static int pxa27x_keypad_probe(struct platform_device *pdev) struct device_node *np = pdev->dev.of_node; struct pxa27x_keypad *keypad; struct input_dev *input_dev; - struct resource *res; int irq, error; /* Driver need build keycode from device tree or pdata */ @@ -728,12 +727,6 @@ static int pxa27x_keypad_probe(struct platform_device *pdev) if (irq < 0) return -ENXIO; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (res == NULL) { - dev_err(&pdev->dev, "failed to get I/O memory\n"); - return -ENXIO; - } - keypad = devm_kzalloc(&pdev->dev, sizeof(*keypad), GFP_KERNEL); if (!keypad) @@ -747,7 +740,7 @@ static int pxa27x_keypad_probe(struct platform_device *pdev) keypad->input_dev = input_dev; keypad->irq = irq; - keypad->mmio_base = devm_ioremap_resource(&pdev->dev, res); + keypad->mmio_base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(keypad->mmio_base)) return PTR_ERR(keypad->mmio_base); diff --git a/drivers/input/keyboard/qt1070.c b/drivers/input/keyboard/qt1070.c index 91aaa9fc43a4..9b093b042bf1 100644 --- a/drivers/input/keyboard/qt1070.c +++ b/drivers/input/keyboard/qt1070.c @@ -149,20 +149,20 @@ static int qt1070_probe(struct i2c_client *client) if (!qt1070_identify(client)) return -ENODEV; - data = kzalloc(sizeof(struct qt1070_data), GFP_KERNEL); - input = input_allocate_device(); - if (!data || !input) { - dev_err(&client->dev, "insufficient memory\n"); - err = -ENOMEM; - goto err_free_mem; - } + data = devm_kzalloc(&client->dev, sizeof(struct qt1070_data), + GFP_KERNEL); + if (!data) + return -ENOMEM; + + input = devm_input_allocate_device(&client->dev); + if (!input) + return -ENOMEM; data->client = client; data->input = input; data->irq = client->irq; input->name = "AT42QT1070 QTouch Sensor"; - input->dev.parent = &client->dev; input->id.bustype = BUS_I2C; /* Add the keycode */ @@ -185,19 +185,20 @@ static int qt1070_probe(struct i2c_client *client) qt1070_write(client, RESET, 1); msleep(QT1070_RESET_TIME); - err = request_threaded_irq(client->irq, NULL, qt1070_interrupt, - IRQF_TRIGGER_NONE | IRQF_ONESHOT, - client->dev.driver->name, data); + err = devm_request_threaded_irq(&client->dev, client->irq, + NULL, qt1070_interrupt, + IRQF_TRIGGER_NONE | IRQF_ONESHOT, + client->dev.driver->name, data); if (err) { dev_err(&client->dev, "fail to request irq\n"); - goto err_free_mem; + return err; } /* Register the input device */ err = input_register_device(data->input); if (err) { dev_err(&client->dev, "Failed to register input device\n"); - goto err_free_irq; + return err; } i2c_set_clientdata(client, data); @@ -206,24 +207,6 @@ static int qt1070_probe(struct i2c_client *client) qt1070_read(client, DET_STATUS); return 0; - -err_free_irq: - free_irq(client->irq, data); -err_free_mem: - input_free_device(input); - kfree(data); - return err; -} - -static void qt1070_remove(struct i2c_client *client) -{ - struct qt1070_data *data = i2c_get_clientdata(client); - - /* Release IRQ */ - free_irq(client->irq, data); - - input_unregister_device(data->input); - kfree(data); } static int qt1070_suspend(struct device *dev) @@ -272,7 +255,6 @@ static struct i2c_driver qt1070_driver = { }, .id_table = qt1070_id, .probe = qt1070_probe, - .remove = qt1070_remove, }; module_i2c_driver(qt1070_driver); diff --git a/drivers/input/keyboard/qt2160.c b/drivers/input/keyboard/qt2160.c index 599ea85cfd30..7e3b09642ab7 100644 --- a/drivers/input/keyboard/qt2160.c +++ b/drivers/input/keyboard/qt2160.c @@ -32,7 +32,7 @@ #define QT2160_NUM_LEDS_X 8 -#define QT2160_CYCLE_INTERVAL (2*HZ) +#define QT2160_CYCLE_INTERVAL 2000 /* msec - 2 sec */ static unsigned char qt2160_key2code[] = { KEY_0, KEY_1, KEY_2, KEY_3, @@ -54,7 +54,6 @@ struct qt2160_led { struct qt2160_data { struct i2c_client *client; struct input_dev *input; - struct delayed_work dwork; unsigned short keycodes[ARRAY_SIZE(qt2160_key2code)]; u16 key_matrix; #ifdef CONFIG_LEDS_CLASS @@ -155,10 +154,10 @@ static int qt2160_read_block(struct i2c_client *client, return 0; } -static int qt2160_get_key_matrix(struct qt2160_data *qt2160) +static void qt2160_get_key_matrix(struct input_dev *input) { + struct qt2160_data *qt2160 = input_get_drvdata(input); struct i2c_client *client = qt2160->client; - struct input_dev *input = qt2160->input; u8 regs[6]; u16 old_matrix, new_matrix; int ret, i, mask; @@ -173,7 +172,7 @@ static int qt2160_get_key_matrix(struct qt2160_data *qt2160) if (ret) { dev_err(&client->dev, "could not perform chip read.\n"); - return ret; + return; } old_matrix = qt2160->key_matrix; @@ -191,37 +190,17 @@ static int qt2160_get_key_matrix(struct qt2160_data *qt2160) } input_sync(input); - - return 0; } -static irqreturn_t qt2160_irq(int irq, void *_qt2160) +static irqreturn_t qt2160_irq(int irq, void *data) { - struct qt2160_data *qt2160 = _qt2160; + struct input_dev *input = data; - mod_delayed_work(system_wq, &qt2160->dwork, 0); + qt2160_get_key_matrix(input); return IRQ_HANDLED; } -static void qt2160_schedule_read(struct qt2160_data *qt2160) -{ - schedule_delayed_work(&qt2160->dwork, QT2160_CYCLE_INTERVAL); -} - -static void qt2160_worker(struct work_struct *work) -{ - struct qt2160_data *qt2160 = - container_of(work, struct qt2160_data, dwork.work); - - dev_dbg(&qt2160->client->dev, "worker\n"); - - qt2160_get_key_matrix(qt2160); - - /* Avoid device lock up by checking every so often */ - qt2160_schedule_read(qt2160); -} - static int qt2160_read(struct i2c_client *client, u8 reg) { int ret; @@ -260,7 +239,7 @@ static int qt2160_write(struct i2c_client *client, u8 reg, u8 data) static int qt2160_register_leds(struct qt2160_data *qt2160) { struct i2c_client *client = qt2160->client; - int ret; + int error; int i; for (i = 0; i < QT2160_NUM_LEDS_X; i++) { @@ -273,9 +252,9 @@ static int qt2160_register_leds(struct qt2160_data *qt2160) led->id = i; led->qt2160 = qt2160; - ret = led_classdev_register(&client->dev, &led->cdev); - if (ret < 0) - return ret; + error = devm_led_classdev_register(&client->dev, &led->cdev); + if (error) + return error; } /* Tur off LEDs */ @@ -286,14 +265,6 @@ static int qt2160_register_leds(struct qt2160_data *qt2160) return 0; } -static void qt2160_unregister_leds(struct qt2160_data *qt2160) -{ - int i; - - for (i = 0; i < QT2160_NUM_LEDS_X; i++) - led_classdev_unregister(&qt2160->leds[i].cdev); -} - #else static inline int qt2160_register_leds(struct qt2160_data *qt2160) @@ -301,10 +272,6 @@ static inline int qt2160_register_leds(struct qt2160_data *qt2160) return 0; } -static inline void qt2160_unregister_leds(struct qt2160_data *qt2160) -{ -} - #endif static bool qt2160_identify(struct i2c_client *client) @@ -345,12 +312,9 @@ static int qt2160_probe(struct i2c_client *client) int i; int error; - /* Check functionality */ - error = i2c_check_functionality(client->adapter, - I2C_FUNC_SMBUS_BYTE); - if (!error) { + if (!i2c_check_functionality(client->adapter, I2C_FUNC_SMBUS_BYTE)) { dev_err(&client->dev, "%s adapter not supported\n", - dev_driver_string(&client->adapter->dev)); + dev_driver_string(&client->adapter->dev)); return -ENODEV; } @@ -358,17 +322,16 @@ static int qt2160_probe(struct i2c_client *client) return -ENODEV; /* Chip is valid and active. Allocate structure */ - qt2160 = kzalloc(sizeof(struct qt2160_data), GFP_KERNEL); - input = input_allocate_device(); - if (!qt2160 || !input) { - dev_err(&client->dev, "insufficient memory\n"); - error = -ENOMEM; - goto err_free_mem; - } + qt2160 = devm_kzalloc(&client->dev, sizeof(*qt2160), GFP_KERNEL); + if (!qt2160) + return -ENOMEM; + + input = devm_input_allocate_device(&client->dev); + if (!input) + return -ENOMEM; qt2160->client = client; qt2160->input = input; - INIT_DELAYED_WORK(&qt2160->dwork, qt2160_worker); input->name = "AT42QT2160 Touch Sense Keyboard"; input->id.bustype = BUS_I2C; @@ -385,66 +348,48 @@ static int qt2160_probe(struct i2c_client *client) } __clear_bit(KEY_RESERVED, input->keybit); + input_set_drvdata(input, qt2160); + /* Calibrate device */ error = qt2160_write(client, QT2160_CMD_CALIBRATE, 1); if (error) { dev_err(&client->dev, "failed to calibrate device\n"); - goto err_free_mem; + return error; } if (client->irq) { - error = request_irq(client->irq, qt2160_irq, - IRQF_TRIGGER_FALLING, "qt2160", qt2160); + error = devm_request_threaded_irq(&client->dev, client->irq, + NULL, qt2160_irq, + IRQF_ONESHOT, + "qt2160", input); if (error) { dev_err(&client->dev, "failed to allocate irq %d\n", client->irq); - goto err_free_mem; + return error; } + } else { + error = input_setup_polling(input, qt2160_get_key_matrix); + if (error) { + dev_err(&client->dev, "Failed to setup polling\n"); + return error; + } + input_set_poll_interval(input, QT2160_CYCLE_INTERVAL); } error = qt2160_register_leds(qt2160); if (error) { dev_err(&client->dev, "Failed to register leds\n"); - goto err_free_irq; + return error; } error = input_register_device(qt2160->input); if (error) { dev_err(&client->dev, "Failed to register input device\n"); - goto err_unregister_leds; + return error; } - i2c_set_clientdata(client, qt2160); - qt2160_schedule_read(qt2160); - return 0; - -err_unregister_leds: - qt2160_unregister_leds(qt2160); -err_free_irq: - if (client->irq) - free_irq(client->irq, qt2160); -err_free_mem: - input_free_device(input); - kfree(qt2160); - return error; -} - -static void qt2160_remove(struct i2c_client *client) -{ - struct qt2160_data *qt2160 = i2c_get_clientdata(client); - - qt2160_unregister_leds(qt2160); - - /* Release IRQ so no queue will be scheduled */ - if (client->irq) - free_irq(client->irq, qt2160); - - cancel_delayed_work_sync(&qt2160->dwork); - - input_unregister_device(qt2160->input); - kfree(qt2160); } static const struct i2c_device_id qt2160_idtable[] = { @@ -461,7 +406,6 @@ static struct i2c_driver qt2160_driver = { .id_table = qt2160_idtable, .probe = qt2160_probe, - .remove = qt2160_remove, }; module_i2c_driver(qt2160_driver); diff --git a/drivers/input/keyboard/sun4i-lradc-keys.c b/drivers/input/keyboard/sun4i-lradc-keys.c index 15c15c0958b0..f304cab0ebdb 100644 --- a/drivers/input/keyboard/sun4i-lradc-keys.c +++ b/drivers/input/keyboard/sun4i-lradc-keys.c @@ -21,10 +21,11 @@ #include #include #include -#include +#include #include #include #include +#include #include #include #include @@ -307,8 +308,7 @@ static int sun4i_lradc_probe(struct platform_device *pdev) input_set_drvdata(lradc->input, lradc); - lradc->base = devm_ioremap_resource(dev, - platform_get_resource(pdev, IORESOURCE_MEM, 0)); + lradc->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(lradc->base)) return PTR_ERR(lradc->base); diff --git a/drivers/input/keyboard/tca6416-keypad.c b/drivers/input/keyboard/tca6416-keypad.c index 2f745cabf4f2..8af59ced1ec2 100644 --- a/drivers/input/keyboard/tca6416-keypad.c +++ b/drivers/input/keyboard/tca6416-keypad.c @@ -24,6 +24,8 @@ #define TCA6416_INVERT 2 #define TCA6416_DIRECTION 3 +#define TCA6416_POLL_INTERVAL 100 /* msec */ + static const struct i2c_device_id tca6416_id[] = { { "tca6416-keys", 16, }, { "tca6408-keys", 8, }, @@ -43,7 +45,6 @@ struct tca6416_keypad_chip { struct i2c_client *client; struct input_dev *input; - struct delayed_work dwork; int io_size; int irqnum; u16 pinmask; @@ -85,9 +86,9 @@ static int tca6416_read_reg(struct tca6416_keypad_chip *chip, int reg, u16 *val) return 0; } -static void tca6416_keys_scan(struct tca6416_keypad_chip *chip) +static void tca6416_keys_scan(struct input_dev *input) { - struct input_dev *input = chip->input; + struct tca6416_keypad_chip *chip = input_get_drvdata(input); u16 reg_val, val; int error, i, pin_index; @@ -122,33 +123,20 @@ static void tca6416_keys_scan(struct tca6416_keypad_chip *chip) */ static irqreturn_t tca6416_keys_isr(int irq, void *dev_id) { - struct tca6416_keypad_chip *chip = dev_id; - - tca6416_keys_scan(chip); + tca6416_keys_scan(dev_id); return IRQ_HANDLED; } -static void tca6416_keys_work_func(struct work_struct *work) -{ - struct tca6416_keypad_chip *chip = - container_of(work, struct tca6416_keypad_chip, dwork.work); - - tca6416_keys_scan(chip); - schedule_delayed_work(&chip->dwork, msecs_to_jiffies(100)); -} - static int tca6416_keys_open(struct input_dev *dev) { struct tca6416_keypad_chip *chip = input_get_drvdata(dev); - /* Get initial device state in case it has switches */ - tca6416_keys_scan(chip); - - if (chip->use_polling) - schedule_delayed_work(&chip->dwork, msecs_to_jiffies(100)); - else - enable_irq(chip->irqnum); + if (!chip->use_polling) { + /* Get initial device state in case it has switches */ + tca6416_keys_scan(dev); + enable_irq(chip->client->irq); + } return 0; } @@ -157,10 +145,8 @@ static void tca6416_keys_close(struct input_dev *dev) { struct tca6416_keypad_chip *chip = input_get_drvdata(dev); - if (chip->use_polling) - cancel_delayed_work_sync(&chip->dwork); - else - disable_irq(chip->irqnum); + if (!chip->use_polling) + disable_irq(chip->client->irq); } static int tca6416_setup_registers(struct tca6416_keypad_chip *chip) @@ -216,12 +202,15 @@ static int tca6416_keypad_probe(struct i2c_client *client) return -EINVAL; } - chip = kzalloc(struct_size(chip, buttons, pdata->nbuttons), GFP_KERNEL); - input = input_allocate_device(); - if (!chip || !input) { - error = -ENOMEM; - goto fail1; - } + chip = devm_kzalloc(&client->dev, + struct_size(chip, buttons, pdata->nbuttons), + GFP_KERNEL); + if (!chip) + return -ENOMEM; + + input = devm_input_allocate_device(&client->dev); + if (!input) + return -ENOMEM; chip->client = client; chip->input = input; @@ -229,11 +218,8 @@ static int tca6416_keypad_probe(struct i2c_client *client) chip->pinmask = pdata->pinmask; chip->use_polling = pdata->use_polling; - INIT_DELAYED_WORK(&chip->dwork, tca6416_keys_work_func); - input->phys = "tca6416-keys/input0"; input->name = client->name; - input->dev.parent = &client->dev; input->open = tca6416_keys_open; input->close = tca6416_keys_close; @@ -263,24 +249,28 @@ static int tca6416_keypad_probe(struct i2c_client *client) */ error = tca6416_setup_registers(chip); if (error) - goto fail1; + return error; - if (!chip->use_polling) { - if (pdata->irq_is_gpio) - chip->irqnum = gpio_to_irq(client->irq); - else - chip->irqnum = client->irq; + if (chip->use_polling) { + error = input_setup_polling(input, tca6416_keys_scan); + if (error) { + dev_err(&client->dev, "Failed to setup polling\n"); + return error; + } - error = request_threaded_irq(chip->irqnum, NULL, - tca6416_keys_isr, - IRQF_TRIGGER_FALLING | - IRQF_ONESHOT | IRQF_NO_AUTOEN, - "tca6416-keypad", chip); + input_set_poll_interval(input, TCA6416_POLL_INTERVAL); + } else { + error = devm_request_threaded_irq(&client->dev, client->irq, + NULL, tca6416_keys_isr, + IRQF_TRIGGER_FALLING | + IRQF_ONESHOT | + IRQF_NO_AUTOEN, + "tca6416-keypad", input); if (error) { dev_dbg(&client->dev, "Unable to claim irq %d; error %d\n", - chip->irqnum, error); - goto fail1; + client->irq, error); + return error; } } @@ -288,70 +278,19 @@ static int tca6416_keypad_probe(struct i2c_client *client) if (error) { dev_dbg(&client->dev, "Unable to register input device, error: %d\n", error); - goto fail2; + return error; } i2c_set_clientdata(client, chip); - device_init_wakeup(&client->dev, 1); - - return 0; - -fail2: - if (!chip->use_polling) { - free_irq(chip->irqnum, chip); - enable_irq(chip->irqnum); - } -fail1: - input_free_device(input); - kfree(chip); - return error; -} - -static void tca6416_keypad_remove(struct i2c_client *client) -{ - struct tca6416_keypad_chip *chip = i2c_get_clientdata(client); - - if (!chip->use_polling) { - free_irq(chip->irqnum, chip); - enable_irq(chip->irqnum); - } - - input_unregister_device(chip->input); - kfree(chip); -} - -static int tca6416_keypad_suspend(struct device *dev) -{ - struct i2c_client *client = to_i2c_client(dev); - struct tca6416_keypad_chip *chip = i2c_get_clientdata(client); - - if (device_may_wakeup(dev)) - enable_irq_wake(chip->irqnum); return 0; } -static int tca6416_keypad_resume(struct device *dev) -{ - struct i2c_client *client = to_i2c_client(dev); - struct tca6416_keypad_chip *chip = i2c_get_clientdata(client); - - if (device_may_wakeup(dev)) - disable_irq_wake(chip->irqnum); - - return 0; -} - -static DEFINE_SIMPLE_DEV_PM_OPS(tca6416_keypad_dev_pm_ops, - tca6416_keypad_suspend, tca6416_keypad_resume); - static struct i2c_driver tca6416_keypad_driver = { .driver = { .name = "tca6416-keypad", - .pm = pm_sleep_ptr(&tca6416_keypad_dev_pm_ops), }, .probe = tca6416_keypad_probe, - .remove = tca6416_keypad_remove, .id_table = tca6416_id, }; diff --git a/drivers/input/keyboard/tegra-kbc.c b/drivers/input/keyboard/tegra-kbc.c index d5a6c7d8eb25..c9a823ea45d0 100644 --- a/drivers/input/keyboard/tegra-kbc.c +++ b/drivers/input/keyboard/tegra-kbc.c @@ -640,7 +640,7 @@ static int tegra_kbc_probe(struct platform_device *pdev) timer_setup(&kbc->timer, tegra_kbc_keypress_timer, 0); - kbc->mmio = devm_platform_get_and_ioremap_resource(pdev, 0, NULL); + kbc->mmio = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(kbc->mmio)) return PTR_ERR(kbc->mmio); diff --git a/drivers/input/keyboard/tm2-touchkey.c b/drivers/input/keyboard/tm2-touchkey.c index 75bd3ea51194..0fd761ae052f 100644 --- a/drivers/input/keyboard/tm2-touchkey.c +++ b/drivers/input/keyboard/tm2-touchkey.c @@ -19,7 +19,6 @@ #include #include #include -#include #include #include diff --git a/drivers/input/misc/Kconfig b/drivers/input/misc/Kconfig index 8a320e6218e3..6ba984d7f0b1 100644 --- a/drivers/input/misc/Kconfig +++ b/drivers/input/misc/Kconfig @@ -791,10 +791,10 @@ config INPUT_IQS626A module will be called iqs626a. config INPUT_IQS7222 - tristate "Azoteq IQS7222A/B/C capacitive touch controller" + tristate "Azoteq IQS7222A/B/C/D capacitive touch controller" depends on I2C help - Say Y to enable support for the Azoteq IQS7222A/B/C family + Say Y to enable support for the Azoteq IQS7222A/B/C/D family of capacitive touch controllers. To compile this driver as a module, choose M here: the diff --git a/drivers/input/misc/cpcap-pwrbutton.c b/drivers/input/misc/cpcap-pwrbutton.c index 879790bbf9fe..85cddb84717a 100644 --- a/drivers/input/misc/cpcap-pwrbutton.c +++ b/drivers/input/misc/cpcap-pwrbutton.c @@ -1,16 +1,8 @@ -/** +// SPDX-License-Identifier: GPL-2.0-only +/* * CPCAP Power Button Input Driver * * Copyright (C) 2017 Sebastian Reichel - * - * This file is subject to the terms and conditions of the GNU General - * Public License. See the file "COPYING" in the main directory of this - * archive for more details. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/drivers/input/misc/da9063_onkey.c b/drivers/input/misc/da9063_onkey.c index b14a389600c9..74808bae326a 100644 --- a/drivers/input/misc/da9063_onkey.c +++ b/drivers/input/misc/da9063_onkey.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -251,6 +252,14 @@ static int da9063_onkey_probe(struct platform_device *pdev) return error; } + error = dev_pm_set_wake_irq(&pdev->dev, irq); + if (error) + dev_warn(&pdev->dev, + "Failed to set IRQ %d as a wake IRQ: %d\n", + irq, error); + else + device_init_wakeup(&pdev->dev, true); + error = input_register_device(onkey->input); if (error) { dev_err(&pdev->dev, diff --git a/drivers/input/misc/gpio-vibra.c b/drivers/input/misc/gpio-vibra.c index 134a1309ba92..ad44b4d18a2a 100644 --- a/drivers/input/misc/gpio-vibra.c +++ b/drivers/input/misc/gpio-vibra.c @@ -18,7 +18,7 @@ #include #include #include -#include +#include #include #include #include @@ -113,22 +113,14 @@ static int gpio_vibrator_probe(struct platform_device *pdev) return -ENOMEM; vibrator->vcc = devm_regulator_get(&pdev->dev, "vcc"); - err = PTR_ERR_OR_ZERO(vibrator->vcc); - if (err) { - if (err != -EPROBE_DEFER) - dev_err(&pdev->dev, "Failed to request regulator: %d\n", - err); - return err; - } + if (IS_ERR(vibrator->vcc)) + return dev_err_probe(&pdev->dev, PTR_ERR(vibrator->vcc), + "Failed to request regulator\n"); vibrator->gpio = devm_gpiod_get(&pdev->dev, "enable", GPIOD_OUT_LOW); - err = PTR_ERR_OR_ZERO(vibrator->gpio); - if (err) { - if (err != -EPROBE_DEFER) - dev_err(&pdev->dev, "Failed to request main gpio: %d\n", - err); - return err; - } + if (IS_ERR(vibrator->gpio)) + return dev_err_probe(&pdev->dev, PTR_ERR(vibrator->gpio), + "Failed to request main gpio\n"); INIT_WORK(&vibrator->play_work, gpio_vibrator_play_work); diff --git a/drivers/input/misc/iqs269a.c b/drivers/input/misc/iqs269a.c index 1272ef7b5794..c0a085639870 100644 --- a/drivers/input/misc/iqs269a.c +++ b/drivers/input/misc/iqs269a.c @@ -17,9 +17,9 @@ #include #include #include +#include #include #include -#include #include #include #include diff --git a/drivers/input/misc/iqs626a.c b/drivers/input/misc/iqs626a.c index 50035c25c3f7..0dab54d3a060 100644 --- a/drivers/input/misc/iqs626a.c +++ b/drivers/input/misc/iqs626a.c @@ -19,8 +19,8 @@ #include #include #include +#include #include -#include #include #include #include diff --git a/drivers/input/misc/iqs7222.c b/drivers/input/misc/iqs7222.c index 096b0925f41b..36aeeae77611 100644 --- a/drivers/input/misc/iqs7222.c +++ b/drivers/input/misc/iqs7222.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-or-later /* - * Azoteq IQS7222A/B/C Capacitive Touch Controller + * Azoteq IQS7222A/B/C/D Capacitive Touch Controller * * Copyright (C) 2022 Jeff LaBundy */ @@ -12,11 +12,12 @@ #include #include #include +#include #include #include #include +#include #include -#include #include #include #include @@ -25,6 +26,7 @@ #define IQS7222_PROD_NUM_A 840 #define IQS7222_PROD_NUM_B 698 #define IQS7222_PROD_NUM_C 863 +#define IQS7222_PROD_NUM_D 1046 #define IQS7222_SYS_STATUS 0x10 #define IQS7222_SYS_STATUS_RESET BIT(3) @@ -54,6 +56,7 @@ #define IQS7222_EVENT_MASK_ATI BIT(12) #define IQS7222_EVENT_MASK_SLDR BIT(10) +#define IQS7222_EVENT_MASK_TPAD IQS7222_EVENT_MASK_SLDR #define IQS7222_EVENT_MASK_TOUCH BIT(1) #define IQS7222_EVENT_MASK_PROX BIT(0) @@ -71,6 +74,7 @@ #define IQS7222_MAX_COLS_CHAN 6 #define IQS7222_MAX_COLS_FILT 2 #define IQS7222_MAX_COLS_SLDR 11 +#define IQS7222_MAX_COLS_TPAD 24 #define IQS7222_MAX_COLS_GPIO 3 #define IQS7222_MAX_COLS_SYS 13 @@ -102,16 +106,18 @@ enum iqs7222_reg_grp_id { IQS7222_REG_GRP_BTN, IQS7222_REG_GRP_CHAN, IQS7222_REG_GRP_SLDR, + IQS7222_REG_GRP_TPAD, IQS7222_REG_GRP_GPIO, IQS7222_REG_GRP_SYS, IQS7222_NUM_REG_GRPS }; static const char * const iqs7222_reg_grp_names[IQS7222_NUM_REG_GRPS] = { - [IQS7222_REG_GRP_CYCLE] = "cycle", - [IQS7222_REG_GRP_CHAN] = "channel", - [IQS7222_REG_GRP_SLDR] = "slider", - [IQS7222_REG_GRP_GPIO] = "gpio", + [IQS7222_REG_GRP_CYCLE] = "cycle-%d", + [IQS7222_REG_GRP_CHAN] = "channel-%d", + [IQS7222_REG_GRP_SLDR] = "slider-%d", + [IQS7222_REG_GRP_TPAD] = "trackpad", + [IQS7222_REG_GRP_GPIO] = "gpio-%d", }; static const unsigned int iqs7222_max_cols[IQS7222_NUM_REG_GRPS] = { @@ -122,6 +128,7 @@ static const unsigned int iqs7222_max_cols[IQS7222_NUM_REG_GRPS] = { [IQS7222_REG_GRP_CHAN] = IQS7222_MAX_COLS_CHAN, [IQS7222_REG_GRP_FILT] = IQS7222_MAX_COLS_FILT, [IQS7222_REG_GRP_SLDR] = IQS7222_MAX_COLS_SLDR, + [IQS7222_REG_GRP_TPAD] = IQS7222_MAX_COLS_TPAD, [IQS7222_REG_GRP_GPIO] = IQS7222_MAX_COLS_GPIO, [IQS7222_REG_GRP_SYS] = IQS7222_MAX_COLS_SYS, }; @@ -130,8 +137,10 @@ static const unsigned int iqs7222_gpio_links[] = { 2, 5, 6, }; struct iqs7222_event_desc { const char *name; + u16 link; u16 mask; u16 val; + u16 strict; u16 enable; enum iqs7222_reg_key_id reg_key; }; @@ -188,6 +197,93 @@ static const struct iqs7222_event_desc iqs7222_sl_events[] = { }, }; +static const struct iqs7222_event_desc iqs7222_tp_events[] = { + { + .name = "event-press", + .link = BIT(7), + }, + { + .name = "event-tap", + .link = BIT(0), + .mask = BIT(0), + .val = BIT(0), + .enable = BIT(0), + .reg_key = IQS7222_REG_KEY_TAP, + }, + { + .name = "event-swipe-x-pos", + .link = BIT(2), + .mask = BIT(2) | BIT(1), + .val = BIT(2), + .strict = BIT(4), + .enable = BIT(1), + .reg_key = IQS7222_REG_KEY_AXIAL, + }, + { + .name = "event-swipe-y-pos", + .link = BIT(3), + .mask = BIT(3) | BIT(1), + .val = BIT(3), + .strict = BIT(3), + .enable = BIT(1), + .reg_key = IQS7222_REG_KEY_AXIAL, + }, + { + .name = "event-swipe-x-neg", + .link = BIT(4), + .mask = BIT(4) | BIT(1), + .val = BIT(4), + .strict = BIT(4), + .enable = BIT(1), + .reg_key = IQS7222_REG_KEY_AXIAL, + }, + { + .name = "event-swipe-y-neg", + .link = BIT(5), + .mask = BIT(5) | BIT(1), + .val = BIT(5), + .strict = BIT(3), + .enable = BIT(1), + .reg_key = IQS7222_REG_KEY_AXIAL, + }, + { + .name = "event-flick-x-pos", + .link = BIT(2), + .mask = BIT(2) | BIT(1), + .val = BIT(2) | BIT(1), + .strict = BIT(4), + .enable = BIT(2), + .reg_key = IQS7222_REG_KEY_AXIAL, + }, + { + .name = "event-flick-y-pos", + .link = BIT(3), + .mask = BIT(3) | BIT(1), + .val = BIT(3) | BIT(1), + .strict = BIT(3), + .enable = BIT(2), + .reg_key = IQS7222_REG_KEY_AXIAL, + }, + { + .name = "event-flick-x-neg", + .link = BIT(4), + .mask = BIT(4) | BIT(1), + .val = BIT(4) | BIT(1), + .strict = BIT(4), + .enable = BIT(2), + .reg_key = IQS7222_REG_KEY_AXIAL, + }, + { + .name = "event-flick-y-neg", + .link = BIT(5), + .mask = BIT(5) | BIT(1), + .val = BIT(5) | BIT(1), + .strict = BIT(3), + .enable = BIT(2), + .reg_key = IQS7222_REG_KEY_AXIAL, + }, +}; + struct iqs7222_reg_grp_desc { u16 base; int num_row; @@ -524,6 +620,62 @@ static const struct iqs7222_dev_desc iqs7222_devs[] = { }, }, }, + { + .prod_num = IQS7222_PROD_NUM_D, + .fw_major = 0, + .fw_minor = 37, + .touch_link = 1770, + .allow_offset = 9, + .event_offset = 10, + .comms_offset = 11, + .reg_grps = { + [IQS7222_REG_GRP_STAT] = { + .base = IQS7222_SYS_STATUS, + .num_row = 1, + .num_col = 7, + }, + [IQS7222_REG_GRP_CYCLE] = { + .base = 0x8000, + .num_row = 7, + .num_col = 2, + }, + [IQS7222_REG_GRP_GLBL] = { + .base = 0x8700, + .num_row = 1, + .num_col = 3, + }, + [IQS7222_REG_GRP_BTN] = { + .base = 0x9000, + .num_row = 14, + .num_col = 3, + }, + [IQS7222_REG_GRP_CHAN] = { + .base = 0xA000, + .num_row = 14, + .num_col = 4, + }, + [IQS7222_REG_GRP_FILT] = { + .base = 0xAE00, + .num_row = 1, + .num_col = 2, + }, + [IQS7222_REG_GRP_TPAD] = { + .base = 0xB000, + .num_row = 1, + .num_col = 24, + }, + [IQS7222_REG_GRP_GPIO] = { + .base = 0xC000, + .num_row = 3, + .num_col = 3, + }, + [IQS7222_REG_GRP_SYS] = { + .base = IQS7222_SYS_SETUP, + .num_row = 1, + .num_col = 12, + }, + }, + }, }; struct iqs7222_prop_desc { @@ -1008,6 +1160,123 @@ static const struct iqs7222_prop_desc iqs7222_props[] = { .val_pitch = 4, .label = "maximum gesture time", }, + { + .name = "azoteq,num-rows", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_offset = 0, + .reg_shift = 4, + .reg_width = 4, + .val_min = 1, + .val_max = 12, + .label = "number of rows", + }, + { + .name = "azoteq,num-cols", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_offset = 0, + .reg_shift = 0, + .reg_width = 4, + .val_min = 1, + .val_max = 12, + .label = "number of columns", + }, + { + .name = "azoteq,lower-cal-y", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_offset = 1, + .reg_shift = 8, + .reg_width = 8, + .label = "lower vertical calibration", + }, + { + .name = "azoteq,lower-cal-x", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_offset = 1, + .reg_shift = 0, + .reg_width = 8, + .label = "lower horizontal calibration", + }, + { + .name = "azoteq,upper-cal-y", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_offset = 2, + .reg_shift = 8, + .reg_width = 8, + .label = "upper vertical calibration", + }, + { + .name = "azoteq,upper-cal-x", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_offset = 2, + .reg_shift = 0, + .reg_width = 8, + .label = "upper horizontal calibration", + }, + { + .name = "azoteq,top-speed", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_offset = 3, + .reg_shift = 8, + .reg_width = 8, + .val_pitch = 4, + .label = "top speed", + }, + { + .name = "azoteq,bottom-speed", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_offset = 3, + .reg_shift = 0, + .reg_width = 8, + .label = "bottom speed", + }, + { + .name = "azoteq,gesture-min-ms", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_key = IQS7222_REG_KEY_TAP, + .reg_offset = 20, + .reg_shift = 8, + .reg_width = 8, + .val_pitch = 16, + .label = "minimum gesture time", + }, + { + .name = "azoteq,gesture-max-ms", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_key = IQS7222_REG_KEY_AXIAL, + .reg_offset = 21, + .reg_shift = 8, + .reg_width = 8, + .val_pitch = 16, + .label = "maximum gesture time", + }, + { + .name = "azoteq,gesture-max-ms", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_key = IQS7222_REG_KEY_TAP, + .reg_offset = 21, + .reg_shift = 0, + .reg_width = 8, + .val_pitch = 16, + .label = "maximum gesture time", + }, + { + .name = "azoteq,gesture-dist", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_key = IQS7222_REG_KEY_TAP, + .reg_offset = 22, + .reg_shift = 0, + .reg_width = 16, + .label = "gesture distance", + }, + { + .name = "azoteq,gesture-dist", + .reg_grp = IQS7222_REG_GRP_TPAD, + .reg_key = IQS7222_REG_KEY_AXIAL, + .reg_offset = 23, + .reg_shift = 0, + .reg_width = 16, + .label = "gesture distance", + }, { .name = "drive-open-drain", .reg_grp = IQS7222_REG_GRP_GPIO, @@ -1091,16 +1360,19 @@ struct iqs7222_private { struct gpio_desc *irq_gpio; struct i2c_client *client; struct input_dev *keypad; + struct touchscreen_properties prop; unsigned int kp_type[IQS7222_MAX_CHAN][ARRAY_SIZE(iqs7222_kp_events)]; unsigned int kp_code[IQS7222_MAX_CHAN][ARRAY_SIZE(iqs7222_kp_events)]; unsigned int sl_code[IQS7222_MAX_SLDR][ARRAY_SIZE(iqs7222_sl_events)]; unsigned int sl_axis[IQS7222_MAX_SLDR]; + unsigned int tp_code[ARRAY_SIZE(iqs7222_tp_events)]; u16 cycle_setup[IQS7222_MAX_CHAN / 2][IQS7222_MAX_COLS_CYCLE]; u16 glbl_setup[IQS7222_MAX_COLS_GLBL]; u16 btn_setup[IQS7222_MAX_CHAN][IQS7222_MAX_COLS_BTN]; u16 chan_setup[IQS7222_MAX_CHAN][IQS7222_MAX_COLS_CHAN]; u16 filt_setup[IQS7222_MAX_COLS_FILT]; u16 sldr_setup[IQS7222_MAX_SLDR][IQS7222_MAX_COLS_SLDR]; + u16 tpad_setup[IQS7222_MAX_COLS_TPAD]; u16 gpio_setup[ARRAY_SIZE(iqs7222_gpio_links)][IQS7222_MAX_COLS_GPIO]; u16 sys_setup[IQS7222_MAX_COLS_SYS]; }; @@ -1127,6 +1399,9 @@ static u16 *iqs7222_setup(struct iqs7222_private *iqs7222, case IQS7222_REG_GRP_SLDR: return iqs7222->sldr_setup[row]; + case IQS7222_REG_GRP_TPAD: + return iqs7222->tpad_setup; + case IQS7222_REG_GRP_GPIO: return iqs7222->gpio_setup[row]; @@ -1381,9 +1656,6 @@ static int iqs7222_ati_trigger(struct iqs7222_private *iqs7222) if (error) return error; - sys_setup &= ~IQS7222_SYS_SETUP_INTF_MODE_MASK; - sys_setup &= ~IQS7222_SYS_SETUP_PWR_MODE_MASK; - for (i = 0; i < IQS7222_NUM_RETRIES; i++) { /* * Trigger ATI from streaming and normal-power modes so that @@ -1561,8 +1833,11 @@ static int iqs7222_dev_init(struct iqs7222_private *iqs7222, int dir) return error; } - if (dir == READ) + if (dir == READ) { + iqs7222->sys_setup[0] &= ~IQS7222_SYS_SETUP_INTF_MODE_MASK; + iqs7222->sys_setup[0] &= ~IQS7222_SYS_SETUP_PWR_MODE_MASK; return 0; + } return iqs7222_ati_trigger(iqs7222); } @@ -1936,6 +2211,14 @@ static int iqs7222_parse_chan(struct iqs7222_private *iqs7222, ref_setup[4] = dev_desc->touch_link; if (fwnode_property_present(chan_node, "azoteq,use-prox")) ref_setup[4] -= 2; + } else if (dev_desc->reg_grps[IQS7222_REG_GRP_TPAD].num_row && + fwnode_property_present(chan_node, + "azoteq,counts-filt-enable")) { + /* + * In the case of IQS7222D, however, the reference mode field + * is partially repurposed as a counts filter enable control. + */ + chan_setup[0] |= IQS7222_CHAN_SETUP_0_REF_MODE_REF; } if (fwnode_property_present(chan_node, "azoteq,rx-enable")) { @@ -2278,6 +2561,136 @@ static int iqs7222_parse_sldr(struct iqs7222_private *iqs7222, IQS7222_REG_KEY_NO_WHEEL); } +static int iqs7222_parse_tpad(struct iqs7222_private *iqs7222, + struct fwnode_handle *tpad_node, int tpad_index) +{ + const struct iqs7222_dev_desc *dev_desc = iqs7222->dev_desc; + struct touchscreen_properties *prop = &iqs7222->prop; + struct i2c_client *client = iqs7222->client; + int num_chan = dev_desc->reg_grps[IQS7222_REG_GRP_CHAN].num_row; + int count, error, i; + u16 *event_mask = &iqs7222->sys_setup[dev_desc->event_offset]; + u16 *tpad_setup = iqs7222->tpad_setup; + unsigned int chan_sel[12]; + + error = iqs7222_parse_props(iqs7222, tpad_node, tpad_index, + IQS7222_REG_GRP_TPAD, + IQS7222_REG_KEY_NONE); + if (error) + return error; + + count = fwnode_property_count_u32(tpad_node, "azoteq,channel-select"); + if (count < 0) { + dev_err(&client->dev, "Failed to count %s channels: %d\n", + fwnode_get_name(tpad_node), count); + return count; + } else if (!count || count > ARRAY_SIZE(chan_sel)) { + dev_err(&client->dev, "Invalid number of %s channels\n", + fwnode_get_name(tpad_node)); + return -EINVAL; + } + + error = fwnode_property_read_u32_array(tpad_node, + "azoteq,channel-select", + chan_sel, count); + if (error) { + dev_err(&client->dev, "Failed to read %s channels: %d\n", + fwnode_get_name(tpad_node), error); + return error; + } + + tpad_setup[6] &= ~GENMASK(num_chan - 1, 0); + + for (i = 0; i < ARRAY_SIZE(chan_sel); i++) { + tpad_setup[8 + i] = 0; + if (i >= count || chan_sel[i] == U8_MAX) + continue; + + if (chan_sel[i] >= num_chan) { + dev_err(&client->dev, "Invalid %s channel: %u\n", + fwnode_get_name(tpad_node), chan_sel[i]); + return -EINVAL; + } + + /* + * The following fields indicate which channels participate in + * the trackpad, as well as each channel's relative placement. + */ + tpad_setup[6] |= BIT(chan_sel[i]); + tpad_setup[8 + i] = chan_sel[i] * 34 + 1072; + } + + tpad_setup[7] = dev_desc->touch_link; + if (fwnode_property_present(tpad_node, "azoteq,use-prox")) + tpad_setup[7] -= 2; + + for (i = 0; i < ARRAY_SIZE(iqs7222_tp_events); i++) + tpad_setup[20] &= ~(iqs7222_tp_events[i].strict | + iqs7222_tp_events[i].enable); + + for (i = 0; i < ARRAY_SIZE(iqs7222_tp_events); i++) { + const char *event_name = iqs7222_tp_events[i].name; + struct fwnode_handle *event_node; + + event_node = fwnode_get_named_child_node(tpad_node, event_name); + if (!event_node) + continue; + + if (fwnode_property_present(event_node, + "azoteq,gesture-angle-tighten")) + tpad_setup[20] |= iqs7222_tp_events[i].strict; + + tpad_setup[20] |= iqs7222_tp_events[i].enable; + + error = iqs7222_parse_event(iqs7222, event_node, tpad_index, + IQS7222_REG_GRP_TPAD, + iqs7222_tp_events[i].reg_key, + iqs7222_tp_events[i].link, 1566, + NULL, + &iqs7222->tp_code[i]); + fwnode_handle_put(event_node); + if (error) + return error; + + if (!dev_desc->event_offset) + continue; + + /* + * The press/release event is determined based on whether the + * coordinate fields report 0xFFFF and solely relies on touch + * or proximity interrupts to be unmasked. + */ + if (i) + *event_mask |= IQS7222_EVENT_MASK_TPAD; + else if (tpad_setup[7] == dev_desc->touch_link) + *event_mask |= IQS7222_EVENT_MASK_TOUCH; + else + *event_mask |= IQS7222_EVENT_MASK_PROX; + } + + if (!iqs7222->tp_code[0]) + return 0; + + input_set_abs_params(iqs7222->keypad, ABS_X, + 0, (tpad_setup[4] ? : 1) - 1, 0, 0); + + input_set_abs_params(iqs7222->keypad, ABS_Y, + 0, (tpad_setup[5] ? : 1) - 1, 0, 0); + + touchscreen_parse_properties(iqs7222->keypad, false, prop); + + if (prop->max_x >= U16_MAX || prop->max_y >= U16_MAX) { + dev_err(&client->dev, "Invalid trackpad size: %u*%u\n", + prop->max_x, prop->max_y); + return -EINVAL; + } + + tpad_setup[4] = prop->max_x + 1; + tpad_setup[5] = prop->max_y + 1; + + return 0; +} + static int (*iqs7222_parse_extra[IQS7222_NUM_REG_GRPS]) (struct iqs7222_private *iqs7222, struct fwnode_handle *reg_grp_node, @@ -2285,6 +2698,7 @@ static int (*iqs7222_parse_extra[IQS7222_NUM_REG_GRPS]) [IQS7222_REG_GRP_CYCLE] = iqs7222_parse_cycle, [IQS7222_REG_GRP_CHAN] = iqs7222_parse_chan, [IQS7222_REG_GRP_SLDR] = iqs7222_parse_sldr, + [IQS7222_REG_GRP_TPAD] = iqs7222_parse_tpad, }; static int iqs7222_parse_reg_grp(struct iqs7222_private *iqs7222, @@ -2298,7 +2712,7 @@ static int iqs7222_parse_reg_grp(struct iqs7222_private *iqs7222, if (iqs7222_reg_grp_names[reg_grp]) { char reg_grp_name[16]; - snprintf(reg_grp_name, sizeof(reg_grp_name), "%s-%d", + snprintf(reg_grp_name, sizeof(reg_grp_name), iqs7222_reg_grp_names[reg_grp], reg_grp_index); reg_grp_node = device_get_named_child_node(&client->dev, @@ -2346,8 +2760,8 @@ static int iqs7222_parse_all(struct iqs7222_private *iqs7222) continue; /* - * The IQS7222C exposes multiple GPIO and must be informed - * as to which GPIO this group represents. + * The IQS7222C and IQS7222D expose multiple GPIO and must be + * informed as to which GPIO this group represents. */ for (j = 0; j < ARRAY_SIZE(iqs7222_gpio_links); j++) gpio_setup[0] &= ~BIT(iqs7222_gpio_links[j]); @@ -2480,6 +2894,41 @@ static int iqs7222_report(struct iqs7222_private *iqs7222) iqs7222->sl_code[i][j], 0); } + for (i = 0; i < dev_desc->reg_grps[IQS7222_REG_GRP_TPAD].num_row; i++) { + u16 tpad_pos_x = le16_to_cpu(status[4]); + u16 tpad_pos_y = le16_to_cpu(status[5]); + u16 state = le16_to_cpu(status[6]); + + input_report_key(iqs7222->keypad, iqs7222->tp_code[0], + tpad_pos_x < U16_MAX); + + if (tpad_pos_x < U16_MAX) + touchscreen_report_pos(iqs7222->keypad, &iqs7222->prop, + tpad_pos_x, tpad_pos_y, false); + + if (!(le16_to_cpu(status[1]) & IQS7222_EVENT_MASK_TPAD)) + continue; + + /* + * Skip the press/release event, as it does not have separate + * status fields and is handled separately. + */ + for (j = 1; j < ARRAY_SIZE(iqs7222_tp_events); j++) { + u16 mask = iqs7222_tp_events[j].mask; + u16 val = iqs7222_tp_events[j].val; + + input_report_key(iqs7222->keypad, + iqs7222->tp_code[j], + (state & mask) == val); + } + + input_sync(iqs7222->keypad); + + for (j = 1; j < ARRAY_SIZE(iqs7222_tp_events); j++) + input_report_key(iqs7222->keypad, + iqs7222->tp_code[j], 0); + } + input_sync(iqs7222->keypad); return 0; @@ -2584,6 +3033,7 @@ static const struct of_device_id iqs7222_of_match[] = { { .compatible = "azoteq,iqs7222a" }, { .compatible = "azoteq,iqs7222b" }, { .compatible = "azoteq,iqs7222c" }, + { .compatible = "azoteq,iqs7222d" }, { } }; MODULE_DEVICE_TABLE(of, iqs7222_of_match); @@ -2598,5 +3048,5 @@ static struct i2c_driver iqs7222_i2c_driver = { module_i2c_driver(iqs7222_i2c_driver); MODULE_AUTHOR("Jeff LaBundy "); -MODULE_DESCRIPTION("Azoteq IQS7222A/B/C Capacitive Touch Controller"); +MODULE_DESCRIPTION("Azoteq IQS7222A/B/C/D Capacitive Touch Controller"); MODULE_LICENSE("GPL"); diff --git a/drivers/input/misc/mma8450.c b/drivers/input/misc/mma8450.c index 76a190b2220b..662b436d765b 100644 --- a/drivers/input/misc/mma8450.c +++ b/drivers/input/misc/mma8450.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #define MMA8450_DRV_NAME "mma8450" diff --git a/drivers/input/misc/pm8941-pwrkey.c b/drivers/input/misc/pm8941-pwrkey.c index 74d77d8aaeff..ba747c5b2b5f 100644 --- a/drivers/input/misc/pm8941-pwrkey.c +++ b/drivers/input/misc/pm8941-pwrkey.c @@ -14,7 +14,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/input/misc/pm8xxx-vibrator.c b/drivers/input/misc/pm8xxx-vibrator.c index 04cb87efd799..5c288fe7accf 100644 --- a/drivers/input/misc/pm8xxx-vibrator.c +++ b/drivers/input/misc/pm8xxx-vibrator.c @@ -7,7 +7,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/input/misc/pmic8xxx-pwrkey.c b/drivers/input/misc/pmic8xxx-pwrkey.c index 89fb137e3715..c406a1cca5c4 100644 --- a/drivers/input/misc/pmic8xxx-pwrkey.c +++ b/drivers/input/misc/pmic8xxx-pwrkey.c @@ -12,7 +12,6 @@ #include #include #include -#include #define PON_CNTL_1 0x1C #define PON_CNTL_PULL_UP BIT(7) diff --git a/drivers/input/misc/pwm-beeper.c b/drivers/input/misc/pwm-beeper.c index 3cf1812384e6..1e731d8397c6 100644 --- a/drivers/input/misc/pwm-beeper.c +++ b/drivers/input/misc/pwm-beeper.c @@ -132,13 +132,8 @@ static int pwm_beeper_probe(struct platform_device *pdev) return -ENOMEM; beeper->pwm = devm_pwm_get(dev, NULL); - if (IS_ERR(beeper->pwm)) { - error = PTR_ERR(beeper->pwm); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to request PWM device: %d\n", - error); - return error; - } + if (IS_ERR(beeper->pwm)) + return dev_err_probe(dev, PTR_ERR(beeper->pwm), "Failed to request PWM device\n"); /* Sync up PWM state and ensure it is off. */ pwm_init_state(beeper->pwm, &state); @@ -151,13 +146,9 @@ static int pwm_beeper_probe(struct platform_device *pdev) } beeper->amplifier = devm_regulator_get(dev, "amp"); - if (IS_ERR(beeper->amplifier)) { - error = PTR_ERR(beeper->amplifier); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to get 'amp' regulator: %d\n", - error); - return error; - } + if (IS_ERR(beeper->amplifier)) + return dev_err_probe(dev, PTR_ERR(beeper->amplifier), + "Failed to get 'amp' regulator\n"); INIT_WORK(&beeper->work, pwm_beeper_work); diff --git a/drivers/input/misc/pwm-vibra.c b/drivers/input/misc/pwm-vibra.c index 2ba035299db8..acac79c488aa 100644 --- a/drivers/input/misc/pwm-vibra.c +++ b/drivers/input/misc/pwm-vibra.c @@ -15,7 +15,7 @@ #include #include #include -#include +#include #include #include #include @@ -140,32 +140,20 @@ static int pwm_vibrator_probe(struct platform_device *pdev) return -ENOMEM; vibrator->vcc = devm_regulator_get(&pdev->dev, "vcc"); - err = PTR_ERR_OR_ZERO(vibrator->vcc); - if (err) { - if (err != -EPROBE_DEFER) - dev_err(&pdev->dev, "Failed to request regulator: %d\n", - err); - return err; - } + if (IS_ERR(vibrator->vcc)) + return dev_err_probe(&pdev->dev, PTR_ERR(vibrator->vcc), + "Failed to request regulator\n"); vibrator->enable_gpio = devm_gpiod_get_optional(&pdev->dev, "enable", GPIOD_OUT_LOW); - err = PTR_ERR_OR_ZERO(vibrator->enable_gpio); - if (err) { - if (err != -EPROBE_DEFER) - dev_err(&pdev->dev, "Failed to request enable gpio: %d\n", - err); - return err; - } + if (IS_ERR(vibrator->enable_gpio)) + return dev_err_probe(&pdev->dev, PTR_ERR(vibrator->enable_gpio), + "Failed to request enable gpio\n"); vibrator->pwm = devm_pwm_get(&pdev->dev, "enable"); - err = PTR_ERR_OR_ZERO(vibrator->pwm); - if (err) { - if (err != -EPROBE_DEFER) - dev_err(&pdev->dev, "Failed to request main pwm: %d\n", - err); - return err; - } + if (IS_ERR(vibrator->pwm)) + return dev_err_probe(&pdev->dev, PTR_ERR(vibrator->pwm), + "Failed to request main pwm\n"); INIT_WORK(&vibrator->play_work, pwm_vibrator_play_work); diff --git a/drivers/input/misc/rotary_encoder.c b/drivers/input/misc/rotary_encoder.c index 22ec62083065..e94cab8133be 100644 --- a/drivers/input/misc/rotary_encoder.c +++ b/drivers/input/misc/rotary_encoder.c @@ -236,12 +236,8 @@ static int rotary_encoder_probe(struct platform_device *pdev) device_property_read_bool(dev, "rotary-encoder,relative-axis"); encoder->gpios = devm_gpiod_get_array(dev, NULL, GPIOD_IN); - if (IS_ERR(encoder->gpios)) { - err = PTR_ERR(encoder->gpios); - if (err != -EPROBE_DEFER) - dev_err(dev, "unable to get gpios: %d\n", err); - return err; - } + if (IS_ERR(encoder->gpios)) + return dev_err_probe(dev, PTR_ERR(encoder->gpios), "unable to get gpios\n"); if (encoder->gpios->ndescs < 2) { dev_err(dev, "not enough gpios found\n"); return -EINVAL; @@ -255,7 +251,6 @@ static int rotary_encoder_probe(struct platform_device *pdev) input->name = pdev->name; input->id.bustype = BUS_HOST; - input->dev.parent = dev; if (encoder->relative_axis) input_set_capability(input, EV_REL, encoder->axis); diff --git a/drivers/input/misc/sparcspkr.c b/drivers/input/misc/sparcspkr.c index cdcb7737c46a..e5dd84725c6e 100644 --- a/drivers/input/misc/sparcspkr.c +++ b/drivers/input/misc/sparcspkr.c @@ -9,7 +9,8 @@ #include #include #include -#include +#include +#include #include #include diff --git a/drivers/input/mouse/elan_i2c_core.c b/drivers/input/mouse/elan_i2c_core.c index 0cff742302a9..148a601396f9 100644 --- a/drivers/input/mouse/elan_i2c_core.c +++ b/drivers/input/mouse/elan_i2c_core.c @@ -1221,13 +1221,8 @@ static int elan_probe(struct i2c_client *client) mutex_init(&data->sysfs_mutex); data->vcc = devm_regulator_get(dev, "vcc"); - if (IS_ERR(data->vcc)) { - error = PTR_ERR(data->vcc); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to get 'vcc' regulator: %d\n", - error); - return error; - } + if (IS_ERR(data->vcc)) + return dev_err_probe(dev, PTR_ERR(data->vcc), "Failed to get 'vcc' regulator\n"); error = regulator_enable(data->vcc); if (error) { diff --git a/drivers/input/mouse/psmouse-smbus.c b/drivers/input/mouse/psmouse-smbus.c index 2a2459b1b4f2..7b13de979908 100644 --- a/drivers/input/mouse/psmouse-smbus.c +++ b/drivers/input/mouse/psmouse-smbus.c @@ -5,6 +5,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include #include #include #include @@ -118,13 +119,18 @@ static psmouse_ret_t psmouse_smbus_process_byte(struct psmouse *psmouse) return PSMOUSE_FULL_PACKET; } +static void psmouse_activate_smbus_mode(struct psmouse_smbus_dev *smbdev) +{ + if (smbdev->need_deactivate) { + psmouse_deactivate(smbdev->psmouse); + /* Give the device time to switch into SMBus mode */ + msleep(30); + } +} + static int psmouse_smbus_reconnect(struct psmouse *psmouse) { - struct psmouse_smbus_dev *smbdev = psmouse->private; - - if (smbdev->need_deactivate) - psmouse_deactivate(psmouse); - + psmouse_activate_smbus_mode(psmouse->private); return 0; } @@ -257,8 +263,7 @@ int psmouse_smbus_init(struct psmouse *psmouse, } } - if (need_deactivate) - psmouse_deactivate(psmouse); + psmouse_activate_smbus_mode(smbdev); psmouse->private = smbdev; psmouse->protocol_handler = psmouse_smbus_process_byte; diff --git a/drivers/input/serio/apbps2.c b/drivers/input/serio/apbps2.c index 513d96e40e0e..3f6866d39b86 100644 --- a/drivers/input/serio/apbps2.c +++ b/drivers/input/serio/apbps2.c @@ -14,11 +14,11 @@ * Contributors: Daniel Hellstrom */ #include -#include #include #include #include #include +#include #include #include #include diff --git a/drivers/input/serio/i8042-acpipnpio.h b/drivers/input/serio/i8042-acpipnpio.h index 028e45bd050b..1724d6cb8649 100644 --- a/drivers/input/serio/i8042-acpipnpio.h +++ b/drivers/input/serio/i8042-acpipnpio.h @@ -1281,6 +1281,13 @@ static const struct dmi_system_id i8042_dmi_quirk_table[] __initconst = { .driver_data = (void *)(SERIO_QUIRK_NOMUX | SERIO_QUIRK_RESET_ALWAYS | SERIO_QUIRK_NOLOOP | SERIO_QUIRK_NOPNP) }, + /* See comment on TUXEDO InfinityBook S17 Gen6 / Clevo NS70MU above */ + { + .matches = { + DMI_MATCH(DMI_BOARD_NAME, "PD5x_7xPNP_PNR_PNN_PNT"), + }, + .driver_data = (void *)(SERIO_QUIRK_NOAUX) + }, { .matches = { DMI_MATCH(DMI_BOARD_NAME, "X170SM"), diff --git a/drivers/input/serio/i8042-sparcio.h b/drivers/input/serio/i8042-sparcio.h index c712c1fe0605..b68793bf05c8 100644 --- a/drivers/input/serio/i8042-sparcio.h +++ b/drivers/input/serio/i8042-sparcio.h @@ -2,7 +2,9 @@ #ifndef _I8042_SPARCIO_H #define _I8042_SPARCIO_H -#include +#include +#include +#include #include #include diff --git a/drivers/input/serio/rpckbd.c b/drivers/input/serio/rpckbd.c index ce420eb1f51b..e8a9709f32eb 100644 --- a/drivers/input/serio/rpckbd.c +++ b/drivers/input/serio/rpckbd.c @@ -101,12 +101,12 @@ static int rpckbd_probe(struct platform_device *dev) int tx_irq, rx_irq; rx_irq = platform_get_irq(dev, 0); - if (rx_irq <= 0) - return rx_irq < 0 ? rx_irq : -ENXIO; + if (rx_irq < 0) + return rx_irq; tx_irq = platform_get_irq(dev, 1); - if (tx_irq <= 0) - return tx_irq < 0 ? tx_irq : -ENXIO; + if (tx_irq < 0) + return tx_irq; serio = kzalloc(sizeof(struct serio), GFP_KERNEL); rpckbd = kzalloc(sizeof(*rpckbd), GFP_KERNEL); diff --git a/drivers/input/serio/xilinx_ps2.c b/drivers/input/serio/xilinx_ps2.c index 960d7601fbc8..f3d28da70b75 100644 --- a/drivers/input/serio/xilinx_ps2.c +++ b/drivers/input/serio/xilinx_ps2.c @@ -14,10 +14,10 @@ #include #include #include +#include #include -#include #include -#include +#include #define DRIVER_NAME "xilinx_ps2" diff --git a/drivers/input/touchscreen/Kconfig b/drivers/input/touchscreen/Kconfig index c2cbd332af1d..e3e2324547b9 100644 --- a/drivers/input/touchscreen/Kconfig +++ b/drivers/input/touchscreen/Kconfig @@ -655,10 +655,10 @@ config TOUCHSCREEN_MTOUCH module will be called mtouch. config TOUCHSCREEN_NOVATEK_NVT_TS - tristate "Novatek NVT-ts touchscreen support" + tristate "Novatek NT11205 touchscreen support" depends on I2C help - Say Y here if you have a Novatek NVT-ts touchscreen. + Say Y here if you have a Novatek NT11205 touchscreen. If unsure, say N. To compile this driver as a module, choose M here: the @@ -1365,6 +1365,16 @@ config TOUCHSCREEN_IQS5XX To compile this driver as a module, choose M here: the module will be called iqs5xx. +config TOUCHSCREEN_IQS7211 + tristate "Azoteq IQS7210A/7211A/E trackpad/touchscreen controller" + depends on I2C + help + Say Y to enable support for the Azoteq IQS7210A/7211A/E + family of trackpad/touchscreen controllers. + + To compile this driver as a module, choose M here: the + module will be called iqs7211. + config TOUCHSCREEN_ZINITIX tristate "Zinitix touchscreen support" depends on I2C diff --git a/drivers/input/touchscreen/Makefile b/drivers/input/touchscreen/Makefile index 159cd5136fdb..62bd24f3ac8e 100644 --- a/drivers/input/touchscreen/Makefile +++ b/drivers/input/touchscreen/Makefile @@ -115,5 +115,6 @@ obj-$(CONFIG_TOUCHSCREEN_COLIBRI_VF50) += colibri-vf50-ts.o obj-$(CONFIG_TOUCHSCREEN_ROHM_BU21023) += rohm_bu21023.o obj-$(CONFIG_TOUCHSCREEN_RASPBERRYPI_FW) += raspberrypi-ts.o obj-$(CONFIG_TOUCHSCREEN_IQS5XX) += iqs5xx.o +obj-$(CONFIG_TOUCHSCREEN_IQS7211) += iqs7211.o obj-$(CONFIG_TOUCHSCREEN_ZINITIX) += zinitix.o obj-$(CONFIG_TOUCHSCREEN_HIMAX_HX83112B) += himax_hx83112b.o diff --git a/drivers/input/touchscreen/bu21013_ts.c b/drivers/input/touchscreen/bu21013_ts.c index 85332cfaa29d..652439a79e21 100644 --- a/drivers/input/touchscreen/bu21013_ts.c +++ b/drivers/input/touchscreen/bu21013_ts.c @@ -410,31 +410,32 @@ static int bu21013_probe(struct i2c_client *client) struct input_dev *in_dev; struct input_absinfo *info; u32 max_x = 0, max_y = 0; + struct device *dev = &client->dev; int error; if (!i2c_check_functionality(client->adapter, I2C_FUNC_SMBUS_BYTE_DATA)) { - dev_err(&client->dev, "i2c smbus byte data not supported\n"); + dev_err(dev, "i2c smbus byte data not supported\n"); return -EIO; } if (!client->irq) { - dev_err(&client->dev, "No IRQ set up\n"); + dev_err(dev, "No IRQ set up\n"); return -EINVAL; } - ts = devm_kzalloc(&client->dev, sizeof(*ts), GFP_KERNEL); + ts = devm_kzalloc(dev, sizeof(*ts), GFP_KERNEL); if (!ts) return -ENOMEM; ts->client = client; - ts->x_flip = device_property_read_bool(&client->dev, "rohm,flip-x"); - ts->y_flip = device_property_read_bool(&client->dev, "rohm,flip-y"); + ts->x_flip = device_property_read_bool(dev, "rohm,flip-x"); + ts->y_flip = device_property_read_bool(dev, "rohm,flip-y"); - in_dev = devm_input_allocate_device(&client->dev); + in_dev = devm_input_allocate_device(dev); if (!in_dev) { - dev_err(&client->dev, "device memory alloc failed\n"); + dev_err(dev, "device memory alloc failed\n"); return -ENOMEM; } ts->in_dev = in_dev; @@ -444,8 +445,8 @@ static int bu21013_probe(struct i2c_client *client) in_dev->name = DRIVER_TP; in_dev->id.bustype = BUS_I2C; - device_property_read_u32(&client->dev, "rohm,touch-max-x", &max_x); - device_property_read_u32(&client->dev, "rohm,touch-max-y", &max_y); + device_property_read_u32(dev, "rohm,touch-max-x", &max_x); + device_property_read_u32(dev, "rohm,touch-max-y", &max_y); input_set_abs_params(in_dev, ABS_MT_POSITION_X, 0, max_x, 0, 0); input_set_abs_params(in_dev, ABS_MT_POSITION_Y, 0, max_y, 0, 0); @@ -454,14 +455,14 @@ static int bu21013_probe(struct i2c_client *client) /* Adjust for the legacy "flip" properties, if present */ if (!ts->props.invert_x && - device_property_read_bool(&client->dev, "rohm,flip-x")) { + device_property_read_bool(dev, "rohm,flip-x")) { info = &in_dev->absinfo[ABS_MT_POSITION_X]; info->maximum -= info->minimum; info->minimum = 0; } if (!ts->props.invert_y && - device_property_read_bool(&client->dev, "rohm,flip-y")) { + device_property_read_bool(dev, "rohm,flip-y")) { info = &in_dev->absinfo[ABS_MT_POSITION_Y]; info->maximum -= info->minimum; info->minimum = 0; @@ -471,55 +472,46 @@ static int bu21013_probe(struct i2c_client *client) INPUT_MT_DIRECT | INPUT_MT_TRACK | INPUT_MT_DROP_UNUSED); if (error) { - dev_err(&client->dev, "failed to initialize MT slots"); + dev_err(dev, "failed to initialize MT slots"); return error; } - ts->regulator = devm_regulator_get(&client->dev, "avdd"); + ts->regulator = devm_regulator_get(dev, "avdd"); if (IS_ERR(ts->regulator)) { - dev_err(&client->dev, "regulator_get failed\n"); + dev_err(dev, "regulator_get failed\n"); return PTR_ERR(ts->regulator); } error = regulator_enable(ts->regulator); if (error) { - dev_err(&client->dev, "regulator enable failed\n"); + dev_err(dev, "regulator enable failed\n"); return error; } - error = devm_add_action_or_reset(&client->dev, bu21013_power_off, ts); + error = devm_add_action_or_reset(dev, bu21013_power_off, ts); if (error) { - dev_err(&client->dev, "failed to install power off handler\n"); + dev_err(dev, "failed to install power off handler\n"); return error; } /* Named "CS" on the chip, DT binding is "reset" */ - ts->cs_gpiod = devm_gpiod_get(&client->dev, "reset", GPIOD_OUT_HIGH); - error = PTR_ERR_OR_ZERO(ts->cs_gpiod); - if (error) { - if (error != -EPROBE_DEFER) - dev_err(&client->dev, "failed to get CS GPIO\n"); - return error; - } + ts->cs_gpiod = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH); + if (IS_ERR(ts->cs_gpiod)) + return dev_err_probe(dev, PTR_ERR(ts->cs_gpiod), "failed to get CS GPIO\n"); + gpiod_set_consumer_name(ts->cs_gpiod, "BU21013 CS"); - error = devm_add_action_or_reset(&client->dev, - bu21013_disable_chip, ts); + error = devm_add_action_or_reset(dev, bu21013_disable_chip, ts); if (error) { - dev_err(&client->dev, - "failed to install chip disable handler\n"); + dev_err(dev, "failed to install chip disable handler\n"); return error; } /* Named "INT" on the chip, DT binding is "touch" */ - ts->int_gpiod = devm_gpiod_get_optional(&client->dev, - "touch", GPIOD_IN); + ts->int_gpiod = devm_gpiod_get_optional(dev, "touch", GPIOD_IN); error = PTR_ERR_OR_ZERO(ts->int_gpiod); - if (error) { - if (error != -EPROBE_DEFER) - dev_err(&client->dev, "failed to get INT GPIO\n"); - return error; - } + if (error) + return dev_err_probe(dev, error, "failed to get INT GPIO\n"); if (ts->int_gpiod) gpiod_set_consumer_name(ts->int_gpiod, "BU21013 INT"); @@ -527,22 +519,20 @@ static int bu21013_probe(struct i2c_client *client) /* configure the touch panel controller */ error = bu21013_init_chip(ts); if (error) { - dev_err(&client->dev, "error in bu21013 config\n"); + dev_err(dev, "error in bu21013 config\n"); return error; } - error = devm_request_threaded_irq(&client->dev, client->irq, - NULL, bu21013_gpio_irq, + error = devm_request_threaded_irq(dev, client->irq, NULL, bu21013_gpio_irq, IRQF_ONESHOT, DRIVER_TP, ts); if (error) { - dev_err(&client->dev, "request irq %d failed\n", - client->irq); + dev_err(dev, "request irq %d failed\n", client->irq); return error; } error = input_register_device(in_dev); if (error) { - dev_err(&client->dev, "failed to register input device\n"); + dev_err(dev, "failed to register input device\n"); return error; } diff --git a/drivers/input/touchscreen/bu21029_ts.c b/drivers/input/touchscreen/bu21029_ts.c index c8126d2efe95..e1dfbd92ab64 100644 --- a/drivers/input/touchscreen/bu21029_ts.c +++ b/drivers/input/touchscreen/bu21029_ts.c @@ -333,6 +333,7 @@ static void bu21029_stop_chip(struct input_dev *dev) static int bu21029_probe(struct i2c_client *client) { + struct device *dev = &client->dev; struct bu21029_ts_data *bu21029; struct input_dev *in_dev; int error; @@ -341,45 +342,33 @@ static int bu21029_probe(struct i2c_client *client) I2C_FUNC_SMBUS_WRITE_BYTE | I2C_FUNC_SMBUS_WRITE_BYTE_DATA | I2C_FUNC_SMBUS_READ_I2C_BLOCK)) { - dev_err(&client->dev, - "i2c functionality support is not sufficient\n"); + dev_err(dev, "i2c functionality support is not sufficient\n"); return -EIO; } - bu21029 = devm_kzalloc(&client->dev, sizeof(*bu21029), GFP_KERNEL); + bu21029 = devm_kzalloc(dev, sizeof(*bu21029), GFP_KERNEL); if (!bu21029) return -ENOMEM; - error = device_property_read_u32(&client->dev, "rohm,x-plate-ohms", - &bu21029->x_plate_ohms); + error = device_property_read_u32(dev, "rohm,x-plate-ohms", &bu21029->x_plate_ohms); if (error) { - dev_err(&client->dev, - "invalid 'x-plate-ohms' supplied: %d\n", error); + dev_err(dev, "invalid 'x-plate-ohms' supplied: %d\n", error); return error; } - bu21029->vdd = devm_regulator_get(&client->dev, "vdd"); - if (IS_ERR(bu21029->vdd)) { - error = PTR_ERR(bu21029->vdd); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "failed to acquire 'vdd' supply: %d\n", error); - return error; - } + bu21029->vdd = devm_regulator_get(dev, "vdd"); + if (IS_ERR(bu21029->vdd)) + return dev_err_probe(dev, PTR_ERR(bu21029->vdd), + "failed to acquire 'vdd' supply\n"); - bu21029->reset_gpios = devm_gpiod_get_optional(&client->dev, - "reset", GPIOD_OUT_HIGH); - if (IS_ERR(bu21029->reset_gpios)) { - error = PTR_ERR(bu21029->reset_gpios); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "failed to acquire 'reset' gpio: %d\n", error); - return error; - } + bu21029->reset_gpios = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_HIGH); + if (IS_ERR(bu21029->reset_gpios)) + return dev_err_probe(dev, PTR_ERR(bu21029->reset_gpios), + "failed to acquire 'reset' gpio\n"); - in_dev = devm_input_allocate_device(&client->dev); + in_dev = devm_input_allocate_device(dev); if (!in_dev) { - dev_err(&client->dev, "unable to allocate input device\n"); + dev_err(dev, "unable to allocate input device\n"); return -ENOMEM; } @@ -400,20 +389,18 @@ static int bu21029_probe(struct i2c_client *client) input_set_drvdata(in_dev, bu21029); - error = devm_request_threaded_irq(&client->dev, client->irq, - NULL, bu21029_touch_soft_irq, + error = devm_request_threaded_irq(dev, client->irq, NULL, + bu21029_touch_soft_irq, IRQF_ONESHOT | IRQF_NO_AUTOEN, DRIVER_NAME, bu21029); if (error) { - dev_err(&client->dev, - "unable to request touch irq: %d\n", error); + dev_err(dev, "unable to request touch irq: %d\n", error); return error; } error = input_register_device(in_dev); if (error) { - dev_err(&client->dev, - "unable to register input device: %d\n", error); + dev_err(dev, "unable to register input device: %d\n", error); return error; } diff --git a/drivers/input/touchscreen/chipone_icn8318.c b/drivers/input/touchscreen/chipone_icn8318.c index 9fbeaf17f00b..d6876d10b252 100644 --- a/drivers/input/touchscreen/chipone_icn8318.c +++ b/drivers/input/touchscreen/chipone_icn8318.c @@ -191,12 +191,8 @@ static int icn8318_probe(struct i2c_client *client) return -ENOMEM; data->wake_gpio = devm_gpiod_get(dev, "wake", GPIOD_OUT_LOW); - if (IS_ERR(data->wake_gpio)) { - error = PTR_ERR(data->wake_gpio); - if (error != -EPROBE_DEFER) - dev_err(dev, "Error getting wake gpio: %d\n", error); - return error; - } + if (IS_ERR(data->wake_gpio)) + return dev_err_probe(dev, PTR_ERR(data->wake_gpio), "Error getting wake gpio\n"); input = devm_input_allocate_device(dev); if (!input) diff --git a/drivers/input/touchscreen/cy8ctma140.c b/drivers/input/touchscreen/cy8ctma140.c index 967ecde23e83..ea3895167b82 100644 --- a/drivers/input/touchscreen/cy8ctma140.c +++ b/drivers/input/touchscreen/cy8ctma140.c @@ -258,12 +258,8 @@ static int cy8ctma140_probe(struct i2c_client *client) ts->regulators[1].supply = "vdd"; error = devm_regulator_bulk_get(dev, ARRAY_SIZE(ts->regulators), ts->regulators); - if (error) { - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to get regulators %d\n", - error); - return error; - } + if (error) + return dev_err_probe(dev, error, "Failed to get regulators\n"); error = cy8ctma140_power_up(ts); if (error) diff --git a/drivers/input/touchscreen/cyttsp5.c b/drivers/input/touchscreen/cyttsp5.c index b461ded946fc..db5a885ecd72 100644 --- a/drivers/input/touchscreen/cyttsp5.c +++ b/drivers/input/touchscreen/cyttsp5.c @@ -18,8 +18,8 @@ #include #include #include +#include #include -#include #include #include diff --git a/drivers/input/touchscreen/edt-ft5x06.c b/drivers/input/touchscreen/edt-ft5x06.c index 795c7dad22bf..457d53337fbb 100644 --- a/drivers/input/touchscreen/edt-ft5x06.c +++ b/drivers/input/touchscreen/edt-ft5x06.c @@ -1168,13 +1168,9 @@ static int edt_ft5x06_ts_probe(struct i2c_client *client) tsdata->max_support_points = chip_data->max_support_points; tsdata->vcc = devm_regulator_get(&client->dev, "vcc"); - if (IS_ERR(tsdata->vcc)) { - error = PTR_ERR(tsdata->vcc); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "failed to request regulator: %d\n", error); - return error; - } + if (IS_ERR(tsdata->vcc)) + return dev_err_probe(&client->dev, PTR_ERR(tsdata->vcc), + "failed to request regulator\n"); tsdata->iovcc = devm_regulator_get(&client->dev, "iovcc"); if (IS_ERR(tsdata->iovcc)) { diff --git a/drivers/input/touchscreen/ektf2127.c b/drivers/input/touchscreen/ektf2127.c index fd8724a3c19f..cc3103b9cbfb 100644 --- a/drivers/input/touchscreen/ektf2127.c +++ b/drivers/input/touchscreen/ektf2127.c @@ -264,12 +264,8 @@ static int ektf2127_probe(struct i2c_client *client) /* This requests the gpio *and* turns on the touchscreen controller */ ts->power_gpios = devm_gpiod_get(dev, "power", GPIOD_OUT_HIGH); - if (IS_ERR(ts->power_gpios)) { - error = PTR_ERR(ts->power_gpios); - if (error != -EPROBE_DEFER) - dev_err(dev, "Error getting power gpio: %d\n", error); - return error; - } + if (IS_ERR(ts->power_gpios)) + return dev_err_probe(dev, PTR_ERR(ts->power_gpios), "Error getting power gpio\n"); input = devm_input_allocate_device(dev); if (!input) diff --git a/drivers/input/touchscreen/elants_i2c.c b/drivers/input/touchscreen/elants_i2c.c index 2da1db64126d..a1af3de9f310 100644 --- a/drivers/input/touchscreen/elants_i2c.c +++ b/drivers/input/touchscreen/elants_i2c.c @@ -1438,24 +1438,14 @@ static int elants_i2c_probe(struct i2c_client *client) i2c_set_clientdata(client, ts); ts->vcc33 = devm_regulator_get(&client->dev, "vcc33"); - if (IS_ERR(ts->vcc33)) { - error = PTR_ERR(ts->vcc33); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "Failed to get 'vcc33' regulator: %d\n", - error); - return error; - } + if (IS_ERR(ts->vcc33)) + return dev_err_probe(&client->dev, PTR_ERR(ts->vcc33), + "Failed to get 'vcc33' regulator\n"); ts->vccio = devm_regulator_get(&client->dev, "vccio"); - if (IS_ERR(ts->vccio)) { - error = PTR_ERR(ts->vccio); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "Failed to get 'vccio' regulator: %d\n", - error); - return error; - } + if (IS_ERR(ts->vccio)) + return dev_err_probe(&client->dev, PTR_ERR(ts->vccio), + "Failed to get 'vccio' regulator\n"); ts->reset_gpio = devm_gpiod_get(&client->dev, "reset", GPIOD_OUT_HIGH); if (IS_ERR(ts->reset_gpio)) { diff --git a/drivers/input/touchscreen/exc3000.c b/drivers/input/touchscreen/exc3000.c index 4af4c1e5d0da..4c0d99aae9e0 100644 --- a/drivers/input/touchscreen/exc3000.c +++ b/drivers/input/touchscreen/exc3000.c @@ -7,6 +7,7 @@ * minimal implementation based on egalax_ts.c and egalax_i2c.c */ +#include #include #include #include @@ -18,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -360,6 +362,12 @@ static int exc3000_probe(struct i2c_client *client) if (IS_ERR(data->reset)) return PTR_ERR(data->reset); + /* For proper reset sequence, enable power while reset asserted */ + error = devm_regulator_get_enable(&client->dev, "vdd"); + if (error && error != -ENODEV) + return dev_err_probe(&client->dev, error, + "failed to request vdd regulator\n"); + if (data->reset) { msleep(EXC3000_RESET_MS); gpiod_set_value_cansleep(data->reset, 0); @@ -454,10 +462,19 @@ static const struct of_device_id exc3000_of_match[] = { MODULE_DEVICE_TABLE(of, exc3000_of_match); #endif +#ifdef CONFIG_ACPI +static const struct acpi_device_id exc3000_acpi_match[] = { + { "EGA00001", .driver_data = (kernel_ulong_t)&exc3000_info[EETI_EXC80H60] }, + { } +}; +MODULE_DEVICE_TABLE(acpi, exc3000_acpi_match); +#endif + static struct i2c_driver exc3000_driver = { .driver = { .name = "exc3000", .of_match_table = of_match_ptr(exc3000_of_match), + .acpi_match_table = ACPI_PTR(exc3000_acpi_match), }, .id_table = exc3000_id, .probe = exc3000_probe, diff --git a/drivers/input/touchscreen/goodix.c b/drivers/input/touchscreen/goodix.c index f5aa240739f9..da9954d6df44 100644 --- a/drivers/input/touchscreen/goodix.c +++ b/drivers/input/touchscreen/goodix.c @@ -935,7 +935,6 @@ static int goodix_add_acpi_gpio_mappings(struct goodix_ts_data *ts) */ static int goodix_get_gpio_config(struct goodix_ts_data *ts) { - int error; struct device *dev; struct gpio_desc *gpiod; bool added_acpi_mappings = false; @@ -951,33 +950,20 @@ static int goodix_get_gpio_config(struct goodix_ts_data *ts) ts->gpiod_rst_flags = GPIOD_IN; ts->avdd28 = devm_regulator_get(dev, "AVDD28"); - if (IS_ERR(ts->avdd28)) { - error = PTR_ERR(ts->avdd28); - if (error != -EPROBE_DEFER) - dev_err(dev, - "Failed to get AVDD28 regulator: %d\n", error); - return error; - } + if (IS_ERR(ts->avdd28)) + return dev_err_probe(dev, PTR_ERR(ts->avdd28), "Failed to get AVDD28 regulator\n"); ts->vddio = devm_regulator_get(dev, "VDDIO"); - if (IS_ERR(ts->vddio)) { - error = PTR_ERR(ts->vddio); - if (error != -EPROBE_DEFER) - dev_err(dev, - "Failed to get VDDIO regulator: %d\n", error); - return error; - } + if (IS_ERR(ts->vddio)) + return dev_err_probe(dev, PTR_ERR(ts->vddio), "Failed to get VDDIO regulator\n"); retry_get_irq_gpio: /* Get the interrupt GPIO pin number */ gpiod = devm_gpiod_get_optional(dev, GOODIX_GPIO_INT_NAME, GPIOD_IN); - if (IS_ERR(gpiod)) { - error = PTR_ERR(gpiod); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to get %s GPIO: %d\n", - GOODIX_GPIO_INT_NAME, error); - return error; - } + if (IS_ERR(gpiod)) + return dev_err_probe(dev, PTR_ERR(gpiod), "Failed to get %s GPIO\n", + GOODIX_GPIO_INT_NAME); + if (!gpiod && has_acpi_companion(dev) && !added_acpi_mappings) { added_acpi_mappings = true; if (goodix_add_acpi_gpio_mappings(ts) == 0) @@ -988,13 +974,9 @@ retry_get_irq_gpio: /* Get the reset line GPIO pin number */ gpiod = devm_gpiod_get_optional(dev, GOODIX_GPIO_RST_NAME, ts->gpiod_rst_flags); - if (IS_ERR(gpiod)) { - error = PTR_ERR(gpiod); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to get %s GPIO: %d\n", - GOODIX_GPIO_RST_NAME, error); - return error; - } + if (IS_ERR(gpiod)) + return dev_err_probe(dev, PTR_ERR(gpiod), "Failed to get %s GPIO\n", + GOODIX_GPIO_RST_NAME); ts->gpiod_rst = gpiod; @@ -1517,6 +1499,7 @@ MODULE_DEVICE_TABLE(i2c, goodix_ts_id); static const struct acpi_device_id goodix_acpi_match[] = { { "GDIX1001", 0 }, { "GDIX1002", 0 }, + { "GDX9110", 0 }, { } }; MODULE_DEVICE_TABLE(acpi, goodix_acpi_match); diff --git a/drivers/input/touchscreen/ili210x.c b/drivers/input/touchscreen/ili210x.c index f7cd773f7292..ad6828e4f2e2 100644 --- a/drivers/input/touchscreen/ili210x.c +++ b/drivers/input/touchscreen/ili210x.c @@ -8,8 +8,8 @@ #include #include #include +#include #include -#include #include #include #include diff --git a/drivers/input/touchscreen/iqs5xx.c b/drivers/input/touchscreen/iqs5xx.c index 0aa9d6492df8..b4768b66eb10 100644 --- a/drivers/input/touchscreen/iqs5xx.c +++ b/drivers/input/touchscreen/iqs5xx.c @@ -23,8 +23,8 @@ #include #include #include +#include #include -#include #include #include diff --git a/drivers/input/touchscreen/iqs7211.c b/drivers/input/touchscreen/iqs7211.c new file mode 100644 index 000000000000..dc084f873762 --- /dev/null +++ b/drivers/input/touchscreen/iqs7211.c @@ -0,0 +1,2557 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Azoteq IQS7210A/7211A/E Trackpad/Touchscreen Controller + * + * Copyright (C) 2023 Jeff LaBundy + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define IQS7211_PROD_NUM 0x00 + +#define IQS7211_EVENT_MASK_ALL GENMASK(14, 8) +#define IQS7211_EVENT_MASK_ALP BIT(13) +#define IQS7211_EVENT_MASK_BTN BIT(12) +#define IQS7211_EVENT_MASK_ATI BIT(11) +#define IQS7211_EVENT_MASK_MOVE BIT(10) +#define IQS7211_EVENT_MASK_GSTR BIT(9) +#define IQS7211_EVENT_MODE BIT(8) + +#define IQS7211_COMMS_ERROR 0xEEEE +#define IQS7211_COMMS_RETRY_MS 50 +#define IQS7211_COMMS_SLEEP_US 100 +#define IQS7211_COMMS_TIMEOUT_US (100 * USEC_PER_MSEC) +#define IQS7211_RESET_TIMEOUT_MS 150 +#define IQS7211_START_TIMEOUT_US (1 * USEC_PER_SEC) + +#define IQS7211_NUM_RETRIES 5 +#define IQS7211_NUM_CRX 8 +#define IQS7211_MAX_CTX 13 + +#define IQS7211_MAX_CONTACTS 2 +#define IQS7211_MAX_CYCLES 21 + +/* + * The following delay is used during instances that must wait for the open- + * drain RDY pin to settle. Its value is calculated as 5*R*C, where R and C + * represent typical datasheet values of 4.7k and 100 nF, respectively. + */ +#define iqs7211_irq_wait() usleep_range(2500, 2600) + +enum iqs7211_dev_id { + IQS7210A, + IQS7211A, + IQS7211E, +}; + +enum iqs7211_comms_mode { + IQS7211_COMMS_MODE_WAIT, + IQS7211_COMMS_MODE_FREE, + IQS7211_COMMS_MODE_FORCE, +}; + +struct iqs7211_reg_field_desc { + struct list_head list; + u8 addr; + u16 mask; + u16 val; +}; + +enum iqs7211_reg_key_id { + IQS7211_REG_KEY_NONE, + IQS7211_REG_KEY_PROX, + IQS7211_REG_KEY_TOUCH, + IQS7211_REG_KEY_TAP, + IQS7211_REG_KEY_HOLD, + IQS7211_REG_KEY_PALM, + IQS7211_REG_KEY_AXIAL_X, + IQS7211_REG_KEY_AXIAL_Y, + IQS7211_REG_KEY_RESERVED +}; + +enum iqs7211_reg_grp_id { + IQS7211_REG_GRP_TP, + IQS7211_REG_GRP_BTN, + IQS7211_REG_GRP_ALP, + IQS7211_REG_GRP_SYS, + IQS7211_NUM_REG_GRPS +}; + +static const char * const iqs7211_reg_grp_names[IQS7211_NUM_REG_GRPS] = { + [IQS7211_REG_GRP_TP] = "trackpad", + [IQS7211_REG_GRP_BTN] = "button", + [IQS7211_REG_GRP_ALP] = "alp", +}; + +static const u16 iqs7211_reg_grp_masks[IQS7211_NUM_REG_GRPS] = { + [IQS7211_REG_GRP_TP] = IQS7211_EVENT_MASK_GSTR, + [IQS7211_REG_GRP_BTN] = IQS7211_EVENT_MASK_BTN, + [IQS7211_REG_GRP_ALP] = IQS7211_EVENT_MASK_ALP, +}; + +struct iqs7211_event_desc { + const char *name; + u16 mask; + u16 enable; + enum iqs7211_reg_grp_id reg_grp; + enum iqs7211_reg_key_id reg_key; +}; + +static const struct iqs7211_event_desc iqs7210a_kp_events[] = { + { + .mask = BIT(10), + .enable = BIT(13) | BIT(12), + .reg_grp = IQS7211_REG_GRP_ALP, + }, + { + .name = "event-prox", + .mask = BIT(2), + .enable = BIT(5) | BIT(4), + .reg_grp = IQS7211_REG_GRP_BTN, + .reg_key = IQS7211_REG_KEY_PROX, + }, + { + .name = "event-touch", + .mask = BIT(3), + .enable = BIT(5) | BIT(4), + .reg_grp = IQS7211_REG_GRP_BTN, + .reg_key = IQS7211_REG_KEY_TOUCH, + }, + { + .name = "event-tap", + .mask = BIT(0), + .enable = BIT(0), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_TAP, + }, + { + .name = "event-hold", + .mask = BIT(1), + .enable = BIT(1), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_HOLD, + }, + { + .name = "event-swipe-x-neg", + .mask = BIT(2), + .enable = BIT(2), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_X, + }, + { + .name = "event-swipe-x-pos", + .mask = BIT(3), + .enable = BIT(3), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_X, + }, + { + .name = "event-swipe-y-pos", + .mask = BIT(4), + .enable = BIT(4), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + }, + { + .name = "event-swipe-y-neg", + .mask = BIT(5), + .enable = BIT(5), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + }, +}; + +static const struct iqs7211_event_desc iqs7211a_kp_events[] = { + { + .mask = BIT(14), + .reg_grp = IQS7211_REG_GRP_ALP, + }, + { + .name = "event-tap", + .mask = BIT(0), + .enable = BIT(0), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_TAP, + }, + { + .name = "event-hold", + .mask = BIT(1), + .enable = BIT(1), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_HOLD, + }, + { + .name = "event-swipe-x-neg", + .mask = BIT(2), + .enable = BIT(2), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_X, + }, + { + .name = "event-swipe-x-pos", + .mask = BIT(3), + .enable = BIT(3), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_X, + }, + { + .name = "event-swipe-y-pos", + .mask = BIT(4), + .enable = BIT(4), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + }, + { + .name = "event-swipe-y-neg", + .mask = BIT(5), + .enable = BIT(5), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + }, +}; + +static const struct iqs7211_event_desc iqs7211e_kp_events[] = { + { + .mask = BIT(14), + .reg_grp = IQS7211_REG_GRP_ALP, + }, + { + .name = "event-tap", + .mask = BIT(0), + .enable = BIT(0), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_TAP, + }, + { + .name = "event-tap-double", + .mask = BIT(1), + .enable = BIT(1), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_TAP, + }, + { + .name = "event-tap-triple", + .mask = BIT(2), + .enable = BIT(2), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_TAP, + }, + { + .name = "event-hold", + .mask = BIT(3), + .enable = BIT(3), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_HOLD, + }, + { + .name = "event-palm", + .mask = BIT(4), + .enable = BIT(4), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_PALM, + }, + { + .name = "event-swipe-x-pos", + .mask = BIT(8), + .enable = BIT(8), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_X, + }, + { + .name = "event-swipe-x-neg", + .mask = BIT(9), + .enable = BIT(9), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_X, + }, + { + .name = "event-swipe-y-pos", + .mask = BIT(10), + .enable = BIT(10), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + }, + { + .name = "event-swipe-y-neg", + .mask = BIT(11), + .enable = BIT(11), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + }, + { + .name = "event-swipe-x-pos-hold", + .mask = BIT(12), + .enable = BIT(12), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_HOLD, + }, + { + .name = "event-swipe-x-neg-hold", + .mask = BIT(13), + .enable = BIT(13), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_HOLD, + }, + { + .name = "event-swipe-y-pos-hold", + .mask = BIT(14), + .enable = BIT(14), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_HOLD, + }, + { + .name = "event-swipe-y-neg-hold", + .mask = BIT(15), + .enable = BIT(15), + .reg_grp = IQS7211_REG_GRP_TP, + .reg_key = IQS7211_REG_KEY_HOLD, + }, +}; + +struct iqs7211_dev_desc { + const char *tp_name; + const char *kp_name; + u16 prod_num; + u16 show_reset; + u16 ati_error[IQS7211_NUM_REG_GRPS]; + u16 ati_start[IQS7211_NUM_REG_GRPS]; + u16 suspend; + u16 ack_reset; + u16 comms_end; + u16 comms_req; + int charge_shift; + int info_offs; + int gesture_offs; + int contact_offs; + u8 sys_stat; + u8 sys_ctrl; + u8 alp_config; + u8 tp_config; + u8 exp_file; + u8 kp_enable[IQS7211_NUM_REG_GRPS]; + u8 gesture_angle; + u8 rx_tx_map; + u8 cycle_alloc[2]; + u8 cycle_limit[2]; + const struct iqs7211_event_desc *kp_events; + int num_kp_events; + int min_crx_alp; + int num_ctx; +}; + +static const struct iqs7211_dev_desc iqs7211_devs[] = { + [IQS7210A] = { + .tp_name = "iqs7210a_trackpad", + .kp_name = "iqs7210a_keys", + .prod_num = 944, + .show_reset = BIT(15), + .ati_error = { + [IQS7211_REG_GRP_TP] = BIT(12), + [IQS7211_REG_GRP_BTN] = BIT(0), + [IQS7211_REG_GRP_ALP] = BIT(8), + }, + .ati_start = { + [IQS7211_REG_GRP_TP] = BIT(13), + [IQS7211_REG_GRP_BTN] = BIT(1), + [IQS7211_REG_GRP_ALP] = BIT(9), + }, + .suspend = BIT(11), + .ack_reset = BIT(7), + .comms_end = BIT(2), + .comms_req = BIT(1), + .charge_shift = 4, + .info_offs = 0, + .gesture_offs = 1, + .contact_offs = 4, + .sys_stat = 0x0A, + .sys_ctrl = 0x35, + .alp_config = 0x39, + .tp_config = 0x4E, + .exp_file = 0x57, + .kp_enable = { + [IQS7211_REG_GRP_TP] = 0x58, + [IQS7211_REG_GRP_BTN] = 0x37, + [IQS7211_REG_GRP_ALP] = 0x37, + }, + .gesture_angle = 0x5F, + .rx_tx_map = 0x60, + .cycle_alloc = { 0x66, 0x75, }, + .cycle_limit = { 10, 6, }, + .kp_events = iqs7210a_kp_events, + .num_kp_events = ARRAY_SIZE(iqs7210a_kp_events), + .min_crx_alp = 4, + .num_ctx = IQS7211_MAX_CTX - 1, + }, + [IQS7211A] = { + .tp_name = "iqs7211a_trackpad", + .kp_name = "iqs7211a_keys", + .prod_num = 763, + .show_reset = BIT(7), + .ati_error = { + [IQS7211_REG_GRP_TP] = BIT(3), + [IQS7211_REG_GRP_ALP] = BIT(5), + }, + .ati_start = { + [IQS7211_REG_GRP_TP] = BIT(5), + [IQS7211_REG_GRP_ALP] = BIT(6), + }, + .ack_reset = BIT(7), + .comms_req = BIT(4), + .charge_shift = 0, + .info_offs = 0, + .gesture_offs = 1, + .contact_offs = 4, + .sys_stat = 0x10, + .sys_ctrl = 0x50, + .tp_config = 0x60, + .alp_config = 0x72, + .exp_file = 0x74, + .kp_enable = { + [IQS7211_REG_GRP_TP] = 0x80, + }, + .gesture_angle = 0x87, + .rx_tx_map = 0x90, + .cycle_alloc = { 0xA0, 0xB0, }, + .cycle_limit = { 10, 8, }, + .kp_events = iqs7211a_kp_events, + .num_kp_events = ARRAY_SIZE(iqs7211a_kp_events), + .num_ctx = IQS7211_MAX_CTX - 1, + }, + [IQS7211E] = { + .tp_name = "iqs7211e_trackpad", + .kp_name = "iqs7211e_keys", + .prod_num = 1112, + .show_reset = BIT(7), + .ati_error = { + [IQS7211_REG_GRP_TP] = BIT(3), + [IQS7211_REG_GRP_ALP] = BIT(5), + }, + .ati_start = { + [IQS7211_REG_GRP_TP] = BIT(5), + [IQS7211_REG_GRP_ALP] = BIT(6), + }, + .suspend = BIT(11), + .ack_reset = BIT(7), + .comms_end = BIT(6), + .comms_req = BIT(4), + .charge_shift = 0, + .info_offs = 1, + .gesture_offs = 0, + .contact_offs = 2, + .sys_stat = 0x0E, + .sys_ctrl = 0x33, + .tp_config = 0x41, + .alp_config = 0x36, + .exp_file = 0x4A, + .kp_enable = { + [IQS7211_REG_GRP_TP] = 0x4B, + }, + .gesture_angle = 0x55, + .rx_tx_map = 0x56, + .cycle_alloc = { 0x5D, 0x6C, }, + .cycle_limit = { 10, 11, }, + .kp_events = iqs7211e_kp_events, + .num_kp_events = ARRAY_SIZE(iqs7211e_kp_events), + .num_ctx = IQS7211_MAX_CTX, + }, +}; + +struct iqs7211_prop_desc { + const char *name; + enum iqs7211_reg_key_id reg_key; + u8 reg_addr[IQS7211_NUM_REG_GRPS][ARRAY_SIZE(iqs7211_devs)]; + int reg_shift; + int reg_width; + int val_pitch; + int val_min; + int val_max; + const char *label; +}; + +static const struct iqs7211_prop_desc iqs7211_props[] = { + { + .name = "azoteq,ati-frac-div-fine", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x1E, + [IQS7211A] = 0x30, + [IQS7211E] = 0x21, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x22, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x23, + [IQS7211A] = 0x36, + [IQS7211E] = 0x25, + }, + }, + .reg_shift = 9, + .reg_width = 5, + .label = "ATI fine fractional divider", + }, + { + .name = "azoteq,ati-frac-mult-coarse", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x1E, + [IQS7211A] = 0x30, + [IQS7211E] = 0x21, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x22, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x23, + [IQS7211A] = 0x36, + [IQS7211E] = 0x25, + }, + }, + .reg_shift = 5, + .reg_width = 4, + .label = "ATI coarse fractional multiplier", + }, + { + .name = "azoteq,ati-frac-div-coarse", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x1E, + [IQS7211A] = 0x30, + [IQS7211E] = 0x21, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x22, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x23, + [IQS7211A] = 0x36, + [IQS7211E] = 0x25, + }, + }, + .reg_shift = 0, + .reg_width = 5, + .label = "ATI coarse fractional divider", + }, + { + .name = "azoteq,ati-comp-div", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x1F, + [IQS7211E] = 0x22, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x24, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7211E] = 0x26, + }, + }, + .reg_shift = 0, + .reg_width = 8, + .val_max = 31, + .label = "ATI compensation divider", + }, + { + .name = "azoteq,ati-comp-div", + .reg_addr = { + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x24, + }, + }, + .reg_shift = 8, + .reg_width = 8, + .val_max = 31, + .label = "ATI compensation divider", + }, + { + .name = "azoteq,ati-comp-div", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7211A] = 0x31, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7211A] = 0x37, + }, + }, + .val_max = 31, + .label = "ATI compensation divider", + }, + { + .name = "azoteq,ati-target", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x20, + [IQS7211A] = 0x32, + [IQS7211E] = 0x23, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x27, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x28, + [IQS7211A] = 0x38, + [IQS7211E] = 0x27, + }, + }, + .label = "ATI target", + }, + { + .name = "azoteq,ati-base", + .reg_addr[IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x26, + }, + .reg_shift = 8, + .reg_width = 8, + .val_pitch = 8, + .label = "ATI base", + }, + { + .name = "azoteq,ati-base", + .reg_addr[IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x26, + }, + .reg_shift = 0, + .reg_width = 8, + .val_pitch = 8, + .label = "ATI base", + }, + { + .name = "azoteq,rate-active-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x29, + [IQS7211A] = 0x40, + [IQS7211E] = 0x28, + }, + .label = "active mode report rate", + }, + { + .name = "azoteq,rate-touch-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x2A, + [IQS7211A] = 0x41, + [IQS7211E] = 0x29, + }, + .label = "idle-touch mode report rate", + }, + { + .name = "azoteq,rate-idle-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x2B, + [IQS7211A] = 0x42, + [IQS7211E] = 0x2A, + }, + .label = "idle mode report rate", + }, + { + .name = "azoteq,rate-lp1-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x2C, + [IQS7211A] = 0x43, + [IQS7211E] = 0x2B, + }, + .label = "low-power mode 1 report rate", + }, + { + .name = "azoteq,rate-lp2-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x2D, + [IQS7211A] = 0x44, + [IQS7211E] = 0x2C, + }, + .label = "low-power mode 2 report rate", + }, + { + .name = "azoteq,timeout-active-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x2E, + [IQS7211A] = 0x45, + [IQS7211E] = 0x2D, + }, + .val_pitch = 1000, + .label = "active mode timeout", + }, + { + .name = "azoteq,timeout-touch-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x2F, + [IQS7211A] = 0x46, + [IQS7211E] = 0x2E, + }, + .val_pitch = 1000, + .label = "idle-touch mode timeout", + }, + { + .name = "azoteq,timeout-idle-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x30, + [IQS7211A] = 0x47, + [IQS7211E] = 0x2F, + }, + .val_pitch = 1000, + .label = "idle mode timeout", + }, + { + .name = "azoteq,timeout-lp1-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x31, + [IQS7211A] = 0x48, + [IQS7211E] = 0x30, + }, + .val_pitch = 1000, + .label = "low-power mode 1 timeout", + }, + { + .name = "azoteq,timeout-lp2-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x32, + [IQS7211E] = 0x31, + }, + .reg_shift = 8, + .reg_width = 8, + .val_pitch = 1000, + .val_max = 60000, + .label = "trackpad reference value update rate", + }, + { + .name = "azoteq,timeout-lp2-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7211A] = 0x49, + }, + .val_pitch = 1000, + .val_max = 60000, + .label = "trackpad reference value update rate", + }, + { + .name = "azoteq,timeout-ati-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x32, + [IQS7211E] = 0x31, + }, + .reg_width = 8, + .val_pitch = 1000, + .val_max = 60000, + .label = "ATI error timeout", + }, + { + .name = "azoteq,timeout-ati-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7211A] = 0x35, + }, + .val_pitch = 1000, + .val_max = 60000, + .label = "ATI error timeout", + }, + { + .name = "azoteq,timeout-comms-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x33, + [IQS7211A] = 0x4A, + [IQS7211E] = 0x32, + }, + .label = "communication timeout", + }, + { + .name = "azoteq,timeout-press-ms", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x34, + }, + .reg_width = 8, + .val_pitch = 1000, + .val_max = 60000, + .label = "press timeout", + }, + { + .name = "azoteq,ati-mode", + .reg_addr[IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x37, + }, + .reg_shift = 15, + .reg_width = 1, + .label = "ATI mode", + }, + { + .name = "azoteq,ati-mode", + .reg_addr[IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x37, + }, + .reg_shift = 7, + .reg_width = 1, + .label = "ATI mode", + }, + { + .name = "azoteq,sense-mode", + .reg_addr[IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x37, + [IQS7211A] = 0x72, + [IQS7211E] = 0x36, + }, + .reg_shift = 8, + .reg_width = 1, + .label = "sensing mode", + }, + { + .name = "azoteq,sense-mode", + .reg_addr[IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x37, + }, + .reg_shift = 0, + .reg_width = 2, + .val_max = 2, + .label = "sensing mode", + }, + { + .name = "azoteq,fosc-freq", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x38, + [IQS7211A] = 0x52, + [IQS7211E] = 0x35, + }, + .reg_shift = 4, + .reg_width = 1, + .label = "core clock frequency selection", + }, + { + .name = "azoteq,fosc-trim", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x38, + [IQS7211A] = 0x52, + [IQS7211E] = 0x35, + }, + .reg_shift = 0, + .reg_width = 4, + .label = "core clock frequency trim", + }, + { + .name = "azoteq,touch-exit", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x3B, + [IQS7211A] = 0x53, + [IQS7211E] = 0x38, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x3E, + }, + }, + .reg_shift = 8, + .reg_width = 8, + .label = "touch exit factor", + }, + { + .name = "azoteq,touch-enter", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x3B, + [IQS7211A] = 0x53, + [IQS7211E] = 0x38, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x3E, + }, + }, + .reg_shift = 0, + .reg_width = 8, + .label = "touch entrance factor", + }, + { + .name = "azoteq,thresh", + .reg_addr = { + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x3C, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x3D, + [IQS7211A] = 0x54, + [IQS7211E] = 0x39, + }, + }, + .label = "threshold", + }, + { + .name = "azoteq,debounce-exit", + .reg_addr = { + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x3F, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x40, + [IQS7211A] = 0x56, + [IQS7211E] = 0x3A, + }, + }, + .reg_shift = 8, + .reg_width = 8, + .label = "debounce exit factor", + }, + { + .name = "azoteq,debounce-enter", + .reg_addr = { + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x3F, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x40, + [IQS7211A] = 0x56, + [IQS7211E] = 0x3A, + }, + }, + .reg_shift = 0, + .reg_width = 8, + .label = "debounce entrance factor", + }, + { + .name = "azoteq,conv-frac", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x48, + [IQS7211A] = 0x58, + [IQS7211E] = 0x3D, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x49, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x4A, + [IQS7211A] = 0x59, + [IQS7211E] = 0x3E, + }, + }, + .reg_shift = 8, + .reg_width = 8, + .label = "conversion frequency fractional divider", + }, + { + .name = "azoteq,conv-period", + .reg_addr = { + [IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x48, + [IQS7211A] = 0x58, + [IQS7211E] = 0x3D, + }, + [IQS7211_REG_GRP_BTN] = { + [IQS7210A] = 0x49, + }, + [IQS7211_REG_GRP_ALP] = { + [IQS7210A] = 0x4A, + [IQS7211A] = 0x59, + [IQS7211E] = 0x3E, + }, + }, + .reg_shift = 0, + .reg_width = 8, + .label = "conversion period", + }, + { + .name = "azoteq,thresh", + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x55, + [IQS7211A] = 0x67, + [IQS7211E] = 0x48, + }, + .reg_shift = 0, + .reg_width = 8, + .label = "threshold", + }, + { + .name = "azoteq,contact-split", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x55, + [IQS7211A] = 0x67, + [IQS7211E] = 0x48, + }, + .reg_shift = 8, + .reg_width = 8, + .label = "contact split factor", + }, + { + .name = "azoteq,trim-x", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x56, + [IQS7211E] = 0x49, + }, + .reg_shift = 0, + .reg_width = 8, + .label = "horizontal trim width", + }, + { + .name = "azoteq,trim-x", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7211A] = 0x68, + }, + .label = "horizontal trim width", + }, + { + .name = "azoteq,trim-y", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7210A] = 0x56, + [IQS7211E] = 0x49, + }, + .reg_shift = 8, + .reg_width = 8, + .label = "vertical trim height", + }, + { + .name = "azoteq,trim-y", + .reg_addr[IQS7211_REG_GRP_SYS] = { + [IQS7211A] = 0x69, + }, + .label = "vertical trim height", + }, + { + .name = "azoteq,gesture-max-ms", + .reg_key = IQS7211_REG_KEY_TAP, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x59, + [IQS7211A] = 0x81, + [IQS7211E] = 0x4C, + }, + .label = "maximum gesture time", + }, + { + .name = "azoteq,gesture-mid-ms", + .reg_key = IQS7211_REG_KEY_TAP, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7211E] = 0x4D, + }, + .label = "repeated gesture time", + }, + { + .name = "azoteq,gesture-dist", + .reg_key = IQS7211_REG_KEY_TAP, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x5A, + [IQS7211A] = 0x82, + [IQS7211E] = 0x4E, + }, + .label = "gesture distance", + }, + { + .name = "azoteq,gesture-dist", + .reg_key = IQS7211_REG_KEY_HOLD, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x5A, + [IQS7211A] = 0x82, + [IQS7211E] = 0x4E, + }, + .label = "gesture distance", + }, + { + .name = "azoteq,gesture-min-ms", + .reg_key = IQS7211_REG_KEY_HOLD, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x5B, + [IQS7211A] = 0x83, + [IQS7211E] = 0x4F, + }, + .label = "minimum gesture time", + }, + { + .name = "azoteq,gesture-max-ms", + .reg_key = IQS7211_REG_KEY_AXIAL_X, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x5C, + [IQS7211A] = 0x84, + [IQS7211E] = 0x50, + }, + .label = "maximum gesture time", + }, + { + .name = "azoteq,gesture-max-ms", + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x5C, + [IQS7211A] = 0x84, + [IQS7211E] = 0x50, + }, + .label = "maximum gesture time", + }, + { + .name = "azoteq,gesture-dist", + .reg_key = IQS7211_REG_KEY_AXIAL_X, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x5D, + [IQS7211A] = 0x85, + [IQS7211E] = 0x51, + }, + .label = "gesture distance", + }, + { + .name = "azoteq,gesture-dist", + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7210A] = 0x5E, + [IQS7211A] = 0x86, + [IQS7211E] = 0x52, + }, + .label = "gesture distance", + }, + { + .name = "azoteq,gesture-dist-rep", + .reg_key = IQS7211_REG_KEY_AXIAL_X, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7211E] = 0x53, + }, + .label = "repeated gesture distance", + }, + { + .name = "azoteq,gesture-dist-rep", + .reg_key = IQS7211_REG_KEY_AXIAL_Y, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7211E] = 0x54, + }, + .label = "repeated gesture distance", + }, + { + .name = "azoteq,thresh", + .reg_key = IQS7211_REG_KEY_PALM, + .reg_addr[IQS7211_REG_GRP_TP] = { + [IQS7211E] = 0x55, + }, + .reg_shift = 8, + .reg_width = 8, + .val_max = 42, + .label = "threshold", + }, +}; + +static const u8 iqs7211_gesture_angle[] = { + 0x00, 0x01, 0x02, 0x03, + 0x04, 0x06, 0x07, 0x08, + 0x09, 0x0A, 0x0B, 0x0C, + 0x0E, 0x0F, 0x10, 0x11, + 0x12, 0x14, 0x15, 0x16, + 0x17, 0x19, 0x1A, 0x1B, + 0x1C, 0x1E, 0x1F, 0x21, + 0x22, 0x23, 0x25, 0x26, + 0x28, 0x2A, 0x2B, 0x2D, + 0x2E, 0x30, 0x32, 0x34, + 0x36, 0x38, 0x3A, 0x3C, + 0x3E, 0x40, 0x42, 0x45, + 0x47, 0x4A, 0x4C, 0x4F, + 0x52, 0x55, 0x58, 0x5B, + 0x5F, 0x63, 0x66, 0x6B, + 0x6F, 0x73, 0x78, 0x7E, + 0x83, 0x89, 0x90, 0x97, + 0x9E, 0xA7, 0xB0, 0xBA, + 0xC5, 0xD1, 0xDF, 0xEF, +}; + +struct iqs7211_ver_info { + __le16 prod_num; + __le16 major; + __le16 minor; + __le32 patch; +} __packed; + +struct iqs7211_touch_data { + __le16 abs_x; + __le16 abs_y; + __le16 pressure; + __le16 area; +} __packed; + +struct iqs7211_tp_config { + u8 tp_settings; + u8 total_rx; + u8 total_tx; + u8 num_contacts; + __le16 max_x; + __le16 max_y; +} __packed; + +struct iqs7211_private { + const struct iqs7211_dev_desc *dev_desc; + struct gpio_desc *reset_gpio; + struct gpio_desc *irq_gpio; + struct i2c_client *client; + struct input_dev *tp_idev; + struct input_dev *kp_idev; + struct iqs7211_ver_info ver_info; + struct iqs7211_tp_config tp_config; + struct touchscreen_properties prop; + struct list_head reg_field_head; + enum iqs7211_comms_mode comms_init; + enum iqs7211_comms_mode comms_mode; + unsigned int num_contacts; + unsigned int kp_code[ARRAY_SIZE(iqs7211e_kp_events)]; + u8 rx_tx_map[IQS7211_MAX_CTX + 1]; + u8 cycle_alloc[2][33]; + u8 exp_file[2]; + u16 event_mask; + u16 ati_start; + u16 gesture_cache; +}; + +static int iqs7211_irq_poll(struct iqs7211_private *iqs7211, u64 timeout_us) +{ + int error, val; + + error = readx_poll_timeout(gpiod_get_value_cansleep, iqs7211->irq_gpio, + val, val, IQS7211_COMMS_SLEEP_US, timeout_us); + + return val < 0 ? val : error; +} + +static int iqs7211_hard_reset(struct iqs7211_private *iqs7211) +{ + if (!iqs7211->reset_gpio) + return 0; + + gpiod_set_value_cansleep(iqs7211->reset_gpio, 1); + + /* + * The following delay ensures the shared RDY/MCLR pin is sampled in + * between periodic assertions by the device and assumes the default + * communication timeout has not been overwritten in OTP memory. + */ + if (iqs7211->reset_gpio == iqs7211->irq_gpio) + msleep(IQS7211_RESET_TIMEOUT_MS); + else + usleep_range(1000, 1100); + + gpiod_set_value_cansleep(iqs7211->reset_gpio, 0); + if (iqs7211->reset_gpio == iqs7211->irq_gpio) + iqs7211_irq_wait(); + + return iqs7211_irq_poll(iqs7211, IQS7211_START_TIMEOUT_US); +} + +static int iqs7211_force_comms(struct iqs7211_private *iqs7211) +{ + u8 msg_buf[] = { 0xFF, }; + int ret; + + switch (iqs7211->comms_mode) { + case IQS7211_COMMS_MODE_WAIT: + return iqs7211_irq_poll(iqs7211, IQS7211_START_TIMEOUT_US); + + case IQS7211_COMMS_MODE_FREE: + return 0; + + case IQS7211_COMMS_MODE_FORCE: + break; + + default: + return -EINVAL; + } + + /* + * The device cannot communicate until it asserts its interrupt (RDY) + * pin. Attempts to do so while RDY is deasserted return an ACK; how- + * ever all write data is ignored, and all read data returns 0xEE. + * + * Unsolicited communication must be preceded by a special force com- + * munication command, after which the device eventually asserts its + * RDY pin and agrees to communicate. + * + * Regardless of whether communication is forced or the result of an + * interrupt, the device automatically deasserts its RDY pin once it + * detects an I2C stop condition, or a timeout expires. + */ + ret = gpiod_get_value_cansleep(iqs7211->irq_gpio); + if (ret < 0) + return ret; + else if (ret > 0) + return 0; + + ret = i2c_master_send(iqs7211->client, msg_buf, sizeof(msg_buf)); + if (ret < (int)sizeof(msg_buf)) { + if (ret >= 0) + ret = -EIO; + + msleep(IQS7211_COMMS_RETRY_MS); + return ret; + } + + iqs7211_irq_wait(); + + return iqs7211_irq_poll(iqs7211, IQS7211_COMMS_TIMEOUT_US); +} + +static int iqs7211_read_burst(struct iqs7211_private *iqs7211, + u8 reg, void *val, u16 val_len) +{ + int ret, i; + struct i2c_client *client = iqs7211->client; + struct i2c_msg msg[] = { + { + .addr = client->addr, + .flags = 0, + .len = sizeof(reg), + .buf = ®, + }, + { + .addr = client->addr, + .flags = I2C_M_RD, + .len = val_len, + .buf = (u8 *)val, + }, + }; + + /* + * The following loop protects against an edge case in which the RDY + * pin is automatically deasserted just as the read is initiated. In + * that case, the read must be retried using forced communication. + */ + for (i = 0; i < IQS7211_NUM_RETRIES; i++) { + ret = iqs7211_force_comms(iqs7211); + if (ret < 0) + continue; + + ret = i2c_transfer(client->adapter, msg, ARRAY_SIZE(msg)); + if (ret < (int)ARRAY_SIZE(msg)) { + if (ret >= 0) + ret = -EIO; + + msleep(IQS7211_COMMS_RETRY_MS); + continue; + } + + if (get_unaligned_le16(msg[1].buf) == IQS7211_COMMS_ERROR) { + ret = -ENODATA; + continue; + } + + ret = 0; + break; + } + + iqs7211_irq_wait(); + + if (ret < 0) + dev_err(&client->dev, + "Failed to read from address 0x%02X: %d\n", reg, ret); + + return ret; +} + +static int iqs7211_read_word(struct iqs7211_private *iqs7211, u8 reg, u16 *val) +{ + __le16 val_buf; + int error; + + error = iqs7211_read_burst(iqs7211, reg, &val_buf, sizeof(val_buf)); + if (error) + return error; + + *val = le16_to_cpu(val_buf); + + return 0; +} + +static int iqs7211_write_burst(struct iqs7211_private *iqs7211, + u8 reg, const void *val, u16 val_len) +{ + int msg_len = sizeof(reg) + val_len; + int ret, i; + struct i2c_client *client = iqs7211->client; + u8 *msg_buf; + + msg_buf = kzalloc(msg_len, GFP_KERNEL); + if (!msg_buf) + return -ENOMEM; + + *msg_buf = reg; + memcpy(msg_buf + sizeof(reg), val, val_len); + + /* + * The following loop protects against an edge case in which the RDY + * pin is automatically asserted just before the force communication + * command is sent. + * + * In that case, the subsequent I2C stop condition tricks the device + * into preemptively deasserting the RDY pin and the command must be + * sent again. + */ + for (i = 0; i < IQS7211_NUM_RETRIES; i++) { + ret = iqs7211_force_comms(iqs7211); + if (ret < 0) + continue; + + ret = i2c_master_send(client, msg_buf, msg_len); + if (ret < msg_len) { + if (ret >= 0) + ret = -EIO; + + msleep(IQS7211_COMMS_RETRY_MS); + continue; + } + + ret = 0; + break; + } + + kfree(msg_buf); + + iqs7211_irq_wait(); + + if (ret < 0) + dev_err(&client->dev, + "Failed to write to address 0x%02X: %d\n", reg, ret); + + return ret; +} + +static int iqs7211_write_word(struct iqs7211_private *iqs7211, u8 reg, u16 val) +{ + __le16 val_buf = cpu_to_le16(val); + + return iqs7211_write_burst(iqs7211, reg, &val_buf, sizeof(val_buf)); +} + +static int iqs7211_start_comms(struct iqs7211_private *iqs7211) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct i2c_client *client = iqs7211->client; + bool forced_comms; + unsigned int val; + u16 comms_setup; + int error; + + /* + * Until forced communication can be enabled, the host must wait for a + * communication window each time it intends to elicit a response from + * the device. + * + * Forced communication is not necessary, however, if the host adapter + * can support clock stretching. In that case, the device freely clock + * stretches until all pending conversions are complete. + */ + forced_comms = device_property_present(&client->dev, + "azoteq,forced-comms"); + + error = device_property_read_u32(&client->dev, + "azoteq,forced-comms-default", &val); + if (error == -EINVAL) { + iqs7211->comms_init = IQS7211_COMMS_MODE_WAIT; + } else if (error) { + dev_err(&client->dev, + "Failed to read default communication mode: %d\n", + error); + return error; + } else if (val) { + iqs7211->comms_init = forced_comms ? IQS7211_COMMS_MODE_FORCE + : IQS7211_COMMS_MODE_WAIT; + } else { + iqs7211->comms_init = forced_comms ? IQS7211_COMMS_MODE_WAIT + : IQS7211_COMMS_MODE_FREE; + } + + iqs7211->comms_mode = iqs7211->comms_init; + + error = iqs7211_hard_reset(iqs7211); + if (error) { + dev_err(&client->dev, "Failed to reset device: %d\n", error); + return error; + } + + error = iqs7211_read_burst(iqs7211, IQS7211_PROD_NUM, + &iqs7211->ver_info, + sizeof(iqs7211->ver_info)); + if (error) + return error; + + if (le16_to_cpu(iqs7211->ver_info.prod_num) != dev_desc->prod_num) { + dev_err(&client->dev, "Invalid product number: %u\n", + le16_to_cpu(iqs7211->ver_info.prod_num)); + return -EINVAL; + } + + error = iqs7211_read_word(iqs7211, dev_desc->sys_ctrl + 1, + &comms_setup); + if (error) + return error; + + if (forced_comms) + comms_setup |= dev_desc->comms_req; + else + comms_setup &= ~dev_desc->comms_req; + + error = iqs7211_write_word(iqs7211, dev_desc->sys_ctrl + 1, + comms_setup | dev_desc->comms_end); + if (error) + return error; + + if (forced_comms) + iqs7211->comms_mode = IQS7211_COMMS_MODE_FORCE; + else + iqs7211->comms_mode = IQS7211_COMMS_MODE_FREE; + + error = iqs7211_read_burst(iqs7211, dev_desc->exp_file, + iqs7211->exp_file, + sizeof(iqs7211->exp_file)); + if (error) + return error; + + error = iqs7211_read_burst(iqs7211, dev_desc->tp_config, + &iqs7211->tp_config, + sizeof(iqs7211->tp_config)); + if (error) + return error; + + error = iqs7211_write_word(iqs7211, dev_desc->sys_ctrl + 1, + comms_setup); + if (error) + return error; + + iqs7211->event_mask = comms_setup & ~IQS7211_EVENT_MASK_ALL; + iqs7211->event_mask |= (IQS7211_EVENT_MASK_ATI | IQS7211_EVENT_MODE); + + return 0; +} + +static int iqs7211_init_device(struct iqs7211_private *iqs7211) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct iqs7211_reg_field_desc *reg_field; + __le16 sys_ctrl[] = { + cpu_to_le16(dev_desc->ack_reset), + cpu_to_le16(iqs7211->event_mask), + }; + int error, i; + + /* + * Acknowledge reset before writing any registers in case the device + * suffers a spurious reset during initialization. The communication + * mode is configured at this time as well. + */ + error = iqs7211_write_burst(iqs7211, dev_desc->sys_ctrl, sys_ctrl, + sizeof(sys_ctrl)); + if (error) + return error; + + if (iqs7211->event_mask & dev_desc->comms_req) + iqs7211->comms_mode = IQS7211_COMMS_MODE_FORCE; + else + iqs7211->comms_mode = IQS7211_COMMS_MODE_FREE; + + /* + * Take advantage of the stop-bit disable function, if available, to + * save the trouble of having to reopen a communication window after + * each read or write. + */ + error = iqs7211_write_word(iqs7211, dev_desc->sys_ctrl + 1, + iqs7211->event_mask | dev_desc->comms_end); + if (error) + return error; + + list_for_each_entry(reg_field, &iqs7211->reg_field_head, list) { + u16 new_val = reg_field->val; + + if (reg_field->mask < U16_MAX) { + u16 old_val; + + error = iqs7211_read_word(iqs7211, reg_field->addr, + &old_val); + if (error) + return error; + + new_val = old_val & ~reg_field->mask; + new_val |= reg_field->val; + + if (new_val == old_val) + continue; + } + + error = iqs7211_write_word(iqs7211, reg_field->addr, new_val); + if (error) + return error; + } + + error = iqs7211_write_burst(iqs7211, dev_desc->tp_config, + &iqs7211->tp_config, + sizeof(iqs7211->tp_config)); + if (error) + return error; + + if (**iqs7211->cycle_alloc) { + error = iqs7211_write_burst(iqs7211, dev_desc->rx_tx_map, + &iqs7211->rx_tx_map, + dev_desc->num_ctx); + if (error) + return error; + + for (i = 0; i < sizeof(dev_desc->cycle_limit); i++) { + error = iqs7211_write_burst(iqs7211, + dev_desc->cycle_alloc[i], + iqs7211->cycle_alloc[i], + dev_desc->cycle_limit[i] * 3); + if (error) + return error; + } + } + + *sys_ctrl = cpu_to_le16(iqs7211->ati_start); + + return iqs7211_write_burst(iqs7211, dev_desc->sys_ctrl, sys_ctrl, + sizeof(sys_ctrl)); +} + +static int iqs7211_add_field(struct iqs7211_private *iqs7211, + struct iqs7211_reg_field_desc new_field) +{ + struct i2c_client *client = iqs7211->client; + struct iqs7211_reg_field_desc *reg_field; + + if (!new_field.addr) + return 0; + + list_for_each_entry(reg_field, &iqs7211->reg_field_head, list) { + if (reg_field->addr != new_field.addr) + continue; + + reg_field->mask |= new_field.mask; + reg_field->val |= new_field.val; + return 0; + } + + reg_field = devm_kzalloc(&client->dev, sizeof(*reg_field), GFP_KERNEL); + if (!reg_field) + return -ENOMEM; + + reg_field->addr = new_field.addr; + reg_field->mask = new_field.mask; + reg_field->val = new_field.val; + + list_add(®_field->list, &iqs7211->reg_field_head); + + return 0; +} + +static int iqs7211_parse_props(struct iqs7211_private *iqs7211, + struct fwnode_handle *reg_grp_node, + enum iqs7211_reg_grp_id reg_grp, + enum iqs7211_reg_key_id reg_key) +{ + struct i2c_client *client = iqs7211->client; + int i; + + for (i = 0; i < ARRAY_SIZE(iqs7211_props); i++) { + const char *name = iqs7211_props[i].name; + u8 reg_addr = iqs7211_props[i].reg_addr[reg_grp] + [iqs7211->dev_desc - + iqs7211_devs]; + int reg_shift = iqs7211_props[i].reg_shift; + int reg_width = iqs7211_props[i].reg_width ? : 16; + int val_pitch = iqs7211_props[i].val_pitch ? : 1; + int val_min = iqs7211_props[i].val_min; + int val_max = iqs7211_props[i].val_max; + const char *label = iqs7211_props[i].label ? : name; + struct iqs7211_reg_field_desc reg_field; + unsigned int val; + int error; + + if (iqs7211_props[i].reg_key != reg_key) + continue; + + if (!reg_addr) + continue; + + error = fwnode_property_read_u32(reg_grp_node, name, &val); + if (error == -EINVAL) { + continue; + } else if (error) { + dev_err(&client->dev, "Failed to read %s %s: %d\n", + fwnode_get_name(reg_grp_node), label, error); + return error; + } + + if (!val_max) + val_max = GENMASK(reg_width - 1, 0) * val_pitch; + + if (val < val_min || val > val_max) { + dev_err(&client->dev, "Invalid %s: %u\n", label, val); + return -EINVAL; + } + + reg_field.addr = reg_addr; + reg_field.mask = GENMASK(reg_shift + reg_width - 1, reg_shift); + reg_field.val = val / val_pitch << reg_shift; + + error = iqs7211_add_field(iqs7211, reg_field); + if (error) + return error; + } + + return 0; +} + +static int iqs7211_parse_event(struct iqs7211_private *iqs7211, + struct fwnode_handle *event_node, + enum iqs7211_reg_grp_id reg_grp, + enum iqs7211_reg_key_id reg_key, + unsigned int *event_code) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct i2c_client *client = iqs7211->client; + struct iqs7211_reg_field_desc reg_field; + unsigned int val; + int error; + + error = iqs7211_parse_props(iqs7211, event_node, reg_grp, reg_key); + if (error) + return error; + + if (reg_key == IQS7211_REG_KEY_AXIAL_X || + reg_key == IQS7211_REG_KEY_AXIAL_Y) { + error = fwnode_property_read_u32(event_node, + "azoteq,gesture-angle", &val); + if (!error) { + if (val >= ARRAY_SIZE(iqs7211_gesture_angle)) { + dev_err(&client->dev, + "Invalid %s gesture angle: %u\n", + fwnode_get_name(event_node), val); + return -EINVAL; + } + + reg_field.addr = dev_desc->gesture_angle; + reg_field.mask = U8_MAX; + reg_field.val = iqs7211_gesture_angle[val]; + + error = iqs7211_add_field(iqs7211, reg_field); + if (error) + return error; + } else if (error != -EINVAL) { + dev_err(&client->dev, + "Failed to read %s gesture angle: %d\n", + fwnode_get_name(event_node), error); + return error; + } + } + + error = fwnode_property_read_u32(event_node, "linux,code", event_code); + if (error == -EINVAL) + error = 0; + else if (error) + dev_err(&client->dev, "Failed to read %s code: %d\n", + fwnode_get_name(event_node), error); + + return error; +} + +static int iqs7211_parse_cycles(struct iqs7211_private *iqs7211, + struct fwnode_handle *tp_node) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct i2c_client *client = iqs7211->client; + int num_cycles = dev_desc->cycle_limit[0] + dev_desc->cycle_limit[1]; + int error, count, i, j, k, cycle_start; + unsigned int cycle_alloc[IQS7211_MAX_CYCLES][2]; + u8 total_rx = iqs7211->tp_config.total_rx; + u8 total_tx = iqs7211->tp_config.total_tx; + + for (i = 0; i < IQS7211_MAX_CYCLES * 2; i++) + *(cycle_alloc[0] + i) = U8_MAX; + + count = fwnode_property_count_u32(tp_node, "azoteq,channel-select"); + if (count == -EINVAL) { + /* + * Assign each sensing cycle's slots (0 and 1) to a channel, + * defined as the intersection between two CRx and CTx pins. + * A channel assignment of 255 means the slot is unused. + */ + for (i = 0, cycle_start = 0; i < total_tx; i++) { + int cycle_stop = 0; + + for (j = 0; j < total_rx; j++) { + /* + * Channels formed by CRx0-3 and CRx4-7 are + * bound to slots 0 and 1, respectively. + */ + int slot = iqs7211->rx_tx_map[j] < 4 ? 0 : 1; + int chan = i * total_rx + j; + + for (k = cycle_start; k < num_cycles; k++) { + if (cycle_alloc[k][slot] < U8_MAX) + continue; + + cycle_alloc[k][slot] = chan; + break; + } + + if (k < num_cycles) { + cycle_stop = max(k, cycle_stop); + continue; + } + + dev_err(&client->dev, + "Insufficient number of cycles\n"); + return -EINVAL; + } + + /* + * Sensing cycles cannot straddle more than one CTx + * pin. As such, the next row's starting cycle must + * be greater than the previous row's highest cycle. + */ + cycle_start = cycle_stop + 1; + } + } else if (count < 0) { + dev_err(&client->dev, "Failed to count channels: %d\n", count); + return count; + } else if (count > num_cycles * 2) { + dev_err(&client->dev, "Insufficient number of cycles\n"); + return -EINVAL; + } else if (count > 0) { + error = fwnode_property_read_u32_array(tp_node, + "azoteq,channel-select", + cycle_alloc[0], count); + if (error) { + dev_err(&client->dev, "Failed to read channels: %d\n", + error); + return error; + } + + for (i = 0; i < count; i++) { + int chan = *(cycle_alloc[0] + i); + + if (chan == U8_MAX) + continue; + + if (chan >= total_rx * total_tx) { + dev_err(&client->dev, "Invalid channel: %d\n", + chan); + return -EINVAL; + } + + for (j = 0; j < count; j++) { + if (j == i || *(cycle_alloc[0] + j) != chan) + continue; + + dev_err(&client->dev, "Duplicate channel: %d\n", + chan); + return -EINVAL; + } + } + } + + /* + * Once the raw channel assignments have been derived, they must be + * packed according to the device's register map. + */ + for (i = 0, cycle_start = 0; i < sizeof(dev_desc->cycle_limit); i++) { + int offs = 0; + + for (j = cycle_start; + j < cycle_start + dev_desc->cycle_limit[i]; j++) { + iqs7211->cycle_alloc[i][offs++] = 0x05; + iqs7211->cycle_alloc[i][offs++] = cycle_alloc[j][0]; + iqs7211->cycle_alloc[i][offs++] = cycle_alloc[j][1]; + } + + cycle_start += dev_desc->cycle_limit[i]; + } + + return 0; +} + +static int iqs7211_parse_tp(struct iqs7211_private *iqs7211, + struct fwnode_handle *tp_node) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct i2c_client *client = iqs7211->client; + unsigned int pins[IQS7211_MAX_CTX]; + int error, count, i, j; + + count = fwnode_property_count_u32(tp_node, "azoteq,rx-enable"); + if (count == -EINVAL) { + return 0; + } else if (count < 0) { + dev_err(&client->dev, "Failed to count CRx pins: %d\n", count); + return count; + } else if (count > IQS7211_NUM_CRX) { + dev_err(&client->dev, "Invalid number of CRx pins\n"); + return -EINVAL; + } + + error = fwnode_property_read_u32_array(tp_node, "azoteq,rx-enable", + pins, count); + if (error) { + dev_err(&client->dev, "Failed to read CRx pins: %d\n", error); + return error; + } + + for (i = 0; i < count; i++) { + if (pins[i] >= IQS7211_NUM_CRX) { + dev_err(&client->dev, "Invalid CRx pin: %u\n", pins[i]); + return -EINVAL; + } + + iqs7211->rx_tx_map[i] = pins[i]; + } + + iqs7211->tp_config.total_rx = count; + + count = fwnode_property_count_u32(tp_node, "azoteq,tx-enable"); + if (count < 0) { + dev_err(&client->dev, "Failed to count CTx pins: %d\n", count); + return count; + } else if (count > dev_desc->num_ctx) { + dev_err(&client->dev, "Invalid number of CTx pins\n"); + return -EINVAL; + } + + error = fwnode_property_read_u32_array(tp_node, "azoteq,tx-enable", + pins, count); + if (error) { + dev_err(&client->dev, "Failed to read CTx pins: %d\n", error); + return error; + } + + for (i = 0; i < count; i++) { + if (pins[i] >= dev_desc->num_ctx) { + dev_err(&client->dev, "Invalid CTx pin: %u\n", pins[i]); + return -EINVAL; + } + + for (j = 0; j < iqs7211->tp_config.total_rx; j++) { + if (iqs7211->rx_tx_map[j] != pins[i]) + continue; + + dev_err(&client->dev, "Conflicting CTx pin: %u\n", + pins[i]); + return -EINVAL; + } + + iqs7211->rx_tx_map[iqs7211->tp_config.total_rx + i] = pins[i]; + } + + iqs7211->tp_config.total_tx = count; + + return iqs7211_parse_cycles(iqs7211, tp_node); +} + +static int iqs7211_parse_alp(struct iqs7211_private *iqs7211, + struct fwnode_handle *alp_node) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct i2c_client *client = iqs7211->client; + struct iqs7211_reg_field_desc reg_field; + int error, count, i; + + count = fwnode_property_count_u32(alp_node, "azoteq,rx-enable"); + if (count < 0 && count != -EINVAL) { + dev_err(&client->dev, "Failed to count CRx pins: %d\n", count); + return count; + } else if (count > IQS7211_NUM_CRX) { + dev_err(&client->dev, "Invalid number of CRx pins\n"); + return -EINVAL; + } else if (count >= 0) { + unsigned int pins[IQS7211_NUM_CRX]; + + error = fwnode_property_read_u32_array(alp_node, + "azoteq,rx-enable", + pins, count); + if (error) { + dev_err(&client->dev, "Failed to read CRx pins: %d\n", + error); + return error; + } + + reg_field.addr = dev_desc->alp_config; + reg_field.mask = GENMASK(IQS7211_NUM_CRX - 1, 0); + reg_field.val = 0; + + for (i = 0; i < count; i++) { + if (pins[i] < dev_desc->min_crx_alp || + pins[i] >= IQS7211_NUM_CRX) { + dev_err(&client->dev, "Invalid CRx pin: %u\n", + pins[i]); + return -EINVAL; + } + + reg_field.val |= BIT(pins[i]); + } + + error = iqs7211_add_field(iqs7211, reg_field); + if (error) + return error; + } + + count = fwnode_property_count_u32(alp_node, "azoteq,tx-enable"); + if (count < 0 && count != -EINVAL) { + dev_err(&client->dev, "Failed to count CTx pins: %d\n", count); + return count; + } else if (count > dev_desc->num_ctx) { + dev_err(&client->dev, "Invalid number of CTx pins\n"); + return -EINVAL; + } else if (count >= 0) { + unsigned int pins[IQS7211_MAX_CTX]; + + error = fwnode_property_read_u32_array(alp_node, + "azoteq,tx-enable", + pins, count); + if (error) { + dev_err(&client->dev, "Failed to read CTx pins: %d\n", + error); + return error; + } + + reg_field.addr = dev_desc->alp_config + 1; + reg_field.mask = GENMASK(dev_desc->num_ctx - 1, 0); + reg_field.val = 0; + + for (i = 0; i < count; i++) { + if (pins[i] >= dev_desc->num_ctx) { + dev_err(&client->dev, "Invalid CTx pin: %u\n", + pins[i]); + return -EINVAL; + } + + reg_field.val |= BIT(pins[i]); + } + + error = iqs7211_add_field(iqs7211, reg_field); + if (error) + return error; + } + + return 0; +} + +static int (*iqs7211_parse_extra[IQS7211_NUM_REG_GRPS]) + (struct iqs7211_private *iqs7211, + struct fwnode_handle *reg_grp_node) = { + [IQS7211_REG_GRP_TP] = iqs7211_parse_tp, + [IQS7211_REG_GRP_ALP] = iqs7211_parse_alp, +}; + +static int iqs7211_parse_reg_grp(struct iqs7211_private *iqs7211, + struct fwnode_handle *reg_grp_node, + enum iqs7211_reg_grp_id reg_grp) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct iqs7211_reg_field_desc reg_field; + int error, i; + + error = iqs7211_parse_props(iqs7211, reg_grp_node, reg_grp, + IQS7211_REG_KEY_NONE); + if (error) + return error; + + if (iqs7211_parse_extra[reg_grp]) { + error = iqs7211_parse_extra[reg_grp](iqs7211, reg_grp_node); + if (error) + return error; + } + + iqs7211->ati_start |= dev_desc->ati_start[reg_grp]; + + reg_field.addr = dev_desc->kp_enable[reg_grp]; + reg_field.mask = 0; + reg_field.val = 0; + + for (i = 0; i < dev_desc->num_kp_events; i++) { + const char *event_name = dev_desc->kp_events[i].name; + struct fwnode_handle *event_node; + + if (dev_desc->kp_events[i].reg_grp != reg_grp) + continue; + + reg_field.mask |= dev_desc->kp_events[i].enable; + + if (event_name) + event_node = fwnode_get_named_child_node(reg_grp_node, + event_name); + else + event_node = fwnode_handle_get(reg_grp_node); + + if (!event_node) + continue; + + error = iqs7211_parse_event(iqs7211, event_node, + dev_desc->kp_events[i].reg_grp, + dev_desc->kp_events[i].reg_key, + &iqs7211->kp_code[i]); + fwnode_handle_put(event_node); + if (error) + return error; + + reg_field.val |= dev_desc->kp_events[i].enable; + + iqs7211->event_mask |= iqs7211_reg_grp_masks[reg_grp]; + } + + return iqs7211_add_field(iqs7211, reg_field); +} + +static int iqs7211_register_kp(struct iqs7211_private *iqs7211) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct input_dev *kp_idev = iqs7211->kp_idev; + struct i2c_client *client = iqs7211->client; + int error, i; + + for (i = 0; i < dev_desc->num_kp_events; i++) + if (iqs7211->kp_code[i]) + break; + + if (i == dev_desc->num_kp_events) + return 0; + + kp_idev = devm_input_allocate_device(&client->dev); + if (!kp_idev) + return -ENOMEM; + + iqs7211->kp_idev = kp_idev; + + kp_idev->name = dev_desc->kp_name; + kp_idev->id.bustype = BUS_I2C; + + for (i = 0; i < dev_desc->num_kp_events; i++) + if (iqs7211->kp_code[i]) + input_set_capability(iqs7211->kp_idev, EV_KEY, + iqs7211->kp_code[i]); + + error = input_register_device(kp_idev); + if (error) + dev_err(&client->dev, "Failed to register %s: %d\n", + kp_idev->name, error); + + return error; +} + +static int iqs7211_register_tp(struct iqs7211_private *iqs7211) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct touchscreen_properties *prop = &iqs7211->prop; + struct input_dev *tp_idev = iqs7211->tp_idev; + struct i2c_client *client = iqs7211->client; + int error; + + error = device_property_read_u32(&client->dev, "azoteq,num-contacts", + &iqs7211->num_contacts); + if (error == -EINVAL) { + return 0; + } else if (error) { + dev_err(&client->dev, "Failed to read number of contacts: %d\n", + error); + return error; + } else if (iqs7211->num_contacts > IQS7211_MAX_CONTACTS) { + dev_err(&client->dev, "Invalid number of contacts: %u\n", + iqs7211->num_contacts); + return -EINVAL; + } + + iqs7211->tp_config.num_contacts = iqs7211->num_contacts ? : 1; + + if (!iqs7211->num_contacts) + return 0; + + iqs7211->event_mask |= IQS7211_EVENT_MASK_MOVE; + + tp_idev = devm_input_allocate_device(&client->dev); + if (!tp_idev) + return -ENOMEM; + + iqs7211->tp_idev = tp_idev; + + tp_idev->name = dev_desc->tp_name; + tp_idev->id.bustype = BUS_I2C; + + input_set_abs_params(tp_idev, ABS_MT_POSITION_X, + 0, le16_to_cpu(iqs7211->tp_config.max_x), 0, 0); + + input_set_abs_params(tp_idev, ABS_MT_POSITION_Y, + 0, le16_to_cpu(iqs7211->tp_config.max_y), 0, 0); + + input_set_abs_params(tp_idev, ABS_MT_PRESSURE, 0, U16_MAX, 0, 0); + + touchscreen_parse_properties(tp_idev, true, prop); + + /* + * The device reserves 0xFFFF for coordinates that correspond to slots + * which are not in a state of touch. + */ + if (prop->max_x >= U16_MAX || prop->max_y >= U16_MAX) { + dev_err(&client->dev, "Invalid trackpad size: %u*%u\n", + prop->max_x, prop->max_y); + return -EINVAL; + } + + iqs7211->tp_config.max_x = cpu_to_le16(prop->max_x); + iqs7211->tp_config.max_y = cpu_to_le16(prop->max_y); + + error = input_mt_init_slots(tp_idev, iqs7211->num_contacts, + INPUT_MT_DIRECT); + if (error) { + dev_err(&client->dev, "Failed to initialize slots: %d\n", + error); + return error; + } + + error = input_register_device(tp_idev); + if (error) + dev_err(&client->dev, "Failed to register %s: %d\n", + tp_idev->name, error); + + return error; +} + +static int iqs7211_report(struct iqs7211_private *iqs7211) +{ + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + struct i2c_client *client = iqs7211->client; + struct iqs7211_touch_data *touch_data; + u16 info_flags, charge_mode, gesture_flags; + __le16 status[12]; + int error, i; + + error = iqs7211_read_burst(iqs7211, dev_desc->sys_stat, status, + dev_desc->contact_offs * sizeof(__le16) + + iqs7211->num_contacts * sizeof(*touch_data)); + if (error) + return error; + + info_flags = le16_to_cpu(status[dev_desc->info_offs]); + + if (info_flags & dev_desc->show_reset) { + dev_err(&client->dev, "Unexpected device reset\n"); + + /* + * The device may or may not expect forced communication after + * it exits hardware reset, so the corresponding state machine + * must be reset as well. + */ + iqs7211->comms_mode = iqs7211->comms_init; + + return iqs7211_init_device(iqs7211); + } + + for (i = 0; i < ARRAY_SIZE(dev_desc->ati_error); i++) { + if (!(info_flags & dev_desc->ati_error[i])) + continue; + + dev_err(&client->dev, "Unexpected %s ATI error\n", + iqs7211_reg_grp_names[i]); + return 0; + } + + for (i = 0; i < iqs7211->num_contacts; i++) { + u16 pressure; + + touch_data = (struct iqs7211_touch_data *) + &status[dev_desc->contact_offs] + i; + pressure = le16_to_cpu(touch_data->pressure); + + input_mt_slot(iqs7211->tp_idev, i); + if (input_mt_report_slot_state(iqs7211->tp_idev, MT_TOOL_FINGER, + pressure != 0)) { + touchscreen_report_pos(iqs7211->tp_idev, &iqs7211->prop, + le16_to_cpu(touch_data->abs_x), + le16_to_cpu(touch_data->abs_y), + true); + input_report_abs(iqs7211->tp_idev, ABS_MT_PRESSURE, + pressure); + } + } + + if (iqs7211->num_contacts) { + input_mt_sync_frame(iqs7211->tp_idev); + input_sync(iqs7211->tp_idev); + } + + if (!iqs7211->kp_idev) + return 0; + + charge_mode = info_flags & GENMASK(dev_desc->charge_shift + 2, + dev_desc->charge_shift); + charge_mode >>= dev_desc->charge_shift; + + /* + * A charging mode higher than 2 (idle mode) indicates the device last + * operated in low-power mode and intends to express an ALP event. + */ + if (info_flags & dev_desc->kp_events->mask && charge_mode > 2) { + input_report_key(iqs7211->kp_idev, *iqs7211->kp_code, 1); + input_sync(iqs7211->kp_idev); + + input_report_key(iqs7211->kp_idev, *iqs7211->kp_code, 0); + } + + for (i = 0; i < dev_desc->num_kp_events; i++) { + if (dev_desc->kp_events[i].reg_grp != IQS7211_REG_GRP_BTN) + continue; + + input_report_key(iqs7211->kp_idev, iqs7211->kp_code[i], + info_flags & dev_desc->kp_events[i].mask); + } + + gesture_flags = le16_to_cpu(status[dev_desc->gesture_offs]); + + for (i = 0; i < dev_desc->num_kp_events; i++) { + enum iqs7211_reg_key_id reg_key = dev_desc->kp_events[i].reg_key; + u16 mask = dev_desc->kp_events[i].mask; + + if (dev_desc->kp_events[i].reg_grp != IQS7211_REG_GRP_TP) + continue; + + if ((gesture_flags ^ iqs7211->gesture_cache) & mask) + input_report_key(iqs7211->kp_idev, iqs7211->kp_code[i], + gesture_flags & mask); + + iqs7211->gesture_cache &= ~mask; + + /* + * Hold and palm gestures persist while the contact remains in + * place; all others are momentary and hence are followed by a + * complementary release event. + */ + if (reg_key == IQS7211_REG_KEY_HOLD || + reg_key == IQS7211_REG_KEY_PALM) { + iqs7211->gesture_cache |= gesture_flags & mask; + gesture_flags &= ~mask; + } + } + + if (gesture_flags) { + input_sync(iqs7211->kp_idev); + + for (i = 0; i < dev_desc->num_kp_events; i++) + if (dev_desc->kp_events[i].reg_grp == IQS7211_REG_GRP_TP && + gesture_flags & dev_desc->kp_events[i].mask) + input_report_key(iqs7211->kp_idev, + iqs7211->kp_code[i], 0); + } + + input_sync(iqs7211->kp_idev); + + return 0; +} + +static irqreturn_t iqs7211_irq(int irq, void *context) +{ + struct iqs7211_private *iqs7211 = context; + + return iqs7211_report(iqs7211) ? IRQ_NONE : IRQ_HANDLED; +} + +static int iqs7211_suspend(struct device *dev) +{ + struct iqs7211_private *iqs7211 = dev_get_drvdata(dev); + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + int error; + + if (!dev_desc->suspend || device_may_wakeup(dev)) + return 0; + + /* + * I2C communication prompts the device to assert its RDY pin if it is + * not already asserted. As such, the interrupt must be disabled so as + * to prevent reentrant interrupts. + */ + disable_irq(gpiod_to_irq(iqs7211->irq_gpio)); + + error = iqs7211_write_word(iqs7211, dev_desc->sys_ctrl, + dev_desc->suspend); + + enable_irq(gpiod_to_irq(iqs7211->irq_gpio)); + + return error; +} + +static int iqs7211_resume(struct device *dev) +{ + struct iqs7211_private *iqs7211 = dev_get_drvdata(dev); + const struct iqs7211_dev_desc *dev_desc = iqs7211->dev_desc; + __le16 sys_ctrl[] = { + 0, + cpu_to_le16(iqs7211->event_mask), + }; + int error; + + if (!dev_desc->suspend || device_may_wakeup(dev)) + return 0; + + disable_irq(gpiod_to_irq(iqs7211->irq_gpio)); + + /* + * Forced communication, if in use, must be explicitly enabled as part + * of the wake-up command. + */ + error = iqs7211_write_burst(iqs7211, dev_desc->sys_ctrl, sys_ctrl, + sizeof(sys_ctrl)); + + enable_irq(gpiod_to_irq(iqs7211->irq_gpio)); + + return error; +} + +static DEFINE_SIMPLE_DEV_PM_OPS(iqs7211_pm, iqs7211_suspend, iqs7211_resume); + +static ssize_t fw_info_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct iqs7211_private *iqs7211 = dev_get_drvdata(dev); + + return scnprintf(buf, PAGE_SIZE, "%u.%u.%u.%u:%u.%u\n", + le16_to_cpu(iqs7211->ver_info.prod_num), + le32_to_cpu(iqs7211->ver_info.patch), + le16_to_cpu(iqs7211->ver_info.major), + le16_to_cpu(iqs7211->ver_info.minor), + iqs7211->exp_file[1], iqs7211->exp_file[0]); +} + +static DEVICE_ATTR_RO(fw_info); + +static struct attribute *iqs7211_attrs[] = { + &dev_attr_fw_info.attr, + NULL +}; +ATTRIBUTE_GROUPS(iqs7211); + +static const struct of_device_id iqs7211_of_match[] = { + { + .compatible = "azoteq,iqs7210a", + .data = &iqs7211_devs[IQS7210A], + }, + { + .compatible = "azoteq,iqs7211a", + .data = &iqs7211_devs[IQS7211A], + }, + { + .compatible = "azoteq,iqs7211e", + .data = &iqs7211_devs[IQS7211E], + }, + { } +}; +MODULE_DEVICE_TABLE(of, iqs7211_of_match); + +static int iqs7211_probe(struct i2c_client *client) +{ + struct iqs7211_private *iqs7211; + enum iqs7211_reg_grp_id reg_grp; + unsigned long irq_flags; + bool shared_irq; + int error, irq; + + iqs7211 = devm_kzalloc(&client->dev, sizeof(*iqs7211), GFP_KERNEL); + if (!iqs7211) + return -ENOMEM; + + i2c_set_clientdata(client, iqs7211); + iqs7211->client = client; + + INIT_LIST_HEAD(&iqs7211->reg_field_head); + + iqs7211->dev_desc = device_get_match_data(&client->dev); + if (!iqs7211->dev_desc) + return -ENODEV; + + shared_irq = iqs7211->dev_desc->num_ctx == IQS7211_MAX_CTX; + + /* + * The RDY pin behaves as an interrupt, but must also be polled ahead + * of unsolicited I2C communication. As such, it is first opened as a + * GPIO and then passed to gpiod_to_irq() to register the interrupt. + * + * If an extra CTx pin is present, the RDY and MCLR pins are combined + * into a single bidirectional pin. In that case, the platform's GPIO + * must be configured as an open-drain output. + */ + iqs7211->irq_gpio = devm_gpiod_get(&client->dev, "irq", + shared_irq ? GPIOD_OUT_LOW + : GPIOD_IN); + if (IS_ERR(iqs7211->irq_gpio)) { + error = PTR_ERR(iqs7211->irq_gpio); + dev_err(&client->dev, "Failed to request IRQ GPIO: %d\n", + error); + return error; + } + + if (shared_irq) { + iqs7211->reset_gpio = iqs7211->irq_gpio; + } else { + iqs7211->reset_gpio = devm_gpiod_get_optional(&client->dev, + "reset", + GPIOD_OUT_HIGH); + if (IS_ERR(iqs7211->reset_gpio)) { + error = PTR_ERR(iqs7211->reset_gpio); + dev_err(&client->dev, + "Failed to request reset GPIO: %d\n", error); + return error; + } + } + + error = iqs7211_start_comms(iqs7211); + if (error) + return error; + + for (reg_grp = 0; reg_grp < IQS7211_NUM_REG_GRPS; reg_grp++) { + const char *reg_grp_name = iqs7211_reg_grp_names[reg_grp]; + struct fwnode_handle *reg_grp_node; + + if (reg_grp_name) + reg_grp_node = device_get_named_child_node(&client->dev, + reg_grp_name); + else + reg_grp_node = fwnode_handle_get(dev_fwnode(&client->dev)); + + if (!reg_grp_node) + continue; + + error = iqs7211_parse_reg_grp(iqs7211, reg_grp_node, reg_grp); + fwnode_handle_put(reg_grp_node); + if (error) + return error; + } + + error = iqs7211_register_kp(iqs7211); + if (error) + return error; + + error = iqs7211_register_tp(iqs7211); + if (error) + return error; + + error = iqs7211_init_device(iqs7211); + if (error) + return error; + + irq = gpiod_to_irq(iqs7211->irq_gpio); + if (irq < 0) + return irq; + + irq_flags = gpiod_is_active_low(iqs7211->irq_gpio) ? IRQF_TRIGGER_LOW + : IRQF_TRIGGER_HIGH; + irq_flags |= IRQF_ONESHOT; + + error = devm_request_threaded_irq(&client->dev, irq, NULL, iqs7211_irq, + irq_flags, client->name, iqs7211); + if (error) + dev_err(&client->dev, "Failed to request IRQ: %d\n", error); + + return error; +} + +static struct i2c_driver iqs7211_i2c_driver = { + .probe = iqs7211_probe, + .driver = { + .name = "iqs7211", + .of_match_table = iqs7211_of_match, + .dev_groups = iqs7211_groups, + .pm = pm_sleep_ptr(&iqs7211_pm), + }, +}; +module_i2c_driver(iqs7211_i2c_driver); + +MODULE_AUTHOR("Jeff LaBundy "); +MODULE_DESCRIPTION("Azoteq IQS7210A/7211A/E Trackpad/Touchscreen Controller"); +MODULE_LICENSE("GPL"); diff --git a/drivers/input/touchscreen/lpc32xx_ts.c b/drivers/input/touchscreen/lpc32xx_ts.c index 15b5cb763526..9bad8b93c039 100644 --- a/drivers/input/touchscreen/lpc32xx_ts.c +++ b/drivers/input/touchscreen/lpc32xx_ts.c @@ -198,54 +198,36 @@ static void lpc32xx_ts_close(struct input_dev *dev) static int lpc32xx_ts_probe(struct platform_device *pdev) { + struct device *dev = &pdev->dev; struct lpc32xx_tsc *tsc; struct input_dev *input; - struct resource *res; - resource_size_t size; int irq; int error; - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (!res) { - dev_err(&pdev->dev, "Can't get memory resource\n"); - return -ENOENT; - } - irq = platform_get_irq(pdev, 0); if (irq < 0) return irq; - tsc = kzalloc(sizeof(*tsc), GFP_KERNEL); - input = input_allocate_device(); - if (!tsc || !input) { - dev_err(&pdev->dev, "failed allocating memory\n"); - error = -ENOMEM; - goto err_free_mem; - } + tsc = devm_kzalloc(dev, sizeof(*tsc), GFP_KERNEL); + if (!tsc) + return -ENOMEM; - tsc->dev = input; tsc->irq = irq; - size = resource_size(res); + tsc->tsc_base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(tsc->tsc_base)) + return PTR_ERR(tsc->tsc_base); - if (!request_mem_region(res->start, size, pdev->name)) { - dev_err(&pdev->dev, "TSC registers are not free\n"); - error = -EBUSY; - goto err_free_mem; - } - - tsc->tsc_base = ioremap(res->start, size); - if (!tsc->tsc_base) { - dev_err(&pdev->dev, "Can't map memory\n"); - error = -ENOMEM; - goto err_release_mem; - } - - tsc->clk = clk_get(&pdev->dev, NULL); + tsc->clk = devm_clk_get(dev, NULL); if (IS_ERR(tsc->clk)) { dev_err(&pdev->dev, "failed getting clock\n"); - error = PTR_ERR(tsc->clk); - goto err_unmap; + return PTR_ERR(tsc->clk); + } + + input = devm_input_allocate_device(dev); + if (!input) { + dev_err(&pdev->dev, "failed allocating input device\n"); + return -ENOMEM; } input->name = MOD_NAME; @@ -254,68 +236,33 @@ static int lpc32xx_ts_probe(struct platform_device *pdev) input->id.vendor = 0x0001; input->id.product = 0x0002; input->id.version = 0x0100; - input->dev.parent = &pdev->dev; input->open = lpc32xx_ts_open; input->close = lpc32xx_ts_close; - input->evbit[0] = BIT_MASK(EV_KEY) | BIT_MASK(EV_ABS); - input->keybit[BIT_WORD(BTN_TOUCH)] = BIT_MASK(BTN_TOUCH); + input_set_capability(input, EV_KEY, BTN_TOUCH); input_set_abs_params(input, ABS_X, LPC32XX_TSC_MIN_XY_VAL, LPC32XX_TSC_MAX_XY_VAL, 0, 0); input_set_abs_params(input, ABS_Y, LPC32XX_TSC_MIN_XY_VAL, LPC32XX_TSC_MAX_XY_VAL, 0, 0); input_set_drvdata(input, tsc); + tsc->dev = input; - error = request_irq(tsc->irq, lpc32xx_ts_interrupt, - 0, pdev->name, tsc); + error = devm_request_irq(dev, tsc->irq, lpc32xx_ts_interrupt, + 0, pdev->name, tsc); if (error) { dev_err(&pdev->dev, "failed requesting interrupt\n"); - goto err_put_clock; + return error; } error = input_register_device(input); if (error) { dev_err(&pdev->dev, "failed registering input device\n"); - goto err_free_irq; + return error; } platform_set_drvdata(pdev, tsc); - device_init_wakeup(&pdev->dev, 1); - - return 0; - -err_free_irq: - free_irq(tsc->irq, tsc); -err_put_clock: - clk_put(tsc->clk); -err_unmap: - iounmap(tsc->tsc_base); -err_release_mem: - release_mem_region(res->start, size); -err_free_mem: - input_free_device(input); - kfree(tsc); - - return error; -} - -static int lpc32xx_ts_remove(struct platform_device *pdev) -{ - struct lpc32xx_tsc *tsc = platform_get_drvdata(pdev); - struct resource *res; - - free_irq(tsc->irq, tsc); - - input_unregister_device(tsc->dev); - - clk_put(tsc->clk); - - iounmap(tsc->tsc_base); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - release_mem_region(res->start, resource_size(res)); - - kfree(tsc); + device_init_wakeup(&pdev->dev, true); return 0; } @@ -384,7 +331,6 @@ MODULE_DEVICE_TABLE(of, lpc32xx_tsc_of_match); static struct platform_driver lpc32xx_ts_driver = { .probe = lpc32xx_ts_probe, - .remove = lpc32xx_ts_remove, .driver = { .name = MOD_NAME, .pm = LPC32XX_TS_PM_OPS, diff --git a/drivers/input/touchscreen/melfas_mip4.c b/drivers/input/touchscreen/melfas_mip4.c index 32896e5085bd..2ac4483fbc25 100644 --- a/drivers/input/touchscreen/melfas_mip4.c +++ b/drivers/input/touchscreen/melfas_mip4.c @@ -1451,13 +1451,8 @@ static int mip4_probe(struct i2c_client *client) ts->gpio_ce = devm_gpiod_get_optional(&client->dev, "ce", GPIOD_OUT_LOW); - if (IS_ERR(ts->gpio_ce)) { - error = PTR_ERR(ts->gpio_ce); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "Failed to get gpio: %d\n", error); - return error; - } + if (IS_ERR(ts->gpio_ce)) + return dev_err_probe(&client->dev, PTR_ERR(ts->gpio_ce), "Failed to get gpio\n"); error = mip4_power_on(ts); if (error) diff --git a/drivers/input/touchscreen/mms114.c b/drivers/input/touchscreen/mms114.c index ac12494c7930..af233b6a16d9 100644 --- a/drivers/input/touchscreen/mms114.c +++ b/drivers/input/touchscreen/mms114.c @@ -7,7 +7,6 @@ #include #include #include -#include #include #include #include @@ -43,6 +42,7 @@ /* Touchscreen absolute values */ #define MMS114_MAX_AREA 0xff +#define MMS114_MAX_TOUCHKEYS 15 #define MMS114_MAX_TOUCH 10 #define MMS114_EVENT_SIZE 8 #define MMS136_EVENT_SIZE 6 @@ -70,6 +70,9 @@ struct mms114_data { unsigned int contact_threshold; unsigned int moving_threshold; + u32 keycodes[MMS114_MAX_TOUCHKEYS]; + int num_keycodes; + /* Use cache data for mode control register(write only) */ u8 cache_mode_control; }; @@ -167,11 +170,6 @@ static void mms114_process_mt(struct mms114_data *data, struct mms114_touch *tou return; } - if (touch->type != MMS114_TYPE_TOUCHSCREEN) { - dev_err(&client->dev, "Wrong touch type (%d)\n", touch->type); - return; - } - id = touch->id - 1; x = touch->x_lo | touch->x_hi << 8; y = touch->y_lo | touch->y_hi << 8; @@ -191,9 +189,33 @@ static void mms114_process_mt(struct mms114_data *data, struct mms114_touch *tou } } +static void mms114_process_touchkey(struct mms114_data *data, + struct mms114_touch *touch) +{ + struct i2c_client *client = data->client; + struct input_dev *input_dev = data->input_dev; + unsigned int keycode_id; + + if (touch->id == 0) + return; + + if (touch->id > data->num_keycodes) { + dev_err(&client->dev, "Wrong touch id for touchkey (%d)\n", + touch->id); + return; + } + + keycode_id = touch->id - 1; + dev_dbg(&client->dev, "keycode id: %d, pressed: %d\n", keycode_id, + touch->pressed); + + input_report_key(input_dev, data->keycodes[keycode_id], touch->pressed); +} + static irqreturn_t mms114_interrupt(int irq, void *dev_id) { struct mms114_data *data = dev_id; + struct i2c_client *client = data->client; struct input_dev *input_dev = data->input_dev; struct mms114_touch touch[MMS114_MAX_TOUCH]; int packet_size; @@ -223,8 +245,22 @@ static irqreturn_t mms114_interrupt(int irq, void *dev_id) if (error < 0) goto out; - for (index = 0; index < touch_size; index++) - mms114_process_mt(data, touch + index); + for (index = 0; index < touch_size; index++) { + switch (touch[index].type) { + case MMS114_TYPE_TOUCHSCREEN: + mms114_process_mt(data, touch + index); + break; + + case MMS114_TYPE_TOUCHKEY: + mms114_process_touchkey(data, touch + index); + break; + + default: + dev_err(&client->dev, "Wrong touch type (%d)\n", + touch[index].type); + break; + } + } input_mt_report_pointer_emulation(data->input_dev, true); input_sync(data->input_dev); @@ -446,6 +482,7 @@ static int mms114_probe(struct i2c_client *client) struct input_dev *input_dev; const void *match_data; int error; + int i; if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) { dev_err(&client->dev, "Not supported I2C adapter\n"); @@ -469,6 +506,42 @@ static int mms114_probe(struct i2c_client *client) data->type = (enum mms_type)match_data; + data->num_keycodes = device_property_count_u32(&client->dev, + "linux,keycodes"); + if (data->num_keycodes == -EINVAL) { + data->num_keycodes = 0; + } else if (data->num_keycodes < 0) { + dev_err(&client->dev, + "Unable to parse linux,keycodes property: %d\n", + data->num_keycodes); + return data->num_keycodes; + } else if (data->num_keycodes > MMS114_MAX_TOUCHKEYS) { + dev_warn(&client->dev, + "Found %d linux,keycodes but max is %d, ignoring the rest\n", + data->num_keycodes, MMS114_MAX_TOUCHKEYS); + data->num_keycodes = MMS114_MAX_TOUCHKEYS; + } + + if (data->num_keycodes > 0) { + error = device_property_read_u32_array(&client->dev, + "linux,keycodes", + data->keycodes, + data->num_keycodes); + if (error) { + dev_err(&client->dev, + "Unable to read linux,keycodes values: %d\n", + error); + return error; + } + + input_dev->keycode = data->keycodes; + input_dev->keycodemax = data->num_keycodes; + input_dev->keycodesize = sizeof(data->keycodes[0]); + for (i = 0; i < data->num_keycodes; i++) + input_set_capability(input_dev, + EV_KEY, data->keycodes[i]); + } + input_set_capability(input_dev, EV_ABS, ABS_MT_POSITION_X); input_set_capability(input_dev, EV_ABS, ABS_MT_POSITION_Y); input_set_abs_params(input_dev, ABS_MT_PRESSURE, 0, 255, 0, 0); diff --git a/drivers/input/touchscreen/novatek-nvt-ts.c b/drivers/input/touchscreen/novatek-nvt-ts.c index 7f7d879aac6d..1a797e410a3f 100644 --- a/drivers/input/touchscreen/novatek-nvt-ts.c +++ b/drivers/input/touchscreen/novatek-nvt-ts.c @@ -1,9 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later /* - * Driver for Novatek i2c touchscreen controller as found on - * the Acer Iconia One 7 B1-750 tablet. The Touchscreen controller - * model-number is unknown. Android calls this a "NVT-ts" touchscreen, - * but that may apply to other Novatek controller models too. + * Driver for Novatek NT11205 i2c touchscreen controller as found + * on the Acer Iconia One 7 B1-750 tablet. * * Copyright (c) 2023 Hans de Goede */ @@ -272,7 +270,7 @@ static int nvt_ts_probe(struct i2c_client *client) error = input_register_device(input); if (error) { - dev_err(dev, "failed to request irq: %d\n", error); + dev_err(dev, "failed to register input device: %d\n", error); return error; } @@ -296,6 +294,6 @@ static struct i2c_driver nvt_ts_driver = { module_i2c_driver(nvt_ts_driver); -MODULE_DESCRIPTION("Novatek NVT-ts touchscreen driver"); +MODULE_DESCRIPTION("Novatek NT11205 touchscreen driver"); MODULE_AUTHOR("Hans de Goede "); MODULE_LICENSE("GPL"); diff --git a/drivers/input/touchscreen/pixcir_i2c_ts.c b/drivers/input/touchscreen/pixcir_i2c_ts.c index 554e179c2e48..4ede0687beb0 100644 --- a/drivers/input/touchscreen/pixcir_i2c_ts.c +++ b/drivers/input/touchscreen/pixcir_i2c_ts.c @@ -13,8 +13,8 @@ #include #include #include -#include #include +#include #include #define PIXCIR_MAX_SLOTS 5 /* Max fingers supported by driver */ @@ -515,41 +515,27 @@ static int pixcir_i2c_ts_probe(struct i2c_client *client) input_set_drvdata(input, tsdata); tsdata->gpio_attb = devm_gpiod_get(dev, "attb", GPIOD_IN); - if (IS_ERR(tsdata->gpio_attb)) { - error = PTR_ERR(tsdata->gpio_attb); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to request ATTB gpio: %d\n", - error); - return error; - } + if (IS_ERR(tsdata->gpio_attb)) + return dev_err_probe(dev, PTR_ERR(tsdata->gpio_attb), + "Failed to request ATTB gpio\n"); tsdata->gpio_reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_LOW); - if (IS_ERR(tsdata->gpio_reset)) { - error = PTR_ERR(tsdata->gpio_reset); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to request RESET gpio: %d\n", - error); - return error; - } + if (IS_ERR(tsdata->gpio_reset)) + return dev_err_probe(dev, PTR_ERR(tsdata->gpio_reset), + "Failed to request RESET gpio\n"); tsdata->gpio_wake = devm_gpiod_get_optional(dev, "wake", GPIOD_OUT_HIGH); - if (IS_ERR(tsdata->gpio_wake)) { - error = PTR_ERR(tsdata->gpio_wake); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to get wake gpio: %d\n", error); - return error; - } + if (IS_ERR(tsdata->gpio_wake)) + return dev_err_probe(dev, PTR_ERR(tsdata->gpio_wake), + "Failed to get wake gpio\n"); tsdata->gpio_enable = devm_gpiod_get_optional(dev, "enable", GPIOD_OUT_HIGH); - if (IS_ERR(tsdata->gpio_enable)) { - error = PTR_ERR(tsdata->gpio_enable); - if (error != -EPROBE_DEFER) - dev_err(dev, "Failed to get enable gpio: %d\n", error); - return error; - } + if (IS_ERR(tsdata->gpio_enable)) + return dev_err_probe(dev, PTR_ERR(tsdata->gpio_enable), + "Failed to get enable gpio\n"); if (tsdata->gpio_enable) msleep(100); diff --git a/drivers/input/touchscreen/raydium_i2c_ts.c b/drivers/input/touchscreen/raydium_i2c_ts.c index 76e7d62d5870..78dd3059d585 100644 --- a/drivers/input/touchscreen/raydium_i2c_ts.c +++ b/drivers/input/touchscreen/raydium_i2c_ts.c @@ -1087,32 +1087,20 @@ static int raydium_i2c_probe(struct i2c_client *client) i2c_set_clientdata(client, ts); ts->avdd = devm_regulator_get(&client->dev, "avdd"); - if (IS_ERR(ts->avdd)) { - error = PTR_ERR(ts->avdd); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "Failed to get 'avdd' regulator: %d\n", error); - return error; - } + if (IS_ERR(ts->avdd)) + return dev_err_probe(&client->dev, PTR_ERR(ts->avdd), + "Failed to get 'avdd' regulator\n"); ts->vccio = devm_regulator_get(&client->dev, "vccio"); - if (IS_ERR(ts->vccio)) { - error = PTR_ERR(ts->vccio); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "Failed to get 'vccio' regulator: %d\n", error); - return error; - } + if (IS_ERR(ts->vccio)) + return dev_err_probe(&client->dev, PTR_ERR(ts->vccio), + "Failed to get 'vccio' regulator\n"); ts->reset_gpio = devm_gpiod_get_optional(&client->dev, "reset", GPIOD_OUT_LOW); - if (IS_ERR(ts->reset_gpio)) { - error = PTR_ERR(ts->reset_gpio); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "failed to get reset gpio: %d\n", error); - return error; - } + if (IS_ERR(ts->reset_gpio)) + return dev_err_probe(&client->dev, PTR_ERR(ts->reset_gpio), + "Failed to get reset gpio\n"); error = raydium_i2c_power_on(ts); if (error) diff --git a/drivers/input/touchscreen/resistive-adc-touch.c b/drivers/input/touchscreen/resistive-adc-touch.c index 6f754a8d30b1..7e761ec73273 100644 --- a/drivers/input/touchscreen/resistive-adc-touch.c +++ b/drivers/input/touchscreen/resistive-adc-touch.c @@ -210,12 +210,8 @@ static int grts_probe(struct platform_device *pdev) /* get the channels from IIO device */ st->iio_chans = devm_iio_channel_get_all(dev); - if (IS_ERR(st->iio_chans)) { - error = PTR_ERR(st->iio_chans); - if (error != -EPROBE_DEFER) - dev_err(dev, "can't get iio channels.\n"); - return error; - } + if (IS_ERR(st->iio_chans)) + return dev_err_probe(dev, PTR_ERR(st->iio_chans), "can't get iio channels\n"); if (!device_property_present(dev, "io-channel-names")) return -ENODEV; diff --git a/drivers/input/touchscreen/silead.c b/drivers/input/touchscreen/silead.c index 9e28f962e059..62f562ad5026 100644 --- a/drivers/input/touchscreen/silead.c +++ b/drivers/input/touchscreen/silead.c @@ -706,11 +706,9 @@ static int silead_ts_probe(struct i2c_client *client) /* Power GPIO pin */ data->gpio_power = devm_gpiod_get_optional(dev, "power", GPIOD_OUT_LOW); - if (IS_ERR(data->gpio_power)) { - if (PTR_ERR(data->gpio_power) != -EPROBE_DEFER) - dev_err(dev, "Shutdown GPIO request failed\n"); - return PTR_ERR(data->gpio_power); - } + if (IS_ERR(data->gpio_power)) + return dev_err_probe(dev, PTR_ERR(data->gpio_power), + "Shutdown GPIO request failed\n"); error = silead_ts_setup(client); if (error) diff --git a/drivers/input/touchscreen/sis_i2c.c b/drivers/input/touchscreen/sis_i2c.c index 426564d0fc39..ed56cb546f39 100644 --- a/drivers/input/touchscreen/sis_i2c.c +++ b/drivers/input/touchscreen/sis_i2c.c @@ -310,23 +310,15 @@ static int sis_ts_probe(struct i2c_client *client) ts->attn_gpio = devm_gpiod_get_optional(&client->dev, "attn", GPIOD_IN); - if (IS_ERR(ts->attn_gpio)) { - error = PTR_ERR(ts->attn_gpio); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "Failed to get attention GPIO: %d\n", error); - return error; - } + if (IS_ERR(ts->attn_gpio)) + return dev_err_probe(&client->dev, PTR_ERR(ts->attn_gpio), + "Failed to get attention GPIO\n"); ts->reset_gpio = devm_gpiod_get_optional(&client->dev, "reset", GPIOD_OUT_LOW); - if (IS_ERR(ts->reset_gpio)) { - error = PTR_ERR(ts->reset_gpio); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, - "Failed to get reset GPIO: %d\n", error); - return error; - } + if (IS_ERR(ts->reset_gpio)) + return dev_err_probe(&client->dev, PTR_ERR(ts->reset_gpio), + "Failed to get reset GPIO\n"); sis_ts_reset(ts); diff --git a/drivers/input/touchscreen/surface3_spi.c b/drivers/input/touchscreen/surface3_spi.c index 31d140248f2e..7efbcd0fde4f 100644 --- a/drivers/input/touchscreen/surface3_spi.c +++ b/drivers/input/touchscreen/surface3_spi.c @@ -221,7 +221,6 @@ static void surface3_spi_power(struct surface3_ts_data *data, bool on) */ static int surface3_spi_get_gpio_config(struct surface3_ts_data *data) { - int error; struct device *dev; struct gpio_desc *gpiod; int i; @@ -231,15 +230,9 @@ static int surface3_spi_get_gpio_config(struct surface3_ts_data *data) /* Get the reset lines GPIO pin number */ for (i = 0; i < 2; i++) { gpiod = devm_gpiod_get_index(dev, NULL, i, GPIOD_OUT_LOW); - if (IS_ERR(gpiod)) { - error = PTR_ERR(gpiod); - if (error != -EPROBE_DEFER) - dev_err(dev, - "Failed to get power GPIO %d: %d\n", - i, - error); - return error; - } + if (IS_ERR(gpiod)) + return dev_err_probe(dev, PTR_ERR(gpiod), + "Failed to get power GPIO %d\n", i); data->gpiod_rst[i] = gpiod; } diff --git a/drivers/input/touchscreen/sx8654.c b/drivers/input/touchscreen/sx8654.c index 0293c493bc79..f5c5881cef6b 100644 --- a/drivers/input/touchscreen/sx8654.c +++ b/drivers/input/touchscreen/sx8654.c @@ -323,13 +323,9 @@ static int sx8654_probe(struct i2c_client *client) sx8654->gpio_reset = devm_gpiod_get_optional(&client->dev, "reset", GPIOD_OUT_HIGH); - if (IS_ERR(sx8654->gpio_reset)) { - error = PTR_ERR(sx8654->gpio_reset); - if (error != -EPROBE_DEFER) - dev_err(&client->dev, "unable to get reset-gpio: %d\n", - error); - return error; - } + if (IS_ERR(sx8654->gpio_reset)) + return dev_err_probe(&client->dev, PTR_ERR(sx8654->gpio_reset), + "unable to get reset-gpio\n"); dev_dbg(&client->dev, "got GPIO reset pin\n"); sx8654->data = device_get_match_data(&client->dev); diff --git a/drivers/input/touchscreen/ti_am335x_tsc.c b/drivers/input/touchscreen/ti_am335x_tsc.c index decf2d24a115..9aa4e35fb4f5 100644 --- a/drivers/input/touchscreen/ti_am335x_tsc.c +++ b/drivers/input/touchscreen/ti_am335x_tsc.c @@ -25,7 +25,6 @@ #include #include #include -#include #include #include diff --git a/drivers/media/dvb-frontends/ascot2e.c b/drivers/media/dvb-frontends/ascot2e.c index 9b00b56230b6..cf8e5f1bd101 100644 --- a/drivers/media/dvb-frontends/ascot2e.c +++ b/drivers/media/dvb-frontends/ascot2e.c @@ -533,7 +533,7 @@ struct dvb_frontend *ascot2e_attach(struct dvb_frontend *fe, priv->i2c_address, priv->i2c); return fe; } -EXPORT_SYMBOL(ascot2e_attach); +EXPORT_SYMBOL_GPL(ascot2e_attach); MODULE_DESCRIPTION("Sony ASCOT2E terr/cab tuner driver"); MODULE_AUTHOR("info@netup.ru"); diff --git a/drivers/media/dvb-frontends/atbm8830.c b/drivers/media/dvb-frontends/atbm8830.c index bdd16b9c5824..778c865085bf 100644 --- a/drivers/media/dvb-frontends/atbm8830.c +++ b/drivers/media/dvb-frontends/atbm8830.c @@ -489,7 +489,7 @@ error_out: return NULL; } -EXPORT_SYMBOL(atbm8830_attach); +EXPORT_SYMBOL_GPL(atbm8830_attach); MODULE_DESCRIPTION("AltoBeam ATBM8830/8831 GB20600 demodulator driver"); MODULE_AUTHOR("David T. L. Wong "); diff --git a/drivers/media/dvb-frontends/au8522_dig.c b/drivers/media/dvb-frontends/au8522_dig.c index 78cafdf27961..230436bf6cbd 100644 --- a/drivers/media/dvb-frontends/au8522_dig.c +++ b/drivers/media/dvb-frontends/au8522_dig.c @@ -879,7 +879,7 @@ error: au8522_release_state(state); return NULL; } -EXPORT_SYMBOL(au8522_attach); +EXPORT_SYMBOL_GPL(au8522_attach); static const struct dvb_frontend_ops au8522_ops = { .delsys = { SYS_ATSC, SYS_DVBC_ANNEX_B }, diff --git a/drivers/media/dvb-frontends/bcm3510.c b/drivers/media/dvb-frontends/bcm3510.c index 68b92b4419cf..b3f5c49accaf 100644 --- a/drivers/media/dvb-frontends/bcm3510.c +++ b/drivers/media/dvb-frontends/bcm3510.c @@ -835,7 +835,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(bcm3510_attach); +EXPORT_SYMBOL_GPL(bcm3510_attach); static const struct dvb_frontend_ops bcm3510_ops = { .delsys = { SYS_ATSC, SYS_DVBC_ANNEX_B }, diff --git a/drivers/media/dvb-frontends/cx22700.c b/drivers/media/dvb-frontends/cx22700.c index b39ff516271b..1d04c0a652b2 100644 --- a/drivers/media/dvb-frontends/cx22700.c +++ b/drivers/media/dvb-frontends/cx22700.c @@ -432,4 +432,4 @@ MODULE_DESCRIPTION("Conexant CX22700 DVB-T Demodulator driver"); MODULE_AUTHOR("Holger Waechtler"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(cx22700_attach); +EXPORT_SYMBOL_GPL(cx22700_attach); diff --git a/drivers/media/dvb-frontends/cx22702.c b/drivers/media/dvb-frontends/cx22702.c index cc6acbf6393d..61ad34b7004b 100644 --- a/drivers/media/dvb-frontends/cx22702.c +++ b/drivers/media/dvb-frontends/cx22702.c @@ -604,7 +604,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(cx22702_attach); +EXPORT_SYMBOL_GPL(cx22702_attach); static const struct dvb_frontend_ops cx22702_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/dvb-frontends/cx24110.c b/drivers/media/dvb-frontends/cx24110.c index 6f99d6a27be2..9aeea089756f 100644 --- a/drivers/media/dvb-frontends/cx24110.c +++ b/drivers/media/dvb-frontends/cx24110.c @@ -653,4 +653,4 @@ MODULE_DESCRIPTION("Conexant CX24110 DVB-S Demodulator driver"); MODULE_AUTHOR("Peter Hettkamp"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(cx24110_attach); +EXPORT_SYMBOL_GPL(cx24110_attach); diff --git a/drivers/media/dvb-frontends/cx24113.c b/drivers/media/dvb-frontends/cx24113.c index dd55d314bf9a..203cb6b3f941 100644 --- a/drivers/media/dvb-frontends/cx24113.c +++ b/drivers/media/dvb-frontends/cx24113.c @@ -590,7 +590,7 @@ error: return NULL; } -EXPORT_SYMBOL(cx24113_attach); +EXPORT_SYMBOL_GPL(cx24113_attach); module_param(debug, int, 0644); MODULE_PARM_DESC(debug, "Activates frontend debugging (default:0)"); diff --git a/drivers/media/dvb-frontends/cx24116.c b/drivers/media/dvb-frontends/cx24116.c index ea8264ccbb4e..8b978a9f74a4 100644 --- a/drivers/media/dvb-frontends/cx24116.c +++ b/drivers/media/dvb-frontends/cx24116.c @@ -1133,7 +1133,7 @@ struct dvb_frontend *cx24116_attach(const struct cx24116_config *config, state->frontend.demodulator_priv = state; return &state->frontend; } -EXPORT_SYMBOL(cx24116_attach); +EXPORT_SYMBOL_GPL(cx24116_attach); /* * Initialise or wake up device diff --git a/drivers/media/dvb-frontends/cx24120.c b/drivers/media/dvb-frontends/cx24120.c index 0f778660c72b..44515fdbe91d 100644 --- a/drivers/media/dvb-frontends/cx24120.c +++ b/drivers/media/dvb-frontends/cx24120.c @@ -305,7 +305,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(cx24120_attach); +EXPORT_SYMBOL_GPL(cx24120_attach); static int cx24120_test_rom(struct cx24120_state *state) { diff --git a/drivers/media/dvb-frontends/cx24123.c b/drivers/media/dvb-frontends/cx24123.c index 3d84ee17e54c..539889e638cc 100644 --- a/drivers/media/dvb-frontends/cx24123.c +++ b/drivers/media/dvb-frontends/cx24123.c @@ -1096,7 +1096,7 @@ error: return NULL; } -EXPORT_SYMBOL(cx24123_attach); +EXPORT_SYMBOL_GPL(cx24123_attach); static const struct dvb_frontend_ops cx24123_ops = { .delsys = { SYS_DVBS }, diff --git a/drivers/media/dvb-frontends/cxd2820r_core.c b/drivers/media/dvb-frontends/cxd2820r_core.c index d7ee294c6833..7feb08dccfa1 100644 --- a/drivers/media/dvb-frontends/cxd2820r_core.c +++ b/drivers/media/dvb-frontends/cxd2820r_core.c @@ -536,7 +536,7 @@ struct dvb_frontend *cxd2820r_attach(const struct cxd2820r_config *config, return pdata.get_dvb_frontend(client); } -EXPORT_SYMBOL(cxd2820r_attach); +EXPORT_SYMBOL_GPL(cxd2820r_attach); static struct dvb_frontend *cxd2820r_get_dvb_frontend(struct i2c_client *client) { diff --git a/drivers/media/dvb-frontends/cxd2841er.c b/drivers/media/dvb-frontends/cxd2841er.c index ef403a9fb753..d925ca24183b 100644 --- a/drivers/media/dvb-frontends/cxd2841er.c +++ b/drivers/media/dvb-frontends/cxd2841er.c @@ -3930,14 +3930,14 @@ struct dvb_frontend *cxd2841er_attach_s(struct cxd2841er_config *cfg, { return cxd2841er_attach(cfg, i2c, SYS_DVBS); } -EXPORT_SYMBOL(cxd2841er_attach_s); +EXPORT_SYMBOL_GPL(cxd2841er_attach_s); struct dvb_frontend *cxd2841er_attach_t_c(struct cxd2841er_config *cfg, struct i2c_adapter *i2c) { return cxd2841er_attach(cfg, i2c, 0); } -EXPORT_SYMBOL(cxd2841er_attach_t_c); +EXPORT_SYMBOL_GPL(cxd2841er_attach_t_c); static const struct dvb_frontend_ops cxd2841er_dvbs_s2_ops = { .delsys = { SYS_DVBS, SYS_DVBS2 }, diff --git a/drivers/media/dvb-frontends/cxd2880/cxd2880_top.c b/drivers/media/dvb-frontends/cxd2880/cxd2880_top.c index f67b6d24b8d4..a06d8368ca79 100644 --- a/drivers/media/dvb-frontends/cxd2880/cxd2880_top.c +++ b/drivers/media/dvb-frontends/cxd2880/cxd2880_top.c @@ -1950,7 +1950,7 @@ struct dvb_frontend *cxd2880_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(cxd2880_attach); +EXPORT_SYMBOL_GPL(cxd2880_attach); MODULE_DESCRIPTION("Sony CXD2880 DVB-T2/T tuner + demod driver"); MODULE_AUTHOR("Sony Semiconductor Solutions Corporation"); diff --git a/drivers/media/dvb-frontends/dib0070.c b/drivers/media/dvb-frontends/dib0070.c index cafb41dba861..9a8e7cdd2a24 100644 --- a/drivers/media/dvb-frontends/dib0070.c +++ b/drivers/media/dvb-frontends/dib0070.c @@ -762,7 +762,7 @@ free_mem: fe->tuner_priv = NULL; return NULL; } -EXPORT_SYMBOL(dib0070_attach); +EXPORT_SYMBOL_GPL(dib0070_attach); MODULE_AUTHOR("Patrick Boettcher "); MODULE_DESCRIPTION("Driver for the DiBcom 0070 base-band RF Tuner"); diff --git a/drivers/media/dvb-frontends/dib0090.c b/drivers/media/dvb-frontends/dib0090.c index 903da33642df..c958bcff026e 100644 --- a/drivers/media/dvb-frontends/dib0090.c +++ b/drivers/media/dvb-frontends/dib0090.c @@ -2634,7 +2634,7 @@ struct dvb_frontend *dib0090_register(struct dvb_frontend *fe, struct i2c_adapte return NULL; } -EXPORT_SYMBOL(dib0090_register); +EXPORT_SYMBOL_GPL(dib0090_register); struct dvb_frontend *dib0090_fw_register(struct dvb_frontend *fe, struct i2c_adapter *i2c, const struct dib0090_config *config) { @@ -2660,7 +2660,7 @@ free_mem: fe->tuner_priv = NULL; return NULL; } -EXPORT_SYMBOL(dib0090_fw_register); +EXPORT_SYMBOL_GPL(dib0090_fw_register); MODULE_AUTHOR("Patrick Boettcher "); MODULE_AUTHOR("Olivier Grenie "); diff --git a/drivers/media/dvb-frontends/dib3000mb.c b/drivers/media/dvb-frontends/dib3000mb.c index a6c2fc4586eb..c598b2a63325 100644 --- a/drivers/media/dvb-frontends/dib3000mb.c +++ b/drivers/media/dvb-frontends/dib3000mb.c @@ -815,4 +815,4 @@ MODULE_AUTHOR(DRIVER_AUTHOR); MODULE_DESCRIPTION(DRIVER_DESC); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(dib3000mb_attach); +EXPORT_SYMBOL_GPL(dib3000mb_attach); diff --git a/drivers/media/dvb-frontends/dib3000mc.c b/drivers/media/dvb-frontends/dib3000mc.c index 2e11a246aae0..c2fca8289aba 100644 --- a/drivers/media/dvb-frontends/dib3000mc.c +++ b/drivers/media/dvb-frontends/dib3000mc.c @@ -935,7 +935,7 @@ error: kfree(st); return NULL; } -EXPORT_SYMBOL(dib3000mc_attach); +EXPORT_SYMBOL_GPL(dib3000mc_attach); static const struct dvb_frontend_ops dib3000mc_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/dvb-frontends/dib7000m.c b/drivers/media/dvb-frontends/dib7000m.c index 97ce97789c9e..fdb22f32e3a1 100644 --- a/drivers/media/dvb-frontends/dib7000m.c +++ b/drivers/media/dvb-frontends/dib7000m.c @@ -1434,7 +1434,7 @@ error: kfree(st); return NULL; } -EXPORT_SYMBOL(dib7000m_attach); +EXPORT_SYMBOL_GPL(dib7000m_attach); static const struct dvb_frontend_ops dib7000m_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/dvb-frontends/dib7000p.c b/drivers/media/dvb-frontends/dib7000p.c index 9273758bf140..444fe1c4bf2d 100644 --- a/drivers/media/dvb-frontends/dib7000p.c +++ b/drivers/media/dvb-frontends/dib7000p.c @@ -2822,7 +2822,7 @@ void *dib7000p_attach(struct dib7000p_ops *ops) return ops; } -EXPORT_SYMBOL(dib7000p_attach); +EXPORT_SYMBOL_GPL(dib7000p_attach); static const struct dvb_frontend_ops dib7000p_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/dvb-frontends/dib8000.c b/drivers/media/dvb-frontends/dib8000.c index 2abda7d1cb6e..2f5165918163 100644 --- a/drivers/media/dvb-frontends/dib8000.c +++ b/drivers/media/dvb-frontends/dib8000.c @@ -4527,7 +4527,7 @@ void *dib8000_attach(struct dib8000_ops *ops) return ops; } -EXPORT_SYMBOL(dib8000_attach); +EXPORT_SYMBOL_GPL(dib8000_attach); MODULE_AUTHOR("Olivier Grenie "); MODULE_DESCRIPTION("Driver for the DiBcom 8000 ISDB-T demodulator"); diff --git a/drivers/media/dvb-frontends/dib9000.c b/drivers/media/dvb-frontends/dib9000.c index 1c57587a917a..83cf6eadd49c 100644 --- a/drivers/media/dvb-frontends/dib9000.c +++ b/drivers/media/dvb-frontends/dib9000.c @@ -2546,7 +2546,7 @@ error: kfree(st); return NULL; } -EXPORT_SYMBOL(dib9000_attach); +EXPORT_SYMBOL_GPL(dib9000_attach); static const struct dvb_frontend_ops dib9000_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/dvb-frontends/drx39xyj/drxj.c b/drivers/media/dvb-frontends/drx39xyj/drxj.c index 68f4e8b5a0ab..a738573c8cd7 100644 --- a/drivers/media/dvb-frontends/drx39xyj/drxj.c +++ b/drivers/media/dvb-frontends/drx39xyj/drxj.c @@ -12372,7 +12372,7 @@ error: return NULL; } -EXPORT_SYMBOL(drx39xxj_attach); +EXPORT_SYMBOL_GPL(drx39xxj_attach); static const struct dvb_frontend_ops drx39xxj_ops = { .delsys = { SYS_ATSC, SYS_DVBC_ANNEX_B }, diff --git a/drivers/media/dvb-frontends/drxd_hard.c b/drivers/media/dvb-frontends/drxd_hard.c index 9860cae65f1c..6a531937f4bb 100644 --- a/drivers/media/dvb-frontends/drxd_hard.c +++ b/drivers/media/dvb-frontends/drxd_hard.c @@ -2939,7 +2939,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(drxd_attach); +EXPORT_SYMBOL_GPL(drxd_attach); MODULE_DESCRIPTION("DRXD driver"); MODULE_AUTHOR("Micronas"); diff --git a/drivers/media/dvb-frontends/drxk_hard.c b/drivers/media/dvb-frontends/drxk_hard.c index 2770baebbbbc..87f3d4f0eb8c 100644 --- a/drivers/media/dvb-frontends/drxk_hard.c +++ b/drivers/media/dvb-frontends/drxk_hard.c @@ -6814,7 +6814,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(drxk_attach); +EXPORT_SYMBOL_GPL(drxk_attach); MODULE_DESCRIPTION("DRX-K driver"); MODULE_AUTHOR("Ralph Metzler"); diff --git a/drivers/media/dvb-frontends/ds3000.c b/drivers/media/dvb-frontends/ds3000.c index 20fcf31af165..515aa7c7baf2 100644 --- a/drivers/media/dvb-frontends/ds3000.c +++ b/drivers/media/dvb-frontends/ds3000.c @@ -859,7 +859,7 @@ struct dvb_frontend *ds3000_attach(const struct ds3000_config *config, ds3000_set_voltage(&state->frontend, SEC_VOLTAGE_OFF); return &state->frontend; } -EXPORT_SYMBOL(ds3000_attach); +EXPORT_SYMBOL_GPL(ds3000_attach); static int ds3000_set_carrier_offset(struct dvb_frontend *fe, s32 carrier_offset_khz) diff --git a/drivers/media/dvb-frontends/dvb-pll.c b/drivers/media/dvb-frontends/dvb-pll.c index 90cb41eacf98..ef697ab6bc2e 100644 --- a/drivers/media/dvb-frontends/dvb-pll.c +++ b/drivers/media/dvb-frontends/dvb-pll.c @@ -866,7 +866,7 @@ out: return NULL; } -EXPORT_SYMBOL(dvb_pll_attach); +EXPORT_SYMBOL_GPL(dvb_pll_attach); static int diff --git a/drivers/media/dvb-frontends/ec100.c b/drivers/media/dvb-frontends/ec100.c index 03bd80666cf8..2ad0a3c2f756 100644 --- a/drivers/media/dvb-frontends/ec100.c +++ b/drivers/media/dvb-frontends/ec100.c @@ -299,7 +299,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(ec100_attach); +EXPORT_SYMBOL_GPL(ec100_attach); static const struct dvb_frontend_ops ec100_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/dvb-frontends/helene.c b/drivers/media/dvb-frontends/helene.c index 68c1a3e0e2ba..f127adee3ebb 100644 --- a/drivers/media/dvb-frontends/helene.c +++ b/drivers/media/dvb-frontends/helene.c @@ -1025,7 +1025,7 @@ struct dvb_frontend *helene_attach_s(struct dvb_frontend *fe, priv->i2c_address, priv->i2c); return fe; } -EXPORT_SYMBOL(helene_attach_s); +EXPORT_SYMBOL_GPL(helene_attach_s); struct dvb_frontend *helene_attach(struct dvb_frontend *fe, const struct helene_config *config, @@ -1061,7 +1061,7 @@ struct dvb_frontend *helene_attach(struct dvb_frontend *fe, priv->i2c_address, priv->i2c); return fe; } -EXPORT_SYMBOL(helene_attach); +EXPORT_SYMBOL_GPL(helene_attach); static int helene_probe(struct i2c_client *client) { diff --git a/drivers/media/dvb-frontends/horus3a.c b/drivers/media/dvb-frontends/horus3a.c index 24bf5cbcc184..0330b78a5b3f 100644 --- a/drivers/media/dvb-frontends/horus3a.c +++ b/drivers/media/dvb-frontends/horus3a.c @@ -395,7 +395,7 @@ struct dvb_frontend *horus3a_attach(struct dvb_frontend *fe, priv->i2c_address, priv->i2c); return fe; } -EXPORT_SYMBOL(horus3a_attach); +EXPORT_SYMBOL_GPL(horus3a_attach); MODULE_DESCRIPTION("Sony HORUS3A satellite tuner driver"); MODULE_AUTHOR("Sergey Kozlov "); diff --git a/drivers/media/dvb-frontends/isl6405.c b/drivers/media/dvb-frontends/isl6405.c index 2cd69b4ff82c..7d28a743f97e 100644 --- a/drivers/media/dvb-frontends/isl6405.c +++ b/drivers/media/dvb-frontends/isl6405.c @@ -141,7 +141,7 @@ struct dvb_frontend *isl6405_attach(struct dvb_frontend *fe, struct i2c_adapter return fe; } -EXPORT_SYMBOL(isl6405_attach); +EXPORT_SYMBOL_GPL(isl6405_attach); MODULE_DESCRIPTION("Driver for lnb supply and control ic isl6405"); MODULE_AUTHOR("Hartmut Hackmann & Oliver Endriss"); diff --git a/drivers/media/dvb-frontends/isl6421.c b/drivers/media/dvb-frontends/isl6421.c index 43b0dfc6f453..2e9f6f12f849 100644 --- a/drivers/media/dvb-frontends/isl6421.c +++ b/drivers/media/dvb-frontends/isl6421.c @@ -213,7 +213,7 @@ struct dvb_frontend *isl6421_attach(struct dvb_frontend *fe, struct i2c_adapter return fe; } -EXPORT_SYMBOL(isl6421_attach); +EXPORT_SYMBOL_GPL(isl6421_attach); MODULE_DESCRIPTION("Driver for lnb supply and control ic isl6421"); MODULE_AUTHOR("Andrew de Quincey & Oliver Endriss"); diff --git a/drivers/media/dvb-frontends/isl6423.c b/drivers/media/dvb-frontends/isl6423.c index 8cd1bb88ce6e..a0d0a3834057 100644 --- a/drivers/media/dvb-frontends/isl6423.c +++ b/drivers/media/dvb-frontends/isl6423.c @@ -289,7 +289,7 @@ exit: fe->sec_priv = NULL; return NULL; } -EXPORT_SYMBOL(isl6423_attach); +EXPORT_SYMBOL_GPL(isl6423_attach); MODULE_DESCRIPTION("ISL6423 SEC"); MODULE_AUTHOR("Manu Abraham"); diff --git a/drivers/media/dvb-frontends/itd1000.c b/drivers/media/dvb-frontends/itd1000.c index 1b33478653d1..f8f362f50e78 100644 --- a/drivers/media/dvb-frontends/itd1000.c +++ b/drivers/media/dvb-frontends/itd1000.c @@ -389,7 +389,7 @@ struct dvb_frontend *itd1000_attach(struct dvb_frontend *fe, struct i2c_adapter return fe; } -EXPORT_SYMBOL(itd1000_attach); +EXPORT_SYMBOL_GPL(itd1000_attach); MODULE_AUTHOR("Patrick Boettcher "); MODULE_DESCRIPTION("Integrant ITD1000 driver"); diff --git a/drivers/media/dvb-frontends/ix2505v.c b/drivers/media/dvb-frontends/ix2505v.c index 73f27105c139..3212e333d472 100644 --- a/drivers/media/dvb-frontends/ix2505v.c +++ b/drivers/media/dvb-frontends/ix2505v.c @@ -302,7 +302,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(ix2505v_attach); +EXPORT_SYMBOL_GPL(ix2505v_attach); module_param_named(debug, ix2505v_debug, int, 0644); MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); diff --git a/drivers/media/dvb-frontends/l64781.c b/drivers/media/dvb-frontends/l64781.c index c5106a1ea1cd..fe5af2453d55 100644 --- a/drivers/media/dvb-frontends/l64781.c +++ b/drivers/media/dvb-frontends/l64781.c @@ -593,4 +593,4 @@ MODULE_DESCRIPTION("LSI L64781 DVB-T Demodulator driver"); MODULE_AUTHOR("Holger Waechtler, Marko Kohtala"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(l64781_attach); +EXPORT_SYMBOL_GPL(l64781_attach); diff --git a/drivers/media/dvb-frontends/lg2160.c b/drivers/media/dvb-frontends/lg2160.c index f343066c297e..fe700aa56bff 100644 --- a/drivers/media/dvb-frontends/lg2160.c +++ b/drivers/media/dvb-frontends/lg2160.c @@ -1426,7 +1426,7 @@ struct dvb_frontend *lg2160_attach(const struct lg2160_config *config, return &state->frontend; } -EXPORT_SYMBOL(lg2160_attach); +EXPORT_SYMBOL_GPL(lg2160_attach); MODULE_DESCRIPTION("LG Electronics LG216x ATSC/MH Demodulator Driver"); MODULE_AUTHOR("Michael Krufky "); diff --git a/drivers/media/dvb-frontends/lgdt3305.c b/drivers/media/dvb-frontends/lgdt3305.c index c15d3735d34c..bdc8311e1c0b 100644 --- a/drivers/media/dvb-frontends/lgdt3305.c +++ b/drivers/media/dvb-frontends/lgdt3305.c @@ -1148,7 +1148,7 @@ fail: kfree(state); return NULL; } -EXPORT_SYMBOL(lgdt3305_attach); +EXPORT_SYMBOL_GPL(lgdt3305_attach); static const struct dvb_frontend_ops lgdt3304_ops = { .delsys = { SYS_ATSC, SYS_DVBC_ANNEX_B }, diff --git a/drivers/media/dvb-frontends/lgdt3306a.c b/drivers/media/dvb-frontends/lgdt3306a.c index 3c6650f6e9a3..263887592415 100644 --- a/drivers/media/dvb-frontends/lgdt3306a.c +++ b/drivers/media/dvb-frontends/lgdt3306a.c @@ -1859,7 +1859,7 @@ fail: kfree(state); return NULL; } -EXPORT_SYMBOL(lgdt3306a_attach); +EXPORT_SYMBOL_GPL(lgdt3306a_attach); #ifdef DBG_DUMP diff --git a/drivers/media/dvb-frontends/lgdt330x.c b/drivers/media/dvb-frontends/lgdt330x.c index 97a10996c7fa..081d6ad3ce72 100644 --- a/drivers/media/dvb-frontends/lgdt330x.c +++ b/drivers/media/dvb-frontends/lgdt330x.c @@ -927,7 +927,7 @@ struct dvb_frontend *lgdt330x_attach(const struct lgdt330x_config *_config, return lgdt330x_get_dvb_frontend(client); } -EXPORT_SYMBOL(lgdt330x_attach); +EXPORT_SYMBOL_GPL(lgdt330x_attach); static const struct dvb_frontend_ops lgdt3302_ops = { .delsys = { SYS_ATSC, SYS_DVBC_ANNEX_B }, diff --git a/drivers/media/dvb-frontends/lgs8gxx.c b/drivers/media/dvb-frontends/lgs8gxx.c index 30014979b985..ffaf60e16ecd 100644 --- a/drivers/media/dvb-frontends/lgs8gxx.c +++ b/drivers/media/dvb-frontends/lgs8gxx.c @@ -1043,7 +1043,7 @@ error_out: return NULL; } -EXPORT_SYMBOL(lgs8gxx_attach); +EXPORT_SYMBOL_GPL(lgs8gxx_attach); MODULE_DESCRIPTION("Legend Silicon LGS8913/LGS8GXX DMB-TH demodulator driver"); MODULE_AUTHOR("David T. L. Wong "); diff --git a/drivers/media/dvb-frontends/lnbh25.c b/drivers/media/dvb-frontends/lnbh25.c index 9ffe06cd787d..41bec050642b 100644 --- a/drivers/media/dvb-frontends/lnbh25.c +++ b/drivers/media/dvb-frontends/lnbh25.c @@ -173,7 +173,7 @@ struct dvb_frontend *lnbh25_attach(struct dvb_frontend *fe, __func__, priv->i2c_address); return fe; } -EXPORT_SYMBOL(lnbh25_attach); +EXPORT_SYMBOL_GPL(lnbh25_attach); MODULE_DESCRIPTION("ST LNBH25 driver"); MODULE_AUTHOR("info@netup.ru"); diff --git a/drivers/media/dvb-frontends/lnbp21.c b/drivers/media/dvb-frontends/lnbp21.c index e564974162d6..32593b1f75a3 100644 --- a/drivers/media/dvb-frontends/lnbp21.c +++ b/drivers/media/dvb-frontends/lnbp21.c @@ -155,7 +155,7 @@ struct dvb_frontend *lnbh24_attach(struct dvb_frontend *fe, return lnbx2x_attach(fe, i2c, override_set, override_clear, i2c_addr, LNBH24_TTX); } -EXPORT_SYMBOL(lnbh24_attach); +EXPORT_SYMBOL_GPL(lnbh24_attach); struct dvb_frontend *lnbp21_attach(struct dvb_frontend *fe, struct i2c_adapter *i2c, u8 override_set, @@ -164,7 +164,7 @@ struct dvb_frontend *lnbp21_attach(struct dvb_frontend *fe, return lnbx2x_attach(fe, i2c, override_set, override_clear, 0x08, LNBP21_ISEL); } -EXPORT_SYMBOL(lnbp21_attach); +EXPORT_SYMBOL_GPL(lnbp21_attach); MODULE_DESCRIPTION("Driver for lnb supply and control ic lnbp21, lnbh24"); MODULE_AUTHOR("Oliver Endriss, Igor M. Liplianin"); diff --git a/drivers/media/dvb-frontends/lnbp22.c b/drivers/media/dvb-frontends/lnbp22.c index b8c7145d4cef..cb4ea5d3fad4 100644 --- a/drivers/media/dvb-frontends/lnbp22.c +++ b/drivers/media/dvb-frontends/lnbp22.c @@ -125,7 +125,7 @@ struct dvb_frontend *lnbp22_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(lnbp22_attach); +EXPORT_SYMBOL_GPL(lnbp22_attach); MODULE_DESCRIPTION("Driver for lnb supply and control ic lnbp22"); MODULE_AUTHOR("Dominik Kuhlen"); diff --git a/drivers/media/dvb-frontends/m88ds3103.c b/drivers/media/dvb-frontends/m88ds3103.c index cf49ac56a37e..cf037b61b226 100644 --- a/drivers/media/dvb-frontends/m88ds3103.c +++ b/drivers/media/dvb-frontends/m88ds3103.c @@ -1695,7 +1695,7 @@ struct dvb_frontend *m88ds3103_attach(const struct m88ds3103_config *cfg, *tuner_i2c_adapter = pdata.get_i2c_adapter(client); return pdata.get_dvb_frontend(client); } -EXPORT_SYMBOL(m88ds3103_attach); +EXPORT_SYMBOL_GPL(m88ds3103_attach); static const struct dvb_frontend_ops m88ds3103_ops = { .delsys = {SYS_DVBS, SYS_DVBS2}, diff --git a/drivers/media/dvb-frontends/m88rs2000.c b/drivers/media/dvb-frontends/m88rs2000.c index b294ba87e934..2aa98203cd65 100644 --- a/drivers/media/dvb-frontends/m88rs2000.c +++ b/drivers/media/dvb-frontends/m88rs2000.c @@ -808,7 +808,7 @@ error: return NULL; } -EXPORT_SYMBOL(m88rs2000_attach); +EXPORT_SYMBOL_GPL(m88rs2000_attach); MODULE_DESCRIPTION("M88RS2000 DVB-S Demodulator driver"); MODULE_AUTHOR("Malcolm Priestley tvboxspy@gmail.com"); diff --git a/drivers/media/dvb-frontends/mb86a16.c b/drivers/media/dvb-frontends/mb86a16.c index 3ec2cb4fa504..0fc45896e7b8 100644 --- a/drivers/media/dvb-frontends/mb86a16.c +++ b/drivers/media/dvb-frontends/mb86a16.c @@ -1853,6 +1853,6 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(mb86a16_attach); +EXPORT_SYMBOL_GPL(mb86a16_attach); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Manu Abraham"); diff --git a/drivers/media/dvb-frontends/mb86a20s.c b/drivers/media/dvb-frontends/mb86a20s.c index 125fed4891ba..f8e4bbee5bd5 100644 --- a/drivers/media/dvb-frontends/mb86a20s.c +++ b/drivers/media/dvb-frontends/mb86a20s.c @@ -2078,7 +2078,7 @@ struct dvb_frontend *mb86a20s_attach(const struct mb86a20s_config *config, dev_info(&i2c->dev, "Detected a Fujitsu mb86a20s frontend\n"); return &state->frontend; } -EXPORT_SYMBOL(mb86a20s_attach); +EXPORT_SYMBOL_GPL(mb86a20s_attach); static const struct dvb_frontend_ops mb86a20s_ops = { .delsys = { SYS_ISDBT }, diff --git a/drivers/media/dvb-frontends/mt312.c b/drivers/media/dvb-frontends/mt312.c index d43a67045dbe..fb867dd8a26b 100644 --- a/drivers/media/dvb-frontends/mt312.c +++ b/drivers/media/dvb-frontends/mt312.c @@ -827,7 +827,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(mt312_attach); +EXPORT_SYMBOL_GPL(mt312_attach); module_param(debug, int, 0644); MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); diff --git a/drivers/media/dvb-frontends/mt352.c b/drivers/media/dvb-frontends/mt352.c index 399d5c519027..1b2889f5cf67 100644 --- a/drivers/media/dvb-frontends/mt352.c +++ b/drivers/media/dvb-frontends/mt352.c @@ -593,4 +593,4 @@ MODULE_DESCRIPTION("Zarlink MT352 DVB-T Demodulator driver"); MODULE_AUTHOR("Holger Waechtler, Daniel Mack, Antonio Mancuso"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(mt352_attach); +EXPORT_SYMBOL_GPL(mt352_attach); diff --git a/drivers/media/dvb-frontends/nxt200x.c b/drivers/media/dvb-frontends/nxt200x.c index 200b6dbc75f8..1c549ada6ebf 100644 --- a/drivers/media/dvb-frontends/nxt200x.c +++ b/drivers/media/dvb-frontends/nxt200x.c @@ -1216,5 +1216,5 @@ MODULE_DESCRIPTION("NXT200X (ATSC 8VSB & ITU-T J.83 AnnexB 64/256 QAM) Demodulat MODULE_AUTHOR("Kirk Lapray, Michael Krufky, Jean-Francois Thibert, and Taylor Jacob"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(nxt200x_attach); +EXPORT_SYMBOL_GPL(nxt200x_attach); diff --git a/drivers/media/dvb-frontends/nxt6000.c b/drivers/media/dvb-frontends/nxt6000.c index 136918f82dda..e8d4940370dd 100644 --- a/drivers/media/dvb-frontends/nxt6000.c +++ b/drivers/media/dvb-frontends/nxt6000.c @@ -621,4 +621,4 @@ MODULE_DESCRIPTION("NxtWave NXT6000 DVB-T demodulator driver"); MODULE_AUTHOR("Florian Schirmer"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(nxt6000_attach); +EXPORT_SYMBOL_GPL(nxt6000_attach); diff --git a/drivers/media/dvb-frontends/or51132.c b/drivers/media/dvb-frontends/or51132.c index 355f3598627b..74e04c7cca1e 100644 --- a/drivers/media/dvb-frontends/or51132.c +++ b/drivers/media/dvb-frontends/or51132.c @@ -605,4 +605,4 @@ MODULE_AUTHOR("Kirk Lapray"); MODULE_AUTHOR("Trent Piepho"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(or51132_attach); +EXPORT_SYMBOL_GPL(or51132_attach); diff --git a/drivers/media/dvb-frontends/or51211.c b/drivers/media/dvb-frontends/or51211.c index ae732dc5116e..2e8e7071a67a 100644 --- a/drivers/media/dvb-frontends/or51211.c +++ b/drivers/media/dvb-frontends/or51211.c @@ -551,5 +551,5 @@ MODULE_DESCRIPTION("Oren OR51211 VSB [pcHDTV HD-2000] Demodulator Driver"); MODULE_AUTHOR("Kirk Lapray"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(or51211_attach); +EXPORT_SYMBOL_GPL(or51211_attach); diff --git a/drivers/media/dvb-frontends/s5h1409.c b/drivers/media/dvb-frontends/s5h1409.c index 3089cc174a6f..28b1dca077ea 100644 --- a/drivers/media/dvb-frontends/s5h1409.c +++ b/drivers/media/dvb-frontends/s5h1409.c @@ -981,7 +981,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(s5h1409_attach); +EXPORT_SYMBOL_GPL(s5h1409_attach); static const struct dvb_frontend_ops s5h1409_ops = { .delsys = { SYS_ATSC, SYS_DVBC_ANNEX_B }, diff --git a/drivers/media/dvb-frontends/s5h1411.c b/drivers/media/dvb-frontends/s5h1411.c index 2563a72e98b7..fc48e659c2d8 100644 --- a/drivers/media/dvb-frontends/s5h1411.c +++ b/drivers/media/dvb-frontends/s5h1411.c @@ -900,7 +900,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(s5h1411_attach); +EXPORT_SYMBOL_GPL(s5h1411_attach); static const struct dvb_frontend_ops s5h1411_ops = { .delsys = { SYS_ATSC, SYS_DVBC_ANNEX_B }, diff --git a/drivers/media/dvb-frontends/s5h1420.c b/drivers/media/dvb-frontends/s5h1420.c index 6bdec2898bc8..d700de1ea6c2 100644 --- a/drivers/media/dvb-frontends/s5h1420.c +++ b/drivers/media/dvb-frontends/s5h1420.c @@ -918,7 +918,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(s5h1420_attach); +EXPORT_SYMBOL_GPL(s5h1420_attach); static const struct dvb_frontend_ops s5h1420_ops = { .delsys = { SYS_DVBS }, diff --git a/drivers/media/dvb-frontends/s5h1432.c b/drivers/media/dvb-frontends/s5h1432.c index 956e8ee4b388..ff5d3bdf3bc6 100644 --- a/drivers/media/dvb-frontends/s5h1432.c +++ b/drivers/media/dvb-frontends/s5h1432.c @@ -355,7 +355,7 @@ struct dvb_frontend *s5h1432_attach(const struct s5h1432_config *config, return &state->frontend; } -EXPORT_SYMBOL(s5h1432_attach); +EXPORT_SYMBOL_GPL(s5h1432_attach); static const struct dvb_frontend_ops s5h1432_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/dvb-frontends/s921.c b/drivers/media/dvb-frontends/s921.c index f118d8e64103..7e461ac159fc 100644 --- a/drivers/media/dvb-frontends/s921.c +++ b/drivers/media/dvb-frontends/s921.c @@ -495,7 +495,7 @@ struct dvb_frontend *s921_attach(const struct s921_config *config, return &state->frontend; } -EXPORT_SYMBOL(s921_attach); +EXPORT_SYMBOL_GPL(s921_attach); static const struct dvb_frontend_ops s921_ops = { .delsys = { SYS_ISDBT }, diff --git a/drivers/media/dvb-frontends/si21xx.c b/drivers/media/dvb-frontends/si21xx.c index 2d29d2c4d434..210ccd356e2b 100644 --- a/drivers/media/dvb-frontends/si21xx.c +++ b/drivers/media/dvb-frontends/si21xx.c @@ -937,7 +937,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(si21xx_attach); +EXPORT_SYMBOL_GPL(si21xx_attach); module_param(debug, int, 0644); MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); diff --git a/drivers/media/dvb-frontends/sp887x.c b/drivers/media/dvb-frontends/sp887x.c index 146e7f2dd3c5..f59c0f96416b 100644 --- a/drivers/media/dvb-frontends/sp887x.c +++ b/drivers/media/dvb-frontends/sp887x.c @@ -624,4 +624,4 @@ MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); MODULE_DESCRIPTION("Spase sp887x DVB-T demodulator driver"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(sp887x_attach); +EXPORT_SYMBOL_GPL(sp887x_attach); diff --git a/drivers/media/dvb-frontends/stb0899_drv.c b/drivers/media/dvb-frontends/stb0899_drv.c index 4ee6c1e1e9f7..2f4d8fb400cd 100644 --- a/drivers/media/dvb-frontends/stb0899_drv.c +++ b/drivers/media/dvb-frontends/stb0899_drv.c @@ -1638,7 +1638,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(stb0899_attach); +EXPORT_SYMBOL_GPL(stb0899_attach); MODULE_PARM_DESC(verbose, "Set Verbosity level"); MODULE_AUTHOR("Manu Abraham"); MODULE_DESCRIPTION("STB0899 Multi-Std frontend"); diff --git a/drivers/media/dvb-frontends/stb6000.c b/drivers/media/dvb-frontends/stb6000.c index 8c9800d577e0..d74e34677b92 100644 --- a/drivers/media/dvb-frontends/stb6000.c +++ b/drivers/media/dvb-frontends/stb6000.c @@ -232,7 +232,7 @@ struct dvb_frontend *stb6000_attach(struct dvb_frontend *fe, int addr, return fe; } -EXPORT_SYMBOL(stb6000_attach); +EXPORT_SYMBOL_GPL(stb6000_attach); module_param(debug, int, 0644); MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); diff --git a/drivers/media/dvb-frontends/stb6100.c b/drivers/media/dvb-frontends/stb6100.c index 698866c4f15a..c5818a15a0d7 100644 --- a/drivers/media/dvb-frontends/stb6100.c +++ b/drivers/media/dvb-frontends/stb6100.c @@ -557,7 +557,7 @@ static void stb6100_release(struct dvb_frontend *fe) kfree(state); } -EXPORT_SYMBOL(stb6100_attach); +EXPORT_SYMBOL_GPL(stb6100_attach); MODULE_PARM_DESC(verbose, "Set Verbosity level"); MODULE_AUTHOR("Manu Abraham"); diff --git a/drivers/media/dvb-frontends/stv0288.c b/drivers/media/dvb-frontends/stv0288.c index 3ae1f3a2f142..a5581bd60f9e 100644 --- a/drivers/media/dvb-frontends/stv0288.c +++ b/drivers/media/dvb-frontends/stv0288.c @@ -590,7 +590,7 @@ error: return NULL; } -EXPORT_SYMBOL(stv0288_attach); +EXPORT_SYMBOL_GPL(stv0288_attach); module_param(debug_legacy_dish_switch, int, 0444); MODULE_PARM_DESC(debug_legacy_dish_switch, diff --git a/drivers/media/dvb-frontends/stv0297.c b/drivers/media/dvb-frontends/stv0297.c index 6d5962d5697a..9d4dbd99a5a7 100644 --- a/drivers/media/dvb-frontends/stv0297.c +++ b/drivers/media/dvb-frontends/stv0297.c @@ -710,4 +710,4 @@ MODULE_DESCRIPTION("ST STV0297 DVB-C Demodulator driver"); MODULE_AUTHOR("Dennis Noermann and Andrew de Quincey"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(stv0297_attach); +EXPORT_SYMBOL_GPL(stv0297_attach); diff --git a/drivers/media/dvb-frontends/stv0299.c b/drivers/media/dvb-frontends/stv0299.c index b5263a0ee5aa..da7ff2c2e8e5 100644 --- a/drivers/media/dvb-frontends/stv0299.c +++ b/drivers/media/dvb-frontends/stv0299.c @@ -752,4 +752,4 @@ MODULE_DESCRIPTION("ST STV0299 DVB Demodulator driver"); MODULE_AUTHOR("Ralph Metzler, Holger Waechtler, Peter Schildmann, Felix Domke, Andreas Oberritter, Andrew de Quincey, Kenneth Aafly"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(stv0299_attach); +EXPORT_SYMBOL_GPL(stv0299_attach); diff --git a/drivers/media/dvb-frontends/stv0367.c b/drivers/media/dvb-frontends/stv0367.c index a93f40617469..48326434488c 100644 --- a/drivers/media/dvb-frontends/stv0367.c +++ b/drivers/media/dvb-frontends/stv0367.c @@ -1750,7 +1750,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(stv0367ter_attach); +EXPORT_SYMBOL_GPL(stv0367ter_attach); static int stv0367cab_gate_ctrl(struct dvb_frontend *fe, int enable) { @@ -2919,7 +2919,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(stv0367cab_attach); +EXPORT_SYMBOL_GPL(stv0367cab_attach); /* * Functions for operation on Digital Devices hardware @@ -3340,7 +3340,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(stv0367ddb_attach); +EXPORT_SYMBOL_GPL(stv0367ddb_attach); MODULE_PARM_DESC(debug, "Set debug"); MODULE_PARM_DESC(i2c_debug, "Set i2c debug"); diff --git a/drivers/media/dvb-frontends/stv0900_core.c b/drivers/media/dvb-frontends/stv0900_core.c index 212312d20ff6..e7b9b9b11d7d 100644 --- a/drivers/media/dvb-frontends/stv0900_core.c +++ b/drivers/media/dvb-frontends/stv0900_core.c @@ -1957,7 +1957,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(stv0900_attach); +EXPORT_SYMBOL_GPL(stv0900_attach); MODULE_PARM_DESC(debug, "Set debug"); diff --git a/drivers/media/dvb-frontends/stv090x.c b/drivers/media/dvb-frontends/stv090x.c index a07dc5fdeb3d..cc45139057ba 100644 --- a/drivers/media/dvb-frontends/stv090x.c +++ b/drivers/media/dvb-frontends/stv090x.c @@ -5071,7 +5071,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(stv090x_attach); +EXPORT_SYMBOL_GPL(stv090x_attach); static const struct i2c_device_id stv090x_id_table[] = { {"stv090x", 0}, diff --git a/drivers/media/dvb-frontends/stv6110.c b/drivers/media/dvb-frontends/stv6110.c index 963f6a896102..1cf9c095dbff 100644 --- a/drivers/media/dvb-frontends/stv6110.c +++ b/drivers/media/dvb-frontends/stv6110.c @@ -427,7 +427,7 @@ struct dvb_frontend *stv6110_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(stv6110_attach); +EXPORT_SYMBOL_GPL(stv6110_attach); module_param(debug, int, 0644); MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); diff --git a/drivers/media/dvb-frontends/stv6110x.c b/drivers/media/dvb-frontends/stv6110x.c index 11653f846c12..c678f47d2449 100644 --- a/drivers/media/dvb-frontends/stv6110x.c +++ b/drivers/media/dvb-frontends/stv6110x.c @@ -467,7 +467,7 @@ const struct stv6110x_devctl *stv6110x_attach(struct dvb_frontend *fe, dev_info(&stv6110x->i2c->dev, "Attaching STV6110x\n"); return stv6110x->devctl; } -EXPORT_SYMBOL(stv6110x_attach); +EXPORT_SYMBOL_GPL(stv6110x_attach); static const struct i2c_device_id stv6110x_id_table[] = { {"stv6110x", 0}, diff --git a/drivers/media/dvb-frontends/tda10021.c b/drivers/media/dvb-frontends/tda10021.c index faa6e54b3372..462e12ab6bd1 100644 --- a/drivers/media/dvb-frontends/tda10021.c +++ b/drivers/media/dvb-frontends/tda10021.c @@ -523,4 +523,4 @@ MODULE_DESCRIPTION("Philips TDA10021 DVB-C demodulator driver"); MODULE_AUTHOR("Ralph Metzler, Holger Waechtler, Markus Schulz"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(tda10021_attach); +EXPORT_SYMBOL_GPL(tda10021_attach); diff --git a/drivers/media/dvb-frontends/tda10023.c b/drivers/media/dvb-frontends/tda10023.c index 8f32edf6b700..4c2541ecd743 100644 --- a/drivers/media/dvb-frontends/tda10023.c +++ b/drivers/media/dvb-frontends/tda10023.c @@ -594,4 +594,4 @@ MODULE_DESCRIPTION("Philips TDA10023 DVB-C demodulator driver"); MODULE_AUTHOR("Georg Acher, Hartmut Birr"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(tda10023_attach); +EXPORT_SYMBOL_GPL(tda10023_attach); diff --git a/drivers/media/dvb-frontends/tda10048.c b/drivers/media/dvb-frontends/tda10048.c index 3cb4e5270e4f..5d5e4e9e4422 100644 --- a/drivers/media/dvb-frontends/tda10048.c +++ b/drivers/media/dvb-frontends/tda10048.c @@ -1138,7 +1138,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(tda10048_attach); +EXPORT_SYMBOL_GPL(tda10048_attach); static const struct dvb_frontend_ops tda10048_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/dvb-frontends/tda1004x.c b/drivers/media/dvb-frontends/tda1004x.c index 83a798ca9b00..6f306db6c615 100644 --- a/drivers/media/dvb-frontends/tda1004x.c +++ b/drivers/media/dvb-frontends/tda1004x.c @@ -1378,5 +1378,5 @@ MODULE_DESCRIPTION("Philips TDA10045H & TDA10046H DVB-T Demodulator"); MODULE_AUTHOR("Andrew de Quincey & Robert Schlabbach"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(tda10045_attach); -EXPORT_SYMBOL(tda10046_attach); +EXPORT_SYMBOL_GPL(tda10045_attach); +EXPORT_SYMBOL_GPL(tda10046_attach); diff --git a/drivers/media/dvb-frontends/tda10086.c b/drivers/media/dvb-frontends/tda10086.c index cdcf97664bba..b449514ae585 100644 --- a/drivers/media/dvb-frontends/tda10086.c +++ b/drivers/media/dvb-frontends/tda10086.c @@ -764,4 +764,4 @@ MODULE_DESCRIPTION("Philips TDA10086 DVB-S Demodulator"); MODULE_AUTHOR("Andrew de Quincey"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(tda10086_attach); +EXPORT_SYMBOL_GPL(tda10086_attach); diff --git a/drivers/media/dvb-frontends/tda665x.c b/drivers/media/dvb-frontends/tda665x.c index 13e8969da7f8..346be5011fb7 100644 --- a/drivers/media/dvb-frontends/tda665x.c +++ b/drivers/media/dvb-frontends/tda665x.c @@ -227,7 +227,7 @@ struct dvb_frontend *tda665x_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(tda665x_attach); +EXPORT_SYMBOL_GPL(tda665x_attach); MODULE_DESCRIPTION("TDA665x driver"); MODULE_AUTHOR("Manu Abraham"); diff --git a/drivers/media/dvb-frontends/tda8083.c b/drivers/media/dvb-frontends/tda8083.c index e3e1c3db2c85..44f53624557b 100644 --- a/drivers/media/dvb-frontends/tda8083.c +++ b/drivers/media/dvb-frontends/tda8083.c @@ -481,4 +481,4 @@ MODULE_DESCRIPTION("Philips TDA8083 DVB-S Demodulator"); MODULE_AUTHOR("Ralph Metzler, Holger Waechtler"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(tda8083_attach); +EXPORT_SYMBOL_GPL(tda8083_attach); diff --git a/drivers/media/dvb-frontends/tda8261.c b/drivers/media/dvb-frontends/tda8261.c index 0d576d41c67d..8b06f92745dc 100644 --- a/drivers/media/dvb-frontends/tda8261.c +++ b/drivers/media/dvb-frontends/tda8261.c @@ -188,7 +188,7 @@ exit: return NULL; } -EXPORT_SYMBOL(tda8261_attach); +EXPORT_SYMBOL_GPL(tda8261_attach); MODULE_AUTHOR("Manu Abraham"); MODULE_DESCRIPTION("TDA8261 8PSK/QPSK Tuner"); diff --git a/drivers/media/dvb-frontends/tda826x.c b/drivers/media/dvb-frontends/tda826x.c index f9703a1dd758..eafcf5f7da3d 100644 --- a/drivers/media/dvb-frontends/tda826x.c +++ b/drivers/media/dvb-frontends/tda826x.c @@ -164,7 +164,7 @@ struct dvb_frontend *tda826x_attach(struct dvb_frontend *fe, int addr, struct i2 return fe; } -EXPORT_SYMBOL(tda826x_attach); +EXPORT_SYMBOL_GPL(tda826x_attach); module_param(debug, int, 0644); MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); diff --git a/drivers/media/dvb-frontends/ts2020.c b/drivers/media/dvb-frontends/ts2020.c index f5b60f827697..a5ebce57f35e 100644 --- a/drivers/media/dvb-frontends/ts2020.c +++ b/drivers/media/dvb-frontends/ts2020.c @@ -525,7 +525,7 @@ struct dvb_frontend *ts2020_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(ts2020_attach); +EXPORT_SYMBOL_GPL(ts2020_attach); /* * We implement own regmap locking due to legacy DVB attach which uses frontend diff --git a/drivers/media/dvb-frontends/tua6100.c b/drivers/media/dvb-frontends/tua6100.c index 2483f614d0e7..41dd9b6d3190 100644 --- a/drivers/media/dvb-frontends/tua6100.c +++ b/drivers/media/dvb-frontends/tua6100.c @@ -186,7 +186,7 @@ struct dvb_frontend *tua6100_attach(struct dvb_frontend *fe, int addr, struct i2 fe->tuner_priv = priv; return fe; } -EXPORT_SYMBOL(tua6100_attach); +EXPORT_SYMBOL_GPL(tua6100_attach); MODULE_DESCRIPTION("DVB tua6100 driver"); MODULE_AUTHOR("Andrew de Quincey"); diff --git a/drivers/media/dvb-frontends/ves1820.c b/drivers/media/dvb-frontends/ves1820.c index 9df14d0be1c1..ee5620e731e9 100644 --- a/drivers/media/dvb-frontends/ves1820.c +++ b/drivers/media/dvb-frontends/ves1820.c @@ -434,4 +434,4 @@ MODULE_DESCRIPTION("VLSI VES1820 DVB-C Demodulator driver"); MODULE_AUTHOR("Ralph Metzler, Holger Waechtler"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(ves1820_attach); +EXPORT_SYMBOL_GPL(ves1820_attach); diff --git a/drivers/media/dvb-frontends/ves1x93.c b/drivers/media/dvb-frontends/ves1x93.c index b74727286302..c60e21d26b88 100644 --- a/drivers/media/dvb-frontends/ves1x93.c +++ b/drivers/media/dvb-frontends/ves1x93.c @@ -540,4 +540,4 @@ MODULE_DESCRIPTION("VLSI VES1x93 DVB-S Demodulator driver"); MODULE_AUTHOR("Ralph Metzler"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(ves1x93_attach); +EXPORT_SYMBOL_GPL(ves1x93_attach); diff --git a/drivers/media/dvb-frontends/zl10036.c b/drivers/media/dvb-frontends/zl10036.c index d392c7cce2ce..7ba575e9c55f 100644 --- a/drivers/media/dvb-frontends/zl10036.c +++ b/drivers/media/dvb-frontends/zl10036.c @@ -496,7 +496,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(zl10036_attach); +EXPORT_SYMBOL_GPL(zl10036_attach); module_param_named(debug, zl10036_debug, int, 0644); MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); diff --git a/drivers/media/dvb-frontends/zl10039.c b/drivers/media/dvb-frontends/zl10039.c index 1335bf78d5b7..a3e4d219400c 100644 --- a/drivers/media/dvb-frontends/zl10039.c +++ b/drivers/media/dvb-frontends/zl10039.c @@ -295,7 +295,7 @@ error: kfree(state); return NULL; } -EXPORT_SYMBOL(zl10039_attach); +EXPORT_SYMBOL_GPL(zl10039_attach); module_param(debug, int, 0644); MODULE_PARM_DESC(debug, "Turn on/off frontend debugging (default:off)."); diff --git a/drivers/media/dvb-frontends/zl10353.c b/drivers/media/dvb-frontends/zl10353.c index 2a2cf20a73d6..8849d05475c2 100644 --- a/drivers/media/dvb-frontends/zl10353.c +++ b/drivers/media/dvb-frontends/zl10353.c @@ -665,4 +665,4 @@ MODULE_DESCRIPTION("Zarlink ZL10353 DVB-T demodulator driver"); MODULE_AUTHOR("Chris Pascoe"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(zl10353_attach); +EXPORT_SYMBOL_GPL(zl10353_attach); diff --git a/drivers/media/pci/bt8xx/dst.c b/drivers/media/pci/bt8xx/dst.c index 3e52a51982d7..110651e47831 100644 --- a/drivers/media/pci/bt8xx/dst.c +++ b/drivers/media/pci/bt8xx/dst.c @@ -1722,7 +1722,7 @@ struct dst_state *dst_attach(struct dst_state *state, struct dvb_adapter *dvb_ad return state; /* Manu (DST is a card not a frontend) */ } -EXPORT_SYMBOL(dst_attach); +EXPORT_SYMBOL_GPL(dst_attach); static const struct dvb_frontend_ops dst_dvbt_ops = { .delsys = { SYS_DVBT }, diff --git a/drivers/media/pci/bt8xx/dst_ca.c b/drivers/media/pci/bt8xx/dst_ca.c index d234a0f404d6..a9cc6e7a57f9 100644 --- a/drivers/media/pci/bt8xx/dst_ca.c +++ b/drivers/media/pci/bt8xx/dst_ca.c @@ -668,7 +668,7 @@ struct dvb_device *dst_ca_attach(struct dst_state *dst, struct dvb_adapter *dvb_ return NULL; } -EXPORT_SYMBOL(dst_ca_attach); +EXPORT_SYMBOL_GPL(dst_ca_attach); MODULE_DESCRIPTION("DST DVB-S/T/C Combo CA driver"); MODULE_AUTHOR("Manu Abraham"); diff --git a/drivers/media/pci/ddbridge/ddbridge-dummy-fe.c b/drivers/media/pci/ddbridge/ddbridge-dummy-fe.c index 6868a0c4fc82..520ebd16b0c4 100644 --- a/drivers/media/pci/ddbridge/ddbridge-dummy-fe.c +++ b/drivers/media/pci/ddbridge/ddbridge-dummy-fe.c @@ -112,7 +112,7 @@ struct dvb_frontend *ddbridge_dummy_fe_qam_attach(void) state->frontend.demodulator_priv = state; return &state->frontend; } -EXPORT_SYMBOL(ddbridge_dummy_fe_qam_attach); +EXPORT_SYMBOL_GPL(ddbridge_dummy_fe_qam_attach); static const struct dvb_frontend_ops ddbridge_dummy_fe_qam_ops = { .delsys = { SYS_DVBC_ANNEX_A }, diff --git a/drivers/media/tuners/fc0011.c b/drivers/media/tuners/fc0011.c index eaa3bbc903d7..3d3b54be2955 100644 --- a/drivers/media/tuners/fc0011.c +++ b/drivers/media/tuners/fc0011.c @@ -499,7 +499,7 @@ struct dvb_frontend *fc0011_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(fc0011_attach); +EXPORT_SYMBOL_GPL(fc0011_attach); MODULE_DESCRIPTION("Fitipower FC0011 silicon tuner driver"); MODULE_AUTHOR("Michael Buesch "); diff --git a/drivers/media/tuners/fc0012.c b/drivers/media/tuners/fc0012.c index 4429d5e8c579..81e65acbdb17 100644 --- a/drivers/media/tuners/fc0012.c +++ b/drivers/media/tuners/fc0012.c @@ -495,7 +495,7 @@ err: return fe; } -EXPORT_SYMBOL(fc0012_attach); +EXPORT_SYMBOL_GPL(fc0012_attach); MODULE_DESCRIPTION("Fitipower FC0012 silicon tuner driver"); MODULE_AUTHOR("Hans-Frieder Vogt "); diff --git a/drivers/media/tuners/fc0013.c b/drivers/media/tuners/fc0013.c index 29dd9b55ff33..1006a2798eef 100644 --- a/drivers/media/tuners/fc0013.c +++ b/drivers/media/tuners/fc0013.c @@ -608,7 +608,7 @@ struct dvb_frontend *fc0013_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(fc0013_attach); +EXPORT_SYMBOL_GPL(fc0013_attach); MODULE_DESCRIPTION("Fitipower FC0013 silicon tuner driver"); MODULE_AUTHOR("Hans-Frieder Vogt "); diff --git a/drivers/media/tuners/max2165.c b/drivers/media/tuners/max2165.c index 1c746bed51fe..1575ab94e1c8 100644 --- a/drivers/media/tuners/max2165.c +++ b/drivers/media/tuners/max2165.c @@ -410,7 +410,7 @@ struct dvb_frontend *max2165_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(max2165_attach); +EXPORT_SYMBOL_GPL(max2165_attach); MODULE_AUTHOR("David T. L. Wong "); MODULE_DESCRIPTION("Maxim MAX2165 silicon tuner driver"); diff --git a/drivers/media/tuners/mc44s803.c b/drivers/media/tuners/mc44s803.c index 0c9161516abd..ed8bdf7ebd99 100644 --- a/drivers/media/tuners/mc44s803.c +++ b/drivers/media/tuners/mc44s803.c @@ -356,7 +356,7 @@ error: kfree(priv); return NULL; } -EXPORT_SYMBOL(mc44s803_attach); +EXPORT_SYMBOL_GPL(mc44s803_attach); MODULE_AUTHOR("Jochen Friedrich"); MODULE_DESCRIPTION("Freescale MC44S803 silicon tuner driver"); diff --git a/drivers/media/tuners/mt2060.c b/drivers/media/tuners/mt2060.c index 0278a9f0aeef..4205ed4cf467 100644 --- a/drivers/media/tuners/mt2060.c +++ b/drivers/media/tuners/mt2060.c @@ -440,7 +440,7 @@ struct dvb_frontend * mt2060_attach(struct dvb_frontend *fe, struct i2c_adapter return fe; } -EXPORT_SYMBOL(mt2060_attach); +EXPORT_SYMBOL_GPL(mt2060_attach); static int mt2060_probe(struct i2c_client *client) { diff --git a/drivers/media/tuners/mt2131.c b/drivers/media/tuners/mt2131.c index 37f50ff6c0bd..eebc06088341 100644 --- a/drivers/media/tuners/mt2131.c +++ b/drivers/media/tuners/mt2131.c @@ -274,7 +274,7 @@ struct dvb_frontend * mt2131_attach(struct dvb_frontend *fe, fe->tuner_priv = priv; return fe; } -EXPORT_SYMBOL(mt2131_attach); +EXPORT_SYMBOL_GPL(mt2131_attach); MODULE_AUTHOR("Steven Toth"); MODULE_DESCRIPTION("Microtune MT2131 silicon tuner driver"); diff --git a/drivers/media/tuners/mt2266.c b/drivers/media/tuners/mt2266.c index 6136f20fa9b7..2e92885a6bcb 100644 --- a/drivers/media/tuners/mt2266.c +++ b/drivers/media/tuners/mt2266.c @@ -336,7 +336,7 @@ struct dvb_frontend * mt2266_attach(struct dvb_frontend *fe, struct i2c_adapter mt2266_calibrate(priv); return fe; } -EXPORT_SYMBOL(mt2266_attach); +EXPORT_SYMBOL_GPL(mt2266_attach); MODULE_AUTHOR("Olivier DANET"); MODULE_DESCRIPTION("Microtune MT2266 silicon tuner driver"); diff --git a/drivers/media/tuners/mxl5005s.c b/drivers/media/tuners/mxl5005s.c index 06dfab9fb8cb..d9bfa257a005 100644 --- a/drivers/media/tuners/mxl5005s.c +++ b/drivers/media/tuners/mxl5005s.c @@ -4120,7 +4120,7 @@ struct dvb_frontend *mxl5005s_attach(struct dvb_frontend *fe, fe->tuner_priv = state; return fe; } -EXPORT_SYMBOL(mxl5005s_attach); +EXPORT_SYMBOL_GPL(mxl5005s_attach); MODULE_DESCRIPTION("MaxLinear MXL5005S silicon tuner driver"); MODULE_AUTHOR("Steven Toth"); diff --git a/drivers/media/tuners/qt1010.c b/drivers/media/tuners/qt1010.c index a7b19863f489..48fc79cd4027 100644 --- a/drivers/media/tuners/qt1010.c +++ b/drivers/media/tuners/qt1010.c @@ -441,7 +441,7 @@ struct dvb_frontend * qt1010_attach(struct dvb_frontend *fe, fe->tuner_priv = priv; return fe; } -EXPORT_SYMBOL(qt1010_attach); +EXPORT_SYMBOL_GPL(qt1010_attach); MODULE_DESCRIPTION("Quantek QT1010 silicon tuner driver"); MODULE_AUTHOR("Antti Palosaari "); diff --git a/drivers/media/tuners/tda18218.c b/drivers/media/tuners/tda18218.c index 4ed94646116f..7d8d84dcb245 100644 --- a/drivers/media/tuners/tda18218.c +++ b/drivers/media/tuners/tda18218.c @@ -336,7 +336,7 @@ struct dvb_frontend *tda18218_attach(struct dvb_frontend *fe, return fe; } -EXPORT_SYMBOL(tda18218_attach); +EXPORT_SYMBOL_GPL(tda18218_attach); MODULE_DESCRIPTION("NXP TDA18218HN silicon tuner driver"); MODULE_AUTHOR("Antti Palosaari "); diff --git a/drivers/media/tuners/xc2028.c b/drivers/media/tuners/xc2028.c index 69c2e1b99bf1..5a967edceca9 100644 --- a/drivers/media/tuners/xc2028.c +++ b/drivers/media/tuners/xc2028.c @@ -1512,7 +1512,7 @@ fail: return NULL; } -EXPORT_SYMBOL(xc2028_attach); +EXPORT_SYMBOL_GPL(xc2028_attach); MODULE_DESCRIPTION("Xceive xc2028/xc3028 tuner driver"); MODULE_AUTHOR("Michel Ludwig "); diff --git a/drivers/media/tuners/xc4000.c b/drivers/media/tuners/xc4000.c index d59b4ab77430..57ded9ff3f04 100644 --- a/drivers/media/tuners/xc4000.c +++ b/drivers/media/tuners/xc4000.c @@ -1742,7 +1742,7 @@ fail2: xc4000_release(fe); return NULL; } -EXPORT_SYMBOL(xc4000_attach); +EXPORT_SYMBOL_GPL(xc4000_attach); MODULE_AUTHOR("Steven Toth, Davide Ferri"); MODULE_DESCRIPTION("Xceive xc4000 silicon tuner driver"); diff --git a/drivers/media/tuners/xc5000.c b/drivers/media/tuners/xc5000.c index 7b7d9fe4f945..2182e5b7b606 100644 --- a/drivers/media/tuners/xc5000.c +++ b/drivers/media/tuners/xc5000.c @@ -1460,7 +1460,7 @@ fail: xc5000_release(fe); return NULL; } -EXPORT_SYMBOL(xc5000_attach); +EXPORT_SYMBOL_GPL(xc5000_attach); MODULE_AUTHOR("Steven Toth"); MODULE_DESCRIPTION("Xceive xc5000 silicon tuner driver"); diff --git a/drivers/net/dsa/microchip/ksz_common.c b/drivers/net/dsa/microchip/ksz_common.c index 6673122266b7..42db7679c360 100644 --- a/drivers/net/dsa/microchip/ksz_common.c +++ b/drivers/net/dsa/microchip/ksz_common.c @@ -2335,13 +2335,27 @@ static u32 ksz_get_phy_flags(struct dsa_switch *ds, int port) { struct ksz_device *dev = ds->priv; - if (dev->chip_id == KSZ8830_CHIP_ID) { + switch (dev->chip_id) { + case KSZ8830_CHIP_ID: /* Silicon Errata Sheet (DS80000830A): * Port 1 does not work with LinkMD Cable-Testing. * Port 1 does not respond to received PAUSE control frames. */ if (!port) return MICREL_KSZ8_P1_ERRATA; + break; + case KSZ9477_CHIP_ID: + /* KSZ9477 Errata DS80000754C + * + * Module 4: Energy Efficient Ethernet (EEE) feature select must + * be manually disabled + * The EEE feature is enabled by default, but it is not fully + * operational. It must be manually disabled through register + * controls. If not disabled, the PHY ports can auto-negotiate + * to enable EEE, and this feature can cause link drops when + * linked to another device supporting EEE. + */ + return MICREL_NO_EEE; } return 0; diff --git a/drivers/net/dsa/sja1105/sja1105.h b/drivers/net/dsa/sja1105/sja1105.h index dee35ba924ad..0617d5ccd3ff 100644 --- a/drivers/net/dsa/sja1105/sja1105.h +++ b/drivers/net/dsa/sja1105/sja1105.h @@ -132,6 +132,8 @@ struct sja1105_info { int max_frame_mem; int num_ports; bool multiple_cascade_ports; + /* Every {port, TXQ} has its own CBS shaper */ + bool fixed_cbs_mapping; enum dsa_tag_protocol tag_proto; const struct sja1105_dynamic_table_ops *dyn_ops; const struct sja1105_table_ops *static_ops; diff --git a/drivers/net/dsa/sja1105/sja1105_main.c b/drivers/net/dsa/sja1105/sja1105_main.c index 331bb1c6676a..a23d980d28f5 100644 --- a/drivers/net/dsa/sja1105/sja1105_main.c +++ b/drivers/net/dsa/sja1105/sja1105_main.c @@ -2115,11 +2115,36 @@ static void sja1105_bridge_leave(struct dsa_switch *ds, int port, } #define BYTES_PER_KBIT (1000LL / 8) +/* Port 0 (the uC port) does not have CBS shapers */ +#define SJA1110_FIXED_CBS(port, prio) ((((port) - 1) * SJA1105_NUM_TC) + (prio)) + +static int sja1105_find_cbs_shaper(struct sja1105_private *priv, + int port, int prio) +{ + int i; + + if (priv->info->fixed_cbs_mapping) { + i = SJA1110_FIXED_CBS(port, prio); + if (i >= 0 && i < priv->info->num_cbs_shapers) + return i; + + return -1; + } + + for (i = 0; i < priv->info->num_cbs_shapers; i++) + if (priv->cbs[i].port == port && priv->cbs[i].prio == prio) + return i; + + return -1; +} static int sja1105_find_unused_cbs_shaper(struct sja1105_private *priv) { int i; + if (priv->info->fixed_cbs_mapping) + return -1; + for (i = 0; i < priv->info->num_cbs_shapers; i++) if (!priv->cbs[i].idle_slope && !priv->cbs[i].send_slope) return i; @@ -2150,14 +2175,20 @@ static int sja1105_setup_tc_cbs(struct dsa_switch *ds, int port, { struct sja1105_private *priv = ds->priv; struct sja1105_cbs_entry *cbs; + s64 port_transmit_rate_kbps; int index; if (!offload->enable) return sja1105_delete_cbs_shaper(priv, port, offload->queue); - index = sja1105_find_unused_cbs_shaper(priv); - if (index < 0) - return -ENOSPC; + /* The user may be replacing an existing shaper */ + index = sja1105_find_cbs_shaper(priv, port, offload->queue); + if (index < 0) { + /* That isn't the case - see if we can allocate a new one */ + index = sja1105_find_unused_cbs_shaper(priv); + if (index < 0) + return -ENOSPC; + } cbs = &priv->cbs[index]; cbs->port = port; @@ -2167,9 +2198,17 @@ static int sja1105_setup_tc_cbs(struct dsa_switch *ds, int port, */ cbs->credit_hi = offload->hicredit; cbs->credit_lo = abs(offload->locredit); - /* User space is in kbits/sec, hardware in bytes/sec */ - cbs->idle_slope = offload->idleslope * BYTES_PER_KBIT; - cbs->send_slope = abs(offload->sendslope * BYTES_PER_KBIT); + /* User space is in kbits/sec, while the hardware in bytes/sec times + * link speed. Since the given offload->sendslope is good only for the + * current link speed anyway, and user space is likely to reprogram it + * when that changes, don't even bother to track the port's link speed, + * but deduce the port transmit rate from idleslope - sendslope. + */ + port_transmit_rate_kbps = offload->idleslope - offload->sendslope; + cbs->idle_slope = div_s64(offload->idleslope * BYTES_PER_KBIT, + port_transmit_rate_kbps); + cbs->send_slope = div_s64(abs(offload->sendslope * BYTES_PER_KBIT), + port_transmit_rate_kbps); /* Convert the negative values from 64-bit 2's complement * to 32-bit 2's complement (for the case of 0x80000000 whose * negative is still negative). diff --git a/drivers/net/dsa/sja1105/sja1105_spi.c b/drivers/net/dsa/sja1105/sja1105_spi.c index 5ce29c8057a4..834b5c1b4db0 100644 --- a/drivers/net/dsa/sja1105/sja1105_spi.c +++ b/drivers/net/dsa/sja1105/sja1105_spi.c @@ -781,6 +781,7 @@ const struct sja1105_info sja1110a_info = { .tag_proto = DSA_TAG_PROTO_SJA1110, .can_limit_mcast_flood = true, .multiple_cascade_ports = true, + .fixed_cbs_mapping = true, .ptp_ts_bits = 32, .ptpegr_ts_bytes = 8, .max_frame_mem = SJA1110_MAX_FRAME_MEMORY, @@ -831,6 +832,7 @@ const struct sja1105_info sja1110b_info = { .tag_proto = DSA_TAG_PROTO_SJA1110, .can_limit_mcast_flood = true, .multiple_cascade_ports = true, + .fixed_cbs_mapping = true, .ptp_ts_bits = 32, .ptpegr_ts_bytes = 8, .max_frame_mem = SJA1110_MAX_FRAME_MEMORY, @@ -881,6 +883,7 @@ const struct sja1105_info sja1110c_info = { .tag_proto = DSA_TAG_PROTO_SJA1110, .can_limit_mcast_flood = true, .multiple_cascade_ports = true, + .fixed_cbs_mapping = true, .ptp_ts_bits = 32, .ptpegr_ts_bytes = 8, .max_frame_mem = SJA1110_MAX_FRAME_MEMORY, @@ -931,6 +934,7 @@ const struct sja1105_info sja1110d_info = { .tag_proto = DSA_TAG_PROTO_SJA1110, .can_limit_mcast_flood = true, .multiple_cascade_ports = true, + .fixed_cbs_mapping = true, .ptp_ts_bits = 32, .ptpegr_ts_bytes = 8, .max_frame_mem = SJA1110_MAX_FRAME_MEMORY, diff --git a/drivers/net/ethernet/freescale/enetc/enetc_pf.c b/drivers/net/ethernet/freescale/enetc/enetc_pf.c index e0a4cb7e3f50..c153dc083aff 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_pf.c +++ b/drivers/net/ethernet/freescale/enetc/enetc_pf.c @@ -1402,7 +1402,7 @@ static void enetc_fixup_clear_rss_rfs(struct pci_dev *pdev) return; si = enetc_psi_create(pdev); - if (si) + if (!IS_ERR(si)) enetc_psi_destroy(pdev); } DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_FREESCALE, ENETC_DEV_ID_PF, diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c index ea0e38b4d9e9..f281e42a7ef9 100644 --- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c +++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c @@ -570,7 +570,10 @@ static int gve_rx_append_frags(struct napi_struct *napi, if (!skb) return -1; - skb_shinfo(rx->ctx.skb_tail)->frag_list = skb; + if (rx->ctx.skb_tail == rx->ctx.skb_head) + skb_shinfo(rx->ctx.skb_head)->frag_list = skb; + else + rx->ctx.skb_tail->next = skb; rx->ctx.skb_tail = skb; num_frags = 0; } diff --git a/drivers/net/ethernet/hisilicon/hns3/hnae3.h b/drivers/net/ethernet/hisilicon/hns3/hnae3.h index a4b43bcd2f0c..aaf1f42624a7 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hnae3.h +++ b/drivers/net/ethernet/hisilicon/hns3/hnae3.h @@ -814,6 +814,7 @@ struct hnae3_tc_info { u8 max_tc; /* Total number of TCs */ u8 num_tc; /* Total number of enabled TCs */ bool mqprio_active; + bool dcb_ets_active; }; #define HNAE3_MAX_DSCP 64 diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c b/drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c index f276b5ecb431..b8508533878b 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_debugfs.c @@ -1045,6 +1045,7 @@ hns3_dbg_dev_specs(struct hnae3_handle *h, char *buf, int len, int *pos) struct hnae3_ae_dev *ae_dev = pci_get_drvdata(h->pdev); struct hnae3_dev_specs *dev_specs = &ae_dev->dev_specs; struct hnae3_knic_private_info *kinfo = &h->kinfo; + struct net_device *dev = kinfo->netdev; *pos += scnprintf(buf + *pos, len - *pos, "dev_spec:\n"); *pos += scnprintf(buf + *pos, len - *pos, "MAC entry num: %u\n", @@ -1087,6 +1088,9 @@ hns3_dbg_dev_specs(struct hnae3_handle *h, char *buf, int len, int *pos) dev_specs->mc_mac_size); *pos += scnprintf(buf + *pos, len - *pos, "MAC statistics number: %u\n", dev_specs->mac_stats_num); + *pos += scnprintf(buf + *pos, len - *pos, + "TX timeout threshold: %d seconds\n", + dev->watchdog_timeo / HZ); } static int hns3_dbg_dev_info(struct hnae3_handle *h, char *buf, int len) @@ -1411,9 +1415,9 @@ int hns3_dbg_init(struct hnae3_handle *handle) return 0; out: - mutex_destroy(&handle->dbgfs_lock); debugfs_remove_recursive(handle->hnae3_dbgfs); handle->hnae3_dbgfs = NULL; + mutex_destroy(&handle->dbgfs_lock); return ret; } @@ -1421,6 +1425,9 @@ void hns3_dbg_uninit(struct hnae3_handle *handle) { u32 i; + debugfs_remove_recursive(handle->hnae3_dbgfs); + handle->hnae3_dbgfs = NULL; + for (i = 0; i < ARRAY_SIZE(hns3_dbg_cmd); i++) if (handle->dbgfs_buf[i]) { kvfree(handle->dbgfs_buf[i]); @@ -1428,8 +1435,6 @@ void hns3_dbg_uninit(struct hnae3_handle *handle) } mutex_destroy(&handle->dbgfs_lock); - debugfs_remove_recursive(handle->hnae3_dbgfs); - handle->hnae3_dbgfs = NULL; } void hns3_dbg_register_debugfs(const char *debugfs_dir_name) diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c index eac2d0573241..b4895c7b3efd 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c @@ -2103,8 +2103,12 @@ static void hns3_tx_doorbell(struct hns3_enet_ring *ring, int num, */ if (test_bit(HNS3_NIC_STATE_TX_PUSH_ENABLE, &priv->state) && num && !ring->pending_buf && num <= HNS3_MAX_PUSH_BD_NUM && doorbell) { + /* This smp_store_release() pairs with smp_load_aquire() in + * hns3_nic_reclaim_desc(). Ensure that the BD valid bit + * is updated. + */ + smp_store_release(&ring->last_to_use, ring->next_to_use); hns3_tx_push_bd(ring, num); - WRITE_ONCE(ring->last_to_use, ring->next_to_use); return; } @@ -2115,6 +2119,11 @@ static void hns3_tx_doorbell(struct hns3_enet_ring *ring, int num, return; } + /* This smp_store_release() pairs with smp_load_aquire() in + * hns3_nic_reclaim_desc(). Ensure that the BD valid bit is updated. + */ + smp_store_release(&ring->last_to_use, ring->next_to_use); + if (ring->tqp->mem_base) hns3_tx_mem_doorbell(ring); else @@ -2122,7 +2131,6 @@ static void hns3_tx_doorbell(struct hns3_enet_ring *ring, int num, ring->tqp->io_base + HNS3_RING_TX_RING_TAIL_REG); ring->pending_buf = 0; - WRITE_ONCE(ring->last_to_use, ring->next_to_use); } static void hns3_tsyn(struct net_device *netdev, struct sk_buff *skb, @@ -3308,8 +3316,6 @@ static void hns3_set_default_feature(struct net_device *netdev) netdev->priv_flags |= IFF_UNICAST_FLT; - netdev->gso_partial_features |= NETIF_F_GSO_GRE_CSUM; - netdev->features |= NETIF_F_HW_VLAN_CTAG_FILTER | NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX | NETIF_F_RXCSUM | NETIF_F_SG | NETIF_F_GSO | @@ -3563,9 +3569,8 @@ static void hns3_reuse_buffer(struct hns3_enet_ring *ring, int i) static bool hns3_nic_reclaim_desc(struct hns3_enet_ring *ring, int *bytes, int *pkts, int budget) { - /* pair with ring->last_to_use update in hns3_tx_doorbell(), - * smp_store_release() is not used in hns3_tx_doorbell() because - * the doorbell operation already have the needed barrier operation. + /* This smp_load_acquire() pairs with smp_store_release() in + * hns3_tx_doorbell(). */ int ltu = smp_load_acquire(&ring->last_to_use); int ntc = ring->next_to_clean; diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_ethtool.c b/drivers/net/ethernet/hisilicon/hns3/hns3_ethtool.c index 36858a72d771..682239f33082 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_ethtool.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_ethtool.c @@ -773,7 +773,9 @@ static int hns3_get_link_ksettings(struct net_device *netdev, hns3_get_ksettings(h, cmd); break; case HNAE3_MEDIA_TYPE_FIBER: - if (module_type == HNAE3_MODULE_TYPE_CR) + if (module_type == HNAE3_MODULE_TYPE_UNKNOWN) + cmd->base.port = PORT_OTHER; + else if (module_type == HNAE3_MODULE_TYPE_CR) cmd->base.port = PORT_DA; else cmd->base.port = PORT_FIBRE; diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_dcb.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_dcb.c index fad5a5ff3cda..b98301e205f7 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_dcb.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_dcb.c @@ -259,7 +259,7 @@ static int hclge_ieee_setets(struct hnae3_handle *h, struct ieee_ets *ets) int ret; if (!(hdev->dcbx_cap & DCB_CAP_DCBX_VER_IEEE) || - hdev->flag & HCLGE_FLAG_MQPRIO_ENABLE) + h->kinfo.tc_info.mqprio_active) return -EINVAL; ret = hclge_ets_validate(hdev, ets, &num_tc, &map_changed); @@ -275,10 +275,7 @@ static int hclge_ieee_setets(struct hnae3_handle *h, struct ieee_ets *ets) } hclge_tm_schd_info_update(hdev, num_tc); - if (num_tc > 1) - hdev->flag |= HCLGE_FLAG_DCB_ENABLE; - else - hdev->flag &= ~HCLGE_FLAG_DCB_ENABLE; + h->kinfo.tc_info.dcb_ets_active = num_tc > 1; ret = hclge_ieee_ets_to_tm_info(hdev, ets); if (ret) @@ -487,7 +484,7 @@ static u8 hclge_getdcbx(struct hnae3_handle *h) struct hclge_vport *vport = hclge_get_vport(h); struct hclge_dev *hdev = vport->back; - if (hdev->flag & HCLGE_FLAG_MQPRIO_ENABLE) + if (h->kinfo.tc_info.mqprio_active) return 0; return hdev->dcbx_cap; @@ -611,7 +608,8 @@ static int hclge_setup_tc(struct hnae3_handle *h, if (!test_bit(HCLGE_STATE_NIC_REGISTERED, &hdev->state)) return -EBUSY; - if (hdev->flag & HCLGE_FLAG_DCB_ENABLE) + kinfo = &vport->nic.kinfo; + if (kinfo->tc_info.dcb_ets_active) return -EINVAL; ret = hclge_mqprio_qopt_check(hdev, mqprio_qopt); @@ -625,7 +623,6 @@ static int hclge_setup_tc(struct hnae3_handle *h, if (ret) return ret; - kinfo = &vport->nic.kinfo; memcpy(&old_tc_info, &kinfo->tc_info, sizeof(old_tc_info)); hclge_sync_mqprio_qopt(&kinfo->tc_info, mqprio_qopt); kinfo->tc_info.mqprio_active = tc > 0; @@ -634,13 +631,6 @@ static int hclge_setup_tc(struct hnae3_handle *h, if (ret) goto err_out; - hdev->flag &= ~HCLGE_FLAG_DCB_ENABLE; - - if (tc > 1) - hdev->flag |= HCLGE_FLAG_MQPRIO_ENABLE; - else - hdev->flag &= ~HCLGE_FLAG_MQPRIO_ENABLE; - return hclge_notify_init_up(hdev); err_out: diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c index f01a7a9ee02c..ff3f8f424ad9 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_debugfs.c @@ -1519,7 +1519,7 @@ static int hclge_dbg_fd_tcam_read(struct hclge_dev *hdev, bool sel_x, struct hclge_desc desc[3]; int pos = 0; int ret, i; - u32 *req; + __le32 *req; hclge_cmd_setup_basic_desc(&desc[0], HCLGE_OPC_FD_TCAM_OP, true); desc[0].flag |= cpu_to_le16(HCLGE_COMM_CMD_FLAG_NEXT); @@ -1544,22 +1544,22 @@ static int hclge_dbg_fd_tcam_read(struct hclge_dev *hdev, bool sel_x, tcam_msg.loc); /* tcam_data0 ~ tcam_data1 */ - req = (u32 *)req1->tcam_data; + req = (__le32 *)req1->tcam_data; for (i = 0; i < 2; i++) pos += scnprintf(tcam_buf + pos, HCLGE_DBG_TCAM_BUF_SIZE - pos, - "%08x\n", *req++); + "%08x\n", le32_to_cpu(*req++)); /* tcam_data2 ~ tcam_data7 */ - req = (u32 *)req2->tcam_data; + req = (__le32 *)req2->tcam_data; for (i = 0; i < 6; i++) pos += scnprintf(tcam_buf + pos, HCLGE_DBG_TCAM_BUF_SIZE - pos, - "%08x\n", *req++); + "%08x\n", le32_to_cpu(*req++)); /* tcam_data8 ~ tcam_data12 */ - req = (u32 *)req3->tcam_data; + req = (__le32 *)req3->tcam_data; for (i = 0; i < 5; i++) pos += scnprintf(tcam_buf + pos, HCLGE_DBG_TCAM_BUF_SIZE - pos, - "%08x\n", *req++); + "%08x\n", le32_to_cpu(*req++)); return ret; } diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c index 0f50dba6cc47..8ca368424436 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.c @@ -11026,6 +11026,7 @@ static void hclge_get_mdix_mode(struct hnae3_handle *handle, static void hclge_info_show(struct hclge_dev *hdev) { + struct hnae3_handle *handle = &hdev->vport->nic; struct device *dev = &hdev->pdev->dev; dev_info(dev, "PF info begin:\n"); @@ -11042,9 +11043,9 @@ static void hclge_info_show(struct hclge_dev *hdev) dev_info(dev, "This is %s PF\n", hdev->flag & HCLGE_FLAG_MAIN ? "main" : "not main"); dev_info(dev, "DCB %s\n", - hdev->flag & HCLGE_FLAG_DCB_ENABLE ? "enable" : "disable"); + handle->kinfo.tc_info.dcb_ets_active ? "enable" : "disable"); dev_info(dev, "MQPRIO %s\n", - hdev->flag & HCLGE_FLAG_MQPRIO_ENABLE ? "enable" : "disable"); + handle->kinfo.tc_info.mqprio_active ? "enable" : "disable"); dev_info(dev, "Default tx spare buffer size: %u\n", hdev->tx_spare_buf_size); diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.h b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.h index ec233ec57222..7bc2049b723d 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.h +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_main.h @@ -919,8 +919,6 @@ struct hclge_dev { #define HCLGE_FLAG_MAIN BIT(0) #define HCLGE_FLAG_DCB_CAPABLE BIT(1) -#define HCLGE_FLAG_DCB_ENABLE BIT(2) -#define HCLGE_FLAG_MQPRIO_ENABLE BIT(3) u32 flag; u32 pkt_buf_size; /* Total pf buf size for tx/rx */ diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 015b78144114..a2b759531cb7 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -34,11 +34,11 @@ struct igb_adapter; /* TX/RX descriptor defines */ #define IGB_DEFAULT_TXD 256 #define IGB_DEFAULT_TX_WORK 128 -#define IGB_MIN_TXD 80 +#define IGB_MIN_TXD 64 #define IGB_MAX_TXD 4096 #define IGB_DEFAULT_RXD 256 -#define IGB_MIN_RXD 80 +#define IGB_MIN_RXD 64 #define IGB_MAX_RXD 4096 #define IGB_DEFAULT_ITR 3 /* dynamic */ diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 1ab787ed254d..13ba9c74bd84 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -3933,8 +3933,9 @@ static void igb_probe_vfs(struct igb_adapter *adapter) struct pci_dev *pdev = adapter->pdev; struct e1000_hw *hw = &adapter->hw; - /* Virtualization features not supported on i210 family. */ - if ((hw->mac.type == e1000_i210) || (hw->mac.type == e1000_i211)) + /* Virtualization features not supported on i210 and 82580 family. */ + if ((hw->mac.type == e1000_i210) || (hw->mac.type == e1000_i211) || + (hw->mac.type == e1000_82580)) return; /* Of the below we really only want the effect of getting diff --git a/drivers/net/ethernet/intel/igbvf/igbvf.h b/drivers/net/ethernet/intel/igbvf/igbvf.h index 57d39ee00b58..7b83678ba83a 100644 --- a/drivers/net/ethernet/intel/igbvf/igbvf.h +++ b/drivers/net/ethernet/intel/igbvf/igbvf.h @@ -39,11 +39,11 @@ enum latency_range { /* Tx/Rx descriptor defines */ #define IGBVF_DEFAULT_TXD 256 #define IGBVF_MAX_TXD 4096 -#define IGBVF_MIN_TXD 80 +#define IGBVF_MIN_TXD 64 #define IGBVF_DEFAULT_RXD 256 #define IGBVF_MAX_RXD 4096 -#define IGBVF_MIN_RXD 80 +#define IGBVF_MIN_RXD 64 #define IGBVF_MIN_ITR_USECS 10 /* 100000 irq/sec */ #define IGBVF_MAX_ITR_USECS 10000 /* 100 irq/sec */ diff --git a/drivers/net/ethernet/intel/igc/igc.h b/drivers/net/ethernet/intel/igc/igc.h index 8ebe6999a528..f48f82d5e274 100644 --- a/drivers/net/ethernet/intel/igc/igc.h +++ b/drivers/net/ethernet/intel/igc/igc.h @@ -379,11 +379,11 @@ static inline u32 igc_rss_type(const union igc_adv_rx_desc *rx_desc) /* TX/RX descriptor defines */ #define IGC_DEFAULT_TXD 256 #define IGC_DEFAULT_TX_WORK 128 -#define IGC_MIN_TXD 80 +#define IGC_MIN_TXD 64 #define IGC_MAX_TXD 4096 #define IGC_DEFAULT_RXD 256 -#define IGC_MIN_RXD 80 +#define IGC_MIN_RXD 64 #define IGC_MAX_RXD 4096 /* Supported Rx Buffer Sizes */ diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c index c2f68678e947..23c2f2ed2fb8 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c @@ -846,6 +846,21 @@ static int nix_aq_enqueue_wait(struct rvu *rvu, struct rvu_block *block, return 0; } +static void nix_get_aq_req_smq(struct rvu *rvu, struct nix_aq_enq_req *req, + u16 *smq, u16 *smq_mask) +{ + struct nix_cn10k_aq_enq_req *aq_req; + + if (!is_rvu_otx2(rvu)) { + aq_req = (struct nix_cn10k_aq_enq_req *)req; + *smq = aq_req->sq.smq; + *smq_mask = aq_req->sq_mask.smq; + } else { + *smq = req->sq.smq; + *smq_mask = req->sq_mask.smq; + } +} + static int rvu_nix_blk_aq_enq_inst(struct rvu *rvu, struct nix_hw *nix_hw, struct nix_aq_enq_req *req, struct nix_aq_enq_rsp *rsp) @@ -857,6 +872,7 @@ static int rvu_nix_blk_aq_enq_inst(struct rvu *rvu, struct nix_hw *nix_hw, struct rvu_block *block; struct admin_queue *aq; struct rvu_pfvf *pfvf; + u16 smq, smq_mask; void *ctx, *mask; bool ena; u64 cfg; @@ -928,13 +944,14 @@ static int rvu_nix_blk_aq_enq_inst(struct rvu *rvu, struct nix_hw *nix_hw, if (rc) return rc; + nix_get_aq_req_smq(rvu, req, &smq, &smq_mask); /* Check if SQ pointed SMQ belongs to this PF/VF or not */ if (req->ctype == NIX_AQ_CTYPE_SQ && ((req->op == NIX_AQ_INSTOP_INIT && req->sq.ena) || (req->op == NIX_AQ_INSTOP_WRITE && - req->sq_mask.ena && req->sq_mask.smq && req->sq.ena))) { + req->sq_mask.ena && req->sq.ena && smq_mask))) { if (!is_valid_txschq(rvu, blkaddr, NIX_TXSCH_LVL_SMQ, - pcifunc, req->sq.smq)) + pcifunc, smq)) return NIX_AF_ERR_AQ_ENQUEUE; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/ct.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/ct.c index 92d3952dfa8b..feeb41693c17 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/ct.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/ct.c @@ -17,8 +17,10 @@ tc_act_parse_ct(struct mlx5e_tc_act_parse_state *parse_state, if (err) return err; - if (mlx5e_is_eswitch_flow(parse_state->flow)) + if (mlx5e_is_eswitch_flow(parse_state->flow)) { attr->esw_attr->split_count = attr->esw_attr->out_count; + parse_state->if_count = 0; + } attr->flags |= MLX5_ATTR_FLAG_CT; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/mirred.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/mirred.c index 291193f7120d..f63402c48028 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/mirred.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/mirred.c @@ -294,6 +294,7 @@ parse_mirred_ovs_master(struct mlx5e_tc_act_parse_state *parse_state, if (err) return err; + parse_state->if_count = 0; esw_attr->out_count++; return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/pedit.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/pedit.c index 3b272bbf4c53..368a95fa77d3 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/pedit.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/pedit.c @@ -98,8 +98,10 @@ tc_act_parse_pedit(struct mlx5e_tc_act_parse_state *parse_state, attr->action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR; - if (ns_type == MLX5_FLOW_NAMESPACE_FDB) + if (ns_type == MLX5_FLOW_NAMESPACE_FDB) { esw_attr->split_count = esw_attr->out_count; + parse_state->if_count = 0; + } return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/redirect_ingress.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/redirect_ingress.c index ad09a8a5f36e..2d1d4a04501b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/redirect_ingress.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/redirect_ingress.c @@ -66,6 +66,7 @@ tc_act_parse_redirect_ingress(struct mlx5e_tc_act_parse_state *parse_state, if (err) return err; + parse_state->if_count = 0; esw_attr->out_count++; return 0; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/vlan.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/vlan.c index c8a3eaf189f6..a13c5e707b83 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/vlan.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/vlan.c @@ -166,6 +166,7 @@ tc_act_parse_vlan(struct mlx5e_tc_act_parse_state *parse_state, return err; esw_attr->split_count = esw_attr->out_count; + parse_state->if_count = 0; return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/vlan_mangle.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/vlan_mangle.c index 310b99230760..f17575b09788 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/vlan_mangle.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc/act/vlan_mangle.c @@ -65,8 +65,10 @@ tc_act_parse_vlan_mangle(struct mlx5e_tc_act_parse_state *parse_state, if (err) return err; - if (ns_type == MLX5_FLOW_NAMESPACE_FDB) + if (ns_type == MLX5_FLOW_NAMESPACE_FDB) { attr->esw_attr->split_count = attr->esw_attr->out_count; + parse_state->if_count = 0; + } return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c index 318083690fcd..c24828b688ac 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c @@ -3936,6 +3936,7 @@ parse_tc_actions(struct mlx5e_tc_act_parse_state *parse_state, } i_split = i + 1; + parse_state->if_count = 0; list_add(&attr->list, &flow->attrs); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c index 6cd7d6497e10..d4cde6555063 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c @@ -1276,12 +1276,19 @@ int mlx5_eswitch_enable_pf_vf_vports(struct mlx5_eswitch *esw, enum mlx5_eswitch_vport_event enabled_events) { + bool pf_needed; int ret; + pf_needed = mlx5_core_is_ecpf_esw_manager(esw->dev) || + esw->mode == MLX5_ESWITCH_LEGACY; + /* Enable PF vport */ - ret = mlx5_eswitch_load_pf_vf_vport(esw, MLX5_VPORT_PF, enabled_events); - if (ret) - return ret; + if (pf_needed) { + ret = mlx5_eswitch_load_pf_vf_vport(esw, MLX5_VPORT_PF, + enabled_events); + if (ret) + return ret; + } /* Enable external host PF HCA */ ret = host_pf_enable_hca(esw->dev); @@ -1317,7 +1324,8 @@ ec_vf_err: ecpf_err: host_pf_disable_hca(esw->dev); pf_hca_err: - mlx5_eswitch_unload_pf_vf_vport(esw, MLX5_VPORT_PF); + if (pf_needed) + mlx5_eswitch_unload_pf_vf_vport(esw, MLX5_VPORT_PF); return ret; } @@ -1335,7 +1343,10 @@ void mlx5_eswitch_disable_pf_vf_vports(struct mlx5_eswitch *esw) } host_pf_disable_hca(esw->dev); - mlx5_eswitch_unload_pf_vf_vport(esw, MLX5_VPORT_PF); + + if (mlx5_core_is_ecpf_esw_manager(esw->dev) || + esw->mode == MLX5_ESWITCH_LEGACY) + mlx5_eswitch_unload_pf_vf_vport(esw, MLX5_VPORT_PF); } static void mlx5_eswitch_get_devlink_param(struct mlx5_eswitch *esw) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c index 752fb0dfb111..b296ac52a439 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c @@ -3216,26 +3216,47 @@ esw_vport_destroy_offloads_acl_tables(struct mlx5_eswitch *esw, esw_acl_ingress_ofld_cleanup(esw, vport); } -static int esw_create_uplink_offloads_acl_tables(struct mlx5_eswitch *esw) +static int esw_create_offloads_acl_tables(struct mlx5_eswitch *esw) { - struct mlx5_vport *vport; + struct mlx5_vport *uplink, *manager; + int ret; - vport = mlx5_eswitch_get_vport(esw, MLX5_VPORT_UPLINK); - if (IS_ERR(vport)) - return PTR_ERR(vport); + uplink = mlx5_eswitch_get_vport(esw, MLX5_VPORT_UPLINK); + if (IS_ERR(uplink)) + return PTR_ERR(uplink); - return esw_vport_create_offloads_acl_tables(esw, vport); + ret = esw_vport_create_offloads_acl_tables(esw, uplink); + if (ret) + return ret; + + manager = mlx5_eswitch_get_vport(esw, esw->manager_vport); + if (IS_ERR(manager)) { + ret = PTR_ERR(manager); + goto err_manager; + } + + ret = esw_vport_create_offloads_acl_tables(esw, manager); + if (ret) + goto err_manager; + + return 0; + +err_manager: + esw_vport_destroy_offloads_acl_tables(esw, uplink); + return ret; } -static void esw_destroy_uplink_offloads_acl_tables(struct mlx5_eswitch *esw) +static void esw_destroy_offloads_acl_tables(struct mlx5_eswitch *esw) { struct mlx5_vport *vport; - vport = mlx5_eswitch_get_vport(esw, MLX5_VPORT_UPLINK); - if (IS_ERR(vport)) - return; + vport = mlx5_eswitch_get_vport(esw, esw->manager_vport); + if (!IS_ERR(vport)) + esw_vport_destroy_offloads_acl_tables(esw, vport); - esw_vport_destroy_offloads_acl_tables(esw, vport); + vport = mlx5_eswitch_get_vport(esw, MLX5_VPORT_UPLINK); + if (!IS_ERR(vport)) + esw_vport_destroy_offloads_acl_tables(esw, vport); } int mlx5_eswitch_reload_reps(struct mlx5_eswitch *esw) @@ -3280,7 +3301,7 @@ static int esw_offloads_steering_init(struct mlx5_eswitch *esw) } esw->fdb_table.offloads.indir = indir; - err = esw_create_uplink_offloads_acl_tables(esw); + err = esw_create_offloads_acl_tables(esw); if (err) goto create_acl_err; @@ -3321,7 +3342,7 @@ create_fdb_err: create_restore_err: esw_destroy_offloads_table(esw); create_offloads_err: - esw_destroy_uplink_offloads_acl_tables(esw); + esw_destroy_offloads_acl_tables(esw); create_acl_err: mlx5_esw_indir_table_destroy(esw->fdb_table.offloads.indir); create_indir_err: @@ -3337,7 +3358,7 @@ static void esw_offloads_steering_cleanup(struct mlx5_eswitch *esw) esw_destroy_offloads_fdb_tables(esw); esw_destroy_restore_table(esw); esw_destroy_offloads_table(esw); - esw_destroy_uplink_offloads_acl_tables(esw); + esw_destroy_offloads_acl_tables(esw); mlx5_esw_indir_table_destroy(esw->fdb_table.offloads.indir); mutex_destroy(&esw->fdb_table.offloads.vports.lock); } diff --git a/drivers/net/ethernet/sfc/rx.c b/drivers/net/ethernet/sfc/rx.c index 2375cef577e4..f77a2d3ef37e 100644 --- a/drivers/net/ethernet/sfc/rx.c +++ b/drivers/net/ethernet/sfc/rx.c @@ -359,26 +359,36 @@ static bool efx_do_xdp(struct efx_nic *efx, struct efx_channel *channel, /* Handle a received packet. Second half: Touches packet payload. */ void __efx_rx_packet(struct efx_channel *channel) { + struct efx_rx_queue *rx_queue = efx_channel_get_rx_queue(channel); struct efx_nic *efx = channel->efx; struct efx_rx_buffer *rx_buf = - efx_rx_buffer(&channel->rx_queue, channel->rx_pkt_index); + efx_rx_buffer(rx_queue, channel->rx_pkt_index); u8 *eh = efx_rx_buf_va(rx_buf); /* Read length from the prefix if necessary. This already * excludes the length of the prefix itself. */ - if (rx_buf->flags & EFX_RX_PKT_PREFIX_LEN) + if (rx_buf->flags & EFX_RX_PKT_PREFIX_LEN) { rx_buf->len = le16_to_cpup((__le16 *) (eh + efx->rx_packet_len_offset)); + /* A known issue may prevent this being filled in; + * if that happens, just drop the packet. + * Must do that in the driver since passing a zero-length + * packet up to the stack may cause a crash. + */ + if (unlikely(!rx_buf->len)) { + efx_free_rx_buffers(rx_queue, rx_buf, + channel->rx_pkt_n_frags); + channel->n_rx_frm_trunc++; + goto out; + } + } /* If we're in loopback test, then pass the packet directly to the * loopback layer, and free the rx_buf here */ if (unlikely(efx->loopback_selftest)) { - struct efx_rx_queue *rx_queue; - efx_loopback_rx_packet(efx, eh, rx_buf->len); - rx_queue = efx_channel_get_rx_queue(channel); efx_free_rx_buffers(rx_queue, rx_buf, channel->rx_pkt_n_frags); goto out; diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c index 35f4b1484029..0f28795e581c 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c @@ -419,9 +419,8 @@ stmmac_probe_config_dt(struct platform_device *pdev, u8 *mac) return ERR_PTR(phy_mode); plat->phy_interface = phy_mode; - plat->mac_interface = stmmac_of_get_mac_mode(np); - if (plat->mac_interface < 0) - plat->mac_interface = plat->phy_interface; + rc = stmmac_of_get_mac_mode(np); + plat->mac_interface = rc < 0 ? plat->phy_interface : rc; /* Some wrapper drivers still rely on phy_node. Let's save it while * they are not converted to phylink. */ diff --git a/drivers/net/macsec.c b/drivers/net/macsec.c index c3f30663070f..b7e151439c48 100644 --- a/drivers/net/macsec.c +++ b/drivers/net/macsec.c @@ -1330,8 +1330,7 @@ static struct crypto_aead *macsec_alloc_tfm(char *key, int key_len, int icv_len) struct crypto_aead *tfm; int ret; - /* Pick a sync gcm(aes) cipher to ensure order is preserved. */ - tfm = crypto_alloc_aead("gcm(aes)", 0, CRYPTO_ALG_ASYNC); + tfm = crypto_alloc_aead("gcm(aes)", 0, 0); if (IS_ERR(tfm)) return tfm; diff --git a/drivers/net/phy/micrel.c b/drivers/net/phy/micrel.c index b6d7981b2d1e..927d3d54658e 100644 --- a/drivers/net/phy/micrel.c +++ b/drivers/net/phy/micrel.c @@ -1800,9 +1800,6 @@ static const struct ksz9477_errata_write ksz9477_errata_writes[] = { /* Transmit waveform amplitude can be improved (1000BASE-T, 100BASE-TX, 10BASE-Te) */ {0x1c, 0x04, 0x00d0}, - /* Energy Efficient Ethernet (EEE) feature select must be manually disabled */ - {0x07, 0x3c, 0x0000}, - /* Register settings are required to meet data sheet supply current specifications */ {0x1c, 0x13, 0x6eff}, {0x1c, 0x14, 0xe6ff}, @@ -1847,6 +1844,12 @@ static int ksz9477_config_init(struct phy_device *phydev) return err; } + /* According to KSZ9477 Errata DS80000754C (Module 4) all EEE modes + * in this switch shall be regarded as broken. + */ + if (phydev->dev_flags & MICREL_NO_EEE) + phydev->eee_broken_modes = -1; + err = genphy_restart_aneg(phydev); if (err) return err; diff --git a/drivers/net/veth.c b/drivers/net/veth.c index d43e62ebc2fc..9c6f4f83f22b 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -344,6 +344,7 @@ static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) { struct veth_priv *rcv_priv, *priv = netdev_priv(dev); struct veth_rq *rq = NULL; + int ret = NETDEV_TX_OK; struct net_device *rcv; int length = skb->len; bool use_napi = false; @@ -378,11 +379,12 @@ static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) } else { drop: atomic64_inc(&priv->dropped); + ret = NET_XMIT_DROP; } rcu_read_unlock(); - return NETDEV_TX_OK; + return ret; } static u64 veth_stats_tx(struct net_device *dev, u64 *packets, u64 *bytes) diff --git a/drivers/nfc/nxp-nci/i2c.c b/drivers/nfc/nxp-nci/i2c.c index dca25a0c2f33..3ae4b41c59ac 100644 --- a/drivers/nfc/nxp-nci/i2c.c +++ b/drivers/nfc/nxp-nci/i2c.c @@ -336,6 +336,7 @@ MODULE_DEVICE_TABLE(of, of_nxp_nci_i2c_match); #ifdef CONFIG_ACPI static const struct acpi_device_id acpi_id[] = { { "NXP1001" }, + { "NXP1002" }, { "NXP7471" }, { } }; diff --git a/drivers/ntb/hw/amd/ntb_hw_amd.c b/drivers/ntb/hw/amd/ntb_hw_amd.c index 4940b6301d83..d687e8c2cc78 100644 --- a/drivers/ntb/hw/amd/ntb_hw_amd.c +++ b/drivers/ntb/hw/amd/ntb_hw_amd.c @@ -941,13 +941,10 @@ static void ndev_init_debugfs(struct amd_ntb_dev *ndev) ndev->debugfs_dir = debugfs_create_dir(pci_name(ndev->ntb.pdev), debugfs_dir); - if (IS_ERR(ndev->debugfs_dir)) - ndev->debugfs_info = NULL; - else - ndev->debugfs_info = - debugfs_create_file("info", S_IRUSR, - ndev->debugfs_dir, ndev, - &amd_ntb_debugfs_info); + ndev->debugfs_info = + debugfs_create_file("info", S_IRUSR, + ndev->debugfs_dir, ndev, + &amd_ntb_debugfs_info); } } diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c index 2abd2235bbca..f9e7847a378e 100644 --- a/drivers/ntb/ntb_transport.c +++ b/drivers/ntb/ntb_transport.c @@ -909,7 +909,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, return 0; } -static void ntb_qp_link_down_reset(struct ntb_transport_qp *qp) +static void ntb_qp_link_context_reset(struct ntb_transport_qp *qp) { qp->link_is_up = false; qp->active = false; @@ -932,6 +932,13 @@ static void ntb_qp_link_down_reset(struct ntb_transport_qp *qp) qp->tx_async = 0; } +static void ntb_qp_link_down_reset(struct ntb_transport_qp *qp) +{ + ntb_qp_link_context_reset(qp); + if (qp->remote_rx_info) + qp->remote_rx_info->entry = qp->rx_max_entry - 1; +} + static void ntb_qp_link_cleanup(struct ntb_transport_qp *qp) { struct ntb_transport_ctx *nt = qp->transport; @@ -1174,7 +1181,7 @@ static int ntb_transport_init_queue(struct ntb_transport_ctx *nt, qp->ndev = nt->ndev; qp->client_ready = false; qp->event_handler = NULL; - ntb_qp_link_down_reset(qp); + ntb_qp_link_context_reset(qp); if (mw_num < qp_count % mw_count) num_qps_mw = qp_count / mw_count + 1; @@ -1894,7 +1901,7 @@ err: static int ntb_process_tx(struct ntb_transport_qp *qp, struct ntb_queue_entry *entry) { - if (qp->tx_index == qp->remote_rx_info->entry) { + if (!ntb_transport_tx_free_entry(qp)) { qp->tx_ring_full++; return -EAGAIN; } @@ -2276,9 +2283,13 @@ int ntb_transport_tx_enqueue(struct ntb_transport_qp *qp, void *cb, void *data, struct ntb_queue_entry *entry; int rc; - if (!qp || !qp->link_is_up || !len) + if (!qp || !len) return -EINVAL; + /* If the qp link is down already, just ignore. */ + if (!qp->link_is_up) + return 0; + entry = ntb_list_rm(&qp->ntb_tx_free_q_lock, &qp->tx_free_q); if (!entry) { qp->tx_err_no_buf++; @@ -2418,7 +2429,7 @@ unsigned int ntb_transport_tx_free_entry(struct ntb_transport_qp *qp) unsigned int head = qp->tx_index; unsigned int tail = qp->remote_rx_info->entry; - return tail > head ? tail - head : qp->tx_max_entry + tail - head; + return tail >= head ? tail - head : qp->tx_max_entry + tail - head; } EXPORT_SYMBOL_GPL(ntb_transport_tx_free_entry); diff --git a/drivers/ntb/test/ntb_perf.c b/drivers/ntb/test/ntb_perf.c index 65e1e5cf1b29..553f1f46bc66 100644 --- a/drivers/ntb/test/ntb_perf.c +++ b/drivers/ntb/test/ntb_perf.c @@ -1355,7 +1355,7 @@ static void perf_setup_dbgfs(struct perf_ctx *perf) struct pci_dev *pdev = perf->ntb->pdev; perf->dbgfs_dir = debugfs_create_dir(pci_name(pdev), perf_dbgfs_topdir); - if (!perf->dbgfs_dir) { + if (IS_ERR(perf->dbgfs_dir)) { dev_warn(&perf->ntb->dev, "DebugFS unsupported\n"); return; } diff --git a/drivers/ntb/test/ntb_tool.c b/drivers/ntb/test/ntb_tool.c index eeeb4b1c97d2..641cb7e05a47 100644 --- a/drivers/ntb/test/ntb_tool.c +++ b/drivers/ntb/test/ntb_tool.c @@ -370,16 +370,9 @@ static ssize_t tool_fn_write(struct tool_ctx *tc, if (*offp) return 0; - buf = kmalloc(size + 1, GFP_KERNEL); - if (!buf) - return -ENOMEM; - - if (copy_from_user(buf, ubuf, size)) { - kfree(buf); - return -EFAULT; - } - - buf[size] = 0; + buf = memdup_user_nul(ubuf, size); + if (IS_ERR(buf)) + return PTR_ERR(buf); n = sscanf(buf, "%c %lli", &cmd, &bits); @@ -1495,8 +1488,6 @@ static void tool_setup_dbgfs(struct tool_ctx *tc) tc->dbgfs_dir = debugfs_create_dir(dev_name(&tc->ntb->dev), tool_dbgfs_topdir); - if (!tc->dbgfs_dir) - return; debugfs_create_file("port", 0600, tc->dbgfs_dir, tc, &tool_port_fops); diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig index 49bd09c7dd0a..e9ae66cc4189 100644 --- a/drivers/pci/Kconfig +++ b/drivers/pci/Kconfig @@ -196,7 +196,7 @@ config PCI_HYPERV config PCI_DYNAMIC_OF_NODES bool "Create Device tree nodes for PCI devices" - depends on OF + depends on OF_IRQ select OF_DYNAMIC help This option enables support for generating device tree nodes for some diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index ab2a4a3a4c06..795534589b98 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -997,6 +997,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) res = window->res; if (!res->flags && !res->start && !res->end) { release_resource(res); + resource_list_destroy_entry(window); continue; } diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 5de09d2eb014..eeec1d6f9023 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3726,7 +3726,7 @@ static void quirk_no_bus_reset(struct pci_dev *dev) */ static void quirk_nvidia_no_bus_reset(struct pci_dev *dev) { - if ((dev->device & 0xffc0) == 0x2340 || dev->device == 0x1eb8) + if ((dev->device & 0xffc0) == 0x2340) quirk_no_bus_reset(dev); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID, diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index e5a2ac4155f6..8fcaa26f0f8a 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -749,6 +749,8 @@ static void armv8pmu_start(struct arm_pmu *cpu_pmu) /* Enable all counters */ armv8pmu_pmcr_write(armv8pmu_pmcr_read() | ARMV8_PMU_PMCR_E); + + kvm_vcpu_pmu_resync_el0(); } static void armv8pmu_stop(struct arm_pmu *cpu_pmu) diff --git a/drivers/perf/cxl_pmu.c b/drivers/perf/cxl_pmu.c index 0a8f597e695b..365d964b0f6a 100644 --- a/drivers/perf/cxl_pmu.c +++ b/drivers/perf/cxl_pmu.c @@ -25,7 +25,7 @@ #include "../cxl/pmu.h" #define CXL_PMU_CAP_REG 0x0 -#define CXL_PMU_CAP_NUM_COUNTERS_MSK GENMASK_ULL(4, 0) +#define CXL_PMU_CAP_NUM_COUNTERS_MSK GENMASK_ULL(5, 0) #define CXL_PMU_CAP_COUNTER_WIDTH_MSK GENMASK_ULL(15, 8) #define CXL_PMU_CAP_NUM_EVN_CAP_REG_SUP_MSK GENMASK_ULL(24, 20) #define CXL_PMU_CAP_FILTERS_SUP_MSK GENMASK_ULL(39, 32) diff --git a/drivers/power/supply/power_supply_core.c b/drivers/power/supply/power_supply_core.c index f26319081698..025bfc0d46a3 100644 --- a/drivers/power/supply/power_supply_core.c +++ b/drivers/power/supply/power_supply_core.c @@ -1255,8 +1255,8 @@ static int psy_register_thermal(struct power_supply *psy) struct thermal_zone_params tzp = { .no_hwmon = IS_ENABLED(CONFIG_POWER_SUPPLY_HWMON) }; - psy->tzd = thermal_zone_device_register(psy->desc->name, - 0, 0, psy, &psy_tzd_ops, &tzp, 0, 0); + psy->tzd = thermal_tripless_zone_device_register(psy->desc->name, + psy, &psy_tzd_ops, &tzp); if (IS_ERR(psy->tzd)) return PTR_ERR(psy->tzd); ret = thermal_zone_device_enable(psy->tzd); diff --git a/drivers/powercap/intel_rapl_common.c b/drivers/powercap/intel_rapl_common.c index 5c2e6d5eea2a..40a2cc649c79 100644 --- a/drivers/powercap/intel_rapl_common.c +++ b/drivers/powercap/intel_rapl_common.c @@ -658,8 +658,6 @@ static struct rapl_primitive_info rpi_msr[NR_RAPL_PRIMITIVES] = { RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), [PL2_CLAMP] = PRIMITIVE_INFO_INIT(PL2_CLAMP, POWER_LIMIT2_CLAMP, 48, RAPL_DOMAIN_REG_LIMIT, ARBITRARY_UNIT, 0), - [PL4_ENABLE] = PRIMITIVE_INFO_INIT(PL4_ENABLE, POWER_LIMIT4_MASK, 0, - RAPL_DOMAIN_REG_PL4, ARBITRARY_UNIT, 0), [TIME_WINDOW1] = PRIMITIVE_INFO_INIT(TIME_WINDOW1, TIME_WINDOW1_MASK, 17, RAPL_DOMAIN_REG_LIMIT, TIME_UNIT, 0), [TIME_WINDOW2] = PRIMITIVE_INFO_INIT(TIME_WINDOW2, TIME_WINDOW2_MASK, 49, @@ -1458,7 +1456,7 @@ static void rapl_detect_powerlimit(struct rapl_domain *rd) } } - if (rapl_read_pl_data(rd, i, PL_ENABLE, false, &val64)) + if (rapl_read_pl_data(rd, i, PL_LIMIT, false, &val64)) rd->rpl[i].name = NULL; } } diff --git a/drivers/pwm/core.c b/drivers/pwm/core.c index 3dacceaef4a9..dc66e3405bf5 100644 --- a/drivers/pwm/core.c +++ b/drivers/pwm/core.c @@ -8,8 +8,8 @@ #include #include +#include #include -#include #include #include #include @@ -127,28 +127,28 @@ static int pwm_device_request(struct pwm_device *pwm, const char *label) } struct pwm_device * -of_pwm_xlate_with_flags(struct pwm_chip *pc, const struct of_phandle_args *args) +of_pwm_xlate_with_flags(struct pwm_chip *chip, const struct of_phandle_args *args) { struct pwm_device *pwm; - if (pc->of_pwm_n_cells < 2) + if (chip->of_pwm_n_cells < 2) return ERR_PTR(-EINVAL); /* flags in the third cell are optional */ if (args->args_count < 2) return ERR_PTR(-EINVAL); - if (args->args[0] >= pc->npwm) + if (args->args[0] >= chip->npwm) return ERR_PTR(-EINVAL); - pwm = pwm_request_from_chip(pc, args->args[0], NULL); + pwm = pwm_request_from_chip(chip, args->args[0], NULL); if (IS_ERR(pwm)) return pwm; pwm->args.period = args->args[1]; pwm->args.polarity = PWM_POLARITY_NORMAL; - if (pc->of_pwm_n_cells >= 3) { + if (chip->of_pwm_n_cells >= 3) { if (args->args_count > 2 && args->args[2] & PWM_POLARITY_INVERTED) pwm->args.polarity = PWM_POLARITY_INVERSED; } @@ -158,18 +158,18 @@ of_pwm_xlate_with_flags(struct pwm_chip *pc, const struct of_phandle_args *args) EXPORT_SYMBOL_GPL(of_pwm_xlate_with_flags); struct pwm_device * -of_pwm_single_xlate(struct pwm_chip *pc, const struct of_phandle_args *args) +of_pwm_single_xlate(struct pwm_chip *chip, const struct of_phandle_args *args) { struct pwm_device *pwm; - if (pc->of_pwm_n_cells < 1) + if (chip->of_pwm_n_cells < 1) return ERR_PTR(-EINVAL); /* validate that one cell is specified, optionally with flags */ if (args->args_count != 1 && args->args_count != 2) return ERR_PTR(-EINVAL); - pwm = pwm_request_from_chip(pc, 0, NULL); + pwm = pwm_request_from_chip(chip, 0, NULL); if (IS_ERR(pwm)) return pwm; @@ -312,22 +312,19 @@ EXPORT_SYMBOL_GPL(pwmchip_add); * pwmchip_remove() - remove a PWM chip * @chip: the PWM chip to remove * - * Removes a PWM chip. This function may return busy if the PWM chip provides - * a PWM device that is still requested. - * - * Returns: 0 on success or a negative error code on failure. + * Removes a PWM chip. */ void pwmchip_remove(struct pwm_chip *chip) { pwmchip_sysfs_unexport(chip); + if (IS_ENABLED(CONFIG_OF)) + of_pwmchip_remove(chip); + mutex_lock(&pwm_lock); list_del_init(&chip->list); - if (IS_ENABLED(CONFIG_OF)) - of_pwmchip_remove(chip); - free_pwms(chip); mutex_unlock(&pwm_lock); @@ -692,7 +689,7 @@ static struct pwm_device *of_pwm_get(struct device *dev, struct device_node *np, struct pwm_device *pwm = NULL; struct of_phandle_args args; struct device_link *dl; - struct pwm_chip *pc; + struct pwm_chip *chip; int index = 0; int err; @@ -709,16 +706,16 @@ static struct pwm_device *of_pwm_get(struct device *dev, struct device_node *np, return ERR_PTR(err); } - pc = fwnode_to_pwmchip(of_fwnode_handle(args.np)); - if (IS_ERR(pc)) { - if (PTR_ERR(pc) != -EPROBE_DEFER) + chip = fwnode_to_pwmchip(of_fwnode_handle(args.np)); + if (IS_ERR(chip)) { + if (PTR_ERR(chip) != -EPROBE_DEFER) pr_err("%s(): PWM chip not found\n", __func__); - pwm = ERR_CAST(pc); + pwm = ERR_CAST(chip); goto put; } - pwm = pc->of_xlate(pc, &args); + pwm = chip->of_xlate(chip, &args); if (IS_ERR(pwm)) goto put; diff --git a/drivers/pwm/pwm-apple.c b/drivers/pwm/pwm-apple.c index a38a62edd713..8e7d67fb5fbe 100644 --- a/drivers/pwm/pwm-apple.c +++ b/drivers/pwm/pwm-apple.c @@ -12,6 +12,7 @@ * - When APPLE_PWM_CTRL is set to 0, the output is constant low */ +#include #include #include #include diff --git a/drivers/pwm/pwm-atmel-hlcdc.c b/drivers/pwm/pwm-atmel-hlcdc.c index 96a709a9d49a..e271d920151e 100644 --- a/drivers/pwm/pwm-atmel-hlcdc.c +++ b/drivers/pwm/pwm-atmel-hlcdc.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -38,11 +39,11 @@ static inline struct atmel_hlcdc_pwm *to_atmel_hlcdc_pwm(struct pwm_chip *chip) return container_of(chip, struct atmel_hlcdc_pwm, chip); } -static int atmel_hlcdc_pwm_apply(struct pwm_chip *c, struct pwm_device *pwm, +static int atmel_hlcdc_pwm_apply(struct pwm_chip *chip, struct pwm_device *pwm, const struct pwm_state *state) { - struct atmel_hlcdc_pwm *chip = to_atmel_hlcdc_pwm(c); - struct atmel_hlcdc *hlcdc = chip->hlcdc; + struct atmel_hlcdc_pwm *atmel = to_atmel_hlcdc_pwm(chip); + struct atmel_hlcdc *hlcdc = atmel->hlcdc; unsigned int status; int ret; @@ -54,7 +55,7 @@ static int atmel_hlcdc_pwm_apply(struct pwm_chip *c, struct pwm_device *pwm, u32 pwmcfg; int pres; - if (!chip->errata || !chip->errata->slow_clk_erratum) { + if (!atmel->errata || !atmel->errata->slow_clk_erratum) { clk_freq = clk_get_rate(new_clk); if (!clk_freq) return -EINVAL; @@ -64,7 +65,7 @@ static int atmel_hlcdc_pwm_apply(struct pwm_chip *c, struct pwm_device *pwm, } /* Errata: cannot use slow clk on some IP revisions */ - if ((chip->errata && chip->errata->slow_clk_erratum) || + if ((atmel->errata && atmel->errata->slow_clk_erratum) || clk_period_ns > state->period) { new_clk = hlcdc->sys_clk; clk_freq = clk_get_rate(new_clk); @@ -77,8 +78,8 @@ static int atmel_hlcdc_pwm_apply(struct pwm_chip *c, struct pwm_device *pwm, for (pres = 0; pres <= ATMEL_HLCDC_PWMPS_MAX; pres++) { /* Errata: cannot divide by 1 on some IP revisions */ - if (!pres && chip->errata && - chip->errata->div1_clk_erratum) + if (!pres && atmel->errata && + atmel->errata->div1_clk_erratum) continue; if ((clk_period_ns << pres) >= state->period) @@ -90,7 +91,7 @@ static int atmel_hlcdc_pwm_apply(struct pwm_chip *c, struct pwm_device *pwm, pwmcfg = ATMEL_HLCDC_PWMPS(pres); - if (new_clk != chip->cur_clk) { + if (new_clk != atmel->cur_clk) { u32 gencfg = 0; int ret; @@ -98,8 +99,8 @@ static int atmel_hlcdc_pwm_apply(struct pwm_chip *c, struct pwm_device *pwm, if (ret) return ret; - clk_disable_unprepare(chip->cur_clk); - chip->cur_clk = new_clk; + clk_disable_unprepare(atmel->cur_clk); + atmel->cur_clk = new_clk; if (new_clk == hlcdc->sys_clk) gencfg = ATMEL_HLCDC_CLKPWMSEL; @@ -160,8 +161,8 @@ static int atmel_hlcdc_pwm_apply(struct pwm_chip *c, struct pwm_device *pwm, if (ret) return ret; - clk_disable_unprepare(chip->cur_clk); - chip->cur_clk = NULL; + clk_disable_unprepare(atmel->cur_clk); + atmel->cur_clk = NULL; } return 0; @@ -183,31 +184,32 @@ static const struct atmel_hlcdc_pwm_errata atmel_hlcdc_pwm_sama5d3_errata = { #ifdef CONFIG_PM_SLEEP static int atmel_hlcdc_pwm_suspend(struct device *dev) { - struct atmel_hlcdc_pwm *chip = dev_get_drvdata(dev); + struct atmel_hlcdc_pwm *atmel = dev_get_drvdata(dev); /* Keep the periph clock enabled if the PWM is still running. */ - if (pwm_is_enabled(&chip->chip.pwms[0])) - clk_disable_unprepare(chip->hlcdc->periph_clk); + if (pwm_is_enabled(&atmel->chip.pwms[0])) + clk_disable_unprepare(atmel->hlcdc->periph_clk); return 0; } static int atmel_hlcdc_pwm_resume(struct device *dev) { - struct atmel_hlcdc_pwm *chip = dev_get_drvdata(dev); + struct atmel_hlcdc_pwm *atmel = dev_get_drvdata(dev); struct pwm_state state; int ret; - pwm_get_state(&chip->chip.pwms[0], &state); + pwm_get_state(&atmel->chip.pwms[0], &state); /* Re-enable the periph clock it was stopped during suspend. */ if (!state.enabled) { - ret = clk_prepare_enable(chip->hlcdc->periph_clk); + ret = clk_prepare_enable(atmel->hlcdc->periph_clk); if (ret) return ret; } - return atmel_hlcdc_pwm_apply(&chip->chip, &chip->chip.pwms[0], &state); + return atmel_hlcdc_pwm_apply(&atmel->chip, &atmel->chip.pwms[0], + &state); } #endif @@ -244,14 +246,14 @@ static int atmel_hlcdc_pwm_probe(struct platform_device *pdev) { const struct of_device_id *match; struct device *dev = &pdev->dev; - struct atmel_hlcdc_pwm *chip; + struct atmel_hlcdc_pwm *atmel; struct atmel_hlcdc *hlcdc; int ret; hlcdc = dev_get_drvdata(dev->parent); - chip = devm_kzalloc(dev, sizeof(*chip), GFP_KERNEL); - if (!chip) + atmel = devm_kzalloc(dev, sizeof(*atmel), GFP_KERNEL); + if (!atmel) return -ENOMEM; ret = clk_prepare_enable(hlcdc->periph_clk); @@ -260,31 +262,31 @@ static int atmel_hlcdc_pwm_probe(struct platform_device *pdev) match = of_match_node(atmel_hlcdc_dt_ids, dev->parent->of_node); if (match) - chip->errata = match->data; + atmel->errata = match->data; - chip->hlcdc = hlcdc; - chip->chip.ops = &atmel_hlcdc_pwm_ops; - chip->chip.dev = dev; - chip->chip.npwm = 1; + atmel->hlcdc = hlcdc; + atmel->chip.ops = &atmel_hlcdc_pwm_ops; + atmel->chip.dev = dev; + atmel->chip.npwm = 1; - ret = pwmchip_add(&chip->chip); + ret = pwmchip_add(&atmel->chip); if (ret) { clk_disable_unprepare(hlcdc->periph_clk); return ret; } - platform_set_drvdata(pdev, chip); + platform_set_drvdata(pdev, atmel); return 0; } static void atmel_hlcdc_pwm_remove(struct platform_device *pdev) { - struct atmel_hlcdc_pwm *chip = platform_get_drvdata(pdev); + struct atmel_hlcdc_pwm *atmel = platform_get_drvdata(pdev); - pwmchip_remove(&chip->chip); + pwmchip_remove(&atmel->chip); - clk_disable_unprepare(chip->hlcdc->periph_clk); + clk_disable_unprepare(atmel->hlcdc->periph_clk); } static const struct of_device_id atmel_hlcdc_pwm_dt_ids[] = { diff --git a/drivers/pwm/pwm-atmel-tcb.c b/drivers/pwm/pwm-atmel-tcb.c index 4a116dc44f6e..c00dd37c5fbd 100644 --- a/drivers/pwm/pwm-atmel-tcb.c +++ b/drivers/pwm/pwm-atmel-tcb.c @@ -19,8 +19,7 @@ #include #include #include -#include -#include +#include #include #include #include @@ -34,7 +33,6 @@ ATMEL_TC_BEEVT | ATMEL_TC_BSWTRG) struct atmel_tcb_pwm_device { - enum pwm_polarity polarity; /* PWM polarity */ unsigned div; /* PWM clock divider */ unsigned duty; /* PWM duty expressed in clk cycles */ unsigned period; /* PWM period expressed in clk cycles */ @@ -57,7 +55,7 @@ struct atmel_tcb_pwm_chip { struct clk *clk; struct clk *gclk; struct clk *slow_clk; - struct atmel_tcb_pwm_device *pwms[NPWM]; + struct atmel_tcb_pwm_device pwms[NPWM]; struct atmel_tcb_channel bkup; }; @@ -68,37 +66,18 @@ static inline struct atmel_tcb_pwm_chip *to_tcb_chip(struct pwm_chip *chip) return container_of(chip, struct atmel_tcb_pwm_chip, chip); } -static int atmel_tcb_pwm_set_polarity(struct pwm_chip *chip, - struct pwm_device *pwm, - enum pwm_polarity polarity) -{ - struct atmel_tcb_pwm_chip *tcbpwmc = to_tcb_chip(chip); - struct atmel_tcb_pwm_device *tcbpwm = tcbpwmc->pwms[pwm->hwpwm]; - - tcbpwm->polarity = polarity; - - return 0; -} - static int atmel_tcb_pwm_request(struct pwm_chip *chip, struct pwm_device *pwm) { struct atmel_tcb_pwm_chip *tcbpwmc = to_tcb_chip(chip); - struct atmel_tcb_pwm_device *tcbpwm; + struct atmel_tcb_pwm_device *tcbpwm = &tcbpwmc->pwms[pwm->hwpwm]; unsigned cmr; int ret; - tcbpwm = devm_kzalloc(chip->dev, sizeof(*tcbpwm), GFP_KERNEL); - if (!tcbpwm) - return -ENOMEM; - ret = clk_prepare_enable(tcbpwmc->clk); - if (ret) { - devm_kfree(chip->dev, tcbpwm); + if (ret) return ret; - } - tcbpwm->polarity = PWM_POLARITY_NORMAL; tcbpwm->duty = 0; tcbpwm->period = 0; tcbpwm->div = 0; @@ -131,27 +110,22 @@ static int atmel_tcb_pwm_request(struct pwm_chip *chip, regmap_write(tcbpwmc->regmap, ATMEL_TC_REG(tcbpwmc->channel, CMR), cmr); spin_unlock(&tcbpwmc->lock); - tcbpwmc->pwms[pwm->hwpwm] = tcbpwm; - return 0; } static void atmel_tcb_pwm_free(struct pwm_chip *chip, struct pwm_device *pwm) { struct atmel_tcb_pwm_chip *tcbpwmc = to_tcb_chip(chip); - struct atmel_tcb_pwm_device *tcbpwm = tcbpwmc->pwms[pwm->hwpwm]; clk_disable_unprepare(tcbpwmc->clk); - tcbpwmc->pwms[pwm->hwpwm] = NULL; - devm_kfree(chip->dev, tcbpwm); } -static void atmel_tcb_pwm_disable(struct pwm_chip *chip, struct pwm_device *pwm) +static void atmel_tcb_pwm_disable(struct pwm_chip *chip, struct pwm_device *pwm, + enum pwm_polarity polarity) { struct atmel_tcb_pwm_chip *tcbpwmc = to_tcb_chip(chip); - struct atmel_tcb_pwm_device *tcbpwm = tcbpwmc->pwms[pwm->hwpwm]; + struct atmel_tcb_pwm_device *tcbpwm = &tcbpwmc->pwms[pwm->hwpwm]; unsigned cmr; - enum pwm_polarity polarity = tcbpwm->polarity; /* * If duty is 0 the timer will be stopped and we have to @@ -203,12 +177,12 @@ static void atmel_tcb_pwm_disable(struct pwm_chip *chip, struct pwm_device *pwm) spin_unlock(&tcbpwmc->lock); } -static int atmel_tcb_pwm_enable(struct pwm_chip *chip, struct pwm_device *pwm) +static int atmel_tcb_pwm_enable(struct pwm_chip *chip, struct pwm_device *pwm, + enum pwm_polarity polarity) { struct atmel_tcb_pwm_chip *tcbpwmc = to_tcb_chip(chip); - struct atmel_tcb_pwm_device *tcbpwm = tcbpwmc->pwms[pwm->hwpwm]; + struct atmel_tcb_pwm_device *tcbpwm = &tcbpwmc->pwms[pwm->hwpwm]; u32 cmr; - enum pwm_polarity polarity = tcbpwm->polarity; /* * If duty is 0 the timer will be stopped and we have to @@ -291,7 +265,7 @@ static int atmel_tcb_pwm_config(struct pwm_chip *chip, struct pwm_device *pwm, int duty_ns, int period_ns) { struct atmel_tcb_pwm_chip *tcbpwmc = to_tcb_chip(chip); - struct atmel_tcb_pwm_device *tcbpwm = tcbpwmc->pwms[pwm->hwpwm]; + struct atmel_tcb_pwm_device *tcbpwm = &tcbpwmc->pwms[pwm->hwpwm]; struct atmel_tcb_pwm_device *atcbpwm = NULL; int i = 0; int slowclk = 0; @@ -338,9 +312,9 @@ static int atmel_tcb_pwm_config(struct pwm_chip *chip, struct pwm_device *pwm, period = div_u64(period_ns, min); if (pwm->hwpwm == 0) - atcbpwm = tcbpwmc->pwms[1]; + atcbpwm = &tcbpwmc->pwms[1]; else - atcbpwm = tcbpwmc->pwms[0]; + atcbpwm = &tcbpwmc->pwms[0]; /* * PWM devices provided by the TCB driver are grouped by 2. @@ -371,11 +345,8 @@ static int atmel_tcb_pwm_apply(struct pwm_chip *chip, struct pwm_device *pwm, int duty_cycle, period; int ret; - /* This function only sets a flag in driver data */ - atmel_tcb_pwm_set_polarity(chip, pwm, state->polarity); - if (!state->enabled) { - atmel_tcb_pwm_disable(chip, pwm); + atmel_tcb_pwm_disable(chip, pwm, state->polarity); return 0; } @@ -386,7 +357,7 @@ static int atmel_tcb_pwm_apply(struct pwm_chip *chip, struct pwm_device *pwm, if (ret) return ret; - return atmel_tcb_pwm_enable(chip, pwm); + return atmel_tcb_pwm_enable(chip, pwm, state->polarity); } static const struct pwm_ops atmel_tcb_pwm_ops = { @@ -422,13 +393,14 @@ static int atmel_tcb_pwm_probe(struct platform_device *pdev) struct atmel_tcb_pwm_chip *tcbpwm; const struct atmel_tcb_config *config; struct device_node *np = pdev->dev.of_node; - struct regmap *regmap; - struct clk *clk, *gclk = NULL; - struct clk *slow_clk; char clk_name[] = "t0_clk"; int err; int channel; + tcbpwm = devm_kzalloc(&pdev->dev, sizeof(*tcbpwm), GFP_KERNEL); + if (tcbpwm == NULL) + return -ENOMEM; + err = of_property_read_u32(np, "reg", &channel); if (err < 0) { dev_err(&pdev->dev, @@ -437,49 +409,43 @@ static int atmel_tcb_pwm_probe(struct platform_device *pdev) return err; } - regmap = syscon_node_to_regmap(np->parent); - if (IS_ERR(regmap)) - return PTR_ERR(regmap); + tcbpwm->regmap = syscon_node_to_regmap(np->parent); + if (IS_ERR(tcbpwm->regmap)) + return PTR_ERR(tcbpwm->regmap); - slow_clk = of_clk_get_by_name(np->parent, "slow_clk"); - if (IS_ERR(slow_clk)) - return PTR_ERR(slow_clk); + tcbpwm->slow_clk = of_clk_get_by_name(np->parent, "slow_clk"); + if (IS_ERR(tcbpwm->slow_clk)) + return PTR_ERR(tcbpwm->slow_clk); clk_name[1] += channel; - clk = of_clk_get_by_name(np->parent, clk_name); - if (IS_ERR(clk)) - clk = of_clk_get_by_name(np->parent, "t0_clk"); - if (IS_ERR(clk)) - return PTR_ERR(clk); + tcbpwm->clk = of_clk_get_by_name(np->parent, clk_name); + if (IS_ERR(tcbpwm->clk)) + tcbpwm->clk = of_clk_get_by_name(np->parent, "t0_clk"); + if (IS_ERR(tcbpwm->clk)) { + err = PTR_ERR(tcbpwm->clk); + goto err_slow_clk; + } match = of_match_node(atmel_tcb_of_match, np->parent); config = match->data; if (config->has_gclk) { - gclk = of_clk_get_by_name(np->parent, "gclk"); - if (IS_ERR(gclk)) - return PTR_ERR(gclk); - } - - tcbpwm = devm_kzalloc(&pdev->dev, sizeof(*tcbpwm), GFP_KERNEL); - if (tcbpwm == NULL) { - err = -ENOMEM; - goto err_slow_clk; + tcbpwm->gclk = of_clk_get_by_name(np->parent, "gclk"); + if (IS_ERR(tcbpwm->gclk)) { + err = PTR_ERR(tcbpwm->gclk); + goto err_clk; + } } tcbpwm->chip.dev = &pdev->dev; tcbpwm->chip.ops = &atmel_tcb_pwm_ops; tcbpwm->chip.npwm = NPWM; tcbpwm->channel = channel; - tcbpwm->regmap = regmap; - tcbpwm->clk = clk; - tcbpwm->gclk = gclk; - tcbpwm->slow_clk = slow_clk; tcbpwm->width = config->counter_width; - err = clk_prepare_enable(slow_clk); + err = clk_prepare_enable(tcbpwm->slow_clk); if (err) - goto err_slow_clk; + goto err_gclk; spin_lock_init(&tcbpwm->lock); @@ -494,8 +460,14 @@ static int atmel_tcb_pwm_probe(struct platform_device *pdev) err_disable_clk: clk_disable_unprepare(tcbpwm->slow_clk); +err_gclk: + clk_put(tcbpwm->gclk); + +err_clk: + clk_put(tcbpwm->clk); + err_slow_clk: - clk_put(slow_clk); + clk_put(tcbpwm->slow_clk); return err; } @@ -507,8 +479,9 @@ static void atmel_tcb_pwm_remove(struct platform_device *pdev) pwmchip_remove(&tcbpwm->chip); clk_disable_unprepare(tcbpwm->slow_clk); - clk_put(tcbpwm->slow_clk); + clk_put(tcbpwm->gclk); clk_put(tcbpwm->clk); + clk_put(tcbpwm->slow_clk); } static const struct of_device_id atmel_tcb_pwm_dt_ids[] = { diff --git a/drivers/pwm/pwm-atmel.c b/drivers/pwm/pwm-atmel.c index 5f7d286871cf..1f73325d1bea 100644 --- a/drivers/pwm/pwm-atmel.c +++ b/drivers/pwm/pwm-atmel.c @@ -25,7 +25,6 @@ #include #include #include -#include #include #include #include @@ -36,7 +35,7 @@ #define PWM_SR 0x0C #define PWM_ISR 0x1C /* Bit field in SR */ -#define PWM_SR_ALL_CH_ON 0x0F +#define PWM_SR_ALL_CH_MASK 0x0F /* The following register is PWM channel related registers */ #define PWM_CH_REG_OFFSET 0x200 @@ -464,6 +463,42 @@ static const struct of_device_id atmel_pwm_dt_ids[] = { }; MODULE_DEVICE_TABLE(of, atmel_pwm_dt_ids); +static int atmel_pwm_enable_clk_if_on(struct atmel_pwm_chip *atmel_pwm, bool on) +{ + unsigned int i, cnt = 0; + unsigned long sr; + int ret = 0; + + sr = atmel_pwm_readl(atmel_pwm, PWM_SR) & PWM_SR_ALL_CH_MASK; + if (!sr) + return 0; + + cnt = bitmap_weight(&sr, atmel_pwm->chip.npwm); + + if (!on) + goto disable_clk; + + for (i = 0; i < cnt; i++) { + ret = clk_enable(atmel_pwm->clk); + if (ret) { + dev_err(atmel_pwm->chip.dev, + "failed to enable clock for pwm %pe\n", + ERR_PTR(ret)); + + cnt = i; + goto disable_clk; + } + } + + return 0; + +disable_clk: + while (cnt--) + clk_disable(atmel_pwm->clk); + + return ret; +} + static int atmel_pwm_probe(struct platform_device *pdev) { struct atmel_pwm_chip *atmel_pwm; @@ -482,42 +517,31 @@ static int atmel_pwm_probe(struct platform_device *pdev) if (IS_ERR(atmel_pwm->base)) return PTR_ERR(atmel_pwm->base); - atmel_pwm->clk = devm_clk_get(&pdev->dev, NULL); + atmel_pwm->clk = devm_clk_get_prepared(&pdev->dev, NULL); if (IS_ERR(atmel_pwm->clk)) - return PTR_ERR(atmel_pwm->clk); - - ret = clk_prepare(atmel_pwm->clk); - if (ret) { - dev_err(&pdev->dev, "failed to prepare PWM clock\n"); - return ret; - } + return dev_err_probe(&pdev->dev, PTR_ERR(atmel_pwm->clk), + "failed to get prepared PWM clock\n"); atmel_pwm->chip.dev = &pdev->dev; atmel_pwm->chip.ops = &atmel_pwm_ops; atmel_pwm->chip.npwm = 4; - ret = pwmchip_add(&atmel_pwm->chip); + ret = atmel_pwm_enable_clk_if_on(atmel_pwm, true); + if (ret < 0) + return ret; + + ret = devm_pwmchip_add(&pdev->dev, &atmel_pwm->chip); if (ret < 0) { - dev_err(&pdev->dev, "failed to add PWM chip %d\n", ret); - goto unprepare_clk; + dev_err_probe(&pdev->dev, ret, "failed to add PWM chip\n"); + goto disable_clk; } - platform_set_drvdata(pdev, atmel_pwm); + return 0; + +disable_clk: + atmel_pwm_enable_clk_if_on(atmel_pwm, false); return ret; - -unprepare_clk: - clk_unprepare(atmel_pwm->clk); - return ret; -} - -static void atmel_pwm_remove(struct platform_device *pdev) -{ - struct atmel_pwm_chip *atmel_pwm = platform_get_drvdata(pdev); - - pwmchip_remove(&atmel_pwm->chip); - - clk_unprepare(atmel_pwm->clk); } static struct platform_driver atmel_pwm_driver = { @@ -526,7 +550,6 @@ static struct platform_driver atmel_pwm_driver = { .of_match_table = of_match_ptr(atmel_pwm_dt_ids), }, .probe = atmel_pwm_probe, - .remove_new = atmel_pwm_remove, }; module_platform_driver(atmel_pwm_driver); diff --git a/drivers/pwm/pwm-bcm-kona.c b/drivers/pwm/pwm-bcm-kona.c index 4fa6e249e4cf..e5b00cc9f7a7 100644 --- a/drivers/pwm/pwm-bcm-kona.c +++ b/drivers/pwm/pwm-bcm-kona.c @@ -61,9 +61,9 @@ struct kona_pwmc { struct clk *clk; }; -static inline struct kona_pwmc *to_kona_pwmc(struct pwm_chip *_chip) +static inline struct kona_pwmc *to_kona_pwmc(struct pwm_chip *chip) { - return container_of(_chip, struct kona_pwmc, chip); + return container_of(chip, struct kona_pwmc, chip); } /* diff --git a/drivers/pwm/pwm-berlin.c b/drivers/pwm/pwm-berlin.c index 0c5992a046b2..0971c666afd1 100644 --- a/drivers/pwm/pwm-berlin.c +++ b/drivers/pwm/pwm-berlin.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/pwm/pwm-crc.c b/drivers/pwm/pwm-crc.c index 4703b4a0b6e4..b9f063dc6b5f 100644 --- a/drivers/pwm/pwm-crc.c +++ b/drivers/pwm/pwm-crc.c @@ -34,9 +34,9 @@ struct crystalcove_pwm { struct regmap *regmap; }; -static inline struct crystalcove_pwm *to_crc_pwm(struct pwm_chip *pc) +static inline struct crystalcove_pwm *to_crc_pwm(struct pwm_chip *chip) { - return container_of(pc, struct crystalcove_pwm, chip); + return container_of(chip, struct crystalcove_pwm, chip); } static int crc_pwm_calc_clk_div(int period_ns) diff --git a/drivers/pwm/pwm-cros-ec.c b/drivers/pwm/pwm-cros-ec.c index 74e863aa1d8d..baaac0c33aa0 100644 --- a/drivers/pwm/pwm-cros-ec.c +++ b/drivers/pwm/pwm-cros-ec.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -37,9 +38,9 @@ struct cros_ec_pwm { u16 duty_cycle; }; -static inline struct cros_ec_pwm_device *pwm_to_cros_ec_pwm(struct pwm_chip *c) +static inline struct cros_ec_pwm_device *pwm_to_cros_ec_pwm(struct pwm_chip *chip) { - return container_of(c, struct cros_ec_pwm_device, chip); + return container_of(chip, struct cros_ec_pwm_device, chip); } static int cros_ec_pwm_request(struct pwm_chip *chip, struct pwm_device *pwm) @@ -218,14 +219,14 @@ static int cros_ec_pwm_get_state(struct pwm_chip *chip, struct pwm_device *pwm, } static struct pwm_device * -cros_ec_pwm_xlate(struct pwm_chip *pc, const struct of_phandle_args *args) +cros_ec_pwm_xlate(struct pwm_chip *chip, const struct of_phandle_args *args) { struct pwm_device *pwm; - if (args->args[0] >= pc->npwm) + if (args->args[0] >= chip->npwm) return ERR_PTR(-EINVAL); - pwm = pwm_request_from_chip(pc, args->args[0], NULL); + pwm = pwm_request_from_chip(chip, args->args[0], NULL); if (IS_ERR(pwm)) return pwm; diff --git a/drivers/pwm/pwm-fsl-ftm.c b/drivers/pwm/pwm-fsl-ftm.c index 5caadbd6194e..b7c6045c5d08 100644 --- a/drivers/pwm/pwm-fsl-ftm.c +++ b/drivers/pwm/pwm-fsl-ftm.c @@ -11,8 +11,7 @@ #include #include #include -#include -#include +#include #include #include #include diff --git a/drivers/pwm/pwm-hibvt.c b/drivers/pwm/pwm-hibvt.c index b95df1a96127..f7ba6fe9a349 100644 --- a/drivers/pwm/pwm-hibvt.c +++ b/drivers/pwm/pwm-hibvt.c @@ -10,7 +10,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/pwm/pwm-imx1.c b/drivers/pwm/pwm-imx1.c index 1f2eb1c8ff6c..0651983bed19 100644 --- a/drivers/pwm/pwm-imx1.c +++ b/drivers/pwm/pwm-imx1.c @@ -14,7 +14,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/pwm/pwm-jz4740.c b/drivers/pwm/pwm-jz4740.c index 3b7067f6cd0d..ef1293f2a897 100644 --- a/drivers/pwm/pwm-jz4740.c +++ b/drivers/pwm/pwm-jz4740.c @@ -15,7 +15,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/pwm/pwm-lp3943.c b/drivers/pwm/pwm-lp3943.c index 35675e4058c6..4b133a17f4be 100644 --- a/drivers/pwm/pwm-lp3943.c +++ b/drivers/pwm/pwm-lp3943.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -24,9 +25,9 @@ struct lp3943_pwm { struct lp3943_platform_data *pdata; }; -static inline struct lp3943_pwm *to_lp3943_pwm(struct pwm_chip *_chip) +static inline struct lp3943_pwm *to_lp3943_pwm(struct pwm_chip *chip) { - return container_of(_chip, struct lp3943_pwm, chip); + return container_of(chip, struct lp3943_pwm, chip); } static struct lp3943_pwm_map * diff --git a/drivers/pwm/pwm-lpc18xx-sct.c b/drivers/pwm/pwm-lpc18xx-sct.c index b9bf5b366f4b..7a19a840bca5 100644 --- a/drivers/pwm/pwm-lpc18xx-sct.c +++ b/drivers/pwm/pwm-lpc18xx-sct.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -366,30 +367,21 @@ static int lpc18xx_pwm_probe(struct platform_device *pdev) if (IS_ERR(lpc18xx_pwm->base)) return PTR_ERR(lpc18xx_pwm->base); - lpc18xx_pwm->pwm_clk = devm_clk_get(&pdev->dev, "pwm"); + lpc18xx_pwm->pwm_clk = devm_clk_get_enabled(&pdev->dev, "pwm"); if (IS_ERR(lpc18xx_pwm->pwm_clk)) return dev_err_probe(&pdev->dev, PTR_ERR(lpc18xx_pwm->pwm_clk), "failed to get pwm clock\n"); - ret = clk_prepare_enable(lpc18xx_pwm->pwm_clk); - if (ret < 0) - return dev_err_probe(&pdev->dev, ret, - "could not prepare or enable pwm clock\n"); - lpc18xx_pwm->clk_rate = clk_get_rate(lpc18xx_pwm->pwm_clk); - if (!lpc18xx_pwm->clk_rate) { - ret = dev_err_probe(&pdev->dev, - -EINVAL, "pwm clock has no frequency\n"); - goto disable_pwmclk; - } + if (!lpc18xx_pwm->clk_rate) + return dev_err_probe(&pdev->dev, + -EINVAL, "pwm clock has no frequency\n"); /* * If clkrate is too fast, the calculations in .apply() might overflow. */ - if (lpc18xx_pwm->clk_rate > NSEC_PER_SEC) { - ret = dev_err_probe(&pdev->dev, -EINVAL, "pwm clock to fast\n"); - goto disable_pwmclk; - } + if (lpc18xx_pwm->clk_rate > NSEC_PER_SEC) + return dev_err_probe(&pdev->dev, -EINVAL, "pwm clock to fast\n"); mutex_init(&lpc18xx_pwm->res_lock); mutex_init(&lpc18xx_pwm->period_lock); @@ -435,18 +427,12 @@ static int lpc18xx_pwm_probe(struct platform_device *pdev) lpc18xx_pwm_writel(lpc18xx_pwm, LPC18XX_PWM_CTRL, val); ret = pwmchip_add(&lpc18xx_pwm->chip); - if (ret < 0) { - dev_err_probe(&pdev->dev, ret, "pwmchip_add failed\n"); - goto disable_pwmclk; - } + if (ret < 0) + return dev_err_probe(&pdev->dev, ret, "pwmchip_add failed\n"); platform_set_drvdata(pdev, lpc18xx_pwm); return 0; - -disable_pwmclk: - clk_disable_unprepare(lpc18xx_pwm->pwm_clk); - return ret; } static void lpc18xx_pwm_remove(struct platform_device *pdev) @@ -459,8 +445,6 @@ static void lpc18xx_pwm_remove(struct platform_device *pdev) val = lpc18xx_pwm_readl(lpc18xx_pwm, LPC18XX_PWM_CTRL); lpc18xx_pwm_writel(lpc18xx_pwm, LPC18XX_PWM_CTRL, val | LPC18XX_PWM_CTRL_HALT); - - clk_disable_unprepare(lpc18xx_pwm->pwm_clk); } static struct platform_driver lpc18xx_pwm_driver = { diff --git a/drivers/pwm/pwm-lpc32xx.c b/drivers/pwm/pwm-lpc32xx.c index 86a0ea0f6955..806f0bb3ad6d 100644 --- a/drivers/pwm/pwm-lpc32xx.c +++ b/drivers/pwm/pwm-lpc32xx.c @@ -51,10 +51,10 @@ static int lpc32xx_pwm_config(struct pwm_chip *chip, struct pwm_device *pwm, if (duty_cycles > 255) duty_cycles = 255; - val = readl(lpc32xx->base + (pwm->hwpwm << 2)); + val = readl(lpc32xx->base); val &= ~0xFFFF; val |= (period_cycles << 8) | duty_cycles; - writel(val, lpc32xx->base + (pwm->hwpwm << 2)); + writel(val, lpc32xx->base); return 0; } @@ -69,9 +69,9 @@ static int lpc32xx_pwm_enable(struct pwm_chip *chip, struct pwm_device *pwm) if (ret) return ret; - val = readl(lpc32xx->base + (pwm->hwpwm << 2)); + val = readl(lpc32xx->base); val |= PWM_ENABLE; - writel(val, lpc32xx->base + (pwm->hwpwm << 2)); + writel(val, lpc32xx->base); return 0; } @@ -81,9 +81,9 @@ static void lpc32xx_pwm_disable(struct pwm_chip *chip, struct pwm_device *pwm) struct lpc32xx_pwm_chip *lpc32xx = to_lpc32xx_pwm_chip(chip); u32 val; - val = readl(lpc32xx->base + (pwm->hwpwm << 2)); + val = readl(lpc32xx->base); val &= ~PWM_ENABLE; - writel(val, lpc32xx->base + (pwm->hwpwm << 2)); + writel(val, lpc32xx->base); clk_disable_unprepare(lpc32xx->clk); } @@ -141,9 +141,9 @@ static int lpc32xx_pwm_probe(struct platform_device *pdev) lpc32xx->chip.npwm = 1; /* If PWM is disabled, configure the output to the default value */ - val = readl(lpc32xx->base + (lpc32xx->chip.pwms[0].hwpwm << 2)); + val = readl(lpc32xx->base); val &= ~PWM_PIN_LEVEL; - writel(val, lpc32xx->base + (lpc32xx->chip.pwms[0].hwpwm << 2)); + writel(val, lpc32xx->base); ret = devm_pwmchip_add(&pdev->dev, &lpc32xx->chip); if (ret < 0) { diff --git a/drivers/pwm/pwm-mediatek.c b/drivers/pwm/pwm-mediatek.c index 7a51d210a877..6adb0ed01906 100644 --- a/drivers/pwm/pwm-mediatek.c +++ b/drivers/pwm/pwm-mediatek.c @@ -14,7 +14,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/pwm/pwm-meson.c b/drivers/pwm/pwm-meson.c index 22f54db3ae8e..25519cddc2a9 100644 --- a/drivers/pwm/pwm-meson.c +++ b/drivers/pwm/pwm-meson.c @@ -37,7 +37,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/pwm/pwm-microchip-core.c b/drivers/pwm/pwm-microchip-core.c index 8750b57684a9..e7525c98105e 100644 --- a/drivers/pwm/pwm-microchip-core.c +++ b/drivers/pwm/pwm-microchip-core.c @@ -37,7 +37,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/pwm/pwm-mtk-disp.c b/drivers/pwm/pwm-mtk-disp.c index 2401b6733241..a83bd6e18b07 100644 --- a/drivers/pwm/pwm-mtk-disp.c +++ b/drivers/pwm/pwm-mtk-disp.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/pwm/pwm-ntxec.c b/drivers/pwm/pwm-ntxec.c index ab63b081df53..7514ea384ec5 100644 --- a/drivers/pwm/pwm-ntxec.c +++ b/drivers/pwm/pwm-ntxec.c @@ -24,7 +24,6 @@ #include struct ntxec_pwm { - struct device *dev; struct ntxec *ec; struct pwm_chip chip; }; @@ -141,14 +140,13 @@ static int ntxec_pwm_probe(struct platform_device *pdev) struct ntxec_pwm *priv; struct pwm_chip *chip; - pdev->dev.of_node = pdev->dev.parent->of_node; + device_set_of_node_from_dev(&pdev->dev, pdev->dev.parent); priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL); if (!priv) return -ENOMEM; priv->ec = ec; - priv->dev = &pdev->dev; chip = &priv->chip; chip->dev = &pdev->dev; diff --git a/drivers/pwm/pwm-pxa.c b/drivers/pwm/pwm-pxa.c index 762429d5647f..1e475ed10180 100644 --- a/drivers/pwm/pwm-pxa.c +++ b/drivers/pwm/pwm-pxa.c @@ -15,6 +15,7 @@ * input clock (PWMCR_SD is set) and the output is driven to inactive. */ +#include #include #include #include @@ -156,13 +157,6 @@ MODULE_DEVICE_TABLE(of, pwm_of_match); #define pwm_of_match NULL #endif -static const struct platform_device_id *pxa_pwm_get_id_dt(struct device *dev) -{ - const struct of_device_id *id = of_match_device(pwm_of_match, dev); - - return id ? id->data : NULL; -} - static int pwm_probe(struct platform_device *pdev) { const struct platform_device_id *id = platform_get_device_id(pdev); @@ -170,7 +164,7 @@ static int pwm_probe(struct platform_device *pdev) int ret = 0; if (IS_ENABLED(CONFIG_OF) && id == NULL) - id = pxa_pwm_get_id_dt(&pdev->dev); + id = of_device_get_match_data(&pdev->dev); if (id == NULL) return -EINVAL; diff --git a/drivers/pwm/pwm-rockchip.c b/drivers/pwm/pwm-rockchip.c index c1a1f2d864b5..03ee18fb82d5 100644 --- a/drivers/pwm/pwm-rockchip.c +++ b/drivers/pwm/pwm-rockchip.c @@ -52,9 +52,9 @@ struct rockchip_pwm_data { u32 enable_conf; }; -static inline struct rockchip_pwm_chip *to_rockchip_pwm_chip(struct pwm_chip *c) +static inline struct rockchip_pwm_chip *to_rockchip_pwm_chip(struct pwm_chip *chip) { - return container_of(c, struct rockchip_pwm_chip, chip); + return container_of(chip, struct rockchip_pwm_chip, chip); } static int rockchip_pwm_get_state(struct pwm_chip *chip, diff --git a/drivers/pwm/pwm-rz-mtu3.c b/drivers/pwm/pwm-rz-mtu3.c index bed8bd671e37..a56cecb0e46e 100644 --- a/drivers/pwm/pwm-rz-mtu3.c +++ b/drivers/pwm/pwm-rz-mtu3.c @@ -40,7 +40,7 @@ * struct rz_mtu3_channel_io_map - MTU3 pwm channel map * * @base_pwm_number: First PWM of a channel - * @num: number of IOs on the HW channel. + * @num_channel_ios: number of IOs on the HW channel. */ struct rz_mtu3_channel_io_map { u8 base_pwm_number; diff --git a/drivers/pwm/pwm-sifive.c b/drivers/pwm/pwm-sifive.c index ae49d67ab2b1..eabddb7c7820 100644 --- a/drivers/pwm/pwm-sifive.c +++ b/drivers/pwm/pwm-sifive.c @@ -13,6 +13,7 @@ */ #include #include +#include #include #include #include @@ -51,9 +52,9 @@ struct pwm_sifive_ddata { }; static inline -struct pwm_sifive_ddata *pwm_sifive_chip_to_ddata(struct pwm_chip *c) +struct pwm_sifive_ddata *pwm_sifive_chip_to_ddata(struct pwm_chip *chip) { - return container_of(c, struct pwm_sifive_ddata, chip); + return container_of(chip, struct pwm_sifive_ddata, chip); } static int pwm_sifive_request(struct pwm_chip *chip, struct pwm_device *pwm) diff --git a/drivers/pwm/pwm-sl28cpld.c b/drivers/pwm/pwm-sl28cpld.c index e64900ad4ba1..9e42e3a74ad6 100644 --- a/drivers/pwm/pwm-sl28cpld.c +++ b/drivers/pwm/pwm-sl28cpld.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -80,12 +81,15 @@ regmap_write((priv)->regmap, (priv)->offset + (reg), (val)) struct sl28cpld_pwm { - struct pwm_chip pwm_chip; + struct pwm_chip chip; struct regmap *regmap; u32 offset; }; -#define sl28cpld_pwm_from_chip(_chip) \ - container_of(_chip, struct sl28cpld_pwm, pwm_chip) + +static inline struct sl28cpld_pwm *sl28cpld_pwm_from_chip(struct pwm_chip *chip) +{ + return container_of(chip, struct sl28cpld_pwm, chip); +} static int sl28cpld_pwm_get_state(struct pwm_chip *chip, struct pwm_device *pwm, @@ -228,12 +232,12 @@ static int sl28cpld_pwm_probe(struct platform_device *pdev) } /* Initialize the pwm_chip structure */ - chip = &priv->pwm_chip; + chip = &priv->chip; chip->dev = &pdev->dev; chip->ops = &sl28cpld_pwm_ops; chip->npwm = 1; - ret = devm_pwmchip_add(&pdev->dev, &priv->pwm_chip); + ret = devm_pwmchip_add(&pdev->dev, chip); if (ret) { dev_err(&pdev->dev, "failed to add PWM chip (%pe)", ERR_PTR(ret)); diff --git a/drivers/pwm/pwm-sprd.c b/drivers/pwm/pwm-sprd.c index d43a6fa3f4e0..1499c8c1fe37 100644 --- a/drivers/pwm/pwm-sprd.c +++ b/drivers/pwm/pwm-sprd.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/pwm/pwm-stm32.c b/drivers/pwm/pwm-stm32.c index 62e397aeb9aa..3d6be7749e23 100644 --- a/drivers/pwm/pwm-stm32.c +++ b/drivers/pwm/pwm-stm32.c @@ -637,7 +637,7 @@ static int stm32_pwm_probe(struct platform_device *pdev) priv->chip.ops = &stm32pwm_ops; priv->chip.npwm = stm32_pwm_detect_channels(priv); - ret = pwmchip_add(&priv->chip); + ret = devm_pwmchip_add(dev, &priv->chip); if (ret < 0) return ret; @@ -646,17 +646,6 @@ static int stm32_pwm_probe(struct platform_device *pdev) return 0; } -static void stm32_pwm_remove(struct platform_device *pdev) -{ - struct stm32_pwm *priv = platform_get_drvdata(pdev); - unsigned int i; - - for (i = 0; i < priv->chip.npwm; i++) - pwm_disable(&priv->chip.pwms[i]); - - pwmchip_remove(&priv->chip); -} - static int __maybe_unused stm32_pwm_suspend(struct device *dev) { struct stm32_pwm *priv = dev_get_drvdata(dev); @@ -701,7 +690,6 @@ MODULE_DEVICE_TABLE(of, stm32_pwm_of_match); static struct platform_driver stm32_pwm_driver = { .probe = stm32_pwm_probe, - .remove_new = stm32_pwm_remove, .driver = { .name = "stm32-pwm", .of_match_table = stm32_pwm_of_match, diff --git a/drivers/pwm/pwm-stmpe.c b/drivers/pwm/pwm-stmpe.c index 5d4a4762ce0c..e205405c4828 100644 --- a/drivers/pwm/pwm-stmpe.c +++ b/drivers/pwm/pwm-stmpe.c @@ -61,8 +61,8 @@ static int stmpe_24xx_pwm_enable(struct pwm_chip *chip, struct pwm_device *pwm) return 0; } -static void stmpe_24xx_pwm_disable(struct pwm_chip *chip, - struct pwm_device *pwm) +static int stmpe_24xx_pwm_disable(struct pwm_chip *chip, + struct pwm_device *pwm) { struct stmpe_pwm *stmpe_pwm = to_stmpe_pwm(chip); u8 value; @@ -72,17 +72,16 @@ static void stmpe_24xx_pwm_disable(struct pwm_chip *chip, if (ret < 0) { dev_err(chip->dev, "error reading PWM#%u control\n", pwm->hwpwm); - return; + return ret; } value = ret & ~BIT(pwm->hwpwm); ret = stmpe_reg_write(stmpe_pwm->stmpe, STMPE24XX_PWMCS, value); - if (ret) { + if (ret) dev_err(chip->dev, "error writing PWM#%u control\n", pwm->hwpwm); - return; - } + return ret; } /* STMPE 24xx PWM instructions */ @@ -111,7 +110,9 @@ static int stmpe_24xx_pwm_config(struct pwm_chip *chip, struct pwm_device *pwm, /* Make sure we are disabled */ if (pwm_is_enabled(pwm)) { - stmpe_24xx_pwm_disable(chip, pwm); + ret = stmpe_24xx_pwm_disable(chip, pwm); + if (ret) + return ret; } else { /* Connect the PWM to the pin */ pin = pwm->hwpwm; @@ -269,7 +270,7 @@ static int stmpe_24xx_pwm_apply(struct pwm_chip *chip, struct pwm_device *pwm, if (!state->enabled) { if (pwm->state.enabled) - stmpe_24xx_pwm_disable(chip, pwm); + return stmpe_24xx_pwm_disable(chip, pwm); return 0; } diff --git a/drivers/pwm/pwm-sun4i.c b/drivers/pwm/pwm-sun4i.c index a8790a8fc53e..c84fcf1a13dc 100644 --- a/drivers/pwm/pwm-sun4i.c +++ b/drivers/pwm/pwm-sun4i.c @@ -17,7 +17,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/pwm/pwm-sunplus.c b/drivers/pwm/pwm-sunplus.c index d6ebe9f03b35..7705c7b86c3a 100644 --- a/drivers/pwm/pwm-sunplus.c +++ b/drivers/pwm/pwm-sunplus.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/pwm/pwm-tegra.c b/drivers/pwm/pwm-tegra.c index 5810abf66e2a..a169a34e0778 100644 --- a/drivers/pwm/pwm-tegra.c +++ b/drivers/pwm/pwm-tegra.c @@ -41,7 +41,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/pwm/pwm-tiecap.c b/drivers/pwm/pwm-tiecap.c index 109449956307..8c94b266c1b2 100644 --- a/drivers/pwm/pwm-tiecap.c +++ b/drivers/pwm/pwm-tiecap.c @@ -12,7 +12,7 @@ #include #include #include -#include +#include /* ECAP registers and bits definitions */ #define CAP1 0x08 diff --git a/drivers/pwm/pwm-tiehrpwm.c b/drivers/pwm/pwm-tiehrpwm.c index bb3959ace6b4..ecbfd7e954ec 100644 --- a/drivers/pwm/pwm-tiehrpwm.c +++ b/drivers/pwm/pwm-tiehrpwm.c @@ -12,7 +12,7 @@ #include #include #include -#include +#include /* EHRPWM registers and bits definitions */ diff --git a/drivers/pwm/pwm-visconti.c b/drivers/pwm/pwm-visconti.c index e3fb79b3e2a7..7f7591a2384c 100644 --- a/drivers/pwm/pwm-visconti.c +++ b/drivers/pwm/pwm-visconti.c @@ -21,7 +21,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/pwm/pwm-vt8500.c b/drivers/pwm/pwm-vt8500.c index d2c48fd98706..6d46db51daac 100644 --- a/drivers/pwm/pwm-vt8500.c +++ b/drivers/pwm/pwm-vt8500.c @@ -6,6 +6,7 @@ * Copyright (C) 2010 Alexey Charkov */ +#include #include #include #include @@ -18,10 +19,6 @@ #include -#include -#include -#include - /* * SoC architecture allocates register space for 4 PWMs but only * 2 are currently implemented. diff --git a/drivers/regulator/tps6287x-regulator.c b/drivers/regulator/tps6287x-regulator.c index d022184a8e7d..9b7c3d77789e 100644 --- a/drivers/regulator/tps6287x-regulator.c +++ b/drivers/regulator/tps6287x-regulator.c @@ -119,7 +119,7 @@ static struct regulator_desc tps6287x_reg = { .ramp_mask = TPS6287X_CTRL1_VRAMP, .ramp_delay_table = tps6287x_ramp_table, .n_ramp_values = ARRAY_SIZE(tps6287x_ramp_table), - .n_voltages = 256, + .n_voltages = 256 * ARRAY_SIZE(tps6287x_voltage_ranges), .linear_ranges = tps6287x_voltage_ranges, .n_linear_ranges = ARRAY_SIZE(tps6287x_voltage_ranges), .linear_range_selectors_bitfield = tps6287x_voltage_range_sel, diff --git a/drivers/regulator/tps6594-regulator.c b/drivers/regulator/tps6594-regulator.c index 25ef102c8270..b7f0c8779757 100644 --- a/drivers/regulator/tps6594-regulator.c +++ b/drivers/regulator/tps6594-regulator.c @@ -384,21 +384,19 @@ static int tps6594_request_reg_irqs(struct platform_device *pdev, if (irq < 0) return -EINVAL; - irq_data[*irq_idx + j].dev = tps->dev; - irq_data[*irq_idx + j].type = irq_type; - irq_data[*irq_idx + j].rdev = rdev; + irq_data[*irq_idx].dev = tps->dev; + irq_data[*irq_idx].type = irq_type; + irq_data[*irq_idx].rdev = rdev; error = devm_request_threaded_irq(tps->dev, irq, NULL, - tps6594_regulator_irq_handler, - IRQF_ONESHOT, - irq_type->irq_name, - &irq_data[*irq_idx]); - (*irq_idx)++; + tps6594_regulator_irq_handler, IRQF_ONESHOT, + irq_type->irq_name, &irq_data[*irq_idx]); if (error) { dev_err(tps->dev, "tps6594 failed to request %s IRQ %d: %d\n", irq_type->irq_name, irq, error); return error; } + (*irq_idx)++; } return 0; } @@ -420,8 +418,8 @@ static int tps6594_regulator_probe(struct platform_device *pdev) int error, i, irq, multi, delta; int irq_idx = 0; int buck_idx = 0; - int ext_reg_irq_nb = 2; - + size_t ext_reg_irq_nb = 2; + size_t reg_irq_nb; enum { MULTI_BUCK12, MULTI_BUCK123, @@ -484,15 +482,16 @@ static int tps6594_regulator_probe(struct platform_device *pdev) } } - if (tps->chip_id == LP8764) + if (tps->chip_id == LP8764) { /* There is only 4 buck on LP8764 */ buck_configured[4] = 1; + reg_irq_nb = size_mul(REGS_INT_NB, (BUCK_NB - 1)); + } else { + reg_irq_nb = size_mul(REGS_INT_NB, (size_add(BUCK_NB, LDO_NB))); + } - irq_data = devm_kmalloc_array(tps->dev, - REGS_INT_NB * sizeof(struct tps6594_regulator_irq_data), - ARRAY_SIZE(tps6594_bucks_irq_types) + - ARRAY_SIZE(tps6594_ldos_irq_types), - GFP_KERNEL); + irq_data = devm_kmalloc_array(tps->dev, reg_irq_nb, + sizeof(struct tps6594_regulator_irq_data), GFP_KERNEL); if (!irq_data) return -ENOMEM; diff --git a/drivers/rtc/Kconfig b/drivers/rtc/Kconfig index 05f4b2d66290..d7502433c78a 100644 --- a/drivers/rtc/Kconfig +++ b/drivers/rtc/Kconfig @@ -904,9 +904,9 @@ config RTC_DRV_PCF2127 select REGMAP_SPI if SPI_MASTER select WATCHDOG_CORE if WATCHDOG help - If you say yes here you get support for the NXP PCF2127/29 RTC + If you say yes here you get support for the NXP PCF2127/29/31 RTC chips with integrated quartz crystal for industrial applications. - Both chips also have watchdog timer and tamper switch detection + These chips also have watchdog timer and tamper switch detection features. PCF2127 has an additional feature of 512 bytes battery backed @@ -1196,6 +1196,7 @@ config RTC_DRV_MSM6242 config RTC_DRV_BQ4802 tristate "TI BQ4802" depends on HAS_IOMEM && HAS_IOPORT + depends on SPARC || COMPILE_TEST help If you say Y here you will get support for the TI BQ4802 RTC chip. diff --git a/drivers/rtc/interface.c b/drivers/rtc/interface.c index 499d89150afc..1b63111cdda2 100644 --- a/drivers/rtc/interface.c +++ b/drivers/rtc/interface.c @@ -376,7 +376,7 @@ int __rtc_read_alarm(struct rtc_device *rtc, struct rtc_wkalrm *alarm) err = rtc_valid_tm(&alarm->time); done: - if (err) + if (err && alarm->enabled) dev_warn(&rtc->dev, "invalid alarm value: %ptR\n", &alarm->time); diff --git a/drivers/rtc/rtc-abx80x.c b/drivers/rtc/rtc-abx80x.c index e08d3181bd2a..fde2b8054c2e 100644 --- a/drivers/rtc/rtc-abx80x.c +++ b/drivers/rtc/rtc-abx80x.c @@ -15,7 +15,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/rtc/rtc-armada38x.c b/drivers/rtc/rtc-armada38x.c index b4139c200676..569c1054d6b0 100644 --- a/drivers/rtc/rtc-armada38x.c +++ b/drivers/rtc/rtc-armada38x.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include @@ -474,7 +473,6 @@ static const struct armada38x_rtc_data armada8k_data = { .alarm = ALARM2, }; -#ifdef CONFIG_OF static const struct of_device_id armada38x_rtc_of_match_table[] = { { .compatible = "marvell,armada-380-rtc", @@ -487,7 +485,6 @@ static const struct of_device_id armada38x_rtc_of_match_table[] = { {} }; MODULE_DEVICE_TABLE(of, armada38x_rtc_of_match_table); -#endif static __init int armada38x_rtc_probe(struct platform_device *pdev) { @@ -577,7 +574,7 @@ static struct platform_driver armada38x_rtc_driver = { .driver = { .name = "armada38x-rtc", .pm = &armada38x_rtc_pm_ops, - .of_match_table = of_match_ptr(armada38x_rtc_of_match_table), + .of_match_table = armada38x_rtc_of_match_table, }, }; diff --git a/drivers/rtc/rtc-aspeed.c b/drivers/rtc/rtc-aspeed.c index a93352ed3aec..880b015eebaf 100644 --- a/drivers/rtc/rtc-aspeed.c +++ b/drivers/rtc/rtc-aspeed.c @@ -118,7 +118,7 @@ MODULE_DEVICE_TABLE(of, aspeed_rtc_match); static struct platform_driver aspeed_rtc_driver = { .driver = { .name = "aspeed-rtc", - .of_match_table = of_match_ptr(aspeed_rtc_match), + .of_match_table = aspeed_rtc_match, }, }; diff --git a/drivers/rtc/rtc-at91rm9200.c b/drivers/rtc/rtc-at91rm9200.c index e9d17232d0a8..add4f71d7b3b 100644 --- a/drivers/rtc/rtc-at91rm9200.c +++ b/drivers/rtc/rtc-at91rm9200.c @@ -22,7 +22,6 @@ #include #include #include -#include #include #include #include @@ -642,7 +641,7 @@ static struct platform_driver at91_rtc_driver = { .driver = { .name = "at91_rtc", .pm = &at91_rtc_pm_ops, - .of_match_table = of_match_ptr(at91_rtc_dt_ids), + .of_match_table = at91_rtc_dt_ids, }, }; diff --git a/drivers/rtc/rtc-at91sam9.c b/drivers/rtc/rtc-at91sam9.c index 610f27dfc462..f93bee96e362 100644 --- a/drivers/rtc/rtc-at91sam9.c +++ b/drivers/rtc/rtc-at91sam9.c @@ -534,7 +534,7 @@ static struct platform_driver at91_rtc_driver = { .driver = { .name = "rtc-at91sam9", .pm = &at91_rtc_pm_ops, - .of_match_table = of_match_ptr(at91_rtc_dt_ids), + .of_match_table = at91_rtc_dt_ids, }, }; diff --git a/drivers/rtc/rtc-cmos.c b/drivers/rtc/rtc-cmos.c index c9416fe8542d..228fb2d11c70 100644 --- a/drivers/rtc/rtc-cmos.c +++ b/drivers/rtc/rtc-cmos.c @@ -913,6 +913,10 @@ static inline void cmos_check_acpi_rtc_status(struct device *dev, #define INITSECTION __init #endif +#define SECS_PER_DAY (24 * 60 * 60) +#define SECS_PER_MONTH (28 * SECS_PER_DAY) +#define SECS_PER_YEAR (365 * SECS_PER_DAY) + static int INITSECTION cmos_do_probe(struct device *dev, struct resource *ports, int rtc_irq) { @@ -1019,6 +1023,13 @@ cmos_do_probe(struct device *dev, struct resource *ports, int rtc_irq) goto cleanup0; } + if (cmos_rtc.mon_alrm) + cmos_rtc.rtc->alarm_offset_max = SECS_PER_YEAR - 1; + else if (cmos_rtc.day_alrm) + cmos_rtc.rtc->alarm_offset_max = SECS_PER_MONTH - 1; + else + cmos_rtc.rtc->alarm_offset_max = SECS_PER_DAY - 1; + rename_region(ports, dev_name(&cmos_rtc.rtc->dev)); if (!mc146818_does_rtc_work()) { diff --git a/drivers/rtc/rtc-cros-ec.c b/drivers/rtc/rtc-cros-ec.c index 998ab8606f0b..0cd397c04ff0 100644 --- a/drivers/rtc/rtc-cros-ec.c +++ b/drivers/rtc/rtc-cros-ec.c @@ -182,21 +182,15 @@ static int cros_ec_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) ret = cros_ec_rtc_set(cros_ec, EC_CMD_RTC_SET_ALARM, alarm_offset); if (ret < 0) { - if (ret == -EINVAL && alarm_offset >= SECS_PER_DAY) { - /* - * RTC chips on some older Chromebooks can only handle - * alarms up to 24h in the future. Try to set an alarm - * below that limit to avoid suspend failures. - */ - ret = cros_ec_rtc_set(cros_ec, EC_CMD_RTC_SET_ALARM, - SECS_PER_DAY - 1); - } - - if (ret < 0) { - dev_err(dev, "error setting alarm in %u seconds: %d\n", - alarm_offset, ret); - return ret; - } + dev_err(dev, "error setting alarm in %u seconds: %d\n", + alarm_offset, ret); + /* + * The EC code returns -EINVAL if the alarm time is too + * far in the future. Convert it to the expected error code. + */ + if (ret == -EINVAL) + ret = -ERANGE; + return ret; } return 0; @@ -355,6 +349,20 @@ static int cros_ec_rtc_probe(struct platform_device *pdev) cros_ec_rtc->rtc->ops = &cros_ec_rtc_ops; cros_ec_rtc->rtc->range_max = U32_MAX; + /* + * The RTC on some older Chromebooks can only handle alarms less than + * 24 hours in the future. The only way to find out is to try to set an + * alarm further in the future. If that fails, assume that the RTC + * connected to the EC can only handle less than 24 hours of alarm + * window. + */ + ret = cros_ec_rtc_set(cros_ec, EC_CMD_RTC_SET_ALARM, SECS_PER_DAY * 2); + if (ret == -EINVAL) + cros_ec_rtc->rtc->alarm_offset_max = SECS_PER_DAY - 1; + + (void)cros_ec_rtc_set(cros_ec, EC_CMD_RTC_SET_ALARM, + EC_RTC_ALARM_CLEAR); + ret = devm_rtc_register_device(cros_ec_rtc->rtc); if (ret) return ret; diff --git a/drivers/rtc/rtc-da9063.c b/drivers/rtc/rtc-da9063.c index ee2efb496174..2f5d60622564 100644 --- a/drivers/rtc/rtc-da9063.c +++ b/drivers/rtc/rtc-da9063.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -496,6 +497,12 @@ static int da9063_rtc_probe(struct platform_device *pdev) dev_err(&pdev->dev, "Failed to request ALARM IRQ %d: %d\n", irq_alarm, ret); + ret = dev_pm_set_wake_irq(&pdev->dev, irq_alarm); + if (ret) + dev_warn(&pdev->dev, + "Failed to set IRQ %d as a wake IRQ: %d\n", + irq_alarm, ret); + device_init_wakeup(&pdev->dev, true); return devm_rtc_register_device(rtc->rtc_dev); diff --git a/drivers/rtc/rtc-ds1305.c b/drivers/rtc/rtc-ds1305.c index ed9360486953..d4de401548b4 100644 --- a/drivers/rtc/rtc-ds1305.c +++ b/drivers/rtc/rtc-ds1305.c @@ -336,8 +336,8 @@ static int ds1305_set_alarm(struct device *dev, struct rtc_wkalrm *alm) /* make sure alarm fires within the next 24 hours */ if (later <= now) return -EINVAL; - if ((later - now) > 24 * 60 * 60) - return -EDOM; + if ((later - now) > ds1305->rtc->alarm_offset_max) + return -ERANGE; /* disable alarm if needed */ if (ds1305->ctrl[0] & DS1305_AEI0) { @@ -691,6 +691,7 @@ static int ds1305_probe(struct spi_device *spi) ds1305->rtc->ops = &ds1305_ops; ds1305->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000; ds1305->rtc->range_max = RTC_TIMESTAMP_END_2099; + ds1305->rtc->alarm_offset_max = 24 * 60 * 60; ds1305_nvmem_cfg.priv = ds1305; status = devm_rtc_register_device(ds1305->rtc); diff --git a/drivers/rtc/rtc-ds1307.c b/drivers/rtc/rtc-ds1307.c index cb5acecc11aa..506b7d1c2397 100644 --- a/drivers/rtc/rtc-ds1307.c +++ b/drivers/rtc/rtc-ds1307.c @@ -1744,7 +1744,7 @@ static int ds1307_probe(struct i2c_client *client) match = device_get_match_data(&client->dev); if (match) { - ds1307->type = (enum ds_type)match; + ds1307->type = (uintptr_t)match; chip = &chips[ds1307->type]; } else if (id) { chip = &chips[id->driver_data]; diff --git a/drivers/rtc/rtc-ds1742.c b/drivers/rtc/rtc-ds1742.c index a5026b0514e7..6ae8b9a294fe 100644 --- a/drivers/rtc/rtc-ds1742.c +++ b/drivers/rtc/rtc-ds1742.c @@ -16,7 +16,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/rtc/rtc-ds2404.c b/drivers/rtc/rtc-ds2404.c index 0480f592307e..3231fd9f61da 100644 --- a/drivers/rtc/rtc-ds2404.c +++ b/drivers/rtc/rtc-ds2404.c @@ -7,9 +7,8 @@ #include #include #include -#include #include -#include +#include #include #include @@ -27,164 +26,140 @@ #define DS2404_CLK 1 #define DS2404_DQ 2 -struct ds2404_gpio { - const char *name; - unsigned int gpio; -}; - struct ds2404 { - struct ds2404_gpio *gpio; + struct device *dev; + struct gpio_desc *rst_gpiod; + struct gpio_desc *clk_gpiod; + struct gpio_desc *dq_gpiod; struct rtc_device *rtc; }; -static struct ds2404_gpio ds2404_gpio[] = { - { "RTC RST", 0 }, - { "RTC CLK", 0 }, - { "RTC DQ", 0 }, -}; - -static int ds2404_gpio_map(struct ds2404 *chip, struct platform_device *pdev, - struct ds2404_platform_data *pdata) +static int ds2404_gpio_map(struct ds2404 *chip, struct platform_device *pdev) { - int i, err; + struct device *dev = &pdev->dev; - ds2404_gpio[DS2404_RST].gpio = pdata->gpio_rst; - ds2404_gpio[DS2404_CLK].gpio = pdata->gpio_clk; - ds2404_gpio[DS2404_DQ].gpio = pdata->gpio_dq; + /* This will de-assert RESET, declare this GPIO as GPIOD_ACTIVE_LOW */ + chip->rst_gpiod = devm_gpiod_get(dev, "rst", GPIOD_OUT_LOW); + if (IS_ERR(chip->rst_gpiod)) + return PTR_ERR(chip->rst_gpiod); - for (i = 0; i < ARRAY_SIZE(ds2404_gpio); i++) { - err = gpio_request(ds2404_gpio[i].gpio, ds2404_gpio[i].name); - if (err) { - dev_err(&pdev->dev, "error mapping gpio %s: %d\n", - ds2404_gpio[i].name, err); - goto err_request; - } - if (i != DS2404_DQ) - gpio_direction_output(ds2404_gpio[i].gpio, 1); - } + chip->clk_gpiod = devm_gpiod_get(dev, "clk", GPIOD_OUT_HIGH); + if (IS_ERR(chip->clk_gpiod)) + return PTR_ERR(chip->clk_gpiod); + + chip->dq_gpiod = devm_gpiod_get(dev, "dq", GPIOD_ASIS); + if (IS_ERR(chip->dq_gpiod)) + return PTR_ERR(chip->dq_gpiod); - chip->gpio = ds2404_gpio; return 0; - -err_request: - while (--i >= 0) - gpio_free(ds2404_gpio[i].gpio); - return err; } -static void ds2404_gpio_unmap(void *data) +static void ds2404_reset(struct ds2404 *chip) { - int i; - - for (i = 0; i < ARRAY_SIZE(ds2404_gpio); i++) - gpio_free(ds2404_gpio[i].gpio); -} - -static void ds2404_reset(struct device *dev) -{ - gpio_set_value(ds2404_gpio[DS2404_RST].gpio, 0); + gpiod_set_value(chip->rst_gpiod, 1); udelay(1000); - gpio_set_value(ds2404_gpio[DS2404_RST].gpio, 1); - gpio_set_value(ds2404_gpio[DS2404_CLK].gpio, 0); - gpio_direction_output(ds2404_gpio[DS2404_DQ].gpio, 0); + gpiod_set_value(chip->rst_gpiod, 0); + gpiod_set_value(chip->clk_gpiod, 0); + gpiod_direction_output(chip->dq_gpiod, 0); udelay(10); } -static void ds2404_write_byte(struct device *dev, u8 byte) +static void ds2404_write_byte(struct ds2404 *chip, u8 byte) { int i; - gpio_direction_output(ds2404_gpio[DS2404_DQ].gpio, 1); + gpiod_direction_output(chip->dq_gpiod, 1); for (i = 0; i < 8; i++) { - gpio_set_value(ds2404_gpio[DS2404_DQ].gpio, byte & (1 << i)); + gpiod_set_value(chip->dq_gpiod, byte & (1 << i)); udelay(10); - gpio_set_value(ds2404_gpio[DS2404_CLK].gpio, 1); + gpiod_set_value(chip->clk_gpiod, 1); udelay(10); - gpio_set_value(ds2404_gpio[DS2404_CLK].gpio, 0); + gpiod_set_value(chip->clk_gpiod, 0); udelay(10); } } -static u8 ds2404_read_byte(struct device *dev) +static u8 ds2404_read_byte(struct ds2404 *chip) { int i; u8 ret = 0; - gpio_direction_input(ds2404_gpio[DS2404_DQ].gpio); + gpiod_direction_input(chip->dq_gpiod); for (i = 0; i < 8; i++) { - gpio_set_value(ds2404_gpio[DS2404_CLK].gpio, 0); + gpiod_set_value(chip->clk_gpiod, 0); udelay(10); - if (gpio_get_value(ds2404_gpio[DS2404_DQ].gpio)) + if (gpiod_get_value(chip->dq_gpiod)) ret |= 1 << i; - gpio_set_value(ds2404_gpio[DS2404_CLK].gpio, 1); + gpiod_set_value(chip->clk_gpiod, 1); udelay(10); } return ret; } -static void ds2404_read_memory(struct device *dev, u16 offset, +static void ds2404_read_memory(struct ds2404 *chip, u16 offset, int length, u8 *out) { - ds2404_reset(dev); - ds2404_write_byte(dev, DS2404_READ_MEMORY_CMD); - ds2404_write_byte(dev, offset & 0xff); - ds2404_write_byte(dev, (offset >> 8) & 0xff); + ds2404_reset(chip); + ds2404_write_byte(chip, DS2404_READ_MEMORY_CMD); + ds2404_write_byte(chip, offset & 0xff); + ds2404_write_byte(chip, (offset >> 8) & 0xff); while (length--) - *out++ = ds2404_read_byte(dev); + *out++ = ds2404_read_byte(chip); } -static void ds2404_write_memory(struct device *dev, u16 offset, +static void ds2404_write_memory(struct ds2404 *chip, u16 offset, int length, u8 *out) { int i; u8 ta01, ta02, es; - ds2404_reset(dev); - ds2404_write_byte(dev, DS2404_WRITE_SCRATCHPAD_CMD); - ds2404_write_byte(dev, offset & 0xff); - ds2404_write_byte(dev, (offset >> 8) & 0xff); + ds2404_reset(chip); + ds2404_write_byte(chip, DS2404_WRITE_SCRATCHPAD_CMD); + ds2404_write_byte(chip, offset & 0xff); + ds2404_write_byte(chip, (offset >> 8) & 0xff); for (i = 0; i < length; i++) - ds2404_write_byte(dev, out[i]); + ds2404_write_byte(chip, out[i]); - ds2404_reset(dev); - ds2404_write_byte(dev, DS2404_READ_SCRATCHPAD_CMD); + ds2404_reset(chip); + ds2404_write_byte(chip, DS2404_READ_SCRATCHPAD_CMD); - ta01 = ds2404_read_byte(dev); - ta02 = ds2404_read_byte(dev); - es = ds2404_read_byte(dev); + ta01 = ds2404_read_byte(chip); + ta02 = ds2404_read_byte(chip); + es = ds2404_read_byte(chip); for (i = 0; i < length; i++) { - if (out[i] != ds2404_read_byte(dev)) { - dev_err(dev, "read invalid data\n"); + if (out[i] != ds2404_read_byte(chip)) { + dev_err(chip->dev, "read invalid data\n"); return; } } - ds2404_reset(dev); - ds2404_write_byte(dev, DS2404_COPY_SCRATCHPAD_CMD); - ds2404_write_byte(dev, ta01); - ds2404_write_byte(dev, ta02); - ds2404_write_byte(dev, es); + ds2404_reset(chip); + ds2404_write_byte(chip, DS2404_COPY_SCRATCHPAD_CMD); + ds2404_write_byte(chip, ta01); + ds2404_write_byte(chip, ta02); + ds2404_write_byte(chip, es); - gpio_direction_input(ds2404_gpio[DS2404_DQ].gpio); - while (gpio_get_value(ds2404_gpio[DS2404_DQ].gpio)) + while (gpiod_get_value(chip->dq_gpiod)) ; } -static void ds2404_enable_osc(struct device *dev) +static void ds2404_enable_osc(struct ds2404 *chip) { u8 in[1] = { 0x10 }; /* enable oscillator */ - ds2404_write_memory(dev, 0x201, 1, in); + + ds2404_write_memory(chip, 0x201, 1, in); } static int ds2404_read_time(struct device *dev, struct rtc_time *dt) { + struct ds2404 *chip = dev_get_drvdata(dev); unsigned long time = 0; __le32 hw_time = 0; - ds2404_read_memory(dev, 0x203, 4, (u8 *)&hw_time); + ds2404_read_memory(chip, 0x203, 4, (u8 *)&hw_time); time = le32_to_cpu(hw_time); rtc_time64_to_tm(time, dt); @@ -193,8 +168,9 @@ static int ds2404_read_time(struct device *dev, struct rtc_time *dt) static int ds2404_set_time(struct device *dev, struct rtc_time *dt) { + struct ds2404 *chip = dev_get_drvdata(dev); u32 time = cpu_to_le32(rtc_tm_to_time64(dt)); - ds2404_write_memory(dev, 0x203, 4, (u8 *)&time); + ds2404_write_memory(chip, 0x203, 4, (u8 *)&time); return 0; } @@ -205,7 +181,6 @@ static const struct rtc_class_ops ds2404_rtc_ops = { static int rtc_probe(struct platform_device *pdev) { - struct ds2404_platform_data *pdata = dev_get_platdata(&pdev->dev); struct ds2404 *chip; int retval = -EBUSY; @@ -213,22 +188,16 @@ static int rtc_probe(struct platform_device *pdev) if (!chip) return -ENOMEM; + chip->dev = &pdev->dev; + chip->rtc = devm_rtc_allocate_device(&pdev->dev); if (IS_ERR(chip->rtc)) return PTR_ERR(chip->rtc); - retval = ds2404_gpio_map(chip, pdev, pdata); + retval = ds2404_gpio_map(chip, pdev); if (retval) return retval; - retval = devm_add_action_or_reset(&pdev->dev, ds2404_gpio_unmap, chip); - if (retval) - return retval; - - dev_info(&pdev->dev, "using GPIOs RST:%d, CLK:%d, DQ:%d\n", - chip->gpio[DS2404_RST].gpio, chip->gpio[DS2404_CLK].gpio, - chip->gpio[DS2404_DQ].gpio); - platform_set_drvdata(pdev, chip); chip->rtc->ops = &ds2404_rtc_ops; @@ -238,7 +207,7 @@ static int rtc_probe(struct platform_device *pdev) if (retval) return retval; - ds2404_enable_osc(&pdev->dev); + ds2404_enable_osc(chip); return 0; } diff --git a/drivers/rtc/rtc-fsl-ftm-alarm.c b/drivers/rtc/rtc-fsl-ftm-alarm.c index 3d7c4077fe1c..a72c4ad0cec6 100644 --- a/drivers/rtc/rtc-fsl-ftm-alarm.c +++ b/drivers/rtc/rtc-fsl-ftm-alarm.c @@ -11,11 +11,8 @@ #include #include #include -#include -#include #include -#include -#include +#include #include #include #include diff --git a/drivers/rtc/rtc-isl12022.c b/drivers/rtc/rtc-isl12022.c index a613257d1574..4eef7afcc8bc 100644 --- a/drivers/rtc/rtc-isl12022.c +++ b/drivers/rtc/rtc-isl12022.c @@ -9,6 +9,8 @@ */ #include +#include +#include #include #include #include @@ -31,6 +33,8 @@ #define ISL12022_REG_SR 0x07 #define ISL12022_REG_INT 0x08 +#define ISL12022_REG_PWR_VBAT 0x0a + #define ISL12022_REG_BETA 0x0d #define ISL12022_REG_TEMP_L 0x28 @@ -41,6 +45,12 @@ #define ISL12022_SR_LBAT75 (1 << 1) #define ISL12022_INT_WRTC (1 << 6) +#define ISL12022_INT_FO_MASK GENMASK(3, 0) +#define ISL12022_INT_FO_OFF 0x0 +#define ISL12022_INT_FO_32K 0x1 + +#define ISL12022_REG_VB85_MASK GENMASK(5, 3) +#define ISL12022_REG_VB75_MASK GENMASK(2, 0) #define ISL12022_BETA_TSE (1 << 7) @@ -141,12 +151,6 @@ static int isl12022_rtc_read_time(struct device *dev, struct rtc_time *tm) if (ret) return ret; - if (buf[ISL12022_REG_SR] & (ISL12022_SR_LBAT85 | ISL12022_SR_LBAT75)) { - dev_warn(dev, - "voltage dropped below %u%%, date and time is not reliable.\n", - buf[ISL12022_REG_SR] & ISL12022_SR_LBAT85 ? 85 : 75); - } - dev_dbg(dev, "raw data is sec=%02x, min=%02x, hr=%02x, mday=%02x, mon=%02x, year=%02x, wday=%02x, sr=%02x, int=%02x", buf[ISL12022_REG_SC], @@ -204,7 +208,34 @@ static int isl12022_rtc_set_time(struct device *dev, struct rtc_time *tm) return regmap_bulk_write(regmap, ISL12022_REG_SC, buf, sizeof(buf)); } +static int isl12022_rtc_ioctl(struct device *dev, unsigned int cmd, unsigned long arg) +{ + struct regmap *regmap = dev_get_drvdata(dev); + u32 user, val; + int ret; + + switch (cmd) { + case RTC_VL_READ: + ret = regmap_read(regmap, ISL12022_REG_SR, &val); + if (ret) + return ret; + + user = 0; + if (val & ISL12022_SR_LBAT85) + user |= RTC_VL_BACKUP_LOW; + + if (val & ISL12022_SR_LBAT75) + user |= RTC_VL_BACKUP_EMPTY; + + return put_user(user, (u32 __user *)arg); + + default: + return -ENOIOCTLCMD; + } +} + static const struct rtc_class_ops isl12022_rtc_ops = { + .ioctl = isl12022_rtc_ioctl, .read_time = isl12022_rtc_read_time, .set_time = isl12022_rtc_set_time, }; @@ -215,10 +246,88 @@ static const struct regmap_config regmap_config = { .use_single_write = true, }; +static int isl12022_register_clock(struct device *dev) +{ + struct regmap *regmap = dev_get_drvdata(dev); + struct clk_hw *hw; + int ret; + + if (!device_property_present(dev, "#clock-cells")) { + /* + * Disabling the F_OUT pin reduces the power + * consumption in battery mode by ~25%. + */ + regmap_update_bits(regmap, ISL12022_REG_INT, ISL12022_INT_FO_MASK, + ISL12022_INT_FO_OFF); + + return 0; + } + + if (!IS_ENABLED(CONFIG_COMMON_CLK)) + return 0; + + /* + * For now, only support a fixed clock of 32768Hz (the reset default). + */ + ret = regmap_update_bits(regmap, ISL12022_REG_INT, + ISL12022_INT_FO_MASK, ISL12022_INT_FO_32K); + if (ret) + return ret; + + hw = devm_clk_hw_register_fixed_rate(dev, "isl12022", NULL, 0, 32768); + if (IS_ERR(hw)) + return PTR_ERR(hw); + + return devm_of_clk_add_hw_provider(dev, of_clk_hw_simple_get, hw); +} + +static const u32 trip_levels[2][7] = { + { 2125000, 2295000, 2550000, 2805000, 3060000, 4250000, 4675000 }, + { 1875000, 2025000, 2250000, 2475000, 2700000, 3750000, 4125000 }, +}; + +static void isl12022_set_trip_levels(struct device *dev) +{ + struct regmap *regmap = dev_get_drvdata(dev); + u32 levels[2] = {0, 0}; + int ret, i, j, x[2]; + u8 val, mask; + + device_property_read_u32_array(dev, "isil,battery-trip-levels-microvolt", + levels, 2); + + for (i = 0; i < 2; i++) { + for (j = 0; j < ARRAY_SIZE(trip_levels[i]) - 1; j++) { + if (levels[i] <= trip_levels[i][j]) + break; + } + x[i] = j; + } + + val = FIELD_PREP(ISL12022_REG_VB85_MASK, x[0]) | + FIELD_PREP(ISL12022_REG_VB75_MASK, x[1]); + mask = ISL12022_REG_VB85_MASK | ISL12022_REG_VB75_MASK; + + ret = regmap_update_bits(regmap, ISL12022_REG_PWR_VBAT, mask, val); + if (ret) + dev_warn(dev, "unable to set battery alarm levels: %d\n", ret); + + /* + * Force a write of the TSE bit in the BETA register, in order + * to trigger an update of the LBAT75 and LBAT85 bits in the + * status register. In battery backup mode, those bits have + * another meaning, so without this, they may contain stale + * values for up to a minute after power-on. + */ + regmap_write_bits(regmap, ISL12022_REG_BETA, + ISL12022_BETA_TSE, ISL12022_BETA_TSE); +} + static int isl12022_probe(struct i2c_client *client) { struct rtc_device *rtc; struct regmap *regmap; + int ret; if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) return -ENODEV; @@ -231,6 +340,11 @@ static int isl12022_probe(struct i2c_client *client) dev_set_drvdata(&client->dev, regmap); + ret = isl12022_register_clock(&client->dev); + if (ret) + return ret; + + isl12022_set_trip_levels(&client->dev); isl12022_hwmon_register(&client->dev); rtc = devm_rtc_allocate_device(&client->dev); diff --git a/drivers/rtc/rtc-isl12026.c b/drivers/rtc/rtc-isl12026.c index 5abff5d348ac..2aabb9151d4c 100644 --- a/drivers/rtc/rtc-isl12026.c +++ b/drivers/rtc/rtc-isl12026.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include @@ -429,7 +428,7 @@ static void isl12026_force_power_modes(struct i2c_client *client) } } -static int isl12026_probe_new(struct i2c_client *client) +static int isl12026_probe(struct i2c_client *client) { struct isl12026 *priv; int ret; @@ -490,7 +489,7 @@ static struct i2c_driver isl12026_driver = { .name = "rtc-isl12026", .of_match_table = isl12026_dt_match, }, - .probe = isl12026_probe_new, + .probe = isl12026_probe, .remove = isl12026_remove, }; diff --git a/drivers/rtc/rtc-isl1208.c b/drivers/rtc/rtc-isl1208.c index b0712b4e3648..e50c23ee1646 100644 --- a/drivers/rtc/rtc-isl1208.c +++ b/drivers/rtc/rtc-isl1208.c @@ -9,7 +9,7 @@ #include #include #include -#include +#include #include #include @@ -188,7 +188,7 @@ isl1208_i2c_validate_client(struct i2c_client *client) static int isl1208_set_xtoscb(struct i2c_client *client, int sr, int xtosb_val) { /* Do nothing if bit is already set to desired value */ - if ((sr & ISL1208_REG_SR_XTOSCB) == xtosb_val) + if (!!(sr & ISL1208_REG_SR_XTOSCB) == xtosb_val) return 0; if (xtosb_val) @@ -862,17 +862,9 @@ isl1208_probe(struct i2c_client *client) i2c_set_clientdata(client, isl1208); /* Determine which chip we have */ - if (client->dev.of_node) { - isl1208->config = of_device_get_match_data(&client->dev); - if (!isl1208->config) - return -ENODEV; - } else { - const struct i2c_device_id *id = i2c_match_id(isl1208_id, client); - - if (!id) - return -ENODEV; - isl1208->config = (struct isl1208_config *)id->driver_data; - } + isl1208->config = i2c_get_match_data(client); + if (!isl1208->config) + return -ENODEV; rc = isl1208_clk_present(client, "xin"); if (rc < 0) @@ -952,7 +944,6 @@ isl1208_probe(struct i2c_client *client) rc = isl1208_setup_irq(client, client->irq); if (rc) return rc; - } else { clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, isl1208->rtc->features); } diff --git a/drivers/rtc/rtc-jz4740.c b/drivers/rtc/rtc-jz4740.c index 36453b008139..bafa7d1b9b88 100644 --- a/drivers/rtc/rtc-jz4740.c +++ b/drivers/rtc/rtc-jz4740.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include @@ -349,7 +349,7 @@ static int jz4740_rtc_probe(struct platform_device *pdev) if (!rtc) return -ENOMEM; - rtc->type = (enum jz4740_rtc_type)device_get_match_data(dev); + rtc->type = (uintptr_t)device_get_match_data(dev); irq = platform_get_irq(pdev, 0); if (irq < 0) diff --git a/drivers/rtc/rtc-lpc24xx.c b/drivers/rtc/rtc-lpc24xx.c index a4612e543f35..df17c48ff086 100644 --- a/drivers/rtc/rtc-lpc24xx.c +++ b/drivers/rtc/rtc-lpc24xx.c @@ -9,9 +9,8 @@ #include #include #include +#include #include -#include -#include #include #include diff --git a/drivers/rtc/rtc-m41t80.c b/drivers/rtc/rtc-m41t80.c index 3cc5151e0986..866489ad56d6 100644 --- a/drivers/rtc/rtc-m41t80.c +++ b/drivers/rtc/rtc-m41t80.c @@ -17,7 +17,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/rtc/rtc-m48t86.c b/drivers/rtc/rtc-m48t86.c index 481c9525b1dd..dd4a62e2d39c 100644 --- a/drivers/rtc/rtc-m48t86.c +++ b/drivers/rtc/rtc-m48t86.c @@ -11,6 +11,7 @@ */ #include +#include #include #include #include @@ -269,9 +270,16 @@ static int m48t86_rtc_probe(struct platform_device *pdev) return 0; } +static const struct of_device_id m48t86_rtc_of_ids[] = { + { .compatible = "st,m48t86" }, + { /* sentinel */ } +}; +MODULE_DEVICE_TABLE(of, m48t86_rtc_of_ids); + static struct platform_driver m48t86_rtc_platform_driver = { .driver = { .name = "rtc-m48t86", + .of_match_table = m48t86_rtc_of_ids, }, .probe = m48t86_rtc_probe, }; diff --git a/drivers/rtc/rtc-mpc5121.c b/drivers/rtc/rtc-mpc5121.c index 07df43e4c4d0..28858fcaea8f 100644 --- a/drivers/rtc/rtc-mpc5121.c +++ b/drivers/rtc/rtc-mpc5121.c @@ -11,10 +11,8 @@ #include #include #include -#include -#include #include -#include +#include #include #include diff --git a/drivers/rtc/rtc-mt6397.c b/drivers/rtc/rtc-mt6397.c index 1d297af80f87..1617063669cc 100644 --- a/drivers/rtc/rtc-mt6397.c +++ b/drivers/rtc/rtc-mt6397.c @@ -9,7 +9,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/rtc/rtc-mt7622.c b/drivers/rtc/rtc-mt7622.c index 81857a457c32..094c649fc137 100644 --- a/drivers/rtc/rtc-mt7622.c +++ b/drivers/rtc/rtc-mt7622.c @@ -7,9 +7,9 @@ #include #include +#include +#include #include -#include -#include #include #include diff --git a/drivers/rtc/rtc-mxc.c b/drivers/rtc/rtc-mxc.c index 762cf03345f1..dbb935dbbd8a 100644 --- a/drivers/rtc/rtc-mxc.c +++ b/drivers/rtc/rtc-mxc.c @@ -11,7 +11,6 @@ #include #include #include -#include #define RTC_INPUT_CLK_32768HZ (0x00 << 5) #define RTC_INPUT_CLK_32000HZ (0x01 << 5) diff --git a/drivers/rtc/rtc-nct3018y.c b/drivers/rtc/rtc-nct3018y.c index a4e3f924837e..ed4e606be8e5 100644 --- a/drivers/rtc/rtc-nct3018y.c +++ b/drivers/rtc/rtc-nct3018y.c @@ -538,7 +538,7 @@ MODULE_DEVICE_TABLE(of, nct3018y_of_match); static struct i2c_driver nct3018y_driver = { .driver = { .name = "rtc-nct3018y", - .of_match_table = of_match_ptr(nct3018y_of_match), + .of_match_table = nct3018y_of_match, }, .probe = nct3018y_probe, .id_table = nct3018y_id, diff --git a/drivers/rtc/rtc-omap.c b/drivers/rtc/rtc-omap.c index 8ae4d7824ec9..5b10ab06cd2e 100644 --- a/drivers/rtc/rtc-omap.c +++ b/drivers/rtc/rtc-omap.c @@ -747,12 +747,12 @@ static int omap_rtc_probe(struct platform_device *pdev) } rtc->irq_timer = platform_get_irq(pdev, 0); - if (rtc->irq_timer <= 0) - return -ENOENT; + if (rtc->irq_timer < 0) + return rtc->irq_timer; rtc->irq_alarm = platform_get_irq(pdev, 1); - if (rtc->irq_alarm <= 0) - return -ENOENT; + if (rtc->irq_alarm < 0) + return rtc->irq_alarm; rtc->clk = devm_clk_get(&pdev->dev, "ext-clk"); if (!IS_ERR(rtc->clk)) diff --git a/drivers/rtc/rtc-pcf2127.c b/drivers/rtc/rtc-pcf2127.c index ee03b04b74ba..9c04c4e1a49c 100644 --- a/drivers/rtc/rtc-pcf2127.c +++ b/drivers/rtc/rtc-pcf2127.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only /* - * An I2C and SPI driver for the NXP PCF2127/29 RTC + * An I2C and SPI driver for the NXP PCF2127/29/31 RTC * Copyright 2013 Til-Technologies * * Author: Renaud Cerrato @@ -8,9 +8,13 @@ * Watchdog and tamper functions * Author: Bruno Thomsen * + * PCF2131 support + * Author: Hugo Villeneuve + * * based on the other drivers in this same directory. * - * Datasheet: https://www.nxp.com/docs/en/data-sheet/PCF2127.pdf + * Datasheets: https://www.nxp.com/docs/en/data-sheet/PCF2127.pdf + * https://www.nxp.com/docs/en/data-sheet/PCF2131DS.pdf */ #include @@ -21,6 +25,7 @@ #include #include #include +#include #include #include @@ -28,6 +33,7 @@ #define PCF2127_REG_CTRL1 0x00 #define PCF2127_BIT_CTRL1_POR_OVRD BIT(3) #define PCF2127_BIT_CTRL1_TSF1 BIT(4) +#define PCF2127_BIT_CTRL1_STOP BIT(5) /* Control register 2 */ #define PCF2127_REG_CTRL2 0x01 #define PCF2127_BIT_CTRL2_AIE BIT(1) @@ -43,20 +49,10 @@ #define PCF2127_BIT_CTRL3_BF BIT(3) #define PCF2127_BIT_CTRL3_BTSE BIT(4) /* Time and date registers */ -#define PCF2127_REG_SC 0x03 +#define PCF2127_REG_TIME_BASE 0x03 #define PCF2127_BIT_SC_OSF BIT(7) -#define PCF2127_REG_MN 0x04 -#define PCF2127_REG_HR 0x05 -#define PCF2127_REG_DM 0x06 -#define PCF2127_REG_DW 0x07 -#define PCF2127_REG_MO 0x08 -#define PCF2127_REG_YR 0x09 /* Alarm registers */ -#define PCF2127_REG_ALARM_SC 0x0A -#define PCF2127_REG_ALARM_MN 0x0B -#define PCF2127_REG_ALARM_HR 0x0C -#define PCF2127_REG_ALARM_DM 0x0D -#define PCF2127_REG_ALARM_DW 0x0E +#define PCF2127_REG_ALARM_BASE 0x0A #define PCF2127_BIT_ALARM_AE BIT(7) /* CLKOUT control register */ #define PCF2127_REG_CLKOUT 0x0f @@ -68,21 +64,15 @@ #define PCF2127_BIT_WD_CTL_CD0 BIT(6) #define PCF2127_BIT_WD_CTL_CD1 BIT(7) #define PCF2127_REG_WD_VAL 0x11 -/* Tamper timestamp registers */ -#define PCF2127_REG_TS_CTRL 0x12 +/* Tamper timestamp1 registers */ +#define PCF2127_REG_TS1_BASE 0x12 #define PCF2127_BIT_TS_CTRL_TSOFF BIT(6) #define PCF2127_BIT_TS_CTRL_TSM BIT(7) -#define PCF2127_REG_TS_SC 0x13 -#define PCF2127_REG_TS_MN 0x14 -#define PCF2127_REG_TS_HR 0x15 -#define PCF2127_REG_TS_DM 0x16 -#define PCF2127_REG_TS_MO 0x17 -#define PCF2127_REG_TS_YR 0x18 /* * RAM registers * PCF2127 has 512 bytes general-purpose static RAM (SRAM) that is * battery backed and can survive a power outage. - * PCF2129 doesn't have this feature. + * PCF2129/31 doesn't have this feature. */ #define PCF2127_REG_RAM_ADDR_MSB 0x1A #define PCF2127_REG_RAM_WRT_CMD 0x1C @@ -90,9 +80,14 @@ /* Watchdog timer value constants */ #define PCF2127_WD_VAL_STOP 0 -#define PCF2127_WD_VAL_MIN 2 -#define PCF2127_WD_VAL_MAX 255 -#define PCF2127_WD_VAL_DEFAULT 60 +/* PCF2127/29 watchdog timer value constants */ +#define PCF2127_WD_CLOCK_HZ_X1000 1000 /* 1Hz */ +#define PCF2127_WD_MIN_HW_HEARTBEAT_MS 500 +/* PCF2131 watchdog timer value constants */ +#define PCF2131_WD_CLOCK_HZ_X1000 250 /* 1/4Hz */ +#define PCF2131_WD_MIN_HW_HEARTBEAT_MS 4000 + +#define PCF2127_WD_DEFAULT_TIMEOUT_S 60 /* Mask for currently enabled interrupts */ #define PCF2127_CTRL1_IRQ_MASK (PCF2127_BIT_CTRL1_TSF1) @@ -101,13 +96,117 @@ PCF2127_BIT_CTRL2_WDTF | \ PCF2127_BIT_CTRL2_TSF2) +#define PCF2127_MAX_TS_SUPPORTED 4 + +/* Control register 4 */ +#define PCF2131_REG_CTRL4 0x03 +#define PCF2131_BIT_CTRL4_TSF4 BIT(4) +#define PCF2131_BIT_CTRL4_TSF3 BIT(5) +#define PCF2131_BIT_CTRL4_TSF2 BIT(6) +#define PCF2131_BIT_CTRL4_TSF1 BIT(7) +/* Control register 5 */ +#define PCF2131_REG_CTRL5 0x04 +#define PCF2131_BIT_CTRL5_TSIE4 BIT(4) +#define PCF2131_BIT_CTRL5_TSIE3 BIT(5) +#define PCF2131_BIT_CTRL5_TSIE2 BIT(6) +#define PCF2131_BIT_CTRL5_TSIE1 BIT(7) +/* Software reset register */ +#define PCF2131_REG_SR_RESET 0x05 +#define PCF2131_SR_RESET_READ_PATTERN (BIT(2) | BIT(5)) +#define PCF2131_SR_RESET_CPR_CMD (PCF2131_SR_RESET_READ_PATTERN | BIT(7)) +/* Time and date registers */ +#define PCF2131_REG_TIME_BASE 0x07 +/* Alarm registers */ +#define PCF2131_REG_ALARM_BASE 0x0E +/* CLKOUT control register */ +#define PCF2131_REG_CLKOUT 0x13 +/* Watchdog registers */ +#define PCF2131_REG_WD_CTL 0x35 +#define PCF2131_REG_WD_VAL 0x36 +/* Tamper timestamp1 registers */ +#define PCF2131_REG_TS1_BASE 0x14 +/* Tamper timestamp2 registers */ +#define PCF2131_REG_TS2_BASE 0x1B +/* Tamper timestamp3 registers */ +#define PCF2131_REG_TS3_BASE 0x22 +/* Tamper timestamp4 registers */ +#define PCF2131_REG_TS4_BASE 0x29 +/* Interrupt mask registers */ +#define PCF2131_REG_INT_A_MASK1 0x31 +#define PCF2131_REG_INT_A_MASK2 0x32 +#define PCF2131_REG_INT_B_MASK1 0x33 +#define PCF2131_REG_INT_B_MASK2 0x34 +#define PCF2131_BIT_INT_BLIE BIT(0) +#define PCF2131_BIT_INT_BIE BIT(1) +#define PCF2131_BIT_INT_AIE BIT(2) +#define PCF2131_BIT_INT_WD_CD BIT(3) +#define PCF2131_BIT_INT_SI BIT(4) +#define PCF2131_BIT_INT_MI BIT(5) +#define PCF2131_CTRL2_IRQ_MASK ( \ + PCF2127_BIT_CTRL2_AF | \ + PCF2127_BIT_CTRL2_WDTF) +#define PCF2131_CTRL4_IRQ_MASK ( \ + PCF2131_BIT_CTRL4_TSF4 | \ + PCF2131_BIT_CTRL4_TSF3 | \ + PCF2131_BIT_CTRL4_TSF2 | \ + PCF2131_BIT_CTRL4_TSF1) + +enum pcf21xx_type { + PCF2127, + PCF2129, + PCF2131, + PCF21XX_LAST_ID +}; + +struct pcf21xx_ts_config { + u8 reg_base; /* Base register to read timestamp values. */ + + /* + * If the TS input pin is driven to GND, an interrupt can be generated + * (supported by all variants). + */ + u8 gnd_detect_reg; /* Interrupt control register address. */ + u8 gnd_detect_bit; /* Interrupt bit. */ + + /* + * If the TS input pin is driven to an intermediate level between GND + * and supply, an interrupt can be generated (optional feature depending + * on variant). + */ + u8 inter_detect_reg; /* Interrupt control register address. */ + u8 inter_detect_bit; /* Interrupt bit. */ + + u8 ie_reg; /* Interrupt enable control register. */ + u8 ie_bit; /* Interrupt enable bit. */ +}; + +struct pcf21xx_config { + int type; /* IC variant */ + int max_register; + unsigned int has_nvmem:1; + unsigned int has_bit_wd_ctl_cd0:1; + unsigned int wd_val_reg_readable:1; /* If watchdog value register can be read. */ + unsigned int has_int_a_b:1; /* PCF2131 supports two interrupt outputs. */ + u8 reg_time_base; /* Time/date base register. */ + u8 regs_alarm_base; /* Alarm function base registers. */ + u8 reg_wd_ctl; /* Watchdog control register. */ + u8 reg_wd_val; /* Watchdog value register. */ + u8 reg_clkout; /* Clkout register. */ + int wdd_clock_hz_x1000; /* Watchdog clock in Hz multiplicated by 1000 */ + int wdd_min_hw_heartbeat_ms; + unsigned int ts_count; + struct pcf21xx_ts_config ts[PCF2127_MAX_TS_SUPPORTED]; + struct attribute_group attribute_group; +}; + struct pcf2127 { struct rtc_device *rtc; struct watchdog_device wdd; struct regmap *regmap; - time64_t ts; - bool ts_valid; + const struct pcf21xx_config *cfg; bool irq_enabled; + time64_t ts[PCF2127_MAX_TS_SUPPORTED]; /* Timestamp values. */ + bool ts_valid[PCF2127_MAX_TS_SUPPORTED]; /* Timestamp valid indication. */ }; /* @@ -117,27 +216,22 @@ struct pcf2127 { static int pcf2127_rtc_read_time(struct device *dev, struct rtc_time *tm) { struct pcf2127 *pcf2127 = dev_get_drvdata(dev); - unsigned char buf[10]; + unsigned char buf[7]; int ret; /* * Avoid reading CTRL2 register as it causes WD_VAL register * value to reset to 0 which means watchdog is stopped. */ - ret = regmap_bulk_read(pcf2127->regmap, PCF2127_REG_CTRL3, - (buf + PCF2127_REG_CTRL3), - ARRAY_SIZE(buf) - PCF2127_REG_CTRL3); + ret = regmap_bulk_read(pcf2127->regmap, pcf2127->cfg->reg_time_base, + buf, sizeof(buf)); if (ret) { dev_err(dev, "%s: read error\n", __func__); return ret; } - if (buf[PCF2127_REG_CTRL3] & PCF2127_BIT_CTRL3_BLF) - dev_info(dev, - "low voltage detected, check/replace RTC battery.\n"); - /* Clock integrity is not guaranteed when OSF flag is set. */ - if (buf[PCF2127_REG_SC] & PCF2127_BIT_SC_OSF) { + if (buf[0] & PCF2127_BIT_SC_OSF) { /* * no need clear the flag here, * it will be cleared once the new date is saved @@ -148,20 +242,17 @@ static int pcf2127_rtc_read_time(struct device *dev, struct rtc_time *tm) } dev_dbg(dev, - "%s: raw data is cr3=%02x, sec=%02x, min=%02x, hr=%02x, " + "%s: raw data is sec=%02x, min=%02x, hr=%02x, " "mday=%02x, wday=%02x, mon=%02x, year=%02x\n", - __func__, buf[PCF2127_REG_CTRL3], buf[PCF2127_REG_SC], - buf[PCF2127_REG_MN], buf[PCF2127_REG_HR], - buf[PCF2127_REG_DM], buf[PCF2127_REG_DW], - buf[PCF2127_REG_MO], buf[PCF2127_REG_YR]); + __func__, buf[0], buf[1], buf[2], buf[3], buf[4], buf[5], buf[6]); - tm->tm_sec = bcd2bin(buf[PCF2127_REG_SC] & 0x7F); - tm->tm_min = bcd2bin(buf[PCF2127_REG_MN] & 0x7F); - tm->tm_hour = bcd2bin(buf[PCF2127_REG_HR] & 0x3F); /* rtc hr 0-23 */ - tm->tm_mday = bcd2bin(buf[PCF2127_REG_DM] & 0x3F); - tm->tm_wday = buf[PCF2127_REG_DW] & 0x07; - tm->tm_mon = bcd2bin(buf[PCF2127_REG_MO] & 0x1F) - 1; /* rtc mn 1-12 */ - tm->tm_year = bcd2bin(buf[PCF2127_REG_YR]); + tm->tm_sec = bcd2bin(buf[0] & 0x7F); + tm->tm_min = bcd2bin(buf[1] & 0x7F); + tm->tm_hour = bcd2bin(buf[2] & 0x3F); + tm->tm_mday = bcd2bin(buf[3] & 0x3F); + tm->tm_wday = buf[4] & 0x07; + tm->tm_mon = bcd2bin(buf[5] & 0x1F) - 1; + tm->tm_year = bcd2bin(buf[6]); tm->tm_year += 100; dev_dbg(dev, "%s: tm is secs=%d, mins=%d, hours=%d, " @@ -198,14 +289,45 @@ static int pcf2127_rtc_set_time(struct device *dev, struct rtc_time *tm) /* year */ buf[i++] = bin2bcd(tm->tm_year - 100); - /* write register's data */ - err = regmap_bulk_write(pcf2127->regmap, PCF2127_REG_SC, buf, i); + /* Write access to time registers: + * PCF2127/29: no special action required. + * PCF2131: requires setting the STOP and CPR bits. STOP bit needs to + * be cleared after time registers are updated. + */ + if (pcf2127->cfg->type == PCF2131) { + err = regmap_update_bits(pcf2127->regmap, PCF2127_REG_CTRL1, + PCF2127_BIT_CTRL1_STOP, + PCF2127_BIT_CTRL1_STOP); + if (err) { + dev_dbg(dev, "setting STOP bit failed\n"); + return err; + } + + err = regmap_write(pcf2127->regmap, PCF2131_REG_SR_RESET, + PCF2131_SR_RESET_CPR_CMD); + if (err) { + dev_dbg(dev, "sending CPR cmd failed\n"); + return err; + } + } + + /* write time register's data */ + err = regmap_bulk_write(pcf2127->regmap, pcf2127->cfg->reg_time_base, buf, i); if (err) { - dev_err(dev, - "%s: err=%d", __func__, err); + dev_dbg(dev, "%s: err=%d", __func__, err); return err; } + if (pcf2127->cfg->type == PCF2131) { + /* Clear STOP bit (PCF2131 only) after write is completed. */ + err = regmap_update_bits(pcf2127->regmap, PCF2127_REG_CTRL1, + PCF2127_BIT_CTRL1_STOP, 0); + if (err) { + dev_dbg(dev, "clearing STOP bit failed\n"); + return err; + } + } + return 0; } @@ -275,9 +397,16 @@ static int pcf2127_nvmem_write(void *priv, unsigned int offset, static int pcf2127_wdt_ping(struct watchdog_device *wdd) { + int wd_val; struct pcf2127 *pcf2127 = watchdog_get_drvdata(wdd); - return regmap_write(pcf2127->regmap, PCF2127_REG_WD_VAL, wdd->timeout); + /* + * Compute counter value of WATCHDG_TIM_VAL to obtain desired period + * in seconds, depending on the source clock frequency. + */ + wd_val = ((wdd->timeout * pcf2127->cfg->wdd_clock_hz_x1000) / 1000) + 1; + + return regmap_write(pcf2127->regmap, pcf2127->cfg->reg_wd_val, wd_val); } /* @@ -311,7 +440,7 @@ static int pcf2127_wdt_stop(struct watchdog_device *wdd) { struct pcf2127 *pcf2127 = watchdog_get_drvdata(wdd); - return regmap_write(pcf2127->regmap, PCF2127_REG_WD_VAL, + return regmap_write(pcf2127->regmap, pcf2127->cfg->reg_wd_val, PCF2127_WD_VAL_STOP); } @@ -339,9 +468,25 @@ static const struct watchdog_ops pcf2127_watchdog_ops = { .set_timeout = pcf2127_wdt_set_timeout, }; +/* + * Compute watchdog period, t, in seconds, from the WATCHDG_TIM_VAL register + * value, n, and the clock frequency, f1000, in Hz x 1000. + * + * The PCF2127/29 datasheet gives t as: + * t = n / f + * The PCF2131 datasheet gives t as: + * t = (n - 1) / f + * For both variants, the watchdog is triggered when the WATCHDG_TIM_VAL reaches + * the value 1, and not zero. Consequently, the equation from the PCF2131 + * datasheet seems to be the correct one for both variants. + */ +static int pcf2127_watchdog_get_period(int n, int f1000) +{ + return (1000 * (n - 1)) / f1000; +} + static int pcf2127_watchdog_init(struct device *dev, struct pcf2127 *pcf2127) { - u32 wdd_timeout; int ret; if (!IS_ENABLED(CONFIG_WATCHDOG) || @@ -351,21 +496,35 @@ static int pcf2127_watchdog_init(struct device *dev, struct pcf2127 *pcf2127) pcf2127->wdd.parent = dev; pcf2127->wdd.info = &pcf2127_wdt_info; pcf2127->wdd.ops = &pcf2127_watchdog_ops; - pcf2127->wdd.min_timeout = PCF2127_WD_VAL_MIN; - pcf2127->wdd.max_timeout = PCF2127_WD_VAL_MAX; - pcf2127->wdd.timeout = PCF2127_WD_VAL_DEFAULT; - pcf2127->wdd.min_hw_heartbeat_ms = 500; + + pcf2127->wdd.min_timeout = + pcf2127_watchdog_get_period( + 2, pcf2127->cfg->wdd_clock_hz_x1000); + pcf2127->wdd.max_timeout = + pcf2127_watchdog_get_period( + 255, pcf2127->cfg->wdd_clock_hz_x1000); + pcf2127->wdd.timeout = PCF2127_WD_DEFAULT_TIMEOUT_S; + + dev_dbg(dev, "%s clock = %d Hz / 1000\n", __func__, + pcf2127->cfg->wdd_clock_hz_x1000); + + pcf2127->wdd.min_hw_heartbeat_ms = pcf2127->cfg->wdd_min_hw_heartbeat_ms; pcf2127->wdd.status = WATCHDOG_NOWAYOUT_INIT_STATUS; watchdog_set_drvdata(&pcf2127->wdd, pcf2127); /* Test if watchdog timer is started by bootloader */ - ret = regmap_read(pcf2127->regmap, PCF2127_REG_WD_VAL, &wdd_timeout); - if (ret) - return ret; + if (pcf2127->cfg->wd_val_reg_readable) { + u32 wdd_timeout; - if (wdd_timeout) - set_bit(WDOG_HW_RUNNING, &pcf2127->wdd.status); + ret = regmap_read(pcf2127->regmap, pcf2127->cfg->reg_wd_val, + &wdd_timeout); + if (ret) + return ret; + + if (wdd_timeout) + set_bit(WDOG_HW_RUNNING, &pcf2127->wdd.status); + } return devm_watchdog_register_device(dev, &pcf2127->wdd); } @@ -386,8 +545,8 @@ static int pcf2127_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm) if (ret) return ret; - ret = regmap_bulk_read(pcf2127->regmap, PCF2127_REG_ALARM_SC, buf, - sizeof(buf)); + ret = regmap_bulk_read(pcf2127->regmap, pcf2127->cfg->regs_alarm_base, + buf, sizeof(buf)); if (ret) return ret; @@ -437,8 +596,8 @@ static int pcf2127_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) buf[3] = bin2bcd(alrm->time.tm_mday); buf[4] = PCF2127_BIT_ALARM_AE; /* Do not match on week day */ - ret = regmap_bulk_write(pcf2127->regmap, PCF2127_REG_ALARM_SC, buf, - sizeof(buf)); + ret = regmap_bulk_write(pcf2127->regmap, pcf2127->cfg->regs_alarm_base, + buf, sizeof(buf)); if (ret) return ret; @@ -446,38 +605,35 @@ static int pcf2127_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) } /* - * This function reads ctrl2 register, caller is responsible for calling - * pcf2127_wdt_active_ping() + * This function reads one timestamp function data, caller is responsible for + * calling pcf2127_wdt_active_ping() */ -static int pcf2127_rtc_ts_read(struct device *dev, time64_t *ts) +static int pcf2127_rtc_ts_read(struct device *dev, time64_t *ts, + int ts_id) { struct pcf2127 *pcf2127 = dev_get_drvdata(dev); struct rtc_time tm; int ret; - unsigned char data[25]; + unsigned char data[7]; - ret = regmap_bulk_read(pcf2127->regmap, PCF2127_REG_CTRL1, data, - sizeof(data)); + ret = regmap_bulk_read(pcf2127->regmap, pcf2127->cfg->ts[ts_id].reg_base, + data, sizeof(data)); if (ret) { dev_err(dev, "%s: read error ret=%d\n", __func__, ret); return ret; } dev_dbg(dev, - "%s: raw data is cr1=%02x, cr2=%02x, cr3=%02x, ts_sc=%02x, ts_mn=%02x, ts_hr=%02x, ts_dm=%02x, ts_mo=%02x, ts_yr=%02x\n", - __func__, data[PCF2127_REG_CTRL1], data[PCF2127_REG_CTRL2], - data[PCF2127_REG_CTRL3], data[PCF2127_REG_TS_SC], - data[PCF2127_REG_TS_MN], data[PCF2127_REG_TS_HR], - data[PCF2127_REG_TS_DM], data[PCF2127_REG_TS_MO], - data[PCF2127_REG_TS_YR]); + "%s: raw data is ts_sc=%02x, ts_mn=%02x, ts_hr=%02x, ts_dm=%02x, ts_mo=%02x, ts_yr=%02x\n", + __func__, data[1], data[2], data[3], data[4], data[5], data[6]); - tm.tm_sec = bcd2bin(data[PCF2127_REG_TS_SC] & 0x7F); - tm.tm_min = bcd2bin(data[PCF2127_REG_TS_MN] & 0x7F); - tm.tm_hour = bcd2bin(data[PCF2127_REG_TS_HR] & 0x3F); - tm.tm_mday = bcd2bin(data[PCF2127_REG_TS_DM] & 0x3F); + tm.tm_sec = bcd2bin(data[1] & 0x7F); + tm.tm_min = bcd2bin(data[2] & 0x7F); + tm.tm_hour = bcd2bin(data[3] & 0x3F); + tm.tm_mday = bcd2bin(data[4] & 0x3F); /* TS_MO register (month) value range: 1-12 */ - tm.tm_mon = bcd2bin(data[PCF2127_REG_TS_MO] & 0x1F) - 1; - tm.tm_year = bcd2bin(data[PCF2127_REG_TS_YR]); + tm.tm_mon = bcd2bin(data[5] & 0x1F) - 1; + tm.tm_year = bcd2bin(data[6]); if (tm.tm_year < 70) tm.tm_year += 100; /* assume we are in 1970...2069 */ @@ -491,47 +647,84 @@ static int pcf2127_rtc_ts_read(struct device *dev, time64_t *ts) return 0; }; -static void pcf2127_rtc_ts_snapshot(struct device *dev) +static void pcf2127_rtc_ts_snapshot(struct device *dev, int ts_id) { struct pcf2127 *pcf2127 = dev_get_drvdata(dev); int ret; - /* Let userspace read the first timestamp */ - if (pcf2127->ts_valid) + if (ts_id >= pcf2127->cfg->ts_count) return; - ret = pcf2127_rtc_ts_read(dev, &pcf2127->ts); + /* Let userspace read the first timestamp */ + if (pcf2127->ts_valid[ts_id]) + return; + + ret = pcf2127_rtc_ts_read(dev, &pcf2127->ts[ts_id], ts_id); if (!ret) - pcf2127->ts_valid = true; + pcf2127->ts_valid[ts_id] = true; } static irqreturn_t pcf2127_rtc_irq(int irq, void *dev) { struct pcf2127 *pcf2127 = dev_get_drvdata(dev); - unsigned int ctrl1, ctrl2; + unsigned int ctrl2; int ret = 0; - ret = regmap_read(pcf2127->regmap, PCF2127_REG_CTRL1, &ctrl1); - if (ret) - return IRQ_NONE; - ret = regmap_read(pcf2127->regmap, PCF2127_REG_CTRL2, &ctrl2); if (ret) return IRQ_NONE; - if (!(ctrl1 & PCF2127_CTRL1_IRQ_MASK || ctrl2 & PCF2127_CTRL2_IRQ_MASK)) - return IRQ_NONE; + if (pcf2127->cfg->ts_count == 1) { + /* PCF2127/29 */ + unsigned int ctrl1; - if (ctrl1 & PCF2127_BIT_CTRL1_TSF1 || ctrl2 & PCF2127_BIT_CTRL2_TSF2) - pcf2127_rtc_ts_snapshot(dev); + ret = regmap_read(pcf2127->regmap, PCF2127_REG_CTRL1, &ctrl1); + if (ret) + return IRQ_NONE; - if (ctrl1 & PCF2127_CTRL1_IRQ_MASK) - regmap_write(pcf2127->regmap, PCF2127_REG_CTRL1, - ctrl1 & ~PCF2127_CTRL1_IRQ_MASK); + if (!(ctrl1 & PCF2127_CTRL1_IRQ_MASK || ctrl2 & PCF2127_CTRL2_IRQ_MASK)) + return IRQ_NONE; - if (ctrl2 & PCF2127_CTRL2_IRQ_MASK) - regmap_write(pcf2127->regmap, PCF2127_REG_CTRL2, - ctrl2 & ~PCF2127_CTRL2_IRQ_MASK); + if (ctrl1 & PCF2127_BIT_CTRL1_TSF1 || ctrl2 & PCF2127_BIT_CTRL2_TSF2) + pcf2127_rtc_ts_snapshot(dev, 0); + + if (ctrl1 & PCF2127_CTRL1_IRQ_MASK) + regmap_write(pcf2127->regmap, PCF2127_REG_CTRL1, + ctrl1 & ~PCF2127_CTRL1_IRQ_MASK); + + if (ctrl2 & PCF2127_CTRL2_IRQ_MASK) + regmap_write(pcf2127->regmap, PCF2127_REG_CTRL2, + ctrl2 & ~PCF2127_CTRL2_IRQ_MASK); + } else { + /* PCF2131. */ + unsigned int ctrl4; + + ret = regmap_read(pcf2127->regmap, PCF2131_REG_CTRL4, &ctrl4); + if (ret) + return IRQ_NONE; + + if (!(ctrl4 & PCF2131_CTRL4_IRQ_MASK || ctrl2 & PCF2131_CTRL2_IRQ_MASK)) + return IRQ_NONE; + + if (ctrl4 & PCF2131_CTRL4_IRQ_MASK) { + int i; + int tsf_bit = PCF2131_BIT_CTRL4_TSF1; /* Start at bit 7. */ + + for (i = 0; i < pcf2127->cfg->ts_count; i++) { + if (ctrl4 & tsf_bit) + pcf2127_rtc_ts_snapshot(dev, i); + + tsf_bit = tsf_bit >> 1; + } + + regmap_write(pcf2127->regmap, PCF2131_REG_CTRL4, + ctrl4 & ~PCF2131_CTRL4_IRQ_MASK); + } + + if (ctrl2 & PCF2131_CTRL2_IRQ_MASK) + regmap_write(pcf2127->regmap, PCF2127_REG_CTRL2, + ctrl2 & ~PCF2131_CTRL2_IRQ_MASK); + } if (ctrl2 & PCF2127_BIT_CTRL2_AF) rtc_update_irq(pcf2127->rtc, 1, RTC_IRQF | RTC_AF); @@ -552,28 +745,41 @@ static const struct rtc_class_ops pcf2127_rtc_ops = { /* sysfs interface */ -static ssize_t timestamp0_store(struct device *dev, - struct device_attribute *attr, - const char *buf, size_t count) +static ssize_t timestamp_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count, int ts_id) { struct pcf2127 *pcf2127 = dev_get_drvdata(dev->parent); int ret; + if (ts_id >= pcf2127->cfg->ts_count) + return 0; + if (pcf2127->irq_enabled) { - pcf2127->ts_valid = false; + pcf2127->ts_valid[ts_id] = false; } else { - ret = regmap_update_bits(pcf2127->regmap, PCF2127_REG_CTRL1, - PCF2127_BIT_CTRL1_TSF1, 0); + /* Always clear GND interrupt bit. */ + ret = regmap_update_bits(pcf2127->regmap, + pcf2127->cfg->ts[ts_id].gnd_detect_reg, + pcf2127->cfg->ts[ts_id].gnd_detect_bit, + 0); + if (ret) { - dev_err(dev, "%s: update ctrl1 ret=%d\n", __func__, ret); + dev_err(dev, "%s: update TS gnd detect ret=%d\n", __func__, ret); return ret; } - ret = regmap_update_bits(pcf2127->regmap, PCF2127_REG_CTRL2, - PCF2127_BIT_CTRL2_TSF2, 0); - if (ret) { - dev_err(dev, "%s: update ctrl2 ret=%d\n", __func__, ret); - return ret; + if (pcf2127->cfg->ts[ts_id].inter_detect_bit) { + /* Clear intermediate level interrupt bit if supported. */ + ret = regmap_update_bits(pcf2127->regmap, + pcf2127->cfg->ts[ts_id].inter_detect_reg, + pcf2127->cfg->ts[ts_id].inter_detect_bit, + 0); + if (ret) { + dev_err(dev, "%s: update TS intermediate level detect ret=%d\n", + __func__, ret); + return ret; + } } ret = pcf2127_wdt_active_ping(&pcf2127->wdd); @@ -582,34 +788,84 @@ static ssize_t timestamp0_store(struct device *dev, } return count; +} + +static ssize_t timestamp0_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + return timestamp_store(dev, attr, buf, count, 0); }; -static ssize_t timestamp0_show(struct device *dev, - struct device_attribute *attr, char *buf) +static ssize_t timestamp1_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + return timestamp_store(dev, attr, buf, count, 1); +}; + +static ssize_t timestamp2_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + return timestamp_store(dev, attr, buf, count, 2); +}; + +static ssize_t timestamp3_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + return timestamp_store(dev, attr, buf, count, 3); +}; + +static ssize_t timestamp_show(struct device *dev, + struct device_attribute *attr, char *buf, + int ts_id) { struct pcf2127 *pcf2127 = dev_get_drvdata(dev->parent); - unsigned int ctrl1, ctrl2; int ret; time64_t ts; + if (ts_id >= pcf2127->cfg->ts_count) + return 0; + if (pcf2127->irq_enabled) { - if (!pcf2127->ts_valid) + if (!pcf2127->ts_valid[ts_id]) return 0; - ts = pcf2127->ts; + ts = pcf2127->ts[ts_id]; } else { - ret = regmap_read(pcf2127->regmap, PCF2127_REG_CTRL1, &ctrl1); + u8 valid_low = 0; + u8 valid_inter = 0; + unsigned int ctrl; + + /* Check if TS input pin is driven to GND, supported by all + * variants. + */ + ret = regmap_read(pcf2127->regmap, + pcf2127->cfg->ts[ts_id].gnd_detect_reg, + &ctrl); if (ret) return 0; - ret = regmap_read(pcf2127->regmap, PCF2127_REG_CTRL2, &ctrl2); - if (ret) + valid_low = ctrl & pcf2127->cfg->ts[ts_id].gnd_detect_bit; + + if (pcf2127->cfg->ts[ts_id].inter_detect_bit) { + /* Check if TS input pin is driven to intermediate level + * between GND and supply, if supported by variant. + */ + ret = regmap_read(pcf2127->regmap, + pcf2127->cfg->ts[ts_id].inter_detect_reg, + &ctrl); + if (ret) + return 0; + + valid_inter = ctrl & pcf2127->cfg->ts[ts_id].inter_detect_bit; + } + + if (!valid_low && !valid_inter) return 0; - if (!(ctrl1 & PCF2127_BIT_CTRL1_TSF1) && - !(ctrl2 & PCF2127_BIT_CTRL2_TSF2)) - return 0; - - ret = pcf2127_rtc_ts_read(dev->parent, &ts); + ret = pcf2127_rtc_ts_read(dev->parent, &ts, ts_id); if (ret) return 0; @@ -618,21 +874,227 @@ static ssize_t timestamp0_show(struct device *dev, return ret; } return sprintf(buf, "%llu\n", (unsigned long long)ts); +} + +static ssize_t timestamp0_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + return timestamp_show(dev, attr, buf, 0); +}; + +static ssize_t timestamp1_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + return timestamp_show(dev, attr, buf, 1); +}; + +static ssize_t timestamp2_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + return timestamp_show(dev, attr, buf, 2); +}; + +static ssize_t timestamp3_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + return timestamp_show(dev, attr, buf, 3); }; static DEVICE_ATTR_RW(timestamp0); +static DEVICE_ATTR_RW(timestamp1); +static DEVICE_ATTR_RW(timestamp2); +static DEVICE_ATTR_RW(timestamp3); static struct attribute *pcf2127_attrs[] = { &dev_attr_timestamp0.attr, NULL }; -static const struct attribute_group pcf2127_attr_group = { - .attrs = pcf2127_attrs, +static struct attribute *pcf2131_attrs[] = { + &dev_attr_timestamp0.attr, + &dev_attr_timestamp1.attr, + &dev_attr_timestamp2.attr, + &dev_attr_timestamp3.attr, + NULL }; +static struct pcf21xx_config pcf21xx_cfg[] = { + [PCF2127] = { + .type = PCF2127, + .max_register = 0x1d, + .has_nvmem = 1, + .has_bit_wd_ctl_cd0 = 1, + .wd_val_reg_readable = 1, + .has_int_a_b = 0, + .reg_time_base = PCF2127_REG_TIME_BASE, + .regs_alarm_base = PCF2127_REG_ALARM_BASE, + .reg_wd_ctl = PCF2127_REG_WD_CTL, + .reg_wd_val = PCF2127_REG_WD_VAL, + .reg_clkout = PCF2127_REG_CLKOUT, + .wdd_clock_hz_x1000 = PCF2127_WD_CLOCK_HZ_X1000, + .wdd_min_hw_heartbeat_ms = PCF2127_WD_MIN_HW_HEARTBEAT_MS, + .ts_count = 1, + .ts[0] = { + .reg_base = PCF2127_REG_TS1_BASE, + .gnd_detect_reg = PCF2127_REG_CTRL1, + .gnd_detect_bit = PCF2127_BIT_CTRL1_TSF1, + .inter_detect_reg = PCF2127_REG_CTRL2, + .inter_detect_bit = PCF2127_BIT_CTRL2_TSF2, + .ie_reg = PCF2127_REG_CTRL2, + .ie_bit = PCF2127_BIT_CTRL2_TSIE, + }, + .attribute_group = { + .attrs = pcf2127_attrs, + }, + }, + [PCF2129] = { + .type = PCF2129, + .max_register = 0x19, + .has_nvmem = 0, + .has_bit_wd_ctl_cd0 = 0, + .wd_val_reg_readable = 1, + .has_int_a_b = 0, + .reg_time_base = PCF2127_REG_TIME_BASE, + .regs_alarm_base = PCF2127_REG_ALARM_BASE, + .reg_wd_ctl = PCF2127_REG_WD_CTL, + .reg_wd_val = PCF2127_REG_WD_VAL, + .reg_clkout = PCF2127_REG_CLKOUT, + .wdd_clock_hz_x1000 = PCF2127_WD_CLOCK_HZ_X1000, + .wdd_min_hw_heartbeat_ms = PCF2127_WD_MIN_HW_HEARTBEAT_MS, + .ts_count = 1, + .ts[0] = { + .reg_base = PCF2127_REG_TS1_BASE, + .gnd_detect_reg = PCF2127_REG_CTRL1, + .gnd_detect_bit = PCF2127_BIT_CTRL1_TSF1, + .inter_detect_reg = PCF2127_REG_CTRL2, + .inter_detect_bit = PCF2127_BIT_CTRL2_TSF2, + .ie_reg = PCF2127_REG_CTRL2, + .ie_bit = PCF2127_BIT_CTRL2_TSIE, + }, + .attribute_group = { + .attrs = pcf2127_attrs, + }, + }, + [PCF2131] = { + .type = PCF2131, + .max_register = 0x36, + .has_nvmem = 0, + .has_bit_wd_ctl_cd0 = 0, + .wd_val_reg_readable = 0, + .has_int_a_b = 1, + .reg_time_base = PCF2131_REG_TIME_BASE, + .regs_alarm_base = PCF2131_REG_ALARM_BASE, + .reg_wd_ctl = PCF2131_REG_WD_CTL, + .reg_wd_val = PCF2131_REG_WD_VAL, + .reg_clkout = PCF2131_REG_CLKOUT, + .wdd_clock_hz_x1000 = PCF2131_WD_CLOCK_HZ_X1000, + .wdd_min_hw_heartbeat_ms = PCF2131_WD_MIN_HW_HEARTBEAT_MS, + .ts_count = 4, + .ts[0] = { + .reg_base = PCF2131_REG_TS1_BASE, + .gnd_detect_reg = PCF2131_REG_CTRL4, + .gnd_detect_bit = PCF2131_BIT_CTRL4_TSF1, + .inter_detect_bit = 0, + .ie_reg = PCF2131_REG_CTRL5, + .ie_bit = PCF2131_BIT_CTRL5_TSIE1, + }, + .ts[1] = { + .reg_base = PCF2131_REG_TS2_BASE, + .gnd_detect_reg = PCF2131_REG_CTRL4, + .gnd_detect_bit = PCF2131_BIT_CTRL4_TSF2, + .inter_detect_bit = 0, + .ie_reg = PCF2131_REG_CTRL5, + .ie_bit = PCF2131_BIT_CTRL5_TSIE2, + }, + .ts[2] = { + .reg_base = PCF2131_REG_TS3_BASE, + .gnd_detect_reg = PCF2131_REG_CTRL4, + .gnd_detect_bit = PCF2131_BIT_CTRL4_TSF3, + .inter_detect_bit = 0, + .ie_reg = PCF2131_REG_CTRL5, + .ie_bit = PCF2131_BIT_CTRL5_TSIE3, + }, + .ts[3] = { + .reg_base = PCF2131_REG_TS4_BASE, + .gnd_detect_reg = PCF2131_REG_CTRL4, + .gnd_detect_bit = PCF2131_BIT_CTRL4_TSF4, + .inter_detect_bit = 0, + .ie_reg = PCF2131_REG_CTRL5, + .ie_bit = PCF2131_BIT_CTRL5_TSIE4, + }, + .attribute_group = { + .attrs = pcf2131_attrs, + }, + }, +}; + +/* + * Enable timestamp function and corresponding interrupt(s). + */ +static int pcf2127_enable_ts(struct device *dev, int ts_id) +{ + struct pcf2127 *pcf2127 = dev_get_drvdata(dev); + int ret; + + if (ts_id >= pcf2127->cfg->ts_count) { + dev_err(dev, "%s: invalid tamper detection ID (%d)\n", + __func__, ts_id); + return -EINVAL; + } + + /* Enable timestamp function. */ + ret = regmap_update_bits(pcf2127->regmap, + pcf2127->cfg->ts[ts_id].reg_base, + PCF2127_BIT_TS_CTRL_TSOFF | + PCF2127_BIT_TS_CTRL_TSM, + PCF2127_BIT_TS_CTRL_TSM); + if (ret) { + dev_err(dev, "%s: tamper detection config (ts%d_ctrl) failed\n", + __func__, ts_id); + return ret; + } + + /* + * Enable interrupt generation when TSF timestamp flag is set. + * Interrupt signals are open-drain outputs and can be left floating if + * unused. + */ + ret = regmap_update_bits(pcf2127->regmap, pcf2127->cfg->ts[ts_id].ie_reg, + pcf2127->cfg->ts[ts_id].ie_bit, + pcf2127->cfg->ts[ts_id].ie_bit); + if (ret) { + dev_err(dev, "%s: tamper detection TSIE%d config failed\n", + __func__, ts_id); + return ret; + } + + return ret; +} + +/* Route all interrupt sources to INT A pin. */ +static int pcf2127_configure_interrupt_pins(struct device *dev) +{ + struct pcf2127 *pcf2127 = dev_get_drvdata(dev); + int ret; + + /* Mask bits need to be cleared to enable corresponding + * interrupt source. + */ + ret = regmap_write(pcf2127->regmap, + PCF2131_REG_INT_A_MASK1, 0); + if (ret) + return ret; + + ret = regmap_write(pcf2127->regmap, + PCF2131_REG_INT_A_MASK2, 0); + if (ret) + return ret; + + return ret; +} + static int pcf2127_probe(struct device *dev, struct regmap *regmap, - int alarm_irq, const char *name, bool is_pcf2127) + int alarm_irq, const struct pcf21xx_config *config) { struct pcf2127 *pcf2127; int ret = 0; @@ -645,6 +1107,7 @@ static int pcf2127_probe(struct device *dev, struct regmap *regmap, return -ENOMEM; pcf2127->regmap = regmap; + pcf2127->cfg = config; dev_set_drvdata(dev, pcf2127); @@ -656,8 +1119,16 @@ static int pcf2127_probe(struct device *dev, struct regmap *regmap, pcf2127->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000; pcf2127->rtc->range_max = RTC_TIMESTAMP_END_2099; pcf2127->rtc->set_start_time = true; /* Sets actual start to 1970 */ - set_bit(RTC_FEATURE_ALARM_RES_2S, pcf2127->rtc->features); - clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, pcf2127->rtc->features); + + /* + * PCF2127/29 do not work correctly when setting alarms at 1s intervals. + * PCF2131 is ok. + */ + if (pcf2127->cfg->type == PCF2127 || pcf2127->cfg->type == PCF2129) { + set_bit(RTC_FEATURE_ALARM_RES_2S, pcf2127->rtc->features); + clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, pcf2127->rtc->features); + } + clear_bit(RTC_FEATURE_ALARM, pcf2127->rtc->features); if (alarm_irq > 0) { @@ -688,7 +1159,16 @@ static int pcf2127_probe(struct device *dev, struct regmap *regmap, set_bit(RTC_FEATURE_ALARM, pcf2127->rtc->features); } - if (is_pcf2127) { + if (pcf2127->cfg->has_int_a_b) { + /* Configure int A/B pins, independently of alarm_irq. */ + ret = pcf2127_configure_interrupt_pins(dev); + if (ret) { + dev_err(dev, "failed to configure interrupt pins\n"); + return ret; + } + } + + if (pcf2127->cfg->has_nvmem) { struct nvmem_config nvmem_cfg = { .priv = pcf2127, .reg_read = pcf2127_nvmem_read, @@ -703,15 +1183,17 @@ static int pcf2127_probe(struct device *dev, struct regmap *regmap, * The "Power-On Reset Override" facility prevents the RTC to do a reset * after power on. For normal operation the PORO must be disabled. */ - regmap_clear_bits(pcf2127->regmap, PCF2127_REG_CTRL1, + ret = regmap_clear_bits(pcf2127->regmap, PCF2127_REG_CTRL1, PCF2127_BIT_CTRL1_POR_OVRD); + if (ret < 0) + return ret; - ret = regmap_read(pcf2127->regmap, PCF2127_REG_CLKOUT, &val); + ret = regmap_read(pcf2127->regmap, pcf2127->cfg->reg_clkout, &val); if (ret < 0) return ret; if (!(val & PCF2127_BIT_CLKOUT_OTPR)) { - ret = regmap_set_bits(pcf2127->regmap, PCF2127_REG_CLKOUT, + ret = regmap_set_bits(pcf2127->regmap, pcf2127->cfg->reg_clkout, PCF2127_BIT_CLKOUT_OTPR); if (ret < 0) return ret; @@ -721,20 +1203,20 @@ static int pcf2127_probe(struct device *dev, struct regmap *regmap, /* * Watchdog timer enabled and reset pin /RST activated when timed out. - * Select 1Hz clock source for watchdog timer. + * Select 1Hz clock source for watchdog timer (1/4Hz for PCF2131). * Note: Countdown timer disabled and not available. - * For pca2129, pcf2129, only bit[7] is for Symbol WD_CD + * For pca2129, pcf2129 and pcf2131, only bit[7] is for Symbol WD_CD * of register watchdg_tim_ctl. The bit[6] is labeled * as T. Bits labeled as T must always be written with * logic 0. */ - ret = regmap_update_bits(pcf2127->regmap, PCF2127_REG_WD_CTL, + ret = regmap_update_bits(pcf2127->regmap, pcf2127->cfg->reg_wd_ctl, PCF2127_BIT_WD_CTL_CD1 | PCF2127_BIT_WD_CTL_CD0 | PCF2127_BIT_WD_CTL_TF1 | PCF2127_BIT_WD_CTL_TF0, PCF2127_BIT_WD_CTL_CD1 | - (is_pcf2127 ? PCF2127_BIT_WD_CTL_CD0 : 0) | + (pcf2127->cfg->has_bit_wd_ctl_cd0 ? PCF2127_BIT_WD_CTL_CD0 : 0) | PCF2127_BIT_WD_CTL_TF1); if (ret) { dev_err(dev, "%s: watchdog config (wd_ctl) failed\n", __func__); @@ -760,34 +1242,15 @@ static int pcf2127_probe(struct device *dev, struct regmap *regmap, } /* - * Enable timestamp function and store timestamp of first trigger - * event until TSF1 and TSF2 interrupt flags are cleared. + * Enable timestamp functions 1 to 4. */ - ret = regmap_update_bits(pcf2127->regmap, PCF2127_REG_TS_CTRL, - PCF2127_BIT_TS_CTRL_TSOFF | - PCF2127_BIT_TS_CTRL_TSM, - PCF2127_BIT_TS_CTRL_TSM); - if (ret) { - dev_err(dev, "%s: tamper detection config (ts_ctrl) failed\n", - __func__); - return ret; + for (int i = 0; i < pcf2127->cfg->ts_count; i++) { + ret = pcf2127_enable_ts(dev, i); + if (ret) + return ret; } - /* - * Enable interrupt generation when TSF1 or TSF2 timestamp flags - * are set. Interrupt signal is an open-drain output and can be - * left floating if unused. - */ - ret = regmap_update_bits(pcf2127->regmap, PCF2127_REG_CTRL2, - PCF2127_BIT_CTRL2_TSIE, - PCF2127_BIT_CTRL2_TSIE); - if (ret) { - dev_err(dev, "%s: tamper detection config (ctrl2) failed\n", - __func__); - return ret; - } - - ret = rtc_add_group(pcf2127->rtc, &pcf2127_attr_group); + ret = rtc_add_group(pcf2127->rtc, &pcf2127->cfg->attribute_group); if (ret) { dev_err(dev, "%s: tamper sysfs registering failed\n", __func__); @@ -799,9 +1262,10 @@ static int pcf2127_probe(struct device *dev, struct regmap *regmap, #ifdef CONFIG_OF static const struct of_device_id pcf2127_of_match[] = { - { .compatible = "nxp,pcf2127" }, - { .compatible = "nxp,pcf2129" }, - { .compatible = "nxp,pca2129" }, + { .compatible = "nxp,pcf2127", .data = &pcf21xx_cfg[PCF2127] }, + { .compatible = "nxp,pcf2129", .data = &pcf21xx_cfg[PCF2129] }, + { .compatible = "nxp,pca2129", .data = &pcf21xx_cfg[PCF2129] }, + { .compatible = "nxp,pcf2131", .data = &pcf21xx_cfg[PCF2131] }, {} }; MODULE_DEVICE_TABLE(of, pcf2127_of_match); @@ -886,26 +1350,41 @@ static const struct regmap_bus pcf2127_i2c_regmap = { static struct i2c_driver pcf2127_i2c_driver; static const struct i2c_device_id pcf2127_i2c_id[] = { - { "pcf2127", 1 }, - { "pcf2129", 0 }, - { "pca2129", 0 }, + { "pcf2127", PCF2127 }, + { "pcf2129", PCF2129 }, + { "pca2129", PCF2129 }, + { "pcf2131", PCF2131 }, { } }; MODULE_DEVICE_TABLE(i2c, pcf2127_i2c_id); static int pcf2127_i2c_probe(struct i2c_client *client) { - const struct i2c_device_id *id = i2c_match_id(pcf2127_i2c_id, client); struct regmap *regmap; - static const struct regmap_config config = { + static struct regmap_config config = { .reg_bits = 8, .val_bits = 8, - .max_register = 0x1d, }; + const struct pcf21xx_config *variant; if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) return -ENODEV; + if (client->dev.of_node) { + variant = of_device_get_match_data(&client->dev); + if (!variant) + return -ENODEV; + } else { + enum pcf21xx_type type = + i2c_match_id(pcf2127_i2c_id, client)->driver_data; + + if (type >= PCF21XX_LAST_ID) + return -ENODEV; + variant = &pcf21xx_cfg[type]; + } + + config.max_register = variant->max_register, + regmap = devm_regmap_init(&client->dev, &pcf2127_i2c_regmap, &client->dev, &config); if (IS_ERR(regmap)) { @@ -914,8 +1393,7 @@ static int pcf2127_i2c_probe(struct i2c_client *client) return PTR_ERR(regmap); } - return pcf2127_probe(&client->dev, regmap, client->irq, - pcf2127_i2c_driver.driver.name, id->driver_data); + return pcf2127_probe(&client->dev, regmap, client->irq, variant); } static struct i2c_driver pcf2127_i2c_driver = { @@ -953,17 +1431,32 @@ static void pcf2127_i2c_unregister_driver(void) #if IS_ENABLED(CONFIG_SPI_MASTER) static struct spi_driver pcf2127_spi_driver; +static const struct spi_device_id pcf2127_spi_id[]; static int pcf2127_spi_probe(struct spi_device *spi) { - static const struct regmap_config config = { + static struct regmap_config config = { .reg_bits = 8, .val_bits = 8, .read_flag_mask = 0xa0, .write_flag_mask = 0x20, - .max_register = 0x1d, }; struct regmap *regmap; + const struct pcf21xx_config *variant; + + if (spi->dev.of_node) { + variant = of_device_get_match_data(&spi->dev); + if (!variant) + return -ENODEV; + } else { + enum pcf21xx_type type = spi_get_device_id(spi)->driver_data; + + if (type >= PCF21XX_LAST_ID) + return -ENODEV; + variant = &pcf21xx_cfg[type]; + } + + config.max_register = variant->max_register, regmap = devm_regmap_init_spi(spi, &config); if (IS_ERR(regmap)) { @@ -972,15 +1465,14 @@ static int pcf2127_spi_probe(struct spi_device *spi) return PTR_ERR(regmap); } - return pcf2127_probe(&spi->dev, regmap, spi->irq, - pcf2127_spi_driver.driver.name, - spi_get_device_id(spi)->driver_data); + return pcf2127_probe(&spi->dev, regmap, spi->irq, variant); } static const struct spi_device_id pcf2127_spi_id[] = { - { "pcf2127", 1 }, - { "pcf2129", 0 }, - { "pca2129", 0 }, + { "pcf2127", PCF2127 }, + { "pcf2129", PCF2129 }, + { "pca2129", PCF2129 }, + { "pcf2131", PCF2131 }, { } }; MODULE_DEVICE_TABLE(spi, pcf2127_spi_id); @@ -1045,5 +1537,5 @@ static void __exit pcf2127_exit(void) module_exit(pcf2127_exit) MODULE_AUTHOR("Renaud Cerrato "); -MODULE_DESCRIPTION("NXP PCF2127/29 RTC driver"); +MODULE_DESCRIPTION("NXP PCF2127/29/31 RTC driver"); MODULE_LICENSE("GPL v2"); diff --git a/drivers/rtc/rtc-pcf85063.c b/drivers/rtc/rtc-pcf85063.c index e517abfaee2a..fdbc07f14036 100644 --- a/drivers/rtc/rtc-pcf85063.c +++ b/drivers/rtc/rtc-pcf85063.c @@ -14,7 +14,7 @@ #include #include #include -#include +#include #include #include @@ -514,49 +514,40 @@ static struct clk *pcf85063_clkout_register_clk(struct pcf85063 *pcf85063) } #endif -enum pcf85063_type { - PCF85063, - PCF85063TP, - PCF85063A, - RV8263, - PCF85063_LAST_ID -}; - -static struct pcf85063_config pcf85063_cfg[] = { - [PCF85063] = { - .regmap = { - .reg_bits = 8, - .val_bits = 8, - .max_register = 0x0a, - }, - }, - [PCF85063TP] = { - .regmap = { - .reg_bits = 8, - .val_bits = 8, - .max_register = 0x0a, - }, - }, - [PCF85063A] = { - .regmap = { - .reg_bits = 8, - .val_bits = 8, - .max_register = 0x11, - }, - .has_alarms = 1, - }, - [RV8263] = { - .regmap = { - .reg_bits = 8, - .val_bits = 8, - .max_register = 0x11, - }, - .has_alarms = 1, - .force_cap_7000 = 1, +static const struct pcf85063_config config_pcf85063 = { + .regmap = { + .reg_bits = 8, + .val_bits = 8, + .max_register = 0x0a, }, }; -static const struct i2c_device_id pcf85063_ids[]; +static const struct pcf85063_config config_pcf85063tp = { + .regmap = { + .reg_bits = 8, + .val_bits = 8, + .max_register = 0x0a, + }, +}; + +static const struct pcf85063_config config_pcf85063a = { + .regmap = { + .reg_bits = 8, + .val_bits = 8, + .max_register = 0x11, + }, + .has_alarms = 1, +}; + +static const struct pcf85063_config config_rv8263 = { + .regmap = { + .reg_bits = 8, + .val_bits = 8, + .max_register = 0x11, + }, + .has_alarms = 1, + .force_cap_7000 = 1, +}; static int pcf85063_probe(struct i2c_client *client) { @@ -579,17 +570,9 @@ static int pcf85063_probe(struct i2c_client *client) if (!pcf85063) return -ENOMEM; - if (client->dev.of_node) { - config = of_device_get_match_data(&client->dev); - if (!config) - return -ENODEV; - } else { - enum pcf85063_type type = - i2c_match_id(pcf85063_ids, client)->driver_data; - if (type >= PCF85063_LAST_ID) - return -ENODEV; - config = &pcf85063_cfg[type]; - } + config = i2c_get_match_data(client); + if (!config) + return -ENODEV; pcf85063->regmap = devm_regmap_init_i2c(client, &config->regmap); if (IS_ERR(pcf85063->regmap)) @@ -655,22 +638,22 @@ static int pcf85063_probe(struct i2c_client *client) } static const struct i2c_device_id pcf85063_ids[] = { - { "pca85073a", PCF85063A }, - { "pcf85063", PCF85063 }, - { "pcf85063tp", PCF85063TP }, - { "pcf85063a", PCF85063A }, - { "rv8263", RV8263 }, + { "pca85073a", .driver_data = (kernel_ulong_t)&config_pcf85063a }, + { "pcf85063", .driver_data = (kernel_ulong_t)&config_pcf85063 }, + { "pcf85063tp", .driver_data = (kernel_ulong_t)&config_pcf85063tp }, + { "pcf85063a", .driver_data = (kernel_ulong_t)&config_pcf85063a }, + { "rv8263", .driver_data = (kernel_ulong_t)&config_rv8263 }, {} }; MODULE_DEVICE_TABLE(i2c, pcf85063_ids); #ifdef CONFIG_OF static const struct of_device_id pcf85063_of_match[] = { - { .compatible = "nxp,pca85073a", .data = &pcf85063_cfg[PCF85063A] }, - { .compatible = "nxp,pcf85063", .data = &pcf85063_cfg[PCF85063] }, - { .compatible = "nxp,pcf85063tp", .data = &pcf85063_cfg[PCF85063TP] }, - { .compatible = "nxp,pcf85063a", .data = &pcf85063_cfg[PCF85063A] }, - { .compatible = "microcrystal,rv8263", .data = &pcf85063_cfg[RV8263] }, + { .compatible = "nxp,pca85073a", .data = &config_pcf85063a }, + { .compatible = "nxp,pcf85063", .data = &config_pcf85063 }, + { .compatible = "nxp,pcf85063tp", .data = &config_pcf85063tp }, + { .compatible = "nxp,pcf85063a", .data = &config_pcf85063a }, + { .compatible = "microcrystal,rv8263", .data = &config_rv8263 }, {} }; MODULE_DEVICE_TABLE(of, pcf85063_of_match); diff --git a/drivers/rtc/rtc-pcf85363.c b/drivers/rtc/rtc-pcf85363.c index 65b8b1338dbb..06194674d71c 100644 --- a/drivers/rtc/rtc-pcf85363.c +++ b/drivers/rtc/rtc-pcf85363.c @@ -15,7 +15,6 @@ #include #include #include -#include #include /* @@ -403,6 +402,7 @@ static int pcf85363_probe(struct i2c_client *client) }, }; int ret, i, err; + bool wakeup_source; if (data) config = data; @@ -432,25 +432,36 @@ static int pcf85363_probe(struct i2c_client *client) pcf85363->rtc->ops = &rtc_ops; pcf85363->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000; pcf85363->rtc->range_max = RTC_TIMESTAMP_END_2099; - clear_bit(RTC_FEATURE_ALARM, pcf85363->rtc->features); + + wakeup_source = device_property_read_bool(&client->dev, + "wakeup-source"); + if (client->irq > 0 || wakeup_source) { + regmap_write(pcf85363->regmap, CTRL_FLAGS, 0); + regmap_update_bits(pcf85363->regmap, CTRL_PIN_IO, + PIN_IO_INTA_OUT, PIN_IO_INTAPM); + } if (client->irq > 0) { unsigned long irqflags = IRQF_TRIGGER_LOW; if (dev_fwnode(&client->dev)) irqflags = 0; - - regmap_write(pcf85363->regmap, CTRL_FLAGS, 0); - regmap_update_bits(pcf85363->regmap, CTRL_PIN_IO, - PIN_IO_INTA_OUT, PIN_IO_INTAPM); ret = devm_request_threaded_irq(&client->dev, client->irq, NULL, pcf85363_rtc_handle_irq, irqflags | IRQF_ONESHOT, "pcf85363", client); - if (ret) - dev_warn(&client->dev, "unable to request IRQ, alarms disabled\n"); - else - set_bit(RTC_FEATURE_ALARM, pcf85363->rtc->features); + if (ret) { + dev_warn(&client->dev, + "unable to request IRQ, alarms disabled\n"); + client->irq = 0; + } + } + + if (client->irq > 0 || wakeup_source) { + device_init_wakeup(&client->dev, true); + set_bit(RTC_FEATURE_ALARM, pcf85363->rtc->features); + } else { + clear_bit(RTC_FEATURE_ALARM, pcf85363->rtc->features); } ret = devm_rtc_register_device(pcf85363->rtc); diff --git a/drivers/rtc/rtc-pxa.c b/drivers/rtc/rtc-pxa.c index eeacf480cf36..e400c78252e8 100644 --- a/drivers/rtc/rtc-pxa.c +++ b/drivers/rtc/rtc-pxa.c @@ -14,7 +14,6 @@ #include #include #include -#include #include "rtc-sa1100.h" diff --git a/drivers/rtc/rtc-rs5c372.c b/drivers/rtc/rtc-rs5c372.c index a5a6c8772ecd..f8fab0205f8c 100644 --- a/drivers/rtc/rtc-rs5c372.c +++ b/drivers/rtc/rtc-rs5c372.c @@ -12,7 +12,7 @@ #include #include #include -#include +#include /* * Ricoh has a family of I2C based RTCs, which differ only slightly from @@ -826,8 +826,7 @@ static int rs5c372_probe(struct i2c_client *client) rs5c372->client = client; i2c_set_clientdata(client, rs5c372); if (client->dev.of_node) { - rs5c372->type = (enum rtc_type) - of_device_get_match_data(&client->dev); + rs5c372->type = (uintptr_t)of_device_get_match_data(&client->dev); } else { const struct i2c_device_id *id = i2c_match_id(rs5c372_id, client); rs5c372->type = id->driver_data; diff --git a/drivers/rtc/rtc-rv3028.c b/drivers/rtc/rtc-rv3028.c index 076e56f4e01a..2f001c59c61d 100644 --- a/drivers/rtc/rtc-rv3028.c +++ b/drivers/rtc/rtc-rv3028.c @@ -17,7 +17,7 @@ #include #include #include -#include +#include #include #include @@ -855,11 +855,68 @@ static const struct regmap_config regmap_config = { .max_register = 0x37, }; +static u8 rv3028_set_trickle_charger(struct rv3028_data *rv3028, + struct i2c_client *client) +{ + int ret, val_old, val; + u32 ohms, chargeable; + + ret = regmap_read(rv3028->regmap, RV3028_BACKUP, &val_old); + if (ret < 0) + return ret; + + /* mask out only trickle charger bits */ + val_old = val_old & (RV3028_BACKUP_TCE | RV3028_BACKUP_TCR_MASK); + val = val_old; + + /* setup trickle charger */ + if (!device_property_read_u32(&client->dev, "trickle-resistor-ohms", + &ohms)) { + int i; + + for (i = 0; i < ARRAY_SIZE(rv3028_trickle_resistors); i++) + if (ohms == rv3028_trickle_resistors[i]) + break; + + if (i < ARRAY_SIZE(rv3028_trickle_resistors)) { + /* enable trickle charger and its resistor */ + val = RV3028_BACKUP_TCE | i; + } else { + dev_warn(&client->dev, "invalid trickle resistor value\n"); + } + } + + if (!device_property_read_u32(&client->dev, "aux-voltage-chargeable", + &chargeable)) { + switch (chargeable) { + case 0: + val &= ~RV3028_BACKUP_TCE; + break; + case 1: + val |= RV3028_BACKUP_TCE; + break; + default: + dev_warn(&client->dev, + "unsupported aux-voltage-chargeable value\n"); + break; + } + } + + /* only update EEPROM if changes are necessary */ + if (val_old != val) { + ret = rv3028_update_cfg(rv3028, RV3028_BACKUP, RV3028_BACKUP_TCE | + RV3028_BACKUP_TCR_MASK, val); + if (ret) + return ret; + } + + return ret; +} + static int rv3028_probe(struct i2c_client *client) { struct rv3028_data *rv3028; int ret, status; - u32 ohms; struct nvmem_config nvmem_cfg = { .name = "rv3028_nvram", .word_size = 1, @@ -937,24 +994,9 @@ static int rv3028_probe(struct i2c_client *client) if (ret) return ret; - /* setup trickle charger */ - if (!device_property_read_u32(&client->dev, "trickle-resistor-ohms", - &ohms)) { - int i; - - for (i = 0; i < ARRAY_SIZE(rv3028_trickle_resistors); i++) - if (ohms == rv3028_trickle_resistors[i]) - break; - - if (i < ARRAY_SIZE(rv3028_trickle_resistors)) { - ret = rv3028_update_cfg(rv3028, RV3028_BACKUP, RV3028_BACKUP_TCE | - RV3028_BACKUP_TCR_MASK, RV3028_BACKUP_TCE | i); - if (ret) - return ret; - } else { - dev_warn(&client->dev, "invalid trickle resistor value\n"); - } - } + ret = rv3028_set_trickle_charger(rv3028, client); + if (ret) + return ret; ret = rtc_add_group(rv3028->rtc, &rv3028_attr_group); if (ret) diff --git a/drivers/rtc/rtc-rv3032.c b/drivers/rtc/rtc-rv3032.c index 6b8eb2039a33..35b2e36b426a 100644 --- a/drivers/rtc/rtc-rv3032.c +++ b/drivers/rtc/rtc-rv3032.c @@ -19,7 +19,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/rtc/rtc-rv8803.c b/drivers/rtc/rtc-rv8803.c index 98679cae13e8..1a3ec1bb5b81 100644 --- a/drivers/rtc/rtc-rv8803.c +++ b/drivers/rtc/rtc-rv8803.c @@ -15,7 +15,7 @@ #include #include #include -#include +#include #include #define RV8803_I2C_TRY_COUNT 4 @@ -645,8 +645,7 @@ static int rv8803_probe(struct i2c_client *client) mutex_init(&rv8803->flags_lock); rv8803->client = client; if (client->dev.of_node) { - rv8803->type = (enum rv8803_type) - of_device_get_match_data(&client->dev); + rv8803->type = (uintptr_t)of_device_get_match_data(&client->dev); } else { const struct i2c_device_id *id = i2c_match_id(rv8803_id, client); diff --git a/drivers/rtc/rtc-rx6110.c b/drivers/rtc/rtc-rx6110.c index 8702db6096ba..834274db8c3f 100644 --- a/drivers/rtc/rtc-rx6110.c +++ b/drivers/rtc/rtc-rx6110.c @@ -13,7 +13,6 @@ #include #include #include -#include #include #include diff --git a/drivers/rtc/rtc-rx8581.c b/drivers/rtc/rtc-rx8581.c index 82881fd2e14a..48efd61a114d 100644 --- a/drivers/rtc/rtc-rx8581.c +++ b/drivers/rtc/rtc-rx8581.c @@ -13,7 +13,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/rtc/rtc-rzn1.c b/drivers/rtc/rtc-rzn1.c index dca736caba85..56ebbd4d0481 100644 --- a/drivers/rtc/rtc-rzn1.c +++ b/drivers/rtc/rtc-rzn1.c @@ -15,7 +15,7 @@ #include #include #include -#include +#include #include #include #include @@ -227,7 +227,7 @@ static int rzn1_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) return ret; /* We cannot set alarms more than one week ahead */ - farest = rtc_tm_to_time64(&tm_now) + (7 * 86400); + farest = rtc_tm_to_time64(&tm_now) + rtc->rtcdev->alarm_offset_max; alarm = rtc_tm_to_time64(tm); if (time_after(alarm, farest)) return -ERANGE; @@ -351,6 +351,7 @@ static int rzn1_rtc_probe(struct platform_device *pdev) rtc->rtcdev->range_min = RTC_TIMESTAMP_BEGIN_2000; rtc->rtcdev->range_max = RTC_TIMESTAMP_END_2099; + rtc->rtcdev->alarm_offset_max = 7 * 86400; rtc->rtcdev->ops = &rzn1_rtc_ops; set_bit(RTC_FEATURE_ALARM_RES_MINUTE, rtc->rtcdev->features); clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, rtc->rtcdev->features); diff --git a/drivers/rtc/rtc-s3c.c b/drivers/rtc/rtc-s3c.c index 70e1a18e5efd..282238818f63 100644 --- a/drivers/rtc/rtc-s3c.c +++ b/drivers/rtc/rtc-s3c.c @@ -23,7 +23,6 @@ #include #include #include -#include #include #include diff --git a/drivers/rtc/rtc-stm32.c b/drivers/rtc/rtc-stm32.c index 3d36e11cff80..76753c71d92e 100644 --- a/drivers/rtc/rtc-stm32.c +++ b/drivers/rtc/rtc-stm32.c @@ -6,11 +6,13 @@ #include #include +#include #include #include #include #include -#include +#include +#include #include #include #include @@ -89,6 +91,9 @@ /* Max STM32 RTC register offset is 0x3FC */ #define UNDEF_REG 0xFFFF +/* STM32 RTC driver time helpers */ +#define SEC_PER_DAY (24 * 60 * 60) + struct stm32_rtc; struct stm32_rtc_registers { @@ -114,6 +119,7 @@ struct stm32_rtc_data { void (*clear_events)(struct stm32_rtc *rtc, unsigned int flags); bool has_pclk; bool need_dbp; + bool need_accuracy; }; struct stm32_rtc { @@ -158,10 +164,9 @@ static int stm32_rtc_enter_init_mode(struct stm32_rtc *rtc) * slowest rtc_ck frequency may be 32kHz and highest should be * 1MHz, we poll every 10 us with a timeout of 100ms. */ - return readl_relaxed_poll_timeout_atomic( - rtc->base + regs->isr, - isr, (isr & STM32_RTC_ISR_INITF), - 10, 100000); + return readl_relaxed_poll_timeout_atomic(rtc->base + regs->isr, isr, + (isr & STM32_RTC_ISR_INITF), + 10, 100000); } return 0; @@ -425,40 +430,42 @@ static int stm32_rtc_alarm_irq_enable(struct device *dev, unsigned int enabled) return 0; } -static int stm32_rtc_valid_alrm(struct stm32_rtc *rtc, struct rtc_time *tm) +static int stm32_rtc_valid_alrm(struct device *dev, struct rtc_time *tm) { - const struct stm32_rtc_registers *regs = &rtc->data->regs; - int cur_day, cur_mon, cur_year, cur_hour, cur_min, cur_sec; - unsigned int dr = readl_relaxed(rtc->base + regs->dr); - unsigned int tr = readl_relaxed(rtc->base + regs->tr); - - cur_day = (dr & STM32_RTC_DR_DATE) >> STM32_RTC_DR_DATE_SHIFT; - cur_mon = (dr & STM32_RTC_DR_MONTH) >> STM32_RTC_DR_MONTH_SHIFT; - cur_year = (dr & STM32_RTC_DR_YEAR) >> STM32_RTC_DR_YEAR_SHIFT; - cur_sec = (tr & STM32_RTC_TR_SEC) >> STM32_RTC_TR_SEC_SHIFT; - cur_min = (tr & STM32_RTC_TR_MIN) >> STM32_RTC_TR_MIN_SHIFT; - cur_hour = (tr & STM32_RTC_TR_HOUR) >> STM32_RTC_TR_HOUR_SHIFT; + static struct rtc_time now; + time64_t max_alarm_time64; + int max_day_forward; + int next_month; + int next_year; /* * Assuming current date is M-D-Y H:M:S. * RTC alarm can't be set on a specific month and year. * So the valid alarm range is: * M-D-Y H:M:S < alarm <= (M+1)-D-Y H:M:S - * with a specific case for December... */ - if ((((tm->tm_year > cur_year) && - (tm->tm_mon == 0x1) && (cur_mon == 0x12)) || - ((tm->tm_year == cur_year) && - (tm->tm_mon <= cur_mon + 1))) && - ((tm->tm_mday > cur_day) || - ((tm->tm_mday == cur_day) && - ((tm->tm_hour > cur_hour) || - ((tm->tm_hour == cur_hour) && (tm->tm_min > cur_min)) || - ((tm->tm_hour == cur_hour) && (tm->tm_min == cur_min) && - (tm->tm_sec >= cur_sec)))))) - return 0; + stm32_rtc_read_time(dev, &now); - return -EINVAL; + /* + * Find the next month and the year of the next month. + * Note: tm_mon and next_month are from 0 to 11 + */ + next_month = now.tm_mon + 1; + if (next_month == 12) { + next_month = 0; + next_year = now.tm_year + 1; + } else { + next_year = now.tm_year; + } + + /* Find the maximum limit of alarm in days. */ + max_day_forward = rtc_month_days(now.tm_mon, now.tm_year) + - now.tm_mday + + min(rtc_month_days(next_month, next_year), now.tm_mday); + + /* Convert to timestamp and compare the alarm time and its upper limit */ + max_alarm_time64 = rtc_tm_to_time64(&now) + max_day_forward * SEC_PER_DAY; + return rtc_tm_to_time64(tm) <= max_alarm_time64 ? 0 : -EINVAL; } static int stm32_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) @@ -469,17 +476,17 @@ static int stm32_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) unsigned int cr, isr, alrmar; int ret = 0; - tm2bcd(tm); - /* * RTC alarm can't be set on a specific date, unless this date is * up to the same day of month next month. */ - if (stm32_rtc_valid_alrm(rtc, tm) < 0) { + if (stm32_rtc_valid_alrm(dev, tm) < 0) { dev_err(dev, "Alarm can be set only on upcoming month.\n"); return -EINVAL; } + tm2bcd(tm); + alrmar = 0; /* tm_year and tm_mon are not used because not supported by RTC */ alrmar |= (tm->tm_mday << STM32_RTC_ALRMXR_DATE_SHIFT) & @@ -545,6 +552,7 @@ static void stm32_rtc_clear_events(struct stm32_rtc *rtc, static const struct stm32_rtc_data stm32_rtc_data = { .has_pclk = false, .need_dbp = true, + .need_accuracy = false, .regs = { .tr = 0x00, .dr = 0x04, @@ -566,6 +574,7 @@ static const struct stm32_rtc_data stm32_rtc_data = { static const struct stm32_rtc_data stm32h7_rtc_data = { .has_pclk = true, .need_dbp = true, + .need_accuracy = false, .regs = { .tr = 0x00, .dr = 0x04, @@ -596,6 +605,7 @@ static void stm32mp1_rtc_clear_events(struct stm32_rtc *rtc, static const struct stm32_rtc_data stm32mp1_data = { .has_pclk = true, .need_dbp = false, + .need_accuracy = true, .regs = { .tr = 0x00, .dr = 0x04, @@ -628,7 +638,7 @@ static int stm32_rtc_init(struct platform_device *pdev, const struct stm32_rtc_registers *regs = &rtc->data->regs; unsigned int prer, pred_a, pred_s, pred_a_max, pred_s_max, cr; unsigned int rate; - int ret = 0; + int ret; rate = clk_get_rate(rtc->rtc_ck); @@ -636,18 +646,32 @@ static int stm32_rtc_init(struct platform_device *pdev, pred_a_max = STM32_RTC_PRER_PRED_A >> STM32_RTC_PRER_PRED_A_SHIFT; pred_s_max = STM32_RTC_PRER_PRED_S >> STM32_RTC_PRER_PRED_S_SHIFT; - for (pred_a = pred_a_max; pred_a + 1 > 0; pred_a--) { - pred_s = (rate / (pred_a + 1)) - 1; + if (rate > (pred_a_max + 1) * (pred_s_max + 1)) { + dev_err(&pdev->dev, "rtc_ck rate is too high: %dHz\n", rate); + return -EINVAL; + } - if (((pred_s + 1) * (pred_a + 1)) == rate) - break; + if (rtc->data->need_accuracy) { + for (pred_a = 0; pred_a <= pred_a_max; pred_a++) { + pred_s = (rate / (pred_a + 1)) - 1; + + if (pred_s <= pred_s_max && ((pred_s + 1) * (pred_a + 1)) == rate) + break; + } + } else { + for (pred_a = pred_a_max; pred_a + 1 > 0; pred_a--) { + pred_s = (rate / (pred_a + 1)) - 1; + + if (((pred_s + 1) * (pred_a + 1)) == rate) + break; + } } /* * Can't find a 1Hz, so give priority to RTC power consumption * by choosing the higher possible value for prediv_a */ - if ((pred_s > pred_s_max) || (pred_a > pred_a_max)) { + if (pred_s > pred_s_max || pred_a > pred_a_max) { pred_a = pred_a_max; pred_s = (rate / (pred_a + 1)) - 1; @@ -656,6 +680,20 @@ static int stm32_rtc_init(struct platform_device *pdev, "fast" : "slow"); } + cr = readl_relaxed(rtc->base + regs->cr); + + prer = readl_relaxed(rtc->base + regs->prer); + prer &= STM32_RTC_PRER_PRED_S | STM32_RTC_PRER_PRED_A; + + pred_s = (pred_s << STM32_RTC_PRER_PRED_S_SHIFT) & + STM32_RTC_PRER_PRED_S; + pred_a = (pred_a << STM32_RTC_PRER_PRED_A_SHIFT) & + STM32_RTC_PRER_PRED_A; + + /* quit if there is nothing to initialize */ + if ((cr & STM32_RTC_CR_FMT) == 0 && prer == (pred_s | pred_a)) + return 0; + stm32_rtc_wpr_unlock(rtc); ret = stm32_rtc_enter_init_mode(rtc); @@ -665,13 +703,10 @@ static int stm32_rtc_init(struct platform_device *pdev, goto end; } - prer = (pred_s << STM32_RTC_PRER_PRED_S_SHIFT) & STM32_RTC_PRER_PRED_S; - writel_relaxed(prer, rtc->base + regs->prer); - prer |= (pred_a << STM32_RTC_PRER_PRED_A_SHIFT) & STM32_RTC_PRER_PRED_A; - writel_relaxed(prer, rtc->base + regs->prer); + writel_relaxed(pred_s, rtc->base + regs->prer); + writel_relaxed(pred_a | pred_s, rtc->base + regs->prer); /* Force 24h time format */ - cr = readl_relaxed(rtc->base + regs->cr); cr &= ~STM32_RTC_CR_FMT; writel_relaxed(cr, rtc->base + regs->cr); @@ -730,16 +765,13 @@ static int stm32_rtc_probe(struct platform_device *pdev) rtc->rtc_ck = devm_clk_get(&pdev->dev, NULL); } else { rtc->pclk = devm_clk_get(&pdev->dev, "pclk"); - if (IS_ERR(rtc->pclk)) { - dev_err(&pdev->dev, "no pclk clock"); - return PTR_ERR(rtc->pclk); - } + if (IS_ERR(rtc->pclk)) + return dev_err_probe(&pdev->dev, PTR_ERR(rtc->pclk), "no pclk clock"); + rtc->rtc_ck = devm_clk_get(&pdev->dev, "rtc_ck"); } - if (IS_ERR(rtc->rtc_ck)) { - dev_err(&pdev->dev, "no rtc_ck clock"); - return PTR_ERR(rtc->rtc_ck); - } + if (IS_ERR(rtc->rtc_ck)) + return dev_err_probe(&pdev->dev, PTR_ERR(rtc->rtc_ck), "no rtc_ck clock"); if (rtc->data->has_pclk) { ret = clk_prepare_enable(rtc->pclk); @@ -859,7 +891,6 @@ static void stm32_rtc_remove(struct platform_device *pdev) device_init_wakeup(&pdev->dev, false); } -#ifdef CONFIG_PM_SLEEP static int stm32_rtc_suspend(struct device *dev) { struct stm32_rtc *rtc = dev_get_drvdata(dev); @@ -890,10 +921,10 @@ static int stm32_rtc_resume(struct device *dev) return ret; } -#endif -static SIMPLE_DEV_PM_OPS(stm32_rtc_pm_ops, - stm32_rtc_suspend, stm32_rtc_resume); +static const struct dev_pm_ops stm32_rtc_pm_ops = { + NOIRQ_SYSTEM_SLEEP_PM_OPS(stm32_rtc_suspend, stm32_rtc_resume) +}; static struct platform_driver stm32_rtc_driver = { .probe = stm32_rtc_probe, diff --git a/drivers/rtc/rtc-stmp3xxx.c b/drivers/rtc/rtc-stmp3xxx.c index 6f11b745f34d..7566d0a44af8 100644 --- a/drivers/rtc/rtc-stmp3xxx.c +++ b/drivers/rtc/rtc-stmp3xxx.c @@ -18,7 +18,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/rtc/rtc-sun6i.c b/drivers/rtc/rtc-sun6i.c index 71548dd59a3a..8e0c66906103 100644 --- a/drivers/rtc/rtc-sun6i.c +++ b/drivers/rtc/rtc-sun6i.c @@ -24,7 +24,6 @@ #include #include #include -#include #include #include #include @@ -847,8 +846,6 @@ static int sun6i_rtc_probe(struct platform_device *pdev) if (ret) return ret; - dev_info(&pdev->dev, "RTC enabled\n"); - return 0; } diff --git a/drivers/rtc/rtc-sunplus.c b/drivers/rtc/rtc-sunplus.c index f33dc301f301..20c7e97c2fc8 100644 --- a/drivers/rtc/rtc-sunplus.c +++ b/drivers/rtc/rtc-sunplus.c @@ -244,7 +244,7 @@ static int sp_rtc_probe(struct platform_device *plat_dev) sp_rtc->irq = platform_get_irq(plat_dev, 0); if (sp_rtc->irq < 0) - return dev_err_probe(&plat_dev->dev, sp_rtc->irq, "platform_get_irq failed\n"); + return sp_rtc->irq; ret = devm_request_irq(&plat_dev->dev, sp_rtc->irq, sp_rtc_irq_handler, IRQF_TRIGGER_RISING, "rtc irq", plat_dev); diff --git a/drivers/rtc/rtc-sunxi.c b/drivers/rtc/rtc-sunxi.c index 5d019e3a835a..5cab9953c44f 100644 --- a/drivers/rtc/rtc-sunxi.c +++ b/drivers/rtc/rtc-sunxi.c @@ -14,8 +14,6 @@ #include #include #include -#include -#include #include #include #include diff --git a/drivers/rtc/rtc-ti-k3.c b/drivers/rtc/rtc-ti-k3.c index 0d90fe923355..ec759d8f7023 100644 --- a/drivers/rtc/rtc-ti-k3.c +++ b/drivers/rtc/rtc-ti-k3.c @@ -9,7 +9,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/rtc/rtc-tps6586x.c b/drivers/rtc/rtc-tps6586x.c index 9f14e2475747..20faf08c254c 100644 --- a/drivers/rtc/rtc-tps6586x.c +++ b/drivers/rtc/rtc-tps6586x.c @@ -252,6 +252,7 @@ static int tps6586x_rtc_probe(struct platform_device *pdev) rtc->rtc->ops = &tps6586x_rtc_ops; rtc->rtc->range_max = (1ULL << 30) - 1; /* 30-bit seconds */ + rtc->rtc->alarm_offset_max = ALM1_VALID_RANGE_IN_SEC; rtc->rtc->start_secs = mktime64(2009, 1, 1, 0, 0, 0); rtc->rtc->set_start_time = true; diff --git a/drivers/rtc/rtc-tps65910.c b/drivers/rtc/rtc-tps65910.c index 75e4c2d777b9..411ff66c0468 100644 --- a/drivers/rtc/rtc-tps65910.c +++ b/drivers/rtc/rtc-tps65910.c @@ -406,11 +406,8 @@ static int tps65910_rtc_probe(struct platform_device *pdev) platform_set_drvdata(pdev, tps_rtc); irq = platform_get_irq(pdev, 0); - if (irq <= 0) { - dev_warn(&pdev->dev, "Wake up is not possible as irq = %d\n", - irq); - return -ENXIO; - } + if (irq < 0) + return irq; ret = devm_request_threaded_irq(&pdev->dev, irq, NULL, tps65910_rtc_interrupt, IRQF_TRIGGER_LOW, diff --git a/drivers/rtc/rtc-twl.c b/drivers/rtc/rtc-twl.c index 81b36948c2fa..13f8ce08243c 100644 --- a/drivers/rtc/rtc-twl.c +++ b/drivers/rtc/rtc-twl.c @@ -487,11 +487,24 @@ static const struct rtc_class_ops twl_rtc_ops = { .alarm_irq_enable = twl_rtc_alarm_irq_enable, }; +static int twl_nvram_read(void *priv, unsigned int offset, void *val, + size_t bytes) +{ + return twl_i2c_read((long)priv, val, offset, bytes); +} + +static int twl_nvram_write(void *priv, unsigned int offset, void *val, + size_t bytes) +{ + return twl_i2c_write((long)priv, val, offset, bytes); +} + /*----------------------------------------------------------------------*/ static int twl_rtc_probe(struct platform_device *pdev) { struct twl_rtc *twl_rtc; + struct nvmem_config nvmem_cfg; struct device_node *np = pdev->dev.of_node; int ret = -EINVAL; int irq = platform_get_irq(pdev, 0); @@ -542,7 +555,6 @@ static int twl_rtc_probe(struct platform_device *pdev) REG_INT_MSK_STS_A); } - dev_info(&pdev->dev, "Enabling TWL-RTC\n"); ret = twl_rtc_write_u8(twl_rtc, BIT_RTC_CTRL_REG_STOP_RTC_M, REG_RTC_CTRL_REG); if (ret < 0) @@ -564,11 +576,8 @@ static int twl_rtc_probe(struct platform_device *pdev) twl_rtc->rtc = devm_rtc_device_register(&pdev->dev, pdev->name, &twl_rtc_ops, THIS_MODULE); - if (IS_ERR(twl_rtc->rtc)) { - dev_err(&pdev->dev, "can't register RTC device, err %ld\n", - PTR_ERR(twl_rtc->rtc)); + if (IS_ERR(twl_rtc->rtc)) return PTR_ERR(twl_rtc->rtc); - } ret = devm_request_threaded_irq(&pdev->dev, irq, NULL, twl_rtc_interrupt, @@ -579,6 +588,30 @@ static int twl_rtc_probe(struct platform_device *pdev) return ret; } + memset(&nvmem_cfg, 0, sizeof(nvmem_cfg)); + nvmem_cfg.name = "twl-secured-"; + nvmem_cfg.type = NVMEM_TYPE_BATTERY_BACKED; + nvmem_cfg.reg_read = twl_nvram_read, + nvmem_cfg.reg_write = twl_nvram_write, + nvmem_cfg.word_size = 1; + nvmem_cfg.stride = 1; + if (twl_class_is_4030()) { + /* 20 bytes SECURED_REG area */ + nvmem_cfg.size = 20; + nvmem_cfg.priv = (void *)TWL_MODULE_SECURED_REG; + devm_rtc_nvmem_register(twl_rtc->rtc, &nvmem_cfg); + /* 8 bytes BACKUP area */ + nvmem_cfg.name = "twl-backup-"; + nvmem_cfg.size = 8; + nvmem_cfg.priv = (void *)TWL4030_MODULE_BACKUP; + devm_rtc_nvmem_register(twl_rtc->rtc, &nvmem_cfg); + } else { + /* 8 bytes SECURED_REG area */ + nvmem_cfg.size = 8; + nvmem_cfg.priv = (void *)TWL_MODULE_SECURED_REG; + devm_rtc_nvmem_register(twl_rtc->rtc, &nvmem_cfg); + } + return 0; } diff --git a/drivers/rtc/rtc-wm8350.c b/drivers/rtc/rtc-wm8350.c index 947f8071803f..3c773cff2b39 100644 --- a/drivers/rtc/rtc-wm8350.c +++ b/drivers/rtc/rtc-wm8350.c @@ -386,8 +386,6 @@ static int wm8350_rtc_probe(struct platform_device *pdev) /* enable the RTC if it's not already enabled */ power5 = wm8350_reg_read(wm8350, WM8350_POWER_MGMT_5); if (!(power5 & WM8350_RTC_TICK_ENA)) { - dev_info(wm8350->dev, "Starting RTC\n"); - wm8350_reg_unlock(wm8350); ret = wm8350_set_bits(wm8350, WM8350_POWER_MGMT_5, @@ -426,11 +424,8 @@ static int wm8350_rtc_probe(struct platform_device *pdev) wm_rtc->rtc = devm_rtc_device_register(&pdev->dev, "wm8350", &wm8350_rtc_ops, THIS_MODULE); - if (IS_ERR(wm_rtc->rtc)) { - ret = PTR_ERR(wm_rtc->rtc); - dev_err(&pdev->dev, "failed to register RTC: %d\n", ret); - return ret; - } + if (IS_ERR(wm_rtc->rtc)) + return PTR_ERR(wm_rtc->rtc); ret = wm8350_register_irq(wm8350, WM8350_IRQ_RTC_SEC, wm8350_rtc_update_handler, 0, diff --git a/drivers/s390/block/dasd_devmap.c b/drivers/s390/block/dasd_devmap.c index 620fab01b710..c4e36650c426 100644 --- a/drivers/s390/block/dasd_devmap.c +++ b/drivers/s390/block/dasd_devmap.c @@ -1378,16 +1378,12 @@ static ssize_t dasd_vendor_show(struct device *dev, static DEVICE_ATTR(vendor, 0444, dasd_vendor_show, NULL); -#define UID_STRLEN ( /* vendor */ 3 + 1 + /* serial */ 14 + 1 +\ - /* SSID */ 4 + 1 + /* unit addr */ 2 + 1 +\ - /* vduit */ 32 + 1) - static ssize_t dasd_uid_show(struct device *dev, struct device_attribute *attr, char *buf) { + char uid_string[DASD_UID_STRLEN]; struct dasd_device *device; struct dasd_uid uid; - char uid_string[UID_STRLEN]; char ua_string[3]; device = dasd_device_from_cdev(to_ccwdev(dev)); diff --git a/drivers/s390/block/dasd_eckd.c b/drivers/s390/block/dasd_eckd.c index 8587e423169e..bd89b032968a 100644 --- a/drivers/s390/block/dasd_eckd.c +++ b/drivers/s390/block/dasd_eckd.c @@ -1079,12 +1079,12 @@ static void dasd_eckd_get_uid_string(struct dasd_conf *conf, create_uid(conf, &uid); if (strlen(uid.vduit) > 0) - snprintf(print_uid, sizeof(*print_uid), + snprintf(print_uid, DASD_UID_STRLEN, "%s.%s.%04x.%02x.%s", uid.vendor, uid.serial, uid.ssid, uid.real_unit_addr, uid.vduit); else - snprintf(print_uid, sizeof(*print_uid), + snprintf(print_uid, DASD_UID_STRLEN, "%s.%s.%04x.%02x", uid.vendor, uid.serial, uid.ssid, uid.real_unit_addr); @@ -1093,8 +1093,8 @@ static void dasd_eckd_get_uid_string(struct dasd_conf *conf, static int dasd_eckd_check_cabling(struct dasd_device *device, void *conf_data, __u8 lpm) { + char print_path_uid[DASD_UID_STRLEN], print_device_uid[DASD_UID_STRLEN]; struct dasd_eckd_private *private = device->private; - char print_path_uid[60], print_device_uid[60]; struct dasd_conf path_conf; path_conf.data = conf_data; @@ -1293,9 +1293,9 @@ static void dasd_eckd_path_available_action(struct dasd_device *device, __u8 path_rcd_buf[DASD_ECKD_RCD_DATA_SIZE]; __u8 lpm, opm, npm, ppm, epm, hpfpm, cablepm; struct dasd_conf_data *conf_data; + char print_uid[DASD_UID_STRLEN]; struct dasd_conf path_conf; unsigned long flags; - char print_uid[60]; int rc, pos; opm = 0; @@ -5855,8 +5855,8 @@ static void dasd_eckd_dump_sense(struct dasd_device *device, static int dasd_eckd_reload_device(struct dasd_device *device) { struct dasd_eckd_private *private = device->private; + char print_uid[DASD_UID_STRLEN]; int rc, old_base; - char print_uid[60]; struct dasd_uid uid; unsigned long flags; diff --git a/drivers/s390/block/dasd_int.h b/drivers/s390/block/dasd_int.h index 0aa56351da72..8a4dbe9d7741 100644 --- a/drivers/s390/block/dasd_int.h +++ b/drivers/s390/block/dasd_int.h @@ -259,6 +259,10 @@ struct dasd_uid { char vduit[33]; }; +#define DASD_UID_STRLEN ( /* vendor */ 3 + 1 + /* serial */ 14 + 1 + \ + /* SSID */ 4 + 1 + /* unit addr */ 2 + 1 + \ + /* vduit */ 32 + 1) + /* * PPRC Status data */ diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c index 06bcb6c78909..4b7ecd4fd431 100644 --- a/drivers/s390/block/dcssblk.c +++ b/drivers/s390/block/dcssblk.c @@ -411,13 +411,13 @@ removeseg: segment_unload(entry->segment_name); } list_del(&dev_info->lh); + up_write(&dcssblk_devices_sem); dax_remove_host(dev_info->gd); kill_dax(dev_info->dax_dev); put_dax(dev_info->dax_dev); del_gendisk(dev_info->gd); put_disk(dev_info->gd); - up_write(&dcssblk_devices_sem); if (device_remove_file_self(dev, attr)) { device_unregister(dev); @@ -790,18 +790,17 @@ dcssblk_remove_store(struct device *dev, struct device_attribute *attr, const ch } list_del(&dev_info->lh); + /* unload all related segments */ + list_for_each_entry(entry, &dev_info->seg_list, lh) + segment_unload(entry->segment_name); + up_write(&dcssblk_devices_sem); + dax_remove_host(dev_info->gd); kill_dax(dev_info->dax_dev); put_dax(dev_info->dax_dev); del_gendisk(dev_info->gd); put_disk(dev_info->gd); - /* unload all related segments */ - list_for_each_entry(entry, &dev_info->seg_list, lh) - segment_unload(entry->segment_name); - - up_write(&dcssblk_devices_sem); - device_unregister(&dev_info->dev); put_device(&dev_info->dev); diff --git a/drivers/s390/char/monreader.c b/drivers/s390/char/monreader.c index 9fa92e45e0ee..7207a7f5842a 100644 --- a/drivers/s390/char/monreader.c +++ b/drivers/s390/char/monreader.c @@ -111,7 +111,7 @@ static inline unsigned long mon_mca_end(struct mon_msg *monmsg) static inline u8 mon_mca_type(struct mon_msg *monmsg, u8 index) { - return *((u8 *) mon_mca_start(monmsg) + monmsg->mca_offset + index); + return *((u8 *)__va(mon_mca_start(monmsg)) + monmsg->mca_offset + index); } static inline u32 mon_mca_size(struct mon_msg *monmsg) @@ -121,12 +121,12 @@ static inline u32 mon_mca_size(struct mon_msg *monmsg) static inline u32 mon_rec_start(struct mon_msg *monmsg) { - return *((u32 *) (mon_mca_start(monmsg) + monmsg->mca_offset + 4)); + return *((u32 *)(__va(mon_mca_start(monmsg)) + monmsg->mca_offset + 4)); } static inline u32 mon_rec_end(struct mon_msg *monmsg) { - return *((u32 *) (mon_mca_start(monmsg) + monmsg->mca_offset + 8)); + return *((u32 *)(__va(mon_mca_start(monmsg)) + monmsg->mca_offset + 8)); } static int mon_check_mca(struct mon_msg *monmsg) @@ -392,8 +392,7 @@ static ssize_t mon_read(struct file *filp, char __user *data, mce_start = mon_mca_start(monmsg) + monmsg->mca_offset; if ((monmsg->pos >= mce_start) && (monmsg->pos < mce_start + 12)) { count = min(count, (size_t) mce_start + 12 - monmsg->pos); - ret = copy_to_user(data, (void *) (unsigned long) monmsg->pos, - count); + ret = copy_to_user(data, __va(monmsg->pos), count); if (ret) return -EFAULT; monmsg->pos += count; @@ -406,8 +405,7 @@ static ssize_t mon_read(struct file *filp, char __user *data, if (monmsg->pos <= mon_rec_end(monmsg)) { count = min(count, (size_t) mon_rec_end(monmsg) - monmsg->pos + 1); - ret = copy_to_user(data, (void *) (unsigned long) monmsg->pos, - count); + ret = copy_to_user(data, __va(monmsg->pos), count); if (ret) return -EFAULT; monmsg->pos += count; diff --git a/drivers/s390/cio/airq.c b/drivers/s390/cio/airq.c index 34967e67249e..a108f2bf5b33 100644 --- a/drivers/s390/cio/airq.c +++ b/drivers/s390/cio/airq.c @@ -49,8 +49,6 @@ int register_adapter_interrupt(struct airq_struct *airq) return -ENOMEM; airq->flags |= AIRQ_PTR_ALLOCATED; } - if (!airq->lsi_mask) - airq->lsi_mask = 0xff; snprintf(dbf_txt, sizeof(dbf_txt), "rairq:%p", airq); CIO_TRACE_EVENT(4, dbf_txt); isc_register(airq->isc); @@ -98,7 +96,7 @@ static irqreturn_t do_airq_interrupt(int irq, void *dummy) head = &airq_lists[tpi_info->isc]; rcu_read_lock(); hlist_for_each_entry_rcu(airq, head, list) - if ((*airq->lsi_ptr & airq->lsi_mask) != 0) + if (*airq->lsi_ptr != 0) airq->handler(airq, tpi_info); rcu_read_unlock(); diff --git a/drivers/s390/crypto/zcrypt_api.c b/drivers/s390/crypto/zcrypt_api.c index 4b23c9f7f3e5..ce04caa7913f 100644 --- a/drivers/s390/crypto/zcrypt_api.c +++ b/drivers/s390/crypto/zcrypt_api.c @@ -366,7 +366,6 @@ static int zcdn_create(const char *name) { dev_t devt; int i, rc = 0; - char nodename[ZCDN_MAX_NAME]; struct zcdn_device *zcdndev; if (mutex_lock_interruptible(&ap_perms_mutex)) @@ -407,13 +406,11 @@ static int zcdn_create(const char *name) zcdndev->device.devt = devt; zcdndev->device.groups = zcdn_dev_attr_groups; if (name[0]) - strncpy(nodename, name, sizeof(nodename)); + rc = dev_set_name(&zcdndev->device, "%s", name); else - snprintf(nodename, sizeof(nodename), - ZCRYPT_NAME "_%d", (int)MINOR(devt)); - nodename[sizeof(nodename) - 1] = '\0'; - if (dev_set_name(&zcdndev->device, nodename)) { - rc = -EINVAL; + rc = dev_set_name(&zcdndev->device, ZCRYPT_NAME "_%d", (int)MINOR(devt)); + if (rc) { + kfree(zcdndev); goto unlockout; } rc = device_register(&zcdndev->device); diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c index 02922768b129..ac67576301bf 100644 --- a/drivers/s390/virtio/virtio_ccw.c +++ b/drivers/s390/virtio/virtio_ccw.c @@ -250,7 +250,6 @@ static struct airq_info *new_airq_info(int index) info->airq.handler = virtio_airq_handler; info->summary_indicator_idx = index; info->airq.lsi_ptr = get_summary_indicator(info); - info->airq.lsi_mask = 0xff; info->airq.isc = VIRTIO_AIRQ_ISC; rc = register_adapter_interrupt(&info->airq); if (rc) { diff --git a/drivers/scsi/aacraid/commsup.c b/drivers/scsi/aacraid/commsup.c index 3f062e4013ab..013a9a334972 100644 --- a/drivers/scsi/aacraid/commsup.c +++ b/drivers/scsi/aacraid/commsup.c @@ -1451,7 +1451,7 @@ retry_next: #endif break; } - scsi_rescan_device(&device->sdev_gendev); + scsi_rescan_device(device); break; default: diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h index e51e92f932fa..93c68931a593 100644 --- a/drivers/scsi/fnic/fnic.h +++ b/drivers/scsi/fnic/fnic.h @@ -27,7 +27,7 @@ #define DRV_NAME "fnic" #define DRV_DESCRIPTION "Cisco FCoE HBA Driver" -#define DRV_VERSION "1.6.0.55" +#define DRV_VERSION "1.6.0.56" #define PFX DRV_NAME ": " #define DFX DRV_NAME "%d: " @@ -236,6 +236,7 @@ struct fnic { unsigned int wq_count; unsigned int cq_count; + struct mutex sgreset_mutex; struct dentry *fnic_stats_debugfs_host; struct dentry *fnic_stats_debugfs_file; struct dentry *fnic_reset_debugfs_file; diff --git a/drivers/scsi/fnic/fnic_scsi.c b/drivers/scsi/fnic/fnic_scsi.c index be89ce96df46..9761b2c9db48 100644 --- a/drivers/scsi/fnic/fnic_scsi.c +++ b/drivers/scsi/fnic/fnic_scsi.c @@ -2167,39 +2167,6 @@ clean_pending_aborts_end: return ret; } -/* - * fnic_scsi_host_start_tag - * Allocates tagid from host's tag list - **/ -static inline int -fnic_scsi_host_start_tag(struct fnic *fnic, struct scsi_cmnd *sc) -{ - struct request *rq = scsi_cmd_to_rq(sc); - struct request_queue *q = rq->q; - struct request *dummy; - - dummy = blk_mq_alloc_request(q, REQ_OP_WRITE, BLK_MQ_REQ_NOWAIT); - if (IS_ERR(dummy)) - return SCSI_NO_TAG; - - rq->tag = dummy->tag; - sc->host_scribble = (unsigned char *)dummy; - - return dummy->tag; -} - -/* - * fnic_scsi_host_end_tag - * frees tag allocated by fnic_scsi_host_start_tag. - **/ -static inline void -fnic_scsi_host_end_tag(struct fnic *fnic, struct scsi_cmnd *sc) -{ - struct request *dummy = (struct request *)sc->host_scribble; - - blk_mq_free_request(dummy); -} - /* * SCSI Eh thread issues a Lun Reset when one or more commands on a LUN * fail to get aborted. It calls driver's eh_device_reset with a SCSI command @@ -2222,7 +2189,6 @@ int fnic_device_reset(struct scsi_cmnd *sc) struct reset_stats *reset_stats; int tag = rq->tag; DECLARE_COMPLETION_ONSTACK(tm_done); - int tag_gen_flag = 0; /*to track tags allocated by fnic driver*/ bool new_sc = 0; /* Wait for rport to unblock */ @@ -2252,17 +2218,17 @@ int fnic_device_reset(struct scsi_cmnd *sc) } fnic_priv(sc)->flags = FNIC_DEVICE_RESET; - /* Allocate tag if not present */ if (unlikely(tag < 0)) { /* - * Really should fix the midlayer to pass in a proper - * request for ioctls... + * For device reset issued through sg3utils, we let + * only one LUN_RESET to go through and use a special + * tag equal to max_tag_id so that we don't have to allocate + * or free it. It won't interact with tags + * allocated by mid layer. */ - tag = fnic_scsi_host_start_tag(fnic, sc); - if (unlikely(tag == SCSI_NO_TAG)) - goto fnic_device_reset_end; - tag_gen_flag = 1; + mutex_lock(&fnic->sgreset_mutex); + tag = fnic->fnic_max_tag_id; new_sc = 1; } io_lock = fnic_io_lock_hash(fnic, sc); @@ -2434,9 +2400,8 @@ fnic_device_reset_end: (u64)sc->cmnd[4] << 8 | sc->cmnd[5]), fnic_flags_and_state(sc)); - /* free tag if it is allocated */ - if (unlikely(tag_gen_flag)) - fnic_scsi_host_end_tag(fnic, sc); + if (new_sc) + mutex_unlock(&fnic->sgreset_mutex); FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, "Returning from device reset %s\n", diff --git a/drivers/scsi/libsas/sas_scsi_host.c b/drivers/scsi/libsas/sas_scsi_host.c index 0c103f4523b8..9047cfcd1072 100644 --- a/drivers/scsi/libsas/sas_scsi_host.c +++ b/drivers/scsi/libsas/sas_scsi_host.c @@ -386,37 +386,7 @@ struct sas_phy *sas_get_local_phy(struct domain_device *dev) } EXPORT_SYMBOL_GPL(sas_get_local_phy); -static void sas_wait_eh(struct domain_device *dev) -{ - struct sas_ha_struct *ha = dev->port->ha; - DEFINE_WAIT(wait); - - if (dev_is_sata(dev)) { - ata_port_wait_eh(dev->sata_dev.ap); - return; - } - retry: - spin_lock_irq(&ha->lock); - - while (test_bit(SAS_DEV_EH_PENDING, &dev->state)) { - prepare_to_wait(&ha->eh_wait_q, &wait, TASK_UNINTERRUPTIBLE); - spin_unlock_irq(&ha->lock); - schedule(); - spin_lock_irq(&ha->lock); - } - finish_wait(&ha->eh_wait_q, &wait); - - spin_unlock_irq(&ha->lock); - - /* make sure SCSI EH is complete */ - if (scsi_host_in_recovery(ha->shost)) { - msleep(10); - goto retry; - } -} - -static int sas_queue_reset(struct domain_device *dev, int reset_type, - u64 lun, int wait) +static int sas_queue_reset(struct domain_device *dev, int reset_type, u64 lun) { struct sas_ha_struct *ha = dev->port->ha; int scheduled = 0, tries = 100; @@ -424,8 +394,6 @@ static int sas_queue_reset(struct domain_device *dev, int reset_type, /* ata: promote lun reset to bus reset */ if (dev_is_sata(dev)) { sas_ata_schedule_reset(dev); - if (wait) - sas_ata_wait_eh(dev); return SUCCESS; } @@ -443,9 +411,6 @@ static int sas_queue_reset(struct domain_device *dev, int reset_type, } spin_unlock_irq(&ha->lock); - if (wait) - sas_wait_eh(dev); - if (scheduled) return SUCCESS; } @@ -498,7 +463,7 @@ int sas_eh_device_reset_handler(struct scsi_cmnd *cmd) struct sas_internal *i = to_sas_internal(host->transportt); if (current != host->ehandler) - return sas_queue_reset(dev, SAS_DEV_LU_RESET, cmd->device->lun, 0); + return sas_queue_reset(dev, SAS_DEV_LU_RESET, cmd->device->lun); int_to_scsilun(cmd->device->lun, &lun); @@ -521,7 +486,7 @@ int sas_eh_target_reset_handler(struct scsi_cmnd *cmd) struct sas_internal *i = to_sas_internal(host->transportt); if (current != host->ehandler) - return sas_queue_reset(dev, SAS_DEV_RESET, 0, 0); + return sas_queue_reset(dev, SAS_DEV_RESET, 0); if (!i->dft->lldd_I_T_nexus_reset) return FAILED; diff --git a/drivers/scsi/mpt3sas/mpi/mpi2.h b/drivers/scsi/mpt3sas/mpi/mpi2.h index ed3923f8db4f..6de35b32223c 100644 --- a/drivers/scsi/mpt3sas/mpi/mpi2.h +++ b/drivers/scsi/mpt3sas/mpi/mpi2.h @@ -199,7 +199,7 @@ * *****************************************************************************/ -typedef volatile struct _MPI2_SYSTEM_INTERFACE_REGS { +typedef struct _MPI2_SYSTEM_INTERFACE_REGS { U32 Doorbell; /*0x00 */ U32 WriteSequence; /*0x04 */ U32 HostDiagnostic; /*0x08 */ diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 53f5492579cb..61a32bf00747 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -138,6 +138,9 @@ _base_get_ioc_facts(struct MPT3SAS_ADAPTER *ioc); static void _base_clear_outstanding_commands(struct MPT3SAS_ADAPTER *ioc); +static u32 +_base_readl_ext_retry(const void __iomem *addr); + /** * mpt3sas_base_check_cmd_timeout - Function * to check timeout and command termination due @@ -201,7 +204,7 @@ module_param_call(mpt3sas_fwfault_debug, _scsih_set_fwfault_debug, * while reading the system interface register. */ static inline u32 -_base_readl_aero(const volatile void __iomem *addr) +_base_readl_aero(const void __iomem *addr) { u32 i = 0, ret_val; @@ -213,8 +216,22 @@ _base_readl_aero(const volatile void __iomem *addr) return ret_val; } +static u32 +_base_readl_ext_retry(const void __iomem *addr) +{ + u32 i, ret_val; + + for (i = 0 ; i < 30 ; i++) { + ret_val = readl(addr); + if (ret_val == 0) + continue; + } + + return ret_val; +} + static inline u32 -_base_readl(const volatile void __iomem *addr) +_base_readl(const void __iomem *addr) { return readl(addr); } @@ -940,7 +957,7 @@ mpt3sas_halt_firmware(struct MPT3SAS_ADAPTER *ioc) dump_stack(); - doorbell = ioc->base_readl(&ioc->chip->Doorbell); + doorbell = ioc->base_readl_ext_retry(&ioc->chip->Doorbell); if ((doorbell & MPI2_IOC_STATE_MASK) == MPI2_IOC_STATE_FAULT) { mpt3sas_print_fault_code(ioc, doorbell & MPI2_DOORBELL_DATA_MASK); @@ -6686,7 +6703,7 @@ mpt3sas_base_get_iocstate(struct MPT3SAS_ADAPTER *ioc, int cooked) { u32 s, sc; - s = ioc->base_readl(&ioc->chip->Doorbell); + s = ioc->base_readl_ext_retry(&ioc->chip->Doorbell); sc = s & MPI2_IOC_STATE_MASK; return cooked ? sc : s; } @@ -6831,7 +6848,7 @@ _base_wait_for_doorbell_ack(struct MPT3SAS_ADAPTER *ioc, int timeout) __func__, count, timeout)); return 0; } else if (int_status & MPI2_HIS_IOC2SYS_DB_STATUS) { - doorbell = ioc->base_readl(&ioc->chip->Doorbell); + doorbell = ioc->base_readl_ext_retry(&ioc->chip->Doorbell); if ((doorbell & MPI2_IOC_STATE_MASK) == MPI2_IOC_STATE_FAULT) { mpt3sas_print_fault_code(ioc, doorbell); @@ -6871,7 +6888,7 @@ _base_wait_for_doorbell_not_used(struct MPT3SAS_ADAPTER *ioc, int timeout) count = 0; cntdn = 1000 * timeout; do { - doorbell_reg = ioc->base_readl(&ioc->chip->Doorbell); + doorbell_reg = ioc->base_readl_ext_retry(&ioc->chip->Doorbell); if (!(doorbell_reg & MPI2_DOORBELL_USED)) { dhsprintk(ioc, ioc_info(ioc, "%s: successful count(%d), timeout(%d)\n", @@ -7019,7 +7036,7 @@ _base_handshake_req_reply_wait(struct MPT3SAS_ADAPTER *ioc, int request_bytes, __le32 *mfp; /* make sure doorbell is not in use */ - if ((ioc->base_readl(&ioc->chip->Doorbell) & MPI2_DOORBELL_USED)) { + if ((ioc->base_readl_ext_retry(&ioc->chip->Doorbell) & MPI2_DOORBELL_USED)) { ioc_err(ioc, "doorbell is in use (line=%d)\n", __LINE__); return -EFAULT; } @@ -7068,7 +7085,7 @@ _base_handshake_req_reply_wait(struct MPT3SAS_ADAPTER *ioc, int request_bytes, } /* read the first two 16-bits, it gives the total length of the reply */ - reply[0] = le16_to_cpu(ioc->base_readl(&ioc->chip->Doorbell) + reply[0] = le16_to_cpu(ioc->base_readl_ext_retry(&ioc->chip->Doorbell) & MPI2_DOORBELL_DATA_MASK); writel(0, &ioc->chip->HostInterruptStatus); if ((_base_wait_for_doorbell_int(ioc, 5))) { @@ -7076,7 +7093,7 @@ _base_handshake_req_reply_wait(struct MPT3SAS_ADAPTER *ioc, int request_bytes, __LINE__); return -EFAULT; } - reply[1] = le16_to_cpu(ioc->base_readl(&ioc->chip->Doorbell) + reply[1] = le16_to_cpu(ioc->base_readl_ext_retry(&ioc->chip->Doorbell) & MPI2_DOORBELL_DATA_MASK); writel(0, &ioc->chip->HostInterruptStatus); @@ -7087,10 +7104,10 @@ _base_handshake_req_reply_wait(struct MPT3SAS_ADAPTER *ioc, int request_bytes, return -EFAULT; } if (i >= reply_bytes/2) /* overflow case */ - ioc->base_readl(&ioc->chip->Doorbell); + ioc->base_readl_ext_retry(&ioc->chip->Doorbell); else reply[i] = le16_to_cpu( - ioc->base_readl(&ioc->chip->Doorbell) + ioc->base_readl_ext_retry(&ioc->chip->Doorbell) & MPI2_DOORBELL_DATA_MASK); writel(0, &ioc->chip->HostInterruptStatus); } @@ -7949,7 +7966,7 @@ _base_diag_reset(struct MPT3SAS_ADAPTER *ioc) goto out; } - host_diagnostic = ioc->base_readl(&ioc->chip->HostDiagnostic); + host_diagnostic = ioc->base_readl_ext_retry(&ioc->chip->HostDiagnostic); drsprintk(ioc, ioc_info(ioc, "wrote magic sequence: count(%d), host_diagnostic(0x%08x)\n", count, host_diagnostic)); @@ -7969,7 +7986,7 @@ _base_diag_reset(struct MPT3SAS_ADAPTER *ioc) for (count = 0; count < (300000000 / MPI2_HARD_RESET_PCIE_SECOND_READ_DELAY_MICRO_SEC); count++) { - host_diagnostic = ioc->base_readl(&ioc->chip->HostDiagnostic); + host_diagnostic = ioc->base_readl_ext_retry(&ioc->chip->HostDiagnostic); if (host_diagnostic == 0xFFFFFFFF) { ioc_info(ioc, @@ -8359,10 +8376,13 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc) ioc->rdpq_array_enable_assigned = 0; ioc->use_32bit_dma = false; ioc->dma_mask = 64; - if (ioc->is_aero_ioc) + if (ioc->is_aero_ioc) { ioc->base_readl = &_base_readl_aero; - else + ioc->base_readl_ext_retry = &_base_readl_ext_retry; + } else { ioc->base_readl = &_base_readl; + ioc->base_readl_ext_retry = &_base_readl; + } r = mpt3sas_base_map_resources(ioc); if (r) goto out_free_resources; diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index 05364aa15ecd..1be0850ca17a 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -994,7 +994,7 @@ typedef void (*NVME_BUILD_PRP)(struct MPT3SAS_ADAPTER *ioc, u16 smid, typedef void (*PUT_SMID_IO_FP_HIP) (struct MPT3SAS_ADAPTER *ioc, u16 smid, u16 funcdep); typedef void (*PUT_SMID_DEFAULT) (struct MPT3SAS_ADAPTER *ioc, u16 smid); -typedef u32 (*BASE_READ_REG) (const volatile void __iomem *addr); +typedef u32 (*BASE_READ_REG) (const void __iomem *addr); /* * To get high iops reply queue's msix index when high iops mode is enabled * else get the msix index of general reply queues. @@ -1618,6 +1618,7 @@ struct MPT3SAS_ADAPTER { u8 diag_trigger_active; u8 atomic_desc_capable; BASE_READ_REG base_readl; + BASE_READ_REG base_readl_ext_retry; struct SL_WH_MASTER_TRIGGER_T diag_trigger_master; struct SL_WH_EVENT_TRIGGERS_T diag_trigger_event; struct SL_WH_SCSI_TRIGGERS_T diag_trigger_scsi; diff --git a/drivers/scsi/mvumi.c b/drivers/scsi/mvumi.c index 97f9d2fa6429..d9d366ec17dc 100644 --- a/drivers/scsi/mvumi.c +++ b/drivers/scsi/mvumi.c @@ -1500,7 +1500,7 @@ static void mvumi_rescan_devices(struct mvumi_hba *mhba, int id) sdev = scsi_device_lookup(mhba->shost, 0, id, 0); if (sdev) { - scsi_rescan_device(&sdev->sdev_gendev); + scsi_rescan_device(sdev); scsi_device_put(sdev); } } diff --git a/drivers/scsi/qedf/qedf.h b/drivers/scsi/qedf/qedf.h index c5c0bbdafc4e..1619cc33034f 100644 --- a/drivers/scsi/qedf/qedf.h +++ b/drivers/scsi/qedf/qedf.h @@ -548,7 +548,6 @@ extern void qedf_get_generic_tlv_data(void *dev, struct qed_generic_tlvs *data); extern void qedf_wq_grcdump(struct work_struct *work); void qedf_stag_change_work(struct work_struct *work); void qedf_ctx_soft_reset(struct fc_lport *lport); -extern void qedf_board_disable_work(struct work_struct *work); extern void qedf_schedule_hw_err_handler(void *dev, enum qed_hw_err_type err_type); diff --git a/drivers/scsi/qedi/qedi_gbl.h b/drivers/scsi/qedi/qedi_gbl.h index 0e316cc24b19..772218445a56 100644 --- a/drivers/scsi/qedi/qedi_gbl.h +++ b/drivers/scsi/qedi/qedi_gbl.h @@ -67,8 +67,6 @@ void qedi_trace_io(struct qedi_ctx *qedi, struct iscsi_task *task, int qedi_alloc_id(struct qedi_portid_tbl *id_tbl, u16 id); u16 qedi_alloc_new_id(struct qedi_portid_tbl *id_tbl); void qedi_free_id(struct qedi_portid_tbl *id_tbl, u16 id); -int qedi_create_sysfs_ctx_attr(struct qedi_ctx *qedi); -void qedi_remove_sysfs_ctx_attr(struct qedi_ctx *qedi); void qedi_clearsq(struct qedi_ctx *qedi, struct qedi_conn *qedi_conn, struct iscsi_task *task); diff --git a/drivers/scsi/qla2xxx/qla_attr.c b/drivers/scsi/qla2xxx/qla_attr.c index b00222459607..44449c70a375 100644 --- a/drivers/scsi/qla2xxx/qla_attr.c +++ b/drivers/scsi/qla2xxx/qla_attr.c @@ -3093,8 +3093,6 @@ qla24xx_vport_create(struct fc_vport *fc_vport, bool disable) vha->flags.difdix_supported = 1; ql_dbg(ql_dbg_user, vha, 0x7082, "Registered for DIF/DIX type 1 and 3 protection.\n"); - if (ql2xenabledif == 1) - prot = SHOST_DIX_TYPE0_PROTECTION; scsi_host_set_prot(vha->host, prot | SHOST_DIF_TYPE1_PROTECTION | SHOST_DIF_TYPE2_PROTECTION diff --git a/drivers/scsi/qla2xxx/qla_dbg.c b/drivers/scsi/qla2xxx/qla_dbg.c index d7e8454304ce..691ef827a5ab 100644 --- a/drivers/scsi/qla2xxx/qla_dbg.c +++ b/drivers/scsi/qla2xxx/qla_dbg.c @@ -12,13 +12,12 @@ * ---------------------------------------------------------------------- * | Module Init and Probe | 0x0199 | | * | Mailbox commands | 0x1206 | 0x11a5-0x11ff | - * | Device Discovery | 0x2134 | 0x210e-0x2115 | - * | | | 0x211c-0x2128 | - * | | | 0x212c-0x2134 | + * | Device Discovery | 0x2134 | 0x2112-0x2115 | + * | | | 0x2127-0x2128 | * | Queue Command and IO tracing | 0x3074 | 0x300b | * | | | 0x3027-0x3028 | * | | | 0x303d-0x3041 | - * | | | 0x302d,0x3033 | + * | | | 0x302e,0x3033 | * | | | 0x3036,0x3038 | * | | | 0x303a | * | DPC Thread | 0x4023 | 0x4002,0x4013 | diff --git a/drivers/scsi/qla2xxx/qla_dbg.h b/drivers/scsi/qla2xxx/qla_dbg.h index 70482b55d240..54f0a412226f 100644 --- a/drivers/scsi/qla2xxx/qla_dbg.h +++ b/drivers/scsi/qla2xxx/qla_dbg.h @@ -368,6 +368,7 @@ ql_log_qp(uint32_t, struct qla_qpair *, int32_t, const char *fmt, ...); #define ql_dbg_tgt_tmr 0x00001000 /* Target mode task management */ #define ql_dbg_tgt_dif 0x00000800 /* Target mode dif */ #define ql_dbg_edif 0x00000400 /* edif and purex debug */ +#define ql_dbg_unsol 0x00000100 /* Unsolicited path debug */ extern int qla27xx_dump_mpi_ram(struct qla_hw_data *, uint32_t, uint32_t *, uint32_t, void **); diff --git a/drivers/scsi/qla2xxx/qla_def.h b/drivers/scsi/qla2xxx/qla_def.h index 336b8c665cb4..deb642607deb 100644 --- a/drivers/scsi/qla2xxx/qla_def.h +++ b/drivers/scsi/qla2xxx/qla_def.h @@ -346,6 +346,12 @@ struct name_list_extended { u8 sent; }; +struct qla_nvme_fc_rjt { + struct fcnvme_ls_rjt *c; + dma_addr_t cdma; + u16 size; +}; + struct els_reject { struct fc_els_ls_rjt *c; dma_addr_t cdma; @@ -503,6 +509,20 @@ struct ct_arg { port_id_t id; }; +struct qla_nvme_lsrjt_pt_arg { + struct fc_port *fcport; + u8 opcode; + u8 vp_idx; + u8 reason; + u8 explanation; + __le16 nport_handle; + u16 control_flags; + __le16 ox_id; + __le32 xchg_address; + u32 tx_byte_count, rx_byte_count; + dma_addr_t tx_addr, rx_addr; +}; + /* * SRB extensions. */ @@ -611,13 +631,16 @@ struct srb_iocb { void *desc; /* These are only used with ls4 requests */ - int cmd_len; - int rsp_len; + __le32 cmd_len; + __le32 rsp_len; dma_addr_t cmd_dma; dma_addr_t rsp_dma; enum nvmefc_fcp_datadir dir; uint32_t dl; uint32_t timeout_sec; + __le32 exchange_address; + __le16 nport_handle; + __le16 ox_id; struct list_head entry; } nvme; struct { @@ -707,6 +730,10 @@ typedef struct srb { struct fc_port *fcport; struct scsi_qla_host *vha; unsigned int start_timer:1; + unsigned int abort:1; + unsigned int aborted:1; + unsigned int completed:1; + unsigned int unsol_rsp:1; uint32_t handle; uint16_t flags; @@ -2542,6 +2569,7 @@ enum rscn_addr_format { typedef struct fc_port { struct list_head list; struct scsi_qla_host *vha; + struct list_head unsol_ctx_head; unsigned int conf_compl_supported:1; unsigned int deleted:2; @@ -3742,6 +3770,16 @@ struct qla_fw_resources { u16 pad; }; +struct qla_fw_res { + u16 iocb_total; + u16 iocb_limit; + atomic_t iocb_used; + + u16 exch_total; + u16 exch_limit; + atomic_t exch_used; +}; + #define QLA_IOCB_PCT_LIMIT 95 struct qla_buf_pool { @@ -3787,6 +3825,12 @@ struct qla_qpair { uint16_t id; /* qp number used with FW */ uint16_t vp_idx; /* vport ID */ + + uint16_t dsd_inuse; + uint16_t dsd_avail; + struct list_head dsd_list; +#define NUM_DSD_CHAIN 4096 + mempool_t *srb_mempool; struct pci_dev *pdev; @@ -4384,7 +4428,6 @@ struct qla_hw_data { uint8_t aen_mbx_count; atomic_t num_pend_mbx_stage1; atomic_t num_pend_mbx_stage2; - atomic_t num_pend_mbx_stage3; uint16_t frame_payload_size; uint32_t login_retry_count; @@ -4714,11 +4757,6 @@ struct qla_hw_data { struct fw_blob *hablob; struct qla82xx_legacy_intr_set nx_legacy_intr; - uint16_t gbl_dsd_inuse; - uint16_t gbl_dsd_avail; - struct list_head gbl_dsd_list; -#define NUM_DSD_CHAIN 4096 - uint8_t fw_type; uint32_t file_prd_off; /* File firmware product offset */ @@ -4800,6 +4838,8 @@ struct qla_hw_data { struct els_reject elsrej; u8 edif_post_stop_cnt_down; struct qla_vp_map *vp_map; + struct qla_nvme_fc_rjt lsrjt; + struct qla_fw_res fwres ____cacheline_aligned; }; #define RX_ELS_SIZE (roundup(sizeof(struct enode) + ELS_MAX_PAYLOAD, SMP_CACHE_BYTES)) @@ -4832,6 +4872,7 @@ struct active_regions { * is variable) starting at "iocb". */ struct purex_item { + void *purls_context; struct list_head list; struct scsi_qla_host *vha; void (*process_item)(struct scsi_qla_host *vha, diff --git a/drivers/scsi/qla2xxx/qla_dfs.c b/drivers/scsi/qla2xxx/qla_dfs.c index 1925cc6897b6..f060e593685d 100644 --- a/drivers/scsi/qla2xxx/qla_dfs.c +++ b/drivers/scsi/qla2xxx/qla_dfs.c @@ -276,6 +276,16 @@ qla_dfs_fw_resource_cnt_show(struct seq_file *s, void *unused) seq_printf(s, "estimate exchange used[%d] high water limit [%d] n", exch_used, ha->base_qpair->fwres.exch_limit); + + if (ql2xenforce_iocb_limit == 2) { + iocbs_used = atomic_read(&ha->fwres.iocb_used); + exch_used = atomic_read(&ha->fwres.exch_used); + seq_printf(s, " estimate iocb2 used [%d] high water limit [%d]\n", + iocbs_used, ha->fwres.iocb_limit); + + seq_printf(s, " estimate exchange2 used[%d] high water limit [%d] \n", + exch_used, ha->fwres.exch_limit); + } } return 0; diff --git a/drivers/scsi/qla2xxx/qla_gbl.h b/drivers/scsi/qla2xxx/qla_gbl.h index 816c0b9ecd0e..09cb9413670a 100644 --- a/drivers/scsi/qla2xxx/qla_gbl.h +++ b/drivers/scsi/qla2xxx/qla_gbl.h @@ -603,7 +603,11 @@ qla2xxx_msix_rsp_q_hs(int irq, void *dev_id); fc_port_t *qla2x00_find_fcport_by_loopid(scsi_qla_host_t *, uint16_t); fc_port_t *qla2x00_find_fcport_by_wwpn(scsi_qla_host_t *, u8 *, u8); fc_port_t *qla2x00_find_fcport_by_nportid(scsi_qla_host_t *, port_id_t *, u8); -void __qla_consume_iocb(struct scsi_qla_host *vha, void **pkt, struct rsp_que **rsp); +void qla24xx_queue_purex_item(scsi_qla_host_t *, struct purex_item *, + void (*process_item)(struct scsi_qla_host *, + struct purex_item *)); +void __qla_consume_iocb(struct scsi_qla_host *, void **, struct rsp_que **); +void qla2xxx_process_purls_iocb(void **pkt, struct rsp_que **rsp); /* * Global Function Prototypes in qla_sup.c source file. @@ -666,9 +670,11 @@ extern int qla2xxx_get_vpd_field(scsi_qla_host_t *, char *, char *, size_t); extern void qla2xxx_flash_npiv_conf(scsi_qla_host_t *); extern int qla24xx_read_fcp_prio_cfg(scsi_qla_host_t *); extern int qla2x00_mailbox_passthru(struct bsg_job *bsg_job); -int __qla_copy_purex_to_buffer(struct scsi_qla_host *vha, void **pkt, - struct rsp_que **rsp, u8 *buf, u32 buf_len); - +int qla2x00_sys_ld_info(struct bsg_job *bsg_job); +int __qla_copy_purex_to_buffer(struct scsi_qla_host *, void **, + struct rsp_que **, u8 *, u32); +struct purex_item *qla27xx_copy_multiple_pkt(struct scsi_qla_host *vha, + void **pkt, struct rsp_que **rsp, bool is_purls, bool byte_order); int qla_mailbox_passthru(scsi_qla_host_t *vha, uint16_t *mbx_in, uint16_t *mbx_out); diff --git a/drivers/scsi/qla2xxx/qla_init.c b/drivers/scsi/qla2xxx/qla_init.c index df623de89255..a314cfc5b263 100644 --- a/drivers/scsi/qla2xxx/qla_init.c +++ b/drivers/scsi/qla2xxx/qla_init.c @@ -2223,6 +2223,8 @@ __qla2x00_async_tm_cmd(struct tmf_arg *arg) rval = QLA_FUNCTION_FAILED; } } + if (tm_iocb->u.tmf.data) + rval = tm_iocb->u.tmf.data; done_free_sp: /* ref: INIT */ @@ -4203,7 +4205,7 @@ void qla_init_iocb_limit(scsi_qla_host_t *vha) u8 i; struct qla_hw_data *ha = vha->hw; - __qla_adjust_iocb_limit(ha->base_qpair); + __qla_adjust_iocb_limit(ha->base_qpair); ha->base_qpair->fwres.iocbs_used = 0; ha->base_qpair->fwres.exch_used = 0; @@ -4214,6 +4216,14 @@ void qla_init_iocb_limit(scsi_qla_host_t *vha) ha->queue_pair_map[i]->fwres.exch_used = 0; } } + + ha->fwres.iocb_total = ha->orig_fw_iocb_count; + ha->fwres.iocb_limit = (ha->orig_fw_iocb_count * QLA_IOCB_PCT_LIMIT) / 100; + ha->fwres.exch_total = ha->orig_fw_xcb_count; + ha->fwres.exch_limit = (ha->orig_fw_xcb_count * QLA_IOCB_PCT_LIMIT) / 100; + + atomic_set(&ha->fwres.iocb_used, 0); + atomic_set(&ha->fwres.exch_used, 0); } void qla_adjust_iocb_limit(scsi_qla_host_t *vha) @@ -5554,6 +5564,7 @@ qla2x00_alloc_fcport(scsi_qla_host_t *vha, gfp_t flags) INIT_WORK(&fcport->reg_work, qla_register_fcport_fn); INIT_LIST_HEAD(&fcport->gnl_entry); INIT_LIST_HEAD(&fcport->list); + INIT_LIST_HEAD(&fcport->unsol_ctx_head); INIT_LIST_HEAD(&fcport->sess_cmd_list); spin_lock_init(&fcport->sess_cmd_lock); @@ -5596,7 +5607,7 @@ static void qla_get_login_template(scsi_qla_host_t *vha) __be32 *q; memset(ha->init_cb, 0, ha->init_cb_size); - sz = min_t(int, sizeof(struct fc_els_csp), ha->init_cb_size); + sz = min_t(int, sizeof(struct fc_els_flogi), ha->init_cb_size); rval = qla24xx_get_port_login_templ(vha, ha->init_cb_dma, ha->init_cb, sz); if (rval != QLA_SUCCESS) { @@ -7390,14 +7401,15 @@ qla2x00_abort_isp_cleanup(scsi_qla_host_t *vha) } /* purge MBox commands */ - if (atomic_read(&ha->num_pend_mbx_stage3)) { + spin_lock_irqsave(&ha->hardware_lock, flags); + if (test_bit(MBX_INTR_WAIT, &ha->mbx_cmd_flags)) { clear_bit(MBX_INTR_WAIT, &ha->mbx_cmd_flags); complete(&ha->mbx_intr_comp); } + spin_unlock_irqrestore(&ha->hardware_lock, flags); i = 0; - while (atomic_read(&ha->num_pend_mbx_stage3) || - atomic_read(&ha->num_pend_mbx_stage2) || + while (atomic_read(&ha->num_pend_mbx_stage2) || atomic_read(&ha->num_pend_mbx_stage1)) { msleep(20); i++; @@ -9643,6 +9655,7 @@ struct qla_qpair *qla2xxx_create_qpair(struct scsi_qla_host *vha, int qos, qpair->vp_idx = vp_idx; qpair->fw_started = ha->flags.fw_started; INIT_LIST_HEAD(&qpair->hints_list); + INIT_LIST_HEAD(&qpair->dsd_list); qpair->chip_reset = ha->base_qpair->chip_reset; qpair->enable_class_2 = ha->base_qpair->enable_class_2; qpair->enable_explicit_conf = @@ -9771,6 +9784,19 @@ int qla2xxx_delete_qpair(struct scsi_qla_host *vha, struct qla_qpair *qpair) if (ret != QLA_SUCCESS) goto fail; + if (!list_empty(&qpair->dsd_list)) { + struct dsd_dma *dsd_ptr, *tdsd_ptr; + + /* clean up allocated prev pool */ + list_for_each_entry_safe(dsd_ptr, tdsd_ptr, + &qpair->dsd_list, list) { + dma_pool_free(ha->dl_dma_pool, dsd_ptr->dsd_addr, + dsd_ptr->dsd_list_dma); + list_del(&dsd_ptr->list); + kfree(dsd_ptr); + } + } + mutex_lock(&ha->mq_lock); ha->queue_pair_map[qpair->id] = NULL; clear_bit(qpair->id, ha->qpair_qid_map); diff --git a/drivers/scsi/qla2xxx/qla_inline.h b/drivers/scsi/qla2xxx/qla_inline.h index 0167e85ba058..0556969f6dc1 100644 --- a/drivers/scsi/qla2xxx/qla_inline.h +++ b/drivers/scsi/qla2xxx/qla_inline.h @@ -386,6 +386,7 @@ enum { RESOURCE_IOCB = BIT_0, RESOURCE_EXCH = BIT_1, /* exchange */ RESOURCE_FORCE = BIT_2, + RESOURCE_HA = BIT_3, }; static inline int @@ -393,7 +394,7 @@ qla_get_fw_resources(struct qla_qpair *qp, struct iocb_resource *iores) { u16 iocbs_used, i; u16 exch_used; - struct qla_hw_data *ha = qp->vha->hw; + struct qla_hw_data *ha = qp->hw; if (!ql2xenforce_iocb_limit) { iores->res_type = RESOURCE_NONE; @@ -428,15 +429,69 @@ qla_get_fw_resources(struct qla_qpair *qp, struct iocb_resource *iores) return -ENOSPC; } } + + if (ql2xenforce_iocb_limit == 2) { + if ((iores->iocb_cnt + atomic_read(&ha->fwres.iocb_used)) >= + ha->fwres.iocb_limit) { + iores->res_type = RESOURCE_NONE; + return -ENOSPC; + } + + if (iores->res_type & RESOURCE_EXCH) { + if ((iores->exch_cnt + atomic_read(&ha->fwres.exch_used)) >= + ha->fwres.exch_limit) { + iores->res_type = RESOURCE_NONE; + return -ENOSPC; + } + } + } + force: qp->fwres.iocbs_used += iores->iocb_cnt; qp->fwres.exch_used += iores->exch_cnt; + if (ql2xenforce_iocb_limit == 2) { + atomic_add(iores->iocb_cnt, &ha->fwres.iocb_used); + atomic_add(iores->exch_cnt, &ha->fwres.exch_used); + iores->res_type |= RESOURCE_HA; + } return 0; } +/* + * decrement to zero. This routine will not decrement below zero + * @v: pointer of type atomic_t + * @amount: amount to decrement from v + */ +static void qla_atomic_dtz(atomic_t *v, int amount) +{ + int c, old, dec; + + c = atomic_read(v); + for (;;) { + dec = c - amount; + if (unlikely(dec < 0)) + dec = 0; + + old = atomic_cmpxchg((v), c, dec); + if (likely(old == c)) + break; + c = old; + } +} + static inline void qla_put_fw_resources(struct qla_qpair *qp, struct iocb_resource *iores) { + struct qla_hw_data *ha = qp->hw; + + if (iores->res_type & RESOURCE_HA) { + if (iores->res_type & RESOURCE_IOCB) + qla_atomic_dtz(&ha->fwres.iocb_used, iores->iocb_cnt); + + if (iores->res_type & RESOURCE_EXCH) + qla_atomic_dtz(&ha->fwres.exch_used, iores->exch_cnt); + } + if (iores->res_type & RESOURCE_IOCB) { if (qp->fwres.iocbs_used >= iores->iocb_cnt) { qp->fwres.iocbs_used -= iores->iocb_cnt; diff --git a/drivers/scsi/qla2xxx/qla_iocb.c b/drivers/scsi/qla2xxx/qla_iocb.c index 1ee9b7d5fc15..df90169f8244 100644 --- a/drivers/scsi/qla2xxx/qla_iocb.c +++ b/drivers/scsi/qla2xxx/qla_iocb.c @@ -11,6 +11,7 @@ #include +static int qla_start_scsi_type6(srb_t *sp); /** * qla2x00_get_cmd_direction() - Determine control_flag data direction. * @sp: SCSI command @@ -590,8 +591,6 @@ qla24xx_build_scsi_type_6_iocbs(srb_t *sp, struct cmd_type_6 *cmd_pkt, uint16_t tot_dsds) { struct dsd64 *cur_dsd = NULL, *next_dsd; - scsi_qla_host_t *vha; - struct qla_hw_data *ha; struct scsi_cmnd *cmd; struct scatterlist *cur_seg; uint8_t avail_dsds; @@ -613,9 +612,6 @@ qla24xx_build_scsi_type_6_iocbs(srb_t *sp, struct cmd_type_6 *cmd_pkt, return 0; } - vha = sp->vha; - ha = vha->hw; - /* Set transfer direction */ if (cmd->sc_data_direction == DMA_TO_DEVICE) { cmd_pkt->control_flags = cpu_to_le16(CF_WRITE_DATA); @@ -636,14 +632,13 @@ qla24xx_build_scsi_type_6_iocbs(srb_t *sp, struct cmd_type_6 *cmd_pkt, tot_dsds -= avail_dsds; dsd_list_len = (avail_dsds + 1) * QLA_DSD_SIZE; - dsd_ptr = list_first_entry(&ha->gbl_dsd_list, - struct dsd_dma, list); + dsd_ptr = list_first_entry(&qpair->dsd_list, struct dsd_dma, list); next_dsd = dsd_ptr->dsd_addr; list_del(&dsd_ptr->list); - ha->gbl_dsd_avail--; + qpair->dsd_avail--; list_add_tail(&dsd_ptr->list, &ctx->dsd_list); ctx->dsd_use_cnt++; - ha->gbl_dsd_inuse++; + qpair->dsd_inuse++; if (first_iocb) { first_iocb = 0; @@ -1722,6 +1717,8 @@ qla24xx_dif_start_scsi(srb_t *sp) if (scsi_get_prot_op(cmd) == SCSI_PROT_NORMAL) { if (cmd->cmd_len <= 16) return qla24xx_start_scsi(sp); + else + return qla_start_scsi_type6(sp); } /* Setup device pointers. */ @@ -2101,6 +2098,8 @@ qla2xxx_dif_start_scsi_mq(srb_t *sp) if (scsi_get_prot_op(cmd) == SCSI_PROT_NORMAL) { if (cmd->cmd_len <= 16) return qla2xxx_start_scsi_mq(sp); + else + return qla_start_scsi_type6(sp); } spin_lock_irqsave(&qpair->qp_lock, flags); @@ -3368,6 +3367,7 @@ qla82xx_start_scsi(srb_t *sp) struct qla_hw_data *ha = vha->hw; struct req_que *req = NULL; struct rsp_que *rsp = NULL; + struct qla_qpair *qpair = sp->qpair; /* Setup device pointers. */ reg = &ha->iobase->isp82; @@ -3416,18 +3416,18 @@ qla82xx_start_scsi(srb_t *sp) uint16_t i; more_dsd_lists = qla24xx_calc_dsd_lists(tot_dsds); - if ((more_dsd_lists + ha->gbl_dsd_inuse) >= NUM_DSD_CHAIN) { + if ((more_dsd_lists + qpair->dsd_inuse) >= NUM_DSD_CHAIN) { ql_dbg(ql_dbg_io, vha, 0x300d, "Num of DSD list %d is than %d for cmd=%p.\n", - more_dsd_lists + ha->gbl_dsd_inuse, NUM_DSD_CHAIN, + more_dsd_lists + qpair->dsd_inuse, NUM_DSD_CHAIN, cmd); goto queuing_error; } - if (more_dsd_lists <= ha->gbl_dsd_avail) + if (more_dsd_lists <= qpair->dsd_avail) goto sufficient_dsds; else - more_dsd_lists -= ha->gbl_dsd_avail; + more_dsd_lists -= qpair->dsd_avail; for (i = 0; i < more_dsd_lists; i++) { dsd_ptr = kzalloc(sizeof(struct dsd_dma), GFP_ATOMIC); @@ -3447,8 +3447,8 @@ qla82xx_start_scsi(srb_t *sp) "for cmd=%p.\n", cmd); goto queuing_error; } - list_add_tail(&dsd_ptr->list, &ha->gbl_dsd_list); - ha->gbl_dsd_avail++; + list_add_tail(&dsd_ptr->list, &qpair->dsd_list); + qpair->dsd_avail++; } sufficient_dsds: @@ -3767,21 +3767,28 @@ qla_nvme_ls(srb_t *sp, struct pt_ls4_request *cmd_pkt) nvme = &sp->u.iocb_cmd; cmd_pkt->entry_type = PT_LS4_REQUEST; cmd_pkt->entry_count = 1; - cmd_pkt->control_flags = cpu_to_le16(CF_LS4_ORIGINATOR << CF_LS4_SHIFT); - cmd_pkt->timeout = cpu_to_le16(nvme->u.nvme.timeout_sec); - cmd_pkt->nport_handle = cpu_to_le16(sp->fcport->loop_id); cmd_pkt->vp_index = sp->fcport->vha->vp_idx; - cmd_pkt->tx_dseg_count = cpu_to_le16(1); - cmd_pkt->tx_byte_count = cpu_to_le32(nvme->u.nvme.cmd_len); - cmd_pkt->dsd[0].length = cpu_to_le32(nvme->u.nvme.cmd_len); - put_unaligned_le64(nvme->u.nvme.cmd_dma, &cmd_pkt->dsd[0].address); + if (sp->unsol_rsp) { + cmd_pkt->control_flags = + cpu_to_le16(CF_LS4_RESPONDER << CF_LS4_SHIFT); + cmd_pkt->nport_handle = nvme->u.nvme.nport_handle; + cmd_pkt->exchange_address = nvme->u.nvme.exchange_address; + } else { + cmd_pkt->control_flags = + cpu_to_le16(CF_LS4_ORIGINATOR << CF_LS4_SHIFT); + cmd_pkt->nport_handle = cpu_to_le16(sp->fcport->loop_id); + cmd_pkt->rx_dseg_count = cpu_to_le16(1); + cmd_pkt->rx_byte_count = nvme->u.nvme.rsp_len; + cmd_pkt->dsd[1].length = nvme->u.nvme.rsp_len; + put_unaligned_le64(nvme->u.nvme.rsp_dma, &cmd_pkt->dsd[1].address); + } - cmd_pkt->rx_dseg_count = cpu_to_le16(1); - cmd_pkt->rx_byte_count = cpu_to_le32(nvme->u.nvme.rsp_len); - cmd_pkt->dsd[1].length = cpu_to_le32(nvme->u.nvme.rsp_len); - put_unaligned_le64(nvme->u.nvme.rsp_dma, &cmd_pkt->dsd[1].address); + cmd_pkt->tx_dseg_count = cpu_to_le16(1); + cmd_pkt->tx_byte_count = nvme->u.nvme.cmd_len; + cmd_pkt->dsd[0].length = nvme->u.nvme.cmd_len; + put_unaligned_le64(nvme->u.nvme.cmd_dma, &cmd_pkt->dsd[0].address); } static void @@ -4198,3 +4205,267 @@ queuing_error: return rval; } + +/** + * qla_start_scsi_type6() - Send a SCSI command to the ISP + * @sp: command to send to the ISP + * + * Returns non-zero if a failure occurred, else zero. + */ +static int +qla_start_scsi_type6(srb_t *sp) +{ + int nseg; + unsigned long flags; + uint32_t *clr_ptr; + uint32_t handle; + struct cmd_type_6 *cmd_pkt; + uint16_t cnt; + uint16_t req_cnt; + uint16_t tot_dsds; + struct req_que *req = NULL; + struct rsp_que *rsp; + struct scsi_cmnd *cmd = GET_CMD_SP(sp); + struct scsi_qla_host *vha = sp->fcport->vha; + struct qla_hw_data *ha = vha->hw; + struct qla_qpair *qpair = sp->qpair; + uint16_t more_dsd_lists = 0; + struct dsd_dma *dsd_ptr; + uint16_t i; + __be32 *fcp_dl; + uint8_t additional_cdb_len; + struct ct6_dsd *ctx; + + /* Acquire qpair specific lock */ + spin_lock_irqsave(&qpair->qp_lock, flags); + + /* Setup qpair pointers */ + req = qpair->req; + rsp = qpair->rsp; + + /* So we know we haven't pci_map'ed anything yet */ + tot_dsds = 0; + + /* Send marker if required */ + if (vha->marker_needed != 0) { + if (__qla2x00_marker(vha, qpair, 0, 0, MK_SYNC_ALL) != QLA_SUCCESS) { + spin_unlock_irqrestore(&qpair->qp_lock, flags); + return QLA_FUNCTION_FAILED; + } + vha->marker_needed = 0; + } + + handle = qla2xxx_get_next_handle(req); + if (handle == 0) + goto queuing_error; + + /* Map the sg table so we have an accurate count of sg entries needed */ + if (scsi_sg_count(cmd)) { + nseg = dma_map_sg(&ha->pdev->dev, scsi_sglist(cmd), + scsi_sg_count(cmd), cmd->sc_data_direction); + if (unlikely(!nseg)) + goto queuing_error; + } else { + nseg = 0; + } + + tot_dsds = nseg; + + /* eventhough driver only need 1 T6 IOCB, FW still convert DSD to Continueation IOCB */ + req_cnt = qla24xx_calc_iocbs(vha, tot_dsds); + + sp->iores.res_type = RESOURCE_IOCB | RESOURCE_EXCH; + sp->iores.exch_cnt = 1; + sp->iores.iocb_cnt = req_cnt; + + if (qla_get_fw_resources(sp->qpair, &sp->iores)) + goto queuing_error; + + more_dsd_lists = qla24xx_calc_dsd_lists(tot_dsds); + if ((more_dsd_lists + qpair->dsd_inuse) >= NUM_DSD_CHAIN) { + ql_dbg(ql_dbg_io, vha, 0x3028, + "Num of DSD list %d is than %d for cmd=%p.\n", + more_dsd_lists + qpair->dsd_inuse, NUM_DSD_CHAIN, cmd); + goto queuing_error; + } + + if (more_dsd_lists <= qpair->dsd_avail) + goto sufficient_dsds; + else + more_dsd_lists -= qpair->dsd_avail; + + for (i = 0; i < more_dsd_lists; i++) { + dsd_ptr = kzalloc(sizeof(*dsd_ptr), GFP_ATOMIC); + if (!dsd_ptr) { + ql_log(ql_log_fatal, vha, 0x3029, + "Failed to allocate memory for dsd_dma for cmd=%p.\n", cmd); + goto queuing_error; + } + INIT_LIST_HEAD(&dsd_ptr->list); + + dsd_ptr->dsd_addr = dma_pool_alloc(ha->dl_dma_pool, + GFP_ATOMIC, &dsd_ptr->dsd_list_dma); + if (!dsd_ptr->dsd_addr) { + kfree(dsd_ptr); + ql_log(ql_log_fatal, vha, 0x302a, + "Failed to allocate memory for dsd_addr for cmd=%p.\n", cmd); + goto queuing_error; + } + list_add_tail(&dsd_ptr->list, &qpair->dsd_list); + qpair->dsd_avail++; + } + +sufficient_dsds: + req_cnt = 1; + + if (req->cnt < (req_cnt + 2)) { + if (IS_SHADOW_REG_CAPABLE(ha)) { + cnt = *req->out_ptr; + } else { + cnt = (uint16_t)rd_reg_dword_relaxed(req->req_q_out); + if (qla2x00_check_reg16_for_disconnect(vha, cnt)) + goto queuing_error; + } + + if (req->ring_index < cnt) + req->cnt = cnt - req->ring_index; + else + req->cnt = req->length - (req->ring_index - cnt); + if (req->cnt < (req_cnt + 2)) + goto queuing_error; + } + + ctx = &sp->u.scmd.ct6_ctx; + + memset(ctx, 0, sizeof(struct ct6_dsd)); + ctx->fcp_cmnd = dma_pool_zalloc(ha->fcp_cmnd_dma_pool, + GFP_ATOMIC, &ctx->fcp_cmnd_dma); + if (!ctx->fcp_cmnd) { + ql_log(ql_log_fatal, vha, 0x3031, + "Failed to allocate fcp_cmnd for cmd=%p.\n", cmd); + goto queuing_error; + } + + /* Initialize the DSD list and dma handle */ + INIT_LIST_HEAD(&ctx->dsd_list); + ctx->dsd_use_cnt = 0; + + if (cmd->cmd_len > 16) { + additional_cdb_len = cmd->cmd_len - 16; + if (cmd->cmd_len % 4 || + cmd->cmd_len > QLA_CDB_BUF_SIZE) { + /* + * SCSI command bigger than 16 bytes must be + * multiple of 4 or too big. + */ + ql_log(ql_log_warn, vha, 0x3033, + "scsi cmd len %d not multiple of 4 for cmd=%p.\n", + cmd->cmd_len, cmd); + goto queuing_error_fcp_cmnd; + } + ctx->fcp_cmnd_len = 12 + cmd->cmd_len + 4; + } else { + additional_cdb_len = 0; + ctx->fcp_cmnd_len = 12 + 16 + 4; + } + + /* Build command packet. */ + req->current_outstanding_cmd = handle; + req->outstanding_cmds[handle] = sp; + sp->handle = handle; + cmd->host_scribble = (unsigned char *)(unsigned long)handle; + req->cnt -= req_cnt; + + cmd_pkt = (struct cmd_type_6 *)req->ring_ptr; + cmd_pkt->handle = make_handle(req->id, handle); + + /* tagged queuing modifier -- default is TSK_SIMPLE (0). */ + clr_ptr = (uint32_t *)cmd_pkt + 2; + memset(clr_ptr, 0, REQUEST_ENTRY_SIZE - 8); + cmd_pkt->dseg_count = cpu_to_le16(tot_dsds); + + /* Set NPORT-ID and LUN number */ + cmd_pkt->nport_handle = cpu_to_le16(sp->fcport->loop_id); + cmd_pkt->port_id[0] = sp->fcport->d_id.b.al_pa; + cmd_pkt->port_id[1] = sp->fcport->d_id.b.area; + cmd_pkt->port_id[2] = sp->fcport->d_id.b.domain; + cmd_pkt->vp_index = sp->vha->vp_idx; + + /* Build IOCB segments */ + qla24xx_build_scsi_type_6_iocbs(sp, cmd_pkt, tot_dsds); + + int_to_scsilun(cmd->device->lun, &cmd_pkt->lun); + host_to_fcp_swap((uint8_t *)&cmd_pkt->lun, sizeof(cmd_pkt->lun)); + + /* build FCP_CMND IU */ + int_to_scsilun(cmd->device->lun, &ctx->fcp_cmnd->lun); + ctx->fcp_cmnd->additional_cdb_len = additional_cdb_len; + + if (cmd->sc_data_direction == DMA_TO_DEVICE) + ctx->fcp_cmnd->additional_cdb_len |= 1; + else if (cmd->sc_data_direction == DMA_FROM_DEVICE) + ctx->fcp_cmnd->additional_cdb_len |= 2; + + /* Populate the FCP_PRIO. */ + if (ha->flags.fcp_prio_enabled) + ctx->fcp_cmnd->task_attribute |= + sp->fcport->fcp_prio << 3; + + memcpy(ctx->fcp_cmnd->cdb, cmd->cmnd, cmd->cmd_len); + + fcp_dl = (__be32 *)(ctx->fcp_cmnd->cdb + 16 + + additional_cdb_len); + *fcp_dl = htonl((uint32_t)scsi_bufflen(cmd)); + + cmd_pkt->fcp_cmnd_dseg_len = cpu_to_le16(ctx->fcp_cmnd_len); + put_unaligned_le64(ctx->fcp_cmnd_dma, + &cmd_pkt->fcp_cmnd_dseg_address); + + sp->flags |= SRB_FCP_CMND_DMA_VALID; + cmd_pkt->byte_count = cpu_to_le32((uint32_t)scsi_bufflen(cmd)); + /* Set total data segment count. */ + cmd_pkt->entry_count = (uint8_t)req_cnt; + + wmb(); + /* Adjust ring index. */ + req->ring_index++; + if (req->ring_index == req->length) { + req->ring_index = 0; + req->ring_ptr = req->ring; + } else { + req->ring_ptr++; + } + + sp->qpair->cmd_cnt++; + sp->flags |= SRB_DMA_VALID; + + /* Set chip new ring index. */ + wrt_reg_dword(req->req_q_in, req->ring_index); + + /* Manage unprocessed RIO/ZIO commands in response queue. */ + if (vha->flags.process_response_queue && + rsp->ring_ptr->signature != RESPONSE_PROCESSED) + qla24xx_process_response_queue(vha, rsp); + + spin_unlock_irqrestore(&qpair->qp_lock, flags); + + return QLA_SUCCESS; + +queuing_error_fcp_cmnd: + dma_pool_free(ha->fcp_cmnd_dma_pool, ctx->fcp_cmnd, ctx->fcp_cmnd_dma); + +queuing_error: + if (tot_dsds) + scsi_dma_unmap(cmd); + + qla_put_fw_resources(sp->qpair, &sp->iores); + + if (sp->u.scmd.crc_ctx) { + mempool_free(sp->u.scmd.crc_ctx, ha->ctx_mempool); + sp->u.scmd.crc_ctx = NULL; + } + + spin_unlock_irqrestore(&qpair->qp_lock, flags); + + return QLA_FUNCTION_FAILED; +} diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c index 1f42a413b598..e98788191897 100644 --- a/drivers/scsi/qla2xxx/qla_isr.c +++ b/drivers/scsi/qla2xxx/qla_isr.c @@ -56,6 +56,22 @@ const char *const port_state_str[] = { [FCS_ONLINE] = "ONLINE" }; +#define SFP_DISABLE_LASER_INITIATED 0x15 /* Sub code of 8070 AEN */ +#define SFP_ENABLE_LASER_INITIATED 0x16 /* Sub code of 8070 AEN */ + +static inline void display_Laser_info(scsi_qla_host_t *vha, + u16 mb1, u16 mb2, u16 mb3) { + + if (mb1 == SFP_DISABLE_LASER_INITIATED) + ql_log(ql_log_warn, vha, 0xf0a2, + "SFP temperature (%d C) reached/exceeded the threshold (%d C). Laser is disabled.\n", + mb3, mb2); + if (mb1 == SFP_ENABLE_LASER_INITIATED) + ql_log(ql_log_warn, vha, 0xf0a3, + "SFP temperature (%d C) reached normal operating level. Laser is enabled.\n", + mb3); +} + static void qla24xx_process_abts(struct scsi_qla_host *vha, struct purex_item *pkt) { @@ -823,6 +839,135 @@ qla83xx_handle_8200_aen(scsi_qla_host_t *vha, uint16_t *mb) } } +/** + * qla27xx_copy_multiple_pkt() - Copy over purex/purls packets that can + * span over multiple IOCBs. + * @vha: SCSI driver HA context + * @pkt: ELS packet + * @rsp: Response queue + * @is_purls: True, for Unsolicited Received FC-NVMe LS rsp IOCB + * false, for Unsolicited Received ELS IOCB + * @byte_order: True, to change the byte ordering of iocb payload + */ +struct purex_item * +qla27xx_copy_multiple_pkt(struct scsi_qla_host *vha, void **pkt, + struct rsp_que **rsp, bool is_purls, + bool byte_order) +{ + struct purex_entry_24xx *purex = NULL; + struct pt_ls4_rx_unsol *purls = NULL; + struct rsp_que *rsp_q = *rsp; + sts_cont_entry_t *new_pkt; + uint16_t no_bytes = 0, total_bytes = 0, pending_bytes = 0; + uint16_t buffer_copy_offset = 0, payload_size = 0; + uint16_t entry_count, entry_count_remaining; + struct purex_item *item; + void *iocb_pkt = NULL; + + if (is_purls) { + purls = *pkt; + total_bytes = (le16_to_cpu(purls->frame_size) & 0x0FFF) - + PURX_ELS_HEADER_SIZE; + entry_count = entry_count_remaining = purls->entry_count; + payload_size = sizeof(purls->payload); + } else { + purex = *pkt; + total_bytes = (le16_to_cpu(purex->frame_size) & 0x0FFF) - + PURX_ELS_HEADER_SIZE; + entry_count = entry_count_remaining = purex->entry_count; + payload_size = sizeof(purex->els_frame_payload); + } + + pending_bytes = total_bytes; + no_bytes = (pending_bytes > payload_size) ? payload_size : + pending_bytes; + ql_dbg(ql_dbg_async, vha, 0x509a, + "%s LS, frame_size 0x%x, entry count %d\n", + (is_purls ? "PURLS" : "FPIN"), total_bytes, entry_count); + + item = qla24xx_alloc_purex_item(vha, total_bytes); + if (!item) + return item; + + iocb_pkt = &item->iocb; + + if (is_purls) + memcpy(iocb_pkt, &purls->payload[0], no_bytes); + else + memcpy(iocb_pkt, &purex->els_frame_payload[0], no_bytes); + buffer_copy_offset += no_bytes; + pending_bytes -= no_bytes; + --entry_count_remaining; + + if (is_purls) + ((response_t *)purls)->signature = RESPONSE_PROCESSED; + else + ((response_t *)purex)->signature = RESPONSE_PROCESSED; + wmb(); + + do { + while ((total_bytes > 0) && (entry_count_remaining > 0)) { + if (rsp_q->ring_ptr->signature == RESPONSE_PROCESSED) { + ql_dbg(ql_dbg_async, vha, 0x5084, + "Ran out of IOCBs, partial data 0x%x\n", + buffer_copy_offset); + cpu_relax(); + continue; + } + + new_pkt = (sts_cont_entry_t *)rsp_q->ring_ptr; + *pkt = new_pkt; + + if (new_pkt->entry_type != STATUS_CONT_TYPE) { + ql_log(ql_log_warn, vha, 0x507a, + "Unexpected IOCB type, partial data 0x%x\n", + buffer_copy_offset); + break; + } + + rsp_q->ring_index++; + if (rsp_q->ring_index == rsp_q->length) { + rsp_q->ring_index = 0; + rsp_q->ring_ptr = rsp_q->ring; + } else { + rsp_q->ring_ptr++; + } + no_bytes = (pending_bytes > sizeof(new_pkt->data)) ? + sizeof(new_pkt->data) : pending_bytes; + if ((buffer_copy_offset + no_bytes) <= total_bytes) { + memcpy(((uint8_t *)iocb_pkt + buffer_copy_offset), + new_pkt->data, no_bytes); + buffer_copy_offset += no_bytes; + pending_bytes -= no_bytes; + --entry_count_remaining; + } else { + ql_log(ql_log_warn, vha, 0x5044, + "Attempt to copy more that we got, optimizing..%x\n", + buffer_copy_offset); + memcpy(((uint8_t *)iocb_pkt + buffer_copy_offset), + new_pkt->data, + total_bytes - buffer_copy_offset); + } + + ((response_t *)new_pkt)->signature = RESPONSE_PROCESSED; + wmb(); + } + + if (pending_bytes != 0 || entry_count_remaining != 0) { + ql_log(ql_log_fatal, vha, 0x508b, + "Dropping partial FPIN, underrun bytes = 0x%x, entry cnts 0x%x\n", + total_bytes, entry_count_remaining); + qla24xx_free_purex_item(item); + return NULL; + } + } while (entry_count_remaining > 0); + + if (byte_order) + host_to_fcp_swap((uint8_t *)&item->iocb, total_bytes); + + return item; +} + int qla2x00_is_a_vp_did(scsi_qla_host_t *vha, uint32_t rscn_entry) { @@ -958,7 +1103,7 @@ initialize_purex_header: return item; } -static void +void qla24xx_queue_purex_item(scsi_qla_host_t *vha, struct purex_item *pkt, void (*process_item)(struct scsi_qla_host *vha, struct purex_item *pkt)) @@ -1798,6 +1943,8 @@ global_port_update: break; case MBA_TEMPERATURE_ALERT: + if (IS_QLA27XX(ha) || IS_QLA28XX(ha)) + display_Laser_info(vha, mb[1], mb[2], mb[3]); ql_dbg(ql_dbg_async, vha, 0x505e, "TEMPERATURE ALERT: %04x %04x %04x\n", mb[1], mb[2], mb[3]); break; @@ -3811,6 +3958,7 @@ void qla24xx_process_response_queue(struct scsi_qla_host *vha, struct qla_hw_data *ha = vha->hw; struct purex_entry_24xx *purex_entry; struct purex_item *pure_item; + struct pt_ls4_rx_unsol *p; u16 rsp_in = 0, cur_ring_index; int is_shadow_hba; @@ -3983,7 +4131,19 @@ process_err: qla28xx_sa_update_iocb_entry(vha, rsp->req, (struct sa_update_28xx *)pkt); break; + case PT_LS4_UNSOL: + p = (void *)pkt; + if (qla_chk_cont_iocb_avail(vha, rsp, (response_t *)pkt, rsp_in)) { + rsp->ring_ptr = (response_t *)pkt; + rsp->ring_index = cur_ring_index; + ql_dbg(ql_dbg_init, vha, 0x2124, + "Defer processing UNSOL LS req opcode %#x...\n", + p->payload[0]); + return; + } + qla2xxx_process_purls_iocb((void **)&pkt, &rsp); + break; default: /* Type Not Supported. */ ql_dbg(ql_dbg_async, vha, 0x5042, diff --git a/drivers/scsi/qla2xxx/qla_mbx.c b/drivers/scsi/qla2xxx/qla_mbx.c index b05f93037875..21ec32b4fb28 100644 --- a/drivers/scsi/qla2xxx/qla_mbx.c +++ b/drivers/scsi/qla2xxx/qla_mbx.c @@ -273,7 +273,6 @@ qla2x00_mailbox_command(scsi_qla_host_t *vha, mbx_cmd_t *mcp) spin_unlock_irqrestore(&ha->hardware_lock, flags); wait_time = jiffies; - atomic_inc(&ha->num_pend_mbx_stage3); if (!wait_for_completion_timeout(&ha->mbx_intr_comp, mcp->tov * HZ)) { ql_dbg(ql_dbg_mbx, vha, 0x117a, @@ -290,7 +289,6 @@ qla2x00_mailbox_command(scsi_qla_host_t *vha, mbx_cmd_t *mcp) spin_unlock_irqrestore(&ha->hardware_lock, flags); atomic_dec(&ha->num_pend_mbx_stage2); - atomic_dec(&ha->num_pend_mbx_stage3); rval = QLA_ABORTED; goto premature_exit; } @@ -302,11 +300,9 @@ qla2x00_mailbox_command(scsi_qla_host_t *vha, mbx_cmd_t *mcp) ha->flags.mbox_busy = 0; spin_unlock_irqrestore(&ha->hardware_lock, flags); atomic_dec(&ha->num_pend_mbx_stage2); - atomic_dec(&ha->num_pend_mbx_stage3); rval = QLA_ABORTED; goto premature_exit; } - atomic_dec(&ha->num_pend_mbx_stage3); if (time_after(jiffies, wait_time + 5 * HZ)) ql_log(ql_log_warn, vha, 0x1015, "cmd=0x%x, waited %d msecs\n", diff --git a/drivers/scsi/qla2xxx/qla_nvme.c b/drivers/scsi/qla2xxx/qla_nvme.c index 9941b38eac93..db753d712991 100644 --- a/drivers/scsi/qla2xxx/qla_nvme.c +++ b/drivers/scsi/qla2xxx/qla_nvme.c @@ -12,6 +12,26 @@ #include static struct nvme_fc_port_template qla_nvme_fc_transport; +static int qla_nvme_ls_reject_iocb(struct scsi_qla_host *vha, + struct qla_qpair *qp, + struct qla_nvme_lsrjt_pt_arg *a, + bool is_xchg_terminate); + +struct qla_nvme_unsol_ctx { + struct list_head elem; + struct scsi_qla_host *vha; + struct fc_port *fcport; + struct srb *sp; + struct nvmefc_ls_rsp lsrsp; + struct nvmefc_ls_rsp *fd_rsp; + struct work_struct lsrsp_work; + struct work_struct abort_work; + __le32 exchange_address; + __le16 nport_handle; + __le16 ox_id; + int comp_status; + spinlock_t cmd_lock; +}; int qla_nvme_register_remote(struct scsi_qla_host *vha, struct fc_port *fcport) { @@ -216,6 +236,55 @@ static void qla_nvme_sp_ls_done(srb_t *sp, int res) schedule_work(&priv->ls_work); } +static void qla_nvme_release_lsrsp_cmd_kref(struct kref *kref) +{ + struct srb *sp = container_of(kref, struct srb, cmd_kref); + struct qla_nvme_unsol_ctx *uctx = sp->priv; + struct nvmefc_ls_rsp *fd_rsp; + unsigned long flags; + + if (!uctx) { + qla2x00_rel_sp(sp); + return; + } + + spin_lock_irqsave(&uctx->cmd_lock, flags); + uctx->sp = NULL; + sp->priv = NULL; + spin_unlock_irqrestore(&uctx->cmd_lock, flags); + + fd_rsp = uctx->fd_rsp; + + list_del(&uctx->elem); + + fd_rsp->done(fd_rsp); + kfree(uctx); + qla2x00_rel_sp(sp); +} + +static void qla_nvme_lsrsp_complete(struct work_struct *work) +{ + struct qla_nvme_unsol_ctx *uctx = + container_of(work, struct qla_nvme_unsol_ctx, lsrsp_work); + + kref_put(&uctx->sp->cmd_kref, qla_nvme_release_lsrsp_cmd_kref); +} + +static void qla_nvme_sp_lsrsp_done(srb_t *sp, int res) +{ + struct qla_nvme_unsol_ctx *uctx = sp->priv; + + if (WARN_ON_ONCE(kref_read(&sp->cmd_kref) == 0)) + return; + + if (res) + res = -EINVAL; + + uctx->comp_status = res; + INIT_WORK(&uctx->lsrsp_work, qla_nvme_lsrsp_complete); + schedule_work(&uctx->lsrsp_work); +} + /* it assumed that QPair lock is held. */ static void qla_nvme_sp_done(srb_t *sp, int res) { @@ -288,6 +357,92 @@ out: kref_put(&sp->cmd_kref, sp->put_fn); } +static int qla_nvme_xmt_ls_rsp(struct nvme_fc_local_port *lport, + struct nvme_fc_remote_port *rport, + struct nvmefc_ls_rsp *fd_resp) +{ + struct qla_nvme_unsol_ctx *uctx = container_of(fd_resp, + struct qla_nvme_unsol_ctx, lsrsp); + struct qla_nvme_rport *qla_rport = rport->private; + fc_port_t *fcport = qla_rport->fcport; + struct scsi_qla_host *vha = uctx->vha; + struct qla_hw_data *ha = vha->hw; + struct qla_nvme_lsrjt_pt_arg a; + struct srb_iocb *nvme; + srb_t *sp; + int rval = QLA_FUNCTION_FAILED; + uint8_t cnt = 0; + + if (!fcport || fcport->deleted) + goto out; + + if (!ha->flags.fw_started) + goto out; + + /* Alloc SRB structure */ + sp = qla2x00_get_sp(vha, fcport, GFP_ATOMIC); + if (!sp) + goto out; + + sp->type = SRB_NVME_LS; + sp->name = "nvme_ls"; + sp->done = qla_nvme_sp_lsrsp_done; + sp->put_fn = qla_nvme_release_lsrsp_cmd_kref; + sp->priv = (void *)uctx; + sp->unsol_rsp = 1; + uctx->sp = sp; + spin_lock_init(&uctx->cmd_lock); + nvme = &sp->u.iocb_cmd; + uctx->fd_rsp = fd_resp; + nvme->u.nvme.desc = fd_resp; + nvme->u.nvme.dir = 0; + nvme->u.nvme.dl = 0; + nvme->u.nvme.timeout_sec = 0; + nvme->u.nvme.cmd_dma = fd_resp->rspdma; + nvme->u.nvme.cmd_len = fd_resp->rsplen; + nvme->u.nvme.rsp_len = 0; + nvme->u.nvme.rsp_dma = 0; + nvme->u.nvme.exchange_address = uctx->exchange_address; + nvme->u.nvme.nport_handle = uctx->nport_handle; + nvme->u.nvme.ox_id = uctx->ox_id; + dma_sync_single_for_device(&ha->pdev->dev, nvme->u.nvme.cmd_dma, + le32_to_cpu(fd_resp->rsplen), DMA_TO_DEVICE); + + ql_dbg(ql_dbg_unsol, vha, 0x2122, + "Unsol lsreq portid=%06x %8phC exchange_address 0x%x ox_id 0x%x hdl 0x%x\n", + fcport->d_id.b24, fcport->port_name, uctx->exchange_address, + uctx->ox_id, uctx->nport_handle); +retry: + rval = qla2x00_start_sp(sp); + switch (rval) { + case QLA_SUCCESS: + break; + case EAGAIN: + msleep(PURLS_MSLEEP_INTERVAL); + cnt++; + if (cnt < PURLS_RETRY_COUNT) + goto retry; + + fallthrough; + default: + ql_dbg(ql_log_warn, vha, 0x2123, + "Failed to xmit Unsol ls response = %d\n", rval); + rval = -EIO; + qla2x00_rel_sp(sp); + goto out; + } + + return 0; +out: + memset((void *)&a, 0, sizeof(a)); + a.vp_idx = vha->vp_idx; + a.nport_handle = uctx->nport_handle; + a.xchg_address = uctx->exchange_address; + qla_nvme_ls_reject_iocb(vha, ha->base_qpair, &a, true); + kfree(uctx); + return rval; +} + static void qla_nvme_ls_abort(struct nvme_fc_local_port *lport, struct nvme_fc_remote_port *rport, struct nvmefc_ls_req *fd) { @@ -355,7 +510,7 @@ static int qla_nvme_ls_req(struct nvme_fc_local_port *lport, nvme->u.nvme.timeout_sec = fd->timeout; nvme->u.nvme.cmd_dma = fd->rqstdma; dma_sync_single_for_device(&ha->pdev->dev, nvme->u.nvme.cmd_dma, - fd->rqstlen, DMA_TO_DEVICE); + le32_to_cpu(fd->rqstlen), DMA_TO_DEVICE); rval = qla2x00_start_sp(sp); if (rval != QLA_SUCCESS) { @@ -720,6 +875,7 @@ static struct nvme_fc_port_template qla_nvme_fc_transport = { .ls_abort = qla_nvme_ls_abort, .fcp_io = qla_nvme_post_cmd, .fcp_abort = qla_nvme_fcp_abort, + .xmt_ls_rsp = qla_nvme_xmt_ls_rsp, .map_queues = qla_nvme_map_queues, .max_hw_queues = DEF_NVME_HW_QUEUES, .max_sgl_segments = 1024, @@ -924,3 +1080,247 @@ inline void qla_wait_nvme_release_cmd_kref(srb_t *orig_sp) return; kref_put(&orig_sp->cmd_kref, orig_sp->put_fn); } + +static void qla_nvme_fc_format_rjt(void *buf, u8 ls_cmd, u8 reason, + u8 explanation, u8 vendor) +{ + struct fcnvme_ls_rjt *rjt = buf; + + rjt->w0.ls_cmd = FCNVME_LSDESC_RQST; + rjt->desc_list_len = fcnvme_lsdesc_len(sizeof(struct fcnvme_ls_rjt)); + rjt->rqst.desc_tag = cpu_to_be32(FCNVME_LSDESC_RQST); + rjt->rqst.desc_len = + fcnvme_lsdesc_len(sizeof(struct fcnvme_lsdesc_rqst)); + rjt->rqst.w0.ls_cmd = ls_cmd; + rjt->rjt.desc_tag = cpu_to_be32(FCNVME_LSDESC_RJT); + rjt->rjt.desc_len = fcnvme_lsdesc_len(sizeof(struct fcnvme_lsdesc_rjt)); + rjt->rjt.reason_code = reason; + rjt->rjt.reason_explanation = explanation; + rjt->rjt.vendor = vendor; +} + +static void qla_nvme_lsrjt_pt_iocb(struct scsi_qla_host *vha, + struct pt_ls4_request *lsrjt_iocb, + struct qla_nvme_lsrjt_pt_arg *a) +{ + lsrjt_iocb->entry_type = PT_LS4_REQUEST; + lsrjt_iocb->entry_count = 1; + lsrjt_iocb->sys_define = 0; + lsrjt_iocb->entry_status = 0; + lsrjt_iocb->handle = QLA_SKIP_HANDLE; + lsrjt_iocb->nport_handle = a->nport_handle; + lsrjt_iocb->exchange_address = a->xchg_address; + lsrjt_iocb->vp_index = a->vp_idx; + + lsrjt_iocb->control_flags = cpu_to_le16(a->control_flags); + + put_unaligned_le64(a->tx_addr, &lsrjt_iocb->dsd[0].address); + lsrjt_iocb->dsd[0].length = cpu_to_le32(a->tx_byte_count); + lsrjt_iocb->tx_dseg_count = cpu_to_le16(1); + lsrjt_iocb->tx_byte_count = cpu_to_le32(a->tx_byte_count); + + put_unaligned_le64(a->rx_addr, &lsrjt_iocb->dsd[1].address); + lsrjt_iocb->dsd[1].length = 0; + lsrjt_iocb->rx_dseg_count = 0; + lsrjt_iocb->rx_byte_count = 0; +} + +static int +qla_nvme_ls_reject_iocb(struct scsi_qla_host *vha, struct qla_qpair *qp, + struct qla_nvme_lsrjt_pt_arg *a, bool is_xchg_terminate) +{ + struct pt_ls4_request *lsrjt_iocb; + + lsrjt_iocb = __qla2x00_alloc_iocbs(qp, NULL); + if (!lsrjt_iocb) { + ql_log(ql_log_warn, vha, 0x210e, + "qla2x00_alloc_iocbs failed.\n"); + return QLA_FUNCTION_FAILED; + } + + if (!is_xchg_terminate) { + qla_nvme_fc_format_rjt((void *)vha->hw->lsrjt.c, a->opcode, + a->reason, a->explanation, 0); + + a->tx_byte_count = sizeof(struct fcnvme_ls_rjt); + a->tx_addr = vha->hw->lsrjt.cdma; + a->control_flags = CF_LS4_RESPONDER << CF_LS4_SHIFT; + + ql_dbg(ql_dbg_unsol, vha, 0x211f, + "Sending nvme fc ls reject ox_id %04x op %04x\n", + a->ox_id, a->opcode); + ql_dump_buffer(ql_dbg_unsol + ql_dbg_verbose, vha, 0x210f, + vha->hw->lsrjt.c, sizeof(*vha->hw->lsrjt.c)); + } else { + a->tx_byte_count = 0; + a->control_flags = CF_LS4_RESPONDER_TERM << CF_LS4_SHIFT; + ql_dbg(ql_dbg_unsol, vha, 0x2110, + "Terminate nvme ls xchg 0x%x\n", a->xchg_address); + } + + qla_nvme_lsrjt_pt_iocb(vha, lsrjt_iocb, a); + /* flush iocb to mem before notifying hw doorbell */ + wmb(); + qla2x00_start_iocbs(vha, qp->req); + return 0; +} + +/* + * qla2xxx_process_purls_pkt() - Pass-up Unsolicited + * Received FC-NVMe Link Service pkt to nvme_fc_rcv_ls_req(). + * LLDD need to provide memory for response buffer, which + * will be used to reference the exchange corresponding + * to the LS when issuing an ls response. LLDD will have to free + * response buffer in lport->ops->xmt_ls_rsp(). + * + * @vha: SCSI qla host + * @item: ptr to purex_item + */ +static void +qla2xxx_process_purls_pkt(struct scsi_qla_host *vha, struct purex_item *item) +{ + struct qla_nvme_unsol_ctx *uctx = item->purls_context; + struct qla_nvme_lsrjt_pt_arg a; + int ret = 1; + +#if (IS_ENABLED(CONFIG_NVME_FC)) + ret = nvme_fc_rcv_ls_req(uctx->fcport->nvme_remote_port, &uctx->lsrsp, + &item->iocb, item->size); +#endif + if (ret) { + ql_dbg(ql_dbg_unsol, vha, 0x2125, "NVMe transport ls_req failed\n"); + memset((void *)&a, 0, sizeof(a)); + a.vp_idx = vha->vp_idx; + a.nport_handle = uctx->nport_handle; + a.xchg_address = uctx->exchange_address; + qla_nvme_ls_reject_iocb(vha, vha->hw->base_qpair, &a, true); + list_del(&uctx->elem); + kfree(uctx); + } +} + +static scsi_qla_host_t * +qla2xxx_get_vha_from_vp_idx(struct qla_hw_data *ha, uint16_t vp_index) +{ + scsi_qla_host_t *base_vha, *vha, *tvp; + unsigned long flags; + + base_vha = pci_get_drvdata(ha->pdev); + + if (!vp_index && !ha->num_vhosts) + return base_vha; + + spin_lock_irqsave(&ha->vport_slock, flags); + list_for_each_entry_safe(vha, tvp, &ha->vp_list, list) { + if (vha->vp_idx == vp_index) { + spin_unlock_irqrestore(&ha->vport_slock, flags); + return vha; + } + } + spin_unlock_irqrestore(&ha->vport_slock, flags); + + return NULL; +} + +void qla2xxx_process_purls_iocb(void **pkt, struct rsp_que **rsp) +{ + struct nvme_fc_remote_port *rport; + struct qla_nvme_rport *qla_rport; + struct qla_nvme_lsrjt_pt_arg a; + struct pt_ls4_rx_unsol *p = *pkt; + struct qla_nvme_unsol_ctx *uctx; + struct rsp_que *rsp_q = *rsp; + struct qla_hw_data *ha; + scsi_qla_host_t *vha; + fc_port_t *fcport = NULL; + struct purex_item *item; + port_id_t d_id = {0}; + port_id_t id = {0}; + u8 *opcode; + bool xmt_reject = false; + + ha = rsp_q->hw; + + vha = qla2xxx_get_vha_from_vp_idx(ha, p->vp_index); + if (!vha) { + ql_log(ql_log_warn, NULL, 0x2110, "Invalid vp index %d\n", p->vp_index); + WARN_ON_ONCE(1); + return; + } + + memset((void *)&a, 0, sizeof(a)); + opcode = (u8 *)&p->payload[0]; + a.opcode = opcode[3]; + a.vp_idx = p->vp_index; + a.nport_handle = p->nport_handle; + a.ox_id = p->ox_id; + a.xchg_address = p->exchange_address; + + id.b.domain = p->s_id.domain; + id.b.area = p->s_id.area; + id.b.al_pa = p->s_id.al_pa; + d_id.b.domain = p->d_id[2]; + d_id.b.area = p->d_id[1]; + d_id.b.al_pa = p->d_id[0]; + + fcport = qla2x00_find_fcport_by_nportid(vha, &id, 0); + if (!fcport) { + ql_dbg(ql_dbg_unsol, vha, 0x211e, + "Failed to find sid=%06x did=%06x\n", + id.b24, d_id.b24); + a.reason = FCNVME_RJT_RC_INV_ASSOC; + a.explanation = FCNVME_RJT_EXP_NONE; + xmt_reject = true; + goto out; + } + rport = fcport->nvme_remote_port; + qla_rport = rport->private; + + item = qla27xx_copy_multiple_pkt(vha, pkt, rsp, true, false); + if (!item) { + a.reason = FCNVME_RJT_RC_LOGIC; + a.explanation = FCNVME_RJT_EXP_NONE; + xmt_reject = true; + goto out; + } + + uctx = kzalloc(sizeof(*uctx), GFP_ATOMIC); + if (!uctx) { + ql_log(ql_log_info, vha, 0x2126, "Failed allocate memory\n"); + a.reason = FCNVME_RJT_RC_LOGIC; + a.explanation = FCNVME_RJT_EXP_NONE; + xmt_reject = true; + kfree(item); + goto out; + } + + uctx->vha = vha; + uctx->fcport = fcport; + uctx->exchange_address = p->exchange_address; + uctx->nport_handle = p->nport_handle; + uctx->ox_id = p->ox_id; + qla_rport->uctx = uctx; + INIT_LIST_HEAD(&uctx->elem); + list_add_tail(&uctx->elem, &fcport->unsol_ctx_head); + item->purls_context = (void *)uctx; + + ql_dbg(ql_dbg_unsol, vha, 0x2121, + "PURLS OP[%01x] size %d xchg addr 0x%x portid %06x\n", + item->iocb.iocb[3], item->size, uctx->exchange_address, + fcport->d_id.b24); + /* +48 0 1 2 3 4 5 6 7 8 9 A B C D E F + * ----- ----------------------------------------------- + * 0000: 00 00 00 05 28 00 00 00 07 00 00 00 08 00 00 00 + * 0010: ab ec 0f cc 00 00 8d 7d 05 00 00 00 10 00 00 00 + * 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 + */ + ql_dump_buffer(ql_dbg_unsol + ql_dbg_verbose, vha, 0x2120, + &item->iocb, item->size); + + qla24xx_queue_purex_item(vha, item, qla2xxx_process_purls_pkt); +out: + if (xmt_reject) { + qla_nvme_ls_reject_iocb(vha, (*rsp)->qpair, &a, false); + __qla_consume_iocb(vha, pkt, rsp); + } +} diff --git a/drivers/scsi/qla2xxx/qla_nvme.h b/drivers/scsi/qla2xxx/qla_nvme.h index d299478371b2..a253ac55171b 100644 --- a/drivers/scsi/qla2xxx/qla_nvme.h +++ b/drivers/scsi/qla2xxx/qla_nvme.h @@ -21,6 +21,7 @@ #define Q2T_NVME_NUM_TAGS 2048 #define QLA_MAX_FC_SEGMENTS 64 +struct qla_nvme_unsol_ctx; struct scsi_qla_host; struct qla_hw_data; struct req_que; @@ -37,6 +38,7 @@ struct nvme_private { struct qla_nvme_rport { struct fc_port *fcport; + struct qla_nvme_unsol_ctx *uctx; }; #define COMMAND_NVME 0x88 /* Command Type FC-NVMe IOCB */ @@ -75,6 +77,9 @@ struct cmd_nvme { struct dsd64 nvme_dsd; }; +#define PURLS_MSLEEP_INTERVAL 1 +#define PURLS_RETRY_COUNT 5 + #define PT_LS4_REQUEST 0x89 /* Link Service pass-through IOCB (request) */ struct pt_ls4_request { uint8_t entry_type; @@ -118,21 +123,19 @@ struct pt_ls4_rx_unsol { __le32 exchange_address; uint8_t d_id[3]; uint8_t r_ctl; - be_id_t s_id; + le_id_t s_id; uint8_t cs_ctl; uint8_t f_ctl[3]; uint8_t type; __le16 seq_cnt; uint8_t df_ctl; uint8_t seq_id; - __le16 rx_id; - __le16 ox_id; - __le32 param; - __le32 desc0; + __le16 rx_id; + __le16 ox_id; + __le32 desc0; #define PT_LS4_PAYLOAD_OFFSET 0x2c #define PT_LS4_FIRST_PACKET_LEN 20 - __le32 desc_len; - __le32 payload[3]; + __le32 payload[5]; }; /* diff --git a/drivers/scsi/qla2xxx/qla_nx.h b/drivers/scsi/qla2xxx/qla_nx.h index 6dc80c8ddf79..5d1bdc15b75c 100644 --- a/drivers/scsi/qla2xxx/qla_nx.h +++ b/drivers/scsi/qla2xxx/qla_nx.h @@ -857,7 +857,9 @@ struct fcp_cmnd { uint8_t task_attribute; uint8_t task_management; uint8_t additional_cdb_len; - uint8_t cdb[260]; /* 256 for CDB len and 4 for FCP_DL */ +#define QLA_CDB_BUF_SIZE 256 +#define QLA_FCP_DL_SIZE 4 + uint8_t cdb[QLA_CDB_BUF_SIZE + QLA_FCP_DL_SIZE]; /* 256 for CDB len and 4 for FCP_DL */ }; struct dsd_dma { diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c index 03bc3a0b45b6..50db08265c51 100644 --- a/drivers/scsi/qla2xxx/qla_os.c +++ b/drivers/scsi/qla2xxx/qla_os.c @@ -44,10 +44,11 @@ module_param(ql2xfulldump_on_mpifail, int, S_IRUGO | S_IWUSR); MODULE_PARM_DESC(ql2xfulldump_on_mpifail, "Set this to take full dump on MPI hang."); -int ql2xenforce_iocb_limit = 1; +int ql2xenforce_iocb_limit = 2; module_param(ql2xenforce_iocb_limit, int, S_IRUGO | S_IWUSR); MODULE_PARM_DESC(ql2xenforce_iocb_limit, - "Enforce IOCB throttling, to avoid FW congestion. (default: 1)"); + "Enforce IOCB throttling, to avoid FW congestion. (default: 2) " + "1: track usage per queue, 2: track usage per adapter"); /* * CT6 CTX allocation cache @@ -432,6 +433,7 @@ static void qla_init_base_qpair(struct scsi_qla_host *vha, struct req_que *req, ha->base_qpair->msix = &ha->msix_entries[QLA_MSIX_RSP_Q]; ha->base_qpair->srb_mempool = ha->srb_mempool; INIT_LIST_HEAD(&ha->base_qpair->hints_list); + INIT_LIST_HEAD(&ha->base_qpair->dsd_list); ha->base_qpair->enable_class_2 = ql2xenableclass2; /* init qpair to this cpu. Will adjust at run time. */ qla_cpu_update(rsp->qpair, raw_smp_processor_id()); @@ -750,9 +752,9 @@ void qla2x00_sp_free_dma(srb_t *sp) dma_pool_free(ha->fcp_cmnd_dma_pool, ctx1->fcp_cmnd, ctx1->fcp_cmnd_dma); - list_splice(&ctx1->dsd_list, &ha->gbl_dsd_list); - ha->gbl_dsd_inuse -= ctx1->dsd_use_cnt; - ha->gbl_dsd_avail += ctx1->dsd_use_cnt; + list_splice(&ctx1->dsd_list, &sp->qpair->dsd_list); + sp->qpair->dsd_inuse -= ctx1->dsd_use_cnt; + sp->qpair->dsd_avail += ctx1->dsd_use_cnt; } if (sp->flags & SRB_GOT_BUF) @@ -836,9 +838,9 @@ void qla2xxx_qpair_sp_free_dma(srb_t *sp) dma_pool_free(ha->fcp_cmnd_dma_pool, ctx1->fcp_cmnd, ctx1->fcp_cmnd_dma); - list_splice(&ctx1->dsd_list, &ha->gbl_dsd_list); - ha->gbl_dsd_inuse -= ctx1->dsd_use_cnt; - ha->gbl_dsd_avail += ctx1->dsd_use_cnt; + list_splice(&ctx1->dsd_list, &sp->qpair->dsd_list); + sp->qpair->dsd_inuse -= ctx1->dsd_use_cnt; + sp->qpair->dsd_avail += ctx1->dsd_use_cnt; sp->flags &= ~SRB_FCP_CMND_DMA_VALID; } @@ -3007,7 +3009,6 @@ qla2x00_probe_one(struct pci_dev *pdev, const struct pci_device_id *id) ha->max_exchg = FW_MAX_EXCHANGES_CNT; atomic_set(&ha->num_pend_mbx_stage1, 0); atomic_set(&ha->num_pend_mbx_stage2, 0); - atomic_set(&ha->num_pend_mbx_stage3, 0); atomic_set(&ha->zio_threshold, DEFAULT_ZIO_THRESHOLD); ha->last_zio_threshold = DEFAULT_ZIO_THRESHOLD; INIT_LIST_HEAD(&ha->tmf_pending); @@ -3288,6 +3289,13 @@ qla2x00_probe_one(struct pci_dev *pdev, const struct pci_device_id *id) host->max_id = ha->max_fibre_devices; host->cmd_per_lun = 3; host->unique_id = host->host_no; + + if (ql2xenabledif && ql2xenabledif != 2) { + ql_log(ql_log_warn, base_vha, 0x302d, + "Invalid value for ql2xenabledif, resetting it to default (2)\n"); + ql2xenabledif = 2; + } + if (IS_T10_PI_CAPABLE(ha) && ql2xenabledif) host->max_cmd_len = 32; else @@ -3524,8 +3532,6 @@ skip_dpc: base_vha->flags.difdix_supported = 1; ql_dbg(ql_dbg_init, base_vha, 0x00f1, "Registering for DIF/DIX type 1 and 3 protection.\n"); - if (ql2xenabledif == 1) - prot = SHOST_DIX_TYPE0_PROTECTION; if (ql2xprotmask) scsi_host_set_prot(host, ql2xprotmask); else @@ -4402,7 +4408,6 @@ qla2x00_mem_alloc(struct qla_hw_data *ha, uint16_t req_len, uint16_t rsp_len, "sf_init_cb=%p.\n", ha->sf_init_cb); } - INIT_LIST_HEAD(&ha->gbl_dsd_list); /* Get consistent memory allocated for Async Port-Database. */ if (!IS_FWI2_CAPABLE(ha)) { @@ -4457,8 +4462,9 @@ qla2x00_mem_alloc(struct qla_hw_data *ha, uint16_t req_len, uint16_t rsp_len, ha->elsrej.size = sizeof(struct fc_els_ls_rjt) + 16; ha->elsrej.c = dma_alloc_coherent(&ha->pdev->dev, - ha->elsrej.size, &ha->elsrej.cdma, GFP_KERNEL); - + ha->elsrej.size, + &ha->elsrej.cdma, + GFP_KERNEL); if (!ha->elsrej.c) { ql_dbg_pci(ql_dbg_init, ha->pdev, 0xffff, "Alloc failed for els reject cmd.\n"); @@ -4467,8 +4473,21 @@ qla2x00_mem_alloc(struct qla_hw_data *ha, uint16_t req_len, uint16_t rsp_len, ha->elsrej.c->er_cmd = ELS_LS_RJT; ha->elsrej.c->er_reason = ELS_RJT_LOGIC; ha->elsrej.c->er_explan = ELS_EXPL_UNAB_DATA; + + ha->lsrjt.size = sizeof(struct fcnvme_ls_rjt); + ha->lsrjt.c = dma_alloc_coherent(&ha->pdev->dev, ha->lsrjt.size, + &ha->lsrjt.cdma, GFP_KERNEL); + if (!ha->lsrjt.c) { + ql_dbg_pci(ql_dbg_init, ha->pdev, 0xffff, + "Alloc failed for nvme fc reject cmd.\n"); + goto fail_lsrjt; + } + return 0; +fail_lsrjt: + dma_free_coherent(&ha->pdev->dev, ha->elsrej.size, + ha->elsrej.c, ha->elsrej.cdma); fail_elsrej: dma_pool_destroy(ha->purex_dma_pool); fail_flt: @@ -4934,18 +4953,16 @@ qla2x00_mem_free(struct qla_hw_data *ha) ha->gid_list = NULL; ha->gid_list_dma = 0; - if (IS_QLA82XX(ha)) { - if (!list_empty(&ha->gbl_dsd_list)) { - struct dsd_dma *dsd_ptr, *tdsd_ptr; + if (!list_empty(&ha->base_qpair->dsd_list)) { + struct dsd_dma *dsd_ptr, *tdsd_ptr; - /* clean up allocated prev pool */ - list_for_each_entry_safe(dsd_ptr, - tdsd_ptr, &ha->gbl_dsd_list, list) { - dma_pool_free(ha->dl_dma_pool, - dsd_ptr->dsd_addr, dsd_ptr->dsd_list_dma); - list_del(&dsd_ptr->list); - kfree(dsd_ptr); - } + /* clean up allocated prev pool */ + list_for_each_entry_safe(dsd_ptr, tdsd_ptr, + &ha->base_qpair->dsd_list, list) { + dma_pool_free(ha->dl_dma_pool, dsd_ptr->dsd_addr, + dsd_ptr->dsd_list_dma); + list_del(&dsd_ptr->list); + kfree(dsd_ptr); } } @@ -5000,6 +5017,12 @@ qla2x00_mem_free(struct qla_hw_data *ha) ha->elsrej.c = NULL; } + if (ha->lsrjt.c) { + dma_free_coherent(&ha->pdev->dev, ha->lsrjt.size, ha->lsrjt.c, + ha->lsrjt.cdma); + ha->lsrjt.c = NULL; + } + ha->init_cb = NULL; ha->init_cb_dma = 0; diff --git a/drivers/scsi/qla2xxx/qla_version.h b/drivers/scsi/qla2xxx/qla_version.h index 81bdf6b03241..d903563e969e 100644 --- a/drivers/scsi/qla2xxx/qla_version.h +++ b/drivers/scsi/qla2xxx/qla_version.h @@ -6,9 +6,9 @@ /* * Driver version */ -#define QLA2XXX_VERSION "10.02.08.500-k" +#define QLA2XXX_VERSION "10.02.09.100-k" #define QLA_DRIVER_MAJOR_VER 10 #define QLA_DRIVER_MINOR_VER 2 -#define QLA_DRIVER_PATCH_VER 8 -#define QLA_DRIVER_BETA_VER 500 +#define QLA_DRIVER_PATCH_VER 9 +#define QLA_DRIVER_BETA_VER 100 diff --git a/drivers/scsi/scsi_debugfs.c b/drivers/scsi/scsi_debugfs.c index 217b70c678c3..f795848b316c 100644 --- a/drivers/scsi/scsi_debugfs.c +++ b/drivers/scsi/scsi_debugfs.c @@ -3,6 +3,7 @@ #include #include #include +#include #include "scsi_debugfs.h" #define SCSI_CMD_FLAG_NAME(name)[const_ilog2(SCMD_##name)] = #name @@ -33,14 +34,33 @@ static int scsi_flags_show(struct seq_file *m, const unsigned long flags, void scsi_show_rq(struct seq_file *m, struct request *rq) { - struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq); + struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq), *cmd2; + struct Scsi_Host *shost = cmd->device->host; int alloc_ms = jiffies_to_msecs(jiffies - cmd->jiffies_at_alloc); int timeout_ms = jiffies_to_msecs(rq->timeout); + const char *list_info = NULL; char buf[80] = "(?)"; + spin_lock_irq(shost->host_lock); + list_for_each_entry(cmd2, &shost->eh_abort_list, eh_entry) { + if (cmd == cmd2) { + list_info = "on eh_abort_list"; + goto unlock; + } + } + list_for_each_entry(cmd2, &shost->eh_cmd_q, eh_entry) { + if (cmd == cmd2) { + list_info = "on eh_cmd_q"; + goto unlock; + } + } +unlock: + spin_unlock_irq(shost->host_lock); + __scsi_format_command(buf, sizeof(buf), cmd->cmnd, cmd->cmd_len); - seq_printf(m, ", .cmd=%s, .retries=%d, .result = %#x, .flags=", buf, - cmd->retries, cmd->result); + seq_printf(m, ", .cmd=%s, .retries=%d, .allowed=%d, .result = %#x, %s%s.flags=", + buf, cmd->retries, cmd->allowed, cmd->result, + list_info ? : "", list_info ? ", " : ""); scsi_flags_show(m, cmd->flags, scsi_cmd_flags, ARRAY_SIZE(scsi_cmd_flags)); seq_printf(m, ", .timeout=%d.%03d, allocated %d.%03d s ago", diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 59176946ab56..c2f647a7c1b0 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -2454,7 +2454,7 @@ static void scsi_evt_emit(struct scsi_device *sdev, struct scsi_event *evt) envp[idx++] = "SDEV_MEDIA_CHANGE=1"; break; case SDEV_EVT_INQUIRY_CHANGE_REPORTED: - scsi_rescan_device(&sdev->sdev_gendev); + scsi_rescan_device(sdev); envp[idx++] = "SDEV_UA=INQUIRY_DATA_HAS_CHANGED"; break; case SDEV_EVT_CAPACITY_CHANGE_REPORTED: diff --git a/drivers/scsi/scsi_priv.h b/drivers/scsi/scsi_priv.h index 118855576ca8..3f0dfb97db6b 100644 --- a/drivers/scsi/scsi_priv.h +++ b/drivers/scsi/scsi_priv.h @@ -137,7 +137,6 @@ extern int scsi_complete_async_scans(void); extern int scsi_scan_host_selected(struct Scsi_Host *, unsigned int, unsigned int, u64, enum scsi_scan_mode); extern void scsi_forget_host(struct Scsi_Host *); -extern void scsi_rescan_device(struct device *); /* scsi_sysctl.c */ #ifdef CONFIG_SYSCTL diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c index aa13feb17c62..52014b2d39e1 100644 --- a/drivers/scsi/scsi_scan.c +++ b/drivers/scsi/scsi_scan.c @@ -1619,9 +1619,9 @@ int scsi_add_device(struct Scsi_Host *host, uint channel, } EXPORT_SYMBOL(scsi_add_device); -void scsi_rescan_device(struct device *dev) +void scsi_rescan_device(struct scsi_device *sdev) { - struct scsi_device *sdev = to_scsi_device(dev); + struct device *dev = &sdev->sdev_gendev; device_lock(dev); diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c index 60317676e45f..24f6eefb6803 100644 --- a/drivers/scsi/scsi_sysfs.c +++ b/drivers/scsi/scsi_sysfs.c @@ -747,7 +747,7 @@ static ssize_t store_rescan_field (struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { - scsi_rescan_device(dev); + scsi_rescan_device(to_scsi_device(dev)); return count; } static DEVICE_ATTR(rescan, S_IWUSR, NULL, store_rescan_field); @@ -840,7 +840,7 @@ store_state_field(struct device *dev, struct device_attribute *attr, * waiting for pending I/O to finish. */ blk_mq_run_hw_queues(sdev->request_queue, true); - scsi_rescan_device(dev); + scsi_rescan_device(sdev); } return ret == 0 ? count : -EINVAL; diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 3c668cfb146d..c92a317ba547 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -104,19 +104,7 @@ static void sd_config_discard(struct scsi_disk *, unsigned int); static void sd_config_write_same(struct scsi_disk *); static int sd_revalidate_disk(struct gendisk *); static void sd_unlock_native_capacity(struct gendisk *disk); -static int sd_probe(struct device *); -static int sd_remove(struct device *); static void sd_shutdown(struct device *); -static int sd_suspend_system(struct device *); -static int sd_suspend_runtime(struct device *); -static int sd_resume_system(struct device *); -static int sd_resume_runtime(struct device *); -static void sd_rescan(struct device *); -static blk_status_t sd_init_command(struct scsi_cmnd *SCpnt); -static void sd_uninit_command(struct scsi_cmnd *SCpnt); -static int sd_done(struct scsi_cmnd *); -static void sd_eh_reset(struct scsi_cmnd *); -static int sd_eh_action(struct scsi_cmnd *, int); static void sd_read_capacity(struct scsi_disk *sdkp, unsigned char *buffer); static void scsi_disk_release(struct device *cdev); @@ -592,33 +580,6 @@ static struct class sd_disk_class = { .dev_groups = sd_disk_groups, }; -static const struct dev_pm_ops sd_pm_ops = { - .suspend = sd_suspend_system, - .resume = sd_resume_system, - .poweroff = sd_suspend_system, - .restore = sd_resume_system, - .runtime_suspend = sd_suspend_runtime, - .runtime_resume = sd_resume_runtime, -}; - -static struct scsi_driver sd_template = { - .gendrv = { - .name = "sd", - .owner = THIS_MODULE, - .probe = sd_probe, - .probe_type = PROBE_PREFER_ASYNCHRONOUS, - .remove = sd_remove, - .shutdown = sd_shutdown, - .pm = &sd_pm_ops, - }, - .rescan = sd_rescan, - .init_command = sd_init_command, - .uninit_command = sd_uninit_command, - .done = sd_done, - .eh_action = sd_eh_action, - .eh_reset = sd_eh_reset, -}; - /* * Don't request a new module, as that could deadlock in multipath * environment. @@ -3929,6 +3890,33 @@ static int sd_resume_runtime(struct device *dev) return sd_resume(dev); } +static const struct dev_pm_ops sd_pm_ops = { + .suspend = sd_suspend_system, + .resume = sd_resume_system, + .poweroff = sd_suspend_system, + .restore = sd_resume_system, + .runtime_suspend = sd_suspend_runtime, + .runtime_resume = sd_resume_runtime, +}; + +static struct scsi_driver sd_template = { + .gendrv = { + .name = "sd", + .owner = THIS_MODULE, + .probe = sd_probe, + .probe_type = PROBE_PREFER_ASYNCHRONOUS, + .remove = sd_remove, + .shutdown = sd_shutdown, + .pm = &sd_pm_ops, + }, + .rescan = sd_rescan, + .init_command = sd_init_command, + .uninit_command = sd_uninit_command, + .done = sd_done, + .eh_action = sd_eh_action, + .eh_reset = sd_eh_reset, +}; + /** * init_sd - entry point for this driver (both when built in or when * a module). diff --git a/drivers/scsi/smartpqi/smartpqi.h b/drivers/scsi/smartpqi/smartpqi.h index e392eaf5b2bf..041940183516 100644 --- a/drivers/scsi/smartpqi/smartpqi.h +++ b/drivers/scsi/smartpqi/smartpqi.h @@ -710,7 +710,7 @@ typedef u32 pqi_index_t; #define SOP_TMF_COMPLETE 0x0 #define SOP_TMF_REJECTED 0x4 #define SOP_TMF_FUNCTION_SUCCEEDED 0x8 -#define SOP_RC_INCORRECT_LOGICAL_UNIT 0x9 +#define SOP_TMF_INCORRECT_LOGICAL_UNIT 0x9 /* additional CDB bytes usage field codes */ #define SOP_ADDITIONAL_CDB_BYTES_0 0 /* 16-byte CDB */ @@ -1085,7 +1085,16 @@ struct pqi_stream_data { u32 last_accessed; }; -#define PQI_MAX_LUNS_PER_DEVICE 256 +#define PQI_MAX_LUNS_PER_DEVICE 256 + +struct pqi_tmf_work { + struct work_struct work_struct; + struct scsi_cmnd *scmd; + struct pqi_ctrl_info *ctrl_info; + struct pqi_scsi_dev *device; + u8 lun; + u8 scsi_opcode; +}; struct pqi_scsi_dev { int devtype; /* as reported by INQUIRY command */ @@ -1111,6 +1120,7 @@ struct pqi_scsi_dev { u8 erase_in_progress : 1; bool aio_enabled; /* only valid for physical disks */ bool in_remove; + bool in_reset[PQI_MAX_LUNS_PER_DEVICE]; bool device_offline; u8 vendor[8]; /* bytes 8-15 of inquiry data */ u8 model[16]; /* bytes 16-31 of inquiry data */ @@ -1149,6 +1159,8 @@ struct pqi_scsi_dev { struct pqi_stream_data stream_data[NUM_STREAMS_PER_LUN]; atomic_t scsi_cmds_outstanding[PQI_MAX_LUNS_PER_DEVICE]; unsigned int raid_bypass_cnt; + + struct pqi_tmf_work tmf_work[PQI_MAX_LUNS_PER_DEVICE]; }; /* VPD inquiry pages */ diff --git a/drivers/scsi/smartpqi/smartpqi_init.c b/drivers/scsi/smartpqi/smartpqi_init.c index 6aaaa7ebca37..9a58df9312fa 100644 --- a/drivers/scsi/smartpqi/smartpqi_init.c +++ b/drivers/scsi/smartpqi/smartpqi_init.c @@ -33,11 +33,11 @@ #define BUILD_TIMESTAMP #endif -#define DRIVER_VERSION "2.1.22-040" +#define DRIVER_VERSION "2.1.24-046" #define DRIVER_MAJOR 2 #define DRIVER_MINOR 1 -#define DRIVER_RELEASE 22 -#define DRIVER_REVISION 40 +#define DRIVER_RELEASE 24 +#define DRIVER_REVISION 46 #define DRIVER_NAME "Microchip SmartPQI Driver (v" \ DRIVER_VERSION BUILD_TIMESTAMP ")" @@ -48,6 +48,8 @@ #define PQI_POST_RESET_DELAY_SECS 5 #define PQI_POST_OFA_RESET_DELAY_UPON_TIMEOUT_SECS 10 +#define PQI_NO_COMPLETION ((void *)-1) + MODULE_AUTHOR("Microchip"); MODULE_DESCRIPTION("Driver for Microchip Smart Family Controller version " DRIVER_VERSION); @@ -96,6 +98,7 @@ static int pqi_ofa_host_memory_update(struct pqi_ctrl_info *ctrl_info); static int pqi_device_wait_for_pending_io(struct pqi_ctrl_info *ctrl_info, struct pqi_scsi_dev *device, u8 lun, unsigned long timeout_msecs); static void pqi_fail_all_outstanding_requests(struct pqi_ctrl_info *ctrl_info); +static void pqi_tmf_worker(struct work_struct *work); /* for flags argument to pqi_submit_raid_request_synchronous() */ #define PQI_SYNC_FLAGS_INTERRUPTABLE 0x1 @@ -455,6 +458,21 @@ static inline bool pqi_device_in_remove(struct pqi_scsi_dev *device) return device->in_remove; } +static inline void pqi_device_reset_start(struct pqi_scsi_dev *device, u8 lun) +{ + device->in_reset[lun] = true; +} + +static inline void pqi_device_reset_done(struct pqi_scsi_dev *device, u8 lun) +{ + device->in_reset[lun] = false; +} + +static inline bool pqi_device_in_reset(struct pqi_scsi_dev *device, u8 lun) +{ + return device->in_reset[lun]; +} + static inline int pqi_event_type_to_event_index(unsigned int event_type) { int index; @@ -2137,6 +2155,15 @@ static inline bool pqi_is_device_added(struct pqi_scsi_dev *device) return device->sdev != NULL; } +static inline void pqi_init_device_tmf_work(struct pqi_scsi_dev *device) +{ + unsigned int lun; + struct pqi_tmf_work *tmf_work; + + for (lun = 0, tmf_work = device->tmf_work; lun < PQI_MAX_LUNS_PER_DEVICE; lun++, tmf_work++) + INIT_WORK(&tmf_work->work_struct, pqi_tmf_worker); +} + static void pqi_update_device_list(struct pqi_ctrl_info *ctrl_info, struct pqi_scsi_dev *new_device_list[], unsigned int num_new_devices) { @@ -2217,6 +2244,7 @@ static void pqi_update_device_list(struct pqi_ctrl_info *ctrl_info, list_add_tail(&device->add_list_entry, &add_list); /* To prevent this device structure from being freed later. */ device->keep_device = true; + pqi_init_device_tmf_work(device); } spin_unlock_irqrestore(&ctrl_info->scsi_device_list_lock, flags); @@ -2257,7 +2285,7 @@ static void pqi_update_device_list(struct pqi_ctrl_info *ctrl_info, device->advertised_queue_depth = device->queue_depth; scsi_change_queue_depth(device->sdev, device->advertised_queue_depth); if (device->rescan) { - scsi_rescan_device(&device->sdev->sdev_gendev); + scsi_rescan_device(device->sdev); device->rescan = false; } } @@ -3330,7 +3358,7 @@ static int pqi_interpret_task_management_response(struct pqi_ctrl_info *ctrl_inf case SOP_TMF_REJECTED: rc = -EAGAIN; break; - case SOP_RC_INCORRECT_LOGICAL_UNIT: + case SOP_TMF_INCORRECT_LOGICAL_UNIT: rc = -ENODEV; break; default: @@ -5628,7 +5656,6 @@ static int pqi_aio_submit_io(struct pqi_ctrl_info *ctrl_info, int rc; struct pqi_io_request *io_request; struct pqi_aio_path_request *request; - struct pqi_scsi_dev *device; io_request = pqi_alloc_io_request(ctrl_info, scmd); if (!io_request) @@ -5648,9 +5675,8 @@ static int pqi_aio_submit_io(struct pqi_ctrl_info *ctrl_info, request->command_priority = io_high_prio; put_unaligned_le16(io_request->index, &request->request_id); request->error_index = request->request_id; - device = scmd->device->hostdata; - if (!pqi_is_logical_device(device) && ctrl_info->multi_lun_device_supported) - put_unaligned_le64(((scmd->device->lun) << 8), &request->lun_number); + if (!raid_bypass && ctrl_info->multi_lun_device_supported) + put_unaligned_le64(scmd->device->lun << 8, &request->lun_number); if (cdb_length > sizeof(request->cdb)) cdb_length = sizeof(request->cdb); request->cdb_length = cdb_length; @@ -5850,6 +5876,7 @@ static inline bool pqi_is_bypass_eligible_request(struct scsi_cmnd *scmd) void pqi_prep_for_scsi_done(struct scsi_cmnd *scmd) { struct pqi_scsi_dev *device; + struct completion *wait; if (!scmd->device) { set_host_byte(scmd, DID_NO_CONNECT); @@ -5863,6 +5890,10 @@ void pqi_prep_for_scsi_done(struct scsi_cmnd *scmd) } atomic_dec(&device->scsi_cmds_outstanding[scmd->device->lun]); + + wait = (struct completion *)xchg(&scmd->host_scribble, NULL); + if (wait != PQI_NO_COMPLETION) + complete(wait); } static bool pqi_is_parity_write_stream(struct pqi_ctrl_info *ctrl_info, @@ -5948,6 +5979,9 @@ static int pqi_scsi_queue_command(struct Scsi_Host *shost, struct scsi_cmnd *scm u16 hw_queue; struct pqi_queue_group *queue_group; bool raid_bypassed; + u8 lun; + + scmd->host_scribble = PQI_NO_COMPLETION; device = scmd->device->hostdata; @@ -5957,7 +5991,9 @@ static int pqi_scsi_queue_command(struct Scsi_Host *shost, struct scsi_cmnd *scm return 0; } - atomic_inc(&device->scsi_cmds_outstanding[scmd->device->lun]); + lun = (u8)scmd->device->lun; + + atomic_inc(&device->scsi_cmds_outstanding[lun]); ctrl_info = shost_to_hba(shost); @@ -5967,7 +6003,7 @@ static int pqi_scsi_queue_command(struct Scsi_Host *shost, struct scsi_cmnd *scm return 0; } - if (pqi_ctrl_blocked(ctrl_info)) { + if (pqi_ctrl_blocked(ctrl_info) || pqi_device_in_reset(device, lun)) { rc = SCSI_MLQUEUE_HOST_BUSY; goto out; } @@ -6002,8 +6038,10 @@ static int pqi_scsi_queue_command(struct Scsi_Host *shost, struct scsi_cmnd *scm } out: - if (rc) - atomic_dec(&device->scsi_cmds_outstanding[scmd->device->lun]); + if (rc) { + scmd->host_scribble = NULL; + atomic_dec(&device->scsi_cmds_outstanding[lun]); + } return rc; } @@ -6097,7 +6135,7 @@ static int pqi_wait_until_inbound_queues_empty(struct pqi_ctrl_info *ctrl_info) } static void pqi_fail_io_queued_for_device(struct pqi_ctrl_info *ctrl_info, - struct pqi_scsi_dev *device) + struct pqi_scsi_dev *device, u8 lun) { unsigned int i; unsigned int path; @@ -6127,6 +6165,9 @@ static void pqi_fail_io_queued_for_device(struct pqi_ctrl_info *ctrl_info, if (scsi_device != device) continue; + if ((u8)scmd->device->lun != lun) + continue; + list_del(&io_request->request_list_entry); set_host_byte(scmd, DID_RESET); pqi_free_io_request(io_request); @@ -6224,15 +6265,13 @@ static int pqi_wait_for_lun_reset_completion(struct pqi_ctrl_info *ctrl_info, #define PQI_LUN_RESET_FIRMWARE_TIMEOUT_SECS 30 -static int pqi_lun_reset(struct pqi_ctrl_info *ctrl_info, struct scsi_cmnd *scmd) +static int pqi_lun_reset(struct pqi_ctrl_info *ctrl_info, struct pqi_scsi_dev *device, u8 lun) { int rc; struct pqi_io_request *io_request; DECLARE_COMPLETION_ONSTACK(wait); struct pqi_task_management_request *request; - struct pqi_scsi_dev *device; - device = scmd->device->hostdata; io_request = pqi_alloc_io_request(ctrl_info, NULL); io_request->io_complete_callback = pqi_lun_reset_complete; io_request->context = &wait; @@ -6247,7 +6286,7 @@ static int pqi_lun_reset(struct pqi_ctrl_info *ctrl_info, struct scsi_cmnd *scmd memcpy(request->lun_number, device->scsi3addr, sizeof(request->lun_number)); if (!pqi_is_logical_device(device) && ctrl_info->multi_lun_device_supported) - request->ml_device_lun_number = (u8)scmd->device->lun; + request->ml_device_lun_number = lun; request->task_management_function = SOP_TASK_MANAGEMENT_LUN_RESET; if (ctrl_info->tmf_iu_timeout_supported) put_unaligned_le16(PQI_LUN_RESET_FIRMWARE_TIMEOUT_SECS, &request->timeout); @@ -6255,7 +6294,7 @@ static int pqi_lun_reset(struct pqi_ctrl_info *ctrl_info, struct scsi_cmnd *scmd pqi_start_io(ctrl_info, &ctrl_info->queue_groups[PQI_DEFAULT_QUEUE_GROUP], RAID_PATH, io_request); - rc = pqi_wait_for_lun_reset_completion(ctrl_info, device, (u8)scmd->device->lun, &wait); + rc = pqi_wait_for_lun_reset_completion(ctrl_info, device, lun, &wait); if (rc == 0) rc = io_request->status; @@ -6269,18 +6308,16 @@ static int pqi_lun_reset(struct pqi_ctrl_info *ctrl_info, struct scsi_cmnd *scmd #define PQI_LUN_RESET_PENDING_IO_TIMEOUT_MSECS (10 * 60 * 1000) #define PQI_LUN_RESET_FAILED_PENDING_IO_TIMEOUT_MSECS (2 * 60 * 1000) -static int pqi_lun_reset_with_retries(struct pqi_ctrl_info *ctrl_info, struct scsi_cmnd *scmd) +static int pqi_lun_reset_with_retries(struct pqi_ctrl_info *ctrl_info, struct pqi_scsi_dev *device, u8 lun) { int reset_rc; int wait_rc; unsigned int retries; unsigned long timeout_msecs; - struct pqi_scsi_dev *device; - device = scmd->device->hostdata; for (retries = 0;;) { - reset_rc = pqi_lun_reset(ctrl_info, scmd); - if (reset_rc == 0 || reset_rc == -ENODEV || ++retries > PQI_LUN_RESET_RETRIES) + reset_rc = pqi_lun_reset(ctrl_info, device, lun); + if (reset_rc == 0 || reset_rc == -ENODEV || reset_rc == -ENXIO || ++retries > PQI_LUN_RESET_RETRIES) break; msleep(PQI_LUN_RESET_RETRY_INTERVAL_MSECS); } @@ -6288,65 +6325,127 @@ static int pqi_lun_reset_with_retries(struct pqi_ctrl_info *ctrl_info, struct sc timeout_msecs = reset_rc ? PQI_LUN_RESET_FAILED_PENDING_IO_TIMEOUT_MSECS : PQI_LUN_RESET_PENDING_IO_TIMEOUT_MSECS; - wait_rc = pqi_device_wait_for_pending_io(ctrl_info, device, scmd->device->lun, timeout_msecs); + wait_rc = pqi_device_wait_for_pending_io(ctrl_info, device, lun, timeout_msecs); if (wait_rc && reset_rc == 0) reset_rc = wait_rc; return reset_rc == 0 ? SUCCESS : FAILED; } -static int pqi_device_reset(struct pqi_ctrl_info *ctrl_info, struct scsi_cmnd *scmd) +static int pqi_device_reset(struct pqi_ctrl_info *ctrl_info, struct pqi_scsi_dev *device, u8 lun) { int rc; - struct pqi_scsi_dev *device; - device = scmd->device->hostdata; pqi_ctrl_block_requests(ctrl_info); pqi_ctrl_wait_until_quiesced(ctrl_info); - pqi_fail_io_queued_for_device(ctrl_info, device); + pqi_fail_io_queued_for_device(ctrl_info, device, lun); rc = pqi_wait_until_inbound_queues_empty(ctrl_info); + pqi_device_reset_start(device, lun); + pqi_ctrl_unblock_requests(ctrl_info); if (rc) rc = FAILED; else - rc = pqi_lun_reset_with_retries(ctrl_info, scmd); - pqi_ctrl_unblock_requests(ctrl_info); + rc = pqi_lun_reset_with_retries(ctrl_info, device, lun); + pqi_device_reset_done(device, lun); + + return rc; +} + +static int pqi_device_reset_handler(struct pqi_ctrl_info *ctrl_info, struct pqi_scsi_dev *device, u8 lun, struct scsi_cmnd *scmd, u8 scsi_opcode) +{ + int rc; + + mutex_lock(&ctrl_info->lun_reset_mutex); + + dev_err(&ctrl_info->pci_dev->dev, + "resetting scsi %d:%d:%d:%u SCSI cmd at %p due to cmd opcode 0x%02x\n", + ctrl_info->scsi_host->host_no, device->bus, device->target, lun, scmd, scsi_opcode); + + pqi_check_ctrl_health(ctrl_info); + if (pqi_ctrl_offline(ctrl_info)) + rc = FAILED; + else + rc = pqi_device_reset(ctrl_info, device, lun); + + dev_err(&ctrl_info->pci_dev->dev, + "reset of scsi %d:%d:%d:%u: %s\n", + ctrl_info->scsi_host->host_no, device->bus, device->target, lun, + rc == SUCCESS ? "SUCCESS" : "FAILED"); + + mutex_unlock(&ctrl_info->lun_reset_mutex); return rc; } static int pqi_eh_device_reset_handler(struct scsi_cmnd *scmd) { - int rc; struct Scsi_Host *shost; struct pqi_ctrl_info *ctrl_info; struct pqi_scsi_dev *device; + u8 scsi_opcode; + + shost = scmd->device->host; + ctrl_info = shost_to_hba(shost); + device = scmd->device->hostdata; + scsi_opcode = scmd->cmd_len > 0 ? scmd->cmnd[0] : 0xff; + + return pqi_device_reset_handler(ctrl_info, device, (u8)scmd->device->lun, scmd, scsi_opcode); +} + +static void pqi_tmf_worker(struct work_struct *work) +{ + struct pqi_tmf_work *tmf_work; + struct scsi_cmnd *scmd; + + tmf_work = container_of(work, struct pqi_tmf_work, work_struct); + scmd = (struct scsi_cmnd *)xchg(&tmf_work->scmd, NULL); + + pqi_device_reset_handler(tmf_work->ctrl_info, tmf_work->device, tmf_work->lun, scmd, tmf_work->scsi_opcode); +} + +static int pqi_eh_abort_handler(struct scsi_cmnd *scmd) +{ + struct Scsi_Host *shost; + struct pqi_ctrl_info *ctrl_info; + struct pqi_scsi_dev *device; + struct pqi_tmf_work *tmf_work; + DECLARE_COMPLETION_ONSTACK(wait); shost = scmd->device->host; ctrl_info = shost_to_hba(shost); device = scmd->device->hostdata; - mutex_lock(&ctrl_info->lun_reset_mutex); + dev_err(&ctrl_info->pci_dev->dev, + "attempting TASK ABORT on scsi %d:%d:%d:%d for SCSI cmd at %p\n", + shost->host_no, device->bus, device->target, (int)scmd->device->lun, scmd); + + if (cmpxchg(&scmd->host_scribble, PQI_NO_COMPLETION, (void *)&wait) == NULL) { + dev_err(&ctrl_info->pci_dev->dev, + "scsi %d:%d:%d:%d for SCSI cmd at %p already completed\n", + shost->host_no, device->bus, device->target, (int)scmd->device->lun, scmd); + scmd->result = DID_RESET << 16; + goto out; + } + + tmf_work = &device->tmf_work[scmd->device->lun]; + + if (cmpxchg(&tmf_work->scmd, NULL, scmd) == NULL) { + tmf_work->ctrl_info = ctrl_info; + tmf_work->device = device; + tmf_work->lun = (u8)scmd->device->lun; + tmf_work->scsi_opcode = scmd->cmd_len > 0 ? scmd->cmnd[0] : 0xff; + schedule_work(&tmf_work->work_struct); + } + + wait_for_completion(&wait); dev_err(&ctrl_info->pci_dev->dev, - "resetting scsi %d:%d:%d:%d due to cmd 0x%02x\n", - shost->host_no, - device->bus, device->target, (u32)scmd->device->lun, - scmd->cmd_len > 0 ? scmd->cmnd[0] : 0xff); + "TASK ABORT on scsi %d:%d:%d:%d for SCSI cmd at %p: SUCCESS\n", + shost->host_no, device->bus, device->target, (int)scmd->device->lun, scmd); - pqi_check_ctrl_health(ctrl_info); - if (pqi_ctrl_offline(ctrl_info)) - rc = FAILED; - else - rc = pqi_device_reset(ctrl_info, scmd); +out: - dev_err(&ctrl_info->pci_dev->dev, - "reset of scsi %d:%d:%d:%d: %s\n", - shost->host_no, device->bus, device->target, (u32)scmd->device->lun, - rc == SUCCESS ? "SUCCESS" : "FAILED"); - - mutex_unlock(&ctrl_info->lun_reset_mutex); - - return rc; + return SUCCESS; } static int pqi_slave_alloc(struct scsi_device *sdev) @@ -6470,21 +6569,21 @@ static int pqi_getpciinfo_ioctl(struct pqi_ctrl_info *ctrl_info, void __user *ar struct pci_dev *pci_dev; u32 subsystem_vendor; u32 subsystem_device; - cciss_pci_info_struct pciinfo; + cciss_pci_info_struct pci_info; if (!arg) return -EINVAL; pci_dev = ctrl_info->pci_dev; - pciinfo.domain = pci_domain_nr(pci_dev->bus); - pciinfo.bus = pci_dev->bus->number; - pciinfo.dev_fn = pci_dev->devfn; + pci_info.domain = pci_domain_nr(pci_dev->bus); + pci_info.bus = pci_dev->bus->number; + pci_info.dev_fn = pci_dev->devfn; subsystem_vendor = pci_dev->subsystem_vendor; subsystem_device = pci_dev->subsystem_device; - pciinfo.board_id = ((subsystem_device << 16) & 0xffff0000) | subsystem_vendor; + pci_info.board_id = ((subsystem_device << 16) & 0xffff0000) | subsystem_vendor; - if (copy_to_user(arg, &pciinfo, sizeof(pciinfo))) + if (copy_to_user(arg, &pci_info, sizeof(pci_info))) return -EFAULT; return 0; @@ -7362,6 +7461,7 @@ static const struct scsi_host_template pqi_driver_template = { .scan_finished = pqi_scan_finished, .this_id = -1, .eh_device_reset_handler = pqi_eh_device_reset_handler, + .eh_abort_handler = pqi_eh_abort_handler, .ioctl = pqi_ioctl, .slave_alloc = pqi_slave_alloc, .slave_configure = pqi_slave_configure, @@ -8904,6 +9004,52 @@ static void pqi_ctrl_offline_worker(struct work_struct *work) pqi_take_ctrl_offline_deferred(ctrl_info); } +static char *pqi_ctrl_shutdown_reason_to_string(enum pqi_ctrl_shutdown_reason ctrl_shutdown_reason) +{ + char *string; + + switch (ctrl_shutdown_reason) { + case PQI_IQ_NOT_DRAINED_TIMEOUT: + string = "inbound queue not drained timeout"; + break; + case PQI_LUN_RESET_TIMEOUT: + string = "LUN reset timeout"; + break; + case PQI_IO_PENDING_POST_LUN_RESET_TIMEOUT: + string = "I/O pending timeout after LUN reset"; + break; + case PQI_NO_HEARTBEAT: + string = "no controller heartbeat detected"; + break; + case PQI_FIRMWARE_KERNEL_NOT_UP: + string = "firmware kernel not ready"; + break; + case PQI_OFA_RESPONSE_TIMEOUT: + string = "OFA response timeout"; + break; + case PQI_INVALID_REQ_ID: + string = "invalid request ID"; + break; + case PQI_UNMATCHED_REQ_ID: + string = "unmatched request ID"; + break; + case PQI_IO_PI_OUT_OF_RANGE: + string = "I/O queue producer index out of range"; + break; + case PQI_EVENT_PI_OUT_OF_RANGE: + string = "event queue producer index out of range"; + break; + case PQI_UNEXPECTED_IU_TYPE: + string = "unexpected IU type"; + break; + default: + string = "unknown reason"; + break; + } + + return string; +} + static void pqi_take_ctrl_offline(struct pqi_ctrl_info *ctrl_info, enum pqi_ctrl_shutdown_reason ctrl_shutdown_reason) { @@ -8916,7 +9062,9 @@ static void pqi_take_ctrl_offline(struct pqi_ctrl_info *ctrl_info, if (!pqi_disable_ctrl_shutdown) sis_shutdown_ctrl(ctrl_info, ctrl_shutdown_reason); pci_disable_device(ctrl_info->pci_dev); - dev_err(&ctrl_info->pci_dev->dev, "controller offline\n"); + dev_err(&ctrl_info->pci_dev->dev, + "controller offline: reason code 0x%x (%s)\n", + ctrl_shutdown_reason, pqi_ctrl_shutdown_reason_to_string(ctrl_shutdown_reason)); schedule_work(&ctrl_info->ctrl_offline_work); } @@ -9062,7 +9210,7 @@ static void pqi_shutdown(struct pci_dev *pci_dev) rc = pqi_flush_cache(ctrl_info, shutdown_event); if (rc) dev_err(&pci_dev->dev, - "unable to flush controller cache\n"); + "unable to flush controller cache during shutdown\n"); pqi_crash_if_pending_command(ctrl_info); pqi_reset(ctrl_info); diff --git a/drivers/scsi/st.c b/drivers/scsi/st.c index 14d7981ddcdd..338aa8c42968 100644 --- a/drivers/scsi/st.c +++ b/drivers/scsi/st.c @@ -414,6 +414,8 @@ static int st_chk_result(struct scsi_tape *STp, struct st_request * SRpnt) if (cmdstatp->have_sense && cmdstatp->sense_hdr.asc == 0 && cmdstatp->sense_hdr.ascq == 0x17) STp->cleaning_req = 1; /* ASC and ASCQ => cleaning requested */ + if (cmdstatp->have_sense && scode == UNIT_ATTENTION && cmdstatp->sense_hdr.asc == 0x29) + STp->pos_unknown = 1; /* ASC => power on / reset */ STp->pos_unknown |= STp->device->was_reset; diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c index 068625556dda..a95936b18f69 100644 --- a/drivers/scsi/storvsc_drv.c +++ b/drivers/scsi/storvsc_drv.c @@ -475,7 +475,7 @@ static void storvsc_device_scan(struct work_struct *work) sdev = scsi_device_lookup(wrk->host, 0, wrk->tgt_id, wrk->lun); if (!sdev) goto done; - scsi_rescan_device(&sdev->sdev_gendev); + scsi_rescan_device(sdev); scsi_device_put(sdev); done: diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c index bd5633667d01..9d1bdcdc1331 100644 --- a/drivers/scsi/virtio_scsi.c +++ b/drivers/scsi/virtio_scsi.c @@ -325,7 +325,7 @@ static void virtscsi_handle_param_change(struct virtio_scsi *vscsi, /* Handle "Parameters changed", "Mode parameters changed", and "Capacity data has changed". */ if (asc == 0x2a && (ascq == 0x00 || ascq == 0x01 || ascq == 0x09)) - scsi_rescan_device(&sdev->sdev_gendev); + scsi_rescan_device(sdev); scsi_device_put(sdev); } diff --git a/drivers/scsi/xen-scsifront.c b/drivers/scsi/xen-scsifront.c index caae61aa2afe..9ec55ddc1204 100644 --- a/drivers/scsi/xen-scsifront.c +++ b/drivers/scsi/xen-scsifront.c @@ -743,7 +743,7 @@ static int scsifront_sdev_configure(struct scsi_device *sdev) if (info->host_active == STATE_ERROR) return -EIO; - if (info && current == info->curr) { + if (current == info->curr) { err = xenbus_printf(XBT_NIL, info->dev->nodename, info->dev_state_path, "%d", XenbusStateConnected); if (err) { @@ -761,7 +761,7 @@ static void scsifront_sdev_destroy(struct scsi_device *sdev) struct vscsifrnt_info *info = shost_priv(sdev->host); int err; - if (info && current == info->curr) { + if (current == info->curr) { err = xenbus_printf(XBT_NIL, info->dev->nodename, info->dev_state_path, "%d", XenbusStateClosed); if (err) @@ -903,7 +903,7 @@ static int scsifront_probe(struct xenbus_device *dev, xenbus_dev_fatal(dev, err, "fail to allocate scsi host"); return err; } - info = (struct vscsifrnt_info *)host->hostdata; + info = shost_priv(host); dev_set_drvdata(&dev->dev, info); info->dev = dev; diff --git a/drivers/soc/renesas/Kconfig b/drivers/soc/renesas/Kconfig index de31589ed054..5a75ab64d1ed 100644 --- a/drivers/soc/renesas/Kconfig +++ b/drivers/soc/renesas/Kconfig @@ -334,6 +334,11 @@ if RISCV config ARCH_R9A07G043 bool "RISC-V Platform support for RZ/Five" select ARCH_RZG2L + select AX45MP_L2_CACHE if RISCV_DMA_NONCOHERENT + select DMA_GLOBAL_POOL + select ERRATA_ANDES if RISCV_SBI + select ERRATA_ANDES_CMO if ERRATA_ANDES + help This enables support for the Renesas RZ/Five SoC. diff --git a/drivers/spi/spi-sun6i.c b/drivers/spi/spi-sun6i.c index 3f5b1556ece0..fddc63309773 100644 --- a/drivers/spi/spi-sun6i.c +++ b/drivers/spi/spi-sun6i.c @@ -106,6 +106,7 @@ struct sun6i_spi { struct reset_control *rstc; struct completion done; + struct completion dma_rx_done; const u8 *tx_buf; u8 *rx_buf; @@ -200,6 +201,13 @@ static size_t sun6i_spi_max_transfer_size(struct spi_device *spi) return SUN6I_MAX_XFER_SIZE - 1; } +static void sun6i_spi_dma_rx_cb(void *param) +{ + struct sun6i_spi *sspi = param; + + complete(&sspi->dma_rx_done); +} + static int sun6i_spi_prepare_dma(struct sun6i_spi *sspi, struct spi_transfer *tfr) { @@ -211,7 +219,7 @@ static int sun6i_spi_prepare_dma(struct sun6i_spi *sspi, struct dma_slave_config rxconf = { .direction = DMA_DEV_TO_MEM, .src_addr = sspi->dma_addr_rx, - .src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES, + .src_addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE, .src_maxburst = 8, }; @@ -224,6 +232,8 @@ static int sun6i_spi_prepare_dma(struct sun6i_spi *sspi, DMA_PREP_INTERRUPT); if (!rxdesc) return -EINVAL; + rxdesc->callback_param = sspi; + rxdesc->callback = sun6i_spi_dma_rx_cb; } txdesc = NULL; @@ -279,6 +289,7 @@ static int sun6i_spi_transfer_one(struct spi_master *master, return -EINVAL; reinit_completion(&sspi->done); + reinit_completion(&sspi->dma_rx_done); sspi->tx_buf = tfr->tx_buf; sspi->rx_buf = tfr->rx_buf; sspi->len = tfr->len; @@ -479,6 +490,22 @@ static int sun6i_spi_transfer_one(struct spi_master *master, start = jiffies; timeout = wait_for_completion_timeout(&sspi->done, msecs_to_jiffies(tx_time)); + + if (!use_dma) { + sun6i_spi_drain_fifo(sspi); + } else { + if (timeout && rx_len) { + /* + * Even though RX on the peripheral side has finished + * RX DMA might still be in flight + */ + timeout = wait_for_completion_timeout(&sspi->dma_rx_done, + timeout); + if (!timeout) + dev_warn(&master->dev, "RX DMA timeout\n"); + } + } + end = jiffies; if (!timeout) { dev_warn(&master->dev, @@ -506,7 +533,6 @@ static irqreturn_t sun6i_spi_handler(int irq, void *dev_id) /* Transfer complete */ if (status & SUN6I_INT_CTL_TC) { sun6i_spi_write(sspi, SUN6I_INT_STA_REG, SUN6I_INT_CTL_TC); - sun6i_spi_drain_fifo(sspi); complete(&sspi->done); return IRQ_HANDLED; } @@ -665,6 +691,7 @@ static int sun6i_spi_probe(struct platform_device *pdev) } init_completion(&sspi->done); + init_completion(&sspi->dma_rx_done); sspi->rstc = devm_reset_control_get_exclusive(&pdev->dev, NULL); if (IS_ERR(sspi->rstc)) { diff --git a/drivers/staging/greybus/pwm.c b/drivers/staging/greybus/pwm.c index f569d371a007..57cc1960d059 100644 --- a/drivers/staging/greybus/pwm.c +++ b/drivers/staging/greybus/pwm.c @@ -266,7 +266,7 @@ static int gb_pwm_probe(struct gbphy_device *gbphy_dev, { struct gb_connection *connection; struct gb_pwm_chip *pwmc; - struct pwm_chip *pwm; + struct pwm_chip *chip; int ret; pwmc = kzalloc(sizeof(*pwmc), GFP_KERNEL); @@ -294,13 +294,13 @@ static int gb_pwm_probe(struct gbphy_device *gbphy_dev, if (ret) goto exit_connection_disable; - pwm = &pwmc->chip; + chip = &pwmc->chip; - pwm->dev = &gbphy_dev->dev; - pwm->ops = &gb_pwm_ops; - pwm->npwm = pwmc->pwm_max + 1; + chip->dev = &gbphy_dev->dev; + chip->ops = &gb_pwm_ops; + chip->npwm = pwmc->pwm_max + 1; - ret = pwmchip_add(pwm); + ret = pwmchip_add(chip); if (ret) { dev_err(&gbphy_dev->dev, "failed to register PWM: %d\n", ret); diff --git a/drivers/staging/media/av7110/sp8870.c b/drivers/staging/media/av7110/sp8870.c index 9767159aeb9b..abf5c72607b6 100644 --- a/drivers/staging/media/av7110/sp8870.c +++ b/drivers/staging/media/av7110/sp8870.c @@ -606,4 +606,4 @@ MODULE_DESCRIPTION("Spase SP8870 DVB-T Demodulator driver"); MODULE_AUTHOR("Juergen Peitz"); MODULE_LICENSE("GPL"); -EXPORT_SYMBOL(sp8870_attach); +EXPORT_SYMBOL_GPL(sp8870_attach); diff --git a/drivers/thermal/armada_thermal.c b/drivers/thermal/armada_thermal.c index 9f6dc4fc9112..f00765bfc22e 100644 --- a/drivers/thermal/armada_thermal.c +++ b/drivers/thermal/armada_thermal.c @@ -876,8 +876,9 @@ static int armada_thermal_probe(struct platform_device *pdev) /* Wait the sensors to be valid */ armada_wait_sensor_validity(priv); - tz = thermal_zone_device_register(priv->zone_name, 0, 0, priv, - &legacy_ops, NULL, 0, 0); + tz = thermal_tripless_zone_device_register(priv->zone_name, + priv, &legacy_ops, + NULL); if (IS_ERR(tz)) { dev_err(&pdev->dev, "Failed to register thermal zone device\n"); diff --git a/drivers/thermal/dove_thermal.c b/drivers/thermal/dove_thermal.c index 9954040d1d2c..7a18cb960bee 100644 --- a/drivers/thermal/dove_thermal.c +++ b/drivers/thermal/dove_thermal.c @@ -139,8 +139,8 @@ static int dove_thermal_probe(struct platform_device *pdev) return ret; } - thermal = thermal_zone_device_register("dove_thermal", 0, 0, - priv, &ops, NULL, 0, 0); + thermal = thermal_tripless_zone_device_register("dove_thermal", priv, + &ops, NULL); if (IS_ERR(thermal)) { dev_err(&pdev->dev, "Failed to register thermal zone device\n"); diff --git a/drivers/thermal/intel/int340x_thermal/int3400_thermal.c b/drivers/thermal/intel/int340x_thermal/int3400_thermal.c index ddd600820f68..ffc2871a021c 100644 --- a/drivers/thermal/intel/int340x_thermal/int3400_thermal.c +++ b/drivers/thermal/intel/int340x_thermal/int3400_thermal.c @@ -609,9 +609,9 @@ static int int3400_thermal_probe(struct platform_device *pdev) evaluate_odvp(priv); - priv->thermal = thermal_zone_device_register("INT3400 Thermal", 0, 0, - priv, &int3400_thermal_ops, - &int3400_thermal_params, 0, 0); + priv->thermal = thermal_tripless_zone_device_register("INT3400 Thermal", priv, + &int3400_thermal_ops, + &int3400_thermal_params); if (IS_ERR(priv->thermal)) { result = PTR_ERR(priv->thermal); goto free_art_trt; diff --git a/drivers/thermal/kirkwood_thermal.c b/drivers/thermal/kirkwood_thermal.c index 668747bd86ef..acb10d24256d 100644 --- a/drivers/thermal/kirkwood_thermal.c +++ b/drivers/thermal/kirkwood_thermal.c @@ -71,8 +71,8 @@ static int kirkwood_thermal_probe(struct platform_device *pdev) if (IS_ERR(priv->sensor)) return PTR_ERR(priv->sensor); - thermal = thermal_zone_device_register("kirkwood_thermal", 0, 0, - priv, &ops, NULL, 0, 0); + thermal = thermal_tripless_zone_device_register("kirkwood_thermal", + priv, &ops, NULL); if (IS_ERR(thermal)) { dev_err(&pdev->dev, "Failed to register thermal zone device\n"); diff --git a/drivers/thermal/spear_thermal.c b/drivers/thermal/spear_thermal.c index 6e78616a576e..96d99289799a 100644 --- a/drivers/thermal/spear_thermal.c +++ b/drivers/thermal/spear_thermal.c @@ -122,8 +122,8 @@ static int spear_thermal_probe(struct platform_device *pdev) stdev->flags = val; writel_relaxed(stdev->flags, stdev->thermal_base); - spear_thermal = thermal_zone_device_register("spear_thermal", 0, 0, - stdev, &ops, NULL, 0, 0); + spear_thermal = thermal_tripless_zone_device_register("spear_thermal", + stdev, &ops, NULL); if (IS_ERR(spear_thermal)) { dev_err(&pdev->dev, "thermal zone device is NULL\n"); ret = PTR_ERR(spear_thermal); diff --git a/drivers/thermal/thermal_core.c b/drivers/thermal/thermal_core.c index 0bdde1ab5d8b..8717a3343512 100644 --- a/drivers/thermal/thermal_core.c +++ b/drivers/thermal/thermal_core.c @@ -1389,16 +1389,16 @@ free_tz: } EXPORT_SYMBOL_GPL(thermal_zone_device_register_with_trips); -struct thermal_zone_device *thermal_zone_device_register(const char *type, int ntrips, int mask, - void *devdata, struct thermal_zone_device_ops *ops, - const struct thermal_zone_params *tzp, int passive_delay, - int polling_delay) +struct thermal_zone_device *thermal_tripless_zone_device_register( + const char *type, + void *devdata, + struct thermal_zone_device_ops *ops, + const struct thermal_zone_params *tzp) { - return thermal_zone_device_register_with_trips(type, NULL, ntrips, mask, - devdata, ops, tzp, - passive_delay, polling_delay); + return thermal_zone_device_register_with_trips(type, NULL, 0, 0, devdata, + ops, tzp, 0, 0); } -EXPORT_SYMBOL_GPL(thermal_zone_device_register); +EXPORT_SYMBOL_GPL(thermal_tripless_zone_device_register); void *thermal_zone_device_priv(struct thermal_zone_device *tzd) { diff --git a/drivers/ufs/core/ufs_bsg.c b/drivers/ufs/core/ufs_bsg.c index 34e423924e06..374e5aae4e7e 100644 --- a/drivers/ufs/core/ufs_bsg.c +++ b/drivers/ufs/core/ufs_bsg.c @@ -76,8 +76,7 @@ static int ufs_bsg_exec_advanced_rpmb_req(struct ufs_hba *hba, struct bsg_job *j int ret; int data_len; - if (hba->ufs_version < ufshci_version(4, 0) || !hba->dev_info.b_advanced_rpmb_en || - !(hba->capabilities & MASK_EHSLUTRD_SUPPORTED)) + if (hba->ufs_version < ufshci_version(4, 0) || !hba->dev_info.b_advanced_rpmb_en) return -EINVAL; if (rpmb_request->ehs_req.length != 2 || rpmb_request->ehs_req.ehs_type != 1) diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c index 96d18c58d06b..80525d148a23 100644 --- a/drivers/ufs/core/ufshcd.c +++ b/drivers/ufs/core/ufshcd.c @@ -7271,11 +7271,17 @@ int ufshcd_advanced_rpmb_req_handler(struct ufs_hba *hba, struct utp_upiu_req *r /* Advanced RPMB starts from UFS 4.0, so its command type is UTP_CMD_TYPE_UFS_STORAGE */ lrbp->command_type = UTP_CMD_TYPE_UFS_STORAGE; - ufshcd_prepare_req_desc_hdr(lrbp, &upiu_flags, dir, 2); + /* + * According to UFSHCI 4.0 specification page 24, if EHSLUTRDS is 0, host controller takes + * EHS length from CMD UPIU, and SW driver use EHS Length field in CMD UPIU. if it is 1, + * HW controller takes EHS length from UTRD. + */ + if (hba->capabilities & MASK_EHSLUTRD_SUPPORTED) + ufshcd_prepare_req_desc_hdr(lrbp, &upiu_flags, dir, 2); + else + ufshcd_prepare_req_desc_hdr(lrbp, &upiu_flags, dir, 0); - /* update the task tag and LUN in the request upiu */ - req_upiu->header.flags = upiu_flags; - req_upiu->header.lun = UFS_UPIU_RPMB_WLUN; + /* update the task tag */ req_upiu->header.task_tag = tag; /* copy the UPIU(contains CDB) request as it is */ diff --git a/drivers/usb/dwc3/dwc3-octeon.c b/drivers/usb/dwc3/dwc3-octeon.c index ff01f2c17452..6010135e1acc 100644 --- a/drivers/usb/dwc3/dwc3-octeon.c +++ b/drivers/usb/dwc3/dwc3-octeon.c @@ -13,7 +13,9 @@ #include #include #include +#include #include +#include /* * USB Control Register diff --git a/drivers/video/backlight/gpio_backlight.c b/drivers/video/backlight/gpio_backlight.c index d3bea42407f1..d28c30b2a35d 100644 --- a/drivers/video/backlight/gpio_backlight.c +++ b/drivers/video/backlight/gpio_backlight.c @@ -87,8 +87,7 @@ static int gpio_backlight_probe(struct platform_device *pdev) /* Not booted with device tree or no phandle link to the node */ bl->props.power = def_value ? FB_BLANK_UNBLANK : FB_BLANK_POWERDOWN; - else if (gpiod_get_direction(gbl->gpiod) == 0 && - gpiod_get_value_cansleep(gbl->gpiod) == 0) + else if (gpiod_get_value_cansleep(gbl->gpiod) == 0) bl->props.power = FB_BLANK_POWERDOWN; else bl->props.power = FB_BLANK_UNBLANK; diff --git a/drivers/video/backlight/led_bl.c b/drivers/video/backlight/led_bl.c index 3259292fda76..032f8bddf872 100644 --- a/drivers/video/backlight/led_bl.c +++ b/drivers/video/backlight/led_bl.c @@ -243,7 +243,7 @@ MODULE_DEVICE_TABLE(of, led_bl_of_match); static struct platform_driver led_bl_driver = { .driver = { .name = "led-backlight", - .of_match_table = of_match_ptr(led_bl_of_match), + .of_match_table = led_bl_of_match, }, .probe = led_bl_probe, .remove_new = led_bl_remove, diff --git a/drivers/video/backlight/lp855x_bl.c b/drivers/video/backlight/lp855x_bl.c index 1c9e921bca14..da1f124db69c 100644 --- a/drivers/video/backlight/lp855x_bl.c +++ b/drivers/video/backlight/lp855x_bl.c @@ -71,6 +71,7 @@ struct lp855x { struct device *dev; struct lp855x_platform_data *pdata; struct pwm_device *pwm; + bool needs_pwm_init; struct regulator *supply; /* regulator for VDD input */ struct regulator *enable; /* regulator for EN/VDDIO input */ }; @@ -216,16 +217,24 @@ err: return ret; } -static void lp855x_pwm_ctrl(struct lp855x *lp, int br, int max_br) +static int lp855x_pwm_ctrl(struct lp855x *lp, int br, int max_br) { struct pwm_state state; - pwm_get_state(lp->pwm, &state); + if (lp->needs_pwm_init) { + pwm_init_state(lp->pwm, &state); + /* Legacy platform data compatibility */ + if (lp->pdata->period_ns > 0) + state.period = lp->pdata->period_ns; + lp->needs_pwm_init = false; + } else { + pwm_get_state(lp->pwm, &state); + } state.duty_cycle = div_u64(br * state.period, max_br); state.enabled = state.duty_cycle; - pwm_apply_state(lp->pwm, &state); + return pwm_apply_state(lp->pwm, &state); } static int lp855x_bl_update_status(struct backlight_device *bl) @@ -237,11 +246,12 @@ static int lp855x_bl_update_status(struct backlight_device *bl) brightness = 0; if (lp->mode == PWM_BASED) - lp855x_pwm_ctrl(lp, brightness, bl->props.max_brightness); + return lp855x_pwm_ctrl(lp, brightness, + bl->props.max_brightness); else if (lp->mode == REGISTER_BASED) - lp855x_write_byte(lp, lp->cfg->reg_brightness, (u8)brightness); - - return 0; + return lp855x_write_byte(lp, lp->cfg->reg_brightness, + (u8)brightness); + return -EINVAL; } static const struct backlight_ops lp855x_bl_ops = { @@ -387,7 +397,6 @@ static int lp855x_probe(struct i2c_client *cl) const struct i2c_device_id *id = i2c_client_get_device_id(cl); const struct acpi_device_id *acpi_id = NULL; struct device *dev = &cl->dev; - struct pwm_state pwmstate; struct lp855x *lp; int ret; @@ -470,15 +479,11 @@ static int lp855x_probe(struct i2c_client *cl) else return dev_err_probe(dev, ret, "getting PWM\n"); + lp->needs_pwm_init = false; lp->mode = REGISTER_BASED; dev_dbg(dev, "mode: register based\n"); } else { - pwm_init_state(lp->pwm, &pwmstate); - /* Legacy platform data compatibility */ - if (lp->pdata->period_ns > 0) - pwmstate.period = lp->pdata->period_ns; - pwm_apply_state(lp->pwm, &pwmstate); - + lp->needs_pwm_init = true; lp->mode = PWM_BASED; dev_dbg(dev, "mode: PWM based\n"); } diff --git a/drivers/video/backlight/qcom-wled.c b/drivers/video/backlight/qcom-wled.c index c6996aa288e6..10129095a4c1 100644 --- a/drivers/video/backlight/qcom-wled.c +++ b/drivers/video/backlight/qcom-wled.c @@ -9,8 +9,8 @@ #include #include #include -#include #include +#include #include /* From DT binding */ diff --git a/drivers/watchdog/Kconfig b/drivers/watchdog/Kconfig index 0cbfb496b9c8..751458959411 100644 --- a/drivers/watchdog/Kconfig +++ b/drivers/watchdog/Kconfig @@ -307,7 +307,7 @@ config XILINX_WATCHDOG config XILINX_WINDOW_WATCHDOG tristate "Xilinx window watchdog timer" depends on HAS_IOMEM - depends on ARM64 + depends on ARM64 || COMPILE_TEST select WATCHDOG_CORE help Window watchdog driver for the versal_wwdt IP core. @@ -343,7 +343,7 @@ config RAVE_SP_WATCHDOG config MLX_WDT tristate "Mellanox Watchdog" - depends on MELLANOX_PLATFORM + depends on MELLANOX_PLATFORM || COMPILE_TEST select WATCHDOG_CORE select REGMAP help @@ -493,7 +493,7 @@ config FTWDT010_WATCHDOG config IXP4XX_WATCHDOG tristate "IXP4xx Watchdog" - depends on ARCH_IXP4XX + depends on ARCH_IXP4XX || (ARM && COMPILE_TEST) select WATCHDOG_CORE help Say Y here if to include support for the watchdog timer @@ -529,7 +529,7 @@ config S3C2410_WATCHDOG config SA1100_WATCHDOG tristate "SA1100/PXA2xx watchdog" - depends on ARCH_SA1100 || ARCH_PXA + depends on ARCH_SA1100 || ARCH_PXA || COMPILE_TEST help Watchdog timer embedded into SA11x0 and PXA2xx chips. This will reboot your system when timeout is reached. @@ -720,7 +720,7 @@ config IMX2_WDT config IMX_SC_WDT tristate "IMX SC Watchdog" depends on HAVE_ARM_SMCCC - depends on IMX_SCU + depends on IMX_SCU || COMPILE_TEST select WATCHDOG_CORE help This is the driver for the system controller watchdog @@ -931,7 +931,7 @@ config ASPEED_WATCHDOG config STM32_WATCHDOG tristate "STM32 Independent WatchDoG (IWDG) support" - depends on ARCH_STM32 + depends on ARCH_STM32 || COMPILE_TEST select WATCHDOG_CORE default y help @@ -1065,7 +1065,7 @@ config ACQUIRE_WDT config ADVANTECH_WDT tristate "Advantech SBC Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help If you are configuring a Linux kernel for the Advantech single-board computer, say `Y' here to support its built-in watchdog timer @@ -1074,14 +1074,16 @@ config ADVANTECH_WDT config ADVANTECH_EC_WDT tristate "Advantech Embedded Controller Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST + select ISA_BUS_API + select WATCHDOG_CORE help This driver supports Advantech products with ITE based Embedded Controller. It does not support Advantech products with other ECs or without EC. config ALIM1535_WDT tristate "ALi M1535 PMU Watchdog Timer" - depends on X86 && PCI + depends on (X86 || COMPILE_TEST) && PCI help This is the driver for the hardware watchdog on the ALi M1535 PMU. @@ -1105,7 +1107,7 @@ config ALIM7101_WDT config EBC_C384_WDT tristate "WinSystems EBC-C384 Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST select ISA_BUS_API select WATCHDOG_CORE help @@ -1115,7 +1117,7 @@ config EBC_C384_WDT config EXAR_WDT tristate "Exar Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST select WATCHDOG_CORE help Enables watchdog timer support for the watchdog timer present @@ -1126,7 +1128,7 @@ config EXAR_WDT config F71808E_WDT tristate "Fintek F718xx, F818xx Super I/O Watchdog" - depends on X86 + depends on X86 || COMPILE_TEST select WATCHDOG_CORE help This is the driver for the hardware watchdog on the Fintek F71808E, @@ -1138,7 +1140,7 @@ config F71808E_WDT config SP5100_TCO tristate "AMD/ATI SP5100 TCO Timer/Watchdog" - depends on X86 && PCI + depends on (X86 || COMPILE_TEST) && PCI select WATCHDOG_CORE help Hardware watchdog driver for the AMD/ATI SP5100 chipset. The TCO @@ -1177,7 +1179,7 @@ config SC520_WDT config SBC_FITPC2_WATCHDOG tristate "Compulab SBC-FITPC2 watchdog" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the built-in watchdog timer on the fit-PC2, fit-PC2i, CM-iAM single-board computers made by Compulab. @@ -1202,7 +1204,7 @@ config SBC_FITPC2_WATCHDOG config EUROTECH_WDT tristate "Eurotech CPU-1220/1410 Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help Enable support for the watchdog timer on the Eurotech CPU-1220 and CPU-1410 cards. These are PC/104 SBCs. Spec sheets and product @@ -1210,7 +1212,7 @@ config EUROTECH_WDT config IB700_WDT tristate "IB700 SBC Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the hardware watchdog on the IB700 Single Board Computer produced by TMC Technology (www.tmc-uk.com). This @@ -1227,7 +1229,7 @@ config IB700_WDT config IBMASR tristate "IBM Automatic Server Restart" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the IBM Automatic Server Restart watchdog timer built-in into some eServer xSeries machines. @@ -1237,7 +1239,7 @@ config IBMASR config WAFER_WDT tristate "ICP Single Board Computer Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help This is a driver for the hardware watchdog on the ICP Single Board Computer. This driver is working on (at least) the following @@ -1259,7 +1261,7 @@ config I6300ESB_WDT config IE6XX_WDT tristate "Intel Atom E6xx Watchdog" - depends on X86 && PCI + depends on (X86 || COMPILE_TEST) && PCI select WATCHDOG_CORE select MFD_CORE select LPC_SCH @@ -1319,7 +1321,7 @@ config ITCO_VENDOR_SUPPORT config IT8712F_WDT tristate "IT8712F (Smart Guardian) Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the built-in watchdog timer on the IT8712F Super I/0 chipset used on many motherboards. @@ -1332,7 +1334,7 @@ config IT8712F_WDT config IT87_WDT tristate "IT87 Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST select WATCHDOG_CORE help This is the driver for the hardware watchdog on the ITE IT8607, @@ -1350,7 +1352,7 @@ config IT87_WDT config HP_WATCHDOG tristate "HP ProLiant iLO2+ Hardware Watchdog Timer" select WATCHDOG_CORE - depends on (ARM64 || X86) && PCI + depends on (ARM64 || X86 || COMPILE_TEST) && PCI help A software monitoring watchdog and NMI handling driver. This driver will detect lockups and provide a stack trace. This is a driver that @@ -1380,7 +1382,7 @@ config KEMPLD_WDT config SC1200_WDT tristate "National Semiconductor PC87307/PC97307 (ala SC1200) Watchdog" - depends on X86 + depends on X86 || COMPILE_TEST help This is a driver for National Semiconductor PC87307/PC97307 hardware watchdog cards as found on the SC1200. This watchdog is mainly used @@ -1403,7 +1405,7 @@ config SCx200_WDT config PC87413_WDT tristate "NS PC87413 watchdog" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the hardware watchdog on the PC87413 chipset This watchdog simply watches your kernel to make sure it doesn't @@ -1417,7 +1419,7 @@ config PC87413_WDT config NV_TCO tristate "nVidia TCO Timer/Watchdog" - depends on X86 && PCI + depends on (X86 || COMPILE_TEST) && PCI help Hardware driver for the TCO timer built into the nVidia Hub family (such as the MCP51). The TCO (Total Cost of Ownership) timer is a @@ -1446,7 +1448,7 @@ config RDC321X_WDT config 60XX_WDT tristate "SBC-60XX Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help This driver can be used with the watchdog timer found on some single board computers, namely the 6010 PII based computer. @@ -1486,7 +1488,7 @@ config SBC7240_WDT config CPU5_WDT tristate "SMA CPU5 Watchdog" - depends on X86 + depends on X86 || COMPILE_TEST help TBD. To compile this driver as a module, choose M here: the @@ -1494,7 +1496,7 @@ config CPU5_WDT config SMSC_SCH311X_WDT tristate "SMSC SCH311X Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the hardware watchdog timer on the SMSC SCH3112, SCH3114 and SCH3116 Super IO chipset @@ -1506,7 +1508,7 @@ config SMSC_SCH311X_WDT config SMSC37B787_WDT tristate "Winbond SMsC37B787 Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the hardware watchdog component on the Winbond SMsC37B787 chipset as used on the NetRunner Mainboard @@ -1526,7 +1528,7 @@ config SMSC37B787_WDT config TQMX86_WDT tristate "TQ-Systems TQMX86 Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST select WATCHDOG_CORE help This is the driver for the hardware watchdog timer in the TQMX86 IO @@ -1539,7 +1541,7 @@ config TQMX86_WDT config VIA_WDT tristate "VIA Watchdog Timer" - depends on X86 && PCI + depends on (X86 || COMPILE_TEST) && PCI select WATCHDOG_CORE help This is the driver for the hardware watchdog timer on VIA @@ -1552,7 +1554,7 @@ config VIA_WDT config W83627HF_WDT tristate "Watchdog timer for W83627HF/W83627DHG and compatibles" - depends on X86 + depends on X86 || COMPILE_TEST select WATCHDOG_CORE help This is the driver for the hardware watchdog on the following @@ -1582,7 +1584,7 @@ config W83627HF_WDT config W83877F_WDT tristate "W83877F (EMACS) Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the hardware watchdog on the W83877F chipset as used in EMACS PC-104 motherboards (and likely others). This @@ -1597,7 +1599,7 @@ config W83877F_WDT config W83977F_WDT tristate "W83977F (PCM-5335) Watchdog Timer" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the hardware watchdog on the W83977F I/O chip as used in AAEON's PCM-5335 SBC (and likely others). This @@ -1610,7 +1612,7 @@ config W83977F_WDT config MACHZ_WDT tristate "ZF MachZ Watchdog" - depends on X86 + depends on X86 || COMPILE_TEST help If you are using a ZF Micro MachZ processor, say Y here, otherwise N. This is the driver for the watchdog timer built-in on that @@ -1623,7 +1625,7 @@ config MACHZ_WDT config SBC_EPX_C3_WATCHDOG tristate "Winsystems SBC EPX-C3 watchdog" - depends on X86 + depends on X86 || COMPILE_TEST help This is the driver for the built-in watchdog timer on the EPX-C3 Single-board computer made by Winsystems, Inc. @@ -1739,7 +1741,7 @@ config INDYDOG config JZ4740_WDT tristate "Ingenic jz4740 SoC hardware watchdog" - depends on MIPS + depends on MIPS || COMPILE_TEST depends on COMMON_CLK select WATCHDOG_CORE select MFD_SYSCON @@ -1798,6 +1800,19 @@ config OCTEON_WDT from the first interrupt, it is then only poked when the device is written. +config MARVELL_GTI_WDT + tristate "Marvell GTI Watchdog driver" + depends on ARCH_THUNDER || (COMPILE_TEST && 64BIT) + default y + select WATCHDOG_CORE + help + Marvell GTI hardware supports watchdog timer. First timeout + works as watchdog pretimeout and installed interrupt handler + will be called on first timeout. Hardware can generate interrupt + to SCP on second timeout but it is not enabled, so second + timeout is ignored. If device poke does not happen then system + will reboot on third timeout. + config BCM2835_WDT tristate "Broadcom BCM2835 hardware watchdog" depends on ARCH_BCM2835 || (OF && COMPILE_TEST) @@ -1823,7 +1838,7 @@ config BCM_KONA_WDT config BCM_KONA_WDT_DEBUG bool "DEBUGFS support for BCM Kona Watchdog" - depends on BCM_KONA_WDT + depends on BCM_KONA_WDT || COMPILE_TEST help If enabled, adds /sys/kernel/debug/bcm_kona_wdt/info which provides access to the driver's internal data structures as well as watchdog @@ -1864,7 +1879,7 @@ config LANTIQ_WDT config LOONGSON1_WDT tristate "Loongson1 SoC hardware watchdog" - depends on MACH_LOONGSON32 + depends on MACH_LOONGSON32 || COMPILE_TEST select WATCHDOG_CORE help Hardware driver for the Loongson1 SoC Watchdog Timer. @@ -1878,7 +1893,7 @@ config RALINK_WDT config GXP_WATCHDOG tristate "HPE GXP watchdog support" - depends on ARCH_HPE_GXP + depends on ARCH_HPE_GXP || COMPILE_TEST select WATCHDOG_CORE help Say Y here to include support for the watchdog timer diff --git a/drivers/watchdog/Makefile b/drivers/watchdog/Makefile index 3633f5b98236..7eab9de311cb 100644 --- a/drivers/watchdog/Makefile +++ b/drivers/watchdog/Makefile @@ -98,6 +98,7 @@ obj-$(CONFIG_VISCONTI_WATCHDOG) += visconti_wdt.o obj-$(CONFIG_MSC313E_WATCHDOG) += msc313e_wdt.o obj-$(CONFIG_APPLE_WATCHDOG) += apple_wdt.o obj-$(CONFIG_SUNPLUS_WATCHDOG) += sunplus_wdt.o +obj-$(CONFIG_MARVELL_GTI_WDT) += marvell_gti_wdt.o # X86 (i386 + ia64 + x86_64) Architecture obj-$(CONFIG_ACQUIRE_WDT) += acquirewdt.o diff --git a/drivers/watchdog/armada_37xx_wdt.c b/drivers/watchdog/armada_37xx_wdt.c index e58652939f8a..8133a5d05647 100644 --- a/drivers/watchdog/armada_37xx_wdt.c +++ b/drivers/watchdog/armada_37xx_wdt.c @@ -14,7 +14,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/watchdog/at91rm9200_wdt.c b/drivers/watchdog/at91rm9200_wdt.c index d20ec27ba354..558015f08c7a 100644 --- a/drivers/watchdog/at91rm9200_wdt.c +++ b/drivers/watchdog/at91rm9200_wdt.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -26,8 +27,6 @@ #include #include #include -#include -#include #define WDT_DEFAULT_TIME 5 /* seconds */ #define WDT_MAX_TIME 256 /* seconds */ diff --git a/drivers/watchdog/cpwd.c b/drivers/watchdog/cpwd.c index 47250f9b68c7..901b94d456db 100644 --- a/drivers/watchdog/cpwd.c +++ b/drivers/watchdog/cpwd.c @@ -31,7 +31,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/watchdog/ftwdt010_wdt.c b/drivers/watchdog/ftwdt010_wdt.c index 442c5bf63ff4..28f5af752c10 100644 --- a/drivers/watchdog/ftwdt010_wdt.c +++ b/drivers/watchdog/ftwdt010_wdt.c @@ -14,7 +14,7 @@ #include #include #include -#include +#include #include #include #include @@ -221,20 +221,18 @@ static const struct dev_pm_ops ftwdt010_wdt_dev_pm_ops = { ftwdt010_wdt_resume) }; -#ifdef CONFIG_OF static const struct of_device_id ftwdt010_wdt_match[] = { { .compatible = "faraday,ftwdt010" }, { .compatible = "cortina,gemini-watchdog" }, {}, }; MODULE_DEVICE_TABLE(of, ftwdt010_wdt_match); -#endif static struct platform_driver ftwdt010_wdt_driver = { .probe = ftwdt010_wdt_probe, .driver = { .name = "ftwdt010-wdt", - .of_match_table = of_match_ptr(ftwdt010_wdt_match), + .of_match_table = ftwdt010_wdt_match, .pm = &ftwdt010_wdt_dev_pm_ops, }, }; diff --git a/drivers/watchdog/gef_wdt.c b/drivers/watchdog/gef_wdt.c index 97afc907f659..6a1db1c783fa 100644 --- a/drivers/watchdog/gef_wdt.c +++ b/drivers/watchdog/gef_wdt.c @@ -31,7 +31,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/watchdog/imx2_wdt.c b/drivers/watchdog/imx2_wdt.c index 6fcc3596103c..42e8ffae18dd 100644 --- a/drivers/watchdog/imx2_wdt.c +++ b/drivers/watchdog/imx2_wdt.c @@ -26,8 +26,7 @@ #include #include #include -#include -#include +#include #include #include #include @@ -375,7 +374,7 @@ static void imx2_wdt_shutdown(struct platform_device *pdev) */ imx2_wdt_set_timeout(wdog, IMX2_WDT_MAX_TIME); imx2_wdt_ping(wdog); - dev_crit(&pdev->dev, "Device shutdown: Expect reboot!\n"); + dev_crit(&pdev->dev, "Device shutdown.\n"); } } diff --git a/drivers/watchdog/imx7ulp_wdt.c b/drivers/watchdog/imx7ulp_wdt.c index 7ca486794ba7..c703586c6e5f 100644 --- a/drivers/watchdog/imx7ulp_wdt.c +++ b/drivers/watchdog/imx7ulp_wdt.c @@ -9,7 +9,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/watchdog/intel-mid_wdt.c b/drivers/watchdog/intel-mid_wdt.c index 9b2173f765c8..fb7fae750181 100644 --- a/drivers/watchdog/intel-mid_wdt.c +++ b/drivers/watchdog/intel-mid_wdt.c @@ -203,3 +203,4 @@ module_platform_driver(mid_wdt_driver); MODULE_AUTHOR("David Cohen "); MODULE_DESCRIPTION("Watchdog Driver for Intel MID platform"); MODULE_LICENSE("GPL"); +MODULE_ALIAS("platform:intel_mid_wdt"); diff --git a/drivers/watchdog/lantiq_wdt.c b/drivers/watchdog/lantiq_wdt.c index 6fab504af88b..a273b97ebcb4 100644 --- a/drivers/watchdog/lantiq_wdt.c +++ b/drivers/watchdog/lantiq_wdt.c @@ -9,7 +9,8 @@ #include #include #include -#include +#include +#include #include #include #include diff --git a/drivers/watchdog/loongson1_wdt.c b/drivers/watchdog/loongson1_wdt.c index 4ac7810a314d..0587ff44d3a1 100644 --- a/drivers/watchdog/loongson1_wdt.c +++ b/drivers/watchdog/loongson1_wdt.c @@ -4,6 +4,7 @@ */ #include +#include #include #include #include diff --git a/drivers/watchdog/marvell_gti_wdt.c b/drivers/watchdog/marvell_gti_wdt.c new file mode 100644 index 000000000000..d7eb8286e11e --- /dev/null +++ b/drivers/watchdog/marvell_gti_wdt.c @@ -0,0 +1,340 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Marvell GTI Watchdog driver + * + * Copyright (C) 2023 Marvell. + */ + +#include +#include +#include +#include +#include +#include +#include + +/* + * Hardware supports following mode of operation: + * 1) Interrupt Only: + * This will generate the interrupt to arm core whenever timeout happens. + * + * 2) Interrupt + del3t (Interrupt to firmware (SCP processor)). + * This will generate interrupt to arm core on 1st timeout happens + * This will generate interrupt to SCP processor on 2nd timeout happens + * + * 3) Interrupt + Interrupt to SCP processor (called delt3t) + reboot. + * This will generate interrupt to arm core on 1st timeout happens + * Will generate interrupt to SCP processor on 2nd timeout happens, + * if interrupt is configured. + * Reboot on 3rd timeout. + * + * Driver will use hardware in mode-3 above so that system can reboot in case + * a hardware hang. Also h/w is configured not to generate SCP interrupt, so + * effectively 2nd timeout is ignored within hardware. + * + * First timeout is effectively watchdog pretimeout. + */ + +/* GTI CWD Watchdog (GTI_CWD_WDOG) Register */ +#define GTI_CWD_WDOG(reg_offset) (0x8 * (reg_offset)) +#define GTI_CWD_WDOG_MODE_INT_DEL3T_RST 0x3 +#define GTI_CWD_WDOG_MODE_MASK GENMASK_ULL(1, 0) +#define GTI_CWD_WDOG_LEN_SHIFT 4 +#define GTI_CWD_WDOG_LEN_MASK GENMASK_ULL(19, 4) +#define GTI_CWD_WDOG_CNT_SHIFT 20 +#define GTI_CWD_WDOG_CNT_MASK GENMASK_ULL(43, 20) + +/* GTI CWD Watchdog Interrupt (GTI_CWD_INT) Register */ +#define GTI_CWD_INT 0x200 +#define GTI_CWD_INT_PENDING_STATUS(bit) BIT_ULL(bit) + +/* GTI CWD Watchdog Interrupt Enable Clear (GTI_CWD_INT_ENA_CLR) Register */ +#define GTI_CWD_INT_ENA_CLR 0x210 +#define GTI_CWD_INT_ENA_CLR_VAL(bit) BIT_ULL(bit) + +/* GTI CWD Watchdog Interrupt Enable Set (GTI_CWD_INT_ENA_SET) Register */ +#define GTI_CWD_INT_ENA_SET 0x218 +#define GTI_CWD_INT_ENA_SET_VAL(bit) BIT_ULL(bit) + +/* GTI CWD Watchdog Poke (GTI_CWD_POKE) Registers */ +#define GTI_CWD_POKE(reg_offset) (0x10000 + 0x8 * (reg_offset)) +#define GTI_CWD_POKE_VAL 1 + +struct gti_match_data { + u32 gti_num_timers; +}; + +static const struct gti_match_data match_data_octeontx2 = { + .gti_num_timers = 54, +}; + +static const struct gti_match_data match_data_cn10k = { + .gti_num_timers = 64, +}; + +struct gti_wdt_priv { + struct watchdog_device wdev; + void __iomem *base; + u32 clock_freq; + struct clk *sclk; + /* wdt_timer_idx used for timer to be used for system watchdog */ + u32 wdt_timer_idx; + const struct gti_match_data *data; +}; + +static irqreturn_t gti_wdt_interrupt(int irq, void *data) +{ + struct watchdog_device *wdev = data; + struct gti_wdt_priv *priv = watchdog_get_drvdata(wdev); + + /* Clear Interrupt Pending Status */ + writeq(GTI_CWD_INT_PENDING_STATUS(priv->wdt_timer_idx), + priv->base + GTI_CWD_INT); + + watchdog_notify_pretimeout(wdev); + + return IRQ_HANDLED; +} + +static int gti_wdt_ping(struct watchdog_device *wdev) +{ + struct gti_wdt_priv *priv = watchdog_get_drvdata(wdev); + + writeq(GTI_CWD_POKE_VAL, + priv->base + GTI_CWD_POKE(priv->wdt_timer_idx)); + + return 0; +} + +static int gti_wdt_start(struct watchdog_device *wdev) +{ + struct gti_wdt_priv *priv = watchdog_get_drvdata(wdev); + u64 regval; + + if (!wdev->pretimeout) + return -EINVAL; + + set_bit(WDOG_HW_RUNNING, &wdev->status); + + /* Clear any pending interrupt */ + writeq(GTI_CWD_INT_PENDING_STATUS(priv->wdt_timer_idx), + priv->base + GTI_CWD_INT); + + /* Enable Interrupt */ + writeq(GTI_CWD_INT_ENA_SET_VAL(priv->wdt_timer_idx), + priv->base + GTI_CWD_INT_ENA_SET); + + /* Set (Interrupt + SCP interrupt (DEL3T) + core domain reset) Mode */ + regval = readq(priv->base + GTI_CWD_WDOG(priv->wdt_timer_idx)); + regval |= GTI_CWD_WDOG_MODE_INT_DEL3T_RST; + writeq(regval, priv->base + GTI_CWD_WDOG(priv->wdt_timer_idx)); + + return 0; +} + +static int gti_wdt_stop(struct watchdog_device *wdev) +{ + struct gti_wdt_priv *priv = watchdog_get_drvdata(wdev); + u64 regval; + + /* Disable Interrupt */ + writeq(GTI_CWD_INT_ENA_CLR_VAL(priv->wdt_timer_idx), + priv->base + GTI_CWD_INT_ENA_CLR); + + /* Set GTI_CWD_WDOG.Mode = 0 to stop the timer */ + regval = readq(priv->base + GTI_CWD_WDOG(priv->wdt_timer_idx)); + regval &= ~GTI_CWD_WDOG_MODE_MASK; + writeq(regval, priv->base + GTI_CWD_WDOG(priv->wdt_timer_idx)); + + return 0; +} + +static int gti_wdt_settimeout(struct watchdog_device *wdev, + unsigned int timeout) +{ + struct gti_wdt_priv *priv = watchdog_get_drvdata(wdev); + u64 timeout_wdog, regval; + + /* Update new timeout */ + wdev->timeout = timeout; + + /* Pretimeout is 1/3 of timeout */ + wdev->pretimeout = timeout / 3; + + /* Get clock cycles from pretimeout */ + timeout_wdog = (u64)priv->clock_freq * wdev->pretimeout; + + /* Watchdog counts in 1024 cycle steps */ + timeout_wdog = timeout_wdog >> 10; + + /* GTI_CWD_WDOG.CNT: reload counter is 16-bit */ + timeout_wdog = (timeout_wdog + 0xff) >> 8; + if (timeout_wdog >= 0x10000) + timeout_wdog = 0xffff; + + /* + * GTI_CWD_WDOG.LEN is 24bit, lower 8-bits should be zero and + * upper 16-bits are same as GTI_CWD_WDOG.CNT + */ + regval = readq(priv->base + GTI_CWD_WDOG(priv->wdt_timer_idx)); + regval &= GTI_CWD_WDOG_MODE_MASK; + regval |= (timeout_wdog << (GTI_CWD_WDOG_CNT_SHIFT + 8)) | + (timeout_wdog << GTI_CWD_WDOG_LEN_SHIFT); + writeq(regval, priv->base + GTI_CWD_WDOG(priv->wdt_timer_idx)); + + return 0; +} + +static int gti_wdt_set_pretimeout(struct watchdog_device *wdev, + unsigned int timeout) +{ + struct gti_wdt_priv *priv = watchdog_get_drvdata(wdev); + struct watchdog_device *wdog_dev = &priv->wdev; + + /* pretimeout should 1/3 of max_timeout */ + if (timeout * 3 <= wdog_dev->max_timeout) + return gti_wdt_settimeout(wdev, timeout * 3); + + return -EINVAL; +} + +static void gti_clk_disable_unprepare(void *data) +{ + clk_disable_unprepare(data); +} + +static int gti_wdt_get_cntfrq(struct platform_device *pdev, + struct gti_wdt_priv *priv) +{ + int err; + + priv->sclk = devm_clk_get_enabled(&pdev->dev, NULL); + if (IS_ERR(priv->sclk)) + return PTR_ERR(priv->sclk); + + err = devm_add_action_or_reset(&pdev->dev, + gti_clk_disable_unprepare, priv->sclk); + if (err) + return err; + + priv->clock_freq = clk_get_rate(priv->sclk); + if (!priv->clock_freq) + return -EINVAL; + + return 0; +} + +static const struct watchdog_info gti_wdt_ident = { + .identity = "Marvell GTI watchdog", + .options = WDIOF_SETTIMEOUT | WDIOF_PRETIMEOUT | WDIOF_KEEPALIVEPING | + WDIOF_MAGICCLOSE | WDIOF_CARDRESET, +}; + +static const struct watchdog_ops gti_wdt_ops = { + .owner = THIS_MODULE, + .start = gti_wdt_start, + .stop = gti_wdt_stop, + .ping = gti_wdt_ping, + .set_timeout = gti_wdt_settimeout, + .set_pretimeout = gti_wdt_set_pretimeout, +}; + +static int gti_wdt_probe(struct platform_device *pdev) +{ + struct gti_wdt_priv *priv; + struct device *dev = &pdev->dev; + struct watchdog_device *wdog_dev; + u64 max_pretimeout; + u32 wdt_idx; + int irq; + int err; + + priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL); + if (!priv) + return -ENOMEM; + + priv->base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(priv->base)) + return dev_err_probe(&pdev->dev, PTR_ERR(priv->base), + "reg property not valid/found\n"); + + err = gti_wdt_get_cntfrq(pdev, priv); + if (err) + return dev_err_probe(&pdev->dev, err, + "GTI clock frequency not valid/found"); + + priv->data = of_device_get_match_data(dev); + + /* default use last timer for watchdog */ + priv->wdt_timer_idx = priv->data->gti_num_timers - 1; + + err = of_property_read_u32(dev->of_node, "marvell,wdt-timer-index", + &wdt_idx); + if (!err) { + if (wdt_idx >= priv->data->gti_num_timers) + return dev_err_probe(&pdev->dev, err, + "GTI wdog timer index not valid"); + + priv->wdt_timer_idx = wdt_idx; + } + + wdog_dev = &priv->wdev; + wdog_dev->info = >i_wdt_ident, + wdog_dev->ops = >i_wdt_ops, + wdog_dev->parent = dev; + /* + * Watchdog counter is 24 bit where lower 8 bits are zeros + * This counter decrements every 1024 clock cycles. + */ + max_pretimeout = (GTI_CWD_WDOG_CNT_MASK >> GTI_CWD_WDOG_CNT_SHIFT); + max_pretimeout &= ~0xFFUL; + max_pretimeout = (max_pretimeout * 1024) / priv->clock_freq; + wdog_dev->pretimeout = max_pretimeout; + + /* Maximum timeout is 3 times the pretimeout */ + wdog_dev->max_timeout = max_pretimeout * 3; + /* Minimum first timeout (pretimeout) is 1, so min_timeout as 3 */ + wdog_dev->min_timeout = 3; + wdog_dev->timeout = wdog_dev->pretimeout; + + watchdog_set_drvdata(wdog_dev, priv); + platform_set_drvdata(pdev, priv); + gti_wdt_settimeout(wdog_dev, wdog_dev->timeout); + watchdog_stop_on_reboot(wdog_dev); + watchdog_stop_on_unregister(wdog_dev); + + err = devm_watchdog_register_device(dev, wdog_dev); + if (err) + return err; + + irq = platform_get_irq(pdev, 0); + if (irq < 0) + return dev_err_probe(&pdev->dev, irq, "IRQ resource not found\n"); + + err = devm_request_irq(dev, irq, gti_wdt_interrupt, 0, + pdev->name, &priv->wdev); + if (err) + return dev_err_probe(dev, err, "Failed to register interrupt handler\n"); + + dev_info(dev, "Watchdog enabled (timeout=%d sec)\n", wdog_dev->timeout); + return 0; +} + +static const struct of_device_id gti_wdt_of_match[] = { + { .compatible = "marvell,cn9670-wdt", .data = &match_data_octeontx2}, + { .compatible = "marvell,cn10624-wdt", .data = &match_data_cn10k}, + { }, +}; +MODULE_DEVICE_TABLE(of, gti_wdt_of_match); + +static struct platform_driver gti_wdt_driver = { + .driver = { + .name = "gti-wdt", + .of_match_table = gti_wdt_of_match, + }, + .probe = gti_wdt_probe, +}; +module_platform_driver(gti_wdt_driver); + +MODULE_AUTHOR("Bharat Bhushan "); +MODULE_DESCRIPTION("Marvell GTI watchdog driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/watchdog/menz69_wdt.c b/drivers/watchdog/menz69_wdt.c index 3c98030b9fcd..c7de30270043 100644 --- a/drivers/watchdog/menz69_wdt.c +++ b/drivers/watchdog/menz69_wdt.c @@ -153,7 +153,6 @@ MODULE_DEVICE_TABLE(mcb, men_z069_ids); static struct mcb_driver men_z069_driver = { .driver = { .name = "z069-wdt", - .owner = THIS_MODULE, }, .probe = men_z069_probe, .remove = men_z069_remove, diff --git a/drivers/watchdog/meson_gxbb_wdt.c b/drivers/watchdog/meson_gxbb_wdt.c index 35d80cb39856..a48622d11ad7 100644 --- a/drivers/watchdog/meson_gxbb_wdt.c +++ b/drivers/watchdog/meson_gxbb_wdt.c @@ -22,7 +22,6 @@ #define GXBB_WDT_CTRL_CLKDIV_EN BIT(25) #define GXBB_WDT_CTRL_CLK_EN BIT(24) -#define GXBB_WDT_CTRL_EE_RESET BIT(21) #define GXBB_WDT_CTRL_EN BIT(18) #define GXBB_WDT_CTRL_DIV_MASK (BIT(18) - 1) @@ -45,6 +44,10 @@ struct meson_gxbb_wdt { struct clk *clk; }; +struct wdt_params { + u32 rst; +}; + static int meson_gxbb_wdt_start(struct watchdog_device *wdt_dev) { struct meson_gxbb_wdt *data = watchdog_get_drvdata(wdt_dev); @@ -140,8 +143,17 @@ static const struct dev_pm_ops meson_gxbb_wdt_pm_ops = { SET_SYSTEM_SLEEP_PM_OPS(meson_gxbb_wdt_suspend, meson_gxbb_wdt_resume) }; +static const struct wdt_params gxbb_params = { + .rst = BIT(21), +}; + +static const struct wdt_params t7_params = { + .rst = BIT(22), +}; + static const struct of_device_id meson_gxbb_wdt_dt_ids[] = { - { .compatible = "amlogic,meson-gxbb-wdt", }, + { .compatible = "amlogic,meson-gxbb-wdt", .data = &gxbb_params, }, + { .compatible = "amlogic,t7-wdt", .data = &t7_params, }, { /* sentinel */ }, }; MODULE_DEVICE_TABLE(of, meson_gxbb_wdt_dt_ids); @@ -150,6 +162,7 @@ static int meson_gxbb_wdt_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct meson_gxbb_wdt *data; + struct wdt_params *params; u32 ctrl_reg; data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL); @@ -164,6 +177,8 @@ static int meson_gxbb_wdt_probe(struct platform_device *pdev) if (IS_ERR(data->clk)) return PTR_ERR(data->clk); + params = (struct wdt_params *)of_device_get_match_data(dev); + platform_set_drvdata(pdev, data); data->wdt_dev.parent = dev; @@ -191,7 +206,7 @@ static int meson_gxbb_wdt_probe(struct platform_device *pdev) /* Setup with 1ms timebase */ ctrl_reg |= ((clk_get_rate(data->clk) / 1000) & GXBB_WDT_CTRL_DIV_MASK) | - GXBB_WDT_CTRL_EE_RESET | + params->rst | GXBB_WDT_CTRL_CLK_EN | GXBB_WDT_CTRL_CLKDIV_EN; diff --git a/drivers/watchdog/meson_wdt.c b/drivers/watchdog/meson_wdt.c index 539feaa1f904..497496f64f55 100644 --- a/drivers/watchdog/meson_wdt.c +++ b/drivers/watchdog/meson_wdt.c @@ -11,11 +11,11 @@ #include #include #include +#include #include #include -#include -#include #include +#include #include #include diff --git a/drivers/watchdog/mpc8xxx_wdt.c b/drivers/watchdog/mpc8xxx_wdt.c index 1c569be72ea2..867f9f311379 100644 --- a/drivers/watchdog/mpc8xxx_wdt.c +++ b/drivers/watchdog/mpc8xxx_wdt.c @@ -16,8 +16,8 @@ #include #include #include -#include -#include +#include +#include #include #include #include diff --git a/drivers/watchdog/mtk_wdt.c b/drivers/watchdog/mtk_wdt.c index a9c437598e7e..b2330b16b497 100644 --- a/drivers/watchdog/mtk_wdt.c +++ b/drivers/watchdog/mtk_wdt.c @@ -25,7 +25,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/watchdog/of_xilinx_wdt.c b/drivers/watchdog/of_xilinx_wdt.c index 2a079ca04aa3..05657dc1d36a 100644 --- a/drivers/watchdog/of_xilinx_wdt.c +++ b/drivers/watchdog/of_xilinx_wdt.c @@ -10,14 +10,13 @@ #include #include #include +#include #include #include #include #include #include #include -#include -#include /* Register offsets for the Wdt device */ #define XWT_TWCSR0_OFFSET 0x0 /* Control/Status Register0 */ diff --git a/drivers/watchdog/pic32-dmt.c b/drivers/watchdog/pic32-dmt.c index bc4ccddc75a3..ab0682492c85 100644 --- a/drivers/watchdog/pic32-dmt.c +++ b/drivers/watchdog/pic32-dmt.c @@ -12,7 +12,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/watchdog/pic32-wdt.c b/drivers/watchdog/pic32-wdt.c index 6d1a00222991..1d282de312ef 100644 --- a/drivers/watchdog/pic32-wdt.c +++ b/drivers/watchdog/pic32-wdt.c @@ -12,7 +12,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/watchdog/pika_wdt.c b/drivers/watchdog/pika_wdt.c index a98abd0d3146..782b8c23d99c 100644 --- a/drivers/watchdog/pika_wdt.c +++ b/drivers/watchdog/pika_wdt.c @@ -23,8 +23,8 @@ #include #include #include +#include #include -#include #define DRV_NAME "PIKA-WDT" diff --git a/drivers/watchdog/pm8916_wdt.c b/drivers/watchdog/pm8916_wdt.c index f4bfbffaf49c..f3fcbeb0852c 100644 --- a/drivers/watchdog/pm8916_wdt.c +++ b/drivers/watchdog/pm8916_wdt.c @@ -266,7 +266,7 @@ static struct platform_driver pm8916_wdt_driver = { .probe = pm8916_wdt_probe, .driver = { .name = "pm8916-wdt", - .of_match_table = of_match_ptr(pm8916_wdt_id_table), + .of_match_table = pm8916_wdt_id_table, .pm = &pm8916_wdt_pm_ops, }, }; diff --git a/drivers/watchdog/qcom-wdt.c b/drivers/watchdog/qcom-wdt.c index d776474dcdf3..9e790f0c2096 100644 --- a/drivers/watchdog/qcom-wdt.c +++ b/drivers/watchdog/qcom-wdt.c @@ -11,7 +11,6 @@ #include #include #include -#include enum wdt_reg { WDT_RST, diff --git a/drivers/watchdog/rave-sp-wdt.c b/drivers/watchdog/rave-sp-wdt.c index 2c95615b6354..5d1c2176d445 100644 --- a/drivers/watchdog/rave-sp-wdt.c +++ b/drivers/watchdog/rave-sp-wdt.c @@ -13,7 +13,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/watchdog/riowd.c b/drivers/watchdog/riowd.c index c04b383e1712..b293792a292a 100644 --- a/drivers/watchdog/riowd.c +++ b/drivers/watchdog/riowd.c @@ -14,7 +14,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/watchdog/rti_wdt.c b/drivers/watchdog/rti_wdt.c index ce8f18e93aa9..8e1be7ba0103 100644 --- a/drivers/watchdog/rti_wdt.c +++ b/drivers/watchdog/rti_wdt.c @@ -14,6 +14,8 @@ #include #include #include +#include +#include #include #include #include @@ -52,6 +54,11 @@ #define DWDST BIT(1) +#define PON_REASON_SOF_NUM 0xBBBBCCCC +#define PON_REASON_MAGIC_NUM 0xDDDDDDDD +#define PON_REASON_EOF_NUM 0xCCCCBBBB +#define RESERVED_MEM_MIN_SIZE 12 + static int heartbeat = DEFAULT_HEARTBEAT; /* @@ -198,6 +205,11 @@ static int rti_wdt_probe(struct platform_device *pdev) struct rti_wdt_device *wdt; struct clk *clk; u32 last_ping = 0; + struct device_node *node; + u32 reserved_mem_size; + struct resource res; + u32 *vaddr; + u64 paddr; wdt = devm_kzalloc(dev, sizeof(*wdt), GFP_KERNEL); if (!wdt) @@ -284,6 +296,42 @@ static int rti_wdt_probe(struct platform_device *pdev) } } + node = of_parse_phandle(pdev->dev.of_node, "memory-region", 0); + if (node) { + ret = of_address_to_resource(node, 0, &res); + if (ret) { + dev_err(dev, "No memory address assigned to the region.\n"); + goto err_iomap; + } + + /* + * If reserved memory is defined for watchdog reset cause. + * Readout the Power-on(PON) reason and pass to bootstatus. + */ + paddr = res.start; + reserved_mem_size = resource_size(&res); + if (reserved_mem_size < RESERVED_MEM_MIN_SIZE) { + dev_err(dev, "The size of reserved memory is too small.\n"); + ret = -EINVAL; + goto err_iomap; + } + + vaddr = memremap(paddr, reserved_mem_size, MEMREMAP_WB); + if (!vaddr) { + dev_err(dev, "Failed to map memory-region.\n"); + ret = -ENOMEM; + goto err_iomap; + } + + if (vaddr[0] == PON_REASON_SOF_NUM && + vaddr[1] == PON_REASON_MAGIC_NUM && + vaddr[2] == PON_REASON_EOF_NUM) { + wdd->bootstatus |= WDIOF_CARDRESET; + } + memset(vaddr, 0, reserved_mem_size); + memunmap(vaddr); + } + watchdog_init_timeout(wdd, heartbeat, dev); ret = watchdog_register_device(wdd); diff --git a/drivers/watchdog/rza_wdt.c b/drivers/watchdog/rza_wdt.c index fe6c2ed35e04..cb4901b3f777 100644 --- a/drivers/watchdog/rza_wdt.c +++ b/drivers/watchdog/rza_wdt.c @@ -9,9 +9,9 @@ #include #include #include +#include #include -#include -#include +#include #include #include diff --git a/drivers/watchdog/rzg2l_wdt.c b/drivers/watchdog/rzg2l_wdt.c index d404953d0e0f..1741f98ca67c 100644 --- a/drivers/watchdog/rzg2l_wdt.c +++ b/drivers/watchdog/rzg2l_wdt.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/watchdog/s3c2410_wdt.c b/drivers/watchdog/s3c2410_wdt.c index 95416a9bdd4b..0b4bd883ff28 100644 --- a/drivers/watchdog/s3c2410_wdt.c +++ b/drivers/watchdog/s3c2410_wdt.c @@ -23,7 +23,6 @@ #include #include #include -#include #include #include #include @@ -379,10 +378,11 @@ static int s3c2410wdt_enable(struct s3c2410_wdt *wdt, bool en) static int s3c2410wdt_keepalive(struct watchdog_device *wdd) { struct s3c2410_wdt *wdt = watchdog_get_drvdata(wdd); + unsigned long flags; - spin_lock(&wdt->lock); + spin_lock_irqsave(&wdt->lock, flags); writel(wdt->count, wdt->reg_base + S3C2410_WTCNT); - spin_unlock(&wdt->lock); + spin_unlock_irqrestore(&wdt->lock, flags); return 0; } @@ -399,10 +399,11 @@ static void __s3c2410wdt_stop(struct s3c2410_wdt *wdt) static int s3c2410wdt_stop(struct watchdog_device *wdd) { struct s3c2410_wdt *wdt = watchdog_get_drvdata(wdd); + unsigned long flags; - spin_lock(&wdt->lock); + spin_lock_irqsave(&wdt->lock, flags); __s3c2410wdt_stop(wdt); - spin_unlock(&wdt->lock); + spin_unlock_irqrestore(&wdt->lock, flags); return 0; } @@ -411,8 +412,9 @@ static int s3c2410wdt_start(struct watchdog_device *wdd) { unsigned long wtcon; struct s3c2410_wdt *wdt = watchdog_get_drvdata(wdd); + unsigned long flags; - spin_lock(&wdt->lock); + spin_lock_irqsave(&wdt->lock, flags); __s3c2410wdt_stop(wdt); @@ -433,7 +435,7 @@ static int s3c2410wdt_start(struct watchdog_device *wdd) writel(wdt->count, wdt->reg_base + S3C2410_WTDAT); writel(wdt->count, wdt->reg_base + S3C2410_WTCNT); writel(wtcon, wdt->reg_base + S3C2410_WTCON); - spin_unlock(&wdt->lock); + spin_unlock_irqrestore(&wdt->lock, flags); return 0; } diff --git a/drivers/watchdog/sama5d4_wdt.c b/drivers/watchdog/sama5d4_wdt.c index aeee934ca51b..13e72918338a 100644 --- a/drivers/watchdog/sama5d4_wdt.c +++ b/drivers/watchdog/sama5d4_wdt.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include #include @@ -255,6 +254,7 @@ static int sama5d4_wdt_probe(struct platform_device *pdev) struct sama5d4_wdt *wdt; void __iomem *regs; u32 irq = 0; + u32 reg; int ret; wdt = devm_kzalloc(dev, sizeof(*wdt), GFP_KERNEL); @@ -305,6 +305,12 @@ static int sama5d4_wdt_probe(struct platform_device *pdev) watchdog_init_timeout(wdd, wdt_timeout, dev); + reg = wdt_read(wdt, AT91_WDT_MR); + if (!(reg & AT91_WDT_WDDIS)) { + wdt->mr &= ~AT91_WDT_WDDIS; + set_bit(WDOG_HW_RUNNING, &wdd->status); + } + ret = sama5d4_wdt_init(wdt); if (ret) return ret; diff --git a/drivers/watchdog/sbsa_gwdt.c b/drivers/watchdog/sbsa_gwdt.c index fd3cfdda4949..421ebcda62e6 100644 --- a/drivers/watchdog/sbsa_gwdt.c +++ b/drivers/watchdog/sbsa_gwdt.c @@ -43,10 +43,9 @@ #include #include #include +#include #include #include -#include -#include #include #include #include diff --git a/drivers/watchdog/starfive-wdt.c b/drivers/watchdog/starfive-wdt.c index 8058fca4d05d..5f501b41faf9 100644 --- a/drivers/watchdog/starfive-wdt.c +++ b/drivers/watchdog/starfive-wdt.c @@ -8,7 +8,8 @@ #include #include #include -#include +#include +#include #include #include #include @@ -526,7 +527,6 @@ static void starfive_wdt_shutdown(struct platform_device *pdev) starfive_wdt_pm_stop(&wdt->wdd); } -#ifdef CONFIG_PM_SLEEP static int starfive_wdt_suspend(struct device *dev) { struct starfive_wdt *wdt = dev_get_drvdata(dev); @@ -556,9 +556,7 @@ static int starfive_wdt_resume(struct device *dev) return starfive_wdt_start(wdt); } -#endif /* CONFIG_PM_SLEEP */ -#ifdef CONFIG_PM static int starfive_wdt_runtime_suspend(struct device *dev) { struct starfive_wdt *wdt = dev_get_drvdata(dev); @@ -574,11 +572,10 @@ static int starfive_wdt_runtime_resume(struct device *dev) return starfive_wdt_enable_clock(wdt); } -#endif /* CONFIG_PM */ static const struct dev_pm_ops starfive_wdt_pm_ops = { - SET_RUNTIME_PM_OPS(starfive_wdt_runtime_suspend, starfive_wdt_runtime_resume, NULL) - SET_SYSTEM_SLEEP_PM_OPS(starfive_wdt_suspend, starfive_wdt_resume) + RUNTIME_PM_OPS(starfive_wdt_runtime_suspend, starfive_wdt_runtime_resume, NULL) + SYSTEM_SLEEP_PM_OPS(starfive_wdt_suspend, starfive_wdt_resume) }; static const struct of_device_id starfive_wdt_match[] = { @@ -594,7 +591,7 @@ static struct platform_driver starfive_wdt_driver = { .shutdown = starfive_wdt_shutdown, .driver = { .name = "starfive-wdt", - .pm = &starfive_wdt_pm_ops, + .pm = pm_ptr(&starfive_wdt_pm_ops), .of_match_table = starfive_wdt_match, }, }; diff --git a/drivers/watchdog/stm32_iwdg.c b/drivers/watchdog/stm32_iwdg.c index 570a71509d2a..d9fd50df9802 100644 --- a/drivers/watchdog/stm32_iwdg.c +++ b/drivers/watchdog/stm32_iwdg.c @@ -17,7 +17,6 @@ #include #include #include -#include #include #include @@ -288,7 +287,7 @@ static struct platform_driver stm32_iwdg_driver = { .probe = stm32_iwdg_probe, .driver = { .name = "iwdg", - .of_match_table = of_match_ptr(stm32_iwdg_of_match), + .of_match_table = stm32_iwdg_of_match, }, }; module_platform_driver(stm32_iwdg_driver); diff --git a/drivers/watchdog/sunxi_wdt.c b/drivers/watchdog/sunxi_wdt.c index 6cf82922d3fb..b85354a99582 100644 --- a/drivers/watchdog/sunxi_wdt.c +++ b/drivers/watchdog/sunxi_wdt.c @@ -18,7 +18,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/watchdog/watchdog_core.c b/drivers/watchdog/watchdog_core.c index d4c5a736fdcb..5b55ccae06d4 100644 --- a/drivers/watchdog/watchdog_core.c +++ b/drivers/watchdog/watchdog_core.c @@ -161,7 +161,7 @@ static int watchdog_reboot_notifier(struct notifier_block *nb, struct watchdog_device *wdd; wdd = container_of(nb, struct watchdog_device, reboot_nb); - if (code == SYS_DOWN || code == SYS_HALT) { + if (code == SYS_DOWN || code == SYS_HALT || code == SYS_POWER_OFF) { if (watchdog_hw_running(wdd)) { int ret; diff --git a/drivers/watchdog/xilinx_wwdt.c b/drivers/watchdog/xilinx_wwdt.c index 2585038d5575..d271e2e8d6e2 100644 --- a/drivers/watchdog/xilinx_wwdt.c +++ b/drivers/watchdog/xilinx_wwdt.c @@ -9,9 +9,10 @@ #include #include #include +#include +#include #include -#include -#include +#include #include /* Max timeout is calculated at 100MHz source clock */ @@ -71,7 +72,7 @@ static int xilinx_wwdt_start(struct watchdog_device *wdd) /* Calculate timeout count */ time_out = xdev->freq * wdd->timeout; - closed_timeout = (time_out * xdev->close_percent) / 100; + closed_timeout = div_u64(time_out * xdev->close_percent, 100); open_timeout = time_out - closed_timeout; wdd->min_hw_heartbeat_ms = xdev->close_percent * 10 * wdd->timeout; diff --git a/fs/ceph/Makefile b/fs/ceph/Makefile index 50c635dc7f71..1f77ca04c426 100644 --- a/fs/ceph/Makefile +++ b/fs/ceph/Makefile @@ -12,3 +12,4 @@ ceph-y := super.o inode.o dir.o file.o locks.o addr.o ioctl.o \ ceph-$(CONFIG_CEPH_FSCACHE) += cache.o ceph-$(CONFIG_CEPH_FS_POSIX_ACL) += acl.o +ceph-$(CONFIG_FS_ENCRYPTION) += crypto.o diff --git a/fs/ceph/acl.c b/fs/ceph/acl.c index c91b293267d7..c53a1d220622 100644 --- a/fs/ceph/acl.c +++ b/fs/ceph/acl.c @@ -140,7 +140,7 @@ int ceph_set_acl(struct mnt_idmap *idmap, struct dentry *dentry, newattrs.ia_ctime = current_time(inode); newattrs.ia_mode = new_mode; newattrs.ia_valid = ATTR_MODE | ATTR_CTIME; - ret = __ceph_setattr(inode, &newattrs); + ret = __ceph_setattr(inode, &newattrs, NULL); if (ret) goto out_free; } @@ -151,7 +151,7 @@ int ceph_set_acl(struct mnt_idmap *idmap, struct dentry *dentry, newattrs.ia_ctime = old_ctime; newattrs.ia_mode = old_mode; newattrs.ia_valid = ATTR_MODE | ATTR_CTIME; - __ceph_setattr(inode, &newattrs); + __ceph_setattr(inode, &newattrs, NULL); } goto out_free; } diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c index 59cbfb80edbd..f4863078f7fe 100644 --- a/fs/ceph/addr.c +++ b/fs/ceph/addr.c @@ -18,6 +18,7 @@ #include "mds_client.h" #include "cache.h" #include "metric.h" +#include "crypto.h" #include #include @@ -242,11 +243,13 @@ static bool ceph_netfs_clamp_length(struct netfs_io_subrequest *subreq) static void finish_netfs_read(struct ceph_osd_request *req) { - struct ceph_fs_client *fsc = ceph_inode_to_client(req->r_inode); + struct inode *inode = req->r_inode; + struct ceph_fs_client *fsc = ceph_inode_to_client(inode); struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0); struct netfs_io_subrequest *subreq = req->r_priv; - int num_pages; + struct ceph_osd_req_op *op = &req->r_ops[0]; int err = req->r_result; + bool sparse = (op->op == CEPH_OSD_OP_SPARSE_READ); ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency, req->r_end_latency, osd_data->length, err); @@ -260,14 +263,29 @@ static void finish_netfs_read(struct ceph_osd_request *req) else if (err == -EBLOCKLISTED) fsc->blocklisted = true; - if (err >= 0 && err < subreq->len) - __set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags); + if (err >= 0) { + if (sparse && err > 0) + err = ceph_sparse_ext_map_end(op); + if (err < subreq->len) + __set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags); + if (IS_ENCRYPTED(inode) && err > 0) { + err = ceph_fscrypt_decrypt_extents(inode, + osd_data->pages, subreq->start, + op->extent.sparse_ext, + op->extent.sparse_ext_cnt); + if (err > subreq->len) + err = subreq->len; + } + } + if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) { + ceph_put_page_vector(osd_data->pages, + calc_pages_for(osd_data->alignment, + osd_data->length), false); + } netfs_subreq_terminated(subreq, err, false); - - num_pages = calc_pages_for(osd_data->alignment, osd_data->length); - ceph_put_page_vector(osd_data->pages, num_pages, false); iput(req->r_inode); + ceph_dec_osd_stopping_blocker(fsc->mdsc); } static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) @@ -334,10 +352,10 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) struct ceph_osd_request *req = NULL; struct ceph_vino vino = ceph_vino(inode); struct iov_iter iter; - struct page **pages; - size_t page_off; int err = 0; u64 len = subreq->len; + bool sparse = IS_ENCRYPTED(inode) || ceph_test_mount_opt(fsc, SPARSEREAD); + u64 off = subreq->start; if (ceph_inode_is_shutdown(inode)) { err = -EIO; @@ -347,8 +365,10 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) if (ceph_has_inline_data(ci) && ceph_netfs_issue_op_inline(subreq)) return; - req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout, vino, subreq->start, &len, - 0, 1, CEPH_OSD_OP_READ, + ceph_fscrypt_adjust_off_and_len(inode, &off, &len); + + req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout, vino, + off, &len, 0, 1, sparse ? CEPH_OSD_OP_SPARSE_READ : CEPH_OSD_OP_READ, CEPH_OSD_FLAG_READ | fsc->client->osdc.client->options->read_from_replica, NULL, ci->i_truncate_seq, ci->i_truncate_size, false); if (IS_ERR(req)) { @@ -357,20 +377,48 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) goto out; } - dout("%s: pos=%llu orig_len=%zu len=%llu\n", __func__, subreq->start, subreq->len, len); - iov_iter_xarray(&iter, ITER_DEST, &rreq->mapping->i_pages, subreq->start, len); - err = iov_iter_get_pages_alloc2(&iter, &pages, len, &page_off); - if (err < 0) { - dout("%s: iov_ter_get_pages_alloc returned %d\n", __func__, err); - goto out; + if (sparse) { + err = ceph_alloc_sparse_ext_map(&req->r_ops[0]); + if (err) + goto out; } - /* should always give us a page-aligned read */ - WARN_ON_ONCE(page_off); - len = err; - err = 0; + dout("%s: pos=%llu orig_len=%zu len=%llu\n", __func__, subreq->start, subreq->len, len); - osd_req_op_extent_osd_data_pages(req, 0, pages, len, 0, false, false); + iov_iter_xarray(&iter, ITER_DEST, &rreq->mapping->i_pages, subreq->start, len); + + /* + * FIXME: For now, use CEPH_OSD_DATA_TYPE_PAGES instead of _ITER for + * encrypted inodes. We'd need infrastructure that handles an iov_iter + * instead of page arrays, and we don't have that as of yet. Once the + * dust settles on the write helpers and encrypt/decrypt routines for + * netfs, we should be able to rework this. + */ + if (IS_ENCRYPTED(inode)) { + struct page **pages; + size_t page_off; + + err = iov_iter_get_pages_alloc2(&iter, &pages, len, &page_off); + if (err < 0) { + dout("%s: iov_ter_get_pages_alloc returned %d\n", + __func__, err); + goto out; + } + + /* should always give us a page-aligned read */ + WARN_ON_ONCE(page_off); + len = err; + err = 0; + + osd_req_op_extent_osd_data_pages(req, 0, pages, len, 0, false, + false); + } else { + osd_req_op_extent_osd_iter(req, 0, &iter); + } + if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) { + err = -EIO; + goto out; + } req->r_callback = finish_netfs_read; req->r_priv = subreq; req->r_inode = inode; @@ -571,10 +619,12 @@ static u64 get_writepages_data_length(struct inode *inode, struct page *page, u64 start) { struct ceph_inode_info *ci = ceph_inode(inode); - struct ceph_snap_context *snapc = page_snap_context(page); + struct ceph_snap_context *snapc; struct ceph_cap_snap *capsnap = NULL; u64 end = i_size_read(inode); + u64 ret; + snapc = page_snap_context(ceph_fscrypt_pagecache_page(page)); if (snapc != ci->i_head_snapc) { bool found = false; spin_lock(&ci->i_ceph_lock); @@ -589,9 +639,12 @@ static u64 get_writepages_data_length(struct inode *inode, spin_unlock(&ci->i_ceph_lock); WARN_ON(!found); } - if (end > page_offset(page) + thp_size(page)) - end = page_offset(page) + thp_size(page); - return end > start ? end - start : 0; + if (end > ceph_fscrypt_page_offset(page) + thp_size(page)) + end = ceph_fscrypt_page_offset(page) + thp_size(page); + ret = end > start ? end - start : 0; + if (ret && fscrypt_is_bounce_page(page)) + ret = round_up(ret, CEPH_FSCRYPT_BLOCK_SIZE); + return ret; } /* @@ -610,10 +663,12 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc) loff_t page_off = page_offset(page); int err; loff_t len = thp_size(page); + loff_t wlen; struct ceph_writeback_ctl ceph_wbc; struct ceph_osd_client *osdc = &fsc->client->osdc; struct ceph_osd_request *req; bool caching = ceph_is_cache_enabled(inode); + struct page *bounce_page = NULL; dout("writepage %p idx %lu\n", page, page->index); @@ -649,31 +704,51 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc) if (ceph_wbc.i_size < page_off + len) len = ceph_wbc.i_size - page_off; + wlen = IS_ENCRYPTED(inode) ? round_up(len, CEPH_FSCRYPT_BLOCK_SIZE) : len; dout("writepage %p page %p index %lu on %llu~%llu snapc %p seq %lld\n", - inode, page, page->index, page_off, len, snapc, snapc->seq); + inode, page, page->index, page_off, wlen, snapc, snapc->seq); if (atomic_long_inc_return(&fsc->writeback_count) > CONGESTION_ON_THRESH(fsc->mount_options->congestion_kb)) fsc->write_congested = true; - req = ceph_osdc_new_request(osdc, &ci->i_layout, ceph_vino(inode), page_off, &len, 0, 1, - CEPH_OSD_OP_WRITE, CEPH_OSD_FLAG_WRITE, snapc, - ceph_wbc.truncate_seq, ceph_wbc.truncate_size, - true); + req = ceph_osdc_new_request(osdc, &ci->i_layout, ceph_vino(inode), + page_off, &wlen, 0, 1, CEPH_OSD_OP_WRITE, + CEPH_OSD_FLAG_WRITE, snapc, + ceph_wbc.truncate_seq, + ceph_wbc.truncate_size, true); if (IS_ERR(req)) { redirty_page_for_writepage(wbc, page); return PTR_ERR(req); } + if (wlen < len) + len = wlen; + set_page_writeback(page); if (caching) ceph_set_page_fscache(page); ceph_fscache_write_to_cache(inode, page_off, len, caching); + if (IS_ENCRYPTED(inode)) { + bounce_page = fscrypt_encrypt_pagecache_blocks(page, + CEPH_FSCRYPT_BLOCK_SIZE, 0, + GFP_NOFS); + if (IS_ERR(bounce_page)) { + redirty_page_for_writepage(wbc, page); + end_page_writeback(page); + ceph_osdc_put_request(req); + return PTR_ERR(bounce_page); + } + } + /* it may be a short write due to an object boundary */ WARN_ON_ONCE(len > thp_size(page)); - osd_req_op_extent_osd_data_pages(req, 0, &page, len, 0, false, false); - dout("writepage %llu~%llu (%llu bytes)\n", page_off, len, len); + osd_req_op_extent_osd_data_pages(req, 0, + bounce_page ? &bounce_page : &page, wlen, 0, + false, false); + dout("writepage %llu~%llu (%llu bytes, %sencrypted)\n", + page_off, len, wlen, IS_ENCRYPTED(inode) ? "" : "not "); req->r_mtime = inode->i_mtime; ceph_osdc_start_request(osdc, req); @@ -681,7 +756,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc) ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency, req->r_end_latency, len, err); - + fscrypt_free_bounce_page(bounce_page); ceph_osdc_put_request(req); if (err == 0) err = len; @@ -800,6 +875,11 @@ static void writepages_finish(struct ceph_osd_request *req) total_pages += num_pages; for (j = 0; j < num_pages; j++) { page = osd_data->pages[j]; + if (fscrypt_is_bounce_page(page)) { + page = fscrypt_pagecache_page(page); + fscrypt_free_bounce_page(osd_data->pages[j]); + osd_data->pages[j] = page; + } BUG_ON(!page); WARN_ON(!PageUptodate(page)); @@ -835,6 +915,7 @@ static void writepages_finish(struct ceph_osd_request *req) else kfree(osd_data->pages); ceph_osdc_put_request(req); + ceph_dec_osd_stopping_blocker(fsc->mdsc); } /* @@ -1070,9 +1151,28 @@ get_more_pages: fsc->mount_options->congestion_kb)) fsc->write_congested = true; - pages[locked_pages++] = page; - fbatch.folios[i] = NULL; + if (IS_ENCRYPTED(inode)) { + pages[locked_pages] = + fscrypt_encrypt_pagecache_blocks(page, + PAGE_SIZE, 0, + locked_pages ? GFP_NOWAIT : GFP_NOFS); + if (IS_ERR(pages[locked_pages])) { + if (PTR_ERR(pages[locked_pages]) == -EINVAL) + pr_err("%s: inode->i_blkbits=%hhu\n", + __func__, inode->i_blkbits); + /* better not fail on first page! */ + BUG_ON(locked_pages == 0); + pages[locked_pages] = NULL; + redirty_page_for_writepage(wbc, page); + unlock_page(page); + break; + } + ++locked_pages; + } else { + pages[locked_pages++] = page; + } + fbatch.folios[i] = NULL; len += thp_size(page); } @@ -1100,7 +1200,7 @@ get_more_pages: } new_request: - offset = page_offset(pages[0]); + offset = ceph_fscrypt_page_offset(pages[0]); len = wsize; req = ceph_osdc_new_request(&fsc->client->osdc, @@ -1121,9 +1221,13 @@ new_request: ceph_wbc.truncate_size, true); BUG_ON(IS_ERR(req)); } - BUG_ON(len < page_offset(pages[locked_pages - 1]) + - thp_size(page) - offset); + BUG_ON(len < ceph_fscrypt_page_offset(pages[locked_pages - 1]) + + thp_size(pages[locked_pages - 1]) - offset); + if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) { + rc = -EIO; + goto release_folios; + } req->r_callback = writepages_finish; req->r_inode = inode; @@ -1132,7 +1236,9 @@ new_request: data_pages = pages; op_idx = 0; for (i = 0; i < locked_pages; i++) { - u64 cur_offset = page_offset(pages[i]); + struct page *page = ceph_fscrypt_pagecache_page(pages[i]); + + u64 cur_offset = page_offset(page); /* * Discontinuity in page range? Ceph can handle that by just passing * multiple extents in the write op. @@ -1161,9 +1267,9 @@ new_request: op_idx++; } - set_page_writeback(pages[i]); + set_page_writeback(page); if (caching) - ceph_set_page_fscache(pages[i]); + ceph_set_page_fscache(page); len += thp_size(page); } ceph_fscache_write_to_cache(inode, offset, len, caching); @@ -1179,8 +1285,16 @@ new_request: offset); len = max(len, min_len); } + if (IS_ENCRYPTED(inode)) + len = round_up(len, CEPH_FSCRYPT_BLOCK_SIZE); + dout("writepages got pages at %llu~%llu\n", offset, len); + if (IS_ENCRYPTED(inode) && + ((offset | len) & ~CEPH_FSCRYPT_BLOCK_MASK)) + pr_warn("%s: bad encrypted write offset=%lld len=%llu\n", + __func__, offset, len); + osd_req_op_extent_osd_data_pages(req, op_idx, data_pages, len, 0, from_pool, false); osd_req_op_extent_update(req, op_idx, len); diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c index 09cd6d334604..14215ec646f7 100644 --- a/fs/ceph/caps.c +++ b/fs/ceph/caps.c @@ -14,6 +14,7 @@ #include "super.h" #include "mds_client.h" #include "cache.h" +#include "crypto.h" #include #include @@ -1216,15 +1217,11 @@ struct cap_msg_args { umode_t mode; bool inline_data; bool wake; + bool encrypted; + u32 fscrypt_auth_len; + u8 fscrypt_auth[sizeof(struct ceph_fscrypt_auth)]; // for context }; -/* - * cap struct size + flock buffer size + inline version + inline data size + - * osd_epoch_barrier + oldest_flush_tid - */ -#define CAP_MSG_SIZE (sizeof(struct ceph_mds_caps) + \ - 4 + 8 + 4 + 4 + 8 + 4 + 4 + 4 + 8 + 8 + 4) - /* Marshal up the cap msg to the MDS */ static void encode_cap_msg(struct ceph_msg *msg, struct cap_msg_args *arg) { @@ -1240,7 +1237,7 @@ static void encode_cap_msg(struct ceph_msg *msg, struct cap_msg_args *arg) arg->size, arg->max_size, arg->xattr_version, arg->xattr_buf ? (int)arg->xattr_buf->vec.iov_len : 0); - msg->hdr.version = cpu_to_le16(10); + msg->hdr.version = cpu_to_le16(12); msg->hdr.tid = cpu_to_le64(arg->flush_tid); fc = msg->front.iov_base; @@ -1257,7 +1254,13 @@ static void encode_cap_msg(struct ceph_msg *msg, struct cap_msg_args *arg) fc->ino = cpu_to_le64(arg->ino); fc->snap_follows = cpu_to_le64(arg->follows); - fc->size = cpu_to_le64(arg->size); +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) + if (arg->encrypted) + fc->size = cpu_to_le64(round_up(arg->size, + CEPH_FSCRYPT_BLOCK_SIZE)); + else +#endif + fc->size = cpu_to_le64(arg->size); fc->max_size = cpu_to_le64(arg->max_size); ceph_encode_timespec64(&fc->mtime, &arg->mtime); ceph_encode_timespec64(&fc->atime, &arg->atime); @@ -1311,6 +1314,27 @@ static void encode_cap_msg(struct ceph_msg *msg, struct cap_msg_args *arg) /* Advisory flags (version 10) */ ceph_encode_32(&p, arg->flags); + + /* dirstats (version 11) - these are r/o on the client */ + ceph_encode_64(&p, 0); + ceph_encode_64(&p, 0); + +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) + /* + * fscrypt_auth and fscrypt_file (version 12) + * + * fscrypt_auth holds the crypto context (if any). fscrypt_file + * tracks the real i_size as an __le64 field (and we use a rounded-up + * i_size in the traditional size field). + */ + ceph_encode_32(&p, arg->fscrypt_auth_len); + ceph_encode_copy(&p, arg->fscrypt_auth, arg->fscrypt_auth_len); + ceph_encode_32(&p, sizeof(__le64)); + ceph_encode_64(&p, arg->size); +#else /* CONFIG_FS_ENCRYPTION */ + ceph_encode_32(&p, 0); + ceph_encode_32(&p, 0); +#endif /* CONFIG_FS_ENCRYPTION */ } /* @@ -1378,7 +1402,6 @@ static void __prep_cap(struct cap_msg_args *arg, struct ceph_cap *cap, arg->follows = flushing ? ci->i_head_snapc->seq : 0; arg->flush_tid = flush_tid; arg->oldest_flush_tid = oldest_flush_tid; - arg->size = i_size_read(inode); ci->i_reported_size = arg->size; arg->max_size = ci->i_wanted_max_size; @@ -1432,8 +1455,39 @@ static void __prep_cap(struct cap_msg_args *arg, struct ceph_cap *cap, } } arg->flags = flags; + arg->encrypted = IS_ENCRYPTED(inode); +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) + if (ci->fscrypt_auth_len && + WARN_ON_ONCE(ci->fscrypt_auth_len > sizeof(struct ceph_fscrypt_auth))) { + /* Don't set this if it's too big */ + arg->fscrypt_auth_len = 0; + } else { + arg->fscrypt_auth_len = ci->fscrypt_auth_len; + memcpy(arg->fscrypt_auth, ci->fscrypt_auth, + min_t(size_t, ci->fscrypt_auth_len, + sizeof(arg->fscrypt_auth))); + } +#endif /* CONFIG_FS_ENCRYPTION */ } +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) +#define CAP_MSG_FIXED_FIELDS (sizeof(struct ceph_mds_caps) + \ + 4 + 8 + 4 + 4 + 8 + 4 + 4 + 4 + 8 + 8 + 4 + 8 + 8 + 4 + 4 + 8) + +static inline int cap_msg_size(struct cap_msg_args *arg) +{ + return CAP_MSG_FIXED_FIELDS + arg->fscrypt_auth_len; +} +#else +#define CAP_MSG_FIXED_FIELDS (sizeof(struct ceph_mds_caps) + \ + 4 + 8 + 4 + 4 + 8 + 4 + 4 + 4 + 8 + 8 + 4 + 8 + 8 + 4 + 4) + +static inline int cap_msg_size(struct cap_msg_args *arg) +{ + return CAP_MSG_FIXED_FIELDS; +} +#endif /* CONFIG_FS_ENCRYPTION */ + /* * Send a cap msg on the given inode. * @@ -1444,7 +1498,8 @@ static void __send_cap(struct cap_msg_args *arg, struct ceph_inode_info *ci) struct ceph_msg *msg; struct inode *inode = &ci->netfs.inode; - msg = ceph_msg_new(CEPH_MSG_CLIENT_CAPS, CAP_MSG_SIZE, GFP_NOFS, false); + msg = ceph_msg_new(CEPH_MSG_CLIENT_CAPS, cap_msg_size(arg), GFP_NOFS, + false); if (!msg) { pr_err("error allocating cap msg: ino (%llx.%llx) flushing %s tid %llu, requeuing cap.\n", ceph_vinop(inode), ceph_cap_string(arg->dirty), @@ -1470,10 +1525,6 @@ static inline int __send_flush_snap(struct inode *inode, struct cap_msg_args arg; struct ceph_msg *msg; - msg = ceph_msg_new(CEPH_MSG_CLIENT_CAPS, CAP_MSG_SIZE, GFP_NOFS, false); - if (!msg) - return -ENOMEM; - arg.session = session; arg.ino = ceph_vino(inode).ino; arg.cid = 0; @@ -1510,6 +1561,15 @@ static inline int __send_flush_snap(struct inode *inode, arg.inline_data = capsnap->inline_data; arg.flags = 0; arg.wake = false; + arg.encrypted = IS_ENCRYPTED(inode); + + /* No fscrypt_auth changes from a capsnap.*/ + arg.fscrypt_auth_len = 0; + + msg = ceph_msg_new(CEPH_MSG_CLIENT_CAPS, cap_msg_size(&arg), + GFP_NOFS, false); + if (!msg) + return -ENOMEM; encode_cap_msg(msg, &arg); ceph_con_send(&arg.session->s_con, msg); @@ -2900,10 +2960,9 @@ int ceph_try_get_caps(struct inode *inode, int need, int want, * due to a small max_size, make sure we check_max_size (and possibly * ask the mds) so we don't get hung up indefinitely. */ -int ceph_get_caps(struct file *filp, int need, int want, loff_t endoff, int *got) +int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need, + int want, loff_t endoff, int *got) { - struct ceph_file_info *fi = filp->private_data; - struct inode *inode = file_inode(filp); struct ceph_inode_info *ci = ceph_inode(inode); struct ceph_fs_client *fsc = ceph_inode_to_client(inode); int ret, _got, flags; @@ -2912,7 +2971,7 @@ int ceph_get_caps(struct file *filp, int need, int want, loff_t endoff, int *got if (ret < 0) return ret; - if ((fi->fmode & CEPH_FILE_MODE_WR) && + if (fi && (fi->fmode & CEPH_FILE_MODE_WR) && fi->filp_gen != READ_ONCE(fsc->filp_gen)) return -EBADF; @@ -2965,7 +3024,7 @@ int ceph_get_caps(struct file *filp, int need, int want, loff_t endoff, int *got continue; } - if ((fi->fmode & CEPH_FILE_MODE_WR) && + if (fi && (fi->fmode & CEPH_FILE_MODE_WR) && fi->filp_gen != READ_ONCE(fsc->filp_gen)) { if (ret >= 0 && _got) ceph_put_cap_refs(ci, _got); @@ -3028,6 +3087,15 @@ int ceph_get_caps(struct file *filp, int need, int want, loff_t endoff, int *got return 0; } +int ceph_get_caps(struct file *filp, int need, int want, loff_t endoff, + int *got) +{ + struct ceph_file_info *fi = filp->private_data; + struct inode *inode = file_inode(filp); + + return __ceph_get_caps(inode, fi, need, want, endoff, got); +} + /* * Take cap refs. Caller must already know we hold at least one ref * on the caps in question or we don't know this is safe. @@ -3323,6 +3391,9 @@ struct cap_extra_info { /* currently issued */ int issued; struct timespec64 btime; + u8 *fscrypt_auth; + u32 fscrypt_auth_len; + u64 fscrypt_file_size; }; /* @@ -3355,6 +3426,14 @@ static void handle_cap_grant(struct inode *inode, bool deleted_inode = false; bool fill_inline = false; + /* + * If there is at least one crypto block then we'll trust + * fscrypt_file_size. If the real length of the file is 0, then + * ignore it (it has probably been truncated down to 0 by the MDS). + */ + if (IS_ENCRYPTED(inode) && size) + size = extra_info->fscrypt_file_size; + dout("handle_cap_grant inode %p cap %p mds%d seq %d %s\n", inode, cap, session->s_mds, seq, ceph_cap_string(newcaps)); dout(" size %llu max_size %llu, i_size %llu\n", size, max_size, @@ -3421,6 +3500,14 @@ static void handle_cap_grant(struct inode *inode, dout("%p mode 0%o uid.gid %d.%d\n", inode, inode->i_mode, from_kuid(&init_user_ns, inode->i_uid), from_kgid(&init_user_ns, inode->i_gid)); +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) + if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || + memcmp(ci->fscrypt_auth, extra_info->fscrypt_auth, + ci->fscrypt_auth_len)) + pr_warn_ratelimited("%s: cap grant attempt to change fscrypt_auth on non-I_NEW inode (old len %d new len %d)\n", + __func__, ci->fscrypt_auth_len, + extra_info->fscrypt_auth_len); +#endif } if ((newcaps & CEPH_CAP_LINK_SHARED) && @@ -3837,7 +3924,8 @@ static void handle_cap_flushsnap_ack(struct inode *inode, u64 flush_tid, */ static bool handle_cap_trunc(struct inode *inode, struct ceph_mds_caps *trunc, - struct ceph_mds_session *session) + struct ceph_mds_session *session, + struct cap_extra_info *extra_info) { struct ceph_inode_info *ci = ceph_inode(inode); int mds = session->s_mds; @@ -3854,8 +3942,16 @@ static bool handle_cap_trunc(struct inode *inode, issued |= implemented | dirty; - dout("handle_cap_trunc inode %p mds%d seq %d to %lld seq %d\n", - inode, mds, seq, truncate_size, truncate_seq); + /* + * If there is at least one crypto block then we'll trust + * fscrypt_file_size. If the real length of the file is 0, then + * ignore it (it has probably been truncated down to 0 by the MDS). + */ + if (IS_ENCRYPTED(inode) && size) + size = extra_info->fscrypt_file_size; + + dout("%s inode %p mds%d seq %d to %lld truncate seq %d\n", + __func__, inode, mds, seq, truncate_size, truncate_seq); queue_trunc = ceph_fill_file_size(inode, issued, truncate_seq, truncate_size, size); return queue_trunc; @@ -4075,6 +4171,52 @@ retry: *target_cap = cap; } +#ifdef CONFIG_FS_ENCRYPTION +static int parse_fscrypt_fields(void **p, void *end, + struct cap_extra_info *extra) +{ + u32 len; + + ceph_decode_32_safe(p, end, extra->fscrypt_auth_len, bad); + if (extra->fscrypt_auth_len) { + ceph_decode_need(p, end, extra->fscrypt_auth_len, bad); + extra->fscrypt_auth = kmalloc(extra->fscrypt_auth_len, + GFP_KERNEL); + if (!extra->fscrypt_auth) + return -ENOMEM; + ceph_decode_copy_safe(p, end, extra->fscrypt_auth, + extra->fscrypt_auth_len, bad); + } + + ceph_decode_32_safe(p, end, len, bad); + if (len >= sizeof(u64)) { + ceph_decode_64_safe(p, end, extra->fscrypt_file_size, bad); + len -= sizeof(u64); + } + ceph_decode_skip_n(p, end, len, bad); + return 0; +bad: + return -EIO; +} +#else +static int parse_fscrypt_fields(void **p, void *end, + struct cap_extra_info *extra) +{ + u32 len; + + /* Don't care about these fields unless we're encryption-capable */ + ceph_decode_32_safe(p, end, len, bad); + if (len) + ceph_decode_skip_n(p, end, len, bad); + ceph_decode_32_safe(p, end, len, bad); + if (len) + ceph_decode_skip_n(p, end, len, bad); + return 0; +bad: + return -EIO; +} +#endif + /* * Handle a caps message from the MDS. * @@ -4105,6 +4247,9 @@ void ceph_handle_caps(struct ceph_mds_session *session, dout("handle_caps from mds%d\n", session->s_mds); + if (!ceph_inc_mds_stopping_blocker(mdsc, session)) + return; + /* decode */ end = msg->front.iov_base + msg->front.iov_len; if (msg->front.iov_len < sizeof(*h)) @@ -4195,13 +4340,17 @@ void ceph_handle_caps(struct ceph_mds_session *session, ceph_decode_64_safe(&p, end, extra_info.nsubdirs, bad); } + if (msg_version >= 12) { + if (parse_fscrypt_fields(&p, end, &extra_info)) + goto bad; + } + /* lookup ino */ inode = ceph_find_inode(mdsc->fsc->sb, vino); dout(" op %s ino %llx.%llx inode %p\n", ceph_cap_op_name(op), vino.ino, vino.snap, inode); mutex_lock(&session->s_mutex); - inc_session_sequence(session); dout(" mds%d seq %lld cap seq %u\n", session->s_mds, session->s_seq, (unsigned)seq); @@ -4292,7 +4441,8 @@ void ceph_handle_caps(struct ceph_mds_session *session, break; case CEPH_CAP_OP_TRUNC: - queue_trunc = handle_cap_trunc(inode, h, session); + queue_trunc = handle_cap_trunc(inode, h, session, + &extra_info); spin_unlock(&ci->i_ceph_lock); if (queue_trunc) ceph_queue_vmtruncate(inode); @@ -4309,12 +4459,15 @@ done: done_unlocked: iput(inode); out: + ceph_dec_mds_stopping_blocker(mdsc); + ceph_put_string(extra_info.pool_ns); /* Defer closing the sessions after s_mutex lock being released */ if (close_sessions) ceph_mdsc_close_sessions(mdsc); + kfree(extra_info.fscrypt_auth); return; flush_cap_releases: @@ -4611,6 +4764,18 @@ int ceph_encode_inode_release(void **p, struct inode *inode, return ret; } +/** + * ceph_encode_dentry_release - encode a dentry release into an outgoing request + * @p: outgoing request buffer + * @dentry: dentry to release + * @dir: dir to release it from + * @mds: mds that we're speaking to + * @drop: caps being dropped + * @unless: unless we have these caps + * + * Encode a dentry release into an outgoing request buffer. Returns 1 if the + * thing was released, or a negative error code otherwise. + */ int ceph_encode_dentry_release(void **p, struct dentry *dentry, struct inode *dir, int mds, int drop, int unless) @@ -4643,13 +4808,25 @@ int ceph_encode_dentry_release(void **p, struct dentry *dentry, if (ret && di->lease_session && di->lease_session->s_mds == mds) { dout("encode_dentry_release %p mds%d seq %d\n", dentry, mds, (int)di->lease_seq); - rel->dname_len = cpu_to_le32(dentry->d_name.len); - memcpy(*p, dentry->d_name.name, dentry->d_name.len); - *p += dentry->d_name.len; rel->dname_seq = cpu_to_le32(di->lease_seq); __ceph_mdsc_drop_dentry_lease(dentry); + spin_unlock(&dentry->d_lock); + if (IS_ENCRYPTED(dir) && fscrypt_has_encryption_key(dir)) { + int ret2 = ceph_encode_encrypted_fname(dir, dentry, *p); + + if (ret2 < 0) + return ret2; + + rel->dname_len = cpu_to_le32(ret2); + *p += ret2; + } else { + rel->dname_len = cpu_to_le32(dentry->d_name.len); + memcpy(*p, dentry->d_name.name, dentry->d_name.len); + *p += dentry->d_name.len; + } + } else { + spin_unlock(&dentry->d_lock); } - spin_unlock(&dentry->d_lock); return ret; } diff --git a/fs/ceph/crypto.c b/fs/ceph/crypto.c new file mode 100644 index 000000000000..e4d5cd56a80b --- /dev/null +++ b/fs/ceph/crypto.c @@ -0,0 +1,673 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * The base64 encode/decode code was copied from fscrypt: + * Copyright (C) 2015, Google, Inc. + * Copyright (C) 2015, Motorola Mobility + * Written by Uday Savagaonkar, 2014. + * Modified by Jaegeuk Kim, 2015. + */ +#include +#include +#include +#include + +#include "super.h" +#include "mds_client.h" +#include "crypto.h" + +/* + * The base64url encoding used by fscrypt includes the '_' character, which may + * cause problems in snapshot names (which can not start with '_'). Thus, we + * used the base64 encoding defined for IMAP mailbox names (RFC 3501) instead, + * which replaces '-' and '_' by '+' and ','. + */ +static const char base64_table[65] = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+,"; + +int ceph_base64_encode(const u8 *src, int srclen, char *dst) +{ + u32 ac = 0; + int bits = 0; + int i; + char *cp = dst; + + for (i = 0; i < srclen; i++) { + ac = (ac << 8) | src[i]; + bits += 8; + do { + bits -= 6; + *cp++ = base64_table[(ac >> bits) & 0x3f]; + } while (bits >= 6); + } + if (bits) + *cp++ = base64_table[(ac << (6 - bits)) & 0x3f]; + return cp - dst; +} + +int ceph_base64_decode(const char *src, int srclen, u8 *dst) +{ + u32 ac = 0; + int bits = 0; + int i; + u8 *bp = dst; + + for (i = 0; i < srclen; i++) { + const char *p = strchr(base64_table, src[i]); + + if (p == NULL || src[i] == 0) + return -1; + ac = (ac << 6) | (p - base64_table); + bits += 6; + if (bits >= 8) { + bits -= 8; + *bp++ = (u8)(ac >> bits); + } + } + if (ac & ((1 << bits) - 1)) + return -1; + return bp - dst; +} + +static int ceph_crypt_get_context(struct inode *inode, void *ctx, size_t len) +{ + struct ceph_inode_info *ci = ceph_inode(inode); + struct ceph_fscrypt_auth *cfa = (struct ceph_fscrypt_auth *)ci->fscrypt_auth; + u32 ctxlen; + + /* Non existent or too short? */ + if (!cfa || (ci->fscrypt_auth_len < (offsetof(struct ceph_fscrypt_auth, cfa_blob) + 1))) + return -ENOBUFS; + + /* Some format we don't recognize? */ + if (le32_to_cpu(cfa->cfa_version) != CEPH_FSCRYPT_AUTH_VERSION) + return -ENOBUFS; + + ctxlen = le32_to_cpu(cfa->cfa_blob_len); + if (len < ctxlen) + return -ERANGE; + + memcpy(ctx, cfa->cfa_blob, ctxlen); + return ctxlen; +} + +static int ceph_crypt_set_context(struct inode *inode, const void *ctx, + size_t len, void *fs_data) +{ + int ret; + struct iattr attr = { }; + struct ceph_iattr cia = { }; + struct ceph_fscrypt_auth *cfa; + + WARN_ON_ONCE(fs_data); + + if (len > FSCRYPT_SET_CONTEXT_MAX_SIZE) + return -EINVAL; + + cfa = kzalloc(sizeof(*cfa), GFP_KERNEL); + if (!cfa) + return -ENOMEM; + + cfa->cfa_version = cpu_to_le32(CEPH_FSCRYPT_AUTH_VERSION); + cfa->cfa_blob_len = cpu_to_le32(len); + memcpy(cfa->cfa_blob, ctx, len); + + cia.fscrypt_auth = cfa; + + ret = __ceph_setattr(inode, &attr, &cia); + if (ret == 0) + inode_set_flags(inode, S_ENCRYPTED, S_ENCRYPTED); + kfree(cia.fscrypt_auth); + return ret; +} + +static bool ceph_crypt_empty_dir(struct inode *inode) +{ + struct ceph_inode_info *ci = ceph_inode(inode); + + return ci->i_rsubdirs + ci->i_rfiles == 1; +} + +static const union fscrypt_policy *ceph_get_dummy_policy(struct super_block *sb) +{ + return ceph_sb_to_client(sb)->fsc_dummy_enc_policy.policy; +} + +static struct fscrypt_operations ceph_fscrypt_ops = { + .get_context = ceph_crypt_get_context, + .set_context = ceph_crypt_set_context, + .get_dummy_policy = ceph_get_dummy_policy, + .empty_dir = ceph_crypt_empty_dir, +}; + +void ceph_fscrypt_set_ops(struct super_block *sb) +{ + fscrypt_set_ops(sb, &ceph_fscrypt_ops); +} + +void ceph_fscrypt_free_dummy_policy(struct ceph_fs_client *fsc) +{ + fscrypt_free_dummy_policy(&fsc->fsc_dummy_enc_policy); +} + +int ceph_fscrypt_prepare_context(struct inode *dir, struct inode *inode, + struct ceph_acl_sec_ctx *as) +{ + int ret, ctxsize; + bool encrypted = false; + struct ceph_inode_info *ci = ceph_inode(inode); + + ret = fscrypt_prepare_new_inode(dir, inode, &encrypted); + if (ret) + return ret; + if (!encrypted) + return 0; + + as->fscrypt_auth = kzalloc(sizeof(*as->fscrypt_auth), GFP_KERNEL); + if (!as->fscrypt_auth) + return -ENOMEM; + + ctxsize = fscrypt_context_for_new_inode(as->fscrypt_auth->cfa_blob, + inode); + if (ctxsize < 0) + return ctxsize; + + as->fscrypt_auth->cfa_version = cpu_to_le32(CEPH_FSCRYPT_AUTH_VERSION); + as->fscrypt_auth->cfa_blob_len = cpu_to_le32(ctxsize); + + WARN_ON_ONCE(ci->fscrypt_auth); + kfree(ci->fscrypt_auth); + ci->fscrypt_auth_len = ceph_fscrypt_auth_len(as->fscrypt_auth); + ci->fscrypt_auth = kmemdup(as->fscrypt_auth, ci->fscrypt_auth_len, + GFP_KERNEL); + if (!ci->fscrypt_auth) + return -ENOMEM; + + inode->i_flags |= S_ENCRYPTED; + + return 0; +} + +void ceph_fscrypt_as_ctx_to_req(struct ceph_mds_request *req, + struct ceph_acl_sec_ctx *as) +{ + swap(req->r_fscrypt_auth, as->fscrypt_auth); +} + +/* + * User-created snapshots can't start with '_'. Snapshots that start with this + * character are special (hint: there aren't real snapshots) and use the + * following format: + * + * __ + * + * where: + * - - the real snapshot name that may need to be decrypted, + * - - the inode number (in decimal) for the actual snapshot + * + * This function parses these snapshot names and returns the inode + * . 'name_len' will also bet set with the + * length. + */ +static struct inode *parse_longname(const struct inode *parent, + const char *name, int *name_len) +{ + struct inode *dir = NULL; + struct ceph_vino vino = { .snap = CEPH_NOSNAP }; + char *inode_number; + char *name_end; + int orig_len = *name_len; + int ret = -EIO; + + /* Skip initial '_' */ + name++; + name_end = strrchr(name, '_'); + if (!name_end) { + dout("Failed to parse long snapshot name: %s\n", name); + return ERR_PTR(-EIO); + } + *name_len = (name_end - name); + if (*name_len <= 0) { + pr_err("Failed to parse long snapshot name\n"); + return ERR_PTR(-EIO); + } + + /* Get the inode number */ + inode_number = kmemdup_nul(name_end + 1, + orig_len - *name_len - 2, + GFP_KERNEL); + if (!inode_number) + return ERR_PTR(-ENOMEM); + ret = kstrtou64(inode_number, 10, &vino.ino); + if (ret) { + dout("Failed to parse inode number: %s\n", name); + dir = ERR_PTR(ret); + goto out; + } + + /* And finally the inode */ + dir = ceph_find_inode(parent->i_sb, vino); + if (!dir) { + /* This can happen if we're not mounting cephfs on the root */ + dir = ceph_get_inode(parent->i_sb, vino, NULL); + if (!dir) + dir = ERR_PTR(-ENOENT); + } + if (IS_ERR(dir)) + dout("Can't find inode %s (%s)\n", inode_number, name); + +out: + kfree(inode_number); + return dir; +} + +int ceph_encode_encrypted_dname(struct inode *parent, struct qstr *d_name, + char *buf) +{ + struct inode *dir = parent; + struct qstr iname; + u32 len; + int name_len; + int elen; + int ret; + u8 *cryptbuf = NULL; + + iname.name = d_name->name; + name_len = d_name->len; + + /* Handle the special case of snapshot names that start with '_' */ + if ((ceph_snap(dir) == CEPH_SNAPDIR) && (name_len > 0) && + (iname.name[0] == '_')) { + dir = parse_longname(parent, iname.name, &name_len); + if (IS_ERR(dir)) + return PTR_ERR(dir); + iname.name++; /* skip initial '_' */ + } + iname.len = name_len; + + if (!fscrypt_has_encryption_key(dir)) { + memcpy(buf, d_name->name, d_name->len); + elen = d_name->len; + goto out; + } + + /* + * Convert cleartext d_name to ciphertext. If result is longer than + * CEPH_NOHASH_NAME_MAX, sha256 the remaining bytes + * + * See: fscrypt_setup_filename + */ + if (!fscrypt_fname_encrypted_size(dir, iname.len, NAME_MAX, &len)) { + elen = -ENAMETOOLONG; + goto out; + } + + /* Allocate a buffer appropriate to hold the result */ + cryptbuf = kmalloc(len > CEPH_NOHASH_NAME_MAX ? NAME_MAX : len, + GFP_KERNEL); + if (!cryptbuf) { + elen = -ENOMEM; + goto out; + } + + ret = fscrypt_fname_encrypt(dir, &iname, cryptbuf, len); + if (ret) { + elen = ret; + goto out; + } + + /* hash the end if the name is long enough */ + if (len > CEPH_NOHASH_NAME_MAX) { + u8 hash[SHA256_DIGEST_SIZE]; + u8 *extra = cryptbuf + CEPH_NOHASH_NAME_MAX; + + /* + * hash the extra bytes and overwrite crypttext beyond that + * point with it + */ + sha256(extra, len - CEPH_NOHASH_NAME_MAX, hash); + memcpy(extra, hash, SHA256_DIGEST_SIZE); + len = CEPH_NOHASH_NAME_MAX + SHA256_DIGEST_SIZE; + } + + /* base64 encode the encrypted name */ + elen = ceph_base64_encode(cryptbuf, len, buf); + dout("base64-encoded ciphertext name = %.*s\n", elen, buf); + + /* To understand the 240 limit, see CEPH_NOHASH_NAME_MAX comments */ + WARN_ON(elen > 240); + if ((elen > 0) && (dir != parent)) { + char tmp_buf[NAME_MAX]; + + elen = snprintf(tmp_buf, sizeof(tmp_buf), "_%.*s_%ld", + elen, buf, dir->i_ino); + memcpy(buf, tmp_buf, elen); + } + +out: + kfree(cryptbuf); + if (dir != parent) { + if ((dir->i_state & I_NEW)) + discard_new_inode(dir); + else + iput(dir); + } + return elen; +} + +int ceph_encode_encrypted_fname(struct inode *parent, struct dentry *dentry, + char *buf) +{ + WARN_ON_ONCE(!fscrypt_has_encryption_key(parent)); + + return ceph_encode_encrypted_dname(parent, &dentry->d_name, buf); +} + +/** + * ceph_fname_to_usr - convert a filename for userland presentation + * @fname: ceph_fname to be converted + * @tname: temporary name buffer to use for conversion (may be NULL) + * @oname: where converted name should be placed + * @is_nokey: set to true if key wasn't available during conversion (may be NULL) + * + * Given a filename (usually from the MDS), format it for presentation to + * userland. If @parent is not encrypted, just pass it back as-is. + * + * Otherwise, base64 decode the string, and then ask fscrypt to format it + * for userland presentation. + * + * Returns 0 on success or negative error code on error. + */ +int ceph_fname_to_usr(const struct ceph_fname *fname, struct fscrypt_str *tname, + struct fscrypt_str *oname, bool *is_nokey) +{ + struct inode *dir = fname->dir; + struct fscrypt_str _tname = FSTR_INIT(NULL, 0); + struct fscrypt_str iname; + char *name = fname->name; + int name_len = fname->name_len; + int ret; + + /* Sanity check that the resulting name will fit in the buffer */ + if (fname->name_len > NAME_MAX || fname->ctext_len > NAME_MAX) + return -EIO; + + /* Handle the special case of snapshot names that start with '_' */ + if ((ceph_snap(dir) == CEPH_SNAPDIR) && (name_len > 0) && + (name[0] == '_')) { + dir = parse_longname(dir, name, &name_len); + if (IS_ERR(dir)) + return PTR_ERR(dir); + name++; /* skip initial '_' */ + } + + if (!IS_ENCRYPTED(dir)) { + oname->name = fname->name; + oname->len = fname->name_len; + ret = 0; + goto out_inode; + } + + ret = ceph_fscrypt_prepare_readdir(dir); + if (ret) + goto out_inode; + + /* + * Use the raw dentry name as sent by the MDS instead of + * generating a nokey name via fscrypt. + */ + if (!fscrypt_has_encryption_key(dir)) { + if (fname->no_copy) + oname->name = fname->name; + else + memcpy(oname->name, fname->name, fname->name_len); + oname->len = fname->name_len; + if (is_nokey) + *is_nokey = true; + ret = 0; + goto out_inode; + } + + if (fname->ctext_len == 0) { + int declen; + + if (!tname) { + ret = fscrypt_fname_alloc_buffer(NAME_MAX, &_tname); + if (ret) + goto out_inode; + tname = &_tname; + } + + declen = ceph_base64_decode(name, name_len, tname->name); + if (declen <= 0) { + ret = -EIO; + goto out; + } + iname.name = tname->name; + iname.len = declen; + } else { + iname.name = fname->ctext; + iname.len = fname->ctext_len; + } + + ret = fscrypt_fname_disk_to_usr(dir, 0, 0, &iname, oname); + if (!ret && (dir != fname->dir)) { + char tmp_buf[CEPH_BASE64_CHARS(NAME_MAX)]; + + name_len = snprintf(tmp_buf, sizeof(tmp_buf), "_%.*s_%ld", + oname->len, oname->name, dir->i_ino); + memcpy(oname->name, tmp_buf, name_len); + oname->len = name_len; + } + +out: + fscrypt_fname_free_buffer(&_tname); +out_inode: + if ((dir != fname->dir) && !IS_ERR(dir)) { + if ((dir->i_state & I_NEW)) + discard_new_inode(dir); + else + iput(dir); + } + return ret; +} + +/** + * ceph_fscrypt_prepare_readdir - simple __fscrypt_prepare_readdir() wrapper + * @dir: directory inode for readdir prep + * + * Simple wrapper around __fscrypt_prepare_readdir() that will mark directory as + * non-complete if this call results in having the directory unlocked. + * + * Returns: + * 1 - if directory was locked and key is now loaded (i.e. dir is unlocked) + * 0 - if directory is still locked + * < 0 - if __fscrypt_prepare_readdir() fails + */ +int ceph_fscrypt_prepare_readdir(struct inode *dir) +{ + bool had_key = fscrypt_has_encryption_key(dir); + int err; + + if (!IS_ENCRYPTED(dir)) + return 0; + + err = __fscrypt_prepare_readdir(dir); + if (err) + return err; + if (!had_key && fscrypt_has_encryption_key(dir)) { + /* directory just got unlocked, mark it as not complete */ + ceph_dir_clear_complete(dir); + return 1; + } + return 0; +} + +int ceph_fscrypt_decrypt_block_inplace(const struct inode *inode, + struct page *page, unsigned int len, + unsigned int offs, u64 lblk_num) +{ + dout("%s: len %u offs %u blk %llu\n", __func__, len, offs, lblk_num); + return fscrypt_decrypt_block_inplace(inode, page, len, offs, lblk_num); +} + +int ceph_fscrypt_encrypt_block_inplace(const struct inode *inode, + struct page *page, unsigned int len, + unsigned int offs, u64 lblk_num, + gfp_t gfp_flags) +{ + dout("%s: len %u offs %u blk %llu\n", __func__, len, offs, lblk_num); + return fscrypt_encrypt_block_inplace(inode, page, len, offs, lblk_num, + gfp_flags); +} + +/** + * ceph_fscrypt_decrypt_pages - decrypt an array of pages + * @inode: pointer to inode associated with these pages + * @page: pointer to page array + * @off: offset into the file that the read data starts + * @len: max length to decrypt + * + * Decrypt an array of fscrypt'ed pages and return the amount of + * data decrypted. Any data in the page prior to the start of the + * first complete block in the read is ignored. Any incomplete + * crypto blocks at the end of the array are ignored (and should + * probably be zeroed by the caller). + * + * Returns the length of the decrypted data or a negative errno. + */ +int ceph_fscrypt_decrypt_pages(struct inode *inode, struct page **page, + u64 off, int len) +{ + int i, num_blocks; + u64 baseblk = off >> CEPH_FSCRYPT_BLOCK_SHIFT; + int ret = 0; + + /* + * We can't deal with partial blocks on an encrypted file, so mask off + * the last bit. + */ + num_blocks = ceph_fscrypt_blocks(off, len & CEPH_FSCRYPT_BLOCK_MASK); + + /* Decrypt each block */ + for (i = 0; i < num_blocks; ++i) { + int blkoff = i << CEPH_FSCRYPT_BLOCK_SHIFT; + int pgidx = blkoff >> PAGE_SHIFT; + unsigned int pgoffs = offset_in_page(blkoff); + int fret; + + fret = ceph_fscrypt_decrypt_block_inplace(inode, page[pgidx], + CEPH_FSCRYPT_BLOCK_SIZE, pgoffs, + baseblk + i); + if (fret < 0) { + if (ret == 0) + ret = fret; + break; + } + ret += CEPH_FSCRYPT_BLOCK_SIZE; + } + return ret; +} + +/** + * ceph_fscrypt_decrypt_extents: decrypt received extents in given buffer + * @inode: inode associated with pages being decrypted + * @page: pointer to page array + * @off: offset into the file that the data in page[0] starts + * @map: pointer to extent array + * @ext_cnt: length of extent array + * + * Given an extent map and a page array, decrypt the received data in-place, + * skipping holes. Returns the offset into buffer of end of last decrypted + * block. + */ +int ceph_fscrypt_decrypt_extents(struct inode *inode, struct page **page, + u64 off, struct ceph_sparse_extent *map, + u32 ext_cnt) +{ + int i, ret = 0; + struct ceph_inode_info *ci = ceph_inode(inode); + u64 objno, objoff; + u32 xlen; + + /* Nothing to do for empty array */ + if (ext_cnt == 0) { + dout("%s: empty array, ret 0\n", __func__); + return 0; + } + + ceph_calc_file_object_mapping(&ci->i_layout, off, map[0].len, + &objno, &objoff, &xlen); + + for (i = 0; i < ext_cnt; ++i) { + struct ceph_sparse_extent *ext = &map[i]; + int pgsoff = ext->off - objoff; + int pgidx = pgsoff >> PAGE_SHIFT; + int fret; + + if ((ext->off | ext->len) & ~CEPH_FSCRYPT_BLOCK_MASK) { + pr_warn("%s: bad encrypted sparse extent idx %d off %llx len %llx\n", + __func__, i, ext->off, ext->len); + return -EIO; + } + fret = ceph_fscrypt_decrypt_pages(inode, &page[pgidx], + off + pgsoff, ext->len); + dout("%s: [%d] 0x%llx~0x%llx fret %d\n", __func__, i, + ext->off, ext->len, fret); + if (fret < 0) { + if (ret == 0) + ret = fret; + break; + } + ret = pgsoff + fret; + } + dout("%s: ret %d\n", __func__, ret); + return ret; +} + +/** + * ceph_fscrypt_encrypt_pages - encrypt an array of pages + * @inode: pointer to inode associated with these pages + * @page: pointer to page array + * @off: offset into the file that the data starts + * @len: max length to encrypt + * @gfp: gfp flags to use for allocation + * + * Decrypt an array of cleartext pages and return the amount of + * data encrypted. Any data in the page prior to the start of the + * first complete block in the read is ignored. Any incomplete + * crypto blocks at the end of the array are ignored. + * + * Returns the length of the encrypted data or a negative errno. + */ +int ceph_fscrypt_encrypt_pages(struct inode *inode, struct page **page, u64 off, + int len, gfp_t gfp) +{ + int i, num_blocks; + u64 baseblk = off >> CEPH_FSCRYPT_BLOCK_SHIFT; + int ret = 0; + + /* + * We can't deal with partial blocks on an encrypted file, so mask off + * the last bit. + */ + num_blocks = ceph_fscrypt_blocks(off, len & CEPH_FSCRYPT_BLOCK_MASK); + + /* Encrypt each block */ + for (i = 0; i < num_blocks; ++i) { + int blkoff = i << CEPH_FSCRYPT_BLOCK_SHIFT; + int pgidx = blkoff >> PAGE_SHIFT; + unsigned int pgoffs = offset_in_page(blkoff); + int fret; + + fret = ceph_fscrypt_encrypt_block_inplace(inode, page[pgidx], + CEPH_FSCRYPT_BLOCK_SIZE, pgoffs, + baseblk + i, gfp); + if (fret < 0) { + if (ret == 0) + ret = fret; + break; + } + ret += CEPH_FSCRYPT_BLOCK_SIZE; + } + return ret; +} diff --git a/fs/ceph/crypto.h b/fs/ceph/crypto.h new file mode 100644 index 000000000000..47e0c319fc68 --- /dev/null +++ b/fs/ceph/crypto.h @@ -0,0 +1,288 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Ceph fscrypt functionality + */ + +#ifndef _CEPH_CRYPTO_H +#define _CEPH_CRYPTO_H + +#include +#include + +#define CEPH_FSCRYPT_BLOCK_SHIFT 12 +#define CEPH_FSCRYPT_BLOCK_SIZE (_AC(1, UL) << CEPH_FSCRYPT_BLOCK_SHIFT) +#define CEPH_FSCRYPT_BLOCK_MASK (~(CEPH_FSCRYPT_BLOCK_SIZE-1)) + +struct ceph_fs_client; +struct ceph_acl_sec_ctx; +struct ceph_mds_request; + +struct ceph_fname { + struct inode *dir; + char *name; // b64 encoded, possibly hashed + unsigned char *ctext; // binary crypttext (if any) + u32 name_len; // length of name buffer + u32 ctext_len; // length of crypttext + bool no_copy; +}; + +/* + * Header for the crypted file when truncating the size, this + * will be sent to MDS, and the MDS will update the encrypted + * last block and then truncate the size. + */ +struct ceph_fscrypt_truncate_size_header { + __u8 ver; + __u8 compat; + + /* + * It will be sizeof(assert_ver + file_offset + block_size) + * if the last block is empty when it's located in a file + * hole. Or the data_len will plus CEPH_FSCRYPT_BLOCK_SIZE. + */ + __le32 data_len; + + __le64 change_attr; + __le64 file_offset; + __le32 block_size; +} __packed; + +struct ceph_fscrypt_auth { + __le32 cfa_version; + __le32 cfa_blob_len; + u8 cfa_blob[FSCRYPT_SET_CONTEXT_MAX_SIZE]; +} __packed; + +#define CEPH_FSCRYPT_AUTH_VERSION 1 +static inline u32 ceph_fscrypt_auth_len(struct ceph_fscrypt_auth *fa) +{ + u32 ctxsize = le32_to_cpu(fa->cfa_blob_len); + + return offsetof(struct ceph_fscrypt_auth, cfa_blob) + ctxsize; +} + +#ifdef CONFIG_FS_ENCRYPTION +/* + * We want to encrypt filenames when creating them, but the encrypted + * versions of those names may have illegal characters in them. To mitigate + * that, we base64 encode them, but that gives us a result that can exceed + * NAME_MAX. + * + * Follow a similar scheme to fscrypt itself, and cap the filename to a + * smaller size. If the ciphertext name is longer than the value below, then + * sha256 hash the remaining bytes. + * + * For the fscrypt_nokey_name struct the dirhash[2] member is useless in ceph + * so the corresponding struct will be: + * + * struct fscrypt_ceph_nokey_name { + * u8 bytes[157]; + * u8 sha256[SHA256_DIGEST_SIZE]; + * }; // 180 bytes => 240 bytes base64-encoded, which is <= NAME_MAX (255) + * + * (240 bytes is the maximum size allowed for snapshot names to take into + * account the format: '__'.) + * + * Note that for long names that end up having their tail portion hashed, we + * must also store the full encrypted name (in the dentry's alternate_name + * field). + */ +#define CEPH_NOHASH_NAME_MAX (180 - SHA256_DIGEST_SIZE) + +#define CEPH_BASE64_CHARS(nbytes) DIV_ROUND_UP((nbytes) * 4, 3) + +int ceph_base64_encode(const u8 *src, int srclen, char *dst); +int ceph_base64_decode(const char *src, int srclen, u8 *dst); + +void ceph_fscrypt_set_ops(struct super_block *sb); + +void ceph_fscrypt_free_dummy_policy(struct ceph_fs_client *fsc); + +int ceph_fscrypt_prepare_context(struct inode *dir, struct inode *inode, + struct ceph_acl_sec_ctx *as); +void ceph_fscrypt_as_ctx_to_req(struct ceph_mds_request *req, + struct ceph_acl_sec_ctx *as); +int ceph_encode_encrypted_dname(struct inode *parent, struct qstr *d_name, + char *buf); +int ceph_encode_encrypted_fname(struct inode *parent, struct dentry *dentry, + char *buf); + +static inline int ceph_fname_alloc_buffer(struct inode *parent, + struct fscrypt_str *fname) +{ + if (!IS_ENCRYPTED(parent)) + return 0; + return fscrypt_fname_alloc_buffer(NAME_MAX, fname); +} + +static inline void ceph_fname_free_buffer(struct inode *parent, + struct fscrypt_str *fname) +{ + if (IS_ENCRYPTED(parent)) + fscrypt_fname_free_buffer(fname); +} + +int ceph_fname_to_usr(const struct ceph_fname *fname, struct fscrypt_str *tname, + struct fscrypt_str *oname, bool *is_nokey); +int ceph_fscrypt_prepare_readdir(struct inode *dir); + +static inline unsigned int ceph_fscrypt_blocks(u64 off, u64 len) +{ + /* crypto blocks cannot span more than one page */ + BUILD_BUG_ON(CEPH_FSCRYPT_BLOCK_SHIFT > PAGE_SHIFT); + + return ((off+len+CEPH_FSCRYPT_BLOCK_SIZE-1) >> CEPH_FSCRYPT_BLOCK_SHIFT) - + (off >> CEPH_FSCRYPT_BLOCK_SHIFT); +} + +/* + * If we have an encrypted inode then we must adjust the offset and + * range of the on-the-wire read to cover an entire encryption block. + * The copy will be done using the original offset and length, after + * we've decrypted the result. + */ +static inline void ceph_fscrypt_adjust_off_and_len(struct inode *inode, + u64 *off, u64 *len) +{ + if (IS_ENCRYPTED(inode)) { + *len = ceph_fscrypt_blocks(*off, *len) * CEPH_FSCRYPT_BLOCK_SIZE; + *off &= CEPH_FSCRYPT_BLOCK_MASK; + } +} + +int ceph_fscrypt_decrypt_block_inplace(const struct inode *inode, + struct page *page, unsigned int len, + unsigned int offs, u64 lblk_num); +int ceph_fscrypt_encrypt_block_inplace(const struct inode *inode, + struct page *page, unsigned int len, + unsigned int offs, u64 lblk_num, + gfp_t gfp_flags); +int ceph_fscrypt_decrypt_pages(struct inode *inode, struct page **page, + u64 off, int len); +int ceph_fscrypt_decrypt_extents(struct inode *inode, struct page **page, + u64 off, struct ceph_sparse_extent *map, + u32 ext_cnt); +int ceph_fscrypt_encrypt_pages(struct inode *inode, struct page **page, u64 off, + int len, gfp_t gfp); + +static inline struct page *ceph_fscrypt_pagecache_page(struct page *page) +{ + return fscrypt_is_bounce_page(page) ? fscrypt_pagecache_page(page) : page; +} + +#else /* CONFIG_FS_ENCRYPTION */ + +static inline void ceph_fscrypt_set_ops(struct super_block *sb) +{ +} + +static inline void ceph_fscrypt_free_dummy_policy(struct ceph_fs_client *fsc) +{ +} + +static inline int ceph_fscrypt_prepare_context(struct inode *dir, + struct inode *inode, + struct ceph_acl_sec_ctx *as) +{ + if (IS_ENCRYPTED(dir)) + return -EOPNOTSUPP; + return 0; +} + +static inline void ceph_fscrypt_as_ctx_to_req(struct ceph_mds_request *req, + struct ceph_acl_sec_ctx *as_ctx) +{ +} + +static inline int ceph_encode_encrypted_dname(struct inode *parent, + struct qstr *d_name, char *buf) +{ + memcpy(buf, d_name->name, d_name->len); + return d_name->len; +} + +static inline int ceph_encode_encrypted_fname(struct inode *parent, + struct dentry *dentry, char *buf) +{ + return -EOPNOTSUPP; +} + +static inline int ceph_fname_alloc_buffer(struct inode *parent, + struct fscrypt_str *fname) +{ + return 0; +} + +static inline void ceph_fname_free_buffer(struct inode *parent, + struct fscrypt_str *fname) +{ +} + +static inline int ceph_fname_to_usr(const struct ceph_fname *fname, + struct fscrypt_str *tname, + struct fscrypt_str *oname, bool *is_nokey) +{ + oname->name = fname->name; + oname->len = fname->name_len; + return 0; +} + +static inline int ceph_fscrypt_prepare_readdir(struct inode *dir) +{ + return 0; +} + +static inline void ceph_fscrypt_adjust_off_and_len(struct inode *inode, + u64 *off, u64 *len) +{ +} + +static inline int ceph_fscrypt_decrypt_block_inplace(const struct inode *inode, + struct page *page, unsigned int len, + unsigned int offs, u64 lblk_num) +{ + return 0; +} + +static inline int ceph_fscrypt_encrypt_block_inplace(const struct inode *inode, + struct page *page, unsigned int len, + unsigned int offs, u64 lblk_num, + gfp_t gfp_flags) +{ + return 0; +} + +static inline int ceph_fscrypt_decrypt_pages(struct inode *inode, + struct page **page, u64 off, + int len) +{ + return 0; +} + +static inline int ceph_fscrypt_decrypt_extents(struct inode *inode, + struct page **page, u64 off, + struct ceph_sparse_extent *map, + u32 ext_cnt) +{ + return 0; +} + +static inline int ceph_fscrypt_encrypt_pages(struct inode *inode, + struct page **page, u64 off, + int len, gfp_t gfp) +{ + return 0; +} + +static inline struct page *ceph_fscrypt_pagecache_page(struct page *page) +{ + return page; +} +#endif /* CONFIG_FS_ENCRYPTION */ + +static inline loff_t ceph_fscrypt_page_offset(struct page *page) +{ + return page_offset(ceph_fscrypt_pagecache_page(page)); +} + +#endif /* _CEPH_CRYPTO_H */ diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index bdcffb04513f..854cbdd66661 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -9,6 +9,7 @@ #include "super.h" #include "mds_client.h" +#include "crypto.h" /* * Directory operations: readdir, lookup, create, link, unlink, @@ -241,7 +242,9 @@ static int __dcache_readdir(struct file *file, struct dir_context *ctx, di = ceph_dentry(dentry); if (d_unhashed(dentry) || d_really_is_negative(dentry) || - di->lease_shared_gen != shared_gen) { + di->lease_shared_gen != shared_gen || + ((dentry->d_flags & DCACHE_NOKEY_NAME) && + fscrypt_has_encryption_key(dir))) { spin_unlock(&dentry->d_lock); dput(dentry); err = -EAGAIN; @@ -340,6 +343,10 @@ static int ceph_readdir(struct file *file, struct dir_context *ctx) ctx->pos = 2; } + err = ceph_fscrypt_prepare_readdir(inode); + if (err < 0) + return err; + spin_lock(&ci->i_ceph_lock); /* request Fx cap. if have Fx, we don't need to release Fs cap * for later create/unlink. */ @@ -389,6 +396,7 @@ more: req = ceph_mdsc_create_request(mdsc, op, USE_AUTH_MDS); if (IS_ERR(req)) return PTR_ERR(req); + err = ceph_alloc_readdir_reply_buffer(req, inode); if (err) { ceph_mdsc_put_request(req); @@ -402,11 +410,21 @@ more: req->r_inode_drop = CEPH_CAP_FILE_EXCL; } if (dfi->last_name) { - req->r_path2 = kstrdup(dfi->last_name, GFP_KERNEL); + struct qstr d_name = { .name = dfi->last_name, + .len = strlen(dfi->last_name) }; + + req->r_path2 = kzalloc(NAME_MAX + 1, GFP_KERNEL); if (!req->r_path2) { ceph_mdsc_put_request(req); return -ENOMEM; } + + err = ceph_encode_encrypted_dname(inode, &d_name, + req->r_path2); + if (err < 0) { + ceph_mdsc_put_request(req); + return err; + } } else if (is_hash_order(ctx->pos)) { req->r_args.readdir.offset_hash = cpu_to_le32(fpos_hash(ctx->pos)); @@ -511,15 +529,20 @@ more: for (; i < rinfo->dir_nr; i++) { struct ceph_mds_reply_dir_entry *rde = rinfo->dir_entries + i; - BUG_ON(rde->offset < ctx->pos); + if (rde->offset < ctx->pos) { + pr_warn("%s: rde->offset 0x%llx ctx->pos 0x%llx\n", + __func__, rde->offset, ctx->pos); + return -EIO; + } + + if (WARN_ON_ONCE(!rde->inode.in)) + return -EIO; ctx->pos = rde->offset; dout("readdir (%d/%d) -> %llx '%.*s' %p\n", i, rinfo->dir_nr, ctx->pos, rde->name_len, rde->name, &rde->inode.in); - BUG_ON(!rde->inode.in); - if (!dir_emit(ctx, rde->name, rde->name_len, ceph_present_ino(inode->i_sb, le64_to_cpu(rde->inode.in->ino)), le32_to_cpu(rde->inode.in->mode) >> 12)) { @@ -532,6 +555,8 @@ more: dout("filldir stopping us...\n"); return 0; } + + /* Reset the lengths to their original allocated vals */ ctx->pos++; } @@ -586,7 +611,6 @@ more: dfi->dir_ordered_count); spin_unlock(&ci->i_ceph_lock); } - dout("readdir %p file %p done.\n", inode, file); return 0; } @@ -760,6 +784,18 @@ static struct dentry *ceph_lookup(struct inode *dir, struct dentry *dentry, if (dentry->d_name.len > NAME_MAX) return ERR_PTR(-ENAMETOOLONG); + if (IS_ENCRYPTED(dir)) { + bool had_key = fscrypt_has_encryption_key(dir); + + err = fscrypt_prepare_lookup_partial(dir, dentry); + if (err < 0) + return ERR_PTR(err); + + /* mark directory as incomplete if it has been unlocked */ + if (!had_key && fscrypt_has_encryption_key(dir)) + ceph_dir_clear_complete(dir); + } + /* can we conclude ENOENT locally? */ if (d_really_is_negative(dentry)) { struct ceph_inode_info *ci = ceph_inode(dir); @@ -865,13 +901,6 @@ static int ceph_mknod(struct mnt_idmap *idmap, struct inode *dir, goto out; } - err = ceph_pre_init_acls(dir, &mode, &as_ctx); - if (err < 0) - goto out; - err = ceph_security_init_secctx(dentry, mode, &as_ctx); - if (err < 0) - goto out; - dout("mknod in dir %p dentry %p mode 0%ho rdev %d\n", dir, dentry, mode, rdev); req = ceph_mdsc_create_request(mdsc, CEPH_MDS_OP_MKNOD, USE_AUTH_MDS); @@ -879,6 +908,17 @@ static int ceph_mknod(struct mnt_idmap *idmap, struct inode *dir, err = PTR_ERR(req); goto out; } + + req->r_new_inode = ceph_new_inode(dir, dentry, &mode, &as_ctx); + if (IS_ERR(req->r_new_inode)) { + err = PTR_ERR(req->r_new_inode); + req->r_new_inode = NULL; + goto out_req; + } + + if (S_ISREG(mode) && IS_ENCRYPTED(dir)) + set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags); + req->r_dentry = dget(dentry); req->r_num_caps = 2; req->r_parent = dir; @@ -889,13 +929,13 @@ static int ceph_mknod(struct mnt_idmap *idmap, struct inode *dir, req->r_dentry_drop = CEPH_CAP_FILE_SHARED | CEPH_CAP_AUTH_EXCL | CEPH_CAP_XATTR_EXCL; req->r_dentry_unless = CEPH_CAP_FILE_EXCL; - if (as_ctx.pagelist) { - req->r_pagelist = as_ctx.pagelist; - as_ctx.pagelist = NULL; - } + + ceph_as_ctx_to_req(req, &as_ctx); + err = ceph_mdsc_do_request(mdsc, dir, req); if (!err && !req->r_reply_info.head->is_dentry) err = ceph_handle_notrace_create(dir, dentry); +out_req: ceph_mdsc_put_request(req); out: if (!err) @@ -912,12 +952,50 @@ static int ceph_create(struct mnt_idmap *idmap, struct inode *dir, return ceph_mknod(idmap, dir, dentry, mode, 0); } +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) +static int prep_encrypted_symlink_target(struct ceph_mds_request *req, + const char *dest) +{ + int err; + int len = strlen(dest); + struct fscrypt_str osd_link = FSTR_INIT(NULL, 0); + + err = fscrypt_prepare_symlink(req->r_parent, dest, len, PATH_MAX, + &osd_link); + if (err) + goto out; + + err = fscrypt_encrypt_symlink(req->r_new_inode, dest, len, &osd_link); + if (err) + goto out; + + req->r_path2 = kmalloc(CEPH_BASE64_CHARS(osd_link.len) + 1, GFP_KERNEL); + if (!req->r_path2) { + err = -ENOMEM; + goto out; + } + + len = ceph_base64_encode(osd_link.name, osd_link.len, req->r_path2); + req->r_path2[len] = '\0'; +out: + fscrypt_fname_free_buffer(&osd_link); + return err; +} +#else +static int prep_encrypted_symlink_target(struct ceph_mds_request *req, + const char *dest) +{ + return -EOPNOTSUPP; +} +#endif + static int ceph_symlink(struct mnt_idmap *idmap, struct inode *dir, struct dentry *dentry, const char *dest) { struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(dir->i_sb); struct ceph_mds_request *req; struct ceph_acl_sec_ctx as_ctx = {}; + umode_t mode = S_IFLNK | 0777; int err; if (ceph_snap(dir) != CEPH_NOSNAP) @@ -932,38 +1010,48 @@ static int ceph_symlink(struct mnt_idmap *idmap, struct inode *dir, goto out; } - err = ceph_security_init_secctx(dentry, S_IFLNK | 0777, &as_ctx); - if (err < 0) - goto out; - dout("symlink in dir %p dentry %p to '%s'\n", dir, dentry, dest); req = ceph_mdsc_create_request(mdsc, CEPH_MDS_OP_SYMLINK, USE_AUTH_MDS); if (IS_ERR(req)) { err = PTR_ERR(req); goto out; } - req->r_path2 = kstrdup(dest, GFP_KERNEL); - if (!req->r_path2) { - err = -ENOMEM; - ceph_mdsc_put_request(req); - goto out; + + req->r_new_inode = ceph_new_inode(dir, dentry, &mode, &as_ctx); + if (IS_ERR(req->r_new_inode)) { + err = PTR_ERR(req->r_new_inode); + req->r_new_inode = NULL; + goto out_req; } + req->r_parent = dir; ihold(dir); + if (IS_ENCRYPTED(req->r_new_inode)) { + err = prep_encrypted_symlink_target(req, dest); + if (err) + goto out_req; + } else { + req->r_path2 = kstrdup(dest, GFP_KERNEL); + if (!req->r_path2) { + err = -ENOMEM; + goto out_req; + } + } + set_bit(CEPH_MDS_R_PARENT_LOCKED, &req->r_req_flags); req->r_dentry = dget(dentry); req->r_num_caps = 2; req->r_dentry_drop = CEPH_CAP_FILE_SHARED | CEPH_CAP_AUTH_EXCL | CEPH_CAP_XATTR_EXCL; req->r_dentry_unless = CEPH_CAP_FILE_EXCL; - if (as_ctx.pagelist) { - req->r_pagelist = as_ctx.pagelist; - as_ctx.pagelist = NULL; - } + + ceph_as_ctx_to_req(req, &as_ctx); + err = ceph_mdsc_do_request(mdsc, dir, req); if (!err && !req->r_reply_info.head->is_dentry) err = ceph_handle_notrace_create(dir, dentry); +out_req: ceph_mdsc_put_request(req); out: if (err) @@ -1003,14 +1091,12 @@ static int ceph_mkdir(struct mnt_idmap *idmap, struct inode *dir, err = -EDQUOT; goto out; } + if ((op == CEPH_MDS_OP_MKSNAP) && IS_ENCRYPTED(dir) && + !fscrypt_has_encryption_key(dir)) { + err = -ENOKEY; + goto out; + } - mode |= S_IFDIR; - err = ceph_pre_init_acls(dir, &mode, &as_ctx); - if (err < 0) - goto out; - err = ceph_security_init_secctx(dentry, mode, &as_ctx); - if (err < 0) - goto out; req = ceph_mdsc_create_request(mdsc, op, USE_AUTH_MDS); if (IS_ERR(req)) { @@ -1018,6 +1104,14 @@ static int ceph_mkdir(struct mnt_idmap *idmap, struct inode *dir, goto out; } + mode |= S_IFDIR; + req->r_new_inode = ceph_new_inode(dir, dentry, &mode, &as_ctx); + if (IS_ERR(req->r_new_inode)) { + err = PTR_ERR(req->r_new_inode); + req->r_new_inode = NULL; + goto out_req; + } + req->r_dentry = dget(dentry); req->r_num_caps = 2; req->r_parent = dir; @@ -1027,15 +1121,15 @@ static int ceph_mkdir(struct mnt_idmap *idmap, struct inode *dir, req->r_dentry_drop = CEPH_CAP_FILE_SHARED | CEPH_CAP_AUTH_EXCL | CEPH_CAP_XATTR_EXCL; req->r_dentry_unless = CEPH_CAP_FILE_EXCL; - if (as_ctx.pagelist) { - req->r_pagelist = as_ctx.pagelist; - as_ctx.pagelist = NULL; - } + + ceph_as_ctx_to_req(req, &as_ctx); + err = ceph_mdsc_do_request(mdsc, dir, req); if (!err && !req->r_reply_info.head->is_target && !req->r_reply_info.head->is_dentry) err = ceph_handle_notrace_create(dir, dentry); +out_req: ceph_mdsc_put_request(req); out: if (!err) @@ -1063,6 +1157,10 @@ static int ceph_link(struct dentry *old_dentry, struct inode *dir, if (ceph_snap(dir) != CEPH_NOSNAP) return -EROFS; + err = fscrypt_prepare_link(old_dentry, dir, dentry); + if (err) + return err; + dout("link in dir %p %llx.%llx old_dentry %p:'%pd' dentry %p:'%pd'\n", dir, ceph_vinop(dir), old_dentry, old_dentry, dentry, dentry); req = ceph_mdsc_create_request(mdsc, CEPH_MDS_OP_LINK, USE_AUTH_MDS); @@ -1310,6 +1408,11 @@ static int ceph_rename(struct mnt_idmap *idmap, struct inode *old_dir, if (err) return err; + err = fscrypt_prepare_rename(old_dir, old_dentry, new_dir, new_dentry, + flags); + if (err) + return err; + dout("rename dir %p dentry %p to dir %p dentry %p\n", old_dir, old_dentry, new_dir, new_dentry); req = ceph_mdsc_create_request(mdsc, op, USE_AUTH_MDS); @@ -1765,6 +1868,10 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) struct inode *dir, *inode; struct ceph_mds_client *mdsc; + valid = fscrypt_d_revalidate(dentry, flags); + if (valid <= 0) + return valid; + if (flags & LOOKUP_RCU) { parent = READ_ONCE(dentry->d_parent); dir = d_inode_rcu(parent); @@ -1777,8 +1884,9 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags) inode = d_inode(dentry); } - dout("d_revalidate %p '%pd' inode %p offset 0x%llx\n", dentry, - dentry, inode, ceph_dentry(dentry)->offset); + dout("d_revalidate %p '%pd' inode %p offset 0x%llx nokey %d\n", dentry, + dentry, inode, ceph_dentry(dentry)->offset, + !!(dentry->d_flags & DCACHE_NOKEY_NAME)); mdsc = ceph_sb_to_client(dir->i_sb)->mdsc; diff --git a/fs/ceph/export.c b/fs/ceph/export.c index f780e4e0d062..8559990a59a5 100644 --- a/fs/ceph/export.c +++ b/fs/ceph/export.c @@ -7,6 +7,7 @@ #include "super.h" #include "mds_client.h" +#include "crypto.h" /* * Basic fh @@ -535,7 +536,9 @@ static int ceph_get_name(struct dentry *parent, char *name, { struct ceph_mds_client *mdsc; struct ceph_mds_request *req; + struct inode *dir = d_inode(parent); struct inode *inode = d_inode(child); + struct ceph_mds_reply_info_parsed *rinfo; int err; if (ceph_snap(inode) != CEPH_NOSNAP) @@ -547,30 +550,47 @@ static int ceph_get_name(struct dentry *parent, char *name, if (IS_ERR(req)) return PTR_ERR(req); - inode_lock(d_inode(parent)); - + inode_lock(dir); req->r_inode = inode; ihold(inode); req->r_ino2 = ceph_vino(d_inode(parent)); - req->r_parent = d_inode(parent); - ihold(req->r_parent); + req->r_parent = dir; + ihold(dir); set_bit(CEPH_MDS_R_PARENT_LOCKED, &req->r_req_flags); req->r_num_caps = 2; err = ceph_mdsc_do_request(mdsc, NULL, req); + inode_unlock(dir); - inode_unlock(d_inode(parent)); + if (err) + goto out; - if (!err) { - struct ceph_mds_reply_info_parsed *rinfo = &req->r_reply_info; + rinfo = &req->r_reply_info; + if (!IS_ENCRYPTED(dir)) { memcpy(name, rinfo->dname, rinfo->dname_len); name[rinfo->dname_len] = 0; - dout("get_name %p ino %llx.%llx name %s\n", - child, ceph_vinop(inode), name); } else { - dout("get_name %p ino %llx.%llx err %d\n", - child, ceph_vinop(inode), err); - } + struct fscrypt_str oname = FSTR_INIT(NULL, 0); + struct ceph_fname fname = { .dir = dir, + .name = rinfo->dname, + .ctext = rinfo->altname, + .name_len = rinfo->dname_len, + .ctext_len = rinfo->altname_len }; + err = ceph_fname_alloc_buffer(dir, &oname); + if (err < 0) + goto out; + + err = ceph_fname_to_usr(&fname, NULL, &oname, NULL); + if (!err) { + memcpy(name, oname.name, oname.len); + name[oname.len] = 0; + } + ceph_fname_free_buffer(dir, &oname); + } +out: + dout("get_name %p ino %llx.%llx err %d %s%s\n", + child, ceph_vinop(inode), err, + err ? "" : "name ", err ? "" : name); ceph_mdsc_put_request(req); return err; } diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 63efe5389783..b1da02f5dbe3 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -366,8 +366,13 @@ int ceph_open(struct inode *inode, struct file *file) /* filter out O_CREAT|O_EXCL; vfs did that already. yuck. */ flags = file->f_flags & ~(O_CREAT|O_EXCL); - if (S_ISDIR(inode->i_mode)) + if (S_ISDIR(inode->i_mode)) { flags = O_DIRECTORY; /* mds likes to know */ + } else if (S_ISREG(inode->i_mode)) { + err = fscrypt_file_open(inode, file); + if (err) + return err; + } dout("open inode %p ino %llx.%llx file %p flags %d (%d)\n", inode, ceph_vinop(inode), file, flags, file->f_flags); @@ -604,7 +609,8 @@ out: ceph_mdsc_release_dir_caps(req); } -static int ceph_finish_async_create(struct inode *dir, struct dentry *dentry, +static int ceph_finish_async_create(struct inode *dir, struct inode *inode, + struct dentry *dentry, struct file *file, umode_t mode, struct ceph_mds_request *req, struct ceph_acl_sec_ctx *as_ctx, @@ -616,7 +622,6 @@ static int ceph_finish_async_create(struct inode *dir, struct dentry *dentry, struct ceph_mds_reply_info_in iinfo = { .in = &in }; struct ceph_inode_info *ci = ceph_inode(dir); struct ceph_dentry_info *di = ceph_dentry(dentry); - struct inode *inode; struct timespec64 now; struct ceph_string *pool_ns; struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(dir->i_sb); @@ -625,10 +630,6 @@ static int ceph_finish_async_create(struct inode *dir, struct dentry *dentry, ktime_get_real_ts64(&now); - inode = ceph_get_inode(dentry->d_sb, vino); - if (IS_ERR(inode)) - return PTR_ERR(inode); - iinfo.inline_version = CEPH_INLINE_NONE; iinfo.change_attr = 1; ceph_encode_timespec64(&iinfo.btime, &now); @@ -686,8 +687,7 @@ static int ceph_finish_async_create(struct inode *dir, struct dentry *dentry, ceph_dir_clear_complete(dir); if (!d_unhashed(dentry)) d_drop(dentry); - if (inode->i_state & I_NEW) - discard_new_inode(inode); + discard_new_inode(inode); } else { struct dentry *dn; @@ -733,6 +733,7 @@ int ceph_atomic_open(struct inode *dir, struct dentry *dentry, struct ceph_fs_client *fsc = ceph_sb_to_client(dir->i_sb); struct ceph_mds_client *mdsc = fsc->mdsc; struct ceph_mds_request *req; + struct inode *new_inode = NULL; struct dentry *dn; struct ceph_acl_sec_ctx as_ctx = {}; bool try_async = ceph_test_mount_opt(fsc, ASYNC_DIROPS); @@ -755,15 +756,16 @@ int ceph_atomic_open(struct inode *dir, struct dentry *dentry, */ flags &= ~O_TRUNC; +retry: if (flags & O_CREAT) { if (ceph_quota_is_max_files_exceeded(dir)) return -EDQUOT; - err = ceph_pre_init_acls(dir, &mode, &as_ctx); - if (err < 0) - return err; - err = ceph_security_init_secctx(dentry, mode, &as_ctx); - if (err < 0) + + new_inode = ceph_new_inode(dir, dentry, &mode, &as_ctx); + if (IS_ERR(new_inode)) { + err = PTR_ERR(new_inode); goto out_ctx; + } /* Async create can't handle more than a page of xattrs */ if (as_ctx.pagelist && !list_is_singular(&as_ctx.pagelist->head)) @@ -772,7 +774,7 @@ int ceph_atomic_open(struct inode *dir, struct dentry *dentry, /* If it's not being looked up, it's negative */ return -ENOENT; } -retry: + /* do the open */ req = prepare_open_request(dir->i_sb, flags, mode); if (IS_ERR(req)) { @@ -787,6 +789,12 @@ retry: req->r_args.open.mask = cpu_to_le32(mask); req->r_parent = dir; ihold(dir); + if (IS_ENCRYPTED(dir)) { + set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags); + err = fscrypt_prepare_lookup_partial(dir, dentry); + if (err < 0) + goto out_req; + } if (flags & O_CREAT) { struct ceph_file_layout lo; @@ -794,32 +802,47 @@ retry: req->r_dentry_drop = CEPH_CAP_FILE_SHARED | CEPH_CAP_AUTH_EXCL | CEPH_CAP_XATTR_EXCL; req->r_dentry_unless = CEPH_CAP_FILE_EXCL; - if (as_ctx.pagelist) { - req->r_pagelist = as_ctx.pagelist; - as_ctx.pagelist = NULL; - } - if (try_async && - (req->r_dir_caps = - try_prep_async_create(dir, dentry, &lo, - &req->r_deleg_ino))) { + + ceph_as_ctx_to_req(req, &as_ctx); + + if (try_async && (req->r_dir_caps = + try_prep_async_create(dir, dentry, &lo, + &req->r_deleg_ino))) { + struct ceph_vino vino = { .ino = req->r_deleg_ino, + .snap = CEPH_NOSNAP }; struct ceph_dentry_info *di = ceph_dentry(dentry); set_bit(CEPH_MDS_R_ASYNC, &req->r_req_flags); req->r_args.open.flags |= cpu_to_le32(CEPH_O_EXCL); req->r_callback = ceph_async_create_cb; + /* Hash inode before RPC */ + new_inode = ceph_get_inode(dir->i_sb, vino, new_inode); + if (IS_ERR(new_inode)) { + err = PTR_ERR(new_inode); + new_inode = NULL; + goto out_req; + } + WARN_ON_ONCE(!(new_inode->i_state & I_NEW)); + spin_lock(&dentry->d_lock); di->flags |= CEPH_DENTRY_ASYNC_CREATE; spin_unlock(&dentry->d_lock); err = ceph_mdsc_submit_request(mdsc, dir, req); if (!err) { - err = ceph_finish_async_create(dir, dentry, - file, mode, req, - &as_ctx, &lo); + err = ceph_finish_async_create(dir, new_inode, + dentry, file, + mode, req, + &as_ctx, &lo); + new_inode = NULL; } else if (err == -EJUKEBOX) { restore_deleg_ino(dir, req->r_deleg_ino); ceph_mdsc_put_request(req); + discard_new_inode(new_inode); + ceph_release_acl_sec_ctx(&as_ctx); + memset(&as_ctx, 0, sizeof(as_ctx)); + new_inode = NULL; try_async = false; ceph_put_string(rcu_dereference_raw(lo.pool_ns)); goto retry; @@ -830,6 +853,8 @@ retry: } set_bit(CEPH_MDS_R_PARENT_LOCKED, &req->r_req_flags); + req->r_new_inode = new_inode; + new_inode = NULL; err = ceph_mdsc_do_request(mdsc, (flags & O_CREAT) ? dir : NULL, req); if (err == -ENOENT) { dentry = ceph_handle_snapdir(req, dentry); @@ -858,6 +883,13 @@ retry: dout("atomic_open finish_no_open on dn %p\n", dn); err = finish_no_open(file, dn); } else { + if (IS_ENCRYPTED(dir) && + !fscrypt_has_permitted_context(dir, d_inode(dentry))) { + pr_warn("Inconsistent encryption context (parent %llx:%llx child %llx:%llx)\n", + ceph_vinop(dir), ceph_vinop(d_inode(dentry))); + goto out_req; + } + dout("atomic_open finish_open on dn %p\n", dn); if (req->r_op == CEPH_MDS_OP_CREATE && req->r_reply_info.has_create_ino) { struct inode *newino = d_inode(dentry); @@ -870,6 +902,7 @@ retry: } out_req: ceph_mdsc_put_request(req); + iput(new_inode); out_ctx: ceph_release_acl_sec_ctx(&as_ctx); dout("atomic_open result=%d\n", err); @@ -924,21 +957,24 @@ enum { * If we get a short result from the OSD, check against i_size; we need to * only return a short read to the caller if we hit EOF. */ -static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, - int *retry_op) +ssize_t __ceph_sync_read(struct inode *inode, loff_t *ki_pos, + struct iov_iter *to, int *retry_op, + u64 *last_objver) { - struct file *file = iocb->ki_filp; - struct inode *inode = file_inode(file); struct ceph_inode_info *ci = ceph_inode(inode); struct ceph_fs_client *fsc = ceph_inode_to_client(inode); struct ceph_osd_client *osdc = &fsc->client->osdc; ssize_t ret; - u64 off = iocb->ki_pos; + u64 off = *ki_pos; u64 len = iov_iter_count(to); u64 i_size = i_size_read(inode); + bool sparse = IS_ENCRYPTED(inode) || ceph_test_mount_opt(fsc, SPARSEREAD); + u64 objver = 0; - dout("sync_read on file %p %llu~%u %s\n", file, off, (unsigned)len, - (file->f_flags & O_DIRECT) ? "O_DIRECT" : ""); + dout("sync_read on inode %p %llx~%llx\n", inode, *ki_pos, len); + + if (ceph_inode_is_shutdown(inode)) + return -EIO; if (!len) return 0; @@ -962,10 +998,21 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, bool more; int idx; size_t left; + struct ceph_osd_req_op *op; + u64 read_off = off; + u64 read_len = len; + + /* determine new offset/length if encrypted */ + ceph_fscrypt_adjust_off_and_len(inode, &read_off, &read_len); + + dout("sync_read orig %llu~%llu reading %llu~%llu", + off, len, read_off, read_len); req = ceph_osdc_new_request(osdc, &ci->i_layout, - ci->i_vino, off, &len, 0, 1, - CEPH_OSD_OP_READ, CEPH_OSD_FLAG_READ, + ci->i_vino, read_off, &read_len, 0, 1, + sparse ? CEPH_OSD_OP_SPARSE_READ : + CEPH_OSD_OP_READ, + CEPH_OSD_FLAG_READ, NULL, ci->i_truncate_seq, ci->i_truncate_size, false); if (IS_ERR(req)) { @@ -973,10 +1020,13 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, break; } + /* adjust len downward if the request truncated the len */ + if (off + len > read_off + read_len) + len = read_off + read_len - off; more = len < iov_iter_count(to); - num_pages = calc_pages_for(off, len); - page_off = off & ~PAGE_MASK; + num_pages = calc_pages_for(read_off, read_len); + page_off = offset_in_page(off); pages = ceph_alloc_page_vector(num_pages, GFP_KERNEL); if (IS_ERR(pages)) { ceph_osdc_put_request(req); @@ -984,29 +1034,75 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, break; } - osd_req_op_extent_osd_data_pages(req, 0, pages, len, page_off, + osd_req_op_extent_osd_data_pages(req, 0, pages, read_len, + offset_in_page(read_off), false, false); + + op = &req->r_ops[0]; + if (sparse) { + ret = ceph_alloc_sparse_ext_map(op); + if (ret) { + ceph_osdc_put_request(req); + break; + } + } + ceph_osdc_start_request(osdc, req); ret = ceph_osdc_wait_request(osdc, req); ceph_update_read_metrics(&fsc->mdsc->metric, req->r_start_latency, req->r_end_latency, - len, ret); + read_len, ret); - ceph_osdc_put_request(req); + if (ret > 0) + objver = req->r_version; i_size = i_size_read(inode); dout("sync_read %llu~%llu got %zd i_size %llu%s\n", off, len, ret, i_size, (more ? " MORE" : "")); - if (ret == -ENOENT) + /* Fix it to go to end of extent map */ + if (sparse && ret >= 0) + ret = ceph_sparse_ext_map_end(op); + else if (ret == -ENOENT) ret = 0; + + if (ret > 0 && IS_ENCRYPTED(inode)) { + int fret; + + fret = ceph_fscrypt_decrypt_extents(inode, pages, + read_off, op->extent.sparse_ext, + op->extent.sparse_ext_cnt); + if (fret < 0) { + ret = fret; + ceph_osdc_put_request(req); + break; + } + + /* account for any partial block at the beginning */ + fret -= (off - read_off); + + /* + * Short read after big offset adjustment? + * Nothing is usable, just call it a zero + * len read. + */ + fret = max(fret, 0); + + /* account for partial block at the end */ + ret = min_t(ssize_t, fret, len); + } + + ceph_osdc_put_request(req); + + /* Short read but not EOF? Zero out the remainder. */ if (ret >= 0 && ret < len && (off + ret < i_size)) { int zlen = min(len - ret, i_size - off - ret); int zoff = page_off + ret; + dout("sync_read zero gap %llu~%llu\n", - off + ret, off + ret + zlen); + off + ret, off + ret + zlen); ceph_zero_page_vector_range(zoff, zlen, pages); ret += zlen; } @@ -1014,15 +1110,16 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, idx = 0; left = ret > 0 ? ret : 0; while (left > 0) { - size_t len, copied; - page_off = off & ~PAGE_MASK; - len = min_t(size_t, left, PAGE_SIZE - page_off); + size_t plen, copied; + + plen = min_t(size_t, left, PAGE_SIZE - page_off); SetPageUptodate(pages[idx]); copied = copy_page_to_iter(pages[idx++], - page_off, len, to); + page_off, plen, to); off += copied; left -= copied; - if (copied < len) { + page_off = 0; + if (copied < plen) { ret = -EFAULT; break; } @@ -1039,21 +1136,37 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, break; } - if (off > iocb->ki_pos) { - if (off >= i_size) { - *retry_op = CHECK_EOF; - ret = i_size - iocb->ki_pos; - iocb->ki_pos = i_size; - } else { - ret = off - iocb->ki_pos; - iocb->ki_pos = off; + if (ret > 0) { + if (off > *ki_pos) { + if (off >= i_size) { + *retry_op = CHECK_EOF; + ret = i_size - *ki_pos; + *ki_pos = i_size; + } else { + ret = off - *ki_pos; + *ki_pos = off; + } } - } + if (last_objver) + *last_objver = objver; + } dout("sync_read result %zd retry_op %d\n", ret, *retry_op); return ret; } +static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, + int *retry_op) +{ + struct file *file = iocb->ki_filp; + struct inode *inode = file_inode(file); + + dout("sync_read on file %p %llx~%zx %s\n", file, iocb->ki_pos, + iov_iter_count(to), (file->f_flags & O_DIRECT) ? "O_DIRECT" : ""); + + return __ceph_sync_read(inode, &iocb->ki_pos, to, retry_op, NULL); +} + struct ceph_aio_request { struct kiocb *iocb; size_t total_len; @@ -1125,8 +1238,10 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req) struct inode *inode = req->r_inode; struct ceph_aio_request *aio_req = req->r_priv; struct ceph_osd_data *osd_data = osd_req_op_extent_osd_data(req, 0); + struct ceph_osd_req_op *op = &req->r_ops[0]; struct ceph_client_metric *metric = &ceph_sb_to_mdsc(inode->i_sb)->metric; unsigned int len = osd_data->bvec_pos.iter.bi_size; + bool sparse = (op->op == CEPH_OSD_OP_SPARSE_READ); BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_BVECS); BUG_ON(!osd_data->num_bvecs); @@ -1147,6 +1262,8 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req) } rc = -ENOMEM; } else if (!aio_req->write) { + if (sparse && rc >= 0) + rc = ceph_sparse_ext_map_end(op); if (rc == -ENOENT) rc = 0; if (rc >= 0 && len > rc) { @@ -1283,6 +1400,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, loff_t pos = iocb->ki_pos; bool write = iov_iter_rw(iter) == WRITE; bool should_dirty = !write && user_backed_iter(iter); + bool sparse = ceph_test_mount_opt(fsc, SPARSEREAD); if (write && ceph_snap(file_inode(file)) != CEPH_NOSNAP) return -EROFS; @@ -1310,6 +1428,8 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, while (iov_iter_count(iter) > 0) { u64 size = iov_iter_count(iter); ssize_t len; + struct ceph_osd_req_op *op; + int readop = sparse ? CEPH_OSD_OP_SPARSE_READ : CEPH_OSD_OP_READ; if (write) size = min_t(u64, size, fsc->mount_options->wsize); @@ -1320,8 +1440,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout, vino, pos, &size, 0, 1, - write ? CEPH_OSD_OP_WRITE : - CEPH_OSD_OP_READ, + write ? CEPH_OSD_OP_WRITE : readop, flags, snapc, ci->i_truncate_seq, ci->i_truncate_size, @@ -1372,6 +1491,14 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, } osd_req_op_extent_osd_data_bvecs(req, 0, bvecs, num_pages, len); + op = &req->r_ops[0]; + if (sparse) { + ret = ceph_alloc_sparse_ext_map(op); + if (ret) { + ceph_osdc_put_request(req); + break; + } + } if (aio_req) { aio_req->total_len += len; @@ -1399,8 +1526,11 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, size = i_size_read(inode); if (!write) { - if (ret == -ENOENT) + if (sparse && ret >= 0) + ret = ceph_sparse_ext_map_end(op); + else if (ret == -ENOENT) ret = 0; + if (ret >= 0 && ret < len && pos + ret < size) { struct iov_iter i; int zlen = min_t(size_t, len - ret, @@ -1481,13 +1611,12 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, struct inode *inode = file_inode(file); struct ceph_inode_info *ci = ceph_inode(inode); struct ceph_fs_client *fsc = ceph_inode_to_client(inode); - struct ceph_vino vino; + struct ceph_osd_client *osdc = &fsc->client->osdc; struct ceph_osd_request *req; struct page **pages; u64 len; int num_pages; int written = 0; - int flags; int ret; bool check_caps = false; struct timespec64 mtime = current_time(inode); @@ -1505,79 +1634,350 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, return ret; ceph_fscache_invalidate(inode, false); - ret = invalidate_inode_pages2_range(inode->i_mapping, - pos >> PAGE_SHIFT, - (pos + count - 1) >> PAGE_SHIFT); - if (ret < 0) - dout("invalidate_inode_pages2_range returned %d\n", ret); - - flags = /* CEPH_OSD_FLAG_ORDERSNAP | */ CEPH_OSD_FLAG_WRITE; while ((len = iov_iter_count(from)) > 0) { size_t left; int n; + u64 write_pos = pos; + u64 write_len = len; + u64 objnum, objoff; + u32 xlen; + u64 assert_ver = 0; + bool rmw; + bool first, last; + struct iov_iter saved_iter = *from; + size_t off; - vino = ceph_vino(inode); - req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout, - vino, pos, &len, 0, 1, - CEPH_OSD_OP_WRITE, flags, snapc, - ci->i_truncate_seq, - ci->i_truncate_size, - false); - if (IS_ERR(req)) { - ret = PTR_ERR(req); - break; - } + ceph_fscrypt_adjust_off_and_len(inode, &write_pos, &write_len); + + /* clamp the length to the end of first object */ + ceph_calc_file_object_mapping(&ci->i_layout, write_pos, + write_len, &objnum, &objoff, + &xlen); + write_len = xlen; + + /* adjust len downward if it goes beyond current object */ + if (pos + len > write_pos + write_len) + len = write_pos + write_len - pos; /* - * write from beginning of first page, - * regardless of io alignment + * If we had to adjust the length or position to align with a + * crypto block, then we must do a read/modify/write cycle. We + * use a version assertion to redrive the thing if something + * changes in between. */ - num_pages = (len + PAGE_SIZE - 1) >> PAGE_SHIFT; + first = pos != write_pos; + last = (pos + len) != (write_pos + write_len); + rmw = first || last; + dout("sync_write ino %llx %lld~%llu adjusted %lld~%llu -- %srmw\n", + ci->i_vino.ino, pos, len, write_pos, write_len, + rmw ? "" : "no "); + + /* + * The data is emplaced into the page as it would be if it were + * in an array of pagecache pages. + */ + num_pages = calc_pages_for(write_pos, write_len); pages = ceph_alloc_page_vector(num_pages, GFP_KERNEL); if (IS_ERR(pages)) { ret = PTR_ERR(pages); - goto out; + break; + } + + /* Do we need to preload the pages? */ + if (rmw) { + u64 first_pos = write_pos; + u64 last_pos = (write_pos + write_len) - CEPH_FSCRYPT_BLOCK_SIZE; + u64 read_len = CEPH_FSCRYPT_BLOCK_SIZE; + struct ceph_osd_req_op *op; + + /* We should only need to do this for encrypted inodes */ + WARN_ON_ONCE(!IS_ENCRYPTED(inode)); + + /* No need to do two reads if first and last blocks are same */ + if (first && last_pos == first_pos) + last = false; + + /* + * Allocate a read request for one or two extents, + * depending on how the request was aligned. + */ + req = ceph_osdc_new_request(osdc, &ci->i_layout, + ci->i_vino, first ? first_pos : last_pos, + &read_len, 0, (first && last) ? 2 : 1, + CEPH_OSD_OP_SPARSE_READ, CEPH_OSD_FLAG_READ, + NULL, ci->i_truncate_seq, + ci->i_truncate_size, false); + if (IS_ERR(req)) { + ceph_release_page_vector(pages, num_pages); + ret = PTR_ERR(req); + break; + } + + /* Something is misaligned! */ + if (read_len != CEPH_FSCRYPT_BLOCK_SIZE) { + ceph_osdc_put_request(req); + ceph_release_page_vector(pages, num_pages); + ret = -EIO; + break; + } + + /* Add extent for first block? */ + op = &req->r_ops[0]; + + if (first) { + osd_req_op_extent_osd_data_pages(req, 0, pages, + CEPH_FSCRYPT_BLOCK_SIZE, + offset_in_page(first_pos), + false, false); + /* We only expect a single extent here */ + ret = __ceph_alloc_sparse_ext_map(op, 1); + if (ret) { + ceph_osdc_put_request(req); + ceph_release_page_vector(pages, num_pages); + break; + } + } + + /* Add extent for last block */ + if (last) { + /* Init the other extent if first extent has been used */ + if (first) { + op = &req->r_ops[1]; + osd_req_op_extent_init(req, 1, + CEPH_OSD_OP_SPARSE_READ, + last_pos, CEPH_FSCRYPT_BLOCK_SIZE, + ci->i_truncate_size, + ci->i_truncate_seq); + } + + ret = __ceph_alloc_sparse_ext_map(op, 1); + if (ret) { + ceph_osdc_put_request(req); + ceph_release_page_vector(pages, num_pages); + break; + } + + osd_req_op_extent_osd_data_pages(req, first ? 1 : 0, + &pages[num_pages - 1], + CEPH_FSCRYPT_BLOCK_SIZE, + offset_in_page(last_pos), + false, false); + } + + ceph_osdc_start_request(osdc, req); + ret = ceph_osdc_wait_request(osdc, req); + + /* FIXME: length field is wrong if there are 2 extents */ + ceph_update_read_metrics(&fsc->mdsc->metric, + req->r_start_latency, + req->r_end_latency, + read_len, ret); + + /* Ok if object is not already present */ + if (ret == -ENOENT) { + /* + * If there is no object, then we can't assert + * on its version. Set it to 0, and we'll use an + * exclusive create instead. + */ + ceph_osdc_put_request(req); + ret = 0; + + /* + * zero out the soon-to-be uncopied parts of the + * first and last pages. + */ + if (first) + zero_user_segment(pages[0], 0, + offset_in_page(first_pos)); + if (last) + zero_user_segment(pages[num_pages - 1], + offset_in_page(last_pos), + PAGE_SIZE); + } else { + if (ret < 0) { + ceph_osdc_put_request(req); + ceph_release_page_vector(pages, num_pages); + break; + } + + op = &req->r_ops[0]; + if (op->extent.sparse_ext_cnt == 0) { + if (first) + zero_user_segment(pages[0], 0, + offset_in_page(first_pos)); + else + zero_user_segment(pages[num_pages - 1], + offset_in_page(last_pos), + PAGE_SIZE); + } else if (op->extent.sparse_ext_cnt != 1 || + ceph_sparse_ext_map_end(op) != + CEPH_FSCRYPT_BLOCK_SIZE) { + ret = -EIO; + ceph_osdc_put_request(req); + ceph_release_page_vector(pages, num_pages); + break; + } + + if (first && last) { + op = &req->r_ops[1]; + if (op->extent.sparse_ext_cnt == 0) { + zero_user_segment(pages[num_pages - 1], + offset_in_page(last_pos), + PAGE_SIZE); + } else if (op->extent.sparse_ext_cnt != 1 || + ceph_sparse_ext_map_end(op) != + CEPH_FSCRYPT_BLOCK_SIZE) { + ret = -EIO; + ceph_osdc_put_request(req); + ceph_release_page_vector(pages, num_pages); + break; + } + } + + /* Grab assert version. It must be non-zero. */ + assert_ver = req->r_version; + WARN_ON_ONCE(ret > 0 && assert_ver == 0); + + ceph_osdc_put_request(req); + if (first) { + ret = ceph_fscrypt_decrypt_block_inplace(inode, + pages[0], CEPH_FSCRYPT_BLOCK_SIZE, + offset_in_page(first_pos), + first_pos >> CEPH_FSCRYPT_BLOCK_SHIFT); + if (ret < 0) { + ceph_release_page_vector(pages, num_pages); + break; + } + } + if (last) { + ret = ceph_fscrypt_decrypt_block_inplace(inode, + pages[num_pages - 1], + CEPH_FSCRYPT_BLOCK_SIZE, + offset_in_page(last_pos), + last_pos >> CEPH_FSCRYPT_BLOCK_SHIFT); + if (ret < 0) { + ceph_release_page_vector(pages, num_pages); + break; + } + } + } } left = len; + off = offset_in_page(pos); for (n = 0; n < num_pages; n++) { - size_t plen = min_t(size_t, left, PAGE_SIZE); - ret = copy_page_from_iter(pages[n], 0, plen, from); + size_t plen = min_t(size_t, left, PAGE_SIZE - off); + + /* copy the data */ + ret = copy_page_from_iter(pages[n], off, plen, from); if (ret != plen) { ret = -EFAULT; break; } + off = 0; left -= ret; } - if (ret < 0) { + dout("sync_write write failed with %d\n", ret); ceph_release_page_vector(pages, num_pages); - goto out; + break; } + if (IS_ENCRYPTED(inode)) { + ret = ceph_fscrypt_encrypt_pages(inode, pages, + write_pos, write_len, + GFP_KERNEL); + if (ret < 0) { + dout("encryption failed with %d\n", ret); + ceph_release_page_vector(pages, num_pages); + break; + } + } + + req = ceph_osdc_new_request(osdc, &ci->i_layout, + ci->i_vino, write_pos, &write_len, + rmw ? 1 : 0, rmw ? 2 : 1, + CEPH_OSD_OP_WRITE, + CEPH_OSD_FLAG_WRITE, + snapc, ci->i_truncate_seq, + ci->i_truncate_size, false); + if (IS_ERR(req)) { + ret = PTR_ERR(req); + ceph_release_page_vector(pages, num_pages); + break; + } + + dout("sync_write write op %lld~%llu\n", write_pos, write_len); + osd_req_op_extent_osd_data_pages(req, rmw ? 1 : 0, pages, write_len, + offset_in_page(write_pos), false, + true); req->r_inode = inode; - - osd_req_op_extent_osd_data_pages(req, 0, pages, len, 0, - false, true); - req->r_mtime = mtime; - ceph_osdc_start_request(&fsc->client->osdc, req); - ret = ceph_osdc_wait_request(&fsc->client->osdc, req); + + /* Set up the assertion */ + if (rmw) { + /* + * Set up the assertion. If we don't have a version + * number, then the object doesn't exist yet. Use an + * exclusive create instead of a version assertion in + * that case. + */ + if (assert_ver) { + osd_req_op_init(req, 0, CEPH_OSD_OP_ASSERT_VER, 0); + req->r_ops[0].assert_ver.ver = assert_ver; + } else { + osd_req_op_init(req, 0, CEPH_OSD_OP_CREATE, + CEPH_OSD_OP_FLAG_EXCL); + } + } + + ceph_osdc_start_request(osdc, req); + ret = ceph_osdc_wait_request(osdc, req); ceph_update_write_metrics(&fsc->mdsc->metric, req->r_start_latency, req->r_end_latency, len, ret); -out: ceph_osdc_put_request(req); if (ret != 0) { + dout("sync_write osd write returned %d\n", ret); + /* Version changed! Must re-do the rmw cycle */ + if ((assert_ver && (ret == -ERANGE || ret == -EOVERFLOW)) || + (!assert_ver && ret == -EEXIST)) { + /* We should only ever see this on a rmw */ + WARN_ON_ONCE(!rmw); + + /* The version should never go backward */ + WARN_ON_ONCE(ret == -EOVERFLOW); + + *from = saved_iter; + + /* FIXME: limit number of times we loop? */ + continue; + } ceph_set_error_write(ci); break; } ceph_clear_error_write(ci); + + /* + * We successfully wrote to a range of the file. Declare + * that region of the pagecache invalid. + */ + ret = invalidate_inode_pages2_range( + inode->i_mapping, + pos >> PAGE_SHIFT, + (pos + len - 1) >> PAGE_SHIFT); + if (ret < 0) { + dout("invalidate_inode_pages2_range returned %d\n", + ret); + ret = 0; + } pos += len; written += len; + dout("sync_write written %d\n", written); if (pos > i_size_read(inode)) { check_caps = ceph_inode_set_size(inode, pos); if (check_caps) @@ -1591,6 +1991,7 @@ out: ret = written; iocb->ki_pos = pos; } + dout("sync_write returning %d\n", ret); return ret; } @@ -1648,7 +2049,9 @@ again: ceph_cap_string(got)); if (!ceph_has_inline_data(ci)) { - if (!retry_op && (iocb->ki_flags & IOCB_DIRECT)) { + if (!retry_op && + (iocb->ki_flags & IOCB_DIRECT) && + !IS_ENCRYPTED(inode)) { ret = ceph_direct_read_write(iocb, to, NULL, NULL); if (ret >= 0 && ret < len) @@ -1934,7 +2337,7 @@ retry_snap: /* we might need to revert back to that point */ data = *from; - if (iocb->ki_flags & IOCB_DIRECT) + if ((iocb->ki_flags & IOCB_DIRECT) && !IS_ENCRYPTED(inode)) written = ceph_direct_read_write(iocb, &data, snapc, &prealloc_cf); else @@ -2165,6 +2568,9 @@ static long ceph_fallocate(struct file *file, int mode, if (!S_ISREG(inode->i_mode)) return -EOPNOTSUPP; + if (IS_ENCRYPTED(inode)) + return -EOPNOTSUPP; + prealloc_cf = ceph_alloc_cap_flush(); if (!prealloc_cf) return -ENOMEM; @@ -2486,6 +2892,10 @@ static ssize_t __ceph_copy_file_range(struct file *src_file, loff_t src_off, return -EOPNOTSUPP; } + /* Every encrypted inode gets its own key, so we can't offload them */ + if (IS_ENCRYPTED(src_inode) || IS_ENCRYPTED(dst_inode)) + return -EOPNOTSUPP; + if (len < src_ci->i_layout.object_size) return -EOPNOTSUPP; /* no remote copy will be done */ diff --git a/fs/ceph/inode.c b/fs/ceph/inode.c index fd05d68e2990..800ab7920513 100644 --- a/fs/ceph/inode.c +++ b/fs/ceph/inode.c @@ -14,10 +14,12 @@ #include #include #include +#include #include "super.h" #include "mds_client.h" #include "cache.h" +#include "crypto.h" #include /* @@ -33,6 +35,7 @@ */ static const struct inode_operations ceph_symlink_iops; +static const struct inode_operations ceph_encrypted_symlink_iops; static void ceph_inode_work(struct work_struct *work); @@ -52,17 +55,99 @@ static int ceph_set_ino_cb(struct inode *inode, void *data) return 0; } -struct inode *ceph_get_inode(struct super_block *sb, struct ceph_vino vino) +/** + * ceph_new_inode - allocate a new inode in advance of an expected create + * @dir: parent directory for new inode + * @dentry: dentry that may eventually point to new inode + * @mode: mode of new inode + * @as_ctx: pointer to inherited security context + * + * Allocate a new inode in advance of an operation to create a new inode. + * This allocates the inode and sets up the acl_sec_ctx with appropriate + * info for the new inode. + * + * Returns a pointer to the new inode or an ERR_PTR. + */ +struct inode *ceph_new_inode(struct inode *dir, struct dentry *dentry, + umode_t *mode, struct ceph_acl_sec_ctx *as_ctx) +{ + int err; + struct inode *inode; + + inode = new_inode(dir->i_sb); + if (!inode) + return ERR_PTR(-ENOMEM); + + if (!S_ISLNK(*mode)) { + err = ceph_pre_init_acls(dir, mode, as_ctx); + if (err < 0) + goto out_err; + } + + inode->i_state = 0; + inode->i_mode = *mode; + + err = ceph_security_init_secctx(dentry, *mode, as_ctx); + if (err < 0) + goto out_err; + + /* + * We'll skip setting fscrypt context for snapshots, leaving that for + * the handle_reply(). + */ + if (ceph_snap(dir) != CEPH_SNAPDIR) { + err = ceph_fscrypt_prepare_context(dir, inode, as_ctx); + if (err) + goto out_err; + } + + return inode; +out_err: + iput(inode); + return ERR_PTR(err); +} + +void ceph_as_ctx_to_req(struct ceph_mds_request *req, + struct ceph_acl_sec_ctx *as_ctx) +{ + if (as_ctx->pagelist) { + req->r_pagelist = as_ctx->pagelist; + as_ctx->pagelist = NULL; + } + ceph_fscrypt_as_ctx_to_req(req, as_ctx); +} + +/** + * ceph_get_inode - find or create/hash a new inode + * @sb: superblock to search and allocate in + * @vino: vino to search for + * @newino: optional new inode to insert if one isn't found (may be NULL) + * + * Search for or insert a new inode into the hash for the given vino, and + * return a reference to it. If new is non-NULL, its reference is consumed. + */ +struct inode *ceph_get_inode(struct super_block *sb, struct ceph_vino vino, + struct inode *newino) { struct inode *inode; if (ceph_vino_is_reserved(vino)) return ERR_PTR(-EREMOTEIO); - inode = iget5_locked(sb, (unsigned long)vino.ino, ceph_ino_compare, - ceph_set_ino_cb, &vino); - if (!inode) + if (newino) { + inode = inode_insert5(newino, (unsigned long)vino.ino, + ceph_ino_compare, ceph_set_ino_cb, &vino); + if (inode != newino) + iput(newino); + } else { + inode = iget5_locked(sb, (unsigned long)vino.ino, + ceph_ino_compare, ceph_set_ino_cb, &vino); + } + + if (!inode) { + dout("No inode found for %llx.%llx\n", vino.ino, vino.snap); return ERR_PTR(-ENOMEM); + } dout("get_inode on %llu=%llx.%llx got %p new %d\n", ceph_present_inode(inode), ceph_vinop(inode), inode, !!(inode->i_state & I_NEW)); @@ -78,8 +163,9 @@ struct inode *ceph_get_snapdir(struct inode *parent) .ino = ceph_ino(parent), .snap = CEPH_SNAPDIR, }; - struct inode *inode = ceph_get_inode(parent->i_sb, vino); + struct inode *inode = ceph_get_inode(parent->i_sb, vino, NULL); struct ceph_inode_info *ci = ceph_inode(inode); + int ret = -ENOTDIR; if (IS_ERR(inode)) return inode; @@ -105,6 +191,24 @@ struct inode *ceph_get_snapdir(struct inode *parent) ci->i_rbytes = 0; ci->i_btime = ceph_inode(parent)->i_btime; +#ifdef CONFIG_FS_ENCRYPTION + /* if encrypted, just borrow fscrypt_auth from parent */ + if (IS_ENCRYPTED(parent)) { + struct ceph_inode_info *pci = ceph_inode(parent); + + ci->fscrypt_auth = kmemdup(pci->fscrypt_auth, + pci->fscrypt_auth_len, + GFP_KERNEL); + if (ci->fscrypt_auth) { + inode->i_flags |= S_ENCRYPTED; + ci->fscrypt_auth_len = pci->fscrypt_auth_len; + } else { + dout("Failed to alloc snapdir fscrypt_auth\n"); + ret = -ENOMEM; + goto err; + } + } +#endif if (inode->i_state & I_NEW) { inode->i_op = &ceph_snapdir_iops; inode->i_fop = &ceph_snapdir_fops; @@ -118,7 +222,7 @@ err: discard_new_inode(inode); else iput(inode); - return ERR_PTR(-ENOTDIR); + return ERR_PTR(ret); } const struct inode_operations ceph_file_iops = { @@ -517,6 +621,7 @@ struct inode *ceph_alloc_inode(struct super_block *sb) ci->i_truncate_seq = 0; ci->i_truncate_size = 0; ci->i_truncate_pending = 0; + ci->i_truncate_pagecache_size = 0; ci->i_max_size = 0; ci->i_reported_size = 0; @@ -547,6 +652,10 @@ struct inode *ceph_alloc_inode(struct super_block *sb) INIT_WORK(&ci->i_work, ceph_inode_work); ci->i_work_mask = 0; memset(&ci->i_btime, '\0', sizeof(ci->i_btime)); +#ifdef CONFIG_FS_ENCRYPTION + ci->fscrypt_auth = NULL; + ci->fscrypt_auth_len = 0; +#endif return &ci->netfs.inode; } @@ -555,6 +664,10 @@ void ceph_free_inode(struct inode *inode) struct ceph_inode_info *ci = ceph_inode(inode); kfree(ci->i_symlink); +#ifdef CONFIG_FS_ENCRYPTION + kfree(ci->fscrypt_auth); +#endif + fscrypt_free_inode(inode); kmem_cache_free(ceph_inode_cachep, ci); } @@ -575,6 +688,7 @@ void ceph_evict_inode(struct inode *inode) clear_inode(inode); ceph_fscache_unregister_inode_cookie(ci); + fscrypt_put_encryption_info(inode); __ceph_remove_caps(ci); @@ -650,7 +764,7 @@ int ceph_fill_file_size(struct inode *inode, int issued, ceph_fscache_update(inode); ci->i_reported_size = size; if (truncate_seq != ci->i_truncate_seq) { - dout("truncate_seq %u -> %u\n", + dout("%s truncate_seq %u -> %u\n", __func__, ci->i_truncate_seq, truncate_seq); ci->i_truncate_seq = truncate_seq; @@ -674,11 +788,26 @@ int ceph_fill_file_size(struct inode *inode, int issued, } } } - if (ceph_seq_cmp(truncate_seq, ci->i_truncate_seq) >= 0 && - ci->i_truncate_size != truncate_size) { - dout("truncate_size %lld -> %llu\n", ci->i_truncate_size, - truncate_size); + + /* + * It's possible that the new sizes of the two consecutive + * size truncations will be in the same fscrypt last block, + * and we need to truncate the corresponding page caches + * anyway. + */ + if (ceph_seq_cmp(truncate_seq, ci->i_truncate_seq) >= 0) { + dout("%s truncate_size %lld -> %llu, encrypted %d\n", __func__, + ci->i_truncate_size, truncate_size, !!IS_ENCRYPTED(inode)); + ci->i_truncate_size = truncate_size; + + if (IS_ENCRYPTED(inode)) { + dout("%s truncate_pagecache_size %lld -> %llu\n", + __func__, ci->i_truncate_pagecache_size, size); + ci->i_truncate_pagecache_size = size; + } else { + ci->i_truncate_pagecache_size = truncate_size; + } } return queue_trunc; } @@ -752,6 +881,34 @@ void ceph_fill_file_time(struct inode *inode, int issued, inode, time_warp_seq, ci->i_time_warp_seq); } +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) +static int decode_encrypted_symlink(const char *encsym, int enclen, u8 **decsym) +{ + int declen; + u8 *sym; + + sym = kmalloc(enclen + 1, GFP_NOFS); + if (!sym) + return -ENOMEM; + + declen = ceph_base64_decode(encsym, enclen, sym); + if (declen < 0) { + pr_err("%s: can't decode symlink (%d). Content: %.*s\n", + __func__, declen, enclen, encsym); + kfree(sym); + return -EIO; + } + sym[declen + 1] = '\0'; + *decsym = sym; + return declen; +} +#else +static int decode_encrypted_symlink(const char *encsym, int symlen, u8 **decsym) +{ + return -EOPNOTSUPP; +} +#endif + /* * Populate an inode based on info from mds. May be called on new or * existing inodes. @@ -857,15 +1014,20 @@ int ceph_fill_inode(struct inode *inode, struct page *locked_page, issued |= __ceph_caps_dirty(ci); new_issued = ~issued & info_caps; - /* directories have fl_stripe_unit set to zero */ - if (le32_to_cpu(info->layout.fl_stripe_unit)) - inode->i_blkbits = - fls(le32_to_cpu(info->layout.fl_stripe_unit)) - 1; - else - inode->i_blkbits = CEPH_BLOCK_SHIFT; - __ceph_update_quota(ci, iinfo->max_bytes, iinfo->max_files); +#ifdef CONFIG_FS_ENCRYPTION + if (iinfo->fscrypt_auth_len && + ((inode->i_state & I_NEW) || (ci->fscrypt_auth_len == 0))) { + kfree(ci->fscrypt_auth); + ci->fscrypt_auth_len = iinfo->fscrypt_auth_len; + ci->fscrypt_auth = iinfo->fscrypt_auth; + iinfo->fscrypt_auth = NULL; + iinfo->fscrypt_auth_len = 0; + inode_set_flags(inode, S_ENCRYPTED, S_ENCRYPTED); + } +#endif + if ((new_version || (new_issued & CEPH_CAP_AUTH_SHARED)) && (issued & CEPH_CAP_AUTH_EXCL) == 0) { inode->i_mode = mode; @@ -878,6 +1040,15 @@ int ceph_fill_inode(struct inode *inode, struct page *locked_page, ceph_decode_timespec64(&ci->i_snap_btime, &iinfo->snap_btime); } + /* directories have fl_stripe_unit set to zero */ + if (IS_ENCRYPTED(inode)) + inode->i_blkbits = CEPH_FSCRYPT_BLOCK_SHIFT; + else if (le32_to_cpu(info->layout.fl_stripe_unit)) + inode->i_blkbits = + fls(le32_to_cpu(info->layout.fl_stripe_unit)) - 1; + else + inode->i_blkbits = CEPH_BLOCK_SHIFT; + if ((new_version || (new_issued & CEPH_CAP_LINK_SHARED)) && (issued & CEPH_CAP_LINK_EXCL) == 0) set_nlink(inode, le32_to_cpu(info->nlink)); @@ -899,6 +1070,7 @@ int ceph_fill_inode(struct inode *inode, struct page *locked_page, if (new_version || (new_issued & (CEPH_CAP_ANY_FILE_RD | CEPH_CAP_ANY_FILE_WR))) { + u64 size = le64_to_cpu(info->size); s64 old_pool = ci->i_layout.pool_id; struct ceph_string *old_ns; @@ -912,10 +1084,22 @@ int ceph_fill_inode(struct inode *inode, struct page *locked_page, pool_ns = old_ns; + if (IS_ENCRYPTED(inode) && size && + iinfo->fscrypt_file_len == sizeof(__le64)) { + u64 fsize = __le64_to_cpu(*(__le64 *)iinfo->fscrypt_file); + + if (size == round_up(fsize, CEPH_FSCRYPT_BLOCK_SIZE)) { + size = fsize; + } else { + pr_warn("fscrypt size mismatch: size=%llu fscrypt_file=%llu, discarding fscrypt_file size.\n", + info->size, size); + } + } + queue_trunc = ceph_fill_file_size(inode, issued, le32_to_cpu(info->truncate_seq), le64_to_cpu(info->truncate_size), - le64_to_cpu(info->size)); + size); /* only update max_size on auth cap */ if ((info->cap.flags & CEPH_CAP_FLAG_AUTH) && ci->i_max_size != le64_to_cpu(info->max_size)) { @@ -975,26 +1159,42 @@ int ceph_fill_inode(struct inode *inode, struct page *locked_page, inode->i_fop = &ceph_file_fops; break; case S_IFLNK: - inode->i_op = &ceph_symlink_iops; if (!ci->i_symlink) { u32 symlen = iinfo->symlink_len; char *sym; spin_unlock(&ci->i_ceph_lock); - if (symlen != i_size_read(inode)) { - pr_err("%s %llx.%llx BAD symlink " - "size %lld\n", __func__, - ceph_vinop(inode), - i_size_read(inode)); + if (IS_ENCRYPTED(inode)) { + if (symlen != i_size_read(inode)) + pr_err("%s %llx.%llx BAD symlink size %lld\n", + __func__, ceph_vinop(inode), + i_size_read(inode)); + + err = decode_encrypted_symlink(iinfo->symlink, + symlen, (u8 **)&sym); + if (err < 0) { + pr_err("%s decoding encrypted symlink failed: %d\n", + __func__, err); + goto out; + } + symlen = err; i_size_write(inode, symlen); inode->i_blocks = calc_inode_blocks(symlen); - } + } else { + if (symlen != i_size_read(inode)) { + pr_err("%s %llx.%llx BAD symlink size %lld\n", + __func__, ceph_vinop(inode), + i_size_read(inode)); + i_size_write(inode, symlen); + inode->i_blocks = calc_inode_blocks(symlen); + } - err = -ENOMEM; - sym = kstrndup(iinfo->symlink, symlen, GFP_NOFS); - if (!sym) - goto out; + err = -ENOMEM; + sym = kstrndup(iinfo->symlink, symlen, GFP_NOFS); + if (!sym) + goto out; + } spin_lock(&ci->i_ceph_lock); if (!ci->i_symlink) @@ -1002,7 +1202,17 @@ int ceph_fill_inode(struct inode *inode, struct page *locked_page, else kfree(sym); /* lost a race */ } - inode->i_link = ci->i_symlink; + + if (IS_ENCRYPTED(inode)) { + /* + * Encrypted symlinks need to be decrypted before we can + * cache their targets in i_link. Don't touch it here. + */ + inode->i_op = &ceph_encrypted_symlink_iops; + } else { + inode->i_link = ci->i_symlink; + inode->i_op = &ceph_symlink_iops; + } break; case S_IFDIR: inode->i_op = &ceph_dir_iops; @@ -1310,8 +1520,15 @@ int ceph_fill_trace(struct super_block *sb, struct ceph_mds_request *req) if (dir && req->r_op == CEPH_MDS_OP_LOOKUPNAME && test_bit(CEPH_MDS_R_PARENT_LOCKED, &req->r_req_flags) && !test_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags)) { + bool is_nokey = false; struct qstr dname; struct dentry *dn, *parent; + struct fscrypt_str oname = FSTR_INIT(NULL, 0); + struct ceph_fname fname = { .dir = dir, + .name = rinfo->dname, + .ctext = rinfo->altname, + .name_len = rinfo->dname_len, + .ctext_len = rinfo->altname_len }; BUG_ON(!rinfo->head->is_target); BUG_ON(req->r_dentry); @@ -1319,8 +1536,20 @@ int ceph_fill_trace(struct super_block *sb, struct ceph_mds_request *req) parent = d_find_any_alias(dir); BUG_ON(!parent); - dname.name = rinfo->dname; - dname.len = rinfo->dname_len; + err = ceph_fname_alloc_buffer(dir, &oname); + if (err < 0) { + dput(parent); + goto done; + } + + err = ceph_fname_to_usr(&fname, NULL, &oname, &is_nokey); + if (err < 0) { + dput(parent); + ceph_fname_free_buffer(dir, &oname); + goto done; + } + dname.name = oname.name; + dname.len = oname.len; dname.hash = full_name_hash(parent, dname.name, dname.len); tvino.ino = le64_to_cpu(rinfo->targeti.in->ino); tvino.snap = le64_to_cpu(rinfo->targeti.in->snapid); @@ -1335,9 +1564,15 @@ retry_lookup: dname.len, dname.name, dn); if (!dn) { dput(parent); + ceph_fname_free_buffer(dir, &oname); err = -ENOMEM; goto done; } + if (is_nokey) { + spin_lock(&dn->d_lock); + dn->d_flags |= DCACHE_NOKEY_NAME; + spin_unlock(&dn->d_lock); + } err = 0; } else if (d_really_is_positive(dn) && (ceph_ino(d_inode(dn)) != tvino.ino || @@ -1349,6 +1584,7 @@ retry_lookup: dput(dn); goto retry_lookup; } + ceph_fname_free_buffer(dir, &oname); req->r_dentry = dn; dput(parent); @@ -1552,7 +1788,7 @@ static int readdir_prepopulate_inodes_only(struct ceph_mds_request *req, vino.ino = le64_to_cpu(rde->inode.in->ino); vino.snap = le64_to_cpu(rde->inode.in->snapid); - in = ceph_get_inode(req->r_dentry->d_sb, vino); + in = ceph_get_inode(req->r_dentry->d_sb, vino, NULL); if (IS_ERR(in)) { err = PTR_ERR(in); dout("new_inode badness got %d\n", err); @@ -1630,7 +1866,8 @@ int ceph_readdir_prepopulate(struct ceph_mds_request *req, struct ceph_mds_session *session) { struct dentry *parent = req->r_dentry; - struct ceph_inode_info *ci = ceph_inode(d_inode(parent)); + struct inode *inode = d_inode(parent); + struct ceph_inode_info *ci = ceph_inode(inode); struct ceph_mds_reply_info_parsed *rinfo = &req->r_reply_info; struct qstr dname; struct dentry *dn; @@ -1704,9 +1941,7 @@ int ceph_readdir_prepopulate(struct ceph_mds_request *req, tvino.snap = le64_to_cpu(rde->inode.in->snapid); if (rinfo->hash_order) { - u32 hash = ceph_str_hash(ci->i_dir_layout.dl_dir_hash, - rde->name, rde->name_len); - hash = ceph_frag_value(hash); + u32 hash = ceph_frag_value(rde->raw_hash); if (hash != last_hash) fpos_offset = 2; last_hash = hash; @@ -1729,6 +1964,11 @@ retry_lookup: err = -ENOMEM; goto out; } + if (rde->is_nokey) { + spin_lock(&dn->d_lock); + dn->d_flags |= DCACHE_NOKEY_NAME; + spin_unlock(&dn->d_lock); + } } else if (d_really_is_positive(dn) && (ceph_ino(d_inode(dn)) != tvino.ino || ceph_snap(d_inode(dn)) != tvino.snap)) { @@ -1754,7 +1994,7 @@ retry_lookup: if (d_really_is_positive(dn)) { in = d_inode(dn); } else { - in = ceph_get_inode(parent->d_sb, tvino); + in = ceph_get_inode(parent->d_sb, tvino, NULL); if (IS_ERR(in)) { dout("new_inode badness\n"); d_drop(dn); @@ -1927,7 +2167,7 @@ void __ceph_do_pending_vmtruncate(struct inode *inode) retry: spin_lock(&ci->i_ceph_lock); if (ci->i_truncate_pending == 0) { - dout("__do_pending_vmtruncate %p none pending\n", inode); + dout("%s %p none pending\n", __func__, inode); spin_unlock(&ci->i_ceph_lock); mutex_unlock(&ci->i_truncate_mutex); return; @@ -1939,8 +2179,7 @@ retry: */ if (ci->i_wrbuffer_ref_head < ci->i_wrbuffer_ref) { spin_unlock(&ci->i_ceph_lock); - dout("__do_pending_vmtruncate %p flushing snaps first\n", - inode); + dout("%s %p flushing snaps first\n", __func__, inode); filemap_write_and_wait_range(&inode->i_data, 0, inode->i_sb->s_maxbytes); goto retry; @@ -1949,9 +2188,9 @@ retry: /* there should be no reader or writer */ WARN_ON_ONCE(ci->i_rd_ref || ci->i_wr_ref); - to = ci->i_truncate_size; + to = ci->i_truncate_pagecache_size; wrbuffer_refs = ci->i_wrbuffer_ref; - dout("__do_pending_vmtruncate %p (%d) to %lld\n", inode, + dout("%s %p (%d) to %lld\n", __func__, inode, ci->i_truncate_pending, to); spin_unlock(&ci->i_ceph_lock); @@ -1959,7 +2198,7 @@ retry: truncate_pagecache(inode, to); spin_lock(&ci->i_ceph_lock); - if (to == ci->i_truncate_size) { + if (to == ci->i_truncate_pagecache_size) { ci->i_truncate_pending = 0; finish = 1; } @@ -2000,6 +2239,32 @@ static void ceph_inode_work(struct work_struct *work) iput(inode); } +static const char *ceph_encrypted_get_link(struct dentry *dentry, + struct inode *inode, + struct delayed_call *done) +{ + struct ceph_inode_info *ci = ceph_inode(inode); + + if (!dentry) + return ERR_PTR(-ECHILD); + + return fscrypt_get_symlink(inode, ci->i_symlink, i_size_read(inode), + done); +} + +static int ceph_encrypted_symlink_getattr(struct mnt_idmap *idmap, + const struct path *path, + struct kstat *stat, u32 request_mask, + unsigned int query_flags) +{ + int ret; + + ret = ceph_getattr(idmap, path, stat, request_mask, query_flags); + if (ret) + return ret; + return fscrypt_symlink_getattr(path, stat); +} + /* * symlinks */ @@ -2010,20 +2275,173 @@ static const struct inode_operations ceph_symlink_iops = { .listxattr = ceph_listxattr, }; -int __ceph_setattr(struct inode *inode, struct iattr *attr) +static const struct inode_operations ceph_encrypted_symlink_iops = { + .get_link = ceph_encrypted_get_link, + .setattr = ceph_setattr, + .getattr = ceph_encrypted_symlink_getattr, + .listxattr = ceph_listxattr, +}; + +/* + * Transfer the encrypted last block to the MDS and the MDS + * will help update it when truncating a smaller size. + * + * We don't support a PAGE_SIZE that is smaller than the + * CEPH_FSCRYPT_BLOCK_SIZE. + */ +static int fill_fscrypt_truncate(struct inode *inode, + struct ceph_mds_request *req, + struct iattr *attr) +{ + struct ceph_inode_info *ci = ceph_inode(inode); + int boff = attr->ia_size % CEPH_FSCRYPT_BLOCK_SIZE; + loff_t pos, orig_pos = round_down(attr->ia_size, + CEPH_FSCRYPT_BLOCK_SIZE); + u64 block = orig_pos >> CEPH_FSCRYPT_BLOCK_SHIFT; + struct ceph_pagelist *pagelist = NULL; + struct kvec iov = {0}; + struct iov_iter iter; + struct page *page = NULL; + struct ceph_fscrypt_truncate_size_header header; + int retry_op = 0; + int len = CEPH_FSCRYPT_BLOCK_SIZE; + loff_t i_size = i_size_read(inode); + int got, ret, issued; + u64 objver; + + ret = __ceph_get_caps(inode, NULL, CEPH_CAP_FILE_RD, 0, -1, &got); + if (ret < 0) + return ret; + + issued = __ceph_caps_issued(ci, NULL); + + dout("%s size %lld -> %lld got cap refs on %s, issued %s\n", __func__, + i_size, attr->ia_size, ceph_cap_string(got), + ceph_cap_string(issued)); + + /* Try to writeback the dirty pagecaches */ + if (issued & (CEPH_CAP_FILE_BUFFER)) { + loff_t lend = orig_pos + CEPH_FSCRYPT_BLOCK_SHIFT - 1; + + ret = filemap_write_and_wait_range(inode->i_mapping, + orig_pos, lend); + if (ret < 0) + goto out; + } + + page = __page_cache_alloc(GFP_KERNEL); + if (page == NULL) { + ret = -ENOMEM; + goto out; + } + + pagelist = ceph_pagelist_alloc(GFP_KERNEL); + if (!pagelist) { + ret = -ENOMEM; + goto out; + } + + iov.iov_base = kmap_local_page(page); + iov.iov_len = len; + iov_iter_kvec(&iter, READ, &iov, 1, len); + + pos = orig_pos; + ret = __ceph_sync_read(inode, &pos, &iter, &retry_op, &objver); + if (ret < 0) + goto out; + + /* Insert the header first */ + header.ver = 1; + header.compat = 1; + header.change_attr = cpu_to_le64(inode_peek_iversion_raw(inode)); + + /* + * Always set the block_size to CEPH_FSCRYPT_BLOCK_SIZE, + * because in MDS it may need this to do the truncate. + */ + header.block_size = cpu_to_le32(CEPH_FSCRYPT_BLOCK_SIZE); + + /* + * If we hit a hole here, we should just skip filling + * the fscrypt for the request, because once the fscrypt + * is enabled, the file will be split into many blocks + * with the size of CEPH_FSCRYPT_BLOCK_SIZE, if there + * has a hole, the hole size should be multiple of block + * size. + * + * If the Rados object doesn't exist, it will be set to 0. + */ + if (!objver) { + dout("%s hit hole, ppos %lld < size %lld\n", __func__, + pos, i_size); + + header.data_len = cpu_to_le32(8 + 8 + 4); + header.file_offset = 0; + ret = 0; + } else { + header.data_len = cpu_to_le32(8 + 8 + 4 + CEPH_FSCRYPT_BLOCK_SIZE); + header.file_offset = cpu_to_le64(orig_pos); + + dout("%s encrypt block boff/bsize %d/%lu\n", __func__, + boff, CEPH_FSCRYPT_BLOCK_SIZE); + + /* truncate and zero out the extra contents for the last block */ + memset(iov.iov_base + boff, 0, PAGE_SIZE - boff); + + /* encrypt the last block */ + ret = ceph_fscrypt_encrypt_block_inplace(inode, page, + CEPH_FSCRYPT_BLOCK_SIZE, + 0, block, + GFP_KERNEL); + if (ret) + goto out; + } + + /* Insert the header */ + ret = ceph_pagelist_append(pagelist, &header, sizeof(header)); + if (ret) + goto out; + + if (header.block_size) { + /* Append the last block contents to pagelist */ + ret = ceph_pagelist_append(pagelist, iov.iov_base, + CEPH_FSCRYPT_BLOCK_SIZE); + if (ret) + goto out; + } + req->r_pagelist = pagelist; +out: + dout("%s %p size dropping cap refs on %s\n", __func__, + inode, ceph_cap_string(got)); + ceph_put_cap_refs(ci, got); + if (iov.iov_base) + kunmap_local(iov.iov_base); + if (page) + __free_pages(page, 0); + if (ret && pagelist) + ceph_pagelist_release(pagelist); + return ret; +} + +int __ceph_setattr(struct inode *inode, struct iattr *attr, + struct ceph_iattr *cia) { struct ceph_inode_info *ci = ceph_inode(inode); unsigned int ia_valid = attr->ia_valid; struct ceph_mds_request *req; struct ceph_mds_client *mdsc = ceph_sb_to_client(inode->i_sb)->mdsc; struct ceph_cap_flush *prealloc_cf; + loff_t isize = i_size_read(inode); int issued; int release = 0, dirtied = 0; int mask = 0; int err = 0; int inode_dirty_flags = 0; bool lock_snap_rwsem = false; + bool fill_fscrypt; + int truncate_retry = 20; /* The RMW will take around 50ms */ +retry: prealloc_cf = ceph_alloc_cap_flush(); if (!prealloc_cf) return -ENOMEM; @@ -2035,6 +2453,7 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr) return PTR_ERR(req); } + fill_fscrypt = false; spin_lock(&ci->i_ceph_lock); issued = __ceph_caps_issued(ci, NULL); @@ -2050,6 +2469,43 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr) } dout("setattr %p issued %s\n", inode, ceph_cap_string(issued)); +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) + if (cia && cia->fscrypt_auth) { + u32 len = ceph_fscrypt_auth_len(cia->fscrypt_auth); + + if (len > sizeof(*cia->fscrypt_auth)) { + err = -EINVAL; + spin_unlock(&ci->i_ceph_lock); + goto out; + } + + dout("setattr %llx:%llx fscrypt_auth len %u to %u)\n", + ceph_vinop(inode), ci->fscrypt_auth_len, len); + + /* It should never be re-set once set */ + WARN_ON_ONCE(ci->fscrypt_auth); + + if (issued & CEPH_CAP_AUTH_EXCL) { + dirtied |= CEPH_CAP_AUTH_EXCL; + kfree(ci->fscrypt_auth); + ci->fscrypt_auth = (u8 *)cia->fscrypt_auth; + ci->fscrypt_auth_len = len; + } else if ((issued & CEPH_CAP_AUTH_SHARED) == 0 || + ci->fscrypt_auth_len != len || + memcmp(ci->fscrypt_auth, cia->fscrypt_auth, len)) { + req->r_fscrypt_auth = cia->fscrypt_auth; + mask |= CEPH_SETATTR_FSCRYPT_AUTH; + release |= CEPH_CAP_AUTH_SHARED; + } + cia->fscrypt_auth = NULL; + } +#else + if (cia && cia->fscrypt_auth) { + err = -EINVAL; + spin_unlock(&ci->i_ceph_lock); + goto out; + } +#endif /* CONFIG_FS_ENCRYPTION */ if (ia_valid & ATTR_UID) { dout("setattr %p uid %d -> %d\n", inode, @@ -2119,10 +2575,27 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr) } } if (ia_valid & ATTR_SIZE) { - loff_t isize = i_size_read(inode); - dout("setattr %p size %lld -> %lld\n", inode, isize, attr->ia_size); - if ((issued & CEPH_CAP_FILE_EXCL) && attr->ia_size >= isize) { + /* + * Only when the new size is smaller and not aligned to + * CEPH_FSCRYPT_BLOCK_SIZE will the RMW is needed. + */ + if (IS_ENCRYPTED(inode) && attr->ia_size < isize && + (attr->ia_size % CEPH_FSCRYPT_BLOCK_SIZE)) { + mask |= CEPH_SETATTR_SIZE; + release |= CEPH_CAP_FILE_SHARED | CEPH_CAP_FILE_EXCL | + CEPH_CAP_FILE_RD | CEPH_CAP_FILE_WR; + set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags); + mask |= CEPH_SETATTR_FSCRYPT_FILE; + req->r_args.setattr.size = + cpu_to_le64(round_up(attr->ia_size, + CEPH_FSCRYPT_BLOCK_SIZE)); + req->r_args.setattr.old_size = + cpu_to_le64(round_up(isize, + CEPH_FSCRYPT_BLOCK_SIZE)); + req->r_fscrypt_file = attr->ia_size; + fill_fscrypt = true; + } else if ((issued & CEPH_CAP_FILE_EXCL) && attr->ia_size >= isize) { if (attr->ia_size > isize) { i_size_write(inode, attr->ia_size); inode->i_blocks = calc_inode_blocks(attr->ia_size); @@ -2132,11 +2605,24 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr) } } else if ((issued & CEPH_CAP_FILE_SHARED) == 0 || attr->ia_size != isize) { - req->r_args.setattr.size = cpu_to_le64(attr->ia_size); - req->r_args.setattr.old_size = cpu_to_le64(isize); mask |= CEPH_SETATTR_SIZE; release |= CEPH_CAP_FILE_SHARED | CEPH_CAP_FILE_EXCL | CEPH_CAP_FILE_RD | CEPH_CAP_FILE_WR; + if (IS_ENCRYPTED(inode) && attr->ia_size) { + set_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags); + mask |= CEPH_SETATTR_FSCRYPT_FILE; + req->r_args.setattr.size = + cpu_to_le64(round_up(attr->ia_size, + CEPH_FSCRYPT_BLOCK_SIZE)); + req->r_args.setattr.old_size = + cpu_to_le64(round_up(isize, + CEPH_FSCRYPT_BLOCK_SIZE)); + req->r_fscrypt_file = attr->ia_size; + } else { + req->r_args.setattr.size = cpu_to_le64(attr->ia_size); + req->r_args.setattr.old_size = cpu_to_le64(isize); + req->r_fscrypt_file = 0; + } } } if (ia_valid & ATTR_MTIME) { @@ -2199,8 +2685,10 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr) release &= issued; spin_unlock(&ci->i_ceph_lock); - if (lock_snap_rwsem) + if (lock_snap_rwsem) { up_read(&mdsc->snap_rwsem); + lock_snap_rwsem = false; + } if (inode_dirty_flags) __mark_inode_dirty(inode, inode_dirty_flags); @@ -2212,8 +2700,29 @@ int __ceph_setattr(struct inode *inode, struct iattr *attr) req->r_args.setattr.mask = cpu_to_le32(mask); req->r_num_caps = 1; req->r_stamp = attr->ia_ctime; + if (fill_fscrypt) { + err = fill_fscrypt_truncate(inode, req, attr); + if (err) + goto out; + } + + /* + * The truncate request will return -EAGAIN when the + * last block has been updated just before the MDS + * successfully gets the xlock for the FILE lock. To + * avoid corrupting the file contents we need to retry + * it. + */ err = ceph_mdsc_do_request(mdsc, NULL, req); + if (err == -EAGAIN && truncate_retry--) { + dout("setattr %p result=%d (%s locally, %d remote), retry it!\n", + inode, err, ceph_cap_string(dirtied), mask); + ceph_mdsc_put_request(req); + ceph_free_cap_flush(prealloc_cf); + goto retry; + } } +out: dout("setattr %p result=%d (%s locally, %d remote)\n", inode, err, ceph_cap_string(dirtied), mask); @@ -2242,6 +2751,10 @@ int ceph_setattr(struct mnt_idmap *idmap, struct dentry *dentry, if (ceph_inode_is_shutdown(inode)) return -ESTALE; + err = fscrypt_prepare_setattr(dentry, attr); + if (err) + return err; + err = setattr_prepare(&nop_mnt_idmap, dentry, attr); if (err != 0) return err; @@ -2254,7 +2767,7 @@ int ceph_setattr(struct mnt_idmap *idmap, struct dentry *dentry, ceph_quota_is_max_bytes_exceeded(inode, attr->ia_size)) return -EDQUOT; - err = __ceph_setattr(inode, attr); + err = __ceph_setattr(inode, attr, NULL); if (err >= 0 && (attr->ia_valid & ATTR_MODE)) err = posix_acl_chmod(&nop_mnt_idmap, dentry, attr->ia_mode); @@ -2525,8 +3038,12 @@ int ceph_getattr(struct mnt_idmap *idmap, const struct path *path, stat->nlink = 1 + 1 + ci->i_subdirs; } - stat->attributes_mask |= STATX_ATTR_CHANGE_MONOTONIC; stat->attributes |= STATX_ATTR_CHANGE_MONOTONIC; + if (IS_ENCRYPTED(inode)) + stat->attributes |= STATX_ATTR_ENCRYPTED; + stat->attributes_mask |= (STATX_ATTR_CHANGE_MONOTONIC | + STATX_ATTR_ENCRYPTED); + stat->result_mask = request_mask & valid_mask; return err; } diff --git a/fs/ceph/ioctl.c b/fs/ceph/ioctl.c index deac817647eb..91a84917d203 100644 --- a/fs/ceph/ioctl.c +++ b/fs/ceph/ioctl.c @@ -6,6 +6,7 @@ #include "mds_client.h" #include "ioctl.h" #include +#include /* * ioctls @@ -268,9 +269,96 @@ static long ceph_ioctl_syncio(struct file *file) return 0; } +static int vet_mds_for_fscrypt(struct file *file) +{ + int i, ret = -EOPNOTSUPP; + struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(file_inode(file)->i_sb); + + mutex_lock(&mdsc->mutex); + for (i = 0; i < mdsc->max_sessions; i++) { + struct ceph_mds_session *s = mdsc->sessions[i]; + + if (!s) + continue; + if (test_bit(CEPHFS_FEATURE_ALTERNATE_NAME, &s->s_features)) + ret = 0; + break; + } + mutex_unlock(&mdsc->mutex); + return ret; +} + +static long ceph_set_encryption_policy(struct file *file, unsigned long arg) +{ + int ret, got = 0; + struct inode *inode = file_inode(file); + struct ceph_inode_info *ci = ceph_inode(inode); + + /* encrypted directories can't have striped layout */ + if (ci->i_layout.stripe_count > 1) + return -EINVAL; + + ret = vet_mds_for_fscrypt(file); + if (ret) + return ret; + + /* + * Ensure we hold these caps so that we _know_ that the rstats check + * in the empty_dir check is reliable. + */ + ret = ceph_get_caps(file, CEPH_CAP_FILE_SHARED, 0, -1, &got); + if (ret) + return ret; + + ret = fscrypt_ioctl_set_policy(file, (const void __user *)arg); + if (got) + ceph_put_cap_refs(ci, got); + + return ret; +} + +static const char *ceph_ioctl_cmd_name(const unsigned int cmd) +{ + switch (cmd) { + case CEPH_IOC_GET_LAYOUT: + return "get_layout"; + case CEPH_IOC_SET_LAYOUT: + return "set_layout"; + case CEPH_IOC_SET_LAYOUT_POLICY: + return "set_layout_policy"; + case CEPH_IOC_GET_DATALOC: + return "get_dataloc"; + case CEPH_IOC_LAZYIO: + return "lazyio"; + case CEPH_IOC_SYNCIO: + return "syncio"; + case FS_IOC_SET_ENCRYPTION_POLICY: + return "set_encryption_policy"; + case FS_IOC_GET_ENCRYPTION_POLICY: + return "get_encryption_policy"; + case FS_IOC_GET_ENCRYPTION_POLICY_EX: + return "get_encryption_policy_ex"; + case FS_IOC_ADD_ENCRYPTION_KEY: + return "add_encryption_key"; + case FS_IOC_REMOVE_ENCRYPTION_KEY: + return "remove_encryption_key"; + case FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS: + return "remove_encryption_key_all_users"; + case FS_IOC_GET_ENCRYPTION_KEY_STATUS: + return "get_encryption_key_status"; + case FS_IOC_GET_ENCRYPTION_NONCE: + return "get_encryption_nonce"; + default: + return "unknown"; + } +} + long ceph_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { - dout("ioctl file %p cmd %u arg %lu\n", file, cmd, arg); + int ret; + + dout("ioctl file %p cmd %s arg %lu\n", file, + ceph_ioctl_cmd_name(cmd), arg); switch (cmd) { case CEPH_IOC_GET_LAYOUT: return ceph_ioctl_get_layout(file, (void __user *)arg); @@ -289,6 +377,43 @@ long ceph_ioctl(struct file *file, unsigned int cmd, unsigned long arg) case CEPH_IOC_SYNCIO: return ceph_ioctl_syncio(file); + + case FS_IOC_SET_ENCRYPTION_POLICY: + return ceph_set_encryption_policy(file, arg); + + case FS_IOC_GET_ENCRYPTION_POLICY: + ret = vet_mds_for_fscrypt(file); + if (ret) + return ret; + return fscrypt_ioctl_get_policy(file, (void __user *)arg); + + case FS_IOC_GET_ENCRYPTION_POLICY_EX: + ret = vet_mds_for_fscrypt(file); + if (ret) + return ret; + return fscrypt_ioctl_get_policy_ex(file, (void __user *)arg); + + case FS_IOC_ADD_ENCRYPTION_KEY: + ret = vet_mds_for_fscrypt(file); + if (ret) + return ret; + return fscrypt_ioctl_add_key(file, (void __user *)arg); + + case FS_IOC_REMOVE_ENCRYPTION_KEY: + return fscrypt_ioctl_remove_key(file, (void __user *)arg); + + case FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS: + return fscrypt_ioctl_remove_key_all_users(file, + (void __user *)arg); + + case FS_IOC_GET_ENCRYPTION_KEY_STATUS: + return fscrypt_ioctl_get_key_status(file, (void __user *)arg); + + case FS_IOC_GET_ENCRYPTION_NONCE: + ret = vet_mds_for_fscrypt(file); + if (ret) + return ret; + return fscrypt_ioctl_get_nonce(file, (void __user *)arg); } return -ENOTTY; diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 5fb367b1d4b0..615db141b6c4 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -15,6 +15,7 @@ #include "super.h" #include "mds_client.h" +#include "crypto.h" #include #include @@ -184,8 +185,54 @@ static int parse_reply_info_in(void **p, void *end, info->rsnaps = 0; } + if (struct_v >= 5) { + u32 alen; + + ceph_decode_32_safe(p, end, alen, bad); + + while (alen--) { + u32 len; + + /* key */ + ceph_decode_32_safe(p, end, len, bad); + ceph_decode_skip_n(p, end, len, bad); + /* value */ + ceph_decode_32_safe(p, end, len, bad); + ceph_decode_skip_n(p, end, len, bad); + } + } + + /* fscrypt flag -- ignore */ + if (struct_v >= 6) + ceph_decode_skip_8(p, end, bad); + + info->fscrypt_auth = NULL; + info->fscrypt_auth_len = 0; + info->fscrypt_file = NULL; + info->fscrypt_file_len = 0; + if (struct_v >= 7) { + ceph_decode_32_safe(p, end, info->fscrypt_auth_len, bad); + if (info->fscrypt_auth_len) { + info->fscrypt_auth = kmalloc(info->fscrypt_auth_len, + GFP_KERNEL); + if (!info->fscrypt_auth) + return -ENOMEM; + ceph_decode_copy_safe(p, end, info->fscrypt_auth, + info->fscrypt_auth_len, bad); + } + ceph_decode_32_safe(p, end, info->fscrypt_file_len, bad); + if (info->fscrypt_file_len) { + info->fscrypt_file = kmalloc(info->fscrypt_file_len, + GFP_KERNEL); + if (!info->fscrypt_file) + return -ENOMEM; + ceph_decode_copy_safe(p, end, info->fscrypt_file, + info->fscrypt_file_len, bad); + } + } *p = end; } else { + /* legacy (unversioned) struct */ if (features & CEPH_FEATURE_MDS_INLINE_DATA) { ceph_decode_64_safe(p, end, info->inline_version, bad); ceph_decode_32_safe(p, end, info->inline_len, bad); @@ -263,27 +310,47 @@ bad: static int parse_reply_info_lease(void **p, void *end, struct ceph_mds_reply_lease **lease, - u64 features) + u64 features, u32 *altname_len, u8 **altname) { + u8 struct_v; + u32 struct_len; + void *lend; + if (features == (u64)-1) { - u8 struct_v, struct_compat; - u32 struct_len; + u8 struct_compat; + ceph_decode_8_safe(p, end, struct_v, bad); ceph_decode_8_safe(p, end, struct_compat, bad); + /* struct_v is expected to be >= 1. we only understand * encoding whose struct_compat == 1. */ if (!struct_v || struct_compat != 1) goto bad; + ceph_decode_32_safe(p, end, struct_len, bad); - ceph_decode_need(p, end, struct_len, bad); - end = *p + struct_len; + } else { + struct_len = sizeof(**lease); + *altname_len = 0; + *altname = NULL; } - ceph_decode_need(p, end, sizeof(**lease), bad); + lend = *p + struct_len; + ceph_decode_need(p, end, struct_len, bad); *lease = *p; *p += sizeof(**lease); - if (features == (u64)-1) - *p = end; + + if (features == (u64)-1) { + if (struct_v >= 2) { + ceph_decode_32_safe(p, end, *altname_len, bad); + ceph_decode_need(p, end, *altname_len, bad); + *altname = *p; + *p += *altname_len; + } else { + *altname = NULL; + *altname_len = 0; + } + } + *p = lend; return 0; bad: return -EIO; @@ -313,7 +380,8 @@ static int parse_reply_info_trace(void **p, void *end, info->dname = *p; *p += info->dname_len; - err = parse_reply_info_lease(p, end, &info->dlease, features); + err = parse_reply_info_lease(p, end, &info->dlease, features, + &info->altname_len, &info->altname); if (err < 0) goto out_bad; } @@ -339,9 +407,10 @@ out_bad: * parse readdir results */ static int parse_reply_info_readdir(void **p, void *end, - struct ceph_mds_reply_info_parsed *info, - u64 features) + struct ceph_mds_request *req, + u64 features) { + struct ceph_mds_reply_info_parsed *info = &req->r_reply_info; u32 num, i = 0; int err; @@ -371,18 +440,87 @@ static int parse_reply_info_readdir(void **p, void *end, info->dir_nr = num; while (num) { + struct inode *inode = d_inode(req->r_dentry); + struct ceph_inode_info *ci = ceph_inode(inode); struct ceph_mds_reply_dir_entry *rde = info->dir_entries + i; + struct fscrypt_str tname = FSTR_INIT(NULL, 0); + struct fscrypt_str oname = FSTR_INIT(NULL, 0); + struct ceph_fname fname; + u32 altname_len, _name_len; + u8 *altname, *_name; + /* dentry */ - ceph_decode_32_safe(p, end, rde->name_len, bad); - ceph_decode_need(p, end, rde->name_len, bad); - rde->name = *p; - *p += rde->name_len; - dout("parsed dir dname '%.*s'\n", rde->name_len, rde->name); + ceph_decode_32_safe(p, end, _name_len, bad); + ceph_decode_need(p, end, _name_len, bad); + _name = *p; + *p += _name_len; + dout("parsed dir dname '%.*s'\n", _name_len, _name); + + if (info->hash_order) + rde->raw_hash = ceph_str_hash(ci->i_dir_layout.dl_dir_hash, + _name, _name_len); /* dentry lease */ - err = parse_reply_info_lease(p, end, &rde->lease, features); + err = parse_reply_info_lease(p, end, &rde->lease, features, + &altname_len, &altname); if (err) goto out_bad; + + /* + * Try to dencrypt the dentry names and update them + * in the ceph_mds_reply_dir_entry struct. + */ + fname.dir = inode; + fname.name = _name; + fname.name_len = _name_len; + fname.ctext = altname; + fname.ctext_len = altname_len; + /* + * The _name_len maybe larger than altname_len, such as + * when the human readable name length is in range of + * (CEPH_NOHASH_NAME_MAX, CEPH_NOHASH_NAME_MAX + SHA256_DIGEST_SIZE), + * then the copy in ceph_fname_to_usr will corrupt the + * data if there has no encryption key. + * + * Just set the no_copy flag and then if there has no + * encryption key the oname.name will be assigned to + * _name always. + */ + fname.no_copy = true; + if (altname_len == 0) { + /* + * Set tname to _name, and this will be used + * to do the base64_decode in-place. It's + * safe because the decoded string should + * always be shorter, which is 3/4 of origin + * string. + */ + tname.name = _name; + + /* + * Set oname to _name too, and this will be + * used to do the dencryption in-place. + */ + oname.name = _name; + oname.len = _name_len; + } else { + /* + * This will do the decryption only in-place + * from altname cryptext directly. + */ + oname.name = altname; + oname.len = altname_len; + } + rde->is_nokey = false; + err = ceph_fname_to_usr(&fname, &tname, &oname, &rde->is_nokey); + if (err) { + pr_err("%s unable to decode %.*s, got %d\n", __func__, + _name_len, _name, err); + goto out_bad; + } + rde->name = oname.name; + rde->name_len = oname.len; + /* inode */ err = parse_reply_info_in(p, end, &rde->inode, features); if (err < 0) @@ -581,15 +719,16 @@ bad: * parse extra results */ static int parse_reply_info_extra(void **p, void *end, - struct ceph_mds_reply_info_parsed *info, + struct ceph_mds_request *req, u64 features, struct ceph_mds_session *s) { + struct ceph_mds_reply_info_parsed *info = &req->r_reply_info; u32 op = le32_to_cpu(info->head->op); if (op == CEPH_MDS_OP_GETFILELOCK) return parse_reply_info_filelock(p, end, info, features); else if (op == CEPH_MDS_OP_READDIR || op == CEPH_MDS_OP_LSSNAP) - return parse_reply_info_readdir(p, end, info, features); + return parse_reply_info_readdir(p, end, req, features); else if (op == CEPH_MDS_OP_CREATE) return parse_reply_info_create(p, end, info, features, s); else if (op == CEPH_MDS_OP_GETVXATTR) @@ -602,9 +741,9 @@ static int parse_reply_info_extra(void **p, void *end, * parse entire mds reply */ static int parse_reply_info(struct ceph_mds_session *s, struct ceph_msg *msg, - struct ceph_mds_reply_info_parsed *info, - u64 features) + struct ceph_mds_request *req, u64 features) { + struct ceph_mds_reply_info_parsed *info = &req->r_reply_info; void *p, *end; u32 len; int err; @@ -626,7 +765,7 @@ static int parse_reply_info(struct ceph_mds_session *s, struct ceph_msg *msg, ceph_decode_32_safe(&p, end, len, bad); if (len > 0) { ceph_decode_need(&p, end, len, bad); - err = parse_reply_info_extra(&p, p+len, info, features, s); + err = parse_reply_info_extra(&p, p+len, req, features, s); if (err < 0) goto out_bad; } @@ -651,8 +790,21 @@ out_bad: static void destroy_reply_info(struct ceph_mds_reply_info_parsed *info) { + int i; + + kfree(info->diri.fscrypt_auth); + kfree(info->diri.fscrypt_file); + kfree(info->targeti.fscrypt_auth); + kfree(info->targeti.fscrypt_file); if (!info->dir_entries) return; + + for (i = 0; i < info->dir_nr; i++) { + struct ceph_mds_reply_dir_entry *rde = info->dir_entries + i; + + kfree(rde->inode.fscrypt_auth); + kfree(rde->inode.fscrypt_file); + } free_pages((unsigned long)info->dir_entries, get_order(info->dir_buf_size)); } @@ -945,6 +1097,7 @@ void ceph_mdsc_release_request(struct kref *kref) iput(req->r_parent); } iput(req->r_target_inode); + iput(req->r_new_inode); if (req->r_dentry) dput(req->r_dentry); if (req->r_old_dentry) @@ -965,6 +1118,8 @@ void ceph_mdsc_release_request(struct kref *kref) put_cred(req->r_cred); if (req->r_pagelist) ceph_pagelist_release(req->r_pagelist); + kfree(req->r_fscrypt_auth); + kfree(req->r_altname); put_request_session(req); ceph_unreserve_caps(req->r_mdsc, &req->r_caps_reservation); WARN_ON_ONCE(!list_empty(&req->r_wait)); @@ -2373,20 +2528,90 @@ static inline u64 __get_oldest_tid(struct ceph_mds_client *mdsc) return mdsc->oldest_tid; } -/* - * Build a dentry's path. Allocate on heap; caller must kfree. Based - * on build_path_from_dentry in fs/cifs/dir.c. +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) +static u8 *get_fscrypt_altname(const struct ceph_mds_request *req, u32 *plen) +{ + struct inode *dir = req->r_parent; + struct dentry *dentry = req->r_dentry; + u8 *cryptbuf = NULL; + u32 len = 0; + int ret = 0; + + /* only encode if we have parent and dentry */ + if (!dir || !dentry) + goto success; + + /* No-op unless this is encrypted */ + if (!IS_ENCRYPTED(dir)) + goto success; + + ret = ceph_fscrypt_prepare_readdir(dir); + if (ret < 0) + return ERR_PTR(ret); + + /* No key? Just ignore it. */ + if (!fscrypt_has_encryption_key(dir)) + goto success; + + if (!fscrypt_fname_encrypted_size(dir, dentry->d_name.len, NAME_MAX, + &len)) { + WARN_ON_ONCE(1); + return ERR_PTR(-ENAMETOOLONG); + } + + /* No need to append altname if name is short enough */ + if (len <= CEPH_NOHASH_NAME_MAX) { + len = 0; + goto success; + } + + cryptbuf = kmalloc(len, GFP_KERNEL); + if (!cryptbuf) + return ERR_PTR(-ENOMEM); + + ret = fscrypt_fname_encrypt(dir, &dentry->d_name, cryptbuf, len); + if (ret) { + kfree(cryptbuf); + return ERR_PTR(ret); + } +success: + *plen = len; + return cryptbuf; +} +#else +static u8 *get_fscrypt_altname(const struct ceph_mds_request *req, u32 *plen) +{ + *plen = 0; + return NULL; +} +#endif + +/** + * ceph_mdsc_build_path - build a path string to a given dentry + * @dentry: dentry to which path should be built + * @plen: returned length of string + * @pbase: returned base inode number + * @for_wire: is this path going to be sent to the MDS? * - * If @stop_on_nosnap, generate path relative to the first non-snapped - * inode. + * Build a string that represents the path to the dentry. This is mostly called + * for two different purposes: + * + * 1) we need to build a path string to send to the MDS (for_wire == true) + * 2) we need a path string for local presentation (e.g. debugfs) + * (for_wire == false) + * + * The path is built in reverse, starting with the dentry. Walk back up toward + * the root, building the path until the first non-snapped inode is reached + * (for_wire) or the root inode is reached (!for_wire). * * Encode hidden .snap dirs as a double /, i.e. * foo/.snap/bar -> foo//bar */ char *ceph_mdsc_build_path(struct dentry *dentry, int *plen, u64 *pbase, - int stop_on_nosnap) + int for_wire) { - struct dentry *temp; + struct dentry *cur; + struct inode *inode; char *path; int pos; unsigned seq; @@ -2403,34 +2628,72 @@ retry: path[pos] = '\0'; seq = read_seqbegin(&rename_lock); - rcu_read_lock(); - temp = dentry; + cur = dget(dentry); for (;;) { - struct inode *inode; + struct dentry *parent; - spin_lock(&temp->d_lock); - inode = d_inode(temp); + spin_lock(&cur->d_lock); + inode = d_inode(cur); if (inode && ceph_snap(inode) == CEPH_SNAPDIR) { dout("build_path path+%d: %p SNAPDIR\n", - pos, temp); - } else if (stop_on_nosnap && inode && dentry != temp && + pos, cur); + spin_unlock(&cur->d_lock); + parent = dget_parent(cur); + } else if (for_wire && inode && dentry != cur && ceph_snap(inode) == CEPH_NOSNAP) { - spin_unlock(&temp->d_lock); + spin_unlock(&cur->d_lock); pos++; /* get rid of any prepended '/' */ break; - } else { - pos -= temp->d_name.len; + } else if (!for_wire || !IS_ENCRYPTED(d_inode(cur->d_parent))) { + pos -= cur->d_name.len; if (pos < 0) { - spin_unlock(&temp->d_lock); + spin_unlock(&cur->d_lock); break; } - memcpy(path + pos, temp->d_name.name, temp->d_name.len); + memcpy(path + pos, cur->d_name.name, cur->d_name.len); + spin_unlock(&cur->d_lock); + parent = dget_parent(cur); + } else { + int len, ret; + char buf[NAME_MAX]; + + /* + * Proactively copy name into buf, in case we need to + * present it as-is. + */ + memcpy(buf, cur->d_name.name, cur->d_name.len); + len = cur->d_name.len; + spin_unlock(&cur->d_lock); + parent = dget_parent(cur); + + ret = ceph_fscrypt_prepare_readdir(d_inode(parent)); + if (ret < 0) { + dput(parent); + dput(cur); + return ERR_PTR(ret); + } + + if (fscrypt_has_encryption_key(d_inode(parent))) { + len = ceph_encode_encrypted_fname(d_inode(parent), + cur, buf); + if (len < 0) { + dput(parent); + dput(cur); + return ERR_PTR(len); + } + } + pos -= len; + if (pos < 0) { + dput(parent); + break; + } + memcpy(path + pos, buf, len); } - spin_unlock(&temp->d_lock); - temp = READ_ONCE(temp->d_parent); + dput(cur); + cur = parent; /* Are we at the root? */ - if (IS_ROOT(temp)) + if (IS_ROOT(cur)) break; /* Are we out of buffer? */ @@ -2439,8 +2702,9 @@ retry: path[pos] = '/'; } - base = ceph_ino(d_inode(temp)); - rcu_read_unlock(); + inode = d_inode(cur); + base = inode ? ceph_ino(inode) : 0; + dput(cur); if (read_seqretry(&rename_lock, seq)) goto retry; @@ -2450,8 +2714,8 @@ retry: * A rename didn't occur, but somehow we didn't end up where * we thought we would. Throw a warning and try again. */ - pr_warn("build_path did not end path lookup where " - "expected, pos is %d\n", pos); + pr_warn("build_path did not end path lookup where expected (pos = %d)\n", + pos); goto retry; } @@ -2471,7 +2735,8 @@ static int build_dentry_path(struct dentry *dentry, struct inode *dir, rcu_read_lock(); if (!dir) dir = d_inode_rcu(dentry->d_parent); - if (dir && parent_locked && ceph_snap(dir) == CEPH_NOSNAP) { + if (dir && parent_locked && ceph_snap(dir) == CEPH_NOSNAP && + !IS_ENCRYPTED(dir)) { *pino = ceph_ino(dir); rcu_read_unlock(); *ppath = dentry->d_name.name; @@ -2539,8 +2804,8 @@ static int set_request_path_attr(struct inode *rinode, struct dentry *rdentry, return r; } -static void encode_timestamp_and_gids(void **p, - const struct ceph_mds_request *req) +static void encode_mclientrequest_tail(void **p, + const struct ceph_mds_request *req) { struct ceph_timespec ts; int i; @@ -2548,11 +2813,43 @@ static void encode_timestamp_and_gids(void **p, ceph_encode_timespec64(&ts, &req->r_stamp); ceph_encode_copy(p, &ts, sizeof(ts)); - /* gid_list */ + /* v4: gid_list */ ceph_encode_32(p, req->r_cred->group_info->ngroups); for (i = 0; i < req->r_cred->group_info->ngroups; i++) ceph_encode_64(p, from_kgid(&init_user_ns, req->r_cred->group_info->gid[i])); + + /* v5: altname */ + ceph_encode_32(p, req->r_altname_len); + ceph_encode_copy(p, req->r_altname, req->r_altname_len); + + /* v6: fscrypt_auth and fscrypt_file */ + if (req->r_fscrypt_auth) { + u32 authlen = ceph_fscrypt_auth_len(req->r_fscrypt_auth); + + ceph_encode_32(p, authlen); + ceph_encode_copy(p, req->r_fscrypt_auth, authlen); + } else { + ceph_encode_32(p, 0); + } + if (test_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags)) { + ceph_encode_32(p, sizeof(__le64)); + ceph_encode_64(p, req->r_fscrypt_file); + } else { + ceph_encode_32(p, 0); + } +} + +static struct ceph_mds_request_head_legacy * +find_legacy_request_head(void *p, u64 features) +{ + bool legacy = !(features & CEPH_FEATURE_FS_BTIME); + struct ceph_mds_request_head_old *ohead; + + if (legacy) + return (struct ceph_mds_request_head_legacy *)p; + ohead = (struct ceph_mds_request_head_old *)p; + return (struct ceph_mds_request_head_legacy *)&ohead->oldest_client_tid; } /* @@ -2565,7 +2862,7 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, int mds = session->s_mds; struct ceph_mds_client *mdsc = session->s_mdsc; struct ceph_msg *msg; - struct ceph_mds_request_head_old *head; + struct ceph_mds_request_head_legacy *lhead; const char *path1 = NULL; const char *path2 = NULL; u64 ino1 = 0, ino2 = 0; @@ -2577,6 +2874,8 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, void *p, *end; int ret; bool legacy = !(session->s_con.peer_features & CEPH_FEATURE_FS_BTIME); + bool old_version = !test_bit(CEPHFS_FEATURE_32BITS_RETRY_FWD, + &session->s_features); ret = set_request_path_attr(req->r_inode, req->r_dentry, req->r_parent, req->r_path1, req->r_ino1.ino, @@ -2601,12 +2900,32 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, goto out_free1; } - len = legacy ? sizeof(*head) : sizeof(struct ceph_mds_request_head); - len += pathlen1 + pathlen2 + 2*(1 + sizeof(u32) + sizeof(u64)) + - sizeof(struct ceph_timespec); - len += sizeof(u32) + (sizeof(u64) * req->r_cred->group_info->ngroups); + req->r_altname = get_fscrypt_altname(req, &req->r_altname_len); + if (IS_ERR(req->r_altname)) { + msg = ERR_CAST(req->r_altname); + req->r_altname = NULL; + goto out_free2; + } - /* calculate (max) length for cap releases */ + /* + * For old cephs without supporting the 32bit retry/fwd feature + * it will copy the raw memories directly when decoding the + * requests. While new cephs will decode the head depending the + * version member, so we need to make sure it will be compatible + * with them both. + */ + if (legacy) + len = sizeof(struct ceph_mds_request_head_legacy); + else if (old_version) + len = sizeof(struct ceph_mds_request_head_old); + else + len = sizeof(struct ceph_mds_request_head); + + /* filepaths */ + len += 2 * (1 + sizeof(u32) + sizeof(u64)); + len += pathlen1 + pathlen2; + + /* cap releases */ len += sizeof(struct ceph_mds_request_release) * (!!req->r_inode_drop + !!req->r_dentry_drop + !!req->r_old_inode_drop + !!req->r_old_dentry_drop); @@ -2616,6 +2935,27 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, if (req->r_old_dentry_drop) len += pathlen2; + /* MClientRequest tail */ + + /* req->r_stamp */ + len += sizeof(struct ceph_timespec); + + /* gid list */ + len += sizeof(u32) + (sizeof(u64) * req->r_cred->group_info->ngroups); + + /* alternate name */ + len += sizeof(u32) + req->r_altname_len; + + /* fscrypt_auth */ + len += sizeof(u32); // fscrypt_auth + if (req->r_fscrypt_auth) + len += ceph_fscrypt_auth_len(req->r_fscrypt_auth); + + /* fscrypt_file */ + len += sizeof(u32); + if (test_bit(CEPH_MDS_R_FSCRYPT_FILE, &req->r_req_flags)) + len += sizeof(__le64); + msg = ceph_msg_new2(CEPH_MSG_CLIENT_REQUEST, len, 1, GFP_NOFS, false); if (!msg) { msg = ERR_PTR(-ENOMEM); @@ -2624,33 +2964,40 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, msg->hdr.tid = cpu_to_le64(req->r_tid); + lhead = find_legacy_request_head(msg->front.iov_base, + session->s_con.peer_features); + /* - * The old ceph_mds_request_head didn't contain a version field, and + * The ceph_mds_request_head_legacy didn't contain a version field, and * one was added when we moved the message version from 3->4. */ if (legacy) { msg->hdr.version = cpu_to_le16(3); - head = msg->front.iov_base; - p = msg->front.iov_base + sizeof(*head); - } else { - struct ceph_mds_request_head *new_head = msg->front.iov_base; + p = msg->front.iov_base + sizeof(*lhead); + } else if (old_version) { + struct ceph_mds_request_head_old *ohead = msg->front.iov_base; msg->hdr.version = cpu_to_le16(4); - new_head->version = cpu_to_le16(CEPH_MDS_REQUEST_HEAD_VERSION); - head = (struct ceph_mds_request_head_old *)&new_head->oldest_client_tid; - p = msg->front.iov_base + sizeof(*new_head); + ohead->version = cpu_to_le16(1); + p = msg->front.iov_base + sizeof(*ohead); + } else { + struct ceph_mds_request_head *nhead = msg->front.iov_base; + + msg->hdr.version = cpu_to_le16(6); + nhead->version = cpu_to_le16(CEPH_MDS_REQUEST_HEAD_VERSION); + p = msg->front.iov_base + sizeof(*nhead); } end = msg->front.iov_base + msg->front.iov_len; - head->mdsmap_epoch = cpu_to_le32(mdsc->mdsmap->m_epoch); - head->op = cpu_to_le32(req->r_op); - head->caller_uid = cpu_to_le32(from_kuid(&init_user_ns, - req->r_cred->fsuid)); - head->caller_gid = cpu_to_le32(from_kgid(&init_user_ns, - req->r_cred->fsgid)); - head->ino = cpu_to_le64(req->r_deleg_ino); - head->args = req->r_args; + lhead->mdsmap_epoch = cpu_to_le32(mdsc->mdsmap->m_epoch); + lhead->op = cpu_to_le32(req->r_op); + lhead->caller_uid = cpu_to_le32(from_kuid(&init_user_ns, + req->r_cred->fsuid)); + lhead->caller_gid = cpu_to_le32(from_kgid(&init_user_ns, + req->r_cred->fsgid)); + lhead->ino = cpu_to_le64(req->r_deleg_ino); + lhead->args = req->r_args; ceph_encode_filepath(&p, end, ino1, path1); ceph_encode_filepath(&p, end, ino2, path2); @@ -2665,15 +3012,23 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, req->r_inode ? req->r_inode : d_inode(req->r_dentry), mds, req->r_inode_drop, req->r_inode_unless, req->r_op == CEPH_MDS_OP_READDIR); - if (req->r_dentry_drop) - releases += ceph_encode_dentry_release(&p, req->r_dentry, + if (req->r_dentry_drop) { + ret = ceph_encode_dentry_release(&p, req->r_dentry, req->r_parent, mds, req->r_dentry_drop, req->r_dentry_unless); - if (req->r_old_dentry_drop) - releases += ceph_encode_dentry_release(&p, req->r_old_dentry, + if (ret < 0) + goto out_err; + releases += ret; + } + if (req->r_old_dentry_drop) { + ret = ceph_encode_dentry_release(&p, req->r_old_dentry, req->r_old_dentry_dir, mds, req->r_old_dentry_drop, req->r_old_dentry_unless); + if (ret < 0) + goto out_err; + releases += ret; + } if (req->r_old_inode_drop) releases += ceph_encode_inode_release(&p, d_inode(req->r_old_dentry), @@ -2684,9 +3039,9 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, p = msg->front.iov_base + req->r_request_release_offset; } - head->num_releases = cpu_to_le16(releases); + lhead->num_releases = cpu_to_le16(releases); - encode_timestamp_and_gids(&p, req); + encode_mclientrequest_tail(&p, req); if (WARN_ON_ONCE(p > end)) { ceph_msg_put(msg); @@ -2715,6 +3070,10 @@ out_free1: ceph_mdsc_free_path((char *)path1, pathlen1); out: return msg; +out_err: + ceph_msg_put(msg); + msg = ERR_PTR(ret); + goto out_free2; } /* @@ -2731,18 +3090,6 @@ static void complete_request(struct ceph_mds_client *mdsc, complete_all(&req->r_completion); } -static struct ceph_mds_request_head_old * -find_old_request_head(void *p, u64 features) -{ - bool legacy = !(features & CEPH_FEATURE_FS_BTIME); - struct ceph_mds_request_head *new_head; - - if (legacy) - return (struct ceph_mds_request_head_old *)p; - new_head = (struct ceph_mds_request_head *)p; - return (struct ceph_mds_request_head_old *)&new_head->oldest_client_tid; -} - /* * called under mdsc->mutex */ @@ -2752,29 +3099,28 @@ static int __prepare_send_request(struct ceph_mds_session *session, { int mds = session->s_mds; struct ceph_mds_client *mdsc = session->s_mdsc; - struct ceph_mds_request_head_old *rhead; + struct ceph_mds_request_head_legacy *lhead; + struct ceph_mds_request_head *nhead; struct ceph_msg *msg; - int flags = 0, max_retry; + int flags = 0, old_max_retry; + bool old_version = !test_bit(CEPHFS_FEATURE_32BITS_RETRY_FWD, + &session->s_features); /* - * The type of 'r_attempts' in kernel 'ceph_mds_request' - * is 'int', while in 'ceph_mds_request_head' the type of - * 'num_retry' is '__u8'. So in case the request retries - * exceeding 256 times, the MDS will receive a incorrect - * retry seq. - * - * In this case it's ususally a bug in MDS and continue - * retrying the request makes no sense. - * - * In future this could be fixed in ceph code, so avoid - * using the hardcode here. + * Avoid inifinite retrying after overflow. The client will + * increase the retry count and if the MDS is old version, + * so we limit to retry at most 256 times. */ - max_retry = sizeof_field(struct ceph_mds_request_head, num_retry); - max_retry = 1 << (max_retry * BITS_PER_BYTE); - if (req->r_attempts >= max_retry) { - pr_warn_ratelimited("%s request tid %llu seq overflow\n", - __func__, req->r_tid); - return -EMULTIHOP; + if (req->r_attempts) { + old_max_retry = sizeof_field(struct ceph_mds_request_head_old, + num_retry); + old_max_retry = 1 << (old_max_retry * BITS_PER_BYTE); + if ((old_version && req->r_attempts >= old_max_retry) || + ((uint32_t)req->r_attempts >= U32_MAX)) { + pr_warn_ratelimited("%s request tid %llu seq overflow\n", + __func__, req->r_tid); + return -EMULTIHOP; + } } req->r_attempts++; @@ -2800,23 +3146,27 @@ static int __prepare_send_request(struct ceph_mds_session *session, * d_move mangles the src name. */ msg = req->r_request; - rhead = find_old_request_head(msg->front.iov_base, - session->s_con.peer_features); + lhead = find_legacy_request_head(msg->front.iov_base, + session->s_con.peer_features); - flags = le32_to_cpu(rhead->flags); + flags = le32_to_cpu(lhead->flags); flags |= CEPH_MDS_FLAG_REPLAY; - rhead->flags = cpu_to_le32(flags); + lhead->flags = cpu_to_le32(flags); if (req->r_target_inode) - rhead->ino = cpu_to_le64(ceph_ino(req->r_target_inode)); + lhead->ino = cpu_to_le64(ceph_ino(req->r_target_inode)); - rhead->num_retry = req->r_attempts - 1; + lhead->num_retry = req->r_attempts - 1; + if (!old_version) { + nhead = (struct ceph_mds_request_head*)msg->front.iov_base; + nhead->ext_num_retry = cpu_to_le32(req->r_attempts - 1); + } /* remove cap/dentry releases from message */ - rhead->num_releases = 0; + lhead->num_releases = 0; p = msg->front.iov_base + req->r_request_release_offset; - encode_timestamp_and_gids(&p, req); + encode_mclientrequest_tail(&p, req); msg->front.iov_len = p - msg->front.iov_base; msg->hdr.front_len = cpu_to_le32(msg->front.iov_len); @@ -2834,18 +3184,23 @@ static int __prepare_send_request(struct ceph_mds_session *session, } req->r_request = msg; - rhead = find_old_request_head(msg->front.iov_base, - session->s_con.peer_features); - rhead->oldest_client_tid = cpu_to_le64(__get_oldest_tid(mdsc)); + lhead = find_legacy_request_head(msg->front.iov_base, + session->s_con.peer_features); + lhead->oldest_client_tid = cpu_to_le64(__get_oldest_tid(mdsc)); if (test_bit(CEPH_MDS_R_GOT_UNSAFE, &req->r_req_flags)) flags |= CEPH_MDS_FLAG_REPLAY; if (test_bit(CEPH_MDS_R_ASYNC, &req->r_req_flags)) flags |= CEPH_MDS_FLAG_ASYNC; if (req->r_parent) flags |= CEPH_MDS_FLAG_WANT_DENTRY; - rhead->flags = cpu_to_le32(flags); - rhead->num_fwd = req->r_num_fwd; - rhead->num_retry = req->r_attempts - 1; + lhead->flags = cpu_to_le32(flags); + lhead->num_fwd = req->r_num_fwd; + lhead->num_retry = req->r_attempts - 1; + if (!old_version) { + nhead = (struct ceph_mds_request_head*)msg->front.iov_base; + nhead->ext_num_fwd = cpu_to_le32(req->r_num_fwd); + nhead->ext_num_retry = cpu_to_le32(req->r_attempts - 1); + } dout(" r_parent = %p\n", req->r_parent); return 0; @@ -3348,22 +3703,35 @@ static void handle_reply(struct ceph_mds_session *session, struct ceph_msg *msg) } dout("handle_reply tid %lld result %d\n", tid, result); - rinfo = &req->r_reply_info; if (test_bit(CEPHFS_FEATURE_REPLY_ENCODING, &session->s_features)) - err = parse_reply_info(session, msg, rinfo, (u64)-1); + err = parse_reply_info(session, msg, req, (u64)-1); else - err = parse_reply_info(session, msg, rinfo, session->s_con.peer_features); + err = parse_reply_info(session, msg, req, + session->s_con.peer_features); mutex_unlock(&mdsc->mutex); /* Must find target inode outside of mutexes to avoid deadlocks */ + rinfo = &req->r_reply_info; if ((err >= 0) && rinfo->head->is_target) { - struct inode *in; + struct inode *in = xchg(&req->r_new_inode, NULL); struct ceph_vino tvino = { .ino = le64_to_cpu(rinfo->targeti.in->ino), .snap = le64_to_cpu(rinfo->targeti.in->snapid) }; - in = ceph_get_inode(mdsc->fsc->sb, tvino); + /* + * If we ended up opening an existing inode, discard + * r_new_inode + */ + if (req->r_op == CEPH_MDS_OP_CREATE && + !req->r_reply_info.has_create_ino) { + /* This should never happen on an async create */ + WARN_ON_ONCE(req->r_deleg_ino); + iput(in); + in = NULL; + } + + in = ceph_get_inode(mdsc->fsc->sb, tvino, in); if (IS_ERR(in)) { err = PTR_ERR(in); mutex_lock(&session->s_mutex); @@ -3406,7 +3774,7 @@ static void handle_reply(struct ceph_mds_session *session, struct ceph_msg *msg) if (err == 0) { if (result == 0 && (req->r_op == CEPH_MDS_OP_READDIR || req->r_op == CEPH_MDS_OP_LSSNAP)) - ceph_readdir_prepopulate(req, req->r_session); + err = ceph_readdir_prepopulate(req, req->r_session); } current->journal_info = NULL; mutex_unlock(&req->r_fill_mutex); @@ -3491,33 +3859,21 @@ static void handle_forward(struct ceph_mds_client *mdsc, if (test_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags)) { dout("forward tid %llu aborted, unregistering\n", tid); __unregister_request(mdsc, req); - } else if (fwd_seq <= req->r_num_fwd) { + } else if (fwd_seq <= req->r_num_fwd || (uint32_t)fwd_seq >= U32_MAX) { /* - * The type of 'num_fwd' in ceph 'MClientRequestForward' - * is 'int32_t', while in 'ceph_mds_request_head' the - * type is '__u8'. So in case the request bounces between - * MDSes exceeding 256 times, the client will get stuck. + * Avoid inifinite retrying after overflow. * - * In this case it's ususally a bug in MDS and continue - * bouncing the request makes no sense. - * - * In future this could be fixed in ceph code, so avoid - * using the hardcode here. + * The MDS will increase the fwd count and in client side + * if the num_fwd is less than the one saved in request + * that means the MDS is an old version and overflowed of + * 8 bits. */ - int max = sizeof_field(struct ceph_mds_request_head, num_fwd); - max = 1 << (max * BITS_PER_BYTE); - if (req->r_num_fwd >= max) { - mutex_lock(&req->r_fill_mutex); - req->r_err = -EMULTIHOP; - set_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags); - mutex_unlock(&req->r_fill_mutex); - aborted = true; - pr_warn_ratelimited("forward tid %llu seq overflow\n", - tid); - } else { - dout("forward tid %llu to mds%d - old seq %d <= %d\n", - tid, next_mds, req->r_num_fwd, fwd_seq); - } + mutex_lock(&req->r_fill_mutex); + req->r_err = -EMULTIHOP; + set_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags); + mutex_unlock(&req->r_fill_mutex); + aborted = true; + pr_warn_ratelimited("forward tid %llu seq overflow\n", tid); } else { /* resend. forward race not possible; mds would drop */ dout("forward tid %llu to mds%d (we resend)\n", tid, next_mds); @@ -4550,6 +4906,9 @@ static void handle_lease(struct ceph_mds_client *mdsc, dout("handle_lease from mds%d\n", mds); + if (!ceph_inc_mds_stopping_blocker(mdsc, session)) + return; + /* decode */ if (msg->front.iov_len < sizeof(*h) + sizeof(u32)) goto bad; @@ -4568,8 +4927,6 @@ static void handle_lease(struct ceph_mds_client *mdsc, dname.len, dname.name); mutex_lock(&session->s_mutex); - inc_session_sequence(session); - if (!inode) { dout("handle_lease no inode %llx\n", vino.ino); goto release; @@ -4631,9 +4988,13 @@ release: out: mutex_unlock(&session->s_mutex); iput(inode); + + ceph_dec_mds_stopping_blocker(mdsc); return; bad: + ceph_dec_mds_stopping_blocker(mdsc); + pr_err("corrupt lease message\n"); ceph_msg_dump(msg); } @@ -4829,6 +5190,9 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) } init_completion(&mdsc->safe_umount_waiters); + spin_lock_init(&mdsc->stopping_lock); + atomic_set(&mdsc->stopping_blockers, 0); + init_completion(&mdsc->stopping_waiter); init_waitqueue_head(&mdsc->session_close_wq); INIT_LIST_HEAD(&mdsc->waiting_for_map); mdsc->quotarealms_inodes = RB_ROOT; diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h index 86d2965e68a1..5a3714bdd64a 100644 --- a/fs/ceph/mds_client.h +++ b/fs/ceph/mds_client.h @@ -32,8 +32,9 @@ enum ceph_feature_type { CEPHFS_FEATURE_ALTERNATE_NAME, CEPHFS_FEATURE_NOTIFY_SESSION_STATE, CEPHFS_FEATURE_OP_GETVXATTR, + CEPHFS_FEATURE_32BITS_RETRY_FWD, - CEPHFS_FEATURE_MAX = CEPHFS_FEATURE_OP_GETVXATTR, + CEPHFS_FEATURE_MAX = CEPHFS_FEATURE_32BITS_RETRY_FWD, }; #define CEPHFS_FEATURES_CLIENT_SUPPORTED { \ @@ -44,8 +45,10 @@ enum ceph_feature_type { CEPHFS_FEATURE_MULTI_RECONNECT, \ CEPHFS_FEATURE_DELEG_INO, \ CEPHFS_FEATURE_METRIC_COLLECT, \ + CEPHFS_FEATURE_ALTERNATE_NAME, \ CEPHFS_FEATURE_NOTIFY_SESSION_STATE, \ CEPHFS_FEATURE_OP_GETVXATTR, \ + CEPHFS_FEATURE_32BITS_RETRY_FWD, \ } /* @@ -86,13 +89,19 @@ struct ceph_mds_reply_info_in { s32 dir_pin; struct ceph_timespec btime; struct ceph_timespec snap_btime; + u8 *fscrypt_auth; + u8 *fscrypt_file; + u32 fscrypt_auth_len; + u32 fscrypt_file_len; u64 rsnaps; u64 change_attr; }; struct ceph_mds_reply_dir_entry { + bool is_nokey; char *name; u32 name_len; + u32 raw_hash; struct ceph_mds_reply_lease *lease; struct ceph_mds_reply_info_in inode; loff_t offset; @@ -116,7 +125,9 @@ struct ceph_mds_reply_info_parsed { struct ceph_mds_reply_info_in diri, targeti; struct ceph_mds_reply_dirfrag *dirfrag; char *dname; + u8 *altname; u32 dname_len; + u32 altname_len; struct ceph_mds_reply_lease *dlease; struct ceph_mds_reply_xattr xattr_info; @@ -263,6 +274,7 @@ struct ceph_mds_request { struct inode *r_parent; /* parent dir inode */ struct inode *r_target_inode; /* resulting inode */ + struct inode *r_new_inode; /* new inode (for creates) */ #define CEPH_MDS_R_DIRECT_IS_HASH (1) /* r_direct_hash is valid */ #define CEPH_MDS_R_ABORTED (2) /* call was aborted */ @@ -272,11 +284,19 @@ struct ceph_mds_request { #define CEPH_MDS_R_DID_PREPOPULATE (6) /* prepopulated readdir */ #define CEPH_MDS_R_PARENT_LOCKED (7) /* is r_parent->i_rwsem wlocked? */ #define CEPH_MDS_R_ASYNC (8) /* async request */ +#define CEPH_MDS_R_FSCRYPT_FILE (9) /* must marshal fscrypt_file field */ unsigned long r_req_flags; struct mutex r_fill_mutex; union ceph_mds_request_args r_args; + + struct ceph_fscrypt_auth *r_fscrypt_auth; + u64 r_fscrypt_file; + + u8 *r_altname; /* fscrypt binary crypttext for long filenames */ + u32 r_altname_len; /* length of r_altname */ + int r_fmode; /* file mode, if expecting cap */ int r_request_release_offset; const struct cred *r_cred; @@ -381,8 +401,9 @@ struct cap_wait { }; enum { - CEPH_MDSC_STOPPING_BEGIN = 1, - CEPH_MDSC_STOPPING_FLUSHED = 2, + CEPH_MDSC_STOPPING_BEGIN = 1, + CEPH_MDSC_STOPPING_FLUSHING = 2, + CEPH_MDSC_STOPPING_FLUSHED = 3, }; /* @@ -401,7 +422,11 @@ struct ceph_mds_client { struct ceph_mds_session **sessions; /* NULL for mds if no session */ atomic_t num_sessions; int max_sessions; /* len of sessions array */ - int stopping; /* true if shutting down */ + + spinlock_t stopping_lock; /* protect snap_empty */ + int stopping; /* the stage of shutting down */ + atomic_t stopping_blockers; + struct completion stopping_waiter; atomic64_t quotarealms_count; /* # realms with quota */ /* @@ -557,7 +582,7 @@ static inline void ceph_mdsc_free_path(char *path, int len) } extern char *ceph_mdsc_build_path(struct dentry *dentry, int *plen, u64 *base, - int stop_on_nosnap); + int for_wire); extern void __ceph_mdsc_drop_dentry_lease(struct dentry *dentry); extern void ceph_mdsc_lease_send_msg(struct ceph_mds_session *session, diff --git a/fs/ceph/quota.c b/fs/ceph/quota.c index 64592adfe48f..f7fcf7f08ec6 100644 --- a/fs/ceph/quota.c +++ b/fs/ceph/quota.c @@ -47,25 +47,23 @@ void ceph_handle_quota(struct ceph_mds_client *mdsc, struct inode *inode; struct ceph_inode_info *ci; + if (!ceph_inc_mds_stopping_blocker(mdsc, session)) + return; + if (msg->front.iov_len < sizeof(*h)) { pr_err("%s corrupt message mds%d len %d\n", __func__, session->s_mds, (int)msg->front.iov_len); ceph_msg_dump(msg); - return; + goto out; } - /* increment msg sequence number */ - mutex_lock(&session->s_mutex); - inc_session_sequence(session); - mutex_unlock(&session->s_mutex); - /* lookup inode */ vino.ino = le64_to_cpu(h->ino); vino.snap = CEPH_NOSNAP; inode = ceph_find_inode(sb, vino); if (!inode) { pr_warn("Failed to find inode %llu\n", vino.ino); - return; + goto out; } ci = ceph_inode(inode); @@ -78,6 +76,8 @@ void ceph_handle_quota(struct ceph_mds_client *mdsc, spin_unlock(&ci->i_ceph_lock); iput(inode); +out: + ceph_dec_mds_stopping_blocker(mdsc); } static struct ceph_quotarealm_inode * diff --git a/fs/ceph/snap.c b/fs/ceph/snap.c index c9920ade15f5..813f21add992 100644 --- a/fs/ceph/snap.c +++ b/fs/ceph/snap.c @@ -1015,6 +1015,9 @@ void ceph_handle_snap(struct ceph_mds_client *mdsc, int locked_rwsem = 0; bool close_sessions = false; + if (!ceph_inc_mds_stopping_blocker(mdsc, session)) + return; + /* decode */ if (msg->front.iov_len < sizeof(*h)) goto bad; @@ -1030,10 +1033,6 @@ void ceph_handle_snap(struct ceph_mds_client *mdsc, dout("%s from mds%d op %s split %llx tracelen %d\n", __func__, mds, ceph_snap_op_name(op), split, trace_len); - mutex_lock(&session->s_mutex); - inc_session_sequence(session); - mutex_unlock(&session->s_mutex); - down_write(&mdsc->snap_rwsem); locked_rwsem = 1; @@ -1151,6 +1150,7 @@ skip_inode: up_write(&mdsc->snap_rwsem); flush_snaps(mdsc); + ceph_dec_mds_stopping_blocker(mdsc); return; bad: @@ -1160,6 +1160,8 @@ out: if (locked_rwsem) up_write(&mdsc->snap_rwsem); + ceph_dec_mds_stopping_blocker(mdsc); + if (close_sessions) ceph_mdsc_close_sessions(mdsc); return; diff --git a/fs/ceph/super.c b/fs/ceph/super.c index a5f52013314d..2d7f5a8d4a92 100644 --- a/fs/ceph/super.c +++ b/fs/ceph/super.c @@ -20,6 +20,7 @@ #include "super.h" #include "mds_client.h" #include "cache.h" +#include "crypto.h" #include #include @@ -46,6 +47,7 @@ static void ceph_put_super(struct super_block *s) struct ceph_fs_client *fsc = ceph_sb_to_client(s); dout("put_super\n"); + ceph_fscrypt_free_dummy_policy(fsc); ceph_mdsc_close_sessions(fsc->mdsc); } @@ -151,6 +153,7 @@ enum { Opt_recover_session, Opt_source, Opt_mon_addr, + Opt_test_dummy_encryption, /* string args above */ Opt_dirstat, Opt_rbytes, @@ -165,6 +168,7 @@ enum { Opt_copyfrom, Opt_wsync, Opt_pagecache, + Opt_sparseread, }; enum ceph_recover_session_mode { @@ -192,6 +196,7 @@ static const struct fs_parameter_spec ceph_mount_parameters[] = { fsparam_string ("fsc", Opt_fscache), // fsc=... fsparam_flag_no ("ino32", Opt_ino32), fsparam_string ("mds_namespace", Opt_mds_namespace), + fsparam_string ("mon_addr", Opt_mon_addr), fsparam_flag_no ("poolperm", Opt_poolperm), fsparam_flag_no ("quotadf", Opt_quotadf), fsparam_u32 ("rasize", Opt_rasize), @@ -203,10 +208,12 @@ static const struct fs_parameter_spec ceph_mount_parameters[] = { fsparam_u32 ("rsize", Opt_rsize), fsparam_string ("snapdirname", Opt_snapdirname), fsparam_string ("source", Opt_source), - fsparam_string ("mon_addr", Opt_mon_addr), + fsparam_flag ("test_dummy_encryption", Opt_test_dummy_encryption), + fsparam_string ("test_dummy_encryption", Opt_test_dummy_encryption), fsparam_u32 ("wsize", Opt_wsize), fsparam_flag_no ("wsync", Opt_wsync), fsparam_flag_no ("pagecache", Opt_pagecache), + fsparam_flag_no ("sparseread", Opt_sparseread), {} }; @@ -576,6 +583,29 @@ static int ceph_parse_mount_param(struct fs_context *fc, else fsopt->flags &= ~CEPH_MOUNT_OPT_NOPAGECACHE; break; + case Opt_sparseread: + if (result.negated) + fsopt->flags &= ~CEPH_MOUNT_OPT_SPARSEREAD; + else + fsopt->flags |= CEPH_MOUNT_OPT_SPARSEREAD; + break; + case Opt_test_dummy_encryption: +#ifdef CONFIG_FS_ENCRYPTION + fscrypt_free_dummy_policy(&fsopt->dummy_enc_policy); + ret = fscrypt_parse_test_dummy_encryption(param, + &fsopt->dummy_enc_policy); + if (ret == -EINVAL) { + warnfc(fc, "Value of option \"%s\" is unrecognized", + param->key); + } else if (ret == -EEXIST) { + warnfc(fc, "Conflicting test_dummy_encryption options"); + ret = -EINVAL; + } +#else + warnfc(fc, + "FS encryption not supported: test_dummy_encryption mount option ignored"); +#endif + break; default: BUG(); } @@ -596,6 +626,7 @@ static void destroy_mount_options(struct ceph_mount_options *args) kfree(args->server_path); kfree(args->fscache_uniq); kfree(args->mon_addr); + fscrypt_free_dummy_policy(&args->dummy_enc_policy); kfree(args); } @@ -710,9 +741,12 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root) if (!(fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS)) seq_puts(m, ",wsync"); - if (fsopt->flags & CEPH_MOUNT_OPT_NOPAGECACHE) seq_puts(m, ",nopagecache"); + if (fsopt->flags & CEPH_MOUNT_OPT_SPARSEREAD) + seq_puts(m, ",sparseread"); + + fscrypt_show_test_dummy_encryption(m, ',', root->d_sb); if (fsopt->wsize != CEPH_MAX_WRITE_SIZE) seq_printf(m, ",wsize=%u", fsopt->wsize); @@ -1052,6 +1086,50 @@ out: return root; } +#ifdef CONFIG_FS_ENCRYPTION +static int ceph_apply_test_dummy_encryption(struct super_block *sb, + struct fs_context *fc, + struct ceph_mount_options *fsopt) +{ + struct ceph_fs_client *fsc = sb->s_fs_info; + + if (!fscrypt_is_dummy_policy_set(&fsopt->dummy_enc_policy)) + return 0; + + /* No changing encryption context on remount. */ + if (fc->purpose == FS_CONTEXT_FOR_RECONFIGURE && + !fscrypt_is_dummy_policy_set(&fsc->fsc_dummy_enc_policy)) { + if (fscrypt_dummy_policies_equal(&fsopt->dummy_enc_policy, + &fsc->fsc_dummy_enc_policy)) + return 0; + errorfc(fc, "Can't set test_dummy_encryption on remount"); + return -EINVAL; + } + + /* Also make sure fsopt doesn't contain a conflicting value. */ + if (fscrypt_is_dummy_policy_set(&fsc->fsc_dummy_enc_policy)) { + if (fscrypt_dummy_policies_equal(&fsopt->dummy_enc_policy, + &fsc->fsc_dummy_enc_policy)) + return 0; + errorfc(fc, "Conflicting test_dummy_encryption options"); + return -EINVAL; + } + + fsc->fsc_dummy_enc_policy = fsopt->dummy_enc_policy; + memset(&fsopt->dummy_enc_policy, 0, sizeof(fsopt->dummy_enc_policy)); + + warnfc(fc, "test_dummy_encryption mode enabled"); + return 0; +} +#else +static int ceph_apply_test_dummy_encryption(struct super_block *sb, + struct fs_context *fc, + struct ceph_mount_options *fsopt) +{ + return 0; +} +#endif + /* * mount: join the ceph cluster, and open root directory. */ @@ -1080,6 +1158,11 @@ static struct dentry *ceph_real_mount(struct ceph_fs_client *fsc, goto out; } + err = ceph_apply_test_dummy_encryption(fsc->sb, fc, + fsc->mount_options); + if (err) + goto out; + dout("mount opening path '%s'\n", path); ceph_fs_debugfs_init(fsc); @@ -1101,6 +1184,7 @@ static struct dentry *ceph_real_mount(struct ceph_fs_client *fsc, out: mutex_unlock(&fsc->client->mount_mutex); + ceph_fscrypt_free_dummy_policy(fsc); return ERR_PTR(err); } @@ -1126,6 +1210,8 @@ static int ceph_set_super(struct super_block *s, struct fs_context *fc) s->s_time_max = U32_MAX; s->s_flags |= SB_NODIRATIME | SB_NOATIME; + ceph_fscrypt_set_ops(s); + ret = set_anon_super_fc(s, fc); if (ret != 0) fsc->sb = NULL; @@ -1287,15 +1373,26 @@ static void ceph_free_fc(struct fs_context *fc) static int ceph_reconfigure_fc(struct fs_context *fc) { + int err; struct ceph_parse_opts_ctx *pctx = fc->fs_private; struct ceph_mount_options *fsopt = pctx->opts; - struct ceph_fs_client *fsc = ceph_sb_to_client(fc->root->d_sb); + struct super_block *sb = fc->root->d_sb; + struct ceph_fs_client *fsc = ceph_sb_to_client(sb); + + err = ceph_apply_test_dummy_encryption(sb, fc, fsopt); + if (err) + return err; if (fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS) ceph_set_mount_opt(fsc, ASYNC_DIROPS); else ceph_clear_mount_opt(fsc, ASYNC_DIROPS); + if (fsopt->flags & CEPH_MOUNT_OPT_SPARSEREAD) + ceph_set_mount_opt(fsc, SPARSEREAD); + else + ceph_clear_mount_opt(fsc, SPARSEREAD); + if (strcmp_null(fsc->mount_options->mon_addr, fsopt->mon_addr)) { kfree(fsc->mount_options->mon_addr); fsc->mount_options->mon_addr = fsopt->mon_addr; @@ -1303,7 +1400,7 @@ static int ceph_reconfigure_fc(struct fs_context *fc) pr_notice("ceph: monitor addresses recorded, but not used for reconnection"); } - sync_filesystem(fc->root->d_sb); + sync_filesystem(sb); return 0; } @@ -1365,25 +1462,101 @@ nomem: return -ENOMEM; } +/* + * Return true if it successfully increases the blocker counter, + * or false if the mdsc is in stopping and flushed state. + */ +static bool __inc_stopping_blocker(struct ceph_mds_client *mdsc) +{ + spin_lock(&mdsc->stopping_lock); + if (mdsc->stopping >= CEPH_MDSC_STOPPING_FLUSHING) { + spin_unlock(&mdsc->stopping_lock); + return false; + } + atomic_inc(&mdsc->stopping_blockers); + spin_unlock(&mdsc->stopping_lock); + return true; +} + +static void __dec_stopping_blocker(struct ceph_mds_client *mdsc) +{ + spin_lock(&mdsc->stopping_lock); + if (!atomic_dec_return(&mdsc->stopping_blockers) && + mdsc->stopping >= CEPH_MDSC_STOPPING_FLUSHING) + complete_all(&mdsc->stopping_waiter); + spin_unlock(&mdsc->stopping_lock); +} + +/* For metadata IO requests */ +bool ceph_inc_mds_stopping_blocker(struct ceph_mds_client *mdsc, + struct ceph_mds_session *session) +{ + mutex_lock(&session->s_mutex); + inc_session_sequence(session); + mutex_unlock(&session->s_mutex); + + return __inc_stopping_blocker(mdsc); +} + +void ceph_dec_mds_stopping_blocker(struct ceph_mds_client *mdsc) +{ + __dec_stopping_blocker(mdsc); +} + +/* For data IO requests */ +bool ceph_inc_osd_stopping_blocker(struct ceph_mds_client *mdsc) +{ + return __inc_stopping_blocker(mdsc); +} + +void ceph_dec_osd_stopping_blocker(struct ceph_mds_client *mdsc) +{ + __dec_stopping_blocker(mdsc); +} + static void ceph_kill_sb(struct super_block *s) { struct ceph_fs_client *fsc = ceph_sb_to_client(s); + struct ceph_mds_client *mdsc = fsc->mdsc; + bool wait; dout("kill_sb %p\n", s); - ceph_mdsc_pre_umount(fsc->mdsc); + ceph_mdsc_pre_umount(mdsc); flush_fs_workqueues(fsc); /* * Though the kill_anon_super() will finally trigger the - * sync_filesystem() anyway, we still need to do it here - * and then bump the stage of shutdown to stop the work - * queue as earlier as possible. + * sync_filesystem() anyway, we still need to do it here and + * then bump the stage of shutdown. This will allow us to + * drop any further message, which will increase the inodes' + * i_count reference counters but makes no sense any more, + * from MDSs. + * + * Without this when evicting the inodes it may fail in the + * kill_anon_super(), which will trigger a warning when + * destroying the fscrypt keyring and then possibly trigger + * a further crash in ceph module when the iput() tries to + * evict the inodes later. */ sync_filesystem(s); - fsc->mdsc->stopping = CEPH_MDSC_STOPPING_FLUSHED; + spin_lock(&mdsc->stopping_lock); + mdsc->stopping = CEPH_MDSC_STOPPING_FLUSHING; + wait = !!atomic_read(&mdsc->stopping_blockers); + spin_unlock(&mdsc->stopping_lock); + if (wait && atomic_read(&mdsc->stopping_blockers)) { + long timeleft = wait_for_completion_killable_timeout( + &mdsc->stopping_waiter, + fsc->client->options->mount_timeout); + if (!timeleft) /* timed out */ + pr_warn("umount timed out, %ld\n", timeleft); + else if (timeleft < 0) /* killed */ + pr_warn("umount was killed, %ld\n", timeleft); + } + + mdsc->stopping = CEPH_MDSC_STOPPING_FLUSHED; kill_anon_super(s); fsc->client->extra_mon_dispatch = NULL; diff --git a/fs/ceph/super.h b/fs/ceph/super.h index 3bfddf34d488..51c7f2b14f6f 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -22,6 +22,7 @@ #include #include +#include "crypto.h" /* large granularity for statfs utilization stats to facilitate * large volume sizes on 32-bit machines. */ @@ -42,6 +43,7 @@ #define CEPH_MOUNT_OPT_NOCOPYFROM (1<<14) /* don't use RADOS 'copy-from' op */ #define CEPH_MOUNT_OPT_ASYNC_DIROPS (1<<15) /* allow async directory ops */ #define CEPH_MOUNT_OPT_NOPAGECACHE (1<<16) /* bypass pagecache altogether */ +#define CEPH_MOUNT_OPT_SPARSEREAD (1<<17) /* always do sparse reads */ #define CEPH_MOUNT_OPT_DEFAULT \ (CEPH_MOUNT_OPT_DCACHE | \ @@ -98,6 +100,7 @@ struct ceph_mount_options { char *server_path; /* default NULL (means "/") */ char *fscache_uniq; /* default NULL */ char *mon_addr; + struct fscrypt_dummy_policy dummy_enc_policy; }; /* mount state */ @@ -154,9 +157,11 @@ struct ceph_fs_client { #ifdef CONFIG_CEPH_FSCACHE struct fscache_volume *fscache; #endif +#ifdef CONFIG_FS_ENCRYPTION + struct fscrypt_dummy_policy fsc_dummy_enc_policy; +#endif }; - /* * File i/o capability. This tracks shared state with the metadata * server that allows us to cache or writeback attributes or to read @@ -419,6 +424,11 @@ struct ceph_inode_info { u32 i_truncate_seq; /* last truncate to smaller size */ u64 i_truncate_size; /* and the size we last truncated down to */ int i_truncate_pending; /* still need to call vmtruncate */ + /* + * For none fscrypt case it equals to i_truncate_size or it will + * equals to fscrypt_file_size + */ + u64 i_truncate_pagecache_size; u64 i_max_size; /* max file size authorized by mds */ u64 i_reported_size; /* (max_)size reported to or requested of mds */ @@ -449,6 +459,13 @@ struct ceph_inode_info { struct work_struct i_work; unsigned long i_work_mask; + +#ifdef CONFIG_FS_ENCRYPTION + u32 fscrypt_auth_len; + u32 fscrypt_file_len; + u8 *fscrypt_auth; + u8 *fscrypt_file; +#endif }; struct ceph_netfs_request_data { @@ -998,6 +1015,7 @@ static inline bool __ceph_have_pending_cap_snap(struct ceph_inode_info *ci) /* inode.c */ struct ceph_mds_reply_info_in; struct ceph_mds_reply_dirfrag; +struct ceph_acl_sec_ctx; extern const struct inode_operations ceph_file_iops; @@ -1005,8 +1023,14 @@ extern struct inode *ceph_alloc_inode(struct super_block *sb); extern void ceph_evict_inode(struct inode *inode); extern void ceph_free_inode(struct inode *inode); +struct inode *ceph_new_inode(struct inode *dir, struct dentry *dentry, + umode_t *mode, struct ceph_acl_sec_ctx *as_ctx); +void ceph_as_ctx_to_req(struct ceph_mds_request *req, + struct ceph_acl_sec_ctx *as_ctx); + extern struct inode *ceph_get_inode(struct super_block *sb, - struct ceph_vino vino); + struct ceph_vino vino, + struct inode *newino); extern struct inode *ceph_get_snapdir(struct inode *parent); extern int ceph_fill_file_size(struct inode *inode, int issued, u32 truncate_seq, u64 truncate_size, u64 size); @@ -1065,7 +1089,13 @@ static inline int ceph_do_getattr(struct inode *inode, int mask, bool force) } extern int ceph_permission(struct mnt_idmap *idmap, struct inode *inode, int mask); -extern int __ceph_setattr(struct inode *inode, struct iattr *attr); + +struct ceph_iattr { + struct ceph_fscrypt_auth *fscrypt_auth; +}; + +extern int __ceph_setattr(struct inode *inode, struct iattr *attr, + struct ceph_iattr *cia); extern int ceph_setattr(struct mnt_idmap *idmap, struct dentry *dentry, struct iattr *attr); extern int ceph_getattr(struct mnt_idmap *idmap, @@ -1099,6 +1129,9 @@ struct ceph_acl_sec_ctx { #ifdef CONFIG_CEPH_FS_SECURITY_LABEL void *sec_ctx; u32 sec_ctxlen; +#endif +#ifdef CONFIG_FS_ENCRYPTION + struct ceph_fscrypt_auth *fscrypt_auth; #endif struct ceph_pagelist *pagelist; }; @@ -1237,6 +1270,8 @@ extern int ceph_encode_dentry_release(void **p, struct dentry *dn, struct inode *dir, int mds, int drop, int unless); +extern int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, + int need, int want, loff_t endoff, int *got); extern int ceph_get_caps(struct file *filp, int need, int want, loff_t endoff, int *got); extern int ceph_try_get_caps(struct inode *inode, @@ -1272,6 +1307,9 @@ extern int ceph_renew_caps(struct inode *inode, int fmode); extern int ceph_open(struct inode *inode, struct file *file); extern int ceph_atomic_open(struct inode *dir, struct dentry *dentry, struct file *file, unsigned flags, umode_t mode); +extern ssize_t __ceph_sync_read(struct inode *inode, loff_t *ki_pos, + struct iov_iter *to, int *retry_op, + u64 *last_objver); extern int ceph_release(struct inode *inode, struct file *filp); extern void ceph_fill_inline_data(struct inode *inode, struct page *locked_page, char *data, size_t len); @@ -1375,4 +1413,9 @@ extern bool ceph_quota_update_statfs(struct ceph_fs_client *fsc, struct kstatfs *buf); extern void ceph_cleanup_quotarealms_inodes(struct ceph_mds_client *mdsc); +bool ceph_inc_mds_stopping_blocker(struct ceph_mds_client *mdsc, + struct ceph_mds_session *session); +void ceph_dec_mds_stopping_blocker(struct ceph_mds_client *mdsc); +bool ceph_inc_osd_stopping_blocker(struct ceph_mds_client *mdsc); +void ceph_dec_osd_stopping_blocker(struct ceph_mds_client *mdsc); #endif /* _FS_CEPH_SUPER_H */ diff --git a/fs/ceph/xattr.c b/fs/ceph/xattr.c index 1cbd84cc82a8..0deae4a0f5f1 100644 --- a/fs/ceph/xattr.c +++ b/fs/ceph/xattr.c @@ -352,6 +352,24 @@ static ssize_t ceph_vxattrcb_auth_mds(struct ceph_inode_info *ci, return ret; } +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) +static bool ceph_vxattrcb_fscrypt_auth_exists(struct ceph_inode_info *ci) +{ + return ci->fscrypt_auth_len; +} + +static ssize_t ceph_vxattrcb_fscrypt_auth(struct ceph_inode_info *ci, + char *val, size_t size) +{ + if (size) { + if (size < ci->fscrypt_auth_len) + return -ERANGE; + memcpy(val, ci->fscrypt_auth, ci->fscrypt_auth_len); + } + return ci->fscrypt_auth_len; +} +#endif /* CONFIG_FS_ENCRYPTION */ + #define CEPH_XATTR_NAME(_type, _name) XATTR_CEPH_PREFIX #_type "." #_name #define CEPH_XATTR_NAME2(_type, _name, _name2) \ XATTR_CEPH_PREFIX #_type "." #_name "." #_name2 @@ -500,6 +518,15 @@ static struct ceph_vxattr ceph_common_vxattrs[] = { .exists_cb = NULL, .flags = VXATTR_FLAG_READONLY, }, +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) + { + .name = "ceph.fscrypt.auth", + .name_size = sizeof("ceph.fscrypt.auth"), + .getxattr_cb = ceph_vxattrcb_fscrypt_auth, + .exists_cb = ceph_vxattrcb_fscrypt_auth_exists, + .flags = VXATTR_FLAG_READONLY, + }, +#endif /* CONFIG_FS_ENCRYPTION */ { .name = NULL, 0 } /* Required table terminator */ }; @@ -1407,6 +1434,9 @@ void ceph_release_acl_sec_ctx(struct ceph_acl_sec_ctx *as_ctx) #endif #ifdef CONFIG_CEPH_FS_SECURITY_LABEL security_release_secctx(as_ctx->sec_ctx, as_ctx->sec_ctxlen); +#endif +#ifdef CONFIG_FS_ENCRYPTION + kfree(as_ctx->fscrypt_auth); #endif if (as_ctx->pagelist) ceph_pagelist_release(as_ctx->pagelist); diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c index 9c4b26aec580..c26d48355cc2 100644 --- a/fs/gfs2/aops.c +++ b/fs/gfs2/aops.c @@ -183,13 +183,13 @@ static int gfs2_writepages(struct address_space *mapping, int ret; /* - * Even if we didn't write any pages here, we might still be holding + * Even if we didn't write enough pages here, we might still be holding * dirty pages in the ail. We forcibly flush the ail because we don't * want balance_dirty_pages() to loop indefinitely trying to write out * pages held in the ail that it can't find. */ ret = iomap_writepages(mapping, wbc, &wpc, &gfs2_writeback_ops); - if (ret == 0) + if (ret == 0 && wbc->nr_to_write > 0) set_bit(SDF_FORCE_AIL_FLUSH, &sdp->sd_flags); return ret; } @@ -272,8 +272,7 @@ continue_unlock: * not be suitable for data integrity * writeout). */ - *done_index = folio->index + - folio_nr_pages(folio); + *done_index = folio_next_index(folio); ret = 1; break; } diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c index f62366be7587..ef7017fb6951 100644 --- a/fs/gfs2/bmap.c +++ b/fs/gfs2/bmap.c @@ -161,7 +161,7 @@ int gfs2_unstuff_dinode(struct gfs2_inode *ip) int error; down_write(&ip->i_rw_mutex); - page = find_or_create_page(inode->i_mapping, 0, GFP_NOFS); + page = grab_cache_page(inode->i_mapping, 0); error = -ENOMEM; if (!page) goto out; diff --git a/fs/gfs2/glock.c b/fs/gfs2/glock.c index 1438e7465e30..9cbf8d98489a 100644 --- a/fs/gfs2/glock.c +++ b/fs/gfs2/glock.c @@ -176,7 +176,7 @@ void gfs2_glock_free(struct gfs2_glock *gl) wake_up_glock(gl); call_rcu(&gl->gl_rcu, gfs2_glock_dealloc); if (atomic_dec_and_test(&sdp->sd_glock_disposal)) - wake_up(&sdp->sd_glock_wait); + wake_up(&sdp->sd_kill_wait); } /** @@ -468,10 +468,10 @@ done: * do_promote - promote as many requests as possible on the current queue * @gl: The glock * - * Returns: 1 if there is a blocked holder at the head of the list + * Returns true on success (i.e., progress was made or there are no waiters). */ -static int do_promote(struct gfs2_glock *gl) +static bool do_promote(struct gfs2_glock *gl) { struct gfs2_holder *gh, *current_gh; @@ -484,10 +484,10 @@ static int do_promote(struct gfs2_glock *gl) * If we get here, it means we may not grant this * holder for some reason. If this holder is at the * head of the list, it means we have a blocked holder - * at the head, so return 1. + * at the head, so return false. */ if (list_is_first(&gh->gh_list, &gl->gl_holders)) - return 1; + return false; do_error(gl, 0); break; } @@ -497,7 +497,7 @@ static int do_promote(struct gfs2_glock *gl) if (!current_gh) current_gh = gh; } - return 0; + return true; } /** @@ -591,10 +591,11 @@ static void finish_xmote(struct gfs2_glock *gl, unsigned int ret) if (gh && !test_bit(GLF_DEMOTE_IN_PROGRESS, &gl->gl_flags)) { /* move to back of queue and try next entry */ if (ret & LM_OUT_CANCELED) { - if ((gh->gh_flags & LM_FLAG_PRIORITY) == 0) - list_move_tail(&gh->gh_list, &gl->gl_holders); + list_move_tail(&gh->gh_list, &gl->gl_holders); gh = find_first_waiter(gl); gl->gl_target = gh->gh_state; + if (do_promote(gl)) + goto out; goto retry; } /* Some error or failed "try lock" - report it */ @@ -679,8 +680,7 @@ __acquires(&gl->gl_lockref.lock) gh && !(gh->gh_flags & LM_FLAG_NOEXP)) goto skip_inval; - lck_flags &= (LM_FLAG_TRY | LM_FLAG_TRY_1CB | LM_FLAG_NOEXP | - LM_FLAG_PRIORITY); + lck_flags &= (LM_FLAG_TRY | LM_FLAG_TRY_1CB | LM_FLAG_NOEXP); GLOCK_BUG_ON(gl, gl->gl_state == target); GLOCK_BUG_ON(gl, gl->gl_state == gl->gl_target); if ((target == LM_ST_UNLOCKED || target == LM_ST_DEFERRED) && @@ -834,7 +834,7 @@ __acquires(&gl->gl_lockref.lock) } else { if (test_bit(GLF_DEMOTE, &gl->gl_flags)) gfs2_demote_wake(gl); - if (do_promote(gl) == 0) + if (do_promote(gl)) goto out_unlock; gh = find_first_waiter(gl); gl->gl_target = gh->gh_state; @@ -1022,7 +1022,7 @@ static void delete_work_func(struct work_struct *work) * step entirely. */ if (gfs2_try_evict(gl)) { - if (test_bit(SDF_DEACTIVATING, &sdp->sd_flags)) + if (test_bit(SDF_KILL, &sdp->sd_flags)) goto out; if (gfs2_queue_verify_evict(gl)) return; @@ -1035,7 +1035,7 @@ static void delete_work_func(struct work_struct *work) GFS2_BLKST_UNLINKED); if (IS_ERR(inode)) { if (PTR_ERR(inode) == -EAGAIN && - !test_bit(SDF_DEACTIVATING, &sdp->sd_flags) && + !test_bit(SDF_KILL, &sdp->sd_flags) && gfs2_queue_verify_evict(gl)) return; } else { @@ -1231,7 +1231,7 @@ int gfs2_glock_get(struct gfs2_sbd *sdp, u64 number, out_free: gfs2_glock_dealloc(&gl->gl_rcu); if (atomic_dec_and_test(&sdp->sd_glock_disposal)) - wake_up(&sdp->sd_glock_wait); + wake_up(&sdp->sd_kill_wait); out: return ret; @@ -1515,27 +1515,20 @@ fail: } if (test_bit(HIF_HOLDER, &gh2->gh_iflags)) continue; - if (unlikely((gh->gh_flags & LM_FLAG_PRIORITY) && !insert_pt)) - insert_pt = &gh2->gh_list; } trace_gfs2_glock_queue(gh, 1); gfs2_glstats_inc(gl, GFS2_LKS_QCOUNT); gfs2_sbstats_inc(gl, GFS2_LKS_QCOUNT); if (likely(insert_pt == NULL)) { list_add_tail(&gh->gh_list, &gl->gl_holders); - if (unlikely(gh->gh_flags & LM_FLAG_PRIORITY)) - goto do_cancel; return; } list_add_tail(&gh->gh_list, insert_pt); -do_cancel: gh = list_first_entry(&gl->gl_holders, struct gfs2_holder, gh_list); - if (!(gh->gh_flags & LM_FLAG_PRIORITY)) { - spin_unlock(&gl->gl_lockref.lock); - if (sdp->sd_lockstruct.ls_ops->lm_cancel) - sdp->sd_lockstruct.ls_ops->lm_cancel(gl); - spin_lock(&gl->gl_lockref.lock); - } + spin_unlock(&gl->gl_lockref.lock); + if (sdp->sd_lockstruct.ls_ops->lm_cancel) + sdp->sd_lockstruct.ls_ops->lm_cancel(gl); + spin_lock(&gl->gl_lockref.lock); return; trap_recursive: @@ -2195,7 +2188,7 @@ void gfs2_gl_hash_clear(struct gfs2_sbd *sdp) flush_workqueue(glock_workqueue); glock_hash_walk(clear_glock, sdp); flush_workqueue(glock_workqueue); - wait_event_timeout(sdp->sd_glock_wait, + wait_event_timeout(sdp->sd_kill_wait, atomic_read(&sdp->sd_glock_disposal) == 0, HZ * 600); glock_hash_walk(dump_glock_func, sdp); @@ -2227,8 +2220,6 @@ static const char *hflags2str(char *buf, u16 flags, unsigned long iflags) *p++ = 'e'; if (flags & LM_FLAG_ANY) *p++ = 'A'; - if (flags & LM_FLAG_PRIORITY) - *p++ = 'p'; if (flags & LM_FLAG_NODE_SCOPE) *p++ = 'n'; if (flags & GL_ASYNC) diff --git a/fs/gfs2/glock.h b/fs/gfs2/glock.h index 1f1ba92c15a8..c8685ca7d2a2 100644 --- a/fs/gfs2/glock.h +++ b/fs/gfs2/glock.h @@ -68,14 +68,6 @@ enum { * also be granted in SHARED. The preferred state is whichever is compatible * with other granted locks, or the specified state if no other locks exist. * - * LM_FLAG_PRIORITY - * Override fairness considerations. Suppose a lock is held in a shared state - * and there is a pending request for the deferred state. A shared lock - * request with the priority flag would be allowed to bypass the deferred - * request and directly join the other shared lock. A shared lock request - * without the priority flag might be forced to wait until the deferred - * requested had acquired and released the lock. - * * LM_FLAG_NODE_SCOPE * This holder agrees to share the lock within this node. In other words, * the glock is held in EX mode according to DLM, but local holders on the @@ -86,7 +78,6 @@ enum { #define LM_FLAG_TRY_1CB 0x0002 #define LM_FLAG_NOEXP 0x0004 #define LM_FLAG_ANY 0x0008 -#define LM_FLAG_PRIORITY 0x0010 #define LM_FLAG_NODE_SCOPE 0x0020 #define GL_ASYNC 0x0040 #define GL_EXACT 0x0080 diff --git a/fs/gfs2/glops.c b/fs/gfs2/glops.c index aecdac3cfbe1..d26759a98b10 100644 --- a/fs/gfs2/glops.c +++ b/fs/gfs2/glops.c @@ -637,7 +637,7 @@ static void iopen_go_callback(struct gfs2_glock *gl, bool remote) struct gfs2_sbd *sdp = gl->gl_name.ln_sbd; if (!remote || sb_rdonly(sdp->sd_vfs) || - test_bit(SDF_DEACTIVATING, &sdp->sd_flags)) + test_bit(SDF_KILL, &sdp->sd_flags)) return; if (gl->gl_demote_state == LM_ST_UNLOCKED && diff --git a/fs/gfs2/incore.h b/fs/gfs2/incore.h index 04f2d78e8658..a8c95c5293c6 100644 --- a/fs/gfs2/incore.h +++ b/fs/gfs2/incore.h @@ -452,7 +452,7 @@ struct gfs2_quota_data { s64 qd_change_sync; unsigned int qd_slot; - unsigned int qd_slot_count; + unsigned int qd_slot_ref; struct buffer_head *qd_bh; struct gfs2_quota_change *qd_bh_qc; @@ -537,6 +537,7 @@ struct gfs2_statfs_change_host { #define GFS2_QUOTA_OFF 0 #define GFS2_QUOTA_ACCOUNT 1 #define GFS2_QUOTA_ON 2 +#define GFS2_QUOTA_QUIET 3 /* on but not complaining */ #define GFS2_DATA_DEFAULT GFS2_DATA_ORDERED #define GFS2_DATA_WRITEBACK 1 @@ -606,7 +607,7 @@ enum { SDF_REMOTE_WITHDRAW = 13, /* Performing remote recovery */ SDF_WITHDRAW_RECOVERY = 14, /* Wait for journal recovery when we are withdrawing */ - SDF_DEACTIVATING = 15, + SDF_KILL = 15, SDF_EVICTING = 16, SDF_FROZEN = 17, }; @@ -716,7 +717,7 @@ struct gfs2_sbd { struct gfs2_glock *sd_rename_gl; struct gfs2_glock *sd_freeze_gl; struct work_struct sd_freeze_work; - wait_queue_head_t sd_glock_wait; + wait_queue_head_t sd_kill_wait; wait_queue_head_t sd_async_glock_wait; atomic_t sd_glock_disposal; struct completion sd_locking_init; diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c index a21ac41d6669..0eac04507904 100644 --- a/fs/gfs2/inode.c +++ b/fs/gfs2/inode.c @@ -276,10 +276,16 @@ struct inode *gfs2_lookup_simple(struct inode *dip, const char *name) * gfs2_lookup_simple callers expect ENOENT * and do not check for NULL. */ - if (inode == NULL) - return ERR_PTR(-ENOENT); - else - return inode; + if (IS_ERR_OR_NULL(inode)) + return inode ? inode : ERR_PTR(-ENOENT); + + /* + * Must not call back into the filesystem when allocating + * pages in the metadata inode's address space. + */ + mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS); + + return inode; } diff --git a/fs/gfs2/lock_dlm.c b/fs/gfs2/lock_dlm.c index 54911294687c..59ab18c79889 100644 --- a/fs/gfs2/lock_dlm.c +++ b/fs/gfs2/lock_dlm.c @@ -222,11 +222,6 @@ static u32 make_flags(struct gfs2_glock *gl, const unsigned int gfs_flags, lkf |= DLM_LKF_NOQUEUEBAST; } - if (gfs_flags & LM_FLAG_PRIORITY) { - lkf |= DLM_LKF_NOORDER; - lkf |= DLM_LKF_HEADQUE; - } - if (gfs_flags & LM_FLAG_ANY) { if (req == DLM_LOCK_PR) lkf |= DLM_LKF_ALTCW; diff --git a/fs/gfs2/log.c b/fs/gfs2/log.c index aa568796207c..e5271ae87d1c 100644 --- a/fs/gfs2/log.c +++ b/fs/gfs2/log.c @@ -1227,6 +1227,21 @@ static void log_refund(struct gfs2_sbd *sdp, struct gfs2_trans *tr) gfs2_log_unlock(sdp); } +static inline int gfs2_jrnl_flush_reqd(struct gfs2_sbd *sdp) +{ + return atomic_read(&sdp->sd_log_pinned) + + atomic_read(&sdp->sd_log_blks_needed) >= + atomic_read(&sdp->sd_log_thresh1); +} + +static inline int gfs2_ail_flush_reqd(struct gfs2_sbd *sdp) +{ + return sdp->sd_jdesc->jd_blocks - + atomic_read(&sdp->sd_log_blks_free) + + atomic_read(&sdp->sd_log_blks_needed) >= + atomic_read(&sdp->sd_log_thresh2); +} + /** * gfs2_log_commit - Commit a transaction to the log * @sdp: the filesystem @@ -1246,9 +1261,7 @@ void gfs2_log_commit(struct gfs2_sbd *sdp, struct gfs2_trans *tr) { log_refund(sdp, tr); - if (atomic_read(&sdp->sd_log_pinned) > atomic_read(&sdp->sd_log_thresh1) || - ((sdp->sd_jdesc->jd_blocks - atomic_read(&sdp->sd_log_blks_free)) > - atomic_read(&sdp->sd_log_thresh2))) + if (gfs2_ail_flush_reqd(sdp) || gfs2_jrnl_flush_reqd(sdp)) wake_up(&sdp->sd_logd_waitq); } @@ -1271,24 +1284,6 @@ static void gfs2_log_shutdown(struct gfs2_sbd *sdp) gfs2_assert_warn(sdp, list_empty(&sdp->sd_ail2_list)); } -static inline int gfs2_jrnl_flush_reqd(struct gfs2_sbd *sdp) -{ - return (atomic_read(&sdp->sd_log_pinned) + - atomic_read(&sdp->sd_log_blks_needed) >= - atomic_read(&sdp->sd_log_thresh1)); -} - -static inline int gfs2_ail_flush_reqd(struct gfs2_sbd *sdp) -{ - unsigned int used_blocks = sdp->sd_jdesc->jd_blocks - atomic_read(&sdp->sd_log_blks_free); - - if (test_and_clear_bit(SDF_FORCE_AIL_FLUSH, &sdp->sd_flags)) - return 1; - - return used_blocks + atomic_read(&sdp->sd_log_blks_needed) >= - atomic_read(&sdp->sd_log_thresh2); -} - /** * gfs2_logd - Update log tail as Active Items get flushed to in-place blocks * @data: Pointer to GFS2 superblock @@ -1301,14 +1296,11 @@ int gfs2_logd(void *data) { struct gfs2_sbd *sdp = data; unsigned long t = 1; - DEFINE_WAIT(wait); while (!kthread_should_stop()) { + if (gfs2_withdrawn(sdp)) + break; - if (gfs2_withdrawn(sdp)) { - msleep_interruptible(HZ); - continue; - } /* Check for errors writing to the journal */ if (sdp->sd_log_error) { gfs2_lm(sdp, @@ -1317,7 +1309,7 @@ int gfs2_logd(void *data) "prevent further damage.\n", sdp->sd_fsname, sdp->sd_log_error); gfs2_withdraw(sdp); - continue; + break; } if (gfs2_jrnl_flush_reqd(sdp) || t == 0) { @@ -1326,7 +1318,9 @@ int gfs2_logd(void *data) GFS2_LFC_LOGD_JFLUSH_REQD); } - if (gfs2_ail_flush_reqd(sdp)) { + if (test_bit(SDF_FORCE_AIL_FLUSH, &sdp->sd_flags) || + gfs2_ail_flush_reqd(sdp)) { + clear_bit(SDF_FORCE_AIL_FLUSH, &sdp->sd_flags); gfs2_ail1_start(sdp); gfs2_ail1_wait(sdp); gfs2_ail1_empty(sdp, 0); @@ -1338,17 +1332,14 @@ int gfs2_logd(void *data) try_to_freeze(); - do { - prepare_to_wait(&sdp->sd_logd_waitq, &wait, - TASK_INTERRUPTIBLE); - if (!gfs2_ail_flush_reqd(sdp) && - !gfs2_jrnl_flush_reqd(sdp) && - !kthread_should_stop()) - t = schedule_timeout(t); - } while(t && !gfs2_ail_flush_reqd(sdp) && - !gfs2_jrnl_flush_reqd(sdp) && - !kthread_should_stop()); - finish_wait(&sdp->sd_logd_waitq, &wait); + t = wait_event_interruptible_timeout(sdp->sd_logd_waitq, + test_bit(SDF_FORCE_AIL_FLUSH, &sdp->sd_flags) || + gfs2_ail_flush_reqd(sdp) || + gfs2_jrnl_flush_reqd(sdp) || + sdp->sd_log_error || + gfs2_withdrawn(sdp) || + kthread_should_stop(), + t); } return 0; diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index 251322b01631..483f69807062 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -456,7 +456,7 @@ static bool gfs2_jhead_pg_srch(struct gfs2_jdesc *jd, * Find the folio with 'index' in the journal's mapping. Search the folio for * the journal head if requested (cleanup == false). Release refs on the * folio so the page cache can reclaim it. We grabbed a - * reference on this folio twice, first when we did a find_or_create_page() + * reference on this folio twice, first when we did a grab_cache_page() * to obtain the folio to add it to the bio and second when we do a * filemap_get_folio() here to get the folio to wait on while I/O on it is being * completed. @@ -481,7 +481,7 @@ static void gfs2_jhead_process_page(struct gfs2_jdesc *jd, unsigned long index, if (!*done) *done = gfs2_jhead_pg_srch(jd, head, &folio->page); - /* filemap_get_folio() and the earlier find_or_create_page() */ + /* filemap_get_folio() and the earlier grab_cache_page() */ folio_put_refs(folio, 2); } @@ -535,8 +535,7 @@ int gfs2_find_jhead(struct gfs2_jdesc *jd, struct gfs2_log_header_host *head, for (; block < je->lblock + je->blocks; block++, dblock++) { if (!page) { - page = find_or_create_page(mapping, - block >> shift, GFP_NOFS); + page = grab_cache_page(mapping, block >> shift); if (!page) { ret = -ENOMEM; done = true; diff --git a/fs/gfs2/main.c b/fs/gfs2/main.c index afcb32854f14..66eb98b690a2 100644 --- a/fs/gfs2/main.c +++ b/fs/gfs2/main.c @@ -152,9 +152,9 @@ static int __init init_gfs2_fs(void) goto fail_shrinker; error = -ENOMEM; - gfs_recovery_wq = alloc_workqueue("gfs_recovery", + gfs2_recovery_wq = alloc_workqueue("gfs2_recovery", WQ_MEM_RECLAIM | WQ_FREEZABLE, 0); - if (!gfs_recovery_wq) + if (!gfs2_recovery_wq) goto fail_wq1; gfs2_control_wq = alloc_workqueue("gfs2_control", @@ -162,7 +162,7 @@ static int __init init_gfs2_fs(void) if (!gfs2_control_wq) goto fail_wq2; - gfs2_freeze_wq = alloc_workqueue("freeze_workqueue", 0, 0); + gfs2_freeze_wq = alloc_workqueue("gfs2_freeze", 0, 0); if (!gfs2_freeze_wq) goto fail_wq3; @@ -194,7 +194,7 @@ fail_mempool: fail_wq3: destroy_workqueue(gfs2_control_wq); fail_wq2: - destroy_workqueue(gfs_recovery_wq); + destroy_workqueue(gfs2_recovery_wq); fail_wq1: unregister_shrinker(&gfs2_qd_shrinker); fail_shrinker: @@ -234,7 +234,7 @@ static void __exit exit_gfs2_fs(void) gfs2_unregister_debugfs(); unregister_filesystem(&gfs2_fs_type); unregister_filesystem(&gfs2meta_fs_type); - destroy_workqueue(gfs_recovery_wq); + destroy_workqueue(gfs2_recovery_wq); destroy_workqueue(gfs2_control_wq); destroy_workqueue(gfs2_freeze_wq); list_lru_destroy(&gfs2_qd_lru); diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c index 8a27957dbfee..33ca04733e93 100644 --- a/fs/gfs2/ops_fstype.c +++ b/fs/gfs2/ops_fstype.c @@ -87,7 +87,7 @@ static struct gfs2_sbd *init_sbd(struct super_block *sb) set_bit(SDF_NOJOURNALID, &sdp->sd_flags); gfs2_tune_init(&sdp->sd_tune); - init_waitqueue_head(&sdp->sd_glock_wait); + init_waitqueue_head(&sdp->sd_kill_wait); init_waitqueue_head(&sdp->sd_async_glock_wait); atomic_set(&sdp->sd_glock_disposal, 0); init_completion(&sdp->sd_locking_init); @@ -1103,29 +1103,49 @@ static int init_threads(struct gfs2_sbd *sdp) struct task_struct *p; int error = 0; - p = kthread_run(gfs2_logd, sdp, "gfs2_logd"); + p = kthread_create(gfs2_logd, sdp, "gfs2_logd/%s", sdp->sd_fsname); if (IS_ERR(p)) { error = PTR_ERR(p); - fs_err(sdp, "can't start logd thread: %d\n", error); + fs_err(sdp, "can't create logd thread: %d\n", error); return error; } + get_task_struct(p); sdp->sd_logd_process = p; - p = kthread_run(gfs2_quotad, sdp, "gfs2_quotad"); + p = kthread_create(gfs2_quotad, sdp, "gfs2_quotad/%s", sdp->sd_fsname); if (IS_ERR(p)) { error = PTR_ERR(p); - fs_err(sdp, "can't start quotad thread: %d\n", error); + fs_err(sdp, "can't create quotad thread: %d\n", error); goto fail; } + get_task_struct(p); sdp->sd_quotad_process = p; + + wake_up_process(sdp->sd_logd_process); + wake_up_process(sdp->sd_quotad_process); return 0; fail: kthread_stop(sdp->sd_logd_process); + put_task_struct(sdp->sd_logd_process); sdp->sd_logd_process = NULL; return error; } +void gfs2_destroy_threads(struct gfs2_sbd *sdp) +{ + if (sdp->sd_logd_process) { + kthread_stop(sdp->sd_logd_process); + put_task_struct(sdp->sd_logd_process); + sdp->sd_logd_process = NULL; + } + if (sdp->sd_quotad_process) { + kthread_stop(sdp->sd_quotad_process); + put_task_struct(sdp->sd_quotad_process); + sdp->sd_quotad_process = NULL; + } +} + /** * gfs2_fill_super - Read in superblock * @sb: The VFS superblock @@ -1276,12 +1296,7 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc) if (error) { gfs2_freeze_unlock(&sdp->sd_freeze_gh); - if (sdp->sd_quotad_process) - kthread_stop(sdp->sd_quotad_process); - sdp->sd_quotad_process = NULL; - if (sdp->sd_logd_process) - kthread_stop(sdp->sd_logd_process); - sdp->sd_logd_process = NULL; + gfs2_destroy_threads(sdp); fs_err(sdp, "can't make FS RW: %d\n", error); goto fail_per_node; } @@ -1381,6 +1396,7 @@ static const struct constant_table gfs2_param_quota[] = { {"off", GFS2_QUOTA_OFF}, {"account", GFS2_QUOTA_ACCOUNT}, {"on", GFS2_QUOTA_ON}, + {"quiet", GFS2_QUOTA_QUIET}, {} }; @@ -1786,9 +1802,9 @@ static void gfs2_kill_sb(struct super_block *sb) /* * Flush and then drain the delete workqueue here (via * destroy_workqueue()) to ensure that any delete work that - * may be running will also see the SDF_DEACTIVATING flag. + * may be running will also see the SDF_KILL flag. */ - set_bit(SDF_DEACTIVATING, &sdp->sd_flags); + set_bit(SDF_KILL, &sdp->sd_flags); gfs2_flush_delete_work(sdp); destroy_workqueue(sdp->sd_delete_wq); diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c index aa5fd06d47bc..171b2713d2e5 100644 --- a/fs/gfs2/quota.c +++ b/fs/gfs2/quota.c @@ -109,38 +109,44 @@ static inline void spin_unlock_bucket(unsigned int hash) static void gfs2_qd_dealloc(struct rcu_head *rcu) { struct gfs2_quota_data *qd = container_of(rcu, struct gfs2_quota_data, qd_rcu); + struct gfs2_sbd *sdp = qd->qd_sbd; + kmem_cache_free(gfs2_quotad_cachep, qd); + if (atomic_dec_and_test(&sdp->sd_quota_count)) + wake_up(&sdp->sd_kill_wait); } -static void gfs2_qd_dispose(struct list_head *list) +static void gfs2_qd_dispose(struct gfs2_quota_data *qd) +{ + struct gfs2_sbd *sdp = qd->qd_sbd; + + spin_lock(&qd_lock); + list_del(&qd->qd_list); + spin_unlock(&qd_lock); + + spin_lock_bucket(qd->qd_hash); + hlist_bl_del_rcu(&qd->qd_hlist); + spin_unlock_bucket(qd->qd_hash); + + if (!gfs2_withdrawn(sdp)) { + gfs2_assert_warn(sdp, !qd->qd_change); + gfs2_assert_warn(sdp, !qd->qd_slot_ref); + gfs2_assert_warn(sdp, !qd->qd_bh_count); + } + + gfs2_glock_put(qd->qd_gl); + call_rcu(&qd->qd_rcu, gfs2_qd_dealloc); +} + +static void gfs2_qd_list_dispose(struct list_head *list) { struct gfs2_quota_data *qd; - struct gfs2_sbd *sdp; while (!list_empty(list)) { qd = list_first_entry(list, struct gfs2_quota_data, qd_lru); - sdp = qd->qd_gl->gl_name.ln_sbd; - list_del(&qd->qd_lru); - /* Free from the filesystem-specific list */ - spin_lock(&qd_lock); - list_del(&qd->qd_list); - spin_unlock(&qd_lock); - - spin_lock_bucket(qd->qd_hash); - hlist_bl_del_rcu(&qd->qd_hlist); - spin_unlock_bucket(qd->qd_hash); - - gfs2_assert_warn(sdp, !qd->qd_change); - gfs2_assert_warn(sdp, !qd->qd_slot_count); - gfs2_assert_warn(sdp, !qd->qd_bh_count); - - gfs2_glock_put(qd->qd_gl); - atomic_dec(&sdp->sd_quota_count); - - /* Delete it from the common reclaim list */ - call_rcu(&qd->qd_rcu, gfs2_qd_dealloc); + gfs2_qd_dispose(qd); } } @@ -149,18 +155,22 @@ static enum lru_status gfs2_qd_isolate(struct list_head *item, struct list_lru_one *lru, spinlock_t *lru_lock, void *arg) { struct list_head *dispose = arg; - struct gfs2_quota_data *qd = list_entry(item, struct gfs2_quota_data, qd_lru); + struct gfs2_quota_data *qd = + list_entry(item, struct gfs2_quota_data, qd_lru); + enum lru_status status; if (!spin_trylock(&qd->qd_lockref.lock)) return LRU_SKIP; + status = LRU_SKIP; if (qd->qd_lockref.count == 0) { lockref_mark_dead(&qd->qd_lockref); list_lru_isolate_move(lru, &qd->qd_lru, dispose); + status = LRU_REMOVED; } spin_unlock(&qd->qd_lockref.lock); - return LRU_REMOVED; + return status; } static unsigned long gfs2_qd_shrink_scan(struct shrinker *shrink, @@ -175,7 +185,7 @@ static unsigned long gfs2_qd_shrink_scan(struct shrinker *shrink, freed = list_lru_shrink_walk(&gfs2_qd_lru, sc, gfs2_qd_isolate, &dispose); - gfs2_qd_dispose(&dispose); + gfs2_qd_list_dispose(&dispose); return freed; } @@ -203,12 +213,7 @@ static u64 qd2index(struct gfs2_quota_data *qd) static u64 qd2offset(struct gfs2_quota_data *qd) { - u64 offset; - - offset = qd2index(qd); - offset *= sizeof(struct gfs2_quota); - - return offset; + return qd2index(qd) * sizeof(struct gfs2_quota); } static struct gfs2_quota_data *qd_alloc(unsigned hash, struct gfs2_sbd *sdp, struct kqid qid) @@ -221,7 +226,7 @@ static struct gfs2_quota_data *qd_alloc(unsigned hash, struct gfs2_sbd *sdp, str return NULL; qd->qd_sbd = sdp; - qd->qd_lockref.count = 1; + qd->qd_lockref.count = 0; spin_lock_init(&qd->qd_lockref.lock); qd->qd_id = qid; qd->qd_slot = -1; @@ -283,6 +288,7 @@ static int qd_get(struct gfs2_sbd *sdp, struct kqid qid, spin_lock_bucket(hash); *qdp = qd = gfs2_qd_search_bucket(hash, sdp, qid); if (qd == NULL) { + new_qd->qd_lockref.count++; *qdp = new_qd; list_add(&new_qd->qd_list, &sdp->sd_quota_list); hlist_bl_add_head_rcu(&new_qd->qd_hlist, &qd_hash_table[hash]); @@ -302,20 +308,31 @@ static int qd_get(struct gfs2_sbd *sdp, struct kqid qid, static void qd_hold(struct gfs2_quota_data *qd) { - struct gfs2_sbd *sdp = qd->qd_gl->gl_name.ln_sbd; + struct gfs2_sbd *sdp = qd->qd_sbd; gfs2_assert(sdp, !__lockref_is_dead(&qd->qd_lockref)); lockref_get(&qd->qd_lockref); } static void qd_put(struct gfs2_quota_data *qd) { + struct gfs2_sbd *sdp; + if (lockref_put_or_lock(&qd->qd_lockref)) return; + BUG_ON(__lockref_is_dead(&qd->qd_lockref)); + sdp = qd->qd_sbd; + if (unlikely(!test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags))) { + lockref_mark_dead(&qd->qd_lockref); + spin_unlock(&qd->qd_lockref.lock); + + gfs2_qd_dispose(qd); + return; + } + qd->qd_lockref.count = 0; list_lru_add(&gfs2_qd_lru, &qd->qd_lru); spin_unlock(&qd->qd_lockref.lock); - } static int slot_get(struct gfs2_quota_data *qd) @@ -325,20 +342,19 @@ static int slot_get(struct gfs2_quota_data *qd) int error = 0; spin_lock(&sdp->sd_bitmap_lock); - if (qd->qd_slot_count != 0) - goto out; - - error = -ENOSPC; - bit = find_first_zero_bit(sdp->sd_quota_bitmap, sdp->sd_quota_slots); - if (bit < sdp->sd_quota_slots) { + if (qd->qd_slot_ref == 0) { + bit = find_first_zero_bit(sdp->sd_quota_bitmap, + sdp->sd_quota_slots); + if (bit >= sdp->sd_quota_slots) { + error = -ENOSPC; + goto out; + } set_bit(bit, sdp->sd_quota_bitmap); qd->qd_slot = bit; - error = 0; -out: - qd->qd_slot_count++; } + qd->qd_slot_ref++; +out: spin_unlock(&sdp->sd_bitmap_lock); - return error; } @@ -347,8 +363,8 @@ static void slot_hold(struct gfs2_quota_data *qd) struct gfs2_sbd *sdp = qd->qd_sbd; spin_lock(&sdp->sd_bitmap_lock); - gfs2_assert(sdp, qd->qd_slot_count); - qd->qd_slot_count++; + gfs2_assert(sdp, qd->qd_slot_ref); + qd->qd_slot_ref++; spin_unlock(&sdp->sd_bitmap_lock); } @@ -357,8 +373,8 @@ static void slot_put(struct gfs2_quota_data *qd) struct gfs2_sbd *sdp = qd->qd_sbd; spin_lock(&sdp->sd_bitmap_lock); - gfs2_assert(sdp, qd->qd_slot_count); - if (!--qd->qd_slot_count) { + gfs2_assert(sdp, qd->qd_slot_ref); + if (!--qd->qd_slot_ref) { BUG_ON(!test_and_clear_bit(qd->qd_slot, sdp->sd_quota_bitmap)); qd->qd_slot = -1; } @@ -367,7 +383,7 @@ static void slot_put(struct gfs2_quota_data *qd) static int bh_get(struct gfs2_quota_data *qd) { - struct gfs2_sbd *sdp = qd->qd_gl->gl_name.ln_sbd; + struct gfs2_sbd *sdp = qd->qd_sbd; struct inode *inode = sdp->sd_qc_inode; struct gfs2_inode *ip = GFS2_I(inode); unsigned int block, offset; @@ -421,7 +437,7 @@ fail: static void bh_put(struct gfs2_quota_data *qd) { - struct gfs2_sbd *sdp = qd->qd_gl->gl_name.ln_sbd; + struct gfs2_sbd *sdp = qd->qd_sbd; mutex_lock(&sdp->sd_quota_mutex); gfs2_assert(sdp, qd->qd_bh_count); @@ -451,6 +467,20 @@ static int qd_check_sync(struct gfs2_sbd *sdp, struct gfs2_quota_data *qd, return 1; } +static int qd_bh_get_or_undo(struct gfs2_sbd *sdp, struct gfs2_quota_data *qd) +{ + int error; + + error = bh_get(qd); + if (!error) + return 0; + + clear_bit(QDF_LOCKED, &qd->qd_flags); + slot_put(qd); + qd_put(qd); + return error; +} + static int qd_fish(struct gfs2_sbd *sdp, struct gfs2_quota_data **qdp) { struct gfs2_quota_data *qd = NULL, *iter; @@ -473,30 +503,29 @@ static int qd_fish(struct gfs2_sbd *sdp, struct gfs2_quota_data **qdp) spin_unlock(&qd_lock); if (qd) { - error = bh_get(qd); - if (error) { - clear_bit(QDF_LOCKED, &qd->qd_flags); - slot_put(qd); - qd_put(qd); + error = qd_bh_get_or_undo(sdp, qd); + if (error) return error; - } + *qdp = qd; } - *qdp = qd; - return 0; } -static void qd_unlock(struct gfs2_quota_data *qd) +static void qdsb_put(struct gfs2_quota_data *qd) { - gfs2_assert_warn(qd->qd_gl->gl_name.ln_sbd, - test_bit(QDF_LOCKED, &qd->qd_flags)); - clear_bit(QDF_LOCKED, &qd->qd_flags); bh_put(qd); slot_put(qd); qd_put(qd); } +static void qd_unlock(struct gfs2_quota_data *qd) +{ + gfs2_assert_warn(qd->qd_sbd, test_bit(QDF_LOCKED, &qd->qd_flags)); + clear_bit(QDF_LOCKED, &qd->qd_flags); + qdsb_put(qd); +} + static int qdsb_get(struct gfs2_sbd *sdp, struct kqid qid, struct gfs2_quota_data **qdp) { @@ -523,13 +552,6 @@ fail: return error; } -static void qdsb_put(struct gfs2_quota_data *qd) -{ - bh_put(qd); - slot_put(qd); - qd_put(qd); -} - /** * gfs2_qa_get - make sure we have a quota allocations data structure, * if necessary @@ -666,7 +688,7 @@ static int sort_qd(const void *a, const void *b) static void do_qc(struct gfs2_quota_data *qd, s64 change, int qc_type) { - struct gfs2_sbd *sdp = qd->qd_gl->gl_name.ln_sbd; + struct gfs2_sbd *sdp = qd->qd_sbd; struct gfs2_inode *ip = GFS2_I(sdp->sd_qc_inode); struct gfs2_quota_change *qc = qd->qd_bh_qc; s64 x; @@ -708,30 +730,29 @@ static void do_qc(struct gfs2_quota_data *qd, s64 change, int qc_type) mutex_unlock(&sdp->sd_quota_mutex); } -static int gfs2_write_buf_to_page(struct gfs2_inode *ip, unsigned long index, +static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, unsigned off, void *buf, unsigned bytes) { + struct gfs2_inode *ip = GFS2_I(sdp->sd_quota_inode); struct inode *inode = &ip->i_inode; - struct gfs2_sbd *sdp = GFS2_SB(inode); struct address_space *mapping = inode->i_mapping; struct page *page; struct buffer_head *bh; u64 blk; unsigned bsize = sdp->sd_sb.sb_bsize, bnum = 0, boff = 0; unsigned to_write = bytes, pg_off = off; - int done = 0; blk = index << (PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift); boff = off % bsize; - page = find_or_create_page(mapping, index, GFP_NOFS); + page = grab_cache_page(mapping, index); if (!page) return -ENOMEM; if (!page_has_buffers(page)) create_empty_buffers(page, bsize, 0); bh = page_buffers(page); - while (!done) { + for(;;) { /* Find the beginning block within the page */ if (pg_off >= ((bnum * bsize) + bsize)) { bh = bh->b_this_page; @@ -751,10 +772,7 @@ static int gfs2_write_buf_to_page(struct gfs2_inode *ip, unsigned long index, set_buffer_uptodate(bh); if (bh_read(bh, REQ_META | REQ_PRIO) < 0) goto unlock_out; - if (gfs2_is_jdata(ip)) - gfs2_trans_add_data(ip->i_gl, bh); - else - gfs2_ordered_add_inode(ip); + gfs2_trans_add_data(ip->i_gl, bh); /* If we need to write to the next block as well */ if (to_write > (bsize - boff)) { @@ -763,7 +781,7 @@ static int gfs2_write_buf_to_page(struct gfs2_inode *ip, unsigned long index, boff = pg_off % bsize; continue; } - done = 1; + break; } /* Write to the page, now that we have setup the buffer(s) */ @@ -780,12 +798,12 @@ unlock_out: return -EIO; } -static int gfs2_write_disk_quota(struct gfs2_inode *ip, struct gfs2_quota *qp, +static int gfs2_write_disk_quota(struct gfs2_sbd *sdp, struct gfs2_quota *qp, loff_t loc) { unsigned long pg_beg; unsigned pg_off, nbytes, overflow = 0; - int pg_oflow = 0, error; + int error; void *ptr; nbytes = sizeof(struct gfs2_quota); @@ -794,17 +812,15 @@ static int gfs2_write_disk_quota(struct gfs2_inode *ip, struct gfs2_quota *qp, pg_off = offset_in_page(loc); /* If the quota straddles a page boundary, split the write in two */ - if ((pg_off + nbytes) > PAGE_SIZE) { - pg_oflow = 1; + if ((pg_off + nbytes) > PAGE_SIZE) overflow = (pg_off + nbytes) - PAGE_SIZE; - } ptr = qp; - error = gfs2_write_buf_to_page(ip, pg_beg, pg_off, ptr, + error = gfs2_write_buf_to_page(sdp, pg_beg, pg_off, ptr, nbytes - overflow); /* If there's an overflow, write the remaining bytes to the next page */ - if (!error && pg_oflow) - error = gfs2_write_buf_to_page(ip, pg_beg + 1, 0, + if (!error && overflow) + error = gfs2_write_buf_to_page(sdp, pg_beg + 1, 0, ptr + nbytes - overflow, overflow); return error; @@ -812,7 +828,7 @@ static int gfs2_write_disk_quota(struct gfs2_inode *ip, struct gfs2_quota *qp, /** * gfs2_adjust_quota - adjust record of current block usage - * @ip: The quota inode + * @sdp: The superblock * @loc: Offset of the entry in the quota file * @change: The amount of usage change to record * @qd: The quota data @@ -824,12 +840,12 @@ static int gfs2_write_disk_quota(struct gfs2_inode *ip, struct gfs2_quota *qp, * Returns: 0 or -ve on error */ -static int gfs2_adjust_quota(struct gfs2_inode *ip, loff_t loc, +static int gfs2_adjust_quota(struct gfs2_sbd *sdp, loff_t loc, s64 change, struct gfs2_quota_data *qd, struct qc_dqblk *fdq) { + struct gfs2_inode *ip = GFS2_I(sdp->sd_quota_inode); struct inode *inode = &ip->i_inode; - struct gfs2_sbd *sdp = GFS2_SB(inode); struct gfs2_quota q; int err; u64 size; @@ -846,7 +862,6 @@ static int gfs2_adjust_quota(struct gfs2_inode *ip, loff_t loc, return err; loc -= sizeof(q); /* gfs2_internal_read would've advanced the loc ptr */ - err = -EIO; be64_add_cpu(&q.qu_value, change); if (((s64)be64_to_cpu(q.qu_value)) < 0) q.qu_value = 0; /* Never go negative on quota usage */ @@ -866,7 +881,7 @@ static int gfs2_adjust_quota(struct gfs2_inode *ip, loff_t loc, } } - err = gfs2_write_disk_quota(ip, &q, loc); + err = gfs2_write_disk_quota(sdp, &q, loc); if (!err) { size = loc + sizeof(struct gfs2_quota); if (size > inode->i_size) @@ -881,7 +896,7 @@ static int gfs2_adjust_quota(struct gfs2_inode *ip, loff_t loc, static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) { - struct gfs2_sbd *sdp = (*qda)->qd_gl->gl_name.ln_sbd; + struct gfs2_sbd *sdp = (*qda)->qd_sbd; struct gfs2_inode *ip = GFS2_I(sdp->sd_quota_inode); struct gfs2_alloc_parms ap = { .aflags = 0, }; unsigned int data_blocks, ind_blocks; @@ -893,18 +908,12 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) unsigned int nalloc = 0, blocks; int error; - error = gfs2_qa_get(ip); - if (error) - return error; - gfs2_write_calc_reserv(ip, sizeof(struct gfs2_quota), &data_blocks, &ind_blocks); ghs = kmalloc_array(num_qd, sizeof(struct gfs2_holder), GFP_NOFS); - if (!ghs) { - error = -ENOMEM; - goto out; - } + if (!ghs) + return -ENOMEM; sort(qda, num_qd, sizeof(struct gfs2_quota_data *), sort_qd, NULL); inode_lock(&ip->i_inode); @@ -953,7 +962,8 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) for (x = 0; x < num_qd; x++) { qd = qda[x]; offset = qd2offset(qd); - error = gfs2_adjust_quota(ip, offset, qd->qd_change_sync, qd, NULL); + error = gfs2_adjust_quota(sdp, offset, qd->qd_change_sync, qd, + NULL); if (error) goto out_end_trans; @@ -961,8 +971,6 @@ static int do_sync(unsigned int num_qd, struct gfs2_quota_data **qda) set_bit(QDF_REFRESH, &qd->qd_flags); } - error = 0; - out_end_trans: gfs2_trans_end(sdp); out_ipres: @@ -976,8 +984,10 @@ out_dq: kfree(ghs); gfs2_log_flush(ip->i_gl->gl_name.ln_sbd, ip->i_gl, GFS2_LOG_HEAD_FLUSH_NORMAL | GFS2_LFC_DO_SYNC); -out: - gfs2_qa_put(ip); + if (!error) { + for (x = 0; x < num_qd; x++) + qda[x]->qd_sync_gen = sdp->sd_quota_sync_gen; + } return error; } @@ -1009,11 +1019,12 @@ static int update_qd(struct gfs2_sbd *sdp, struct gfs2_quota_data *qd) static int do_glock(struct gfs2_quota_data *qd, int force_refresh, struct gfs2_holder *q_gh) { - struct gfs2_sbd *sdp = qd->qd_gl->gl_name.ln_sbd; + struct gfs2_sbd *sdp = qd->qd_sbd; struct gfs2_inode *ip = GFS2_I(sdp->sd_quota_inode); struct gfs2_holder i_gh; int error; + gfs2_assert_warn(sdp, sdp == qd->qd_gl->gl_name.ln_sbd); restart: error = gfs2_glock_nq_init(qd->qd_gl, LM_ST_SHARED, 0, q_gh); if (error) @@ -1059,9 +1070,10 @@ int gfs2_quota_lock(struct gfs2_inode *ip, kuid_t uid, kgid_t gid) struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode); struct gfs2_quota_data *qd; u32 x; - int error = 0; + int error; - if (sdp->sd_args.ar_quota != GFS2_QUOTA_ON) + if (sdp->sd_args.ar_quota != GFS2_QUOTA_ON && + sdp->sd_args.ar_quota != GFS2_QUOTA_QUIET) return 0; error = gfs2_quota_hold(ip, uid, gid); @@ -1089,16 +1101,15 @@ int gfs2_quota_lock(struct gfs2_inode *ip, kuid_t uid, kgid_t gid) return error; } -static int need_sync(struct gfs2_quota_data *qd) +static bool need_sync(struct gfs2_quota_data *qd) { - struct gfs2_sbd *sdp = qd->qd_gl->gl_name.ln_sbd; + struct gfs2_sbd *sdp = qd->qd_sbd; struct gfs2_tune *gt = &sdp->sd_tune; s64 value; unsigned int num, den; - int do_sync = 1; if (!qd->qd_qb.qb_limit) - return 0; + return false; spin_lock(&qd_lock); value = qd->qd_change; @@ -1109,26 +1120,26 @@ static int need_sync(struct gfs2_quota_data *qd) den = gt->gt_quota_scale_den; spin_unlock(>->gt_spin); - if (value < 0) - do_sync = 0; + if (value <= 0) + return false; else if ((s64)be64_to_cpu(qd->qd_qb.qb_value) >= (s64)be64_to_cpu(qd->qd_qb.qb_limit)) - do_sync = 0; + return false; else { value *= gfs2_jindex_size(sdp) * num; value = div_s64(value, den); value += (s64)be64_to_cpu(qd->qd_qb.qb_value); if (value < (s64)be64_to_cpu(qd->qd_qb.qb_limit)) - do_sync = 0; + return false; } - return do_sync; + return true; } void gfs2_quota_unlock(struct gfs2_inode *ip) { struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode); - struct gfs2_quota_data *qda[4]; + struct gfs2_quota_data *qda[2 * GFS2_MAXQUOTAS]; unsigned int count = 0; u32 x; int found; @@ -1138,7 +1149,7 @@ void gfs2_quota_unlock(struct gfs2_inode *ip) for (x = 0; x < ip->i_qadata->qa_qd_num; x++) { struct gfs2_quota_data *qd; - int sync; + bool sync; qd = ip->i_qadata->qa_qd[x]; sync = need_sync(qd); @@ -1154,15 +1165,8 @@ void gfs2_quota_unlock(struct gfs2_inode *ip) if (!found) continue; - gfs2_assert_warn(sdp, qd->qd_change_sync); - if (bh_get(qd)) { - clear_bit(QDF_LOCKED, &qd->qd_flags); - slot_put(qd); - qd_put(qd); - continue; - } - - qda[count++] = qd; + if (!qd_bh_get_or_undo(sdp, qd)) + qda[count++] = qd; } if (count) { @@ -1178,12 +1182,13 @@ void gfs2_quota_unlock(struct gfs2_inode *ip) static int print_message(struct gfs2_quota_data *qd, char *type) { - struct gfs2_sbd *sdp = qd->qd_gl->gl_name.ln_sbd; + struct gfs2_sbd *sdp = qd->qd_sbd; - fs_info(sdp, "quota %s for %s %u\n", - type, - (qd->qd_id.type == USRQUOTA) ? "user" : "group", - from_kqid(&init_user_ns, qd->qd_id)); + if (sdp->sd_args.ar_quota != GFS2_QUOTA_QUIET) + fs_info(sdp, "quota %s for %s %u\n", + type, + (qd->qd_id.type == USRQUOTA) ? "user" : "group", + from_kqid(&init_user_ns, qd->qd_id)); return 0; } @@ -1269,7 +1274,8 @@ void gfs2_quota_change(struct gfs2_inode *ip, s64 change, u32 x; struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode); - if (sdp->sd_args.ar_quota != GFS2_QUOTA_ON || + if ((sdp->sd_args.ar_quota != GFS2_QUOTA_ON && + sdp->sd_args.ar_quota != GFS2_QUOTA_QUIET) || gfs2_assert_warn(sdp, change)) return; if (ip->i_diskflags & GFS2_DIF_SYSTEM) @@ -1288,6 +1294,24 @@ void gfs2_quota_change(struct gfs2_inode *ip, s64 change, } } +static bool qd_changed(struct gfs2_sbd *sdp) +{ + struct gfs2_quota_data *qd; + bool changed = false; + + spin_lock(&qd_lock); + list_for_each_entry(qd, &sdp->sd_quota_list, qd_list) { + if (test_bit(QDF_LOCKED, &qd->qd_flags) || + !test_bit(QDF_CHANGE, &qd->qd_flags)) + continue; + + changed = true; + break; + } + spin_unlock(&qd_lock); + return changed; +} + int gfs2_quota_sync(struct super_block *sb, int type) { struct gfs2_sbd *sdp = sb->s_fs_info; @@ -1297,6 +1321,9 @@ int gfs2_quota_sync(struct super_block *sb, int type) unsigned int x; int error = 0; + if (!qd_changed(sdp)) + return 0; + qda = kcalloc(max_qd, sizeof(struct gfs2_quota_data *), GFP_KERNEL); if (!qda) return -ENOMEM; @@ -1318,10 +1345,6 @@ int gfs2_quota_sync(struct super_block *sb, int type) if (num_qd) { if (!error) error = do_sync(num_qd, qda); - if (!error) - for (x = 0; x < num_qd; x++) - qda[x]->qd_sync_gen = - sdp->sd_quota_sync_gen; for (x = 0; x < num_qd; x++) qd_unlock(qda[x]); @@ -1423,7 +1446,7 @@ int gfs2_quota_init(struct gfs2_sbd *sdp) set_bit(QDF_CHANGE, &qd->qd_flags); qd->qd_change = qc_change; qd->qd_slot = slot; - qd->qd_slot_count = 1; + qd->qd_slot_ref = 1; spin_lock(&qd_lock); BUG_ON(test_and_set_bit(slot, sdp->sd_quota_bitmap)); @@ -1455,36 +1478,35 @@ fail: void gfs2_quota_cleanup(struct gfs2_sbd *sdp) { - struct list_head *head = &sdp->sd_quota_list; struct gfs2_quota_data *qd; + LIST_HEAD(dispose); + int count; + + BUG_ON(test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags)); spin_lock(&qd_lock); - while (!list_empty(head)) { - qd = list_last_entry(head, struct gfs2_quota_data, qd_list); + list_for_each_entry(qd, &sdp->sd_quota_list, qd_list) { + spin_lock(&qd->qd_lockref.lock); + if (qd->qd_lockref.count != 0) { + spin_unlock(&qd->qd_lockref.lock); + continue; + } + lockref_mark_dead(&qd->qd_lockref); + spin_unlock(&qd->qd_lockref.lock); - list_del(&qd->qd_list); - - /* Also remove if this qd exists in the reclaim list */ list_lru_del(&gfs2_qd_lru, &qd->qd_lru); - atomic_dec(&sdp->sd_quota_count); - spin_unlock(&qd_lock); - - spin_lock_bucket(qd->qd_hash); - hlist_bl_del_rcu(&qd->qd_hlist); - spin_unlock_bucket(qd->qd_hash); - - gfs2_assert_warn(sdp, !qd->qd_change); - gfs2_assert_warn(sdp, !qd->qd_slot_count); - gfs2_assert_warn(sdp, !qd->qd_bh_count); - - gfs2_glock_put(qd->qd_gl); - call_rcu(&qd->qd_rcu, gfs2_qd_dealloc); - - spin_lock(&qd_lock); + list_add(&qd->qd_lru, &dispose); } spin_unlock(&qd_lock); - gfs2_assert_warn(sdp, !atomic_read(&sdp->sd_quota_count)); + gfs2_qd_list_dispose(&dispose); + + wait_event_timeout(sdp->sd_kill_wait, + (count = atomic_read(&sdp->sd_quota_count)) == 0, + HZ * 60); + + if (count != 0) + fs_err(sdp, "%d left-over quota data objects\n", count); kvfree(sdp->sd_quota_bitmap); sdp->sd_quota_bitmap = NULL; @@ -1536,12 +1558,11 @@ int gfs2_quotad(void *data) unsigned long statfs_timeo = 0; unsigned long quotad_timeo = 0; unsigned long t = 0; - DEFINE_WAIT(wait); while (!kthread_should_stop()) { - if (gfs2_withdrawn(sdp)) - goto bypass; + break; + /* Update the master statfs file */ if (sdp->sd_statfs_force_sync) { int error = gfs2_statfs_sync(sdp->sd_vfs, 0); @@ -1559,15 +1580,16 @@ int gfs2_quotad(void *data) try_to_freeze(); -bypass: t = min(quotad_timeo, statfs_timeo); - prepare_to_wait(&sdp->sd_quota_wait, &wait, TASK_INTERRUPTIBLE); - if (!sdp->sd_statfs_force_sync) - t -= schedule_timeout(t); - else + t = wait_event_interruptible_timeout(sdp->sd_quota_wait, + sdp->sd_statfs_force_sync || + gfs2_withdrawn(sdp) || + kthread_should_stop(), + t); + + if (sdp->sd_statfs_force_sync) t = 0; - finish_wait(&sdp->sd_quota_wait, &wait); } return 0; @@ -1580,6 +1602,8 @@ static int gfs2_quota_get_state(struct super_block *sb, struct qc_state *state) memset(state, 0, sizeof(*state)); switch (sdp->sd_args.ar_quota) { + case GFS2_QUOTA_QUIET: + fallthrough; case GFS2_QUOTA_ON: state->s_state[USRQUOTA].flags |= QCI_LIMITS_ENFORCED; state->s_state[GRPQUOTA].flags |= QCI_LIMITS_ENFORCED; @@ -1726,7 +1750,7 @@ static int gfs2_set_dqblk(struct super_block *sb, struct kqid qid, goto out_release; /* Apply changes */ - error = gfs2_adjust_quota(ip, offset, 0, qd, fdq); + error = gfs2_adjust_quota(sdp, offset, 0, qd, fdq); if (!error) clear_bit(QDF_QMSG_QUIET, &qd->qd_flags); diff --git a/fs/gfs2/recovery.c b/fs/gfs2/recovery.c index 9c7a9f640bad..5aae02669a40 100644 --- a/fs/gfs2/recovery.c +++ b/fs/gfs2/recovery.c @@ -27,7 +27,7 @@ #include "util.h" #include "dir.h" -struct workqueue_struct *gfs_recovery_wq; +struct workqueue_struct *gfs2_recovery_wq; int gfs2_replay_read_block(struct gfs2_jdesc *jd, unsigned int blk, struct buffer_head **bh) @@ -570,7 +570,7 @@ int gfs2_recover_journal(struct gfs2_jdesc *jd, bool wait) return -EBUSY; /* we have JDF_RECOVERY, queue should always succeed */ - rv = queue_work(gfs_recovery_wq, &jd->jd_work); + rv = queue_work(gfs2_recovery_wq, &jd->jd_work); BUG_ON(!rv); if (wait) diff --git a/fs/gfs2/recovery.h b/fs/gfs2/recovery.h index 0d30f8e804f4..7a0c9d0b7503 100644 --- a/fs/gfs2/recovery.h +++ b/fs/gfs2/recovery.h @@ -9,7 +9,7 @@ #include "incore.h" -extern struct workqueue_struct *gfs_recovery_wq; +extern struct workqueue_struct *gfs2_recovery_wq; static inline void gfs2_replay_incr_blk(struct gfs2_jdesc *jd, u32 *blk) { diff --git a/fs/gfs2/super.c b/fs/gfs2/super.c index 2f701335e8ee..02d93da21b2b 100644 --- a/fs/gfs2/super.c +++ b/fs/gfs2/super.c @@ -546,20 +546,10 @@ void gfs2_make_fs_ro(struct gfs2_sbd *sdp) { int log_write_allowed = test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags); - if (!test_bit(SDF_DEACTIVATING, &sdp->sd_flags)) + if (!test_bit(SDF_KILL, &sdp->sd_flags)) gfs2_flush_delete_work(sdp); - if (!log_write_allowed && current == sdp->sd_quotad_process) - fs_warn(sdp, "The quotad daemon is withdrawing.\n"); - else if (sdp->sd_quotad_process) - kthread_stop(sdp->sd_quotad_process); - sdp->sd_quotad_process = NULL; - - if (!log_write_allowed && current == sdp->sd_logd_process) - fs_warn(sdp, "The logd daemon is withdrawing.\n"); - else if (sdp->sd_logd_process) - kthread_stop(sdp->sd_logd_process); - sdp->sd_logd_process = NULL; + gfs2_destroy_threads(sdp); if (log_write_allowed) { gfs2_quota_sync(sdp->sd_vfs, 0); @@ -580,15 +570,8 @@ void gfs2_make_fs_ro(struct gfs2_sbd *sdp) gfs2_log_is_empty(sdp), HZ * 5); gfs2_assert_warn(sdp, gfs2_log_is_empty(sdp)); - } else { - wait_event_timeout(sdp->sd_log_waitq, - gfs2_log_is_empty(sdp), - HZ * 5); } gfs2_quota_cleanup(sdp); - - if (!log_write_allowed) - sdp->sd_vfs->s_flags |= SB_RDONLY; } /** @@ -622,6 +605,10 @@ restart: if (!sb_rdonly(sb)) { gfs2_make_fs_ro(sdp); } + if (gfs2_withdrawn(sdp)) { + gfs2_destroy_threads(sdp); + gfs2_quota_cleanup(sdp); + } WARN_ON(gfs2_withdrawing(sdp)); /* At this point, we're through modifying the disk */ @@ -1134,6 +1121,9 @@ static int gfs2_show_options(struct seq_file *s, struct dentry *root) case GFS2_QUOTA_ON: state = "on"; break; + case GFS2_QUOTA_QUIET: + state = "quiet"; + break; default: state = "unknown"; break; diff --git a/fs/gfs2/super.h b/fs/gfs2/super.h index bba58629bc45..ab9c83106932 100644 --- a/fs/gfs2/super.h +++ b/fs/gfs2/super.h @@ -36,6 +36,7 @@ extern int gfs2_lookup_in_master_dir(struct gfs2_sbd *sdp, char *filename, extern int gfs2_make_fs_rw(struct gfs2_sbd *sdp); extern void gfs2_make_fs_ro(struct gfs2_sbd *sdp); extern void gfs2_online_uevent(struct gfs2_sbd *sdp); +extern void gfs2_destroy_threads(struct gfs2_sbd *sdp); extern int gfs2_statfs_init(struct gfs2_sbd *sdp); extern void gfs2_statfs_change(struct gfs2_sbd *sdp, s64 total, s64 free, s64 dinodes); diff --git a/fs/gfs2/sys.c b/fs/gfs2/sys.c index c60bc7f628e1..60a0206890c5 100644 --- a/fs/gfs2/sys.c +++ b/fs/gfs2/sys.c @@ -98,7 +98,10 @@ static ssize_t status_show(struct gfs2_sbd *sdp, char *buf) "sd_log_flush_head: %d\n" "sd_log_flush_tail: %d\n" "sd_log_blks_reserved: %d\n" - "sd_log_revokes_available: %d\n", + "sd_log_revokes_available: %d\n" + "sd_log_pinned: %d\n" + "sd_log_thresh1: %d\n" + "sd_log_thresh2: %d\n", test_bit(SDF_JOURNAL_CHECKED, &f), test_bit(SDF_JOURNAL_LIVE, &f), (sdp->sd_jdesc ? sdp->sd_jdesc->jd_jid : 0), @@ -118,7 +121,7 @@ static ssize_t status_show(struct gfs2_sbd *sdp, char *buf) test_bit(SDF_WITHDRAW_IN_PROG, &f), test_bit(SDF_REMOTE_WITHDRAW, &f), test_bit(SDF_WITHDRAW_RECOVERY, &f), - test_bit(SDF_DEACTIVATING, &f), + test_bit(SDF_KILL, &f), sdp->sd_log_error, rwsem_is_locked(&sdp->sd_log_flush_lock), sdp->sd_log_num_revoke, @@ -128,7 +131,10 @@ static ssize_t status_show(struct gfs2_sbd *sdp, char *buf) sdp->sd_log_flush_head, sdp->sd_log_flush_tail, sdp->sd_log_blks_reserved, - atomic_read(&sdp->sd_log_revokes_available)); + atomic_read(&sdp->sd_log_revokes_available), + atomic_read(&sdp->sd_log_pinned), + atomic_read(&sdp->sd_log_thresh1), + atomic_read(&sdp->sd_log_thresh2)); return s; } diff --git a/fs/gfs2/util.c b/fs/gfs2/util.c index dac22b1c1a2e..da29fafb6272 100644 --- a/fs/gfs2/util.c +++ b/fs/gfs2/util.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -150,7 +151,14 @@ static void signal_our_withdraw(struct gfs2_sbd *sdp) if (!sb_rdonly(sdp->sd_vfs)) { bool locked = mutex_trylock(&sdp->sd_freeze_mutex); - gfs2_make_fs_ro(sdp); + wake_up(&sdp->sd_logd_waitq); + wake_up(&sdp->sd_quota_wait); + + wait_event_timeout(sdp->sd_log_waitq, + gfs2_log_is_empty(sdp), + HZ * 5); + + sdp->sd_vfs->s_flags |= SB_RDONLY; if (locked) mutex_unlock(&sdp->sd_freeze_mutex); @@ -315,19 +323,19 @@ int gfs2_withdraw(struct gfs2_sbd *sdp) struct lm_lockstruct *ls = &sdp->sd_lockstruct; const struct lm_lockops *lm = ls->ls_ops; - if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW && - test_and_set_bit(SDF_WITHDRAWN, &sdp->sd_flags)) { - if (!test_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags)) - return -1; - - wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_IN_PROG, - TASK_UNINTERRUPTIBLE); - return -1; - } - - set_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags); - if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW) { + unsigned long old = READ_ONCE(sdp->sd_flags), new; + + do { + if (old & BIT(SDF_WITHDRAWN)) { + wait_on_bit(&sdp->sd_flags, + SDF_WITHDRAW_IN_PROG, + TASK_UNINTERRUPTIBLE); + return -1; + } + new = old | BIT(SDF_WITHDRAWN) | BIT(SDF_WITHDRAW_IN_PROG); + } while (unlikely(!try_cmpxchg(&sdp->sd_flags, &old, new))); + fs_err(sdp, "about to withdraw this file system\n"); BUG_ON(sdp->sd_args.ar_debug); diff --git a/fs/nls/Kconfig b/fs/nls/Kconfig index a0d0e2f7ec83..2a601af6f3bd 100644 --- a/fs/nls/Kconfig +++ b/fs/nls/Kconfig @@ -618,11 +618,6 @@ config NLS_UTF8 the Unicode/ISO9646 universal character set. config NLS_UCS2_UTILS - tristate "NLS UCS-2 UTILS" - help - Set of older UCS-2 conversion utilities and tables used by some - filesystems including SMB/CIFS. This includes upper case conversion - tables. This will automatically be selected when the filesystem - that uses it is selected. + tristate endif # NLS diff --git a/fs/ntfs3/super.c b/fs/ntfs3/super.c index 5fffddea554f..cfec5e0c7f66 100644 --- a/fs/ntfs3/super.c +++ b/fs/ntfs3/super.c @@ -571,12 +571,8 @@ static void init_once(void *foo) /* * Noinline to reduce binary size. */ -static noinline void ntfs3_free_sbi(struct ntfs_sb_info *sbi) +static noinline void ntfs3_put_sbi(struct ntfs_sb_info *sbi) { - kfree(sbi->new_rec); - kvfree(ntfs_put_shared(sbi->upcase)); - kfree(sbi->def_table); - wnd_close(&sbi->mft.bitmap); wnd_close(&sbi->used.bitmap); @@ -601,6 +597,13 @@ static noinline void ntfs3_free_sbi(struct ntfs_sb_info *sbi) indx_clear(&sbi->security.index_sdh); indx_clear(&sbi->reparse.index_r); indx_clear(&sbi->objid.index_o); +} + +static void ntfs3_free_sbi(struct ntfs_sb_info *sbi) +{ + kfree(sbi->new_rec); + kvfree(ntfs_put_shared(sbi->upcase)); + kfree(sbi->def_table); kfree(sbi->compress.lznt); #ifdef CONFIG_NTFS3_LZX_XPRESS xpress_free_decompressor(sbi->compress.xpress); @@ -625,6 +628,7 @@ static void ntfs_put_super(struct super_block *sb) /* Mark rw ntfs as clear, if possible. */ ntfs_set_state(sbi, NTFS_DIRTY_CLEAR); + ntfs3_put_sbi(sbi); } static int ntfs_statfs(struct dentry *dentry, struct kstatfs *buf) @@ -1644,8 +1648,10 @@ static void ntfs_fs_free(struct fs_context *fc) struct ntfs_mount_options *opts = fc->fs_private; struct ntfs_sb_info *sbi = fc->s_fs_info; - if (sbi) + if (sbi) { + ntfs3_put_sbi(sbi); ntfs3_free_sbi(sbi); + } if (opts) put_mount_options(opts); diff --git a/fs/smb/client/cached_dir.c b/fs/smb/client/cached_dir.c index 2d5e9a9d5b8b..b17f067e4ada 100644 --- a/fs/smb/client/cached_dir.c +++ b/fs/smb/client/cached_dir.c @@ -18,7 +18,8 @@ static void smb2_close_cached_fid(struct kref *ref); static struct cached_fid *find_or_create_cached_dir(struct cached_fids *cfids, const char *path, - bool lookup_only) + bool lookup_only, + __u32 max_cached_dirs) { struct cached_fid *cfid; @@ -43,7 +44,7 @@ static struct cached_fid *find_or_create_cached_dir(struct cached_fids *cfids, spin_unlock(&cfids->cfid_list_lock); return NULL; } - if (cfids->num_entries >= MAX_CACHED_FIDS) { + if (cfids->num_entries >= max_cached_dirs) { spin_unlock(&cfids->cfid_list_lock); return NULL; } @@ -145,7 +146,7 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon, const char *npath; if (tcon == NULL || tcon->cfids == NULL || tcon->nohandlecache || - is_smb1_server(tcon->ses->server)) + is_smb1_server(tcon->ses->server) || (dir_cache_timeout == 0)) return -EOPNOTSUPP; ses = tcon->ses; @@ -162,7 +163,7 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon, if (!utf16_path) return -ENOMEM; - cfid = find_or_create_cached_dir(cfids, path, lookup_only); + cfid = find_or_create_cached_dir(cfids, path, lookup_only, tcon->max_cached_dirs); if (cfid == NULL) { kfree(utf16_path); return -ENOENT; @@ -582,7 +583,7 @@ cifs_cfids_laundromat_thread(void *p) return 0; spin_lock(&cfids->cfid_list_lock); list_for_each_entry_safe(cfid, q, &cfids->entries, entry) { - if (time_after(jiffies, cfid->time + HZ * 30)) { + if (time_after(jiffies, cfid->time + HZ * dir_cache_timeout)) { list_del(&cfid->entry); list_add(&cfid->entry, &entry); cfids->num_entries--; diff --git a/fs/smb/client/cached_dir.h b/fs/smb/client/cached_dir.h index facc9b154d00..a82ff2cea789 100644 --- a/fs/smb/client/cached_dir.h +++ b/fs/smb/client/cached_dir.h @@ -49,7 +49,7 @@ struct cached_fid { struct cached_dirents dirents; }; -#define MAX_CACHED_FIDS 16 +/* default MAX_CACHED_FIDS is 16 */ struct cached_fids { /* Must be held when: * - accessing the cfids->entries list diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c index 73c44e097a69..22869cda1356 100644 --- a/fs/smb/client/cifsfs.c +++ b/fs/smb/client/cifsfs.c @@ -117,6 +117,10 @@ module_param(cifs_max_pending, uint, 0444); MODULE_PARM_DESC(cifs_max_pending, "Simultaneous requests to server for " "CIFS/SMB1 dialect (N/A for SMB3) " "Default: 32767 Range: 2 to 32767."); +unsigned int dir_cache_timeout = 30; +module_param(dir_cache_timeout, uint, 0644); +MODULE_PARM_DESC(dir_cache_timeout, "Number of seconds to cache directory contents for which we have a lease. Default: 30 " + "Range: 1 to 65000 seconds, 0 to disable caching dir contents"); #ifdef CONFIG_CIFS_STATS2 unsigned int slow_rsp_threshold = 1; module_param(slow_rsp_threshold, uint, 0644); @@ -695,6 +699,8 @@ cifs_show_options(struct seq_file *s, struct dentry *root) seq_printf(s, ",snapshot=%llu", tcon->snapshot_time); if (tcon->handle_timeout) seq_printf(s, ",handletimeout=%u", tcon->handle_timeout); + if (tcon->max_cached_dirs != MAX_CACHED_FIDS) + seq_printf(s, ",max_cached_dirs=%u", tcon->max_cached_dirs); /* * Display file and directory attribute timeout in seconds. @@ -1679,6 +1685,12 @@ init_cifs(void) CIFS_MAX_REQ); } + /* Limit max to about 18 hours, and setting to zero disables directory entry caching */ + if (dir_cache_timeout > 65000) { + dir_cache_timeout = 65000; + cifs_dbg(VFS, "dir_cache_timeout set to max of 65000 seconds\n"); + } + cifsiod_wq = alloc_workqueue("cifsiod", WQ_FREEZABLE|WQ_MEM_RECLAIM, 0); if (!cifsiod_wq) { rc = -ENOMEM; diff --git a/fs/smb/client/cifsfs.h b/fs/smb/client/cifsfs.h index 532c38fe07cd..41daebd220ff 100644 --- a/fs/smb/client/cifsfs.h +++ b/fs/smb/client/cifsfs.h @@ -152,6 +152,6 @@ extern const struct export_operations cifs_export_ops; #endif /* CONFIG_CIFS_NFSD_EXPORT */ /* when changing internal version - update following two lines at same time */ -#define SMB3_PRODUCT_BUILD 44 -#define CIFS_VERSION "2.44" +#define SMB3_PRODUCT_BUILD 45 +#define CIFS_VERSION "2.45" #endif /* _CIFSFS_H */ diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h index 259e231f8b4f..032d8716f671 100644 --- a/fs/smb/client/cifsglob.h +++ b/fs/smb/client/cifsglob.h @@ -1210,6 +1210,7 @@ struct cifs_tcon { __u32 max_chunks; __u32 max_bytes_chunk; __u32 max_bytes_copy; + __u32 max_cached_dirs; #ifdef CONFIG_CIFS_FSCACHE u64 resource_id; /* server resource id */ struct fscache_volume *fscache; /* cookie for share */ @@ -2016,6 +2017,7 @@ extern unsigned int CIFSMaxBufSize; /* max size not including hdr */ extern unsigned int cifs_min_rcv; /* min size of big ntwrk buf pool */ extern unsigned int cifs_min_small; /* min size of small buf pool */ extern unsigned int cifs_max_pending; /* MAX requests at once to server*/ +extern unsigned int dir_cache_timeout; /* max time for directory lease caching of dir */ extern bool disable_legacy_dialects; /* forbid vers=1.0 and vers=2.0 mounts */ extern atomic_t mid_count; diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c index 3bd71f982170..687754791bf0 100644 --- a/fs/smb/client/connect.c +++ b/fs/smb/client/connect.c @@ -2657,6 +2657,7 @@ cifs_get_tcon(struct cifs_ses *ses, struct smb3_fs_context *ctx) tcon->retry = ctx->retry; tcon->nocase = ctx->nocase; tcon->broken_sparse_sup = ctx->no_sparse; + tcon->max_cached_dirs = ctx->max_cached_dirs; if (ses->server->capabilities & SMB2_GLOBAL_CAP_DIRECTORY_LEASING) tcon->nohandlecache = ctx->nohandlecache; else diff --git a/fs/smb/client/fs_context.c b/fs/smb/client/fs_context.c index 67e16c2ac90e..e45ce31bbda7 100644 --- a/fs/smb/client/fs_context.c +++ b/fs/smb/client/fs_context.c @@ -150,6 +150,7 @@ const struct fs_parameter_spec smb3_fs_parameters[] = { fsparam_u32("closetimeo", Opt_closetimeo), fsparam_u32("echo_interval", Opt_echo_interval), fsparam_u32("max_credits", Opt_max_credits), + fsparam_u32("max_cached_dirs", Opt_max_cached_dirs), fsparam_u32("handletimeout", Opt_handletimeout), fsparam_u64("snapshot", Opt_snapshot), fsparam_u32("max_channels", Opt_max_channels), @@ -1165,6 +1166,14 @@ static int smb3_fs_context_parse_param(struct fs_context *fc, if (result.uint_32 > 1) ctx->multichannel = true; break; + case Opt_max_cached_dirs: + if (result.uint_32 < 1) { + cifs_errorf(fc, "%s: Invalid max_cached_dirs, needs to be 1 or more\n", + __func__); + goto cifs_parse_mount_err; + } + ctx->max_cached_dirs = result.uint_32; + break; case Opt_handletimeout: ctx->handle_timeout = result.uint_32; if (ctx->handle_timeout > SMB3_MAX_HANDLE_TIMEOUT) { @@ -1592,7 +1601,7 @@ int smb3_init_fs_context(struct fs_context *fc) ctx->acregmax = CIFS_DEF_ACTIMEO; ctx->acdirmax = CIFS_DEF_ACTIMEO; ctx->closetimeo = SMB3_DEF_DCLOSETIMEO; - + ctx->max_cached_dirs = MAX_CACHED_FIDS; /* Most clients set timeout to 0, allows server to use its default */ ctx->handle_timeout = 0; /* See MS-SMB2 spec section 2.2.14.2.12 */ diff --git a/fs/smb/client/fs_context.h b/fs/smb/client/fs_context.h index f4eaf8558902..9d8d34af0211 100644 --- a/fs/smb/client/fs_context.h +++ b/fs/smb/client/fs_context.h @@ -128,6 +128,7 @@ enum cifs_param { Opt_closetimeo, Opt_echo_interval, Opt_max_credits, + Opt_max_cached_dirs, Opt_snapshot, Opt_max_channels, Opt_handletimeout, @@ -261,6 +262,7 @@ struct smb3_fs_context { __u32 handle_timeout; /* persistent and durable handle timeout in ms */ unsigned int max_credits; /* smb3 max_credits 10 < credits < 60000 */ unsigned int max_channels; + unsigned int max_cached_dirs; __u16 compression; /* compression algorithm 0xFFFF default 0=disabled */ bool rootfs:1; /* if it's a SMB root file system */ bool witness:1; /* use witness protocol */ @@ -287,7 +289,7 @@ extern void smb3_update_mnt_flags(struct cifs_sb_info *cifs_sb); */ #define SMB3_MAX_DCLOSETIMEO (1 << 30) #define SMB3_DEF_DCLOSETIMEO (1 * HZ) /* even 1 sec enough to help eg open/write/close/open/read */ - +#define MAX_CACHED_FIDS 16 extern char *cifs_sanitize_prepath(char *prepath, gfp_t gfp); #endif diff --git a/fs/smb/client/fscache.c b/fs/smb/client/fscache.c index 3677525ee993..e5cad149f5a2 100644 --- a/fs/smb/client/fscache.c +++ b/fs/smb/client/fscache.c @@ -48,7 +48,7 @@ int cifs_fscache_get_super_cookie(struct cifs_tcon *tcon) sharename = extract_sharename(tcon->tree_name); if (IS_ERR(sharename)) { cifs_dbg(FYI, "%s: couldn't extract sharename\n", __func__); - return -EINVAL; + return PTR_ERR(sharename); } slen = strlen(sharename); diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c index e3dd698854d6..d9eda2e958b4 100644 --- a/fs/smb/client/smb2ops.c +++ b/fs/smb/client/smb2ops.c @@ -2683,6 +2683,7 @@ smb2_queryfs(const unsigned int xid, struct cifs_tcon *tcon, smb2_copy_fs_info_to_kstatfs(info, buf); qfs_exit: + trace_smb3_qfs_done(xid, tcon->tid, tcon->ses->Suid, tcon->tree_name, rc); free_rsp_buf(buftype, rsp_iov.iov_base); return rc; } diff --git a/fs/smb/client/trace.h b/fs/smb/client/trace.h index e671bd16f00c..a7e4755bed0f 100644 --- a/fs/smb/client/trace.h +++ b/fs/smb/client/trace.h @@ -691,7 +691,7 @@ DEFINE_EVENT(smb3_tcon_class, smb3_##name, \ TP_ARGS(xid, tid, sesid, unc_name, rc)) DEFINE_SMB3_TCON_EVENT(tcon); - +DEFINE_SMB3_TCON_EVENT(qfs_done); /* * For smb2/smb3 open (including create and mkdir) calls diff --git a/fs/smb/common/smb2pdu.h b/fs/smb/common/smb2pdu.h index 2680251b9aac..319fb9ffc6a0 100644 --- a/fs/smb/common/smb2pdu.h +++ b/fs/smb/common/smb2pdu.h @@ -406,7 +406,7 @@ struct smb2_tree_disconnect_rsp { /* Capabilities flags */ #define SMB2_GLOBAL_CAP_DFS 0x00000001 #define SMB2_GLOBAL_CAP_LEASING 0x00000002 /* Resp only New to SMB2.1 */ -#define SMB2_GLOBAL_CAP_LARGE_MTU 0X00000004 /* Resp only New to SMB2.1 */ +#define SMB2_GLOBAL_CAP_LARGE_MTU 0x00000004 /* Resp only New to SMB2.1 */ #define SMB2_GLOBAL_CAP_MULTI_CHANNEL 0x00000008 /* New to SMB3 */ #define SMB2_GLOBAL_CAP_PERSISTENT_HANDLES 0x00000010 /* New to SMB3 */ #define SMB2_GLOBAL_CAP_DIRECTORY_LEASING 0x00000020 /* New to SMB3 */ diff --git a/fs/smb/server/Kconfig b/fs/smb/server/Kconfig index 793151ddd60e..cabe6a843c6a 100644 --- a/fs/smb/server/Kconfig +++ b/fs/smb/server/Kconfig @@ -1,5 +1,5 @@ config SMB_SERVER - tristate "SMB3 server support (EXPERIMENTAL)" + tristate "SMB3 server support" depends on INET depends on MULTIUSER depends on FILE_LOCKING diff --git a/fs/smb/server/server.c b/fs/smb/server/server.c index 801cd0929209..5ab2f52f9b35 100644 --- a/fs/smb/server/server.c +++ b/fs/smb/server/server.c @@ -590,8 +590,6 @@ static int __init ksmbd_server_init(void) if (ret) goto err_crypto_destroy; - pr_warn_once("The ksmbd server is experimental\n"); - return 0; err_crypto_destroy: diff --git a/fs/stat.c b/fs/stat.c index 136711ae72fb..6822ac77aec2 100644 --- a/fs/stat.c +++ b/fs/stat.c @@ -311,6 +311,23 @@ int vfs_fstatat(int dfd, const char __user *filename, int statx_flags = flags | AT_NO_AUTOMOUNT; struct filename *name; + /* + * Work around glibc turning fstat() into fstatat(AT_EMPTY_PATH) + * + * If AT_EMPTY_PATH is set, we expect the common case to be that + * empty path, and avoid doing all the extra pathname work. + */ + if (dfd >= 0 && flags == AT_EMPTY_PATH) { + char c; + + ret = get_user(c, filename); + if (unlikely(ret)) + return ret; + + if (likely(!c)) + return vfs_fstat(dfd, stat); + } + name = getname_flags(filename, getname_statx_lookup_flags(statx_flags), NULL); ret = vfs_statx(dfd, name, statx_flags, stat, STATX_BASIC_STATS); putname(name); diff --git a/include/kvm/arm_pmu.h b/include/kvm/arm_pmu.h index 847da6fc2713..31029f4f7be8 100644 --- a/include/kvm/arm_pmu.h +++ b/include/kvm/arm_pmu.h @@ -12,7 +12,7 @@ #define ARMV8_PMU_CYCLE_IDX (ARMV8_PMU_MAX_COUNTERS - 1) -#ifdef CONFIG_HW_PERF_EVENTS +#if IS_ENABLED(CONFIG_HW_PERF_EVENTS) && IS_ENABLED(CONFIG_KVM) struct kvm_pmc { u8 idx; /* index into the pmu->pmc array */ @@ -74,6 +74,7 @@ int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu); struct kvm_pmu_events *kvm_get_pmu_events(void); void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu); void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu); +void kvm_vcpu_pmu_resync_el0(void); #define kvm_vcpu_has_pmu(vcpu) \ (test_bit(KVM_ARM_VCPU_PMU_V3, (vcpu)->arch.features)) @@ -171,6 +172,7 @@ static inline u8 kvm_arm_pmu_get_pmuver_limit(void) { return 0; } +static inline void kvm_vcpu_pmu_resync_el0(void) {} #endif diff --git a/include/linux/audit.h b/include/linux/audit.h index 6a3a9e122bb5..51b1b7054a23 100644 --- a/include/linux/audit.h +++ b/include/linux/audit.h @@ -117,6 +117,8 @@ enum audit_nfcfgop { AUDIT_NFT_OP_OBJ_RESET, AUDIT_NFT_OP_FLOWTABLE_REGISTER, AUDIT_NFT_OP_FLOWTABLE_UNREGISTER, + AUDIT_NFT_OP_SETELEM_RESET, + AUDIT_NFT_OP_RULE_RESET, AUDIT_NFT_OP_INVALID, }; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 12596af59c00..024e8b28c34b 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -438,7 +438,7 @@ static inline void bpf_long_memcpy(void *dst, const void *src, u32 size) size /= sizeof(long); while (size--) - *ldst++ = *lsrc++; + data_race(*ldst++ = *lsrc++); } /* copy everything but bpf_spin_lock, bpf_timer, and kptrs. There could be one of each. */ diff --git a/include/linux/ceph/ceph_fs.h b/include/linux/ceph/ceph_fs.h index 49586ff26152..5f2301ee88bc 100644 --- a/include/linux/ceph/ceph_fs.h +++ b/include/linux/ceph/ceph_fs.h @@ -359,14 +359,19 @@ enum { extern const char *ceph_mds_op_name(int op); - -#define CEPH_SETATTR_MODE 1 -#define CEPH_SETATTR_UID 2 -#define CEPH_SETATTR_GID 4 -#define CEPH_SETATTR_MTIME 8 -#define CEPH_SETATTR_ATIME 16 -#define CEPH_SETATTR_SIZE 32 -#define CEPH_SETATTR_CTIME 64 +#define CEPH_SETATTR_MODE (1 << 0) +#define CEPH_SETATTR_UID (1 << 1) +#define CEPH_SETATTR_GID (1 << 2) +#define CEPH_SETATTR_MTIME (1 << 3) +#define CEPH_SETATTR_ATIME (1 << 4) +#define CEPH_SETATTR_SIZE (1 << 5) +#define CEPH_SETATTR_CTIME (1 << 6) +#define CEPH_SETATTR_MTIME_NOW (1 << 7) +#define CEPH_SETATTR_ATIME_NOW (1 << 8) +#define CEPH_SETATTR_BTIME (1 << 9) +#define CEPH_SETATTR_KILL_SGUID (1 << 10) +#define CEPH_SETATTR_FSCRYPT_AUTH (1 << 11) +#define CEPH_SETATTR_FSCRYPT_FILE (1 << 12) /* * Ceph setxattr request flags. @@ -462,24 +467,26 @@ union ceph_mds_request_args { } __attribute__ ((packed)); union ceph_mds_request_args_ext { - union ceph_mds_request_args old; - struct { - __le32 mode; - __le32 uid; - __le32 gid; - struct ceph_timespec mtime; - struct ceph_timespec atime; - __le64 size, old_size; /* old_size needed by truncate */ - __le32 mask; /* CEPH_SETATTR_* */ - struct ceph_timespec btime; - } __attribute__ ((packed)) setattr_ext; + union { + union ceph_mds_request_args old; + struct { + __le32 mode; + __le32 uid; + __le32 gid; + struct ceph_timespec mtime; + struct ceph_timespec atime; + __le64 size, old_size; /* old_size needed by truncate */ + __le32 mask; /* CEPH_SETATTR_* */ + struct ceph_timespec btime; + } __attribute__ ((packed)) setattr_ext; + }; }; #define CEPH_MDS_FLAG_REPLAY 1 /* this is a replayed op */ #define CEPH_MDS_FLAG_WANT_DENTRY 2 /* want dentry in reply */ #define CEPH_MDS_FLAG_ASYNC 4 /* request is asynchronous */ -struct ceph_mds_request_head_old { +struct ceph_mds_request_head_legacy { __le64 oldest_client_tid; __le32 mdsmap_epoch; /* on client */ __le32 flags; /* CEPH_MDS_FLAG_* */ @@ -492,9 +499,9 @@ struct ceph_mds_request_head_old { union ceph_mds_request_args args; } __attribute__ ((packed)); -#define CEPH_MDS_REQUEST_HEAD_VERSION 1 +#define CEPH_MDS_REQUEST_HEAD_VERSION 2 -struct ceph_mds_request_head { +struct ceph_mds_request_head_old { __le16 version; /* struct version */ __le64 oldest_client_tid; __le32 mdsmap_epoch; /* on client */ @@ -508,6 +515,23 @@ struct ceph_mds_request_head { union ceph_mds_request_args_ext args; } __attribute__ ((packed)); +struct ceph_mds_request_head { + __le16 version; /* struct version */ + __le64 oldest_client_tid; + __le32 mdsmap_epoch; /* on client */ + __le32 flags; /* CEPH_MDS_FLAG_* */ + __u8 num_retry, num_fwd; /* legacy count retry and fwd attempts */ + __le16 num_releases; /* # include cap/lease release records */ + __le32 op; /* mds op code */ + __le32 caller_uid, caller_gid; + __le64 ino; /* use this ino for openc, mkdir, mknod, + etc. (if replaying) */ + union ceph_mds_request_args_ext args; + + __le32 ext_num_retry; /* new count retry attempts */ + __le32 ext_num_fwd; /* new count fwd attempts */ +} __attribute__ ((packed)); + /* cap/lease release record */ struct ceph_mds_request_release { __le64 ino, cap_id; /* ino and unique cap id */ diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h index 99c1726be6ee..2eaaabbe98cb 100644 --- a/include/linux/ceph/messenger.h +++ b/include/linux/ceph/messenger.h @@ -17,6 +17,7 @@ struct ceph_msg; struct ceph_connection; +struct ceph_msg_data_cursor; /* * Ceph defines these callbacks for handling connection events. @@ -70,6 +71,30 @@ struct ceph_connection_operations { int used_proto, int result, const int *allowed_protos, int proto_cnt, const int *allowed_modes, int mode_cnt); + + /** + * sparse_read: read sparse data + * @con: connection we're reading from + * @cursor: data cursor for reading extents + * @buf: optional buffer to read into + * + * This should be called more than once, each time setting up to + * receive an extent into the current cursor position, and zeroing + * the holes between them. + * + * Returns amount of data to be read (in bytes), 0 if reading is + * complete, or -errno if there was an error. + * + * If @buf is set on a >0 return, then the data should be read into + * the provided buffer. Otherwise, it should be read into the cursor. + * + * The sparse read operation is expected to initialize the cursor + * with a length covering up to the end of the last extent. + */ + int (*sparse_read)(struct ceph_connection *con, + struct ceph_msg_data_cursor *cursor, + char **buf); + }; /* use format string %s%lld */ @@ -98,6 +123,7 @@ enum ceph_msg_data_type { CEPH_MSG_DATA_BIO, /* data source/destination is a bio list */ #endif /* CONFIG_BLOCK */ CEPH_MSG_DATA_BVECS, /* data source/destination is a bio_vec array */ + CEPH_MSG_DATA_ITER, /* data source/destination is an iov_iter */ }; #ifdef CONFIG_BLOCK @@ -199,6 +225,7 @@ struct ceph_msg_data { bool own_pages; }; struct ceph_pagelist *pagelist; + struct iov_iter iter; }; }; @@ -207,6 +234,7 @@ struct ceph_msg_data_cursor { struct ceph_msg_data *data; /* current data item */ size_t resid; /* bytes not yet consumed */ + int sr_resid; /* residual sparse_read len */ bool need_crc; /* crc update needed */ union { #ifdef CONFIG_BLOCK @@ -222,6 +250,10 @@ struct ceph_msg_data_cursor { struct page *page; /* page from list */ size_t offset; /* bytes from list */ }; + struct { + struct iov_iter iov_iter; + unsigned int lastlen; + }; }; }; @@ -251,6 +283,7 @@ struct ceph_msg { struct kref kref; bool more_to_follow; bool needs_out_seq; + bool sparse_read; int front_alloc_len; struct ceph_msgpool *pool; @@ -309,6 +342,10 @@ struct ceph_connection_v1_info { int in_base_pos; /* bytes read */ + /* sparse reads */ + struct kvec in_sr_kvec; /* current location to receive into */ + u64 in_sr_len; /* amount of data in this extent */ + /* message in temps */ u8 in_tag; /* protocol control byte */ struct ceph_msg_header in_hdr; @@ -395,6 +432,7 @@ struct ceph_connection_v2_info { void *conn_bufs[16]; int conn_buf_cnt; + int data_len_remain; struct kvec in_sign_kvecs[8]; struct kvec out_sign_kvecs[8]; @@ -573,6 +611,8 @@ void ceph_msg_data_add_bio(struct ceph_msg *msg, struct ceph_bio_iter *bio_pos, #endif /* CONFIG_BLOCK */ void ceph_msg_data_add_bvecs(struct ceph_msg *msg, struct ceph_bvec_iter *bvec_pos); +void ceph_msg_data_add_iter(struct ceph_msg *msg, + struct iov_iter *iter); struct ceph_msg *ceph_msg_new2(int type, int front_len, int max_data_items, gfp_t flags, bool can_fail); diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h index fb6be72104df..bf9823956758 100644 --- a/include/linux/ceph/osd_client.h +++ b/include/linux/ceph/osd_client.h @@ -29,14 +29,62 @@ typedef void (*ceph_osdc_callback_t)(struct ceph_osd_request *); #define CEPH_HOMELESS_OSD -1 -/* a given osd we're communicating with */ +/* + * A single extent in a SPARSE_READ reply. + * + * Note that these come from the OSD as little-endian values. On BE arches, + * we convert them in-place after receipt. + */ +struct ceph_sparse_extent { + u64 off; + u64 len; +} __packed; + +/* Sparse read state machine state values */ +enum ceph_sparse_read_state { + CEPH_SPARSE_READ_HDR = 0, + CEPH_SPARSE_READ_EXTENTS, + CEPH_SPARSE_READ_DATA_LEN, + CEPH_SPARSE_READ_DATA, +}; + +/* + * A SPARSE_READ reply is a 32-bit count of extents, followed by an array of + * 64-bit offset/length pairs, and then all of the actual file data + * concatenated after it (sans holes). + * + * Unfortunately, we don't know how long the extent array is until we've + * started reading the data section of the reply. The caller should send down + * a destination buffer for the array, but we'll alloc one if it's too small + * or if the caller doesn't. + */ +struct ceph_sparse_read { + enum ceph_sparse_read_state sr_state; /* state machine state */ + u64 sr_req_off; /* orig request offset */ + u64 sr_req_len; /* orig request length */ + u64 sr_pos; /* current pos in buffer */ + int sr_index; /* current extent index */ + __le32 sr_datalen; /* length of actual data */ + u32 sr_count; /* extent count in reply */ + int sr_ext_len; /* length of extent array */ + struct ceph_sparse_extent *sr_extent; /* extent array */ +}; + +/* + * A given osd we're communicating with. + * + * Note that the o_requests tree can be searched while holding the "lock" mutex + * or the "o_requests_lock" spinlock. Insertion or removal requires both! + */ struct ceph_osd { refcount_t o_ref; + int o_sparse_op_idx; struct ceph_osd_client *o_osdc; int o_osd; int o_incarnation; struct rb_node o_node; struct ceph_connection o_con; + spinlock_t o_requests_lock; struct rb_root o_requests; struct rb_root o_linger_requests; struct rb_root o_backoff_mappings; @@ -46,6 +94,7 @@ struct ceph_osd { unsigned long lru_ttl; struct list_head o_keepalive_item; struct mutex lock; + struct ceph_sparse_read o_sparse_read; }; #define CEPH_OSD_SLAB_OPS 2 @@ -59,6 +108,7 @@ enum ceph_osd_data_type { CEPH_OSD_DATA_TYPE_BIO, #endif /* CONFIG_BLOCK */ CEPH_OSD_DATA_TYPE_BVECS, + CEPH_OSD_DATA_TYPE_ITER, }; struct ceph_osd_data { @@ -82,6 +132,7 @@ struct ceph_osd_data { struct ceph_bvec_iter bvec_pos; u32 num_bvecs; }; + struct iov_iter iter; }; }; @@ -98,6 +149,8 @@ struct ceph_osd_req_op { u64 offset, length; u64 truncate_size; u32 truncate_seq; + int sparse_ext_cnt; + struct ceph_sparse_extent *sparse_ext; struct ceph_osd_data osd_data; } extent; struct { @@ -145,6 +198,9 @@ struct ceph_osd_req_op { u32 src_fadvise_flags; struct ceph_osd_data osd_data; } copy_from; + struct { + u64 ver; + } assert_ver; }; }; @@ -199,6 +255,7 @@ struct ceph_osd_request { struct ceph_osd_client *r_osdc; struct kref r_kref; bool r_mempool; + bool r_linger; /* don't resend on failure */ struct completion r_completion; /* private to osd_client.c */ ceph_osdc_callback_t r_callback; @@ -211,9 +268,9 @@ struct ceph_osd_request { struct ceph_snap_context *r_snapc; /* for writes */ struct timespec64 r_mtime; /* ditto */ u64 r_data_offset; /* ditto */ - bool r_linger; /* don't resend on failure */ /* internal */ + u64 r_version; /* data version sent in reply */ unsigned long r_stamp; /* jiffies, send or check time */ unsigned long r_start_stamp; /* jiffies */ ktime_t r_start_latency; /* ktime_t */ @@ -450,6 +507,8 @@ void osd_req_op_extent_osd_data_bvecs(struct ceph_osd_request *osd_req, void osd_req_op_extent_osd_data_bvec_pos(struct ceph_osd_request *osd_req, unsigned int which, struct ceph_bvec_iter *bvec_pos); +void osd_req_op_extent_osd_iter(struct ceph_osd_request *osd_req, + unsigned int which, struct iov_iter *iter); extern void osd_req_op_cls_request_data_pagelist(struct ceph_osd_request *, unsigned int which, @@ -504,6 +563,20 @@ extern struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *, u32 truncate_seq, u64 truncate_size, bool use_mempool); +int __ceph_alloc_sparse_ext_map(struct ceph_osd_req_op *op, int cnt); + +/* + * How big an extent array should we preallocate for a sparse read? This is + * just a starting value. If we get more than this back from the OSD, the + * receiver will reallocate. + */ +#define CEPH_SPARSE_EXT_ARRAY_INITIAL 16 + +static inline int ceph_alloc_sparse_ext_map(struct ceph_osd_req_op *op) +{ + return __ceph_alloc_sparse_ext_map(op, CEPH_SPARSE_EXT_ARRAY_INITIAL); +} + extern void ceph_osdc_get_request(struct ceph_osd_request *req); extern void ceph_osdc_put_request(struct ceph_osd_request *req); @@ -558,5 +631,19 @@ int ceph_osdc_list_watchers(struct ceph_osd_client *osdc, struct ceph_object_locator *oloc, struct ceph_watch_item **watchers, u32 *num_watchers); -#endif +/* Find offset into the buffer of the end of the extent map */ +static inline u64 ceph_sparse_ext_map_end(struct ceph_osd_req_op *op) +{ + struct ceph_sparse_extent *ext; + + /* No extents? No data */ + if (op->extent.sparse_ext_cnt == 0) + return 0; + + ext = &op->extent.sparse_ext[op->extent.sparse_ext_cnt - 1]; + + return ext->off + ext->len - op->extent.offset; +} + +#endif diff --git a/include/linux/ceph/rados.h b/include/linux/ceph/rados.h index 43a7a1573b51..73c3efbec36c 100644 --- a/include/linux/ceph/rados.h +++ b/include/linux/ceph/rados.h @@ -523,6 +523,10 @@ struct ceph_osd_op { struct { __le64 cookie; } __attribute__ ((packed)) notify; + struct { + __le64 unused; + __le64 ver; + } __attribute__ ((packed)) assert_ver; struct { __le64 offset, length; __le64 src_offset; diff --git a/include/linux/gameport.h b/include/linux/gameport.h index 0a221e768ea4..07e370113b2b 100644 --- a/include/linux/gameport.h +++ b/include/linux/gameport.h @@ -63,7 +63,7 @@ struct gameport_driver { int gameport_open(struct gameport *gameport, struct gameport_driver *drv, int mode); void gameport_close(struct gameport *gameport); -#if defined(CONFIG_GAMEPORT) || (defined(MODULE) && defined(CONFIG_GAMEPORT_MODULE)) +#if IS_REACHABLE(CONFIG_GAMEPORT) void __gameport_register_port(struct gameport *gameport, struct module *owner); /* use a define to avoid include chaining to get THIS_MODULE */ diff --git a/include/linux/ipv6.h b/include/linux/ipv6.h index 1cc11d475516..8faff387a760 100644 --- a/include/linux/ipv6.h +++ b/include/linux/ipv6.h @@ -148,6 +148,7 @@ struct inet6_skb_parm { #define IP6SKB_JUMBOGRAM 128 #define IP6SKB_SEG6 256 #define IP6SKB_FAKEJUMBO 512 +#define IP6SKB_MULTIPATH 1024 }; #if defined(CONFIG_NET_L3_MASTER_DEV) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 819b6bc8ac08..3df5499f7936 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -54,11 +54,13 @@ extern p4d_t kasan_early_shadow_p4d[MAX_PTRS_PER_P4D]; int kasan_populate_early_shadow(const void *shadow_start, const void *shadow_end); +#ifndef __HAVE_ARCH_SHADOW_MAP static inline void *kasan_mem_to_shadow(const void *addr) { return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT) + KASAN_SHADOW_OFFSET; } +#endif int kasan_add_zero_shadow(void *start, unsigned long size); void kasan_remove_zero_shadow(void *start, unsigned long size); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 9d3ac7720da9..fb6c6109fdca 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -190,8 +190,6 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req, bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req); bool kvm_make_all_cpus_request_except(struct kvm *kvm, unsigned int req, struct kvm_vcpu *except); -bool kvm_make_cpus_request_mask(struct kvm *kvm, unsigned int req, - unsigned long *vcpu_bitmap); #define KVM_USERSPACE_IRQ_SOURCE_ID 0 #define KVM_IRQFD_RESAMPLE_IRQ_SOURCE_ID 1 @@ -256,11 +254,15 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #endif #ifdef KVM_ARCH_WANT_MMU_NOTIFIER +union kvm_mmu_notifier_arg { + pte_t pte; +}; + struct kvm_gfn_range { struct kvm_memory_slot *slot; gfn_t start; gfn_t end; - pte_t pte; + union kvm_mmu_notifier_arg arg; bool may_block; }; bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); @@ -865,6 +867,25 @@ static inline void kvm_vm_bugged(struct kvm *kvm) unlikely(__ret); \ }) +/* + * Note, "data corruption" refers to corruption of host kernel data structures, + * not guest data. Guest data corruption, suspected or confirmed, that is tied + * and contained to a single VM should *never* BUG() and potentially panic the + * host, i.e. use this variant of KVM_BUG() if and only if a KVM data structure + * is corrupted and that corruption can have a cascading effect to other parts + * of the hosts and/or to other VMs. + */ +#define KVM_BUG_ON_DATA_CORRUPTION(cond, kvm) \ +({ \ + bool __ret = !!(cond); \ + \ + if (IS_ENABLED(CONFIG_BUG_ON_DATA_CORRUPTION)) \ + BUG_ON(__ret); \ + else if (WARN_ON_ONCE(__ret && !(kvm)->vm_bugged)) \ + kvm_vm_bugged(kvm); \ + unlikely(__ret); \ +}) + static inline void kvm_vcpu_srcu_read_lock(struct kvm_vcpu *vcpu) { #ifdef CONFIG_PROVE_RCU @@ -1359,6 +1380,9 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target); void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu, bool yield_to_kernel_mode); void kvm_flush_remote_tlbs(struct kvm *kvm); +void kvm_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages); +void kvm_flush_remote_tlbs_memslot(struct kvm *kvm, + const struct kvm_memory_slot *memslot); #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE int kvm_mmu_topup_memory_cache(struct kvm_mmu_memory_cache *mc, int min); @@ -1387,10 +1411,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, unsigned long mask); void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot); -#ifdef CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT -void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, - const struct kvm_memory_slot *memslot); -#else /* !CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ +#ifndef CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log); int kvm_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log, int *is_dirty, struct kvm_memory_slot **memslot); @@ -1479,11 +1500,23 @@ static inline void kvm_arch_free_vm(struct kvm *kvm) } #endif -#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLB -static inline int kvm_arch_flush_remote_tlb(struct kvm *kvm) +#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS +static inline int kvm_arch_flush_remote_tlbs(struct kvm *kvm) { return -ENOTSUPP; } +#else +int kvm_arch_flush_remote_tlbs(struct kvm *kvm); +#endif + +#ifndef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE +static inline int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, + gfn_t gfn, u64 nr_pages) +{ + return -EOPNOTSUPP; +} +#else +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t gfn, u64 nr_pages); #endif #ifdef __KVM_HAVE_ARCH_NONCOHERENT_DMA @@ -2148,8 +2181,6 @@ struct kvm_device_ops { int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma); }; -void kvm_device_get(struct kvm_device *dev); -void kvm_device_put(struct kvm_device *dev); struct kvm_device *kvm_device_from_filp(struct file *filp); int kvm_register_device_ops(const struct kvm_device_ops *ops, u32 type); void kvm_unregister_device_ops(u32 type); diff --git a/include/linux/micrel_phy.h b/include/linux/micrel_phy.h index 8bef1ab62bba..4e27ca7c49de 100644 --- a/include/linux/micrel_phy.h +++ b/include/linux/micrel_phy.h @@ -41,9 +41,10 @@ #define PHY_ID_KSZ9477 0x00221631 /* struct phy_device dev_flags definitions */ -#define MICREL_PHY_50MHZ_CLK 0x00000001 -#define MICREL_PHY_FXEN 0x00000002 -#define MICREL_KSZ8_P1_ERRATA 0x00000003 +#define MICREL_PHY_50MHZ_CLK BIT(0) +#define MICREL_PHY_FXEN BIT(1) +#define MICREL_KSZ8_P1_ERRATA BIT(2) +#define MICREL_NO_EEE BIT(3) #define MICREL_KSZ9021_EXTREG_CTRL 0xB #define MICREL_KSZ9021_EXTREG_DATA_WRITE 0xC diff --git a/include/linux/nvme-fc-driver.h b/include/linux/nvme-fc-driver.h index 4109f1bd6128..f6ef8cf5d774 100644 --- a/include/linux/nvme-fc-driver.h +++ b/include/linux/nvme-fc-driver.h @@ -53,10 +53,10 @@ struct nvmefc_ls_req { void *rqstaddr; dma_addr_t rqstdma; - u32 rqstlen; + __le32 rqstlen; void *rspaddr; dma_addr_t rspdma; - u32 rsplen; + __le32 rsplen; u32 timeout; void *private; @@ -120,7 +120,7 @@ struct nvmefc_ls_req { struct nvmefc_ls_rsp { void *rspbuf; dma_addr_t rspdma; - u16 rsplen; + __le32 rsplen; void (*done)(struct nvmefc_ls_rsp *rsp); void *nvme_fc_private; /* LLDD is not to access !! */ diff --git a/include/linux/of.h b/include/linux/of.h index ed679819c279..6a9ddf20e79a 100644 --- a/include/linux/of.h +++ b/include/linux/of.h @@ -1676,8 +1676,8 @@ int of_overlay_notifier_unregister(struct notifier_block *nb); #else -static inline int of_overlay_fdt_apply(void *overlay_fdt, u32 overlay_fdt_size, - int *ovcs_id) +static inline int of_overlay_fdt_apply(const void *overlay_fdt, u32 overlay_fdt_size, + int *ovcs_id, struct device_node *target_base) { return -ENOTSUPP; } diff --git a/include/linux/oid_registry.h b/include/linux/oid_registry.h index 0f4a8903922a..f86a08ba0207 100644 --- a/include/linux/oid_registry.h +++ b/include/linux/oid_registry.h @@ -67,6 +67,7 @@ enum OID { OID_msOutlookExpress, /* 1.3.6.1.4.1.311.16.4 */ OID_ntlmssp, /* 1.3.6.1.4.1.311.2.2.10 */ + OID_negoex, /* 1.3.6.1.4.1.311.2.2.30 */ OID_spnego, /* 1.3.6.1.5.5.2 */ diff --git a/include/linux/phylink.h b/include/linux/phylink.h index 7d07f8736431..2b886ea654bb 100644 --- a/include/linux/phylink.h +++ b/include/linux/phylink.h @@ -600,7 +600,7 @@ void pcs_get_state(struct phylink_pcs *pcs, * * The %neg_mode argument should be tested via the phylink_mode_*() family of * functions, or for PCS that set pcs->neg_mode true, should be tested - * against the %PHYLINK_PCS_NEG_* definitions. + * against the PHYLINK_PCS_NEG_* definitions. */ int pcs_config(struct phylink_pcs *pcs, unsigned int neg_mode, phy_interface_t interface, const unsigned long *advertising, @@ -630,7 +630,7 @@ void pcs_an_restart(struct phylink_pcs *pcs); * * The %mode argument should be tested via the phylink_mode_*() family of * functions, or for PCS that set pcs->neg_mode true, should be tested - * against the %PHYLINK_PCS_NEG_* definitions. + * against the PHYLINK_PCS_NEG_* definitions. */ void pcs_link_up(struct phylink_pcs *pcs, unsigned int neg_mode, phy_interface_t interface, int speed, int duplex); diff --git a/include/linux/platform_data/rtc-ds2404.h b/include/linux/platform_data/rtc-ds2404.h deleted file mode 100644 index 22c53825528f..000000000000 --- a/include/linux/platform_data/rtc-ds2404.h +++ /dev/null @@ -1,20 +0,0 @@ -/* - * ds2404.h - platform data structure for the DS2404 RTC. - * - * This file is subject to the terms and conditions of the GNU General Public - * License. See the file "COPYING" in the main directory of this archive - * for more details. - * - * Copyright (C) 2012 Sven Schnelle - */ - -#ifndef __LINUX_DS2404_H -#define __LINUX_DS2404_H - -struct ds2404_platform_data { - - unsigned int gpio_rst; - unsigned int gpio_clk; - unsigned int gpio_dq; -}; -#endif diff --git a/include/linux/pwm.h b/include/linux/pwm.h index 04ae1d9073a7..d2f9f690a9c1 100644 --- a/include/linux/pwm.h +++ b/include/linux/pwm.h @@ -298,7 +298,7 @@ struct pwm_chip { int base; unsigned int npwm; - struct pwm_device * (*of_xlate)(struct pwm_chip *pc, + struct pwm_device * (*of_xlate)(struct pwm_chip *chip, const struct of_phandle_args *args); unsigned int of_pwm_n_cells; @@ -395,9 +395,9 @@ struct pwm_device *pwm_request_from_chip(struct pwm_chip *chip, unsigned int index, const char *label); -struct pwm_device *of_pwm_xlate_with_flags(struct pwm_chip *pc, +struct pwm_device *of_pwm_xlate_with_flags(struct pwm_chip *chip, const struct of_phandle_args *args); -struct pwm_device *of_pwm_single_xlate(struct pwm_chip *pc, +struct pwm_device *of_pwm_single_xlate(struct pwm_chip *chip, const struct of_phandle_args *args); struct pwm_device *pwm_get(struct device *dev, const char *con_id); diff --git a/include/linux/raid/pq.h b/include/linux/raid/pq.h index f29aaaf2eb21..006e18decfad 100644 --- a/include/linux/raid/pq.h +++ b/include/linux/raid/pq.h @@ -108,6 +108,8 @@ extern const struct raid6_calls raid6_vpermxor1; extern const struct raid6_calls raid6_vpermxor2; extern const struct raid6_calls raid6_vpermxor4; extern const struct raid6_calls raid6_vpermxor8; +extern const struct raid6_calls raid6_lsx; +extern const struct raid6_calls raid6_lasx; struct raid6_recov_calls { void (*data2)(int, size_t, int, int, void **); @@ -123,6 +125,8 @@ extern const struct raid6_recov_calls raid6_recov_avx2; extern const struct raid6_recov_calls raid6_recov_avx512; extern const struct raid6_recov_calls raid6_recov_s390xc; extern const struct raid6_recov_calls raid6_recov_neon; +extern const struct raid6_recov_calls raid6_recov_lsx; +extern const struct raid6_recov_calls raid6_recov_lasx; extern const struct raid6_calls raid6_neonx1; extern const struct raid6_calls raid6_neonx2; diff --git a/include/linux/rtc.h b/include/linux/rtc.h index 1fd9c6a21ebe..4c0bcbeb1f00 100644 --- a/include/linux/rtc.h +++ b/include/linux/rtc.h @@ -146,6 +146,7 @@ struct rtc_device { time64_t range_min; timeu64_t range_max; + timeu64_t alarm_offset_max; time64_t start_secs; time64_t offset_secs; bool set_start_time; diff --git a/include/linux/tca6416_keypad.h b/include/linux/tca6416_keypad.h index b0d36a9934cc..5cf6f6f82aa7 100644 --- a/include/linux/tca6416_keypad.h +++ b/include/linux/tca6416_keypad.h @@ -25,7 +25,6 @@ struct tca6416_keys_platform_data { unsigned int rep:1; /* enable input subsystem auto repeat */ uint16_t pinmask; uint16_t invert; - int irq_is_gpio; int use_polling; /* use polling if Interrupt is not connected*/ }; #endif diff --git a/include/linux/thermal.h b/include/linux/thermal.h index eb17495c8acc..c99440aac1a1 100644 --- a/include/linux/thermal.h +++ b/include/linux/thermal.h @@ -300,16 +300,22 @@ int thermal_acpi_critical_trip_temp(struct acpi_device *adev, int *ret_temp); #endif #ifdef CONFIG_THERMAL -struct thermal_zone_device *thermal_zone_device_register(const char *, int, int, - void *, struct thermal_zone_device_ops *, - const struct thermal_zone_params *, int, int); +struct thermal_zone_device *thermal_zone_device_register_with_trips( + const char *type, + struct thermal_trip *trips, + int num_trips, int mask, + void *devdata, + struct thermal_zone_device_ops *ops, + const struct thermal_zone_params *tzp, + int passive_delay, int polling_delay); -void thermal_zone_device_unregister(struct thermal_zone_device *); +struct thermal_zone_device *thermal_tripless_zone_device_register( + const char *type, + void *devdata, + struct thermal_zone_device_ops *ops, + const struct thermal_zone_params *tzp); -struct thermal_zone_device * -thermal_zone_device_register_with_trips(const char *, struct thermal_trip *, int, int, - void *, struct thermal_zone_device_ops *, - const struct thermal_zone_params *, int, int); +void thermal_zone_device_unregister(struct thermal_zone_device *tz); void *thermal_zone_device_priv(struct thermal_zone_device *tzd); const char *thermal_zone_device_type(struct thermal_zone_device *tzd); @@ -350,15 +356,26 @@ int thermal_zone_device_enable(struct thermal_zone_device *tz); int thermal_zone_device_disable(struct thermal_zone_device *tz); void thermal_zone_device_critical(struct thermal_zone_device *tz); #else -static inline struct thermal_zone_device *thermal_zone_device_register( - const char *type, int trips, int mask, void *devdata, - struct thermal_zone_device_ops *ops, - const struct thermal_zone_params *tzp, - int passive_delay, int polling_delay) +static inline struct thermal_zone_device *thermal_zone_device_register_with_trips( + const char *type, + struct thermal_trip *trips, + int num_trips, int mask, + void *devdata, + struct thermal_zone_device_ops *ops, + const struct thermal_zone_params *tzp, + int passive_delay, int polling_delay) { return ERR_PTR(-ENODEV); } -static inline void thermal_zone_device_unregister( - struct thermal_zone_device *tz) + +static inline struct thermal_zone_device *thermal_tripless_zone_device_register( + const char *type, + void *devdata, + struct thermal_zone_device_ops *ops, + const struct thermal_zone_params *tzp) +{ return ERR_PTR(-ENODEV); } + +static inline void thermal_zone_device_unregister(struct thermal_zone_device *tz) { } + static inline struct thermal_cooling_device * thermal_cooling_device_register(const char *type, void *devdata, const struct thermal_cooling_device_ops *ops) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index 741703b45f61..cb571dfcf4b1 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -856,6 +856,9 @@ static inline int __must_check xa_insert_irq(struct xarray *xa, * stores the index into the @id pointer, then stores the entry at * that index. A concurrent lookup will not see an uninitialised @id. * + * Must only be operated on an xarray initialized with flag XA_FLAGS_ALLOC set + * in xa_init_flags(). + * * Context: Any context. Takes and releases the xa_lock. May sleep if * the @gfp flags permit. * Return: 0 on success, -ENOMEM if memory could not be allocated or @@ -886,6 +889,9 @@ static inline __must_check int xa_alloc(struct xarray *xa, u32 *id, * stores the index into the @id pointer, then stores the entry at * that index. A concurrent lookup will not see an uninitialised @id. * + * Must only be operated on an xarray initialized with flag XA_FLAGS_ALLOC set + * in xa_init_flags(). + * * Context: Any context. Takes and releases the xa_lock while * disabling softirqs. May sleep if the @gfp flags permit. * Return: 0 on success, -ENOMEM if memory could not be allocated or @@ -916,6 +922,9 @@ static inline int __must_check xa_alloc_bh(struct xarray *xa, u32 *id, * stores the index into the @id pointer, then stores the entry at * that index. A concurrent lookup will not see an uninitialised @id. * + * Must only be operated on an xarray initialized with flag XA_FLAGS_ALLOC set + * in xa_init_flags(). + * * Context: Process context. Takes and releases the xa_lock while * disabling interrupts. May sleep if the @gfp flags permit. * Return: 0 on success, -ENOMEM if memory could not be allocated or @@ -949,6 +958,9 @@ static inline int __must_check xa_alloc_irq(struct xarray *xa, u32 *id, * The search for an empty entry will start at @next and will wrap * around if necessary. * + * Must only be operated on an xarray initialized with flag XA_FLAGS_ALLOC set + * in xa_init_flags(). + * * Context: Any context. Takes and releases the xa_lock. May sleep if * the @gfp flags permit. * Return: 0 if the allocation succeeded without wrapping. 1 if the @@ -983,6 +995,9 @@ static inline int xa_alloc_cyclic(struct xarray *xa, u32 *id, void *entry, * The search for an empty entry will start at @next and will wrap * around if necessary. * + * Must only be operated on an xarray initialized with flag XA_FLAGS_ALLOC set + * in xa_init_flags(). + * * Context: Any context. Takes and releases the xa_lock while * disabling softirqs. May sleep if the @gfp flags permit. * Return: 0 if the allocation succeeded without wrapping. 1 if the @@ -1017,6 +1032,9 @@ static inline int xa_alloc_cyclic_bh(struct xarray *xa, u32 *id, void *entry, * The search for an empty entry will start at @next and will wrap * around if necessary. * + * Must only be operated on an xarray initialized with flag XA_FLAGS_ALLOC set + * in xa_init_flags(). + * * Context: Process context. Takes and releases the xa_lock while * disabling interrupts. May sleep if the @gfp flags permit. * Return: 0 if the allocation succeeded without wrapping. 1 if the diff --git a/include/net/ip.h b/include/net/ip.h index 19adacd5ece0..3489a1cca5e7 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -57,6 +57,7 @@ struct inet_skb_parm { #define IPSKB_FRAG_PMTU BIT(6) #define IPSKB_L3SLAVE BIT(7) #define IPSKB_NOPOLICY BIT(8) +#define IPSKB_MULTIPATH BIT(9) u16 frag_max_size; }; @@ -94,7 +95,7 @@ static inline void ipcm_init_sk(struct ipcm_cookie *ipcm, ipcm_init(ipcm); ipcm->sockc.mark = READ_ONCE(inet->sk.sk_mark); - ipcm->sockc.tsflags = inet->sk.sk_tsflags; + ipcm->sockc.tsflags = READ_ONCE(inet->sk.sk_tsflags); ipcm->oif = READ_ONCE(inet->sk.sk_bound_dev_if); ipcm->addr = inet->inet_saddr; ipcm->protocol = inet->inet_num; diff --git a/include/net/ip6_fib.h b/include/net/ip6_fib.h index c9ff23cf313e..1ba9f4ddf2f6 100644 --- a/include/net/ip6_fib.h +++ b/include/net/ip6_fib.h @@ -642,7 +642,10 @@ static inline bool fib6_rules_early_flow_dissect(struct net *net, if (!net->ipv6.fib6_rules_require_fldissect) return false; - skb_flow_dissect_flow_keys(skb, flkeys, flag); + memset(flkeys, 0, sizeof(*flkeys)); + __skb_flow_dissect(net, skb, &flow_keys_dissector, + flkeys, NULL, 0, 0, 0, flag); + fl6->fl6_sport = flkeys->ports.src; fl6->fl6_dport = flkeys->ports.dst; fl6->flowi6_proto = flkeys->basic.ip_proto; diff --git a/include/net/ip_fib.h b/include/net/ip_fib.h index a378eff827c7..f0c13864180e 100644 --- a/include/net/ip_fib.h +++ b/include/net/ip_fib.h @@ -418,7 +418,10 @@ static inline bool fib4_rules_early_flow_dissect(struct net *net, if (!net->ipv4.fib_rules_require_fldissect) return false; - skb_flow_dissect_flow_keys(skb, flkeys, flag); + memset(flkeys, 0, sizeof(*flkeys)); + __skb_flow_dissect(net, skb, &flow_keys_dissector, + flkeys, NULL, 0, 0, 0, flag); + fl4->fl4_sport = flkeys->ports.src; fl4->fl4_dport = flkeys->ports.dst; fl4->flowi4_proto = flkeys->basic.ip_proto; diff --git a/include/net/ip_tunnels.h b/include/net/ip_tunnels.h index e8750b4ef7e1..f346b4efbc30 100644 --- a/include/net/ip_tunnels.h +++ b/include/net/ip_tunnels.h @@ -483,15 +483,14 @@ static inline void iptunnel_xmit_stats(struct net_device *dev, int pkt_len) u64_stats_inc(&tstats->tx_packets); u64_stats_update_end(&tstats->syncp); put_cpu_ptr(tstats); - } else { - struct net_device_stats *err_stats = &dev->stats; + return; + } - if (pkt_len < 0) { - err_stats->tx_errors++; - err_stats->tx_aborted_errors++; - } else { - err_stats->tx_dropped++; - } + if (pkt_len < 0) { + DEV_STATS_INC(dev, tx_errors); + DEV_STATS_INC(dev, tx_aborted_errors); + } else { + DEV_STATS_INC(dev, tx_dropped); } } diff --git a/include/net/scm.h b/include/net/scm.h index c5bcdf65f55c..e8c76b4be2fe 100644 --- a/include/net/scm.h +++ b/include/net/scm.h @@ -9,6 +9,7 @@ #include #include #include +#include /* Well, we should have at least one descriptor open * to accept passed FDs 8) @@ -123,14 +124,17 @@ static inline bool scm_has_secdata(struct socket *sock) static __inline__ void scm_pidfd_recv(struct msghdr *msg, struct scm_cookie *scm) { struct file *pidfd_file = NULL; - int pidfd; + int len, pidfd; - /* - * put_cmsg() doesn't return an error if CMSG is truncated, + /* put_cmsg() doesn't return an error if CMSG is truncated, * that's why we need to opencode these checks here. */ - if ((msg->msg_controllen <= sizeof(struct cmsghdr)) || - (msg->msg_controllen - sizeof(struct cmsghdr)) < sizeof(int)) { + if (msg->msg_flags & MSG_CMSG_COMPAT) + len = sizeof(struct compat_cmsghdr) + sizeof(int); + else + len = sizeof(struct cmsghdr) + sizeof(int); + + if (msg->msg_controllen < len) { msg->msg_flags |= MSG_CTRUNC; return; } diff --git a/include/net/sock.h b/include/net/sock.h index 11d503417591..b770261fbdaf 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1053,6 +1053,12 @@ static inline void sk_wmem_queued_add(struct sock *sk, int val) WRITE_ONCE(sk->sk_wmem_queued, sk->sk_wmem_queued + val); } +static inline void sk_forward_alloc_add(struct sock *sk, int val) +{ + /* Paired with lockless reads of sk->sk_forward_alloc */ + WRITE_ONCE(sk->sk_forward_alloc, sk->sk_forward_alloc + val); +} + void sk_stream_write_space(struct sock *sk); /* OOB backlog add */ @@ -1377,7 +1383,7 @@ static inline int sk_forward_alloc_get(const struct sock *sk) if (sk->sk_prot->forward_alloc_get) return sk->sk_prot->forward_alloc_get(sk); #endif - return sk->sk_forward_alloc; + return READ_ONCE(sk->sk_forward_alloc); } static inline bool __sk_stream_memory_free(const struct sock *sk, int wake) @@ -1673,14 +1679,14 @@ static inline void sk_mem_charge(struct sock *sk, int size) { if (!sk_has_account(sk)) return; - sk->sk_forward_alloc -= size; + sk_forward_alloc_add(sk, -size); } static inline void sk_mem_uncharge(struct sock *sk, int size) { if (!sk_has_account(sk)) return; - sk->sk_forward_alloc += size; + sk_forward_alloc_add(sk, size); sk_mem_reclaim(sk); } @@ -1900,7 +1906,9 @@ struct sockcm_cookie { static inline void sockcm_init(struct sockcm_cookie *sockc, const struct sock *sk) { - *sockc = (struct sockcm_cookie) { .tsflags = sk->sk_tsflags }; + *sockc = (struct sockcm_cookie) { + .tsflags = READ_ONCE(sk->sk_tsflags) + }; } int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, @@ -2695,9 +2703,9 @@ void __sock_recv_wifi_status(struct msghdr *msg, struct sock *sk, static inline void sock_recv_timestamp(struct msghdr *msg, struct sock *sk, struct sk_buff *skb) { - ktime_t kt = skb->tstamp; struct skb_shared_hwtstamps *hwtstamps = skb_hwtstamps(skb); - + u32 tsflags = READ_ONCE(sk->sk_tsflags); + ktime_t kt = skb->tstamp; /* * generate control messages if * - receive time stamping in software requested @@ -2705,10 +2713,10 @@ sock_recv_timestamp(struct msghdr *msg, struct sock *sk, struct sk_buff *skb) * - hardware time stamps available and wanted */ if (sock_flag(sk, SOCK_RCVTSTAMP) || - (sk->sk_tsflags & SOF_TIMESTAMPING_RX_SOFTWARE) || - (kt && sk->sk_tsflags & SOF_TIMESTAMPING_SOFTWARE) || + (tsflags & SOF_TIMESTAMPING_RX_SOFTWARE) || + (kt && tsflags & SOF_TIMESTAMPING_SOFTWARE) || (hwtstamps->hwtstamp && - (sk->sk_tsflags & SOF_TIMESTAMPING_RAW_HARDWARE))) + (tsflags & SOF_TIMESTAMPING_RAW_HARDWARE))) __sock_recv_timestamp(msg, sk, skb); else sock_write_timestamp(sk, kt); @@ -2730,7 +2738,8 @@ static inline void sock_recv_cmsgs(struct msghdr *msg, struct sock *sk, #define TSFLAGS_ANY (SOF_TIMESTAMPING_SOFTWARE | \ SOF_TIMESTAMPING_RAW_HARDWARE) - if (sk->sk_flags & FLAGS_RECV_CMSGS || sk->sk_tsflags & TSFLAGS_ANY) + if (sk->sk_flags & FLAGS_RECV_CMSGS || + READ_ONCE(sk->sk_tsflags) & TSFLAGS_ANY) __sock_recv_cmsgs(msg, sk, skb); else if (unlikely(sock_flag(sk, SOCK_TIMESTAMP))) sock_write_timestamp(sk, skb->tstamp); diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h index a2b8d30c4c80..49f768d0ff37 100644 --- a/include/scsi/scsi_host.h +++ b/include/scsi/scsi_host.h @@ -764,7 +764,7 @@ scsi_template_proc_dir(const struct scsi_host_template *sht); #define scsi_template_proc_dir(sht) NULL #endif extern void scsi_scan_host(struct Scsi_Host *); -extern void scsi_rescan_device(struct device *); +extern void scsi_rescan_device(struct scsi_device *); extern void scsi_remove_host(struct Scsi_Host *); extern struct Scsi_Host *scsi_host_get(struct Scsi_Host *); extern int scsi_host_busy(struct Scsi_Host *shost); diff --git a/include/sound/dmaengine_pcm.h b/include/sound/dmaengine_pcm.h index c9a8bce9a785..d70c55f17df7 100644 --- a/include/sound/dmaengine_pcm.h +++ b/include/sound/dmaengine_pcm.h @@ -142,7 +142,7 @@ struct snd_dmaengine_pcm_config { struct snd_pcm_substream *substream); int (*process)(struct snd_pcm_substream *substream, int channel, unsigned long hwoff, - struct iov_iter *buf, unsigned long bytes); + unsigned long bytes); dma_filter_fn compat_filter_fn; struct device *dma_dev; const char *chan_names[SNDRV_PCM_STREAM_LAST + 1]; diff --git a/include/sound/soc-component.h b/include/sound/soc-component.h index 17bea3144551..ceca69b46a82 100644 --- a/include/sound/soc-component.h +++ b/include/sound/soc-component.h @@ -139,7 +139,7 @@ struct snd_soc_component_driver { struct snd_pcm_audio_tstamp_report *audio_tstamp_report); int (*copy)(struct snd_soc_component *component, struct snd_pcm_substream *substream, int channel, - unsigned long pos, struct iov_iter *buf, + unsigned long pos, struct iov_iter *iter, unsigned long bytes); struct page *(*page)(struct snd_soc_component *component, struct snd_pcm_substream *substream, @@ -511,7 +511,7 @@ int snd_soc_pcm_component_ioctl(struct snd_pcm_substream *substream, int snd_soc_pcm_component_sync_stop(struct snd_pcm_substream *substream); int snd_soc_pcm_component_copy(struct snd_pcm_substream *substream, int channel, unsigned long pos, - struct iov_iter *buf, unsigned long bytes); + struct iov_iter *iter, unsigned long bytes); struct page *snd_soc_pcm_component_page(struct snd_pcm_substream *substream, unsigned long offset); int snd_soc_pcm_component_mmap(struct snd_pcm_substream *substream, diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index ee0bcff14b69..9b731976ce2f 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -445,6 +445,8 @@ typedef struct elf64_shdr { #define NT_MIPS_DSP 0x800 /* MIPS DSP ASE registers */ #define NT_MIPS_FP_MODE 0x801 /* MIPS floating-point mode */ #define NT_MIPS_MSA 0x802 /* MIPS SIMD registers */ +#define NT_RISCV_CSR 0x900 /* RISC-V Control and Status Registers */ +#define NT_RISCV_VECTOR 0x901 /* RISC-V vector registers */ #define NT_LOONGARCH_CPUCFG 0xa00 /* LoongArch CPU config registers */ #define NT_LOONGARCH_CSR 0xa01 /* LoongArch control and status registers */ #define NT_LOONGARCH_LSX 0xa02 /* LoongArch Loongson SIMD Extension registers */ diff --git a/include/uapi/linux/netfilter/nf_tables.h b/include/uapi/linux/netfilter/nf_tables.h index 8466c2a9938f..ca30232b7bc8 100644 --- a/include/uapi/linux/netfilter/nf_tables.h +++ b/include/uapi/linux/netfilter/nf_tables.h @@ -263,6 +263,7 @@ enum nft_chain_attributes { * @NFTA_RULE_USERDATA: user data (NLA_BINARY, NFT_USERDATA_MAXLEN) * @NFTA_RULE_ID: uniquely identifies a rule in a transaction (NLA_U32) * @NFTA_RULE_POSITION_ID: transaction unique identifier of the previous rule (NLA_U32) + * @NFTA_RULE_CHAIN_ID: add the rule to chain by ID, alternative to @NFTA_RULE_CHAIN (NLA_U32) */ enum nft_rule_attributes { NFTA_RULE_UNSPEC, diff --git a/include/uapi/scsi/scsi_bsg_ufs.h b/include/uapi/scsi/scsi_bsg_ufs.h index 7c7975f9905e..03f2beadf201 100644 --- a/include/uapi/scsi/scsi_bsg_ufs.h +++ b/include/uapi/scsi/scsi_bsg_ufs.h @@ -83,7 +83,7 @@ struct utp_upiu_header { union { __u8 tm_function; __u8 query_function; - }; + } __attribute__((packed)); __u8 response; __u8 status; __u8 ehs_length; diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index 300455b4bc12..c53678875416 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -93,6 +93,8 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *f) struct io_uring_sqe *sqe; unsigned int sq_idx; + if (ctx->flags & IORING_SETUP_NO_SQARRAY) + break; sq_idx = READ_ONCE(ctx->sq_array[entry & sq_mask]); if (sq_idx > sq_mask) continue; diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 62f345587df5..1ecc8c748768 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -174,6 +174,16 @@ static void io_worker_ref_put(struct io_wq *wq) complete(&wq->worker_done); } +bool io_wq_worker_stopped(void) +{ + struct io_worker *worker = current->worker_private; + + if (WARN_ON_ONCE(!io_wq_current_is_worker())) + return true; + + return test_bit(IO_WQ_BIT_EXIT, &worker->wq->state); +} + static void io_worker_cancel_cb(struct io_worker *worker) { struct io_wq_acct *acct = io_wq_get_acct(worker); diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 06d9ca90c577..2b2a6406dd8e 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -52,6 +52,7 @@ void io_wq_hash_work(struct io_wq_work *work, void *val); int io_wq_cpu_affinity(struct io_uring_task *tctx, cpumask_var_t mask); int io_wq_max_workers(struct io_wq *wq, int *new_count); +bool io_wq_worker_stopped(void); static inline bool io_wq_is_hashed(struct io_wq_work *work) { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 140d9650f637..8a50c5682c98 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -150,6 +150,31 @@ static void io_queue_sqe(struct io_kiocb *req); struct kmem_cache *req_cachep; +static int __read_mostly sysctl_io_uring_disabled; +static int __read_mostly sysctl_io_uring_group = -1; + +#ifdef CONFIG_SYSCTL +static struct ctl_table kernel_io_uring_disabled_table[] = { + { + .procname = "io_uring_disabled", + .data = &sysctl_io_uring_disabled, + .maxlen = sizeof(sysctl_io_uring_disabled), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_TWO, + }, + { + .procname = "io_uring_group", + .data = &sysctl_io_uring_group, + .maxlen = sizeof(gid_t), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + {}, +}; +#endif + struct sock *io_uring_get_socket(struct file *file) { #if defined(CONFIG_UNIX) @@ -883,7 +908,7 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) struct io_uring_cqe *cqe = &ctx->completion_cqes[i]; if (!io_fill_cqe_aux(ctx, cqe->user_data, cqe->res, cqe->flags)) { - if (ctx->task_complete) { + if (ctx->lockless_cq) { spin_lock(&ctx->completion_lock); io_cqring_event_overflow(ctx, cqe->user_data, cqe->res, cqe->flags, 0, 0); @@ -1541,7 +1566,7 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) if (!(req->flags & REQ_F_CQE_SKIP) && unlikely(!io_fill_cqe_req(ctx, req))) { - if (ctx->task_complete) { + if (ctx->lockless_cq) { spin_lock(&ctx->completion_lock); io_req_cqe_overflow(req); spin_unlock(&ctx->completion_lock); @@ -1985,6 +2010,8 @@ fail: if (!needs_poll) { if (!(req->ctx->flags & IORING_SETUP_IOPOLL)) break; + if (io_wq_worker_stopped()) + break; cond_resched(); continue; } @@ -4073,9 +4100,30 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params) return io_uring_create(entries, &p, params); } +static inline bool io_uring_allowed(void) +{ + int disabled = READ_ONCE(sysctl_io_uring_disabled); + kgid_t io_uring_group; + + if (disabled == 2) + return false; + + if (disabled == 0 || capable(CAP_SYS_ADMIN)) + return true; + + io_uring_group = make_kgid(&init_user_ns, sysctl_io_uring_group); + if (!gid_valid(io_uring_group)) + return false; + + return in_group_p(io_uring_group); +} + SYSCALL_DEFINE2(io_uring_setup, u32, entries, struct io_uring_params __user *, params) { + if (!io_uring_allowed()) + return -EPERM; + return io_uring_setup(entries, params); } @@ -4669,6 +4717,10 @@ static int __init io_uring_init(void) offsetof(struct io_kiocb, cmd.data), sizeof_field(struct io_kiocb, cmd.data), NULL); +#ifdef CONFIG_SYSCTL + register_sysctl_init("kernel", kernel_io_uring_disabled_table); +#endif + return 0; }; __initcall(io_uring_init); diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c index ee2d2c687fda..bd6c2c7959a5 100644 --- a/io_uring/sqpoll.c +++ b/io_uring/sqpoll.c @@ -430,7 +430,9 @@ __cold int io_sqpoll_wq_cpu_affinity(struct io_ring_ctx *ctx, if (sqd) { io_sq_thread_park(sqd); - ret = io_wq_cpu_affinity(sqd->thread->io_uring, mask); + /* Don't set affinity for a dying thread */ + if (sqd->thread) + ret = io_wq_cpu_affinity(sqd->thread->io_uring, mask); io_sq_thread_unpark(sqd); } diff --git a/kernel/auditsc.c b/kernel/auditsc.c index b0cb7631e48b..21d2fa815e78 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -143,6 +143,8 @@ static const struct audit_nfcfgop_tab audit_nfcfgs[] = { { AUDIT_NFT_OP_OBJ_RESET, "nft_reset_obj" }, { AUDIT_NFT_OP_FLOWTABLE_REGISTER, "nft_register_flowtable" }, { AUDIT_NFT_OP_FLOWTABLE_UNREGISTER, "nft_unregister_flowtable" }, + { AUDIT_NFT_OP_SETELEM_RESET, "nft_reset_setelem" }, + { AUDIT_NFT_OP_RULE_RESET, "nft_reset_rule" }, { AUDIT_NFT_OP_INVALID, "nft_invalid" }, }; diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index b5149cfce7d4..146824cc9689 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -553,7 +553,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, void *value, u64 map_flags, gfp_t gfp_flags) { struct bpf_local_storage_data *old_sdata = NULL; - struct bpf_local_storage_elem *selem = NULL; + struct bpf_local_storage_elem *alloc_selem, *selem = NULL; struct bpf_local_storage *local_storage; unsigned long flags; int err; @@ -607,11 +607,12 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, } } - if (gfp_flags == GFP_KERNEL) { - selem = bpf_selem_alloc(smap, owner, value, true, gfp_flags); - if (!selem) - return ERR_PTR(-ENOMEM); - } + /* A lookup has just been done before and concluded a new selem is + * needed. The chance of an unnecessary alloc is unlikely. + */ + alloc_selem = selem = bpf_selem_alloc(smap, owner, value, true, gfp_flags); + if (!alloc_selem) + return ERR_PTR(-ENOMEM); raw_spin_lock_irqsave(&local_storage->lock, flags); @@ -623,13 +624,13 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, * simple. */ err = -EAGAIN; - goto unlock_err; + goto unlock; } old_sdata = bpf_local_storage_lookup(local_storage, smap, false); err = check_flags(old_sdata, map_flags); if (err) - goto unlock_err; + goto unlock; if (old_sdata && (map_flags & BPF_F_LOCK)) { copy_map_value_locked(&smap->map, old_sdata->data, value, @@ -638,23 +639,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, goto unlock; } - if (gfp_flags != GFP_KERNEL) { - /* local_storage->lock is held. Hence, we are sure - * we can unlink and uncharge the old_sdata successfully - * later. Hence, instead of charging the new selem now - * and then uncharge the old selem later (which may cause - * a potential but unnecessary charge failure), avoid taking - * a charge at all here (the "!old_sdata" check) and the - * old_sdata will not be uncharged later during - * bpf_selem_unlink_storage_nolock(). - */ - selem = bpf_selem_alloc(smap, owner, value, !old_sdata, gfp_flags); - if (!selem) { - err = -ENOMEM; - goto unlock_err; - } - } - + alloc_selem = NULL; /* First, link the new selem to the map */ bpf_selem_link_map(smap, selem); @@ -665,20 +650,16 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, if (old_sdata) { bpf_selem_unlink_map(SELEM(old_sdata)); bpf_selem_unlink_storage_nolock(local_storage, SELEM(old_sdata), - false, false); + true, false); } unlock: raw_spin_unlock_irqrestore(&local_storage->lock, flags); - return SDATA(selem); - -unlock_err: - raw_spin_unlock_irqrestore(&local_storage->lock, flags); - if (selem) { + if (alloc_selem) { mem_uncharge(smap, owner, smap->elem_size); - bpf_selem_free(selem, smap, true); + bpf_selem_free(alloc_selem, smap, true); } - return ERR_PTR(err); + return err ? ERR_PTR(err) : SDATA(selem); } static u16 bpf_local_storage_cache_idx_get(struct bpf_local_storage_cache *cache) @@ -779,7 +760,7 @@ void bpf_local_storage_destroy(struct bpf_local_storage *local_storage) * of the loop will set the free_cgroup_storage to true. */ free_storage = bpf_selem_unlink_storage_nolock( - local_storage, selem, false, true); + local_storage, selem, true, true); } raw_spin_unlock_irqrestore(&local_storage->lock, flags); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 0f8f036d8bd1..4e3ce0542e31 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -870,7 +870,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins GFP_KERNEL); if (!pack) return NULL; - pack->ptr = module_alloc(BPF_PROG_PACK_SIZE); + pack->ptr = bpf_jit_alloc_exec(BPF_PROG_PACK_SIZE); if (!pack->ptr) { kfree(pack); return NULL; @@ -894,7 +894,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns) mutex_lock(&pack_mutex); if (size > BPF_PROG_PACK_SIZE) { size = round_up(size, PAGE_SIZE); - ptr = module_alloc(size); + ptr = bpf_jit_alloc_exec(size); if (ptr) { bpf_fill_ill_insns(ptr, size); set_vm_flush_reset_perms(ptr); @@ -932,7 +932,7 @@ void bpf_prog_pack_free(struct bpf_binary_header *hdr) mutex_lock(&pack_mutex); if (hdr->size > BPF_PROG_PACK_SIZE) { - module_memfree(hdr); + bpf_jit_free_exec(hdr); goto out; } @@ -956,7 +956,7 @@ void bpf_prog_pack_free(struct bpf_binary_header *hdr) if (bitmap_find_next_zero_area(pack->bitmap, BPF_PROG_CHUNK_COUNT, 0, BPF_PROG_CHUNK_COUNT, 0) == 0) { list_del(&pack->list); - module_memfree(pack->ptr); + bpf_jit_free_exec(pack->ptr); kfree(pack); } out: diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 14d38c8aa636..2fa2df861e92 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -5506,9 +5506,9 @@ int kern_sys_bpf(int cmd, union bpf_attr *attr, unsigned int size) } run_ctx.bpf_cookie = 0; - run_ctx.saved_run_ctx = NULL; if (!__bpf_prog_enter_sleepable_recur(prog, &run_ctx)) { /* recursion detected */ + __bpf_prog_exit_sleepable_recur(prog, 0, &run_ctx); bpf_prog_put(prog); return -EBUSY; } diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 78acf28d4873..53ff50cac61e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -926,13 +926,12 @@ u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog, migrate_disable(); might_fault(); + run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx); + if (unlikely(this_cpu_inc_return(*(prog->active)) != 1)) { bpf_prog_inc_misses_counter(prog); return 0; } - - run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx); - return bpf_prog_start_time(); } diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index 4c1e9a3c0ab6..f488997b0717 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -160,7 +160,7 @@ if DMA_CMA config DMA_NUMA_CMA bool "Enable separate DMA Contiguous Memory Area for NUMA Node" - default NUMA + depends on NUMA help Enable this option to get numa CMA areas so that NUMA devices can get local memory by DMA coherent APIs. diff --git a/kernel/dma/debug.c b/kernel/dma/debug.c index f011409eccb5..49d40d4a3aa3 100644 --- a/kernel/dma/debug.c +++ b/kernel/dma/debug.c @@ -637,15 +637,19 @@ static struct dma_debug_entry *__dma_entry_alloc(void) return entry; } -static void __dma_entry_alloc_check_leak(void) +/* + * This should be called outside of free_entries_lock scope to avoid potential + * deadlocks with serial consoles that use DMA. + */ +static void __dma_entry_alloc_check_leak(u32 nr_entries) { - u32 tmp = nr_total_entries % nr_prealloc_entries; + u32 tmp = nr_entries % nr_prealloc_entries; /* Shout each time we tick over some multiple of the initial pool */ if (tmp < DMA_DEBUG_DYNAMIC_ENTRIES) { pr_info("dma_debug_entry pool grown to %u (%u00%%)\n", - nr_total_entries, - (nr_total_entries / nr_prealloc_entries)); + nr_entries, + (nr_entries / nr_prealloc_entries)); } } @@ -656,8 +660,10 @@ static void __dma_entry_alloc_check_leak(void) */ static struct dma_debug_entry *dma_entry_alloc(void) { + bool alloc_check_leak = false; struct dma_debug_entry *entry; unsigned long flags; + u32 nr_entries; spin_lock_irqsave(&free_entries_lock, flags); if (num_free_entries == 0) { @@ -667,13 +673,17 @@ static struct dma_debug_entry *dma_entry_alloc(void) pr_err("debugging out of memory - disabling\n"); return NULL; } - __dma_entry_alloc_check_leak(); + alloc_check_leak = true; + nr_entries = nr_total_entries; } entry = __dma_entry_alloc(); spin_unlock_irqrestore(&free_entries_lock, flags); + if (alloc_check_leak) + __dma_entry_alloc_check_leak(nr_entries); + #ifdef CONFIG_STACKTRACE entry->stack_len = stack_trace_save(entry->stack_entries, ARRAY_SIZE(entry->stack_entries), diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index 1acec2e22827..b481c48a31a6 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -135,9 +135,9 @@ encrypt_mapping: remove_mapping: #ifdef CONFIG_DMA_DIRECT_REMAP dma_common_free_remap(addr, pool_size); -#endif -free_page: __maybe_unused +free_page: __free_pages(page, order); +#endif out: return ret; } diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index 47fd5b6f968e..18ea2398bf56 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -540,14 +540,12 @@ char *log_buf_addr_get(void) { return log_buf; } -EXPORT_SYMBOL_GPL(log_buf_addr_get); /* Return log buffer size */ u32 log_buf_len_get(void) { return log_buf_len; } -EXPORT_SYMBOL_GPL(log_buf_len_get); /* * Define how much of the log buffer we could take at maximum. The value diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 319cfbeb0738..fa307f93fa2e 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2237,6 +2237,17 @@ config TEST_DIV64 If unsure, say N. +config TEST_IOV_ITER + tristate "Test iov_iter operation" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Enable this to turn on testing of the operation of the I/O iterator + (iov_iter). This test is executed only once during system boot (so + affects only boot time), or at module load time. + + If unsure, say N. + config KPROBES_SANITY_TEST tristate "Kprobes sanity tests" if !KUNIT_ALL_TESTS depends on DEBUG_KERNEL diff --git a/lib/Makefile b/lib/Makefile index 2e08397f6210..740109b6e2c8 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -64,6 +64,7 @@ obj-$(CONFIG_TEST_BITOPS) += test_bitops.o CFLAGS_test_bitops.o += -Werror obj-$(CONFIG_CPUMASK_KUNIT_TEST) += cpumask_kunit.o obj-$(CONFIG_TEST_SYSCTL) += test_sysctl.o +obj-$(CONFIG_TEST_IOV_ITER) += kunit_iov_iter.o obj-$(CONFIG_HASH_KUNIT_TEST) += test_hash.o obj-$(CONFIG_TEST_IDA) += test_ida.o obj-$(CONFIG_TEST_UBSAN) += test_ubsan.o diff --git a/lib/idr.c b/lib/idr.c index 7ecdfdb5309e..13f2758c2377 100644 --- a/lib/idr.c +++ b/lib/idr.c @@ -100,7 +100,7 @@ EXPORT_SYMBOL_GPL(idr_alloc); * @end: The maximum ID (exclusive). * @gfp: Memory allocation flags. * - * Allocates an unused ID in the range specified by @nextid and @end. If + * Allocates an unused ID in the range specified by @start and @end. If * @end is <= 0, it is treated as one larger than %INT_MAX. This allows * callers to use @start + N as @end as long as N is within integer range. * The search for an unused ID will start at the last ID allocated and will diff --git a/lib/iov_iter.c b/lib/iov_iter.c index b31597b0ca20..27234a820eeb 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1654,14 +1654,14 @@ static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i, size_t *offset0) { struct page **p, *page; - size_t skip = i->iov_offset, offset; + size_t skip = i->iov_offset, offset, size; int k; for (;;) { if (i->nr_segs == 0) return 0; - maxsize = min(maxsize, i->bvec->bv_len - skip); - if (maxsize) + size = min(maxsize, i->bvec->bv_len - skip); + if (size) break; i->iov_offset = 0; i->nr_segs--; @@ -1674,16 +1674,16 @@ static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i, offset = skip % PAGE_SIZE; *offset0 = offset; - maxpages = want_pages_array(pages, maxsize, offset, maxpages); + maxpages = want_pages_array(pages, size, offset, maxpages); if (!maxpages) return -ENOMEM; p = *pages; for (k = 0; k < maxpages; k++) p[k] = page + k; - maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); - iov_iter_advance(i, maxsize); - return maxsize; + size = min_t(size_t, size, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, size); + return size; } /* @@ -1698,14 +1698,14 @@ static ssize_t iov_iter_extract_kvec_pages(struct iov_iter *i, { struct page **p, *page; const void *kaddr; - size_t skip = i->iov_offset, offset, len; + size_t skip = i->iov_offset, offset, len, size; int k; for (;;) { if (i->nr_segs == 0) return 0; - maxsize = min(maxsize, i->kvec->iov_len - skip); - if (maxsize) + size = min(maxsize, i->kvec->iov_len - skip); + if (size) break; i->iov_offset = 0; i->nr_segs--; @@ -1717,13 +1717,13 @@ static ssize_t iov_iter_extract_kvec_pages(struct iov_iter *i, offset = (unsigned long)kaddr & ~PAGE_MASK; *offset0 = offset; - maxpages = want_pages_array(pages, maxsize, offset, maxpages); + maxpages = want_pages_array(pages, size, offset, maxpages); if (!maxpages) return -ENOMEM; p = *pages; kaddr -= offset; - len = offset + maxsize; + len = offset + size; for (k = 0; k < maxpages; k++) { size_t seg = min_t(size_t, len, PAGE_SIZE); @@ -1737,9 +1737,9 @@ static ssize_t iov_iter_extract_kvec_pages(struct iov_iter *i, kaddr += PAGE_SIZE; } - maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); - iov_iter_advance(i, maxsize); - return maxsize; + size = min_t(size_t, size, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, size); + return size; } /* diff --git a/lib/kunit_iov_iter.c b/lib/kunit_iov_iter.c new file mode 100644 index 000000000000..859b67c4d697 --- /dev/null +++ b/lib/kunit_iov_iter.c @@ -0,0 +1,777 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* I/O iterator tests. This can only test kernel-backed iterator types. + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include + +MODULE_DESCRIPTION("iov_iter testing"); +MODULE_AUTHOR("David Howells "); +MODULE_LICENSE("GPL"); + +struct kvec_test_range { + int from, to; +}; + +static const struct kvec_test_range kvec_test_ranges[] = { + { 0x00002, 0x00002 }, + { 0x00027, 0x03000 }, + { 0x05193, 0x18794 }, + { 0x20000, 0x20000 }, + { 0x20000, 0x24000 }, + { 0x24000, 0x27001 }, + { 0x29000, 0xffffb }, + { 0xffffd, 0xffffe }, + { -1 } +}; + +static inline u8 pattern(unsigned long x) +{ + return x & 0xff; +} + +static void iov_kunit_unmap(void *data) +{ + vunmap(data); +} + +static void *__init iov_kunit_create_buffer(struct kunit *test, + struct page ***ppages, + size_t npages) +{ + struct page **pages; + unsigned long got; + void *buffer; + + pages = kunit_kcalloc(test, npages, sizeof(struct page *), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pages); + *ppages = pages; + + got = alloc_pages_bulk_array(GFP_KERNEL, npages, pages); + if (got != npages) { + release_pages(pages, got); + KUNIT_ASSERT_EQ(test, got, npages); + } + + buffer = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buffer); + + kunit_add_action_or_reset(test, iov_kunit_unmap, buffer); + return buffer; +} + +static void __init iov_kunit_load_kvec(struct kunit *test, + struct iov_iter *iter, int dir, + struct kvec *kvec, unsigned int kvmax, + void *buffer, size_t bufsize, + const struct kvec_test_range *pr) +{ + size_t size = 0; + int i; + + for (i = 0; i < kvmax; i++, pr++) { + if (pr->from < 0) + break; + KUNIT_ASSERT_GE(test, pr->to, pr->from); + KUNIT_ASSERT_LE(test, pr->to, bufsize); + kvec[i].iov_base = buffer + pr->from; + kvec[i].iov_len = pr->to - pr->from; + size += pr->to - pr->from; + } + KUNIT_ASSERT_LE(test, size, bufsize); + + iov_iter_kvec(iter, dir, kvec, i, size); +} + +/* + * Test copying to a ITER_KVEC-type iterator. + */ +static void __init iov_kunit_copy_to_kvec(struct kunit *test) +{ + const struct kvec_test_range *pr; + struct iov_iter iter; + struct page **spages, **bpages; + struct kvec kvec[8]; + u8 *scratch, *buffer; + size_t bufsize, npages, size, copied; + int i, patt; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + scratch = iov_kunit_create_buffer(test, &spages, npages); + for (i = 0; i < bufsize; i++) + scratch[i] = pattern(i); + + buffer = iov_kunit_create_buffer(test, &bpages, npages); + memset(buffer, 0, bufsize); + + iov_kunit_load_kvec(test, &iter, READ, kvec, ARRAY_SIZE(kvec), + buffer, bufsize, kvec_test_ranges); + size = iter.count; + + copied = copy_to_iter(scratch, size, &iter); + + KUNIT_EXPECT_EQ(test, copied, size); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_EXPECT_EQ(test, iter.nr_segs, 0); + + /* Build the expected image in the scratch buffer. */ + patt = 0; + memset(scratch, 0, bufsize); + for (pr = kvec_test_ranges; pr->from >= 0; pr++) + for (i = pr->from; i < pr->to; i++) + scratch[i] = pattern(patt++); + + /* Compare the images */ + for (i = 0; i < bufsize; i++) { + KUNIT_EXPECT_EQ_MSG(test, buffer[i], scratch[i], "at i=%x", i); + if (buffer[i] != scratch[i]) + return; + } + + KUNIT_SUCCEED(); +} + +/* + * Test copying from a ITER_KVEC-type iterator. + */ +static void __init iov_kunit_copy_from_kvec(struct kunit *test) +{ + const struct kvec_test_range *pr; + struct iov_iter iter; + struct page **spages, **bpages; + struct kvec kvec[8]; + u8 *scratch, *buffer; + size_t bufsize, npages, size, copied; + int i, j; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + buffer = iov_kunit_create_buffer(test, &bpages, npages); + for (i = 0; i < bufsize; i++) + buffer[i] = pattern(i); + + scratch = iov_kunit_create_buffer(test, &spages, npages); + memset(scratch, 0, bufsize); + + iov_kunit_load_kvec(test, &iter, WRITE, kvec, ARRAY_SIZE(kvec), + buffer, bufsize, kvec_test_ranges); + size = min(iter.count, bufsize); + + copied = copy_from_iter(scratch, size, &iter); + + KUNIT_EXPECT_EQ(test, copied, size); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_EXPECT_EQ(test, iter.nr_segs, 0); + + /* Build the expected image in the main buffer. */ + i = 0; + memset(buffer, 0, bufsize); + for (pr = kvec_test_ranges; pr->from >= 0; pr++) { + for (j = pr->from; j < pr->to; j++) { + buffer[i++] = pattern(j); + if (i >= bufsize) + goto stop; + } + } +stop: + + /* Compare the images */ + for (i = 0; i < bufsize; i++) { + KUNIT_EXPECT_EQ_MSG(test, scratch[i], buffer[i], "at i=%x", i); + if (scratch[i] != buffer[i]) + return; + } + + KUNIT_SUCCEED(); +} + +struct bvec_test_range { + int page, from, to; +}; + +static const struct bvec_test_range bvec_test_ranges[] = { + { 0, 0x0002, 0x0002 }, + { 1, 0x0027, 0x0893 }, + { 2, 0x0193, 0x0794 }, + { 3, 0x0000, 0x1000 }, + { 4, 0x0000, 0x1000 }, + { 5, 0x0000, 0x1000 }, + { 6, 0x0000, 0x0ffb }, + { 6, 0x0ffd, 0x0ffe }, + { -1, -1, -1 } +}; + +static void __init iov_kunit_load_bvec(struct kunit *test, + struct iov_iter *iter, int dir, + struct bio_vec *bvec, unsigned int bvmax, + struct page **pages, size_t npages, + size_t bufsize, + const struct bvec_test_range *pr) +{ + struct page *can_merge = NULL, *page; + size_t size = 0; + int i; + + for (i = 0; i < bvmax; i++, pr++) { + if (pr->from < 0) + break; + KUNIT_ASSERT_LT(test, pr->page, npages); + KUNIT_ASSERT_LT(test, pr->page * PAGE_SIZE, bufsize); + KUNIT_ASSERT_GE(test, pr->from, 0); + KUNIT_ASSERT_GE(test, pr->to, pr->from); + KUNIT_ASSERT_LE(test, pr->to, PAGE_SIZE); + + page = pages[pr->page]; + if (pr->from == 0 && pr->from != pr->to && page == can_merge) { + i--; + bvec[i].bv_len += pr->to; + } else { + bvec_set_page(&bvec[i], page, pr->to - pr->from, pr->from); + } + + size += pr->to - pr->from; + if ((pr->to & ~PAGE_MASK) == 0) + can_merge = page + pr->to / PAGE_SIZE; + else + can_merge = NULL; + } + + iov_iter_bvec(iter, dir, bvec, i, size); +} + +/* + * Test copying to a ITER_BVEC-type iterator. + */ +static void __init iov_kunit_copy_to_bvec(struct kunit *test) +{ + const struct bvec_test_range *pr; + struct iov_iter iter; + struct bio_vec bvec[8]; + struct page **spages, **bpages; + u8 *scratch, *buffer; + size_t bufsize, npages, size, copied; + int i, b, patt; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + scratch = iov_kunit_create_buffer(test, &spages, npages); + for (i = 0; i < bufsize; i++) + scratch[i] = pattern(i); + + buffer = iov_kunit_create_buffer(test, &bpages, npages); + memset(buffer, 0, bufsize); + + iov_kunit_load_bvec(test, &iter, READ, bvec, ARRAY_SIZE(bvec), + bpages, npages, bufsize, bvec_test_ranges); + size = iter.count; + + copied = copy_to_iter(scratch, size, &iter); + + KUNIT_EXPECT_EQ(test, copied, size); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_EXPECT_EQ(test, iter.nr_segs, 0); + + /* Build the expected image in the scratch buffer. */ + b = 0; + patt = 0; + memset(scratch, 0, bufsize); + for (pr = bvec_test_ranges; pr->from >= 0; pr++, b++) { + u8 *p = scratch + pr->page * PAGE_SIZE; + + for (i = pr->from; i < pr->to; i++) + p[i] = pattern(patt++); + } + + /* Compare the images */ + for (i = 0; i < bufsize; i++) { + KUNIT_EXPECT_EQ_MSG(test, buffer[i], scratch[i], "at i=%x", i); + if (buffer[i] != scratch[i]) + return; + } + + KUNIT_SUCCEED(); +} + +/* + * Test copying from a ITER_BVEC-type iterator. + */ +static void __init iov_kunit_copy_from_bvec(struct kunit *test) +{ + const struct bvec_test_range *pr; + struct iov_iter iter; + struct bio_vec bvec[8]; + struct page **spages, **bpages; + u8 *scratch, *buffer; + size_t bufsize, npages, size, copied; + int i, j; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + buffer = iov_kunit_create_buffer(test, &bpages, npages); + for (i = 0; i < bufsize; i++) + buffer[i] = pattern(i); + + scratch = iov_kunit_create_buffer(test, &spages, npages); + memset(scratch, 0, bufsize); + + iov_kunit_load_bvec(test, &iter, WRITE, bvec, ARRAY_SIZE(bvec), + bpages, npages, bufsize, bvec_test_ranges); + size = iter.count; + + copied = copy_from_iter(scratch, size, &iter); + + KUNIT_EXPECT_EQ(test, copied, size); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_EXPECT_EQ(test, iter.nr_segs, 0); + + /* Build the expected image in the main buffer. */ + i = 0; + memset(buffer, 0, bufsize); + for (pr = bvec_test_ranges; pr->from >= 0; pr++) { + size_t patt = pr->page * PAGE_SIZE; + + for (j = pr->from; j < pr->to; j++) { + buffer[i++] = pattern(patt + j); + if (i >= bufsize) + goto stop; + } + } +stop: + + /* Compare the images */ + for (i = 0; i < bufsize; i++) { + KUNIT_EXPECT_EQ_MSG(test, scratch[i], buffer[i], "at i=%x", i); + if (scratch[i] != buffer[i]) + return; + } + + KUNIT_SUCCEED(); +} + +static void iov_kunit_destroy_xarray(void *data) +{ + struct xarray *xarray = data; + + xa_destroy(xarray); + kfree(xarray); +} + +static void __init iov_kunit_load_xarray(struct kunit *test, + struct iov_iter *iter, int dir, + struct xarray *xarray, + struct page **pages, size_t npages) +{ + size_t size = 0; + int i; + + for (i = 0; i < npages; i++) { + void *x = xa_store(xarray, i, pages[i], GFP_KERNEL); + + KUNIT_ASSERT_FALSE(test, xa_is_err(x)); + size += PAGE_SIZE; + } + iov_iter_xarray(iter, dir, xarray, 0, size); +} + +static struct xarray *iov_kunit_create_xarray(struct kunit *test) +{ + struct xarray *xarray; + + xarray = kzalloc(sizeof(struct xarray), GFP_KERNEL); + xa_init(xarray); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, xarray); + kunit_add_action_or_reset(test, iov_kunit_destroy_xarray, xarray); + return xarray; +} + +/* + * Test copying to a ITER_XARRAY-type iterator. + */ +static void __init iov_kunit_copy_to_xarray(struct kunit *test) +{ + const struct kvec_test_range *pr; + struct iov_iter iter; + struct xarray *xarray; + struct page **spages, **bpages; + u8 *scratch, *buffer; + size_t bufsize, npages, size, copied; + int i, patt; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + xarray = iov_kunit_create_xarray(test); + + scratch = iov_kunit_create_buffer(test, &spages, npages); + for (i = 0; i < bufsize; i++) + scratch[i] = pattern(i); + + buffer = iov_kunit_create_buffer(test, &bpages, npages); + memset(buffer, 0, bufsize); + + iov_kunit_load_xarray(test, &iter, READ, xarray, bpages, npages); + + i = 0; + for (pr = kvec_test_ranges; pr->from >= 0; pr++) { + size = pr->to - pr->from; + KUNIT_ASSERT_LE(test, pr->to, bufsize); + + iov_iter_xarray(&iter, READ, xarray, pr->from, size); + copied = copy_to_iter(scratch + i, size, &iter); + + KUNIT_EXPECT_EQ(test, copied, size); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_EXPECT_EQ(test, iter.iov_offset, size); + i += size; + } + + /* Build the expected image in the scratch buffer. */ + patt = 0; + memset(scratch, 0, bufsize); + for (pr = kvec_test_ranges; pr->from >= 0; pr++) + for (i = pr->from; i < pr->to; i++) + scratch[i] = pattern(patt++); + + /* Compare the images */ + for (i = 0; i < bufsize; i++) { + KUNIT_EXPECT_EQ_MSG(test, buffer[i], scratch[i], "at i=%x", i); + if (buffer[i] != scratch[i]) + return; + } + + KUNIT_SUCCEED(); +} + +/* + * Test copying from a ITER_XARRAY-type iterator. + */ +static void __init iov_kunit_copy_from_xarray(struct kunit *test) +{ + const struct kvec_test_range *pr; + struct iov_iter iter; + struct xarray *xarray; + struct page **spages, **bpages; + u8 *scratch, *buffer; + size_t bufsize, npages, size, copied; + int i, j; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + xarray = iov_kunit_create_xarray(test); + + buffer = iov_kunit_create_buffer(test, &bpages, npages); + for (i = 0; i < bufsize; i++) + buffer[i] = pattern(i); + + scratch = iov_kunit_create_buffer(test, &spages, npages); + memset(scratch, 0, bufsize); + + iov_kunit_load_xarray(test, &iter, READ, xarray, bpages, npages); + + i = 0; + for (pr = kvec_test_ranges; pr->from >= 0; pr++) { + size = pr->to - pr->from; + KUNIT_ASSERT_LE(test, pr->to, bufsize); + + iov_iter_xarray(&iter, WRITE, xarray, pr->from, size); + copied = copy_from_iter(scratch + i, size, &iter); + + KUNIT_EXPECT_EQ(test, copied, size); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_EXPECT_EQ(test, iter.iov_offset, size); + i += size; + } + + /* Build the expected image in the main buffer. */ + i = 0; + memset(buffer, 0, bufsize); + for (pr = kvec_test_ranges; pr->from >= 0; pr++) { + for (j = pr->from; j < pr->to; j++) { + buffer[i++] = pattern(j); + if (i >= bufsize) + goto stop; + } + } +stop: + + /* Compare the images */ + for (i = 0; i < bufsize; i++) { + KUNIT_EXPECT_EQ_MSG(test, scratch[i], buffer[i], "at i=%x", i); + if (scratch[i] != buffer[i]) + return; + } + + KUNIT_SUCCEED(); +} + +/* + * Test the extraction of ITER_KVEC-type iterators. + */ +static void __init iov_kunit_extract_pages_kvec(struct kunit *test) +{ + const struct kvec_test_range *pr; + struct iov_iter iter; + struct page **bpages, *pagelist[8], **pages = pagelist; + struct kvec kvec[8]; + u8 *buffer; + ssize_t len; + size_t bufsize, size = 0, npages; + int i, from; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + buffer = iov_kunit_create_buffer(test, &bpages, npages); + + iov_kunit_load_kvec(test, &iter, READ, kvec, ARRAY_SIZE(kvec), + buffer, bufsize, kvec_test_ranges); + size = iter.count; + + pr = kvec_test_ranges; + from = pr->from; + do { + size_t offset0 = LONG_MAX; + + for (i = 0; i < ARRAY_SIZE(pagelist); i++) + pagelist[i] = (void *)(unsigned long)0xaa55aa55aa55aa55ULL; + + len = iov_iter_extract_pages(&iter, &pages, 100 * 1024, + ARRAY_SIZE(pagelist), 0, &offset0); + KUNIT_EXPECT_GE(test, len, 0); + if (len < 0) + break; + KUNIT_EXPECT_GE(test, (ssize_t)offset0, 0); + KUNIT_EXPECT_LT(test, offset0, PAGE_SIZE); + KUNIT_EXPECT_LE(test, len, size); + KUNIT_EXPECT_EQ(test, iter.count, size - len); + size -= len; + + if (len == 0) + break; + + for (i = 0; i < ARRAY_SIZE(pagelist); i++) { + struct page *p; + ssize_t part = min_t(ssize_t, len, PAGE_SIZE - offset0); + int ix; + + KUNIT_ASSERT_GE(test, part, 0); + while (from == pr->to) { + pr++; + from = pr->from; + if (from < 0) + goto stop; + } + ix = from / PAGE_SIZE; + KUNIT_ASSERT_LT(test, ix, npages); + p = bpages[ix]; + KUNIT_EXPECT_PTR_EQ(test, pagelist[i], p); + KUNIT_EXPECT_EQ(test, offset0, from % PAGE_SIZE); + from += part; + len -= part; + KUNIT_ASSERT_GE(test, len, 0); + if (len == 0) + break; + offset0 = 0; + } + + if (test->status == KUNIT_FAILURE) + break; + } while (iov_iter_count(&iter) > 0); + +stop: + KUNIT_EXPECT_EQ(test, size, 0); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_SUCCEED(); +} + +/* + * Test the extraction of ITER_BVEC-type iterators. + */ +static void __init iov_kunit_extract_pages_bvec(struct kunit *test) +{ + const struct bvec_test_range *pr; + struct iov_iter iter; + struct page **bpages, *pagelist[8], **pages = pagelist; + struct bio_vec bvec[8]; + ssize_t len; + size_t bufsize, size = 0, npages; + int i, from; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + iov_kunit_create_buffer(test, &bpages, npages); + iov_kunit_load_bvec(test, &iter, READ, bvec, ARRAY_SIZE(bvec), + bpages, npages, bufsize, bvec_test_ranges); + size = iter.count; + + pr = bvec_test_ranges; + from = pr->from; + do { + size_t offset0 = LONG_MAX; + + for (i = 0; i < ARRAY_SIZE(pagelist); i++) + pagelist[i] = (void *)(unsigned long)0xaa55aa55aa55aa55ULL; + + len = iov_iter_extract_pages(&iter, &pages, 100 * 1024, + ARRAY_SIZE(pagelist), 0, &offset0); + KUNIT_EXPECT_GE(test, len, 0); + if (len < 0) + break; + KUNIT_EXPECT_GE(test, (ssize_t)offset0, 0); + KUNIT_EXPECT_LT(test, offset0, PAGE_SIZE); + KUNIT_EXPECT_LE(test, len, size); + KUNIT_EXPECT_EQ(test, iter.count, size - len); + size -= len; + + if (len == 0) + break; + + for (i = 0; i < ARRAY_SIZE(pagelist); i++) { + struct page *p; + ssize_t part = min_t(ssize_t, len, PAGE_SIZE - offset0); + int ix; + + KUNIT_ASSERT_GE(test, part, 0); + while (from == pr->to) { + pr++; + from = pr->from; + if (from < 0) + goto stop; + } + ix = pr->page + from / PAGE_SIZE; + KUNIT_ASSERT_LT(test, ix, npages); + p = bpages[ix]; + KUNIT_EXPECT_PTR_EQ(test, pagelist[i], p); + KUNIT_EXPECT_EQ(test, offset0, from % PAGE_SIZE); + from += part; + len -= part; + KUNIT_ASSERT_GE(test, len, 0); + if (len == 0) + break; + offset0 = 0; + } + + if (test->status == KUNIT_FAILURE) + break; + } while (iov_iter_count(&iter) > 0); + +stop: + KUNIT_EXPECT_EQ(test, size, 0); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_SUCCEED(); +} + +/* + * Test the extraction of ITER_XARRAY-type iterators. + */ +static void __init iov_kunit_extract_pages_xarray(struct kunit *test) +{ + const struct kvec_test_range *pr; + struct iov_iter iter; + struct xarray *xarray; + struct page **bpages, *pagelist[8], **pages = pagelist; + ssize_t len; + size_t bufsize, size = 0, npages; + int i, from; + + bufsize = 0x100000; + npages = bufsize / PAGE_SIZE; + + xarray = iov_kunit_create_xarray(test); + + iov_kunit_create_buffer(test, &bpages, npages); + iov_kunit_load_xarray(test, &iter, READ, xarray, bpages, npages); + + for (pr = kvec_test_ranges; pr->from >= 0; pr++) { + from = pr->from; + size = pr->to - from; + KUNIT_ASSERT_LE(test, pr->to, bufsize); + + iov_iter_xarray(&iter, WRITE, xarray, from, size); + + do { + size_t offset0 = LONG_MAX; + + for (i = 0; i < ARRAY_SIZE(pagelist); i++) + pagelist[i] = (void *)(unsigned long)0xaa55aa55aa55aa55ULL; + + len = iov_iter_extract_pages(&iter, &pages, 100 * 1024, + ARRAY_SIZE(pagelist), 0, &offset0); + KUNIT_EXPECT_GE(test, len, 0); + if (len < 0) + break; + KUNIT_EXPECT_LE(test, len, size); + KUNIT_EXPECT_EQ(test, iter.count, size - len); + if (len == 0) + break; + size -= len; + KUNIT_EXPECT_GE(test, (ssize_t)offset0, 0); + KUNIT_EXPECT_LT(test, offset0, PAGE_SIZE); + + for (i = 0; i < ARRAY_SIZE(pagelist); i++) { + struct page *p; + ssize_t part = min_t(ssize_t, len, PAGE_SIZE - offset0); + int ix; + + KUNIT_ASSERT_GE(test, part, 0); + ix = from / PAGE_SIZE; + KUNIT_ASSERT_LT(test, ix, npages); + p = bpages[ix]; + KUNIT_EXPECT_PTR_EQ(test, pagelist[i], p); + KUNIT_EXPECT_EQ(test, offset0, from % PAGE_SIZE); + from += part; + len -= part; + KUNIT_ASSERT_GE(test, len, 0); + if (len == 0) + break; + offset0 = 0; + } + + if (test->status == KUNIT_FAILURE) + goto stop; + } while (iov_iter_count(&iter) > 0); + + KUNIT_EXPECT_EQ(test, size, 0); + KUNIT_EXPECT_EQ(test, iter.count, 0); + KUNIT_EXPECT_EQ(test, iter.iov_offset, pr->to - pr->from); + } + +stop: + KUNIT_SUCCEED(); +} + +static struct kunit_case __refdata iov_kunit_cases[] = { + KUNIT_CASE(iov_kunit_copy_to_kvec), + KUNIT_CASE(iov_kunit_copy_from_kvec), + KUNIT_CASE(iov_kunit_copy_to_bvec), + KUNIT_CASE(iov_kunit_copy_from_bvec), + KUNIT_CASE(iov_kunit_copy_to_xarray), + KUNIT_CASE(iov_kunit_copy_from_xarray), + KUNIT_CASE(iov_kunit_extract_pages_kvec), + KUNIT_CASE(iov_kunit_extract_pages_bvec), + KUNIT_CASE(iov_kunit_extract_pages_xarray), + {} +}; + +static struct kunit_suite iov_kunit_suite = { + .name = "iov_iter", + .test_cases = iov_kunit_cases, +}; + +kunit_test_suites(&iov_kunit_suite); diff --git a/lib/raid6/Makefile b/lib/raid6/Makefile index 45e17619422b..035b0a4db476 100644 --- a/lib/raid6/Makefile +++ b/lib/raid6/Makefile @@ -9,6 +9,7 @@ raid6_pq-$(CONFIG_ALTIVEC) += altivec1.o altivec2.o altivec4.o altivec8.o \ vpermxor1.o vpermxor2.o vpermxor4.o vpermxor8.o raid6_pq-$(CONFIG_KERNEL_MODE_NEON) += neon.o neon1.o neon2.o neon4.o neon8.o recov_neon.o recov_neon_inner.o raid6_pq-$(CONFIG_S390) += s390vx8.o recov_s390xc.o +raid6_pq-$(CONFIG_LOONGARCH) += loongarch_simd.o recov_loongarch_simd.o hostprogs += mktables diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c index a22a05c9af8a..0ec534faf019 100644 --- a/lib/raid6/algos.c +++ b/lib/raid6/algos.c @@ -73,6 +73,14 @@ const struct raid6_calls * const raid6_algos[] = { &raid6_neonx2, &raid6_neonx1, #endif +#ifdef CONFIG_LOONGARCH +#ifdef CONFIG_CPU_HAS_LASX + &raid6_lasx, +#endif +#ifdef CONFIG_CPU_HAS_LSX + &raid6_lsx, +#endif +#endif #if defined(__ia64__) &raid6_intx32, &raid6_intx16, @@ -103,6 +111,14 @@ const struct raid6_recov_calls *const raid6_recov_algos[] = { #endif #if defined(CONFIG_KERNEL_MODE_NEON) &raid6_recov_neon, +#endif +#ifdef CONFIG_LOONGARCH +#ifdef CONFIG_CPU_HAS_LASX + &raid6_recov_lasx, +#endif +#ifdef CONFIG_CPU_HAS_LSX + &raid6_recov_lsx, +#endif #endif &raid6_recov_intx1, NULL diff --git a/lib/raid6/loongarch.h b/lib/raid6/loongarch.h new file mode 100644 index 000000000000..acfc33ce7056 --- /dev/null +++ b/lib/raid6/loongarch.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2023 WANG Xuerui + * + * raid6/loongarch.h + * + * Definitions common to LoongArch RAID-6 code only + */ + +#ifndef _LIB_RAID6_LOONGARCH_H +#define _LIB_RAID6_LOONGARCH_H + +#ifdef __KERNEL__ + +#include +#include + +#else /* for user-space testing */ + +#include + +/* have to supply these defines for glibc 2.37- and musl */ +#ifndef HWCAP_LOONGARCH_LSX +#define HWCAP_LOONGARCH_LSX (1 << 4) +#endif +#ifndef HWCAP_LOONGARCH_LASX +#define HWCAP_LOONGARCH_LASX (1 << 5) +#endif + +#define kernel_fpu_begin() +#define kernel_fpu_end() + +#define cpu_has_lsx (getauxval(AT_HWCAP) & HWCAP_LOONGARCH_LSX) +#define cpu_has_lasx (getauxval(AT_HWCAP) & HWCAP_LOONGARCH_LASX) + +#endif /* __KERNEL__ */ + +#endif /* _LIB_RAID6_LOONGARCH_H */ diff --git a/lib/raid6/loongarch_simd.c b/lib/raid6/loongarch_simd.c new file mode 100644 index 000000000000..aa5d9f924ca3 --- /dev/null +++ b/lib/raid6/loongarch_simd.c @@ -0,0 +1,422 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * RAID6 syndrome calculations in LoongArch SIMD (LSX & LASX) + * + * Copyright 2023 WANG Xuerui + * + * Based on the generic RAID-6 code (int.uc): + * + * Copyright 2002-2004 H. Peter Anvin + */ + +#include +#include "loongarch.h" + +/* + * The vector algorithms are currently priority 0, which means the generic + * scalar algorithms are not being disabled if vector support is present. + * This is like the similar LoongArch RAID5 XOR code, with the main reason + * repeated here: it cannot be ruled out at this point of time, that some + * future (maybe reduced) models could run the vector algorithms slower than + * the scalar ones, maybe for errata or micro-op reasons. It may be + * appropriate to revisit this after one or two more uarch generations. + */ + +#ifdef CONFIG_CPU_HAS_LSX +#define NSIZE 16 + +static int raid6_has_lsx(void) +{ + return cpu_has_lsx; +} + +static void raid6_lsx_gen_syndrome(int disks, size_t bytes, void **ptrs) +{ + u8 **dptr = (u8 **)ptrs; + u8 *p, *q; + int d, z, z0; + + z0 = disks - 3; /* Highest data disk */ + p = dptr[z0+1]; /* XOR parity */ + q = dptr[z0+2]; /* RS syndrome */ + + kernel_fpu_begin(); + + /* + * $vr0, $vr1, $vr2, $vr3: wp + * $vr4, $vr5, $vr6, $vr7: wq + * $vr8, $vr9, $vr10, $vr11: wd + * $vr12, $vr13, $vr14, $vr15: w2 + * $vr16, $vr17, $vr18, $vr19: w1 + */ + for (d = 0; d < bytes; d += NSIZE*4) { + /* wq$$ = wp$$ = *(unative_t *)&dptr[z0][d+$$*NSIZE]; */ + asm volatile("vld $vr0, %0" : : "m"(dptr[z0][d+0*NSIZE])); + asm volatile("vld $vr1, %0" : : "m"(dptr[z0][d+1*NSIZE])); + asm volatile("vld $vr2, %0" : : "m"(dptr[z0][d+2*NSIZE])); + asm volatile("vld $vr3, %0" : : "m"(dptr[z0][d+3*NSIZE])); + asm volatile("vori.b $vr4, $vr0, 0"); + asm volatile("vori.b $vr5, $vr1, 0"); + asm volatile("vori.b $vr6, $vr2, 0"); + asm volatile("vori.b $vr7, $vr3, 0"); + for (z = z0-1; z >= 0; z--) { + /* wd$$ = *(unative_t *)&dptr[z][d+$$*NSIZE]; */ + asm volatile("vld $vr8, %0" : : "m"(dptr[z][d+0*NSIZE])); + asm volatile("vld $vr9, %0" : : "m"(dptr[z][d+1*NSIZE])); + asm volatile("vld $vr10, %0" : : "m"(dptr[z][d+2*NSIZE])); + asm volatile("vld $vr11, %0" : : "m"(dptr[z][d+3*NSIZE])); + /* wp$$ ^= wd$$; */ + asm volatile("vxor.v $vr0, $vr0, $vr8"); + asm volatile("vxor.v $vr1, $vr1, $vr9"); + asm volatile("vxor.v $vr2, $vr2, $vr10"); + asm volatile("vxor.v $vr3, $vr3, $vr11"); + /* w2$$ = MASK(wq$$); */ + asm volatile("vslti.b $vr12, $vr4, 0"); + asm volatile("vslti.b $vr13, $vr5, 0"); + asm volatile("vslti.b $vr14, $vr6, 0"); + asm volatile("vslti.b $vr15, $vr7, 0"); + /* w1$$ = SHLBYTE(wq$$); */ + asm volatile("vslli.b $vr16, $vr4, 1"); + asm volatile("vslli.b $vr17, $vr5, 1"); + asm volatile("vslli.b $vr18, $vr6, 1"); + asm volatile("vslli.b $vr19, $vr7, 1"); + /* w2$$ &= NBYTES(0x1d); */ + asm volatile("vandi.b $vr12, $vr12, 0x1d"); + asm volatile("vandi.b $vr13, $vr13, 0x1d"); + asm volatile("vandi.b $vr14, $vr14, 0x1d"); + asm volatile("vandi.b $vr15, $vr15, 0x1d"); + /* w1$$ ^= w2$$; */ + asm volatile("vxor.v $vr16, $vr16, $vr12"); + asm volatile("vxor.v $vr17, $vr17, $vr13"); + asm volatile("vxor.v $vr18, $vr18, $vr14"); + asm volatile("vxor.v $vr19, $vr19, $vr15"); + /* wq$$ = w1$$ ^ wd$$; */ + asm volatile("vxor.v $vr4, $vr16, $vr8"); + asm volatile("vxor.v $vr5, $vr17, $vr9"); + asm volatile("vxor.v $vr6, $vr18, $vr10"); + asm volatile("vxor.v $vr7, $vr19, $vr11"); + } + /* *(unative_t *)&p[d+NSIZE*$$] = wp$$; */ + asm volatile("vst $vr0, %0" : "=m"(p[d+NSIZE*0])); + asm volatile("vst $vr1, %0" : "=m"(p[d+NSIZE*1])); + asm volatile("vst $vr2, %0" : "=m"(p[d+NSIZE*2])); + asm volatile("vst $vr3, %0" : "=m"(p[d+NSIZE*3])); + /* *(unative_t *)&q[d+NSIZE*$$] = wq$$; */ + asm volatile("vst $vr4, %0" : "=m"(q[d+NSIZE*0])); + asm volatile("vst $vr5, %0" : "=m"(q[d+NSIZE*1])); + asm volatile("vst $vr6, %0" : "=m"(q[d+NSIZE*2])); + asm volatile("vst $vr7, %0" : "=m"(q[d+NSIZE*3])); + } + + kernel_fpu_end(); +} + +static void raid6_lsx_xor_syndrome(int disks, int start, int stop, + size_t bytes, void **ptrs) +{ + u8 **dptr = (u8 **)ptrs; + u8 *p, *q; + int d, z, z0; + + z0 = stop; /* P/Q right side optimization */ + p = dptr[disks-2]; /* XOR parity */ + q = dptr[disks-1]; /* RS syndrome */ + + kernel_fpu_begin(); + + /* + * $vr0, $vr1, $vr2, $vr3: wp + * $vr4, $vr5, $vr6, $vr7: wq + * $vr8, $vr9, $vr10, $vr11: wd + * $vr12, $vr13, $vr14, $vr15: w2 + * $vr16, $vr17, $vr18, $vr19: w1 + */ + for (d = 0; d < bytes; d += NSIZE*4) { + /* P/Q data pages */ + /* wq$$ = wp$$ = *(unative_t *)&dptr[z0][d+$$*NSIZE]; */ + asm volatile("vld $vr0, %0" : : "m"(dptr[z0][d+0*NSIZE])); + asm volatile("vld $vr1, %0" : : "m"(dptr[z0][d+1*NSIZE])); + asm volatile("vld $vr2, %0" : : "m"(dptr[z0][d+2*NSIZE])); + asm volatile("vld $vr3, %0" : : "m"(dptr[z0][d+3*NSIZE])); + asm volatile("vori.b $vr4, $vr0, 0"); + asm volatile("vori.b $vr5, $vr1, 0"); + asm volatile("vori.b $vr6, $vr2, 0"); + asm volatile("vori.b $vr7, $vr3, 0"); + for (z = z0-1; z >= start; z--) { + /* wd$$ = *(unative_t *)&dptr[z][d+$$*NSIZE]; */ + asm volatile("vld $vr8, %0" : : "m"(dptr[z][d+0*NSIZE])); + asm volatile("vld $vr9, %0" : : "m"(dptr[z][d+1*NSIZE])); + asm volatile("vld $vr10, %0" : : "m"(dptr[z][d+2*NSIZE])); + asm volatile("vld $vr11, %0" : : "m"(dptr[z][d+3*NSIZE])); + /* wp$$ ^= wd$$; */ + asm volatile("vxor.v $vr0, $vr0, $vr8"); + asm volatile("vxor.v $vr1, $vr1, $vr9"); + asm volatile("vxor.v $vr2, $vr2, $vr10"); + asm volatile("vxor.v $vr3, $vr3, $vr11"); + /* w2$$ = MASK(wq$$); */ + asm volatile("vslti.b $vr12, $vr4, 0"); + asm volatile("vslti.b $vr13, $vr5, 0"); + asm volatile("vslti.b $vr14, $vr6, 0"); + asm volatile("vslti.b $vr15, $vr7, 0"); + /* w1$$ = SHLBYTE(wq$$); */ + asm volatile("vslli.b $vr16, $vr4, 1"); + asm volatile("vslli.b $vr17, $vr5, 1"); + asm volatile("vslli.b $vr18, $vr6, 1"); + asm volatile("vslli.b $vr19, $vr7, 1"); + /* w2$$ &= NBYTES(0x1d); */ + asm volatile("vandi.b $vr12, $vr12, 0x1d"); + asm volatile("vandi.b $vr13, $vr13, 0x1d"); + asm volatile("vandi.b $vr14, $vr14, 0x1d"); + asm volatile("vandi.b $vr15, $vr15, 0x1d"); + /* w1$$ ^= w2$$; */ + asm volatile("vxor.v $vr16, $vr16, $vr12"); + asm volatile("vxor.v $vr17, $vr17, $vr13"); + asm volatile("vxor.v $vr18, $vr18, $vr14"); + asm volatile("vxor.v $vr19, $vr19, $vr15"); + /* wq$$ = w1$$ ^ wd$$; */ + asm volatile("vxor.v $vr4, $vr16, $vr8"); + asm volatile("vxor.v $vr5, $vr17, $vr9"); + asm volatile("vxor.v $vr6, $vr18, $vr10"); + asm volatile("vxor.v $vr7, $vr19, $vr11"); + } + + /* P/Q left side optimization */ + for (z = start-1; z >= 0; z--) { + /* w2$$ = MASK(wq$$); */ + asm volatile("vslti.b $vr12, $vr4, 0"); + asm volatile("vslti.b $vr13, $vr5, 0"); + asm volatile("vslti.b $vr14, $vr6, 0"); + asm volatile("vslti.b $vr15, $vr7, 0"); + /* w1$$ = SHLBYTE(wq$$); */ + asm volatile("vslli.b $vr16, $vr4, 1"); + asm volatile("vslli.b $vr17, $vr5, 1"); + asm volatile("vslli.b $vr18, $vr6, 1"); + asm volatile("vslli.b $vr19, $vr7, 1"); + /* w2$$ &= NBYTES(0x1d); */ + asm volatile("vandi.b $vr12, $vr12, 0x1d"); + asm volatile("vandi.b $vr13, $vr13, 0x1d"); + asm volatile("vandi.b $vr14, $vr14, 0x1d"); + asm volatile("vandi.b $vr15, $vr15, 0x1d"); + /* wq$$ = w1$$ ^ w2$$; */ + asm volatile("vxor.v $vr4, $vr16, $vr12"); + asm volatile("vxor.v $vr5, $vr17, $vr13"); + asm volatile("vxor.v $vr6, $vr18, $vr14"); + asm volatile("vxor.v $vr7, $vr19, $vr15"); + } + /* + * *(unative_t *)&p[d+NSIZE*$$] ^= wp$$; + * *(unative_t *)&q[d+NSIZE*$$] ^= wq$$; + */ + asm volatile( + "vld $vr20, %0\n\t" + "vld $vr21, %1\n\t" + "vld $vr22, %2\n\t" + "vld $vr23, %3\n\t" + "vld $vr24, %4\n\t" + "vld $vr25, %5\n\t" + "vld $vr26, %6\n\t" + "vld $vr27, %7\n\t" + "vxor.v $vr20, $vr20, $vr0\n\t" + "vxor.v $vr21, $vr21, $vr1\n\t" + "vxor.v $vr22, $vr22, $vr2\n\t" + "vxor.v $vr23, $vr23, $vr3\n\t" + "vxor.v $vr24, $vr24, $vr4\n\t" + "vxor.v $vr25, $vr25, $vr5\n\t" + "vxor.v $vr26, $vr26, $vr6\n\t" + "vxor.v $vr27, $vr27, $vr7\n\t" + "vst $vr20, %0\n\t" + "vst $vr21, %1\n\t" + "vst $vr22, %2\n\t" + "vst $vr23, %3\n\t" + "vst $vr24, %4\n\t" + "vst $vr25, %5\n\t" + "vst $vr26, %6\n\t" + "vst $vr27, %7\n\t" + : "+m"(p[d+NSIZE*0]), "+m"(p[d+NSIZE*1]), + "+m"(p[d+NSIZE*2]), "+m"(p[d+NSIZE*3]), + "+m"(q[d+NSIZE*0]), "+m"(q[d+NSIZE*1]), + "+m"(q[d+NSIZE*2]), "+m"(q[d+NSIZE*3]) + ); + } + + kernel_fpu_end(); +} + +const struct raid6_calls raid6_lsx = { + raid6_lsx_gen_syndrome, + raid6_lsx_xor_syndrome, + raid6_has_lsx, + "lsx", + .priority = 0 /* see the comment near the top of the file for reason */ +}; + +#undef NSIZE +#endif /* CONFIG_CPU_HAS_LSX */ + +#ifdef CONFIG_CPU_HAS_LASX +#define NSIZE 32 + +static int raid6_has_lasx(void) +{ + return cpu_has_lasx; +} + +static void raid6_lasx_gen_syndrome(int disks, size_t bytes, void **ptrs) +{ + u8 **dptr = (u8 **)ptrs; + u8 *p, *q; + int d, z, z0; + + z0 = disks - 3; /* Highest data disk */ + p = dptr[z0+1]; /* XOR parity */ + q = dptr[z0+2]; /* RS syndrome */ + + kernel_fpu_begin(); + + /* + * $xr0, $xr1: wp + * $xr2, $xr3: wq + * $xr4, $xr5: wd + * $xr6, $xr7: w2 + * $xr8, $xr9: w1 + */ + for (d = 0; d < bytes; d += NSIZE*2) { + /* wq$$ = wp$$ = *(unative_t *)&dptr[z0][d+$$*NSIZE]; */ + asm volatile("xvld $xr0, %0" : : "m"(dptr[z0][d+0*NSIZE])); + asm volatile("xvld $xr1, %0" : : "m"(dptr[z0][d+1*NSIZE])); + asm volatile("xvori.b $xr2, $xr0, 0"); + asm volatile("xvori.b $xr3, $xr1, 0"); + for (z = z0-1; z >= 0; z--) { + /* wd$$ = *(unative_t *)&dptr[z][d+$$*NSIZE]; */ + asm volatile("xvld $xr4, %0" : : "m"(dptr[z][d+0*NSIZE])); + asm volatile("xvld $xr5, %0" : : "m"(dptr[z][d+1*NSIZE])); + /* wp$$ ^= wd$$; */ + asm volatile("xvxor.v $xr0, $xr0, $xr4"); + asm volatile("xvxor.v $xr1, $xr1, $xr5"); + /* w2$$ = MASK(wq$$); */ + asm volatile("xvslti.b $xr6, $xr2, 0"); + asm volatile("xvslti.b $xr7, $xr3, 0"); + /* w1$$ = SHLBYTE(wq$$); */ + asm volatile("xvslli.b $xr8, $xr2, 1"); + asm volatile("xvslli.b $xr9, $xr3, 1"); + /* w2$$ &= NBYTES(0x1d); */ + asm volatile("xvandi.b $xr6, $xr6, 0x1d"); + asm volatile("xvandi.b $xr7, $xr7, 0x1d"); + /* w1$$ ^= w2$$; */ + asm volatile("xvxor.v $xr8, $xr8, $xr6"); + asm volatile("xvxor.v $xr9, $xr9, $xr7"); + /* wq$$ = w1$$ ^ wd$$; */ + asm volatile("xvxor.v $xr2, $xr8, $xr4"); + asm volatile("xvxor.v $xr3, $xr9, $xr5"); + } + /* *(unative_t *)&p[d+NSIZE*$$] = wp$$; */ + asm volatile("xvst $xr0, %0" : "=m"(p[d+NSIZE*0])); + asm volatile("xvst $xr1, %0" : "=m"(p[d+NSIZE*1])); + /* *(unative_t *)&q[d+NSIZE*$$] = wq$$; */ + asm volatile("xvst $xr2, %0" : "=m"(q[d+NSIZE*0])); + asm volatile("xvst $xr3, %0" : "=m"(q[d+NSIZE*1])); + } + + kernel_fpu_end(); +} + +static void raid6_lasx_xor_syndrome(int disks, int start, int stop, + size_t bytes, void **ptrs) +{ + u8 **dptr = (u8 **)ptrs; + u8 *p, *q; + int d, z, z0; + + z0 = stop; /* P/Q right side optimization */ + p = dptr[disks-2]; /* XOR parity */ + q = dptr[disks-1]; /* RS syndrome */ + + kernel_fpu_begin(); + + /* + * $xr0, $xr1: wp + * $xr2, $xr3: wq + * $xr4, $xr5: wd + * $xr6, $xr7: w2 + * $xr8, $xr9: w1 + */ + for (d = 0; d < bytes; d += NSIZE*2) { + /* P/Q data pages */ + /* wq$$ = wp$$ = *(unative_t *)&dptr[z0][d+$$*NSIZE]; */ + asm volatile("xvld $xr0, %0" : : "m"(dptr[z0][d+0*NSIZE])); + asm volatile("xvld $xr1, %0" : : "m"(dptr[z0][d+1*NSIZE])); + asm volatile("xvori.b $xr2, $xr0, 0"); + asm volatile("xvori.b $xr3, $xr1, 0"); + for (z = z0-1; z >= start; z--) { + /* wd$$ = *(unative_t *)&dptr[z][d+$$*NSIZE]; */ + asm volatile("xvld $xr4, %0" : : "m"(dptr[z][d+0*NSIZE])); + asm volatile("xvld $xr5, %0" : : "m"(dptr[z][d+1*NSIZE])); + /* wp$$ ^= wd$$; */ + asm volatile("xvxor.v $xr0, $xr0, $xr4"); + asm volatile("xvxor.v $xr1, $xr1, $xr5"); + /* w2$$ = MASK(wq$$); */ + asm volatile("xvslti.b $xr6, $xr2, 0"); + asm volatile("xvslti.b $xr7, $xr3, 0"); + /* w1$$ = SHLBYTE(wq$$); */ + asm volatile("xvslli.b $xr8, $xr2, 1"); + asm volatile("xvslli.b $xr9, $xr3, 1"); + /* w2$$ &= NBYTES(0x1d); */ + asm volatile("xvandi.b $xr6, $xr6, 0x1d"); + asm volatile("xvandi.b $xr7, $xr7, 0x1d"); + /* w1$$ ^= w2$$; */ + asm volatile("xvxor.v $xr8, $xr8, $xr6"); + asm volatile("xvxor.v $xr9, $xr9, $xr7"); + /* wq$$ = w1$$ ^ wd$$; */ + asm volatile("xvxor.v $xr2, $xr8, $xr4"); + asm volatile("xvxor.v $xr3, $xr9, $xr5"); + } + + /* P/Q left side optimization */ + for (z = start-1; z >= 0; z--) { + /* w2$$ = MASK(wq$$); */ + asm volatile("xvslti.b $xr6, $xr2, 0"); + asm volatile("xvslti.b $xr7, $xr3, 0"); + /* w1$$ = SHLBYTE(wq$$); */ + asm volatile("xvslli.b $xr8, $xr2, 1"); + asm volatile("xvslli.b $xr9, $xr3, 1"); + /* w2$$ &= NBYTES(0x1d); */ + asm volatile("xvandi.b $xr6, $xr6, 0x1d"); + asm volatile("xvandi.b $xr7, $xr7, 0x1d"); + /* wq$$ = w1$$ ^ w2$$; */ + asm volatile("xvxor.v $xr2, $xr8, $xr6"); + asm volatile("xvxor.v $xr3, $xr9, $xr7"); + } + /* + * *(unative_t *)&p[d+NSIZE*$$] ^= wp$$; + * *(unative_t *)&q[d+NSIZE*$$] ^= wq$$; + */ + asm volatile( + "xvld $xr10, %0\n\t" + "xvld $xr11, %1\n\t" + "xvld $xr12, %2\n\t" + "xvld $xr13, %3\n\t" + "xvxor.v $xr10, $xr10, $xr0\n\t" + "xvxor.v $xr11, $xr11, $xr1\n\t" + "xvxor.v $xr12, $xr12, $xr2\n\t" + "xvxor.v $xr13, $xr13, $xr3\n\t" + "xvst $xr10, %0\n\t" + "xvst $xr11, %1\n\t" + "xvst $xr12, %2\n\t" + "xvst $xr13, %3\n\t" + : "+m"(p[d+NSIZE*0]), "+m"(p[d+NSIZE*1]), + "+m"(q[d+NSIZE*0]), "+m"(q[d+NSIZE*1]) + ); + } + + kernel_fpu_end(); +} + +const struct raid6_calls raid6_lasx = { + raid6_lasx_gen_syndrome, + raid6_lasx_xor_syndrome, + raid6_has_lasx, + "lasx", + .priority = 0 /* see the comment near the top of the file for reason */ +}; +#undef NSIZE +#endif /* CONFIG_CPU_HAS_LASX */ diff --git a/lib/raid6/recov_loongarch_simd.c b/lib/raid6/recov_loongarch_simd.c new file mode 100644 index 000000000000..94aeac85e6f7 --- /dev/null +++ b/lib/raid6/recov_loongarch_simd.c @@ -0,0 +1,513 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * RAID6 recovery algorithms in LoongArch SIMD (LSX & LASX) + * + * Copyright (C) 2023 WANG Xuerui + * + * Originally based on recov_avx2.c and recov_ssse3.c: + * + * Copyright (C) 2012 Intel Corporation + * Author: Jim Kukunas + */ + +#include +#include "loongarch.h" + +/* + * Unlike with the syndrome calculation algorithms, there's no boot-time + * selection of recovery algorithms by benchmarking, so we have to specify + * the priorities and hope the future cores will all have decent vector + * support (i.e. no LASX slower than LSX, or even scalar code). + */ + +#ifdef CONFIG_CPU_HAS_LSX +static int raid6_has_lsx(void) +{ + return cpu_has_lsx; +} + +static void raid6_2data_recov_lsx(int disks, size_t bytes, int faila, + int failb, void **ptrs) +{ + u8 *p, *q, *dp, *dq; + const u8 *pbmul; /* P multiplier table for B data */ + const u8 *qmul; /* Q multiplier table (for both) */ + + p = (u8 *)ptrs[disks - 2]; + q = (u8 *)ptrs[disks - 1]; + + /* + * Compute syndrome with zero for the missing data pages + * Use the dead data pages as temporary storage for + * delta p and delta q + */ + dp = (u8 *)ptrs[faila]; + ptrs[faila] = (void *)raid6_empty_zero_page; + ptrs[disks - 2] = dp; + dq = (u8 *)ptrs[failb]; + ptrs[failb] = (void *)raid6_empty_zero_page; + ptrs[disks - 1] = dq; + + raid6_call.gen_syndrome(disks, bytes, ptrs); + + /* Restore pointer table */ + ptrs[faila] = dp; + ptrs[failb] = dq; + ptrs[disks - 2] = p; + ptrs[disks - 1] = q; + + /* Now, pick the proper data tables */ + pbmul = raid6_vgfmul[raid6_gfexi[failb - faila]]; + qmul = raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila] ^ raid6_gfexp[failb]]]; + + kernel_fpu_begin(); + + /* + * vr20, vr21: qmul + * vr22, vr23: pbmul + */ + asm volatile("vld $vr20, %0" : : "m" (qmul[0])); + asm volatile("vld $vr21, %0" : : "m" (qmul[16])); + asm volatile("vld $vr22, %0" : : "m" (pbmul[0])); + asm volatile("vld $vr23, %0" : : "m" (pbmul[16])); + + while (bytes) { + /* vr4 - vr7: Q */ + asm volatile("vld $vr4, %0" : : "m" (q[0])); + asm volatile("vld $vr5, %0" : : "m" (q[16])); + asm volatile("vld $vr6, %0" : : "m" (q[32])); + asm volatile("vld $vr7, %0" : : "m" (q[48])); + /* vr4 - vr7: Q + Qxy */ + asm volatile("vld $vr8, %0" : : "m" (dq[0])); + asm volatile("vld $vr9, %0" : : "m" (dq[16])); + asm volatile("vld $vr10, %0" : : "m" (dq[32])); + asm volatile("vld $vr11, %0" : : "m" (dq[48])); + asm volatile("vxor.v $vr4, $vr4, $vr8"); + asm volatile("vxor.v $vr5, $vr5, $vr9"); + asm volatile("vxor.v $vr6, $vr6, $vr10"); + asm volatile("vxor.v $vr7, $vr7, $vr11"); + /* vr0 - vr3: P */ + asm volatile("vld $vr0, %0" : : "m" (p[0])); + asm volatile("vld $vr1, %0" : : "m" (p[16])); + asm volatile("vld $vr2, %0" : : "m" (p[32])); + asm volatile("vld $vr3, %0" : : "m" (p[48])); + /* vr0 - vr3: P + Pxy */ + asm volatile("vld $vr8, %0" : : "m" (dp[0])); + asm volatile("vld $vr9, %0" : : "m" (dp[16])); + asm volatile("vld $vr10, %0" : : "m" (dp[32])); + asm volatile("vld $vr11, %0" : : "m" (dp[48])); + asm volatile("vxor.v $vr0, $vr0, $vr8"); + asm volatile("vxor.v $vr1, $vr1, $vr9"); + asm volatile("vxor.v $vr2, $vr2, $vr10"); + asm volatile("vxor.v $vr3, $vr3, $vr11"); + + /* vr8 - vr11: higher 4 bits of each byte of (Q + Qxy) */ + asm volatile("vsrli.b $vr8, $vr4, 4"); + asm volatile("vsrli.b $vr9, $vr5, 4"); + asm volatile("vsrli.b $vr10, $vr6, 4"); + asm volatile("vsrli.b $vr11, $vr7, 4"); + /* vr4 - vr7: lower 4 bits of each byte of (Q + Qxy) */ + asm volatile("vandi.b $vr4, $vr4, 0x0f"); + asm volatile("vandi.b $vr5, $vr5, 0x0f"); + asm volatile("vandi.b $vr6, $vr6, 0x0f"); + asm volatile("vandi.b $vr7, $vr7, 0x0f"); + /* lookup from qmul[0] */ + asm volatile("vshuf.b $vr4, $vr20, $vr20, $vr4"); + asm volatile("vshuf.b $vr5, $vr20, $vr20, $vr5"); + asm volatile("vshuf.b $vr6, $vr20, $vr20, $vr6"); + asm volatile("vshuf.b $vr7, $vr20, $vr20, $vr7"); + /* lookup from qmul[16] */ + asm volatile("vshuf.b $vr8, $vr21, $vr21, $vr8"); + asm volatile("vshuf.b $vr9, $vr21, $vr21, $vr9"); + asm volatile("vshuf.b $vr10, $vr21, $vr21, $vr10"); + asm volatile("vshuf.b $vr11, $vr21, $vr21, $vr11"); + /* vr16 - vr19: B(Q + Qxy) */ + asm volatile("vxor.v $vr16, $vr8, $vr4"); + asm volatile("vxor.v $vr17, $vr9, $vr5"); + asm volatile("vxor.v $vr18, $vr10, $vr6"); + asm volatile("vxor.v $vr19, $vr11, $vr7"); + + /* vr4 - vr7: higher 4 bits of each byte of (P + Pxy) */ + asm volatile("vsrli.b $vr4, $vr0, 4"); + asm volatile("vsrli.b $vr5, $vr1, 4"); + asm volatile("vsrli.b $vr6, $vr2, 4"); + asm volatile("vsrli.b $vr7, $vr3, 4"); + /* vr12 - vr15: lower 4 bits of each byte of (P + Pxy) */ + asm volatile("vandi.b $vr12, $vr0, 0x0f"); + asm volatile("vandi.b $vr13, $vr1, 0x0f"); + asm volatile("vandi.b $vr14, $vr2, 0x0f"); + asm volatile("vandi.b $vr15, $vr3, 0x0f"); + /* lookup from pbmul[0] */ + asm volatile("vshuf.b $vr12, $vr22, $vr22, $vr12"); + asm volatile("vshuf.b $vr13, $vr22, $vr22, $vr13"); + asm volatile("vshuf.b $vr14, $vr22, $vr22, $vr14"); + asm volatile("vshuf.b $vr15, $vr22, $vr22, $vr15"); + /* lookup from pbmul[16] */ + asm volatile("vshuf.b $vr4, $vr23, $vr23, $vr4"); + asm volatile("vshuf.b $vr5, $vr23, $vr23, $vr5"); + asm volatile("vshuf.b $vr6, $vr23, $vr23, $vr6"); + asm volatile("vshuf.b $vr7, $vr23, $vr23, $vr7"); + /* vr4 - vr7: A(P + Pxy) */ + asm volatile("vxor.v $vr4, $vr4, $vr12"); + asm volatile("vxor.v $vr5, $vr5, $vr13"); + asm volatile("vxor.v $vr6, $vr6, $vr14"); + asm volatile("vxor.v $vr7, $vr7, $vr15"); + + /* vr4 - vr7: A(P + Pxy) + B(Q + Qxy) = Dx */ + asm volatile("vxor.v $vr4, $vr4, $vr16"); + asm volatile("vxor.v $vr5, $vr5, $vr17"); + asm volatile("vxor.v $vr6, $vr6, $vr18"); + asm volatile("vxor.v $vr7, $vr7, $vr19"); + asm volatile("vst $vr4, %0" : "=m" (dq[0])); + asm volatile("vst $vr5, %0" : "=m" (dq[16])); + asm volatile("vst $vr6, %0" : "=m" (dq[32])); + asm volatile("vst $vr7, %0" : "=m" (dq[48])); + + /* vr0 - vr3: P + Pxy + Dx = Dy */ + asm volatile("vxor.v $vr0, $vr0, $vr4"); + asm volatile("vxor.v $vr1, $vr1, $vr5"); + asm volatile("vxor.v $vr2, $vr2, $vr6"); + asm volatile("vxor.v $vr3, $vr3, $vr7"); + asm volatile("vst $vr0, %0" : "=m" (dp[0])); + asm volatile("vst $vr1, %0" : "=m" (dp[16])); + asm volatile("vst $vr2, %0" : "=m" (dp[32])); + asm volatile("vst $vr3, %0" : "=m" (dp[48])); + + bytes -= 64; + p += 64; + q += 64; + dp += 64; + dq += 64; + } + + kernel_fpu_end(); +} + +static void raid6_datap_recov_lsx(int disks, size_t bytes, int faila, + void **ptrs) +{ + u8 *p, *q, *dq; + const u8 *qmul; /* Q multiplier table */ + + p = (u8 *)ptrs[disks - 2]; + q = (u8 *)ptrs[disks - 1]; + + /* + * Compute syndrome with zero for the missing data page + * Use the dead data page as temporary storage for delta q + */ + dq = (u8 *)ptrs[faila]; + ptrs[faila] = (void *)raid6_empty_zero_page; + ptrs[disks - 1] = dq; + + raid6_call.gen_syndrome(disks, bytes, ptrs); + + /* Restore pointer table */ + ptrs[faila] = dq; + ptrs[disks - 1] = q; + + /* Now, pick the proper data tables */ + qmul = raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila]]]; + + kernel_fpu_begin(); + + /* vr22, vr23: qmul */ + asm volatile("vld $vr22, %0" : : "m" (qmul[0])); + asm volatile("vld $vr23, %0" : : "m" (qmul[16])); + + while (bytes) { + /* vr0 - vr3: P + Dx */ + asm volatile("vld $vr0, %0" : : "m" (p[0])); + asm volatile("vld $vr1, %0" : : "m" (p[16])); + asm volatile("vld $vr2, %0" : : "m" (p[32])); + asm volatile("vld $vr3, %0" : : "m" (p[48])); + /* vr4 - vr7: Qx */ + asm volatile("vld $vr4, %0" : : "m" (dq[0])); + asm volatile("vld $vr5, %0" : : "m" (dq[16])); + asm volatile("vld $vr6, %0" : : "m" (dq[32])); + asm volatile("vld $vr7, %0" : : "m" (dq[48])); + /* vr4 - vr7: Q + Qx */ + asm volatile("vld $vr8, %0" : : "m" (q[0])); + asm volatile("vld $vr9, %0" : : "m" (q[16])); + asm volatile("vld $vr10, %0" : : "m" (q[32])); + asm volatile("vld $vr11, %0" : : "m" (q[48])); + asm volatile("vxor.v $vr4, $vr4, $vr8"); + asm volatile("vxor.v $vr5, $vr5, $vr9"); + asm volatile("vxor.v $vr6, $vr6, $vr10"); + asm volatile("vxor.v $vr7, $vr7, $vr11"); + + /* vr8 - vr11: higher 4 bits of each byte of (Q + Qx) */ + asm volatile("vsrli.b $vr8, $vr4, 4"); + asm volatile("vsrli.b $vr9, $vr5, 4"); + asm volatile("vsrli.b $vr10, $vr6, 4"); + asm volatile("vsrli.b $vr11, $vr7, 4"); + /* vr4 - vr7: lower 4 bits of each byte of (Q + Qx) */ + asm volatile("vandi.b $vr4, $vr4, 0x0f"); + asm volatile("vandi.b $vr5, $vr5, 0x0f"); + asm volatile("vandi.b $vr6, $vr6, 0x0f"); + asm volatile("vandi.b $vr7, $vr7, 0x0f"); + /* lookup from qmul[0] */ + asm volatile("vshuf.b $vr4, $vr22, $vr22, $vr4"); + asm volatile("vshuf.b $vr5, $vr22, $vr22, $vr5"); + asm volatile("vshuf.b $vr6, $vr22, $vr22, $vr6"); + asm volatile("vshuf.b $vr7, $vr22, $vr22, $vr7"); + /* lookup from qmul[16] */ + asm volatile("vshuf.b $vr8, $vr23, $vr23, $vr8"); + asm volatile("vshuf.b $vr9, $vr23, $vr23, $vr9"); + asm volatile("vshuf.b $vr10, $vr23, $vr23, $vr10"); + asm volatile("vshuf.b $vr11, $vr23, $vr23, $vr11"); + /* vr4 - vr7: qmul(Q + Qx) = Dx */ + asm volatile("vxor.v $vr4, $vr4, $vr8"); + asm volatile("vxor.v $vr5, $vr5, $vr9"); + asm volatile("vxor.v $vr6, $vr6, $vr10"); + asm volatile("vxor.v $vr7, $vr7, $vr11"); + asm volatile("vst $vr4, %0" : "=m" (dq[0])); + asm volatile("vst $vr5, %0" : "=m" (dq[16])); + asm volatile("vst $vr6, %0" : "=m" (dq[32])); + asm volatile("vst $vr7, %0" : "=m" (dq[48])); + + /* vr0 - vr3: P + Dx + Dx = P */ + asm volatile("vxor.v $vr0, $vr0, $vr4"); + asm volatile("vxor.v $vr1, $vr1, $vr5"); + asm volatile("vxor.v $vr2, $vr2, $vr6"); + asm volatile("vxor.v $vr3, $vr3, $vr7"); + asm volatile("vst $vr0, %0" : "=m" (p[0])); + asm volatile("vst $vr1, %0" : "=m" (p[16])); + asm volatile("vst $vr2, %0" : "=m" (p[32])); + asm volatile("vst $vr3, %0" : "=m" (p[48])); + + bytes -= 64; + p += 64; + q += 64; + dq += 64; + } + + kernel_fpu_end(); +} + +const struct raid6_recov_calls raid6_recov_lsx = { + .data2 = raid6_2data_recov_lsx, + .datap = raid6_datap_recov_lsx, + .valid = raid6_has_lsx, + .name = "lsx", + .priority = 1, +}; +#endif /* CONFIG_CPU_HAS_LSX */ + +#ifdef CONFIG_CPU_HAS_LASX +static int raid6_has_lasx(void) +{ + return cpu_has_lasx; +} + +static void raid6_2data_recov_lasx(int disks, size_t bytes, int faila, + int failb, void **ptrs) +{ + u8 *p, *q, *dp, *dq; + const u8 *pbmul; /* P multiplier table for B data */ + const u8 *qmul; /* Q multiplier table (for both) */ + + p = (u8 *)ptrs[disks - 2]; + q = (u8 *)ptrs[disks - 1]; + + /* + * Compute syndrome with zero for the missing data pages + * Use the dead data pages as temporary storage for + * delta p and delta q + */ + dp = (u8 *)ptrs[faila]; + ptrs[faila] = (void *)raid6_empty_zero_page; + ptrs[disks - 2] = dp; + dq = (u8 *)ptrs[failb]; + ptrs[failb] = (void *)raid6_empty_zero_page; + ptrs[disks - 1] = dq; + + raid6_call.gen_syndrome(disks, bytes, ptrs); + + /* Restore pointer table */ + ptrs[faila] = dp; + ptrs[failb] = dq; + ptrs[disks - 2] = p; + ptrs[disks - 1] = q; + + /* Now, pick the proper data tables */ + pbmul = raid6_vgfmul[raid6_gfexi[failb - faila]]; + qmul = raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila] ^ raid6_gfexp[failb]]]; + + kernel_fpu_begin(); + + /* + * xr20, xr21: qmul + * xr22, xr23: pbmul + */ + asm volatile("vld $vr20, %0" : : "m" (qmul[0])); + asm volatile("vld $vr21, %0" : : "m" (qmul[16])); + asm volatile("vld $vr22, %0" : : "m" (pbmul[0])); + asm volatile("vld $vr23, %0" : : "m" (pbmul[16])); + asm volatile("xvreplve0.q $xr20, $xr20"); + asm volatile("xvreplve0.q $xr21, $xr21"); + asm volatile("xvreplve0.q $xr22, $xr22"); + asm volatile("xvreplve0.q $xr23, $xr23"); + + while (bytes) { + /* xr0, xr1: Q */ + asm volatile("xvld $xr0, %0" : : "m" (q[0])); + asm volatile("xvld $xr1, %0" : : "m" (q[32])); + /* xr0, xr1: Q + Qxy */ + asm volatile("xvld $xr4, %0" : : "m" (dq[0])); + asm volatile("xvld $xr5, %0" : : "m" (dq[32])); + asm volatile("xvxor.v $xr0, $xr0, $xr4"); + asm volatile("xvxor.v $xr1, $xr1, $xr5"); + /* xr2, xr3: P */ + asm volatile("xvld $xr2, %0" : : "m" (p[0])); + asm volatile("xvld $xr3, %0" : : "m" (p[32])); + /* xr2, xr3: P + Pxy */ + asm volatile("xvld $xr4, %0" : : "m" (dp[0])); + asm volatile("xvld $xr5, %0" : : "m" (dp[32])); + asm volatile("xvxor.v $xr2, $xr2, $xr4"); + asm volatile("xvxor.v $xr3, $xr3, $xr5"); + + /* xr4, xr5: higher 4 bits of each byte of (Q + Qxy) */ + asm volatile("xvsrli.b $xr4, $xr0, 4"); + asm volatile("xvsrli.b $xr5, $xr1, 4"); + /* xr0, xr1: lower 4 bits of each byte of (Q + Qxy) */ + asm volatile("xvandi.b $xr0, $xr0, 0x0f"); + asm volatile("xvandi.b $xr1, $xr1, 0x0f"); + /* lookup from qmul[0] */ + asm volatile("xvshuf.b $xr0, $xr20, $xr20, $xr0"); + asm volatile("xvshuf.b $xr1, $xr20, $xr20, $xr1"); + /* lookup from qmul[16] */ + asm volatile("xvshuf.b $xr4, $xr21, $xr21, $xr4"); + asm volatile("xvshuf.b $xr5, $xr21, $xr21, $xr5"); + /* xr6, xr7: B(Q + Qxy) */ + asm volatile("xvxor.v $xr6, $xr4, $xr0"); + asm volatile("xvxor.v $xr7, $xr5, $xr1"); + + /* xr4, xr5: higher 4 bits of each byte of (P + Pxy) */ + asm volatile("xvsrli.b $xr4, $xr2, 4"); + asm volatile("xvsrli.b $xr5, $xr3, 4"); + /* xr0, xr1: lower 4 bits of each byte of (P + Pxy) */ + asm volatile("xvandi.b $xr0, $xr2, 0x0f"); + asm volatile("xvandi.b $xr1, $xr3, 0x0f"); + /* lookup from pbmul[0] */ + asm volatile("xvshuf.b $xr0, $xr22, $xr22, $xr0"); + asm volatile("xvshuf.b $xr1, $xr22, $xr22, $xr1"); + /* lookup from pbmul[16] */ + asm volatile("xvshuf.b $xr4, $xr23, $xr23, $xr4"); + asm volatile("xvshuf.b $xr5, $xr23, $xr23, $xr5"); + /* xr0, xr1: A(P + Pxy) */ + asm volatile("xvxor.v $xr0, $xr0, $xr4"); + asm volatile("xvxor.v $xr1, $xr1, $xr5"); + + /* xr0, xr1: A(P + Pxy) + B(Q + Qxy) = Dx */ + asm volatile("xvxor.v $xr0, $xr0, $xr6"); + asm volatile("xvxor.v $xr1, $xr1, $xr7"); + + /* xr2, xr3: P + Pxy + Dx = Dy */ + asm volatile("xvxor.v $xr2, $xr2, $xr0"); + asm volatile("xvxor.v $xr3, $xr3, $xr1"); + + asm volatile("xvst $xr0, %0" : "=m" (dq[0])); + asm volatile("xvst $xr1, %0" : "=m" (dq[32])); + asm volatile("xvst $xr2, %0" : "=m" (dp[0])); + asm volatile("xvst $xr3, %0" : "=m" (dp[32])); + + bytes -= 64; + p += 64; + q += 64; + dp += 64; + dq += 64; + } + + kernel_fpu_end(); +} + +static void raid6_datap_recov_lasx(int disks, size_t bytes, int faila, + void **ptrs) +{ + u8 *p, *q, *dq; + const u8 *qmul; /* Q multiplier table */ + + p = (u8 *)ptrs[disks - 2]; + q = (u8 *)ptrs[disks - 1]; + + /* + * Compute syndrome with zero for the missing data page + * Use the dead data page as temporary storage for delta q + */ + dq = (u8 *)ptrs[faila]; + ptrs[faila] = (void *)raid6_empty_zero_page; + ptrs[disks - 1] = dq; + + raid6_call.gen_syndrome(disks, bytes, ptrs); + + /* Restore pointer table */ + ptrs[faila] = dq; + ptrs[disks - 1] = q; + + /* Now, pick the proper data tables */ + qmul = raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila]]]; + + kernel_fpu_begin(); + + /* xr22, xr23: qmul */ + asm volatile("vld $vr22, %0" : : "m" (qmul[0])); + asm volatile("xvreplve0.q $xr22, $xr22"); + asm volatile("vld $vr23, %0" : : "m" (qmul[16])); + asm volatile("xvreplve0.q $xr23, $xr23"); + + while (bytes) { + /* xr0, xr1: P + Dx */ + asm volatile("xvld $xr0, %0" : : "m" (p[0])); + asm volatile("xvld $xr1, %0" : : "m" (p[32])); + /* xr2, xr3: Qx */ + asm volatile("xvld $xr2, %0" : : "m" (dq[0])); + asm volatile("xvld $xr3, %0" : : "m" (dq[32])); + /* xr2, xr3: Q + Qx */ + asm volatile("xvld $xr4, %0" : : "m" (q[0])); + asm volatile("xvld $xr5, %0" : : "m" (q[32])); + asm volatile("xvxor.v $xr2, $xr2, $xr4"); + asm volatile("xvxor.v $xr3, $xr3, $xr5"); + + /* xr4, xr5: higher 4 bits of each byte of (Q + Qx) */ + asm volatile("xvsrli.b $xr4, $xr2, 4"); + asm volatile("xvsrli.b $xr5, $xr3, 4"); + /* xr2, xr3: lower 4 bits of each byte of (Q + Qx) */ + asm volatile("xvandi.b $xr2, $xr2, 0x0f"); + asm volatile("xvandi.b $xr3, $xr3, 0x0f"); + /* lookup from qmul[0] */ + asm volatile("xvshuf.b $xr2, $xr22, $xr22, $xr2"); + asm volatile("xvshuf.b $xr3, $xr22, $xr22, $xr3"); + /* lookup from qmul[16] */ + asm volatile("xvshuf.b $xr4, $xr23, $xr23, $xr4"); + asm volatile("xvshuf.b $xr5, $xr23, $xr23, $xr5"); + /* xr2, xr3: qmul(Q + Qx) = Dx */ + asm volatile("xvxor.v $xr2, $xr2, $xr4"); + asm volatile("xvxor.v $xr3, $xr3, $xr5"); + + /* xr0, xr1: P + Dx + Dx = P */ + asm volatile("xvxor.v $xr0, $xr0, $xr2"); + asm volatile("xvxor.v $xr1, $xr1, $xr3"); + + asm volatile("xvst $xr2, %0" : "=m" (dq[0])); + asm volatile("xvst $xr3, %0" : "=m" (dq[32])); + asm volatile("xvst $xr0, %0" : "=m" (p[0])); + asm volatile("xvst $xr1, %0" : "=m" (p[32])); + + bytes -= 64; + p += 64; + q += 64; + dq += 64; + } + + kernel_fpu_end(); +} + +const struct raid6_recov_calls raid6_recov_lasx = { + .data2 = raid6_2data_recov_lasx, + .datap = raid6_datap_recov_lasx, + .valid = raid6_has_lasx, + .name = "lasx", + .priority = 2, +}; +#endif /* CONFIG_CPU_HAS_LASX */ diff --git a/lib/raid6/test/Makefile b/lib/raid6/test/Makefile index 1f693ea3b980..2abe0076a636 100644 --- a/lib/raid6/test/Makefile +++ b/lib/raid6/test/Makefile @@ -41,6 +41,16 @@ ifeq ($(findstring ppc,$(ARCH)),ppc) gcc -c -x c - >/dev/null && rm ./-.o && echo yes) endif +ifeq ($(ARCH),loongarch64) + CFLAGS += -I../../../arch/loongarch/include -DCONFIG_LOONGARCH=1 + CFLAGS += $(shell echo 'vld $$vr0, $$zero, 0' | \ + gcc -c -x assembler - >/dev/null 2>&1 && \ + rm ./-.o && echo -DCONFIG_CPU_HAS_LSX=1) + CFLAGS += $(shell echo 'xvld $$xr0, $$zero, 0' | \ + gcc -c -x assembler - >/dev/null 2>&1 && \ + rm ./-.o && echo -DCONFIG_CPU_HAS_LASX=1) +endif + ifeq ($(IS_X86),yes) OBJS += mmx.o sse1.o sse2.o avx2.o recov_ssse3.o recov_avx2.o avx512.o recov_avx512.o CFLAGS += -DCONFIG_X86 @@ -54,6 +64,8 @@ else ifeq ($(HAS_ALTIVEC),yes) CFLAGS += -DCONFIG_ALTIVEC OBJS += altivec1.o altivec2.o altivec4.o altivec8.o \ vpermxor1.o vpermxor2.o vpermxor4.o vpermxor8.o +else ifeq ($(ARCH),loongarch64) + OBJS += loongarch_simd.o recov_loongarch_simd.o endif .c.o: diff --git a/lib/xarray.c b/lib/xarray.c index 2071a3718f4e..39f07bfc4dcc 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -206,7 +206,7 @@ static void *xas_descend(struct xa_state *xas, struct xa_node *node) void *entry = xa_entry(xas->xa, node, offset); xas->xa_node = node; - if (xa_is_sibling(entry)) { + while (xa_is_sibling(entry)) { offset = xa_to_sibling(entry); entry = xa_entry(xas->xa, node, offset); if (node->shift && xa_is_node(entry)) @@ -1802,6 +1802,9 @@ EXPORT_SYMBOL(xa_get_order); * stores the index into the @id pointer, then stores the entry at * that index. A concurrent lookup will not see an uninitialised @id. * + * Must only be operated on an xarray initialized with flag XA_FLAGS_ALLOC set + * in xa_init_flags(). + * * Context: Any context. Expects xa_lock to be held on entry. May * release and reacquire xa_lock if @gfp flags permit. * Return: 0 on success, -ENOMEM if memory could not be allocated or @@ -1850,6 +1853,9 @@ EXPORT_SYMBOL(__xa_alloc); * The search for an empty entry will start at @next and will wrap * around if necessary. * + * Must only be operated on an xarray initialized with flag XA_FLAGS_ALLOC set + * in xa_init_flags(). + * * Context: Any context. Expects xa_lock to be held on entry. May * release and reacquire xa_lock if @gfp flags permit. * Return: 0 if the allocation succeeded without wrapping. 1 if the diff --git a/mm/kasan/init.c b/mm/kasan/init.c index dcfec277e839..89895f38f722 100644 --- a/mm/kasan/init.c +++ b/mm/kasan/init.c @@ -139,6 +139,10 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr, return 0; } +void __weak __meminit pmd_init(void *addr) +{ +} + static int __ref zero_pud_populate(p4d_t *p4d, unsigned long addr, unsigned long end) { @@ -166,8 +170,9 @@ static int __ref zero_pud_populate(p4d_t *p4d, unsigned long addr, if (!p) return -ENOMEM; } else { - pud_populate(&init_mm, pud, - early_alloc(PAGE_SIZE, NUMA_NO_NODE)); + p = early_alloc(PAGE_SIZE, NUMA_NO_NODE); + pmd_init(p); + pud_populate(&init_mm, pud, p); } } zero_pmd_populate(pud, addr, next); @@ -176,6 +181,10 @@ static int __ref zero_pud_populate(p4d_t *p4d, unsigned long addr, return 0; } +void __weak __meminit pud_init(void *addr) +{ +} + static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr, unsigned long end) { @@ -207,8 +216,9 @@ static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr, if (!p) return -ENOMEM; } else { - p4d_populate(&init_mm, p4d, - early_alloc(PAGE_SIZE, NUMA_NO_NODE)); + p = early_alloc(PAGE_SIZE, NUMA_NO_NODE); + pud_init(p); + p4d_populate(&init_mm, p4d, p); } } zero_pud_populate(p4d, addr, next); diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 2e973b36fe07..f70e3d7a602e 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -291,16 +291,22 @@ struct kasan_stack_ring { #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) +#ifndef __HAVE_ARCH_SHADOW_MAP static inline const void *kasan_shadow_to_mem(const void *shadow_addr) { return (void *)(((unsigned long)shadow_addr - KASAN_SHADOW_OFFSET) << KASAN_SHADOW_SCALE_SHIFT); } +#endif static __always_inline bool addr_has_metadata(const void *addr) { +#ifdef __HAVE_ARCH_SHADOW_MAP + return (kasan_mem_to_shadow((void *)addr) != NULL); +#else return (kasan_reset_tag(addr) >= kasan_shadow_to_mem((void *)KASAN_SHADOW_START)); +#endif } /** diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 96fd0411f5c5..3872528d0963 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -574,13 +574,14 @@ static void rcu_guarded_free(struct rcu_head *h) */ static unsigned long kfence_init_pool(void) { - unsigned long addr = (unsigned long)__kfence_pool; + unsigned long addr; struct page *pages; int i; if (!arch_kfence_init_pool()) - return addr; + return (unsigned long)__kfence_pool; + addr = (unsigned long)__kfence_pool; pages = virt_to_page(__kfence_pool); /* diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 57a7a64b84ed..0841f8d82419 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -543,6 +543,7 @@ struct bpf_fentry_test_t { int noinline bpf_fentry_test7(struct bpf_fentry_test_t *arg) { + asm volatile (""); return (long)arg; } diff --git a/net/can/j1939/socket.c b/net/can/j1939/socket.c index feaec4ad6d16..b28c976f52a0 100644 --- a/net/can/j1939/socket.c +++ b/net/can/j1939/socket.c @@ -974,6 +974,7 @@ static void __j1939_sk_errqueue(struct j1939_session *session, struct sock *sk, struct sock_exterr_skb *serr; struct sk_buff *skb; char *state = "UNK"; + u32 tsflags; int err; jsk = j1939_sk(sk); @@ -981,13 +982,14 @@ static void __j1939_sk_errqueue(struct j1939_session *session, struct sock *sk, if (!(jsk->state & J1939_SOCK_ERRQUEUE)) return; + tsflags = READ_ONCE(sk->sk_tsflags); switch (type) { case J1939_ERRQUEUE_TX_ACK: - if (!(sk->sk_tsflags & SOF_TIMESTAMPING_TX_ACK)) + if (!(tsflags & SOF_TIMESTAMPING_TX_ACK)) return; break; case J1939_ERRQUEUE_TX_SCHED: - if (!(sk->sk_tsflags & SOF_TIMESTAMPING_TX_SCHED)) + if (!(tsflags & SOF_TIMESTAMPING_TX_SCHED)) return; break; case J1939_ERRQUEUE_TX_ABORT: @@ -997,7 +999,7 @@ static void __j1939_sk_errqueue(struct j1939_session *session, struct sock *sk, case J1939_ERRQUEUE_RX_DPO: fallthrough; case J1939_ERRQUEUE_RX_ABORT: - if (!(sk->sk_tsflags & SOF_TIMESTAMPING_RX_SOFTWARE)) + if (!(tsflags & SOF_TIMESTAMPING_RX_SOFTWARE)) return; break; default: @@ -1054,7 +1056,7 @@ static void __j1939_sk_errqueue(struct j1939_session *session, struct sock *sk, } serr->opt_stats = true; - if (sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID) + if (tsflags & SOF_TIMESTAMPING_OPT_ID) serr->ee.ee_data = session->tskey; netdev_dbg(session->priv->ndev, "%s: 0x%p tskey: %i, state: %s\n", diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c index 5eb4898cccd4..10a41cd9c523 100644 --- a/net/ceph/messenger.c +++ b/net/ceph/messenger.c @@ -969,6 +969,62 @@ static bool ceph_msg_data_pagelist_advance(struct ceph_msg_data_cursor *cursor, return true; } +static void ceph_msg_data_iter_cursor_init(struct ceph_msg_data_cursor *cursor, + size_t length) +{ + struct ceph_msg_data *data = cursor->data; + + cursor->iov_iter = data->iter; + cursor->lastlen = 0; + iov_iter_truncate(&cursor->iov_iter, length); + cursor->resid = iov_iter_count(&cursor->iov_iter); +} + +static struct page *ceph_msg_data_iter_next(struct ceph_msg_data_cursor *cursor, + size_t *page_offset, size_t *length) +{ + struct page *page; + ssize_t len; + + if (cursor->lastlen) + iov_iter_revert(&cursor->iov_iter, cursor->lastlen); + + len = iov_iter_get_pages2(&cursor->iov_iter, &page, PAGE_SIZE, + 1, page_offset); + BUG_ON(len < 0); + + cursor->lastlen = len; + + /* + * FIXME: The assumption is that the pages represented by the iov_iter + * are pinned, with the references held by the upper-level + * callers, or by virtue of being under writeback. Eventually, + * we'll get an iov_iter_get_pages2 variant that doesn't take + * page refs. Until then, just put the page ref. + */ + VM_BUG_ON_PAGE(!PageWriteback(page) && page_count(page) < 2, page); + put_page(page); + + *length = min_t(size_t, len, cursor->resid); + return page; +} + +static bool ceph_msg_data_iter_advance(struct ceph_msg_data_cursor *cursor, + size_t bytes) +{ + BUG_ON(bytes > cursor->resid); + cursor->resid -= bytes; + + if (bytes < cursor->lastlen) { + cursor->lastlen -= bytes; + } else { + iov_iter_advance(&cursor->iov_iter, bytes - cursor->lastlen); + cursor->lastlen = 0; + } + + return cursor->resid; +} + /* * Message data is handled (sent or received) in pieces, where each * piece resides on a single page. The network layer might not @@ -996,6 +1052,9 @@ static void __ceph_msg_data_cursor_init(struct ceph_msg_data_cursor *cursor) case CEPH_MSG_DATA_BVECS: ceph_msg_data_bvecs_cursor_init(cursor, length); break; + case CEPH_MSG_DATA_ITER: + ceph_msg_data_iter_cursor_init(cursor, length); + break; case CEPH_MSG_DATA_NONE: default: /* BUG(); */ @@ -1013,6 +1072,7 @@ void ceph_msg_data_cursor_init(struct ceph_msg_data_cursor *cursor, cursor->total_resid = length; cursor->data = msg->data; + cursor->sr_resid = 0; __ceph_msg_data_cursor_init(cursor); } @@ -1042,6 +1102,9 @@ struct page *ceph_msg_data_next(struct ceph_msg_data_cursor *cursor, case CEPH_MSG_DATA_BVECS: page = ceph_msg_data_bvecs_next(cursor, page_offset, length); break; + case CEPH_MSG_DATA_ITER: + page = ceph_msg_data_iter_next(cursor, page_offset, length); + break; case CEPH_MSG_DATA_NONE: default: page = NULL; @@ -1080,6 +1143,9 @@ void ceph_msg_data_advance(struct ceph_msg_data_cursor *cursor, size_t bytes) case CEPH_MSG_DATA_BVECS: new_piece = ceph_msg_data_bvecs_advance(cursor, bytes); break; + case CEPH_MSG_DATA_ITER: + new_piece = ceph_msg_data_iter_advance(cursor, bytes); + break; case CEPH_MSG_DATA_NONE: default: BUG(); @@ -1879,6 +1945,18 @@ void ceph_msg_data_add_bvecs(struct ceph_msg *msg, } EXPORT_SYMBOL(ceph_msg_data_add_bvecs); +void ceph_msg_data_add_iter(struct ceph_msg *msg, + struct iov_iter *iter) +{ + struct ceph_msg_data *data; + + data = ceph_msg_data_add(msg); + data->type = CEPH_MSG_DATA_ITER; + data->iter = *iter; + + msg->data_length += iov_iter_count(&data->iter); +} + /* * construct a new message with given type, size * the new msg has a ref count of 1. diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c index 3d57bb48a2b4..f9a50d7f0d20 100644 --- a/net/ceph/messenger_v1.c +++ b/net/ceph/messenger_v1.c @@ -159,9 +159,9 @@ static size_t sizeof_footer(struct ceph_connection *con) static void prepare_message_data(struct ceph_msg *msg, u32 data_len) { - /* Initialize data cursor */ - - ceph_msg_data_cursor_init(&msg->cursor, msg, data_len); + /* Initialize data cursor if it's not a sparse read */ + if (!msg->sparse_read) + ceph_msg_data_cursor_init(&msg->cursor, msg, data_len); } /* @@ -960,9 +960,9 @@ static void process_ack(struct ceph_connection *con) prepare_read_tag(con); } -static int read_partial_message_section(struct ceph_connection *con, - struct kvec *section, - unsigned int sec_len, u32 *crc) +static int read_partial_message_chunk(struct ceph_connection *con, + struct kvec *section, + unsigned int sec_len, u32 *crc) { int ret, left; @@ -978,11 +978,91 @@ static int read_partial_message_section(struct ceph_connection *con, section->iov_len += ret; } if (section->iov_len == sec_len) - *crc = crc32c(0, section->iov_base, section->iov_len); + *crc = crc32c(*crc, section->iov_base, section->iov_len); return 1; } +static inline int read_partial_message_section(struct ceph_connection *con, + struct kvec *section, + unsigned int sec_len, u32 *crc) +{ + *crc = 0; + return read_partial_message_chunk(con, section, sec_len, crc); +} + +static int read_sparse_msg_extent(struct ceph_connection *con, u32 *crc) +{ + struct ceph_msg_data_cursor *cursor = &con->in_msg->cursor; + bool do_bounce = ceph_test_opt(from_msgr(con->msgr), RXBOUNCE); + + if (do_bounce && unlikely(!con->bounce_page)) { + con->bounce_page = alloc_page(GFP_NOIO); + if (!con->bounce_page) { + pr_err("failed to allocate bounce page\n"); + return -ENOMEM; + } + } + + while (cursor->sr_resid > 0) { + struct page *page, *rpage; + size_t off, len; + int ret; + + page = ceph_msg_data_next(cursor, &off, &len); + rpage = do_bounce ? con->bounce_page : page; + + /* clamp to what remains in extent */ + len = min_t(int, len, cursor->sr_resid); + ret = ceph_tcp_recvpage(con->sock, rpage, (int)off, len); + if (ret <= 0) + return ret; + *crc = ceph_crc32c_page(*crc, rpage, off, ret); + ceph_msg_data_advance(cursor, (size_t)ret); + cursor->sr_resid -= ret; + if (do_bounce) + memcpy_page(page, off, rpage, off, ret); + } + return 1; +} + +static int read_sparse_msg_data(struct ceph_connection *con) +{ + struct ceph_msg_data_cursor *cursor = &con->in_msg->cursor; + bool do_datacrc = !ceph_test_opt(from_msgr(con->msgr), NOCRC); + u32 crc = 0; + int ret = 1; + + if (do_datacrc) + crc = con->in_data_crc; + + do { + if (con->v1.in_sr_kvec.iov_base) + ret = read_partial_message_chunk(con, + &con->v1.in_sr_kvec, + con->v1.in_sr_len, + &crc); + else if (cursor->sr_resid > 0) + ret = read_sparse_msg_extent(con, &crc); + + if (ret <= 0) { + if (do_datacrc) + con->in_data_crc = crc; + return ret; + } + + memset(&con->v1.in_sr_kvec, 0, sizeof(con->v1.in_sr_kvec)); + ret = con->ops->sparse_read(con, cursor, + (char **)&con->v1.in_sr_kvec.iov_base); + con->v1.in_sr_len = ret; + } while (ret > 0); + + if (do_datacrc) + con->in_data_crc = crc; + + return ret < 0 ? ret : 1; /* must return > 0 to indicate success */ +} + static int read_partial_msg_data(struct ceph_connection *con) { struct ceph_msg_data_cursor *cursor = &con->in_msg->cursor; @@ -1173,7 +1253,9 @@ static int read_partial_message(struct ceph_connection *con) if (!m->num_data_items) return -EIO; - if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) + if (m->sparse_read) + ret = read_sparse_msg_data(con); + else if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) ret = read_partial_msg_data_bounce(con); else ret = read_partial_msg_data(con); diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c index 1df1d29dee92..d09a39ff2cf0 100644 --- a/net/ceph/messenger_v2.c +++ b/net/ceph/messenger_v2.c @@ -8,9 +8,9 @@ #include #include -#include /* for crypto_memneq() */ #include #include +#include #include #include #include @@ -52,14 +52,16 @@ #define FRAME_LATE_STATUS_COMPLETE 0xe #define FRAME_LATE_STATUS_ABORTED_MASK 0xf -#define IN_S_HANDLE_PREAMBLE 1 -#define IN_S_HANDLE_CONTROL 2 -#define IN_S_HANDLE_CONTROL_REMAINDER 3 -#define IN_S_PREPARE_READ_DATA 4 -#define IN_S_PREPARE_READ_DATA_CONT 5 -#define IN_S_PREPARE_READ_ENC_PAGE 6 -#define IN_S_HANDLE_EPILOGUE 7 -#define IN_S_FINISH_SKIP 8 +#define IN_S_HANDLE_PREAMBLE 1 +#define IN_S_HANDLE_CONTROL 2 +#define IN_S_HANDLE_CONTROL_REMAINDER 3 +#define IN_S_PREPARE_READ_DATA 4 +#define IN_S_PREPARE_READ_DATA_CONT 5 +#define IN_S_PREPARE_READ_ENC_PAGE 6 +#define IN_S_PREPARE_SPARSE_DATA 7 +#define IN_S_PREPARE_SPARSE_DATA_CONT 8 +#define IN_S_HANDLE_EPILOGUE 9 +#define IN_S_FINISH_SKIP 10 #define OUT_S_QUEUE_DATA 1 #define OUT_S_QUEUE_DATA_CONT 2 @@ -967,12 +969,48 @@ static void init_sgs_cursor(struct scatterlist **sg, } } +/** + * init_sgs_pages: set up scatterlist on an array of page pointers + * @sg: scatterlist to populate + * @pages: pointer to page array + * @dpos: position in the array to start (bytes) + * @dlen: len to add to sg (bytes) + * @pad: pointer to pad destination (if any) + * + * Populate the scatterlist from the page array, starting at an arbitrary + * byte in the array and running for a specified length. + */ +static void init_sgs_pages(struct scatterlist **sg, struct page **pages, + int dpos, int dlen, u8 *pad) +{ + int idx = dpos >> PAGE_SHIFT; + int off = offset_in_page(dpos); + int resid = dlen; + + do { + int len = min(resid, (int)PAGE_SIZE - off); + + sg_set_page(*sg, pages[idx], len, off); + *sg = sg_next(*sg); + off = 0; + ++idx; + resid -= len; + } while (resid); + + if (need_padding(dlen)) { + sg_set_buf(*sg, pad, padding_len(dlen)); + *sg = sg_next(*sg); + } +} + static int setup_message_sgs(struct sg_table *sgt, struct ceph_msg *msg, u8 *front_pad, u8 *middle_pad, u8 *data_pad, - void *epilogue, bool add_tag) + void *epilogue, struct page **pages, int dpos, + bool add_tag) { struct ceph_msg_data_cursor cursor; struct scatterlist *cur_sg; + int dlen = data_len(msg); int sg_cnt; int ret; @@ -986,9 +1024,15 @@ static int setup_message_sgs(struct sg_table *sgt, struct ceph_msg *msg, if (middle_len(msg)) sg_cnt += calc_sg_cnt(msg->middle->vec.iov_base, middle_len(msg)); - if (data_len(msg)) { - ceph_msg_data_cursor_init(&cursor, msg, data_len(msg)); - sg_cnt += calc_sg_cnt_cursor(&cursor); + if (dlen) { + if (pages) { + sg_cnt += calc_pages_for(dpos, dlen); + if (need_padding(dlen)) + sg_cnt++; + } else { + ceph_msg_data_cursor_init(&cursor, msg, dlen); + sg_cnt += calc_sg_cnt_cursor(&cursor); + } } ret = sg_alloc_table(sgt, sg_cnt, GFP_NOIO); @@ -1002,9 +1046,13 @@ static int setup_message_sgs(struct sg_table *sgt, struct ceph_msg *msg, if (middle_len(msg)) init_sgs(&cur_sg, msg->middle->vec.iov_base, middle_len(msg), middle_pad); - if (data_len(msg)) { - ceph_msg_data_cursor_init(&cursor, msg, data_len(msg)); - init_sgs_cursor(&cur_sg, &cursor, data_pad); + if (dlen) { + if (pages) { + init_sgs_pages(&cur_sg, pages, dpos, dlen, data_pad); + } else { + ceph_msg_data_cursor_init(&cursor, msg, dlen); + init_sgs_cursor(&cur_sg, &cursor, data_pad); + } } WARN_ON(!sg_is_last(cur_sg)); @@ -1039,10 +1087,53 @@ static int decrypt_control_remainder(struct ceph_connection *con) padded_len(rem_len) + CEPH_GCM_TAG_LEN); } +/* Process sparse read data that lives in a buffer */ +static int process_v2_sparse_read(struct ceph_connection *con, + struct page **pages, int spos) +{ + struct ceph_msg_data_cursor *cursor = &con->v2.in_cursor; + int ret; + + for (;;) { + char *buf = NULL; + + ret = con->ops->sparse_read(con, cursor, &buf); + if (ret <= 0) + return ret; + + dout("%s: sparse_read return %x buf %p\n", __func__, ret, buf); + + do { + int idx = spos >> PAGE_SHIFT; + int soff = offset_in_page(spos); + struct page *spage = con->v2.in_enc_pages[idx]; + int len = min_t(int, ret, PAGE_SIZE - soff); + + if (buf) { + memcpy_from_page(buf, spage, soff, len); + buf += len; + } else { + struct bio_vec bv; + + get_bvec_at(cursor, &bv); + len = min_t(int, len, bv.bv_len); + memcpy_page(bv.bv_page, bv.bv_offset, + spage, soff, len); + ceph_msg_data_advance(cursor, len); + } + spos += len; + ret -= len; + } while (ret); + } +} + static int decrypt_tail(struct ceph_connection *con) { struct sg_table enc_sgt = {}; struct sg_table sgt = {}; + struct page **pages = NULL; + bool sparse = con->in_msg->sparse_read; + int dpos = 0; int tail_len; int ret; @@ -1053,9 +1144,14 @@ static int decrypt_tail(struct ceph_connection *con) if (ret) goto out; + if (sparse) { + dpos = padded_len(front_len(con->in_msg) + padded_len(middle_len(con->in_msg))); + pages = con->v2.in_enc_pages; + } + ret = setup_message_sgs(&sgt, con->in_msg, FRONT_PAD(con->v2.in_buf), - MIDDLE_PAD(con->v2.in_buf), DATA_PAD(con->v2.in_buf), - con->v2.in_buf, true); + MIDDLE_PAD(con->v2.in_buf), DATA_PAD(con->v2.in_buf), + con->v2.in_buf, pages, dpos, true); if (ret) goto out; @@ -1065,6 +1161,12 @@ static int decrypt_tail(struct ceph_connection *con) if (ret) goto out; + if (sparse && data_len(con->in_msg)) { + ret = process_v2_sparse_read(con, con->v2.in_enc_pages, dpos); + if (ret) + goto out; + } + WARN_ON(!con->v2.in_enc_page_cnt); ceph_release_page_vector(con->v2.in_enc_pages, con->v2.in_enc_page_cnt); @@ -1588,7 +1690,7 @@ static int prepare_message_secure(struct ceph_connection *con) encode_epilogue_secure(con, false); ret = setup_message_sgs(&sgt, con->out_msg, zerop, zerop, zerop, - &con->v2.out_epil, false); + &con->v2.out_epil, NULL, 0, false); if (ret) goto out; @@ -1825,6 +1927,123 @@ static void prepare_read_data_cont(struct ceph_connection *con) con->v2.in_state = IN_S_HANDLE_EPILOGUE; } +static int prepare_sparse_read_cont(struct ceph_connection *con) +{ + int ret; + struct bio_vec bv; + char *buf = NULL; + struct ceph_msg_data_cursor *cursor = &con->v2.in_cursor; + + WARN_ON(con->v2.in_state != IN_S_PREPARE_SPARSE_DATA_CONT); + + if (iov_iter_is_bvec(&con->v2.in_iter)) { + if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) { + con->in_data_crc = crc32c(con->in_data_crc, + page_address(con->bounce_page), + con->v2.in_bvec.bv_len); + get_bvec_at(cursor, &bv); + memcpy_to_page(bv.bv_page, bv.bv_offset, + page_address(con->bounce_page), + con->v2.in_bvec.bv_len); + } else { + con->in_data_crc = ceph_crc32c_page(con->in_data_crc, + con->v2.in_bvec.bv_page, + con->v2.in_bvec.bv_offset, + con->v2.in_bvec.bv_len); + } + + ceph_msg_data_advance(cursor, con->v2.in_bvec.bv_len); + cursor->sr_resid -= con->v2.in_bvec.bv_len; + dout("%s: advance by 0x%x sr_resid 0x%x\n", __func__, + con->v2.in_bvec.bv_len, cursor->sr_resid); + WARN_ON_ONCE(cursor->sr_resid > cursor->total_resid); + if (cursor->sr_resid) { + get_bvec_at(cursor, &bv); + if (bv.bv_len > cursor->sr_resid) + bv.bv_len = cursor->sr_resid; + if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) { + bv.bv_page = con->bounce_page; + bv.bv_offset = 0; + } + set_in_bvec(con, &bv); + con->v2.data_len_remain -= bv.bv_len; + return 0; + } + } else if (iov_iter_is_kvec(&con->v2.in_iter)) { + /* On first call, we have no kvec so don't compute crc */ + if (con->v2.in_kvec_cnt) { + WARN_ON_ONCE(con->v2.in_kvec_cnt > 1); + con->in_data_crc = crc32c(con->in_data_crc, + con->v2.in_kvecs[0].iov_base, + con->v2.in_kvecs[0].iov_len); + } + } else { + return -EIO; + } + + /* get next extent */ + ret = con->ops->sparse_read(con, cursor, &buf); + if (ret <= 0) { + if (ret < 0) + return ret; + + reset_in_kvecs(con); + add_in_kvec(con, con->v2.in_buf, CEPH_EPILOGUE_PLAIN_LEN); + con->v2.in_state = IN_S_HANDLE_EPILOGUE; + return 0; + } + + if (buf) { + /* receive into buffer */ + reset_in_kvecs(con); + add_in_kvec(con, buf, ret); + con->v2.data_len_remain -= ret; + return 0; + } + + if (ret > cursor->total_resid) { + pr_warn("%s: ret 0x%x total_resid 0x%zx resid 0x%zx\n", + __func__, ret, cursor->total_resid, cursor->resid); + return -EIO; + } + get_bvec_at(cursor, &bv); + if (bv.bv_len > cursor->sr_resid) + bv.bv_len = cursor->sr_resid; + if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) { + if (unlikely(!con->bounce_page)) { + con->bounce_page = alloc_page(GFP_NOIO); + if (!con->bounce_page) { + pr_err("failed to allocate bounce page\n"); + return -ENOMEM; + } + } + + bv.bv_page = con->bounce_page; + bv.bv_offset = 0; + } + set_in_bvec(con, &bv); + con->v2.data_len_remain -= ret; + return ret; +} + +static int prepare_sparse_read_data(struct ceph_connection *con) +{ + struct ceph_msg *msg = con->in_msg; + + dout("%s: starting sparse read\n", __func__); + + if (WARN_ON_ONCE(!con->ops->sparse_read)) + return -EOPNOTSUPP; + + if (!con_secure(con)) + con->in_data_crc = -1; + + reset_in_kvecs(con); + con->v2.in_state = IN_S_PREPARE_SPARSE_DATA_CONT; + con->v2.data_len_remain = data_len(msg); + return prepare_sparse_read_cont(con); +} + static int prepare_read_tail_plain(struct ceph_connection *con) { struct ceph_msg *msg = con->in_msg; @@ -1845,7 +2064,10 @@ static int prepare_read_tail_plain(struct ceph_connection *con) } if (data_len(msg)) { - con->v2.in_state = IN_S_PREPARE_READ_DATA; + if (msg->sparse_read) + con->v2.in_state = IN_S_PREPARE_SPARSE_DATA; + else + con->v2.in_state = IN_S_PREPARE_READ_DATA; } else { add_in_kvec(con, con->v2.in_buf, CEPH_EPILOGUE_PLAIN_LEN); con->v2.in_state = IN_S_HANDLE_EPILOGUE; @@ -2898,6 +3120,12 @@ static int populate_in_iter(struct ceph_connection *con) prepare_read_enc_page(con); ret = 0; break; + case IN_S_PREPARE_SPARSE_DATA: + ret = prepare_sparse_read_data(con); + break; + case IN_S_PREPARE_SPARSE_DATA_CONT: + ret = prepare_sparse_read_cont(con); + break; case IN_S_HANDLE_EPILOGUE: ret = handle_epilogue(con); break; @@ -3489,6 +3717,23 @@ static void revoke_at_prepare_read_enc_page(struct ceph_connection *con) con->v2.in_state = IN_S_FINISH_SKIP; } +static void revoke_at_prepare_sparse_data(struct ceph_connection *con) +{ + int resid; /* current piece of data */ + int remaining; + + WARN_ON(con_secure(con)); + WARN_ON(!data_len(con->in_msg)); + WARN_ON(!iov_iter_is_bvec(&con->v2.in_iter)); + resid = iov_iter_count(&con->v2.in_iter); + dout("%s con %p resid %d\n", __func__, con, resid); + + remaining = CEPH_EPILOGUE_PLAIN_LEN + con->v2.data_len_remain; + con->v2.in_iter.count -= resid; + set_in_skip(con, resid + remaining); + con->v2.in_state = IN_S_FINISH_SKIP; +} + static void revoke_at_handle_epilogue(struct ceph_connection *con) { int resid; @@ -3505,6 +3750,7 @@ static void revoke_at_handle_epilogue(struct ceph_connection *con) void ceph_con_v2_revoke_incoming(struct ceph_connection *con) { switch (con->v2.in_state) { + case IN_S_PREPARE_SPARSE_DATA: case IN_S_PREPARE_READ_DATA: revoke_at_prepare_read_data(con); break; @@ -3514,6 +3760,9 @@ void ceph_con_v2_revoke_incoming(struct ceph_connection *con) case IN_S_PREPARE_READ_ENC_PAGE: revoke_at_prepare_read_enc_page(con); break; + case IN_S_PREPARE_SPARSE_DATA_CONT: + revoke_at_prepare_sparse_data(con); + break; case IN_S_HANDLE_EPILOGUE: revoke_at_handle_epilogue(con); break; diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c index 658a6f2320cf..d3a759e052c8 100644 --- a/net/ceph/osd_client.c +++ b/net/ceph/osd_client.c @@ -171,6 +171,13 @@ static void ceph_osd_data_bvecs_init(struct ceph_osd_data *osd_data, osd_data->num_bvecs = num_bvecs; } +static void ceph_osd_iter_init(struct ceph_osd_data *osd_data, + struct iov_iter *iter) +{ + osd_data->type = CEPH_OSD_DATA_TYPE_ITER; + osd_data->iter = *iter; +} + static struct ceph_osd_data * osd_req_op_raw_data_in(struct ceph_osd_request *osd_req, unsigned int which) { @@ -264,6 +271,22 @@ void osd_req_op_extent_osd_data_bvec_pos(struct ceph_osd_request *osd_req, } EXPORT_SYMBOL(osd_req_op_extent_osd_data_bvec_pos); +/** + * osd_req_op_extent_osd_iter - Set up an operation with an iterator buffer + * @osd_req: The request to set up + * @which: Index of the operation in which to set the iter + * @iter: The buffer iterator + */ +void osd_req_op_extent_osd_iter(struct ceph_osd_request *osd_req, + unsigned int which, struct iov_iter *iter) +{ + struct ceph_osd_data *osd_data; + + osd_data = osd_req_op_data(osd_req, which, extent, osd_data); + ceph_osd_iter_init(osd_data, iter); +} +EXPORT_SYMBOL(osd_req_op_extent_osd_iter); + static void osd_req_op_cls_request_info_pagelist( struct ceph_osd_request *osd_req, unsigned int which, struct ceph_pagelist *pagelist) @@ -346,6 +369,8 @@ static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data) #endif /* CONFIG_BLOCK */ case CEPH_OSD_DATA_TYPE_BVECS: return osd_data->bvec_pos.iter.bi_size; + case CEPH_OSD_DATA_TYPE_ITER: + return iov_iter_count(&osd_data->iter); default: WARN(true, "unrecognized data type %d\n", (int)osd_data->type); return 0; @@ -376,8 +401,10 @@ static void osd_req_op_data_release(struct ceph_osd_request *osd_req, switch (op->op) { case CEPH_OSD_OP_READ: + case CEPH_OSD_OP_SPARSE_READ: case CEPH_OSD_OP_WRITE: case CEPH_OSD_OP_WRITEFULL: + kfree(op->extent.sparse_ext); ceph_osd_data_release(&op->extent.osd_data); break; case CEPH_OSD_OP_CALL: @@ -669,6 +696,7 @@ static void get_num_data_items(struct ceph_osd_request *req, /* reply */ case CEPH_OSD_OP_STAT: case CEPH_OSD_OP_READ: + case CEPH_OSD_OP_SPARSE_READ: case CEPH_OSD_OP_LIST_WATCHERS: *num_reply_data_items += 1; break; @@ -738,7 +766,7 @@ void osd_req_op_extent_init(struct ceph_osd_request *osd_req, BUG_ON(opcode != CEPH_OSD_OP_READ && opcode != CEPH_OSD_OP_WRITE && opcode != CEPH_OSD_OP_WRITEFULL && opcode != CEPH_OSD_OP_ZERO && - opcode != CEPH_OSD_OP_TRUNCATE); + opcode != CEPH_OSD_OP_TRUNCATE && opcode != CEPH_OSD_OP_SPARSE_READ); op->extent.offset = offset; op->extent.length = length; @@ -951,6 +979,8 @@ static void ceph_osdc_msg_data_add(struct ceph_msg *msg, #endif } else if (osd_data->type == CEPH_OSD_DATA_TYPE_BVECS) { ceph_msg_data_add_bvecs(msg, &osd_data->bvec_pos); + } else if (osd_data->type == CEPH_OSD_DATA_TYPE_ITER) { + ceph_msg_data_add_iter(msg, &osd_data->iter); } else { BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE); } @@ -963,6 +993,7 @@ static u32 osd_req_encode_op(struct ceph_osd_op *dst, case CEPH_OSD_OP_STAT: break; case CEPH_OSD_OP_READ: + case CEPH_OSD_OP_SPARSE_READ: case CEPH_OSD_OP_WRITE: case CEPH_OSD_OP_WRITEFULL: case CEPH_OSD_OP_ZERO: @@ -1017,6 +1048,10 @@ static u32 osd_req_encode_op(struct ceph_osd_op *dst, dst->copy_from.src_fadvise_flags = cpu_to_le32(src->copy_from.src_fadvise_flags); break; + case CEPH_OSD_OP_ASSERT_VER: + dst->assert_ver.unused = cpu_to_le64(0); + dst->assert_ver.ver = cpu_to_le64(src->assert_ver.ver); + break; default: pr_err("unsupported osd opcode %s\n", ceph_osd_op_name(src->op)); @@ -1059,7 +1094,8 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc, BUG_ON(opcode != CEPH_OSD_OP_READ && opcode != CEPH_OSD_OP_WRITE && opcode != CEPH_OSD_OP_ZERO && opcode != CEPH_OSD_OP_TRUNCATE && - opcode != CEPH_OSD_OP_CREATE && opcode != CEPH_OSD_OP_DELETE); + opcode != CEPH_OSD_OP_CREATE && opcode != CEPH_OSD_OP_DELETE && + opcode != CEPH_OSD_OP_SPARSE_READ); req = ceph_osdc_alloc_request(osdc, snapc, num_ops, use_mempool, GFP_NOFS); @@ -1100,15 +1136,30 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc, if (flags & CEPH_OSD_FLAG_WRITE) req->r_data_offset = off; - if (num_ops > 1) + if (num_ops > 1) { + int num_req_ops, num_rep_ops; + /* - * This is a special case for ceph_writepages_start(), but it - * also covers ceph_uninline_data(). If more multi-op request - * use cases emerge, we will need a separate helper. + * If this is a multi-op write request, assume that we'll need + * request ops. If it's a multi-op read then assume we'll need + * reply ops. Anything else and call it -EINVAL. */ - r = __ceph_osdc_alloc_messages(req, GFP_NOFS, num_ops, 0); - else + if (flags & CEPH_OSD_FLAG_WRITE) { + num_req_ops = num_ops; + num_rep_ops = 0; + } else if (flags & CEPH_OSD_FLAG_READ) { + num_req_ops = 0; + num_rep_ops = num_ops; + } else { + r = -EINVAL; + goto fail; + } + + r = __ceph_osdc_alloc_messages(req, GFP_NOFS, num_req_ops, + num_rep_ops); + } else { r = ceph_osdc_alloc_messages(req, GFP_NOFS); + } if (r) goto fail; @@ -1120,6 +1171,18 @@ fail: } EXPORT_SYMBOL(ceph_osdc_new_request); +int __ceph_alloc_sparse_ext_map(struct ceph_osd_req_op *op, int cnt) +{ + op->extent.sparse_ext_cnt = cnt; + op->extent.sparse_ext = kmalloc_array(cnt, + sizeof(*op->extent.sparse_ext), + GFP_NOFS); + if (!op->extent.sparse_ext) + return -ENOMEM; + return 0; +} +EXPORT_SYMBOL(__ceph_alloc_sparse_ext_map); + /* * We keep osd requests in an rbtree, sorted by ->r_tid. */ @@ -1177,6 +1240,7 @@ static void osd_init(struct ceph_osd *osd) { refcount_set(&osd->o_ref, 1); RB_CLEAR_NODE(&osd->o_node); + spin_lock_init(&osd->o_requests_lock); osd->o_requests = RB_ROOT; osd->o_linger_requests = RB_ROOT; osd->o_backoff_mappings = RB_ROOT; @@ -1187,6 +1251,13 @@ static void osd_init(struct ceph_osd *osd) mutex_init(&osd->lock); } +static void ceph_init_sparse_read(struct ceph_sparse_read *sr) +{ + kfree(sr->sr_extent); + memset(sr, '\0', sizeof(*sr)); + sr->sr_state = CEPH_SPARSE_READ_HDR; +} + static void osd_cleanup(struct ceph_osd *osd) { WARN_ON(!RB_EMPTY_NODE(&osd->o_node)); @@ -1197,6 +1268,8 @@ static void osd_cleanup(struct ceph_osd *osd) WARN_ON(!list_empty(&osd->o_osd_lru)); WARN_ON(!list_empty(&osd->o_keepalive_item)); + ceph_init_sparse_read(&osd->o_sparse_read); + if (osd->o_auth.authorizer) { WARN_ON(osd_homeless(osd)); ceph_auth_destroy_authorizer(osd->o_auth.authorizer); @@ -1216,6 +1289,9 @@ static struct ceph_osd *create_osd(struct ceph_osd_client *osdc, int onum) osd_init(osd); osd->o_osdc = osdc; osd->o_osd = onum; + osd->o_sparse_op_idx = -1; + + ceph_init_sparse_read(&osd->o_sparse_read); ceph_con_init(&osd->o_con, osd, &osd_con_ops, &osdc->client->msgr); @@ -1406,7 +1482,9 @@ static void link_request(struct ceph_osd *osd, struct ceph_osd_request *req) atomic_inc(&osd->o_osdc->num_homeless); get_osd(osd); + spin_lock(&osd->o_requests_lock); insert_request(&osd->o_requests, req); + spin_unlock(&osd->o_requests_lock); req->r_osd = osd; } @@ -1418,7 +1496,9 @@ static void unlink_request(struct ceph_osd *osd, struct ceph_osd_request *req) req, req->r_tid); req->r_osd = NULL; + spin_lock(&osd->o_requests_lock); erase_request(&osd->o_requests, req); + spin_unlock(&osd->o_requests_lock); put_osd(osd); if (!osd_homeless(osd)) @@ -2016,6 +2096,7 @@ static void setup_request_data(struct ceph_osd_request *req) &op->raw_data_in); break; case CEPH_OSD_OP_READ: + case CEPH_OSD_OP_SPARSE_READ: ceph_osdc_msg_data_add(reply_msg, &op->extent.osd_data); break; @@ -2435,8 +2516,10 @@ static void finish_request(struct ceph_osd_request *req) req->r_end_latency = ktime_get(); - if (req->r_osd) + if (req->r_osd) { + ceph_init_sparse_read(&req->r_osd->o_sparse_read); unlink_request(req->r_osd, req); + } atomic_dec(&osdc->num_requests); /* @@ -3795,6 +3878,7 @@ static void handle_reply(struct ceph_osd *osd, struct ceph_msg *msg) * one (type of) reply back. */ WARN_ON(!(m.flags & CEPH_OSD_FLAG_ONDISK)); + req->r_version = m.user_version; req->r_result = m.result ?: data_len; finish_request(req); mutex_unlock(&osd->lock); @@ -5348,6 +5432,24 @@ static void osd_dispatch(struct ceph_connection *con, struct ceph_msg *msg) ceph_msg_put(msg); } +/* How much sparse data was requested? */ +static u64 sparse_data_requested(struct ceph_osd_request *req) +{ + u64 len = 0; + + if (req->r_flags & CEPH_OSD_FLAG_READ) { + int i; + + for (i = 0; i < req->r_num_ops; ++i) { + struct ceph_osd_req_op *op = &req->r_ops[i]; + + if (op->op == CEPH_OSD_OP_SPARSE_READ) + len += op->extent.length; + } + } + return len; +} + /* * Lookup and return message for incoming reply. Don't try to do * anything about a larger than preallocated data portion of the @@ -5364,6 +5466,7 @@ static struct ceph_msg *get_reply(struct ceph_connection *con, int front_len = le32_to_cpu(hdr->front_len); int data_len = le32_to_cpu(hdr->data_len); u64 tid = le64_to_cpu(hdr->tid); + u64 srlen; down_read(&osdc->lock); if (!osd_registered(osd)) { @@ -5396,7 +5499,8 @@ static struct ceph_msg *get_reply(struct ceph_connection *con, req->r_reply = m; } - if (data_len > req->r_reply->data_length) { + srlen = sparse_data_requested(req); + if (!srlen && data_len > req->r_reply->data_length) { pr_warn("%s osd%d tid %llu data %d > preallocated %zu, skipping\n", __func__, osd->o_osd, req->r_tid, data_len, req->r_reply->data_length); @@ -5406,6 +5510,8 @@ static struct ceph_msg *get_reply(struct ceph_connection *con, } m = ceph_msg_get(req->r_reply); + m->sparse_read = (bool)srlen; + dout("get_reply tid %lld %p\n", tid, m); out_unlock_session: @@ -5638,9 +5744,217 @@ static int osd_check_message_signature(struct ceph_msg *msg) return ceph_auth_check_message_signature(auth, msg); } +static void advance_cursor(struct ceph_msg_data_cursor *cursor, size_t len, + bool zero) +{ + while (len) { + struct page *page; + size_t poff, plen; + + page = ceph_msg_data_next(cursor, &poff, &plen); + if (plen > len) + plen = len; + if (zero) + zero_user_segment(page, poff, poff + plen); + len -= plen; + ceph_msg_data_advance(cursor, plen); + } +} + +static int prep_next_sparse_read(struct ceph_connection *con, + struct ceph_msg_data_cursor *cursor) +{ + struct ceph_osd *o = con->private; + struct ceph_sparse_read *sr = &o->o_sparse_read; + struct ceph_osd_request *req; + struct ceph_osd_req_op *op; + + spin_lock(&o->o_requests_lock); + req = lookup_request(&o->o_requests, le64_to_cpu(con->in_msg->hdr.tid)); + if (!req) { + spin_unlock(&o->o_requests_lock); + return -EBADR; + } + + if (o->o_sparse_op_idx < 0) { + u64 srlen = sparse_data_requested(req); + + dout("%s: [%d] starting new sparse read req. srlen=0x%llx\n", + __func__, o->o_osd, srlen); + ceph_msg_data_cursor_init(cursor, con->in_msg, srlen); + } else { + u64 end; + + op = &req->r_ops[o->o_sparse_op_idx]; + + WARN_ON_ONCE(op->extent.sparse_ext); + + /* hand back buffer we took earlier */ + op->extent.sparse_ext = sr->sr_extent; + sr->sr_extent = NULL; + op->extent.sparse_ext_cnt = sr->sr_count; + sr->sr_ext_len = 0; + dout("%s: [%d] completed extent array len %d cursor->resid %zd\n", + __func__, o->o_osd, op->extent.sparse_ext_cnt, cursor->resid); + /* Advance to end of data for this operation */ + end = ceph_sparse_ext_map_end(op); + if (end < sr->sr_req_len) + advance_cursor(cursor, sr->sr_req_len - end, false); + } + + ceph_init_sparse_read(sr); + + /* find next op in this request (if any) */ + while (++o->o_sparse_op_idx < req->r_num_ops) { + op = &req->r_ops[o->o_sparse_op_idx]; + if (op->op == CEPH_OSD_OP_SPARSE_READ) + goto found; + } + + /* reset for next sparse read request */ + spin_unlock(&o->o_requests_lock); + o->o_sparse_op_idx = -1; + return 0; +found: + sr->sr_req_off = op->extent.offset; + sr->sr_req_len = op->extent.length; + sr->sr_pos = sr->sr_req_off; + dout("%s: [%d] new sparse read op at idx %d 0x%llx~0x%llx\n", __func__, + o->o_osd, o->o_sparse_op_idx, sr->sr_req_off, sr->sr_req_len); + + /* hand off request's sparse extent map buffer */ + sr->sr_ext_len = op->extent.sparse_ext_cnt; + op->extent.sparse_ext_cnt = 0; + sr->sr_extent = op->extent.sparse_ext; + op->extent.sparse_ext = NULL; + + spin_unlock(&o->o_requests_lock); + return 1; +} + +#ifdef __BIG_ENDIAN +static inline void convert_extent_map(struct ceph_sparse_read *sr) +{ + int i; + + for (i = 0; i < sr->sr_count; i++) { + struct ceph_sparse_extent *ext = &sr->sr_extent[i]; + + ext->off = le64_to_cpu((__force __le64)ext->off); + ext->len = le64_to_cpu((__force __le64)ext->len); + } +} +#else +static inline void convert_extent_map(struct ceph_sparse_read *sr) +{ +} +#endif + +#define MAX_EXTENTS 4096 + +static int osd_sparse_read(struct ceph_connection *con, + struct ceph_msg_data_cursor *cursor, + char **pbuf) +{ + struct ceph_osd *o = con->private; + struct ceph_sparse_read *sr = &o->o_sparse_read; + u32 count = sr->sr_count; + u64 eoff, elen; + int ret; + + switch (sr->sr_state) { + case CEPH_SPARSE_READ_HDR: +next_op: + ret = prep_next_sparse_read(con, cursor); + if (ret <= 0) + return ret; + + /* number of extents */ + ret = sizeof(sr->sr_count); + *pbuf = (char *)&sr->sr_count; + sr->sr_state = CEPH_SPARSE_READ_EXTENTS; + break; + case CEPH_SPARSE_READ_EXTENTS: + /* Convert sr_count to host-endian */ + count = le32_to_cpu((__force __le32)sr->sr_count); + sr->sr_count = count; + dout("[%d] got %u extents\n", o->o_osd, count); + + if (count > 0) { + if (!sr->sr_extent || count > sr->sr_ext_len) { + /* + * Apply a hard cap to the number of extents. + * If we have more, assume something is wrong. + */ + if (count > MAX_EXTENTS) { + dout("%s: OSD returned 0x%x extents in a single reply!\n", + __func__, count); + return -EREMOTEIO; + } + + /* no extent array provided, or too short */ + kfree(sr->sr_extent); + sr->sr_extent = kmalloc_array(count, + sizeof(*sr->sr_extent), + GFP_NOIO); + if (!sr->sr_extent) + return -ENOMEM; + sr->sr_ext_len = count; + } + ret = count * sizeof(*sr->sr_extent); + *pbuf = (char *)sr->sr_extent; + sr->sr_state = CEPH_SPARSE_READ_DATA_LEN; + break; + } + /* No extents? Read data len */ + fallthrough; + case CEPH_SPARSE_READ_DATA_LEN: + convert_extent_map(sr); + ret = sizeof(sr->sr_datalen); + *pbuf = (char *)&sr->sr_datalen; + sr->sr_state = CEPH_SPARSE_READ_DATA; + break; + case CEPH_SPARSE_READ_DATA: + if (sr->sr_index >= count) { + sr->sr_state = CEPH_SPARSE_READ_HDR; + goto next_op; + } + + eoff = sr->sr_extent[sr->sr_index].off; + elen = sr->sr_extent[sr->sr_index].len; + + dout("[%d] ext %d off 0x%llx len 0x%llx\n", + o->o_osd, sr->sr_index, eoff, elen); + + if (elen > INT_MAX) { + dout("Sparse read extent length too long (0x%llx)\n", + elen); + return -EREMOTEIO; + } + + /* zero out anything from sr_pos to start of extent */ + if (sr->sr_pos < eoff) + advance_cursor(cursor, eoff - sr->sr_pos, true); + + /* Set position to end of extent */ + sr->sr_pos = eoff + elen; + + /* send back the new length and nullify the ptr */ + cursor->sr_resid = elen; + ret = elen; + *pbuf = NULL; + + /* Bump the array index */ + ++sr->sr_index; + break; + } + return ret; +} + static const struct ceph_connection_operations osd_con_ops = { .get = osd_get_con, .put = osd_put_con, + .sparse_read = osd_sparse_read, .alloc_msg = osd_alloc_msg, .dispatch = osd_dispatch, .fault = osd_fault, diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 89d15ceaf9af..b3b3af0e7844 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -1831,8 +1831,7 @@ u32 __skb_get_hash_symmetric(const struct sk_buff *skb) memset(&keys, 0, sizeof(keys)); __skb_flow_dissect(NULL, skb, &flow_keys_dissector_symmetric, - &keys, NULL, 0, 0, 0, - FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL); + &keys, NULL, 0, 0, 0, 0); return __flow_hash_from_keys(&keys, &hashrnd); } diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 45707059082f..4eaf7ed0d1f4 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -550,7 +550,7 @@ static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node, bool *pfmemalloc) { bool ret_pfmemalloc = false; - unsigned int obj_size; + size_t obj_size; void *obj; obj_size = SKB_HEAD_ALIGN(*size); @@ -567,7 +567,13 @@ static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node, obj = kmem_cache_alloc_node(skb_small_head_cache, flags, node); goto out; } - *size = obj_size = kmalloc_size_roundup(obj_size); + + obj_size = kmalloc_size_roundup(obj_size); + /* The following cast might truncate high-order bits of obj_size, this + * is harmless because kmalloc(obj_size >= 2^32) will fail anyway. + */ + *size = (unsigned int)obj_size; + /* * Try a regular allocation, when that fails and we're not entitled * to the reserves, fail. @@ -4423,21 +4429,20 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb, struct sk_buff *segs = NULL; struct sk_buff *tail = NULL; struct sk_buff *list_skb = skb_shinfo(head_skb)->frag_list; - skb_frag_t *frag = skb_shinfo(head_skb)->frags; unsigned int mss = skb_shinfo(head_skb)->gso_size; unsigned int doffset = head_skb->data - skb_mac_header(head_skb); - struct sk_buff *frag_skb = head_skb; unsigned int offset = doffset; unsigned int tnl_hlen = skb_tnl_header_len(head_skb); unsigned int partial_segs = 0; unsigned int headroom; unsigned int len = head_skb->len; + struct sk_buff *frag_skb; + skb_frag_t *frag; __be16 proto; bool csum, sg; - int nfrags = skb_shinfo(head_skb)->nr_frags; int err = -ENOMEM; int i = 0; - int pos; + int nfrags, pos; if ((skb_shinfo(head_skb)->gso_type & SKB_GSO_DODGY) && mss != GSO_BY_FRAGS && mss != skb_headlen(head_skb)) { @@ -4514,6 +4519,13 @@ normal: headroom = skb_headroom(head_skb); pos = skb_headlen(head_skb); + if (skb_orphan_frags(head_skb, GFP_ATOMIC)) + return ERR_PTR(-ENOMEM); + + nfrags = skb_shinfo(head_skb)->nr_frags; + frag = skb_shinfo(head_skb)->frags; + frag_skb = head_skb; + do { struct sk_buff *nskb; skb_frag_t *nskb_frag; @@ -4534,6 +4546,10 @@ normal: (skb_headlen(list_skb) == len || sg)) { BUG_ON(skb_headlen(list_skb) > len); + nskb = skb_clone(list_skb, GFP_ATOMIC); + if (unlikely(!nskb)) + goto err; + i = 0; nfrags = skb_shinfo(list_skb)->nr_frags; frag = skb_shinfo(list_skb)->frags; @@ -4552,12 +4568,8 @@ normal: frag++; } - nskb = skb_clone(list_skb, GFP_ATOMIC); list_skb = list_skb->next; - if (unlikely(!nskb)) - goto err; - if (unlikely(pskb_trim(nskb, len))) { kfree_skb(nskb); goto err; @@ -4633,12 +4645,16 @@ normal: skb_shinfo(nskb)->flags |= skb_shinfo(head_skb)->flags & SKBFL_SHARED_FRAG; - if (skb_orphan_frags(frag_skb, GFP_ATOMIC) || - skb_zerocopy_clone(nskb, frag_skb, GFP_ATOMIC)) + if (skb_zerocopy_clone(nskb, frag_skb, GFP_ATOMIC)) goto err; while (pos < offset + len) { if (i >= nfrags) { + if (skb_orphan_frags(list_skb, GFP_ATOMIC) || + skb_zerocopy_clone(nskb, list_skb, + GFP_ATOMIC)) + goto err; + i = 0; nfrags = skb_shinfo(list_skb)->nr_frags; frag = skb_shinfo(list_skb)->frags; @@ -4652,10 +4668,6 @@ normal: i--; frag--; } - if (skb_orphan_frags(frag_skb, GFP_ATOMIC) || - skb_zerocopy_clone(nskb, frag_skb, - GFP_ATOMIC)) - goto err; list_skb = list_skb->next; } @@ -5207,7 +5219,7 @@ static void __skb_complete_tx_timestamp(struct sk_buff *skb, serr->ee.ee_info = tstype; serr->opt_stats = opt_stats; serr->header.h4.iif = skb->dev ? skb->dev->ifindex : 0; - if (sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID) { + if (READ_ONCE(sk->sk_tsflags) & SOF_TIMESTAMPING_OPT_ID) { serr->ee.ee_data = skb_shinfo(skb)->tskey; if (sk_is_tcp(sk)) serr->ee.ee_data -= atomic_read(&sk->sk_tskey); @@ -5263,21 +5275,23 @@ void __skb_tstamp_tx(struct sk_buff *orig_skb, { struct sk_buff *skb; bool tsonly, opt_stats = false; + u32 tsflags; if (!sk) return; - if (!hwtstamps && !(sk->sk_tsflags & SOF_TIMESTAMPING_OPT_TX_SWHW) && + tsflags = READ_ONCE(sk->sk_tsflags); + if (!hwtstamps && !(tsflags & SOF_TIMESTAMPING_OPT_TX_SWHW) && skb_shinfo(orig_skb)->tx_flags & SKBTX_IN_PROGRESS) return; - tsonly = sk->sk_tsflags & SOF_TIMESTAMPING_OPT_TSONLY; + tsonly = tsflags & SOF_TIMESTAMPING_OPT_TSONLY; if (!skb_may_tx_timestamp(sk, tsonly)) return; if (tsonly) { #ifdef CONFIG_INET - if ((sk->sk_tsflags & SOF_TIMESTAMPING_OPT_STATS) && + if ((tsflags & SOF_TIMESTAMPING_OPT_STATS) && sk_is_tcp(sk)) { skb = tcp_get_timestamping_opt_stats(sk, orig_skb, ack_skb); diff --git a/net/core/skmsg.c b/net/core/skmsg.c index a0659fc29bcc..6c31eefbd777 100644 --- a/net/core/skmsg.c +++ b/net/core/skmsg.c @@ -612,12 +612,18 @@ static int sk_psock_skb_ingress_self(struct sk_psock *psock, struct sk_buff *skb static int sk_psock_handle_skb(struct sk_psock *psock, struct sk_buff *skb, u32 off, u32 len, bool ingress) { + int err = 0; + if (!ingress) { if (!sock_writeable(psock->sk)) return -EAGAIN; return skb_send_sock(psock->sk, skb, off, len); } - return sk_psock_skb_ingress(psock, skb, off, len); + skb_get(skb); + err = sk_psock_skb_ingress(psock, skb, off, len); + if (err < 0) + kfree_skb(skb); + return err; } static void sk_psock_skb_state(struct sk_psock *psock, @@ -685,9 +691,7 @@ static void sk_psock_backlog(struct work_struct *work) } while (len); skb = skb_dequeue(&psock->ingress_skb); - if (!ingress) { - kfree_skb(skb); - } + kfree_skb(skb); } end: mutex_unlock(&psock->work_mutex); diff --git a/net/core/sock.c b/net/core/sock.c index b6d15dfe737f..2b38e67ca7ad 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -766,7 +766,8 @@ bool sk_mc_loop(struct sock *sk) return false; if (!sk) return true; - switch (sk->sk_family) { + /* IPV6_ADDRFORM can change sk->sk_family under us. */ + switch (READ_ONCE(sk->sk_family)) { case AF_INET: return inet_test_bit(MC_LOOP, sk); #if IS_ENABLED(CONFIG_IPV6) @@ -894,7 +895,7 @@ static int sock_timestamping_bind_phc(struct sock *sk, int phc_index) if (!match) return -EINVAL; - sk->sk_bind_phc = phc_index; + WRITE_ONCE(sk->sk_bind_phc, phc_index); return 0; } @@ -937,7 +938,7 @@ int sock_set_timestamping(struct sock *sk, int optname, return ret; } - sk->sk_tsflags = val; + WRITE_ONCE(sk->sk_tsflags, val); sock_valbool_flag(sk, SOCK_TSTAMP_NEW, optname == SO_TIMESTAMPING_NEW); if (val & SOF_TIMESTAMPING_RX_SOFTWARE) @@ -1045,7 +1046,7 @@ static int sock_reserve_memory(struct sock *sk, int bytes) mem_cgroup_uncharge_skmem(sk->sk_memcg, pages); return -ENOMEM; } - sk->sk_forward_alloc += pages << PAGE_SHIFT; + sk_forward_alloc_add(sk, pages << PAGE_SHIFT); WRITE_ONCE(sk->sk_reserved_mem, sk->sk_reserved_mem + (pages << PAGE_SHIFT)); @@ -1719,8 +1720,8 @@ int sk_getsockopt(struct sock *sk, int level, int optname, case SO_TIMESTAMPING_OLD: lv = sizeof(v.timestamping); - v.timestamping.flags = sk->sk_tsflags; - v.timestamping.bind_phc = sk->sk_bind_phc; + v.timestamping.flags = READ_ONCE(sk->sk_tsflags); + v.timestamping.bind_phc = READ_ONCE(sk->sk_bind_phc); break; case SO_RCVTIMEO_OLD: @@ -2747,9 +2748,9 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo) prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE); if (refcount_read(&sk->sk_wmem_alloc) < READ_ONCE(sk->sk_sndbuf)) break; - if (sk->sk_shutdown & SEND_SHUTDOWN) + if (READ_ONCE(sk->sk_shutdown) & SEND_SHUTDOWN) break; - if (sk->sk_err) + if (READ_ONCE(sk->sk_err)) break; timeo = schedule_timeout(timeo); } @@ -2777,7 +2778,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, goto failure; err = -EPIPE; - if (sk->sk_shutdown & SEND_SHUTDOWN) + if (READ_ONCE(sk->sk_shutdown) & SEND_SHUTDOWN) goto failure; if (sk_wmem_alloc_get(sk) < READ_ONCE(sk->sk_sndbuf)) @@ -3139,10 +3140,10 @@ int __sk_mem_schedule(struct sock *sk, int size, int kind) { int ret, amt = sk_mem_pages(size); - sk->sk_forward_alloc += amt << PAGE_SHIFT; + sk_forward_alloc_add(sk, amt << PAGE_SHIFT); ret = __sk_mem_raise_allocated(sk, size, amt, kind); if (!ret) - sk->sk_forward_alloc -= amt << PAGE_SHIFT; + sk_forward_alloc_add(sk, -(amt << PAGE_SHIFT)); return ret; } EXPORT_SYMBOL(__sk_mem_schedule); @@ -3174,7 +3175,7 @@ void __sk_mem_reduce_allocated(struct sock *sk, int amount) void __sk_mem_reclaim(struct sock *sk, int amount) { amount >>= PAGE_SHIFT; - sk->sk_forward_alloc -= amount << PAGE_SHIFT; + sk_forward_alloc_add(sk, -(amount << PAGE_SHIFT)); __sk_mem_reduce_allocated(sk, amount); } EXPORT_SYMBOL(__sk_mem_reclaim); @@ -3753,7 +3754,7 @@ void sk_get_meminfo(const struct sock *sk, u32 *mem) mem[SK_MEMINFO_RCVBUF] = READ_ONCE(sk->sk_rcvbuf); mem[SK_MEMINFO_WMEM_ALLOC] = sk_wmem_alloc_get(sk); mem[SK_MEMINFO_SNDBUF] = READ_ONCE(sk->sk_sndbuf); - mem[SK_MEMINFO_FWD_ALLOC] = sk->sk_forward_alloc; + mem[SK_MEMINFO_FWD_ALLOC] = sk_forward_alloc_get(sk); mem[SK_MEMINFO_WMEM_QUEUED] = READ_ONCE(sk->sk_wmem_queued); mem[SK_MEMINFO_OPTMEM] = atomic_read(&sk->sk_omem_alloc); mem[SK_MEMINFO_BACKLOG] = READ_ONCE(sk->sk_backlog.len); diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 8f07fea39d9e..cb11750b1df5 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -18,7 +18,7 @@ struct bpf_stab { struct bpf_map map; struct sock **sks; struct sk_psock_progs progs; - raw_spinlock_t lock; + spinlock_t lock; }; #define SOCK_CREATE_FLAG_MASK \ @@ -44,7 +44,7 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) return ERR_PTR(-ENOMEM); bpf_map_init_from_attr(&stab->map, attr); - raw_spin_lock_init(&stab->lock); + spin_lock_init(&stab->lock); stab->sks = bpf_map_area_alloc((u64) stab->map.max_entries * sizeof(struct sock *), @@ -411,7 +411,7 @@ static int __sock_map_delete(struct bpf_stab *stab, struct sock *sk_test, struct sock *sk; int err = 0; - raw_spin_lock_bh(&stab->lock); + spin_lock_bh(&stab->lock); sk = *psk; if (!sk_test || sk_test == sk) sk = xchg(psk, NULL); @@ -421,7 +421,7 @@ static int __sock_map_delete(struct bpf_stab *stab, struct sock *sk_test, else err = -EINVAL; - raw_spin_unlock_bh(&stab->lock); + spin_unlock_bh(&stab->lock); return err; } @@ -487,7 +487,7 @@ static int sock_map_update_common(struct bpf_map *map, u32 idx, psock = sk_psock(sk); WARN_ON_ONCE(!psock); - raw_spin_lock_bh(&stab->lock); + spin_lock_bh(&stab->lock); osk = stab->sks[idx]; if (osk && flags == BPF_NOEXIST) { ret = -EEXIST; @@ -501,10 +501,10 @@ static int sock_map_update_common(struct bpf_map *map, u32 idx, stab->sks[idx] = sk; if (osk) sock_map_unref(osk, &stab->sks[idx]); - raw_spin_unlock_bh(&stab->lock); + spin_unlock_bh(&stab->lock); return 0; out_unlock: - raw_spin_unlock_bh(&stab->lock); + spin_unlock_bh(&stab->lock); if (psock) sk_psock_put(sk, psock); out_free: @@ -835,7 +835,7 @@ struct bpf_shtab_elem { struct bpf_shtab_bucket { struct hlist_head head; - raw_spinlock_t lock; + spinlock_t lock; }; struct bpf_shtab { @@ -910,7 +910,7 @@ static void sock_hash_delete_from_link(struct bpf_map *map, struct sock *sk, * is okay since it's going away only after RCU grace period. * However, we need to check whether it's still present. */ - raw_spin_lock_bh(&bucket->lock); + spin_lock_bh(&bucket->lock); elem_probe = sock_hash_lookup_elem_raw(&bucket->head, elem->hash, elem->key, map->key_size); if (elem_probe && elem_probe == elem) { @@ -918,7 +918,7 @@ static void sock_hash_delete_from_link(struct bpf_map *map, struct sock *sk, sock_map_unref(elem->sk, elem); sock_hash_free_elem(htab, elem); } - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); } static long sock_hash_delete_elem(struct bpf_map *map, void *key) @@ -932,7 +932,7 @@ static long sock_hash_delete_elem(struct bpf_map *map, void *key) hash = sock_hash_bucket_hash(key, key_size); bucket = sock_hash_select_bucket(htab, hash); - raw_spin_lock_bh(&bucket->lock); + spin_lock_bh(&bucket->lock); elem = sock_hash_lookup_elem_raw(&bucket->head, hash, key, key_size); if (elem) { hlist_del_rcu(&elem->node); @@ -940,7 +940,7 @@ static long sock_hash_delete_elem(struct bpf_map *map, void *key) sock_hash_free_elem(htab, elem); ret = 0; } - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); return ret; } @@ -1000,7 +1000,7 @@ static int sock_hash_update_common(struct bpf_map *map, void *key, hash = sock_hash_bucket_hash(key, key_size); bucket = sock_hash_select_bucket(htab, hash); - raw_spin_lock_bh(&bucket->lock); + spin_lock_bh(&bucket->lock); elem = sock_hash_lookup_elem_raw(&bucket->head, hash, key, key_size); if (elem && flags == BPF_NOEXIST) { ret = -EEXIST; @@ -1026,10 +1026,10 @@ static int sock_hash_update_common(struct bpf_map *map, void *key, sock_map_unref(elem->sk, elem); sock_hash_free_elem(htab, elem); } - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); return 0; out_unlock: - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); sk_psock_put(sk, psock); out_free: sk_psock_free_link(link); @@ -1115,7 +1115,7 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) for (i = 0; i < htab->buckets_num; i++) { INIT_HLIST_HEAD(&htab->buckets[i].head); - raw_spin_lock_init(&htab->buckets[i].lock); + spin_lock_init(&htab->buckets[i].lock); } return &htab->map; @@ -1147,11 +1147,11 @@ static void sock_hash_free(struct bpf_map *map) * exists, psock exists and holds a ref to socket. That * lets us to grab a socket ref too. */ - raw_spin_lock_bh(&bucket->lock); + spin_lock_bh(&bucket->lock); hlist_for_each_entry(elem, &bucket->head, node) sock_hold(elem->sk); hlist_move_list(&bucket->head, &unlink_list); - raw_spin_unlock_bh(&bucket->lock); + spin_unlock_bh(&bucket->lock); /* Process removed entries out of atomic context to * block for socket lock before deleting the psock's diff --git a/net/handshake/netlink.c b/net/handshake/netlink.c index 1086653e1fad..d0bc1dd8e65a 100644 --- a/net/handshake/netlink.c +++ b/net/handshake/netlink.c @@ -157,26 +157,24 @@ out_status: int handshake_nl_done_doit(struct sk_buff *skb, struct genl_info *info) { struct net *net = sock_net(skb->sk); - struct handshake_req *req = NULL; - struct socket *sock = NULL; + struct handshake_req *req; + struct socket *sock; int fd, status, err; if (GENL_REQ_ATTR_CHECK(info, HANDSHAKE_A_DONE_SOCKFD)) return -EINVAL; fd = nla_get_u32(info->attrs[HANDSHAKE_A_DONE_SOCKFD]); - err = 0; sock = sockfd_lookup(fd, &err); - if (err) { - err = -EBADF; - goto out_status; - } + if (!sock) + return err; req = handshake_req_hash_lookup(sock->sk); if (!req) { err = -EBUSY; + trace_handshake_cmd_done_err(net, req, sock->sk, err); fput(sock->file); - goto out_status; + return err; } trace_handshake_cmd_done(net, req, sock->sk, fd); @@ -188,10 +186,6 @@ int handshake_nl_done_doit(struct sk_buff *skb, struct genl_info *info) handshake_complete(req, status, info); fput(sock->file); return 0; - -out_status: - trace_handshake_cmd_done_err(net, req, sock->sk, err); - return err; } static unsigned int handshake_net_id; diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c index 65ba18a91865..eafa4a033515 100644 --- a/net/ipv4/fib_semantics.c +++ b/net/ipv4/fib_semantics.c @@ -278,7 +278,8 @@ void fib_release_info(struct fib_info *fi) hlist_del(&nexthop_nh->nh_hash); } endfor_nexthops(fi) } - fi->fib_dead = 1; + /* Paired with READ_ONCE() from fib_table_lookup() */ + WRITE_ONCE(fi->fib_dead, 1); fib_info_put(fi); } spin_unlock_bh(&fib_info_lock); @@ -1581,6 +1582,7 @@ struct fib_info *fib_create_info(struct fib_config *cfg, link_it: ofi = fib_find_info(fi); if (ofi) { + /* fib_table_lookup() should not see @fi yet. */ fi->fib_dead = 1; free_fib_info(fi); refcount_inc(&ofi->fib_treeref); @@ -1619,6 +1621,7 @@ err_inval: failure: if (fi) { + /* fib_table_lookup() should not see @fi yet. */ fi->fib_dead = 1; free_fib_info(fi); } diff --git a/net/ipv4/fib_trie.c b/net/ipv4/fib_trie.c index 74d403dbd2b4..d13fb9e76b97 100644 --- a/net/ipv4/fib_trie.c +++ b/net/ipv4/fib_trie.c @@ -1582,7 +1582,8 @@ found: if (fa->fa_dscp && inet_dscp_to_dsfield(fa->fa_dscp) != flp->flowi4_tos) continue; - if (fi->fib_dead) + /* Paired with WRITE_ONCE() in fib_release_info() */ + if (READ_ONCE(fi->fib_dead)) continue; if (fa->fa_info->fib_scope < flp->flowi4_scope) continue; diff --git a/net/ipv4/igmp.c b/net/ipv4/igmp.c index 0c9e768e5628..418e5fb58fd3 100644 --- a/net/ipv4/igmp.c +++ b/net/ipv4/igmp.c @@ -353,8 +353,9 @@ static struct sk_buff *igmpv3_newpack(struct net_device *dev, unsigned int mtu) struct flowi4 fl4; int hlen = LL_RESERVED_SPACE(dev); int tlen = dev->needed_tailroom; - unsigned int size = mtu; + unsigned int size; + size = min(mtu, IP_MAX_MTU); while (1) { skb = alloc_skb(size + hlen + tlen, GFP_ATOMIC | __GFP_NOWARN); diff --git a/net/ipv4/ip_forward.c b/net/ipv4/ip_forward.c index e18931a6d153..66fac1216d46 100644 --- a/net/ipv4/ip_forward.c +++ b/net/ipv4/ip_forward.c @@ -67,7 +67,6 @@ static int ip_forward_finish(struct net *net, struct sock *sk, struct sk_buff *s struct ip_options *opt = &(IPCB(skb)->opt); __IP_INC_STATS(net, IPSTATS_MIB_OUTFORWDATAGRAMS); - __IP_ADD_STATS(net, IPSTATS_MIB_OUTOCTETS, skb->len); #ifdef CONFIG_NET_SWITCHDEV if (skb->offload_l3_fwd_mark) { diff --git a/net/ipv4/ip_input.c b/net/ipv4/ip_input.c index fe9ead9ee863..5e9c8156656a 100644 --- a/net/ipv4/ip_input.c +++ b/net/ipv4/ip_input.c @@ -584,7 +584,8 @@ static void ip_sublist_rcv_finish(struct list_head *head) static struct sk_buff *ip_extract_route_hint(const struct net *net, struct sk_buff *skb, int rt_type) { - if (fib4_has_custom_rules(net) || rt_type == RTN_BROADCAST) + if (fib4_has_custom_rules(net) || rt_type == RTN_BROADCAST || + IPCB(skb)->flags & IPSKB_MULTIPATH) return NULL; return skb; diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 43ba4b77b248..4ab877cf6d35 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -207,6 +207,9 @@ static int ip_finish_output2(struct net *net, struct sock *sk, struct sk_buff *s } else if (rt->rt_type == RTN_BROADCAST) IP_UPD_PO_STATS(net, IPSTATS_MIB_OUTBCAST, skb->len); + /* OUTOCTETS should be counted after fragment */ + IP_UPD_PO_STATS(net, IPSTATS_MIB_OUT, skb->len); + if (unlikely(skb_headroom(skb) < hh_len && dev->header_ops)) { skb = skb_expand_head(skb, hh_len); if (!skb) @@ -366,8 +369,6 @@ int ip_mc_output(struct net *net, struct sock *sk, struct sk_buff *skb) /* * If the indicated interface is up and running, send the packet. */ - IP_UPD_PO_STATS(net, IPSTATS_MIB_OUT, skb->len); - skb->dev = dev; skb->protocol = htons(ETH_P_IP); @@ -424,8 +425,6 @@ int ip_output(struct net *net, struct sock *sk, struct sk_buff *skb) { struct net_device *dev = skb_dst(skb)->dev, *indev = skb->dev; - IP_UPD_PO_STATS(net, IPSTATS_MIB_OUT, skb->len); - skb->dev = dev; skb->protocol = htons(ETH_P_IP); @@ -982,7 +981,7 @@ static int __ip_append_data(struct sock *sk, paged = !!cork->gso_size; if (cork->tx_flags & SKBTX_ANY_TSTAMP && - sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID) + READ_ONCE(sk->sk_tsflags) & SOF_TIMESTAMPING_OPT_ID) tskey = atomic_inc_return(&sk->sk_tskey) - 1; hh_len = LL_RESERVED_SPACE(rt->dst.dev); diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c index d1c73660b844..cce9cb25f3b3 100644 --- a/net/ipv4/ip_sockglue.c +++ b/net/ipv4/ip_sockglue.c @@ -511,7 +511,7 @@ static bool ipv4_datagram_support_cmsg(const struct sock *sk, * or without payload (SOF_TIMESTAMPING_OPT_TSONLY). */ info = PKTINFO_SKB_CB(skb); - if (!(sk->sk_tsflags & SOF_TIMESTAMPING_OPT_CMSG) || + if (!(READ_ONCE(sk->sk_tsflags) & SOF_TIMESTAMPING_OPT_CMSG) || !info->ipi_ifindex) return false; diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c index 3f0c6d602fb7..9e222a57bc2b 100644 --- a/net/ipv4/ipmr.c +++ b/net/ipv4/ipmr.c @@ -1804,7 +1804,6 @@ static inline int ipmr_forward_finish(struct net *net, struct sock *sk, struct ip_options *opt = &(IPCB(skb)->opt); IP_INC_STATS(net, IPSTATS_MIB_OUTFORWDATAGRAMS); - IP_ADD_STATS(net, IPSTATS_MIB_OUTOCTETS, skb->len); if (unlikely(opt->optlen)) ip_forward_options(skb); diff --git a/net/ipv4/route.c b/net/ipv4/route.c index d8c99bdc6170..66f419e7f9a7 100644 --- a/net/ipv4/route.c +++ b/net/ipv4/route.c @@ -2144,6 +2144,7 @@ static int ip_mkroute_input(struct sk_buff *skb, int h = fib_multipath_hash(res->fi->fib_net, NULL, skb, hkeys); fib_select_multipath(res, h); + IPCB(skb)->flags |= IPSKB_MULTIPATH; } #endif diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index b1559481898d..0c3040a63ebd 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2256,14 +2256,14 @@ void tcp_recv_timestamp(struct msghdr *msg, const struct sock *sk, } } - if (sk->sk_tsflags & SOF_TIMESTAMPING_SOFTWARE) + if (READ_ONCE(sk->sk_tsflags) & SOF_TIMESTAMPING_SOFTWARE) has_timestamping = true; else tss->ts[0] = (struct timespec64) {0}; } if (tss->ts[2].tv_sec || tss->ts[2].tv_nsec) { - if (sk->sk_tsflags & SOF_TIMESTAMPING_RAW_HARDWARE) + if (READ_ONCE(sk->sk_tsflags) & SOF_TIMESTAMPING_RAW_HARDWARE) has_timestamping = true; else tss->ts[2] = (struct timespec64) {0}; diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index e6b4fbd642f7..ccfc8bbf7455 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3474,7 +3474,7 @@ void sk_forced_mem_schedule(struct sock *sk, int size) if (delta <= 0) return; amt = sk_mem_pages(delta); - sk->sk_forward_alloc += amt << PAGE_SHIFT; + sk_forward_alloc_add(sk, amt << PAGE_SHIFT); sk_memory_allocated_add(sk, amt); if (mem_cgroup_sockets_enabled && sk->sk_memcg) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 0794a2c46a56..f39b9c844580 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1414,9 +1414,9 @@ static void udp_rmem_release(struct sock *sk, int size, int partial, spin_lock(&sk_queue->lock); - sk->sk_forward_alloc += size; + sk_forward_alloc_add(sk, size); amt = (sk->sk_forward_alloc - partial) & ~(PAGE_SIZE - 1); - sk->sk_forward_alloc -= amt; + sk_forward_alloc_add(sk, -amt); if (amt) __sk_mem_reduce_allocated(sk, amt >> PAGE_SHIFT); @@ -1527,7 +1527,7 @@ int __udp_enqueue_schedule_skb(struct sock *sk, struct sk_buff *skb) goto uncharge_drop; } - sk->sk_forward_alloc -= size; + sk_forward_alloc_add(sk, -size); /* no need to setup a destructor, we will explicitly release the * forward allocated memory on dequeue diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c index 934fec67785a..fdf1a3daec5f 100644 --- a/net/ipv6/addrconf.c +++ b/net/ipv6/addrconf.c @@ -1380,7 +1380,7 @@ retry: * idev->desync_factor if it's larger */ cnf_temp_preferred_lft = READ_ONCE(idev->cnf.temp_prefered_lft); - max_desync_factor = min_t(__u32, + max_desync_factor = min_t(long, idev->cnf.max_desync_factor, cnf_temp_preferred_lft - regen_advance); diff --git a/net/ipv6/ip6_input.c b/net/ipv6/ip6_input.c index d94041bb4287..b8378814532c 100644 --- a/net/ipv6/ip6_input.c +++ b/net/ipv6/ip6_input.c @@ -99,7 +99,8 @@ static bool ip6_can_use_hint(const struct sk_buff *skb, static struct sk_buff *ip6_extract_route_hint(const struct net *net, struct sk_buff *skb) { - if (fib6_routes_require_src(net) || fib6_has_custom_rules(net)) + if (fib6_routes_require_src(net) || fib6_has_custom_rules(net) || + IP6CB(skb)->flags & IP6SKB_MULTIPATH) return NULL; return skb; diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 0665e8b09968..54fc4c711f2c 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -451,7 +451,6 @@ static inline int ip6_forward_finish(struct net *net, struct sock *sk, struct dst_entry *dst = skb_dst(skb); __IP6_INC_STATS(net, ip6_dst_idev(dst), IPSTATS_MIB_OUTFORWDATAGRAMS); - __IP6_ADD_STATS(net, ip6_dst_idev(dst), IPSTATS_MIB_OUTOCTETS, skb->len); #ifdef CONFIG_NET_SWITCHDEV if (skb->offload_l3_fwd_mark) { @@ -1502,7 +1501,7 @@ static int __ip6_append_data(struct sock *sk, orig_mtu = mtu; if (cork->tx_flags & SKBTX_ANY_TSTAMP && - sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID) + READ_ONCE(sk->sk_tsflags) & SOF_TIMESTAMPING_OPT_ID) tskey = atomic_inc_return(&sk->sk_tskey) - 1; hh_len = LL_RESERVED_SPACE(rt->dst.dev); diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c index 67a3b8f6e72b..30ca064b76ef 100644 --- a/net/ipv6/ip6mr.c +++ b/net/ipv6/ip6mr.c @@ -2010,8 +2010,6 @@ static inline int ip6mr_forward2_finish(struct net *net, struct sock *sk, struct { IP6_INC_STATS(net, ip6_dst_idev(skb_dst(skb)), IPSTATS_MIB_OUTFORWDATAGRAMS); - IP6_ADD_STATS(net, ip6_dst_idev(skb_dst(skb)), - IPSTATS_MIB_OUTOCTETS, skb->len); return dst_output(net, sk, skb); } diff --git a/net/ipv6/ping.c b/net/ipv6/ping.c index 1b2772834972..5831aaa53d75 100644 --- a/net/ipv6/ping.c +++ b/net/ipv6/ping.c @@ -119,7 +119,7 @@ static int ping_v6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) return -EINVAL; ipcm6_init_sk(&ipc6, np); - ipc6.sockc.tsflags = sk->sk_tsflags; + ipc6.sockc.tsflags = READ_ONCE(sk->sk_tsflags); ipc6.sockc.mark = READ_ONCE(sk->sk_mark); fl6.flowi6_oif = oif; diff --git a/net/ipv6/raw.c b/net/ipv6/raw.c index 0eae7661a85c..42fcec3ecf5e 100644 --- a/net/ipv6/raw.c +++ b/net/ipv6/raw.c @@ -772,7 +772,7 @@ static int rawv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) fl6.flowi6_uid = sk->sk_uid; ipcm6_init(&ipc6); - ipc6.sockc.tsflags = sk->sk_tsflags; + ipc6.sockc.tsflags = READ_ONCE(sk->sk_tsflags); ipc6.sockc.mark = fl6.flowi6_mark; if (sin6) { diff --git a/net/ipv6/route.c b/net/ipv6/route.c index 2b95ebaa41fd..18469d8acee5 100644 --- a/net/ipv6/route.c +++ b/net/ipv6/route.c @@ -423,6 +423,9 @@ void fib6_select_path(const struct net *net, struct fib6_result *res, if (match->nh && have_oif_match && res->nh) return; + if (skb) + IP6CB(skb)->flags |= IP6SKB_MULTIPATH; + /* We might have already computed the hash for ICMPv6 errors. In such * case it will always be non-zero. Otherwise now is the time to do it. */ diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index ebc6ae47cfea..86b5d509a468 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1339,7 +1339,7 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) ipcm6_init(&ipc6); ipc6.gso_size = READ_ONCE(up->gso_size); - ipc6.sockc.tsflags = sk->sk_tsflags; + ipc6.sockc.tsflags = READ_ONCE(sk->sk_tsflags); ipc6.sockc.mark = READ_ONCE(sk->sk_mark); /* destination address check */ diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c index 393f01b2a7e6..4580f61426bb 100644 --- a/net/kcm/kcmsock.c +++ b/net/kcm/kcmsock.c @@ -1859,6 +1859,8 @@ static __net_exit void kcm_exit_net(struct net *net) * that all multiplexors and psocks have been destroyed. */ WARN_ON(!list_empty(&knet->mux_list)); + + mutex_destroy(&knet->mutex); } static struct pernet_operations kcm_net_ops = { diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 933b257eee02..a7fc16f5175d 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -134,9 +134,15 @@ static void mptcp_drop(struct sock *sk, struct sk_buff *skb) __kfree_skb(skb); } +static void mptcp_rmem_fwd_alloc_add(struct sock *sk, int size) +{ + WRITE_ONCE(mptcp_sk(sk)->rmem_fwd_alloc, + mptcp_sk(sk)->rmem_fwd_alloc + size); +} + static void mptcp_rmem_charge(struct sock *sk, int size) { - mptcp_sk(sk)->rmem_fwd_alloc -= size; + mptcp_rmem_fwd_alloc_add(sk, -size); } static bool mptcp_try_coalesce(struct sock *sk, struct sk_buff *to, @@ -177,7 +183,7 @@ static bool mptcp_ooo_try_coalesce(struct mptcp_sock *msk, struct sk_buff *to, static void __mptcp_rmem_reclaim(struct sock *sk, int amount) { amount >>= PAGE_SHIFT; - mptcp_sk(sk)->rmem_fwd_alloc -= amount << PAGE_SHIFT; + mptcp_rmem_charge(sk, amount << PAGE_SHIFT); __sk_mem_reduce_allocated(sk, amount); } @@ -186,7 +192,7 @@ static void mptcp_rmem_uncharge(struct sock *sk, int size) struct mptcp_sock *msk = mptcp_sk(sk); int reclaimable; - msk->rmem_fwd_alloc += size; + mptcp_rmem_fwd_alloc_add(sk, size); reclaimable = msk->rmem_fwd_alloc - sk_unused_reserved_mem(sk); /* see sk_mem_uncharge() for the rationale behind the following schema */ @@ -341,7 +347,7 @@ static bool mptcp_rmem_schedule(struct sock *sk, struct sock *ssk, int size) if (!__sk_mem_raise_allocated(sk, size, amt, SK_MEM_RECV)) return false; - msk->rmem_fwd_alloc += amount; + mptcp_rmem_fwd_alloc_add(sk, amount); return true; } @@ -1800,7 +1806,7 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) } /* data successfully copied into the write queue */ - sk->sk_forward_alloc -= total_ts; + sk_forward_alloc_add(sk, -total_ts); copied += psize; dfrag->data_len += psize; frag_truesize += psize; @@ -3257,8 +3263,8 @@ void mptcp_destroy_common(struct mptcp_sock *msk, unsigned int flags) /* move all the rx fwd alloc into the sk_mem_reclaim_final in * inet_sock_destruct() will dispose it */ - sk->sk_forward_alloc += msk->rmem_fwd_alloc; - msk->rmem_fwd_alloc = 0; + sk_forward_alloc_add(sk, msk->rmem_fwd_alloc); + WRITE_ONCE(msk->rmem_fwd_alloc, 0); mptcp_token_destroy(msk); mptcp_pm_free_anno_list(msk); mptcp_free_local_addr_list(msk); @@ -3522,7 +3528,8 @@ static void mptcp_shutdown(struct sock *sk, int how) static int mptcp_forward_alloc_get(const struct sock *sk) { - return sk->sk_forward_alloc + mptcp_sk(sk)->rmem_fwd_alloc; + return READ_ONCE(sk->sk_forward_alloc) + + READ_ONCE(mptcp_sk(sk)->rmem_fwd_alloc); } static int mptcp_ioctl_outq(const struct mptcp_sock *msk, u64 v) diff --git a/net/netfilter/ipset/ip_set_hash_netportnet.c b/net/netfilter/ipset/ip_set_hash_netportnet.c index 005a7ce87217..bf4f91b78e1d 100644 --- a/net/netfilter/ipset/ip_set_hash_netportnet.c +++ b/net/netfilter/ipset/ip_set_hash_netportnet.c @@ -36,6 +36,7 @@ MODULE_ALIAS("ip_set_hash:net,port,net"); #define IP_SET_HASH_WITH_PROTO #define IP_SET_HASH_WITH_NETS #define IPSET_NET_COUNT 2 +#define IP_SET_HASH_WITH_NET0 /* IPv4 variant */ diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c index 41b826dff6f5..e429ebba74b3 100644 --- a/net/netfilter/nf_tables_api.c +++ b/net/netfilter/nf_tables_api.c @@ -102,6 +102,7 @@ static const u8 nft2audit_op[NFT_MSG_MAX] = { // enum nf_tables_msg_types [NFT_MSG_NEWFLOWTABLE] = AUDIT_NFT_OP_FLOWTABLE_REGISTER, [NFT_MSG_GETFLOWTABLE] = AUDIT_NFT_OP_INVALID, [NFT_MSG_DELFLOWTABLE] = AUDIT_NFT_OP_FLOWTABLE_UNREGISTER, + [NFT_MSG_GETSETELEM_RESET] = AUDIT_NFT_OP_SETELEM_RESET, }; static void nft_validate_state_update(struct nft_table *table, u8 new_validate_state) @@ -3421,6 +3422,18 @@ err: nfnetlink_set_err(ctx->net, ctx->portid, NFNLGRP_NFTABLES, -ENOBUFS); } +static void audit_log_rule_reset(const struct nft_table *table, + unsigned int base_seq, + unsigned int nentries) +{ + char *buf = kasprintf(GFP_ATOMIC, "%s:%u", + table->name, base_seq); + + audit_log_nfcfg(buf, table->family, nentries, + AUDIT_NFT_OP_RULE_RESET, GFP_ATOMIC); + kfree(buf); +} + struct nft_rule_dump_ctx { char *table; char *chain; @@ -3467,6 +3480,10 @@ cont: cont_skip: (*idx)++; } + + if (reset && *idx) + audit_log_rule_reset(table, cb->seq, *idx); + return 0; } @@ -3634,6 +3651,9 @@ static int nf_tables_getrule(struct sk_buff *skb, const struct nfnl_info *info, if (err < 0) goto err_fill_rule_info; + if (reset) + audit_log_rule_reset(table, nft_pernet(net)->base_seq, 1); + return nfnetlink_unicast(skb2, net, NETLINK_CB(skb).portid); err_fill_rule_info: @@ -5624,13 +5644,25 @@ static int nf_tables_dump_setelem(const struct nft_ctx *ctx, return nf_tables_fill_setelem(args->skb, set, elem, args->reset); } +static void audit_log_nft_set_reset(const struct nft_table *table, + unsigned int base_seq, + unsigned int nentries) +{ + char *buf = kasprintf(GFP_ATOMIC, "%s:%u", table->name, base_seq); + + audit_log_nfcfg(buf, table->family, nentries, + AUDIT_NFT_OP_SETELEM_RESET, GFP_ATOMIC); + kfree(buf); +} + struct nft_set_dump_ctx { const struct nft_set *set; struct nft_ctx ctx; }; static int nft_set_catchall_dump(struct net *net, struct sk_buff *skb, - const struct nft_set *set, bool reset) + const struct nft_set *set, bool reset, + unsigned int base_seq) { struct nft_set_elem_catchall *catchall; u8 genmask = nft_genmask_cur(net); @@ -5646,6 +5678,8 @@ static int nft_set_catchall_dump(struct net *net, struct sk_buff *skb, elem.priv = catchall->elem; ret = nf_tables_fill_setelem(skb, set, &elem, reset); + if (reset && !ret) + audit_log_nft_set_reset(set->table, base_seq, 1); break; } @@ -5725,12 +5759,17 @@ static int nf_tables_dump_set(struct sk_buff *skb, struct netlink_callback *cb) set->ops->walk(&dump_ctx->ctx, set, &args.iter); if (!args.iter.err && args.iter.count == cb->args[0]) - args.iter.err = nft_set_catchall_dump(net, skb, set, reset); - rcu_read_unlock(); - + args.iter.err = nft_set_catchall_dump(net, skb, set, + reset, cb->seq); nla_nest_end(skb, nest); nlmsg_end(skb, nlh); + if (reset && args.iter.count > args.iter.skip) + audit_log_nft_set_reset(table, cb->seq, + args.iter.count - args.iter.skip); + + rcu_read_unlock(); + if (args.iter.err && args.iter.err != -EMSGSIZE) return args.iter.err; if (args.iter.count == cb->args[0]) @@ -5955,13 +5994,13 @@ static int nf_tables_getsetelem(struct sk_buff *skb, struct netlink_ext_ack *extack = info->extack; u8 genmask = nft_genmask_cur(info->net); u8 family = info->nfmsg->nfgen_family; + int rem, err = 0, nelems = 0; struct net *net = info->net; struct nft_table *table; struct nft_set *set; struct nlattr *attr; struct nft_ctx ctx; bool reset = false; - int rem, err = 0; table = nft_table_lookup(net, nla[NFTA_SET_ELEM_LIST_TABLE], family, genmask, 0); @@ -6004,8 +6043,13 @@ static int nf_tables_getsetelem(struct sk_buff *skb, NL_SET_BAD_ATTR(extack, attr); break; } + nelems++; } + if (reset) + audit_log_nft_set_reset(table, nft_pernet(net)->base_seq, + nelems); + return err; } diff --git a/net/netfilter/nfnetlink_osf.c b/net/netfilter/nfnetlink_osf.c index 8f1bfa6ccc2d..50723ba08289 100644 --- a/net/netfilter/nfnetlink_osf.c +++ b/net/netfilter/nfnetlink_osf.c @@ -315,6 +315,14 @@ static int nfnl_osf_add_callback(struct sk_buff *skb, f = nla_data(osf_attrs[OSF_ATTR_FINGER]); + if (f->opt_num > ARRAY_SIZE(f->opt)) + return -EINVAL; + + if (!memchr(f->genre, 0, MAXGENRELEN) || + !memchr(f->subtype, 0, MAXGENRELEN) || + !memchr(f->version, 0, MAXGENRELEN)) + return -EINVAL; + kf = kmalloc(sizeof(struct nf_osf_finger), GFP_KERNEL); if (!kf) return -ENOMEM; diff --git a/net/netfilter/nft_exthdr.c b/net/netfilter/nft_exthdr.c index 7f856ceb3a66..3fbaa7bf41f9 100644 --- a/net/netfilter/nft_exthdr.c +++ b/net/netfilter/nft_exthdr.c @@ -35,6 +35,14 @@ static unsigned int optlen(const u8 *opt, unsigned int offset) return opt[offset + 1]; } +static int nft_skb_copy_to_reg(const struct sk_buff *skb, int offset, u32 *dest, unsigned int len) +{ + if (len % NFT_REG32_SIZE) + dest[len / NFT_REG32_SIZE] = 0; + + return skb_copy_bits(skb, offset, dest, len); +} + static void nft_exthdr_ipv6_eval(const struct nft_expr *expr, struct nft_regs *regs, const struct nft_pktinfo *pkt) @@ -56,8 +64,7 @@ static void nft_exthdr_ipv6_eval(const struct nft_expr *expr, } offset += priv->offset; - dest[priv->len / NFT_REG32_SIZE] = 0; - if (skb_copy_bits(pkt->skb, offset, dest, priv->len) < 0) + if (nft_skb_copy_to_reg(pkt->skb, offset, dest, priv->len) < 0) goto err; return; err: @@ -153,8 +160,7 @@ static void nft_exthdr_ipv4_eval(const struct nft_expr *expr, } offset += priv->offset; - dest[priv->len / NFT_REG32_SIZE] = 0; - if (skb_copy_bits(pkt->skb, offset, dest, priv->len) < 0) + if (nft_skb_copy_to_reg(pkt->skb, offset, dest, priv->len) < 0) goto err; return; err: @@ -210,7 +216,8 @@ static void nft_exthdr_tcp_eval(const struct nft_expr *expr, if (priv->flags & NFT_EXTHDR_F_PRESENT) { *dest = 1; } else { - dest[priv->len / NFT_REG32_SIZE] = 0; + if (priv->len % NFT_REG32_SIZE) + dest[priv->len / NFT_REG32_SIZE] = 0; memcpy(dest, opt + offset, priv->len); } @@ -238,7 +245,12 @@ static void nft_exthdr_tcp_set_eval(const struct nft_expr *expr, if (!tcph) goto err; + if (skb_ensure_writable(pkt->skb, nft_thoff(pkt) + tcphdr_len)) + goto err; + + tcph = (struct tcphdr *)(pkt->skb->data + nft_thoff(pkt)); opt = (u8 *)tcph; + for (i = sizeof(*tcph); i < tcphdr_len - 1; i += optl) { union { __be16 v16; @@ -253,15 +265,6 @@ static void nft_exthdr_tcp_set_eval(const struct nft_expr *expr, if (i + optl > tcphdr_len || priv->len + priv->offset > optl) goto err; - if (skb_ensure_writable(pkt->skb, - nft_thoff(pkt) + i + priv->len)) - goto err; - - tcph = nft_tcp_header_pointer(pkt, sizeof(buff), buff, - &tcphdr_len); - if (!tcph) - goto err; - offset = i + priv->offset; switch (priv->len) { @@ -325,9 +328,9 @@ static void nft_exthdr_tcp_strip_eval(const struct nft_expr *expr, if (skb_ensure_writable(pkt->skb, nft_thoff(pkt) + tcphdr_len)) goto drop; - opt = (u8 *)nft_tcp_header_pointer(pkt, sizeof(buff), buff, &tcphdr_len); - if (!opt) - goto err; + tcph = (struct tcphdr *)(pkt->skb->data + nft_thoff(pkt)); + opt = (u8 *)tcph; + for (i = sizeof(*tcph); i < tcphdr_len - 1; i += optl) { unsigned int j; @@ -392,9 +395,8 @@ static void nft_exthdr_sctp_eval(const struct nft_expr *expr, offset + ntohs(sch->length) > pkt->skb->len) break; - dest[priv->len / NFT_REG32_SIZE] = 0; - if (skb_copy_bits(pkt->skb, offset + priv->offset, - dest, priv->len) < 0) + if (nft_skb_copy_to_reg(pkt->skb, offset + priv->offset, + dest, priv->len) < 0) break; return; } diff --git a/net/netfilter/nft_set_rbtree.c b/net/netfilter/nft_set_rbtree.c index c6435e709231..f250b5399344 100644 --- a/net/netfilter/nft_set_rbtree.c +++ b/net/netfilter/nft_set_rbtree.c @@ -312,6 +312,7 @@ static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set, struct nft_rbtree_elem *rbe, *rbe_le = NULL, *rbe_ge = NULL; struct rb_node *node, *next, *parent, **p, *first = NULL; struct nft_rbtree *priv = nft_set_priv(set); + u8 cur_genmask = nft_genmask_cur(net); u8 genmask = nft_genmask_next(net); int d, err; @@ -357,8 +358,11 @@ static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set, if (!nft_set_elem_active(&rbe->ext, genmask)) continue; - /* perform garbage collection to avoid bogus overlap reports. */ - if (nft_set_elem_expired(&rbe->ext)) { + /* perform garbage collection to avoid bogus overlap reports + * but skip new elements in this transaction. + */ + if (nft_set_elem_expired(&rbe->ext) && + nft_set_elem_active(&rbe->ext, cur_genmask)) { err = nft_rbtree_gc_elem(set, priv, rbe, genmask); if (err < 0) return err; diff --git a/net/netfilter/xt_sctp.c b/net/netfilter/xt_sctp.c index e8961094a282..b46a6a512058 100644 --- a/net/netfilter/xt_sctp.c +++ b/net/netfilter/xt_sctp.c @@ -149,6 +149,8 @@ static int sctp_mt_check(const struct xt_mtchk_param *par) { const struct xt_sctp_info *info = par->matchinfo; + if (info->flag_count > ARRAY_SIZE(info->flag_info)) + return -EINVAL; if (info->flags & ~XT_SCTP_VALID_FLAGS) return -EINVAL; if (info->invflags & ~XT_SCTP_VALID_FLAGS) diff --git a/net/netfilter/xt_u32.c b/net/netfilter/xt_u32.c index 177b40d08098..117d4615d668 100644 --- a/net/netfilter/xt_u32.c +++ b/net/netfilter/xt_u32.c @@ -96,11 +96,32 @@ static bool u32_mt(const struct sk_buff *skb, struct xt_action_param *par) return ret ^ data->invert; } +static int u32_mt_checkentry(const struct xt_mtchk_param *par) +{ + const struct xt_u32 *data = par->matchinfo; + const struct xt_u32_test *ct; + unsigned int i; + + if (data->ntests > ARRAY_SIZE(data->tests)) + return -EINVAL; + + for (i = 0; i < data->ntests; ++i) { + ct = &data->tests[i]; + + if (ct->nnums > ARRAY_SIZE(ct->location) || + ct->nvalues > ARRAY_SIZE(ct->value)) + return -EINVAL; + } + + return 0; +} + static struct xt_match xt_u32_mt_reg __read_mostly = { .name = "u32", .revision = 0, .family = NFPROTO_UNSPEC, .match = u32_mt, + .checkentry = u32_mt_checkentry, .matchsize = sizeof(struct xt_u32), .me = THIS_MODULE, }; diff --git a/net/sched/sch_fq_pie.c b/net/sched/sch_fq_pie.c index 591d87d5e5c0..68e6acd0f130 100644 --- a/net/sched/sch_fq_pie.c +++ b/net/sched/sch_fq_pie.c @@ -61,6 +61,7 @@ struct fq_pie_sched_data { struct pie_params p_params; u32 ecn_prob; u32 flows_cnt; + u32 flows_cursor; u32 quantum; u32 memory_limit; u32 new_flow_count; @@ -375,22 +376,32 @@ flow_error: static void fq_pie_timer(struct timer_list *t) { struct fq_pie_sched_data *q = from_timer(q, t, adapt_timer); + unsigned long next, tupdate; struct Qdisc *sch = q->sch; spinlock_t *root_lock; /* to lock qdisc for probability calculations */ - u32 idx; + int max_cnt, i; rcu_read_lock(); root_lock = qdisc_lock(qdisc_root_sleeping(sch)); spin_lock(root_lock); - for (idx = 0; idx < q->flows_cnt; idx++) - pie_calculate_probability(&q->p_params, &q->flows[idx].vars, - q->flows[idx].backlog); - - /* reset the timer to fire after 'tupdate' jiffies. */ - if (q->p_params.tupdate) - mod_timer(&q->adapt_timer, jiffies + q->p_params.tupdate); + /* Limit this expensive loop to 2048 flows per round. */ + max_cnt = min_t(int, q->flows_cnt - q->flows_cursor, 2048); + for (i = 0; i < max_cnt; i++) { + pie_calculate_probability(&q->p_params, + &q->flows[q->flows_cursor].vars, + q->flows[q->flows_cursor].backlog); + q->flows_cursor++; + } + tupdate = q->p_params.tupdate; + next = 0; + if (q->flows_cursor >= q->flows_cnt) { + q->flows_cursor = 0; + next = tupdate; + } + if (tupdate) + mod_timer(&q->adapt_timer, jiffies + next); spin_unlock(root_lock); rcu_read_unlock(); } diff --git a/net/sched/sch_plug.c b/net/sched/sch_plug.c index ea8c4a7174bb..35f49edf63db 100644 --- a/net/sched/sch_plug.c +++ b/net/sched/sch_plug.c @@ -207,7 +207,7 @@ static struct Qdisc_ops plug_qdisc_ops __read_mostly = { .priv_size = sizeof(struct plug_sched_data), .enqueue = plug_enqueue, .dequeue = plug_dequeue, - .peek = qdisc_peek_head, + .peek = qdisc_peek_dequeued, .init = plug_init, .change = plug_change, .reset = qdisc_reset_queue, diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c index 1a25752f1a9a..546c10adcacd 100644 --- a/net/sched/sch_qfq.c +++ b/net/sched/sch_qfq.c @@ -974,10 +974,13 @@ static void qfq_update_eligible(struct qfq_sched *q) } /* Dequeue head packet of the head class in the DRR queue of the aggregate. */ -static void agg_dequeue(struct qfq_aggregate *agg, - struct qfq_class *cl, unsigned int len) +static struct sk_buff *agg_dequeue(struct qfq_aggregate *agg, + struct qfq_class *cl, unsigned int len) { - qdisc_dequeue_peeked(cl->qdisc); + struct sk_buff *skb = qdisc_dequeue_peeked(cl->qdisc); + + if (!skb) + return NULL; cl->deficit -= (int) len; @@ -987,6 +990,8 @@ static void agg_dequeue(struct qfq_aggregate *agg, cl->deficit += agg->lmax; list_move_tail(&cl->alist, &agg->active); } + + return skb; } static inline struct sk_buff *qfq_peek_skb(struct qfq_aggregate *agg, @@ -1132,11 +1137,18 @@ static struct sk_buff *qfq_dequeue(struct Qdisc *sch) if (!skb) return NULL; - qdisc_qstats_backlog_dec(sch, skb); sch->q.qlen--; + + skb = agg_dequeue(in_serv_agg, cl, len); + + if (!skb) { + sch->q.qlen++; + return NULL; + } + + qdisc_qstats_backlog_dec(sch, skb); qdisc_bstats_update(sch, skb); - agg_dequeue(in_serv_agg, cl, len); /* If lmax is lowered, through qfq_change_class, for a class * owning pending packets with larger size than the new value * of lmax, then the following condition may hold. diff --git a/net/sctp/proc.c b/net/sctp/proc.c index f13d6a34f32f..ec00ee75d59a 100644 --- a/net/sctp/proc.c +++ b/net/sctp/proc.c @@ -282,7 +282,7 @@ static int sctp_assocs_seq_show(struct seq_file *seq, void *v) assoc->init_retries, assoc->shutdown_retries, assoc->rtx_data_chunks, refcount_read(&sk->sk_wmem_alloc), - sk->sk_wmem_queued, + READ_ONCE(sk->sk_wmem_queued), sk->sk_sndbuf, sk->sk_rcvbuf); seq_printf(seq, "\n"); diff --git a/net/sctp/socket.c b/net/sctp/socket.c index fd0631e70d46..ab943e8fb1db 100644 --- a/net/sctp/socket.c +++ b/net/sctp/socket.c @@ -69,7 +69,7 @@ #include /* Forward declarations for internal helper functions. */ -static bool sctp_writeable(struct sock *sk); +static bool sctp_writeable(const struct sock *sk); static void sctp_wfree(struct sk_buff *skb); static int sctp_wait_for_sndbuf(struct sctp_association *asoc, long *timeo_p, size_t msg_len); @@ -140,7 +140,7 @@ static inline void sctp_set_owner_w(struct sctp_chunk *chunk) refcount_add(sizeof(struct sctp_chunk), &sk->sk_wmem_alloc); asoc->sndbuf_used += chunk->skb->truesize + sizeof(struct sctp_chunk); - sk->sk_wmem_queued += chunk->skb->truesize + sizeof(struct sctp_chunk); + sk_wmem_queued_add(sk, chunk->skb->truesize + sizeof(struct sctp_chunk)); sk_mem_charge(sk, chunk->skb->truesize); } @@ -9144,7 +9144,7 @@ static void sctp_wfree(struct sk_buff *skb) struct sock *sk = asoc->base.sk; sk_mem_uncharge(sk, skb->truesize); - sk->sk_wmem_queued -= skb->truesize + sizeof(struct sctp_chunk); + sk_wmem_queued_add(sk, -(skb->truesize + sizeof(struct sctp_chunk))); asoc->sndbuf_used -= skb->truesize + sizeof(struct sctp_chunk); WARN_ON(refcount_sub_and_test(sizeof(struct sctp_chunk), &sk->sk_wmem_alloc)); @@ -9299,9 +9299,9 @@ void sctp_write_space(struct sock *sk) * UDP-style sockets or TCP-style sockets, this code should work. * - Daisy */ -static bool sctp_writeable(struct sock *sk) +static bool sctp_writeable(const struct sock *sk) { - return sk->sk_sndbuf > sk->sk_wmem_queued; + return READ_ONCE(sk->sk_sndbuf) > READ_ONCE(sk->sk_wmem_queued); } /* Wait for an association to go into ESTABLISHED state. If timeout is 0, diff --git a/net/socket.c b/net/socket.c index 77f28328e387..c8b08b32f097 100644 --- a/net/socket.c +++ b/net/socket.c @@ -827,7 +827,7 @@ static bool skb_is_swtx_tstamp(const struct sk_buff *skb, int false_tstamp) static ktime_t get_timestamp(struct sock *sk, struct sk_buff *skb, int *if_index) { - bool cycles = sk->sk_tsflags & SOF_TIMESTAMPING_BIND_PHC; + bool cycles = READ_ONCE(sk->sk_tsflags) & SOF_TIMESTAMPING_BIND_PHC; struct skb_shared_hwtstamps *shhwtstamps = skb_hwtstamps(skb); struct net_device *orig_dev; ktime_t hwtstamp; @@ -879,12 +879,12 @@ void __sock_recv_timestamp(struct msghdr *msg, struct sock *sk, int need_software_tstamp = sock_flag(sk, SOCK_RCVTSTAMP); int new_tstamp = sock_flag(sk, SOCK_TSTAMP_NEW); struct scm_timestamping_internal tss; - int empty = 1, false_tstamp = 0; struct skb_shared_hwtstamps *shhwtstamps = skb_hwtstamps(skb); int if_index; ktime_t hwtstamp; + u32 tsflags; /* Race occurred between timestamp enabling and packet receiving. Fill in the current time for now. */ @@ -926,11 +926,12 @@ void __sock_recv_timestamp(struct msghdr *msg, struct sock *sk, } memset(&tss, 0, sizeof(tss)); - if ((sk->sk_tsflags & SOF_TIMESTAMPING_SOFTWARE) && + tsflags = READ_ONCE(sk->sk_tsflags); + if ((tsflags & SOF_TIMESTAMPING_SOFTWARE) && ktime_to_timespec64_cond(skb->tstamp, tss.ts + 0)) empty = 0; if (shhwtstamps && - (sk->sk_tsflags & SOF_TIMESTAMPING_RAW_HARDWARE) && + (tsflags & SOF_TIMESTAMPING_RAW_HARDWARE) && !skb_is_swtx_tstamp(skb, false_tstamp)) { if_index = 0; if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP_NETDEV) @@ -938,14 +939,14 @@ void __sock_recv_timestamp(struct msghdr *msg, struct sock *sk, else hwtstamp = shhwtstamps->hwtstamp; - if (sk->sk_tsflags & SOF_TIMESTAMPING_BIND_PHC) + if (tsflags & SOF_TIMESTAMPING_BIND_PHC) hwtstamp = ptp_convert_timestamp(&hwtstamp, - sk->sk_bind_phc); + READ_ONCE(sk->sk_bind_phc)); if (ktime_to_timespec64_cond(hwtstamp, tss.ts + 2)) { empty = 0; - if ((sk->sk_tsflags & SOF_TIMESTAMPING_OPT_PKTINFO) && + if ((tsflags & SOF_TIMESTAMPING_OPT_PKTINFO) && !skb_is_err_queue(skb)) put_ts_pktinfo(msg, skb, if_index); } diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 86930a8ed012..3e8a04a13668 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -680,7 +680,7 @@ static void unix_release_sock(struct sock *sk, int embrion) * What the above comment does talk about? --ANK(980817) */ - if (unix_tot_inflight) + if (READ_ONCE(unix_tot_inflight)) unix_gc(); /* Garbage collect fds */ } diff --git a/net/unix/scm.c b/net/unix/scm.c index e9dde7176c8a..6ff628f2349f 100644 --- a/net/unix/scm.c +++ b/net/unix/scm.c @@ -64,7 +64,7 @@ void unix_inflight(struct user_struct *user, struct file *fp) /* Paired with READ_ONCE() in wait_for_unix_gc() */ WRITE_ONCE(unix_tot_inflight, unix_tot_inflight + 1); } - user->unix_inflight++; + WRITE_ONCE(user->unix_inflight, user->unix_inflight + 1); spin_unlock(&unix_gc_lock); } @@ -85,7 +85,7 @@ void unix_notinflight(struct user_struct *user, struct file *fp) /* Paired with READ_ONCE() in wait_for_unix_gc() */ WRITE_ONCE(unix_tot_inflight, unix_tot_inflight - 1); } - user->unix_inflight--; + WRITE_ONCE(user->unix_inflight, user->unix_inflight - 1); spin_unlock(&unix_gc_lock); } @@ -99,7 +99,7 @@ static inline bool too_many_unix_fds(struct task_struct *p) { struct user_struct *user = current_user(); - if (unlikely(user->unix_inflight > task_rlimit(p, RLIMIT_NOFILE))) + if (unlikely(READ_ONCE(user->unix_inflight) > task_rlimit(p, RLIMIT_NOFILE))) return !capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN); return false; } diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index fcfc8472f73d..55f8b9b0e06d 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -602,7 +602,7 @@ static struct sk_buff *xsk_build_skb_zerocopy(struct xdp_sock *xs, for (copied = 0, i = skb_shinfo(skb)->nr_frags; copied < len; i++) { if (unlikely(i >= MAX_SKB_FRAGS)) - return ERR_PTR(-EFAULT); + return ERR_PTR(-EOVERFLOW); page = pool->umem->pgs[addr >> PAGE_SHIFT]; get_page(page); @@ -655,15 +655,17 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, skb_put(skb, len); err = skb_store_bits(skb, 0, buffer, len); - if (unlikely(err)) + if (unlikely(err)) { + kfree_skb(skb); goto free_err; + } } else { int nr_frags = skb_shinfo(skb)->nr_frags; struct page *page; u8 *vaddr; if (unlikely(nr_frags == (MAX_SKB_FRAGS - 1) && xp_mb_desc(desc))) { - err = -EFAULT; + err = -EOVERFLOW; goto free_err; } @@ -690,12 +692,14 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, return skb; free_err: - if (err == -EAGAIN) { - xsk_cq_cancel_locked(xs, 1); - } else { - xsk_set_destructor_arg(skb); - xsk_drop_skb(skb); + if (err == -EOVERFLOW) { + /* Drop the packet */ + xsk_set_destructor_arg(xs->skb); + xsk_drop_skb(xs->skb); xskq_cons_release(xs->tx); + } else { + /* Let application retry */ + xsk_cq_cancel_locked(xs, 1); } return ERR_PTR(err); @@ -738,7 +742,7 @@ static int __xsk_generic_xmit(struct sock *sk) skb = xsk_build_skb(xs, &desc); if (IS_ERR(skb)) { err = PTR_ERR(skb); - if (err == -EAGAIN) + if (err != -EOVERFLOW) goto out; err = 0; continue; diff --git a/net/xdp/xsk_diag.c b/net/xdp/xsk_diag.c index c014217f5fa7..22b36c8143cf 100644 --- a/net/xdp/xsk_diag.c +++ b/net/xdp/xsk_diag.c @@ -111,6 +111,9 @@ static int xsk_diag_fill(struct sock *sk, struct sk_buff *nlskb, sock_diag_save_cookie(sk, msg->xdiag_cookie); mutex_lock(&xs->mutex); + if (READ_ONCE(xs->state) == XSK_UNBOUND) + goto out_nlmsg_trim; + if ((req->xdiag_show & XDP_SHOW_INFO) && xsk_diag_put_info(xs, nlskb)) goto out_nlmsg_trim; diff --git a/scripts/bpf_doc.py b/scripts/bpf_doc.py index eaae2ce78381..61b7dddedc46 100755 --- a/scripts/bpf_doc.py +++ b/scripts/bpf_doc.py @@ -59,9 +59,9 @@ class Helper(APIElement): Break down helper function protocol into smaller chunks: return type, name, distincts arguments. """ - arg_re = re.compile('((\w+ )*?(\w+|...))( (\**)(\w+))?$') + arg_re = re.compile(r'((\w+ )*?(\w+|...))( (\**)(\w+))?$') res = {} - proto_re = re.compile('(.+) (\**)(\w+)\(((([^,]+)(, )?){1,5})\)$') + proto_re = re.compile(r'(.+) (\**)(\w+)\(((([^,]+)(, )?){1,5})\)$') capture = proto_re.match(self.proto) res['ret_type'] = capture.group(1) @@ -114,11 +114,11 @@ class HeaderParser(object): return Helper(proto=proto, desc=desc, ret=ret) def parse_symbol(self): - p = re.compile(' \* ?(BPF\w+)$') + p = re.compile(r' \* ?(BPF\w+)$') capture = p.match(self.line) if not capture: raise NoSyscallCommandFound - end_re = re.compile(' \* ?NOTES$') + end_re = re.compile(r' \* ?NOTES$') end = end_re.match(self.line) if end: raise NoSyscallCommandFound @@ -133,7 +133,7 @@ class HeaderParser(object): # - Same as above, with "const" and/or "struct" in front of type # - "..." (undefined number of arguments, for bpf_trace_printk()) # There is at least one term ("void"), and at most five arguments. - p = re.compile(' \* ?((.+) \**\w+\((((const )?(struct )?(\w+|\.\.\.)( \**\w+)?)(, )?){1,5}\))$') + p = re.compile(r' \* ?((.+) \**\w+\((((const )?(struct )?(\w+|\.\.\.)( \**\w+)?)(, )?){1,5}\))$') capture = p.match(self.line) if not capture: raise NoHelperFound @@ -141,7 +141,7 @@ class HeaderParser(object): return capture.group(1) def parse_desc(self, proto): - p = re.compile(' \* ?(?:\t| {5,8})Description$') + p = re.compile(r' \* ?(?:\t| {5,8})Description$') capture = p.match(self.line) if not capture: raise Exception("No description section found for " + proto) @@ -154,7 +154,7 @@ class HeaderParser(object): if self.line == ' *\n': desc += '\n' else: - p = re.compile(' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') + p = re.compile(r' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') capture = p.match(self.line) if capture: desc_present = True @@ -167,7 +167,7 @@ class HeaderParser(object): return desc def parse_ret(self, proto): - p = re.compile(' \* ?(?:\t| {5,8})Return$') + p = re.compile(r' \* ?(?:\t| {5,8})Return$') capture = p.match(self.line) if not capture: raise Exception("No return section found for " + proto) @@ -180,7 +180,7 @@ class HeaderParser(object): if self.line == ' *\n': ret += '\n' else: - p = re.compile(' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') + p = re.compile(r' \* ?(?:\t| {5,8})(?:\t| {8})(.*)') capture = p.match(self.line) if capture: ret_present = True @@ -219,12 +219,12 @@ class HeaderParser(object): self.seek_to('enum bpf_cmd {', 'Could not find start of bpf_cmd enum', 0) # Searches for either one or more BPF\w+ enums - bpf_p = re.compile('\s*(BPF\w+)+') + bpf_p = re.compile(r'\s*(BPF\w+)+') # Searches for an enum entry assigned to another entry, # for e.g. BPF_PROG_RUN = BPF_PROG_TEST_RUN, which is # not documented hence should be skipped in check to # determine if the right number of syscalls are documented - assign_p = re.compile('\s*(BPF\w+)\s*=\s*(BPF\w+)') + assign_p = re.compile(r'\s*(BPF\w+)\s*=\s*(BPF\w+)') bpf_cmd_str = '' while True: capture = assign_p.match(self.line) @@ -239,7 +239,7 @@ class HeaderParser(object): break self.line = self.reader.readline() # Find the number of occurences of BPF\w+ - self.enum_syscalls = re.findall('(BPF\w+)+', bpf_cmd_str) + self.enum_syscalls = re.findall(r'(BPF\w+)+', bpf_cmd_str) def parse_desc_helpers(self): self.seek_to(helpersDocStart, @@ -263,7 +263,7 @@ class HeaderParser(object): self.seek_to('#define ___BPF_FUNC_MAPPER(FN, ctx...)', 'Could not find start of eBPF helper definition list') # Searches for one FN(\w+) define or a backslash for newline - p = re.compile('\s*FN\((\w+), (\d+), ##ctx\)|\\\\') + p = re.compile(r'\s*FN\((\w+), (\d+), ##ctx\)|\\\\') fn_defines_str = '' i = 0 while True: @@ -278,7 +278,7 @@ class HeaderParser(object): break self.line = self.reader.readline() # Find the number of occurences of FN(\w+) - self.define_unique_helpers = re.findall('FN\(\w+, \d+, ##ctx\)', fn_defines_str) + self.define_unique_helpers = re.findall(r'FN\(\w+, \d+, ##ctx\)', fn_defines_str) def validate_helpers(self): last_helper = '' @@ -425,7 +425,7 @@ class PrinterRST(Printer): try: cmd = ['git', 'log', '-1', '--pretty=format:%cs', '--no-patch', '-L', - '/{}/,/\*\//:include/uapi/linux/bpf.h'.format(delimiter)] + '/{}/,/\\*\\//:include/uapi/linux/bpf.h'.format(delimiter)] date = subprocess.run(cmd, cwd=linuxRoot, capture_output=True, check=True) return date.stdout.decode().rstrip() @@ -516,7 +516,7 @@ as "Dual BSD/GPL", may be used). Some helper functions are only accessible to programs that are compatible with the GNU Privacy License (GPL). In order to use such helpers, the eBPF program must be loaded with the correct -license string passed (via **attr**) to the **bpf**\ () system call, and this +license string passed (via **attr**) to the **bpf**\\ () system call, and this generally translates into the C source code of the program containing a line similar to the following: @@ -550,7 +550,7 @@ may be interested in: * The bpftool utility can be used to probe the availability of helper functions on the system (as well as supported program and map types, and a number of other parameters). To do so, run **bpftool feature probe** (see - **bpftool-feature**\ (8) for details). Add the **unprivileged** keyword to + **bpftool-feature**\\ (8) for details). Add the **unprivileged** keyword to list features available to unprivileged users. Compatibility between helper functions and program types can generally be found @@ -562,23 +562,23 @@ other functions, themselves allowing access to additional helpers. The requirement for GPL license is also in those **struct bpf_func_proto**. Compatibility between helper functions and map types can be found in the -**check_map_func_compatibility**\ () function in file *kernel/bpf/verifier.c*. +**check_map_func_compatibility**\\ () function in file *kernel/bpf/verifier.c*. Helper functions that invalidate the checks on **data** and **data_end** pointers for network processing are listed in function -**bpf_helper_changes_pkt_data**\ () in file *net/core/filter.c*. +**bpf_helper_changes_pkt_data**\\ () in file *net/core/filter.c*. SEE ALSO ======== -**bpf**\ (2), -**bpftool**\ (8), -**cgroups**\ (7), -**ip**\ (8), -**perf_event_open**\ (2), -**sendmsg**\ (2), -**socket**\ (7), -**tc-bpf**\ (8)''' +**bpf**\\ (2), +**bpftool**\\ (8), +**cgroups**\\ (7), +**ip**\\ (8), +**perf_event_open**\\ (2), +**sendmsg**\\ (2), +**socket**\\ (7), +**tc-bpf**\\ (8)''' print(footer) def print_proto(self, helper): @@ -598,7 +598,7 @@ SEE ALSO one_arg = '{}{}'.format(comma, a['type']) if a['name']: if a['star']: - one_arg += ' {}**\ '.format(a['star'].replace('*', '\\*')) + one_arg += ' {}**\\ '.format(a['star'].replace('*', '\\*')) else: one_arg += '** ' one_arg += '*{}*\\ **'.format(a['name']) diff --git a/scripts/headers_install.sh b/scripts/headers_install.sh index afdddc82f02b..56d3c338d91d 100755 --- a/scripts/headers_install.sh +++ b/scripts/headers_install.sh @@ -81,7 +81,6 @@ arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_NO arch/nios2/include/uapi/asm/swab.h:CONFIG_NIOS2_CI_SWAB_SUPPORT arch/x86/include/uapi/asm/auxvec.h:CONFIG_IA32_EMULATION arch/x86/include/uapi/asm/auxvec.h:CONFIG_X86_64 -arch/x86/include/uapi/asm/mman.h:CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS " for c in $configs diff --git a/security/landlock/ruleset.h b/security/landlock/ruleset.h index d43231b783e4..55b1df8f66a8 100644 --- a/security/landlock/ruleset.h +++ b/security/landlock/ruleset.h @@ -67,7 +67,7 @@ struct landlock_rule { * @layers: Stack of layers, from the latest to the newest, implemented * as a flexible array member (FAM). */ - struct landlock_layer layers[]; + struct landlock_layer layers[] __counted_by(num_layers); }; /** diff --git a/sound/core/pcm_lib.c b/sound/core/pcm_lib.c index 4859fb1caec9..a11cd7d6295f 100644 --- a/sound/core/pcm_lib.c +++ b/sound/core/pcm_lib.c @@ -1992,8 +1992,8 @@ static int default_write_copy(struct snd_pcm_substream *substream, int channel, unsigned long hwoff, struct iov_iter *iter, unsigned long bytes) { - if (!copy_from_iter(get_dma_ptr(substream->runtime, channel, hwoff), - bytes, iter)) + if (copy_from_iter(get_dma_ptr(substream->runtime, channel, hwoff), + bytes, iter) != bytes) return -EFAULT; return 0; } @@ -2025,8 +2025,8 @@ static int default_read_copy(struct snd_pcm_substream *substream, int channel, unsigned long hwoff, struct iov_iter *iter, unsigned long bytes) { - if (!copy_to_iter(get_dma_ptr(substream->runtime, channel, hwoff), - bytes, iter)) + if (copy_to_iter(get_dma_ptr(substream->runtime, channel, hwoff), + bytes, iter) != bytes) return -EFAULT; return 0; } diff --git a/sound/core/seq/seq_memory.c b/sound/core/seq/seq_memory.c index 174585bf59d2..b603bb93f896 100644 --- a/sound/core/seq/seq_memory.c +++ b/sound/core/seq/seq_memory.c @@ -187,8 +187,13 @@ int snd_seq_expand_var_event(const struct snd_seq_event *event, int count, char err = expand_var_event(event, 0, len, buf, in_kernel); if (err < 0) return err; - if (len != newlen) - memset(buf + len, 0, newlen - len); + if (len != newlen) { + if (in_kernel) + memset(buf + len, 0, newlen - len); + else if (clear_user((__force void __user *)buf + len, + newlen - len)) + return -EFAULT; + } return newlen; } EXPORT_SYMBOL(snd_seq_expand_var_event); diff --git a/sound/isa/sb/emu8000_pcm.c b/sound/isa/sb/emu8000_pcm.c index c05935c2edc4..9234d4fe8ada 100644 --- a/sound/isa/sb/emu8000_pcm.c +++ b/sound/isa/sb/emu8000_pcm.c @@ -456,7 +456,7 @@ static int emu8k_pcm_silence(struct snd_pcm_substream *subs, /* convert to word unit */ pos = (pos << 1) + rec->loop_start[voice]; count <<= 1; - LOOP_WRITE(rec, pos, USER_SOCKPTR(NULL), count); + LOOP_WRITE(rec, pos, NULL, count); return 0; } diff --git a/sound/pci/hda/patch_cs8409.c b/sound/pci/hda/patch_cs8409.c index 0ba1fbcbb21e..627899959ffe 100644 --- a/sound/pci/hda/patch_cs8409.c +++ b/sound/pci/hda/patch_cs8409.c @@ -888,7 +888,7 @@ static void cs42l42_resume(struct sub_codec *cs42l42) /* Initialize CS42L42 companion codec */ cs8409_i2c_bulk_write(cs42l42, cs42l42->init_seq, cs42l42->init_seq_num); - usleep_range(30000, 35000); + msleep(CS42L42_INIT_TIMEOUT_MS); /* Clear interrupts, by reading interrupt status registers */ cs8409_i2c_bulk_read(cs42l42, irq_regs, ARRAY_SIZE(irq_regs)); diff --git a/sound/pci/hda/patch_cs8409.h b/sound/pci/hda/patch_cs8409.h index 2a8dfb4ff046..937e9387abdc 100644 --- a/sound/pci/hda/patch_cs8409.h +++ b/sound/pci/hda/patch_cs8409.h @@ -229,6 +229,7 @@ enum cs8409_coefficient_index_registers { #define CS42L42_I2C_SLEEP_US (2000) #define CS42L42_PDN_TIMEOUT_US (250000) #define CS42L42_PDN_SLEEP_US (2000) +#define CS42L42_INIT_TIMEOUT_MS (45) #define CS42L42_FULL_SCALE_VOL_MASK (2) #define CS42L42_FULL_SCALE_VOL_0DB (1) #define CS42L42_FULL_SCALE_VOL_MINUS6DB (0) diff --git a/sound/pci/hda/patch_realtek.c b/sound/pci/hda/patch_realtek.c index a07df6f92960..b7e78bfcffd8 100644 --- a/sound/pci/hda/patch_realtek.c +++ b/sound/pci/hda/patch_realtek.c @@ -7057,6 +7057,27 @@ static void alc295_fixup_dell_inspiron_top_speakers(struct hda_codec *codec, } } +/* Forcibly assign NID 0x03 to HP while NID 0x02 to SPK */ +static void alc287_fixup_bind_dacs(struct hda_codec *codec, + const struct hda_fixup *fix, int action) +{ + struct alc_spec *spec = codec->spec; + static const hda_nid_t conn[] = { 0x02, 0x03 }; /* exclude 0x06 */ + static const hda_nid_t preferred_pairs[] = { + 0x17, 0x02, 0x21, 0x03, 0 + }; + + if (action != HDA_FIXUP_ACT_PRE_PROBE) + return; + + snd_hda_override_conn_list(codec, 0x17, ARRAY_SIZE(conn), conn); + spec->gen.preferred_dacs = preferred_pairs; + spec->gen.auto_mute_via_amp = 1; + snd_hda_codec_write_cache(codec, 0x14, 0, AC_VERB_SET_PIN_WIDGET_CONTROL, + 0x0); /* Make sure 0x14 was disable */ +} + + enum { ALC269_FIXUP_GPIO2, ALC269_FIXUP_SONY_VAIO, @@ -7319,6 +7340,7 @@ enum { ALC287_FIXUP_TAS2781_I2C, ALC245_FIXUP_HP_MUTE_LED_COEFBIT, ALC245_FIXUP_HP_X360_MUTE_LEDS, + ALC287_FIXUP_THINKPAD_I2S_SPK, }; /* A special fixup for Lenovo C940 and Yoga Duet 7; @@ -9413,6 +9435,10 @@ static const struct hda_fixup alc269_fixups[] = { .chained = true, .chain_id = ALC245_FIXUP_HP_GPIO_LED }, + [ALC287_FIXUP_THINKPAD_I2S_SPK] = { + .type = HDA_FIXUP_FUNC, + .v.func = alc287_fixup_bind_dacs, + }, }; static const struct snd_pci_quirk alc269_fixup_tbl[] = { @@ -10544,6 +10570,10 @@ static const struct snd_hda_pin_quirk alc269_pin_fixup_tbl[] = { {0x17, 0x90170111}, {0x19, 0x03a11030}, {0x21, 0x03211020}), + SND_HDA_PIN_QUIRK(0x10ec0287, 0x17aa, "Lenovo", ALC287_FIXUP_THINKPAD_I2S_SPK, + {0x17, 0x90170110}, + {0x19, 0x03a11030}, + {0x21, 0x03211020}), SND_HDA_PIN_QUIRK(0x10ec0286, 0x1025, "Acer", ALC286_FIXUP_ACER_AIO_MIC_NO_PRESENCE, {0x12, 0x90a60130}, {0x17, 0x90170110}, diff --git a/sound/pci/hda/tas2781_hda_i2c.c b/sound/pci/hda/tas2781_hda_i2c.c index 37114fd61a38..fb802802939e 100644 --- a/sound/pci/hda/tas2781_hda_i2c.c +++ b/sound/pci/hda/tas2781_hda_i2c.c @@ -173,16 +173,6 @@ static int tasdevice_get_profile_id(struct snd_kcontrol *kcontrol, return 0; } -static int tasdevice_hda_clamp(int val, int max) -{ - if (val > max) - val = max; - - if (val < 0) - val = 0; - return val; -} - static int tasdevice_set_profile_id(struct snd_kcontrol *kcontrol, struct snd_ctl_elem_value *ucontrol) { @@ -191,7 +181,7 @@ static int tasdevice_set_profile_id(struct snd_kcontrol *kcontrol, int max = tas_priv->rcabin.ncfgs - 1; int val, ret = 0; - val = tasdevice_hda_clamp(nr_profile, max); + val = clamp(nr_profile, 0, max); if (tas_priv->rcabin.profile_cfg_id != val) { tas_priv->rcabin.profile_cfg_id = val; @@ -248,7 +238,7 @@ static int tasdevice_program_put(struct snd_kcontrol *kcontrol, int max = tas_fw->nr_programs - 1; int val, ret = 0; - val = tasdevice_hda_clamp(nr_program, max); + val = clamp(nr_program, 0, max); if (tas_priv->cur_prog != val) { tas_priv->cur_prog = val; @@ -277,7 +267,7 @@ static int tasdevice_config_put(struct snd_kcontrol *kcontrol, int max = tas_fw->nr_configurations - 1; int val, ret = 0; - val = tasdevice_hda_clamp(nr_config, max); + val = clamp(nr_config, 0, max); if (tas_priv->cur_conf != val) { tas_priv->cur_conf = val; diff --git a/sound/soc/amd/yc/acp6x-mach.c b/sound/soc/amd/yc/acp6x-mach.c index b304b3562c82..3ec15b46fa35 100644 --- a/sound/soc/amd/yc/acp6x-mach.c +++ b/sound/soc/amd/yc/acp6x-mach.c @@ -213,6 +213,13 @@ static const struct dmi_system_id yc_acp_quirk_table[] = { DMI_MATCH(DMI_PRODUCT_NAME, "21J6"), } }, + { + .driver_data = &acp6x_card, + .matches = { + DMI_MATCH(DMI_BOARD_VENDOR, "LENOVO"), + DMI_MATCH(DMI_PRODUCT_NAME, "82TL"), + } + }, { .driver_data = &acp6x_card, .matches = { @@ -325,6 +332,13 @@ static const struct dmi_system_id yc_acp_quirk_table[] = { DMI_MATCH(DMI_BOARD_NAME, "8A22"), } }, + { + .driver_data = &acp6x_card, + .matches = { + DMI_MATCH(DMI_BOARD_VENDOR, "HP"), + DMI_MATCH(DMI_BOARD_NAME, "8A3E"), + } + }, { .driver_data = &acp6x_card, .matches = { diff --git a/sound/soc/atmel/mchp-pdmc.c b/sound/soc/atmel/mchp-pdmc.c index afe213a71212..dcc4e14b3dde 100644 --- a/sound/soc/atmel/mchp-pdmc.c +++ b/sound/soc/atmel/mchp-pdmc.c @@ -954,7 +954,7 @@ static int mchp_pdmc_dt_init(struct mchp_pdmc *dd) /* used to clean the channel index found on RHR's MSB */ static int mchp_pdmc_process(struct snd_pcm_substream *substream, int channel, unsigned long hwoff, - struct iov_iter *buf, unsigned long bytes) + unsigned long bytes) { struct snd_pcm_runtime *runtime = substream->runtime; u8 *dma_ptr = runtime->dma_area + hwoff + diff --git a/sound/soc/codecs/Kconfig b/sound/soc/codecs/Kconfig index 95b5bd883215..f1e1dbc509f6 100644 --- a/sound/soc/codecs/Kconfig +++ b/sound/soc/codecs/Kconfig @@ -1968,11 +1968,15 @@ config SND_SOC_UDA1380 tristate depends on I2C +config SND_SOC_WCD_CLASSH + tristate + config SND_SOC_WCD9335 tristate "WCD9335 Codec" depends on SLIMBUS select REGMAP_SLIMBUS select REGMAP_IRQ + select SND_SOC_WCD_CLASSH help The WCD9335 is a standalone Hi-Fi audio CODEC IC, supports Qualcomm Technologies, Inc. (QTI) multimedia solutions, @@ -1987,6 +1991,7 @@ config SND_SOC_WCD934X depends on SLIMBUS select REGMAP_IRQ select REGMAP_SLIMBUS + select SND_SOC_WCD_CLASSH select SND_SOC_WCD_MBHC depends on MFD_WCD934X || COMPILE_TEST help @@ -1997,6 +2002,7 @@ config SND_SOC_WCD938X depends on SND_SOC_WCD938X_SDW tristate depends on SOUNDWIRE || !SOUNDWIRE + select SND_SOC_WCD_CLASSH config SND_SOC_WCD938X_SDW tristate "WCD9380/WCD9385 Codec - SDW" diff --git a/sound/soc/codecs/Makefile b/sound/soc/codecs/Makefile index c8502a49b40a..a87e56938ce5 100644 --- a/sound/soc/codecs/Makefile +++ b/sound/soc/codecs/Makefile @@ -303,10 +303,11 @@ snd-soc-twl4030-objs := twl4030.o snd-soc-twl6040-objs := twl6040.o snd-soc-uda1334-objs := uda1334.o snd-soc-uda1380-objs := uda1380.o +snd-soc-wcd-classh-objs := wcd-clsh-v2.o snd-soc-wcd-mbhc-objs := wcd-mbhc-v2.o -snd-soc-wcd9335-objs := wcd-clsh-v2.o wcd9335.o -snd-soc-wcd934x-objs := wcd-clsh-v2.o wcd934x.o -snd-soc-wcd938x-objs := wcd938x.o wcd-clsh-v2.o +snd-soc-wcd9335-objs := wcd9335.o +snd-soc-wcd934x-objs := wcd934x.o +snd-soc-wcd938x-objs := wcd938x.o snd-soc-wcd938x-sdw-objs := wcd938x-sdw.o snd-soc-wl1273-objs := wl1273.o snd-soc-wm-adsp-objs := wm_adsp.o @@ -685,6 +686,7 @@ obj-$(CONFIG_SND_SOC_TWL4030) += snd-soc-twl4030.o obj-$(CONFIG_SND_SOC_TWL6040) += snd-soc-twl6040.o obj-$(CONFIG_SND_SOC_UDA1334) += snd-soc-uda1334.o obj-$(CONFIG_SND_SOC_UDA1380) += snd-soc-uda1380.o +obj-$(CONFIG_SND_SOC_WCD_CLASSH) += snd-soc-wcd-classh.o obj-$(CONFIG_SND_SOC_WCD_MBHC) += snd-soc-wcd-mbhc.o obj-$(CONFIG_SND_SOC_WCD9335) += snd-soc-wcd9335.o obj-$(CONFIG_SND_SOC_WCD934X) += snd-soc-wcd934x.o diff --git a/sound/soc/codecs/cs35l45.c b/sound/soc/codecs/cs35l45.c index d1edb9876c10..be4f4229576c 100644 --- a/sound/soc/codecs/cs35l45.c +++ b/sound/soc/codecs/cs35l45.c @@ -279,7 +279,7 @@ static const struct snd_kcontrol_new cs35l45_dsp_muxes[] = { }; static const struct snd_kcontrol_new cs35l45_dac_muxes[] = { - SOC_DAPM_ENUM("DACPCM1 Source", cs35l45_dacpcm_enums[0]), + SOC_DAPM_ENUM("DACPCM Source", cs35l45_dacpcm_enums[0]), }; static const struct snd_soc_dapm_widget cs35l45_dapm_widgets[] = { @@ -333,7 +333,7 @@ static const struct snd_soc_dapm_widget cs35l45_dapm_widgets[] = { SND_SOC_DAPM_MUX("DSP_RX7 Source", SND_SOC_NOPM, 0, 0, &cs35l45_dsp_muxes[6]), SND_SOC_DAPM_MUX("DSP_RX8 Source", SND_SOC_NOPM, 0, 0, &cs35l45_dsp_muxes[7]), - SND_SOC_DAPM_MUX("DACPCM1 Source", SND_SOC_NOPM, 0, 0, &cs35l45_dac_muxes[0]), + SND_SOC_DAPM_MUX("DACPCM Source", SND_SOC_NOPM, 0, 0, &cs35l45_dac_muxes[0]), SND_SOC_DAPM_OUT_DRV("AMP", SND_SOC_NOPM, 0, 0, NULL, 0), @@ -403,7 +403,7 @@ static const struct snd_soc_dapm_route cs35l45_dapm_routes[] = { { "ASP_RX1", NULL, "ASP_EN" }, { "ASP_RX2", NULL, "ASP_EN" }, - { "AMP", NULL, "DACPCM1 Source"}, + { "AMP", NULL, "DACPCM Source"}, { "AMP", NULL, "GLOBAL_EN"}, CS35L45_DSP_MUX_ROUTE("DSP_RX1"), @@ -427,7 +427,7 @@ static const struct snd_soc_dapm_route cs35l45_dapm_routes[] = { {"DSP1 Preload", NULL, "DSP1 Preloader"}, {"DSP1", NULL, "DSP1 Preloader"}, - CS35L45_DAC_MUX_ROUTE("DACPCM1"), + CS35L45_DAC_MUX_ROUTE("DACPCM"), { "SPK", NULL, "AMP"}, }; @@ -969,7 +969,7 @@ static irqreturn_t cs35l45_dsp_virt2_mbox_cb(int irq, void *data) ret = regmap_read(cs35l45->regmap, CS35L45_DSP_VIRT2_MBOX_3, &mbox_val); if (!ret && mbox_val) - ret = cs35l45_dsp_virt2_mbox3_irq_handle(cs35l45, mbox_val & CS35L45_MBOX3_CMD_MASK, + cs35l45_dsp_virt2_mbox3_irq_handle(cs35l45, mbox_val & CS35L45_MBOX3_CMD_MASK, (mbox_val & CS35L45_MBOX3_DATA_MASK) >> CS35L45_MBOX3_DATA_SHIFT); /* Handle DSP trace log IRQ */ @@ -1078,6 +1078,7 @@ static int cs35l45_initialize(struct cs35l45_private *cs35l45) switch (dev_id[0]) { case 0x35A450: + case 0x35A460: break; default: dev_err(cs35l45->dev, "Bad DEVID 0x%x\n", dev_id[0]); diff --git a/sound/soc/codecs/cs35l56-shared.c b/sound/soc/codecs/cs35l56-shared.c index ae373f335ea8..98b1e63360ae 100644 --- a/sound/soc/codecs/cs35l56-shared.c +++ b/sound/soc/codecs/cs35l56-shared.c @@ -243,26 +243,27 @@ int cs35l56_wait_for_firmware_boot(struct cs35l56_base *cs35l56_base) { unsigned int reg; unsigned int val; - int ret; + int read_ret, poll_ret; if (cs35l56_base->rev < CS35L56_REVID_B0) reg = CS35L56_DSP1_HALO_STATE_A1; else reg = CS35L56_DSP1_HALO_STATE; - ret = regmap_read_poll_timeout(cs35l56_base->regmap, reg, - val, - (val < 0xFFFF) && (val >= CS35L56_HALO_STATE_BOOT_DONE), - CS35L56_HALO_STATE_POLL_US, - CS35L56_HALO_STATE_TIMEOUT_US); + /* + * This can't be a regmap_read_poll_timeout() because cs35l56 will NAK + * I2C until it has booted which would terminate the poll + */ + poll_ret = read_poll_timeout(regmap_read, read_ret, + (val < 0xFFFF) && (val >= CS35L56_HALO_STATE_BOOT_DONE), + CS35L56_HALO_STATE_POLL_US, + CS35L56_HALO_STATE_TIMEOUT_US, + false, + cs35l56_base->regmap, reg, &val); - if ((ret < 0) && (ret != -ETIMEDOUT)) { - dev_err(cs35l56_base->dev, "Failed to read HALO_STATE: %d\n", ret); - return ret; - } - - if ((ret == -ETIMEDOUT) || (val != CS35L56_HALO_STATE_BOOT_DONE)) { - dev_err(cs35l56_base->dev, "Firmware boot fail: HALO_STATE=%#x\n", val); + if (poll_ret) { + dev_err(cs35l56_base->dev, "Firmware boot timed out(%d): HALO_STATE=%#x\n", + read_ret, val); return -EIO; } diff --git a/sound/soc/codecs/cs42l43.c b/sound/soc/codecs/cs42l43.c index 24e718e51174..1a95c370fc4c 100644 --- a/sound/soc/codecs/cs42l43.c +++ b/sound/soc/codecs/cs42l43.c @@ -2205,7 +2205,8 @@ static int cs42l43_codec_probe(struct platform_device *pdev) // Don't use devm as we need to get against the MFD device priv->mclk = clk_get_optional(cs42l43->dev, "mclk"); if (IS_ERR(priv->mclk)) { - dev_err_probe(priv->dev, PTR_ERR(priv->mclk), "Failed to get mclk\n"); + ret = PTR_ERR(priv->mclk); + dev_err_probe(priv->dev, ret, "Failed to get mclk\n"); goto err_pm; } diff --git a/sound/soc/codecs/rt5645.c b/sound/soc/codecs/rt5645.c index 038d93e20883..1a137ca3f496 100644 --- a/sound/soc/codecs/rt5645.c +++ b/sound/soc/codecs/rt5645.c @@ -3269,13 +3269,17 @@ static int rt5645_component_set_jack(struct snd_soc_component *component, { struct snd_soc_jack *mic_jack = NULL; struct snd_soc_jack *btn_jack = NULL; - int *type = (int *)data; + int type; - if (*type & SND_JACK_MICROPHONE) - mic_jack = hs_jack; - if (*type & (SND_JACK_BTN_0 | SND_JACK_BTN_1 | - SND_JACK_BTN_2 | SND_JACK_BTN_3)) - btn_jack = hs_jack; + if (hs_jack) { + type = *(int *)data; + + if (type & SND_JACK_MICROPHONE) + mic_jack = hs_jack; + if (type & (SND_JACK_BTN_0 | SND_JACK_BTN_1 | + SND_JACK_BTN_2 | SND_JACK_BTN_3)) + btn_jack = hs_jack; + } return rt5645_set_jack_detect(component, hs_jack, mic_jack, btn_jack); } diff --git a/sound/soc/codecs/wcd-clsh-v2.c b/sound/soc/codecs/wcd-clsh-v2.c index a75db27e5205..d96e23ec43d4 100644 --- a/sound/soc/codecs/wcd-clsh-v2.c +++ b/sound/soc/codecs/wcd-clsh-v2.c @@ -355,6 +355,7 @@ void wcd_clsh_set_hph_mode(struct wcd_clsh_ctrl *ctrl, int mode) wcd_clsh_v2_set_hph_mode(comp, mode); } +EXPORT_SYMBOL_GPL(wcd_clsh_set_hph_mode); static void wcd_clsh_set_flyback_current(struct snd_soc_component *comp, int mode) @@ -869,11 +870,13 @@ int wcd_clsh_ctrl_set_state(struct wcd_clsh_ctrl *ctrl, return 0; } +EXPORT_SYMBOL_GPL(wcd_clsh_ctrl_set_state); int wcd_clsh_ctrl_get_state(struct wcd_clsh_ctrl *ctrl) { return ctrl->state; } +EXPORT_SYMBOL_GPL(wcd_clsh_ctrl_get_state); struct wcd_clsh_ctrl *wcd_clsh_ctrl_alloc(struct snd_soc_component *comp, int version) @@ -890,8 +893,13 @@ struct wcd_clsh_ctrl *wcd_clsh_ctrl_alloc(struct snd_soc_component *comp, return ctrl; } +EXPORT_SYMBOL_GPL(wcd_clsh_ctrl_alloc); void wcd_clsh_ctrl_free(struct wcd_clsh_ctrl *ctrl) { kfree(ctrl); } +EXPORT_SYMBOL_GPL(wcd_clsh_ctrl_free); + +MODULE_DESCRIPTION("WCD93XX Class-H driver"); +MODULE_LICENSE("GPL"); diff --git a/sound/soc/intel/avs/pcm.c b/sound/soc/intel/avs/pcm.c index 1fbb2c2fadb5..8565a530706d 100644 --- a/sound/soc/intel/avs/pcm.c +++ b/sound/soc/intel/avs/pcm.c @@ -796,6 +796,28 @@ static int avs_component_probe(struct snd_soc_component *component) ret = avs_load_topology(component, filename); kfree(filename); + if (ret == -ENOENT && !strncmp(mach->tplg_filename, "hda-", 4)) { + unsigned int vendor_id; + + if (sscanf(mach->tplg_filename, "hda-%08x-tplg.bin", &vendor_id) != 1) + return ret; + + if (((vendor_id >> 16) & 0xFFFF) == 0x8086) + mach->tplg_filename = devm_kasprintf(adev->dev, GFP_KERNEL, + "hda-8086-generic-tplg.bin"); + else + mach->tplg_filename = devm_kasprintf(adev->dev, GFP_KERNEL, + "hda-generic-tplg.bin"); + + filename = kasprintf(GFP_KERNEL, "%s/%s", component->driver->topology_name_prefix, + mach->tplg_filename); + if (!filename) + return -ENOMEM; + + dev_info(card->dev, "trying to load fallback topology %s\n", mach->tplg_filename); + ret = avs_load_topology(component, filename); + kfree(filename); + } if (ret < 0) return ret; diff --git a/sound/soc/soc-component.c b/sound/soc/soc-component.c index f18406dfa1e4..ba7c0ae82e00 100644 --- a/sound/soc/soc-component.c +++ b/sound/soc/soc-component.c @@ -1054,7 +1054,7 @@ int snd_soc_pcm_component_sync_stop(struct snd_pcm_substream *substream) int snd_soc_pcm_component_copy(struct snd_pcm_substream *substream, int channel, unsigned long pos, - struct iov_iter *buf, unsigned long bytes) + struct iov_iter *iter, unsigned long bytes) { struct snd_soc_pcm_runtime *rtd = asoc_substream_to_rtd(substream); struct snd_soc_component *component; @@ -1065,7 +1065,7 @@ int snd_soc_pcm_component_copy(struct snd_pcm_substream *substream, if (component->driver->copy) return soc_component_ret(component, component->driver->copy(component, substream, - channel, pos, buf, bytes)); + channel, pos, iter, bytes)); return -EINVAL; } diff --git a/sound/soc/soc-generic-dmaengine-pcm.c b/sound/soc/soc-generic-dmaengine-pcm.c index ff2166525dbc..d0653d775c87 100644 --- a/sound/soc/soc-generic-dmaengine-pcm.c +++ b/sound/soc/soc-generic-dmaengine-pcm.c @@ -290,29 +290,29 @@ static snd_pcm_uframes_t dmaengine_pcm_pointer( static int dmaengine_copy(struct snd_soc_component *component, struct snd_pcm_substream *substream, int channel, unsigned long hwoff, - struct iov_iter *buf, unsigned long bytes) + struct iov_iter *iter, unsigned long bytes) { struct snd_pcm_runtime *runtime = substream->runtime; struct dmaengine_pcm *pcm = soc_component_to_pcm(component); int (*process)(struct snd_pcm_substream *substream, int channel, unsigned long hwoff, - struct iov_iter *buf, unsigned long bytes) = pcm->config->process; + unsigned long bytes) = pcm->config->process; bool is_playback = substream->stream == SNDRV_PCM_STREAM_PLAYBACK; void *dma_ptr = runtime->dma_area + hwoff + channel * (runtime->dma_bytes / runtime->channels); if (is_playback) - if (copy_from_iter(dma_ptr, bytes, buf) != bytes) + if (copy_from_iter(dma_ptr, bytes, iter) != bytes) return -EFAULT; if (process) { - int ret = process(substream, channel, hwoff, buf, bytes); + int ret = process(substream, channel, hwoff, bytes); if (ret < 0) return ret; } if (!is_playback) - if (copy_to_iter(dma_ptr, bytes, buf) != bytes) + if (copy_to_iter(dma_ptr, bytes, iter) != bytes) return -EFAULT; return 0; diff --git a/sound/soc/stm/stm32_sai_sub.c b/sound/soc/stm/stm32_sai_sub.c index f9b5d5969155..0acc848c1f00 100644 --- a/sound/soc/stm/stm32_sai_sub.c +++ b/sound/soc/stm/stm32_sai_sub.c @@ -1246,7 +1246,7 @@ static const struct snd_soc_dai_ops stm32_sai_pcm_dai_ops2 = { static int stm32_sai_pcm_process_spdif(struct snd_pcm_substream *substream, int channel, unsigned long hwoff, - struct iov_iter *buf, unsigned long bytes) + unsigned long bytes) { struct snd_pcm_runtime *runtime = substream->runtime; struct snd_soc_pcm_runtime *rtd = asoc_substream_to_rtd(substream); diff --git a/sound/usb/midi2.c b/sound/usb/midi2.c index a27e244650c8..1ec177fe284e 100644 --- a/sound/usb/midi2.c +++ b/sound/usb/midi2.c @@ -265,7 +265,7 @@ static void free_midi_urbs(struct snd_usb_midi2_endpoint *ep) if (!ep) return; - for (i = 0; i < ep->num_urbs; ++i) { + for (i = 0; i < NUM_URBS; ++i) { ctx = &ep->urbs[i]; if (!ctx->urb) break; @@ -279,6 +279,7 @@ static void free_midi_urbs(struct snd_usb_midi2_endpoint *ep) } /* allocate URBs for an EP */ +/* the callers should handle allocation errors via free_midi_urbs() */ static int alloc_midi_urbs(struct snd_usb_midi2_endpoint *ep) { struct snd_usb_midi2_urb *ctx; @@ -351,8 +352,10 @@ static int snd_usb_midi_v2_open(struct snd_ump_endpoint *ump, int dir) return -EIO; if (ep->direction == STR_OUT) { err = alloc_midi_urbs(ep); - if (err) + if (err) { + free_midi_urbs(ep); return err; + } } return 0; } diff --git a/tools/arch/x86/include/asm/cpufeatures.h b/tools/arch/x86/include/asm/cpufeatures.h index 1f6d904c6481..798e60b5454b 100644 --- a/tools/arch/x86/include/asm/cpufeatures.h +++ b/tools/arch/x86/include/asm/cpufeatures.h @@ -434,6 +434,7 @@ #define X86_FEATURE_SEV_ES (19*32+ 3) /* AMD Secure Encrypted Virtualization - Encrypted State */ #define X86_FEATURE_V_TSC_AUX (19*32+ 9) /* "" Virtual TSC_AUX */ #define X86_FEATURE_SME_COHERENT (19*32+10) /* "" AMD hardware-enforced cache coherency */ +#define X86_FEATURE_DEBUG_SWAP (19*32+14) /* AMD SEV-ES full debug state swap support */ /* AMD-defined Extended Feature 2 EAX, CPUID level 0x80000021 (EAX), word 20 */ #define X86_FEATURE_NO_NESTED_DATA_BP (20*32+ 0) /* "" No Nested Data Breakpoints */ diff --git a/tools/bpf/bpftool/link.c b/tools/bpf/bpftool/link.c index 0b214f6ab5c8..2e5c231e08ac 100644 --- a/tools/bpf/bpftool/link.c +++ b/tools/bpf/bpftool/link.c @@ -83,7 +83,7 @@ const char *evsel__hw_cache_result[PERF_COUNT_HW_CACHE_RESULT_MAX] = { #define perf_event_name(array, id) ({ \ const char *event_str = NULL; \ \ - if ((id) >= 0 && (id) < ARRAY_SIZE(array)) \ + if ((id) < ARRAY_SIZE(array)) \ event_str = array[id]; \ event_str; \ }) diff --git a/tools/build/Makefile.build b/tools/build/Makefile.build index 89430338a3d9..fac42486a8cf 100644 --- a/tools/build/Makefile.build +++ b/tools/build/Makefile.build @@ -117,6 +117,16 @@ $(OUTPUT)%.s: %.c FORCE $(call rule_mkdir) $(call if_changed_dep,cc_s_c) +# bison and flex files are generated in the OUTPUT directory +# so it needs a separate rule to depend on them properly +$(OUTPUT)%-bison.o: $(OUTPUT)%-bison.c FORCE + $(call rule_mkdir) + $(call if_changed_dep,$(host)cc_o_c) + +$(OUTPUT)%-flex.o: $(OUTPUT)%-flex.c FORCE + $(call rule_mkdir) + $(call if_changed_dep,$(host)cc_o_c) + # Gather build data: # obj-y - list of build objects # subdir-y - list of directories to nest diff --git a/tools/build/feature/Makefile b/tools/build/feature/Makefile index f0c5de018a95..dad79ede4e0a 100644 --- a/tools/build/feature/Makefile +++ b/tools/build/feature/Makefile @@ -340,7 +340,7 @@ $(OUTPUT)test-jvmti-cmlr.bin: $(BUILD) $(OUTPUT)test-llvm.bin: - $(BUILDXX) -std=gnu++14 \ + $(BUILDXX) -std=gnu++17 \ -I$(shell $(LLVM_CONFIG) --includedir) \ -L$(shell $(LLVM_CONFIG) --libdir) \ $(shell $(LLVM_CONFIG) --libs Core BPF) \ @@ -348,17 +348,15 @@ $(OUTPUT)test-llvm.bin: > $(@:.bin=.make.output) 2>&1 $(OUTPUT)test-llvm-version.bin: - $(BUILDXX) -std=gnu++14 \ + $(BUILDXX) -std=gnu++17 \ -I$(shell $(LLVM_CONFIG) --includedir) \ > $(@:.bin=.make.output) 2>&1 $(OUTPUT)test-clang.bin: - $(BUILDXX) -std=gnu++14 \ + $(BUILDXX) -std=gnu++17 \ -I$(shell $(LLVM_CONFIG) --includedir) \ -L$(shell $(LLVM_CONFIG) --libdir) \ - -Wl,--start-group -lclangBasic -lclangDriver \ - -lclangFrontend -lclangEdit -lclangLex \ - -lclangAST -Wl,--end-group \ + -Wl,--start-group -lclang-cpp -Wl,--end-group \ $(shell $(LLVM_CONFIG) --libs Core option) \ $(shell $(LLVM_CONFIG) --system-libs) \ > $(@:.bin=.make.output) 2>&1 diff --git a/tools/build/feature/test-clang.cpp b/tools/build/feature/test-clang.cpp deleted file mode 100644 index 7d87075cd1c5..000000000000 --- a/tools/build/feature/test-clang.cpp +++ /dev/null @@ -1,28 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include "clang/Basic/Version.h" -#if CLANG_VERSION_MAJOR < 8 -#include "clang/Basic/VirtualFileSystem.h" -#endif -#include "clang/Driver/Driver.h" -#include "clang/Frontend/TextDiagnosticPrinter.h" -#include "llvm/ADT/IntrusiveRefCntPtr.h" -#include "llvm/Support/ManagedStatic.h" -#if CLANG_VERSION_MAJOR >= 8 -#include "llvm/Support/VirtualFileSystem.h" -#endif -#include "llvm/Support/raw_ostream.h" - -using namespace clang; -using namespace clang::driver; - -int main() -{ - IntrusiveRefCntPtr DiagID(new DiagnosticIDs()); - IntrusiveRefCntPtr DiagOpts = new DiagnosticOptions(); - - DiagnosticsEngine Diags(DiagID, &*DiagOpts); - Driver TheDriver("test", "bpf-pc-linux", Diags); - - llvm::llvm_shutdown(); - return 0; -} diff --git a/tools/build/feature/test-cxx.cpp b/tools/build/feature/test-cxx.cpp deleted file mode 100644 index 396aaedd2418..000000000000 --- a/tools/build/feature/test-cxx.cpp +++ /dev/null @@ -1,16 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include -#include - -static void print_str(std::string s) -{ - std::cout << s << std::endl; -} - -int main() -{ - std::string s("Hello World!"); - print_str(std::move(s)); - std::cout << "|" << s << "|" << std::endl; - return 0; -} diff --git a/tools/build/feature/test-llvm-version.cpp b/tools/build/feature/test-llvm-version.cpp deleted file mode 100644 index 8a091625446a..000000000000 --- a/tools/build/feature/test-llvm-version.cpp +++ /dev/null @@ -1,12 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include -#include "llvm/Config/llvm-config.h" - -#define NUM_VERSION (((LLVM_VERSION_MAJOR) << 16) + (LLVM_VERSION_MINOR << 8) + LLVM_VERSION_PATCH) -#define pass int main() {printf("%x\n", NUM_VERSION); return 0;} - -#if NUM_VERSION >= 0x030900 -pass -#else -# error This LLVM is not tested yet. -#endif diff --git a/tools/build/feature/test-llvm.cpp b/tools/build/feature/test-llvm.cpp deleted file mode 100644 index 88a3d1bdd9f6..000000000000 --- a/tools/build/feature/test-llvm.cpp +++ /dev/null @@ -1,14 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include "llvm/Support/ManagedStatic.h" -#include "llvm/Support/raw_ostream.h" -#define NUM_VERSION (((LLVM_VERSION_MAJOR) << 16) + (LLVM_VERSION_MINOR << 8) + LLVM_VERSION_PATCH) - -#if NUM_VERSION < 0x030900 -# error "LLVM version too low" -#endif -int main() -{ - llvm::errs() << "Hello World!\n"; - llvm::llvm_shutdown(); - return 0; -} diff --git a/tools/lib/perf/include/perf/event.h b/tools/lib/perf/include/perf/event.h index ba2dcf64f4e6..ae64090184d3 100644 --- a/tools/lib/perf/include/perf/event.h +++ b/tools/lib/perf/include/perf/event.h @@ -148,9 +148,19 @@ struct perf_record_switch { struct perf_record_header_attr { struct perf_event_header header; struct perf_event_attr attr; - __u64 id[]; + /* + * Array of u64 id follows here but we cannot use a flexible array + * because size of attr in the data can be different then current + * version. Please use perf_record_header_attr_id() below. + * + * __u64 id[]; // do not use this + */ }; +/* Returns the pointer to id array based on the actual attr size. */ +#define perf_record_header_attr_id(evt) \ + ((void *)&(evt)->attr.attr + (evt)->attr.attr.size) + enum { PERF_CPU_MAP__CPUS = 0, PERF_CPU_MAP__MASK = 1, diff --git a/tools/perf/Documentation/perf-bench.txt b/tools/perf/Documentation/perf-bench.txt index f04f0eaded98..ca5789625cd2 100644 --- a/tools/perf/Documentation/perf-bench.txt +++ b/tools/perf/Documentation/perf-bench.txt @@ -67,6 +67,9 @@ SUBSYSTEM 'internals':: Benchmark internal perf functionality. +'uprobe':: + Benchmark overhead of uprobe + BPF. + 'all':: All benchmark subsystems. diff --git a/tools/perf/Documentation/perf-config.txt b/tools/perf/Documentation/perf-config.txt index 1478068ad5dd..0b4e79dbd3f6 100644 --- a/tools/perf/Documentation/perf-config.txt +++ b/tools/perf/Documentation/perf-config.txt @@ -125,9 +125,6 @@ Given a $HOME/.perfconfig like this: group = true skip-empty = true - [llvm] - dump-obj = true - clang-opt = -g You can hide source code of annotate feature setting the config to false with @@ -657,36 +654,6 @@ ftrace.*:: -F option is not specified. Possible values are 'function' and 'function_graph'. -llvm.*:: - llvm.clang-path:: - Path to clang. If omit, search it from $PATH. - - llvm.clang-bpf-cmd-template:: - Cmdline template. Below lines show its default value. Environment - variable is used to pass options. - "$CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS "\ - "-DLINUX_VERSION_CODE=$LINUX_VERSION_CODE " \ - "$CLANG_OPTIONS $PERF_BPF_INC_OPTIONS $KERNEL_INC_OPTIONS " \ - "-Wno-unused-value -Wno-pointer-sign " \ - "-working-directory $WORKING_DIR " \ - "-c \"$CLANG_SOURCE\" --target=bpf $CLANG_EMIT_LLVM -O2 -o - $LLVM_OPTIONS_PIPE" - - llvm.clang-opt:: - Options passed to clang. - - llvm.kbuild-dir:: - kbuild directory. If not set, use /lib/modules/`uname -r`/build. - If set to "" deliberately, skip kernel header auto-detector. - - llvm.kbuild-opts:: - Options passed to 'make' when detecting kernel header options. - - llvm.dump-obj:: - Enable perf dump BPF object files compiled by LLVM. - - llvm.opts:: - Options passed to llc. - samples.*:: samples.context:: diff --git a/tools/perf/Documentation/perf-dlfilter.txt b/tools/perf/Documentation/perf-dlfilter.txt index fb22e3b31dc5..8887cc20a809 100644 --- a/tools/perf/Documentation/perf-dlfilter.txt +++ b/tools/perf/Documentation/perf-dlfilter.txt @@ -64,6 +64,12 @@ internal filtering. If implemented, 'filter_description' should return a one-line description of the filter, and optionally a longer description. +Do not assume the 'sample' argument is valid (dereferenceable) +after 'filter_event' and 'filter_event_early' return. + +Do not assume data referenced by pointers in struct perf_dlfilter_sample +is valid (dereferenceable) after 'filter_event' and 'filter_event_early' return. + The perf_dlfilter_sample structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -150,7 +156,8 @@ struct perf_dlfilter_fns { const char *(*srcline)(void *ctx, __u32 *line_number); struct perf_event_attr *(*attr)(void *ctx); __s32 (*object_code)(void *ctx, __u64 ip, void *buf, __u32 len); - void *(*reserved[120])(void *); + void (*al_cleanup)(void *ctx, struct perf_dlfilter_al *al); + void *(*reserved[119])(void *); }; ---- @@ -161,7 +168,8 @@ struct perf_dlfilter_fns { 'args' returns arguments from --dlarg options. 'resolve_address' provides information about 'address'. al->size must be set -before calling. Returns 0 on success, -1 otherwise. +before calling. Returns 0 on success, -1 otherwise. Call al_cleanup() (if present, +see below) when 'al' data is no longer needed. 'insn' returns instruction bytes and length. @@ -171,6 +179,12 @@ before calling. Returns 0 on success, -1 otherwise. 'object_code' reads object code and returns the number of bytes read. +'al_cleanup' must be called (if present, so check perf_dlfilter_fns.al_cleanup != NULL) +after resolve_address() to free any associated resources. + +Do not assume pointers obtained via perf_dlfilter_fns are valid (dereferenceable) +after 'filter_event' and 'filter_event_early' return. + The perf_dlfilter_al structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -197,9 +211,13 @@ struct perf_dlfilter_al { /* Below members are only populated by resolve_ip() */ __u8 filtered; /* true if this sample event will be filtered out */ const char *comm; + void *priv; /* Private data. Do not change */ }; ---- +Do not assume data referenced by pointers in struct perf_dlfilter_al +is valid (dereferenceable) after 'filter_event' and 'filter_event_early' return. + perf_dlfilter_sample flags ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/tools/perf/Documentation/perf-ftrace.txt b/tools/perf/Documentation/perf-ftrace.txt index df4595563801..d780b93fcf87 100644 --- a/tools/perf/Documentation/perf-ftrace.txt +++ b/tools/perf/Documentation/perf-ftrace.txt @@ -96,8 +96,9 @@ OPTIONS for 'perf ftrace trace' --func-opts:: List of options allowed to set: - call-graph - Display kernel stack trace for function tracer. - irq-info - Display irq context info for function tracer. + + - call-graph - Display kernel stack trace for function tracer. + - irq-info - Display irq context info for function tracer. -G:: --graph-funcs=:: @@ -118,11 +119,12 @@ OPTIONS for 'perf ftrace trace' --graph-opts:: List of options allowed to set: - nosleep-time - Measure on-CPU time only for function_graph tracer. - noirqs - Ignore functions that happen inside interrupt. - verbose - Show process names, PIDs, timestamps, etc. - thresh= - Setup trace duration threshold in microseconds. - depth= - Set max depth for function graph tracer to follow. + + - nosleep-time - Measure on-CPU time only for function_graph tracer. + - noirqs - Ignore functions that happen inside interrupt. + - verbose - Show process names, PIDs, timestamps, etc. + - thresh= - Setup trace duration threshold in microseconds. + - depth= - Set max depth for function graph tracer to follow. OPTIONS for 'perf ftrace latency' diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt index 680396c56bd1..d5217be012d7 100644 --- a/tools/perf/Documentation/perf-record.txt +++ b/tools/perf/Documentation/perf-record.txt @@ -99,20 +99,6 @@ OPTIONS If you want to profile write accesses in [0x1000~1008), just set 'mem:0x1000/8:w'. - - a BPF source file (ending in .c) or a precompiled object file (ending - in .o) selects one or more BPF events. - The BPF program can attach to various perf events based on the ELF section - names. - - When processing a '.c' file, perf searches an installed LLVM to compile it - into an object file first. Optional clang options can be passed via the - '--clang-opt' command line option, e.g.: - - perf record --clang-opt "-DLINUX_VERSION_CODE=0x50000" \ - -e tests/bpf-script-example.c - - Note: '--clang-opt' must be placed before '--event/-e'. - - a group of events surrounded by a pair of brace ("{event1,event2,...}"). Each event is separated by commas and the group should be quoted to prevent the shell interpretation. You also need to use --group on @@ -523,9 +509,10 @@ CLOCK_BOOTTIME, CLOCK_REALTIME and CLOCK_TAI. Select AUX area tracing Snapshot Mode. This option is valid only with an AUX area tracing event. Optionally, certain snapshot capturing parameters can be specified in a string that follows this option: - 'e': take one last snapshot on exit; guarantees that there is at least one + + - 'e': take one last snapshot on exit; guarantees that there is at least one snapshot in the output file; - : if the PMU supports this, specify the desired snapshot size. + - : if the PMU supports this, specify the desired snapshot size. In Snapshot Mode trace data is captured only when signal SIGUSR2 is received and on exit if the above 'e' option is given. @@ -547,14 +534,6 @@ PERF_RECORD_SWITCH_CPU_WIDE. In some cases (e.g. Intel PT, CoreSight or Arm SPE) switch events will be enabled automatically, which can be suppressed by by the option --no-switch-events. ---clang-path=PATH:: -Path to clang binary to use for compiling BPF scriptlets. -(enabled when BPF support is on) - ---clang-opt=OPTIONS:: -Options passed to clang when compiling BPF scriptlets. -(enabled when BPF support is on) - --vmlinux=PATH:: Specify vmlinux path which has debuginfo. (enabled when BPF prologue is on) @@ -572,8 +551,9 @@ providing implementation for Posix AIO API. --affinity=mode:: Set affinity mask of trace reading thread according to the policy defined by 'mode' value: - node - thread affinity mask is set to NUMA node cpu mask of the processed mmap buffer - cpu - thread affinity mask is set to cpu of the processed mmap buffer + + - node - thread affinity mask is set to NUMA node cpu mask of the processed mmap buffer + - cpu - thread affinity mask is set to cpu of the processed mmap buffer --mmap-flush=number:: @@ -625,16 +605,17 @@ Record timestamp boundary (time of first/last samples). --switch-output[=mode]:: Generate multiple perf.data files, timestamp prefixed, switching to a new one based on 'mode' value: - "signal" - when receiving a SIGUSR2 (default value) or - - when reaching the size threshold, size is expected to - be a number with appended unit character - B/K/M/G -