tmakin/ack-tegra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge 84e57d2922 ("Merge tag 'exfat-for-6.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat") into android-mainline

Browse Source 
Steps on the way to 6.2-rc1

Change-Id: Iff35e073bb8f517d7476ce0967a556872f4cca2d
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
This commit is contained in:
Greg Kroah-Hartman
2023-01-21 11:17:39 +00:00
parent dee3378e18 851f657a86
commit 90a29d14c3
1281 changed files with 61137 additions and 22428 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.clang-format
+3
View File
@@ -441,8 +441,11 @@ ForEachMacros:
- 'inet_lhash2_for_each_icsk'
- 'inet_lhash2_for_each_icsk_continue'
- 'inet_lhash2_for_each_icsk_rcu'
- 'interval_tree_for_each_double_span'
- 'interval_tree_for_each_span'
- 'intlist__for_each_entry'
- 'intlist__for_each_entry_safe'
- 'iopt_for_each_contig_area'
- 'kcore_copy__for_each_phdr'
- 'key_for_each'
- 'key_for_each_safe'
Documentation/ABI/testing/sysfs-fs-f2fs
+41 -6
View File
@@ -99,6 +99,12 @@ Description: Controls the issue rate of discard commands that consist of small
checkpoint is triggered, and issued during the checkpoint.
By default, it is disabled with 0.
What: /sys/fs/f2fs/<disk>/max_ordered_discard
Date: October 2022
Contact: "Yangtao Li" <frank.li@vivo.com>
Description: Controls the maximum ordered discard, the unit size is one block(4KB).
Set it to 16 by default.
What: /sys/fs/f2fs/<disk>/max_discard_request
Date: December 2021
Contact: "Konstantin Vyshetsky" <vkon@google.com>
@@ -132,7 +138,8 @@ Contact: "Chao Yu" <yuchao0@huawei.com>
Description: Controls discard granularity of inner discard thread. Inner thread
will not issue discards with size that is smaller than granularity.
The unit size is one block(4KB), now only support configuring
in range of [1, 512]. Default value is 4(=16KB).
in range of [1, 512]. Default value is 16.
For small devices, default value is 1.
What: /sys/fs/f2fs/<disk>/umount_discard_timeout
Date: January 2019
@@ -235,7 +242,7 @@ Description: Shows total written kbytes issued to disk.
What: /sys/fs/f2fs/<disk>/features
Date: July 2017
Contact: "Jaegeuk Kim" <jaegeuk@kernel.org>
Description: <deprecated: should use /sys/fs/f2fs/<disk>/feature_list/
Description: <deprecated: should use /sys/fs/f2fs/<disk>/feature_list/>
Shows all enabled features in current device.
Supported features:
encryption, blkzoned, extra_attr, projquota, inode_checksum,
@@ -592,10 +599,10 @@ Description: With "mode=fragment:block" mount options, we can scatter block allo
in the length of 1..<max_fragment_hole> by turns. This value can be set
between 1..512 and the default value is 4.
What: /sys/fs/f2fs/<disk>/gc_urgent_high_remaining
Date: December 2021
Contact: "Daeho Jeong" <daehojeong@google.com>
Description: You can set the trial count limit for GC urgent high mode with this value.
What: /sys/fs/f2fs/<disk>/gc_remaining_trials
Date: October 2022
Contact: "Yangtao Li" <frank.li@vivo.com>
Description: You can set the trial count limit for GC urgent and idle mode with this value.
If GC thread gets to the limit, the mode will turn back to GC normal mode.
By default, the value is zero, which means there is no limit like before.
@@ -634,3 +641,31 @@ Date: July 2022
Contact: "Daeho Jeong" <daehojeong@google.com>
Description: Show the accumulated total revoked atomic write block count after boot.
If you write "0" here, you can initialize to "0".
What: /sys/fs/f2fs/<disk>/gc_mode
Date: October 2022
Contact: "Yangtao Li" <frank.li@vivo.com>
Description: Show the current gc_mode as a string.
This is a read-only entry.
What: /sys/fs/f2fs/<disk>/discard_urgent_util
Date: November 2022
Contact: "Yangtao Li" <frank.li@vivo.com>
Description: When space utilization exceeds this, do background DISCARD aggressively.
Does DISCARD forcibly in a period of given min_discard_issue_time when the number
of discards is not 0 and set discard granularity to 1.
Default: 80
What: /sys/fs/f2fs/<disk>/hot_data_age_threshold
Date: November 2022
Contact: "Ping Xiong" <xiongping1@xiaomi.com>
Description: When DATA SEPARATION is on, it controls the age threshold to indicate
the data blocks as hot. By default it was initialized as 262144 blocks
(equals to 1GB).
What: /sys/fs/f2fs/<disk>/warm_data_age_threshold
Date: November 2022
Contact: "Ping Xiong" <xiongping1@xiaomi.com>
Description: When DATA SEPARATION is on, it controls the age threshold to indicate
the data blocks as warm. By default it was initialized as 2621440 blocks
(equals to 10GB).
Documentation/ABI/testing/sysfs-kernel-oops_count
+6
View File
@@ -0,0 +1,6 @@
What: /sys/kernel/oops_count
Date: November 2022
KernelVersion: 6.2.0
Contact: Linux Kernel Hardening List <linux-hardening@vger.kernel.org>
Description:
Shows how many times the system has Oopsed since last boot.
Documentation/ABI/testing/sysfs-kernel-warn_count
+6
View File
@@ -0,0 +1,6 @@
What: /sys/kernel/oops_count
Date: November 2022
KernelVersion: 6.2.0
Contact: Linux Kernel Hardening List <linux-hardening@vger.kernel.org>
Description:
Shows how many times the system has Warned since last boot.
Documentation/admin-guide/cifs/usage.rst
+1 -1
View File
@@ -858,7 +858,7 @@ CIFS kernel module parameters
These module parameters can be specified or modified either during the time of
module loading or during the runtime by using the interface::
/proc/module/cifs/parameters/<param>
/sys/module/cifs/parameters/<param>
i.e.::
Documentation/admin-guide/kdump/vmcoreinfo.rst
+29
View File
@@ -595,3 +595,32 @@ X2TLB
-----
Indicates whether the crashed kernel enabled SH extended mode.
RISCV64
=======
VA_BITS
-------
The maximum number of bits for virtual addresses. Used to compute the
virtual memory ranges.
PAGE_OFFSET
-----------
Indicates the virtual kernel start address of the direct-mapped RAM region.
phys_ram_base
-------------
Indicates the start physical RAM address.
MODULES_VADDR|MODULES_END|VMALLOC_START|VMALLOC_END|VMEMMAP_START|VMEMMAP_END|KERNEL_LINK_ADDR
----------------------------------------------------------------------------------------------
Used to get the correct ranges:
* MODULES_VADDR ~ MODULES_END : Kernel module space.
* VMALLOC_START ~ VMALLOC_END : vmalloc() / ioremap() space.
* VMEMMAP_START ~ VMEMMAP_END : vmemmap space, used for struct page array.
* KERNEL_LINK_ADDR : start address of Kernel link and BPF
Documentation/admin-guide/sysctl/kernel.rst
+19
View File
@@ -670,6 +670,15 @@ This is the default behavior.
an oops event is detected.
oops_limit
==========
Number of kernel oopses after which the kernel should panic when
``panic_on_oops`` is not set. Setting this to 0 disables checking
the count. Setting this to 1 has the same effect as setting
``panic_on_oops=1``. The default value is 10000.
osrelease, ostype & version
===========================
@@ -1526,6 +1535,16 @@ entry will default to 2 instead of 0.
2 Unprivileged calls to ``bpf()`` are disabled
= =============================================================
warn_limit
==========
Number of kernel warnings after which the kernel should panic when
``panic_on_warn`` is not set. Setting this to 0 disables checking
the warning count. Setting this to 1 has the same effect as setting
``panic_on_warn=1``. The default value is 0.
watchdog
========
Documentation/core-api/kernel-api.rst
+3
View File
@@ -36,6 +36,9 @@ String Conversions
String Manipulation
-------------------
.. kernel-doc:: include/linux/fortify-string.h
:internal:
.. kernel-doc:: lib/string.c
:export:
Documentation/crypto/devel-algos.rst
+1 -1
View File
@@ -172,7 +172,7 @@ Here are schematics of how these functions are called when operated from
other part of the kernel. Note that the .setkey() call might happen
before or after any of these schematics happen, but must not happen
during any of these are in-flight. Please note that calling .init()
followed immediately by .finish() is also a perfectly valid
followed immediately by .final() is also a perfectly valid
transformation.
::
Documentation/crypto/userspace-if.rst
+12 -3
View File
@@ -131,9 +131,9 @@ from the kernel crypto API. If the buffer is too small for the message
digest, the flag MSG_TRUNC is set by the kernel.
In order to set a message digest key, the calling application must use
the setsockopt() option of ALG_SET_KEY. If the key is not set the HMAC
operation is performed without the initial HMAC state change caused by
the key.
the setsockopt() option of ALG_SET_KEY or ALG_SET_KEY_BY_KEY_SERIAL. If the
key is not set the HMAC operation is performed without the initial HMAC state
change caused by the key.
Symmetric Cipher API
--------------------
@@ -382,6 +382,15 @@ mentioned optname:
- the RNG cipher type to provide the seed
- ALG_SET_KEY_BY_KEY_SERIAL -- Setting the key via keyring key_serial_t.
This operation behaves the same as ALG_SET_KEY. The decrypted
data is copied from a keyring key, and uses that data as the
key for symmetric encryption.
The passed in key_serial_t must have the KEY_(POS|USR|GRP|OTH)_SEARCH
permission set, otherwise -EPERM is returned. Supports key types: user,
logon, encrypted, and trusted.
- ALG_SET_AEAD_AUTHSIZE -- Setting the authentication tag size for
AEAD ciphers. For a encryption operation, the authentication tag of
the given size will be generated. For a decryption operation, the
Documentation/devicetree/bindings/crypto/rockchip,rk3288-crypto.yaml
+127
View File
@@ -0,0 +1,127 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/crypto/rockchip,rk3288-crypto.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Rockchip Electronics Security Accelerator
maintainers:
- Heiko Stuebner <heiko@sntech.de>
properties:
compatible:
enum:
- rockchip,rk3288-crypto
- rockchip,rk3328-crypto
- rockchip,rk3399-crypto
reg:
maxItems: 1
interrupts:
maxItems: 1
clocks:
minItems: 3
maxItems: 4
clock-names:
minItems: 3
maxItems: 4
resets:
minItems: 1
maxItems: 3
reset-names:
minItems: 1
maxItems: 3
allOf:
- if:
properties:
compatible:
contains:
const: rockchip,rk3288-crypto
then:
properties:
clocks:
minItems: 4
clock-names:
items:
- const: aclk
- const: hclk
- const: sclk
- const: apb_pclk
resets:
maxItems: 1
reset-names:
items:
- const: crypto-rst
- if:
properties:
compatible:
contains:
const: rockchip,rk3328-crypto
then:
properties:
clocks:
maxItems: 3
clock-names:
items:
- const: hclk_master
- const: hclk_slave
- const: sclk
resets:
maxItems: 1
reset-names:
items:
- const: crypto-rst
- if:
properties:
compatible:
contains:
const: rockchip,rk3399-crypto
then:
properties:
clocks:
maxItems: 3
clock-names:
items:
- const: hclk_master
- const: hclk_slave
- const: sclk
resets:
minItems: 3
reset-names:
items:
- const: master
- const: slave
- const: crypto-rst
required:
- compatible
- reg
- interrupts
- clocks
- clock-names
- resets
- reset-names
additionalProperties: false
examples:
- |
#include <dt-bindings/interrupt-controller/arm-gic.h>
#include <dt-bindings/clock/rk3288-cru.h>
crypto@ff8a0000 {
compatible = "rockchip,rk3288-crypto";
reg = <0xff8a0000 0x4000>;
interrupts = <GIC_SPI 48 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&cru ACLK_CRYPTO>, <&cru HCLK_CRYPTO>,
<&cru SCLK_CRYPTO>, <&cru ACLK_DMAC1>;
clock-names = "aclk", "hclk", "sclk", "apb_pclk";
resets = <&cru SRST_CRYPTO>;
reset-names = "crypto-rst";
};
Documentation/devicetree/bindings/crypto/rockchip-crypto.txt
-28
View File
@@ -1,28 +0,0 @@
Rockchip Electronics And Security Accelerator
Required properties:
- compatible: Should be "rockchip,rk3288-crypto"
- reg: Base physical address of the engine and length of memory mapped
region
- interrupts: Interrupt number
- clocks: Reference to the clocks about crypto
- clock-names: "aclk" used to clock data
"hclk" used to clock data
"sclk" used to clock crypto accelerator
"apb_pclk" used to clock dma
- resets: Must contain an entry for each entry in reset-names.
See ../reset/reset.txt for details.
- reset-names: Must include the name "crypto-rst".
Examples:
crypto: cypto-controller@ff8a0000 {
compatible = "rockchip,rk3288-crypto";
reg = <0xff8a0000 0x4000>;
interrupts = <GIC_SPI 48 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&cru ACLK_CRYPTO>, <&cru HCLK_CRYPTO>,
<&cru SCLK_CRYPTO>, <&cru ACLK_DMAC1>;
clock-names = "aclk", "hclk", "sclk", "apb_pclk";
resets = <&cru SRST_CRYPTO>;
reset-names = "crypto-rst";
};
Documentation/devicetree/bindings/crypto/st,stm32-cryp.yaml
+19
View File
@@ -6,12 +6,18 @@ $schema: http://devicetree.org/meta-schemas/core.yaml#
title: STMicroelectronics STM32 CRYP bindings
description: The STM32 CRYP block is built on the CRYP block found in
the STn8820 SoC introduced in 2007, and subsequently used in the U8500
SoC in 2010.
maintainers:
- Lionel Debieve <lionel.debieve@foss.st.com>
properties:
compatible:
enum:
- st,stn8820-cryp
- stericsson,ux500-cryp
- st,stm32f756-cryp
- st,stm32mp1-cryp
@@ -27,6 +33,19 @@ properties:
resets:
maxItems: 1
dmas:
items:
- description: mem2cryp DMA channel
- description: cryp2mem DMA channel
dma-names:
items:
- const: mem2cryp
- const: cryp2mem
power-domains:
maxItems: 1
required:
- compatible
- reg
Documentation/devicetree/bindings/gpio/gpio-davinci.yaml
+1 -1
View File
@@ -35,7 +35,7 @@ properties:
gpio-line-names:
description: strings describing the names of each gpio line.
minItems: 1
maxItems: 100
maxItems: 144
"#gpio-cells":
const: 2
Documentation/devicetree/bindings/gpio/gpio-latch.yaml
+94
View File
@@ -0,0 +1,94 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/gpio/gpio-latch.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: GPIO latch controller
maintainers:
- Sascha Hauer <s.hauer@pengutronix.de>
description: |
This binding describes a GPIO multiplexer based on latches connected to
other GPIOs, like this:
CLK0 ----------------------. ,--------.
CLK1 -------------------. `--------|> #0 |
| | |
OUT0 ----------------+--|-----------|D0 Q0|-----|<
OUT1 --------------+-|--|-----------|D1 Q1|-----|<
OUT2 ------------+-|-|--|-----------|D2 Q2|-----|<
OUT3 ----------+-|-|-|--|-----------|D3 Q3|-----|<
OUT4 --------+-|-|-|-|--|-----------|D4 Q4|-----|<
OUT5 ------+-|-|-|-|-|--|-----------|D5 Q5|-----|<
OUT6 ----+-|-|-|-|-|-|--|-----------|D6 Q6|-----|<
OUT7 --+-|-|-|-|-|-|-|--|-----------|D7 Q7|-----|<
| | | | | | | | | `--------'
| | | | | | | | |
| | | | | | | | | ,--------.
| | | | | | | | `-----------|> #1 |
| | | | | | | | | |
| | | | | | | `--------------|D0 Q0|-----|<
| | | | | | `----------------|D1 Q1|-----|<
| | | | | `------------------|D2 Q2|-----|<
| | | | `--------------------|D3 Q3|-----|<
| | | `----------------------|D4 Q4|-----|<
| | `------------------------|D5 Q5|-----|<
| `--------------------------|D6 Q6|-----|<
`----------------------------|D7 Q7|-----|<
`--------'
The number of clk-gpios and latched-gpios is not fixed. The actual number
of number of latches and the number of inputs per latch is derived from
the number of GPIOs given in the corresponding device tree properties.
properties:
compatible:
const: gpio-latch
"#gpio-cells":
const: 2
clk-gpios:
description: Array of GPIOs to be used to clock a latch
latched-gpios:
description: Array of GPIOs to be used as inputs per latch
setup-duration-ns:
description: Delay in nanoseconds to wait after the latch inputs have been
set up
clock-duration-ns:
description: Delay in nanoseconds to wait between clock output changes
gpio-controller: true
gpio-line-names: true
required:
- compatible
- "#gpio-cells"
- gpio-controller
- clk-gpios
- latched-gpios
additionalProperties: false
examples:
- |
gpio-latch {
#gpio-cells = <2>;
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_di_do_leds>;
compatible = "gpio-latch";
gpio-controller;
setup-duration-ns = <100>;
clock-duration-ns = <100>;
clk-gpios = <&gpio3 7 0>, <&gpio3 8 0>;
latched-gpios = <&gpio3 21 0>, <&gpio3 22 0>,
<&gpio3 23 0>, <&gpio3 24 0>,
<&gpio3 25 0>, <&gpio3 26 0>,
<&gpio3 27 0>, <&gpio3 28 0>;
};
Documentation/devicetree/bindings/gpio/gpio-pca9570.yaml
+1
View File
@@ -12,6 +12,7 @@ maintainers:
properties:
compatible:
enum:
- dlg,slg7xl45106
- nxp,pca9570
- nxp,pca9571
Documentation/devicetree/bindings/gpio/hisilicon,ascend910-gpio.yaml
+56
View File
@@ -0,0 +1,56 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/gpio/hisilicon,ascend910-gpio.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: HiSilicon common GPIO controller
maintainers:
- Jay Fang <f.fangjian@huawei.com>
description:
The HiSilicon common GPIO controller can be used for many different
types of SoC such as Huawei Ascend AI series chips.
properties:
compatible:
const: hisilicon,ascend910-gpio
reg:
maxItems: 1
interrupts:
maxItems: 1
gpio-controller: true
"#gpio-cells":
const: 2
ngpios:
minimum: 1
maximum: 32
required:
- compatible
- reg
- interrupts
- gpio-controller
- "#gpio-cells"
- ngpios
additionalProperties: false
examples:
- |
#include <dt-bindings/interrupt-controller/arm-gic.h>
gpio@840d0000 {
compatible = "hisilicon,ascend910-gpio";
reg = <0x840d0000 0x1000>;
ngpios = <32>;
gpio-controller;
#gpio-cells = <2>;
interrupts = <GIC_SPI 33 IRQ_TYPE_LEVEL_HIGH>;
};
Documentation/devicetree/bindings/i2c/hisilicon,ascend910-i2c.yaml
+73
View File
@@ -0,0 +1,73 @@
# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/i2c/hisilicon,ascend910-i2c.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: HiSilicon common I2C controller
maintainers:
- Yicong Yang <yangyicong@hisilicon.com>
description:
The HiSilicon common I2C controller can be used for many different
types of SoC such as Huawei Ascend AI series chips.
allOf:
- $ref: /schemas/i2c/i2c-controller.yaml#
properties:
compatible:
const: hisilicon,ascend910-i2c
reg:
maxItems: 1
interrupts:
maxItems: 1
clocks:
maxItems: 1
clock-frequency:
default: 400000
i2c-sda-falling-time-ns:
default: 343
i2c-scl-falling-time-ns:
default: 203
i2c-sda-hold-time-ns:
default: 830
i2c-scl-rising-time-ns:
default: 365
i2c-digital-filter-width-ns:
default: 0
required:
- compatible
- reg
- interrupts
unevaluatedProperties: false
examples:
- |
#include <dt-bindings/interrupt-controller/arm-gic.h>
i2c@38b0000 {
compatible = "hisilicon,ascend910-i2c";
reg = <0x38b0000 0x10000>;
interrupts = <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>;
i2c-sda-falling-time-ns = <56>;
i2c-scl-falling-time-ns = <56>;
i2c-sda-hold-time-ns = <56>;
i2c-scl-rising-time-ns = <56>;
i2c-digital-filter;
i2c-digital-filter-width-ns = <0x0>;
clocks = <&alg_clk>;
clock-frequency = <400000>;
};
Documentation/devicetree/bindings/i2c/i2c-mt65xx.yaml
+1
View File
@@ -23,6 +23,7 @@ properties:
- const: mediatek,mt6577-i2c
- const: mediatek,mt6589-i2c
- const: mediatek,mt7622-i2c
- const: mediatek,mt7986-i2c
- const: mediatek,mt8168-i2c
- const: mediatek,mt8173-i2c
- const: mediatek,mt8183-i2c
Documentation/devicetree/bindings/i2c/marvell,mv64xxx-i2c.yaml
+1
View File
@@ -19,6 +19,7 @@ properties:
- const: allwinner,sun6i-a31-i2c
- items:
- enum:
- allwinner,suniv-f1c100s-i2c
- allwinner,sun8i-a23-i2c
- allwinner,sun8i-a83t-i2c
- allwinner,sun8i-v536-i2c
Documentation/devicetree/bindings/i2c/qcom,i2c-geni-qcom.yaml
+54 -10
View File
@@ -10,18 +10,19 @@ maintainers:
- Andy Gross <agross@kernel.org>
- Bjorn Andersson <bjorn.andersson@linaro.org>
allOf:
- $ref: /schemas/i2c/i2c-controller.yaml#
properties:
compatible:
const: qcom,geni-i2c
enum:
- qcom,geni-i2c
- qcom,geni-i2c-master-hub
clocks:
maxItems: 1
minItems: 1
maxItems: 2
clock-names:
const: se
minItems: 1
maxItems: 2
clock-frequency:
default: 100000
@@ -35,13 +36,12 @@ properties:
- const: rx
interconnects:
minItems: 2
maxItems: 3
interconnect-names:
items:
- const: qup-core
- const: qup-config
- const: qup-memory
minItems: 2
maxItems: 3
interrupts:
maxItems: 1
@@ -71,6 +71,50 @@ required:
- clock-names
- reg
allOf:
- $ref: /schemas/i2c/i2c-controller.yaml#
- if:
properties:
compatible:
contains:
const: qcom,geni-i2c-master-hub
then:
properties:
clocks:
minItems: 2
clock-names:
items:
- const: se
- const: core
dmas: false
dma-names: false
interconnects:
maxItems: 2
interconnect-names:
items:
- const: qup-core
- const: qup-config
else:
properties:
clocks:
maxItems: 1
clock-names:
const: se
interconnects:
minItems: 3
interconnect-names:
items:
- const: qup-core
- const: qup-config
- const: qup-memory
unevaluatedProperties: false
examples:
Documentation/devicetree/bindings/i2c/renesas,riic.yaml
+1 -1
View File
@@ -19,7 +19,7 @@ properties:
- enum:
- renesas,riic-r7s72100 # RZ/A1H
- renesas,riic-r7s9210 # RZ/A2M
- renesas,riic-r9a07g043 # RZ/G2UL
- renesas,riic-r9a07g043 # RZ/G2UL and RZ/Five
- renesas,riic-r9a07g044 # RZ/G2{L,LC}
- renesas,riic-r9a07g054 # RZ/V2L
- const: renesas,riic-rz # RZ/A or RZ/G2L
Documentation/devicetree/bindings/pci/baikal,bt1-pcie.yaml
+168
View File
@@ -0,0 +1,168 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/pci/baikal,bt1-pcie.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Baikal-T1 PCIe Root Port Controller
maintainers:
- Serge Semin <fancer.lancer@gmail.com>
description:
Embedded into Baikal-T1 SoC Root Complex controller with a single port
activated. It's based on the DWC RC PCIe v4.60a IP-core, which is configured
to have just a single Root Port function and is capable of establishing the
link up to Gen.3 speed on x4 lanes. It doesn't have embedded clock and reset
control module, so the proper interface initialization is supposed to be
performed by software. There four in- and four outbound iATU regions
which can be used to emit all required TLP types on the PCIe bus.
allOf:
- $ref: /schemas/pci/snps,dw-pcie.yaml#
properties:
compatible:
const: baikal,bt1-pcie
reg:
description:
DBI, DBI2 and at least 4KB outbound iATU-capable region for the
peripheral devices CFG-space access.
maxItems: 3
reg-names:
items:
- const: dbi
- const: dbi2
- const: config
interrupts:
description:
MSI, AER, PME, Hot-plug, Link Bandwidth Management, Link Equalization
request and eight Read/Write eDMA IRQ lines are available.
maxItems: 14
interrupt-names:
items:
- const: dma0
- const: dma1
- const: dma2
- const: dma3
- const: dma4
- const: dma5
- const: dma6
- const: dma7
- const: msi
- const: aer
- const: pme
- const: hp
- const: bw_mg
- const: l_eq
clocks:
description:
DBI (attached to the APB bus), AXI-bus master and slave interfaces
are fed up by the dedicated application clocks. A common reference
clock signal is supposed to be attached to the corresponding Ref-pad
of the SoC. It will be redistributed amongst the controller core
sub-modules (pipe, core, aux, etc).
maxItems: 4
clock-names:
items:
- const: dbi
- const: mstr
- const: slv
- const: ref
resets:
description:
A comprehensive controller reset logic is supposed to be implemented
by software, so almost all the possible application and core reset
signals are exposed via the system CCU module.
maxItems: 9
reset-names:
items:
- const: mstr
- const: slv
- const: pwr
- const: hot
- const: phy
- const: core
- const: pipe
- const: sticky
- const: non-sticky
baikal,bt1-syscon:
$ref: /schemas/types.yaml#/definitions/phandle
description:
Phandle to the Baikal-T1 System Controller DT node. It's required to
access some additional PM, Reset-related and LTSSM signals.
num-lanes:
maximum: 4
max-link-speed:
maximum: 3
required:
- compatible
- reg
- reg-names
- interrupts
- interrupt-names
unevaluatedProperties: false
examples:
- |
#include <dt-bindings/interrupt-controller/mips-gic.h>
#include <dt-bindings/gpio/gpio.h>
pcie@1f052000 {
compatible = "baikal,bt1-pcie";
device_type = "pci";
reg = <0x1f052000 0x1000>, <0x1f053000 0x1000>, <0x1bdbf000 0x1000>;
reg-names = "dbi", "dbi2", "config";
#address-cells = <3>;
#size-cells = <2>;
ranges = <0x81000000 0 0x00000000 0x1bdb0000 0 0x00008000>,
<0x82000000 0 0x20000000 0x08000000 0 0x13db0000>;
bus-range = <0x0 0xff>;
interrupts = <GIC_SHARED 80 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 81 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 82 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 83 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 84 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 85 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 86 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 87 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 88 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 89 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 90 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 91 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 92 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SHARED 93 IRQ_TYPE_LEVEL_HIGH>;
interrupt-names = "dma0", "dma1", "dma2", "dma3",
"dma4", "dma5", "dma6", "dma7",
"msi", "aer", "pme", "hp", "bw_mg",
"l_eq";
clocks = <&ccu_sys 1>, <&ccu_axi 6>, <&ccu_axi 7>, <&clk_pcie>;
clock-names = "dbi", "mstr", "slv", "ref";
resets = <&ccu_axi 6>, <&ccu_axi 7>, <&ccu_sys 7>, <&ccu_sys 10>,
<&ccu_sys 4>, <&ccu_sys 6>, <&ccu_sys 5>, <&ccu_sys 8>,
<&ccu_sys 9>;
reset-names = "mstr", "slv", "pwr", "hot", "phy", "core", "pipe",
"sticky", "non-sticky";
reset-gpios = <&port0 0 GPIO_ACTIVE_LOW>;
num-lanes = <4>;
max-link-speed = <3>;
};
...
Documentation/devicetree/bindings/pci/fsl,imx6q-pcie.yaml
+42 -4
View File
@@ -14,9 +14,6 @@ description: |+
This PCIe host controller is based on the Synopsys DesignWare PCIe IP
and thus inherits all the common properties defined in snps,dw-pcie.yaml.
allOf:
- $ref: /schemas/pci/snps,dw-pcie.yaml#
properties:
compatible:
enum:
@@ -61,7 +58,7 @@ properties:
- const: pcie
- const: pcie_bus
- const: pcie_phy
- const: pcie_inbound_axi for imx6sx-pcie, pcie_aux for imx8mq-pcie
- enum: [ pcie_inbound_axi, pcie_aux ]
num-lanes:
const: 1
@@ -175,6 +172,47 @@ required:
- clocks
- clock-names
allOf:
- $ref: /schemas/pci/snps,dw-pcie.yaml#
- if:
properties:
compatible:
contains:
const: fsl,imx6sx-pcie
then:
properties:
clock-names:
items:
- {}
- {}
- {}
- const: pcie_inbound_axi
- if:
properties:
compatible:
contains:
const: fsl,imx8mq-pcie
then:
properties:
clock-names:
items:
- {}
- {}
- {}
- const: pcie_aux
- if:
properties:
compatible:
not:
contains:
enum:
- fsl,imx6sx-pcie
- fsl,imx8mq-pcie
then:
properties:
clock-names:
maxItems: 3
unevaluatedProperties: false
examples:
Documentation/devicetree/bindings/pci/mediatek-pcie-gen3.yaml
+63 -14
View File
@@ -43,14 +43,12 @@ description: |+
each set has its own address for MSI message, and supports 32 MSI vectors
to generate interrupt.
allOf:
- $ref: /schemas/pci/pci-bus.yaml#
properties:
compatible:
oneOf:
- items:
- enum:
- mediatek,mt7986-pcie
- mediatek,mt8188-pcie
- mediatek,mt8195-pcie
- const: mediatek,mt8192-pcie
@@ -70,29 +68,29 @@ properties:
minItems: 1
maxItems: 8
iommu-map:
maxItems: 1
iommu-map-mask:
const: 0
resets:
minItems: 1
maxItems: 2
reset-names:
minItems: 1
maxItems: 2
items:
- const: phy
- const: mac
enum: [ phy, mac ]
clocks:
minItems: 4
maxItems: 6
clock-names:
items:
- const: pl_250m
- const: tl_26m
- const: tl_96m
- const: tl_32k
- const: peri_26m
- enum:
- top_133m # for MT8192
- peri_mem # for MT8188/MT8195
minItems: 4
maxItems: 6
assigned-clocks:
maxItems: 1
@@ -107,6 +105,9 @@ properties:
items:
- const: pcie-phy
power-domains:
maxItems: 1
'#interrupt-cells':
const: 1
@@ -138,6 +139,54 @@ required:
- '#interrupt-cells'
- interrupt-controller
allOf:
- $ref: /schemas/pci/pci-bus.yaml#
- if:
properties:
compatible:
const: mediatek,mt8192-pcie
then:
properties:
clock-names:
items:
- const: pl_250m
- const: tl_26m
- const: tl_96m
- const: tl_32k
- const: peri_26m
- const: top_133m
- if:
properties:
compatible:
contains:
enum:
- mediatek,mt8188-pcie
- mediatek,mt8195-pcie
then:
properties:
clock-names:
items:
- const: pl_250m
- const: tl_26m
- const: tl_96m
- const: tl_32k
- const: peri_26m
- const: peri_mem
- if:
properties:
compatible:
contains:
enum:
- mediatek,mt7986-pcie
then:
properties:
clock-names:
items:
- const: pl_250m
- const: tl_26m
- const: peri_26m
- const: top_133m
unevaluatedProperties: false
examples:
Documentation/devicetree/bindings/pci/qcom,pcie.yaml
+22
View File
@@ -62,6 +62,16 @@ properties:
minItems: 3
maxItems: 13
dma-coherent: true
interconnects:
maxItems: 2
interconnect-names:
items:
- const: pcie-mem
- const: cpu-pcie
resets:
minItems: 1
maxItems: 12
@@ -631,6 +641,18 @@ allOf:
items:
- const: pci # PCIe core reset
- if:
properties:
compatible:
contains:
enum:
- qcom,pcie-sa8540p
- qcom,pcie-sc8280xp
then:
required:
- interconnects
- interconnect-names
- if:
not:
properties:
Documentation/devicetree/bindings/pci/rockchip-dw-pcie.yaml
+2 -2
View File
@@ -14,10 +14,10 @@ maintainers:
description: |+
RK3568 SoC PCIe host controller is based on the Synopsys DesignWare
PCIe IP and thus inherits all the common properties defined in
designware-pcie.txt.
snps,dw-pcie.yaml.
allOf:
- $ref: /schemas/pci/pci-bus.yaml#
- $ref: /schemas/pci/snps,dw-pcie.yaml#
properties:
compatible:
Documentation/devicetree/bindings/pci/snps,dw-pcie-common.yaml
+266
View File
@@ -0,0 +1,266 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/pci/snps,dw-pcie-common.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Synopsys DWC PCIe RP/EP controller
maintainers:
- Jingoo Han <jingoohan1@gmail.com>
- Gustavo Pimentel <gustavo.pimentel@synopsys.com>
description:
Generic Synopsys DesignWare PCIe Root Port and Endpoint controller
properties.
select: false
properties:
reg:
description:
DWC PCIe CSR space is normally accessed over the dedicated Data Bus
Interface - DBI. In accordance with the reference manual the register
configuration space belongs to the Configuration-Dependent Module (CDM)
and is split up into several sub-parts Standard PCIe configuration
space, Port Logic Registers (PL), Shadow Config-space Registers,
iATU/eDMA registers. The particular sub-space is selected by the
CDM/ELBI (dbi_cs) and CS2 (dbi_cs2) signals (selector bits). Such
configuration provides a flexible interface for the system engineers to
either map the particular space at a desired MMIO address or just leave
them in a contiguous memory space if pure Native or AXI Bridge DBI access
is selected. Note the PCIe CFG-space, PL and Shadow registers are
specific for each activated function, while the rest of the sub-spaces
are common for all of them (if there are more than one).
minItems: 2
maxItems: 6
reg-names:
minItems: 2
maxItems: 6
interrupts:
description:
There are two main sub-blocks which are normally capable of
generating interrupts. It's System Information Interface and MSI
interface. While the former one has some common for the Host and
Endpoint controllers IRQ-signals, the later interface is obviously
Root Complex specific since it's responsible for the incoming MSI
messages signalling. The System Information IRQ signals are mainly
responsible for reporting the generic PCIe hierarchy and Root
Complex events like VPD IO request, general AER, PME, Hot-plug, link
bandwidth change, link equalization request, INTx asserted/deasserted
Message detection, embedded DMA Tx/Rx/Error.
minItems: 1
maxItems: 26
interrupt-names:
minItems: 1
maxItems: 26
clocks:
description:
DWC PCIe reference manual explicitly defines a set of the clocks required
to get the controller working correctly. In general all of them can
be divided into two groups':' application and core clocks. Note the
platforms may have some of the clock sources unspecified in case if the
corresponding domains are fed up from a common clock source.
minItems: 1
maxItems: 7
clock-names:
minItems: 1
maxItems: 7
items:
oneOf:
- description:
Data Bus Interface (DBI) clock. Clock signal for the AXI-bus
interface of the Configuration-Dependent Module, which is
basically the set of the controller CSRs.
const: dbi
- description:
Application AXI-bus Master interface clock. Basically this is
a clock for the controller DMA interface (PCI-to-CPU).
const: mstr
- description:
Application AXI-bus Slave interface clock. This is a clock for
the CPU-to-PCI memory IO interface.
const: slv
- description:
Controller Core-PCS PIPE interface clock. It's normally
supplied by an external PCS-PHY.
const: pipe
- description:
Controller Primary clock. It's assumed that all controller input
signals (except resets) are synchronous to this clock.
const: core
- description:
Auxiliary clock for the controller PMC domain. The controller
partitioning implies having some parts to operate with this
clock in some power management states.
const: aux
- description:
Generic reference clock. In case if there are several
interfaces fed up with a common clock source it's advisable to
define it with this name (for instance pipe, core and aux can
be connected to a single source of the periodic signal).
const: ref
- description:
Clock for the PHY registers interface. Originally this is
a PHY-viewport-based interface, but some platform may have
specifically designed one.
const: phy_reg
- description:
Vendor-specific clock names. Consider using the generic names
above for new bindings.
oneOf:
- description: See native 'dbi' clock for details
enum: [ pcie, pcie_apb_sys, aclk_dbi ]
- description: See native 'mstr/slv' clock for details
enum: [ pcie_bus, pcie_inbound_axi, pcie_aclk, aclk_mst, aclk_slv ]
- description: See native 'pipe' clock for details
enum: [ pcie_phy, pcie_phy_ref, link ]
- description: See native 'aux' clock for details
enum: [ pcie_aux ]
- description: See native 'ref' clock for details.
enum: [ gio ]
- description: See nativs 'phy_reg' clock for details
enum: [ pcie_apb_phy, pclk ]
resets:
description:
DWC PCIe reference manual explicitly defines a set of the reset
signals required to be de-asserted to properly activate the controller
sub-parts. All of these signals can be divided into two sub-groups':'
application and core resets with respect to the main sub-domains they
are supposed to reset. Note the platforms may have some of these signals
unspecified in case if they are automatically handled or aggregated into
a comprehensive control module.
minItems: 1
maxItems: 10
reset-names:
minItems: 1
maxItems: 10
items:
oneOf:
- description: Data Bus Interface (DBI) domain reset
const: dbi
- description: AXI-bus Master interface reset
const: mstr
- description: AXI-bus Slave interface reset
const: slv
- description: Application-dependent interface reset
const: app
- description: Controller Non-sticky CSR flags reset
const: non-sticky
- description: Controller sticky CSR flags reset
const: sticky
- description: PIPE-interface (Core-PCS) logic reset
const: pipe
- description:
Controller primary reset (resets everything except PMC module)
const: core
- description: PCS/PHY block reset
const: phy
- description: PMC hot reset signal
const: hot
- description: Cold reset signal
const: pwr
- description:
Vendor-specific reset names. Consider using the generic names
above for new bindings.
oneOf:
- description: See native 'app' reset for details
enum: [ apps, gio, apb ]
- description: See native 'phy' reset for details
enum: [ pciephy, link ]
- description: See native 'pwr' reset for details
enum: [ turnoff ]
phys:
description:
There can be up to the number of possible lanes PHYs specified placed in
the phandle array in the line-based order. Obviously each the specified
PHYs are supposed to be able to work in the PCIe mode with a speed
implied by the DWC PCIe controller they are attached to.
minItems: 1
maxItems: 16
phy-names:
minItems: 1
maxItems: 16
oneOf:
- description: Generic PHY names
items:
pattern: '^pcie[0-9]+$'
- description:
Vendor-specific PHY names. Consider using the generic
names above for new bindings.
items:
oneOf:
- pattern: '^pcie(-?phy[0-9]*)?$'
- pattern: '^p2u-[0-7]$'
reset-gpio:
deprecated: true
description:
Reference to the GPIO-controlled PERST# signal. It is used to reset all
the peripheral devices available on the PCIe bus.
maxItems: 1
reset-gpios:
description:
Reference to the GPIO-controlled PERST# signal. It is used to reset all
the peripheral devices available on the PCIe bus.
maxItems: 1
max-link-speed:
maximum: 5
num-lanes:
description:
Number of PCIe link lanes to use. Can be omitted if the already brought
up link is supposed to be preserved.
maximum: 16
num-ob-windows:
$ref: /schemas/types.yaml#/definitions/uint32
deprecated: true
description:
Number of outbound address translation windows. This parameter can be
auto-detected based on the iATU memory writability. So there is no
point in having a dedicated DT-property for it.
maximum: 256
num-ib-windows:
$ref: /schemas/types.yaml#/definitions/uint32
deprecated: true
description:
Number of inbound address translation windows. In the same way as
for the outbound AT windows, this parameter can be auto-detected based
on the iATU memory writability. There is no point having a dedicated
DT-property for it either.
maximum: 256
num-viewport:
$ref: /schemas/types.yaml#/definitions/uint32
deprecated: true
description:
Number of outbound view ports configured in hardware. It's the same as
the number of outbound AT windows.
maximum: 256
snps,enable-cdm-check:
$ref: /schemas/types.yaml#/definitions/flag
description:
Enable automatic checking of CDM (Configuration Dependent Module)
registers for data corruption. CDM registers include standard PCIe
configuration space registers, Port Logic registers, DMA and iATU
registers. This feature has been available since DWC PCIe v4.80a.
dma-coherent: true
additionalProperties: true
...
Documentation/devicetree/bindings/pci/snps,dw-pcie-ep.yaml
+156 -50
View File
@@ -13,76 +13,182 @@ maintainers:
description: |
Synopsys DesignWare PCIe host controller endpoint
# Please create a separate DT-schema for your DWC PCIe Endpoint controller
# and make sure it's assigned with the vendor-specific compatible string.
select:
properties:
compatible:
const: snps,dw-pcie-ep
required:
- compatible
allOf:
- $ref: /schemas/pci/pci-ep.yaml#
- $ref: /schemas/pci/snps,dw-pcie-common.yaml#
properties:
compatible:
anyOf:
- {}
- const: snps,dw-pcie-ep
reg:
description: |
It should contain Data Bus Interface (dbi) and config registers for all
versions.
For designware core version >= 4.80, it may contain ATU address space.
description:
DBI, DBI2 reg-spaces and outbound memory window are required for the
normal controller functioning. iATU memory IO region is also required
if the space is unrolled (IP-core version >= 4.80a).
minItems: 2
maxItems: 4
maxItems: 5
reg-names:
minItems: 2
maxItems: 4
maxItems: 5
items:
enum: [dbi, dbi2, config, atu, addr_space, link, atu_dma, appl]
oneOf:
- description:
Basic DWC PCIe controller configuration-space accessible over
the DBI interface. This memory space is either activated with
CDM/ELBI = 0 and CS2 = 0 or is a contiguous memory region
with all spaces. Note iATU/eDMA CSRs are indirectly accessible
via the PL viewports on the DWC PCIe controllers older than
v4.80a.
const: dbi
- description:
Shadow DWC PCIe config-space registers. This space is selected
by setting CDM/ELBI = 0 and CS2 = 1. This is an intermix of
the PCI-SIG PCIe CFG-space with the shadow registers for some
PCI Header space, PCI Standard and Extended Structures. It's
mainly relevant for the end-point controller configuration,
but still there are some shadow registers available for the
Root Port mode too.
const: dbi2
- description:
External Local Bus registers. It's an application-dependent
registers normally defined by the platform engineers. The space
can be selected by setting CDM/ELBI = 1 and CS2 = 0 wires or can
be accessed over some platform-specific means (for instance
as a part of a system controller).
enum: [ elbi, app ]
- description:
iATU/eDMA registers common for all device functions. It's an
unrolled memory space with the internal Address Translation
Unit and Enhanced DMA, which is selected by setting CDM/ELBI = 1
and CS2 = 1. For IP-core releases prior v4.80a, these registers
have been programmed via an indirect addressing scheme using a
set of viewport CSRs mapped into the PL space. Note iATU is
normally mapped to the 0x0 address of this region, while eDMA
is available at 0x80000 base address.
const: atu
- description:
Platform-specific eDMA registers. Some platforms may have eDMA
CSRs mapped in a non-standard base address. The registers offset
can be changed or the MS/LS-bits of the address can be attached
in an additional RTL block before the MEM-IO transactions reach
the DW PCIe slave interface.
const: dma
- description:
PHY/PCS configuration registers. Some platforms can have the
PCS and PHY CSRs accessible over a dedicated memory mapped
region, but mainly these registers are indirectly accessible
either by means of the embedded PHY viewport schema or by some
platform-specific method.
const: phy
- description:
Outbound iATU-capable memory-region which will be used to
generate various application-specific traffic on the PCIe bus
hierarchy. It's usage scenario depends on the endpoint
functionality, for instance it can be used to create MSI(X)
messages.
const: addr_space
- description:
Vendor-specific CSR names. Consider using the generic names above
for new bindings.
oneOf:
- description: See native 'elbi/app' CSR region for details.
enum: [ link, appl ]
- description: See native 'atu' CSR region for details.
enum: [ atu_dma ]
allOf:
- contains:
const: dbi
- contains:
const: addr_space
reset-gpio:
description: GPIO pin number of PERST# signal
maxItems: 1
deprecated: true
interrupts:
description:
There is no mandatory IRQ signals for the normal controller functioning,
but in addition to the native set the platforms may have a link- or
PM-related IRQs specified.
minItems: 1
maxItems: 20
reset-gpios:
description: GPIO controlled connection to PERST# signal
maxItems: 1
interrupt-names:
minItems: 1
maxItems: 20
items:
oneOf:
- description:
Controller request to read or write virtual product data
from/to the VPD capability registers.
const: vpd
- description:
Link Equalization Request flag is set in the Link Status 2
register (applicable if the corresponding IRQ is enabled in
the Link Control 3 register).
const: l_eq
- description:
Indicates that the eDMA Tx/Rx transfer is complete or that an
error has occurred on the corresponding channel. eDMA can have
eight Tx (Write) and Rx (Read) eDMA channels thus supporting up
to 16 IRQ signals all together. Write eDMA channels shall go
first in the ordered row as per default edma_int[*] bus setup.
pattern: '^dma([0-9]|1[0-5])?$'
- description:
PCIe protocol correctable error or a Data Path protection
correctable error is detected by the automotive/safety
feature.
const: sft_ce
- description:
Indicates that the internal safety mechanism has detected an
uncorrectable error.
const: sft_ue
- description:
Application-specific IRQ raised depending on the vendor-specific
events basis.
const: app
- description:
Vendor-specific IRQ names. Consider using the generic names above
for new bindings.
oneOf:
- description: See native "app" IRQ for details
enum: [ intr ]
snps,enable-cdm-check:
type: boolean
description: |
This is a boolean property and if present enables
automatic checking of CDM (Configuration Dependent Module) registers
for data corruption. CDM registers include standard PCIe configuration
space registers, Port Logic registers, DMA and iATU (internal Address
Translation Unit) registers.
num-ib-windows:
$ref: /schemas/types.yaml#/definitions/uint32
maximum: 256
description: number of inbound address translation windows
deprecated: true
num-ob-windows:
$ref: /schemas/types.yaml#/definitions/uint32
maximum: 256
description: number of outbound address translation windows
deprecated: true
max-functions:
maximum: 32
required:
- compatible
- reg
- reg-names
- compatible
additionalProperties: true
examples:
- |
bus {
#address-cells = <1>;
#size-cells = <1>;
pcie-ep@dfd00000 {
compatible = "snps,dw-pcie-ep";
reg = <0xdfc00000 0x0001000>, /* IP registers 1 */
<0xdfc01000 0x0001000>, /* IP registers 2 */
<0xd0000000 0x2000000>; /* Configuration space */
reg-names = "dbi", "dbi2", "addr_space";
};
pcie-ep@dfd00000 {
compatible = "snps,dw-pcie-ep";
reg = <0xdfc00000 0x0001000>, /* IP registers 1 */
<0xdfc01000 0x0001000>, /* IP registers 2 */
<0xd0000000 0x2000000>; /* Configuration space */
reg-names = "dbi", "dbi2", "addr_space";
interrupts = <23>, <24>;
interrupt-names = "dma0", "dma1";
clocks = <&sys_clk 12>, <&sys_clk 24>;
clock-names = "dbi", "ref";
resets = <&sys_rst 12>, <&sys_rst 24>;
reset-names = "dbi", "phy";
phys = <&pcie_phy0>, <&pcie_phy1>, <&pcie_phy2>, <&pcie_phy3>;
phy-names = "pcie0", "pcie1", "pcie2", "pcie3";
max-link-speed = <3>;
max-functions = /bits/ 8 <4>;
};
Documentation/devicetree/bindings/pci/snps,dw-pcie.yaml
+193 -65
View File
@@ -13,20 +13,25 @@ maintainers:
description: |
Synopsys DesignWare PCIe host controller
# Please create a separate DT-schema for your DWC PCIe Root Port controller
# and make sure it's assigned with the vendor-specific compatible string.
select:
properties:
compatible:
const: snps,dw-pcie
required:
- compatible
allOf:
- $ref: /schemas/pci/pci-bus.yaml#
- $ref: /schemas/pci/snps,dw-pcie-common.yaml#
properties:
compatible:
anyOf:
- {}
- const: snps,dw-pcie
reg:
description: |
It should contain Data Bus Interface (dbi) and config registers for all
versions.
For designware core version >= 4.80, it may contain ATU address space.
description:
At least DBI reg-space and peripheral devices CFG-space outbound window
are required for the normal controller work. iATU memory IO region is
also required if the space is unrolled (IP-core version >= 4.80a).
minItems: 2
maxItems: 5
@@ -34,71 +39,194 @@ properties:
minItems: 2
maxItems: 5
items:
enum: [ dbi, dbi2, config, atu, atu_dma, app, appl, elbi, mgmt, ctrl,
parf, cfg, link, ulreg, smu, mpu, apb, phy, ecam ]
oneOf:
- description:
Basic DWC PCIe controller configuration-space accessible over
the DBI interface. This memory space is either activated with
CDM/ELBI = 0 and CS2 = 0 or is a contiguous memory region
with all spaces. Note iATU/eDMA CSRs are indirectly accessible
via the PL viewports on the DWC PCIe controllers older than
v4.80a.
const: dbi
- description:
Shadow DWC PCIe config-space registers. This space is selected
by setting CDM/ELBI = 0 and CS2 = 1. This is an intermix of
the PCI-SIG PCIe CFG-space with the shadow registers for some
PCI Header space, PCI Standard and Extended Structures. It's
mainly relevant for the end-point controller configuration,
but still there are some shadow registers available for the
Root Port mode too.
const: dbi2
- description:
External Local Bus registers. It's an application-dependent
registers normally defined by the platform engineers. The space
can be selected by setting CDM/ELBI = 1 and CS2 = 0 wires or can
be accessed over some platform-specific means (for instance
as a part of a system controller).
enum: [ elbi, app ]
- description:
iATU/eDMA registers common for all device functions. It's an
unrolled memory space with the internal Address Translation
Unit and Enhanced DMA, which is selected by setting CDM/ELBI = 1
and CS2 = 1. For IP-core releases prior v4.80a, these registers
have been programmed via an indirect addressing scheme using a
set of viewport CSRs mapped into the PL space. Note iATU is
normally mapped to the 0x0 address of this region, while eDMA
is available at 0x80000 base address.
const: atu
- description:
Platform-specific eDMA registers. Some platforms may have eDMA
CSRs mapped in a non-standard base address. The registers offset
can be changed or the MS/LS-bits of the address can be attached
in an additional RTL block before the MEM-IO transactions reach
the DW PCIe slave interface.
const: dma
- description:
PHY/PCS configuration registers. Some platforms can have the
PCS and PHY CSRs accessible over a dedicated memory mapped
region, but mainly these registers are indirectly accessible
either by means of the embedded PHY viewport schema or by some
platform-specific method.
const: phy
- description:
Outbound iATU-capable memory-region which will be used to access
the peripheral PCIe devices configuration space.
const: config
- description:
Vendor-specific CSR names. Consider using the generic names above
for new bindings.
oneOf:
- description: See native 'elbi/app' CSR region for details.
enum: [ apb, mgmt, link, ulreg, appl ]
- description: See native 'atu' CSR region for details.
enum: [ atu_dma ]
- description: Syscon-related CSR regions.
enum: [ smu, mpu ]
- description: Tegra234 aperture
enum: [ ecam ]
allOf:
- contains:
const: dbi
- contains:
const: config
num-lanes:
description: |
number of lanes to use (this property should be specified unless
the link is brought already up in firmware)
maximum: 16
interrupts:
description:
DWC PCIe Root Port/Complex specific IRQ signals. At least MSI interrupt
signal is supposed to be specified for the host controller.
minItems: 1
maxItems: 26
reset-gpio:
description: GPIO pin number of PERST# signal
maxItems: 1
deprecated: true
reset-gpios:
description: GPIO controlled connection to PERST# signal
maxItems: 1
interrupts: true
interrupt-names: true
clocks: true
snps,enable-cdm-check:
type: boolean
description: |
This is a boolean property and if present enables
automatic checking of CDM (Configuration Dependent Module) registers
for data corruption. CDM registers include standard PCIe configuration
space registers, Port Logic registers, DMA and iATU (internal Address
Translation Unit) registers.
num-viewport:
$ref: /schemas/types.yaml#/definitions/uint32
maximum: 256
description: |
number of view ports configured in hardware. If a platform
does not specify it, the driver autodetects it.
deprecated: true
interrupt-names:
minItems: 1
maxItems: 26
items:
oneOf:
- description:
Controller request to read or write virtual product data
from/to the VPD capability registers.
const: vpd
- description:
Link Equalization Request flag is set in the Link Status 2
register (applicable if the corresponding IRQ is enabled in
the Link Control 3 register).
const: l_eq
- description:
Indicates that the eDMA Tx/Rx transfer is complete or that an
error has occurred on the corresponding channel. eDMA can have
eight Tx (Write) and Rx (Read) eDMA channels thus supporting up
to 16 IRQ signals all together. Write eDMA channels shall go
first in the ordered row as per default edma_int[*] bus setup.
pattern: '^dma([0-9]|1[0-5])?$'
- description:
PCIe protocol correctable error or a Data Path protection
correctable error is detected by the automotive/safety
feature.
const: sft_ce
- description:
Indicates that the internal safety mechanism has detected an
uncorrectable error.
const: sft_ue
- description:
Application-specific IRQ raised depending on the vendor-specific
events basis.
const: app
- description:
DSP AXI MSI Interrupt detected. It gets de-asserted when there is
no more MSI interrupt pending. The interrupt is relevant to the
iMSI-RX - Integrated MSI Receiver (AXI bridge).
const: msi
- description:
Legacy A/B/C/D interrupt signal. Basically it's triggered by
receiving a Assert_INT{A,B,C,D}/Desassert_INT{A,B,C,D} message
from the downstream device.
pattern: "^int(a|b|c|d)$"
- description:
Error condition detected and a flag is set in the Root Error Status
register of the AER capability. It's asserted when the RC
internally generated an error or an error message is received by
the RC.
const: aer
- description:
PME message is received by the port. That means having the PME
status bit set in the Root Status register (the event is
supposed to be unmasked in the Root Control register).
const: pme
- description:
Hot-plug event is detected. That is a bit has been set in the
Slot Status register and the corresponding event is enabled in
the Slot Control register.
const: hp
- description:
Link Autonomous Bandwidth Status flag has been set in the Link
Status register (the event is supposed to be unmasked in the
Link Control register).
const: bw_au
- description:
Bandwidth Management Status flag has been set in the Link
Status register (the event is supposed to be unmasked in the
Link Control register).
const: bw_mg
- description:
Vendor-specific IRQ names. Consider using the generic names above
for new bindings.
oneOf:
- description: See native "app" IRQ for details
enum: [ intr ]
allOf:
- contains:
const: msi
additionalProperties: true
required:
- compatible
- reg
- reg-names
- compatible
examples:
- |
bus {
#address-cells = <1>;
#size-cells = <1>;
pcie@dfc00000 {
device_type = "pci";
compatible = "snps,dw-pcie";
reg = <0xdfc00000 0x0001000>, /* IP registers */
<0xd0000000 0x0002000>; /* Configuration space */
reg-names = "dbi", "config";
#address-cells = <3>;
#size-cells = <2>;
ranges = <0x81000000 0 0x00000000 0xde000000 0 0x00010000>,
<0x82000000 0 0xd0400000 0xd0400000 0 0x0d000000>;
interrupts = <25>, <24>;
#interrupt-cells = <1>;
num-lanes = <1>;
};
pcie@dfc00000 {
compatible = "snps,dw-pcie";
device_type = "pci";
reg = <0xdfc00000 0x0001000>, /* IP registers */
<0xd0000000 0x0002000>; /* Configuration space */
reg-names = "dbi", "config";
#address-cells = <3>;
#size-cells = <2>;
ranges = <0x81000000 0 0x00000000 0xde000000 0 0x00010000>,
<0x82000000 0 0xd0400000 0xd0400000 0 0x0d000000>;
bus-range = <0x0 0xff>;
interrupts = <25>, <24>;
interrupt-names = "msi", "hp";
#interrupt-cells = <1>;
reset-gpios = <&port0 0 1>;
phys = <&pcie_phy>;
phy-names = "pcie";
num-lanes = <1>;
max-link-speed = <3>;
};
Documentation/devicetree/bindings/pci/ti,j721e-pci-ep.yaml
+7
View File
@@ -58,6 +58,13 @@ properties:
dma-coherent:
description: Indicates that the PCIe IP block can ensure the coherency
interrupts:
maxItems: 1
interrupt-names:
items:
- const: link_state
required:
- compatible
- reg
Documentation/devicetree/bindings/pci/ti,j721e-pci-host.yaml
+22
View File
@@ -73,9 +73,31 @@ properties:
- const: 0xb00f
- items:
- const: 0xb010
- items:
- const: 0xb013
msi-map: true
interrupts:
maxItems: 1
interrupt-names:
items:
- const: link_state
interrupt-controller:
type: object
additionalProperties: false
properties:
interrupt-controller: true
'#interrupt-cells':
const: 1
interrupts:
maxItems: 1
required:
- compatible
- reg
Documentation/devicetree/bindings/pci/toshiba,visconti-pcie.yaml
+4 -3
View File
@@ -36,7 +36,7 @@ properties:
- const: mpu
interrupts:
maxItems: 1
maxItems: 2
clocks:
items:
@@ -94,8 +94,9 @@ examples:
#interrupt-cells = <1>;
ranges = <0x81000000 0 0x40000000 0 0x40000000 0 0x00010000>,
<0x82000000 0 0x50000000 0 0x50000000 0 0x20000000>;
interrupts = <GIC_SPI 215 IRQ_TYPE_LEVEL_HIGH>;
interrupt-names = "intr";
interrupts = <GIC_SPI 211 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 215 IRQ_TYPE_LEVEL_HIGH>;
interrupt-names = "msi", "intr";
interrupt-map-mask = <0 0 0 7>;
interrupt-map =
<0 0 0 1 &gic GIC_SPI 215 IRQ_TYPE_LEVEL_HIGH
Documentation/devicetree/bindings/rng/nuvoton,npcm-rng.yaml
+3 -1
View File
@@ -16,7 +16,9 @@ maintainers:
properties:
compatible:
const: nuvoton,npcm750-rng
enum:
- nuvoton,npcm750-rng
- nuvoton,npcm845-rng
reg:
maxItems: 1
Documentation/devicetree/bindings/soc/qcom/qcom,geni-se.yaml
+38 -6
View File
@@ -21,20 +21,19 @@ properties:
compatible:
enum:
- qcom,geni-se-qup
- qcom,geni-se-i2c-master-hub
reg:
description: QUP wrapper common register address and length.
maxItems: 1
clock-names:
items:
- const: m-ahb
- const: s-ahb
minItems: 1
maxItems: 2
clocks:
items:
- description: Master AHB Clock
- description: Slave AHB Clock
minItems: 1
maxItems: 2
"#address-cells":
const: 2
@@ -81,6 +80,39 @@ patternProperties:
description: GENI Serial Engine based UART Controller.
$ref: /schemas/serial/qcom,serial-geni-qcom.yaml#
allOf:
- if:
properties:
compatible:
contains:
const: qcom,geni-se-i2c-master-hub
then:
properties:
clock-names:
items:
- const: s-ahb
clocks:
items:
- description: Slave AHB Clock
iommus: false
patternProperties:
"spi@[0-9a-f]+$": false
"serial@[0-9a-f]+$": false
else:
properties:
clock-names:
items:
- const: m-ahb
- const: s-ahb
clocks:
items:
- description: Master AHB Clock
- description: Slave AHB Clock
additionalProperties: false
examples:
Documentation/devicetree/bindings/thermal/generic-adc-thermal.yaml
+84
View File
@@ -0,0 +1,84 @@
# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
%YAML 1.2
---
$id: http://devicetree.org/schemas/thermal/generic-adc-thermal.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: General Purpose Analog To Digital Converter (ADC) based thermal sensor
maintainers:
- Laxman Dewangan <ldewangan@nvidia.com>
description:
On some of platforms, thermal sensor like thermistors are connected to
one of ADC channel and sensor resistance is read via voltage across the
sensor resistor. The voltage read across the sensor is mapped to
temperature using voltage-temperature lookup table.
properties:
compatible:
const: generic-adc-thermal
'#thermal-sensor-cells':
const: 0
io-channels:
maxItems: 1
io-channel-names:
const: sensor-channel
temperature-lookup-table:
description: |
Lookup table to map the relation between ADC value and temperature.
When ADC is read, the value is looked up on the table to get the
equivalent temperature.
If not specified, driver assumes the ADC channel gives milliCelsius
directly.
$ref: /schemas/types.yaml#/definitions/int32-matrix
items:
items:
- description: Temperature in milliCelsius
- description: ADC read value
required:
- compatible
- '#thermal-sensor-cells'
- io-channels
- io-channel-names
additionalProperties: false
examples:
- |
#include <dt-bindings/thermal/thermal.h>
thermal-sensor {
compatible = "generic-adc-thermal";
#thermal-sensor-cells = <0>;
io-channels = <&ads1015 1>;
io-channel-names = "sensor-channel";
temperature-lookup-table = <
(-40000) 2578
(-39000) 2577
(-38000) 2576
(-37000) 2575
(-36000) 2574
(-35000) 2573
(-34000) 2572
(-33000) 2571
(-32000) 2569
(-31000) 2568
(-30000) 2567
/* skip */
118000 254
119000 247
120000 240
121000 233
122000 226
123000 220
124000 214
125000 208>;
};
...
Documentation/devicetree/bindings/thermal/imx8mm-thermal.yaml
+7
View File
@@ -32,6 +32,13 @@ properties:
clocks:
maxItems: 1
nvmem-cells:
maxItems: 1
description: Phandle to the calibration data provided by ocotp
nvmem-cell-names:
const: calib
"#thermal-sensor-cells":
description: |
Number of cells required to uniquely identify the thermal
Documentation/devicetree/bindings/thermal/mediatek-thermal.txt
+2
View File
@@ -13,6 +13,8 @@ Required properties:
- "mediatek,mt2701-thermal" : For MT2701 family of SoCs
- "mediatek,mt2712-thermal" : For MT2712 family of SoCs
- "mediatek,mt7622-thermal" : For MT7622 SoC
- "mediatek,mt7981-thermal", "mediatek,mt7986-thermal" : For MT7981 SoC
- "mediatek,mt7986-thermal" : For MT7986 SoC
- "mediatek,mt8183-thermal" : For MT8183 family of SoCs
- "mediatek,mt8516-thermal", "mediatek,mt2701-thermal : For MT8516 family of SoCs
- reg: Address range of the thermal controller
Documentation/devicetree/bindings/thermal/qcom-tsens.yaml
+74 -15
View File
@@ -58,8 +58,14 @@ properties:
- qcom,sm8150-tsens
- qcom,sm8250-tsens
- qcom,sm8350-tsens
- qcom,sm8450-tsens
- qcom,sm8550-tsens
- const: qcom,tsens-v2
- description: v2 of TSENS with combined interrupt
enum:
- qcom,ipq8074-tsens
reg:
items:
- description: TM registers
@@ -67,15 +73,11 @@ properties:
interrupts:
minItems: 1
items:
- description: Combined interrupt if upper or lower threshold crossed
- description: Interrupt if critical threshold crossed
maxItems: 2
interrupt-names:
minItems: 1
items:
- const: uplow
- const: critical
maxItems: 2
nvmem-cells:
minItems: 1
@@ -129,22 +131,64 @@ allOf:
then:
properties:
interrupts:
maxItems: 1
items:
- description: Combined interrupt if upper or lower threshold crossed
interrupt-names:
maxItems: 1
else:
properties:
interrupts:
minItems: 2
interrupt-names:
minItems: 2
items:
- const: uplow
- if:
properties:
compatible:
contains:
enum:
- qcom,msm8953-tsens
- qcom,msm8996-tsens
- qcom,msm8998-tsens
- qcom,sc7180-tsens
- qcom,sc7280-tsens
- qcom,sc8180x-tsens
- qcom,sc8280xp-tsens
- qcom,sdm630-tsens
- qcom,sdm845-tsens
- qcom,sm6350-tsens
- qcom,sm8150-tsens
- qcom,sm8250-tsens
- qcom,sm8350-tsens
- qcom,sm8450-tsens
- qcom,tsens-v2
then:
properties:
interrupts:
items:
- description: Combined interrupt if upper or lower threshold crossed
- description: Interrupt if critical threshold crossed
interrupt-names:
items:
- const: uplow
- const: critical
- if:
properties:
compatible:
contains:
enum:
- qcom,ipq8074-tsens
then:
properties:
interrupts:
items:
- description: Combined interrupt if upper, lower or critical thresholds crossed
interrupt-names:
items:
- const: combined
- if:
properties:
compatible:
contains:
enum:
- qcom,ipq8074-tsens
- qcom,tsens-v0_1
- qcom,tsens-v1
- qcom,tsens-v2
@@ -227,4 +271,19 @@ examples:
#qcom,sensors = <13>;
#thermal-sensor-cells = <1>;
};
- |
#include <dt-bindings/interrupt-controller/arm-gic.h>
// Example 4 (for any IPQ8074 based SoC-s):
tsens4: thermal-sensor@4a9000 {
compatible = "qcom,ipq8074-tsens";
reg = <0x4a9000 0x1000>,
<0x4a8000 0x1000>;
interrupts = <GIC_SPI 184 IRQ_TYPE_LEVEL_HIGH>;
interrupt-names = "combined";
#qcom,sensors = <16>;
#thermal-sensor-cells = <1>;
};
...
Documentation/devicetree/bindings/thermal/rzg2l-thermal.yaml
+1 -1
View File
@@ -17,7 +17,7 @@ properties:
compatible:
items:
- enum:
- renesas,r9a07g043-tsu # RZ/G2UL
- renesas,r9a07g043-tsu # RZ/G2UL and RZ/Five
- renesas,r9a07g044-tsu # RZ/G2{L,LC}
- renesas,r9a07g054-tsu # RZ/V2L
- const: renesas,rzg2l-tsu
Documentation/devicetree/bindings/thermal/thermal-generic-adc.txt
-95
View File
@@ -1,95 +0,0 @@
General Purpose Analog To Digital Converter (ADC) based thermal sensor.
On some of platforms, thermal sensor like thermistors are connected to
one of ADC channel and sensor resistance is read via voltage across the
sensor resistor. The voltage read across the sensor is mapped to
temperature using voltage-temperature lookup table.
Required properties:
===================
- compatible: Must be "generic-adc-thermal".
- #thermal-sensor-cells: Should be 1. See Documentation/devicetree/bindings/thermal/thermal-sensor.yaml for a description
of this property.
Optional properties:
===================
- temperature-lookup-table: Two dimensional array of Integer; lookup table
to map the relation between ADC value and
temperature. When ADC is read, the value is
looked up on the table to get the equivalent
temperature.
The first value of the each row of array is the
temperature in milliCelsius and second value of
the each row of array is the ADC read value.
If not specified, driver assumes the ADC channel
gives milliCelsius directly.
Example :
#include <dt-bindings/thermal/thermal.h>
i2c@7000c400 {
ads1015: ads1015@4a {
reg = <0x4a>;
compatible = "ads1015";
sampling-frequency = <3300>;
#io-channel-cells = <1>;
};
};
tboard_thermistor: thermal-sensor {
compatible = "generic-adc-thermal";
#thermal-sensor-cells = <0>;
io-channels = <&ads1015 1>;
io-channel-names = "sensor-channel";
temperature-lookup-table = < (-40000) 2578
(-39000) 2577
(-38000) 2576
(-37000) 2575
(-36000) 2574
(-35000) 2573
(-34000) 2572
(-33000) 2571
(-32000) 2569
(-31000) 2568
(-30000) 2567
::::::::::
118000 254
119000 247
120000 240
121000 233
122000 226
123000 220
124000 214
125000 208>;
};
dummy_cool_dev: dummy-cool-dev {
compatible = "dummy-cooling-dev";
#cooling-cells = <2>; /* min followed by max */
};
thermal-zones {
Tboard {
polling-delay = <15000>; /* milliseconds */
polling-delay-passive = <0>; /* milliseconds */
thermal-sensors = <&tboard_thermistor>;
trips {
therm_est_trip: therm_est_trip {
temperature = <40000>;
type = "active";
hysteresis = <1000>;
};
};
cooling-maps {
map0 {
trip = <&therm_est_trip>;
cooling-device = <&dummy_cool_dev THERMAL_NO_LIMIT THERMAL_NO_LIMIT>;
contribution = <100>;
};
};
};
};
Documentation/devicetree/bindings/thermal/ti,j72xx-thermal.yaml
+34 -1
View File
@@ -9,6 +9,19 @@ title: Texas Instruments J72XX VTM (DTS) binding
maintainers:
- Keerthy <j-keerthy@ti.com>
description: |
The TI K3 family of SoCs typically have a Voltage & Thermal
Management (VTM) device to control up to 8 temperature diode
sensors to measure silicon junction temperatures from different
hotspots of the chip as well as provide temperature, interrupt
and alerting information.
The following polynomial equation can then be used to convert
value returned by this device into a temperature in Celsius
Temp(C) = (-9.2627e-12) * x^4 + (6.0373e-08) * x^3 + \
(-1.7058e-04) * x^2 + (3.2512e-01) * x + (-4.9003e+01)
properties:
compatible:
enum:
@@ -19,7 +32,12 @@ properties:
items:
- description: VTM cfg1 register space
- description: VTM cfg2 register space
- description: VTM efuse register space
- description: |
A software trimming method must be applied to some Jacinto
devices to function properly. This eFuse region provides
the information needed for these SoCs to report
temperatures accurately.
minItems: 2
power-domains:
maxItems: 1
@@ -27,6 +45,21 @@ properties:
"#thermal-sensor-cells":
const: 1
allOf:
- if:
properties:
compatible:
contains:
const: ti,j721e-vtm
then:
properties:
reg:
minItems: 3
else:
properties:
reg:
maxItems: 2
required:
- compatible
- reg
Documentation/driver-api/gpio/legacy.rst
-5
View File
@@ -558,11 +558,6 @@ Platform Support
To force-enable this framework, a platform's Kconfig will "select" GPIOLIB,
else it is up to the user to configure support for GPIO.
It may also provide a custom value for ARCH_NR_GPIOS, so that it better
reflects the number of GPIOs in actual use on that platform, without
wasting static table space. (It should count both built-in/SoC GPIOs and
also ones on GPIO expanders.
If neither of these options are selected, the platform does not support
GPIOs through GPIO-lib and the code cannot be enabled by the user.
Documentation/features/vm/huge-vmap/arch-support.txt
+1 -1
View File
@@ -21,7 +21,7 @@
| openrisc: | TODO |
| parisc: | TODO |
| powerpc: | ok |
| riscv: | TODO |
| riscv: | ok |
| s390: | TODO |
| sh: | TODO |
| sparc: | TODO |
Documentation/filesystems/f2fs.rst
+12 -1
View File
@@ -25,10 +25,14 @@ a consistency checking tool (fsck.f2fs), and a debugging tool (dump.f2fs).
- git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git
For reporting bugs and sending patches, please use the following mailing list:
For sending patches, please use the following mailing list:
- linux-f2fs-devel@lists.sourceforge.net
For reporting bugs, please use the following f2fs bug tracker link:
- https://bugzilla.kernel.org/enter_bug.cgi?product=File%20System&component=f2fs
Background and Design issues
============================
@@ -154,6 +158,8 @@ nobarrier This option can be used if underlying storage guarantees
If this option is set, no cache_flush commands are issued
but f2fs still guarantees the write ordering of all the
data writes.
barrier If this option is set, cache_flush commands are allowed to be
issued.
fastboot This option is used when a system wants to reduce mount
time as much as possible, even though normal performance
can be sacrificed.
@@ -199,6 +205,7 @@ fault_type=%d Support configuring fault injection type, should be
FAULT_SLAB_ALLOC 0x000008000
FAULT_DQUOT_INIT 0x000010000
FAULT_LOCK_OP 0x000020000
FAULT_BLKADDR 0x000040000
=================== ===========
mode=%s Control block allocation mode which supports "adaptive"
and "lfs". In "lfs" mode, there should be no random
@@ -340,6 +347,10 @@ memory=%s Control memory mode. This supports "normal" and "low" modes.
Because of the nature of low memory devices, in this mode, f2fs
will try to save memory sometimes by sacrificing performance.
"normal" mode is the default mode and same as before.
age_extent_cache Enable an age extent cache based on rb-tree. It records
data block update frequency of the extent per inode, in
order to provide better temperature hints for data block
allocation.
======================== ============================================================
Debugfs Entries
Documentation/riscv/patch-acceptance.rst
+14 -8
View File
@@ -20,16 +20,22 @@ Submit Checklist Addendum
-------------------------
We'll only accept patches for new modules or extensions if the
specifications for those modules or extensions are listed as being
"Frozen" or "Ratified" by the RISC-V Foundation. (Developers may, of
course, maintain their own Linux kernel trees that contain code for
any draft extensions that they wish.)
unlikely to be incompatibly changed in the future. For
specifications from the RISC-V foundation this means "Frozen" or
"Ratified", for the UEFI forum specifications this means a published
ECR. (Developers may, of course, maintain their own Linux kernel trees
that contain code for any draft extensions that they wish.)
Additionally, the RISC-V specification allows implementors to create
Additionally, the RISC-V specification allows implementers to create
their own custom extensions. These custom extensions aren't required
to go through any review or ratification process by the RISC-V
Foundation. To avoid the maintenance complexity and potential
performance impact of adding kernel code for implementor-specific
RISC-V extensions, we'll only to accept patches for extensions that
have been officially frozen or ratified by the RISC-V Foundation.
(Implementors, may, of course, maintain their own Linux kernel trees
containing code for any custom extensions that they wish.)
RISC-V extensions, we'll only consider patches for extensions that either:
- Have been officially frozen or ratified by the RISC-V Foundation, or
- Have been implemented in hardware that is widely available, per standard
Linux practice.
(Implementers, may, of course, maintain their own Linux kernel trees containing
code for any custom extensions that they wish.)
Documentation/userspace-api/index.rst
+1
View File
@@ -25,6 +25,7 @@ place where this information is gathered.
ebpf/index
ioctl/index
iommu
iommufd
media/index
netlink/index
sysfs-platform_profile
Documentation/userspace-api/ioctl/ioctl-number.rst
+1
View File
@@ -105,6 +105,7 @@ Code Seq# Include File Comments
'8' all SNP8023 advanced NIC card
<mailto:mcr@solidum.com>
';' 64-7F linux/vfio.h
';' 80-FF linux/iommufd.h
'=' 00-3f uapi/linux/ptp_clock.h <mailto:richardcochran@gmail.com>
'@' 00-0F linux/radeonfb.h conflict!
'@' 00-0F drivers/video/aty/aty128fb.c conflict!
Documentation/userspace-api/iommufd.rst
+223
View File
@@ -0,0 +1,223 @@
.. SPDX-License-Identifier: GPL-2.0+
=======
IOMMUFD
=======
:Author: Jason Gunthorpe
:Author: Kevin Tian
Overview
========
IOMMUFD is the user API to control the IOMMU subsystem as it relates to managing
IO page tables from userspace using file descriptors. It intends to be general
and consumable by any driver that wants to expose DMA to userspace. These
drivers are eventually expected to deprecate any internal IOMMU logic
they may already/historically implement (e.g. vfio_iommu_type1.c).
At minimum iommufd provides universal support of managing I/O address spaces and
I/O page tables for all IOMMUs, with room in the design to add non-generic
features to cater to specific hardware functionality.
In this context the capital letter (IOMMUFD) refers to the subsystem while the
small letter (iommufd) refers to the file descriptors created via /dev/iommu for
use by userspace.
Key Concepts
============
User Visible Objects
--------------------
Following IOMMUFD objects are exposed to userspace:
- IOMMUFD_OBJ_IOAS, representing an I/O address space (IOAS), allowing map/unmap
of user space memory into ranges of I/O Virtual Address (IOVA).
The IOAS is a functional replacement for the VFIO container, and like the VFIO
container it copies an IOVA map to a list of iommu_domains held within it.
- IOMMUFD_OBJ_DEVICE, representing a device that is bound to iommufd by an
external driver.
- IOMMUFD_OBJ_HW_PAGETABLE, representing an actual hardware I/O page table
(i.e. a single struct iommu_domain) managed by the iommu driver.
The IOAS has a list of HW_PAGETABLES that share the same IOVA mapping and
it will synchronize its mapping with each member HW_PAGETABLE.
All user-visible objects are destroyed via the IOMMU_DESTROY uAPI.
The diagram below shows relationship between user-visible objects and kernel
datastructures (external to iommufd), with numbers referred to operations
creating the objects and links::
_________________________________________________________
| iommufd |
| [1] |
| _________________ |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | [3] [2] |
| | | ____________ __________ |
| | IOAS |<--| |<------| | |
| | | |HW_PAGETABLE| | DEVICE | |
| | | |____________| |__________| |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| |_________________| | | |
| | | | |
|_________|___________________|___________________|_______|
| | |
| _____v______ _______v_____
| PFN storage | | | |
|------------>|iommu_domain| |struct device|
|____________| |_____________|
1. IOMMUFD_OBJ_IOAS is created via the IOMMU_IOAS_ALLOC uAPI. An iommufd can
hold multiple IOAS objects. IOAS is the most generic object and does not
expose interfaces that are specific to single IOMMU drivers. All operations
on the IOAS must operate equally on each of the iommu_domains inside of it.
2. IOMMUFD_OBJ_DEVICE is created when an external driver calls the IOMMUFD kAPI
to bind a device to an iommufd. The driver is expected to implement a set of
ioctls to allow userspace to initiate the binding operation. Successful
completion of this operation establishes the desired DMA ownership over the
device. The driver must also set the driver_managed_dma flag and must not
touch the device until this operation succeeds.
3. IOMMUFD_OBJ_HW_PAGETABLE is created when an external driver calls the IOMMUFD
kAPI to attach a bound device to an IOAS. Similarly the external driver uAPI
allows userspace to initiate the attaching operation. If a compatible
pagetable already exists then it is reused for the attachment. Otherwise a
new pagetable object and iommu_domain is created. Successful completion of
this operation sets up the linkages among IOAS, device and iommu_domain. Once
this completes the device could do DMA.
Every iommu_domain inside the IOAS is also represented to userspace as a
HW_PAGETABLE object.
.. note::
Future IOMMUFD updates will provide an API to create and manipulate the
HW_PAGETABLE directly.
A device can only bind to an iommufd due to DMA ownership claim and attach to at
most one IOAS object (no support of PASID yet).
Kernel Datastructure
--------------------
User visible objects are backed by following datastructures:
- iommufd_ioas for IOMMUFD_OBJ_IOAS.
- iommufd_device for IOMMUFD_OBJ_DEVICE.
- iommufd_hw_pagetable for IOMMUFD_OBJ_HW_PAGETABLE.
Several terminologies when looking at these datastructures:
- Automatic domain - refers to an iommu domain created automatically when
attaching a device to an IOAS object. This is compatible to the semantics of
VFIO type1.
- Manual domain - refers to an iommu domain designated by the user as the
target pagetable to be attached to by a device. Though currently there are
no uAPIs to directly create such domain, the datastructure and algorithms
are ready for handling that use case.
- In-kernel user - refers to something like a VFIO mdev that is using the
IOMMUFD access interface to access the IOAS. This starts by creating an
iommufd_access object that is similar to the domain binding a physical device
would do. The access object will then allow converting IOVA ranges into struct
page * lists, or doing direct read/write to an IOVA.
iommufd_ioas serves as the metadata datastructure to manage how IOVA ranges are
mapped to memory pages, composed of:
- struct io_pagetable holding the IOVA map
- struct iopt_area's representing populated portions of IOVA
- struct iopt_pages representing the storage of PFNs
- struct iommu_domain representing the IO page table in the IOMMU
- struct iopt_pages_access representing in-kernel users of PFNs
- struct xarray pinned_pfns holding a list of pages pinned by in-kernel users
Each iopt_pages represents a logical linear array of full PFNs. The PFNs are
ultimately derived from userspace VAs via an mm_struct. Once they have been
pinned the PFNs are stored in IOPTEs of an iommu_domain or inside the pinned_pfns
xarray if they have been pinned through an iommufd_access.
PFN have to be copied between all combinations of storage locations, depending
on what domains are present and what kinds of in-kernel "software access" users
exist. The mechanism ensures that a page is pinned only once.
An io_pagetable is composed of iopt_areas pointing at iopt_pages, along with a
list of iommu_domains that mirror the IOVA to PFN map.
Multiple io_pagetable-s, through their iopt_area-s, can share a single
iopt_pages which avoids multi-pinning and double accounting of page
consumption.
iommufd_ioas is sharable between subsystems, e.g. VFIO and VDPA, as long as
devices managed by different subsystems are bound to a same iommufd.
IOMMUFD User API
================
.. kernel-doc:: include/uapi/linux/iommufd.h
IOMMUFD Kernel API
==================
The IOMMUFD kAPI is device-centric with group-related tricks managed behind the
scene. This allows the external drivers calling such kAPI to implement a simple
device-centric uAPI for connecting its device to an iommufd, instead of
explicitly imposing the group semantics in its uAPI as VFIO does.
.. kernel-doc:: drivers/iommu/iommufd/device.c
:export:
.. kernel-doc:: drivers/iommu/iommufd/main.c
:export:
VFIO and IOMMUFD
----------------
Connecting a VFIO device to iommufd can be done in two ways.
First is a VFIO compatible way by directly implementing the /dev/vfio/vfio
container IOCTLs by mapping them into io_pagetable operations. Doing so allows
the use of iommufd in legacy VFIO applications by symlinking /dev/vfio/vfio to
/dev/iommufd or extending VFIO to SET_CONTAINER using an iommufd instead of a
container fd.
The second approach directly extends VFIO to support a new set of device-centric
user API based on aforementioned IOMMUFD kernel API. It requires userspace
change but better matches the IOMMUFD API semantics and easier to support new
iommufd features when comparing it to the first approach.
Currently both approaches are still work-in-progress.
There are still a few gaps to be resolved to catch up with VFIO type1, as
documented in iommufd_vfio_check_extension().
Future TODOs
============
Currently IOMMUFD supports only kernel-managed I/O page table, similar to VFIO
type1. New features on the radar include:
- Binding iommu_domain's to PASID/SSID
- Userspace page tables, for ARM, x86 and S390
- Kernel bypass'd invalidation of user page tables
- Re-use of the KVM page table in the IOMMU
- Dirty page tracking in the IOMMU
- Runtime Increase/Decrease of IOPTE size
- PRI support with faults resolved in userspace
Documentation/virt/kvm/api.rst
+165 -109
View File
@@ -272,18 +272,6 @@ the VCPU file descriptor can be mmap-ed, including:
KVM_CAP_DIRTY_LOG_RING, see section 8.3.
4.6 KVM_SET_MEMORY_REGION
-------------------------
:Capability: basic
:Architectures: all
:Type: vm ioctl
:Parameters: struct kvm_memory_region (in)
:Returns: 0 on success, -1 on error
This ioctl is obsolete and has been removed.
4.7 KVM_CREATE_VCPU
-------------------
@@ -368,17 +356,6 @@ see the description of the capability.
Note that the Xen shared info page, if configured, shall always be assumed
to be dirty. KVM will not explicitly mark it such.
4.9 KVM_SET_MEMORY_ALIAS
------------------------
:Capability: basic
:Architectures: x86
:Type: vm ioctl
:Parameters: struct kvm_memory_alias (in)
:Returns: 0 (success), -1 (error)
This ioctl is obsolete and has been removed.
4.10 KVM_RUN
------------
@@ -1332,7 +1309,7 @@ yet and must be cleared on entry.
__u64 userspace_addr; /* start of the userspace allocated memory */
};
/* for kvm_memory_region::flags */
/* for kvm_userspace_memory_region::flags */
#define KVM_MEM_LOG_DIRTY_PAGES (1UL << 0)
#define KVM_MEM_READONLY (1UL << 1)
@@ -1377,10 +1354,6 @@ the memory region are automatically reflected into the guest. For example, an
mmap() that affects the region will be made visible immediately. Another
example is madvise(MADV_DROP).
It is recommended to use this API instead of the KVM_SET_MEMORY_REGION ioctl.
The KVM_SET_MEMORY_REGION does not allow fine grained control over memory
allocation and is deprecated.
4.36 KVM_SET_TSS_ADDR
---------------------
@@ -3293,6 +3266,7 @@ valid entries found.
----------------------
:Capability: KVM_CAP_DEVICE_CTRL
:Architectures: all
:Type: vm ioctl
:Parameters: struct kvm_create_device (in/out)
:Returns: 0 on success, -1 on error
@@ -3333,6 +3307,7 @@ number.
:Capability: KVM_CAP_DEVICE_CTRL, KVM_CAP_VM_ATTRIBUTES for vm device,
KVM_CAP_VCPU_ATTRIBUTES for vcpu device
KVM_CAP_SYS_ATTRIBUTES for system (/dev/kvm) device (no set)
:Architectures: x86, arm64, s390
:Type: device ioctl, vm ioctl, vcpu ioctl
:Parameters: struct kvm_device_attr
:Returns: 0 on success, -1 on error
@@ -4104,80 +4079,71 @@ flags values for ``struct kvm_msr_filter_range``:
``KVM_MSR_FILTER_READ``
Filter read accesses to MSRs using the given bitmap. A 0 in the bitmap
indicates that a read should immediately fail, while a 1 indicates that
a read for a particular MSR should be handled regardless of the default
indicates that read accesses should be denied, while a 1 indicates that
a read for a particular MSR should be allowed regardless of the default
filter action.
``KVM_MSR_FILTER_WRITE``
Filter write accesses to MSRs using the given bitmap. A 0 in the bitmap
indicates that a write should immediately fail, while a 1 indicates that
a write for a particular MSR should be handled regardless of the default
indicates that write accesses should be denied, while a 1 indicates that
a write for a particular MSR should be allowed regardless of the default
filter action.
``KVM_MSR_FILTER_READ | KVM_MSR_FILTER_WRITE``
Filter both read and write accesses to MSRs using the given bitmap. A 0
in the bitmap indicates that both reads and writes should immediately fail,
while a 1 indicates that reads and writes for a particular MSR are not
filtered by this range.
flags values for ``struct kvm_msr_filter``:
``KVM_MSR_FILTER_DEFAULT_ALLOW``
If no filter range matches an MSR index that is getting accessed, KVM will
fall back to allowing access to the MSR.
allow accesses to all MSRs by default.
``KVM_MSR_FILTER_DEFAULT_DENY``
If no filter range matches an MSR index that is getting accessed, KVM will
fall back to rejecting access to the MSR. In this mode, all MSRs that should
be processed by KVM need to explicitly be marked as allowed in the bitmaps.
deny accesses to all MSRs by default.
This ioctl allows user space to define up to 16 bitmaps of MSR ranges to
specify whether a certain MSR access should be explicitly filtered for or not.
This ioctl allows userspace to define up to 16 bitmaps of MSR ranges to deny
guest MSR accesses that would normally be allowed by KVM. If an MSR is not
covered by a specific range, the "default" filtering behavior applies. Each
bitmap range covers MSRs from [base .. base+nmsrs).
If this ioctl has never been invoked, MSR accesses are not guarded and the
default KVM in-kernel emulation behavior is fully preserved.
If an MSR access is denied by userspace, the resulting KVM behavior depends on
whether or not KVM_CAP_X86_USER_SPACE_MSR's KVM_MSR_EXIT_REASON_FILTER is
enabled. If KVM_MSR_EXIT_REASON_FILTER is enabled, KVM will exit to userspace
on denied accesses, i.e. userspace effectively intercepts the MSR access. If
KVM_MSR_EXIT_REASON_FILTER is not enabled, KVM will inject a #GP into the guest
on denied accesses.
If an MSR access is allowed by userspace, KVM will emulate and/or virtualize
the access in accordance with the vCPU model. Note, KVM may still ultimately
inject a #GP if an access is allowed by userspace, e.g. if KVM doesn't support
the MSR, or to follow architectural behavior for the MSR.
By default, KVM operates in KVM_MSR_FILTER_DEFAULT_ALLOW mode with no MSR range
filters.
Calling this ioctl with an empty set of ranges (all nmsrs == 0) disables MSR
filtering. In that mode, ``KVM_MSR_FILTER_DEFAULT_DENY`` is invalid and causes
an error.
As soon as the filtering is in place, every MSR access is processed through
the filtering except for accesses to the x2APIC MSRs (from 0x800 to 0x8ff);
x2APIC MSRs are always allowed, independent of the ``default_allow`` setting,
and their behavior depends on the ``X2APIC_ENABLE`` bit of the APIC base
register.
.. warning::
MSR accesses coming from nested vmentry/vmexit are not filtered.
MSR accesses as part of nested VM-Enter/VM-Exit are not filtered.
This includes both writes to individual VMCS fields and reads/writes
through the MSR lists pointed to by the VMCS.
If a bit is within one of the defined ranges, read and write accesses are
guarded by the bitmap's value for the MSR index if the kind of access
is included in the ``struct kvm_msr_filter_range`` flags. If no range
cover this particular access, the behavior is determined by the flags
field in the kvm_msr_filter struct: ``KVM_MSR_FILTER_DEFAULT_ALLOW``
and ``KVM_MSR_FILTER_DEFAULT_DENY``.
Each bitmap range specifies a range of MSRs to potentially allow access on.
The range goes from MSR index [base .. base+nmsrs]. The flags field
indicates whether reads, writes or both reads and writes are filtered
by setting a 1 bit in the bitmap for the corresponding MSR index.
If an MSR access is not permitted through the filtering, it generates a
#GP inside the guest. When combined with KVM_CAP_X86_USER_SPACE_MSR, that
allows user space to deflect and potentially handle various MSR accesses
into user space.
x2APIC MSR accesses cannot be filtered (KVM silently ignores filters that
cover any x2APIC MSRs).
Note, invoking this ioctl while a vCPU is running is inherently racy. However,
KVM does guarantee that vCPUs will see either the previous filter or the new
filter, e.g. MSRs with identical settings in both the old and new filter will
have deterministic behavior.
Similarly, if userspace wishes to intercept on denied accesses,
KVM_MSR_EXIT_REASON_FILTER must be enabled before activating any filters, and
left enabled until after all filters are deactivated. Failure to do so may
result in KVM injecting a #GP instead of exiting to userspace.
4.98 KVM_CREATE_SPAPR_TCE_64
----------------------------
@@ -5163,10 +5129,13 @@ KVM_PV_ENABLE
===== =============================
KVM_PV_DISABLE
Deregister the VM from the Ultravisor and reclaim the memory that
had been donated to the Ultravisor, making it usable by the kernel
again. All registered VCPUs are converted back to non-protected
ones.
Deregister the VM from the Ultravisor and reclaim the memory that had
been donated to the Ultravisor, making it usable by the kernel again.
All registered VCPUs are converted back to non-protected ones. If a
previous protected VM had been prepared for asynchonous teardown with
KVM_PV_ASYNC_CLEANUP_PREPARE and not subsequently torn down with
KVM_PV_ASYNC_CLEANUP_PERFORM, it will be torn down in this call
together with the current protected VM.
KVM_PV_VM_SET_SEC_PARMS
Pass the image header from VM memory to the Ultravisor in
@@ -5289,6 +5258,36 @@ KVM_PV_DUMP
authentication tag all of which are needed to decrypt the dump at a
later time.
KVM_PV_ASYNC_CLEANUP_PREPARE
:Capability: KVM_CAP_S390_PROTECTED_ASYNC_DISABLE
Prepare the current protected VM for asynchronous teardown. Most
resources used by the current protected VM will be set aside for a
subsequent asynchronous teardown. The current protected VM will then
resume execution immediately as non-protected. There can be at most
one protected VM prepared for asynchronous teardown at any time. If
a protected VM had already been prepared for teardown without
subsequently calling KVM_PV_ASYNC_CLEANUP_PERFORM, this call will
fail. In that case, the userspace process should issue a normal
KVM_PV_DISABLE. The resources set aside with this call will need to
be cleaned up with a subsequent call to KVM_PV_ASYNC_CLEANUP_PERFORM
or KVM_PV_DISABLE, otherwise they will be cleaned up when KVM
terminates. KVM_PV_ASYNC_CLEANUP_PREPARE can be called again as soon
as cleanup starts, i.e. before KVM_PV_ASYNC_CLEANUP_PERFORM finishes.
KVM_PV_ASYNC_CLEANUP_PERFORM
:Capability: KVM_CAP_S390_PROTECTED_ASYNC_DISABLE
Tear down the protected VM previously prepared for teardown with
KVM_PV_ASYNC_CLEANUP_PREPARE. The resources that had been set aside
will be freed during the execution of this command. This PV command
should ideally be issued by userspace from a separate thread. If a
fatal signal is received (or the process terminates naturally), the
command will terminate immediately without completing, and the normal
KVM shutdown procedure will take care of cleaning up all remaining
protected VMs, including the ones whose teardown was interrupted by
process termination.
4.126 KVM_XEN_HVM_SET_ATTR
--------------------------
@@ -5306,6 +5305,7 @@ KVM_PV_DUMP
union {
__u8 long_mode;
__u8 vector;
__u8 runstate_update_flag;
struct {
__u64 gfn;
} shared_info;
@@ -5383,6 +5383,14 @@ KVM_XEN_ATTR_TYPE_XEN_VERSION
event channel delivery, so responding within the kernel without
exiting to userspace is beneficial.
KVM_XEN_ATTR_TYPE_RUNSTATE_UPDATE_FLAG
This attribute is available when the KVM_CAP_XEN_HVM ioctl indicates
support for KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG. It enables the
XEN_RUNSTATE_UPDATE flag which allows guest vCPUs to safely read
other vCPUs' vcpu_runstate_info. Xen guests enable this feature via
the VM_ASST_TYPE_runstate_update_flag of the HYPERVISOR_vm_assist
hypercall.
4.127 KVM_XEN_HVM_GET_ATTR
--------------------------
@@ -6440,31 +6448,35 @@ if it decides to decode and emulate the instruction.
Used on x86 systems. When the VM capability KVM_CAP_X86_USER_SPACE_MSR is
enabled, MSR accesses to registers that would invoke a #GP by KVM kernel code
will instead trigger a KVM_EXIT_X86_RDMSR exit for reads and KVM_EXIT_X86_WRMSR
may instead trigger a KVM_EXIT_X86_RDMSR exit for reads and KVM_EXIT_X86_WRMSR
exit for writes.
The "reason" field specifies why the MSR trap occurred. User space will only
receive MSR exit traps when a particular reason was requested during through
The "reason" field specifies why the MSR interception occurred. Userspace will
only receive MSR exits when a particular reason was requested during through
ENABLE_CAP. Currently valid exit reasons are:
KVM_MSR_EXIT_REASON_UNKNOWN - access to MSR that is unknown to KVM
KVM_MSR_EXIT_REASON_INVAL - access to invalid MSRs or reserved bits
KVM_MSR_EXIT_REASON_FILTER - access blocked by KVM_X86_SET_MSR_FILTER
============================ ========================================
KVM_MSR_EXIT_REASON_UNKNOWN access to MSR that is unknown to KVM
KVM_MSR_EXIT_REASON_INVAL access to invalid MSRs or reserved bits
KVM_MSR_EXIT_REASON_FILTER access blocked by KVM_X86_SET_MSR_FILTER
============================ ========================================
For KVM_EXIT_X86_RDMSR, the "index" field tells user space which MSR the guest
wants to read. To respond to this request with a successful read, user space
For KVM_EXIT_X86_RDMSR, the "index" field tells userspace which MSR the guest
wants to read. To respond to this request with a successful read, userspace
writes the respective data into the "data" field and must continue guest
execution to ensure the read data is transferred into guest register state.
If the RDMSR request was unsuccessful, user space indicates that with a "1" in
If the RDMSR request was unsuccessful, userspace indicates that with a "1" in
the "error" field. This will inject a #GP into the guest when the VCPU is
executed again.
For KVM_EXIT_X86_WRMSR, the "index" field tells user space which MSR the guest
wants to write. Once finished processing the event, user space must continue
vCPU execution. If the MSR write was unsuccessful, user space also sets the
For KVM_EXIT_X86_WRMSR, the "index" field tells userspace which MSR the guest
wants to write. Once finished processing the event, userspace must continue
vCPU execution. If the MSR write was unsuccessful, userspace also sets the
"error" field to "1".
See KVM_X86_SET_MSR_FILTER for details on the interaction with MSR filtering.
::
@@ -7229,19 +7241,29 @@ polling.
:Parameters: args[0] contains the mask of KVM_MSR_EXIT_REASON_* events to report
:Returns: 0 on success; -1 on error
This capability enables trapping of #GP invoking RDMSR and WRMSR instructions
into user space.
This capability allows userspace to intercept RDMSR and WRMSR instructions if
access to an MSR is denied. By default, KVM injects #GP on denied accesses.
When a guest requests to read or write an MSR, KVM may not implement all MSRs
that are relevant to a respective system. It also does not differentiate by
CPU type.
To allow more fine grained control over MSR handling, user space may enable
To allow more fine grained control over MSR handling, userspace may enable
this capability. With it enabled, MSR accesses that match the mask specified in
args[0] and trigger a #GP event inside the guest by KVM will instead trigger
KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR exit notifications which user space
can then handle to implement model specific MSR handling and/or user notifications
to inform a user that an MSR was not handled.
args[0] and would trigger a #GP inside the guest will instead trigger
KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR exit notifications. Userspace
can then implement model specific MSR handling and/or user notifications
to inform a user that an MSR was not emulated/virtualized by KVM.
The valid mask flags are:
============================ ===============================================
KVM_MSR_EXIT_REASON_UNKNOWN intercept accesses to unknown (to KVM) MSRs
KVM_MSR_EXIT_REASON_INVAL intercept accesses that are architecturally
invalid according to the vCPU model and/or mode
KVM_MSR_EXIT_REASON_FILTER intercept accesses that are denied by userspace
via KVM_X86_SET_MSR_FILTER
============================ ===============================================
7.22 KVM_CAP_X86_BUS_LOCK_EXIT
-------------------------------
@@ -7384,8 +7406,9 @@ hibernation of the host; however the VMM needs to manually save/restore the
tags as appropriate if the VM is migrated.
When this capability is enabled all memory in memslots must be mapped as
not-shareable (no MAP_SHARED), attempts to create a memslot with a
MAP_SHARED mmap will result in an -EINVAL return.
``MAP_ANONYMOUS`` or with a RAM-based file mapping (``tmpfs``, ``memfd``),
attempts to create a memslot with an invalid mmap will result in an
-EINVAL return.
When enabled the VMM may make use of the ``KVM_ARM_MTE_COPY_TAGS`` ioctl to
perform a bulk copy of tags to/from the guest.
@@ -7901,7 +7924,7 @@ KVM_EXIT_X86_WRMSR exit notifications.
This capability indicates that KVM supports that accesses to user defined MSRs
may be rejected. With this capability exposed, KVM exports new VM ioctl
KVM_X86_SET_MSR_FILTER which user space can call to specify bitmaps of MSR
ranges that KVM should reject access to.
ranges that KVM should deny access to.
In combination with KVM_CAP_X86_USER_SPACE_MSR, this allows user space to
trap and emulate MSRs that are outside of the scope of KVM as well as
@@ -7920,7 +7943,7 @@ regardless of what has actually been exposed through the CPUID leaf.
8.29 KVM_CAP_DIRTY_LOG_RING/KVM_CAP_DIRTY_LOG_RING_ACQ_REL
----------------------------------------------------------
:Architectures: x86
:Architectures: x86, arm64
:Parameters: args[0] - size of the dirty log ring
KVM is capable of tracking dirty memory using ring buffers that are
@@ -8002,13 +8025,6 @@ flushing is done by the KVM_GET_DIRTY_LOG ioctl). To achieve that, one
needs to kick the vcpu out of KVM_RUN using a signal. The resulting
vmexit ensures that all dirty GFNs are flushed to the dirty rings.
NOTE: the capability KVM_CAP_DIRTY_LOG_RING and the corresponding
ioctl KVM_RESET_DIRTY_RINGS are mutual exclusive to the existing ioctls
KVM_GET_DIRTY_LOG and KVM_CLEAR_DIRTY_LOG. After enabling
KVM_CAP_DIRTY_LOG_RING with an acceptable dirty ring size, the virtual
machine will switch to ring-buffer dirty page tracking and further
KVM_GET_DIRTY_LOG or KVM_CLEAR_DIRTY_LOG ioctls will fail.
NOTE: KVM_CAP_DIRTY_LOG_RING_ACQ_REL is the only capability that
should be exposed by weakly ordered architecture, in order to indicate
the additional memory ordering requirements imposed on userspace when
@@ -8017,6 +8033,33 @@ Architecture with TSO-like ordering (such as x86) are allowed to
expose both KVM_CAP_DIRTY_LOG_RING and KVM_CAP_DIRTY_LOG_RING_ACQ_REL
to userspace.
After enabling the dirty rings, the userspace needs to detect the
capability of KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP to see whether the
ring structures can be backed by per-slot bitmaps. With this capability
advertised, it means the architecture can dirty guest pages without
vcpu/ring context, so that some of the dirty information will still be
maintained in the bitmap structure. KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP
can't be enabled if the capability of KVM_CAP_DIRTY_LOG_RING_ACQ_REL
hasn't been enabled, or any memslot has been existing.
Note that the bitmap here is only a backup of the ring structure. The
use of the ring and bitmap combination is only beneficial if there is
only a very small amount of memory that is dirtied out of vcpu/ring
context. Otherwise, the stand-alone per-slot bitmap mechanism needs to
be considered.
To collect dirty bits in the backup bitmap, userspace can use the same
KVM_GET_DIRTY_LOG ioctl. KVM_CLEAR_DIRTY_LOG isn't needed as long as all
the generation of the dirty bits is done in a single pass. Collecting
the dirty bitmap should be the very last thing that the VMM does before
considering the state as complete. VMM needs to ensure that the dirty
state is final and avoid missing dirty pages from another ioctl ordered
after the bitmap collection.
NOTE: One example of using the backup bitmap is saving arm64 vgic/its
tables through KVM_DEV_ARM_{VGIC_GRP_CTRL, ITS_SAVE_TABLES} command on
KVM device "kvm-arm-vgic-its" when dirty ring is enabled.
8.30 KVM_CAP_XEN_HVM
--------------------
@@ -8025,12 +8068,13 @@ to userspace.
This capability indicates the features that Xen supports for hosting Xen
PVHVM guests. Valid flags are::
#define KVM_XEN_HVM_CONFIG_HYPERCALL_MSR (1 << 0)
#define KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL (1 << 1)
#define KVM_XEN_HVM_CONFIG_SHARED_INFO (1 << 2)
#define KVM_XEN_HVM_CONFIG_RUNSTATE (1 << 3)
#define KVM_XEN_HVM_CONFIG_EVTCHN_2LEVEL (1 << 4)
#define KVM_XEN_HVM_CONFIG_EVTCHN_SEND (1 << 5)
#define KVM_XEN_HVM_CONFIG_HYPERCALL_MSR (1 << 0)
#define KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL (1 << 1)
#define KVM_XEN_HVM_CONFIG_SHARED_INFO (1 << 2)
#define KVM_XEN_HVM_CONFIG_RUNSTATE (1 << 3)
#define KVM_XEN_HVM_CONFIG_EVTCHN_2LEVEL (1 << 4)
#define KVM_XEN_HVM_CONFIG_EVTCHN_SEND (1 << 5)
#define KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG (1 << 6)
The KVM_XEN_HVM_CONFIG_HYPERCALL_MSR flag indicates that the KVM_XEN_HVM_CONFIG
ioctl is available, for the guest to set its hypercall page.
@@ -8062,6 +8106,18 @@ KVM_XEN_VCPU_ATTR_TYPE_VCPU_ID/TIMER/UPCALL_VECTOR vCPU attributes.
related to event channel delivery, timers, and the XENVER_version
interception.
The KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG flag indicates that KVM supports
the KVM_XEN_ATTR_TYPE_RUNSTATE_UPDATE_FLAG attribute in the KVM_XEN_SET_ATTR
and KVM_XEN_GET_ATTR ioctls. This controls whether KVM will set the
XEN_RUNSTATE_UPDATE flag in guest memory mapped vcpu_runstate_info during
updates of the runstate information. Note that versions of KVM which support
the RUNSTATE feature above, but not thie RUNSTATE_UPDATE_FLAG feature, will
always set the XEN_RUNSTATE_UPDATE flag when updating the guest structure,
which is perhaps counterintuitive. When this flag is advertised, KVM will
behave more correctly, not using the XEN_RUNSTATE_UPDATE flag until/unless
specifically enabled (by the guest making the hypercall, causing the VMM
to enable the KVM_XEN_ATTR_TYPE_RUNSTATE_UPDATE_FLAG attribute).
8.31 KVM_CAP_PPC_MULTITCE
-------------------------
Documentation/virt/kvm/arm/pvtime.rst
+8 -6
View File
@@ -23,21 +23,23 @@ the PV_TIME_FEATURES hypercall should be probed using the SMCCC 1.1
ARCH_FEATURES mechanism before calling it.
PV_TIME_FEATURES
============= ======== ==========
============= ======== =================================================
Function ID: (uint32) 0xC5000020
PV_call_id: (uint32) The function to query for support.
Currently only PV_TIME_ST is supported.
Return value: (int64) NOT_SUPPORTED (-1) or SUCCESS (0) if the relevant
PV-time feature is supported by the hypervisor.
============= ======== ==========
============= ======== =================================================
PV_TIME_ST
============= ======== ==========
============= ======== ==============================================
Function ID: (uint32) 0xC5000021
Return value: (int64) IPA of the stolen time data structure for this
VCPU. On failure:
NOT_SUPPORTED (-1)
============= ======== ==========
============= ======== ==============================================
The IPA returned by PV_TIME_ST should be mapped by the guest as normal memory
with inner and outer write back caching attributes, in the inner shareable
@@ -76,5 +78,5 @@ It is advisable that one or more 64k pages are set aside for the purpose of
these structures and not used for other purposes, this enables the guest to map
the region using 64k pages and avoids conflicting attributes with other memory.
For the user space interface see Documentation/virt/kvm/devices/vcpu.rst
section "3. GROUP: KVM_ARM_VCPU_PVTIME_CTRL".
For the user space interface see
:ref:`Documentation/virt/kvm/devices/vcpu.rst <kvm_arm_vcpu_pvtime_ctrl>`.
Documentation/virt/kvm/devices/arm-vgic-its.rst
+4 -1
View File
@@ -52,7 +52,10 @@ KVM_DEV_ARM_VGIC_GRP_CTRL
KVM_DEV_ARM_ITS_SAVE_TABLES
save the ITS table data into guest RAM, at the location provisioned
by the guest in corresponding registers/table entries.
by the guest in corresponding registers/table entries. Should userspace
require a form of dirty tracking to identify which pages are modified
by the saving process, it should use a bitmap even if using another
mechanism to track the memory dirtied by the vCPUs.
The layout of the tables in guest memory defines an ABI. The entries
are laid out in little endian format as described in the last paragraph.
Documentation/virt/kvm/devices/vcpu.rst
+2
View File
@@ -171,6 +171,8 @@ configured values on other VCPUs. Userspace should configure the interrupt
numbers on at least one VCPU after creating all VCPUs and before running any
VCPUs.
.. _kvm_arm_vcpu_pvtime_ctrl:
3. GROUP: KVM_ARM_VCPU_PVTIME_CTRL
==================================
MAINTAINERS
+55 -4
View File
@@ -312,6 +312,13 @@ L: linux-iio@vger.kernel.org
S: Maintained
F: drivers/counter/104-quad-8.c
ACCES IDIO-16 GPIO LIBRARY
M: William Breathitt Gray <william.gray@linaro.org>
L: linux-gpio@vger.kernel.org
S: Maintained
F: drivers/gpio/gpio-idio-16.c
F: drivers/gpio/gpio-idio-16.h
ACCES PCI-IDIO-16 GPIO DRIVER
M: William Breathitt Gray <william.gray@linaro.org>
L: linux-gpio@vger.kernel.org
@@ -7889,6 +7896,7 @@ M: Chao Yu <chao@kernel.org>
L: linux-f2fs-devel@lists.sourceforge.net
S: Maintained
W: https://f2fs.wiki.kernel.org/
B: https://bugzilla.kernel.org/enter_bug.cgi?product=File%20System&component=f2fs
T: git git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git
F: Documentation/ABI/testing/sysfs-fs-f2fs
F: Documentation/filesystems/f2fs.rst
@@ -8105,6 +8113,8 @@ S: Supported
T: git git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git for-next/hardening
F: include/linux/fortify-string.h
F: lib/fortify_kunit.c
F: lib/memcpy_kunit.c
F: lib/strscpy_kunit.c
F: lib/test_fortify/*
F: scripts/test_fortify.sh
K: \b__NO_FORTIFY\b
@@ -9263,6 +9273,7 @@ HISILICON GPIO DRIVER
M: Jay Fang <f.fangjian@huawei.com>
L: linux-gpio@vger.kernel.org
S: Maintained
F: Documentation/devicetree/bindings/gpio/hisilicon,ascend910-gpio.yaml
F: drivers/gpio/gpio-hisi.c
HISILICON HIGH PERFORMANCE RSA ENGINE DRIVER (HPRE)
@@ -9279,6 +9290,7 @@ M: Yicong Yang <yangyicong@hisilicon.com>
L: linux-i2c@vger.kernel.org
S: Maintained
W: https://www.hisilicon.com
F: Documentation/devicetree/bindings/i2c/hisilicon,ascend910-i2c.yaml
F: drivers/i2c/busses/i2c-hisi.c
HISILICON LPC BUS DRIVER
@@ -9770,8 +9782,7 @@ F: Documentation/devicetree/bindings/i3c/cdns,i3c-master.yaml
F: drivers/i3c/master/i3c-master-cdns.c
I3C DRIVER FOR SYNOPSYS DESIGNWARE
M: Vitor Soares <vitor.soares@synopsys.com>
S: Maintained
S: Orphan
F: Documentation/devicetree/bindings/i3c/snps,dw-i3c-master.yaml
F: drivers/i3c/master/dw*
@@ -10800,6 +10811,18 @@ F: drivers/iommu/dma-iommu.h
F: drivers/iommu/iova.c
F: include/linux/iova.h
IOMMUFD
M: Jason Gunthorpe <jgg@nvidia.com>
M: Kevin Tian <kevin.tian@intel.com>
L: iommu@lists.linux.dev
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git
F: Documentation/userspace-api/iommufd.rst
F: drivers/iommu/iommufd/
F: include/linux/iommufd.h
F: include/uapi/linux/iommufd.h
F: tools/testing/selftests/iommu/
IOMMU SUBSYSTEM
M: Joerg Roedel <joro@8bytes.org>
M: Will Deacon <will@kernel.org>
@@ -11204,6 +11227,8 @@ M: Kees Cook <keescook@chromium.org>
L: linux-hardening@vger.kernel.org
S: Supported
T: git git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git for-next/hardening
F: Documentation/ABI/testing/sysfs-kernel-oops_count
F: Documentation/ABI/testing/sysfs-kernel-warn_count
F: include/linux/overflow.h
F: include/linux/randomize_kstack.h
F: mm/usercopy.c
@@ -11421,6 +11446,16 @@ F: arch/x86/kvm/svm/hyperv.*
F: arch/x86/kvm/svm/svm_onhyperv.*
F: arch/x86/kvm/vmx/evmcs.*
KVM X86 Xen (KVM/Xen)
M: David Woodhouse <dwmw2@infradead.org>
M: Paul Durrant <paul@xen.org>
M: Sean Christopherson <seanjc@google.com>
M: Paolo Bonzini <pbonzini@redhat.com>
L: kvm@vger.kernel.org
S: Supported
T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
F: arch/x86/kvm/xen.*
KERNFS
M: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
M: Tejun Heo <tj@kernel.org>
@@ -13801,6 +13836,15 @@ F: drivers/scsi/smartpqi/smartpqi*.[ch]
F: include/linux/cciss*.h
F: include/uapi/linux/cciss*.h
MICROSOFT MANA RDMA DRIVER
M: Long Li <longli@microsoft.com>
M: Ajay Sharma <sharmaajay@microsoft.com>
L: linux-rdma@vger.kernel.org
S: Supported
F: drivers/infiniband/hw/mana/
F: include/net/mana
F: include/uapi/rdma/mana-abi.h
MICROSOFT SURFACE AGGREGATOR TABLET-MODE SWITCH
M: Maximilian Luz <luzmaximilian@gmail.com>
L: platform-driver-x86@vger.kernel.org
@@ -17924,6 +17968,13 @@ F: Documentation/ABI/*/sysfs-driver-hid-roccat*
F: drivers/hid/hid-roccat*
F: include/linux/hid-roccat*
ROCKCHIP CRYPTO DRIVERS
M: Corentin Labbe <clabbe@baylibre.com>
L: linux-crypto@vger.kernel.org
S: Maintained
F: Documentation/devicetree/bindings/crypto/rockchip,rk3288-crypto.yaml
F: drivers/crypto/rockchip/
ROCKCHIP I2S TDM DRIVER
M: Nicolas Frattaroli <frattaroli.nicolas@gmail.com>
L: linux-rockchip@lists.infradead.org
@@ -19037,7 +19088,7 @@ M: Jason A. Donenfeld <Jason@zx2c4.com>
S: Maintained
F: include/linux/siphash.h
F: lib/siphash.c
F: lib/test_siphash.c
F: lib/siphash_kunit.c
SIS 190 ETHERNET DRIVER
M: Francois Romieu <romieu@fr.zoreil.com>
@@ -21738,7 +21789,7 @@ M: Alex Williamson <alex.williamson@redhat.com>
R: Cornelia Huck <cohuck@redhat.com>
L: kvm@vger.kernel.org
S: Maintained
T: git git://github.com/awilliam/linux-vfio.git
T: git https://github.com/awilliam/linux-vfio.git
F: Documentation/ABI/testing/sysfs-devices-vfio-dev
F: Documentation/driver-api/vfio.rst
F: drivers/vfio/
Makefile
+3 -3
View File
@@ -1032,8 +1032,8 @@ KBUILD_CFLAGS += $(CC_FLAGS_CFI)
export CC_FLAGS_CFI
endif
ifdef CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B
KBUILD_CFLAGS += -falign-functions=64
ifneq ($(CONFIG_FUNCTION_ALIGNMENT),0)
KBUILD_CFLAGS += -falign-functions=$(CONFIG_FUNCTION_ALIGNMENT)
endif
# arch Makefile may override CC so keep this after arch Makefile is included
@@ -1146,7 +1146,7 @@ endif
# We never want expected sections to be placed heuristically by the
# linker. All sections should be explicitly named in the linker script.
ifdef CONFIG_LD_ORPHAN_WARN
LDFLAGS_vmlinux += --orphan-handling=warn
LDFLAGS_vmlinux += --orphan-handling=$(CONFIG_LD_ORPHAN_WARN_LEVEL)
endif
# Align the bit size of userspace programs with the kernel
arch/Kconfig
+24
View File
@@ -1438,4 +1438,28 @@ source "kernel/gcov/Kconfig"
source "scripts/gcc-plugins/Kconfig"
config FUNCTION_ALIGNMENT_4B
bool
config FUNCTION_ALIGNMENT_8B
bool
config FUNCTION_ALIGNMENT_16B
bool
config FUNCTION_ALIGNMENT_32B
bool
config FUNCTION_ALIGNMENT_64B
bool
config FUNCTION_ALIGNMENT
int
default 64 if FUNCTION_ALIGNMENT_64B
default 32 if FUNCTION_ALIGNMENT_32B
default 16 if FUNCTION_ALIGNMENT_16B
default 8 if FUNCTION_ALIGNMENT_8B
default 4 if FUNCTION_ALIGNMENT_4B
default 0
endmenu
arch/arm/Kconfig
-21
View File
@@ -1158,27 +1158,6 @@ config ARM_PSCI
0022A ("Power State Coordination Interface System Software on
ARM processors").
# The GPIO number here must be sorted by descending number. In case of
# a multiplatform kernel, we just want the highest value required by the
# selected platforms.
config ARCH_NR_GPIO
int
default 2048 if ARCH_INTEL_SOCFPGA
default 1024 if ARCH_BRCMSTB || ARCH_RENESAS || ARCH_TEGRA || \
ARCH_ZYNQ || ARCH_ASPEED
default 512 if ARCH_EXYNOS || ARCH_KEYSTONE || SOC_OMAP5 || \
SOC_DRA7XX || ARCH_S3C24XX || ARCH_S3C64XX || ARCH_S5PV210
default 416 if ARCH_SUNXI
default 392 if ARCH_U8500
default 352 if ARCH_VT8500
default 288 if ARCH_ROCKCHIP
default 264 if MACH_H4700
default 0
help
Maximum number of GPIOs in the system.
If unsure, leave the default value.
config HZ_FIXED
int
default 128 if SOC_AT91RM9200
arch/arm/boot/compressed/Makefile
+1 -1
View File
@@ -124,7 +124,7 @@ LDFLAGS_vmlinux += --no-undefined
LDFLAGS_vmlinux += -X
# Report orphan sections
ifdef CONFIG_LD_ORPHAN_WARN
LDFLAGS_vmlinux += --orphan-handling=warn
LDFLAGS_vmlinux += --orphan-handling=$(CONFIG_LD_ORPHAN_WARN_LEVEL)
endif
# Next argument is a linker script
LDFLAGS_vmlinux += -T
arch/arm/crypto/Kconfig
+1 -1
View File
@@ -18,7 +18,7 @@ config CRYPTO_GHASH_ARM_CE
depends on KERNEL_MODE_NEON
select CRYPTO_HASH
select CRYPTO_CRYPTD
select CRYPTO_GF128MUL
select CRYPTO_LIB_GF128MUL
help
GCM GHASH function (NIST SP800-38D)
arch/arm/crypto/aes-cipher-glue.c
+1 -1
View File
@@ -7,7 +7,7 @@
*/
#include <crypto/aes.h>
#include <linux/crypto.h>
#include <crypto/algapi.h>
#include <linux/module.h>
asmlinkage void __aes_arm_encrypt(u32 *rk, int rounds, const u8 *in, u8 *out);
arch/arm/crypto/nh-neon-core.S
+1 -1
View File
@@ -69,7 +69,7 @@
/*
* void nh_neon(const u32 *key, const u8 *message, size_t message_len,
* u8 hash[NH_HASH_BYTES])
* __le64 hash[NH_NUM_PASSES])
*
* It's guaranteed that message_len % 16 == 0.
*/
arch/arm/crypto/nhpoly1305-neon-glue.c
+2 -9
View File
@@ -14,14 +14,7 @@
#include <linux/module.h>
asmlinkage void nh_neon(const u32 *key, const u8 *message, size_t message_len,
u8 hash[NH_HASH_BYTES]);
/* wrapper to avoid indirect call to assembly, which doesn't work with CFI */
static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,
__le64 hash[NH_NUM_PASSES])
{
nh_neon(key, message, message_len, (u8 *)hash);
}
__le64 hash[NH_NUM_PASSES]);
static int nhpoly1305_neon_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
@@ -33,7 +26,7 @@ static int nhpoly1305_neon_update(struct shash_desc *desc,
unsigned int n = min_t(unsigned int, srclen, SZ_4K);
kernel_neon_begin();
crypto_nhpoly1305_update_helper(desc, src, n, _nh_neon);
crypto_nhpoly1305_update_helper(desc, src, n, nh_neon);
kernel_neon_end();
src += n;
srclen -= n;
arch/arm/include/asm/gpio.h
-1
View File
@@ -2,7 +2,6 @@
#ifndef _ARCH_ARM_GPIO_H
#define _ARCH_ARM_GPIO_H
/* Note: this may rely upon the value of ARCH_NR_GPIOS set in mach/gpio.h */
#include <asm-generic/gpio.h>
/* The trivial gpiolib dispatchers */
arch/arm64/Kconfig
+1 -12
View File
@@ -1988,6 +1988,7 @@ config ARM64_MTE
depends on ARM64_PAN
select ARCH_HAS_SUBPAGE_FAULTS
select ARCH_USES_HIGH_VMA_FLAGS
select ARCH_USES_PG_ARCH_X
help
Memory Tagging (part of the ARMv8.5 Extensions) provides
architectural support for run-time, always-on detection of
@@ -2168,18 +2169,6 @@ config STACKPROTECTOR_PER_TASK
def_bool y
depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG
# The GPIO number here must be sorted by descending number. In case of
# a multiplatform kernel, we just want the highest value required by the
# selected platforms.
config ARCH_NR_GPIO
int
default 2048 if ARCH_APPLE
default 0
help
Maximum number of GPIOs in the system.
If unsure, leave the default value.
config UNWIND_PATCH_PAC_INTO_SCS
bool "Enable shadow call stack dynamically using code patching"
# needs Clang with https://reviews.llvm.org/D111780 incorporated
arch/arm64/crypto/Kconfig
+47 -2
View File
@@ -6,8 +6,8 @@ config CRYPTO_GHASH_ARM64_CE
tristate "Hash functions: GHASH (ARMv8 Crypto Extensions)"
depends on KERNEL_MODE_NEON
select CRYPTO_HASH
select CRYPTO_GF128MUL
select CRYPTO_LIB_AES
select CRYPTO_LIB_GF128MUL
select CRYPTO_AEAD
help
GCM GHASH function (NIST SP800-38D)
@@ -96,6 +96,17 @@ config CRYPTO_SHA3_ARM64
Architecture: arm64 using:
- ARMv8.2 Crypto Extensions
config CRYPTO_SM3_NEON
tristate "Hash functions: SM3 (NEON)"
depends on KERNEL_MODE_NEON
select CRYPTO_HASH
select CRYPTO_SM3
help
SM3 (ShangMi 3) secure hash function (OSCCA GM/T 0004-2012)
Architecture: arm64 using:
- NEON (Advanced SIMD) extensions
config CRYPTO_SM3_ARM64_CE
tristate "Hash functions: SM3 (ARMv8.2 Crypto Extensions)"
depends on KERNEL_MODE_NEON
@@ -220,7 +231,7 @@ config CRYPTO_SM4_ARM64_CE
- NEON (Advanced SIMD) extensions
config CRYPTO_SM4_ARM64_CE_BLK
tristate "Ciphers: SM4, modes: ECB/CBC/CFB/CTR (ARMv8 Crypto Extensions)"
tristate "Ciphers: SM4, modes: ECB/CBC/CFB/CTR/XTS (ARMv8 Crypto Extensions)"
depends on KERNEL_MODE_NEON
select CRYPTO_SKCIPHER
select CRYPTO_SM4
@@ -231,6 +242,8 @@ config CRYPTO_SM4_ARM64_CE_BLK
- CBC (Cipher Block Chaining) mode (NIST SP800-38A)
- CFB (Cipher Feedback) mode (NIST SP800-38A)
- CTR (Counter) mode (NIST SP800-38A)
- XTS (XOR Encrypt XOR with ciphertext stealing) mode (NIST SP800-38E
and IEEE 1619)
Architecture: arm64 using:
- ARMv8 Crypto Extensions
@@ -268,6 +281,38 @@ config CRYPTO_AES_ARM64_CE_CCM
- ARMv8 Crypto Extensions
- NEON (Advanced SIMD) extensions
config CRYPTO_SM4_ARM64_CE_CCM
tristate "AEAD cipher: SM4 in CCM mode (ARMv8 Crypto Extensions)"
depends on KERNEL_MODE_NEON
select CRYPTO_ALGAPI
select CRYPTO_AEAD
select CRYPTO_SM4
select CRYPTO_SM4_ARM64_CE_BLK
help
AEAD cipher: SM4 cipher algorithms (OSCCA GB/T 32907-2016) with
CCM (Counter with Cipher Block Chaining-Message Authentication Code)
authenticated encryption mode (NIST SP800-38C)
Architecture: arm64 using:
- ARMv8 Crypto Extensions
- NEON (Advanced SIMD) extensions
config CRYPTO_SM4_ARM64_CE_GCM
tristate "AEAD cipher: SM4 in GCM mode (ARMv8 Crypto Extensions)"
depends on KERNEL_MODE_NEON
select CRYPTO_ALGAPI
select CRYPTO_AEAD
select CRYPTO_SM4
select CRYPTO_SM4_ARM64_CE_BLK
help
AEAD cipher: SM4 cipher algorithms (OSCCA GB/T 32907-2016) with
GCM (Galois/Counter Mode) authenticated encryption mode (NIST SP800-38D)
Architecture: arm64 using:
- ARMv8 Crypto Extensions
- PMULL (Polynomial Multiply Long) instructions
- NEON (Advanced SIMD) extensions
config CRYPTO_CRCT10DIF_ARM64_CE
tristate "CRCT10DIF (PMULL)"
depends on KERNEL_MODE_NEON && CRC_T10DIF
arch/arm64/crypto/Makefile
+9
View File
@@ -17,6 +17,9 @@ sha512-ce-y := sha512-ce-glue.o sha512-ce-core.o
obj-$(CONFIG_CRYPTO_SHA3_ARM64) += sha3-ce.o
sha3-ce-y := sha3-ce-glue.o sha3-ce-core.o
obj-$(CONFIG_CRYPTO_SM3_NEON) += sm3-neon.o
sm3-neon-y := sm3-neon-glue.o sm3-neon-core.o
obj-$(CONFIG_CRYPTO_SM3_ARM64_CE) += sm3-ce.o
sm3-ce-y := sm3-ce-glue.o sm3-ce-core.o
@@ -26,6 +29,12 @@ sm4-ce-cipher-y := sm4-ce-cipher-glue.o sm4-ce-cipher-core.o
obj-$(CONFIG_CRYPTO_SM4_ARM64_CE_BLK) += sm4-ce.o
sm4-ce-y := sm4-ce-glue.o sm4-ce-core.o
obj-$(CONFIG_CRYPTO_SM4_ARM64_CE_CCM) += sm4-ce-ccm.o
sm4-ce-ccm-y := sm4-ce-ccm-glue.o sm4-ce-ccm-core.o
obj-$(CONFIG_CRYPTO_SM4_ARM64_CE_GCM) += sm4-ce-gcm.o
sm4-ce-gcm-y := sm4-ce-gcm-glue.o sm4-ce-gcm-core.o
obj-$(CONFIG_CRYPTO_SM4_ARM64_NEON_BLK) += sm4-neon.o
sm4-neon-y := sm4-neon-glue.o sm4-neon-core.o
arch/arm64/crypto/aes-ce-glue.c
+1 -1
View File
@@ -9,9 +9,9 @@
#include <asm/simd.h>
#include <asm/unaligned.h>
#include <crypto/aes.h>
#include <crypto/algapi.h>
#include <crypto/internal/simd.h>
#include <linux/cpufeature.h>
#include <linux/crypto.h>
#include <linux/module.h>
#include "aes-ce-setkey.h"
arch/arm64/crypto/aes-cipher-glue.c
+1 -1
View File
@@ -6,7 +6,7 @@
*/
#include <crypto/aes.h>
#include <linux/crypto.h>
#include <crypto/algapi.h>
#include <linux/module.h>
asmlinkage void __aes_arm64_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds);
arch/arm64/crypto/aes-modes.S
+12 -22
View File
@@ -52,8 +52,7 @@ SYM_FUNC_END(aes_decrypt_block5x)
*/
AES_FUNC_START(aes_ecb_encrypt)
stp x29, x30, [sp, #-16]!
mov x29, sp
frame_push 0
enc_prepare w3, x2, x5
@@ -77,14 +76,13 @@ ST5( st1 {v4.16b}, [x0], #16 )
subs w4, w4, #1
bne .Lecbencloop
.Lecbencout:
ldp x29, x30, [sp], #16
frame_pop
ret
AES_FUNC_END(aes_ecb_encrypt)
AES_FUNC_START(aes_ecb_decrypt)
stp x29, x30, [sp, #-16]!
mov x29, sp
frame_push 0
dec_prepare w3, x2, x5
@@ -108,7 +106,7 @@ ST5( st1 {v4.16b}, [x0], #16 )
subs w4, w4, #1
bne .Lecbdecloop
.Lecbdecout:
ldp x29, x30, [sp], #16
frame_pop
ret
AES_FUNC_END(aes_ecb_decrypt)
@@ -171,9 +169,6 @@ AES_FUNC_END(aes_cbc_encrypt)
AES_FUNC_END(aes_essiv_cbc_encrypt)
AES_FUNC_START(aes_essiv_cbc_decrypt)
stp x29, x30, [sp, #-16]!
mov x29, sp
ld1 {cbciv.16b}, [x5] /* get iv */
mov w8, #14 /* AES-256: 14 rounds */
@@ -182,11 +177,9 @@ AES_FUNC_START(aes_essiv_cbc_decrypt)
b .Lessivcbcdecstart
AES_FUNC_START(aes_cbc_decrypt)
stp x29, x30, [sp, #-16]!
mov x29, sp
ld1 {cbciv.16b}, [x5] /* get iv */
.Lessivcbcdecstart:
frame_push 0
dec_prepare w3, x2, x6
.LcbcdecloopNx:
@@ -236,7 +229,7 @@ ST5( st1 {v4.16b}, [x0], #16 )
bne .Lcbcdecloop
.Lcbcdecout:
st1 {cbciv.16b}, [x5] /* return iv */
ldp x29, x30, [sp], #16
frame_pop
ret
AES_FUNC_END(aes_cbc_decrypt)
AES_FUNC_END(aes_essiv_cbc_decrypt)
@@ -337,8 +330,7 @@ AES_FUNC_END(aes_cbc_cts_decrypt)
BLOCKS .req x13
BLOCKS_W .req w13
stp x29, x30, [sp, #-16]!
mov x29, sp
frame_push 0
enc_prepare ROUNDS_W, KEY, IV_PART
ld1 {vctr.16b}, [IV]
@@ -481,7 +473,7 @@ ST5( st1 {v4.16b}, [OUT], #16 )
.if !\xctr
st1 {vctr.16b}, [IV] /* return next CTR value */
.endif
ldp x29, x30, [sp], #16
frame_pop
ret
.Lctrtail\xctr:
@@ -645,8 +637,7 @@ AES_FUNC_END(aes_xctr_encrypt)
.endm
AES_FUNC_START(aes_xts_encrypt)
stp x29, x30, [sp, #-16]!
mov x29, sp
frame_push 0
ld1 {v4.16b}, [x6]
xts_load_mask v8
@@ -704,7 +695,7 @@ AES_FUNC_START(aes_xts_encrypt)
st1 {v0.16b}, [x0]
.Lxtsencret:
st1 {v4.16b}, [x6]
ldp x29, x30, [sp], #16
frame_pop
ret
.LxtsencctsNx:
@@ -732,8 +723,7 @@ AES_FUNC_START(aes_xts_encrypt)
AES_FUNC_END(aes_xts_encrypt)
AES_FUNC_START(aes_xts_decrypt)
stp x29, x30, [sp, #-16]!
mov x29, sp
frame_push 0
/* subtract 16 bytes if we are doing CTS */
sub w8, w4, #0x10
@@ -794,7 +784,7 @@ AES_FUNC_START(aes_xts_decrypt)
b .Lxtsdecloop
.Lxtsdecout:
st1 {v4.16b}, [x6]
ldp x29, x30, [sp], #16
frame_pop
ret
.Lxtsdeccts:
arch/arm64/crypto/aes-neonbs-core.S
+7 -9
View File
@@ -760,7 +760,7 @@ SYM_FUNC_START_LOCAL(__xts_crypt8)
eor v6.16b, v6.16b, v31.16b
eor v7.16b, v7.16b, v16.16b
stp q16, q17, [sp, #16]
stp q16, q17, [x6]
mov bskey, x2
mov rounds, x3
@@ -768,8 +768,8 @@ SYM_FUNC_START_LOCAL(__xts_crypt8)
SYM_FUNC_END(__xts_crypt8)
.macro __xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7
stp x29, x30, [sp, #-48]!
mov x29, sp
frame_push 0, 32
add x6, sp, #.Lframe_local_offset
ld1 {v25.16b}, [x5]
@@ -781,7 +781,7 @@ SYM_FUNC_END(__xts_crypt8)
eor v18.16b, \o2\().16b, v27.16b
eor v19.16b, \o3\().16b, v28.16b
ldp q24, q25, [sp, #16]
ldp q24, q25, [x6]
eor v20.16b, \o4\().16b, v29.16b
eor v21.16b, \o5\().16b, v30.16b
@@ -795,7 +795,7 @@ SYM_FUNC_END(__xts_crypt8)
b.gt 0b
st1 {v25.16b}, [x5]
ldp x29, x30, [sp], #48
frame_pop
ret
.endm
@@ -820,9 +820,7 @@ SYM_FUNC_END(aesbs_xts_decrypt)
* int rounds, int blocks, u8 iv[])
*/
SYM_FUNC_START(aesbs_ctr_encrypt)
stp x29, x30, [sp, #-16]!
mov x29, sp
frame_push 0
ldp x7, x8, [x5]
ld1 {v0.16b}, [x5]
CPU_LE( rev x7, x7 )
@@ -862,6 +860,6 @@ CPU_LE( rev x8, x8 )
b.gt 0b
st1 {v0.16b}, [x5]
ldp x29, x30, [sp], #16
frame_pop
ret
SYM_FUNC_END(aesbs_ctr_encrypt)
arch/arm64/crypto/crct10dif-ce-core.S
+2 -3
View File
@@ -429,7 +429,7 @@ CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 )
umov w0, v0.h[0]
.ifc \p, p8
ldp x29, x30, [sp], #16
frame_pop
.endif
ret
@@ -466,8 +466,7 @@ CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
// Assumes len >= 16.
//
SYM_FUNC_START(crc_t10dif_pmull_p8)
stp x29, x30, [sp, #-16]!
mov x29, sp
frame_push 1
crc_t10dif_pmull p8
SYM_FUNC_END(crc_t10dif_pmull_p8)
arch/arm64/crypto/ghash-ce-core.S
+3 -5
View File
@@ -436,9 +436,7 @@ SYM_FUNC_END(pmull_ghash_update_p8)
.align 6
.macro pmull_gcm_do_crypt, enc
stp x29, x30, [sp, #-32]!
mov x29, sp
str x19, [sp, #24]
frame_push 1
load_round_keys x7, x6, x8
@@ -529,7 +527,7 @@ CPU_LE( rev w8, w8 )
.endif
bne 0b
3: ldp x19, x10, [sp, #24]
3: ldr x10, [sp, #.Lframe_local_offset]
cbz x10, 5f // output tag?
ld1 {INP3.16b}, [x10] // load lengths[]
@@ -562,7 +560,7 @@ CPU_LE( rev w8, w8 )
smov w0, v0.b[0] // return b0
.endif
4: ldp x29, x30, [sp], #32
4: frame_pop
ret
5:
arch/arm64/crypto/ghash-ce-glue.c
+1 -1
View File
@@ -508,7 +508,7 @@ static void __exit ghash_ce_mod_exit(void)
crypto_unregister_shash(&ghash_alg);
}
static const struct cpu_feature ghash_cpu_feature[] = {
static const struct cpu_feature __maybe_unused ghash_cpu_feature[] = {
{ cpu_feature(PMULL) }, { }
};
MODULE_DEVICE_TABLE(cpu, ghash_cpu_feature);
arch/arm64/crypto/nh-neon-core.S
+3 -2
View File
@@ -8,6 +8,7 @@
*/
#include <linux/linkage.h>
#include <linux/cfi_types.h>
KEY .req x0
MESSAGE .req x1
@@ -58,11 +59,11 @@
/*
* void nh_neon(const u32 *key, const u8 *message, size_t message_len,
* u8 hash[NH_HASH_BYTES])
* __le64 hash[NH_NUM_PASSES])
*
* It's guaranteed that message_len % 16 == 0.
*/
SYM_FUNC_START(nh_neon)
SYM_TYPED_FUNC_START(nh_neon)
ld1 {K0.4s,K1.4s}, [KEY], #32
movi PASS0_SUMS.2d, #0
arch/arm64/crypto/nhpoly1305-neon-glue.c
+2 -9
View File
@@ -14,14 +14,7 @@
#include <linux/module.h>
asmlinkage void nh_neon(const u32 *key, const u8 *message, size_t message_len,
u8 hash[NH_HASH_BYTES]);
/* wrapper to avoid indirect call to assembly, which doesn't work with CFI */
static void _nh_neon(const u32 *key, const u8 *message, size_t message_len,
__le64 hash[NH_NUM_PASSES])
{
nh_neon(key, message, message_len, (u8 *)hash);
}
__le64 hash[NH_NUM_PASSES]);
static int nhpoly1305_neon_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
@@ -33,7 +26,7 @@ static int nhpoly1305_neon_update(struct shash_desc *desc,
unsigned int n = min_t(unsigned int, srclen, SZ_4K);
kernel_neon_begin();
crypto_nhpoly1305_update_helper(desc, src, n, _nh_neon);
crypto_nhpoly1305_update_helper(desc, src, n, nh_neon);
kernel_neon_end();
src += n;
srclen -= n;
arch/arm64/crypto/sm3-ce-glue.c
+1 -1
View File
@@ -84,7 +84,7 @@ static struct shash_alg sm3_alg = {
.base.cra_driver_name = "sm3-ce",
.base.cra_blocksize = SM3_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
.base.cra_priority = 200,
.base.cra_priority = 400,
};
static int __init sm3_ce_mod_init(void)
arch/arm64/crypto/sm3-neon-core.S
+601
View File
@@ -0,0 +1,601 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* sm3-neon-core.S - SM3 secure hash using NEON instructions
*
* Linux/arm64 port of the libgcrypt SM3 implementation for AArch64
*
* Copyright (C) 2021 Jussi Kivilinna <jussi.kivilinna@iki.fi>
* Copyright (c) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#include <linux/linkage.h>
#include <linux/cfi_types.h>
#include <asm/assembler.h>
/* Context structure */
#define state_h0 0
#define state_h1 4
#define state_h2 8
#define state_h3 12
#define state_h4 16
#define state_h5 20
#define state_h6 24
#define state_h7 28
/* Stack structure */
#define STACK_W_SIZE (32 * 2 * 3)
#define STACK_W (0)
#define STACK_SIZE (STACK_W + STACK_W_SIZE)
/* Register macros */
#define RSTATE x0
#define RDATA x1
#define RNBLKS x2
#define RKPTR x28
#define RFRAME x29
#define ra w3
#define rb w4
#define rc w5
#define rd w6
#define re w7
#define rf w8
#define rg w9
#define rh w10
#define t0 w11
#define t1 w12
#define t2 w13
#define t3 w14
#define t4 w15
#define t5 w16
#define t6 w17
#define k_even w19
#define k_odd w20
#define addr0 x21
#define addr1 x22
#define s0 w23
#define s1 w24
#define s2 w25
#define s3 w26
#define W0 v0
#define W1 v1
#define W2 v2
#define W3 v3
#define W4 v4
#define W5 v5
#define XTMP0 v6
#define XTMP1 v7
#define XTMP2 v16
#define XTMP3 v17
#define XTMP4 v18
#define XTMP5 v19
#define XTMP6 v20
/* Helper macros. */
#define _(...) /*_*/
#define clear_vec(x) \
movi x.8h, #0;
#define rolw(o, a, n) \
ror o, a, #(32 - n);
/* Round function macros. */
#define GG1_1(x, y, z, o, t) \
eor o, x, y;
#define GG1_2(x, y, z, o, t) \
eor o, o, z;
#define GG1_3(x, y, z, o, t)
#define FF1_1(x, y, z, o, t) GG1_1(x, y, z, o, t)
#define FF1_2(x, y, z, o, t)
#define FF1_3(x, y, z, o, t) GG1_2(x, y, z, o, t)
#define GG2_1(x, y, z, o, t) \
bic o, z, x;
#define GG2_2(x, y, z, o, t) \
and t, y, x;
#define GG2_3(x, y, z, o, t) \
eor o, o, t;
#define FF2_1(x, y, z, o, t) \
eor o, x, y;
#define FF2_2(x, y, z, o, t) \
and t, x, y; \
and o, o, z;
#define FF2_3(x, y, z, o, t) \
eor o, o, t;
#define R(i, a, b, c, d, e, f, g, h, k, K_LOAD, round, widx, wtype, IOP, iop_param) \
K_LOAD(round); \
ldr t5, [sp, #(wtype##_W1_ADDR(round, widx))]; \
rolw(t0, a, 12); /* rol(a, 12) => t0 */ \
IOP(1, iop_param); \
FF##i##_1(a, b, c, t1, t2); \
ldr t6, [sp, #(wtype##_W1W2_ADDR(round, widx))]; \
add k, k, e; \
IOP(2, iop_param); \
GG##i##_1(e, f, g, t3, t4); \
FF##i##_2(a, b, c, t1, t2); \
IOP(3, iop_param); \
add k, k, t0; \
add h, h, t5; \
add d, d, t6; /* w1w2 + d => d */ \
IOP(4, iop_param); \
rolw(k, k, 7); /* rol (t0 + e + t), 7) => k */ \
GG##i##_2(e, f, g, t3, t4); \
add h, h, k; /* h + w1 + k => h */ \
IOP(5, iop_param); \
FF##i##_3(a, b, c, t1, t2); \
eor t0, t0, k; /* k ^ t0 => t0 */ \
GG##i##_3(e, f, g, t3, t4); \
add d, d, t1; /* FF(a,b,c) + d => d */ \
IOP(6, iop_param); \
add t3, t3, h; /* GG(e,f,g) + h => t3 */ \
rolw(b, b, 9); /* rol(b, 9) => b */ \
eor h, t3, t3, ror #(32-9); \
IOP(7, iop_param); \
add d, d, t0; /* t0 + d => d */ \
rolw(f, f, 19); /* rol(f, 19) => f */ \
IOP(8, iop_param); \
eor h, h, t3, ror #(32-17); /* P0(t3) => h */
#define R1(a, b, c, d, e, f, g, h, k, K_LOAD, round, widx, wtype, IOP, iop_param) \
R(1, ##a, ##b, ##c, ##d, ##e, ##f, ##g, ##h, ##k, K_LOAD, round, widx, wtype, IOP, iop_param)
#define R2(a, b, c, d, e, f, g, h, k, K_LOAD, round, widx, wtype, IOP, iop_param) \
R(2, ##a, ##b, ##c, ##d, ##e, ##f, ##g, ##h, ##k, K_LOAD, round, widx, wtype, IOP, iop_param)
#define KL(round) \
ldp k_even, k_odd, [RKPTR, #(4*(round))];
/* Input expansion macros. */
/* Byte-swapped input address. */
#define IW_W_ADDR(round, widx, offs) \
(STACK_W + ((round) / 4) * 64 + (offs) + ((widx) * 4))
/* Expanded input address. */
#define XW_W_ADDR(round, widx, offs) \
(STACK_W + ((((round) / 3) - 4) % 2) * 64 + (offs) + ((widx) * 4))
/* Rounds 1-12, byte-swapped input block addresses. */
#define IW_W1_ADDR(round, widx) IW_W_ADDR(round, widx, 32)
#define IW_W1W2_ADDR(round, widx) IW_W_ADDR(round, widx, 48)
/* Rounds 1-12, expanded input block addresses. */
#define XW_W1_ADDR(round, widx) XW_W_ADDR(round, widx, 0)
#define XW_W1W2_ADDR(round, widx) XW_W_ADDR(round, widx, 16)
/* Input block loading.
* Interleaving within round function needed for in-order CPUs. */
#define LOAD_W_VEC_1_1() \
add addr0, sp, #IW_W1_ADDR(0, 0);
#define LOAD_W_VEC_1_2() \
add addr1, sp, #IW_W1_ADDR(4, 0);
#define LOAD_W_VEC_1_3() \
ld1 {W0.16b}, [RDATA], #16;
#define LOAD_W_VEC_1_4() \
ld1 {W1.16b}, [RDATA], #16;
#define LOAD_W_VEC_1_5() \
ld1 {W2.16b}, [RDATA], #16;
#define LOAD_W_VEC_1_6() \
ld1 {W3.16b}, [RDATA], #16;
#define LOAD_W_VEC_1_7() \
rev32 XTMP0.16b, W0.16b;
#define LOAD_W_VEC_1_8() \
rev32 XTMP1.16b, W1.16b;
#define LOAD_W_VEC_2_1() \
rev32 XTMP2.16b, W2.16b;
#define LOAD_W_VEC_2_2() \
rev32 XTMP3.16b, W3.16b;
#define LOAD_W_VEC_2_3() \
eor XTMP4.16b, XTMP1.16b, XTMP0.16b;
#define LOAD_W_VEC_2_4() \
eor XTMP5.16b, XTMP2.16b, XTMP1.16b;
#define LOAD_W_VEC_2_5() \
st1 {XTMP0.16b}, [addr0], #16;
#define LOAD_W_VEC_2_6() \
st1 {XTMP4.16b}, [addr0]; \
add addr0, sp, #IW_W1_ADDR(8, 0);
#define LOAD_W_VEC_2_7() \
eor XTMP6.16b, XTMP3.16b, XTMP2.16b;
#define LOAD_W_VEC_2_8() \
ext W0.16b, XTMP0.16b, XTMP0.16b, #8; /* W0: xx, w0, xx, xx */
#define LOAD_W_VEC_3_1() \
mov W2.16b, XTMP1.16b; /* W2: xx, w6, w5, w4 */
#define LOAD_W_VEC_3_2() \
st1 {XTMP1.16b}, [addr1], #16;
#define LOAD_W_VEC_3_3() \
st1 {XTMP5.16b}, [addr1]; \
ext W1.16b, XTMP0.16b, XTMP0.16b, #4; /* W1: xx, w3, w2, w1 */
#define LOAD_W_VEC_3_4() \
ext W3.16b, XTMP1.16b, XTMP2.16b, #12; /* W3: xx, w9, w8, w7 */
#define LOAD_W_VEC_3_5() \
ext W4.16b, XTMP2.16b, XTMP3.16b, #8; /* W4: xx, w12, w11, w10 */
#define LOAD_W_VEC_3_6() \
st1 {XTMP2.16b}, [addr0], #16;
#define LOAD_W_VEC_3_7() \
st1 {XTMP6.16b}, [addr0];
#define LOAD_W_VEC_3_8() \
ext W5.16b, XTMP3.16b, XTMP3.16b, #4; /* W5: xx, w15, w14, w13 */
#define LOAD_W_VEC_1(iop_num, ...) \
LOAD_W_VEC_1_##iop_num()
#define LOAD_W_VEC_2(iop_num, ...) \
LOAD_W_VEC_2_##iop_num()
#define LOAD_W_VEC_3(iop_num, ...) \
LOAD_W_VEC_3_##iop_num()
/* Message scheduling. Note: 3 words per vector register.
* Interleaving within round function needed for in-order CPUs. */
#define SCHED_W_1_1(round, w0, w1, w2, w3, w4, w5) \
/* Load (w[i - 16]) => XTMP0 */ \
/* Load (w[i - 13]) => XTMP5 */ \
ext XTMP0.16b, w0.16b, w0.16b, #12; /* XTMP0: w0, xx, xx, xx */
#define SCHED_W_1_2(round, w0, w1, w2, w3, w4, w5) \
ext XTMP5.16b, w1.16b, w1.16b, #12;
#define SCHED_W_1_3(round, w0, w1, w2, w3, w4, w5) \
ext XTMP0.16b, XTMP0.16b, w1.16b, #12; /* XTMP0: xx, w2, w1, w0 */
#define SCHED_W_1_4(round, w0, w1, w2, w3, w4, w5) \
ext XTMP5.16b, XTMP5.16b, w2.16b, #12;
#define SCHED_W_1_5(round, w0, w1, w2, w3, w4, w5) \
/* w[i - 9] == w3 */ \
/* W3 ^ XTMP0 => XTMP0 */ \
eor XTMP0.16b, XTMP0.16b, w3.16b;
#define SCHED_W_1_6(round, w0, w1, w2, w3, w4, w5) \
/* w[i - 3] == w5 */ \
/* rol(XMM5, 15) ^ XTMP0 => XTMP0 */ \
/* rol(XTMP5, 7) => XTMP1 */ \
add addr0, sp, #XW_W1_ADDR((round), 0); \
shl XTMP2.4s, w5.4s, #15;
#define SCHED_W_1_7(round, w0, w1, w2, w3, w4, w5) \
shl XTMP1.4s, XTMP5.4s, #7;
#define SCHED_W_1_8(round, w0, w1, w2, w3, w4, w5) \
sri XTMP2.4s, w5.4s, #(32-15);
#define SCHED_W_2_1(round, w0, w1, w2, w3, w4, w5) \
sri XTMP1.4s, XTMP5.4s, #(32-7);
#define SCHED_W_2_2(round, w0, w1, w2, w3, w4, w5) \
eor XTMP0.16b, XTMP0.16b, XTMP2.16b;
#define SCHED_W_2_3(round, w0, w1, w2, w3, w4, w5) \
/* w[i - 6] == W4 */ \
/* W4 ^ XTMP1 => XTMP1 */ \
eor XTMP1.16b, XTMP1.16b, w4.16b;
#define SCHED_W_2_4(round, w0, w1, w2, w3, w4, w5) \
/* P1(XTMP0) ^ XTMP1 => W0 */ \
shl XTMP3.4s, XTMP0.4s, #15;
#define SCHED_W_2_5(round, w0, w1, w2, w3, w4, w5) \
shl XTMP4.4s, XTMP0.4s, #23;
#define SCHED_W_2_6(round, w0, w1, w2, w3, w4, w5) \
eor w0.16b, XTMP1.16b, XTMP0.16b;
#define SCHED_W_2_7(round, w0, w1, w2, w3, w4, w5) \
sri XTMP3.4s, XTMP0.4s, #(32-15);
#define SCHED_W_2_8(round, w0, w1, w2, w3, w4, w5) \
sri XTMP4.4s, XTMP0.4s, #(32-23);
#define SCHED_W_3_1(round, w0, w1, w2, w3, w4, w5) \
eor w0.16b, w0.16b, XTMP3.16b;
#define SCHED_W_3_2(round, w0, w1, w2, w3, w4, w5) \
/* Load (w[i - 3]) => XTMP2 */ \
ext XTMP2.16b, w4.16b, w4.16b, #12;
#define SCHED_W_3_3(round, w0, w1, w2, w3, w4, w5) \
eor w0.16b, w0.16b, XTMP4.16b;
#define SCHED_W_3_4(round, w0, w1, w2, w3, w4, w5) \
ext XTMP2.16b, XTMP2.16b, w5.16b, #12;
#define SCHED_W_3_5(round, w0, w1, w2, w3, w4, w5) \
/* W1 ^ W2 => XTMP3 */ \
eor XTMP3.16b, XTMP2.16b, w0.16b;
#define SCHED_W_3_6(round, w0, w1, w2, w3, w4, w5)
#define SCHED_W_3_7(round, w0, w1, w2, w3, w4, w5) \
st1 {XTMP2.16b-XTMP3.16b}, [addr0];
#define SCHED_W_3_8(round, w0, w1, w2, w3, w4, w5)
#define SCHED_W_W0W1W2W3W4W5_1(iop_num, round) \
SCHED_W_1_##iop_num(round, W0, W1, W2, W3, W4, W5)
#define SCHED_W_W0W1W2W3W4W5_2(iop_num, round) \
SCHED_W_2_##iop_num(round, W0, W1, W2, W3, W4, W5)
#define SCHED_W_W0W1W2W3W4W5_3(iop_num, round) \
SCHED_W_3_##iop_num(round, W0, W1, W2, W3, W4, W5)
#define SCHED_W_W1W2W3W4W5W0_1(iop_num, round) \
SCHED_W_1_##iop_num(round, W1, W2, W3, W4, W5, W0)
#define SCHED_W_W1W2W3W4W5W0_2(iop_num, round) \
SCHED_W_2_##iop_num(round, W1, W2, W3, W4, W5, W0)
#define SCHED_W_W1W2W3W4W5W0_3(iop_num, round) \
SCHED_W_3_##iop_num(round, W1, W2, W3, W4, W5, W0)
#define SCHED_W_W2W3W4W5W0W1_1(iop_num, round) \
SCHED_W_1_##iop_num(round, W2, W3, W4, W5, W0, W1)
#define SCHED_W_W2W3W4W5W0W1_2(iop_num, round) \
SCHED_W_2_##iop_num(round, W2, W3, W4, W5, W0, W1)
#define SCHED_W_W2W3W4W5W0W1_3(iop_num, round) \
SCHED_W_3_##iop_num(round, W2, W3, W4, W5, W0, W1)
#define SCHED_W_W3W4W5W0W1W2_1(iop_num, round) \
SCHED_W_1_##iop_num(round, W3, W4, W5, W0, W1, W2)
#define SCHED_W_W3W4W5W0W1W2_2(iop_num, round) \
SCHED_W_2_##iop_num(round, W3, W4, W5, W0, W1, W2)
#define SCHED_W_W3W4W5W0W1W2_3(iop_num, round) \
SCHED_W_3_##iop_num(round, W3, W4, W5, W0, W1, W2)
#define SCHED_W_W4W5W0W1W2W3_1(iop_num, round) \
SCHED_W_1_##iop_num(round, W4, W5, W0, W1, W2, W3)
#define SCHED_W_W4W5W0W1W2W3_2(iop_num, round) \
SCHED_W_2_##iop_num(round, W4, W5, W0, W1, W2, W3)
#define SCHED_W_W4W5W0W1W2W3_3(iop_num, round) \
SCHED_W_3_##iop_num(round, W4, W5, W0, W1, W2, W3)
#define SCHED_W_W5W0W1W2W3W4_1(iop_num, round) \
SCHED_W_1_##iop_num(round, W5, W0, W1, W2, W3, W4)
#define SCHED_W_W5W0W1W2W3W4_2(iop_num, round) \
SCHED_W_2_##iop_num(round, W5, W0, W1, W2, W3, W4)
#define SCHED_W_W5W0W1W2W3W4_3(iop_num, round) \
SCHED_W_3_##iop_num(round, W5, W0, W1, W2, W3, W4)
/*
* Transform blocks*64 bytes (blocks*16 32-bit words) at 'src'.
*
* void sm3_neon_transform(struct sm3_state *sst, u8 const *src,
* int blocks)
*/
.text
.align 3
SYM_TYPED_FUNC_START(sm3_neon_transform)
ldp ra, rb, [RSTATE, #0]
ldp rc, rd, [RSTATE, #8]
ldp re, rf, [RSTATE, #16]
ldp rg, rh, [RSTATE, #24]
stp x28, x29, [sp, #-16]!
stp x19, x20, [sp, #-16]!
stp x21, x22, [sp, #-16]!
stp x23, x24, [sp, #-16]!
stp x25, x26, [sp, #-16]!
mov RFRAME, sp
sub addr0, sp, #STACK_SIZE
adr_l RKPTR, .LKtable
and sp, addr0, #(~63)
/* Preload first block. */
LOAD_W_VEC_1(1, 0)
LOAD_W_VEC_1(2, 0)
LOAD_W_VEC_1(3, 0)
LOAD_W_VEC_1(4, 0)
LOAD_W_VEC_1(5, 0)
LOAD_W_VEC_1(6, 0)
LOAD_W_VEC_1(7, 0)
LOAD_W_VEC_1(8, 0)
LOAD_W_VEC_2(1, 0)
LOAD_W_VEC_2(2, 0)
LOAD_W_VEC_2(3, 0)
LOAD_W_VEC_2(4, 0)
LOAD_W_VEC_2(5, 0)
LOAD_W_VEC_2(6, 0)
LOAD_W_VEC_2(7, 0)
LOAD_W_VEC_2(8, 0)
LOAD_W_VEC_3(1, 0)
LOAD_W_VEC_3(2, 0)
LOAD_W_VEC_3(3, 0)
LOAD_W_VEC_3(4, 0)
LOAD_W_VEC_3(5, 0)
LOAD_W_VEC_3(6, 0)
LOAD_W_VEC_3(7, 0)
LOAD_W_VEC_3(8, 0)
.balign 16
.Loop:
/* Transform 0-3 */
R1(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 0, 0, IW, _, 0)
R1(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 1, 1, IW, _, 0)
R1(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 2, 2, IW, _, 0)
R1(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 3, 3, IW, _, 0)
/* Transform 4-7 + Precalc 12-14 */
R1(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 4, 0, IW, _, 0)
R1(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 5, 1, IW, _, 0)
R1(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 6, 2, IW, SCHED_W_W0W1W2W3W4W5_1, 12)
R1(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 7, 3, IW, SCHED_W_W0W1W2W3W4W5_2, 12)
/* Transform 8-11 + Precalc 12-17 */
R1(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 8, 0, IW, SCHED_W_W0W1W2W3W4W5_3, 12)
R1(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 9, 1, IW, SCHED_W_W1W2W3W4W5W0_1, 15)
R1(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 10, 2, IW, SCHED_W_W1W2W3W4W5W0_2, 15)
R1(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 11, 3, IW, SCHED_W_W1W2W3W4W5W0_3, 15)
/* Transform 12-14 + Precalc 18-20 */
R1(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 12, 0, XW, SCHED_W_W2W3W4W5W0W1_1, 18)
R1(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 13, 1, XW, SCHED_W_W2W3W4W5W0W1_2, 18)
R1(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 14, 2, XW, SCHED_W_W2W3W4W5W0W1_3, 18)
/* Transform 15-17 + Precalc 21-23 */
R1(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 15, 0, XW, SCHED_W_W3W4W5W0W1W2_1, 21)
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 16, 1, XW, SCHED_W_W3W4W5W0W1W2_2, 21)
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 17, 2, XW, SCHED_W_W3W4W5W0W1W2_3, 21)
/* Transform 18-20 + Precalc 24-26 */
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 18, 0, XW, SCHED_W_W4W5W0W1W2W3_1, 24)
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 19, 1, XW, SCHED_W_W4W5W0W1W2W3_2, 24)
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 20, 2, XW, SCHED_W_W4W5W0W1W2W3_3, 24)
/* Transform 21-23 + Precalc 27-29 */
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 21, 0, XW, SCHED_W_W5W0W1W2W3W4_1, 27)
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 22, 1, XW, SCHED_W_W5W0W1W2W3W4_2, 27)
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 23, 2, XW, SCHED_W_W5W0W1W2W3W4_3, 27)
/* Transform 24-26 + Precalc 30-32 */
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 24, 0, XW, SCHED_W_W0W1W2W3W4W5_1, 30)
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 25, 1, XW, SCHED_W_W0W1W2W3W4W5_2, 30)
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 26, 2, XW, SCHED_W_W0W1W2W3W4W5_3, 30)
/* Transform 27-29 + Precalc 33-35 */
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 27, 0, XW, SCHED_W_W1W2W3W4W5W0_1, 33)
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 28, 1, XW, SCHED_W_W1W2W3W4W5W0_2, 33)
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 29, 2, XW, SCHED_W_W1W2W3W4W5W0_3, 33)
/* Transform 30-32 + Precalc 36-38 */
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 30, 0, XW, SCHED_W_W2W3W4W5W0W1_1, 36)
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 31, 1, XW, SCHED_W_W2W3W4W5W0W1_2, 36)
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 32, 2, XW, SCHED_W_W2W3W4W5W0W1_3, 36)
/* Transform 33-35 + Precalc 39-41 */
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 33, 0, XW, SCHED_W_W3W4W5W0W1W2_1, 39)
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 34, 1, XW, SCHED_W_W3W4W5W0W1W2_2, 39)
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 35, 2, XW, SCHED_W_W3W4W5W0W1W2_3, 39)
/* Transform 36-38 + Precalc 42-44 */
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 36, 0, XW, SCHED_W_W4W5W0W1W2W3_1, 42)
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 37, 1, XW, SCHED_W_W4W5W0W1W2W3_2, 42)
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 38, 2, XW, SCHED_W_W4W5W0W1W2W3_3, 42)
/* Transform 39-41 + Precalc 45-47 */
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 39, 0, XW, SCHED_W_W5W0W1W2W3W4_1, 45)
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 40, 1, XW, SCHED_W_W5W0W1W2W3W4_2, 45)
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 41, 2, XW, SCHED_W_W5W0W1W2W3W4_3, 45)
/* Transform 42-44 + Precalc 48-50 */
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 42, 0, XW, SCHED_W_W0W1W2W3W4W5_1, 48)
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 43, 1, XW, SCHED_W_W0W1W2W3W4W5_2, 48)
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 44, 2, XW, SCHED_W_W0W1W2W3W4W5_3, 48)
/* Transform 45-47 + Precalc 51-53 */
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 45, 0, XW, SCHED_W_W1W2W3W4W5W0_1, 51)
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 46, 1, XW, SCHED_W_W1W2W3W4W5W0_2, 51)
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 47, 2, XW, SCHED_W_W1W2W3W4W5W0_3, 51)
/* Transform 48-50 + Precalc 54-56 */
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 48, 0, XW, SCHED_W_W2W3W4W5W0W1_1, 54)
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 49, 1, XW, SCHED_W_W2W3W4W5W0W1_2, 54)
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 50, 2, XW, SCHED_W_W2W3W4W5W0W1_3, 54)
/* Transform 51-53 + Precalc 57-59 */
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 51, 0, XW, SCHED_W_W3W4W5W0W1W2_1, 57)
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 52, 1, XW, SCHED_W_W3W4W5W0W1W2_2, 57)
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 53, 2, XW, SCHED_W_W3W4W5W0W1W2_3, 57)
/* Transform 54-56 + Precalc 60-62 */
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 54, 0, XW, SCHED_W_W4W5W0W1W2W3_1, 60)
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 55, 1, XW, SCHED_W_W4W5W0W1W2W3_2, 60)
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 56, 2, XW, SCHED_W_W4W5W0W1W2W3_3, 60)
/* Transform 57-59 + Precalc 63 */
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 57, 0, XW, SCHED_W_W5W0W1W2W3W4_1, 63)
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 58, 1, XW, SCHED_W_W5W0W1W2W3W4_2, 63)
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 59, 2, XW, SCHED_W_W5W0W1W2W3W4_3, 63)
/* Transform 60 */
R2(ra, rb, rc, rd, re, rf, rg, rh, k_even, KL, 60, 0, XW, _, _)
subs RNBLKS, RNBLKS, #1
b.eq .Lend
/* Transform 61-63 + Preload next block */
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 61, 1, XW, LOAD_W_VEC_1, _)
ldp s0, s1, [RSTATE, #0]
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 62, 2, XW, LOAD_W_VEC_2, _)
ldp s2, s3, [RSTATE, #8]
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 63, 0, XW, LOAD_W_VEC_3, _)
/* Update the chaining variables. */
eor ra, ra, s0
eor rb, rb, s1
ldp s0, s1, [RSTATE, #16]
eor rc, rc, s2
ldp k_even, k_odd, [RSTATE, #24]
eor rd, rd, s3
eor re, re, s0
stp ra, rb, [RSTATE, #0]
eor rf, rf, s1
stp rc, rd, [RSTATE, #8]
eor rg, rg, k_even
stp re, rf, [RSTATE, #16]
eor rh, rh, k_odd
stp rg, rh, [RSTATE, #24]
b .Loop
.Lend:
/* Transform 61-63 */
R2(rd, ra, rb, rc, rh, re, rf, rg, k_odd, _, 61, 1, XW, _, _)
ldp s0, s1, [RSTATE, #0]
R2(rc, rd, ra, rb, rg, rh, re, rf, k_even, KL, 62, 2, XW, _, _)
ldp s2, s3, [RSTATE, #8]
R2(rb, rc, rd, ra, rf, rg, rh, re, k_odd, _, 63, 0, XW, _, _)
/* Update the chaining variables. */
eor ra, ra, s0
clear_vec(W0)
eor rb, rb, s1
clear_vec(W1)
ldp s0, s1, [RSTATE, #16]
clear_vec(W2)
eor rc, rc, s2
clear_vec(W3)
ldp k_even, k_odd, [RSTATE, #24]
clear_vec(W4)
eor rd, rd, s3
clear_vec(W5)
eor re, re, s0
clear_vec(XTMP0)
stp ra, rb, [RSTATE, #0]
clear_vec(XTMP1)
eor rf, rf, s1
clear_vec(XTMP2)
stp rc, rd, [RSTATE, #8]
clear_vec(XTMP3)
eor rg, rg, k_even
clear_vec(XTMP4)
stp re, rf, [RSTATE, #16]
clear_vec(XTMP5)
eor rh, rh, k_odd
clear_vec(XTMP6)
stp rg, rh, [RSTATE, #24]
/* Clear message expansion area */
add addr0, sp, #STACK_W
st1 {W0.16b-W3.16b}, [addr0], #64
st1 {W0.16b-W3.16b}, [addr0], #64
st1 {W0.16b-W3.16b}, [addr0]
mov sp, RFRAME
ldp x25, x26, [sp], #16
ldp x23, x24, [sp], #16
ldp x21, x22, [sp], #16
ldp x19, x20, [sp], #16
ldp x28, x29, [sp], #16
ret
SYM_FUNC_END(sm3_neon_transform)
.section ".rodata", "a"
.align 4
.LKtable:
.long 0x79cc4519, 0xf3988a32, 0xe7311465, 0xce6228cb
.long 0x9cc45197, 0x3988a32f, 0x7311465e, 0xe6228cbc
.long 0xcc451979, 0x988a32f3, 0x311465e7, 0x6228cbce
.long 0xc451979c, 0x88a32f39, 0x11465e73, 0x228cbce6
.long 0x9d8a7a87, 0x3b14f50f, 0x7629ea1e, 0xec53d43c
.long 0xd8a7a879, 0xb14f50f3, 0x629ea1e7, 0xc53d43ce
.long 0x8a7a879d, 0x14f50f3b, 0x29ea1e76, 0x53d43cec
.long 0xa7a879d8, 0x4f50f3b1, 0x9ea1e762, 0x3d43cec5
.long 0x7a879d8a, 0xf50f3b14, 0xea1e7629, 0xd43cec53
.long 0xa879d8a7, 0x50f3b14f, 0xa1e7629e, 0x43cec53d
.long 0x879d8a7a, 0x0f3b14f5, 0x1e7629ea, 0x3cec53d4
.long 0x79d8a7a8, 0xf3b14f50, 0xe7629ea1, 0xcec53d43
.long 0x9d8a7a87, 0x3b14f50f, 0x7629ea1e, 0xec53d43c
.long 0xd8a7a879, 0xb14f50f3, 0x629ea1e7, 0xc53d43ce
.long 0x8a7a879d, 0x14f50f3b, 0x29ea1e76, 0x53d43cec
.long 0xa7a879d8, 0x4f50f3b1, 0x9ea1e762, 0x3d43cec5
arch/arm64/crypto/sm3-neon-glue.c
+103
View File
@@ -0,0 +1,103 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* sm3-neon-glue.c - SM3 secure hash using NEON instructions
*
* Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#include <asm/neon.h>
#include <asm/simd.h>
#include <asm/unaligned.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/simd.h>
#include <crypto/sm3.h>
#include <crypto/sm3_base.h>
#include <linux/cpufeature.h>
#include <linux/crypto.h>
#include <linux/module.h>
asmlinkage void sm3_neon_transform(struct sm3_state *sst, u8 const *src,
int blocks);
static int sm3_neon_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
if (!crypto_simd_usable()) {
sm3_update(shash_desc_ctx(desc), data, len);
return 0;
}
kernel_neon_begin();
sm3_base_do_update(desc, data, len, sm3_neon_transform);
kernel_neon_end();
return 0;
}
static int sm3_neon_final(struct shash_desc *desc, u8 *out)
{
if (!crypto_simd_usable()) {
sm3_final(shash_desc_ctx(desc), out);
return 0;
}
kernel_neon_begin();
sm3_base_do_finalize(desc, sm3_neon_transform);
kernel_neon_end();
return sm3_base_finish(desc, out);
}
static int sm3_neon_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
if (!crypto_simd_usable()) {
struct sm3_state *sctx = shash_desc_ctx(desc);
if (len)
sm3_update(sctx, data, len);
sm3_final(sctx, out);
return 0;
}
kernel_neon_begin();
if (len)
sm3_base_do_update(desc, data, len, sm3_neon_transform);
sm3_base_do_finalize(desc, sm3_neon_transform);
kernel_neon_end();
return sm3_base_finish(desc, out);
}
static struct shash_alg sm3_alg = {
.digestsize = SM3_DIGEST_SIZE,
.init = sm3_base_init,
.update = sm3_neon_update,
.final = sm3_neon_final,
.finup = sm3_neon_finup,
.descsize = sizeof(struct sm3_state),
.base.cra_name = "sm3",
.base.cra_driver_name = "sm3-neon",
.base.cra_blocksize = SM3_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
.base.cra_priority = 200,
};
static int __init sm3_neon_init(void)
{
return crypto_register_shash(&sm3_alg);
}
static void __exit sm3_neon_fini(void)
{
crypto_unregister_shash(&sm3_alg);
}
module_init(sm3_neon_init);
module_exit(sm3_neon_fini);
MODULE_DESCRIPTION("SM3 secure hash using NEON instructions");
MODULE_AUTHOR("Jussi Kivilinna <jussi.kivilinna@iki.fi>");
MODULE_AUTHOR("Tianjia Zhang <tianjia.zhang@linux.alibaba.com>");
MODULE_LICENSE("GPL v2");
arch/arm64/crypto/sm4-ce-asm.h
+209
View File
@@ -0,0 +1,209 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* SM4 helper macros for Crypto Extensions
* Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#define SM4_PREPARE(ptr) \
ld1 {v24.16b-v27.16b}, [ptr], #64; \
ld1 {v28.16b-v31.16b}, [ptr];
#define SM4_CRYPT_BLK_BE(b0) \
sm4e b0.4s, v24.4s; \
sm4e b0.4s, v25.4s; \
sm4e b0.4s, v26.4s; \
sm4e b0.4s, v27.4s; \
sm4e b0.4s, v28.4s; \
sm4e b0.4s, v29.4s; \
sm4e b0.4s, v30.4s; \
sm4e b0.4s, v31.4s; \
rev64 b0.4s, b0.4s; \
ext b0.16b, b0.16b, b0.16b, #8; \
rev32 b0.16b, b0.16b;
#define SM4_CRYPT_BLK(b0) \
rev32 b0.16b, b0.16b; \
SM4_CRYPT_BLK_BE(b0);
#define SM4_CRYPT_BLK2_BE(b0, b1) \
sm4e b0.4s, v24.4s; \
sm4e b1.4s, v24.4s; \
sm4e b0.4s, v25.4s; \
sm4e b1.4s, v25.4s; \
sm4e b0.4s, v26.4s; \
sm4e b1.4s, v26.4s; \
sm4e b0.4s, v27.4s; \
sm4e b1.4s, v27.4s; \
sm4e b0.4s, v28.4s; \
sm4e b1.4s, v28.4s; \
sm4e b0.4s, v29.4s; \
sm4e b1.4s, v29.4s; \
sm4e b0.4s, v30.4s; \
sm4e b1.4s, v30.4s; \
sm4e b0.4s, v31.4s; \
sm4e b1.4s, v31.4s; \
rev64 b0.4s, b0.4s; \
rev64 b1.4s, b1.4s; \
ext b0.16b, b0.16b, b0.16b, #8; \
ext b1.16b, b1.16b, b1.16b, #8; \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
#define SM4_CRYPT_BLK2(b0, b1) \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
SM4_CRYPT_BLK2_BE(b0, b1);
#define SM4_CRYPT_BLK4_BE(b0, b1, b2, b3) \
sm4e b0.4s, v24.4s; \
sm4e b1.4s, v24.4s; \
sm4e b2.4s, v24.4s; \
sm4e b3.4s, v24.4s; \
sm4e b0.4s, v25.4s; \
sm4e b1.4s, v25.4s; \
sm4e b2.4s, v25.4s; \
sm4e b3.4s, v25.4s; \
sm4e b0.4s, v26.4s; \
sm4e b1.4s, v26.4s; \
sm4e b2.4s, v26.4s; \
sm4e b3.4s, v26.4s; \
sm4e b0.4s, v27.4s; \
sm4e b1.4s, v27.4s; \
sm4e b2.4s, v27.4s; \
sm4e b3.4s, v27.4s; \
sm4e b0.4s, v28.4s; \
sm4e b1.4s, v28.4s; \
sm4e b2.4s, v28.4s; \
sm4e b3.4s, v28.4s; \
sm4e b0.4s, v29.4s; \
sm4e b1.4s, v29.4s; \
sm4e b2.4s, v29.4s; \
sm4e b3.4s, v29.4s; \
sm4e b0.4s, v30.4s; \
sm4e b1.4s, v30.4s; \
sm4e b2.4s, v30.4s; \
sm4e b3.4s, v30.4s; \
sm4e b0.4s, v31.4s; \
sm4e b1.4s, v31.4s; \
sm4e b2.4s, v31.4s; \
sm4e b3.4s, v31.4s; \
rev64 b0.4s, b0.4s; \
rev64 b1.4s, b1.4s; \
rev64 b2.4s, b2.4s; \
rev64 b3.4s, b3.4s; \
ext b0.16b, b0.16b, b0.16b, #8; \
ext b1.16b, b1.16b, b1.16b, #8; \
ext b2.16b, b2.16b, b2.16b, #8; \
ext b3.16b, b3.16b, b3.16b, #8; \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
rev32 b3.16b, b3.16b;
#define SM4_CRYPT_BLK4(b0, b1, b2, b3) \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
rev32 b3.16b, b3.16b; \
SM4_CRYPT_BLK4_BE(b0, b1, b2, b3);
#define SM4_CRYPT_BLK8_BE(b0, b1, b2, b3, b4, b5, b6, b7) \
sm4e b0.4s, v24.4s; \
sm4e b1.4s, v24.4s; \
sm4e b2.4s, v24.4s; \
sm4e b3.4s, v24.4s; \
sm4e b4.4s, v24.4s; \
sm4e b5.4s, v24.4s; \
sm4e b6.4s, v24.4s; \
sm4e b7.4s, v24.4s; \
sm4e b0.4s, v25.4s; \
sm4e b1.4s, v25.4s; \
sm4e b2.4s, v25.4s; \
sm4e b3.4s, v25.4s; \
sm4e b4.4s, v25.4s; \
sm4e b5.4s, v25.4s; \
sm4e b6.4s, v25.4s; \
sm4e b7.4s, v25.4s; \
sm4e b0.4s, v26.4s; \
sm4e b1.4s, v26.4s; \
sm4e b2.4s, v26.4s; \
sm4e b3.4s, v26.4s; \
sm4e b4.4s, v26.4s; \
sm4e b5.4s, v26.4s; \
sm4e b6.4s, v26.4s; \
sm4e b7.4s, v26.4s; \
sm4e b0.4s, v27.4s; \
sm4e b1.4s, v27.4s; \
sm4e b2.4s, v27.4s; \
sm4e b3.4s, v27.4s; \
sm4e b4.4s, v27.4s; \
sm4e b5.4s, v27.4s; \
sm4e b6.4s, v27.4s; \
sm4e b7.4s, v27.4s; \
sm4e b0.4s, v28.4s; \
sm4e b1.4s, v28.4s; \
sm4e b2.4s, v28.4s; \
sm4e b3.4s, v28.4s; \
sm4e b4.4s, v28.4s; \
sm4e b5.4s, v28.4s; \
sm4e b6.4s, v28.4s; \
sm4e b7.4s, v28.4s; \
sm4e b0.4s, v29.4s; \
sm4e b1.4s, v29.4s; \
sm4e b2.4s, v29.4s; \
sm4e b3.4s, v29.4s; \
sm4e b4.4s, v29.4s; \
sm4e b5.4s, v29.4s; \
sm4e b6.4s, v29.4s; \
sm4e b7.4s, v29.4s; \
sm4e b0.4s, v30.4s; \
sm4e b1.4s, v30.4s; \
sm4e b2.4s, v30.4s; \
sm4e b3.4s, v30.4s; \
sm4e b4.4s, v30.4s; \
sm4e b5.4s, v30.4s; \
sm4e b6.4s, v30.4s; \
sm4e b7.4s, v30.4s; \
sm4e b0.4s, v31.4s; \
sm4e b1.4s, v31.4s; \
sm4e b2.4s, v31.4s; \
sm4e b3.4s, v31.4s; \
sm4e b4.4s, v31.4s; \
sm4e b5.4s, v31.4s; \
sm4e b6.4s, v31.4s; \
sm4e b7.4s, v31.4s; \
rev64 b0.4s, b0.4s; \
rev64 b1.4s, b1.4s; \
rev64 b2.4s, b2.4s; \
rev64 b3.4s, b3.4s; \
rev64 b4.4s, b4.4s; \
rev64 b5.4s, b5.4s; \
rev64 b6.4s, b6.4s; \
rev64 b7.4s, b7.4s; \
ext b0.16b, b0.16b, b0.16b, #8; \
ext b1.16b, b1.16b, b1.16b, #8; \
ext b2.16b, b2.16b, b2.16b, #8; \
ext b3.16b, b3.16b, b3.16b, #8; \
ext b4.16b, b4.16b, b4.16b, #8; \
ext b5.16b, b5.16b, b5.16b, #8; \
ext b6.16b, b6.16b, b6.16b, #8; \
ext b7.16b, b7.16b, b7.16b, #8; \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
rev32 b3.16b, b3.16b; \
rev32 b4.16b, b4.16b; \
rev32 b5.16b, b5.16b; \
rev32 b6.16b, b6.16b; \
rev32 b7.16b, b7.16b;
#define SM4_CRYPT_BLK8(b0, b1, b2, b3, b4, b5, b6, b7) \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
rev32 b3.16b, b3.16b; \
rev32 b4.16b, b4.16b; \
rev32 b5.16b, b5.16b; \
rev32 b6.16b, b6.16b; \
rev32 b7.16b, b7.16b; \
SM4_CRYPT_BLK8_BE(b0, b1, b2, b3, b4, b5, b6, b7);
arch/arm64/crypto/sm4-ce-ccm-core.S
+328
View File
@@ -0,0 +1,328 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* SM4-CCM AEAD Algorithm using ARMv8 Crypto Extensions
* as specified in rfc8998
* https://datatracker.ietf.org/doc/html/rfc8998
*
* Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#include <linux/linkage.h>
#include <asm/assembler.h>
#include "sm4-ce-asm.h"
.arch armv8-a+crypto
.irp b, 0, 1, 8, 9, 10, 11, 12, 13, 14, 15, 16, 24, 25, 26, 27, 28, 29, 30, 31
.set .Lv\b\().4s, \b
.endr
.macro sm4e, vd, vn
.inst 0xcec08400 | (.L\vn << 5) | .L\vd
.endm
/* Register macros */
#define RMAC v16
/* Helper macros. */
#define inc_le128(vctr) \
mov vctr.d[1], x8; \
mov vctr.d[0], x7; \
adds x8, x8, #1; \
rev64 vctr.16b, vctr.16b; \
adc x7, x7, xzr;
.align 3
SYM_FUNC_START(sm4_ce_cbcmac_update)
/* input:
* x0: round key array, CTX
* x1: mac
* x2: src
* w3: nblocks
*/
SM4_PREPARE(x0)
ld1 {RMAC.16b}, [x1]
.Lcbcmac_loop_4x:
cmp w3, #4
blt .Lcbcmac_loop_1x
sub w3, w3, #4
ld1 {v0.16b-v3.16b}, [x2], #64
SM4_CRYPT_BLK(RMAC)
eor RMAC.16b, RMAC.16b, v0.16b
SM4_CRYPT_BLK(RMAC)
eor RMAC.16b, RMAC.16b, v1.16b
SM4_CRYPT_BLK(RMAC)
eor RMAC.16b, RMAC.16b, v2.16b
SM4_CRYPT_BLK(RMAC)
eor RMAC.16b, RMAC.16b, v3.16b
cbz w3, .Lcbcmac_end
b .Lcbcmac_loop_4x
.Lcbcmac_loop_1x:
sub w3, w3, #1
ld1 {v0.16b}, [x2], #16
SM4_CRYPT_BLK(RMAC)
eor RMAC.16b, RMAC.16b, v0.16b
cbnz w3, .Lcbcmac_loop_1x
.Lcbcmac_end:
st1 {RMAC.16b}, [x1]
ret
SYM_FUNC_END(sm4_ce_cbcmac_update)
.align 3
SYM_FUNC_START(sm4_ce_ccm_final)
/* input:
* x0: round key array, CTX
* x1: ctr0 (big endian, 128 bit)
* x2: mac
*/
SM4_PREPARE(x0)
ld1 {RMAC.16b}, [x2]
ld1 {v0.16b}, [x1]
SM4_CRYPT_BLK2(RMAC, v0)
/* en-/decrypt the mac with ctr0 */
eor RMAC.16b, RMAC.16b, v0.16b
st1 {RMAC.16b}, [x2]
ret
SYM_FUNC_END(sm4_ce_ccm_final)
.align 3
SYM_FUNC_START(sm4_ce_ccm_enc)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* x3: ctr (big endian, 128 bit)
* w4: nbytes
* x5: mac
*/
SM4_PREPARE(x0)
ldp x7, x8, [x3]
rev x7, x7
rev x8, x8
ld1 {RMAC.16b}, [x5]
.Lccm_enc_loop_4x:
cmp w4, #(4 * 16)
blt .Lccm_enc_loop_1x
sub w4, w4, #(4 * 16)
/* construct CTRs */
inc_le128(v8) /* +0 */
inc_le128(v9) /* +1 */
inc_le128(v10) /* +2 */
inc_le128(v11) /* +3 */
ld1 {v0.16b-v3.16b}, [x2], #64
SM4_CRYPT_BLK2(v8, RMAC)
eor v8.16b, v8.16b, v0.16b
eor RMAC.16b, RMAC.16b, v0.16b
SM4_CRYPT_BLK2(v9, RMAC)
eor v9.16b, v9.16b, v1.16b
eor RMAC.16b, RMAC.16b, v1.16b
SM4_CRYPT_BLK2(v10, RMAC)
eor v10.16b, v10.16b, v2.16b
eor RMAC.16b, RMAC.16b, v2.16b
SM4_CRYPT_BLK2(v11, RMAC)
eor v11.16b, v11.16b, v3.16b
eor RMAC.16b, RMAC.16b, v3.16b
st1 {v8.16b-v11.16b}, [x1], #64
cbz w4, .Lccm_enc_end
b .Lccm_enc_loop_4x
.Lccm_enc_loop_1x:
cmp w4, #16
blt .Lccm_enc_tail
sub w4, w4, #16
/* construct CTRs */
inc_le128(v8)
ld1 {v0.16b}, [x2], #16
SM4_CRYPT_BLK2(v8, RMAC)
eor v8.16b, v8.16b, v0.16b
eor RMAC.16b, RMAC.16b, v0.16b
st1 {v8.16b}, [x1], #16
cbz w4, .Lccm_enc_end
b .Lccm_enc_loop_1x
.Lccm_enc_tail:
/* construct CTRs */
inc_le128(v8)
SM4_CRYPT_BLK2(RMAC, v8)
/* store new MAC */
st1 {RMAC.16b}, [x5]
.Lccm_enc_tail_loop:
ldrb w0, [x2], #1 /* get 1 byte from input */
umov w9, v8.b[0] /* get top crypted CTR byte */
umov w6, RMAC.b[0] /* get top MAC byte */
eor w9, w9, w0 /* w9 = CTR ^ input */
eor w6, w6, w0 /* w6 = MAC ^ input */
strb w9, [x1], #1 /* store out byte */
strb w6, [x5], #1 /* store MAC byte */
subs w4, w4, #1
beq .Lccm_enc_ret
/* shift out one byte */
ext RMAC.16b, RMAC.16b, RMAC.16b, #1
ext v8.16b, v8.16b, v8.16b, #1
b .Lccm_enc_tail_loop
.Lccm_enc_end:
/* store new MAC */
st1 {RMAC.16b}, [x5]
/* store new CTR */
rev x7, x7
rev x8, x8
stp x7, x8, [x3]
.Lccm_enc_ret:
ret
SYM_FUNC_END(sm4_ce_ccm_enc)
.align 3
SYM_FUNC_START(sm4_ce_ccm_dec)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* x3: ctr (big endian, 128 bit)
* w4: nbytes
* x5: mac
*/
SM4_PREPARE(x0)
ldp x7, x8, [x3]
rev x7, x7
rev x8, x8
ld1 {RMAC.16b}, [x5]
.Lccm_dec_loop_4x:
cmp w4, #(4 * 16)
blt .Lccm_dec_loop_1x
sub w4, w4, #(4 * 16)
/* construct CTRs */
inc_le128(v8) /* +0 */
inc_le128(v9) /* +1 */
inc_le128(v10) /* +2 */
inc_le128(v11) /* +3 */
ld1 {v0.16b-v3.16b}, [x2], #64
SM4_CRYPT_BLK2(v8, RMAC)
eor v8.16b, v8.16b, v0.16b
eor RMAC.16b, RMAC.16b, v8.16b
SM4_CRYPT_BLK2(v9, RMAC)
eor v9.16b, v9.16b, v1.16b
eor RMAC.16b, RMAC.16b, v9.16b
SM4_CRYPT_BLK2(v10, RMAC)
eor v10.16b, v10.16b, v2.16b
eor RMAC.16b, RMAC.16b, v10.16b
SM4_CRYPT_BLK2(v11, RMAC)
eor v11.16b, v11.16b, v3.16b
eor RMAC.16b, RMAC.16b, v11.16b
st1 {v8.16b-v11.16b}, [x1], #64
cbz w4, .Lccm_dec_end
b .Lccm_dec_loop_4x
.Lccm_dec_loop_1x:
cmp w4, #16
blt .Lccm_dec_tail
sub w4, w4, #16
/* construct CTRs */
inc_le128(v8)
ld1 {v0.16b}, [x2], #16
SM4_CRYPT_BLK2(v8, RMAC)
eor v8.16b, v8.16b, v0.16b
eor RMAC.16b, RMAC.16b, v8.16b
st1 {v8.16b}, [x1], #16
cbz w4, .Lccm_dec_end
b .Lccm_dec_loop_1x
.Lccm_dec_tail:
/* construct CTRs */
inc_le128(v8)
SM4_CRYPT_BLK2(RMAC, v8)
/* store new MAC */
st1 {RMAC.16b}, [x5]
.Lccm_dec_tail_loop:
ldrb w0, [x2], #1 /* get 1 byte from input */
umov w9, v8.b[0] /* get top crypted CTR byte */
umov w6, RMAC.b[0] /* get top MAC byte */
eor w9, w9, w0 /* w9 = CTR ^ input */
eor w6, w6, w9 /* w6 = MAC ^ output */
strb w9, [x1], #1 /* store out byte */
strb w6, [x5], #1 /* store MAC byte */
subs w4, w4, #1
beq .Lccm_dec_ret
/* shift out one byte */
ext RMAC.16b, RMAC.16b, RMAC.16b, #1
ext v8.16b, v8.16b, v8.16b, #1
b .Lccm_dec_tail_loop
.Lccm_dec_end:
/* store new MAC */
st1 {RMAC.16b}, [x5]
/* store new CTR */
rev x7, x7
rev x8, x8
stp x7, x8, [x3]
.Lccm_dec_ret:
ret
SYM_FUNC_END(sm4_ce_ccm_dec)
arch/arm64/crypto/sm4-ce-ccm-glue.c
+303
View File
@@ -0,0 +1,303 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* SM4-CCM AEAD Algorithm using ARMv8 Crypto Extensions
* as specified in rfc8998
* https://datatracker.ietf.org/doc/html/rfc8998
*
* Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#include <linux/module.h>
#include <linux/crypto.h>
#include <linux/kernel.h>
#include <linux/cpufeature.h>
#include <asm/neon.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/skcipher.h>
#include <crypto/sm4.h>
#include "sm4-ce.h"
asmlinkage void sm4_ce_cbcmac_update(const u32 *rkey_enc, u8 *mac,
const u8 *src, unsigned int nblocks);
asmlinkage void sm4_ce_ccm_enc(const u32 *rkey_enc, u8 *dst, const u8 *src,
u8 *iv, unsigned int nbytes, u8 *mac);
asmlinkage void sm4_ce_ccm_dec(const u32 *rkey_enc, u8 *dst, const u8 *src,
u8 *iv, unsigned int nbytes, u8 *mac);
asmlinkage void sm4_ce_ccm_final(const u32 *rkey_enc, u8 *iv, u8 *mac);
static int ccm_setkey(struct crypto_aead *tfm, const u8 *key,
unsigned int key_len)
{
struct sm4_ctx *ctx = crypto_aead_ctx(tfm);
if (key_len != SM4_KEY_SIZE)
return -EINVAL;
kernel_neon_begin();
sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec,
crypto_sm4_fk, crypto_sm4_ck);
kernel_neon_end();
return 0;
}
static int ccm_setauthsize(struct crypto_aead *tfm, unsigned int authsize)
{
if ((authsize & 1) || authsize < 4)
return -EINVAL;
return 0;
}
static int ccm_format_input(u8 info[], struct aead_request *req,
unsigned int msglen)
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
unsigned int l = req->iv[0] + 1;
unsigned int m;
__be32 len;
/* verify that CCM dimension 'L': 2 <= L <= 8 */
if (l < 2 || l > 8)
return -EINVAL;
if (l < 4 && msglen >> (8 * l))
return -EOVERFLOW;
memset(&req->iv[SM4_BLOCK_SIZE - l], 0, l);
memcpy(info, req->iv, SM4_BLOCK_SIZE);
m = crypto_aead_authsize(aead);
/* format flags field per RFC 3610/NIST 800-38C */
*info |= ((m - 2) / 2) << 3;
if (req->assoclen)
*info |= (1 << 6);
/*
* format message length field,
* Linux uses a u32 type to represent msglen
*/
if (l >= 4)
l = 4;
len = cpu_to_be32(msglen);
memcpy(&info[SM4_BLOCK_SIZE - l], (u8 *)&len + 4 - l, l);
return 0;
}
static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[])
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
struct sm4_ctx *ctx = crypto_aead_ctx(aead);
struct __packed { __be16 l; __be32 h; } aadlen;
u32 assoclen = req->assoclen;
struct scatter_walk walk;
unsigned int len;
if (assoclen < 0xff00) {
aadlen.l = cpu_to_be16(assoclen);
len = 2;
} else {
aadlen.l = cpu_to_be16(0xfffe);
put_unaligned_be32(assoclen, &aadlen.h);
len = 6;
}
sm4_ce_crypt_block(ctx->rkey_enc, mac, mac);
crypto_xor(mac, (const u8 *)&aadlen, len);
scatterwalk_start(&walk, req->src);
do {
u32 n = scatterwalk_clamp(&walk, assoclen);
u8 *p, *ptr;
if (!n) {
scatterwalk_start(&walk, sg_next(walk.sg));
n = scatterwalk_clamp(&walk, assoclen);
}
p = ptr = scatterwalk_map(&walk);
assoclen -= n;
scatterwalk_advance(&walk, n);
while (n > 0) {
unsigned int l, nblocks;
if (len == SM4_BLOCK_SIZE) {
if (n < SM4_BLOCK_SIZE) {
sm4_ce_crypt_block(ctx->rkey_enc,
mac, mac);
len = 0;
} else {
nblocks = n / SM4_BLOCK_SIZE;
sm4_ce_cbcmac_update(ctx->rkey_enc,
mac, ptr, nblocks);
ptr += nblocks * SM4_BLOCK_SIZE;
n %= SM4_BLOCK_SIZE;
continue;
}
}
l = min(n, SM4_BLOCK_SIZE - len);
if (l) {
crypto_xor(mac + len, ptr, l);
len += l;
ptr += l;
n -= l;
}
}
scatterwalk_unmap(p);
scatterwalk_done(&walk, 0, assoclen);
} while (assoclen);
}
static int ccm_crypt(struct aead_request *req, struct skcipher_walk *walk,
u32 *rkey_enc, u8 mac[],
void (*sm4_ce_ccm_crypt)(const u32 *rkey_enc, u8 *dst,
const u8 *src, u8 *iv,
unsigned int nbytes, u8 *mac))
{
u8 __aligned(8) ctr0[SM4_BLOCK_SIZE];
int err;
/* preserve the initial ctr0 for the TAG */
memcpy(ctr0, walk->iv, SM4_BLOCK_SIZE);
crypto_inc(walk->iv, SM4_BLOCK_SIZE);
kernel_neon_begin();
if (req->assoclen)
ccm_calculate_auth_mac(req, mac);
do {
unsigned int tail = walk->nbytes % SM4_BLOCK_SIZE;
const u8 *src = walk->src.virt.addr;
u8 *dst = walk->dst.virt.addr;
if (walk->nbytes == walk->total)
tail = 0;
if (walk->nbytes - tail)
sm4_ce_ccm_crypt(rkey_enc, dst, src, walk->iv,
walk->nbytes - tail, mac);
if (walk->nbytes == walk->total)
sm4_ce_ccm_final(rkey_enc, ctr0, mac);
kernel_neon_end();
if (walk->nbytes) {
err = skcipher_walk_done(walk, tail);
if (err)
return err;
if (walk->nbytes)
kernel_neon_begin();
}
} while (walk->nbytes > 0);
return 0;
}
static int ccm_encrypt(struct aead_request *req)
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
struct sm4_ctx *ctx = crypto_aead_ctx(aead);
u8 __aligned(8) mac[SM4_BLOCK_SIZE];
struct skcipher_walk walk;
int err;
err = ccm_format_input(mac, req, req->cryptlen);
if (err)
return err;
err = skcipher_walk_aead_encrypt(&walk, req, false);
if (err)
return err;
err = ccm_crypt(req, &walk, ctx->rkey_enc, mac, sm4_ce_ccm_enc);
if (err)
return err;
/* copy authtag to end of dst */
scatterwalk_map_and_copy(mac, req->dst, req->assoclen + req->cryptlen,
crypto_aead_authsize(aead), 1);
return 0;
}
static int ccm_decrypt(struct aead_request *req)
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
unsigned int authsize = crypto_aead_authsize(aead);
struct sm4_ctx *ctx = crypto_aead_ctx(aead);
u8 __aligned(8) mac[SM4_BLOCK_SIZE];
u8 authtag[SM4_BLOCK_SIZE];
struct skcipher_walk walk;
int err;
err = ccm_format_input(mac, req, req->cryptlen - authsize);
if (err)
return err;
err = skcipher_walk_aead_decrypt(&walk, req, false);
if (err)
return err;
err = ccm_crypt(req, &walk, ctx->rkey_enc, mac, sm4_ce_ccm_dec);
if (err)
return err;
/* compare calculated auth tag with the stored one */
scatterwalk_map_and_copy(authtag, req->src,
req->assoclen + req->cryptlen - authsize,
authsize, 0);
if (crypto_memneq(authtag, mac, authsize))
return -EBADMSG;
return 0;
}
static struct aead_alg sm4_ccm_alg = {
.base = {
.cra_name = "ccm(sm4)",
.cra_driver_name = "ccm-sm4-ce",
.cra_priority = 400,
.cra_blocksize = 1,
.cra_ctxsize = sizeof(struct sm4_ctx),
.cra_module = THIS_MODULE,
},
.ivsize = SM4_BLOCK_SIZE,
.chunksize = SM4_BLOCK_SIZE,
.maxauthsize = SM4_BLOCK_SIZE,
.setkey = ccm_setkey,
.setauthsize = ccm_setauthsize,
.encrypt = ccm_encrypt,
.decrypt = ccm_decrypt,
};
static int __init sm4_ce_ccm_init(void)
{
return crypto_register_aead(&sm4_ccm_alg);
}
static void __exit sm4_ce_ccm_exit(void)
{
crypto_unregister_aead(&sm4_ccm_alg);
}
module_cpu_feature_match(SM4, sm4_ce_ccm_init);
module_exit(sm4_ce_ccm_exit);
MODULE_DESCRIPTION("Synchronous SM4 in CCM mode using ARMv8 Crypto Extensions");
MODULE_ALIAS_CRYPTO("ccm(sm4)");
MODULE_AUTHOR("Tianjia Zhang <tianjia.zhang@linux.alibaba.com>");
MODULE_LICENSE("GPL v2");
arch/arm64/crypto/sm4-ce-cipher-glue.c
+1 -1
View File
@@ -2,11 +2,11 @@
#include <asm/neon.h>
#include <asm/simd.h>
#include <crypto/algapi.h>
#include <crypto/sm4.h>
#include <crypto/internal/simd.h>
#include <linux/module.h>
#include <linux/cpufeature.h>
#include <linux/crypto.h>
#include <linux/types.h>
MODULE_ALIAS_CRYPTO("sm4");
arch/arm64/crypto/sm4-ce-core.S
+815 -382
View File
File diff suppressed because it is too large Load Diff
arch/arm64/crypto/sm4-ce-gcm-core.S
+741
View File
@@ -0,0 +1,741 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* SM4-GCM AEAD Algorithm using ARMv8 Crypto Extensions
* as specified in rfc8998
* https://datatracker.ietf.org/doc/html/rfc8998
*
* Copyright (C) 2016 Jussi Kivilinna <jussi.kivilinna@iki.fi>
* Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#include <linux/linkage.h>
#include <asm/assembler.h>
#include "sm4-ce-asm.h"
.arch armv8-a+crypto
.irp b, 0, 1, 2, 3, 24, 25, 26, 27, 28, 29, 30, 31
.set .Lv\b\().4s, \b
.endr
.macro sm4e, vd, vn
.inst 0xcec08400 | (.L\vn << 5) | .L\vd
.endm
/* Register macros */
/* Used for both encryption and decryption */
#define RHASH v21
#define RRCONST v22
#define RZERO v23
/* Helper macros. */
/*
* input: m0, m1
* output: r0:r1 (low 128-bits in r0, high in r1)
*/
#define PMUL_128x128(r0, r1, m0, m1, T0, T1) \
ext T0.16b, m1.16b, m1.16b, #8; \
pmull r0.1q, m0.1d, m1.1d; \
pmull T1.1q, m0.1d, T0.1d; \
pmull2 T0.1q, m0.2d, T0.2d; \
pmull2 r1.1q, m0.2d, m1.2d; \
eor T0.16b, T0.16b, T1.16b; \
ext T1.16b, RZERO.16b, T0.16b, #8; \
ext T0.16b, T0.16b, RZERO.16b, #8; \
eor r0.16b, r0.16b, T1.16b; \
eor r1.16b, r1.16b, T0.16b;
#define PMUL_128x128_4x(r0, r1, m0, m1, T0, T1, \
r2, r3, m2, m3, T2, T3, \
r4, r5, m4, m5, T4, T5, \
r6, r7, m6, m7, T6, T7) \
ext T0.16b, m1.16b, m1.16b, #8; \
ext T2.16b, m3.16b, m3.16b, #8; \
ext T4.16b, m5.16b, m5.16b, #8; \
ext T6.16b, m7.16b, m7.16b, #8; \
pmull r0.1q, m0.1d, m1.1d; \
pmull r2.1q, m2.1d, m3.1d; \
pmull r4.1q, m4.1d, m5.1d; \
pmull r6.1q, m6.1d, m7.1d; \
pmull T1.1q, m0.1d, T0.1d; \
pmull T3.1q, m2.1d, T2.1d; \
pmull T5.1q, m4.1d, T4.1d; \
pmull T7.1q, m6.1d, T6.1d; \
pmull2 T0.1q, m0.2d, T0.2d; \
pmull2 T2.1q, m2.2d, T2.2d; \
pmull2 T4.1q, m4.2d, T4.2d; \
pmull2 T6.1q, m6.2d, T6.2d; \
pmull2 r1.1q, m0.2d, m1.2d; \
pmull2 r3.1q, m2.2d, m3.2d; \
pmull2 r5.1q, m4.2d, m5.2d; \
pmull2 r7.1q, m6.2d, m7.2d; \
eor T0.16b, T0.16b, T1.16b; \
eor T2.16b, T2.16b, T3.16b; \
eor T4.16b, T4.16b, T5.16b; \
eor T6.16b, T6.16b, T7.16b; \
ext T1.16b, RZERO.16b, T0.16b, #8; \
ext T3.16b, RZERO.16b, T2.16b, #8; \
ext T5.16b, RZERO.16b, T4.16b, #8; \
ext T7.16b, RZERO.16b, T6.16b, #8; \
ext T0.16b, T0.16b, RZERO.16b, #8; \
ext T2.16b, T2.16b, RZERO.16b, #8; \
ext T4.16b, T4.16b, RZERO.16b, #8; \
ext T6.16b, T6.16b, RZERO.16b, #8; \
eor r0.16b, r0.16b, T1.16b; \
eor r2.16b, r2.16b, T3.16b; \
eor r4.16b, r4.16b, T5.16b; \
eor r6.16b, r6.16b, T7.16b; \
eor r1.16b, r1.16b, T0.16b; \
eor r3.16b, r3.16b, T2.16b; \
eor r5.16b, r5.16b, T4.16b; \
eor r7.16b, r7.16b, T6.16b;
/*
* input: r0:r1 (low 128-bits in r0, high in r1)
* output: a
*/
#define REDUCTION(a, r0, r1, rconst, T0, T1) \
pmull2 T0.1q, r1.2d, rconst.2d; \
ext T1.16b, T0.16b, RZERO.16b, #8; \
ext T0.16b, RZERO.16b, T0.16b, #8; \
eor r1.16b, r1.16b, T1.16b; \
eor r0.16b, r0.16b, T0.16b; \
pmull T0.1q, r1.1d, rconst.1d; \
eor a.16b, r0.16b, T0.16b;
#define SM4_CRYPT_PMUL_128x128_BLK(b0, r0, r1, m0, m1, T0, T1) \
rev32 b0.16b, b0.16b; \
ext T0.16b, m1.16b, m1.16b, #8; \
sm4e b0.4s, v24.4s; \
pmull r0.1q, m0.1d, m1.1d; \
sm4e b0.4s, v25.4s; \
pmull T1.1q, m0.1d, T0.1d; \
sm4e b0.4s, v26.4s; \
pmull2 T0.1q, m0.2d, T0.2d; \
sm4e b0.4s, v27.4s; \
pmull2 r1.1q, m0.2d, m1.2d; \
sm4e b0.4s, v28.4s; \
eor T0.16b, T0.16b, T1.16b; \
sm4e b0.4s, v29.4s; \
ext T1.16b, RZERO.16b, T0.16b, #8; \
sm4e b0.4s, v30.4s; \
ext T0.16b, T0.16b, RZERO.16b, #8; \
sm4e b0.4s, v31.4s; \
eor r0.16b, r0.16b, T1.16b; \
rev64 b0.4s, b0.4s; \
eor r1.16b, r1.16b, T0.16b; \
ext b0.16b, b0.16b, b0.16b, #8; \
rev32 b0.16b, b0.16b;
#define SM4_CRYPT_PMUL_128x128_BLK3(b0, b1, b2, \
r0, r1, m0, m1, T0, T1, \
r2, r3, m2, m3, T2, T3, \
r4, r5, m4, m5, T4, T5) \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
ext T0.16b, m1.16b, m1.16b, #8; \
ext T2.16b, m3.16b, m3.16b, #8; \
ext T4.16b, m5.16b, m5.16b, #8; \
sm4e b0.4s, v24.4s; \
sm4e b1.4s, v24.4s; \
sm4e b2.4s, v24.4s; \
pmull r0.1q, m0.1d, m1.1d; \
pmull r2.1q, m2.1d, m3.1d; \
pmull r4.1q, m4.1d, m5.1d; \
sm4e b0.4s, v25.4s; \
sm4e b1.4s, v25.4s; \
sm4e b2.4s, v25.4s; \
pmull T1.1q, m0.1d, T0.1d; \
pmull T3.1q, m2.1d, T2.1d; \
pmull T5.1q, m4.1d, T4.1d; \
sm4e b0.4s, v26.4s; \
sm4e b1.4s, v26.4s; \
sm4e b2.4s, v26.4s; \
pmull2 T0.1q, m0.2d, T0.2d; \
pmull2 T2.1q, m2.2d, T2.2d; \
pmull2 T4.1q, m4.2d, T4.2d; \
sm4e b0.4s, v27.4s; \
sm4e b1.4s, v27.4s; \
sm4e b2.4s, v27.4s; \
pmull2 r1.1q, m0.2d, m1.2d; \
pmull2 r3.1q, m2.2d, m3.2d; \
pmull2 r5.1q, m4.2d, m5.2d; \
sm4e b0.4s, v28.4s; \
sm4e b1.4s, v28.4s; \
sm4e b2.4s, v28.4s; \
eor T0.16b, T0.16b, T1.16b; \
eor T2.16b, T2.16b, T3.16b; \
eor T4.16b, T4.16b, T5.16b; \
sm4e b0.4s, v29.4s; \
sm4e b1.4s, v29.4s; \
sm4e b2.4s, v29.4s; \
ext T1.16b, RZERO.16b, T0.16b, #8; \
ext T3.16b, RZERO.16b, T2.16b, #8; \
ext T5.16b, RZERO.16b, T4.16b, #8; \
sm4e b0.4s, v30.4s; \
sm4e b1.4s, v30.4s; \
sm4e b2.4s, v30.4s; \
ext T0.16b, T0.16b, RZERO.16b, #8; \
ext T2.16b, T2.16b, RZERO.16b, #8; \
ext T4.16b, T4.16b, RZERO.16b, #8; \
sm4e b0.4s, v31.4s; \
sm4e b1.4s, v31.4s; \
sm4e b2.4s, v31.4s; \
eor r0.16b, r0.16b, T1.16b; \
eor r2.16b, r2.16b, T3.16b; \
eor r4.16b, r4.16b, T5.16b; \
rev64 b0.4s, b0.4s; \
rev64 b1.4s, b1.4s; \
rev64 b2.4s, b2.4s; \
eor r1.16b, r1.16b, T0.16b; \
eor r3.16b, r3.16b, T2.16b; \
eor r5.16b, r5.16b, T4.16b; \
ext b0.16b, b0.16b, b0.16b, #8; \
ext b1.16b, b1.16b, b1.16b, #8; \
ext b2.16b, b2.16b, b2.16b, #8; \
eor r0.16b, r0.16b, r2.16b; \
eor r1.16b, r1.16b, r3.16b; \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
eor r0.16b, r0.16b, r4.16b; \
eor r1.16b, r1.16b, r5.16b;
#define inc32_le128(vctr) \
mov vctr.d[1], x9; \
add w6, w9, #1; \
mov vctr.d[0], x8; \
bfi x9, x6, #0, #32; \
rev64 vctr.16b, vctr.16b;
#define GTAG_HASH_LENGTHS(vctr0, vlen) \
ld1 {vlen.16b}, [x7]; \
/* construct CTR0 */ \
/* the lower 32-bits of initial IV is always be32(1) */ \
mov x6, #0x1; \
bfi x9, x6, #0, #32; \
mov vctr0.d[0], x8; \
mov vctr0.d[1], x9; \
rbit vlen.16b, vlen.16b; \
rev64 vctr0.16b, vctr0.16b; \
/* authtag = GCTR(CTR0, GHASH) */ \
eor RHASH.16b, RHASH.16b, vlen.16b; \
SM4_CRYPT_PMUL_128x128_BLK(vctr0, RR0, RR1, RHASH, RH1, \
RTMP0, RTMP1); \
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP2, RTMP3); \
rbit RHASH.16b, RHASH.16b; \
eor RHASH.16b, RHASH.16b, vctr0.16b;
/* Register macros for encrypt and ghash */
/* can be the same as input v0-v3 */
#define RR1 v0
#define RR3 v1
#define RR5 v2
#define RR7 v3
#define RR0 v4
#define RR2 v5
#define RR4 v6
#define RR6 v7
#define RTMP0 v8
#define RTMP1 v9
#define RTMP2 v10
#define RTMP3 v11
#define RTMP4 v12
#define RTMP5 v13
#define RTMP6 v14
#define RTMP7 v15
#define RH1 v16
#define RH2 v17
#define RH3 v18
#define RH4 v19
.align 3
SYM_FUNC_START(sm4_ce_pmull_ghash_setup)
/* input:
* x0: round key array, CTX
* x1: ghash table
*/
SM4_PREPARE(x0)
adr_l x2, .Lghash_rconst
ld1r {RRCONST.2d}, [x2]
eor RZERO.16b, RZERO.16b, RZERO.16b
/* H = E(K, 0^128) */
rev32 v0.16b, RZERO.16b
SM4_CRYPT_BLK_BE(v0)
/* H ^ 1 */
rbit RH1.16b, v0.16b
/* H ^ 2 */
PMUL_128x128(RR0, RR1, RH1, RH1, RTMP0, RTMP1)
REDUCTION(RH2, RR0, RR1, RRCONST, RTMP2, RTMP3)
/* H ^ 3 */
PMUL_128x128(RR0, RR1, RH2, RH1, RTMP0, RTMP1)
REDUCTION(RH3, RR0, RR1, RRCONST, RTMP2, RTMP3)
/* H ^ 4 */
PMUL_128x128(RR0, RR1, RH2, RH2, RTMP0, RTMP1)
REDUCTION(RH4, RR0, RR1, RRCONST, RTMP2, RTMP3)
st1 {RH1.16b-RH4.16b}, [x1]
ret
SYM_FUNC_END(sm4_ce_pmull_ghash_setup)
.align 3
SYM_FUNC_START(pmull_ghash_update)
/* input:
* x0: ghash table
* x1: ghash result
* x2: src
* w3: nblocks
*/
ld1 {RH1.16b-RH4.16b}, [x0]
ld1 {RHASH.16b}, [x1]
rbit RHASH.16b, RHASH.16b
adr_l x4, .Lghash_rconst
ld1r {RRCONST.2d}, [x4]
eor RZERO.16b, RZERO.16b, RZERO.16b
.Lghash_loop_4x:
cmp w3, #4
blt .Lghash_loop_1x
sub w3, w3, #4
ld1 {v0.16b-v3.16b}, [x2], #64
rbit v0.16b, v0.16b
rbit v1.16b, v1.16b
rbit v2.16b, v2.16b
rbit v3.16b, v3.16b
/*
* (in0 ^ HASH) * H^4 => rr0:rr1
* (in1) * H^3 => rr2:rr3
* (in2) * H^2 => rr4:rr5
* (in3) * H^1 => rr6:rr7
*/
eor RHASH.16b, RHASH.16b, v0.16b
PMUL_128x128_4x(RR0, RR1, RHASH, RH4, RTMP0, RTMP1,
RR2, RR3, v1, RH3, RTMP2, RTMP3,
RR4, RR5, v2, RH2, RTMP4, RTMP5,
RR6, RR7, v3, RH1, RTMP6, RTMP7)
eor RR0.16b, RR0.16b, RR2.16b
eor RR1.16b, RR1.16b, RR3.16b
eor RR0.16b, RR0.16b, RR4.16b
eor RR1.16b, RR1.16b, RR5.16b
eor RR0.16b, RR0.16b, RR6.16b
eor RR1.16b, RR1.16b, RR7.16b
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP0, RTMP1)
cbz w3, .Lghash_end
b .Lghash_loop_4x
.Lghash_loop_1x:
sub w3, w3, #1
ld1 {v0.16b}, [x2], #16
rbit v0.16b, v0.16b
eor RHASH.16b, RHASH.16b, v0.16b
PMUL_128x128(RR0, RR1, RHASH, RH1, RTMP0, RTMP1)
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP2, RTMP3)
cbnz w3, .Lghash_loop_1x
.Lghash_end:
rbit RHASH.16b, RHASH.16b
st1 {RHASH.2d}, [x1]
ret
SYM_FUNC_END(pmull_ghash_update)
.align 3
SYM_FUNC_START(sm4_ce_pmull_gcm_enc)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* x3: ctr (big endian, 128 bit)
* w4: nbytes
* x5: ghash result
* x6: ghash table
* x7: lengths (only for last block)
*/
SM4_PREPARE(x0)
ldp x8, x9, [x3]
rev x8, x8
rev x9, x9
ld1 {RH1.16b-RH4.16b}, [x6]
ld1 {RHASH.16b}, [x5]
rbit RHASH.16b, RHASH.16b
adr_l x6, .Lghash_rconst
ld1r {RRCONST.2d}, [x6]
eor RZERO.16b, RZERO.16b, RZERO.16b
cbz w4, .Lgcm_enc_hash_len
.Lgcm_enc_loop_4x:
cmp w4, #(4 * 16)
blt .Lgcm_enc_loop_1x
sub w4, w4, #(4 * 16)
/* construct CTRs */
inc32_le128(v0) /* +0 */
inc32_le128(v1) /* +1 */
inc32_le128(v2) /* +2 */
inc32_le128(v3) /* +3 */
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64
SM4_CRYPT_BLK4(v0, v1, v2, v3)
eor v0.16b, v0.16b, RTMP0.16b
eor v1.16b, v1.16b, RTMP1.16b
eor v2.16b, v2.16b, RTMP2.16b
eor v3.16b, v3.16b, RTMP3.16b
st1 {v0.16b-v3.16b}, [x1], #64
/* ghash update */
rbit v0.16b, v0.16b
rbit v1.16b, v1.16b
rbit v2.16b, v2.16b
rbit v3.16b, v3.16b
/*
* (in0 ^ HASH) * H^4 => rr0:rr1
* (in1) * H^3 => rr2:rr3
* (in2) * H^2 => rr4:rr5
* (in3) * H^1 => rr6:rr7
*/
eor RHASH.16b, RHASH.16b, v0.16b
PMUL_128x128_4x(RR0, RR1, RHASH, RH4, RTMP0, RTMP1,
RR2, RR3, v1, RH3, RTMP2, RTMP3,
RR4, RR5, v2, RH2, RTMP4, RTMP5,
RR6, RR7, v3, RH1, RTMP6, RTMP7)
eor RR0.16b, RR0.16b, RR2.16b
eor RR1.16b, RR1.16b, RR3.16b
eor RR0.16b, RR0.16b, RR4.16b
eor RR1.16b, RR1.16b, RR5.16b
eor RR0.16b, RR0.16b, RR6.16b
eor RR1.16b, RR1.16b, RR7.16b
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP0, RTMP1)
cbz w4, .Lgcm_enc_hash_len
b .Lgcm_enc_loop_4x
.Lgcm_enc_loop_1x:
cmp w4, #16
blt .Lgcm_enc_tail
sub w4, w4, #16
/* construct CTRs */
inc32_le128(v0)
ld1 {RTMP0.16b}, [x2], #16
SM4_CRYPT_BLK(v0)
eor v0.16b, v0.16b, RTMP0.16b
st1 {v0.16b}, [x1], #16
/* ghash update */
rbit v0.16b, v0.16b
eor RHASH.16b, RHASH.16b, v0.16b
PMUL_128x128(RR0, RR1, RHASH, RH1, RTMP0, RTMP1)
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP2, RTMP3)
cbz w4, .Lgcm_enc_hash_len
b .Lgcm_enc_loop_1x
.Lgcm_enc_tail:
/* construct CTRs */
inc32_le128(v0)
SM4_CRYPT_BLK(v0)
/* load permute table */
adr_l x0, .Lcts_permute_table
add x0, x0, #32
sub x0, x0, w4, uxtw
ld1 {v3.16b}, [x0]
.Lgcm_enc_tail_loop:
/* do encrypt */
ldrb w0, [x2], #1 /* get 1 byte from input */
umov w6, v0.b[0] /* get top crypted byte */
eor w6, w6, w0 /* w6 = CTR ^ input */
strb w6, [x1], #1 /* store out byte */
/* shift right out one byte */
ext v0.16b, v0.16b, v0.16b, #1
/* the last ciphertext is placed in high bytes */
ins v0.b[15], w6
subs w4, w4, #1
bne .Lgcm_enc_tail_loop
/* padding last block with zeros */
tbl v0.16b, {v0.16b}, v3.16b
/* ghash update */
rbit v0.16b, v0.16b
eor RHASH.16b, RHASH.16b, v0.16b
PMUL_128x128(RR0, RR1, RHASH, RH1, RTMP0, RTMP1)
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP2, RTMP3)
.Lgcm_enc_hash_len:
cbz x7, .Lgcm_enc_end
GTAG_HASH_LENGTHS(v1, v3)
b .Lgcm_enc_ret
.Lgcm_enc_end:
/* store new CTR */
rev x8, x8
rev x9, x9
stp x8, x9, [x3]
rbit RHASH.16b, RHASH.16b
.Lgcm_enc_ret:
/* store new MAC */
st1 {RHASH.2d}, [x5]
ret
SYM_FUNC_END(sm4_ce_pmull_gcm_enc)
#undef RR1
#undef RR3
#undef RR5
#undef RR7
#undef RR0
#undef RR2
#undef RR4
#undef RR6
#undef RTMP0
#undef RTMP1
#undef RTMP2
#undef RTMP3
#undef RTMP4
#undef RTMP5
#undef RTMP6
#undef RTMP7
#undef RH1
#undef RH2
#undef RH3
#undef RH4
/* Register macros for decrypt */
/* v0-v2 for building CTRs, v3-v5 for saving inputs */
#define RR1 v6
#define RR3 v7
#define RR5 v8
#define RR0 v9
#define RR2 v10
#define RR4 v11
#define RTMP0 v12
#define RTMP1 v13
#define RTMP2 v14
#define RTMP3 v15
#define RTMP4 v16
#define RTMP5 v17
#define RH1 v18
#define RH2 v19
#define RH3 v20
.align 3
SYM_FUNC_START(sm4_ce_pmull_gcm_dec)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* x3: ctr (big endian, 128 bit)
* w4: nbytes
* x5: ghash result
* x6: ghash table
* x7: lengths (only for last block)
*/
SM4_PREPARE(x0)
ldp x8, x9, [x3]
rev x8, x8
rev x9, x9
ld1 {RH1.16b-RH3.16b}, [x6]
ld1 {RHASH.16b}, [x5]
rbit RHASH.16b, RHASH.16b
adr_l x6, .Lghash_rconst
ld1r {RRCONST.2d}, [x6]
eor RZERO.16b, RZERO.16b, RZERO.16b
cbz w4, .Lgcm_dec_hash_len
.Lgcm_dec_loop_3x:
cmp w4, #(3 * 16)
blt .Lgcm_dec_loop_1x
sub w4, w4, #(3 * 16)
ld1 {v3.16b-v5.16b}, [x2], #(3 * 16)
/* construct CTRs */
inc32_le128(v0) /* +0 */
rbit v6.16b, v3.16b
inc32_le128(v1) /* +1 */
rbit v7.16b, v4.16b
inc32_le128(v2) /* +2 */
rbit v8.16b, v5.16b
eor RHASH.16b, RHASH.16b, v6.16b
/* decrypt & ghash update */
SM4_CRYPT_PMUL_128x128_BLK3(v0, v1, v2,
RR0, RR1, RHASH, RH3, RTMP0, RTMP1,
RR2, RR3, v7, RH2, RTMP2, RTMP3,
RR4, RR5, v8, RH1, RTMP4, RTMP5)
eor v0.16b, v0.16b, v3.16b
eor v1.16b, v1.16b, v4.16b
eor v2.16b, v2.16b, v5.16b
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP0, RTMP1)
st1 {v0.16b-v2.16b}, [x1], #(3 * 16)
cbz w4, .Lgcm_dec_hash_len
b .Lgcm_dec_loop_3x
.Lgcm_dec_loop_1x:
cmp w4, #16
blt .Lgcm_dec_tail
sub w4, w4, #16
ld1 {v3.16b}, [x2], #16
/* construct CTRs */
inc32_le128(v0)
rbit v6.16b, v3.16b
eor RHASH.16b, RHASH.16b, v6.16b
SM4_CRYPT_PMUL_128x128_BLK(v0, RR0, RR1, RHASH, RH1, RTMP0, RTMP1)
eor v0.16b, v0.16b, v3.16b
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP2, RTMP3)
st1 {v0.16b}, [x1], #16
cbz w4, .Lgcm_dec_hash_len
b .Lgcm_dec_loop_1x
.Lgcm_dec_tail:
/* construct CTRs */
inc32_le128(v0)
SM4_CRYPT_BLK(v0)
/* load permute table */
adr_l x0, .Lcts_permute_table
add x0, x0, #32
sub x0, x0, w4, uxtw
ld1 {v3.16b}, [x0]
.Lgcm_dec_tail_loop:
/* do decrypt */
ldrb w0, [x2], #1 /* get 1 byte from input */
umov w6, v0.b[0] /* get top crypted byte */
eor w6, w6, w0 /* w6 = CTR ^ input */
strb w6, [x1], #1 /* store out byte */
/* shift right out one byte */
ext v0.16b, v0.16b, v0.16b, #1
/* the last ciphertext is placed in high bytes */
ins v0.b[15], w0
subs w4, w4, #1
bne .Lgcm_dec_tail_loop
/* padding last block with zeros */
tbl v0.16b, {v0.16b}, v3.16b
/* ghash update */
rbit v0.16b, v0.16b
eor RHASH.16b, RHASH.16b, v0.16b
PMUL_128x128(RR0, RR1, RHASH, RH1, RTMP0, RTMP1)
REDUCTION(RHASH, RR0, RR1, RRCONST, RTMP2, RTMP3)
.Lgcm_dec_hash_len:
cbz x7, .Lgcm_dec_end
GTAG_HASH_LENGTHS(v1, v3)
b .Lgcm_dec_ret
.Lgcm_dec_end:
/* store new CTR */
rev x8, x8
rev x9, x9
stp x8, x9, [x3]
rbit RHASH.16b, RHASH.16b
.Lgcm_dec_ret:
/* store new MAC */
st1 {RHASH.2d}, [x5]
ret
SYM_FUNC_END(sm4_ce_pmull_gcm_dec)
.section ".rodata", "a"
.align 4
.Lcts_permute_table:
.byte 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff
.byte 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff
.byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7
.byte 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf
.byte 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff
.byte 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff
.Lghash_rconst:
.quad 0x87
arch/arm64/crypto/sm4-ce-gcm-glue.c
+286
View File
@@ -0,0 +1,286 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* SM4-GCM AEAD Algorithm using ARMv8 Crypto Extensions
* as specified in rfc8998
* https://datatracker.ietf.org/doc/html/rfc8998
*
* Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#include <linux/module.h>
#include <linux/crypto.h>
#include <linux/kernel.h>
#include <linux/cpufeature.h>
#include <asm/neon.h>
#include <crypto/b128ops.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/skcipher.h>
#include <crypto/sm4.h>
#include "sm4-ce.h"
asmlinkage void sm4_ce_pmull_ghash_setup(const u32 *rkey_enc, u8 *ghash_table);
asmlinkage void pmull_ghash_update(const u8 *ghash_table, u8 *ghash,
const u8 *src, unsigned int nblocks);
asmlinkage void sm4_ce_pmull_gcm_enc(const u32 *rkey_enc, u8 *dst,
const u8 *src, u8 *iv,
unsigned int nbytes, u8 *ghash,
const u8 *ghash_table, const u8 *lengths);
asmlinkage void sm4_ce_pmull_gcm_dec(const u32 *rkey_enc, u8 *dst,
const u8 *src, u8 *iv,
unsigned int nbytes, u8 *ghash,
const u8 *ghash_table, const u8 *lengths);
#define GHASH_BLOCK_SIZE 16
#define GCM_IV_SIZE 12
struct sm4_gcm_ctx {
struct sm4_ctx key;
u8 ghash_table[16 * 4];
};
static int gcm_setkey(struct crypto_aead *tfm, const u8 *key,
unsigned int key_len)
{
struct sm4_gcm_ctx *ctx = crypto_aead_ctx(tfm);
if (key_len != SM4_KEY_SIZE)
return -EINVAL;
kernel_neon_begin();
sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec,
crypto_sm4_fk, crypto_sm4_ck);
sm4_ce_pmull_ghash_setup(ctx->key.rkey_enc, ctx->ghash_table);
kernel_neon_end();
return 0;
}
static int gcm_setauthsize(struct crypto_aead *tfm, unsigned int authsize)
{
switch (authsize) {
case 4:
case 8:
case 12 ... 16:
return 0;
default:
return -EINVAL;
}
}
static void gcm_calculate_auth_mac(struct aead_request *req, u8 ghash[])
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
struct sm4_gcm_ctx *ctx = crypto_aead_ctx(aead);
u8 __aligned(8) buffer[GHASH_BLOCK_SIZE];
u32 assoclen = req->assoclen;
struct scatter_walk walk;
unsigned int buflen = 0;
scatterwalk_start(&walk, req->src);
do {
u32 n = scatterwalk_clamp(&walk, assoclen);
u8 *p, *ptr;
if (!n) {
scatterwalk_start(&walk, sg_next(walk.sg));
n = scatterwalk_clamp(&walk, assoclen);
}
p = ptr = scatterwalk_map(&walk);
assoclen -= n;
scatterwalk_advance(&walk, n);
if (n + buflen < GHASH_BLOCK_SIZE) {
memcpy(&buffer[buflen], ptr, n);
buflen += n;
} else {
unsigned int nblocks;
if (buflen) {
unsigned int l = GHASH_BLOCK_SIZE - buflen;
memcpy(&buffer[buflen], ptr, l);
ptr += l;
n -= l;
pmull_ghash_update(ctx->ghash_table, ghash,
buffer, 1);
}
nblocks = n / GHASH_BLOCK_SIZE;
if (nblocks) {
pmull_ghash_update(ctx->ghash_table, ghash,
ptr, nblocks);
ptr += nblocks * GHASH_BLOCK_SIZE;
}
buflen = n % GHASH_BLOCK_SIZE;
if (buflen)
memcpy(&buffer[0], ptr, buflen);
}
scatterwalk_unmap(p);
scatterwalk_done(&walk, 0, assoclen);
} while (assoclen);
/* padding with '0' */
if (buflen) {
memset(&buffer[buflen], 0, GHASH_BLOCK_SIZE - buflen);
pmull_ghash_update(ctx->ghash_table, ghash, buffer, 1);
}
}
static int gcm_crypt(struct aead_request *req, struct skcipher_walk *walk,
struct sm4_gcm_ctx *ctx, u8 ghash[],
void (*sm4_ce_pmull_gcm_crypt)(const u32 *rkey_enc,
u8 *dst, const u8 *src, u8 *iv,
unsigned int nbytes, u8 *ghash,
const u8 *ghash_table, const u8 *lengths))
{
u8 __aligned(8) iv[SM4_BLOCK_SIZE];
be128 __aligned(8) lengths;
int err;
memset(ghash, 0, SM4_BLOCK_SIZE);
lengths.a = cpu_to_be64(req->assoclen * 8);
lengths.b = cpu_to_be64(walk->total * 8);
memcpy(iv, walk->iv, GCM_IV_SIZE);
put_unaligned_be32(2, iv + GCM_IV_SIZE);
kernel_neon_begin();
if (req->assoclen)
gcm_calculate_auth_mac(req, ghash);
do {
unsigned int tail = walk->nbytes % SM4_BLOCK_SIZE;
const u8 *src = walk->src.virt.addr;
u8 *dst = walk->dst.virt.addr;
if (walk->nbytes == walk->total) {
tail = 0;
sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv,
walk->nbytes, ghash,
ctx->ghash_table,
(const u8 *)&lengths);
} else if (walk->nbytes - tail) {
sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv,
walk->nbytes - tail, ghash,
ctx->ghash_table, NULL);
}
kernel_neon_end();
err = skcipher_walk_done(walk, tail);
if (err)
return err;
if (walk->nbytes)
kernel_neon_begin();
} while (walk->nbytes > 0);
return 0;
}
static int gcm_encrypt(struct aead_request *req)
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
struct sm4_gcm_ctx *ctx = crypto_aead_ctx(aead);
u8 __aligned(8) ghash[SM4_BLOCK_SIZE];
struct skcipher_walk walk;
int err;
err = skcipher_walk_aead_encrypt(&walk, req, false);
if (err)
return err;
err = gcm_crypt(req, &walk, ctx, ghash, sm4_ce_pmull_gcm_enc);
if (err)
return err;
/* copy authtag to end of dst */
scatterwalk_map_and_copy(ghash, req->dst, req->assoclen + req->cryptlen,
crypto_aead_authsize(aead), 1);
return 0;
}
static int gcm_decrypt(struct aead_request *req)
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
unsigned int authsize = crypto_aead_authsize(aead);
struct sm4_gcm_ctx *ctx = crypto_aead_ctx(aead);
u8 __aligned(8) ghash[SM4_BLOCK_SIZE];
u8 authtag[SM4_BLOCK_SIZE];
struct skcipher_walk walk;
int err;
err = skcipher_walk_aead_decrypt(&walk, req, false);
if (err)
return err;
err = gcm_crypt(req, &walk, ctx, ghash, sm4_ce_pmull_gcm_dec);
if (err)
return err;
/* compare calculated auth tag with the stored one */
scatterwalk_map_and_copy(authtag, req->src,
req->assoclen + req->cryptlen - authsize,
authsize, 0);
if (crypto_memneq(authtag, ghash, authsize))
return -EBADMSG;
return 0;
}
static struct aead_alg sm4_gcm_alg = {
.base = {
.cra_name = "gcm(sm4)",
.cra_driver_name = "gcm-sm4-ce",
.cra_priority = 400,
.cra_blocksize = 1,
.cra_ctxsize = sizeof(struct sm4_gcm_ctx),
.cra_module = THIS_MODULE,
},
.ivsize = GCM_IV_SIZE,
.chunksize = SM4_BLOCK_SIZE,
.maxauthsize = SM4_BLOCK_SIZE,
.setkey = gcm_setkey,
.setauthsize = gcm_setauthsize,
.encrypt = gcm_encrypt,
.decrypt = gcm_decrypt,
};
static int __init sm4_ce_gcm_init(void)
{
if (!cpu_have_named_feature(PMULL))
return -ENODEV;
return crypto_register_aead(&sm4_gcm_alg);
}
static void __exit sm4_ce_gcm_exit(void)
{
crypto_unregister_aead(&sm4_gcm_alg);
}
static const struct cpu_feature __maybe_unused sm4_ce_gcm_cpu_feature[] = {
{ cpu_feature(PMULL) },
{}
};
MODULE_DEVICE_TABLE(cpu, sm4_ce_gcm_cpu_feature);
module_cpu_feature_match(SM4, sm4_ce_gcm_init);
module_exit(sm4_ce_gcm_exit);
MODULE_DESCRIPTION("Synchronous SM4 in GCM mode using ARMv8 Crypto Extensions");
MODULE_ALIAS_CRYPTO("gcm(sm4)");
MODULE_AUTHOR("Tianjia Zhang <tianjia.zhang@linux.alibaba.com>");
MODULE_LICENSE("GPL v2");
arch/arm64/crypto/sm4-ce-glue.c
+544 -31
View File
@@ -14,8 +14,12 @@
#include <linux/cpufeature.h>
#include <asm/neon.h>
#include <asm/simd.h>
#include <crypto/b128ops.h>
#include <crypto/internal/simd.h>
#include <crypto/internal/skcipher.h>
#include <crypto/internal/hash.h>
#include <crypto/scatterwalk.h>
#include <crypto/xts.h>
#include <crypto/sm4.h>
#define BYTES2BLKS(nbytes) ((nbytes) >> 4)
@@ -26,15 +30,48 @@ asmlinkage void sm4_ce_crypt_block(const u32 *rkey, u8 *dst, const u8 *src);
asmlinkage void sm4_ce_crypt(const u32 *rkey, u8 *dst, const u8 *src,
unsigned int nblks);
asmlinkage void sm4_ce_cbc_enc(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblks);
u8 *iv, unsigned int nblocks);
asmlinkage void sm4_ce_cbc_dec(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblks);
u8 *iv, unsigned int nblocks);
asmlinkage void sm4_ce_cbc_cts_enc(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nbytes);
asmlinkage void sm4_ce_cbc_cts_dec(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nbytes);
asmlinkage void sm4_ce_cfb_enc(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblks);
asmlinkage void sm4_ce_cfb_dec(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblks);
asmlinkage void sm4_ce_ctr_enc(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblks);
asmlinkage void sm4_ce_xts_enc(const u32 *rkey1, u8 *dst, const u8 *src,
u8 *tweak, unsigned int nbytes,
const u32 *rkey2_enc);
asmlinkage void sm4_ce_xts_dec(const u32 *rkey1, u8 *dst, const u8 *src,
u8 *tweak, unsigned int nbytes,
const u32 *rkey2_enc);
asmlinkage void sm4_ce_mac_update(const u32 *rkey_enc, u8 *digest,
const u8 *src, unsigned int nblocks,
bool enc_before, bool enc_after);
EXPORT_SYMBOL(sm4_ce_expand_key);
EXPORT_SYMBOL(sm4_ce_crypt_block);
EXPORT_SYMBOL(sm4_ce_cbc_enc);
EXPORT_SYMBOL(sm4_ce_cfb_enc);
struct sm4_xts_ctx {
struct sm4_ctx key1;
struct sm4_ctx key2;
};
struct sm4_mac_tfm_ctx {
struct sm4_ctx key;
u8 __aligned(8) consts[];
};
struct sm4_mac_desc_ctx {
unsigned int len;
u8 digest[SM4_BLOCK_SIZE];
};
static int sm4_setkey(struct crypto_skcipher *tfm, const u8 *key,
unsigned int key_len)
@@ -44,8 +81,33 @@ static int sm4_setkey(struct crypto_skcipher *tfm, const u8 *key,
if (key_len != SM4_KEY_SIZE)
return -EINVAL;
kernel_neon_begin();
sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec,
crypto_sm4_fk, crypto_sm4_ck);
kernel_neon_end();
return 0;
}
static int sm4_xts_setkey(struct crypto_skcipher *tfm, const u8 *key,
unsigned int key_len)
{
struct sm4_xts_ctx *ctx = crypto_skcipher_ctx(tfm);
int ret;
if (key_len != SM4_KEY_SIZE * 2)
return -EINVAL;
ret = xts_verify_key(tfm, key, key_len);
if (ret)
return ret;
kernel_neon_begin();
sm4_ce_expand_key(key, ctx->key1.rkey_enc,
ctx->key1.rkey_dec, crypto_sm4_fk, crypto_sm4_ck);
sm4_ce_expand_key(&key[SM4_KEY_SIZE], ctx->key2.rkey_enc,
ctx->key2.rkey_dec, crypto_sm4_fk, crypto_sm4_ck);
kernel_neon_end();
return 0;
}
@@ -94,66 +156,128 @@ static int sm4_ecb_decrypt(struct skcipher_request *req)
return sm4_ecb_do_crypt(req, ctx->rkey_dec);
}
static int sm4_cbc_encrypt(struct skcipher_request *req)
static int sm4_cbc_crypt(struct skcipher_request *req,
struct sm4_ctx *ctx, bool encrypt)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm);
struct skcipher_walk walk;
unsigned int nbytes;
int err;
err = skcipher_walk_virt(&walk, req, false);
if (err)
return err;
while ((nbytes = walk.nbytes) > 0) {
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
unsigned int nblks;
unsigned int nblocks;
kernel_neon_begin();
nblocks = nbytes / SM4_BLOCK_SIZE;
if (nblocks) {
kernel_neon_begin();
nblks = BYTES2BLKS(nbytes);
if (nblks) {
sm4_ce_cbc_enc(ctx->rkey_enc, dst, src, walk.iv, nblks);
nbytes -= nblks * SM4_BLOCK_SIZE;
if (encrypt)
sm4_ce_cbc_enc(ctx->rkey_enc, dst, src,
walk.iv, nblocks);
else
sm4_ce_cbc_dec(ctx->rkey_dec, dst, src,
walk.iv, nblocks);
kernel_neon_end();
}
kernel_neon_end();
err = skcipher_walk_done(&walk, nbytes);
err = skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE);
}
return err;
}
static int sm4_cbc_encrypt(struct skcipher_request *req)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm);
return sm4_cbc_crypt(req, ctx, true);
}
static int sm4_cbc_decrypt(struct skcipher_request *req)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm);
return sm4_cbc_crypt(req, ctx, false);
}
static int sm4_cbc_cts_crypt(struct skcipher_request *req, bool encrypt)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct sm4_ctx *ctx = crypto_skcipher_ctx(tfm);
struct scatterlist *src = req->src;
struct scatterlist *dst = req->dst;
struct scatterlist sg_src[2], sg_dst[2];
struct skcipher_request subreq;
struct skcipher_walk walk;
unsigned int nbytes;
int cbc_blocks;
int err;
err = skcipher_walk_virt(&walk, req, false);
if (req->cryptlen < SM4_BLOCK_SIZE)
return -EINVAL;
while ((nbytes = walk.nbytes) > 0) {
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
unsigned int nblks;
if (req->cryptlen == SM4_BLOCK_SIZE)
return sm4_cbc_crypt(req, ctx, encrypt);
kernel_neon_begin();
skcipher_request_set_tfm(&subreq, tfm);
skcipher_request_set_callback(&subreq, skcipher_request_flags(req),
NULL, NULL);
nblks = BYTES2BLKS(nbytes);
if (nblks) {
sm4_ce_cbc_dec(ctx->rkey_dec, dst, src, walk.iv, nblks);
nbytes -= nblks * SM4_BLOCK_SIZE;
}
/* handle the CBC cryption part */
cbc_blocks = DIV_ROUND_UP(req->cryptlen, SM4_BLOCK_SIZE) - 2;
if (cbc_blocks) {
skcipher_request_set_crypt(&subreq, src, dst,
cbc_blocks * SM4_BLOCK_SIZE,
req->iv);
kernel_neon_end();
err = sm4_cbc_crypt(&subreq, ctx, encrypt);
if (err)
return err;
err = skcipher_walk_done(&walk, nbytes);
dst = src = scatterwalk_ffwd(sg_src, src, subreq.cryptlen);
if (req->dst != req->src)
dst = scatterwalk_ffwd(sg_dst, req->dst,
subreq.cryptlen);
}
return err;
/* handle ciphertext stealing */
skcipher_request_set_crypt(&subreq, src, dst,
req->cryptlen - cbc_blocks * SM4_BLOCK_SIZE,
req->iv);
err = skcipher_walk_virt(&walk, &subreq, false);
if (err)
return err;
kernel_neon_begin();
if (encrypt)
sm4_ce_cbc_cts_enc(ctx->rkey_enc, walk.dst.virt.addr,
walk.src.virt.addr, walk.iv, walk.nbytes);
else
sm4_ce_cbc_cts_dec(ctx->rkey_dec, walk.dst.virt.addr,
walk.src.virt.addr, walk.iv, walk.nbytes);
kernel_neon_end();
return skcipher_walk_done(&walk, 0);
}
static int sm4_cbc_cts_encrypt(struct skcipher_request *req)
{
return sm4_cbc_cts_crypt(req, true);
}
static int sm4_cbc_cts_decrypt(struct skcipher_request *req)
{
return sm4_cbc_cts_crypt(req, false);
}
static int sm4_cfb_encrypt(struct skcipher_request *req)
@@ -283,6 +407,111 @@ static int sm4_ctr_crypt(struct skcipher_request *req)
return err;
}
static int sm4_xts_crypt(struct skcipher_request *req, bool encrypt)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct sm4_xts_ctx *ctx = crypto_skcipher_ctx(tfm);
int tail = req->cryptlen % SM4_BLOCK_SIZE;
const u32 *rkey2_enc = ctx->key2.rkey_enc;
struct scatterlist sg_src[2], sg_dst[2];
struct skcipher_request subreq;
struct scatterlist *src, *dst;
struct skcipher_walk walk;
unsigned int nbytes;
int err;
if (req->cryptlen < SM4_BLOCK_SIZE)
return -EINVAL;
err = skcipher_walk_virt(&walk, req, false);
if (err)
return err;
if (unlikely(tail > 0 && walk.nbytes < walk.total)) {
int nblocks = DIV_ROUND_UP(req->cryptlen, SM4_BLOCK_SIZE) - 2;
skcipher_walk_abort(&walk);
skcipher_request_set_tfm(&subreq, tfm);
skcipher_request_set_callback(&subreq,
skcipher_request_flags(req),
NULL, NULL);
skcipher_request_set_crypt(&subreq, req->src, req->dst,
nblocks * SM4_BLOCK_SIZE, req->iv);
err = skcipher_walk_virt(&walk, &subreq, false);
if (err)
return err;
} else {
tail = 0;
}
while ((nbytes = walk.nbytes) >= SM4_BLOCK_SIZE) {
if (nbytes < walk.total)
nbytes &= ~(SM4_BLOCK_SIZE - 1);
kernel_neon_begin();
if (encrypt)
sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr,
walk.src.virt.addr, walk.iv, nbytes,
rkey2_enc);
else
sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr,
walk.src.virt.addr, walk.iv, nbytes,
rkey2_enc);
kernel_neon_end();
rkey2_enc = NULL;
err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
if (err)
return err;
}
if (likely(tail == 0))
return 0;
/* handle ciphertext stealing */
dst = src = scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen);
if (req->dst != req->src)
dst = scatterwalk_ffwd(sg_dst, req->dst, subreq.cryptlen);
skcipher_request_set_crypt(&subreq, src, dst, SM4_BLOCK_SIZE + tail,
req->iv);
err = skcipher_walk_virt(&walk, &subreq, false);
if (err)
return err;
kernel_neon_begin();
if (encrypt)
sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr,
walk.src.virt.addr, walk.iv, walk.nbytes,
rkey2_enc);
else
sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr,
walk.src.virt.addr, walk.iv, walk.nbytes,
rkey2_enc);
kernel_neon_end();
return skcipher_walk_done(&walk, 0);
}
static int sm4_xts_encrypt(struct skcipher_request *req)
{
return sm4_xts_crypt(req, true);
}
static int sm4_xts_decrypt(struct skcipher_request *req)
{
return sm4_xts_crypt(req, false);
}
static struct skcipher_alg sm4_algs[] = {
{
.base = {
@@ -345,28 +574,312 @@ static struct skcipher_alg sm4_algs[] = {
.setkey = sm4_setkey,
.encrypt = sm4_ctr_crypt,
.decrypt = sm4_ctr_crypt,
}, {
.base = {
.cra_name = "cts(cbc(sm4))",
.cra_driver_name = "cts-cbc-sm4-ce",
.cra_priority = 400,
.cra_blocksize = SM4_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct sm4_ctx),
.cra_module = THIS_MODULE,
},
.min_keysize = SM4_KEY_SIZE,
.max_keysize = SM4_KEY_SIZE,
.ivsize = SM4_BLOCK_SIZE,
.walksize = SM4_BLOCK_SIZE * 2,
.setkey = sm4_setkey,
.encrypt = sm4_cbc_cts_encrypt,
.decrypt = sm4_cbc_cts_decrypt,
}, {
.base = {
.cra_name = "xts(sm4)",
.cra_driver_name = "xts-sm4-ce",
.cra_priority = 400,
.cra_blocksize = SM4_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct sm4_xts_ctx),
.cra_module = THIS_MODULE,
},
.min_keysize = SM4_KEY_SIZE * 2,
.max_keysize = SM4_KEY_SIZE * 2,
.ivsize = SM4_BLOCK_SIZE,
.walksize = SM4_BLOCK_SIZE * 2,
.setkey = sm4_xts_setkey,
.encrypt = sm4_xts_encrypt,
.decrypt = sm4_xts_decrypt,
}
};
static int sm4_cbcmac_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int key_len)
{
struct sm4_mac_tfm_ctx *ctx = crypto_shash_ctx(tfm);
if (key_len != SM4_KEY_SIZE)
return -EINVAL;
kernel_neon_begin();
sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec,
crypto_sm4_fk, crypto_sm4_ck);
kernel_neon_end();
return 0;
}
static int sm4_cmac_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int key_len)
{
struct sm4_mac_tfm_ctx *ctx = crypto_shash_ctx(tfm);
be128 *consts = (be128 *)ctx->consts;
u64 a, b;
if (key_len != SM4_KEY_SIZE)
return -EINVAL;
memset(consts, 0, SM4_BLOCK_SIZE);
kernel_neon_begin();
sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec,
crypto_sm4_fk, crypto_sm4_ck);
/* encrypt the zero block */
sm4_ce_crypt_block(ctx->key.rkey_enc, (u8 *)consts, (const u8 *)consts);
kernel_neon_end();
/* gf(2^128) multiply zero-ciphertext with u and u^2 */
a = be64_to_cpu(consts[0].a);
b = be64_to_cpu(consts[0].b);
consts[0].a = cpu_to_be64((a << 1) | (b >> 63));
consts[0].b = cpu_to_be64((b << 1) ^ ((a >> 63) ? 0x87 : 0));
a = be64_to_cpu(consts[0].a);
b = be64_to_cpu(consts[0].b);
consts[1].a = cpu_to_be64((a << 1) | (b >> 63));
consts[1].b = cpu_to_be64((b << 1) ^ ((a >> 63) ? 0x87 : 0));
return 0;
}
static int sm4_xcbc_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int key_len)
{
struct sm4_mac_tfm_ctx *ctx = crypto_shash_ctx(tfm);
u8 __aligned(8) key2[SM4_BLOCK_SIZE];
static u8 const ks[3][SM4_BLOCK_SIZE] = {
{ [0 ... SM4_BLOCK_SIZE - 1] = 0x1},
{ [0 ... SM4_BLOCK_SIZE - 1] = 0x2},
{ [0 ... SM4_BLOCK_SIZE - 1] = 0x3},
};
if (key_len != SM4_KEY_SIZE)
return -EINVAL;
kernel_neon_begin();
sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec,
crypto_sm4_fk, crypto_sm4_ck);
sm4_ce_crypt_block(ctx->key.rkey_enc, key2, ks[0]);
sm4_ce_crypt(ctx->key.rkey_enc, ctx->consts, ks[1], 2);
sm4_ce_expand_key(key2, ctx->key.rkey_enc, ctx->key.rkey_dec,
crypto_sm4_fk, crypto_sm4_ck);
kernel_neon_end();
return 0;
}
static int sm4_mac_init(struct shash_desc *desc)
{
struct sm4_mac_desc_ctx *ctx = shash_desc_ctx(desc);
memset(ctx->digest, 0, SM4_BLOCK_SIZE);
ctx->len = 0;
return 0;
}
static int sm4_mac_update(struct shash_desc *desc, const u8 *p,
unsigned int len)
{
struct sm4_mac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
struct sm4_mac_desc_ctx *ctx = shash_desc_ctx(desc);
unsigned int l, nblocks;
if (len == 0)
return 0;
if (ctx->len || ctx->len + len < SM4_BLOCK_SIZE) {
l = min(len, SM4_BLOCK_SIZE - ctx->len);
crypto_xor(ctx->digest + ctx->len, p, l);
ctx->len += l;
len -= l;
p += l;
}
if (len && (ctx->len % SM4_BLOCK_SIZE) == 0) {
kernel_neon_begin();
if (len < SM4_BLOCK_SIZE && ctx->len == SM4_BLOCK_SIZE) {
sm4_ce_crypt_block(tctx->key.rkey_enc,
ctx->digest, ctx->digest);
ctx->len = 0;
} else {
nblocks = len / SM4_BLOCK_SIZE;
len %= SM4_BLOCK_SIZE;
sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, p,
nblocks, (ctx->len == SM4_BLOCK_SIZE),
(len != 0));
p += nblocks * SM4_BLOCK_SIZE;
if (len == 0)
ctx->len = SM4_BLOCK_SIZE;
}
kernel_neon_end();
if (len) {
crypto_xor(ctx->digest, p, len);
ctx->len = len;
}
}
return 0;
}
static int sm4_cmac_final(struct shash_desc *desc, u8 *out)
{
struct sm4_mac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
struct sm4_mac_desc_ctx *ctx = shash_desc_ctx(desc);
const u8 *consts = tctx->consts;
if (ctx->len != SM4_BLOCK_SIZE) {
ctx->digest[ctx->len] ^= 0x80;
consts += SM4_BLOCK_SIZE;
}
kernel_neon_begin();
sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, consts, 1,
false, true);
kernel_neon_end();
memcpy(out, ctx->digest, SM4_BLOCK_SIZE);
return 0;
}
static int sm4_cbcmac_final(struct shash_desc *desc, u8 *out)
{
struct sm4_mac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
struct sm4_mac_desc_ctx *ctx = shash_desc_ctx(desc);
if (ctx->len) {
kernel_neon_begin();
sm4_ce_crypt_block(tctx->key.rkey_enc, ctx->digest,
ctx->digest);
kernel_neon_end();
}
memcpy(out, ctx->digest, SM4_BLOCK_SIZE);
return 0;
}
static struct shash_alg sm4_mac_algs[] = {
{
.base = {
.cra_name = "cmac(sm4)",
.cra_driver_name = "cmac-sm4-ce",
.cra_priority = 400,
.cra_blocksize = SM4_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct sm4_mac_tfm_ctx)
+ SM4_BLOCK_SIZE * 2,
.cra_module = THIS_MODULE,
},
.digestsize = SM4_BLOCK_SIZE,
.init = sm4_mac_init,
.update = sm4_mac_update,
.final = sm4_cmac_final,
.setkey = sm4_cmac_setkey,
.descsize = sizeof(struct sm4_mac_desc_ctx),
}, {
.base = {
.cra_name = "xcbc(sm4)",
.cra_driver_name = "xcbc-sm4-ce",
.cra_priority = 400,
.cra_blocksize = SM4_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct sm4_mac_tfm_ctx)
+ SM4_BLOCK_SIZE * 2,
.cra_module = THIS_MODULE,
},
.digestsize = SM4_BLOCK_SIZE,
.init = sm4_mac_init,
.update = sm4_mac_update,
.final = sm4_cmac_final,
.setkey = sm4_xcbc_setkey,
.descsize = sizeof(struct sm4_mac_desc_ctx),
}, {
.base = {
.cra_name = "cbcmac(sm4)",
.cra_driver_name = "cbcmac-sm4-ce",
.cra_priority = 400,
.cra_blocksize = 1,
.cra_ctxsize = sizeof(struct sm4_mac_tfm_ctx),
.cra_module = THIS_MODULE,
},
.digestsize = SM4_BLOCK_SIZE,
.init = sm4_mac_init,
.update = sm4_mac_update,
.final = sm4_cbcmac_final,
.setkey = sm4_cbcmac_setkey,
.descsize = sizeof(struct sm4_mac_desc_ctx),
}
};
static int __init sm4_init(void)
{
return crypto_register_skciphers(sm4_algs, ARRAY_SIZE(sm4_algs));
int err;
err = crypto_register_skciphers(sm4_algs, ARRAY_SIZE(sm4_algs));
if (err)
return err;
err = crypto_register_shashes(sm4_mac_algs, ARRAY_SIZE(sm4_mac_algs));
if (err)
goto out_err;
return 0;
out_err:
crypto_unregister_skciphers(sm4_algs, ARRAY_SIZE(sm4_algs));
return err;
}
static void __exit sm4_exit(void)
{
crypto_unregister_shashes(sm4_mac_algs, ARRAY_SIZE(sm4_mac_algs));
crypto_unregister_skciphers(sm4_algs, ARRAY_SIZE(sm4_algs));
}
module_cpu_feature_match(SM4, sm4_init);
module_exit(sm4_exit);
MODULE_DESCRIPTION("SM4 ECB/CBC/CFB/CTR using ARMv8 Crypto Extensions");
MODULE_DESCRIPTION("SM4 ECB/CBC/CFB/CTR/XTS using ARMv8 Crypto Extensions");
MODULE_ALIAS_CRYPTO("sm4-ce");
MODULE_ALIAS_CRYPTO("sm4");
MODULE_ALIAS_CRYPTO("ecb(sm4)");
MODULE_ALIAS_CRYPTO("cbc(sm4)");
MODULE_ALIAS_CRYPTO("cfb(sm4)");
MODULE_ALIAS_CRYPTO("ctr(sm4)");
MODULE_ALIAS_CRYPTO("cts(cbc(sm4))");
MODULE_ALIAS_CRYPTO("xts(sm4)");
MODULE_ALIAS_CRYPTO("cmac(sm4)");
MODULE_ALIAS_CRYPTO("xcbc(sm4)");
MODULE_ALIAS_CRYPTO("cbcmac(sm4)");
MODULE_AUTHOR("Tianjia Zhang <tianjia.zhang@linux.alibaba.com>");
MODULE_LICENSE("GPL v2");
arch/arm64/crypto/sm4-ce.h
+16
View File
@@ -0,0 +1,16 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* SM4 common functions for Crypto Extensions
* Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
void sm4_ce_expand_key(const u8 *key, u32 *rkey_enc, u32 *rkey_dec,
const u32 *fk, const u32 *ck);
void sm4_ce_crypt_block(const u32 *rkey, u8 *dst, const u8 *src);
void sm4_ce_cbc_enc(const u32 *rkey_enc, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblocks);
void sm4_ce_cfb_enc(const u32 *rkey_enc, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblocks);
arch/arm64/crypto/sm4-neon-core.S
+398 -206
View File
@@ -18,6 +18,11 @@
#define RTMP2 v10
#define RTMP3 v11
#define RTMP4 v12
#define RTMP5 v13
#define RTMP6 v14
#define RTMP7 v15
#define RX0 v12
#define RX1 v13
#define RKEY v14
@@ -25,7 +30,7 @@
/* Helper macros. */
#define PREPARE \
#define SM4_PREPARE() \
adr_l x5, crypto_sm4_sbox; \
ld1 {v16.16b-v19.16b}, [x5], #64; \
ld1 {v20.16b-v23.16b}, [x5], #64; \
@@ -42,7 +47,25 @@
zip1 s2.2d, RTMP2.2d, RTMP3.2d; \
zip2 s3.2d, RTMP2.2d, RTMP3.2d;
#define rotate_clockwise_90(s0, s1, s2, s3) \
#define transpose_4x4_2x(s0, s1, s2, s3, s4, s5, s6, s7) \
zip1 RTMP0.4s, s0.4s, s1.4s; \
zip1 RTMP1.4s, s2.4s, s3.4s; \
zip2 RTMP2.4s, s0.4s, s1.4s; \
zip2 RTMP3.4s, s2.4s, s3.4s; \
zip1 RTMP4.4s, s4.4s, s5.4s; \
zip1 RTMP5.4s, s6.4s, s7.4s; \
zip2 RTMP6.4s, s4.4s, s5.4s; \
zip2 RTMP7.4s, s6.4s, s7.4s; \
zip1 s0.2d, RTMP0.2d, RTMP1.2d; \
zip2 s1.2d, RTMP0.2d, RTMP1.2d; \
zip1 s2.2d, RTMP2.2d, RTMP3.2d; \
zip2 s3.2d, RTMP2.2d, RTMP3.2d; \
zip1 s4.2d, RTMP4.2d, RTMP5.2d; \
zip2 s5.2d, RTMP4.2d, RTMP5.2d; \
zip1 s6.2d, RTMP6.2d, RTMP7.2d; \
zip2 s7.2d, RTMP6.2d, RTMP7.2d;
#define rotate_clockwise_4x4(s0, s1, s2, s3) \
zip1 RTMP0.4s, s1.4s, s0.4s; \
zip2 RTMP1.4s, s1.4s, s0.4s; \
zip1 RTMP2.4s, s3.4s, s2.4s; \
@@ -52,6 +75,24 @@
zip1 s2.2d, RTMP3.2d, RTMP1.2d; \
zip2 s3.2d, RTMP3.2d, RTMP1.2d;
#define rotate_clockwise_4x4_2x(s0, s1, s2, s3, s4, s5, s6, s7) \
zip1 RTMP0.4s, s1.4s, s0.4s; \
zip1 RTMP2.4s, s3.4s, s2.4s; \
zip2 RTMP1.4s, s1.4s, s0.4s; \
zip2 RTMP3.4s, s3.4s, s2.4s; \
zip1 RTMP4.4s, s5.4s, s4.4s; \
zip1 RTMP6.4s, s7.4s, s6.4s; \
zip2 RTMP5.4s, s5.4s, s4.4s; \
zip2 RTMP7.4s, s7.4s, s6.4s; \
zip1 s0.2d, RTMP2.2d, RTMP0.2d; \
zip2 s1.2d, RTMP2.2d, RTMP0.2d; \
zip1 s2.2d, RTMP3.2d, RTMP1.2d; \
zip2 s3.2d, RTMP3.2d, RTMP1.2d; \
zip1 s4.2d, RTMP6.2d, RTMP4.2d; \
zip2 s5.2d, RTMP6.2d, RTMP4.2d; \
zip1 s6.2d, RTMP7.2d, RTMP5.2d; \
zip2 s7.2d, RTMP7.2d, RTMP5.2d;
#define ROUND4(round, s0, s1, s2, s3) \
dup RX0.4s, RKEY.s[round]; \
/* rk ^ s1 ^ s2 ^ s3 */ \
@@ -87,14 +128,7 @@
/* s0 ^= RTMP3 */ \
eor s0.16b, s0.16b, RTMP3.16b;
#define SM4_CRYPT_BLK4(b0, b1, b2, b3) \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
rev32 b3.16b, b3.16b; \
\
transpose_4x4(b0, b1, b2, b3); \
\
#define SM4_CRYPT_BLK4_BE(b0, b1, b2, b3) \
mov x6, 8; \
4: \
ld1 {RKEY.4s}, [x0], #16; \
@@ -107,15 +141,23 @@
\
bne 4b; \
\
rotate_clockwise_90(b0, b1, b2, b3); \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
rev32 b3.16b, b3.16b; \
\
rotate_clockwise_4x4(b0, b1, b2, b3); \
\
/* repoint to rkey */ \
sub x0, x0, #128;
#define SM4_CRYPT_BLK4(b0, b1, b2, b3) \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
rev32 b3.16b, b3.16b; \
SM4_CRYPT_BLK4_BE(b0, b1, b2, b3);
#define ROUND8(round, s0, s1, s2, s3, t0, t1, t2, t3) \
/* rk ^ s1 ^ s2 ^ s3 */ \
dup RX0.4s, RKEY.s[round]; \
@@ -175,7 +217,7 @@
eor s0.16b, s0.16b, RTMP0.16b; \
eor t0.16b, t0.16b, RTMP1.16b;
#define SM4_CRYPT_BLK8(b0, b1, b2, b3, b4, b5, b6, b7) \
#define SM4_CRYPT_BLK8_norotate(b0, b1, b2, b3, b4, b5, b6, b7) \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
@@ -185,9 +227,6 @@
rev32 b6.16b, b6.16b; \
rev32 b7.16b, b7.16b; \
\
transpose_4x4(b0, b1, b2, b3); \
transpose_4x4(b4, b5, b6, b7); \
\
mov x6, 8; \
8: \
ld1 {RKEY.4s}, [x0], #16; \
@@ -200,8 +239,6 @@
\
bne 8b; \
\
rotate_clockwise_90(b0, b1, b2, b3); \
rotate_clockwise_90(b4, b5, b6, b7); \
rev32 b0.16b, b0.16b; \
rev32 b1.16b, b1.16b; \
rev32 b2.16b, b2.16b; \
@@ -214,274 +251,429 @@
/* repoint to rkey */ \
sub x0, x0, #128;
#define SM4_CRYPT_BLK8(b0, b1, b2, b3, b4, b5, b6, b7) \
SM4_CRYPT_BLK8_norotate(b0, b1, b2, b3, b4, b5, b6, b7); \
rotate_clockwise_4x4_2x(b0, b1, b2, b3, b4, b5, b6, b7); \
.align 3
SYM_FUNC_START_LOCAL(__sm4_neon_crypt_blk1_4)
SYM_FUNC_START(sm4_neon_crypt)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* w3: num blocks (1..4)
* w3: nblocks
*/
PREPARE;
SM4_PREPARE()
ld1 {v0.16b}, [x2], #16;
mov v1.16b, v0.16b;
mov v2.16b, v0.16b;
mov v3.16b, v0.16b;
cmp w3, #2;
blt .Lblk4_load_input_done;
ld1 {v1.16b}, [x2], #16;
beq .Lblk4_load_input_done;
ld1 {v2.16b}, [x2], #16;
cmp w3, #3;
beq .Lblk4_load_input_done;
ld1 {v3.16b}, [x2];
.Lcrypt_loop_8x:
sub w3, w3, #8
tbnz w3, #31, .Lcrypt_4x
.Lblk4_load_input_done:
SM4_CRYPT_BLK4(v0, v1, v2, v3);
ld4 {v0.4s-v3.4s}, [x2], #64
ld4 {v4.4s-v7.4s}, [x2], #64
st1 {v0.16b}, [x1], #16;
cmp w3, #2;
blt .Lblk4_store_output_done;
st1 {v1.16b}, [x1], #16;
beq .Lblk4_store_output_done;
st1 {v2.16b}, [x1], #16;
cmp w3, #3;
beq .Lblk4_store_output_done;
st1 {v3.16b}, [x1];
SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7)
.Lblk4_store_output_done:
ret;
SYM_FUNC_END(__sm4_neon_crypt_blk1_4)
st1 {v0.16b-v3.16b}, [x1], #64
st1 {v4.16b-v7.16b}, [x1], #64
.align 3
SYM_FUNC_START(sm4_neon_crypt_blk1_8)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* w3: num blocks (1..8)
*/
cmp w3, #5;
blt __sm4_neon_crypt_blk1_4;
cbz w3, .Lcrypt_end
b .Lcrypt_loop_8x
PREPARE;
.Lcrypt_4x:
add w3, w3, #8
cmp w3, #4
blt .Lcrypt_tail
ld1 {v0.16b-v3.16b}, [x2], #64;
ld1 {v4.16b}, [x2], #16;
mov v5.16b, v4.16b;
mov v6.16b, v4.16b;
mov v7.16b, v4.16b;
beq .Lblk8_load_input_done;
ld1 {v5.16b}, [x2], #16;
cmp w3, #7;
blt .Lblk8_load_input_done;
ld1 {v6.16b}, [x2], #16;
beq .Lblk8_load_input_done;
ld1 {v7.16b}, [x2];
sub w3, w3, #4
.Lblk8_load_input_done:
SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7);
ld4 {v0.4s-v3.4s}, [x2], #64
cmp w3, #6;
st1 {v0.16b-v3.16b}, [x1], #64;
st1 {v4.16b}, [x1], #16;
blt .Lblk8_store_output_done;
st1 {v5.16b}, [x1], #16;
beq .Lblk8_store_output_done;
st1 {v6.16b}, [x1], #16;
cmp w3, #7;
beq .Lblk8_store_output_done;
st1 {v7.16b}, [x1];
SM4_CRYPT_BLK4(v0, v1, v2, v3)
.Lblk8_store_output_done:
ret;
SYM_FUNC_END(sm4_neon_crypt_blk1_8)
st1 {v0.16b-v3.16b}, [x1], #64
.align 3
SYM_FUNC_START(sm4_neon_crypt_blk8)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* w3: nblocks (multiples of 8)
*/
PREPARE;
cbz w3, .Lcrypt_end
.Lcrypt_loop_blk:
subs w3, w3, #8;
bmi .Lcrypt_end;
.Lcrypt_tail:
cmp w3, #2
ld1 {v0.16b}, [x2], #16
blt .Lcrypt_tail_load_done
ld1 {v1.16b}, [x2], #16
beq .Lcrypt_tail_load_done
ld1 {v2.16b}, [x2], #16
ld1 {v0.16b-v3.16b}, [x2], #64;
ld1 {v4.16b-v7.16b}, [x2], #64;
.Lcrypt_tail_load_done:
transpose_4x4(v0, v1, v2, v3)
SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7);
SM4_CRYPT_BLK4(v0, v1, v2, v3)
st1 {v0.16b-v3.16b}, [x1], #64;
st1 {v4.16b-v7.16b}, [x1], #64;
b .Lcrypt_loop_blk;
cmp w3, #2
st1 {v0.16b}, [x1], #16
blt .Lcrypt_end
st1 {v1.16b}, [x1], #16
beq .Lcrypt_end
st1 {v2.16b}, [x1], #16
.Lcrypt_end:
ret;
SYM_FUNC_END(sm4_neon_crypt_blk8)
ret
SYM_FUNC_END(sm4_neon_crypt)
.align 3
SYM_FUNC_START(sm4_neon_cbc_dec_blk8)
SYM_FUNC_START(sm4_neon_cbc_dec)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* x3: iv (big endian, 128 bit)
* w4: nblocks (multiples of 8)
* w4: nblocks
*/
PREPARE;
SM4_PREPARE()
ld1 {RIV.16b}, [x3];
ld1 {RIV.16b}, [x3]
.Lcbc_loop_blk:
subs w4, w4, #8;
bmi .Lcbc_end;
.Lcbc_dec_loop_8x:
sub w4, w4, #8
tbnz w4, #31, .Lcbc_dec_4x
ld1 {v0.16b-v3.16b}, [x2], #64;
ld1 {v4.16b-v7.16b}, [x2];
ld4 {v0.4s-v3.4s}, [x2], #64
ld4 {v4.4s-v7.4s}, [x2]
SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7);
SM4_CRYPT_BLK8_norotate(v0, v1, v2, v3, v4, v5, v6, v7)
sub x2, x2, #64;
eor v0.16b, v0.16b, RIV.16b;
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64;
eor v1.16b, v1.16b, RTMP0.16b;
eor v2.16b, v2.16b, RTMP1.16b;
eor v3.16b, v3.16b, RTMP2.16b;
st1 {v0.16b-v3.16b}, [x1], #64;
/* Avoid overwriting the RIV register */
rotate_clockwise_4x4(v0, v1, v2, v3)
rotate_clockwise_4x4(v4, v5, v6, v7)
eor v4.16b, v4.16b, RTMP3.16b;
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64;
eor v5.16b, v5.16b, RTMP0.16b;
eor v6.16b, v6.16b, RTMP1.16b;
eor v7.16b, v7.16b, RTMP2.16b;
sub x2, x2, #64
mov RIV.16b, RTMP3.16b;
st1 {v4.16b-v7.16b}, [x1], #64;
eor v0.16b, v0.16b, RIV.16b
b .Lcbc_loop_blk;
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64
ld1 {RTMP4.16b-RTMP7.16b}, [x2], #64
.Lcbc_end:
eor v1.16b, v1.16b, RTMP0.16b
eor v2.16b, v2.16b, RTMP1.16b
eor v3.16b, v3.16b, RTMP2.16b
eor v4.16b, v4.16b, RTMP3.16b
eor v5.16b, v5.16b, RTMP4.16b
eor v6.16b, v6.16b, RTMP5.16b
eor v7.16b, v7.16b, RTMP6.16b
mov RIV.16b, RTMP7.16b
st1 {v0.16b-v3.16b}, [x1], #64
st1 {v4.16b-v7.16b}, [x1], #64
cbz w4, .Lcbc_dec_end
b .Lcbc_dec_loop_8x
.Lcbc_dec_4x:
add w4, w4, #8
cmp w4, #4
blt .Lcbc_dec_tail
sub w4, w4, #4
ld1 {v0.16b-v3.16b}, [x2], #64
rev32 v4.16b, v0.16b
rev32 v5.16b, v1.16b
rev32 v6.16b, v2.16b
rev32 v7.16b, v3.16b
transpose_4x4(v4, v5, v6, v7)
SM4_CRYPT_BLK4_BE(v4, v5, v6, v7)
eor v4.16b, v4.16b, RIV.16b
eor v5.16b, v5.16b, v0.16b
eor v6.16b, v6.16b, v1.16b
eor v7.16b, v7.16b, v2.16b
mov RIV.16b, v3.16b
st1 {v4.16b-v7.16b}, [x1], #64
cbz w4, .Lcbc_dec_end
.Lcbc_dec_tail:
cmp w4, #2
ld1 {v0.16b}, [x2], #16
blt .Lcbc_dec_tail_load_done
ld1 {v1.16b}, [x2], #16
beq .Lcbc_dec_tail_load_done
ld1 {v2.16b}, [x2], #16
.Lcbc_dec_tail_load_done:
rev32 v4.16b, v0.16b
rev32 v5.16b, v1.16b
rev32 v6.16b, v2.16b
transpose_4x4(v4, v5, v6, v7)
SM4_CRYPT_BLK4_BE(v4, v5, v6, v7)
cmp w4, #2
eor v4.16b, v4.16b, RIV.16b
mov RIV.16b, v0.16b
st1 {v4.16b}, [x1], #16
blt .Lcbc_dec_end
eor v5.16b, v5.16b, v0.16b
mov RIV.16b, v1.16b
st1 {v5.16b}, [x1], #16
beq .Lcbc_dec_end
eor v6.16b, v6.16b, v1.16b
mov RIV.16b, v2.16b
st1 {v6.16b}, [x1], #16
.Lcbc_dec_end:
/* store new IV */
st1 {RIV.16b}, [x3];
st1 {RIV.16b}, [x3]
ret;
SYM_FUNC_END(sm4_neon_cbc_dec_blk8)
ret
SYM_FUNC_END(sm4_neon_cbc_dec)
.align 3
SYM_FUNC_START(sm4_neon_cfb_dec_blk8)
SYM_FUNC_START(sm4_neon_cfb_dec)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* x3: iv (big endian, 128 bit)
* w4: nblocks (multiples of 8)
* w4: nblocks
*/
PREPARE;
SM4_PREPARE()
ld1 {v0.16b}, [x3];
ld1 {v0.16b}, [x3]
.Lcfb_loop_blk:
subs w4, w4, #8;
bmi .Lcfb_end;
.Lcfb_dec_loop_8x:
sub w4, w4, #8
tbnz w4, #31, .Lcfb_dec_4x
ld1 {v1.16b, v2.16b, v3.16b}, [x2], #48;
ld1 {v4.16b-v7.16b}, [x2];
ld1 {v1.16b-v3.16b}, [x2], #48
ld4 {v4.4s-v7.4s}, [x2]
SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7);
transpose_4x4(v0, v1, v2, v3)
sub x2, x2, #48;
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64;
eor v0.16b, v0.16b, RTMP0.16b;
eor v1.16b, v1.16b, RTMP1.16b;
eor v2.16b, v2.16b, RTMP2.16b;
eor v3.16b, v3.16b, RTMP3.16b;
st1 {v0.16b-v3.16b}, [x1], #64;
SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7)
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64;
eor v4.16b, v4.16b, RTMP0.16b;
eor v5.16b, v5.16b, RTMP1.16b;
eor v6.16b, v6.16b, RTMP2.16b;
eor v7.16b, v7.16b, RTMP3.16b;
st1 {v4.16b-v7.16b}, [x1], #64;
sub x2, x2, #48
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64
ld1 {RTMP4.16b-RTMP7.16b}, [x2], #64
mov v0.16b, RTMP3.16b;
eor v0.16b, v0.16b, RTMP0.16b
eor v1.16b, v1.16b, RTMP1.16b
eor v2.16b, v2.16b, RTMP2.16b
eor v3.16b, v3.16b, RTMP3.16b
eor v4.16b, v4.16b, RTMP4.16b
eor v5.16b, v5.16b, RTMP5.16b
eor v6.16b, v6.16b, RTMP6.16b
eor v7.16b, v7.16b, RTMP7.16b
b .Lcfb_loop_blk;
st1 {v0.16b-v3.16b}, [x1], #64
st1 {v4.16b-v7.16b}, [x1], #64
.Lcfb_end:
mov v0.16b, RTMP7.16b
cbz w4, .Lcfb_dec_end
b .Lcfb_dec_loop_8x
.Lcfb_dec_4x:
add w4, w4, #8
cmp w4, #4
blt .Lcfb_dec_tail
sub w4, w4, #4
ld1 {v4.16b-v7.16b}, [x2], #64
rev32 v0.16b, v0.16b /* v0 is IV register */
rev32 v1.16b, v4.16b
rev32 v2.16b, v5.16b
rev32 v3.16b, v6.16b
transpose_4x4(v0, v1, v2, v3)
SM4_CRYPT_BLK4_BE(v0, v1, v2, v3)
eor v0.16b, v0.16b, v4.16b
eor v1.16b, v1.16b, v5.16b
eor v2.16b, v2.16b, v6.16b
eor v3.16b, v3.16b, v7.16b
st1 {v0.16b-v3.16b}, [x1], #64
mov v0.16b, v7.16b
cbz w4, .Lcfb_dec_end
.Lcfb_dec_tail:
cmp w4, #2
ld1 {v4.16b}, [x2], #16
blt .Lcfb_dec_tail_load_done
ld1 {v5.16b}, [x2], #16
beq .Lcfb_dec_tail_load_done
ld1 {v6.16b}, [x2], #16
.Lcfb_dec_tail_load_done:
rev32 v0.16b, v0.16b /* v0 is IV register */
rev32 v1.16b, v4.16b
rev32 v2.16b, v5.16b
transpose_4x4(v0, v1, v2, v3)
SM4_CRYPT_BLK4_BE(v0, v1, v2, v3)
cmp w4, #2
eor v0.16b, v0.16b, v4.16b
st1 {v0.16b}, [x1], #16
mov v0.16b, v4.16b
blt .Lcfb_dec_end
eor v1.16b, v1.16b, v5.16b
st1 {v1.16b}, [x1], #16
mov v0.16b, v5.16b
beq .Lcfb_dec_end
eor v2.16b, v2.16b, v6.16b
st1 {v2.16b}, [x1], #16
mov v0.16b, v6.16b
.Lcfb_dec_end:
/* store new IV */
st1 {v0.16b}, [x3];
st1 {v0.16b}, [x3]
ret;
SYM_FUNC_END(sm4_neon_cfb_dec_blk8)
ret
SYM_FUNC_END(sm4_neon_cfb_dec)
.align 3
SYM_FUNC_START(sm4_neon_ctr_enc_blk8)
SYM_FUNC_START(sm4_neon_ctr_crypt)
/* input:
* x0: round key array, CTX
* x1: dst
* x2: src
* x3: ctr (big endian, 128 bit)
* w4: nblocks (multiples of 8)
* w4: nblocks
*/
PREPARE;
SM4_PREPARE()
ldp x7, x8, [x3];
rev x7, x7;
rev x8, x8;
ldp x7, x8, [x3]
rev x7, x7
rev x8, x8
.Lctr_loop_blk:
subs w4, w4, #8;
bmi .Lctr_end;
.Lctr_crypt_loop_8x:
sub w4, w4, #8
tbnz w4, #31, .Lctr_crypt_4x
#define inc_le128(vctr) \
mov vctr.d[1], x8; \
mov vctr.d[0], x7; \
adds x8, x8, #1; \
adc x7, x7, xzr; \
rev64 vctr.16b, vctr.16b;
#define inc_le128(vctr) \
mov vctr.d[1], x8; \
mov vctr.d[0], x7; \
adds x8, x8, #1; \
rev64 vctr.16b, vctr.16b; \
adc x7, x7, xzr;
/* construct CTRs */
inc_le128(v0); /* +0 */
inc_le128(v1); /* +1 */
inc_le128(v2); /* +2 */
inc_le128(v3); /* +3 */
inc_le128(v4); /* +4 */
inc_le128(v5); /* +5 */
inc_le128(v6); /* +6 */
inc_le128(v7); /* +7 */
inc_le128(v0) /* +0 */
inc_le128(v1) /* +1 */
inc_le128(v2) /* +2 */
inc_le128(v3) /* +3 */
inc_le128(v4) /* +4 */
inc_le128(v5) /* +5 */
inc_le128(v6) /* +6 */
inc_le128(v7) /* +7 */
SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7);
transpose_4x4_2x(v0, v1, v2, v3, v4, v5, v6, v7)
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64;
eor v0.16b, v0.16b, RTMP0.16b;
eor v1.16b, v1.16b, RTMP1.16b;
eor v2.16b, v2.16b, RTMP2.16b;
eor v3.16b, v3.16b, RTMP3.16b;
st1 {v0.16b-v3.16b}, [x1], #64;
SM4_CRYPT_BLK8(v0, v1, v2, v3, v4, v5, v6, v7)
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64;
eor v4.16b, v4.16b, RTMP0.16b;
eor v5.16b, v5.16b, RTMP1.16b;
eor v6.16b, v6.16b, RTMP2.16b;
eor v7.16b, v7.16b, RTMP3.16b;
st1 {v4.16b-v7.16b}, [x1], #64;
ld1 {RTMP0.16b-RTMP3.16b}, [x2], #64
ld1 {RTMP4.16b-RTMP7.16b}, [x2], #64
b .Lctr_loop_blk;
eor v0.16b, v0.16b, RTMP0.16b
eor v1.16b, v1.16b, RTMP1.16b
eor v2.16b, v2.16b, RTMP2.16b
eor v3.16b, v3.16b, RTMP3.16b
eor v4.16b, v4.16b, RTMP4.16b
eor v5.16b, v5.16b, RTMP5.16b
eor v6.16b, v6.16b, RTMP6.16b
eor v7.16b, v7.16b, RTMP7.16b
.Lctr_end:
st1 {v0.16b-v3.16b}, [x1], #64
st1 {v4.16b-v7.16b}, [x1], #64
cbz w4, .Lctr_crypt_end
b .Lctr_crypt_loop_8x
.Lctr_crypt_4x:
add w4, w4, #8
cmp w4, #4
blt .Lctr_crypt_tail
sub w4, w4, #4
/* construct CTRs */
inc_le128(v0) /* +0 */
inc_le128(v1) /* +1 */
inc_le128(v2) /* +2 */
inc_le128(v3) /* +3 */
ld1 {v4.16b-v7.16b}, [x2], #64
transpose_4x4(v0, v1, v2, v3)
SM4_CRYPT_BLK4(v0, v1, v2, v3)
eor v0.16b, v0.16b, v4.16b
eor v1.16b, v1.16b, v5.16b
eor v2.16b, v2.16b, v6.16b
eor v3.16b, v3.16b, v7.16b
st1 {v0.16b-v3.16b}, [x1], #64
cbz w4, .Lctr_crypt_end
.Lctr_crypt_tail:
/* inc_le128 will change the sign bit */
ld1 {v4.16b}, [x2], #16
inc_le128(v0)
cmp w4, #2
blt .Lctr_crypt_tail_load_done
ld1 {v5.16b}, [x2], #16
inc_le128(v1)
cmp w4, #2
beq .Lctr_crypt_tail_load_done
ld1 {v6.16b}, [x2], #16
inc_le128(v2)
.Lctr_crypt_tail_load_done:
transpose_4x4(v0, v1, v2, v3)
SM4_CRYPT_BLK4(v0, v1, v2, v3)
cmp w4, #2
eor v0.16b, v0.16b, v4.16b
st1 {v0.16b}, [x1], #16
blt .Lctr_crypt_end
eor v1.16b, v1.16b, v5.16b
st1 {v1.16b}, [x1], #16
beq .Lctr_crypt_end
eor v2.16b, v2.16b, v6.16b
st1 {v2.16b}, [x1], #16
.Lctr_crypt_end:
/* store new CTR */
rev x7, x7;
rev x8, x8;
stp x7, x8, [x3];
rev x7, x7
rev x8, x8
stp x7, x8, [x3]
ret;
SYM_FUNC_END(sm4_neon_ctr_enc_blk8)
ret
SYM_FUNC_END(sm4_neon_ctr_crypt)
arch/arm64/crypto/sm4-neon-glue.c
+49 -131
View File
@@ -18,19 +18,14 @@
#include <crypto/internal/skcipher.h>
#include <crypto/sm4.h>
#define BYTES2BLKS(nbytes) ((nbytes) >> 4)
#define BYTES2BLK8(nbytes) (((nbytes) >> 4) & ~(8 - 1))
asmlinkage void sm4_neon_crypt_blk1_8(const u32 *rkey, u8 *dst, const u8 *src,
unsigned int nblks);
asmlinkage void sm4_neon_crypt_blk8(const u32 *rkey, u8 *dst, const u8 *src,
unsigned int nblks);
asmlinkage void sm4_neon_cbc_dec_blk8(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblks);
asmlinkage void sm4_neon_cfb_dec_blk8(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblks);
asmlinkage void sm4_neon_ctr_enc_blk8(const u32 *rkey, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblks);
asmlinkage void sm4_neon_crypt(const u32 *rkey, u8 *dst, const u8 *src,
unsigned int nblocks);
asmlinkage void sm4_neon_cbc_dec(const u32 *rkey_dec, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblocks);
asmlinkage void sm4_neon_cfb_dec(const u32 *rkey_enc, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblocks);
asmlinkage void sm4_neon_ctr_crypt(const u32 *rkey_enc, u8 *dst, const u8 *src,
u8 *iv, unsigned int nblocks);
static int sm4_setkey(struct crypto_skcipher *tfm, const u8 *key,
unsigned int key_len)
@@ -51,27 +46,18 @@ static int sm4_ecb_do_crypt(struct skcipher_request *req, const u32 *rkey)
while ((nbytes = walk.nbytes) > 0) {
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
unsigned int nblks;
unsigned int nblocks;
kernel_neon_begin();
nblocks = nbytes / SM4_BLOCK_SIZE;
if (nblocks) {
kernel_neon_begin();
nblks = BYTES2BLK8(nbytes);
if (nblks) {
sm4_neon_crypt_blk8(rkey, dst, src, nblks);
dst += nblks * SM4_BLOCK_SIZE;
src += nblks * SM4_BLOCK_SIZE;
nbytes -= nblks * SM4_BLOCK_SIZE;
sm4_neon_crypt(rkey, dst, src, nblocks);
kernel_neon_end();
}
nblks = BYTES2BLKS(nbytes);
if (nblks) {
sm4_neon_crypt_blk1_8(rkey, dst, src, nblks);
nbytes -= nblks * SM4_BLOCK_SIZE;
}
kernel_neon_end();
err = skcipher_walk_done(&walk, nbytes);
err = skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE);
}
return err;
@@ -138,48 +124,19 @@ static int sm4_cbc_decrypt(struct skcipher_request *req)
while ((nbytes = walk.nbytes) > 0) {
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
unsigned int nblks;
unsigned int nblocks;
kernel_neon_begin();
nblocks = nbytes / SM4_BLOCK_SIZE;
if (nblocks) {
kernel_neon_begin();
nblks = BYTES2BLK8(nbytes);
if (nblks) {
sm4_neon_cbc_dec_blk8(ctx->rkey_dec, dst, src,
walk.iv, nblks);
dst += nblks * SM4_BLOCK_SIZE;
src += nblks * SM4_BLOCK_SIZE;
nbytes -= nblks * SM4_BLOCK_SIZE;
sm4_neon_cbc_dec(ctx->rkey_dec, dst, src,
walk.iv, nblocks);
kernel_neon_end();
}
nblks = BYTES2BLKS(nbytes);
if (nblks) {
u8 keystream[SM4_BLOCK_SIZE * 8];
u8 iv[SM4_BLOCK_SIZE];
int i;
sm4_neon_crypt_blk1_8(ctx->rkey_dec, keystream,
src, nblks);
src += ((int)nblks - 2) * SM4_BLOCK_SIZE;
dst += (nblks - 1) * SM4_BLOCK_SIZE;
memcpy(iv, src + SM4_BLOCK_SIZE, SM4_BLOCK_SIZE);
for (i = nblks - 1; i > 0; i--) {
crypto_xor_cpy(dst, src,
&keystream[i * SM4_BLOCK_SIZE],
SM4_BLOCK_SIZE);
src -= SM4_BLOCK_SIZE;
dst -= SM4_BLOCK_SIZE;
}
crypto_xor_cpy(dst, walk.iv,
keystream, SM4_BLOCK_SIZE);
memcpy(walk.iv, iv, SM4_BLOCK_SIZE);
nbytes -= nblks * SM4_BLOCK_SIZE;
}
kernel_neon_end();
err = skcipher_walk_done(&walk, nbytes);
err = skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE);
}
return err;
@@ -238,42 +195,22 @@ static int sm4_cfb_decrypt(struct skcipher_request *req)
while ((nbytes = walk.nbytes) > 0) {
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
unsigned int nblks;
unsigned int nblocks;
kernel_neon_begin();
nblocks = nbytes / SM4_BLOCK_SIZE;
if (nblocks) {
kernel_neon_begin();
nblks = BYTES2BLK8(nbytes);
if (nblks) {
sm4_neon_cfb_dec_blk8(ctx->rkey_enc, dst, src,
walk.iv, nblks);
dst += nblks * SM4_BLOCK_SIZE;
src += nblks * SM4_BLOCK_SIZE;
nbytes -= nblks * SM4_BLOCK_SIZE;
sm4_neon_cfb_dec(ctx->rkey_enc, dst, src,
walk.iv, nblocks);
kernel_neon_end();
dst += nblocks * SM4_BLOCK_SIZE;
src += nblocks * SM4_BLOCK_SIZE;
nbytes -= nblocks * SM4_BLOCK_SIZE;
}
nblks = BYTES2BLKS(nbytes);
if (nblks) {
u8 keystream[SM4_BLOCK_SIZE * 8];
memcpy(keystream, walk.iv, SM4_BLOCK_SIZE);
if (nblks > 1)
memcpy(&keystream[SM4_BLOCK_SIZE], src,
(nblks - 1) * SM4_BLOCK_SIZE);
memcpy(walk.iv, src + (nblks - 1) * SM4_BLOCK_SIZE,
SM4_BLOCK_SIZE);
sm4_neon_crypt_blk1_8(ctx->rkey_enc, keystream,
keystream, nblks);
crypto_xor_cpy(dst, src, keystream,
nblks * SM4_BLOCK_SIZE);
dst += nblks * SM4_BLOCK_SIZE;
src += nblks * SM4_BLOCK_SIZE;
nbytes -= nblks * SM4_BLOCK_SIZE;
}
kernel_neon_end();
/* tail */
if (walk.nbytes == walk.total && nbytes > 0) {
u8 keystream[SM4_BLOCK_SIZE];
@@ -302,41 +239,22 @@ static int sm4_ctr_crypt(struct skcipher_request *req)
while ((nbytes = walk.nbytes) > 0) {
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
unsigned int nblks;
unsigned int nblocks;
kernel_neon_begin();
nblocks = nbytes / SM4_BLOCK_SIZE;
if (nblocks) {
kernel_neon_begin();
nblks = BYTES2BLK8(nbytes);
if (nblks) {
sm4_neon_ctr_enc_blk8(ctx->rkey_enc, dst, src,
walk.iv, nblks);
dst += nblks * SM4_BLOCK_SIZE;
src += nblks * SM4_BLOCK_SIZE;
nbytes -= nblks * SM4_BLOCK_SIZE;
sm4_neon_ctr_crypt(ctx->rkey_enc, dst, src,
walk.iv, nblocks);
kernel_neon_end();
dst += nblocks * SM4_BLOCK_SIZE;
src += nblocks * SM4_BLOCK_SIZE;
nbytes -= nblocks * SM4_BLOCK_SIZE;
}
nblks = BYTES2BLKS(nbytes);
if (nblks) {
u8 keystream[SM4_BLOCK_SIZE * 8];
int i;
for (i = 0; i < nblks; i++) {
memcpy(&keystream[i * SM4_BLOCK_SIZE],
walk.iv, SM4_BLOCK_SIZE);
crypto_inc(walk.iv, SM4_BLOCK_SIZE);
}
sm4_neon_crypt_blk1_8(ctx->rkey_enc, keystream,
keystream, nblks);
crypto_xor_cpy(dst, src, keystream,
nblks * SM4_BLOCK_SIZE);
dst += nblks * SM4_BLOCK_SIZE;
src += nblks * SM4_BLOCK_SIZE;
nbytes -= nblks * SM4_BLOCK_SIZE;
}
kernel_neon_end();
/* tail */
if (walk.nbytes == walk.total && nbytes > 0) {
u8 keystream[SM4_BLOCK_SIZE];
arch/arm64/include/asm/kvm_arm.h
+6 -2
View File
@@ -135,7 +135,7 @@
* 40 bits wide (T0SZ = 24). Systems with a PARange smaller than 40 bits are
* not known to exist and will break with this configuration.
*
* The VTCR_EL2 is configured per VM and is initialised in kvm_arm_setup_stage2().
* The VTCR_EL2 is configured per VM and is initialised in kvm_init_stage2_mmu.
*
* Note that when using 4K pages, we concatenate two first level page tables
* together. With 16K pages, we concatenate 16 first level page tables.
@@ -340,9 +340,13 @@
* We have
* PAR [PA_Shift - 1 : 12] = PA [PA_Shift - 1 : 12]
* HPFAR [PA_Shift - 9 : 4] = FIPA [PA_Shift - 1 : 12]
*
* Always assume 52 bit PA since at this point, we don't know how many PA bits
* the page table has been set up for. This should be safe since unused address
* bits in PAR are res0.
*/
#define PAR_TO_HPFAR(par) \
(((par) & GENMASK_ULL(PHYS_MASK_SHIFT - 1, 12)) >> 8)
(((par) & GENMASK_ULL(52 - 1, 12)) >> 8)
#define ECN(x) { ESR_ELx_EC_##x, #x }
arch/arm64/include/asm/kvm_asm.h
+5 -2
View File
@@ -76,6 +76,9 @@ enum __kvm_host_smccc_func {
__KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs,
__KVM_HOST_SMCCC_FUNC___vgic_v3_restore_aprs,
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_init_traps,
__KVM_HOST_SMCCC_FUNC___pkvm_init_vm,
__KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu,
__KVM_HOST_SMCCC_FUNC___pkvm_teardown_vm,
};
#define DECLARE_KVM_VHE_SYM(sym) extern char sym[]
@@ -106,7 +109,7 @@ enum __kvm_host_smccc_func {
#define per_cpu_ptr_nvhe_sym(sym, cpu) \
({ \
unsigned long base, off; \
base = kvm_arm_hyp_percpu_base[cpu]; \
base = kvm_nvhe_sym(kvm_arm_hyp_percpu_base)[cpu]; \
off = (unsigned long)&CHOOSE_NVHE_SYM(sym) - \
(unsigned long)&CHOOSE_NVHE_SYM(__per_cpu_start); \
base ? (typeof(CHOOSE_NVHE_SYM(sym))*)(base + off) : NULL; \
@@ -211,7 +214,7 @@ DECLARE_KVM_HYP_SYM(__kvm_hyp_vector);
#define __kvm_hyp_init CHOOSE_NVHE_SYM(__kvm_hyp_init)
#define __kvm_hyp_vector CHOOSE_HYP_SYM(__kvm_hyp_vector)
extern unsigned long kvm_arm_hyp_percpu_base[NR_CPUS];
extern unsigned long kvm_nvhe_sym(kvm_arm_hyp_percpu_base)[];
DECLARE_KVM_NVHE_SYM(__per_cpu_start);
DECLARE_KVM_NVHE_SYM(__per_cpu_end);
arch/arm64/include/asm/kvm_host.h
+74 -2
View File
@@ -73,6 +73,63 @@ u32 __attribute_const__ kvm_target_cpu(void);
int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu);
struct kvm_hyp_memcache {
phys_addr_t head;
unsigned long nr_pages;
};
static inline void push_hyp_memcache(struct kvm_hyp_memcache *mc,
phys_addr_t *p,
phys_addr_t (*to_pa)(void *virt))
{
*p = mc->head;
mc->head = to_pa(p);
mc->nr_pages++;
}
static inline void *pop_hyp_memcache(struct kvm_hyp_memcache *mc,
void *(*to_va)(phys_addr_t phys))
{
phys_addr_t *p = to_va(mc->head);
if (!mc->nr_pages)
return NULL;
mc->head = *p;
mc->nr_pages--;
return p;
}
static inline int __topup_hyp_memcache(struct kvm_hyp_memcache *mc,
unsigned long min_pages,
void *(*alloc_fn)(void *arg),
phys_addr_t (*to_pa)(void *virt),
void *arg)
{
while (mc->nr_pages < min_pages) {
phys_addr_t *p = alloc_fn(arg);
if (!p)
return -ENOMEM;
push_hyp_memcache(mc, p, to_pa);
}
return 0;
}
static inline void __free_hyp_memcache(struct kvm_hyp_memcache *mc,
void (*free_fn)(void *virt, void *arg),
void *(*to_va)(phys_addr_t phys),
void *arg)
{
while (mc->nr_pages)
free_fn(pop_hyp_memcache(mc, to_va), arg);
}
void free_hyp_memcache(struct kvm_hyp_memcache *mc);
int topup_hyp_memcache(struct kvm_hyp_memcache *mc, unsigned long min_pages);
struct kvm_vmid {
atomic64_t id;
};
@@ -115,6 +172,13 @@ struct kvm_smccc_features {
unsigned long vendor_hyp_bmap;
};
typedef unsigned int pkvm_handle_t;
struct kvm_protected_vm {
pkvm_handle_t handle;
struct kvm_hyp_memcache teardown_mc;
};
struct kvm_arch {
struct kvm_s2_mmu mmu;
@@ -163,9 +227,19 @@ struct kvm_arch {
u8 pfr0_csv2;
u8 pfr0_csv3;
struct {
u8 imp:4;
u8 unimp:4;
} dfr0_pmuver;
/* Hypercall features firmware registers' descriptor */
struct kvm_smccc_features smccc_feat;
/*
* For an untrusted host VM, 'pkvm.handle' is used to lookup
* the associated pKVM instance in the hypervisor.
*/
struct kvm_protected_vm pkvm;
};
struct kvm_vcpu_fault_info {
@@ -925,8 +999,6 @@ int kvm_set_ipa_limit(void);
#define __KVM_HAVE_ARCH_VM_ALLOC
struct kvm *kvm_arch_alloc_vm(void);
int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type);
static inline bool kvm_vm_is_protected(struct kvm *kvm)
{
return false;
arch/arm64/include/asm/kvm_hyp.h
+3
View File
@@ -123,4 +123,7 @@ extern u64 kvm_nvhe_sym(id_aa64mmfr0_el1_sys_val);
extern u64 kvm_nvhe_sym(id_aa64mmfr1_el1_sys_val);
extern u64 kvm_nvhe_sym(id_aa64mmfr2_el1_sys_val);
extern unsigned long kvm_nvhe_sym(__icache_flags);
extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits);
#endif /* __ARM64_KVM_HYP_H__ */
arch/arm64/include/asm/kvm_mmu.h
+1 -1
View File
@@ -166,7 +166,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
void free_hyp_pgds(void);
void stage2_unmap_vm(struct kvm *kvm);
int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu);
int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long type);
void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu);
int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
phys_addr_t pa, unsigned long size, bool writable);
arch/arm64/include/asm/kvm_pgtable.h
+154 -35
View File
@@ -42,6 +42,8 @@ typedef u64 kvm_pte_t;
#define KVM_PTE_ADDR_MASK GENMASK(47, PAGE_SHIFT)
#define KVM_PTE_ADDR_51_48 GENMASK(15, 12)
#define KVM_PHYS_INVALID (-1ULL)
static inline bool kvm_pte_valid(kvm_pte_t pte)
{
return pte & KVM_PTE_VALID;
@@ -57,6 +59,18 @@ static inline u64 kvm_pte_to_phys(kvm_pte_t pte)
return pa;
}
static inline kvm_pte_t kvm_phys_to_pte(u64 pa)
{
kvm_pte_t pte = pa & KVM_PTE_ADDR_MASK;
if (PAGE_SHIFT == 16) {
pa &= GENMASK(51, 48);
pte |= FIELD_PREP(KVM_PTE_ADDR_51_48, pa >> 48);
}
return pte;
}
static inline u64 kvm_granule_shift(u32 level)
{
/* Assumes KVM_PGTABLE_MAX_LEVELS is 4 */
@@ -85,6 +99,8 @@ static inline bool kvm_level_supports_block_mapping(u32 level)
* allocation is physically contiguous.
* @free_pages_exact: Free an exact number of memory pages previously
* allocated by zalloc_pages_exact.
* @free_removed_table: Free a removed paging structure by unlinking and
* dropping references.
* @get_page: Increment the refcount on a page.
* @put_page: Decrement the refcount on a page. When the
* refcount reaches 0 the page is automatically
@@ -103,6 +119,7 @@ struct kvm_pgtable_mm_ops {
void* (*zalloc_page)(void *arg);
void* (*zalloc_pages_exact)(size_t size);
void (*free_pages_exact)(void *addr, size_t size);
void (*free_removed_table)(void *addr, u32 level);
void (*get_page)(void *addr);
void (*put_page)(void *addr);
int (*page_count)(void *addr);
@@ -161,6 +178,121 @@ enum kvm_pgtable_prot {
typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end,
enum kvm_pgtable_prot prot);
/**
* enum kvm_pgtable_walk_flags - Flags to control a depth-first page-table walk.
* @KVM_PGTABLE_WALK_LEAF: Visit leaf entries, including invalid
* entries.
* @KVM_PGTABLE_WALK_TABLE_PRE: Visit table entries before their
* children.
* @KVM_PGTABLE_WALK_TABLE_POST: Visit table entries after their
* children.
* @KVM_PGTABLE_WALK_SHARED: Indicates the page-tables may be shared
* with other software walkers.
*/
enum kvm_pgtable_walk_flags {
KVM_PGTABLE_WALK_LEAF = BIT(0),
KVM_PGTABLE_WALK_TABLE_PRE = BIT(1),
KVM_PGTABLE_WALK_TABLE_POST = BIT(2),
KVM_PGTABLE_WALK_SHARED = BIT(3),
};
struct kvm_pgtable_visit_ctx {
kvm_pte_t *ptep;
kvm_pte_t old;
void *arg;
struct kvm_pgtable_mm_ops *mm_ops;
u64 addr;
u64 end;
u32 level;
enum kvm_pgtable_walk_flags flags;
};
typedef int (*kvm_pgtable_visitor_fn_t)(const struct kvm_pgtable_visit_ctx *ctx,
enum kvm_pgtable_walk_flags visit);
static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *ctx)
{
return ctx->flags & KVM_PGTABLE_WALK_SHARED;
}
/**
* struct kvm_pgtable_walker - Hook into a page-table walk.
* @cb: Callback function to invoke during the walk.
* @arg: Argument passed to the callback function.
* @flags: Bitwise-OR of flags to identify the entry types on which to
* invoke the callback function.
*/
struct kvm_pgtable_walker {
const kvm_pgtable_visitor_fn_t cb;
void * const arg;
const enum kvm_pgtable_walk_flags flags;
};
/*
* RCU cannot be used in a non-kernel context such as the hyp. As such, page
* table walkers used in hyp do not call into RCU and instead use other
* synchronization mechanisms (such as a spinlock).
*/
#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
typedef kvm_pte_t *kvm_pteref_t;
static inline kvm_pte_t *kvm_dereference_pteref(struct kvm_pgtable_walker *walker,
kvm_pteref_t pteref)
{
return pteref;
}
static inline int kvm_pgtable_walk_begin(struct kvm_pgtable_walker *walker)
{
/*
* Due to the lack of RCU (or a similar protection scheme), only
* non-shared table walkers are allowed in the hypervisor.
*/
if (walker->flags & KVM_PGTABLE_WALK_SHARED)
return -EPERM;
return 0;
}
static inline void kvm_pgtable_walk_end(struct kvm_pgtable_walker *walker) {}
static inline bool kvm_pgtable_walk_lock_held(void)
{
return true;
}
#else
typedef kvm_pte_t __rcu *kvm_pteref_t;
static inline kvm_pte_t *kvm_dereference_pteref(struct kvm_pgtable_walker *walker,
kvm_pteref_t pteref)
{
return rcu_dereference_check(pteref, !(walker->flags & KVM_PGTABLE_WALK_SHARED));
}
static inline int kvm_pgtable_walk_begin(struct kvm_pgtable_walker *walker)
{
if (walker->flags & KVM_PGTABLE_WALK_SHARED)
rcu_read_lock();
return 0;
}
static inline void kvm_pgtable_walk_end(struct kvm_pgtable_walker *walker)
{
if (walker->flags & KVM_PGTABLE_WALK_SHARED)
rcu_read_unlock();
}
static inline bool kvm_pgtable_walk_lock_held(void)
{
return rcu_read_lock_held();
}
#endif
/**
* struct kvm_pgtable - KVM page-table.
* @ia_bits: Maximum input address size, in bits.
@@ -175,7 +307,7 @@ typedef bool (*kvm_pgtable_force_pte_cb_t)(u64 addr, u64 end,
struct kvm_pgtable {
u32 ia_bits;
u32 start_level;
kvm_pte_t *pgd;
kvm_pteref_t pgd;
struct kvm_pgtable_mm_ops *mm_ops;
/* Stage-2 only */
@@ -184,39 +316,6 @@ struct kvm_pgtable {
kvm_pgtable_force_pte_cb_t force_pte_cb;
};
/**
* enum kvm_pgtable_walk_flags - Flags to control a depth-first page-table walk.
* @KVM_PGTABLE_WALK_LEAF: Visit leaf entries, including invalid
* entries.
* @KVM_PGTABLE_WALK_TABLE_PRE: Visit table entries before their
* children.
* @KVM_PGTABLE_WALK_TABLE_POST: Visit table entries after their
* children.
*/
enum kvm_pgtable_walk_flags {
KVM_PGTABLE_WALK_LEAF = BIT(0),
KVM_PGTABLE_WALK_TABLE_PRE = BIT(1),
KVM_PGTABLE_WALK_TABLE_POST = BIT(2),
};
typedef int (*kvm_pgtable_visitor_fn_t)(u64 addr, u64 end, u32 level,
kvm_pte_t *ptep,
enum kvm_pgtable_walk_flags flag,
void * const arg);
/**
* struct kvm_pgtable_walker - Hook into a page-table walk.
* @cb: Callback function to invoke during the walk.
* @arg: Argument passed to the callback function.
* @flags: Bitwise-OR of flags to identify the entry types on which to
* invoke the callback function.
*/
struct kvm_pgtable_walker {
const kvm_pgtable_visitor_fn_t cb;
void * const arg;
const enum kvm_pgtable_walk_flags flags;
};
/**
* kvm_pgtable_hyp_init() - Initialise a hypervisor stage-1 page-table.
* @pgt: Uninitialised page-table structure to initialise.
@@ -296,6 +395,14 @@ u64 kvm_pgtable_hyp_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size);
*/
u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift);
/**
* kvm_pgtable_stage2_pgd_size() - Helper to compute size of a stage-2 PGD
* @vtcr: Content of the VTCR register.
*
* Return: the size (in bytes) of the stage-2 PGD
*/
size_t kvm_pgtable_stage2_pgd_size(u64 vtcr);
/**
* __kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table.
* @pgt: Uninitialised page-table structure to initialise.
@@ -324,6 +431,17 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
*/
void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
/**
* kvm_pgtable_stage2_free_removed() - Free a removed stage-2 paging structure.
* @mm_ops: Memory management callbacks.
* @pgtable: Unlinked stage-2 paging structure to be freed.
* @level: Level of the stage-2 paging structure to be freed.
*
* The page-table is assumed to be unreachable by any hardware walkers prior to
* freeing and therefore no TLB invalidation is performed.
*/
void kvm_pgtable_stage2_free_removed(struct kvm_pgtable_mm_ops *mm_ops, void *pgtable, u32 level);
/**
* kvm_pgtable_stage2_map() - Install a mapping in a guest stage-2 page-table.
* @pgt: Page-table structure initialised by kvm_pgtable_stage2_init*().
@@ -333,6 +451,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
* @prot: Permissions and attributes for the mapping.
* @mc: Cache of pre-allocated and zeroed memory from which to allocate
* page-table pages.
* @flags: Flags to control the page-table walk (ex. a shared walk)
*
* The offset of @addr within a page is ignored, @size is rounded-up to
* the next page boundary and @phys is rounded-down to the previous page
@@ -354,7 +473,7 @@ void kvm_pgtable_stage2_destroy(struct kvm_pgtable *pgt);
*/
int kvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
u64 phys, enum kvm_pgtable_prot prot,
void *mc);
void *mc, enum kvm_pgtable_walk_flags flags);
/**
* kvm_pgtable_stage2_set_owner() - Unmap and annotate pages in the IPA space to
arch/arm64/include/asm/kvm_pkvm.h
+38
View File
@@ -9,11 +9,49 @@
#include <linux/memblock.h>
#include <asm/kvm_pgtable.h>
/* Maximum number of VMs that can co-exist under pKVM. */
#define KVM_MAX_PVMS 255
#define HYP_MEMBLOCK_REGIONS 128
int pkvm_init_host_vm(struct kvm *kvm);
int pkvm_create_hyp_vm(struct kvm *kvm);
void pkvm_destroy_hyp_vm(struct kvm *kvm);
extern struct memblock_region kvm_nvhe_sym(hyp_memory)[];
extern unsigned int kvm_nvhe_sym(hyp_memblock_nr);
static inline unsigned long
hyp_vmemmap_memblock_size(struct memblock_region *reg, size_t vmemmap_entry_size)
{
unsigned long nr_pages = reg->size >> PAGE_SHIFT;
unsigned long start, end;
start = (reg->base >> PAGE_SHIFT) * vmemmap_entry_size;
end = start + nr_pages * vmemmap_entry_size;
start = ALIGN_DOWN(start, PAGE_SIZE);
end = ALIGN(end, PAGE_SIZE);
return end - start;
}
static inline unsigned long hyp_vmemmap_pages(size_t vmemmap_entry_size)
{
unsigned long res = 0, i;
for (i = 0; i < kvm_nvhe_sym(hyp_memblock_nr); i++) {
res += hyp_vmemmap_memblock_size(&kvm_nvhe_sym(hyp_memory)[i],
vmemmap_entry_size);
}
return res >> PAGE_SHIFT;
}
static inline unsigned long hyp_vm_table_pages(void)
{
return PAGE_ALIGN(KVM_MAX_PVMS * sizeof(void *)) >> PAGE_SHIFT;
}
static inline unsigned long __hyp_pgtable_max_pages(unsigned long nr_pages)
{
unsigned long total = 0, i;
arch/arm64/include/asm/mte.h
+64 -1
View File
@@ -25,7 +25,7 @@ unsigned long mte_copy_tags_to_user(void __user *to, void *from,
unsigned long n);
int mte_save_tags(struct page *page);
void mte_save_page_tags(const void *page_addr, void *tag_storage);
bool mte_restore_tags(swp_entry_t entry, struct page *page);
void mte_restore_tags(swp_entry_t entry, struct page *page);
void mte_restore_page_tags(void *page_addr, const void *tag_storage);
void mte_invalidate_tags(int type, pgoff_t offset);
void mte_invalidate_tags_area(int type);
@@ -36,6 +36,58 @@ void mte_free_tag_storage(char *storage);
/* track which pages have valid allocation tags */
#define PG_mte_tagged PG_arch_2
/* simple lock to avoid multiple threads tagging the same page */
#define PG_mte_lock PG_arch_3
static inline void set_page_mte_tagged(struct page *page)
{
/*
* Ensure that the tags written prior to this function are visible
* before the page flags update.
*/
smp_wmb();
set_bit(PG_mte_tagged, &page->flags);
}
static inline bool page_mte_tagged(struct page *page)
{
bool ret = test_bit(PG_mte_tagged, &page->flags);
/*
* If the page is tagged, ensure ordering with a likely subsequent
* read of the tags.
*/
if (ret)
smp_rmb();
return ret;
}
/*
* Lock the page for tagging and return 'true' if the page can be tagged,
* 'false' if already tagged. PG_mte_tagged is never cleared and therefore the
* locking only happens once for page initialisation.
*
* The page MTE lock state:
*
* Locked: PG_mte_lock && !PG_mte_tagged
* Unlocked: !PG_mte_lock || PG_mte_tagged
*
* Acquire semantics only if the page is tagged (returning 'false').
*/
static inline bool try_page_mte_tagging(struct page *page)
{
if (!test_and_set_bit(PG_mte_lock, &page->flags))
return true;
/*
* The tags are either being initialised or may have been initialised
* already. Check if the PG_mte_tagged flag has been set or wait
* otherwise.
*/
smp_cond_load_acquire(&page->flags, VAL & (1UL << PG_mte_tagged));
return false;
}
void mte_zero_clear_page_tags(void *addr);
void mte_sync_tags(pte_t old_pte, pte_t pte);
@@ -56,6 +108,17 @@ size_t mte_probe_user_range(const char __user *uaddr, size_t size);
/* unused if !CONFIG_ARM64_MTE, silence the compiler */
#define PG_mte_tagged 0
static inline void set_page_mte_tagged(struct page *page)
{
}
static inline bool page_mte_tagged(struct page *page)
{
return false;
}
static inline bool try_page_mte_tagging(struct page *page)
{
return false;
}
static inline void mte_zero_clear_page_tags(void *addr)
{
}
arch/arm64/include/asm/pgtable.h
+2 -2
View File
@@ -1046,8 +1046,8 @@ static inline void arch_swap_invalidate_area(int type)
#define __HAVE_ARCH_SWAP_RESTORE
static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio)
{
if (system_supports_mte() && mte_restore_tags(entry, &folio->page))
set_bit(PG_mte_tagged, &folio->flags);
if (system_supports_mte())
mte_restore_tags(entry, &folio->page);
}
#endif /* CONFIG_ARM64_MTE */
arch/arm64/include/uapi/asm/kvm.h
+1
View File
@@ -43,6 +43,7 @@
#define __KVM_HAVE_VCPU_EVENTS
#define KVM_COALESCED_MMIO_PAGE_OFFSET 1
#define KVM_DIRTY_LOG_PAGE_OFFSET 64
#define KVM_REG_SIZE(id) \
(1U << (((id) & KVM_REG_SIZE_MASK) >> KVM_REG_SIZE_SHIFT))
arch/arm64/kernel/cpufeature.c
+3 -1
View File
@@ -2077,8 +2077,10 @@ static void cpu_enable_mte(struct arm64_cpu_capabilities const *cap)
* Clear the tags in the zero page. This needs to be done via the
* linear map which has the Tagged attribute.
*/
if (!test_and_set_bit(PG_mte_tagged, &ZERO_PAGE(0)->flags))
if (try_page_mte_tagging(ZERO_PAGE(0))) {
mte_clear_page_tags(lm_alias(empty_zero_page));
set_page_mte_tagged(ZERO_PAGE(0));
}
kasan_init_hw_tags_cpu();
}

Some files were not shown because too many files have changed in this diff Show More