Merge "Merge 3822a7c409 ("Merge tag 'mm-stable-2023-02-20-13-37' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm") into android-mainline" into android-mainline
This commit is contained in:
Treehugger Robot
@@ -182,3 +182,42 @@ Date: November 2021
|
||||
Contact: Jarkko Sakkinen <jarkko@kernel.org>
|
||||
Description:
|
||||
The total amount of SGX physical memory in bytes.
|
||||
|
||||
What: /sys/devices/system/node/nodeX/memory_failure/total
|
||||
Date: January 2023
|
||||
Contact: Jiaqi Yan <jiaqiyan@google.com>
|
||||
Description:
|
||||
The total number of raw poisoned pages (pages containing
|
||||
corrupted data due to memory errors) on a NUMA node.
|
||||
|
||||
What: /sys/devices/system/node/nodeX/memory_failure/ignored
|
||||
Date: January 2023
|
||||
Contact: Jiaqi Yan <jiaqiyan@google.com>
|
||||
Description:
|
||||
Of the raw poisoned pages on a NUMA node, how many pages are
|
||||
ignored by memory error recovery attempt, usually because
|
||||
support for this type of pages is unavailable, and kernel
|
||||
gives up the recovery.
|
||||
|
||||
What: /sys/devices/system/node/nodeX/memory_failure/failed
|
||||
Date: January 2023
|
||||
Contact: Jiaqi Yan <jiaqiyan@google.com>
|
||||
Description:
|
||||
Of the raw poisoned pages on a NUMA node, how many pages are
|
||||
failed by memory error recovery attempt. This usually means
|
||||
a key recovery operation failed.
|
||||
|
||||
What: /sys/devices/system/node/nodeX/memory_failure/delayed
|
||||
Date: January 2023
|
||||
Contact: Jiaqi Yan <jiaqiyan@google.com>
|
||||
Description:
|
||||
Of the raw poisoned pages on a NUMA node, how many pages are
|
||||
delayed by memory error recovery attempt. Delayed poisoned
|
||||
pages usually will be retried by kernel.
|
||||
|
||||
What: /sys/devices/system/node/nodeX/memory_failure/recovered
|
||||
Date: January 2023
|
||||
Contact: Jiaqi Yan <jiaqiyan@google.com>
|
||||
Description:
|
||||
Of the raw poisoned pages on a NUMA node, how many pages are
|
||||
recovered by memory error recovery attempt.
|
||||
|
||||
@@ -258,6 +258,35 @@ Contact: SeongJae Park <sj@kernel.org>
|
||||
Description: Writing to and reading from this file sets and gets the low
|
||||
watermark of the scheme in permil.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/nr_filters
|
||||
Date: Dec 2022
|
||||
Contact: SeongJae Park <sj@kernel.org>
|
||||
Description: Writing a number 'N' to this file creates the number of
|
||||
directories for setting filters of the scheme named '0' to
|
||||
'N-1' under the filters/ directory.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/type
|
||||
Date: Dec 2022
|
||||
Contact: SeongJae Park <sj@kernel.org>
|
||||
Description: Writing to and reading from this file sets and gets the type of
|
||||
the memory of the interest. 'anon' for anonymous pages, or
|
||||
'memcg' for specific memory cgroup can be written and read.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/memcg_path
|
||||
Date: Dec 2022
|
||||
Contact: SeongJae Park <sj@kernel.org>
|
||||
Description: If 'memcg' is written to the 'type' file, writing to and
|
||||
reading from this file sets and gets the path to the memory
|
||||
cgroup of the interest.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/matching
|
||||
Date: Dec 2022
|
||||
Contact: SeongJae Park <sj@kernel.org>
|
||||
Description: Writing 'Y' or 'N' to this file sets whether to filter out
|
||||
pages that do or do not match to the 'type' and 'memcg_path',
|
||||
respectively. Filter out means the action of the scheme will
|
||||
not be applied to.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/nr_tried
|
||||
Date: Mar 2022
|
||||
Contact: SeongJae Park <sj@kernel.org>
|
||||
|
||||
@@ -87,6 +87,8 @@ Brief summary of control files.
|
||||
memory.swappiness set/show swappiness parameter of vmscan
|
||||
(See sysctl's vm.swappiness)
|
||||
memory.move_charge_at_immigrate set/show controls of moving charges
|
||||
This knob is deprecated and shouldn't be
|
||||
used.
|
||||
memory.oom_control set/show oom controls.
|
||||
memory.numa_stat show the number of memory usage per numa
|
||||
node
|
||||
@@ -727,8 +729,15 @@ If we want to change this to 1G, we can at any time use::
|
||||
|
||||
.. _cgroup-v1-memory-move-charges:
|
||||
|
||||
8. Move charges at task migration
|
||||
=================================
|
||||
8. Move charges at task migration (DEPRECATED!)
|
||||
===============================================
|
||||
|
||||
THIS IS DEPRECATED!
|
||||
|
||||
It's expensive and unreliable! It's better practice to launch workload
|
||||
tasks directly from inside their target cgroup. Use dedicated workload
|
||||
cgroups to allow fine-grained policy adjustments without having to
|
||||
move physical pages between control domains.
|
||||
|
||||
Users can move charges associated with a task along with task migration, that
|
||||
is, uncharge task's pages from the old cgroup and charge them to the new cgroup.
|
||||
|
||||
@@ -205,6 +205,15 @@ The end physical address of memory region that DAMON_RECLAIM will do work
|
||||
against. That is, DAMON_RECLAIM will find cold memory regions in this region
|
||||
and reclaims. By default, biggest System RAM is used as the region.
|
||||
|
||||
skip_anon
|
||||
---------
|
||||
|
||||
Skip anonymous pages reclamation.
|
||||
|
||||
If this parameter is set as ``Y``, DAMON_RECLAIM does not reclaim anonymous
|
||||
pages. By default, ``N``.
|
||||
|
||||
|
||||
kdamond_pid
|
||||
-----------
|
||||
|
||||
|
||||
@@ -25,10 +25,12 @@ DAMON provides below interfaces for different users.
|
||||
interface provides only simple :ref:`statistics <damos_stats>` for the
|
||||
monitoring results. For detailed monitoring results, DAMON provides a
|
||||
:ref:`tracepoint <tracepoint>`.
|
||||
- *debugfs interface.*
|
||||
- *debugfs interface. (DEPRECATED!)*
|
||||
:ref:`This <debugfs_interface>` is almost identical to :ref:`sysfs interface
|
||||
<sysfs_interface>`. This will be removed after next LTS kernel is released,
|
||||
so users should move to the :ref:`sysfs interface <sysfs_interface>`.
|
||||
<sysfs_interface>`. This is deprecated, so users should move to the
|
||||
:ref:`sysfs interface <sysfs_interface>`. If you depend on this and cannot
|
||||
move, please report your usecase to damon@lists.linux.dev and
|
||||
linux-mm@kvack.org.
|
||||
- *Kernel Space Programming Interface.*
|
||||
:doc:`This </mm/damon/api>` is for kernel space programmers. Using this,
|
||||
users can utilize every feature of DAMON most flexibly and efficiently by
|
||||
@@ -87,6 +89,8 @@ comma (","). ::
|
||||
│ │ │ │ │ │ │ quotas/ms,bytes,reset_interval_ms
|
||||
│ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil
|
||||
│ │ │ │ │ │ │ watermarks/metric,interval_us,high,mid,low
|
||||
│ │ │ │ │ │ │ filters/nr_filters
|
||||
│ │ │ │ │ │ │ │ 0/type,matching,memcg_id
|
||||
│ │ │ │ │ │ │ stats/nr_tried,sz_tried,nr_applied,sz_applied,qt_exceeds
|
||||
│ │ │ │ │ │ │ tried_regions/
|
||||
│ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age
|
||||
@@ -151,6 +155,8 @@ number (``N``) to the file creates the number of child directories named as
|
||||
moment, only one context per kdamond is supported, so only ``0`` or ``1`` can
|
||||
be written to the file.
|
||||
|
||||
.. _sysfs_contexts:
|
||||
|
||||
contexts/<N>/
|
||||
-------------
|
||||
|
||||
@@ -268,21 +274,32 @@ schemes/<N>/
|
||||
------------
|
||||
|
||||
In each scheme directory, five directories (``access_pattern``, ``quotas``,
|
||||
``watermarks``, ``stats``, and ``tried_regions``) and one file (``action``)
|
||||
exist.
|
||||
``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and one file
|
||||
(``action``) exist.
|
||||
|
||||
The ``action`` file is for setting and getting what action you want to apply to
|
||||
memory regions having specific access pattern of the interest. The keywords
|
||||
that can be written to and read from the file and their meaning are as below.
|
||||
|
||||
- ``willneed``: Call ``madvise()`` for the region with ``MADV_WILLNEED``
|
||||
- ``cold``: Call ``madvise()`` for the region with ``MADV_COLD``
|
||||
- ``pageout``: Call ``madvise()`` for the region with ``MADV_PAGEOUT``
|
||||
- ``hugepage``: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``
|
||||
- ``nohugepage``: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``
|
||||
Note that support of each action depends on the running DAMON operations set
|
||||
`implementation <sysfs_contexts>`.
|
||||
|
||||
- ``willneed``: Call ``madvise()`` for the region with ``MADV_WILLNEED``.
|
||||
Supported by ``vaddr`` and ``fvaddr`` operations set.
|
||||
- ``cold``: Call ``madvise()`` for the region with ``MADV_COLD``.
|
||||
Supported by ``vaddr`` and ``fvaddr`` operations set.
|
||||
- ``pageout``: Call ``madvise()`` for the region with ``MADV_PAGEOUT``.
|
||||
Supported by ``vaddr``, ``fvaddr`` and ``paddr`` operations set.
|
||||
- ``hugepage``: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``.
|
||||
Supported by ``vaddr`` and ``fvaddr`` operations set.
|
||||
- ``nohugepage``: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``.
|
||||
Supported by ``vaddr`` and ``fvaddr`` operations set.
|
||||
- ``lru_prio``: Prioritize the region on its LRU lists.
|
||||
Supported by ``paddr`` operations set.
|
||||
- ``lru_deprio``: Deprioritize the region on its LRU lists.
|
||||
- ``stat``: Do nothing but count the statistics
|
||||
Supported by ``paddr`` operations set.
|
||||
- ``stat``: Do nothing but count the statistics.
|
||||
Supported by all operations sets.
|
||||
|
||||
schemes/<N>/access_pattern/
|
||||
---------------------------
|
||||
@@ -347,6 +364,46 @@ as below.
|
||||
|
||||
The ``interval`` should written in microseconds unit.
|
||||
|
||||
schemes/<N>/filters/
|
||||
--------------------
|
||||
|
||||
Users could know something more than the kernel for specific types of memory.
|
||||
In the case, users could do their own management for the memory and hence
|
||||
doesn't want DAMOS bothers that. Users could limit DAMOS by setting the access
|
||||
pattern of the scheme and/or the monitoring regions for the purpose, but that
|
||||
can be inefficient in some cases. In such cases, users could set non-access
|
||||
pattern driven filters using files in this directory.
|
||||
|
||||
In the beginning, this directory has only one file, ``nr_filters``. Writing a
|
||||
number (``N``) to the file creates the number of child directories named ``0``
|
||||
to ``N-1``. Each directory represents each filter. The filters are evaluated
|
||||
in the numeric order.
|
||||
|
||||
Each filter directory contains three files, namely ``type``, ``matcing``, and
|
||||
``memcg_path``. You can write one of two special keywords, ``anon`` for
|
||||
anonymous pages, or ``memcg`` for specific memory cgroup filtering. In case of
|
||||
the memory cgroup filtering, you can specify the memory cgroup of the interest
|
||||
by writing the path of the memory cgroup from the cgroups mount point to
|
||||
``memcg_path`` file. You can write ``Y`` or ``N`` to ``matching`` file to
|
||||
filter out pages that does or does not match to the type, respectively. Then,
|
||||
the scheme's action will not be applied to the pages that specified to be
|
||||
filtered out.
|
||||
|
||||
For example, below restricts a DAMOS action to be applied to only non-anonymous
|
||||
pages of all memory cgroups except ``/having_care_already``.::
|
||||
|
||||
# echo 2 > nr_filters
|
||||
# # filter out anonymous pages
|
||||
echo anon > 0/type
|
||||
echo Y > 0/matching
|
||||
# # further filter out all cgroups except one at '/having_care_already'
|
||||
echo memcg > 1/type
|
||||
echo /having_care_already > 1/memcg_path
|
||||
echo N > 1/matching
|
||||
|
||||
Note that filters are currently supported only when ``paddr``
|
||||
`implementation <sysfs_contexts>` is being used.
|
||||
|
||||
.. _sysfs_schemes_stats:
|
||||
|
||||
schemes/<N>/stats/
|
||||
@@ -432,13 +489,17 @@ the files as above. Above is only for an example.
|
||||
|
||||
.. _debugfs_interface:
|
||||
|
||||
debugfs Interface
|
||||
=================
|
||||
debugfs Interface (DEPRECATED!)
|
||||
===============================
|
||||
|
||||
.. note::
|
||||
|
||||
DAMON debugfs interface will be removed after next LTS kernel is released, so
|
||||
users should move to the :ref:`sysfs interface <sysfs_interface>`.
|
||||
THIS IS DEPRECATED!
|
||||
|
||||
DAMON debugfs interface is deprecated, so users should move to the
|
||||
:ref:`sysfs interface <sysfs_interface>`. If you depend on this and cannot
|
||||
move, please report your usecase to damon@lists.linux.dev and
|
||||
linux-mm@kvack.org.
|
||||
|
||||
DAMON exports eight files, ``attrs``, ``target_ids``, ``init_regions``,
|
||||
``schemes``, ``monitor_on``, ``kdamond_pid``, ``mk_contexts`` and
|
||||
@@ -574,11 +635,15 @@ The ``<action>`` is a predefined integer for memory management actions, which
|
||||
DAMON will apply to the regions having the target access pattern. The
|
||||
supported numbers and their meanings are as below.
|
||||
|
||||
- 0: Call ``madvise()`` for the region with ``MADV_WILLNEED``
|
||||
- 1: Call ``madvise()`` for the region with ``MADV_COLD``
|
||||
- 2: Call ``madvise()`` for the region with ``MADV_PAGEOUT``
|
||||
- 3: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``
|
||||
- 4: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``
|
||||
- 0: Call ``madvise()`` for the region with ``MADV_WILLNEED``. Ignored if
|
||||
``target`` is ``paddr``.
|
||||
- 1: Call ``madvise()`` for the region with ``MADV_COLD``. Ignored if
|
||||
``target`` is ``paddr``.
|
||||
- 2: Call ``madvise()`` for the region with ``MADV_PAGEOUT``.
|
||||
- 3: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``. Ignored if
|
||||
``target`` is ``paddr``.
|
||||
- 4: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``. Ignored if
|
||||
``target`` is ``paddr``.
|
||||
- 5: Do nothing but count the statistics
|
||||
|
||||
Quota
|
||||
|
||||
@@ -459,13 +459,13 @@ Examples
|
||||
.. _map_hugetlb:
|
||||
|
||||
``map_hugetlb``
|
||||
see tools/testing/selftests/vm/map_hugetlb.c
|
||||
see tools/testing/selftests/mm/map_hugetlb.c
|
||||
|
||||
``hugepage-shm``
|
||||
see tools/testing/selftests/vm/hugepage-shm.c
|
||||
see tools/testing/selftests/mm/hugepage-shm.c
|
||||
|
||||
``hugepage-mmap``
|
||||
see tools/testing/selftests/vm/hugepage-mmap.c
|
||||
see tools/testing/selftests/mm/hugepage-mmap.c
|
||||
|
||||
The `libhugetlbfs`_ library provides a wide range of userspace tools
|
||||
to help with huge page usability, environment setup, and control.
|
||||
|
||||
@@ -63,7 +63,7 @@ workload one should:
|
||||
are not reclaimable, he or she can filter them out using
|
||||
``/proc/kpageflags``.
|
||||
|
||||
The page-types tool in the tools/vm directory can be used to assist in this.
|
||||
The page-types tool in the tools/mm directory can be used to assist in this.
|
||||
If the tool is run initially with the appropriate option, it will mark all the
|
||||
queried pages as idle. Subsequent runs of the tool can then show which pages have
|
||||
their idle flag cleared in the interim.
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
=============
|
||||
=======================
|
||||
NUMA Memory Performance
|
||||
=======================
|
||||
|
||||
NUMA Locality
|
||||
=============
|
||||
|
||||
@@ -59,7 +62,6 @@ that are CPUs and hence suitable for generic task scheduling, and
|
||||
IO initiators such as GPUs and NICs. Unlike access class 0, only
|
||||
nodes containing CPUs are considered.
|
||||
|
||||
================
|
||||
NUMA Performance
|
||||
================
|
||||
|
||||
@@ -94,7 +96,6 @@ for the platform.
|
||||
Access class 1 takes the same form but only includes values for CPU to
|
||||
memory activity.
|
||||
|
||||
==========
|
||||
NUMA Cache
|
||||
==========
|
||||
|
||||
@@ -168,7 +169,6 @@ The "size" is the number of bytes provided by this cache level.
|
||||
The "write_policy" will be 0 for write-back, and non-zero for
|
||||
write-through caching.
|
||||
|
||||
========
|
||||
See Also
|
||||
========
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ There are four components to pagemap:
|
||||
* ``/proc/kpagecount``. This file contains a 64-bit count of the number of
|
||||
times each page is mapped, indexed by PFN.
|
||||
|
||||
The page-types tool in the tools/vm directory can be used to query the
|
||||
The page-types tool in the tools/mm directory can be used to query the
|
||||
number of times a page is mapped.
|
||||
|
||||
* ``/proc/kpageflags``. This file contains a 64-bit set of flags for each
|
||||
@@ -170,7 +170,7 @@ LRU related page flags
|
||||
14 - SWAPBACKED
|
||||
The page is backed by swap/RAM.
|
||||
|
||||
The page-types tool in the tools/vm directory can be used to query the
|
||||
The page-types tool in the tools/mm directory can be used to query the
|
||||
above flags.
|
||||
|
||||
Using pagemap to do something useful
|
||||
|
||||
@@ -55,18 +55,17 @@ flags the caller provides. The caller is required to pass in a non-null struct
|
||||
pages* array, and the function then pins pages by incrementing each by a special
|
||||
value: GUP_PIN_COUNTING_BIAS.
|
||||
|
||||
For compound pages, the GUP_PIN_COUNTING_BIAS scheme is not used. Instead,
|
||||
an exact form of pin counting is achieved, by using the 2nd struct page
|
||||
in the compound page. A new struct page field, compound_pincount, has
|
||||
been added in order to support this.
|
||||
For large folios, the GUP_PIN_COUNTING_BIAS scheme is not used. Instead,
|
||||
the extra space available in the struct folio is used to store the
|
||||
pincount directly.
|
||||
|
||||
This approach for compound pages avoids the counting upper limit problems that
|
||||
are discussed below. Those limitations would have been aggravated severely by
|
||||
huge pages, because each tail page adds a refcount to the head page. And in
|
||||
fact, testing revealed that, without a separate compound_pincount field,
|
||||
page overflows were seen in some huge page stress tests.
|
||||
This approach for large folios avoids the counting upper limit problems
|
||||
that are discussed below. Those limitations would have been aggravated
|
||||
severely by huge pages, because each tail page adds a refcount to the
|
||||
head page. And in fact, testing revealed that, without a separate pincount
|
||||
field, refcount overflows were seen in some huge page stress tests.
|
||||
|
||||
This also means that huge pages and compound pages do not suffer
|
||||
This also means that huge pages and large folios do not suffer
|
||||
from the false positives problem that is mentioned below.::
|
||||
|
||||
Function
|
||||
@@ -221,7 +220,7 @@ Unit testing
|
||||
============
|
||||
This file::
|
||||
|
||||
tools/testing/selftests/vm/gup_test.c
|
||||
tools/testing/selftests/mm/gup_test.c
|
||||
|
||||
has the following new calls to exercise the new pin*() wrapper functions:
|
||||
|
||||
@@ -264,9 +263,9 @@ place.)
|
||||
Other diagnostics
|
||||
=================
|
||||
|
||||
dump_page() has been enhanced slightly, to handle these new counting
|
||||
fields, and to better report on compound pages in general. Specifically,
|
||||
for compound pages, the exact (compound_pincount) pincount is reported.
|
||||
dump_page() has been enhanced slightly to handle these new counting
|
||||
fields, and to better report on large folios in general. Specifically,
|
||||
for large folios, the exact pincount is reported.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
@@ -140,6 +140,23 @@ disabling KASAN altogether or controlling its features:
|
||||
- ``kasan.vmalloc=off`` or ``=on`` disables or enables tagging of vmalloc
|
||||
allocations (default: ``on``).
|
||||
|
||||
- ``kasan.page_alloc.sample=<sampling interval>`` makes KASAN tag only every
|
||||
Nth page_alloc allocation with the order equal or greater than
|
||||
``kasan.page_alloc.sample.order``, where N is the value of the ``sample``
|
||||
parameter (default: ``1``, or tag every such allocation).
|
||||
This parameter is intended to mitigate the performance overhead introduced
|
||||
by KASAN.
|
||||
Note that enabling this parameter makes Hardware Tag-Based KASAN skip checks
|
||||
of allocations chosen by sampling and thus miss bad accesses to these
|
||||
allocations. Use the default value for accurate bug detection.
|
||||
|
||||
- ``kasan.page_alloc.sample.order=<minimum page order>`` specifies the minimum
|
||||
order of allocations that are affected by sampling (default: ``3``).
|
||||
Only applies when ``kasan.page_alloc.sample`` is set to a value greater
|
||||
than ``1``.
|
||||
This parameter is intended to allow sampling only large page_alloc
|
||||
allocations, which is the biggest source of the performance overhead.
|
||||
|
||||
Error reports
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ Memory Balancing
|
||||
|
||||
Started Jan 2000 by Kanoj Sarcar <kanoj@sgi.com>
|
||||
|
||||
Memory balancing is needed for !__GFP_ATOMIC and !__GFP_KSWAPD_RECLAIM as
|
||||
Memory balancing is needed for !__GFP_HIGH and !__GFP_KSWAPD_RECLAIM as
|
||||
well as for non __GFP_IO allocations.
|
||||
|
||||
The first reason why a caller may avoid reclaim is that the caller can not
|
||||
|
||||
@@ -4,8 +4,9 @@
|
||||
DAMON: Data Access MONitor
|
||||
==========================
|
||||
|
||||
DAMON is a data access monitoring framework subsystem for the Linux kernel.
|
||||
The core mechanisms of DAMON (refer to :doc:`design` for the detail) make it
|
||||
DAMON is a Linux kernel subsystem that provides a framework for data access
|
||||
monitoring and the monitoring results based system operations. The core
|
||||
monitoring mechanisms of DAMON (refer to :doc:`design` for the detail) make it
|
||||
|
||||
- *accurate* (the monitoring output is useful enough for DRAM level memory
|
||||
management; It might not appropriate for CPU Cache levels, though),
|
||||
@@ -14,12 +15,16 @@ The core mechanisms of DAMON (refer to :doc:`design` for the detail) make it
|
||||
- *scalable* (the upper-bound of the overhead is in constant range regardless
|
||||
of the size of target workloads).
|
||||
|
||||
Using this framework, therefore, the kernel's memory management mechanisms can
|
||||
make advanced decisions. Experimental memory management optimization works
|
||||
that incurring high data accesses monitoring overhead could implemented again.
|
||||
In user space, meanwhile, users who have some special workloads can write
|
||||
personalized applications for better understanding and optimizations of their
|
||||
workloads and systems.
|
||||
Using this framework, therefore, the kernel can operate system in an
|
||||
access-aware fashion. Because the features are also exposed to the user space,
|
||||
users who have special information about their workloads can write personalized
|
||||
applications for better understanding and optimizations of their workloads and
|
||||
systems.
|
||||
|
||||
For easier development of such systems, DAMON provides a feature called DAMOS
|
||||
(DAMon-based Operation Schemes) in addition to the monitoring. Using the
|
||||
feature, DAMON users in both kernel and user spaces can do access-aware system
|
||||
operations with no code but simple configurations.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
@@ -27,3 +32,4 @@ workloads and systems.
|
||||
faq
|
||||
design
|
||||
api
|
||||
maintainer-profile
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
.. SPDX-License-Identifier: GPL-2.0
|
||||
|
||||
DAMON Maintainer Entry Profile
|
||||
==============================
|
||||
|
||||
The DAMON subsystem covers the files that listed in 'DATA ACCESS MONITOR'
|
||||
section of 'MAINTAINERS' file.
|
||||
|
||||
The mailing lists for the subsystem are damon@lists.linux.dev and
|
||||
linux-mm@kvack.org. Patches should be made against the mm-unstable tree [1]_
|
||||
whenever possible and posted to the mailing lists.
|
||||
|
||||
SCM Trees
|
||||
---------
|
||||
|
||||
There are multiple Linux trees for DAMON development. Patches under
|
||||
development or testing are queued in damon/next [2]_ by the DAMON maintainer.
|
||||
Suffieicntly reviewed patches will be queued in mm-unstable [1]_ by the memory
|
||||
management subsystem maintainer. After more sufficient tests, the patches will
|
||||
be queued in mm-stable [3]_ , and finally pull-requested to the mainline by the
|
||||
memory management subsystem maintainer.
|
||||
|
||||
Note again the patches for review should be made against the mm-unstable
|
||||
tree[1] whenever possible. damon/next is only for preview of others' works in
|
||||
progress.
|
||||
|
||||
Submit checklist addendum
|
||||
-------------------------
|
||||
|
||||
When making DAMON changes, you should do below.
|
||||
|
||||
- Build changes related outputs including kernel and documents.
|
||||
- Ensure the builds introduce no new errors or warnings.
|
||||
- Run and ensure no new failures for DAMON selftests [4]_ and kunittests [5]_ .
|
||||
|
||||
Further doing below and putting the results will be helpful.
|
||||
|
||||
- Run damon-tests/corr [6]_ for normal changes.
|
||||
- Run damon-tests/perf [7]_ for performance changes.
|
||||
|
||||
Key cycle dates
|
||||
---------------
|
||||
|
||||
Patches can be sent anytime. Key cycle dates of the mm-unstable[1] and
|
||||
mm-stable[3] trees depend on the memory management subsystem maintainer.
|
||||
|
||||
Review cadence
|
||||
--------------
|
||||
|
||||
The DAMON maintainer does the work on the usual work hour (09:00 to 17:00,
|
||||
Mon-Fri) in PST. The response to patches will occasionally be slow. Do not
|
||||
hesitate to send a ping if you have not heard back within a week of sending a
|
||||
patch.
|
||||
|
||||
|
||||
.. [1] https://git.kernel.org/akpm/mm/h/mm-unstable
|
||||
.. [2] https://git.kernel.org/sj/h/damon/next
|
||||
.. [3] https://git.kernel.org/akpm/mm/h/mm-stable
|
||||
.. [4] https://github.com/awslabs/damon-tests/blob/master/corr/run.sh#L49
|
||||
.. [5] https://github.com/awslabs/damon-tests/blob/master/corr/tests/kunit.sh
|
||||
.. [6] https://github.com/awslabs/damon-tests/tree/master/corr
|
||||
.. [7] https://github.com/awslabs/damon-tests/tree/master/perf
|
||||
@@ -55,7 +55,8 @@ list shows them in order of preference of use.
|
||||
It can be invoked from any context (including interrupts) but the mappings
|
||||
can only be used in the context which acquired them.
|
||||
|
||||
This function should be preferred, where feasible, over all the others.
|
||||
This function should always be used, whereas kmap_atomic() and kmap() have
|
||||
been deprecated.
|
||||
|
||||
These mappings are thread-local and CPU-local, meaning that the mapping
|
||||
can only be accessed from within this thread and the thread is bound to the
|
||||
@@ -80,7 +81,7 @@ list shows them in order of preference of use.
|
||||
for pages which are known to not come from ZONE_HIGHMEM. However, it is
|
||||
always safe to use kmap_local_page() / kunmap_local().
|
||||
|
||||
While it is significantly faster than kmap(), for the higmem case it
|
||||
While it is significantly faster than kmap(), for the highmem case it
|
||||
comes with restrictions about the pointers validity. Contrary to kmap()
|
||||
mappings, the local mappings are only valid in the context of the caller
|
||||
and cannot be handed to other contexts. This implies that users must
|
||||
@@ -98,10 +99,21 @@ list shows them in order of preference of use.
|
||||
(included in the "Functions" section) for details on how to manage nested
|
||||
mappings.
|
||||
|
||||
* kmap_atomic(). This permits a very short duration mapping of a single
|
||||
page. Since the mapping is restricted to the CPU that issued it, it
|
||||
performs well, but the issuing task is therefore required to stay on that
|
||||
CPU until it has finished, lest some other task displace its mappings.
|
||||
* kmap_atomic(). This function has been deprecated; use kmap_local_page().
|
||||
|
||||
NOTE: Conversions to kmap_local_page() must take care to follow the mapping
|
||||
restrictions imposed on kmap_local_page(). Furthermore, the code between
|
||||
calls to kmap_atomic() and kunmap_atomic() may implicitly depend on the side
|
||||
effects of atomic mappings, i.e. disabling page faults or preemption, or both.
|
||||
In that case, explicit calls to pagefault_disable() or preempt_disable() or
|
||||
both must be made in conjunction with the use of kmap_local_page().
|
||||
|
||||
[Legacy documentation]
|
||||
|
||||
This permits a very short duration mapping of a single page. Since the
|
||||
mapping is restricted to the CPU that issued it, it performs well, but
|
||||
the issuing task is therefore required to stay on that CPU until it has
|
||||
finished, lest some other task displace its mappings.
|
||||
|
||||
kmap_atomic() may also be used by interrupt contexts, since it does not
|
||||
sleep and the callers too may not sleep until after kunmap_atomic() is
|
||||
@@ -113,11 +125,20 @@ list shows them in order of preference of use.
|
||||
|
||||
It is assumed that k[un]map_atomic() won't fail.
|
||||
|
||||
* kmap(). This should be used to make short duration mapping of a single
|
||||
page with no restrictions on preemption or migration. It comes with an
|
||||
overhead as mapping space is restricted and protected by a global lock
|
||||
for synchronization. When mapping is no longer needed, the address that
|
||||
the page was mapped to must be released with kunmap().
|
||||
* kmap(). This function has been deprecated; use kmap_local_page().
|
||||
|
||||
NOTE: Conversions to kmap_local_page() must take care to follow the mapping
|
||||
restrictions imposed on kmap_local_page(). In particular, it is necessary to
|
||||
make sure that the kernel virtual memory pointer is only valid in the thread
|
||||
that obtained it.
|
||||
|
||||
[Legacy documentation]
|
||||
|
||||
This should be used to make short duration mapping of a single page with no
|
||||
restrictions on preemption or migration. It comes with an overhead as mapping
|
||||
space is restricted and protected by a global lock for synchronization. When
|
||||
mapping is no longer needed, the address that the page was mapped to must be
|
||||
released with kunmap().
|
||||
|
||||
Mapping changes must be propagated across all the CPUs. kmap() also
|
||||
requires global TLB invalidation when the kmap's pool wraps and it might
|
||||
|
||||
@@ -179,14 +179,14 @@ Consuming Reservations/Allocating a Huge Page
|
||||
|
||||
Reservations are consumed when huge pages associated with the reservations
|
||||
are allocated and instantiated in the corresponding mapping. The allocation
|
||||
is performed within the routine alloc_huge_page()::
|
||||
is performed within the routine alloc_hugetlb_folio()::
|
||||
|
||||
struct page *alloc_huge_page(struct vm_area_struct *vma,
|
||||
struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
|
||||
unsigned long addr, int avoid_reserve)
|
||||
|
||||
alloc_huge_page is passed a VMA pointer and a virtual address, so it can
|
||||
alloc_hugetlb_folio is passed a VMA pointer and a virtual address, so it can
|
||||
consult the reservation map to determine if a reservation exists. In addition,
|
||||
alloc_huge_page takes the argument avoid_reserve which indicates reserves
|
||||
alloc_hugetlb_folio takes the argument avoid_reserve which indicates reserves
|
||||
should not be used even if it appears they have been set aside for the
|
||||
specified address. The avoid_reserve argument is most often used in the case
|
||||
of Copy on Write and Page Migration where additional copies of an existing
|
||||
@@ -206,7 +206,8 @@ a reservation for the allocation. After determining whether a reservation
|
||||
exists and can be used for the allocation, the routine dequeue_huge_page_vma()
|
||||
is called. This routine takes two arguments related to reservations:
|
||||
|
||||
- avoid_reserve, this is the same value/argument passed to alloc_huge_page()
|
||||
- avoid_reserve, this is the same value/argument passed to
|
||||
alloc_hugetlb_folio().
|
||||
- chg, even though this argument is of type long only the values 0 or 1 are
|
||||
passed to dequeue_huge_page_vma. If the value is 0, it indicates a
|
||||
reservation exists (see the section "Memory Policy and Reservations" for
|
||||
@@ -231,9 +232,9 @@ the scope reservations. Even if a surplus page is allocated, the same
|
||||
reservation based adjustments as above will be made: SetPagePrivate(page) and
|
||||
resv_huge_pages--.
|
||||
|
||||
After obtaining a new huge page, (page)->private is set to the value of
|
||||
the subpool associated with the page if it exists. This will be used for
|
||||
subpool accounting when the page is freed.
|
||||
After obtaining a new hugetlb folio, (folio)->_hugetlb_subpool is set to the
|
||||
value of the subpool associated with the page if it exists. This will be used
|
||||
for subpool accounting when the folio is freed.
|
||||
|
||||
The routine vma_commit_reservation() is then called to adjust the reserve
|
||||
map based on the consumption of the reservation. In general, this involves
|
||||
@@ -244,8 +245,8 @@ was no reservation in a shared mapping or this was a private mapping a new
|
||||
entry must be created.
|
||||
|
||||
It is possible that the reserve map could have been changed between the call
|
||||
to vma_needs_reservation() at the beginning of alloc_huge_page() and the
|
||||
call to vma_commit_reservation() after the page was allocated. This would
|
||||
to vma_needs_reservation() at the beginning of alloc_hugetlb_folio() and the
|
||||
call to vma_commit_reservation() after the folio was allocated. This would
|
||||
be possible if hugetlb_reserve_pages was called for the same page in a shared
|
||||
mapping. In such cases, the reservation count and subpool free page count
|
||||
will be off by one. This rare condition can be identified by comparing the
|
||||
|
||||
@@ -89,15 +89,15 @@ variables are monotonically increasing.
|
||||
|
||||
Generation numbers are truncated into ``order_base_2(MAX_NR_GENS+1)``
|
||||
bits in order to fit into the gen counter in ``folio->flags``. Each
|
||||
truncated generation number is an index to ``lrugen->lists[]``. The
|
||||
truncated generation number is an index to ``lrugen->folios[]``. The
|
||||
sliding window technique is used to track at least ``MIN_NR_GENS`` and
|
||||
at most ``MAX_NR_GENS`` generations. The gen counter stores a value
|
||||
within ``[1, MAX_NR_GENS]`` while a page is on one of
|
||||
``lrugen->lists[]``; otherwise it stores zero.
|
||||
``lrugen->folios[]``; otherwise it stores zero.
|
||||
|
||||
Each generation is divided into multiple tiers. A page accessed ``N``
|
||||
times through file descriptors is in tier ``order_base_2(N)``. Unlike
|
||||
generations, tiers do not have dedicated ``lrugen->lists[]``. In
|
||||
generations, tiers do not have dedicated ``lrugen->folios[]``. In
|
||||
contrast to moving across generations, which requires the LRU lock,
|
||||
moving across tiers only involves atomic operations on
|
||||
``folio->flags`` and therefore has a negligible cost. A feedback loop
|
||||
@@ -127,7 +127,7 @@ page mapped by this PTE to ``(max_seq%MAX_NR_GENS)+1``.
|
||||
Eviction
|
||||
--------
|
||||
The eviction consumes old generations. Given an ``lruvec``, it
|
||||
increments ``min_seq`` when ``lrugen->lists[]`` indexed by
|
||||
increments ``min_seq`` when ``lrugen->folios[]`` indexed by
|
||||
``min_seq%MAX_NR_GENS`` becomes empty. To select a type and a tier to
|
||||
evict from, it first compares ``min_seq[]`` to select the older type.
|
||||
If both types are equally old, it selects the one whose first tier has
|
||||
@@ -141,9 +141,85 @@ loop has detected outlying refaults from the tier this page is in. To
|
||||
this end, the feedback loop uses the first tier as the baseline, for
|
||||
the reason stated earlier.
|
||||
|
||||
Working set protection
|
||||
----------------------
|
||||
Each generation is timestamped at birth. If ``lru_gen_min_ttl`` is
|
||||
set, an ``lruvec`` is protected from the eviction when its oldest
|
||||
generation was born within ``lru_gen_min_ttl`` milliseconds. In other
|
||||
words, it prevents the working set of ``lru_gen_min_ttl`` milliseconds
|
||||
from getting evicted. The OOM killer is triggered if this working set
|
||||
cannot be kept in memory.
|
||||
|
||||
This time-based approach has the following advantages:
|
||||
|
||||
1. It is easier to configure because it is agnostic to applications
|
||||
and memory sizes.
|
||||
2. It is more reliable because it is directly wired to the OOM killer.
|
||||
|
||||
Rmap/PT walk feedback
|
||||
---------------------
|
||||
Searching the rmap for PTEs mapping each page on an LRU list (to test
|
||||
and clear the accessed bit) can be expensive because pages from
|
||||
different VMAs (PA space) are not cache friendly to the rmap (VA
|
||||
space). For workloads mostly using mapped pages, searching the rmap
|
||||
can incur the highest CPU cost in the reclaim path.
|
||||
|
||||
``lru_gen_look_around()`` exploits spatial locality to reduce the
|
||||
trips into the rmap. It scans the adjacent PTEs of a young PTE and
|
||||
promotes hot pages. If the scan was done cacheline efficiently, it
|
||||
adds the PMD entry pointing to the PTE table to the Bloom filter. This
|
||||
forms a feedback loop between the eviction and the aging.
|
||||
|
||||
Bloom Filters
|
||||
-------------
|
||||
Bloom filters are a space and memory efficient data structure for set
|
||||
membership test, i.e., test if an element is not in the set or may be
|
||||
in the set.
|
||||
|
||||
In the eviction path, specifically, in ``lru_gen_look_around()``, if a
|
||||
PMD has a sufficient number of hot pages, its address is placed in the
|
||||
filter. In the aging path, set membership means that the PTE range
|
||||
will be scanned for young pages.
|
||||
|
||||
Note that Bloom filters are probabilistic on set membership. If a test
|
||||
is false positive, the cost is an additional scan of a range of PTEs,
|
||||
which may yield hot pages anyway. Parameters of the filter itself can
|
||||
control the false positive rate in the limit.
|
||||
|
||||
Memcg LRU
|
||||
---------
|
||||
An memcg LRU is a per-node LRU of memcgs. It is also an LRU of LRUs,
|
||||
since each node and memcg combination has an LRU of folios (see
|
||||
``mem_cgroup_lruvec()``). Its goal is to improve the scalability of
|
||||
global reclaim, which is critical to system-wide memory overcommit in
|
||||
data centers. Note that memcg LRU only applies to global reclaim.
|
||||
|
||||
The basic structure of an memcg LRU can be understood by an analogy to
|
||||
the active/inactive LRU (of folios):
|
||||
|
||||
1. It has the young and the old (generations), i.e., the counterparts
|
||||
to the active and the inactive;
|
||||
2. The increment of ``max_seq`` triggers promotion, i.e., the
|
||||
counterpart to activation;
|
||||
3. Other events trigger similar operations, e.g., offlining an memcg
|
||||
triggers demotion, i.e., the counterpart to deactivation.
|
||||
|
||||
In terms of global reclaim, it has two distinct features:
|
||||
|
||||
1. Sharding, which allows each thread to start at a random memcg (in
|
||||
the old generation) and improves parallelism;
|
||||
2. Eventual fairness, which allows direct reclaim to bail out at will
|
||||
and reduces latency without affecting fairness over some time.
|
||||
|
||||
In terms of traversing memcgs during global reclaim, it improves the
|
||||
best-case complexity from O(n) to O(1) and does not affect the
|
||||
worst-case complexity O(n). Therefore, on average, it has a sublinear
|
||||
complexity.
|
||||
|
||||
Summary
|
||||
-------
|
||||
The multi-gen LRU can be disassembled into the following parts:
|
||||
The multi-gen LRU (of folios) can be disassembled into the following
|
||||
parts:
|
||||
|
||||
* Generations
|
||||
* Rmap walks
|
||||
|
||||
@@ -59,7 +59,7 @@ Usage
|
||||
|
||||
1) Build user-space helper::
|
||||
|
||||
cd tools/vm
|
||||
cd tools/mm
|
||||
make page_owner_sort
|
||||
|
||||
2) Enable page owner: add "page_owner=on" to boot cmdline.
|
||||
|
||||
@@ -19,7 +19,7 @@ slabs that have data in them. See "slabinfo -h" for more options when
|
||||
running the command. ``slabinfo`` can be compiled with
|
||||
::
|
||||
|
||||
gcc -o slabinfo tools/vm/slabinfo.c
|
||||
gcc -o slabinfo tools/mm/slabinfo.c
|
||||
|
||||
Some of the modes of operation of ``slabinfo`` require that slub debugging
|
||||
be enabled on the command line. F.e. no tracking information will be
|
||||
|
||||
@@ -110,20 +110,20 @@ Refcounts and transparent huge pages
|
||||
Refcounting on THP is mostly consistent with refcounting on other compound
|
||||
pages:
|
||||
|
||||
- get_page()/put_page() and GUP operate on head page's ->_refcount.
|
||||
- get_page()/put_page() and GUP operate on the folio->_refcount.
|
||||
|
||||
- ->_refcount in tail pages is always zero: get_page_unless_zero() never
|
||||
succeeds on tail pages.
|
||||
|
||||
- map/unmap of PMD entry for the whole compound page increment/decrement
|
||||
->compound_mapcount, stored in the first tail page of the compound page;
|
||||
and also increment/decrement ->subpages_mapcount (also in the first tail)
|
||||
by COMPOUND_MAPPED when compound_mapcount goes from -1 to 0 or 0 to -1.
|
||||
- map/unmap of a PMD entry for the whole THP increment/decrement
|
||||
folio->_entire_mapcount and also increment/decrement
|
||||
folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount
|
||||
goes from -1 to 0 or 0 to -1.
|
||||
|
||||
- map/unmap of sub-pages with PTE entry increment/decrement ->_mapcount
|
||||
on relevant sub-page of the compound page, and also increment/decrement
|
||||
->subpages_mapcount, stored in first tail page of the compound page, when
|
||||
_mapcount goes from -1 to 0 or 0 to -1: counting sub-pages mapped by PTE.
|
||||
- map/unmap of individual pages with PTE entry increment/decrement
|
||||
page->_mapcount and also increment/decrement folio->_nr_pages_mapped
|
||||
when page->_mapcount goes from -1 to 0 or 0 to -1 as this counts
|
||||
the number of pages mapped by PTE.
|
||||
|
||||
split_huge_page internally has to distribute the refcounts in the head
|
||||
page to the tail pages before clearing all PG_head/tail bits from the page
|
||||
@@ -151,8 +151,8 @@ clear where references should go after split: it will stay on the head page.
|
||||
Note that split_huge_pmd() doesn't have any limitations on refcounting:
|
||||
pmd can be split at any point and never fails.
|
||||
|
||||
Partial unmap and deferred_split_huge_page()
|
||||
============================================
|
||||
Partial unmap and deferred_split_folio()
|
||||
========================================
|
||||
|
||||
Unmapping part of THP (with munmap() or other way) is not going to free
|
||||
memory immediately. Instead, we detect that a subpage of THP is not in use
|
||||
@@ -164,6 +164,6 @@ the place where we can detect partial unmap. It also might be
|
||||
counterproductive since in many cases partial unmap happens during exit(2) if
|
||||
a THP crosses a VMA boundary.
|
||||
|
||||
The function deferred_split_huge_page() is used to queue a page for splitting.
|
||||
The function deferred_split_folio() is used to queue a folio for splitting.
|
||||
The splitting itself will happen when we get memory pressure via shrinker
|
||||
interface.
|
||||
|
||||
@@ -10,7 +10,7 @@ Introduction
|
||||
|
||||
This document describes the Linux memory manager's "Unevictable LRU"
|
||||
infrastructure and the use of this to manage several types of "unevictable"
|
||||
pages.
|
||||
folios.
|
||||
|
||||
The document attempts to provide the overall rationale behind this mechanism
|
||||
and the rationale for some of the design decisions that drove the
|
||||
@@ -25,8 +25,8 @@ The Unevictable LRU
|
||||
===================
|
||||
|
||||
The Unevictable LRU facility adds an additional LRU list to track unevictable
|
||||
pages and to hide these pages from vmscan. This mechanism is based on a patch
|
||||
by Larry Woodman of Red Hat to address several scalability problems with page
|
||||
folios and to hide these folios from vmscan. This mechanism is based on a patch
|
||||
by Larry Woodman of Red Hat to address several scalability problems with folio
|
||||
reclaim in Linux. The problems have been observed at customer sites on large
|
||||
memory x86_64 systems.
|
||||
|
||||
@@ -50,40 +50,41 @@ The infrastructure may also be able to handle other conditions that make pages
|
||||
unevictable, either by definition or by circumstance, in the future.
|
||||
|
||||
|
||||
The Unevictable LRU Page List
|
||||
-----------------------------
|
||||
The Unevictable LRU Folio List
|
||||
------------------------------
|
||||
|
||||
The Unevictable LRU page list is a lie. It was never an LRU-ordered list, but a
|
||||
companion to the LRU-ordered anonymous and file, active and inactive page lists;
|
||||
and now it is not even a page list. But following familiar convention, here in
|
||||
this document and in the source, we often imagine it as a fifth LRU page list.
|
||||
The Unevictable LRU folio list is a lie. It was never an LRU-ordered
|
||||
list, but a companion to the LRU-ordered anonymous and file, active and
|
||||
inactive folio lists; and now it is not even a folio list. But following
|
||||
familiar convention, here in this document and in the source, we often
|
||||
imagine it as a fifth LRU folio list.
|
||||
|
||||
The Unevictable LRU infrastructure consists of an additional, per-node, LRU list
|
||||
called the "unevictable" list and an associated page flag, PG_unevictable, to
|
||||
indicate that the page is being managed on the unevictable list.
|
||||
called the "unevictable" list and an associated folio flag, PG_unevictable, to
|
||||
indicate that the folio is being managed on the unevictable list.
|
||||
|
||||
The PG_unevictable flag is analogous to, and mutually exclusive with, the
|
||||
PG_active flag in that it indicates on which LRU list a page resides when
|
||||
PG_active flag in that it indicates on which LRU list a folio resides when
|
||||
PG_lru is set.
|
||||
|
||||
The Unevictable LRU infrastructure maintains unevictable pages as if they were
|
||||
The Unevictable LRU infrastructure maintains unevictable folios as if they were
|
||||
on an additional LRU list for a few reasons:
|
||||
|
||||
(1) We get to "treat unevictable pages just like we treat other pages in the
|
||||
(1) We get to "treat unevictable folios just like we treat other folios in the
|
||||
system - which means we get to use the same code to manipulate them, the
|
||||
same code to isolate them (for migrate, etc.), the same code to keep track
|
||||
of the statistics, etc..." [Rik van Riel]
|
||||
|
||||
(2) We want to be able to migrate unevictable pages between nodes for memory
|
||||
(2) We want to be able to migrate unevictable folios between nodes for memory
|
||||
defragmentation, workload management and memory hotplug. The Linux kernel
|
||||
can only migrate pages that it can successfully isolate from the LRU
|
||||
can only migrate folios that it can successfully isolate from the LRU
|
||||
lists (or "Movable" pages: outside of consideration here). If we were to
|
||||
maintain pages elsewhere than on an LRU-like list, where they can be
|
||||
detected by isolate_lru_page(), we would prevent their migration.
|
||||
maintain folios elsewhere than on an LRU-like list, where they can be
|
||||
detected by folio_isolate_lru(), we would prevent their migration.
|
||||
|
||||
The unevictable list does not differentiate between file-backed and anonymous,
|
||||
swap-backed pages. This differentiation is only important while the pages are,
|
||||
in fact, evictable.
|
||||
The unevictable list does not differentiate between file-backed and
|
||||
anonymous, swap-backed folios. This differentiation is only important
|
||||
while the folios are, in fact, evictable.
|
||||
|
||||
The unevictable list benefits from the "arrayification" of the per-node LRU
|
||||
lists and statistics originally proposed and posted by Christoph Lameter.
|
||||
@@ -156,7 +157,7 @@ These are currently used in three places in the kernel:
|
||||
Detecting Unevictable Pages
|
||||
---------------------------
|
||||
|
||||
The function page_evictable() in mm/internal.h determines whether a page is
|
||||
The function folio_evictable() in mm/internal.h determines whether a folio is
|
||||
evictable or not using the query function outlined above [see section
|
||||
:ref:`Marking address spaces unevictable <mark_addr_space_unevict>`]
|
||||
to check the AS_UNEVICTABLE flag.
|
||||
@@ -165,7 +166,7 @@ For address spaces that are so marked after being populated (as SHM regions
|
||||
might be), the lock action (e.g. SHM_LOCK) can be lazy, and need not populate
|
||||
the page tables for the region as does, for example, mlock(), nor need it make
|
||||
any special effort to push any pages in the SHM_LOCK'd area to the unevictable
|
||||
list. Instead, vmscan will do this if and when it encounters the pages during
|
||||
list. Instead, vmscan will do this if and when it encounters the folios during
|
||||
a reclamation scan.
|
||||
|
||||
On an unlock action (such as SHM_UNLOCK), the unlocker (e.g. shmctl()) must scan
|
||||
@@ -174,41 +175,43 @@ condition is keeping them unevictable. If an unevictable region is destroyed,
|
||||
the pages are also "rescued" from the unevictable list in the process of
|
||||
freeing them.
|
||||
|
||||
page_evictable() also checks for mlocked pages by testing an additional page
|
||||
flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is
|
||||
faulted into a VM_LOCKED VMA, or found in a VMA being VM_LOCKED.
|
||||
folio_evictable() also checks for mlocked folios by calling
|
||||
folio_test_mlocked(), which is set when a folio is faulted into a
|
||||
VM_LOCKED VMA, or found in a VMA being VM_LOCKED.
|
||||
|
||||
|
||||
Vmscan's Handling of Unevictable Pages
|
||||
--------------------------------------
|
||||
Vmscan's Handling of Unevictable Folios
|
||||
---------------------------------------
|
||||
|
||||
If unevictable pages are culled in the fault path, or moved to the unevictable
|
||||
list at mlock() or mmap() time, vmscan will not encounter the pages until they
|
||||
If unevictable folios are culled in the fault path, or moved to the unevictable
|
||||
list at mlock() or mmap() time, vmscan will not encounter the folios until they
|
||||
have become evictable again (via munlock() for example) and have been "rescued"
|
||||
from the unevictable list. However, there may be situations where we decide,
|
||||
for the sake of expediency, to leave an unevictable page on one of the regular
|
||||
for the sake of expediency, to leave an unevictable folio on one of the regular
|
||||
active/inactive LRU lists for vmscan to deal with. vmscan checks for such
|
||||
pages in all of the shrink_{active|inactive|page}_list() functions and will
|
||||
"cull" such pages that it encounters: that is, it diverts those pages to the
|
||||
folios in all of the shrink_{active|inactive|page}_list() functions and will
|
||||
"cull" such folios that it encounters: that is, it diverts those folios to the
|
||||
unevictable list for the memory cgroup and node being scanned.
|
||||
|
||||
There may be situations where a page is mapped into a VM_LOCKED VMA, but the
|
||||
page is not marked as PG_mlocked. Such pages will make it all the way to
|
||||
shrink_active_list() or shrink_page_list() where they will be detected when
|
||||
vmscan walks the reverse map in folio_referenced() or try_to_unmap(). The page
|
||||
is culled to the unevictable list when it is released by the shrinker.
|
||||
There may be situations where a folio is mapped into a VM_LOCKED VMA,
|
||||
but the folio does not have the mlocked flag set. Such folios will make
|
||||
it all the way to shrink_active_list() or shrink_page_list() where they
|
||||
will be detected when vmscan walks the reverse map in folio_referenced()
|
||||
or try_to_unmap(). The folio is culled to the unevictable list when it
|
||||
is released by the shrinker.
|
||||
|
||||
To "cull" an unevictable page, vmscan simply puts the page back on the LRU list
|
||||
using putback_lru_page() - the inverse operation to isolate_lru_page() - after
|
||||
dropping the page lock. Because the condition which makes the page unevictable
|
||||
may change once the page is unlocked, __pagevec_lru_add_fn() will recheck the
|
||||
unevictable state of a page before placing it on the unevictable list.
|
||||
To "cull" an unevictable folio, vmscan simply puts the folio back on
|
||||
the LRU list using folio_putback_lru() - the inverse operation to
|
||||
folio_isolate_lru() - after dropping the folio lock. Because the
|
||||
condition which makes the folio unevictable may change once the folio
|
||||
is unlocked, __pagevec_lru_add_fn() will recheck the unevictable state
|
||||
of a folio before placing it on the unevictable list.
|
||||
|
||||
|
||||
MLOCKED Pages
|
||||
=============
|
||||
|
||||
The unevictable page list is also useful for mlock(), in addition to ramfs and
|
||||
The unevictable folio list is also useful for mlock(), in addition to ramfs and
|
||||
SYSV SHM. Note that mlock() is only available in CONFIG_MMU=y situations; in
|
||||
NOMMU situations, all mappings are effectively mlocked.
|
||||
|
||||
@@ -293,7 +296,7 @@ treated as a no-op and mlock_fixup() simply returns.
|
||||
If the VMA passes some filtering as described in "Filtering Special VMAs"
|
||||
below, mlock_fixup() will attempt to merge the VMA with its neighbors or split
|
||||
off a subset of the VMA if the range does not cover the entire VMA. Any pages
|
||||
already present in the VMA are then marked as mlocked by mlock_page() via
|
||||
already present in the VMA are then marked as mlocked by mlock_folio() via
|
||||
mlock_pte_range() via walk_page_range() via mlock_vma_pages_range().
|
||||
|
||||
Before returning from the system call, do_mlock() or mlockall() will call
|
||||
@@ -306,22 +309,22 @@ do end up getting faulted into this VM_LOCKED VMA, they will be handled in the
|
||||
fault path - which is also how mlock2()'s MLOCK_ONFAULT areas are handled.
|
||||
|
||||
For each PTE (or PMD) being faulted into a VMA, the page add rmap function
|
||||
calls mlock_vma_page(), which calls mlock_page() when the VMA is VM_LOCKED
|
||||
calls mlock_vma_folio(), which calls mlock_folio() when the VMA is VM_LOCKED
|
||||
(unless it is a PTE mapping of a part of a transparent huge page). Or when
|
||||
it is a newly allocated anonymous page, lru_cache_add_inactive_or_unevictable()
|
||||
calls mlock_new_page() instead: similar to mlock_page(), but can make better
|
||||
it is a newly allocated anonymous page, folio_add_lru_vma() calls
|
||||
mlock_new_folio() instead: similar to mlock_folio(), but can make better
|
||||
judgments, since this page is held exclusively and known not to be on LRU yet.
|
||||
|
||||
mlock_page() sets PageMlocked immediately, then places the page on the CPU's
|
||||
mlock pagevec, to batch up the rest of the work to be done under lru_lock by
|
||||
__mlock_page(). __mlock_page() sets PageUnevictable, initializes mlock_count
|
||||
mlock_folio() sets PG_mlocked immediately, then places the page on the CPU's
|
||||
mlock folio batch, to batch up the rest of the work to be done under lru_lock by
|
||||
__mlock_folio(). __mlock_folio() sets PG_unevictable, initializes mlock_count
|
||||
and moves the page to unevictable state ("the unevictable LRU", but with
|
||||
mlock_count in place of LRU threading). Or if the page was already PageLRU
|
||||
and PageUnevictable and PageMlocked, it simply increments the mlock_count.
|
||||
mlock_count in place of LRU threading). Or if the page was already PG_lru
|
||||
and PG_unevictable and PG_mlocked, it simply increments the mlock_count.
|
||||
|
||||
But in practice that may not work ideally: the page may not yet be on an LRU, or
|
||||
it may have been temporarily isolated from LRU. In such cases the mlock_count
|
||||
field cannot be touched, but will be set to 0 later when __pagevec_lru_add_fn()
|
||||
field cannot be touched, but will be set to 0 later when __munlock_folio()
|
||||
returns the page to "LRU". Races prohibit mlock_count from being set to 1 then:
|
||||
rather than risk stranding a page indefinitely as unevictable, always err with
|
||||
mlock_count on the low side, so that when munlocked the page will be rescued to
|
||||
@@ -368,20 +371,21 @@ Because of the VMA filtering discussed above, VM_LOCKED will not be set in
|
||||
any "special" VMAs. So, those VMAs will be ignored for munlock.
|
||||
|
||||
If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the
|
||||
specified range. All pages in the VMA are then munlocked by munlock_page() via
|
||||
specified range. All pages in the VMA are then munlocked by munlock_folio() via
|
||||
mlock_pte_range() via walk_page_range() via mlock_vma_pages_range() - the same
|
||||
function used when mlocking a VMA range, with new flags for the VMA indicating
|
||||
that it is munlock() being performed.
|
||||
|
||||
munlock_page() uses the mlock pagevec to batch up work to be done under
|
||||
lru_lock by __munlock_page(). __munlock_page() decrements the page's
|
||||
mlock_count, and when that reaches 0 it clears PageMlocked and clears
|
||||
PageUnevictable, moving the page from unevictable state to inactive LRU.
|
||||
munlock_folio() uses the mlock pagevec to batch up work to be done
|
||||
under lru_lock by __munlock_folio(). __munlock_folio() decrements the
|
||||
folio's mlock_count, and when that reaches 0 it clears the mlocked flag
|
||||
and clears the unevictable flag, moving the folio from unevictable state
|
||||
to the inactive LRU.
|
||||
|
||||
But in practice that may not work ideally: the page may not yet have reached
|
||||
But in practice that may not work ideally: the folio may not yet have reached
|
||||
"the unevictable LRU", or it may have been temporarily isolated from it. In
|
||||
those cases its mlock_count field is unusable and must be assumed to be 0: so
|
||||
that the page will be rescued to an evictable LRU, then perhaps be mlocked
|
||||
that the folio will be rescued to an evictable LRU, then perhaps be mlocked
|
||||
again later if vmscan finds it in a VM_LOCKED VMA.
|
||||
|
||||
|
||||
@@ -408,7 +412,7 @@ However, since mlock_vma_pages_range() starts by setting VM_LOCKED on a VMA,
|
||||
before mlocking any pages already present, if one of those pages were migrated
|
||||
before mlock_pte_range() reached it, it would get counted twice in mlock_count.
|
||||
To prevent that, mlock_vma_pages_range() temporarily marks the VMA as VM_IO,
|
||||
so that mlock_vma_page() will skip it.
|
||||
so that mlock_vma_folio() will skip it.
|
||||
|
||||
To complete page migration, we place the old and new pages back onto the LRU
|
||||
afterwards. The "unneeded" page - old page on success, new page on failure -
|
||||
@@ -481,18 +485,19 @@ Before the unevictable/mlock changes, mlocking did not mark the pages in any
|
||||
way, so unmapping them required no processing.
|
||||
|
||||
For each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls
|
||||
munlock_vma_page(), which calls munlock_page() when the VMA is VM_LOCKED
|
||||
munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED
|
||||
(unless it was a PTE mapping of a part of a transparent huge page).
|
||||
|
||||
munlock_page() uses the mlock pagevec to batch up work to be done under
|
||||
lru_lock by __munlock_page(). __munlock_page() decrements the page's
|
||||
mlock_count, and when that reaches 0 it clears PageMlocked and clears
|
||||
PageUnevictable, moving the page from unevictable state to inactive LRU.
|
||||
munlock_folio() uses the mlock pagevec to batch up work to be done
|
||||
under lru_lock by __munlock_folio(). __munlock_folio() decrements the
|
||||
folio's mlock_count, and when that reaches 0 it clears the mlocked flag
|
||||
and clears the unevictable flag, moving the folio from unevictable state
|
||||
to the inactive LRU.
|
||||
|
||||
But in practice that may not work ideally: the page may not yet have reached
|
||||
But in practice that may not work ideally: the folio may not yet have reached
|
||||
"the unevictable LRU", or it may have been temporarily isolated from it. In
|
||||
those cases its mlock_count field is unusable and must be assumed to be 0: so
|
||||
that the page will be rescued to an evictable LRU, then perhaps be mlocked
|
||||
that the folio will be rescued to an evictable LRU, then perhaps be mlocked
|
||||
again later if vmscan finds it in a VM_LOCKED VMA.
|
||||
|
||||
|
||||
@@ -505,7 +510,7 @@ which had been Copied-On-Write from the file pages now being truncated.
|
||||
|
||||
Mlocked pages can be munlocked and deleted in this way: like with munmap(),
|
||||
for each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls
|
||||
munlock_vma_page(), which calls munlock_page() when the VMA is VM_LOCKED
|
||||
munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED
|
||||
(unless it was a PTE mapping of a part of a transparent huge page).
|
||||
|
||||
However, if there is a racing munlock(), since mlock_vma_pages_range() starts
|
||||
@@ -513,7 +518,7 @@ munlocking by clearing VM_LOCKED from a VMA, before munlocking all the pages
|
||||
present, if one of those pages were unmapped by truncation or hole punch before
|
||||
mlock_pte_range() reached it, it would not be recognized as mlocked by this VMA,
|
||||
and would not be counted out of mlock_count. In this rare case, a page may
|
||||
still appear as PageMlocked after it has been fully unmapped: and it is left to
|
||||
still appear as PG_mlocked after it has been fully unmapped: and it is left to
|
||||
release_pages() (or __page_cache_release()) to clear it and update statistics
|
||||
before freeing (this event is counted in /proc/vmstat unevictable_pgs_cleared,
|
||||
which is usually 0).
|
||||
@@ -525,7 +530,7 @@ Page Reclaim in shrink_*_list()
|
||||
vmscan's shrink_active_list() culls any obviously unevictable pages -
|
||||
i.e. !page_evictable(page) pages - diverting those to the unevictable list.
|
||||
However, shrink_active_list() only sees unevictable pages that made it onto the
|
||||
active/inactive LRU lists. Note that these pages do not have PageUnevictable
|
||||
active/inactive LRU lists. Note that these pages do not have PG_unevictable
|
||||
set - otherwise they would be on the unevictable list and shrink_active_list()
|
||||
would never see them.
|
||||
|
||||
@@ -547,6 +552,6 @@ and node unevictable list.
|
||||
|
||||
rmap's folio_referenced_one(), called via vmscan's shrink_active_list() or
|
||||
shrink_page_list(), and rmap's try_to_unmap_one() called via shrink_page_list(),
|
||||
check for (3) pages still mapped into VM_LOCKED VMAs, and call mlock_vma_page()
|
||||
check for (3) pages still mapped into VM_LOCKED VMAs, and call mlock_vma_folio()
|
||||
to correct them. Such pages are culled to the unevictable list when released
|
||||
by the shrinker.
|
||||
|
||||
@@ -78,3 +78,171 @@ Similarly, we assign zspage to:
|
||||
* ZS_ALMOST_FULL when n > N / f
|
||||
* ZS_EMPTY when n == 0
|
||||
* ZS_FULL when n == N
|
||||
|
||||
|
||||
Internals
|
||||
=========
|
||||
|
||||
zsmalloc has 255 size classes, each of which can hold a number of zspages.
|
||||
Each zspage can contain up to ZSMALLOC_CHAIN_SIZE physical (0-order) pages.
|
||||
The optimal zspage chain size for each size class is calculated during the
|
||||
creation of the zsmalloc pool (see calculate_zspage_chain_size()).
|
||||
|
||||
As an optimization, zsmalloc merges size classes that have similar
|
||||
characteristics in terms of the number of pages per zspage and the number
|
||||
of objects that each zspage can store.
|
||||
|
||||
For instance, consider the following size classes:::
|
||||
|
||||
class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable
|
||||
...
|
||||
94 1536 0 0 0 0 0 3 0
|
||||
100 1632 0 0 0 0 0 2 0
|
||||
...
|
||||
|
||||
|
||||
Size classes #95-99 are merged with size class #100. This means that when we
|
||||
need to store an object of size, say, 1568 bytes, we end up using size class
|
||||
#100 instead of size class #96. Size class #100 is meant for objects of size
|
||||
1632 bytes, so each object of size 1568 bytes wastes 1632-1568=64 bytes.
|
||||
|
||||
Size class #100 consists of zspages with 2 physical pages each, which can
|
||||
hold a total of 5 objects. If we need to store 13 objects of size 1568, we
|
||||
end up allocating three zspages, or 6 physical pages.
|
||||
|
||||
However, if we take a closer look at size class #96 (which is meant for
|
||||
objects of size 1568 bytes) and trace `calculate_zspage_chain_size()`, we
|
||||
find that the most optimal zspage configuration for this class is a chain
|
||||
of 5 physical pages:::
|
||||
|
||||
pages per zspage wasted bytes used%
|
||||
1 960 76
|
||||
2 352 95
|
||||
3 1312 89
|
||||
4 704 95
|
||||
5 96 99
|
||||
|
||||
This means that a class #96 configuration with 5 physical pages can store 13
|
||||
objects of size 1568 in a single zspage, using a total of 5 physical pages.
|
||||
This is more efficient than the class #100 configuration, which would use 6
|
||||
physical pages to store the same number of objects.
|
||||
|
||||
As the zspage chain size for class #96 increases, its key characteristics
|
||||
such as pages per-zspage and objects per-zspage also change. This leads to
|
||||
dewer class mergers, resulting in a more compact grouping of classes, which
|
||||
reduces memory wastage.
|
||||
|
||||
Let's take a closer look at the bottom of `/sys/kernel/debug/zsmalloc/zramX/classes`:::
|
||||
|
||||
class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable
|
||||
...
|
||||
202 3264 0 0 0 0 0 4 0
|
||||
254 4096 0 0 0 0 0 1 0
|
||||
...
|
||||
|
||||
Size class #202 stores objects of size 3264 bytes and has a maximum of 4 pages
|
||||
per zspage. Any object larger than 3264 bytes is considered huge and belongs
|
||||
to size class #254, which stores each object in its own physical page (objects
|
||||
in huge classes do not share pages).
|
||||
|
||||
Increasing the size of the chain of zspages also results in a higher watermark
|
||||
for the huge size class and fewer huge classes overall. This allows for more
|
||||
efficient storage of large objects.
|
||||
|
||||
For zspage chain size of 8, huge class watermark becomes 3632 bytes:::
|
||||
|
||||
class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable
|
||||
...
|
||||
202 3264 0 0 0 0 0 4 0
|
||||
211 3408 0 0 0 0 0 5 0
|
||||
217 3504 0 0 0 0 0 6 0
|
||||
222 3584 0 0 0 0 0 7 0
|
||||
225 3632 0 0 0 0 0 8 0
|
||||
254 4096 0 0 0 0 0 1 0
|
||||
...
|
||||
|
||||
For zspage chain size of 16, huge class watermark becomes 3840 bytes:::
|
||||
|
||||
class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable
|
||||
...
|
||||
202 3264 0 0 0 0 0 4 0
|
||||
206 3328 0 0 0 0 0 13 0
|
||||
207 3344 0 0 0 0 0 9 0
|
||||
208 3360 0 0 0 0 0 14 0
|
||||
211 3408 0 0 0 0 0 5 0
|
||||
212 3424 0 0 0 0 0 16 0
|
||||
214 3456 0 0 0 0 0 11 0
|
||||
217 3504 0 0 0 0 0 6 0
|
||||
219 3536 0 0 0 0 0 13 0
|
||||
222 3584 0 0 0 0 0 7 0
|
||||
223 3600 0 0 0 0 0 15 0
|
||||
225 3632 0 0 0 0 0 8 0
|
||||
228 3680 0 0 0 0 0 9 0
|
||||
230 3712 0 0 0 0 0 10 0
|
||||
232 3744 0 0 0 0 0 11 0
|
||||
234 3776 0 0 0 0 0 12 0
|
||||
235 3792 0 0 0 0 0 13 0
|
||||
236 3808 0 0 0 0 0 14 0
|
||||
238 3840 0 0 0 0 0 15 0
|
||||
254 4096 0 0 0 0 0 1 0
|
||||
...
|
||||
|
||||
Overall the combined zspage chain size effect on zsmalloc pool configuration:::
|
||||
|
||||
pages per zspage number of size classes (clusters) huge size class watermark
|
||||
4 69 3264
|
||||
5 86 3408
|
||||
6 93 3504
|
||||
7 112 3584
|
||||
8 123 3632
|
||||
9 140 3680
|
||||
10 143 3712
|
||||
11 159 3744
|
||||
12 164 3776
|
||||
13 180 3792
|
||||
14 183 3808
|
||||
15 188 3840
|
||||
16 191 3840
|
||||
|
||||
|
||||
A synthetic test
|
||||
----------------
|
||||
|
||||
zram as a build artifacts storage (Linux kernel compilation).
|
||||
|
||||
* `CONFIG_ZSMALLOC_CHAIN_SIZE=4`
|
||||
|
||||
zsmalloc classes stats:::
|
||||
|
||||
class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable
|
||||
...
|
||||
Total 13 51 413836 412973 159955 3
|
||||
|
||||
zram mm_stat:::
|
||||
|
||||
1691783168 628083717 655175680 0 655175680 60 0 34048 34049
|
||||
|
||||
|
||||
* `CONFIG_ZSMALLOC_CHAIN_SIZE=8`
|
||||
|
||||
zsmalloc classes stats:::
|
||||
|
||||
class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage freeable
|
||||
...
|
||||
Total 18 87 414852 412978 156666 0
|
||||
|
||||
zram mm_stat:::
|
||||
|
||||
1691803648 627793930 641703936 0 641703936 60 0 33591 33591
|
||||
|
||||
Using larger zspage chains may result in using fewer physical pages, as seen
|
||||
in the example where the number of physical pages used decreased from 159955
|
||||
to 156666, at the same time maximum zsmalloc pool memory usage went down from
|
||||
655175680 to 641703936 bytes.
|
||||
|
||||
However, this advantage may be offset by the potential for increased system
|
||||
memory pressure (as some zspages have larger chain sizes) in cases where there
|
||||
is heavy internal fragmentation and zspool compaction is unable to relocate
|
||||
objects and release zspages. In these cases, it is recommended to decrease
|
||||
the limit on the size of the zspage chains (as specified by the
|
||||
CONFIG_ZSMALLOC_CHAIN_SIZE option).
|
||||
|
||||
@@ -142,14 +142,14 @@ HPAGE_RESV_OWNER标志被设置，以表明该VMA拥有预留。
|
||||
消耗预留/分配一个巨页
|
||||
===========================
|
||||
|
||||
当与预留相关的巨页在相应的映射中被分配和实例化时，预留就被消耗了。该分配是在函数alloc_huge_page()
|
||||
当与预留相关的巨页在相应的映射中被分配和实例化时，预留就被消耗了。该分配是在函数alloc_hugetlb_folio()
|
||||
中进行的::
|
||||
|
||||
struct page *alloc_huge_page(struct vm_area_struct *vma,
|
||||
struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
|
||||
unsigned long addr, int avoid_reserve)
|
||||
|
||||
alloc_huge_page被传递给一个VMA指针和一个虚拟地址，因此它可以查阅预留映射以确定是否存在预留。
|
||||
此外，alloc_huge_page需要一个参数avoid_reserve，该参数表示即使看起来已经为指定的地址预留了
|
||||
alloc_hugetlb_folio被传递给一个VMA指针和一个虚拟地址，因此它可以查阅预留映射以确定是否存在预留。
|
||||
此外，alloc_hugetlb_folio需要一个参数avoid_reserve，该参数表示即使看起来已经为指定的地址预留了
|
||||
预留，也不应该使用预留。avoid_reserve参数最常被用于写时拷贝和页面迁移的情况下，即现有页面的额
|
||||
外拷贝被分配。
|
||||
|
||||
@@ -162,7 +162,7 @@ vma_needs_reservation()返回的值通常为0或1。如果该地址存在预留
|
||||
确定预留是否存在并可用于分配后，调用dequeue_huge_page_vma()函数。这个函数需要两个与预留有关
|
||||
的参数：
|
||||
|
||||
- avoid_reserve，这是传递给alloc_huge_page()的同一个值/参数。
|
||||
- avoid_reserve，这是传递给alloc_hugetlb_folio()的同一个值/参数。
|
||||
- chg，尽管这个参数的类型是long，但只有0或1的值被传递给dequeue_huge_page_vma。如果该值为0，
|
||||
则表明存在预留（关于可能的问题，请参见 “预留和内存策略” 一节）。如果值
|
||||
为1，则表示不存在预留，如果可能的话，必须从全局空闲池中取出该页。
|
||||
@@ -179,7 +179,7 @@ free_huge_pages的值被递减。如果有一个与该页相关的预留，将
|
||||
的剩余巨页和超额分配的问题。即使分配了一个多余的页面，也会进行与上面一样的基于预留的调整:
|
||||
SetPagePrivate(page) 和 resv_huge_pages--.
|
||||
|
||||
在获得一个新的巨页后，(page)->private被设置为与该页面相关的子池的值，如果它存在的话。当页
|
||||
在获得一个新的巨页后，(folio)->_hugetlb_subpool被设置为与该页面相关的子池的值，如果它存在的话。当页
|
||||
面被释放时，这将被用于子池的计数。
|
||||
|
||||
然后调用函数vma_commit_reservation()，根据预留的消耗情况调整预留映射。一般来说，这涉及
|
||||
@@ -199,7 +199,7 @@ SetPagePrivate(page)和resv_huge_pages-。
|
||||
已经存在，所以不做任何改变。然而，如果共享映射中没有预留，或者这是一个私有映射，则必须创建
|
||||
一个新的条目。
|
||||
|
||||
在alloc_huge_page()开始调用vma_needs_reservation()和页面分配后调用
|
||||
在alloc_hugetlb_folio()开始调用vma_needs_reservation()和页面分配后调用
|
||||
vma_commit_reservation()之间，预留映射有可能被改变。如果hugetlb_reserve_pages在共
|
||||
享映射中为同一页面被调用，这将是可能的。在这种情况下，预留计数和子池空闲页计数会有一个偏差。
|
||||
这种罕见的情况可以通过比较vma_needs_reservation和vma_commit_reservation的返回值来
|
||||
|
||||
@@ -51,7 +51,7 @@ page owner在默认情况下是禁用的。所以，如果你想使用它，你
|
||||
|
||||
1) 构建用户空间的帮助::
|
||||
|
||||
cd tools/vm
|
||||
cd tools/mm
|
||||
make page_owner_sort
|
||||
|
||||
2) 启用page owner: 添加 "page_owner=on" 到 boot cmdline.
|
||||
|
||||
+10
-4
@@ -5657,6 +5657,11 @@ M: SeongJae Park <sj@kernel.org>
|
||||
L: damon@lists.linux.dev
|
||||
L: linux-mm@kvack.org
|
||||
S: Maintained
|
||||
W: https://damonitor.github.io
|
||||
P: Documentation/mm/damon/maintainer-profile.rst
|
||||
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|
||||
T: quilt git://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new
|
||||
T: git git://git.kernel.org/pub/scm/linux/kernel/git/sj/linux.git damon/next
|
||||
F: Documentation/ABI/testing/sysfs-kernel-mm-damon
|
||||
F: Documentation/admin-guide/mm/damon/
|
||||
F: Documentation/mm/damon/
|
||||
@@ -9340,7 +9345,7 @@ F: Documentation/mm/hmm.rst
|
||||
F: include/linux/hmm*
|
||||
F: lib/test_hmm*
|
||||
F: mm/hmm*
|
||||
F: tools/testing/selftests/vm/*hmm*
|
||||
F: tools/testing/selftests/mm/*hmm*
|
||||
|
||||
HOST AP DRIVER
|
||||
M: Jouni Malinen <j@w1.fi>
|
||||
@@ -13385,7 +13390,7 @@ M: Andrew Morton <akpm@linux-foundation.org>
|
||||
L: linux-mm@kvack.org
|
||||
S: Maintained
|
||||
W: http://www.linux-mm.org
|
||||
T: git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|
||||
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|
||||
T: quilt git://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new
|
||||
F: include/linux/gfp.h
|
||||
F: include/linux/gfp_types.h
|
||||
@@ -13394,7 +13399,8 @@ F: include/linux/mm.h
|
||||
F: include/linux/mmzone.h
|
||||
F: include/linux/pagewalk.h
|
||||
F: mm/
|
||||
F: tools/testing/selftests/vm/
|
||||
F: tools/mm/
|
||||
F: tools/testing/selftests/mm/
|
||||
|
||||
VMALLOC
|
||||
M: Andrew Morton <akpm@linux-foundation.org>
|
||||
@@ -13403,7 +13409,7 @@ R: Christoph Hellwig <hch@infradead.org>
|
||||
L: linux-mm@kvack.org
|
||||
S: Maintained
|
||||
W: http://www.linux-mm.org
|
||||
T: git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|
||||
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|
||||
F: include/linux/vmalloc.h
|
||||
F: mm/vmalloc.c
|
||||
|
||||
|
||||
@@ -17,9 +17,8 @@
|
||||
extern void clear_page(void *page);
|
||||
#define clear_user_page(page, vaddr, pg) clear_page(page)
|
||||
|
||||
#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
|
||||
alloc_page_vma(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vaddr)
|
||||
#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
|
||||
#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
|
||||
vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
|
||||
|
||||
extern void copy_page(void * _to, void * _from);
|
||||
#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
|
||||
@@ -87,10 +86,6 @@ typedef struct page *pgtable_t;
|
||||
#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
|
||||
#define virt_addr_valid(kaddr) pfn_valid((__pa(kaddr) >> PAGE_SHIFT))
|
||||
|
||||
#ifdef CONFIG_FLATMEM
|
||||
#define pfn_valid(pfn) ((pfn) < max_mapnr)
|
||||
#endif /* CONFIG_FLATMEM */
|
||||
|
||||
#include <asm-generic/memory_model.h>
|
||||
#include <asm-generic/getorder.h>
|
||||
|
||||
|
||||
@@ -74,6 +74,9 @@ struct vm_area_struct;
|
||||
#define _PAGE_DIRTY 0x20000
|
||||
#define _PAGE_ACCESSED 0x40000
|
||||
|
||||
/* We borrow bit 39 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE 0x8000000000UL
|
||||
|
||||
/*
|
||||
* NOTE! The "accessed" bit isn't necessarily exact: it can be kept exactly
|
||||
* by software (use the KRE/URE/KWE/UWE bits appropriately), but I'll fake it.
|
||||
@@ -301,18 +304,47 @@ extern inline void update_mmu_cache(struct vm_area_struct * vma,
|
||||
}
|
||||
|
||||
/*
|
||||
* Non-present pages: high 24 bits are offset, next 8 bits type,
|
||||
* low 32 bits zero.
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3
|
||||
* 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2
|
||||
* <------------------- offset ------------------> E <--- type -->
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <--------------------------- zeroes -------------------------->
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*/
|
||||
extern inline pte_t mk_swap_pte(unsigned long type, unsigned long offset)
|
||||
{ pte_t pte; pte_val(pte) = (type << 32) | (offset << 40); return pte; }
|
||||
{ pte_t pte; pte_val(pte) = ((type & 0x7f) << 32) | (offset << 40); return pte; }
|
||||
|
||||
#define __swp_type(x) (((x).val >> 32) & 0xff)
|
||||
#define __swp_type(x) (((x).val >> 32) & 0x7f)
|
||||
#define __swp_offset(x) ((x).val >> 40)
|
||||
#define __swp_entry(type, off) ((swp_entry_t) { pte_val(mk_swap_pte((type), (off))) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %016lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#define pmd_ERROR(e) \
|
||||
|
||||
@@ -109,7 +109,6 @@ extern int pfn_valid(unsigned long pfn);
|
||||
#else /* CONFIG_HIGHMEM */
|
||||
|
||||
#define ARCH_PFN_OFFSET virt_to_pfn(CONFIG_LINUX_RAM_BASE)
|
||||
#define pfn_valid(pfn) (((pfn) - ARCH_PFN_OFFSET) < max_mapnr)
|
||||
|
||||
#endif /* CONFIG_HIGHMEM */
|
||||
|
||||
|
||||
@@ -26,6 +26,9 @@
|
||||
#define _PAGE_GLOBAL (1 << 8) /* ASID agnostic (H) */
|
||||
#define _PAGE_PRESENT (1 << 9) /* PTE/TLB Valid (H) */
|
||||
|
||||
/* We borrow bit 5 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE _PAGE_DIRTY
|
||||
|
||||
#ifdef CONFIG_ARC_MMU_V4
|
||||
#define _PAGE_HW_SZ (1 << 10) /* Normal/super (H) */
|
||||
#else
|
||||
@@ -106,9 +109,18 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,
|
||||
void update_mmu_cache(struct vm_area_struct *vma, unsigned long address,
|
||||
pte_t *ptep);
|
||||
|
||||
/* Encode swap {type,off} tuple into PTE
|
||||
* We reserve 13 bits for 5-bit @type, keeping bits 12-5 zero, ensuring that
|
||||
* PAGE_PRESENT is zero in a PTE holding swap "identifier"
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <-------------- offset -------------> <--- zero --> E < type ->
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* The zero'ed bits include _PAGE_PRESENT.
|
||||
*/
|
||||
#define __swp_entry(type, off) ((swp_entry_t) \
|
||||
{ ((type) & 0x1f) | ((off) << 13) })
|
||||
@@ -120,6 +132,14 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address,
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
PTE_BIT_FUNC(swp_mkexclusive, |= (_PAGE_SWP_EXCLUSIVE));
|
||||
PTE_BIT_FUNC(swp_clear_exclusive, &= ~(_PAGE_SWP_EXCLUSIVE));
|
||||
|
||||
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
|
||||
#include <asm/hugepage.h>
|
||||
#endif
|
||||
|
||||
@@ -386,6 +386,4 @@ static inline unsigned long __virt_to_idmap(unsigned long x)
|
||||
|
||||
#endif
|
||||
|
||||
#include <asm-generic/memory_model.h>
|
||||
|
||||
#endif
|
||||
|
||||
@@ -158,6 +158,7 @@ typedef struct page *pgtable_t;
|
||||
|
||||
#ifdef CONFIG_HAVE_ARCH_PFN_VALID
|
||||
extern int pfn_valid(unsigned long);
|
||||
#define pfn_valid pfn_valid
|
||||
#endif
|
||||
|
||||
#include <asm/memory.h>
|
||||
@@ -167,5 +168,6 @@ extern int pfn_valid(unsigned long);
|
||||
#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_TSK_EXEC
|
||||
|
||||
#include <asm-generic/getorder.h>
|
||||
#include <asm-generic/memory_model.h>
|
||||
|
||||
#endif
|
||||
|
||||
@@ -126,6 +126,9 @@
|
||||
#define L_PTE_SHARED (_AT(pteval_t, 1) << 10) /* shared(v6), coherent(xsc3) */
|
||||
#define L_PTE_NONE (_AT(pteval_t, 1) << 11)
|
||||
|
||||
/* We borrow bit 7 to store the exclusive marker in swap PTEs. */
|
||||
#define L_PTE_SWP_EXCLUSIVE L_PTE_RDONLY
|
||||
|
||||
/*
|
||||
* These are the memory types, defined to be compatible with
|
||||
* pre-ARMv6 CPUs cacheable and bufferable bits: n/a,n/a,C,B
|
||||
|
||||
@@ -76,6 +76,9 @@
|
||||
#define L_PTE_NONE (_AT(pteval_t, 1) << 57) /* PROT_NONE */
|
||||
#define L_PTE_RDONLY (_AT(pteval_t, 1) << 58) /* READ ONLY */
|
||||
|
||||
/* We borrow bit 7 to store the exclusive marker in swap PTEs. */
|
||||
#define L_PTE_SWP_EXCLUSIVE (_AT(pteval_t, 1) << 7)
|
||||
|
||||
#define L_PMD_SECT_VALID (_AT(pmdval_t, 1) << 0)
|
||||
#define L_PMD_SECT_DIRTY (_AT(pmdval_t, 1) << 55)
|
||||
#define L_PMD_SECT_NONE (_AT(pmdval_t, 1) << 57)
|
||||
|
||||
@@ -271,27 +271,47 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
|
||||
}
|
||||
|
||||
/*
|
||||
* Encode and decode a swap entry. Swap entries are stored in the Linux
|
||||
* page tables as follows:
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <--------------- offset ------------------------> < type -> 0 0
|
||||
* <------------------- offset ------------------> E < type -> 0 0
|
||||
*
|
||||
* This gives us up to 31 swap files and 128GB per swap file. Note that
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*
|
||||
* This gives us up to 31 swap files and 64GB per swap file. Note that
|
||||
* the offset field is always non-zero.
|
||||
*/
|
||||
#define __SWP_TYPE_SHIFT 2
|
||||
#define __SWP_TYPE_BITS 5
|
||||
#define __SWP_TYPE_MASK ((1 << __SWP_TYPE_BITS) - 1)
|
||||
#define __SWP_OFFSET_SHIFT (__SWP_TYPE_BITS + __SWP_TYPE_SHIFT)
|
||||
#define __SWP_OFFSET_SHIFT (__SWP_TYPE_BITS + __SWP_TYPE_SHIFT + 1)
|
||||
|
||||
#define __swp_type(x) (((x).val >> __SWP_TYPE_SHIFT) & __SWP_TYPE_MASK)
|
||||
#define __swp_offset(x) ((x).val >> __SWP_OFFSET_SHIFT)
|
||||
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) << __SWP_TYPE_SHIFT) | ((offset) << __SWP_OFFSET_SHIFT) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (((type) & __SWP_TYPE_MASK) << __SWP_TYPE_SHIFT) | \
|
||||
((offset) << __SWP_OFFSET_SHIFT) })
|
||||
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(swp) __pte((swp).val | PTE_TYPE_FAULT)
|
||||
#define __swp_entry_to_pte(swp) __pte((swp).val)
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_isset(pte, L_PTE_SWP_EXCLUSIVE);
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
return set_pte_bit(pte, __pgprot(L_PTE_SWP_EXCLUSIVE));
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
return clear_pte_bit(pte, __pgprot(L_PTE_SWP_EXCLUSIVE));
|
||||
}
|
||||
|
||||
/*
|
||||
* It is an error for the kernel to have more swap files than we can
|
||||
|
||||
@@ -315,7 +315,7 @@ static int __init gate_vma_init(void)
|
||||
gate_vma.vm_page_prot = PAGE_READONLY_EXEC;
|
||||
gate_vma.vm_start = 0xffff0000;
|
||||
gate_vma.vm_end = 0xffff0000 + PAGE_SIZE;
|
||||
gate_vma.vm_flags = VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYEXEC;
|
||||
vm_flags_init(&gate_vma, VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYEXEC);
|
||||
return 0;
|
||||
}
|
||||
arch_initcall(gate_vma_init);
|
||||
|
||||
@@ -29,9 +29,9 @@ void copy_user_highpage(struct page *to, struct page *from,
|
||||
void copy_highpage(struct page *to, struct page *from);
|
||||
#define __HAVE_ARCH_COPY_HIGHPAGE
|
||||
|
||||
struct page *alloc_zeroed_user_highpage_movable(struct vm_area_struct *vma,
|
||||
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
|
||||
unsigned long vaddr);
|
||||
#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
|
||||
#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio
|
||||
|
||||
void tag_clear_highpage(struct page *to);
|
||||
#define __HAVE_ARCH_TAG_CLEAR_HIGHPAGE
|
||||
|
||||
@@ -421,7 +421,6 @@ static inline pgprot_t mk_pmd_sect_prot(pgprot_t prot)
|
||||
return __pgprot((pgprot_val(prot) & ~PMD_TABLE_BIT) | PMD_TYPE_SECT);
|
||||
}
|
||||
|
||||
#define __HAVE_ARCH_PTE_SWP_EXCLUSIVE
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
return set_pte_bit(pte, __pgprot(PTE_SWP_EXCLUSIVE));
|
||||
|
||||
@@ -138,13 +138,11 @@ int vdso_join_timens(struct task_struct *task, struct time_namespace *ns)
|
||||
mmap_read_lock(mm);
|
||||
|
||||
for_each_vma(vmi, vma) {
|
||||
unsigned long size = vma->vm_end - vma->vm_start;
|
||||
|
||||
if (vma_is_special_mapping(vma, vdso_info[VDSO_ABI_AA64].dm))
|
||||
zap_page_range(vma, vma->vm_start, size);
|
||||
zap_vma_pages(vma);
|
||||
#ifdef CONFIG_COMPAT_VDSO
|
||||
if (vma_is_special_mapping(vma, vdso_info[VDSO_ABI_AA32].dm))
|
||||
zap_page_range(vma, vma->vm_start, size);
|
||||
zap_vma_pages(vma);
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
@@ -925,7 +925,7 @@ NOKPROBE_SYMBOL(do_debug_exception);
|
||||
/*
|
||||
* Used during anonymous page fault handling.
|
||||
*/
|
||||
struct page *alloc_zeroed_user_highpage_movable(struct vm_area_struct *vma,
|
||||
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
|
||||
unsigned long vaddr)
|
||||
{
|
||||
gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO;
|
||||
@@ -938,7 +938,7 @@ struct page *alloc_zeroed_user_highpage_movable(struct vm_area_struct *vma,
|
||||
if (vma->vm_flags & VM_MTE)
|
||||
flags |= __GFP_ZEROTAGS;
|
||||
|
||||
return alloc_page_vma(flags, vma, vaddr);
|
||||
return vma_alloc_folio(flags, 0, vma, vaddr, false);
|
||||
}
|
||||
|
||||
void tag_clear_highpage(struct page *page)
|
||||
|
||||
@@ -10,6 +10,9 @@
|
||||
#define _PAGE_ACCESSED (1<<3)
|
||||
#define _PAGE_MODIFIED (1<<4)
|
||||
|
||||
/* We borrow bit 9 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (1<<9)
|
||||
|
||||
/* implemented in hardware */
|
||||
#define _PAGE_GLOBAL (1<<6)
|
||||
#define _PAGE_VALID (1<<7)
|
||||
@@ -26,7 +29,8 @@
|
||||
#define _PAGE_PROT_NONE _PAGE_READ
|
||||
|
||||
/*
|
||||
* Encode and decode a swap entry
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTE:
|
||||
* bit 0: _PAGE_PRESENT (zero)
|
||||
@@ -35,15 +39,16 @@
|
||||
* bit 6: _PAGE_GLOBAL (zero)
|
||||
* bit 7: _PAGE_VALID (zero)
|
||||
* bit 8: swap type[4]
|
||||
* bit 9 - 31: swap offset
|
||||
* bit 9: exclusive marker
|
||||
* bit 10 - 31: swap offset
|
||||
*/
|
||||
#define __swp_type(x) ((((x).val >> 2) & 0xf) | \
|
||||
(((x).val >> 4) & 0x10))
|
||||
#define __swp_offset(x) ((x).val >> 9)
|
||||
#define __swp_offset(x) ((x).val >> 10)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { \
|
||||
((type & 0xf) << 2) | \
|
||||
((type & 0x10) << 4) | \
|
||||
((offset) << 9)})
|
||||
((offset) << 10)})
|
||||
|
||||
#define HAVE_ARCH_UNMAPPED_AREA
|
||||
|
||||
|
||||
@@ -10,6 +10,9 @@
|
||||
#define _PAGE_PRESENT (1<<10)
|
||||
#define _PAGE_MODIFIED (1<<11)
|
||||
|
||||
/* We borrow bit 7 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (1<<7)
|
||||
|
||||
/* implemented in hardware */
|
||||
#define _PAGE_GLOBAL (1<<0)
|
||||
#define _PAGE_VALID (1<<1)
|
||||
@@ -26,23 +29,25 @@
|
||||
#define _PAGE_PROT_NONE _PAGE_WRITE
|
||||
|
||||
/*
|
||||
* Encode and decode a swap entry
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTE:
|
||||
* bit 0: _PAGE_GLOBAL (zero)
|
||||
* bit 1: _PAGE_VALID (zero)
|
||||
* bit 2 - 6: swap type
|
||||
* bit 7 - 8: swap offset[0 - 1]
|
||||
* bit 7: exclusive marker
|
||||
* bit 8: swap offset[0]
|
||||
* bit 9: _PAGE_WRITE (zero)
|
||||
* bit 10: _PAGE_PRESENT (zero)
|
||||
* bit 11 - 31: swap offset[2 - 22]
|
||||
* bit 11 - 31: swap offset[1 - 21]
|
||||
*/
|
||||
#define __swp_type(x) (((x).val >> 2) & 0x1f)
|
||||
#define __swp_offset(x) ((((x).val >> 7) & 0x3) | \
|
||||
(((x).val >> 9) & 0x7ffffc))
|
||||
#define __swp_offset(x) ((((x).val >> 8) & 0x1) | \
|
||||
(((x).val >> 10) & 0x3ffffe))
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { \
|
||||
((type & 0x1f) << 2) | \
|
||||
((offset & 0x3) << 7) | \
|
||||
((offset & 0x7ffffc) << 9)})
|
||||
((offset & 0x1) << 8) | \
|
||||
((offset & 0x3ffffe) << 10)})
|
||||
|
||||
#endif /* __ASM_CSKY_PGTABLE_BITS_H */
|
||||
|
||||
@@ -39,7 +39,6 @@
|
||||
|
||||
#define virt_addr_valid(kaddr) ((void *)(kaddr) >= (void *)PAGE_OFFSET && \
|
||||
(void *)(kaddr) < high_memory)
|
||||
#define pfn_valid(pfn) ((pfn) >= ARCH_PFN_OFFSET && ((pfn) - ARCH_PFN_OFFSET) < max_mapnr)
|
||||
|
||||
extern void *memset(void *dest, int c, size_t l);
|
||||
extern void *memcpy(void *to, const void *from, size_t l);
|
||||
|
||||
@@ -200,6 +200,23 @@ static inline pte_t pte_mkyoung(pte_t pte)
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
#define __HAVE_PHYS_MEM_ACCESS_PROT
|
||||
struct file;
|
||||
extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
|
||||
|
||||
@@ -95,7 +95,6 @@ struct page;
|
||||
/* Default vm area behavior is non-executable. */
|
||||
#define VM_DATA_DEFAULT_FLAGS VM_DATA_FLAGS_NON_EXEC
|
||||
|
||||
#define pfn_valid(pfn) ((pfn) < max_mapnr)
|
||||
#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
|
||||
|
||||
/* Need to not use a define for linesize; may move this to another file. */
|
||||
|
||||
@@ -61,6 +61,9 @@ extern unsigned long empty_zero_page;
|
||||
* So we'll put up with a bit of inefficiency for now...
|
||||
*/
|
||||
|
||||
/* We borrow bit 6 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (1<<6)
|
||||
|
||||
/*
|
||||
* Top "FOURTH" level (pgd), which for the Hexagon VM is really
|
||||
* only the second from the bottom, pgd and pud both being collapsed.
|
||||
@@ -359,9 +362,12 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
#define ZERO_PAGE(vaddr) (virt_to_page(&empty_zero_page))
|
||||
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Swap/file PTE definitions. If _PAGE_PRESENT is zero, the rest of the PTE is
|
||||
* interpreted as swap information. The remaining free bits are interpreted as
|
||||
* swap type/offset tuple. Rather than have the TLB fill handler test
|
||||
* listed below. Rather than have the TLB fill handler test
|
||||
* _PAGE_PRESENT, we're going to reserve the permissions bits and set them to
|
||||
* all zeros for swap entries, which speeds up the miss handler at the cost of
|
||||
* 3 bits of offset. That trade-off can be revisited if necessary, but Hexagon
|
||||
@@ -371,9 +377,10 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
* Format of swap PTE:
|
||||
* bit 0: Present (zero)
|
||||
* bits 1-5: swap type (arch independent layer uses 5 bits max)
|
||||
* bits 6-9: bits 3:0 of offset
|
||||
* bit 6: exclusive marker
|
||||
* bits 7-9: bits 2:0 of offset
|
||||
* bits 10-12: effectively _PAGE_PROTNONE (all zero)
|
||||
* bits 13-31: bits 22:4 of swap offset
|
||||
* bits 13-31: bits 21:3 of swap offset
|
||||
*
|
||||
* The split offset makes some of the following macros a little gnarly,
|
||||
* but there's plenty of precedent for this sort of thing.
|
||||
@@ -383,11 +390,28 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
#define __swp_type(swp_pte) (((swp_pte).val >> 1) & 0x1f)
|
||||
|
||||
#define __swp_offset(swp_pte) \
|
||||
((((swp_pte).val >> 6) & 0xf) | (((swp_pte).val >> 9) & 0x7ffff0))
|
||||
((((swp_pte).val >> 7) & 0x7) | (((swp_pte).val >> 10) & 0x3ffff8))
|
||||
|
||||
#define __swp_entry(type, offset) \
|
||||
((swp_entry_t) { \
|
||||
((type << 1) | \
|
||||
((offset & 0x7ffff0) << 9) | ((offset & 0xf) << 6)) })
|
||||
(((type & 0x1f) << 1) | \
|
||||
((offset & 0x3ffff8) << 10) | ((offset & 0x7) << 7)) })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
@@ -82,25 +82,19 @@ do { \
|
||||
} while (0)
|
||||
|
||||
|
||||
#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
|
||||
#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
|
||||
({ \
|
||||
struct page *page = alloc_page_vma( \
|
||||
GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vaddr); \
|
||||
if (page) \
|
||||
flush_dcache_page(page); \
|
||||
page; \
|
||||
struct folio *folio = vma_alloc_folio( \
|
||||
GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false); \
|
||||
if (folio) \
|
||||
flush_dcache_folio(folio); \
|
||||
folio; \
|
||||
})
|
||||
|
||||
#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
|
||||
|
||||
#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
|
||||
|
||||
#include <asm-generic/memory_model.h>
|
||||
|
||||
#ifdef CONFIG_FLATMEM
|
||||
# define pfn_valid(pfn) ((pfn) < max_mapnr)
|
||||
#endif
|
||||
|
||||
#define page_to_phys(page) (page_to_pfn(page) << PAGE_SHIFT)
|
||||
#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
|
||||
#define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
|
||||
|
||||
@@ -58,6 +58,9 @@
|
||||
#define _PAGE_ED (__IA64_UL(1) << 52) /* exception deferral */
|
||||
#define _PAGE_PROTNONE (__IA64_UL(1) << 63)
|
||||
|
||||
/* We borrow bit 7 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (1 << 7)
|
||||
|
||||
#define _PFN_MASK _PAGE_PPN_MASK
|
||||
/* Mask of bits which may be changed by pte_modify(); the odd bits are there for _PAGE_PROTNONE */
|
||||
#define _PAGE_CHG_MASK (_PAGE_P | _PAGE_PROTNONE | _PAGE_PL_MASK | _PAGE_AR_MASK | _PAGE_ED)
|
||||
@@ -399,6 +402,9 @@ extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
|
||||
extern void paging_init (void);
|
||||
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Note: The macros below rely on the fact that MAX_SWAPFILES_SHIFT <= number of
|
||||
* bits in the swap-type field of the swap pte. It would be nice to
|
||||
* enforce that, but we can't easily include <linux/swap.h> here.
|
||||
@@ -406,16 +412,35 @@ extern void paging_init (void);
|
||||
*
|
||||
* Format of swap pte:
|
||||
* bit 0 : present bit (must be zero)
|
||||
* bits 1- 7: swap-type
|
||||
* bits 1- 6: swap type
|
||||
* bit 7 : exclusive marker
|
||||
* bits 8-62: swap offset
|
||||
* bit 63 : _PAGE_PROTNONE bit
|
||||
*/
|
||||
#define __swp_type(entry) (((entry).val >> 1) & 0x7f)
|
||||
#define __swp_type(entry) (((entry).val >> 1) & 0x3f)
|
||||
#define __swp_offset(entry) (((entry).val << 1) >> 9)
|
||||
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) << 1) | ((long) (offset) << 8) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { ((type & 0x3f) << 1) | \
|
||||
((long) (offset) << 8) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
/*
|
||||
* ZERO_PAGE is a global shared page that is always zero: used
|
||||
* for zero-mapped memory areas etc..
|
||||
|
||||
+4
-4
@@ -109,7 +109,7 @@ ia64_init_addr_space (void)
|
||||
vma_set_anonymous(vma);
|
||||
vma->vm_start = current->thread.rbs_bot & PAGE_MASK;
|
||||
vma->vm_end = vma->vm_start + PAGE_SIZE;
|
||||
vma->vm_flags = VM_DATA_DEFAULT_FLAGS|VM_GROWSUP|VM_ACCOUNT;
|
||||
vm_flags_init(vma, VM_DATA_DEFAULT_FLAGS|VM_GROWSUP|VM_ACCOUNT);
|
||||
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
|
||||
mmap_write_lock(current->mm);
|
||||
if (insert_vm_struct(current->mm, vma)) {
|
||||
@@ -127,8 +127,8 @@ ia64_init_addr_space (void)
|
||||
vma_set_anonymous(vma);
|
||||
vma->vm_end = PAGE_SIZE;
|
||||
vma->vm_page_prot = __pgprot(pgprot_val(PAGE_READONLY) | _PAGE_MA_NAT);
|
||||
vma->vm_flags = VM_READ | VM_MAYREAD | VM_IO |
|
||||
VM_DONTEXPAND | VM_DONTDUMP;
|
||||
vm_flags_init(vma, VM_READ | VM_MAYREAD | VM_IO |
|
||||
VM_DONTEXPAND | VM_DONTDUMP);
|
||||
mmap_write_lock(current->mm);
|
||||
if (insert_vm_struct(current->mm, vma)) {
|
||||
mmap_write_unlock(current->mm);
|
||||
@@ -272,7 +272,7 @@ static int __init gate_vma_init(void)
|
||||
vma_init(&gate_vma, NULL);
|
||||
gate_vma.vm_start = FIXADDR_USER_START;
|
||||
gate_vma.vm_end = FIXADDR_USER_END;
|
||||
gate_vma.vm_flags = VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC;
|
||||
vm_flags_init(&gate_vma, VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC);
|
||||
gate_vma.vm_page_prot = __pgprot(__ACCESS_BITS | _PAGE_PL_3 | _PAGE_AR_RX);
|
||||
|
||||
return 0;
|
||||
|
||||
@@ -82,19 +82,6 @@ typedef struct { unsigned long pgprot; } pgprot_t;
|
||||
|
||||
#define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
|
||||
|
||||
#ifdef CONFIG_FLATMEM
|
||||
|
||||
static inline int pfn_valid(unsigned long pfn)
|
||||
{
|
||||
/* avoid <linux/mm.h> include hell */
|
||||
extern unsigned long max_mapnr;
|
||||
unsigned long pfn_offset = ARCH_PFN_OFFSET;
|
||||
|
||||
return pfn >= pfn_offset && pfn < max_mapnr;
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
#define virt_to_pfn(kaddr) PFN_DOWN(PHYSADDR(kaddr))
|
||||
#define virt_to_page(kaddr) pfn_to_page(virt_to_pfn(kaddr))
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
#define _PAGE_SPECIAL_SHIFT 11
|
||||
#define _PAGE_HGLOBAL_SHIFT 12 /* HGlobal is a PMD bit */
|
||||
#define _PAGE_PFN_SHIFT 12
|
||||
#define _PAGE_SWP_EXCLUSIVE_SHIFT 23
|
||||
#define _PAGE_PFN_END_SHIFT 48
|
||||
#define _PAGE_NO_READ_SHIFT 61
|
||||
#define _PAGE_NO_EXEC_SHIFT 62
|
||||
@@ -33,6 +34,9 @@
|
||||
#define _PAGE_PROTNONE (_ULCAST_(1) << _PAGE_PROTNONE_SHIFT)
|
||||
#define _PAGE_SPECIAL (_ULCAST_(1) << _PAGE_SPECIAL_SHIFT)
|
||||
|
||||
/* We borrow bit 23 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (_ULCAST_(1) << _PAGE_SWP_EXCLUSIVE_SHIFT)
|
||||
|
||||
/* Used by TLB hardware (placed in EntryLo*) */
|
||||
#define _PAGE_VALID (_ULCAST_(1) << _PAGE_VALID_SHIFT)
|
||||
#define _PAGE_DIRTY (_ULCAST_(1) << _PAGE_DIRTY_SHIFT)
|
||||
|
||||
@@ -249,13 +249,26 @@ extern void pud_init(void *addr);
|
||||
extern void pmd_init(void *addr);
|
||||
|
||||
/*
|
||||
* Non-present pages: high 40 bits are offset, next 8 bits type,
|
||||
* low 16 bits zero.
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3
|
||||
* 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2
|
||||
* <--------------------------- offset ---------------------------
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* --------------> E <--- type ---> <---------- zeroes ---------->
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* The zero'ed bits include _PAGE_PRESENT and _PAGE_PROTNONE.
|
||||
*/
|
||||
static inline pte_t mk_swap_pte(unsigned long type, unsigned long offset)
|
||||
{ pte_t pte; pte_val(pte) = (type << 16) | (offset << 24); return pte; }
|
||||
{ pte_t pte; pte_val(pte) = ((type & 0x7f) << 16) | (offset << 24); return pte; }
|
||||
|
||||
#define __swp_type(x) (((x).val >> 16) & 0xff)
|
||||
#define __swp_type(x) (((x).val >> 16) & 0x7f)
|
||||
#define __swp_offset(x) ((x).val >> 24)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { pte_val(mk_swap_pte((type), (offset))) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
@@ -263,6 +276,23 @@ static inline pte_t mk_swap_pte(unsigned long type, unsigned long offset)
|
||||
#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
|
||||
#define __swp_entry_to_pmd(x) ((pmd_t) { (x).val | _PAGE_HUGE })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
extern void paging_init(void);
|
||||
|
||||
#define pte_none(pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))
|
||||
|
||||
@@ -149,7 +149,7 @@ static inline void tlb_flush(struct mmu_gather *tlb)
|
||||
struct vm_area_struct vma;
|
||||
|
||||
vma.vm_mm = tlb->mm;
|
||||
vma.vm_flags = 0;
|
||||
vm_flags_init(&vma, 0);
|
||||
if (tlb->fullmm) {
|
||||
flush_tlb_mm(tlb->mm);
|
||||
return;
|
||||
|
||||
@@ -46,6 +46,9 @@
|
||||
#define _CACHEMASK040 (~0x060)
|
||||
#define _PAGE_GLOBAL040 0x400 /* 68040 global bit, used for kva descs */
|
||||
|
||||
/* We borrow bit 24 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE CF_PAGE_NOCACHE
|
||||
|
||||
/*
|
||||
* Externally used page protection values.
|
||||
*/
|
||||
@@ -254,15 +257,41 @@ static inline pte_t pte_mkcache(pte_t pte)
|
||||
extern pgd_t kernel_pg_dir[PTRS_PER_PGD];
|
||||
|
||||
/*
|
||||
* Encode and de-code a swap entry (must be !pte_none(e) && !pte_present(e))
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <------------------ offset -------------> 0 0 0 E <-- type --->
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*/
|
||||
#define __swp_type(x) ((x).val & 0xFF)
|
||||
#define __swp_type(x) ((x).val & 0x7f)
|
||||
#define __swp_offset(x) ((x).val >> 11)
|
||||
#define __swp_entry(typ, off) ((swp_entry_t) { (typ) | \
|
||||
#define __swp_entry(typ, off) ((swp_entry_t) { ((typ) & 0x7f) | \
|
||||
(off << 11) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) (__pte((x).val))
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
#define pmd_pfn(pmd) (pmd_val(pmd) >> PAGE_SHIFT)
|
||||
#define pmd_page(pmd) (pfn_to_page(pmd_val(pmd) >> PAGE_SHIFT))
|
||||
|
||||
|
||||
@@ -41,6 +41,9 @@
|
||||
|
||||
#define _PAGE_PROTNONE 0x004
|
||||
|
||||
/* We borrow bit 11 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE 0x800
|
||||
|
||||
#ifndef __ASSEMBLY__
|
||||
|
||||
/* This is the cache mode to be used for pages containing page descriptors for
|
||||
@@ -124,7 +127,7 @@ static inline void pud_set(pud_t *pudp, pmd_t *pmdp)
|
||||
* expects pmd_page() to exists, only to then DCE it all. Provide a dummy to
|
||||
* make the compiler happy.
|
||||
*/
|
||||
#define pmd_page(pmd) NULL
|
||||
#define pmd_page(pmd) ((struct page *)NULL)
|
||||
|
||||
|
||||
#define pud_none(pud) (!pud_val(pud))
|
||||
@@ -169,12 +172,40 @@ static inline pte_t pte_mkcache(pte_t pte)
|
||||
#define swapper_pg_dir kernel_pg_dir
|
||||
extern pgd_t kernel_pg_dir[128];
|
||||
|
||||
/* Encode and de-code a swap entry (must be !pte_none(e) && !pte_present(e)) */
|
||||
#define __swp_type(x) (((x).val >> 4) & 0xff)
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <----------------- offset ------------> E <-- type ---> 0 0 0 0
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*/
|
||||
#define __swp_type(x) (((x).val >> 4) & 0x7f)
|
||||
#define __swp_offset(x) ((x).val >> 12)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { ((type) << 4) | ((offset) << 12) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (((type) & 0x7f) << 4) | ((offset) << 12) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
#endif /* !__ASSEMBLY__ */
|
||||
#endif /* _MOTOROLA_PGTABLE_H */
|
||||
|
||||
@@ -62,11 +62,7 @@ extern unsigned long _ramend;
|
||||
#include <asm/page_no.h>
|
||||
#endif
|
||||
|
||||
#ifndef CONFIG_MMU
|
||||
#define __phys_to_pfn(paddr) ((unsigned long)((paddr) >> PAGE_SHIFT))
|
||||
#define __pfn_to_phys(pfn) PFN_PHYS(pfn)
|
||||
#endif
|
||||
|
||||
#include <asm-generic/getorder.h>
|
||||
#include <asm-generic/memory_model.h>
|
||||
|
||||
#endif /* _M68K_PAGE_H */
|
||||
|
||||
@@ -134,7 +134,6 @@ extern int m68k_virt_to_node_shift;
|
||||
})
|
||||
|
||||
#define ARCH_PFN_OFFSET (m68k_memory[0].addr >> PAGE_SHIFT)
|
||||
#include <asm-generic/memory_model.h>
|
||||
|
||||
#define virt_addr_valid(kaddr) ((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
|
||||
#define pfn_valid(pfn) virt_addr_valid(pfn_to_virt(pfn))
|
||||
|
||||
@@ -13,9 +13,8 @@ extern unsigned long memory_end;
|
||||
#define clear_user_page(page, vaddr, pg) clear_page(page)
|
||||
#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
|
||||
|
||||
#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
|
||||
alloc_page_vma(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vaddr)
|
||||
#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
|
||||
#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
|
||||
vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
|
||||
|
||||
#define __pa(vaddr) ((unsigned long)(vaddr))
|
||||
#define __va(paddr) ((void *)((unsigned long)(paddr)))
|
||||
@@ -26,13 +25,11 @@ extern unsigned long memory_end;
|
||||
#define virt_to_page(addr) (mem_map + (((unsigned long)(addr)-PAGE_OFFSET) >> PAGE_SHIFT))
|
||||
#define page_to_virt(page) __va(((((page) - mem_map) << PAGE_SHIFT) + PAGE_OFFSET))
|
||||
|
||||
#define pfn_to_page(pfn) virt_to_page(pfn_to_virt(pfn))
|
||||
#define page_to_pfn(page) virt_to_pfn(page_to_virt(page))
|
||||
#define pfn_valid(pfn) ((pfn) < max_mapnr)
|
||||
|
||||
#define virt_addr_valid(kaddr) (((unsigned long)(kaddr) >= PAGE_OFFSET) && \
|
||||
((unsigned long)(kaddr) < memory_end))
|
||||
|
||||
#define ARCH_PFN_OFFSET PHYS_PFN(PAGE_OFFSET_RAW)
|
||||
|
||||
#endif /* __ASSEMBLY__ */
|
||||
|
||||
#endif /* _M68K_PAGE_NO_H */
|
||||
|
||||
@@ -31,12 +31,6 @@
|
||||
extern void paging_init(void);
|
||||
#define swapper_pg_dir ((pgd_t *) 0)
|
||||
|
||||
#define __swp_type(x) (0)
|
||||
#define __swp_offset(x) (0)
|
||||
#define __swp_entry(typ,off) ((swp_entry_t) { ((typ) | ((off) << 7)) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
/*
|
||||
* ZERO_PAGE is a global shared page that is always zero: used
|
||||
* for zero-mapped memory areas etc..
|
||||
|
||||
@@ -71,6 +71,9 @@
|
||||
#define SUN3_PMD_MASK (0x0000003F)
|
||||
#define SUN3_PMD_MAGIC (0x0000002B)
|
||||
|
||||
/* We borrow bit 6 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE 0x040
|
||||
|
||||
#ifndef __ASSEMBLY__
|
||||
|
||||
/*
|
||||
@@ -152,12 +155,41 @@ static inline pte_t pte_mkcache(pte_t pte) { return pte; }
|
||||
extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
|
||||
extern pgd_t kernel_pg_dir[PTRS_PER_PGD];
|
||||
|
||||
/* Macros to (de)construct the fake PTEs representing swap pages. */
|
||||
#define __swp_type(x) ((x).val & 0x7F)
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* 0 <--------------------- offset ----------------> E <- type -->
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*/
|
||||
#define __swp_type(x) ((x).val & 0x3f)
|
||||
#define __swp_offset(x) (((x).val) >> 7)
|
||||
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) | ((offset) << 7)) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (((type) & 0x3f) | \
|
||||
(((offset) << 7) & ~SUN3_PAGE_VALID)) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
#endif /* !__ASSEMBLY__ */
|
||||
#endif /* !_SUN3_PGTABLE_H */
|
||||
|
||||
@@ -112,7 +112,6 @@ extern int page_is_ram(unsigned long pfn);
|
||||
# define page_to_phys(page) (page_to_pfn(page) << PAGE_SHIFT)
|
||||
|
||||
# define ARCH_PFN_OFFSET (memory_start >> PAGE_SHIFT)
|
||||
# define pfn_valid(pfn) ((pfn) >= ARCH_PFN_OFFSET && (pfn) < (max_mapnr + ARCH_PFN_OFFSET))
|
||||
# endif /* __ASSEMBLY__ */
|
||||
|
||||
#define virt_addr_valid(vaddr) (pfn_valid(virt_to_pfn(vaddr)))
|
||||
|
||||
@@ -131,10 +131,10 @@ extern pte_t *va_to_pte(unsigned long address);
|
||||
* of the 16 available. Bit 24-26 of the TLB are cleared in the TLB
|
||||
* miss handler. Bit 27 is PAGE_USER, thus selecting the correct
|
||||
* zone.
|
||||
* - PRESENT *must* be in the bottom two bits because swap cache
|
||||
* entries use the top 30 bits. Because 4xx doesn't support SMP
|
||||
* anyway, M is irrelevant so we borrow it for PAGE_PRESENT. Bit 30
|
||||
* is cleared in the TLB miss handler before the TLB entry is loaded.
|
||||
* - PRESENT *must* be in the bottom two bits because swap PTEs use the top
|
||||
* 30 bits. Because 4xx doesn't support SMP anyway, M is irrelevant so we
|
||||
* borrow it for PAGE_PRESENT. Bit 30 is cleared in the TLB miss handler
|
||||
* before the TLB entry is loaded.
|
||||
* - All other bits of the PTE are loaded into TLBLO without
|
||||
* * modification, leaving us only the bits 20, 21, 24, 25, 26, 30 for
|
||||
* software PTE bits. We actually use bits 21, 24, 25, and
|
||||
@@ -155,6 +155,9 @@ extern pte_t *va_to_pte(unsigned long address);
|
||||
#define _PAGE_ACCESSED 0x400 /* software: R: page referenced */
|
||||
#define _PMD_PRESENT PAGE_MASK
|
||||
|
||||
/* We borrow bit 24 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE _PAGE_DIRTY
|
||||
|
||||
/*
|
||||
* Some bits are unused...
|
||||
*/
|
||||
@@ -393,18 +396,39 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
|
||||
|
||||
/*
|
||||
* Encode and decode a swap entry.
|
||||
* Note that the bits we use in a PTE for representing a swap entry
|
||||
* must not include the _PAGE_PRESENT bit, or the _PAGE_HASHPTE bit
|
||||
* (if used). -- paulus
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
|
||||
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||||
* <------------------ offset -------------------> E < type -> 0 0
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*/
|
||||
#define __swp_type(entry) ((entry).val & 0x3f)
|
||||
#define __swp_type(entry) ((entry).val & 0x1f)
|
||||
#define __swp_offset(entry) ((entry).val >> 6)
|
||||
#define __swp_entry(type, offset) \
|
||||
((swp_entry_t) { (type) | ((offset) << 6) })
|
||||
((swp_entry_t) { ((type) & 0x1f) | ((offset) << 6) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 2 })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val << 2 })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
extern unsigned long iopa(unsigned long addr);
|
||||
|
||||
/* Values for nocacheflag and cmode */
|
||||
|
||||
@@ -224,34 +224,6 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x);
|
||||
|
||||
#define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
|
||||
|
||||
#ifdef CONFIG_FLATMEM
|
||||
|
||||
static inline int pfn_valid(unsigned long pfn)
|
||||
{
|
||||
/* avoid <linux/mm.h> include hell */
|
||||
extern unsigned long max_mapnr;
|
||||
unsigned long pfn_offset = ARCH_PFN_OFFSET;
|
||||
|
||||
return pfn >= pfn_offset && pfn < max_mapnr;
|
||||
}
|
||||
|
||||
#elif defined(CONFIG_SPARSEMEM)
|
||||
|
||||
/* pfn_valid is defined in linux/mmzone.h */
|
||||
|
||||
#elif defined(CONFIG_NUMA)
|
||||
|
||||
#define pfn_valid(pfn) \
|
||||
({ \
|
||||
unsigned long __pfn = (pfn); \
|
||||
int __n = pfn_to_nid(__pfn); \
|
||||
((__n >= 0) ? (__pfn < NODE_DATA(__n)->node_start_pfn + \
|
||||
NODE_DATA(__n)->node_spanned_pages) \
|
||||
: 0); \
|
||||
})
|
||||
|
||||
#endif
|
||||
|
||||
#define virt_to_pfn(kaddr) PFN_DOWN(virt_to_phys((void *)(kaddr)))
|
||||
#define virt_to_page(kaddr) pfn_to_page(virt_to_pfn(kaddr))
|
||||
|
||||
|
||||
@@ -191,49 +191,113 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_t prot)
|
||||
|
||||
#define pte_page(x) pfn_to_page(pte_pfn(x))
|
||||
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*/
|
||||
#if defined(CONFIG_CPU_R3K_TLB)
|
||||
|
||||
/* Swap entries must have VALID bit cleared. */
|
||||
/*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <----------- offset ------------> < type -> V G E 0 0 0 0 0 0 P
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* _PAGE_PRESENT (P), _PAGE_VALID (V) and_PAGE_GLOBAL (G) have to remain
|
||||
* unused.
|
||||
*/
|
||||
#define __swp_type(x) (((x).val >> 10) & 0x1f)
|
||||
#define __swp_offset(x) ((x).val >> 15)
|
||||
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) << 10) | ((offset) << 15) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (((type) & 0x1f) << 10) | ((offset) << 15) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
/* We borrow bit 7 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (1 << 7)
|
||||
|
||||
#else
|
||||
|
||||
#if defined(CONFIG_XPA)
|
||||
|
||||
/* Swap entries must have VALID and GLOBAL bits cleared. */
|
||||
/*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3
|
||||
* 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2
|
||||
* 0 0 0 0 0 0 E P <------------------ zeroes ------------------->
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <----------------- offset ------------------> < type -> V G 0 0
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* _PAGE_PRESENT (P), _PAGE_VALID (V) and_PAGE_GLOBAL (G) have to remain
|
||||
* unused.
|
||||
*/
|
||||
#define __swp_type(x) (((x).val >> 4) & 0x1f)
|
||||
#define __swp_offset(x) ((x).val >> 9)
|
||||
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) << 4) | ((offset) << 9) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (((type) & 0x1f) << 4) | ((offset) << 9) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { (pte).pte_high })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { 0, (x).val })
|
||||
|
||||
/*
|
||||
* We borrow bit 57 (bit 25 in the low PTE) to store the exclusive marker in
|
||||
* swap PTEs.
|
||||
*/
|
||||
#define _PAGE_SWP_EXCLUSIVE (1 << 25)
|
||||
|
||||
#elif defined(CONFIG_PHYS_ADDR_T_64BIT) && defined(CONFIG_CPU_MIPS32)
|
||||
|
||||
/* Swap entries must have VALID and GLOBAL bits cleared. */
|
||||
/*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3
|
||||
* 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2
|
||||
* <------------------ zeroes -------------------> E P 0 0 0 0 0 0
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <------------------- offset --------------------> < type -> V G
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* _PAGE_PRESENT (P), _PAGE_VALID (V) and_PAGE_GLOBAL (G) have to remain
|
||||
* unused.
|
||||
*/
|
||||
#define __swp_type(x) (((x).val >> 2) & 0x1f)
|
||||
#define __swp_offset(x) ((x).val >> 7)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { ((type) << 2) | ((offset) << 7) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (((type) & 0x1f) << 2) | ((offset) << 7) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { (pte).pte_high })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { 0, (x).val })
|
||||
|
||||
/*
|
||||
* We borrow bit 39 (bit 7 in the low PTE) to store the exclusive marker in swap
|
||||
* PTEs.
|
||||
*/
|
||||
#define _PAGE_SWP_EXCLUSIVE (1 << 7)
|
||||
|
||||
#else
|
||||
/*
|
||||
* Constraints:
|
||||
* _PAGE_PRESENT at bit 0
|
||||
* _PAGE_MODIFIED at bit 4
|
||||
* _PAGE_GLOBAL at bit 6
|
||||
* _PAGE_VALID at bit 7
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <------------- offset --------------> < type -> 0 0 0 0 0 0 E P
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* _PAGE_PRESENT (P), _PAGE_VALID (V) and_PAGE_GLOBAL (G) have to remain
|
||||
* unused. The location of V and G varies.
|
||||
*/
|
||||
#define __swp_type(x) (((x).val >> 8) & 0x1f)
|
||||
#define __swp_offset(x) ((x).val >> 13)
|
||||
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) << 8) | ((offset) << 13) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { ((type) << 8) | ((offset) << 13) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
/* We borrow bit 1 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (1 << 1)
|
||||
|
||||
#endif /* defined(CONFIG_PHYS_ADDR_T_64BIT) && defined(CONFIG_CPU_MIPS32) */
|
||||
|
||||
#endif /* defined(CONFIG_CPU_R3K_TLB) */
|
||||
|
||||
@@ -320,16 +320,31 @@ extern void pud_init(void *addr);
|
||||
extern void pmd_init(void *addr);
|
||||
|
||||
/*
|
||||
* Non-present pages: high 40 bits are offset, next 8 bits type,
|
||||
* low 16 bits zero.
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3
|
||||
* 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2
|
||||
* <--------------------------- offset ---------------------------
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* --------------> E <-- type ---> <---------- zeroes ----------->
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*/
|
||||
static inline pte_t mk_swap_pte(unsigned long type, unsigned long offset)
|
||||
{ pte_t pte; pte_val(pte) = (type << 16) | (offset << 24); return pte; }
|
||||
{ pte_t pte; pte_val(pte) = ((type & 0x7f) << 16) | (offset << 24); return pte; }
|
||||
|
||||
#define __swp_type(x) (((x).val >> 16) & 0xff)
|
||||
#define __swp_type(x) (((x).val >> 16) & 0x7f)
|
||||
#define __swp_offset(x) ((x).val >> 24)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { pte_val(mk_swap_pte((type), (offset))) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
/* We borrow bit 23 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (1 << 23)
|
||||
|
||||
#endif /* _ASM_PGTABLE_64_H */
|
||||
|
||||
@@ -528,6 +528,41 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
|
||||
}
|
||||
#endif
|
||||
|
||||
#if defined(CONFIG_PHYS_ADDR_T_64BIT) && defined(CONFIG_CPU_MIPS32)
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte.pte_low & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte.pte_low |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte.pte_low &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
#else
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
#endif
|
||||
|
||||
extern void __update_tlb(struct vm_area_struct *vma, unsigned long address,
|
||||
pte_t pte);
|
||||
|
||||
@@ -86,15 +86,6 @@ extern struct page *mem_map;
|
||||
|
||||
# define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
|
||||
|
||||
static inline bool pfn_valid(unsigned long pfn)
|
||||
{
|
||||
/* avoid <linux/mm.h> include hell */
|
||||
extern unsigned long max_mapnr;
|
||||
unsigned long pfn_offset = ARCH_PFN_OFFSET;
|
||||
|
||||
return pfn >= pfn_offset && pfn < max_mapnr;
|
||||
}
|
||||
|
||||
# define virt_to_page(vaddr) pfn_to_page(PFN_DOWN(virt_to_phys(vaddr)))
|
||||
# define virt_addr_valid(vaddr) pfn_valid(PFN_DOWN(virt_to_phys(vaddr)))
|
||||
|
||||
|
||||
@@ -31,4 +31,7 @@
|
||||
#define _PAGE_ACCESSED (1<<26) /* page referenced */
|
||||
#define _PAGE_DIRTY (1<<27) /* dirty page */
|
||||
|
||||
/* We borrow bit 31 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE (1<<31)
|
||||
|
||||
#endif /* _ASM_NIOS2_PGTABLE_BITS_H */
|
||||
|
||||
@@ -232,23 +232,44 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
__FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
/*
|
||||
* Encode and decode a swap entry (must be !pte_none(pte) && !pte_present(pte):
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* 31 30 29 28 27 26 25 24 23 22 21 20 19 18 ... 1 0
|
||||
* 0 0 0 0 type. 0 0 0 0 0 0 offset.........
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* This gives us up to 2**2 = 4 swap files and 2**20 * 4K = 4G per swap file.
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* E < type -> 0 0 0 0 0 0 <-------------- offset --------------->
|
||||
*
|
||||
* Note that the offset field is always non-zero, thus !pte_none(pte) is always
|
||||
* true.
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*
|
||||
* Note that the offset field is always non-zero if the swap type is 0, thus
|
||||
* !pte_none() is always true.
|
||||
*/
|
||||
#define __swp_type(swp) (((swp).val >> 26) & 0x3)
|
||||
#define __swp_type(swp) (((swp).val >> 26) & 0x1f)
|
||||
#define __swp_offset(swp) ((swp).val & 0xfffff)
|
||||
#define __swp_entry(type, off) ((swp_entry_t) { (((type) & 0x3) << 26) \
|
||||
#define __swp_entry(type, off) ((swp_entry_t) { (((type) & 0x1f) << 26) \
|
||||
| ((off) & 0xfffff) })
|
||||
#define __swp_entry_to_pte(swp) ((pte_t) { (swp).val })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
extern void __init paging_init(void);
|
||||
extern void __init mmu_init(void);
|
||||
|
||||
|
||||
@@ -80,8 +80,6 @@ typedef struct page *pgtable_t;
|
||||
|
||||
#define page_to_phys(page) ((dma_addr_t)page_to_pfn(page) << PAGE_SHIFT)
|
||||
|
||||
#define pfn_valid(pfn) ((pfn) < max_mapnr)
|
||||
|
||||
#define virt_addr_valid(kaddr) (pfn_valid(virt_to_pfn(kaddr)))
|
||||
|
||||
#endif /* __ASSEMBLY__ */
|
||||
|
||||
@@ -154,6 +154,9 @@ extern void paging_init(void);
|
||||
#define _KERNPG_TABLE \
|
||||
(_PAGE_BASE | _PAGE_SRE | _PAGE_SWE | _PAGE_ACCESSED | _PAGE_DIRTY)
|
||||
|
||||
/* We borrow bit 11 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE _PAGE_U_SHARED
|
||||
|
||||
#define PAGE_NONE __pgprot(_PAGE_ALL)
|
||||
#define PAGE_READONLY __pgprot(_PAGE_ALL | _PAGE_URE | _PAGE_SRE)
|
||||
#define PAGE_READONLY_X __pgprot(_PAGE_ALL | _PAGE_URE | _PAGE_SRE | _PAGE_EXEC)
|
||||
@@ -385,16 +388,43 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,
|
||||
|
||||
/* __PHX__ FIXME, SWAP, this probably doesn't work */
|
||||
|
||||
/* Encode and de-code a swap entry (must be !pte_none(e) && !pte_present(e)) */
|
||||
/* Since the PAGE_PRESENT bit is bit 4, we can use the bits above */
|
||||
|
||||
#define __swp_type(x) (((x).val >> 5) & 0x7f)
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <-------------- offset ---------------> E <- type --> 0 0 0 0 0
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* The zero'ed bits include _PAGE_PRESENT.
|
||||
*/
|
||||
#define __swp_type(x) (((x).val >> 5) & 0x3f)
|
||||
#define __swp_offset(x) ((x).val >> 12)
|
||||
#define __swp_entry(type, offset) \
|
||||
((swp_entry_t) { ((type) << 5) | ((offset) << 12) })
|
||||
((swp_entry_t) { (((type) & 0x3f) << 5) | ((offset) << 12) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
typedef pte_t *pte_addr_t;
|
||||
|
||||
#endif /* __ASSEMBLY__ */
|
||||
|
||||
@@ -155,10 +155,6 @@ extern int npmem_ranges;
|
||||
#define __pa(x) ((unsigned long)(x)-PAGE_OFFSET)
|
||||
#define __va(x) ((void *)((unsigned long)(x)+PAGE_OFFSET))
|
||||
|
||||
#ifndef CONFIG_SPARSEMEM
|
||||
#define pfn_valid(pfn) ((pfn) < max_mapnr)
|
||||
#endif
|
||||
|
||||
#ifdef CONFIG_HUGETLB_PAGE
|
||||
#define HPAGE_SHIFT PMD_SHIFT /* fixed for transparent huge pages */
|
||||
#define HPAGE_SIZE ((1UL) << HPAGE_SHIFT)
|
||||
|
||||
@@ -218,6 +218,9 @@ extern void __update_cache(pte_t pte);
|
||||
#define _PAGE_KERNEL_RWX (_PAGE_KERNEL_EXEC | _PAGE_WRITE)
|
||||
#define _PAGE_KERNEL (_PAGE_KERNEL_RO | _PAGE_WRITE)
|
||||
|
||||
/* We borrow bit 23 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE _PAGE_ACCESSED
|
||||
|
||||
/* The pgd/pmd contains a ptr (in phys addr space); since all pgds/pmds
|
||||
* are page-aligned, we don't care about the PAGE_OFFSET bits, except
|
||||
* for a few meta-information bits, so we shift the address to be
|
||||
@@ -394,17 +397,48 @@ extern void paging_init (void);
|
||||
|
||||
#define update_mmu_cache(vms,addr,ptep) __update_cache(*ptep)
|
||||
|
||||
/* Encode and de-code a swap entry */
|
||||
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs (32bit):
|
||||
*
|
||||
* 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
|
||||
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||||
* <---------------- offset -----------------> P E <ofs> < type ->
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* _PAGE_PRESENT (P) must be 0.
|
||||
*
|
||||
* For the 64bit version, the offset is extended by 32bit.
|
||||
*/
|
||||
#define __swp_type(x) ((x).val & 0x1f)
|
||||
#define __swp_offset(x) ( (((x).val >> 6) & 0x7) | \
|
||||
(((x).val >> 8) & ~0x7) )
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (type) | \
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { \
|
||||
((type) & 0x1f) | \
|
||||
((offset & 0x7) << 6) | \
|
||||
((offset & ~0x7) << 8) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) |= _PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
pte_val(pte) &= ~_PAGE_SWP_EXCLUSIVE;
|
||||
return pte;
|
||||
}
|
||||
|
||||
static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep)
|
||||
{
|
||||
pte_t pte;
|
||||
|
||||
@@ -42,6 +42,9 @@
|
||||
#define _PMD_PRESENT_MASK (PAGE_MASK)
|
||||
#define _PMD_BAD (~PAGE_MASK)
|
||||
|
||||
/* We borrow the _PAGE_USER bit to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE _PAGE_USER
|
||||
|
||||
/* And here we include common definitions */
|
||||
|
||||
#define _PAGE_KERNEL_RO 0
|
||||
@@ -363,17 +366,41 @@ static inline void __ptep_set_access_flags(struct vm_area_struct *vma,
|
||||
#define pmd_page(pmd) pfn_to_page(pmd_pfn(pmd))
|
||||
|
||||
/*
|
||||
* Encode and decode a swap entry.
|
||||
* Note that the bits we use in a PTE for representing a swap entry
|
||||
* must not include the _PAGE_PRESENT bit or the _PAGE_HASHPTE bit (if used).
|
||||
* -- paulus
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs (32bit PTEs):
|
||||
*
|
||||
* 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
|
||||
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||||
* <----------------- offset --------------------> < type -> E H P
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
* _PAGE_PRESENT (P) and __PAGE_HASHPTE (H) must be 0.
|
||||
*
|
||||
* For 64bit PTEs, the offset is extended by 32bit.
|
||||
*/
|
||||
#define __swp_type(entry) ((entry).val & 0x1f)
|
||||
#define __swp_offset(entry) ((entry).val >> 5)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (type) | ((offset) << 5) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { ((type) & 0x1f) | ((offset) << 5) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
return __pte(pte_val(pte) | _PAGE_SWP_EXCLUSIVE);
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
return __pte(pte_val(pte) & ~_PAGE_SWP_EXCLUSIVE);
|
||||
}
|
||||
|
||||
/* Generic accessors to PTE bits */
|
||||
static inline int pte_write(pte_t pte) { return !!(pte_val(pte) & _PAGE_RW);}
|
||||
static inline int pte_read(pte_t pte) { return 1; }
|
||||
|
||||
@@ -717,7 +717,6 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
|
||||
}
|
||||
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
|
||||
|
||||
#define __HAVE_ARCH_PTE_SWP_EXCLUSIVE
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
return __pte_raw(pte_raw(pte) | cpu_to_be64(_PAGE_SWP_EXCLUSIVE));
|
||||
|
||||
@@ -360,18 +360,30 @@ static inline int pte_young(pte_t pte)
|
||||
#endif
|
||||
|
||||
#define pmd_page(pmd) pfn_to_page(pmd_pfn(pmd))
|
||||
|
||||
/*
|
||||
* Encode and decode a swap entry.
|
||||
* Note that the bits we use in a PTE for representing a swap entry
|
||||
* must not include the _PAGE_PRESENT bit.
|
||||
* -- paulus
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs (32bit PTEs):
|
||||
*
|
||||
* 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
|
||||
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||||
* <------------------ offset -------------------> < type -> E 0 0
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*
|
||||
* For 64bit PTEs, the offset is extended by 32bit.
|
||||
*/
|
||||
#define __swp_type(entry) ((entry).val & 0x1f)
|
||||
#define __swp_offset(entry) ((entry).val >> 5)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { (type) | ((offset) << 5) })
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { ((type) & 0x1f) | ((offset) << 5) })
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
|
||||
|
||||
/* We borrow LSB 2 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE 0x000004
|
||||
|
||||
#endif /* !__ASSEMBLY__ */
|
||||
|
||||
#endif /* __ASM_POWERPC_NOHASH_32_PGTABLE_H */
|
||||
|
||||
@@ -27,9 +27,9 @@
|
||||
* of the 16 available. Bit 24-26 of the TLB are cleared in the TLB
|
||||
* miss handler. Bit 27 is PAGE_USER, thus selecting the correct
|
||||
* zone.
|
||||
* - PRESENT *must* be in the bottom two bits because swap cache
|
||||
* entries use the top 30 bits. Because 40x doesn't support SMP
|
||||
* anyway, M is irrelevant so we borrow it for PAGE_PRESENT. Bit 30
|
||||
* - PRESENT *must* be in the bottom two bits because swap PTEs
|
||||
* use the top 30 bits. Because 40x doesn't support SMP anyway, M is
|
||||
* irrelevant so we borrow it for PAGE_PRESENT. Bit 30
|
||||
* is cleared in the TLB miss handler before the TLB entry is loaded.
|
||||
* - All other bits of the PTE are loaded into TLBLO without
|
||||
* modification, leaving us only the bits 20, 21, 24, 25, 26, 30 for
|
||||
|
||||
@@ -56,20 +56,10 @@
|
||||
* above bits. Note that the bit values are CPU specific, not architecture
|
||||
* specific.
|
||||
*
|
||||
* The kernel PTE entry holds an arch-dependent swp_entry structure under
|
||||
* certain situations. In other words, in such situations some portion of
|
||||
* the PTE bits are used as a swp_entry. In the PPC implementation, the
|
||||
* 3-24th LSB are shared with swp_entry, however the 0-2nd three LSB still
|
||||
* hold protection values. That means the three protection bits are
|
||||
* reserved for both PTE and SWAP entry at the most significant three
|
||||
* LSBs.
|
||||
*
|
||||
* There are three protection bits available for SWAP entry:
|
||||
* _PAGE_PRESENT
|
||||
* _PAGE_HASHPTE (if HW has)
|
||||
*
|
||||
* So those three bits have to be inside of 0-2nd LSB of PTE.
|
||||
*
|
||||
* The kernel PTE entry can be an ordinary PTE mapping a page or a special swap
|
||||
* PTE. In case of a swap PTE, LSB 2-24 are used to store information regarding
|
||||
* the swap entry. However LSB 0-1 still hold protection values, for example,
|
||||
* to distinguish swap PTEs from ordinary PTEs, and must be used with care.
|
||||
*/
|
||||
|
||||
#define _PAGE_PRESENT 0x00000001 /* S: PTE valid */
|
||||
|
||||
@@ -11,8 +11,8 @@
|
||||
32 33 34 35 36 ... 50 51 52 53 54 55 56 57 58 59 60 61 62 63
|
||||
RPN...................... 0 0 U0 U1 U2 U3 UX SX UW SW UR SR
|
||||
|
||||
- PRESENT *must* be in the bottom three bits because swap cache
|
||||
entries use the top 29 bits.
|
||||
- PRESENT *must* be in the bottom two bits because swap PTEs use
|
||||
the top 30 bits.
|
||||
|
||||
*/
|
||||
|
||||
|
||||
@@ -276,22 +276,40 @@ static inline void __ptep_set_access_flags(struct vm_area_struct *vma,
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
/* Encode and de-code a swap entry */
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3
|
||||
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||||
* <-------------------------- offset ----------------------------
|
||||
*
|
||||
* 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 6 6 6 6
|
||||
* 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
|
||||
* --------------> <----------- zero ------------> E < type -> 0 0
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*/
|
||||
#define MAX_SWAPFILES_CHECK() do { \
|
||||
BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS); \
|
||||
} while (0)
|
||||
|
||||
#define SWP_TYPE_BITS 5
|
||||
#define __swp_type(x) (((x).val >> _PAGE_BIT_SWAP_TYPE) \
|
||||
#define __swp_type(x) (((x).val >> 2) \
|
||||
& ((1UL << SWP_TYPE_BITS) - 1))
|
||||
#define __swp_offset(x) ((x).val >> PTE_RPN_SHIFT)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) { \
|
||||
((type) << _PAGE_BIT_SWAP_TYPE) \
|
||||
(((type) & 0x1f) << 2) \
|
||||
| ((offset) << PTE_RPN_SHIFT) })
|
||||
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val((pte)) })
|
||||
#define __swp_entry_to_pte(x) __pte((x).val)
|
||||
|
||||
/* We borrow MSB 56 (LSB 7) to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE 0x80
|
||||
|
||||
int map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot);
|
||||
void unmap_kernel_page(unsigned long va);
|
||||
extern int __meminit vmemmap_create_mapping(unsigned long start,
|
||||
|
||||
@@ -151,6 +151,21 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
|
||||
return __pte((pte_val(pte) & _PAGE_CHG_MASK) | pgprot_val(newprot));
|
||||
}
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
return __pte(pte_val(pte) | _PAGE_SWP_EXCLUSIVE);
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
return __pte(pte_val(pte) & ~_PAGE_SWP_EXCLUSIVE);
|
||||
}
|
||||
|
||||
/* Insert a PTE, top-level function is out of line. It uses an inline
|
||||
* low level function in the respective pgtable-* files
|
||||
*/
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
/* Architected bits */
|
||||
#define _PAGE_PRESENT 0x000001 /* software: pte contains a translation */
|
||||
#define _PAGE_SW1 0x000002
|
||||
#define _PAGE_BIT_SWAP_TYPE 2
|
||||
#define _PAGE_BAP_SR 0x000004
|
||||
#define _PAGE_BAP_UR 0x000008
|
||||
#define _PAGE_BAP_SW 0x000010
|
||||
|
||||
@@ -117,15 +117,6 @@ extern long long virt_phys_offset;
|
||||
|
||||
#ifdef CONFIG_FLATMEM
|
||||
#define ARCH_PFN_OFFSET ((unsigned long)(MEMORY_START >> PAGE_SHIFT))
|
||||
#ifndef __ASSEMBLY__
|
||||
extern unsigned long max_mapnr;
|
||||
static inline bool pfn_valid(unsigned long pfn)
|
||||
{
|
||||
unsigned long min_pfn = ARCH_PFN_OFFSET;
|
||||
|
||||
return pfn >= min_pfn && pfn < max_mapnr;
|
||||
}
|
||||
#endif
|
||||
#endif
|
||||
|
||||
#define virt_to_pfn(kaddr) (__pa(kaddr) >> PAGE_SHIFT)
|
||||
|
||||
@@ -132,7 +132,7 @@ void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size)
|
||||
* address decoding but I'd rather not deal with those outside of the
|
||||
* reserved 64K legacy region.
|
||||
*/
|
||||
area = __get_vm_area_caller(size, 0, PHB_IO_BASE, PHB_IO_END,
|
||||
area = __get_vm_area_caller(size, VM_IOREMAP, PHB_IO_BASE, PHB_IO_END,
|
||||
__builtin_return_address(0));
|
||||
if (!area)
|
||||
return NULL;
|
||||
|
||||
@@ -120,10 +120,8 @@ int vdso_join_timens(struct task_struct *task, struct time_namespace *ns)
|
||||
|
||||
mmap_read_lock(mm);
|
||||
for_each_vma(vmi, vma) {
|
||||
unsigned long size = vma->vm_end - vma->vm_start;
|
||||
|
||||
if (vma_is_special_mapping(vma, &vvar_spec))
|
||||
zap_page_range(vma, vma->vm_start, size);
|
||||
zap_vma_pages(vma);
|
||||
}
|
||||
mmap_read_unlock(mm);
|
||||
|
||||
|
||||
@@ -393,6 +393,7 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm,
|
||||
{
|
||||
unsigned long gfn = memslot->base_gfn;
|
||||
unsigned long end, start = gfn_to_hva(kvm, gfn);
|
||||
unsigned long vm_flags;
|
||||
int ret = 0;
|
||||
struct vm_area_struct *vma;
|
||||
int merge_flag = (merge) ? MADV_MERGEABLE : MADV_UNMERGEABLE;
|
||||
@@ -409,12 +410,15 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm,
|
||||
ret = H_STATE;
|
||||
break;
|
||||
}
|
||||
/* Copy vm_flags to avoid partial modifications in ksm_madvise */
|
||||
vm_flags = vma->vm_flags;
|
||||
ret = ksm_madvise(vma, vma->vm_start, vma->vm_end,
|
||||
merge_flag, &vma->vm_flags);
|
||||
merge_flag, &vm_flags);
|
||||
if (ret) {
|
||||
ret = H_STATE;
|
||||
break;
|
||||
}
|
||||
vm_flags_reset(vma, vm_flags);
|
||||
start = vma->vm_end;
|
||||
} while (end > vma->vm_end);
|
||||
|
||||
|
||||
@@ -324,7 +324,7 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
|
||||
|
||||
/*
|
||||
|
||||
@@ -156,7 +156,7 @@ static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr,
|
||||
* VM_NOHUGEPAGE and split them.
|
||||
*/
|
||||
for_each_vma_range(vmi, vma, addr + len) {
|
||||
vma->vm_flags |= VM_NOHUGEPAGE;
|
||||
vm_flags_set(vma, VM_NOHUGEPAGE);
|
||||
walk_page_vma(vma, &subpage_walk_ops, NULL);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -414,7 +414,7 @@ static vm_fault_t vas_mmap_fault(struct vm_fault *vmf)
|
||||
/*
|
||||
* When the LPAR lost credits due to core removal or during
|
||||
* migration, invalidate the existing mapping for the current
|
||||
* paste addresses and set windows in-active (zap_page_range in
|
||||
* paste addresses and set windows in-active (zap_vma_pages in
|
||||
* reconfig_close_windows()).
|
||||
* New mapping will be done later after migration or new credits
|
||||
* available. So continue to receive faults if the user space
|
||||
@@ -525,7 +525,7 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma)
|
||||
pfn = paste_addr >> PAGE_SHIFT;
|
||||
|
||||
/* flags, page_prot from cxl_mmap(), except we want cachable */
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_cached(vma->vm_page_prot);
|
||||
|
||||
prot = __pgprot(pgprot_val(vma->vm_page_prot) | _PAGE_DIRTY);
|
||||
|
||||
@@ -291,7 +291,7 @@ static int spufs_mem_mmap(struct file *file, struct vm_area_struct *vma)
|
||||
if (!(vma->vm_flags & VM_SHARED))
|
||||
return -EINVAL;
|
||||
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
|
||||
|
||||
vma->vm_ops = &spufs_mem_mmap_vmops;
|
||||
@@ -381,7 +381,7 @@ static int spufs_cntl_mmap(struct file *file, struct vm_area_struct *vma)
|
||||
if (!(vma->vm_flags & VM_SHARED))
|
||||
return -EINVAL;
|
||||
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
|
||||
|
||||
vma->vm_ops = &spufs_cntl_mmap_vmops;
|
||||
@@ -1043,7 +1043,7 @@ static int spufs_signal1_mmap(struct file *file, struct vm_area_struct *vma)
|
||||
if (!(vma->vm_flags & VM_SHARED))
|
||||
return -EINVAL;
|
||||
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
|
||||
|
||||
vma->vm_ops = &spufs_signal1_mmap_vmops;
|
||||
@@ -1179,7 +1179,7 @@ static int spufs_signal2_mmap(struct file *file, struct vm_area_struct *vma)
|
||||
if (!(vma->vm_flags & VM_SHARED))
|
||||
return -EINVAL;
|
||||
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
|
||||
|
||||
vma->vm_ops = &spufs_signal2_mmap_vmops;
|
||||
@@ -1302,7 +1302,7 @@ static int spufs_mss_mmap(struct file *file, struct vm_area_struct *vma)
|
||||
if (!(vma->vm_flags & VM_SHARED))
|
||||
return -EINVAL;
|
||||
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
|
||||
|
||||
vma->vm_ops = &spufs_mss_mmap_vmops;
|
||||
@@ -1364,7 +1364,7 @@ static int spufs_psmap_mmap(struct file *file, struct vm_area_struct *vma)
|
||||
if (!(vma->vm_flags & VM_SHARED))
|
||||
return -EINVAL;
|
||||
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
|
||||
|
||||
vma->vm_ops = &spufs_psmap_mmap_vmops;
|
||||
@@ -1424,7 +1424,7 @@ static int spufs_mfc_mmap(struct file *file, struct vm_area_struct *vma)
|
||||
if (!(vma->vm_flags & VM_SHARED))
|
||||
return -EINVAL;
|
||||
|
||||
vma->vm_flags |= VM_IO | VM_PFNMAP;
|
||||
vm_flags_set(vma, VM_IO | VM_PFNMAP);
|
||||
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
|
||||
|
||||
vma->vm_ops = &spufs_mfc_mmap_vmops;
|
||||
|
||||
@@ -760,8 +760,7 @@ static int reconfig_close_windows(struct vas_caps *vcap, int excess_creds,
|
||||
* is done before the original mmap() and after the ioctl.
|
||||
*/
|
||||
if (vma)
|
||||
zap_page_range(vma, vma->vm_start,
|
||||
vma->vm_end - vma->vm_start);
|
||||
zap_vma_pages(vma);
|
||||
|
||||
mmap_write_unlock(task_ref->mm);
|
||||
mutex_unlock(&task_ref->mmap_mutex);
|
||||
|
||||
@@ -171,11 +171,6 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x);
|
||||
|
||||
#define sym_to_pfn(x) __phys_to_pfn(__pa_symbol(x))
|
||||
|
||||
#ifdef CONFIG_FLATMEM
|
||||
#define pfn_valid(pfn) \
|
||||
(((pfn) >= ARCH_PFN_OFFSET) && (((pfn) - ARCH_PFN_OFFSET) < max_mapnr))
|
||||
#endif
|
||||
|
||||
#endif /* __ASSEMBLY__ */
|
||||
|
||||
#define virt_addr_valid(vaddr) ({ \
|
||||
|
||||
@@ -27,6 +27,9 @@
|
||||
*/
|
||||
#define _PAGE_PROT_NONE _PAGE_GLOBAL
|
||||
|
||||
/* Used for swap PTEs only. */
|
||||
#define _PAGE_SWP_EXCLUSIVE _PAGE_ACCESSED
|
||||
|
||||
#define _PAGE_PFN_SHIFT 10
|
||||
|
||||
/*
|
||||
|
||||
@@ -728,16 +728,18 @@ extern pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
|
||||
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
|
||||
|
||||
/*
|
||||
* Encode and decode a swap entry
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Format of swap PTE:
|
||||
* bit 0: _PAGE_PRESENT (zero)
|
||||
* bit 1 to 3: _PAGE_LEAF (zero)
|
||||
* bit 5: _PAGE_PROT_NONE (zero)
|
||||
* bits 6 to 10: swap type
|
||||
* bits 10 to XLEN-1: swap offset
|
||||
* bit 6: exclusive marker
|
||||
* bits 7 to 11: swap type
|
||||
* bits 11 to XLEN-1: swap offset
|
||||
*/
|
||||
#define __SWP_TYPE_SHIFT 6
|
||||
#define __SWP_TYPE_SHIFT 7
|
||||
#define __SWP_TYPE_BITS 5
|
||||
#define __SWP_TYPE_MASK ((1UL << __SWP_TYPE_BITS) - 1)
|
||||
#define __SWP_OFFSET_SHIFT (__SWP_TYPE_BITS + __SWP_TYPE_SHIFT)
|
||||
@@ -748,11 +750,27 @@ extern pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
|
||||
#define __swp_type(x) (((x).val >> __SWP_TYPE_SHIFT) & __SWP_TYPE_MASK)
|
||||
#define __swp_offset(x) ((x).val >> __SWP_OFFSET_SHIFT)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t) \
|
||||
{ ((type) << __SWP_TYPE_SHIFT) | ((offset) << __SWP_OFFSET_SHIFT) })
|
||||
{ (((type) & __SWP_TYPE_MASK) << __SWP_TYPE_SHIFT) | \
|
||||
((offset) << __SWP_OFFSET_SHIFT) })
|
||||
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_mkexclusive(pte_t pte)
|
||||
{
|
||||
return __pte(pte_val(pte) | _PAGE_SWP_EXCLUSIVE);
|
||||
}
|
||||
|
||||
static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
{
|
||||
return __pte(pte_val(pte) & ~_PAGE_SWP_EXCLUSIVE);
|
||||
}
|
||||
|
||||
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
|
||||
#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
|
||||
#define __swp_entry_to_pmd(swp) __pmd((swp).val)
|
||||
|
||||
@@ -124,13 +124,11 @@ int vdso_join_timens(struct task_struct *task, struct time_namespace *ns)
|
||||
mmap_read_lock(mm);
|
||||
|
||||
for_each_vma(vmi, vma) {
|
||||
unsigned long size = vma->vm_end - vma->vm_start;
|
||||
|
||||
if (vma_is_special_mapping(vma, vdso_info.dm))
|
||||
zap_page_range(vma, vma->vm_start, size);
|
||||
zap_vma_pages(vma);
|
||||
#ifdef CONFIG_COMPAT
|
||||
if (vma_is_special_mapping(vma, compat_vdso_info.dm))
|
||||
zap_page_range(vma, vma->vm_start, size);
|
||||
zap_vma_pages(vma);
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
@@ -73,9 +73,8 @@ static inline void copy_page(void *to, void *from)
|
||||
#define clear_user_page(page, vaddr, pg) clear_page(page)
|
||||
#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
|
||||
|
||||
#define alloc_zeroed_user_highpage_movable(vma, vaddr) \
|
||||
alloc_page_vma(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, vma, vaddr)
|
||||
#define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE_MOVABLE
|
||||
#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
|
||||
vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
|
||||
|
||||
/*
|
||||
* These are used to make use of C type-checking..
|
||||
|
||||
@@ -827,7 +827,6 @@ static inline int pmd_protnone(pmd_t pmd)
|
||||
}
|
||||
#endif
|
||||
|
||||
#define __HAVE_ARCH_PTE_SWP_EXCLUSIVE
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte_val(pte) & _PAGE_SWP_EXCLUSIVE;
|
||||
|
||||
@@ -59,11 +59,9 @@ int vdso_join_timens(struct task_struct *task, struct time_namespace *ns)
|
||||
|
||||
mmap_read_lock(mm);
|
||||
for_each_vma(vmi, vma) {
|
||||
unsigned long size = vma->vm_end - vma->vm_start;
|
||||
|
||||
if (!vma_is_special_mapping(vma, &vvar_mapping))
|
||||
continue;
|
||||
zap_page_range(vma, vma->vm_start, size);
|
||||
zap_vma_pages(vma);
|
||||
break;
|
||||
}
|
||||
mmap_read_unlock(mm);
|
||||
|
||||
+7
-4
@@ -722,7 +722,7 @@ void gmap_discard(struct gmap *gmap, unsigned long from, unsigned long to)
|
||||
if (is_vm_hugetlb_page(vma))
|
||||
continue;
|
||||
size = min(to - gaddr, PMD_SIZE - (gaddr & ~PMD_MASK));
|
||||
zap_page_range(vma, vmaddr, size);
|
||||
zap_page_range_single(vma, vmaddr, size, NULL);
|
||||
}
|
||||
mmap_read_unlock(gmap->mm);
|
||||
}
|
||||
@@ -2522,8 +2522,7 @@ static inline void thp_split_mm(struct mm_struct *mm)
|
||||
VMA_ITERATOR(vmi, mm, 0);
|
||||
|
||||
for_each_vma(vmi, vma) {
|
||||
vma->vm_flags &= ~VM_HUGEPAGE;
|
||||
vma->vm_flags |= VM_NOHUGEPAGE;
|
||||
vm_flags_mod(vma, VM_NOHUGEPAGE, VM_HUGEPAGE);
|
||||
walk_page_vma(vma, &thp_split_walk_ops, NULL);
|
||||
}
|
||||
mm->def_flags |= VM_NOHUGEPAGE;
|
||||
@@ -2588,14 +2587,18 @@ int gmap_mark_unmergeable(void)
|
||||
{
|
||||
struct mm_struct *mm = current->mm;
|
||||
struct vm_area_struct *vma;
|
||||
unsigned long vm_flags;
|
||||
int ret;
|
||||
VMA_ITERATOR(vmi, mm, 0);
|
||||
|
||||
for_each_vma(vmi, vma) {
|
||||
/* Copy vm_flags to avoid partial modifications in ksm_madvise */
|
||||
vm_flags = vma->vm_flags;
|
||||
ret = ksm_madvise(vma, vma->vm_start, vma->vm_end,
|
||||
MADV_UNMERGEABLE, &vma->vm_flags);
|
||||
MADV_UNMERGEABLE, &vm_flags);
|
||||
if (ret)
|
||||
return ret;
|
||||
vm_flags_reset(vma, vm_flags);
|
||||
}
|
||||
mm->def_flags &= ~VM_MERGEABLE;
|
||||
return 0;
|
||||
|
||||
@@ -169,9 +169,6 @@ typedef struct page *pgtable_t;
|
||||
#define PFN_START (__MEMORY_START >> PAGE_SHIFT)
|
||||
#define ARCH_PFN_OFFSET (PFN_START)
|
||||
#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
|
||||
#ifdef CONFIG_FLATMEM
|
||||
#define pfn_valid(pfn) ((pfn) >= min_low_pfn && (pfn) < max_low_pfn)
|
||||
#endif
|
||||
#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
|
||||
|
||||
#include <asm-generic/memory_model.h>
|
||||
|
||||
@@ -423,40 +423,69 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
#endif
|
||||
|
||||
/*
|
||||
* Encode and de-code a swap entry
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
*
|
||||
* Constraints:
|
||||
* _PAGE_PRESENT at bit 8
|
||||
* _PAGE_PROTNONE at bit 9
|
||||
*
|
||||
* For the normal case, we encode the swap type into bits 0:7 and the
|
||||
* swap offset into bits 10:30. For the 64-bit PTE case, we keep the
|
||||
* preserved bits in the low 32-bits and use the upper 32 as the swap
|
||||
* offset (along with a 5-bit type), following the same approach as x86
|
||||
* PAE. This keeps the logic quite simple.
|
||||
* For the normal case, we encode the swap type and offset into the swap PTE
|
||||
* such that bits 8 and 9 stay zero. For the 64-bit PTE case, we use the
|
||||
* upper 32 for the swap offset and swap type, following the same approach as
|
||||
* x86 PAE. This keeps the logic quite simple.
|
||||
*
|
||||
* As is evident by the Alpha code, if we ever get a 64-bit unsigned
|
||||
* long (swp_entry_t) to match up with the 64-bit PTEs, this all becomes
|
||||
* much cleaner..
|
||||
*
|
||||
* NOTE: We should set ZEROs at the position of _PAGE_PRESENT
|
||||
* and _PAGE_PROTNONE bits
|
||||
*/
|
||||
|
||||
#ifdef CONFIG_X2TLB
|
||||
/*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3
|
||||
* 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2
|
||||
* <--------------------- offset ----------------------> < type ->
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <------------------- zeroes --------------------> E 0 0 0 0 0 0
|
||||
*/
|
||||
#define __swp_type(x) ((x).val & 0x1f)
|
||||
#define __swp_offset(x) ((x).val >> 5)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t){ (type) | (offset) << 5})
|
||||
#define __swp_entry(type, offset) ((swp_entry_t){ ((type) & 0x1f) | (offset) << 5})
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t){ (pte).pte_high })
|
||||
#define __swp_entry_to_pte(x) ((pte_t){ 0, (x).val })
|
||||
|
||||
#else
|
||||
#define __swp_type(x) ((x).val & 0xff)
|
||||
/*
|
||||
* Format of swap PTEs:
|
||||
*
|
||||
* 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1
|
||||
* 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
|
||||
* <--------------- offset ----------------> 0 0 0 0 E < type -> 0
|
||||
*
|
||||
* E is the exclusive marker that is not stored in swap entries.
|
||||
*/
|
||||
#define __swp_type(x) ((x).val & 0x1f)
|
||||
#define __swp_offset(x) ((x).val >> 10)
|
||||
#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) <<10})
|
||||
#define __swp_entry(type, offset) ((swp_entry_t){((type) & 0x1f) | (offset) << 10})
|
||||
|
||||
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 1 })
|
||||
#define __swp_entry_to_pte(x) ((pte_t) { (x).val << 1 })
|
||||
#endif
|
||||
|
||||
/* In both cases, we borrow bit 6 to store the exclusive marker in swap PTEs. */
|
||||
#define _PAGE_SWP_EXCLUSIVE _PAGE_USER
|
||||
|
||||
static inline int pte_swp_exclusive(pte_t pte)
|
||||
{
|
||||
return pte.pte_low & _PAGE_SWP_EXCLUSIVE;
|
||||
}
|
||||
|
||||
PTE_BIT_FUNC(low, swp_mkexclusive, |= _PAGE_SWP_EXCLUSIVE);
|
||||
PTE_BIT_FUNC(low, swp_clear_exclusive, &= ~_PAGE_SWP_EXCLUSIVE);
|
||||
|
||||
#endif /* __ASSEMBLY__ */
|
||||
#endif /* __ASM_SH_PGTABLE_32_H */
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user