tmakin/ack-tegra
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge 5bdb4078e1 ("Merge tag 'sched_ext-for-7.1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext") into android-mainline

Browse Source 
Steps on the way to v7.1-rc1

Change-Id: Id90e55d00688bee471df782d351a3ff512177599
Signed-off-by: Carlos Llamas <cmllamas@google.com>
This commit is contained in:
Carlos Llamas committed 2026-05-19 17:33:38 +00:00
2 parents 3c49f2cbba 5bdb4078e1
commit 8c7d581ebc
67 files changed
+6305 -1541
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Documentation/admin-guide/kernel-parameters.txt
+2 -1
View File
@@ -8566,7 +8566,8 @@ Kernel parameters
workqueue.default_affinity_scope=
Select the default affinity scope to use for unbound
workqueues. Can be one of "cpu", "smt", "cache",
"numa" and "system". Default is "cache". For more
"cache_shard", "numa" and "system". Default is
"cache_shard". For more
information, see the Affinity Scopes section in
Documentation/core-api/workqueue.rst.
Documentation/core-api/workqueue.rst
+10 -4
View File
@@ -378,9 +378,9 @@ Affinity Scopes
An unbound workqueue groups CPUs according to its affinity scope to improve
cache locality. For example, if a workqueue is using the default affinity
scope of "cache", it will group CPUs according to last level cache
boundaries. A work item queued on the workqueue will be assigned to a worker
on one of the CPUs which share the last level cache with the issuing CPU.
scope of "cache_shard", it will group CPUs into sub-LLC shards. A work item
queued on the workqueue will be assigned to a worker on one of the CPUs
within the same shard as the issuing CPU.
Once started, the worker may or may not be allowed to move outside the scope
depending on the ``affinity_strict`` setting of the scope.
@@ -402,7 +402,13 @@ Workqueue currently supports the following affinity scopes.
``cache``
CPUs are grouped according to cache boundaries. Which specific cache
boundary is used is determined by the arch code. L3 is used in a lot of
cases. This is the default affinity scope.
cases.
``cache_shard``
CPUs are grouped into sub-LLC shards of at most ``wq_cache_shard_size``
cores (default 8, tunable via the ``workqueue.cache_shard_size`` boot
parameter). Shards are always split on core (SMT group) boundaries.
This is the default affinity scope.
``numa``
CPUs are grouped according to NUMA boundaries.
Documentation/driver-api/driver-model/devres.rst
+4
View File
@@ -464,3 +464,7 @@ SPI
WATCHDOG
devm_watchdog_register_device()
WORKQUEUE
devm_alloc_workqueue()
devm_alloc_ordered_workqueue()
Documentation/scheduler/sched-ext.rst
+188 -19
View File
@@ -93,6 +93,55 @@ scheduler has been loaded):
# cat /sys/kernel/sched_ext/enable_seq
1
Each running scheduler also exposes a per-scheduler ``events`` file under
``/sys/kernel/sched_ext/<scheduler-name>/events`` that tracks diagnostic
counters. Each counter occupies one ``name value`` line:
.. code-block:: none
# cat /sys/kernel/sched_ext/simple/events
SCX_EV_SELECT_CPU_FALLBACK 0
SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE 0
SCX_EV_DISPATCH_KEEP_LAST 123
SCX_EV_ENQ_SKIP_EXITING 0
SCX_EV_ENQ_SKIP_MIGRATION_DISABLED 0
SCX_EV_REENQ_IMMED 0
SCX_EV_REENQ_LOCAL_REPEAT 0
SCX_EV_REFILL_SLICE_DFL 456789
SCX_EV_BYPASS_DURATION 0
SCX_EV_BYPASS_DISPATCH 0
SCX_EV_BYPASS_ACTIVATE 0
SCX_EV_INSERT_NOT_OWNED 0
SCX_EV_SUB_BYPASS_DISPATCH 0
The counters are described in ``kernel/sched/ext_internal.h``; briefly:
* ``SCX_EV_SELECT_CPU_FALLBACK``: ops.select_cpu() returned a CPU unusable by
the task and the core scheduler silently picked a fallback CPU.
* ``SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE``: a local-DSQ dispatch was redirected
to the global DSQ because the target CPU went offline.
* ``SCX_EV_DISPATCH_KEEP_LAST``: a task continued running because no other
task was available (only when ``SCX_OPS_ENQ_LAST`` is not set).
* ``SCX_EV_ENQ_SKIP_EXITING``: an exiting task was dispatched to the local DSQ
directly, bypassing ops.enqueue() (only when ``SCX_OPS_ENQ_EXITING`` is not set).
* ``SCX_EV_ENQ_SKIP_MIGRATION_DISABLED``: a migration-disabled task was
dispatched to its local DSQ directly (only when
``SCX_OPS_ENQ_MIGRATION_DISABLED`` is not set).
* ``SCX_EV_REENQ_IMMED``: a task dispatched with ``SCX_ENQ_IMMED`` was
re-enqueued because the target CPU was not available for immediate execution.
* ``SCX_EV_REENQ_LOCAL_REPEAT``: a reenqueue of the local DSQ triggered
another reenqueue; recurring counts indicate incorrect ``SCX_ENQ_REENQ``
handling in the BPF scheduler.
* ``SCX_EV_REFILL_SLICE_DFL``: a task's time slice was refilled with the
default value (``SCX_SLICE_DFL``).
* ``SCX_EV_BYPASS_DURATION``: total nanoseconds spent in bypass mode.
* ``SCX_EV_BYPASS_DISPATCH``: number of tasks dispatched while in bypass mode.
* ``SCX_EV_BYPASS_ACTIVATE``: number of times bypass mode was activated.
* ``SCX_EV_INSERT_NOT_OWNED``: attempted to insert a task not owned by this
scheduler into a DSQ; such attempts are silently ignored.
* ``SCX_EV_SUB_BYPASS_DISPATCH``: tasks dispatched from sub-scheduler bypass
DSQs (only relevant with ``CONFIG_EXT_SUB_SCHED``).
``tools/sched_ext/scx_show_state.py`` is a drgn script which shows more
detailed information:
@@ -228,16 +277,23 @@ The following briefly shows how a waking task is scheduled and executed.
scheduler can wake up any cpu using the ``scx_bpf_kick_cpu()`` helper,
using ``ops.select_cpu()`` judiciously can be simpler and more efficient.
A task can be immediately inserted into a DSQ from ``ops.select_cpu()``
by calling ``scx_bpf_dsq_insert()``. If the task is inserted into
``SCX_DSQ_LOCAL`` from ``ops.select_cpu()``, it will be inserted into the
local DSQ of whichever CPU is returned from ``ops.select_cpu()``.
Additionally, inserting directly from ``ops.select_cpu()`` will cause the
``ops.enqueue()`` callback to be skipped.
Note that the scheduler core will ignore an invalid CPU selection, for
example, if it's outside the allowed cpumask of the task.
A task can be immediately inserted into a DSQ from ``ops.select_cpu()``
by calling ``scx_bpf_dsq_insert()`` or ``scx_bpf_dsq_insert_vtime()``.
If the task is inserted into ``SCX_DSQ_LOCAL`` from
``ops.select_cpu()``, it will be added to the local DSQ of whichever CPU
is returned from ``ops.select_cpu()``. Additionally, inserting directly
from ``ops.select_cpu()`` will cause the ``ops.enqueue()`` callback to
be skipped.
Any other attempt to store a task in BPF-internal data structures from
``ops.select_cpu()`` does not prevent ``ops.enqueue()`` from being
invoked. This is discouraged, as it can introduce racy behavior or
inconsistent state.
2. Once the target CPU is selected, ``ops.enqueue()`` is invoked (unless the
task was inserted directly from ``ops.select_cpu()``). ``ops.enqueue()``
can make one of the following decisions:
@@ -251,6 +307,61 @@ The following briefly shows how a waking task is scheduled and executed.
* Queue the task on the BPF side.
**Task State Tracking and ops.dequeue() Semantics**
A task is in the "BPF scheduler's custody" when the BPF scheduler is
responsible for managing its lifecycle. A task enters custody when it is
dispatched to a user DSQ or stored in the BPF scheduler's internal data
structures. Custody is entered only from ``ops.enqueue()`` for those
operations. The only exception is dispatching to a user DSQ from
``ops.select_cpu()``: although the task is not yet technically in BPF
scheduler custody at that point, the dispatch has the same semantic
effect as dispatching from ``ops.enqueue()`` for custody-related
purposes.
Once ``ops.enqueue()`` is called, the task may or may not enter custody
depending on what the scheduler does:
* **Directly dispatched to terminal DSQs** (``SCX_DSQ_LOCAL``,
``SCX_DSQ_LOCAL_ON | cpu``, or ``SCX_DSQ_GLOBAL``): the BPF scheduler
is done with the task - it either goes straight to a CPU's local run
queue or to the global DSQ as a fallback. The task never enters (or
exits) BPF custody, and ``ops.dequeue()`` will not be called.
* **Dispatch to user-created DSQs** (custom DSQs): the task enters the
BPF scheduler's custody. When the task later leaves BPF custody
(dispatched to a terminal DSQ, picked by core-sched, or dequeued for
sleep/property changes), ``ops.dequeue()`` will be called exactly
once.
* **Stored in BPF data structures** (e.g., internal BPF queues): the
task is in BPF custody. ``ops.dequeue()`` will be called when it
leaves (e.g., when ``ops.dispatch()`` moves it to a terminal DSQ, or
on property change / sleep).
When a task leaves BPF scheduler custody, ``ops.dequeue()`` is invoked.
The dequeue can happen for different reasons, distinguished by flags:
1. **Regular dispatch**: when a task in BPF custody is dispatched to a
terminal DSQ from ``ops.dispatch()`` (leaving BPF custody for
execution), ``ops.dequeue()`` is triggered without any special flags.
2. **Core scheduling pick**: when ``CONFIG_SCHED_CORE`` is enabled and
core scheduling picks a task for execution while it's still in BPF
custody, ``ops.dequeue()`` is called with the
``SCX_DEQ_CORE_SCHED_EXEC`` flag.
3. **Scheduling property change**: when a task property changes (via
operations like ``sched_setaffinity()``, ``sched_setscheduler()``,
priority changes, CPU migrations, etc.) while the task is still in
BPF custody, ``ops.dequeue()`` is called with the
``SCX_DEQ_SCHED_CHANGE`` flag set in ``deq_flags``.
**Important**: Once a task has left BPF custody (e.g., after being
dispatched to a terminal DSQ), property changes will not trigger
``ops.dequeue()``, since the task is no longer managed by the BPF
scheduler.
3. When a CPU is ready to schedule, it first looks at its local DSQ. If
empty, it then looks at the global DSQ. If there still isn't a task to
run, ``ops.dispatch()`` is invoked which can use the following two
@@ -264,9 +375,9 @@ The following briefly shows how a waking task is scheduled and executed.
rather than performing them immediately. There can be up to
``ops.dispatch_max_batch`` pending tasks.
* ``scx_bpf_move_to_local()`` moves a task from the specified non-local
* ``scx_bpf_dsq_move_to_local()`` moves a task from the specified non-local
DSQ to the dispatching DSQ. This function cannot be called with any BPF
locks held. ``scx_bpf_move_to_local()`` flushes the pending insertions
locks held. ``scx_bpf_dsq_move_to_local()`` flushes the pending insertions
tasks before trying to move from the specified DSQ.
4. After ``ops.dispatch()`` returns, if there are tasks in the local DSQ,
@@ -297,8 +408,8 @@ for more information.
Task Lifecycle
--------------
The following pseudo-code summarizes the entire lifecycle of a task managed
by a sched_ext scheduler:
The following pseudo-code presents a rough overview of the entire lifecycle
of a task managed by a sched_ext scheduler:
.. code-block:: c
@@ -311,22 +422,37 @@ by a sched_ext scheduler:
ops.runnable(); /* Task becomes ready to run */
while (task is runnable) {
if (task is not in a DSQ && task->scx.slice == 0) {
while (task_is_runnable(task)) {
if (task is not in a DSQ || task->scx.slice == 0) {
ops.enqueue(); /* Task can be added to a DSQ */
/* Task property change (i.e., affinity, nice, etc.)? */
if (sched_change(task)) {
ops.dequeue(); /* Exiting BPF scheduler custody */
ops.quiescent();
/* Property change callback, e.g. ops.set_weight() */
ops.runnable();
continue;
}
/* Any usable CPU becomes available */
ops.dispatch(); /* Task is moved to a local DSQ */
ops.dispatch(); /* Task is moved to a local DSQ */
ops.dequeue(); /* Exiting BPF scheduler custody */
}
ops.running(); /* Task starts running on its assigned CPU */
while (task->scx.slice > 0 && task is runnable)
while (task_is_runnable(task) && task->scx.slice > 0) {
ops.tick(); /* Called every 1/HZ seconds */
if (task->scx.slice == 0)
ops.dispatch(); /* task->scx.slice can be refilled */
}
ops.stopping(); /* Task stops running (time slice expires or wait) */
/* Task's CPU becomes available */
ops.dispatch(); /* task->scx.slice can be refilled */
}
ops.quiescent(); /* Task releases its assigned CPU (wait) */
@@ -335,6 +461,30 @@ by a sched_ext scheduler:
ops.disable(); /* Disable BPF scheduling for the task */
ops.exit_task(); /* Task is destroyed */
Note that the above pseudo-code does not cover all possible state transitions
and edge cases, to name a few examples:
* ``ops.dispatch()`` may fail to move the task to a local DSQ due to a racing
property change on that task, in which case ``ops.dispatch()`` will be
retried.
* The task may be direct-dispatched to a local DSQ from ``ops.enqueue()``,
in which case ``ops.dispatch()`` and ``ops.dequeue()`` are skipped and we go
straight to ``ops.running()``.
* Property changes may occur at virtually any point during the task's lifecycle,
not just when the task is queued and waiting to be dispatched. For example,
changing a property of a running task will lead to the callback sequence
``ops.stopping()`` -> ``ops.quiescent()`` -> (property change callback) ->
``ops.runnable()`` -> ``ops.running()``.
* A sched_ext task can be preempted by a task from a higher-priority scheduling
class, in which case it will exit the tick-dispatch loop even though it is runnable
and has a non-zero slice.
See the "Scheduling Cycle" section for a more detailed description of how
a freshly woken up task gets on a CPU.
Where to Look
=============
@@ -377,6 +527,25 @@ Where to Look
scheduling. Tasks with CPU affinity are direct-dispatched in FIFO order;
all others are scheduled in user space by a simple vruntime scheduler.
Module Parameters
=================
sched_ext exposes two module parameters under the ``sched_ext.`` prefix that
control bypass-mode behaviour. These knobs are primarily for debugging; there
is usually no reason to change them during normal operation. They can be read
and written at runtime (mode 0600) via
``/sys/module/sched_ext/parameters/``.
``sched_ext.slice_bypass_us`` (default: 5000 µs)
The time slice assigned to all tasks when the scheduler is in bypass mode,
i.e. during BPF scheduler load, unload, and error recovery. Valid range is
100 µs to 100 ms.
``sched_ext.bypass_lb_intv_us`` (default: 500000 µs)
The interval at which the bypass-mode load balancer redistributes tasks
across CPUs. Set to 0 to disable load balancing during bypass mode. Valid
range is 0 to 10 s.
ABI Instability
===============
MAINTAINERS
+1
View File
@@ -24503,6 +24503,7 @@ F: Documentation/admin-guide/mm/slab.rst
F: Documentation/mm/slab.rst
F: include/linux/mempool.h
F: include/linux/slab.h
F: lib/tests/slub_kunit.c
F: mm/failslab.c
F: mm/mempool.c
F: mm/slab.h
drivers/firmware/efi/efi.c
+1 -1
View File
@@ -423,7 +423,7 @@ static int __init efisubsys_init(void)
* ordered workqueue (which creates only one execution context)
* should suffice for all our needs.
*/
efi_rts_wq = alloc_ordered_workqueue("efi_rts_wq", 0);
efi_rts_wq = alloc_ordered_workqueue("efi_runtime", WQ_SYSFS);
if (!efi_rts_wq) {
pr_err("Creating efi_rts_wq failed, EFI runtime services disabled.\n");
clear_bit(EFI_RUNTIME_SERVICES, &efi.flags);
include/linux/cgroup-defs.h
+5
View File
@@ -17,6 +17,7 @@
#include <linux/refcount.h>
#include <linux/percpu-refcount.h>
#include <linux/percpu-rwsem.h>
#include <linux/sched.h>
#include <linux/u64_stats_sync.h>
#include <linux/workqueue.h>
#include <linux/bpf-cgroup-defs.h>
@@ -167,6 +168,7 @@ struct cgroup_file {
struct kernfs_node *kn;
unsigned long notified_at;
struct timer_list notify_timer;
spinlock_t lock;
};
/*
@@ -627,6 +629,9 @@ struct cgroup {
#ifdef CONFIG_BPF_SYSCALL
struct bpf_local_storage __rcu *bpf_cgrp_storage;
#endif
#ifdef CONFIG_EXT_SUB_SCHED
struct scx_sched __rcu *scx_sched;
#endif
/* All ancestors including self */
union {
include/linux/cgroup.h
+63 -2
View File
@@ -42,6 +42,14 @@ struct kernel_clone_args;
#ifdef CONFIG_CGROUPS
/*
* To avoid confusing the compiler (and generating warnings) with code
* that attempts to access what would be a 0-element array (i.e. sized
* to a potentially empty array when CGROUP_SUBSYS_COUNT == 0), this
* constant expression can be added.
*/
#define CGROUP_HAS_SUBSYS_CONFIG (CGROUP_SUBSYS_COUNT > 0)
enum css_task_iter_flags {
CSS_TASK_ITER_PROCS = (1U << 0), /* walk only threadgroup leaders */
CSS_TASK_ITER_THREADED = (1U << 1), /* walk all threaded css_sets in the domain */
@@ -76,6 +84,7 @@ enum cgroup_lifetime_events {
extern struct file_system_type cgroup_fs_type;
extern struct cgroup_root cgrp_dfl_root;
extern struct css_set init_css_set;
extern struct mutex cgroup_mutex;
extern spinlock_t css_set_lock;
extern struct blocking_notifier_head cgroup_lifetime_notifier;
@@ -103,6 +112,8 @@ extern struct blocking_notifier_head cgroup_lifetime_notifier;
#define cgroup_subsys_on_dfl(ss) \
static_branch_likely(&ss ## _on_dfl_key)
bool cgroup_on_dfl(const struct cgroup *cgrp);
bool css_has_online_children(struct cgroup_subsys_state *css);
struct cgroup_subsys_state *css_from_id(int id, struct cgroup_subsys *ss);
struct cgroup_subsys_state *cgroup_e_css(struct cgroup *cgroup,
@@ -274,6 +285,32 @@ void css_task_iter_end(struct css_task_iter *it);
for ((pos) = css_next_descendant_post(NULL, (css)); (pos); \
(pos) = css_next_descendant_post((pos), (css)))
/* iterate over child cgrps, lock should be held throughout iteration */
#define cgroup_for_each_live_child(child, cgrp) \
list_for_each_entry((child), &(cgrp)->self.children, self.sibling) \
if (({ lockdep_assert_held(&cgroup_mutex); \
cgroup_is_dead(child); })) \
; \
else
/* walk live descendants in pre order */
#define cgroup_for_each_live_descendant_pre(dsct, d_css, cgrp) \
css_for_each_descendant_pre((d_css), cgroup_css((cgrp), NULL)) \
if (({ lockdep_assert_held(&cgroup_mutex); \
(dsct) = (d_css)->cgroup; \
cgroup_is_dead(dsct); })) \
; \
else
/* walk live descendants in postorder */
#define cgroup_for_each_live_descendant_post(dsct, d_css, cgrp) \
css_for_each_descendant_post((d_css), cgroup_css((cgrp), NULL)) \
if (({ lockdep_assert_held(&cgroup_mutex); \
(dsct) = (d_css)->cgroup; \
cgroup_is_dead(dsct); })) \
; \
else
/**
* cgroup_taskset_for_each - iterate cgroup_taskset
* @task: the loop cursor
@@ -336,6 +373,27 @@ static inline u64 cgroup_id(const struct cgroup *cgrp)
return cgrp->kn->id;
}
/**
* cgroup_css - obtain a cgroup's css for the specified subsystem
* @cgrp: the cgroup of interest
* @ss: the subsystem of interest (%NULL returns @cgrp->self)
*
* Return @cgrp's css (cgroup_subsys_state) associated with @ss. This
* function must be called either under cgroup_mutex or rcu_read_lock() and
* the caller is responsible for pinning the returned css if it wants to
* keep accessing it outside the said locks. This function may return
* %NULL if @cgrp doesn't have @subsys_id enabled.
*/
static inline struct cgroup_subsys_state *cgroup_css(struct cgroup *cgrp,
struct cgroup_subsys *ss)
{
if (CGROUP_HAS_SUBSYS_CONFIG && ss)
return rcu_dereference_check(cgrp->subsys[ss->id],
lockdep_is_held(&cgroup_mutex));
else
return &cgrp->self;
}
/**
* css_is_dying - test whether the specified css is dying
* @css: target css
@@ -372,6 +430,11 @@ static inline bool css_is_self(struct cgroup_subsys_state *css)
return false;
}
static inline bool cgroup_is_dead(const struct cgroup *cgrp)
{
return !(cgrp->self.flags & CSS_ONLINE);
}
static inline void cgroup_get(struct cgroup *cgrp)
{
css_get(&cgrp->self);
@@ -387,8 +450,6 @@ static inline void cgroup_put(struct cgroup *cgrp)
css_put(&cgrp->self);
}
extern struct mutex cgroup_mutex;
static inline void cgroup_lock(void)
{
mutex_lock(&cgroup_mutex);
include/linux/sched/ext.h
+64 -45
View File
@@ -62,6 +62,16 @@ enum scx_dsq_id_flags {
SCX_DSQ_LOCAL_CPU_MASK = 0xffffffffLLU,
};
struct scx_deferred_reenq_user {
struct list_head node;
u64 flags;
};
struct scx_dsq_pcpu {
struct scx_dispatch_q *dsq;
struct scx_deferred_reenq_user deferred_reenq_user;
};
/*
* A dispatch queue (DSQ) can be either a FIFO or p->scx.dsq_vtime ordered
* queue. A built-in DSQ is always a FIFO. The built-in local DSQs are used to
@@ -78,30 +88,58 @@ struct scx_dispatch_q {
u64 id;
struct rhash_head hash_node;
struct llist_node free_node;
struct scx_sched *sched;
struct scx_dsq_pcpu __percpu *pcpu;
struct rcu_head rcu;
};
/* scx_entity.flags */
/* sched_ext_entity.flags */
enum scx_ent_flags {
SCX_TASK_QUEUED = 1 << 0, /* on ext runqueue */
SCX_TASK_IN_CUSTODY = 1 << 1, /* in custody, needs ops.dequeue() when leaving */
SCX_TASK_RESET_RUNNABLE_AT = 1 << 2, /* runnable_at should be reset */
SCX_TASK_DEQD_FOR_SLEEP = 1 << 3, /* last dequeue was for SLEEP */
SCX_TASK_SUB_INIT = 1 << 4, /* task being initialized for a sub sched */
SCX_TASK_IMMED = 1 << 5, /* task is on local DSQ with %SCX_ENQ_IMMED */
SCX_TASK_STATE_SHIFT = 8, /* bit 8 and 9 are used to carry scx_task_state */
/*
* Bits 8 and 9 are used to carry task state:
*
* NONE ops.init_task() not called yet
* INIT ops.init_task() succeeded, but task can be cancelled
* READY fully initialized, but not in sched_ext
* ENABLED fully initialized and in sched_ext
*/
SCX_TASK_STATE_SHIFT = 8, /* bits 8 and 9 are used to carry task state */
SCX_TASK_STATE_BITS = 2,
SCX_TASK_STATE_MASK = ((1 << SCX_TASK_STATE_BITS) - 1) << SCX_TASK_STATE_SHIFT,
SCX_TASK_CURSOR = 1 << 31, /* iteration cursor, not a task */
};
SCX_TASK_NONE = 0 << SCX_TASK_STATE_SHIFT,
SCX_TASK_INIT = 1 << SCX_TASK_STATE_SHIFT,
SCX_TASK_READY = 2 << SCX_TASK_STATE_SHIFT,
SCX_TASK_ENABLED = 3 << SCX_TASK_STATE_SHIFT,
/* scx_entity.flags & SCX_TASK_STATE_MASK */
enum scx_task_state {
SCX_TASK_NONE, /* ops.init_task() not called yet */
SCX_TASK_INIT, /* ops.init_task() succeeded, but task can be cancelled */
SCX_TASK_READY, /* fully initialized, but not in sched_ext */
SCX_TASK_ENABLED, /* fully initialized and in sched_ext */
/*
* Bits 12 and 13 are used to carry reenqueue reason. In addition to
* %SCX_ENQ_REENQ flag, ops.enqueue() can also test for
* %SCX_TASK_REENQ_REASON_NONE to distinguish reenqueues.
*
* NONE not being reenqueued
* KFUNC reenqueued by scx_bpf_dsq_reenq() and friends
* IMMED reenqueued due to failed ENQ_IMMED
* PREEMPTED preempted while running
*/
SCX_TASK_REENQ_REASON_SHIFT = 12,
SCX_TASK_REENQ_REASON_BITS = 2,
SCX_TASK_REENQ_REASON_MASK = ((1 << SCX_TASK_REENQ_REASON_BITS) - 1) << SCX_TASK_REENQ_REASON_SHIFT,
SCX_TASK_NR_STATES,
SCX_TASK_REENQ_NONE = 0 << SCX_TASK_REENQ_REASON_SHIFT,
SCX_TASK_REENQ_KFUNC = 1 << SCX_TASK_REENQ_REASON_SHIFT,
SCX_TASK_REENQ_IMMED = 2 << SCX_TASK_REENQ_REASON_SHIFT,
SCX_TASK_REENQ_PREEMPTED = 3 << SCX_TASK_REENQ_REASON_SHIFT,
/* iteration cursor, not a task */
SCX_TASK_CURSOR = 1 << 31,
};
/* scx_entity.dsq_flags */
@@ -109,33 +147,6 @@ enum scx_ent_dsq_flags {
SCX_TASK_DSQ_ON_PRIQ = 1 << 0, /* task is queued on the priority queue of a dsq */
};
/*
* Mask bits for scx_entity.kf_mask. Not all kfuncs can be called from
* everywhere and the following bits track which kfunc sets are currently
* allowed for %current. This simple per-task tracking works because SCX ops
* nest in a limited way. BPF will likely implement a way to allow and disallow
* kfuncs depending on the calling context which will replace this manual
* mechanism. See scx_kf_allow().
*/
enum scx_kf_mask {
SCX_KF_UNLOCKED = 0, /* sleepable and not rq locked */
/* ENQUEUE and DISPATCH may be nested inside CPU_RELEASE */
SCX_KF_CPU_RELEASE = 1 << 0, /* ops.cpu_release() */
/*
* ops.dispatch() may release rq lock temporarily and thus ENQUEUE and
* SELECT_CPU may be nested inside. ops.dequeue (in REST) may also be
* nested inside DISPATCH.
*/
SCX_KF_DISPATCH = 1 << 1, /* ops.dispatch() */
SCX_KF_ENQUEUE = 1 << 2, /* ops.enqueue() and ops.select_cpu() */
SCX_KF_SELECT_CPU = 1 << 3, /* ops.select_cpu() */
SCX_KF_REST = 1 << 4, /* other rq-locked operations */
__SCX_KF_RQ_LOCKED = SCX_KF_CPU_RELEASE | SCX_KF_DISPATCH |
SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
__SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
};
enum scx_dsq_lnode_flags {
SCX_DSQ_LNODE_ITER_CURSOR = 1 << 0,
@@ -149,19 +160,31 @@ struct scx_dsq_list_node {
u32 priv; /* can be used by iter cursor */
};
#define INIT_DSQ_LIST_CURSOR(__node, __flags, __priv) \
#define INIT_DSQ_LIST_CURSOR(__cursor, __dsq, __flags) \
(struct scx_dsq_list_node) { \
.node = LIST_HEAD_INIT((__node).node), \
.node = LIST_HEAD_INIT((__cursor).node), \
.flags = SCX_DSQ_LNODE_ITER_CURSOR | (__flags), \
.priv = (__priv), \
.priv = READ_ONCE((__dsq)->seq), \
}
struct scx_sched;
/*
* The following is embedded in task_struct and contains all fields necessary
* for a task to be scheduled by SCX.
*/
struct sched_ext_entity {
#ifdef CONFIG_CGROUPS
/*
* Associated scx_sched. Updated either during fork or while holding
* both p->pi_lock and rq lock.
*/
struct scx_sched __rcu *sched;
#endif
struct scx_dispatch_q *dsq;
atomic_long_t ops_state;
u64 ddsp_dsq_id;
u64 ddsp_enq_flags;
struct scx_dsq_list_node dsq_list; /* dispatch order */
struct rb_node dsq_priq; /* p->scx.dsq_vtime order */
u32 dsq_seq;
@@ -171,9 +194,7 @@ struct sched_ext_entity {
s32 sticky_cpu;
s32 holding_cpu;
s32 selected_cpu;
u32 kf_mask; /* see scx_kf_mask above */
struct task_struct *kf_tasks[2]; /* see SCX_CALL_OP_TASK() */
atomic_long_t ops_state;
struct list_head runnable_node; /* rq->scx.runnable_list */
unsigned long runnable_at;
@@ -181,8 +202,6 @@ struct sched_ext_entity {
#ifdef CONFIG_SCHED_CORE
u64 core_sched_at; /* see scx_prio_less() */
#endif
u64 ddsp_dsq_id;
u64 ddsp_enq_flags;
/* BPF scheduler modifiable fields */
include/linux/workqueue.h
+38 -9
View File
@@ -131,8 +131,9 @@ struct rcu_work {
enum wq_affn_scope {
WQ_AFFN_DFL, /* use system default */
WQ_AFFN_CPU, /* one pod per CPU */
WQ_AFFN_SMT, /* one pod poer SMT */
WQ_AFFN_SMT, /* one pod per SMT */
WQ_AFFN_CACHE, /* one pod per LLC */
WQ_AFFN_CACHE_SHARD, /* synthetic sub-LLC shards */
WQ_AFFN_NUMA, /* one pod per NUMA node */
WQ_AFFN_SYSTEM, /* one pod across the whole system */
@@ -440,6 +441,9 @@ enum wq_consts {
* system_long_wq is similar to system_percpu_wq but may host long running
* works. Queue flushing might take relatively long.
*
* system_dfl_long_wq is similar to system_dfl_wq but it may host long running
* works.
*
* system_dfl_wq is unbound workqueue. Workers are not bound to
* any specific CPU, not concurrency managed, and all queued works are
* executed immediately as long as max_active limit is not reached and
@@ -468,6 +472,7 @@ extern struct workqueue_struct *system_power_efficient_wq;
extern struct workqueue_struct *system_freezable_power_efficient_wq;
extern struct workqueue_struct *system_bh_wq;
extern struct workqueue_struct *system_bh_highpri_wq;
extern struct workqueue_struct *system_dfl_long_wq;
void workqueue_softirq_action(bool highpri);
void workqueue_softirq_dead(unsigned int cpu);
@@ -512,6 +517,26 @@ __printf(1, 4) struct workqueue_struct *
alloc_workqueue_noprof(const char *fmt, unsigned int flags, int max_active, ...);
#define alloc_workqueue(...) alloc_hooks(alloc_workqueue_noprof(__VA_ARGS__))
/**
* devm_alloc_workqueue - Resource-managed allocate a workqueue
* @dev: Device to allocate workqueue for
* @fmt: printf format for the name of the workqueue
* @flags: WQ_* flags
* @max_active: max in-flight work items, 0 for default
* @...: args for @fmt
*
* Resource managed workqueue, see alloc_workqueue() for details.
*
* The workqueue will be automatically destroyed on driver detach. Typically
* this should be used in drivers already relying on devm interafaces.
*
* RETURNS:
* Pointer to the allocated workqueue on success, %NULL on failure.
*/
__printf(2, 5) struct workqueue_struct *
devm_alloc_workqueue(struct device *dev, const char *fmt, unsigned int flags,
int max_active, ...);
#ifdef CONFIG_LOCKDEP
/**
* alloc_workqueue_lockdep_map - allocate a workqueue with user-defined lockdep_map
@@ -568,6 +593,8 @@ alloc_workqueue_lockdep_map(const char *fmt, unsigned int flags, int max_active,
*/
#define alloc_ordered_workqueue(fmt, flags, args...) \
alloc_workqueue(fmt, WQ_UNBOUND | __WQ_ORDERED | (flags), 1, ##args)
#define devm_alloc_ordered_workqueue(dev, fmt, flags, args...) \
devm_alloc_workqueue(dev, fmt, WQ_UNBOUND | __WQ_ORDERED | (flags), 1, ##args)
#define create_workqueue(name) \
alloc_workqueue("%s", __WQ_LEGACY | WQ_MEM_RECLAIM | WQ_PERCPU, 1, (name))
@@ -712,14 +739,14 @@ static inline bool schedule_work_on(int cpu, struct work_struct *work)
}
/**
* schedule_work - put work task in global workqueue
* schedule_work - put work task in per-CPU workqueue
* @work: job to be done
*
* Returns %false if @work was already on the kernel-global workqueue and
* Returns %false if @work was already on the system per-CPU workqueue and
* %true otherwise.
*
* This puts a job in the kernel-global workqueue if it was not already
* queued and leaves it in the same position on the kernel-global
* This puts a job in the system per-CPU workqueue if it was not already
* queued and leaves it in the same position on the system per-CPU
* workqueue otherwise.
*
* Shares the same memory-ordering properties of queue_work(), cf. the
@@ -783,6 +810,8 @@ extern void __warn_flushing_systemwide_wq(void)
_wq == system_highpri_wq) || \
(__builtin_constant_p(_wq == system_long_wq) && \
_wq == system_long_wq) || \
(__builtin_constant_p(_wq == system_dfl_long_wq) && \
_wq == system_dfl_long_wq) || \
(__builtin_constant_p(_wq == system_dfl_wq) && \
_wq == system_dfl_wq) || \
(__builtin_constant_p(_wq == system_freezable_wq) && \
@@ -796,12 +825,12 @@ extern void __warn_flushing_systemwide_wq(void)
})
/**
* schedule_delayed_work_on - queue work in global workqueue on CPU after delay
* schedule_delayed_work_on - queue work in per-CPU workqueue on CPU after delay
* @cpu: cpu to use
* @dwork: job to be done
* @delay: number of jiffies to wait
*
* After waiting for a given time this puts a job in the kernel-global
* After waiting for a given time this puts a job in the system per-CPU
* workqueue on the specified CPU.
*/
static inline bool schedule_delayed_work_on(int cpu, struct delayed_work *dwork,
@@ -811,11 +840,11 @@ static inline bool schedule_delayed_work_on(int cpu, struct delayed_work *dwork,
}
/**
* schedule_delayed_work - put work task in global workqueue after delay
* schedule_delayed_work - put work task in per-CPU workqueue after delay
* @dwork: job to be done
* @delay: number of jiffies to wait or 0 for immediate execution
*
* After waiting for a given time this puts a job in the kernel-global
* After waiting for a given time this puts a job in the system per-CPU
* workqueue.
*/
static inline bool schedule_delayed_work(struct delayed_work *dwork,
init/Kconfig
+4
View File
@@ -1190,6 +1190,10 @@ config EXT_GROUP_SCHED
endif #CGROUP_SCHED
config EXT_SUB_SCHED
def_bool y
depends on SCHED_CLASS_EXT && CGROUPS
config SCHED_MM_CID
def_bool y
depends on SMP && RSEQ
kernel/cgroup/cgroup-internal.h
-6
View File
@@ -184,11 +184,6 @@ extern bool cgrp_dfl_visible;
for ((ssid) = 0; (ssid) < CGROUP_SUBSYS_COUNT && \
(((ss) = cgroup_subsys[ssid]) || true); (ssid)++)
static inline bool cgroup_is_dead(const struct cgroup *cgrp)
{
return !(cgrp->self.flags & CSS_ONLINE);
}
static inline bool notify_on_release(const struct cgroup *cgrp)
{
return test_bit(CGRP_NOTIFY_ON_RELEASE, &cgrp->flags);
@@ -222,7 +217,6 @@ static inline void get_css_set(struct css_set *cset)
}
bool cgroup_ssid_enabled(int ssid);
bool cgroup_on_dfl(const struct cgroup *cgrp);
struct cgroup_root *cgroup_root_from_kf(struct kernfs_root *kf_root);
struct cgroup *task_cgroup_from_root(struct task_struct *task,
kernel/cgroup/cgroup.c
+29 -81
View File
@@ -71,14 +71,6 @@
/* let's not notify more than 100 times per second */
#define CGROUP_FILE_NOTIFY_MIN_INTV DIV_ROUND_UP(HZ, 100)
/*
* To avoid confusing the compiler (and generating warnings) with code
* that attempts to access what would be a 0-element array (i.e. sized
* to a potentially empty array when CGROUP_SUBSYS_COUNT == 0), this
* constant expression can be added.
*/
#define CGROUP_HAS_SUBSYS_CONFIG (CGROUP_SUBSYS_COUNT > 0)
/*
* cgroup_mutex is the master lock. Any modification to cgroup or its
* hierarchy must be performed while holding it.
@@ -110,12 +102,6 @@ static bool cgroup_debug __read_mostly;
*/
static DEFINE_SPINLOCK(cgroup_idr_lock);
/*
* Protects cgroup_file->kn for !self csses. It synchronizes notifications
* against file removal/re-creation across css hiding.
*/
static DEFINE_SPINLOCK(cgroup_file_kn_lock);
DEFINE_PERCPU_RWSEM(cgroup_threadgroup_rwsem);
#define cgroup_assert_mutex_or_rcu_locked() \
@@ -512,27 +498,6 @@ static u32 cgroup_ss_mask(struct cgroup *cgrp)
return cgrp->root->subsys_mask;
}
/**
* cgroup_css - obtain a cgroup's css for the specified subsystem
* @cgrp: the cgroup of interest
* @ss: the subsystem of interest (%NULL returns @cgrp->self)
*
* Return @cgrp's css (cgroup_subsys_state) associated with @ss. This
* function must be called either under cgroup_mutex or rcu_read_lock() and
* the caller is responsible for pinning the returned css if it wants to
* keep accessing it outside the said locks. This function may return
* %NULL if @cgrp doesn't have @subsys_id enabled.
*/
static struct cgroup_subsys_state *cgroup_css(struct cgroup *cgrp,
struct cgroup_subsys *ss)
{
if (CGROUP_HAS_SUBSYS_CONFIG && ss)
return rcu_dereference_check(cgrp->subsys[ss->id],
lockdep_is_held(&cgroup_mutex));
else
return &cgrp->self;
}
/**
* cgroup_e_css_by_mask - obtain a cgroup's effective css for the specified ss
* @cgrp: the cgroup of interest
@@ -744,32 +709,6 @@ EXPORT_SYMBOL_GPL(of_css);
} \
} while (false)
/* iterate over child cgrps, lock should be held throughout iteration */
#define cgroup_for_each_live_child(child, cgrp) \
list_for_each_entry((child), &(cgrp)->self.children, self.sibling) \
if (({ lockdep_assert_held(&cgroup_mutex); \
cgroup_is_dead(child); })) \
; \
else
/* walk live descendants in pre order */
#define cgroup_for_each_live_descendant_pre(dsct, d_css, cgrp) \
css_for_each_descendant_pre((d_css), cgroup_css((cgrp), NULL)) \
if (({ lockdep_assert_held(&cgroup_mutex); \
(dsct) = (d_css)->cgroup; \
cgroup_is_dead(dsct); })) \
; \
else
/* walk live descendants in postorder */
#define cgroup_for_each_live_descendant_post(dsct, d_css, cgrp) \
css_for_each_descendant_post((d_css), cgroup_css((cgrp), NULL)) \
if (({ lockdep_assert_held(&cgroup_mutex); \
(dsct) = (d_css)->cgroup; \
cgroup_is_dead(dsct); })) \
; \
else
/*
* The default css_set - used by init and its children prior to any
* hierarchies being mounted. It contains a pointer to the root state
@@ -1751,9 +1690,9 @@ static void cgroup_rm_file(struct cgroup *cgrp, const struct cftype *cft)
struct cgroup_subsys_state *css = cgroup_css(cgrp, cft->ss);
struct cgroup_file *cfile = (void *)css + cft->file_offset;
spin_lock_irq(&cgroup_file_kn_lock);
cfile->kn = NULL;
spin_unlock_irq(&cgroup_file_kn_lock);
spin_lock_irq(&cfile->lock);
WRITE_ONCE(cfile->kn, NULL);
spin_unlock_irq(&cfile->lock);
timer_delete_sync(&cfile->notify_timer);
}
@@ -4435,10 +4374,8 @@ static int cgroup_add_file(struct cgroup_subsys_state *css, struct cgroup *cgrp,
struct cgroup_file *cfile = (void *)css + cft->file_offset;
timer_setup(&cfile->notify_timer, cgroup_file_notify_timer, 0);
spin_lock_irq(&cgroup_file_kn_lock);
spin_lock_init(&cfile->lock);
cfile->kn = kn;
spin_unlock_irq(&cgroup_file_kn_lock);
}
return 0;
@@ -4693,21 +4630,32 @@ int cgroup_add_legacy_cftypes(struct cgroup_subsys *ss, struct cftype *cfts)
*/
void cgroup_file_notify(struct cgroup_file *cfile)
{
unsigned long flags;
unsigned long flags, last, next;
struct kernfs_node *kn = NULL;
spin_lock_irqsave(&cgroup_file_kn_lock, flags);
if (cfile->kn) {
unsigned long last = cfile->notified_at;
unsigned long next = last + CGROUP_FILE_NOTIFY_MIN_INTV;
if (!READ_ONCE(cfile->kn))
return;
if (time_in_range(jiffies, last, next)) {
timer_reduce(&cfile->notify_timer, next);
} else {
kernfs_notify(cfile->kn);
cfile->notified_at = jiffies;
}
last = READ_ONCE(cfile->notified_at);
next = last + CGROUP_FILE_NOTIFY_MIN_INTV;
if (time_in_range(jiffies, last, next)) {
timer_reduce(&cfile->notify_timer, next);
if (timer_pending(&cfile->notify_timer))
return;
}
spin_lock_irqsave(&cfile->lock, flags);
if (cfile->kn) {
kn = cfile->kn;
kernfs_get(kn);
WRITE_ONCE(cfile->notified_at, jiffies);
}
spin_unlock_irqrestore(&cfile->lock, flags);
if (kn) {
kernfs_notify(kn);
kernfs_put(kn);
}
spin_unlock_irqrestore(&cgroup_file_kn_lock, flags);
}
EXPORT_SYMBOL_GPL(cgroup_file_notify);
@@ -4720,10 +4668,10 @@ void cgroup_file_show(struct cgroup_file *cfile, bool show)
{
struct kernfs_node *kn;
spin_lock_irq(&cgroup_file_kn_lock);
spin_lock_irq(&cfile->lock);
kn = cfile->kn;
kernfs_get(kn);
spin_unlock_irq(&cgroup_file_kn_lock);
spin_unlock_irq(&cfile->lock);
if (kn)
kernfs_show(kn, show);
kernel/cgroup/dmem.c
+2 -3
View File
@@ -707,8 +707,7 @@ static int dmem_cgroup_region_capacity_show(struct seq_file *sf, void *v)
return 0;
}
static int dmemcg_parse_limit(char *options, struct dmem_cgroup_region *region,
u64 *new_limit)
static int dmemcg_parse_limit(char *options, u64 *new_limit)
{
char *end;
@@ -762,7 +761,7 @@ static ssize_t dmemcg_limit_write(struct kernfs_open_file *of,
if (!region)
return -EINVAL;
err = dmemcg_parse_limit(options, region, &new_limit);
err = dmemcg_parse_limit(options, &new_limit);
if (err < 0)
goto out_put;
kernel/cgroup/rdma.c
+1 -1
View File
@@ -173,7 +173,7 @@ uncharge_cg_locked(struct rdma_cgroup *cg,
* the system.
*/
if (unlikely(!rpool)) {
pr_warn("Invalid device %p or rdma cgroup %p\n", cg, device);
pr_warn("Invalid device %p or rdma cgroup %p\n", device, cg);
return;
}
kernel/fork.c
+5 -1
View File
@@ -2528,8 +2528,12 @@ __latent_entropy struct task_struct *copy_process(
fd_install(pidfd, pidfile);
proc_fork_connector(p);
sched_post_fork(p);
/*
* sched_ext needs @p to be associated with its cgroup in its post_fork
* hook. cgroup_post_fork() should come before sched_post_fork().
*/
cgroup_post_fork(p, args);
sched_post_fork(p);
perf_event_fork(p);
trace_task_newtask(p, clone_flags);
kernel/sched/core.c
+1 -1
View File
@@ -4866,7 +4866,7 @@ int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)
p->sched_class->task_fork(p);
raw_spin_unlock_irqrestore(&p->pi_lock, flags);
return scx_fork(p);
return scx_fork(p, kargs);
}
void sched_cancel_fork(struct task_struct *p)
kernel/sched/ext.c
+3062 -937
View File
File diff suppressed because it is too large. Load diff
kernel/sched/ext.h
+2 -2
View File
@@ -11,7 +11,7 @@
void scx_tick(struct rq *rq);
void init_scx_entity(struct sched_ext_entity *scx);
void scx_pre_fork(struct task_struct *p);
int scx_fork(struct task_struct *p);
int scx_fork(struct task_struct *p, struct kernel_clone_args *kargs);
void scx_post_fork(struct task_struct *p);
void scx_cancel_fork(struct task_struct *p);
bool scx_can_stop_tick(struct rq *rq);
@@ -44,7 +44,7 @@ bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,
static inline void scx_tick(struct rq *rq) {}
static inline void scx_pre_fork(struct task_struct *p) {}
static inline int scx_fork(struct task_struct *p) { return 0; }
static inline int scx_fork(struct task_struct *p, struct kernel_clone_args *kargs) { return 0; }
static inline void scx_post_fork(struct task_struct *p) {}
static inline void scx_cancel_fork(struct task_struct *p) {}
static inline u32 scx_cpuperf_target(s32 cpu) { return 0; }
kernel/sched/ext_idle.c
+132 -67
View File
@@ -368,7 +368,7 @@ void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops)
/*
* Enable NUMA optimization only when there are multiple NUMA domains
* among the online CPUs and the NUMA domains don't perfectly overlaps
* among the online CPUs and the NUMA domains don't perfectly overlap
* with the LLC domains.
*
* If all CPUs belong to the same NUMA node and the same LLC domain,
@@ -424,18 +424,24 @@ static inline bool task_affinity_all(const struct task_struct *p)
* - prefer the last used CPU to take advantage of cached data (L1, L2) and
* branch prediction optimizations.
*
* 3. Pick a CPU within the same LLC (Last-Level Cache):
* 3. Prefer @prev_cpu's SMT sibling:
* - if @prev_cpu is busy and no fully idle core is available, try to
* place the task on an idle SMT sibling of @prev_cpu; keeping the
* task on the same core makes migration cheaper, preserves L1 cache
* locality and reduces wakeup latency.
*
* 4. Pick a CPU within the same LLC (Last-Level Cache):
* - if the above conditions aren't met, pick a CPU that shares the same
* LLC, if the LLC domain is a subset of @cpus_allowed, to maintain
* cache locality.
*
* 4. Pick a CPU within the same NUMA node, if enabled:
* 5. Pick a CPU within the same NUMA node, if enabled:
* - choose a CPU from the same NUMA node, if the node cpumask is a
* subset of @cpus_allowed, to reduce memory access latency.
*
* 5. Pick any idle CPU within the @cpus_allowed domain.
* 6. Pick any idle CPU within the @cpus_allowed domain.
*
* Step 3 and 4 are performed only if the system has, respectively,
* Step 4 and 5 are performed only if the system has, respectively,
* multiple LLCs / multiple NUMA nodes (see scx_selcpu_topo_llc and
* scx_selcpu_topo_numa) and they don't contain the same subset of CPUs.
*
@@ -616,6 +622,20 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
goto out_unlock;
}
#ifdef CONFIG_SCHED_SMT
/*
* Use @prev_cpu's sibling if it's idle.
*/
if (sched_smt_active()) {
for_each_cpu_and(cpu, cpu_smt_mask(prev_cpu), allowed) {
if (cpu == prev_cpu)
continue;
if (scx_idle_test_and_clear_cpu(cpu))
goto out_unlock;
}
}
#endif
/*
* Search for any idle CPU in the same LLC domain.
*/
@@ -767,8 +787,9 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify)
* either enqueue() sees the idle bit or update_idle() sees the task
* that enqueue() queued.
*/
if (SCX_HAS_OP(sch, update_idle) && do_notify && !scx_rq_bypassing(rq))
SCX_CALL_OP(sch, SCX_KF_REST, update_idle, rq, cpu_of(rq), idle);
if (SCX_HAS_OP(sch, update_idle) && do_notify &&
!scx_bypassing(sch, cpu_of(rq)))
SCX_CALL_OP(sch, update_idle, rq, cpu_of(rq), idle);
}
static void reset_idle_masks(struct sched_ext_ops *ops)
@@ -892,8 +913,8 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,
s32 prev_cpu, u64 wake_flags,
const struct cpumask *allowed, u64 flags)
{
struct rq *rq;
struct rq_flags rf;
unsigned long irq_flags;
bool we_locked = false;
s32 cpu;
if (!ops_cpu_valid(sch, prev_cpu, NULL))
@@ -903,27 +924,20 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,
return -EBUSY;
/*
* If called from an unlocked context, acquire the task's rq lock,
* so that we can safely access p->cpus_ptr and p->nr_cpus_allowed.
* Accessing p->cpus_ptr / p->nr_cpus_allowed needs either @p's rq
* lock or @p's pi_lock. Three cases:
*
* Otherwise, allow to use this kfunc only from ops.select_cpu()
* and ops.select_enqueue().
* - inside ops.select_cpu(): try_to_wake_up() holds @p's pi_lock.
* - other rq-locked SCX op: scx_locked_rq() points at the held rq.
* - truly unlocked (UNLOCKED ops, SYSCALL, non-SCX struct_ops):
* nothing held, take pi_lock ourselves.
*/
if (scx_kf_allowed_if_unlocked()) {
rq = task_rq_lock(p, &rf);
} else {
if (!scx_kf_allowed(sch, SCX_KF_SELECT_CPU | SCX_KF_ENQUEUE))
return -EPERM;
rq = scx_locked_rq();
}
/*
* Validate locking correctness to access p->cpus_ptr and
* p->nr_cpus_allowed: if we're holding an rq lock, we're safe;
* otherwise, assert that p->pi_lock is held.
*/
if (!rq)
if (this_rq()->scx.in_select_cpu) {
lockdep_assert_held(&p->pi_lock);
} else if (!scx_locked_rq()) {
raw_spin_lock_irqsave(&p->pi_lock, irq_flags);
we_locked = true;
}
/*
* This may also be called from ops.enqueue(), so we need to handle
@@ -942,8 +956,8 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,
allowed ?: p->cpus_ptr, flags);
}
if (scx_kf_allowed_if_unlocked())
task_rq_unlock(rq, p, &rf);
if (we_locked)
raw_spin_unlock_irqrestore(&p->pi_lock, irq_flags);
return cpu;
}
@@ -952,14 +966,15 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,
* scx_bpf_cpu_node - Return the NUMA node the given @cpu belongs to, or
* trigger an error if @cpu is invalid
* @cpu: target CPU
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*/
__bpf_kfunc int scx_bpf_cpu_node(s32 cpu)
__bpf_kfunc s32 scx_bpf_cpu_node(s32 cpu, const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch) || !ops_cpu_valid(sch, cpu, NULL))
return NUMA_NO_NODE;
return cpu_to_node(cpu);
@@ -971,6 +986,7 @@ __bpf_kfunc int scx_bpf_cpu_node(s32 cpu)
* @prev_cpu: CPU @p was on previously
* @wake_flags: %SCX_WAKE_* flags
* @is_idle: out parameter indicating whether the returned CPU is idle
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Can be called from ops.select_cpu(), ops.enqueue(), or from an unlocked
* context such as a BPF test_run() call, as long as built-in CPU selection
@@ -981,14 +997,15 @@ __bpf_kfunc int scx_bpf_cpu_node(s32 cpu)
* currently idle and thus a good candidate for direct dispatching.
*/
__bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu,
u64 wake_flags, bool *is_idle)
u64 wake_flags, bool *is_idle,
const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
s32 cpu;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return -ENODEV;
@@ -1016,6 +1033,7 @@ struct scx_bpf_select_cpu_and_args {
* @args->prev_cpu: CPU @p was on previously
* @args->wake_flags: %SCX_WAKE_* flags
* @args->flags: %SCX_PICK_IDLE* flags
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Wrapper kfunc that takes arguments via struct to work around BPF's 5 argument
* limit. BPF programs should use scx_bpf_select_cpu_and() which is provided
@@ -1034,13 +1052,14 @@ struct scx_bpf_select_cpu_and_args {
*/
__bpf_kfunc s32
__scx_bpf_select_cpu_and(struct task_struct *p, const struct cpumask *cpus_allowed,
struct scx_bpf_select_cpu_and_args *args)
struct scx_bpf_select_cpu_and_args *args,
const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return -ENODEV;
@@ -1062,6 +1081,17 @@ __bpf_kfunc s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64
if (unlikely(!sch))
return -ENODEV;
#ifdef CONFIG_EXT_SUB_SCHED
/*
* Disallow if any sub-scheds are attached. There is no way to tell
* which scheduler called us, just error out @p's scheduler.
*/
if (unlikely(!list_empty(&sch->children))) {
scx_error(scx_task_sched(p), "__scx_bpf_select_cpu_and() must be used");
return -EINVAL;
}
#endif
return select_cpu_from_kfunc(sch, p, prev_cpu, wake_flags,
cpus_allowed, flags);
}
@@ -1070,18 +1100,20 @@ __bpf_kfunc s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64
* scx_bpf_get_idle_cpumask_node - Get a referenced kptr to the
* idle-tracking per-CPU cpumask of a target NUMA node.
* @node: target NUMA node
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Returns an empty cpumask if idle tracking is not enabled, if @node is
* not valid, or running on a UP kernel. In this case the actual error will
* be reported to the BPF scheduler via scx_error().
*/
__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask_node(int node)
__bpf_kfunc const struct cpumask *
scx_bpf_get_idle_cpumask_node(s32 node, const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return cpu_none_mask;
@@ -1095,17 +1127,18 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask_node(int node)
/**
* scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking
* per-CPU cpumask.
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Returns an empty mask if idle tracking is not enabled, or running on a
* UP kernel.
*/
__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void)
__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return cpu_none_mask;
@@ -1125,18 +1158,20 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void)
* idle-tracking, per-physical-core cpumask of a target NUMA node. Can be
* used to determine if an entire physical core is free.
* @node: target NUMA node
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Returns an empty cpumask if idle tracking is not enabled, if @node is
* not valid, or running on a UP kernel. In this case the actual error will
* be reported to the BPF scheduler via scx_error().
*/
__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask_node(int node)
__bpf_kfunc const struct cpumask *
scx_bpf_get_idle_smtmask_node(s32 node, const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return cpu_none_mask;
@@ -1154,17 +1189,18 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask_node(int node)
* scx_bpf_get_idle_smtmask - Get a referenced kptr to the idle-tracking,
* per-physical-core cpumask. Can be used to determine if an entire physical
* core is free.
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Returns an empty mask if idle tracking is not enabled, or running on a
* UP kernel.
*/
__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void)
__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return cpu_none_mask;
@@ -1200,6 +1236,7 @@ __bpf_kfunc void scx_bpf_put_idle_cpumask(const struct cpumask *idle_mask)
/**
* scx_bpf_test_and_clear_cpu_idle - Test and clear @cpu's idle state
* @cpu: cpu to test and clear idle for
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Returns %true if @cpu was idle and its idle state was successfully cleared.
* %false otherwise.
@@ -1207,13 +1244,13 @@ __bpf_kfunc void scx_bpf_put_idle_cpumask(const struct cpumask *idle_mask)
* Unavailable if ops.update_idle() is implemented and
* %SCX_OPS_KEEP_BUILTIN_IDLE is not set.
*/
__bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu)
__bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu, const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return false;
@@ -1231,6 +1268,7 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu)
* @cpus_allowed: Allowed cpumask
* @node: target NUMA node
* @flags: %SCX_PICK_IDLE_* flags
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Pick and claim an idle cpu in @cpus_allowed from the NUMA node @node.
*
@@ -1246,13 +1284,14 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu)
* %SCX_OPS_BUILTIN_IDLE_PER_NODE is not set.
*/
__bpf_kfunc s32 scx_bpf_pick_idle_cpu_node(const struct cpumask *cpus_allowed,
int node, u64 flags)
s32 node, u64 flags,
const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return -ENODEV;
@@ -1267,6 +1306,7 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu_node(const struct cpumask *cpus_allowed,
* scx_bpf_pick_idle_cpu - Pick and claim an idle cpu
* @cpus_allowed: Allowed cpumask
* @flags: %SCX_PICK_IDLE_CPU_* flags
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Pick and claim an idle cpu in @cpus_allowed. Returns the picked idle cpu
* number on success. -%EBUSY if no matching cpu was found.
@@ -1286,13 +1326,13 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu_node(const struct cpumask *cpus_allowed,
* scx_bpf_pick_idle_cpu_node() instead.
*/
__bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed,
u64 flags)
u64 flags, const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return -ENODEV;
@@ -1313,6 +1353,7 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed,
* @cpus_allowed: Allowed cpumask
* @node: target NUMA node
* @flags: %SCX_PICK_IDLE_CPU_* flags
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Pick and claim an idle cpu in @cpus_allowed. If none is available, pick any
* CPU in @cpus_allowed. Guaranteed to succeed and returns the picked idle cpu
@@ -1329,14 +1370,15 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed,
* CPU.
*/
__bpf_kfunc s32 scx_bpf_pick_any_cpu_node(const struct cpumask *cpus_allowed,
int node, u64 flags)
s32 node, u64 flags,
const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
s32 cpu;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return -ENODEV;
@@ -1362,6 +1404,7 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu_node(const struct cpumask *cpus_allowed,
* scx_bpf_pick_any_cpu - Pick and claim an idle cpu if available or pick any CPU
* @cpus_allowed: Allowed cpumask
* @flags: %SCX_PICK_IDLE_CPU_* flags
* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs
*
* Pick and claim an idle cpu in @cpus_allowed. If none is available, pick any
* CPU in @cpus_allowed. Guaranteed to succeed and returns the picked idle cpu
@@ -1376,14 +1419,14 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu_node(const struct cpumask *cpus_allowed,
* scx_bpf_pick_any_cpu_node() instead.
*/
__bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed,
u64 flags)
u64 flags, const struct bpf_prog_aux *aux)
{
struct scx_sched *sch;
s32 cpu;
guard(rcu)();
sch = rcu_dereference(scx_root);
sch = scx_prog_sched(aux);
if (unlikely(!sch))
return -ENODEV;
@@ -1408,20 +1451,17 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed,
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(scx_kfunc_ids_idle)
BTF_ID_FLAGS(func, scx_bpf_cpu_node)
BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask_node, KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask_node, KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_cpu_node, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask_node, KF_IMPLICIT_ARGS | KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_IMPLICIT_ARGS | KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask_node, KF_IMPLICIT_ARGS | KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_IMPLICIT_ARGS | KF_ACQUIRE)
BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE)
BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle)
BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu_node, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu_node, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU)
BTF_ID_FLAGS(func, __scx_bpf_select_cpu_and, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_select_cpu_and, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu_node, KF_IMPLICIT_ARGS | KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_IMPLICIT_ARGS | KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu_node, KF_IMPLICIT_ARGS | KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_IMPLICIT_ARGS | KF_RCU)
BTF_KFUNCS_END(scx_kfunc_ids_idle)
static const struct btf_kfunc_id_set scx_kfunc_set_idle = {
@@ -1429,13 +1469,38 @@ static const struct btf_kfunc_id_set scx_kfunc_set_idle = {
.set = &scx_kfunc_ids_idle,
};
/*
* The select_cpu kfuncs internally call task_rq_lock() when invoked from an
* rq-unlocked context, and thus cannot be safely called from arbitrary tracing
* contexts where @p's pi_lock state is unknown. Keep them out of
* BPF_PROG_TYPE_TRACING by registering them in their own set which is exposed
* only to STRUCT_OPS and SYSCALL programs.
*
* These kfuncs are also members of scx_kfunc_ids_unlocked (see ext.c) because
* they're callable from unlocked contexts in addition to ops.select_cpu() and
* ops.enqueue().
*/
BTF_KFUNCS_START(scx_kfunc_ids_select_cpu)
BTF_ID_FLAGS(func, __scx_bpf_select_cpu_and, KF_IMPLICIT_ARGS | KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_select_cpu_and, KF_RCU)
BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_IMPLICIT_ARGS | KF_RCU)
BTF_KFUNCS_END(scx_kfunc_ids_select_cpu)
static const struct btf_kfunc_id_set scx_kfunc_set_select_cpu = {
.owner = THIS_MODULE,
.set = &scx_kfunc_ids_select_cpu,
.filter = scx_kfunc_context_filter,
};
int scx_idle_init(void)
{
int ret;
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &scx_kfunc_set_idle) ||
register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &scx_kfunc_set_idle) ||
register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &scx_kfunc_set_idle);
register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &scx_kfunc_set_idle) ||
register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &scx_kfunc_set_select_cpu) ||
register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &scx_kfunc_set_select_cpu);
return ret;
}
kernel/sched/ext_idle.h
+2
View File
@@ -12,6 +12,8 @@
struct sched_ext_ops;
extern struct btf_id_set8 scx_kfunc_ids_select_cpu;
void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops);
void scx_idle_init_masks(void);
kernel/sched/ext_internal.h
+321 -23
View File
@@ -6,6 +6,7 @@
* Copyright (c) 2025 Tejun Heo <tj@kernel.org>
*/
#define SCX_OP_IDX(op) (offsetof(struct sched_ext_ops, op) / sizeof(void (*)(void)))
#define SCX_MOFF_IDX(moff) ((moff) / sizeof(void (*)(void)))
enum scx_consts {
SCX_DSP_DFL_MAX_BATCH = 32,
@@ -24,10 +25,16 @@ enum scx_consts {
*/
SCX_TASK_ITER_BATCH = 32,
SCX_BYPASS_HOST_NTH = 2,
SCX_BYPASS_LB_DFL_INTV_US = 500 * USEC_PER_MSEC,
SCX_BYPASS_LB_DONOR_PCT = 125,
SCX_BYPASS_LB_MIN_DELTA_DIV = 4,
SCX_BYPASS_LB_BATCH = 256,
SCX_REENQ_LOCAL_MAX_REPEAT = 256,
SCX_SUB_MAX_DEPTH = 4,
};
enum scx_exit_kind {
@@ -38,6 +45,7 @@ enum scx_exit_kind {
SCX_EXIT_UNREG_BPF, /* BPF-initiated unregistration */
SCX_EXIT_UNREG_KERN, /* kernel-initiated unregistration */
SCX_EXIT_SYSRQ, /* requested by 'S' sysrq */
SCX_EXIT_PARENT, /* parent exiting */
SCX_EXIT_ERROR = 1024, /* runtime error, error msg contains details */
SCX_EXIT_ERROR_BPF, /* ERROR but triggered through scx_bpf_error() */
@@ -62,6 +70,7 @@ enum scx_exit_kind {
enum scx_exit_code {
/* Reasons */
SCX_ECODE_RSN_HOTPLUG = 1LLU << 32,
SCX_ECODE_RSN_CGROUP_OFFLINE = 2LLU << 32,
/* Actions */
SCX_ECODE_ACT_RESTART = 1LLU << 48,
@@ -175,9 +184,10 @@ enum scx_ops_flags {
SCX_OPS_BUILTIN_IDLE_PER_NODE = 1LLU << 6,
/*
* CPU cgroup support flags
* If set, %SCX_ENQ_IMMED is assumed to be set on all local DSQ
* enqueues.
*/
SCX_OPS_HAS_CGROUP_WEIGHT = 1LLU << 16, /* DEPRECATED, will be removed on 6.18 */
SCX_OPS_ALWAYS_ENQ_IMMED = 1LLU << 7,
SCX_OPS_ALL_FLAGS = SCX_OPS_KEEP_BUILTIN_IDLE |
SCX_OPS_ENQ_LAST |
@@ -186,7 +196,7 @@ enum scx_ops_flags {
SCX_OPS_ALLOW_QUEUED_WAKEUP |
SCX_OPS_SWITCH_PARTIAL |
SCX_OPS_BUILTIN_IDLE_PER_NODE |
SCX_OPS_HAS_CGROUP_WEIGHT,
SCX_OPS_ALWAYS_ENQ_IMMED,
/* high 8 bits are internal, don't include in SCX_OPS_ALL_FLAGS */
__SCX_OPS_INTERNAL_MASK = 0xffLLU << 56,
@@ -213,7 +223,7 @@ struct scx_exit_task_args {
bool cancelled;
};
/* argument container for ops->cgroup_init() */
/* argument container for ops.cgroup_init() */
struct scx_cgroup_init_args {
/* the weight of the cgroup [1..10000] */
u32 weight;
@@ -236,12 +246,12 @@ enum scx_cpu_preempt_reason {
};
/*
* Argument container for ops->cpu_acquire(). Currently empty, but may be
* Argument container for ops.cpu_acquire(). Currently empty, but may be
* expanded in the future.
*/
struct scx_cpu_acquire_args {};
/* argument container for ops->cpu_release() */
/* argument container for ops.cpu_release() */
struct scx_cpu_release_args {
/* the reason the CPU was preempted */
enum scx_cpu_preempt_reason reason;
@@ -250,9 +260,7 @@ struct scx_cpu_release_args {
struct task_struct *task;
};
/*
* Informational context provided to dump operations.
*/
/* informational context provided to dump operations */
struct scx_dump_ctx {
enum scx_exit_kind kind;
s64 exit_code;
@@ -261,6 +269,18 @@ struct scx_dump_ctx {
u64 at_jiffies;
};
/* argument container for ops.sub_attach() */
struct scx_sub_attach_args {
struct sched_ext_ops *ops;
char *cgroup_path;
};
/* argument container for ops.sub_detach() */
struct scx_sub_detach_args {
struct sched_ext_ops *ops;
char *cgroup_path;
};
/**
* struct sched_ext_ops - Operation table for BPF scheduler implementation
*
@@ -721,6 +741,20 @@ struct sched_ext_ops {
#endif /* CONFIG_EXT_GROUP_SCHED */
/**
* @sub_attach: Attach a sub-scheduler
* @args: argument container, see the struct definition
*
* Return 0 to accept the sub-scheduler. -errno to reject.
*/
s32 (*sub_attach)(struct scx_sub_attach_args *args);
/**
* @sub_detach: Detach a sub-scheduler
* @args: argument container, see the struct definition
*/
void (*sub_detach)(struct scx_sub_detach_args *args);
/*
* All online ops must come before ops.cpu_online().
*/
@@ -762,6 +796,10 @@ struct sched_ext_ops {
*/
void (*exit)(struct scx_exit_info *info);
/*
* Data fields must comes after all ops fields.
*/
/**
* @dispatch_max_batch: Max nr of tasks that dispatch() can dispatch
*/
@@ -796,6 +834,12 @@ struct sched_ext_ops {
*/
u64 hotplug_seq;
/**
* @cgroup_id: When >1, attach the scheduler as a sub-scheduler on the
* specified cgroup.
*/
u64 sub_cgroup_id;
/**
* @name: BPF scheduler's name
*
@@ -806,7 +850,7 @@ struct sched_ext_ops {
char name[SCX_OPS_NAME_LEN];
/* internal use only, must be NULL */
void *priv;
void __rcu *priv;
};
enum scx_opi {
@@ -853,6 +897,24 @@ struct scx_event_stats {
*/
s64 SCX_EV_ENQ_SKIP_MIGRATION_DISABLED;
/*
* The number of times a task, enqueued on a local DSQ with
* SCX_ENQ_IMMED, was re-enqueued because the CPU was not available for
* immediate execution.
*/
s64 SCX_EV_REENQ_IMMED;
/*
* The number of times a reenq of local DSQ caused another reenq of
* local DSQ. This can happen when %SCX_ENQ_IMMED races against a higher
* priority class task even if the BPF scheduler always satisfies the
* prerequisites for %SCX_ENQ_IMMED at the time of enqueue. However,
* that scenario is very unlikely and this count going up regularly
* indicates that the BPF scheduler is handling %SCX_ENQ_REENQ
* incorrectly causing recursive reenqueues.
*/
s64 SCX_EV_REENQ_LOCAL_REPEAT;
/*
* Total number of times a task's time slice was refilled with the
* default value (SCX_SLICE_DFL).
@@ -873,15 +935,77 @@ struct scx_event_stats {
* The number of times the bypassing mode has been activated.
*/
s64 SCX_EV_BYPASS_ACTIVATE;
/*
* The number of times the scheduler attempted to insert a task that it
* doesn't own into a DSQ. Such attempts are ignored.
*
* As BPF schedulers are allowed to ignore dequeues, it's difficult to
* tell whether such an attempt is from a scheduler malfunction or an
* ignored dequeue around sub-sched enabling. If this count keeps going
* up regardless of sub-sched enabling, it likely indicates a bug in the
* scheduler.
*/
s64 SCX_EV_INSERT_NOT_OWNED;
/*
* The number of times tasks from bypassing descendants are scheduled
* from sub_bypass_dsq's.
*/
s64 SCX_EV_SUB_BYPASS_DISPATCH;
};
struct scx_sched;
enum scx_sched_pcpu_flags {
SCX_SCHED_PCPU_BYPASSING = 1LLU << 0,
};
/* dispatch buf */
struct scx_dsp_buf_ent {
struct task_struct *task;
unsigned long qseq;
u64 dsq_id;
u64 enq_flags;
};
struct scx_dsp_ctx {
struct rq *rq;
u32 cursor;
u32 nr_tasks;
struct scx_dsp_buf_ent buf[];
};
struct scx_deferred_reenq_local {
struct list_head node;
u64 flags;
u64 seq;
u32 cnt;
};
struct scx_sched_pcpu {
struct scx_sched *sch;
u64 flags; /* protected by rq lock */
/*
* The event counters are in a per-CPU variable to minimize the
* accounting overhead. A system-wide view on the event counter is
* constructed when requested by scx_bpf_events().
*/
struct scx_event_stats event_stats;
struct scx_deferred_reenq_local deferred_reenq_local;
struct scx_dispatch_q bypass_dsq;
#ifdef CONFIG_EXT_SUB_SCHED
u32 bypass_host_seq;
#endif
/* must be the last entry - contains flex array */
struct scx_dsp_ctx dsp_ctx;
};
struct scx_sched_pnode {
struct scx_dispatch_q global_dsq;
};
struct scx_sched {
@@ -897,15 +1021,50 @@ struct scx_sched {
* per-node split isn't sufficient, it can be further split.
*/
struct rhashtable dsq_hash;
struct scx_dispatch_q **global_dsqs;
struct scx_sched_pnode **pnode;
struct scx_sched_pcpu __percpu *pcpu;
u64 slice_dfl;
u64 bypass_timestamp;
s32 bypass_depth;
/* bypass dispatch path enable state, see bypass_dsp_enabled() */
unsigned long bypass_dsp_claim;
atomic_t bypass_dsp_enable_depth;
bool aborting;
bool dump_disabled; /* protected by scx_dump_lock */
u32 dsp_max_batch;
s32 level;
/*
* Updates to the following warned bitfields can race causing RMW issues
* but it doesn't really matter.
*/
bool warned_zero_slice:1;
bool warned_deprecated_rq:1;
bool warned_unassoc_progs:1;
struct list_head all;
#ifdef CONFIG_EXT_SUB_SCHED
struct rhash_head hash_node;
struct list_head children;
struct list_head sibling;
struct cgroup *cgrp;
char *cgrp_path;
struct kset *sub_kset;
bool sub_attached;
#endif /* CONFIG_EXT_SUB_SCHED */
/*
* The maximum amount of time in jiffies that a task may be runnable
* without being scheduled on a CPU. If this timeout is exceeded, it
* will trigger scx_error().
*/
unsigned long watchdog_timeout;
atomic_t exit_kind;
struct scx_exit_info *exit_info;
@@ -913,9 +1072,13 @@ struct scx_sched {
struct kobject kobj;
struct kthread_worker *helper;
struct irq_work error_irq_work;
struct irq_work disable_irq_work;
struct kthread_work disable_work;
struct timer_list bypass_lb_timer;
struct rcu_work rcu_work;
/* all ancestors including self */
struct scx_sched *ancestors[];
};
enum scx_wake_flags {
@@ -942,13 +1105,27 @@ enum scx_enq_flags {
SCX_ENQ_PREEMPT = 1LLU << 32,
/*
* The task being enqueued was previously enqueued on the current CPU's
* %SCX_DSQ_LOCAL, but was removed from it in a call to the
* scx_bpf_reenqueue_local() kfunc. If scx_bpf_reenqueue_local() was
* invoked in a ->cpu_release() callback, and the task is again
* dispatched back to %SCX_LOCAL_DSQ by this current ->enqueue(), the
* task will not be scheduled on the CPU until at least the next invocation
* of the ->cpu_acquire() callback.
* Only allowed on local DSQs. Guarantees that the task either gets
* on the CPU immediately and stays on it, or gets reenqueued back
* to the BPF scheduler. It will never linger on a local DSQ or be
* silently put back after preemption.
*
* The protection persists until the next fresh enqueue - it
* survives SAVE/RESTORE cycles, slice extensions and preemption.
* If the task can't stay on the CPU for any reason, it gets
* reenqueued back to the BPF scheduler.
*
* Exiting and migration-disabled tasks bypass ops.enqueue() and
* are placed directly on a local DSQ without IMMED protection
* unless %SCX_OPS_ENQ_EXITING and %SCX_OPS_ENQ_MIGRATION_DISABLED
* are set respectively.
*/
SCX_ENQ_IMMED = 1LLU << 33,
/*
* The task being enqueued was previously enqueued on a DSQ, but was
* removed and is being re-enqueued. See SCX_TASK_REENQ_* flags to find
* out why a given task is being reenqueued.
*/
SCX_ENQ_REENQ = 1LLU << 40,
@@ -969,6 +1146,7 @@ enum scx_enq_flags {
SCX_ENQ_CLEAR_OPSS = 1LLU << 56,
SCX_ENQ_DSQ_PRIQ = 1LLU << 57,
SCX_ENQ_NESTED = 1LLU << 58,
SCX_ENQ_GDSQ_FALLBACK = 1LLU << 59, /* fell back to global DSQ */
};
enum scx_deq_flags {
@@ -982,6 +1160,28 @@ enum scx_deq_flags {
* it hasn't been dispatched yet. Dequeue from the BPF side.
*/
SCX_DEQ_CORE_SCHED_EXEC = 1LLU << 32,
/*
* The task is being dequeued due to a property change (e.g.,
* sched_setaffinity(), sched_setscheduler(), set_user_nice(),
* etc.).
*/
SCX_DEQ_SCHED_CHANGE = 1LLU << 33,
};
enum scx_reenq_flags {
/* low 16bits determine which tasks should be reenqueued */
SCX_REENQ_ANY = 1LLU << 0, /* all tasks */
__SCX_REENQ_FILTER_MASK = 0xffffLLU,
__SCX_REENQ_USER_MASK = SCX_REENQ_ANY,
/* bits 32-35 used by task_should_reenq() */
SCX_REENQ_TSR_RQ_OPEN = 1LLU << 32,
SCX_REENQ_TSR_NOT_FIRST = 1LLU << 33,
__SCX_REENQ_TSR_MASK = 0xfLLU << 32,
};
enum scx_pick_idle_cpu_flags {
@@ -1161,8 +1361,11 @@ enum scx_ops_state {
#define SCX_OPSS_STATE_MASK ((1LU << SCX_OPSS_QSEQ_SHIFT) - 1)
#define SCX_OPSS_QSEQ_MASK (~SCX_OPSS_STATE_MASK)
extern struct scx_sched __rcu *scx_root;
DECLARE_PER_CPU(struct rq *, scx_locked_rq_state);
int scx_kfunc_context_filter(const struct bpf_prog *prog, u32 kfunc_id);
/*
* Return the rq currently locked from an scx callback, or NULL if no rq is
* locked.
@@ -1172,12 +1375,107 @@ static inline struct rq *scx_locked_rq(void)
return __this_cpu_read(scx_locked_rq_state);
}
static inline bool scx_kf_allowed_if_unlocked(void)
static inline bool scx_bypassing(struct scx_sched *sch, s32 cpu)
{
return !current->scx.kf_mask;
return unlikely(per_cpu_ptr(sch->pcpu, cpu)->flags &
SCX_SCHED_PCPU_BYPASSING);
}
static inline bool scx_rq_bypassing(struct rq *rq)
#ifdef CONFIG_EXT_SUB_SCHED
/**
* scx_task_sched - Find scx_sched scheduling a task
* @p: task of interest
*
* Return @p's scheduler instance. Must be called with @p's pi_lock or rq lock
* held.
*/
static inline struct scx_sched *scx_task_sched(const struct task_struct *p)
{
return unlikely(rq->scx.flags & SCX_RQ_BYPASSING);
return rcu_dereference_protected(p->scx.sched,
lockdep_is_held(&p->pi_lock) ||
lockdep_is_held(__rq_lockp(task_rq(p))));
}
/**
* scx_task_sched_rcu - Find scx_sched scheduling a task
* @p: task of interest
*
* Return @p's scheduler instance. The returned scx_sched is RCU protected.
*/
static inline struct scx_sched *scx_task_sched_rcu(const struct task_struct *p)
{
return rcu_dereference_all(p->scx.sched);
}
/**
* scx_task_on_sched - Is a task on the specified sched?
* @sch: sched to test against
* @p: task of interest
*
* Returns %true if @p is on @sch, %false otherwise.
*/
static inline bool scx_task_on_sched(struct scx_sched *sch,
const struct task_struct *p)
{
return rcu_access_pointer(p->scx.sched) == sch;
}
/**
* scx_prog_sched - Find scx_sched associated with a BPF prog
* @aux: aux passed in from BPF to a kfunc
*
* To be called from kfuncs. Return the scheduler instance associated with the
* BPF program given the implicit kfunc argument aux. The returned scx_sched is
* RCU protected.
*/
static inline struct scx_sched *scx_prog_sched(const struct bpf_prog_aux *aux)
{
struct sched_ext_ops *ops;
struct scx_sched *root;
ops = bpf_prog_get_assoc_struct_ops(aux);
if (likely(ops))
return rcu_dereference_all(ops->priv);
root = rcu_dereference_all(scx_root);
if (root) {
/*
* COMPAT-v6.19: Schedulers built before sub-sched support was
* introduced may have unassociated non-struct_ops programs.
*/
if (!root->ops.sub_attach)
return root;
if (!root->warned_unassoc_progs) {
printk_deferred(KERN_WARNING "sched_ext: Unassociated program %s (id %d)\n",
aux->name, aux->id);
root->warned_unassoc_progs = true;
}
}
return NULL;
}
#else /* CONFIG_EXT_SUB_SCHED */
static inline struct scx_sched *scx_task_sched(const struct task_struct *p)
{
return rcu_dereference_protected(scx_root,
lockdep_is_held(&p->pi_lock) ||
lockdep_is_held(__rq_lockp(task_rq(p))));
}
static inline struct scx_sched *scx_task_sched_rcu(const struct task_struct *p)
{
return rcu_dereference_all(scx_root);
}
static inline bool scx_task_on_sched(struct scx_sched *sch,
const struct task_struct *p)
{
return true;
}
static struct scx_sched *scx_prog_sched(const struct bpf_prog_aux *aux)
{
return rcu_dereference_all(scx_root);
}
#endif /* CONFIG_EXT_SUB_SCHED */
kernel/sched/sched.h
+9 -3
View File
@@ -791,7 +791,6 @@ enum scx_rq_flags {
SCX_RQ_ONLINE = 1 << 0,
SCX_RQ_CAN_STOP_TICK = 1 << 1,
SCX_RQ_BAL_KEEP = 1 << 3, /* balance decided to keep current */
SCX_RQ_BYPASSING = 1 << 4,
SCX_RQ_CLK_VALID = 1 << 5, /* RQ clock is fresh and valid */
SCX_RQ_BAL_CB_PENDING = 1 << 6, /* must queue a cb after dispatching */
@@ -807,8 +806,10 @@ struct scx_rq {
u64 extra_enq_flags; /* see move_task_to_local_dsq() */
u32 nr_running;
u32 cpuperf_target; /* [0, SCHED_CAPACITY_SCALE] */
bool in_select_cpu;
bool cpu_released;
u32 flags;
u32 nr_immed; /* ENQ_IMMED tasks on local_dsq */
u64 clock; /* current per-rq clock -- see scx_bpf_now() */
cpumask_var_t cpus_to_kick;
cpumask_var_t cpus_to_kick_if_idle;
@@ -817,12 +818,17 @@ struct scx_rq {
cpumask_var_t cpus_to_sync;
bool kick_sync_pending;
unsigned long kick_sync;
local_t reenq_local_deferred;
struct task_struct *sub_dispatch_prev;
raw_spinlock_t deferred_reenq_lock;
u64 deferred_reenq_locals_seq;
struct list_head deferred_reenq_locals; /* scheds requesting reenq of local DSQ */
struct list_head deferred_reenq_users; /* user DSQs requesting reenq */
struct balance_callback deferred_bal_cb;
struct balance_callback kick_sync_bal_cb;
struct irq_work deferred_irq_work;
struct irq_work kick_cpus_irq_work;
struct scx_dispatch_q bypass_dsq;
};
#endif /* CONFIG_SCHED_CLASS_EXT */
kernel/workqueue.c
+261 -24
View File
@@ -41,6 +41,7 @@
#include <linux/mempolicy.h>
#include <linux/freezer.h>
#include <linux/debug_locks.h>
#include <linux/device/devres.h>
#include <linux/lockdep.h>
#include <linux/idr.h>
#include <linux/jhash.h>
@@ -130,6 +131,14 @@ enum wq_internal_consts {
WORKER_ID_LEN = 10 + WQ_NAME_LEN, /* "kworker/R-" + WQ_NAME_LEN */
};
/* Layout of shards within one LLC pod */
struct llc_shard_layout {
int nr_large_shards; /* number of large shards (cores_per_shard + 1) */
int cores_per_shard; /* base number of cores per default shard */
int nr_shards; /* total number of shards */
/* nr_default shards = (nr_shards - nr_large_shards) */
};
/*
* We don't want to trap softirq for too long. See MAX_SOFTIRQ_TIME and
* MAX_SOFTIRQ_RESTART in kernel/softirq.c. These are macros because
@@ -404,11 +413,12 @@ struct work_offq_data {
u32 flags;
};
static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = {
static const char * const wq_affn_names[WQ_AFFN_NR_TYPES] = {
[WQ_AFFN_DFL] = "default",
[WQ_AFFN_CPU] = "cpu",
[WQ_AFFN_SMT] = "smt",
[WQ_AFFN_CACHE] = "cache",
[WQ_AFFN_CACHE_SHARD] = "cache_shard",
[WQ_AFFN_NUMA] = "numa",
[WQ_AFFN_SYSTEM] = "system",
};
@@ -431,13 +441,16 @@ module_param_named(cpu_intensive_warning_thresh, wq_cpu_intensive_warning_thresh
static bool wq_power_efficient = IS_ENABLED(CONFIG_WQ_POWER_EFFICIENT_DEFAULT);
module_param_named(power_efficient, wq_power_efficient, bool, 0444);
static unsigned int wq_cache_shard_size = 8;
module_param_named(cache_shard_size, wq_cache_shard_size, uint, 0444);
static bool wq_online; /* can kworkers be created yet? */
static bool wq_topo_initialized __read_mostly = false;
static struct kmem_cache *pwq_cache;
static struct wq_pod_type wq_pod_types[WQ_AFFN_NR_TYPES];
static enum wq_affn_scope wq_affn_dfl = WQ_AFFN_CACHE;
static enum wq_affn_scope wq_affn_dfl = WQ_AFFN_CACHE_SHARD;
/* buf for wq_update_unbound_pod_attrs(), protected by CPU hotplug exclusion */
static struct workqueue_attrs *unbound_wq_update_pwq_attrs_buf;
@@ -530,6 +543,8 @@ struct workqueue_struct *system_bh_wq;
EXPORT_SYMBOL_GPL(system_bh_wq);
struct workqueue_struct *system_bh_highpri_wq;
EXPORT_SYMBOL_GPL(system_bh_highpri_wq);
struct workqueue_struct *system_dfl_long_wq __ro_after_init;
EXPORT_SYMBOL_GPL(system_dfl_long_wq);
static int worker_thread(void *__worker);
static void workqueue_sysfs_unregister(struct workqueue_struct *wq);
@@ -2519,7 +2534,6 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq,
struct timer_list *timer = &dwork->timer;
struct work_struct *work = &dwork->work;
WARN_ON_ONCE(!wq);
WARN_ON_ONCE(timer->function != delayed_work_timer_fn);
WARN_ON_ONCE(timer_pending(timer));
WARN_ON_ONCE(!list_empty(&work->entry));
@@ -5635,8 +5649,16 @@ enomem:
for_each_possible_cpu(cpu) {
struct pool_workqueue *pwq = *per_cpu_ptr(wq->cpu_pwq, cpu);
if (pwq)
if (pwq) {
/*
* Unlink pwq from wq->pwqs since link_pwq()
* may have already added it. wq->mutex is not
* needed as the wq has not been published yet.
*/
if (!list_empty(&pwq->pwqs_node))
list_del_rcu(&pwq->pwqs_node);
kmem_cache_free(pwq_cache, pwq);
}
}
free_percpu(wq->cpu_pwq);
wq->cpu_pwq = NULL;
@@ -5904,6 +5926,33 @@ struct workqueue_struct *alloc_workqueue_noprof(const char *fmt,
}
EXPORT_SYMBOL_GPL(alloc_workqueue_noprof);
static void devm_workqueue_release(void *res)
{
destroy_workqueue(res);
}
__printf(2, 5) struct workqueue_struct *
devm_alloc_workqueue(struct device *dev, const char *fmt, unsigned int flags,
int max_active, ...)
{
struct workqueue_struct *wq;
va_list args;
int ret;
va_start(args, max_active);
wq = alloc_workqueue(fmt, flags, max_active, args);
va_end(args);
if (!wq)
return NULL;
ret = devm_add_action_or_reset(dev, devm_workqueue_release, wq);
if (ret)
return NULL;
return wq;
}
EXPORT_SYMBOL_GPL(devm_alloc_workqueue);
#ifdef CONFIG_LOCKDEP
__printf(1, 5)
struct workqueue_struct *
@@ -7059,7 +7108,7 @@ int workqueue_unbound_housekeeping_update(const struct cpumask *hk)
/*
* If the operation fails, it will fall back to
* wq_requested_unbound_cpumask which is initially set to
* (HK_TYPE_WQ ∩ HK_TYPE_DOMAIN) house keeping mask and rewritten
* HK_TYPE_DOMAIN house keeping mask and rewritten
* by any subsequent write to workqueue/cpumask sysfs file.
*/
if (!cpumask_and(cpumask, wq_requested_unbound_cpumask, hk))
@@ -7078,13 +7127,7 @@ int workqueue_unbound_housekeeping_update(const struct cpumask *hk)
static int parse_affn_scope(const char *val)
{
int i;
for (i = 0; i < ARRAY_SIZE(wq_affn_names); i++) {
if (!strncasecmp(val, wq_affn_names[i], strlen(wq_affn_names[i])))
return i;
}
return -EINVAL;
return sysfs_match_string(wq_affn_names, val);
}
static int wq_affn_dfl_set(const char *val, const struct kernel_param *kp)
@@ -7191,7 +7234,26 @@ static struct attribute *wq_sysfs_attrs[] = {
&dev_attr_max_active.attr,
NULL,
};
ATTRIBUTE_GROUPS(wq_sysfs);
static umode_t wq_sysfs_is_visible(struct kobject *kobj, struct attribute *a, int n)
{
struct device *dev = kobj_to_dev(kobj);
struct workqueue_struct *wq = dev_to_wq(dev);
/*
* Adjusting max_active breaks ordering guarantee. Changing it has no
* effect on BH worker. Limit max_active to RO in such case.
*/
if (wq->flags & (WQ_BH | __WQ_ORDERED))
return 0444;
return a->mode;
}
static const struct attribute_group wq_sysfs_group = {
.is_visible = wq_sysfs_is_visible,
.attrs = wq_sysfs_attrs,
};
__ATTRIBUTE_GROUPS(wq_sysfs);
static ssize_t wq_nice_show(struct device *dev, struct device_attribute *attr,
char *buf)
@@ -7494,13 +7556,6 @@ int workqueue_sysfs_register(struct workqueue_struct *wq)
struct wq_device *wq_dev;
int ret;
/*
* Adjusting max_active breaks ordering guarantee. Disallow exposing
* ordered workqueues.
*/
if (WARN_ON(wq->flags & __WQ_ORDERED))
return -EINVAL;
wq->wq_dev = wq_dev = kzalloc_obj(*wq_dev);
if (!wq_dev)
return -ENOMEM;
@@ -7877,8 +7932,8 @@ void __init workqueue_init_early(void)
{
struct wq_pod_type *pt = &wq_pod_types[WQ_AFFN_SYSTEM];
int std_nice[NR_STD_WORKER_POOLS] = { 0, HIGHPRI_NICE_LEVEL };
void (*irq_work_fns[2])(struct irq_work *) = { bh_pool_kick_normal,
bh_pool_kick_highpri };
void (*irq_work_fns[NR_STD_WORKER_POOLS])(struct irq_work *) =
{ bh_pool_kick_normal, bh_pool_kick_highpri };
int i, cpu;
BUILD_BUG_ON(__alignof__(struct pool_workqueue) < __alignof__(long long));
@@ -7890,7 +7945,6 @@ void __init workqueue_init_early(void)
cpumask_copy(wq_online_cpumask, cpu_online_mask);
cpumask_copy(wq_unbound_cpumask, cpu_possible_mask);
restrict_unbound_cpumask("HK_TYPE_WQ", housekeeping_cpumask(HK_TYPE_WQ));
restrict_unbound_cpumask("HK_TYPE_DOMAIN", housekeeping_cpumask(HK_TYPE_DOMAIN));
if (!cpumask_empty(&wq_cmdline_cpumask))
restrict_unbound_cpumask("workqueue.unbound_cpus", &wq_cmdline_cpumask);
@@ -7974,11 +8028,12 @@ void __init workqueue_init_early(void)
system_bh_wq = alloc_workqueue("events_bh", WQ_BH | WQ_PERCPU, 0);
system_bh_highpri_wq = alloc_workqueue("events_bh_highpri",
WQ_BH | WQ_HIGHPRI | WQ_PERCPU, 0);
system_dfl_long_wq = alloc_workqueue("events_dfl_long", WQ_UNBOUND, WQ_MAX_ACTIVE);
BUG_ON(!system_wq || !system_percpu_wq|| !system_highpri_wq || !system_long_wq ||
!system_unbound_wq || !system_freezable_wq || !system_dfl_wq ||
!system_power_efficient_wq ||
!system_freezable_power_efficient_wq ||
!system_bh_wq || !system_bh_highpri_wq);
!system_bh_wq || !system_bh_highpri_wq || !system_dfl_long_wq);
}
static void __init wq_cpu_intensive_thresh_init(void)
@@ -8144,6 +8199,186 @@ static bool __init cpus_share_numa(int cpu0, int cpu1)
return cpu_to_node(cpu0) == cpu_to_node(cpu1);
}
/* Maps each CPU to its shard index within the LLC pod it belongs to */
static int cpu_shard_id[NR_CPUS] __initdata;
/**
* llc_count_cores - count distinct cores (SMT groups) within an LLC pod
* @pod_cpus: the cpumask of CPUs in the LLC pod
* @smt_pods: the SMT pod type, used to identify sibling groups
*
* A core is represented by the lowest-numbered CPU in its SMT group. Returns
* the number of distinct cores found in @pod_cpus.
*/
static int __init llc_count_cores(const struct cpumask *pod_cpus,
struct wq_pod_type *smt_pods)
{
const struct cpumask *sibling_cpus;
int nr_cores = 0, c;
/*
* Count distinct cores by only counting the first CPU in each
* SMT sibling group.
*/
for_each_cpu(c, pod_cpus) {
sibling_cpus = smt_pods->pod_cpus[smt_pods->cpu_pod[c]];
if (cpumask_first(sibling_cpus) == c)
nr_cores++;
}
return nr_cores;
}
/*
* llc_shard_size - number of cores in a given shard
*
* Cores are spread as evenly as possible. The first @nr_large_shards shards are
* "large shards" with (cores_per_shard + 1) cores; the rest are "default
* shards" with cores_per_shard cores.
*/
static int __init llc_shard_size(int shard_id, int cores_per_shard, int nr_large_shards)
{
/* The first @nr_large_shards shards are large shards */
if (shard_id < nr_large_shards)
return cores_per_shard + 1;
/* The remaining shards are default shards */
return cores_per_shard;
}
/*
* llc_calc_shard_layout - compute the shard layout for an LLC pod
* @nr_cores: number of distinct cores in the LLC pod
*
* Chooses the number of shards that keeps average shard size closest to
* wq_cache_shard_size. Returns a struct describing the total number of shards,
* the base size of each, and how many are large shards.
*/
static struct llc_shard_layout __init llc_calc_shard_layout(int nr_cores)
{
struct llc_shard_layout layout;
/* Ensure at least one shard; pick the count closest to the target size */
layout.nr_shards = max(1, DIV_ROUND_CLOSEST(nr_cores, wq_cache_shard_size));
layout.cores_per_shard = nr_cores / layout.nr_shards;
layout.nr_large_shards = nr_cores % layout.nr_shards;
return layout;
}
/*
* llc_shard_is_full - check whether a shard has reached its core capacity
* @cores_in_shard: number of cores already assigned to this shard
* @shard_id: index of the shard being checked
* @layout: the shard layout computed by llc_calc_shard_layout()
*
* Returns true if @cores_in_shard equals the expected size for @shard_id.
*/
static bool __init llc_shard_is_full(int cores_in_shard, int shard_id,
const struct llc_shard_layout *layout)
{
return cores_in_shard == llc_shard_size(shard_id, layout->cores_per_shard,
layout->nr_large_shards);
}
/**
* llc_populate_cpu_shard_id - populate cpu_shard_id[] for each CPU in an LLC pod
* @pod_cpus: the cpumask of CPUs in the LLC pod
* @smt_pods: the SMT pod type, used to identify sibling groups
* @nr_cores: number of distinct cores in @pod_cpus (from llc_count_cores())
*
* Walks @pod_cpus in order. At each SMT group leader, advances to the next
* shard once the current shard is full. Results are written to cpu_shard_id[].
*/
static void __init llc_populate_cpu_shard_id(const struct cpumask *pod_cpus,
struct wq_pod_type *smt_pods,
int nr_cores)
{
struct llc_shard_layout layout = llc_calc_shard_layout(nr_cores);
const struct cpumask *sibling_cpus;
/* Count the number of cores in the current shard_id */
int cores_in_shard = 0;
unsigned int leader;
/* This is a cursor for the shards. Go from zero to nr_shards - 1*/
int shard_id = 0;
int c;
/* Iterate at every CPU for a given LLC pod, and assign it a shard */
for_each_cpu(c, pod_cpus) {
sibling_cpus = smt_pods->pod_cpus[smt_pods->cpu_pod[c]];
if (cpumask_first(sibling_cpus) == c) {
/* This is the CPU leader for the siblings */
if (llc_shard_is_full(cores_in_shard, shard_id, &layout)) {
shard_id++;
cores_in_shard = 0;
}
cores_in_shard++;
cpu_shard_id[c] = shard_id;
} else {
/*
* The siblings' shard MUST be the same as the leader.
* never split threads in the same core.
*/
leader = cpumask_first(sibling_cpus);
/*
* This check silences a Warray-bounds warning on UP
* configs where NR_CPUS=1 makes cpu_shard_id[]
* a single-element array, and the compiler can't
* prove the index is always 0.
*/
if (WARN_ON_ONCE(leader >= nr_cpu_ids))
continue;
cpu_shard_id[c] = cpu_shard_id[leader];
}
}
WARN_ON_ONCE(shard_id != (layout.nr_shards - 1));
}
/**
* precompute_cache_shard_ids - assign each CPU its shard index within its LLC
*
* Iterates over all LLC pods. For each pod, counts distinct cores then assigns
* shard indices to all CPUs in the pod. Must be called after WQ_AFFN_CACHE and
* WQ_AFFN_SMT have been initialized.
*/
static void __init precompute_cache_shard_ids(void)
{
struct wq_pod_type *llc_pods = &wq_pod_types[WQ_AFFN_CACHE];
struct wq_pod_type *smt_pods = &wq_pod_types[WQ_AFFN_SMT];
const struct cpumask *cpus_sharing_llc;
int nr_cores;
int pod;
if (!wq_cache_shard_size) {
pr_warn("workqueue: cache_shard_size must be > 0, setting to 1\n");
wq_cache_shard_size = 1;
}
for (pod = 0; pod < llc_pods->nr_pods; pod++) {
cpus_sharing_llc = llc_pods->pod_cpus[pod];
/* Number of cores in this given LLC */
nr_cores = llc_count_cores(cpus_sharing_llc, smt_pods);
llc_populate_cpu_shard_id(cpus_sharing_llc, smt_pods, nr_cores);
}
}
/*
* cpus_share_cache_shard - test whether two CPUs belong to the same cache shard
*
* Two CPUs share a cache shard if they are in the same LLC and have the same
* shard index. Used as the pod affinity callback for WQ_AFFN_CACHE_SHARD.
*/
static bool __init cpus_share_cache_shard(int cpu0, int cpu1)
{
if (!cpus_share_cache(cpu0, cpu1))
return false;
return cpu_shard_id[cpu0] == cpu_shard_id[cpu1];
}
/**
* workqueue_init_topology - initialize CPU pods for unbound workqueues
*
@@ -8159,6 +8394,8 @@ void __init workqueue_init_topology(void)
init_pod_type(&wq_pod_types[WQ_AFFN_CPU], cpus_dont_share);
init_pod_type(&wq_pod_types[WQ_AFFN_SMT], cpus_share_smt);
init_pod_type(&wq_pod_types[WQ_AFFN_CACHE], cpus_share_cache);
precompute_cache_shard_ids();
init_pod_type(&wq_pod_types[WQ_AFFN_CACHE_SHARD], cpus_share_cache_shard);
init_pod_type(&wq_pod_types[WQ_AFFN_NUMA], cpus_share_numa);
wq_topo_initialized = true;
lib/Kconfig.debug
+10
View File
@@ -2642,6 +2642,16 @@ config TEST_VMALLOC
If unsure, say N.
config TEST_WORKQUEUE
tristate "Test module for stress/performance analysis of workqueue"
default n
help
This builds the "test_workqueue" module for benchmarking
workqueue throughput under contention. Useful for evaluating
affinity scope changes (e.g., cache_shard vs cache).
If unsure, say N.
config TEST_BPF
tristate "Test BPF filter functionality"
depends on m && NET
lib/Makefile
+1
View File
@@ -79,6 +79,7 @@ UBSAN_SANITIZE_test_ubsan.o := y
obj-$(CONFIG_TEST_KSTRTOX) += test-kstrtox.o
obj-$(CONFIG_TEST_LKM) += test_module.o
obj-$(CONFIG_TEST_VMALLOC) += test_vmalloc.o
obj-$(CONFIG_TEST_WORKQUEUE) += test_workqueue.o
obj-$(CONFIG_TEST_RHASHTABLE) += test_rhashtable.o
obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_keys.o
obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_key_base.o
lib/test_workqueue.c
+294
View File
@@ -0,0 +1,294 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Test module for stress and performance analysis of workqueue.
*
* Benchmarks queue_work() throughput on an unbound workqueue to measure
* pool->lock contention under different affinity scope configurations
* (e.g., cache vs cache_shard).
*
* The affinity scope is changed between runs via the workqueue's sysfs
* affinity_scope attribute (WQ_SYSFS).
*
* Copyright (c) 2026 Meta Platforms, Inc. and affiliates
* Copyright (c) 2026 Breno Leitao <leitao@debian.org>
*
*/
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/workqueue.h>
#include <linux/kthread.h>
#include <linux/moduleparam.h>
#include <linux/completion.h>
#include <linux/atomic.h>
#include <linux/slab.h>
#include <linux/ktime.h>
#include <linux/cpumask.h>
#include <linux/sched.h>
#include <linux/sort.h>
#include <linux/fs.h>
#define WQ_NAME "bench_wq"
#define SCOPE_PATH "/sys/bus/workqueue/devices/" WQ_NAME "/affinity_scope"
static int nr_threads;
module_param(nr_threads, int, 0444);
MODULE_PARM_DESC(nr_threads,
"Number of threads to spawn (default: 0 = num_online_cpus())");
static int wq_items = 50000;
module_param(wq_items, int, 0444);
MODULE_PARM_DESC(wq_items,
"Number of work items each thread queues (default: 50000)");
static struct workqueue_struct *bench_wq;
static atomic_t threads_done;
static DECLARE_COMPLETION(start_comp);
static DECLARE_COMPLETION(all_done_comp);
struct thread_ctx {
struct completion work_done;
struct work_struct work;
u64 *latencies;
int cpu;
int items;
};
static void bench_work_fn(struct work_struct *work)
{
struct thread_ctx *ctx = container_of(work, struct thread_ctx, work);
complete(&ctx->work_done);
}
static int bench_kthread_fn(void *data)
{
struct thread_ctx *ctx = data;
ktime_t t_start, t_end;
int i;
/* Wait for all threads to be ready */
wait_for_completion(&start_comp);
if (kthread_should_stop())
return 0;
for (i = 0; i < ctx->items; i++) {
reinit_completion(&ctx->work_done);
INIT_WORK(&ctx->work, bench_work_fn);
t_start = ktime_get();
queue_work(bench_wq, &ctx->work);
t_end = ktime_get();
ctx->latencies[i] = ktime_to_ns(ktime_sub(t_end, t_start));
wait_for_completion(&ctx->work_done);
}
if (atomic_dec_and_test(&threads_done))
complete(&all_done_comp);
/*
* Wait for kthread_stop() so the module text isn't freed
* while we're still executing.
*/
while (!kthread_should_stop())
schedule();
return 0;
}
static int cmp_u64(const void *a, const void *b)
{
u64 va = *(const u64 *)a;
u64 vb = *(const u64 *)b;
if (va < vb)
return -1;
if (va > vb)
return 1;
return 0;
}
static int __init set_affn_scope(const char *scope)
{
struct file *f;
loff_t pos = 0;
ssize_t ret;
f = filp_open(SCOPE_PATH, O_WRONLY, 0);
if (IS_ERR(f)) {
pr_err("test_workqueue: open %s failed: %ld\n",
SCOPE_PATH, PTR_ERR(f));
return PTR_ERR(f);
}
ret = kernel_write(f, scope, strlen(scope), &pos);
filp_close(f, NULL);
if (ret < 0) {
pr_err("test_workqueue: write '%s' failed: %zd\n", scope, ret);
return ret;
}
return 0;
}
static int __init run_bench(int n_threads, const char *scope, const char *label)
{
struct task_struct **tasks;
unsigned long total_items;
struct thread_ctx *ctxs;
u64 *all_latencies;
ktime_t start, end;
int cpu, i, j, ret;
s64 elapsed_us;
ret = set_affn_scope(scope);
if (ret)
return ret;
ctxs = kcalloc(n_threads, sizeof(*ctxs), GFP_KERNEL);
if (!ctxs)
return -ENOMEM;
tasks = kcalloc(n_threads, sizeof(*tasks), GFP_KERNEL);
if (!tasks) {
kfree(ctxs);
return -ENOMEM;
}
total_items = (unsigned long)n_threads * wq_items;
all_latencies = kvmalloc_array(total_items, sizeof(u64), GFP_KERNEL);
if (!all_latencies) {
kfree(tasks);
kfree(ctxs);
return -ENOMEM;
}
/* Allocate per-thread latency arrays */
for (i = 0; i < n_threads; i++) {
ctxs[i].latencies = kvmalloc_array(wq_items, sizeof(u64),
GFP_KERNEL);
if (!ctxs[i].latencies) {
while (--i >= 0)
kvfree(ctxs[i].latencies);
kvfree(all_latencies);
kfree(tasks);
kfree(ctxs);
return -ENOMEM;
}
}
atomic_set(&threads_done, n_threads);
reinit_completion(&all_done_comp);
reinit_completion(&start_comp);
/* Create kthreads, each bound to a different online CPU */
i = 0;
for_each_online_cpu(cpu) {
if (i >= n_threads)
break;
ctxs[i].cpu = cpu;
ctxs[i].items = wq_items;
init_completion(&ctxs[i].work_done);
tasks[i] = kthread_create(bench_kthread_fn, &ctxs[i],
"wq_bench/%d", cpu);
if (IS_ERR(tasks[i])) {
ret = PTR_ERR(tasks[i]);
pr_err("test_workqueue: failed to create kthread %d: %d\n",
i, ret);
/* Unblock threads waiting on start_comp before stopping them */
complete_all(&start_comp);
while (--i >= 0)
kthread_stop(tasks[i]);
goto out_free;
}
kthread_bind(tasks[i], cpu);
wake_up_process(tasks[i]);
i++;
}
/* Start timing and release all threads */
start = ktime_get();
complete_all(&start_comp);
/* Wait for all threads to finish the benchmark */
wait_for_completion(&all_done_comp);
/* Drain any remaining work */
flush_workqueue(bench_wq);
/* Ensure all kthreads have fully exited before module memory is freed */
for (i = 0; i < n_threads; i++)
kthread_stop(tasks[i]);
end = ktime_get();
elapsed_us = ktime_us_delta(end, start);
/* Merge all per-thread latencies and sort for percentile calculation */
j = 0;
for (i = 0; i < n_threads; i++) {
memcpy(&all_latencies[j], ctxs[i].latencies,
wq_items * sizeof(u64));
j += wq_items;
}
sort(all_latencies, total_items, sizeof(u64), cmp_u64, NULL);
pr_info("test_workqueue: %-16s %llu items/sec\tp50=%llu\tp90=%llu\tp95=%llu ns\n",
label,
elapsed_us ? div_u64(total_items * 1000000ULL, elapsed_us) : 0,
all_latencies[total_items * 50 / 100],
all_latencies[total_items * 90 / 100],
all_latencies[total_items * 95 / 100]);
ret = 0;
out_free:
for (i = 0; i < n_threads; i++)
kvfree(ctxs[i].latencies);
kvfree(all_latencies);
kfree(tasks);
kfree(ctxs);
return ret;
}
static const char * const bench_scopes[] = {
"cpu", "smt", "cache_shard", "cache", "numa", "system",
};
static int __init test_workqueue_init(void)
{
int n_threads = min(nr_threads ?: num_online_cpus(), num_online_cpus());
int i;
if (wq_items <= 0) {
pr_err("test_workqueue: wq_items must be > 0\n");
return -EINVAL;
}
bench_wq = alloc_workqueue(WQ_NAME, WQ_UNBOUND | WQ_SYSFS, 0);
if (!bench_wq)
return -ENOMEM;
pr_info("test_workqueue: running %d threads, %d items/thread\n",
n_threads, wq_items);
for (i = 0; i < ARRAY_SIZE(bench_scopes); i++)
run_bench(n_threads, bench_scopes[i], bench_scopes[i]);
destroy_workqueue(bench_wq);
/* Return -EAGAIN so the module doesn't stay loaded after the benchmark */
return -EAGAIN;
}
module_init(test_workqueue_init);
MODULE_AUTHOR("Breno Leitao <leitao@debian.org>");
MODULE_DESCRIPTION("Stress/performance benchmark for workqueue subsystem");
MODULE_LICENSE("GPL");
lib/tests/slub_kunit.c
+92
View File
@@ -7,6 +7,7 @@
#include <linux/kernel.h>
#include <linux/rcupdate.h>
#include <linux/delay.h>
#include <linux/perf_event.h>
#include "../mm/slab.h"
static struct kunit_resource resource;
@@ -291,6 +292,94 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)
kmem_cache_destroy(s);
}
#ifdef CONFIG_PERF_EVENTS
#define NR_ITERATIONS 1000
#define NR_OBJECTS 1000
static void *objects[NR_OBJECTS];
struct test_nolock_context {
struct kunit *test;
int callback_count;
int alloc_ok;
int alloc_fail;
struct perf_event *event;
};
static struct perf_event_attr hw_attr = {
.type = PERF_TYPE_HARDWARE,
.config = PERF_COUNT_HW_CPU_CYCLES,
.size = sizeof(struct perf_event_attr),
.pinned = 1,
.disabled = 1,
.freq = 1,
.sample_freq = 100000,
};
static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
struct perf_sample_data *data,
struct pt_regs *regs)
{
void *objp;
gfp_t gfp;
struct test_nolock_context *ctx = event->overflow_handler_context;
/* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */
gfp = (ctx->callback_count % 2) ? 0 : __GFP_ACCOUNT;
objp = kmalloc_nolock(64, gfp, NUMA_NO_NODE);
if (objp)
ctx->alloc_ok++;
else
ctx->alloc_fail++;
kfree_nolock(objp);
ctx->callback_count++;
}
static void test_kmalloc_kfree_nolock(struct kunit *test)
{
int i, j;
struct test_nolock_context ctx = { .test = test };
struct perf_event *event;
bool alloc_fail = false;
event = perf_event_create_kernel_counter(&hw_attr, -1, current,
overflow_handler_test_kmalloc_kfree_nolock,
&ctx);
if (IS_ERR(event))
kunit_skip(test, "Failed to create perf event");
ctx.event = event;
perf_event_enable(ctx.event);
for (i = 0; i < NR_ITERATIONS; i++) {
for (j = 0; j < NR_OBJECTS; j++) {
gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
objects[j] = kmalloc(64, gfp);
if (!objects[j]) {
j--;
while (j >= 0)
kfree(objects[j--]);
alloc_fail = true;
goto cleanup;
}
}
for (j = 0; j < NR_OBJECTS; j++)
kfree(objects[j]);
}
cleanup:
perf_event_disable(ctx.event);
perf_event_release_kernel(ctx.event);
kunit_info(test, "callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail);
if (alloc_fail)
kunit_skip(test, "Allocation failed");
KUNIT_EXPECT_EQ(test, 0, slab_errors);
}
#endif
static int test_init(struct kunit *test)
{
slab_errors = 0;
@@ -315,6 +404,9 @@ static struct kunit_case test_cases[] = {
KUNIT_CASE(test_kfree_rcu_wq_destroy),
KUNIT_CASE(test_leak_destroy),
KUNIT_CASE(test_krealloc_redzone_zeroing),
#ifdef CONFIG_PERF_EVENTS
KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock),
#endif
{}
};
mm/Kconfig
+1
View File
@@ -172,6 +172,7 @@ config SLUB
config KVFREE_RCU_BATCHED
def_bool y
depends on !SLUB_TINY && !TINY_RCU
depends on !RCU_STRICT_GRACE_PERIOD
config SLUB_TINY
bool "Configure for minimal memory footprint"
mm/slab.h
+6 -1
View File
@@ -192,6 +192,11 @@ struct kmem_cache_order_objects {
unsigned int x;
};
struct kmem_cache_per_node_ptrs {
struct node_barn *barn;
struct kmem_cache_node *node;
};
/*
* Slab cache management.
*/
@@ -248,7 +253,7 @@ struct kmem_cache {
struct kmem_cache_stats __percpu *cpu_stats;
#endif
struct kmem_cache_node *node[MAX_NUMNODES];
struct kmem_cache_per_node_ptrs per_node[MAX_NUMNODES];
};
/*
mm/slub.c
+223 -130
View File
@@ -58,7 +58,7 @@
* 0. cpu_hotplug_lock
* 1. slab_mutex (Global Mutex)
* 2a. kmem_cache->cpu_sheaves->lock (Local trylock)
* 2b. node->barn->lock (Spinlock)
* 2b. barn->lock (Spinlock)
* 2c. node->list_lock (Spinlock)
* 3. slab_lock(slab) (Only on some arches)
* 4. object_map_lock (Only for debugging)
@@ -135,7 +135,7 @@
* or spare sheaf can handle the allocation or free, there is no other
* overhead.
*
* node->barn->lock (spinlock)
* barn->lock (spinlock)
*
* This lock protects the operations on per-NUMA-node barn. It can quickly
* serve an empty or full sheaf if available, and avoid more expensive refill
@@ -419,26 +419,24 @@ struct kmem_cache_node {
atomic_long_t total_objects;
struct list_head full;
#endif
struct node_barn *barn;
};
static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
{
return s->node[node];
return s->per_node[node].node;
}
static inline struct node_barn *get_barn_node(struct kmem_cache *s, int node)
{
return s->per_node[node].barn;
}
/*
* Get the barn of the current cpu's closest memory node. It may not exist on
* systems with memoryless nodes but without CONFIG_HAVE_MEMORYLESS_NODES
* Get the barn of the current cpu's NUMA node. It may be a memoryless node.
*/
static inline struct node_barn *get_barn(struct kmem_cache *s)
{
struct kmem_cache_node *n = get_node(s, numa_mem_id());
if (!n)
return NULL;
return n->barn;
return get_barn_node(s, numa_node_id());
}
/*
@@ -457,6 +455,12 @@ static inline struct node_barn *get_barn(struct kmem_cache *s)
*/
static nodemask_t slab_nodes;
/*
* Similar to slab_nodes but for where we have node_barn allocated.
* Corresponds to N_ONLINE nodes.
*/
static nodemask_t slab_barn_nodes;
/*
* Workqueue used for flushing cpu and kfree_rcu sheaves.
*/
@@ -2851,24 +2855,6 @@ static int refill_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
return 0;
}
static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf);
static struct slab_sheaf *alloc_full_sheaf(struct kmem_cache *s, gfp_t gfp)
{
struct slab_sheaf *sheaf = alloc_empty_sheaf(s, gfp);
if (!sheaf)
return NULL;
if (refill_sheaf(s, sheaf, gfp | __GFP_NOMEMALLOC | __GFP_NOWARN)) {
sheaf_flush_unused(s, sheaf);
free_empty_sheaf(s, sheaf);
return NULL;
}
return sheaf;
}
/*
* Maximum number of objects freed during a single flush of main pcs sheaf.
* Translates directly to an on-stack array size.
@@ -4111,6 +4097,51 @@ void flush_all_rcu_sheaves(void)
rcu_barrier();
}
static int slub_cpu_setup(unsigned int cpu)
{
int nid = cpu_to_node(cpu);
struct kmem_cache *s;
int ret = 0;
/*
* we never clear a nid so it's safe to do a quick check before taking
* the mutex, and then recheck to handle parallel cpu hotplug safely
*/
if (node_isset(nid, slab_barn_nodes))
return 0;
mutex_lock(&slab_mutex);
if (node_isset(nid, slab_barn_nodes))
goto out;
list_for_each_entry(s, &slab_caches, list) {
struct node_barn *barn;
/*
* barn might already exist if a previous callback failed midway
*/
if (!cache_has_sheaves(s) || get_barn_node(s, nid))
continue;
barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, nid);
if (!barn) {
ret = -ENOMEM;
goto out;
}
barn_init(barn);
s->per_node[nid].barn = barn;
}
node_set(nid, slab_barn_nodes);
out:
mutex_unlock(&slab_mutex);
return ret;
}
/*
* Use the cpu notifier to insure that the cpu slabs are flushed when
* necessary.
@@ -4640,34 +4671,35 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
if (!allow_spin)
return NULL;
if (empty) {
if (!refill_sheaf(s, empty, gfp | __GFP_NOMEMALLOC | __GFP_NOWARN)) {
full = empty;
} else {
/*
* we must be very low on memory so don't bother
* with the barn
*/
sheaf_flush_unused(s, empty);
free_empty_sheaf(s, empty);
}
} else {
full = alloc_full_sheaf(s, gfp);
if (!empty) {
empty = alloc_empty_sheaf(s, gfp);
if (!empty)
return NULL;
}
if (!full)
if (refill_sheaf(s, empty, gfp | __GFP_NOMEMALLOC | __GFP_NOWARN)) {
/*
* we must be very low on memory so don't bother
* with the barn
*/
sheaf_flush_unused(s, empty);
free_empty_sheaf(s, empty);
return NULL;
}
full = empty;
empty = NULL;
if (!local_trylock(&s->cpu_sheaves->lock))
goto barn_put;
pcs = this_cpu_ptr(s->cpu_sheaves);
/*
* If we are returning empty sheaf, we either got it from the
* barn or had to allocate one. If we are returning a full
* sheaf, it's due to racing or being migrated to a different
* cpu. Breaching the barn's sheaf limits should be thus rare
* enough so just ignore them to simplify the recovery.
* If we put any empty or full sheaf to the barn below, it's due to
* racing or being migrated to a different cpu. Breaching the barn's
* sheaf limits should be thus rare enough so just ignore them to
* simplify the recovery.
*/
if (pcs->main->size == 0) {
@@ -5117,12 +5149,15 @@ void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
}
/*
* refill a sheaf previously returned by kmem_cache_prefill_sheaf to at least
* the given size
* Refill a sheaf previously returned by kmem_cache_prefill_sheaf to at least
* the given size.
*
* the sheaf might be replaced by a new one when requesting more than
* s->sheaf_capacity objects if such replacement is necessary, but the refill
* fails (returning -ENOMEM), the existing sheaf is left intact
* Return: 0 on success. The sheaf will contain at least @size objects.
* The sheaf might have been replaced with a new one if more than
* sheaf->capacity objects are requested.
*
* Return: -ENOMEM on failure. Some objects might have been added to the sheaf
* but the sheaf will not be replaced.
*
* In practice we always refill to full sheaf's capacity.
*/
@@ -5817,7 +5852,6 @@ bool free_to_pcs(struct kmem_cache *s, void *object, bool allow_spin)
static void rcu_free_sheaf(struct rcu_head *head)
{
struct kmem_cache_node *n;
struct slab_sheaf *sheaf;
struct node_barn *barn = NULL;
struct kmem_cache *s;
@@ -5840,12 +5874,10 @@ static void rcu_free_sheaf(struct rcu_head *head)
if (__rcu_free_sheaf_prepare(s, sheaf))
goto flush;
n = get_node(s, sheaf->node);
if (!n)
barn = get_barn_node(s, sheaf->node);
if (!barn)
goto flush;
barn = n->barn;
/* due to slab_free_hook() */
if (unlikely(sheaf->size == 0))
goto empty;
@@ -5967,7 +5999,7 @@ do_free:
rcu_sheaf = NULL;
} else {
pcs->rcu_free = NULL;
rcu_sheaf->node = numa_mem_id();
rcu_sheaf->node = numa_node_id();
}
/*
@@ -5989,6 +6021,57 @@ fail:
return false;
}
static __always_inline bool can_free_to_pcs(struct slab *slab)
{
int slab_node;
int numa_node;
if (!IS_ENABLED(CONFIG_NUMA))
goto check_pfmemalloc;
slab_node = slab_nid(slab);
#ifdef CONFIG_HAVE_MEMORYLESS_NODES
/*
* numa_mem_id() points to the closest node with memory so only allow
* objects from that node to the percpu sheaves
*/
numa_node = numa_mem_id();
if (likely(slab_node == numa_node))
goto check_pfmemalloc;
#else
/*
* numa_mem_id() is only a wrapper to numa_node_id() which is where this
* cpu belongs to, but it might be a memoryless node anyway. We don't
* know what the closest node is.
*/
numa_node = numa_node_id();
/* freed object is from this cpu's node, proceed */
if (likely(slab_node == numa_node))
goto check_pfmemalloc;
/*
* Freed object isn't from this cpu's node, but that node is memoryless
* or only has ZONE_MOVABLE memory, which slab cannot allocate from.
* Proceed as it's better to cache remote objects than falling back to
* the slowpath for everything. The allocation side can never obtain
* a local object anyway, if none exist. We don't have numa_mem_id() to
* point to the closest node as we would on a proper memoryless node
* setup.
*/
if (unlikely(!node_state(numa_node, N_NORMAL_MEMORY)))
goto check_pfmemalloc;
#endif
return false;
check_pfmemalloc:
return likely(!slab_test_pfmemalloc(slab));
}
/*
* Bulk free objects to the percpu sheaves.
* Unlike free_to_pcs() this includes the calls to all necessary hooks
@@ -6003,7 +6086,6 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
struct node_barn *barn;
void *remote_objects[PCS_BATCH_MAX];
unsigned int remote_nr = 0;
int node = numa_mem_id();
next_remote_batch:
while (i < size) {
@@ -6017,8 +6099,7 @@ next_remote_batch:
continue;
}
if (unlikely((IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != node)
|| slab_test_pfmemalloc(slab))) {
if (unlikely(!can_free_to_pcs(slab))) {
remote_objects[remote_nr] = p[i];
p[i] = p[--size];
if (++remote_nr >= PCS_BATCH_MAX)
@@ -6194,11 +6275,8 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,
if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false)))
return;
if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id())
&& likely(!slab_test_pfmemalloc(slab))) {
if (likely(free_to_pcs(s, object, true)))
return;
}
if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, object, true)))
return;
__slab_free(s, slab, object, object, 1, addr);
stat(s, FREE_SLOWPATH);
@@ -6569,10 +6647,8 @@ void kfree_nolock(const void *object)
*/
kasan_slab_free(s, x, false, false, /* skip quarantine */true);
if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id())) {
if (likely(free_to_pcs(s, x, false)))
return;
}
if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, x, false)))
return;
/*
* __slab_free() can locklessly cmpxchg16 into a slab, but then it might
@@ -7456,7 +7532,7 @@ static inline int calculate_order(unsigned int size)
}
static void
init_kmem_cache_node(struct kmem_cache_node *n, struct node_barn *barn)
init_kmem_cache_node(struct kmem_cache_node *n)
{
n->nr_partial = 0;
spin_lock_init(&n->list_lock);
@@ -7466,9 +7542,6 @@ init_kmem_cache_node(struct kmem_cache_node *n, struct node_barn *barn)
atomic_long_set(&n->total_objects, 0);
INIT_LIST_HEAD(&n->full);
#endif
n->barn = barn;
if (barn)
barn_init(barn);
}
#ifdef CONFIG_SLUB_STATS
@@ -7563,8 +7636,8 @@ static void early_kmem_cache_node_alloc(int node)
n = kasan_slab_alloc(kmem_cache_node, n, GFP_KERNEL, false);
slab->freelist = get_freepointer(kmem_cache_node, n);
slab->inuse = 1;
kmem_cache_node->node[node] = n;
init_kmem_cache_node(n, NULL);
kmem_cache_node->per_node[node].node = n;
init_kmem_cache_node(n);
inc_slabs_node(kmem_cache_node, node, slab->objects);
/*
@@ -7579,15 +7652,20 @@ static void free_kmem_cache_nodes(struct kmem_cache *s)
int node;
struct kmem_cache_node *n;
for_each_kmem_cache_node(s, node, n) {
if (n->barn) {
WARN_ON(n->barn->nr_full);
WARN_ON(n->barn->nr_empty);
kfree(n->barn);
n->barn = NULL;
}
for_each_node(node) {
struct node_barn *barn = get_barn_node(s, node);
s->node[node] = NULL;
if (!barn)
continue;
WARN_ON(barn->nr_full);
WARN_ON(barn->nr_empty);
kfree(barn);
s->per_node[node].barn = NULL;
}
for_each_kmem_cache_node(s, node, n) {
s->per_node[node].node = NULL;
kmem_cache_free(kmem_cache_node, n);
}
}
@@ -7608,31 +7686,36 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)
for_each_node_mask(node, slab_nodes) {
struct kmem_cache_node *n;
struct node_barn *barn = NULL;
if (slab_state == DOWN) {
early_kmem_cache_node_alloc(node);
continue;
}
if (cache_has_sheaves(s)) {
barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, node);
if (!barn)
return 0;
}
n = kmem_cache_alloc_node(kmem_cache_node,
GFP_KERNEL, node);
if (!n) {
kfree(barn);
if (!n)
return 0;
}
init_kmem_cache_node(n, barn);
s->node[node] = n;
init_kmem_cache_node(n);
s->per_node[node].node = n;
}
if (slab_state == DOWN || !cache_has_sheaves(s))
return 1;
for_each_node_mask(node, slab_barn_nodes) {
struct node_barn *barn;
barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, node);
if (!barn)
return 0;
barn_init(barn);
s->per_node[node].barn = barn;
}
return 1;
}
@@ -7921,10 +8004,15 @@ int __kmem_cache_shutdown(struct kmem_cache *s)
if (cache_has_sheaves(s))
rcu_barrier();
for_each_node(node) {
struct node_barn *barn = get_barn_node(s, node);
if (barn)
barn_shrink(s, barn);
}
/* Attempt to free all objects */
for_each_kmem_cache_node(s, node, n) {
if (n->barn)
barn_shrink(s, n->barn);
free_partial(s, n);
if (n->nr_partial || node_nr_slabs(n))
return 1;
@@ -8134,14 +8222,18 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s)
unsigned long flags;
int ret = 0;
for_each_node(node) {
struct node_barn *barn = get_barn_node(s, node);
if (barn)
barn_shrink(s, barn);
}
for_each_kmem_cache_node(s, node, n) {
INIT_LIST_HEAD(&discard);
for (i = 0; i < SHRINK_PROMOTE_MAX; i++)
INIT_LIST_HEAD(promote + i);
if (n->barn)
barn_shrink(s, n->barn);
spin_lock_irqsave(&n->list_lock, flags);
/*
@@ -8230,7 +8322,8 @@ static int slab_mem_going_online_callback(int nid)
if (get_node(s, nid))
continue;
if (cache_has_sheaves(s)) {
if (cache_has_sheaves(s) && !get_barn_node(s, nid)) {
barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, nid);
if (!barn) {
@@ -8251,15 +8344,20 @@ static int slab_mem_going_online_callback(int nid)
goto out;
}
init_kmem_cache_node(n, barn);
init_kmem_cache_node(n);
s->per_node[nid].node = n;
s->node[nid] = n;
if (barn) {
barn_init(barn);
s->per_node[nid].barn = barn;
}
}
/*
* Any cache created after this point will also have kmem_cache_node
* initialized for the new node.
* and barn initialized for the new node.
*/
node_set(nid, slab_nodes);
node_set(nid, slab_barn_nodes);
out:
mutex_unlock(&slab_mutex);
return ret;
@@ -8338,7 +8436,7 @@ static void __init bootstrap_cache_sheaves(struct kmem_cache *s)
if (!capacity)
return;
for_each_node_mask(node, slab_nodes) {
for_each_node_mask(node, slab_barn_nodes) {
struct node_barn *barn;
barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, node);
@@ -8349,7 +8447,7 @@ static void __init bootstrap_cache_sheaves(struct kmem_cache *s)
}
barn_init(barn);
get_node(s, node)->barn = barn;
s->per_node[node].barn = barn;
}
for_each_possible_cpu(cpu) {
@@ -8410,6 +8508,9 @@ void __init kmem_cache_init(void)
for_each_node_state(node, N_MEMORY)
node_set(node, slab_nodes);
for_each_online_node(node)
node_set(node, slab_barn_nodes);
create_boot_cache(kmem_cache_node, "kmem_cache_node",
sizeof(struct kmem_cache_node),
SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0);
@@ -8420,8 +8521,8 @@ void __init kmem_cache_init(void)
slab_state = PARTIAL;
create_boot_cache(kmem_cache, "kmem_cache",
offsetof(struct kmem_cache, node) +
nr_node_ids * sizeof(struct kmem_cache_node *),
offsetof(struct kmem_cache, per_node) +
nr_node_ids * sizeof(struct kmem_cache_per_node_ptrs),
SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0);
kmem_cache = bootstrap(&boot_kmem_cache);
@@ -8436,7 +8537,7 @@ void __init kmem_cache_init(void)
/* Setup random freelists for each cache */
init_freelist_randomization();
cpuhp_setup_state_nocalls(CPUHP_SLUB_DEAD, "slub:dead", NULL,
cpuhp_setup_state_nocalls(CPUHP_SLUB_DEAD, "slub:dead", slub_cpu_setup,
slub_cpu_dead);
pr_info("SLUB: HWalign=%d, Order=%u-%u, MinObjects=%u, CPUs=%u, Nodes=%u\n",
@@ -8903,7 +9004,7 @@ static ssize_t show_slab_objects(struct kmem_cache *s,
return len;
}
#define to_slab_attr(n) container_of(n, struct slab_attribute, attr)
#define to_slab_attr(n) container_of_const(n, struct slab_attribute, attr)
#define to_slab(n) container_of(n, struct kmem_cache, kobj)
struct slab_attribute {
@@ -8913,10 +9014,10 @@ struct slab_attribute {
};
#define SLAB_ATTR_RO(_name) \
static struct slab_attribute _name##_attr = __ATTR_RO_MODE(_name, 0400)
static const struct slab_attribute _name##_attr = __ATTR_RO_MODE(_name, 0400)
#define SLAB_ATTR(_name) \
static struct slab_attribute _name##_attr = __ATTR_RW_MODE(_name, 0600)
static const struct slab_attribute _name##_attr = __ATTR_RW_MODE(_name, 0600)
static ssize_t slab_size_show(struct kmem_cache *s, char *buf)
{
@@ -9310,7 +9411,7 @@ static ssize_t skip_kfence_store(struct kmem_cache *s,
SLAB_ATTR(skip_kfence);
#endif
static struct attribute *slab_attrs[] = {
static const struct attribute *const slab_attrs[] = {
&slab_size_attr.attr,
&object_size_attr.attr,
&objs_per_slab_attr.attr,
@@ -9387,15 +9488,13 @@ static struct attribute *slab_attrs[] = {
NULL
};
static const struct attribute_group slab_attr_group = {
.attrs = slab_attrs,
};
ATTRIBUTE_GROUPS(slab);
static ssize_t slab_attr_show(struct kobject *kobj,
struct attribute *attr,
char *buf)
{
struct slab_attribute *attribute;
const struct slab_attribute *attribute;
struct kmem_cache *s;
attribute = to_slab_attr(attr);
@@ -9411,7 +9510,7 @@ static ssize_t slab_attr_store(struct kobject *kobj,
struct attribute *attr,
const char *buf, size_t len)
{
struct slab_attribute *attribute;
const struct slab_attribute *attribute;
struct kmem_cache *s;
attribute = to_slab_attr(attr);
@@ -9436,6 +9535,7 @@ static const struct sysfs_ops slab_sysfs_ops = {
static const struct kobj_type slab_ktype = {
.sysfs_ops = &slab_sysfs_ops,
.release = kmem_cache_release,
.default_groups = slab_groups,
};
static struct kset *slab_kset;
@@ -9523,10 +9623,6 @@ static int sysfs_slab_add(struct kmem_cache *s)
if (err)
goto out;
err = sysfs_create_group(&s->kobj, &slab_attr_group);
if (err)
goto out_del_kobj;
if (!unmergeable) {
/* Setup first alias */
sysfs_slab_alias(s, s->name);
@@ -9535,9 +9631,6 @@ out:
if (!unmergeable)
kfree(name);
return err;
out_del_kobj:
kobject_del(&s->kobj);
goto out;
}
void sysfs_slab_unlink(struct kmem_cache *s)
tools/sched_ext/include/scx/bpf_arena_common.bpf.h
+6 -2
View File
@@ -15,7 +15,9 @@
#endif
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && !defined(BPF_ARENA_FORCE_ASM)
#ifndef __arena
#define __arena __attribute__((address_space(1)))
#endif
#define __arena_global __attribute__((address_space(1)))
#define cast_kern(ptr) /* nop for bpf prog. emitted by LLVM */
#define cast_user(ptr) /* nop for bpf prog. emitted by LLVM */
@@ -81,12 +83,13 @@
void __arena* bpf_arena_alloc_pages(void *map, void __arena *addr, __u32 page_cnt,
int node_id, __u64 flags) __ksym __weak;
void bpf_arena_free_pages(void *map, void __arena *ptr, __u32 page_cnt) __ksym __weak;
int bpf_arena_reserve_pages(void *map, void __arena *ptr, __u32 page_cnt) __ksym __weak;
/*
* Note that cond_break can only be portably used in the body of a breakable
* construct, whereas can_loop can be used anywhere.
*/
#ifdef TEST
#ifdef SCX_BPF_UNITTEST
#define can_loop true
#define __cond_break(expr) expr
#else
@@ -165,7 +168,7 @@ void bpf_arena_free_pages(void *map, void __arena *ptr, __u32 page_cnt) __ksym _
})
#endif /* __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ */
#endif /* __BPF_FEATURE_MAY_GOTO */
#endif /* TEST */
#endif /* SCX_BPF_UNITTEST */
#define cond_break __cond_break(break)
#define cond_break_label(label) __cond_break(goto label)
@@ -173,3 +176,4 @@ void bpf_arena_free_pages(void *map, void __arena *ptr, __u32 page_cnt) __ksym _
void bpf_preempt_disable(void) __weak __ksym;
void bpf_preempt_enable(void) __weak __ksym;
ssize_t bpf_arena_mapping_nr_pages(void *p__map) __weak __ksym;
tools/sched_ext/include/scx/common.bpf.h
+277
View File
@@ -291,6 +291,50 @@ BPF_PROG(name, ##args)
})
#endif /* ARRAY_ELEM_PTR */
/**
* __sink - Hide @expr's value from the compiler and BPF verifier
* @expr: The expression whose value should be opacified
*
* No-op at runtime. The empty inline assembly with a read-write constraint
* ("+g") has two effects at compile/verify time:
*
* 1. Compiler: treats @expr as both read and written, preventing dead-code
* elimination and keeping @expr (and any side effects that produced it)
* alive.
*
* 2. BPF verifier: forgets the precise value/range of @expr ("makes it
* imprecise"). The verifier normally tracks exact ranges for every register
* and stack slot. While useful, precision means each distinct value creates a
* separate verifier state. Inside loops this leads to state explosion - each
* iteration carries different precise values so states never merge and the
* verifier explores every iteration individually.
*
* Example - preventing loop state explosion::
*
* u32 nr_intersects = 0, nr_covered = 0;
* __sink(nr_intersects);
* __sink(nr_covered);
* bpf_for(i, 0, nr_nodes) {
* if (intersects(cpumask, node_mask[i]))
* nr_intersects++;
* if (covers(cpumask, node_mask[i]))
* nr_covered++;
* }
*
* Without __sink(), the verifier tracks every possible (nr_intersects,
* nr_covered) pair across iterations, causing "BPF program is too large". With
* __sink(), the values become unknown scalars so all iterations collapse into
* one reusable state.
*
* Example - keeping a reference alive::
*
* struct task_struct *t = bpf_task_acquire(task);
* __sink(t);
*
* Follows the convention from BPF selftests (bpf_misc.h).
*/
#define __sink(expr) asm volatile ("" : "+g"(expr))
/*
* BPF declarations and helpers
*/
@@ -336,6 +380,7 @@ void bpf_task_release(struct task_struct *p) __ksym;
/* cgroup */
struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level) __ksym;
struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp) __ksym;
void bpf_cgroup_release(struct cgroup *cgrp) __ksym;
struct cgroup *bpf_cgroup_from_id(u64 cgid) __ksym;
@@ -741,6 +786,73 @@ static inline u64 __sqrt_u64(u64 x)
return r;
}
/*
* ctzll -- Counts trailing zeros in an unsigned long long. If the input value
* is zero, the return value is undefined.
*/
static inline int ctzll(u64 v)
{
#if (!defined(__BPF__) && defined(__SCX_TARGET_ARCH_x86)) || \
(defined(__BPF__) && defined(__clang_major__) && __clang_major__ >= 19)
/*
* Use the ctz builtin when: (1) building for native x86, or
* (2) building for BPF with clang >= 19 (BPF backend supports
* the intrinsic from clang 19 onward; earlier versions hit
* "unimplemented opcode" in the backend).
*/
return __builtin_ctzll(v);
#else
/*
* If neither the target architecture nor the toolchains support ctzll,
* use software-based emulation. Let's use the De Bruijn sequence-based
* approach to find LSB fastly. See the details of De Bruijn sequence:
*
* https://en.wikipedia.org/wiki/De_Bruijn_sequence
* https://www.chessprogramming.org/BitScan#De_Bruijn_Multiplication
*/
const int lookup_table[64] = {
0, 1, 48, 2, 57, 49, 28, 3, 61, 58, 50, 42, 38, 29, 17, 4,
62, 55, 59, 36, 53, 51, 43, 22, 45, 39, 33, 30, 24, 18, 12, 5,
63, 47, 56, 27, 60, 41, 37, 16, 54, 35, 52, 21, 44, 32, 23, 11,
46, 26, 40, 15, 34, 20, 31, 10, 25, 14, 19, 9, 13, 8, 7, 6,
};
const u64 DEBRUIJN_CONSTANT = 0x03f79d71b4cb0a89ULL;
unsigned int index;
u64 lowest_bit;
const int *lt;
if (v == 0)
return -1;
/*
* Isolate the least significant bit (LSB).
* For example, if v = 0b...10100, then v & -v = 0b...00100
*/
lowest_bit = v & -v;
/*
* Each isolated bit produces a unique 6-bit value, guaranteed by the
* De Bruijn property. Calculate a unique index into the lookup table
* using the magic constant and a right shift.
*
* Multiplying by the 64-bit constant "spreads out" that 1-bit into a
* unique pattern in the top 6 bits. This uniqueness property is
* exactly what a De Bruijn sequence guarantees: Every possible 6-bit
* pattern (in top bits) occurs exactly once for each LSB position. So,
* the constant 0x03f79d71b4cb0a89ULL is carefully chosen to be a
* De Bruijn sequence, ensuring no collisions in the table index.
*/
index = (lowest_bit * DEBRUIJN_CONSTANT) >> 58;
/*
* Lookup in a precomputed table. No collision is guaranteed by the
* De Bruijn property.
*/
lt = MEMBER_VPTR(lookup_table, [index]);
return (lt)? *lt : -1;
#endif
}
/*
* Return a value proportionally scaled to the task's weight.
*/
@@ -758,6 +870,171 @@ static inline u64 scale_by_task_weight_inverse(const struct task_struct *p, u64
}
/*
* Get a random u64 from the kernel's pseudo-random generator.
*/
static inline u64 get_prandom_u64()
{
return ((u64)bpf_get_prandom_u32() << 32) | bpf_get_prandom_u32();
}
/*
* Define the shadow structure to avoid a compilation error when
* vmlinux.h does not enable necessary kernel configs. The ___local
* suffix is a CO-RE convention that tells the loader to match this
* against the base struct rq in the kernel. The attribute
* preserve_access_index tells the compiler to generate a CO-RE
* relocation for these fields.
*/
struct rq___local {
/*
* A monotonically increasing clock per CPU. It is rq->clock minus
* cumulative IRQ time and hypervisor steal time. Unlike rq->clock,
* it does not advance during IRQ processing or hypervisor preemption.
* It does advance during idle (the idle task counts as a running task
* for this purpose).
*/
u64 clock_task;
/*
* Invariant version of clock_task scaled by CPU capacity and
* frequency. For example, clock_pelt advances 2x slower on a CPU
* with half the capacity.
*
* At idle exit, rq->clock_pelt jumps forward to resync with
* clock_task. The kernel's rq_clock_pelt() corrects for this jump
* by subtracting lost_idle_time, yielding a clock that appears
* continuous across idle transitions. scx_clock_pelt() mirrors
* rq_clock_pelt() by performing the same subtraction.
*/
u64 clock_pelt;
/*
* Accumulates the magnitude of each clock_pelt jump at idle exit.
* Subtracting this from clock_pelt gives rq_clock_pelt(): a
* continuous, capacity-invariant clock suitable for both task
* execution time stamping and cross-idle measurements.
*/
unsigned long lost_idle_time;
/*
* Shadow of paravirt_steal_clock() (the hypervisor's cumulative
* stolen time counter). Stays frozen while the hypervisor preempts
* the vCPU; catches up the next time update_rq_clock_task() is
* called. The delta is the stolen time not yet subtracted from
* clock_task.
*
* Unlike irqtime->total (a plain kernel-side field), the live stolen
* time counter lives in hypervisor-specific shared memory and has no
* kernel-side equivalent readable from BPF in a hypervisor-agnostic
* way. This field is therefore the only portable BPF-accessible
* approximation of cumulative steal time.
*
* Available only when CONFIG_PARAVIRT_TIME_ACCOUNTING is on.
*/
u64 prev_steal_time_rq;
} __attribute__((preserve_access_index));
extern struct rq runqueues __ksym;
/*
* Define the shadow structure to avoid a compilation error when
* vmlinux.h does not enable necessary kernel configs.
*/
struct irqtime___local {
/*
* Cumulative IRQ time counter for this CPU, in nanoseconds. Advances
* immediately at the exit of every hardirq and non-ksoftirqd softirq
* via irqtime_account_irq(). ksoftirqd time is counted as normal
* task time and is NOT included. NMI time is also NOT included.
*
* The companion field irqtime->sync (struct u64_stats_sync) protects
* against 64-bit tearing on 32-bit architectures. On 64-bit kernels,
* u64_stats_sync is an empty struct and all seqcount operations are
* no-ops, so a plain BPF_CORE_READ of this field is safe.
*
* Available only when CONFIG_IRQ_TIME_ACCOUNTING is on.
*/
u64 total;
} __attribute__((preserve_access_index));
/*
* cpu_irqtime is a per-CPU variable defined only when
* CONFIG_IRQ_TIME_ACCOUNTING is on. Declare it as __weak so the BPF
* loader sets its address to 0 (rather than failing) when the symbol
* is absent from the running kernel.
*/
extern struct irqtime___local cpu_irqtime __ksym __weak;
static inline struct rq___local *get_current_rq(u32 cpu)
{
/*
* This is a workaround to get an rq pointer since we decided to
* deprecate scx_bpf_cpu_rq().
*
* WARNING: The caller must hold the rq lock for @cpu. This is
* guaranteed when called from scheduling callbacks (ops.running,
* ops.stopping, ops.enqueue, ops.dequeue, ops.dispatch, etc.).
* There is no runtime check available in BPF for kernel spinlock
* state — correctness is enforced by calling context only.
*/
return (void *)bpf_per_cpu_ptr(&runqueues, cpu);
}
static inline u64 scx_clock_task(u32 cpu)
{
struct rq___local *rq = get_current_rq(cpu);
/* Equivalent to the kernel's rq_clock_task(). */
return rq ? rq->clock_task : 0;
}
static inline u64 scx_clock_pelt(u32 cpu)
{
struct rq___local *rq = get_current_rq(cpu);
/*
* Equivalent to the kernel's rq_clock_pelt(): subtracts
* lost_idle_time from clock_pelt to absorb the jump that occurs
* when clock_pelt resyncs with clock_task at idle exit. The result
* is a continuous, capacity-invariant clock safe for both task
* execution time stamping and cross-idle measurements.
*/
return rq ? (rq->clock_pelt - rq->lost_idle_time) : 0;
}
static inline u64 scx_clock_virt(u32 cpu)
{
struct rq___local *rq;
/*
* Check field existence before calling get_current_rq() so we avoid
* the per_cpu lookup entirely on kernels built without
* CONFIG_PARAVIRT_TIME_ACCOUNTING.
*/
if (!bpf_core_field_exists(((struct rq___local *)0)->prev_steal_time_rq))
return 0;
/* Lagging shadow of the kernel's paravirt_steal_clock(). */
rq = get_current_rq(cpu);
return rq ? BPF_CORE_READ(rq, prev_steal_time_rq) : 0;
}
static inline u64 scx_clock_irq(u32 cpu)
{
struct irqtime___local *irqt;
/*
* bpf_core_type_exists() resolves at load time: if struct irqtime is
* absent from kernel BTF (CONFIG_IRQ_TIME_ACCOUNTING off), the loader
* patches this into an unconditional return 0, making the
* bpf_per_cpu_ptr() call below dead code that the verifier never sees.
*/
if (!bpf_core_type_exists(struct irqtime___local))
return 0;
/* Equivalent to the kernel's irq_time_read(). */
irqt = bpf_per_cpu_ptr(&cpu_irqtime, cpu);
return irqt ? BPF_CORE_READ(irqt, total) : 0;
}
#include "compat.bpf.h"
#include "enums.bpf.h"
tools/sched_ext/include/scx/common.h
+1 -4
View File
@@ -67,6 +67,7 @@ typedef int64_t s64;
bpf_map__set_value_size((__skel)->maps.elfsec##_##arr, \
sizeof((__skel)->elfsec##_##arr->arr[0]) * (n)); \
(__skel)->elfsec##_##arr = \
(typeof((__skel)->elfsec##_##arr)) \
bpf_map__initial_value((__skel)->maps.elfsec##_##arr, &__sz); \
} while (0)
@@ -74,10 +75,6 @@ typedef int64_t s64;
#include "compat.h"
#include "enums.h"
/* not available when building kernel tools/sched_ext */
#if __has_include(<lib/sdt_task_defs.h>)
#include "bpf_arena_common.h"
#include <lib/sdt_task_defs.h>
#endif
#endif /* __SCHED_EXT_COMMON_H */
tools/sched_ext/include/scx/compat.bpf.h
+52 -5
View File
@@ -28,8 +28,11 @@ struct cgroup *scx_bpf_task_cgroup___new(struct task_struct *p) __ksym __weak;
*
* scx_bpf_dispatch_from_dsq() and friends were added during v6.12 by
* 4c30f5ce4f7a ("sched_ext: Implement scx_bpf_dispatch[_vtime]_from_dsq()").
*
* v7.1: scx_bpf_dsq_move_to_local___v2() to add @enq_flags.
*/
bool scx_bpf_dsq_move_to_local___new(u64 dsq_id) __ksym __weak;
bool scx_bpf_dsq_move_to_local___v2(u64 dsq_id, u64 enq_flags) __ksym __weak;
bool scx_bpf_dsq_move_to_local___v1(u64 dsq_id) __ksym __weak;
void scx_bpf_dsq_move_set_slice___new(struct bpf_iter_scx_dsq *it__iter, u64 slice) __ksym __weak;
void scx_bpf_dsq_move_set_vtime___new(struct bpf_iter_scx_dsq *it__iter, u64 vtime) __ksym __weak;
bool scx_bpf_dsq_move___new(struct bpf_iter_scx_dsq *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __ksym __weak;
@@ -41,10 +44,12 @@ void scx_bpf_dispatch_from_dsq_set_vtime___old(struct bpf_iter_scx_dsq *it__iter
bool scx_bpf_dispatch_from_dsq___old(struct bpf_iter_scx_dsq *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __ksym __weak;
bool scx_bpf_dispatch_vtime_from_dsq___old(struct bpf_iter_scx_dsq *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __ksym __weak;
#define scx_bpf_dsq_move_to_local(dsq_id) \
(bpf_ksym_exists(scx_bpf_dsq_move_to_local___new) ? \
scx_bpf_dsq_move_to_local___new((dsq_id)) : \
scx_bpf_consume___old((dsq_id)))
#define scx_bpf_dsq_move_to_local(dsq_id, enq_flags) \
(bpf_ksym_exists(scx_bpf_dsq_move_to_local___v2) ? \
scx_bpf_dsq_move_to_local___v2((dsq_id), (enq_flags)) : \
(bpf_ksym_exists(scx_bpf_dsq_move_to_local___v1) ? \
scx_bpf_dsq_move_to_local___v1((dsq_id)) : \
scx_bpf_consume___old((dsq_id))))
#define scx_bpf_dsq_move_set_slice(it__iter, slice) \
(bpf_ksym_exists(scx_bpf_dsq_move_set_slice___new) ? \
@@ -103,6 +108,19 @@ static inline struct task_struct *__COMPAT_scx_bpf_dsq_peek(u64 dsq_id)
return p;
}
/*
* v7.1: scx_bpf_sub_dispatch() for sub-sched dispatch. Preserve until
* we drop the compat layer for older kernels that lack the kfunc.
*/
bool scx_bpf_sub_dispatch___compat(u64 cgroup_id) __ksym __weak;
static inline bool scx_bpf_sub_dispatch(u64 cgroup_id)
{
if (bpf_ksym_exists(scx_bpf_sub_dispatch___compat))
return scx_bpf_sub_dispatch___compat(cgroup_id);
return false;
}
/**
* __COMPAT_is_enq_cpu_selected - Test if SCX_ENQ_CPU_SELECTED is on
* in a compatible way. We will preserve this __COMPAT helper until v6.16.
@@ -266,6 +284,14 @@ scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
}
}
/*
* scx_bpf_select_cpu_and() is now an inline wrapper. Use this instead of
* bpf_ksym_exists(scx_bpf_select_cpu_and) to test availability.
*/
#define __COMPAT_HAS_scx_bpf_select_cpu_and \
(bpf_core_type_exists(struct scx_bpf_select_cpu_and_args) || \
bpf_ksym_exists(scx_bpf_select_cpu_and___compat))
/**
* scx_bpf_dsq_insert_vtime - Insert a task into the vtime priority queue of a DSQ
* @p: task_struct to insert
@@ -375,6 +401,27 @@ static inline void scx_bpf_reenqueue_local(void)
scx_bpf_reenqueue_local___v1();
}
/*
* v6.20: New scx_bpf_dsq_reenq() that allows re-enqueues on more DSQs. This
* will eventually deprecate scx_bpf_reenqueue_local().
*/
void scx_bpf_dsq_reenq___compat(u64 dsq_id, u64 reenq_flags, const struct bpf_prog_aux *aux__prog) __ksym __weak;
static inline bool __COMPAT_has_generic_reenq(void)
{
return bpf_ksym_exists(scx_bpf_dsq_reenq___compat);
}
static inline void scx_bpf_dsq_reenq(u64 dsq_id, u64 reenq_flags)
{
if (bpf_ksym_exists(scx_bpf_dsq_reenq___compat))
scx_bpf_dsq_reenq___compat(dsq_id, reenq_flags, NULL);
else if (dsq_id == SCX_DSQ_LOCAL && reenq_flags == 0)
scx_bpf_reenqueue_local();
else
scx_bpf_error("kernel too old to reenqueue foreign local or user DSQs");
}
/*
* Define sched_ext_ops. This may be expanded to define multiple variants for
* backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH().
tools/sched_ext/include/scx/compat.h
+51 -1
View File
@@ -8,6 +8,7 @@
#define __SCX_COMPAT_H
#include <bpf/btf.h>
#include <bpf/libbpf.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
@@ -115,6 +116,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
#define SCX_OPS_ENQ_MIGRATION_DISABLED SCX_OPS_FLAG(SCX_OPS_ENQ_MIGRATION_DISABLED)
#define SCX_OPS_ALLOW_QUEUED_WAKEUP SCX_OPS_FLAG(SCX_OPS_ALLOW_QUEUED_WAKEUP)
#define SCX_OPS_BUILTIN_IDLE_PER_NODE SCX_OPS_FLAG(SCX_OPS_BUILTIN_IDLE_PER_NODE)
#define SCX_OPS_ALWAYS_ENQ_IMMED SCX_OPS_FLAG(SCX_OPS_ALWAYS_ENQ_IMMED)
#define SCX_PICK_IDLE_FLAG(name) __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", #name)
@@ -158,6 +160,7 @@ static inline long scx_hotplug_seq(void)
* COMPAT:
* - v6.17: ops.cgroup_set_bandwidth()
* - v6.19: ops.cgroup_set_idle()
* - v7.1: ops.sub_attach(), ops.sub_detach(), ops.sub_cgroup_id
*/
#define SCX_OPS_OPEN(__ops_name, __scx_name) ({ \
struct __scx_name *__skel; \
@@ -179,18 +182,65 @@ static inline long scx_hotplug_seq(void)
fprintf(stderr, "WARNING: kernel doesn't support ops.cgroup_set_idle()\n"); \
__skel->struct_ops.__ops_name->cgroup_set_idle = NULL; \
} \
if (__skel->struct_ops.__ops_name->sub_attach && \
!__COMPAT_struct_has_field("sched_ext_ops", "sub_attach")) { \
fprintf(stderr, "WARNING: kernel doesn't support ops.sub_attach()\n"); \
__skel->struct_ops.__ops_name->sub_attach = NULL; \
} \
if (__skel->struct_ops.__ops_name->sub_detach && \
!__COMPAT_struct_has_field("sched_ext_ops", "sub_detach")) { \
fprintf(stderr, "WARNING: kernel doesn't support ops.sub_detach()\n"); \
__skel->struct_ops.__ops_name->sub_detach = NULL; \
} \
if (__skel->struct_ops.__ops_name->sub_cgroup_id > 0 && \
!__COMPAT_struct_has_field("sched_ext_ops", "sub_cgroup_id")) { \
fprintf(stderr, "WARNING: kernel doesn't support ops.sub_cgroup_id\n"); \
__skel->struct_ops.__ops_name->sub_cgroup_id = 0; \
} \
__skel; \
})
/*
* Associate non-struct_ops BPF programs with the scheduler's struct_ops map so
* that scx_prog_sched() can determine which scheduler a BPF program belongs
* to. Requires libbpf >= 1.7.
*/
#if LIBBPF_MAJOR_VERSION > 1 || \
(LIBBPF_MAJOR_VERSION == 1 && LIBBPF_MINOR_VERSION >= 7)
static inline void __scx_ops_assoc_prog(struct bpf_program *prog,
struct bpf_map *map,
const char *ops_name)
{
s32 err = bpf_program__assoc_struct_ops(prog, map, NULL);
if (err)
fprintf(stderr,
"ERROR: Failed to associate %s with %s: %d\n",
bpf_program__name(prog), ops_name, err);
}
#else
static inline void __scx_ops_assoc_prog(struct bpf_program *prog,
struct bpf_map *map,
const char *ops_name)
{
}
#endif
#define SCX_OPS_LOAD(__skel, __ops_name, __scx_name, __uei_name) ({ \
struct bpf_program *__prog; \
UEI_SET_SIZE(__skel, __ops_name, __uei_name); \
SCX_BUG_ON(__scx_name##__load((__skel)), "Failed to load skel"); \
bpf_object__for_each_program(__prog, (__skel)->obj) { \
if (bpf_program__type(__prog) == BPF_PROG_TYPE_STRUCT_OPS) \
continue; \
__scx_ops_assoc_prog(__prog, (__skel)->maps.__ops_name, \
#__ops_name); \
} \
})
/*
* New versions of bpftool now emit additional link placeholders for BPF maps,
* and set up BPF skeleton in such a way that libbpf will auto-attach BPF maps
* automatically, assumming libbpf is recent enough (v1.5+). Old libbpf will do
* automatically, assuming libbpf is recent enough (v1.5+). Old libbpf will do
* nothing with those links and won't attempt to auto-attach maps.
*
* To maintain compatibility with older libbpf while avoiding trying to attach
tools/sched_ext/include/scx/enum_defs.autogen.h
+48 -13
View File
@@ -14,18 +14,27 @@
#define HAVE_SCX_EXIT_MSG_LEN
#define HAVE_SCX_EXIT_DUMP_DFL_LEN
#define HAVE_SCX_CPUPERF_ONE
#define HAVE_SCX_OPS_TASK_ITER_BATCH
#define HAVE_SCX_TASK_ITER_BATCH
#define HAVE_SCX_BYPASS_HOST_NTH
#define HAVE_SCX_BYPASS_LB_DFL_INTV_US
#define HAVE_SCX_BYPASS_LB_DONOR_PCT
#define HAVE_SCX_BYPASS_LB_MIN_DELTA_DIV
#define HAVE_SCX_BYPASS_LB_BATCH
#define HAVE_SCX_REENQ_LOCAL_MAX_REPEAT
#define HAVE_SCX_SUB_MAX_DEPTH
#define HAVE_SCX_CPU_PREEMPT_RT
#define HAVE_SCX_CPU_PREEMPT_DL
#define HAVE_SCX_CPU_PREEMPT_STOP
#define HAVE_SCX_CPU_PREEMPT_UNKNOWN
#define HAVE_SCX_DEQ_SLEEP
#define HAVE_SCX_DEQ_CORE_SCHED_EXEC
#define HAVE_SCX_DEQ_SCHED_CHANGE
#define HAVE_SCX_DSQ_FLAG_BUILTIN
#define HAVE_SCX_DSQ_FLAG_LOCAL_ON
#define HAVE_SCX_DSQ_INVALID
#define HAVE_SCX_DSQ_GLOBAL
#define HAVE_SCX_DSQ_LOCAL
#define HAVE_SCX_DSQ_BYPASS
#define HAVE_SCX_DSQ_LOCAL_ON
#define HAVE_SCX_DSQ_LOCAL_CPU_MASK
#define HAVE_SCX_DSQ_ITER_REV
@@ -35,31 +44,55 @@
#define HAVE___SCX_DSQ_ITER_ALL_FLAGS
#define HAVE_SCX_DSQ_LNODE_ITER_CURSOR
#define HAVE___SCX_DSQ_LNODE_PRIV_SHIFT
#define HAVE_SCX_ENABLING
#define HAVE_SCX_ENABLED
#define HAVE_SCX_DISABLING
#define HAVE_SCX_DISABLED
#define HAVE_SCX_ENQ_WAKEUP
#define HAVE_SCX_ENQ_HEAD
#define HAVE_SCX_ENQ_CPU_SELECTED
#define HAVE_SCX_ENQ_PREEMPT
#define HAVE_SCX_ENQ_IMMED
#define HAVE_SCX_ENQ_REENQ
#define HAVE_SCX_ENQ_LAST
#define HAVE___SCX_ENQ_INTERNAL_MASK
#define HAVE_SCX_ENQ_CLEAR_OPSS
#define HAVE_SCX_ENQ_DSQ_PRIQ
#define HAVE_SCX_ENQ_NESTED
#define HAVE_SCX_ENQ_GDSQ_FALLBACK
#define HAVE_SCX_TASK_DSQ_ON_PRIQ
#define HAVE_SCX_TASK_QUEUED
#define HAVE_SCX_TASK_IN_CUSTODY
#define HAVE_SCX_TASK_RESET_RUNNABLE_AT
#define HAVE_SCX_TASK_DEQD_FOR_SLEEP
#define HAVE_SCX_TASK_SUB_INIT
#define HAVE_SCX_TASK_IMMED
#define HAVE_SCX_TASK_STATE_SHIFT
#define HAVE_SCX_TASK_STATE_BITS
#define HAVE_SCX_TASK_STATE_MASK
#define HAVE_SCX_TASK_NONE
#define HAVE_SCX_TASK_INIT
#define HAVE_SCX_TASK_READY
#define HAVE_SCX_TASK_ENABLED
#define HAVE_SCX_TASK_REENQ_REASON_SHIFT
#define HAVE_SCX_TASK_REENQ_REASON_BITS
#define HAVE_SCX_TASK_REENQ_REASON_MASK
#define HAVE_SCX_TASK_REENQ_NONE
#define HAVE_SCX_TASK_REENQ_KFUNC
#define HAVE_SCX_TASK_REENQ_IMMED
#define HAVE_SCX_TASK_REENQ_PREEMPTED
#define HAVE_SCX_TASK_CURSOR
#define HAVE_SCX_ECODE_RSN_HOTPLUG
#define HAVE_SCX_ECODE_RSN_CGROUP_OFFLINE
#define HAVE_SCX_ECODE_ACT_RESTART
#define HAVE_SCX_EFLAG_INITIALIZED
#define HAVE_SCX_EXIT_NONE
#define HAVE_SCX_EXIT_DONE
#define HAVE_SCX_EXIT_UNREG
#define HAVE_SCX_EXIT_UNREG_BPF
#define HAVE_SCX_EXIT_UNREG_KERN
#define HAVE_SCX_EXIT_SYSRQ
#define HAVE_SCX_EXIT_PARENT
#define HAVE_SCX_EXIT_ERROR
#define HAVE_SCX_EXIT_ERROR_BPF
#define HAVE_SCX_EXIT_ERROR_STALL
@@ -80,40 +113,42 @@
#define HAVE_SCX_OPI_CPU_HOTPLUG_BEGIN
#define HAVE_SCX_OPI_CPU_HOTPLUG_END
#define HAVE_SCX_OPI_END
#define HAVE_SCX_OPS_ENABLING
#define HAVE_SCX_OPS_ENABLED
#define HAVE_SCX_OPS_DISABLING
#define HAVE_SCX_OPS_DISABLED
#define HAVE_SCX_OPS_KEEP_BUILTIN_IDLE
#define HAVE_SCX_OPS_ENQ_LAST
#define HAVE_SCX_OPS_ENQ_EXITING
#define HAVE_SCX_OPS_SWITCH_PARTIAL
#define HAVE_SCX_OPS_ENQ_MIGRATION_DISABLED
#define HAVE_SCX_OPS_ALLOW_QUEUED_WAKEUP
#define HAVE_SCX_OPS_HAS_CGROUP_WEIGHT
#define HAVE_SCX_OPS_BUILTIN_IDLE_PER_NODE
#define HAVE_SCX_OPS_ALWAYS_ENQ_IMMED
#define HAVE_SCX_OPS_ALL_FLAGS
#define HAVE___SCX_OPS_INTERNAL_MASK
#define HAVE_SCX_OPS_HAS_CPU_PREEMPT
#define HAVE_SCX_OPSS_NONE
#define HAVE_SCX_OPSS_QUEUEING
#define HAVE_SCX_OPSS_QUEUED
#define HAVE_SCX_OPSS_DISPATCHING
#define HAVE_SCX_OPSS_QSEQ_SHIFT
#define HAVE_SCX_PICK_IDLE_CORE
#define HAVE_SCX_PICK_IDLE_IN_NODE
#define HAVE_SCX_OPS_NAME_LEN
#define HAVE_SCX_SLICE_DFL
#define HAVE_SCX_SLICE_BYPASS
#define HAVE_SCX_SLICE_INF
#define HAVE_SCX_REENQ_ANY
#define HAVE___SCX_REENQ_FILTER_MASK
#define HAVE___SCX_REENQ_USER_MASK
#define HAVE_SCX_REENQ_TSR_RQ_OPEN
#define HAVE_SCX_REENQ_TSR_NOT_FIRST
#define HAVE___SCX_REENQ_TSR_MASK
#define HAVE_SCX_RQ_ONLINE
#define HAVE_SCX_RQ_CAN_STOP_TICK
#define HAVE_SCX_RQ_BAL_PENDING
#define HAVE_SCX_RQ_BAL_KEEP
#define HAVE_SCX_RQ_BYPASSING
#define HAVE_SCX_RQ_CLK_VALID
#define HAVE_SCX_RQ_BAL_CB_PENDING
#define HAVE_SCX_RQ_IN_WAKEUP
#define HAVE_SCX_RQ_IN_BALANCE
#define HAVE_SCX_TASK_NONE
#define HAVE_SCX_TASK_INIT
#define HAVE_SCX_TASK_READY
#define HAVE_SCX_TASK_ENABLED
#define HAVE_SCX_TASK_NR_STATES
#define HAVE_SCX_SCHED_PCPU_BYPASSING
#define HAVE_SCX_TG_ONLINE
#define HAVE_SCX_TG_INITED
#define HAVE_SCX_WAKE_FORK
tools/sched_ext/include/scx/enums.autogen.bpf.h
+11
View File
@@ -67,6 +67,12 @@ const volatile u64 __SCX_TASK_RESET_RUNNABLE_AT __weak;
const volatile u64 __SCX_TASK_DEQD_FOR_SLEEP __weak;
#define SCX_TASK_DEQD_FOR_SLEEP __SCX_TASK_DEQD_FOR_SLEEP
const volatile u64 __SCX_TASK_SUB_INIT __weak;
#define SCX_TASK_SUB_INIT __SCX_TASK_SUB_INIT
const volatile u64 __SCX_TASK_IMMED __weak;
#define SCX_TASK_IMMED __SCX_TASK_IMMED
const volatile u64 __SCX_TASK_STATE_SHIFT __weak;
#define SCX_TASK_STATE_SHIFT __SCX_TASK_STATE_SHIFT
@@ -115,6 +121,9 @@ const volatile u64 __SCX_ENQ_HEAD __weak;
const volatile u64 __SCX_ENQ_PREEMPT __weak;
#define SCX_ENQ_PREEMPT __SCX_ENQ_PREEMPT
const volatile u64 __SCX_ENQ_IMMED __weak;
#define SCX_ENQ_IMMED __SCX_ENQ_IMMED
const volatile u64 __SCX_ENQ_REENQ __weak;
#define SCX_ENQ_REENQ __SCX_ENQ_REENQ
@@ -127,3 +136,5 @@ const volatile u64 __SCX_ENQ_CLEAR_OPSS __weak;
const volatile u64 __SCX_ENQ_DSQ_PRIQ __weak;
#define SCX_ENQ_DSQ_PRIQ __SCX_ENQ_DSQ_PRIQ
const volatile u64 __SCX_DEQ_SCHED_CHANGE __weak;
#define SCX_DEQ_SCHED_CHANGE __SCX_DEQ_SCHED_CHANGE
tools/sched_ext/include/scx/enums.autogen.h
+4
View File
@@ -26,6 +26,8 @@
SCX_ENUM_SET(skel, scx_ent_flags, SCX_TASK_QUEUED); \
SCX_ENUM_SET(skel, scx_ent_flags, SCX_TASK_RESET_RUNNABLE_AT); \
SCX_ENUM_SET(skel, scx_ent_flags, SCX_TASK_DEQD_FOR_SLEEP); \
SCX_ENUM_SET(skel, scx_ent_flags, SCX_TASK_SUB_INIT); \
SCX_ENUM_SET(skel, scx_ent_flags, SCX_TASK_IMMED); \
SCX_ENUM_SET(skel, scx_ent_flags, SCX_TASK_STATE_SHIFT); \
SCX_ENUM_SET(skel, scx_ent_flags, SCX_TASK_STATE_BITS); \
SCX_ENUM_SET(skel, scx_ent_flags, SCX_TASK_STATE_MASK); \
@@ -42,8 +44,10 @@
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_WAKEUP); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_HEAD); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_PREEMPT); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_IMMED); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_REENQ); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_LAST); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_CLEAR_OPSS); \
SCX_ENUM_SET(skel, scx_enq_flags, SCX_ENQ_DSQ_PRIQ); \
SCX_ENUM_SET(skel, scx_deq_flags, SCX_DEQ_SCHED_CHANGE); \
} while (0)
tools/sched_ext/include/scx/enums.h
+1 -1
View File
@@ -9,7 +9,7 @@
#ifndef __SCX_ENUMS_H
#define __SCX_ENUMS_H
static inline void __ENUM_set(u64 *val, char *type, char *name)
static inline void __ENUM_set(u64 *val, const char *type, const char *name)
{
bool res;
tools/sched_ext/scx_central.bpf.c
+44 -22
View File
@@ -60,6 +60,7 @@ const volatile u32 nr_cpu_ids = 1; /* !0 for veristat, set during init */
const volatile u64 slice_ns;
bool timer_pinned = true;
bool timer_started;
u64 nr_total, nr_locals, nr_queued, nr_lost_pids;
u64 nr_timers, nr_dispatches, nr_mismatches, nr_retries;
u64 nr_overflows;
@@ -179,9 +180,47 @@ static bool dispatch_to_cpu(s32 cpu)
return false;
}
static void start_central_timer(void)
{
struct bpf_timer *timer;
u32 key = 0;
int ret;
if (likely(timer_started))
return;
timer = bpf_map_lookup_elem(&central_timer, &key);
if (!timer) {
scx_bpf_error("failed to lookup central timer");
return;
}
ret = bpf_timer_start(timer, TIMER_INTERVAL_NS, BPF_F_TIMER_CPU_PIN);
/*
* BPF_F_TIMER_CPU_PIN is pretty new (>=6.7). If we're running in a
* kernel which doesn't have it, bpf_timer_start() will return -EINVAL.
* Retry without the PIN. This would be the perfect use case for
* bpf_core_enum_value_exists() but the enum type doesn't have a name
* and can't be used with bpf_core_enum_value_exists(). Oh well...
*/
if (ret == -EINVAL) {
timer_pinned = false;
ret = bpf_timer_start(timer, TIMER_INTERVAL_NS, 0);
}
if (ret) {
scx_bpf_error("bpf_timer_start failed (%d)", ret);
return;
}
timer_started = true;
}
void BPF_STRUCT_OPS(central_dispatch, s32 cpu, struct task_struct *prev)
{
if (cpu == central_cpu) {
start_central_timer();
/* dispatch for all other CPUs first */
__sync_fetch_and_add(&nr_dispatches, 1);
@@ -214,13 +253,13 @@ void BPF_STRUCT_OPS(central_dispatch, s32 cpu, struct task_struct *prev)
}
/* look for a task to run on the central CPU */
if (scx_bpf_dsq_move_to_local(FALLBACK_DSQ_ID))
if (scx_bpf_dsq_move_to_local(FALLBACK_DSQ_ID, 0))
return;
dispatch_to_cpu(central_cpu);
} else {
bool *gimme;
if (scx_bpf_dsq_move_to_local(FALLBACK_DSQ_ID))
if (scx_bpf_dsq_move_to_local(FALLBACK_DSQ_ID, 0))
return;
gimme = ARRAY_ELEM_PTR(cpu_gimme_task, cpu, nr_cpu_ids);
@@ -310,29 +349,12 @@ int BPF_STRUCT_OPS_SLEEPABLE(central_init)
if (!timer)
return -ESRCH;
if (bpf_get_smp_processor_id() != central_cpu) {
scx_bpf_error("init from non-central CPU");
return -EINVAL;
}
bpf_timer_init(timer, &central_timer, CLOCK_MONOTONIC);
bpf_timer_set_callback(timer, central_timerfn);
ret = bpf_timer_start(timer, TIMER_INTERVAL_NS, BPF_F_TIMER_CPU_PIN);
/*
* BPF_F_TIMER_CPU_PIN is pretty new (>=6.7). If we're running in a
* kernel which doesn't have it, bpf_timer_start() will return -EINVAL.
* Retry without the PIN. This would be the perfect use case for
* bpf_core_enum_value_exists() but the enum type doesn't have a name
* and can't be used with bpf_core_enum_value_exists(). Oh well...
*/
if (ret == -EINVAL) {
timer_pinned = false;
ret = bpf_timer_start(timer, TIMER_INTERVAL_NS, 0);
}
if (ret)
scx_bpf_error("bpf_timer_start failed (%d)", ret);
return ret;
scx_bpf_kick_cpu(central_cpu, 0);
return 0;
}
void BPF_STRUCT_OPS(central_exit, struct scx_exit_info *ei)
tools/sched_ext/scx_central.c
+1 -25
View File
@@ -5,7 +5,6 @@
* Copyright (c) 2022 David Vernet <dvernet@meta.com>
*/
#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <unistd.h>
#include <inttypes.h>
@@ -21,7 +20,7 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
"Usage: %s [-s SLICE_US] [-c CPU]\n"
"Usage: %s [-s SLICE_US] [-c CPU] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -c CPU Override the central CPU (default: 0)\n"
@@ -49,8 +48,6 @@ int main(int argc, char **argv)
struct bpf_link *link;
__u64 seq = 0, ecode;
__s32 opt;
cpu_set_t *cpuset;
size_t cpuset_size;
libbpf_set_print(libbpf_print_fn);
signal(SIGINT, sigint_handler);
@@ -96,27 +93,6 @@ restart:
SCX_OPS_LOAD(skel, central_ops, scx_central, uei);
/*
* Affinitize the loading thread to the central CPU, as:
* - That's where the BPF timer is first invoked in the BPF program.
* - We probably don't want this user space component to take up a core
* from a task that would benefit from avoiding preemption on one of
* the tickless cores.
*
* Until BPF supports pinning the timer, it's not guaranteed that it
* will always be invoked on the central CPU. In practice, this
* suffices the majority of the time.
*/
cpuset = CPU_ALLOC(skel->rodata->nr_cpu_ids);
SCX_BUG_ON(!cpuset, "Failed to allocate cpuset");
cpuset_size = CPU_ALLOC_SIZE(skel->rodata->nr_cpu_ids);
CPU_ZERO_S(cpuset_size, cpuset);
CPU_SET_S(skel->rodata->central_cpu, cpuset_size, cpuset);
SCX_BUG_ON(sched_setaffinity(0, cpuset_size, cpuset),
"Failed to affinitize to central CPU %d (max %d)",
skel->rodata->central_cpu, skel->rodata->nr_cpu_ids - 1);
CPU_FREE(cpuset);
link = SCX_OPS_ATTACH(skel, central_ops, scx_central);
if (!skel->data->timer_pinned)
tools/sched_ext/scx_cpu0.bpf.c
+1 -1
View File
@@ -66,7 +66,7 @@ void BPF_STRUCT_OPS(cpu0_enqueue, struct task_struct *p, u64 enq_flags)
void BPF_STRUCT_OPS(cpu0_dispatch, s32 cpu, struct task_struct *prev)
{
if (cpu == 0)
scx_bpf_dsq_move_to_local(DSQ_CPU0);
scx_bpf_dsq_move_to_local(DSQ_CPU0, 0);
}
s32 BPF_STRUCT_OPS_SLEEPABLE(cpu0_init)
tools/sched_ext/scx_flatcg.bpf.c
+13 -11
View File
@@ -18,7 +18,7 @@
* 100/(100+100) == 1/2. At its parent level, A is competing against D and A's
* share in that competition is 100/(200+100) == 1/3. B's eventual share in the
* system can be calculated by multiplying the two shares, 1/2 * 1/3 == 1/6. C's
* eventual shaer is the same at 1/6. D is only competing at the top level and
* eventual share is the same at 1/6. D is only competing at the top level and
* its share is 200/(100+200) == 2/3.
*
* So, instead of hierarchically scheduling level-by-level, we can consider it
@@ -551,9 +551,11 @@ void BPF_STRUCT_OPS(fcg_stopping, struct task_struct *p, bool runnable)
* too much, determine the execution time by taking explicit timestamps
* instead of depending on @p->scx.slice.
*/
if (!fifo_sched)
p->scx.dsq_vtime +=
(SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
if (!fifo_sched) {
u64 delta = scale_by_task_weight_inverse(p, SCX_SLICE_DFL - p->scx.slice);
scx_bpf_task_set_dsq_vtime(p, p->scx.dsq_vtime + delta);
}
taskc = bpf_task_storage_get(&task_ctx, p, 0, 0);
if (!taskc) {
@@ -660,7 +662,7 @@ static bool try_pick_next_cgroup(u64 *cgidp)
goto out_free;
}
if (!scx_bpf_dsq_move_to_local(cgid)) {
if (!scx_bpf_dsq_move_to_local(cgid, 0)) {
bpf_cgroup_release(cgrp);
stat_inc(FCG_STAT_PNC_EMPTY);
goto out_stash;
@@ -740,7 +742,7 @@ void BPF_STRUCT_OPS(fcg_dispatch, s32 cpu, struct task_struct *prev)
goto pick_next_cgroup;
if (time_before(now, cpuc->cur_at + cgrp_slice_ns)) {
if (scx_bpf_dsq_move_to_local(cpuc->cur_cgid)) {
if (scx_bpf_dsq_move_to_local(cpuc->cur_cgid, 0)) {
stat_inc(FCG_STAT_CNS_KEEP);
return;
}
@@ -780,7 +782,7 @@ void BPF_STRUCT_OPS(fcg_dispatch, s32 cpu, struct task_struct *prev)
pick_next_cgroup:
cpuc->cur_at = now;
if (scx_bpf_dsq_move_to_local(FALLBACK_DSQ)) {
if (scx_bpf_dsq_move_to_local(FALLBACK_DSQ, 0)) {
cpuc->cur_cgid = 0;
return;
}
@@ -822,7 +824,7 @@ s32 BPF_STRUCT_OPS(fcg_init_task, struct task_struct *p,
if (!(cgc = find_cgrp_ctx(args->cgroup)))
return -ENOENT;
p->scx.dsq_vtime = cgc->tvtime_now;
scx_bpf_task_set_dsq_vtime(p, cgc->tvtime_now);
return 0;
}
@@ -919,12 +921,12 @@ void BPF_STRUCT_OPS(fcg_cgroup_move, struct task_struct *p,
struct fcg_cgrp_ctx *from_cgc, *to_cgc;
s64 delta;
/* find_cgrp_ctx() triggers scx_ops_error() on lookup failures */
/* find_cgrp_ctx() triggers scx_bpf_error() on lookup failures */
if (!(from_cgc = find_cgrp_ctx(from)) || !(to_cgc = find_cgrp_ctx(to)))
return;
delta = time_delta(p->scx.dsq_vtime, from_cgc->tvtime_now);
p->scx.dsq_vtime = to_cgc->tvtime_now + delta;
scx_bpf_task_set_dsq_vtime(p, to_cgc->tvtime_now + delta);
}
s32 BPF_STRUCT_OPS_SLEEPABLE(fcg_init)
@@ -960,5 +962,5 @@ SCX_OPS_DEFINE(flatcg_ops,
.cgroup_move = (void *)fcg_cgroup_move,
.init = (void *)fcg_init,
.exit = (void *)fcg_exit,
.flags = SCX_OPS_HAS_CGROUP_WEIGHT | SCX_OPS_ENQ_EXITING,
.flags = SCX_OPS_ENQ_EXITING,
.name = "flatcg");
tools/sched_ext/scx_pair.c
+13 -3
View File
@@ -21,7 +21,7 @@ const char help_fmt[] =
"\n"
"See the top-level comment in .bpf.c for more details.\n"
"\n"
"Usage: %s [-S STRIDE]\n"
"Usage: %s [-S STRIDE] [-v]\n"
"\n"
" -S STRIDE Override CPU pair stride (default: nr_cpus_ids / 2)\n"
" -v Print libbpf debug messages\n"
@@ -48,6 +48,7 @@ int main(int argc, char **argv)
struct bpf_link *link;
__u64 seq = 0, ecode;
__s32 stride, i, opt, outer_fd;
__u32 pair_id = 0;
libbpf_set_print(libbpf_print_fn);
signal(SIGINT, sigint_handler);
@@ -82,6 +83,14 @@ restart:
scx_pair__destroy(skel);
return -1;
}
if (skel->rodata->nr_cpu_ids & 1) {
fprintf(stderr, "scx_pair requires an even CPU count, got %u\n",
skel->rodata->nr_cpu_ids);
scx_pair__destroy(skel);
return -1;
}
bpf_map__set_max_entries(skel->maps.pair_ctx, skel->rodata->nr_cpu_ids / 2);
/* Resize arrays so their element count is equal to cpu count. */
@@ -109,10 +118,11 @@ restart:
skel->rodata_pair_cpu->pair_cpu[i] = j;
skel->rodata_pair_cpu->pair_cpu[j] = i;
skel->rodata_pair_id->pair_id[i] = i;
skel->rodata_pair_id->pair_id[j] = i;
skel->rodata_pair_id->pair_id[i] = pair_id;
skel->rodata_pair_id->pair_id[j] = pair_id;
skel->rodata_in_pair_idx->in_pair_idx[i] = 0;
skel->rodata_in_pair_idx->in_pair_idx[j] = 1;
pair_id++;
printf("[%d, %d] ", i, j);
}
tools/sched_ext/scx_qmap.bpf.c
+172 -42
View File
@@ -11,8 +11,6 @@
*
* - BPF-side queueing using PIDs.
* - Sleepable per-task storage allocation using ops.prep_enable().
* - Using ops.cpu_release() to handle a higher priority scheduling class taking
* the CPU away.
* - Core-sched support.
*
* This scheduler is primarily for demonstration and testing of sched_ext
@@ -26,8 +24,11 @@
enum consts {
ONE_SEC_IN_NS = 1000000000,
ONE_MSEC_IN_NS = 1000000,
LOWPRI_INTV_NS = 10 * ONE_MSEC_IN_NS,
SHARED_DSQ = 0,
HIGHPRI_DSQ = 1,
LOWPRI_DSQ = 2,
HIGHPRI_WEIGHT = 8668, /* this is what -20 maps to */
};
@@ -41,12 +42,18 @@ const volatile u32 dsp_batch;
const volatile bool highpri_boosting;
const volatile bool print_dsqs_and_events;
const volatile bool print_msgs;
const volatile u64 sub_cgroup_id;
const volatile s32 disallow_tgid;
const volatile bool suppress_dump;
const volatile bool always_enq_immed;
const volatile u32 immed_stress_nth;
u64 nr_highpri_queued;
u32 test_error_cnt;
#define MAX_SUB_SCHEDS 8
u64 sub_sched_cgroup_ids[MAX_SUB_SCHEDS];
UEI_DEFINE(uei);
struct qmap {
@@ -127,7 +134,7 @@ struct {
} cpu_ctx_stor SEC(".maps");
/* Statistics */
u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_dequeued, nr_ddsp_from_enq;
u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_reenqueued_cpu0, nr_dequeued, nr_ddsp_from_enq;
u64 nr_core_sched_execed;
u64 nr_expedited_local, nr_expedited_remote, nr_expedited_lost, nr_expedited_from_timer;
u32 cpuperf_min, cpuperf_avg, cpuperf_max;
@@ -137,8 +144,10 @@ static s32 pick_direct_dispatch_cpu(struct task_struct *p, s32 prev_cpu)
{
s32 cpu;
if (p->nr_cpus_allowed == 1 ||
scx_bpf_test_and_clear_cpu_idle(prev_cpu))
if (!always_enq_immed && p->nr_cpus_allowed == 1)
return prev_cpu;
if (scx_bpf_test_and_clear_cpu_idle(prev_cpu))
return prev_cpu;
cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0);
@@ -168,6 +177,9 @@ s32 BPF_STRUCT_OPS(qmap_select_cpu, struct task_struct *p,
if (!(tctx = lookup_task_ctx(p)))
return -ESRCH;
if (p->scx.weight < 2 && !(p->flags & PF_KTHREAD))
return prev_cpu;
cpu = pick_direct_dispatch_cpu(p, prev_cpu);
if (cpu >= 0) {
@@ -202,8 +214,11 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
void *ring;
s32 cpu;
if (enq_flags & SCX_ENQ_REENQ)
if (enq_flags & SCX_ENQ_REENQ) {
__sync_fetch_and_add(&nr_reenqueued, 1);
if (scx_bpf_task_cpu(p) == 0)
__sync_fetch_and_add(&nr_reenqueued_cpu0, 1);
}
if (p->flags & PF_KTHREAD) {
if (stall_kernel_nth && !(++kernel_cnt % stall_kernel_nth))
@@ -225,6 +240,22 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
*/
tctx->core_sched_seq = core_sched_tail_seqs[idx]++;
/*
* IMMED stress testing: Every immed_stress_nth'th enqueue, dispatch
* directly to prev_cpu's local DSQ even when busy to force dsq->nr > 1
* and exercise the kernel IMMED reenqueue trigger paths.
*/
if (immed_stress_nth && !(enq_flags & SCX_ENQ_REENQ)) {
static u32 immed_stress_cnt;
if (!(++immed_stress_cnt % immed_stress_nth)) {
tctx->force_local = false;
scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | scx_bpf_task_cpu(p),
slice_ns, enq_flags);
return;
}
}
/*
* If qmap_select_cpu() is telling us to or this is the last runnable
* task on the CPU, enqueue locally.
@@ -235,6 +266,13 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)
return;
}
/* see lowpri_timerfn() */
if (__COMPAT_has_generic_reenq() &&
p->scx.weight < 2 && !(p->flags & PF_KTHREAD) && !(enq_flags & SCX_ENQ_REENQ)) {
scx_bpf_dsq_insert(p, LOWPRI_DSQ, slice_ns, enq_flags);
return;
}
/* if select_cpu() wasn't called, try direct dispatch */
if (!__COMPAT_is_enq_cpu_selected(enq_flags) &&
(cpu = pick_direct_dispatch_cpu(p, scx_bpf_task_cpu(p))) >= 0) {
@@ -375,7 +413,7 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
if (dispatch_highpri(false))
return;
if (!nr_highpri_queued && scx_bpf_dsq_move_to_local(SHARED_DSQ))
if (!nr_highpri_queued && scx_bpf_dsq_move_to_local(SHARED_DSQ, 0))
return;
if (dsp_inf_loop_after && nr_dispatched > dsp_inf_loop_after) {
@@ -433,6 +471,46 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
__sync_fetch_and_add(&nr_dispatched, 1);
scx_bpf_dsq_insert(p, SHARED_DSQ, slice_ns, 0);
/*
* scx_qmap uses a global BPF queue that any CPU's
* dispatch can pop from. If this CPU popped a task that
* can't run here, it gets stranded on SHARED_DSQ after
* consume_dispatch_q() skips it. Kick the task's home
* CPU so it drains SHARED_DSQ.
*
* There's a race between the pop and the flush of the
* buffered dsq_insert:
*
* CPU 0 (dispatching) CPU 1 (home, idle)
* ~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
* pop from BPF queue
* dsq_insert(buffered)
* balance:
* SHARED_DSQ empty
* BPF queue empty
* -> goes idle
* flush -> on SHARED
* kick CPU 1
* wakes, drains task
*
* The kick prevents indefinite stalls but a per-CPU
* kthread like ksoftirqd can be briefly stranded when
* its home CPU enters idle with softirq pending,
* triggering:
*
* "NOHZ tick-stop error: local softirq work is pending, handler #N!!!"
*
* from report_idle_softirq(). The kick lands shortly
* after and the home CPU drains the task. This could be
* avoided by e.g. dispatching pinned tasks to local or
* global DSQs, but the current code is left as-is to
* document this class of issue -- other schedulers
* seeing similar warnings can use this as a reference.
*/
if (!bpf_cpumask_test_cpu(cpu, p->cpus_ptr))
scx_bpf_kick_cpu(scx_bpf_task_cpu(p), 0);
bpf_task_release(p);
batch--;
@@ -440,7 +518,7 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
if (!batch || !scx_bpf_dispatch_nr_slots()) {
if (dispatch_highpri(false))
return;
scx_bpf_dsq_move_to_local(SHARED_DSQ);
scx_bpf_dsq_move_to_local(SHARED_DSQ, 0);
return;
}
if (!cpuc->dsp_cnt)
@@ -450,6 +528,12 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
cpuc->dsp_cnt = 0;
}
for (i = 0; i < MAX_SUB_SCHEDS; i++) {
if (sub_sched_cgroup_ids[i] &&
scx_bpf_sub_dispatch(sub_sched_cgroup_ids[i]))
return;
}
/*
* No other tasks. @prev will keep running. Update its core_sched_seq as
* if the task were enqueued and dispatched immediately.
@@ -532,36 +616,11 @@ bool BPF_STRUCT_OPS(qmap_core_sched_before,
return task_qdist(a) > task_qdist(b);
}
SEC("tp_btf/sched_switch")
int BPF_PROG(qmap_sched_switch, bool preempt, struct task_struct *prev,
struct task_struct *next, unsigned long prev_state)
{
if (!__COMPAT_scx_bpf_reenqueue_local_from_anywhere())
return 0;
/*
* If @cpu is taken by a higher priority scheduling class, it is no
* longer available for executing sched_ext tasks. As we don't want the
* tasks in @cpu's local dsq to sit there until @cpu becomes available
* again, re-enqueue them into the global dsq. See %SCX_ENQ_REENQ
* handling in qmap_enqueue().
*/
switch (next->policy) {
case 1: /* SCHED_FIFO */
case 2: /* SCHED_RR */
case 6: /* SCHED_DEADLINE */
scx_bpf_reenqueue_local();
}
return 0;
}
void BPF_STRUCT_OPS(qmap_cpu_release, s32 cpu, struct scx_cpu_release_args *args)
{
/* see qmap_sched_switch() to learn how to do this on newer kernels */
if (!__COMPAT_scx_bpf_reenqueue_local_from_anywhere())
scx_bpf_reenqueue_local();
}
/*
* sched_switch tracepoint and cpu_release handlers are no longer needed.
* With SCX_OPS_ALWAYS_ENQ_IMMED, wakeup_preempt_scx() reenqueues IMMED
* tasks when a higher-priority scheduling class takes the CPU.
*/
s32 BPF_STRUCT_OPS(qmap_init_task, struct task_struct *p,
struct scx_init_task_args *args)
@@ -856,13 +915,35 @@ static int monitor_timerfn(void *map, int *key, struct bpf_timer *timer)
return 0;
}
struct lowpri_timer {
struct bpf_timer timer;
};
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, 1);
__type(key, u32);
__type(value, struct lowpri_timer);
} lowpri_timer SEC(".maps");
/*
* Nice 19 tasks are put into the lowpri DSQ. Every 10ms, reenq is triggered and
* the tasks are transferred to SHARED_DSQ.
*/
static int lowpri_timerfn(void *map, int *key, struct bpf_timer *timer)
{
scx_bpf_dsq_reenq(LOWPRI_DSQ, 0);
bpf_timer_start(timer, LOWPRI_INTV_NS, 0);
return 0;
}
s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)
{
u32 key = 0;
struct bpf_timer *timer;
s32 ret;
if (print_msgs)
if (print_msgs && !sub_cgroup_id)
print_cpus();
ret = scx_bpf_create_dsq(SHARED_DSQ, -1);
@@ -877,14 +958,32 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)
return ret;
}
ret = scx_bpf_create_dsq(LOWPRI_DSQ, -1);
if (ret)
return ret;
timer = bpf_map_lookup_elem(&monitor_timer, &key);
if (!timer)
return -ESRCH;
bpf_timer_init(timer, &monitor_timer, CLOCK_MONOTONIC);
bpf_timer_set_callback(timer, monitor_timerfn);
ret = bpf_timer_start(timer, ONE_SEC_IN_NS, 0);
if (ret)
return ret;
return bpf_timer_start(timer, ONE_SEC_IN_NS, 0);
if (__COMPAT_has_generic_reenq()) {
/* see lowpri_timerfn() */
timer = bpf_map_lookup_elem(&lowpri_timer, &key);
if (!timer)
return -ESRCH;
bpf_timer_init(timer, &lowpri_timer, CLOCK_MONOTONIC);
bpf_timer_set_callback(timer, lowpri_timerfn);
ret = bpf_timer_start(timer, LOWPRI_INTV_NS, 0);
if (ret)
return ret;
}
return 0;
}
void BPF_STRUCT_OPS(qmap_exit, struct scx_exit_info *ei)
@@ -892,6 +991,36 @@ void BPF_STRUCT_OPS(qmap_exit, struct scx_exit_info *ei)
UEI_RECORD(uei, ei);
}
s32 BPF_STRUCT_OPS(qmap_sub_attach, struct scx_sub_attach_args *args)
{
s32 i;
for (i = 0; i < MAX_SUB_SCHEDS; i++) {
if (!sub_sched_cgroup_ids[i]) {
sub_sched_cgroup_ids[i] = args->ops->sub_cgroup_id;
bpf_printk("attaching sub-sched[%d] on %s",
i, args->cgroup_path);
return 0;
}
}
return -ENOSPC;
}
void BPF_STRUCT_OPS(qmap_sub_detach, struct scx_sub_detach_args *args)
{
s32 i;
for (i = 0; i < MAX_SUB_SCHEDS; i++) {
if (sub_sched_cgroup_ids[i] == args->ops->sub_cgroup_id) {
sub_sched_cgroup_ids[i] = 0;
bpf_printk("detaching sub-sched[%d] on %s",
i, args->cgroup_path);
break;
}
}
}
SCX_OPS_DEFINE(qmap_ops,
.select_cpu = (void *)qmap_select_cpu,
.enqueue = (void *)qmap_enqueue,
@@ -899,7 +1028,6 @@ SCX_OPS_DEFINE(qmap_ops,
.dispatch = (void *)qmap_dispatch,
.tick = (void *)qmap_tick,
.core_sched_before = (void *)qmap_core_sched_before,
.cpu_release = (void *)qmap_cpu_release,
.init_task = (void *)qmap_init_task,
.dump = (void *)qmap_dump,
.dump_cpu = (void *)qmap_dump_cpu,
@@ -907,6 +1035,8 @@ SCX_OPS_DEFINE(qmap_ops,
.cgroup_init = (void *)qmap_cgroup_init,
.cgroup_set_weight = (void *)qmap_cgroup_set_weight,
.cgroup_set_bandwidth = (void *)qmap_cgroup_set_bandwidth,
.sub_attach = (void *)qmap_sub_attach,
.sub_detach = (void *)qmap_sub_detach,
.cpu_online = (void *)qmap_cpu_online,
.cpu_offline = (void *)qmap_cpu_offline,
.init = (void *)qmap_init,
tools/sched_ext/scx_qmap.c
+25 -4
View File
@@ -10,6 +10,7 @@
#include <inttypes.h>
#include <signal.h>
#include <libgen.h>
#include <sys/stat.h>
#include <bpf/bpf.h>
#include <scx/common.h>
#include "scx_qmap.bpf.skel.h"
@@ -20,7 +21,7 @@ const char help_fmt[] =
"See the top-level comment in .bpf.c for more details.\n"
"\n"
"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n"
" [-P] [-M] [-d PID] [-D LEN] [-p] [-v]\n"
" [-P] [-M] [-H] [-d PID] [-D LEN] [-S] [-p] [-I] [-F COUNT] [-v]\n"
"\n"
" -s SLICE_US Override slice duration\n"
" -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
@@ -35,6 +36,8 @@ const char help_fmt[] =
" -D LEN Set scx_exit_info.dump buffer length\n"
" -S Suppress qmap-specific debug dump\n"
" -p Switch only tasks on SCHED_EXT policy instead of all\n"
" -I Turn on SCX_OPS_ALWAYS_ENQ_IMMED\n"
" -F COUNT IMMED stress: force every COUNT'th enqueue to a busy local DSQ (use with -I)\n"
" -v Print libbpf debug messages\n"
" -h Display this help and exit\n";
@@ -67,7 +70,7 @@ int main(int argc, char **argv)
skel->rodata->slice_ns = __COMPAT_ENUM_OR_ZERO("scx_public_consts", "SCX_SLICE_DFL");
while ((opt = getopt(argc, argv, "s:e:t:T:l:b:PMHd:D:Spvh")) != -1) {
while ((opt = getopt(argc, argv, "s:e:t:T:l:b:PMHc:d:D:SpIF:vh")) != -1) {
switch (opt) {
case 's':
skel->rodata->slice_ns = strtoull(optarg, NULL, 0) * 1000;
@@ -96,6 +99,16 @@ int main(int argc, char **argv)
case 'H':
skel->rodata->highpri_boosting = true;
break;
case 'c': {
struct stat st;
if (stat(optarg, &st) < 0) {
perror("stat");
return 1;
}
skel->struct_ops.qmap_ops->sub_cgroup_id = st.st_ino;
skel->rodata->sub_cgroup_id = st.st_ino;
break;
}
case 'd':
skel->rodata->disallow_tgid = strtol(optarg, NULL, 0);
if (skel->rodata->disallow_tgid < 0)
@@ -110,6 +123,13 @@ int main(int argc, char **argv)
case 'p':
skel->struct_ops.qmap_ops->flags |= SCX_OPS_SWITCH_PARTIAL;
break;
case 'I':
skel->rodata->always_enq_immed = true;
skel->struct_ops.qmap_ops->flags |= SCX_OPS_ALWAYS_ENQ_IMMED;
break;
case 'F':
skel->rodata->immed_stress_nth = strtoul(optarg, NULL, 0);
break;
case 'v':
verbose = true;
break;
@@ -126,9 +146,10 @@ int main(int argc, char **argv)
long nr_enqueued = skel->bss->nr_enqueued;
long nr_dispatched = skel->bss->nr_dispatched;
printf("stats : enq=%lu dsp=%lu delta=%ld reenq=%"PRIu64" deq=%"PRIu64" core=%"PRIu64" enq_ddsp=%"PRIu64"\n",
printf("stats : enq=%lu dsp=%lu delta=%ld reenq/cpu0=%"PRIu64"/%"PRIu64" deq=%"PRIu64" core=%"PRIu64" enq_ddsp=%"PRIu64"\n",
nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched,
skel->bss->nr_reenqueued, skel->bss->nr_dequeued,
skel->bss->nr_reenqueued, skel->bss->nr_reenqueued_cpu0,
skel->bss->nr_dequeued,
skel->bss->nr_core_sched_execed,
skel->bss->nr_ddsp_from_enq);
printf(" exp_local=%"PRIu64" exp_remote=%"PRIu64" exp_timer=%"PRIu64" exp_lost=%"PRIu64"\n",
tools/sched_ext/scx_sdt.bpf.c
+3 -2
View File
@@ -317,7 +317,8 @@ int scx_alloc_free_idx(struct scx_allocator *alloc, __u64 idx)
};
/* Zero out one word at a time. */
for (i = zero; i < alloc->pool.elem_size / 8 && can_loop; i++) {
for (i = zero; i < (alloc->pool.elem_size - sizeof(struct sdt_data)) / 8
&& can_loop; i++) {
data->payload[i] = 0;
}
}
@@ -643,7 +644,7 @@ void BPF_STRUCT_OPS(sdt_enqueue, struct task_struct *p, u64 enq_flags)
void BPF_STRUCT_OPS(sdt_dispatch, s32 cpu, struct task_struct *prev)
{
scx_bpf_dsq_move_to_local(SHARED_DSQ);
scx_bpf_dsq_move_to_local(SHARED_DSQ, 0);
}
s32 BPF_STRUCT_OPS_SLEEPABLE(sdt_init_task, struct task_struct *p,
tools/sched_ext/scx_sdt.c
+1 -1
View File
@@ -20,7 +20,7 @@ const char help_fmt[] =
"\n"
"Modified version of scx_simple that demonstrates arena-based data structures.\n"
"\n"
"Usage: %s [-f] [-v]\n"
"Usage: %s [-v]\n"
"\n"
" -v Print libbpf debug messages\n"
" -h Display this help and exit\n";
tools/sched_ext/scx_simple.bpf.c
+5 -3
View File
@@ -89,7 +89,7 @@ void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
void BPF_STRUCT_OPS(simple_dispatch, s32 cpu, struct task_struct *prev)
{
scx_bpf_dsq_move_to_local(SHARED_DSQ);
scx_bpf_dsq_move_to_local(SHARED_DSQ, 0);
}
void BPF_STRUCT_OPS(simple_running, struct task_struct *p)
@@ -121,12 +121,14 @@ void BPF_STRUCT_OPS(simple_stopping, struct task_struct *p, bool runnable)
* too much, determine the execution time by taking explicit timestamps
* instead of depending on @p->scx.slice.
*/
p->scx.dsq_vtime += (SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
u64 delta = scale_by_task_weight_inverse(p, SCX_SLICE_DFL - p->scx.slice);
scx_bpf_task_set_dsq_vtime(p, p->scx.dsq_vtime + delta);
}
void BPF_STRUCT_OPS(simple_enable, struct task_struct *p)
{
p->scx.dsq_vtime = vtime_now;
scx_bpf_task_set_dsq_vtime(p, vtime_now);
}
s32 BPF_STRUCT_OPS_SLEEPABLE(simple_init)
tools/sched_ext/scx_userland.c
+1 -1
View File
@@ -38,7 +38,7 @@ const char help_fmt[] =
"\n"
"Try to reduce `sysctl kernel.pid_max` if this program triggers OOMs.\n"
"\n"
"Usage: %s [-b BATCH]\n"
"Usage: %s [-b BATCH] [-v]\n"
"\n"
" -b BATCH The number of tasks to batch when dispatching (default: 8)\n"
" -v Print libbpf debug messages\n"
tools/testing/selftests/sched_ext/Makefile
+1
View File
@@ -163,6 +163,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs
auto-test-targets := \
create_dsq \
dequeue \
enq_last_no_enq_fails \
ddsp_bogus_dsq_fail \
ddsp_vtimelocal_fail \
tools/testing/selftests/sched_ext/dequeue.bpf.c
+389
View File
@@ -0,0 +1,389 @@
// SPDX-License-Identifier: GPL-2.0
/*
* A scheduler that validates ops.dequeue() is called correctly:
* - Tasks dispatched to terminal DSQs (local, global) bypass the BPF
* scheduler entirely: no ops.dequeue() should be called
* - Tasks dispatched to user DSQs from ops.enqueue() enter BPF custody:
* ops.dequeue() must be called when they leave custody
* - Every ops.enqueue() dispatch to non-terminal DSQs is followed by
* exactly one ops.dequeue() (validate 1:1 pairing and state machine)
*
* Copyright (c) 2026 NVIDIA Corporation.
*/
#include <scx/common.bpf.h>
#define SHARED_DSQ 0
/*
* BPF internal queue.
*
* Tasks are stored here and consumed from ops.dispatch(), validating that
* tasks on BPF internal structures still get ops.dequeue() when they
* leave.
*/
struct {
__uint(type, BPF_MAP_TYPE_QUEUE);
__uint(max_entries, 32768);
__type(value, s32);
} global_queue SEC(".maps");
char _license[] SEC("license") = "GPL";
UEI_DEFINE(uei);
/*
* Counters to track the lifecycle of tasks:
* - enqueue_cnt: Number of times ops.enqueue() was called
* - dequeue_cnt: Number of times ops.dequeue() was called (any type)
* - dispatch_dequeue_cnt: Number of regular dispatch dequeues (no flag)
* - change_dequeue_cnt: Number of property change dequeues
* - bpf_queue_full: Number of times the BPF internal queue was full
*/
u64 enqueue_cnt, dequeue_cnt, dispatch_dequeue_cnt, change_dequeue_cnt, bpf_queue_full;
/*
* Test scenarios:
* 0) Dispatch to local DSQ from ops.select_cpu() (terminal DSQ, bypasses BPF
* scheduler, no dequeue callbacks)
* 1) Dispatch to global DSQ from ops.select_cpu() (terminal DSQ, bypasses BPF
* scheduler, no dequeue callbacks)
* 2) Dispatch to shared user DSQ from ops.select_cpu() (enters BPF scheduler,
* dequeue callbacks expected)
* 3) Dispatch to local DSQ from ops.enqueue() (terminal DSQ, bypasses BPF
* scheduler, no dequeue callbacks)
* 4) Dispatch to global DSQ from ops.enqueue() (terminal DSQ, bypasses BPF
* scheduler, no dequeue callbacks)
* 5) Dispatch to shared user DSQ from ops.enqueue() (enters BPF scheduler,
* dequeue callbacks expected)
* 6) BPF internal queue from ops.enqueue(): store task PIDs in ops.enqueue(),
* consume in ops.dispatch() and dispatch to local DSQ (validates dequeue
* for tasks stored in internal BPF data structures)
*/
u32 test_scenario;
/*
* Per-task state to track lifecycle and validate workflow semantics.
* State transitions:
* NONE -> ENQUEUED (on enqueue)
* NONE -> DISPATCHED (on direct dispatch to terminal DSQ)
* ENQUEUED -> DISPATCHED (on dispatch dequeue)
* DISPATCHED -> NONE (on property change dequeue or re-enqueue)
* ENQUEUED -> NONE (on property change dequeue before dispatch)
*/
enum task_state {
TASK_NONE = 0,
TASK_ENQUEUED,
TASK_DISPATCHED,
};
struct task_ctx {
enum task_state state; /* Current state in the workflow */
u64 enqueue_seq; /* Sequence number for debugging */
};
struct {
__uint(type, BPF_MAP_TYPE_TASK_STORAGE);
__uint(map_flags, BPF_F_NO_PREALLOC);
__type(key, int);
__type(value, struct task_ctx);
} task_ctx_stor SEC(".maps");
static struct task_ctx *try_lookup_task_ctx(struct task_struct *p)
{
return bpf_task_storage_get(&task_ctx_stor, p, 0, 0);
}
s32 BPF_STRUCT_OPS(dequeue_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
{
struct task_ctx *tctx;
tctx = try_lookup_task_ctx(p);
if (!tctx)
return prev_cpu;
switch (test_scenario) {
case 0:
/*
* Direct dispatch to the local DSQ.
*
* Task bypasses BPF scheduler entirely: no enqueue
* tracking, no ops.dequeue() callbacks.
*/
scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
tctx->state = TASK_DISPATCHED;
break;
case 1:
/*
* Direct dispatch to the global DSQ.
*
* Task bypasses BPF scheduler entirely: no enqueue
* tracking, no ops.dequeue() callbacks.
*/
scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, 0);
tctx->state = TASK_DISPATCHED;
break;
case 2:
/*
* Dispatch to a shared user DSQ.
*
* Task enters BPF scheduler management: track
* enqueue/dequeue lifecycle and validate state
* transitions.
*/
if (tctx->state == TASK_ENQUEUED)
scx_bpf_error("%d (%s): enqueue while in ENQUEUED state seq=%llu",
p->pid, p->comm, tctx->enqueue_seq);
scx_bpf_dsq_insert(p, SHARED_DSQ, SCX_SLICE_DFL, 0);
__sync_fetch_and_add(&enqueue_cnt, 1);
tctx->state = TASK_ENQUEUED;
tctx->enqueue_seq++;
break;
}
return prev_cpu;
}
void BPF_STRUCT_OPS(dequeue_enqueue, struct task_struct *p, u64 enq_flags)
{
struct task_ctx *tctx;
s32 pid = p->pid;
tctx = try_lookup_task_ctx(p);
if (!tctx)
return;
switch (test_scenario) {
case 3:
/*
* Direct dispatch to the local DSQ.
*
* Task bypasses BPF scheduler entirely: no enqueue
* tracking, no ops.dequeue() callbacks.
*/
scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, enq_flags);
tctx->state = TASK_DISPATCHED;
break;
case 4:
/*
* Direct dispatch to the global DSQ.
*
* Task bypasses BPF scheduler entirely: no enqueue
* tracking, no ops.dequeue() callbacks.
*/
scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
tctx->state = TASK_DISPATCHED;
break;
case 5:
/*
* Dispatch to shared user DSQ.
*
* Task enters BPF scheduler management: track
* enqueue/dequeue lifecycle and validate state
* transitions.
*/
if (tctx->state == TASK_ENQUEUED)
scx_bpf_error("%d (%s): enqueue while in ENQUEUED state seq=%llu",
p->pid, p->comm, tctx->enqueue_seq);
scx_bpf_dsq_insert(p, SHARED_DSQ, SCX_SLICE_DFL, enq_flags);
__sync_fetch_and_add(&enqueue_cnt, 1);
tctx->state = TASK_ENQUEUED;
tctx->enqueue_seq++;
break;
case 6:
/*
* Store task in BPF internal queue.
*
* Task enters BPF scheduler management: track
* enqueue/dequeue lifecycle and validate state
* transitions.
*/
if (tctx->state == TASK_ENQUEUED)
scx_bpf_error("%d (%s): enqueue while in ENQUEUED state seq=%llu",
p->pid, p->comm, tctx->enqueue_seq);
if (bpf_map_push_elem(&global_queue, &pid, 0)) {
scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
__sync_fetch_and_add(&bpf_queue_full, 1);
tctx->state = TASK_DISPATCHED;
} else {
__sync_fetch_and_add(&enqueue_cnt, 1);
tctx->state = TASK_ENQUEUED;
tctx->enqueue_seq++;
}
break;
default:
/* For all other scenarios, dispatch to the global DSQ */
scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
tctx->state = TASK_DISPATCHED;
break;
}
scx_bpf_kick_cpu(scx_bpf_task_cpu(p), SCX_KICK_IDLE);
}
void BPF_STRUCT_OPS(dequeue_dequeue, struct task_struct *p, u64 deq_flags)
{
struct task_ctx *tctx;
__sync_fetch_and_add(&dequeue_cnt, 1);
tctx = try_lookup_task_ctx(p);
if (!tctx)
return;
/*
* For scenarios 0, 1, 3, and 4 (terminal DSQs: local and global),
* ops.dequeue() should never be called because tasks bypass the
* BPF scheduler entirely. If we get here, it's a kernel bug.
*/
if (test_scenario == 0 || test_scenario == 3) {
scx_bpf_error("%d (%s): dequeue called for local DSQ scenario",
p->pid, p->comm);
return;
}
if (test_scenario == 1 || test_scenario == 4) {
scx_bpf_error("%d (%s): dequeue called for global DSQ scenario",
p->pid, p->comm);
return;
}
if (deq_flags & SCX_DEQ_SCHED_CHANGE) {
/*
* Property change interrupting the workflow. Valid from
* both ENQUEUED and DISPATCHED states. Transitions task
* back to NONE state.
*/
__sync_fetch_and_add(&change_dequeue_cnt, 1);
/* Validate state transition */
if (tctx->state != TASK_ENQUEUED && tctx->state != TASK_DISPATCHED)
scx_bpf_error("%d (%s): invalid property change dequeue state=%d seq=%llu",
p->pid, p->comm, tctx->state, tctx->enqueue_seq);
/*
* Transition back to NONE: task outside scheduler control.
*
* Scenario 6: dispatch() checks tctx->state after popping a
* PID, if the task is in state NONE, it was dequeued by
* property change and must not be dispatched (this
* prevents "target CPU not allowed").
*/
tctx->state = TASK_NONE;
} else {
/*
* Regular dispatch dequeue: kernel is moving the task from
* BPF custody to a terminal DSQ. Normally we come from
* ENQUEUED state. We can also see TASK_NONE if the task
* was dequeued by property change (SCX_DEQ_SCHED_CHANGE)
* while it was already on a DSQ (dispatched but not yet
* consumed); in that case we just leave state as NONE.
*/
__sync_fetch_and_add(&dispatch_dequeue_cnt, 1);
/*
* Must be ENQUEUED (normal path) or NONE (already dequeued
* by property change while on a DSQ).
*/
if (tctx->state != TASK_ENQUEUED && tctx->state != TASK_NONE)
scx_bpf_error("%d (%s): dispatch dequeue from state %d seq=%llu",
p->pid, p->comm, tctx->state, tctx->enqueue_seq);
if (tctx->state == TASK_ENQUEUED)
tctx->state = TASK_DISPATCHED;
/* NONE: leave as-is, task was already property-change dequeued */
}
}
void BPF_STRUCT_OPS(dequeue_dispatch, s32 cpu, struct task_struct *prev)
{
if (test_scenario == 6) {
struct task_ctx *tctx;
struct task_struct *p;
s32 pid;
if (bpf_map_pop_elem(&global_queue, &pid))
return;
p = bpf_task_from_pid(pid);
if (!p)
return;
/*
* If the task was dequeued by property change
* (ops.dequeue() set tctx->state = TASK_NONE), skip
* dispatch.
*/
tctx = try_lookup_task_ctx(p);
if (!tctx || tctx->state == TASK_NONE) {
bpf_task_release(p);
return;
}
/*
* Dispatch to this CPU's local DSQ if allowed, otherwise
* fallback to the global DSQ.
*/
if (bpf_cpumask_test_cpu(cpu, p->cpus_ptr))
scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cpu, SCX_SLICE_DFL, 0);
else
scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, 0);
bpf_task_release(p);
} else {
scx_bpf_dsq_move_to_local(SHARED_DSQ, 0);
}
}
s32 BPF_STRUCT_OPS(dequeue_init_task, struct task_struct *p,
struct scx_init_task_args *args)
{
struct task_ctx *tctx;
tctx = bpf_task_storage_get(&task_ctx_stor, p, 0,
BPF_LOCAL_STORAGE_GET_F_CREATE);
if (!tctx)
return -ENOMEM;
return 0;
}
s32 BPF_STRUCT_OPS_SLEEPABLE(dequeue_init)
{
s32 ret;
ret = scx_bpf_create_dsq(SHARED_DSQ, -1);
if (ret)
return ret;
return 0;
}
void BPF_STRUCT_OPS(dequeue_exit, struct scx_exit_info *ei)
{
UEI_RECORD(uei, ei);
}
SEC(".struct_ops.link")
struct sched_ext_ops dequeue_ops = {
.select_cpu = (void *)dequeue_select_cpu,
.enqueue = (void *)dequeue_enqueue,
.dequeue = (void *)dequeue_dequeue,
.dispatch = (void *)dequeue_dispatch,
.init_task = (void *)dequeue_init_task,
.init = (void *)dequeue_init,
.exit = (void *)dequeue_exit,
.flags = SCX_OPS_ENQ_LAST,
.name = "dequeue_test",
};
tools/testing/selftests/sched_ext/dequeue.c
+274
View File
@@ -0,0 +1,274 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (c) 2025 NVIDIA Corporation.
*/
#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
#include <time.h>
#include <bpf/bpf.h>
#include <scx/common.h>
#include <sys/wait.h>
#include <sched.h>
#include <pthread.h>
#include "scx_test.h"
#include "dequeue.bpf.skel.h"
#define NUM_WORKERS 8
#define AFFINITY_HAMMER_MS 500
/*
* Worker function that creates enqueue/dequeue events via CPU work and
* sleep.
*/
static void worker_fn(int id)
{
int i;
volatile int sum = 0;
for (i = 0; i < 1000; i++) {
volatile int j;
/* Do some work to trigger scheduling events */
for (j = 0; j < 10000; j++)
sum += j;
/* Sleep to trigger dequeue */
usleep(1000 + (id * 100));
}
exit(0);
}
/*
* This thread changes workers' affinity from outside so that some changes
* hit tasks while they are still in the scheduler's queue and trigger
* property-change dequeues.
*/
static void *affinity_hammer_fn(void *arg)
{
pid_t *pids = arg;
cpu_set_t cpuset;
int i = 0, n = NUM_WORKERS;
struct timespec start, now;
clock_gettime(CLOCK_MONOTONIC, &start);
while (1) {
int w = i % n;
int cpu = (i / n) % 4;
CPU_ZERO(&cpuset);
CPU_SET(cpu, &cpuset);
sched_setaffinity(pids[w], sizeof(cpuset), &cpuset);
i++;
/* Check elapsed time every 256 iterations to limit gettime cost */
if ((i & 255) == 0) {
long long elapsed_ms;
clock_gettime(CLOCK_MONOTONIC, &now);
elapsed_ms = (now.tv_sec - start.tv_sec) * 1000LL +
(now.tv_nsec - start.tv_nsec) / 1000000;
if (elapsed_ms >= AFFINITY_HAMMER_MS)
break;
}
}
return NULL;
}
static enum scx_test_status run_scenario(struct dequeue *skel, u32 scenario,
const char *scenario_name)
{
struct bpf_link *link;
pid_t pids[NUM_WORKERS];
pthread_t hammer;
int i, status;
u64 enq_start, deq_start,
dispatch_deq_start, change_deq_start, bpf_queue_full_start;
u64 enq_delta, deq_delta,
dispatch_deq_delta, change_deq_delta, bpf_queue_full_delta;
/* Set the test scenario */
skel->bss->test_scenario = scenario;
/* Record starting counts */
enq_start = skel->bss->enqueue_cnt;
deq_start = skel->bss->dequeue_cnt;
dispatch_deq_start = skel->bss->dispatch_dequeue_cnt;
change_deq_start = skel->bss->change_dequeue_cnt;
bpf_queue_full_start = skel->bss->bpf_queue_full;
link = bpf_map__attach_struct_ops(skel->maps.dequeue_ops);
SCX_FAIL_IF(!link, "Failed to attach struct_ops for scenario %s", scenario_name);
/* Fork worker processes to generate enqueue/dequeue events */
for (i = 0; i < NUM_WORKERS; i++) {
pids[i] = fork();
SCX_FAIL_IF(pids[i] < 0, "Failed to fork worker %d", i);
if (pids[i] == 0) {
worker_fn(i);
/* Should not reach here */
exit(1);
}
}
/*
* Run an "affinity hammer" so that some property changes hit tasks
* while they are still in BPF custody (e.g., in user DSQ or BPF
* queue), triggering SCX_DEQ_SCHED_CHANGE dequeues.
*/
SCX_FAIL_IF(pthread_create(&hammer, NULL, affinity_hammer_fn, pids) != 0,
"Failed to create affinity hammer thread");
pthread_join(hammer, NULL);
/* Wait for all workers to complete */
for (i = 0; i < NUM_WORKERS; i++) {
SCX_FAIL_IF(waitpid(pids[i], &status, 0) != pids[i],
"Failed to wait for worker %d", i);
SCX_FAIL_IF(status != 0, "Worker %d exited with status %d", i, status);
}
bpf_link__destroy(link);
SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_UNREG));
/* Calculate deltas */
enq_delta = skel->bss->enqueue_cnt - enq_start;
deq_delta = skel->bss->dequeue_cnt - deq_start;
dispatch_deq_delta = skel->bss->dispatch_dequeue_cnt - dispatch_deq_start;
change_deq_delta = skel->bss->change_dequeue_cnt - change_deq_start;
bpf_queue_full_delta = skel->bss->bpf_queue_full - bpf_queue_full_start;
printf("%s:\n", scenario_name);
printf(" enqueues: %lu\n", (unsigned long)enq_delta);
printf(" dequeues: %lu (dispatch: %lu, property_change: %lu)\n",
(unsigned long)deq_delta,
(unsigned long)dispatch_deq_delta,
(unsigned long)change_deq_delta);
printf(" BPF queue full: %lu\n", (unsigned long)bpf_queue_full_delta);
/*
* Validate enqueue/dequeue lifecycle tracking.
*
* For scenarios 0, 1, 3, 4 (local and global DSQs from
* ops.select_cpu() and ops.enqueue()), both enqueues and dequeues
* should be 0 because tasks bypass the BPF scheduler entirely:
* tasks never enter BPF scheduler's custody.
*
* For scenarios 2, 5, 6 (user DSQ or BPF internal queue) we expect
* both enqueues and dequeues.
*
* The BPF code does strict state machine validation with
* scx_bpf_error() to ensure the workflow semantics are correct.
*
* If we reach this point without errors, the semantics are
* validated correctly.
*/
if (scenario == 0 || scenario == 1 ||
scenario == 3 || scenario == 4) {
/* Tasks bypass BPF scheduler completely */
SCX_EQ(enq_delta, 0);
SCX_EQ(deq_delta, 0);
SCX_EQ(dispatch_deq_delta, 0);
SCX_EQ(change_deq_delta, 0);
} else {
/*
* User DSQ from ops.enqueue() or ops.select_cpu(): tasks
* enter BPF scheduler's custody.
*
* Also validate 1:1 enqueue/dequeue pairing.
*/
SCX_GT(enq_delta, 0);
SCX_GT(deq_delta, 0);
SCX_EQ(enq_delta, deq_delta);
}
return SCX_TEST_PASS;
}
static enum scx_test_status setup(void **ctx)
{
struct dequeue *skel;
skel = dequeue__open();
SCX_FAIL_IF(!skel, "Failed to open skel");
SCX_ENUM_INIT(skel);
SCX_FAIL_IF(dequeue__load(skel), "Failed to load skel");
*ctx = skel;
return SCX_TEST_PASS;
}
static enum scx_test_status run(void *ctx)
{
struct dequeue *skel = ctx;
enum scx_test_status status;
status = run_scenario(skel, 0, "Scenario 0: Local DSQ from ops.select_cpu()");
if (status != SCX_TEST_PASS)
return status;
status = run_scenario(skel, 1, "Scenario 1: Global DSQ from ops.select_cpu()");
if (status != SCX_TEST_PASS)
return status;
status = run_scenario(skel, 2, "Scenario 2: User DSQ from ops.select_cpu()");
if (status != SCX_TEST_PASS)
return status;
status = run_scenario(skel, 3, "Scenario 3: Local DSQ from ops.enqueue()");
if (status != SCX_TEST_PASS)
return status;
status = run_scenario(skel, 4, "Scenario 4: Global DSQ from ops.enqueue()");
if (status != SCX_TEST_PASS)
return status;
status = run_scenario(skel, 5, "Scenario 5: User DSQ from ops.enqueue()");
if (status != SCX_TEST_PASS)
return status;
status = run_scenario(skel, 6, "Scenario 6: BPF queue from ops.enqueue()");
if (status != SCX_TEST_PASS)
return status;
printf("\n=== Summary ===\n");
printf("Total enqueues: %lu\n", (unsigned long)skel->bss->enqueue_cnt);
printf("Total dequeues: %lu\n", (unsigned long)skel->bss->dequeue_cnt);
printf(" Dispatch dequeues: %lu (no flag, normal workflow)\n",
(unsigned long)skel->bss->dispatch_dequeue_cnt);
printf(" Property change dequeues: %lu (SCX_DEQ_SCHED_CHANGE flag)\n",
(unsigned long)skel->bss->change_dequeue_cnt);
printf(" BPF queue full: %lu\n",
(unsigned long)skel->bss->bpf_queue_full);
printf("\nAll scenarios passed - no state machine violations detected\n");
printf("-> Validated: Local DSQ dispatch bypasses BPF scheduler\n");
printf("-> Validated: Global DSQ dispatch bypasses BPF scheduler\n");
printf("-> Validated: User DSQ dispatch triggers ops.dequeue() callbacks\n");
printf("-> Validated: Dispatch dequeues have no flags (normal workflow)\n");
printf("-> Validated: Property change dequeues have SCX_DEQ_SCHED_CHANGE flag\n");
printf("-> Validated: No duplicate enqueues or invalid state transitions\n");
return SCX_TEST_PASS;
}
static void cleanup(void *ctx)
{
struct dequeue *skel = ctx;
dequeue__destroy(skel);
}
struct scx_test dequeue_test = {
.name = "dequeue",
.description = "Verify ops.dequeue() semantics",
.setup = setup,
.run = run,
.cleanup = cleanup,
};
REGISTER_SCX_TEST(&dequeue_test)
tools/testing/selftests/sched_ext/exit.bpf.c
+1 -1
View File
@@ -41,7 +41,7 @@ void BPF_STRUCT_OPS(exit_dispatch, s32 cpu, struct task_struct *p)
if (exit_point == EXIT_DISPATCH)
EXIT_CLEANLY();
scx_bpf_dsq_move_to_local(DSQ_ID);
scx_bpf_dsq_move_to_local(DSQ_ID, 0);
}
void BPF_STRUCT_OPS(exit_enable, struct task_struct *p)
tools/testing/selftests/sched_ext/exit.c
+1 -1
View File
@@ -33,7 +33,7 @@ static enum scx_test_status run(void *ctx)
skel = exit__open();
SCX_ENUM_INIT(skel);
skel->rodata->exit_point = tc;
exit__load(skel);
SCX_FAIL_IF(exit__load(skel), "Failed to load skel");
link = bpf_map__attach_struct_ops(skel->maps.exit_ops);
if (!link) {
SCX_ERR("Failed to attach scheduler");
tools/testing/selftests/sched_ext/exit_test.h
+1 -1
View File
@@ -17,4 +17,4 @@ enum exit_test_case {
NUM_EXITS,
};
#endif // # __EXIT_TEST_H__
#endif // __EXIT_TEST_H__
tools/testing/selftests/sched_ext/maximal.bpf.c
+7 -10
View File
@@ -30,7 +30,7 @@ void BPF_STRUCT_OPS(maximal_dequeue, struct task_struct *p, u64 deq_flags)
void BPF_STRUCT_OPS(maximal_dispatch, s32 cpu, struct task_struct *prev)
{
scx_bpf_dsq_move_to_local(DSQ_ID);
scx_bpf_dsq_move_to_local(DSQ_ID, 0);
}
void BPF_STRUCT_OPS(maximal_runnable, struct task_struct *p, u64 enq_flags)
@@ -67,13 +67,12 @@ void BPF_STRUCT_OPS(maximal_set_cpumask, struct task_struct *p,
void BPF_STRUCT_OPS(maximal_update_idle, s32 cpu, bool idle)
{}
void BPF_STRUCT_OPS(maximal_cpu_acquire, s32 cpu,
struct scx_cpu_acquire_args *args)
{}
void BPF_STRUCT_OPS(maximal_cpu_release, s32 cpu,
struct scx_cpu_release_args *args)
{}
SEC("tp_btf/sched_switch")
int BPF_PROG(maximal_sched_switch, bool preempt, struct task_struct *prev,
struct task_struct *next, unsigned int prev_state)
{
return 0;
}
void BPF_STRUCT_OPS(maximal_cpu_online, s32 cpu)
{}
@@ -150,8 +149,6 @@ struct sched_ext_ops maximal_ops = {
.set_weight = (void *) maximal_set_weight,
.set_cpumask = (void *) maximal_set_cpumask,
.update_idle = (void *) maximal_update_idle,
.cpu_acquire = (void *) maximal_cpu_acquire,
.cpu_release = (void *) maximal_cpu_release,
.cpu_online = (void *) maximal_cpu_online,
.cpu_offline = (void *) maximal_cpu_offline,
.init_task = (void *) maximal_init_task,
tools/testing/selftests/sched_ext/maximal.c
+3
View File
@@ -19,6 +19,9 @@ static enum scx_test_status setup(void **ctx)
SCX_ENUM_INIT(skel);
SCX_FAIL_IF(maximal__load(skel), "Failed to load skel");
bpf_map__set_autoattach(skel->maps.maximal_ops, false);
SCX_FAIL_IF(maximal__attach(skel), "Failed to attach skel");
*ctx = skel;
return SCX_TEST_PASS;
tools/testing/selftests/sched_ext/numa.bpf.c
+1 -1
View File
@@ -68,7 +68,7 @@ void BPF_STRUCT_OPS(numa_dispatch, s32 cpu, struct task_struct *prev)
{
int node = __COMPAT_scx_bpf_cpu_node(cpu);
scx_bpf_dsq_move_to_local(node);
scx_bpf_dsq_move_to_local(node, 0);
}
s32 BPF_STRUCT_OPS_SLEEPABLE(numa_init)
tools/testing/selftests/sched_ext/peek_dsq.bpf.c
+5 -5
View File
@@ -95,7 +95,7 @@ static int scan_dsq_pool(void)
record_peek_result(task->pid);
/* Try to move this task to local */
if (!moved && scx_bpf_dsq_move_to_local(dsq_id) == 0) {
if (!moved && scx_bpf_dsq_move_to_local(dsq_id, 0) == 0) {
moved = 1;
break;
}
@@ -156,19 +156,19 @@ void BPF_STRUCT_OPS(peek_dsq_dispatch, s32 cpu, struct task_struct *prev)
dsq_peek_result2_pid = peek_result ? peek_result->pid : -1;
/* Now consume the task since we've peeked at it */
scx_bpf_dsq_move_to_local(test_dsq_id);
scx_bpf_dsq_move_to_local(test_dsq_id, 0);
/* Mark phase 1 as complete */
phase1_complete = 1;
bpf_printk("Phase 1 complete, starting phase 2 stress testing");
} else if (!phase1_complete) {
/* Still in phase 1, use real DSQ */
scx_bpf_dsq_move_to_local(real_dsq_id);
scx_bpf_dsq_move_to_local(real_dsq_id, 0);
} else {
/* Phase 2: Scan all DSQs in the pool and try to move a task */
if (!scan_dsq_pool()) {
/* No tasks found in DSQ pool, fall back to real DSQ */
scx_bpf_dsq_move_to_local(real_dsq_id);
scx_bpf_dsq_move_to_local(real_dsq_id, 0);
}
}
}
@@ -197,7 +197,7 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(peek_dsq_init)
}
err = scx_bpf_create_dsq(real_dsq_id, -1);
if (err) {
scx_bpf_error("Failed to create DSQ %d: %d", test_dsq_id, err);
scx_bpf_error("Failed to create DSQ %d: %d", real_dsq_id, err);
return err;
}
tools/testing/selftests/sched_ext/reload_loop.c
+3
View File
@@ -23,6 +23,9 @@ static enum scx_test_status setup(void **ctx)
SCX_ENUM_INIT(skel);
SCX_FAIL_IF(maximal__load(skel), "Failed to load skel");
bpf_map__set_autoattach(skel->maps.maximal_ops, false);
SCX_FAIL_IF(maximal__attach(skel), "Failed to attach skel");
return SCX_TEST_PASS;
}
tools/testing/selftests/sched_ext/rt_stall.c
+5
View File
@@ -119,6 +119,11 @@ static enum scx_test_status setup(void **ctx)
{
struct rt_stall *skel;
if (!__COMPAT_struct_has_field("rq", "ext_server")) {
fprintf(stderr, "SKIP: ext DL server not supported\n");
return SCX_TEST_SKIP;
}
skel = rt_stall__open();
SCX_FAIL_IF(!skel, "Failed to open");
SCX_ENUM_INIT(skel);
tools/testing/selftests/sched_ext/runner.c
+36 -4
View File
@@ -18,7 +18,7 @@ const char help_fmt[] =
"It's required for the testcases to be serial, as only a single host-wide sched_ext\n"
"scheduler may be loaded at any given time."
"\n"
"Usage: %s [-t TEST] [-h]\n"
"Usage: %s [-t TEST] [-s] [-l] [-q]\n"
"\n"
" -t TEST Only run tests whose name includes this string\n"
" -s Include print output for skipped tests\n"
@@ -133,6 +133,8 @@ static bool test_valid(const struct scx_test *test)
int main(int argc, char **argv)
{
const char *filter = NULL;
const char *failed_tests[MAX_SCX_TESTS];
const char *skipped_tests[MAX_SCX_TESTS];
unsigned testnum = 0, i;
unsigned passed = 0, skipped = 0, failed = 0;
int opt;
@@ -162,6 +164,26 @@ int main(int argc, char **argv)
}
}
if (optind < argc) {
fprintf(stderr, "Unexpected argument '%s'. Use -t to filter tests.\n",
argv[optind]);
return 1;
}
if (filter) {
for (i = 0; i < __scx_num_tests; i++) {
if (!should_skip_test(&__scx_tests[i], filter))
break;
}
if (i == __scx_num_tests) {
fprintf(stderr, "No tests matched filter '%s'\n", filter);
fprintf(stderr, "Available tests (use -l to list):\n");
for (i = 0; i < __scx_num_tests; i++)
fprintf(stderr, " %s\n", __scx_tests[i].name);
return 1;
}
}
for (i = 0; i < __scx_num_tests; i++) {
enum scx_test_status status;
struct scx_test *test = &__scx_tests[i];
@@ -198,10 +220,10 @@ int main(int argc, char **argv)
passed++;
break;
case SCX_TEST_SKIP:
skipped++;
skipped_tests[skipped++] = test->name;
break;
case SCX_TEST_FAIL:
failed++;
failed_tests[failed++] = test->name;
break;
}
}
@@ -210,8 +232,18 @@ int main(int argc, char **argv)
printf("PASSED: %u\n", passed);
printf("SKIPPED: %u\n", skipped);
printf("FAILED: %u\n", failed);
if (skipped > 0) {
printf("\nSkipped tests:\n");
for (i = 0; i < skipped; i++)
printf(" - %s\n", skipped_tests[i]);
}
if (failed > 0) {
printf("\nFailed tests:\n");
for (i = 0; i < failed; i++)
printf(" - %s\n", failed_tests[i]);
}
return 0;
return failed > 0 ? 1 : 0;
}
void scx_test_register(struct scx_test *test)
tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c
+5 -3
View File
@@ -53,7 +53,7 @@ ddsp:
void BPF_STRUCT_OPS(select_cpu_vtime_dispatch, s32 cpu, struct task_struct *p)
{
if (scx_bpf_dsq_move_to_local(VTIME_DSQ))
if (scx_bpf_dsq_move_to_local(VTIME_DSQ, 0))
consumed = true;
}
@@ -66,12 +66,14 @@ void BPF_STRUCT_OPS(select_cpu_vtime_running, struct task_struct *p)
void BPF_STRUCT_OPS(select_cpu_vtime_stopping, struct task_struct *p,
bool runnable)
{
p->scx.dsq_vtime += (SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
u64 delta = scale_by_task_weight_inverse(p, SCX_SLICE_DFL - p->scx.slice);
scx_bpf_task_set_dsq_vtime(p, p->scx.dsq_vtime + delta);
}
void BPF_STRUCT_OPS(select_cpu_vtime_enable, struct task_struct *p)
{
p->scx.dsq_vtime = vtime_now;
scx_bpf_task_set_dsq_vtime(p, vtime_now);
}
s32 BPF_STRUCT_OPS_SLEEPABLE(select_cpu_vtime_init)
tools/testing/selftests/sched_ext/util.h
+1 -1
View File
@@ -10,4 +10,4 @@
long file_read_long(const char *path);
int file_write_long(const char *path, long val);
#endif // __SCX_TEST_H__
#endif // __SCX_TEST_UTIL_H__
tools/workqueue/wq_dump.py
+8 -12
View File
@@ -107,6 +107,7 @@ WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM']
WQ_AFFN_CPU = prog['WQ_AFFN_CPU']
WQ_AFFN_SMT = prog['WQ_AFFN_SMT']
WQ_AFFN_CACHE = prog['WQ_AFFN_CACHE']
WQ_AFFN_CACHE_SHARD = prog['WQ_AFFN_CACHE_SHARD']
WQ_AFFN_NUMA = prog['WQ_AFFN_NUMA']
WQ_AFFN_SYSTEM = prog['WQ_AFFN_SYSTEM']
@@ -138,7 +139,7 @@ def print_pod_type(pt):
print(f' [{cpu}]={pt.cpu_pod[cpu].value_()}', end='')
print('')
for affn in [WQ_AFFN_CPU, WQ_AFFN_SMT, WQ_AFFN_CACHE, WQ_AFFN_NUMA, WQ_AFFN_SYSTEM]:
for affn in [WQ_AFFN_CPU, WQ_AFFN_SMT, WQ_AFFN_CACHE, WQ_AFFN_CACHE_SHARD, WQ_AFFN_NUMA, WQ_AFFN_SYSTEM]:
print('')
print(f'{wq_affn_names[affn].string_().decode().upper()}{" (default)" if affn == wq_affn_dfl else ""}')
print_pod_type(wq_pod_types[affn])
@@ -227,15 +228,10 @@ if 'node_to_cpumask_map' in prog:
print(f'NODE[{node:02}]={cpumask_str(node_to_cpumask_map[node])}')
print('')
print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ min max', end='')
first = True
print(f'[{"workqueue":^{WQ_NAME_LEN-1}} {"min":>4} {"max":>4}', end='')
for node in for_each_node():
if first:
print(f' NODE {node}', end='')
first = False
else:
print(f' {node:7}', end='')
print(f' {"dfl":>7} ]')
print(f' {"NODE " + str(node):>9}', end='')
print(f' {"dfl":>9} ]')
print('')
for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
@@ -243,11 +239,11 @@ if 'node_to_cpumask_map' in prog:
continue
print(f'{wq.name.string_().decode():{WQ_NAME_LEN}} ', end='')
print(f'{wq.min_active.value_():3} {wq.max_active.value_():3}', end='')
print(f'{wq.min_active.value_():4} {wq.max_active.value_():4}', end='')
for node in for_each_node():
nna = wq.node_nr_active[node]
print(f' {nna.nr.counter.value_():3}/{nna.max.value_():3}', end='')
print(f' {f"{nna.nr.counter.value_()}/{nna.max.value_()}":>9}', end='')
nna = wq.node_nr_active[nr_node_ids]
print(f' {nna.nr.counter.value_():3}/{nna.max.value_():3}')
print(f' {f"{nna.nr.counter.value_()}/{nna.max.value_()}":>9}')
else:
printf(f'node_to_cpumask_map not present, is NUMA enabled?')