From 7c25a0bd4bf7139944c5893ff61211f4b5a3455e Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:32 +0200 Subject: [PATCH 01/69] eventpoll: expand top-of-file overview / locking doc The existing ~40-line "LOCKING:" banner covered the three-level lock hierarchy (epnested_mutex > ep->mtx > ep->lock) but nothing else. Lifetime rules, the ready-list state machine, the three removal paths, and the POLLFREE contract are implicit in the code. The recent UAF series (a6dc643c6931, 07712db80857, 8c2e52ebbe88, f2e467a48287) rode on invariants that were only implicit. Codify them at the top of the file: the subsystem overview, the lock hierarchy and its mutex_lock_nested() subclass convention (reworded from the old banner), a field-protection table for struct eventpoll and struct epitem that names the two faces of the rbn/rcu union (rbn under ep->mtx while linked into ep->rbr; rcu touched only by kfree_rcu(epi) on the free path), the ovflist sentinel encoding and scan-flip invariants, the three removal paths (A ep_remove, B ep_clear_and_put, C eventpoll_release_file) and the epi_fget() pin that orchestrates A vs C, and the POLLFREE store-release / load-acquire handshake. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-1-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 199 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 162 insertions(+), 37 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index a3090b446af1..5896f705a3ac 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -41,45 +41,170 @@ #include /* - * LOCKING: - * There are three level of locking required by epoll : + * fs/eventpoll.c - Efficient event polling ("epoll") kernel implementation. * - * 1) epnested_mutex (mutex) - * 2) ep->mtx (mutex) - * 3) ep->lock (spinlock) * - * The acquire order is the one listed above, from 1 to 3. - * We need a spinlock (ep->lock) because we manipulate objects - * from inside the poll callback, that might be triggered from - * a wake_up() that in turn might be called from IRQ context. - * So we can't sleep inside the poll callback and hence we need - * a spinlock. During the event transfer loop (from kernel to - * user space) we could end up sleeping due a copy_to_user(), so - * we need a lock that will allow us to sleep. This lock is a - * mutex (ep->mtx). It is acquired during the event transfer loop, - * during epoll_ctl(EPOLL_CTL_DEL) and during eventpoll_release_file(). - * The epnested_mutex is acquired when inserting an epoll fd onto another - * epoll fd. We do this so that we walk the epoll tree and ensure that this - * insertion does not create a cycle of epoll file descriptors, which - * could lead to deadlock. We need a global mutex to prevent two - * simultaneous inserts (A into B and B into A) from racing and - * constructing a cycle without either insert observing that it is - * going to. - * It is necessary to acquire multiple "ep->mtx"es at once in the - * case when one epoll fd is added to another. In this case, we - * always acquire the locks in the order of nesting (i.e. after - * epoll_ctl(e1, EPOLL_CTL_ADD, e2), e1->mtx will always be acquired - * before e2->mtx). Since we disallow cycles of epoll file - * descriptors, this ensures that the mutexes are well-ordered. In - * order to communicate this nesting to lockdep, when walking a tree - * of epoll file descriptors, we use the current recursion depth as - * the lockdep subkey. - * It is possible to drop the "ep->mtx" and to use the global - * mutex "epnested_mutex" (together with "ep->lock") to have it working, - * but having "ep->mtx" will make the interface more scalable. - * Events that require holding "epnested_mutex" are very rare, while for - * normal operations the epoll private "ep->mtx" will guarantee - * a better scalability. + * Overview + * -------- + * + * Each epoll_create(2) returns an anonymous [eventpoll] file whose + * ->private_data is a struct eventpoll. Each EPOLL_CTL_ADD installs + * a struct epitem linking one (watched file, fd) pair back to that + * eventpoll via the watched file's f_op->poll() wait queue(s). When + * the watched file signals readiness, ep_poll_callback() fires and + * marks the epitem ready. epoll_wait(2) drains the ready list under + * ep->mtx, re-queueing items in level-triggered mode. + * + * epoll instances can watch other epoll instances up to EP_MAX_NESTS + * deep; cycles are forbidden and detected at EPOLL_CTL_ADD time. + * + * + * Locking + * ------- + * + * Three levels, acquired from outer to inner: + * + * epnested_mutex (global; rare; taken only for EPOLL_CTL_ADD + * loop / path checks) + * > ep->mtx (per-eventpoll; sleepable; serializes most ops) + * > ep->lock (per-eventpoll; IRQ-safe spinlock) + * + * file->f_lock (per-file; NOT IRQ-safe; guards f_ep hlist ops; + * nested inside ep->mtx, outside ep->lock) + * + * Rationale: + * - ep->lock is a spinlock because ep_poll_callback() is called from + * wake_up() which may run in hard-IRQ context. All ep->lock + * critical sections use spin_lock_irqsave(). + * - ep->mtx is a sleepable mutex because the event delivery loop + * calls copy_to_user(), and ep_insert() may sleep in + * kmem_cache_alloc() and f_op->poll(). + * - epnested_mutex is global because cycle detection needs a global + * view of the epoll topology; a per-object scheme would let two + * concurrent inserts (A into B, B into A) construct a cycle + * without either observer seeing it. + * - Per-ep ep->mtx is preferred for scalability elsewhere. Events + * that require epnested_mutex are rare. + * + * When EPOLL_CTL_ADD nests one eventpoll inside another we acquire + * ep->mtx on both: outer first, target second. Since cycles are + * forbidden the set of live ep->mtx holds is always a strict chain, + * communicated to lockdep via mutex_lock_nested() subclasses derived + * from the current recursion depth. + * + * + * Field protection + * ---------------- + * + * struct eventpoll: + * mtx - self + * rbr - ep->mtx + * ovflist, rdllist - ep->lock (IRQ-safe) + * wq - ep->lock for queue mutation + * poll_wait - internal waitqueue spinlock + * refs - file->f_lock for adds; ep->mtx for removes; + * RCU for readers (hlist_del_rcu + kfree_rcu(ep)) + * ws - ep->mtx + * gen, loop_check_depth - epnested_mutex + * file, user - immutable after setup + * refcount - atomic (refcount_t) + * napi_* - READ_ONCE / WRITE_ONCE + * + * struct epitem: + * rbn / rcu union - rbn: ep->mtx (while epi is linked in ep->rbr). + * rcu: written only by kfree_rcu(epi) on the free + * path; otherwise untouched by epoll code. + * rdllink, next - ep->lock + * ffd, ep - immutable after ep_insert() + * pwqlist - ep->mtx for writes; POLLFREE clears pwq->whead + * via smp_store_release(), see below + * fllink - file->f_lock for mutation; hlist_del_rcu + + * kfree_rcu(epi) for safe RCU readers + * ws - RCU (rcu_assign_pointer / + * rcu_dereference_check(mtx)) + * event - ep->mtx for writes; lockless read in + * ep_poll_callback pairs with smp_mb() in + * ep_modify() + * + * + * Ready-list state machine + * ------------------------ + * + * Readiness is tracked in two lists under ep->lock: + * + * rdllist - doubly-linked FIFO; the "current" ready list. + * ovflist - singly-linked LIFO; used during a scan to catch + * events that arrive while rdllist is being iterated + * without ep->lock. + * + * Encoded in ep->ovflist: + * EP_UNACTIVE_PTR - no scan active; callback appends to rdllist. + * NULL - scan active, no spill yet. + * pointer to epi - scan active with spilled items (LIFO). + * + * Encoded in epi->next: + * EP_UNACTIVE_PTR - epi is not on ovflist. + * otherwise - next epi on ovflist (NULL at tail). + * + * ep_start_scan() flips "not scanning" to "scanning" and splices + * rdllist into a caller-local txlist. ep_done_scan() drains ovflist + * back to rdllist (list_add head-insert reverses LIFO to FIFO), + * flips back to "not scanning", and re-splices any items the caller + * left in txlist (e.g., level-triggered re-queues). + * + * + * Removal paths + * ------------- + * + * Three paths dispose of epitems and/or eventpolls: + * + * A. ep_remove() - EPOLL_CTL_DEL and ep_insert() + * rollback. Caller holds ep->mtx. + * B. ep_clear_and_put() - close of the epoll fd itself + * (ep_eventpoll_release). + * C. eventpoll_release_file() - close of a watched file, invoked + * from __fput(). + * + * Coordination: + * A and C exclude each other via the watched file's refcount. + * A pins the file with epi_fget() before touching file->f_ep or + * file->f_lock; if the pin fails, __fput() is in flight and C + * will clean this epi up. See the epi_fget() block comment. + * A and B both hold ep->mtx serially. B walks the rbtree with + * rb_next() captured before ep_remove() erases the current node. + * B and C both take ep->mtx; the loser sees fewer entries or an + * empty file->f_ep. + * + * Within every path the internal order is strict: + * ep_unregister_pollwait() - drain pwqlist; synchronizes with any + * in-flight ep_poll_callback via the + * watched wait-queue head's lock. + * ep_remove_file() - hlist_del_rcu of epi->fllink and, + * if last watcher, clear file->f_ep, + * under file->f_lock. + * ep_remove_epi() - rb_erase, rdllist unlink (ep->lock), + * wakeup_source_unregister, + * kfree_rcu(epi). + * + * kfree_rcu(epi) defers the free past RCU readers in + * reverse_path_check_proc(); kfree_rcu(ep) defers past readers in + * ep_get_upwards_depth_proc(). + * + * + * POLLFREE handshake + * ------------------ + * + * When a subsystem tears down a wait-queue head that an epitem is + * registered on (binder, signalfd, ...), it wakes the callback with + * POLLFREE and must RCU-defer the head's free. The store/load pair: + * + * ep_poll_callback() POLLFREE branch: + * smp_store_release(&pwq->whead, NULL) + * + * ep_remove_wait_queue(): + * smp_load_acquire(&pwq->whead) + * + * See those sites for the full argument. */ /* Epoll private bits inside the event mask */ From 35e7fa083ab25f68c7cd57b9427f65d8794ab7c2 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:33 +0200 Subject: [PATCH 02/69] eventpoll: document loop-check / path-check globals The globals that support EPOLL_CTL_ADD's cycle and path-length checks are scattered: epnested_mutex, loop_check_gen, inserting_into, and tfile_check_list sit at the top of the file; path_count[] and path_limits[] are declared inline with the path-check code further down. Their interaction -- the "ep->gen == loop_check_gen" trigger in do_epoll_ctl(), the two loop_check_gen++ bumps that sandwich a check, the EP_UNACTIVE_PTR sentinel on tfile_check_list, the -ELOOP back-edge detection via inserting_into -- is not documented anywhere. The area has had three recent fixes (CVE-2025-38349, the unbounded recursion fix, and the overflow fix) whose logic depends on these invariants. Collect the description in one block alongside the declarations, cross-reference the path_count[] declaration that lives with the path-check code, and name the fix commits so future readers can find the context. Also add a short comment on struct epitems_head describing its dual use (wrapper for non-epoll file->f_ep versus pointing into &ep->refs for the epoll-watches-epoll case), which the old comment on tfile_check_list had accidentally attached to the struct. Comment-only; no functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-2-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 56 ++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 50 insertions(+), 6 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 5896f705a3ac..477fcbc8e95e 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -372,12 +372,54 @@ struct ep_pqueue { /* Maximum number of epoll watched descriptors, per user */ static long max_user_watches __read_mostly; -/* Used for cycles detection */ +/* + * Cycle and path-length checks at EPOLL_CTL_ADD + * --------------------------------------------- + * + * When EPOLL_CTL_ADD creates a link that either targets an eventpoll + * file or extends an existing chain of eventpolls, two checks run: + * + * 1. no cycle is being formed -- ep_loop_check() walks downward + * from the candidate target, and ep_get_upwards_depth_proc() + * walks upward from the outer ep, both bounded by EP_MAX_NESTS. + * 2. no file accumulates more than path_limits[depth] wakeup paths + * of a given length -- reverse_path_check(). + * + * Both need a global view of the epoll topology and must be atomic + * with the insertion, so the scratch state below is all serialized by + * one global mutex, epnested_mutex. Non-nested inserts skip this + * machinery entirely and take only ep->mtx. + * + * epnested_mutex Serializes the whole check; also protects every + * other variable in this block plus path_count[] + * (declared with the path-check code further + * down). + * loop_check_gen Monotonic stamp, bumped once at the start of a + * check and once at the end. ep->gen caches the + * value under which ep was last visited by + * ep_loop_check_proc() or + * ep_get_upwards_depth_proc(); the post-check + * bump ensures those cached stamps can no longer + * equal loop_check_gen, so the + * "ep->gen == loop_check_gen" trigger in + * do_epoll_ctl() only fires while another check + * is in flight. + * inserting_into Outer eventpoll pointer for the lifetime of one + * ep_loop_check(); ep_loop_check_proc() fails + * with -ELOOP if the downward walk reaches it. + * tfile_check_list Singly-linked list of epitems_head objects + * collected by ep_loop_check_proc() during the + * walk, consumed by reverse_path_check() + * afterwards. Sentinel EP_UNACTIVE_PTR means no + * check is in flight. + * + * Commits fdcfce93073d ("eventpoll: Fix integer overflow in + * ep_loop_check_proc()") and f2e467a48287 ("eventpoll: Fix + * semi-unbounded recursion") hardened the walk; any refactor must + * preserve both bail-outs. + */ static DEFINE_MUTEX(epnested_mutex); - static u64 loop_check_gen = 0; - -/* Used to check for epoll file descriptor inclusion loops */ static struct eventpoll *inserting_into; /* Slab cache used to allocate "struct epitem" */ @@ -387,8 +429,10 @@ static struct kmem_cache *epi_cache __ro_after_init; static struct kmem_cache *pwq_cache __ro_after_init; /* - * List of files with newly added links, where we may need to limit the number - * of emanating paths. Protected by the epnested_mutex. + * Wrapper anchor for file->f_ep when the watched file is not itself an + * eventpoll; for the epoll-watches-epoll case, file->f_ep points at + * &watched_ep->refs directly. The ->next field threads + * tfile_check_list during one EPOLL_CTL_ADD path check. */ struct epitems_head { struct hlist_head epitems; From 14af80c3255590a7b0936c29f1b88127f9f9fc2d Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:34 +0200 Subject: [PATCH 03/69] eventpoll: clarify POLLFREE handshake comments ep_remove_wait_queue() and the POLLFREE branch of ep_poll_callback() are the two halves of a release/acquire handshake that lets a subsystem (binder, signalfd, ...) tear down a wait-queue head from under a registered epitem. The existing local comments documented the race but did not name the protocol or refer readers from one side to the other. After the previous commit added a "POLLFREE handshake" section to the top-of-file banner, these sites can point at the banner and at each other. Rework the two comment blocks so that each side is labelled "acquire side" or "release side", references the banner, and explains its role in the protocol. On the release side fuse the two former comments into one narrative: list_del_init() tolerates a second delete from a racing ep_remove_wait_queue(), and the smp_store_release() is what lets that racing remover discover the teardown. Comment-only; no functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-3-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 477fcbc8e95e..1d1fd6464c38 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -828,10 +828,15 @@ static void ep_remove_wait_queue(struct eppoll_entry *pwq) rcu_read_lock(); /* - * If it is cleared by POLLFREE, it should be rcu-safe. - * If we read NULL we need a barrier paired with - * smp_store_release() in ep_poll_callback(), otherwise - * we rely on whead->lock. + * POLLFREE handshake, acquire side; see "POLLFREE handshake" + * at the top of this file. + * + * A NULL load is paired with the smp_store_release(&whead, NULL) + * in ep_poll_callback()'s POLLFREE branch: the teardown is + * complete and we must not touch whead again. On a non-NULL load + * rcu_read_lock() keeps the waitqueue memory alive (POLLFREE + * firers RCU-defer the free) and whead->lock inside + * remove_wait_queue() serializes us against the store side. */ whead = smp_load_acquire(&pwq->whead); if (whead) @@ -1505,17 +1510,24 @@ out_unlock: if (pollflags & POLLFREE) { /* - * If we race with ep_remove_wait_queue() it can miss - * ->whead = NULL and do another remove_wait_queue() after - * us, so we can't use __remove_wait_queue(). + * POLLFREE handshake, release side; see "POLLFREE handshake" + * at the top of this file. + * + * Unlink our wait entry with list_del_init rather than + * __remove_wait_queue: a concurrent ep_remove_wait_queue() + * that already loaded a non-NULL whead may still call + * remove_wait_queue() after us, and list_del_init() tolerates + * the second delete. + * + * smp_store_release(&whead, NULL) publishes the teardown to + * ep_remove_wait_queue()'s smp_load_acquire(). Before this + * store, a racing ep_clear_and_put() / ep_remove() reaches + * ep_remove_wait_queue() which sees whead != NULL and takes + * whead->lock -- the same lock held by our caller, so it + * serializes behind us. Once whead is zeroed, nothing else + * protects ep / epi / wait. */ list_del_init(&wait->entry); - /* - * ->whead != NULL protects us from the race with - * ep_clear_and_put() or ep_remove(), ep_remove_wait_queue() - * takes whead->lock held by the caller. Once we nullify it, - * nothing protects ep/epi or even wait. - */ smp_store_release(&ep_pwq_from_wait(wait)->whead, NULL); } From a573cb40f9819c3fe81a43eb10170f8fc8eddc5e Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:35 +0200 Subject: [PATCH 04/69] eventpoll: refresh epi_fget() / ep_remove_file() comments Two comments drifted from the code they sit on. epi_fget()'s block comment still referenced atomic_long_inc_not_zero, which has been file_ref_get() for a while, and described only one of the function's two roles: safe dereference of epi->ffd.file under ep->mtx. Since commit a6dc643c6931 ("eventpoll: fix ep_remove struct eventpoll / struct file UAF") the refcount bump also serves as a pin that blocks __fput() from starting, which is what lets ep_remove() touch file->f_lock and file->f_ep without racing eventpoll_release_file(). Update the block to name both roles and the commit that introduced the pin role. ep_remove_file()'s one-line "See eventpoll_release() for details" pointed at an inline in include/linux/eventpoll.h but said nothing about what those details were. Replace it with a short explanation: we publish NULL so the eventpoll_release() fastpath can skip the slow path, and this is safe because every f_ep writer either holds a pin via epi_fget() or is __fput() itself. Comment-only; no functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-4-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 1d1fd6464c38..1039d9737ce9 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -991,22 +991,23 @@ static void ep_free(struct eventpoll *ep) } /* - * The ffd.file pointer may be in the process of being torn down due to - * being closed, but we may not have finished eventpoll_release() yet. + * Pin @epi->ffd.file for operations that require both safe dereference + * and exclusion from __fput(). * - * Normally, even with the atomic_long_inc_not_zero, the file may have - * been free'd and then gotten re-allocated to something else (since - * files are not RCU-delayed, they are SLAB_TYPESAFE_BY_RCU). + * struct file uses SLAB_TYPESAFE_BY_RCU, so a freed slot can be + * reassigned at any time. The bare load of epi->ffd.file is safe here + * because the caller holds ep->mtx and eventpoll_release_file() blocks + * on that mutex while tearing down the epi, so the backing file + * allocation cannot be freed and reused under us. An rcu_read_lock() + * is therefore unnecessary for the load. * - * But for epoll, users hold the ep->mtx mutex, and as such any file in - * the process of being free'd will block in eventpoll_release_file() - * and thus the underlying file allocation will not be free'd, and the - * file re-use cannot happen. - * - * For the same reason we can avoid a rcu_read_lock() around the - * operation - 'ffd.file' cannot go away even if the refcount has - * reached zero (but we must still not call out to ->poll() functions - * etc). + * A successful file_ref_get() additionally blocks __fput() from + * starting on this file: once the refcount has reached zero it cannot + * come back. ep_remove() relies on that to touch file->f_lock and + * file->f_ep without racing eventpoll_release_file() (see commit + * a6dc643c6931). A NULL return means __fput() is already in flight; + * the caller must bail without touching the file, and + * eventpoll_release_file() will clean the epi up from its side. */ static struct file *epi_fget(const struct epitem *epi) { @@ -1032,7 +1033,13 @@ static void ep_remove_file(struct eventpoll *ep, struct epitem *epi, spin_lock(&file->f_lock); head = file->f_ep; if (hlist_is_singular_node(&epi->fllink, head)) { - /* See eventpoll_release() for details. */ + /* + * Last watcher: publish NULL so the eventpoll_release() + * fastpath in include/linux/eventpoll.h can skip the slow + * path on a future __fput(). Safe because every f_ep writer + * either holds a pin on @file via epi_fget() or is __fput() + * itself -- see the comment in eventpoll_release(). + */ WRITE_ONCE(file->f_ep, NULL); if (!is_file_epoll(file)) { struct epitems_head *v; From 858c14b1f0a4aa3faebe1b9af757f20518ea78be Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:36 +0200 Subject: [PATCH 05/69] eventpoll: document ep_clear_and_put() two-pass pattern ep_clear_and_put() walks the rbtree twice: once to drain each epi's pwqlist, then again to ep_remove() each entry. The split is load-bearing -- fusing the passes into one loop would let a poll callback still queued on some epi_i fire after epi_{i+k} has already been freed -- but the previous comments described each pass in isolation and did not explain the ordering invariant or the cooperation with removal path C (eventpoll_release_file). Add a function-level docblock that labels this as path B from the top-of-file "Removal paths" section, names the two passes and the ordering invariant, explains the pwqlist drain as synchronization with in-flight ep_poll_callback() via whead->lock, describes the C-path hand-off when epi_fget() returns NULL, and states the ep->refcount invariant that keeps ep_remove()'s WARN_ON_ONCE safe across the loop. Also tighten the per-pass comments to one line each and fix the minor grammar bug in the poll_wait release comment ("these file" -> "poll-on-ep"). No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-5-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 1039d9737ce9..b6a14c69c482 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1103,20 +1103,47 @@ static void ep_remove(struct eventpoll *ep, struct epitem *epi) WARN_ON_ONCE(ep_refcount_dec_and_test(ep)); } +/* + * Removal path B (see "Removal paths" in the top-of-file banner): + * close of the epoll fd itself, reached via ep_eventpoll_release(). + * + * Under ep->mtx we walk the rbtree twice: + * + * Pass 1 drains each epi's pwqlist via ep_unregister_pollwait(). + * This takes each watched waitqueue head's lock and so + * synchronizes with any in-flight ep_poll_callback(), so + * after the pass ends no callback can still be holding or + * about to dereference any epi on this ep. + * + * Pass 2 runs ep_remove() on each epi. The per-epi pwqlist is + * already empty, but the rest of ep_remove() still runs + * (epi_fget() pin, f_ep clear under f_lock, rbtree erase, + * rdllist unlink, kfree_rcu). + * + * Pass 1 must strictly precede Pass 2: fusing them would let a + * callback queued on epi_i still fire after epi_{i+k} was freed. + * + * A concurrent eventpoll_release_file() (path C) serializes against + * us on ep->mtx; in Pass 2, ep_remove() transparently hands off any + * epi whose watched file is in __fput() by bailing when epi_fget() + * returns NULL, and C will clean that epi up on its side. + * + * ep->refcount is held > 0 throughout by the ep file's own share; + * we drop that share after the walk and free the eventpoll if we + * were last. + */ static void ep_clear_and_put(struct eventpoll *ep) { struct rb_node *rbp, *next; struct epitem *epi; - /* We need to release all tasks waiting for these file */ + /* Release any threads blocked in poll-on-ep. */ if (waitqueue_active(&ep->poll_wait)) ep_poll_safewake(ep, NULL, 0); mutex_lock(&ep->mtx); - /* - * Walks through the whole tree by unregistering poll callbacks. - */ + /* Pass 1: drain pwqlists; synchronizes with in-flight callbacks. */ for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = rb_next(rbp)) { epi = rb_entry(rbp, struct epitem, rbn); @@ -1124,14 +1151,7 @@ static void ep_clear_and_put(struct eventpoll *ep) cond_resched(); } - /* - * Walks through the whole tree and try to free each "struct epitem". - * Note that ep_remove() will not remove the epitem in case of a - * racing eventpoll_release_file(); the latter will do the removal. - * At this point we are sure no poll callbacks will be lingering around. - * Since we still own a reference to the eventpoll struct, the loop can't - * dispose it. - */ + /* Pass 2: remove each epi. rb_next() is captured before erase. */ for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = next) { next = rb_next(rbp); epi = rb_entry(rbp, struct epitem, rbn); From 31c95f087e5df2c508f0f52d13d42f44d78c6510 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:37 +0200 Subject: [PATCH 06/69] eventpoll: rename ep_refcount_dec_and_test() to ep_put() ep_refcount_dec_and_test() mirrors refcount_dec_and_test() verbatim, which reads fine at a call site like if (ep_refcount_dec_and_test(ep)) ep_free(ep); but awkward at WARN_ON_ONCE(ep_refcount_dec_and_test(ep)); and does not pair cleanly with ep_get(). Rename to the idiomatic ep_put() and reword the kerneldoc to spell out the return-value contract (caller is responsible for ep_free() iff the return is true). Leave ep_put() as a bool-returning wrapper -- we cannot fold ep_free() into it because ep_remove() calls it under ep->mtx and the mutex would still be held when ep_free()'s mutex_destroy() ran (see commit 8c2e52ebbe88 "eventpoll: don't decrement ep refcount while still holding the ep mutex"). No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-6-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index b6a14c69c482..da31a3ac6057 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -969,9 +969,10 @@ static void ep_get(struct eventpoll *ep) } /* - * Returns true if the event poll can be disposed + * Drop a reference to @ep; returns true iff it was the last, in which + * case the caller is responsible for ep_free(). */ -static bool ep_refcount_dec_and_test(struct eventpoll *ep) +static bool ep_put(struct eventpoll *ep) { if (!refcount_dec_and_test(&ep->refcount)) return false; @@ -1100,7 +1101,7 @@ static void ep_remove(struct eventpoll *ep, struct epitem *epi) ep_remove_file(ep, epi, file); ep_remove_epi(ep, epi); - WARN_ON_ONCE(ep_refcount_dec_and_test(ep)); + WARN_ON_ONCE(ep_put(ep)); } /* @@ -1160,7 +1161,7 @@ static void ep_clear_and_put(struct eventpoll *ep) } mutex_unlock(&ep->mtx); - if (ep_refcount_dec_and_test(ep)) + if (ep_put(ep)) ep_free(ep); } @@ -1339,7 +1340,7 @@ again: mutex_unlock(&ep->mtx); - if (ep_refcount_dec_and_test(ep)) + if (ep_put(ep)) ep_free(ep); goto again; } From c17eb7a5285a6d77c16397ad3b6223aca7a87d56 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:38 +0200 Subject: [PATCH 07/69] eventpoll: drop unused depth argument from epoll_mutex_lock() epoll_mutex_lock() has three callers, all in do_epoll_ctl(), and every one passes depth == 0. The argument has been dead since the helper was introduced. Drop it. Because a zero subclass makes mutex_lock_nested() equivalent to mutex_lock(), switch the blocking path to the simpler primitive as well. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-7-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index da31a3ac6057..ba1017c72167 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -2432,16 +2432,13 @@ static inline void ep_take_care_of_epollwakeup(struct epoll_event *epev) } #endif -static inline int epoll_mutex_lock(struct mutex *mutex, int depth, - bool nonblock) +static inline int epoll_mutex_lock(struct mutex *mutex, bool nonblock) { if (!nonblock) { - mutex_lock_nested(mutex, depth); + mutex_lock(mutex); return 0; } - if (mutex_trylock(mutex)) - return 0; - return -EAGAIN; + return mutex_trylock(mutex) ? 0 : -EAGAIN; } int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, @@ -2513,14 +2510,14 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, * deep wakeup paths from forming in parallel through multiple * EPOLL_CTL_ADD operations. */ - error = epoll_mutex_lock(&ep->mtx, 0, nonblock); + error = epoll_mutex_lock(&ep->mtx, nonblock); if (error) goto error_tgt_fput; if (op == EPOLL_CTL_ADD) { if (READ_ONCE(fd_file(f)->f_ep) || ep->gen == loop_check_gen || is_file_epoll(fd_file(tf))) { mutex_unlock(&ep->mtx); - error = epoll_mutex_lock(&epnested_mutex, 0, nonblock); + error = epoll_mutex_lock(&epnested_mutex, nonblock); if (error) goto error_tgt_fput; loop_check_gen++; @@ -2531,7 +2528,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, if (ep_loop_check(ep, tep) != 0) goto error_tgt_fput; } - error = epoll_mutex_lock(&ep->mtx, 0, nonblock); + error = epoll_mutex_lock(&ep->mtx, nonblock); if (error) goto error_tgt_fput; } From 6a3f1a494bc91d7976cf0d2b200bb3f1a22eef64 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:39 +0200 Subject: [PATCH 08/69] eventpoll: rename attach_epitem() to ep_attach_file() ep_remove_file() tears down the f_ep linkage that attach_epitem() establishes, so the pair should look like one. Rename to ep_attach_file() for the "ep_*" + subject symmetry and to match the naming used elsewhere in the file (ep_insert, ep_modify, ep_remove, ep_remove_file, ep_remove_epi, ep_unregister_pollwait). Pure rename; no functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-8-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index ba1017c72167..1fe9f1772a28 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1735,7 +1735,7 @@ static noinline void ep_destroy_wakeup_source(struct epitem *epi) wakeup_source_unregister(ws); } -static int attach_epitem(struct file *file, struct epitem *epi) +static int ep_attach_file(struct file *file, struct epitem *epi) { struct epitems_head *to_free = NULL; struct hlist_head *head = NULL; @@ -1806,7 +1806,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, if (tep) mutex_lock_nested(&tep->mtx, 1); /* Add the current item to the list of active epoll hook for this file */ - if (unlikely(attach_epitem(tfile, epi) < 0)) { + if (unlikely(ep_attach_file(tfile, epi) < 0)) { if (tep) mutex_unlock(&tep->mtx); kmem_cache_free(epi_cache, epi); From 7de03099264ac71d1fed8af9632bfea5af7bbf49 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:40 +0200 Subject: [PATCH 09/69] eventpoll: relocate KCMP helpers near compat syscalls ep_find_tfd() and get_epoll_tfile_raw_ptr() are only used when CONFIG_KCMP=y. They implement the lookup side of the kcmp(2) KCMP_EPOLL_TFD query. The helpers currently live between ep_find() and ep_poll_callback(), interrupting the run of hot-path code (callback, wait-queue setup, path check, insert, modify, send_events, poll) with a feature-gated block. Move the #ifdef CONFIG_KCMP block next to #ifdef CONFIG_COMPAT, which is also a peripheral ABI extension. Hot-path code becomes a contiguous span, and the userspace-adjacent extensions cluster at the end of the file just before eventpoll_init(). Pure code movement; diff is 44 removed and 44 added, all within one block. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-9-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 88 +++++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 44 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 1fe9f1772a28..fde2396342b6 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1399,50 +1399,6 @@ static struct epitem *ep_find(struct eventpoll *ep, struct file *file, int fd) return epir; } -#ifdef CONFIG_KCMP -static struct epitem *ep_find_tfd(struct eventpoll *ep, int tfd, unsigned long toff) -{ - struct rb_node *rbp; - struct epitem *epi; - - for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = rb_next(rbp)) { - epi = rb_entry(rbp, struct epitem, rbn); - if (epi->ffd.fd == tfd) { - if (toff == 0) - return epi; - else - toff--; - } - cond_resched(); - } - - return NULL; -} - -struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd, - unsigned long toff) -{ - struct file *file_raw; - struct eventpoll *ep; - struct epitem *epi; - - if (!is_file_epoll(file)) - return ERR_PTR(-EINVAL); - - ep = file->private_data; - - mutex_lock(&ep->mtx); - epi = ep_find_tfd(ep, tfd, toff); - if (epi) - file_raw = epi->ffd.file; - else - file_raw = ERR_PTR(-ENOENT); - mutex_unlock(&ep->mtx); - - return file_raw; -} -#endif /* CONFIG_KCMP */ - /* * This is the callback that is passed to the wait queue wakeup * mechanism. It is called by the stored file descriptors when they @@ -2733,6 +2689,50 @@ SYSCALL_DEFINE6(epoll_pwait2, int, epfd, struct epoll_event __user *, events, sigmask, sigsetsize); } +#ifdef CONFIG_KCMP +static struct epitem *ep_find_tfd(struct eventpoll *ep, int tfd, unsigned long toff) +{ + struct rb_node *rbp; + struct epitem *epi; + + for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = rb_next(rbp)) { + epi = rb_entry(rbp, struct epitem, rbn); + if (epi->ffd.fd == tfd) { + if (toff == 0) + return epi; + else + toff--; + } + cond_resched(); + } + + return NULL; +} + +struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd, + unsigned long toff) +{ + struct file *file_raw; + struct eventpoll *ep; + struct epitem *epi; + + if (!is_file_epoll(file)) + return ERR_PTR(-EINVAL); + + ep = file->private_data; + + mutex_lock(&ep->mtx); + epi = ep_find_tfd(ep, tfd, toff); + if (epi) + file_raw = epi->ffd.file; + else + file_raw = ERR_PTR(-ENOENT); + mutex_unlock(&ep->mtx); + + return file_raw; +} +#endif /* CONFIG_KCMP */ + #ifdef CONFIG_COMPAT static int do_compat_epoll_pwait(int epfd, struct epoll_event __user *events, int maxevents, struct timespec64 *timeout, From e0e35f4cb983a55a36e79e9b2a20ca0e0688fae6 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:41 +0200 Subject: [PATCH 10/69] eventpoll: split ep_insert() into alloc + register stages ep_insert() was 130 lines and mixed four concerns in one body: user quota charge and epitem allocation, attach-into-file-hlist plus rbtree insert plus target-ep locking, reverse-path + EPOLLWAKEUP + poll-queue install with rollback, and ready-list publication. Factor the first two concerns into named helpers so the body reduces to orchestration. ep_alloc_epitem() charges the user's epoll_watches quota, allocates a fresh epitem, and initializes its fields. On failure it returns ERR_PTR(-ENOSPC) or ERR_PTR(-ENOMEM); on success the epi is not yet linked into anything. ep_register_epitem() installs @epi into @tfile's f_ep hlist and @ep's rbtree, optionally chains @tfile onto tfile_check_list for the path check, takes the tep->mtx nested lock for the epoll-watches- epoll case, and finally takes the ep_get() reference that pairs with ep_remove()'s ep_put() in ep_insert()'s error paths. On failure it frees the epi and decrements epoll_watches to match ep_alloc_epitem(). ep_insert()'s remaining body is the rollback-via-ep_remove() chain (reverse_path_check, EPOLLWAKEUP source creation, ep_ptable_queue_proc allocation) and the ready-list / wake publication. Remove a few stale comments that duplicated function-level documentation or described obvious code. No functional change; rollback boundaries unchanged -- every error path after ep_register_epitem() still calls ep_remove(), preserving the ep->refcount invariant that keeps ep_remove()'s WARN_ON_ONCE safe. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-10-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 143 +++++++++++++++++++++++++++++++------------------ 1 file changed, 90 insertions(+), 53 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index fde2396342b6..e4a4e92d329f 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1725,6 +1725,87 @@ allocate: return 0; } +/* + * Charge the user's epoll_watches quota, allocate a fresh epitem for + * @tfile/@fd, and initialize its fields. The returned item is not yet + * linked into any data structure; the caller must install it via + * ep_register_epitem() (which takes over on success) or kmem_cache_free() + * it and decrement epoll_watches on its own. + * + * Returns ERR_PTR(-ENOSPC) if the quota is exceeded, ERR_PTR(-ENOMEM) + * if the slab allocation fails. + */ +static struct epitem *ep_alloc_epitem(struct eventpoll *ep, + const struct epoll_event *event, + struct file *tfile, int fd) +{ + struct epitem *epi; + + if (unlikely(percpu_counter_compare(&ep->user->epoll_watches, + max_user_watches) >= 0)) + return ERR_PTR(-ENOSPC); + percpu_counter_inc(&ep->user->epoll_watches); + + epi = kmem_cache_zalloc(epi_cache, GFP_KERNEL); + if (unlikely(!epi)) { + percpu_counter_dec(&ep->user->epoll_watches); + return ERR_PTR(-ENOMEM); + } + + INIT_LIST_HEAD(&epi->rdllink); + epi->ep = ep; + ep_set_ffd(&epi->ffd, tfile, fd); + epi->event = *event; + epi->next = EP_UNACTIVE_PTR; + + return epi; +} + +/* + * Install @epi into its target file's f_ep hlist and into @ep's rbtree, + * taking one additional reference on @ep for the lifetime of the item. + * + * If @tep is non-NULL, the target file is itself an eventpoll; we hold + * tep->mtx at subclass 1 across the attach + rbtree insert to serialize + * with the target side. RB tree ops are protected by @ep->mtx, which + * the caller already holds. + * + * On failure the epi is freed and the epoll_watches counter decremented, + * matching ep_alloc_epitem()'s allocation. After this returns + * successfully, ep_insert()'s later error paths use ep_remove() for + * unwind; that cannot drop @ep's refcount to zero because the ep file + * itself still holds the original reference. + */ +static int ep_register_epitem(struct eventpoll *ep, struct epitem *epi, + struct eventpoll *tep, int full_check) +{ + struct file *tfile = epi->ffd.file; + int error; + + if (tep) + mutex_lock_nested(&tep->mtx, 1); + + error = ep_attach_file(tfile, epi); + if (unlikely(error)) { + if (tep) + mutex_unlock(&tep->mtx); + kmem_cache_free(epi_cache, epi); + percpu_counter_dec(&ep->user->epoll_watches); + return error; + } + + if (full_check && !tep) + list_file(tfile); + + ep_rbtree_insert(ep, epi); + + if (tep) + mutex_unlock(&tep->mtx); + + ep_get(ep); + return 0; +} + /* * Must be called with "mtx" held. */ @@ -1742,52 +1823,15 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, lockdep_assert_irqs_enabled(); - if (unlikely(percpu_counter_compare(&ep->user->epoll_watches, - max_user_watches) >= 0)) - return -ENOSPC; - percpu_counter_inc(&ep->user->epoll_watches); + epi = ep_alloc_epitem(ep, event, tfile, fd); + if (IS_ERR(epi)) + return PTR_ERR(epi); - if (!(epi = kmem_cache_zalloc(epi_cache, GFP_KERNEL))) { - percpu_counter_dec(&ep->user->epoll_watches); - return -ENOMEM; - } + error = ep_register_epitem(ep, epi, tep, full_check); + if (error) + return error; - /* Item initialization follow here ... */ - INIT_LIST_HEAD(&epi->rdllink); - epi->ep = ep; - ep_set_ffd(&epi->ffd, tfile, fd); - epi->event = *event; - epi->next = EP_UNACTIVE_PTR; - - if (tep) - mutex_lock_nested(&tep->mtx, 1); - /* Add the current item to the list of active epoll hook for this file */ - if (unlikely(ep_attach_file(tfile, epi) < 0)) { - if (tep) - mutex_unlock(&tep->mtx); - kmem_cache_free(epi_cache, epi); - percpu_counter_dec(&ep->user->epoll_watches); - return -ENOMEM; - } - - if (full_check && !tep) - list_file(tfile); - - /* - * Add the current item to the RB tree. All RB tree operations are - * protected by "mtx", and ep_insert() is called with "mtx" held. - */ - ep_rbtree_insert(ep, epi); - if (tep) - mutex_unlock(&tep->mtx); - - /* - * ep_remove() calls in the later error paths can't lead to - * ep_free() as the ep file itself still holds an ep reference. - */ - ep_get(ep); - - /* now check if we've created too many backpaths */ + /* Reject the insert if the new link would create too many back-paths. */ if (unlikely(full_check && reverse_path_check())) { ep_remove(ep, epi); return -EINVAL; @@ -1814,28 +1858,21 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, */ revents = ep_item_poll(epi, &epq.pt, 1); - /* - * We have to check if something went wrong during the poll wait queue - * install process. Namely an allocation for a wait queue failed due - * high memory pressure. - */ + /* ep_ptable_queue_proc() signals allocation failure by clearing epq.epi. */ if (unlikely(!epq.epi)) { ep_remove(ep, epi); return -ENOMEM; } - /* We have to drop the new item inside our item list to keep track of it */ + /* Drop the new item onto the ready list if it is already ready. */ spin_lock_irq(&ep->lock); - /* record NAPI ID of new item if present */ ep_set_busy_poll_napi_id(epi); - /* If the file is already "ready" we drop it inside the ready list */ if (revents && !ep_is_linked(epi)) { list_add_tail(&epi->rdllink, &ep->rdllist); ep_pm_stay_awake(epi); - /* Notify waiting tasks that events are available */ if (waitqueue_active(&ep->wq)) wake_up(&ep->wq); if (waitqueue_active(&ep->poll_wait)) From 08fe3679a86d8c069ba93ec6211a1df82a8a8ad5 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:42 +0200 Subject: [PATCH 11/69] eventpoll: split ep_clear_and_put() into drain helpers ep_clear_and_put()'s two-pass walk is the main way an ep file close tears down its state, and the ordering between the passes is load-bearing (see previous commit's docblock). Give each pass its own function so the ordering is enforced by the call sequence in ep_clear_and_put() rather than by convention inside one body. ep_drain_pollwaits() carries out Pass 1: walk the rbtree and ep_unregister_pollwait() each epi. The function-level comment names it as Pass 1 and spells out the synchronization contract with ep_poll_callback(). ep_drain_tree() carries out Pass 2: walk the rbtree and ep_remove() each epi, capturing rb_next() before each erase. The comment names it as Pass 2 and documents the hand-off with a concurrent eventpoll_release_file() (removal path C). ep_clear_and_put() keeps the poll-on-ep wakeup, ep->mtx bracketing, and ep_put() + conditional ep_free(), and its docblock shrinks to the high-level summary; the per-pass detail moved into the helpers. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-11-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 87 +++++++++++++++++++++++++++++--------------------- 1 file changed, 51 insertions(+), 36 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index e4a4e92d329f..eeddd05ba529 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1105,62 +1105,77 @@ static void ep_remove(struct eventpoll *ep, struct epitem *epi) } /* - * Removal path B (see "Removal paths" in the top-of-file banner): - * close of the epoll fd itself, reached via ep_eventpoll_release(). - * - * Under ep->mtx we walk the rbtree twice: - * - * Pass 1 drains each epi's pwqlist via ep_unregister_pollwait(). - * This takes each watched waitqueue head's lock and so - * synchronizes with any in-flight ep_poll_callback(), so - * after the pass ends no callback can still be holding or - * about to dereference any epi on this ep. - * - * Pass 2 runs ep_remove() on each epi. The per-epi pwqlist is - * already empty, but the rest of ep_remove() still runs - * (epi_fget() pin, f_ep clear under f_lock, rbtree erase, - * rdllist unlink, kfree_rcu). - * - * Pass 1 must strictly precede Pass 2: fusing them would let a - * callback queued on epi_i still fire after epi_{i+k} was freed. - * - * A concurrent eventpoll_release_file() (path C) serializes against - * us on ep->mtx; in Pass 2, ep_remove() transparently hands off any - * epi whose watched file is in __fput() by bailing when epi_fget() - * returns NULL, and C will clean that epi up on its side. - * - * ep->refcount is held > 0 throughout by the ep file's own share; - * we drop that share after the walk and free the eventpoll if we - * were last. + * Pass 1 of ep_clear_and_put(): drain every epi's pwqlist. + * ep_unregister_pollwait() takes each watched wait-queue head's lock, + * which synchronizes with any in-flight ep_poll_callback(); after + * this returns no callback can still be about to dereference an epi + * on this ep. Must strictly precede ep_drain_tree() -- fusing the + * two walks would let a callback queued on epi_i still fire after + * epi_{i+k} had already been freed. */ -static void ep_clear_and_put(struct eventpoll *ep) +static void ep_drain_pollwaits(struct eventpoll *ep) { - struct rb_node *rbp, *next; + struct rb_node *rbp; struct epitem *epi; - /* Release any threads blocked in poll-on-ep. */ - if (waitqueue_active(&ep->poll_wait)) - ep_poll_safewake(ep, NULL, 0); + lockdep_assert_held(&ep->mtx); - mutex_lock(&ep->mtx); - - /* Pass 1: drain pwqlists; synchronizes with in-flight callbacks. */ for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = rb_next(rbp)) { epi = rb_entry(rbp, struct epitem, rbn); ep_unregister_pollwait(ep, epi); cond_resched(); } +} + +/* + * Pass 2 of ep_clear_and_put(): ep_remove() every epi. The per-epi + * pwqlist is already empty (ep_drain_pollwaits ran), but the rest of + * ep_remove() still runs: epi_fget() pin, f_ep clear under f_lock, + * rbtree erase, rdllist unlink, kfree_rcu(epi). rb_next() is captured + * before each erase so the iteration is stable. + * + * A concurrent eventpoll_release_file() (removal path C) on a watched + * file serializes with us via ep->mtx; ep_remove() transparently + * hands off any epi whose file is in __fput() by bailing when + * epi_fget() returns NULL, and path C will clean that epi up. + */ +static void ep_drain_tree(struct eventpoll *ep) +{ + struct rb_node *rbp, *next; + struct epitem *epi; + + lockdep_assert_held(&ep->mtx); - /* Pass 2: remove each epi. rb_next() is captured before erase. */ for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = next) { next = rb_next(rbp); epi = rb_entry(rbp, struct epitem, rbn); ep_remove(ep, epi); cond_resched(); } +} +/* + * Removal path B (see "Removal paths" in the top-of-file banner): + * close of the epoll fd itself, reached via ep_eventpoll_release(). + * + * Two passes under ep->mtx: first ep_drain_pollwaits() quiesces + * in-flight callbacks, then ep_drain_tree() frees the epis. The + * ep->refcount is kept > 0 across the walk by the ep file's own + * share, which we drop below; ep_free() runs iff we were the last + * holder after the tree drained. + */ +static void ep_clear_and_put(struct eventpoll *ep) +{ + /* Release any threads blocked in poll-on-ep. */ + if (waitqueue_active(&ep->poll_wait)) + ep_poll_safewake(ep, NULL, 0); + + mutex_lock(&ep->mtx); + ep_drain_pollwaits(ep); + ep_drain_tree(ep); mutex_unlock(&ep->mtx); + if (ep_put(ep)) ep_free(ep); } From 499a5e7f4a57fa08a297c627d007a55069acac9a Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:43 +0200 Subject: [PATCH 12/69] eventpoll: extract ep_deliver_event() from ep_send_events() ep_send_events()'s body covered two concerns: per-item work (PM wakeup-source bookkeeping, re-poll, copy_to_user, level-trigger re-queue, EPOLLONESHOT mask clear) and the scan-level accumulator (maxevents cap, EFAULT preservation, txlist/rdllist splice). Extract the per-item work as ep_deliver_event(), which returns a tri-state int: 1 one event was delivered; caller advances the counter, 0 re-poll produced no caller-requested events (item drops out of the ready list; a future callback will re-queue), -EFAULT copy_to_user() faulted; item is already re-inserted at the head of the txlist so ep_done_scan() splices it back to rdllist. The per-item comments (PM ordering, the "sole writer to rdllist" invariant for the LT re-queue, the EFAULT semantics) move into ep_deliver_event(). ep_send_events() reduces to the fatal-signal short-circuit, scan bracket, and a short txlist walk that accumulates the deliveries and preserves the "first error wins" EFAULT contract (res = delivered only if no event was previously delivered; otherwise the success count is returned and -EFAULT is reported on the next call). No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-12-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 138 ++++++++++++++++++++++++++++++------------------- 1 file changed, 84 insertions(+), 54 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index eeddd05ba529..6d4167a347ab 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1979,6 +1979,82 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, return 0; } +/* + * Attempt to deliver one event for @epi into @*uevents. + * + * Returns 1 if an event was delivered (with *uevents advanced to the + * next slot), 0 if the re-poll reported no caller-requested events + * (@epi drops out of the ready list; a future callback will re-add + * it), or -EFAULT if copy_to_user() faulted (in which case @epi is + * re-inserted at the head of @txlist so ep_done_scan() merges it + * back to rdllist for the next attempt). + * + * PM bookkeeping and level-triggered re-queue are handled here. + * Caller holds ep->mtx and the scan is active. + */ +static int ep_deliver_event(struct eventpoll *ep, struct epitem *epi, + poll_table *pt, + struct epoll_event __user **uevents, + struct list_head *txlist) +{ + struct epoll_event __user *next; + struct wakeup_source *ws; + __poll_t revents; + + /* + * Activate ep->ws before deactivating epi->ws to prevent + * triggering auto-suspend here (in case we reactivate epi->ws + * below). Rearranging to delay the deactivation would let + * epi->ws drift out of sync with ep_is_linked(). + */ + ws = ep_wakeup_source(epi); + if (ws) { + if (ws->active) + __pm_stay_awake(ep->ws); + __pm_relax(ws); + } + + list_del_init(&epi->rdllink); + + /* + * Re-poll under ep->mtx so userspace cannot change the item + * out from under us. If no caller-requested events remain, + * @epi stays off the ready list; the poll callback will + * re-queue it when events next appear. + */ + revents = ep_item_poll(epi, pt, 1); + if (!revents) + return 0; + + next = epoll_put_uevent(revents, epi->event.data, *uevents); + if (!next) { + /* + * copy_to_user() faulted: put the item back so + * ep_done_scan() splices it onto rdllist for the next + * attempt. + */ + list_add(&epi->rdllink, txlist); + ep_pm_stay_awake(epi); + return -EFAULT; + } + *uevents = next; + + if (epi->event.events & EPOLLONESHOT) { + epi->event.events &= EP_PRIVATE_BITS; + } else if (!(epi->event.events & EPOLLET)) { + /* + * Level-triggered: re-queue so the next epoll_wait() + * rechecks availability. We are the sole writer to + * rdllist here -- epoll_ctl() callers are locked out + * by ep->mtx, and the poll callback queues to ovflist + * during scans. + */ + list_add_tail(&epi->rdllink, &ep->rdllist); + ep_pm_stay_awake(epi); + } + return 1; +} + static int ep_send_events(struct eventpoll *ep, struct epoll_event __user *events, int maxevents) { @@ -2001,70 +2077,24 @@ static int ep_send_events(struct eventpoll *ep, ep_start_scan(ep, &txlist); /* - * We can loop without lock because we are passed a task private list. - * Items cannot vanish during the loop we are holding ep->mtx. + * We can loop without lock because we are passed a task-private + * txlist; items cannot vanish while we hold ep->mtx. */ list_for_each_entry_safe(epi, tmp, &txlist, rdllink) { - struct wakeup_source *ws; - __poll_t revents; + int delivered; if (res >= maxevents) break; - /* - * Activate ep->ws before deactivating epi->ws to prevent - * triggering auto-suspend here (in case we reactive epi->ws - * below). - * - * This could be rearranged to delay the deactivation of epi->ws - * instead, but then epi->ws would temporarily be out of sync - * with ep_is_linked(). - */ - ws = ep_wakeup_source(epi); - if (ws) { - if (ws->active) - __pm_stay_awake(ep->ws); - __pm_relax(ws); - } - - list_del_init(&epi->rdllink); - - /* - * If the event mask intersect the caller-requested one, - * deliver the event to userspace. Again, we are holding ep->mtx, - * so no operations coming from userspace can change the item. - */ - revents = ep_item_poll(epi, &pt, 1); - if (!revents) - continue; - - events = epoll_put_uevent(revents, epi->event.data, events); - if (!events) { - list_add(&epi->rdllink, &txlist); - ep_pm_stay_awake(epi); + delivered = ep_deliver_event(ep, epi, &pt, &events, &txlist); + if (delivered < 0) { if (!res) - res = -EFAULT; + res = delivered; break; } - res++; - if (epi->event.events & EPOLLONESHOT) - epi->event.events &= EP_PRIVATE_BITS; - else if (!(epi->event.events & EPOLLET)) { - /* - * If this file has been added with Level - * Trigger mode, we need to insert back inside - * the ready list, so that the next call to - * epoll_wait() will check again the events - * availability. At this point, no one can insert - * into ep->rdllist besides us. The epoll_ctl() - * callers are locked out by - * ep_send_events() holding "mtx" and the - * poll callback will queue them in ep->ovflist. - */ - list_add_tail(&epi->rdllink, &ep->rdllist); - ep_pm_stay_awake(epi); - } + res += delivered; } + ep_done_scan(ep, &txlist); mutex_unlock(&ep->mtx); From 8d5278e05b7426eea986db62fa7de3c0cfaaef8f Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:44 +0200 Subject: [PATCH 13/69] eventpoll: extract lock dance from do_epoll_ctl() into ep_ctl_lock() do_epoll_ctl() interleaved three concerns in one body: input validation, the ep->mtx + epnested_mutex acquisition dance for EPOLL_CTL_ADD on potentially-nested topologies, and the op dispatch with final unlock. The middle concern is the error-prone one; the error_tgt_fput label existed mainly to orchestrate it. Extract the acquisition as ep_ctl_lock() and the release as ep_ctl_unlock(). ep_ctl_lock() always takes ep->mtx and, for EPOLL_CTL_ADD on a topology that can change, additionally runs the loop / path check under epnested_mutex. The return value is a ternary: 0 ep->mtx held. 1 ep->mtx AND epnested_mutex held (full-check mode). -errno failure, no locks held. The non-negative value doubles as the @full_check argument to ep_insert() and as the argument to ep_ctl_unlock(), so the caller neither needs an out-parameter nor a separate boolean: full_check = ep_ctl_lock(ep, op, epfile, tfile, nonblock); if (full_check < 0) return full_check; ... ep_ctl_unlock(ep, full_check); ep_ctl_unlock() drops ep->mtx and, if full_check == 1, clears tfile_check_list, bumps loop_check_gen, and drops epnested_mutex -- mirroring the old error_tgt_fput block. With that in place do_epoll_ctl()'s preconditions become direct returns (no locks held, nothing to clean up), the acquisition is a single call, the op dispatch is unchanged, and the epilogue is a single ep_ctl_unlock() before return. The error_tgt_fput label goes away. The two loop_check_gen bumps (one at the start of the full check, one after) are preserved inside ep_ctl_lock() / ep_ctl_unlock(), keeping the invariant that ep->gen stamps left on per-eventpoll caches never equal loop_check_gen after the check completes. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-13-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 152 +++++++++++++++++++++++++++++-------------------- 1 file changed, 91 insertions(+), 61 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 6d4167a347ab..d49457dc8c7f 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -2479,14 +2479,92 @@ static inline int epoll_mutex_lock(struct mutex *mutex, bool nonblock) return mutex_trylock(mutex) ? 0 : -EAGAIN; } +/* + * Acquire the locks required for do_epoll_ctl() on @ep for @op. + * + * Always takes ep->mtx. For EPOLL_CTL_ADD, additionally runs the + * loop / path check under epnested_mutex when the topology can + * change: @ep is already watched (epfile->f_ep non-NULL), @ep was + * recently loop-checked (ep->gen == loop_check_gen), or @tfile is + * itself an eventpoll. + * + * Return value encodes both outcome and lock state: + * + * 0 success; ep->mtx held. + * 1 success; ep->mtx held AND the full check ran under + * epnested_mutex (which is also still held). The value + * doubles as the @full_check argument to ep_insert(). + * -errno failure; no locks held. + * + * The caller releases what was taken with ep_ctl_unlock(ep, ret). + * + * Holding epnested_mutex on add is what prevents two racing + * EPOLL_CTL_ADDs on different eps from building a cycle without + * either walker observing it. + */ +static int ep_ctl_lock(struct eventpoll *ep, int op, + struct file *epfile, struct file *tfile, + bool nonblock) +{ + struct eventpoll *tep; + int error; + + error = epoll_mutex_lock(&ep->mtx, nonblock); + if (error) + return error; + + if (op != EPOLL_CTL_ADD) + return 0; + if (!READ_ONCE(epfile->f_ep) && ep->gen != loop_check_gen && + !is_file_epoll(tfile)) + return 0; + + /* Full check needed: drop ep->mtx so we can take epnested_mutex. */ + mutex_unlock(&ep->mtx); + error = epoll_mutex_lock(&epnested_mutex, nonblock); + if (error) + return error; + + loop_check_gen++; + + if (is_file_epoll(tfile)) { + tep = tfile->private_data; + if (ep_loop_check(ep, tep) != 0) { + error = -ELOOP; + goto err_unlock_nested; + } + } + + error = epoll_mutex_lock(&ep->mtx, nonblock); + if (error) + goto err_unlock_nested; + + return 1; + +err_unlock_nested: + clear_tfile_check_list(); + loop_check_gen++; + mutex_unlock(&epnested_mutex); + return error; +} + +static void ep_ctl_unlock(struct eventpoll *ep, int full_check) +{ + mutex_unlock(&ep->mtx); + if (full_check) { + clear_tfile_check_list(); + loop_check_gen++; + mutex_unlock(&epnested_mutex); + } +} + int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, bool nonblock) { int error; - int full_check = 0; + int full_check; struct eventpoll *ep; struct epitem *epi; - struct eventpoll *tep = NULL; CLASS(fd, f)(epfd); if (fd_empty(f)) @@ -2506,76 +2584,34 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, ep_take_care_of_epollwakeup(epds); /* - * We have to check that the file structure underneath the file descriptor - * the user passed to us _is_ an eventpoll file. And also we do not permit + * The @epfd file must itself be an eventpoll, and we do not permit * adding an epoll file descriptor inside itself. */ - error = -EINVAL; if (fd_file(f) == fd_file(tf) || !is_file_epoll(fd_file(f))) - goto error_tgt_fput; + return -EINVAL; /* * epoll adds to the wakeup queue at EPOLL_CTL_ADD time only, * so EPOLLEXCLUSIVE is not allowed for a EPOLL_CTL_MOD operation. - * Also, we do not currently supported nested exclusive wakeups. + * Also, nested exclusive wakeups are not supported. */ if (ep_op_has_event(op) && (epds->events & EPOLLEXCLUSIVE)) { if (op == EPOLL_CTL_MOD) - goto error_tgt_fput; + return -EINVAL; if (op == EPOLL_CTL_ADD && (is_file_epoll(fd_file(tf)) || (epds->events & ~EPOLLEXCLUSIVE_OK_BITS))) - goto error_tgt_fput; + return -EINVAL; } - /* - * At this point it is safe to assume that the "private_data" contains - * our own data structure. - */ ep = fd_file(f)->private_data; - /* - * When we insert an epoll file descriptor inside another epoll file - * descriptor, there is the chance of creating closed loops, which are - * better be handled here, than in more critical paths. While we are - * checking for loops we also determine the list of files reachable - * and hang them on the tfile_check_list, so we can check that we - * haven't created too many possible wakeup paths. - * - * We do not need to take the global 'epumutex' on EPOLL_CTL_ADD when - * the epoll file descriptor is attaching directly to a wakeup source, - * unless the epoll file descriptor is nested. The purpose of taking the - * 'epnested_mutex' on add is to prevent complex toplogies such as loops and - * deep wakeup paths from forming in parallel through multiple - * EPOLL_CTL_ADD operations. - */ - error = epoll_mutex_lock(&ep->mtx, nonblock); - if (error) - goto error_tgt_fput; - if (op == EPOLL_CTL_ADD) { - if (READ_ONCE(fd_file(f)->f_ep) || ep->gen == loop_check_gen || - is_file_epoll(fd_file(tf))) { - mutex_unlock(&ep->mtx); - error = epoll_mutex_lock(&epnested_mutex, nonblock); - if (error) - goto error_tgt_fput; - loop_check_gen++; - full_check = 1; - if (is_file_epoll(fd_file(tf))) { - tep = fd_file(tf)->private_data; - error = -ELOOP; - if (ep_loop_check(ep, tep) != 0) - goto error_tgt_fput; - } - error = epoll_mutex_lock(&ep->mtx, nonblock); - if (error) - goto error_tgt_fput; - } - } + full_check = ep_ctl_lock(ep, op, fd_file(f), fd_file(tf), nonblock); + if (full_check < 0) + return full_check; /* - * Try to lookup the file inside our RB tree. Since we grabbed "mtx" - * above, we can be sure to be able to use the item looked up by - * ep_find() till we release the mutex. + * Look the target up in ep's RB tree. We hold ep->mtx, so the + * item stays valid until we release. */ epi = ep_find(ep, fd_file(tf), fd); @@ -2610,14 +2646,8 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, error = -ENOENT; break; } - mutex_unlock(&ep->mtx); -error_tgt_fput: - if (full_check) { - clear_tfile_check_list(); - loop_check_gen++; - mutex_unlock(&epnested_mutex); - } + ep_ctl_unlock(ep, full_check); return error; } From 0dcb726466a556f273beaadfb76f12d1b0087dd7 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:45 +0200 Subject: [PATCH 14/69] eventpoll: wrap EP_UNACTIVE_PTR in typed sentinel helpers ep->ovflist and epi->next both use EP_UNACTIVE_PTR (a cast to (void *)-1) as a sentinel, with distinct meanings at each site: ep->ovflist == EP_UNACTIVE_PTR no scan in progress epi->next == EP_UNACTIVE_PTR epi not on ovflist Call sites had to know the sentinel's value and, by convention, what it meant in each context. Hide both behind inline helpers: ep_is_scanning(ep) predicate for "scan in progress" ep_enter_scan(ep) WRITE_ONCE flip to NULL (scan start) ep_exit_scan(ep) WRITE_ONCE flip to sentinel (scan end) epi_on_ovflist(epi) predicate for "epi is on ovflist" epi_clear_ovflist(epi) clear epi's ovflist link slot Convert ep_events_available(), ep_start_scan(), ep_done_scan(), ep_poll_callback(), and ep_alloc_epitem() to use the wrappers. The ovflist state-machine transitions are now named, not encoded in sentinel comparisons, and the top-of-file "Ready-list state machine" section is the single place that spells out the sentinel's meaning. ep_alloc() keeps the raw "ep->ovflist = EP_UNACTIVE_PTR" init (no concurrent access at that point) with an inline "not scanning" comment, and the tfile_check_list sentinel is left alone -- it will disappear entirely when the loop-check globals move into a stack-allocated ep_ctl_ctx in a later commit. Also rework ep_done_scan()'s for-loop: the combined initializer + update clause that advanced nepi AND cleared epi->next in one step was clever but hard to read; splitting the update into two statements inside the body makes the epi_clear_ovflist() call visible. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-14-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 73 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 52 insertions(+), 21 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index d49457dc8c7f..4199ef8e42e5 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -541,6 +541,43 @@ static inline struct epitem *ep_item_from_wait(wait_queue_entry_t *p) return container_of(p, struct eppoll_entry, wait)->base; } +/* + * Ready-list / ovflist state (see "Ready-list state machine" in the + * top-of-file banner for the full state machine). EP_UNACTIVE_PTR is + * the sentinel; these wrappers name each transition and each test so + * call sites do not need to know the sentinel's value. + */ + +/* True iff @ep is between ep_enter_scan() and ep_exit_scan(). */ +static inline bool ep_is_scanning(struct eventpoll *ep) +{ + return READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR; +} + +/* Called by ep_start_scan(): divert ep_poll_callback() to ovflist. */ +static inline void ep_enter_scan(struct eventpoll *ep) +{ + WRITE_ONCE(ep->ovflist, NULL); +} + +/* Called by ep_done_scan(): redirect ep_poll_callback() back to rdllist. */ +static inline void ep_exit_scan(struct eventpoll *ep) +{ + WRITE_ONCE(ep->ovflist, EP_UNACTIVE_PTR); +} + +/* True iff @epi is currently linked on its ep's ovflist. */ +static inline bool epi_on_ovflist(const struct epitem *epi) +{ + return epi->next != EP_UNACTIVE_PTR; +} + +/* Mark @epi as not on any ovflist (init and post-drain). */ +static inline void epi_clear_ovflist(struct epitem *epi) +{ + epi->next = EP_UNACTIVE_PTR; +} + /** * ep_events_available - Checks if ready events might be available. * @@ -551,8 +588,7 @@ static inline struct epitem *ep_item_from_wait(wait_queue_entry_t *p) */ static inline int ep_events_available(struct eventpoll *ep) { - return !list_empty_careful(&ep->rdllist) || - READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR; + return !list_empty_careful(&ep->rdllist) || ep_is_scanning(ep); } #ifdef CONFIG_NET_RX_BUSY_POLL @@ -910,7 +946,7 @@ static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist) lockdep_assert_irqs_enabled(); spin_lock_irq(&ep->lock); list_splice_init(&ep->rdllist, txlist); - WRITE_ONCE(ep->ovflist, NULL); + ep_enter_scan(ep); spin_unlock_irq(&ep->lock); } @@ -925,29 +961,24 @@ static void ep_done_scan(struct eventpoll *ep, * other events might have been queued by the poll callback. * We re-insert them inside the main ready-list here. */ - for (nepi = READ_ONCE(ep->ovflist); (epi = nepi) != NULL; - nepi = epi->next, epi->next = EP_UNACTIVE_PTR) { + for (nepi = READ_ONCE(ep->ovflist); (epi = nepi) != NULL; ) { + nepi = epi->next; + epi_clear_ovflist(epi); /* - * We need to check if the item is already in the list. - * During the "sproc" callback execution time, items are - * queued into ->ovflist but the "txlist" might already - * contain them, and the list_splice() below takes care of them. + * Skip items that the caller already returned via @txlist + * -- the list_splice() below takes care of those. */ if (!ep_is_linked(epi)) { /* - * ->ovflist is LIFO, so we have to reverse it in order - * to keep in FIFO. + * ovflist is LIFO; list_add() head-insert here + * reverses the iteration order into FIFO. */ list_add(&epi->rdllink, &ep->rdllist); ep_pm_stay_awake(epi); } } - /* - * We need to set back ep->ovflist to EP_UNACTIVE_PTR, so that after - * releasing the lock, events will be queued in the normal way inside - * ep->rdllist. - */ - WRITE_ONCE(ep->ovflist, EP_UNACTIVE_PTR); + /* Back out of scan mode; callbacks target ep->rdllist again. */ + ep_exit_scan(ep); /* * Quickly re-inject items left on "txlist". @@ -1376,7 +1407,7 @@ static int ep_alloc(struct eventpoll **pep) init_waitqueue_head(&ep->poll_wait); INIT_LIST_HEAD(&ep->rdllist); ep->rbr = RB_ROOT_CACHED; - ep->ovflist = EP_UNACTIVE_PTR; + ep->ovflist = EP_UNACTIVE_PTR; /* not scanning */ ep->user = get_current_user(); refcount_set(&ep->refcount, 1); @@ -1456,8 +1487,8 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v * semantics). All the events that happen during that period of time are * chained in ep->ovflist and requeued later on. */ - if (READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR) { - if (epi->next == EP_UNACTIVE_PTR) { + if (ep_is_scanning(ep)) { + if (!epi_on_ovflist(epi)) { epi->next = READ_ONCE(ep->ovflist); WRITE_ONCE(ep->ovflist, epi); ep_pm_stay_awake_rcu(epi); @@ -1771,7 +1802,7 @@ static struct epitem *ep_alloc_epitem(struct eventpoll *ep, epi->ep = ep; ep_set_ffd(&epi->ffd, tfile, fd); epi->event = *event; - epi->next = EP_UNACTIVE_PTR; + epi_clear_ovflist(epi); return epi; } From f38567bb63ae029e3b63fcb99b6a2dcc6f421e69 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:46 +0200 Subject: [PATCH 15/69] eventpoll: rename epi->next and txlist for clarity Two list-related names were confusing in isolation: struct epitem::next A singly-linked link slot used only when an epi is queued on ep->ovflist during an ep_start_scan/ep_done_scan window. The bare name "next" suggests a generic list link and doesn't say which list it belongs to. txlist The caller-local list_head used by ep_send_events() and __ep_eventpoll_poll() to hold the batch of items stolen from ep->rdllist for the current scan. "txlist" ("transmission list") is abbreviated and overloaded: it doesn't distinguish itself from ep->rdllist or ep->ovflist at a glance. Rename for what each actually is: struct epitem::next -> struct epitem::ovflist_next local txlist -> scan_batch With these in place: - epi->ovflist_next reads as "this is the ep->ovflist link slot", matching the rdllink pattern above it. - scan_batch reads as "the batch currently being scanned", clearly distinct from rdllist (canonical ready list) and ovflist (scan-window overflow). ep->rdllist and ep->ovflist struct field names are preserved -- they are long-standing interface-facing identifiers, and the new inline helpers (ep_is_scanning, epi_on_ovflist, ...) already hide the sentinel semantics at call sites. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-15-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 62 ++++++++++++++++++++++++++------------------------ 1 file changed, 32 insertions(+), 30 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 4199ef8e42e5..7ed4b47279ff 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -142,15 +142,15 @@ * NULL - scan active, no spill yet. * pointer to epi - scan active with spilled items (LIFO). * - * Encoded in epi->next: + * Encoded in epi->ovflist_next: * EP_UNACTIVE_PTR - epi is not on ovflist. * otherwise - next epi on ovflist (NULL at tail). * * ep_start_scan() flips "not scanning" to "scanning" and splices - * rdllist into a caller-local txlist. ep_done_scan() drains ovflist + * rdllist into a caller-local scan_batch. ep_done_scan() drains ovflist * back to rdllist (list_add head-insert reverses LIFO to FIFO), * flips back to "not scanning", and re-splices any items the caller - * left in txlist (e.g., level-triggered re-queues). + * left in scan_batch (e.g., level-triggered re-queues). * * * Removal paths @@ -261,14 +261,16 @@ struct epitem { struct rcu_head rcu; }; - /* List header used to link this structure to the eventpoll ready list */ + /* Link on the owning eventpoll's ready list (ep->rdllist). */ struct list_head rdllink; /* - * Works together "struct eventpoll"->ovflist in keeping the - * single linked chain of items. + * Link on the owning eventpoll's scan-overflow list (ep->ovflist), + * EP_UNACTIVE_PTR when not linked. See epi_on_ovflist() / + * epi_clear_ovflist() and the "Ready-list state machine" section + * in the top-of-file banner. */ - struct epitem *next; + struct epitem *ovflist_next; /* The file descriptor information this item refers to */ struct epoll_filefd ffd; @@ -569,13 +571,13 @@ static inline void ep_exit_scan(struct eventpoll *ep) /* True iff @epi is currently linked on its ep's ovflist. */ static inline bool epi_on_ovflist(const struct epitem *epi) { - return epi->next != EP_UNACTIVE_PTR; + return epi->ovflist_next != EP_UNACTIVE_PTR; } /* Mark @epi as not on any ovflist (init and post-drain). */ static inline void epi_clear_ovflist(struct epitem *epi) { - epi->next = EP_UNACTIVE_PTR; + epi->ovflist_next = EP_UNACTIVE_PTR; } /** @@ -933,7 +935,7 @@ static inline void ep_pm_stay_awake_rcu(struct epitem *epi) * ep->mutex needs to be held because we could be hit by * eventpoll_release_file() and epoll_ctl(). */ -static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist) +static void ep_start_scan(struct eventpoll *ep, struct list_head *scan_batch) { /* * Steal the ready list, and re-init the original one to the @@ -945,13 +947,13 @@ static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist) */ lockdep_assert_irqs_enabled(); spin_lock_irq(&ep->lock); - list_splice_init(&ep->rdllist, txlist); + list_splice_init(&ep->rdllist, scan_batch); ep_enter_scan(ep); spin_unlock_irq(&ep->lock); } static void ep_done_scan(struct eventpoll *ep, - struct list_head *txlist) + struct list_head *scan_batch) { struct epitem *epi, *nepi; @@ -962,10 +964,10 @@ static void ep_done_scan(struct eventpoll *ep, * We re-insert them inside the main ready-list here. */ for (nepi = READ_ONCE(ep->ovflist); (epi = nepi) != NULL; ) { - nepi = epi->next; + nepi = epi->ovflist_next; epi_clear_ovflist(epi); /* - * Skip items that the caller already returned via @txlist + * Skip items that the caller already returned via @scan_batch * -- the list_splice() below takes care of those. */ if (!ep_is_linked(epi)) { @@ -981,9 +983,9 @@ static void ep_done_scan(struct eventpoll *ep, ep_exit_scan(ep); /* - * Quickly re-inject items left on "txlist". + * Quickly re-inject items left on "scan_batch". */ - list_splice(txlist, &ep->rdllist); + list_splice(scan_batch, &ep->rdllist); __pm_relax(ep->ws); if (!list_empty(&ep->rdllist)) { @@ -1247,7 +1249,7 @@ static __poll_t ep_item_poll(const struct epitem *epi, poll_table *pt, int depth static __poll_t __ep_eventpoll_poll(struct file *file, poll_table *wait, int depth) { struct eventpoll *ep = file->private_data; - LIST_HEAD(txlist); + LIST_HEAD(scan_batch); struct epitem *epi, *tmp; poll_table pt; __poll_t res = 0; @@ -1262,8 +1264,8 @@ static __poll_t __ep_eventpoll_poll(struct file *file, poll_table *wait, int dep * the ready list. */ mutex_lock_nested(&ep->mtx, depth); - ep_start_scan(ep, &txlist); - list_for_each_entry_safe(epi, tmp, &txlist, rdllink) { + ep_start_scan(ep, &scan_batch); + list_for_each_entry_safe(epi, tmp, &scan_batch, rdllink) { if (ep_item_poll(epi, &pt, depth + 1)) { res = EPOLLIN | EPOLLRDNORM; break; @@ -1277,7 +1279,7 @@ static __poll_t __ep_eventpoll_poll(struct file *file, poll_table *wait, int dep list_del_init(&epi->rdllink); } } - ep_done_scan(ep, &txlist); + ep_done_scan(ep, &scan_batch); mutex_unlock(&ep->mtx); return res; } @@ -1489,7 +1491,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v */ if (ep_is_scanning(ep)) { if (!epi_on_ovflist(epi)) { - epi->next = READ_ONCE(ep->ovflist); + epi->ovflist_next = READ_ONCE(ep->ovflist); WRITE_ONCE(ep->ovflist, epi); ep_pm_stay_awake_rcu(epi); } @@ -2017,7 +2019,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, * next slot), 0 if the re-poll reported no caller-requested events * (@epi drops out of the ready list; a future callback will re-add * it), or -EFAULT if copy_to_user() faulted (in which case @epi is - * re-inserted at the head of @txlist so ep_done_scan() merges it + * re-inserted at the head of @scan_batch so ep_done_scan() merges it * back to rdllist for the next attempt). * * PM bookkeeping and level-triggered re-queue are handled here. @@ -2026,7 +2028,7 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi, static int ep_deliver_event(struct eventpoll *ep, struct epitem *epi, poll_table *pt, struct epoll_event __user **uevents, - struct list_head *txlist) + struct list_head *scan_batch) { struct epoll_event __user *next; struct wakeup_source *ws; @@ -2064,7 +2066,7 @@ static int ep_deliver_event(struct eventpoll *ep, struct epitem *epi, * ep_done_scan() splices it onto rdllist for the next * attempt. */ - list_add(&epi->rdllink, txlist); + list_add(&epi->rdllink, scan_batch); ep_pm_stay_awake(epi); return -EFAULT; } @@ -2090,7 +2092,7 @@ static int ep_send_events(struct eventpoll *ep, struct epoll_event __user *events, int maxevents) { struct epitem *epi, *tmp; - LIST_HEAD(txlist); + LIST_HEAD(scan_batch); poll_table pt; int res = 0; @@ -2105,19 +2107,19 @@ static int ep_send_events(struct eventpoll *ep, init_poll_funcptr(&pt, NULL); mutex_lock(&ep->mtx); - ep_start_scan(ep, &txlist); + ep_start_scan(ep, &scan_batch); /* * We can loop without lock because we are passed a task-private - * txlist; items cannot vanish while we hold ep->mtx. + * scan_batch; items cannot vanish while we hold ep->mtx. */ - list_for_each_entry_safe(epi, tmp, &txlist, rdllink) { + list_for_each_entry_safe(epi, tmp, &scan_batch, rdllink) { int delivered; if (res >= maxevents) break; - delivered = ep_deliver_event(ep, epi, &pt, &events, &txlist); + delivered = ep_deliver_event(ep, epi, &pt, &events, &scan_batch); if (delivered < 0) { if (!res) res = delivered; @@ -2126,7 +2128,7 @@ static int ep_send_events(struct eventpoll *ep, res += delivered; } - ep_done_scan(ep, &txlist); + ep_done_scan(ep, &scan_batch); mutex_unlock(&ep->mtx); return res; From 23431ae93a21d485f66a177304de353113dcdd28 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:47 +0200 Subject: [PATCH 16/69] eventpoll: use bool for predicate helpers Three inline predicates -- is_file_epoll(), ep_is_linked(), ep_events_available() -- were declared to return int even though their only use is as a truthy test and their bodies are already boolean expressions. ep_has_wakeup_source(), in the same file, returns bool, so the convention was already inconsistent. Convert all three to return bool. Rewrite ep_events_available()'s verbose kerneldoc to the same one-line style the rest of the predicates use now. ep_poll()'s local eavail variable stores the result of ep_events_available() (already boolean), ep_busy_loop() (returns bool), and list_empty() (int but tested as boolean). Split it out of the combined int declaration and give it bool type; replace the one "eavail = 1" after a wakeup with "eavail = true" to match. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-16-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 7ed4b47279ff..201e688304b3 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -505,7 +505,7 @@ static void __init epoll_sysctls_init(void) static const struct file_operations eventpoll_fops; -static inline int is_file_epoll(struct file *f) +static inline bool is_file_epoll(struct file *f) { return f->f_op == &eventpoll_fops; } @@ -526,8 +526,8 @@ static inline int ep_cmp_ffd(struct epoll_filefd *p1, (p1->file < p2->file ? -1 : p1->fd - p2->fd)); } -/* Tells us if the item is currently linked */ -static inline int ep_is_linked(struct epitem *epi) +/* True iff @epi is on its owning ep's ready list. */ +static inline bool ep_is_linked(struct epitem *epi) { return !list_empty(&epi->rdllink); } @@ -580,15 +580,8 @@ static inline void epi_clear_ovflist(struct epitem *epi) epi->ovflist_next = EP_UNACTIVE_PTR; } -/** - * ep_events_available - Checks if ready events might be available. - * - * @ep: Pointer to the eventpoll context. - * - * Return: a value different than %zero if ready events are available, - * or %zero otherwise. - */ -static inline int ep_events_available(struct eventpoll *ep) +/* True iff @ep has ready events that epoll_wait() might harvest. */ +static inline bool ep_events_available(struct eventpoll *ep) { return !list_empty_careful(&ep->rdllist) || ep_is_scanning(ep); } @@ -2218,7 +2211,8 @@ static int ep_schedule_timeout(ktime_t *to) static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, int maxevents, struct timespec64 *timeout) { - int res, eavail, timed_out = 0; + int res, timed_out = 0; + bool eavail; u64 slack = 0; wait_queue_entry_t wait; ktime_t expires, *to = NULL; @@ -2316,7 +2310,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * If timed out and still on the wait queue, recheck eavail * carefully under lock, below. */ - eavail = 1; + eavail = true; if (!list_empty_careful(&wait.entry)) { spin_lock_irq(&ep->lock); From e09c77d940034e3f0c4ae1c18cc655ea9ec2cc41 Mon Sep 17 00:00:00 2001 From: Christian Brauner Date: Fri, 24 Apr 2026 15:46:48 +0200 Subject: [PATCH 17/69] eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx Three globals were shared between the loop check and the path check paths: tfile_check_list (chain of epitems_head to walk afterwards), path_count[] (per-depth wakeup-path tally) and inserting_into (cycle-detection sentinel). All three are scratch state used only during a single EPOLL_CTL_ADD full_check, yet they sit at file scope and rely on epnested_mutex for exclusion. The area has had three bugs in the last year -- CVE-2025-38349, f2e467a48287 ("eventpoll: Fix semi-unbounded recursion"), and fdcfce93073d ("eventpoll: Fix integer overflow in ep_loop_check_proc()") -- all rooted in the shared-mutable-global pattern being hard to reason about. Collect the three into a stack-allocated struct ep_ctl_ctx: struct ep_ctl_ctx { struct eventpoll *inserting_into; struct epitems_head *tfile_check_list; int path_count[PATH_ARR_SIZE]; }; do_epoll_ctl() zero-initializes one on its stack and plumbs it through ep_ctl_lock() / ep_ctl_unlock() / ep_insert() / ep_register_epitem() / list_file() / ep_loop_check() / ep_loop_check_proc() / reverse_path_check() / reverse_path_check_proc() / path_count_inc() / path_count_init() / clear_tfile_check_list(). Non-nested inserts leave the ctx zeroed and skip the machinery entirely. With the scratch state in ctx: - tfile_check_list no longer has an EP_UNACTIVE_PTR sentinel -- NULL is the obvious "empty" value and the zero-init handles it for free; - path_count[] is no longer an array global that could be touched in unexpected orderings; - inserting_into is scoped to the exact call that set it. loop_check_gen stays as a file-scope monotonic counter, because the stamp left on ep->gen by a completed walk must not equal the stamp of a future walk -- something a stack-local value cannot guarantee across calls. It remains protected by epnested_mutex for the bump and read lockless for the "do we need a full check" trigger in ep_ctl_lock(). Every bail-out that existed before (the ELOOP on cycle, the path limit check, the unbounded-recursion cap, the +1 overflow guard) is preserved verbatim; only the data they operate on moved from file scope to the stack ctx. No functional change. Signed-off-by: Christian Brauner (Amutable) Link: https://patch.msgid.link/20260424-work-epoll-rework-v1-17-249ed00a20f3@kernel.org Signed-off-by: Christian Brauner --- fs/eventpoll.c | 204 ++++++++++++++++++++++++++++--------------------- 1 file changed, 116 insertions(+), 88 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 201e688304b3..b839cc02eb0e 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -388,32 +388,25 @@ static long max_user_watches __read_mostly; * of a given length -- reverse_path_check(). * * Both need a global view of the epoll topology and must be atomic - * with the insertion, so the scratch state below is all serialized by - * one global mutex, epnested_mutex. Non-nested inserts skip this - * machinery entirely and take only ep->mtx. + * with the insertion, so the check is serialized by epnested_mutex + * and carries its scratch state on a stack-allocated struct + * ep_ctl_ctx scoped to one do_epoll_ctl() call. Non-nested inserts + * skip this machinery entirely and take only ep->mtx. * - * epnested_mutex Serializes the whole check; also protects every - * other variable in this block plus path_count[] - * (declared with the path-check code further - * down). - * loop_check_gen Monotonic stamp, bumped once at the start of a - * check and once at the end. ep->gen caches the - * value under which ep was last visited by + * epnested_mutex Serializes the whole check. + * loop_check_gen Global monotonic stamp, bumped at the start of + * a check and again at the end. ep->gen caches + * the value under which ep was last visited by * ep_loop_check_proc() or * ep_get_upwards_depth_proc(); the post-check * bump ensures those cached stamps can no longer * equal loop_check_gen, so the * "ep->gen == loop_check_gen" trigger in - * do_epoll_ctl() only fires while another check + * ep_ctl_lock() only fires while another check * is in flight. - * inserting_into Outer eventpoll pointer for the lifetime of one - * ep_loop_check(); ep_loop_check_proc() fails - * with -ELOOP if the downward walk reaches it. - * tfile_check_list Singly-linked list of epitems_head objects - * collected by ep_loop_check_proc() during the - * walk, consumed by reverse_path_check() - * afterwards. Sentinel EP_UNACTIVE_PTR means no - * check is in flight. + * + * struct ep_ctl_ctx carries the rest (inserting_into, tfile_check_list, + * path_count[]) through the walk; see its declaration below. * * Commits fdcfce93073d ("eventpoll: Fix integer overflow in * ep_loop_check_proc()") and f2e467a48287 ("eventpoll: Fix @@ -422,7 +415,36 @@ static long max_user_watches __read_mostly; */ static DEFINE_MUTEX(epnested_mutex); static u64 loop_check_gen = 0; -static struct eventpoll *inserting_into; + +#define PATH_ARR_SIZE 5 + +/* + * Per-do_epoll_ctl() scratch for the loop / path checks. Allocated on + * the caller's stack; populated by ep_ctl_lock() and the downward + * walk; consumed by reverse_path_check(); released by ep_ctl_unlock(). + * Only valid while the caller holds epnested_mutex. + */ +struct ep_ctl_ctx { + /* + * Outer eventpoll for one ep_loop_check(); if the downward walk + * reaches it the insert would form a cycle. + */ + struct eventpoll *inserting_into; + + /* + * Singly-linked list of epitems_head objects collected during + * ep_loop_check_proc(), then walked by reverse_path_check(). + * NULL means empty. + */ + struct epitems_head *tfile_check_list; + + /* + * Per-depth wakeup-path tally used by reverse_path_check_proc(); + * reinitialized to zero at the start of each reverse_path_check() + * iteration. + */ + int path_count[PATH_ARR_SIZE]; +}; /* Slab cache used to allocate "struct epitem" */ static struct kmem_cache *epi_cache __ro_after_init; @@ -434,13 +456,12 @@ static struct kmem_cache *pwq_cache __ro_after_init; * Wrapper anchor for file->f_ep when the watched file is not itself an * eventpoll; for the epoll-watches-epoll case, file->f_ep points at * &watched_ep->refs directly. The ->next field threads - * tfile_check_list during one EPOLL_CTL_ADD path check. + * ctx->tfile_check_list during one EPOLL_CTL_ADD path check. */ struct epitems_head { struct hlist_head epitems; struct epitems_head *next; }; -static struct epitems_head *tfile_check_list = EP_UNACTIVE_PTR; static struct kmem_cache *ephead_cache __ro_after_init; @@ -450,14 +471,14 @@ static inline void free_ephead(struct epitems_head *head) kmem_cache_free(ephead_cache, head); } -static void list_file(struct file *file) +static void list_file(struct file *file, struct ep_ctl_ctx *ctx) { struct epitems_head *head; head = container_of(file->f_ep, struct epitems_head, epitems); if (!head->next) { - head->next = tfile_check_list; - tfile_check_list = head; + head->next = ctx->tfile_check_list; + ctx->tfile_check_list = head; } } @@ -1613,41 +1634,40 @@ static void ep_rbtree_insert(struct eventpoll *ep, struct epitem *epi) -#define PATH_ARR_SIZE 5 /* - * These are the number paths of length 1 to 5, that we are allowing to emanate - * from a single file of interest. For example, we allow 1000 paths of length - * 1, to emanate from each file of interest. This essentially represents the - * potential wakeup paths, which need to be limited in order to avoid massive - * uncontrolled wakeup storms. The common use case should be a single ep which - * is connected to n file sources. In this case each file source has 1 path - * of length 1. Thus, the numbers below should be more than sufficient. These - * path limits are enforced during an EPOLL_CTL_ADD operation, since a modify - * and delete can't add additional paths. Protected by the epnested_mutex. + * Upper bound on wakeup paths emanating from any one watched file, + * indexed by path depth (1..PATH_ARR_SIZE). For example, we allow + * 1000 paths of length 1 from each watched file. These caps limit + * the wakeup amplification that can be built from epoll-watches- + * epoll topologies without rejecting reasonable usage. + * + * Enforced at EPOLL_CTL_ADD; CTL_MOD and CTL_DEL cannot add paths. + * The running tallies live in ctx->path_count[] and are protected by + * epnested_mutex. */ static const int path_limits[PATH_ARR_SIZE] = { 1000, 500, 100, 50, 10 }; -static int path_count[PATH_ARR_SIZE]; -static int path_count_inc(int nests) +static int path_count_inc(struct ep_ctl_ctx *ctx, int nests) { /* Allow an arbitrary number of depth 1 paths */ if (nests == 0) return 0; - if (++path_count[nests] > path_limits[nests]) + if (++ctx->path_count[nests] > path_limits[nests]) return -1; return 0; } -static void path_count_init(void) +static void path_count_init(struct ep_ctl_ctx *ctx) { int i; for (i = 0; i < PATH_ARR_SIZE; i++) - path_count[i] = 0; + ctx->path_count[i] = 0; } -static int reverse_path_check_proc(struct hlist_head *refs, int depth) +static int reverse_path_check_proc(struct ep_ctl_ctx *ctx, + struct hlist_head *refs, int depth) { int error = 0; struct epitem *epi; @@ -1659,9 +1679,9 @@ static int reverse_path_check_proc(struct hlist_head *refs, int depth) hlist_for_each_entry_rcu(epi, refs, fllink) { struct hlist_head *refs = &epi->ep->refs; if (hlist_empty(refs)) - error = path_count_inc(depth); + error = path_count_inc(ctx, depth); else - error = reverse_path_check_proc(refs, depth + 1); + error = reverse_path_check_proc(ctx, refs, depth + 1); if (error != 0) break; } @@ -1669,24 +1689,23 @@ static int reverse_path_check_proc(struct hlist_head *refs, int depth) } /** - * reverse_path_check - The tfile_check_list is list of epitem_head, which have - * links that are proposed to be newly added. We need to - * make sure that those added links don't add too many - * paths such that we will spend all our time waking up - * eventpoll objects. + * reverse_path_check - ctx->tfile_check_list is a list of epitems_head + * anchoring files with newly proposed links; make + * sure those links don't push any path-length bucket + * over its limit in path_limits[]. * * Return: %zero if the proposed links don't create too many paths, * %-1 otherwise. */ -static int reverse_path_check(void) +static int reverse_path_check(struct ep_ctl_ctx *ctx) { struct epitems_head *p; - for (p = tfile_check_list; p != EP_UNACTIVE_PTR; p = p->next) { + for (p = ctx->tfile_check_list; p; p = p->next) { int error; - path_count_init(); + path_count_init(ctx); rcu_read_lock(); - error = reverse_path_check_proc(&p->epitems, 0); + error = reverse_path_check_proc(ctx, &p->epitems, 0); rcu_read_unlock(); if (error) return error; @@ -1817,8 +1836,9 @@ static struct epitem *ep_alloc_epitem(struct eventpoll *ep, * unwind; that cannot drop @ep's refcount to zero because the ep file * itself still holds the original reference. */ -static int ep_register_epitem(struct eventpoll *ep, struct epitem *epi, - struct eventpoll *tep, int full_check) +static int ep_register_epitem(struct ep_ctl_ctx *ctx, struct eventpoll *ep, + struct epitem *epi, struct eventpoll *tep, + int full_check) { struct file *tfile = epi->ffd.file; int error; @@ -1836,7 +1856,7 @@ static int ep_register_epitem(struct eventpoll *ep, struct epitem *epi, } if (full_check && !tep) - list_file(tfile); + list_file(tfile, ctx); ep_rbtree_insert(ep, epi); @@ -1850,8 +1870,9 @@ static int ep_register_epitem(struct eventpoll *ep, struct epitem *epi, /* * Must be called with "mtx" held. */ -static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, - struct file *tfile, int fd, int full_check) +static int ep_insert(struct ep_ctl_ctx *ctx, struct eventpoll *ep, + const struct epoll_event *event, struct file *tfile, + int fd, int full_check) { int error, pwake = 0; __poll_t revents; @@ -1868,12 +1889,12 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, if (IS_ERR(epi)) return PTR_ERR(epi); - error = ep_register_epitem(ep, epi, tep, full_check); + error = ep_register_epitem(ctx, ep, epi, tep, full_check); if (error) return error; /* Reject the insert if the new link would create too many back-paths. */ - if (unlikely(full_check && reverse_path_check())) { + if (unlikely(full_check && reverse_path_check(ctx))) { ep_remove(ep, epi); return -EINVAL; } @@ -2340,7 +2361,8 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * Return: depth of the subtree, or a value bigger than EP_MAX_NESTS if we found * a loop or went too deep. */ -static int ep_loop_check_proc(struct eventpoll *ep, int depth) +static int ep_loop_check_proc(struct ep_ctl_ctx *ctx, + struct eventpoll *ep, int depth) { int result = 0; struct rb_node *rbp; @@ -2356,22 +2378,23 @@ static int ep_loop_check_proc(struct eventpoll *ep, int depth) if (unlikely(is_file_epoll(epi->ffd.file))) { struct eventpoll *ep_tovisit; ep_tovisit = epi->ffd.file->private_data; - if (ep_tovisit == inserting_into || depth > EP_MAX_NESTS) + if (ep_tovisit == ctx->inserting_into || + depth > EP_MAX_NESTS) result = EP_MAX_NESTS+1; else - result = max(result, ep_loop_check_proc(ep_tovisit, depth + 1) + 1); + result = max(result, + ep_loop_check_proc(ctx, ep_tovisit, + depth + 1) + 1); if (result > EP_MAX_NESTS) break; } else { /* - * If we've reached a file that is not associated with - * an ep, then we need to check if the newly added - * links are going to add too many wakeup paths. We do - * this by adding it to the tfile_check_list, if it's - * not already there, and calling reverse_path_check() - * during ep_insert(). + * A non-epoll leaf. Queue it for the companion + * reverse_path_check() that runs after this walk so + * any new links we propose don't add too many wakeup + * paths. */ - list_file(epi->ffd.file); + list_file(epi->ffd.file, ctx); } } ep->loop_check_depth = result; @@ -2400,22 +2423,24 @@ static int ep_get_upwards_depth_proc(struct eventpoll *ep, int depth) * into another epoll file (represented by @ep) does not create * closed loops or too deep chains. * - * @ep: Pointer to the epoll we are inserting into. - * @to: Pointer to the epoll to be inserted. + * @ctx: Per-CTL_ADD scratch context. + * @ep: Pointer to the epoll we are inserting into. + * @to: Pointer to the epoll to be inserted. * * Return: %zero if adding the epoll @to inside the epoll @from * does not violate the constraints, or %-1 otherwise. */ -static int ep_loop_check(struct eventpoll *ep, struct eventpoll *to) +static int ep_loop_check(struct ep_ctl_ctx *ctx, struct eventpoll *ep, + struct eventpoll *to) { int depth, upwards_depth; - inserting_into = ep; + ctx->inserting_into = ep; /* * Check how deep down we can get from @to, and whether it is possible * to loop up to @ep. */ - depth = ep_loop_check_proc(to, 0); + depth = ep_loop_check_proc(ctx, to, 0); if (depth > EP_MAX_NESTS) return -1; /* Check how far up we can go from @ep. */ @@ -2426,12 +2451,12 @@ static int ep_loop_check(struct eventpoll *ep, struct eventpoll *to) return (depth+1+upwards_depth > EP_MAX_NESTS) ? -1 : 0; } -static void clear_tfile_check_list(void) +static void clear_tfile_check_list(struct ep_ctl_ctx *ctx) { rcu_read_lock(); - while (tfile_check_list != EP_UNACTIVE_PTR) { - struct epitems_head *head = tfile_check_list; - tfile_check_list = head->next; + while (ctx->tfile_check_list) { + struct epitems_head *head = ctx->tfile_check_list; + ctx->tfile_check_list = head->next; unlist_file(head); } rcu_read_unlock(); @@ -2529,9 +2554,8 @@ static inline int epoll_mutex_lock(struct mutex *mutex, bool nonblock) * EPOLL_CTL_ADDs on different eps from building a cycle without * either walker observing it. */ -static int ep_ctl_lock(struct eventpoll *ep, int op, - struct file *epfile, struct file *tfile, - bool nonblock) +static int ep_ctl_lock(struct ep_ctl_ctx *ctx, struct eventpoll *ep, int op, + struct file *epfile, struct file *tfile, bool nonblock) { struct eventpoll *tep; int error; @@ -2556,7 +2580,7 @@ static int ep_ctl_lock(struct eventpoll *ep, int op, if (is_file_epoll(tfile)) { tep = tfile->private_data; - if (ep_loop_check(ep, tep) != 0) { + if (ep_loop_check(ctx, ep, tep) != 0) { error = -ELOOP; goto err_unlock_nested; } @@ -2569,17 +2593,18 @@ static int ep_ctl_lock(struct eventpoll *ep, int op, return 1; err_unlock_nested: - clear_tfile_check_list(); + clear_tfile_check_list(ctx); loop_check_gen++; mutex_unlock(&epnested_mutex); return error; } -static void ep_ctl_unlock(struct eventpoll *ep, int full_check) +static void ep_ctl_unlock(struct ep_ctl_ctx *ctx, struct eventpoll *ep, + int full_check) { mutex_unlock(&ep->mtx); if (full_check) { - clear_tfile_check_list(); + clear_tfile_check_list(ctx); loop_check_gen++; mutex_unlock(&epnested_mutex); } @@ -2592,6 +2617,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, int full_check; struct eventpoll *ep; struct epitem *epi; + struct ep_ctl_ctx ctx = { }; CLASS(fd, f)(epfd); if (fd_empty(f)) @@ -2632,7 +2658,8 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, ep = fd_file(f)->private_data; - full_check = ep_ctl_lock(ep, op, fd_file(f), fd_file(tf), nonblock); + full_check = ep_ctl_lock(&ctx, ep, op, fd_file(f), fd_file(tf), + nonblock); if (full_check < 0) return full_check; @@ -2647,7 +2674,8 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, case EPOLL_CTL_ADD: if (!epi) { epds->events |= EPOLLERR | EPOLLHUP; - error = ep_insert(ep, epds, fd_file(tf), fd, full_check); + error = ep_insert(&ctx, ep, epds, fd_file(tf), fd, + full_check); } else error = -EEXIST; break; @@ -2674,7 +2702,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, break; } - ep_ctl_unlock(ep, full_check); + ep_ctl_unlock(&ctx, ep, full_check); return error; } From bd6a1379a41acfc38edf15e2c15aecb0694a7fba Mon Sep 17 00:00:00 2001 From: Fengnan Chang Date: Mon, 20 Apr 2026 14:16:30 +0800 Subject: [PATCH 18/69] iomap: avoid memset iomap when iter is done When iomap_iter() finishes its iteration (returns <= 0), it is no longer necessary to memset the entire iomap and srcmap structures. In high-IOPS scenarios (like 4k randread NVMe polling with io_uring), where the majority of I/Os complete in a single extent map, this wasted memory write bandwidth, as the caller will just discard the iterator. Use this command to test: taskset -c 30 ./t/io_uring -p1 -d512 -b4096 -s32 -c32 -F1 -B1 -R1 -X1 -n1 -P1 /mnt/testfile IOPS improve about 5% on ext4 and XFS. However, we MUST still call iomap_iter_reset_iomap() to release the folio_batch if IOMAP_F_FOLIO_BATCH is set, otherwise we leak page references. Therefore, split the cleanup logic: always release the folio_batch, but skip the memset() when ret <= 0. Signed-off-by: Fengnan Chang Link: https://patch.msgid.link/20260420061630.62077-1-changfengnan@bytedance.com Reviewed-by: Brian Foster Reviewed-by: Christoph Hellwig Signed-off-by: Christian Brauner --- fs/iomap/iter.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c index c04796f6e57f..e4a29829591a 100644 --- a/fs/iomap/iter.c +++ b/fs/iomap/iter.c @@ -6,17 +6,13 @@ #include #include "trace.h" -static inline void iomap_iter_reset_iomap(struct iomap_iter *iter) +static inline void iomap_iter_clean_fbatch(struct iomap_iter *iter) { if (iter->iomap.flags & IOMAP_F_FOLIO_BATCH) { folio_batch_release(iter->fbatch); folio_batch_reinit(iter->fbatch); iter->iomap.flags &= ~IOMAP_F_FOLIO_BATCH; } - - iter->status = 0; - memset(&iter->iomap, 0, sizeof(iter->iomap)); - memset(&iter->srcmap, 0, sizeof(iter->srcmap)); } /* Advance the current iterator position and decrement the remaining length */ @@ -102,10 +98,14 @@ int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops) ret = 0; else ret = 1; - iomap_iter_reset_iomap(iter); + iomap_iter_clean_fbatch(iter); + iter->status = 0; if (ret <= 0) return ret; + memset(&iter->iomap, 0, sizeof(iter->iomap)); + memset(&iter->srcmap, 0, sizeof(iter->srcmap)); + begin: ret = ops->iomap_begin(iter->inode, iter->pos, iter->len, iter->flags, &iter->iomap, &iter->srcmap); From ecf22101ba7a398eb1aefbb4eaba5fdddf7b08e4 Mon Sep 17 00:00:00 2001 From: Chi Zhiling Date: Mon, 11 May 2026 17:40:07 +0800 Subject: [PATCH 19/69] iomap: add dirty page control to iomap_zero_iter This patch prepares the iomap framework for exFAT's upcoming migration to iomap. During testing of the exFAT iomap branch with xfstests generic/299 on a VM with 8GB RAM and a 40GB disk, system unresponsiveness was observed. iomap_zero_iter() lacked dirty page throttling, which could cause memory pressure when exFAT's valid_size mechanism triggers large-scale zeroing operations during writes beyond valid_size. Align iomap_zero_iter() with iomap_write_iter() by adding balance_dirty_pages_ratelimited() to throttle dirty page generation during large zeroing operations Signed-off-by: Chi Zhiling Link: https://patch.msgid.link/20260511094007.728011-1-chizhiling@163.com Reviewed-by: Christoph Hellwig Cc: Namjae Jeon Signed-off-by: Christian Brauner --- fs/iomap/buffered-io.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index d7b648421a70..437a7c527c4a 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -1543,6 +1543,8 @@ static int iomap_zero_iter(struct iomap_iter *iter, bool *did_zero, size_t offset; bool ret; + balance_dirty_pages_ratelimited(iter->inode->i_mapping); + bytes = min_t(u64, SIZE_MAX, bytes); status = iomap_write_begin(iter, write_ops, &folio, &offset, &bytes); From 5a78a15537d4f260275d404edf7da1dd2d7157db Mon Sep 17 00:00:00 2001 From: Namjae Jeon Date: Mon, 11 May 2026 23:11:51 +0900 Subject: [PATCH 20/69] iomap: remove over-strict inline data boundary check The current iomap_inline_data_valid() check ensures that inline data does not cross a PAGE_SIZE boundary. However, this is an unnecessarily strict constraint. If a filesystem provides a valid iomap::inline_data pointer and iomap::length, we should trust that the caller has mapped sufficient memory for the range, even if it spans across page boundaries. Removing this check allows filesystems to point directly to their internal data structures without forced page-alignment or additional redundant allocations. This remove iomap_inline_data_valid() and its callers in buffered and direct I/O paths. Suggested-by: Matthew Wilcox Reviewed-by: Christoph Hellwig Reviewed-by: Gao Xiang Signed-off-by: Namjae Jeon Link: https://patch.msgid.link/20260511141151.6021-1-linkinjeon@kernel.org Signed-off-by: Christian Brauner --- fs/iomap/buffered-io.c | 1 - fs/iomap/direct-io.c | 3 --- include/linux/iomap.h | 10 ---------- 3 files changed, 14 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 437a7c527c4a..c98cf19fbe08 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -1058,7 +1058,6 @@ static bool iomap_write_end_inline(const struct iomap_iter *iter, void *addr; WARN_ON_ONCE(!folio_test_uptodate(folio)); - BUG_ON(!iomap_inline_data_valid(iomap)); if (WARN_ON_ONCE(!iomap->inline_data)) return false; diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index b0a6549b3848..a4459c5abb07 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -603,9 +603,6 @@ static int iomap_dio_inline_iter(struct iomap_iter *iomi, struct iomap_dio *dio) if (WARN_ON_ONCE(!inline_data)) return -EIO; - if (WARN_ON_ONCE(!iomap_inline_data_valid(iomap))) - return -EIO; - if (dio->flags & IOMAP_DIO_WRITE) { loff_t size = iomi->inode->i_size; diff --git a/include/linux/iomap.h b/include/linux/iomap.h index 2c5685adf3a9..52bfd6783417 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -142,16 +142,6 @@ static inline void *iomap_inline_data(const struct iomap *iomap, loff_t pos) return iomap->inline_data + pos - iomap->offset; } -/* - * Check if the mapping's length is within the valid range for inline data. - * This is used to guard against accessing data beyond the page inline_data - * points at. - */ -static inline bool iomap_inline_data_valid(const struct iomap *iomap) -{ - return iomap->length <= PAGE_SIZE - offset_in_page(iomap->inline_data); -} - /* * When get_folio succeeds, put_folio will always be called to do any * cleanup work necessary. put_folio is responsible for unlocking and putting From 9caef5b2a1fc22c2e24159a3d0fd74a47aed94ab Mon Sep 17 00:00:00 2001 From: Jens Axboe Date: Thu, 14 May 2026 08:07:17 -0600 Subject: [PATCH 21/69] eventpoll: pass struct epoll_filefd through ep_find() and ep_insert() Have ep_find() and ep_insert() take a struct epoll_filefd rather than a file/fd tuple. Kill off ep_set_ffd() as it's now no longer needed. No functional change. This is a prep patch for adding a file based do_epoll_ctl() variant. Signed-off-by: Jens Axboe Link: https://patch.msgid.link/20260514140817.623026-2-axboe@kernel.dk Reviewed-by: Christian Brauner Signed-off-by: Christian Brauner --- fs/eventpoll.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index a3090b446af1..f464f2f39e0e 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -339,14 +339,6 @@ static inline int is_file_epoll(struct file *f) return f->f_op == &eventpoll_fops; } -/* Setup the structure that is used as key for the RB tree */ -static inline void ep_set_ffd(struct epoll_filefd *ffd, - struct file *file, int fd) -{ - ffd->file = file; - ffd->fd = fd; -} - /* Compare RB tree keys */ static inline int ep_cmp_ffd(struct epoll_filefd *p1, struct epoll_filefd *p2) @@ -1173,17 +1165,15 @@ static int ep_alloc(struct eventpoll **pep) * are protected by the "mtx" mutex, and ep_find() must be called with * "mtx" held. */ -static struct epitem *ep_find(struct eventpoll *ep, struct file *file, int fd) +static struct epitem *ep_find(struct eventpoll *ep, struct epoll_filefd *tf) { int kcmp; struct rb_node *rbp; struct epitem *epi, *epir = NULL; - struct epoll_filefd ffd; - ep_set_ffd(&ffd, file, fd); for (rbp = ep->rbr.rb_root.rb_node; rbp; ) { epi = rb_entry(rbp, struct epitem, rbn); - kcmp = ep_cmp_ffd(&ffd, &epi->ffd); + kcmp = ep_cmp_ffd(tf, &epi->ffd); if (kcmp > 0) rbp = rbp->rb_right; else if (kcmp < 0) @@ -1564,7 +1554,7 @@ allocate: * Must be called with "mtx" held. */ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, - struct file *tfile, int fd, int full_check) + struct epoll_filefd *tf, int full_check) { int error, pwake = 0; __poll_t revents; @@ -1572,8 +1562,8 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, struct ep_pqueue epq; struct eventpoll *tep = NULL; - if (is_file_epoll(tfile)) - tep = tfile->private_data; + if (is_file_epoll(tf->file)) + tep = tf->file->private_data; lockdep_assert_irqs_enabled(); @@ -1590,14 +1580,14 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, /* Item initialization follow here ... */ INIT_LIST_HEAD(&epi->rdllink); epi->ep = ep; - ep_set_ffd(&epi->ffd, tfile, fd); + epi->ffd = *tf; epi->event = *event; epi->next = EP_UNACTIVE_PTR; if (tep) mutex_lock_nested(&tep->mtx, 1); /* Add the current item to the list of active epoll hook for this file */ - if (unlikely(attach_epitem(tfile, epi) < 0)) { + if (unlikely(attach_epitem(tf->file, epi) < 0)) { if (tep) mutex_unlock(&tep->mtx); kmem_cache_free(epi_cache, epi); @@ -1606,7 +1596,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, } if (full_check && !tep) - list_file(tfile); + list_file(tf->file); /* * Add the current item to the RB tree. All RB tree operations are @@ -2243,6 +2233,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, struct eventpoll *ep; struct epitem *epi; struct eventpoll *tep = NULL; + struct epoll_filefd efd; CLASS(fd, f)(epfd); if (fd_empty(f)) @@ -2253,6 +2244,9 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, if (fd_empty(tf)) return -EBADF; + efd.file = fd_file(tf); + efd.fd = fd; + /* The target file descriptor must support poll */ if (!file_can_poll(fd_file(tf))) return -EPERM; @@ -2333,14 +2327,14 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, * above, we can be sure to be able to use the item looked up by * ep_find() till we release the mutex. */ - epi = ep_find(ep, fd_file(tf), fd); + epi = ep_find(ep, &efd); error = -EINVAL; switch (op) { case EPOLL_CTL_ADD: if (!epi) { epds->events |= EPOLLERR | EPOLLHUP; - error = ep_insert(ep, epds, fd_file(tf), fd, full_check); + error = ep_insert(ep, epds, &efd, full_check); } else error = -EEXIST; break; From 5de2759f2b7c925f187e552cae47775acd5f4b40 Mon Sep 17 00:00:00 2001 From: Jens Axboe Date: Thu, 14 May 2026 08:07:18 -0600 Subject: [PATCH 22/69] eventpoll: export is_file_epoll() Make is_file_epoll() available outside of epoll. This is in preparation from using it from io_uring. Signed-off-by: Jens Axboe Link: https://patch.msgid.link/20260514140817.623026-3-axboe@kernel.dk Reviewed-by: Christian Brauner Signed-off-by: Christian Brauner --- fs/eventpoll.c | 2 +- include/linux/eventpoll.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index f464f2f39e0e..9ea6a2bd3d87 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -334,7 +334,7 @@ static void __init epoll_sysctls_init(void) static const struct file_operations eventpoll_fops; -static inline int is_file_epoll(struct file *f) +int is_file_epoll(struct file *f) { return f->f_op == &eventpoll_fops; } diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h index 728fb5dee5ed..7bf30e9f90d7 100644 --- a/include/linux/eventpoll.h +++ b/include/linux/eventpoll.h @@ -63,6 +63,7 @@ static inline void eventpoll_release(struct file *file) int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, bool nonblock); +int is_file_epoll(struct file *f); /* Tells if the epoll_ctl(2) operation needs an event copy from userspace */ static inline int ep_op_has_event(int op) From 0995baf261ce8fc141768911d97ba76e854e26cf Mon Sep 17 00:00:00 2001 From: Jens Axboe Date: Thu, 14 May 2026 08:07:19 -0600 Subject: [PATCH 23/69] eventpoll: add file based control interface Add do_epoll_ctl_file(), which takes a pre-resolved epoll file and a struct epoll_filefd for the target rather than two integer file descriptors. do_epoll_ctl() remains as a thin wrapper. In preparation for using the file based interface from io_uring. Signed-off-by: Jens Axboe Link: https://patch.msgid.link/20260514140817.623026-4-axboe@kernel.dk Signed-off-by: Christian Brauner --- fs/eventpoll.c | 62 ++++++++++++++++++++------------------- include/linux/eventpoll.h | 7 +++++ 2 files changed, 39 insertions(+), 30 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 9ea6a2bd3d87..1c7001866340 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -99,11 +99,6 @@ #define EP_ITEM_COST (sizeof(struct epitem) + sizeof(struct eppoll_entry)) -struct epoll_filefd { - struct file *file; - int fd; -} __packed; - /* Wait structure used by the poll hooks */ struct eppoll_entry { /* List header used to link this structure to the "struct epitem" */ @@ -2225,30 +2220,17 @@ static inline int epoll_mutex_lock(struct mutex *mutex, int depth, return -EAGAIN; } -int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, - bool nonblock) +int do_epoll_ctl_file(struct file *f, int op, struct epoll_filefd *tf, + struct epoll_event *epds, bool nonblock) { int error; int full_check = 0; struct eventpoll *ep; struct epitem *epi; struct eventpoll *tep = NULL; - struct epoll_filefd efd; - - CLASS(fd, f)(epfd); - if (fd_empty(f)) - return -EBADF; - - /* Get the "struct file *" for the target file */ - CLASS(fd, tf)(fd); - if (fd_empty(tf)) - return -EBADF; - - efd.file = fd_file(tf); - efd.fd = fd; /* The target file descriptor must support poll */ - if (!file_can_poll(fd_file(tf))) + if (!file_can_poll(tf->file)) return -EPERM; /* Check if EPOLLWAKEUP is allowed */ @@ -2261,7 +2243,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, * adding an epoll file descriptor inside itself. */ error = -EINVAL; - if (fd_file(f) == fd_file(tf) || !is_file_epoll(fd_file(f))) + if (f == tf->file || !is_file_epoll(f)) goto error_tgt_fput; /* @@ -2272,7 +2254,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, if (ep_op_has_event(op) && (epds->events & EPOLLEXCLUSIVE)) { if (op == EPOLL_CTL_MOD) goto error_tgt_fput; - if (op == EPOLL_CTL_ADD && (is_file_epoll(fd_file(tf)) || + if (op == EPOLL_CTL_ADD && (is_file_epoll(tf->file) || (epds->events & ~EPOLLEXCLUSIVE_OK_BITS))) goto error_tgt_fput; } @@ -2281,7 +2263,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, * At this point it is safe to assume that the "private_data" contains * our own data structure. */ - ep = fd_file(f)->private_data; + ep = f->private_data; /* * When we insert an epoll file descriptor inside another epoll file @@ -2302,16 +2284,16 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, if (error) goto error_tgt_fput; if (op == EPOLL_CTL_ADD) { - if (READ_ONCE(fd_file(f)->f_ep) || ep->gen == loop_check_gen || - is_file_epoll(fd_file(tf))) { + if (READ_ONCE(f->f_ep) || ep->gen == loop_check_gen || + is_file_epoll(tf->file)) { mutex_unlock(&ep->mtx); error = epoll_mutex_lock(&epnested_mutex, 0, nonblock); if (error) goto error_tgt_fput; loop_check_gen++; full_check = 1; - if (is_file_epoll(fd_file(tf))) { - tep = fd_file(tf)->private_data; + if (is_file_epoll(tf->file)) { + tep = tf->file->private_data; error = -ELOOP; if (ep_loop_check(ep, tep) != 0) goto error_tgt_fput; @@ -2327,14 +2309,14 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, * above, we can be sure to be able to use the item looked up by * ep_find() till we release the mutex. */ - epi = ep_find(ep, &efd); + epi = ep_find(ep, tf); error = -EINVAL; switch (op) { case EPOLL_CTL_ADD: if (!epi) { epds->events |= EPOLLERR | EPOLLHUP; - error = ep_insert(ep, epds, &efd, full_check); + error = ep_insert(ep, epds, tf, full_check); } else error = -EEXIST; break; @@ -2369,6 +2351,26 @@ error_tgt_fput: mutex_unlock(&epnested_mutex); } return error; + +} + +int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, + bool nonblock) +{ + struct epoll_filefd efd; + + CLASS(fd, f)(epfd); + if (fd_empty(f)) + return -EBADF; + + /* Get the "struct file *" for the target file */ + CLASS(fd, tf)(fd); + if (fd_empty(tf)) + return -EBADF; + + efd.file = fd_file(tf); + efd.fd = fd; + return do_epoll_ctl_file(fd_file(f), op, &efd, epds, nonblock); } /* diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h index 7bf30e9f90d7..4a6fe989810b 100644 --- a/include/linux/eventpoll.h +++ b/include/linux/eventpoll.h @@ -61,6 +61,13 @@ static inline void eventpoll_release(struct file *file) eventpoll_release_file(file); } +struct epoll_filefd { + struct file *file; + int fd; +} __packed; + +int do_epoll_ctl_file(struct file *f, int op, struct epoll_filefd *tf, + struct epoll_event *epds, bool nonblock); int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, bool nonblock); int is_file_epoll(struct file *f); From 463aba8090738bcd956297f7341b6874e76ae664 Mon Sep 17 00:00:00 2001 From: Jens Axboe Date: Thu, 14 May 2026 08:07:20 -0600 Subject: [PATCH 24/69] eventpoll: rename struct epoll_filefd to epoll_key This more accurately describes what purpose this structure serves, as a lookup key. Suggested-by: Christian Brauner Signed-off-by: Jens Axboe Link: https://patch.msgid.link/20260514140817.623026-5-axboe@kernel.dk Signed-off-by: Christian Brauner --- fs/eventpoll.c | 13 ++++++------- include/linux/eventpoll.h | 4 ++-- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 1c7001866340..7535b10f8c6a 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -141,7 +141,7 @@ struct epitem { struct epitem *next; /* The file descriptor information this item refers to */ - struct epoll_filefd ffd; + struct epoll_key ffd; /* List containing poll wait queues */ struct eppoll_entry *pwqlist; @@ -335,8 +335,7 @@ int is_file_epoll(struct file *f) } /* Compare RB tree keys */ -static inline int ep_cmp_ffd(struct epoll_filefd *p1, - struct epoll_filefd *p2) +static inline int ep_cmp_ffd(struct epoll_key *p1, struct epoll_key *p2) { return (p1->file > p2->file ? +1: (p1->file < p2->file ? -1 : p1->fd - p2->fd)); @@ -1160,7 +1159,7 @@ static int ep_alloc(struct eventpoll **pep) * are protected by the "mtx" mutex, and ep_find() must be called with * "mtx" held. */ -static struct epitem *ep_find(struct eventpoll *ep, struct epoll_filefd *tf) +static struct epitem *ep_find(struct eventpoll *ep, struct epoll_key *tf) { int kcmp; struct rb_node *rbp; @@ -1549,7 +1548,7 @@ allocate: * Must be called with "mtx" held. */ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event, - struct epoll_filefd *tf, int full_check) + struct epoll_key *tf, int full_check) { int error, pwake = 0; __poll_t revents; @@ -2220,7 +2219,7 @@ static inline int epoll_mutex_lock(struct mutex *mutex, int depth, return -EAGAIN; } -int do_epoll_ctl_file(struct file *f, int op, struct epoll_filefd *tf, +int do_epoll_ctl_file(struct file *f, int op, struct epoll_key *tf, struct epoll_event *epds, bool nonblock) { int error; @@ -2357,7 +2356,7 @@ error_tgt_fput: int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, bool nonblock) { - struct epoll_filefd efd; + struct epoll_key efd; CLASS(fd, f)(epfd); if (fd_empty(f)) diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h index 4a6fe989810b..c214c374fefc 100644 --- a/include/linux/eventpoll.h +++ b/include/linux/eventpoll.h @@ -61,12 +61,12 @@ static inline void eventpoll_release(struct file *file) eventpoll_release_file(file); } -struct epoll_filefd { +struct epoll_key { struct file *file; int fd; } __packed; -int do_epoll_ctl_file(struct file *f, int op, struct epoll_filefd *tf, +int do_epoll_ctl_file(struct file *f, int op, struct epoll_key *tf, struct epoll_event *epds, bool nonblock); int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, bool nonblock); From 47f28b493daf9049307494689e515205f1c377af Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 18 May 2026 08:29:13 +0200 Subject: [PATCH 25/69] iomap: don't make REQ_POLLED imply REQ_NOWAIT As described in commit 2bc057692599 ("block: don't make REQ_POLLED imply REQ_NOWAIT"), which fixed the same issue for the block device node, there are valid cases to poll for I/O completion without REQ_NOWAIT. Additionally, sing REQ_NOWAIT for file system writes is currently not supported as file systems writes are not idempotent and would need a retry of just the bio and not the entire operation to be fully supported. Switch iomap to set REQ_POLLED and remove the now unused bio_set_polled helper. Signed-off-by: Christoph Hellwig Link: https://patch.msgid.link/20260518062917.506483-1-hch@lst.de Signed-off-by: Christian Brauner --- fs/iomap/direct-io.c | 2 +- include/linux/bio.h | 14 -------------- 2 files changed, 1 insertion(+), 15 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index a4459c5abb07..3df100fe0bf6 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -69,7 +69,7 @@ static void iomap_dio_submit_bio(const struct iomap_iter *iter, /* Sync dio can't be polled reliably */ if ((iocb->ki_flags & IOCB_HIPRI) && !is_sync_kiocb(iocb)) { - bio_set_polled(bio, iocb); + bio->bi_opf |= REQ_POLLED; WRITE_ONCE(iocb->private, bio); } diff --git a/include/linux/bio.h b/include/linux/bio.h index 97d747320b35..fbccb2f264b0 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -702,20 +702,6 @@ static inline bool bioset_initialized(struct bio_set *bs) return bs->bio_slab != NULL; } -/* - * Mark a bio as polled. Note that for async polled IO, the caller must - * expect -EWOULDBLOCK if we cannot allocate a request (or other resources). - * We cannot block waiting for requests on polled IO, as those completions - * must be found by the caller. This is different than IRQ driven IO, where - * it's safe to wait for IO to complete. - */ -static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb) -{ - bio->bi_opf |= REQ_POLLED; - if (kiocb->ki_flags & IOCB_NOWAIT) - bio->bi_opf |= REQ_NOWAIT; -} - static inline void bio_clear_polled(struct bio *bio) { bio->bi_opf &= ~REQ_POLLED; From 70a03a385de2b8f0fa54dbc70bdc3ed176853d1c Mon Sep 17 00:00:00 2001 From: Randy Dunlap Date: Mon, 18 May 2026 21:23:14 -0700 Subject: [PATCH 26/69] eventpoll: add missing kernel-doc for @ctx function parameters Add the missing kernel-doc comments to prevent kernel-doc build warnings while building the documentation. WARNING: fs/eventpoll.c:1684 function parameter 'ctx' not described in 'reverse_path_check' WARNING: fs/eventpoll.c:2349 function parameter 'ctx' not described in 'ep_loop_check_proc' Fixes: e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") Signed-off-by: Randy Dunlap Link: https://patch.msgid.link/20260519042314.124041-1-rdunlap@infradead.org Signed-off-by: Christian Brauner (Amutable) --- fs/eventpoll.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index a569e98d4a99..baa97d0edade 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1677,6 +1677,7 @@ static int reverse_path_check_proc(struct ep_ctl_ctx *ctx, * anchoring files with newly proposed links; make * sure those links don't push any path-length bucket * over its limit in path_limits[]. + * @ctx: Per-do_epoll_ctl() scratch for the loop / path checks. * * Return: %zero if the proposed links don't create too many paths, * %-1 otherwise. @@ -2339,6 +2340,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * epoll file does not create closed loops, and * determine the depth of the subtree starting at @ep * + * @ctx: Per-do_epoll_ctl() scratch for the loop / path checks. * @ep: the &struct eventpoll to be currently checked. * @depth: Current depth of the path being checked. * From 827382c088e9ac18a9e05c0a238a2ff919bc4755 Mon Sep 17 00:00:00 2001 From: Nam Cao Date: Tue, 2 Jun 2026 19:51:45 +0200 Subject: [PATCH 27/69] selftests/eventpoll: Add test for multiple waiters Add a test whichs creates 64 threads who all epoll_wait() on the same eventpoll. The source eventfd is written but never read, therefore all the threads should always see an EPOLLIN event. This test fails because of a kernel bug, which will be fixed by a follow-up commit. Signed-off-by: Nam Cao Link: https://patch.msgid.link/b11947013563875c046c0b0959c29fd95eeebd34.1780422138.git.namcao@linutronix.de Signed-off-by: Christian Brauner (Amutable) --- .../filesystems/epoll/epoll_wakeup_test.c | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/tools/testing/selftests/filesystems/epoll/epoll_wakeup_test.c b/tools/testing/selftests/filesystems/epoll/epoll_wakeup_test.c index 8bc57a2ef966..f6f1a7ff01b0 100644 --- a/tools/testing/selftests/filesystems/epoll/epoll_wakeup_test.c +++ b/tools/testing/selftests/filesystems/epoll/epoll_wakeup_test.c @@ -3493,4 +3493,49 @@ TEST(epoll64) close(ctx.sfd[1]); } +static void *epoll65_wait(void *ctx_) +{ + struct epoll_mtcontext *ctx = ctx_; + struct epoll_event event; + + for (int i = 0; i < 100000; ++i) { + if (!epoll_wait(ctx->efd[0], &event, 1, 0)) + return (void *)ENODATA; + } + + return (void *)0; +} + +TEST(epoll65) +{ + struct epoll_mtcontext ctx; + struct epoll_event event; + int64_t dummy_data = 99; + pthread_t threads[64]; + uintptr_t ret; + int i, err; + + ctx.efd[0] = epoll_create(1); + ASSERT_GE(ctx.efd[0], 0); + ctx.efd[1] = eventfd(0, 0); + ASSERT_GE(ctx.efd[1], 0); + + event.events = EPOLLIN; + err = epoll_ctl(ctx.efd[0], EPOLL_CTL_ADD, ctx.efd[1], &event); + ASSERT_EQ(err, 0); + + write(ctx.efd[1], &dummy_data, sizeof(dummy_data)); + + for (i = 0; i < ARRAY_SIZE(threads); ++i) + ASSERT_EQ(pthread_create(&threads[i], NULL, epoll65_wait, &ctx), 0); + + for (i = 0; i < ARRAY_SIZE(threads); ++i) { + ASSERT_EQ(pthread_join(threads[i], (void **)&ret), 0); + ASSERT_EQ(ret, 0); + } + + close(ctx.efd[0]); + close(ctx.efd[1]); +} + TEST_HARNESS_MAIN From 0c4aefe3c2d0f272a2ad73699a12d4446ffdbe7b Mon Sep 17 00:00:00 2001 From: Nam Cao Date: Tue, 2 Jun 2026 19:51:46 +0200 Subject: [PATCH 28/69] eventpoll: Fix epoll_wait() report false negative ep_events_available() checks for available events by looking at ep->rdllist and ep_is_scanning(). However, this is done without a lock and can report false negative if ep_start_scan() or ep_done_scan() are executed by another task concurrently. For example: _________________________________________________________________________ |ep_start_scan() | list_splice_init(&ep->rdllist, ...) ep_events_available() | !list_empty_careful(&ep->rdllist)| || ep_is_scanning(ep) | | ep_enter_scan(ep) ___________________________________|_____________________________________ Another example: _________________________________________________________________________ ep_events_available() | |ep_start_scan() | list_splice_init(&ep->rdllist, ...) | ep_enter_scan(ep) !list_empty_careful(&ep->rdllist)| |ep_done_scan() | ep_exit_scan(ep) | list_splice(..., &ep->rdllist) || ep_is_scanning(ep) | ___________________________________|_____________________________________ In the above examples, ep_events_available() sees no event despite events being available. In case epoll_wait() is called with timeout=0, epoll_wait() will wrongly return "no event" to user. Introduce a sequence lock to resolve this issue. Measuring the time consumption of 10 million loop iterations doing epoll_wait(), the following performance drop is observed: timeout #event before after diff 0ms 0 3727ms 3974ms +6.6% 0ms 1 8099ms 9134ms +13% 1ms 1 13525ms 13586ms +0.45% Considering the use case of epoll_wait() (wait for events, do something with the events, repeat), it should only contribute to a small portion of user's CPU consumption. Therefore this performance drop is not alarming. Fixes: c5a282e9635e ("fs/epoll: reduce the scope of wq lock in epoll_wait()") Suggested-by: Mateusz Guzik Signed-off-by: Nam Cao Link: https://patch.msgid.link/4363cd8e34a21d4f0d257be1b33e84dc25030fdf.1780422138.git.namcao@linutronix.de Signed-off-by: Christian Brauner (Amutable) --- fs/eventpoll.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index baa97d0edade..df364a8783b5 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -38,6 +38,7 @@ #include #include #include +#include #include /* @@ -312,6 +313,9 @@ struct eventpoll { /* Lock which protects rdllist and ovflist */ spinlock_t lock; + /* Protect switching between rdllist and ovflist */ + seqcount_spinlock_t seq; + /* RB tree root used to store monitored fd structs */ struct rb_root_cached rbr; @@ -590,7 +594,10 @@ static inline void epi_clear_ovflist(struct epitem *epi) /* True iff @ep has ready events that epoll_wait() might harvest. */ static inline bool ep_events_available(struct eventpoll *ep) { - return !list_empty_careful(&ep->rdllist) || ep_is_scanning(ep); + unsigned int seq = read_seqcount_begin(&ep->seq); + + return !list_empty_careful(&ep->rdllist) || ep_is_scanning(ep) || + read_seqcount_retry(&ep->seq, seq); } #ifdef CONFIG_NET_RX_BUSY_POLL @@ -947,8 +954,12 @@ static void ep_start_scan(struct eventpoll *ep, struct list_head *scan_batch) */ lockdep_assert_irqs_enabled(); spin_lock_irq(&ep->lock); + write_seqcount_begin(&ep->seq); + list_splice_init(&ep->rdllist, scan_batch); ep_enter_scan(ep); + + write_seqcount_end(&ep->seq); spin_unlock_irq(&ep->lock); } @@ -979,6 +990,9 @@ static void ep_done_scan(struct eventpoll *ep, ep_pm_stay_awake(epi); } } + + write_seqcount_begin(&ep->seq); + /* Back out of scan mode; callbacks target ep->rdllist again. */ ep_exit_scan(ep); @@ -986,6 +1000,9 @@ static void ep_done_scan(struct eventpoll *ep, * Quickly re-inject items left on "scan_batch". */ list_splice(scan_batch, &ep->rdllist); + + write_seqcount_end(&ep->seq); + __pm_relax(ep->ws); if (!list_empty(&ep->rdllist)) { @@ -1405,6 +1422,7 @@ static int ep_alloc(struct eventpoll **pep) mutex_init(&ep->mtx); spin_lock_init(&ep->lock); + seqcount_spinlock_init(&ep->seq, &ep->lock); init_waitqueue_head(&ep->wq); init_waitqueue_head(&ep->poll_wait); INIT_LIST_HEAD(&ep->rdllist); From 6b1a162ba1a2dfd1ceac24029e4d0c2443173ed2 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:14 +0100 Subject: [PATCH 29/69] buffer: Remove forward declaration of submit_bh_wbc() Rearrange functions to avoid this forward declaration. Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-2-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 150 +++++++++++++++++++++++++--------------------------- 1 file changed, 73 insertions(+), 77 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index b0b3792b1496..d4e9a10d2a26 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -54,9 +54,6 @@ #include "internal.h" -static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh, - enum rw_hint hint, struct writeback_control *wbc); - #define BH_ENTRY(list) list_entry((list), struct buffer_head, b_assoc_buffers) inline void touch_buffer(struct buffer_head *bh) @@ -1157,6 +1154,79 @@ void __bforget(struct buffer_head *bh) } EXPORT_SYMBOL(__bforget); +static void end_bio_bh_io_sync(struct bio *bio) +{ + struct buffer_head *bh = bio->bi_private; + + if (unlikely(bio_flagged(bio, BIO_QUIET))) + set_bit(BH_Quiet, &bh->b_state); + + bh->b_end_io(bh, !bio->bi_status); + bio_put(bio); +} + +static void buffer_set_crypto_ctx(struct bio *bio, const struct buffer_head *bh, + gfp_t gfp_mask) +{ + const struct address_space *mapping = folio_mapping(bh->b_folio); + + /* + * The ext4 journal (jbd2) can submit a buffer_head it directly created + * for a non-pagecache page. fscrypt doesn't care about these. + */ + if (!mapping) + return; + fscrypt_set_bio_crypt_ctx(bio, mapping->host, + folio_pos(bh->b_folio) + bh_offset(bh), gfp_mask); +} + +static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh, + enum rw_hint write_hint, struct writeback_control *wbc) +{ + const enum req_op op = opf & REQ_OP_MASK; + struct bio *bio; + + BUG_ON(!buffer_locked(bh)); + BUG_ON(!buffer_mapped(bh)); + BUG_ON(!bh->b_end_io); + BUG_ON(buffer_delay(bh)); + BUG_ON(buffer_unwritten(bh)); + + /* + * Only clear out a write error when rewriting + */ + if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE)) + clear_buffer_write_io_error(bh); + + if (buffer_meta(bh)) + opf |= REQ_META; + if (buffer_prio(bh)) + opf |= REQ_PRIO; + + bio = bio_alloc(bh->b_bdev, 1, opf, GFP_NOIO); + + if (IS_ENABLED(CONFIG_FS_ENCRYPTION)) + buffer_set_crypto_ctx(bio, bh, GFP_NOIO); + + bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9); + bio->bi_write_hint = write_hint; + + bio_add_folio_nofail(bio, bh->b_folio, bh->b_size, bh_offset(bh)); + + bio->bi_end_io = end_bio_bh_io_sync; + bio->bi_private = bh; + + /* Take care of bh's that straddle the end of the device */ + guard_bio_eod(bio); + + if (wbc) { + wbc_init_bio(wbc, bio); + wbc_account_cgroup_owner(wbc, bh->b_folio, bh->b_size); + } + + blk_crypto_submit_bio(bio); +} + static struct buffer_head *__bread_slow(struct buffer_head *bh) { lock_buffer(bh); @@ -2663,80 +2733,6 @@ sector_t generic_block_bmap(struct address_space *mapping, sector_t block, } EXPORT_SYMBOL(generic_block_bmap); -static void end_bio_bh_io_sync(struct bio *bio) -{ - struct buffer_head *bh = bio->bi_private; - - if (unlikely(bio_flagged(bio, BIO_QUIET))) - set_bit(BH_Quiet, &bh->b_state); - - bh->b_end_io(bh, !bio->bi_status); - bio_put(bio); -} - -static void buffer_set_crypto_ctx(struct bio *bio, const struct buffer_head *bh, - gfp_t gfp_mask) -{ - const struct address_space *mapping = folio_mapping(bh->b_folio); - - /* - * The ext4 journal (jbd2) can submit a buffer_head it directly created - * for a non-pagecache page. fscrypt doesn't care about these. - */ - if (!mapping) - return; - fscrypt_set_bio_crypt_ctx(bio, mapping->host, - folio_pos(bh->b_folio) + bh_offset(bh), gfp_mask); -} - -static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh, - enum rw_hint write_hint, - struct writeback_control *wbc) -{ - const enum req_op op = opf & REQ_OP_MASK; - struct bio *bio; - - BUG_ON(!buffer_locked(bh)); - BUG_ON(!buffer_mapped(bh)); - BUG_ON(!bh->b_end_io); - BUG_ON(buffer_delay(bh)); - BUG_ON(buffer_unwritten(bh)); - - /* - * Only clear out a write error when rewriting - */ - if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE)) - clear_buffer_write_io_error(bh); - - if (buffer_meta(bh)) - opf |= REQ_META; - if (buffer_prio(bh)) - opf |= REQ_PRIO; - - bio = bio_alloc(bh->b_bdev, 1, opf, GFP_NOIO); - - if (IS_ENABLED(CONFIG_FS_ENCRYPTION)) - buffer_set_crypto_ctx(bio, bh, GFP_NOIO); - - bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9); - bio->bi_write_hint = write_hint; - - bio_add_folio_nofail(bio, bh->b_folio, bh->b_size, bh_offset(bh)); - - bio->bi_end_io = end_bio_bh_io_sync; - bio->bi_private = bh; - - /* Take care of bh's that straddle the end of the device */ - guard_bio_eod(bio); - - if (wbc) { - wbc_init_bio(wbc, bio); - wbc_account_cgroup_owner(wbc, bh->b_folio, bh->b_size); - } - - blk_crypto_submit_bio(bio); -} - void submit_bh(blk_opf_t opf, struct buffer_head *bh) { submit_bh_wbc(opf, bh, WRITE_LIFE_NOT_SET, NULL); From 4b7381326424809577eb43bf635ea1d43a095d89 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:15 +0100 Subject: [PATCH 30/69] buffer: Add bh_submit() bh_submit() takes a bio_end_io allowing users to avoid the indirect function call through bh->b_end_io, and eventually allowing us to remove bh->b_end_io. Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-3-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 31 +++++++++++++++++++++++++++---- include/linux/buffer_head.h | 1 + 2 files changed, 28 insertions(+), 4 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index d4e9a10d2a26..ea926963529e 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1180,15 +1180,15 @@ static void buffer_set_crypto_ctx(struct bio *bio, const struct buffer_head *bh, folio_pos(bh->b_folio) + bh_offset(bh), gfp_mask); } -static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh, - enum rw_hint write_hint, struct writeback_control *wbc) +static void __bh_submit(struct buffer_head *bh, blk_opf_t opf, + enum rw_hint write_hint, struct writeback_control *wbc, + bio_end_io_t end_bio) { const enum req_op op = opf & REQ_OP_MASK; struct bio *bio; BUG_ON(!buffer_locked(bh)); BUG_ON(!buffer_mapped(bh)); - BUG_ON(!bh->b_end_io); BUG_ON(buffer_delay(bh)); BUG_ON(buffer_unwritten(bh)); @@ -1213,7 +1213,7 @@ static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh, bio_add_folio_nofail(bio, bh->b_folio, bh->b_size, bh_offset(bh)); - bio->bi_end_io = end_bio_bh_io_sync; + bio->bi_end_io = end_bio; bio->bi_private = bh; /* Take care of bh's that straddle the end of the device */ @@ -1227,6 +1227,29 @@ static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh, blk_crypto_submit_bio(bio); } +static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh, + enum rw_hint write_hint, + struct writeback_control *wbc) +{ + BUG_ON(!bh->b_end_io); + __bh_submit(bh, opf, write_hint, wbc, end_bio_bh_io_sync); +} + +/** + * bh_submit - Start I/O against a buffer head + * @bh: The buffer head to perform I/O on. + * @opf: Operation and flags for bio. + * @end_io: The routine to call when I/O has completed. + * + * If you need to do I/O on an individual bh (instead of allowing the + * page cache to do I/O on the folio that it is in), call this function. + */ +void bh_submit(struct buffer_head *bh, blk_opf_t opf, bio_end_io_t end_io) +{ + __bh_submit(bh, opf, WRITE_LIFE_NOT_SET, NULL, end_io); +} +EXPORT_SYMBOL(bh_submit); + static struct buffer_head *__bread_slow(struct buffer_head *bh) { lock_buffer(bh); diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index e4939e33b4b5..d59980e4adda 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -240,6 +240,7 @@ int sync_dirty_buffer(struct buffer_head *bh); int __sync_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags); void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags); void submit_bh(blk_opf_t, struct buffer_head *); +void bh_submit(struct buffer_head *, blk_opf_t, bio_end_io_t); void write_boundary_block(struct block_device *bdev, sector_t bblock, unsigned blocksize); int bh_uptodate_or_lock(struct buffer_head *bh); From df21e33ff9c4ab20eb2a98273c2d9fed300aba26 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:16 +0100 Subject: [PATCH 31/69] buffer: Remove mark_buffer_async_write_endio() All callers of mark_buffer_async_write_endio() pass end_buffer_async_write, so we can inline mark_buffer_async_write_endio() into mark_buffer_async_write() and just call that instead. Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-4-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index ea926963529e..4dcce64ef006 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -443,16 +443,10 @@ static void mark_buffer_async_read(struct buffer_head *bh) set_buffer_async_read(bh); } -static void mark_buffer_async_write_endio(struct buffer_head *bh, - bh_end_io_t *handler) -{ - bh->b_end_io = handler; - set_buffer_async_write(bh); -} - void mark_buffer_async_write(struct buffer_head *bh) { - mark_buffer_async_write_endio(bh, end_buffer_async_write); + bh->b_end_io = end_buffer_async_write; + set_buffer_async_write(bh); } EXPORT_SYMBOL(mark_buffer_async_write); @@ -1903,8 +1897,7 @@ int __block_write_full_folio(struct inode *inode, struct folio *folio, continue; } if (test_clear_buffer_dirty(bh)) { - mark_buffer_async_write_endio(bh, - end_buffer_async_write); + mark_buffer_async_write(bh); } else { unlock_buffer(bh); } @@ -1958,8 +1951,7 @@ recover: if (buffer_mapped(bh) && buffer_dirty(bh) && !buffer_delay(bh)) { lock_buffer(bh); - mark_buffer_async_write_endio(bh, - end_buffer_async_write); + mark_buffer_async_write(bh); } else { /* * The buffer may have been set dirty during From 86ecf5704ed3c731284b9c60e46dcc5d115ebe7c Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:17 +0100 Subject: [PATCH 32/69] buffer: Add bh_end_read(), bh_end_write() and bh_end_async_write() These are the bio_end_io_t versions of end_buffer_read_sync(), end_buffer_write_sync() and end_buffer_async_write(). They do not contain a put_bh() call as it is no longer necessary. Also add the helper function bio_endio_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-5-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 89 ++++++++++++++++++++++++++++++++++--- include/linux/buffer_head.h | 6 +++ 2 files changed, 88 insertions(+), 7 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 4dcce64ef006..e22a94afa385 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -129,6 +129,34 @@ static void buffer_io_error(struct buffer_head *bh, char *msg) bh->b_bdev, (unsigned long long)bh->b_blocknr, msg); } +/** + * bio_endio_bh - Discard the bio used to submit a buffer. + * @bio: The bio. + * @bhp: Where to return the buffer_head. + * + * Call this in your bio_end_io handler to retrieve the buffer_head + * submitted in bh_submit(). If you did not call bh_submit(), do not + * call this function; it will return garbage. + * + * This function consumes the bio refcount which will probably free the + * bio. + * + * Return: True if the I/O succeeded. + */ +bool bio_endio_bh(struct bio *bio, struct buffer_head **bhp) +{ + bool success = bio->bi_status == BLK_STS_OK; + struct buffer_head *bh = bio->bi_private; + + if (unlikely(bio_flagged(bio, BIO_QUIET))) + set_bit(BH_Quiet, &bh->b_state); + bio_put(bio); + + *bhp = bh; + return success; +} +EXPORT_SYMBOL(bio_endio_bh); + /* * End-of-IO handler helper function which does not touch the bh after * unlocking it. @@ -159,7 +187,22 @@ void end_buffer_read_sync(struct buffer_head *bh, int uptodate) } EXPORT_SYMBOL(end_buffer_read_sync); -void end_buffer_write_sync(struct buffer_head *bh, int uptodate) +/** + * bh_end_read - I/O end handler for reads + * @bio: The bio being completed. + * + * Pass this function to bh_submit() if you're reading into the buffer, + * unless you need your own special I/O end handler. + */ +void bh_end_read(struct bio *bio) +{ + struct buffer_head *bh; + bool uptodate = bio_endio_bh(bio, &bh); + __end_buffer_read_notouch(bh, uptodate); +} +EXPORT_SYMBOL(bh_end_read); + +static void __end_buffer_write_sync(struct buffer_head *bh, int uptodate) { if (uptodate) { set_buffer_uptodate(bh); @@ -169,10 +212,30 @@ void end_buffer_write_sync(struct buffer_head *bh, int uptodate) clear_buffer_uptodate(bh); } unlock_buffer(bh); +} + +void end_buffer_write_sync(struct buffer_head *bh, int uptodate) +{ + __end_buffer_write_sync(bh, uptodate); put_bh(bh); } EXPORT_SYMBOL(end_buffer_write_sync); +/** + * bh_end_write - I/O end handler for writes + * @bio: The bio being completed. + * + * Pass this function to bh_submit() if you're writing from the buffer, + * unless you need your own special I/O end handler. + */ +void bh_end_write(struct bio *bio) +{ + struct buffer_head *bh; + bool success = bio_endio_bh(bio, &bh); + __end_buffer_write_sync(bh, success); +} +EXPORT_SYMBOL(bh_end_write); + static struct buffer_head * __find_get_block_slow(struct block_device *bdev, sector_t block, bool atomic) { @@ -416,6 +479,21 @@ still_busy: spin_unlock_irqrestore(&first->b_uptodate_lock, flags); } +/** + * bh_end_async_write - I/O end handler for async folio writes + * @bio: The bio being completed. + * + * Pass this function to bh_submit() if you're doing the equivalent of + * block_write_full_folio(). + */ +void bh_end_async_write(struct bio *bio) +{ + struct buffer_head *bh; + bool success = bio_endio_bh(bio, &bh); + end_buffer_async_write(bh, success); +} +EXPORT_SYMBOL(bh_end_async_write); + /* * If a page's buffers are under async readin (end_buffer_async_read * completion) then there is a possibility that another thread of @@ -1150,13 +1228,10 @@ EXPORT_SYMBOL(__bforget); static void end_bio_bh_io_sync(struct bio *bio) { - struct buffer_head *bh = bio->bi_private; + struct buffer_head *bh; + bool uptodate = bio_endio_bh(bio, &bh); - if (unlikely(bio_flagged(bio, BIO_QUIET))) - set_bit(BH_Quiet, &bh->b_state); - - bh->b_end_io(bh, !bio->bi_status); - bio_put(bio); + bh->b_end_io(bh, uptodate); } static void buffer_set_crypto_ctx(struct bio *bio, const struct buffer_head *bh, diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index d59980e4adda..b0a31a90fa79 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -204,6 +204,12 @@ struct buffer_head *create_empty_buffers(struct folio *folio, unsigned long blocksize, unsigned long b_state); void end_buffer_read_sync(struct buffer_head *bh, int uptodate); void end_buffer_write_sync(struct buffer_head *bh, int uptodate); +bool bio_endio_bh(struct bio *bio, struct buffer_head **bhp); + +/* Completion routines suitable for passing to bh_submit() */ +void bh_end_read(struct bio *bio); +void bh_end_write(struct bio *bio); +void bh_end_async_write(struct bio *bio); /* Things to do with metadata buffers list */ void mmb_mark_buffer_dirty(struct buffer_head *bh, struct mapping_metadata_bhs *mmb); From b50e811b8fc9b1f28cfba1b74d5c1ad54e521107 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:18 +0100 Subject: [PATCH 33/69] buffer: Convert write_dirty_buffer to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-6-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index e22a94afa385..6329f2b554da 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -2836,9 +2836,7 @@ void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags) unlock_buffer(bh); return; } - bh->b_end_io = end_buffer_write_sync; - get_bh(bh); - submit_bh(REQ_OP_WRITE | op_flags, bh); + bh_submit(bh, REQ_OP_WRITE | op_flags, bh_end_write); } EXPORT_SYMBOL(write_dirty_buffer); From 8b7846fe5b27c11d883a05afe3481f39252a9e65 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:19 +0100 Subject: [PATCH 34/69] buffer: Convert __bread_slow to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-7-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 6329f2b554da..f8428f802b68 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1326,9 +1326,7 @@ static struct buffer_head *__bread_slow(struct buffer_head *bh) unlock_buffer(bh); return bh; } else { - get_bh(bh); - bh->b_end_io = end_buffer_read_sync; - submit_bh(REQ_OP_READ, bh); + bh_submit(bh, REQ_OP_READ, bh_end_read); wait_on_buffer(bh); if (buffer_uptodate(bh)) return bh; From ac13eb47988fa871abcc3f34df946b5c039b3481 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:20 +0100 Subject: [PATCH 35/69] buffer: Convert __sync_dirty_buffer to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-8-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index f8428f802b68..deabf7479325 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -2857,9 +2857,7 @@ int __sync_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags) return -EIO; } - get_bh(bh); - bh->b_end_io = end_buffer_write_sync; - submit_bh(REQ_OP_WRITE | op_flags, bh); + bh_submit(bh, REQ_OP_WRITE | op_flags, bh_end_write); wait_on_buffer(bh); if (!buffer_uptodate(bh)) return -EIO; From a70db2d6f0bb9a3cf5b49a4bb30d34da955f051c Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:21 +0100 Subject: [PATCH 36/69] buffer: Convert __bh_read to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-9-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index deabf7479325..97c8b9966060 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -3089,9 +3089,7 @@ int __bh_read(struct buffer_head *bh, blk_opf_t op_flags, bool wait) BUG_ON(!buffer_locked(bh)); - get_bh(bh); - bh->b_end_io = end_buffer_read_sync; - submit_bh(REQ_OP_READ | op_flags, bh); + bh_submit(bh, REQ_OP_READ | op_flags, bh_end_read); if (wait) { wait_on_buffer(bh); if (!buffer_uptodate(bh)) From 44d325ebba175e73711a4bd90f30e1bc192242d9 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:22 +0100 Subject: [PATCH 37/69] buffer: Convert __bh_read_batch to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jan Kara Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-10-willy@infradead.org Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 97c8b9966060..d96320c18acb 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -3131,9 +3131,7 @@ void __bh_read_batch(int nr, struct buffer_head *bhs[], continue; } - bh->b_end_io = end_buffer_read_sync; - get_bh(bh); - submit_bh(REQ_OP_READ | op_flags, bh); + bh_submit(bh, REQ_OP_READ | op_flags, bh_end_read); } } EXPORT_SYMBOL(__bh_read_batch); From 5059fb945ca895af80506c7a90d46b7308d6f6c5 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:23 +0100 Subject: [PATCH 38/69] buffer: Convert block_read_full_folio to bh_submit() Avoid an extra indirect function call by using bh_submit() instead of submit_bh(). Since mark_buffer_async_read() would collapse to a single function call, inline it into block_read_full_folio() along with its extensive comment. Convert end_buffer_async_read_io() to bh_end_async_read(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-11-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 65 ++++++++++++++++++++++++++--------------------------- 1 file changed, 32 insertions(+), 33 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index d96320c18acb..b668a73d48ec 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -402,11 +402,13 @@ static void decrypt_bh(struct work_struct *work) } /* - * I/O completion handler for block_read_full_folio() - pages + * I/O completion handler for block_read_full_folio() - folios * which come unlocked at the end of I/O. */ -static void end_buffer_async_read_io(struct buffer_head *bh, int uptodate) +static void bh_end_async_read(struct bio *bio) { + struct buffer_head *bh; + bool uptodate = bio_endio_bh(bio, &bh); struct inode *inode = bh->b_folio->mapping->host; bool decrypt = fscrypt_inode_uses_fs_layer_crypto(inode); struct fsverity_info *vi = NULL; @@ -431,7 +433,7 @@ static void end_buffer_async_read_io(struct buffer_head *bh, int uptodate) } return; } - uptodate = 0; + uptodate = false; } end_buffer_async_read(bh, uptodate); } @@ -494,33 +496,6 @@ void bh_end_async_write(struct bio *bio) } EXPORT_SYMBOL(bh_end_async_write); -/* - * If a page's buffers are under async readin (end_buffer_async_read - * completion) then there is a possibility that another thread of - * control could lock one of the buffers after it has completed - * but while some of the other buffers have not completed. This - * locked buffer would confuse end_buffer_async_read() into not unlocking - * the page. So the absence of BH_Async_Read tells end_buffer_async_read() - * that this buffer is not under async I/O. - * - * The page comes unlocked when it has no locked buffer_async buffers - * left. - * - * PageLocked prevents anyone starting new async I/O reads any of - * the buffers. - * - * PageWriteback is used to prevent simultaneous writeout of the same - * page. - * - * PageLocked prevents anyone from starting writeback of a page which is - * under read I/O (PageWriteback is only ever set against a locked page). - */ -static void mark_buffer_async_read(struct buffer_head *bh) -{ - bh->b_end_io = end_buffer_async_read_io; - set_buffer_async_read(bh); -} - void mark_buffer_async_write(struct buffer_head *bh) { bh->b_end_io = end_buffer_async_write; @@ -2497,9 +2472,33 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) continue; } - mark_buffer_async_read(bh); + /* + * If a folio's buffers are under async readin + * (end_buffer_async_read completion) then there is a + * possibility that another thread of control could lock + * one of the buffers after it has completed but while + * some of the other buffers have not completed. This + * locked buffer would confuse end_buffer_async_read() + * into not unlocking the folio. So the absence of + * BH_Async_Read tells end_buffer_async_read() that this + * buffer is not under async I/O. + * + * The folio comes unlocked when it has no locked + * buffer_async buffers left. + * + * The folio lock prevents anyone starting new async + * I/O reads into any of the buffers. + * + * The writeback flag is used to prevent simultaneous + * writeout of the same folio. + * + * The folio lock prevents anyone from starting writeback + * of a folio which is under read I/O (the writeback + * flag is only ever set on a locked folio). + */ + set_buffer_async_read(bh); if (prev) - submit_bh(REQ_OP_READ, prev); + bh_submit(prev, REQ_OP_READ, bh_end_async_read); prev = bh; } while (iblock++, (bh = bh->b_this_page) != head); @@ -2513,7 +2512,7 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) * in this folio. */ if (prev) - submit_bh(REQ_OP_READ, prev); + bh_submit(prev, REQ_OP_READ, bh_end_async_read); else folio_end_read(folio, !page_error); From cc81507980f2d482ed5a0e9e3483306158f06665 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:24 +0100 Subject: [PATCH 39/69] buffer: Convert __block_write_full_folio to __bh_submit() Avoid an extra indirect function call by using __bh_submit() instead of submit_bh_wbc(). Since there is only one caller of submit_bh_wbc() left, inline it into submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-12-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index b668a73d48ec..50bcef50a4a4 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1271,14 +1271,6 @@ static void __bh_submit(struct buffer_head *bh, blk_opf_t opf, blk_crypto_submit_bio(bio); } -static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh, - enum rw_hint write_hint, - struct writeback_control *wbc) -{ - BUG_ON(!bh->b_end_io); - __bh_submit(bh, opf, write_hint, wbc, end_bio_bh_io_sync); -} - /** * bh_submit - Start I/O against a buffer head * @bh: The buffer head to perform I/O on. @@ -1945,7 +1937,7 @@ int __block_write_full_folio(struct inode *inode, struct folio *folio, continue; } if (test_clear_buffer_dirty(bh)) { - mark_buffer_async_write(bh); + set_buffer_async_write(bh); } else { unlock_buffer(bh); } @@ -1961,8 +1953,9 @@ int __block_write_full_folio(struct inode *inode, struct folio *folio, do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh_wbc(REQ_OP_WRITE | write_flags, bh, - inode->i_write_hint, wbc); + __bh_submit(bh, REQ_OP_WRITE | write_flags, + inode->i_write_hint, wbc, + bh_end_async_write); nr_underway++; } bh = next; @@ -1999,7 +1992,7 @@ recover: if (buffer_mapped(bh) && buffer_dirty(bh) && !buffer_delay(bh)) { lock_buffer(bh); - mark_buffer_async_write(bh); + set_buffer_async_write(bh); } else { /* * The buffer may have been set dirty during @@ -2015,8 +2008,9 @@ recover: struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { clear_buffer_dirty(bh); - submit_bh_wbc(REQ_OP_WRITE | write_flags, bh, - inode->i_write_hint, wbc); + __bh_submit(bh, REQ_OP_WRITE | write_flags, + inode->i_write_hint, wbc, + bh_end_async_write); nr_underway++; } bh = next; @@ -2822,7 +2816,8 @@ EXPORT_SYMBOL(generic_block_bmap); void submit_bh(blk_opf_t opf, struct buffer_head *bh) { - submit_bh_wbc(opf, bh, WRITE_LIFE_NOT_SET, NULL); + BUG_ON(!bh->b_end_io); + __bh_submit(bh, opf, WRITE_LIFE_NOT_SET, NULL, end_bio_bh_io_sync); } EXPORT_SYMBOL(submit_bh); From 576c783cc0d6395feaa56d4a89f2b7b4933a5a3d Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:25 +0100 Subject: [PATCH 40/69] ext4; Convert __ext4_read_bh() to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by converting ext4_end_bitmap_read() from bh_end_io_t to bio_end_io_t and calling bh_submit(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-13-willy@infradead.org Reviewed-by: Jan Kara Cc: linux-ext4@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/ext4/ext4.h | 10 +++++----- fs/ext4/ialloc.c | 6 ++++-- fs/ext4/super.c | 12 ++++++------ 3 files changed, 15 insertions(+), 13 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 94283a991e5c..6af11f0ff1c5 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -2959,7 +2959,7 @@ extern unsigned long ext4_count_dirs(struct super_block *); extern void ext4_mark_bitmap_end(int start_bit, int end_bit, char *bitmap); extern int ext4_init_inode_table(struct super_block *sb, ext4_group_t group, int barrier); -extern void ext4_end_bitmap_read(struct buffer_head *bh, int uptodate); +void ext4_end_bitmap_read(struct bio *bio); /* fast_commit.c */ int ext4_fc_info_show(struct seq_file *seq, void *v); @@ -3184,10 +3184,10 @@ extern struct buffer_head *ext4_sb_bread_unmovable(struct super_block *sb, sector_t block); extern struct buffer_head *ext4_sb_bread_nofail(struct super_block *sb, sector_t block); -extern void ext4_read_bh_nowait(struct buffer_head *bh, blk_opf_t op_flags, - bh_end_io_t *end_io, bool simu_fail); -extern int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags, - bh_end_io_t *end_io, bool simu_fail); +void ext4_read_bh_nowait(struct buffer_head *bh, blk_opf_t op_flags, + bio_end_io_t end_io, bool simu_fail); +int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags, + bio_end_io_t end_io, bool simu_fail); extern int ext4_read_bh_lock(struct buffer_head *bh, blk_opf_t op_flags, bool wait); extern void ext4_sb_breadahead_unmovable(struct super_block *sb, sector_t block); extern int ext4_seq_options_show(struct seq_file *seq, void *offset); diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c index 3fd8f0099852..9aacd629a683 100644 --- a/fs/ext4/ialloc.c +++ b/fs/ext4/ialloc.c @@ -66,14 +66,16 @@ void ext4_mark_bitmap_end(int start_bit, int end_bit, char *bitmap) memset(bitmap + (i >> 3), 0xff, (end_bit - i) >> 3); } -void ext4_end_bitmap_read(struct buffer_head *bh, int uptodate) +void ext4_end_bitmap_read(struct bio *bio) { + struct buffer_head *bh; + bool uptodate = bio_endio_bh(bio, &bh); + if (uptodate) { set_buffer_uptodate(bh); set_bitmap_uptodate(bh); } unlock_buffer(bh); - put_bh(bh); } static int ext4_validate_inode_bitmap(struct super_block *sb, diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 6a77db4d3124..bc7faedcb8e4 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -161,7 +161,7 @@ MODULE_ALIAS("ext3"); static inline void __ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags, - bh_end_io_t *end_io, bool simu_fail) + bio_end_io_t end_io, bool simu_fail) { if (simu_fail) { clear_buffer_uptodate(bh); @@ -176,13 +176,13 @@ static inline void __ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags, */ clear_buffer_verified(bh); - bh->b_end_io = end_io ? end_io : end_buffer_read_sync; - get_bh(bh); - submit_bh(REQ_OP_READ | op_flags, bh); + if (!end_io) + end_io = bh_end_read; + bh_submit(bh, REQ_OP_READ | op_flags, end_io); } void ext4_read_bh_nowait(struct buffer_head *bh, blk_opf_t op_flags, - bh_end_io_t *end_io, bool simu_fail) + bio_end_io_t end_io, bool simu_fail) { BUG_ON(!buffer_locked(bh)); @@ -194,7 +194,7 @@ void ext4_read_bh_nowait(struct buffer_head *bh, blk_opf_t op_flags, } int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags, - bh_end_io_t *end_io, bool simu_fail) + bio_end_io_t end_io, bool simu_fail) { BUG_ON(!buffer_locked(bh)); From cf3c71eed0b7c089f96905fdfe3cec994737f24f Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:26 +0100 Subject: [PATCH 41/69] ext4: Convert ext4_fc_submit_bh() to bh_submit() Avoid an extra indirect function call by converting ext4_end_buffer_io_sync() from bh_end_io_t to bio_end_io_t and calling bh_submit(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-14-willy@infradead.org Reviewed-by: Jan Kara Cc: linux-ext4@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/ext4/fast_commit.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/fs/ext4/fast_commit.c b/fs/ext4/fast_commit.c index b3c22636251d..5773b85e43cb 100644 --- a/fs/ext4/fast_commit.c +++ b/fs/ext4/fast_commit.c @@ -184,8 +184,11 @@ #include static struct kmem_cache *ext4_fc_dentry_cachep; -static void ext4_end_buffer_io_sync(struct buffer_head *bh, int uptodate) +static void ext4_end_buffer_io_sync(struct bio *bio) { + struct buffer_head *bh; + bool uptodate = bio_endio_bh(bio, &bh); + BUFFER_TRACE(bh, ""); if (uptodate) { ext4_debug("%s: Block %lld up-to-date", @@ -659,8 +662,7 @@ static void ext4_fc_submit_bh(struct super_block *sb, bool is_tail) lock_buffer(bh); set_buffer_dirty(bh); set_buffer_uptodate(bh); - bh->b_end_io = ext4_end_buffer_io_sync; - submit_bh(REQ_OP_WRITE | write_flags, bh); + bh_submit(bh, REQ_OP_WRITE | write_flags, ext4_end_buffer_io_sync); EXT4_SB(sb)->s_fc_bh = NULL; } From 786a9941c727d5cebdc40557bb28117c9d0c21ad Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:27 +0100 Subject: [PATCH 42/69] ext4: Convert write_mmp_block_thawed() to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-15-willy@infradead.org Reviewed-by: Jan Kara Cc: linux-ext4@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/ext4/mmp.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fs/ext4/mmp.c b/fs/ext4/mmp.c index 6f57c181ff77..7ce361484b38 100644 --- a/fs/ext4/mmp.c +++ b/fs/ext4/mmp.c @@ -46,9 +46,8 @@ static int write_mmp_block_thawed(struct super_block *sb, ext4_mmp_csum_set(sb, mmp); lock_buffer(bh); - bh->b_end_io = end_buffer_write_sync; - get_bh(bh); - submit_bh(REQ_OP_WRITE | REQ_SYNC | REQ_META | REQ_PRIO, bh); + bh_submit(bh, REQ_OP_WRITE | REQ_SYNC | REQ_META | REQ_PRIO, + bh_end_write); wait_on_buffer(bh); if (unlikely(!buffer_uptodate(bh))) return -EIO; From aa17449d60c7f088d4ffb77044d88c4692fc76bb Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:28 +0100 Subject: [PATCH 43/69] ext4: Convert ext4_commit_super() to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-16-willy@infradead.org Acked-by: Theodore Ts'o Reviewed-by: Jan Kara Cc: linux-ext4@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/ext4/super.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/fs/ext4/super.c b/fs/ext4/super.c index bc7faedcb8e4..7283108d7609 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -6316,12 +6316,10 @@ static int ext4_commit_super(struct super_block *sb) clear_buffer_write_io_error(sbh); set_buffer_uptodate(sbh); } - get_bh(sbh); /* Clear potential dirty bit if it was journalled update */ clear_buffer_dirty(sbh); - sbh->b_end_io = end_buffer_write_sync; - submit_bh(REQ_OP_WRITE | REQ_SYNC | - (test_opt(sb, BARRIER) ? REQ_FUA : 0), sbh); + bh_submit(sbh, REQ_OP_WRITE | REQ_SYNC | + (test_opt(sb, BARRIER) ? REQ_FUA : 0), bh_end_write); wait_on_buffer(sbh); if (buffer_write_io_error(sbh)) { ext4_msg(sb, KERN_ERR, "I/O error while writing " From e4b09ba06e6f884dbbfc6f43ecc079ef6864c4e2 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:29 +0100 Subject: [PATCH 44/69] jbd2: Convert journal commit to bh_submit() Avoid an extra indirect function call by using bh_submit() instead of submit_bh() in journal_submit_commit_record() and jbd2_journal_commit_transaction(). These both use journal_end_buffer_io_sync(), so it's more straightforward to do them both at once. Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-17-willy@infradead.org Acked-by: Theodore Ts'o Reviewed-by: Jan Kara Cc: linux-ext4@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/jbd2/commit.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c index 8cf61e7185c4..4e91593d27e5 100644 --- a/fs/jbd2/commit.c +++ b/fs/jbd2/commit.c @@ -29,8 +29,10 @@ /* * IO end handler for temporary buffer_heads handling writes to the journal. */ -static void journal_end_buffer_io_sync(struct buffer_head *bh, int uptodate) +static void journal_end_buffer_io_sync(struct bio *bio) { + struct buffer_head *bh; + bool uptodate = bio_endio_bh(bio, &bh); struct buffer_head *orig_bh = bh->b_private; BUFFER_TRACE(bh, ""); @@ -147,13 +149,12 @@ static int journal_submit_commit_record(journal_t *journal, lock_buffer(bh); clear_buffer_dirty(bh); set_buffer_uptodate(bh); - bh->b_end_io = journal_end_buffer_io_sync; if (journal->j_flags & JBD2_BARRIER && !jbd2_has_feature_async_commit(journal)) write_flags |= REQ_PREFLUSH | REQ_FUA; - submit_bh(write_flags, bh); + bh_submit(bh, write_flags, journal_end_buffer_io_sync); *cbh = bh; return 0; } @@ -751,9 +752,9 @@ start_journal_io: lock_buffer(bh); clear_buffer_dirty(bh); set_buffer_uptodate(bh); - bh->b_end_io = journal_end_buffer_io_sync; - submit_bh(REQ_OP_WRITE | JBD2_JOURNAL_REQ_FLAGS, - bh); + bh_submit(bh, + REQ_OP_WRITE | JBD2_JOURNAL_REQ_FLAGS, + journal_end_buffer_io_sync); } cond_resched(); From 8a30f324e2cd93605038ee20ff25d0dec58839b6 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:30 +0100 Subject: [PATCH 45/69] jbd2: Convert jbd2_write_superblock() to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-18-willy@infradead.org Acked-by: Theodore Ts'o Reviewed-by: Jan Kara Cc: linux-ext4@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/jbd2/journal.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c index 4f397fcdb13c..2040af8c84cb 100644 --- a/fs/jbd2/journal.c +++ b/fs/jbd2/journal.c @@ -1820,9 +1820,7 @@ static int jbd2_write_superblock(journal_t *journal, blk_opf_t write_flags) } if (jbd2_journal_has_csum_v2or3(journal)) sb->s_checksum = jbd2_superblock_csum(sb); - get_bh(bh); - bh->b_end_io = end_buffer_write_sync; - submit_bh(REQ_OP_WRITE | write_flags, bh); + bh_submit(bh, REQ_OP_WRITE | write_flags, bh_end_write); wait_on_buffer(bh); if (buffer_write_io_error(bh)) { clear_buffer_write_io_error(bh); From 36ec1a90fe117ff2255a497eb3b73e6587b2ed70 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:31 +0100 Subject: [PATCH 46/69] ocfs2: Convert ocfs2_write_block to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-19-willy@infradead.org Reviewed-by: Joseph Qi Reviewed-by: Jan Kara Cc: ocfs2-devel@lists.linux.dev Signed-off-by: Christian Brauner (Amutable) --- fs/ocfs2/buffer_head_io.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/ocfs2/buffer_head_io.c b/fs/ocfs2/buffer_head_io.c index 701d27d908d4..714daa4bbda2 100644 --- a/fs/ocfs2/buffer_head_io.c +++ b/fs/ocfs2/buffer_head_io.c @@ -62,9 +62,7 @@ int ocfs2_write_block(struct ocfs2_super *osb, struct buffer_head *bh, /* remove from dirty list before I/O. */ clear_buffer_dirty(bh); - get_bh(bh); /* for end_buffer_write_sync() */ - bh->b_end_io = end_buffer_write_sync; - submit_bh(REQ_OP_WRITE, bh); + bh_submit(bh, REQ_OP_WRITE, bh_end_write); wait_on_buffer(bh); From 4b08d56b09d75efb0788dbaea619cf0f5d509576 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:32 +0100 Subject: [PATCH 47/69] ocfs2: Convert ocfs2_read_block to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-20-willy@infradead.org Reviewed-by: Joseph Qi Reviewed-by: Jan Kara Cc: ocfs2-devel@lists.linux.dev Signed-off-by: Christian Brauner (Amutable) --- fs/ocfs2/buffer_head_io.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/ocfs2/buffer_head_io.c b/fs/ocfs2/buffer_head_io.c index 714daa4bbda2..9edaeef5c4b9 100644 --- a/fs/ocfs2/buffer_head_io.c +++ b/fs/ocfs2/buffer_head_io.c @@ -143,9 +143,7 @@ int ocfs2_read_blocks_sync(struct ocfs2_super *osb, u64 block, #endif } - get_bh(bh); /* for end_buffer_read_sync() */ - bh->b_end_io = end_buffer_read_sync; - submit_bh(REQ_OP_READ, bh); + bh_submit(bh, REQ_OP_READ, bh_end_read); } read_failure: From 4416bb661b1cb40d78837f630755d2b8440cf4f3 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:33 +0100 Subject: [PATCH 48/69] ocfs2: Convert ocfs2_read_blocks to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-21-willy@infradead.org Reviewed-by: Joseph Qi Reviewed-by: Jan Kara Cc: ocfs2-devel@lists.linux.dev Signed-off-by: Christian Brauner (Amutable) --- fs/ocfs2/buffer_head_io.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/ocfs2/buffer_head_io.c b/fs/ocfs2/buffer_head_io.c index 9edaeef5c4b9..185ff4dcba75 100644 --- a/fs/ocfs2/buffer_head_io.c +++ b/fs/ocfs2/buffer_head_io.c @@ -319,11 +319,9 @@ int ocfs2_read_blocks(struct ocfs2_caching_info *ci, u64 block, int nr, continue; } - get_bh(bh); /* for end_buffer_read_sync() */ if (validate) set_buffer_needs_validate(bh); - bh->b_end_io = end_buffer_read_sync; - submit_bh(REQ_OP_READ, bh); + bh_submit(bh, REQ_OP_READ, bh_end_read); continue; } } From 184dec3146911b923a7eff11545f85797c2b9f81 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:34 +0100 Subject: [PATCH 49/69] ocfs2: Convert ocfs2_write_super_or_backup to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-22-willy@infradead.org Reviewed-by: Joseph Qi Reviewed-by: Jan Kara Cc: ocfs2-devel@lists.linux.dev Signed-off-by: Christian Brauner (Amutable) --- fs/ocfs2/buffer_head_io.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/ocfs2/buffer_head_io.c b/fs/ocfs2/buffer_head_io.c index 185ff4dcba75..8e67eba70674 100644 --- a/fs/ocfs2/buffer_head_io.c +++ b/fs/ocfs2/buffer_head_io.c @@ -440,10 +440,8 @@ int ocfs2_write_super_or_backup(struct ocfs2_super *osb, /* remove from dirty list before I/O. */ clear_buffer_dirty(bh); - get_bh(bh); /* for end_buffer_write_sync() */ - bh->b_end_io = end_buffer_write_sync; ocfs2_compute_meta_ecc(osb->sb, bh->b_data, &di->i_check); - submit_bh(REQ_OP_WRITE, bh); + bh_submit(bh, REQ_OP_WRITE, bh_end_write); wait_on_buffer(bh); From f8ff032ce5f0c08348b55468330a67c93b58848b Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:35 +0100 Subject: [PATCH 50/69] gfs2: Convert gfs2_metapath_ra to bh_submit() Avoid an extra indirect function call by using bh_submit() instead of submit_bh(). Also simplify the control flow now that the buffer refcount is not put by bh_end_read(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-23-willy@infradead.org Reviewed-by: Jan Kara Cc: gfs2@lists.linux.dev Signed-off-by: Christian Brauner (Amutable) --- fs/gfs2/bmap.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c index b3d7fcd95f03..d158c4b7413d 100644 --- a/fs/gfs2/bmap.c +++ b/fs/gfs2/bmap.c @@ -304,14 +304,15 @@ static void gfs2_metapath_ra(struct gfs2_glock *gl, __be64 *start, __be64 *end) rabh = gfs2_getbuf(gl, be64_to_cpu(*t), CREATE); if (trylock_buffer(rabh)) { if (!buffer_uptodate(rabh)) { - rabh->b_end_io = end_buffer_read_sync; - submit_bh(REQ_OP_READ | REQ_RAHEAD | REQ_META | - REQ_PRIO, rabh); - continue; + bh_submit(rabh, + REQ_OP_READ | REQ_RAHEAD | REQ_META | + REQ_PRIO, + bh_end_read); + } else { + unlock_buffer(rabh); } - unlock_buffer(rabh); } - brelse(rabh); + put_bh(rabh); } } From 6390838da4901e4a332c11b1d6ddc20f167005f9 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:36 +0100 Subject: [PATCH 51/69] gfs2: Convert gfs2_dir_readahead to bh_submit() Avoid an extra indirect function call by using bh_submit() instead of submit_bh(). Also simplify the control flow now that the buffer refcount is not put by bh_end_read(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-24-willy@infradead.org Reviewed-by: Jan Kara Cc: gfs2@lists.linux.dev Signed-off-by: Christian Brauner (Amutable) --- fs/gfs2/dir.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/fs/gfs2/dir.c b/fs/gfs2/dir.c index 022dbb31e0d9..0237b36b9eb1 100644 --- a/fs/gfs2/dir.c +++ b/fs/gfs2/dir.c @@ -1505,15 +1505,13 @@ static void gfs2_dir_readahead(struct inode *inode, unsigned hsize, u32 index, if (trylock_buffer(bh)) { if (buffer_uptodate(bh)) { unlock_buffer(bh); - brelse(bh); - continue; + } else { + bh_submit(bh, REQ_OP_READ | REQ_RAHEAD | + REQ_META | REQ_PRIO, + bh_end_read); } - bh->b_end_io = end_buffer_read_sync; - submit_bh(REQ_OP_READ | REQ_RAHEAD | REQ_META | - REQ_PRIO, bh); - continue; } - brelse(bh); + put_bh(bh); } } From 037f3833ba9f893d2a84c9a9fd74d78201bb2b35 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:37 +0100 Subject: [PATCH 52/69] gfs2: Remove use of b_end_io in gfs2_meta_read_endio() All buffer heads submitted by gfs2_submit_bhs() use end_buffer_read_sync() so we can call it directly. Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-25-willy@infradead.org Reviewed-by: Jan Kara Cc: gfs2@lists.linux.dev Signed-off-by: Christian Brauner (Amutable) --- fs/gfs2/meta_io.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index d407dd476e72..df5f05054f6e 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -212,7 +212,7 @@ static void gfs2_meta_read_endio(struct bio *bio) do { struct buffer_head *next = bh->b_this_page; len -= bh->b_size; - bh->b_end_io(bh, !bio->bi_status); + end_buffer_read_sync(bh, bio->bi_status == BLK_STS_OK); bh = next; } while (bh && len); } @@ -275,7 +275,6 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags, unlock_buffer(bh); flags &= ~DIO_WAIT; } else { - bh->b_end_io = end_buffer_read_sync; get_bh(bh); bhs[num++] = bh; } @@ -288,7 +287,6 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags, unlock_buffer(bh); brelse(bh); } else { - bh->b_end_io = end_buffer_read_sync; bhs[num++] = bh; } } From d755b0a9949b46f207ed079378c40e0181f398b1 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:38 +0100 Subject: [PATCH 53/69] gfs2: Convert gfs2_aspace_write_folio to bh_submit() Avoid an extra indirect function call by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-26-willy@infradead.org Reviewed-by: Jan Kara Cc: gfs2@lists.linux.dev Signed-off-by: Christian Brauner (Amutable) --- fs/gfs2/meta_io.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index df5f05054f6e..4895bb010cfb 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -59,7 +59,7 @@ static void gfs2_aspace_write_folio(struct folio *folio, continue; } if (test_clear_buffer_dirty(bh)) { - mark_buffer_async_write(bh); + set_buffer_async_write(bh); } else { unlock_buffer(bh); } @@ -75,7 +75,8 @@ static void gfs2_aspace_write_folio(struct folio *folio, do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh(REQ_OP_WRITE | write_flags, bh); + bh_submit(bh, REQ_OP_WRITE | write_flags, + bh_end_async_write); nr_underway++; } bh = next; @@ -221,7 +222,7 @@ static void gfs2_meta_read_endio(struct bio *bio) /* * Submit several consecutive buffer head I/O requests as a single bio I/O - * request. (See submit_bh_wbc.) + * request. (See bh_submit.) */ static void gfs2_submit_bhs(blk_opf_t opf, struct buffer_head *bhs[], int num) { From e6eff926482d0f6ebe038a15348158682f5711a0 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:39 +0100 Subject: [PATCH 54/69] buffer: Remove mark_buffer_async_write() There are no more callers of this function, so delete it. end_buffer_async_write() then has only one caller left, so inline it into bh_end_async_write(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-27-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 38 ++++++++++++------------------------- include/linux/buffer_head.h | 1 - 2 files changed, 12 insertions(+), 27 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 50bcef50a4a4..d35f584a06f9 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -438,12 +438,19 @@ static void bh_end_async_read(struct bio *bio) end_buffer_async_read(bh, uptodate); } -/* - * Completion handler for block_write_full_folio() - folios which are unlocked - * during I/O, and which have the writeback flag cleared upon I/O completion. +/** + * bh_end_async_write - I/O end handler for async folio writes + * @bio: The bio being completed. + * + * Pass this function to bh_submit() if you're doing the equivalent of + * block_write_full_folio(). That is, the folio is unlocked, and will + * have its writeback flag cleared once all async write buffers have + * completed. */ -static void end_buffer_async_write(struct buffer_head *bh, int uptodate) +void bh_end_async_write(struct bio *bio) { + struct buffer_head *bh; + bool success = bio_endio_bh(bio, &bh); unsigned long flags; struct buffer_head *first; struct buffer_head *tmp; @@ -452,7 +459,7 @@ static void end_buffer_async_write(struct buffer_head *bh, int uptodate) BUG_ON(!buffer_async_write(bh)); folio = bh->b_folio; - if (uptodate) { + if (success) { set_buffer_uptodate(bh); } else { buffer_io_error(bh, ", lost async page write"); @@ -480,29 +487,8 @@ static void end_buffer_async_write(struct buffer_head *bh, int uptodate) still_busy: spin_unlock_irqrestore(&first->b_uptodate_lock, flags); } - -/** - * bh_end_async_write - I/O end handler for async folio writes - * @bio: The bio being completed. - * - * Pass this function to bh_submit() if you're doing the equivalent of - * block_write_full_folio(). - */ -void bh_end_async_write(struct bio *bio) -{ - struct buffer_head *bh; - bool success = bio_endio_bh(bio, &bh); - end_buffer_async_write(bh, success); -} EXPORT_SYMBOL(bh_end_async_write); -void mark_buffer_async_write(struct buffer_head *bh) -{ - bh->b_end_io = end_buffer_async_write; - set_buffer_async_write(bh); -} -EXPORT_SYMBOL(mark_buffer_async_write); - /* * fs/buffer.c contains helper functions for buffer-backed address space's diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index b0a31a90fa79..f0a9f3f05ac1 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -224,7 +224,6 @@ static inline void clean_bdev_bh_alias(struct buffer_head *bh) clean_bdev_aliases(bh->b_bdev, bh->b_blocknr, 1); } -void mark_buffer_async_write(struct buffer_head *bh); void __wait_on_buffer(struct buffer_head *); wait_queue_head_t *bh_waitq_head(struct buffer_head *bh); struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block, From 673c1546cdd73728bc353bdadbbeea6e2dbef42a Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:40 +0100 Subject: [PATCH 55/69] nilfs2: Convert nilfs_btnode_submit_block to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-28-willy@infradead.org Acked-by: Ryusuke Konishi Reviewed-by: Viacheslav Dubeyko Reviewed-by: Jan Kara Cc: linux-nilfs@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/nilfs2/btnode.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/nilfs2/btnode.c b/fs/nilfs2/btnode.c index 2e553d698d0f..680e4009a30d 100644 --- a/fs/nilfs2/btnode.c +++ b/fs/nilfs2/btnode.c @@ -134,9 +134,7 @@ int nilfs_btnode_submit_block(struct address_space *btnc, __u64 blocknr, } set_buffer_mapped(bh); bh->b_blocknr = pblocknr; /* set block address for read */ - bh->b_end_io = end_buffer_read_sync; - get_bh(bh); - submit_bh(opf, bh); + bh_submit(bh, opf, bh_end_read); bh->b_blocknr = blocknr; /* set back to the given block address */ *submit_ptr = pblocknr; err = 0; From 143a7cea4280562a0cdc3ca43e6e1ced4f8ec61d Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:41 +0100 Subject: [PATCH 56/69] nilfs2: Convert nilfs_gccache_submit_read_data to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-29-willy@infradead.org Acked-by: Ryusuke Konishi Reviewed-by: Viacheslav Dubeyko Reviewed-by: Jan Kara Cc: linux-nilfs@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/nilfs2/gcinode.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/nilfs2/gcinode.c b/fs/nilfs2/gcinode.c index 62d4c1b787e9..85379ac23ae2 100644 --- a/fs/nilfs2/gcinode.c +++ b/fs/nilfs2/gcinode.c @@ -83,9 +83,7 @@ int nilfs_gccache_submit_read_data(struct inode *inode, sector_t blkoff, if (!buffer_mapped(bh)) set_buffer_mapped(bh); bh->b_blocknr = pbn; - bh->b_end_io = end_buffer_read_sync; - get_bh(bh); - submit_bh(REQ_OP_READ, bh); + bh_submit(bh, REQ_OP_READ, bh_end_read); if (vbn) bh->b_blocknr = vbn; out: From b9fa6916b1141e3648779e99039b02d4c57225f6 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:42 +0100 Subject: [PATCH 57/69] nilfs2: Convert nilfs_mdt_submit_block to bh_submit() Avoid an extra indirect function call and changing the buffer refcount by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-30-willy@infradead.org Acked-by: Ryusuke Konishi Reviewed-by: Viacheslav Dubeyko Reviewed-by: Jan Kara Cc: linux-nilfs@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/nilfs2/mdt.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/nilfs2/mdt.c b/fs/nilfs2/mdt.c index 09adb40c65e5..2a435349fd21 100644 --- a/fs/nilfs2/mdt.c +++ b/fs/nilfs2/mdt.c @@ -148,9 +148,7 @@ nilfs_mdt_submit_block(struct inode *inode, unsigned long blkoff, blk_opf_t opf, } map_bh(bh, inode->i_sb, (sector_t)blknum); - bh->b_end_io = end_buffer_read_sync; - get_bh(bh); - submit_bh(opf, bh); + bh_submit(bh, opf, bh_end_read); ret = 0; trace_nilfs2_mdt_submit_block(inode, inode->i_ino, blkoff, From 2dc07082e3f5d51ae3bf293f9c39c192a1227242 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:43 +0100 Subject: [PATCH 58/69] md-bitmap: Convert read_file_page and write_file_page to bh_submit() Avoid an extra indirect function call by using bh_submit() instead of submit_bh(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-31-willy@infradead.org Reviewed-by: Jan Kara Cc: linux-raid@vger.kernel.org Signed-off-by: Christian Brauner (Amutable) --- drivers/md/md-bitmap.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c index 83378c033c72..ffc9891be22b 100644 --- a/drivers/md/md-bitmap.c +++ b/drivers/md/md-bitmap.c @@ -501,6 +501,18 @@ static void write_sb_page(struct bitmap *bitmap, unsigned long pg_index, static void md_bitmap_file_kick(struct bitmap *bitmap); #ifdef CONFIG_MD_BITMAP_FILE +static void end_bitmap_write(struct bio *bio) +{ + struct buffer_head *bh; + bool uptodate = bio_endio_bh(bio, &bh); + struct bitmap *bitmap = bh->b_private; + + if (!uptodate) + set_bit(BITMAP_WRITE_ERROR, &bitmap->flags); + if (atomic_dec_and_test(&bitmap->pending_writes)) + wake_up(&bitmap->write_wait); +} + static void write_file_page(struct bitmap *bitmap, struct page *page, int wait) { struct buffer_head *bh = page_buffers(page); @@ -509,7 +521,7 @@ static void write_file_page(struct bitmap *bitmap, struct page *page, int wait) atomic_inc(&bitmap->pending_writes); set_buffer_locked(bh); set_buffer_mapped(bh); - submit_bh(REQ_OP_WRITE | REQ_SYNC, bh); + bh_submit(bh, REQ_OP_WRITE | REQ_SYNC, end_bitmap_write); bh = bh->b_this_page; } @@ -518,16 +530,6 @@ static void write_file_page(struct bitmap *bitmap, struct page *page, int wait) atomic_read(&bitmap->pending_writes) == 0); } -static void end_bitmap_write(struct buffer_head *bh, int uptodate) -{ - struct bitmap *bitmap = bh->b_private; - - if (!uptodate) - set_bit(BITMAP_WRITE_ERROR, &bitmap->flags); - if (atomic_dec_and_test(&bitmap->pending_writes)) - wake_up(&bitmap->write_wait); -} - static void free_buffers(struct page *page) { struct buffer_head *bh; @@ -591,12 +593,11 @@ static int read_file_page(struct file *file, unsigned long index, else count -= blocksize; - bh->b_end_io = end_bitmap_write; bh->b_private = bitmap; atomic_inc(&bitmap->pending_writes); set_buffer_locked(bh); set_buffer_mapped(bh); - submit_bh(REQ_OP_READ, bh); + bh_submit(bh, REQ_OP_READ, end_bitmap_write); } blk_cur++; bh = bh->b_this_page; From 2911b935825523a71a71eecfe908902d181c041e Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:44 +0100 Subject: [PATCH 59/69] buffer: Remove submit_bh() No users are left; remove this API. Also remove/fix comments mentioning it, and end_bio_bh_io_sync() as it's now unused. Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-32-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- Documentation/trace/ftrace.rst | 4 ++-- fs/buffer.c | 27 ++++++--------------------- include/linux/buffer_head.h | 3 +-- mm/vmscan.c | 2 +- 4 files changed, 10 insertions(+), 26 deletions(-) diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst index b9efb148a5c2..2ed1b96e440b 100644 --- a/Documentation/trace/ftrace.rst +++ b/Documentation/trace/ftrace.rst @@ -1624,7 +1624,7 @@ function-trace, we get a much larger output:: => blk_queue_bio => submit_bio_noacct => submit_bio - => submit_bh + => bh_submit => __ext3_get_inode_loc => ext3_iget => ext3_lookup @@ -1909,7 +1909,7 @@ tracers. => blk_queue_bio => submit_bio_noacct => submit_bio - => submit_bh + => bh_submit => ext3_bread => ext3_dir_bread => htree_dirblock_to_tree diff --git a/fs/buffer.c b/fs/buffer.c index d35f584a06f9..3df0ea1a6342 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1187,14 +1187,6 @@ void __bforget(struct buffer_head *bh) } EXPORT_SYMBOL(__bforget); -static void end_bio_bh_io_sync(struct bio *bio) -{ - struct buffer_head *bh; - bool uptodate = bio_endio_bh(bio, &bh); - - bh->b_end_io(bh, uptodate); -} - static void buffer_set_crypto_ctx(struct bio *bio, const struct buffer_head *bh, gfp_t gfp_mask) { @@ -1829,15 +1821,15 @@ static struct buffer_head *folio_create_buffers(struct folio *folio, /* * While block_write_full_folio is writing back the dirty buffers under - * the page lock, whoever dirtied the buffers may decide to clean them + * the folio lock, whoever dirtied the buffers may decide to clean them * again at any time. We handle that by only looking at the buffer * state inside lock_buffer(). * * If block_write_full_folio() is called for regular writeback - * (wbc->sync_mode == WB_SYNC_NONE) then it will redirty a page which has a - * locked buffer. This only can happen if someone has written the buffer - * directly, with submit_bh(). At the address_space level PageWriteback - * prevents this contention from occurring. + * (wbc->sync_mode == WB_SYNC_NONE) then it will redirty a folio which + * has a locked buffer. This only can happen if someone has written + * the buffer directly, with bh_submit(). At the address_space level + * the folio writeback flag prevents this contention from occurring. * * If block_write_full_folio() is called with wbc->sync_mode == * WB_SYNC_ALL, the writes are posted using REQ_SYNC; this @@ -1954,7 +1946,7 @@ done: /* * The folio was marked dirty, but the buffers were * clean. Someone wrote them back by hand with - * write_dirty_buffer/submit_bh. A rare case. + * write_dirty_buffer/bh_submit. A rare case. */ folio_end_writeback(folio); @@ -2800,13 +2792,6 @@ sector_t generic_block_bmap(struct address_space *mapping, sector_t block, } EXPORT_SYMBOL(generic_block_bmap); -void submit_bh(blk_opf_t opf, struct buffer_head *bh) -{ - BUG_ON(!bh->b_end_io); - __bh_submit(bh, opf, WRITE_LIFE_NOT_SET, NULL, end_bio_bh_io_sync); -} -EXPORT_SYMBOL(submit_bh); - void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags) { lock_buffer(bh); diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index f0a9f3f05ac1..7629130d42c4 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -55,7 +55,7 @@ typedef void (bh_end_io_t)(struct buffer_head *bh, int uptodate); * is the bio, and buffer_heads are used for extracting block * mappings (via a get_block_t call), for tracking state within * a folio (via a folio_mapping) and for wrapping bio submission - * for backward compatibility reasons (e.g. submit_bh). + * for backward compatibility reasons (e.g. bh_submit). */ struct buffer_head { unsigned long b_state; /* buffer state bitmap (see above) */ @@ -244,7 +244,6 @@ void __lock_buffer(struct buffer_head *bh); int sync_dirty_buffer(struct buffer_head *bh); int __sync_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags); void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags); -void submit_bh(blk_opf_t, struct buffer_head *); void bh_submit(struct buffer_head *, blk_opf_t, bio_end_io_t); void write_boundary_block(struct block_device *bdev, sector_t bblock, unsigned blocksize); diff --git a/mm/vmscan.c b/mm/vmscan.c index bd1b1aa12581..67231d3189ef 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1449,7 +1449,7 @@ retry: * is possible for a folio to have the dirty flag set, * but it is actually clean (all its buffers are clean). * This happens if the buffers were written out directly, - * with submit_bh(). ext3 will do this, as well as + * with bh_submit(). ext3 will do this, as well as * the blockdev mapping. filemap_release_folio() will * discover that cleanness and will drop the buffers * and mark the folio clean - it can be freed. From ac75b922bb67cc8edb52006c9346dc0ca91d04c8 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:45 +0100 Subject: [PATCH 60/69] buffer: Remove b_end_io This shrinks buffer_head by 8 bytes, letting us pack more buffer heads per slab. With a Debian config, it shrinks from 104 bytes to 96 bytes which is 42 objects per 4KiB page rather than 39, a 7% reduction in the amount of memory used. Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-33-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- Documentation/filesystems/locking.rst | 14 -------------- drivers/md/raid5.h | 6 +++--- fs/buffer.c | 1 - include/linux/buffer_head.h | 4 +--- 4 files changed, 4 insertions(+), 21 deletions(-) diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst index 8421ea21bd35..a27aca42fd85 100644 --- a/Documentation/filesystems/locking.rst +++ b/Documentation/filesystems/locking.rst @@ -416,20 +416,6 @@ lm_open_conflict yes no no lm_breaker_timedout yes no no ====================== ============= ================= ========= -buffer_head -=========== - -prototypes:: - - void (*b_end_io)(struct buffer_head *bh, int uptodate); - -locking rules: - -called from interrupts. In other words, extreme care is needed here. -bh is locked, but that's all warranties we have here. Currently only RAID1, -highmem, fs/buffer.c, and fs/ntfs/aops.c are providing these. Block devices -call this method upon the IO completion. - block_device_operations ======================= prototypes:: diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h index 1c7b710fc9c1..1dfa60a41d91 100644 --- a/drivers/md/raid5.h +++ b/drivers/md/raid5.h @@ -38,7 +38,7 @@ * Clean -> Dirty - on compute_parity to satisfy write/sync (RECONSTRUCT or RMW) * * The Want->Empty, Want->Clean, Dirty->Clean, transitions - * all happen in b_end_io at interrupt time. + * all happen in end_io at interrupt time. * Each sets the Uptodate bit before releasing the Lock bit. * This leaves one multi-stage transition: * Want->Dirty->Clean @@ -64,7 +64,7 @@ * together, but we are not guaranteed of that so we allow for more. * * If a buffer is on the read list when the associated cache buffer is - * Uptodate, the data is copied into the read buffer and it's b_end_io + * Uptodate, the data is copied into the read buffer and it's end_io * routine is called. This may happen in the end_request routine only * if the buffer has just successfully been read. end_request should * remove the buffers from the list and then set the Uptodate bit on @@ -76,7 +76,7 @@ * into the cache buffer, which is then marked dirty, and moved onto a * third list, the written list (bh_written). Once both the parity * block and the cached buffer are successfully written, any buffer on - * a written list can be returned with b_end_io. + * a written list can be returned with end_io. * * The write list and read list both act as fifos. The read list, * write list and written list are protected by the device_lock. diff --git a/fs/buffer.c b/fs/buffer.c index 3df0ea1a6342..ccda92920175 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -946,7 +946,6 @@ static sector_t folio_init_buffers(struct folio *folio, do { if (!buffer_mapped(bh)) { - bh->b_end_io = NULL; bh->b_private = NULL; bh->b_bdev = bdev; bh->b_blocknr = block; diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index 7629130d42c4..1ee56c9f2327 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -46,7 +46,6 @@ enum bh_state_bits { struct page; struct buffer_head; struct address_space; -typedef void (bh_end_io_t)(struct buffer_head *bh, int uptodate); /* * Historically, a buffer_head was used to map a single block @@ -70,8 +69,7 @@ struct buffer_head { char *b_data; /* pointer to data within the page */ struct block_device *b_bdev; - bh_end_io_t *b_end_io; /* I/O completion */ - void *b_private; /* reserved for b_end_io */ + void *b_private; /* reserved for bio_end_io */ struct list_head b_assoc_buffers; /* associated with another mapping */ struct mapping_metadata_bhs *b_mmb; /* head of the list of metadata bhs * this buffer is associated with */ From 15dfe154126743baaf3be669cdec893f3f174118 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:46 +0100 Subject: [PATCH 61/69] buffer: Change calling convention for end_buffer_read_sync() Unify end_buffer_read_sync() and __end_buffer_read_notouch() by requiring the caller put the refcount on the buffer. The only caller is in the gfs2_meta_read() path, and there we can put the refcount after locking the buffer. Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-34-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 28 +++++++++------------------- fs/gfs2/meta_io.c | 2 +- 2 files changed, 10 insertions(+), 20 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index ccda92920175..db604c0e54d4 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -157,15 +157,15 @@ bool bio_endio_bh(struct bio *bio, struct buffer_head **bhp) } EXPORT_SYMBOL(bio_endio_bh); -/* - * End-of-IO handler helper function which does not touch the bh after - * unlocking it. - * Note: unlock_buffer() sort-of does touch the bh after unlocking it, but - * a race there is benign: unlock_buffer() only use the bh's address for - * hashing after unlocking the buffer, so it doesn't actually touch the bh - * itself. +/** + * end_buffer_read_sync - Handle buffer reads finishing + * @bh: The buffer. + * @uptodate: True if the read was successful. + * + * If a buffer is read through a mechanism that isn't bh_submit(), you + * can call this function to finish the read. */ -static void __end_buffer_read_notouch(struct buffer_head *bh, int uptodate) +void end_buffer_read_sync(struct buffer_head *bh, int uptodate) { if (uptodate) { set_buffer_uptodate(bh); @@ -175,16 +175,6 @@ static void __end_buffer_read_notouch(struct buffer_head *bh, int uptodate) } unlock_buffer(bh); } - -/* - * Default synchronous end-of-IO handler.. Just mark it up-to-date and - * unlock the buffer. - */ -void end_buffer_read_sync(struct buffer_head *bh, int uptodate) -{ - put_bh(bh); - __end_buffer_read_notouch(bh, uptodate); -} EXPORT_SYMBOL(end_buffer_read_sync); /** @@ -198,7 +188,7 @@ void bh_end_read(struct bio *bio) { struct buffer_head *bh; bool uptodate = bio_endio_bh(bio, &bh); - __end_buffer_read_notouch(bh, uptodate); + end_buffer_read_sync(bh, uptodate); } EXPORT_SYMBOL(bh_end_read); diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index 4895bb010cfb..a87cfbf0df38 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -286,10 +286,10 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags, lock_buffer(bh); if (buffer_uptodate(bh)) { unlock_buffer(bh); - brelse(bh); } else { bhs[num++] = bh; } + brelse(bh); } gfs2_submit_bhs(REQ_OP_READ | REQ_META | REQ_PRIO, bhs, num); From b20f15420f786a029be09b1d9b81695581d1a122 Mon Sep 17 00:00:00 2001 From: "Matthew Wilcox (Oracle)" Date: Thu, 28 May 2026 18:31:47 +0100 Subject: [PATCH 62/69] buffer: Remove end_buffer_write_sync() It has no callers left, so delete it. Inline __end_buffer_write_sync() into bh_end_write(). Signed-off-by: Matthew Wilcox (Oracle) Link: https://patch.msgid.link/20260528173150.1093780-35-willy@infradead.org Reviewed-by: Jan Kara Signed-off-by: Christian Brauner (Amutable) --- fs/buffer.c | 29 +++++++++-------------------- include/linux/buffer_head.h | 1 - 2 files changed, 9 insertions(+), 21 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index db604c0e54d4..7ed8dd77d221 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -192,25 +192,6 @@ void bh_end_read(struct bio *bio) } EXPORT_SYMBOL(bh_end_read); -static void __end_buffer_write_sync(struct buffer_head *bh, int uptodate) -{ - if (uptodate) { - set_buffer_uptodate(bh); - } else { - buffer_io_error(bh, ", lost sync page write"); - mark_buffer_write_io_error(bh); - clear_buffer_uptodate(bh); - } - unlock_buffer(bh); -} - -void end_buffer_write_sync(struct buffer_head *bh, int uptodate) -{ - __end_buffer_write_sync(bh, uptodate); - put_bh(bh); -} -EXPORT_SYMBOL(end_buffer_write_sync); - /** * bh_end_write - I/O end handler for writes * @bio: The bio being completed. @@ -222,7 +203,15 @@ void bh_end_write(struct bio *bio) { struct buffer_head *bh; bool success = bio_endio_bh(bio, &bh); - __end_buffer_write_sync(bh, success); + + if (success) { + set_buffer_uptodate(bh); + } else { + buffer_io_error(bh, ", lost sync page write"); + mark_buffer_write_io_error(bh); + clear_buffer_uptodate(bh); + } + unlock_buffer(bh); } EXPORT_SYMBOL(bh_end_write); diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index 1ee56c9f2327..8b23bc9a244c 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -201,7 +201,6 @@ struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size); struct buffer_head *create_empty_buffers(struct folio *folio, unsigned long blocksize, unsigned long b_state); void end_buffer_read_sync(struct buffer_head *bh, int uptodate); -void end_buffer_write_sync(struct buffer_head *bh, int uptodate); bool bio_endio_bh(struct bio *bio, struct buffer_head **bhp); /* Completion routines suitable for passing to bh_submit() */ From a1e9718b406bc4e6d0c63c7b999d06febbdc4091 Mon Sep 17 00:00:00 2001 From: Zhan Wei Date: Fri, 29 May 2026 22:25:33 +0800 Subject: [PATCH 63/69] eventpoll: restore EP_UNACTIVE_PTR sentinel for ctx->tfile_check_list Commit e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") moved tfile_check_list from a file-scope global into the stack-allocated struct ep_ctl_ctx, and in doing so replaced the EP_UNACTIVE_PTR sentinel with NULL on the grounds that "NULL is the obvious 'empty' value and zero-init handles it for free", describing the change as "No functional change". It is not. epitems_head->next is overloaded with two roles: 1. the "next" pointer that threads a head onto ctx->tfile_check_list; 2. a membership flag: ep_remove_file() uses !smp_load_acquire(&v->next) to mean "this head is not on any pending ctx->tfile_check_list and is therefore safe to free". Before that change the EP_UNACTIVE_PTR sentinel kept the two roles disjoint: a head on the list always had a non-NULL ->next (another head, or the sentinel at the tail), so ->next == NULL was equivalent to "never listed". With the sentinel gone the list is NULL-terminated, so the tail head's ->next is NULL as well. ep_remove_file()'s gate can no longer distinguish "never listed" from "listed at the tail", and misfires on the tail head. The reader (reverse_path_check_proc) holds epnested_mutex + rcu_read_lock; the freer (ep_remove_file) holds ep->mtx + file->f_lock. The two sides share no mutex -- the sentinel was the invariant the gate relied on to know it could skip the read side. With it gone, ep_remove_file() frees the tail head while reverse_path_check_proc() is still walking it, producing the slab-use-after-free read. The syzbot reproducer hits this within seconds on a multi-CPU VM. Restore the sentinel: initialize ctx.tfile_check_list to EP_UNACTIVE_PTR in do_epoll_ctl_file(), and terminate the walk on "!= EP_UNACTIVE_PTR" in reverse_path_check() and clear_tfile_check_list(). The tail head's ->next becomes the sentinel again rather than NULL, so ep_remove_file()'s gate regains its exclusivity and stops misfiring on the tail. ep_remove_file() itself is unchanged. This restores the invariant the file-scope tfile_check_list relied on before that change while preserving the ctx packaging it introduced. Reported-by: syzbot+e70e1b6cba8714543f7c@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=e70e1b6cba8714543f7c Fixes: e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") Suggested-by: Christian Brauner Link: https://lore.kernel.org/all/20260528-rotwild-summt-kuhhandel-7276ef4c33b7@brauner.io/ Signed-off-by: Zhan Wei Link: https://patch.msgid.link/20260529142533.23696-1-zhanwei919@gmail.com Signed-off-by: Christian Brauner (Amutable) --- fs/eventpoll.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index df364a8783b5..0e65c7431dfc 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -433,7 +433,11 @@ struct ep_ctl_ctx { /* * Singly-linked list of epitems_head objects collected during * ep_loop_check_proc(), then walked by reverse_path_check(). - * NULL means empty. + * Terminated by EP_UNACTIVE_PTR, not NULL: epitems_head->next + * doubles as a membership flag (a NULL ->next means "not on this + * list", see ep_remove_file()), so the list uses a non-NULL + * sentinel to keep the tail head distinguishable from an unlisted + * one. */ struct epitems_head *tfile_check_list; @@ -1704,7 +1708,7 @@ static int reverse_path_check(struct ep_ctl_ctx *ctx) { struct epitems_head *p; - for (p = ctx->tfile_check_list; p; p = p->next) { + for (p = ctx->tfile_check_list; p != EP_UNACTIVE_PTR; p = p->next) { int error; path_count_init(ctx); rcu_read_lock(); @@ -2458,7 +2462,7 @@ static int ep_loop_check(struct ep_ctl_ctx *ctx, struct eventpoll *ep, static void clear_tfile_check_list(struct ep_ctl_ctx *ctx) { rcu_read_lock(); - while (ctx->tfile_check_list) { + while (ctx->tfile_check_list != EP_UNACTIVE_PTR) { struct epitems_head *head = ctx->tfile_check_list; ctx->tfile_check_list = head->next; unlist_file(head); @@ -2621,7 +2625,9 @@ int do_epoll_ctl_file(struct file *f, int op, struct epoll_key *tf, int full_check; struct eventpoll *ep; struct epitem *epi; - struct ep_ctl_ctx ctx = { }; + struct ep_ctl_ctx ctx = { + .tfile_check_list = EP_UNACTIVE_PTR, + }; /* The target file descriptor must support poll */ if (!file_can_poll(tf->file)) From 198c3b90e600d442f8333d254a23c5da989c5cb4 Mon Sep 17 00:00:00 2001 From: Namjae Jeon Date: Mon, 18 May 2026 20:46:55 +0900 Subject: [PATCH 64/69] iomap: introduce IOMAP_F_ZERO_TAIL flag In filesystems that maintain a separate Valid Data Length, such as exFAT and NTFS, a partial write may start at or beyond the current valid_size and extend it. In this case, the region after the previous valid_size but within the same filesystem block is considered unwritten. This patch introduces IOMAP_F_ZERO_TAIL. When this flag is set in iomap, __iomap_write_begin() will zero only the tail portion while preserving any valid data before it in the same block. Without this tail zeroing, stale data in the unwritten portion of the block can remain in the page cache. Subsequent reads can then return incorrect contents from that region. Acked-by: Christoph Hellwig Signed-off-by: Namjae Jeon Link: https://patch.msgid.link/20260518114705.9601-2-linkinjeon@kernel.org Acked-by: "Darrick J. Wong" Reviewed-by: "Darrick J. Wong" Signed-off-by: Christian Brauner (Amutable) --- fs/iomap/buffered-io.c | 4 ++++ include/linux/iomap.h | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index c98cf19fbe08..d6451c4208d4 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -836,6 +836,7 @@ static int __iomap_write_begin(const struct iomap_iter *iter, return -EIO; folio_zero_segments(folio, poff, from, to, poff + plen); } else { + const struct iomap *iomap = iomap_iter_srcmap(iter); int status; if (iter->flags & IOMAP_NOWAIT) @@ -853,6 +854,9 @@ static int __iomap_write_begin(const struct iomap_iter *iter, len, status, GFP_NOFS); if (status) return status; + + if (iomap->flags & IOMAP_F_ZERO_TAIL) + folio_zero_segment(folio, to, poff + plen); } iomap_set_range_uptodate(folio, poff, plen); } while ((block_start += plen) < block_end); diff --git a/include/linux/iomap.h b/include/linux/iomap.h index 52bfd6783417..cea6bbc97b6e 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -67,6 +67,9 @@ struct vm_fault; * bio, i.e. set REQ_ATOMIC. * * IOMAP_F_INTEGRITY indicates that the filesystems handles integrity metadata. + * + * IOMAP_F_ZERO_TAIL indicates the remainder of the block after the data + * written should be zeroed. */ #define IOMAP_F_NEW (1U << 0) #define IOMAP_F_DIRTY (1U << 1) @@ -86,6 +89,7 @@ struct vm_fault; #else #define IOMAP_F_INTEGRITY 0 #endif /* CONFIG_BLK_DEV_INTEGRITY */ +#define IOMAP_F_ZERO_TAIL (1U << 10) /* * Flag reserved for file system specific usage From 07d09774e2bfa21dedcee3ef45892bb20827b12c Mon Sep 17 00:00:00 2001 From: Andrey Albershteyn Date: Wed, 20 May 2026 14:37:02 +0200 Subject: [PATCH 65/69] fsverity: generate and store zero-block hash Compute the hash of one filesystem block's worth of zeros. A filesystem implementation can decide to elide merkle tree blocks containing only this hash and synthesize the contents at read time. Let's pretend that there's a file containing 131 data block and whose merkle tree looks roughly like this: root +--leaf0 | +--data0 | +--data1 | +--... | `--data128 `--leaf1 +--data129 +--data130 `--data131 If data[0-128] are sparse holes, then leaf0 will contain a repeating sequence of @zero_digest. Therefore, leaf0 need not be written to disk because its contents can be synthesized. A subsequent xfs patch will use this to reduce the size of the merkle tree when dealing with sparse gold master disk images and the like. Note that this works only on the first-level (data holes). fsverity doesn't store/generate zero_digest for any higher levels. Add a helper to pre-fill folio with hashes of empty blocks. This will be used by iomap to synthesize blocks full of zero hashes on the fly. Signed-off-by: Darrick J. Wong Acked-by: Eric Biggers Signed-off-by: Andrey Albershteyn Link: https://patch.msgid.link/20260520123722.405752-5-aalbersh@kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/verity/fsverity_private.h | 3 +++ fs/verity/measure.c | 4 ++-- fs/verity/open.c | 3 +++ fs/verity/pagecache.c | 22 ++++++++++++++++++++++ include/linux/fsverity.h | 8 ++++++++ 5 files changed, 38 insertions(+), 2 deletions(-) diff --git a/fs/verity/fsverity_private.h b/fs/verity/fsverity_private.h index 6e6854c19078..881d46f25e08 100644 --- a/fs/verity/fsverity_private.h +++ b/fs/verity/fsverity_private.h @@ -53,6 +53,9 @@ struct merkle_tree_params { u64 tree_size; /* Merkle tree size in bytes */ unsigned long tree_pages; /* Merkle tree size in pages */ + /* the hash of an all-zeroes block */ + u8 zero_digest[FS_VERITY_MAX_DIGEST_SIZE]; + /* * Starting block index for each tree level, ordered from leaf level (0) * to root level ('num_levels - 1') diff --git a/fs/verity/measure.c b/fs/verity/measure.c index 6a35623ebdf0..818083507885 100644 --- a/fs/verity/measure.c +++ b/fs/verity/measure.c @@ -68,8 +68,8 @@ EXPORT_SYMBOL_GPL(fsverity_ioctl_measure); * @alg: (out) the digest's algorithm, as a FS_VERITY_HASH_ALG_* value * @halg: (out) the digest's algorithm, as a HASH_ALGO_* value * - * Retrieves the fsverity digest of the given file. The file must have been - * opened at least once since the inode was last loaded into the inode cache; + * Retrieves the fsverity digest of the given file. The + * fsverity_ensure_verity_info() must be called on the inode beforehand; * otherwise this function will not recognize when fsverity is enabled. * * The file's fsverity digest consists of @raw_digest in combination with either diff --git a/fs/verity/open.c b/fs/verity/open.c index dfa0d1afe0fe..d0c56a7faa3b 100644 --- a/fs/verity/open.c +++ b/fs/verity/open.c @@ -153,6 +153,9 @@ int fsverity_init_merkle_tree_params(struct merkle_tree_params *params, goto out_err; } + fsverity_hash_block(params, page_address(ZERO_PAGE(0)), + params->zero_digest); + params->tree_size = offset << log_blocksize; params->tree_pages = PAGE_ALIGN(params->tree_size) >> PAGE_SHIFT; return 0; diff --git a/fs/verity/pagecache.c b/fs/verity/pagecache.c index 1819314ecaa3..99f5f53eea98 100644 --- a/fs/verity/pagecache.c +++ b/fs/verity/pagecache.c @@ -2,6 +2,7 @@ /* * Copyright 2019 Google LLC */ +#include "fsverity_private.h" #include #include @@ -56,3 +57,24 @@ void generic_readahead_merkle_tree(struct inode *inode, pgoff_t index, folio_put(folio); } EXPORT_SYMBOL_GPL(generic_readahead_merkle_tree); + +/** + * fsverity_fill_zerohash() - fill folio with hashes of zero data block + * @folio: folio to fill + * @offset: offset in the folio to start + * @len: length of the range to fill with hashes + * @vi: fsverity info + */ +void fsverity_fill_zerohash(struct folio *folio, size_t offset, size_t len, + struct fsverity_info *vi) +{ + size_t off = offset; + + WARN_ON_ONCE(!IS_ALIGNED(offset, vi->tree_params.digest_size)); + WARN_ON_ONCE(!IS_ALIGNED(len, vi->tree_params.digest_size)); + + for (; off < (offset + len); off += vi->tree_params.digest_size) + memcpy_to_folio(folio, off, vi->tree_params.zero_digest, + vi->tree_params.digest_size); +} +EXPORT_SYMBOL_GPL(fsverity_fill_zerohash); diff --git a/include/linux/fsverity.h b/include/linux/fsverity.h index a8f9aa75b792..6c467ded9751 100644 --- a/include/linux/fsverity.h +++ b/include/linux/fsverity.h @@ -201,6 +201,8 @@ bool fsverity_verify_blocks(struct fsverity_info *vi, struct folio *folio, size_t len, size_t offset); void fsverity_verify_bio(struct fsverity_info *vi, struct bio *bio); void fsverity_enqueue_verify_work(struct work_struct *work); +void fsverity_fill_zerohash(struct folio *folio, size_t offset, size_t len, + struct fsverity_info *vi); #else /* !CONFIG_FS_VERITY */ @@ -281,6 +283,12 @@ static inline void fsverity_enqueue_verify_work(struct work_struct *work) WARN_ON_ONCE(1); } +static inline void fsverity_fill_zerohash(struct folio *folio, size_t offset, + size_t len, struct fsverity_info *vi) +{ + WARN_ON_ONCE(1); +} + #endif /* !CONFIG_FS_VERITY */ static inline bool fsverity_verify_folio(struct fsverity_info *vi, From 63e242afa4661632786a2129cf297a135a6995ba Mon Sep 17 00:00:00 2001 From: Andrey Albershteyn Date: Wed, 20 May 2026 14:37:05 +0200 Subject: [PATCH 66/69] iomap: introduce IOMAP_F_FSVERITY and teach writeback to handle fsverity This flag indicates that I/O is for fsverity metadata. In the write path skip i_size check and i_size updates as metadata is past EOF. In writeback don't update i_size and continue writeback if even folio is beyond EOF. In read path don't zero fsverity folios, again they are past EOF. The iomap_block_needs_zeroing() is also called from write path. For folios of larger order we don't want to zero out pages in the folio as these could contain other merkle tree blocks. For fsverity, filesystem will request to read PAGE_SIZE memory regions. For data folios, iomap will zero the rest of the folio for anything which is beyond EOF. We don't want this for fsverity folios. Christian Brauner says: Changed IOMAP_F_FSVERITY from (1U << 10) to (1U << 11) to avoid colliding with IOMAP_F_ZERO_TAIL, which already uses (1U << 10). Signed-off-by: Andrey Albershteyn Link: https://patch.msgid.link/20260520123722.405752-8-aalbersh@kernel.org Reviewed-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig Signed-off-by: Christian Brauner (Amutable) --- fs/iomap/buffered-io.c | 43 +++++++++++++++++++++++++++++++++--------- fs/iomap/trace.h | 3 ++- include/linux/iomap.h | 8 ++++++++ 3 files changed, 44 insertions(+), 10 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index d6451c4208d4..d9462f28afbd 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -353,9 +353,26 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter, { const struct iomap *srcmap = iomap_iter_srcmap(iter); - return srcmap->type != IOMAP_MAPPED || - (srcmap->flags & IOMAP_F_NEW) || - pos >= i_size_read(iter->inode); + /* + * If this block has not been written, there's nothing to read + */ + if (srcmap->type != IOMAP_MAPPED) + return true; + + /* + * Newly allocated blocks have not been written + */ + if (srcmap->flags & IOMAP_F_NEW) + return true; + + /* + * fsverity metadata is stored past i_size, we need to read it instead + * of zeroing + */ + if (srcmap->flags & IOMAP_F_FSVERITY) + return false; + + return pos >= i_size_read(iter->inode); } /** @@ -1170,13 +1187,14 @@ retry: * unlock and release the folio. */ old_size = iter->inode->i_size; - if (pos + written > old_size) { + if (pos + written > old_size && + !(iter->iomap.flags & IOMAP_F_FSVERITY)) { i_size_write(iter->inode, pos + written); iter->iomap.flags |= IOMAP_F_SIZE_CHANGED; } __iomap_put_folio(iter, write_ops, written, folio); - if (old_size < pos) + if (old_size < pos && !(iter->iomap.flags & IOMAP_F_FSVERITY)) pagecache_isize_extended(iter->inode, old_size, pos); cond_resched(); @@ -1802,13 +1820,20 @@ static int iomap_writeback_range(struct iomap_writepage_ctx *wpc, * Check interaction of the folio with the file end. * * If the folio is entirely beyond i_size, return false. If it straddles - * i_size, adjust end_pos and zero all data beyond i_size. + * i_size, adjust end_pos and zero all data beyond i_size. Don't skip fsverity + * folios as those are beyond i_size. */ -static bool iomap_writeback_handle_eof(struct folio *folio, struct inode *inode, - u64 *end_pos) +static bool iomap_writeback_handle_eof(struct folio *folio, + struct iomap_writepage_ctx *wpc, u64 *end_pos) { + struct inode *inode = wpc->inode; u64 isize = i_size_read(inode); + if (wpc->iomap.flags & IOMAP_F_FSVERITY) { + WARN_ON_ONCE(folio_pos(folio) < isize); + return true; + } + if (*end_pos > isize) { size_t poff = offset_in_folio(folio, isize); pgoff_t end_index = isize >> PAGE_SHIFT; @@ -1874,7 +1899,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio) trace_iomap_writeback_folio(inode, pos, folio_size(folio)); - if (!iomap_writeback_handle_eof(folio, inode, &end_pos)) + if (!iomap_writeback_handle_eof(folio, wpc, &end_pos)) return 0; WARN_ON_ONCE(end_pos <= pos); diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h index 097773c6db80..e4dd25b27656 100644 --- a/fs/iomap/trace.h +++ b/fs/iomap/trace.h @@ -118,7 +118,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter); { IOMAP_F_ATOMIC_BIO, "ATOMIC_BIO" }, \ { IOMAP_F_PRIVATE, "PRIVATE" }, \ { IOMAP_F_SIZE_CHANGED, "SIZE_CHANGED" }, \ - { IOMAP_F_STALE, "STALE" } + { IOMAP_F_STALE, "STALE" }, \ + { IOMAP_F_FSVERITY, "FSVERITY" } #define IOMAP_DIO_STRINGS \ diff --git a/include/linux/iomap.h b/include/linux/iomap.h index cea6bbc97b6e..5b2b40e81b61 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -91,6 +91,14 @@ struct vm_fault; #endif /* CONFIG_BLK_DEV_INTEGRITY */ #define IOMAP_F_ZERO_TAIL (1U << 10) +/* + * Indicates reads and writes of fsverity metadata. + * + * Fsverity metadata is stored after the regular file data and thus beyond + * i_size. + */ +#define IOMAP_F_FSVERITY (1U << 11) + /* * Flag reserved for file system specific usage */ From 1d140731753a277be36637300a0f3faa396edec1 Mon Sep 17 00:00:00 2001 From: Andrey Albershteyn Date: Wed, 20 May 2026 14:37:06 +0200 Subject: [PATCH 67/69] iomap: teach iomap to read files with fsverity Obtain fsverity info for folios with file data and fsverity metadata. Filesystem can pass vi down to ioend and then to fsverity for verification. This is different from other filesystems ext4, f2fs, btrfs supporting fsverity, these filesystems don't need fsverity_info for reading fsverity metadata. While reading merkle tree iomap requires fsverity info to synthesize hashes for zeroed data block. fsverity metadata has two kinds of holes - ones in merkle tree and one after fsverity descriptor. Merkle tree holes are blocks full of hashes of zeroed data blocks. These are not stored on the disk but synthesized on the fly. This saves a bit of space for sparse files. Due to this iomap also need to lookup fsverity_info for folios with fsverity metadata. ->vi has a hash of the zeroed data block which will be used to fill the merkle tree block. The hole past descriptor is interpreted as end of metadata region. As we don't have EOF here we use this hole as an indication that rest of the folio is empty. This patch marks rest of the folio beyond fsverity descriptor as uptodate. For file data, fsverity needs to verify consistency of the whole file against the root hash, hashes of holes are included in the merkle tree. Verify them too. Issue reading of fsverity merkle tree on the fsverity inodes. This way metadata will be available at I/O completion time. Reviewed-by: Darrick J. Wong Reviewed-by: Christoph Hellwig Signed-off-by: Andrey Albershteyn Link: https://patch.msgid.link/20260520123722.405752-9-aalbersh@kernel.org Signed-off-by: Christian Brauner (Amutable) --- fs/iomap/buffered-io.c | 41 +++++++++++++++++++++++++++++++++++++++-- fs/iomap/ioend.c | 1 + include/linux/iomap.h | 2 ++ 3 files changed, 42 insertions(+), 2 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index d9462f28afbd..f9cc5a32e222 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "internal.h" #include "trace.h" @@ -561,9 +562,27 @@ static int iomap_read_folio_iter(struct iomap_iter *iter, if (plen == 0) return 0; - /* zero post-eof blocks as the page may be mapped */ - if (iomap_block_needs_zeroing(iter, pos)) { + /* + * Handling of fsverity "holes". We hit this for two case: + * 1. No need to go further, the hole after fsverity + * descriptor is the end of the fsverity metadata. + * + * 2. This folio contains merkle tree blocks which need to be + * synthesized. If we already have fsverity info (ctx->vi) + * synthesize these blocks. + */ + if ((iomap->flags & IOMAP_F_FSVERITY) && + iomap->type == IOMAP_HOLE) { + if (ctx->vi) + fsverity_fill_zerohash(folio, poff, plen, + ctx->vi); + iomap_set_range_uptodate(folio, poff, plen); + } else if (iomap_block_needs_zeroing(iter, pos)) { + /* zero post-eof blocks as the page may be mapped */ folio_zero_range(folio, poff, plen); + if (ctx->vi && + !fsverity_verify_blocks(ctx->vi, folio, plen, poff)) + return -EIO; iomap_set_range_uptodate(folio, poff, plen); } else { if (!*bytes_submitted) @@ -614,6 +633,15 @@ void iomap_read_folio(const struct iomap_ops *ops, trace_iomap_readpage(iter.inode, 1); + /* + * Fetch fsverity_info for both data and fsverity metadata, as iomap + * needs zeroed hash for merkle tree block synthesis + */ + ctx->vi = fsverity_get_info(iter.inode); + if (ctx->vi && iter.pos < i_size_read(iter.inode)) + fsverity_readahead(ctx->vi, folio->index, + folio_nr_pages(folio)); + while ((ret = iomap_iter(&iter, ops)) > 0) iter.status = iomap_read_folio_iter(&iter, ctx, &bytes_submitted); @@ -681,6 +709,15 @@ void iomap_readahead(const struct iomap_ops *ops, trace_iomap_readahead(rac->mapping->host, readahead_count(rac)); + /* + * Fetch fsverity_info for both data and fsverity metadata, as iomap + * needs zeroed hash for merkle tree block synthesis + */ + ctx->vi = fsverity_get_info(iter.inode); + if (ctx->vi && iter.pos < i_size_read(iter.inode)) + fsverity_readahead(ctx->vi, readahead_index(rac), + readahead_count(rac)); + while (iomap_iter(&iter, ops) > 0) iter.status = iomap_readahead_iter(&iter, ctx, &cur_bytes_submitted); diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c index acf3cf98b23a..f7c3e0c70fd7 100644 --- a/fs/iomap/ioend.c +++ b/fs/iomap/ioend.c @@ -28,6 +28,7 @@ struct iomap_ioend *iomap_init_ioend(struct inode *inode, ioend->io_offset = file_offset; ioend->io_size = bio->bi_iter.bi_size; ioend->io_sector = bio->bi_iter.bi_sector; + ioend->io_vi = NULL; ioend->io_private = NULL; return ioend; } diff --git a/include/linux/iomap.h b/include/linux/iomap.h index 5b2b40e81b61..1be2e16b696a 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -429,6 +429,7 @@ struct iomap_ioend { loff_t io_offset; /* offset in the file */ sector_t io_sector; /* start sector of ioend */ void *io_private; /* file system private data */ + struct fsverity_info *io_vi; /* fsverity info */ struct bio io_bio; /* MUST BE LAST! */ }; @@ -503,6 +504,7 @@ struct iomap_read_folio_ctx { struct readahead_control *rac; void *read_ctx; loff_t read_ctx_file_offset; + struct fsverity_info *vi; }; struct iomap_read_ops { From 36a36c4cac914510123071fb58270f6380faed1b Mon Sep 17 00:00:00 2001 From: Andrey Albershteyn Date: Wed, 20 May 2026 14:37:07 +0200 Subject: [PATCH 68/69] iomap: introduce iomap_fsverity_write() for writing fsverity metadata This is just a wrapper around iomap_file_buffered_write() to create necessary iterator over metadata. Reviewed-by: Christoph Hellwig Reviewed-by: Darrick J. Wong Signed-off-by: Andrey Albershteyn Link: https://patch.msgid.link/20260520123722.405752-10-aalbersh@kernel.org Reviewed-by: "Darrick J. Wong" Signed-off-by: Christian Brauner (Amutable) --- fs/iomap/buffered-io.c | 25 +++++++++++++++++++++++++ include/linux/iomap.h | 3 +++ 2 files changed, 28 insertions(+) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index f9cc5a32e222..dab4aacff313 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -1290,6 +1290,31 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i, } EXPORT_SYMBOL_GPL(iomap_file_buffered_write); +int iomap_fsverity_write(struct file *file, loff_t pos, size_t length, + const void *buf, const struct iomap_ops *ops, + const struct iomap_write_ops *write_ops) +{ + int ret; + struct iov_iter iiter; + struct kvec kvec = { + .iov_base = (void *)buf, + .iov_len = length, + }; + struct kiocb iocb = { + .ki_filp = file, + .ki_ioprio = get_current_ioprio(), + .ki_pos = pos, + }; + + iov_iter_kvec(&iiter, WRITE, &kvec, 1, length); + + ret = iomap_file_buffered_write(&iocb, &iiter, ops, write_ops, NULL); + if (ret < 0) + return ret; + return ret == length ? 0 : -EIO; +} +EXPORT_SYMBOL_GPL(iomap_fsverity_write); + static void iomap_write_delalloc_ifs_punch(struct inode *inode, struct folio *folio, loff_t start_byte, loff_t end_byte, struct iomap *iomap, iomap_punch_t punch) diff --git a/include/linux/iomap.h b/include/linux/iomap.h index 1be2e16b696a..3582ed1fe236 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -353,6 +353,9 @@ static inline bool iomap_want_unshare_iter(const struct iomap_iter *iter) ssize_t iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *from, const struct iomap_ops *ops, const struct iomap_write_ops *write_ops, void *private); +int iomap_fsverity_write(struct file *file, loff_t pos, size_t length, + const void *buf, const struct iomap_ops *ops, + const struct iomap_write_ops *write_ops); void iomap_read_folio(const struct iomap_ops *ops, struct iomap_read_folio_ctx *ctx, void *private); void iomap_readahead(const struct iomap_ops *ops, From b7bae6880e8de2a5f693c18d87ad5cc26f157eb2 Mon Sep 17 00:00:00 2001 From: Namjae Jeon Date: Wed, 3 Jun 2026 23:40:31 +0900 Subject: [PATCH 69/69] iomap: Add IOMAP_F_ZERO_TAIL flag to trace event strings Add IOMAP_F_ZERO_TAIL to the flag string mapping in iomap trace events. This allows the new flag to be properly displayed in ftrace output when iomap operations use it. Signed-off-by: Namjae Jeon Link: https://patch.msgid.link/20260603144031.7370-1-linkinjeon@kernel.org Reviewed-by: "Darrick J. Wong" Signed-off-by: Christian Brauner (Amutable) --- fs/iomap/trace.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h index e4dd25b27656..e1ea8392cf47 100644 --- a/fs/iomap/trace.h +++ b/fs/iomap/trace.h @@ -119,7 +119,8 @@ DEFINE_RANGE_EVENT(iomap_zero_iter); { IOMAP_F_PRIVATE, "PRIVATE" }, \ { IOMAP_F_SIZE_CHANGED, "SIZE_CHANGED" }, \ { IOMAP_F_STALE, "STALE" }, \ - { IOMAP_F_FSVERITY, "FSVERITY" } + { IOMAP_F_FSVERITY, "FSVERITY" }, \ + { IOMAP_F_ZERO_TAIL, "ZERO TAIL" } #define IOMAP_DIO_STRINGS \