diff options
| author | Eric Dumazet <edumazet@google.com> | 2025-11-16 20:27:17 +0000 |
|---|---|---|
| committer | Jakub Kicinski <kuba@kernel.org> | 2025-11-18 18:25:47 -0800 |
| commit | 21664814b89e1268bc48e9f641b813746a7dbaae (patch) | |
| tree | dc0c9b7fd978032be974648c29329e2f3b865d5c /net/core | |
| parent | 294e63825966319e815e469931cfbfc437004d0f (diff) | |
net: use napi_skb_cache even in process context
This is a followup of commit e20dfbad8aab ("net: fix napi_consume_skb()
with alien skbs").
Now the per-cpu napi_skb_cache is populated from TX completion path,
we can make use of this cache, especially for cpus not used
from a driver NAPI poll (primary user of napi_cache).
We can use the napi_skb_cache only if current context is not from hard irq.
With this patch, I consistently reach 130 Mpps on my UDP tx stress test
and reduce SLUB spinlock contention to smaller values.
Note there is still some SLUB contention for skb->head allocations.
I had to tune /sys/kernel/slab/skbuff_small_head/cpu_partial
and /sys/kernel/slab/skbuff_small_head/min_partial depending
on the platform taxonomy.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Jason Xing <kerneljasonxing@gmail.com>
Tested-by: Jason Xing <kerneljasonxing@gmail.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20251116202717.1542829-4-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Diffstat (limited to 'net/core')
| -rw-r--r-- | net/core/skbuff.c | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 8a0a4ca7fa5d..9feea830a4db 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -666,7 +666,12 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, skb = napi_skb_cache_get(true); if (unlikely(!skb)) return NULL; + } else if (!in_hardirq() && !irqs_disabled()) { + local_bh_disable(); + skb = napi_skb_cache_get(false); + local_bh_enable(); } + if (!skb) { fallback: skb = kmem_cache_alloc_node(cache, gfp_mask & ~GFP_DMA, node); |