Ixgbe xdp - #7
Closed
Magda-Pytel wants to merge 20 commits into
Closed
Conversation
Similarly as in commit 5384467 ("iavf: kill "legacy-rx" for good"), drop skb construction logic in favor of only using napi_build_skb() as a superior option that reduces the need to allocate and copy memory. As IXGBEVF_PRIV_FLAGS_LEGACY_RX is the only private flag in ixgbevf, entirely remove private flags support from the driver. When compared to iavf changes, ixgbevf has a single complication: MAC type 82599 cannot finely limit the DMA write size with RXDCTL.RLPML, only 1024 increments through SRRCTL are available, see commit fe68195 ("ixgbevf: Require large buffers for build_skb on 82599VF") and commit 2bafa8f ("ixgbe: don't set RXDCTL.RLPML for 82599"). Therefore, this is a special case requiring legacy RX unless large buffers are used. For now, solve this by always using large buffers for this MAC type. Suggested-by: Alexander Lobakin <aleksander.lobakin@intel.com> Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com> Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Again, same as in the related iavf commit 920d86f ("iavf: drop page splitting and recycling"), as an intermediate step, drop the page sharing and recycling logic in a preparation to offload it to page_pool. Instead of the previous sharing and recycling, just allocate a new page every time. Suggested-by: Alexander Lobakin <aleksander.lobakin@intel.com> Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com> Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Use page_pool buffers by the means of libeth in the Rx queues, this significantly reduces code complexity of the driver itself. Suggested-by: Alexander Lobakin <aleksander.lobakin@intel.com> Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com> Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Implement XDP support for received fragmented packets, this requires using some helpers from libeth_xdp. Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Use libeth to support XDP_TX action for segmented packets. Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
To fully support XDP_REDIRECT, utilize more libeth helpers in XDP Rx path, hence save cached_ntu in the ring structure instead of stack. ixgbevf-supported VFs usually have few queues, so use libeth_xdpsq_lock functionality for XDP queue sharing. Adjust filling-in of XDP Tx descriptors to use data from xdp frame. Otherwise, simply use libeth helpers to implement .ndo_xdp_xmit(). While at it, fix a typo in libeth docs. Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Introduce pseudo header split support in the ixgbevf driver, specifically targeting ixgbe_mac_82599_vf. On older hardware (e.g. ixgbe_mac_82599_vf), RX DMA write size can only be limited in 1K increments. This causes issues when attempting to fit multiple packets per page, as a DMA write may overwrite the headroom of the next packet. To address this, introduce pseudo header split support, where the hardware copies the full L2 header into a dedicated header buffer. This avoids the need for HR/TR alignment and allows safe skb construction from the header buffer without risking overwrites. Given that once packet is too big to fit into a single page, the behaviour is the same for all supported HW, use pseudo header split only for smaller packets. Signed-off-by: Natalia Wochtman <natalia.wochtman@intel.com> Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com> Co-developed-by: Larysa Zaremba <larysa.zaremba@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Currently, when MTU is changed, page pool is not reconfigured, which leads to usage of suboptimal buffer sizes. Always destroy page pool when cleaning the ring up and create it anew when we first allocate Rx buffers. Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
xskxceiver attempts to change MTU after attaching XDP program, ixgbevf rejects the request leading to test being failed. Support MTU change operation even when XDP program is already attached, perform the same frame size check as when attaching a program. Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com> Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
AF_XDP ZC Rx path is also required to implement skb creation. Move all common functions to a header file as inlines. Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Implement xsk_buff_pool configuration and supporting functionality, such as a single queue pair reconfiguration. Also, properly initialize Rx buffers. Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Add code that handles Tx ZC queues inside of napi_poll(), utilize libeth. As NIC's multiple buffer conventions do not play nicely with AF_XDP's, leave handling of segments for later. Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Add code that handles AF_XDP ZC Rx queues inside of napi_poll(), utilize libeth helpers. Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
To finalize basic AF_XDP implementation, set features and add .ndo_xsk_wakeup() handler. Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Transmitting multi-buffer AF_XDP packets is not very straightforward given HW limitations in ixgbevf, namely that the first data descriptor must contain the length of the whole packet. Use private data of an sqe to store the length of an unfinished packet so far and the first descriptor index. Once EoP zero-copy descriptor is processed, write the accumulated length into the saved first descriptor. Signed-off-by: Larysa Zaremba <larysa.zaremba@intel.com>
Similarly, like for virtual function in ixgbevf, remove ixgbe_construct_skb(), the legacy-rx private flag, and all ring_uses_build_skb() conditionals. build_skb is now the only RX code path. This is a preparation for the conversion to libeth and page_pool based Rx buffer management. Signed-off-by: Magdalena Pytel <magdalena.pytel@intel.com>
Stop sharing pages between multiple Rx buffers. Convert ixgbe Rx path from page-sharing/recycling to a simple one-page-per-packet model. Remove logic that refers to page-sharing buffer management. This is a preparatory step before converting the Rx path to page_pool and libeth. Fixed XDP_DROP/ABORTED: reaches ixgbe_put_rx_buffer() but page is never freed. Signed-off-by: Magdalena Pytel <magdalena.pytel@intel.com>
Convert ixgbe Rx path from manual page-per-packet DMA management to libeth/page_pool infrastructure, and integrates libeth XDP helpers. Each packet gets its own buffer instead of sharing pages between packets. Signed-off-by: Magdalena Pytel <magdalena.pytel@intel.com>
* Added branch prediction hints to the ixgbe Rx path and clean up the packet accounting. * Moved total_rx_bytes and total_rx_packets accounting together. Signed-off-by: Magdalena Pytel <magdalena.pytel@intel.com>
Implement XDP multi-buffer support for Rx fragmented packets
using libeth_xdp.
* Added MODULE_IMPORT_NS("LIBETH_XDP")
and #include <net/libeth/types.h>
* Replaced struct sk_buff *skb in ixgbe_ring with
struct libeth_xdp_buff_stash
and ixgbe_clean_rx_ring skb cleanup with libeth_xdp_return_stash
* Removed ixgbe_add_rx_frag()
* Reorganized ixgbe_clean_rx_irq() and ixgbe_is_non_eop()
* Updated ixgbe_xdp_setup() to allow multi-buffer XDP programs
* Added NETDEV_XDP_ACT_RX_SG in xdp_features
Signed-off-by: Magdalena Pytel <magdalena.pytel@intel.com>walking-machine pushed a commit
that referenced
this pull request
Jul 29, 2026
Fix a race-condition use-after-free in tunnel_key_release_params(). The function releases the metadata_dst of the old params synchronously via dst_release() while deferring the params struct free with kfree_rcu(). A concurrent tunnel_key_act() reader on the datapath may still hold the old params pointer (under rcu_read_lock_bh) and proceed to call dst_clone(¶ms->tcft_enc_metadata->dst) after the writer's dst_release has already pushed the dst's rcuref to RCUREF_DEAD. zdi-disclosures@trendmicro.com produced a poc which i (and Victor) verified that KASAN reports: ================================================================== BUG: KASAN: slab-use-after-free in instrument_atomic_read_write include/linux/instrumented.h:112 BUG: KASAN: slab-use-after-free in atomic_sub_return_release include/linux/atomic/atomic-instrumented.h:326 BUG: KASAN: slab-use-after-free in __rcuref_put include/linux/rcuref.h:109 BUG: KASAN: slab-use-after-free in rcuref_put include/linux/rcuref.h:173 BUG: KASAN: slab-use-after-free in dst_release+0x5b/0x370 net/core/dst.c:168 Write of size 4 at addr ffff88806158de40 by task poc/9388 CPU: 0 UID: 0 PID: 9388 Comm: poc Tainted: G W 7.1.0-rc7 #7 PREEMPT(lazy) Tainted: [W]=WARN Hardware name: QEMU Ubuntu 25.10 PC v2 (i440FX + PIIX, + 10.1 machine, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 Call Trace: <TASK> __dump_stack lib/dump_stack.c:94 dump_stack_lvl+0x100/0x190 lib/dump_stack.c:120 print_address_description mm/kasan/report.c:378 print_report+0x139/0x4ad mm/kasan/report.c:482 kasan_report+0xe4/0x1d0 mm/kasan/report.c:595 check_region_inline mm/kasan/generic.c:186 kasan_check_range+0x125/0x200 mm/kasan/generic.c:200 instrument_atomic_read_write include/linux/instrumented.h:112 atomic_sub_return_release include/linux/atomic/atomic-instrumented.h:326 __rcuref_put include/linux/rcuref.h:109 rcuref_put include/linux/rcuref.h:173 dst_release+0x5b/0x370 net/core/dst.c:168 refdst_drop include/net/dst.h:272 skb_dst_drop include/net/dst.h:284 skb_release_head_state+0x293/0x400 net/core/skbuff.c:1163 skb_release_all net/core/skbuff.c:1187 [..] Allocated by task 9391: kasan_save_stack+0x30/0x50 mm/kasan/common.c:57 kasan_save_track+0x14/0x30 mm/kasan/common.c:78 poison_kmalloc_redzone mm/kasan/common.c:398 __kasan_kmalloc+0x9a/0xb0 mm/kasan/common.c:415 kasan_kmalloc include/linux/kasan.h:263 __do_kmalloc_node mm/slub.c:5296 __kmalloc_noprof+0x2f1/0x830 mm/slub.c:5308 kmalloc_noprof include/linux/slab.h:954 kzalloc_noprof include/linux/slab.h:1188 offload_action_alloc+0x2f/0x130 net/core/flow_offload.c:35 tcf_action_offload_add_ex+0x1ba/0x880 net/sched/act_api.c:258 tcf_action_offload_add net/sched/act_api.c:293 tcf_action_init+0x66e/0xa20 net/sched/act_api.c:1547 tcf_action_add+0xf6/0x5d0 net/sched/act_api.c:2101 [..] Freed by task 9391: kasan_save_stack+0x30/0x50 mm/kasan/common.c:57 kasan_save_track+0x14/0x30 mm/kasan/common.c:78 kasan_save_free_info+0x3b/0x70 mm/kasan/generic.c:584 poison_slab_object mm/kasan/common.c:253 __kasan_slab_free+0x6b/0x90 mm/kasan/common.c:285 kasan_slab_free include/linux/kasan.h:235 slab_free_hook mm/slub.c:2689 slab_free mm/slub.c:6251 kfree+0x21f/0x6b0 mm/slub.c:6566 tcf_action_offload_add_ex+0x4ad/0x880 net/sched/act_api.c:284 tcf_action_offload_add net/sched/act_api.c:293 tcf_action_init+0x66e/0xa20 net/sched/act_api.c:1547 tcf_action_add+0xf6/0x5d0 net/sched/act_api.c:2101 The buggy address belongs to the object at ffff88806158de00 which belongs to the cache kmalloc-256 of size 256 The buggy address is located 64 bytes inside of freed 256-byte region [ffff88806158de00, ffff88806158df00) The buggy address belongs to the physical page: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0xffff88806158d600 pfn:0x6158c head: order:1 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0 flags: 0x4fff00000000240(workingset|head|node=1|zone=1|lastcpupid=0x7ff) page_type: f5(slab) raw: 04fff00000000240 ffff88801c841b40 ffffea0001856290 ffffea0001856190 raw: ffff88806158d600 0000000800100009 00000000f5000000 0000000000000000 head: 04fff00000000240 ffff88801c841b40 ffffea0001856290 ffffea0001856190 head: ffff88806158d600 0000000800100009 00000000f5000000 0000000000000000 head: 04fff00000000001 ffffffffffffff81 00000000ffffffff 00000000ffffffff head: ffffffffffffffff 0000000000000000 00000000ffffffff 0000000000000002 page dumped because: kasan: bad access detected page_owner tracks the page as allocated page last allocated via order 1, migratetype Unmovable, gfp_mask 0xd2820(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 9391, tgid 9378 (poc), ts 123227323196, free_ts 0 set_page_owner include/linux/page_owner.h:32 post_alloc_hook+0xfe/0x140 mm/page_alloc.c:1853 prep_new_page mm/page_alloc.c:1861 get_page_from_freelist+0x110c/0x2fc0 mm/page_alloc.c:3941 __alloc_frozen_pages_noprof+0x263/0x2bc0 mm/page_alloc.c:5221 alloc_slab_page mm/slub.c:3278 allocate_slab mm/slub.c:3467 new_slab+0xa6/0x690 mm/slub.c:3525 refill_objects+0x271/0x420 mm/slub.c:7272 refill_sheaf mm/slub.c:2816 __pcs_replace_empty_main+0x373/0x630 mm/slub.c:4652 alloc_from_pcs mm/slub.c:4750 slab_alloc_node mm/slub.c:4884 __do_kmalloc_node mm/slub.c:5295 __kmalloc_noprof+0x66d/0x830 mm/slub.c:5308 kmalloc_noprof include/linux/slab.h:954 metadata_dst_alloc+0x26/0x90 net/core/dst.c:298 tun_rx_dst include/net/dst_metadata.h:144 __ip_tun_set_dst include/net/dst_metadata.h:208 tunnel_key_init+0xb01/0x1b90 net/sched/act_tunnel_key.c:451 tcf_action_init_1+0x46b/0x6c0 net/sched/act_api.c:1428 tcf_action_init+0x448/0xa20 net/sched/act_api.c:1503 tcf_action_add+0xf6/0x5d0 net/sched/act_api.c:2101 [..] ================================================================== Fix by moving dst_release() into a custom RCU callback that runs after the grace period, matching the lifetime of the containing params struct. Readers in the datapath therefore always find a live rcuref when calling dst_clone(). Fixes: 9174c3d ("net/sched: act_tunnel_key: fix memory leak in case of action replace") Reported-by: zdi-disclosures@trendmicro.com Tested-by: Victor Nogueira <victor@mojatatu.com> Signed-off-by: Jamal Hadi Salim <jhs@mojatatu.com> Reviewed-by: Davide Caratti <dcaratti@redhat.com> Link: https://patch.msgid.link/20260711150537.7946-1-jhs@mojatatu.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.