I confirmed this on 6.7 that this was introduced by commit 28628fa952fefc7f2072ce6e8016968cc452b1ba with following changes: static inline void @@ -1397,6 +1394,9 @@ static int ip_set_swap(struct sk_buff *skb, const struct nfnl_info *info, ip_set(inst, to_id) = from; write_unlock_bh(&ip_set_ref_lock); + /* Make sure all readers of the old set pointers are completed. */ + synchronize_rcu(); + return 0; } synchronize_rcu causes the delay, and its usage here is very confusing, there is no reclaimer code after it. FYI David