rmmod br_netfilter or set net.bridge.bridge-nf-call-arptables=0 net.bridge.bridge-nf-call-iptables=0 net.bridge.bridge-nf-call-ip6tables=0
Alright, the possibility that bridged (non-routed) packets don't need to go through filtering rules didn't occur to me at all
Thanks a lot for pointing it out, no physdev-is-bridged needed for me now.