mac802_hwsim: libnl does not detect genl hwsim protocol in default ns

[Benjamin Beichler <Benjamin.Beichler@xxxxxxxxxxxxxx>]

Hi,

I got a strange problem with newer kernel (post 4.8 stable) with hwsim.
Our current code works like this:

modprobe mac80211_hwsim radios=10

./costum_wmediumd

This will create all hwsim_interfaces in the first hwsim "netgroup"
within the default network namespace. But the code to lookup the genl
"MAC80211_HWSIM" protocol fails. Moreover some calls fails, other not ...

    //family = genl_ctrl_search_by_name(cache, "MAC80211_HWSIM"); -->
fails directly (return NULL)

    int hwsim_netlink_id = genl_ctrl_resolve(sock,"MAC80211_HWSIM");   
    if (hwsim_netlink_id == 0)
    {
        printf("Family MAC80211_HWSIM ID not found\n");
        exit(EXIT_FAILURE);
    }
    printf("Family ID found (%d), search for structure
\n\n",hwsim_netlink_id);
    family = genl_ctrl_search(cache, hwsim_netlink_id);
    if (family == NULL)
    {
        printf("Family MAC80211_HWSIM family not found\n");
        exit(EXIT_FAILURE);
    }

this will output:

"Family ID found (28), seatch for structure

Family MAC80211_HWSIM family not found"

But when I start the code in some network namespace, it works (but works
with no interfaces, as they stay in default network namespace). Of
course a workaround is easy, but it definitely breaks some legacy code,
working the same way. I reviewed the recent changes of hwsim regarding
namespaces, but I found no bug. The initial hwsim-devices are created in
netgroup 0 and the default namespace and I don't see any differentiation
of the netlink protocol registration, which depends on network
namespaces. Maybe there is a bug in libnl, but currently I have no Idea,
how to track down the issue.

Any suggestions ?