Re: Corosync instances seems to ignore each other despite many UDP chat without firewall

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello.

Thank you for your help; unfortunately, even with your clues, it doesn't work. Here come my config with your suggestions applied :

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:25:90:77:84:de 
          inet adr:37.59.18.208  Bcast:37.59.18.255  Masque:255.255.255.0
          adr inet6: fe80::225:90ff:fe77:84de/64 Scope:Lien
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:562019 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1325787 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000
          RX bytes:87811137 (83.7 MiB)  TX bytes:221866389 (211.5 MiB)
          Interruption:16 Mémoire:fbce0000-fbd00000

lo        Link encap:Boucle locale 
          inet adr:127.0.0.1  Masque:255.0.0.0
          adr inet6: ::1/128 Scope:Hôte
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8139 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:0
          RX bytes:828638 (809.2 KiB)  TX bytes:828638 (809.2 KiB)

tap0      Link encap:Ethernet  HWaddr 7a:5c:2a:32:ee:30 
          inet adr:10.88.0.2  Bcast:10.88.0.255  Masque:255.255.255.0
          adr inet6: fe80::785c:2aff:fe32:ee30/64 Scope:Lien
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:51971 errors:0 dropped:0 overruns:0 frame:0
          TX packets:42362 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:100
          RX bytes:9387727 (8.9 MiB)  TX bytes:8169107 (7.7 MiB)

cat corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank

totem {
        version: 2
        secauth: off
        interface {
                member {
                        memberaddr: 10.88.0.1
                }
                member {
                        memberaddr: 10.88.0.2
                }
                ringnumber: 0
                bindnetaddr: 10.88.0.0
                mcastport: 5405
                ttl: 1
        }
        transport: udpu
}

logging {
        fileline: off
        to_logfile: yes
        to_syslog: yes
        debug: on
        logfile: /var/log/corosync.log
        debug: off
        timestamp: on
        logger_subsys {
                subsys: AMF
                debug: off
        }
}

As you can see, I established an OpenVPN TAP connection in order to have a VLAN between these distant machines (geographically distant, like 400 km or 250 mi, to improve cluster reliability) to ensure they share a subnetwork. I checked connectivity, and the node are able to join each other via this VPN (at least via SSH). I also disabled the firewall by the time I solve this problem, as there are no content served and no established connection with our facility, and to make sure firewall is not interfering.
# Node 1
iptables -nvL
Chain INPUT (policy ACCEPT 143K packets, 24M bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain OUTPUT (policy ACCEPT 159K packets, 28M bytes)
 pkts bytes target     prot opt in     out     source               destination

# Node 2
iptables -nvL
Chain INPUT (policy ACCEPT 144K packets, 26M bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain OUTPUT (policy ACCEPT 125K packets, 23M bytes)
 pkts bytes target     prot opt in     out     source               destination

Unfortunately, it still doesn't work.

Nevertheless, instances stopped filling their logs with connectivity warnings; instead, I found these messages after making the changes you suggested :
# Node 1
Jun 07 10:24:07 corosync [MAIN  ] Corosync Cluster Engine ('1.4.2'): started and ready to provide service.
Jun 07 10:24:07 corosync [MAIN  ] Corosync built-in features: nss
Jun 07 10:24:07 corosync [MAIN  ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
Jun 07 10:24:07 corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
Jun 07 10:24:07 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jun 07 10:24:07 corosync [TOTEM ] The network interface [10.88.0.1] is now up.
Jun 07 10:24:07 corosync [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Jun 07 10:24:07 corosync [SERV  ] Service engine loaded: corosync configuration service
Jun 07 10:24:07 corosync [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Jun 07 10:24:07 corosync [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Jun 07 10:24:07 corosync [SERV  ] Service engine loaded: corosync profile loading service
Jun 07 10:24:07 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Jun 07 10:24:07 corosync [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Jun 07 10:24:07 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jun 07 10:24:07 corosync [CPG   ] chosen downlist: sender r(0) ip(10.88.0.1) ; members(old:0 left:0)
Jun 07 10:24:07 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Jun 07 10:24:07 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jun 07 10:24:08 corosync [CPG   ] chosen downlist: sender r(0) ip(10.88.0.1) ; members(old:1 left:0)
Jun 07 10:24:08 corosync [MAIN  ] Completed service synchronization, ready to provide service.

# Node 2
Jun 07 10:23:51 corosync [MAIN  ] Corosync Cluster Engine ('1.4.2'): started and ready to provide service.
Jun 07 10:23:51 corosync [MAIN  ] Corosync built-in features: nss
Jun 07 10:23:51 corosync [MAIN  ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
Jun 07 10:23:51 corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
Jun 07 10:23:51 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jun 07 10:23:51 corosync [TOTEM ] The network interface [10.88.0.2] is now up.
Jun 07 10:23:51 corosync [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Jun 07 10:23:51 corosync [SERV  ] Service engine loaded: corosync configuration service
Jun 07 10:23:51 corosync [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Jun 07 10:23:51 corosync [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Jun 07 10:23:51 corosync [SERV  ] Service engine loaded: corosync profile loading service
Jun 07 10:23:51 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Jun 07 10:23:51 corosync [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Jun 07 10:23:51 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jun 07 10:23:51 corosync [CPG   ] chosen downlist: sender r(0) ip(10.88.0.2) ; members(old:0 left:0)
Jun 07 10:23:51 corosync [MAIN  ] Completed service synchronization, ready to provide service.
Jun 07 10:23:59 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jun 07 10:24:00 corosync [CPG   ] chosen downlist: sender r(0) ip(10.88.0.1) ; members(old:1 left:0)
Jun 07 10:24:00 corosync [MAIN  ] Completed service synchronization, ready to provide service.

In addition, when I check listening services, I get this :

netstat -lnptu
Connexions Internet actives (seulement serveurs)
Proto Recv-Q Send-Q Adresse locale          Adresse distante        Etat        PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3012/nginx     
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      17881/named    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      16480/sshd     
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      17881/named    
tcp6       0      0 ::1:53                  :::*                    LISTEN      17881/named    
tcp6       0      0 :::22                   :::*                    LISTEN      16480/sshd     
tcp6       0      0 ::1:953                 :::*                    LISTEN      17881/named    
udp        0      0 0.0.0.0:58265           0.0.0.0:*                           3630/corosync  
udp        0      0 127.0.0.1:53            0.0.0.0:*                           17881/named    
udp        0      0 127.0.0.1:921           0.0.0.0:*                           17788/lwresd   
udp        0      0 0.0.0.0:35009           0.0.0.0:*                           3630/corosync  
udp        0      0 10.88.0.2:5405          0.0.0.0:*                           3630/corosync  
udp6       0      0 ::1:53                  :::*                                17881/named    

Is it me or were they able to open an listening socket, detect each other and are waiting for prividing services? In this case, why crm_mon --one-shot -V still responds "Connection to cluster failed: connection failed"?

Still a little config issue, I assume, but where?

Thank you in advance.

Regards.

Le 07/06/2012 10:17, Jan Friesse a écrit :
This is expected behavior, and even more makes me sure that whole problem is really hidden in nonexisting local member addr in your config.

Honza

David Guyot napsal(a):
Hello again, everybody.

I just noticed that, when I tried to set secauth to off, during the
period of time in which one node accepted secured connections one the
other unsecured connections, the network fault message were replaced by
these :
Jun 06 17:16:17 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:17 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:17 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:17 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:17 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:17 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:17 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:17 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:17 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:17 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:17 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:17 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:17 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:17 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data
Jun 06 17:16:18 corosync [TOTEM ] Received message has invalid digest...
ignoring.
Jun 06 17:16:18 corosync [TOTEM ] Invalid packet data

If this is relevant...

Thank you in advance.

Regards.

Le 06/06/2012 17:05, David Guyot a écrit :
Hello, everybody.

I'm trying to establish a 2-node Debian Squeeze x64 cluster with
Corosync and Pacemaker, but I'm hanged with a strange issue : despite a
lot of UDP chatting between the nodes (so network is OK but), each
Corosync instance seems to ignore each other : the other node is never
detected, and crm_mon --one-shot -V only says "Connection to cluster
failed: connection failed". But the strangest in there is that both
Corosync nodes are filling their logs with error messages saying "Totem
is unable to form a cluster because of an operating system or network
fault. The most common cause of this message is that the local firewall
is configured improperly.". I tcpdumped all traffic between the hosts,
and I have 2-way traffic between them. I tried to use backports versions
of all Corosync- and Pacemaker-related packages, without improvement.

I must add that, due to my hosting company network policy, I was forced
to use UPD-Unicast instead of multicast, because multicast is blocked.

Here comes my config :
corosync.conf :
# Please read the corosync.conf.5 manual page
compatibility: whitetank

totem {
         version: 2
         secauth: on
         interface {
                 member {
                         memberaddr: 176.31.238.131
                 }
                 ringnumber: 0
                 bindnetaddr: 37.59.18.208
                 mcastport: 5405
                 ttl: 1
         }
         transport: udpu
}

logging {
         fileline: off
         to_logfile: yes
         to_syslog: yes
         debug: on
         logfile: /var/log/corosync.log
         debug: off
         timestamp: on
         logger_subsys {
                 subsys: AMF
                 debug: off
         }
}

Log messages :
Jun 06 16:35:14 corosync [MAIN  ] Corosync Cluster Engine ('1.4.2'):
started and ready to provide service.
Jun 06 16:35:14 corosync [MAIN  ] Corosync built-in features: nss
Jun 06 16:35:14 corosync [MAIN  ] Successfully read main configuration
file '/etc/corosync/corosync.conf'.
Jun 06 16:35:14 corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
Jun 06 16:35:14 corosync [TOTEM ] Initializing transmit/receive
security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jun 06 16:35:14 corosync [TOTEM ] The network interface [37.59.18.208]
is now up.
Jun 06 16:35:14 corosync [SERV  ] Service engine loaded: corosync
extended virtual synchrony service
Jun 06 16:35:14 corosync [SERV  ] Service engine loaded: corosync
configuration service
Jun 06 16:35:14 corosync [SERV  ] Service engine loaded: corosync
cluster closed process group service v1.01
Jun 06 16:35:14 corosync [SERV  ] Service engine loaded: corosync
cluster config database access v1.01
Jun 06 16:35:14 corosync [SERV  ] Service engine loaded: corosync
profile loading service
Jun 06 16:35:14 corosync [SERV  ] Service engine loaded: corosync
cluster quorum service v0.1
Jun 06 16:35:14 corosync [MAIN  ] Compatibility mode set to whitetank.
Using V1 and V2 of the synchronization engine.
Jun 06 16:35:23 corosync [TOTEM ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause
of this message is that the local firewall is configured improperly.
Jun 06 16:35:25 corosync [TOTEM ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause
of this message is that the local firewall is configured improperly.
Jun 06 16:35:27 corosync [TOTEM ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause
of this message is that the local firewall is configured improperly.
Jun 06 16:35:30 corosync [TOTEM ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause
of this message is that the local firewall is configured improperly.

# uname -a
Linux Vindemiatrix 3.2.13-grsec-xxxx-grs-ipv6-64 #1 SMP Thu Mar 29
09:48:59 UTC 2012 x86_64 GNU/Linux

# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source
destination
     0     0 ACCEPT     all  --  tun0   *       0.0.0.0/0
0.0.0.0/0
     0     0 ACCEPT     all  --  lo     *       0.0.0.0/0
0.0.0.0/0
     0     0            tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp dpt:22 state NEW recent: SET name: SSH side: source
     0     0 LOGDROP    tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp dpt:22 state NEW recent: UPDATE seconds: 60
hit_count: 6 TTL-Match name: SSH side: source
     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp dpt:22 state NEW
     0     0 LOGDROP    tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp flags:0x17/0x02 multiport dports 80,443 #conn/32
100
     1    48 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp flags:0x17/0x02 multiport dports 80,443
     0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpt:21 flags:0x17/0x02 limit: avg 5/min burst 50
recent: SET name: FTP side: source
     0     0 LOGDROP    tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpt:21 flags:0x17/0x02 recent: UPDATE seconds:
60 hit_count: 6 TTL-Match name: FTP side: source
     0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpt:21 flags:0x17/0x02
     0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpts:50000:50500 state RELATED,ESTABLISHED
     0     0 ACCEPT     tcp  --  eth0   *       176.31.238.131
0.0.0.0/0           tcp dpt:1194
11867 3145K ACCEPT     udp  --  *      *       0.0.0.0/0
0.0.0.0/0           udp dpt:5405 /* Corosync */
    35  9516 ACCEPT     all  --  eth0   *       0.0.0.0/0
0.0.0.0/0           state NEW limit: avg 30/sec burst 200
     0     0 LOGDROP    tcp  --  eth0   *       0.0.0.0/0
0.0.0.0/0           tcp dpt:80 STRING match "w00tw00t.at.ISC.SANS." ALGO
name bm TO 65535
     0     0 ACCEPT     icmp --  *      *       0.0.0.0/0
0.0.0.0/0           limit: avg 10/sec burst 5
     0     0 LOGDROP    icmp --  *      *       0.0.0.0/0
0.0.0.0/0
  1031 70356 ACCEPT     all  --  *      *       0.0.0.0/0
0.0.0.0/0           state RELATED,ESTABLISHED
     3   132 LOGDROP    all  --  *      *       0.0.0.0/0
0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source
destination
     0     0 LOGDROP    all  --  *      *       0.0.0.0/0
0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source
destination
     0     0 ACCEPT     all  --  *      tun0    0.0.0.0/0
0.0.0.0/0
     0     0 ACCEPT     all  --  *      lo      0.0.0.0/0
0.0.0.0/0
     0     0 LOGDROP    tcp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           tcp dpt:80 owner UID match 33
     0     0 LOGDROP    udp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           udp dpt:80 owner UID match 33
     0     0 LOGDROP    tcp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           tcp dpt:443 owner UID match 33
     0     0 LOGDROP    udp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           udp dpt:443 owner UID match 33
     0     0 ACCEPT     tcp  --  *      eth0    0.0.0.0/0
176.31.238.131      tcp dpt:1194
11871 3146K ACCEPT     udp  --  *      *       0.0.0.0/0
0.0.0.0/0           udp dpt:5405 /* Corosync */
     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp dpt:22
     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0           tcp dpt:25
     0     0 ACCEPT     tcp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           tcp dpt:43
     0     0 ACCEPT     tcp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           tcp dpt:53
     0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           udp dpt:53
     0     0 ACCEPT     tcp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           tcp dpt:80
     0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           udp dpt:123
     0     0 ACCEPT     tcp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           tcp dpt:443
     0     0 ACCEPT     tcp  --  *      eth0    0.0.0.0/0
0.0.0.0/0           tcp dpt:873
    11   924 ACCEPT     icmp --  *      *       0.0.0.0/0
0.0.0.0/0
  1071  712K ACCEPT     all  --  *      *       0.0.0.0/0
0.0.0.0/0           state RELATED,ESTABLISHED
    67 14013 LOGDROP    all  --  *      *       0.0.0.0/0
0.0.0.0/0

Chain LOGDROP (12 references)
  pkts bytes target     prot opt in     out     source
destination
    57 11655 LOG        all  --  *      *       0.0.0.0/0
0.0.0.0/0           limit: avg 1/sec burst 5 LOG flags 0 level 5 prefix
`iptables rejected: '
    70 14145 DROP       all  --  *      *       0.0.0.0/0
0.0.0.0/0

# corosync -v
Corosync Cluster Engine, version '1.4.2'
Copyright (c) 2006-2009 Red Hat, Inc.

I've been trying to solve this problem the 2 last days, without any
result. Any help welcome.

Thank ou in advance!

Regards.






_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss



Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss

[Index of Archives]     [Linux Clusters]     [Corosync Project]     [Linux USB Devel]     [Linux Audio Users]     [Photo]     [Yosemite News]    [Yosemite Photos]    [Linux Kernel]     [Linux SCSI]     [X.Org]

  Powered by Linux