Hi everyone, Happy New Year! We have the first weekly call of the new year today, in 2 hours from now. At that time, I would like to propose the following agenda. Short form inline here, then longer commentary follows. I would additionally like to propose that we have a followup meeting either tomorrow (Fri) or on Monday. Therefore, please let us know on the call what would work. Short version: 0). Progress over the holidays - We know this wasn't great, but let's just cover it briefly. 1). Build system status - Just a quick update on what is currently up, and what is not. 2). Short term infrastructure fixes - Bigger discussion about getting building back on track. - Notice not "long term", save that for later please. 3). Communication - Better co-ordination (specific topic for FUDCon) - Planning around outages, who has the ball, and docs 4). Staging before deploying - Need to get into a habit of staging new ideas - Focus is to retain uptime on the hub so that builders don't waste many cycles on aborted builds at least. 5). FUDCon. - Raspberry Pi "coupon" plan update? - Clearly we need to focus on rawhide and build system 6). Other business. And now the longer version: 0). Progress over the holidays - We know this wasn't great, but let's just cover it briefly. 1). Build system status - Just a quick update on what is currently up, and what is not. - Let's understand where we are before we discuss fixing things. 2). Short term infrastructure fixes - We need to get to rawhide. We don't need to have an optimized Cloud-hardened Enterprise Grade solution this week. That means anything that works is fine for the next few weeks. Let's be real because we've got FUDCon very soon, and a branch is about to happen, so every day spent dreaming up ideas is lost time. We need to focus on pragmatic real solutions that can be achieved with the resources available, not go "blue sky". - It is known that the NFSv4 approach is unlikely to work as-is due to problems with xattrs on NFSv4 (apparently). Therefore, we should try to find an alternative solution. - There are two problems here that are largely disjoint: 1). Improving IOPS so we don't drown in IO (largely independently of higher layers) 2). What is provided to the individual builders (loopback, ISCSI, network block device, other) To improve upon 1, the following ideas have been floated: - Switch to using SSDs (Red Hat willing to help fund) - Increase the spindle count (again, funds available) - Switch to local storage on the builders This is my understanding of the realities: - The SSD option is likely to be a quick, easy win. Chris says he may have numbers for us this morning. - The spindle count option is secondary, can just add some new disks, and that's fairly quickly done too. - Local storage for now means using the SD Cards, which we are willing to sacrifice if they die early as a result. The suggestion is to setup a few "small" builders with local storage while the NFS is worked. To improve upon 2, the following ideas have been floated: - Switch to using NFSv4 directly from builders (does not seem to actually be possible with the xattr reqs.) - Retain existing (kludge, but well used) loopback - Adopt ISCSI based individual partition mounts - Adopt Network Block Device individual mounts My suggestion is that we keep the loopback option *unless* it is known that switching to ISCSI will also magically solve the hardware IOPS issue on the server (which it won't). I believe it is fastest and easiest to shove some SSDs in the NFS box, keep serving loopback mounted files via NFS. It has been used successfully in the past, and we only need to get back there. 3). Communication - There has been a little "spirited conversation" on IRC of late, largely I think because we need to collectively work on our communication. Not a blame game, just let's figure out what we would all like in the way of future discussion. - Suggest that we have a formalized process for planning any significant project changes, whether infrastructure or not, bring them up on the *mailing list* (and IRC for initial input, but mailing list after that). We should document exactly what we plan to do in a particular situation so that we all agree on what it is that we are doing :) - Start with the build system fixing. Let's figure out on this call what it is that we want, then write it up, then do it. We should absolutely write up what it is first. 4). Staging before deploying - Formal process for staging and testing. The current build system is also a great learning platform, and that should be encouraged. At the same time, we're growing increasing dependencies upon it, so we should move to a setup wherein significant changes are staged with a small number of test builders first, like in the Phoenix Fedora infrastructure. Red Hat is willing to help fund some hardware if needed. 5). FUDCon - Any updates from Eben on the Raspberry Pi front? If not, we should get that resolved before early next week. If we can at least agree what the rate and terms will be, we can take names at FUDCon and arrange the actual coupon codes to be sent to them by email later on, that's not an issue. - Let's use the FUDCon high bandwidth opportunity effectively. - Need to plan rawhide, primary arch plans. 6). Other business? Meeting Logistics ----------------- Leaders: ctyler (Chris Tyler), jsmith (Jared Smith), jonmasters (me) IRC: #fedora-arm (using the bot to track the meeting details) Conference code: 617-759-1337 (no passcode required) Toll Free Dial-In Number (US & Canada): (800) 451-8679 International Dial-In Number: +1-(212) 729-5016 Global Access Numbers Tollfree: Argentina Dial-In #: 8004441016 Australia Dial-In #: 1800337169 Austria Dial-In #: 0800005898 Bahamas Dial-In #: 18002054776 Bahrain Dial-In #: 80004377 Belgium Dial-In #: 080048325 Brazil Dial-In #: 08008921002 Bulgaria Dial-In #: 008001100236 Chile Dial-In #: 800370228 Colombia Dial-In #: 018009134033 Costa Rica Dial-In #: 08000131048 Cyprus Dial-In #: 80095297 Czech Republic Dial-In #: 800 701 035 Denmark Dial-In #: 80887114 Dominican Republic Dial-In #: 18887512313 Estonia Dial-In #: 8000100232 Finland Dial-In #: 0800117116 France Dial-In #: 0805632867 Germany Dial-In #: 08006647541 Greece Dial-In #: 00800127562 Hong Kong Dial-In #: 800930349 Hungary Dial-In #: 0680016796 Iceland Dial-In #: 8008967 India Dial-In #: 0008006501533 Indonesia Dial-In #: 0018030179162 Ireland Dial-In #: 1800932401 Israel Dial-In #: 1809462557 Italy Dial-In #: 800985897 Jamaica Dial-In #: 18002050328 Japan Dial-In #: 0120934453 Korea (South) Dial-In #: 007986517393 Latvia Dial-In #: 80003339 Lithuania Dial-In #: 880030479 Luxembourg Dial-In #: 80026595 Malaysia Dial-In #: 1800814451 Mexico Dial-In #: 0018664590915 New Zealand Dial-In #: 0800888167 Norway Dial-In #: 80012994 Panama Dial-In #: 008002269184 Philippines Dial-In #: 180011100991 Poland Dial-In #: 008001210187 Portugal Dial-In #: 800814625 Russian Federation Dial-In #: 81080028341012 Saint Kitts and Nevis Dial-In #: 18002059252 Singapore Dial-In #: 8006162235 Slovak Republic Dial-In #: 0800001441 South Africa Dial-In #: 0800981148 Spain Dial-In #: 800300524 Sweden Dial-In #: 200896860 Switzerland Dial-In #: 800650077 Taiwan Dial-In #: 00801127141 Thailand Dial-In #: 001800656966 Trinidad and Tobago Dial-In #: 18002024615 United Arab Emirates Dial-In #: 8000650591 United Kingdom Dial-In #: 08006948057 United States Dial-In #: 8004518679 Uruguay Dial-In #: 00040190315 Venezuela Dial-In #: 08001627182 Jon. _______________________________________________ arm mailing list arm@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/arm
