Now that the project has started producing packages that are larger than 2GB (mostly the ceph-debuginfo ones), it has caused issues on the build system. It was noted on Tuesday that builds were failing with "OverflowErrors" and a Python traceback (part of the process that moves binaries around). The fix involved updating the client to do multipart uploads, and updating the database TYPE that capture the size (in bytes) of binaries which was set to INT. The max integer is 2**31 which coincides with roughly 2GB. The TYPE was updated to BIGINT. Propagating the change takes a while because it is done in a way to avoid disrupting existing builds, so nodes are taken down one at a time when there aren't any pending builds. All the heavy lifting was done by David Galloway (thanks David!) tl;dr Builds failing this week were because the build system couldn't handle binaries larger than 2GB, this was solved late last night by patching the uploading tool and updating the databases that store package metadata. -Alfredo
