Changes in EPEL mirror content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



As several people have noted, the file directory structure of EPEL has
changed recently. This layout may require changes in both (1) scripts
written with hard-coded locations, and (2) mirrors which were unable
to get daily updates from the main mirrors.  While the changes were
communicated in meetings, I did not adequately comprehend their
effects to let mirrors and EPEL users know about it. This meant this
announcement was delayed over two weeks.

===============
 What Happened
===============

The updates in the build system were to add new features and make the
release engineering code more manageble. The old release style used by
EPEL in EL-6 and EL-7 was different from how all other releases were
done and caused several problems for the release code and mirrors.

1. Due to all the files of the release being in one directory, any
   code which needed to stat (2) the directory caused the server to go
   over thousands of files before returning. With EPEL being a large
   amount of downloads, this negatively impacted systems. Servers
   mirroring the data could find long delays in rsyncing the data
   down.

2. The code that generated this was a 'special' case in the Fedora
   releng release process which was fragile and tended to cause
   problems for updates and releases in both EPEL and the normal
   release.

3. The layouts were different from the current Fedora release so that
   people grabbing software from multiple places also had to special
   case their scripts.

During the updates to the release system with a new version of pungi,
it was decided to remove this special case and have all software
Fedora created laid out in the same structure by the build tools. This
would hopefully make things much more maintainable and improve
performance.

In order to safely transition, there would be a time where the old
files would remain on the server in the old trees and also be
hardlinked to their new location. This was intended to allow for
mirrors to get the files with the minimum amount of bandwidth. However
there were some problems which showed up.

1. As I said before, I didn't grasp that the change was going to
   affect EPEL and didn't communicate this to the lists.

2. The transition time for removing the hardlinks was in days versus
   weeks. While most mirrors do daily updates, some only do weekly or
   monthly rsync's. They missed the hardlinks completely and had to
   download data twice.

3. In the usual rule of three, various top level mirrors
   (mirrors.kernel.org and some others) had un-related mirroring
   problems at the end of October. When these servers caught up with
   the new layout, the hardlinked files were gone. This meant that
   mirrors taking data from a couple of tier1 sites had large uploads.

=================================
 How to deal with current things
=================================

The current layout structure should be 'solid' for the next couple of
years. With the break down of packages into alphabetical subtrees, the
'load' per server should not require a re-ordering in the near
future.

If you have written scripts which downloaded a specific file from the
mirrors, (aka http://dl.fedoraproject.org/pub/archive/epel/5/i386/epel-release-5-4.noarch.rpm
or some similar link), you should instead use a stable linked package
like http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
The epel-release packages get updated regularly to get new macros or
other changes so linking to a specific file is very error prone.

Otherwise one should use yum/dnf related commands to get the files
from the mirrors. This is useful for mirror sites which may alter the
directory structure themselves and thus only the repodata is 'safe' to
figure out what to download.


-- 
Stephen J Smoogen.
_______________________________________________
epel-devel mailing list -- epel-devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to epel-devel-leave@xxxxxxxxxxxxxxxxxxxxxxx




[Index of Archives]     [Fedora Announce]     [Fedora News]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Maintainers]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Legacy]     [Fedora Desktop]     [Fedora Fonts]     [ATA RAID]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [Fedora Package Announce]     [SSH]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Centos]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora QA]     [Fedora Triage]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Tux]     [Yosemite News]     [Linux Apps]     [Gnome Users]     [KDE Users]     [Fedora Tools]     [Fedora Art]     [Fedora Docs]     [Maemo Users]     [Asterisk PBX]     [Fedora Sparc]     [Fedora Universal Network Connector]     [Fedora ARM]

  Powered by Linux