Re: Mirrormanager thoughts

[Matt Domsch <Matt_Domsch@xxxxxxxx>]

On Sun, Jan 07, 2007 at 11:38:31PM -0600, Matt Domsch wrote:
> https://hosted.fedoraproject.org/projects/mirrormanager/wiki/WikiStart
> 
> I've redone the mirrormanager schema and initialization data.  I
> haven't redone the controllers or templates at all yet, so this won't
> run except in the tg-admin shell and tg-admin toolbox (catwalk is
> nice).  Please take a look.  I append the schema file for review.  The
> hierarchy is as follows:

More schema updates.  Three new objects.

Product is a top-level content name, e.g. 'rhel' or 'fedora'.

Products can have many ProductVersions.  Some versions are 'test' versions
(e.g. 6.90).  This needed as the path to fedora-test-6.90 looks like
fedora/linux/core/test/6.90/ while the path to fedora-core-6 looks
like fedora/linux/core/6/ (the extra 'test' indirection there...)

Category is meant to be able to capture the standard directory
structure for 'core', 'extras', 'updates', 'updates-testing', 'test',
and 'epel' cleanly.  It uses $VERSION and $ARCH in the path names to
be replaced when creating a Content.


> Content is the highest level unit of data that can be synced.

Content refers to the combination of a Category, a ProductVersion, and
an Arch, and is still the highest level unit of data that can be synced.
These have names like fedora-core-6-i386. (product, category, version, arch).

The other objects are unchanged fundamentally.
Your comments would be appreciated.

Thanks,
Matt

-- 
Matt Domsch
Software Architect
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com


from sqlobject import *

from turbogears.database import PackageHub


hub = PackageHub("mirrors")
__connection__ = hub

from mirrors.identity_models import *

class Site(SQLObject):
    name = StringCol(alternateID=True)
    robot_email = StringCol(default=None)
    admin_active = BoolCol(default=False)
    user_active = BoolCol(default=True)
    private = BoolCol(default=False)
    admins = MultipleJoin('SiteAdmin')
    hosts  = MultipleJoin('Host')
    ips    = MultipleJoin('SiteIP')
    

class SiteAdmin(SQLObject):
    username = StringCol()
    site = ForeignKey('Site')

# IP addresses to go in the rsync ACLs
class SiteIP(SQLObject):
    site = ForeignKey('Site')
    address = StringCol()

class Host(SQLObject):
    name = StringCol()
    site = ForeignKey('Site')
    country = StringCol(default=None)
    internet2 = BoolCol(default=False)
    pull_from = ForeignKey('Site')
    mirrors = MultipleJoin('Mirror')
    private_rsyncs = MultipleJoin('HostPrivateRsync')
    ip_blocks = MultipleJoin('HostIPBlocks')

# for other mirrors to pull from in tiering
class HostPrivateRsync(SQLObject):
    host = ForeignKey('Host')
    url = StringCol()
    user = StringCol(default=None)
    password = StringCol(default=None)

# some hosts only serve some IP CIDR blocks
# such as private mirrors behind
# company firewalls.  This lets them first try
# the global yum mirrorlist redirector which will
# return them their local mirror URL
class HostIPBlocks(SQLObject):
    host = ForeignKey('Host')
    cidr    = StringCol()


class Arch(SQLObject):
    name = StringCol(alternateID=True)

class Category(SQLObject):
    # e.g. core, extras, updates, updates-testing, test, ...
    name = StringCol(alternateID=True)
    # with $VERSION and $ARCH for later substitution
    path = StringCol()
    canonicalhost = StringCol()
    sourcepkgpath = StringCol()
    sourceisopath = StringCol(default=None)
    contents = MultipleJoin('Content')

class Product(SQLObject):
    name = StringCol(alternateID=True)
    versions = MultipleJoin('ProductVersion')

class ProductVersion(SQLObject):
    name = StringCol()
    product = ForeignKey('Product')
    isTest = BoolCol(default=False)


class Content(SQLObject):
    # e.g. fedora-core-6-i386
    name = StringCol(alternateID=True)
    category = ForeignKey('Category')
    # 5, 6, development
    version = ForeignKey('ProductVersion')
    # default subdir starting below /
    # e.g. 'pub/fedora/core/6/$ARCH/'
    path = StringCol()
    arch = ForeignKey('Arch')
    # these are paths below $ARCH/
    # e.g. iso/, os/, debuginfo/
    # of course Extras doesn't have these ;-(
    isos  = StringCol(default=None)
    binarypackages  = StringCol(default=None)
    repodata  = StringCol(default=None)
    debuginfo = StringCol(default='debug/')
    repoview  = StringCol(default=None)
    mirrors = MultipleJoin('Mirror')

# one per Host per Content
# fortunately these will be created/modified
# by a crawler program rather than manually
class Mirror(SQLObject):
    host = ForeignKey('Host')
    content  = ForeignKey('Content')
    urls = MultipleJoin('MirrorURL')
    # sync status
    dvd_isos_synced    = BoolCol(default=False)
    cd_isos_synced     = BoolCol(default=False)
    binarypackages_synced  = BoolCol(default=False)
    debuginfo_synced = BoolCol(default=False)
    repoview_synced  = BoolCol(default=False)

# One per protocol/path one can get at this same data
# checking one MirrorURL is the same as checking them all
class MirrorURL(SQLObject):
    mirror = ForeignKey('Mirror')
    # The equivalent of the dir structure on the masters
    # e.g. http://mirrors.kernel.org/fedora/core/6/i386/os/
    path      = StringCol(default=None)