I thought this was exactly the JSON blob we will implement, If you look at the model of PDC
Those endpoints are covered by relations Release, Compose, RPM through some others.
The PDC approach is more or less a normalized RDBMS model.
This should be replaceable by a much simpler denormalized approach and create one entry for each compose with all the data in one place.
It is also mentioned in the arc docs https://fedora-arc.readthedocs.io/en/latest/pdc/users.html#fedoraqa-fedfind
On Tue, Sep 12, 2023 at 2:18 AM Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote:
On Mon, 2023-09-11 at 09:40 -0700, Kevin Fenzi wrote:
> These are the things that fedfind/qa users? Do we have examples of this
> data?
Okay, here is (again) a list of the stuff fedfind uses. You can see
sample data for each compose in the web UI itself, PDC actually has a
rather good interactive API view.
1. WHAT: https://pdc.fedoraproject.org/rest_api/v1/composes/
HOW: ?release=fedora-NN&compose_label=label
?compose_id=cid
WHY: To find a compose's compose ID from its label, or a compose's
label from its compose ID. These functions are used on various paths.
The most important one in real life is probably when we are creating
and updating validation event pages for candidate composes, between the
time the compose exists and the time it is synced to alt.fp.o - during
this time wikitcms may need to look the compose up by label, but for
fedfind to actually find it, cid_from_label has to work, because it can
only 'natively' find a compose by label if it's been synced to
alt.fp.o; if the compose has not yet been synced, the only way fedfind
can find the compose is to use cid_from_label then locate it on
kojipkgs under its compose ID.
WORKAROUND: it would be difficult to work around this if we did not
have the ability. For wikitcms we could try 'embedding' the compose ID
in the event pages somehow. For some other less important uses there is
probably no workaround.
2. WHAT: https://pdc.fedoraproject.org/rest_api/v1/compose-images/
HOW: compose-images/(composeid)
WHY: To provide image metadata for nightly composes that have been
garbage-collected, and releases in the mirror system after they have
been split across directories and their metadata stripped
WORKAROUND: if there is no store of metadata by compose ID then
fedfind just can't do this any more. It's not critical functionality to
anything important I'm aware of, though - it's just a nice feature that
can be used to e.g. run long-term analysis of image size changes across
multiple releases, stuff like that. If this goes away I will just drop
this feature from fedfind and you will no longer be able to find the
real metadata for these composes (it would provide synthesized metadata
for mirrored releases, but not garbage-collected nightlies). Note,
retrieving the original metadata for mirrored releases depends on the
cid_from_label feature (see 1).
3. WHAT: https://pdc.fedoraproject.org/rest_api/v1/releases/
HOW: ?version=39&name=Fedora (for e.g.)
WHY: To find the compose that was "previous" to any given compose.
The first result for this PDC query for any given release is a dict
with a handy 'compose_set' key whose value is a list of all the
composes for that release, in reverse order by date. So we can easily
find the compose that came 'before' any given compose. This is, I
*think*, only used by the 'check-compose' script which sends out those
"compose check report" emails.
WORKAROUND: fedfind has a hideous alternative implementation of this
which involves just blindly trying possible decrements and seeing if
they exist. e.g. if you ask for the compose previous to Fedora-39-
20230911.n.1, it will try 20230911.n.0, if that doesn't exist, it will
try 20230910.n.5 (we start counting backwards from 5 because, hey,
gotta start somewhere, and doing more than 5 composes on one day is
unlikely), then 20230910.n.4, and so on, until it hits something that
exists. So, if we can't get a nice data set like this any more...we'll
just have to use that. Bleh. We might also just drop check-compose and
kill the feature, I suppose. I don't pay as much attention to those
reports as I used to.
4. WHAT: https://pdc.fedoraproject.org/rest_api/v1/rpms/
HOW: ?compose=cid&arch=src&name=^package1$&name=^package2$....
WHY: To get the NEVR (name, epoch, version, release) for a given set
of packages from a compose. This can be used by relvalconsumer (the
thing that creates the wiki validation events) to decide whether any
"important" packages changed between the last tested compose and the
compose it's currently deciding whether to create a validation event
for: if it's been more than 3 days but less than 14 days since the last
event, it will create a new event *if* any important package's version
changed.
WORKAROUND: Looking back on the history of this, I don't think
anything's actually using it right now. There is an ugly alternative
version of this one, too, which scrapes dl.fedoraproject.org's HTML
output (told you it was ugly!). I initially replaced that version with
the PDC one for some compose types, but then ran into problems with
that, and ultimately decided to provide both side-by-side; I think
relvalconsumer is the only thing that actually uses this feature, and
it currently uses the ugly HTML scraping version, not the PDC version.
I probably meant to try and make it use the PDC version but never got
around to it. The ugly version is working okay in practice, so I
suppose the workaround here would just be to drop the pretty PDC
version and rely on HTML scraping forever (sob).
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@xxxxxxxxxxxxx
https://www.happyassassin.net
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Tomas Hrcka
fas: humaton
libera.CHAT: jednorozec
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
