On 3/16/20 6:25 PM, Neal Gompa wrote:
On Mon, Mar 16, 2020 at 11:24 AM Ben Cotton <bcotton@xxxxxxxxxx> wrote:
https://fedoraproject.org/wiki/Changes/Sqlite_Rpmdb
== Summary ==
Change format of the RPM database from Berkeley DB to a new Sqlite format.
== Owner ==
* Name: [[User:pmatilai| Panu Matilainen]] [[User:ffesti|Florian Festi]]
* Email: pmatilai@xxxxxxxxxx ffesti@xxxxxxxxxx
== Detailed Description ==
The current rpm database implementation is based on Berkeley DB 5.x, a
version which is unmaintained upstream for several years now. Berkeley
DB 6.x is license incompatible so moving to that is not an option. In
addition, the existing rpmdb implementation is notoriously unreliable
as it's not transactional and has no other means to detect
inconsistencies either.
Changing to a more sustainable database implementation is long
overdue. We propose to change the default rpmdb format to the new
sqlite based implementation. Support for current BDB format will be
retained in Fedora 33, and phased out to read-only support in Fedora
34.
== Benefit to Fedora ==
* A far more robust rpm database implementation
* Getting rid of Berkeley DB dependency in one of the core components
== Scope ==
* Proposal owners:
** Once [[Changes/RPM-4.16|RPM 4.16]] lands and passes initial
shakedown, change the default rpmdb configuration to sqlite
** Address any bugs and issues in the database backend found by wider
testing base
** Help other developers to address Berkeley DB dependencies
* Other developers:
** Test for hidden Berkeley DB dependencies in other software, address
them as found and needed
* Release engineering: [https://pagure.io/releng/issue/9308 #9308]
* Policies and guidelines: Policies and guidelines are not affected
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
=== Upgrading ===
* Ability to upgrade is not affected
* After upgrade completes, manual action (rpmdb --rebuilddb) will
probably be needed to convert to sqlite. Alternatively user can change
configuration to stay on BDB.
=== Compatibility ===
* Container/chroot use-cases will be affected: older rpm versions will
be unable to query/manipulate the rpmdb from outside the chroot
* Koji/COPR may need to override the database format (back to) BDB for
the time being
== How To Test ==
* Rpmdb gets thoroughly exercised as a matter of normal system
operation, performing installs, updates, package builds etc
* Of specific interest here is torture testing: forcibly killing rpm
in various stages of execution - database should stay consistent and
operational (other system state is out of scope)
* Test database conversions from one backend to another (rpmdb
--rebuilddb --define "_db_backend <backend>")
== User Experience ==
* In normal operation, users should see little or no change
* Behavior in error situations is much more robust: forcibly killed
transaction no longer causes database inconsistency or corruption
== Dependencies ==
* This change depends on [[Changes/RPM-4.16|RPM 4.16]], support for
sqlite rpmdb is not present in older versions
* RPM will grow a new dependency on sqlite-libs
* Technically the rpmdb format is an internal implementation detail of
RPM and the data is only accessible through the librpm API, but some
software is making assumptions both about the format and/or in
particular, file naming. These are being tracked at
https://bugzilla.redhat.com/show_bug.cgi?id=1766120
* Upgrade tooling could/should perform rpmdb rebuild at end, this
would be a good thing to do regardless of this change
== Contingency Plan ==
* Contingency mechanism:
** Revert the default database back to Berkeley DB backend in the
package. Running 'rpmdb --rebuilddb' on hosts is currently required to
actually convert the database, but means to automate conversion in
specific conditions is being discussed upstream.
** The rpm-team does not expect problems with the database backend
itself, but we are aware that postponing may be needed due to
infrastructure or other tooling not being ready, primarily due to
inability to access the database from older releases.
* Contingency deadline: Beta freeze
* Blocks release? Yes
== Documentation ==
* [https://rpm.org/wiki/Releases/4.16.0 | RPM 4.16 release notes]
== Release Notes ==
* After upgrading from an older release, rpm operations will issue
warnings about database backend configuration not matching what's on
disk. Users should run 'rpmdb --rebuilddb' at earliest opportunity, or
change configuration to stay on Berkeley DB backend (eg 'echo
%_db_backend bdb > /etc/rpm/macros.db')
* The details are subject to change, the database rebuild may be done
by upgrade tooling
I'm glad to *finally* see this happen, so congratulations to the RPM
team for finally making this a reality! I look forward to trying this
out in Rawhide as soon as possible. 😊
FWIW, those who want an early taste, you can try my rpm-snapshot repo:
https://copr.fedorainfracloud.org/coprs/pmatilai/rpm-snapshot/
I run those snapshots on my own laptop at all times so it's not supposed
or expected to eat your disk or anything like that, but caveat emptor.
- Panu -
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
