PyRPM Tool descriptions
=======================

In order to understand what each of the tools really does this section
describes the behaviour of the tools in more detail for each tool.


pyrpmcheck
----------

There are currently 4 basic tests in pyrpmcheck. The first checks for all
packages if any two dirs differ in user/group/mode. The second check looks for
duplicate provides, meaning if 2 rpms provide the same thing. The third test
looks for dangling symlinks, that is, symlinks that point to nowhere. The
last check simply checks if any package names appear in the various given dirs.
This check is only useful for FC and FC-Extras, as there should be no overlap
there.

Check some common problems with packages and interdependencies for
Fedora Core Devel:
-----------------------
pyrpmcheck /mirror/fedora/development/i386/os/Packages
-----------------------


pyrpmcheckinstall
-----------------

In order to test problems with installs or removals this tool can be used to
test a whole repository against missing prereqs, failing scripts and other
common problems. It does so by iterating over all packages of the given
repos and installing and removing them just like a yum install <package>/yum
remove <package> would have done. Doing this in an empty buildroot allows one
to find missing dependencies and other script failures easily.

Check a complete FC4 repository:

-----------------------
pyrpmcheckinstall -y -v -c fc4-i386.conf -r /mnt/build/test &> /tmp/installtest
-----------------------

To only get the errors and have a minimal log make it quiet:

-----------------------
pyrpmcheckinstall -y -q -c fc4-i386.conf -r /mnt/build/test &> /tmp/installtest
-----------------------

To work on a directory use this:

-----------------------
DEST=/mnt/build/test
for d in dev dev/shm dev/pts proc sys ; do
  mkdir -p $DEST/$d
  mount --bind /$d $DEST/$d
done
mkdir -p $DEST/etc
touch $DEST/etc/fstab
pyrpmcheckinstall -y -c /dev/null -r $DEST /mirror/fedora/development/i386/os/Packages &> /tmp/installtest
for d in sys proc dev/pts dev/shm dev ; do
  umount $DEST/$d
done
# grep for "ERROR" in the logs
-----------------------

If you have a multiple CPUs and a lot of ram you can run the whole test in a
tmpfs and store the restults in logfiles like this:

-----------------------
TOP=/mnt/build/tmpfs
NAME=fc-extras-devel-i386
DEST=$TOP/$NAME
NUMCPU=8
MEM=7G
mkdir -p $TOP
mount -t tmpfs -o size=$MEM none $TOP/
for i in `seq 0 $NUMCPU`; do
  D=$DEST.$i
  for d in dev dev/shm dev/pts proc sys ; do
    mkdir -p $D/$d
    mount --bind /$d $D/$d
  done

  mkdir -p $D/etc
  touch $D/etc/fstab
done
pyrpmcheckinstall --ignoresize -y -v -c /root/$NAME.conf -r $DEST numcpu=$NUMCPU logfile=/var/tmp/$NAME.log
for i in `seq 0 $NUMCPU`; do
  killall pyrpmcheckinstall
  sleep 1
done
for i in `seq 0 $NUMCPU`; do
  D=$DEST.$i
  for d in sys proc dev/pts dev/shm dev ; do
    umount $D/$d
  done
done
umount $TOP

# grep for "Error" in the logs
-----------------------


pyrpmcheckrepo
--------------

This tool should actually be renamed (and will so soon) to pyrpmcheckinstall as
that's what it does. What the tool does it to simulate an installation (just
like you could do with `pyrpmyum \--test`) and then simulates an update to the
second set of rpms or repository. A typical test would be to use a FC-3
or FC-4 yum.conf repo as a install base and then use a FC-devel repo or
rpm tree as the update tree. The nice thing about the simulation is that it
doesn't actually perform the installation and so doesn't need any disk space.
Additionally you can use the (undocumented ;)) option `\--arch` where you can
specify any arch you like. The host arch doesn't matter for the simulation
as the whole code "thinks" it runs under arch. So simulated installs and
updates for all kinds of archs can be performed this way on one single machine.

Check if an update from Fedora Core 4 to Fedora Core Devel works or what
problems come up:
-----------------------
pyrpmcheckrepo -r /mnt/foo install /mirror/fedora/4/i386/os/Fedora/RPMS update /mirror/fedora/development/i386/os/Packages
-----------------------


pyrpmcreaterepo
---------------

Using the original createrepo as a basis we integrated our pyrpm code into it
instead of using rpmlib.

Create a repo for FC-4:

-----------------------
pyrpmcreaterepo /mirror/fedora/4/i386/os/
-----------------------


pyrpmdbconvert
--------------

Converts one rpm database to another. The 2 currently supported databases are
rpmdb (DB4 based) and a SQLite rpm database. It can be used to either convert
a rpm database from one format to another or to recreate a rpm database.

Convert your system rpmdb into a sqlite database:
-----------------------
pyrpmconvertdb /var/lib/rpm sqlitedb://var/lib/rpm.sqlite
-----------------------

Rebuild your database:

-----------------------
pyrpmconvertdb /var/lib/rpm /var/lib/rpm.rebuild
-----------------------



pyrpmdbverify
-------------

This tool actually really reads all DB4 files in /var/lib/rpm (or, if you use
the `\--dbpath` option, in that directory) and tries to cross check the entries
with the rpm header that is found in Packages. The pyrpm-devel documentation
contains a lot more detail about the structure of
link:pyrpm-devel.txt#rpmdb[rpmdb]. Simply put we check if for every entry in
every file there is a
corresponding entry in Packages and vice versa. There are some exceptions and
special cases, but all known ones are handled correctly. The one check that is
very unreliable is the SHA1 header checksum tests as this test has to rely on a
very specific ordering of the rpm binary header and about the tags that appear
in the original rpm. If those assumptions don't hold this checksum can't be
computed correctly.
Additional binary packages or a yum config file with yum repositories can be
specified to test the rpmdb headers against those packages for additional
verification.

Link to a page with more information on rpmdb repair information is available
at link:http://www.rpm.org/hintskinks/repairdb/[].

Check the rpmdb of the current system with internal cross checks
-----------------------
pyrpmdbverify
-----------------------

Same, but this time we use external rpms to verify against, too
-----------------------
pyrpmdbverify /mirror/fedora/development/i386/os/Packages
-----------------------

Verify our rpm database in our FC-4 buildroot using external rpms as well
-----------------------
pyrpmdbverify --dbpath /mnt/build/fc4-i386 /mirror/fedora/4/i386/os/Fedora/RPMS
-----------------------

Verify our rpm database in our FC-4 buildroot using external rpms using a yum
repository:
-----------------------
pyrpmdbverify -c fc4-i386.conf --dbpath /mnt/build/fc4-i386
-----------------------


pyrpminstall
------------

Simply put this tool allows you to install, update and erase rpms just like
you would with rpm itself. It supports most of the common options from rpm
and is mostly using the same syntax, too. As it's "only" rpm though there is
no depresolver in this tool, so if you have any missing dependencies you'll
have to fiddle them out yourself, just as you had to do with rpm.


pyrpmkickstart
--------------

Usage: pyrpmkickstart <options> <kickstartfile>
                      [[<disk name>:]<disk image>|<disk device>]*

This tool can install Red Hat Enterprise Linux (>=3) or Fedora Core
distributions on hard disks, partitions or disk images. pyrpmkickstart uses
a kickstart file for setup and installation of the system. Here is a very
simple kickstart file:

----------------------------------------------------------------------------
lang en_US
langsupport en_US.UTF-8
keyboard us
mouse
timezone EST
rootpw test
install
nfs --server=server --dir=/dist/fedora/core/4/i386/os
#url --url=file:///dist/fedora/core/4/i386/os
bootloader --location=mbr
#zerombr yes
#clearpart --all --initlabel
partition swap --size=100
partition /boot --fstype=ext3 --size=100
partition / --fstype=ext3 --size=1 --grow
auth  --useshadow  --enablemd5
network --bootproto=dhcp --device=eth0
firewall --enabled --ssh
selinux --enforcing
%packages --resolvedeps
@base
@x
----------------------------------------------------------------------------

You can create kickstart files with "redhat-config-kickstart" or
"system-config-kickstart" according to your needs.

In the kickstart example, pyrpmkickstart will create new partitions in an IDE
hard disk or disk image. If you want to use existing partitions, then append
"--usepart=/dev/hdaX" to the partition entry. If a partition should not be
formatted, then add "--noformat".

The source has to be a installable tree. It is not possible to install from
iso images. There are currently two supported installation methods:
NFS install and url-file install. Contrary to the kickstart documentation, it
is possible to install from a local installation tree with
"url --url=file://path", where path is a full path name with a leading slash.

The destination is either a hard disk, a partition on a hard disk or a
complete disk image (usable e.g. with XEN, QEMU, VMware). If you are using a
hard disk, please be careful with "zerombr" and "clearpart" in the kickstart
file. These will remove some or all partitions on this hard drive according
to the usage. Do not use them, if you want to install on a single partition
on your hard drive. You would have to install the bootloader into the root
partition of your new install as well: "bootloader --location=partition".

A disk image is similar to a real hard disk for the installer. It assumes,
that it is a IDE hard disk and maps it to hda. Qemu emulates a disk image as
a IDE hard drive internally, so you can use it as is. If you want to use sda
instead, you have to map the disk image with "sda:disk.img". This is essential
for installation in vmware disk images, because vmware defaults to SCSI disks.
Please keep in mind that you have to preallocate the hard disk in vmware to be
usable with pyrpmkickstart. Here is the command to easily generate a 2G disk
image:

        dd if=/dev/zero of=disk2G.img bs=1M count=2048

Disk image installations are usable by XEN, QEMU and VMware. Please remember
to increase loop device amount for large disk image installs in
/etc/modprobe.conf:

        options loop max_loop=64

Here is the command to install into the disk image "disk2G.img" with the
kickstart file "ks.cfg":

        pyrpmkickstart ks.cfg disk2G.img

Here is the command to use a disk image with QEMU.

        qemu -m 256 disk2G.img

The default virtual memory size in qemu is 128M, therefore it is advisable to
increase it.

Say you have just completed a normal kickstart installation with qemu.
Create a second IDE disk and check if your kickstart file can be used
to install onto the second IDE disk with:

        pyrpmkickstart --no-stage2 ks.cfg hda:/dev/hdb

pyrpmkickstart first downloads all rpm packages before the installation
begins. This can be changed with the option "--no-cache".

pyrpmkickstart now has SELinux support. You can install a SELinux guest even if 
your host system has no SELinux support. This is done via a autrelabeling on
first boot of the guest.


Fedora Core 7 and later:

If you need ata disk drive support, you have to add a device entry for the
driver to the kickstart file:

    device scsi ata_piix


pyrpmrandomizer
---------------

This is our main stress test tool. With it you can run literally thousands of
update and erase operations in a buildroot and after several days scan the
output for any problems. It works simply by alternating between trying to
randomly install one rpm package (including needed dependent rpms) and
randomly removing one package. There are several special options which help
circumvent some strange problems. The first is `\--servicehack`
which simply changes `/sbin/service` to "exit 0" as especially some of the
postuninstall scripts really misbehave otherwise and you don't need any
services started or stopped in a buildroot anyway for testing. The second is
the `\--autoerase` option which I'll describe a little more in detail for
pyrpmyum.

Run 10000 install and erase operations in the /mnt/build/test buildroot and
save the resulting output in /tmp/stresstest
-----------------------
pyrpmrandomizer --servicehack -y -v -c fc3-updates-i386.conf -r /mnt/build/test 10000 &>/tmp/stresstest
-----------------------


pyrpmrebuilddb
--------------

Ever had to use `rpm \--rebuilddb`? Well, here is our version of it. It tries
to do the same thing as rpm where it takes /var/lib/rpm/Packages and rebuilds
the database with that information. The default path for the new db is ./rpmdb
so that your original one won't be overwritten.
Currently `/var/lib/rpm/Pubkeys` is not yet rebuild.


pyrpmspecinfo
-------------

As we often have to work with specfiles, too, we've written a small tool with
which we can extract the different sections from a specfile in order to use
that info via pipe in other tools.


pyrpmverify
-----------

A tool to verify installed rpm packages. Command-line arguments specify
packages to verify. When no arguments are specified, all installed packages are
verified.

With the `--diff` option `pyrpmverify` attempts to download the original
packages and outputs diffs of the changed files. Changes will be shown for
all nonempty regular files which are marked as `%config`.

Some packages have content verification explicitly disabled for some files
(e. g.  `/etc/nsswitch.conf` in FC5 glibc-2.4-4), you can use the
`--verifyallconfig` option to force verification (and diff output) for such
files.

Multilib installations are currently untested and SELinux context testing
depends on link:https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188106[].


pyrpmyum
--------

Our main workhorse and real puppy. This tool, as the name already suggests is
basically a yum replacement of sorts. It uses the same command line options as
yum and uses the same config files, too. It contains a complete depresolver
and allows complex installs, updates and erases from systems. We have used it
to install FC1 and then step by step update to FC1-updates, FC2, FC2-updates,
FC3, FC3-updates and finally FC4. Some of the updates need the `\--autoerase`
option though because some dependencies just can't be met and packages need
to be removed in order to fulfill them. The autoerase option tries exactly
that by semi-intelligently selecting the package that needs to be removed in
order for the update to work properly without user intervention. For the
described update chain only about 6-7 packages were deleted up to FC3, and all
of them would have to be removed manually anyway. But a word of warning here:
The autoerase option might run amok if things go really wrong or you try to
update to a completely broken tree and would then try to remove almost all of
your packages. This has never happened for real trees for us, but just imagine
a broken repository and things could go really wrong. That's why this option
isn't directly listed in the help and should only be used for a `\--test` run
or without the -y option where you get listed separately from the "normal"
transaction the packages that pyrpmyum would autoerase for you.

Install a full Fedora Core Development tree in `/mnt/build/fc-devel-i386`
buildroot:
-----------------------
pyrpmyum -y -v -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 install "*"
-----------------------

Update kdebase in `/mnt/build/fc-devel-i386` buildroot to latest version,
resolving all dependencies:
-----------------------
pyrpmyum -y -v -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 update kdebase
-----------------------

Same, only if we get some unresolvable problems try to remove packages that
don't fit automatically:
-----------------------
pyrpmyum -y -v --autoerase -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 update kdebase
-----------------------

Make a dry-run of the above command to be sure that not everything gets
automatically deleted :)
-----------------------
pyrpmyum -y -v --test --autoerase -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 update kdebase
-----------------------

Remove glibc and all depending packages from our buildroot :)
-----------------------
pyrpmyum -y -v -c fc-devel-i386.conf -r /mnt/build/fc-devel-i386 remove glibc
-----------------------


oldpyrpm.py
-----------

This script does not depend on the rest of the pyrpm module, but is all
copied into one file. It is not as complete as the new pyrpm code, but
can still be used for many use cases. oldpyrpm.py only depends on
python-2.2 or newer and you can use parts of it even if one of the addon
python modules are not available. Also feel free to check the script for
more hidden usage cases.


Diffing Source Rpms
~~~~~~~~~~~~~~~~~~~

You can list the changes between two source packages. If you specify
`\--explode` it tries to also show the changes within `\*.tar.gz` files
if they have changed.
-----------------------
oldpyrpm.py [--explode] --diff 1.src.rpm 2.src.rpm
-----------------------


Extracting Rpm Data
~~~~~~~~~~~~~~~~~~~

You can extract source or normal binary packages. This always overwrites
existing files and does not start any scripts in binary packages, so it
is often only used for source packages:
-----------------------
oldpyrpm.py [--buildroot=/chroot] --extract *.rpm
-----------------------


Excluded Arch Checking
~~~~~~~~~~~~~~~~~~~~~~

You can check source rpm packages and mark the ones which are excluded from
building on certain architectures.
-----------------------
oldpyrpm.py --checkarch /mirror/fedora/development/source/SRPMS
-----------------------


Checking rpmdb in /var/lib/rpm
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can check the rpm database `rpmdb`. The complete information that rpm
needs is included in `/var/lib/rpm/Packages` and the opengpg data in
`/var/lib/rpm/Pubkeys`. All other db4 files in the rpmdb directory contain
duplicate data from `Packages` to speed up information lookup.

This is how the rpmdb is being checked:

 - Verify all data from `/var/lib/rpm/Packages` is the same as in all other
   db4 files (two-way check).
 - Most of the rpm header data does have a sha1 checksum. For most newer rpm
   packages in rpmdb we can try to verify that sha1 to be correct.
 - If scripts fail often duplicate rpm packages stay installed. Apart from
   kernel packages and a few other special packages, you get a warning about
   all duplicated packages.
 - Check all dependency information for `Requires:`, `Provides:`,
   `Conflicts:` and `Obsoletes:` is correctly matching up and if the
   installed rpm packages do have any fileconflicts.
 - If you specify a yum configuration file, all installed files are checked
   against the yum repositories. Which rpm packages do not match, which
   installed packages are not available in one of the repos. You can specify
   more than on yum config file by repeating the `-c` option.
 - If installed rpms are compared against a repository, then a check is done
   if pyrpm would also write the same rpmdb entry to compare if our functions
   work the same as original `/bin/rpm` ones.
 - Only rpmdb files in `/var/lib/rpm` are accessed read-only, no actual
   installed files from the rpm packages are verified.
   If checks against repositories are done, those are also read-only and
   all tests can be run as non-root user.
 - The check can be run on any architecture for any other architecture with
   automatic detection of the right endian setting of the db files as well
   as automatic arch setting by looking at the installed kernels in the rpmdb.
   So you can copy the rpmdb files onto a central server to check them all.
 - Check if only packages with the correct architecture are installed. Make
   sure no arch-dependent rpm and a noarch package are installed at the
   same time. Warn if we have more than one package installed for any
   given name/arch pair (with the exception of "installonly" packages like
   the kernel). Also warn if we have more than one package with the same
   name that has different epoch/version/release. This is checking multilib
   (also called compatarch) installations for some additional sanity.
 - `-q` and `\--quiet` output less progress information. `-v` or `\--verbose`
   add more output information. `\--rpmdbpath /var/lib/rpm/` can specify the
   directory which contains the database files. `--nofileconflicts` disables
   the checks for fileconflicts.
 - For special setups you might want to specify `\--releasever 4`,
   `\--arch i686` or `\--buildroot /mnt/build` (buildroot is actually only
   used to read the yum config files.

Example usages to check your rpm database `rpmdb`:
-----------------------
oldpyrpm.py [--verbose|-v|--quiet|-q] [--rpmdbpath=/var/lib/rpm/] --checkrpmdb
oldpyrpm.py --checkrpmdb [--enablerepos] [--fileconflicts] [-v]
-----------------------


Createrepo
~~~~~~~~~~

You can generate repo metadata like `createrepo` does. You need to copy the
`comps.xml` file yourself into the repodata directory if you need one.
Also xml pretty printing is hardcoded (`-p` option for upstream createrepo).
Createrepo files are compatible with the output from createrepo-0.4.3.
-----------------------
cp comps.xml /mirror/fedora/development/i386/repodata/comps.xml
oldpyrpm.py --createrepo /mirror/fedora/development/i386/
-----------------------




Package Verification
~~~~~~~~~~~~~~~~~~~~

Per default `oldpyrpm.py` is only reading in rpm packages and doing some
sanity checks on them. It can read in rpm packages from Red Hat Linux 5.2
and newer. The `pyrpmyum` code only works on RHEL3 or newer due to how
missing epoch are treated in dependency information.
Here some options that can be useful:

 - If you specify a directory, automatically all files in that dir are read.
 - `\--nodigest` skips verifying the md5 checksum of the header and payload
   data and the sha1 checksum of the header data.
 - `\--nopayload` skips reading in the cpio data of all files
   (called `payload`).
 - Add `-c /etc/yum.conf` to read in yum repositories.
 - `\--strict` is adding checks which are only useful for the Fedora Core
   development tree.
 - `\--verbose` or `-v` increases output logging information. This is nearly
   unused in the current code, so doesn't really change anything.
 - `\--quiet` or `-q` sets the verbose level to 0.
 - Specify a certain architecture via `\--arch i686`. Default value is detected
   via uname(2).
 - To override the release version information add `\--releasever 4`.
   The default is the version number of the installed redhat-release or
   fedora-release package (used rpm packages can be specified via
   `\--distroverpkg="redhat-release fedora-release"`).
 - `-y` to never prompt for yes/no questions. This is unused until now.
 - `\--noverify` is skipping the special debug tests and then mainly all
   rpm packages are only read in. If also `\--small` is specified, not all
   rpm tags are read in, only the ones in `importanttags`.
 - Specify a cache directory to hold data from yum repositories via
   `\--cachedir ~/.pyrpm/cache/`.
 - Dependency and fileconflict checking is done if either `\--strict` or
   `\--checkdeps` is added. `\--nofileconflicts` is disabling fileconflict
   checking.
 - `\--runorderer` is ordering the set of rpms according to their dependencies
   in a way they should get installed.

Example usages:
-----------------------
find /mirror/ -name "*.rpm" -type f -print0 2>/dev/null | xargs -0 oldpyrpm.py --nodigest --nopayload
locate '*.rpm' | xargs oldpyrpm.py --nodigest --nopayload
oldpyrpm.py --strict --nodigest --nopayload --fileconflicts --completerepo /mirror/fedora/development/{i386,source}
-----------------------

Only read in the needed rpm tags and do dependency checking without further
debug checks. This shows the full speed of the current implementation and
finishes in under 14 seconds for FC-development on some laptop:
-----------------------
time oldpyrpm.py --checkdeps --small --nodigest --nopayload --noverify /mirror/fedora/development/i386/os/Packages
time oldpyrpm.py --checkdeps --small --nodigest --nopayload --noverify /mirror/fedora/4/i386/os/Fedora/RPMS /mirror/fedora/updates/4/i386 /mirror/fedora-extras/4/i386
-----------------------

