Upgrade Path Test¶
scripts/upgrade-path-test.py checks that an install made by an older release
keeps working when its container is re-created from an image built from this
tree, the way Unraid users upgrade: same /config, same backup directory, new
image.
make test-upgrade # build this tree, upgrade from 0.2.4
make test-upgrade ARGS="--new-image arkive:dev" # use an image you already built
make test-upgrade ARGS="--keep" # leave everything for debugging
scripts/upgrade-path-test.py --help
Needs Docker and Python 3.9+ (standard library only). Run it before tagging a
release, and after changes to migrations (backend/app/core/database.py), the
keyfile guard, settings or secret storage, dump naming, job semantics, or how
restic is invoked. On GitHub it runs in the "Release tests" workflow
(.github/workflows/release-tests.yml): on pull requests that touch the image,
the schema, the dump and restore code or the version files, and on demand from
the Actions tab, upgrading from 0.2.4 and from the two releases before the
tree. Pushing a branch named release-test/<anything> runs it on that tree
without a pull request. Prefer that to a machine that runs other things: the test needs an image
built from the tree and either a privileged daemon or a Docker host of its own,
and on a busy host most of the time goes into copying the images into the
private daemon.
An old release that validates notification event names (0.4.0 and later)
refuses the retired events the 0.2.4 UI offered; the test then creates its
channel with backup.failed only and checks that the upgrade keeps it.
What it does¶
- Starts the old release (default
islamdiaa/arkive:0.2.4,--old-imageto change) as the Unraid template runs it: as root, with a small fake flash drive (go,ident.cfgand a marker file) mounted read-only at/boot-config, so Arkive detects Unraid and backs the flash up. It also gets a sample directory mounted read-only, a backup volume, and a throwaway Postgres container holding a marker row. - Completes setup through the old release's own API with a local target,
adds a notification channel with events the old UI offered, runs the full
backup job, and records the snapshot, the dump file, the flash archive
(0.2.4 wrote
.tar.gz), the.keyfilehash and the schema version. - Stops it and starts the new image on the same volumes.
- Checks that the new image:
- starts, with the keyfile guard accepting the old
.keyfile(unchanged) - applied every schema migration in its code, including the data migrations (the old channel gains the new failure events)
- accepts the old API key, keeps the run history, doesn't re-enter setup, and still detects Unraid
- lists the old snapshot from its cache and after a refresh from the repo
- restores files from the old snapshot byte-for-byte
- restores the old database dump, which then loads into a fresh Postgres with the original rows
- restores the old
.tar.gzflash archive, which holds the flash files - passes a restore drill on the old snapshot (written before dump manifests existed, so the drill infers its dumps)
- runs a new backup with status
successthat dumps the database, archives the flash, writes a new snapshot and keeps the old one; that dump restores with a row written after the upgrade - runs the other jobs the old setup created (
db_dump,flash) with statussuccess; only theflashjob archives the flash, and its new.tararchive restores with the flash files; then reportsstatus: ok - refuses to start when the
.keyfileis missing (so the guard really checks the old database), and starts again once it is back
Isolation¶
Arkive needs the Docker socket, and 0.2.4 discovers and dumps every database
it can see. By default the test therefore runs everything inside a private,
privileged docker:dind daemon that holds only the test containers, so the
real containers on the machine are out of its reach. The images are copied
into that daemon, which can take several minutes on a busy host.
--docker host uses the current daemon instead (faster, no privileged
container). It refuses to run when that daemon has any container the test
didn't create, so it is only for throwaway machines such as CI runners.
The API is published on 127.0.0.1:8310 (--port), and the container that
publishes it (the private daemon, or Arkive with --docker host) is on a
network of its own (<prefix>net). That matters because 0.2.4 leaves its API
open until setup: on the default bridge, other containers could reach it
through the container's own address. Processes on the host itself, including
host-network containers, can still reach it; 0.2.4's API is open to them only
until the test completes setup, a few seconds after it starts.
Test data lives in named Docker volumes, not host directories. Everything the
test creates is named with --name-prefix (default arkive-upgrade-test-)
and removed at the end, including the image it built (--new-image tests an
existing image and leaves it alone). Before creating anything, a run claims
the prefix by creating <prefix>net, which fails if the network exists; so a
second run with the same prefix stops instead of adopting (and later
removing) the first run's containers and volumes. Use a different
--name-prefix (and --port) to run two at once.
When it fails¶
The failing check is printed with FAIL:, followed by the last log lines of
the old and new Arkive containers. Rerun with --keep to inspect the
containers and volumes; afterwards remove them with
docker rm -f <prefix>dind && docker volume rm <prefix>dind-lib && docker network rm <prefix>net
(dind mode), or the <prefix>* containers, volumes and network (host mode),
and the <prefix>new:local image if the test built it. A run that was killed
before it could clean up leaves the same objects behind, and the next run
with that prefix refuses to start until they are gone.