Project requirements and setup
One major barrier to reproducibility is documenting and repeating the steps it took to get ready to run a project. If one relies too much on system-level dependencies, this can lead to reproducibility issues because a full system is hard to define with sufficient detail. Imagine trying to remember every change you've ever made to your computer and determining which will impact running your project!
Our goal is then to minimize system-level dependencies and configuration as much as possible, and what remains should be generally applicable to many projects, e.g., Docker, uv, and of course Calkit itself. Conversely, relying on system-wide installations of things like Python packages is a bad idea. For software libraries and tools more specific to a project, use environments.
Requirements can be declared in a project's calkit.yaml file
as a list in the requirements section,
and these will be checked before running the pipeline when
calkit run is called.
This way, when someone else tries to run your project,
they will be notified and can fix the issue before trying again,
which is more convenient than telling them to run through a
list of setup steps in a README.
A requirement can be an app, an environmental variable, a one-time setup step, or a constraint on the machine itself, like how many CPUs it has. Environmental variables are useful for configuration of a project that needs to be unique on each user's machine, which can also be used to avoid committing secrets to the repo.
Requirements are what has to exist outside a project's environments:
on the machine a stage runs on, e.g., an app installed system-wide or a
number of CPUs, or in the session there, e.g., an env var or a login.
Calkit builds and locks environments,
but only checks requirements, at most offering to install or set them up.
Where a requirement is declared says which stages need it:
the project's apply to every stage,
a machine environment's, i.e., a system, slurm or pbs environment's,
to every stage that runs on that machine,
and a stage's to that stage alone.
A stage's and its machine environment's are checked on the machine where
the stage runs, which is the one running calkit run unless the stage
runs in a machine environment with a host.
The project's are checked on the machine running calkit run.
Note
This section used to be called dependencies, and that name still
works, with a deprecation warning. It was renamed because half of
what belongs here isn't something you can install: a CPU count or an
amount of memory is required, not depended upon. Set one key or the
other; setting both is an error, since they'd be two places for the
same list to drift apart.
The example below, taken from
this project
shows both a unique configuration variable (STRAVA_CLIENT_ID)
and a secret (STRAVA_CLIENT_SECRET)
that allow a different user to use copy and reuse the project without
changing anything.
requirements:
- docker
- kind: env-var
name: STRAVA_CLIENT_ID
notes: >
The STRAVA_CLIENT_ID and STRAVA_CLIENT_SECRET environmental
variables can be set in the .env file after creating a Strava
application at https://www.strava.com/settings/api
- kind: env-var
name: STRAVA_CLIENT_SECRET
As we can see in the notes for STRAVA_CLIENT_ID,
a .env file, which is kept out of version control,
can be used to define these variables.
calkit set-env-var can be used as a shortcut to set one of these
in lieu of directly editing .env.
calkit check env-vars can also be run to check for missing variables,
prompting the user for their values and setting them in .env.
Additional (non-secret) environmental variables can be set at the project
level in calkit.yaml in the env_vars map:
These will then be set for calls to the run and xenv commands.
When calkit run (or calkit check reqs) encounters a missing
env-var requirement in an interactive shell, it prompts the user
for a value, writes it to .env, and exports it for the rest of the
run.
calkit status does the same, except that a variable left unset is
reported rather than treated as fatal, since status reports rather than
enforces.
calkit check reqs is also available as calkit check requirements,
and, under the key's old name, as calkit check deps and
calkit check dependencies.
A per-variable default may be declared so that pressing Enter accepts
the default:
In non-interactive contexts (CI), the same missing variable still raises a clear error so failures aren't silent.
Machine requirements
It is possible to define requirements of the system on which the project is being run, e.g., how many CPU cores, how much memory, etc. For example:
requirements:
- kind: cpu-count
min: 8
- kind: memory-gb
min: 32
- kind: os
equals: [Linux, Darwin]
- kind: calkit-python-version
version_spec: ">=3.11"
cpu-count and memory-gb take min and/or max.
Everything else takes equals---matched case-insensitively, and a list
means any one of them will do---matches, a glob like *.cluster.edu
that is matched the same way, or a version_spec for properties that
are versions.
The properties available are the machine-describing ones from the table
in environments;
versions of installed tools are reached as an app requirement with a
version_spec instead, which is one way to say it rather than two.
An entry that constrains nothing is rejected.
If you don't want to constrain a property but do want stages to rerun
when it changes, that's what a system environment's
lock is for:
requirements gate, locks pin.
Checking these means reading the machine, which is what
calkit describe system reports.
A property the machine doesn't report is an error rather than a silent
pass---an unanswerable question isn't a satisfied one.
Stage or environment specific requirements
Requirements in the project's requirements section apply to the whole
project, so a machine that doesn't meet them can't run anything.
A requirement that only some stages have can go on those stages, or on
an environment that describes a machine, i.e., a system, slurm or
pbs environment, which applies it to every stage using that
environment.
Other environments, e.g., conda or uv-venv, describe what's
installed rather than the machine, so they can't have requirements;
pair one with a machine instead, as in cluster:py:
environments:
ml:
kind: system
requirements:
- matlab
cluster:
kind: system
requirements:
- kind: hostname
matches: "*.cluster.edu"
py:
kind: uv-venv
path: requirements.txt
pipeline:
stages:
fetch:
kind: python-script
script_path: scripts/fetch.py
environment: cluster:py
requirements:
- kind: env-var
name: DATA_TOKEN
On a machine where a stage's requirements, or its environments', aren't
met, calkit run skips that stage rather than failing.
The same goes for a stage whose switch environment picks nothing
there, or whose system environment is locked to another machine.
A slurm or pbs environment without a host requires its
scheduler's sbatch or qsub on this machine.
Its outputs are kept, so stages that use them still run.
A stage that needs an output a skipped stage hasn't made, e.g., one
made on a cluster and not pulled, is skipped too, as are the stages
after it.
It stays out of date in calkit status, which says why it can't run
there, and the run ends by listing the out-of-date stages it skipped.
Requirements of an environment on another machine, e.g., a system
environment with a host, are checked on that machine by calkit run.
calkit status and calkit check reqs don't connect to other machines,
so they don't check those.
On a terminal, before skipping a stage for requirements this machine
doesn't meet, calkit run offers what's missing if the stage would run,
i.e., it's out of date, follows one that is, or is forced,
the same way it does for the project's requirements:
installing an app, running a setup command, or asking for a variable.
A stage that's up to date is never asked about,
and a requirement many stages share is asked about once per run.
This comes after the environments of the other stages are checked,
since checking one can put a stage out of date.
The environments of a stage whose requirements weren't met can't be
checked until they are, so a stage is judged out of date on what they
last recorded, and they're checked once its requirements are met.
If that puts another stage using them out of date, it's offered next.
Declining, or an install that fails, skips the stage as before,
as does running without a terminal, e.g., in CI.
calkit check reqs checks everything a machine needs to run the whole
project: the project's requirements, and those of every stage and
environment, except a switch's options it doesn't pick here,
reporting each one that isn't met by where it's declared.
With --stale, it only checks what calkit run needs now:
the project's requirements, and those of the stages calkit status
lists as out of date and of their environments.
Both cover subprojects too.
Some skips still fail the run, after the stages that can run have run:
- One of the stages named on the command line, e.g.,
calkit run fetch, is out of date and can't run. - A stage is out of date and can't run because something is wrong rather than because the project says so, e.g., its machine can't be reached.
- Any stage is out of date and can't run, with
calkit run --strict, e.g., in CI for a project whose stages should all run there. Thecalkit/runaction takes it asrun_args: --strict.
A stage that iterates can use its values
in its requirements, so each item only needs what it uses,
and an item whose requirements aren't met is skipped on its own.
Quote a value in braces, since YAML reads {engine} alone as a mapping:
pipeline:
stages:
round-trip:
kind: python-script
script_path: scripts/round-trip.py
args:
- --engine={engine}
iterate_over:
- arg_name: engine
values: [word, libreoffice]
requirements:
- "{engine}"
Calkit finds Microsoft Word and LibreOffice where they install,
so word and libreoffice work as app requirements.
Pinning the Calkit CLI version
A project can declare which version of the Calkit CLI it needs by
listing calkit itself as a requirement with a
PEP 440 version specifier:
Equivalent flat-dict form, useful when you want to add a notes field:
If the running CLI doesn't satisfy the spec, calkit run aborts with
a fix-it message pointing at two options: re-run against a pinned
version using --use-version (see below), or upgrade in place with
calkit upgrade.
Running against a specific Calkit version
Use the top-level --use-version flag to re-invoke the CLI under a
specific calkit-python release without changing your installation.
This is the easiest way to bring a clone up to a working baseline:
Under the hood this re-execs as:
so it requires uv (specifically uvx)
on PATH.
You can pass a bare version (0.38, treated as an exact pin) or a
PEP 440 specifier (>=0.38, ==0.38.1, etc.).
Arguments after --use-version <ver> are forwarded to the child
process verbatim;
use -- if you need to pass through a flag that would otherwise be
parsed by the parent (e.g. calkit --use-version 0.3 -- --version).
Auto-installing apps
For a set of well-known apps, Calkit ships with a registry of
upstream installers and can offer to run them when the app is missing.
On an interactive terminal calkit run (and calkit check reqs)
will prompt before installing;
in CI the same path prints the install command as a fix-it and exits
non-zero.
Installing is always opt-in: nothing runs without a yes at the prompt
or --yes on the command line.
Apps currently in the registry:
| Name | Installer |
|---|---|
pixi |
curl -fsSL https://pixi.sh/install.sh \| sh (and PowerShell on Windows) |
uv |
curl -LsSf https://astral.sh/uv/install.sh \| sh |
rustup, cargo |
upstream rustup script on Unix, winget on Windows |
juliaup, julia |
upstream juliaup script on Unix, winget on Windows |
nix |
Determinate Systems installer on Unix; WSL2-only on Windows |
conda, mamba |
Miniforge in batch mode, into ~/miniforge3 |
git |
Homebrew on macOS, winget on Windows |
docker |
Homebrew cask on macOS, get.docker.com on Linux, winget on Windows |
colima |
Homebrew on macOS, with the Docker CLI unless one is already installed |
code |
Homebrew cask on macOS, winget on Windows |
R, Rscript |
Homebrew cask on macOS, winget on Windows |
node, npm, npx |
pixi global install nodejs, so pixi first; also calkit install nodejs |
brew |
the upstream Homebrew script, macOS only |
choco |
the upstream Chocolatey script in an elevated shell, Windows only |
Run calkit list installers for the live list.
On Linux, Git, VS Code, and R come from the system package manager,
so those entries print what to run rather than guessing which one.
Colima isn't needed on Linux, where Docker runs natively,
so there it points to Docker Engine instead,
and it doesn't run on Windows, so there it points to Docker Desktop.
After installing it, colima start starts its VM.
Some entries depend on another, e.g., Git on macOS is installed with Homebrew. Those prerequisites are prompted for first, one at a time, so saying yes to one thing never silently installs another.
You can also trigger an install directly:
calkit install pixi # prompts before running
calkit install pixi --yes # non-interactive (scripts, CI provisioning)
After a successful install, Calkit prepends the installer's known
output directory (e.g. ~/.pixi/bin) to PATH for the current
process, so the very next requirement check sees the new binary
without requiring a shell restart.
Every app Calkit installs is recorded in ~/.calkit/installed.json,
with the date and the command that was run,
and calkit list installers marks those entries.
That way there's always a record of what Calkit changed on the machine.
A project can say how to install an app itself with a setup_command,
which is offered the same way,
in place of any installer in the registry,
e.g., for an app the registry doesn't have:
requirements:
- kind: app
name: gh
setup_command: brew install gh
description: Downloads the raw data from the lab's GitHub releases.
A missing app's description, if it has one,
is shown when Calkit asks to install it and in the error if it's still
missing.
Setup requirements
Some preconditions aren't files or environment variables---they're
one-time per-machine actions like gh auth login or
huggingface-cli login.
The setup requirement kind captures these declaratively:
requirements:
- pixi
- kind: setup
name: Authenticate GitHub CLI
check_command: calkit xenv -n analysis -- gh auth status
setup_command: calkit xenv -n analysis -- gh auth login
description: >
Authenticate the GitHub CLI, which is used by the data-fetching
stage of the pipeline. Run this once per clone; the pipeline can
then pull data from GitHub without further prompts.
Each setup requirement declares:
check_command: a shell command whose exit code determines whether the requirement is satisfied (exit0= satisfied). Required.setup_command: optional. On an interactive terminal Calkit asks before running it; in CI Calkit prints it as a fix-it and aborts.description: optional human-readable explanation; used in error messages andcalkit list.name: optional. If omitted, Calkit synthesizes a stable name from a hash ofcheck_command, so anonymous setup steps still get a predictable identifier in error messages.cache_ttl: optional. See below.
To run a check_command or setup_command inside one of the project's
environments, prefix it with calkit xenv -n <env> -- explicitly --
there is no implicit wrap, because explicit is easier to debug when a
command fails.
Caching setup checks
Probing a network-bound check (like gh auth status) on every
calkit run is wasteful and slow.
Calkit caches successful setup checks under
.calkit/local/dep-checks.sqlite (which is .gitignored) for one day
by default.
The cache invalidates automatically whenever the check_command itself
changes, so editing calkit.yaml never silently relies on a stale
"passed" result.
To override the TTL for a single requirement, set cache_ttl to a duration
string (30s, 5m, 2h, 7d, 1w) or a bare integer number of
seconds.
cache_ttl: 0 disables caching for that requirement:
requirements:
- kind: setup
name: AWS credentials are valid
check_command: aws sts get-caller-identity
cache_ttl: 1h # re-check at most hourly
- kind: setup
name: Daily license check
check_command: ./scripts/check-license.sh
cache_ttl: 0 # always re-probe
To force a re-probe of every cached setup requirement on a single
invocation, pass --no-cache:
Ordering and the requirement flow
Within a single calkit run or calkit check reqs, Calkit processes
requirements in four phases regardless of the order they appear in
calkit.yaml:
- machine properties: First, because a machine too small to run the project at all should say so before anything is installed on it.
env-var: Prompted next so installers and setup steps can read newly-set variables.app: Environment managers likepixianduvmust exist before any setup step that runs inside one of those environments. Missing apps with asetup_commandor a registered installer trigger the auto-install prompt here.setup: Last, sincecheck_commandtypically wrapscalkit xenvand depends on both apps and env vars.
This ordering means a single fresh-clone calkit run can prompt for
secrets, install pixi, build the environment, and authenticate the
GitHub CLI without any manual setup steps in between.