Environments

A computational environment describes the necessary conditions for code to run properly. Ensuring that every stage in your pipeline is run within a defined environment is a great way to improve reproducibility. Typically these will be created inside the global system environment, where the system environment has an environment manager installed. Because the system environment itself is somewhat of a "primary artifact" (cf. reproducibility), i.e., it is manually manipulated by the user, we want to minimize its uniqueness or changes to it, since there's a risk we may not properly document its state. This means we want to limit foundational dependencies to very common tools like Git, common shells like Bash, and environment managers.

There are many different environment management tools out there to choose from, and Calkit attempts to provide a similar interface for all of them. Calkit also attempts to enforce their usage in such a way that all important information about the environment is captured locally in the project in so-called "lock files." This way, the project can be moved to other machines without needing to worry about manually installing packages.

Calkit provides a means for defining or declaring environments inside a project's calkit.yaml file. There is also a command line utility calkit xenv for executing a command in one of these, which ensures that the environment matches its specification before execution.

Environment types and definitions

Calkit supports the following environment types:

• Docker
• Conda
• venv (included in the Python standard library)
• uv (both venv and project-based)
• Pixi
• renv
• Julia
• MATLAB
• Nix (flake-based)
• SLURM
• system (the machine itself, local or reached over SSH)

Environment definitions live in the project's calkit.yaml file in the environments section. Most environments will have a path property pointing to a file that lists the necessary dependencies--the "spec." For example, a Python virtual environment or "venv" can be defined as a simple list of dependencies in a requirements.txt file, which might look like:

pandas>=2
polars==0.17.1
matplotlib

Automatic detection

For common environment types, Calkit will register a new environment upon its first use with calkit xenv. For example, if you run:

calkit xenv python scripts/run.py

Calkit will attempt to find an environment spec, create the environment, save it in calkit.yaml, export a lock file, and run the command in that environment. If there are multiple env specs, e.g., requirements.txt and environment.yml, you can provide the path, e.g.,

calkit xenv -p environment.yml python scripts/run.py

and Calkit will use that one to create the environment and save the path in calkit.yaml.

Checking, syncing, and executing

An environment can be checked that it matches its specification with:

calkit check env --name {env-name}

This will produce a "lock file" (inside the project's .calkit/env-locks directory if the environment manager doesn't export lock files by default), which uniquely identifies the actual environment that was created to help diagnose reproducibility issues down the road.

A command can be executed in an environment with:

calkit xenv --name {env-name} -- {command}

Before the command is executed, Calkit will check that the environment matches its specification, and if it needs to be updated, that will be done before execution.

All project environments can be checked at once with:

calkit check envs

Inspecting environment paths

To see where an environment's specification and lock file live, use:

calkit describe env --name {env-name}

Adding --json prints the same information as machine-readable JSON, which is handy for other tools that need to locate these files:

calkit describe env --name {env-name} --json

The output is a single line, shown formatted here for readability (pipe it through something like jq to format it yourself):

{
  "kind": "uv-venv",
  "spec_path": "requirements.txt",
  "lock_path": ".calkit/env-locks/my-env/linux-64.txt",
  "prefix": null,
  "python": "3.13"
}

All keys are always present, so a null value means the field doesn't apply to that kind of environment. To describe every environment at once, keyed by name, use calkit describe envs [--json].

Choosing an environment type

So which type of environment should you use? The short answer is: any. Any environment is better than none, where none means installing dependencies in the global host machine environment. If you want the long answer, keep reading.

Docker is probably the most reproducible out of any environment type, since a Docker image includes information about the operating system. If it's convenient, e.g., if an image already contains all the necessary dependencies, go with a Docker environment. However, in some cases Docker may be a bit heavier than necessary.

If you're running Python code, a uv-venv environment is a good default choice. uv is very easy to install and very fast.

If you have non-Python dependencies that depend on complex compiled binaries (as scientific and engineering oriented tooling often does) and a uv-venv can't be built on your machine, A Conda environment is a good choice. However, Pixi has access to the same packages and is a bit faster. It's sort of like uv for Conda packages, and is similarly very easy to install.

If you're working on a machine for which you don't have control to install dependencies, or working as part of a team, a plain old Python venv could be the best option.

Again, try not to get too hung up on the decision of which environment type to use. Try one and see how it goes. Calkit should make the experience similar for all types.

Examples

Creating any type of environment from the Calkit CLI follows a similar pattern starting with calkit new. You can view the help output with calkit new --help and filter it down to environment-related commands with calkit new --help | grep env.

Docker

A new Docker environment can be added to the project with calkit new docker-env. A Docker environment can use an existing image, e.g., from Docker Hub, or it can create a new image, e.g., from a Dockerfile stored in the project repo.

Let's say you want to add an OpenFOAM environment to your project. This can be achieved with something like:

calkit new docker-env --image microfluidica/openfoam:2412 --name foam

Then you can run a command in that environment with:

calkit xenv -n foam -- icoFoam -help

You can similarly jump into an interactive bash terminal with:

calkit xenv -n foam bash

Some Docker images are CLI tools with an image entrypoint already defined (for example minlag/mermaid-cli). In that case, use command_mode: entrypoint so Calkit passes your command arguments directly to the container entrypoint instead of running shell -c ....

# In calkit.yaml
environments:
  mermaid:
    kind: docker
    image: minlag/mermaid-cli
    wdir: /data
    command_mode: entrypoint

Then execute with:

calkit xenv -n mermaid -- \
  -i figures/my-mermaid-diagram.mmd \
  -o figures/my-mermaid-diagram.pdf

But what if there isn't an image out there that has everything you need already installed into it? In this case, you can define and build a new derived image in the project by using the --from parameter, optionally adding predefined "layers" to the image with --add-layer. This will produce a Dockerfile defining the image, and when that environment is run with calkit xenv, that image will be built and a lock file produced.

For example, running:

calkit new docker-env \
    --from microfluidica/openfoam:2412 \
    --name foam2 \
    --add-layer miniforge

will create a Dockerfile in the project and add the environment named foam2 to the calkit.yaml file. Calling calkit xenv -n foam2 bash will cause the image to be built and a lock file Dockerfile-lock.json to be created. Note that the Dockerfile path can be controlled with the --path option.

You can go in and modify the Dockerfile, e.g., to add more installation commands, and another call to calkit xenv -n foam2 will kick off a rebuild automatically, since the lock file will no longer match the Dockerfile.

If you're copying local files into the Docker image, you can declare these dependencies in the environment definition so the content of those will be tracked as well:

# In calkit.yaml
environments:
  foam2:
    kind: docker
    image: foam2
    inputs:
      - src/mySolver.C

This highlights Calkit's declarative design philosophy. Simply declare the environment and use it in a pipeline stage and Calkit will ensure it is built and up to date. There is no need to think about building images as a separate step.

Caching images in a registry

An image built from a Dockerfile can take a long time to build, and once it's gone from the local Docker image store, rebuilding it is the only way to get it back. Worse, a rebuild isn't guaranteed to produce the same image, since the packages the Dockerfile installs move on over time.

Setting registry on the environment lets calkit push publish the image, so that Calkit can pull it back by digest whenever it's missing rather than rebuilding it:

# In calkit.yaml
environments:
  foam2:
    kind: docker
    path: Dockerfile
    image: foam2
    registry: ghcr.io

image can be left out when there's a path to build from, in which case the image is named after the project and the environment, e.g., someone/some-project.foam2, matching how the environment's Jupyter kernel is named. An environment defined purely by someone else's image has to name it.

That name comes from owner and name in calkit.yaml, or from the Git remote if they're not set. A project with neither says nothing about what it's called, and Calkit asks for image, or for owner and name, rather than naming the image after the directory the project happens to sit in, which would rename the image whenever the directory moved.

ghcr.io on its own resolves to the project's namespace in the GitHub Container Registry, e.g., ghcr.io/someone/some-project, naming the registry rather than leaving it to be worked out, but any registry prefix works, and leaving registry unset or null keeps images local. The lock file records the image's digest, so what gets pulled back is exactly what was built, and Calkit checks the image's layers against the lock after pulling.

calkit push sends the images of every environment with a registry set, alongside Git and DVC, skipping any the registry already has. An environment built before its registry was configured is pushed as-is, without being rebuilt first. A project with no image to send never reaches a registry at all: an environment with no registry, or one whose image was never built on this machine, is settled locally rather than with a round-trip and a request for credentials. Writing digests into lock files is a check's job, not a push's, so a push leaves the lock files alone. Pass --no-docker to skip images, or name what to send: calkit push docker publishes the images and nothing else, which is handy mid-work when the code isn't ready to go out with them. Checking an environment builds or pulls whatever the project needs to run, and leaves publishing to calkit push. The digest goes into the lock file as soon as the image is built, since a manifest is content-addressed: the digest an image is built with is the one it has once it's pushed, so pushing it leaves the lock alone.

To rebuild or repull an image and write fresh lock files, e.g., after an image's tag has been moved out from under the digest in the lock, run:

calkit update docker-env --name foam2 --lock

Locking multiple platforms

A Docker environment is locked per architecture, in .calkit/env-locks/{name}. Calkit reads the image's manifest from the registry and writes a lock file for each platform it provides, not just the one it's running on, so moving a project between an arm64 laptop and an amd64 server doesn't invalidate every stage in the environment.

Images from a registry are multi-platform already. An image built from a Dockerfile is built for one platform by default; to build and lock more than one, declare them:

# In calkit.yaml
environments:
  foam2:
    kind: docker
    path: Dockerfile
    image: foam2
    registry: ghcr.io
    build_platforms:
      - linux/amd64
      - linux/arm64

Building for multiple platforms requires a registry, since a multi-platform image can only be kept in one.

How images are fetched

The lock file for a Docker environment records the exact image its stages ran in: its layers, and the digests it can be pulled back by. Whenever an environment is checked, e.g., as part of calkit run, Calkit works from that record rather than assuming a rebuild will do, since building the same Dockerfile again produces a different image once the packages it installs have moved on.

If the image named in calkit.yaml is already in the local Docker image store and its layers match the lock, there's nothing to do. Otherwise Calkit tries, in order:

1. Pull the digest in the lock, from the environment's registry if it has one, or from wherever the image came from originally. This is the usual case for an image deleted by a docker system prune, or for a collaborator who's never built it. The layers of whatever comes back are checked against the lock, so a tag that's been moved out from under the digest can't quietly substitute a different image.
2. Fetch it from a project release. Releases record the images they archived (see Archiving Docker images), so if no registry can serve the image, Calkit looks through the project's releases for one whose layers match the lock, downloads it from Zenodo, loads it into Docker, and checks its layers again. This is what keeps an old version of a project reproducible after a registry has dropped the image, or after the account that published it is gone.
3. Build it, for an environment with a Dockerfile, or pull it by tag for one named after an existing image. Only at this point does the environment get an image that isn't the one the lock describes, and the lock is rewritten to record what was actually built.

The effect is that deleting an image locally costs a download rather than a rebuild, and doesn't invalidate any stage that used it, since the lock file doesn't change.

A downloaded release image is cached under .calkit/local/container-images so it isn't fetched twice. Fetching from a release needs the release to have been published, since that's where the file is downloaded from.

uv

uv can create both project and venv virtual environments. Project environments are defined by a pyproject.toml file, while venv environments are defined by a requirements.txt file.

To create a new uv project environment, inside a project directory run something like:

calkit new uv-env -n my-env "polars>=1.0" matplotlib

By default, this will create a pyproject.toml file in .calkit/envs/my-env/pyproject.toml, but the path can be controlled with the --path option.

To create a new uv venv, simply replace uv-env with uv-venv in the above command and a requirements.txt file will be created instead.

calkit xenv -n my-env python -c "import matplotlib, print(matplotlib.__version__)"

If you were to run something like:

calkit xenv -n my-env python -c "import pandas, print(pandas.__version__)"

it would fail, since pandas is not present in the spec file (pyproject.toml or requirements.txt). However, if you add it in there, calling the above command again will succeed because Calkit automatically checks or syncs the environment before execution.

venv

A venv environment, which uses Python's built-in venv module, can be used nearly identically to the uv example above. Simply replace uv-venv with venv in the calkit new call.

Conda

As you might expect, Conda environments again work nearly identically to uv-venv and venv environments.

You can create a new Conda environment with something like:

calkit new conda-env -n my-conda-env numpy matplotlib --pip pandas

Note that in this case, we specified one package, pandas, to be installed from the Python Package Index (PyPI) with pip using the --pip option.

The new Conda environment spec will be written to environment.yml by default, which can be controlled with the --path option.

A prefix for the environment can be specified to keep all packages under the project directory, e.g., by adding --prefix .conda-envs/my-conda-env. If this option is omitted, the environment will become part of Conda's system-wide collection of environments with a name like {project_name}-{env_name}, where the project name is added to avoid conflicts.

Similar to other environment types, any time a command is executed with calkit xenv, this environment will be checked and created or updated as necessary.

Calling:

calkit xenv -n my-conda-env -- which python

will create it. If you add any dependencies to environment.yml, calling that same command will cause the environment to be rebuilt before execution, and an updated environment-lock.yml file will be created. Again this highlights Calkit's declarative design philosophy. Declare the environment and what command should be executed inside, and Calkit will handle the rest.

Julia

Julia environments have paths that point to a Project.toml file. Creating a new Julia environment is similar to creating a Python environment:

calkit new julia-env \
    --name my-julia-env \
    --path ./envs/my-julia-env/Project.toml \
    --julia 1.11 \
    WaterLily \
    Makie

With Julia environments, it's possible to execute a command:

calkit xenv -n my-julia-env -- -e "println(\"hello world\");"

or a script:

calkit xenv -n my-julia-env -- my_julia_script.jl arg1 arg2

Running Julia this was will ensure the global environment is ignored, meaning you can be sure if it's successful on your machine, it will be successful on others.

SLURM

SLURM is a job scheduler commonly used for high performance computing (HPC). A SLURM environment can be defined in calkit.yaml as follows:

environments:
  my-hpc-cluster:
    kind: slurm
    host: hpc.myinstitute.org

See the HPC guide for how to use SLURM (and PBS) environments in pipeline stages.

System

A system environment is the machine as it is, with nothing built, installed, or isolated by Calkit. It's an escape hatch for software Calkit doesn't manage, e.g., a site-wide module system or a hand-built toolchain.

The simplest form is the machine you're on:

environments:
  local:
    kind: system
    lock:
      - os
      - python-version

Nothing is pinned by default, since opting out of isolation is the whole point of this kind. The lock property is how a project says which properties of the machine its results actually depend on. Locked properties are recorded in the environment's lock file the first time it's checked. On a machine where one of them differs, calkit run skips the stages using the environment rather than mixing results from different machines, until the lock is updated with calkit update env --lock. With relock: auto, the lock follows the machine instead, and the stages rerun.

The properties that can be locked are:

Any app's version can be locked too, as <app>-version, e.g., gcc-version. A version is recorded as its number up to the patch part, e.g., 27.3.1 from Docker version 27.3.1, build ce12230, or 6.8.0 from a Linux kernel release 6.8.0-45-generic. Any version can end in -major or -minor to lock only those parts, e.g., python-version-minor records 3.12 for Python 3.12.8, so only a move to another minor version counts as a change. -patch is the same as no ending, and -full keeps a suffix attached to the number, e.g., 11.4.0-1ubuntu1~22.04 for Ubuntu's GCC 11.4.0.

Run calkit describe system to see what these are on the machine you're on.

The built-in _system environment is shorthand for this kind on localhost with nothing locked.

Following the machine, or staying on one

Every stage using an environment depends on its lock file, so relock decides what happens to those stages when the lock and the machine disagree.

With relock: auto, results follow the machine. A change to a locked property rewrites the lock, so calkit status lists the stages using the environment as out of date, and calkit run reruns them. This suits results that depend on the version of a tool Calkit doesn't manage, e.g., a compiler from a site-wide module system:

environments:
  local:
    kind: system
    lock:
      - gcc-version
    relock: auto

Upgrading GCC, or moving to a machine with a different version of it, reruns the stages that use local.

With the default, relock: manual, results stay with the machine that made them, which suits benchmarks:

environments:
  bench-machine:
    kind: system
    lock:
      - os
      - machine
      - processor
      - cpu-count
      - memory-gb

On another machine, stages that are up to date stay up to date, and ones that need to run are skipped. calkit status keeps listing those as out of date, saying they can't run on this machine, until they run on the locked machine, or until calkit update env -n bench-machine --lock makes this machine the locked one, which puts every stage using the environment out of date at once. Either way, results from two machines are never mixed.

The lock is written the first time the environment is checked, so check it on the machine results should come from and commit the lock file. Otherwise, whichever machine checks it first becomes the locked one. A property added to lock later is recorded from the first machine that matches the rest of the lock, putting every stage using it out of date, like adding a dependency to a package spec keeps the versions already locked. On a machine that doesn't match, it's a mismatch like any other. To keep a stage's results no matter where the project runs, freeze the stage with frozen: true.

Requirements

A system environment can also declare requirements---what has to be true of that machine before stages run on it:

environments:
  cluster:
    kind: system
    host: hpc.example.edu
    requirements:
      - kind: cpu-count
        min: 16
      - julia>=1.10
    lock:
      - cpu-count
      - julia-version

requirements and lock answer different questions, which is why a property can appear in both. A requirement is a precondition: on a machine that doesn't meet it, the stages using the environment are skipped, and calkit status says what was found and what was needed. A lock pins a value results depend on: the first value seen is recorded, and a machine with a different one skips those stages too, unless the lock is updated or set to follow the machine. So the example above means "don't run on fewer than 16 CPUs," and separately "don't mix results from machines with different numbers of CPUs."

If you don't care what a property is but do care that it stays the same, lock it and leave it out of requirements. A requirement that constrains nothing is rejected, since it asserts nothing.

Requirements are checked on the machine the environment names. For a host that isn't this one, that means an app is looked for on that host's PATH, a variable is read from the shell a login gets there, and a setup requirement's check_command runs there. Nothing is offered as a fix in that case---installing something on another machine belongs to whoever administers it---so Calkit reports what was missing and where. Checking a requirement about the machine itself, like cpu-count, needs Calkit installed on that host, since that's what reports its properties.

The project's own top-level requirements describe the host you're driving from, which is the _system environment. They're checked on every calkit run, wherever stages end up running, because that machine still has to be able to drive the pipeline.

Running on another machine

A system environment's host names the machine the work belongs on. SSH is how a machine is reached, not a kind of environment in its own right, so there's no separate ssh kind: if host names the machine you're on, the stage runs right there, and otherwise Calkit connects over ssh and copies files with scp. This is useful, e.g., for offloading work to a cluster login node or a cloud VM with a more powerful GPU.

It is assumed that dependencies on the other machine are managed separately, unless you pair it with an inner environment (see below).

environments:
  cluster:
    kind: system
    host: "10.225.22.25"

The host is the only thing you have to declare. Everything else has a sensible default:

• user is left to SSH, which resolves it from ~/.ssh/config or falls back to your current account. Repeating it here would only be a second place for it to be wrong.
• wdir---the project's workspace on that machine, a clone of the project and the directory stages run in---defaults to ~/.calkit/workspaces/<hub>/<owner>/<name>. A relative path is taken from the connecting user's home directory. It's qualified by hub and owner because a host is shared: two projects named example-ssh from different owners are different projects. It's hidden because Calkit checks the workspace out with --force, so it must not look like somewhere you'd keep your own work.
• ssh_key is left to SSH and its agent. If a particular host needs a particular key, that belongs in ~/.ssh/config, which already answers "which key for which host" and is where people look for it. The field is for cases where that isn't available, such as CI dropping a key at a known path.

So the fuller form, if you do need to be explicit, is:

environments:
  cluster:
    kind: system
    host: "10.225.22.25"
    user: my-user-name
    wdir: /home/my-user-name/calkit/example
    ssh_key: ~/.ssh/id_ed25519

Getting set up

Check that the host is actually reachable before running anything:

calkit check env -n cluster

calkit run does the same check for every environment in the pipeline, so you don't have to remember to. Either way it connects without allowing a password prompt, so an unauthorized key is reported now rather than hanging halfway through a pipeline.

In a terminal, that check walks you through whatever is missing, asking before each step:

• Any environment variable the definition refers to, such as a host written as ${CK_SSH_HOST}, is prompted for and saved to .env, so it's only asked once. .env is added to .gitignore if it isn't already.
• If you have no SSH key, it offers to create one (ed25519, no passphrase, so stages can run unattended).
• If this machine isn't authorized on the host yet, it offers to run ssh-copy-id, then re-checks rather than assuming it worked.
• If Calkit isn't installed on the host, it offers to install it there. That's needed to activate an inner environment or to read the machine's properties for a lock.

Nothing happens without you agreeing to it, since creating a key and authorizing a machine both change things outside the project.

Without a terminal---in CI, say---none of this is attempted, because there's nobody to answer. It fails instead with the exact commands to run, e.g.:

ssh-copy-id -i ~/.ssh/id_ed25519 my-user-name@10.225.22.25
ssh my-user-name@10.225.22.25 'curl -LsSf install.calkit.org | sh'

To execute a command in this environment, we can add a stage like this to our pipeline in calkit.yaml:

pipeline:
  stages:
    run-simulation:
      kind: shell-script
      environment: cluster
      script_path: script.sh
      outputs:
        - results

How the workspace is kept in sync

Notice that nothing above says which files to copy back and forth, and nothing in the compiled pipeline does either. Calkit works it out, because a list of paths written down anywhere is a list that can fall behind the pipeline---at which point the stage quietly runs against stale inputs, which is the failure you'd least want here.

Before the command runs, Calkit captures your working tree---including edits you haven't committed---as a Git snapshot, pushes it straight to the workspace, and checks it out there detached. No branch is created on either side, so several people (or several clones) can share one workspace without their branch names colliding, and cleaning up afterwards is a single reserved namespace rather than a set of names someone has to recognize. Data that DVC tracks is ignored by Git, so it can't ride along in the snapshot. It travels through the workspace's own DVC cache instead, which Calkit addresses as a DVC remote: only the objects the workspace is missing cross the wire, deduplicated by content.

Afterwards, the workspace is asked what the run produced---anything it reports as changed or newly appeared since the snapshot it was given---and that is what comes back. dvc.lock is deliberately never carried back: your DVC writes its own from what it hashes locally.

The workspace is reused between runs, which is what keeps environments, the DVC cache, and the Git history warm---a fresh one would rebuild all three every time. Because it's a single checkout at a single commit, a run holds a lock on it, and a second run that wants the same workspace is told who has it rather than checking out over them. The lock sits beside the workspace, not inside it, and is released once a run's outputs have been collected; if the run is interrupted while the remote job is still going, the lock stays, because the workspace really is still busy.

One consequence worth knowing: if the project changes locally while a stage is running elsewhere, Calkit refuses to collect the results rather than recording them. DVC hashes a stage's dependencies from your local files once the command returns, so recording a result in that situation would write a dvc.lock pairing inputs that were never used with outputs they never produced---and unlike a stale stage, which simply reruns, a lock file like that goes on looking up to date indefinitely.

lock works the same way here as it does locally, except that the properties recorded are the host's---what a stage's results depend on is the machine it actually ran on. Calkit reads them from that machine when the environment is checked, which means Calkit has to be installed there. It already is if you pair the environment with a runtime (see below), since that's what activates the inner environment on the far end.

Pairing with a runtime

Because a system environment says where a stage runs rather than what it runs in, it can wrap another environment the same way a SLURM environment can, using the composite <outer>:<inner> syntax:

pipeline:
  stages:
    simulate:
      kind: python-script
      environment: cluster:py
      script_path: simulate.py

Calkit dispatches to cluster first, then activates the py environment once there, so the workspace on that machine needs both Calkit and the project.

MATLAB

Adding a MATLAB environment to a project will cause Calkit to automatically generate a Docker image based on its version and products attributes. A MATLAB_LICENSE_SERVER environmental variable must be set so the container can properly contact a license server. This can be done with:

calkit set-env-var MATLAB_LICENSE_SERVER <XXXX@some.server.edu>

Note that environmental variables set this way will be ignored by Git, and so will need to be set on each new machine on which the project is to be run.

A MATLAB environment (and a pipeline stage that uses it) looks like:

# In calkit.yaml
environments:
  my-matlab-2024b:
    kind: matlab
    version: R2024b
    products:
      - Simulink
      - Global_Optimization_Toolbox
      - Parallel_Computing_Toolbox

pipeline:
  stages:
    my-matlab-script:
      kind: matlab-script
      script_path: scripts/run_sim.m
      environment: my-matlab-2024b
      inputs:
        - config/my-sim-config.json
      outputs:
        - results/sim-results.h5

Pixi

Pixi environments typically have the path pixi.toml:

environments:
  my-pixi:
    kind: pixi
    path: pixi.toml

Nix

Calkit supports Nix environments via flakes. Reproducibility comes from flake.lock, which pins every input (including nixpkgs) to an exact revision. Calkit tracks flake.lock as a DVC dependency, so pipeline stages re-run when the environment changes.

Create one with:

calkit new nix-env --name my-nix-env python3 R uv

In calkit.yaml:

environments:
  my-nix-env:
    kind: nix
    path: flake.nix

Projects can contain multiple Nix envs. The first one lands at the repo root (flake.nix); subsequent ones get nested under .calkit/envs/{name}/flake.nix so each env has its own independent flake.lock. You can override the path with --path if you want a different layout.

To enter a specific dev shell from the flake (instead of the default), set shell:

environments:
  my-nix-env:
    kind: nix
    path: envs/my-nix-env/flake.nix
    shell: r-shell

Run a command in a Nix environment:

calkit xenv -n my-nix-env -- python --version

Add more packages to an existing Nix env (this edits the flake's packages = with pkgs; [ ... ] list, refreshes flake.lock, and commits):

calkit update env --name my-nix-env --add-package R --add-package polars

On Linux and macOS, install Nix with calkit install nix — this runs the Determinate Systems installer, which enables flakes by default. Nix is not supported natively on Windows; run Calkit inside WSL2 and install Nix there.

Calkit itself is also available as a flake — see Nix in the installation guide to add the Calkit CLI to your own Nix environments via inputs.calkit.url = "github:calkit/calkit".

R

R environments can be managed with Conda, Pixi, or renv (recommended). The env spec path for an renv environment is typically a DESCRIPTION file.

environments:
  r:
    kind: renv
    path: DESCRIPTION

The imports inside DESCRIPTION are used to create and sync the environment:

Package: CalkitProject
Version: 0.0.1
Title: Auto-generated R environment
Imports: tidyverse

System environments

A system environment runs things on the machine as it is, with nothing built, installed, or isolated. It's the escape hatch for software Calkit doesn't manage, e.g., a site-wide module system or a hand-built toolchain.

Nothing about the machine is pinned by default, since opting out of isolation is the point of this kind. lock is how a project says which properties its results actually depend on:

environments:
  cluster:
    kind: system
    host: gpu-node-1.example.edu
    lock:
      - os
      - julia-version

Locked properties are written to the environment's lock file, which stages depend on, the first time the environment is checked. On a machine where one of them differs, the lock is left alone, so results already computed stay current, and stages that need to run are skipped there rather than mixing in results from a different machine. Run calkit update env -n NAME --lock to make the new machine the one results come from, or set relock: auto for the lock to follow whichever machine runs the stages. Locking a property the machine can't supply, e.g., a tool that isn't installed, is an error rather than a recorded null---a stage that claims to be pinned to something it isn't is worse than one that pins nothing.

Running a project on more than one machine

A project often runs on more than one machine, e.g., a laptop for the paper and a cluster for the simulations. These settings decide what happens where:

• requirements on a stage or an environment say what a machine needs for it to run there. Elsewhere, its stages are skipped and keep their outputs, so the rest of the pipeline still runs (see requirements).
• A switch environment picks one of several environments by machine, e.g., a SLURM cluster where sbatch exists and this machine otherwise, without rerunning stages when the project moves between them.
• lock on a system environment records the machine properties results depend on. A machine that differs doesn't rerun those stages unless the lock is updated, or set to follow the machine with relock: auto.
• Lock files for uv-venv, venv and conda environments are written for every platform at once where possible, with the same package versions where each platform has them, so a collaborator on another operating system installs from them rather than relocking. Changing the spec relocks every platform, and a machine updates its environment to match its lock the next time it's checked. Which spec the locks came from is recorded in .calkit/env-spec-hashes.json, which is committed along with them.

calkit status says which stages can't run on the current machine and why.

Environment kind reference

Environment definitions belong in the environments section of calkit.yaml.

conda

Model class: CondaEnvironment

docker

Model class: DockerEnvironment

julia

Model class: JuliaEnvironment

matlab

Model class: MatlabEnvironment

nix

Model class: NixEnvironment

pbs

Model class: PBSEnvironment

pixi

Model class: PixiEnvironment

renv

Model class: REnvironment

slurm

Model class: SlurmEnvironment

switch

Model class: SwitchEnvironment

One of several environments, picked by the machine Calkit runs on.

switch lists the options in order, and the first whose when conditions this machine all meets is picked. The last option can leave out when to be the default. Each condition is one name: value pair:

• app_exists: an app on PATH.
• env_var_set: an environmental variable that is set.
• env_var_equals: NAME=value for an environmental variable.
• hostname_matches: a glob the hostname matches, ignoring case.
• os_is: linux, macos or windows.
• machine_id_equals: this machine's ID, as calkit describe system shows it.

A list of values means any one of them. To pick an environment when either of two conditions holds, list it as two options. For example:

cluster:
  kind: switch
  switch:
    - when:
        - os_is: linux
        - app_exists: sbatch
      use: any-slurm
    - when:
        - env_var_equals: NERSC_HOST=perlmutter
      use: perlmutter
    - when:
        - hostname_matches: ["*.gps.caltech.edu", "*.hpc.caltech.edu"]
      use: clima
    - use: clima-remote

The conditions are checked on the machine Calkit runs on, which is where the pick is made, even when the option picked is another machine.

An environment only the switch uses can be defined in place instead of named, e.g., use: {kind: slurm}. It's named after the switch and its position, e.g., cluster.1 for the first option, which is what calkit xenv -n and messages call it.

The options are either all machines (system, slurm or pbs), so the switch can be the outer half of a composite environment like cluster:py, or all runtimes, e.g., one per operating system. An option can't be another switch, or _system; to pick this machine as it is, define an environment with kind: system and use that.

Stages depend on the switch's definition and its options', so editing them reruns those stages, but not on which option was picked, so moving between machines the switch covers doesn't. A stage whose switch picks nothing on this machine can't run here, and is skipped.

An option's lock is part of the switch's too, so relocking the option, with relock: auto or calkit update env --lock, reruns the switch's stages. The first time a switch is checked, its options with a host are locked from there. An option without one is locked the first time it's checked on its own machine, which reruns nothing, since only a machine taking up its option has changed.

system

Model class: SystemEnvironment

The machine as it is, with nothing built, installed, or isolated.

An escape hatch for software Calkit doesn't manage, e.g., a site-wide module system or a hand-built toolchain. Nothing is pinned by default, since opting out of isolation is the whole point of this kind, so lock is how a project says which properties of the machine its results actually depend on.

Locked properties are recorded in the environment's lock file, which stages depend on, the first time the environment is checked. A property added to lock later is recorded from the first machine matching the rest. After that, the lock is not rewritten when the project moves to a machine where they differ, so stages that are up to date stay up to date. Checking the environment there warns about the difference, and a stage that needs to run is skipped by calkit run, since its result would not be comparable with the existing ones. To make the new machine the one results come from, run calkit update env -n NAME --lock, which updates the lock and invalidates the stages that depend on it. With relock: auto, that happens on its own, for results that should follow the machine.

requirements is the other half, and answers a different question. It says what must be true of this machine---apps that must be installed, variables that must be set, at least this many CPUs---and is checked before anything runs, on the machine the environment names. A stage whose requirements aren't met there is skipped, keeping its outputs, while a locked property records what results were computed with. One gates, the other pins, so a property that matters both ways is written in both places.

default_setup is what has to be done on this machine before a stage can run: sourcing a site setup script, loading modules, putting a hand-built toolchain on the PATH. It runs in the same shell as the stage's own command, so a variable it sets or a function it defines is in scope for the stage, whether or not it was exported---only a child process needs that. This is why it can't be a setup requirement: those run in a shell of their own and are cached, since they check whether something has been done rather than doing it every time.

It is not recorded in the environment's lock file. The pipeline compiler merges it with each stage's own setup and writes the result beside the pipeline, which the stage depends on---so changing it reruns exactly the stages that run it, and not the ones whose env_default_setup means they never do.

host names the machine. SSH is how a machine is reached, not a kind of environment, so there is no separate ssh kind: a system env whose host isn't this machine is reached over SSH, and one whose host is this machine runs here, the same way a SLURM env does. The built-in _system environment is shorthand for this kind on localhost with nothing locked.

machine_id says which machine, where host only says what it answers to. Names are renamed, resolve differently from different networks, and are reused; a project that means one particular machine can name it here instead and have that survive all of it. It replaces the name in deciding whether this is that machine, and is checked again on the far end when it isn't---so a host that has come to point at a different box is reported rather than run on. host is still what reaches it, so both are worth declaring for a machine that isn't this one. Run calkit describe system on a machine to read its ID.

Declaring one says where to run, which is a separate question from whether results depend on the machine: moving a project to a new one and updating this need not invalidate everything computed on the old one. Whether it does is left to lock, where machine-id is available for projects whose results really are machine-specific.

wdir is the project's workspace on that host---the directory the stage runs in. It defaults to ~/.calkit/workspaces/<hub>/<owner>/<name>, so a project that just names a host lands somewhere predictable rather than having to spell out a path that is the same on every machine anyway. Qualified by hub and owner because a host is shared, and hidden because transfers check out with --force: a path that looks like the user's own checkout is one whose edits would be silently destroyed.

What moves in and out of that workspace is deliberately not declared here. An environment doesn't know which files a stage reads, so a list kept alongside it can fall behind the pipeline and quietly run against stale inputs; the paths are taken from the stage instead.

uv

Model class: UvEnvironment

uv-venv

Model class: UvVenvEnvironment

venv

Model class: VenvEnvironment