# nixos-hetzner-demo Continuous, rapid NixOS deployments to Hetzner Cloud with FlakeHub and OpenTofu ## Continuous, rapid, NixOS deployments to Hetzner Cloud # Continuous, rapid, NixOS deployments to Hetzner Cloud OpenTofu. > > [![doc](https://docs.outskirtslabs.com/nixos-hetzner-demo/next/)(https://img.shields.io/badge/doc-outskirtslabs-orange.svg)] > https://docs.outskirtslabs.com/open-source-vital-signs#static[image:https://img.shields.io/badge/status-static-blue.svg[status: > static]] > >
📌 NOTE
> This repo was based on https://determinate.systems[Determinate > Systems]'s https://github.com/determinatesystems/demo[repo for AWS > AMIs]. This is a proof-of-concept repo maintained by me and not DetSys. > I use something like this in prod, so it works, but don’t expect this > repo to be updated regularly. >
> > This project shows you how to continuously deploy a > [NixOS](https://zero-to-nix.com/concepts/nixos) configuration to a > [Hetzner Cloud](https://www.hetzner.com/cloud) server using > [OpenTofu](https://opentofu.org) and [FlakeHub](https://flakehub.com) in > seconds. > > == Differences from the AWS demo > > This repo is a port of the > [Determinate Systems AWS demo](https://github.com/determinatesystems/demo) > to Hetzner Cloud. Key differences: > > | | | | > | --- | --- | --- | > | Aspect | AWS Demo | Hetzner Demo | > | Demo app | EtherCalc (removed from nixpkgs) | CryptPad | > | Base image | Pre-built AMIs from Determinate Systems | Custom image built from [nixos-hetzner](https://github.com/outskirtslabs/nixos-hetzner) and uploaded | > | FlakeHub auth | IAM role (`determinate-nixd login aws`) | API token (`determinate-nixd login token`) | > | Deployment method | AWS Systems Manager (SSM) | SSH with deploy key | > | Networking | VPC, subnets, security groups | Simple firewall rules | > | GitHub auth | OIDC federation with AWS | SSH deploy key as secret | > > The core FlakeHub workflow remains the same: build closures in CI, > publish to FlakeHub Cache, and apply pre-built configurations in > seconds. > > However there are a few caveats: > > Since Hetzner Cloud does not have a public marketplace for cloud images, > you have to build the cloud image yourself (with the help of > [nixos-hetzner](https://github.com/outskirtslabs/nixos-hetzner)) and > upload it. Hetzner Cloud API keys are scoped to a project, so you will > need to upload the image on a per-project basis. > > We rely on the tool > [hcloud-upload-image](https://github.com/apricote/hcloud-upload-image) to > perform the image creation. It can take awhile. In my experience > uploading a 4GB image can take 8-12 minutes (assuming a fast pipe and > bandwidth is not the bottleneck). > > If you build the image on a system using FlakeHub Cache, then the cloud > image will be available to other systems from which you want to deploy > (that are also using FlakeHub Cache). > > Once you have a NixOS image uploaded to HCloud, it is straightforward to > create a nixos server from the snapshot. You can clickops at > console.hetzner.com, use the hcloud cli tool, or use terraform/opentofu. > > In this demo we use opentofu to provision the server, firewall, ssh key, > etc. > > The deployment process involves fetching a pre-built NixOS > [closure](https://zero-to-nix.com/concepts/closures) from > [FlakeHub](https://flakehub.com) and applying it to the Hetzner Cloud > server, streamlining the deployment process and ensuring consistency > across deployments. > > Timings (YMMV): > > | | | > | --- | --- | > | Step | Time | > | Image upload | ~9 minutes | > | Initial tofu apply | ~4 minutes | > | - Hetzner resources creation | 3m5s | > | - FlakeHub nixos provisioning | ~60s | > | Subsequent deploy (GitHub Actions) | ~3 minutes | > | - Build {plus} publish | 2m55s | > | - fh apply via SSH | 14s | > > == Sign-up for the FlakeHub beta > > To experience this streamlined NixOS deployment pipeline for yourself, > [sign up for the FlakeHub beta](https://determinate.systems) at > https://determinate.systems. FlakeHub provides the enterprise-grade Nix > infrastructure needed to fully use these advanced deployment techniques. > > == Prerequisites > > * Paid [Hetzner Cloud account](https://www.hetzner.com/cloud) with an API > token > * Paid [FlakeHub account](https://flakehub.com) with an API token > * [Detsys Nix](https://docs.determinate.systems/determinate-nix) with > flakes enabled > * [OpenTofu](https://opentofu.org) (available in the dev shell) > Project status: > **[Static](https://docs.outskirtslabs.com/open-source-vital-signs#static)**. > > == Getting Started > > This demo deploys [CryptPad](https://cryptpad.net), a collaborative > document editing platform, to a Hetzner Cloud server. > > Deployment is a two-step process: > > 1. [Manual deployment](#manual-deployment) - Run OpenTofu locally to > create the server infrastructure (required first) > 2. [Automated deployment](#automated-deployment-with-github-actions) - > Once the server exists, push to GitHub for CI/CD deployments >
💡 TIP
> For a full rundown of how everything in the demo works, see > [How it works](#how-it-works) below. >
> > === Manual deployment > > ==== 1. Build and upload the base image > > First, build and upload a NixOS base image to Hetzner Cloud. This only > needs to be done once (or when you want to update the base image). > > ```shell > # Enter the dev shell > nix develop > > # Create .env file with your tokens (keeps them out of shell history) > cat > .env << 'EOF' > HCLOUD_TOKEN=your-hetzner-token > FLAKEHUB_TOKEN=your-flakehub-token > EOF > chmod 600 .env > > # Load and export tokens, then build/upload the image > set -a && source .env && set +a > ./scripts/upload-image.sh > > # Note the image ID from the output (e.g., 123456789) > ``` > > ==== 2. Generate a deploy key > > Generate an SSH key for GitHub Actions deployments: > > ```shell > ssh-keygen -t ed25519 -C "github-actions-deploy" -f deploy_key -N "" > ``` > > ==== 3. Configure OpenTofu > > ```shell > cd setup > > # Copy the example config > cp terraform.tfvars.example terraform.tfvars > > # Edit terraform.tfvars with your values: > # - hcloud_token: Your Hetzner Cloud API token > # - hcloud_image_id: The image ID from step 1 > # - flakehub_token: Your FlakeHub API token > # - deploy_ssh_public_key: Contents of deploy_key.pub > # - deploy_ssh_private_key: Contents of deploy_key (private key, for first-boot provisioning) > # - (optional) ssh_public_key: Your personal SSH key for manual access > ``` > > ==== 4. Create the infrastructure > > ```shell > # Initialize OpenTofu > tofu init > > # Validate the configuration > tofu validate > > # Create the resources > tofu apply -auto-approve > > # Get the website URL > export CRYPTPAD_URL=$(tofu output --json | jq -r .website.value) > > # Open the website (wait ~60 seconds for first deployment) > open "${CRYPTPAD_URL}" > > # (optional) When you're done, destroy the resources > tofu destroy -auto-approve > ``` > > === Automated deployment with GitHub Actions > > Once the server exists, GitHub Actions can automatically deploy NixOS > configuration changes. Configure the following repository secrets: > > | | | > | --- | --- | > | Secret | Description | > | `DEPLOY++_++SSH++_++PRIVATE++_++KEY` | The contents of the `deploy++_++key` file (private key) | > | `HETZNER++_++SERVER++_++IP` | The server IP from `tofu output server++_++ip` | > > Using the `gh` CLI after completing link:#manual-deployment[manual > deployment]: > > ```shell > # Create a production environment (optional) > gh api repos/{owner}/{repo}/environments/production --method PUT > > # Set secrets > cat deploy_key | gh secret set DEPLOY_SSH_PRIVATE_KEY > tofu output -raw server_ip | gh secret set HETZNER_SERVER_IP > ``` > > The workflow will automatically build, publish to FlakeHub, and deploy > on pushes to main. > > == Documentation > > * [Docs](https://docs.outskirtslabs.com/nixos-hetzner-demo/next/) > * https://docs.outskirtslabs.com/nixos-hetzner-demo/next/api[API > Reference] > * https://github.com/outskirtslabs/nixos-hetzner-demo/issues[Support via > GitHub Issues] > == How it works > > Here’s a high level walkthrough of what’s going on > > Initial setup (one-time) > > 1. Build NixOS base image locally using > [nixos-hetzner](https://github.com/outskirtslabs/nixos-hetzner) > 2. Upload image to Hetzner Cloud via hcloud-upload-image > 3. Generate SSH deploy key for GitHub Actions > 4. Configure terraform.tfvars with tokens, image ID, and keys > First deployment (tofu apply) > > 1. OpenTofu creates Hetzner resources (SSH key, firewall, server) > 2. Server boots with base NixOS image (includes determinate-nixd and fh) > 3. Provisioner SSHs into server > 4. Authenticates with FlakeHub using `determinate-nixd login token` > 5. Runs `fh apply nixos ++<++flakeref++>++` to pull config from FlakeHub > 6. Server downloads pre-built closure and switches to new configuration > Subsequent deployments (via GitHub Actions) > > 1. Developer pushes changes to flake.nix > 2. CI builds the NixOS closure > 3. flakehub-push publishes closure to FlakeHub (with output paths cached) > 4. Deploy job SSHs to server with the exact flakeref > 5. Server runs `fh apply nixos` - downloads closure from FlakeHub Cache > 6. Configuration applied in seconds (no building on server) > === Nix flake > > The [`flake.nix`](./flake.nix) defines the NixOS configuration for > the demo system: > > * Inputs: > * `nixpkgs`: Custom nixpkgs fork with Hetzner Cloud tools (can be > reverted to upstream nixpkgs once > [PR #375551](https://github.com/NixOS/nixpkgs/pull/375551) is merged > * [`nixos-hetzner`](https://github.com/outskirtslabs/nixos-hetzner): > Hetzner Cloud image building tools > * `determinate`: Determinate Nix distribution from FlakeHub > * `fh`: FlakeHub CLI from FlakeHub > * Outputs: > * `nixosConfigurations.cryptpad-demo`: The CryptPad server > configuration > * `devShells.default`: Development environment with required tools > === OpenTofu configuration > > The [`setup/`](./setup/) directory contains OpenTofu configuration > for Hetzner Cloud: > > * `providers.tf`: Hetzner Cloud provider configuration > * `variables.tf`: Input variables (API tokens, image ID, SSH keys, etc.) > * `main.tf`: Server, firewall, and SSH key resources > * `outputs.tf`: Server IP and website URL > After server creation, Terraform provisions the server via SSH: > > 1. Authenticates with FlakeHub using `determinate-nixd login token` > 2. Applies the NixOS configuration using `fh apply nixos` > === GitHub Actions workflow > > The [`.github/workflows/ci.yml`](./.github/workflows/ci.yml) > workflow: > > 1. Build job: Builds the NixOS closure and publishes to FlakeHub > 2. Deploy job: SSHs to the server and runs `fh apply nixos` with the new > closure > === Continuous deployment > > Continuous deployments work by: > > 1. Pushing changes to the `flake.nix` > 2. GitHub Actions builds the new closure > 3. The closure is published to FlakeHub Cache > 4. The deploy job SSHs to the server and applies the new configuration > To demonstrate, make a change to the CryptPad configuration in > `flake.nix` and push the changes. > > === Triggering rollbacks > > Use the `workflow++_++dispatch` event to manually trigger a deployment > of a previous version. > > Why FlakeHub? > > Applying fully evaluated NixOS closures via > [FlakeHub](https://flakehub.com) differs from typical deployments using > Nix in several key ways: > > _Deployment speed_ > > * FlakeHub deployment: The NixOS configuration is evaluated and built > ahead of time. As the closure is pre-built and cached, the deployment > process is faster. The server only needs to download and apply the > pre-built closure. > * Typical Nix deployment: The evaluation and build process happens > during deployment, which can be time-consuming. > _Resource utilization_ > > * FlakeHub deployment: Offloads the computationally intensive tasks of > evaluation and building to a controlled environment (e.g., a CI/CD > pipeline), freeing up resources on the target server. > * Typical Nix deployment: The target server must handle the evaluation > and build process, which can be resource-intensive. > _Scalability_ > > * FlakeHub deployment: The pre-built and cached nature allows for rapid > instance provisioning, making it ideal for auto-scaling scenarios. > * Typical Nix deployment: The time required for evaluation and building > on each new instance can introduce significant delays. > In summary, applying a fully evaluated NixOS closure from > [FlakeHub](https://flakehub.com) ensures that the exact same configuration > is deployed every time, as the closure is a fixed, immutable artifact.