Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

Running a Nomad Node

Difficulty: Medium (linux server management, cli usage)

Setup Time: ~1 hr

Technical Requirements

SGX Setup (Ubuntu 24.04 LTS)

  1. Ensure SGX is enabled in BIOS

Note: On OVH, make sure to enable Remote KVM or IPMI when provisioning the machine to be able to access the BIOS

  1. SSH into your server
ssh ubuntu@my.server.ip

Note: It is highly recommended to do some basic hardening on the instance.

  • Create a new user and delete the ubuntu user
  • Setup ~/.ssh/authorized_keys and disable password login for ssh
  • Routinely update the system packages
  1. Setup Intel's package signing key and install aesmd
curl -fsSL https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | sudo apt-key add -
sudo add-apt-repository "deb https://download.01.org/intel-sgx/sgx_repo/ubuntu noble main"
sudo apt-get update
sudo apt-get install -y sgx-aesm-service libsgx-dcap-default-qpl libsgx-enclave-common \
  libsgx-aesm-quote-ex-plugin libsgx-aesm-ecdsa-plugin libsgx-dcap-ql libsgx-dcap-quote-verify
sudo systemctl enable aesmd
  1. Configure the quote provider to use Intel PCS for attestations
sudo cp /etc/sgx_default_qcnl.conf /etc/sgx_default_qcnl.conf.bkp
sudo sed -s -i 's/localhost:8081/api.trustedservices.intel.com/' /etc/sgx_default_qcnl.conf
  1. Install required build dependencies
sudo apt-get update
sudo apt-get install -y build-essential libssl-dev protobuf-compiler pkg-config
  1. Install the Rust programming language
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.bashrc
rustup default stable
  1. Install sgxs tools from fortanix
cargo install sgxs-tools
  1. Add the current user to the sgx group
sudo groupadd sgx_prv
sudo groupadd sgx
sudo usermod -aG sgx_prv $USER
sudo usermod -aG sgx $USER

Logout and re-login to ensure the group is fully added to your user.

  1. Verify sgx installation

From the sgxs-tools crate we installed, we can verify if the machine has been provisioned correctly by running:

sgx-detect

You should see an output similar to the one below:

Detecting SGX, this may take a minute...
✔  SGX instruction set
  ✔  CPU support
  ✔  CPU configuration
  ✔  Enclave attributes
  ✔  Enclave Page Cache
  SGX features
    ✔  SGX2  ✔  EXINFO  ✔  ENCLV  ✔  OVERSUB  ✔  KSS
    Total EPC size: 378.0MiB
✔  Flexible launch control
  ✔  CPU support
  ？ CPU configuration
  ✔  Able to launch production mode enclave
✔  SGX system software
  ✔  SGX kernel device (/dev/sgx_enclave)
  ✔  libsgx_enclave_common
  ✔  AESM service
  ✔  Able to launch enclaves
    ✔  Debug mode
    ✔  Production mode
    ✔  Production mode (Intel whitelisted)
 
You're all set to start running SGX programs!
  1. Install Docker
curl -fsSL https://get.docker.com | bash

Docker compose

Create a directory for your node with the following structure:

nomad/
├─ data/
│  └─ config.toml
└─ docker-compose.yml
mkdir nomad && cd nomad
mkdir data
touch docker-compose.yml

Example docker-compose.yml

The following docker compose file runs the nomad node with the required sgx devices connected, configuration and state mounted, node auto-restart, and a healthcheck script:

services:
  nomad:
    container_name: nomad
    hostname: nomad
    image: "..." # Unreleased for now
    command: ["run", "-v"]
    devices:
      # Allow access to sgx devices
      - /dev/sgx_enclave:/dev/sgx_enclave
      - /dev/sgx_provision:/dev/sgx_provision
    volumes:
      # Mount aesmd dir for socket access
      - /var/run/aesmd:/var/run/aesmd
      # Mount quote provider configuration
      - /etc/sgx_default_qcnl.conf:/etc/sgx_default_qcnl.conf
      # Mount configuration and state directory
      - ./data:/root/.config/nomad
    environment:
      # Enable sgx logs
      - SGX_DEBUG=1
      # Enable rust backtraces
      - RUST_BACKTRACE=0
    ports:
      # Forward networking ports from configuration
      - "8000:8000"
      - "9000:9000"
    healthcheck:
      # Healthcheck endpoint integration
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    restart: unless-stopped

Telemetry

Telemetry can be exported by adding standard OTEL_EXPORTER environment variables to the compose file:

For example, using Signoz Cloud:

services:
  nomad:
    ...
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.us.signoz.cloud
      - OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=<key>

You may also create a .env file next to the docker-compose.yml containing the variables, and expose it to the node:

services:
  nomad:
    ...
    env_file: .env

Generating config.toml

Next to your docker compose file, the data/ directory is exposed to the container for configuration and enclave state.

Generate the configuration using the nomad config command:

docker compose run nomad config --network ethereum -w

If successful, a config file should be created at data/config.toml.

Example config.toml

The generated configuration should look like this:

[p2p]
# List of peer multiaddrs to bootstrap from
bootstrap = ["/ip4/15.235.50.174/tcp/9000"]
# Interval for searching for more network peers
bootstrap_interval = "5m"
# Port to listen on for the node's libp2p tcp server
# Must match the port forwarded in docker-compose.yml
tcp = 9000
 
[api]
# Port to listen on for the node's REST API server
# Must match the port forwarded in docker-compose.yml
port = 8000
 
[eth]
network = "ethereum"
# Ethereum RPC endpoint
geth_rpc = "https://ethereum-rpc.publicnode.com/"
builder_rpc = "https://ethereum-rpc.publicnode.com/"
# Number of active EOA accounts to manage
num_accounts = 8
# Minimum ETH balance to maintain per account
min_eth = 0.001
# Mirage platform compliance signers
whitelist_signers = ["9070a3c16a130e1db8f27414f3c6081c99798c7c856670ae49ac2f10b3faa4a4"]
polling_interval = "4s"
polling_max_attempts = 30
use_faucet = false
 
[eth.uniswap]
# Uniswap V2 router for automatic ETH replenishment
router = "0x7a250d5630b4cf539739df2c5dacb4c659f2488d"
max_slippage_percent = 5
swap_deadline = "20m"
# Amount of ETH to swap for when balance is low
target_eth_amount = 0.005
 
[eth.token.USDC]
# Token address to swap from for ETH
address = "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"
# Minimum balance to reserve (in human-readable units)
min_balance = 10000.0
# Enable swapping from this token
swap = true
 
[otlp]
# OpenTelemetry export settings
# Configure exporter via OTEL_EXPORTER environment variables
logs = false
traces = false
metrics = true
 
[enclave]
# SGX enclave binary paths
enclave_path = "enclave.sgxs"
signature_path = "mirage.sig"
seal_path = "."
# Bootstrap nodes to fetch enclave key from
nodes = ["15.235.50.174:8000"]
debug_keys = []

Funding the node

The node generates multiple deposit addresses. View them with the deposit subcommand:

docker compose run nomad deposit

This will print a list of addresses that the node controls, which will be cycled into EOA addresses used for executing user transactions.

  1. Send funds directly from a CEX (ie. Coinbase, Kraken, Binance). Do not send from a personal wallet or another on-chain address.
  2. Use random amounts for each deposit address.

The node will move the deposited funds into encumbered accounts and use them to execute user signals.

Backing up node keys

Starting and updating the node

Once the configuration has been created and funds have been deposited, the node can be started and managed using normal docker compose commands:

docker compose up -d --pull always

It is required to re-run this command with --pull always periodically, to ensure the node software is up-to-date. Failure to do so may lead to missing out on signals and rewards.

Stopping and withdrawing from the node

At any point the node operator can request funds to be withdrawn from the node's accounts. Simply halt the node process, and run one of the the following commands:

# Stop the node first
docker compose down
 
# For all funds
docker compose run nomad withdraw --all <RECIPIENT>
 
# For a specific amount of funds.
# Amount MUST include token decimals, for example 5000 USDC with 6 decimals:
docker compose run nomad withdraw --contract <TOKEN_ADDRESS> --amount 5000000000 <RECIPIENT>

The node will begin the withdraw process inside the enclave, sending the platform any outstanding fees, producing Mirage signals, and broadcasting them to other nodes for final transfers.