Developer Guide

Welcome to the developer guide for the miden node :)

This is intended to serve as a basic introduction to the codebase as well as covering relevant concepts and recording architectural decisions.

This is not intended for dApp developers or users of the node, but for development of the node itself.

It is also a good idea to familiarise yourself with the operator manual.

Contributing to Miden Node

Thanks for taking the time to contribute. We want to make contributing to this project as easy and transparent as possible.

Before you begin

Start by commenting your interest in the issue you want to address. This lets us assign the issue to you, prevents multiple people from repeating the same work, and gives us a place to add any context you may need.

Most contributions should branch from and target next. Maintenance work may instead target a release branch; see the branching policy for details. If you are unsure which branch to use, confirm the target on the issue before starting work.

Typos and low-effort contributions

We don't accept PRs for typo fixes as these are often scanned for AI "contributors". If you find typos please open an issue instead.

Commits

Try keep your commit names and messages related to the content. This provides reviewers with context if they need to step through your changes by commit.

This does not need to be perfect because we generally squash merge a PR - the commit naming is therefore only relevant for the review process.

Pre-PR checklist

Before submitting a PR, ensure that you're up to date by rebasing onto the branch your pull request targets, and that tests and lints pass by running:

# Runs the various lints
make lint
# Runs the test suite
make test

Post-PR

Please don't rebase your branch once the PR has been opened. In other words - only append new commits. This lets reviewers have a consistent view of your changes for follow-up reviews. Reviewers may request a rebase once they're ready in order to merge your changes in.

Any contributions you make will be under the MIT Software License

In short, when you submit code changes, your submissions are understood to be under the same MIT License that covers the project. Feel free to contact the maintainers if that's a concern.

Branching policy

The repository has two kinds of protected branches:

• next is the active development branch and the default target for pull requests.
• release/vX.Y is a maintained branch for a specific minor release line, for example release/v0.16.

Creating a release branch

next may continue to represent the latest stable release line until development requires a breaking change. Before that change is merged, maintainers create the corresponding release/vX.Y branch from next. The release branch preserves the compatible release line while breaking development continues on next.

The version portion of a release branch contains only the major and minor version. Each component is a non-negative integer without leading zeroes. Only repository administrators create new release branches.

Applying changes

After a release branch is created, it permanently diverges from next. Releases are not promoted by merging next into a release branch or by merging a release branch back into next.

Changes should target the branch where they are required:

• New development targets next.
• Maintenance changes target the applicable release/vX.Y branch.
• A change needed on multiple branches is applied to each branch through its own pull request, normally by backporting the original change.

Branch protection

The Protected branches ruleset applies to next and all release/vX.Y branches. It prevents deletion and non-fast-forward updates, requires pull requests, and requires the repository's test status check.

Administrators can bypass reviews and required checks through a pull request when necessary, but cannot bypass the pull request requirement. This preserves an audit trail for every protected branch change.

Release process

A release is created by tagging the commit to release. It does not require a merge between next and a release branch. The commit may be on next while it still represents that release line, or on the applicable release/vX.Y branch after the branches have diverged.

Release tags

Release tags use one of these forms:

• vX.Y.Z for a stable release, for example v0.16.2.
• vX.Y.Z-suffix for a prerelease, for example v0.17.0-rc.1.

X, Y, and Z are non-negative integers without leading zeroes. A prerelease suffix must start with - and contain at least one character after it.

The Release tags ruleset is the source of truth for the accepted tag format and restricts who can create, update, or delete release tags.

Creating a release

1. Ensure the target commit is in a publishable state. Every publishable workspace package must use the same version.
2. Create the release tag on that commit and push it to GitHub. The tag without its leading v must exactly match the workspace package version.
3. The Release workflow validates the tag and package versions, checks crate builds and the minimum supported Rust version, dry-runs crate, Docker image, and Compose publishing, then publishes the Docker images and Compose application.
4. After those checks pass, the workflow creates the GitHub release and release notes, then starts the crates.io and Debian publishing workflows.

The root docker-compose.yml includes the component models under compose/, keeping direct local Compose commands and profiles independent of additional -f arguments. Local includes cannot be published directly, so the .github/actions/publish-compose action first renders the complete, all-profile model without interpolating its variables or normalizing project resource names. It then either dry-runs or publishes that flattened model.

The bundled three-validator development genesis is an inline Compose config named genesis. Consumers replace that resource with a file through a Compose override when they need a custom genesis configuration. The same override works with the repository model and the published OCI application.

The workflow uses a broad v* trigger because GitHub Actions does not use the same pattern language as repository rulesets. Its preflight step verifies that this trigger still matches the target of the Release tags ruleset, while the ruleset itself enforces the release-tag format.

Navigating the codebase

The code is organised using a Rust workspace with separate crates for the node and remote prover binaries, a crate for each node component, a couple of gRPC-related codegen crates, and a catch-all utilities crate.

The primary execution artifacts are the node and remote prover binaries. The library crates are not intended for external usage, but instead simply serve to enforce code organisation and decoupling.

We have a top-level proto crate, which contains the external and internal gRPC and protobuf schemas. It also exposes the tonic/prost file descriptors for each gRPC service for convenience. We then have an internal proto crate in ./crates, which uses the above file descriptors to generate the actual service traits, and also defines some domain objects and other gRPC shared utilities and definitions.

[!NOTE] > miden-protocol is an important dependency which contains the core Miden protocol definitions e.g. accounts, notes, transactions etc.

Monitoring

Developer level overview of how we aim to use tracing and open-telemetry to provide monitoring and telemetry for the node.

Please begin by reading through the monitoring operator guide as this will provide some much needed context.

Approach and philosophy

We want to trace important information such that we can quickly recognise issues (monitoring & alerting) and identify the cause. Conventionally this has been achieved via metrics and logs respectively, however a more modern approach is using wide-events/traces and post-processing these instead. We're using the OpenTelemetry standard for this, however we are only using the trace pillar and avoid metrics and logs.

We wish to emit these traces without compromising on code quality and readability. This is also a downside to including metrics - these are usually emitted inline with the code, causing noise and obscuring the business logic. Ideally we want to rely almost entirely on tracing::#[instrument] to create spans as these live outide the function body.

There are of course exceptions to the rule - usually the root span itself is created manually e.g. a new root span for each block building iteration. Inner spans should ideally keep to #[instrument] where possible.

Local Compose routing

The local Compose model always runs an OpenTelemetry Collector, and every Miden service exports to its stable in-stack address. This indirection is necessary because a Compose profile can start Tempo but cannot conditionally change the environment of the already-running Miden services.

The collector fans each trace out through two independent failover connectors. One prefers the Tempo service and the other prefers the caller-defined OTEL_EXPORTER_OTLP_ENDPOINT. An unavailable destination falls back to a /dev/null file exporter, and the collector periodically probes the preferred destination so it can resume forwarding when the destination becomes available. The caller branch defaults to a secondary loopback OTLP receiver that feeds the same null exporter, avoiding connection errors when no external endpoint is configured.

This setup deliberately favors isolation over durable delivery. Traces are dropped while a destination is unavailable, so neither an optional Compose profile nor an external collector can interfere with the node services.

Relevant crates

We've attempted to lock most of the OpenTelemetry crates behind our own abstractions in the utils crate. There are a lot of these crates and it can be difficult to keep them all separate when writing new code. We also hope this will provide a more consistent result as we build out our monitoring.

tracing is the defacto standard for logging and tracing within the Rust ecosystem. OpenTelemetry has decided to avoid fracturing the ecosystem and instead attempts to bridge between tracing and the OpenTelemetry standard in-so-far as is possible. All this to say that there are some rough edges where the two combine - this should improve over time.

Important concepts

OpenTelemetry standards & documentation

https://opentelemetry.io/docs

There is a lot. You don't need all of it - look things up as and when you stumble into confusion.

It is probably worth reading through the naming conventions to get a sense of style.

Footguns and common issues

tracing requires data to be known statically e.g. you cannot add span attributes dynamically. tracing-opentelemetry provides a span extension trait which works around this limitation - however this dynamic information is only visible to the OpenTelemetry processing i.e. tracing_subscriber won't see this at all.

In general, you'll find that tracing subscribers are blind to any extensions or OpenTelemetry specific concepts. The reverse is of course not true because OpenTelemetry is integrating with tracing.

Another pain point is error stacks - or rather lack thereof. #[tracing::instrument(err)] correctly marks the span as an error, however unfortunately the macro only uses the Display or Debug implementation of the error. This means you are missing the error reports entirely. tracing_opentelemetry reuses the stringified error data provided by tracing so currently there is no work-around for this. Using Debug via ?err at least shows some information but one still misses the actual error messages which is quite bad.

Manually instrumenting code (i.e. without #[instrument]) can be rather error prone because async calls must be manually instrumented each time. And non-async code also requires holding the span.

Distributed context

We track traces across our components by injecting the parent span ID into the gRPC client's request metadata. The server side then extracts this and uses this as the parent span ID for its processing.

Read more in the official OpenTelemetry documentation.

Choosing spans

A root span should represent a set of operations that belong together. It also shouldn't live forever as span information is usually only sent once the span closes i.e. a root span around the entire node makes no sense as the operation runs forever.

A good convention to follow is creating child spans for timing information you may want when debugging a failure or slow operation. As an example, it may make sense to instrument a mutex locking function to visualize the contention on it. Or separating the database file IO from the sqlite statement creation. Essentially operations which you would otherwise consider logging the timings for should be separate spans. While you may find this changes the code you might otherwise create, we've found this actually results in fairly good structure since it follows your business logic sense.

Inclusions and naming conventions

Where possible, attempt to find and use the naming conventions specified by the standard, ideally via the opentelemetry-semantic-conventions crate.

Include information you'd want to see when debugging - make life easy for your future self looking at data at 3AM on a Saturday. Also consider what information may be useful when correlating data e.g. client IP.

Node components

The node is split into five distinct components that communicate via gRPC. See the operator architecture guide for an overview of each component.

The following sections will describe the inner architecture of each component.

RPC Component

This is by far the simplest component. Essentially this is a thin gRPC server which proxies all requests to the store and block-producer components.

Its main function is to pre-validate all requests before sending them on. This means malformed or non-sensical requests get rejected before reaching the store and block-producer, reducing their load. Notably this also includes verifying the proofs of submitting transactions. This allows the block-producer to skip proof verification (it trusts the RPC component), reducing the load in this critical component.

RPC Versioning and the HTTP ACCEPT header

The RPC component allows clients to negotiate their desired Miden RPC version using the well-known HTTP ACCEPT header, using the following format:

application/vnd.miden; version=<version-req>; genesis=<genesis-commitment>

The version lets the client specify their supported version and the server will attempt to comply if it can. At this early stage, only client versions which are semver compatible with the server version are likely to be accepted i.e. the server in all likely only supports a single version.

The genesis property is intended to let the client confirm they are on the correct network, by specifying the network's genesis commitment. This guards against operating on the wrong network, as well as against network resets.

Query limits (GetLimits)

The RPC service exposes a GetLimits endpoint which returns the query parameter limits enforced by the server for multi-value parameters (e.g. number of nullifiers, note tags, note IDs, account IDs).

These limits are defined centrally in miden_node_utils::limiter and are enforced at the RPC boundary (and also inside the store) to keep database queries bounded and to keep response payloads within the ~4 MB budget.

GENERAL_REQUEST_LIMIT is currently 1000, and endpoint-specific limits are:

Additional internal-only limits in miden_node_utils::limiter (not surfaced by GetLimits) include:

Error Handling

The RPC component uses domain-specific error enums for structured error reporting instead of proto-generated error types. This provides better control over error codes and makes error handling more maintainable.

Error Architecture

Error handling follows this pattern:

1. Domain Errors: Business logic errors are defined in domain-specific enums
2. gRPC Conversion: Domain errors are converted to gRPC Status objects with structured details
3. Error Details: Specific error codes are embedded in Status.details as single bytes

SubmitProvenTx Errors

Transaction submission errors are:

#![allow(unused)]
fn main() {
enum SubmitProvenTxGrpcError {
    Internal = 0,
    DeserializationFailed = 1,
    InvalidTransactionProof = 2,
    IncorrectAccountInitialCommitment = 3,
    InputNotesAlreadyConsumed = 4,
    UnauthenticatedNotesNotFound = 5,
    OutputNotesAlreadyExist = 6,
    TransactionExpired = 7,
}
}

Error codes are embedded as single bytes in Status.details

Store component

This component persists the chain state in a sqlite database. It also stores each block's raw data as a file.

Merkle data structures are kept in-memory and are rebuilt on startup. Other data like account, note and nullifier information is always read from disk. We will need to revisit this in the future but for now this is performant enough.

Migrations

We have database migration support in place but don't actively use it yet. There is only the latest schema, and we reset chain state (aka nuke the existing database) on each release.

RocksDB tree storage

The account and nullifier trees are persisted in separate RocksDB instances under <data-directory>/accounttree and <data-directory>/nullifiertree, managed by RocksDbStorage from miden-crypto. Column families: leaves, st16–st56 (subtrees at each depth), metadata (root/counts), depth16 (cached depth-16 hashes for fast startup).

Compaction parallelism and background jobs are set to rayon::current_num_threads() automatically. WAL sync per write is disabled for throughput; a 512 MiB WAL cap bounds recovery time. Bloom filter bits vary by depth (8.0–12.0) and memtables are 128 MiB per column family. See RocksDbStorage::open for the full fixed configuration. Runtime-tuneable parameters are documented in the operator usage guide.

Architecture

The store consists mainly of state management plus a gRPC server which answers requests from the RPC component. In sequencer mode, the block-producer uses the store state in-process for block inputs and block application.

Block Producer Component

The block-producer is responsible for ordering transactions into batches, and batches into blocks, and creating the proofs for batches. Proving is usually outsourced to a remote prover but can be done locally if throughput isn't essential, e.g. for test purposes on a local node.

The core of the block-producer revolves around the mempool which forms a DAG of all in-flight transactions and batches. It also ensures all invariants of the transactions are upheld e.g. account's current state matches the transaction's initial state, that all input notes are valid and unconsumed and that the transaction hasn't expired.

Batch production

Transactions are selected from the mempool periodically to form batches. This batch is then proven and submitted back to the mempool where it can be included in a block.

Block production

Proven batches are selected from the mempool periodically to form the next block. The block is then built and sent to the validator for verification and signing. This signed block is then submitted to the store where it gets proven and committed. Proof generation in production is typically outsourced to a remote machine with appropriate resources. For convenience, it is also possible to perform proving in-process. This is useful when running a local node for test purposes.

Once the block is committed, all transactions and batches in the block are marked in the mempool as committed.

Mempool data pruning

The mempool keeps the N most recent blocks locally, to allow incoming transactions a grace period so we can verify their state against the store, and the local state deltas in the mempool. Without this overlap, we would constantly be racing transaction check against the store with newly committed blocks.

After each now block, the N+1th oldest block and its batches and transactionsa are pruned from the mempool state.

Transaction lifecycle

1. Transaction arrives at RPC component
2. Transaction proof is verified
3. Transaction arrives at block-producer
4. Transaction delta is verified
   • Does the account state match
   • Do all input notes exist and are unconsumed
   • Output notes are unique
   • Transaction is not expired
5. Wait until all parent transactions are in a batch
6. Be selected as part of a batch
7. Proven as part of a batch
8. Wait until all parent batches are in a block
9. Be selected as part of a block
10. Committed

Note that its possible for transactions to be rejected/dropped even after they've been accepted, at any point in the above lifecycle (which effectively shows the happy path). This can occur if:

• The transaction expires before being included in a block.
• Any parent transaction is dropped (which will revert the state, invalidating child transactions).
• It causes proving or any part of block/batch creation to fail repeatedly. This is a fail-safe against unforeseen bugs, removing problematic (but potentially valid) transactions from the mempool to prevent outages.

Network Transaction Builder Component

The network transaction builder (NTB) is responsible for driving the state of network accounts.

What is a network account

Network accounts are a special type of fully public account which contains no authentication and whose state can therefore be updated by anyone (in theory). Such accounts are required when publicly mutable state is needed.

An issue with publicly mutable state is that transactions against an account must be sequential and require the previous account commitment in order to create the transaction proof. This conflicts with Miden's client side proving and concurrency model since users would race each other to submit transactions against such an account.

Instead our solution is to have the network be responsible for driving the account state forward, and users can interact with the account only indirectly using notes. Notes don't require a specific ordering and can be created concurrently without worrying about conflicts. We call these network notes and they always target a specific network account.

A network transaction is a transaction which consumes and applies a set of network notes to a network account. There is nothing special about the transaction itself - it can only be identified by the fact that it updates the state of a network account.

Limitations

At present, we artificially limit this such that only this component may create transactions against network accounts. This is enforced at the RPC layer by disallowing network transactions entirely in that component. The NTB skirts around this by submitting its transactions directly to the block-producer.

This limitation is there to prevent complicating the NTBs implementation while the protocol and definitions of network accounts, notes and transactions mature.

Implementation

The NTB uses an actor-per-account model managed by a central Coordinator. On startup the coordinator syncs all known network accounts and their unconsumed notes from the store. It then follows the committed block stream from the RPC service for updates which would impact network account state.

For each network account, the coordinator spawns a dedicated AccountActor. Each actor runs in its own async task and is responsible for creating transactions that consume network notes targeting its account. On startup, each actor waits until its account has been committed to the chain before producing any transactions. This means newly created network accounts will idle until their creation transaction is included in a block. Once the committed state is available, the actor reads its state from the database and re-evaluates whenever notified by the coordinator.

Actors that have been idle (no available notes to consume) for longer than the idle timeout will be deactivated. The idle timeout is configurable via the --ntx-builder.idle-timeout CLI argument (default: 5 minutes).

Deactivated actors are re-spawned when committed-chain processing detects new notes targeting their account.

Each actors crash count is tracked, and once the count reaches a configurable threshold, the account is deactivated and no new actor will be spawned for it. This prevents resource exhaustion from a persistently failing account. The threshold is configurable via the --ntx-builder.max-account-crashes CLI argument (default: 10).

The block-producer remains blissfully unaware of network transactions. From its perspective a network transaction is simply the same as any other.

gRPC Server

The NTX exposes an internal gRPC server for querying its state. The RPC component proxies public requests to this server. In bundled mode the server is started automatically on a random port and wired to the RPC; in distributed mode operators must pass the NTB's address to the RPC via --ntx-builder.url (or MIDEN_NODE_NTX_BUILDER_URL).

Currently the only endpoint is GetNetworkNoteStatus(note_id) which returns the lifecycle status of a network note (pending, processed, or discarded), along with the latest execution error, attempt count, and block number of the last attempt. This is useful for debugging notes that fail to be consumed.

Validator Component

The validator is responsible for verifying each new block and signing it if correct.

This signature is required before a block may be committed on chain, and thus acts as an independent safe guard.

The validator is therefore run completely separate from the main node operations, and is operated by a separate entity. The validator's public key is published (or at least will be for mainnet).

Dual purpose: training wheels

The validator has a 2nd purpose while Miden is maturing. To prevent private state from being lost, and to guard from potential bugs in the VM/cryptography primitives, Miden will launch with training wheels. Notably, we require users to include the private input data along with their transactions. This means users will have privacy on the network but not from the validator set.

As part of the transaction submission process, each transaction, its proof, and private inputs, are sent to the validator, which re-executes the transaction, thereby verifying it and its proof are correct. This also lets us store the private data as part of our training wheels.

Block verification

The validator ensures that each new block is sequential with the previously signed block. i.e. header.parent_commitment == last_block.commitment. It also checks that the block contains only transactions that it has previously seen and verified.

Once verified, the block is signed and returned to the sender.

Transaction encryption key

In addition to its per-validator signing key, every validator is provisioned with the same shared transaction encryption keypair, an Ed25519 key that miden-crypto uses for X25519 key agreement in its IES scheme. Clients use it to encrypt the private transaction inputs they submit, so that any validator in the set can decrypt them.

The GetTransactionEncryptionKey endpoint returns the shared public key together with an IES scheme identifier, an opaque key ID, and a list of validator attestations, currently holding one signature from this validator's own signing key over an attestation commitment (the TransactionEncryptionKey proto message documents the exact payload). The commitment carries a domain tag that separates attestations from block header signatures, and the genesis commitment so an attestation cannot replay across networks. The signature proves to clients that a chain-recognized validator vouches for the key, so the key can be served through an untrusted RPC.

This scheme does not protect the inputs from parties holding the shared secret and has no forward secrecy. It is the first phase of the transaction input encryption design: later phases move the key material to threshold and TEE-managed setups.

Submission path

SubmitProvenTransaction carries a SealedTransactionInputs envelope: a key_id in the clear plus the ciphertext of a serialized SealedMessage. The validator rebuilds the associated data from its own scheme, key id and genesis commitment, plus the transaction id it parses from the accompanying plaintext ProvenTransaction. Nothing the submitter controls enters the associated data, so a mismatched key_id cannot influence which key is tried: it only lets the validator answer failed_precondition ("re-fetch the key") instead of an indistinguishable authentication failure.

The unseal happens before the serve lock is taken, so a slow or hung decrypt backend cannot starve the exclusive lock that a backup block subscription needs. The cost is that an already-validated resubmission pays for the unseal before being short-circuited.

After the proof, re-execution, and header checks pass, the validator encrypts the validated inputs under a fresh content key. Golden EHTDH1 protects that content key with the validators' threshold key. The validator stores only the transaction ID and the protected record. It does not store the client envelope or plaintext. A rejected transaction never creates a record.

Oddities and FAQs

Common questions and head scratchers.

Chain MMR

The chain MMR always lags behind the blockchain by one block because otherwise there would be a cyclic dependency between the chain MMR and the block hash:

• chain MMR contains each block's hash as a leaf
• block hash calculation includes the chain MMR's root

To work-around this the inclusion of a block hash in the chain MMR is delayed by one block. Or put differently, block N is responsible for inserting block N-1 into the chain MMR. This does not break blockchain linkage because the block header (and therefore hash) still includes the previous block's hash.