Long-Running Operations¶

This recipe shows how an application can expose a remote long-running operation through Core tools without confusing the agent Run with the remote operation.

Core owns the local Run, tool invocation, policy, and checkpoint projections. The remote execution system owns the operation ID, accepted/running state, cancel confirmation, and final effect evidence. They are related by stable references, not one shared state machine.

Required Distinctions¶

Concept	Meaning
Run canceled	The local agent loop stopped; the remote operation is not necessarily stopped
Cancel requested	A request was sent to the execution authority; cancellation is not confirmed
Cancel confirmed	The execution authority reports the operation stopped
Accepted	The execution authority accepted the operation
Completed	The execution authority reports the operation finished
Effect verified	Environment or business evidence confirms the expected effect

Do not collapse these into one boolean. In particular, a Run error does not prove that the remote operation failed, and a completion event does not prove a business effect committed.

Pattern¶

Use a stable business idempotency key. A new Run ID or model tool-call ID is not automatically suitable.
Submit through a tool; persist the returned operation reference.
On a lost response or restart, query the execution authority by the idempotency key or operation ID before submitting again.
Send cancellation separately and record the authority's confirmation.
Keep final effect evidence in the executing system or an effect ledger.
Do not claim exactly-once from a checkpoint or an in-process deduplication map.

Tested Example¶

The runnable example is the separate L4 recipe module examples/recipes/long-running-go. From that directory:

GOWORK=off go test -mod=readonly -run LongRunning -v ./...

Check	Expected outcome
Duplicate submit with the same key	same operation ID; one effect
Lost response recovery	query/submit with the same key returns the existing operation; no duplicate
Cancel request	state is `cancel_requested`, not `canceled`
Cancel confirmation	authority moves the operation to `canceled` once
Run cancellation	local Run stops; remote state remains `accepted`; no false remote-cancel claim
File-backed restart	operation, idempotency key, cancel request, and confirmation survive a new service instance
Effect verification	`completed` is not `effect_verified`; verification requires evidence and is persisted
Effect receipt	evidence must carry a verifier-issued receipt; forged receipts are rejected and leave the state at `completed`
HTTP response loss	commit succeeds but the response is lost; retry with the same key returns the existing operation
HTTP restart	a new loopback service loads the file and answers status/recovery queries
Cross-process restart	a separate test process serves HTTP; it is killed and restarted with the same file, and the main process recovers the existing operation
Partial temp file	a leftover `.tmp` file is ignored; the last committed state and idempotency key remain intact
Fencing token	a worker holding an old epoch cannot complete; the new owner increments the epoch and completes once
Durable fencing	owner, epoch, and completion survive restart; stale epochs are still rejected
Lease expiry	an expired lease cannot complete; a new owner acquires the next epoch
Lease renewal	renewal before expiry extends the lease and preserves completion
Lease revocation	revocation blocks the old owner; a new owner can reacquire
Lease restart	expiry and revocation survive a restart
Concurrent writers	two independent file writers can overwrite each other; the boundary test records the lost update and is not a pass
Independent verifier process	the execution service calls a separate verifier process and does not hold the signing secret; forged receipts are rejected
Bearer/TLS boundary	missing or wrong bearer token is rejected before any effect; a test TLS certificate is used
Fault before rename	a simulated crash before the rename boundary leaves the last committed state; the temp file is ignored
Fault after rename	a simulated crash after the rename boundary preserves the newly committed state
Valid uncommitted temp	a valid but uncommitted temp file does not override the committed file
Corrupt committed file	startup fails closed instead of silently starting empty
Process crash before rename	the standalone service exits at the commit boundary; restart keeps the last committed state
Process crash after rename	the standalone service exits after commit; restart keeps the new operation

The fixture can persist to a JSON file and expose it over a loopback HTTP service so response loss and restart can be tested through a real transport boundary. It still demonstrates the contract shape, not a durable worker, lease/fencing implementation, or production effect ledger.

A larger load/fault reproduction is available:

PRAXO_LOAD_SUBMISSIONS=100 go test -run TestStandaloneServiceLoadFault -v ./cmd/longrunning-service
python scripts/test_longrunning_service.py

The first command submits 100 concurrent operations, kills the service after the first acknowledgement, restarts it, and verifies every acknowledged operation. The second runs module tests, the load/fault test, and the published-package recipe from a fresh checkout.

Standalone Service¶

The example module also builds a standalone service:

GOWORK=off go build -mod=readonly -o longrunning-service ./cmd/longrunning-service
./longrunning-service -addr 127.0.0.1:8080 -state ./operations.json -token "$PRAXO_TOKEN"

It exposes /healthz, /readyz, /metrics, and the operation endpoints, supports -tls-cert/-tls-key, and shuts down on interrupt. Configuration can come from flags or PRAXO_STATE, PRAXO_TOKEN, PRAXO_VERIFIER_SECRET, and PRAXO_LOCK_DSN. Token and verifier secrets can also use -token-file and -verifier-secret-file (the verifier uses -token-file/-secret-file); file values are re-read on each request, so replacing the file rotates credentials without a restart. Credential issuance remains an external identity-provider responsibility. The cmd integration test builds the binary, starts it on a real TCP port, checks health, readiness, submit, status, concurrent submissions, token rotation, and metrics. A separate cmd/longrunning-verifier serves /verify with its own bearer token and signing secret, plus health/readiness endpoints. This is still a local fixture: it has no production identity, key rotation, database, or deployment automation.

Lease Backends¶

Mutations are serialized through a configurable lease:

Backend	Flags	Notes
File	`-lock-backend file` (default)	`-lock-path`, `-lock-ttl`; stale locks expire
SQLite	`-lock-backend sqlite -lock-dsn <dsn>`	creates a `praxo_leases` table
Postgres	`-lock-backend postgres -lock-dsn <dsn>`	creates a `praxo_leases` table
None	`-lock-backend none`	single writer only; the boundary test records lost updates

The SQL table stores name, owner, and expires_at in epoch milliseconds. A mutation acquires the lease, reloads the state file, applies the change, commits, and releases. The concurrent-writer store test uses the file lease; SQL lease serialization is tested with SQLite; the Postgres path uses the same statements and driver but needs PRAXO_POSTGRES_DSN for its gated tests. Live Postgres was not completed in the current Windows environment because the local server exited with 0xC0000142, so Postgres remains unverified here. This is example-level coordination, not a production lease service.

Monitoring¶

/metrics exposes praxo_longrunning_http_requests_total, praxo_longrunning_http_errors_total, and praxo_longrunning_operations_total. Example Prometheus scrape and alert rules are in monitoring/prometheus.yml, monitoring/alerts.yml, and monitoring/prometheus.args (15-day retention). python scripts/test_longrunning_monitoring.py validates the configs and metric names. No live collector, retention store, or notification channel is included.

Operations¶

Keep the listen address private or behind TLS/gateway. Health, readiness, and /metrics are unauthenticated.
Pass secrets through PRAXO_TOKEN, PRAXO_VERIFIER_SECRET, and PRAXO_LOCK_DSN instead of command history, or use token/secret files.
Replace token/secret files to rotate credentials without restart. Issuance, revocation notification, and overlap policy remain external.
Restrict the state directory to the service account, back up the committed state file, and keep temp/lock files on the same filesystem.
Prefer file for one host, SQLite for a single shared volume, and Postgres for multiple hosts. TTLs are not a production lease service.
Scrape /metrics and alert on errors/readiness. Use the example monitoring/ config for scrape interval, retention, and alert rules; the counters themselves have no collector, retention, or alerting built in.
Run cmd/longrunning-verifier separately with its own PRAXO_VERIFIER_SECRET and PRAXO_VERIFIER_TOKEN, and keep the secret out of the execution service.

Limits¶

The file store writes a temporary file, syncs it, and renames it over the main file. Deterministic tests inject failure before and after the rename boundary, verify that a valid uncommitted temp file cannot override committed state, and require corrupt committed state to fail closed. This is not a full crash-consistency proof: the directory is not synced, OS-level power loss and committed-file corruption recovery are not exercised, and concurrent multi-process writers remain unsupported.
The HTTP service is a loopback test fixture. Its bearer token is static and its TLS certificate is a test certificate; it has no identity provider, token rotation, tenant binding, rate limiting, or production deployment guarantee.
Restart includes a killed child process with the same file. Process-level crash tests exit the standalone service before and after the rename boundary; OS-level power loss and committed-file corruption recovery remain untested.
In-process or single-file deduplication is not a production exactly-once mechanism. A real service must persist the idempotency key and result with defined concurrency and retention semantics.
A status query must go to the authoritative execution system, not a cached checkpoint.
A cancel request is not a cancel confirmation; the execution authority must report the final state.
The verifier is a separate loopback process fixture, but it uses a static secret and token. A real deployment needs key management, rotation, revocation, and an independently operated verification authority.
The fencing fixture now models lease expiry, renewal, and revocation in a JSON file. The standalone service can serialize mutations with a file, SQLite, or Postgres lease; without a lease, a boundary test records that two independent writers can lose an update. Production lease coordination, trusted clocks, and database operations remain deployment responsibilities.
Effect verification, production leases/fencing, reconciliation, and retention belong to the execution system or a durable workflow product.
This recipe does not provide exactly-once execution.

See governed execution design, the security model, and when not to use Core.