Jobs & lifecycle

Both manifest generation and trajectory generation are async. manifest-create and trajectory-create return job records first; the durable manifest or trajectory only exists once the worker finishes.

The two job kinds

Kind	Produces	Resource type
manifest_generation	One WorldManifest when the job completes	manifest
trajectory_generation	One or more WorldTrajectory records (per --count)	trajectory or trajectory_batch

Single-trajectory jobs record a trajectory resource; multi-trajectory jobs (--count > 1) record a trajectory_batch. The produced resource IDs are carried on the completed job’s result payload.

Jobs go through the same status progression: queued → running → completed | failed | cancelled.

Waiting

job-wait polls until the job reaches a terminal status:

dn worlds job-wait <job-id>

It prints the terminal record and exits non-zero for any status that isn’t completed, so it’s safe to use in scripts:

dn worlds job-wait "$job_id" || { echo "generation failed"; exit 1; }

--poll-interval-sec adjusts the polling rate (default 5s). --timeout-sec bounds the wait; the command exits with an error if the timeout elapses, but the job itself keeps running on the server.

Listing and inspecting

dn worlds job-list --status running
dn worlds job-list --kind trajectory_generation
dn worlds job-get <job-id>

job-list paginates and filters by kind, status, project, or creator. job-get returns the full record including progress, the produced resource ID (once the job completes), and any error message.

The web app’s Worlds → Jobs tab shows the same list with live polling every ten seconds — useful for watching a batch of trajectory jobs at once.

Cancellation

Cancellation differs by status:

• Queued jobs cancel immediately. The worker never picks them up.
• Running jobs record a cancellation request. The worker drops its lease at the next safe point and the job settles to cancelled after cleanup — which can take a few seconds while the backend tears down the sandbox.

dn worlds job-cancel <job-id>

Running jobs carry a short lease that the worker heartbeats. If the worker loses its lease (crash, deployment, network partition), the job is requeued rather than left silently hanging.

Note

Cancellation is a request, not an instant kill. If the worker is mid-generation when you cancel, the job may still produce partial artifacts before settling. Inspect the terminal job record for any partial resource ID.

Debugging missing resources

If you submitted manifest-create or trajectory-create and can’t find the result, check the job before assuming the resource failed to exist. Most of the time the job is still running or terminated with an error.

The flow:

# Given a job ID from a create command
dn worlds job-get <job-id>

• status=queued or running — not finished yet. Keep waiting or job-wait.
• status=completed — the resource_id points at the produced manifest or trajectory. The result payload on completed trajectory jobs also carries dataset_ref, trajectory_ids, and sample artifact paths — the same values the Jobs tab surfaces.
• status=failed — the error field has the reason.
• status=cancelled — either user-initiated or a worker cleanup; check error for context.

Heartbeats and workers

Worker leases cap at five minutes with heartbeats, so a dead worker frees its jobs within one lease window. Jobs pinned to sandboxes (trajectory jobs, especially in agent mode) are linked via WorldJobSandbox records — useful for correlating a job to its backing sandbox if you need to inspect sandbox state during a run.

What’s next

• CLI reference: dn worlds
• Workspace-scoped behavior: Manifests — projects