Jobs & lifecycle

Both manifest generation and trajectory generation are async. manifest-create and trajectory-create return job records first; the durable manifest or trajectory only exists once the worker finishes.

The two job kinds

Kind	Produces	Resource type
manifest_generation	One WorldManifest when the job completes	manifest
trajectory_generation	One or more WorldTrajectory records (per --count)	trajectory or trajectory_batch

Single-trajectory jobs record a trajectory resource; multi-trajectory jobs (--count > 1) record a trajectory_batch. The produced resource IDs are carried on the completed job’s result payload.

Jobs go through the same status progression: queued → running → completed | failed | cancelled.

Waiting

job-wait polls until the job reaches a terminal status:

dn worlds job-wait <job-id>

It prints the terminal record and exits non-zero for any status that isn’t completed, so it’s safe to use in scripts:

dn worlds job-wait "$job_id" || { echo "generation failed"; exit 1; }

--poll-interval-sec adjusts the polling rate (default 5s). --timeout-sec bounds the wait; the command exits with an error if the timeout elapses, but the job itself keeps running on the server.

Listing and inspecting

dn worlds job-list --status running
dn worlds job-list --kind trajectory_generation
dn worlds job-get <job-id>

job-list paginates and filters by kind, status, project, or creator. job-get returns the full record including progress, the produced resource ID (once the job completes), and any error message.

In the web app, a project’s Labs tab lists jobs among its runs, under Manifests or Trajectories for what each job makes. A job keeps its own row while it’s queued, running, failed or cancelled; once it completes, the manifest or trajectories it made take its place, and their detail links back to the job. Filter Manifests or Trajectories by Status to find jobs in a given state. The list refreshes every ten seconds while any job is queued or running — useful for watching a batch of trajectory jobs at once. The current navigation’s Worlds → Jobs tab shows the same jobs in a list of their own until late November 2026, when the new navigation becomes permanent. On a self-hosted deployment, the tab goes away when it upgrades to that release.

Open a completed trajectory generation job and select Download ATIF to export its published trajectory dataset. The button appears after dataset publication succeeds. A trajectory job’s detail also counts how many of its trajectories reached the goal.

Cancellation

Cancellation differs by status:

• Queued jobs cancel immediately. The worker never picks them up.
• Running jobs record a cancellation request. The worker drops its lease at the next safe point and the job settles to cancelled after cleanup — which can take a few seconds while the backend tears down the sandbox.

dn worlds job-cancel <job-id>

In the web app, select Cancel job on a queued or running job’s detail. The job reads Cancelling until the worker stops it.

Running jobs carry a short lease that the worker heartbeats. If the worker loses its lease (crash, deployment, network partition), the job is requeued rather than left silently hanging.

Note

Cancellation is a request, not an instant kill. If the worker is mid-generation when you cancel, the job may still produce partial artifacts before settling. Inspect the terminal job record for any partial resource ID.

Debugging missing resources

If you submitted manifest-create or trajectory-create and can’t find the result, check the job before assuming the resource failed to exist. Most of the time the job is still running or terminated with an error.

The flow:

# Given a job ID from a create command
dn worlds job-get <job-id>

• status=queued or running — not finished yet. Keep waiting or job-wait.
• status=completed — the resource_id points at the produced manifest or trajectory. The result payload on completed trajectory jobs also carries dataset_ref, trajectory_ids, and sample artifact paths — the same values a job’s detail shows in the web app.
• status=failed — the error field has the reason.
• status=cancelled — either user-initiated or a worker cleanup; check error for context.

Heartbeats and workers

Worker leases cap at five minutes with heartbeats, so a dead worker frees its jobs within one lease window. Jobs pinned to sandboxes (trajectory jobs, especially in agent mode) are linked via WorldJobSandbox records — useful for correlating a job to its backing sandbox if you need to inspect sandbox state during a run.

What’s next

• CLI reference: dn worlds
• Workspace-scoped behavior: Manifests — projects