Trainer control

start, wait, cancel, abortSignal, and how reconnects work.

createTrainer returns a Trainer object with three methods. arkor start and Studio's "Run training" button both call start() followed by wait(). You only call them yourself when you wire training into your own code (a server, a script, a custom CLI).

Signature#

interface Trainer {
  readonly name: string;
  start(): Promise<{ jobId: string }>;
  wait(): Promise<TrainingResult>;
  cancel(): Promise<void>;
}

interface TrainingResult {
  job: TrainingJob;
  artifacts: unknown[];
}

start()#

const { jobId } = await trainer.start();

• Submits the job to the cloud API and resolves once the backend has accepted it. The returned jobId is the same id you see in Studio and in the SDK's TrainingJob.id.
• Idempotent: calling start() a second time on the same trainer returns the same jobId without resubmitting (packages/arkor/src/core/trainer.ts:275-289).
• Does not open the event stream and does not dispatch any callbacks. wait() is what does that.

wait()#

const { job, artifacts } = await trainer.wait();

• Opens the SSE event stream for the run, dispatches each frame to your callbacks, and resolves with the terminal TrainingResult when the stream reports training.completed or training.failed.
• Calls start() for you if you have not called it yet.
• All five lifecycle callbacks fire from inside wait(). If you call start() without wait(), no callbacks run, even though the run continues on the backend.
• Reconnects on transient SSE errors (see below).

cancel()#

await trainer.cancel();

• If start() has not been called yet, cancel() is a no-op (early return at :388-389).
• Otherwise it sends a cancel request to the backend.
• Best-effort. The SDK does not short-circuit on terminal status; if the run already completed, failed, or was cancelled, the backend may return a non-2xx and cancel() rejects. Wrap in try / catch if you call it speculatively.

abortSignal#

const controller = new AbortController();

const trainer = createTrainer({
  name: "with-timeout",
  model: "unsloth/gemma-4-E4B-it",
  dataset: { type: "huggingface", name: "arkorlab/triage-demo" },
  abortSignal: controller.signal,
});

// Later, from anywhere:
controller.abort();

abortSignal controls only your local wait() loop. When the signal aborts:

1. The pending SSE fetch is aborted (trainer.ts:325-328).
2. Any active reconnect-backoff delay rejects with signal.reason (trainer.ts:178).
3. handleFailure re-throws when the signal is aborted (trainer.ts:308).
4. wait() therefore rejects, not resolves, when you abort.

It does not call cancel() and does not send anything to the backend. The job keeps using GPU time on the managed side.

If you want both effects (stop waiting locally and stop the run on the backend), do them separately:

try {
  await trainer.wait();
} catch (err) {
  if (controller.signal.aborted) {
    // expected: we asked wait() to stop
  } else {
    throw err;
  }
}
await trainer.cancel(); // best-effort; see above

Use abortSignal for "I no longer care about waiting on this run" (request timeout, parent process exit). Use cancel() for "stop the run on the backend".

Reconnects#

wait() keeps the SSE stream alive across transient failures by default:

• A clean stream EOF after at least one real, parseable data frame (keepalive ping frames and malformed frames do not count as progress) triggers an immediate reconnect at the base delay (initialReconnectDelayMs, default 1000 ms) without counting against the failure budget. The stream resumes via Last-Event-ID.
• A connect error, or a stream EOF that yielded no parseable data frame (including one that sent only keepalive pings or unparseable garbage), counts as a failure and goes through handleFailure: exponential backoff via initialReconnectDelayMs * 2 ** attempt, with the per-attempt delay clamped at maxReconnectDelayMs (default 60 000 ms) and the consecutive-failure count capped at maxReconnectAttempts. This is deliberate: a broken intermediary that only ever emits a ping (or garbage) and then EOFs is counted so it can't loop forever at the base delay.
• Permanent connect errors are the exception to the retry loop: a 401 / 403 (auth), 410 (gone), or 426 (upgrade required) response rejects wait() immediately, since it would fail identically on every reconnect. A 404 is treated as transient and retries (a just-created job's event stream may not be visible yet), as do 408, 429, and 5xx.
• maxReconnectAttempts defaults to undefined (unlimited consecutive failures). It is not configurable through TrainerInput; the only way to set it (along with reconnectDelayMs and maxReconnectDelayMs) is the second context argument to createTrainer, annotated @internal and subject to change. For most projects this means transient SSE failures are silently retried for as long as the job runs.

This path is only for transport failures. A thrown user callback does not go through it: it rejects wait() immediately, without any reconnect or retry (see Lifecycle callbacks § Exception handling). Routing a throw through reconnect would resume past the failing event's already-advanced Last-Event-ID, swallowing your error. If you need deterministic, non-fatal error handling, catch inside the callback rather than relying on wait() to reject.

Two-process pattern#

A common shape for non-CLI use is to keep a long-lived trainer reference and let your own code orchestrate start, wait, and cancel:

import { createTrainer } from "arkor";

const controller = new AbortController();
process.on("SIGINT", () => controller.abort());

const trainer = createTrainer({
  /* ... */
  abortSignal: controller.signal, // wired so abort() actually rejects wait()
});

const { jobId } = await trainer.start();
console.log(`Started ${jobId}`);

try {
  const { artifacts } = await trainer.wait();
  console.log(`Finished with ${artifacts.length} artifact(s).`);
} catch (err) {
  if (controller.signal.aborted) {
    await trainer.cancel().catch(() => {});
    throw new Error("Aborted by signal");
  }
  throw err;
}

This is functionally what arkor start does, minus the entry resolution from runTrainer.

See also#

• createTrainer for the input shape that produces this Trainer
• Lifecycle callbacks for what fires from inside wait()
• runTrainer for the entry-resolution helper that wraps start() + wait()
• Programmatic runs recipe for a full server / script wiring