Skip to content

Public API

For supported Hugging Face Trainer, Lightning, and RF-DETR training, start with traceml run train.py. The launcher attaches the integration; you do not need the Python setup calls below. See Integrations for supported paths and explicit setup in other frameworks.

Use this reference for custom-loop instrumentation, summary export, and advanced launches. Import the stable core API from traceml_ai:

import traceml_ai as traceml

The reference below documents every symbol in traceml_ai.__all__.

Stable Core API

traceml.__version__

A string identifying the installed TraceML version. It is useful when recording the environment for a run or bug report.

Lifecycle

init

init(*, mode: str = 'auto', patch_dataloader: Optional[bool] = None, patch_forward: Optional[bool] = None, patch_backward: Optional[bool] = None, patch_h2d: Optional[bool] = None, disabled: Optional[bool] = None, ui_mode: Optional[str] = None, interval: Optional[float] = None, logs_dir: Optional[str] = None, enable_logging: Optional[bool] = None, session_id: Optional[str] = None, aggregator_host: Optional[str] = None, aggregator_port: Optional[int] = None, connect_timeout_sec: float = 10.0, connect_retry_interval_sec: float = 0.25, on_missing_aggregator: Optional[str] = None) -> TraceMLInitConfig

Initialize TraceML and start its runtime for this process.

Parameters:

Name Type Description Default
mode str

Instrumentation policy: "auto" (default), "manual", or "selective". "custom" is accepted as an alias for selective.

'auto'
patch_dataloader Optional[bool]

Per-feature automatic-instrumentation settings. They may be supplied only with mode="selective".

None
patch_forward Optional[bool]

Per-feature automatic-instrumentation settings. They may be supplied only with mode="selective".

None
patch_backward Optional[bool]

Per-feature automatic-instrumentation settings. They may be supplied only with mode="selective".

None
patch_h2d Optional[bool]

Per-feature automatic-instrumentation settings. They may be supplied only with mode="selective".

None
disabled Optional[bool]

Disable TraceML entirely for this process.

None
ui_mode Optional[str]

Display mode: "summary", "cli", or "dashboard".

None
interval Optional[float]

Sampling interval in seconds for runtime telemetry.

None
logs_dir Optional[str]

Directory for session artifacts.

None
enable_logging Optional[bool]

Enable TraceML file logging.

None
session_id Optional[str]

Identifier shared by workers that write one run's artifacts.

None
aggregator_host Optional[str]

Aggregator endpoint for direct launches. traceml run configures it automatically.

None
aggregator_port Optional[str]

Aggregator endpoint for direct launches. traceml run configures it automatically.

None
connect_timeout_sec float

Bounded connection wait and retry interval for the aggregator.

10.0
connect_retry_interval_sec float

Bounded connection wait and retry interval for the aggregator.

10.0
on_missing_aggregator Optional[str]

Missing-aggregator policy: "warn" prints one stderr warning and continues with tracing disabled; "raise" fails the call. Resolution is explicit argument, then TRACEML_ON_MISSING_AGGREGATOR, then "warn".

None

Returns:

Type Description
TraceMLInitConfig

The effective instrumentation configuration. Fail-open initialization returns a disabled configuration.

Raises:

Type Description
ValueError

If mode or selective patch settings are invalid.

RuntimeError

If initialization conflicts with an existing configuration, or when on_missing_aggregator="raise" is selected and setup fails.

start

start(*, mode: str = 'auto', patch_dataloader: Optional[bool] = None, patch_forward: Optional[bool] = None, patch_backward: Optional[bool] = None, patch_h2d: Optional[bool] = None, disabled: Optional[bool] = None, ui_mode: Optional[str] = None, interval: Optional[float] = None, logs_dir: Optional[str] = None, enable_logging: Optional[bool] = None, session_id: Optional[str] = None, aggregator_host: Optional[str] = None, aggregator_port: Optional[int] = None, connect_timeout_sec: float = 10.0, connect_retry_interval_sec: float = 0.25, on_missing_aggregator: Optional[str] = None) -> TraceMLInitConfig

Backward-compatible alias for :func:init.

This function accepts the same parameters and has the same missing- aggregator behavior as :func:init. New code may use either name; prefer init for consistency with framework integration entry points.

Step boundary

trace_step

trace_step(model: Module) -> ContextManager[None]

Return the context manager that defines one training-step boundary.

Parameters:

Name Type Description Default
model Module

The PyTorch model being trained. TraceML uses it to collect step-memory measurements and associate automatic phase timing with the step.

required

Returns:

Type Description
ContextManager[None]

Use around the work from optimizer.zero_grad(...) through optimizer.step().

Notes

When tracing is disabled, the context manager is a no-op. Exceptions from the training body still propagate normally.

End-of-run summaries

summary

summary(*, timeout_sec: float = 30.0, poll_interval_sec: float = 0.1, print_text: bool = False, rank0_only: bool = True) -> Optional[Dict[str, Any]]

Return a compact tracker-friendly summary for the active session.

Parameters:

Name Type Description Default
timeout_sec float

Maximum time to wait for the aggregator to publish the summary.

30.0
poll_interval_sec float

Delay between checks for the aggregator response.

0.1
print_text bool

Print the matching final_summary.txt artifact when available.

False
rank0_only bool

Return None on non-primary ranks when true.

True

Returns:

Type Description
dict or None

A flat projection of the final summary for experiment trackers, or None for a non-primary rank when rank0_only=True.

Raises:

Type Description
RuntimeError

If session history is unavailable, the aggregator returns an error, or the request times out.

final_summary

final_summary(*, timeout_sec: float = 30.0, poll_interval_sec: float = 0.1, print_text: bool = False, rank0_only: bool = True) -> Optional[Dict[str, Any]]

Return the full final-summary payload for the active session.

Parameters:

Name Type Description Default
timeout_sec float

Maximum time to wait for the aggregator to publish the summary.

30.0
poll_interval_sec float

Delay between checks for the aggregator response.

0.1
print_text bool

Print the matching final_summary.txt artifact when available.

False
rank0_only bool

Return None on non-primary ranks when true.

True

Returns:

Type Description
dict or None

The parsed final_summary.json payload, or None for a non-primary rank when rank0_only=True.

Raises:

Type Description
RuntimeError

If session history is unavailable, the aggregator returns an error, or the request times out.

Manual instrumentation helpers

Use these for manual or selective instrumentation. init(mode="auto") already owns the matching PyTorch paths, so wrappers reject duplicate instrumentation. The exception is wrap_dataloader_fetch(...) for a custom non-PyTorch iterator that automatic DataLoader instrumentation cannot observe. When TraceML is disabled, each wrapper is an identity no-op. Otherwise, invalid targets raise their documented TypeError before initialization and ownership are checked.

wrap_dataloader_fetch

wrap_dataloader_fetch(obj: Any) -> Any

Wrap a loader or iterator to time step-scoped batch fetches.

Parameters:

Name Type Description Default
obj Any

A loader implementing __iter__ or an iterator implementing __next__.

required

Returns:

Type Description
Any

A loader or iterator proxy that records fetch time.

Raises:

Type Description
TypeError

If obj is neither a loader nor an iterator.

RuntimeError

If TraceML has not been initialized, or if automatic instrumentation already owns the supplied torch DataLoader or iterator.

wrap_forward

wrap_forward(model: Module) -> nn.Module

Wrap one model instance's forward(...) for step-scoped timing.

Parameters:

Name Type Description Default
model Module

The PyTorch module to instrument.

required

Returns:

Type Description
Module

The same model instance after its forward method is wrapped in place.

Raises:

Type Description
TypeError

If model is not a PyTorch module with a callable forward.

RuntimeError

If TraceML has not been initialized, automatic forward instrumentation is active, or the instance cannot be wrapped safely.

wrap_backward

wrap_backward(loss: Any) -> Any

Wrap a loss-like object so .backward(...) records timing.

Parameters:

Name Type Description Default
loss Any

An object with a callable backward method, normally a loss tensor.

required

Returns:

Type Description
Any

A proxy that forwards attributes and times its backward call.

Raises:

Type Description
TypeError

If loss has no callable backward method.

RuntimeError

If TraceML has not been initialized or automatic backward instrumentation is active.

wrap_optimizer

wrap_optimizer(optimizer: Any) -> Any

Wrap one optimizer instance's .step(...) for timing.

Parameters:

Name Type Description Default
optimizer Any

An optimizer with a callable step method.

required

Returns:

Type Description
Any

The same optimizer instance after its step method is wrapped in place.

Raises:

Type Description
TypeError

If optimizer has no callable step method.

RuntimeError

If TraceML has not been initialized, automatic optimizer instrumentation is active, or the instance cannot be wrapped safely.

wrap_h2d

wrap_h2d(obj: Any) -> Any

Wrap an object so qualifying CPU-to-CUDA .to(...) calls are timed.

Parameters:

Name Type Description Default
obj Any

A tensor or batch object with a callable .to(...) method.

required

Returns:

Type Description
Any

A proxy that forwards attributes and returns the transferred object on its first .to(...) call.

Raises:

Type Description
TypeError

If obj has no callable .to(...) method.

RuntimeError

If TraceML has not been initialized or automatic H2D instrumentation is active.

Notes

Use this after initialization in manual mode, or in selective mode with automatic H2D timing disabled.

CLI

TraceML installs the traceml command:

traceml run train.py                  # training diagnosis and saved reports
traceml run train.py --mode=cli       # live terminal view
traceml run train.py --mode=dashboard # live browser view
traceml watch train.py                # system/process visibility
traceml serve                         # standalone aggregator for direct launches

Dashboard mode requires pip install "traceml-ai[dashboard]".

Summary mode is the default for every topology. Live CLI and dashboard modes are intended for single-node runs. watch does not install automatic trainer instrumentation or provide step timing.

Use traceml run --help for launch flags. See Distributed Training for multi-node commands, Compare Runs for saved report comparison, and Regression Guard for CI thresholds.

Training output

run and watch save training stdout and stderr by default under logs/<run-name>/nodes/node_<node-rank>/. Summary and dashboard modes mirror both streams; CLI mode shows a bounded stderr excerpt after failure.

Use --no-save-training-output to inherit the terminal instead. Captured streams are pipes, so isatty() is false. TraceML does not replace Python's sys.stdout or sys.stderr. Aggregator diagnostics are saved separately. See Training Crashes for artifact paths and failure behavior.

History and configuration

run, watch, and serve accept --history-retention DURATION. The default is 30m; bare values are seconds, with s, m, h, and d suffixes accepted. Use --no-history on run or watch only with a live display mode. Summary mode, HTML reports, and summary APIs require history.

The retention setting is also available as history_retention in traceml.yaml and TRACEML_HISTORY_RETENTION. CLI settings take precedence over environment variables, YAML, and built-in defaults, in that order.

How history retention works

Step Time and Step Memory are pruned through a shared step boundary aligned across every expected rank in both streams and older than the chosen duration. System, Process, and GPU history use that step's timestamp. Later arrivals at or before a deleted step or timestamp are dropped before insertion.

Direct Launch with traceml serve

For a direct python or torchrun launch, start the aggregator separately and use the manual setup for your integration in the script. Custom PyTorch loops use traceml.init() and traceml.trace_step(); framework integrations may require their own initializer and callback or handler.

# terminal 1
traceml serve --aggregator-host 127.0.0.1 --aggregator-port 29765

# terminal 2: script already contains its explicit TraceML setup
python train.py

serve does not launch training or attach framework callbacks. Stop it after training to finalize its reports. Training stdout/stderr remain owned by your terminal, scheduler, or container runtime.

Multi-node direct launches and serve flags

Bind the aggregator on a reachable address and set its endpoint on every training node:

traceml serve --aggregator-bind-host 0.0.0.0 --aggregator-host <node0-ip> \
  --aggregator-port 29765 --nnodes <N> --nproc-per-node <M>

TRACEML_AGGREGATOR_HOST=<node0-ip> TRACEML_AGGREGATOR_PORT=29765 \
  torchrun ... train.py

Workers default to 127.0.0.1, so non-aggregator nodes need the reachable node-0 endpoint. Configure the expected world size to match your workers.

Flag Meaning
--aggregator-host Worker connection address; default 127.0.0.1.
--aggregator-bind-host Bind address; use 0.0.0.0 for multi-node.
--aggregator-port Telemetry port; default 29765.
--nnodes / --nproc-per-node Expected world size for finalization.
--mode summary (default), cli, or dashboard.
--logs-dir Session log directory.
--run-name / --session-id Shared run identity; choose a fresh name.
--history-retention History duration; default 30m.

An existing run folder is not overwritten. Telemetry with a different explicit worker session ID is ignored. Workers that generate their own ID, such as a plain python train.py, are still admitted.

Missing-aggregator behavior

Launcher startup and in-script initialization have different defaults:

Entry point Default if the aggregator is unavailable
traceml run / traceml watch Stop before launching training.
Explicit traceml.init() Retry within a bounded timeout, warn, and continue with tracing disabled.

To let the launcher continue without telemetry:

traceml run train.py --on-missing-aggregator=warn

To require telemetry during explicit setup:

traceml.init(on_missing_aggregator="raise")

The policy resolves from the explicit flag or argument, then TRACEML_ON_MISSING_AGGREGATOR, then the entry point's default. It is not a traceml.yaml setting. Calls that require an aggregator, including traceml.summary(), still fail when telemetry is unavailable.

An occupied telemetry endpoint is also a startup failure. Stop the earlier aggregator or choose another --aggregator-port.

After training starts, telemetry failures are reported separately and do not replace the training exit code. Only the aggregator-owning launcher reports final telemetry health; in multi-node runs this is node 0. See Training Crashes for the manifest and diagnostics.

Direct-launch configuration

Runtime settings resolve from explicit init() arguments, environment variables, traceml.yaml, then built-in defaults. For display mode, the Python argument is ui_mode; init(mode=...) selects instrumentation behavior.

Aggregator host and port resolve from explicit arguments, environment variables, then defaults. Run identity and aggregator endpoints are not read from YAML. See the function references above for accepted arguments.

Framework Integrations

Framework integrations are separate from the stable core API above. Use the matching integration guide for installation and runtime requirements.

Hugging Face

For a standard transformers.Trainer, use traceml run train.py; the launcher attaches TraceMLTrainerCallback when training starts. Direct launches and custom training loops can still call the integration init() and register the callback manually. See the Hugging Face guide for setup and limitations.

traceml_ai.integrations.huggingface.init

init()

Initialize TraceML for Hugging Face Trainer runs.

For a direct launch, call once before constructing the Trainer, then register TraceMLTrainerCallback. init() makes TraceML's process-wide instrumentation explicit: PyTorch DataLoader fetch timing, the H2D Tensor.to patch, and the forward/backward/optimizer auto-timers that trace_step arms inside each bracketed step.

The callback is a per-step bracket and cannot install these process-wide patches on its own; the auto-timers it arms are no-ops unless the matching patch is installed. For manual setup, init() ensures that the DataLoader fetch patch in particular is installed deterministically rather than relying on import order. traceml run calls this path automatically for a standard Trainer. It also installs the narrow Trainer lifecycle guard that aborts unfinished steps before an automatic batch-size retry, plus the collection hook needed to isolate training Input Wait and observe Accelerate's pre-callback H2D transfers. This mirrors the PyTorch Lightning integration's init(); HF uses mode="auto" because trace_step drives forward/backward timing through the patch-gated auto-timers, whereas Lightning's callback owns that timing directly.

traceml_ai.integrations.huggingface.TraceMLTrainerCallback

TraceMLTrainerCallback()

Bases: TrainerCallback if HAS_TRANSFORMERS else object

Hugging Face Trainer integration for TraceML.

Register with Trainer(..., callbacks=[TraceMLTrainerCallback()]).

The callback brackets TraceML's trace_step context manager: it opens trace_step in on_step_begin and completes it in on_step_end. The lifecycle guard installed by :func:init aborts an open context when the Trainer attempt exits early. trace_step owns the step memory tracker, step counter, auto-timers, and capture publication.

One completed TraceML step corresponds to one HF accumulation/update boundary (on_step_end), including when AMP skips the parameter update. on_substep_end does not advance the counter: forward and backward events from the actual micro-batches in the group fold into one step, including a shorter final group. Optimizer events describe calls that actually run; their count does not drive TraceML's step counter.

TraceML step IDs remain process-local, so their increments match HF's global_step increments during recorded, completed groups; their absolute values need not match after checkpoint resume. See the HF integration docs for the input-timing limitations and lifecycle behavior.

PyTorch Lightning

Run a standard Lightning script with traceml run train.py for automatic initialization and callback attachment. Both Lightning namespaces are supported. For direct launches, the manual path uses both init() and TraceMLCallback(). Compatible existing setup is reused under automatic launch. See the Lightning guide for step semantics and limits.

traceml_ai.integrations.lightning.init

init()

Initialize TraceML for PyTorch Lightning runs.

Lightning owns the training loop, so TraceMLCallback owns step boundaries, capture completion, and framework hook integration. The integration init enables DataLoader fetch timing plus the H2D Tensor.to patch. The callback opens the traced step around Lightning's batch transfer (so H2D is inside it), turns H2D timing on only there, and observes outermost module calls during training_step for forward timing. Fetches of non-training loaders are excluded.

traceml_ai.integrations.lightning.TraceMLCallback

TraceMLCallback()

Bases: Callback

Official TraceML Callback for PyTorch Lightning.

One TraceML step is one Lightning optimizer-update boundary. Without gradient accumulation this is one training batch. With automatic gradient accumulation, the capture remains open across the group's micro-batches; their repeated timing events are aggregated by StepTimeSampler and the memory tracker reports the peak across the group. Fetches of validation, sanity-check, test and predict loaders are kept out of Input Wait. A group that raises is discarded, not published.

on_exception

on_exception(trainer, pl_module, exception)

Abandon the current incomplete step without publishing it.

Lightning calls this hook, not teardown, when training raises. The open timing regions are closed; every micro-batch accumulated since the prior update, plus the pending memory snapshot, is discarded. The step counter is left where it was, the wrappers are restored, and the user's exception propagates untouched.

RF-DETR

Run an ordinary RF-DETR model.train() script with traceml run train.py for automatic attachment. Manual setup needs only rfdetr.init() before training; the integration supplies its specialized callback. Compatible existing setup is reused. See the RF-DETR guide for supported modes.

traceml_ai.integrations.rfdetr.init

init()

Enable TraceML for subsequent RF-DETR model.train() calls.

traceml run activates this adapter automatically. For manual setup, call once before training in each worker. RF-DETR remains optional. The process-wide factory hook is idempotent and leaves standalone evaluation unchanged; per-fit model and transfer wrappers are restored by the callback.

Supports eager detection on CPU/CUDA with single-device or torchrun DDP execution. The native rfdetr fit CLI does not use this factory hook. Returns the effective TraceML initialization configuration.

MONAI

Call init() before building the trainer, then pass TraceMLHandler() in train_handlers. See the MONAI guide for the seam each phase is measured from.

traceml_ai.integrations.monai.init

init()

Initialize TraceML for MONAI SupervisedTrainer runs.

Call once before building the trainer, then pass TraceMLHandler() in train_handlers. H2D timing is armed; the DataLoader fetch patch is not. The handler times Input Wait from the engine's own fetch events on the training thread, so a loader that fetches on a background thread, such as ThreadDataLoader, is timed by what the loop waited.

traceml_ai.integrations.monai.TraceMLHandler

TraceMLHandler()

MONAI handler that times SupervisedTrainer steps for TraceML.

It traces one trainer that runs SupervisedTrainer._iteration itself, including a subclass that inherits it. Any other engine gets one warning and nothing attached. Each run first drops step events recorded earlier in the process. Every exit closes open regions, drops an unfinished step and restores prepare_batch. Nothing is registered on Events.EXCEPTION_RAISED, so the user's exception always propagates.

attach

attach(engine) -> None

Called by MONAI for each engine the handler is listed on.

Ray Train

Use TraceMLTorchTrainer and its configuration for explicit worker-side setup. See the Ray Train guide.

traceml_ai.integrations.ray.TraceMLTorchTrainer

TraceMLTorchTrainer(train_loop_per_worker: TrainLoop, *, train_loop_config: Optional[Dict[str, Any]] = None, traceml_config: Optional[TraceMLRayConfig] = None, **torch_trainer_kwargs: Any)

TraceML wrapper for ray.train.torch.TorchTrainer.

The wrapper deliberately uses composition instead of subclassing. Ray's TorchTrainer still owns training orchestration; TraceML only adds:

  • one aggregator actor before fit()
  • one runtime wrapper inside each worker
  • best-effort aggregator shutdown after fit() completes or fails

last_endpoint property

last_endpoint: Optional[AggregatorEndpoint]

Aggregator endpoint used by the most recent fit() call.

fit

fit() -> Any

Run Ray Train with TraceML telemetry enabled.

traceml_ai.integrations.ray.TraceMLRayConfig dataclass

TraceMLRayConfig(mode: str = 'summary', profile: str = 'run', init_mode: str = 'auto', patch_dataloader: Optional[bool] = None, patch_forward: Optional[bool] = None, patch_backward: Optional[bool] = None, patch_h2d: Optional[bool] = None, logs_dir: str = './logs', session_id: str = '', sampler_interval_sec: float = DEFAULT_INTERVAL_SEC, history_retention: DurationValue = '30m', bind_host: str = '0.0.0.0', port: int = 0, stop_timeout_sec: float = 5.0)

TraceML settings used by the Ray Train integration.

Parameters:

Name Type Description Default
mode str

TraceML display/reporting mode. "summary" is the default for Ray because distributed worker logs are often noisy.

'summary'
profile str

TraceML sampler profile. Public Ray integration uses the normal "run" profile.

'run'
init_mode str

Instrumentation mode passed to traceml.init() inside each worker.

'auto'
patch_dataloader Optional[bool]

Selective-mode-only override for DataLoader fetch patching.

None
patch_forward Optional[bool]

Selective-mode-only override for forward-pass patching.

None
patch_backward Optional[bool]

Selective-mode-only override for backward-pass patching.

None
patch_h2d Optional[bool]

Selective-mode-only override for host-to-device transfer patching.

None
logs_dir str

Directory where TraceML writes session logs and summary artifacts.

'./logs'
session_id str

Optional explicit TraceML session id. If omitted, a unique Ray session id is generated for each fit() call.

''
sampler_interval_sec float

Background sampler cadence in seconds. The aggregator actor uses the same value for its live UI refresh cadence; TCP ingestion is immediate.

DEFAULT_INTERVAL_SEC
history_retention DurationValue

Raw telemetry analysis history as seconds or a duration such as "30m" or "2h".

'30m'
bind_host str

Host interface used by the aggregator actor. Use "0.0.0.0" for multi-node Ray clusters so workers on other nodes can connect.

'0.0.0.0'
port int

Aggregator TCP port. 0 asks the OS to pick a free port.

0
stop_timeout_sec float

Best-effort timeout for aggregator shutdown.

5.0