Public API¶
For supported Hugging Face Trainer, Lightning, and RF-DETR training, start
with traceml run train.py. The launcher attaches the integration; you do not
need the Python setup calls below. See Integrations for
supported paths and explicit setup in other frameworks.
Use this reference for custom-loop instrumentation, summary export, and
advanced launches. Import the stable core API from traceml_ai:
import traceml_ai as traceml
The reference below documents every symbol in traceml_ai.__all__.
Stable Core API¶
traceml.__version__¶
A string identifying the installed TraceML version. It is useful when recording the environment for a run or bug report.
Lifecycle¶
init
¶
init(*, mode: str = 'auto', patch_dataloader: Optional[bool] = None, patch_forward: Optional[bool] = None, patch_backward: Optional[bool] = None, patch_h2d: Optional[bool] = None, disabled: Optional[bool] = None, ui_mode: Optional[str] = None, interval: Optional[float] = None, logs_dir: Optional[str] = None, enable_logging: Optional[bool] = None, session_id: Optional[str] = None, aggregator_host: Optional[str] = None, aggregator_port: Optional[int] = None, connect_timeout_sec: float = 10.0, connect_retry_interval_sec: float = 0.25, on_missing_aggregator: Optional[str] = None) -> TraceMLInitConfig
Initialize TraceML and start its runtime for this process.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mode
|
str
|
Instrumentation policy: |
'auto'
|
patch_dataloader
|
Optional[bool]
|
Per-feature automatic-instrumentation settings. They may be supplied
only with |
None
|
patch_forward
|
Optional[bool]
|
Per-feature automatic-instrumentation settings. They may be supplied
only with |
None
|
patch_backward
|
Optional[bool]
|
Per-feature automatic-instrumentation settings. They may be supplied
only with |
None
|
patch_h2d
|
Optional[bool]
|
Per-feature automatic-instrumentation settings. They may be supplied
only with |
None
|
disabled
|
Optional[bool]
|
Disable TraceML entirely for this process. |
None
|
ui_mode
|
Optional[str]
|
Display mode: |
None
|
interval
|
Optional[float]
|
Sampling interval in seconds for runtime telemetry. |
None
|
logs_dir
|
Optional[str]
|
Directory for session artifacts. |
None
|
enable_logging
|
Optional[bool]
|
Enable TraceML file logging. |
None
|
session_id
|
Optional[str]
|
Identifier shared by workers that write one run's artifacts. |
None
|
aggregator_host
|
Optional[str]
|
Aggregator endpoint for direct launches. |
None
|
aggregator_port
|
Optional[str]
|
Aggregator endpoint for direct launches. |
None
|
connect_timeout_sec
|
float
|
Bounded connection wait and retry interval for the aggregator. |
10.0
|
connect_retry_interval_sec
|
float
|
Bounded connection wait and retry interval for the aggregator. |
10.0
|
on_missing_aggregator
|
Optional[str]
|
Missing-aggregator policy: |
None
|
Returns:
| Type | Description |
|---|---|
TraceMLInitConfig
|
The effective instrumentation configuration. Fail-open initialization returns a disabled configuration. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If mode or selective patch settings are invalid. |
RuntimeError
|
If initialization conflicts with an existing configuration, or when
|
start
¶
start(*, mode: str = 'auto', patch_dataloader: Optional[bool] = None, patch_forward: Optional[bool] = None, patch_backward: Optional[bool] = None, patch_h2d: Optional[bool] = None, disabled: Optional[bool] = None, ui_mode: Optional[str] = None, interval: Optional[float] = None, logs_dir: Optional[str] = None, enable_logging: Optional[bool] = None, session_id: Optional[str] = None, aggregator_host: Optional[str] = None, aggregator_port: Optional[int] = None, connect_timeout_sec: float = 10.0, connect_retry_interval_sec: float = 0.25, on_missing_aggregator: Optional[str] = None) -> TraceMLInitConfig
Backward-compatible alias for :func:init.
This function accepts the same parameters and has the same missing-
aggregator behavior as :func:init. New code may use either name; prefer
init for consistency with framework integration entry points.
Step boundary¶
trace_step
¶
trace_step(model: Module) -> ContextManager[None]
Return the context manager that defines one training-step boundary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
The PyTorch model being trained. TraceML uses it to collect step-memory measurements and associate automatic phase timing with the step. |
required |
Returns:
| Type | Description |
|---|---|
ContextManager[None]
|
Use around the work from |
Notes
When tracing is disabled, the context manager is a no-op. Exceptions from the training body still propagate normally.
End-of-run summaries¶
summary
¶
summary(*, timeout_sec: float = 30.0, poll_interval_sec: float = 0.1, print_text: bool = False, rank0_only: bool = True) -> Optional[Dict[str, Any]]
Return a compact tracker-friendly summary for the active session.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout_sec
|
float
|
Maximum time to wait for the aggregator to publish the summary. |
30.0
|
poll_interval_sec
|
float
|
Delay between checks for the aggregator response. |
0.1
|
print_text
|
bool
|
Print the matching |
False
|
rank0_only
|
bool
|
Return |
True
|
Returns:
| Type | Description |
|---|---|
dict or None
|
A flat projection of the final summary for experiment trackers, or
|
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If session history is unavailable, the aggregator returns an error, or the request times out. |
final_summary
¶
final_summary(*, timeout_sec: float = 30.0, poll_interval_sec: float = 0.1, print_text: bool = False, rank0_only: bool = True) -> Optional[Dict[str, Any]]
Return the full final-summary payload for the active session.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout_sec
|
float
|
Maximum time to wait for the aggregator to publish the summary. |
30.0
|
poll_interval_sec
|
float
|
Delay between checks for the aggregator response. |
0.1
|
print_text
|
bool
|
Print the matching |
False
|
rank0_only
|
bool
|
Return |
True
|
Returns:
| Type | Description |
|---|---|
dict or None
|
The parsed |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If session history is unavailable, the aggregator returns an error, or the request times out. |
Manual instrumentation helpers¶
Use these for manual or selective instrumentation. init(mode="auto") already
owns the matching PyTorch paths, so wrappers reject duplicate instrumentation.
The exception is wrap_dataloader_fetch(...) for a custom non-PyTorch iterator
that automatic DataLoader instrumentation cannot observe. When TraceML is
disabled, each wrapper is an identity no-op. Otherwise, invalid targets raise
their documented TypeError before initialization and ownership are checked.
wrap_dataloader_fetch
¶
wrap_dataloader_fetch(obj: Any) -> Any
Wrap a loader or iterator to time step-scoped batch fetches.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
obj
|
Any
|
A loader implementing |
required |
Returns:
| Type | Description |
|---|---|
Any
|
A loader or iterator proxy that records fetch time. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
RuntimeError
|
If TraceML has not been initialized, or if automatic instrumentation
already owns the supplied torch |
wrap_forward
¶
wrap_forward(model: Module) -> nn.Module
Wrap one model instance's forward(...) for step-scoped timing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
The PyTorch module to instrument. |
required |
Returns:
| Type | Description |
|---|---|
Module
|
The same model instance after its |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
RuntimeError
|
If TraceML has not been initialized, automatic forward instrumentation is active, or the instance cannot be wrapped safely. |
wrap_backward
¶
wrap_backward(loss: Any) -> Any
Wrap a loss-like object so .backward(...) records timing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
loss
|
Any
|
An object with a callable |
required |
Returns:
| Type | Description |
|---|---|
Any
|
A proxy that forwards attributes and times its |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
RuntimeError
|
If TraceML has not been initialized or automatic backward instrumentation is active. |
wrap_optimizer
¶
wrap_optimizer(optimizer: Any) -> Any
Wrap one optimizer instance's .step(...) for timing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
optimizer
|
Any
|
An optimizer with a callable |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The same optimizer instance after its |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
RuntimeError
|
If TraceML has not been initialized, automatic optimizer instrumentation is active, or the instance cannot be wrapped safely. |
wrap_h2d
¶
wrap_h2d(obj: Any) -> Any
Wrap an object so qualifying CPU-to-CUDA .to(...) calls are timed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
obj
|
Any
|
A tensor or batch object with a callable |
required |
Returns:
| Type | Description |
|---|---|
Any
|
A proxy that forwards attributes and returns the transferred object on
its first |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
RuntimeError
|
If TraceML has not been initialized or automatic H2D instrumentation is active. |
Notes
Use this after initialization in manual mode, or in selective mode with automatic H2D timing disabled.
CLI¶
TraceML installs the traceml command:
traceml run train.py # training diagnosis and saved reports
traceml run train.py --mode=cli # live terminal view
traceml run train.py --mode=dashboard # live browser view
traceml watch train.py # system/process visibility
traceml serve # standalone aggregator for direct launches
Dashboard mode requires pip install "traceml-ai[dashboard]".
Summary mode is the default for every topology. Live CLI and dashboard modes
are intended for single-node runs. watch does not install automatic trainer
instrumentation or provide step timing.
Use traceml run --help for launch flags. See
Distributed Training for multi-node commands,
Compare Runs for saved report comparison, and
Regression Guard for CI thresholds.
Training output¶
run and watch save training stdout and stderr by default under
logs/<run-name>/nodes/node_<node-rank>/. Summary and dashboard modes mirror
both streams; CLI mode shows a bounded stderr excerpt after failure.
Use --no-save-training-output to inherit the terminal instead. Captured
streams are pipes, so isatty() is false. TraceML does not replace Python's
sys.stdout or sys.stderr. Aggregator diagnostics are saved separately.
See Training Crashes for artifact paths and failure
behavior.
History and configuration¶
run, watch, and serve accept --history-retention DURATION. The default
is 30m; bare values are seconds, with s, m, h, and d suffixes accepted.
Use --no-history on run or watch only with a live display mode.
Summary mode, HTML reports, and summary APIs require history.
The retention setting is also available as history_retention in
traceml.yaml and TRACEML_HISTORY_RETENTION. CLI settings take precedence
over environment variables, YAML, and built-in defaults, in that order.
How history retention works
Step Time and Step Memory are pruned through a shared step boundary aligned across every expected rank in both streams and older than the chosen duration. System, Process, and GPU history use that step's timestamp. Later arrivals at or before a deleted step or timestamp are dropped before insertion.
Direct Launch with traceml serve¶
For a direct python or torchrun launch, start the aggregator separately
and use the manual setup for your integration in the script.
Custom PyTorch loops use traceml.init() and traceml.trace_step(); framework
integrations may require their own initializer and callback or handler.
# terminal 1
traceml serve --aggregator-host 127.0.0.1 --aggregator-port 29765
# terminal 2: script already contains its explicit TraceML setup
python train.py
serve does not launch training or attach framework callbacks. Stop it after
training to finalize its reports. Training stdout/stderr remain owned by your
terminal, scheduler, or container runtime.
Multi-node direct launches and serve flags
Bind the aggregator on a reachable address and set its endpoint on every training node:
traceml serve --aggregator-bind-host 0.0.0.0 --aggregator-host <node0-ip> \
--aggregator-port 29765 --nnodes <N> --nproc-per-node <M>
TRACEML_AGGREGATOR_HOST=<node0-ip> TRACEML_AGGREGATOR_PORT=29765 \
torchrun ... train.py
Workers default to 127.0.0.1, so non-aggregator nodes need the reachable
node-0 endpoint. Configure the expected world size to match your workers.
| Flag | Meaning |
|---|---|
--aggregator-host |
Worker connection address; default 127.0.0.1. |
--aggregator-bind-host |
Bind address; use 0.0.0.0 for multi-node. |
--aggregator-port |
Telemetry port; default 29765. |
--nnodes / --nproc-per-node |
Expected world size for finalization. |
--mode |
summary (default), cli, or dashboard. |
--logs-dir |
Session log directory. |
--run-name / --session-id |
Shared run identity; choose a fresh name. |
--history-retention |
History duration; default 30m. |
An existing run folder is not overwritten. Telemetry with a different
explicit worker session ID is ignored. Workers that generate their own ID,
such as a plain python train.py, are still admitted.
Missing-aggregator behavior¶
Launcher startup and in-script initialization have different defaults:
| Entry point | Default if the aggregator is unavailable |
|---|---|
traceml run / traceml watch |
Stop before launching training. |
Explicit traceml.init() |
Retry within a bounded timeout, warn, and continue with tracing disabled. |
To let the launcher continue without telemetry:
traceml run train.py --on-missing-aggregator=warn
To require telemetry during explicit setup:
traceml.init(on_missing_aggregator="raise")
The policy resolves from the explicit flag or argument, then
TRACEML_ON_MISSING_AGGREGATOR, then the entry point's default. It is not a
traceml.yaml setting. Calls that require an aggregator, including
traceml.summary(), still fail when telemetry is unavailable.
An occupied telemetry endpoint is also a startup failure. Stop the earlier
aggregator or choose another --aggregator-port.
After training starts, telemetry failures are reported separately and do not replace the training exit code. Only the aggregator-owning launcher reports final telemetry health; in multi-node runs this is node 0. See Training Crashes for the manifest and diagnostics.
Direct-launch configuration¶
Runtime settings resolve from explicit init() arguments, environment
variables, traceml.yaml, then built-in defaults. For display mode, the Python
argument is ui_mode; init(mode=...) selects instrumentation behavior.
Aggregator host and port resolve from explicit arguments, environment variables, then defaults. Run identity and aggregator endpoints are not read from YAML. See the function references above for accepted arguments.
Framework Integrations¶
Framework integrations are separate from the stable core API above. Use the matching integration guide for installation and runtime requirements.
Hugging Face¶
For a standard transformers.Trainer, use traceml run train.py; the launcher
attaches TraceMLTrainerCallback when training starts. Direct launches and
custom training loops can still call the integration init() and register the
callback manually. See the Hugging Face guide
for setup and limitations.
traceml_ai.integrations.huggingface.init
¶
init()
Initialize TraceML for Hugging Face Trainer runs.
For a direct launch, call once before constructing the Trainer, then
register TraceMLTrainerCallback. init() makes TraceML's process-wide
instrumentation explicit: PyTorch DataLoader fetch timing, the H2D
Tensor.to patch, and the forward/backward/optimizer auto-timers that
trace_step arms inside each bracketed step.
The callback is a per-step bracket and cannot install these process-wide
patches on its own; the auto-timers it arms are no-ops unless the matching
patch is installed. For manual setup, init() ensures that the
DataLoader fetch patch in particular is installed deterministically rather
than relying on import order. traceml run calls this path automatically
for a standard Trainer. It also installs the narrow Trainer lifecycle
guard that aborts unfinished steps before an automatic batch-size retry,
plus the collection hook needed to isolate training Input Wait and observe
Accelerate's pre-callback H2D transfers. This mirrors the PyTorch Lightning
integration's init(); HF uses mode="auto" because trace_step
drives forward/backward timing through the patch-gated auto-timers, whereas
Lightning's callback owns that timing directly.
traceml_ai.integrations.huggingface.TraceMLTrainerCallback
¶
TraceMLTrainerCallback()
Bases: TrainerCallback if HAS_TRANSFORMERS else object
Hugging Face Trainer integration for TraceML.
Register with Trainer(..., callbacks=[TraceMLTrainerCallback()]).
The callback brackets TraceML's trace_step context manager: it opens
trace_step in on_step_begin and completes it in on_step_end.
The lifecycle guard installed by :func:init aborts an open context when
the Trainer attempt exits early. trace_step owns the step memory
tracker, step counter, auto-timers, and capture publication.
One completed TraceML step corresponds to one HF accumulation/update
boundary (on_step_end), including when AMP skips the parameter update.
on_substep_end does not advance the counter: forward and backward
events from the actual micro-batches in the group fold into one step,
including a shorter final group. Optimizer events describe calls that
actually run; their count does not drive TraceML's step counter.
TraceML step IDs remain process-local, so their increments match HF's
global_step increments during recorded, completed groups; their
absolute values need not match after checkpoint resume. See the HF
integration docs for the input-timing limitations and lifecycle behavior.
PyTorch Lightning¶
Run a standard Lightning script with traceml run train.py for automatic
initialization and callback attachment. Both Lightning namespaces are supported.
For direct launches, the manual path uses both init() and TraceMLCallback().
Compatible existing setup is reused under automatic launch. See the
Lightning guide for step semantics and limits.
traceml_ai.integrations.lightning.init
¶
init()
Initialize TraceML for PyTorch Lightning runs.
Lightning owns the training loop, so TraceMLCallback owns step boundaries, capture completion, and framework hook integration. The integration init enables DataLoader fetch timing plus the H2D Tensor.to patch. The callback opens the traced step around Lightning's batch transfer (so H2D is inside it), turns H2D timing on only there, and observes outermost module calls during training_step for forward timing. Fetches of non-training loaders are excluded.
traceml_ai.integrations.lightning.TraceMLCallback
¶
TraceMLCallback()
Bases: Callback
Official TraceML Callback for PyTorch Lightning.
One TraceML step is one Lightning optimizer-update boundary. Without
gradient accumulation this is one training batch. With automatic gradient
accumulation, the capture remains open across the group's micro-batches;
their repeated timing events are aggregated by StepTimeSampler and the
memory tracker reports the peak across the group. Fetches of validation,
sanity-check, test and predict loaders are kept out of Input Wait. A group
that raises is discarded, not published.
on_exception
¶
on_exception(trainer, pl_module, exception)
Abandon the current incomplete step without publishing it.
Lightning calls this hook, not teardown, when training raises.
The open timing regions are closed; every micro-batch accumulated
since the prior update, plus the pending memory snapshot, is discarded.
The step counter is left where it was, the wrappers are restored, and
the user's exception propagates untouched.
RF-DETR¶
Run an ordinary RF-DETR model.train() script with traceml run train.py for
automatic attachment. Manual setup needs only rfdetr.init() before training;
the integration supplies its specialized callback. Compatible existing setup
is reused. See the RF-DETR guide for supported modes.
traceml_ai.integrations.rfdetr.init
¶
init()
Enable TraceML for subsequent RF-DETR model.train() calls.
traceml run activates this adapter automatically. For manual setup,
call once before training in each worker. RF-DETR remains optional. The
process-wide factory hook is idempotent and leaves standalone evaluation unchanged;
per-fit model and transfer wrappers are restored by the callback.
Supports eager detection on CPU/CUDA with single-device or torchrun DDP
execution. The native rfdetr fit CLI does not use this factory hook.
Returns the effective TraceML initialization configuration.
MONAI¶
Call init() before building the trainer, then pass TraceMLHandler() in
train_handlers. See the MONAI guide for the seam
each phase is measured from.
traceml_ai.integrations.monai.init
¶
init()
Initialize TraceML for MONAI SupervisedTrainer runs.
Call once before building the trainer, then pass TraceMLHandler() in
train_handlers. H2D timing is armed; the DataLoader fetch patch is
not. The handler times Input Wait from the engine's own fetch events on
the training thread, so a loader that fetches on a background thread,
such as ThreadDataLoader, is timed by what the loop waited.
traceml_ai.integrations.monai.TraceMLHandler
¶
TraceMLHandler()
MONAI handler that times SupervisedTrainer steps for TraceML.
It traces one trainer that runs SupervisedTrainer._iteration itself,
including a subclass that inherits it. Any other engine gets one warning
and nothing attached. Each run first drops step events recorded earlier
in the process. Every exit closes open regions, drops an unfinished step
and restores prepare_batch. Nothing is registered on
Events.EXCEPTION_RAISED, so the user's exception always propagates.
Ray Train¶
Use TraceMLTorchTrainer and its configuration for explicit worker-side
setup. See the Ray Train guide.
traceml_ai.integrations.ray.TraceMLTorchTrainer
¶
TraceMLTorchTrainer(train_loop_per_worker: TrainLoop, *, train_loop_config: Optional[Dict[str, Any]] = None, traceml_config: Optional[TraceMLRayConfig] = None, **torch_trainer_kwargs: Any)
TraceML wrapper for ray.train.torch.TorchTrainer.
The wrapper deliberately uses composition instead of subclassing. Ray's
TorchTrainer still owns training orchestration; TraceML only adds:
- one aggregator actor before
fit() - one runtime wrapper inside each worker
- best-effort aggregator shutdown after
fit()completes or fails
traceml_ai.integrations.ray.TraceMLRayConfig
dataclass
¶
TraceMLRayConfig(mode: str = 'summary', profile: str = 'run', init_mode: str = 'auto', patch_dataloader: Optional[bool] = None, patch_forward: Optional[bool] = None, patch_backward: Optional[bool] = None, patch_h2d: Optional[bool] = None, logs_dir: str = './logs', session_id: str = '', sampler_interval_sec: float = DEFAULT_INTERVAL_SEC, history_retention: DurationValue = '30m', bind_host: str = '0.0.0.0', port: int = 0, stop_timeout_sec: float = 5.0)
TraceML settings used by the Ray Train integration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mode
|
str
|
TraceML display/reporting mode. |
'summary'
|
profile
|
str
|
TraceML sampler profile. Public Ray integration uses the normal
|
'run'
|
init_mode
|
str
|
Instrumentation mode passed to |
'auto'
|
patch_dataloader
|
Optional[bool]
|
Selective-mode-only override for DataLoader fetch patching. |
None
|
patch_forward
|
Optional[bool]
|
Selective-mode-only override for forward-pass patching. |
None
|
patch_backward
|
Optional[bool]
|
Selective-mode-only override for backward-pass patching. |
None
|
patch_h2d
|
Optional[bool]
|
Selective-mode-only override for host-to-device transfer patching. |
None
|
logs_dir
|
str
|
Directory where TraceML writes session logs and summary artifacts. |
'./logs'
|
session_id
|
str
|
Optional explicit TraceML session id. If omitted, a unique Ray session
id is generated for each |
''
|
sampler_interval_sec
|
float
|
Background sampler cadence in seconds. The aggregator actor uses the same value for its live UI refresh cadence; TCP ingestion is immediate. |
DEFAULT_INTERVAL_SEC
|
history_retention
|
DurationValue
|
Raw telemetry analysis history as seconds or a duration such as
|
'30m'
|
bind_host
|
str
|
Host interface used by the aggregator actor. Use |
'0.0.0.0'
|
port
|
int
|
Aggregator TCP port. |
0
|
stop_timeout_sec
|
float
|
Best-effort timeout for aggregator shutdown. |
5.0
|