Use TraceML With Your Stack¶
Run your existing Hugging Face Trainer, PyTorch Lightning, or RF-DETR script
with traceml run train.py. TraceML instruments standard training
automatically. For custom PyTorch loops, MONAI, Ray, Accelerate, and DeepSpeed,
follow the setup in the matching guide below.
Choose Your Integration¶
Choose your framework to open its setup guide. Zero-code launch uses traceml run train.py for the standard training paths.
| Framework | Setup | Example |
|---|---|---|
| Hugging Face Trainer | Zero-code launch | Example |
| PyTorch Lightning | Zero-code launch | Example |
| RF-DETR | Zero-code launch | Example |
| Custom PyTorch loop | Explicit step boundary | Example |
| Hugging Face Accelerate | Explicit step boundary | Example |
| MONAI | Handler setup | Example |
| Ray Train | Trainer adapter | Example |
| DeepSpeed | Explicit step boundary | Example |
| W&B / MLflow | Summary export | Example |
Integration support matrix¶
Tested coverage, versions, and limitations
CI tested identifies configurations checked by the linked job and tests. Documented recipe and experimental identify coverage without equivalent end-to-end validation. See each framework guide for the requirements that apply to your run.
| Stack | Recommended API | Tested dependency range | CPU / GPU | Single- / multi-process | Multi-node | Emitted signals | Example | Validation level | Limitations |
|---|---|---|---|---|---|---|---|---|---|
| Hugging Face Trainer | traceml run train.py (standard Trainer); init() + TraceMLTrainerCallback for manual setup |
CI resolves torch>=2.5.0, transformers>=4.46.1, and accelerate>=0.26.0; not a pinned compatibility matrix. | CPU: CI tested — real Trainer run GPU: Not claimed — no GPU job |
Single: CI tested — CPU only Multi: CI tested — two CPU/Gloo ranks |
Experimental — no reproducible job | CPU launcher tests assert step, forward, backward, optimizer, and DataLoader fetch events. The automatic two-rank test verifies both ranks and aligned optimizer steps. GPU H2D and memory signals are not asserted in CI. | huggingface_trainer_minimal.py | CI tested CI job: test-integration; automatic single-process and two-rank real Trainer launcher tests |
Hugging Face limitations · #141 · #320 · #323 |
| PyTorch Lightning | traceml run train.py (standard Trainer.fit); init() + TraceMLCallback for manual setup |
CI resolves lightning>=2.6.0 (not pinned; this is not a broad version matrix). The cited tests run under both the lightning and the legacy pytorch_lightning namespace. | CPU: CI tested — Ubuntu integration job GPU: Documented recipe — no GPU job |
Single: CI tested — real Trainer.fit() CPU run Multi: CI tested — two CPU/Gloo ranks |
Experimental — no reproducible job | CPU tests assert step, forward, backward, optimizer, and DataLoader fetch events. The automatic two-rank test verifies both ranks and aligned optimizer steps. H2D is GPU-only and not asserted in CI. | lightning_minimal.py | CI tested CI job: test-integration; real Trainer.fit() automatic single-process and two-rank launcher tests |
Lightning guide · #141 · #320 · #323 |
| RF-DETR | traceml run train.py (standard model.train); rfdetr.init() for manual setup |
Dedicated CPU CI pins rfdetr[train]==1.10.1. A reproducible one- and four-T4 case study uses RF-DETR development commit 0ed5be8 with torch 2.9.1; this is not a broad compatibility range. | CPU: CI tested — dedicated RF-DETR CPU job GPU: Documented recipe — one- and four-T4 case study; no CUDA CI |
Single: CI tested — parity and real Nano smoke Multi: CI tested — two CPU/Gloo ranks in CI; four-T4 NCCL case study |
Experimental — no physical multi-node validation | CPU tests assert training step, inner-model forward, backward, optimizer and DataLoader fetch, excluding evaluation. The T4 case study records H2D and CUDA phase timing on one and four GPUs; CUDA memory remains unvalidated. | rfdetr_minimal.py | CI tested CI job: test-rfdetr; RF-DETR parity, Nano and Gloo smoke tests |
RF-DETR limitations · #141 · #323 |
| Custom PyTorch loop | traceml.init() + traceml.trace_step(model) |
CI resolves torch>=2.5.0 (not pinned; this is not a broad version matrix). | CPU: CI tested — Ubuntu integration job GPU: Not claimed — no GPU job |
Single: CI tested — launcher smoke run Multi: Not claimed — no matrix evidence |
Not claimed — no matrix evidence | The cited smoke test asserts a final summary; it does not declare a per-signal contract. | quickstart.py | CI tested CI job: test-integration; real PyTorch launcher smoke test |
Distributed Training · #141 · #320 · #323 |
| Hugging Face Accelerate | traceml.init() + traceml.trace_step(accelerator.unwrap_model(model)) |
CI resolves torch>=2.5.0 and accelerate>=0.26.0; not a pinned compatibility matrix. | CPU: CI tested — real Accelerator.prepare/backward run GPU: Not claimed — no GPU job |
Single: CI tested — CPU only Multi: Documented recipe — no real distributed job |
Experimental — no reproducible job | step, forward, backward, optimizer, and dataloader fetch on the cited real Accelerate run. | accelerate_minimal.py | CI tested CI job: test-integration; real Accelerate prepare/backward test |
Accelerate limitations · #141 · #320 · #323 |
| MONAI | traceml_ai.integrations.monai.init() + TraceMLHandler in train_handlers |
A dedicated CPU CI job pins monai==1.6. with pytorch-ignite==0.5., and reruns the same tests at the declared floor pytorch-ignite==0.4.11. The one-T4 spleen notebook ran MONAI 1.6.0, Ignite 0.5.5 and torch 2.11.0+cu130. This is not a broad compatibility matrix. | CPU: CI tested — dedicated MONAI CPU job GPU: Documented recipe — one-T4 spleen notebook; no CUDA CI |
Single: CI tested — real SupervisedTrainer runs, plus the documented example end to end Multi: Not claimed — no distributed job |
Not claimed — no distributed job | tests/integrations/test_telemetry_conformance.py asserts step time, forward, backward, optimizer and the engine's own fetch bracket on a CPU run. The same job runs examples/integrations/monai_minimal.py under traceml run, with and without gradient accumulation, and fails if the summary loses Input Wait, forward, backward or optimizer time, or if its step count disagrees with the optimizer updates the trainer made. H2D and step memory are GPU-only and not asserted in CI. The one-T4 spleen notebook's table shows H2D for every run, and each run's summary carries step memory; neither value is validated. | monai_minimal.py | CI tested CI job: test-monai; real MONAI SupervisedTrainer tests, rerun at the Ignite floor |
MONAI limitations · #141 · #320 · #323 · #463 |
| Ray Train | TraceMLTorchTrainer + TraceMLRayConfig |
Documented install requirement: ray[train]>=2.0.0. No real-Ray CI-tested range. | CPU: Documented recipe — no real-Ray job GPU: Documented recipe — no GPU job |
Single: Documented recipe — local Ray recipe Multi: Documented recipe — worker recipe only |
Experimental — no reproducible job | Guide documents step, forward, backward, optimizer, and wrapped Ray Data fetch timing; no real-Ray CI signal assertion exists. | torchtrainer_minimal.py | documented recipe | Ray guide · #141 · #320 · #323 |
| DeepSpeed | traceml.init() + traceml.trace_step(model_engine.module) |
Documented install requirement: deepspeed>=0.10.0. No real-DeepSpeed CI-tested range. | CPU: Unsupported — the guide requires CUDA GPU: Documented recipe — no CUDA CI job |
Single: Documented recipe — no real engine run Multi: Documented recipe — no real distributed job |
Experimental — no reproducible job | Guide documents DataLoader fetch, H2D, forward, backward, optimizer, step, and memory; no real-DeepSpeed CI signal assertion exists. | deepspeed_minimal.py | documented recipe | DeepSpeed limitations · #141 · #320 · #323 |
| W&B / MLflow | traceml.summary() |
No W&B or MLflow dependency is installed or run in CI. | CPU: Documented recipe — tracker client not run GPU: Not applicable — tracker export is device-independent |
Single: Documented recipe — no tracker-client CI run Multi: Not claimed — no matrix evidence |
Not claimed — no matrix evidence | No tracker telemetry is emitted by TraceML; the documented integration exports TraceML's compact summary dictionary. | summary_logging_minimal.py | documented recipe | W&B / MLflow guide · #323 |
Evidence limits¶
- This matrix makes no GPU, multi-process, or multi-node claim without a corresponding reproducible job.
- Tested configurations and recorded examples do not establish compatibility with every version, device, or training strategy.