Update transformers backend name (#2075)

3outeille · web-flow · commit ad9f188abe81 · 2025-11-22T21:42:29.000-08:00
following Huggingface efforts in VLLM (cf vllm-project/vllm#28725), we would like to uniformize the naming and make sure that people think we use the HF models only
diff --git a/.ci/docker/common/install_conda.sh b/.ci/docker/common/install_conda.sh
@@ -43,7 +43,7 @@ install_pip_dependencies() {
   pip_install -r /opt/conda/requirements.txt
   pip_install -r /opt/conda/requirements-flux.txt
   pip_install -r /opt/conda/requirements-vlm.txt
-  pip_install -r /opt/conda/requirements-transformers-backend.txt
+  pip_install -r /opt/conda/requirements-transformers-modeling-backend.txt
   popd
 }
 
diff --git a/.ci/docker/requirements-transformers-modeling-backend.txt b/.ci/docker/requirements-transformers-modeling-backend.txt
diff --git a/.ci/docker/ubuntu/Dockerfile b/.ci/docker/ubuntu/Dockerfile
@@ -33,7 +33,7 @@ COPY requirements-dev.txt /opt/conda/
 COPY requirements.txt /opt/conda/
 COPY requirements-flux.txt /opt/conda/
 COPY requirements-vlm.txt /opt/conda/
-COPY requirements-transformers-backend.txt /opt/conda/
+COPY requirements-transformers-modeling-backend.txt /opt/conda/
 COPY conda-env-ci.txt /opt/conda/
 COPY ./common/install_conda.sh install_conda.sh
 COPY ./common/utils.sh utils.sh
diff --git a/.github/workflows/integration_test_8gpu_transformers_modeling_backend.yaml b/.github/workflows/integration_test_8gpu_transformers_modeling_backend.yaml
@@ -1,13 +1,13 @@
-name: Transformers Backend 8 GPU Integration Tests
+name: Transformers Modeling Backend 8 GPU Integration Tests
 
 on:
   push:
     branches: [ main ]
     paths:
-      - 'torchtitan/experiments/transformers_backend/**'
+      - 'torchtitan/experiments/transformers_modeling_backend/**'
   pull_request:
     paths:
-      - 'torchtitan/experiments/transformers_backend/**'
+      - 'torchtitan/experiments/transformers_modeling_backend/**'
   schedule:
     # Runs every 12 hours
     - cron: '0 */12 * * *'
@@ -50,4 +50,4 @@ jobs:
         USE_CPP=0 python -m pip install --pre torchao --index-url https://download.pytorch.org/whl/nightly/cu126
 
         mkdir artifacts-to-be-uploaded
-        python -m torchtitan.experiments.transformers_backend.tests.integration_tests artifacts-to-be-uploaded --ngpu 8
+        python -m torchtitan.experiments.transformers_modeling_backend.tests.integration_tests artifacts-to-be-uploaded --ngpu 8
diff --git a/torchtitan/experiments/README.md b/torchtitan/experiments/README.md
@@ -31,4 +31,4 @@ We provide this `experiments/` folder to host experiments that add significant v
 | [moe_symm_mem_kernels](./moe_symm_mem_kernels/) | TBA | [@kwen2501](https://github.com/kwen2501) |
 | [gpt_oss](./gpt_oss/) | TBA | [@jianiw](https://github.com/jianiw) |
 | [compiler_toolkit](./compiler_toolkit/) | [![Compiler Toolkit 8 GPU Integration Tests](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu_compiler_toolkit.yaml/badge.svg?branch=main)](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu_compiler_toolkit.yaml?query=branch%3Amain) | [@SherlockNoMad](https://github.com/SherlockNoMad) [@yiming0416](https://github.com/yiming0416) |
-| [transformers_backend](./transformers_backend/) | [![Transformers backend 8 GPU Integration Tests](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu_transformers_backend.yaml/badge.svg?branch=main)](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu_transformers_backend.yaml?query=branch%3Amain) | [@3outeille](https://github.com/3outeille) |
+| [transformers_modeling_backend](./transformers_modeling_backend/) | [![Transformers modeling backend 8 GPU Integration Tests](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu_transformers_modeling_backend.yaml/badge.svg?branch=main)](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu_transformers_modeling_backend.yaml?query=branch%3Amain) | [@3outeille](https://github.com/3outeille) |
diff --git a/torchtitan/experiments/__init__.py b/torchtitan/experiments/__init__.py
@@ -12,6 +12,6 @@
         "vlm",
         "compiler_toolkit.deepseek_v3",
         "compiler_toolkit.llama3",
-        "transformers_backend",
+        "transformers_modeling_backend",
     ]
 )
diff --git a/torchtitan/experiments/transformers_modeling_backend/README.md b/torchtitan/experiments/transformers_modeling_backend/README.md
@@ -1,23 +1,25 @@
-# Huggingface Transformers backend
+# Huggingface Transformers Modeling backend
+
+This enables HF transformers models to be trained with `4D parallelism + torch.compile`
 
 ## Quick start
 
 - Requirements `transformers==4.57.1`
 
-- Config: `torchtitan/torchtitan/experiments/transformers_backend/configs/qwen3.toml`
+- Config: `torchtitan/torchtitan/experiments/transformers_modeling_backend/configs/qwen3.toml`
 ```diff
 ...
 [model]
 - name = "llama3"
-+ name = "transformers_backend"
++ name = "transformers_modeling_backend"
 flavor = "debugmodel"
 hf_assets_path = "./tests/assets/tokenizer"
 
 +[hf_transformers]
 +model = "Qwen/Qwen3-4B-Instruct-2507"
 ...
 ```
-- Train: `LOG_RANK=7 CONFIG_FILE=<YOUR_PATH>/torchtitan/experiments/transformers_backend/configs/qwen3.toml ./run_train.sh --job.custom_config_module=torchtitan.experiments.transformers_backend.job_config --compile.enable`
+- Train: `LOG_RANK=7 CONFIG_FILE=<YOUR_PATH>/torchtitan/experiments/transformers_modeling_backend/configs/qwen3.toml ./run_train.sh --job.custom_config_module=torchtitan.experiments.transformers_modeling_backend.job_config --compile.enable`
     - Make sure you have created the tokenizers beforehand
 <img width="1334" height="453" alt="image" src="https://github.com/user-attachments/assets/da459448-027b-4af9-8176-6a3e433a272c" />
 
diff --git a/torchtitan/experiments/transformers_modeling_backend/__init__.py b/torchtitan/experiments/transformers_modeling_backend/__init__.py
diff --git a/torchtitan/experiments/transformers_modeling_backend/configs/debug_model.toml b/torchtitan/experiments/transformers_modeling_backend/configs/debug_model.toml
@@ -20,7 +20,7 @@ save_tb_folder = "tb"
 enable_wandb = false
 
 [model]
-name = "transformers_backend"
+name = "transformers_modeling_backend"
 flavor = "debugmodel"
 # test folder with tokenizer.json, for debug purpose only
 hf_assets_path = "./tests/assets/tokenizer"
diff --git a/torchtitan/experiments/transformers_modeling_backend/configs/full.toml b/torchtitan/experiments/transformers_modeling_backend/configs/full.toml
@@ -20,7 +20,7 @@ save_tb_folder = "tb"
 enable_wandb = false
 
 [model]
-name = "transformers_backend"
+name = "transformers_modeling_backend"
 flavor = "full"
 # test folder with tokenizer.json, for debug purpose only
 hf_assets_path = "./tests/assets/tokenizer"
diff --git a/torchtitan/experiments/transformers_modeling_backend/infra/parallelize.py b/torchtitan/experiments/transformers_modeling_backend/infra/parallelize.py
@@ -22,7 +22,7 @@
 from torchtitan.distributed.activation_checkpoint import apply_ac
 
 from torchtitan.distributed.tensor_parallel import maybe_enable_async_tp
-from torchtitan.experiments.transformers_backend.job_config import JobConfig
+from torchtitan.experiments.transformers_modeling_backend.job_config import JobConfig
 from torchtitan.models.llama3.infra.parallelize import apply_compile, apply_ddp
 from torchtitan.tools.logging import logger
 
diff --git a/torchtitan/experiments/transformers_modeling_backend/infra/pipeline.py b/torchtitan/experiments/transformers_modeling_backend/infra/pipeline.py
@@ -21,7 +21,7 @@
 from torchtitan.components.loss import LossFunction
 from torchtitan.distributed import ParallelDims
 from torchtitan.distributed.pipeline_parallel import build_pipeline_schedule
-from torchtitan.experiments.transformers_backend.job_config import JobConfig
+from torchtitan.experiments.transformers_modeling_backend.job_config import JobConfig
 from torchtitan.protocols.train_spec import BaseModelArgs, ParallelizeFunction
 from torchtitan.tools.logging import logger
 
diff --git a/torchtitan/experiments/transformers_modeling_backend/job_config.py b/torchtitan/experiments/transformers_modeling_backend/job_config.py
diff --git a/torchtitan/experiments/transformers_modeling_backend/model/args.py b/torchtitan/experiments/transformers_modeling_backend/model/args.py
diff --git a/torchtitan/experiments/transformers_modeling_backend/model/model.py b/torchtitan/experiments/transformers_modeling_backend/model/model.py
diff --git a/torchtitan/experiments/transformers_modeling_backend/tests/integration_tests.py b/torchtitan/experiments/transformers_modeling_backend/tests/integration_tests.py
@@ -11,7 +11,7 @@
 from tests.integration_tests.run_tests import run_tests
 
 
-def build_transformers_backend_test_list() -> list[OverrideDefinitions]:
+def build_transformers_modeling_backend_test_list() -> list[OverrideDefinitions]:
     """
     key is the config file name and value is a list of OverrideDefinitions
     that is used to generate variations of integration tests based on the
@@ -21,8 +21,8 @@ def build_transformers_backend_test_list() -> list[OverrideDefinitions]:
         OverrideDefinitions(
             [
                 [
-                    "--model.name transformers_backend",
-                    "--job.custom_config_module=torchtitan.experiments.transformers_backend.job_config",
+                    "--model.name transformers_modeling_backend",
+                    "--job.custom_config_module=torchtitan.experiments.transformers_modeling_backend.job_config",
                     "--hf_transformers.model Qwen/Qwen2.5-7B",
                     "--parallelism.data_parallel_shard_degree 2",
                     "--parallelism.tensor_parallel_degree 2",
@@ -31,15 +31,15 @@ def build_transformers_backend_test_list() -> list[OverrideDefinitions]:
                 ],
             ],
             "Transformers Backend FSDP+TP+PP",
-            "transformers_backend_fsdp+tp+pp",
+            "transformers_modeling_backend_fsdp+tp+pp",
             ngpu=8,
         ),
     ]
     return integration_tests_flavors
 
 
 _TEST_SUITES_FUNCTION = {
-    "transformers_backend": build_transformers_backend_test_list,
+    "transformers_modeling_backend": build_transformers_modeling_backend_test_list,
 }
 
 
@@ -64,7 +64,7 @@ def main():
     if os.listdir(args.output_dir):
         raise RuntimeError("Please provide an empty output directory.")
 
-    test_list = _TEST_SUITES_FUNCTION["transformers_backend"]()
+    test_list = _TEST_SUITES_FUNCTION["transformers_modeling_backend"]()
     run_tests(args, test_list)
 
 

Original file line number	Diff line number	Diff line change
`@@ -43,7 +43,7 @@ install_pip_dependencies() {`
`43`	`43`	`pip_install -r /opt/conda/requirements.txt`
`44`	`44`	`pip_install -r /opt/conda/requirements-flux.txt`
`45`	`45`	`pip_install -r /opt/conda/requirements-vlm.txt`
`46`		`- pip_install -r /opt/conda/requirements-transformers-backend.txt`
	`46`	`+ pip_install -r /opt/conda/requirements-transformers-modeling-backend.txt`
`47`	`47`	`popd`
`48`	`48`	`}`
`49`	`49`
Original file line number	Diff line number	Diff line change
`@@ -12,6 +12,6 @@`
`12`	`12`	`"vlm",`
`13`	`13`	`"compiler_toolkit.deepseek_v3",`
`14`	`14`	`"compiler_toolkit.llama3",`
`15`		`- "transformers_backend",`
	`15`	`+ "transformers_modeling_backend",`
`16`	`16`	`]`
`17`	`17`	`)`