Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add contact group #21

Merged
merged 3 commits into from
Sep 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions .github/workflows/unit_test.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,23 @@
name: Unit Tests

on:
on:
workflow_dispatch:
pull_request:
branches:
- main
tags:
- r[0-9]+.[0-9]+
paths-ignore:
- 'docs/**'
- '**.md'
push:
branches:
- main
tags:
- r[0-9]+.[0-9]+
paths-ignore:
- 'docs/**'
- '**.md'


jobs:
Expand All @@ -26,5 +36,5 @@ jobs:
git config --global --add safe.directory $PWD && \
pip install -e . && \
bash tests/run_ut.sh'
env:
env:
UT_IMAGE: ${{ secrets.UT_IMAGE }}
25 changes: 17 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/alibabapai/torchacc/blob/main/LICENSE)

# TorchAcc

**TorchAcc** is an AI training acceleration framework developed by Alibaba Cloud’s PAI.
**TorchAcc** is an AI training acceleration framework developed by Alibaba Cloud’s PAI team.

TorchAcc is built on [PyTorch/XLA](https://github.com/pytorch/xla) and provides an easy-to-use interface to accelerate the training of PyTorch models. At the same time, TorchAcc has implemented extensive optimizations for distributed training, memory management, and computation specifically for GPUs, ultimately achieving improved ease of use, better GPU training performance, and enhanced scalability for distributed training.

[**Documentation**](https://torchacc.readthedocs.io/en/latest/)

## Highlighted Features

* Rich distributed Parallelism
* Rich distributed parallelism strategies
* Data Parallelism
* Fully Sharded Data Parallelism
* Tensor Parallelism
Expand All @@ -20,7 +20,7 @@ TorchAcc is built on [PyTorch/XLA](https://github.com/pytorch/xla) and provides
* [Ulysess](https://arxiv.org/abs/2309.14509)
* [Ring Attention](https://arxiv.org/abs/2310.01889)
* FlashSequence (2D Sequence Parallelism)
* Low Memory Cost
* Memory efficient
* High Performance
* Easy-to-use API

Expand Down Expand Up @@ -53,32 +53,41 @@ sudo docker run --gpus all --net host --ipc host --shm-size 10G -it --rm --cap-
see the [contribution guide](docs/source/contributing.md).


## LLMs training examples

### Getting Started
## Getting Started

We present a straightforward example for training a Transformer model using TorchAcc, illustrating the usage of the TorchAcc API.
You can quickly initiate training a Transformer model with TorchAcc by executing the following command:
``` shell
torchrun --nproc_per_node=4 benchmarks/transformer.py --bf16 --acc --disable_loss_print --fsdp_size=4 --gc
```

## LLMs training examples

### Utilizing HuggingFace Transformers

If you are familiar with HuggingFace Transformers's Trainer, you can easily accelerate a Transformer model using TorchAcc, see the [huggingface transformers](docs/source/tutorials/hf_transformers.md)

### LLMs training acceleration with FlashModels

If you want to try the latest features of Torchacc or want to use the TorchAcc interface more flexibly for model acceleration, you can use our LLM acceleration library, FlashModels. It integrates various distributed implementations of commonly used open-source LLM models and provides a wealth of examples.
If you want to try the latest features of Torchacc or want to use the TorchAcc interface more flexibly for model acceleration, you can use our LLM acceleration library, FlashModels. FlashModels integrates various distributed implementations of commonly used open-source LLMs and provides a wealth of examples and benchmarks.

https://github.com/AlibabaPAI/FlashModels

### SFT using modelscope/swift
coming soon..


## Contributing
see the [contribution guide](docs/source/contributing.md).


## Contact Us

You can contact us by adding our DingTalk group:

<p align="center">
<img width="30%" src=docs/figures/group.png />
</p>

## License
[Apache License 2.0](LICENSE)
Binary file added docs/figures/group.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
import datetime

project = 'PAI-TorchAcc'
project = 'TorchAcc'
copyright = '2024, alibaba-cloud'
author = 'Alibaba Cloud'
release = '2.3.0'
Expand Down
4 changes: 2 additions & 2 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
.. PAI-TorchAcc documentation master file, created by
.. TorchAcc documentation master file, created by
sphinx-quickstart on Fri Nov 3 16:11:35 2023.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.

Welcome to PAI-TorchAcc's documentation!
Welcome to TorchAcc's documentation!
========================================

.. toctree::
Expand Down
Loading