tag:github.com,2008:https://github.com/Oneflow-Inc/libai/releases 2024-02-07T06:53:39Z tag:github.com,2008:Repository/420927199/v0.3.0 2024-03-11T06:50:01Z <h3>v0.3.0 (03/11/2024)</h3> <p><strong>New Features:</strong></p> <ul> <li>Support mock transformers, see <a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/mock_transformers#readme">Mock transformers</a></li> <li>Support lm-evaluation-harness for model evaluation</li> <li>User Experience Optimization</li> </ul> <p><strong>New Supported Models:</strong></p> <ul> <li>These models are natively supported by libai</li> </ul> <table> <tbody> <tr> <th width="130"> Models </th> <th valign="bottom" align="center" width="140"> 2D(tp+pp) Inference</th> <th valign="bottom" align="center" width="140"> 3D Parallel Training </th> </tr> <tr> <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/BLOOM"> <b> BLOOM </b> </a></td> <td align="center">✔</td> <td align="center">-</td> </tr> <tr> <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/ChatGLM"> <b> ChatGLM </b> </a></td> <td align="center">✔</td> <td align="center">✔</td> </tr> <tr> <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/Couplets"> <b> Couplets </b> </a></td> <td align="center">✔</td> <td align="center">✔</td> </tr> <tr> <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/DALLE2"> <b> DALLE2 </b> </a></td> <td align="center">✔</td> <td align="center">-</td> </tr> <tr> <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/Llama"> <b> Llama2 </b> </a></td> <td align="center">✔</td> <td align="center">✔</td> </tr> <tr> <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/MAE"> <b> MAE </b> </a></td> <td align="center">✔</td> <td align="center">✔</td> </tr> <tr> <td align="center"><a href="https://github.com/Oneflow-Inc/libai/tree/main/projects/Stable_Diffusion"> <b> Stable_Diffusion </b> </a></td> <td align="center">-</td> <td align="center">-</td> </tr> </tbody> </table> <p><strong>New Mock Models:</strong></p> <ul> <li>These models are extended and implemented by libai through mocking transformers.</li> </ul> <table> <tbody> <tr> <th width="130"> Models </th> <th valign="bottom" align="center" width="140">Tensor Parallel</th> <th valign="bottom" align="center" width="150">Pipeline Parallel</th> </tr> <tr> <td align="center"><a href="https://huggingface.co/docs/transformers/v4.26.1/en/model_doc/bloom#overview" rel="nofollow"> <b> BLOOM </b> </a></td> <td align="center">✔</td> <td align="center">-</td> </tr> <tr> <td align="center"><a href="https://github.com/openai/gpt-2/blob/master/model_card.md"> <b> GPT2 </b> </a></td> <td align="center">✔</td> <td align="center">-</td> </tr> <tr> <td align="center"><a href="https://huggingface.co/docs/transformers/v4.28.0/en/model_doc/llama#overview" rel="nofollow"> <b> LLAMA </b> </a></td> <td align="center">✔</td> <td align="center">-</td> </tr> <tr> <td align="center"><a href="https://huggingface.co/docs/transformers/main/en/model_doc/llama2" rel="nofollow"> <b> LLAMA2 </b> </a></td> <td align="center">✔</td> <td align="center">-</td> </tr> <tr> <td align="center"><a href="https://huggingface.co/baichuan-inc/Baichuan-7B" rel="nofollow"> <b> Baichuan </b> </a></td> <td align="center">✔</td> <td align="center">-</td> </tr> <tr> <td align="center"><a href="https://huggingface.co/docs/transformers/v4.26.1/en/model_doc/opt#overview" rel="nofollow"> <b> OPT </b> </a></td> <td align="center">✔</td> <td align="center">-</td> </tr> </tbody> </table> xiezipeng-ML tag:github.com,2008:Repository/420927199/v0.2.0 2022-07-08T03:06:22Z <h3>v0.2.0 (07/07/2022)</h3> <p><strong>New Features:</strong></p> <ul> <li>Support evaluation enabled and set <code>eval_iter</code></li> <li>Support customized sampler in <code>config.py</code></li> <li>Support rdma for pipeline-model-parallel</li> <li>Support multi fused kernel <ul> <li>fused_scale_mask_softmax_dropout</li> <li>fused_scale_tril_softmax_mask_scale</li> <li>fused_self_attention in branch <code>libai_bench</code></li> </ul> </li> <li>User Experience Optimization</li> <li>Optimization for training throughput, see <a href="https://libai.readthedocs.io/en/latest/tutorials/get_started/Benchmark.html" rel="nofollow">benchmark</a> for more details</li> </ul> <p><strong>New Supported Models:</strong></p> <ul> <li>Support 3D parallel <a href="https://arxiv.org/abs/1907.11692" rel="nofollow">Roberta</a> model</li> <li>Support 2D parallel (data parallel + tensor model parallel) <a href="https://arxiv.org/abs/2104.08821" rel="nofollow">SimCSE</a> model</li> <li>Support Data parallel <a href="https://arxiv.org/abs/2111.06377" rel="nofollow">MAE</a> model</li> <li>Support Data parallel <a href="https://arxiv.org/abs/2104.02057" rel="nofollow">MOCOV3</a> model</li> </ul> CPFLAME tag:github.com,2008:Repository/420927199/v0.1.1 2022-03-22T07:33:48Z <p>stable release v0.1.1</p> L1aoXingyu tag:github.com,2008:Repository/420927199/v0.1.0 2022-03-23T05:29:04Z <p>Some major features:</p> <ul> <li>Data Parallelism</li> <li>1D Tensor Parallelism</li> <li>Pipeline Parallelism</li> <li>Unified Distributed Layers for single-GPU and multi-GPU</li> <li>"LazyConfig" system for more flexible syntax and no predefined structures</li> <li>Easy-to-use trainer and engine</li> <li>CV &amp; NLP Data Preprocessing</li> <li>Mixed Precision Training</li> <li>Activation Checkpointing</li> <li>Gradient Accumulation</li> <li>Gradient Clipping</li> <li>Zero Redundancy Optimizer (ZeRO)</li> </ul> <h3>Model_Zoo</h3> <ul> <li>Bert (3D Parallelism)</li> <li>GPT-2 (3D Parallelism)</li> <li>ViT (3D Parallelism)</li> <li>Swin-Transformer (data parallelism)</li> <li>Add finetune task in <code>projects/</code></li> <li>Add text classification in <code>projects</code></li> </ul> L1aoXingyu tag:github.com,2008:Repository/420927199/v0.0.1.4 2022-03-22T07:08:38Z <p>test release v0.0.1.4</p> L1aoXingyu tag:github.com,2008:Repository/420927199/v0.0.1.3 2022-01-25T08:30:31Z <p>test stable release v0.0.1.3</p> Ldpe2G