Skip to content

Latest commit

 

History

History
56 lines (34 loc) · 2.36 KB

readme.md

File metadata and controls

56 lines (34 loc) · 2.36 KB
Raw

Transformers from Scratch for English-to-Arabic Translation

This repository contains an implementation of a Transformer model built from scratch using PyTorch. The model is trained for English-to-Arabic translation using the news commentary dataset.

Key Features

  • Custom Transformer Architecture: Implemented from scratch in PyTorch, inspired by Mr. Umar Jamil's code and his YouTube video.
  • Monitoring and Logging: Utilized PyTorch Ignite for efficient monitoring, logging, and checkpointing during training.
  • Dataset Handling: Leveraged Hugging Face's datasets library for easy access to the news_commentary dataset and tokenization.

Repository packages structure

  • model: Contains the architecture of the Transformer model. This directory includes the implementation of the model components.
  • config: Holds configuration files and hyperparameters for training. Adjust these files to modify the training settings and parameters.
  • train: Contains scripts and modules related to training the model. This is where the training process is defined and executed.
  • utils: Includes utility functions and classes, particularly those related to PyTorch Ignite handlers for monitoring, logging, and checkpointing.

Getting Started

To get started with this repository, follow these steps:

  1. Clone the Repository:

    git clone https://github.com/Mo-Ouail-Ocf/transformers-from-scratch.git
cd transformers-from-scratch

  2. Set Up Your Environment:

    Create a Conda environment from the provided env.yml file:

    conda env create -f env.yml

    Activate the Conda environment:

    conda activate transformers-env

    Note: This code is compatible with Python 3.11.

Future Tasks

  • Results and Visualization: Implement visualization tools for attention scores to better understand model performance and behavior.

Acknowledgements

  • Special thanks to Mr. Umar Jamil for his exceptional resources and tutorials, which provided valuable insights into implementing and understanding Transformer models.

  • For more information and in-depth tutorials, visit Mr. Umar Jamil's YouTube channel.