Skip to content

Latest commit

 

History

History
109 lines (77 loc) · 2.59 KB

README.md

File metadata and controls

109 lines (77 loc) · 2.59 KB
Raw

PyTorch T5 inference

Description

This document has instructions for running T5 inference using Intel-optimized PyTorch.

Bare Metal

General setup

Follow link to install Conda and build Pytorch, IPEX, and Jemalloc.

Model Specific Setup

  • Install dependency
pip install transformers
pip install -r requirements.txt
  • Setup the output dir to store the log
    export OUTPUT_DIR=$Where_to_save_log
  • Setup runnning precision
    export PRECISION=$Data_type(fp32, int8)
  • Setup model name running
    export MODEL_NAME=$Model_name(t5-small, t5-base, t5-large, t5-3b and t5-11b)
  • Setup predict samples to do inference
    export MAX_PREDICT_SAMPLES=$Max_predict_samples
  • Setup cores number to use
    export CORES_PER_INSTANCE=$Cores_use
  • Set Jemalloc Preload for better performance

The jemalloc should be built from the General setup section.

    export LD_PRELOAD="path/lib/libjemalloc.so":$LD_PRELOAD
    export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
  • Set IOMP preload for better performance

IOMP should be installed in your conda env from the General setup section.

    export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
  • Set ENV to use AMX if you are using SPR
    export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX

Quick Start Scripts

backend performance
IPEX bash run_inference.sh ipex
Offical Pytorch bash run_inference.sh pytorch

Run the model

Follow the instructions above to setup your bare metal environment, download and preprocess the dataset, and do the model specific setup. Once all the setup is done, the Model Zoo can be used to run a quickstart script. Ensure that you have an enviornment variable set to point to an output directory.

# Clone the model zoo repo and set the MODEL_DIR
git clone https://github.com/IntelAI/models.git
cd models
export MODEL_DIR=$(pwd)

# Env vars
export OUTPUT_DIR=<path to an output directory>
export PRECISION=int8
export MODEL_NAME=t5-small
export MAX_PREDICT_SAMPLES=30
export CORES_PER_INSTANCE=4
# Run a quickstart script (for example, ipex inference)
bash run_inference.sh ipex

License

LICENSE