Table of Contents

• 1. Introduction
• 2. Model Architecture
• 3. Key Modifications
• 4. Model Parameters
• 5. System Architecture
• 6. Cross-Validation Strategy
• 7. Installation
• 8. Usage

1. Introduction

SAINT+ is a Transformer-based knowledge tracing model designed to predict student performance by analyzing exercise history. Built on the classical Transformer encoder-decoder architecture, SAINT+ processes sequences of student interactions to identify learning patterns and forecast future outcomes.

The model employs a dual-stream approach:

• Encoder: Processes exercise embeddings using self-attention to identify relevant patterns in the exercise sequence
• Decoder: Analyzes response embeddings and applies encoder-decoder attention to correlate past performance with future questions

This architecture enables the model to capture temporal dependencies and learning trajectories, making it effective for knowledge tracing tasks in educational technology applications.

Folder PATH listing
+---assets                         <-- Assets directory
│       Saint_plus-banner.png      <-- Project banner image
│
+---scripts                        <-- Training scripts
│       train.py                   <-- Main training script
│
+---src                            <-- Source code package
│       __init__.py                <-- Package initialization
│       data_generator.py          <-- Data loading and sequences
│       model.py                   <-- SAINT+ model implementation
│       parser.py                  <-- Hyperparameter parser
│       pre_process.py             <-- Data preprocessing
│       utils.py                   <-- Utility functions
│
+---weights                        <-- Model weights directory
│       README.md                  <-- Weights documentation
│       saint_2_128.pt             <-- Pre-trained model weights
│
        .gitignore                 <-- Git exclusions
        LICENSE                    <-- License information
        README.md                  <-- Project documentation
        requirements.txt           <-- Python dependencies
        setup.py                   <-- Package setup script

2. Model Architecture

2.1 Overview

SAINT+ follows a Transformer encoder-decoder structure with specialized embeddings for educational data. The encoder processes question-related features, while the decoder handles response-related features, enabling the model to learn the relationship between exercises and student performance.

graph TB
    subgraph Input["Input Features"]
        CID[Content ID]
        PART[Part ID]
        EXPL[Explanation Flag]
        TLAG[Time Lag]
        ETIME[Elapsed Time]
        ACORR[Answer Correctness]
        UANS[User Answer]
    end
    
    subgraph EncoderEmbed["Encoder Embeddings"]
        CID --> CID_EMB[Content Embedding]
        PART --> PART_EMB[Part Embedding]
        EXPL --> EXPL_EMB[Explanation Embedding]
        TLAG --> TLAG_EMB["Time Lag Embedding<br/>(log scaled)"]
        CID_EMB --> CONCAT1[Concatenate]
        PART_EMB --> CONCAT1
        EXPL_EMB --> CONCAT1
        TLAG_EMB --> CONCAT1
        CONCAT1 --> DENSE1[Dense Layer]
        DENSE1 --> ENC_EMB[Encoder Embeddings]
    end
    
    subgraph DecoderEmbed["Decoder Embeddings"]
        TLAG --> TLAG_EMB2[Time Lag Embedding]
        ETIME --> ETIME_EMB["Elapsed Time Embedding<br/>(log scaled)"]
        ACORR --> ACORR_EMB[Answer Correctness Embedding]
        UANS --> UANS_EMB[User Answer Embedding]
        TLAG_EMB2 --> CONCAT2[Concatenate]
        ETIME_EMB --> CONCAT2
        ACORR_EMB --> CONCAT2
        UANS_EMB --> CONCAT2
        CONCAT2 --> DENSE2[Dense Layer]
        DENSE2 --> DEC_EMB[Decoder Embeddings]
    end
    
    subgraph Position["Position Encoding"]
        POS[Position IDs] --> POS_EMB[Position Embedding]
        ENC_EMB --> ADD_POS1[Add Position]
        POS_EMB --> ADD_POS1
        DEC_EMB --> ADD_POS2[Add Position]
        POS_EMB --> ADD_POS2
    end
    
    subgraph Transformer["Transformer Layers"]
        ADD_POS1 --> ENC_TRANS[Encoder<br/>Self-Attention]
        ADD_POS2 --> DEC_TRANS[Decoder<br/>Self-Attention +<br/>Encoder-Decoder Attention]
        ENC_TRANS --> ENC_OUT[Encoder Output]
        ENC_OUT --> DEC_TRANS
        DEC_TRANS --> DEC_OUT[Decoder Output]
    end
    
    subgraph Output["Output Layer"]
        DEC_OUT --> LAYER_NORM1[Layer Norm]
        LAYER_NORM1 --> FFN[Feed-Forward Network]
        FFN --> LAYER_NORM2[Layer Norm + Residual]
        LAYER_NORM2 --> FINAL[Linear Layer]
        FINAL --> SIGMOID[Sigmoid]
        SIGMOID --> PRED[Prediction]
    end
    
    MASK[Causal Mask] --> ENC_TRANS
    MASK --> DEC_TRANS

Loading

2.2 Component Details

Embedding Layers:

• Content Embedding: Maps question IDs to dense representations
• Part Embedding: Encodes question categories/parts
• Time Embeddings: Two linear layers process time lag and elapsed time (log-scaled)
• Answer Embeddings: Encodes answer correctness and user answer choices
• Explanation Embedding: Indicates whether explanations were viewed

Embedding Combination:

• Encoder embeddings: Concatenates content, part, explanation, and time lag embeddings
• Decoder embeddings: Concatenates time lag, elapsed time, answer correctness, and user answer embeddings
• Both streams use dense layers to project concatenated embeddings to model dimension
• Position embeddings are added (not concatenated) to both streams

Attention Mechanisms:

• Self-Attention (Encoder): Identifies relationships within the exercise sequence
• Self-Attention (Decoder): Learns patterns in response sequences
• Encoder-Decoder Attention: Correlates exercise patterns (encoder) with performance patterns (decoder)
• Causal Masking: Applied to all attention layers to prevent future information leakage

2.3 Causal Masking

A causal mask is applied to all encoder and decoder layers to ensure that predictions at position i only depend on information from positions ≤ i. This prevents data leakage and maintains temporal causality.

graph LR
    subgraph MaskMatrix["Causal Mask Matrix"]
        direction TB
        M1["1 0 0 0<br/>1 1 0 0<br/>1 1 1 0<br/>1 1 1 1"]
    end
    
    subgraph Attention["Attention Mechanism"]
        Q[Query] --> ATT[Attention]
        K[Key] --> ATT
        V[Value] --> ATT
        M1 --> ATT
        ATT --> OUT[Output]
    end
    
    subgraph Effect["Effect"]
        E1["Position 1: Sees only position 1"]
        E2["Position 2: Sees positions 1-2"]
        E3["Position 3: Sees positions 1-3"]
        E4["Position 4: Sees positions 1-4"]
    end
    
    MaskMatrix --> Attention
    Attention --> Effect

Loading

The mask creates an upper triangular matrix where:

• 1 indicates allowed attention (past and current positions)
• 0 indicates masked attention (future positions)

2.4 Time Feature Processing

Time features are processed using natural logarithm scaling to improve convergence and handle the wide range of time values.

flowchart LR
    subgraph Input["Raw Time Features"]
        TL[Time Lag<br/>minutes]
        ET[Elapsed Time<br/>seconds]
    end
    
    subgraph Preprocessing["Preprocessing"]
        TL --> CLIP1["Clip: 0-1440 min<br/>(1 day max)"]
        ET --> CLIP2["Clip: 0-300 sec"]
    end
    
    subgraph Model["Model Forward Pass"]
        CLIP1 --> LOG1["log(time_lag + 1)"]
        CLIP2 --> LOG2["log(elapsed_time + 1)"]
    end
    
    subgraph Embedding["Embedding"]
        LOG1 --> LINEAR1[Linear Layer<br/>1 → d_model]
        LOG2 --> LINEAR2[Linear Layer<br/>1 → d_model]
        LINEAR1 --> TL_EMB[Time Lag Embedding]
        LINEAR2 --> ET_EMB[Elapsed Time Embedding]
    end
    
    subgraph Usage["Usage"]
        TL_EMB --> ENC[Encoder Stream]
        TL_EMB --> DEC[Decoder Stream]
        ET_EMB --> DEC
    end

Loading

Time Feature Details:

• Time Lag: Time difference between consecutive questions
  • Clipped to 0-1440 minutes during preprocessing
  • Log-scaled in model: log(time_lag + 1)
• Elapsed Time: Time spent on each question
  • Clipped to 0-300 seconds during preprocessing
  • Log-scaled in model: log(elapsed_time + 1)
• Both features are log-scaled to normalize the distribution and improve convergence
• Time lag appears in both encoder and decoder; elapsed time only in decoder

3. Key Modifications

SAINT+ introduces several enhancements over the base SAINT architecture:

1. Prior Question Explanation (prior_question_had_explanation)

   • Added to encoder input
   • Provides context about whether the student viewed explanations
   • Helps model understand learning behavior patterns

2. Prior User Answer (prior_user_answer)

   • Added to decoder input
   • Captures answer choice patterns (e.g., repeated same choice may indicate guessing)
   • Enables detection of systematic response patterns

3. Time Feature Scaling

   • Both time features scaled by natural logarithm
   • Improves model convergence and handles wide time ranges
   • Formula: log(time_feature + 1)

4. Embedding Combination Strategy

   • Uses concatenation instead of addition for combining embeddings
   • Preserves distinct information from each embedding type
   • Position embeddings added (not concatenated) with learnable weighting

5. Normalized Learning Rate Schedule

   • Implements the learning rate schedule from the Transformer paper
   • Formula: lr = factor * d_model^(-0.5) * min(step^(-0.5), step * warmup^(-1.5))
   • Provides stable training with warmup phase

4. Model Parameters

Parameter	Value
Number of Attention Layers	2
Number of Heads	4
Embedding Dimension	128
Forward Linear Layer Dimension	512
Dropout	0.1
Max Sequence Length	100
Batch Size	512
Warm Steps	4000
Split Ratio	0.95

5. System Architecture

The SAINT+ system consists of several interconnected components that handle data preprocessing, sequence generation, model training, and inference.

graph TB
    subgraph DataPrep["Data Preprocessing"]
        RAW[Raw Data<br/>train.pkl] --> PREPROC[pre_process.py]
        QUES[Questions CSV] --> PREPROC
        PREPROC --> TIME_DICT[time_dict.pkl.zip]
        PREPROC --> TRAIN_GRP[train_group.pkl.zip]
        PREPROC --> VAL_GRP[val_group.pkl.zip]
    end
    
    subgraph DataGen["Data Generation"]
        TRAIN_GRP --> DATAGEN[data_generator.py<br/>Riiid_Sequence]
        VAL_GRP --> DATAGEN
        DATAGEN --> SEQUENCES[Sequences<br/>Fixed Length]
    end
    
    subgraph Training["Model Training"]
        SEQUENCES --> TRAIN[train.py]
        PARSER[parser.py<br/>Hyperparameters] --> TRAIN
        MODEL_DEF[model.py<br/>SaintPlus] --> TRAIN
        TRAIN --> WEIGHTS[saint.pt<br/>Best Model]
    end
    
    subgraph Inference["Inference"]
        WEIGHTS --> INFER[Load Model]
        NEW_DATA[New Sequences] --> INFER
        INFER --> PREDICTIONS[Predictions]
    end
    
    subgraph Utils["Utilities"]
        UTILS[utils.py<br/>get_time_lag] --> PREPROC
    end
    
    DataPrep --> DataGen
    DataGen --> Training
    Training --> Inference
    Utils --> DataPrep

Loading

5.1 Component Responsibilities

• pre_process.py: Handles data preprocessing, feature engineering, virtual timestamp generation, and train/validation splitting
• data_generator.py: Implements Riiid_Sequence dataset class that creates fixed-length sequences from grouped user data
• model.py: Contains the SaintPlus model implementation, FFN module, and NoamOpt optimizer
• train.py: Orchestrates the training loop, validation, and model checkpointing
• parser.py: Defines command-line arguments for hyperparameter configuration
• utils.py: Provides utility functions like get_time_lag for feature computation

6. Cross-Validation Strategy

The preprocessing strategy employs a time series cross-validation approach to ensure proper temporal splitting of training and validation data.

6.1 Challenge

The original timestamp feature only indicates elapsed time since a user's first event, not absolute time. This makes it impossible to directly split data by time across all users.

6.2 Solution: Virtual Timestamp

The strategy creates virtual start times for each user to enable proper time-based splitting:

flowchart TD
    subgraph Step1["Step 1: Find Maximum"]
        ALL_USERS[All User Timestamps] --> MAX_TS[Find Max Timestamp<br/>Across All Users]
    end
    
    subgraph Step2["Step 2: Calculate Intervals"]
        MAX_TS --> CALC["For Each User:<br/>interval = Max_TS - user_max_ts"]
        CALC --> INTERVALS[User Start Intervals]
    end
    
    subgraph Step3["Step 3: Generate Virtual Times"]
        INTERVALS --> RAND["Random Time Selection<br/>within interval"]
        RAND --> VIRTUAL_START[Virtual Start Time]
        USER_TS[User Timestamps] --> ADD["Add Virtual Start<br/>to User Timestamps"]
        VIRTUAL_START --> ADD
        ADD --> VIRTUAL_TS[Virtual Timestamps]
    end
    
    subgraph Step4["Step 4: Split by Time"]
        VIRTUAL_TS --> SORT[Sort by Virtual Timestamp]
        SORT --> SPLIT["Split at split_ratio<br/>(default: 0.95)"]
        SPLIT --> TRAIN_DATA[Training Data]
        SPLIT --> VAL_DATA[Validation Data]
    end
    
    Step1 --> Step2
    Step2 --> Step3
    Step3 --> Step4

Loading

6.3 Benefits

• Temporal Validity: Ensures validation data comes from later time periods than training data
• User Balance: Prevents over-representation of heavy users in validation set
• Realistic Evaluation: Mimics real-world scenarios where predictions are made on future data

7. Installation

7.1 Prerequisites

• Python 3.7 or higher
• PyTorch 1.9.0 or higher
• CUDA (optional, for GPU acceleration)

7.2 Install Dependencies

# Clone the repository
git clone https://github.com/Organization-non-question-proficiency-evaluation/Video-Proficiency-SAINT_Plus.git
cd Video-Proficiency-SAINT_Plus

# Install dependencies
pip install -r requirements.txt

# Or install as a package
pip install -e .

7.3 Verify Installation

python -c "import torch; from saint_plus import SaintPlus; print('Installation successful!')"

8. Usage

8.1 Data Preprocessing

Before training, prepare your data using the preprocessing script:

Option 1: Run as a module (after installation):

python -m saint_plus.pre_process

Option 2: Run directly:

python src/pre_process.py

Important: Update the train_path and ques_path variables in src/pre_process.py before running.

The preprocessing script will generate:

• train_group.pkl.zip: Training data grouped by user
• val_group.pkl.zip: Validation data grouped by user
• time_dict.pkl.zip: Time dictionary for inference

8.2 Training

1. Configure Hyperparameters (optional): Adjust parameters in src/parser.py or pass via command-line arguments

2. Run Training:

python scripts/train.py

With custom parameters:

python scripts/train.py --num_layers 2 --num_heads 4 --d_model 128 --batch_size 512

The training script will:

• Load preprocessed training and validation data
• Initialize the SAINT+ model with specified parameters
• Train using the NoamOpt learning rate schedule
• Save the best model based on validation AUC
• Output training progress, loss, and metrics

8.3 Using Pre-trained Weights

To load and use a pre-trained model:

import torch
from saint_plus import SaintPlus

# Initialize model with same architecture as training
model = SaintPlus(
    seq_len=100,
    num_layers=2,
    d_ffn=512,
    d_model=128,
    num_heads=4,
    max_len=1000,
    n_questions=13523,
    n_parts=7,
    n_tasks=10000,
    n_ans=4,
    dropout=0.1
)

# Load pre-trained weights
model.load_state_dict(torch.load("weights/saint_2_128.pt"))
model.eval()

# Use model for inference
# ... prepare input tensors ...
# predictions = model(content_ids, parts, time_lag, ques_elapsed_time, 
#                     answer_correct, ques_had_explian, user_answer)