core implementation

Author: Zihan Wang

.DS_Store

@@ -0,0 +1,206 @@
+# AIM
+
+This repository contains the implementations of our paper:  
+**"AI Model Modulation with Logits Redistribution"**.
+
+## Introduction
+
+Large-scale models are typically adapted to meet the diverse requirements of model owners and users. However, maintaining multiple specialized versions of the model is inefficient. In response, we propose AIM, a novel model modulation paradigm that enables a single model to exhibit diverse behaviors to meet the specific end requirements. AIM enables two key modulation modes: utility and focus modulations. The former provides model owners with dynamic control over output quality to deliver varying utility levels, and the latter offers users precise control to shift model’s focused input features. AIM introduces a logits redistribution strategy that operates in a training data-agnostic and retraining-free manner. We establish a formal foundation to ensure AIM’s regulation capability, based on the statistical properties of logits ordering via joint probability distributions. Our evaluation confirms AIM’s practicality and versatility for AI model modulation, with tasks spanning image classification, semantic segmentation and text generation, and prevalent architectures including ResNet, SegFormer and Llama.
+
+## CV Task1: Image Classification
+### 1. Project Structure
+
+```
+img_classification
+├── dataset/
+└── model/
+└── val.py
+            
+```
+
+### 2. Quick Start
+* Install dependencies
+```python
+pip install torch torchvision matplotlib 
+```
+* Run evaluation (CIFAR-10/100 example)
+For CIFAR10/CIFAR100, we use pretrained models from: ***[PyTorch CIFAR Models](https://github.com/chenyaofo/pytorch-cifar-models).*** 
+```python
+python img_classification/val.py --dataset cifar10 --model resnet56 --batch_size 128 --num_perturb_steps 101 --perturb_step 0.2 --plot --save_plot
+```
+- Results from Our paper for reference:
+<br><img width="380" alt="image" src="https://github.com/user-attachments/assets/77c6881f-23b6-4679-8ff1-50f628e12329" />
+
+
+
+## CV Task2: Semantic Segmentation
+### 1. Project Structure
+
+```
+semantic-segmentation-main/
+├── assests/                 # Test images
+├── configs/                # Configuration files
+├── data/                   # Dataset root
+│   └── (example) ADEChallengeData2016/
+│       ├── annotations/
+│       └── images/
+├── tools/perturbation/
+│          ├── models.py                      # Perturbation models
+│          ├── evaluation.py                  # Metric evaluation
+│          └── infer_perturbation.py          # Inference script
+└── output/                 # Results directory
+            
+```
+
+### 2. Setup Instructions
+- Please download base from ***[Semantic Segmentation](https://github.com/sithu31296/semantic-segmentation/tree/main)*** or Clone base repository
+```bash
+git clone https://github.com/sithu31296/semantic-segmentation.git
+```
+- Set up env
+```python
+pip install -e .
+```
+- Navigate to the base directory
+```bash
+cd semantic-segmentation
+```
+```bash
+git clone https://github.com/CurtisYoung18/AIM.git
+```
+- ***Copy and Paste*** our segmentaion py files to semantic-segmentation-main/tools/
+```bash
+cp -r AIM-main/semantic_segmentation/perturbation tools/
+```
+- Download datasets(eg.***[ADE20K datasets](http://sceneparsing.csail.mit.edu/)***). Place the dataset in the root directory and update the dataset path in configs accordingly
+- Download the backbone model(eg. ***[Backbones](https://github.com/sithu31296/semantic-segmentation/blob/main/docs/BACKBONES.md), [Pretrained Models](https://github.com/sithu31296/semantic-segmentation/blob/main/docs/MODELS.md)***). Update the configuration file accordingly
+- Add configs to desired yaml files:
+```python
+PERTURBATION:
+  METHOD: 'single'           # Options: 'single' or 'multi'
+  RATIO: 0.2                # Base perturbation ratio
+  STEP_SIZE: 0.4            # Ratio increment step
+  STEP_LIMIT: 10            # Number of perturbation steps
+  TARGET_IDX: 136           # Class index for single perturbation
+  POSITIVE: true            # Perturbation direction
+  TARGETS:                  # For multi-class perturbation
+    - index: 136
+      positive: true
+    - index: 20
+      positive: false
+```
+- (Optional)Download inference pictures(eg. ***[Kitty](https://www.kaggle.com/datasets/klemenko/kitti-dataset)***) for testing and place them to assests(eg. /ade)
+
+
+### 3.Run
+<details open>
+  <summary><strong>Inference</strong></summary>
+
+To make an inference, edit the parameters of the config file from below.
+* Change `MODEL` >> `NAME` and `BACKBONE` to your desired pretrained model.
+* Change `DATASET` >> `NAME` to the dataset name depending on the pretrained model.
+* Set `TEST` >> `MODEL_PATH` to pretrained weights of the testing model.
+* Change `TEST` >> `FILE` to the file or image folder path you want to test.
+* Testing results will be saved in `SAVE_DIR`.
+
+- Example:
+```python
+python tools/perturbation/infer_perturbation.py \
+  --cfg configs/ade20k.yaml \
+  --pert_ratio 0.2 \
+  --pert_method single
+```
+</details>
+
+<details open>
+<summary><strong>Evaluation</strong></summary>
+
+```python
+tools/perturbation/evaluation.py --cfg configs/ade20k.yaml
+```
+</details>
+
+- Results from Our paper for reference:
+<br> ![image](https://github.com/user-attachments/assets/115bab21-27c3-4bdc-85dd-1e12a5fd1f50)
+
+
+### 4. Core Components
+
+| File                  | Description                                                   |
+|-----------------------|---------------------------------------------------------------|
+| `models.py`          | Implements SingleClassPerturbator and MultiClassPerturbator   |
+| `evaluation.py`      | Evaluation is performed over a range of perturbation ratios on the "person" class.        |
+| `infer_perturbation.py` | Visualization pipeline with perturbation controls         |
+
+
+## NLP Task: Modulation Modultion during Text Generation
+We tailored the OpenCompass framework to include a new class `HuggingFaceNoiseModel` which introduces the capability to inject Gaussian noise into the logits during the generation process of a Hugging Face model, thus enable model modulation for AI Models.  
+
+## Key Modifications
+
+Gaussian Noise Injection:
+
+The `GaussianNoiseLogitsProcessor` class is responsible for adding Gaussian noise to the logits (scores) during the generation process. The noise is sampled from a normal distribution with a configurable mean and standard deviation (std).
+
+Integration with Generation Configuration:
+
+The HuggingFaceNoiseModel class checks for the presence of a noise_std parameter in the generation_kwargs. If found, it initializes the GaussianNoiseLogitsProcessor with the specified standard deviation and adds it to the LogitsProcessorList.
+
+The noise_std parameter is then removed from generation_kwargs to avoid conflicts during the generation process.
+
+OpencCompass Config:
+
+THe main script is eval_noise_llama3.py, which contains the model, datasets, and std config. You can reproduce our experiments following Usage step by step.
+
+
+## Usage
+To use the HuggingFaceNoiseModel class, follow these steps:
+
+### 🛠️ Installation
+
+Below are the steps for quick installation and datasets preparation.
+
+#### 💻 Environment Setup
+
+We highly recommend using conda to manage your python environment.
+
+- #### Create your virtual environment
+
+  ```bash
+  conda create --name opencompass python=3.10 -y
+  conda activate opencompass
+  ```
+
+- #### Install OpenCompass via pip
+
+  ```bash
+    pip install -U opencompass
+
+    ## Full installation (with support for more datasets)
+    # pip install "opencompass[full]"
+
+    ## Environment with model acceleration frameworks
+    ## Manage different acceleration frameworks using virtual environments
+    ## since they usually have dependency conflicts with each other.
+    # pip install "opencompass[lmdeploy]"
+    # pip install "opencompass[vllm]"
+
+    ## API evaluation (i.e. Openai, Qwen)
+    # pip install "opencompass[api]"
+  ```
+
+- #### Install OpenCompass from source
+
+  If you want to use opencompass's latest features, or develop new features, you can also build it from source
+
+  ```bash
+    git clone https://github.com/open-compass/opencompass opencompass
+    cd opencompass
+    pip install -e .
+    # pip install -e ".[full]"
+    # pip install -e ".[vllm]"
+  ```
+- ### Run command
+  ```bash
+    python -u run.py configs/eval_noise_llama3.py
+  ```

README.md

@@ -0,0 +1,5 @@
+[codespell]
+skip = *.ipynb
+count =
+quiet-level = 3
+ignore-words-list = nd, ans, ques, rouge, softwares, wit

custom_opencompass/.codespellrc

@@ -0,0 +1,94 @@
+name: 🐞 Bug report
+description: Create a report to help us improve
+labels: ["bug"]
+title: "[Bug] "
+body:
+  - type: markdown
+    attributes:
+      value: |
+        For general questions or idea discussions, please post it to our [**Forum**](https://github.com/open-compass/opencompass/discussions).
+        If you have already identified the reason, we strongly appreciate you creating a new PR according to [the tutorial](https://opencompass.readthedocs.io/en/master/community/CONTRIBUTING.html)!
+        If you need our help, please fill in the following form to help us to identify the bug.
+
+  - type: checkboxes
+    attributes:
+      label: Prerequisite
+      description: Please check the following items before creating a new issue.
+      options:
+      - label: I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
+        required: true
+      - label: The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).
+        required: true
+
+  - type: dropdown
+    id: task
+    attributes:
+      label: Type
+      description: The problem arises when
+      options:
+        - I'm evaluating with the officially supported tasks/models/datasets.
+        - I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.
+    validations:
+      required: true
+
+  - type: textarea
+    id: environment
+    validations:
+      required: true
+    attributes:
+      label: Environment
+      description: |
+        Please run `python -c "import opencompass.utils;import pprint;pprint.pprint(dict(opencompass.utils.collect_env()))"` to collect necessary environment information and paste it here.
+      placeholder: |
+        ```python
+        # The output the above command
+        ```
+
+  - type: textarea
+    attributes:
+      label: Reproduces the problem - code/configuration sample
+      description: |
+        Please provide a code or configuration sample that reproduces the problem you ran into. It can be a Colab link or just a code snippet.
+      placeholder: |
+        ```python
+        # Sample code to reproduce the problem
+        ```
+    validations:
+      required: true
+
+  - type: textarea
+    attributes:
+      label: Reproduces the problem - command or script
+      description: |
+        What command or script did you run?
+      placeholder: |
+        ```shell
+        The command or script you run.
+        ```
+    validations:
+      required: true
+
+  - type: textarea
+    attributes:
+      label: Reproduces the problem - error message
+      description: |
+        Please provide the error message or logs you got, with the full traceback.
+
+        Tip: You can attach images or log files by dragging them into the text area..
+      placeholder: |
+        ```
+        The error message or logs you got, with the full traceback.
+        ```
+    validations:
+      required: true
+
+  - type: textarea
+    id: other
+    attributes:
+      label: Other information
+      description: |
+        Tell us anything else you think we should know.
+
+        1. What's your expected result?
+        2. What dataset did you use?
+        3. What do you think might be the reason?

custom_opencompass/.github/ISSUE_TEMPLATE/1_bug-report.yml

@@ -0,0 +1,30 @@
+name: 🚀 Feature request
+description: Suggest an idea for this project
+labels: ["enhancement"]
+title: "[Feature] "
+body:
+  - type: markdown
+    attributes:
+      value: |
+        For general questions or idea discussions, please post it to our [**Forum**](https://github.com/open-compass/opencompass/discussions).
+        If you have already implemented the feature, we strongly appreciate you creating a new PR according to [the tutorial](https://opencompass.readthedocs.io/en/master/community/CONTRIBUTING.html)!
+
+  - type: textarea
+    id: describe
+    validations:
+      required: true
+    attributes:
+      label: Describe the feature
+      description: |
+        What kind of feature do you want OpenCompass to add. If there is an official code release or third-party implementation, please also provide the information here, which would be very helpful.
+      placeholder: |
+        A clear and concise description of the motivation of the feature.
+        Ex1. It is inconvenient when \[....\].
+        Ex2. There is a recent paper \[....\], which is very helpful for \[....\].
+
+  - type: checkboxes
+    id: pr
+    attributes:
+      label: Will you implement it?
+      options:
+        - label: I would like to implement this feature and create a PR!