Skip to content

Latest commit

 

History

History
27 lines (16 loc) · 2.36 KB

add_new_modules.md

File metadata and controls

27 lines (16 loc) · 2.36 KB
Raw

Each module in promptbench can be easily extended. In the following, we provide basic guidelines for customizing your own datasets, models, prompt engineering methods, and evaluation metrics.

Add new datasets

Adding new datasets involves two steps:

  • Implementing a New Dataset Class: Datasets are supposed to be implemented in dataload/dataset.py and inherit from the Dataset class. For your custom dataset, implement the __init__ method to load your dataset. We recommend organizing your data samples as dictionaries to facilitate the input process.
  • Adding an Interface: After customizing the dataset class, register it in the DataLoader class within dataload.py.

Add new models

Similar to adding new datasets, the addition of new models also consists of two steps.

  • Implementing a New Model Class: Models should be implemented in models/model.py, inheriting from the LLMModel class. In your customized model, you should implement self.tokenizer and self.model. You may also customize your own predict function for inference. If the predict function is not customized, the default predict function inherited from LLMModel will be used.
  • Adding an Interface: After customizing the model class, register it in the _create_model function within the class LLMModel and MODEL_LIST dictionary in __init__.py.

Add new prompt engineering methods

Adding new methods in prompt engineering is similar to steps of adding new datasets and models.

  • Implementing a New Methods Class: Methods should be implemented in \ prompt\_engineering Module. Firstly, create a new .py file for your methods. Then implement two functions: \_\_init\_\_ and query. For unified management, two points need be noticed: 1. all methods should inherits from Base class that has common code for prompt engineering methods. 2. prompts used in methods should be stored in prompts/method\_oriented.py.
  • Adding an Interface: After implementing a new methods, register it in the METHOD\_MAP that is used to map method names to their corresponding class.

Add new metrics and input/output process functions

New evaluation metrics should be implemented as static functions in class Eval within the metrics module. Similarly, new input/output process functions should be implemented as static functions in class InputProcess and class OutputProcess in the utils module.