v1.4.0 (#43)

* Feature/workflows (#8)

* chore: add codeowners file

* chore: add python poetry action and docs workflow

* chore: update pre-commit file

* chore: update docs

* chore: update logo

* chore: add cicd pipeline for automated deployment

* chore: update poetry version

* chore: fix action versioning

* chore: add gitattributes to ignore line count in jupyter notebooks

* chore: add and update docstrings

* chore: fix end of files

* chore: update action versions

* Update README.md

---------

Co-authored-by: mo374z <schlager.mo@t-online.de>

* Fix/workflows (#11)

* chore: fix workflow execution

* chore: fix version check in CICD pipeline

* Opro implementation (#7)

* update gitignore

* initial implementation of opro

* formatting of prompt template

* added opro test run

* opro refinements

* fixed sampling error

* add docs to opro

* fix pre commit issues#

* fix pre commit issues#

* fixed end of line

* Patch/pre commit config (#10)

* fixed pre commit config and removed end of file line breaks in tempaltes

* added /

* Feature/prompt generation (#12)

* added prompt_creation.py

* change version

* Create LICENSE (#14)

* Refactor/remove deepinfra (#16)

* Remove deepinfra file

* change langchain-community version

* Usability patches (#15)

* renamed get_tasks to get_task and change functionality accordingly. moved templates and data_sets

* init

* move templates to templates.py

* Add nested asyncio to make it useable in notebooks

* Update README.md

* changed getting_started.ipynb and created helper functions

* added sampling of initial population

* fixed config

* fixed callbacks

* adjust runs

* fix run evaluation api token

* fix naming convention in opro, remove on epoch end for logger callback, fixed to allow for numeric values in class names

* Update promptolution/llms/api_llm.py

Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

* fixed comments

* Update pyproject.toml

* resolve comments

---------

Co-authored-by: mo374z <schlager.mo@t-online.de>
Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>
Co-authored-by: Moritz Schlager <87517800+mo374z@users.noreply.github.com>

* Feature/examplar selection (#17)

* implemented random selector

* added random search selector

* increased version count

* fix typos

* Update promptolution/predictors/base_predictor.py

Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

* Update promptolution/tasks/classification_tasks.py

Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

* resolve comments

* resolve comments

---------

Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

* Chore/docs release notes (#18)

* Update release-notes.md

* Fix release note links

* revert Chore/docs release notes (#18)"

This reverts commit e23dd74.

* revert last commit

* updated release notes and read me

* Feature/read from df (#21)

* Delete Experiment files

* Removed config necessities

* improved opro meta-prompts

* added read from data frame feature

* changed required python version to 3.9

* Update pyproject.toml

* Update release-notes.md

* merge

* merge

* resolve merge mistakes

* delete duplicated lines

* Update release-notes.md (#24)

* Fix/dependencies (#28)

* delete poetry.lock and upgrade transformers dependency

* Update release-notes.md

* Add vllm as feature and a llm_test_run_script

* small fixes in vllm class

* differentiate between vllm and api inference

* set up experiment over multiple tasks and prompts

* change csv saving

* add base llm super class

* add changes from PR review

* change some VLLM params

* fix tensor parallel size to 1

* experiment with batch size

* experiment with larger batch sizes

* add continuous batch llm

* remove arg

* remove continuous batch inference try

* add batching to vllm

* add batching in script

* Add release notes and increase version number

* remove llm_test_run.py script

* change system prompt

* Fix/vllm (#33)

* add token count, flexible batch size and kwargs to vllm class

* add testing script for implementation

* fix batch size calculation

* small changes

* add revision test

* add argument to parser

* max model len to int

* remove script

* Change version and Release notes

* changed callback behaviour and impelemented token count callback

* added super inits

* allow for splits not based on white space (such as new line break etc)

* include task descriptions

* add tokenizer based token count to vllm class

* update test run script

* use classifiers accordingly

* small fix

* add storage path

* helpers should use classificator

* use different model

* changes in opro test

* change get_predictor function

* fix callback calling

* change optimizer test run script

* small alignments

* small alignments

* small alignments

* some changes to match the current optimizer implementation

* changes in template and config

* allow for batching of prompt creation

* update release notes and version

* extend csvcallback functionality

* change callback csv export

* change step time calculation

* small changes

* remove llm_test_run script

* update release notes

* fix issues in token stepswise calculation

* small fix

---------

Co-authored-by: finitearth <t.zehle@gmail.com>

* implement changes from review

* add typing to token count callback

* Feature/deterministic (#35)

* make vllm class deterministic
* fixes in prompt creation

* Fix/prompt creation (#36)

* fixes in the "MarkerBasedClassificator"
* generalize the Classificator
* add verbosity and callback handling in EvoPromptGA
* add timestamp to the callback

* add arguements to test script

* added some feature notes

* Fix/template (#39)

* v1.3.1 (#37)

#### Added features
* new features for the VLLM Wrapper (accept seeding to ensure reproducibility)
* fixes in the "MarkerBasedClassificator"
* fixes in prompt creation and task description handling
* generalize the Classificator
* add verbosity and callback handling in EvoPromptGA
* add timestamp to the callback
* removed datasets from repo
* changed task creation (now by default with a dataset)

* add generation prompt to vllm input

* allow for parquet as fileoutput callback

* added sys_prompts

* change usage of csv callbacks

* add system prompt to token counts

* fix merge issues

* drag system prompts from api to task

* added release notes

---------

Co-authored-by: Moritz Schlager <87517800+mo374z@users.noreply.github.com>

* Fix/opro (#38)

* v1.3.1 (#37)

#### Added features
* new features for the VLLM Wrapper (accept seeding to ensure reproducibility)
* fixes in the "MarkerBasedClassificator"
* fixes in prompt creation and task description handling
* generalize the Classificator
* add verbosity and callback handling in EvoPromptGA
* add timestamp to the callback
* removed datasets from repo
* changed task creation (now by default with a dataset)

* opro reimplementation according to the paper

* fine opro implementation

* opro test scripts alignment

* implement opro review

* small fix in score handling

* adjust hyperparameters

* add early stopping at convergence to opro

* Update promptolution/optimizers/opro.py

Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

---------

Co-authored-by: Moritz Schlager <87517800+mo374z@users.noreply.github.com>
Co-authored-by: mo374z <schlager.mo@t-online.de>
Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

* Refactor/generic api llm (#41)

* v1.3.2 (#40)

#### Added features
* Allow for configuration and evaluation of system prompts in all LLM-Classes
* CSV Callback is now FileOutputCallback and able to write Parquet files
* Fixed LLM-Call templates in VLLM
* refined OPRO-implementation to be closer to the paper

* implement api calls

* removed default for system messages

* roll back renaming

---------

Co-authored-by: mo374z <schlager.mo@t-online.de>

* Refactor/interfaces (#42)

* add token count, flexible batch size and kwargs to vllm class

* add testing script for implementation

* fix batch size calculation

* small changes

* add revision test

* add argument to parser

* max model len to int

* remove script

* Change version and Release notes

* changed callback behaviour and impelemented token count callback

* init

* small corrections

* changes to prevent merge conflicts

* small changes

* first tests

* add api_test

* a lot has changed for gangsters

* fix

* added

* removed

* fix

* Update promptolution/predictors/__init__.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* add testing to ci pipeline

* Update action.yml

* fix test dependencies in pipeline

* Add further test dependencies

* Refactor dependency groups

* Add getting started notebook to documentation

* Add workflow call trigger to docs workflow

* Add CI and Docs status badges

* Add temporary file for docs testing

* Remove temporary file for docs testing

* Update notebooks/getting_started.ipynb

Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

* Update notebooks/getting_started.ipynb

Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

* comments

* end2end

* added

* new

* more

* change

* update dependencies

* added literals to get optimizer

* added release notes

---------

Co-authored-by: mo374z <schlager.mo@t-online.de>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Timo Heiß <ti-heiss@t-online.de>
Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>

* Remove obsolete Docstring

* remove redundant init

* remove redundant init

* resolve merge chaos

* formatting

* remove obsolete reproduce experiments part in readme

* Update README.md

---------

Co-authored-by: Timo Heiß <87521684+timo282@users.noreply.github.com>
Co-authored-by: mo374z <schlager.mo@t-online.de>
Co-authored-by: Moritz Schlager <87517800+mo374z@users.noreply.github.com>
Co-authored-by: Timo Heiß <ti-heiss@t-online.de>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Author: 4 people

.coverage

@@ -34,5 +34,6 @@ runs:
         path: .venv
         key: ${{ runner.os }}-poetry-${{ inputs.poetry-version }}-${{ hashFiles('**/poetry.lock') }}
     - name: Install dependencies
-      run: poetry install --no-interaction --no-root --only ${{ inputs.groups }}
+      run:
+        poetry install --no-interaction --no-root --only ${{ inputs.groups }}
       shell: bash

.github/actions/python-poetry/action.yml

@@ -21,11 +21,11 @@ jobs:
       - name: Set up Python and Poetry
         uses: ./.github/actions/python-poetry
         with:
-          groups: dev
+          groups: main,dev,test
       - name: Run pre-commit
         uses: pre-commit/action@v3.0.1
-      # - name: Run tests
-      #   run: poetry run python -m pytest
+      - name: Run tests
+        run: poetry run python -m pytest
 
   build:
     runs-on: ubuntu-latest

.github/workflows/ci.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+  workflow_call:
 
 permissions:
   contents: write
@@ -22,6 +23,10 @@ jobs:
         with:
           groups: "docs"
 
+      - name: Generate notebook examples
+        run: |
+          poetry run jupyter nbconvert --to markdown --allow-errors --output-dir docs/examples notebooks/*.ipynb
+
       - name: Deploy docs
         run: |
           poetry run mkdocs gh-deploy --force

.github/workflows/docs.yml

@@ -9,3 +9,5 @@ dist/
 outputs/
 results/
 poetry.lock
+CLAUDE.md
+**/CLAUDE.local.md

.gitignore

@@ -1,19 +1,33 @@
 ![promptolution](https://github.com/user-attachments/assets/84c050bd-61a1-4f2e-bc4e-874d9b4a69af)
+
+[![CI](https://github.com/finitearth/promptolution/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/finitearth/promptolution/actions/workflows/ci.yml)
+[![Docs](https://github.com/finitearth/promptolution/actions/workflows/docs.yml/badge.svg?branch=main)](https://github.com/finitearth/promptolution/actions/workflows/docs.yml)
+
 # Promptolution
 Promptolution is a library that provides a modular and extensible framework for implementing prompt tuning experiments. It offers a user-friendly interface to assemble the core components for various prompt optimization tasks.
 
 In addition, this repository contains our experiments for the paper "Towards Cost-Effective Prompt Tuning: Evaluating the Effects of Model Size, Model Family and Task Descriptions in EvoPrompt".
 
 This project was developed by [Timo Heiß](https://www.linkedin.com/in/timo-heiss/), [Moritz Schlager](https://www.linkedin.com/in/moritz-schlager/) and [Tom Zehle](https://www.linkedin.com/in/tom-zehle/).
 
+## Getting Started
+Take a look at our getting started notebook: [getting_started.ipynb](https://github.com/finitearth/promptolution/blob/main/notebooks/getting_started.ipynb)
+
 ## Installation
 
 Use pip to install our library:
 
 ```
-pip install promptolution
+pip install promptolution[api]
+```
+
+If you want to run your prompt optimization locally, either via transformers or vLLM, consider running:
+
+```
+pip install promptolution[vllm,transformers]
 ```
 
+
 Alternatively, clone the repository, run
 
 ```
@@ -58,17 +72,5 @@ Create API Keys for the models you want to use:
 - Detailed logging and callback system for optimization analysis
 
 
-## Getting Started
-Take a look at our getting started notebook: [getting_started.py](https://github.com/finitearth/promptolution/blob/main/notebooks/getting_started.ipynb)
-## Reproduce our Experiments
-
-We provide scripts and configs for all our experiments. Run experiments based on config via:
-
-```
-poetry run python scripts/experiment_runs.py --experiment "configs/<my_experiment>.ini"
-```
-where `<my_experiment>.ini` is a config based on our templates.
-
-
 
-This project was developed for seminar "AutoML in the age of large pre-trained models" at LMU Munich.
+This project was developed for the "AutoML in the age of large pre-trained models" seminar at LMU Munich.

README.md

@@ -1,6 +1,6 @@
 # Config
 
-::: promptolution.config.Config
+::: promptolution.config.ExperimentConfig
     options:
       show_root_heading: true
       show_root_full_path: false