Skip to content

Commit aea25f8

Browse files
mo374zmo374z
authored
Add test coverage report to ci workflow (#46)
* add test coverage report to ci workflow * add coverage to readme automatically * remove condition * small fixes * Update coverage badge and report in README [skip ci] * fixes * change update readme function * Update coverage badge in README [skip ci] * remove duplicates in readme * Update coverage badge in README [skip ci] * add new badges to readme * Update coverage badge in README [skip ci] --------- Co-authored-by: mo374z <mo374z@users.noreply.github.com>
1 parent cac2db2 commit aea25f8

File tree

2 files changed

+64
-17
lines changed

2 files changed

+64
-17
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 53 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
name: CI
2-
32
on:
43
push:
54
branches:
@@ -12,37 +11,81 @@ on:
1211
workflow_call:
1312
workflow_dispatch:
1413

14+
permissions:
15+
contents: write
16+
checks: write
17+
pull-requests: write
18+
1519
jobs:
1620
test:
1721
runs-on: ubuntu-latest
1822
steps:
19-
- name: checkout
23+
- name: Checkout code
2024
uses: actions/checkout@v3
25+
2126
- name: Set up Python and Poetry
2227
uses: ./.github/actions/python-poetry
2328
with:
2429
groups: main,dev,test
30+
2531
- name: Run pre-commit
2632
uses: pre-commit/action@v3.0.1
27-
- name: Run tests
28-
run: poetry run python -m pytest
33+
34+
- name: Run tests with coverage
35+
run: |
36+
poetry run python -m pytest --junitxml=pytest.xml --cov-report=term-missing:skip-covered --cov=. tests/ | tee pytest-coverage.txt
37+
38+
- name: Generate coverage report & comment on PR
39+
id: coverageComment
40+
uses: MishaKav/pytest-coverage-comment@main
41+
with:
42+
pytest-coverage-path: ./pytest-coverage.txt
43+
junitxml-path: ./pytest.xml
44+
hide-report: true
45+
46+
- name: Update README with coverage badge
47+
run: |
48+
cat > update_readme.py << 'EOF'
49+
import re
50+
coverage = "${{ steps.coverageComment.outputs.coverage }}"
51+
color = "${{ steps.coverageComment.outputs.color }}"
52+
coverage_url = coverage.replace('%', '%25')
53+
badge_md = f"![Coverage](https://img.shields.io/badge/Coverage-{coverage_url}-{color})"
54+
with open('README.md', 'r') as f:
55+
content = f.read()
56+
new_content = re.sub(
57+
r'!\[Coverage\]\(https://img\.shields\.io/badge/Coverage-[^)]+\)',
58+
badge_md,
59+
content
60+
)
61+
with open('README.md', 'w') as f:
62+
f.write(new_content)
63+
EOF
64+
python update_readme.py
65+
66+
- name: Commit coverage updates to README
67+
uses: stefanzweifel/git-auto-commit-action@v4
68+
with:
69+
commit_message: "Update coverage badge in README [skip ci]"
70+
file_pattern: README.md
71+
commit_user_name: "github-actions[bot]"
72+
commit_user_email: "github-actions[bot]@users.noreply.github.com"
2973

3074
build:
3175
runs-on: ubuntu-latest
3276
needs: test
33-
3477
steps:
35-
- name: checkout
78+
- name: Checkout code
3679
uses: actions/checkout@v3
37-
80+
3881
- name: Set up Python and Poetry
3982
uses: ./.github/actions/python-poetry
40-
83+
4184
- name: Build wheel
4285
run: poetry build --format wheel
43-
86+
4487
- name: Upload wheel
4588
uses: actions/upload-artifact@v4
4689
with:
4790
name: ${{ github.event.repository.name }}
48-
path: dist/
91+
path: dist/

‎README.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,15 @@
11
![promptolution](https://github.com/user-attachments/assets/84c050bd-61a1-4f2e-bc4e-874d9b4a69af)
22

3+
4+
![Coverage](https://img.shields.io/badge/Coverage-90%25-brightgreen)
35
[![CI](https://github.com/finitearth/promptolution/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/finitearth/promptolution/actions/workflows/ci.yml)
46
[![Docs](https://github.com/finitearth/promptolution/actions/workflows/docs.yml/badge.svg?branch=main)](https://github.com/finitearth/promptolution/actions/workflows/docs.yml)
7+
![Code Style](https://img.shields.io/badge/Code%20Style-black-black)
8+
![Python Versions](https://img.shields.io/badge/Python%20Versions-≥3.9-blue)
59

610
# Promptolution
7-
Promptolution is a library that provides a modular and extensible framework for implementing prompt tuning experiments. It offers a user-friendly interface to assemble the core components for various prompt optimization tasks.
811

9-
In addition, this repository contains our experiments for the paper "Towards Cost-Effective Prompt Tuning: Evaluating the Effects of Model Size, Model Family and Task Descriptions in EvoPrompt".
12+
Promptolution is a library that provides a modular and extensible framework for implementing prompt tuning experiments. It offers a user-friendly interface to assemble the core components for various prompt optimization tasks.
1013

1114
This project was developed by [Timo Heiß](https://www.linkedin.com/in/timo-heiss/), [Moritz Schlager](https://www.linkedin.com/in/moritz-schlager/) and [Tom Zehle](https://www.linkedin.com/in/tom-zehle/).
1215

@@ -48,11 +51,12 @@ Create API Keys for the models you want to use:
4851
- DeepInfra (for Llama): store token in deepinfratoken.txt
4952

5053
## Optimization Algorithms to choose from
51-
| **Name** | **# init population** | **Exploration** | **Costs** | **Convergence Speed** | **Parallelizable** | **Utilizes Failure Cases** |
52-
|:--------:|:---------------------:|:---------------:|:---------:|:---------------------:|:------------------:|:---------------------:|
53-
| EvoPrompt DE | 8-12 | 👍 | 💲 | ⚡⚡ | ✅ | ❌ |
54-
| EvoPrompt GA | 8-12 | 👍 | 💲 | ⚡⚡ | ✅ | ❌ |
55-
| OPro | 0 | 👎 | 💲💲 | ⚡ | ❌ | ❌ |
54+
| **Name** | **Paper** | **init prompts** | **Exploration** | **Costs** | **Convergence Speed** | **Parallelizable** | **Utilizes Fewshot Examples** |
55+
|:--------:|:-------:|:--------------:|:---------------:|:---------:|:---------------------:|:------------------:|:---------------------:|
56+
| `CAPO` | [Zehle et al.](https://arxiv.org/abs/2504.16005)| *required* | 👍 | 💲 | ⚡⚡ | ✅ | ✅ |
57+
| `EvoPromptDE` | [Guo et al.](https://arxiv.org/abs/2309.08532) | *required* | 👍 | 💲💲 | ⚡⚡ | ✅ | ❌ |
58+
| `EvoPromptGA` | [Guo et al.](https://arxiv.org/abs/2309.08532) | *required* | 👍 | 💲💲 | ⚡⚡ | ✅ | ❌ |
59+
| `OPRO` | [Yang et al.](https://arxiv.org/abs/2309.03409)| *optional* | 👎 | 💲💲 | ⚡ | ❌ | ❌ |
5660

5761
## Core Components
5862

0 commit comments

Comments
 (0)