Skip to content

Feature/examplar selection #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Oct 13, 2024
Merged

Feature/examplar selection #17

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4cb7a65
implemented random selector
finitearth Oct 12, 2024
4eb0fb0
added random search selector
finitearth Oct 13, 2024
99bb9ba
increased version count
finitearth Oct 13, 2024
fe4b68b
fix typos
finitearth Oct 13, 2024
5f5298f
Update promptolution/predictors/base_predictor.py
finitearth Oct 13, 2024
71e7555
Update promptolution/tasks/classification_tasks.py
finitearth Oct 13, 2024
bf31cb1
resolve comments
finitearth Oct 13, 2024
e2fed79
resolve comments
finitearth Oct 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 64 additions & 82 deletions notebooks/getting_started.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -64,9 +64,18 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 1,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"c:\\Users\\tzehl\\Documents\\programming\\promptolution\\.venv\\Lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
"source": [
"from promptolution.helpers import run_experiment\n",
"from promptolution.config import Config"
Expand All @@ -81,7 +90,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -90,25 +99,28 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"config = Config(\n",
" task_name=\"agnews\",\n",
" ds_path=\"../data_sets/cls/agnews/\",\n",
" n_steps=3,\n",
" optimizer=\"evopromptga\",\n",
" task_name=\"subj\",\n",
" ds_path=\"../data_sets/cls/subj/\",\n",
" n_steps=8,\n",
" optimizer=\"evopromptde\",\n",
" meta_llm=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
" evaluation_llm=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
" downstream_llm=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
" api_token=token,\n",
" prepend_exemplars=True,\n",
" exemplar_selector=\"random_search\",\n",
" n_exemplars=3,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -117,7 +129,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 5,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -147,104 +159,74 @@
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Classify the news story into one of the follow...</td>\n",
" <td>0.95</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>You will be required to classify a news articl...</td>\n",
" <td>0.90</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Classify the topic of the following news as \"W...</td>\n",
" <td>0.90</td>\n",
" <th>1</th>\n",
" <td>evaluate each sentence as either objective or ...</td>\n",
" <td>0.80</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Classify news articles into categories (World,...</td>\n",
" <td>0.90</td>\n",
" <th>8</th>\n",
" <td>As a linguist, analyze a statement from a movi...</td>\n",
" <td>0.80</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Classify the given news article into one of th...</td>\n",
" <td>0.85</td>\n",
" <th>3</th>\n",
" <td>identify whether the given sentence was expres...</td>\n",
" <td>0.65</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Your job is to determine whether a news articl...</td>\n",
" <td>0.85</td>\n",
" <th>5</th>\n",
" <td>Analyze the textual content of a given stateme...</td>\n",
" <td>0.65</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Categorize the provided news article according...</td>\n",
" <td>0.85</td>\n",
" <th>9</th>\n",
" <td>determine the classification of each sentence ...</td>\n",
" <td>0.60</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Categorize the news article into one of four c...</td>\n",
" <td>0.80</td>\n",
" <th>0</th>\n",
" <td>evaluate each statement as either subjective o...</td>\n",
" <td>0.50</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Your responsibility is to accurately categoriz...</td>\n",
" <td>0.80</td>\n",
" <td>Classify the sentence according to its subject...</td>\n",
" <td>0.40</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Identify the primary theme of a news article a...</td>\n",
" <td>0.80</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>In this task, you are given a news article. Yo...</td>\n",
" <td>0.80</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Accurately categorize news articles into World...</td>\n",
" <td>0.80</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Accurately categorize the provided news articl...</td>\n",
" <td>0.75</td>\n",
" <th>6</th>\n",
" <td>As a classifier, interpret phrases in movie re...</td>\n",
" <td>0.35</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Determine the theme of the news item. Choose f...</td>\n",
" <td>0.75</td>\n",
" <th>7</th>\n",
" <td>and\\n\\nshae is about to return to bed when she...</td>\n",
" <td>0.35</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Determine the primary theme of the news articl...</td>\n",
" <td>0.70</td>\n",
" <th>4</th>\n",
" <td>Analyze reviews and label them as subjective o...</td>\n",
" <td>0.30</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" prompt score\n",
"0 Classify the news story into one of the follow... 0.95\n",
"7 You will be required to classify a news articl... 0.90\n",
"11 Classify the topic of the following news as \"W... 0.90\n",
"12 Classify news articles into categories (World,... 0.90\n",
"4 Classify the given news article into one of th... 0.85\n",
"6 Your job is to determine whether a news articl... 0.85\n",
"13 Categorize the provided news article according... 0.85\n",
"1 Categorize the news article into one of four c... 0.80\n",
"2 Your responsibility is to accurately categoriz... 0.80\n",
"3 Identify the primary theme of a news article a... 0.80\n",
"8 In this task, you are given a news article. Yo... 0.80\n",
"14 Accurately categorize news articles into World... 0.80\n",
"5 Accurately categorize the provided news articl... 0.75\n",
"9 Determine the theme of the news item. Choose f... 0.75\n",
"10 Determine the primary theme of the news articl... 0.70"
" prompt score\n",
"1 evaluate each sentence as either objective or ... 0.80\n",
"8 As a linguist, analyze a statement from a movi... 0.80\n",
"3 identify whether the given sentence was expres... 0.65\n",
"5 Analyze the textual content of a given stateme... 0.65\n",
"9 determine the classification of each sentence ... 0.60\n",
"0 evaluate each statement as either subjective o... 0.50\n",
"2 Classify the sentence according to its subject... 0.40\n",
"6 As a classifier, interpret phrases in movie re... 0.35\n",
"7 and\\n\\nshae is about to return to bed when she... 0.35\n",
"4 Analyze reviews and label them as subjective o... 0.30"
]
},
"execution_count": 11,
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
Expand Down
Loading
Loading