Skip to content

Rename intros #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 22, 2023
Merged

Rename intros #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
243 changes: 122 additions & 121 deletions docs/source/notebooks/extraction/intro.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,122 +1,123 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "7e8fc49a-e8b2-404b-a059-e9f668c460e5",
"metadata": {},
"source": [
"# Extraction Tasks\n",
"[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain-benchmarks/blob/main/docs/source/notebooks/extraction/intro.ipynb)\n",
"\n",
"These tasks refer to an LLM's ability to extract structured output from an unstructured source, such as emails, websites, or other text. Below are a list of supported datasets.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "86912590-a90a-4351-8ab4-89192cdee1e7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table>\n",
"<thead>\n",
"<tr><th>Name </th><th>Type </th><th>Dataset ID </th><th>Description </th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>Email Extraction</td><td>ExtractionTask</td><td><a href=\"https://smith.langchain.com/public/36bdfe7d-3cd1-4b36-b957-d12d95810a2b/d\" target=\"_blank\" rel=\"noopener\">36bdfe7d-3cd1-4b36-b957-d12d95810a2b</a></td><td>A dataset of 42 real emails deduped from a spam folder, with semantic HTML tags removed, as well as a script for initial extraction and formatting of other emails from an arbitrary .mbox file like the one exported by Gmail.\n",
"\n",
"Some additional cleanup of the data was done by hand after the initial pass.\n",
"\n",
"See https://github.com/jacoblee93/oss-model-extraction-evals. </td></tr>\n",
"</tbody>\n",
"</table>"
],
"text/plain": [
"Registry(tasks=[ExtractionTask(name='Email Extraction', dataset_id='https://smith.langchain.com/public/36bdfe7d-3cd1-4b36-b957-d12d95810a2b/d', description='A dataset of 42 real emails deduped from a spam folder, with semantic HTML tags removed, as well as a script for initial extraction and formatting of other emails from an arbitrary .mbox file like the one exported by Gmail.\\n\\nSome additional cleanup of the data was done by hand after the initial pass.\\n\\nSee https://github.com/jacoblee93/oss-model-extraction-evals.\\n ', schema=<class 'langchain_benchmarks.extraction.tasks.email_task.Email'>, instructions=ChatPromptTemplate(input_variables=['email'], messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], template='You are an expert researcher.')), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['email'], template='What can you tell me about the following email? Make sure to extract the question in the correct format. Here is the email:\\n ```\\n{email}\\n```'))]))])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_benchmarks import registry\n",
"\n",
"registry.filter(Type=\"ExtractionTask\")"
]
},
{
"cell_type": "markdown",
"id": "e771e544-b8f5-4359-8fd7-c89b71fbe460",
"metadata": {},
"source": [
"### Task resources\n",
"\n",
"In addition to the dataset_id, name, and description, each extraction task provides the following:\n",
"\n",
"- `schema` - a pydantic base model defining the schema (or schemas) the model should extract\n",
"\n",
"\n",
"### Dataset schema\n",
"\n",
"Each task corresponds to a LangSmith dataset with the following schema:\n",
"\n",
"Inputs:\n",
"- `input: str` - the input text\n",
"\n",
"Outputs\n",
"- `output: str` - the expected extraction result, as a json object\n"
]
},
{
"cell_type": "markdown",
"id": "d04e05f3-3f20-4fed-bb98-3eb072213bbd",
"metadata": {},
"source": [
"### Evaluation\n",
"\n",
"The extraction tasks also have an evaluation config, which defines default LangSmith evaluators to apply when benchmarking your architecture.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "9c7865bd-8251-4579-85a3-f9085d96f497",
"metadata": {},
"outputs": [],
"source": [
"from langchain.chat_models import ChatOpenAI\n",
"\n",
"from langchain_benchmarks.extraction import get_eval_config\n",
"\n",
"eval_llm = ChatOpenAI(model=\"gpt-4\", model_kwargs={\"seed\": 42})\n",
"eval_config = get_eval_config(eval_llm)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.2"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
"cells": [
{
"cell_type": "markdown",
"id": "7e8fc49a-e8b2-404b-a059-e9f668c460e5",
"metadata": {},
"source": [
"# Introduction\n",
"\n",
"[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain-benchmarks/blob/main/docs/source/notebooks/extraction/intro.ipynb)\n",
"\n",
"These tasks refer to an LLM's ability to extract structured output from an unstructured source, such as emails, websites, or other text. Below are a list of supported datasets.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "86912590-a90a-4351-8ab4-89192cdee1e7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table>\n",
"<thead>\n",
"<tr><th>Name </th><th>Type </th><th>Dataset ID </th><th>Description </th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>Email Extraction</td><td>ExtractionTask</td><td><a href=\"https://smith.langchain.com/public/36bdfe7d-3cd1-4b36-b957-d12d95810a2b/d\" target=\"_blank\" rel=\"noopener\">36bdfe7d-3cd1-4b36-b957-d12d95810a2b</a></td><td>A dataset of 42 real emails deduped from a spam folder, with semantic HTML tags removed, as well as a script for initial extraction and formatting of other emails from an arbitrary .mbox file like the one exported by Gmail.\n",
"\n",
"Some additional cleanup of the data was done by hand after the initial pass.\n",
"\n",
"See https://github.com/jacoblee93/oss-model-extraction-evals. </td></tr>\n",
"</tbody>\n",
"</table>"
],
"text/plain": [
"Registry(tasks=[ExtractionTask(name='Email Extraction', dataset_id='https://smith.langchain.com/public/36bdfe7d-3cd1-4b36-b957-d12d95810a2b/d', description='A dataset of 42 real emails deduped from a spam folder, with semantic HTML tags removed, as well as a script for initial extraction and formatting of other emails from an arbitrary .mbox file like the one exported by Gmail.\\n\\nSome additional cleanup of the data was done by hand after the initial pass.\\n\\nSee https://github.com/jacoblee93/oss-model-extraction-evals.\\n ', schema=<class 'langchain_benchmarks.extraction.tasks.email_task.Email'>, instructions=ChatPromptTemplate(input_variables=['email'], messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], template='You are an expert researcher.')), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['email'], template='What can you tell me about the following email? Make sure to extract the question in the correct format. Here is the email:\\n ```\\n{email}\\n```'))]))])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_benchmarks import registry\n",
"\n",
"registry.filter(Type=\"ExtractionTask\")"
]
},
{
"cell_type": "markdown",
"id": "e771e544-b8f5-4359-8fd7-c89b71fbe460",
"metadata": {},
"source": [
"### Task resources\n",
"\n",
"In addition to the dataset_id, name, and description, each extraction task provides the following:\n",
"\n",
"- `schema` - a pydantic base model defining the schema (or schemas) the model should extract\n",
"\n",
"\n",
"### Dataset schema\n",
"\n",
"Each task corresponds to a LangSmith dataset with the following schema:\n",
"\n",
"Inputs:\n",
"- `input: str` - the input text\n",
"\n",
"Outputs\n",
"- `output: str` - the expected extraction result, as a json object\n"
]
},
{
"cell_type": "markdown",
"id": "d04e05f3-3f20-4fed-bb98-3eb072213bbd",
"metadata": {},
"source": [
"### Evaluation\n",
"\n",
"The extraction tasks also have an evaluation config, which defines default LangSmith evaluators to apply when benchmarking your architecture.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "9c7865bd-8251-4579-85a3-f9085d96f497",
"metadata": {},
"outputs": [],
"source": [
"from langchain.chat_models import ChatOpenAI\n",
"\n",
"from langchain_benchmarks.extraction import get_eval_config\n",
"\n",
"eval_llm = ChatOpenAI(model=\"gpt-4\", model_kwargs={\"seed\": 42})\n",
"eval_config = get_eval_config(eval_llm)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading