Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Structured Prompting: Overcoming Length Limits in In-Context Learning #805

Open
1 task
ShellLM opened this issue Apr 12, 2024 · 1 comment
Open
1 task
Labels
human-verified <INST>NEVER PICK THIS LABEL</INST> in-context-learning Examples of few-shot prompts for in-context learning. llm Large Language Models llm-experiments experiments with large language models MachineLearning ML Models, Training and Inference Papers Research papers prompt-engineering Developing and optimizing prompts to efficiently use language models for various applications and re

Comments

@ShellLM
Copy link
Collaborator

ShellLM commented Apr 12, 2024

Structured Prompting: Overcoming Length Limits in In-Context Learning

Snippet

"Structured Prompting: Scaling In-Context Learning to 1,000 Examples

Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, Furu Wei Large language models have exhibited intriguing in-context learning capability, achieving promising zero- and few-shot performance without updating the parameters. However, conventional in-context learning is usually restricted by length constraints, rendering it ineffective to absorb supervision from a large number of examples. In order to go beyond few shots, we introduce structured prompting that breaks the length limit and scales in-context learning to thousands of examples. Specifically, demonstration examples are separately encoded with well-designed position embeddings, and then they are jointly attended by the test example using a rescaled attention mechanism. So we can scale the number of exemplars with linear complexity instead of quadratic complexity with respect to length. Experimental results on a diverse set of tasks show that our approach improves end-task performance and reduces evaluation variance over conventional in-context learning as the number of demonstration examples increases. Code has been released at this https URL.

Comments: 14 pages

Suggested labels

None

@ShellLM ShellLM added the Papers Research papers label Apr 12, 2024
@ShellLM
Copy link
Collaborator Author

ShellLM commented Apr 12, 2024

Related content

#706 similarity score: 0.89
#546 similarity score: 0.86
#363 similarity score: 0.84
#551 similarity score: 0.84
#536 similarity score: 0.83
#317 similarity score: 0.83

@irthomasthomas irthomasthomas added MachineLearning ML Models, Training and Inference prompt-engineering Developing and optimizing prompts to efficiently use language models for various applications and re llm Large Language Models llm-experiments experiments with large language models in-context-learning Examples of few-shot prompts for in-context learning. human-verified <INST>NEVER PICK THIS LABEL</INST> labels Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
human-verified <INST>NEVER PICK THIS LABEL</INST> in-context-learning Examples of few-shot prompts for in-context learning. llm Large Language Models llm-experiments experiments with large language models MachineLearning ML Models, Training and Inference Papers Research papers prompt-engineering Developing and optimizing prompts to efficiently use language models for various applications and re
Projects
None yet
Development

No branches or pull requests

2 participants