Skip to content

Commit b7f832e

Browse files
kongdskongds
committed
update
1 parent a5fd840 commit b7f832e

File tree

1 file changed

+204
-24
lines changed

1 file changed

+204
-24
lines changed

‎README.md

Lines changed: 204 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -3,37 +3,217 @@
33
## Overview
44
Large language models (LLMs) have recently garnered significant interest. With in-context learning, LLMs achieve impressive results in various natural language tasks. However, the application of LLMs to sentence embeddings remains an area of ongoing research. In this work, we propose an in-context learning-based method aimed at improving sentence embeddings performance. Our approach involves adapting the previous prompt-based representation method for autore- gressive models, constructing a demonstration set that enables LLMs to perform in-context learning, and scaling up the LLMs to different model sizes. Through extensive experiments, in-context learning enables LLMs to generate high-quality sentence embeddings without any fine-tuning. It helps LLMs achieve performance comparable to current contrastive learning methods. By scaling model size, we find scaling to more than tens of billion parameters harms the performance on semantic textual similarity (STS) tasks. However, the largest model outperforms other counterparts and achieves the new state-of-the-art result on transfer tasks. We also fine-tune LLMs with current contrastive learning approach, and the 2.7B OPT model, incorporating our prompt-based method, surpasses the performance of 4.8B ST5, achieving the new state-of-the-art results on STS tasks.
55

6-
## Results on STS Tasks with in-context learing (without fine-tuning)
7-
8-
| Model | STS12 | STS13 | STS14 | STS15 | STS16 | STSb | SICK-R | Avg. |
9-
|----------|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:------:|:-----:|
10-
| OPT 125M | 62.22 | 73.10 | 61.84 | 71.09 | 72.08 | 67.80 | 64.10 | 67.46 |
11-
| OPT 350M | 63.87 | 73.85 | 63.41 | 72.45 | 73.13 | 70.84 | 65.61 | 69.02 |
12-
| OPT 1.3B | 72.78 | 83.77 | 73.61 | 83.42 | 80.60 | 78.80 | 69.69 | 77.52 |
13-
| OPT 2.7B | 68.49 | 84.72 | 75.15 | 83.62 | 81.34 | 80.94 | 72.97 | 78.18 |
14-
| OPT 6.7B | 70.65 | 84.51 | 75.01 | 83.51 | 82.00 | 81.12 | 76.77 | 79.08 |
15-
| OPT 13B | 71.99 | 85.22 | 76.04 | 82.23 | 81.38 | 81.42 | 75.00 | 79.04 |
16-
| OPT 30B | 69.99 | 83.35 | 74.75 | 83.14 | 82.42 | 81.45 | 77.46 | 78.94 |
17-
| OPT 66B | 69.93 | 83.29 | 74.88 | 80.10 | 81.11 | 81.76 | 76.26 | 78.19 |
18-
6+
## Results on STS Tasks with in-context learning (without fine-tuning)
7+
8+
<table align="center">
9+
<thead>
10+
<tr>
11+
<th>Model</th>
12+
<th align="center">STS12</th>
13+
<th align="center">STS13</th>
14+
<th align="center">STS14</th>
15+
<th align="center">STS15</th>
16+
<th align="center">STS16</th>
17+
<th align="center">STSb</th>
18+
<th align="center">SICK-R</th>
19+
<th align="center">Avg.</th>
20+
</tr>
21+
</thead>
22+
<tbody>
23+
<tr>
24+
<td>OPT 125M</td>
25+
<td align="center">62.22</td>
26+
<td align="center">73.10</td>
27+
<td align="center">61.84</td>
28+
<td align="center">71.09</td>
29+
<td align="center">72.08</td>
30+
<td align="center">67.80</td>
31+
<td align="center">64.10</td>
32+
<td align="center">67.46</td>
33+
</tr>
34+
<tr>
35+
<td>OPT 350M</td>
36+
<td align="center">63.87</td>
37+
<td align="center">73.85</td>
38+
<td align="center">63.41</td>
39+
<td align="center">72.45</td>
40+
<td align="center">73.13</td>
41+
<td align="center">70.84</td>
42+
<td align="center">65.61</td>
43+
<td align="center">69.02</td>
44+
</tr>
45+
<tr>
46+
<td>OPT 1.3B</td>
47+
<td align="center">72.78</td>
48+
<td align="center">83.77</td>
49+
<td align="center">73.61</td>
50+
<td align="center">83.42</td>
51+
<td align="center">80.60</td>
52+
<td align="center">78.80</td>
53+
<td align="center">69.69</td>
54+
<td align="center">77.52</td>
55+
</tr>
56+
<tr>
57+
<td>OPT 2.7B</td>
58+
<td align="center">68.49</td>
59+
<td align="center">84.72</td>
60+
<td align="center">75.15</td>
61+
<td align="center">83.62</td>
62+
<td align="center">81.34</td>
63+
<td align="center">80.94</td>
64+
<td align="center">72.97</td>
65+
<td align="center">78.18</td>
66+
</tr>
67+
<tr>
68+
<td>OPT 6.7B</td>
69+
<td align="center">70.65</td>
70+
<td align="center">84.51</td>
71+
<td align="center">75.01</td>
72+
<td align="center">83.51</td>
73+
<td align="center">82.00</td>
74+
<td align="center">81.12</td>
75+
<td align="center">76.77</td>
76+
<td align="center">79.08</td>
77+
</tr>
78+
<tr>
79+
<td>OPT 13B</td>
80+
<td align="center">71.99</td>
81+
<td align="center">85.22</td>
82+
<td align="center">76.04</td>
83+
<td align="center">82.23</td>
84+
<td align="center">81.38</td>
85+
<td align="center">81.42</td>
86+
<td align="center">75.00</td>
87+
<td align="center">79.04</td>
88+
</tr>
89+
<tr>
90+
<td>OPT 30B</td>
91+
<td align="center">69.99</td>
92+
<td align="center">83.35</td>
93+
<td align="center">74.75</td>
94+
<td align="center">83.14</td>
95+
<td align="center">82.42</td>
96+
<td align="center">81.45</td>
97+
<td align="center">77.46</td>
98+
<td align="center">78.94</td>
99+
</tr>
100+
<tr>
101+
<td>OPT 66B</td>
102+
<td align="center">69.93</td>
103+
<td align="center">83.29</td>
104+
<td align="center">74.88</td>
105+
<td align="center">80.10</td>
106+
<td align="center">81.11</td>
107+
<td align="center">81.76</td>
108+
<td align="center">76.26</td>
109+
<td align="center">78.19</td>
110+
</tr>
111+
</tbody>
112+
</table>
113+
19114
To evaluate the above results, please run the following script,
20115
```sh
21116
bash run_icl.sh [opt-125m|opt-350m|opt-1.3b|opt-2.7b|opt-6.7b|opt-13b|opt-30b|opt-66b]
22117
```
23118

24-
## Results on STS Tasks with contrastive learing (with fine-tuning)
119+
## Results on STS Tasks with contrastive learning (with fine-tuning)
25120

26-
| Model | STS12 | STS13 | STS14 | STS15 | STS16 | STSb | SICK-R | Avg. |
27-
|-------------------------------------------------------------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:------:|:-----:|
28-
| [royokong/prompteol-opt-1.3b](https://huggingface.co/royokong/prompteol-opt-1.3b) | 79.01 | 89.26 | 84.10 | 88.30 | 84.62 | 87.71 | 80.52 | 84.79 |
29-
| [royokong/prompteol-opt-2.7b](https://huggingface.co/royokong/prompteol-opt-2.7b) | 79.49 | 89.64 | 84.80 | 89.51 | 85.91 | 88.33 | 81.64 | 85.62 |
30-
| [royokong/prompteol-opt-6.7b](https://huggingface.co/royokong/prompteol-opt-6.7b) | 80.14 | 90.02 | 84.94 | 89.78 | 85.84 | 88.75 | 81.29 | 85.82 |
31-
| [royokong/prompteol-opt-13b](https://huggingface.co/royokong/prompteol-opt-13b) | 80.20 | 90.24 | 85.34 | 89.52 | 85.90 | 88.56 | 82.06 | 85.97 |
32-
| | | | | | | | | |
33-
| [royokong/prompteol-llama-7b](https://huggingface.co/royokong/prompteol-llama-7b) | 79.16 | 90.22 | 85.40 | 88.99 | 86.25 | 88.37 | 81.51 | 85.70 |
34-
| [royokong/prompteol-llama-13b](https://huggingface.co/royokong/prompteol-llama-13b) | 78.63 | 90.03 | 85.46 | 89.48 | 86.18 | 88.45 | 82.69 | 85.85 |
121+
<table align="center">
122+
<thead>
123+
<tr>
124+
<th align="center">Model</th>
125+
<th align="center">STS12</th>
126+
<th align="center">STS13</th>
127+
<th align="center">STS14</th>
128+
<th align="center">STS15</th>
129+
<th align="center">STS16</th>
130+
<th align="center">STSb</th>
131+
<th align="center">SICK-R</th>
132+
<th align="center">Avg.</th>
133+
</tr>
134+
</thead>
135+
<tbody>
136+
<tr>
137+
<td align="center"><a href="https://huggingface.co/royokong/prompteol-opt-1.3b" rel="nofollow" style="font-size: 0.95em;">royokong/prompteol-opt-1.3b</a></td>
138+
<td align="center">79.01</td>
139+
<td align="center">89.26</td>
140+
<td align="center">84.10</td>
141+
<td align="center">88.30</td>
142+
<td align="center">84.62</td>
143+
<td align="center">87.71</td>
144+
<td align="center">80.52</td>
145+
<td align="center">84.79</td>
146+
</tr>
147+
<tr>
148+
<td align="center"><a href="https://huggingface.co/royokong/prompteol-opt-2.7b" rel="nofollow" style="font-size: 0.95em;">royokong/prompteol-opt-2.7b</a></td>
149+
<td align="center">79.49</td>
150+
<td align="center">89.64</td>
151+
<td align="center">84.80</td>
152+
<td align="center">89.51</td>
153+
<td align="center">85.91</td>
154+
<td align="center">88.33</td>
155+
<td align="center">81.64</td>
156+
<td align="center">85.62</td>
157+
</tr>
158+
<tr>
159+
<td align="center"><a href="https://huggingface.co/royokong/prompteol-opt-6.7b" rel="nofollow" style="font-size: 0.95em;">royokong/prompteol-opt-6.7b</a></td>
160+
<td align="center">80.14</td>
161+
<td align="center">90.02</td>
162+
<td align="center">84.94</td>
163+
<td align="center">89.78</td>
164+
<td align="center">85.84</td>
165+
<td align="center">88.75</td>
166+
<td align="center">81.29</td>
167+
<td align="center">85.82</td>
168+
</tr>
169+
<tr>
170+
<td align="center"><a href="https://huggingface.co/royokong/prompteol-opt-13b" rel="nofollow" style="font-size: 0.95em;">royokong/prompteol-opt-13b</a></td>
171+
<td align="center">80.20</td>
172+
<td align="center">90.24</td>
173+
<td align="center">85.34</td>
174+
<td align="center">89.52</td>
175+
<td align="center">85.90</td>
176+
<td align="center">88.56</td>
177+
<td align="center">82.06</td>
178+
<td align="center">85.97</td>
179+
</tr>
180+
<tr>
181+
<td></td>
182+
<td align="center"></td>
183+
<td align="center"></td>
184+
<td align="center"></td>
185+
<td align="center"></td>
186+
<td align="center"></td>
187+
<td align="center"></td>
188+
<td align="center"></td>
189+
<td align="center"></td>
190+
</tr>
191+
<tr>
192+
<td align="center"><a href="https://huggingface.co/royokong/prompteol-llama-7b" rel="nofollow" style="font-size: 0.95em;">royokong/prompteol-llama-7b</a></td>
193+
<td align="center">79.16</td>
194+
<td align="center">90.22</td>
195+
<td align="center">85.40</td>
196+
<td align="center">88.99</td>
197+
<td align="center">86.25</td>
198+
<td align="center">88.37</td>
199+
<td align="center">81.51</td>
200+
<td align="center">85.70</td>
201+
</tr>
202+
<tr>
203+
<td align="center"><a href="https://huggingface.co/royokong/prompteol-llama-13b" rel="nofollow" style="font-size: 0.95em;">royokong/prompteol-llama-13b</a></td>
204+
<td align="center">78.63</td>
205+
<td align="center">90.03</td>
206+
<td align="center">85.46</td>
207+
<td align="center">89.48</td>
208+
<td align="center">86.18</td>
209+
<td align="center">88.45</td>
210+
<td align="center">82.69</td>
211+
<td align="center">85.85</td>
212+
</tr>
213+
</tbody>
214+
</table>
35215

36-
To evaluate the above results, please run the following script,
216+
To evaluate the above results, please run the following script:
37217
```sh
38218
MODEL_PATH=facebook/opt-2.7b # or decapoda-research/llama-x-hf x model size 7b 13b
39219
LORA=royokong/prompteol-opt-2.7b # or royokong/prompteol-llama-x x model size 7b 13b

0 commit comments

Comments
 (0)