You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We follow [WizardCoder](https://github.com/nlpxucan/WizardLM/blob/main/demo/wizardLM_demo.py) and provide the script to build a local demo server with [gradio](https://www.gradio.app). Refer to [/demo](demo/README.md) for more information.
128
128
129
-
Here are some interesting examples showing **Magicoder**'s improvements over the base model:
We create the following two examples by making two original HumanEval problems more complicated. While both **Magicoder-*S*-DS-6.7B** and **deepseek-coder-6.7b-base**, which is **Magicoder-*S*-DS-6.7B**'s base model, can solve the original HumanEval problem, only our **Magicoder-*S*-DS-6.7B** can solve the new complicated problems.
135
-
136
-
### Example 1
137
-
Original Problem:
138
-
139
-
- Write a function to, given list of integers, return list in **"strange"** order. **"Strange"** sorting, is when you start with the minimum value, then maximum of the remaining integers, then minimum and so on.
140
-
141
-
New Problem:
142
-
143
-
- Write a function to, given list of integers, return list in **"ascending"** order. **"Ascending"** sorting, is when you start with the minimum value, then maximum of the remaining integers, then minimum and so on.
144
-
145
-
This problem is challenging because we change the name of a new way, which is **not** the ascending order, to sort integers from "strange" to "ascending". The model should understand that the word "ascending" here has a new meaning based on both the context and the fact that it is surrounded by quotation marks. As is shown in the following responses to the new problem from two models, **Magicoder-*S*-DS-6.7B** successfully understands this complicated requirement, while **deepseek-coder-6.7b-base** is misled and sort integers in ascending order instead.
146
-
147
-
Response to New Problem:
148
-
```python
149
-
# Magicoder-S-DS-6.7B (Correct)
150
-
defascending_sort(lst):
151
-
sorted_lst = []
152
-
while lst:
153
-
min_val =min(lst)
154
-
sorted_lst.append(min_val)
155
-
lst.remove(min_val)
156
-
if lst:
157
-
max_val =max(lst)
158
-
sorted_lst.append(max_val)
159
-
lst.remove(max_val)
160
-
return sorted_lst
161
-
162
-
# deepseek-coder-6.7b-base (Wrong)
163
-
defsort_ascending(lst):
164
-
lst.sort()
165
-
return lst
166
-
```
167
-
168
-
### Example 2
169
-
Original Problem:
170
-
171
-
- Write a function that takes an integer a and returns True if this ingeger is a cube of some integer number. **Note: you may assume the input is always valid**.
172
-
173
-
New Problem:
174
-
175
-
- Write a function that takes an integer a and returns True if this ingeger is a cube of some integer number. **Note: you should check whether the input is valid**.
176
-
177
-
This problem is challenging because we ask the model to check the inputs' validity rather than assuming the input is always valid. While **Magicoder-*S*-DS-6.7B** successfully check the validity of the input, **deepseek-coder-6.7b-base** wrongly sets `a < 0` as the criterion of invalidity and thus fails to solve the problem.
178
-
179
-
Response to New Problem:
180
-
```python
181
-
# Magicoder-S-DS-6.7B (Correct)
182
-
defis_cube(a):
183
-
ifnotisinstance(a, int):
184
-
returnFalse
185
-
if a <0:
186
-
a =-a
187
-
returnround(a ** (1. /3)) **3== a
188
-
189
-
# deepseek-coder-6.7b-base (Wrong)
190
-
defis_cube(a):
191
-
if a <0:
192
-
returnFalse
193
-
else:
194
-
for i inrange(1, a):
195
-
if i**3== a:
196
-
returnTrue
197
-
returnFalse
198
-
```
199
-
200
-
</details>
201
-
202
-
203
-
204
-
<details>
205
-
<summary> <strong> Magicoder's Ability to Use External Libraries </strong> </summary>
206
-
207
-
We create the following example that requires models to use external libraries for the certain task. While our **Magicoder-*S*-DS-6.7B** successfully follows the instruction in the example, **deepseek-coder-6.7b-base**, which is **Magicoder-*S*-DS-6.7B**'s base model, tends to miss some requirements in the instruction.
208
-
209
-
Prompt:
210
-
211
-
- Write a **gradio application** for the following use case: Take an input image and return a 45 degree clockwise rotated image. You should also add text description under the output showing the rotation degree.
212
-
213
-
214
-
This instruction is challenging because our **Magicoder**'s fine-tuning dataset **does not** contain the library "gradio" that is necessary for this task. Here are the gradio applications that **Magicoder-*S*-DS-6.7B** and **deepseek-coder-6.7b-base** construct respectively:
215
-
216
-
-**Magicoder-*S*-DS-6.7B**: **Correct!** As required in the instruction, it **adds the text description** under the output, and successfully performs the 45-degree rotation on the input image in the **clockwise** direction.
217
-
218
-
Interface:
219
-

220
-
221
-
222
-
-**Deepseek-coder-6.7b-base**: Wrong. Obviously, it **misses the text description** under the output. Even worse, it wrongly performs the 45-degree rotation on the input image in the **counterclockwise** direction.
0 commit comments