Skip to content

Commit bf9c0a6

Browse files
minor(README)
1 parent 681ecb0 commit bf9c0a6

File tree

1 file changed

+0
-101
lines changed

1 file changed

+0
-101
lines changed

README.md

Lines changed: 0 additions & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -126,107 +126,6 @@ print(todo.get_tasks()) # Output: []
126126

127127
We follow [WizardCoder](https://github.com/nlpxucan/WizardLM/blob/main/demo/wizardLM_demo.py) and provide the script to build a local demo server with [gradio](https://www.gradio.app). Refer to [/demo](demo/README.md) for more information.
128128

129-
Here are some interesting examples showing **Magicoder**'s improvements over the base model:
130-
131-
<details>
132-
<summary> <strong> Magicoder's Understanding Ability </strong> </summary>
133-
134-
We create the following two examples by making two original HumanEval problems more complicated. While both **Magicoder-*S*-DS-6.7B** and **deepseek-coder-6.7b-base**, which is **Magicoder-*S*-DS-6.7B**'s base model, can solve the original HumanEval problem, only our **Magicoder-*S*-DS-6.7B** can solve the new complicated problems.
135-
136-
### Example 1
137-
Original Problem:
138-
139-
- Write a function to, given list of integers, return list in **"strange"** order. **"Strange"** sorting, is when you start with the minimum value, then maximum of the remaining integers, then minimum and so on.
140-
141-
New Problem:
142-
143-
- Write a function to, given list of integers, return list in **"ascending"** order. **"Ascending"** sorting, is when you start with the minimum value, then maximum of the remaining integers, then minimum and so on.
144-
145-
This problem is challenging because we change the name of a new way, which is **not** the ascending order, to sort integers from "strange" to "ascending". The model should understand that the word "ascending" here has a new meaning based on both the context and the fact that it is surrounded by quotation marks. As is shown in the following responses to the new problem from two models, **Magicoder-*S*-DS-6.7B** successfully understands this complicated requirement, while **deepseek-coder-6.7b-base** is misled and sort integers in ascending order instead.
146-
147-
Response to New Problem:
148-
```python
149-
# Magicoder-S-DS-6.7B (Correct)
150-
def ascending_sort(lst):
151-
sorted_lst = []
152-
while lst:
153-
min_val = min(lst)
154-
sorted_lst.append(min_val)
155-
lst.remove(min_val)
156-
if lst:
157-
max_val = max(lst)
158-
sorted_lst.append(max_val)
159-
lst.remove(max_val)
160-
return sorted_lst
161-
162-
# deepseek-coder-6.7b-base (Wrong)
163-
def sort_ascending(lst):
164-
lst.sort()
165-
return lst
166-
```
167-
168-
### Example 2
169-
Original Problem:
170-
171-
- Write a function that takes an integer a and returns True if this ingeger is a cube of some integer number. **Note: you may assume the input is always valid**.
172-
173-
New Problem:
174-
175-
- Write a function that takes an integer a and returns True if this ingeger is a cube of some integer number. **Note: you should check whether the input is valid**.
176-
177-
This problem is challenging because we ask the model to check the inputs' validity rather than assuming the input is always valid. While **Magicoder-*S*-DS-6.7B** successfully check the validity of the input, **deepseek-coder-6.7b-base** wrongly sets `a < 0` as the criterion of invalidity and thus fails to solve the problem.
178-
179-
Response to New Problem:
180-
```python
181-
# Magicoder-S-DS-6.7B (Correct)
182-
def is_cube(a):
183-
if not isinstance(a, int):
184-
return False
185-
if a < 0:
186-
a = -a
187-
return round(a ** (1. / 3)) ** 3 == a
188-
189-
# deepseek-coder-6.7b-base (Wrong)
190-
def is_cube(a):
191-
if a < 0:
192-
return False
193-
else:
194-
for i in range(1, a):
195-
if i**3 == a:
196-
return True
197-
return False
198-
```
199-
200-
</details>
201-
202-
203-
204-
<details>
205-
<summary> <strong> Magicoder's Ability to Use External Libraries </strong> </summary>
206-
207-
We create the following example that requires models to use external libraries for the certain task. While our **Magicoder-*S*-DS-6.7B** successfully follows the instruction in the example, **deepseek-coder-6.7b-base**, which is **Magicoder-*S*-DS-6.7B**'s base model, tends to miss some requirements in the instruction.
208-
209-
Prompt:
210-
211-
- Write a **gradio application** for the following use case: Take an input image and return a 45 degree clockwise rotated image. You should also add text description under the output showing the rotation degree.
212-
213-
214-
This instruction is challenging because our **Magicoder**'s fine-tuning dataset **does not** contain the library "gradio" that is necessary for this task. Here are the gradio applications that **Magicoder-*S*-DS-6.7B** and **deepseek-coder-6.7b-base** construct respectively:
215-
216-
- **Magicoder-*S*-DS-6.7B**: **Correct!** As required in the instruction, it **adds the text description** under the output, and successfully performs the 45-degree rotation on the input image in the **clockwise** direction.
217-
218-
Interface:
219-
![Magicoder](assets/magicoder-s-ds.png)
220-
221-
222-
- **Deepseek-coder-6.7b-base**: Wrong. Obviously, it **misses the text description** under the output. Even worse, it wrongly performs the 45-degree rotation on the input image in the **counterclockwise** direction.
223-
224-
Interface:
225-
![deepseek-coder-6.7b-base](assets/ds-coder-base.png)
226-
227-
</details>
228-
229-
230129
## 📝 Citation
231130

232131
```bibtex

0 commit comments

Comments
 (0)