Regression in generating text with Phi-3-mini-4k-instruct with a long prompt (gibberish in v4.42+)

[mwootten-abeyon]

System Info

• transformers version: 4.44.1
• Platform: Linux-6.5.0-1023-aws-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.24.1
• Safetensors version: 0.4.2
• Accelerate version: 0.30.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.4.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: no
• Using GPU in script?: yes
• GPU type: Tesla T4

Who can help?

@ArthurZucker @zucchini-nlp

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

input_text copied from the associated ONNX-runtime bug ticket (see below)

import torch

from transformers import GenerationConfig, AutoModelForCausalLM, AutoTokenizer
model_identifier = 'microsoft/Phi-3-mini-4k-instruct'
model = AutoModelForCausalLM.from_pretrained(model_identifier, device_map='cuda', torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_identifier)

input_text = """
<|user|>
Tell me about Paris, France.<|end|>
<|assistant|>
Paris, the capital city of France, is renowned for its rich history, iconic landmarks, and vibrant culture. Known as "The City of Light," Paris is situated in the north-central part of the country along the Seine River.

Here are some key aspects of Paris:

1. Landmarks: Paris is home to numerous famous landmarks, including the Eiffel Tower, the Louvre Museum, Notre-Dame Cathedral, and the Champs-Élysées. The Eiffel Tower, built in 1889, is an iconic symbol of Paris and attracts millions of tourists each year. The Louvre Museum, the world's largest art museum, houses thousands of works of art, including the Mona Lisa and the Venus de Milo.

2. History: Paris has a rich history dating back to the 3rd century BC, when it was founded by a Celtic tribe called the Parisii. Over the centuries, the city has been influenced by various cultures, including the Romans, the Franks, and the Normans. The French Revolution in the late 18th century marked a significant turning point in Paris's history, leading to the establishment of the modern French Republic.

3. Culture: Paris is a global center for art, fashion, gastronomy, and culture. The city is home to numerous museums, including the Centre Pompidou, Musée d'Orsay, and Musée Rodin. Paris is also known for its fashion industry, with many famous designers having their origins in the city. The city's cuisine is also highly regarded, with a focus on fresh ingredients, and a wide variety of dishes, including French classics like coq au vin, boeuf bourguignon, and crêpes.

4. Architecture: Parisian architecture is characterized by its diverse styles, ranging from Gothic and Romanesque to Art Nouveau and Art Deco. The city's famous Haussmannian buildings, designed by Baron Haussmann in the mid-19th century, are known for their uniform facades, wrought-iron balconies, and large windows.

5. Transportation: Paris has an extensive public transportation system, including the Paris Métro, RER (suburban trains), and buses. The city's iconic yellow taxis are also a popular mode of transportation.

6. Language: The official language of Paris is French, and the city's residents are known for their charm and politeness.

7. Festivals and Events: Paris hosts numerous festivals and events throughout the year, including the annual Bastille Day celebrations, the Paris Fashion Week, and the famous annual New Year's Eve fireworks on the Eiffel Tower.

8. Geography: Paris is located in the north-central part of France, with the Seine River running through the city. The city's geography is characterized by rolling hills and picturesque parks, such as the Bois de Boulogne and the Jardin des Tuileries.

9. Population: As of 2021, Paris has an estimated population of around 2.2 million residents, with the metropolitan area housing over 12 million people.

In summary, Paris is a city steeped in history, culture, and art, with a unique blend of architectural styles and a vibrant atmosphere that continues to captivate millions of visitors each year.<|end|>
<|user|>
Please give me a list of 5 architectural landmarks in Paris, France.<|end|>
<|assistant|>
1. Eiffel Tower: Designed by Gustave Eiffel and completed in 1889, the Eiffel Tower is an iconic symbol of Paris and France. Standing at 324 meters tall, it was the tallest man-made structure in the world until the completion of the Chrysler Building in New York in 1930. The Eiffel Tower is made of wrought iron and offers visitors stunning views of the city from its three levels.

2. Notre-Dame Cathedral: Located on the Île de la Cité, Notre-Dame Cathedral is a masterpiece of French Gothic architecture. Construction began in the 12th century and continued for over 200 years, with the cathedral's completion in the 14th century. The cathedral is famous for its intricate facade, stained-glass windows, and the iconic gargoyles and chimeras.

3. Louvre Museum: Originally built as a fortress in the 12th century, the Louvre Museum is now the world's largest art museum and a historic monument in Paris. The museum's most famous landmark is the iconic glass pyramid entrance, designed by architect I. M. Pei in the 1980s. The Louvre houses over 380,000 works of art, including the Mona Lisa and the Venus de Milo.

4. Sacré-Cœur Basilica: The Sacré-Cœur Basilica, also known as the Basilique du Sacré-Cœur, is a Roman Catholic church and minor basilica located at the summit of the butte Montmartre, the highest point in Paris. The basilica was designed by Paul Abadie and dedicated in 1914. Its white domes and lavender-colored travertine stone make it a distinctive landmark in the Paris skyline.

5. Arc de Triomphe: The Arc de Triomphe is a monumental structure located at the western end of the Champs-Élysées. Commissioned by Napoleon in 1806, the Arc was designed by Jean-François-Thérèse Chalgrin and completed in 1836. The monument honors those who fought and died for France during the French Revolutionary and Napoleonic Wars. The Arc features sculptural reliefs and inscriptions, and its façade is adorned with the names of 357 generals and 660 soldiers.

These five architectural landmarks showcase the diverse styles and historical periods of Paris, from Gothic to Neoclassical, and from the 19th to the 20th centuries. Each landmark has its unique features and contributes to the city's rich architectural heritage.<|end|>
<|user|>
Please give me a list of 10 famous items displayed in the Louvre Museum. Thanks!<|end|>
<|assistant|>
1. Mona Lisa: The Mona Lisa, painted by Leonardo da Vinci in the early 16th century, is arguably the most famous painting in the world. The portrait is known for its enigmatic smile and masterful use of sfumato, a technique that creates a soft, hazy effect.

2. Venus de Milo: This ancient Greek statue, believed to have been created around 130-100 BC, is a masterpiece of Hellenistic sculpture. The Venus de Milo is renowned for its graceful beauty and the mystery surrounding its missing arms.

3. Winged Victory of Samothrace: This Hellenistic sculpture, dating back to the 2nd century BC, depicts the Greek goddess Nike, the personification of victory. The sculpture is celebrated for its dynamic movement and intricate details.

4. Liberty Leading the People: This iconic painting by Eugène Delacroix, created in 1830, commemorates the July Revolution in France. The artwork depicts a woman personifying Liberty leading a group of revolutionaries over the bodies of the fallen.

5. The Wedding at Cana: A 1516 painting by Veronese, The Wedding at Cana is a large-scale work that depicts the biblical story of Jesus turning water into wine at a wedding feast. The painting is known for its vibrant colors and intricate details.

6. The Raft of the Medusa: This 1819 painting by Théodore Géricault is a powerful depiction of the aftermath of the shipwreck of the French frigate Méduse. The painting is famous for its dramatic composition and emotional intensity.

7. The Coronation of Napoleon: This 1805 painting by Jacques-Louis David portrays the coronation of Napoleon Bonaparte as Emperor of the French. The artwork is a masterpiece of neoclassical style and captures the pomp and ceremony of the event.

8. The Death of Socrates: A 1914 painting by Pablo Picasso, The Death of Socrates depicts the moment when the ancient Greek philosopher Socrates drinks the poison hemlock, as ordered by the Athenian government. The painting is a powerful expression of the existential themes associated with Socrates' death.

9. The Turkish Bath: A 1887 painting by Puvis de Chavannes, The Turkish Bath is a large-scale work that depicts a scene of relaxation and leisity in a traditional Turkish bath. The painting is known for its subdued color palette and serene atmosphere.

10. The Wedding at Cana: A 1901 painting by Giovanni Boldini, The Wedding at Cana is a lively depiction of a festive Italian wedding. The painting is characterized by Boldini's signature flamboyant style and his ability to capture the essence of the subjects with a sense of movement and energy.

These 10 famous items in the Louvre Museum showcase a diverse range of artistic styles and periods, from ancient Greek and Roman sculptures to Renaissance and 19th-century paintings. The collection at the Louvre is one of the world's largest and most important museums, housing over 800,000 works of art, and these famous items are just a small selection of the museum's vast collection.<|end|>
<|user|>
Please give me a list of 10 famous French dishes.<|end|>
<|assistant|>
"""

tokenized = tokenizer(input_text, return_tensors='pt').to('cuda')
output = model.generate(**tokenized, max_new_tokens=1000)
print(tokenizer.batch_decode(output, skip_special_tokens=True)[0])

The first few tokens are reasonable, but the response quickly turns to gibberish:

1. Coq au vin: Coq au vin is a classic French dish that consists of chicken cooked in red wine. The dish is typically braised in red wine and served with bacon, and it is a dish. The dish. The French cuisine is a dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The dish. The city. The dish. The city. The of France. Paris. Paris. The city. Paris. Paris. The city. Paris. The city. Paris Paris. Paris. The city. Paris. The city. Paris, the city. Paris, France. Paris, the city. Paris, the city. Paris, the city. Paris, Paris, the city. The city, Paris, Paris, Paris, Paris, Paris, Parisine Paris. of France. Paris. of the Seine. The Eouveen France. Parisinean. The Eiffreaux. The Eroses. The Eiffra Paris, the Eiffreon Paris, the E. The Eggre France, the Eiffre, the Eau, Paris, Paris, Paris, Paris. Paris, Paris, the Paris, the Eighth Paris, the Eaking Paris, Parises, Parisian Paris, Paris, Parisine Parisine Paris, Paris, Paris, France, France, France, Paris, Parisesparis, France, the city France, France, Paris, France, Paris, France, France, Paris, France. France. France. France. France. France, France of France. Parises. Paris, Paris, Paris, Paris, Paris of France. Paris, Parisian France of France of France. France, France, Paris, Paris, France, Paris, France, and the Eunicartic. The E98, the Frenchespars, Paris, France, France, France, France, France, France, the E Seine, France, France, Paris. The city, Paris. The city-artries theater. The Loueshe, Paris, the Eauts, Parises, the Esphere, France, France, France, France, France, France, the Frenchespars, the city-en, Paris, the French, the city. Theater,
sight, theater of France of France. France. The Seine. The elle France. The unarchon France, the cityenac France, France, Franceen, 7ives, France, France, France, France France France, France, France, France, France, France.unen,
Par-separenighten, theater, the Frenches,
-selle, France, France, France Franceen,sight, Paris, for the, Parises, Parises, Parisen, France, and for the Parisen, the Frenchsighten, the Frenchen, France, and, France, France, France, the Seine the Seine, the Frenchsagianasterns, France,
s,s,selle, France, the Frenchsons of the Frenchselle,suns,sun France,
The Orsights, France,sightsellebelle,sights, 8enights,sightsse,sunakes,s.sorun-Dreccendernses and,says,sight, theorays,s and the citys,sights,s0sights, the Frenchen,s, the
swe, the by France,s,selle, and, and,sightakes,s,s to France,s, as the ndauts, France, Franceen, France,unagautheas of the Seine,s, mostoraunora, France, Franceenonauton,s,sunautunelles,sunartheautsunununheunun-sunseanunautenimeasternsartunoraunays,sunsighten France-Relleasternsouth-art-beccanks,sunartunserbyunresunautyunicelle,s bysweora by by by8artenartun bys8artun by Franceunreighs and French,sights ands,s bys by-byunart by its,sights,sightemsqueen ands,spars by-

Expected behavior

The output should not be gibberish. For comparison, with an earlier version of transformers the output is good:

1. Coq au Vin: Coq au Vin is a classic French dish that translates to "rooster in wine." The dish consists of chicken braised with wine, lardons, mushrooms, and garlic. It is a hearty and flavorful dish that is often served with potatoes or rice.

2. Boeuf Bourguignon: Boeuf Bourguignon is a traditional French beef stew that is made with beef braised in red wine, typically Burgundy, along with mushrooms, onions, and carrots. The dish is known for its rich and savory flavor and is often served with potatoes or noodles.

3. Ratatouille: Ratatouille is a vegetable stew that originated in Nice, France. The dish is made with tomatoes, eggplant, zucchini, bell peppers, onions, and garlic, and is seasoned with herbs such as thyme, basil, and oregano. Ratatouille is a healthy and flavorful dish that is often served as a side or as a main course.

4. Quiche Lorraine: Quiche Lorraine is a savory pie that originated in the Lorraine region of France. The dish is made with a pastry crust filled with a mixture of eggs, cream, and bacon or ham. Quiche Lorraine is a versatile dish that can be served as a breakfast, lunch, or dinner.

5. Bouillabaisse: Bouillabaisse is a traditional Provençal fish stew that originated in the port city of Marseille. The dish is made with a variety of fish, shellfish, and vegetables, and is seasoned with saffron, fennel, and orange peel. Bouillabaisse is a flavorful and aromatic dish that is often served with crusty bread.

6. Cassoulet: Cassoulet is a hearty bean stew that originated in the south of France. The dish is made with white beans, meat (usually pork, duck, or sausage), and vegetables such as carrots, onions, and garlic. Cassoulet is a comforting and filling dish that is often served with crusty bread.

7. Tarte Tatin: Tarte Tatin is a classic French upside-down apple tart that is made with caramelized apples and puff pastry. The dish is named after the Tatin sisters, who are credited with its creation. Tarte Tatin is a delicious and elegant dessert that is often served with vanilla ice cream.

8. Crème Brûlée: Crème Brûlée is a creamy custard dessert that is topped with a layer of caramelized sugar. The dish is made with heavy cream, egg yolks, sugar, and vanilla, and is known for its rich and creamy texture. Crème Brûlée is a popular dessert that is often served in fine dining restaurants.

9. Croque Monsieur: Croque Monsieur is a classic French sandwich that is made with ham, cheese, and a béchamel sauce. The dish is topped with a layer of cheese and broiled until golden brown. Croque Monsieur is a delicious and satisfying meal that is often served as a lunch or dinner.

10. Crêpes: Crêpes are thin, delicate pancakes that are a staple of French cuisine. Crêpes can be filled with a variety of sweet or savory ingredients, such as Nutella, jam, cheese, ham, or eggs. Crêpes are a versatile and delicious dish that can be enjoyed for breakfast, lunch, or dinner.

---

I first noticed this when performance dropped dramatically after a routine upgrade of transformers. Using git bisect, I was able to establish that the commit introducing the issue was 730a440. In the course of investigating this, I found similar issues in other transformer libraries; seemingly the same issue exists in both onnxruntime and llama.cpp.

In my experience, the number of tokens needed in the prompt to make the results nonsense isn't exactly 2048, but a bit higher; in the prompt I was using when I first noticed this (which I unfortunately can't share), I saw a breakdown after 2265 tokens.

[ArthurZucker]

Hey! Thanks for the great reproducer, this is super bad I'll see if I can reproduce and find a fix! Thanks for reporting!

[zucchini-nlp]

I think the degradation is related to the self.config.sliding_window which is no longer used when constructing the attention mask.

Prob the modeling code should be very similar to Mistral which uses SlidingWindowCache (https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py). I am quite short on bandwidth so feel free if anyone want to work on this

[ArthurZucker]

This can also be related to this: #33129

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[zucchini-nlp]

not stale, PR will be merged soon and the bug will be fixed