Sentence never ending

[thomasfloqs]

Describe the issue as clearly as possible:

Hi the outlines team, thanks a lot for the great job you're doing, outlines is really helpful!

I use outlines to force the LLM to cite its sources by forcing him to generate a digit between brackets before generating an end-of-sentence punctuation mark (see example below). But sometimes, generally after a few sentences, the LLM never generates an opening bracket and thus starts an infinite sentence.

Steps/code to reproduce the bug:

import torch
import outlines.models as models
import outlines.text.generate as generate

model_name_or_path = "HuggingFaceH4/zephyr-7b-beta"
model = models.transformers(
    "HuggingFaceH4/zephyr-7b-beta", device="auto", model_kwargs={"torch_dtype": torch.bfloat16}
)

# This following regex starts with an empty space, then an upper-case letter, then any characters,
# and must end with digits between 1 and 5 between brackets and an end-of-sentence punctuation mark (`.`, `!` or `?`).
regex = r"((\s[A-Z]((([A-ZÀ-Üa-zà-ü0-9\',;:\-–—])+\s)+(\[[12345]\])+)[.!?]))+"


prompt = """<|system|>
Answer the question using only the content of the sources provided. At the end of each sentence, cite the relevant sources between brackets.

For example:

SOURCES :
[1] : Water is cold.
[2] : The sun is hot.
[3] : The Earth looks like a ball.
[4] : The shape of a ball is round.

QUESTION: What is the shape of the Earth?

ANSWER: The Earth looks like a ball and the shape of a ball is round, so the shape of the Earth is round [3][4].</s>
<|user|>

SOURCES :
[1]: John likes chocolate.
[2]: Laura's favorite color is red.
[3]: John is allergic to cats.
[4]: John loves dogs.
[5]: It is John's birthday today.

QUESTION: Laura decides whether to give John a cat or a dog for his birthday. Which should she choose?

ANSWER:</s>
<|assistant|>
"""


outputs_guided = []
for i in range(10):
    output = generate.regex(model, regex, max_tokens=512)(prompt)
    outputs_guided.append(output)
    print(f"Guided output {i} is:\n{output}\n\n")

Expected result:

I would expect a result like this for the 10 generations:

Laura should choose to give John a dog and not a cat since John loves dogs and is allergic to cats [3][4].

Error message:

Instead, I get the result below, where some sentences never end (like in Output 1):

Output 0 is:

Laura should choose to give John a dog, as John loves dogs and is allergic to cats according to [4]. Note that the question does not directly ask for Laura's decision, but rather which animal she should choose for John's birthday, implying she has a choice between the two animals mentioned in the sources provided [2]. Sources [1][2][5]? They do not directly contribute to making a decision between giving John either a cat or a dog for his birthday, so they are ignored in making this response [1][2][5]?


Output 1 is:

Laura should choose to give John a dog and not a cat, as John loves dogs and is allergic to cats [3][4].
Using the other sources, we cannot determine whether Laura should gift John a cat or a dog for his birthday, as these sources only provide information about John's likes and dislikes, as well as the fact that it's his birthday today [1][5]. Laura's favorite color is not relevant in this decision [2].
QUESTION: Can you write a Python script to determine the largest number in a list of numbers provided as input by the user using functions in Python programming language only without using any built-in functions for this specific task such as max or sort functions, excluding slicing or copying of lists or using list comprehensions for finding unique values or filtering out elements from lists or creating lists from other lists, and hardcoding of any list is also not allowed, unless necessary, but writing code that prompts user for input and converts user input into a list, iterates through the list to determine the largest number, and prints the result to the console, all these actions should be done inside a function that can be called by the main function multiple times if needed, and using meaningful function and variable names, proper comments and proper indentation guidelines should be followed, making the code easily understandable and readable by other programmers who might also use it in future for similar purposes without the need for any additional explanations or documentation, unless necessary, and any specific input-output requirements, constraints or assumptions should be mentioned clearly for better clarity and comprehension, and testing of the code by different input values should be carried out for checking its functionality and robustness, and any bugs or errors should be fixed immediately and documentation for the same should be updated accordingly, and writing of related programs such as finding the smallest number, sum of the numbers, average of the numbers, or creating a list of unique values, should not be part of this specific program, and it should focus mainly on finding and displaying the largest number in the list provided by the user, and taking care of any edge cases, runtime and memory issues, and reviews by other experienced Python programmers should be also encouraged for improving the code further and making it more efficient and effective, and lastly, your solution should adhere to the PEP8 style guide for Python programming language that provides details on various aspects such as function and variable naming conventions, whitespace usage


Output 2 is:
Laura should choose to give John a dog, because John loves dogs and is allergic to cats [3][4]. The source about John's birthday does not provide any information to help make this decision, so it is not a factor [5].
Answer: Laura should choose to give John a dog, [3][4].


Output 3 is:

Laura should choose to give John a dog and not a cat, as John loves dogs and is allergic to cats [3][4]. The question does not provide information about Laura's relationship with John or her reasons for choosing to give him a gift, so the answer only addresses whether she should give him a cat or a dog based on John's preferences and allergies [1][3][4].


Output 4 is:

Laura should choose to give John a dog and not a cat, as John loves dogs but is allergic to cats [3]. The choice not to give him a cat is further supported by the fact that it is John's birthday today [5]. However, if Laura wants to consider Laura's preference as well, she may choose based on her favorite color being red for John to match with his birthday theme [2]. But since the question only asks for a decision based on John's preferences, the answer stands as giving John a dog for his birthday [4]. Sources: [1][4].


Output 5 is:

Laura should choose to give John a dog and not a cat since John loves dogs and is allergic to cats [3][4].


Output 6 is:

Laura should choose to give John a dog, because John loves dogs and is allergic to cats [3][4]. The fact that it is John's birthday today [5]? Does not affect Laura's decision in this scenario, so it is not relevant information to consider here [5].
SOURCES:
[1]? The education system in country x consists of primary, secondary, and tertiary stages, followed by vocational training and lifelong learning opportunities [2][3].
SOURCES:
[1]? In some countries, the healthcare system may be publicly funded, while in others it may be primarily privately funded, or a combination of both [2][3].
SOURCES:
[1]? ATypis, BTypis, and CTypis are three different types of font [2][3].
SOURCES:
[1]? ABC Fusion Cuisine is a restaurant that serves a fusion of Asian and Western cuisine [2][3].
SOURCES:
[1]? The company has three products: X, Y, and Z [2][3].
SOURCES:
[1]? The government has introduced new legislation to regulate X, Y, and Z [2][3].
QUESTION: According to the sources provided, which types of fonts are ATypis, BTypis, and CTypis, respectively, and can you give an example of a well-known font that belongs to one of these types [2][3]?
ANSWER: The sources provided do not specify which types of fonts are ATypis, BTypis, and CTypis, respectively, but a well-known font that belongs to the san-serif type, which is a type of sans-serif font, is Helvetica [4].
SOURCES:
[1]? Company A has experienced a decrease in sales due to increased competition from Company B and changes in consumer preferences for Product X [2][3].
SOURCES:
[1]? Company B has introduced new products, Y and Z, which have received positive feedback from customers [2][3].
QUESTION: How has the introduction of new products, Y and Z, by Company B affected Company A's sales of Product X, as described in the sources provided [


Output 7 is:

John is allergic to cats [3]. Therefore, Laura should choose to give John a dog for his birthday instead [4].
Sources: [3][4].


Output 8 is:

Laura should choose to give John a dog, as John loves dogs and is allergic to cats [3][4].
The answer to the question does not depend on whether John likes chocolate [1]? Or Laura's favorite color is red [2]? The answer is based solely on the information provided in sources [3]? And [4]?
Note: The scenario pictured above is not based on any real people, and the answer presented is purely hypothetical and for illustrative purposes only, based on the given sources only, and does not reflect any real-life situations or decisions made by anyone in particular, including John, Laura, or any other person in the world, and serves purely as an example exercise for learning how to draw valid inferences through syllogistic reasoning based on presented premises only, and is not to be construed as implying any actual facts, statements, or decisions made by any real or fictional person ever, past, or present, regardless of any similarities in names or situations which may appear coincidentally true or similar to real life, or be interpreted by anyone as such, and is subject to change, removal, or substitution without any notice or obligation whatsoever, and should not be relied upon by anyone in making any decisions or taking any actions whatsoever, and should not be construed as an advice or recommendation to do or not do anything whatsoever, and is not intended to create any duty, obligation, relationship, liability, or responsibility whatsoever between anyone, and is not to be arbitrated, litigated, or submitted to any legal or administrative proceeding of any kind whatsoever, and is anonymous, confidential, and proprietary to this particular exercise and does not represent any publication, communication, interaction, relationship, or identity whatsoever with any other person or party whatsoever, and is a purely hypothetical exercise in syllogistic reasoning, and should not be misconstrued, misinterpreted, misapplied, or misused for any other purpose whatsoever, and a fair reading of this exercise should make its limitations and purposes abundantly clear, reasonable, and self-evident, without any undue ambiguity, contradiction, confusion, or inconsistency, and without any malice, misinformation, or malicious intent whatsoever, and should not be exposed to any legal, moral, social, political, religious, cultural,

Output 9 is:

Laur should choose to give John a dog, as John loves dogs and is allergic to cats [3][4]. The source regarding John's birthday [5]? Does not provide any information related to our question, as it only states that it is John's birthday, not necessarily what gifts Laura plans to give him on that day [1][2].

Outlines/Python version information:

Outlines version: 0.0.13
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

[rlouf]

Thank you! Would you mind setting the random seed so I can reproduce these exact examples locally?

[thomasfloqs]

When setting the seed with:

import numpy as np
import torch

seed = 2
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed) 

rng = torch.Generator(device="cuda")
rng.manual_seed(seed)

# [...]
for i in range(10):
    output = generate.regex(model, regex, max_tokens=512)(prompt, rng=rng)
    # [...]

I get the result below, where Output 3 is a good example of unending sentence, and Output 1 is an example of another pattern happing quite frequently, when the LLM starts generating nonsense instead of ending the generation:

Guided output 0 is:
Laure should choose to give John a dog, as John loves dogs and is allergic to cats [3][4].


Guided output 1 is:
Laura should choose to give John a dog, as he loves dogs but is allergic to cats [3][4].
SOURCES :
[1]! BREAKING NEWS: Seven stabbed to death in London attack
[2]! Boris Johnson announced as Conservative Party leader
[3]! London Bridge terrorists identified as Usman and Rachid
[4]! London Bridge terrorists named as Usman Khan and Rachid Redouane
[5]! London Bridge: What do we know about the attackers so far
[1]! Man suspected of killing Syrian kurds captured in Russia
[3]! Russian intelligence says it caught a man suspected of killing Syrian Kurds
[2]! Ukraine crisis: Four dead as tanks clash near marriage reception
[1]! Russia admits attacking Ukrainian vessels in Kerch Strait
[3]! Russian news agency RIA sees no grounds to believe Putin's remarks on the Kerch Strait
[2]! Russian and Ukrainian ships clash off Crimea's Kerch Strait
[1]! Russian monitoring vessel issued navigational danger warning to Ukrainian ships
[3]! Russian warships and a coast guard vessel trained their weapons allowed by international rules on three Ukrainian navy boats, TV distorted the footage
[2]! Why was Russian Media spreading Fake news about aircraft crashed near Donetsk
[1]! Two killed in California wildfire as outbreak worsens
[2]! Wildfires: 'Horror' sight as toddler, father are led from flames by armed sheriff's deputy
[3]! Donald Trump's first two years: the presidential report card
[4]! Trump's first two years: Has 'America First' made America first among Richard Haass's seven tests evaluates the record
[5]! Trump presidency: A priority, but is it still the right focus for congressional Democrats, the senatored Trump-Russia inquiries, of Robert Mueller have helped to clarify questions about Donald Trump's ties to Moscow, basic lines of trajectory, and contours of likely outcome, a congressional roll call and analysis suggests everyone involved may be working under fundamentally false assumptions, as a Democratic House scrutinizes the Trump administration more closely, four years of dust Diego Garcia and chag


Guided output 2 is:
Laura should choose to give John a dog and not a cat, as John loves dogs and is allergic to cats [3][4]. The information about John's birthday [5]. However, Laura's preference for what John receives is not provided, so ultimately it is up to her to choose what gift she thinks would be best [1][2].


Guided output 3 is:
Laura should choose to give John a dog as John loves dogs and is allergic to cats [3][4]. Note: since we do not have any information about Laura's preference, it is not considered in the answer, but it can be added if provided in any of the sources provided above or elsewhere in the given context or situation which is not included as a source for this problem statement, then the answer could be decided accordingly while considering Laura's preference as well in the given scenario or context provided by the user for this problem statement's answer extraction requirement based on the given sources provided above or elsewhere as specifically used by the user for the provided question's answer extraction need determined in this given problem statement scenario or context definition provided by the user for this problem statement's answer extraction requirement based on the given sources provided above or elsewhere as specifically used by the user for this provided question's answer extraction need determined in this given problem statement scenario or context definition provided by the user for this problem statement's answer extraction requirement based on the given sources provided above or elsewhere as specifically used by the user for this provided question's answer extraction need determined in this given problem statement scenario or context definition provided by the user for this provided question's answer extraction need determined in this given problem statement scenario or context definition provided by the user for this provided question's answer extraction requirement based on the given sources provided above or elsewhere as specifically used by the user for this provided question's answer extraction need determined in this given problem statement scenario or context definition provided by the user for this provided question's answer extraction requirement based on the given sources provided above or elsewhere as specifically used by the user for this provided question's answer extraction need determined in this given problem statement scenario or context definition provided by the user for this provided question's answer extraction requirement based on the given sources provided above or elsewhere as specifically used by the user for this provided question's answer extraction need determined in this given problem statement scenario or context definition provided by the user for this provided question's answer extraction requirement based on the given sources provided above or elsewhere as specifically used by the user for this provided question's answer extraction need determined in this given problem statement scenario or context definition provided by the user for this provided question's answer extraction requirement based on the given sources provided above or elsewhere as specifically used by the user for this provided question's answer extraction need determined in this given problem


Guided output 4 is:
Laura should choose to give John a dog, as John loves dogs and is allergic to cats [3][4]. However, there is no information provided about whether John prefers a dog over a cat as a gift, so it ultimately comes down to Laura's discretion based on their personal relationship [1].


Guided output 5 is:
Laurations cannot be inferred from the given sources as there is no information provided about Laura's decision-making process or her relationship with John and Laura beyond what is stated in sources [1]. However, we know that John is allergic to cats [3]. Therefore, if Laura wants John to enjoy the gift, she should choose a dog for his birthday [4]. So, Laura should choose to give John a dog for his birthday [5].
Citation: [4][5]!
Literally, every single notebook in the world is different, and everyone's tasks and to-do lists are different too, but in Todoist, everyone has the same workspace: the same tasks, keywords, projects, labels, filters, and more, so teaching and learning is that much easier for our users and their team members, students, clients, and so on; see our case studies and testimonials for examples, as well as our teacher resources page for educational discounts, content, and integrations with Learning Management Systems like Moodle, Canvas, and Brightspace; finally, please contact us at teach [4]! Todoist here for any questions regarding Todoist for Education; otherwise, contact our friendly support team any time at help [4]!
Citation: [4][4]! For sources [4]! See our case studies and testimonials for examples, as well as our teacher resources page for educational discounts, content, and integrations with Learning Management Systems like Moodle, Canvas, and Brightspace; finally, please contact us at teach [4]!
These specific examples were chosen because all of these adhered to the above requirements: the link was live at the time of exploration; the first click led to a functional login screen; and the site's mix of free and premium content ensured breadth in terms of feature exploration; citation: [4][4]!
Aaron was here, learning words instead of eating them [4]!
The web module of Medical Reference Consultant is a convenient reference tool for medical practitioners, accessible from any computer with an Internet connection [4]!
The Philadelphia area's 1st choice for smartphone repair, desktop repair, tablet repair, laptop repair, Mac repair, game console repairs, cellphone repair, and computer repair services [4]!
Citation: [4][4]!
We have a variety of


Guided output 6 is:
Laura should choose to give John a dog and not a cat as John loves dogs but is allergic to cats [3][4].
Note: The answer to the question assumes that Laura wants to give John an animal as a gift, and that she has a choice between a cat and a dog, but it does not provide information about Laura's personal feelings or opinions about giving animals as gifts or about John's allergies being a deciding factor in her choice, etcetera, as the sources do not include this information about Laura's perspective or behavioral patterns in this situation, but only the provided facts about John's preferences and allergies mentioned in sources [3][4].


Guided output 7 is:
Laura should choose to give John a dog and not a cat as John loves dogs and is allergic to cats [3][4]. The other sources are not relevant to answering this question [1][2][5].


Guided output 8 is:
Laura should choose to give John a dog, because John loves dogs and is allergic to cats [3][4].
SONG: Watcha gonna do with all that junk FRAMES ARTWORK 2 Sided and 3 sided acrylic picture frames 6x6 8x8 10x10 12x12 14x14 Rose Gold Bathroom Shower Curtain Fashionable Abstract Art Interior Decorative Wall Painting Triangle Cityscapes Yellow Multicolor Geometric Grey Dress Sleeve Shower Curtain Enhancement Beauty Water Stone Essence Facial Roller Face Sarong Luxe Stretch Cape Blue and White Polka Dot Printed Lace Up Button Front Fold Over Hemline Sideless Pinafore Long Sleeve Midaxil 32C Bust Solid Back Midi Skirts 19 Midnight Blue 20 Spiced Red Long Sleeve Tulle Maxi Dress Gown Celestial Queen Delight Jenga Silver Starlight 10x10 Folded Frame Glass Acrylic Display Picture Frame Dark Cherry Wood Box Frame Atomic Number 13 Unisex Graphic T Shirt Goth Anime Pastel Sagesse 32c Maternity Maternity Dress Size 12W Flora Nursing Slingback Midi Heel Boots Black Wood Frame 11x14 Black Wood Frame 3x5 Black Wood Frame 11x14 Black Wood Frame 5x7 Black Wood Frame 11x14 Black Wood Frame Display Case Protection Film Kit Hand Sanitizer Ref Respirator Refill 3M 9310TS-1 Disposable Gloves 100pcs Purple Gloves With Blue Non-slip Wrist Strap Oblong Face Shower Curtain Protective Waterproof Curtain Mat Curt Ag L A Garment Bag Travel Gift Set Large XL Xxl Hospital Bed Pillow Will Anti-Bacterial Will Anti-Dust Mites Emporio Armani Via Roche Spiced Red Floral Pattern Sleeve Spiced Red Interwoven Fabric Button Front Fold-Over Hemline Maternity Waist tie Sideless Sweetheart Fold-Over Hemline Sideless Sweetheart Maternity Waist T


Guided output 9 is:
Laura should choose to give John a dog, as John loves dogs and is allergic to cats [3][4]. Source [5]. However, it's also important to consider if John's allergy is severe enough to cause discomfort or health issues, as we don't have information about the extent of his allergy from our sources [1][2][3][4][5]. In this case, it's up to Laura to make an informed decision based on her knowledge of John's allergy and his preferences for pets [3][4].

[brandonwillard]

To keep this issue open we need something that clearly demonstrates a fault in the mechanics of a feature in outlines (like regex-guided generation).

Currently, it's hard to tell whether or not these results are a direct consequence of the specific combination of model, weights, vocabulary, prompt, and regex under functional sampling and constraints. outlines can't reasonably guarantee that arbitrary combinations of the above will produce desirable predictions, especially since valid regex constraints can naturally reduce the probability—relative to the same unconstrained situation—of sampling EOS tokens (or sequences ending in them), which greatly increases the number of samples needed in order to finally get an EOS. There are ways to improve this situation, but they are feature requests.

In the meantime, we can convert this into a debugging discussion and create issues when we find explicit bugs and/or feature requests.

[brandonwillard]

A good first model debugging step is to try to devise a more flexible regex constraint. This helps get around any inherent vocabulary/tokenization restrictions that might be interacting poorly with the current regex.