Reasoning Patterns

This repository contains Python code designed to detect and categorize "Reasoning Patterns" within text generated by large language models. These patterns are indicators of the model's reasoning process, highlighting moments of self-correction, verification, and strategic planning.

The project is based on the analysis of 10,000 responses from the Qwen3-32B model.

📜 File Description

The repository consists of a single file:

• patterns.py: This file contains the core logic of the project. It includes:

  • Three extensive regular expression strings: backtrack_pattern, verification_pattern, and subgoal_set_pattern.
  • A function, catch_rethink_patterns(text), which takes a string as input and returns the counts and matched phrases for each pattern type.

🧠 The Rethinking Patterns

The patterns are classified into three distinct categories, each representing a different aspect of the reasoning process.

🔄 Backtracking

This pattern captures instances where the model recognizes an error, abandons a flawed approach, or revises its strategy. It signifies self-correction and the ability to navigate away from dead ends.

Number of Expressions: 532

✅ Verification

This pattern identifies phrases associated with checking results, confirming assumptions, and validating solutions. It reflects a process of quality control and confidence building in the generated answer.

Number of Expressions: 334

🎯 Sub-goal Setting

This pattern detects when the model breaks a complex problem down into smaller, more manageable steps. It includes phrases related to planning, outlining a strategy, and structuring the solution path.

Number of Expressions: 1036

⚙️ How It Works & Usage

The catch_rethink_patterns() function in patterns.py uses the defined regular expressions to find and count all occurrences of the rethinking patterns in a given text.

Example

Here is how you can use the function to analyze a piece of text:

from patterns import catch_rethink_patterns

sample_text = """
Let's start by finding the derivative. Wait, that's the wrong approach, I made a mistake.
Let me backtrack and try a different method. Okay, the new plan is to first solve for x,
and then plug that value back in to verify the answer. Let's check the result. Yes, it's correct.
"""

# Get the counts and the specific matches for each pattern
backtrack_count, verification_count, subgoal_count, backtrack_matches, verification_matches, subgoal_matches = catch_rethink_patterns(sample_text)

print(f"Backtracking patterns found: {backtrack_count}")
print(f"Matches: {backtrack_matches}")
print("-" * 20)
print(f"Verification patterns found: {verification_count}")
print(f"Matches: {verification_matches}")
print("-" * 20)
print(f"Sub-goal setting patterns found: {subgoal_count}")
print(f"Matches: {subgoal_matches}")