go-stedy — Real-time Emotional Vector Analysis for Text Streams

What problem does this solve?

Text carries emotional signals that simple positive/negative polarity misses. A news headline "Market dropped 20% in disaster collapse" and "Market showed spectacular 20% profit growth" might both be "strong sentiment" — but they tell completely different stories. Single-dimension polarity (good/bad) can't distinguish fear from anger, trust from anticipation, or detect when the emotional trajectory of a text shifts mid-stream.

go-stedy solves these problems:

Multi-dimensional emotion: decomposes text into 10 NRC EmoLex dimensions (Joy, Trust, Positive, Anticipation, Anger, Fear, Sadness, Disgust, Negative, Surprise) instead of a single polarity score.
Streaming emotional trajectory: tracks how sentiment evolves token by token, not just the aggregate. The final output is the emotional vector at the LAST token — i.e., the freshest emotional state.
Anomaly detection: flags tokens where the smoothing process has pulled the emotional vector too far from what the input word actually means. This catches contrastive negation ("not happy" — the "not" is an anomaly), ironic constructions, emotional pivots in text.
Volatility and inversion measurement: quantifies how much the emotional signal oscillates (DramaIndex), counts micro-inversions (sharp curvature in the scalar signal) and macro-inversions (valence zero-crossings within a 16-token horizon).

Go-STEDY stands for Go Single-pass Topology of Emotion Dynamics, Yikes! Single-pass here means that STEDY has a roughly linear O(n) complexity. Don't judge, I name my things myself.

Real-world use cases

Domain	What go-stedy measures
Financial news streams	Emotional trajectory of market sentiment through a news feed. High DramaIndex+macro inversions = uncertainty/volatility.
Social media monitoring	Track emotional vectors across posts. Detect anger→fear transitions (crisis signal) or Joy→Trust building (brand affinity).
Customer support transcripts	Measure journey from frustration to resolution. Anomalies at resolution point = genuine sentiment change vs polite closure.
Content moderation	Detect hate speech (Anger/Negative spikes) vs criticism (Fear/Sadness). The 10-dim profile is richer than binary toxic/nontoxic.
Literary analysis	Track emotional arcs of characters or chapters. Anomaly detection finds narrative pivots, ironic passages.
Chatbot UX	Measure user emotional trajectory within a conversation. Micro-inversions signal confusion or frustration with the bot.
Political speech / debate	Macro-inversions track stance changes. DramaIndex measures rhetorical intensity.

What it is NOT

go-stedy is not a general NLP library. It has no POS tagging, no parsing, no NER, no topic modeling, no LLM integration. It takes text, looks up each word in a fixed emotion lexicon, applies exponential smoothing, and reports the emotional trajectory. It is:

~2000× faster than an LLM-based approach (microseconds per text, not hundreds of milliseconds)
Deterministic: same text always produces the same result
Zero-alloc after warmup (no garbage collection pause)
Safe for real-time streaming at tens of thousands of texts per second per core

Performance comparison with other approaches

All measurements single-core, excluding GPU acceleration:

Approach	Example	Texts/sec (1 core)	Class	Deterministic
go-stedy en-trie ¹	this library	~46,700	🚀 nanoseconds	✓
go-stedy en-mph ¹	this library	~45,400	🚀 nanoseconds	✓
VADER (Python)	NLTK lexicon	~15,000-30,000	⚡ microseconds	✓
TextBlob (Python)	pattern-en lexicon	~5,000-10,000	⚡ microseconds	✓
spaCy TextCategorizer	textcat pipeline	~5,000-10,000	⚡ microseconds	✗
BERT / RoBERTa sentiment	transformers (batch=1)	~100-500	🐢 milliseconds	✗
LLM API (GPT-4o, 200 tok)	chat completion	~1-5	🐌 seconds	✗

go-stedy outperforms other lexicon-based tools by 3-6× and transformer models by 200-900× per core. On a Zen+ desktop CPU, the fastest backend handles ~46k texts/second (~130 words avg) — enough to process the entire daily Twitter firehose (~500M tweets) on 4 cores in under 45 minutes.

¹ All go-stedy benchmarks built with -gcflags=-l=4. See "Compiler flags."

Quick start

package main

import (
    "fmt"
    stedy "github.com/drupaldoesnotexists/go-stedy"
    enmph "github.com/drupaldoesnotexists/go-stedy/lexicons/en-mph"
)

func main() {
    e := stedy.NewDefault(enmph.NewMphLexicon())

    r := e.Ingest("The market showed beautiful profit and spectacular growth")
    fmt.Printf("Valence: %+.3f  Drama: %.3f  Anomalies: %d\n",
        r.Valence, r.DramaIndex, len(r.Anomalies))

    r = e.Ingest("But suddenly we hit total disaster and collapse")
    fmt.Printf("Valence: %+.3f  Drama: %.3f  Anomalies: %d\n",
        r.Valence, r.DramaIndex, len(r.Anomalies))
}

Valence: +0.432  Drama: 0.812  Anomalies: 2
Valence: -0.651  Drama: 2.341  Anomalies: 3

Note: you pass the SAME engine both texts. go-stedy reuses its internal buffers between calls — zero allocations after the first run. See "Engine lifecycle" section for details.

CLI tool

Build and run the interactive sentiment CLI:

go build -o stedy ./cmd/stedy-cli/

# Analyze from stdin
echo "The market showed beautiful profit and spectacular growth" | ./stedy -pretty

# Analyze from flag
./stedy -text "This is a total disaster and complete failure" -pretty

# JSON output (default)
echo "I love this beautiful world" | ./stedy

# Pick a different lexicon backend
echo "Hello world" | ./stedy -lexicon en-trie

# List available backends
./stedy -list

Subcommands

The CLI has three subcommands:

`analyze` (default)

Analyze text sentiment. This is the default mode — any unrecognized first argument is treated as analyze flags:

./stedy -text "Beautiful day" -pretty
echo "Terrible news" | ./stedy

`label` — Interactive dataset labeling via pairwise comparison

Launches a TUI for labeling texts with valence and drama scores through pairwise comparison (Elo ranking):

./stedy label -dataset unlabeled.csv -out labeled.csv -n 200

-dataset: Path to input CSV (text column required)
-out: Output path for labeled dataset (default: labeled_output.csv)
-n: Number of pairwise comparisons (default: 2× dataset size)
TUI controls: click to select, j/k to navigate, = to skip pair

`optimize` — Random search parameter optimization

Finds optimal engine parameters (alpha, thetaMicro, thetaDelta, weightMicro, weightMacro) via random search against labeled datasets:

./stedy optimize -train labeled.csv -test heldout.csv -out results.txt -n 5000

-train: Labeled training dataset (CSV with valence and drama columns)
-test: Optional held-out test set for validation
-out: Save results to file
-lang: Language code (en, ru)
-lexicon: Specific lexicon backend
-n: Number of random search iterations (default: 2000)

The optimizer measures Pearson correlation and pairwise accuracy for both valence and drama, then reports the best combined score.

Performance tip: Build with -tags thread_unsafe_mutability for ~1100× faster optimization (mutate params in-place instead of recreating the engine each iteration):

go build -tags thread_unsafe_mutability -o stedy ./cmd/stedy-cli/
./stedy optimize -train labeled.csv -n 5000

See "Thread-unsafe mutability" section for details and thread-safety warning.

See cmd/stedy-cli/ for the full source.

How it works — from 30,000 feet

Text → [Tokenization] → tokens+caps → [PASS 1: Lexicon Lookup +
Exponential Smoothing + Anomaly Detection] → smoothed field vectors +
anomaly bits → [PASS 2: Volatility, Inversions] → [Anomaly Collection]
→ Sentiment

Tokenization: splits text into runs of letters+digits. ALLCAPS words (>2 chars) get a 1.75× emotional multiplier — shouting amplifies emotion.

PASS 1: For each token, look up its 10-dimensional emotional vector from the NRC lexicon. Multiply by caps multiplier. Blend with previous token's smoothed field using exponential smoothing (controlled by α). Compare the smoothed result to the raw lexicon vector — if they differ too much (squared distance > θ_Δ²), mark the token as anomalous.

PASS 2: Walk the scalar projection of the smoothed fields. Measure volatility (sum of absolute first derivative), micro-inversions (second derivative curvature > θ_μ), and macro-inversions (valence zero-crossings within a 16-token horizon). Compute DramaIndex = volatility × (1 + micro + macro).

Result: The final field vector (emotional state after reading the last token), the scalar valence, DramaIndex, and a list of anomalous tokens with their smoothed vectors.

Thread-unsafe mutability

By default, Stedy[L] is immutable after construction — you cannot change parameters (alpha, thetaMicro, thetaDelta, weightMicro, weightMacro) without creating a new engine via New or NewDefault.

For performance-critical batch processing (e.g., parameter optimization), the thread_unsafe_mutability build tag enables in-place mutation:

//go:build thread_unsafe_mutability

import stedy "github.com/drupaldoesnotexists/go-stedy"

var e stedy.ParamMutator = stedy.NewDefault(enmph.NewMphLexicon())
e.SetAlpha(0.3)
e.SetThetaMicro(0.7)
e.SetThetaDelta(0.15)
e.SetWeightMicro(0.5)
e.SetWeightMacro(1.5)
result := e.Ingest(text)

Thread safety warning

ParamMutator methods are NOT thread-safe. Concurrent calls to SetAlpha, SetThetaMicro, SetThetaDelta, SetWeightMicro, or SetWeightMacro while Ingest is running produce undefined behaviour (data races on internal float32 fields).

Safe usage patterns:

Single goroutine: mutate between Ingest calls, never during
Per-goroutine engine: one *Stedy[L] per goroutine, mutate independently
External mutex: guard mutation+Ingest with sync.Mutex

Without the build tag, the engine is fully immutable and safe to share across goroutines (Lexicon backends are read-only after construction).

Build tag

go build -tags thread_unsafe_mutability ./cmd/stedy-cli/
go test -tags thread_unsafe_mutability -bench=BenchmarkMutability ./...

The thread_unsafe_mutability build tag enables the ParamMutator interface and the Set* methods on *Stedy[L]. Without the tag, these types and methods don't exist — the engine is immutable.

Algorithm in detail

Tokenization (`parseTokens`)

Defines a "word" as a maximal contiguous run of letters and digits. Everything else is a delimiter (whitespace, punctuation, symbols). Pure digit-only tokens are discarded.

Four pieces of information are collected per token:

The token string (passed downstream to lexicon Lookup)
A multiplier m (1.0 or 1.75). Set to 1.75 if the token has >2 letters AND every letter is uppercase. This encodes the intuition that ALLCAPS text is emotionally amplified, as in "DISASTER" vs "disaster".
The token is stored in the reuse buffer for anomaly result reporting

The implementation has four paths selected by probing the first non-ASCII byte at the very first high-byte occurrence: The implementation has four paths selected by probing the first non-ASCII byte at the very first high-byte occurrence:

ASCII fast path (parseTokensASCII): If every byte in the input text has its high bit clear (< 0x80), the entire tokenizer uses byte-level comparisons (c >= 'a' && c <= 'z') instead of unicode.IsLetter. This is ~3× faster for typical English text and avoids the Unicode classifier's table lookups.

Cyrillic UTF-8 path (parseTokensCyrillic): Activated when a 0xD0 or 0xD1 byte is followed by a valid UTF-8 continuation byte (0x80-0xBF). This path processes 2-byte Cyrillic UTF-8 sequences with direct byte-range checks — no rune decoding. It handles ASCII-in-Cyrillic-mixed text and tracks case via bit-twiddle on the second byte. Falls back to the Unicode path for non-Cyrillic non-ASCII scripts (CJK, Arabic, etc.).

KOI8-R path (parseTokensKOI8): Activated when high bytes are detected that do not form valid UTF-8 multi-byte sequences — the engine probes the first non-ASCII byte; if it's in the KOI-8 Cyrillic range (0xC0-0xFF) and lacks a valid UTF-8 continuation, the entire text is routed here. KOI-8 is a single-byte encoding where Cyrillic letters occupy 0xC0-0xFF and case folding is XOR 0x20 (same instruction as ASCII). This path is as fast as the ASCII path for Cyrillic text and is the fastest tokenizer for Russian input.

Unicode path (parseTokensUnicode): Falls back to unicode.IsLetter / unicode.IsDigit + unicode.IsUpper via range-loop over runes. Correct for any valid UTF-8 input including CJK, accented Latin, Greek, etc. Used when the first non-ASCII byte suggests non-Cyrillic multi-byte UTF-8. any valid UTF-8 input including CJK, accented Latin, Greek, etc. Used when the first non-ASCII byte suggests non-Cyrillic multi-byte UTF-8.

The first token is unrolled from the main PASS 1 loop for a small (but benchmarkable) throughput gain.

PASS 1 — Exponential smoothing field computation

For token 0:

f₀ = b₀ · m₀

where b₀ is the lexicon vector for token 0, m₀ is the caps multiplier.

For tokens i ≥ 1:

fᵢ = (bᵢ · mᵢ) + α · fᵢ₋₁

Each of the 10 dimensions is computed independently — this is a 10-FMA (floating multiply-add) per token. The formula is an exponential moving average where the raw lexicon vector at token i is the "signal" and the previous field is the "prior context." α controls the blend.

A scalar projection is computed alongside the field:

sᵢ = (Joy + Trust + Positive + Anticipation) − (Anger + Fear + Sadness + Disgust + Negative)

This collapses the 10-dim vector into a single valence score for PASS 2 (volatility and inversion detection). The full 10-dim vector is preserved in the final Sentiment.

What this means practically

α = 0: Each token stands alone. The field at token i depends only on token i. No context bleeding. Passions change instantly.
α → 1: Each token is heavily influenced by everything that came before. The field at token i is the accumulated emotional state of the entire text up to that point, weighted toward recent tokens.
Default α = 0.5: Smooth but responsive. A short emotional burst (one angry word among friendly words) shows up but doesn't overwhelm. Good general-purpose setting.

The design deliberately uses the LAST token's field as the output valence, not the average across all tokens. This means:

A text that starts positive and ends negative returns negative valence ("I was happy, but now I'm sad" → negative)
A text that starts negative and ends positive returns positive valence ("It was terrible, but it's all better now" → positive)
This matches how humans read: the most recent emotional state dominates

Anomaly detection (inline in PASS 1)

Anomaly detection lived in PASS 2 in an earlier version. It was moved into PASS 1 to eliminate a redundant lexicon lookup pass.

For each token i, after computing the smoothed field fᵢ, the squared Euclidean distance between fᵢ and the base lexicon vector bᵢ is computed:

‖fᵢ − bᵢ‖² = Σⱼ(fᵢⱼ − bᵢⱼ)²

If this distance exceeds θ_Δ², the token is flagged as anomalous.

What anomalies detect

An anomaly means: "the emotional smoothing process has pulled this word's field away from what the lexicon says this word actually means." This happens when:

Contrastive negation: "not happy" — the "happy" is preceded by "not" which has Negative=1. The smoothing pulls "happy" toward negative, but the raw lexicon says "happy" is strongly positive. That tension = anomaly.
Emotional pivots: "The product was great. Then it broke." — "broke" is strongly negative, but the prior context ("great") is positive. The smoothing partially carries the positive forward, creating a field that differs from what "broke" alone would produce.
Ironic constructions: "Great service, said nobody ever." — "great" gets pulled toward negative by what follows. The smoothed field differs from the lexicon entry for "great."
ALLCAPS contrast: A SHOUTED word next to quiet text creates a field that differs from the un-multiplied base vector.

Anomalies are stored as a bitfield ([]uint64) to minimize allocation and collection overhead. Collection uses bits.TrailingZeros64 to iterate only over set bits, skipping zero words in O(number of anomalies) rather than O(tokens).

PASS 2 — Volatility and inversions

PASS 2 operates only on the cached scalar sᵢ from PASS 1. No lexicon lookups. No field vector compute. No allocations.

Volatility

totalVolatility = Σᵢ|sᵢ₊₁ − sᵢ|

The sum of absolute first differences of the scalar valence across all adjacent token pairs. This measures how much the emotional signal oscillates. A monotonic text (gradually more positive or negative) has low volatility. A text that swings back and forth has high volatility.

Micro-inversions (curvature)

For each token i (1 ≤ i ≤ n-2), the expected midpoint between its neighbors is computed:

expectedMid = (sᵢ₋₁ + sᵢ₊₁) / 2
curvature = |sᵢ − expectedMid|

If curvature > θ_μ, token i is a micro-inversion: a local "kink" in the emotional signal where a single token deviates sharply from the trend line.

What micro-inversions measure

A single angry word in an otherwise calm sentence → micro-inversion
"I loved it but OMG it broke" — "OMG" might be a micro-inversion
Hedging words ("actually", "however") in the middle of a statement
A sarcastic "great" followed by criticism

Micro-inversions increase the DramaIndex. A text with many micro-inversions feels emotionally "jumpy" or unstable.

Macro-inversions (zero-crossings)

For each position i, look ahead up to 16 tokens (configurable by horizon constant). If sᵢ and sᵢ₊₁₆ have opposite signs (one positive, one negative), a binary search finds the exact zero-crossing token. Deduplication by position ensures each crossing is counted once.

What macro-inversions measure

A zero-crossing means the emotional valence changed sign — from positive to negative or vice versa. This is a major emotional shift, such as:

"Everything was perfect. Then the scandal broke."
"The market was bearish for months. But the new policy changed everything."
"I hate this. Wait, I actually love it."

DramaIndex

DramaIndex = totalVolatility × (1 + microInversions + macroInversions)

DramaIndex combines three signals into one number. High DramaIndex means the text is emotionally turbulent: polarity swings, sharp turns, unexpected contradictions. Low DramaIndex means the text is emotionally stable: consistent valence, few pivots or contradictions.

DramaIndex range	Characteristic
< 1.0	Mild, consistent text. News report, factual statement.
1.0 - 3.0	Moderate emotional variation. Opinion piece with some counterpoints.
3.0 - 10.0	High drama. Argumentative text, emotional storytelling.
> 10.0	Extreme turbulence. Crisis narrative, heated debate, traumatic recounting.

(These ranges depend on token count — a 300-word text will naturally have higher DramaIndex than a 10-word text. The TextSize benchmarks show the scaling.)

WeightedDramaIndex

WeightedDramaIndex = DramaIndex / n_tokens

Normalizes drama by text length. Use this instead of raw DramaIndex when comparing texts of different lengths:

WeightedDramaIndex < 0.01: Consistent tone. No emotional turbulence.
0.01 - 0.03: Moderate. Opinion piece with some counterpoints.
0.03 - 0.10: High drama per token. Concise but emotionally charged.
> 0.10: Extreme. Short but very turbulent (e.g., "I loved it then it exploded" = short text, sharp pivot → high per-token drama).

WeightedDramaIndex is always zero when text is empty (n=0 guard, though Ingest already returns empty result for empty input).

Anomaly result collection

After PASS 2, the bitfield isAnomalyBits contains one bit per token (packed into uint64 words). The collection phase iterates over non-zero words using bits.TrailingZeros64 to find each set bit in O(1) per bit. Each anomaly fills an Anomaly struct with the token's string and its smoothed field vector.

The anomaly slice is written into a reuse buffer (s.anomalyBuffer) that is valid only until the next Ingest call on the same Stedy. For persistent results, use IngestCopy().

Anomaly collection performance is proportional to the actual number of anomalies, not the total token count. At thetaDelta=max (θ_Δ=10), zero anomalies out of 1000 tokens cost ~16 iterations (two uint64 words). At thetaDelta=0.20, ~132 anomalies cost collection time proportional to that count.

Anomalies as compression mechanism

Anomalies were originally conceived as a storage compression strategy for large text corpora paired with a known lexicon.

The insight: most tokens in a text closely match their lexicon vectors after smoothing. Only the anomalous tokens — where field diverges from lexicon — carry unique emotional information that cannot be reconstructed from the dictionary alone.

Storage strategy:

Full text storage (naive):     100 GB for 1M × 100KB documents
Anomaly-only compression:       store (index, anomaly_bitfield) per document
                                reconstruct full trajectory by replaying
                                Ingest with lexicon-only lookups plus
                                anomaly overrides at flagged positions

What you gain:

Anomaly bitfield is 1 bit per token: 1000 tokens = 125 bits, vs storing the full text or full field trajectory (10 × float32 × 1000 = 40 KB).
Reconstruction is exact: The smoothed field at each token depends only on the lexicon vector and the previous field. If all non-anomalous tokens are reconstructed from the lexicon (they match) and anomalous tokens are stored explicitly (word + field), replaying Ingest produces identical results.
Selective decompression: Only the windows around anomalies need to be reconstructed for many analysis tasks (emotional pivots, contrastive negation, high-tension passages).

Pragmatic note: If raw text is already stored elsewhere (e.g., a database), there is no compression win — just store the anomaly positions and re-run Ingest. The compression value appears when you want to discard raw text and keep only a queryable emotional fingerprint.

Zero-copy output model

Ingest() returns a Sentiment whose Anomalies slice points into the engine's internal anomalyBuffer. This means:

Zero allocations per call after warmup
The returned Sentiment is VALID ONLY until the next Ingest() on the same engine
For concurrent access: one Stedy per goroutine, or use IngestCopy()

result := engine.Ingest(text)
// result.Anomalies shares memory with engine.anomalyBuffer

persistent := engine.IngestCopy(text)
// persistent.Anomalies is an independent copy

The 18→11 B/op improvement reported in benchmarks (see below) comes from this: previously anomaly results were allocated per-call and then garbage collected. Now they are recycled in place.

Parameters — what they mean for YOUR problem

alpha (default 0.5)

What it does: Controls how much emotional context bleeds from one token to the next.

Technical: Exponential smoothing factor. fᵢ = (bᵢ · mᵢ) + α · fᵢ₋₁. At α=0, fᵢ = bᵢ · mᵢ (no smoothing). At α=1, fᵢ = (bᵢ · mᵢ) + fᵢ₋₁ (full accumulation, no decay).

What it means for your data:

Text type	Recommended α	Why
Tweets / short social posts	0.2 - 0.4	Short texts don't build much context. Low α keeps each token independent. A positive tweet with one negative word → negative, as it should be.
News headlines	0.3 - 0.5	Moderate smoothing catches multi-word phrases ("not bad" = positive-ish, not "not"=negative+"bad"=negative). Default 0.5 works.
News articles / blog posts	0.5 - 0.7	Longer texts benefit from context accumulation. Earlier emotional framing influences later interpretation.
Essays / literary text	0.6 - 0.8	High α creates smooth emotional arcs. The field at token 100 still remembers what happened at token 10. Good for detecting narrative tone shifts.
Chat / dialog turns	0.3 - 0.5	Each turn may be short, but within a turn, moderate smoothing helps.
Financial filings	0.4 - 0.6	Balance. Too much smoothing buries sudden risk disclosures. Too little misses building concern across paragraphs.

Edge case — α = 0: Alpha zero means no anomalies either. Lookup: if fᵢ = bᵢ · mᵢ, then fᵢ − bᵢ = bᵢ·(mᵢ−1). For mᵢ=1 (most tokens), the anomaly vector is zero → no anomalies even at θ_Δ=0.001. This is correct: if there's no smoothing, there's no tension between raw and smoothed. Only ALLCAPS tokens (m=1.75) can be anomalous at α=0.

Edge case — α = 1: If you set α=1 and feed a very long text, the field values can grow unbounded (each token adds to the previous without decay). The field at token 1000 may be much larger than any individual lexicon value. This is intentional — it represents "accumulated emotional charge" — but the absolute values may not be comparable across texts of different lengths at α=1. For most use cases, α=0.5 is safer.

thetaMicro (default 0.8)

What it does: Sets the sensitivity of micro-inversion (curvature) detection.

Technical: A token at position i is a micro-inversion if |sᵢ − (sᵢ₋₁ + sᵢ₊₁)/2| > θ_μ. Higher θ_μ = fewer micro-inversions flagged.

What it means for your data:

Text type	Recommended θ_μ	Why
High-precision emotional analysis	0.5 - 0.7	Catch every small kink in the emotional signal. More micro-inversions → higher DramaIndex → finer granularity of emotional turbulence.
Default / general purpose	0.8	Catches moderate-to-strong curvature. A single angry word in calm text will be flagged. Tiny fluctuations from function words ("the", "and") won't be.
Noisy text / mixed sources	1.0 - 1.5	If your texts have frequent but meaningless valence fluctuations (e.g., social media with mixed content), raise θ_μ to avoid noise being counted as micro-inversions.
Essays / literary text	0.6 - 0.9	Literary texts benefit from finer curvature detection (stylistic choices create meaningful micro-kinks).

What happens at extremes: θ_μ=0 flags every token (all curvature is detected). θ_μ=10 flags nothing (no curvature is sharp enough). The default θ_μ=0.8 is calibrated for the typical valence range of the NRC lexicon (vectors have values 0 or 1, so the scalar range is roughly [-5, +4]).

thetaDelta (default 0.20)

What it does: Sets the sensitivity of anomaly detection — how much the smoothed field can differ from the raw lexicon vector before a token is flagged as "anomalous."

Technical: θ_Δ² is compared against ‖fᵢ − bᵢ‖². Higher θ_Δ = fewer anomalies.

What it means for your data:

Use case	Recommended θ_Δ	Why
Catch ALL emotional shifts	0.0 - 0.05	Every token that has any smoothing effect from neighbors is flagged. High recall, possibly noisy. Useful for debugging or training.
General anomaly detection	0.15 - 0.30	Default 0.20 catches contrastive negation, clear emotional pivots, ironic constructions. Not too many false positives.
Conservative / high-precision	0.50 - 1.00	Only very strong divergences are flagged. 70% fewer anomalies than default, but the remaining ones are high-confidence.
No anomalies needed	10 (or any large value)	Effectively disables anomaly detection. Saves ~7% runtime and keeps allocations at 0. The anomaly slice will be empty.

Benchmark evidence (thetaDelta sweep, EnMph ProductionStream, 1000 texts, ~130 words avg):

BenchmarkGoStedy_EnMph_ThetaDelta_020  22008 ns/op  132 anomalies   3 B/op  0 allocs
BenchmarkGoStedy_EnMph_ThetaDelta_050  21427 ns/op   99 anomalies  19 B/op  0 allocs
BenchmarkGoStedy_EnMph_ThetaDelta_100  21110 ns/op   39 anomalies  10 B/op  0 allocs
BenchmarkGoStedy_EnMph_ThetaDelta_Max  19676 ns/op    0 anomalies   7 B/op  0 allocs

Notice that thetaDelta 0.20→1.00 reduces anomalies 70% but throughput barely changes (21385→21110 ns/op). The reason: anomaly detection compute (the squared distance check) happens regardless — it's inline in PASS 1. Only the collection phase at the end is proportional to anomaly count. At 132 anomalies out of ~130000 total tokens across 1000 texts, collection is a tiny fraction of total time. The thetaDelta_Max case (0 anomalies) skips most squared- distance checks during PASS 1 too (because the comparison is always false for the small-squared distance), explaining the more noticeable speedup.

Parameter presets by use case

// Social media monitoring — responsive, no context bleed, catch all shifts
stedy.New(lex, 0.3, 0.7, 0.10)

// Financial news — moderate context, high-precision anomalies
stedy.New(lex, 0.5, 0.8, 0.40)

// Essay/literary analysis — strong smoothing, fine curvature
stedy.New(lex, 0.7, 0.6, 0.15)

// Chat — responsive, no micro-noise
stedy.New(lex, 0.3, 1.0, 0.20)

// Maximum throughput (no anomalies) — for bulk classification
stedy.New(lex, 0.5, 0.8, 10.0)

API reference

Types

// 10-dimensional emotional vector from NRC EmoLex
type LexiconVector struct {
    Joy, Trust, Positive, Anticipation      float32
    Anger, Fear, Sadness, Disgust, Negative float32
    Surprise                                float32
}

// Fields are inlined in struct layout for optimal packing
// (all float32s are contiguous with no padding gaps).

// Single anomalous token with its smoothed field vector
type Anomaly struct {
	OffsetStart uint32
	OffsetEnd   uint32

	Joy, Trust, Positive, Anticipation      float32
	Anger, Fear, Sadness, Disgust, Negative float32
	Surprise                                float32
}

// Result of a single Ingest or IngestCopy call
type Sentiment struct {
	DramaIndex        float32   // totalVolatility × (1 + micro + macro inversions)
	WeightedDramaIndex float32   // DramaIndex / n_tokens — length-normalized drama
	Valence           float32   // s_{n-1}: scalar projection of last token's field

	// 10 emotional dimensions — the smoothed field of the LAST token
	Joy, Trust, Positive, Anticipation      float32
	Anger, Fear, Sadness, Disgust, Negative float32
	Surprise                                float32

	Anomalies []Anomaly // tokens where smoothed field diverges from lexicon
}

Lexicon interface

type Lexicon interface {
    Lookup(word string) (vec LexiconVector, found bool)
}

Implement this interface for your own lexicon backend. Any source of 10-dimensional per-word vectors can be plugged in. The found return value lets the engine distinguish "word exists but has zero vector" from "word not found." In either case, a missing word contributes zero to all dimensions.

Engine construction

func New[L Lexicon](l L, alpha, thetaMicro, thetaDelta float32) *Stedy[L]
func NewDefault[L Lexicon](l L) *Stedy[L]  // alpha=0.5, thetaMicro=0.8, thetaDelta=0.20

NewDefault is equivalent to New(l, 0.5, 0.8, 0.20).

The type parameter L is a concrete type implementing Lexicon. With generics, the Lookup call on l is devirtualized — the compiler inlines it directly instead of going through interface dispatch. This yields ~3-4% throughput gain on the full pipeline, especially for trie backends (tight lookup loops).

New allocates all internal buffers to an initial capacity of 64 tokens. Buffers grow automatically via ensureCapacity when needed.

Ingester interface (runtime polymorphism)

When the concrete lexicon type isn't known until runtime (e.g., CLI selecting by language flag), use the Ingester interface:

type Ingester interface {
    Ingest(text string) Sentiment
    IngestCopy(text string) Sentiment
}

Every *Stedy[L] automatically satisfies Ingester. Example:

var engine stedy.Ingester
switch lang {
case "en":
    engine = stedy.NewDefault(enmph.NewMphLexicon())
case "ru":
    engine = stedy.NewDefault(rumph.NewMphLexicon())
}
result := engine.Ingest("text") // single interface dispatch, per-document

One interface call per document (not per token) — runtime overhead is negligible.

Ingest

func (s *Stedy[L]) Ingest(text string) Sentiment    // zero-copy anomalies
func (s *Stedy[L]) IngestCopy(text string) Sentiment // safe-to-persist anomalies

Ingest: The primary entry point. Tokenizes, runs PASS 1 and PASS 2, collects anomalies into the reuse buffer, returns Sentiment. Zero allocations per call after warmup (the first call may grow buffers). Anomalies slice is valid only until the next Ingest() on the same engine.

IngestCopy: Same as Ingest but deep-copies the anomaly slice. The returned Sentiment is fully independent and safe to persist across future Ingest calls. The anomaly copy is the only allocation difference between Ingest and IngestCopy.

Stedy internal types (not part of public API, visible in package only)

type tokenData struct {
    base   LexiconVector // cached lexicon lookup (PASS1 → PASS2 reuse)
    field  LexiconVector // smoothed field after PASS1
    scalar float32        // posScore − negScore
}

The tokenData struct replaced three separate slices (rawFieldBuffer, rawScalarBuffer, baseVecBuffer) to improve cache locality. Keeping base, field, and scalar in adjacent memory reduces cache misses during PASS 2's scalar-only access pattern (the struct is smaller than three heap-allocated slices).

Lexicon backends

go-stedy comes with English and Russian lexicon backends, both reading from the NRC EmoLex dataset (embedded via //go:embed at compile time). They share a parser (lexicons.ParseEmbedFile) that converts the TSV format (word, emotion, association_flag) into a map[string]LexiconVector.

Extended lexicons (`*-ext-mph`)

Extended variants (en-ext-mph, ru-ext-mph, ru-koi8-ext-mph) use morphological expansion (pymorphy2 for Russian, suffix-based for English) to cover conjugated forms, plurals, and derived words. They are gzip-compressed at rest and transparently decompressed during init via ParseEmbedFile.

Lexicon	Words (×base)	Init time	Binary size
`en-mph`	14k (1×)	~30ms	2.6 MB
`en-ext-mph`	73k (5×)	~180ms	1.9 MB.gz
`ru-mph`	11k (1×)	~97ms	3.8 MB
`ru-ext-mph`	39k (3.5×)	~147ms	1.3 MB.gz

Extended lexicons are opt-in — use -lexicon ru-ext-mph explicitly. They are not recommended for production: for best coverage, lemmatize input text and use the standard lexicons instead. The extended variants exist for quick experimentation without preprocessing.

Both backends are:

Zero-alloc on Lookup (no allocations during text processing)
Thread-safe after construction (the data structure is read-only at runtime)
Created once and shared across goroutines (one Stedy per goroutine)

en-mph: Hash table with linear probing

Location: lexicons/en-mph/

Implementation:

Murmur3 32-bit hash (adapted from github.com/cespare/mph, MIT license)
Open-addressing hash table with linear probing at 50% load factor
All keys and values stored inline in []hashEntry for cache locality
No pointer dereferencing during lookup (keys and vecs are value types)
Average ~1.5 probes per lookup (known and unknown)
Hash table size = nextPow2(n * 2) where n = number of lexicon entries

When to use: General purpose. Slightly higher per-lookup cost (~29 ns known, ~22 ns unknown) but better throughput in the full engine pipeline because the hash table fits in fewer cache lines than the trie.

Size overhead: ~2× the raw data (50% load factor). The hash table is built at NewMphLexicon() time and stored in the MphLexicon struct.

en-trie: 256-ary trie

Location: lexicons/en-trie/

Implementation:

256-ary trie (each node has 256 child pointers indexed by byte value)
Case-normalizing lookup: uppercase bytes are lowercased via b |= 0x20
Prefix-matching: if an exact match isn't found but a prefix exists in the lexicon, the prefix's vector is returned with found=true. This means "markets" matches "market" if "markets" isn't in the lexicon.
Hash-consed structure: many nodes share subtrees via pointer equality

When to use: Better for known-word lookups (~17 ns vs ~29 ns). The trie's prefix-matching behavior means that plural forms, verb conjugations, and derived words often match their base form. This can improve lexical coverage without adding every variant to the lexicon.

Size overhead: Large. Each of the ~15k entries creates up to N nodes (N = word length). Each node has 256 pointers (2048 bytes on 64-bit). Many nodes are sparse. Total memory is significantly higher than the hash table.

Performance characteristics (from micro-benchmarks, Ryzen 3 3200G, -gcflags=-l=4):

Backend	Known lookup	Unknown lookup	ProductionStream	0 allocs
en-mph	26.92 ns/op	22.41 ns/op	22008 ns/op	✓
en-trie	17.04 ns/op	10.28 ns/op	21398 ns/op	✓

Interesting: the English trie is faster than mph on per-lookup, but the full pipeline shows similar throughput. Reason: Ingest runtime is dominated by PASS 1 field computation, PASS 2 volatility, and tokenization — not just lexicon lookup. The trie's lookup advantage is diluted across total per-token work. The KOI-8 trie maintains a lookup advantage (~1.7× vs UTF-8) but the full pipeline throughput is similar — tokenization and field computation remain the dominant costs regardless of encoding.

Which backend to choose

Criterion	en-mph	en-trie	ru-mph / ru-trie	ru-koi8-*
Memory constrained	✓	✗	✓ (mph) / ✗ (trie)	✓ (mph) / ✗ (trie)
Maximum throughput	✓	—	—	✓ (faster lookup)
Lexical coverage	—	✓	— (mph) / ✓ (trie)	✓ (trie)
Russian input (UTF-8)	—	—	✓	— (needs KOI-8)
Russian input (KOI-8)	—	—	—	✓ (fastest lookup)
Predictable behavior	✓	—	✓ (mph) / — (trie)	✓ (mph) / — (trie)

For English: en-mph is the safer default. Use en-trie for prefix matching. For Russian in UTF-8: ru-mph for exact match, ru-trie for prefix matching. For maximum possible throughput: en-trie is the fastest in the full pipeline (~21 µs per text). The KOI-8 trie offers ~1.7× faster known-word lookup for Russian text but similar pipeline throughput — advantage is per-lookup, not pipeline.

Russian language backends

go-stedy includes four Russian lexicon backends covering two encodings and two data structures:

Backend	Encoding	Structure	Description
`ru-mph`	UTF-8	MPH	Russian sentiment (UTF-8), exact-match hash
`ru-trie`	UTF-8	Trie	Russian sentiment (UTF-8), prefix-matching
`ru-koi8-mph`	KOI8-R	MPH	Russian sentiment (KOI8-R), exact-match hash
`ru-koi8-trie`	KOI8-R	Trie	Russian sentiment (KOI8-R), prefix-matching
`ru-ext-mph`	UTF-8	MPH	Expanded (39k words), compressed, exact-match
`ru-koi8-ext-mph`	KOI8-R	MPH	Expanded (KOI8-R), compressed, exact-match
`en-ext-mph`	ASCII	MPH	Expanded (73k words), compressed, exact-match

All four read from the Russian NRC EmoLex dataset (~40,000 entries, embedded via //go:embed).

UTF-8 vs KOI-8 — the performance gap

Cyrillic text is 2 bytes per character in UTF-8 but only 1 byte per character in KOI-8. This halves trie traversal depth and eliminates complex UTF-8 multi-byte parsing branches. KOI-8 case folding is a single XOR 0x20 — the same instruction as ASCII.

Lexicon	Lookup Known	Lookup Unknown	ProductionStream
en-trie (ASCII)	17.04 ns/op	10.28 ns/op	21,398 ns/op
ru-trie (UTF-8)	65.98 ns/op	4.80 ns/op	27,042 ns/op
ru-koi8-trie	39.47 ns/op	5.33 ns/op	28,704 ns/op
ru-mph (UTF-8)	31.05 ns/op	21.71 ns/op	23,287 ns/op
ru-koi8-mph	27.62 ns/op	21.21 ns/op	30,119 ns/op

ru-koi8-trie is ~1.7× faster than UTF-8 Russian on known-word lookup (39 ns vs 66 ns). Production stream throughput is similar (~29 µs per text) — the pipeline bottleneck is tokenization and field computation, not per-lookup speed.

Storage in KOI-8 — practical recommendation

If your data pipeline processes Russian text, store it in KOI-8 encoding and use ru-koi8-* backends. The single-byte encoding benefits are:

Trie performance: 39 ns/op vs 66 ns/op for UTF-8 Cyrillic — the trie traverses half as many nodes per word.
Tokenization speed: byte-level parser for KOI-8 (same code path as ASCII) avoids UTF-8 detection and multi-byte decoding entirely.
Storage footprint: KOI-8 text is ~50% smaller than UTF-8 for Cyrillic content.
SIMD-ready transcoding: UTF-8 → KOI-8 conversion is a trivial byte remap (256-entry LUT or SIMD shuffle). Even databases like ClickHouse support KOI-8 natively (koi8_r collation), making it practical to store raw text in KOI-8 at the DB level and stream it directly to go-stedy without preprocessing.

The detection is automatic: parseTokens() probes the first non-ASCII byte and routes to the correct tokenizer (UTF-8 Cyrillic, KOI-8, or generic Unicode). Feed KOI-8 text to Ingest() — it works transparently.

When to use each backend

Criterion	Choose ru-* (UTF-8)	Choose ru-koi8-* (KOI-8)
Input text encoding	UTF-8	KOI-8
Maximum throughput	—	✓ (~1.7× lookup)
Storage efficiency	✗ (2 bytes/char)	✓ (1 byte/char)
Compatibility	✓ (universal)	— (KOI-8 only)
Prefix matching (trie)	✓	✓
Exact match (MPH)	✓	✓

If your text is already in UTF-8 and you cannot re-encode, use ru-* backends. If you control the storage layer, use KOI-8 and ru-koi8-*.

Adding a new language lexicon (code generation)

go-stedy provides a code generation tool lexgen that creates a complete lexicon module from an NRC EmoLex data file for any language.

Template structure (lexicons/_template/):

_template/
├── mph/
│   ├── lexicon.go.tmpl     # Lexicon type + New + Lookup
│   ├── table.go.tmpl       # HashTable (murmur3 hash, linear probing)
│   └── bench_test.go.tmpl   # Lookup + ProductionStream benchmarks
└── trie/
    ├── lexicon.go.tmpl      # Lexicon type + New + Lookup
    └── bench_test.go.tmpl   # Lookup + ProductionStream benchmarks

Generation tool (lexicons/cmd/lexgen/main.go):

# Generate Spanish MPH lexicon
go run ./lexicons/cmd/lexgen --lang es --type mph --data path/to/nrc_es.txt

# Generate Spanish Trie lexicon
go run ./lexicons/cmd/lexgen --lang es --type trie --data path/to/nrc_es.txt

This creates:

lexicons/es-mph/ (or lexicons/es-trie/)
- lexicon.go — generates EsMphLexicon type (from template)
- table.go — generates EsMphTable type with murmur3 hash + linear probing (from template)
- bench_test.go — Lookup known/unknown + ProductionStream benchmarks
- nrc_emolex_es.txt — copy of the input data file
- go.mod — proper module path (github.com/drupaldoesnotexists/go-stedy/lexicons/es-mph)

Template variables:

Variable	Example (es, mph)	Example (de, trie)
`{{.Lang}}`	`es`	`de`
`{{.Impl}}`	`mph`	`trie`
`{{.ImplName}}`	`Mph`	`Trie`
`{{.Package}}`	`es`	`de`
`{{.ImportRef}}`	`esmph`	`detrie`
`{{.DataFile}}`	`nrc_emolex_es.txt`	`nrc_emolex_de.txt`

After generation, add the new module to go.work:

go 1.24.2

use (
    .
    ./lexicons
    ./lexicons/en-mph
    ./lexicons/en-trie
    ./lexicons/es-mph        # new
)

Shared parser (lexicons/parser.go):

The ParseEmbedFile(data, negatives) function reads the NRC TSV format (word\temotion\tassociation_flag), associates each emotion with its dimension in LexiconVector, and applies the negative word overrides (no, not, never, without, but get Negative=1).

Architecture

Module structure (Go workspace)

go-stedy/                      # Module: github.com/drupaldoesnotexists/go-stedy
├── stedy.go                   # Core engine: Stedy, LexiconVector, Sentiment, Anomaly
├── stedy_internal_test.go     # 31 unit tests (internal package access)
├── stedy_bench_test.go        # 18 integration benchmarks (production stream, text sizes,
│                              #   case sweep, lifecycle, thetaDelta sweep, unicode)
├── stedy_bench_components_test.go  # 11 component benchmarks (parseTokens, reset,
│                                   #   ensureCapacity, addToken, field computation,
│                                   #   sumSquares, abs, anomaly bits, unicode cost)
│
├── lexicons/                  # Module: github.com/drupaldoesnotexists/go-stedy/lexicons
│   ├── parser.go              # ParseEmbedFile — shared NRC TSV parser
│   ├── _template/             # Code generation templates
│   │   ├── mph/               #   MPH lexicon + table + bench templates
│   │   └── trie/              #   Trie lexicon + bench templates
│   └── cmd/lexgen/main.go     # Code generator tool
│
├── lexicons/en-mph/           # Module: github.com/drupaldoesnotexists/go-stedy/lexicons/en-mph
│   ├── lexicon.go             # MphLexicon type
│   ├── table.go               # HashTable (murmur3 + linear probing)
│   ├── bench_test.go          # Lookup + ProductionStream benchmarks
│   └── nrc_emolex_en.txt      # Embedded English NRC data
│
├── lexicons/en-trie/          # Module: github.com/drupaldoesnotexists/go-stedy/lexicons/en-trie
│   ├── lexicon.go             # TrieLexicon type (256-ary trie)
│   ├── bench_test.go          # Lookup + ProductionStream benchmarks
│   └── nrc_emolex_en.txt      # Embedded English NRC data
│
├── lexicons/ru-mph/           # Module: github.com/drupaldoesnotexists/go-stedy/lexicons/ru-mph
│   ├── lexicon.go             # Russian MPH (UTF-8)
│   ├── table.go               # HashTable (murmur3 + linear probing)
│   ├── bench_test.go          # Russian benchmarks
│   └── nrc_emolex_ru.txt      # Embedded Russian NRC data
│
├── lexicons/ru-trie/          # Module: github.com/drupaldoesnotexists/go-stedy/lexicons/ru-trie
│   ├── lexicon.go             # Russian trie (UTF-8, Cyrillic-optimized folding)
│   ├── bench_test.go          # Russian benchmarks
│   ├── cyrillic_lookup_test.go# Cyrillic case-folding tests
│   └── nrc_emolex_ru.txt      # Embedded Russian NRC data
│
├── lexicons/ru-koi8-mph/      # Module: github.com/drupaldoesnotexists/go-stedy/lexicons/ru-koi8-mph
│   ├── lexicon.go             # Russian MPH (KOI8-R)
│   ├── table.go               # HashTable (murmur3 + linear probing)
│   ├── bench_test.go          # KOI-8 benchmarks
│   └── nrc_emolex_ru_koi8.txt # Embedded Russian NRC data (KOI8-R)
│
├── lexicons/ru-koi8-trie/     # Module: github.com/drupaldoesnotexists/go-stedy/lexicons/ru-koi8-trie
│   ├── lexicon.go             # Russian trie (KOI8-R, single-byte case folding)
│   ├── bench_test.go          # KOI-8 benchmarks
│   └── nrc_emolex_ru_koi8.txt # Embedded Russian NRC data (KOI8-R)
│
├── go.mod                     # Module: github.com/drupaldoesnotexists/go-stedy
├── go.work                    # Go workspace defining all 8 modules
├── LICENSE                    # MPL-2.0
├── go.work                    # Go workspace defining all 8 modules
├── README.md                  # This file
└── .gitignore                 # Binaries, IDE files, OS artifacts

Module dependency graph

go-stedy (root)
  ├── lexicons/ (shared parser)
  │   └── imported by all lexicon modules (ParseEmbedFile)
  │   └── imported by all lexicon modules (ParseEmbedFile)
  ├── lexicons/en-mph/
  │   └── imports root (LexiconVector, Lexicon interface)
  ├── lexicons/en-trie/
  │   └── imports root
  ├── lexicons/ru-mph/
  │   └── imports root
  ├── lexicons/ru-trie/
  │   └── imports root
  ├── lexicons/ru-koi8-mph/
  │   └── imports root
  └── lexicons/ru-koi8-trie/
      └── imports root
  ├── lexicons/en-trie/
  │   └── imports root
  ├── lexicons/ru-mph/
  │   └── imports root
  ├── lexicons/ru-trie/
  │   └── imports root
  ├── lexicons/ru-koi8-mph/
  │   └── imports root
  └── lexicons/ru-koi8-trie/
      └── imports root

All modules use go 1.21 (or the minimum version your project needs). The workspace root (go.work) tracks the active toolchain version.

Internal data structures

Stedy[L]
├── lexicon           L                  — concrete lookup backend (generic)
├── alpha             float32            — exponential smoothing factor
├── thetaMicro        float32            — micro-inversion curvature threshold
├── thetaDeltaSqr     float32            — anomaly distance threshold (squared)
├── tokensBuffer      []string           — token strings from tokenization
├── capsBuffer        []float32          — caps multiplier per token (1.0 or 1.75)
├── tokenStartOffsets []uint32           — token first char byte offsets
├── tokenEndOffsets   []uint32           — token last char byte offsets
├── tokens            []tokenData        — PASS 1 results (base, field, scalar)
├── inversionBuffer   []int              — deduplicated macro-inversion positions
├── isAnomalyBits     []uint64           — anomaly bitfield
└── anomalyBuffer     []Anomaly          — anomaly result reuse buffer

All buffers are slice-based and allocated once. They grow only on the first text that exceeds the previous maximum token count. After warmup with the expected maximum text size, all Ingest calls are zero-allocation.

Optimization highlights

buffer merge (b008e79): three separate slices (rawFieldBuffer, rawScalarBuffer, baseVecBuffer) → single []tokenData. Cache locality improvement: PASS 2 reads tokens[i].scalar while PASS 1 writes tokens[i].field — same struct, adjacent cache line.

ASCII fast path (parseTokens): single-byte high-bit check selects byte-level or Unicode tokenizer. ~3× faster tokenization for ASCII text.

Anomaly bitfield (isAnomalyBits): uint64 bitmask with bits.TrailingZeros64 iteration. O(anomalies) collection instead of O(tokens).

Inline anomaly detection (moved from PASS 2 to PASS 1): eliminates duplicate lexicon Lookup. Combined with baseVecBuffer (now embedded in tokenData), saves the #1 bottleneck identified in profiling (~38% of Ingest).

Flat hash table (6c2ead7): replaced github.com/cespare/mph (2-level perfect hash with key indirection) with murmur3 + linear probing at 50% load. Keys and values in contiguous slice. ~2-4% pipeline throughput gain.

Generics stenciling (Stedy[L]): The engine is generic over the Lexicon constraint. At compile time, each concrete L gets its own specialized Ingest — the Lookup call goes directly to the concrete type, not through an interface. This enables inlining across the call boundary and future PGO optimization. Measured ~4% throughput gain on trie backends (tight loop benefits most from devirtualization).

Compiler flags: -gcflags=-l=4 increases the inline budget. Measured ~9% throughput gain on ProductionStream benchmark. See "Performance tuning" for build command.

Testing

Unit tests (31 tests)

File: stedy_internal_test.go (package stedy — internal access to unexported methods like parseTokens).

Tokenizer tests (10):

TestParseTokens_Empty — empty string → 0 tokens
TestParseTokens_PunctOnly — no letters → 0 tokens
TestParseTokens_Basic — 3 words → 3 tokens
TestParseTokens_MultipleSpaces — extra whitespace → correct count
TestParseTokens_Newlines — newline-separated → correct count
TestParseTokens_Numbers — "123" (all digits) skipped
TestParseTokens_DigitInWord — "happy2" is one token
TestParseTokens_AllCaps — ALLCAPS >2 chars → multiplier=1.75
TestParseTokens_ShortCaps — ≤2 letter ALLCAPS → multiplier=1.0
TestParseTokens_MixedCase — Title Case → multiplier=1.0
TestParseTokens_Unicode — accented Latin → correct count
TestParseTokens_Cyrillic — non-Latin script → correct count

Ingest tests (9):

TestIngest_Empty — zero Sentiment
TestIngest_SinglePositiveWord — Joy > 0, Valence > 0
TestIngest_SingleNegativeWord — Sadness > 0, Valence < 0
TestIngest_CapsMultiplierApplied — 1.75× multiplier applied
TestIngest_MultipleWords — multiple positive words → Joy > 0
TestIngest_PositiveThenNegative — DramaIndex > 0 at shift
TestIngest_NegativeThenPositive — DramaIndex > 0 at shift
TestIngest_NeutralText — low DramaIndex
TestIngest_Consistency — same text → same result (uses IngestCopy)
TestIngest_UnknownWords — all unknown → zero vector
TestIngest_MixedKnownUnknown — known words still produce signal

Threshold tests (5):

TestThreshold_ThetaDeltaZero — 0 → all tokens anomalous
TestThreshold_ThetaDeltaInfinite — θ_Δ=1000 → 0 anomalies
TestThreshold_AlphaZero — α=0 → no context bleed
TestThreshold_AlphaOne — α=1 → full accumulation
TestThreshold_AlphaZeroNoAnomalies — α=0 → no anomalies (correct)

Lifecycle tests (3):

TestEngine_Recycle — large-then-small → same as fresh
TestEngine_RepeatedRecycle — 100 cycles → consistent
TestEngine_FreshVsRecycled — unknown words after large text → zero signal

Anomaly tests (3):

TestAnomaly_WordMatches — anomaly words are from input
TestAnomaly_ZeroThreshold — all tokens anomalous at θ_Δ=0
TestAnomaly_EmotionalSwing — transition words flagged

Field-level tests (2):

TestIngest_AllEmotionalDimensions — 10 dims populated
TestIngest_ValenceSign — positive/negative text → correct sign

Test lexicon (testLexicon): 11 manually crafted entries covering positive, negative, neutral, high-surprise, and mixed-emotion words. Lookup doesn't case-normalize (unlike real lexicons) — tests can use uppercase entries when needed.

Running tests

go test ./...               # all 31 tests in all modules
go test -v ./...            # verbose output

Benchmarks

Platform

AMD Ryzen 3 3200G (Radeon Vega Graphics), Linux x86_64, Go 1.24.2. Build includes -gcflags=-l=4 (aggressive inlining) unless noted.

Integration benchmarks (stedy_bench_test.go)

Production-like stream: 1000 synthetic texts, each 10-300 words, 10% ALLCAPS.

Production stream (Engine reused across all calls):

BenchmarkGoStedy_EnMph_ProductionStream-4    6169   21006 ns/op   19 B/op  0 allocs/op
BenchmarkGoStedy_EnTrie_ProductionStream-4   6116   20686 ns/op   24 B/op  0 allocs/op

Both backends within ~1.5% of each other. ~21 µs per text (~130 words), ~6200 texts per 100ms bench run. Heap allocations come only from first-call buffer growth (amortized to ~11-24 B/op across all 1000 texts).

Text size sweep:

Text size           MPH             Trie
Tiny (2w)        180.2 ns/op      188.7 ns/op
Small (8w)       774.4 ns/op      697.7 ns/op
Medium (50-60w)  5246 ns/op       4979 ns/op
Large (300-400w) 49556 ns/op      47355 ns/op

Scales linearly with token count, as expected. The Trie backend shows a slight advantage on per-token processing.

Case sensitivity (same 8-word text, different casing):

Case        MPH         Trie
AllLower   764.5 ns/op  754.6 ns/op
AllUpper   735.5 ns/op  752.0 ns/op
Mixed      712.4 ns/op  728.9 ns/op

Case has minimal impact. ALLCAPS in a short text doesn't create enough anomalies to measure. The slight variation is measurement noise.

Engine lifecycle:

Fresh engine (+New)        4271 ns/op  11192 B/op  7 allocs/op
Recycled engine            757.6 ns/op      0 B/op  0 allocs/op

~5.6× faster to recycle. The 7 allocations in a fresh engine are buffer initialization (tokensBuffer, capsBuffer, tokens, inversionBuffer, isAnomalyBits start, anomalyBuffer start, plus the strings for the 8-word input). After warmup, zero allocations.

Memory reuse (large-then-small):

BenchmarkGoStedy_EnMph_ReuseAfterLarge-4  168.8 ns/op  0 B/op  0 allocs/op

Process a 500-word text to warm buffers, then benchmark a 2-word text. 168.8 ns/op — essentially just tokenization + 2 lookups. Confirms that ensureCapacity doesn't shrink buffers, and the existing capacity is reused.

Unicode text (mixed ASCII + accented + Greek + CJK + Cyrillic):

EnMph  1348 ns/op
EnTrie 1199 ns/op

~1.8× slower than same-length ASCII text (the Unicode tokenizer path vs ASCII fast path). Still sub-2µs for mixed-script text.

ThetaDelta sweep (MPH, ProductionStream, varying θ_Δ):

θ_Δ=0.20  22787 ns/op  131.8 anomalies/op  19 B/op  0 allocs
θ_Δ=0.30  34938 ns/op  118.6 anomalies/op  19 B/op  0 allocs
θ_Δ=0.40  21927 ns/op  108.4 anomalies/op  24 B/op  0 allocs
θ_Δ=0.50  21427 ns/op   99.1 anomalies/op  19 B/op  0 allocs
θ_Δ=0.75  20980 ns/op   60.7 anomalies/op  19 B/op  0 allocs
θ_Δ=1.00  21110 ns/op   39.2 anomalies/op  10 B/op  0 allocs
θ_Δ=10.0  19676 ns/op    0.0 anomalies/op   7 B/op  0 allocs

(θ_Δ=0.30 showing 34938 ns is likely a measurement outlier — the test binary may have seen CPU frequency scaling during that run. The trend across other values is clear.)

ThetaDelta sweep (Trie):

θ_Δ=0.20  20618 ns/op  144 anomalies/op  15 B/op  0 allocs
θ_Δ=10.0  18709 ns/op    0 anomalies/op   7 B/op  0 allocs

Thread-unsafe mutability benchmarks (stedy_bench_mutability_test.go)

Built with -tags thread_unsafe_mutability. Compares mutating params in-place vs recreating the engine for each param set (1000 texts, 5 param sets cycled):

BenchmarkMutability_MutateInPlace-4     53734   22130 ns/op     2 B/op   0 allocs/op
BenchmarkMutability_RecreateEngine-4       48  24500000 ns/op  14 MB/op  141907 allocs/op

MutateInPlace is ~1100× faster and ~7 million × fewer allocations than recreating the engine. The recreate path allocates ~14 MB and 142k objects per iteration (new lexicon hash table, new engine buffers). The mutate path is zero-alloc after warmup.

Use -tags thread_unsafe_mutability for parameter optimization, grid search, or any workload that changes params between Ingest calls. See "Thread-unsafe mutability" section for thread-safety requirements.

Component benchmarks (stedy_bench_components_test.go)

Published component-level micro-benchmarks measuring individual operations in isolation. These data are captured for performance regression testing and to inform optimization priorities.

Tokenization:

BenchmarkParseTokens_Short-4          121.8 ns/op   0 B/op  0 allocs/op  (5 words)
BenchmarkParseTokens_Long-4          28037 ns/op  10 B/op  0 allocs/op  (100 clauses)
BenchmarkParseTokens_AllUpper-4       229.6 ns/op   0 B/op  0 allocs/op  (8 ALLCAPS)
BenchmarkParseTokens_Unicode-4        808.6 ns/op   0 B/op  0 allocs/op  (mixed UTF-8)

Internal operations:

BenchmarkReset-4                     2.147 ns/op   0 B/op  0 allocs/op
BenchmarkEnsureCapacity_NoGrow-4     2.182 ns/op   0 B/op
BenchmarkEnsureCapacity_Grow-4       2.332 ns/op   2 B/op
BenchmarkAddToken-4                  3.826 ns/op   0 B/op  0 allocs/op
BenchmarkAddToken_AllCaps-4          4.628 ns/op   0 B/op  0 allocs/op
BenchmarkFieldComputation_TenFields-4  114.1 ns/op  0 B/op  0 allocs/op

PASS 2 micro-benchmarks:

BenchmarkSumSquares-4                2.099 ns/op   0 B/op  0 allocs/op  (10-dim squared diff)
BenchmarkAbsFloat64Conversion-4      2.100 ns/op   0 B/op  0 allocs/op  (abs32 vs math.Abs)
BenchmarkAnomalyBitIterate-4         55.79 ns/op   0 B/op  0 allocs/op  (2× uint64 bit scan)

Unicode classifier cost:

BenchmarkUnicodeIsLetter_ASCII-4     12.87 ns/op   0 B/op  0 allocs/op  (6 ASCII runes)
BenchmarkUnicodeIsLetter_NonASCII-4  37.79 ns/op   0 B/op  0 allocs/op  (6 non-ASCII runes)
BenchmarkASCIIByteCheck-4             5.022 ns/op  0 B/op  0 allocs/op  (6 ASCII bytes)

The ByteCheck (5 ns) vs isLetter_ASCII (12.9 ns) confirms that the ASCII fast path (~2.5× faster) is justified.

Rand source:

BenchmarkRandFloat32-4               5.927 ns/op   0 B/op  0 allocs/op

(Relevant because the benchmark dataset generator uses rand.Float32() for 10% ALLCAPS selection — not part of Ingest itself.)

Lexicon module benchmarks

en-mph (lexicons/en-mph/bench_test.go, 1s):

BenchmarkMphLexicon_Lookup_Known-4          40204184   26.92 ns/op  0 B/op  0 allocs
BenchmarkMphLexicon_Lookup_Unknown-4        54874694   22.41 ns/op  0 B/op  0 allocs
BenchmarkMphLexicon_ProductionStream-4         60331   19634 ns/op  1 B/op  0 allocs

en-trie (lexicons/en-trie/bench_test.go, 1s):

BenchmarkTrieLexicon_Lookup_Known-4          66107965   17.04 ns/op  0 B/op  0 allocs
BenchmarkTrieLexicon_Lookup_Unknown-4       100000000   10.28 ns/op  0 B/op  0 allocs
BenchmarkTrieLexicon_ProductionStream-4         69054   17143 ns/op  1 B/op  0 allocs

ru-mph (lexicons/ru-mph/bench_test.go, UTF-8 Russian, 1s):

BenchmarkMphLexicon_Lookup_Known-4          38457487   31.05 ns/op  0 B/op  0 allocs
BenchmarkMphLexicon_Lookup_Unknown-4        55854384   21.71 ns/op  0 B/op  0 allocs
BenchmarkMphLexicon_ProductionStream-4         51118   23287 ns/op  3 B/op  0 allocs

ru-trie (lexicons/ru-trie/bench_test.go, UTF-8 Russian, Cyrillic-optimized, 1s):

BenchmarkTrieLexicon_Lookup_Known-4          18418015   65.98 ns/op  0 B/op  0 allocs
BenchmarkTrieLexicon_Lookup_Unknown-4       246297561    4.80 ns/op  0 B/op  0 allocs
BenchmarkTrieLexicon_ProductionStream-4         43966   27042 ns/op  3 B/op  0 allocs

ru-koi8-mph (lexicons/ru-koi8-mph/bench_test.go, KOI8-R Russian, 1s):

BenchmarkMphLexicon_Lookup_Known-4          43253229   27.62 ns/op  0 B/op  0 allocs
BenchmarkMphLexicon_Lookup_Unknown-4        56431576   21.21 ns/op  0 B/op  0 allocs
BenchmarkMphLexicon_ProductionStream-4         40084   30119 ns/op  6 B/op  0 allocs

ru-koi8-trie (lexicons/ru-koi8-trie/bench_test.go, KOI8-R Russian, 1s):

BenchmarkTrieLexicon_Lookup_Known-4          29618655   39.47 ns/op  0 B/op  0 allocs
BenchmarkTrieLexicon_Lookup_Unknown-4       224693245    5.33 ns/op  0 B/op  0 allocs
BenchmarkTrieLexicon_ProductionStream-4         42183   28704 ns/op  6 B/op  0 allocs

Known lookup: ru-koi8-trie is ~1.7× faster than UTF-8 ru-trie (39 ns vs 66 ns). The KOI-8 trie benefits from single-byte Cyrillic (half the trie depth per word) and trivial XOR-0x20 case folding. However, the full pipeline throughput is similar across all Russian backends (~27-30 µs per text) — the bottleneck is tokenization and field computation, not per-lookup speed.

Performance tuning

1. Reuse the engine

The single most impactful optimization. New() allocates all internal buffers. Creating a new Stedy per text costs ~4271 ns + 11192 bytes. Recycling the same engine costs ~757 ns + 0 bytes.

// BAD — creates new engine per text
for _, text := range texts {
    e := stedy.NewDefault(enmph.NewMphLexicon())
    result := e.Ingest(text)
}

// GOOD — reuse
e := stedy.NewDefault(enmph.NewMphLexicon())
for _, text := range texts {
    result := e.Ingest(text)
}

// BEST — warm up with your largest expected text first
e := stedy.NewDefault(enmph.NewMphLexicon())
_ = e.Ingest(largestExpectedText) // pre-grow buffers
// steady state: zero allocations

Concurrency model: One Stedy per goroutine. Lexicon backends are read-only after construction and can be shared. The per-goroutine engine cost is just NewDefault(lex).

2. Compiler flags

go test -bench=. -gcflags=-l=4 ./...
go build -gcflags=-l=4 .

-gcflags=-l=4 enables aggressive inlining. Measured ~9% throughput gain on ProductionStream. The inliner is more willing to inline hot functions (especially the small helpers like abs32, addToken, and the field computation expressions).

Note: -l=4 increases compile time and binary size slightly. The binary still dwarfs typical Go binaries — the NRC EmoLex embedded data (~140 KB) dominates.

3. ThetaDelta tuning (disabling anomalies)

If you don't need anomaly metadata for your use case:

e := stedy.New(lex, 0.5, 0.8, 10.0) // thetaDelta=10 effectively disables
result := e.Ingest(text)
// result.Anomalies is empty

Saves ~7% on throughput and keeps allocations at 0 (no anomaly collection needed). The anomaly cursor loop in PASS 1 still runs one comparison per token, but the comparison distSqr > thetaDeltaSqr is always false, and the bit is never set.

4. SIMD opportunity (not yet implemented)

The 10-FMA hot loop in PASS 1 (~9% of Ingest time) is SIMD-friendly:

currentField.Joy = (baseVec.Joy * m) + (a * prevField.Joy)
// 10 consecutive float32 operations, no loop-carried dependency per dimension

SSE2 processes 4× float32 per instruction. AVX2 processes 8× float32. A plan9 assembly dispatch (field_compute_amd64.s) could accelerate the hot loop ~3× with SSE2, for ~6% total pipeline gain. Not yet implemented — the assembly surface is small but requires plan9 expertise for Go's assembler constraints.

5. Choose the right lexicon backend

en-mph: compact (~2× raw data), predictable (exact match), ~29 ns lookup
en-trie: larger memory, prefix-matching, ~17 ns lookup

In the full pipeline, the difference is usually (!) ~1.5% of Ingest time. Choose by feature (prefix-matching vs exact-match) rather than microbenchmarks.

Text preprocessing — how input preparation affects backend choice

The two lexicon backends respond differently to text preprocessing. Choose your pipeline based on whether you lemmatize/stem or pass raw text.

Raw text (no preprocessing)

Every token is looked up as-is in the lexicon. The trie backend is preferred here because:

Prefix matching: "markets" matches "market" entry. The trie stores "market" but a lookup for "markets" falls through the 256-ary tree: m → a → r → k → e → t → s. At s there is no child, but there IS a value at t (the shortest prefix). The trie returns the t node's vector.
Case normalization built in: The trie normalizes input during lookup, so "MARKET", "Market", "market" all resolve to the same entry.
Limitation: Inflected forms distant from the base word ("went" → "go") are never matched. The trie matches prefixes only — it has no morphological awareness.

With the MPH backend on raw text, lookup fails for any inflected form not in the lexicon. If the lexicon has "market" but the text says "markets", you get a miss (zero vector contribution). For English, the built-in lexicon covers ~14000 common lemmas — most inflected forms exist, but not all.

Lemmatized text

Apply a lemmatizer (e.g., spaCy, Stanford CoreNLP, Gosentiwordnet) before Ingest:

"He ran marketing campaigns" → stem/lemmatize → "He run market campaign"

Now both backends see the exact lexicon entries:

MPH backend wins: Exact hash lookup at ~29 ns, compact storage (~16 MB total for table+keys), deterministic performance.
Trie lost its advantage: No prefix matching needed, every lookup is exact. The trie's ~17 ns lookup is faster but the difference is ~1.5% of Ingest time — irrelevant.
Recommendation: Use mph+lemmatizer for maximum throughput and smallest binary size.

Token count effect: Lemmatization and stemming reduce average word length, which directly affects tokenization performance. The tokenizer scans byte-by-byte — shorter words mean fewer iterations per token. For ASCII text, each byte takes ~3-5 ns to process. A lemmatizer that trims 2-3 characters per word saves 10-15 ns per token. Over 1000 tokens, this adds up to ~10-15 µs — significant relative to the ~20 µs total Ingest time. The effect is more pronounced for Cyrillic UTF-8 text (2 bytes/char parsed at ~6-8 ns/byte vs KOI-8's ~3-5 ns/byte). KOI-8 encoding already halves byte count per character, so the marginal gain from stemming is smaller.

Stemmed text

Similar to lemmatized: stems are shorter and more likely to match lexicon prefixes. The trie backend handles stems naturally (stem is a prefix of the full word). MPH needs exact stem-to-entry match.

KOI-8 encoded text (Russian)

For Russian text pipelines, consider storing and processing text in KOI-8 encoding rather than UTF-8. The ru-koi8-* backends are designed for this:

Single byte per Cyrillic character: UTF-8 uses 2 bytes per Cyrillic character; KOI-8 uses 1 byte. This halves trie traversal depth and reduces tokenization byte-scan time by ~50%.
Trivial case folding: KOI-8 uppercase→lowercase is a single XOR 0x20 instruction (same as ASCII). UTF-8 Cyrillic requires 3 different folding patterns depending on the letter's position in the alphabet.
Tokenization: parseTokensKOI8 uses the same byte-level loop as the ASCII fast path — no rune decoding, no continuation byte checks.
Per-lookup speed: ru-koi8-trie known-word lookup is ~1.7× faster than UTF-8 ru-trie (39 ns vs 66 ns). The full pipeline throughput is similar (~28 µs per text) — the bottleneck is tokenization and field computation.

Detection is automatic: The engine probes the first non-ASCII byte in parseTokens() and selects KOI-8 tokenization when bytes don't form valid UTF-8 Cyrillic sequences. Just call Ingest(text) — no separate API needed.

Practical recommendation: If you store Russian text in a database (e.g., ClickHouse with koi8_r collation), stream it directly to go-stedy without re-encoding. The KOI-8-optimized backends offer faster per-word lookup, though pipeline throughput is similar to UTF-8 variants. Transcoding UTF-8 → KOI-8 at read time can itself be vectorized (256-entry LUT or SIMD shuffle), making the round-trip (DB→transcode→Ingest) competitive.

ALLCAPS handling

Both backends receive normalized input from the tokenizer. The tokenizer detects ALLCAPS words (measured as ≥75% uppercase letters in the word) and the caps multiplier is applied AFTER the lexicon lookup. The lexicon lookup itself is lowercase. ALLCAPS is a runtime effect on the field, not a lexicon concern.

Stop word removal

Not recommended for emotional analysis. "Function words" (a, the, is, and) carry no emotional signal, but removing them changes token indices and shifts the smoothing window. The zero-field contribution of stop words (their lexicon lookup returns all zeros) means they act as neutral smoothing carriers — they propagate context without adding their own signal. This is valuable for PASS 2's volatility computation.

If you remove stop words pre-Ingest, expect:

Fewer total tokens → lower DramaIndex (less distance accumulated)
More abrupt emotional transitions (no neutral carriers between pivots)
Higher WeightedDramaIndex per token (skipped neutral tokens removed)

Go version requirements

go-stedy requires Go 1.24 or later for one feature:

Benchmark API — `b.Loop()`

// Go 1.24+ (used throughout go-stedy benchmarks)
func BenchmarkFoo(b *testing.B) {
    for b.Loop() {
        // ...
    }
}

b.Loop() replaces the manual b.N iteration loop. It manages the iteration count, reporting state, and cleanup automatically. This is the canonical benchmark form starting in Go 1.24.

Other Go features used (all stable since earlier versions)

Feature	Since	Where used
Embedded files (`//go:embed`)	Go 1.16	Lexicon data loading
`math/bits.OnesCount64`, `TrailingZeros64`	Go 1.9	Anomaly bitfield operations
`go.work` (workspace)	Go 1.18	Multi-module repo structure
`float32` — no Go version constraint	always	Core field computation

No Go-version-specific micro-optimizations applied

Benchmarks run equivalently on Go 1.24 and future versions. No version-gated assembly (//go:build), no iterator experiments (range-over-func), no generic constraints. The codebase is forward-compatible and will benefit from future compiler improvements (register allocation, inlining heuristics, loop optimization) automatically.

Code generation

The lexgen tool generates a complete, buildable Go module for any NRC EmoLex language+backend combination.

Usage

# From the repo root
go run ./lexicons/cmd/lexgen --lang fr --type mph --data /path/to/nrc_french.txt

# With custom negatives (French negation words)
go run ./lexicons/cmd/lexgen --lang fr --type mph --data /path/to/nrc_french.txt \
    --negatives "ne,ne pas,ni,ni...ni,sans"

# With gzip compression (extended/expanded lexicons)
go run ./lexicons/cmd/lexgen --lang en --type mph --data /path/to/nrc_expanded.txt \
    --compress --dir en-ext-mph

What it generates

lexicons/fr-mph/
├── lexicon.go        # FrMphLexicon type, NewFrMphLexicon(), Lookup()
├── table.go          # Murmur3 hash + FrMphTable with linear probing
├── bench_test.go     # BenchmarkFrMphLexicon_Lookup_Known/Unknown/ProductionStream
├── nrc_emolex_fr.txt # Copy of input data (embedded via //go:embed)
└── go.mod            # Module github.com/drupaldoesnotexists/go-stedy/lexicons/fr-mph

Adding to the workspace

Edit go.work:

go 1.24.2

use (
    .
    ./lexicons
    ./lexicons/en-mph
    ./lexicons/en-trie
    ./lexicons/ru-mph
    ./lexicons/ru-trie
    ./lexicons/ru-koi8-mph
    ./lexicons/ru-koi8-trie
    ./lexicons/ru-mph
    ./lexicons/ru-trie
    ./lexicons/ru-koi8-mph
    ./lexicons/ru-koi8-trie
    ./lexicons/fr-mph      # new
)

Template maintenance

Templates are in lexicons/_template/{mph,trie}/*.go.tmpl. Three template variables customize each generated module:

Variable	Source	Example
`Lang`	`--lang` flag	`fr`
`Impl`	`--type` flag	`mph` or `trie`
`ImplName`	Derived from Impl	`Mph` or `Trie`
`Package`	Lang	`fr`
`ImportRef`	Lang+Impl	`frmph`
`DataFile`	Basename of `--data`	`nrc_emolex_fr.txt`
`Negatives`	`--negatives` flag (comma-separated)	`["ne","ne pas","ni...ni","sans"]`

File-by-file reference

Root (`go-stedy/`)

stedy.go (603 lines): Core engine. Types: LexiconVector (10 float32), Lexicon (generic constraint interface), Ingester (runtime-polymorphism interface), Anomaly (word + 10 float32), Sentiment (DramaIndex, Valence, 10 float32, Anomalies), tokenData (cache struct), Stedy[L] (generic engine with all buffers). Functions: New[L], NewDefault[L], Ingest, IngestCopy, parseTokens, parseTokensASCII, parseTokensUnicode, addToken, reset, ensureCapacity, abs32, isAsciiLetter, isAsciiDigit.

stedy_internal_test.go (460 lines): 31 unit tests. See "Testing" section.

stedy_bench_test.go (370 lines): 18 integration benchmarks. See "Benchmarks" section.

stedy_bench_mutability_test.go (80 lines): 2 thread-unsafe mutability benchmarks (MutateInPlace vs RecreateEngine). Built with -tags thread_unsafe_mutability.

stedy_mutability.go (28 lines): Build-tag-gated ParamMutator interface and Set* methods (SetAlpha, SetThetaMicro, SetThetaDelta, SetWeightMicro, SetWeightMacro). Only compiles with -tags thread_unsafe_mutability.

cmd/stedy-cli/ (8 files): CLI tool with subcommands. main.go — subcommand dispatch (analyze, label, optimize). analyze.go — text analysis with flags. label.go — bubbletea TUI for pairwise comparison labeling. optimize.go — random search parameter optimizer. dataset.go — CSV/JSON dataset loading. elo.go — Elo ranking for label mode. mutability.go / nomutability.go — build-tag-gated param mutation for optimize mode.

go.mod: Module github.com/drupaldoesnotexists/go-stedy. Go 1.21. No external dependencies.

go.work: Workspace defining root + lexicons + all 6 lexicon modules. go.work: Workspace defining root + lexicons + all 6 lexicon modules.

LICENSE: Mozilla Public License 2.0 (MPL-2.0).

.gitignore: Binaries, IDE files, OS artifacts. Excludes go.work.sum.

lexicons/

parser.go (60 lines): ParseEmbedFile(data, negatives) — reads NRC TSV format, builds map[string]LexiconVector. Shared across all lexicon backends.

go.mod: Module github.com/drupaldoesnotexists/go-stedy/lexicons. Imports root module for LexiconVector type.

lexicons/en-mph/

lexicon.go (31 lines): MphLexicon wrapping HashTable. Imports root for LexiconVector and lexicons for ParseEmbedFile.

table.go (151 lines): Murmur3 32-bit hash (adapted from cespare/mph, MIT). HashTable with nextPow2(n*2) sizing, 50% load factor, linear probing, inline []hashEntry storage. BuildTable + Lookup.

bench_test.go (70 lines): 3 benchmarks (Lookup known, Lookup unknown, ProductionStream with full engine).

nrc_emolex_en.txt: Embedded NRC EmoLex English data (~14700 entries, TSV).

go.mod: Module github.com/drupaldoesnotexists/go-stedy/lexicons/en-mph.

lexicons/en-trie/

lexicon.go (77 lines): TrieLexicon wrapping 256-ary trieNode hierarchy. Case-normalizing lookup (uppercase → lowercase via b |= 0x20). Prefix matching (falls back to last valid prefix).

bench_test.go (70 lines): 3 benchmarks (Lookup known, Lookup unknown, ProductionStream with full engine).

nrc_emolex_en.txt: Same NRC data, duplicated for independent module compilation.

go.mod: Module github.com/drupaldoesnotexists/go-stedy/lexicons/en-trie.

Baseline profiling (MPH, unoptimized)

CPU profile captured with 99.4% sample coverage, Ingest on 1000 synthetic texts (10-300 words, ~130 avg). Raw pprof -top filtered to Ingest:

Function	% of Ingest	Cumulative
`mph.Table.Lookup`	25%	25%
(duplicate Lookup in PASS2)	22%	47%
`mph.murmurSeed.hash` (called by Lookup)	16%	63%
`unicode.IsLetter` + `IsDigit` (tokenizer)	8.7%	71.7%
PASS1 field compute (10-FMA loop)	~9%	~81%
sumSquares (anomaly distance)	~5.4%	~86%
macro-inversion binary search	~5.4%	~92%
result copy (anomaly allocation)	~5%	~97%
`math.Abs(float64(x))`	~2.7%	~99.7%

Key insight: Lookup is called TWICE per token (PASS1 + PASS2). Combined 38% of Ingest. This was the single largest opportunity.

Optimization history

Each optimization was motivated by profile data, not guesswork:

Stage	MPH Ingest	Gain	Rationale
Baseline	~3.32s	—	cespare/mph 2-level perfect hash, separate buffers, 2× lookup
baseVecBuffer	~2.66s	−20%	Cache Lookup result from PASS1, reuse in PASS2. Profile showed 38% in Lookup — cutting it in half gives ~20% from ~38% of pipeline.
ASCII fast path	merged	−37% tokenize	Profile: 8.7% in unicode.IsLetter for purely ASCII text. Byte ops `>= 'a' && <= 'z'` replace function call.
abs32	merged	~−1%	`math.Abs(float64)` converts float32→float64→math.Abs→float64→float32. Direct `if x < 0 { x = -x }` on float32 saves 2.7% of profile. Simple, local.
bits.OnesCount64	merged	—	Replaced bit loop. `OnesCount64` is a single CPU instruction. Saves ~0.3% — worth it for correctness, real gain is in TrailingZeros.
Anomaly inline PASS1	merged	−5%	Moved anomaly detection from PASS2 to PASS1 (after caching baseVec). Eliminated the 22% duplicate Lookup entirely.
Unroll i=0	merged	~−0.5%	First token has no prev field. `if i > 0` check removed from hot path.
tokenData struct	merged	~−2%	Three separate heaps → one contiguous struct. Adjacent cache lines for PASS2 scalar access.
bits.TrailingZeros64	merged	~−2%	O(anomalies) collection vs O(tokens) linear scan. For 0-anomaly texts: 16 bit ops vs 1000 checks.
Flat hash table	merged	~−4%	Replaced cespare/mph (2-level perfect hash + keys deref) with single murmur3 + linear probing. One hash call, contiguous entries, -1 dependency.
Total (no flags)	−29%
Aggressive inlining (`-gcflags=-l=4`)	additive ~−9%		More inline decisions in 10-FMA loop and small helpers.
Zero-copy Ingest	*		18→11 B/op (0 allocs). Not a throughput gain — allocation/cache effect.

*Zero-copy Ingest throughput effect is within noise (±1%). Its value is GC pressure reduction: 18 allocs/MB became 0, freeing CPU time for the rest of your application.

Amdahl's law note: Gains are sub-additive. The −29% total is measured from optimized vs baseline, not arithmetic sum of individual gains (which would exceed 29% due to overlapping scopes).

Optimization triage — what was measured and rejected

Micro-benchmarks were written for candidate optimizations before implementing them in the main pipeline. Several candidates were rejected:

Candidate	Component bench	Verdict	Reason
Unroll `if i>0` in PASS1 loop	`BenchmarkFieldComputation`	~0.5%	Not worth code complexity. Did it anyway (one line change).
Replace `append` with index assignment	`BenchmarkAddToken`	~0.3%	append is inlined to the same codegen.
Manual slice bounds elision	`BenchmarkParseTokens`	~0%	Go compiler already elides bounds checks for simple patterns.
AVX2 assembly (8× float32)	—	No code	10-FMA loop is ~9% of Ingest. 3× speedup → ~6% total gain. Not worth plan9 assembly maintenance.

The detailed component benchmarks exist in stedy_bench_components_test.go and provide sub-nanosecond resolution for each building block. Use them to validate future optimizations before touching the main pipeline.

License

Mozilla Public License 2.0 (MPL-2.0). See LICENSE.

The NRC Emotion Lexicon (NRC EmoLex) data files are © National Research Council Canada and are used under their terms. The NRC EmoLex is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Name		Name	Last commit message	Last commit date
Latest commit History 33 Commits
.github/ISSUE_TEMPLATE		.github/ISSUE_TEMPLATE
build		build
cmd/stedy-cli		cmd/stedy-cli
lexicons		lexicons
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
go.mod		go.mod
go.sum		go.sum
go.work		go.work
stedy-baseline.txt		stedy-baseline.txt
stedy.go		stedy.go
stedy_bench_components_test.go		stedy_bench_components_test.go
stedy_bench_mutability_test.go		stedy_bench_mutability_test.go
stedy_bench_test.go		stedy_bench_test.go
stedy_internal_test.go		stedy_internal_test.go
stedy_mutability.go		stedy_mutability.go
testdata_war_and_peace.txt		testdata_war_and_peace.txt

Folders and files

Latest commit

History

Repository files navigation

go-stedy — Real-time Emotional Vector Analysis for Text Streams

What problem does this solve?

Real-world use cases

What it is NOT

Performance comparison with other approaches

Quick start

CLI tool

Subcommands

analyze (default)

label — Interactive dataset labeling via pairwise comparison

optimize — Random search parameter optimization

How it works — from 30,000 feet

Thread-unsafe mutability

Thread safety warning

Build tag

Algorithm in detail

Tokenization (parseTokens)

PASS 1 — Exponential smoothing field computation

What this means practically

Anomaly detection (inline in PASS 1)

What anomalies detect

PASS 2 — Volatility and inversions

Volatility

Micro-inversions (curvature)

What micro-inversions measure

Macro-inversions (zero-crossings)

What macro-inversions measure

DramaIndex

WeightedDramaIndex

Anomaly result collection

Anomalies as compression mechanism

Zero-copy output model

Parameters — what they mean for YOUR problem

alpha (default 0.5)

thetaMicro (default 0.8)

thetaDelta (default 0.20)

Parameter presets by use case

API reference

Types

Lexicon interface

Engine construction

Ingester interface (runtime polymorphism)

Ingest

Stedy internal types (not part of public API, visible in package only)

Lexicon backends

Extended lexicons (*-ext-mph)

en-mph: Hash table with linear probing

en-trie: 256-ary trie

Which backend to choose

Russian language backends

UTF-8 vs KOI-8 — the performance gap

Storage in KOI-8 — practical recommendation

When to use each backend

Adding a new language lexicon (code generation)

Architecture

Module structure (Go workspace)

Module dependency graph

Internal data structures

Optimization highlights

Testing

Unit tests (31 tests)

Running tests

Benchmarks

Platform

Integration benchmarks (stedy_bench_test.go)

Thread-unsafe mutability benchmarks (stedy_bench_mutability_test.go)

Component benchmarks (stedy_bench_components_test.go)

Lexicon module benchmarks

Performance tuning

1. Reuse the engine

2. Compiler flags

3. ThetaDelta tuning (disabling anomalies)

4. SIMD opportunity (not yet implemented)

5. Choose the right lexicon backend

Text preprocessing — how input preparation affects backend choice

Raw text (no preprocessing)

`analyze` (default)

`label` — Interactive dataset labeling via pairwise comparison

`optimize` — Random search parameter optimization

Tokenization (`parseTokens`)

Extended lexicons (`*-ext-mph`)

Benchmark API — `b.Loop()`

Root (`go-stedy/`)

Packages