-
Notifications
You must be signed in to change notification settings - Fork 0
Add Parser with format detection and smart splitting #14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
23163d2
feat: Add Parser for format detection and word splitting
google-labs-jules[bot] b14952c
refactor: Use feature flags for parser configuration
google-labs-jules[bot] 9745186
refactor: Switch to property-based parsing configuration
google-labs-jules[bot] dfc95c7
refactor: Introduce ParserOption interface and custom split logic
google-labs-jules[bot] d4a019a
refactor: Extract DefaultSplitCamelCase and document SmartAcronyms
google-labs-jules[bot] d053d91
doc: Add WordClassifier type and clarify classify method
google-labs-jules[bot] 5dc150e
test: Add Custom SubPart Logic test case
google-labs-jules[bot] 1cc96e4
test: Clean up parser tests and verify delimiter logic
google-labs-jules[bot] cfe7c2e
refactor: Implement multi-layered parsing architecture
google-labs-jules[bot] 73864cc
refactor: Finalize SubPart architecture and docs
google-labs-jules[bot] c7b3871
style: Run go fmt and address acronym classification feedback
google-labs-jules[bot] f1c458d
docs: Add Parsing section to README
google-labs-jules[bot] 799a9e2
test: Update Contextual Dash test expectation
google-labs-jules[bot] 6783002
test: Add XMLReader test case for SmartAcronyms
google-labs-jules[bot] db7b23f
refactor: Update Parse signature and add docs
google-labs-jules[bot] e5ffcbe
fix: Remove unreachable case in Parse type switch
google-labs-jules[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,210 @@ | ||
| package strings2 | ||
|
|
||
| import ( | ||
| "strings" | ||
| "unicode" | ||
| ) | ||
|
|
||
| // Parse parses the input string into a slice of Words based on detection or provided options. | ||
| // It follows the pipeline: String -> SubParts -> Parts -> Words. | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jules Document SmartAcronym well please
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've added documentation to the |
||
| // | ||
| // opts can be: | ||
| // - ParserOption interface | ||
| // - Partitioner function | ||
| // - ParserSmartAcronyms bool | ||
| func Parse(input string, opts ...any) ([]Word, error) { | ||
| // Level 5: Scan | ||
| subs, stats := StringToSubParts(input) | ||
|
|
||
| p := &ParserConfig{ | ||
| SmartAcronyms: true, | ||
| } | ||
|
|
||
| for _, opt := range opts { | ||
| switch o := opt.(type) { | ||
| case Partitioner: | ||
| p.Partitioner = o | ||
| case ParserOption: | ||
| o.Apply(p) | ||
| } | ||
| } | ||
|
|
||
| // Level 4: Partition | ||
| // If partitioner is not set, try to detect | ||
| partitioner := p.Partitioner | ||
| if partitioner == nil { | ||
| partitioner = DetectPartitioner(stats) | ||
| } | ||
|
|
||
| parts := SubPartsToParts(subs, partitioner) | ||
|
|
||
| // Level 3: Words | ||
| words := PartsToWords(parts, p) | ||
|
|
||
| return words, nil | ||
| } | ||
|
|
||
| // ParserConfig holds configuration for the parsing pipeline. | ||
| type ParserConfig struct { | ||
| Partitioner Partitioner | ||
| // SmartAcronyms controls whether all-uppercase words (longer than 1 char) | ||
| // should be treated as AcronymWord instead of UpperCaseWord. | ||
| // Defaults to true. | ||
| SmartAcronyms bool | ||
| } | ||
|
|
||
| // ParserOption configures the parser. | ||
| type ParserOption interface { | ||
| Apply(*ParserConfig) | ||
| } | ||
|
|
||
| type funcParserOption func(*ParserConfig) | ||
|
|
||
| func (f funcParserOption) Apply(p *ParserConfig) { f(p) } | ||
|
|
||
| // ParserSmartAcronyms is a typed option for SmartAcronyms configuration. | ||
| // It allows passing a boolean-like type directly to Parse. | ||
| type ParserSmartAcronyms bool | ||
|
|
||
| func (b ParserSmartAcronyms) Apply(p *ParserConfig) { | ||
| p.SmartAcronyms = bool(b) | ||
| } | ||
|
|
||
| // WithPartitioner sets a specific partitioner strategy. | ||
| func WithPartitioner(pt Partitioner) ParserOption { | ||
| return funcParserOption(func(p *ParserConfig) { | ||
| p.Partitioner = pt | ||
| }) | ||
| } | ||
|
|
||
| // WithSmartAcronyms enables or disables smart acronym detection. | ||
| func WithSmartAcronyms(enabled bool) ParserOption { | ||
| return funcParserOption(func(p *ParserConfig) { | ||
| p.SmartAcronyms = enabled | ||
| }) | ||
| } | ||
|
|
||
| // DetectPartitioner uses stats to guess the best partitioner. | ||
| func DetectPartitioner(stats Stats) Partitioner { | ||
| // Heuristic: | ||
| // If spaces > 0, likely Sentence (Sentence usually beats Kebab/Snake if mixed) | ||
| if stats.Spaces > 0 { | ||
| return func(subs []SubPart) []Part { | ||
| // Space partitioner | ||
| var parts []Part | ||
| var current []SubPart | ||
| for _, s := range subs { | ||
| if s.IsSpace() { | ||
| if len(current) > 0 { | ||
| parts = append(parts, &WordPart{BasePart{Subs: current}}) | ||
| current = nil | ||
| } | ||
| } else { | ||
| current = append(current, s) | ||
| } | ||
| } | ||
| if len(current) > 0 { | ||
| parts = append(parts, &WordPart{BasePart{Subs: current}}) | ||
| } | ||
| return parts | ||
| } | ||
| } | ||
| // If underscores > 0, likely SnakeCase | ||
| if stats.SymbolCounts['_'] > 0 { | ||
| return SnakeCasePartitioner | ||
| } | ||
| // If hyphens > 0, likely KebabCase | ||
| if stats.SymbolCounts['-'] > 0 { | ||
| return KebabCasePartitioner | ||
| } | ||
|
|
||
| // Default to CamelCase | ||
| return CamelCasePartitioner | ||
| } | ||
|
|
||
| // PartsToWords converts Parts to Words using classification logic. | ||
| func PartsToWords(parts []Part, config *ParserConfig) []Word { | ||
| var words []Word | ||
| for _, part := range parts { | ||
| words = append(words, ClassifyPart(part, config)) | ||
| } | ||
| return words | ||
| } | ||
|
|
||
| // ClassifyPart converts a Part into a Word. | ||
| func ClassifyPart(part Part, config *ParserConfig) Word { | ||
| s := part.String() | ||
| if s == "" { | ||
| return ExactCaseWord("") | ||
| } | ||
|
|
||
| // Check for dots -> Acronym | ||
| if strings.Contains(s, ".") { | ||
| return AcronymWord(s) | ||
| } | ||
|
|
||
| // Check casing | ||
| isAllUpper := true | ||
| isAllLower := true | ||
| isTitle := false | ||
|
|
||
| runes := []rune(s) | ||
| if len(runes) > 0 && unicode.IsUpper(runes[0]) { | ||
| isTitle = true | ||
| } | ||
|
|
||
| for i, r := range runes { | ||
| if !unicode.IsUpper(r) && unicode.IsLetter(r) { | ||
| isAllUpper = false | ||
| } | ||
| if !unicode.IsLower(r) && unicode.IsLetter(r) { | ||
| isAllLower = false | ||
| } | ||
| if i > 0 && unicode.IsUpper(r) { | ||
| isTitle = false | ||
| } | ||
| } | ||
|
|
||
| if isAllUpper { | ||
| // Use SmartAcronyms config or default | ||
| smartAcronyms := true | ||
| if config != nil { | ||
| smartAcronyms = config.SmartAcronyms | ||
| } | ||
|
|
||
| if smartAcronyms && len(runes) > 1 { | ||
| return AcronymWord(s) | ||
| } | ||
| return UpperCaseWord(s) | ||
| } | ||
|
|
||
| if isAllLower { | ||
| return SingleCaseWord(s) | ||
| } | ||
|
|
||
| if isTitle { | ||
| return FirstUpperCaseWord(s) | ||
| } | ||
|
|
||
| return ExactCaseWord(s) | ||
| } | ||
|
|
||
| // Level 1 / 2 Helpers | ||
|
|
||
| func ParseSnakeCase(input string) []Word { | ||
| subs, _ := StringToSubParts(input) | ||
| parts := SubPartsToParts(subs, SnakeCasePartitioner) | ||
| return PartsToWords(parts, nil) | ||
| } | ||
|
|
||
| func ParseCamelCase(input string) []Word { | ||
| subs, _ := StringToSubParts(input) | ||
| parts := SubPartsToParts(subs, CamelCasePartitioner) | ||
| return PartsToWords(parts, nil) | ||
| } | ||
|
|
||
| func ParseKebabCase(input string) []Word { | ||
| subs, _ := StringToSubParts(input) | ||
| parts := SubPartsToParts(subs, KebabCasePartitioner) | ||
| return PartsToWords(parts, nil) | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jules please talk more extensively about how this works including listing and explaining all types.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've updated the README to explain the parsing architecture, types, and levels in more detail.