Improved tokenizer #94
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR updates test snapshots to reflect changes in tokenization behavior where adjacent text tokens are now being merged into single tokens. The change appears to consolidate token sequences that previously split text/symbols/paths/commands into individual tokens, now combining them into fewer, larger tokens for improved efficiency.
Reviewed Changes
Copilot reviewed 241 out of 253 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/grammar/snapshots/r#mod__grammar__alpha050__unfinished_function_call.snap | Updates snapshots for path and filename tokenization to merge tokens like "std"/"/"/"array" into "std/array" |
| tests/grammar/snapshots/r#mod__grammar__alpha050__unfinished_function_call-2.snap | Similar path token merging updates for parsed output |
| tests/grammar/snapshots/r#mod__grammar__alpha050__stdlib_text.snap | Extensive token merging for shell commands, escape sequences, and text patterns |
| tests/grammar/snapshots/r#mod__grammar__alpha050__stdlib_text-2.snap | Corresponding parsed output updates with merged text nodes |
| tests/grammar/snapshots/r#mod__grammar__alpha050__stdlib_math.snap | Token merging for mathematical expressions and shell commands |
| tests/grammar/snapshots/r#mod__grammar__alpha050__stdlib_http.snap | Merges command-line options and arguments into single tokens |
| tests/grammar/snapshots/r#mod__grammar__alpha050__stdlib_date.snap | Date format string and command token consolidation |
| tests/grammar/snapshots/r#mod__grammar__alpha050__numbers.snap | New snapshot file for number literal parsing |
| tests/grammar/snapshots/r#mod__grammar__alpha050__mv_files.snap | Path token merging for file operations |
| tests/grammar/snapshots/r#mod__grammar__alpha050__comments_in_ifs.snap | Text consolidation in conditional statements |
| Alpha040/035/034 snapshots | Similar token merging updates for older grammar versions |
| src/utils.rs | Formatting change to split import statement into multiple lines |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 296 out of 407 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
No description provided.