Warning

The development to use pegen for Xonsh is halted mostly because of the following reasons.

• The development activity on pegen has slowed down. Recent Python versions have significant grammar changes; If pegen provides the Python specific changes, then it can be reused without much maintenance burden. Other option is to copy grammar from CPython repo itself and transpile the C code to Python
• I have heavily modified the original Python generator to reduce the amount of code generated. Even though the memory usage has been reduced for small files, the performance wasn't upto the mark. So I've decided to focus on reusing Ruff's parser.

—- Noortheen Raja J

xonsh-parser

Xonsh parser using CPython's PEGen grammar

---

Documentation: https://jnoortheen.github.io/xonsh-parser/

Source Code: https://github.com/jnoortheen/xonsh-parser

Benchmarks: https://xonsh.github.io/peg-parser/dev/bench/

---

Development

Setup environment

• You can use any PEP-621 supported tools like PDM, Hatch ... to manage the development environment and production build.

# install development deps
uv sync

• We use pre-commit for linting the code.

# setup linters,formatters etc.,
pre-commit install

• We use Taskfile to manage development tasks.

task test

# to watch for changes and run tests
task test --watch -- -x --ff

# release a new version
task release

Workflow

1. Explore the PEG grammar notations from the following pages a. https://we-like-parsers.github.io/pegen/ b. https://devguide.python.org/internals/parser/
2. Edit the xonsh.gram while adding any new actions to subheader.py
3. Test the new changes with task test as the parser.py will get autogenerated
4. You can force regeneration with task generate -f

License

This project is licensed under the terms of the MIT license.

Credits

• Thanks to pegen project for the Python version of the CPython's PEG parser
• Thanks to intellij-pegen-plugin