Discuss approach for submitting Ruby version support changes

[piotrhoppe]

Hi @enebo,

I would like to discuss how to provide my latest changes to the JRuby Parser project.

With the assistance of AI, I have added support for Ruby 3.3.x. This involves extensive changes to the parser.

My preference would be to submit all of these changes as a single pull request, but I am also happy to split them into several smaller pull requests if that would make them easier to review.

Would you prefer a single pull request, or a series of smaller pull requests, with each one covering support for a specific Ruby version (e.g. 2.4.x, 2.5.x, and so on up to 3.3.x)?

I would appreciate your guidance on the preferred approach before I prepare the pull request(s).

I would also like to mention that I have used this new parser in my other project, Community Ruby, where you previously contributed as a member of Oracle.

[enebo]

@piotrhoppe I think as one PR is ok. This project was moved to its own specifically because we did not want the deal with the complexity or overhead of maintain position info in JRuby proper. To my knowledge we are so far behind in ruby compat on this project I doubt anyone is using this (barring an ancient version where they are probably ok using the older artifact).

I think one PR is ok and once that is landed I suspect iteration on this project will be finer grain. I will be curious to see how it will differ (or not from historical versions of JRuby propers parser. I assume it will have to have the same productions and lexer weirdness.

[enebo]

@piotrhoppe Oh and yeah I took over support for a while until it became too much work and I seemed to have very few users. I also remember I was slowly refactoring the APIs used to fit more into jruby-parser rather than writing lots of code which assumed jruby parsing API was not changeable. The previous maintainers never really approached us with their needs. I suspect there are still a lot of opportunity for that still.

[piotrhoppe]

Thanks for your response! Great to hear that it can be done as a single pull request. I'll start preparing it.

[piotrhoppe]

@enebo This pull request moves JRuby Parser closer to supporting newer Ruby releases. It currently covers Ruby versions up to 3.3, with only the two newest versions, 3.4 and 4.0, still missing.

I'm planning to try adding support for these two versions in my free time. From what I have seen, Ruby 3.3 does not seem to be too far from Ruby 4.0 in terms of the parser changes required.

There are no new "extra" features added in this pull request beyond following the existing parser style. For me, the main goal is to have this support available for the Community Ruby plugin, where I need a parser that can handle newer Ruby versions.

[headius]

Happy to see this getting updated again!

I want to mention that we have turned the native Prism parser into a JVM-hosted WASM library and would like to eventually make that the official Ruby parser library for JVM. The API and AST would be a bit different.

[enebo]

The Prism API is something you will have to deal with from 4.1+. It is quite a bit different and due to all the lower level AST usage in NB Ruby it might be worth considering either a) not supporting previous versions but then changing a lot of source code anyways b) abstracting things enough so old and new work together. I think b) will not be simple but I suspect a) is a pretty bold choice.

[enebo]

I should add c) which would be just nursing along the bison/LALR grammar since the influx of new Ruby syntax has slowed down a lot. This could actually be the simplest option but being able to use Prism would mean never having to fix syntax bugs that make a valid AST (Prism has a lot of people working on it).

[headius]

I think b) would only be useful if there is a need to parse multiple versions of Ruby code with a single copy of the library. Given our experiences maintaining multiple versions at the same time in a single code base, my preference would definitely be for a).

As this moves towards prism, we may also be able to reduce the amount of code duplication from JRuby to this parser.