Adopt new structure for proper grammar generation

[gravemalte]

This PR is a draft/suggestion PR. Collaboration with @MiriamGaus and @coemgen1992.

We discovered that this repository is in a very poor state, both in terms of documentation and dependencies. While we want to promote UVL, achieving this requires a proper structure and good understandability of the repository. Currently, the information in the README is outdated, the dependencies are outdated, and it is difficult to generate parsers for different languages. The CI (as far as I know) is not working properly. For these reasons, we decided to clean up the entire repository and document it thoroughly.

This effort introduces changes to the structure, the documentation, and the grammar files. I have split the grammar into lexer- and parser-relevant files to ensure modular behavior when generating UVL for different programming languages. Additionally, integrating new languages is now easier than ever. The CI can easily generate the parser/lexer, and in the future, we plan to introduce a proper test environment.

The tests will be moved to a new repository, and we will integrate them as a git submodule

Please provide feedback and bear with us if you encounter any mistakes.

[SundermannC]

Thanks a lot for putting this together. I think the repository is in a much better shape with your changes. I have some small comments, but all of those can also be performed later. From my tests, the parser generation for each language appears to work as expected.

[SundermannC]

Thanks a lot for the quick answers. I guess we could merge this very soon. The main hindrance are the merge conflicts. Can you resolve them @gravemalte

[gravemalte]

Thanks a lot for the quick answers. I guess we could merge this very soon. The main hindrance are the merge conflicts. Can you resolve them @gravemalte

Sure I will resolve them. Next week I have some time to do the rest.
Before merging I would like to have more people to try out the changes because they are breaking and critical. So please, to all of the UVL people try to build this locally.

[SundermannC]

Makes sense. I asked @jagalindo and @rotkiv93 for further reviews as they appear to be the main users/developers for the Python and JS parsers respectively.

[jagalindo]

This PR makes sense to me. I'd agree on having different readmes for different targets if needed. We just got the uvl org in pypi, so we cab move the pypi package to the shared org. The organization seems right to me. What do you think @rotkiv93?

[rotkiv93]

I checked our JS version, and we need to change the package.json file in order to be able to build the grammar properly.

We need to change this line:

uvl-parser/js/package.json

Line 7 in bfecb77

"build-grammar": "antlr4 -Dlanguage=JavaScript -o src/lib -Xexact-output-dir ../uvl/UVLJavaScript.g4",

With this command:
antlr4 -lib ../uvl/ -Dlanguage=JavaScript -o ./src/lib ../uvl/JavaScript/UVLJavaScriptLexer.g4 ../uvl/JavaScript/UVLJavaScriptParser.g4

Also, we checked and the grammar doesnt compile properly with older antlr4 versions, for example, v4.7.2 does not work properly, we should put a disclaimer somewher (probably in the JS README)

[gravemalte]

I checked our JS version, and we need to change the package.json file in order to be able to build the grammar properly.

We need to change this line:

uvl-parser/js/package.json

Line 7 in bfecb77

"build-grammar": "antlr4 -Dlanguage=JavaScript -o src/lib -Xexact-output-dir ../uvl/UVLJavaScript.g4",

With this command: antlr4 -lib ../uvl/ -Dlanguage=JavaScript -o ./src/lib ../uvl/JavaScript/UVLJavaScriptLexer.g4 ../uvl/JavaScript/UVLJavaScriptParser.g4

Also, we checked and the grammar doesnt compile properly with older antlr4 versions, for example, v4.7.2 does not work properly, we should put a disclaimer somewher (probably in the JS README)

I will change this thanks for pointing out. Speaking of JS, I have a question about the export statement in the index.js.
The export of the UVL ANTLR UVLJavaScriptParser.js is marked there, is there any reason why? Because the parser itself is quite useless without the UVLJavaScriptCustomLexer. Also the UVLJavaScriptParser is exported in the FeatureModel.js again.

[rotkiv93]

@gravemalte You are right—it's weird to have it like this. There's no problem with deleting it from the index.js file. Also, I don't think it's relevant to have it exported in the FeatureModel.js file either. I think we can delete it for now, and if a case arises where it needs to be exported, we can change it in the future.

[gravemalte]

I resolved all the conflicts and addressed the comments.

But please test this again.

I tried to run the test suite from @MiriamGaus in PR Universal-Variability-Language/java-fm-metamodel#15 with my changes, but two tests are failing. I don't know if they failed before. One is about the math expression that a result is wrong, the other is about the de.vill.parsing.ParsingTests.checkLanguageLevelInclude. But error is not very helpful. I didn't have time to investigate these errors, and I'm also not sure if they are from my parser changes here.

[MiriamGaus]

In the old version the test on the de.vill.parsing.ParsingTests.checkLanguageLevelInclude runs and the other test is fixed in the second open PR for the UVL-Parser. There this is a parser problem.

[gravemalte]

@MiriamGaus I will merge now your changes in my branch and try again.

[gravemalte]

@MiriamGaus the Expression fixed the math error like you said, but the lanaguage_level test is still failing. Any ideas?

[gravemalte]

Found the cause, was on me. I screwed some symbols up.

Test are passing.

[rotkiv93]

The JS part LGTM

[jagalindo]

I'd try to investigate if there is any mechanism to update automatically the version numbers so it doesn't crash in the ci actions.