Ease installation and support developer mode (eval, test...)

[yanntm]

Hi,
We are currently working with some students at connecting our own refactoring front-end to your engine ; the idea is that we rewrite loops when we can to use the Stream API, so that your own tool finds more points where it could apply the parallelStream transformation.

But we've been having issues with installation and the various dependencies in your code base. To limit headaches particularly for my students who are just learning about plugins/eclipse dev, I built a (hopefully) cleaner environment based off your code.

Basically my fork of your project https://github.com/yanntm/Optimize-Java-8-Streams-Refactoring/ has no sematic differences, but :

• a lot of changes to the maven build/travis, features and updatesite definitions
• some changes in Manifest where appropriate to refactor dependencies.
• I also discarded most of the lib/ folders (in favor of properly handled Plug-in dependencies),
• dropped any "imported package" dependency (these are not dealt with by maven/eclipse itself)
• edited the build path variables appropriately,
• and emptied the "update site" project that contained a bunch of artifacts that Maven is supposed to regenerate/download.

( I've also had to build a new maven enabled build for the "metrics" dependency here https://github.com/yanntm/metrics2-gh since the project on sourceforge offers Eclipse 3.X format update site that maven cannot properly treat as a p2 repo. In the process I updated a bit of code that used API moved from "swt.custom" to "swt.widgets" but that was very minor. So we now depend on that particular build of "metrics" intead of sourceforge)

So I'm using travis-ci to build and run the stuff, though the Jacoco/reporting thing I'm not sure about, the rest works fine and produces an update site hosted on github pages, here :
https://yanntm.github.io/Optimize-Java-8-Streams-Refactoring/

So you get :

• a single update site, no need to tell anyone anymore about all those (nasty) dependencies
• two category/features instead of one : the first one is just the core functionality you had on your update site, the second one contains "test" and "eval" plugins. The idea is that once you install this second feature, opening the eval or test project in eclipse works directly, as all dependencies are found in the current platform
• versioning using maven snapshot : the versions (I bumped to 2.0.0-SNAPSHOT) evolve at each build, so that a client will always get the latest version of the plugins, even if he has some older version installed.

I just tested using a clean eclipse "2020-03 Java Developer", install the PDE from normal eclipse site, then install both features from update site https://yanntm.github.io/Optimize-Java-8-Streams-Refactoring/ , clone (my fork of) the repo, open "eval" (or any of the projects in the repo, but eval and test have the worst dependencies) => no compilation/issues, it works out of the box.

I'd be happy to submit a PR with these changes, if you like them, but I thought I'd post an issue first to discuss it with you.

Thanks for making this code base available,
Best regards
Yann Thierry-Mieg
(Sorbonne Université)

[yanntm]

Just an edit, I think it's easier to also get PDE and the JDT sources (developper ressources) so that we are more comfortable, and so it works whatever release of Eclipse was chosen initially.
So the "dev" feature now also brings JDT source + PDE to the platform.
It takes times for new users to realize they actually want "developer ressources" from the "Updates" part of official eclipse, so that ctrl-clicking e.g. CompilationUnit brings to the code.

This removes one more step from earlier procedure : "install the PDE from normal eclipse site" is now not necessary.

[yanntm]

Please note that I'm willing to spend some time with you to explain the changes I made in detail and how/why so you can then maintain the code base. We can setup a chat on Jitsi or something similar maybe so I can share my screen, if you'd like that email me Yann.Thierry-Mieg@lip6.fr , I'm at +6h from NYC though so morning for you is better for me.

[khatchad]

@yanntm,

Thank you for your interest in this project! I have replied inline below:

We are currently working with some students at connecting our own refactoring front-end to your engine ; the idea is that we rewrite loops when we can to use the Stream API,

Sounds great. You may also want to check out [1,2].

But we've been having issues with installation and the various dependencies in your code base.

I imagine that the issues are to do with developing with the code base rather than from a users perspective, correct?

To limit headaches particularly for my students who are just learning about plugins/eclipse dev, I built a (hopefully) cleaner environment based off your code.

Basically my fork of your project https://github.com/yanntm/Optimize-Java-8-Streams-Refactoring/ has no sematic differences, but :

• a lot of changes to the maven build/travis, features and updatesite definitions
• some changes in Manifest where appropriate to refactor dependencies.
• I also discarded most of the lib/ folders (in favor of properly handled Plug-in dependencies),
• dropped any "imported package" dependency (these are not dealt with by maven/eclipse itself)
• edited the build path variables appropriately,
• and emptied the "update site" project that contained a bunch of artifacts that Maven is supposed to regenerate/download.

The update site project is for users using the plug-in rather than developers developing against the plug-in's code base. However, I agree that the update site project has several problems. One of those was that we found a bug where update site dependencies were not being processed. As such, we wound up including the dependencies in this update site. Another is that we really should be using tycho to generate the update site rather than releasing the plug-in manually through the Eclipse GUI.

( I've also had to build a new maven enabled build for the "metrics" dependency here https://github.com/yanntm/metrics2-gh since the project on sourceforge offers Eclipse 3.X format update site that maven cannot properly treat as a p2 repo.

The metrics plug-in is only used for our evaluation plug-in, which may not be useful for your students. That being said, I am surprised that the current maven build is passing without listing the metrics update site in the parent pom.xml.

In turns out that the evaluation plug-in is not being built. I think were just focused on using maven to run the test suite, which is why we didn't include it. I've filed this issue.

In the process I updated a bit of code that used API moved from "swt.custom" to "swt.widgets" but that was very minor. So we now depend on that particular build of "metrics" intead of sourceforge)

So I'm using travis-ci to build and run the stuff, though the Jacoco/reporting thing I'm not sure about,

That's for code coverage. My guess is that you would need to use your own repo token for coverals.io here for your fork to build correctly.

the rest works fine and produces an update site hosted on github pages, here :
https://yanntm.github.io/Optimize-Java-8-Streams-Refactoring/

I received a 404 for that URL. The update site on the upstream repository just uses GitHub for hosting. See https://github.com/ponder-lab/Optimize-Java-8-Streams-Refactoring#update-site. It seems to work pretty well without the need to deploy a GitHub pages site. Is there an advantage to the latter? That being said, we would like to eventually transition to a tycho-built update site, and I am unsure yet if it would work the same way.

So you get :

• a single update site, no need to tell anyone anymore about all those (nasty) dependencies

Sorry, again, I am a bit confused as to whether this is in respect to users of the plug-in or developers coding against the plug-in. I am guessing from your situation that it is the latter.

From a user's standpoint, we only list the dependencies here for convenience. They are not explicitly required to install the plug-in.

Dependencies for developing are listed in CONTRIBUTING.md. There are two sets of those, one more focused on Eclipse things and the other more on related projects. For the former, I was under the impression that running mvn install would grab most of those. Is that not the case? Is this what you are having trouble with? We have had trouble with the Eclipse Test Framework in the past (once because the project was renamed).

• two category/features instead of one : the first one is just the core functionality you had on your update site, the second one contains "test" and "eval" plugins.

Right, we didn't include those as they are not "customer" facing. In other words, we could not imagine a reason why developers working with the plug-in themselves would use the update site to obtain these plug-ins. But, now I am beginning to understand the scenario.

The idea is that once you install this second feature, opening the eval or test project in eclipse works directly, as all dependencies are found in the current platform

OK, by opening them you mean through the "Plug-ins" view rather than by viewing the source code in the project explorer, for example.

• versioning using maven snapshot : the versions (I bumped to 2.0.0-SNAPSHOT) evolve at each build, so that a client will always get the latest version of the plugins, even if he has some older version installed.

Right. Currently, they are mixed. The idea was to only update the plug-ins that have changed. But I see that Tycho updates all of the versions, and that this is sort of the norm.

I just tested using a clean eclipse "2020-03 Java Developer", install the PDE from normal eclipse site, then install both features from update site https://yanntm.github.io/Optimize-Java-8-Streams-Refactoring/ , clone (my fork of) the repo, open "eval" (or any of the projects in the repo, but eval and test have the worst dependencies) => no compilation/issues, it works out of the box.

I'd be happy to submit a PR with these changes, if you like them, but I thought I'd post an issue first to discuss it with you.

Thanks for this! I see that you issued a PR and I will check that out ASAP. One question: would building instructions also need to be updated in CONTRIBUTING.md?

[1] L. Franklin, A. Gyori, J. Lahoda and D. Dig, "LambdaFicator: From imperative to functional programming through automated refactoring," 2013 35th International Conference on Software Engineering (ICSE), San Francisco, CA, 2013, pp. 1287-1290. http://mir.cs.illinois.edu/~gyori/pubs/icse13tool-LambdaFicator.pdf

[2] Alex Gyori, Lyle Franklin, Danny Dig, and Jan Lahoda. 2013. Crossing the gap from imperative to functional programming through refactoring. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013). Association for Computing Machinery, New York, NY, USA, 543–553. DOI:https://doi.org/10.1145/2491411.2491461. http://mir.cs.illinois.edu/gyori/pubs/LambdaFicator_FSE13.pdf

[khatchad]

Please note that I'm willing to spend some time with you to explain the changes I made in detail and how/why so you can then maintain the code base. We can setup a chat on Jitsi or something similar maybe so I can share my screen, if you'd like that email me Yann.Thierry-Mieg@lip6.fr , I'm at +6h from NYC though so morning for you is better for me.

Thanks, @yanntm. Let me have a look at the PR and get back to you.

[yanntm]

I imagine that the issues are to do with developing with the code base rather than from a users perspective, correct?

Yes most of this effort is for developpers, your update site works fine for just executing the refactoring. This functionality is now offered by one of the two features.

I received a 404 for that URL. The update site on the upstream repository just uses GitHub for hosting.

It's an eclipse update site not a web page, adding e.g. content.xml.xz to the address or pointing eclipse to it will work.
What you get on github pages is simply the branch called gh-pages, you can browse the contents here https://github.com/yanntm/Optimize-Java-8-Streams-Refactoring/tree/gh-pages

That being said, we would like to eventually transition to a tycho-built update site, and I am unsure yet if it would work the same way.

So yes this is built by tycho + travis, relvant configuration is here https://github.com/yanntm/Optimize-Java-8-Streams-Refactoring/blob/master/.travis.yml#L37-L43

Deploying to github pages is my usual choice, but we can deploy to a particular branch if you prefer, I have code examples for that, e.g. https://github.com/lip6/libDDD/blob/master/.travis.yml#L56-L63 this code deploys to build a branch in https://github.com/lip6/libDDD/tree/osx for MacOS, so this setup not using gh-pages is possible if you prefer it. This gives us addresses for clients with some "raw" in them like you currently use for your update sites e.g. https://github.com/lip6/libDDD/raw/osx/osx.tgz

Thanks for this! I see that you issued a PR and I will check that out ASAP. One question: would building instructions also need to be updated in CONTRIBUTING.md?

Yes I guess they become extremely simplified, the procedure becomes "install Dev feature from update site", no projects or other dependencies need to be listed, since they are packaged along with the dev feature, so then simply clone and import into eclipse the project you want to work with. Maven is reading all it's config from the tycho/eclipse configuration of the plugins, so no need for maven install either.

refs

Thanks for pointing these out, I'd not come across them yet.

[khatchad]

I imagine that the issues are to do with developing with the code base rather than from a users perspective, correct?

Yes most of this effort is for developpers, your update site works fine for just executing the refactoring. This functionality is now offered by one of the two features.

Great. Thanks for clearing that up.

I received a 404 for that URL. The update site on the upstream repository just uses GitHub for hosting.

It's an eclipse update site not a web page, adding e.g. content.xml to the address or pointing eclipse to it will work.

Oh right. I always get tripped up by that.

That being said, we would like to eventually transition to a tycho-built update site, and I am unsure yet if it would work the same way.

So yes this is built by tycho + travis, relvant configuration is here https://github.com/yanntm/Optimize-Java-8-Streams-Refactoring/blob/master/.travis.yml#L37-L43

Cool! I didn't know you could do that. Does it create a new version and copy the packages to the update site upon each commit?

Deploying to github pages is my usual choice, but we can deploy to a particular branch if you prefer, I have code examples for that, e.g. https://github.com/lip6/libDDD/blob/master/.travis.yml#L56-L63 this code deploys to build a branch in https://github.com/lip6/libDDD/tree/osx for MacOS, so this setup not using gh-pages is possible if you prefer it. This gives us addresses for clients with some "raw" in them like you currently use for your update sites e.g. https://github.com/lip6/libDDD/raw/osx/osx.tgz

Thanks for the link. I agree that deploying to gh-pages via travis is pretty convenient. The only concern I have about pages is that existing users may already be using the existing (raw) update site on the master branch. I wouldn't want to cut them off. I'll have a look at the code above.

Thanks for this! I see that you issued a PR and I will check that out ASAP. One question: would building instructions also need to be updated in CONTRIBUTING.md?

Yes I guess they become extremely simplified, the procedure becomes "install Dev feature from update site", no projects or other dependencies need to be listed, since they are packaged along with the dev feature, so then simply clone and import into eclipse the project you want to work with. Maven is reading all it's config from the tycho/eclipse configuration of the plugins, so no need for maven install either.

Sounds great. I suppose this situation is a bit similar to what the Eclipse platform has to deal with themselves. On one hand, you have the Eclipse committers and on the other you have developers writing against the plug-ins and interfaces.

refs

Thanks for pointing these out, I'd not come across them yet.

Np.

[yanntm]

Cool! I didn't know you could do that. Does it create a new version and copy the packages to the update site upon each commit?

So yes the update site is regenerated from scratch at every commit (note that there is not a single JAR file in the repo currently), and I edited my answer slightly so it's clearer that the deployment is really just to github on a particular branch, deploying to branch gh-pages is one possibility that exposes the site on a "github.io" address, but we can deploy to any branch really.

[yanntm]

Hi,
I've been busy with other stuff, but our student's internship is coming to an end, so if the PR will go through it would be better to do it now while I'm still focused on this. Is there anything I can add to make it easier to adopt ? Again, I'm fine with spending a shared screen session explaining how/what the changes are to you, I can configure to make two separate update sites for dev or normal users, maybe update the contributing section, I don't know, just tell me if anything is particularly blocking for you. It's not blocking or anything for us (we are running off my fork), but if you'd like to adopt the changes we should do it soon.
cheers
yann

[khatchad]

Sorry, I haven't had much time to follow up. We are closing our semester here as well. I should have some time later this week to have a look. Thanks again for the contribution!

[yanntm]

Hi, yeah sorry for the poke, there is no real immediate rush, when I said soon I really meant that we should deal with this by end of June, this is indeed a busy period and distant evaluation is a pretty difficult issue (who are we really grading ?), we are all rushed into bizarre situations with the pandemic
best regards
yannn

[yanntm]

Hi,
I see you're looking at that PR, there are a couple things I now do differently with more experience on GH actions + I found some new Tycho tricks I wasn't aware of that ease use of maven dependencies. But the PR should still be good and still be able to build and grab all dependencies. If you do merge I'll try to find the time to update to modern syntax and update target platform to e.g. 2022-03 release.