Upgrade Scala version to be compatible with JDK 8

[AlexanderKaraberov]

Currently specified version is <scala.version>2.9.1</scala.version> which is quite old and is not compatible with JDK 8:

[ERROR] error: error while loading AnnotatedElement, class file '/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)' is broken
[INFO] (bad constant pool tag 18 at byte 76)
[ERROR] error: error while loading CharSequence, class file '/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken
[INFO] (bad constant pool tag 18 at byte 10)

I checked Cloudant Maven repo and it doesn't contain newer versions. Is it possible to upgrade Scala to at least 2.10.2 or 2.11 ?

[jiangphcn]

It is possible, but it is not just to modify one configurable entry. We need to modify scala-lang and its dependency, such as overlock, etc, and also consider to upgrade approach of logging and metrics.

Some preliminary work was done and the result is promising.

https://github.com/boundary/overlock/compare/master...jiangphcn:support-scala-2.11?expand=1
https://github.com/boundary/scalang/compare/master...cloudant:scalang-support-scala-2.11?expand=1
https://github.com/cloudant/clouseau/compare/support-scala-2.11?expand=1

[arturog]

I also need to bring clouseau to a recent JDK. Last I looked we needed to remove a logging library from boundary, update scalang and overlock (also from boundary) and get everything to build with latest JDK. IMO we must fork these dependencies. Are we the only ones using them? o_O

CouchDB has decided to move to FoundationDB:
https://lists.apache.org/thread.html/99c1a51918818d23a0049883c1233c4b0e76c0a71fba37c37f572e61@%3Cdev.couchdb.apache.org%3E

I have a question for cloudant:
When are we expecting to complete our move to CouchDB + FoundationDB combo? 1month, 1year, 5years, 10years? It would be great to have a rough idea. We also needed to update the version of Lucene, but, is it worth the effort? If FoundationDB is still some time away I would be happy to start working on clouseau / scalang.

[jiangphcn]

Hi @arturog To my understanding, CouchDB 3.0 still uses dreyfus + clouseau while CouchDB 4.0 will be based on CouchDB + FoundationDB combo. They will co-exist over times. If you plan to officially update the version of Lucene, it will give benefit to CouchDB 3.0. I am doing similar thing to update spidermonkey version. To me, I am glad to see that you start working on clouseau/scalang. I am able of assisting you because you can see above commits I worked on.

[arturog]

Hi @jiangphcn, thanks for the info, that's very useful information.

Some of this update work was already done a couple of years ago:
https://github.com/dmunch/clouseau

I think we can cherry-pick some of the commits there and bring the forks from cloudant up to speed. Once scala and jdk are updated, then we can try updating Lucene.

[arturog]

Hello again @jiangphcn, some good news. I have successfully move forward overlock and scalang from cloudant/boundary to scala_2.13. If you could start looking at the code and give some feedback that would be great. I still have lots of deprecations and warnings but I feel I've made some progress. Most deprecations left are around Symbol so I'm not too worried about. Please have a look at the tests and if you can make them pass that would be a huge step forward.

• NOTE!!! I will rebase these branches between today and tomorrow so please do not work on top of them just yet! Just clone and have a look for now. Thanks!

Instructions:

mkdir /tmp/scalang_2.13
cd /tmp/scalang_2.13
git clone https://github.com/arturog/overlock.git
cd overlock
mvn install:install-file -Dfile=./target/overlock-scala_2.13-0.9.0-SNAPSHOT.jar -DpomFile=./pom.xml

cd /tmp/scalang_2.13
git clone https://github.com/arturog/scalang.git
git checkout scala-2-13
mvn verify

Let me know what you think. Next steps would be:

1. Remove Symbol and all deprecations
2. Rebase and cleanup the branches
3. Set versions for scalang and overlock to 1.0.0-SNAPSHOT, ideally under cloudant (please confirm we're happy to do this). Bye bye boundary.
4. Update clouseau
5. (for me) Release on a new system I have, ideally in the next week or so :-O!

Once this is working 100% I'll start updating Lucene

[jiangphcn]

Great process! I will give comments after some check.

[arturog]

Great, thanks! :-) After looking at the code, we need to have a chat about the use of "Symbol" in scalang - I went ahead and removed it in the latest commit. I feel in this case it was the right thing to do, but again I'm not a Scala expert.

(...clouseau will follow)

[arturog]

Hello @jiangphcn, I have updated clouseau and I can run it with scala_2_13. But I am now stuck and really need some guidance. I have a couple of tests that don't pass but I think it's a "Scala" thing - the code seems to be OK.

I am now stuck and really need a bit of guidance to get this done. Can you give it a quick test on your end?

https://github.com/arturog/clouseau

Instructions (after installing overlock and scalang above)

cd /tmp
git clone https://github.com/arturog/clouseau
cd clouseau
git checkout scala-2-13
mvn scala:run

Thanks!

[arturog]

I reverted the removal of "Symbol", I misunderstood that we were translating them to Erlang terms. The three branches above (overlock, scalang and clouseau) are now working with couchdb-master as of today. Also, I pushed --force (and will do so again).

[arturog]

OK, so where do we stand. I have updated dependencies to get couchdb-master + clouseu + dreyfus working with scala 2.13 and newer JDK. I have a few questions and comments though, it would be great to get some information on the below:

Clouseau

• A big patch applied in May (00a1e51) nuked some code (A tiny patch I did years ago). (BAD?)
• Clouseau has no version tags - How do I know what commit to use with dreyfus? Are we always working on master?
• Is cloudant using the https://github.com/cloudant-labs/clouseau repo for their own deployments? do I have the latest code?
• Clouseau refers to partitioned dbs, which makes me think it is expecting a couchdb running from master. Can someone clarify what commit is stable for the combo couchdb-dreyfus-clouseau?

=> Main question: What commit is the working, stable version of clouseau that I should branch from to apply all scala-2.13 and Lucene upgrades?

Dreyfus

• Dreyfus master clearly doesn't apply to stable couchdb-2.3.1. What commit of dreyfus corresponds to stable-couchdb-2.3.1?
• Dreyfus has seen many patches in the past year, but they are all incompatible with couchdb-2.3.1. Are these expecting master to be used?
• Dreyfus has a branch 'dreyfus-new-and-old-clouseau'. Is clouseau from github "old-clouseau"? This one left me a bit astonished at first... Is there another clouseau?

=> Main question: What commit of dreyfus should we use?

CouchDB
=> Main question: What commit is the stable couchdb to work with all the above?

I personally want to stick to a reliable couchdb-dreyfus-clouseau combo as I myself run patched versions of CouchDB-2.3.1. I would be happy to move forward our patches to master but without tags and knowing what is stable we are in the dark. CouchDB-2.3.1 removed some functionality we used (starting processes via the config), which was a bit annoying. We shrugged it off and moved to couchdb-2.3.1 but I feel we are again at square one branching from the "stable" couchdb-2.3.1. Also, looking at commits and future plans, CouchDB seems very keen on dropping features. Committing to "stable" releases seems to leave you on a path that is already deprecated (what happened from 2.2 to 2.3).

Thanks!

Arturo

[jiangphcn]

@arturog, my original plan is to give a try over weekend due to other project commitment recently. To your questions, my general answer is that the master in every repository is preferred base branch to continue.

Right now, we are using

• clouseau: https://github.com/cloudant-labs/clouseau
• dreyfus: https://github.com/apache/couchdb/tree/master/src/dreyfus
• couchdb: https://github.com/apache/couchdb/

above is being applied to cloudant.

Regarding to your questions:

• Clouseau

  A big patch applied in May (00a1e51) nuked some code (A tiny patch I did years ago). (BAD?)
  Clouseau has no version tags - How do I know what commit to use with dreyfus? Are we always working on master?

[Peng Hui]: we should consider master
>Is cloudant using the https://github.com/cloudant-labs/clouseau repo for their own deployments? do I have the latest code?

[Peng Hui]: yes, please use this repository to get latest codes
>Clouseau refers to partitioned dbs, which makes me think it is expecting a couchdb running from master. Can someone clarify what commit is stable for the combo couchdb-dreyfus-clouseau?

[Peng Hui]: j8013 in cloudant is using 6c35acd8af3dad40f932ca179dbcbf0d5428498e from apache/couchdb to get codes for dreyfys. The clouseau is master in cloudant-labs/clouseau

=> Main question: What commit is the working, stable version of clouseau that I should branch from to apply all scala-2.13 and Lucene upgrades?

[Peng Hui]: in general, we should start from master plus the change for scalang

Dreyfus

Dreyfus master clearly doesn't apply to stable couchdb-2.3.1. What commit of dreyfus corresponds to stable-couchdb-2.3.1?
Dreyfus has seen many patches in the past year, but they are all incompatible with couchdb-2.3.1. Are these expecting master to be used?
Dreyfus has a branch 'dreyfus-new-and-old-clouseau'. Is clouseau from github "old-clouseau"? This one left me a bit astonished at first... Is there another clouseau?

=> Main question: What commit of dreyfus should we use?
=> Main question: What commit is the stable couchdb to work with all the above?

[Peng Hui]: please consider https://github.com/apache/couchdb/tree/master/src/dreyfus
CouchDB

[curlycabbage]

@arturog, if there's anything this java noob can do to help, please let me know. I am trying to get clouseau running on a raspberry pi cluster, which is quite challenging with the java 1.6 dependency.

I have tried following your build steps, starting with overlock, and I am stuck on this error:

mvn install:install-file -Dfile=./target/overlock-scala_2.13-0.9.0-SNAPSHOT.jar -DpomFile=./pom.xml
... [snip] ....
[ERROR] The specified file '/tmp/scalang_2.13/overlock/target/overlock-scala_2.13-0.9.0-SNAPSHOT.jar' not exists

If you enable issues in your overlock or clouseau repos, I'll repost it there so we don't add more noise to this thread. Thanks!

[arturog]

Hi, basically you need to build overlock first, add it to your local repo and then build closeau. I am away from my laptop at the moment but will post some.more help. Cheers A.

…

On 23 November 2019 17:55:54 GMT+00:00, Peter Radocchia ***@***.***> wrote: @arturog, if there's anything this java noob can do to help, please let me know. I am trying to get clouseau running on a raspberry pi cluster, which is quite challenging with the java 1.6 dependency. I have tried following your build steps, starting with overlock, and I am stuck on this error: ``` mvn install:install-file -Dfile=./target/overlock-scala_2.13-0.9.0-SNAPSHOT.jar -DpomFile=./pom.xml ... [snip] .... [ERROR] The specified file '/tmp/scalang_2.13/overlock/target/overlock-scala_2.13-0.9.0-SNAPSHOT.jar' not exists ``` If you enable issues your overlock or clouseau repos, I'll repost it there so we don't add more noise to this thread. Thanks! -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #22 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[arturog]

@curlycabbage we can bring your issue over here: arturog/scalang#1