Payara 5.2022.2+ Update

[poikilotherm]

A new release for Payara has been released: https://github.com/payara/Payara/releases/tag/payara-server-5.2021.6

As Dataverse 5.6 has been released not long ago and many likely didn't update yet, should we change the release notes to guide people to use that version? It makes the logging problem reported by @donsizemore in #8052 obsolete. It also contains some updated dependencies (sec fixes).

Happy to create a PR against develop for the update and another one for the release notes/docs.

WDYT @scolapasta @djbrooke @donsizemore @pdurbin @landreev @kcondon @qqmyers

[donsizemore]

I've been running Dataverse 5.6 on Payara 5.2021.6 on an internal test instance for the past couple days. The only SEVEREs I've hit are the expected entries about EZID and the memory leak on Payara shutdown.

I'm all in favor of security fixes, but am leery of stress on the user community, as in-place Payara upgrades seem to cause problems for community installations. The logging bug is a fairly minor one; I only cobbled the setting into dataverse-ansible because one of Odum's programmers prefers that log format.

Looks like 5.2021.6 adds OIDC Per-session configuration and multitenancy, if this is useful to Dataverse?

[poikilotherm]

Payara 5.2021.7 has been released https://github.com/payara/Payara/releases/tag/payara-server-5.2021.7

It contains a security fix that doesn't apply to us (we do not publish to root context, do we?).

[poikilotherm]

Payara 5.2021.8 has been released https://github.com/payara/Payara/releases/tag/payara-server-5.2021.8

It contains a relevant security fix within a dependency with varying ratings 🤷‍♂️.

(Added info in https://github.com/IQSS/dataverse-security/issues/44)

[pdurbin]

The following is from @qqmyers in #8230 which we are consolidating into this issue.

Update to payara-5.2021.8 with upgrade instructions

At some point we'll want to update to payara 5.2021.8 (or later). This issue is a placeholder for that and a place to document the following so we don't forget when a payara update happens. I think this will require some release notes/upgrade instructions for whatever Dataverse release it is associated with:

It looks like 5.2021.8 includes and update of the H2 database to v 1.4.200 which can expose an issue with older H2 databases, specifically h2database/h2database#2078 . In making updates at QDR, I ran into this and was unable to get Dataverse to restart after an update due to this, seeing stack traces about the EJBTimers that include org.h2.message.DbException 'unable to read at position ...' errors and 'java.lang.IllegalStateException: Unsupported type 17'.

The solution was to remove the contents of <domain>/lib/databases prior to restarting, along with deleting the contents of the <domain>/generated and <domain>/osgi-cache directories. A few notes:

• I did not undeploy Dataverse, just stopped the service/stopped the domain. It's possible that undeploy/redeploy would allow some other way to handle things
• Deleting just the lib/databases dir contents caused errors in stopping the domain as the EJBTimers and org.glassfish.flashlight.impl.provider.FlashlightProbeProviderFactory.unregisterProbeProvider failed to shutdown properly (since their db info was gone). This resulted in failure to restart the domain and left the server with Dataverse not running. Removing the generated and/or osgi-cache directory content and restarting fixed this (presumably the rest of the state of the EJBTimers/Flashlight is cached in those locations.) Restarting after all of those were cleared allowed a good restart.
• I also experimented with removing the <jvm-options>-javaagent:/usr/local/payara5/glassfish/lib/monitor/flashlight-agent.jar</jvm-options> line in domain.xml . As far as I can tell this didn't help but I mention it in case a restart without that line did something important (i.e. along with clearing the dirs).
• At QDR, I noticed a lib/databases/ejbtimer subdirectory along with ejbtimer.lock.db ejbtimer.mv.db and ejbtimer.trace.db files associated with H2 (the last only if there's been an error). Looking a bit, the subdirectory is from the old Derby instance - our instructions to copy the domain forward leave the old unused Derby db on the disk. The fix here to clear lib/databases gets rid of it. (However, new installations may not have the Derby cruft at all).

In terms of instructions, I think we've had releases where clearing the generated/osgi-cache dirs has been mentioned. I think this just adds one more (lib/databases).

[donsizemore]

I just upgraded a test instance from 5.2021.8 to 5.2021.9 and Jim's findings hold true: one must remove lib/databases for Payara to return on launch.

[poikilotherm]

In the meantime, Payara 5.2021.9 and 5.2021.10 have been released.

[poikilotherm]

Also note that around Q2/2022 Payara 6 is the only supported community version - 5 will be enterprise only. See #8305

[djbrooke]

Hey @poikilotherm - if you'd be willing to create a PR for this upgrade, that would be great. If not, we'll bring it into a sprint in the near future. Thanks!

[djbrooke]

• Plan is to move to 5.2021.X in this issue and to 6 in Q2/2022.
• Some concern around the quick turnaround from 5 to 6 and the end of community support as soon as 6 is released (or at least so close to the release of version 6).

[poikilotherm]

Well we could wait for 6. We should upgrade to Jakarta EE 9/10 and Java 17 #8094 and test on 6 Beta. Starting this now might give a benefit. Depends on how likely it is to have a critical sec issue in one of the next 5 releases.

Not sure how many releases of 5 we will see before the switch to enterprise only...

Could be Dataverse 6 then 🙈

(Sorry for not having created a PR, quite busy with concept paper for HERMES right now...)

[qqmyers]

FWIW: Discussion in slack today about UVA updating payara to 5.2021.10 and hitting what so far looks like the issue above (from Nov 10th). Whether we require a newer version or not, we may want to put info in the release notes(or elsewhere) about this since payara makes it easiest to get the latest version and harder to dig for 5.2021.7, etc.

[pdurbin]

S3 logging behavior may come into play: 5.8 on S3 storage: thumbnails anger the S3 Library #8219. Hopefully the upgrade will fix it.

[poikilotherm]

It is likely that Payara 5.2022.2 will include a bug fix for payara/Payara#5322

This has been fixed a few days ago, so this fix is hopefully in one of the last releases we will see from Payara 5 community.

I will definitely update containers to use this newer version.

[scolapasta]

Closing this, as will focus energy on upgrading directly to Payara 6: #8305