If jetty.org is down, impossible to start Jetty server (12.0.5) - multiple dtd to eclipse and jetty

[ArfyFR]

Jetty version(s)
jetty-12.0.5

Jetty Environment
ee10

Java version/vendor (use: java -version)
openjdk 17.0.9 2023-10-17

OS type/version
Linux or Windows

Description
xml in webapps, like

<?xml version="1.0"  encoding="UTF-8"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "https://jetty.org/configure_10_0.dtd">
<Configure class="org.eclipse.jetty.server.handler.ContextHandler">
...

While https://jetty.org/ is down, I got error

2024-10-18 10:41:07.163 [main] WARN  o.e.jetty.deploy.DeploymentManager - Unable to reach node goal: started
java.net.SocketException: Unexpected end of file from server
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:954)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:761)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:951)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:761)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1688)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:677)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1397)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1363)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:257)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1152)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1040)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:943)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:542)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:889)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:825)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224)
	at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:637)
	at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
	at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:254)
	at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:303)
	at org.eclipse.jetty.xml.XmlConfiguration.<init>(XmlConfiguration.java:278)
	at org.eclipse.jetty.deploy.providers.ContextProvider$1.<init>(ContextProvider.java:350)
	at org.eclipse.jetty.deploy.providers.ContextProvider.createContextHandler(ContextProvider.java:349)
	at org.eclipse.jetty.deploy.App.getContextHandler(App.java:108)
	at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:33)
	at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:183)
	at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:522)
	at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:162)
	at org.eclipse.jetty.deploy.providers.ScanningAppProvider.pathAdded(ScanningAppProvider.java:293)
	at org.eclipse.jetty.deploy.providers.ContextProvider.pathAdded(ContextProvider.java:520)
	at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.pathAdded(ScanningAppProvider.java:69)
	at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:902)
	at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:868)
	at org.eclipse.jetty.util.Scanner.scan(Scanner.java:775)
	at org.eclipse.jetty.util.Scanner.startScanning(Scanner.java:664)
	at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:636)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
	at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:264)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:592)
	at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:263)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.server.Server.start(Server.java:622)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
	at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:468)
	at org.eclipse.jetty.server.Server.doStart(Server.java:563)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:2030)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
	at org.eclipse.jetty.start.Main.start(Main.java:519)
	at org.eclipse.jetty.start.Main.main(Main.java:76)
2024-10-18 10:41:07.184 [main] WARN  o.eclipse.jetty.xml.XmlConfiguration - Unable to execute XmlConfiguration
java.net.SocketException: Unexpected end of file from server
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:954)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:761)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:951)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:761)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1688)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:677)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1397)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1363)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:257)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1152)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1040)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:943)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:542)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:889)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:825)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224)
	at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:637)
	at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
	at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:254)
	at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:303)
	at org.eclipse.jetty.xml.XmlConfiguration.<init>(XmlConfiguration.java:278)
	at org.eclipse.jetty.deploy.providers.ContextProvider$1.<init>(ContextProvider.java:350)
	at org.eclipse.jetty.deploy.providers.ContextProvider.createContextHandler(ContextProvider.java:349)
	at org.eclipse.jetty.deploy.App.getContextHandler(App.java:108)
	at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:33)
	at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:183)
	at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:522)
	at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:162)
	at org.eclipse.jetty.deploy.providers.ScanningAppProvider.pathAdded(ScanningAppProvider.java:293)
	at org.eclipse.jetty.deploy.providers.ContextProvider.pathAdded(ContextProvider.java:520)
	at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.pathAdded(ScanningAppProvider.java:69)
	at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:902)
	at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:868)
	at org.eclipse.jetty.util.Scanner.scan(Scanner.java:775)
	at org.eclipse.jetty.util.Scanner.startScanning(Scanner.java:664)
	at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:636)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
	at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:264)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:592)
	at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:263)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.server.Server.start(Server.java:622)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
	at org.eclipse.jetty.server.Handler$Abstract.doStart(Handler.java:468)
	at org.eclipse.jetty.server.Server.doStart(Server.java:563)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:2030)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:221)
	at org.eclipse.jetty.start.Main.start(Main.java:519)
	at org.eclipse.jetty.start.Main.main(Main.java:76)

How to reproduce?
Shutdown jetty.org ? ;)

[ArfyFR]

Note: XML in webapps with links to eclipse.dev

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "https://eclipse.dev/jetty/configure_10_0.dtd">

was working (after removing those with https://jetty.org/configure_10_0.dtd)

[joakime]

The existence or not of jetty.org (or any of the other valid hostnames in the DTD references) shouldn't matter.
Those should be mapped properly for local discovery via the XML catalog present in Jetty.

I'll test this out and see if a fix is necessary for jetty-12.0.x HEAD.

Btw, you should upgrade to Jetty 12.0.14, you are missing several fixes for various security advisories.

[ArfyFR]

The XML which failed, was a "org.eclipse.jetty.fcgi.proxy.FastCGIProxyHandler" one:

<?xml version="1.0"  encoding="UTF-8"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "https://jetty.org/configure_10_0.dtd">
<Configure class="org.eclipse.jetty.server.handler.ContextHandler">

  <Set name="contextPath">/fastcgi</Set>

      <Set name="handler">
        <New class="org.eclipse.jetty.server.handler.PathMappingsHandler">
          <Call name="addMapping">
            <Arg>
              <New class="org.eclipse.jetty.http.pathmap.ServletPathSpec"><Arg>/</Arg></New>
            </Arg>
            <Arg>
              <New class="org.eclipse.jetty.fcgi.proxy.FastCGIProxyHandler">
                <Arg>(http?)://([^/]+)(/fastcgi)(.*)</Arg>
                <Arg>http://localhost:7899$4</Arg>
                <Arg>/gtc</Arg>
                <Set name="scriptPattern"><Call class="java.util.regex.Pattern" name="compile"><Arg>()(/?.+)$</Arg></Call></Set>
                <Set name="fastCGISecure">false</Set>
              </New>
            </Arg>
          </Call>
        </New>
      </Set>

</Configure>
...

it may perhaps help you to reproduce

[joakime]

PUBLIC "-//Mort Bay Consulting//DTD Configure//EN"

That looks wrong, I need to double check.

[ArfyFR]

PUBLIC "-//Mort Bay Consulting//DTD Configure//EN"

That looks wrong, need to double check.

It comes from here: https://jetty.org/docs/jetty/12/operations-guide/protocols/index.html#fcgi-configure
I wrote my fastcgi xml from the Wordpress example there

[joakime]

It comes from here: https://jetty.org/docs/jetty/12/operations-guide/protocols/index.html#fcgi-configure
I wrote my fastcgi xml from the Wordpress example there

That PUBLIC definition isn't registered in the xml catalog.
I'll go add them.
That is the likely cause of the issue, not the URI/URL.

It should read ...

<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "https://jetty.org/configure_10_0.dtd">

[ArfyFR]

This URI/URL worked for months ... just not while jetty.org was down

If I change in the dtd jetty.org to jetty.or, removing the 'g', I can reproduce the problem. I mean like that:

<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "https://jetty.or/configure_10_0.dtd">

[joakime]

This URI/URL worked for months ... just not while jetty.org was down

If I change in the dtd jetty.org to jetty.or, removing the 'g', I can reproduce the problem. I mean like that:

<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "https://jetty.or/configure_10_0.dtd">

Removing the g in org will make that a different System ID that wont match the XML catalog.

https://github.com/jetty/jetty.project/blob/jetty-12.0.x/jetty-core/jetty-xml/src/main/resources/org/eclipse/jetty/xml/catalog-configure.xml

Can you run this and report back the output?

$ cd /path/to/my-jetty-base
$ java -jar /opt/jetty-home/start.jar --list-config

Feel free to overwrite sensitive things with XXXXX just don't delete any of the lines in the output.
I want to confirm your environment setup before I implement a possible solution.

[ArfyFR]

Enabled Modules:
----------------

JVM Version & Properties:
-------------------------
 java.home = C:\tools\64bits\java\jdk-17.0.9+9
 java.vm.vendor = Eclipse Adoptium
 java.vm.version = 17.0.9+9
 java.vm.name = OpenJDK 64-Bit Server VM
 java.vm.info = mixed mode, sharing
 java.runtime.name = OpenJDK Runtime Environment
 java.runtime.version = 17.0.9+9
 java.io.tmpdir = C:\Users\borsc\AppData\Local\Temp\
 user.dir = C:\DEV\GIT\gtc
 user.language = fr
 user.country = FR

Jetty Version & Properties:
---------------------------
 jetty.version = 12.0.5
 jetty.tag.version = jetty-12.0.5
 jetty.build = 3aed62e4959bb8c01f5975fe81e078e3ff626126
 jetty.home = C:\tools\noarch\jetty\12.0.5
 jetty.base = C:\DEV\GIT\gtc

Config Search Order:
--------------------
 <command-line>
 ${jetty.base} -> C:\DEV\GIT\gtc
 ${jetty.home} -> C:\tools\noarch\jetty\12.0.5

System Properties:
------------------
 (no system properties specified)

Properties: Jetty
-----------------
 java.version = 17.0.9
 java.version.major = 17
 java.version.micro = 9
 java.version.minor = 0
 java.version.platform = 17
 jetty.base = C:\DEV\GIT\gtc
 jetty.base.uri = file:///C:/DEV/GIT/gtc
 jetty.home = C:\tools\noarch\jetty\12.0.5
 jetty.home.uri = file:///C:/tools/noarch/jetty/12.0.5
 runtime.feature.alpn = true

Classpath: Jetty
----------------
No classpath entries and/or version information available show.

Active XMLs: Jetty
------------------
 (no xml files specified)

[joakime]

You have nothing configured in your ${jetty.base} ?

Are you using the jetty-home and jetty-base split?
Or something else?

Your stacktrace shows that you are using Jetty's start.jar, so I'm a bit confused now.

org.eclipse.jetty.start.Main.main(Main.java:76)