OpenGrok home page not refreshing when index updated (NEW)

[ilovemaui]

Hi, I noticed this issue with the new version of OpenGrok (0.13-rc4). This seems to be different from
#1175

When I create the index, everything seems to be created ok, and the data appears to be correct, except that the timestamp doesn't appear to update unless I bounce the index.
I am running on Linux Hardware, Redhat version 6.6
Using OpenGrok version 0.13-rc4
Universal CTAGS version 0.0.0
JAVA Version - 1.8.0_72
apache-tomcat version 8.5.4

Let me know if there is any other data you need. If you can figure out a fix quickly, that would be most appreciated. thanks.

[vladak]

Marking as stopper since it seems to be introduced by recent changes.

[tulinkry]

I guess your repository is not public so I can not easily reproduce it...? That would really help.

Have you read the flow in my comment in #1175? Does it apply to you?

Can you please check the logs? Looking for some information:

1. is there any addFile or removeFile call? (if not, the timestamp is not updated)
2. these log messages should not be present (dirty)
3. these log messages should not be present (timestamp)
4. can you check that this code is called?
5. these log messages should not be present (send conf)
6. is this message present in tomcat log:

[ilovemaui]

hi Krystof, so to answer your questions.

1. yes, files updated
2. not seeing that message.
3. not present
4. how to check if that code is called?
5. not present
6. not present

fyi, the timestamp file is updated.
here is current:
$ ls -l timestamp
-rw-r--r-- 1 test test 0 Nov 21 16:33 timestamp

index $ ls -l
total 60485916
-rw-r--r-- 1 robotest general 18265382995 Nov 21 16:14 _c8yl.fdt
-rw-r--r-- 1 test test 4794192 Nov 21 16:14 _c8yl.fdx
-rw-r--r-- 1 test test 1145 Nov 21 16:32 _c8yl.fnm
-rw-r--r-- 1 test test 38642033 Nov 21 16:32 _c8yl.nvd
-rw-r--r-- 1 test test 158 Nov 21 16:32 _c8yl.nvm
-rw-r--r-- 1 test test 579 Nov 21 16:32 _c8yl.si
-rw-r--r-- 1 test test 5945897247 Nov 21 16:32 _c8yl_Lucene50_0.doc
-rw-r--r-- 1 test test 32999114719 Nov 21 16:32 _c8yl_Lucene50_0.pos
-rw-r--r-- 1 test test 4392903039 Nov 21 16:32 _c8yl_Lucene50_0.tim
-rw-r--r-- 1 test test 73532490 Nov 21 16:32 _c8yl_Lucene50_0.tip
-rw-r--r-- 1 test test 217232818 Nov 21 16:32 _c8yl_Lucene54_0.dvd
-rw-r--r-- 1 test test 256 Nov 21 16:32 _c8yl_Lucene54_0.dvm
-rw-r--r-- 1 test test 142 Nov 21 16:33 segments_elus

Let me know if there is any other info you need.

[tulinkry]

According to what you wrote the problem is between these steps:
5. these log messages should not be present (send conf) - this is ok
6. is this message present in tomcat log: - this is not ok (must be present in tomcat log)

Which means that the webapp doesn't get any configuration updates. (As you would notice the timestamp forced update is just 2 lines above). If you see Configuration update routine done, check log output for errors. in the logs after the index then you can be sure that the indexer also tried to send the new configuration to the webapp.

If there is none of the error messages in 5., that means that the indexer successfully sent the configuration out. But it has never been received by the webapp. I don't know where to start, try to analyze:

1. Webapp listens on some port (set in web.xml) and the indexer sends the configuration to the very same port (set on each run from env variable) (default 2424). (However this should fail with an exception in 5.)
2. [Available only in >=RC4] You can try the new messaging addition #1195 messages addition to see if you can reach the webapp (it uses the same listener on the same port) - the basic usage is that it displays a message on the front page (wiki + examples)
3. Network connectivity - can the indexer reach it? can your computer reach the webapp on the 2424 port? (traceroute)
4. Network traffic - where the outgoing packets go and why they don't reach the destination (wireshark for ex.)

[ilovemaui]

something else I noticed.. so when I start up apache-tomcat I see this warning.. actually happens 3 times in a startup, which I did with a small test case..
29-Nov-2016 11:01:56.979 WARNING [localhost-startStop-1] org.opensolaris.opengrok.configuration.RuntimeEnvironment.startConfigurationListenerThread I/O error when waiting for config:
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.bind(ServerSocket.java:329)
at org.opensolaris.opengrok.configuration.RuntimeEnvironment.startConfigurationListenerThread(RuntimeEnvironment.java:1430)
at org.opensolaris.opengrok.web.WebappListener.contextInitialized(WebappListener.java:86)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5178)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:152)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1403)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

29-Nov-2016 11:01:56.980 SEVERE [localhost-startStop-1] org.opensolaris.opengrok.web.WebappListener.contextInitialized OpenGrok: Failed to start configuration listener thread
29-Nov-2016 11:01:57.433 WARNING [localhost-startStop-1] org.opensolaris.opengrok.configuration.RuntimeEnvironment.startConfigurationListenerThread I/O error when waiting for config:
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.bind(ServerSocket.java:329)
at org.opensolaris.opengrok.configuration.RuntimeEnvironment.startConfigurationListenerThread(RuntimeEnvironment.java:1430)
at org.opensolaris.opengrok.web.WebappListener.contextInitialized(WebappListener.java:86)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5178)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:152)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1403)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

29-Nov-2016 11:01:57.433 SEVERE [localhost-startStop-1] org.opensolaris.opengrok.web.WebappListener.contextInitialized OpenGrok: Failed to start configuration listener thread
29-Nov-2016 11:01:57.881 WARNING [localhost-startStop-1] org.opensolaris.opengrok.configuration.RuntimeEnvironment.startConfigurationListenerThread I/O error when waiting for config:
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.bind(ServerSocket.java:329)
at org.opensolaris.opengrok.configuration.RuntimeEnvironment.startConfigurationListenerThread(RuntimeEnvironment.java:1430)
at org.opensolaris.opengrok.web.WebappListener.contextInitialized(WebappListener.java:86)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5178)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:152)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1403)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

29-Nov-2016 11:01:57.882 SEVERE [localhost-startStop-1] org.opensolaris.opengrok.web.WebappListener.contextInitialized OpenGrok: Failed to start configuration listener thread

Also, here is some info from the apache-tomcat conf/server.xml file

<!--The connectors can use a shared executor, you can define one or more named thread pools-->
<!--
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
    maxThreads="150" minSpareThreads="4"/>
-->


<!-- A "Connector" represents an endpoint by which requests are received
     and responses are returned. Documentation at :
     Java HTTP Connector: /docs/config/http.html
     Java AJP  Connector: /docs/config/ajp.html
     APR (HTTP/AJP) Connector: /docs/apr.html
     Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
-->
<Connector port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           maxHttpHeaderSize="65536"
           redirectPort="8444" />
<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
           port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           maxHttpHeaderSize="65536"
           redirectPort="8444" />
-->
<!-- Define a SSL/TLS HTTP/1.1 Connector on port 8444
     This connector uses the NIO implementation with the JSSE engine. When
     using the JSSE engine, the JSSE configuration attributes must be used.
-->
<!--
<Connector port="8444" protocol="org.apache.coyote.http11.Http11NioProtocol"
           maxThreads="150" SSLEnabled="true">
    <SSLHostConfig>
        <Certificate certificateKeystoreFile="conf/localhost-rsa.jks"
                     type="RSA" />
    </SSLHostConfig>
</Connector>
-->
<!-- Define a SSL/TLS HTTP/1.1 Connector on port 8444 with HTTP/2
     This connector uses the APR/native implementation. When using the
     APR/native implementation or the OpenSSL engine with NIO or NIO2 then
     the OpenSSL configuration attributes must be used.
-->
<!--
<Connector port="8444" protocol="org.apache.coyote.http11.Http11AprProtocol"
           maxThreads="150" SSLEnabled="true" >
    <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
    <SSLHostConfig>
        <Certificate certificateKeyFile="conf/localhost-rsa-key.pem"
                     certificateFile="conf/localhost-rsa-cert.pem"
                     certificateChainFile="conf/localhost-rsa-chain.pem"
                     type="RSA" />
    </SSLHostConfig>
</Connector>
-->
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8008" protocol="AJP/1.3" redirectPort="8444" />


<!-- An Engine represents the entry point (within Catalina) that processes
     every request.  The Engine implementation for Tomcat stand alone
     analyzes the HTTP headers included with the request, and passes them
     on to the appropriate Host (virtual host).
     Documentation at /docs/config/engine.html -->

<!-- You should set jvmRoute to support load-balancing via AJP ie :
<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
-->
<Engine name="Catalina" defaultHost="localhost">

  <!--For clustering, please take a look at documentation at:
      /docs/cluster-howto.html  (simple how to)
      /docs/config/cluster.html (reference documentation) -->
  <!--
  <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
  -->

  <!-- Use the LockOutRealm to prevent attempts to guess user passwords
       via a brute-force attack -->
  <Realm className="org.apache.catalina.realm.LockOutRealm">
    <!-- This Realm uses the UserDatabase configured in the global JNDI
         resources under the key "UserDatabase".  Any edits
         that are performed against this UserDatabase are immediately
         available for use by the Realm.  -->
    <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
           resourceName="UserDatabase"/>
  </Realm>

  <Host name="localhost"  appBase="webapps"
        unpackWARs="true" autoDeploy="true">

    <!-- SingleSignOn valve, share authentication between web applications
         Documentation at: /docs/config/valve.html -->
    <!--
    <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
    -->

    <!-- Access log processes all example.
         Documentation at: /docs/config/valve.html
         Note: The pattern used is equivalent to using pattern="common" -->
    <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
           prefix="localhost_access_log" suffix=".txt"
           pattern="%h %l %u %t &quot;%r&quot; %s %b" />

so not sure if there is anything here that could be amiss?

[tulinkry]

This java.net.BindException: Address already in use means that your application couldn't start a socket on 2424 port (because there is already another one). This all explains why the timestamp is not updated - the configuration is not sent to your current application.

Possible cause:

1. (it happens to me also) Sometimes when I redeploy application the tomcat does not correctly finish the old listener (so it is still active while not being useful). Try investigate open ports on your machine and kill the process which holds the socket (if not the case 2. - read next) and restart tomcat

$ lsof -i

2. You have more than one instance of opengrok running at the same time (but not by an accident) and they are not configured properly for the configuration updates, which is this env variable set to localhost:<correct port number> while doing OpenGrok deploy and OpenGrok index. Restart tomcat.

3. You have more than one instance of opengrok running at the same time by an accident. Investigate the tomcat webapps directory for anything suspicious and then restart tomcat.

[ilovemaui]

Still not solving this. I have 2 versions running, but on 2 different ports, 8080, and 58080, but even if I only use one instance, I still get this error. So I guess I am still not sure what I am doing wrong. I will pose a couple of questions?

• in the indexer, so I need to specify anything specific, like with -U option? (currently I set nothing)
• in source.war - WEB-INF/web.xml - I specify localhost:2424 (is this correct)?
• i tried setting this in my env: export OPENGROK_WEBAPP_CFGADDR=localhost:2424 (doesn't help)
• in apache conf/server.xml - set this:
  
  and:
  

I cancel the current instance, and verify by making sure it isn't running. then every time I start up, I get the errors listed in previous runs.. eg. org.opensolaris.opengrok.configuration.RuntimeEnvironment.startConfigurationListenerThread I/O error when waiting for config:

So not sure if this is related to my timestamp issue, or something I am just configuring wrong. last version of Opengrok I ran (where I didn't see this issue, is both 0.9, and 0.11.1)

So anyone that help me figure out what to set would be lifesaver! :-)

[tulinkry]

The scheme is like this for two instances:

tomcat
   |---- 1. instance (serving port 8080)
   |        - this instance starts a configuration listener (a special thread with open socket)
   |          - by default on port 2424 (so localhost:2424)
   |---- 2. instance (serving port 58080)
           - this instance tries to start a configuration listener (a special thread with open socket)
              - by default on port 2424 (so localhost:2424) <= fails binding the socket

1. When you use indexer, it by default sends the results (configuration update) to the localhost:2424 so eventually both of your indexers sends the update to the first instance. Therefore you have to set the -U option for the second instance to localhost:2425 (or whatever port you use in step 2).

2. The webapplication must be configured in the same way. Both instances can't have the same portnumber. Set the second to localhost:2425 (or whatever port you want)

3. I told you to set the env variable OPENGROK_WEBAPP_CFGADDR however that only takes an effect when you use the opengrok shell wrapper. It does nothing when you use the opengrok.jar file directly.

The bind exception you got is there because the second instance would like to start the listener on port 2424 but it can't because it's already used by the first instance.

Also the timestamp is not updated in the second instance because the message most probably goes to the first instance.

Do not forget to kill all applications running on the 2424 port and restart tomcat after you make some changes.

[ilovemaui]

Thank you. I have gotten this to work! I didn't realize some of the new features of OpenGrok that I was never using before. :-D

[tulinkry]

Caused by misconfiguration.

Glad to hear this :)