OpenGrok home page doesn't show projects in group during configuration upload

[danielviorreta]

Describe the bug
The home page is not updated with the last index timestamp using the -U param in the indexer or after calling the configuration api method using the webapp context (http://localhost:8080/source).

To Reproduce
Calling the api method:

curl -X PUT -H "Content-Type: application/xml"  --data "@/var/opengrok/etc/configuration.xml" http://localhost:8080/source/api/v1/configuration?reindex=true

produces the next output in tomcat:

20-Apr-2020 09:58:13.240 INFO [http-nio-127.0.0.1-8080-exec-5] org.opengrok.indexer.util.Statistics.report done invalidating 4119 repositories (took 52.275 seconds)
20-Apr-2020 10:00:30.336 INFO [http-nio-127.0.0.1-8080-exec-5] org.opengrok.indexer.configuration.RuntimeEnvironment.applyConfig Configuration updated
20-Apr-2020 10:00:30.336 INFO [http-nio-127.0.0.1-8080-exec-5] org.opengrok.indexer.configuration.WatchDogService.stop Watchdog stoped
20-Apr-2020 10:00:30.336 WARNING [http-nio-127.0.0.1-8080-exec-5] org.opengrok.indexer.framework.PluginFramework.reload Plugin directory not found or not readable: /var/opengrok/data/../plugins. All requests allowed.
20-Apr-2020 10:00:30.336 INFO [http-nio-127.0.0.1-8080-exec-5] org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.initSuggester Suggester disabled

But the home page is not refreshed with the date of the timestamp file. The date of the timestamp file in the DATA directory is properly updated. Same effect adding the -U param in the indexer.

The home page is properly updated after calling the same api method but without the context (source). But the page doesn't show any project during the api call:
curl -X PUT -H "Content-Type: application/xml" --data "@/var/opengrok/etc/configuration.xml" http://localhost:8080//api/v1/configuration?reindex=true

[vladak]

Do you use per project reindex by a chance ?

[vladak]

I am asking because my workflow involves running indexer for each project in parallel which avoids to set the configuration at the end of the reindex (see https://github.com/oracle/opengrok/wiki/Per-project-management-and-workflow). Instead the timestamp file is touched once the project indexing is over and that works (#1670).

[danielviorreta]

I index all the data at once.
Is there any way to force the timestamp to be reloaded as you suggested in
#1670 (comment)? It looks like the /opengrok/dist/bin/sync.py doesn't accept the -p param anymore.

[vladak]

The timestamp file's modified time stamp is only updated during reindex in IndexDatabase#update() via env.setIndexTimestamp(). The configuration controller calls just env.applyConfig() that refreshes the timestamp in memory via refreshDateForLastIndexRun() by stat'ing the file. Updating the configuration via API cannot change the timestamp of the file.

As for sending the API call to the app server without context I don't see how this could work without also deploying the app to ROOT.

[vladak]

Returning the the original problem (the timestamp file is touched however the changes do not manifest in the UI): when running the indexer with -U, the configuration is sent to the webapp at the end of the indexing using the configuration API end point. This in turn refreshes the time stamp.
Then, foot.jspf calls cfg.getEnv().getDateForLastIndexRun() to retrieve the refreshed timestamp when any page is reloaded.

I tried the basic indexer incantation and cannot reproduce the reported problem.

For completeness, the timestamp file is touched only if there were any changes to the index (there is dirty flag in IndexDatabase).

[danielviorreta]

I have my app deployed to ROOT so adding the source context didn't fix my problem as it was expected. Sorry for the confusion, I misinterpreted the logs.

So, my original problem is that adding -U http://localhost:8080 to the indexer grok UI doesn't show any project during projects invalidations. Indexer logs:

2020-04-21 05:26:16.122-0400 CONFIG t1 ExtendedLogger.config: Selected ExecutorServiceProvider implementation [org.glassfish.jersey.client.DefaultClientAsyncExecutorProvider] to be used for injection of executor qualified by [org.glassfish.jersey.client.ClientAsyncExecutor] annotation.
2020-04-21 05:26:16.122-0400 CONFIG t1 ExtendedLogger.config: Selected ScheduledExecutorServiceProvider implementation [org.glassfish.jersey.client.DefaultClientBackgroundSchedulerProvider] to be used for injection of scheduler qualified by [org.glassfish.jersey.client.ClientBackgroundScheduler] annotation.
2020-04-21 05:26:16.123-0400 CONFIG t1 HttpUrlConnector.<init>: Restricted headers are not enabled using [sun.net.http.allowRestrictedHeaders] system property (setting only takes effect on connections created after the property has been set/changed).
2020-04-21 05:26:16.125-0400 FINEST t1 HttpURLConnection.plainConnect0: ProxySelector Request for http://localhost:8080/api/v1/configuration?reindex=true
2020-04-21 05:26:16.125-0400 FINEST t1 HttpClient.logFinest: KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://localhost:8080/api/v1/projects/jdk7u60/indexed)
2020-04-21 05:26:16.125-0400 FINEST t1 HttpURLConnection.plainConnect0: Proxy used: DIRECT
2020-04-21 05:26:16.127-0400 FINE t1 HttpURLConnection.writeRequests: sun.net.www.MessageHeader@54bc3bcf7 pairs: {PUT /api/v1/configuration?reindex=true HTTP/1.1: null}{Content-Type: application/xml}{User-Agent: Jersey/2.29 (HttpUrlConnection 11.0.3)}{Host: localhost:8080}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 2103065}
2020-04-21 05:27:25.994-0400 FINE t1 HttpURLConnection.getInputStream0: sun.net.www.MessageHeader@d3d50946 pairs: {null: HTTP/1.1 204}{Cache-Control: private}{Expires: Thu, 01 Jan 1970 00:00:00 GMT}{Date: Tue, 21 Apr 2020 09:27:25 GMT}{Keep-Alive: timeout=20}{Connection: keep-alive}
2020-04-21 05:27:25.994-0400 INFO t1 Indexer.sendToConfigHost: Configuration update routine done, check log output for errors.`

In tomcat logs I have this:

`21-Apr-2020 05:27:14.581 INFO [http-nio-127.0.0.1-8080-exec-603] org.opengrok.indexer.util.Statistics.report done invalidating 1487 repositories (took 58.290 seconds)`

During that minute, grok UI doesn't show any project. Is this expected or do you think I have any configuration problem? I tried to deploy my webapp using the source context path but I get the same problem.

Thanks!

[vladak]

The old repositories should show during invalidation. How does the index page look like without repositories ?

What is the version you are running ?

[danielviorreta]

I am running version 1.3.12. I reproduce the problem every time that I call:
/bin/curl -X PUT -H "Content-Type: application/xml" --data "@/var/opengrok/etc/configuration.xml" http://localhost:8080/source/api/v1/configuration?reindex=true

[danielviorreta]

@vladak did you manage to reproduce the problem? Do you have any suggestion about that?
Thanks!

[danielviorreta]

@vladak I think the problem is in

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/configuration/RuntimeEnvironment.java

Line 1587 in 3b65bef

group.getRepositories().clear();

All my projects are included in a group. Uploading a new configuration clears all the repositories from all the groups.
Projects that are not part of group are still listed during the upload.

Does it make sense?

[tarzanek]

@tulinkry ^ any obvious prob that you can see when groups are merged?

[tulinkry]

It does not merge them, it replaces the old ones but it shouldn't take a minute even for thousand of projects.

[danielviorreta]

it takes ~ 5 minutes for 5000 projects and 500 groups.
The repositories in groups disappeared from the UI when I see this log in tomcat:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/history/HistoryGuru.java

Line 862 in 2173ed7

LOGGER.log(Level.FINE, "invalidating {0} repositories", repos.size());

(maybe because we are clearing the repos in https://github.com/oracle/opengrok/blob/2173ed7b20b4d6ae704b761403b2af7f52426e38/opengrok-indexer/src/main/java/org/opengrok/indexer/history/HistoryGuru.java#L854)

And they only come back after

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/configuration/RuntimeEnvironment.java

Line 1760 in 3b65bef

LOGGER.log(Level.INFO, "Configuration updated");