ES-Rally in kubernetes pod not sending report to target cluster

[Meghanakj]

Hello Team,
We are running an ESRally benchmark inside a kubernetes pod. The pod is configured with a PersistentVolume mounted at /rally/.rally, where the benchmark data and reports are stored.
We can see the benchmark running and completing inside the kubernetes cluster. we are using a custom rally.ini configuration to send the benchmark report to another Elasticsearch cluster. However, the benchmark completes, the report is not being sent to the target cluster as expected.

We would like to highlight that we are currently using the Docker-based Rally image elastic/rally:2.12.0

we need support to:
Understand why the report is not sent and validate if additional steps or permission are required in this setup.

please find attached files:

esrally_updated1.txt

esrally-logs-pvc.txt

esrally-report-ini-configmap(1).txt

With Regards,
Meghana K J

[gareth-ellis]

Do you have access to the logs? The config you have provided looks ok from what I can see - the fact that you get the report out suggests either that the data is being saved to the cluster, but for whatever reason you cannot see it, or that the data is being stored locally and your rally.ini file is being ignored.

There are some items missing from rally.ini file - e.g track location, but if that was causing an issue we would likely see another error.

Please also make sure you obfuscate any sensitive information in files that you add to the issue

[Meghanakj]

yes, we have access to logs.
in kibana, we are searching with index pattern "rally-metrics-* " in targeted cluster.

esrallylogs.txt

we have gone through official rally documentation, we didn't find track location information to add under report section in rally.ini.
Configuration - Rally 2.12.0 documentation

[gareth-ellis]

In your rally-home pvc there should be a logs directory, within there there should be a rally.log file - this is what we need to see, as it will say what rally was doing - the log you pasted I think is the stdout from the pod (I should have been clearer).

In your screenshot of kibana, i'm not sure if its a typo - but you have .rally-metrics-* - you should be searching without the leading .

[Meghanakj]

there is no typo error with index pattern. even we tried with rally-metrics index

we have log directory, but inside directory we couldn't see any rally.log file.

[gareth-ellis]

OK - in your most recent screenshot, you're now missing the -*..

Try

GET rally-metrics-*/_search

Another thing that seems odd, is that you are also missing a benchmarks folder within .rally. This suggests either the FS has been cleared since you last ran a race, or a different folder is being used as the home directory

I've just had a go at rerunning, and RALLY_HOME should be set to /rally, not /rally/.rally.
Whats actually happening then, is its recreating a new rally.ini under /rally/.rally/.rally/rally.ini

Once you set RALLY_HOME to /rally, then you get an error regarding the rally.ini file.

I suggest you include the following, as well:

[meta]
config.version = 17

[system]
env.name = local

[node]
root.dir = ${CONFIG_DIR}/benchmarks
src.root.dir = ${CONFIG_DIR}/benchmarks/src

[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = ${CONFIG_DIR}/benchmarks/data

[reporting]
<include the details from your config >


[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = false

[distributions]

[Meghanakj]

Hello,
We added the necessary configuration to the file. We noticed that the benchmark completed successfully, but the index was not created.
PFA of configuration.

ini-updated.txt

rally-updated.txt

esrally-logs-pvc.txt

[gareth-ellis]

Can you provide the /rally/.rally/logs/rally.log file?

Also, what do you get if you run the following on your metrics cluster

GET _cat/indices/rally-*

[Meghanakj]

esrallylogs.txt

When we ran GET _cat/indices/rally-*, not getting any result.

[gareth-ellis]

The logs you provided are from stdout from the pod, rather than the logs in /rally/.rally/logs/rally.log (These should be being saved in your pvc, so starting a new pod should have access to logs from previous runs)

Edit:

I've just checked your files you uploaded before, and you have updated the mount locations for your pvc and for rally.ini - they still need to be mounted at e.g /rally/.rally/rally.ini and /rally/.rally/logs - please update that and rerun,

[Meghanakj]

rally.log

[Meghanakj]

now we mounted as rally/.rally/rally.ini and /rally/.rally/logs

rally-updated.txt

[gareth-ellis]

ok, do you have a copy of the logs from running with updated config?

Again, this is the file in /rally/.rally/logs/rally.log - not the pod logs