[BUG] Valid git version marked as invalid

[ze]

Describe the bug
Upon trying to list a workload repository, I get this error.
[ERROR] Cannot list. Your git version is [['git version 2.32.0']] but Benchmark requires at least git 1.9. Please update git.

To Reproduce

• I installed git via sudo yum install git where git --version responds with git version 2.32.0
• Run this command opensearch-benchmark list workloads --workload-repository="literally anything"
• It will fail

Expected behavior
It should work as expected and list the workload repository, because my git version is valid.

Logs

2022-05-04 16:34:40,693 -not-actor-/PID:19982 osbenchmark.benchmark INFO OS [uname_result(system='Linux', node='ip-10-16-1-42.ec2.internal', release='4.14.275-207.503.amzn2.x86_64', version='#1 SMP Wed Apr 13 20:24:50 UTC 2022', machine='x86_64', processor='x86_64')]
2022-05-04 16:34:40,694 -not-actor-/PID:19982 osbenchmark.benchmark INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50857456, name='cpython', version=sys.version_info(major=3, minor=8, micro=5, releaselevel='final', serial=0))]
2022-05-04 16:34:40,694 -not-actor-/PID:19982 osbenchmark.benchmark INFO Benchmark version [0.0.2]
2022-05-04 16:34:40,694 -not-actor-/PID:19982 osbenchmark.utils.net INFO Connecting directly to the Internet (no proxy support).
2022-05-04 16:34:40,722 -not-actor-/PID:19982 osbenchmark.benchmark INFO Detected a working Internet connection.
2022-05-04 16:34:40,729 -not-actor-/PID:19982 osbenchmark.benchmark ERROR Cannot run subcommand [list].
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.8/site-packages/osbenchmark/benchmark.py", line 825, in dispatch_sub_command
    dispatch_list(cfg)
  File "/home/ec2-user/.local/lib/python3.8/site-packages/osbenchmark/benchmark.py", line 613, in dispatch_list
    workload.list_workloads(cfg)
  File "/home/ec2-user/.local/lib/python3.8/site-packages/osbenchmark/workload/loader.py", line 118, in list_workloads
    available_workloads = workloads(cfg)
  File "/home/ec2-user/.local/lib/python3.8/site-packages/osbenchmark/workload/loader.py", line 113, in workloads
    repo = workload_repo(cfg)
  File "/home/ec2-user/.local/lib/python3.8/site-packages/osbenchmark/workload/loader.py", line 290, in workload_repo
    return GitWorkloadRepository(cfg, fetch, update)
  File "/home/ec2-user/.local/lib/python3.8/site-packages/osbenchmark/workload/loader.py", line 331, in __init__
    self.repo.update(distribution_version)
  File "/home/ec2-user/.local/lib/python3.8/site-packages/osbenchmark/utils/repo.py", line 99, in update
    branch = versions.best_match(git.branches(self.repo_dir, remote=False), distribution_version)
  File "/home/ec2-user/.local/lib/python3.8/site-packages/osbenchmark/utils/git.py", line 42, in probe
    raise exceptions.SystemSetupError("Your git version is [%s] but Benchmark requires at least git 1.9. "
osbenchmark.exceptions.SystemSetupError: Your git version is [['git version 2.32.0']] but Benchmark requires at least git 1.9. Please update git.

More Context (please complete the following information):

• Workload(Share link for custom workloads) Doesn't apply
• Service(E.g OpenSearch) Would be opensearch
• Version (E.g. 1.0) Version 0.0.2 downloaded via pip.

Additional context
I haven't dug into the code, but it looks like it tries a match expecting the git version to return an exact semver instead of git version ...?

[domruf]

I also get this error. I even build the docker container and tried it in the container. But I still got the error

$ opensearch-benchmark execute_test --offline --distribution-version=1.2.0 --workload=geonames --test-mode --pipeline=benchmark-only
...
[ERROR] Cannot execute_test. Error in test execution orchestrator (Your git version is [['git version 2.20.1']] but Benchmark requires at least git 1.9. Please update git.)
...

[domruf]

FYI after I manually did

git clone https://github.com/opensearch-project/opensearch-benchmark-workloads /benchmark/.benchmark/benchmarks/workloads/default

in the container I now get the error

[ERROR] Cannot execute_test. Error in worker_coordinator (OpenSearch REST API layer is not available.)

[IanHoang]

Hi @ze and @domruf,

Thanks for your interest in OpenSearch-Benchmark and for sharing this! I was able to list the workloads repository without receiving the git error. I've tried the following methods and attached their outputs.

---

Outputs

1. Using the default workloads repository that comes with OpenSearch-Benchmark

hoangia@3c22fbd0d988 ~ % opensearch-benchmark --version         
opensearch-benchmark 0.0.2
hoangia@3c22fbd0d988 ~ % opensearch-benchmark list workloads

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[WARNING] Local changes in [/Users/hoangia/.benchmark/benchmarks/workloads/default] prevent workloads update from remote. Please commit your changes.
Available workloads:

Name           Description                                                                                                        Documents      Compressed Size    Uncompressed Size    Default TestProcedure         All TestProcedures
-------------  -----------------------------------------------------------------------------------------------------------------  -------------  -----------------  -------------------  ----------------------------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
geonames       POIs from Geonames                                                                                                 11,396,503     252.9 MB           3.3 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts,significant-text
percolator     Percolator benchmark based on AOL queries                                                                          2,000,000      121.1 kB           104.9 MB             append-no-conflicts           append-no-conflicts
juno_workload  HTTP server log data                                                                                               8,334,000,000  N/A                1003.1 GB            append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,search-queries
http_logs      HTTP server log data                                                                                               247,249,096    1.2 GB             31.1 GB              append-no-conflicts           append-no-conflicts,runtime-fields,append-no-conflicts-index-only,append-sorted-no-conflicts,append-index-only-with-ingest-pipeline,update,append-no-conflicts-index-reindex-only
geoshape       Shapes from PlanetOSM                                                                                              60,523,283     13.4 GB            45.4 GB              append-no-conflicts           append-no-conflicts
geopoint       Point coordinates from PlanetOSM                                                                                   60,844,404     482.1 MB           2.3 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
nyc_taxis      Taxi rides in New York in 2015                                                                                     165,346,692    4.5 GB             74.3 GB              append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts-index-only,update,append-ml,date-histogram
geopointshape  Point coordinates from PlanetOSM indexed as geoshapes                                                              60,844,404     470.8 MB           2.6 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
so             Indexing benchmark using up to questions and answers from StackOverflow                                            36,062,278     8.9 GB             33.1 GB              append-no-conflicts           append-no-conflicts
eventdata      This benchmark indexes HTTP access logs generated based sample logs from the elastic.co website using a generator  20,000,000     756.0 MB           15.3 GB              append-no-conflicts           append-no-conflicts,transform
nested         StackOverflow Q&A stored as nested docs                                                                            11,203,029     663.3 MB           3.4 GB               nested-search-test-procedure  nested-search-test-procedure,index-only
noaa           Global daily weather measurements from NOAA                                                                        33,659,481     949.4 MB           9.0 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,top_metrics,aggs
pmc            Full text benchmark with academic papers from PMC                                                                  574,199        5.5 GB             21.7 GB              append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts

-------------------------------
[INFO] SUCCESS (took 1 seconds)
-------------------------------

2. Created a new repository called test-repository and git cloned opensearch-benchmark-workloads

hoangia@3c22fbd0d988 workloads % mkdir test-repository
hoangia@3c22fbd0d988 workloads % ls
default		test-repository
hoangia@3c22fbd0d988 workloads % cd test-repository
hoangia@3c22fbd0d988 test-repository % git clone https://github.com/opensearch-project/opensearch-benchmark-workloads ./
Cloning into '.'...
remote: Enumerating objects: 3876, done.
remote: Counting objects: 100% (505/505), done.
remote: Compressing objects: 100% (120/120), done.
remote: Total 3876 (delta 408), reused 396 (delta 361), pack-reused 3371
Receiving objects: 100% (3876/3876), 1.29 MiB | 1.23 MiB/s, done.
Resolving deltas: 100% (2571/2571), done.
hoangia@3c22fbd0d988 default % cd ~
hoangia@3c22fbd0d988 ~ % opensearch-benchmark list workloads --workload-repository="test-repository"

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

Available workloads:

Name           Description                                                                                                        Documents    Compressed Size    Uncompressed Size    Default TestProcedure         All TestProcedures
-------------  -----------------------------------------------------------------------------------------------------------------  -----------  -----------------  -------------------  ----------------------------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------
geonames       POIs from Geonames                                                                                                 11,396,503   252.9 MB           3.3 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts,significant-text
percolator     Percolator benchmark based on AOL queries                                                                          2,000,000    121.1 kB           104.9 MB             append-no-conflicts           append-no-conflicts
http_logs      HTTP server log data                                                                                               247,249,096  1.2 GB             31.1 GB              append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-index-only-with-ingest-pipeline,update,append-no-conflicts-index-reindex-only
geoshape       Shapes from PlanetOSM                                                                                              60,523,283   13.4 GB            45.4 GB              append-no-conflicts           append-no-conflicts
geopoint       Point coordinates from PlanetOSM                                                                                   60,844,404   482.1 MB           2.3 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
nyc_taxis      Taxi rides in New York in 2015                                                                                     165,346,692  4.5 GB             74.3 GB              append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts-index-only,update
geopointshape  Point coordinates from PlanetOSM indexed as geoshapes                                                              60,844,404   470.8 MB           2.6 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
so             Indexing benchmark using up to questions and answers from StackOverflow                                            36,062,278   8.9 GB             33.1 GB              append-no-conflicts           append-no-conflicts
eventdata      This benchmark indexes HTTP access logs generated based sample logs from the elastic.co website using a generator  20,000,000   756.0 MB           15.3 GB              append-no-conflicts           append-no-conflicts,transform
nested         StackOverflow Q&A stored as nested docs                                                                            11,203,029   663.3 MB           3.4 GB               nested-search-test-procedure  nested-search-test-procedure,index-only
noaa           Global daily weather measurements from NOAA                                                                        33,659,481   949.4 MB           9.0 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,top_metrics,aggs
pmc            Full text benchmark with academic papers from PMC                                                                  574,199      5.5 GB             21.7 GB              append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts

-------------------------------
[INFO] SUCCESS (took 1 seconds)
-------------------------------

---

My setup:

• opensearch-benchmark version: 0.0.2
• git version: 2.33.0
• operating system: MacOS, Linux

Follow Up Questions

Just to follow up with @ze, are you still experiencing this issue? If so, I have a few more questions:

1. Prior to receiving this error, did you install opensearch-benchmark through pip or via git cloning the repository?
2. What operating system are you attempting to run opensearch-benchmark on?

[IanHoang]

@domruf Could you share exact steps to reproduce your situation as well as more context on your setup?

Based on my experience, I've only seen this issue occur when users invoke a cloned repository of opensearch-benchmark that contains changes.

[engechas]

This exception comes from here:

opensearch-benchmark/osbenchmark/utils/git.py

Line 42 in ad608cc

raise exceptions.SystemSetupError("Your git version is [%s] but Benchmark requires at least git 1.9. "

The code that triggers this exception path is the if statement on line 35, which checks the exit status of running:

git -C </path/to/something> --version

The purpose of this is to verify that the git version is above 1.9, where the -C flag was introduced. If the git version is below 1.9, then the above command will return a non-zero exit code and trigger the exception shown by @ze .

However in this case it seems that there is a different cause for git -C </path/to/something> --version returning a non-zero exit code (potentially something related to the path missing?). We will need repro steps to track down exactly what this is as @IanHoang mentions.

As part of the solution to this issue, we should take a look at refactoring the probe method to distinguish between version issues and other issues so that we can provide a meaningful error message

[ze]

Apologies for the late response. I'm currently on work vacation, so I don't have access to the machine I sourced the issue on.
It was a fresh box on EC2 with amzn2 linux. I ran the commands I said above to repro (installed git with yum, pip installed opensearch-benchmark)

To be purely speculative, it may be the binary from yum was built with specific flags that cause issues with git -C path --version. The exception could be altered to indicate the path used if somebody could be done to make it more explicit.

[narsimplyme]

Is there any updates on this? I've tried Git 1.9.5 and 2.39.0 and it seems the same

• Fresh install on a bare metal using pip3 install opensearch-benchmark

The result is same. any workarounds?
[ERROR] Cannot list. Your git version is [['git version 1.9.5']] but Benchmark requires at least git 1.9. Please update git.

[sharp-pixel]

The error message is misleading, the error can happen when git doesn't have permissions on the requested folder (like ~/.benchmark).
We need to make the error message more explicit.

[IanHoang]

Apologies for the late response and thank you for your patience. We've started tacking several OSB issues in the last few months and many of which have been resolved (including a git related one here #246) in starting in OSB v0.2.0 (released March 2023).

I haven't been able to successfully reproduce the issue. I spun up a single Amazon Linux 2 EC2 instance and did the following:

• Installed Git via sudo yum instal git --> installed git version 2.39.2
• Installed pip --> installed pip3 inversion 23.1.2
• Installed opensearch-benchmark via python3 -m pip install opensearch-benchmark --> installed opensearch-benchmark 0.4.1

I have run the list subcommand several times and have not encountered the issue

opensearch-benchmark list workloads --workload-repository=default

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

Available workloads:

Name           Description                                                                                                        Documents    Compressed Size    Uncompressed Size    Default TestProcedure         All TestProcedures
-------------  -----------------------------------------------------------------------------------------------------------------  -----------  -----------------  -------------------  ----------------------------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------
eventdata      This benchmark indexes HTTP access logs generated based sample logs from the elastic.co website using a generator  20,000,000   756.0 MB           15.3 GB              append-no-conflicts           append-no-conflicts,transform
geonames       POIs from Geonames                                                                                                 11,396,503   252.9 MB           3.3 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts,significant-text
geopoint       Point coordinates from PlanetOSM                                                                                   60,844,404   482.1 MB           2.3 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geopointshape  Point coordinates from PlanetOSM indexed as geoshapes                                                              60,844,404   470.8 MB           2.6 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geoshape       Shapes from PlanetOSM                                                                                              60,523,283   13.4 GB            45.4 GB              append-no-conflicts           append-no-conflicts
http_logs      HTTP server log data                                                                                               247,249,096  1.2 GB             31.1 GB              append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-index-only-with-ingest-pipeline,update,append-no-conflicts-index-reindex-only
nested         StackOverflow Q&A stored as nested docs                                                                            11,203,029   663.3 MB           3.4 GB               nested-search-test-procedure  nested-search-test-procedure,index-only
noaa           Global daily weather measurements from NOAA                                                                        33,659,481   949.4 MB           9.0 GB               append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,top_metrics,aggs
nyc_taxis      Taxi rides in New York in 2015                                                                                     165,346,692  4.5 GB             74.3 GB              append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts-index-only,update,searchable-snapshot
percolator     Percolator benchmark based on AOL queries                                                                          2,000,000    121.1 kB           104.9 MB             append-no-conflicts           append-no-conflicts
pmc            Full text benchmark with academic papers from PMC                                                                  574,199      5.5 GB             21.7 GB              append-no-conflicts           append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts
so             Indexing benchmark using up to questions and answers from StackOverflow                                            36,062,278   8.9 GB             33.1 GB              append-no-conflicts           append-no-conflicts

-------------------------------
[INFO] SUCCESS (took 0 seconds)
-------------------------------

I have also run the execute_test subcommand like @domruf did and I did not receive the issue. Also, --pipeline=benchmark-only should not be used unless users have a local OpenSearch cluster running or have provided the cluster endpoint to --target-hosts parameter.

I'll close this issue for now since many changes have been applied and merged into OSB since March 2023 and this issue has not had activity since January. Feel free to reopen the issue if the issue still exists in the latest version of OSB.