Add Jenkins Docker Parallel Support and Restructure CICD Environment for build/assemble/test

[peterzhuamazon]

Description

Add Jenkins Docker Parallel Support and Restructure CICD Environment for build/assemble/test

1. Modify Dockerfile to include necessary dependencies, include both x64 and arm64 build capability. Remove JDK and MAVEN as Jenkins dependencies, add to dockerfile
2. Added centos alongside the original al2 base image, as k-NN needs to be built on CentOS for older glibc version (2.26 vs 2.17) than the one on al2.
3. Upload docker images based on the dockerfiles to https://hub.docker.com/r/opensearchstaging/ci-runner.
4. Removed tools/standard-* folders as they are obsolete with the introduction of https://github.com/opensearch-project/opensearch-build/tree/main/bundle-workflow/src/test_workflow.
5. Move the standard deploy single node cluster script to scripts/ folder as it is still handy to use for testing on local.
6. Move all the dockerfiles to bundle-workflow/ folder as they are used for all bundle related actions.
7. Added entries to README and Documentations for usages.

Thanks.

Issues Resolved

#399 #209 opensearch-project/opensearch-infra#109

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #588 (59c5f79) into main (0badd64) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #588   +/-   ##
=======================================
  Coverage   85.22%   85.22%           
=======================================
  Files          61       61           
  Lines        1672     1672           
=======================================
  Hits         1425     1425           
  Misses        247      247           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0badd64...59c5f79. Read the comment docs.

[peternied]

This is awesome!

[dblock]

This is a great step forward, I would 🚢 it.

[peternied]

Looking great - none of my questions are blocking, but I'd love more details

[peterzhuamazon]

Hi @dblock @gaiksaya please check and approve, comment if you still have any concerns. Thanks.
I also add a new line of code to allow docker adding env var for JAVA_HOME.
This will unblock k-NN lib make issues as jni.ini is not found.

Thanks.

[dblock]

I do wonder whether we have something to learn from the likes of GitHub actions which don’t include Java or Maven in base images. This makes it easy to test matrixes of things.