Missing bwc test for indexing data for index created with older version

[jainankitk]

Is your feature request related to a problem? Please describe

Recent changes - #19480 in the DateFieldMapper resulted into backward incompatibility, while indexing data from 3.3.0 into an index created on 3.2.0 / 2.19.2 version

Describe the solution you'd like

We probably try to upgrade the existing indices and run search, but do not explicitly index additional data into the upgraded index. We should be able to catch such regressions by doing so.

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

No response

[getsaurabh02]

Thanks @jainankitk ! This is a valuable request, especially since our backward compatibility tests missed a critical upgrade flow for indexing, particularly around certain field types in OS 3.3 release demanding an urgent need for a patch release.

Adding @asimmahmood1 who is planning to add support for timestamp field in the indexing flow. Also adding @Bukhtawar @andrross for visibility and to discuss any additional improvements we may need in this area.

[asimmahmood1]

I was looking at two the existing tests:

1. OpenSearch/qa/mixed-cluster/src/test/java/org/opensearch/backwards/IndexingIT.java

   Line 77 in d34b352

   request.setJsonEntity("{\"test\": \"test_" + randomAlphaOfLength(2) + "\", \"sortfield\": \""+ randomIntBetween(0, numDocs) + "\"}");

2. OpenSearch/qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/IndexingIT.java

   Line 391 in d34b352

   private void bulk(String index, String valueSuffix, int count) throws IOException {

My guess is these tests were created to assert different version of nodes can communicate with each other, but not necessary that same node can be upgraded. Am I correct?

Both of these only index a String/text and number, and not date or many of other types: https://docs.opensearch.org/latest/mappings/supported-field-types/index/

In the short term I can add date a few more fields to above test. Long term should we create a new test setup that just asserts that same node can be upgraded, so that it's easier to debug when it fails. If there are such tests already, let me know. @Bukhtawar.

[andrross]

@asimmahmood1 Here is my understanding of what the mixed-cluster and rolling-upgrade test suites do:

• Mixed cluster: Creates a 4 node cluster, 2 nodes running the old version and 2 nodes running the new version. Then all the tests are run against that cluster. Note this includes the MixedClusterClientYamlTestSuiteIT test that runs all the yaml test scenarios defined in rest-api-spec.
• Rolling upgrade: Creates a 3 node cluster, all running the old version and then:
  • runs the tests
  • upgrades 1 node, re-runs the tests
  • upgrades another node, re-runs the tests
  • upgrades the last node, re-runs the tests

The rolling upgrade suite also contains a YAML test suite, but defines its own yaml tests.

[andrross]

@jainankitk @asimmahmood1 I want to raise another half-baked idea here. The existing test suites certainly provide a lot of value, but they are hideously complicated with loads of complicated groovy code and thousands of lines of custom gradle code. This leads to a lot of pain when things go wrong, particularly with Gradle upgrades because we are so deeply integrated into the Gradle codebase.

Instead of expanding this, how about tackling the problem externally? These types of tests could operate on published artifacts (previous releases and the latest snapshot version of the next release). We'd have to figure out how to integrate this into our CI ecosystem, but it could probably be another component in the distribution integration test workflows. We in fact already have something that looks like an external test framework in the opensearch-api-specification. I don't think it has any ability to orchestrate upgrades or mixed versions, but it could, and it could eventually replace a big chunk of the complexity that is currently in this repo.

[jainankitk]

Instead of expanding this, how about tackling the problem externally? These types of tests could operate on published artifacts (previous releases and the latest snapshot version of the next release). We'd have to figure out how to integrate this into our CI ecosystem, but it could probably be another component in the distribution integration test workflows. We in fact already have something that looks like an external test framework in the opensearch-api-specification.

I have limited understanding of the distribution integration test workflows, but want to ensure that the tests are not expensive to run if with every commit / PR. IMO, it is even okay to run these tests less frequently say nightly, but ensure that we cover wider variety of test cases using real world data using, say opensearch-benchmark-workloads.