Renaming, and Backwards Compatibility for Indices

[dblock]

To track for #12

[hsiang9431-amzn]

Hi, for what dashboards team has done, changing the use for .kibana indices to .opensearch_dashboards, it appears to me supporting only an one way indices upgrade from opendistro to opensearch will make more sense.

Is this acceptable and should be applied to rest of the project?

Otherwise maintaining support for 2 sets of indices may cause major overhead in terms of code logic and testing. Please keep us posted on the decisions, it will help our team make decision down the road.

Thanks,

[saratvemulapalli]

Sure definitely. Yes the indices are permanently renamed.
There are 2 parts to the answer:

1. .opensearch_dashboards is the index which we will use going forward.
2. Dashboards team and plugins should have backward compatibility support for .kibana in order to support drop in replacement for customers.

Similarly it's the same with opendistro indices.

[dblock]

When an OpenSearch with a plugin joins a cluster, check old name, and create the new one if that doesn't exist.

[cliu123]

When an OpenSearch with a plugin joins a cluster, check old name, and create the new one if that doesn't exist.

@dblock @saratvemulapalli
Is there a POC for this? We do not see an effective way to support both indices in plugins. If there is a POC for any solution, could you please provide?

[hsiang9431-amzn]

When an OpenSearch with a plugin joins a cluster, check old name, and create the new one if that doesn't exist.

Does this mean a new OpenSearch node will perform indices upgrade once it first join a cluster?

This approach makes more sense IMO and it is also be expandable to other plugins.
Maybe we should have a function in OpenSearch core that do this work and let plugins invoke it during plugin startup?

something like

public CreateAndCopy(final String oldIndex, final String newIndex) {
    // create new index
    // check if old index exist
    // if yes, copy the content to new index
}

If it's possible, adding the copy operation as a hook on index write for legacy indices should help keeping things coherent.

Is there a POC for this? We do not see an effective way to support both indices in plugins. If there is a POC for any solution, could you please provide?

Going with this approach, the additional tests should be surrounding added functions and on-write hooks. It should reduce the complexity comparing to supporting both.

[dblock]

When an OpenSearch with a plugin joins a cluster, check old name, and create the new one if that doesn't exist.

@dblock @saratvemulapalli
Is there a POC for this? We do not see an effective way to support both indices in plugins. If there is a POC for any solution, could you please provide?

I really just started looking into this, sorry :(

[dblock]

When an OpenSearch with a plugin joins a cluster, check old name, and create the new one if that doesn't exist.

Does this mean a new OpenSearch node will perform indices upgrade once it first join a cluster?

I don't think so, I think all Sarat is saying is that there's this if exists(old) { create alias or do nothing } else { create new } logic somewhere.

Duplicating indexes live at startup/joining a cluster just to rename them seems potentially extremely expensive and error-prone.

[saratvemulapalli]

We have decided not to rename indices as it is a lot of work and increases the complexity.
I believe the only open question would be how do we take care of .kibana index, which is probably already being handled by opensearch-project/OpenSearch-Dashboards#334

@dblock can we close this issue?