Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update config index mappings to use correct field types #2710

Merged
merged 3 commits into from
Jul 24, 2024
Merged

update config index mappings to use correct field types #2710

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
391bcf9
update config index mappings to use with correct field types
rbhavna Jul 23, 2024
5513c27
enable dynamic mapping for config index
rbhavna Jul 23, 2024
64dbbd2
change last_update_time field
rbhavna Jul 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions common/src/main/java/org/opensearch/ml/common/CommonValue.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@

import java.util.Set;

import static org.opensearch.ml.common.MLConfig.CONFIG_TYPE_FIELD;
import static org.opensearch.ml.common.MLConfig.LAST_UPDATED_TIME_FIELD;
import static org.opensearch.ml.common.MLConfig.ML_CONFIGURATION_FIELD;
import static org.opensearch.ml.common.conversation.ConversationalIndexConstants.APPLICATION_TYPE_FIELD;
import static org.opensearch.ml.common.conversation.ConversationalIndexConstants.INTERACTIONS_ADDITIONAL_INFO_FIELD;
import static org.opensearch.ml.common.conversation.ConversationalIndexConstants.INTERACTIONS_CONVERSATION_ID_FIELD;
Expand Down Expand Up @@ -407,22 +410,21 @@ public class CommonValue {
+ " \"_meta\": {\"schema_version\": "
+ ML_CONFIG_INDEX_SCHEMA_VERSION
+ "},\n"
+ " \"dynamic\": \"strict\",\n"
+ " \"properties\": {\n"
+ " \""
+ MASTER_KEY
+ "\": {\"type\": \"keyword\"},\n"
+ " \""
+ MLConfig.TYPE_FIELD
+ CONFIG_TYPE_FIELD
+ "\" : {\"type\":\"keyword\"},\n"
+ " \""
+ MLConfig.CONFIGURATION_FIELD
+ ML_CONFIGURATION_FIELD
+ "\" : {\"type\": \"flat_object\"},\n"
+ " \""
+ CREATE_TIME_FIELD
+ "\": {\"type\": \"date\", \"format\": \"strict_date_time||epoch_millis\"},\n"
+ " \""
+ LAST_UPDATE_TIME_FIELD
+ LAST_UPDATED_TIME_FIELD
+ "\": {\"type\": \"date\", \"format\": \"strict_date_time||epoch_millis\"}\n"
+ " }\n"
+ "}";
Expand Down
29 changes: 24 additions & 5 deletions common/src/main/java/org/opensearch/ml/common/MLConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,18 @@ public class MLConfig implements ToXContentObject, Writeable {

public static final String TYPE_FIELD = "type";

public static final String CONFIG_TYPE_FIELD = "config_type";

public static final String CONFIGURATION_FIELD = "configuration";

public static final String ML_CONFIGURATION_FIELD = "ml_configuration";

public static final String CREATE_TIME_FIELD = "create_time";
public static final String LAST_UPDATE_TIME_FIELD = "last_update_time";

public static final String LAST_UPDATED_TIME_FIELD = "last_updated_time";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to reuse the old Field names for CX experience BWC, rather than defining new ones?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using old names and trying to update their types is failing. OpenSearch does not allow conversion between certain types and flat_object is one of them



@Setter
private String type;

Expand Down Expand Up @@ -78,10 +85,10 @@ public void writeTo(StreamOutput out) throws IOException {
public XContentBuilder toXContent(XContentBuilder xContentBuilder, Params params) throws IOException {
XContentBuilder builder = xContentBuilder.startObject();
if (type != null) {
builder.field(TYPE_FIELD, type);
builder.field(CONFIG_TYPE_FIELD, type);
Copy link
Collaborator

@ylwu-amzn ylwu-amzn Sep 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this PR

    "type" : "os_query_assist_ppl_agent",
    "configuration" : {
      "agent_id" : "qdFHuZEBJv7K-ogaOTOB"
    }

Now we have

    "config_type" : "os_query_assist_ppl_agent",
    "ml_configuration" : {
      "agent_id" : "qdFHuZEBJv7K-ogaOTOB"
    }

This is a breaking change.

How about changing to

if (type != null) {
            builder.field(TYPE_FIELD, type);
}
if (configType != null) {
            builder.field(CONFIG_TYPE_FIELD, type);
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for line 91, that's also breaking

}
if (configuration != null) {
builder.field(CONFIGURATION_FIELD, configuration);
builder.field(ML_CONFIGURATION_FIELD, configuration);
}
if (createTime != null) {
builder.field(CREATE_TIME_FIELD, createTime.toEpochMilli());
Expand All @@ -99,9 +106,12 @@ public static MLConfig fromStream(StreamInput in) throws IOException {

public static MLConfig parse(XContentParser parser) throws IOException {
String type = null;
String configType = null;
Configuration configuration = null;
Configuration mlConfiguration = null;
Instant createTime = null;
Instant lastUpdateTime = null;
Instant lastUpdatedTime = null;

ensureExpectedToken(XContentParser.Token.START_OBJECT, parser.currentToken(), parser);
while (parser.nextToken() != XContentParser.Token.END_OBJECT) {
Expand All @@ -112,25 +122,34 @@ public static MLConfig parse(XContentParser parser) throws IOException {
case TYPE_FIELD:
type = parser.text();
break;
case CONFIG_TYPE_FIELD:
configType = parser.text();
break;
case CONFIGURATION_FIELD:
configuration = Configuration.parse(parser);
break;
case ML_CONFIGURATION_FIELD:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another field added in 2.13, LAST_UPDATE_TIME_FIELD, let's use similar way to fix?

mlConfiguration = Configuration.parse(parser);
break;
case CREATE_TIME_FIELD:
createTime = Instant.ofEpochMilli(parser.longValue());
break;
case LAST_UPDATE_TIME_FIELD:
lastUpdateTime = Instant.ofEpochMilli(parser.longValue());
break;
case LAST_UPDATED_TIME_FIELD:
lastUpdatedTime = Instant.ofEpochMilli(parser.longValue());
break;
default:
parser.skipChildren();
break;
}
}
return MLConfig.builder()
.type(type)
.configuration(configuration)
.type(configType == null ? type : configType)
Copy link
Collaborator

@ylwu-amzn ylwu-amzn Sep 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parse new field value to new field

.type(type)
.configType(configType)

.configuration(mlConfiguration == null ? configuration : mlConfiguration)
.createTime(createTime)
.lastUpdateTime(lastUpdateTime)
.lastUpdateTime(lastUpdatedTime == null ? lastUpdateTime : lastUpdatedTime)
.build();
}
}
Loading