Skip to content

Fix DateMathExpressionResolverTests tests #37037 #37059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1195,7 +1195,7 @@ private void validateBulkResponses(int nbItems, boolean[] errors, BulkResponse b
public void testUrlEncode() throws IOException {
String indexPattern = "<logstash-{now/M}>";
String expectedIndex = "logstash-" +
DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(DateTimeZone.UTC).monthOfYear().roundFloorCopy());
DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(DateTimeZone.UTC).monthOfYear().roundFloorCopy());
{
IndexRequest indexRequest = new IndexRequest(indexPattern).id("id#1");
indexRequest.source("field", "value");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ To run this example, first follow the steps in

The seat data contains:

* A date in the format `YYYY-MM-DD` where the second digit of both month and day
* A date in the format `yyyy-MM-DD` where the second digit of both month and day
is optional.
* A time in the format HH:MM* where the second digit of both hours and minutes
is optional. The star (*) represents either the `String` `AM` or `PM`.
Expand Down
10 changes: 5 additions & 5 deletions docs/reference/api-conventions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Where:
[horizontal]
`static_name`:: is the static text part of the name
`date_math_expr`:: is a dynamic date math expression that computes the date dynamically
`date_format`:: is the optional format in which the computed date should be rendered. Defaults to `YYYY.MM.dd`.
`date_format`:: is the optional format in which the computed date should be rendered. Defaults to `yyyy.MM.dd`.
`time_zone`:: is the optional time zone . Defaults to `utc`.

Date math expressions are resolved locale-independent. Consequently, it is not possible to use any other
Expand Down Expand Up @@ -132,9 +132,9 @@ they resolve to given the current time is 22rd March 2024 noon utc.
| Expression |Resolves to
| `<logstash-{now/d}>` | `logstash-2024.03.22`
| `<logstash-{now/M}>` | `logstash-2024.03.01`
| `<logstash-{now/M{YYYY.MM}}>` | `logstash-2024.03`
| `<logstash-{now/M-1M{YYYY.MM}}>` | `logstash-2024.02`
| `<logstash-{now/d{YYYY.MM.dd\|+12:00}}>` | `logstash-2024.03.23`
| `<logstash-{now/M{yyyy.MM}}>` | `logstash-2024.03`
| `<logstash-{now/M-1M{yyyy.MM}}>` | `logstash-2024.02`
| `<logstash-{now/d{yyyy.MM.dd\|+12:00}}>` | `logstash-2024.03.23`
|======

To use the characters `{` and `}` in the static part of an index name template, escape them
Expand All @@ -144,7 +144,7 @@ with a backslash `\`, for example:

The following example shows a search request that searches the Logstash indices for the past
three days, assuming the indices use the default Logstash index name format,
`logstash-YYYY.MM.dd`.
`logstash-yyyy.MM.dd`.

[source,js]
----------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/monitoring/http-export.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ xpack.monitoring.exporters:
headers: <7>
My-Proxy-Header: abc123
My-Other-Thing: [ def456, ... ]
index.name.time_format: YYYY-MM <8>
index.name.time_format: yyyy-MM <8>

----------------------------------
<1> A `local` exporter defined explicitly whose arbitrary name is `my_local`.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/settings/monitoring-settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ anything defined here.
`index.name.time_format`::

A mechanism for changing the default date suffix for the, by default, daily Monitoring indices.
The default value is `YYYY.MM.DD`, which is why the indices are created daily.
The default value is `yyyy.MM.DD`, which is why the indices are created daily.

`use_ingest`::

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/sql/functions/date-time.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The table below shows the mapping between {es} and {es-sql}:
s|{es}
s|{es-sql}
2+h| Index/Table datetime math
2+|<index-{now/M{YYYY.MM}}>
2+|<index-{now/M{yyyy.MM}}>
2+h| Query date/time math
| 1y | INTERVAL 1 YEAR
| 2M | INTERVAL 2 MONTH
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -355,8 +355,8 @@ public void testUrlEncoder() {
}

public void testUrlEncoderWithParam() throws Exception {
assertScript("{{#url}}{{index}}{{/url}}", singletonMap("index", "<logstash-{now/d{YYYY.MM.dd|+12:00}}>"),
equalTo("%3Clogstash-%7Bnow%2Fd%7BYYYY.MM.dd%7C%2B12%3A00%7D%7D%3E"));
assertScript("{{#url}}{{index}}{{/url}}", singletonMap("index", "<logstash-{now/d{yyyy.MM.dd|+12:00}}>"),
equalTo("%3Clogstash-%7Bnow%2Fd%7Byyyy.MM.dd%7C%2B12%3A00%7D%7D%3E"));

final String random = randomAlphaOfLength(10);
assertScript("{{#url}}prefix_{{s}}{{/url}}", singletonMap("s", random),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -716,7 +716,7 @@ private static Mapper.Builder<?,?> createBuilderFromDynamicValue(final ParseCont
} else if (parseableAsLong == false && parseableAsDouble == false && context.root().dateDetection()) {
// We refuse to match pure numbers, which are too likely to be
// false positives with date formats that include eg.
// `epoch_millis` or `YYYY`
// `epoch_millis` or `yyyy`
for (DateFormatter dateTimeFormatter : context.root().dynamicDateTimeFormatters()) {
try {
dateTimeFormatter.parse(text);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -276,42 +276,42 @@ public void testRolloverOnExistingIndex() throws Exception {
public void testRolloverWithDateMath() {
ZonedDateTime now = ZonedDateTime.now(ZoneOffset.UTC);
assumeTrue("only works on the same day", now.plusMinutes(5).getDayOfYear() == now.getDayOfYear());
String index = "test-" + DateFormatter.forPattern("YYYY.MM.dd").format(now) + "-1";
String index = "test-" + DateFormatter.forPattern("yyyy.MM.dd").format(now) + "-1";
String dateMathExp = "<test-{now/d}-1>";
assertAcked(prepareCreate(dateMathExp).addAlias(new Alias("test_alias")).get());
ensureGreen(index);
// now we modify the provided name such that we can test that the pattern is carried on
client().admin().indices().prepareClose(index).get();
client().admin().indices().prepareUpdateSettings(index).setSettings(Settings.builder()
.put(IndexMetaData.SETTING_INDEX_PROVIDED_NAME,
"<test-{now/M{YYYY.MM}}-1>")).get();
"<test-{now/M{yyyy.MM}}-1>")).get();

client().admin().indices().prepareOpen(index).get();
ensureGreen(index);
RolloverResponse response = client().admin().indices().prepareRolloverIndex("test_alias").get();
assertThat(response.getOldIndex(), equalTo(index));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM").format(now) + "-000002"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM").format(now) + "-000002"));
assertThat(response.isDryRun(), equalTo(false));
assertThat(response.isRolledOver(), equalTo(true));
assertThat(response.getConditionStatus().size(), equalTo(0));

response = client().admin().indices().prepareRolloverIndex("test_alias").get();
assertThat(response.getOldIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM").format(now) + "-000002"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM").format(now) + "-000003"));
assertThat(response.getOldIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM").format(now) + "-000002"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM").format(now) + "-000003"));
assertThat(response.isDryRun(), equalTo(false));
assertThat(response.isRolledOver(), equalTo(true));
assertThat(response.getConditionStatus().size(), equalTo(0));

GetSettingsResponse getSettingsResponse = client().admin().indices().prepareGetSettings(response.getOldIndex(),
response.getNewIndex()).get();
assertEquals("<test-{now/M{YYYY.MM}}-000002>", getSettingsResponse.getSetting(response.getOldIndex(),
assertEquals("<test-{now/M{yyyy.MM}}-000002>", getSettingsResponse.getSetting(response.getOldIndex(),
IndexMetaData.SETTING_INDEX_PROVIDED_NAME));
assertEquals("<test-{now/M{YYYY.MM}}-000003>", getSettingsResponse.getSetting(response.getNewIndex(),
assertEquals("<test-{now/M{yyyy.MM}}-000003>", getSettingsResponse.getSetting(response.getNewIndex(),
IndexMetaData.SETTING_INDEX_PROVIDED_NAME));

response = client().admin().indices().prepareRolloverIndex("test_alias").setNewIndexName("<test-{now/d}-000004>").get();
assertThat(response.getOldIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM").format(now) + "-000003"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("YYYY.MM.dd").format(now) + "-000004"));
assertThat(response.getOldIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM").format(now) + "-000003"));
assertThat(response.getNewIndex(), equalTo("test-" + DateFormatter.forPattern("yyyy.MM.dd").format(now) + "-000004"));
assertThat(response.isDryRun(), equalTo(false));
assertThat(response.isRolledOver(), equalTo(true));
assertThat(response.getConditionStatus().size(), equalTo(0));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,11 @@ public void testExpression() throws Exception {
List<String> result = expressionResolver.resolve(context, indexExpressions);
assertThat(result.size(), equalTo(3));
assertThat(result.get(0),
equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
equalTo(".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
assertThat(result.get(1),
equalTo(".watch_history-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
equalTo(".watch_history-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
assertThat(result.get(2),
equalTo("logstash-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
equalTo("logstash-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
}

public void testEmpty() throws Exception {
Expand All @@ -86,12 +86,11 @@ public void testExpression_MultiParts() throws Exception {
List<String> result = expressionResolver.resolve(context, Arrays.asList("<.text1-{now/d}-text2-{now/M}>"));
assertThat(result.size(), equalTo(1));
assertThat(result.get(0), equalTo(".text1-"
+ DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))
+ DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))
+ "-text2-"
+ DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC).withDayOfMonth(1))));
+ DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC).withDayOfMonth(1))));
}

@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/37037")
public void testExpression_CustomFormat() throws Exception {
List<String> results = expressionResolver.resolve(context, Arrays.asList("<.marvel-{now/d{yyyy.MM.dd}}>"));
assertThat(results.size(), equalTo(1));
Expand All @@ -106,7 +105,6 @@ public void testExpression_EscapeStatic() throws Exception {
equalTo(".mar{v}el-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
}

@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/37037")
public void testExpression_EscapeDateFormat() throws Exception {
List<String> result = expressionResolver.resolve(context, Arrays.asList("<.marvel-{now/d{'\\{year\\}'yyyy}}>"));
assertThat(result.size(), equalTo(1));
Expand All @@ -116,18 +114,17 @@ public void testExpression_EscapeDateFormat() throws Exception {

public void testExpression_MixedArray() throws Exception {
List<String> result = expressionResolver.resolve(context, Arrays.asList(
"name1", "<.marvel-{now/d}>", "name2", "<.logstash-{now/M{YYYY.MM}}>"
"name1", "<.marvel-{now/d}>", "name2", "<.logstash-{now/M{yyyy.MM}}>"
));
assertThat(result.size(), equalTo(4));
assertThat(result.get(0), equalTo("name1"));
assertThat(result.get(1),
equalTo(".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
equalTo(".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(new DateTime(context.getStartTime(), UTC))));
assertThat(result.get(2), equalTo("name2"));
assertThat(result.get(3), equalTo(".logstash-" +
DateTimeFormat.forPattern("YYYY.MM").print(new DateTime(context.getStartTime(), UTC).withDayOfMonth(1))));
DateTimeFormat.forPattern("yyyy.MM").print(new DateTime(context.getStartTime(), UTC).withDayOfMonth(1))));
}

@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/37037")
public void testExpression_CustomTimeZoneInIndexName() throws Exception {
DateTimeZone timeZone;
int hoursOffset;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -662,7 +662,7 @@ public void testsTimeZoneParsing() {
// Formatter used to print and parse the sample date.
// Printing the date works but parsing it back fails
// with Joda 2.9.4
DateTimeFormatter formatter = DateTimeFormat.forPattern("YYYY-MM-dd'T'HH:mm:ss " + randomFrom("ZZZ", "[ZZZ]", "'['ZZZ']'"));
DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss " + randomFrom("ZZZ", "[ZZZ]", "'['ZZZ']'"));

String dateTimeAsString = formatter.print(expected);
assertThat(dateTimeAsString, startsWith("2016-11-10T05:37:59 "));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@ public class DateMathIndexExpressionsIntegrationIT extends ESIntegTestCase {

public void testIndexNameDateMathExpressions() {
DateTime now = new DateTime(DateTimeZone.UTC);
String index1 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now);
String index2 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.minusDays(1));
String index3 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.minusDays(2));
String index1 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now);
String index2 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now.minusDays(1));
String index3 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now.minusDays(2));
createIndex(index1, index2, index3);

GetSettingsResponse getSettingsResponse = client().admin().indices().prepareGetSettings(index1, index2, index3).get();
Expand Down Expand Up @@ -110,9 +110,9 @@ public void testIndexNameDateMathExpressions() {

public void testAutoCreateIndexWithDateMathExpression() throws Exception {
DateTime now = new DateTime(DateTimeZone.UTC);
String index1 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now);
String index2 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.minusDays(1));
String index3 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.minusDays(2));
String index1 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now);
String index2 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now.minusDays(1));
String index3 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now.minusDays(2));

String dateMathExp1 = "<.marvel-{now/d}>";
String dateMathExp2 = "<.marvel-{now/d-1d}>";
Expand All @@ -134,9 +134,9 @@ public void testAutoCreateIndexWithDateMathExpression() throws Exception {

public void testCreateIndexWithDateMathExpression() throws Exception {
DateTime now = new DateTime(DateTimeZone.UTC);
String index1 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now);
String index2 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.minusDays(1));
String index3 = ".marvel-" + DateTimeFormat.forPattern("YYYY.MM.dd").print(now.minusDays(2));
String index1 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now);
String index2 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now.minusDays(1));
String index3 = ".marvel-" + DateTimeFormat.forPattern("yyyy.MM.dd").print(now.minusDays(2));

String dateMathExp1 = "<.marvel-{now/d}>";
String dateMathExp2 = "<.marvel-{now/d-1d}>";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -232,8 +232,8 @@ public void testYearOfEra() {
}

public void testToString1() {
assertMethodDeprecation(() -> assertThat(javaTime.toString("YYYY/MM/dd HH:mm:ss.SSS"),
equalTo(jodaTime.toString("YYYY/MM/dd HH:mm:ss.SSS"))), "toString(String)", "a DateTimeFormatter");
assertMethodDeprecation(() -> assertThat(javaTime.toString("yyyy/MM/dd HH:mm:ss.SSS"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused by reading this. Shouldnt one be in capital letters and one not in order for this to work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java time cares significantly about Y vs y. Afaict, Joda time only uses Y vs y to exclude 0 and negative numbers (actually, the documentation I can find differs about whether or not 0 is a valid value, but...).

That said, y in joda is the "year" w/o the constraint, which is a closer map to Java y.

equalTo(jodaTime.toString("yyyy/MM/dd HH:mm:ss.SSS"))), "toString(String)", "a DateTimeFormatter");
}

public void testToString2() {
Expand Down
2 changes: 1 addition & 1 deletion x-pack/docs/en/watcher/release-notes.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ October 28, 2015
they did before the upgrade. For example if `watcher.dynamic_indices.time_zone`
setting was set to `+01:00` and a watch has the following index name
`<logstash-{now/d}>` then after the upgrade you need to update this watch to
use the following index name `<logstash-{now/d{YYYY.MM.dd|+01:00}}>`.
use the following index name `<logstash-{now/d{yyyy.MM.dd|+01:00}}>`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are old release notes, that might be outdated in other ways as well, I am not sure if it is worth updating?


.New Features
* Added new <<actions-hipchat, HipChat Action>>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@
*/
public class Cron implements ToXContentFragment {
protected static final TimeZone UTC = DateTimeZone.UTC.toTimeZone();
protected static final DateTimeFormatter formatter = DateTimeFormat.forPattern("YYYY-MM-dd'T'HH:mm:ss");
protected static final DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss");

private static final int SECOND = 0;
private static final int MINUTE = 1;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ public abstract class Exporter implements AutoCloseable {
Setting.affixKeySetting("xpack.monitoring.exporters.","index.name.time_format",
key -> Setting.simpleString(key, Property.Dynamic, Property.NodeScope));

private static final String INDEX_FORMAT = "YYYY.MM.dd";
private static final String INDEX_FORMAT = "yyyy.MM.dd";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the monitoring side of things is fixed in #35809


protected final Config config;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ private void checkMonitoringDocs() {
.get("xpack.monitoring.exporters._local.index.name.time_format");
assertEquals(indexTimeFormat, customTimeFormat);
if (customTimeFormat == null) {
customTimeFormat = "YYYY.MM.dd";
customTimeFormat = "yyyy.MM.dd";
}

DateTimeFormatter dateParser = ISODateTimeFormat.dateTime().withZoneUTC();
Expand Down
Loading