Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding CONVERT_TZ and DATETIME functions to SQL and PPL #848

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
1e64b3d
Added `CONVERT_TZ` to the PPL lexer/parser and the SQL lexer/parser.
MitchellGale Aug 10, 2022
9349b6f
Added `convert_tz` to the BuiltinFunctionName.java.
MitchellGale Aug 10, 2022
eedd00b
Added convert_tz to DateTimeFunction.java register, private FunctionR…
MitchellGale Aug 12, 2022
b00d3de
Added IT PPL and SQL tests for various conditions including time zone…
MitchellGale Aug 16, 2022
2766e8d
Added DATETIME to OpenSearchPPLParser.g4 and OpenSearchSQLParser.g4.
MitchellGale Aug 16, 2022
7da57c0
Rebase merge conflict resolution.
MitchellGale Sep 16, 2022
454e600
Added ppl doctest for convert_tz
MitchellGale Aug 17, 2022
3e83284
Completed implementation for datetime and convert_Tz.
MitchellGale Aug 19, 2022
74b728f
Removed SQL test from PPL IT test
MitchellGale Aug 19, 2022
01b0e23
Removed redundant convert to string.
MitchellGale Aug 19, 2022
d233bf9
Fixed doctests and reverted changes to adddate function in DateTimeFu…
MitchellGale Aug 22, 2022
619fbd2
Fixed doctest.
MitchellGale Aug 22, 2022
5dbafbe
Fixed doctest.
MitchellGale Aug 22, 2022
ea9311a
Added additional integration tests for PPL and SQL tests for convert_tz.
MitchellGale Aug 22, 2022
a4e872c
Added null for timezones outside of basic range for DATETIME.
MitchellGale Aug 23, 2022
7713d5b
Added test cases for null with the datetime function. Made DateTime f…
MitchellGale Aug 25, 2022
cc797c7
Seperated out test from DateTimeFunction.java.
MitchellGale Aug 30, 2022
106ba99
Updated tests. Fixed exception to be less general.
MitchellGale Sep 1, 2022
e02aeed
Removed changes.
MitchellGale Sep 1, 2022
e6b72b6
Fixed rel timezone issue.
MitchellGale Sep 1, 2022
497e6e5
Added support for non valid datetime to return null for convert_tz.
MitchellGale Sep 1, 2022
42fab1f
Made exception more verbose.
MitchellGale Sep 1, 2022
dfc28c6
Removed unneeded format changes in DateTimeFunction.java.
MitchellGale Sep 1, 2022
410c0dd
Added more doctests.
MitchellGale Sep 1, 2022
5b1b670
Removed formatting changes.
MitchellGale Sep 1, 2022
ae2b864
Reverting sql/ppl DateTimeFunctionsIT.java.
MitchellGale Sep 1, 2022
76fd965
Reverted changes to DateTimeFunctionTest.java
MitchellGale Sep 2, 2022
afb2347
Added more information about invalid date for convert_tz
MitchellGale Sep 2, 2022
e0d4c1c
Converted "from Field" and "To Field" to use "Fieldn" where n is the …
MitchellGale Sep 2, 2022
de0c857
Added date validation. Added test cases in IT to cover cases. Added t…
MitchellGale Sep 7, 2022
af3eb4c
Fixed date validation function. Broke up some unit tests.
MitchellGale Sep 8, 2022
0e4accc
Fixed formatting.
MitchellGale Sep 8, 2022
ee67db2
Added DateTime tests, broke up functions.
MitchellGale Sep 8, 2022
7a0f930
Added space in DateTimeTest.java.
MitchellGale Sep 8, 2022
e16fcaa
Tidied up code and tests.
MitchellGale Sep 9, 2022
24b22b2
Fixed local date time rst test.
MitchellGale Sep 12, 2022
17a5642
Removed nested try/catch exceptions.
MitchellGale Sep 12, 2022
1af5b76
removed exprConvertTZ function call from within try catch statement.
MitchellGale Sep 12, 2022
3312f26
Reverted change from parse localdate
MitchellGale Sep 12, 2022
3e302da
Removed extra casting around fromTz variable.
MitchellGale Sep 12, 2022
6b72f23
Updated wording for functions.rst
MitchellGale Sep 12, 2022
bbf47ff
Updated wording for datetime.rst to describe null for conert_tz
MitchellGale Sep 12, 2022
d5a63bd
Added more test cases for functions.rst
MitchellGale Sep 12, 2022
a82df6a
Fixed doctests.
MitchellGale Sep 12, 2022
5cf21a7
Removed extra import.
MitchellGale Sep 12, 2022
5f7c7f8
Renamed isValidTimeZone function to isValidMySqlTimeZoneId.
MitchellGale Sep 13, 2022
b012512
Has ExprDatetimeValue doing work for exprConvertTZ call.
MitchellGale Sep 14, 2022
a1e301b
Deleted fromTZ
MitchellGale Sep 14, 2022
f6ecf8d
Moved fixed variables to top of class.
MitchellGale Sep 15, 2022
528c129
Added Null to support of exprDateTime.
MitchellGale Sep 15, 2022
0869b92
Added missing expr functions for makedate/time
MitchellGale Sep 16, 2022
f6a4500
cleaning up after rebase merge.
MitchellGale Sep 16, 2022
a7f77f3
Rebase merge conflict resolution.
MitchellGale Sep 16, 2022
2ee52d8
Fixed missing DATETIME in SQL Parser.
MitchellGale Sep 19, 2022
4cde931
Fixed IT test.
MitchellGale Sep 19, 2022
1fbed3f
Addressed PR comments
MitchellGale Sep 26, 2022
d28906e
Added missing variables after rebase
MitchellGale Sep 28, 2022
9b9250e
Fixed checkstyle errors after rebase.
MitchellGale Sep 28, 2022
eb0c894
Moved formatter for date time over.
MitchellGale Sep 28, 2022
3d64678
Changed function resolved to default
MitchellGale Sep 28, 2022
abae961
Removed unneeded code
MitchellGale Sep 28, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated tests. Fixed exception to be less general.
Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>
  • Loading branch information
MitchellGale committed Sep 28, 2022
commit 106ba99299360c14ec41821d1b766261c5f26b48
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
import java.math.RoundingMode;
import java.text.DecimalFormat;
import java.time.Instant;
import java.time.DateTimeException;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.ZoneId;
Expand All @@ -38,6 +39,7 @@
import java.time.format.DateTimeParseException;
import java.time.format.TextStyle;
import java.util.Locale;
import java.util.Set;
import java.util.TimeZone;
import java.util.concurrent.TimeUnit;
import javax.annotation.Nullable;
Expand Down Expand Up @@ -613,19 +615,23 @@ private ExprValue exprAddDateDays(ExprValue date, ExprValue days) {
* @return DateTime that has been converted to the to_tz timezone.
*/
private ExprValue exprConvertTZ(ExprValue startingDateTime, ExprValue fromTz, ExprValue toTz) {
ZoneId convertedFromTz = ZoneId.of(fromTz.stringValue());
ZoneId convertedToTz = ZoneId.of(toTz.stringValue());
try {
ZoneId convertedFromTz = ZoneId.of(fromTz.stringValue());
ZoneId convertedToTz = ZoneId.of(toTz.stringValue());

if (!isTimeZoneValid(convertedFromTz)
if (!isTimeZoneValid(convertedFromTz)
|| !isTimeZoneValid(convertedToTz)) {
return ExprNullValue.of();
}
return ExprNullValue.of();
}

ZonedDateTime zonedDateTime =
startingDateTime.datetimeValue().atZone(convertedFromTz);
ZonedDateTime zonedDateTime =
startingDateTime.datetimeValue().atZone(convertedFromTz);

return new ExprDatetimeValue(
zonedDateTime.withZoneSameInstant(convertedToTz).toLocalDateTime());
return new ExprDatetimeValue(
zonedDateTime.withZoneSameInstant(convertedToTz).toLocalDateTime());
} catch (DateTimeException e) {
return ExprNullValue.of();
}
}

/**
Expand Down Expand Up @@ -668,7 +674,7 @@ private ExprValue exprDateTime(ExprValue dateTime, ExprValue timeZone) {
new ExprDatetimeValue(zdtWithZoneOffset.toLocalDateTime()),
new ExprStringValue(String.valueOf(fromTZ)),
new ExprStringValue(String.valueOf(ZoneId.of(timeZone.stringValue()))));
} catch (Exception e) {
} catch (DateTimeParseException e) {
LocalDateTime ldtFormatted = LocalDateTime.parse(dateTime.stringValue(), formatDT);

convertTZResult = exprConvertTZ(
Expand Down Expand Up @@ -1076,14 +1082,10 @@ private Boolean isTimeZoneValid(ZoneId zone) {
ZonedDateTime passedTzValidator =
defaultDateTime.atZone(defaultTz).withZoneSameInstant(zone).withZoneSameLocal(defaultTz);

if ((passedTzValidator.isBefore(maxTzValidator)
return (passedTzValidator.isBefore(maxTzValidator)
|| passedTzValidator.isEqual(maxTzValidator))
&& (passedTzValidator.isAfter(minTzValidator)
|| passedTzValidator.isEqual(minTzValidator))) {
return true;
}

return false;
|| passedTzValidator.isEqual(minTzValidator));
}

/**
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/


package org.opensearch.sql.expression.datetime;


import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.when;
import static org.opensearch.sql.data.model.ExprValueUtils.missingValue;
import static org.opensearch.sql.data.model.ExprValueUtils.nullValue;
import static org.opensearch.sql.data.type.ExprCoreType.DATETIME;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.opensearch.sql.data.model.ExprDatetimeValue;
import org.opensearch.sql.data.model.ExprValue;
import org.opensearch.sql.expression.DSL;
import org.opensearch.sql.expression.Expression;
import org.opensearch.sql.expression.ExpressionTestBase;
import org.opensearch.sql.expression.FunctionExpression;
import org.opensearch.sql.expression.env.Environment;



@ExtendWith(MockitoExtension.class)
class ConvertTZTest extends ExpressionTestBase {

@Mock
Environment<Expression, ExprValue> env;

@Mock
Expression nullRef;

@Mock
Expression missingRef;

@BeforeEach
public void setup() {
when(nullRef.valueOf(env)).thenReturn(nullValue());
when(missingRef.valueOf(env)).thenReturn(missingValue());
}

@Test
public void convert_tz() {
when(nullRef.type()).thenReturn(DATETIME);
when(missingRef.type()).thenReturn(DATETIME);
assertEquals(nullValue(), eval(dsl.date(nullRef)));
assertEquals(missingValue(), eval(dsl.date(missingRef)));

FunctionExpression expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("+10:00"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-16 08:00:00"), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("+16:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("-16:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+15:00"),
DSL.literal("+01:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("-15:00"),
DSL.literal("+01:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("-12:00"),
DSL.literal("+15:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("+14:00"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-16 12:00:00"), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:01"),
DSL.literal("-13:59"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-15 08:00:00"), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+)()"),
DSL.literal("+12:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("fesasgrsgr"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));
}

private ExprValue eval(Expression expression) {
return expression.valueOf(env);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@
import org.opensearch.sql.data.model.ExprDateValue;
import org.opensearch.sql.data.model.ExprDatetimeValue;
import org.opensearch.sql.data.model.ExprLongValue;
import org.opensearch.sql.data.model.ExprNullValue;
import org.opensearch.sql.data.model.ExprTimeValue;
import org.opensearch.sql.data.model.ExprTimestampValue;
import org.opensearch.sql.data.model.ExprValue;
Expand Down Expand Up @@ -225,63 +224,6 @@ public void adddate() {
assertEquals(missingValue(), eval(dsl.adddate(nullRef, missingRef)));
}

@Test
public void convert_tz() {
when(nullRef.type()).thenReturn(DATETIME);
when(missingRef.type()).thenReturn(DATETIME);
assertEquals(nullValue(), eval(dsl.date(nullRef)));
assertEquals(missingValue(), eval(dsl.date(missingRef)));

FunctionExpression expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("+10:00"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-16 08:00:00"), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("+16:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("-16:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+15:00"),
DSL.literal("+01:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("-15:00"),
DSL.literal("+01:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("-12:00"),
DSL.literal("+15:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.convert_tz(dsl.datetime(
DSL.literal("2008-05-15 22:00:00")),
DSL.literal("+00:00"),
DSL.literal("+14:00"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-16 12:00:00"), expr.valueOf(env));
}

@Test
public void date() {
when(nullRef.type()).thenReturn(DATE);
Expand Down Expand Up @@ -329,12 +271,6 @@ public void date_add() {
assertEquals(new ExprDatetimeValue("2020-08-26 01:00:00"), expr.valueOf(env));
assertEquals("date_add(\"2020-08-26\", interval(1, \"hour\"))", expr.toString());

expr = dsl
.date_add(DSL.literal("2020-08-26"), dsl.interval(DSL.literal(0), DSL.literal("hour")));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2020-08-26 00:00:00"), expr.valueOf(env));
assertEquals("date_add(\"2020-08-26\", interval(0, \"hour\"))", expr.toString());

when(nullRef.type()).thenReturn(DATE);
assertEquals(nullValue(), eval(dsl.date_add(nullRef, DSL.literal(1L))));
assertEquals(nullValue(),
Expand Down Expand Up @@ -425,52 +361,6 @@ public void date_sub() {
assertEquals(missingValue(), eval(dsl.date_sub(nullRef, missingRef)));
}

@Test
public void date_time() {
when(nullRef.type()).thenReturn(DATETIME);
when(missingRef.type()).thenReturn(DATETIME);
assertEquals(nullValue(), eval(dsl.date(nullRef)));
assertEquals(missingValue(), eval(dsl.date(missingRef)));

FunctionExpression expr = dsl.datetime(DSL.literal("2008-05-15 22:00:00"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-15 22:00:00"), expr.valueOf(env));

expr = dsl.datetime(DSL.literal("2008-05-15 22:00:00+01:00"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-15 22:00:00"), expr.valueOf(env));

expr = dsl.datetime(DSL.literal("2008-05-15 22:00:00+01:00"),
DSL.literal("America/Los_Angeles"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-15 14:00:00"), expr.valueOf(env));

expr = dsl.datetime(DSL.literal("2008-05-15 22:00:00"),
DSL.literal("America/Los_Angeles"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-15 15:00:00"), expr.valueOf(env));

expr = dsl.datetime(DSL.literal("2008-05-15 22:00:00-11:00"),
DSL.literal("America/Los_Angeles"));
assertEquals(DATETIME, expr.type());
assertEquals(new ExprDatetimeValue("2008-05-16 02:00:00"), expr.valueOf(env));

expr = dsl.datetime(DSL.literal("2008-05-15 22:00:00-12:00"),
DSL.literal("+15:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.datetime(DSL.literal("2008-05-15 22:00:00-13:00"),
DSL.literal("+10:00"));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));

expr = dsl.datetime(DSL.literal("2008-05-15 22:00:00-11:00"),
DSL.literal(nullValue()));
assertEquals(DATETIME, expr.type());
assertEquals(nullValue(), expr.valueOf(env));
}

@Test
public void day() {
when(nullRef.type()).thenReturn(DATE);
Expand Down Expand Up @@ -1113,4 +1003,4 @@ void testDateFormat(DateFormatTester dft) {
private ExprValue eval(Expression expression) {
return expression.valueOf(env);
}
}
}
Loading