Skip to content
This repository was archived by the owner on Aug 2, 2022. It is now read-only.

Commit f33c0e6

Browse files
lyndonbautopenghuojoshuali925jordanw-bqchloe-zh
authored
DATE_FORMAT function (#764)
* Bug fix, support long type for aggregation (#522) * Bug fix, support long type for aggregation * change to datetime to JDBC format * Opendistro Release 1.9.0 (#532) * prepare odfe 1.9 * Fix all ES 7.8 compile and build errors * Revert changes as Lombok is working now * Update CustomExternalTestCluster.java * Fix license headers check * Use splitFieldsByMetadata to separate fields when calling SearchHit constructor * More fixes for ODFE 1.9 * Remove todo statement * Add ODFE 1.9.0 release notes * Rename release notes to use 4 digit versions (#547) * Revert changes ahead of develop branch in master (#551) * Revert "Rename release notes to use 4 digit versions (#547)" This reverts commit 33c6d3e. * Revert "Opendistro Release 1.9.0 (#532)" This reverts commit 254f2e0. * Revert "Bug fix, support long type for aggregation (#522)" This reverts commit fb2ed91. * Merge all SQL repos and adjust workflows (#549) (#554) * merge all sql repos * fix test and build workflows * fix workbench and odbc path * fix workbench and odbc path * restructure workbench dir and fix workflows * fix workbench workflow * fix workbench workflow * fix workbench workflow * fix workbench workflow * fix workbench workflow * revert workbench directory structure * fix workbench workflow * fix workbench workflow * fix workbench workflow * fix workbench workflow * update workbench workflow for release * Delete .github/ in sql-workbench directory * Add cypress to sql-workbench * Sync latest ODBC commits * Sync latest workbench commits (will add cypress in separate PR) * Add ignored ODBC libs * add date and time support (#560) * add date and time support * update doc * update doc * Revert "add date and time support (#560)" (#567) This reverts commit 4b33a2f. * add error details for all server communication errors (#645) - add null check to avoid crashing if details not initialized * Revert "add error details for all server communication errors (#645)" (#653) This reverts commit c11125d. * Fix download link in package description (#729) * add functions day, month, quarter, year * fix build error * fix doctest error * fix doctest build error * fix doctest * add dayofmonth() * add dayofyear() * add dayofweek() * fix dayofweek logic & add unit test * fix doctest for dayofweek() * add dayname * add monthname * fix checkstyle build error * fix build error * fix doctest for monthname * add hour() * add minute() * add second * add microsecond * fix datetime & timestamp issue for microsecond * add time_to_sec * add subdate & date_sub * fix doctest error * fix build error * add KeywordsCanBeId for dayofweek * add to_days * add from_days() * arrange by alphabetical order * add manual IT * add string input for date functions * fix microsecond * update doc * add date_add * add week * address PR comments * update tests & doc * fix doc format * update tests for adddate * move string conversion to ExprStringValue * add string type in doc * edge case * fix case 5 & 7 * add IT * update doc * fix table * rename * add string type * nit: add newline * fix type in comment * nit * add test cases for datetime function in ExpeStringValue * removing implicit def for keyword in parser * add dayofweek * [1] Merged rupals week branch in * add unit tests for null, missing values * nit * [1] Added integration tests. * [1] Working on ppl integration tests. * [1] Updated for integration tests * [1] Fixed schema verification * [1] Adding documentation. * [1] Fixing documentation * [1] Simplified a bunch of logic * [1] Reducing changes that are from spacing * [1] Removed extra merge line * address PR comment * [1] Updates * [1] Removed some unwanted changes. * [1] Minor whitespace adjustements * [1] Updating based on code review Co-authored-by: Peng Huo <penghuo@gmail.com> Co-authored-by: Joshua <joshuali925@gmail.com> Co-authored-by: Joshua Li <lijshu@amazon.com> Co-authored-by: Jordan Wilson <37088125+jordanw-bq@users.noreply.github.com> Co-authored-by: Chloe <chloezh1102@gmail.com> Co-authored-by: chloe-zh <fizhang@amazon.com> Co-authored-by: Sayali Gaikawad <61760125+gaiksaya@users.noreply.github.com> Co-authored-by: Rupal Mahajan <>
1 parent d75b7f1 commit f33c0e6

File tree

13 files changed

+450
-20
lines changed

13 files changed

+450
-20
lines changed

‎core/src/main/java/com/amazon/opendistroforelasticsearch/sql/data/model/ExprStringValue.java‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,7 @@
2121
import java.time.LocalDate;
2222
import java.time.LocalDateTime;
2323
import java.time.LocalTime;
24-
import java.time.format.DateTimeParseException;
2524
import java.util.Objects;
26-
import lombok.EqualsAndHashCode;
2725
import lombok.RequiredArgsConstructor;
2826

2927
/**

‎core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/DSL.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -329,6 +329,10 @@ public FunctionExpression timestamp(Expression... expressions) {
329329
return function(BuiltinFunctionName.TIMESTAMP, expressions);
330330
}
331331

332+
public FunctionExpression date_format(Expression... expressions) {
333+
return function(BuiltinFunctionName.DATE_FORMAT, expressions);
334+
}
335+
332336
public FunctionExpression to_days(Expression... expressions) {
333337
return function(BuiltinFunctionName.TO_DAYS, expressions);
334338
}

‎core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/CalendarLookup.java‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,9 @@ private static Calendar getCalendar(int mode, LocalDate date) {
4747

4848
/**
4949
* Set first day of week, minimal days in first week and date in calendar.
50-
* @param firstDayOfWeek the given first day of the week.
50+
* @param firstDayOfWeek the given first day of the week.
5151
* @param minimalDaysInWeek the given minimal days required in the first week of the year.
52-
* @param date the given date.
52+
* @param date the given date.
5353
*/
5454
private static Calendar getCalendar(int firstDayOfWeek, int minimalDaysInWeek, LocalDate date) {
5555
Calendar calendar = Calendar.getInstance();
@@ -74,4 +74,19 @@ static int getWeekNumber(int mode, LocalDate date) {
7474
}
7575
return weekNumber;
7676
}
77-
}
77+
78+
/**
79+
* Returns year for date according to mode.
80+
* @param mode Integer for mode. Valid mode values are 0 to 7.
81+
* @param date LocalDate for date.
82+
*/
83+
static int getYearNumber(int mode, LocalDate date) {
84+
Calendar calendar = getCalendar(mode, date);
85+
int weekNumber = getWeekNumber(mode, date);
86+
int yearNumber = calendar.get(Calendar.YEAR);
87+
if ((weekNumber > 51) && (calendar.get(Calendar.DAY_OF_MONTH) < 7)) {
88+
yearNumber--;
89+
}
90+
return yearNumber;
91+
}
92+
}

‎core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFormatterUtil.java‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
package com.amazon.opendistroforelasticsearch.sql.expression.datetime;
2+
3+
import com.amazon.opendistroforelasticsearch.sql.data.model.ExprStringValue;
4+
import com.amazon.opendistroforelasticsearch.sql.data.model.ExprValue;
5+
import com.google.common.collect.ImmutableMap;
6+
7+
import java.time.LocalDateTime;
8+
import java.time.format.DateTimeFormatter;
9+
import java.util.Locale;
10+
import java.util.Map;
11+
import java.util.regex.Matcher;
12+
import java.util.regex.Pattern;
13+
14+
/**
15+
* This class converts a SQL style DATE_FORMAT format specifier and converts it to a
16+
* Java SimpleDateTime format.
17+
*/
18+
class DateTimeFormatterUtil {
19+
private static final int SUFFIX_SPECIAL_START_TH = 11;
20+
private static final int SUFFIX_SPECIAL_END_TH = 13;
21+
private static final String SUFFIX_SPECIAL_TH = "th";
22+
private static final Map<Integer, String> SUFFIX_CONVERTER =
23+
ImmutableMap.<Integer, String>builder()
24+
.put(1, "st").put(2, "nd").put(3, "rd").build();
25+
26+
// The following have special cases that need handling outside of the format options provided
27+
// by the DateTimeFormatter class.
28+
interface DateTimeFormatHandler {
29+
String getFormat(LocalDateTime date);
30+
}
31+
32+
private static final Map<String, DateTimeFormatHandler> HANDLERS =
33+
ImmutableMap.<String, DateTimeFormatHandler>builder()
34+
.put("%a", (date) -> "EEE") // %a => EEE - Abbreviated weekday name (Sun..Sat)
35+
.put("%b", (date) -> "LLL") // %b => LLL - Abbreviated month name (Jan..Dec)
36+
.put("%c", (date) -> "MM") // %c => MM - Month, numeric (0..12)
37+
.put("%d", (date) -> "dd") // %d => dd - Day of the month, numeric (00..31)
38+
.put("%e", (date) -> "d") // %e => d - Day of the month, numeric (0..31)
39+
.put("%H", (date) -> "HH") // %H => HH - (00..23)
40+
.put("%h", (date) -> "hh") // %h => hh - (01..12)
41+
.put("%I", (date) -> "hh") // %I => hh - (01..12)
42+
.put("%i", (date) -> "mm") // %i => mm - Minutes, numeric (00..59)
43+
.put("%j", (date) -> "DDD") // %j => DDD - (001..366)
44+
.put("%k", (date) -> "H") // %k => H - (0..23)
45+
.put("%l", (date) -> "h") // %l => h - (1..12)
46+
.put("%p", (date) -> "a") // %p => a - AM or PM
47+
.put("%M", (date) -> "LLLL") // %M => LLLL - Month name (January..December)
48+
.put("%m", (date) -> "MM") // %m => MM - Month, numeric (00..12)
49+
.put("%r", (date) -> "hh:mm:ss a") // %r => hh:mm:ss a - hh:mm:ss followed by AM or PM
50+
.put("%S", (date) -> "ss") // %S => ss - Seconds (00..59)
51+
.put("%s", (date) -> "ss") // %s => ss - Seconds (00..59)
52+
.put("%T", (date) -> "HH:mm:ss") // %T => HH:mm:ss
53+
.put("%W", (date) -> "EEEE") // %W => EEEE - Weekday name (Sunday..Saturday)
54+
.put("%Y", (date) -> "yyyy") // %Y => yyyy - Year, numeric, 4 digits
55+
.put("%y", (date) -> "yy") // %y => yy - Year, numeric, 2 digits
56+
// The following are not directly supported by DateTimeFormatter.
57+
.put("%D", (date) -> // %w - Day of month with English suffix
58+
String.format("'%d%s'", date.getDayOfMonth(), getSuffix(date.getDayOfMonth())))
59+
.put("%f", (date) -> // %f - Microseconds
60+
String.format("'%d'", (date.getNano() / 1000)))
61+
.put("%w", (date) -> // %w - Day of week (0 indexed)
62+
String.format("'%d'", date.getDayOfWeek().getValue()))
63+
.put("%U", (date) -> // %U Week where Sunday is the first day - WEEK() mode 0
64+
String.format("'%d'", CalendarLookup.getWeekNumber(0, date.toLocalDate())))
65+
.put("%u", (date) -> // %u Week where Monday is the first day - WEEK() mode 1
66+
String.format("'%d'", CalendarLookup.getWeekNumber(1, date.toLocalDate())))
67+
.put("%V", (date) -> // %V Week where Sunday is the first day - WEEK() mode 2 used with %X
68+
String.format("'%d'", CalendarLookup.getWeekNumber(2, date.toLocalDate())))
69+
.put("%v", (date) -> // %v Week where Monday is the first day - WEEK() mode 3 used with %x
70+
String.format("'%d'", CalendarLookup.getWeekNumber(3, date.toLocalDate())))
71+
.put("%X", (date) -> // %X Year for week where Sunday is the first day, 4 digits used with %V
72+
String.format("'%d'", CalendarLookup.getYearNumber(2, date.toLocalDate())))
73+
.put("%x", (date) -> // %x Year for week where Monday is the first day, 4 digits used with %v
74+
String.format("'%d'", CalendarLookup.getYearNumber(3, date.toLocalDate())))
75+
.build();
76+
77+
private static final Pattern pattern = Pattern.compile("%.");
78+
private static final String MOD_LITERAL = "%";
79+
80+
private DateTimeFormatterUtil() {
81+
}
82+
83+
/**
84+
* Format the date using the date format String.
85+
* @param dateExpr the date ExprValue of Date/Datetime/Timestamp/String type.
86+
* @param formatExpr the format ExprValue of String type.
87+
* @return Date formatted using format and returned as a String.
88+
*/
89+
static ExprValue getFormattedDate(ExprValue dateExpr, ExprValue formatExpr) {
90+
final LocalDateTime date = dateExpr.datetimeValue();
91+
final Matcher matcher = pattern.matcher(formatExpr.stringValue());
92+
final StringBuffer format = new StringBuffer();
93+
while (matcher.find()) {
94+
matcher.appendReplacement(format,
95+
HANDLERS.getOrDefault(matcher.group(), (d) ->
96+
String.format("'%s'", matcher.group().replaceFirst(MOD_LITERAL, "")))
97+
.getFormat(date));
98+
}
99+
matcher.appendTail(format);
100+
101+
// English Locale matches SQL requirements.
102+
// 'AM'/'PM' instead of 'a.m.'/'p.m.'
103+
// 'Sat' instead of 'Sat.' etc
104+
return new ExprStringValue(date.format(
105+
DateTimeFormatter.ofPattern(format.toString(), Locale.ENGLISH)));
106+
}
107+
108+
/**
109+
* Returns English suffix of incoming value.
110+
* @param val Incoming value.
111+
* @return English suffix as String (st, nd, rd, th)
112+
*/
113+
private static String getSuffix(int val) {
114+
// The numbers 11, 12, and 13 do not follow general suffix rules.
115+
if ((SUFFIX_SPECIAL_START_TH <= val) && (val <= SUFFIX_SPECIAL_END_TH)) {
116+
return SUFFIX_SPECIAL_TH;
117+
}
118+
return SUFFIX_CONVERTER.getOrDefault(val % 10, SUFFIX_SPECIAL_TH);
119+
}
120+
}

‎core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ public void register(BuiltinFunctionRepository repository) {
8585
repository.register(time());
8686
repository.register(time_to_sec());
8787
repository.register(timestamp());
88+
repository.register(date_format());
8889
repository.register(to_days());
8990
repository.register(week());
9091
repository.register(year());
@@ -400,6 +401,27 @@ private FunctionResolver year() {
400401
);
401402
}
402403

404+
/**
405+
* Formats date according to format specifier. First argument is date, second is format.
406+
* Detailed supported signatures:
407+
* (STRING, STRING) -> STRING
408+
* (DATE, STRING) -> STRING
409+
* (DATETIME, STRING) -> STRING
410+
* (TIMESTAMP, STRING) -> STRING
411+
*/
412+
private FunctionResolver date_format() {
413+
return define(BuiltinFunctionName.DATE_FORMAT.getName(),
414+
impl(nullMissingHandling(DateTimeFormatterUtil::getFormattedDate),
415+
STRING, STRING, STRING),
416+
impl(nullMissingHandling(DateTimeFormatterUtil::getFormattedDate),
417+
STRING, DATE, STRING),
418+
impl(nullMissingHandling(DateTimeFormatterUtil::getFormattedDate),
419+
STRING, DATETIME, STRING),
420+
impl(nullMissingHandling(DateTimeFormatterUtil::getFormattedDate),
421+
STRING, TIMESTAMP, STRING)
422+
);
423+
}
424+
403425
/**
404426
* ADDDATE function implementation for ExprValue.
405427
*

‎core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/function/BuiltinFunctionName.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ public enum BuiltinFunctionName {
7272
TIME(FunctionName.of("time")),
7373
TIME_TO_SEC(FunctionName.of("time_to_sec")),
7474
TIMESTAMP(FunctionName.of("timestamp")),
75+
DATE_FORMAT(FunctionName.of("date_format")),
7576
TO_DAYS(FunctionName.of("to_days")),
7677
WEEK(FunctionName.of("week")),
7778
YEAR(FunctionName.of("year")),

‎core/src/test/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunctionTest.java‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@
4747
import com.amazon.opendistroforelasticsearch.sql.expression.ExpressionTestBase;
4848
import com.amazon.opendistroforelasticsearch.sql.expression.FunctionExpression;
4949
import com.amazon.opendistroforelasticsearch.sql.expression.env.Environment;
50+
import com.google.common.collect.ImmutableList;
51+
import java.util.List;
52+
import lombok.AllArgsConstructor;
5053
import org.junit.jupiter.api.BeforeEach;
5154
import org.junit.jupiter.api.Test;
5255
import org.junit.jupiter.api.extension.ExtendWith;
@@ -71,6 +74,89 @@ public void setup() {
7174
when(missingRef.valueOf(env)).thenReturn(missingValue());
7275
}
7376

77+
final List<DateFormatTester> dateFormatTesters = ImmutableList.of(
78+
new DateFormatTester("1998-01-31 13:14:15.012345",
79+
ImmutableList.of("%H","%I","%k","%l","%i","%p","%r","%S","%T"," %M",
80+
"%W","%D","%Y","%y","%a","%b","%j","%m","%d","%h","%s","%w","%f",
81+
"%q","%"),
82+
ImmutableList.of("13","01","13","1","14","PM","01:14:15 PM","15","13:14:15"," January",
83+
"Saturday","31st","1998","98","Sat","Jan","031","01","31","01","15","6","12345",
84+
"q","%")
85+
),
86+
new DateFormatTester("1999-12-01",
87+
ImmutableList.of("%D"),
88+
ImmutableList.of("1st")
89+
),
90+
new DateFormatTester("1999-12-02",
91+
ImmutableList.of("%D"),
92+
ImmutableList.of("2nd")
93+
),
94+
new DateFormatTester("1999-12-03",
95+
ImmutableList.of("%D"),
96+
ImmutableList.of("3rd")
97+
),
98+
new DateFormatTester("1999-12-04",
99+
ImmutableList.of("%D"),
100+
ImmutableList.of("4th")
101+
),
102+
new DateFormatTester("1999-12-11",
103+
ImmutableList.of("%D"),
104+
ImmutableList.of("11th")
105+
),
106+
new DateFormatTester("1999-12-12",
107+
ImmutableList.of("%D"),
108+
ImmutableList.of("12th")
109+
),
110+
new DateFormatTester("1999-12-13",
111+
ImmutableList.of("%D"),
112+
ImmutableList.of("13th")
113+
),
114+
new DateFormatTester("1999-12-31",
115+
ImmutableList.of("%x","%v","%X","%V","%u","%U"),
116+
ImmutableList.of("1999", "52", "1999", "52", "52", "52")
117+
),
118+
new DateFormatTester("2000-01-01",
119+
ImmutableList.of("%x","%v","%X","%V","%u","%U"),
120+
ImmutableList.of("1999", "52", "1999", "52", "0", "0")
121+
),
122+
new DateFormatTester("1998-12-31",
123+
ImmutableList.of("%x","%v","%X","%V","%u","%U"),
124+
ImmutableList.of("1998", "52", "1998", "52", "52", "52")
125+
),
126+
new DateFormatTester("1999-01-01",
127+
ImmutableList.of("%x","%v","%X","%V","%u","%U"),
128+
ImmutableList.of("1998", "52", "1998", "52", "0", "0")
129+
),
130+
new DateFormatTester("2020-01-04",
131+
ImmutableList.of("%x","%X"),
132+
ImmutableList.of("2020", "2019")
133+
),
134+
new DateFormatTester("2008-12-31",
135+
ImmutableList.of("%v","%V","%u","%U"),
136+
ImmutableList.of("53","52","53","52")
137+
)
138+
);
139+
140+
@AllArgsConstructor
141+
private class DateFormatTester {
142+
private final String date;
143+
private final List<String> formatterList;
144+
private final List<String> formattedList;
145+
private static final String DELIMITER = "|";
146+
147+
String getFormatter() {
148+
return String.join(DELIMITER, formatterList);
149+
}
150+
151+
String getFormatted() {
152+
return String.join(DELIMITER, formattedList);
153+
}
154+
155+
FunctionExpression getDateFormatExpression() {
156+
return dsl.date_format(DSL.literal(date), DSL.literal(getFormatter()));
157+
}
158+
}
159+
74160
@Test
75161
public void adddate() {
76162
FunctionExpression expr = dsl.adddate(dsl.date(DSL.literal("2020-08-26")), DSL.literal(7));
@@ -872,6 +958,48 @@ public void year() {
872958
assertEquals(integerValue(2020), eval(expression));
873959
}
874960

961+
@Test
962+
public void date_format() {
963+
dateFormatTesters.forEach(this::testDateFormat);
964+
String timestamp = "1998-01-31 13:14:15.012345";
965+
String timestampFormat = "%a %b %c %D %d %e %f %H %h %I %i %j %k %l %M "
966+
+ "%m %p %r %S %s %T %% %P";
967+
String timestampFormatted = "Sat Jan 01 31st 31 31 12345 13 01 01 14 031 13 1 "
968+
+ "January 01 PM 01:14:15 PM 15 15 13:14:15 % P";
969+
970+
FunctionExpression expr = dsl.date_format(DSL.literal(timestamp), DSL.literal(timestampFormat));
971+
assertEquals(STRING, expr.type());
972+
assertEquals(timestampFormatted, eval(expr).stringValue());
973+
974+
when(nullRef.type()).thenReturn(DATE);
975+
when(missingRef.type()).thenReturn(DATE);
976+
assertEquals(nullValue(), eval(dsl.date_format(nullRef, DSL.literal(""))));
977+
assertEquals(missingValue(), eval(dsl.date_format(missingRef, DSL.literal(""))));
978+
979+
when(nullRef.type()).thenReturn(DATETIME);
980+
when(missingRef.type()).thenReturn(DATETIME);
981+
assertEquals(nullValue(), eval(dsl.date_format(nullRef, DSL.literal(""))));
982+
assertEquals(missingValue(), eval(dsl.date_format(missingRef, DSL.literal(""))));
983+
984+
when(nullRef.type()).thenReturn(TIMESTAMP);
985+
when(missingRef.type()).thenReturn(TIMESTAMP);
986+
assertEquals(nullValue(), eval(dsl.date_format(nullRef, DSL.literal(""))));
987+
assertEquals(missingValue(), eval(dsl.date_format(missingRef, DSL.literal(""))));
988+
989+
when(nullRef.type()).thenReturn(STRING);
990+
when(missingRef.type()).thenReturn(STRING);
991+
assertEquals(nullValue(), eval(dsl.date_format(nullRef, DSL.literal(""))));
992+
assertEquals(missingValue(), eval(dsl.date_format(missingRef, DSL.literal(""))));
993+
assertEquals(nullValue(), eval(dsl.date_format(DSL.literal(""), nullRef)));
994+
assertEquals(missingValue(), eval(dsl.date_format(DSL.literal(""), missingRef)));
995+
}
996+
997+
void testDateFormat(DateFormatTester dft) {
998+
FunctionExpression expr = dft.getDateFormatExpression();
999+
assertEquals(STRING, expr.type());
1000+
assertEquals(dft.getFormatted(), eval(expr).stringValue());
1001+
}
1002+
8751003
private ExprValue eval(Expression expression) {
8761004
return expression.valueOf(env);
8771005
}

0 commit comments

Comments
 (0)