format setting supported for date type #11608
Conversation
evanevanevanevannnn
changed the title
evanevanevanevannnn
changed the title
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
| return true; | ||
| } | ||
|
|
||
| if (name == "data.date.format") { |
There was a problem hiding this comment.
Для parquet не дописана часть кода: https://github.com/ydb-platform/ydb/blob/main/ydb/library/yql/providers/s3/actors/yql_arrow_column_converters.cpp#L557
There was a problem hiding this comment.
дописал
There was a problem hiding this comment.
Оказалось не удобным генерировать parquet файлы из вне. Посмотри в сторону этих тестов https://github.com/ydb-platform/ydb/blob/main/ydb/tests/fq/s3/test_format_setting.py#L1052 Это уже новые тесты которые удобно писать и читать
There was a problem hiding this comment.
добавил
| settings.Add(ctx.NewList(read.Pos(), std::move(pair))); | ||
| } | ||
|
|
||
| if (topicKeyParser.GetDateFormat()) { |
There was a problem hiding this comment.
А здесь для date нету общееизвестных форматов POSIX/ISO как для Timestamp?
There was a problem hiding this comment.
POSIX/ISO для даты выглядят одинакого - %Y-%m-%d
по совместительству это совпадает с дефолтным парсингом, поэтому показалось бессмысленно добавлять несколько одинаковых форматов
| memset(&input_tm, 0, sizeof(tm)); | ||
| input_tm.tm_mday = 1; | ||
|
|
||
| auto ptr = strptime(buf.position(), format.c_str(), &input_tm); |
There was a problem hiding this comment.
А нету другой функции которая бы сразу date бы проверяла? А то потом идут различные преобразования + 1900. Там никаки overflow быть не может? Еще не понятно почему mon на 1 сдвигается
There was a problem hiding this comment.
не нашел функции, которая сразу в LocalDate/DayNum сможет распарсить согласно формату
преобразования такие потому что в struct tm нумерация месяцев с 0, а лет с 1900. а в LocalDate соответственно с единицы и нуля
int tm_mon; /* Month. [0-11] */
int tm_year; /* Year - 1900. */
| { | ||
| LocalDate local_date; | ||
| readDateTextFormatImpl(local_date, buf, format); | ||
| /// When the parameter is out of rule or out of range, Date32 uses 1925-01-01 as the default value (-DateLUT::instance().getDayNumOffsetEpoch(), -16436) and Date uses 1970-01-01. |
There was a problem hiding this comment.
А мы точно хотим такое поведение? Может тут ошибку нужно выдавать. В Datetime какое поведение сейчас?
There was a problem hiding this comment.
в datetime не помню, но в этом же файле для date и date32 (просто без настроек формата, на несколько строк выше) идентичное поведение
| ) | ||
|
|
||
| sql = f''' | ||
| INSERT INTO bindings.`{storage_sink_binding_name}` |
There was a problem hiding this comment.
Можешь попробовать прочитать то что записали и провалидировать результат?
There was a problem hiding this comment.
данные сравниваются с каноничными в /canondata/...
There was a problem hiding this comment.
Это сложно проверять. Точно ли канонические данные правильные? Это мне кажется еще одна ошибка была в соседних тестах так делать)
|
⚪ Test history | Ya make output | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
| } | ||
|
|
||
| template <bool isOptional> | ||
| std::shared_ptr<arrow::Array> ArrowStringAsYqlDate(const std::shared_ptr<arrow::DataType>& targetType, const std::shared_ptr<arrow::Array>& value, const NDB::FormatSettings& formatSettings) { |
There was a problem hiding this comment.
Давай еще остальные конвертеры поддержим по аналогии с Datetime/Timestamp?
|
⚪ Test history | Ya make output | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ Test history | Ya make output | Test bloat
⚪ Test history | Ya make output | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
2 participants
Changelog entry
fix for YQ-2950
Changelog category