-
Notifications
You must be signed in to change notification settings - Fork 3.9k
sql: fix serialization of datums and prefix casts #12836
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
What was the rationale for having the format flag negated? (why did you chose Review status: 0 of 11 files reviewed at latest revision, 1 unresolved discussion. pkg/sql/parser/datum.go, line 396 at r1 (raw file):
Don't do this special casing here. Do it just once for all ambiguous datum types in Comments from Reviewable |
I figured that without them the expression may not be valid and we shouldn't print out invalid expressions. I don't have a strong opinion here though.. Review status: 0 of 11 files reviewed at latest revision, 1 unresolved discussion. Comments from Reviewable |
I think we ought to distinguish "pretty printing an expr for a human" and "serializing an expr in a format that is deserializable and also as much as possible readable by a human". Perhaps we should have different function names for these two things, and use them in the appropriate contexts. |
pkg/sql/parser/datum.go, line 396 at r1 (raw file): Previously, knz (kena) wrote…
But there is no generic way to do that since it applies only to some nodes (Datums with some types).. So we would have a big Or are you suggesting to include an explicit cast for all constants? (although that still requires a switch to only apply it to Datum nodes no?) Comments from Reviewable |
409cbf9
to
9a58315
Compare
Ok, take two. Please ignore the first two commits (I sent out a separate PR for those). I reversed the meaning of the flag; it's now only enabled as part of a new I also found that we can't elide the Review status: 0 of 14 files reviewed at latest revision, 1 unresolved discussion, some commit checks pending. Comments from Reviewable |
The changes here look good in general, but we should walk through all of the places where Also, we are starting to overlap with #12611, which has related complications. It might not be worth worrying about yet, but does the second part of @knz's proposed plan there influence your design here at all? Perhaps instead of enforcing datum types through a formatting mode, we have a new tree transformation that can either annotate all expressions with Reviewed 10 of 10 files at r4. pkg/sql/create.go, line 387 at r4 (raw file):
I think we need to do the same thing with persisted expressions on pkg/sql/parser/datum.go, line 396 at r1 (raw file): Previously, RaduBerinde wrote…
I personally am fine with this approach FWIW. pkg/sql/parser/type_check.go, line 211 at r4 (raw file):
long line Comments from Reviewable |
There is definitely overlap with #12611 - this change solves a big part of the problem, that of interpreting literals/constants with the same types. The other concerns - around a function changing its return type - are something that we hold more control over and might not be problems in practice; in any case, we could add on top of the infrastructure proposed here to output function return types as well. Review status: 10 of 14 files reviewed at latest revision, 3 unresolved discussions, some commit checks failed. Comments from Reviewable |
641e4be
to
3968361
Compare
Review status: 7 of 17 files reviewed at latest revision, 3 unresolved discussions. pkg/sql/create.go, line 387 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Thanks, great point! Done for default and check. I don't think FK references have any expressions. pkg/sql/parser/type_check.go, line 211 at r4 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. Comments from Reviewable |
Mostly LGTM. Will want to have a look again when all the tests pass. Reviewed 4 of 12 files at r2, 3 of 6 files at r3, 1 of 10 files at r4, 6 of 6 files at r5. pkg/sql/parser/datum.go, line 396 at r1 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Ok I've decided I'm fine either way too. Comments from Reviewable |
3968361
to
7ff83ec
Compare
Thanks! Updated. The fixes around |
7ff83ec
to
ddb01f3
Compare
Reviewed 1 of 6 files at r5, 7 of 7 files at r6. Comments from Reviewable |
I don't think it's necessary for the cases we've seen but I figured it's an incorrect way to format a cast in general and given that we have a lot of code that manipulates expressions it's hard to say for sure that the enclosed expression will always stay a constant. By the way, I was looking at writing some tests and I am realizing this doesn't quite work as intended.. It turns out that the Review status: all files reviewed at latest revision, 1 unresolved discussion, all commit checks successful. Comments from Reviewable |
Ah good catch. We should almost certainly prefer type annotations to casts
in this situation then. The semantics might work out to be the same, but
each carries a different connotation.
…On Thu, Jan 12, 2017, 7:40 PM RaduBerinde ***@***.***> wrote:
I don't think it's necessary for the cases we've seen but I figured it's
an incorrect way to format a cast in general and given that we have a lot
of code that manipulates expressions it's hard to say for sure that the
enclosed expression will always stay a constant.
By the way, castPrefixParens seems incorrect to me - formats INTERVAL (
.. ) '1h' as INTERVAL ('1h').
I was looking at writing some tests and I am realizing this doesn't quite
work as intended.. It turns out that the TYPENAME xx notation only works
if xx is a string. Things like DECIMAL 1.2 don't parse.. So I think I
should replace the annotations I added with actual casts (1.2::DECIMAL),
what do you think?
------------------------------
Review status: all files reviewed at latest revision, 1 unresolved
discussion, all commit checks successful.
------------------------------
*Comments from Reviewable
<https://reviewable.io:443/reviews/cockroachdb/cockroach/12836#-:-KaK9FKAh3sr25I4_moC:bemyqkp>*
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#12836 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFL7-O74Q1Sht2_SXMLRmv430ZlGxgQtks5rRsgRgaJpZM4Lf4_I>
.
|
So Review status: all files reviewed at latest revision, 1 unresolved discussion, all commit checks successful. Comments from Reviewable |
Right. And actually the semantics wouldn't even be the same. Casts would only desire the type, while annotations would require it. We want the latter here. If we're going to do this approach, we can probably unify the implementation ala @knz's suggestion about |
What do you mean casts only desire the type? If the cast isn't possible, that is an "invalid cast" error no? One thing I'm wondering about is whether we will end up with two annotations which doesn't work (unlike casts) - would it be possible to ever (after some expression manipulation perhaps) have a I was also thinking of switching these Review status: all files reviewed at latest revision, 1 unresolved discussion, all commit checks successful. Comments from Reviewable |
What I mean is that a I can't think of any situation where it would be possible to get a I had the same idea a little while ago about switching the |
Yeah I thought about it a bit and I don't think the case I mentioned could happen either. Sorry, you guys mentioned Review status: all files reviewed at latest revision, 1 unresolved discussion, all commit checks successful. Comments from Reviewable |
Wait, it seems we will actually have the exact same problem. Should we make Comments from Reviewable |
Argh.. I was just about to post a new version but I found the following case Review status: all files reviewed at latest revision, 1 unresolved discussion, all commit checks successful. Comments from Reviewable |
ddb01f3
to
98f257f
Compare
Ok, please check the latest version (and my ramblings above for context):
Review status: 8 of 17 files reviewed at latest revision, 1 unresolved discussion. Comments from Reviewable |
98f257f
to
5f87a27
Compare
Reviewed 8 of 9 files at r7. pkg/sql/parser/format.go, line 138 at r7 (raw file):
We do for I would lean towards making this a method ( pkg/sql/parser/type_check_test.go, line 38 at r7 (raw file):
Why is this being normalized? pkg/sql/parser/type_check_test.go, line 46 at r7 (raw file):
The reason for the different case is because your
to
Comments from Reviewable |
pkg/sql/parser/type_check_test.go, line 38 at r7 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
I could ask you the same question sir! Looks like Comments from Reviewable |
pkg/sql/parser/format.go, line 138 at r7 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
I figured that if the expression type checks with I am hesitating between adding a new method and pushing this back down into Comments from Reviewable |
Review status: 16 of 17 files reviewed at latest revision, 3 unresolved discussions, all commit checks successful. pkg/sql/parser/format.go, line 138 at r7 (raw file): Previously, RaduBerinde wrote…
I'm pretty sure you're correct in the case where nothing changes between formatting and parsing, although I still have #12611 on the back of my mind, in which case ambiguity could be introduced. And that's a good point, though I do think a new method would help separate concerns. I'm fine with either approach, so go with whatever you like best. pkg/sql/parser/type_check_test.go, line 38 at r7 (raw file): Previously, RaduBerinde wrote…
Huh.. would you look at that. I guess overload resolution won't really work with NULL operands. I forsee an overhaul of NULL handling in the near future, but in the meantime, carry on! Comments from Reviewable |
5f87a27
to
cb1a3fb
Compare
Updated, TFTR and for the great suggestions! Review status: 14 of 17 files reviewed at latest revision, 3 unresolved discussions, some commit checks pending. pkg/sql/parser/format.go, line 138 at r7 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/parser/type_check_test.go, line 46 at r7 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. Comments from Reviewable |
Hm, I now see this for
Similar issue - Review status: 14 of 17 files reviewed at latest revision, 3 unresolved discussions, some commit checks failed. Comments from Reviewable |
|got: CREATE VIEW T5 AS SELECT s, count(*) FROM d.t GROUP BY s HAVING
count(*) > 3:::INT expected: CREATE VIEW T5 AS SELECT s, count(*) FROM
d.t GROUP BY s HAVING count(*) > 3 |
Similar issue - |BinaryExpr.TypeCheck| modifies the expression in place
and the operand becomes a |DInt|. Do you think this is ok from a user
perspective?
OK with me
…--
Raphael 'kena' Poss
|
I could serialize the expression before type-checking to go around this issue. But it seems that for #12611 we would actually need to go in the opposite direction and actually serialize the type-checked expression, right? Should I do that in this change? |
cb1a3fb
to
9bd50dd
Compare
Never mind this, doing either is not easy at all, I will not pursue that in this change. |
The failing test actually seems like a good thing to me. It means that the ambiguity of the untyped numeric literal will only be considered once, and from then on, will always be guaranteed to be treated as the same type. Review status: 14 of 18 files reviewed at latest revision, 3 unresolved discussions, all commit checks successful. Comments from Reviewable |
Is the latest version good to go? |
Review status: 14 of 18 files reviewed at latest revision, 3 unresolved discussions, all commit checks successful. pkg/sql/parser/datum.go, line 166 at r8 (raw file):
nit: I'd move all of these directly above or below their type's respective pkg/sql/parser/type_check.go, line 211 at r8 (raw file):
We can't do this for column types that can have parameters but don't in this case? Like Comments from Reviewable |
9bd50dd
to
320bd4f
Compare
This change addresses a few related problems (described in more detail in cockroachdb#12532): - ambiguous datum types after serialization - invalid prefix casts after expression processing (like `INTERVAL INTERVAL `1.5s`'). Changes: - a new FmtFlag adds type annotations to datums and placeholders. This ensures that expressions that contain `DDInterval`, `DDate` can be formatted and re-parsed correctly. This replaces ad-hoc type prepending like `INTERVAL '..'` or `DATE '..'`. - `CastExpr.TypeCheck` now returns the underlying type-checked expression directly when it is a Datum that we know for sure is of identical type. - `castPrefix` (renamed to `castPrepend`) now only functions if the underlying expression is a string literal (preventing incorrect expressions like `DECIMAL '1.0':::STRING`). `castPrefixParens` is removed (it was incorrect). - `AnnotateTypeExpr.TypeCheck` now returns the underlying type-checked expression rather than changing the `AnnotateTypeExpr` in-place and returning it (which could result in double annotations with the new `FmtFlag`). `AnnotateTypeExpr` is no longer a `TypedExpr` so it is never part of a type-checked expression (which makes sense). - type checking tests are extended to verify the serialized type-checked expression, which verifies both serialization and type resolution. Note that changing the prepend syntax to a `AnnotateTypeExpr` was considered but that doesn't work for things like `DECIMAL '2.0'`. Fixes cockroachdb#12532. Helps with cockroachdb#12611.
Review status: 14 of 18 files reviewed at latest revision, 3 unresolved discussions. pkg/sql/parser/datum.go, line 166 at r8 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Done. pkg/sql/parser/type_check.go, line 211 at r8 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
We could, do you think it's worth it? I didn't do it because it would be a bunch of cases that depend on each type... pehaps we should add a method to Comments from Reviewable |
Review status: 14 of 18 files reviewed at latest revision, 1 unresolved discussion, all commit checks successful. pkg/sql/parser/type_check.go, line 211 at r8 (raw file): Previously, RaduBerinde wrote…
I don't think so. Let's push this as is. I actually need to open an issue for CastExpr to handle column type parameters anyway. Comments from Reviewable |
Review status: 14 of 18 files reviewed at latest revision, 1 unresolved discussion, all commit checks successful. pkg/sql/parser/type_check.go, line 211 at r8 (raw file): Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
Follow up issue #13024. Comments from Reviewable |
Removing the "prefix" syntax for
CastExpr
s; we now prepend the type as part ofDatum formatting (conditional on a new
FmtFlag
). This ensures that expressionsthat contain
DDInterval
,DDate
etc (without aCastExpr
) can be formattedand re-parsed correctly.
To avoid a redundant cast, we elide the
CastExpr
during type checking if theargument is a constant that can become the required type.
See #12532 for more context.
Fixed #12532.
Note: the current change breaks a lot of tests that encode expressions (like TestSimplifyExpr, parse tests), I want to get feedback on the change before I go through correcting them.
This change is