Skip to content

Conversation

@apstndb
Copy link
Collaborator

@apstndb apstndb commented Aug 17, 2024

This PR adds decoder logic for PROTO and ENUM.

spanner> WITH t AS (SELECT CAST('genre: POP, nationality: "Japan"' AS spanner.examples.music.SingerInfo) AS singer)
      -> SELECT t.singer, t.singer.genre, CAST(t.singer AS STRING) AS singer_string, CAST(t.singer.genre AS STRING) AS genre_string
      -> FROM t;
+--------------+-------+---------------------------------+--------------+
| singer       | genre | singer_string                   | genre_string |
+--------------+-------+---------------------------------+--------------+
| GgVKYXBhbiAA | 0     | nationality: "Japan" genre: POP | POP          |
+--------------+-------+---------------------------------+--------------+

Update dependency because it depends on "spanner: Add support for Proto Columns" in spanner client v1.62.0

fixes #174

@apstndb
Copy link
Collaborator Author

apstndb commented Aug 17, 2024

I am wondering that proto test case can be undeterministic(not canonical).

@apstndb apstndb requested a review from yfuruyama August 17, 2024 13:37
Copy link
Collaborator

@yfuruyama yfuruyama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for filing this PR. I left a comment about how spanner-cli should handle the proto column. Let's discuss in the comment.

decoder_test.go Outdated
}
}

func TestDecodeColumnProto(t *testing.T) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need these test cases?

I think spanner-cli should handle the proto column just as an arbitrary byte sequences (similar to BYTE column) and I'm not sure if we should take care of the validness of the parsed content (I feel it's a job for Spanner server test case).

Copy link
Collaborator Author

@apstndb apstndb Aug 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. It is a fair comment.
This is a legacy of trying to put this test case into a table-driven test for TestDecodeColumn.

What we really want to test here is the following:

  • No error occurs
  • It is not mistakenly treated as NULL or an empty byte string
  • The input byte sequence does not change

Therefore, this test should simply test that the byte string is passed as is via spanner.GenericColumnValue.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is what we need.
8752da0

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have noticed it can be written by table driven test using spanner.GenericColumnValue.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rewrited af031a4

@apstndb apstndb requested a review from yfuruyama August 22, 2024 14:54
@apstndb
Copy link
Collaborator Author

apstndb commented Aug 22, 2024

If you don't want to contain testdata/protos, I can rewrite ENUM test cases using hand-writtenspanner.GenericColumnValue.

@yfuruyama
Copy link
Collaborator

yfuruyama commented Aug 26, 2024

Sorry I don't have enough time to review this. Next ETA: end of this week.

@yfuruyama
Copy link
Collaborator

Sorry for the delayed review.

If you don't want to contain testdata/protos, I can rewrite ENUM test cases using hand-written spanner.GenericColumnValue.

This is another approach, but how about using existing enum type which implements protoreflect.Enum interface from external module? For example, descriptorpb.Edition can be used for this purpose.

This is just for testing, so I think any enum is fine.

decoder_test.go Outdated
}

// It should only contain test cases which can't be tested in TestDecodeColumn
func TestDecodeColumnGCV(t *testing.T) {
Copy link
Collaborator

@yfuruyama yfuruyama Aug 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need to create a separate test function because spanner.NewRow accepts the spanner.GenericColumnValue as well.

ref: https://github.com/googleapis/google-cloud-go/blob/633dc86f615ee32c1ad1ec704bf6bc858a4d535c/spanner/value.go#L4424-L4428

Copy link
Collaborator Author

@apstndb apstndb Sep 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that I don't want to test spanner.NewRow and (*spanner.Row).Column, so TestDecodeColumnGCV is more cleaner unit tests.
But I don't want to rewrite all existing test cases in TestDecodeColumn to equivalent hand written spanner.GenericColumnValue, so I agree to merge new test cases into TestDecodeColumn.

decoder_test.go Outdated
Comment on lines 326 to 330
{
desc: "null proto",
value: (*protos.SingerInfo)(nil),
want: "NULL",
},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can replace this with the following test to stop dependency on the imported proto?

		{
			desc: "null proto",
			value: spanner.GenericColumnValue{
				Type: &sppb.Type{
					Code: sppb.TypeCode_PROTO,
				},
				Value: structpb.NewNullValue(),
			},
			want: "NULL",
		},

@apstndb
Copy link
Collaborator Author

apstndb commented Sep 1, 2024

This is another approach, but how about using existing enum type which implements protoreflect.Enum interface from external module? For example, descriptorpb.Edition can be used for this purpose.

All options can test we want to test.
If I use a existing enum type, I feel a enum type in well-known protobuf types like Syntax is better than other types.

@apstndb
Copy link
Collaborator Author

apstndb commented Sep 1, 2024

I leave ProtoTypeFqn=examples.spanner.music.SingerInfo because I think this field won't be empty in real data.

@apstndb apstndb requested a review from yfuruyama September 1, 2024 15:30
Copy link
Collaborator

@yfuruyama yfuruyama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you for your work on this PR! I think now the implementation is much clearer.

@yfuruyama yfuruyama merged commit abc1cf9 into cloudspannerecosystem:master Sep 3, 2024
@apstndb apstndb deleted the support-proto-enum branch September 5, 2024 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Protocol Buffers

2 participants