[Bug]: AppendRows can return nil result for a non-error case

[GlenDC]

Contact Details

glen.decauwsemaecker@otainsight.com

What happened?

https://github.com/OTA-Insight/bqwriter/blob/8cb7bcd2f88826def3daafa154b590dc23c3cd4a/internal/bigquery/storage/client.go#L144 can retrun (nil, nil) which smells like an anti-pattern to me? For now we capture this case by returning a new error in an unreleased patch ourselves:

8cb7bcd

Weird thing is that this only appears in a new service written by a new joiner while existing services using bqwriter have never seen such a crash (because the library would crash if you give a nil result to that ch, given it does not expect that).

We need to investigate if this is because of a mistake on our end or just something that needs to be documented or something that might need to be fixed or is already fixed on bigquery's side.

Version

0.7.0 (Latest)

What platform are you seeing the problem on?

MacOS, Linux

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[rajatkb-sc]

Is this bug resolved or any way we can avoid it when using the library ??

[GlenDC]

If you point to the master branch of this repo it should return you an error in case this happens.

The next release of this library will contain this fix or contains the patch to fix this upstream.

I hope this helps you :)

[rajatkb-sc]

Yes I did that. This helps thanx. But with this library I have been facing some trouble getting bigquery publication to actually work. Unsure if its the Google bigquery library which is not providing the error properly or is it this library itself not communicating it.

Any clues on debugging this. I wanted to avoid writing all the boiler plate code form the official doc for using the default streaming type for the API so ended up using this lib.

I am using the same code provided in example for Storage API.

[GlenDC]

I’m afraid that that might be because the data you’re writing to BQ is not tin the correct format for one or multiple fields. I’ve seen this happen in some teams before.

Feel free to share a snippet of Go code of the structures and functions used to serialise and write, alongside the scheme of the table you’re trying to write to. I’m happy to give you a hand on it by reviewing the requested code and schema.

Google is quite vague and silent in such error cases. I’ll look into if it’s possible in the next release of this library to be more clear about such I’m compatibility issues.

[rajatkb-sc]

I actually ditched the library for now. But I think some little tweaks for better error communication and that definitely would improve the library .. Let me provide a snipper of code which I think might be missing in the library

Libarary error handling :

https://github.com/OTA-Insight/bqwriter/blob/8cb7bcd2f88826def3daafa154b590dc23c3cd4a/internal/bigquery/storage/client.go#L144

But I have been using this one

        result, err := managedStream.AppendRows(ctx, rows)
	if err != nil {
		
	} else if result != nil {
		_, err := result.GetResult(ctx)
     
              if err != nil {
			              attempts, _ := result.TotalAttempts(ctx)
			              if apiErr, ok := apierror.FromError(err); ok {
				              storageErr := &storagepb.StorageError{}
				              if e := apiErr.Details().ExtractProtoMessage(storageErr); e != nil {
					              // handle error
				              }
			              }
		              }
}

This generally gives a good insight into the problem. fond the code form official golang doc for the bigquery API : link

[GlenDC]

Opening this issue as open to others in the community as we internally do not have time to look at this issue with sufficient priority. Happy to guide / mentor the one wishing to pick this up.