test(bigquery): run CI on the Arrow query results stack (do not merge)

[ohmayr]

Not for review or merge. This draft exists only to run the full CI (unit, cover, lint, mypy, Kokoro system tests) on the Arrow query results PRs, since those checks only run for PRs that target main, and the stacked PRs target each other:

• feat(bigquery-storage): add Arrow options and cancel to read_rows #18643
• refactor(bigquery): move Arrow query result reading to _arrow_query #18644
• feat(bigquery): validate Arrow queries before sending them #18645
• fix(bigquery): send the same Arrow options on jobs.query and ReadRows #18646
• feat(bigquery): add timestamp_precision for Arrow query results #18649
• fix(bigquery): keep the Arrow first page and wait without rows #18650
• fix(bigquery): apply max_results to the whole Arrow result #18651
• feat(bigquery): share one BigQuery Storage client per Client #18652
• feat(bigquery): read Arrow results from explicit destinations #18654

The branch points at the top of that stack and is updated as more PRs are added. Review the individual PRs instead.

[gemini-code-assist]

Code Review

This pull request introduces support for reading BigQuery query results in Apache Arrow format with serialization options (such as buffer compression) across both google-cloud-bigquery and google-cloud-bigquery-storage. It adds a new _arrow_query module to handle validation, serialization options, and downloading of Arrow record batches, integrates these options into query_and_wait and RowIterator, and implements a cancel() method on ReadRowsStream to allow early termination of streams. Feedback on these changes suggests raising a ValueError if an Arrow record batch is received without a schema to prevent silent data loss, refactoring the optional parameter handling in _read_rows_options for better encapsulation, and adding upfront validation for project, location, and job_id to avoid malformed stream names.

[gemini-code-assist]

If response.arrow_record_batch is present but pa_schema is None, the record batch will be silently skipped. Since there is no fallback mechanism when streaming from the BigQuery Read API, this can lead to silent data loss where some rows are omitted from the final results without raising any error. Raising a ValueError explicitly when a batch is received without a schema prevents silent data corruption.

Suggested change

	if (
	response.arrow_record_batch
	and response.arrow_record_batch.serialized_record_batch
	and pa_schema is not None
	):
	batch = pyarrow.ipc.read_record_batch(
	pyarrow.py_buffer(response.arrow_record_batch.serialized_record_batch),
	pa_schema,
	)
	yield batch
	if (
	response.arrow_record_batch
	and response.arrow_record_batch.serialized_record_batch
	):
	if pa_schema is None:
	raise ValueError("An Arrow record batch was received, but the schema is missing.")
	batch = pyarrow.ipc.read_record_batch(
	pyarrow.py_buffer(response.arrow_record_batch.serialized_record_batch),
	pa_schema,
	)
	yield batch

[gemini-code-assist]

When processing optional parameters with a helper function like serialization_options, prefer placing the null or None check within the function body rather than at the call site to simplify the calling code and improve encapsulation. Therefore, we should pass options directly to serialization_options instead of performing a manual inline OR check.

Suggested change

	def _read_rows_options(options: Optional[Dict[str, str]]) -> Dict[str, str]:
	"""Convert ``jobs.query`` ``arrowSerializationOptions`` to the
	``ArrowSerializationOptions`` fields that ``ReadRows`` takes."""
	options = options or serialization_options(None)
	return {
	"buffer_compression": options["bufferCompression"],
	"picos_timestamp_precision": options["picosTimestampPrecision"],
	}
	def _read_rows_options(options: Optional[Dict[str, str]]) -> Dict[str, str]:
	"""Convert ``jobs.query`` ``arrowSerializationOptions`` to the
	``ArrowSerializationOptions`` fields that ``ReadRows`` takes."""
	options = serialization_options(options)
	return {
	"buffer_compression": options.get("bufferCompression", "COMPRESSION_UNSPECIFIED"),
	"picos_timestamp_precision": options.get("picosTimestampPrecision", "TIMESTAMP_PRECISION_MICROS"),
	}

References

1. When processing optional parameters with a helper function, prefer placing the null or None check within the function body rather than at the call site to simplify the calling code and improve encapsulation.

[gemini-code-assist]

If project, location, or row_iterator._job_id are None or empty, the constructed stream_name will contain "None", leading to a confusing gRPC or path formatting error from the API. Adding explicit validation checks upfront will provide a much clearer error message to the user.

    project = row_iterator._project or (
        row_iterator.client.project if row_iterator.client else None
    )
    location = row_iterator._location or (
        row_iterator.client.location if row_iterator.client else None
    )
    if not project:
        raise ValueError("Project is required to stream Arrow results.")
    if not location:
        raise ValueError("Location is required to stream Arrow results.")
    if not row_iterator._job_id:
        raise ValueError("Job ID is required to stream Arrow results.")
    stream_name = f"projects/{project}/locations/{location}/jobs/{row_iterator._job_id}/streams/_default"