[C++][Go][Java][FlightRPC] Add support for result set expiration

[kou]

Describe the enhancement requested

Based on the proposal in https://docs.google.com/document/d/1jhPyPZSOo2iy0LqIJVUs9KWPyFULVFJXTILDfkadx2g/edit# .
See also the discussion thread: https://lists.apache.org/thread/247z3t06mf132nocngc1jkp3oqglz7jp

Currently, it is undefined whether a client can call DoGet more than once. Clients may want to retry requests, and servers may not want to persist a query result forever.

Proposal

Add an expiration time to FlightEndpoint. If present, clients may assume they can retry DoGet requests. Otherwise, clients should avoid retrying DoGet requests.

message FlightEndpoint {
  // In UTC.
  google.protobuf.Timestamp expiration_time = 3;
}

This proposal is not a full retry protocol.

Also, add “pre-defined” actions to Flight RPC for working with result sets. These are pre-defined Protobuf messages with standardized encodings for use with DoAction:

• CancelQuery CancelFlightInfo: Asynchronously cancel the execution of a distributed query. (Replaces the equivalent Flight SQL action.)
• RefreshQuery RefreshFlightEndpoint: Request an extension of the expiration of a FlightEndpoint.
• CloseQuery CloseFlightInfo: Close a FlightInfo so that the server can clean up resources early. (Only CancelFlightInfo will be enough.)

This lets the ADBC/JDBC/ODBC drivers for Flight SQL explicitly manage result set lifetimes. These can be used with Flight SQL as regular actions.

Prior Art

• Dremio: job results are cleaned up after a set time.
• Google BigQuery Storage: read sessions include an explicit expiration time.
• Snowflake: result sets persist for 24 hours.

Component(s)

C++, FlightRPC

[kou]

https://lists.apache.org/thread/71pp95q6yklodm6lfjttswr3slfowdrb

About CloseQuery:

maybe it would be nice to name it 'CloseFlightInfo', to be matched with GetFlightInfo.

I agree with this. I'll use CancelFlightInfo/RefreshFlightInfo/CloseFlightInfo.

[kou]

Ah, should RefreshQuery extend a expiration time of a FlightEndpoint? Or all expiration times of FlightInfo?
If the former, we should use RefereshFlightEndpoint, RefreshFlightInfo otherwise.

[kou]

RefreshFlightEndpoint will be better because it's difficult that we return a suitable response for refreshing multiple FlightEndpoints. (If one of multiple FlightEndpoints is only failed, what response is suitable?)

[lidavidm]

RefreshFlightEndpoint should be more flexible, agreed.

[kou]

Discussion:
[DISCUSS][Format][Flight] Result set expiration support
https://lists.apache.org/thread/48fqd554gkqrrld8k13l3b8trz5gk7ow

[kou]

Changes based on discussion on the mailing list:

https://lists.apache.org/thread/ofw6bydhyhcqqn3t8013gx0k6fbw8lvv

• Remove CloseFlightInfo (if nobody objects it)
• RefreshFlightEndpoint ->
  RenewFlightEndpoint
• RenewFlightEndpoint(FlightEndpoint) ->
  RenewFlightEndpoint(RenewFlightEndpointRequest)
• CancelFlightInfo(FlightInfo) ->
  CancelFlightInfo(CancelFlightInfoRequest)