Response headers?

[gotdan]

Are the response headers exposed in some way? I'm experimenting with using this extension to pull in data directly from the github API and they return a "link" header for paging.

[lmangani]

Hello @gotdan please share a full example

[gotdan]

Sure @lmangani! For example, one could use the request below to get the recently merged pull requests in the duckdb repo. It's only able to retrieve the first page of API results (in this case a max of 100 pull requests), since the url to get the next page of API results is passed to the client in a "link" response header. If there was a way to get a map of the response headers in addition to the response body (maybe (res->>'headers')::MAP, similar to the way the status code is handled?), it seems like I should be able to write a recursive CTE to make a series of requests that get pages of data until the full set of pull requests is retrieved.

Note that you currently need to hard code a github API token in the request Authorization header for this query to work correctly (maybe in the future, the duckdb secrets manager could be used in some way for this?).

	WITH raw_pulls AS (
		SELECT http_get(
			'https://api.github.com/repos/duckdb/duckdb/pulls',
			headers => MAP {
				'Accept': 'application/vnd.github+json',
				'Authorization': 'Bearer {token},
				'X-GitHub-Api-Version': '2022-11-28'
			},
			params => MAP {
				'state': 'closed',
				'sort': 'updated',
				'direction': 'desc',
				'per_page': '100'
			}
		) AS res
	),
	pulls AS (
		SELECT
			(res->>'status')::INT AS status,
			(res->>'reason') AS reason,
			unnest ( (res->>'body')::JSON[] ) AS body
		FROM
			raw_pulls
	),
	pull_detail AS (
		SELECT from_json(body, 
			'{"headers": "JSON", "merged_at": "DATETIME", "title":"VARCHAR", "body": "VARCHAR"}'
		) AS props
		FROM pulls
	)
	SELECT props.* FROM pull_detail WHERE props.merged_at IS NOT NULL