Total downloads does not count all releases

[liamg]

Are you experiencing an issue with...

shields.io

🐞 Description

The number in the total downloads badge for our repo has been fluctuating lately. We believe the issue is the badge is only counting downloads for the most recent 500 releases, instead of paging over all API result pages for every release.

🔗 Link to the badge

https://img.shields.io/github/downloads/aquasecurity/tfsec/total

💡 Possible Solution

Iterate over each page of the GitHub API call results using per_page=1000 and page=N.

I think the issue is here:

shields/services/github/github-downloads.service.js

Lines 246 to 251 in 23c0406

	const allReleases = await this._requestJson({
	schema: releaseArraySchema,
	url: `/repos/${user}/${repo}/releases`,
	options: { qs: { per_page: 500 } },
	errorMessages: errorMessagesFor('repo not found'),
	})

[chris48s]

I think the max value for per_page on this endpoint is 100, not 1000 (or 500)

https://api.github.com/repos/aquasecurity/tfsec/releases?per_page=500

returns the first 100 items

[calebcartwright]

I'm not sure how common this will be in practice, but we may need something similar to the approach we had to take for the Docker badges of firing off the requests for the pages in parallel. If we have to go through the 5+ page sequence synchronously then I wouldn't be surprised if we run into timeout issues

[calebcartwright]

The other challenge is that while we could theoretically have a max ceiling of two sequential calls with the v3 API (since we can calculate the calls for each page and fire them in parallel). However, I'm not sure the same is viable with graphql, as the latter will require us to actually iterate through each page one by one. This is unfortunate as it's tough to rationalize increased investment in v3 at this point.