Programmatic Access to CRS PDF/HTML Documents Referenced by API

[AndrVelich]

Dear Congress.gov API Team,

We are currently using the Congress.gov API to ingest structured metadata for CRS (Congressional Research Service) reports. The API endpoints for structured data function as expected and meet our requirements.

However, we are encountering issues when attempting to programmatically retrieve the associated PDF and HTML documents referenced in the API responses. The URLs provided point to the congress.gov domain, which appears to be protected by Cloudflare. Automated requests to download these files are blocked or challenged, preventing reliable programmatic access.

We would appreciate clarification on the following:

Is there an officially supported mechanism for programmatic retrieval of CRS PDF and HTML documents?

Are alternative endpoints, bulk download options, or authenticated access methods available for accessing these materials?

If access to congress.gov-hosted documents is intentionally restricted, is there a recommended approach for compliant automated retrieval?

Our use case involves systematic ingestion of publicly available CRS materials for research and indexing purposes. We are prepared to adhere to any applicable rate limits, authentication requirements, or usage policies.

We would appreciate your guidance on how best to proceed.

[rbram]

Hello. Thanks, we will investigate this. For now, please lower your requests, and we will follow up with other solutions if available.

[AndrVelich]

Hi @rbram , thank you for your feedback. Just to let you know, I’m currently unable to complete even a single request.

[rbram]

Thanks. Could you send me an email (rbram@loc.gov) with the following info: what programming language you are using, a code snippet (if possible), and your IP address?
Also, could you let us know if you are getting an error message or error response and any details?

[AndrVelich]

We’re currently unable to access the required files due to Cloudflare protection. Instead of the expected content, we’re receiving a Cloudflare challenge page in the response. Since the challenge cannot be solved programmatically, this is preventing us from proceeding with the integration.
I've just sent all the info to your email. Please let me know if I can help with any other information.
Thank you!

[seand-glocod]

Hello, I am also encoutering this problem when trying to use the API.
I am too from a company that focuses on regulatory sources, and does want to fetch the pdf that relate to each established law.
When I fetch from api.congress.gov, the returned metadata directs me to the "normal" website of congress.gov to download the PDF document, where I get hit with a 429 on every request I send, and I am unable to download even a single PDF document.
If there is anything else I can provide to make it easier to help please let me know.
Thanks in advance.