Querying for `researchers` can result in HTTP 408 exceptions

[edsu]

I was wondering if anyone else has been experiencing intermittent HTTP 408 errors when doing queries?

File "/opt/airflow/rialto_airflow/dags/harvest.py", line 70, in dimensions_harvest
   return dimensions.harvest(snapshot, limit=harvest_limit)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/airflow/rialto_airflow/harvest/dimensions.py", line 45, in harvest
   for dimensions_pub_json in orcid_publications(orcid_query_str):
 File "/opt/airflow/rialto_airflow/harvest/dimensions.py", line 79, in orcid_publications
   yield from publications_from_dois(dois)
 File "/opt/airflow/rialto_airflow/harvest/dimensions.py", line 114, in publications_from_dois
   result = query_with_retry(q, retry=5)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/opt/airflow/rialto_airflow/harvest/dimensions.py", line 171, in query_with_retry
   raise e
 File "/opt/airflow/rialto_airflow/harvest/dimensions.py", line 165, in query_with_retry
   return dsl().query(q, retry=retry)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/dimcli/core/api.py", line 187, in query
   return self.query(
          ^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/dimcli/core/api.py", line 187, in query
   return self.query(
          ^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/dimcli/core/api.py", line 187, in query
   return self.query(
          ^^^^^^^^^^^
 [Previous line repeated 2 more times]
 File "/home/airflow/.local/lib/python3.12/site-packages/dimcli/core/api.py", line 196, in query
   response.raise_for_status()
 File "/home/airflow/.local/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
   raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 408 Client Error: Request Timeout for url: https://app.dimensions.ai/api/dsl

[peetucket]

Note: we updated our code to use an API key instead of username/password and it seemed to work better.

old code:

 dimcli.login(
        "username",
        "password",
        "https://app.dimensions.ai"
    )

new code:

dimcli.login(
     key="api-key",  
     endpoint="https://app.dimensions.ai/api/dsl/v2"
   )

[peetucket]

Never mind - we are still seeing 408s, when querying publications for this ORCID: 0000-0001-6711-3286 (though it maybe just a coincidence and is instead arising after a certain number of queries)

[edsu]

If it's helpful this code seems to reliably cause the 408 error with dimcli v1.4:

import os
import dimcli

dimcli.login(
    key=os.environ.get("DIMENSIONS_API_KEY"),
    endpoint="https://app.dimensions.ai/api/dsl/v2",
)

dsl = dimcli.Dsl()

q = '''
search publications where doi in ["10.3847/1538-4357/ad9749","10.1103/physrevd.111.042005","10.3847/1538-4357/ad8de0","10.1364/fio.2024.jtu4a.2","10.3847/1538-4357/ad65ce","10.1364/cleo_si.2024.sm1d.3","10.1103/physrevd.110.042001","10.3847/1538-4357/ad3e83","10.3847/2041-8213/ad5beb"]
return publications [researchers]
limit 1000
'''

result = dsl.query(q)

Either replacing the researchers field for another field like title or removing one of the DOIs from the list causes the query to succeed. I'm a bit confused because there doesn't appear to be any researchers data coming back for any of these DOIs.

This appears to be a problem with the Dimensions API service rather than the dimcli client, so please advise if it is better to report it elsewhere.

[edsu]

It looks like this is returning a more helpful error now:

Connected to: <https://app.dimensions.ai/api/dsl/v2> - DSL v2.11
Method: manual login
Returned Errors: 1
Time: 0.96s
Service unavailable
1 EvaluationError found

Your query is too long e.g. because it includes too many filters. Please review it by keeping in mind the guidelines on https://docs.dimensions.ai/dsl/faq.html#queries-and-errors [code: 3]