FYI: Nutanix API Authentication fails when making many requests at once

[maxaudron]

When making multiple requests to the API with basic auth on each request, about half get denied.
Authenticating once and caching with session auth works normaly.
Same result when using prismcentral or Cluster's prism directly.
Authentication backend is LDAP.

One of these two errors are returned:

{"state": "ERROR", "code": 401, "message_list": [{"reason": "AUTHENTICATION_REQUIRED", "message": "Authentication required.", "details": "Basic realm=\"Intent Gateway Login Required\""}], "api_version": "3.1"}

{
  "api_version": "3.1",
  "code": 403,
  "message_list": [
    {
      "message": "No permission to access the resource. <Request 'http://prismcentral.example.com:9440/v3/images/9e6b62a9-f397-4839-8eed-c7bae60447de' [GET]>",
      "reason": "ACCESS_DENIED"
    }
  ],
  "state": "ERROR"
}

the API logs show following error:

2019-11-29 00:39:02 UWSGI 212.82.224.202 [GET]:/v3/images/9e6b62a9-f397-4839-8eed-c7bae60447de issued by: <> took 101621 msecs, response status: 200
2019-11-29 00:39:04 UWSGI - [POST]:/v3/fanout_proxy?remote_cluster_uuid=000573a1-39c7-1202-0000-000000019b5b&url_path=remote_rpc_request%3Fservice_name%3Dnutanix.catalog.CatalogRpcService%26port%3D2007%26base_url%3D%252Frpc&method=POST&content_type=application%2Foctet-stream&tenant_uuid=00000000-0000-0000-0000-000000000000 issued by: admin took 965 msecs, response status: 200
2019-11-29 00:39:05 INFO category.py:115 Looking up category by uuid: c63a4536-c2fa-4d83-9fa2-29ca2b4fb508 
2019-11-29 00:39:05 INFO category.py:115 Looking up category by uuid: c63a4536-c2fa-4d83-9fa2-29ca2b4fb508 
2019-11-29 00:39:05 ERROR athena_auth.py:128 User is not allowed to access the system without access control policy. 
2019-11-29 00:39:05 ERROR athena_auth.py:144 Error: Traceback (most recent call last):
  File "/home/jenkins.svc/workspace/postcommit-jobs/nos/5.11/x86_64-pc-release-euphrates-5.11-stable/builds/build-euphrates-5.11-stable-release/python-tree/bdist.linux-x86_64/egg/aplos/lib/auth/athena_auth.py", line 131, in authenticate
BasicAuthRequiredError: {'api_version': '3.1',
 'code': 401,
 'message_list': [{'details': 'Basic realm="Intent Gateway Login Required"',
                   'message': 'Authentication required.',
                   'reason': 'AUTHENTICATION_REQUIRED'}],
 'state': 'ERROR'}2019-11-29 00:39:05 ERROR auth.py:93 Traceback (most recent call last):
  File "/home/jenkins.svc/workspace/postcommit-jobs/nos/5.11/x86_64-pc-release-euphrates-5.11-stable/builds/build-euphrates-5.11-stable-release/python-tree/bdist.linux-x86_64/egg/aplos/lib/auth/auth.py", line 83, in session_authenticate
  File "/home/jenkins.svc/workspace/postcommit-jobs/nos/5.11/x86_64-pc-release-euphrates-5.11-stable/builds/build-euphrates-5.11-stable-release/python-tree/bdist.linux-x86_64/egg/aplos/lib/auth/athena_auth.py", line 131, in authenticate
BasicAuthRequiredError: {'api_version': '3.1',
 'code': 401,
 'message_list': [{'details': 'Basic realm="Intent Gateway Login Required"',
                   'message': 'Authentication required.',
                   'reason': 'AUTHENTICATION_REQUIRED'}],
 'state': 'ERROR'}- UWSGI - [GET]:/v3/images/9e6b62a9-f397-4839-8eed-c7bae60447de - took 101981 msecs, response status: 401

Steps to reproduce

spam some API requests all at once:

$ for i in $(seq 1 50); do curl -u $TF_VAR_username:$TF_VAR_password https://prismcentral.example.com:9440/api/nutanix/v3/images/9e6b62a9-f397-4839-8eed-c7bae60447de; done

PR #88 mitigates this somewhat.

[PacoDw]

Hello @maxaudron
Does this error happen when you are using Terraform or using just the API directly (using the client)?

[maxaudron]

This happens with both terraform and the curl api command i posted. As said the PR #88 works around this.

The actual issue is with the nutanix API and i have a case open tracking this.

[PacoDw]

Thank you so much, I will comment this to the Nutanix team as well

[PacoDw]

we will close this issue due that it's not part of terraform