Fix use of HTTP credentials #220
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The current implementation doesn't properly pass username/password credentials when connecting to Elasticsearch using HTTP (as opposed to connecting with AWS credentials). The version 7.x Python elasticsearch library doesn't support the http_auth parameter currently specified, see code here: https://github.com/elastic/elasticsearch-py/blob/e275834941ea053c300e2c4bc494fccd5265f127/elasticsearch/client/__init__.py#L63 Instead, the username/password must be passed as part of the HTTP hostname. Additionally, the use_ssl parameter needs to be set if connecting via HTTPS.
higs4281
approved these changes
Aug 26, 2025
Member
Author
|
@higs4281 this is a step toward the new environment; in that environment, we connect to AWS OpenSearch using username/password credentials. I'm happy to also include an upgrade to elasticsearch==7.13.4, I just wanted to keep the changes minimal. Would you prefer me to include that here? |
Member
You can leave it out and I'll push a separate PR for something we should have done long ago. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current implementation doesn't properly pass username/password credentials when connecting to Elasticsearch using HTTP (as opposed to connecting with AWS credentials).
The version 7.x Python elasticsearch library doesn't support the http_auth parameter currently specified, see code here.
Instead, the username/password must be passed as part of the HTTP hostname. Additionally, the
use_sslparameter needs to be set if connecting via HTTPS.To test, try connecting to an ES server using ES_HOST/ES_PORT both with and without ES_USER/ES_PASSWORD. I've confirmed this on our infrastructure.
See internal DEVPLAT-1569 for additional context, and cfpb/ccdb-data-pipeline#75 for a similar fix in ccdb-data-pipeline.