Add support for HTTP Basic Auth #440

rnpridgeon · 2018-08-18T06:49:59Z

No description provided.

edenhill

Good stuff, some minor comments.
Also missing an integration test.

confluent_kafka/avro/__init__.py

confluent_kafka/avro/cached_schema_registry_client.py

tests/avro/test_cached_client.py

confluent_kafka/avro/cached_schema_registry_client.py

rnpridgeon · 2018-08-30T18:43:53Z

Now with integration tests!

confluent_kafka/avro/__init__.py

edenhill

Almost there, good stuff!

confluent_kafka/avro/__init__.py

confluent_kafka/avro/cached_schema_registry_client.py

docker/bin/certify.sh

tests/README.md

tests/avro/test_cached_client.py

tests/run.sh

tests/testconf.json

rnpridgeon · 2018-10-08T14:34:06Z

@edenhill mind giving this another pass

confluent_kafka/avro/__init__.py

confluent_kafka/avro/cached_schema_registry_client.py

edenhill · 2018-10-08T15:12:49Z

confluent_kafka/avro/cached_schema_registry_client.py

@@ -94,9 +152,15 @@ def _send_request(self, url, method='GET', body=None, headers={}):
        _headers.update(headers)

        response = self._session.request(method, url, headers=_headers, json=body)
+
+        # Returned by Jetty not SR so the payload is not json encoded


Might be better to wrap the json parsing in a try-block for all 4xx

Any comment on this? Still seems like a valid concern

Sorry missed this one. I like this idea I did something similar in the CCloud CLI as well.

tests/integration/integration_test.py

tests/integration/run.sh

tests/README.md

rnpridgeon · 2018-11-19T05:53:05Z

@edenhill, mind giving this another go. I have fast-forwarded to rebase the PR on master and addressed all of your comments.

edenhill

Almost there, buddy!
Would be nice with an avro example, with auth support (maybe commented out), in examples/

confluent_kafka/avro/__init__.py

confluent_kafka/avro/cached_schema_registry_client.py

edenhill · 2018-11-20T07:06:16Z

confluent_kafka/avro/cached_schema_registry_client.py

@@ -94,9 +152,15 @@ def _send_request(self, url, method='GET', body=None, headers={}):
        _headers.update(headers)

        response = self._session.request(method, url, headers=_headers, json=body)
+
+        # Returned by Jetty not SR so the payload is not json encoded


Any comment on this? Still seems like a valid concern

docker/.env

docker/bin/certify.sh

tests/integration/integration_test.py

tests/run.sh

tools/windows-build.bat

rnpridgeon · 2018-11-21T21:22:13Z

@edenhill, hopefully this cleans up the rest.

confluent_kafka/avro/cached_schema_registry_client.py

edenhill · 2018-11-26T13:47:30Z

confluent_kafka/avro/cached_schema_registry_client.py

+        Deletes the specified subject and its associated compatibility level if registered.
+        It is recommended to use this API only when a topic needs to be recycled or in development environments
+
+        @:param: subject: subject name


I'm not sure about this doxygen style, make sure the docs are generated correctly.

It follows the conventions within the rest of the file. Would you like me to correct all of them or just this one?

edenhill · 2018-11-26T13:48:01Z

confluent_kafka/avro/cached_schema_registry_client.py

+        It is recommended to use this API only when a topic needs to be recycled or in development environments
+
+        @:param: subject: subject name
+        @:returns: version (int) - version of the schema deleted under this subject


there is an '@rtype' for specifying the return type

confluent_kafka/avro/cached_schema_registry_client.py

edenhill · 2018-11-26T13:49:33Z

examples/serialization/__init__.py

@@ -0,0 +1 @@
+__import__('pkg_resources').declare_namespace(__name__)


what's this for?

https://setuptools.readthedocs.io/en/latest/pkg_resources.html#namespace-package-support

avoid conflicts between confluent_kafka.avro and examples.avro

examples/serialization/avro/__main__.py

examples/setup.py

rnpridgeon · 2018-11-27T19:55:57Z

@edenhill docs build fine although the CachedSchemaRegistryClient appears to be omitted by autodoc. This has actually always been the case though so I'd like to request that we move that into another PR.

#495

examples/serialization/avro/avro-cli.py

edenhill · 2018-11-28T10:53:43Z

examples/serialization/avro/avro-cli.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python


nit: the examples/seriali../avro.. path may be overdoing it (but it is fine)

Having faced this again, I really think it should be in examples/ with the other examples, I don't see a reason for the separation

examples/serialization/avro/avro-cli.py

edenhill · 2018-11-30T15:12:19Z

examples/serialization/avro/avro-cli.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python


Having faced this again, I really think it should be in examples/ with the other examples, I don't see a reason for the separation

examples/serialization/avro/avro-cli.py

edenhill

Smaller nits, almost there!

examples/avro-cli.py

edenhill · 2018-12-01T10:19:54Z

examples/avro-cli.py

+""")
+
+
+# User stores the deserialized user Avro record.


These should be doc strings

edenhill · 2018-12-01T10:20:13Z

examples/avro-cli.py

+        self.favorite_number = favorite_number
+        self.favorite_color = favorite_color
+
+    # The Avro Python library does not support code generation.


examples/avro-cli.py

edenhill · 2018-12-01T10:23:40Z

examples/avro-cli.py

+            record.favorite_number = int(input("Enter favorite number: "))
+            record.favorite_color = input("Enter favorite color: ")
+
+            producer.produce(topic=topic, partition=0, value=record.to_dict(), callback=on_delivery)


suggest not specifying the partition, is atypical, and not needed with subscribe()

ack, leftover from usage of assign

we could use this opportunity to demonstrate how you would do externalized custom partitioning.

But the python client does not support pluggable partitioners, and it doesn't seem relevant for show-casing avro.

I meant how you could still use a custom partitioner externally (i.e. explicitly set the partition with the result of some hashing function). You are right though that is out of place here.

examples/avro-cli.py

confluent_kafka/avro/__init__.py

examples/avro-cli.py

edenhill

one small change with brackets, then good to go!

examples/avro-cli.py

edenhill · 2018-12-03T20:48:04Z

LGTM!

edenhill · 2018-12-03T20:49:01Z

Might want to consider fixing the history of recent commit-review-ping-pong commits.

…contexts.

ADDale · 2019-01-03T16:42:46Z

I am still getting the below error. please confirm if this has been fixed and can you share working exmaple for the same.
Running MacOS, Python3.6.5
Installed packages with
pip install confluent-kafka
pip install confluent-kafka[avro]

run the example avro producer script from https://github.com/confluentinc/confluent-kafka-python/blob/master/README.md having edited for my broker, schema registry and topic name.

error stack is :
Traceback (most recent call last):
File "/Users/07HO38/Documents/Python/OEHCS/kafka_produce_avro.py", line 43, in
avroProducer.produce(topic='idcs-5bea815f69d54c6daa5701a57b5914fa-JDBCUSERS', value=value, key=key)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/confluent_kafka/avro/init.py", line 80, in produce
value = self._serializer.encode_record_with_schema(topic, value_schema, value)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/confluent_kafka/avro/serializer/message_serializer.py", line 98, in encode_record_with_schema
schema_id = self.registry_client.register(subject, schema)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/confluent_kafka/avro/cached_schema_registry_client.py", line 141, in register
result, code = self._send_request(url, method='POST', body=body)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/confluent_kafka/avro/cached_schema_registry_client.py", line 97, in _send_request
return response.json(), response.status_code
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/models.py", line 892, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

ADDale · 2019-01-03T16:50:58Z

I am still getting the error. Can you please share the working code for the same?
run the example avro producer script from https://github.com/confluentinc/confluent-kafka-python/blob/master/README.md having edited for my broker, schema registry and topic name.

error stack is :
Traceback (most recent call last):
File "/Users/07HO38/Documents/Python/OEHCS/kafka_produce_avro.py", line 43, in
avroProducer.produce(topic='idcs-5bea815f69d54c6daa5701a57b5914fa-JDBCUSERS', value=value, key=key)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/confluent_kafka/avro/init.py", line 80, in produce
value = self._serializer.encode_record_with_schema(topic, value_schema, value)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/confluent_kafka/avro/serializer/message_serializer.py", line 98, in encode_record_with_schema
schema_id = self.registry_client.register(subject, schema)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/confluent_kafka/avro/cached_schema_registry_client.py", line 141, in register
result, code = self._send_request(url, method='POST', body=body)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/confluent_kafka/avro/cached_schema_registry_client.py", line 97, in _send_request
return response.json(), response.status_code
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/models.py", line 892, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

rnpridgeon requested a review from edenhill August 18, 2018 06:49

edenhill suggested changes Aug 20, 2018

View reviewed changes

mhowlett reviewed Aug 20, 2018

View reviewed changes

confluent_kafka/avro/cached_schema_registry_client.py Outdated Show resolved Hide resolved

mhowlett reviewed Sep 4, 2018

View reviewed changes

confluent_kafka/avro/__init__.py Outdated Show resolved Hide resolved

edenhill suggested changes Sep 14, 2018

View reviewed changes

edenhill suggested changes Oct 8, 2018

View reviewed changes

rnpridgeon force-pushed the basic_auth branch from ca83f84 to 5414e04 Compare November 9, 2018 17:03

rnpridgeon force-pushed the basic_auth branch from ea4a682 to 92b5c19 Compare November 19, 2018 04:06

edenhill suggested changes Nov 20, 2018

View reviewed changes

This was referenced Nov 22, 2018

How to send basic authentication parameters #489

Closed

"json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)" when running example avro producer #490

Closed

Fix for issue #443. #444

Closed

edenhill suggested changes Nov 26, 2018

View reviewed changes

rnpridgeon added 12 commits November 26, 2018 14:55

Add docker-compose cluster for use in integration tests

5b4cfc3

Update CachedSchemaRegistryClient constructor to use configuration dict

eb1b9b1

Add HTTP basic auth support to CachedSchemaRegistryClient

108a891

Add requests dependency to install requirements

9ffa273

Add Avro Serialization example

93b07a1

Move integration tests from example to tests

a99034e

Add runner script for running tests

f957b5c

Update tox.ini with new integration test location

234e13a

Add Delete schema support to CachedSchemaRegistryClient

87d67e4

Fix get_compatibility response parsing in CachedSchemaRegistryClient

01d5d57

Remove integration tests from examples directory

63693b3

Fix delete_subject ClientError format string

107911d

rnpridgeon force-pushed the basic_auth branch from c4e374f to 107911d Compare November 27, 2018 02:48

rnpridgeon mentioned this pull request Nov 27, 2018

Remove two default.topic.config uses #493

Closed

Update doc string format for Avro components

a6bcc87

edenhill suggested changes Nov 28, 2018

View reviewed changes

edenhill reviewed Nov 29, 2018

View reviewed changes

examples/serialization/avro/avro-cli.py Outdated Show resolved Hide resolved

edenhill reviewed Nov 29, 2018

View reviewed changes

examples/serialization/avro/avro-cli.py Outdated Show resolved Hide resolved

edenhill reviewed Nov 29, 2018

View reviewed changes

examples/serialization/avro/avro-cli.py Outdated Show resolved Hide resolved

edenhill reviewed Nov 29, 2018

View reviewed changes

examples/serialization/avro/avro-cli.py Outdated Show resolved Hide resolved

edenhill reviewed Nov 29, 2018

View reviewed changes

examples/serialization/avro/avro-cli.py Outdated Show resolved Hide resolved

edenhill reviewed Nov 29, 2018

View reviewed changes

examples/serialization/avro/avro-cli.py Outdated Show resolved Hide resolved

edenhill suggested changes Nov 30, 2018

View reviewed changes

edenhill suggested changes Dec 1, 2018

View reviewed changes

edenhill reviewed Dec 1, 2018

View reviewed changes

confluent_kafka/avro/__init__.py Show resolved Hide resolved

edenhill suggested changes Dec 1, 2018

View reviewed changes

edenhill suggested changes Dec 3, 2018

View reviewed changes

examples/avro-cli.py Outdated Show resolved Hide resolved

examples/avro-cli.py Outdated Show resolved Hide resolved

rnpridgeon mentioned this pull request Dec 3, 2018

V1.0 deprecations #498

Merged

edenhill approved these changes Dec 3, 2018

View reviewed changes

Add avro-cli example, rethrow deserialization exception with message …

6c81cbe

…contexts.

rnpridgeon force-pushed the basic_auth branch from 35e7eb1 to 6c81cbe Compare December 4, 2018 07:47

rnpridgeon merged commit df649a4 into master Dec 4, 2018

rnpridgeon deleted the basic_auth branch December 4, 2018 08:12

		@@ -0,0 +1 @@
		__import__('pkg_resources').declare_namespace(__name__)

Add support for HTTP Basic Auth #440

Add support for HTTP Basic Auth #440

Conversation

rnpridgeon commented Aug 18, 2018

edenhill left a comment

Choose a reason for hiding this comment

rnpridgeon commented Aug 30, 2018

edenhill left a comment

Choose a reason for hiding this comment

rnpridgeon commented Oct 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rnpridgeon commented Nov 19, 2018

edenhill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rnpridgeon commented Nov 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rnpridgeon commented Nov 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edenhill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edenhill left a comment

Choose a reason for hiding this comment

edenhill commented Dec 3, 2018

edenhill commented Dec 3, 2018

ADDale commented Jan 3, 2019

ADDale commented Jan 3, 2019

rnpridgeon commented Nov 27, 2018 •

edited

Loading