Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl #1918

swenzel · 2019-10-02T16:55:53Z

This change is

swenzel · 2019-10-02T16:57:57Z

Documentation is still missing, I'll add something on Friday.

swenzel · 2019-10-17T10:42:04Z

@dpkp @jeffwidman ready to be reviewed now.
That one test that is still failing seems to only fail some of the times. My best guess is that the broker itself hasn't completely fetched the data yet and that a small sleep might fix this.
But I think I have already fixed more than I should 😅
Looking forward to reading your feedback :)

ofek · 2019-11-25T17:27:49Z

@dpkp @jeffwidman We are extremely interested in this as well. Could you please take a look?

israelglar · 2019-12-06T16:28:04Z

Would be great to have this feature. Is there a timeframe for when we can merge it?

Thanks!

dpkp

Thanks so much for this PR and the excellent work on test fixtures!

dpkp · 2019-12-29T01:07:26Z

.travis.yml

@@ -25,7 +25,7 @@ addons:
 cache:
  directories:
    - $HOME/.cache/pip
-    - servers/
+    - servers/dist


good catch!

dpkp · 2019-12-29T01:08:37Z

kafka/client_async.py

-            return found
-
-        return None
+        return found


dpkp · 2019-12-29T01:11:57Z

test/__init__.py

-            pass
-
-logging.getLogger(__name__).addHandler(NullHandler())
+logging.basicConfig(level=logging.INFO)


What's your thinking on the logging changes here?

dpkp · 2019-12-29T01:15:01Z

kafka/conn.py

@@ -98,6 +107,69 @@ class ConnectionStates(object):
    AUTHENTICATING = '<authenticating>'


+class ScramClient:


Thoughts on moving this to its own file ?

dpkp · 2019-12-29T01:17:45Z

kafka/conn.py

+                close = False
+            else:
+                try:
+                    client_first = scram_client.first_message().encode()


should we specify an encoding explicitly? is 'utf-8' always the default? does it matter?

dpkp · 2019-12-29T01:19:23Z

kafka/conn.py

+                    self._send_bytes_blocking(size + client_first)
+
+                    (data_len,) = struct.unpack('>i', self._recv_bytes_blocking(4))
+                    server_first = self._recv_bytes_blocking(data_len).decode()


similarly re: codec. would prefer to specific 'utf-8' explicitly

dpkp · 2019-12-29T01:19:33Z

kafka/conn.py

+                    server_first = self._recv_bytes_blocking(data_len).decode()
+                    scram_client.process_server_first_message(server_first)
+
+                    client_final = scram_client.final_message().encode()


and here - utf-8

dpkp · 2019-12-29T01:19:48Z

kafka/conn.py

+                    self._send_bytes_blocking(size + client_final)
+
+                    (data_len,) = struct.unpack('>i', self._recv_bytes_blocking(4))
+                    server_final = self._recv_bytes_blocking(data_len).decode()


and finally also here

dpkp · 2019-12-29T01:25:55Z

test/fixtures.py

+        if sasl_mechanism is not None:
+            self.sasl_mechanism = sasl_mechanism.upper()
+        else:
+            self.sasl_mechanism = None


the else clause seems redundant?

dpkp · 2019-12-29T01:33:12Z

test/fixtures.py

+                # wait and try again
+                # on travis the brokers sometimes take a while to find themselves
+                time.sleep(0.5)
+                self._create_topic_via_admin_api(topic_name, num_partitions, replication_factor, timeout_ms)


might prefer an outer loop to enable as many retries as needed until timeout

ofek · 2020-01-09T13:48:43Z

@dpkp Hello! Will this be released soon?

ofek · 2020-01-16T01:10:44Z

@dpkp @jeffwidman Any update?

ofek · 2020-01-16T18:52:06Z

Alternatively, can I please do something to help?

jeffwidman · 2020-01-16T19:55:54Z

Hey @ofek I don't have the rights to cut a new release, but I just pinged @dpkp and he said on IM that he's currently traveling w/o his laptop for the next week so can't look at it until he returns. Sorry about that.

If you weren't shipping to end-users and just using internally then it'd be trivial to pin to a specific commit hash rather than needing a release... but alas, you are shipping to end users.

ofek · 2020-01-16T20:10:52Z

No worries, thanks!

ofek · 2020-01-28T21:32:48Z

Hello again! Any update?

jfpatenaude · 2020-02-06T18:19:02Z

I'm also waiting on this feature, @dpkp can you give us an update on the expected timeline?

terence-bigtt · 2020-05-19T19:52:37Z

looking forward for that too !

ofek · 2020-05-19T19:58:27Z

@terence-bigtt This is released.

Neustradamus · 2022-01-11T00:06:06Z

@swenzel, @dpkp: Thanks a lot for all about SCRAM.

Can you look, a guy has found a bug?

poll never stops with incorrect password for SCRAM-SHA-256 #2238

Linked to:

State of Play scram-sasl/info#1

Swen Wenzel added 29 commits October 16, 2019 15:10

add missing dev requirement 'mock'

ce61dfa

extend KafkaFixture to support scram and plain sasl mechanisms

b565067

write test for PLAIN sasl mechanism

fd8f11c

normalize KafkaFixture.get_clients .get_consumers and .get_producers

67d5276

refactor KafkaFixture.get_clients .get_consumers and .get_producers

471029b

add get_topic_names to KafkaFixture

f0b0389

fix bug in KafkaFixture.create_topics

d6761bd

create function to turn special chars to underscore

6077c6c

improve KafkaFixture.get_consumers

37d100d

improve sasl tests and enable tests for scram

d297767

implement scram

a97386d

make sure to wait for scram credentials during kafka startup

3c3c5a0

ensure compatibility with older python versions

9785c05

enable logging for tests

0570d91

add more output to SpawnedService

d7fe59b

add more checks if SpawnedService is actually running

22b6190

switch to actual jaas config file for kafka

84b57ed

remove sasl mechanism settings from server.properties for 1.1.1

01f48d7

change scram_pattern in KafkaFixture

3f399d3

minor refactoring of least_loaded_node

fecdc46

update docstrings with information about new sasl mechanisms

9a99f95

minor refactoring of test_sasl_integration.py

60702c6

trying to make it run in travis

b61d68f

go back to raising runtime error in wait_for

e641521

reduce kafka startup timeout and log config file content

720daa3

reduce tests to get quicker feedback

de6a360

output config file contents earlier

1f98ee4

further investiagte settings

fa1ca01

try to get rid of emtpy log lines

e41c0be

Swen Wenzel added 9 commits October 16, 2019 18:01

dump logs on sasl_kafka tear down

8aca0ee

try shorter topic name for sasl test

d5e9a0f

enable debug mode for kafka

a3aadf3

refactor KafkaFixture._create_topics and include retry logic

c0b0437

fix bug in KafkaFixture._send_request

ca3c304

another minor bugfix

e780de8

revert changes introduced for debugging

1fe4c17

minor bugfix

bce9e41

increase Kafka startup timeout again

54922a0

dpkp approved these changes Dec 29, 2019

View reviewed changes

This was referenced Dec 29, 2019

SASL SCRAM support #1920

Closed

kafka 2.4.0 #1972

Merged

dpkp merged commit ee1c4a4 into dpkp:master Dec 29, 2019

SukiCZ mentioned this pull request Jan 16, 2020

Support for sasl mechanism: SCRAM-SHA-256 aio-libs/aiokafka#571

Closed

swenzel deleted the enable-sasl branch March 2, 2020 15:18

This was referenced Jan 10, 2022

Scram mechanism support #1486

Closed

Support for sasl mechanism: SCRAM-SHA-256 #1882

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl #1918

Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl #1918

swenzel commented Oct 2, 2019 •

edited by dpkp

Loading

swenzel commented Oct 2, 2019

swenzel commented Oct 17, 2019

ofek commented Nov 25, 2019

israelglar commented Dec 6, 2019

dpkp left a comment

dpkp Dec 29, 2019

dpkp Dec 29, 2019

dpkp Dec 29, 2019

dpkp Dec 29, 2019

dpkp Dec 29, 2019

dpkp Dec 29, 2019

dpkp Dec 29, 2019

dpkp Dec 29, 2019

dpkp Dec 29, 2019

dpkp Dec 29, 2019

ofek commented Jan 9, 2020

ofek commented Jan 16, 2020

ofek commented Jan 16, 2020

jeffwidman commented Jan 16, 2020 •

edited

Loading

ofek commented Jan 16, 2020

ofek commented Jan 28, 2020

jfpatenaude commented Feb 6, 2020

terence-bigtt commented May 19, 2020

ofek commented May 19, 2020

Neustradamus commented Jan 11, 2022

		@@ -98,6 +107,69 @@ class ConnectionStates(object):
		AUTHENTICATING = '<authenticating>'


		class ScramClient:

Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl #1918

Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl #1918

Conversation

swenzel commented Oct 2, 2019 • edited by dpkp Loading

swenzel commented Oct 2, 2019

swenzel commented Oct 17, 2019

ofek commented Nov 25, 2019

israelglar commented Dec 6, 2019

dpkp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ofek commented Jan 9, 2020

ofek commented Jan 16, 2020

ofek commented Jan 16, 2020

jeffwidman commented Jan 16, 2020 • edited Loading

ofek commented Jan 16, 2020

ofek commented Jan 28, 2020

jfpatenaude commented Feb 6, 2020

terence-bigtt commented May 19, 2020

ofek commented May 19, 2020

Neustradamus commented Jan 11, 2022

swenzel commented Oct 2, 2019 •

edited by dpkp

Loading

jeffwidman commented Jan 16, 2020 •

edited

Loading