Registry request service #4

granders · 2015-01-27T19:13:35Z

This branch contains a few failover tests - clean kill, hard kill, clean bounce, and hard bounce. @ewencp @nehanarkhede @junrao Feedback is welcome, as are ideas for other failure modes that we should check.

A few thoughts/questions come to mind -

The http helper functions in schema_registry_utils.py are useful but currently need to be called locally. I suppose if a command line hook were added for http_request, the http requests could come from specified nodes by calling node.account.ssh(<http_command>) This might be useful if we want http requests coming in from multiple different machines.
I may want to add a failure mode where kafka brokers are killed.
In general, if we want to use ducttape results to check regressions, among other things, at some point we may want to archive results?

ewencp · 2015-01-27T22:52:54Z

ducttape/services/core.py

+        if clean_shutdown:
+            node.account.ssh("/opt/schema-registry/bin/schema-registry-stop", allow_fail=True)
+        else:
+            cmd = """TARGET=`ps ax | grep -i schema-registry | grep java | grep -v grep | awk '{print $1}'`;"""


Maybe preemptively refactor this into Service with a parameter to control which process you grep for? Seems like any test of failures is going to have to use this basic pattern.

…killing the next master.

…still need to fix subclassing PerformanceService.

ewencp · 2015-02-03T19:45:20Z

@granders A bunch of things were marked done but I don't see new commits. Is this ready for review and you forgot to push or is this still a WIP?

granders · 2015-02-03T19:47:09Z

@ewencp Ah, sorry for the confusion, still WIP, but I wanted an easy way to track which items I had already addressed.

…ter-schemas-service to everything-runs-test. Some small refactoring.

granders · 2015-02-04T18:20:03Z

@ewencp
I think I addressed your comments, ready for re-review.

A few notable additions/changes:

Added kill_process() in remoteaccount as a reusable tool
Added wait_for_http_service() in remoteaccount - pings a rest service until it's awake
Instead of specifying number of schemas in RegisterSchemasService, now the failover processes signal that the background threads can terminate by setting register_schemas_service.ready_to_finish to True

Other notes

The reporting for failover may still need some work

ewencp · 2015-02-04T19:38:52Z

ducttape/tests/sr_leader_failover.py

+            except:
+                success = False
+                summary += "%d was reported successful but actually failed\n" % id
+        if success:


No need to change anything for this patch, but you may find it cleaner (and more Pythonic) in the future to use one-liners like summary += "Success.\n" if success else "Failure.\n"

ewencp · 2015-02-04T19:38:58Z

@granders Looks good, +1!

@ewencp

Registry request service Thanks for reviewing @ewencp

Initial patch for KIP-25 Note that to install ducktape, do *not* use pip to install ducktape. Instead: ``` $ git clone gitgithub.com:confluentinc/ducktape.git $ cd ducktape $ python setup.py install ``` Author: Geoff Anderson <geoff@confluent.io> Author: Geoff <granders@gmail.com> Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ewen, Gwen, Jun, Guozhang Closes #70 from granders/KAFKA-2276 and squashes the following commits: a62fb6c [Geoff Anderson] fixed checkstyle errors a70f0f8 [Geoff Anderson] Merged in upstream trunk. 8b62019 [Geoff Anderson] Merged in upstream trunk. 47b7b64 [Geoff Anderson] Created separate tools jar so that the clients package does not pull in dependencies on the Jackson JSON tools or argparse4j. a9e6a14 [Geoff Anderson] Merged in upstream changes d18db7b [Geoff Anderson] fixed :rat errors (needed to add licenses) 321fdf8 [Geoff Anderson] Ignore tests/ and vagrant/ directories when running rat build task 795fc75 [Geoff Anderson] Merged in changes from upstream trunk. 1d93f06 [Geoff Anderson] Updated provisioning to use java 7 in light of KAFKA-2316 2ea4e29 [Geoff Anderson] Tweaked README, changed default log collection behavior on VerifiableProducer 0eb6fdc [Geoff Anderson] Merged in system-tests 69dd7be [Geoff Anderson] Merged in trunk 4034dd6 [Geoff Anderson] Merged in upstream trunk ede6450 [Geoff] Merge pull request #4 from confluentinc/move_muckrake 7751545 [Geoff Anderson] Corrected license headers e6d532f [Geoff Anderson] java 7 -> java 6 8c61e2d [Geoff Anderson] Reverted jdk back to 6 f14c507 [Geoff Anderson] Removed mode = "test" from Vagrantfile and Vagrantfile.local examples. Updated testing README to clarify aws setup. 98b7253 [Geoff Anderson] Updated consumer tests to pre-populate kafka logs e6a41f1 [Geoff Anderson] removed stray println b15b24f [Geoff Anderson] leftover KafkaBenchmark in super call 0f75187 [Geoff Anderson] Rmoved stray allow_fail. kafka_benchmark_test -> benchmark_test f469f84 [Geoff Anderson] Tweaked readme, added example Vagrantfile.local 3d73857 [Geoff Anderson] Merged downstream changes 42dcdb1 [Geoff Anderson] Tweaked behavior of stop_node, clean_node to generally fail fast 7f7c3e0 [Geoff Anderson] Updated setup.py for kafkatest c60125c [Geoff Anderson] TestEndToEndLatency -> EndToEndLatency 4f476fe [Geoff Anderson] Moved aws scripts to vagrant directory 5af88fc [Geoff Anderson] Updated README to include aws quickstart e5edf03 [Geoff Anderson] Updated example aws Vagrantfile.local 96533c3 [Geoff] Update aws-access-keys-commands 25a413d [Geoff] Update aws-example-Vagrantfile.local 884b20e [Geoff Anderson] Moved a bunch of files to kafkatest directory fc7c81c [Geoff Anderson] added setup.py 632be12 [Geoff] Merge pull request #3 from confluentinc/verbose-client 51a94fd [Geoff Anderson] Use argparse4j instead of joptsimple. ThroughputThrottler now has more intuitive behavior when targetThroughput is 0. a80a428 [Geoff Anderson] Added shell program for VerifiableProducer. d586fb0 [Geoff Anderson] Updated comments to reflect that throttler is not message-specific 6842ed1 [Geoff Anderson] left out a file from last commit 1228eef [Geoff Anderson] Renamed throttler 9100417 [Geoff Anderson] Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable. 0a5de8e [Geoff Anderson] Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements. 475423b [Geoff Anderson] Convert class to string before adding to json object. bc009f2 [Geoff Anderson] Got rid of VerboseProducer in core (moved to clients) c0526fe [Geoff Anderson] Updates per review comments. 8b4b1f2 [Geoff Anderson] Minor updates to VerboseProducer 2777712 [Geoff Anderson] Added some metadata to producer output. da94b8c [Geoff Anderson] Added number of messages option. 07cd1c6 [Geoff Anderson] Added simple producer which prints status of produced messages to stdout. a278988 [Geoff Anderson] fixed typos f1914c3 [Liquan Pei] Merge pull request #2 from confluentinc/system_tests 81e4156 [Liquan Pei] Bootstrap Kafka system tests

add `@ok_to_fail` decorator

…#416) * add logging (#1) * add logging * add logging * add logging * Perform SIGKILL on hanging threads (#3) * remove debug statements (#4) * remove UT, test using systests (#6)

Geoff Anderson added 5 commits January 23, 2015 13:24

Functional partial port of schema registry rest utils methods.

7166678

Ignore ipython notebook files.

d7d57eb

Functional failover tests.

4bee883

Restored everything runs test.

676aeab

Minor changes.

05c398f

ewencp reviewed Jan 27, 2015
View reviewed changes

Geoff Anderson added 6 commits January 27, 2015 15:23

Add a timeout while waiting for schema registry service to start.

889c6db

Added wait_for_http_service to address issue #3

96c8593

In bounce failover tests, wait for previous master to wake up before …

f055213

…killing the next master.

Minor cleanup.

8913089

Some refactoring of KafkaRestService and SchemaRegistryService.

99c6c8b

Restores test functionality, adds some of Ewens suggestions - though …

20553cc

…still need to fix subclassing PerformanceService.

Geoff Anderson added 2 commits February 3, 2015 14:57

Gave tests more control over when to stop register schemas service.

0bbe474

Merged in master.

04a4fea

granders mentioned this pull request Feb 3, 2015

Refactor RegisterSchemasService to share logic with PerformanceService #11

Closed

Geoff Anderson added 2 commits February 3, 2015 16:12

Added headers for kafka-rest to fix everything-runs-test. Added regis…

0fb0648

…ter-schemas-service to everything-runs-test. Some small refactoring.

Updated summary reporting.

c1a5bbf

ewencp reviewed Feb 4, 2015
View reviewed changes

granders added a commit that referenced this pull request Feb 4, 2015

Merge pull request #4 from confluentinc/registry-request-service

2e5fbee

Registry request service Thanks for reviewing @ewencp

granders merged commit 2e5fbee into master Feb 4, 2015

granders deleted the registry-request-service branch February 4, 2015 19:42

granders mentioned this pull request Feb 18, 2015

Use simple HTTP requests instead of time.sleep to more reliably detect when services are ready #3

Closed

dotnwat pushed a commit to dotnwat/ducktape that referenced this pull request Mar 8, 2023

Merge pull request confluentinc#4 from andrewhsu/ok_to_fail

a60d8e9

add `@ok_to_fail` decorator

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Registry request service #4

Registry request service #4

granders commented Jan 27, 2015

ewencp Jan 27, 2015

granders Feb 3, 2015

ewencp commented Feb 3, 2015

granders commented Feb 3, 2015

granders commented Feb 4, 2015

ewencp Feb 4, 2015

ewencp commented Feb 4, 2015

Registry request service #4

Registry request service #4

Conversation

granders commented Jan 27, 2015

ewencp Jan 27, 2015

Choose a reason for hiding this comment

granders Feb 3, 2015

Choose a reason for hiding this comment

ewencp commented Feb 3, 2015

granders commented Feb 3, 2015

granders commented Feb 4, 2015

ewencp Feb 4, 2015

Choose a reason for hiding this comment

ewencp commented Feb 4, 2015