[Performance Test] Add Test Clients for Apache RocketMQ on Dockers #39

luyaoluo · 2020-06-16T04:21:16Z

test rocketmq with different number of topics,producers and consumers.

Test for different number of topics,producers and consumers

xieus

Some early feedback. Thank you for your contribution.

xieus · 2020-06-16T11:23:42Z

tools/rocketmq-test/client/src/main/java/demo/ProducerRunnable.java

+    @Override
+    public void run(){
+        try {
+            //设置生产者组名


@lly00 I do understand the comments :-) As this repository is a public repo, in order to be accessed by a broader community, I would recommend to replace the comments in English. Thank you.

xieus · 2020-06-16T11:28:16Z

tools/rocketmq-test/client/pom.xml

+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+


NOTE：This comment is not for pom.xml

Please remove the target folder in the PR as it is auto-generated. Instead, you can create a .gitignore file which could avoid checking any files or directories that you don't want to check in.

An example: https://github.com/futurewei-cloud/alcor/blob/master/.gitignore

xieus · 2020-06-16T11:30:31Z

tools/rocketmq-test/docker-rocketmq/base/scripts/runserver-customize.sh

@@ -0,0 +1,142 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more


Quick question: do we modify some existing scripts which are under ASF license? Could you add the source script link as the beginning of this file as well? Thank you.

haboy52581

add some comments, focus on the perf, also please remove binary files

haboy52581 · 2020-06-18T01:51:41Z

tools/rocketmq-test/client/src/main/java/demo/ConsumerRunnable.java

+        String temp_str = new String(temp);
+        return temp_str;
+    }
+    // private static void startConsumer(String topicName) throws Exception {


Please remove commented code

haboy52581 · 2020-06-18T01:52:09Z

tools/rocketmq-test/client/src/main/java/demo/ConsumerThreadPool.java

+        int consumerThreadCount = 20;
+        int topicNumber = 20;
+
+        String url = "localhost:9876";


make it read from central config

haboy52581 · 2020-06-18T01:53:38Z

tools/rocketmq-test/client/src/main/java/demo/ProducerRunnable.java

+        System.out.println(topicName + " " + "Start produce");
+        while (true) {
+            Message message = new Message(topicName,"TagTest",createSpecificSizeString(size).getBytes());
+            long startTime = System.currentTimeMillis();


please log thread info

haboy52581 · 2020-06-18T01:53:55Z

tools/rocketmq-test/client/src/main/java/demo/ProducerRunnable.java

+            producer.send(message);
+            long endTime = System.currentTimeMillis();
+            System.out.println(topicName + " " + "send message in %d",endTime-startTime);
+            Thread.sleep(1000);


why wait here?

haboy52581 · 2020-06-18T01:54:43Z

...test/client/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst

@@ -0,0 +1,7 @@
+/home/sdn02/rocketmq/Client/src/main/java/demo/ProducerThreadPool.java


make one home env instead of hard coding path

haboy52581 · 2020-06-18T01:56:08Z

tools/rocketmq-test/docker-rocketmq/base/Dockerfile

+# limitations under the License.
+#
+
+FROM centos:7


please merge Docker ops when possible to speed up and save layers

haboy52581 · 2020-06-18T01:57:05Z

tools/rocketmq-test/docker-rocketmq/base/scripts/runbroker-customize.sh

+# Set for `JAVA_OPT`.
+JAVA_OPT="${JAVA_OPT} -server -Xms${Xms} -Xmx${Xmx} -Xmn${Xmn}"
+JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:SurvivorRatio=8"
+JAVA_OPT="${JAVA_OPT} -verbose:gc -Xloggc:/dev/shm/mq_gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy"


we don't need verbose:gc here before we hit problem, which would slow down the perf

haboy52581 · 2020-06-18T01:57:28Z

tools/rocketmq-test/docker-rocketmq/base/scripts/runserver-customize.sh

+#===========================================================================================
+calculate_heap_sizes()
+{
+    case "`uname`" in


haboy52581 · 2020-06-18T02:06:38Z

tools/rocketmq-test/client/src/main/java/demo/ConsumerRunnable.java

+            consumer.registerMessageListener(new MessageListenerConcurrently() {
+                @Override
+                public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
+                    System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);


we don't need this msg , only the statistics

haboy52581 · 2020-06-18T02:07:35Z

tools/rocketmq-test/docker-rocketmq/rmq/rmq/brokerconf/broker.conf

+defaultTopicQueueNums=4
+
+#是否允许 Broker 自动创建Topic，建议线下开启，线上关闭 ！！！这里仔细看是false，false，false
+#原因下篇博客见~ 哈哈哈哈


what 's 哈哈哈哈

Let us move those comments completely from the PR. @lly00

xieus · 2020-10-09T06:25:06Z

@lly00 Thanks for uploading the test results 👍

@chenpiaoping This is also related to the MQ scaling path perf testing.

luyaoluo added 2 commits June 16, 2020 12:15

add test for rocketmq on docker

1464284

Test for different number of topics,producers and consumers

Update ConsumerRunnable.java

7c06ce3

xieus requested a review from haboy52581 June 16, 2020 10:59

xieus assigned luyaoluo Jun 16, 2020

xieus added feature New feature development good first issue Good for newcomers labels Jun 16, 2020

xieus added this to the v0.5 beta release milestone Jun 16, 2020

xieus changed the title ~~Add test for rocketmq on dockers~~ [Performance Test] Add Test Clients for RocketMQ on Dockers Jun 16, 2020

xieus changed the title ~~[Performance Test] Add Test Clients for RocketMQ on Dockers~~ [Performance Test] Add Test Clients for Apache RocketMQ on Dockers Jun 16, 2020

xieus reviewed Jun 16, 2020

View reviewed changes

xieus mentioned this pull request Jun 17, 2020

add test code for rocketmq on docker futurewei-cloud/alcor#250

Closed

haboy52581 reviewed Jun 18, 2020

View reviewed changes

luyaoluo added 2 commits July 30, 2020 00:10

re-commit test on rocketmq

f8b8394

Add the test result for RocketMQ and Pulsar

62a369b

luyaoluo added 3 commits October 9, 2020 14:45

Update MQTestResult.md

60ddfc3

add pulsar design doc

346d88a

Merge branch 'master' of github.com:lly00/alcor-int

0881a48

		@@ -0,0 +1,142 @@
		#!/bin/bash

		# Licensed to the Apache Software Foundation (ASF) under one or more

		@@ -0,0 +1,7 @@
		/home/sdn02/rocketmq/Client/src/main/java/demo/ProducerThreadPool.java

[Performance Test] Add Test Clients for Apache RocketMQ on Dockers #39

Are you sure you want to change the base?

[Performance Test] Add Test Clients for Apache RocketMQ on Dockers #39

Uh oh!

Conversation

luyaoluo commented Jun 16, 2020

Uh oh!

xieus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haboy52581 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xieus commented Oct 9, 2020

Uh oh!

Uh oh!