[Enhancement] refactor allin1 image #28240

kevincai · 2023-07-29T04:56:28Z

add supervisor and nginx into allin1
add broker into allin1
robust register BE BROKER into FE
be able to use stream load directly talking to nginx http port

Fixes #13319

What type of PR is this:

Checklist:

I have added test cases for my bug fix or my new feature
This pr will affect users' behaviors
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

imay

Wonderful job, how much additional volume are added to the image size after this change?

docker/dockerfiles/allin1/services/director/run.sh

kevincai · 2023-07-30T03:50:09Z

about 450MB fs_broker size, I guess around 100-200MB additional storage after compress.

kevincai · 2023-07-30T04:02:48Z

docker/dockerfiles/allin1/services/director/run.sh

+    loginfo "check if need to add BE into FE service ..."
+    while true
+    do
+        result=`exec_sql_with_retry "SHOW BACKENDS;"`


result can be possibly mixtured with error messages, need fix it here.

* add supervisor and nginx into allin1 * add broker into allin1 * robust register BE BROKER into FE Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

* support aarch platform as well Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

yandongxiao · 2023-07-31T06:20:59Z

docker/dockerfiles/allin1/allin1-ubi.Dockerfile

-RUN yum install -y java-1.8.0-openjdk-devel tzdata openssl curl vim ca-certificates fontconfig gzip tar less hostname procps-ng lsof && \
-    rpm -ivh https://repo.mysql.com/mysql57-community-release-el7.rpm && \
+RUN yum install -y java-1.8.0-openjdk-devel tzdata openssl curl vim ca-certificates fontconfig gzip tar less hostname procps-ng lsof python3-pip nginx nc && \
+    rpm -ivh https://repo.mysql.com//mysql80-community-release-el8-7.noarch.rpm && \


yandongxiao · 2023-07-31T06:25:51Z

docker/dockerfiles/be/be-ubi.Dockerfile

@@ -28,9 +28,9 @@ FROM registry.access.redhat.com/ubi8/ubi:8.7
 ARG STARROCKS_ROOT=/opt/starrocks

 RUN yum install -y java-1.8.0-openjdk-devel tzdata openssl curl vim ca-certificates fontconfig gzip tar less hostname procps-ng lsof && \
-    rpm -ivh https://repo.mysql.com/mysql57-community-release-el7.rpm && \
+    rpm -ivh https://repo.mysql.com//mysql80-community-release-el8-7.noarch.rpm &&  \


double slash

yandongxiao · 2023-07-31T06:26:14Z

docker/dockerfiles/fe/fe-ubi.Dockerfile

@@ -28,9 +28,9 @@ FROM registry.access.redhat.com/ubi8/ubi:8.7
 ARG STARROCKS_ROOT=/opt/starrocks

 RUN yum install -y java-1.8.0-openjdk-devel tzdata openssl curl vim ca-certificates fontconfig gzip tar less hostname procps-ng lsof && \
-    rpm -ivh https://repo.mysql.com/mysql57-community-release-el7.rpm && \
+    rpm -ivh https://repo.mysql.com//mysql80-community-release-el8-7.noarch.rpm && \


double slash

yandongxiao · 2023-07-31T06:57:44Z

docker/dockerfiles/allin1/services/director/run.sh

+loginfo
+loginfo "Enjoy the journal to StarRocks blazing-fast lake-house engine!"
+
+while true


this while loop is used to make run.sh run as a service, to avoid to exit?

Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

wanpengfei-git · 2023-07-31T07:03:07Z

[FE PR Coverage Check]

😍 pass : 0 / 0 (0%)

wanpengfei-git · 2023-07-31T08:16:39Z

@Mergifyio backport branch-3.1

wanpengfei-git · 2023-07-31T08:16:41Z

@Mergifyio backport branch-3.0

wanpengfei-git · 2023-07-31T08:16:42Z

@Mergifyio backport branch-2.5

mergify · 2023-07-31T08:16:45Z

backport branch-3.1

✅ Backports have been created

#28298 [Enhancement] refactor allin1 image (backport #28240) has been created for branch branch-3.1

mergify · 2023-07-31T08:16:47Z

backport branch-3.0

✅ Backports have been created

#28299 [Enhancement] refactor allin1 image (backport #28240) has been created for branch branch-3.0

mergify · 2023-07-31T08:16:49Z

backport branch-2.5

✅ Backports have been created

#28300 [Enhancement] refactor allin1 image (backport #28240) has been created for branch branch-2.5 but encountered conflicts

* add supervisor and nginx into allin1 * add broker into allin1 * robust register BE BROKER into FE * be able to use stream load directly talking to nginx http port Fixes #13319 Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com> (cherry picked from commit 011bf17)

* add supervisor and nginx into allin1 * add broker into allin1 * robust register BE BROKER into FE * be able to use stream load directly talking to nginx http port Fixes #13319 Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com> (cherry picked from commit 011bf17) # Conflicts: # .dockerignore

* add supervisor and nginx into allin1 * add broker into allin1 * robust register BE BROKER into FE * be able to use stream load directly talking to nginx http port Fixes #13319 Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com> (cherry picked from commit 011bf17)

* add supervisor and nginx into allin1 * add broker into allin1 * robust register BE BROKER into FE * be able to use stream load directly talking to nginx http port Fixes #13319 (cherry picked from commit 011bf17) Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

github-actions bot added the 3.1 label Jul 29, 2023

mergify bot assigned kevincai Jul 29, 2023

kevincai force-pushed the allin1-enhancement branch from 15fbead to 93c3d7c Compare July 29, 2023 05:02

imay previously approved these changes Jul 29, 2023

View reviewed changes

kevincai dismissed imay’s stale review via b91332c July 29, 2023 09:32

github-actions bot added 3.0 2.5 labels Jul 29, 2023

imay reviewed Jul 30, 2023

View reviewed changes

docker/dockerfiles/allin1/services/director/run.sh Show resolved Hide resolved

kevincai force-pushed the allin1-enhancement branch from b91332c to 5605444 Compare July 30, 2023 03:55

kevincai commented Jul 30, 2023

View reviewed changes

imay previously approved these changes Jul 30, 2023

View reviewed changes

kevincai dismissed imay’s stale review via 7a42b22 July 30, 2023 09:56

kevincai mentioned this pull request Jul 30, 2023

Stream load Parquet file #13319

Open

kevincai force-pushed the allin1-enhancement branch 3 times, most recently from bbdec9c to 1502fa1 Compare July 31, 2023 03:44

kevincai added 5 commits July 31, 2023 14:37

[Enhancement] refactor allin1 image

1f4c8f6

* add supervisor and nginx into allin1 * add broker into allin1 * robust register BE BROKER into FE Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

use mysql-client-community 8 client for ubi

d236fda

* support aarch platform as well Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

fix allin1 with broker build from local dev path

9fb2da3

Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

add license header

b6850a5

Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

fix logerr polutes result

89d93c4

Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

yandongxiao reviewed Jul 31, 2023

View reviewed changes

fix double slash in url link

1eb0f84

Signed-off-by: Kevin Xiaohua Cai <caixiaohua@starrocks.com>

kevincai force-pushed the allin1-enhancement branch from 1502fa1 to 1eb0f84 Compare July 31, 2023 07:01

yandongxiao approved these changes Jul 31, 2023

View reviewed changes

sduzh approved these changes Jul 31, 2023

View reviewed changes

wyb approved these changes Jul 31, 2023

View reviewed changes

wyb merged commit 011bf17 into StarRocks:main Jul 31, 2023

github-actions bot removed the 3.1 label Jul 31, 2023

github-actions bot removed the 3.0 label Jul 31, 2023

github-actions bot removed the 2.5 label Jul 31, 2023

This was referenced Jul 31, 2023

[Enhancement] refactor allin1 image (backport #28240) #28298

Merged

[Enhancement] refactor allin1 image (backport #28240) #28299

Merged

[Enhancement] refactor allin1 image (backport #28240) #28300

Closed

kevincai deleted the allin1-enhancement branch August 4, 2023 14:21

This was referenced Apr 19, 2024

[minor][cdc][docs] Fix outdated StarRocks quickstart guide apache/flink-cdc#3238

Merged

[FLINK-35092][cdc][starrocks] Add starrocks integration test cases apache/flink-cdc#3231

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] refactor allin1 image #28240

[Enhancement] refactor allin1 image #28240

kevincai commented Jul 29, 2023 •

edited

Loading

imay left a comment

kevincai commented Jul 30, 2023

kevincai Jul 30, 2023

yandongxiao Jul 31, 2023

kevincai Jul 31, 2023

yandongxiao Jul 31, 2023

kevincai Jul 31, 2023

yandongxiao Jul 31, 2023

kevincai Jul 31, 2023

yandongxiao Jul 31, 2023

kevincai Jul 31, 2023

wanpengfei-git commented Jul 31, 2023

wanpengfei-git commented Jul 31, 2023

wanpengfei-git commented Jul 31, 2023

wanpengfei-git commented Jul 31, 2023

mergify bot commented Jul 31, 2023 •

edited

Loading

mergify bot commented Jul 31, 2023 •

edited

Loading

mergify bot commented Jul 31, 2023 •

edited

Loading

[Enhancement] refactor allin1 image #28240

[Enhancement] refactor allin1 image #28240

Conversation

kevincai commented Jul 29, 2023 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

imay left a comment

Choose a reason for hiding this comment

kevincai commented Jul 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wanpengfei-git commented Jul 31, 2023

[FE PR Coverage Check]

wanpengfei-git commented Jul 31, 2023

wanpengfei-git commented Jul 31, 2023

wanpengfei-git commented Jul 31, 2023

mergify bot commented Jul 31, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Jul 31, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Jul 31, 2023 • edited Loading

✅ Backports have been created

kevincai commented Jul 29, 2023 •

edited

Loading

mergify bot commented Jul 31, 2023 •

edited

Loading

mergify bot commented Jul 31, 2023 •

edited

Loading

mergify bot commented Jul 31, 2023 •

edited

Loading