Skip to content

Conversation

@elek
Copy link
Member

@elek elek commented Apr 10, 2019

In HDDS-872 we added Dockerfile and skaffold definition to run dev builds on kubernetes. But it would be great to include example k8s resource definitions helping the deployment of ozone to any kubernetes cluster.

In this patch I will

  1. Add k8s resources files to the release tar file to deploy basic ozone cluster
  2. Add Dockerfile to the release tar file to create custom ozone image any time
  3. Add additional maven profiles to build and push development docker images.
  4. We don't need skaffold any more as the maven based approach is more flexible (we can support multiple k8s definitions)

To easily support multiple type of configuration (simple ozone, minikube, csi) we need a basic set of k8s resources files and additional transformations to generate the ready-to-use files for each specific use-cases.

The easiest way to do this is adopting the existing structure from https://github.com/flokkr/k8s and use https://github.com/elek/flekszible tool. But the tool itself is not required at runtime as we generate all the required k8s resources files during the development and add the results to the version control.

See: https://issues.apache.org/jira/browse/HDDS-1412

@elek elek added the ozone label Apr 10, 2019
metadata:
name: datanode
labels:
app.kubernetes.io/component: ozone

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:end of line

metadata:
name: s3g
labels:
app.kubernetes.io/component: ozone

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:end of line

annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9876"
prometheus.io/path: "/prom"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:end of line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 32 Docker mode activated.
_ Prechecks _
0 yamllint 0 yamllint was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 38 Maven dependency ordering for branch
+1 mvninstall 1123 trunk passed
+1 compile 132 trunk passed
+1 mvnsite 88 trunk passed
+1 shadedclient 634 branch has no errors when building and testing our client artifacts.
+1 javadoc 68 trunk passed
_ Patch Compile Tests _
0 mvndep 17 Maven dependency ordering for patch
-1 mvninstall 23 dist in the patch failed.
+1 compile 114 the patch passed
+1 javac 114 the patch passed
+1 hadolint 0 There were no new hadolint issues.
+1 mvnsite 54 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 shelldocs 13 The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236)
-1 whitespace 0 The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 xml 2 The patch has no ill-formed XML file.
+1 shadedclient 728 patch has no errors when building and testing our client artifacts.
+1 javadoc 60 the patch passed
_ Other Tests _
+1 unit 39 common in the patch passed.
+1 unit 26 dist in the patch passed.
+1 asflicense 33 The patch does not generate ASF License warnings.
3427
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-719/1/artifact/out/Dockerfile
GITHUB PR #719
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml hadolint shellcheck shelldocs yamllint
uname Linux c86b1f6071eb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 65deb1a
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
shellcheck v0.4.6
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-719/1/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-719/1/artifact/out/whitespace-eol.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-719/1/testReport/
Max. process+thread count 467 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/dist U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-719/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

metadata:
name: datanode
labels:
app.kubernetes.io/component: ozone

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:end of line

metadata:
name: s3g
labels:
app.kubernetes.io/component: ozone

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:end of line

annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9876"
prometheus.io/path: "/prom"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:end of line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 26 Docker mode activated.
_ Prechecks _
0 yamllint 0 yamllint was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 53 Maven dependency ordering for branch
+1 mvninstall 1103 trunk passed
-1 compile 40 hadoop-ozone in trunk failed.
+1 mvnsite 88 trunk passed
+1 shadedclient 659 branch has no errors when building and testing our client artifacts.
+1 javadoc 58 trunk passed
_ Patch Compile Tests _
0 mvndep 12 Maven dependency ordering for patch
-1 mvninstall 22 dist in the patch failed.
-1 compile 35 hadoop-ozone in the patch failed.
-1 javac 35 hadoop-ozone in the patch failed.
+1 hadolint 0 There were no new hadolint issues.
+1 mvnsite 53 the patch passed
+1 shellcheck 1 There were no new shellcheck issues.
+1 shelldocs 14 The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236)
-1 whitespace 0 The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 xml 3 The patch has no ill-formed XML file.
+1 shadedclient 730 patch has no errors when building and testing our client artifacts.
+1 javadoc 54 the patch passed
_ Other Tests _
+1 unit 41 common in the patch passed.
+1 unit 23 dist in the patch passed.
+1 asflicense 27 The patch does not generate ASF License warnings.
3207
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-719/2/artifact/out/Dockerfile
GITHUB PR #719
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml hadolint shellcheck shelldocs yamllint
uname Linux 9b78ecfa43fa 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / d608be6
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-719/2/artifact/out/branch-compile-hadoop-ozone.txt
shellcheck v0.4.6
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-719/2/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-719/2/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-719/2/artifact/out/patch-compile-hadoop-ozone.txt
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-719/2/artifact/out/whitespace-eol.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-719/2/testReport/
Max. process+thread count 447 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/dist U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-719/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@elek
Copy link
Member Author

elek commented May 2, 2019

This is reviewed as part of #783 by @anuengineer (that patch based on this commit the PR contained both the commits). Merging it right now.

@elek elek closed this in 2ab318b May 2, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 512 Docker mode activated.
_ Prechecks _
0 yamllint 1 yamllint was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
0 mvndep 36 Maven dependency ordering for branch
+1 mvninstall 1041 trunk passed
+1 compile 120 trunk passed
+1 mvnsite 93 trunk passed
+1 shadedclient 670 branch has no errors when building and testing our client artifacts.
+1 javadoc 58 trunk passed
_ Patch Compile Tests _
0 mvndep 12 Maven dependency ordering for patch
-1 mvninstall 18 dist in the patch failed.
+1 compile 106 the patch passed
+1 javac 106 the patch passed
+1 hadolint 0 There were no new hadolint issues.
+1 mvnsite 53 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 shelldocs 16 The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236)
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 2 The patch has no ill-formed XML file.
+1 shadedclient 702 patch has no errors when building and testing our client artifacts.
+1 javadoc 55 the patch passed
_ Other Tests _
+1 unit 39 common in the patch passed.
+1 unit 24 dist in the patch passed.
+1 asflicense 29 The patch does not generate ASF License warnings.
3761
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-719/3/artifact/out/Dockerfile
GITHUB PR #719
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml hadolint shellcheck shelldocs yamllint
uname Linux 2107fdc79bba 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / f682a17
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
shellcheck v0.4.6
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-719/3/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-719/3/testReport/
Max. process+thread count 411 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/dist U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-719/3/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
Author: Jagadish <jvenkatraman@linkedin.com>

Reviewers: Jagadish <jagadish@apache.org>

Closes apache#719 from vjagadish1989/website-reorg16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants