Apache NiFi on EKS #99

codesometech · 2023-02-03T17:02:40Z

What does this PR do?

🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.

This PR enables deployment on Apache NiFi on EKS using EKS Blueprints.

#80

Motivation

AWS Data Lab team observed customers looking for guidance on running Apache NiFi on EKS. Currently customers are running Apache NiFi on EC2. Running Apache NiFi on EKS simplifies operations (deployments, observability, etc.) for customers who are already running other workloads on Amazon EKS

More

Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
Yes, I have added a new example under examples to support my PR
Yes, I have created another PR to add a new Add-on for Terraform EKS Blueprints repo (if applicable)
Yes, I have updated the website/docs or website/blog section for this feature
[ X] Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

E2E Test successfully complete before merge?

Additional Notes

A working example and documentation for the website is currently under development.

victorgu-github · 2023-02-07T16:28:52Z

website/docs/streaming-platforms-eks/nifi.md

+
+# Apache NiFi on EKS
+
+### Coming soon..


could you pls add instruction about how to deploy this example here?
I tried terraform init and apply but see ACM cert is required. Don't know how to proceed.

streaming/nifi/addons.tf

codesometech · 2023-02-16T15:25:53Z

@season1946 NiFi.md is updated with instructions

victorgu-github · 2023-02-16T21:16:05Z

@codesometech hosted zone and acm cert are not optional.

Since the doc says it is optional, I directly copy/paste the code below but got error

export TF_VAR_eks_cluster_domain="example.com"
export TF_VAR_acm_certificate_domain="*.example.com"
export TF_VAR_nifi_sub_domain="nifi"
export TF_VAR_nifi_username="admin"

codesometech · 2023-02-21T15:21:36Z

@season1946 Documentation is fixed. To proceed you will need a hosted zone in Route53 and an ACM cert for that domain.

victorgu-github · 2023-02-21T19:36:41Z

@codesometech thanks for the explanation. looks good to me.
@vara-bonthu

askulkarni2 · 2023-02-20T15:15:57Z

streaming/nifi/addons.tf

+  enable_amazon_eks_vpc_cni            = true
+  enable_amazon_eks_coredns            = true
+  enable_amazon_eks_kube_proxy         = true
+  enable_amazon_eks_aws_ebs_csi_driver = true


Is this actually used? Looks like the default in-tree gp2 StorageClass is used in helm chart.

eks_aws_ebs_csi_driver is now mandatory form eks v1.23 even with default gp2.

askulkarni2 · 2023-02-20T15:17:30Z

streaming/nifi/addons.tf

+    name       = "metrics-server"
+    repository = "https://kubernetes-sigs.github.io/metrics-server/" # (Optional) Repository URL where to locate the requested chart.
+    chart      = "metrics-server"
+    version    = "3.8.2"
+    namespace  = "kube-system"
+    timeout    = "300"


We should try to keep the code less verbose and remove the default values. Please do this for all add-ons below.

askulkarni2 · 2023-02-21T23:54:36Z

streaming/nifi/addons.tf

+  special          = true
+  override_special = "!#$%&*()-_=+[]{}<>:?"
+}
+#tfsec:ignore:aws-ssm-secret-use-customer-key


nit: missing newline

askulkarni2 · 2023-02-21T23:54:51Z

streaming/nifi/addons.tf

+  length  = 16
+  special = false
+}
+#tfsec:ignore:aws-ssm-secret-use-customer-key


nit: missing newline

askulkarni2 · 2023-02-21T23:55:06Z

streaming/nifi/addons.tf

+  length  = 16
+  special = false
+}
+#tfsec:ignore:aws-ssm-secret-use-customer-key


nit: missing newline

askulkarni2 · 2023-02-22T00:12:10Z

streaming/nifi/vpc.tf

+  }
+
+  private_subnet_tags = {
+    "kubernetes.io/cluster/${local.name}" = "shared"


this tag is no longer needed

tag removed

askulkarni2 · 2023-02-22T00:13:10Z

website/docs/streaming-platforms/nifi.md

@@ -0,0 +1,205 @@
+---


Remove references to "ray" in the doc.

askulkarni2 · 2023-02-22T00:15:57Z

website/docs/streaming-platforms/nifi.md

+sidebar_label: Apache NiFi on EKS
+---
+
+# Apache NiFi on EKS


Please add some details on what Nifi is and when should customers consider Nifi. References to some pertinent documentation would also be useful here.

Added intro section

askulkarni2 · 2023-02-22T00:16:56Z

website/docs/streaming-platforms/nifi.md

+
+Coming soon..
+
+### Monitoring


Is there a grafana dashboard?

Grafana dashboard link added

askulkarni2 · 2023-02-22T00:17:58Z

website/docs/streaming-platforms/nifi.md

+
+### Examples
+
+Coming soon..


Can we add a simple example now? Without example the blueprint does not make much sense.

Example added

codesometech · 2023-03-03T20:42:52Z

@askulkarni2 I made the suggested changes and also moved to the new pattern for eks cluster.

vara-bonthu

@codesometech Excellent work on this PR Praveen 🥇 We are good to merge this PR if you double check the comments from Apoorva and ensure the checks are passing.

codesometech · 2023-03-07T06:43:18Z

@codesometech Excellent work on this PR Praveen 🥇 We are good to merge this PR if you double check the comments from Apoorva and ensure the checks are passing.

@vara-bonthu I added a couple of missing new lines. TFlint is failing in emr-eks-ack-crossplane and ray. Rest of the checks are passing.

Praveen Nerellapalli and others added 11 commits January 30, 2023 21:16

Monitoring and external-dns changes

3b69fe3

Domain filter fix

e152d8f

Merge branch 'awslabs:main' into main

7fc9eba

Docs and removed S3 for logs

1bdce22

Merge branch 'main' of https://github.com/codesometech/data-on-eks

ef330a8

Moved sensitive key to secrets manager

30db93d

Moved nifi to streaming

c232f8e

Added comments to values.yaml

18911c1

Changed domain names to required

791bf12

Changed description for nifi subdomain variable

3115595

Precommit changes

6420b88

codesometech requested a review from a team as a code owner February 3, 2023 17:02

codesometech temporarily deployed to DoEKS Test February 3, 2023 17:02 — with GitHub Actions Inactive

victorgu-github reviewed Feb 7, 2023

View reviewed changes

streaming/nifi/addons.tf Show resolved Hide resolved

Merge branch 'awslabs:main' into main

64e772f

codesometech temporarily deployed to DoEKS Test February 15, 2023 15:21 — with GitHub Actions Inactive

NiFi documentation

6aa8db6

codesometech temporarily deployed to DoEKS Test February 15, 2023 16:08 — with GitHub Actions Inactive

Updated NiFi documentation

57b1169

askulkarni2 reviewed Feb 22, 2023

View reviewed changes

Merge branch 'awslabs:main' into main

573f574

codesometech temporarily deployed to DoEKS Test February 24, 2023 17:33 — with GitHub Actions Inactive

Praveen Nerellapalli added 3 commits February 24, 2023 13:44

Merge branch 'main' of https://github.com/codesometech/data-on-eks

cfe9546

PR comments changes

5d4f303

Formatting

927856d

Praveen Nerellapalli and others added 4 commits February 28, 2023 16:05

Formatting

2e3ff0b

Pre commit changes

37b8117

Website changes

31e1ee2

Merge branch 'awslabs:main' into main

61d1812

codesometech temporarily deployed to DoEKS Test March 3, 2023 19:57 — with GitHub Actions Inactive

Praveen Nerellapalli added 3 commits March 3, 2023 15:04

Website changes

67db536

Website changes

fc0c249

Merge branch 'main' of https://github.com/codesometech/data-on-eks

9d6164c

codesometech temporarily deployed to DoEKS Test March 3, 2023 20:06 — with GitHub Actions Inactive

Grafana dashboard link

e17a64e

codesometech temporarily deployed to DoEKS Test March 3, 2023 20:41 — with GitHub Actions Inactive

vara-bonthu approved these changes Mar 5, 2023

View reviewed changes

Praveen Nerellapalli and others added 2 commits March 7, 2023 07:29

new line

b061615

Merge branch 'awslabs:main' into main

14c0239

codesometech temporarily deployed to DoEKS Test March 7, 2023 06:29 — with GitHub Actions Inactive

Merge branch 'main' of https://github.com/codesometech/data-on-eks

ca991be

codesometech temporarily deployed to DoEKS Test March 7, 2023 06:35 — with GitHub Actions Inactive

vara-bonthu merged commit 8d3806e into awslabs:main Mar 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Apache NiFi on EKS #99

Apache NiFi on EKS #99

codesometech commented Feb 3, 2023

victorgu-github Feb 7, 2023

codesometech commented Feb 16, 2023

victorgu-github commented Feb 16, 2023

codesometech commented Feb 21, 2023

victorgu-github commented Feb 21, 2023

askulkarni2 Feb 20, 2023

vara-bonthu Mar 5, 2023

askulkarni2 Feb 20, 2023

askulkarni2 Feb 21, 2023

askulkarni2 Feb 21, 2023

askulkarni2 Feb 21, 2023

askulkarni2 Feb 22, 2023

codesometech Mar 7, 2023

askulkarni2 Feb 22, 2023

codesometech Mar 7, 2023

askulkarni2 Feb 22, 2023

codesometech Mar 3, 2023

askulkarni2 Feb 22, 2023

codesometech Mar 3, 2023

askulkarni2 Feb 22, 2023

codesometech Mar 3, 2023

codesometech commented Mar 3, 2023

vara-bonthu left a comment

codesometech commented Mar 7, 2023

Apache NiFi on EKS #99

Apache NiFi on EKS #99

Conversation

codesometech commented Feb 3, 2023

What does this PR do?

Motivation

More

For Moderators

Additional Notes

Choose a reason for hiding this comment

codesometech commented Feb 16, 2023

victorgu-github commented Feb 16, 2023

codesometech commented Feb 21, 2023

victorgu-github commented Feb 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codesometech commented Mar 3, 2023

vara-bonthu left a comment

Choose a reason for hiding this comment

codesometech commented Mar 7, 2023