README.adoc

CDF Workshops

Introduction

In this hands-on workshop, you will build a full OT to IT workflow for an IoT Predictive Maintenance use case. Below is the architecture diagram, showing all the components you will setup over the lab exercises. While the diagram divides the components according to their location (factory, regional or datacenter level) in this workshop all such components will reside in one single host.

Workshops

Important

If this is your first time going through this content, please read the rest of this README introduction before jumping to the Labs.

If you already familiar with the instructions in the README, time to start working and see some interesting stuff! Pick your lab and let’s get started!

• Ingesting data from the edge

• NiFi and Streams Processing

• Creating Dashboards with Cloudera Data Viz

• Querying streams with SQL

• Change Data Capture (CDC) with Flink/SSB

• Streams Replication

• Spark and Fast Analytics with Kudu

• CDSW Experiments and Models

Use cases:

• Credit card fraud detection

Before you start

• Everything is Case-Sensitive.

• Check all your connections and spellings

• If you hit any problems like, for example, MiNiFi Not Sending Messages, check for clues in the Troubleshooting section below.

Pre-requisites

• Laptop with a supported OS (Windows 7 not supported).

• A modern browser like Google Chrome or Firefox (IE not supported).

Connecting to your cluster

You instructor will give access to a registration link where you can request a cluster. You should have 2 addresses for you one-node cluster: the public DNS name and the public IP address. With those addresses you can test the following connectivity to your cluster:

Note	The credentials for all the services below that require authentication are admin/Supersecret1 (capital "S").

1. Ensure you can connect to the following service using your browser:

   Service	URL	Comments
   Cloudera Manager	http://<public_dns>:7180/	Port 7183 when TLS is enabled
   Atlas	http://<public_dns>:31000/	Port 31443 when TLS is enabled
   CDSW	http://cdsw.<public_IP>.nip.io/	admin/Supersecret1
   CDP Data Visualization	http://viz.cdsw.<public_IP>.nip.io/
   Edge Flow Manager	http://<public_dns>:10088/efm/ui/
   Flink Dashboard	http://<public_dns>:8078/
   Hue	http://<public_dns>:8889/
   Knox	https://<public_dns>:9443/gateway/homepage/home/	Only when security is enabled
   NiFi	http://<public_dns>:8080/nifi/	Port 8443 when TLS is enabled
   NiFi Registry	http://<public_dns>:18080/nifi-registry/	Port 18433 when TLS is enabled
   Ranger	https://<public_dns>:6182/	Only when security is enabled
   Schema Registry	http://<public_dns>:7788/	Port 7790 when TLS is enabled
   SMM	http://<public_dns>:9991/
   SQL Stream Builder (SSB)	http://<public_dns>:18121/	For SSB versions older than 1.7 use this instead: http://<public_dns>:18121/

2. Login into Cloudera Manager and familiarize yourself with the services installed

3. Login into Hue. As you are the first user to login into Hue, you are granted admin privileges. At this point, you won’t need to do anything on Hue, but by logging in, CDH has created your HDFS user and folder, which you will need for the next lab.

Below a screenshot of Chrome open with 8 tabs, one for each service.

(Optional) SSH access

This access is not required for the workshop labs. You can skip this section.

SSH access is only required if you need to troubleshoot issues or want to poke around your clusters. The procedure to connect via SSH depends on the type of computer you’re using:

SSH into the cluster from the Web UI

From the registration link, you can click on the link at the right side to connect to the cluster from a web based SSH client with the credential centos/Supersecret1.

SSH into the cluster from Linux/Macos

From the registration link, download the PEM key required to access to your cluster with SSH. Run the following command:

chmod 400 workshop.pem
ssh -i workshop.pem centos@you-ip-address

SSH into the cluster from Windows

From the registration link, download the PEM key required to access to your cluster with SSH. We will use PuTTY to connect to the cluster. However, Putty doesn’t accept PEM key. Follow these instructions to convert your PEM key into a PPK key and connect to the cluster

Convert your key with PuTTYgen:

1. Use PuTTYgen to convert .PEM file to .PPK file.

2. Start PuTTYgen and select “Load”

3. Select your .PEM file.

4. Putty will convert the .PEM format to .PPK format.

5. Select “Save Private Key” A passphrase is not required but can be used if additional security is required.

Connect with PuTTY:

1. Launch PuTTY and enter the host IP address.

2. Navigate to Connection/SSH/Auth

3. Click “Browse” and select the .PPK file you exported from PuTTYgen.

4. Click “Open.”

Resources

• Original blog by Abdelkrim Hadjidj

• This workshop is based on the following work by Fabio Ghirardello:

  • https://github.com/fabiog1901/IoT-predictive-maintenance

  • https://github.com/fabiog1901/OneNodeCDHCluster

• Cloudera Documentation

Troubleshooting

General

• Everything is Case-Sensitive.

• Check all your connections and spellings

MiNiFi Not Sending Messages

• Make sure you pick HTTP, not RAW, in Cloud Connection to NiFi

• Make sure there are no spaces before or after Destination ID, URL, Names, Topics, Brokers, etc…​

• Make sure there are no spaces anywhere!

• Everything is Case-Sensitive. For example, the bucket name in NiFi Registry is IoT, not iot.

• Check /opt/cloudera/cem/minifi/logs/minifi-app.log if you can’t find an issue

• You must have HDFS User Created via HUE, Go there First

CEM doesn’t pick up new NARs

1. Delete the agent manifest manually using the EFM API:

2. Verify each class has the same agent manifest ID:

   http://hostname:10088/efm/api/agent-classes
   [{"name":"iot1","agentManifests":["agent-manifest-id"]},{"name":"iot4","agentManifests":["agent-manifest-id"]}]

3. Confirm the manifest doesn’t have the NAR you installed

   http://hostname:10088/efm/api/agent-manifests?class=iot4
   [{"identifier":"agent-manifest-id","agentType":"minifi-java","version":"1","buildInfo":{"timestamp":1556628651811,"compiler":"JDK 8"},"bundles":[{"group":"default","artifact":"system","version":"unversioned","componentManifest":{"controllerServices":[],"processors":

4. Call the API endpoint:

   http://hostname:10088/efm/swagger/

5. Hit the DELETE - Delete the agent manifest specified by id button, and in the id field, enter `agent-manifest-id