Skip to content

Commit c2dcd80

Browse files
gabydreckschmidtgabydreckschmidt
committed
Updates for 10/09 release
1 parent b14d169 commit c2dcd80

12 files changed

+1146
-11
lines changed

‎Known-issues.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
copyright:
44
years: 2017, 2020
5-
lastupdated: "2020-07-01"
5+
lastupdated: "2020-09-10"
66

77
subcollection: AnalyticsEngine
88

@@ -39,5 +39,6 @@ List of known issues in the release:
3939
|Spark History Server UI - StdOut and StdErr logs | The drilldown links to StdOut and StdErr logs from the Stages or Executors tabs are broken. |AE 1.2 <br/> Created before end November 2019| SSH to the cluster and run `yarn logs --applicationId <appId>` |
4040
| Broken rolling restart of worker daemons | The rolling restart of the slave components for the HDFS, Yarn, HBase, Ambari Metrics and Spark2 services is broken, resulting in an HTTP 403 error. | All versions | For now, a workaround is to restart the respective service as a whole from service action menu by selecting `Restart All`.|
4141
| Livy REST API | If you use the Livy REST API `GET /batches/{batchId}/log`, the logs are available only for a few hours after submitting the job. If you try to retrieve the logs many hours after submitting the job, the following error is displayed: `HTTP ERROR 500 : Problem accessing /gateway/default/livy/v1/batches. Reason: Server Error` | AE 1.1 | |
42-
| Adding additional libraries | Livy or IBM Cloud CLI commands fail after installing additional packages, for example PyMySQL, manually on all nodes of a cluster because the package can't be found. | AE 1.1 clusters created before June 2019 | Navigate to **Ambari UI > Spark2 > Configs > Custom spark2-defaults > Add Property ** and enter the following 2 lines: <br/> <br/> `spark.yarn.appMasterEnv.PYSPARK3_PYTHON=/home/common/conda/anaconda3/bin/python` <br/> <br/> `spark.yarn.appMasterEnv.PYSPARK_PYTHON=/home/common/conda/anaconda2/bin/python` <br/> <br/> Restart the Spark service when prompted. This command forces the use of Anaconda Python instead of System Python. |
42+
| Adding additional libraries | Livy or IBM Cloud CLI commands fail after installing additional packages, for example PyMySQL, manually on all nodes of a cluster because the package can't be found. | AE 1.1 clusters created before June 2019 | Navigate to **Ambari UI > Spark2 > Configs > Custom spark2-defaults > Add Property** and enter the following 2 lines: <br/> <br/> `spark.yarn.appMasterEnv.PYSPARK3_PYTHON=/home/common/conda/anaconda3/bin/python` <br/> <br/> `spark.yarn.appMasterEnv.PYSPARK_PYTHON=/home/common/conda/anaconda2/bin/python` <br/> <br/> Restart the Spark service when prompted. This command forces the use of Anaconda Python instead of System Python. |
4343
| | If you install a newer version of a Python package than is already on the cluster and try to use it, the following error is displayed: `ImportError: No module named.` | AE 1.1 clusters created before June 2019 |Force the path to take the latest version by entering the following in your Python script: <br/> For Anaconda3: `import sys; sys.path.insert(0, '/home/wce/clsadmin/pipAnaconda3Packages')` <br/> <br/> For Anaconda2: `import sys; sys.path.insert(0, '/home/wce/clsadmin/pipAnaconda2Packages')` |
44+
| Disk space issues | HDFS audit logs are not rotated and so fill up the disk space, which disrupts the normal functioning of a cluster. | AE 1.2 clusters created before 10 September 2020 | Navigate to **Ambari UI > HDFS > Config > Advanced** and search in the filter for `hdfs-log4j`. At the bottom of the text box, add the following two lines that set the maximum log file size and the backup frequency to 30 days: <br/><br/>`log4j.appender.DRFAAUDIT.MaxFileSize={{hadoop_log_max_backup_size}}MB`<br/><br/>`log4j.appender.DRFAAUDIT.MaxBackupIndex={{hadoop_log_number_of_backup_files}}`|

‎aggregating-logs.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
22

33
copyright:
4-
years: 2017, 2019
5-
lastupdated: "2019-10-08"
4+
years: 2017, 2020
5+
lastupdated: "2020-08-30"
66

77
subcollection: AnalyticsEngine
88

@@ -112,15 +112,20 @@ You can use the following component names:
112112
- `hadoop-mapreduce`
113113
- `hadoop-yarn`
114114
- `hbase`
115+
- `hdfs`
116+
- `hdfs-audit`
115117
- `hive`
116118
- `jnbg`
117119
- `knox`
120+
- `knox-audit`
118121
- `livy2`
119122
- `spark2`
120123

121124
**Data node components**
122125
- `hadoop-mapreduce`
123126
- `hadoop-yarn`
127+
- `hdfs`
128+
- `hdfs-audit`
124129
- `spark2`
125130
- `yarn-apps`
126131

‎data-skipping.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22

33
copyright:
4-
years: 2017, 20120
4+
years: 2017, 2020
55
lastupdated: "2020-06-23"
66

77
subcollection: AnalyticsEngine

‎lazy-timeseries-evaluation.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
3+
copyright:
4+
years: 2017, 2020
5+
lastupdated: "2020-08-30"
6+
7+
subcollection: AnalyticsEngine
8+
9+
---
10+
11+
<!-- Attribute definitions -->
12+
{:new_window: target="_blank"}
13+
{:shortdesc: .shortdesc}
14+
{:codeblock: .codeblock}
15+
{:screen: .screen}
16+
{:pre: .pre}
17+
18+
# Lazy evaluation
19+
{: #lazy-evaluation}
20+
21+
Lazy evaluation is an evaluation strategy that delays the evaluation of an expression until its value is needed. When combined with memoization, lazy evaluation strategy avoids repeated evaluations and can reduce the running time of certain functions by a significant factor.
22+
23+
The time series library uses lazy evaluation to process data. Notionally an execution graph is constructed on time series data whose evaluation is triggered only when its output is materialized. Assuming an object is moving in a one dimensional space, whose location is captured by x(t). You can determine the harsh acceleration/braking (`h(t)`) of this object by using its velocity (`v(t)`) and acceleration (`a(t)`) time series as follows:
24+
25+
```python
26+
# 1d location timeseries
27+
x(t) = input location timeseries
28+
29+
# velocity - first derivative of x(t)
30+
v(t) = x(t) - x(t-1)
31+
32+
# acceleration - second derivative of x(t)
33+
a(t) = v(t) - v(t-1)
34+
35+
# harsh acceleration/braking using thresholds on acceleration
36+
h(t) = +1 if a(t) > threshold_acceleration
37+
= -1 if a(t) < threshold_deceleration
38+
= 0 otherwise
39+
```
40+
41+
This results in a simple execution graph of the form:
42+
43+
```
44+
x(t) --> v(t) --> a(t) --> h(t)
45+
```
46+
47+
Evaluations are triggered only when an action is performed, such as `compute h(5...10)`, i.e. `compute h(5), ..., h(10)`. The library captures narrow temporal dependencies between time series. In this example, `h(5...10)` requires `a(5...10)`, which in turn requires `v(4...10)`, which then requires `x(3...10)`. Only the relevant portions of `a(t)`, `v(t)` and `x(t)` are evaluated.
48+
49+
```
50+
h(5...10) <-- a(5...10) <-- v(4...10) <-- x(3...10)
51+
```
52+
Furthermore, evaluations are memoized and can thus be reused in subsequent actions on `h`. For example, when a request for `h(7...12)` follows a request for `h(5...10)`, the memoized values `h(7...10)` would be leveraged; further, `h(11...12)` would be evaluated using `a(11...12), v(10...12)` and `x(9...12)`, which would in turn leverage `v(10)` and `x(9...10)` memoized from the prior computation.
53+
54+
In a more general example, you could define a smoothened velocity timeseries as follows:
55+
56+
```python
57+
# 1d location timeseries
58+
x(t) = input location timeseries
59+
60+
# velocity - first derivative of x(t)
61+
v(t) = x(t) - x(t-1)
62+
63+
# smoothened velocity
64+
# alpha is the smoothing factor
65+
# n is a smoothing history
66+
v_smooth(t) = (v(t)*1.0 + v(t-1)*alpha + ... + v(t-n)*alpha^n) / (1 + alpha + ... + alpha^n)
67+
68+
# acceleration - second derivative of x(t)
69+
a(t) = v_smooth(t) - v_smooth(t-1)
70+
```
71+
72+
In this example `h(l...u)` has the following temporal dependency. Evaluation of `h(l...u)` would strictly adhere to this temporal dependency with memoization.
73+
74+
```
75+
h(l...u) <-- a(l...u) <-- v_smooth(l-1...u) <-- v(l-n-1...u) <-- x(l-n-2...u)
76+
```
77+
78+
## An Example
79+
{: #time-series-example}
80+
81+
The following example shows a python code snippet that implements harsh acceleration on a simple in-memory time series. The library includes several built-in transforms. In this example the difference transform is applied twice to the location time series to compute acceleration time series. A map operation is applied to the acceleration time series using a harsh lambda function, which is defined after the code sample, that maps acceleration to either `+1` (harsh acceleration), `-1` (harsh braking) and `0` (otherwise). The filter operation selects only instances wherein either harsh acceleration or harsh braking is observed. Prior to calling `get_values`, an execution graph is created, but no computations are performed. On calling `get_values(5, 10)`, the evaluation is performed with memoization on the narrowest possible temporal dependency in the execution graph.
82+
83+
```python
84+
import tspy
85+
from tspy.builders.functions import transformers
86+
87+
x = tspy.time_series.list([1.0, 2.0, 4.0, 7.0, 11.0, 16.0, 22.0, 29.0, 28.0, 30.0, 29.0, 30.0, 30.0])
88+
v = x.transform(transformers.difference())
89+
a = v.transform(transformers.difference())
90+
h = a.map(harsh).filter(lambda h: h != 0)
91+
92+
print(h[5, 10])
93+
```
94+
95+
The harsh lambda is defined as follows:
96+
97+
```python
98+
def harsh(a):
99+
threshold_acceleration = 2.0
100+
threshold_braking = -2.0
101+
102+
if (a > threshold_acceleration):
103+
return +1
104+
elif (a < threshold_braking):
105+
return -1
106+
else:
107+
return 0
108+
```

‎parquet-encryption.md‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
copyright:
44
years: 2017, 2020
5-
lastupdated: "2020-07-02"
5+
lastupdated: "2020-09-10"
66

77
subcollection: AnalyticsEngine
88

@@ -63,14 +63,13 @@ Key features include:
6363
To enable Parquet encryption in {{site.data.keyword.iae_full_notm}}, set the following Spark classpath properties to point to the Parquet jar files that implement Parquet modular encryption, and to the key management jar file:
6464
6565
1. Navigate to **Ambari > Spark > Config -> Custom spark2-default**.
66-
1. Add the following two parameters to point explicitly to the location of the JAR files. Make sure that you edit the paths to use the actual version of jar files on the cluster.
66+
1. Add the following two parameters to point explicitly to the location of the JAR files.
6767
6868
Alternatively, you can get the JAR files applied as part of the cluster creation process. See [Advanced Provisioning](/docs/AnalyticsEngine?topic=AnalyticsEngine-advanced-provisioning-options){: external}.
6969
7070
```
71-
spark.driver.extraClassPath=/home/common/lib/parquetEncryption/ibm-parquet-kms-<latestversion>-jar-with-dependencies.jar:/home/common/lib/parquetEncryption/parquet-format-<latestversion>.jar:/home/common/lib/parquetEncryption/parquet-hadoop-<latestversion>.jar
72-
73-
spark.executor.extraClassPath=/home/common/lib/parquetEncryption/ibm-parquet-<latestversion>-jar-with-dependencies.jar:/home/common/lib/parquetEncryption/parquet-format-<latestversion>.jar:/home/common/lib/parquetEncryption/parquet-hadoop-<latestversion>.jar
71+
spark.driver.extraClassPath=/home/common/lib/parquetEncryption/ibmparquetkms.jar:/home/common/lib/parquetEncryption/parquetformat.jar:/home/common/lib/parquetEncryption/parquethadoop.jar
72+
spark.executor.extraClassPath=/home/common/lib/parquetEncryption/ibmparquetkms.jar:/home/common/lib/parquetEncryption/parquetformat.jar:/home/common/lib/parquetEncryption/parquethadoop.jar
7473
```
7574
7675
## Mandatory parameters

‎release-notes.md‎

Lines changed: 51 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
copyright:
44
years: 2017, 2020
5-
lastupdated: "2020-07-23"
5+
lastupdated: "2020-09-10"
66

77
subcollection: AnalyticsEngine
88

@@ -23,6 +23,56 @@ Use these notes to learn about the latest features, additions and changes to {{s
2323
{: shortdesc}
2424
## {{site.data.keyword.iae_full_notm}} information
2525

26+
### 10 September 2020
27+
28+
- **[AE-1.2.v28.4]** The following security patch was applied to the underlying VMs for IBM Java CVE: [CVE-2019-17639](https://www.ibm.com/blogs/psirt/security-bulletin-multiple-vulnerabilities-may-affect-ibm-sdk-java-technology-edition-3/)
29+
- You can now configure log aggregation for the HDFS component. See [Configuring log aggregation](/docs/AnalyticsEngine?topic=AnalyticsEngine-log-aggregation).
30+
- A fix was added that prevents HDFS audit logs from filling up disk space, which was caused by a misconfiguration of the log4j rotation property that disrupted the way clusters should work.
31+
- You can now use the time series library in your Spark applications, which provides a rich time series data model and imputation functions for transforming, reducing, segmenting, joining, and forecasting time series. SQL extensions to time series are also provided. See [Time series library](/docs/AnalyticsEngine?topic=AnalyticsEngine-time-series).
32+
33+
### 20 August 2020
34+
35+
- As of this deployment, Analytics Engine cluster’s build version will be available in `/home/common/aeversion.txt` on the nodes of the cluster. You can check this file after you SSH to the cluster. This will help in tracking fixes that were made available against a particular version of Analytics Engine. For example, this deployment version is AE-1.2.v28.3.
36+
37+
- **[AE-1.2.v28.3]** Fixes were applied for the following security vulnerabilities on the OS level packages of cluster nodes. Apart from these fixes, a few config vulnerabilities at the OS level were also applied.
38+
39+
[CVE-2019-11729](https://access.redhat.com/errata/RHSA-2019:4190),
40+
[CVE-2019-11745](https://access.redhat.com/errata/RHSA-2019:4190),
41+
[CVE-2020-8616](https://access.redhat.com/errata/RHSA-2020:2344),
42+
[CVE-2020-8617](https://access.redhat.com/errata/RHSA-2020:2344),
43+
[CVE-2019-12735](https://access.redhat.com/errata/RHSA-2019:1619),
44+
[CVE-2020-11008](https://access.redhat.com/errata/RHSA-2020:2337),
45+
[CVE-2020-12049](https://access.redhat.com/errata/RHSA-2020:2894),
46+
[CVE-2020-1967](https://gitlab.alpinelinux.org/alpine/aports/-/issues/11429),
47+
[CVE-2020-3810](https://ubuntu.com/security/notices/USN-4359-1),
48+
[CVE-2019-5188](http://www.ubuntu.com/usn/usn-4249-1),
49+
[CVE-2019-5094](http://www.ubuntu.com/usn/usn-4142-1),
50+
[usn-4038-3](http://www.ubuntu.com/usn/usn-4038-3),
51+
[CVE-2017-12133](http://www.ubuntu.com/usn/usn-4416-1),
52+
[CVE-2017-18269](http://www.ubuntu.com/usn/usn-4416-1),
53+
[CVE-2018-11236](http://www.ubuntu.com/usn/usn-4416-1),
54+
[CVE-2018-11237](http://www.ubuntu.com/usn/usn-4416-1),
55+
[CVE-2018-19591](http://www.ubuntu.com/usn/usn-4416-1),
56+
[CVE-2018-6485](http://www.ubuntu.com/usn/usn-4416-1),
57+
[CVE-2019-19126](http://www.ubuntu.com/usn/usn-4416-1),
58+
[CVE-2019-9169](http://www.ubuntu.com/usn/usn-4416-1),
59+
[CVE-2020-10029](http://www.ubuntu.com/usn/usn-4416-1),
60+
[CVE-2020-1751](http://www.ubuntu.com/usn/usn-4416-1),
61+
[CVE-2020-1752](http://www.ubuntu.com/usn/usn-4416-1),
62+
[CVE-2019-13627](http://www.ubuntu.com/usn/usn-4236-2),
63+
[CVE-2018-16888](http://www.ubuntu.com/usn/usn-4269-1),
64+
[CVE-2019-20386](http://www.ubuntu.com/usn/usn-4269-1),
65+
[CVE-2019-3843](http://www.ubuntu.com/usn/usn-4269-1),
66+
[CVE-2019-3844](http://www.ubuntu.com/usn/usn-4269-1),
67+
[CVE-2020-1712](http://www.ubuntu.com/usn/usn-4269-1),
68+
[CVE-2016-9840](http://www.ubuntu.com/usn/usn-4246-1),
69+
[CVE-2016-9841](http://www.ubuntu.com/usn/usn-4246-1),
70+
[CVE-2016-9842](http://www.ubuntu.com/usn/usn-4246-1),
71+
[CVE-2016-9843](http://www.ubuntu.com/usn/usn-4246-1),
72+
[CVE-2019-9924](http://www.ubuntu.com/usn/usn-4058-1)
73+
74+
- There is no security patch application for existing clusters. Create a new cluster to get these fixes.
75+
2676
### 23 July 2020
2777

2878
- Default values for the `gateway.socket.*` parameters were added to fix errors related to the length of messages when using Spark WebSockets.

0 commit comments

Comments
 (0)