1. Eskimo Introduction

A state of the art Big Data Infrastructure and Management Web Console to build, manage and operate Big Data 2.0 Analytics clusters

Eskimo is in a certain way the Operating System of your Big Data Cluster:

• A plug and play, working out of the Box, Big Data Analytics platform fulfilling enterprise environment requirements.

• A state of the art Big Data 2.0 platform

  • based on Docker, Marathon, Mesos and Systemd

  • packaging Gluster, Spark, Kafka, Flink, Nifi and ElasticSearch

  • with all the administration and management consoles such as Cerebro, Kibana, Zeppelin, Kafka-Manager, Grafana and Prometheus.

• An Administration Application aimed at drastically simplifying the deployment, administration and operation of your Big Data Cluster

• A Data Science Laboratory and Production environment where Data Analytics is both

  • developed and

  • operated in production

Eskimo is as well:

• a collection of ready to use docker containers packaging fine-tuned and highly customized plug and play services with all the nuts and bolds required to make them work well together.

• a framework for building and deploying Big Data and NoSQL services based on docker and systemd

Eskimo Packages all the componentds above and takes care of all the nuts and bolts required to make them work together smoothly and without the user or administrator needing to take care of anything.

1.1. Eskimo CE Project Development Status

Travis CI Integration Build Status

2. Information

Reach http://www.eskimo.sh for more information on Eskimo or look at the documentation in the folder doc.

3. Building eskimo

3.1. Requirements

Eskimo uses plain old Apache Maven for building (https://maven.apache.org/) as well as JDK 8+ (https://openjdk.java.net/) for both building and execution.

Minimum requirements are as follows:

• JDK 8 or greater

• Apache maven 3.5 or greater.

Every other dependency will be downloaded during the maven build process.

Theoretically, Eskimo can be built on any Operating System providing a command line and supporting Java 8+ (Java 9 recommended) and Maven 3+. However, the eskimo source tree makes an extensive use of symbolic links wich - unfortunately - are not supported on Microsoft Windows.
For this reason, building eskimo on windows is absolutely possible but it is highly discouraged to develop eskimo on Windows.

3.2. Building using maven

In order to build eskimo, simply run the following command in this very folder (the one containing this readme.adoc file):

Build Eskimo

mvn clean install

As a result of the build process, Eskimo is available as an extractable archive in eskimo-version-bin.zip or eskimo-version-bin.tar.gz in the folder target.

3.3. Specific hints and troubleshooting information regarding eskimo building

You might want to read carefully the following additional information related to building eskimo.

3.3.1. Put maven and Java in PATH

Of course, for the above command to work, you need to have java and mvn in your path.

Use for instance the following commands on Linux:

Put maven and java in PATH on Linux

export JAVA_HOME=/usr/local/lib/jdk-9
export MAVEN_HOME=/usr/local/lib/apache-maven-3.5.3
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

(You might want to put above commands in your /etc/profile or /etc/bash.bashrc)

And for instance the following commands on Windows

Put maven and java in PATH on Windows

set JAVA_HOME=C:\programs\jdk-9
set MAVEN_HOME=C:\programs\apache-maven-3.6.1
set PATH=%MAVEN_HOME%\bin;%JAVA_HOME%\bin;%PATH%

3.3.2. Install maven dependency trilead-ssh2

Eskimo used the library trilead-ssh2 version build-217-jenkins-16 which is not available in standard maven repositories.
This library is bundled with the eskimo sources in the folder lib.
One can installl it in his own local maven repository using the following command:

Install trilead-ssh2 library

cd libs
bash install_libs.sh

3.3.3. OpenJDK 9 empty cacerts

The Open JDK 9 comes with a quite annoying issue. The lib/security/cacert file referencing the SSL certifications authorities is empty.
This prevents most of the time both maven from successfully accessing maven repositories as well as JRuby from working flawlessly.

Symptoms of this problem are as follows
Either the maven build fails at the time of fetching dependencies:

Maven fails to fetch dependencies

Z:\data\\eskimo>mvn clean install
[INFO] Scanning for projects...
[INFO]
[INFO] --------------< org.springframework.boot:eskimo-console >---------------
[INFO] Building Eskimo Console 0.1-SNAPSHOT
[INFO] --------------------------------[ war ]---------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/3.1.0/maven-assembly-plugin-3.1.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.335 s
[INFO] Finished at: 2019-07-14T16:19:54+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-assembly-plugin:3.1.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-assembly-plugin:jar:3.1.0: Could not transfer artifact org.apache.maven.plugins:maven-assembly-plugin:pom:3.1.0 from/to central (https://repo.maven.apache.org/maven2): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

Or asciidoctor fails within JRuby at the time of generating the documentation:

asciidoctor fails on Jruby

badtrash@badbooknew:/data//eskimo$ mvn clean install
[INFO] Scanning for projects...
[INFO]
[INFO] --------------< org.springframework.boot:eskimo-console >---------------
[INFO] Building Eskimo Console 0.1-SNAPSHOT
[INFO] --------------------------------[ war ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ eskimo-console ---
[INFO] Deleting /data/eskimo/target
[INFO]
[INFO] --- asciidoctor-maven-plugin:1.6.0:process-asciidoc (service-dev-guide-pdf) @ eskimo-console ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 40.533 s
[INFO] Finished at: 2019-07-14T17:20:57+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.asciidoctor:asciidoctor-maven-plugin:1.6.0:process-asciidoc (service-dev-guide-pdf) on project eskimo-console: Execution service-dev-guide-pdf of goal org.asciidoctor:asciidoctor-maven-plugin:1.6.0:process-asciidoc failed: (NameError) cannot load (ext) (org.jruby.ext.openssl.OpenSSL): InvocationTargetException: (OpenSSL::X509::StoreError) setting default path failed: the trustAnchors parameter must be non-empty -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Happily the fix is quite straightforward, just copy a valid cacert file to your jdk-9/lib/security folder.

For instance on debian or ubuntu, you can generate a valid cacert file for the system bundled java and then copy this valid cacert file to your jdk-9.

generate cacert and copy it to jdk-9

badtrash@badbooknew:/data//eskimo$ sudo update-ca-certificates -f
...
badtrash@badbooknew:/data//eskimo$ cd /usr/local/lib/jdk-9/lib/security/
badtrash@badbooknew:/usr/local/lib/jdk-9/lib/security$ sudo rm cacerts
badtrash@badbooknew:/usr/local/lib/jdk-9/lib/security$ sudo ln -s /etc/ssl/certs/java/cacerts

And the problem shall be solved.

On windows, you need to find a valid cacert file and copy it over to your jdk-9/lib/security folder.
A sample (generated on debian) cacert file is provided with eskimo sources in the folder test_lab

Using custom cacert file fix for maven

If for any reason you cannot change the cacert file of your JDK, you can still manage to find a way to have maven working by following the following procedure, for instance for Windows:

1. Use a browser (I used IE) to go to https://repo.maven.apache.org/

   • Click on lock icon and choose "View Certificate"

   • Go to the "Details" tab and choose "Save to File"

   • Choose type "Base 64 X.509 (.CER)" and save it somewhere, e.g in C:\data\maven_cert.cer

2. Create a trusted keystore

   • keytool -import -file C:\data\maven_cert.cer -keystore c:\data\mavenKeystore

   • (give "changeit" as password)

3. One can now fetch dependencies from maven successfully by using:

   • mvn clean install -Djavax.net.ssl.trustStore=C:\data\mavenKeystore -Djavax.net.ssl.trustStorePassword=changeit

Unfortunately that wouldn’t work for adressing JRuby’s issue when generating the eskimo documentation and you would need to comment the asciidoctor part out of the maven build.

3.3.4. Building HTML documentation

Optionnaly, the HTML documentation of eskimo (in addition to the PDF documentation which is always built) can be built using the htmlDoc maven profile:

Build Eskimo with HTML Documentation

mvn clean install -PhtmlDoc

3.4. Eskimo Build result

The build results at the end of the maven build process is located in the folder `target.
The results are two archive files:

• A zip archive : eskimo-version-bin.zip

• A tarball archive : eskimo-version-bin.tar.gz

You can find pre-built packages of Eskimo on https://www.eskimo.sh.

4. Running Eskimo

This section gives instructions about running eskimo.

Eskimo itself can run on any Operating System supporting a JVM (Linux, Windows, Mac OSX) but cluster nodes are only supported running Linux.

4.1. Prerequisites

4.1.1. System requirements

In order to run eskimo, one needs to have

• At least 8Gb of disk storage space on the machine running Eskimo

• At least one linux machine available on the network (can be the same machine than the one running Eskimo) with either a debian-based (Ubuntu, Debian, etc.), red-hat-based (CentOS, Fedora, etc.) or OpenSUSE Operating System.

4.1.2. Available JVM with Java in PATH

In order to run eskimo, one needs to have java in the path.

Use for instance the following commands on Linux:

Put java in PATH on Linux

export JAVA_HOME=/usr/local/lib/jdk-9
export PATH=$JAVA_HOME/bin:$PATH

(You might want to put above commands in your /etc/profile or /etc/bash.bashrc)

And for instance the following commands on Windows

Put java in PATH on Windows

set JAVA_HOME=C:\programs\jdk-9
set PATH=%JAVA_HOME%\bin;%PATH%

4.2. Extract archive.

After building eskimo using maven, the zip and tarball archives are located in the folder target.

One of these archives needs to be extracted on the local filesystem.

Then in the folder bin under the newly extracted eskimo binary distribution folder, one can find two scripts:

• a script eskimo.bat to execute eskimo on Windows

• a script eskimo.sh to execute eskimo on Linux.

4.3. Access eskimo

With eskimo properly started using the above scripts, one can reach eskimo using http://machine_ip:9090.
The default port number is 9090. This can be changed in configuration file eskimo.properties.

The default login / password credentials are admin / password.

4.4. Typical startup issues

Several issues can happen upon first eskimo startup.
This section describes common issues and ways to resolved them.

4.4.1. eskimo-users.json cannot be written

If you meet an error as the following on startup:

Impossible to write eskimo-users.json

Caused by: ch.niceideas.common.utils.FileException: ./eskimo-users.json (Unauthorized access)
        at ch.niceideas.common.utils.FileUtils.writeFile(FileUtils.java:154)
        at ch.niceideas.eskimo.security.JSONBackedUserDetailsManager.<init>(JSONBackedUserDetailsManager.java:81)
        at ch.niceideas.eskimo.configurations.WebSecurityConfiguration.userDetailsService(WebSecurityConfiguration.java:127)
        ... 50 more
Caused by: java.io.FileNotFoundException: ./eskimo-users.json (Unauthorized access)
        at java.base/java.io.FileOutputStream.open0(Native Method)
        at java.base/java.io.FileOutputStream.open(FileOutputStream.java:276)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:220)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:170)
        at java.base/java.io.FileWriter.<init>(FileWriter.java:90)
        at ch.niceideas.common.utils.FileUtils.writeFile(FileUtils.java:149)
        ... 52 more

Eskimo uses a local file to define users and access credentials. Upon first startup, if that file doesn’t exist already, it is created by eskimo (with the default credentials above) at the path pointed to by the property security.userJsonFile in eskimo.properties.

If you experience the error above or something alike, change that propery to point to a location where the first version of the file can successfully be created.

5. Eskimo source distribution layout

The directory structure of the eskimo source distribution is as follows:

• doc contains the source documentation in asciidoc format

• libs contains dependencies not available in standard maven repositories

• packages_dev contains the docker images development framework and packages

• packages_distrib is the destination folder in which downloaded or built images are placed

• services_setup contains the services installation framework and packages

• src contains the source files to build eskimo

• test_lab contains various tools to build VMs aimed at testing eskimo

6. Test Laboratory

The folder test_lab folder in the Eskimo CE project root folder contains a Vagrant framework used to create virtual machines to test Eskimo.

The supported virtual machines are declared in the file VagrantFile in the variable nodes:

Vagrant Virtual Machines declaration

# Define cluster nodes
nodes = [
  { :hostname => 'deb-node1',  :box => 'debian/buster64',
                               :ip => '192.168.10.11', :ram => 8000 },
  { :hostname => 'deb-node2',  :box => 'ubuntu/xenial64',
                               :ip => '192.168.10.12', :ram => 6000 },
  { :hostname => 'cent-node1', :box => 'centos/7',
                               :ip => '192.168.10.13', :ram => 6000},
  { :hostname => 'cent-node2', :box => 'fedora/29-cloud-base',
                               :ip => '192.168.10.14', :ram => 6000 }
]

The defined VMs use different Operating Systems to test Eskimo’s installation on different OSes.
One can edit this variable to declare one’s own VM’s as required.

An individual VM is then started with vargrant using VirtualBox as follows:

Start deb-node1 with VirtualBox

$ cd ./test_lab/vagrant/
$ vagrant up deb-node1

Vagrant can also start VM’s using libvirt / QEMU / kvm if the required libraries and provider are properly installed:

Start deb-node1 with libvirt

$ cd ./test_lab/vagrant/
$ vagrant up deb-node1 --provider=libvirt

7. Further informations

Look at the following files for more information

• The Eskimo User Guide

• readme.adoc in the folder packages_dev for information about the docker packages building framework

• readme.adoc in the folder services_setup for information about the services installation and operation framework.

Appendix A: Copyright and License

Eskimo is Copyright 2019 eskimo.sh - All rights reserved.
Author : http://www.eskimo.sh

Eskimo is available under a dual licensing model : commercial and GNU AGPL.
If you did not acquire a commercial licence for Eskimo, you can still use it and consider it free software under the terms of the GNU Affero Public License. You can redistribute it and/or modify it under the terms of the GNU Affero Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
Compliance to each and every aspect of the GNU Affero Public License is mandatory for users who did no acquire a commercial license.

Eskimo is distributed as a free software under GNU AGPL in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero Public License for more details.

You should have received a copy of the GNU Affero Public License along with Eskimo. If not, see https://www.gnu.org/licenses/ or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA, 02110-1301 USA.

You can be released from the requirements of the license by purchasing a commercial license. Buying such a commercial license is mandatory as soon as :

• you develop activities involving Eskimo without disclosing the source code of your own product, software, platform, use cases or scripts.

• you deploy eskimo as part of a commercial product, platform or software.

For more information, please contact eskimo.sh at https://www.eskimo.sh

The above copyright notice and this licensing notice shall be included in all copies or substantial portions of the Software.