BIGTOP-3959: RPM packaging for TRINO

bigtop-packages/src/common/trino/developers.rst

@@ -0,0 +1,28 @@
+For Developers: Building a component from Git repository
+
+Prerequisites
+
+You will need git installed.
+You will need java 17 installed for Trino
+Java 8 for bigtop
+You will need to use gradlew which is included in the source code. (Right in the root of the project folder)
+This project's gradlew has more documentation here
+Use git to download BigTop :
+git clone https://github.com/apache/bigtop.git
+
+move into the root project folder:
+cd bigtop
+
+To fetch source from a Git repository, there're two ways to achieve this: a). modify ./bigtop.bom and add JSON snippets to your component/package, or b). specify properties at command line
+
+bigtop.bom
+Add following JSON snippets to the desired component/package:
+
+git     { repo = ""; ref = ""; dir = ""; commit_hash = "" }
+repo - SSH, HTTP or local path to Git repo.
+ref - branch, tag or commit hash to check out.
+dir - [OPTIONAL] directory name to write source into.
+commit_hash - [OPTIONAL] a commit hash to reset to.
+Some packages have different names for source directory and source tarball (hbase-0.98.5-src.tar.gz contains hbase-0.98.5 directory). By default source will be fetched in a directory named by tarball { source = TARBALL_SRC } without .t* extension. To explicitly set directory name use the dir option.
+
+When commit_hash specified, the repo to build the package will be reset to the commit hash.

bigtop-packages/src/common/trino/do-component-build

@@ -0,0 +1,41 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -ex
+
+
+. `dirname $0`/bigtop.bom
+mkdir trino
+# Apply the Puppet manifest to download and extract JDK
+sudo puppet apply jdk17_download.pp
+# Set JAVA_HOME for this session to the JDK downloaded by the Puppet manifest
+export JAVA_HOME=/opt/jdk-17.0.2
+echo JAVA_HOME=`pwd`/`ls | grep "jdk-*"` >> ~/.bash_profile
+source ~/.bash_profile
+
+# Setting versions for trino build
+mvn clean -pl !plugin/trino-kafka,!testing/trino-benchto-benchmarks,!docs install -DskipTests -Dcheckstyle.skipExec -Dmaven.wagon.http.ssl.allowall=true
+sed -i '$ d' ~/.bash_profile
+source ~/.bash_profile
+rm -rf `pwd`/`ls | grep "jdk-*"`
+
+mkdir -p build/trino
+mkdir -p build/trino-cli
+
+# Server
+tar -C build/trino --strip-components=1 -xzf core/trino-server/target/*.tar.gz
+# CLI
+cp -ra client/trino-cli/target/*-executable.jar build/trino-cli/trino

bigtop-packages/src/common/trino/getting-started.rst

@@ -0,0 +1,20 @@
+Getting started with bigtop packaging.
+
+1) You will have to package rpm.
+
+2) Here are some steps you can follow to bring a new bigtop package in.
+
+determine where your source is, and add it to bigtop.bom
+update the bigtop-packages/src/common/<your-package> folder to have your component,
+and the do-component-build for it (which usually just builds a jar).
+Why is there a "common" directory? Simply because deb and rpm packaging share some tasks (like do-component-build,
+which just usually runs a mvn or gradle command), and so we keep a common install directory which they can both leverage for packaging.
+(for RPM) now add a .spec file into bigtop-packages/src/<your-package>/... into the appropriate directory
+(i.e. bigtop-packages/src/rpm/tachyon/SPECS/tachyon.spec).
+Obviously, your tachyon.spec file will use whats in common/ in a RPM specific way, to install the RPM package.
+create a rules file using do-component-build
+
+Test it with gradle <your-package>-rpm for the others.
+Finally add a smoke test! This is as easy as adding a new groovy file to bitop-tests/smoke-tests/<your-package>/TestThisStuff.groovy,
+following conventions that others have created.
+3) As always, we will improve on the directions above, but this should help to get you started. .

bigtop-packages/src/common/trino/install_trino.sh

@@ -0,0 +1,193 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+usage() {
+  echo "
+usage: $0 <options>
+  Required not-so-options:
+     --build-dir=DIR             path to dist.dir
+     --source-dir=DIR            path to package shared files dir
+     --prefix=PREFIX             path to install into
+  Optional options:
+     --doc-dir=DIR               path to install docs into [/usr/share/doc/trino]
+     --lib-dir=DIR               path to install trino home [/usr/lib/trino]
+     --installed-lib-dir=DIR     path where lib-dir will end up on target system
+     --bin-dir=DIR               path to install bins [/usr/bin]
+     ... [ see source for more similar options ]
+  "
+  exit 1
+}
+
+OPTS=$(getopt \
+  -n $0 \
+  -o '' \
+  -l 'prefix:' \
+  -l 'lib-dir:' \
+  -l 'installed-lib-dir:' \
+  -l 'bin-dir:' \
+  -l 'trino-version:' \
+  -l 'source-dir:' \
+  -l 'cli-dir:' \
+  -l 'build-dir:' -- "$@")
+
+if [ $? != 0 ] ; then
+    usage
+fi
+
+eval set -- "$OPTS"
+while true ; do
+    case "$1" in
+        --prefix)
+        PREFIX=$2 ; shift 2
+        ;;
+        --build-dir)
+        BUILD_DIR=$2 ; shift 2
+        ;;
+        --source-dir)
+        SOURCE_DIR=$2 ; shift 2
+        ;;
+        --cli-dir)
+        CLI_BUILD_DIR=$2 ; shift 2
+        ;;
+        --lib-dir)
+        LIB_DIR=$2 ; shift 2
+        ;;
+        --installed-lib-dir)
+        INSTALLED_LIB_DIR=$2 ; shift 2
+        ;;
+        --bin-dir)
+        BIN_DIR=$2 ; shift 2
+        ;;
+        --)
+        shift ; break
+        ;;
+        *)
+        echo "Unknown option: $1"
+        usage
+        exit 1
+        ;;
+    esac
+done
+
+for var in PREFIX BUILD_DIR SOURCE_DIR ; do
+  if [ -z "$(eval "echo \$$var")" ]; then
+    echo Missing param: $var
+    usage
+  fi
+done
+
+if [ -f "$SOURCE_DIR/bigtop.bom" ]; then
+  . $SOURCE_DIR/bigtop.bom
+fi
+
+MAN_DIR=${MAN_DIR:-/usr/share/man}/man1
+DOC_DIR=${DOC_DIR:-/usr/share/doc/trino}
+CLI_DIR=${CLI_DIR:-/usr/lib/trino-cli}
+LIB_DIR=${TRINO_DIR:-/usr/lib/trino}
+VAR_DIR=${VAR_DIR:-/var/lib/trino}
+LOG_DIR=${LOG_DIR:-/var/log/trino}
+RUN_DIR=${RUN_DIR:-/var/run/trino}
+INSTALLED_LIB_DIR=${INSTALLED_LIB_DIR:-/usr/lib/trino}
+BIN_DIR=${BIN_DIR:-/usr/bin}
+CONF_DIR=${CONF_DIR:-/etc/trino}
+CONF_DIST_DIR=${CONF_DIST_DIR:-/etc/trino.dist}
+DEFAULT_DIR=${DEFAULT_DIR:-/etc/default}
+
+install -d -m 0755 $PREFIX/$CONF_DIST_DIR
+install -d -m 0755 $PREFIX/$LIB_DIR
+install -d -m 0755 $PREFIX/$CLI_DIR
+install -d -m 0755 $PREFIX/$DOC_DIR
+install -d -m 0755 $PREFIX/$VAR_DIR
+install -d -m 0755 $PREFIX/$LOG_DIR
+install -d -m 0755 $PREFIX/$RUN_DIR
+install -d -m 0755 $PREFIX/$DEFAULT_DIR
+
+cp -ra ${BUILD_DIR}/* $PREFIX/$LIB_DIR/
+cp -ra ${CLI_BUILD_DIR}/* $PREFIX/$CLI_DIR/
+
+chmod +x $PREFIX/$LIB_DIR/bin/launcher
+
+install -d -m 0755 $PREFIX/$CONF_DIST_DIR
+install -d -m 0755 $PREFIX/$CONF_DIST_DIR/catalog
+
+cat > $PREFIX/$CONF_DIST_DIR/node.properties <<EOF
+node.environment=production
+node.id=ffffffff-ffff-ffff-ffff-ffffffffffff
+node.data-dir=/var/trino/data
+EOF
+
+cat > $PREFIX/$CONF_DIST_DIR/jvm.config <<EOF
+-server
+-Xmx16G
+-XX:G1HeapRegionSize=32M
+-XX:+ExplicitGCInvokesConcurrent
+-XX:+ExitOnOutOfMemoryError
+-XX:+HeapDumpOnOutOfMemoryError
+-XX:-OmitStackTraceInFastThrow
+-XX:ReservedCodeCacheSize=512M
+-XX:PerMethodRecompilationCutoff=10000
+-XX:PerBytecodeRecompilationCutoff=10000
+-Djdk.attach.allowAttachSelf=true
+-Djdk.nio.maxCachedBufferSize=2000000
+-XX:+UnlockDiagnosticVMOptions
+EOF
+
+cat > $PREFIX/$CONF_DIST_DIR/config.properties <<EOF
+# A single machine for testing that will function as both a coordinator and worker
+coordinator=true
+node-scheduler.include-coordinator=true
+http-server.http.port=8080
+query.max-memory=4GB
+query.max-memory-per-node=1GB
+discovery-server.enabled=true
+discovery.uri=http://localhost:8080
+# Minimal configuration for the coordinator:
+#coordinator=true
+#node-scheduler.include-coordinator=false
+#http-server.http.port=8080
+#query.max-memory=4GB
+#query.max-memory-per-node=1GB
+#discovery-server.enabled=true
+#discovery.uri=http://example.net:8080
+# Minimal configuration for the workers:
+#coordinator=false
+#http-server.http.port=8080
+#query.max-memory=4GB
+#query.max-memory-per-node=1GB
+#discovery.uri=http://example.net:8080
+http-server.log.path=/var/log/presto/http-request.log
+EOF
+
+cat > $PREFIX/$CONF_DIST_DIR/log.properties <<EOF
+io.trino=INFO
+EOF
+
+cat > $PREFIX/$CONF_DIST_DIR/catalog/tpch.properties <<EOF
+connector.name=tpch
+EOF
+
+cat > $PREFIX/$CONF_DIST_DIR/catalog/tpcds.properties <<EOF
+connector.name=tpcds
+EOF
+
+cat > $PREFIX/$CONF_DIST_DIR/catalog/jmx.properties <<EOF
+connector.name=jmx
+EOF
+
+ln -s ${CONF_DIR} $PREFIX/$LIB_DIR/etc