Skip to content

[HOPSFS-258] Manually trigger hive build #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

[HOPSFS-258] Manually trigger hive build #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f1404a8
HOPSFS-258: Bump Hops Hadoop Version to 3.2.0.17-EE-SNAPSHOT
smkniazi May 21, 2025
bb5c648
HOPSFS-258: bumped hive version to 3.0.0.13.9
smkniazi May 21, 2025
b0542aa
Add basic bake
vatj Jun 24, 2025
c43c679
Hive version
vatj Jun 24, 2025
461fab2
Silence wget
vatj Jun 24, 2025
8080516
Push updated version
vatj Jun 24, 2025
35f72cf
Push trigger
vatj Jun 24, 2025
7aa55dc
Fix jira tag and commit hash
vatj Jun 24, 2025
9edee2c
Modify to allow download from arg nexus_url
vatj Jun 25, 2025
5e54be3
Update to allow not publishing the package
vatj Jun 26, 2025
a2c6393
Allow change to tar package name
vatj Jun 26, 2025
93e4480
Update tags
vatj Jun 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion accumulo-handler/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion beeline/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion classification/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion cli/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion contrib/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
20 changes: 11 additions & 9 deletions dockerfiles/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# syntax=docker/dockerfile:experimental
FROM openjdk:8
ARG HIVE_VERSION=3.0.0.13.6
ARG HADOOP_VERSION_EE=3.2.0.16-EE-RC0
ARG HIVE_TAR_NAME=hive-packaging-${HIVE_VERSION}-bin.tar.gz
ARG NEXUS_HIVE_URL="https://nexus.hops.works/repository/hive-artifacts/io/hops/hive/hive-packaging"
ARG HADOOP_VERSION_EE=3.2.0.17-EE-SNAPSHOT
ARG PROMETHEUS_EXPORTER_VERSION="0.12.0"
ARG PROMETHEUS_EXPORTER_VERSION="0.12.0"
ARG MYSQL_CONNECTOR_VERSION="8.0.21"
Expand All @@ -19,7 +21,7 @@ RUN groupadd -g ${HADOOP_GROUP_ID} ${HADOOP_GROUP_NAME} && \
# Download HADOOP
RUN --mount=type=secret,id=wgetrc,dst=/root/.wgetrc \
mkdir -p /srv/hops && \
wget https://nexus.hops.works/repository/hopshadoop/hops-$HADOOP_VERSION_EE.tgz && \
wget -q https://nexus.hops.works/repository/hopshadoop/hops-$HADOOP_VERSION_EE.tgz && \
tar -C /srv/hops/ -zxf hops-$HADOOP_VERSION_EE.tgz && \
ln -s /srv/hops/hadoop-$HADOOP_VERSION_EE /srv/hops/hadoop && \
rm -rf /srv/hops/hadoop/etc && \
Expand All @@ -35,12 +37,12 @@ ENV HADOOP_HOME=/srv/hops/hadoop
ENV HADOOP_CONF_DIR="$HADOOP_HOME/conf"

# Download hive
# https://nexus.hops.works/repository/hive-artifacts/io/hops/hive/hive-packaging/3.0.0.13.8/hive-packaging-3.0.0.13.8-bin.tar.gz
# https://nexus.hops.works/repository/hive-artifacts/io/hops/hive/hive-packaging/3.0.0.13.9/hive-packaging-3.0.0.13.9-bin.tar.gz
RUN --mount=type=secret,id=wgetrc,dst=/root/.wgetrc \
wget https://nexus.hops.works/repository/hive-artifacts/io/hops/hive/hive-packaging/$HIVE_VERSION/hive-packaging-$HIVE_VERSION-bin.tar.gz && \
tar -C /srv/hops/ -zxf hive-packaging-$HIVE_VERSION-bin.tar.gz && \
wget -q ${NEXUS_HIVE_URL}/$HIVE_VERSION/${HIVE_TAR_NAME} && \
tar -C /srv/hops/ -zxf ${HIVE_TAR_NAME} && \
ln -s /srv/hops/apache-hive-$HIVE_VERSION-bin /srv/hops/hive && \
rm hive-packaging-$HIVE_VERSION-bin.tar.gz
rm ${HIVE_TAR_NAME}

RUN rm -rf /srv/hops/hive/conf/*
RUN chmod -R 755 -R /srv/hops/apache-hive-$HIVE_VERSION-bin
Expand All @@ -54,13 +56,13 @@ ENV HOPSWORKS_JARS=/srv/hops/hopsworks-jars
ENV TEZ_JARS=/srv/hops/tez-jars
RUN mkdir -p ${HOPSWORKS_JARS}
# https://repo.hops.works/master/prometheus/jmx_prometheus_javaagent-0.12.0.jar
RUN wget -O ${HOPSWORKS_JARS}/jmx_prometheus_javaagent-${PROMETHEUS_EXPORTER_VERSION}.jar https://repo.hops.works/master/prometheus/jmx_prometheus_javaagent-${PROMETHEUS_EXPORTER_VERSION}.jar
RUN wget -qO ${HOPSWORKS_JARS}/jmx_prometheus_javaagent-${PROMETHEUS_EXPORTER_VERSION}.jar https://repo.hops.works/master/prometheus/jmx_prometheus_javaagent-${PROMETHEUS_EXPORTER_VERSION}.jar
# https://repo.hops.works/master/mysql-connector-java-8.0.21-bin.jar
RUN wget -O ${HOPSWORKS_JARS}/mysql-connector-java-${MYSQL_CONNECTOR_VERSION}-bin.jar https://repo.hops.works/master/mysql-connector-java-${MYSQL_CONNECTOR_VERSION}-bin.jar
RUN wget -qO ${HOPSWORKS_JARS}/mysql-connector-java-${MYSQL_CONNECTOR_VERSION}-bin.jar https://repo.hops.works/master/mysql-connector-java-${MYSQL_CONNECTOR_VERSION}-bin.jar

RUN chmod -R 755 ${HOPSWORKS_JARS}

RUN wget -O ${HOPSWORKS_JARS}/hudi-hadoop-mr-bundle-$HUDI_BUNDLE_VERSION.jar http://archiva.hops.works/repository/Hops/io/hops/hudi/hudi-hadoop-mr-bundle/$HUDI_BUNDLE_VERSION/hudi-hadoop-mr-bundle-$HUDI_BUNDLE_VERSION.jar
RUN wget -qO ${HOPSWORKS_JARS}/hudi-hadoop-mr-bundle-$HUDI_BUNDLE_VERSION.jar http://archiva.hops.works/repository/Hops/io/hops/hudi/hudi-hadoop-mr-bundle/$HUDI_BUNDLE_VERSION/hudi-hadoop-mr-bundle-$HUDI_BUNDLE_VERSION.jar

RUN mkdir -p $TEZ_JARS && \
wget -O ${TEZ_JARS}/apache-tez-${TEZ_VERSION}.tar.gz https://repo.hops.works/master/apache-tez-${TEZ_VERSION}.tar.gz
Expand Down
4 changes: 2 additions & 2 deletions dockerfiles/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ This folder contains the files we copy to the image when building
**Note** we are downloading the hive packaging from the hopsworks [archiva](https://archiva.hops.works/#welcome). This is way easier than building inside docker.

- User: `hive`
- HIVE_VERSION: `3.0.0.13.8`. This should come automatically from reading the pom file
- HADOOP_VERSION_EE: `3.2.0.16-EE-RC0`
- HIVE_VERSION: `3.0.0.13.9`. This should come automatically from reading the pom file
- HADOOP_VERSION_EE: `3.2.0.17-EE-SNAPSHOT`
- PROMETHEUS_EXPORTER_VERSION: '0.12.0'
- MYSQL_CONNECTOR_VERSION: '8.0.21'
- HUDI_BUNDLE_VERSION: '0.12.3.0'
Expand Down
49 changes: 49 additions & 0 deletions dockerfiles/docker-bake.hcl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
variable "HIVE_VERSION" {
default = "$HIVE_VERSION"
}

variable "NEXUS_HIVE_URL" {
default = "$NEXUS_HIVE_URL"
}

variable "HIVE_TAR_NAME" {
default = "$HIVE_TAR_NAME"
}

variable "REGISTRY" {
default = "$REGISTRY"
}

variable "REGISTRY_PROJECT" {
default = "$REGISTRY_PROJECT"
}

variable "COMMIT_HASH" {
default = "$COMMIT_HASH"
}

variable "JIRA_TAG" {
default = "$JIRA_TAG"
}


target "hive" {
dockerfile = "./Dockerfile"
platforms = ["linux/amd64"]
output = ["type=registry"]
attest = ["type=sbom"]
args = {
HIVE_VERSION = "${HIVE_VERSION}",
NEXUS_HIVE_URL = "${NEXUS_HIVE_URL}",
HIVE_TAR_NAME = "${HIVE_TAR_NAME}",
}
tags = [
"${REGISTRY}/${REGISTRY_PROJECT}/hopsworks/hive:${HIVE_VERSION}",
"${REGISTRY}/${REGISTRY_PROJECT}/hopsworks/hive:${HIVE_VERSION}-${COMMIT_HASH}",
"${REGISTRY}/${REGISTRY_PROJECT}/hopsworks/hive:${JIRA_TAG}",
]
cache-from= ["type=registry,ref=${REGISTRY}/${REGISTRY_PROJECT}/hopsworks/hive:cache-${JIRA_TAG}"]
cache-to = ["type=registry,ref=${REGISTRY}/${REGISTRY_PROJECT}/hopsworks/hive:cache-${JIRA_TAG},mode=max,image-manifest=true"]
secret = ["id=wgetrc,src=./.wgetrc"]
}

2 changes: 1 addition & 1 deletion hcatalog/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
<parent>
<groupId>io.hops.hive.hcatalog</groupId>
<artifactId>hive-hcatalog</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hcatalog/hcatalog-pig-adapter/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
<parent>
<groupId>io.hops.hive.hcatalog</groupId>
<artifactId>hive-hcatalog</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hcatalog/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hcatalog/server-extensions/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
<parent>
<groupId>io.hops.hive.hcatalog</groupId>
<artifactId>hive-hcatalog</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hcatalog/streaming/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>io.hops.hive.hcatalog</groupId>
<artifactId>hive-hcatalog</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hcatalog/webhcat/java-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
<parent>
<groupId>io.hops.hive.hcatalog</groupId>
<artifactId>hive-hcatalog</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hcatalog/webhcat/svr/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
<parent>
<groupId>io.hops.hive.hcatalog</groupId>
<artifactId>hive-hcatalog</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hops-jdbc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hplsql/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion itests/hive-minikdc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>org.apache.hive</groupId>
<artifactId>hive-it</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
4 changes: 2 additions & 2 deletions itests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

<groupId>org.apache.hive</groupId>
<artifactId>hive-it</artifactId>
<packaging>pom</packaging>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<name>Hive Integration - Parent</name>

<properties>
Expand Down
2 changes: 1 addition & 1 deletion itests/qtest-druid/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
<parent>
<artifactId>hive-it</artifactId>
<groupId>org.apache.hive</groupId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion jdbc-handler/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion jdbc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion kryo-registrator/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>hive</artifactId>
<groupId>io.hops.hive</groupId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
</parent>

<artifactId>hive-kryo-registrator</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion llap-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion llap-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion llap-ext-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion llap-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion llap-tez/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion metastore/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion packaging/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
4 changes: 2 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

<groupId>io.hops.hive</groupId>
<artifactId>hive</artifactId>
<version>3.0.0.13.8</version>
<version>3.0.0.13.9</version>
<packaging>pom</packaging>

<name>Hive</name>
Expand Down Expand Up @@ -142,7 +142,7 @@
<flatbuffers.version>1.9.0</flatbuffers.version>
<guava.version>27.0-jre</guava.version>
<groovy.version>2.4.11</groovy.version>
<hadoop.version>3.2.0.16-EE-RC0</hadoop.version>
<hadoop.version>3.2.0.17-EE-SNAPSHOT</hadoop.version>
<h2database.version>1.3.166</h2database.version>
<hadoop.bin.path>${basedir}/${hive.path.to.root}/testutils/hadoop</hadoop.bin.path>
<hamcrest.version>1.3</hamcrest.version>
Expand Down
Loading