Skip to content

Commit

Permalink
[improvement](release) add release download scripts (#18703)
Browse files Browse the repository at this point in the history
For now, there are 3 packages for the release binaries of Doris: https://doris.apache.org/download
And user may be confused about how to download and deploy these packages.

So I provide a download script for each release, and user can simply download the script and run it, like:

```
> sh download_x64_apache.sh

Begin to download FE from "https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.3-rc02/apache-doris-fe-1.2.3-bin-x86_64.tar.xz" to "apache-doris-1.2.3-bin/" ...
Total size: 408078012 Bytes
#################################################### 100.0%
Begin to download BE from "https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.3-rc02/apache-doris-be-1.2.3-bin-x86_64.tar.xz" to "apache-doris-1.2.3-bin/" ...
Total size: 606211324 Bytes
#################################################### 100.0%
Begin to download DEPS from "https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.3-rc02/apache-doris-dependencies-1.2.3-bin-x86_64.tar.xz" to "apache-doris-1.2.3-bin/" ...
Total size: 253869148 Bytes
#################################################### 100.0%
Begin to assemble the binaries ...
Move java-udf-jar-with-dependencies.jar to be/lib/ ...
Download complete!
You can now deploy Apache Doris from apache-doris-1.2.3-bin/
```

The script will do the rest.

This script will later be published on the Download page of Apache Doris website, so that user can easily get
it and use it.

Currently only for Linux platform. Other platform is untested.
  • Loading branch information
morningman authored Apr 16, 2023
1 parent 69ae14f commit 70a418c
Show file tree
Hide file tree
Showing 10 changed files with 345 additions and 2 deletions.
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ package-lock.json
**/build/
**/target/
**/node_modules/
**/dist/
**/log/

# ignore project file
Expand Down
6 changes: 5 additions & 1 deletion dist/README
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ This directory is used to organize the list of third-party dependent licenses re

This directory holds the license files related to third-party dependencies.

2. `tools`
2. `tools/`

This directory stores tools related to license organization.

3. `download_scripts/`

This directory stores scripts for downloading release binaries.
36 changes: 36 additions & 0 deletions dist/download_scripts/1.2.3/download_arm_apache.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail

FE="apache-doris-fe-1.2.3-bin-arm"
BE="apache-doris-be-1.2.3-bin-arm"
DEPS="apache-doris-dependencies-1.2.3-bin-arm"
DOWNLOAD_LINK_PREFIX="https://dlcdn.apache.org/doris/1.2/1.2.3-rc02/"
DOWNLOAD_DIR="apache-doris-1.2.3-bin"

# Check and download download_base.sh
DOWNLOAD_BASE_SCRIPTS="download_base.sh"

if [[ ! -f "${DOWNLOAD_BASE_SCRIPTS}" ]]; then
curl -O https://raw.githubusercontent.com/apache/doris/master/dist/download_scripts/download_base.sh &&
chmod a+x "${DOWNLOAD_BASE_SCRIPTS}"
fi

# Begin to download
./"${DOWNLOAD_BASE_SCRIPTS}" "${FE}" "${BE}" "${DEPS}" "${DOWNLOAD_LINK_PREFIX}" "${DOWNLOAD_DIR}"
36 changes: 36 additions & 0 deletions dist/download_scripts/1.2.3/download_arm_tsinghua.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail

FE="apache-doris-fe-1.2.3-bin-arm"
BE="apache-doris-be-1.2.3-bin-arm"
DEPS="apache-doris-dependencies-1.2.3-bin-arm"
DOWNLOAD_LINK_PREFIX="https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.3-rc02/"
DOWNLOAD_DIR="apache-doris-1.2.3-bin"

# Check and download download_base.sh
DOWNLOAD_BASE_SCRIPTS="download_base.sh"

if [[ ! -f "${DOWNLOAD_BASE_SCRIPTS}" ]]; then
curl -O https://raw.githubusercontent.com/apache/doris/master/dist/download_scripts/download_base.sh &&
chmod a+x "${DOWNLOAD_BASE_SCRIPTS}"
fi

# Begin to download
./"${DOWNLOAD_BASE_SCRIPTS}" "${FE}" "${BE}" "${DEPS}" "${DOWNLOAD_LINK_PREFIX}" "${DOWNLOAD_DIR}"
36 changes: 36 additions & 0 deletions dist/download_scripts/1.2.3/download_x64_apache.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail

FE="apache-doris-fe-1.2.3-bin-x86_64"
BE="apache-doris-be-1.2.3-bin-x86_64"
DEPS="apache-doris-dependencies-1.2.3-bin-x86_64"
DOWNLOAD_LINK_PREFIX="https://dlcdn.apache.org/doris/1.2/1.2.3-rc02/"
DOWNLOAD_DIR="apache-doris-1.2.3-bin"

# Check and download download_base.sh
DOWNLOAD_BASE_SCRIPTS="download_base.sh"

if [[ ! -f "${DOWNLOAD_BASE_SCRIPTS}" ]]; then
curl -O https://raw.githubusercontent.com/apache/doris/master/dist/download_scripts/download_base.sh &&
chmod a+x "${DOWNLOAD_BASE_SCRIPTS}"
fi

# Begin to download
./"${DOWNLOAD_BASE_SCRIPTS}" "${FE}" "${BE}" "${DEPS}" "${DOWNLOAD_LINK_PREFIX}" "${DOWNLOAD_DIR}"
36 changes: 36 additions & 0 deletions dist/download_scripts/1.2.3/download_x64_noavx2_apache.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail

FE="apache-doris-fe-1.2.3-bin-x86_64"
BE="apache-doris-be-1.2.3-bin-x86_64-noavx2"
DEPS="apache-doris-dependencies-1.2.3-bin-x86_64"
DOWNLOAD_LINK_PREFIX="https://dlcdn.apache.org/doris/1.2/1.2.3-rc02/"
DOWNLOAD_DIR="apache-doris-1.2.3-bin"

# Check and download download_base.sh
DOWNLOAD_BASE_SCRIPTS="download_base.sh"

if [[ ! -f "${DOWNLOAD_BASE_SCRIPTS}" ]]; then
curl -O https://raw.githubusercontent.com/apache/doris/master/dist/download_scripts/download_base.sh &&
chmod a+x "${DOWNLOAD_BASE_SCRIPTS}"
fi

# Begin to download
./"${DOWNLOAD_BASE_SCRIPTS}" "${FE}" "${BE}" "${DEPS}" "${DOWNLOAD_LINK_PREFIX}" "${DOWNLOAD_DIR}"
36 changes: 36 additions & 0 deletions dist/download_scripts/1.2.3/download_x64_noavx2_tsinghua.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail

FE="apache-doris-fe-1.2.3-bin-x86_64"
BE="apache-doris-be-1.2.3-bin-x86_64-noavx2"
DEPS="apache-doris-dependencies-1.2.3-bin-x86_64"
DOWNLOAD_LINK_PREFIX="https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.3-rc02/"
DOWNLOAD_DIR="apache-doris-1.2.3-bin"

# Check and download download_base.sh
DOWNLOAD_BASE_SCRIPTS="download_base.sh"

if [[ ! -f "${DOWNLOAD_BASE_SCRIPTS}" ]]; then
curl -O https://raw.githubusercontent.com/apache/doris/master/dist/download_scripts/download_base.sh &&
chmod a+x "${DOWNLOAD_BASE_SCRIPTS}"
fi

# Begin to download
./"${DOWNLOAD_BASE_SCRIPTS}" "${FE}" "${BE}" "${DEPS}" "${DOWNLOAD_LINK_PREFIX}" "${DOWNLOAD_DIR}"
36 changes: 36 additions & 0 deletions dist/download_scripts/1.2.3/download_x64_tsinghua.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -eo pipefail

FE="apache-doris-fe-1.2.3-bin-x86_64"
BE="apache-doris-be-1.2.3-bin-x86_64"
DEPS="apache-doris-dependencies-1.2.3-bin-x86_64"
DOWNLOAD_LINK_PREFIX="https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.3-rc02/"
DOWNLOAD_DIR="apache-doris-1.2.3-bin"

# Check and download download_base.sh
DOWNLOAD_BASE_SCRIPTS="download_base.sh"

if [[ ! -f "${DOWNLOAD_BASE_SCRIPTS}" ]]; then
curl -O https://raw.githubusercontent.com/apache/doris/master/dist/download_scripts/download_base.sh &&
chmod a+x "${DOWNLOAD_BASE_SCRIPTS}"
fi

# Begin to download
./"${DOWNLOAD_BASE_SCRIPTS}" "${FE}" "${BE}" "${DEPS}" "${DOWNLOAD_LINK_PREFIX}" "${DOWNLOAD_DIR}"
36 changes: 36 additions & 0 deletions dist/download_scripts/README
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

This download scripts are used to download release bianries of Apache Doris.

The offical release bianries of Apache Doris has 3 packages for FE, BE and dependencies.

These scripts help download all packages of sepcified release, and do some reorgnization to make the package deploy ready.

- `download_bash.sh`

This is the base script. It receive parameter of which release to download to do the rest.

- Subdirectories

Each subdirectory is for a certain release, for 1.2.3, as an example:

It will provide serveral scripts for different platform and different CPU type.

For each new release, the release manager should create a new subdirectory here for that release,

and prepare the download scirpts, which contains the pacakges that need to be downloaded.
88 changes: 88 additions & 0 deletions dist/download_scripts/download_base.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

###############################################################################
# This script is used to download Apache Doris release binaries
# This is the base script of all download scripts.
# For certain release, it should create a script and wrap
# this script, eg:
# sh download_1.2.3.sh \
# apache-doris-fe-1.2.3-bin-x86_64 \
# apache-doris-be-1.2.3-bin-x86_64 \
# apache-doris-dependencies-1.2.3-bin-x86_64 \
# https://mirrors.tuna.tsinghua.edu.cn/apache/doris/1.2/1.2.3-rc02/ \
# apache-doris-1.2.3-bin
###############################################################################

set -eo pipefail

FE="$1"
BE="$2"
DEPS="$3"
DOWNLOAD_LINK_PREFIX="$4"
DOWNLOAD_DIR="$5"
FILE_SUFFIX=".tar.xz"

# Check if curl cmd exists
if [[ -n $(which curl >/dev/null 2>&1) ]]; then
echo "curl command not found on the system"
exit 1
fi

# Check download dir
if [[ -f "${DOWNLOAD_DIR}" || -d "${DOWNLOAD_DIR}" ]]; then
read -r -p "Download dir ${DOWNLOAD_DIR} already exists. Overwrite? [y/n] " resp
if [[ "${resp}" = "y" || "${resp}" = "Y" ]]; then
rm -rf "${DOWNLOAD_DIR}"
echo "Origin ${DOWNLOAD_DIR} has been removed and created a new one".
else
echo "Please remove the ${DOWNLOAD_DIR} before downloading."
exit 0
fi
fi

mkdir "${DOWNLOAD_DIR}"

# Begin to download
FE_LINK="${DOWNLOAD_LINK_PREFIX}${FE}${FILE_SUFFIX}"
BE_LINK="${DOWNLOAD_LINK_PREFIX}${BE}${FILE_SUFFIX}"
DEPS_LINK="${DOWNLOAD_LINK_PREFIX}${DEPS}${FILE_SUFFIX}"

download() {
MODULE="$1"
LINK="$2"
DIR="$3"
echo "Begin to download ${MODULE} from \"${LINK}\" to \"${DIR}/\" ..."
total_size=$(curl -sI "${LINK}" | grep -i Content-Length | awk '{print $2}' | tr -d '\r')
echo "Total size: ${total_size} Bytes"
echo "curl -# ${LINK} | tar xJ -C ${DIR}/"
curl -# "${LINK}" | tar xJ -C "${DIR}/"
}

download "FE" "${FE_LINK}" "${DOWNLOAD_DIR}"
download "BE" "${BE_LINK}" "${DOWNLOAD_DIR}"
download "DEPS" "${DEPS_LINK}" "${DOWNLOAD_DIR}"

# Assemble
echo "Begin to assemble the binaries ..."

echo "Move java-udf-jar-with-dependencies.jar to be/lib/ ..."
mv "${DOWNLOAD_DIR}/${DEPS}/java-udf-jar-with-dependencies.jar" "${DOWNLOAD_DIR}/${BE}/lib"

echo "Download complete!"
echo "You can now deploy Apache Doris from ${DOWNLOAD_DIR}/"

0 comments on commit 70a418c

Please sign in to comment.