Skip to content

Modified workflow to create new direct-query directory for async-query-core and add last commit_id to the metadata file #3648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

Modified workflow to create new direct-query directory for async-query-core and add last commit_id to the metadata file #3648

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5d1ead5
Modified workflow to create new direct-query directory for async-quer…
ahkcs May 21, 2025
c294392
Extract version info from build.gradle
ahkcs May 21, 2025
76c4871
Merge branch 'main' into feat/workflow_change
ahkcs May 21, 2025
4475905
fix refactoring
ahkcs May 21, 2025
9f73793
Merge branch 'feat/workflow_change' of github.com:KaiHuang020719/sql …
ahkcs May 21, 2025
2ac32d5
revision
ahkcs May 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 171 additions & 22 deletions .github/workflows/maven-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,44 +19,193 @@ jobs:
id-token: write
contents: write

# Set credentials once at the job level
env:
SNAPSHOT_REPO_URL: https://aws.oss.sonatype.org/content/repositories/snapshots/

steps:
- uses: actions/setup-java@v3
with:
distribution: temurin # Temurin is a distribution of adoptium
distribution: temurin
java-version: 21
- uses: actions/checkout@v3
- uses: aws-actions/configure-aws-credentials@v1.7.0
with:
role-to-assume: ${{ secrets.PUBLISH_SNAPSHOTS_ROLE }}
aws-region: us-east-1

# Create the initial direct-query directory structure
- name: Create direct-query directory structure in repository
# Get and mask credentials once in a dedicated step
- name: Setup publishing credentials
id: creds
run: |
# Get credentials for publishing
export SONATYPE_USERNAME=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-username --query SecretString --output text)
export SONATYPE_PASSWORD=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-password --query SecretString --output text)
SONATYPE_USERNAME=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-username --query SecretString --output text)
SONATYPE_PASSWORD=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-password --query SecretString --output text)
echo "::add-mask::$SONATYPE_USERNAME"
echo "::add-mask::$SONATYPE_PASSWORD"
echo "SONATYPE_USERNAME=$SONATYPE_USERNAME" >> $GITHUB_ENV
echo "SONATYPE_PASSWORD=$SONATYPE_PASSWORD" >> $GITHUB_ENV

# Capture the commit ID for metadata purposes
- name: Set commit ID
id: set_commit
run: |
COMMIT_ID=$(git log -1 --format='%H')
echo "commit_id=${COMMIT_ID}" >> $GITHUB_OUTPUT
echo "Using commit ID: ${COMMIT_ID}"

# Extract version information from build.gradle
- name: Extract version from build.gradle
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think the current shadow jar building depends on opensearch version, can you double check?

Copy link
Contributor Author

@ahkcs ahkcs May 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the version is used to name the directory that we upload our jar to

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's decouple it from opensearch release

id: extract_version
run: |
# Extract the base OpenSearch version from build.gradle
OPENSEARCH_VERSION=$(grep -o 'opensearch_version = System.getProperty("opensearch.version", "[^"]*' build.gradle | sed 's/.*"\(.*\)"/\1/')

# Extract the base version without snapshot or other qualifiers
BASE_VERSION=$(echo $OPENSEARCH_VERSION | cut -d'-' -f1)

# OpenSearch SQL version follows pattern: <base_version>.0
VERSION="${BASE_VERSION}.0"

# Check if it's a snapshot (true by default in build.gradle)
IS_SNAPSHOT=$(grep -o 'isSnapshot = "[^"]*' build.gradle | sed 's/.*"\(.*\)"/\1/')
if [[ "$IS_SNAPSHOT" == "true" ]]; then
VERSION="${VERSION}-SNAPSHOT"
fi

# Check for version qualifier
BUILD_VERSION_QUALIFIER=$(grep -o 'buildVersionQualifier = System.getProperty("build.version_qualifier", "[^"]*' build.gradle | sed 's/.*"\(.*\)"/\1/')
if [[ -n "$BUILD_VERSION_QUALIFIER" ]]; then
# Insert qualifier before -SNAPSHOT if present
if [[ "$VERSION" == *"-SNAPSHOT"* ]]; then
VERSION="${VERSION/-SNAPSHOT/}"
VERSION="${VERSION}-${BUILD_VERSION_QUALIFIER}-SNAPSHOT"
else
VERSION="${VERSION}-${BUILD_VERSION_QUALIFIER}"
fi
fi

echo "VERSION=${VERSION}" >> $GITHUB_OUTPUT
echo "Extracted version: ${VERSION}"

- name: Build and publish shadow JAR
run: |
# Build the shadow JAR
./gradlew :async-query-core:shadowJar

# Define constants
ARTIFACT_ID="async-query-core"
DIR_NAME="direct-query"
GROUP_PATH="org/opensearch"
VERSION="${{ steps.extract_version.outputs.VERSION }}"

# Find the generated shadow JAR
SHADOW_JAR=$(find ./async-query-core/build/libs/ -name "*-all.jar" | head -n 1)

if [ -z "$SHADOW_JAR" ]; then
echo "Error: Shadow JAR not found!"
exit 1
fi

# Create directory structure in local Maven repository
MAVEN_LOCAL_PATH="${HOME}/.m2/repository/${GROUP_PATH}/${DIR_NAME}/${VERSION}"
mkdir -p "${MAVEN_LOCAL_PATH}"

# Copy the shadow JAR to the local Maven repository with proper naming
MAVEN_JAR_NAME="${ARTIFACT_ID}-${VERSION}.jar"
cp "${SHADOW_JAR}" "${MAVEN_LOCAL_PATH}/${MAVEN_JAR_NAME}"

# Generate a POM file
cat > "${MAVEN_LOCAL_PATH}/${ARTIFACT_ID}-${VERSION}.pom" << EOF
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.opensearch</groupId>
<artifactId>${ARTIFACT_ID}</artifactId>
<version>${VERSION}</version>
</project>
EOF

echo "Shadow JAR and POM published to local Maven repository for version ${VERSION}"

# Checkout opensearch-build-libraries repository for publishing scripts
- uses: actions/checkout@v4
with:
repository: 'opensearch-project/opensearch-build-libraries'
path: 'build'

- name: Generate SHA checksums for JAR and POM files
run: |
for i in `find ${HOME}/.m2/repository/org/opensearch/ -name "*.pom" -type f`; do sha512sum "$i" | awk '{print $1}' >> "$i.sha512"; done
for i in `find ${HOME}/.m2/repository/org/opensearch/ -name "*.jar" -type f`; do sha512sum "$i" | awk '{print $1}' >> "$i.sha512"; done
for i in `find ${HOME}/.m2/repository/org/opensearch/ -name "*.pom" -type f`; do sha256sum "$i" | awk '{print $1}' >> "$i.sha256"; done
for i in `find ${HOME}/.m2/repository/org/opensearch/ -name "*.jar" -type f`; do sha256sum "$i" | awk '{print $1}' >> "$i.sha256"; done

- name: Install XML tools
run: sudo apt-get update && sudo apt-get install -y xmlstarlet

- name: Publish snapshots to maven
run: |
# Publish snapshots to maven
cd build/resources/publish/
cp -a $HOME/.m2/repository/* ./
./publish-snapshot.sh ./

- name: Update version metadata with commit ID
run: |
COMMIT_ID="${{ steps.set_commit.outputs.commit_id }}"
DIR_NAME="direct-query"

VERSION="${{ steps.extract_version.outputs.VERSION }}"

# Add commit ID to version-specific metadata file
echo "Processing commit ID for version: ${VERSION}"

# Create a placeholder file
TEMP_DIR=$(mktemp -d)
echo "Directory placeholder - $(date)" > "${TEMP_DIR}/.placeholder"
METADATA_FILE="${TEMP_DIR}/maven-metadata.xml"

# Download metadata from repository
META_URL="${SNAPSHOT_REPO_URL}org/opensearch/${DIR_NAME}/${VERSION}/maven-metadata.xml"
echo "Downloading metadata from ${META_URL}"

# Try to download the metadata file
curl -s -u "${SONATYPE_USERNAME}:${SONATYPE_PASSWORD}" -o "${METADATA_FILE}" "${META_URL}"

# Upload the placeholder file to create the directory structure
echo "Creating initial directory structure..."
curl -X PUT -u "${SONATYPE_USERNAME}:${SONATYPE_PASSWORD}" \
--upload-file "${TEMP_DIR}/.placeholder" \
"https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/direct-query/.placeholder"
# If successful, modify and upload back
if [ -s "${METADATA_FILE}" ]; then
echo "Modifying metadata for ${VERSION}"
cp "${METADATA_FILE}" "${METADATA_FILE}.bak"

awk -v commit="${COMMIT_ID}" '
/<versioning>/ {
print $0
print " <commitId>" commit "</commitId>"
next
}
{print}
' "${METADATA_FILE}.bak" > "${METADATA_FILE}"

# Upload modified file back
curl -X PUT -u "${SONATYPE_USERNAME}:${SONATYPE_PASSWORD}" --upload-file "${METADATA_FILE}" "${META_URL}"

# Update the SHA checksums (the publishing script might not have created these)
cd "${TEMP_DIR}"
sha256sum "maven-metadata.xml" | awk '{print $1}' > "maven-metadata.xml.sha256"
sha512sum "maven-metadata.xml" | awk '{print $1}' > "maven-metadata.xml.sha512"

# Upload the checksums
curl -X PUT -u "${SONATYPE_USERNAME}:${SONATYPE_PASSWORD}" --upload-file "maven-metadata.xml.sha256" "${META_URL}.sha256"
curl -X PUT -u "${SONATYPE_USERNAME}:${SONATYPE_PASSWORD}" --upload-file "maven-metadata.xml.sha512" "${META_URL}.sha512"
cd -

echo "Updated metadata and checksums for ${VERSION}"
else
echo "Failed to download metadata for ${VERSION}"
exit 1
fi

# Clean up
rm -rf "${TEMP_DIR}"
echo "Directory structure created"

- name: publish snapshots to maven
run: |
export SONATYPE_USERNAME=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-username --query SecretString --output text)
export SONATYPE_PASSWORD=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-password --query SecretString --output text)
echo "::add-mask::$SONATYPE_USERNAME"
echo "::add-mask::$SONATYPE_PASSWORD"
./gradlew publishPluginZipPublicationToSnapshotsRepository

echo "Version metadata updated with commit ID"
Loading