Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

JakartaEE New Infra ReStaging Job

This job should resolve current problem with retention period of OSSRH staging repository. It rebuilds and re-deploys artifacts from existing version tag.

Job Configuration

Source Code Management

TAG variable must be passed to Source Code Management section of Jenkins job configuration.

  • Branches to build: Branch Specifier (blank for 'any'): tags/$TAG

Job Parameters

This project is parameterized: checked

Name Type Default Description
TAG String Version tag to build.
DRY_RUN Boolean false Do not publish artifacts to OSSRH.

Shell Script

Maven targets being called are clean deploy. This is the proper sequence to be used, but may not work for any project. Another already proven set of targets is clean package source:jar javadoc:jar gpg:sign install:install nexus-staging:deploy.

There is jakartatransaction string hard coded in nexus plugin rc-list calls for parent < 1.0.5. It must be changed to proper deployment ID prefix of current project.

Job Environment

Script must be executed by bash because it contains bash specific code for arrays manipulation. Java SE version used to build the project depends on each project so it must be set properly. 

#!/bin/bash -ex

TOOLS_PREFIX='/opt/tools'
JAVA_PREFIX="${TOOLS_PREFIX}/java/openjdk"
MVN_HOME="${TOOLS_PREFIX}/apache-maven/latest"
JAVA_HOME="${JAVA_PREFIX}/jdk-11/latest"
PATH="${MVN_HOME}/bin:${JAVA_HOME}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Maven plugins
HELP_PLUGIN='org.apache.maven.plugins:maven-help-plugin:3.1.0'

Project pom.xml in a subdirectory

This section is optional and makes sense only when project pom.xml is not in root of ${WORKSPACE}. 

# Directory with project top level pom.xml
BUILD_DIR="${WORKSPACE}/some-directory"

cd ${BUILD_DIR}

Job Parameters Processing

Value of TAG must be set and must be non empty. Build must fail if those conditions are not met. Project and parent version numbers are retrieved from Maven POM.

Maven target to be executed depends on DRY_RUN value. Artifacts deployment must be suppressed when DRY_RUN is set to true. 

# Tag value is mandatory, fail the build if missing.
if [ -z "${TAG}" ]; then
  echo '-[ Missing required release tag! ]----------------------------------------------'
  exit 1
fi

# Compute release version
RELEASE_VERSION=`mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.version 2> /dev/null | grep -E '^[0-9]+(\.[0-9]+)+$'`
# Compute parent version
PARENT_VERSION=`mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.parent.version 2> /dev/null | grep -E '^[0-9]+(\.[0-9]+)+$'`

echo "Release tag:     ${TAG}"
echo "Release version: ${RELEASE_VERSION}"
echo "Parent version:  ${PARENT_VERSION}"

# Execute install target only on dry run.
if [ ${DRY_RUN} = 'true' ]; then
  echo '-[ Dry run turned on ]----------------------------------------------------------'
  MVN_DEPLOY_ARGS='install'
else
  MVN_DEPLOY_ARGS='deploy'
fi

Parent Version Evaluation

Unfortunately parent version < 1.0.5 does not contain nexus-staging-maven-plugin configuration so nexus deployment must be handled manually for those already deprecated versions. Some older projects releases may contain older parent and this script must handle this situation properly.

Versions compare works with 2 to 4 version components. Parent versioning always uses 3 components, but it's not a mandatory rule. Versions compare is following https://semver.org/ rules with limitation that each version component is not greater than 999.

Nexus staging plugin targets and their execution are set depending on parent version. 

# Maven listing depends on parent. Bash specific code to do some safe enough version comparsion.
# Limitations: version string has 2-4 version components, each component has max. 3 digits.
# Maybe this code can be simplified a bit later.
set -f
PAR_COMPONENTS=(${PARENT_VERSION//\./ })
PAR_COMP_LEN=${#PAR_COMPONENTS[@]}
case "${PAR_COMP_LEN}" in
  2)
    PAR_VER="$(printf '%03d%03d000000' ${PAR_COMPONENTS[0]} ${PAR_COMPONENTS[1]})"
  ;;
  3)
    PAR_VER="$(printf '%03d%03d%03d000' ${PAR_COMPONENTS[0]} ${PAR_COMPONENTS[1]} ${PAR_COMPONENTS[2]})"
  ;;
  4)
    PAR_VER="$(printf '%03d%03d%03d%03d' ${PAR_COMPONENTS[0]} ${PAR_COMPONENTS[1]} ${PAR_COMPONENTS[2]} ${PAR_COMPONENTS[3]})"
  ;;
  *)
    echo '-[ Error ]----------------------------------------------------------------------'
    echo "Invalid version numbers count in parent: ${NEXT_COMP_LEN}, exitting."
    exit 1
  ;;
esac
# Nexux plugin calls depend on parent version
if [ "${PAR_VER}" -ge '001000005000' ]; then
  LIST_CMD="nexus-staging:rc-list"
  DROP_CMD="nexus-staging:rc-drop"
else
  echo '-[ Warning: Old parent detected, fallback to external Nexus plugin call ]-------'
  NEXUS_PLUGIN='org.sonatype.plugins:nexus-staging-maven-plugin:1.6.7'
  NEXUS_PLUGIN_PARAMS='-DnexusUrl=https://oss.sonatype.org/ -DserverId=ossrh'
  LIST_CMD="${NEXUS_PLUGIN_PARAMS} ${NEXUS_PLUGIN}:rc-list"
  DROP_CMD="${NEXUS_PLUGIN_PARAMS} ${NEXUS_PLUGIN}:rc-drop"
  CLOSE_CMD="${NEXUS_PLUGIN_PARAMS} ${NEXUS_PLUGIN}:rc-close"
fi

Workaround: GPG initialization

GPG setup is missing in Docker image and GPG keys are passed in KEYRING environment variables. Content of this variable must be processed and stored into GPG key store. 

# Workaround: GPG initialization
gpg --batch --import ${KEYRING}
for fpr in $(gpg --list-keys --with-colons  | awk -F: '/fpr:/ {print $10}' | sort -u);
do
  echo -e "5\ny\n" |  gpg --batch --command-fd 0 --expert --edit-key $fpr trust;
done

Build and Deploy Artifacts

Nexus deployments are identified by keys stored as deployment description. Each key should be unique identifier of deployed project and its version so it contains groupId, artifactId and version strings separated by ':'.

Old Nexus deployments with the same groupId:artifactId:version key are dropped. All artifacts are built and deployed to the staging repository. Deployment is manually closed for parent < 1.0.5. 

# Project identifiers
ARTIFACT_ID=$(mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.artifactId | grep -Ev '(^\[)')
GROUP_ID=$(mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.groupId | grep -Ev '(^\[)')
STAGING_DESC="${GROUP_ID}:${ARTIFACT_ID}:${RELEASE_VERSION}"
STAGING_KEY="$(echo ${STAGING_DESC} | sed -e 's/\./\\\./g')"

# Delete old artifacts from Nexus
echo '-[ Drop old staging repository deployments ]------------------------------------'
for staging_key in $(mvn -B ${LIST_CMD} | egrep "^\[INFO\] [A-Z,a-z,-]+-[0-9]+\s+[A-Z]+\s+${STAGING_KEY}" | awk '{print $2}'); do
  echo "Repository ID: ${staging_key}"
  # Do not drop artifacts on dry run, just print ID to be dropped
  if [ ${DRY_RUN} != 'true' ]; then
    mvn -U -C -B \
        -DstagingRepositoryId="${staging_key}" \
        ${DROP_CMD}
  fi
done

# Deploy new artifacts
echo '-[ Deploy artifacts to staging repository ]-------------------------------------'
mvn -U -C -B \
    -DskipTests -Ddoclint=none -Poss-release,staging \
    -DstagingDescription="${STAGING_DESC}" \
    clean ${MVN_DEPLOY_ARGS}

if [ "${PAR_VER}" -lt '001000005000' -a ${DRY_RUN} != 'true' ]; then
  echo '-[ Warning: Old parent detected, closing deployment manually]-------------------'
  STAGING_REPO_ID=$(mvn -B ${LIST_CMD} | \
                    egrep '^\[INFO\] jakartatransaction\-[0-9]+[ ]+OPEN[ ]+Implicitly created.' | \
                    awk '{print $2}' | head -1)
  if [ -n "${STAGING_REPO_ID}" ]; then
    echo '-[ Closing staging repository ]-------------------------------------'
    echo "Repository ID: ${STAGING_REPO_ID}"
    mvn -U -C -B -Poss-release -Pstaging ${CLOSE_CMD} \
        -DstagingDescription="${STAGING_DESC}" \
        -DstagingRepositoryId=${STAGING_REPO_ID}
  fi
fi

Simple Version for parent >= 1.0.5

This simple version of the script can be used when all releases depend on parent version 1.0.5 or later. 

#!/bin/bash -ex

TOOLS_PREFIX='/opt/tools'
JAVA_PREFIX="${TOOLS_PREFIX}/java/openjdk"
MVN_HOME="${TOOLS_PREFIX}/apache-maven/latest"
JAVA_HOME="${JAVA_PREFIX}/jdk-11/latest"
PATH="${MVN_HOME}/bin:${JAVA_HOME}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Maven plugins
HELP_PLUGIN='org.apache.maven.plugins:maven-help-plugin:3.1.0'

# Tag value is mandatory, fail the build if missing.
if [ -z "${TAG}" ]; then
  echo '-[ Missing required release tag! ]----------------------------------------------'
  exit 1
fi

# Compute release version
RELEASE_VERSION=`mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.version 2> /dev/null | grep -E '^[0-9]+(\.[0-9]+)+$'`
# Compute parent version
PARENT_VERSION=`mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.parent.version 2> /dev/null | grep -E '^[0-9]+(\.[0-9]+)+$'`

echo "Release tag:     ${TAG}"
echo "Release version: ${RELEASE_VERSION}"
echo "Parent version:  ${PARENT_VERSION}"

# Execute install target only on dry run.
if [ ${DRY_RUN} = 'true' ]; then
  echo '-[ Dry run turned on ]----------------------------------------------------------'
  MVN_DEPLOY_ARGS='install'
else
  MVN_DEPLOY_ARGS='deploy'
fi

# Workaround: GPG initialization
gpg --batch --import ${KEYRING}
for fpr in $(gpg --list-keys --with-colons  | awk -F: '/fpr:/ {print $10}' | sort -u);
do
  echo -e "5\ny\n" |  gpg --batch --command-fd 0 --expert --edit-key $fpr trust;
done

# Project identifiers
ARTIFACT_ID=$(mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.artifactId | grep -Ev '(^\[)')
GROUP_ID=$(mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.groupId | grep -Ev '(^\[)')
STAGING_DESC="${GROUP_ID}:${ARTIFACT_ID}:${RELEASE_VERSION}"
STAGING_KEY="$(echo ${STAGING_DESC} | sed -e 's/\./\\\./g')"

# Delete old artifacts from Nexus
echo '-[ Drop old staging repository deployments ]------------------------------------'
for staging_key in $(mvn -B nexus-staging:rc-list | egrep "^\[INFO\] [A-Z,a-z,-]+-[0-9]+\s+[A-Z]+\s+${STAGING_KEY}" | awk '{print $2}'); do
  echo "Repository ID: ${staging_key}"
  # Do not drop artifacts on dry run, just print ID to be dropped
  if [ ${DRY_RUN} != 'true' ]; then
    mvn -U -C -B \
        -DstagingRepositoryId="${staging_key}" \
        nexus-staging:rc-drop
  fi
done

# Deploy new artifacts
echo '-[ Deploy artifacts to staging repository ]-------------------------------------'
mvn -U -C -B \
    -DskipTests -Ddoclint=none -Poss-release,staging \
    -DstagingDescription="${STAGING_DESC}" \
    clean ${MVN_DEPLOY_ARGS}

Whole Script

#!/bin/bash -ex

TOOLS_PREFIX='/opt/tools'
JAVA_PREFIX="${TOOLS_PREFIX}/java/oracle"
MVN_HOME="${TOOLS_PREFIX}/apache-maven/latest"
JAVA_HOME="${JAVA_PREFIX}/jdk-8/latest"
PATH="${MVN_HOME}/bin:${JAVA_HOME}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Maven plugins
HELP_PLUGIN='org.apache.maven.plugins:maven-help-plugin:3.1.0'

# Tag value is mandatory, fail the build if missing.
if [ -z "${TAG}" ]; then
  echo '-[ Missing required release tag! ]----------------------------------------------'
  exit 1
fi

# Compute release version
RELEASE_VERSION=`mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.version 2> /dev/null | grep -E '^[0-9]+(\.[0-9]+)+$'`
# Compute parent version
PARENT_VERSION=`mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.parent.version 2> /dev/null | grep -E '^[0-9]+(\.[0-9]+)+$'`

echo "Release tag:     ${TAG}"
echo "Release version: ${RELEASE_VERSION}"
echo "Parent version:  ${PARENT_VERSION}"

# Execute install target only on dry run.
if [ ${DRY_RUN} = 'true' ]; then
  echo '-[ Dry run turned on ]----------------------------------------------------------'
  MVN_DEPLOY_ARGS='install'
else
  MVN_DEPLOY_ARGS='deploy'
fi

# Maven listing depends on parent. Bash specific code to do some safe enough version comparsion.
# Limitations: version string has 2-4 version components, each component has max. 3 digits.
# Maybe this code can be simplified a bit later.
set -f
PAR_COMPONENTS=(${PARENT_VERSION//\./ })
PAR_COMP_LEN=${#PAR_COMPONENTS[@]}
case "${PAR_COMP_LEN}" in
  2)
    PAR_VER="$(printf '%03d%03d000000' ${PAR_COMPONENTS[0]} ${PAR_COMPONENTS[1]})"
  ;;
  3)
    PAR_VER="$(printf '%03d%03d%03d000' ${PAR_COMPONENTS[0]} ${PAR_COMPONENTS[1]} ${PAR_COMPONENTS[2]})"
  ;;
  4)
    PAR_VER="$(printf '%03d%03d%03d%03d' ${PAR_COMPONENTS[0]} ${PAR_COMPONENTS[1]} ${PAR_COMPONENTS[2]} ${PAR_COMPONENTS[3]})"
  ;;
  *)
    echo '-[ Error ]----------------------------------------------------------------------'
    echo "Invalid version numbers count in parent: ${NEXT_COMP_LEN}, exitting."
    exit 1
  ;;
esac
# Nexux plugin calls depend on parent version
if [ "${PAR_VER}" -ge '001000005000' ]; then
  LIST_CMD="nexus-staging:rc-list"
  DROP_CMD="nexus-staging:rc-drop"
else
  echo '-[ Warning: Old parent detected, fallback to external Nexus plugin call ]-------'
  NEXUS_PLUGIN='org.sonatype.plugins:nexus-staging-maven-plugin:1.6.7'
  NEXUS_PLUGIN_PARAMS='-DnexusUrl=https://oss.sonatype.org/ -DserverId=ossrh'
  LIST_CMD="${NEXUS_PLUGIN_PARAMS} ${NEXUS_PLUGIN}:rc-list"
  DROP_CMD="${NEXUS_PLUGIN_PARAMS} ${NEXUS_PLUGIN}:rc-drop"
  CLOSE_CMD="${NEXUS_PLUGIN_PARAMS} ${NEXUS_PLUGIN}:rc-close"
fi

# Workaround: GPG initialization
gpg --batch --import ${KEYRING}
for fpr in $(gpg --list-keys --with-colons  | awk -F: '/fpr:/ {print $10}' | sort -u);
do
  echo -e "5\ny\n" |  gpg --batch --command-fd 0 --expert --edit-key $fpr trust;
done

# Project identifiers
ARTIFACT_ID=$(mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.artifactId | grep -Ev '(^\[)')
GROUP_ID=$(mvn -B ${HELP_PLUGIN}:evaluate -Dexpression=project.groupId | grep -Ev '(^\[)')
STAGING_DESC="${GROUP_ID}:${ARTIFACT_ID}:${RELEASE_VERSION}"
STAGING_KEY="$(echo ${STAGING_DESC} | sed -e 's/\./\\\./g')"

# Delete old artifacts from Nexus
echo '-[ Drop old staging repository deployments ]------------------------------------'
for staging_key in $(mvn -B ${LIST_CMD} | egrep "^\[INFO\] [A-Z,a-z,-]+-[0-9]+\s+[A-Z]+\s+${STAGING_KEY}" | awk '{print $2}'); do
  echo "Repository ID: ${staging_key}"
  # Do not drop artifacts on dry run, just print ID to be dropped
  if [ ${DRY_RUN} != 'true' ]; then
    mvn -U -C -B \
        -DstagingRepositoryId="${staging_key}" \
        ${DROP_CMD}
  fi
done

# Deploy new artifacts
echo '-[ Deploy artifacts to staging repository ]-------------------------------------'
mvn -U -C -B \
    -DskipTests -Ddoclint=none -Poss-release,staging \
    -DstagingDescription="${STAGING_DESC}" \
    clean ${MVN_DEPLOY_ARGS}

if [ "${PAR_VER}" -lt '001000005000' -a ${DRY_RUN} != 'true' ]; then
  echo '-[ Warning: Old parent detected, closing deployment manually]-------------------'
  STAGING_REPO_ID=$(mvn -B ${LIST_CMD} | \
                    egrep '^\[INFO\] jakartatransaction\-[0-9]+[ ]+OPEN[ ]+Implicitly created.' | \
                    awk '{print $2}' | head -1)
  if [ -n "${STAGING_REPO_ID}" ]; then
    echo '-[ Closing staging repository ]-------------------------------------'
    echo "Repository ID: ${STAGING_REPO_ID}"
    mvn -U -C -B -Poss-release -Pstaging ${CLOSE_CMD} \
        -DstagingDescription="${STAGING_DESC}" \
        -DstagingRepositoryId=${STAGING_REPO_ID}
  fi
fi
Retrieved from "https://wiki.eclipse.org/index.php?title=JakartaEE_New_Infra_ReStaging_Job&oldid=429869"

This page was last modified 08:03, 27 December 2018 by Eclipsepedia anonymous user Unnamed Poltroon.

Back to the top