Adding the generic_package.sh script file #44
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| - job: | ||
| name: debian-package-builder | ||
| node: master | ||
| description: glusterfs package script to build both debian and ubuntu | ||
| project-type: freestyle | ||
|
|
||
| scm: | ||
| - glusterfs | ||
|
|
||
| parameters: | ||
| - string: | ||
| default: v7.0 | ||
| description: Refspec to "git checkout" as the release code snapshot. Typically | ||
| a git annotated tag like "v6.0". Specifying a branch name will use the | ||
| current HEAD of that branch. | ||
| name: GERRIT_REFSPEC | ||
| - string: | ||
| default: 7 | ||
| description: Series number for the package to be built against. | ||
| name: SERIES | ||
| - string: | ||
| default: 0 | ||
| description: Version number for the package to be built against. | ||
| name: VERSION | ||
| - string: | ||
| default: | ||
| description: Release number for the package to be built against. | ||
| Leave it empty if you are building above series 5. | ||
|
There was a problem hiding this comment. Mhh, what is series 5, you mean version 5 branch ? is it still supported, or can we drp it ? There was a problem hiding this comment. So, the comment is still here, what is series 5 ? Do you mean "release 5" ? (and if we can drop, should it be removed from the description ?) |
||
| name: RELEASE | ||
| - string: | ||
| default: all | ||
| description: Use the default "all" or leave it empty to build | ||
| it for both Debian and Ubuntu. If a specific distribution is needed, | ||
| please name it as "debian" or "ubuntu". | ||
| name: OS | ||
| - string: | ||
| default: | ||
| description: Leave it empty to build it for both Debian and Ubuntu. | ||
| If a specific distribution is selected above, then mention the | ||
| specific flavor here. is needed. Flavors can be named as "xenial" or | ||
| numbered as "16.04" for ubuntu. For debian it can named as "bullseye" | ||
| or it can be numbered as "11". | ||
| name: FLAVOR | ||
| - string: | ||
| default: | ||
| description: The latest series with respect to all the existing series. | ||
| name: LATEST_SERIES | ||
| - string: | ||
| default: | ||
| Description: The latest version of latest series. | ||
| name: LATEST_VERSION | ||
| - string: | ||
| default: [email protected],[email protected] | ||
| description: Mail addresses to send the announcement of the packages too. | ||
| name: ANNOUNCE_EMAIL | ||
|
|
||
| builders: | ||
| - shell: /opt/qa/debian-ubuntu-package.sh | ||
|
There was a problem hiding this comment. Location is not correct. You have added the file in build-gluster-org/scripts/debian-ubuntu-package.sh |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| #!/bin/bash | ||
|
|
||
| #for both the debian and ubuntu packaging of various versions we have to | ||
| #ssh to a particular ubuntu machine and package it using pbuilder. | ||
|
|
||
| set -xe | ||
|
|
||
| while [ $# -eq 5 ] | ||
| do | ||
| echo "building everything" | ||
| echo "packing debian distribution" | ||
| ~/build-gluster-org/scripts/generic_package.sh debian stretch $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| ~/build-gluster-org/scripts/generic_package.sh debian buster $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh debian bullseye $SERIES $VERSION $RELEASE | ||
| echo "packing ubuntu distribution" | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu xenial $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu bionic $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu disco $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu eoan $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu focal $SERIES $VERSION $RELEASE | ||
| done | ||
|
|
||
| while [ $# -gt 5 ] | ||
| do | ||
| if [ "$OS" == "all" ]; then | ||
| echo "packing all distribution" | ||
| echo "packing debian distribution" | ||
| ~/build-gluster-org/scripts/generic_package.sh debian stretch $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| ~/build-gluster-org/scripts/generic_package.sh debian buster $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh debian bullseye $SERIES $VERSION $RELEASE | ||
| echo "packing ubuntu distribution" | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu xenial $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu bionic $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu disco $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu eoan $SERIES $VERSION $RELEASE | ||
| ~/build-gluster-org/scripts/generic_package.sh ubuntu focal $SERIES $VERSION $RELEASE | ||
| elif [ "$OS" == "debian" ]; then | ||
| echo "packing debian alone" | ||
| if [ "$FLAVOR" == "stretch" ] || [ "$FLAVOR" == "9" ]; then | ||
|
There was a problem hiding this comment. I think it would be cleaner to have 1 single if with the 6 possibles values than 3 separate ones. There was a problem hiding this comment. can you please elaborate this comment There was a problem hiding this comment. Well, here we have 3 lines with if, they all run the same script with the same argument, the only difference is the echo. This seems harder to read while : case $FLAVOR in is shorter |
||
| echo "packing debian stretch alone" | ||
| ../scripts/generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| fi | ||
| if [ "$FLAVOR" == "buster" ] || [ "$FLAVOR" == "10" ]; then | ||
| echo "packing debian buster alone" | ||
| ../scripts/generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| fi | ||
| if [ "$FLAVOR" == "bullseye" ] || [ "$FLAVOR" == "11" ]; then | ||
| echo "packing debian bullseye alone" | ||
| ../scripts//generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| fi | ||
| elif [ "$OS" == "ubuntu" ]; then | ||
| echo "packing ubuntu alone" | ||
| if [ "$FLAVOR" == "xenial" ] || [ "$FLAVOR" == "16.04" ]; then | ||
| echo "packing xenial alone" | ||
| ../scripts/generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| fi | ||
| if [ "$FLAVOR" == "bionic" ] || [ "$FLAVOR" == "18.04" ]; then | ||
| echo "packing bionic alone" | ||
| ../scripts/generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| fi | ||
| if [ "$FLAVOR" == "disco" ] || [ "$FLAVOR" == "19.04" ]; then | ||
| echo "packing disco alone" | ||
| ../scripts/generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| fi | ||
| if [ "$FLAVOR" == "eoan" ] || [ "$FLAVOR" == "19.10" ]; then | ||
| echo "packing eoan alone" | ||
| ../scripts/generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| fi | ||
| if [ "$FLAVOR" == "focal" ] || [ "$FLAVOR" == "20.04" ] || [ "$FLAVOR" == "focal fossa"]; then | ||
| echo "packing focal alone" | ||
| ../scripts/generic_package.sh $OS $FLAVOR $SERIES $VERSION $RELEASE $LATEST_SERIES $LATEST_VERSION | ||
| fi | ||
| if [ "$FLAVOR" == "hirsute" ] || [ "$FLAVOR" == "21.04" ] || [ "$FLAVOR" == "hirsute hippo"]; then | ||
| fi | ||
| done | ||
|
|
||
| RET=$? | ||
|
|
||
| exit $RET | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,169 @@ | ||
| #!/bin/bash | ||
|
|
||
| #immediately exit if any command has a non-zero exit status | ||
| set -euxo pipefail | ||
| #enable debugging | ||
| set -x | ||
|
|
||
| ################################################################################################ | ||
| #OS (e.g. Ubuntu/Debian) | ||
| #Series (e.g. 4.1) | ||
| #Version (e.g. 4.1.0) | ||
| #Release (e.g. 1) | ||
| #Flavor(e.g. Ubuntu - xenial/bionic/eoan/focal/groovy, Debian - buster/stretch/bullseye) | ||
|
|
||
| # to run use: 'bash -x generic_package.sh debian stretch 6 4.1.0 1' | ||
|
|
||
| #Key generation: https://help.ubuntu.com/community/GnuPrivacyGuardHowto#Generating_an_OpenPGP_Key | ||
| #Addditional info: sending of pubkey to a keyserv is been done using the MIT keyserv | ||
| ################################################################################################# | ||
| os=$1 | ||
| flavor=$2 | ||
| series=$3 | ||
| version=$4 | ||
| release=$5 | ||
| latest_series=$6 | ||
| latest_version=$7 | ||
|
|
||
| #Keys required in debian builds | ||
| declare -a debuild_keys | ||
| debuild_keys="F9C958A3AEE0D2184FAD1CBD43607F0DC2F8238C" | ||
|
|
||
| declare -a pbuild_keys | ||
| pbuild_keys="BF11C87C" | ||
|
|
||
| # Check for OS(Ubuntu or Debian) | ||
| if [ "$os" == "ubuntu" ]; then | ||
| mirror="http://ubuntu.osuosl.org/ubuntu/" | ||
| debuild_key=4F5B5CA5 | ||
| elif [ "$os" == "debian" ]; then | ||
| mirror="http://ftp.us.debian.org/debian/" | ||
| if [ "$flavor" == "stretch" ]; then | ||
| debuild_key=${pbuild_keys} | ||
| else | ||
| debuild_key=${debuild_keys} | ||
| fi | ||
| pbuild_key=${pbuild_keys} | ||
| else | ||
| echo "Exiting: OS should be debian or ubuntu. Please provide the right one" | ||
| exit | ||
| fi | ||
|
|
||
| mkdir ${os}-${flavor}-Glusterfs-${version} | ||
|
|
||
| cd ${os}-${flavor}-Glusterfs-${version} | ||
|
|
||
| mkdir build packages | ||
|
|
||
| echo "Building glusterfs-${version}-${release} for ${flavor}" | ||
|
|
||
| cd build | ||
|
|
||
| TGZS=(`ls ~/glusterfs-${version}-?-*/build/glusterfs-${version}.tar.gz`) | ||
| echo ${TGZS[0]} | ||
|
|
||
| if [ -z ${TGZS[0]} ]; then | ||
| echo "wget https://download.gluster.org/pub/gluster/glusterfs/${series}/${version}/glusterfs-${version}.tar.gz" | ||
| wget https://download.gluster.org/pub/gluster/glusterfs/${series}/${version}/glusterfs-${version}.tar.gz | ||
| else | ||
| echo "found ${TGZS[0]}, using it..." | ||
| cp ${TGZS[0]} . | ||
| fi | ||
|
|
||
| echo "Creating link file.." | ||
| ln -s glusterfs-${version}.tar.gz glusterfs_${version}.orig.tar.gz | ||
|
|
||
| echo "Untaring.." | ||
| tar xpf glusterfs-${version}.tar.gz | ||
|
|
||
| # Changelogs needed for building are maintained in a separate repo. | ||
| # the repo has to be clone and updated properly so we can copy the changelogs so far. | ||
|
|
||
| echo "Cloning the glusterfs-debian repo" | ||
| git clone https://github.com/gluster/glusterfs-debian.git | ||
|
|
||
| cd glusterfs-debian/ | ||
|
|
||
| git checkout -b ${flavor}-${series}-local origin/${flavor}-glusterfs-${series} | ||
|
|
||
| if [ "$os" == "ubuntu" ]; then | ||
| dch --distribution ${flavor} -u medium -v ${version}-${os}1~${flavor}1 "GlusterFS ${version} GA" | ||
| elif [ "$os" == "debian" ]; then | ||
| dch --distribution ${flavor} -u low -v (${version}-1 ${flavor}) "GlusterFS ${version} GA" | ||
| fi | ||
|
|
||
| git commit -a -m "Glusterfs ${version} G.A (${flavor})" | ||
|
|
||
| echo "Copying Changelog to source" | ||
| cp -a debian ../glusterfs-${version}/ | ||
|
|
||
| echo "Building source package.." | ||
| cd ../glusterfs-${version} | ||
| debuild -S -sa -k${debuild_key} | ||
|
There was a problem hiding this comment. Where would the signing key stored ? (since -k requires that) There was a problem hiding this comment. debuild_key variable contains the key. There was a problem hiding this comment. I wasn't clear on my question. From what I understand, -k is the key identifier. But the actual private key is somewhere else (I think in ~/.gnupg), and so we need to discuss how that part is going to be managed. There was a problem hiding this comment. we are initialising the values for debuild keys in https://github.com/gluster/build-jobs/pull/44/files/b32e0625d90b8c89432e753aab502d5e72b64b46#diff-02d2c32f5282eea2a7412f831420ae8c2f55ce0ed17671f6878e110c258b2e6bR29 The key creation is something we have maintained in a particular machine, whose access is not given to wider audience. There was a problem hiding this comment. Then, where is that machine ? There was a problem hiding this comment. Here is the draft of mojo doc that contains machine details: https://source.redhat.com/.motion/communitiesatredhat/crosscuttingco/pjmcop/prodev/study_grp/study_group_sign_up/A883E22D-0D6C-EB11-80F2-000D3A020FEB/.object There was a problem hiding this comment. |
||
|
|
||
| echo "Uploading the packages.." | ||
| if [ "$os" == "ubuntu" ]; then | ||
| cd .. | ||
| dput ppa:gluster/glusterfs-${series} glusterfs_${version}-${os}1~${flavor}1_source.changes | ||
|
There was a problem hiding this comment. Does it requires some authentication ? I do not see how it is done. There was a problem hiding this comment. No, AIFIK. @kalebskeithley any more insights? There was a problem hiding this comment. Well, if could be without authentication, but then, we need some kind of signature, cause I do not think anyone can push debian package for us, no ? There was a problem hiding this comment. yes, we would need to follow https://help.launchpad.net/Packaging/PPA/Uploading to get required authentiacation There was a problem hiding this comment. The documentation do not tell much, but if that use ssh, then we need to have the key available to the builder, which is not declared in the job yaml file. There is example here: https://github.com/gluster/build-jobs/blob/master/build-gluster-org/jobs/centos7-regression.yml#L64 for adding a credential There was a problem hiding this comment. The OpenPGP keys are used for signing as described here $ gpg --list-keys on the machine rhs-vm-17.storage-dev.lab.eng.bos.redhat.com under the user glusterpackager, shows the already existing keys. (documented the same in the mojo doc draft: let me know if any more information is needed to be documented) There was a problem hiding this comment. But that's a internal VM, and Jenkins is hosted outside of the lan. I may miss something obvious, but Jenkins can't connect to that server, and I think Product Security would strongly dislike that a external server (build.gluster.org) is able to remotely execute any code on a internal system ( rhs-vm-17.storage-dev.lab.eng.bos.redhat.com ). So we need to have the key as secret in the job, and store that in Jenkins. We can't use RH internal system. There was a problem hiding this comment. yes, we can store the key as secret in the job. We can add it in jenkins machine, I can help in getting it from rhs-vm-17.storage-dev.lab.eng.bos.redhat.com |
||
|
|
||
| echo "Done" | ||
| exit | ||
Shwetha-Acharya marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| fi | ||
|
|
||
| # we are using the same builder machine to build so we are running the "pbuilder | ||
| # create" everytime to create the chroot according to the os and flavor we want to build. | ||
| echo "creating chroot for ${os} ${flavor}" | ||
| sudo pbuilder create --distribution ${flavor} --mirror ${mirror} --debootstrapopts --keyring=/usr/share/keyrings/${os}-archive-keyring.gpg | ||
|
|
||
| echo "Building glusterfs-${version} for ${os} ${flavor} using the chroot and .dsc we created" | ||
|
|
||
| # have to use the .dsc file inside the ${os}${flavor} folder | ||
| sudo pbuilder build ~/${os}-${flavor}-Glusterfs-${version}/build/glusterfs_${version}-${release}.dsc | tee build.log | ||
|
|
||
| #move the packages to packages directory. | ||
| cp /var/cache/pbuilder/result/glusterfs*${version}-${release}*.deb ~/${os}-${flavor}-Glusterfs-${version}/packages/ | ||
| rm -rf /var/cache/pbuilder/result/glusterfs*${version}-${release}*.deb | ||
|
There was a problem hiding this comment. Why not mv instead of cp + rm ? |
||
|
|
||
| if [ "$flavor" != "stretch" ]; then | ||
|
There was a problem hiding this comment. So, why is stretch special here ? There was a problem hiding this comment. It is because /var/cache/pbuilder/result/libg*${version}-${release}*.deb are not created only in stretch. In buster and bullseye, they will be created |
||
| mv /var/cache/pbuilder/result/libg*${version}-${release}*.deb ~/${os}-${flavor}-Glusterfs-${version}/packages/ | ||
| fi | ||
| /usr/share/debdelta/dpkg-sig -v -k ${pbuild_key} --sign builder ~/${os}-${flavor}-Glusterfs-${version}/packages/glusterfs-*${version}-${release}*.deb | ||
|
|
||
| cd /var/www/repos/apt/debian/ | ||
|
|
||
| rm -rf pool/* dists/* db/* | ||
|
There was a problem hiding this comment. Not sure jenkins can erase file there. There was a problem hiding this comment. We can have a trail run to verify it. There was a problem hiding this comment. Also, if we create file here, shouldn't it be cleaned with a trap, like the rest ? |
||
|
|
||
| cp ~/conf.distributions/${series} conf/distributions | ||
Shwetha-Acharya marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # distribute Debian packages using apt | ||
| for i in ~/${os}-${flavor}-Glusterfs-${version}/packages/glusterfs-*${version}-${release}*; do reprepro includedeb $flavor $i; done | ||
| if [ "$flavor" != "stretch" ]; then | ||
| for i in ~/${os}-${flavor}-Glusterfs-${version}/packages/libg*${version}-${release}*.deb; do reprepro includedeb $flavor $i; done | ||
| fi | ||
| reprepro includedsc ${flavor} ~/${os}-${flavor}-Glusterfs-${version}/build/glusterfs_${version}-${release}.dsc | ||
|
|
||
| tar czf ~/${os}-${flavor}-Glusterfs-${version}/${flavor}-apt-amd64-${version}.tgz pool/ dists/ | ||
|
|
||
| echo "Pushing Changelog changes.." | ||
| git push origin ${flavor}-${series}-local:${flavor}-glusterfs-${series} | ||
|
There was a problem hiding this comment. This is not going to work if the clone is done over HTTP There was a problem hiding this comment. I am sorry, but I still see https for the clone. I was not precise enough, I want to point that we can't push over http, including https. And if we need to push a new changelog, we need a ssh key. There was a problem hiding this comment. clone is now changed to ssh |
||
|
|
||
| cd ~/${os}-${flavor}-Glusterfs-${version} | ||
|
|
||
| #copy the tar.gz file produced by the build to download.rht.gluster.org:/var/www/scratch | ||
| scp $flavor-apt-amd64-$version.tgz [email protected]:/var/www/scratch | ||
|
|
||
| ssh [email protected] /var/www/html/pub/gluster/unpacking-script.sh series version os flavor latest_version latest_series | ||
|
There was a problem hiding this comment. Shouldn't it be "$version", etc ? |
||
|
|
||
| cd .. | ||
| function finish { | ||
| # cleanup code | ||
| echo "removing the chroot" | ||
| sudo rm -rf /var/cache/pbuilder/base.tgz | ||
|
|
||
| #removing folders created while packaging | ||
| rm -rf ~/${os}-${flavor}-Glusterfs-${version} | ||
| } | ||
| trap finish EXIT | ||
|
There was a problem hiding this comment. Traps should be placed at the start of the script. Otherwise, it be used only when the script is over, which is not useful. |
||
| trap finish SIGQUIT | ||
|
|
||
| echo "Done." | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use the label
debian10for this job. We have a machine existing on jenkins https://build.gluster.org/computer/builder-deb10-1.int.rht.gluster.org/There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@deepshikhaaa done!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@deepshikhaaa please validate the liburing-devel support as well.