Manual Build Of RPMs For Big PanDA - CERN Twiki

Manual build of RPMs for Big PanDA

• Introduction
• Manual build of ATLAS BigPanDAmon RPMs
  • Manual build of bigpandamon-atlas
    • Automatic bigpandamon-atlas RPM build
      • Automatic bigpandamon-atlas RPM build - trunk
      • Automatic bigpandamon-atlas RPM build - branches/devel
    • OBSOLETE - Manual RPM build for bigpandamon-atlas
    • Update RPM configs in the devel branch
    • Merge devel branch to trunk
      • Merge troubleshooting
    • Create tag of bigpandamon-atlas-0.0.4
    • Build RPM
    • Copy RPMs to the repository
  • Manual build of bigpandamon-core for ATLAS
    • Update RPM configs in the devel branch
    • Merge devel branch to trunk
    • Create tag of bigpandamon-core-0.0.4
    • Build RPM
      • Automatic bigpandamon-core RPM build
        • Automatic bigpandamon-core RPM build - trunk
        • Automatic bigpandamon-core RPM build - branches/devel
      • OBSOLETE - Manual RPM build for bigpandamon-core
    • Copy RPMs to the repository
  • Update RPM repos
  • Deployment to the integration/production machines
    • Deployment to the integration machine
    • Deployment to the production machine

Introduction

• This page is evolving. Please keep checking.
• This page contains information about manual build of RPMs for BigPanDA. Keep checking BigPanDARPMbuildSystem.
• Feel free to contact JaroslavaSchovancova and TorreWenaus with questions.

Manual build of ATLAS BigPanDAmon RPMs

• Let us assume that we need to build both bigpandamon-atlas and bigpandamon-core RPMs.

Manual build of bigpandamon-atlas

Automatic bigpandamon-atlas RPM build

Automatic bigpandamon-atlas RPM build - trunk

• Upon every commit to https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon-atlas an RPM is built.
• Trunk RPM is available https://pandabuild.cern.ch/x86_64/ as bigpandamon-atlas-<stable version>-1.noarch.rpm
  • <stable version> is the most recent (current, new) stable version of the bigpandamon-atlas package
• Naming convention steering for trunk: in trunk/bigpandamon-atlas/setup.cfg
  • version = VALUE, where VALUE is the new stable version number
  • release_type = stable
• Naming convention steering for devel: in devel/bigpandamon-atlas/setup.cfg
  • version = VALUE, where VALUE is the newest stable version number
  • release_type = dev

Automatic bigpandamon-atlas RPM build - branches/devel

• Upon every commit to https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/branches/devel/bigpandamon-atlas an RPM is built.
• Devel RPM is available https://pandabuild.cern.ch/unstable/ as bigpandamon-atlas-<stable version>.dev_r<svn revision of the commit>-1.noarch.rpm
  • <stable version> is the most recent stable version of the bigpandamon-atlas package
  • <svn revision of the commit> is the revision number of the latest commit into branches/devel of bigpandamon-atlas.
• Naming convention steering: in devel/bigpandamon-atlas/setup.cfg
  • version = VALUE, where VALUE is the latest stable version number
  • release_type = dev

OBSOLETE - Manual RPM build for bigpandamon-atlas

Update RPM configs in the devel branch

• Check the latest version available, do not forget to increment version string.
  • https://pandabuild.cern.ch/x86_64/
  • https://pandabuild.cern.ch/SRPMS/
• cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas # cd panda-bigmon/trunk/bigpandamon-atlas
• edit file branches/devel/bigpandamon-atlas/setup.cfg
  • update "requires" if necessary # svn diff setup.cfg Index: setup.cfg =================================================================== --- setup.cfg (revision 18781) +++ setup.cfg (working copy) @@ -4,5 +4,6 @@ provides = bigpandamon-atlas release = 1 packager = Panda Team <hn-atlas-panda-pathena@cern.ch> -requires = python bigpandamon-core +requires = python + bigpandamon-core >= 0.0.4
• edit file branches/devel/bigpandamon-atlas/setup.py
  • update/check whatever is marked with #FIXME, i.e. release_version, expected_extensions, packages, data_files
• commit changes to RPM configs # svn ci -m "bigpandamon-atlas/branches/devel: prepared RPM setup."

Merge devel branch to trunk

• cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas # cd panda-bigmon/trunk/bigpandamon-atlas
• update local copy of trunk of bigpandamon-atlas # svn up
• merge branches/devel/bigpandamon-atlas branch to trunk/bigpandamon-atlas
  • postpone conflicts for setup.py and setup.cfg # svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-atlas --- Merging differences between repository URLs into '.': U atlas/prodtask/views.py U atlas/prodtask/request_views.py U atlas/prodtask/urls.py U atlas/prodtask/forms.py U atlas/prodtask/templates/prodtask/_index.html U atlas/prodtask/templates/prodtask/_reqdatatable.html C setup.cfg C setup.py
  • resolve conflicts for setup.py and setup.cfg # svn resolve --accept theirs-full setup.cfg Resolved conflicted state of 'setup.cfg' # svn resolve --accept theirs-full setup.py Resolved conflicted state of 'setup.py'
• now go through changes before committing the merge, and adapt what is necessary, resolve collisions etc.
  • diff from svnweb can be useful as an additional check: e.g. https://svnweb.cern.ch/trac/panda/changeset?new=18778%40panda-bigmon%2Fbranches%2Fdevel&old=18772%40panda-bigmon%2Ftrunk
• commit the merge # svn ci -m "Merged devel/bigpandamon-atlas to trunk/bigpandamon-atlas, RC for bigpandamon-atlas-0.0.4"

Merge troubleshooting

• If someone accidentally committed into trunk instead of branches/devel, you may get error message such as # svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-atlas svn: Reintegrate can only be used if revisions 18773 through 19007 were previously merged from svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas to the reintegrate source, but this is not the case: panda-bigmon/branches/devel/bigpandamon-atlas/atlas Missing ranges: /panda-bigmon/trunk/bigpandamon-atlas/atlas:18773-18862

The resolution of this issue is described in http://stackoverflow.com/a/2648969:

• • go to trunk/bigpandamon-atlas and perform svn propdel svn:mergeinfo -R svn revert . svn ci -m "Removed mergeinfo"
  • go to branches/devel/bigpandamon-atlas and perform svn propdel svn:mergeinfo -R svn revert . svn ci -m "Removed mergeinfo"
  • go back to trunk/bigpandamon-atlas and restart the merge.

Create tag of bigpandamon-atlas-0.0.4

• cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas # cd panda-bigmon/trunk/bigpandamon-atlas
• create a new tag svn copy \ svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas \ svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-atlas/bigpandamon-atlas-0.0.4 \ -m "bigpandamon-atlas-0.0.4"

Build RPM

• login to aipanda004 as jschovan/sbelov # [lxvoadm]# ssh jschovan@aipanda004
• cd to a workdir # MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR" # mkdir -p $MYWORKDIR # cd $MYWORKDIR
• prepare .rpmmacros # echo "%_unpackaged_files_terminate_build 0" >> ~/.rpmmacros
• checkout the tagged code # svn co svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-atlas/bigpandamon-atlas-0.0.4
• cd to the tagged directory # cd $MYWORKDIR/bigpandamon-atlas-0.0.4 # ls $MYWORKDIR/bigpandamon-atlas-0.0.4 atlas INSTALL.txt MANIFEST.in README.txt setup.cfg setup.py
• build RPM of bigpandamon-atlas-0.0.4 # QA_SKIP_BUILD_ROOT=1 python setup.py bdist_rpm ... lots of output, and then ... moving build/bdist.linux-x86_64/rpm/SRPMS/bigpandamon-atlas-0.0.4-1.src.rpm -> dist moving build/bdist.linux-x86_64/rpm/RPMS/noarch/bigpandamon-atlas-0.0.4-1.noarch.rpm -> dist # ls $MYWORKDIR/bigpandamon-atlas-0.0.4 atlas build dist INSTALL.txt MANIFEST MANIFEST.in README.txt setup.cfg setup.py # ls $MYWORKDIR/bigpandamon-atlas-0.0.4/dist -1 bigpandamon-atlas-0.0.4-1.noarch.rpm bigpandamon-atlas-0.0.4-1.src.rpm bigpandamon-atlas-0.0.4.tar.gz
• check files in the noarch RPM
  • make sure that none of the python modules is missing (make sure every python module has init.py committed in SVN)
  • make sure that none of the templates files is missing
  • make sure that the static files are not missing # rpm -qpl dist/bigpandamon-atlas-0.0.4-1.noarch.rpm | less
• remove .rpmmacros # rm ~/.rpmmacros

Copy RPMs to the repository

• sudo su on aipanda004
• cd to the dist directory # MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR" # cd $MYWORKDIR/bigpandamon-atlas-0.0.4/dist
• cp src.rpm to /data/build/SRPMS # cp bigpandamon-atlas-0.0.4-1.src.rpm /data/build/SRPMS/
• cp noarch.rpm to /data/build/x86_64 # cp bigpandamon-atlas-0.0.4-1.noarch.rpm /data/build/x86_64

Manual build of bigpandamon-core for ATLAS

Update RPM configs in the devel branch

• Check the latest version available, do not forget to increment version string.
  • https://pandabuild.cern.ch/x86_64/
  • https://pandabuild.cern.ch/SRPMS/
• cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core # cd panda-bigmon/trunk/bigpandamon-core
• edit file branches/devel/bigpandamon-core/setup.py
  • update/check whatever is marked with #FIXME, i.e. release_version, expected_extensions, packages, data_files
• commit changes to RPM configs # svn ci -m "bigpandamon-core/branches/devel: prepared RPM setup."

Merge devel branch to trunk

• cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core # cd panda-bigmon/trunk/bigpandamon-core
• update local copy of trunk of bigpandamon-core # svn up
• merge branches/devel/bigpandamon-core branch to trunk/bigpandamon-core
  • postpone conflicts for setup.py and setup.cfg # svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-core --- Merging differences between repository URLs into '.': ... updates ... C setup.py
  • resolve conflicts for setup.py (and setup.cfg) # svn resolve --accept theirs-full setup.py Resolved conflicted state of 'setup.py'
• now go through changes before committing the merge, and adapt what is necessary, resolve collisions etc.
  • diff from svnweb can be useful as an additional check: e.g. https://svnweb.cern.ch/trac/panda/changeset?new=18778%40panda-bigmon%2Fbranches%2Fdevel&old=18772%40panda-bigmon%2Ftrunk
• commit the merge # svn ci -m "Merged devel/bigpandamon-core to trunk/bigpandamon-core, RC for bigpandamon-core-0.0.4"

Create tag of bigpandamon-core-0.0.4

• cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core # cd panda-bigmon/trunk/bigpandamon-core
• create a new tag svn copy \ svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core \ svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-core/bigpandamon-core-0.0.4 \ -m "bigpandamon-core-0.0.4"

Build RPM

Automatic bigpandamon-core RPM build

Automatic bigpandamon-core RPM build - trunk

• Upon every commit to https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon-core an RPM is built.
• Trunk RPM is available https://pandabuild.cern.ch/x86_64/ as bigpandamon-core-<stable version>-1.noarch.rpm
  • <stable version> is the most recent (current, new) stable version of the bigpandamon-core package
• Naming convention steering for trunk: in trunk/bigpandamon-core/setup.cfg
  • version = VALUE, where VALUE is the new stable version number
  • release_type = stable
• Naming convention steering for devel: in devel/bigpandamon-core/setup.cfg
  • version = VALUE, where VALUE is the newest stable version number
  • release_type = dev

Automatic bigpandamon-core RPM build - branches/devel

• Upon every commit to https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/branches/devel/bigpandamon-core an RPM is built.
• Devel RPM is available https://pandabuild.cern.ch/unstable/ as bigpandamon-core-<stable version>.dev_r<svn revision of the commit>-1.noarch.rpm
  • <stable version> is the most recent stable version of the bigpandamon-core package
  • <svn revision of the commit> is the revision number of the latest commit into branches/devel of bigpandamon-core.
• Naming convention steering: in devel/bigpandamon-core/setup.cfg
  • version = VALUE, where VALUE is the latest stable version number
  • release_type = dev

OBSOLETE - Manual RPM build for bigpandamon-core

• login to aipanda004 as jschovan/sbelov # [lxvoadm]# ssh jschovan@aipanda004
• cd to a workdir # MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR" # mkdir -p $MYWORKDIR # cd $MYWORKDIR
• prepare .rpmmacros # echo "%_unpackaged_files_terminate_build 0" >> ~/.rpmmacros
• checkout the tagged code # svn co svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-core/bigpandamon-core-0.0.4
• cd to the tagged directory # cd $MYWORKDIR/bigpandamon-core-0.0.4 # ls $MYWORKDIR/bigpandamon-core-0.0.4 core INSTALL.txt MANIFEST.in README.txt setup.cfg setup.py
• build RPM of bigpandamon-core-0.0.4 # QA_SKIP_BUILD_ROOT=1 python setup.py bdist_rpm ... lots of output, and then ... moving build/bdist.linux-x86_64/rpm/SRPMS/bigpandamon-core-0.0.4-1.src.rpm -> dist moving build/bdist.linux-x86_64/rpm/RPMS/noarch/bigpandamon-core-0.0.4-1.noarch.rpm -> dist # ls $MYWORKDIR/bigpandamon-core-0.0.4 core build dist INSTALL.txt MANIFEST MANIFEST.in README.txt setup.cfg setup.py # ls $MYWORKDIR/bigpandamon-core-0.0.4/dist -1 bigpandamon-core-0.0.4-1.noarch.rpm bigpandamon-core-0.0.4-1.src.rpm bigpandamon-core-0.0.4.tar.gz
• check files in the noarch RPM
  • make sure that none of the python modules is missing (make sure every python module has init.py committed in SVN)
  • make sure that none of the templates files is missing
  • make sure that the static files are not missing # rpm -qpl dist/bigpandamon-core-0.0.4-1.noarch.rpm | less
• remove .rpmmacros # rm ~/.rpmmacros

Copy RPMs to the repository

• sudo su on aipanda004
• cd to the dist directory # MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR" # cd $MYWORKDIR/bigpandamon-core-0.0.4/dist
• cp src.rpm to /data/build/SRPMS # cp bigpandamon-core-0.0.4-1.src.rpm /data/build/SRPMS/
• cp noarch.rpm to /data/build/x86_64 # cp bigpandamon-core-0.0.4-1.noarch.rpm /data/build/x86_64

Update RPM repos

• sudo su on aipanda004 (the pandabuild.cern.ch machine)

• cd to dir with repobuild scripts # cd /data/build-not-public
• run the repobuild scripts # ./repo_SRPMS # ./repo_x86_64

Deployment to the integration/production machines

Deployment to the integration machine

• sudo su on aipanda043 - integration
• to be continued...

Deployment to the production machine

• sudo su on aipanda022 - production # yum clean all # yum upgrade
• make sure local.py is in place and is correct # /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/core/common/settings/local.py # /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/atlas/settings/local.py
• activate the virtualenv
  • make sure "pip freeze" contains all the virtualenv dependencies # cd /data/virtualenv/django1.6.1__python2.6.6__atlas # source bin/activate # export PYTHONPATH=/data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath:$PYTHONPATH
• try out ./manage.py runserver # cd /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/atlas # python manage.py runserver
• run ./manage.py collectstatic # python manage.py collectstatic
• restart the httpd service and change the owner of the logs files to apache:apache # service httpd restart # chown apache:apache -R /data/atlpan/bigpandamon/var/log/* /data/bigpandamon_virtualhosts/atlas/logs/* /data/bigpandamon_virtualhosts/jedimon/logs/*

Major updates: -- JaroslavaSchovancova - 24 Apr 2014

Responsible: JaroslavaSchovancova