invenio-devscripts repository contains a collection of scripts
useful when hacking on Invenio.
The following devscripts are suitable for Invenio 2.x based development environments.
Permits to bootstrap a fresh Invenio 2 development environment on a newly installed OS, say a Vagrant VM box. Use with care! Usually used only once.
Firstly, start a new, fresh, empty OS, such as Ubuntu/Trusty64 Vagrant virtual machine:
laptop> mkdir -p ~/private/vm/invenio2trusty64
laptop> cd ~/private/vm/invenio2trusty64
laptop> vim Vagrantfile # enter following content:
Vagrant.configure("2") do |config|
config.vm.box = "trusty64"
config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box"
config.vm.hostname = 'localhost.localdomain'
config.vm.network :forwarded_port, host: 8080, guest: 8080
config.vm.network :forwarded_port, host: 8443, guest: 8443
config.vm.provider :virtualbox do |vb|
vb.customize ["modifyvm", :id, "--memory", "2048"]
vb.customize ["modifyvm", :id, "--cpus", 2]
end
end
laptop> vagrant up
Secondly, connect to the new VM and launch Invenio kickstarter:
laptop> vagrant ssh vm> wget https://raw.githubusercontent.com/tiborsimko/invenio-devscripts/master/invenio2-kickstart vm> chmod u+x ./invenio2-kickstart vm> ./invenio2-kickstart --yes-i-know --yes-i-really-know
Thirdly, go brew some tee, come back in twenty minutes, enjoy!
laptop> firefox http://0.0.0.0:8080/
You now have Invenio 2 and its demo site installed, using some
default parameters suitable for Invenio developments. If you would
like to change some of these parameters, you can set some environment
variables before launching the kickstarter script; please see the
beginning of invenio2-kickstart script for advice.
The following devscripts are suitable mostly for Invenio 1.x based development environments.
- invenio-kickstart
- kick-start Invenio developments on a new OS (use with care!)
- invenio-setup-environment
- kick-start Invenio development environment (bashrc,/opt)
- invenio-setup-prerequisites
- kick-start Invenio prerequisites (Apache/MySQL/Python/etc)
- invenio-initialise-virtualenv
- initialise virtualenv for Invenio development
- invenio-recreate-demo-site
- recreate demo site
- invenio-retest-demo-site
- rerun test suite on recreated demo site
- invenio-make-install
- install (parts of) source tree and restart Invenio WSGI application
- invenio-check-kwalitee
- check code kwalitee of source files
- invenio-check-branch
- check code kwalitee changes between branches
- invenio-backup-site
- create backup file for the given Invenio site instance
- invenio-restore-site
- restore Invenio site instance from the backup file
- invenio-create-deploy-recipe
- create deploy recipes
- invenio-check-AUTHORS
- check whether all committers are well present in AUTHORS or THANKS files
- invenio-check-POTFILES
- check whether all sources files that contain I18N phrases are well present in POTFILES.in
- invenio-delete-old-git-tags
- delete old CVS-times tags from git repository
- invenio-code-browser
- generate and publish Invenio code browser pages
- invenio-symlink-installer
- GNU install drop-in replacement that could be used to symlink files instead of copying them.
Put the scripts somewhere under your PATH. For example, if you
clone this repository under $HOME/private/src:
cd $HOME/private/src git clone https://github.com/tiborsimko/invenio-devscripts.git
then in your $HOME/.bashrc you can say:
export PATH=$HOME/private/src/invenio-devscripts:$PATH
You will need to configure some of the following environment variables:
| Variable | Documentation | Default value |
| CFG_INVENIO_SRCDIR | where are Invenio sources? | ~/private/src/invenio |
| CFG_INVENIO_PREFIX | where is Invenio installed? | /opt/invenio |
| CFG_INVENIO_HOSTNAME | what is allowed hostname? | pcuds06 |
| CFG_INVENIO_DOMAINNAME | what is our domain name? | cern.ch |
| CFG_INVENIO_PORT_HTTP | what is our HTTP port number? | 80 |
| CFG_INVENIO_PORT_HTTPS | what is our HTTPS port number? | 443 |
| CFG_INVENIO_USER | under which user ID Invenio runs? | www-data |
| CFG_INVENIO_ADMIN | who is admin of this instance? | tibor.simko@cern.ch |
| CFG_INVENIO_DATABASE_NAME | what is database name? | invenio |
| CFG_INVENIO_DATABASE_USER | what is database user? | invenio |
| CFG_INVENIO_DATABASE_PASS | what is database user password? | my123p$ss |
| CFG_INVENIO_APACHECTL | how to restart Apache? | /etc/init.d/apache2 |
| CFG_INVENIO_MYSQLCTL | how to restart MySQL? | /etc/init.d/mysql |
| CFG_INVENIO_VIRTUALENVS | where is virtualenv home? | ~/.virtualenvs |
| CFG_INVENIO_SYMLINK_INSTALLER | install via symlinks? | 0 |
| CFG_INVENIO_SITE_LANGS | which UI languages to enable? | all; e.g. use “en,fr,it,de” to enable only some |
Here is a minimal example of what you can put in your $HOME/.bashrc:
export CFG_INVENIO_HOSTNAME=doc export CFG_INVENIO_DOMAINNAME=example.org export CFG_INVENIO_ADMIN=john.doe@example.org
Note that you can use invenio-kickstart helper script that will
populate $HOME/.bashrc for you with some sensible defaults. This is
useful especially for VM boxes.
Note that if you don’t set up CFG variables, you can still call devscripts “on the spot”, for example:
CFG_INVENIO_HOSTNAME=newbox CFG_INVENIO_USER=apache invenio-retest-demo-site --yes-i-know
If you have a newly installed OS where you would like to bootstrap Invenio development environment in a fully automated, unassisted manner, you can do:
invenio-kickstart --yes-i-know --yes-i-really-know
Some thirty minutes later, the box should sport a fully-functioning Invenio demo site.
This assumes that the user has full sudo rights on the given box.
See below for the full help on invenio-kickstart devscript.
If you have never installed Invenio before, you may want to do it manually the first time, as a learning exercise. Please follow pages such as Invenio on Debian.
Some devscripts, especially ones that recreate Invenio demo site for
you, further assume that you have certain sudo rights to execute
certain commands. Example for Debian GNU/Linux:
$ cat /etc/sudoers.d/johndoe
johndoe ALL=(www-data) NOPASSWD: ALL, \
(root) NOPASSWD: /bin/rm -rf /opt/invenio/var/tmp/ooffice-tmp-files, \
(root) NOPASSWD: /bin/mkdir -p /opt/invenio/var/tmp/ooffice-tmp-files, \
(root) NOPASSWD: /bin/chown -R nobody /opt/invenio/var/tmp/ooffice-tmp-files, \
(root) NOPASSWD: /bin/chmod -R 755 /opt/invenio/var/tmp/ooffice-tmp-files, \
(root) NOPASSWD: /etc/init.d/apache2, \
(root) NOPASSWD: /etc/init.d/mysql
Kick-starts Invenio developments on a newly installed OS, say a vagrant VM box. Use with care! Usually used only once.
Suppose you have a new VM where you would like to bootstrap Invenio development environment and set up the Invenio demo site in a fully automated, unassisted manner. The kickstarter allows you to achieve this via the following one-liner:
git clone https://github.com/tiborsimko/invenio-devscripts /tmp/invenio-devscripts /tmp/invenio-devscripts/invenio-kickstart --yes-i-know --yes-i-really-know
Please use with extreme care. Never on any production node!
The kickstarter assumes that the user has full sudo rights on the
given box:
$ cat /etc/sudoers.d/johndoe johndoe ALL=(ALL:ALL) NOPASSWD: ALL
The kick-starter will set up johndoe’s bashrc, set up /opt/invenio,
set up Python symlinks, set up Apache, MySQL, Python packages and
other pre-requisites, clone Invenio sources into ~/private/src
directory, install Invenio and instantiate ready-to-be-used Invenio
demo site. All in fully automated, unassisted way.
Usually tested on CentOS, Debian, FreeBSD, Scientific Linux, Ubuntu.
Note that you can later chain on with invenio-initialise-virtualenv
so that the same box could run maint, master and next branches
of Invenio. (This could be later added as a new option to the
kickstarter.)
Note that you can also pass a wanted branch value as an argument to
invenio-kickstart in order to install some wanted Invenio branch
development environment, such as maint-1.1. This is useful e.g. on
Scientific Linux 5 installations where only Invenio v1.1 or previous
release series are working. (Because SL5 uses Python-2.4.) So, in
order to install Invenio v1.1 based development environment on these
boxes, you should add maint-1.1 branch argument:
invenio-kickstart --yes-i-know --yes-i-really-know maint-1.1
Kick-starts Invenio development environment. Alters bashrc, sets up
/opt/invenio, sets up Python invenio symlinks. Usually called by
invenio-kickstart, never directly.
Kick-starts Invenio prerequisites’ installation and configuration.
Downloads and configures Apache, MySQL, Python packages, system etc.
Usually called by invenio-kickstart, never directly.
If you have installed Invenio on your box, e.g. by using
invenio-kickstart or by following InvenioOnDebian instructions, then
you may find it cumbersome to switch between various branches. Say
you are developing a feature based off the master branch and you
would like to quick-fix a bug occurred in the maint-1.1 branch, as
well as to check this behaviour on the brand new next branch, all
without having to recreate your Invenio demo sites.
The solution is to set up several virtualenv environments, each with
its own installation place and its own database, so that you can
quickly switch between them. The devscript
invenio-initialise-virtualenv will assist you in customising your
environment for Invenio developments.
Some prerequisites, starting out of a system installed in InvenioOnDebian manner:
sudo aptitude install virtualenvwrapper
rm -rf /opt/invenio # this will become symlink laterHere is how you can create a new virtualenv environment called
invenio-master (using system Apache and system Python packages) with
fresh new Invenio demo site on it:
# Invenio sources are cloned in ~/private/src/invenio
mkvirtualenv invenio-maint-1.0
invenio-initialise-virtualenv --yes-i-know
deactivate && workon invenio-maint-1.0
invenio-recreate-demo-site --yes-i-knowYou can use the above commands to up several virtualenv environments
named invenio-maint-1.0, invenio-maint-1.1, invenio-master,
invenio-next, each corresponding to the respective branch.
Here is how you can quickly switch between them:
workon invenio-maint-1.1 # browser will show we are running 'maint-1.1' site
workon invenio-next # browser will show we are running 'next' siteHere is typical hacking session with switching between environments:
## (1) let's start by working on some-new-feature-a in 'master' branch
workon invenio-master # switch to master virtualenv
git checkout -b some-new-feature-a master # start working on a feature
cd modules/webfoo/lib/
vim webfoo_templates.py # edit some files
invenio-make-install # install changes
firefox # check some-new-feature-a in browser
## (2) phone rings, there is a bug in Invenio v1.1.0 that we have to quick fix
git commit -a -m xxx # stash unsaved work on some-new-feature-a
workon invenio-maint-1.1 # switch to maint-1.1 virtualenv
git checkout -b fix-for-webbar maint-1.1 # start working on a bug fix for WebBar
cd modules/webbar/lib/
vim webbar_dblayer.py # edit some files
invenio-make-install # install changes
firefox # check whether WebBar is OK now
git commit -a -m 'WebBar: fix for baz' # commit the fix
## (3) a visitor enters and wonders about the brand new search facets from 'next' branch
workon invenio-next
firefox # will show 'next' demo site with facetsOnce you have installed Invenio for the first time on a box, you can
use this devscript to reinstall the Atlantis demo site anew. Please
beware, because invenio-recreate-demo-site will erase your database
tables and recreate your /opt/invenio anew.
Installing Invenio demo site from scratch requires having an Internet
connection and may take up to 15 minutes. For a quicker technique to
restore a vanilla Invenio demo site from a previously installed one,
please see invenio-backup-site and invenio-restore-site devscripts
below.
This devscript launches all unit/regression/web test suite cases on your installation, compares results against the last run, and warns you in case of differences. It is useful to see whether your branch did not accidentally break some tests. The script assumes running on the usual Atlantis demo site conditions, and may be destructive, so please beware.
This is the most frequently used devscript. Depending on where in the Invenio source tree it is called from, the devscript installs the current part of the source tree and restarts Invenio WSGI application. For example, when you are hacking on WebMessage Python sources files, you can deploy your changes to the running Invenio instance in no time. (Well, in 0.11 sec, on my box.)
Because this devscript is so frequently used, it is advantageous to
create shortcuts for it, for example a shell alias called mi (for
“make install”):
alias mi="$HOME/private/src/invenio-devscripts/invenio-make-install"
and a hot key for your preferred editor; an example for Emacs:
(defun tibor-invenio-make-install ()
"Launch invenio-make-install script on the current buffer."
(interactive)
(save-buffer)
(shell-command "~/private/src/invenio-devscripts/invenio-make-install"))
(global-set-key (kbd "C-c i") 'tibor-invenio-make-install)This allows you to press C-c i to install your edits.
Optional experimental component permitting the edition of files in the
source directory instead of using invenio-make-install devscript to
put them to the target place. Useful for developers who are not using
C-c i style of shortcuts mentioned above. (Say Eclipse/PyDev
instead of Emacs.)
If you have set CFG_INVENIO_SYMLINK_INSTALLER=1, note that the
./configure script will be executed with INSTALL=`which
invenio-symlink-installer`, which will cause the
invenio-symlink-installer script to be used instead of the
system-wide GNU install utility. Upon “make install” this script
will cause the symlinking of source files (directories are still
created in the regular way). Note that this will mean that if you
modify a file in the install tree, the file will be actually changed
in the source tree and vice-versa (so beware of doing “git checkout
other-branch” as this will change in place the content of the deployed
tree).
This devscript is also very frequently used. When hacking on say
webmessage_dblayer.py, calling this script will detect the most
common code kwalitee problems:
invenio-check-kwalitee --check-some webmessage_dblayer.py
Because this devscript is so frequently used, you may again want to
create a short shell alias for it, say kw (=for “kwalitee”):
alias kw="$HOME/private/src/invenio-devscripts/invenio-check-kwalitee --check-some"
and a hot key for your preferred editor; an example for Emacs:
(defun tibor-invenio-check-kwalitee ()
"Launch Invenio code kwalitee check on the current buffer."
(interactive)
(let* ((input-file (buffer-file-name (current-buffer)))
(command (concat "~/private/src/invenio-devscripts/invenio-check-kwalitee "
"--check-some "
input-file)))
(save-some-buffers (not compilation-ask-about-save) nil)
(if (and input-file (string-equal (file-name-extension input-file) "py"))
(compilation-start command)
(message "[ERROR] Cannot run Invenio code kwalitee check on non-Python buffers."))))
(global-set-key (kbd "C-c k") 'tibor-invenio-check-kwalitee)This allows you to press C-c k to see the list of potential code
kwalitee problems in your buffer and to press the usual C-x ` to
jump from one problematic location to the next in order to fix them.
If you work on a feature branch named say new-feature that stems from master and you modify plenty of files, add some new files, and delete some other files, then you may want to check how the overall code kwalitee changed in your branch with respect to master. You can run:
invenio-check-branch master new-featureThis devscript will perform kwalitee check on files that were modified in the new-feature branch when compared to the master branch and it will inform you of kwalitee report differences. The comparison disregards line numbers, so that typical code moving/adding/deleting situations are handled relatively nicely, even if the starting branch code is dirty. However, doing comparison in this “fuzzy” way may also leads to false positives, so beware. If/when we are free of kwalitee issues everywhere, we can do stricter comparison here.
You should run invenio-check-branch on your feature branches before
every merge request.
Creates backup file for the given Invenio site instance. Basically
dumps the current database content and creates tarball of every file
under /opt/invenio. The final backup file is named like
inveniomaint11-site-backup-2013-07-28-13-05-21.tar and can be
restored via invenio-restore-site devscript, see below.
Restores Invenio site instance from the backup file previously created
by invenio-backup-site devscript, see above. Basically, removes
everything under /opt/invenio, and recreates database tables from
the dump.
Creates deployment recipe out of an Invenio commit or a range of commits. Here are typical use cases:
invenio-create-deploy-recipe --cds
invenio-create-deploy-recipe --cds HEAD~10..
invenio-create-deploy-recipe --inspire --via-filecopy 48c7348..52fa18f
invenio-create-deploy-recipe --inspire HEAD,HEADNotes:
- The script understands CDS and INSPIRE site conditions as two
concrete site examples. You can use
--cdsor--inspirecommand line options to specify deployment on either CDS or INSPIRE. - The script takes
SHA1of the commit to deploy, orSHA1a..SHA1bcommit range to deploy. If this argument is missing, then it deploys the latest single commit on the current branch. - Note that the commit range may contain a comma – such as in the
fourth
HEAD,HEADexample above – in which case the commit range before the comma will be taken from the Invenio repository, and the commit range after the comma will be taken from the appropriate overlay repository (CDS or INSPIRE). In case the comma is not specified, the current repository is taken, either Invenio master repository or the INSPIRE overlay. - The generated recipe is
org-mode-formatted and is to be inspected by humans. E.g. in case of DB changes, the recipe will contain a warning at the end of the recipe, and a human is supposed provide appropriateALTER TABLEstatements and the like. - The generated recipe can use either file copy instructions
(
--via-filecopy) or autotools installation instructions (--via-install). You may preferably use the former in order to deploy small patches. - Note that the recipes may be generated on a machine that does not run the same destination overlay. E.g. one can generate Invenio deployment recipes for a remote INSPIRE service on a locally run Atlantis demo site. However, when generated INSPIRE overlay recipes, it is helpful to do so on an INSPIRE site, because the script may try to find location of some expected files.
Once the deployment recipe is proof-read by human, the generated code snippets can be copy-pasted onto appropriate worker nodes.
Note that this devscript constitutes a human-assisted semi-automatic deployment technique. This is mostly because of the bleeding edge nature of the master branch deployment that is seeked out here. If we would like to deploy release maintenance branches, then we could target more fully automated deployment mechanisms, via post-commit hooks or via Fabric.
Checks whether all git committers are well listed in the AUTHORS or THANKS file.
This is used mostly before making releases by people wearing system integration and release management hats.
Checks I18N usage in Invenio. Firstly, checks whether each file from
POTFILES.in exists and whether it contains _(. Secondly, checks
each Invenio source file that contains _( whether it is well present
in POTFILES.in.
This is used mostly before making releases or massive PO file updates by people wearing system integration and release management hats.
A helper script to delete old CVS-era tags from Invenio source code repository. Some developers still have the old tags present in their personal repositories, so when you fetch from them, the old CVS style tags may reappear from time to time. When this happens, running this script will delete them.
Generate and publish Invenio code browser pages. Assumes having
installed Invenio locally first. Call this script with
--generate-code-browser-pages CLI option and check its output in a
web browser. If everything looks fine, then publish generated code
browser pages on the code browser canonical web site by calling this
script with --publish-code-browser-pages CLI option. This script is
used from time to time by the Head Developer.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, see http://www.gnu.org/licenses/.