cms-2016-pileup-dataset: scripts needed for pileup dataset record

.gitignore

@@ -74,3 +74,8 @@ cms-2016-collision-datasets/inputs/hlt-config-store
 cms-2016-collision-datasets/inputs/das-json-store
 cms-2016-collision-datasets/inputs/das-json-config-store
 cms-2016-collision-datasets/outputs/*.json
+cms-2016-pileup-dataset/inputs/config-store
+cms-2016-pileup-dataset/inputs/das-json-store
+cms-2016-pileup-dataset/inputs/mcm-store
+cms-2016-pileup-dataset/outputs/
+cms-2016-pileup-dataset/cookies.txt

README.rst

@@ -50,6 +50,7 @@ Specific data ingestion and curation campaigns:
 - `cms-2015-collision-datasets-hi-ppref <cms-2015-collision-datasets-hi-ppref>`_ - helper scripts for CMS 2015 heavy ion release (proton-proton reference collision datasets)
 - `cms-2015-simulated-datasets <cms-2015-simulated-datasets>`_ -- helper scripts for the CMS 2015 open data release (simulated datasets)
 - `cms-2016-collision-datasets <cms-2016-collision-datasets>`_ -- helper scripts for the CMS 2016 open data release (collision datasets)
+- `cms-2016-pileup-dataset <cms-2016-pileup-dataset>`_ -- helper scripts for the CMS 2016 open data release (pileup dataset)
 - `cms-2016-simulated-datasets <cms-2016-simulated-datasets>`_ -- helper scripts for the CMS 2016 open data release (simulated datasets)
 - `cms-YYYY-luminosity <cms-YYYY-luminosity>`_ -- helper scripts for the CMS luminosity information records (any year)
 - `cms-YYYY-run-numbers <cms-YYYY-run-numbers>`_ -- helper scripts for enriching CMS dataset run numbers (any year)

cms-2016-pileup-dataset/README.md

@@ -0,0 +1,73 @@
+# cms-2016-pileup-dataset
+
+This directory contains helper scripts used to prepare CMS 2016 open data
+release regarding pile-up dataset.
+
+- `code/` folder contains the python code;
+- `inputs/` folder contains input text files with the list of datasets for each
+  year and input files;
+- `outputs/` folder contains generated JSON records to be included as the CERN
+  Open Data portal fixtures.
+
+Every step necessary to produce the final `*.json` files is handled by the
+`code/interface.py` script. Details about it can be queried with the command:
+
+```console
+$ python3 code/interface.py --help
+```
+
+Please make sure to get the VOMS proxy file before running these scripts:
+
+```console
+$ voms-proxy-init --voms cms --rfc --valid 190:00
+```
+
+Please make sure to set the EOS instance to EOSPUBLIC before running these scripts:
+
+```console
+$ export EOS_MGM_URL=root://eospublic.cern.ch
+```
+Please make sure to have a valid `userkey.nodes.pem` certificate present in
+`$HOME/.globus`. If not, you have to run the following on top of the regular
+CMS certificate documentation:
+
+```console
+$ cd $HOME/.globus
+$ ls userkey.nodes.pem
+$ openssl pkcs12 -in myCert.p12 -nocerts -nodes -out userkey.nodes.pem  # if not present
+$ cd -
+```
+
+First step is to create EOS file index cache:
+
+```console
+$ python3 ./code/interface.py --create-eos-indexes inputs/CMS-2016-premix.txt
+```
+
+This requires the index data files to be placed in their final location. However, for
+early testing on LXPLUS, all steps can be run without the EOS file index cache
+by means of adding the command-line option `--ignore-eos-store` to the commands below.
+
+We can now build sample records by doing:
+
+```console
+$ python3 ./code/interface.py --create-das-json-store --ignore-eos-store inputs/CMS-2016-pileup-dataset.txt
+
+$ auth-get-sso-cookie -u  https://cms-pdmv.cern.ch/mcm -o cookies.txt
+$ python3 ./code/interface.py --create-mcm-store --ignore-eos-store inputs/CMS-2016-pileup-dataset.txt
+
+$ python3 ./code/interface.py --get-conf-files --ignore-eos-store inputs/CMS-2016-pileup-dataset.txt
+
+$ python3 ./code/interface.py --create-records --ignore-eos-store inputs/CMS-2016-premix.txt
+```
+
+Each step builds a subdirectory with a cache (`das-json-store`, `mcm-store` and
+`config-store`). They are large, do not upload them to the repository, respect
+the `.gitignore`.
+
+The output JSON files for the dataset records will be generated in the
+`outputs` directory.
+
+The three configuration files from `./inputs/config-store` are to be copied to
+`/eos/opendata/cms/configuration-files/MonteCarlo2016/`. Don't forget to add
+the `*.py` extension.