Scripts – Data curation and processing logic for the Swedish Parliament Corpus

General setup and use

Setting up an environment

Set up a conda environment : Follow the steps here.

With the environment active, install the pyriksdagen module, either from PyPi

pip install pyriksdagen

or from a local copy in the pyriksdagen repo

pip install .

The LazyArchive

The LazyArchive() class attempts to connect to the KB labs in the lazyest way possible. If you'll use the scripts often, it's worthwhile to set 3 environment variables:

KBLMYLAB=https://betalab.kb.se
KBLUSER=
KBLPASS=

They can be added to the environment variables, e.g. ~/miniconda3/envs/tf/etc/conda/activate.d/env_vars.sh. If these are not present, you will be prompted for the username and password.

Curating data

Most scripts take --start YEAR and --end YEAR arguments to define a span of time to operate on. Other options are noted in with the file below.

-1. Create new curation branch from dev.

git checkout -b curation-<decade_start_year>s dev

1. Generate an input csv by querying protocol packages using scripts/query2csv.py

   • this creates input/protocols/scanned.csv or input/protocols/digital_originals.csv, to be read by scripts/pipeline.py
   • with the -m option the script will create year directories in corpus/protocols/ if they don't already exist
   • obs., unlike the other scripts use of --start and --end to define a range of dates is exclusive of the end year – updated to behave like the other scripts – obs. 2, a potential problem is that this doesn't handle the two-year formats - 199495

2. Compile parlaclarin for years queried in (1) with scripts/pipeline.py – make sure input/raw/ exists.

3. Look for introductions with scripts/classify_intros.py

   • this creates input/segmentation/intros.csv
   • had to add /home/bob/miniconda3/envs/tf/lib/python3.9/site-packages/nvidia/cublas/lib/ to $LD_LIBRARY_PATH

4. Run scripts/resegment.py to segment and label introductions in corpus/protocols/<year>/*.xml files

5. Run scripts/add_uuid.py to make sure any new segments have a uuid.

6. Run scripts/find_dates.py to find marginal notes with dates and add dates to metadata.

7. Run scripts/build_classifier.py (the classifier doesn't need to be built every time) different args!?

   • --datapath : needs a file currently at input/curation/classifier_data.csv (but how is this file generated? it's a mystery... it just exists)
   • --epochs (can use the default)
   • writes to the segment-classifier/ ... how does it relate to years of protocols? it doesn't – it's apparently trained generally and scripts/reclassify.py allows to specify which years are operated on

8. Run scripts/reclassify.py to reclassify utterances and notes

   • nb. build_classifier writes to segment-classifier/, but this reads from input/segment-classifier/, so the output needs to be moved, or we can fix the discrepancy
   • do this one year at a time for dolan's sakie
     • for year in {START..END}; do python scripts/reclassify.py -s $year -e $year; done

9. Run add_uuid.py again.

10. Run scripts/dollar_sign_replace.py to replace dollar signs.

11. Run scripts/fix_capitalized_dashes.py.

12. Run scripts/wikidata_process.py (makes metadata available for redetect.py)

13. Run scripts/redetect.py.

14. Run scripts/split_into_sections.py.

Quality Control

1. generate a sample for by decade with sample_pages_new.py.

   • This generates a csv file in input/quality_control/sample_<decade-start-year>.csv and a list of protocols in the sample input/quality_control/sample_<decade-start-year>.txt

2. Add (git-add_QC-sample.sh for the lazy) and commit the sample to working branch.

3. Populate the quality control csv file with populate-QC-sample-test.py

   • sample protocols need to be on the local machine where the script is run. Since it pops open protocols in github an originals in betalab in a browser, this script doesn't play nice with working over ssh
   • QC should distinguish between the same segment classes that scripts/reclassify.py produces and . Other classes may become relevant later.

4. Does data pass QC test? If yes, add and push the rest of the protocols.