Updating the readme, and fixing the commit flag for insert_new.

Author: SimonGoring

.gitignore

@@ -32,3 +32,23 @@ helpers/archives/
 helpers/figshareUpload/settings.yaml
 
 helpers/figshareUpload/lib/__pycache__/
+
+helpers/.env
+
+Proposals/publications/.env
+
+Proposals/publications/data/neotoma_citations_202410070925.csv
+
+Proposals/publications/data/neotoma_publications_202410151545.csv
+
+Proposals/publications/json_pubs.json
+
+Proposals/publications/output.csv
+
+*.csv#
+
+Proposals/speleothem_data/.env
+
+Proposals/publications/src/publications/__pycache__/
+
+Proposals/publications/data/raw_doi.csv

Proposals/publications/README.md

@@ -4,12 +4,20 @@ This project is intended to help add DOIs to existing publications, support meta
 
 ## Updating Existing Neotoma Records
 
+These scripts are run (for the most part) directly against the Neotoma Database. Because of this, we use an `.env` file when running the scripts. The `.env` file should contain the connection string to the database, as a JSON object, related to `DBAUTH`. For example:
+
+```env
+DBAUTH={"host":"localhost","port":5555,"user":"PUBLICATIONGUY","password":"PUBLICATIONSCOOL","database":"neotoma"}
+```
+
+Without this connection string the scripts will not work properly. For development work, please use `neotomatank` as the database for connection.
+
 ### Finding new DOIs
 
 For existing publications that do not include DOIs we can scan the Neotoma Publications database from the commandline:
 
-```python
-uv run src/find_potential_dois.py --limit 100 --skip 100 --output ./data/offset100.csv
+```bash
+uv run src/find_potential_dois.py --limit 30000 --output ./data/offset100.csv
 ```
 
 This will return a CSV file (saved in the `--output` directory) with the Neotoma `publicationid`, current `citation`, the `doi` stored in Neotoma (generally empty) and then columns for the `newdoi`, obtained from a CrossRef search, as well as the `bibtex` citation.
@@ -32,9 +40,41 @@ The `--commit` flag allows us to test the run, to ensure that we don't accidenta
 
 Otherwise the upload will end with the statement:
 
-```
+```bash
 The --commit flag was set to False, rolling back operation.
 ```
 
 ## Inserting New Publicaitons from DOIs
 
+To facilitate bulk uploads of records we can use a file with raw DOI strings and use CrossRef to resolve the citation for us. The functions we use are in the [`publications`](./src/publications/) folder, both [`return_bibtex.py`](./src/publications/return_bibtex.py) -- which takes the DOI and returns a formatted BibTex citation -- and [`add_citation`](./src/publications/add_citation.py), which takes the BibTex citation and formats it using APA style.
+
+Given a text file with DOIs and, potentially empty spaces (to support simply copying a column from a spreadsheet), we process each unique entry.
+
+```csv
+10.1017/S0033822200001089
+
+10.1017/S0033822200020452
+10.1017/S0033822200001089
+10.1017/S0033822200001089
+10.1017/S0033822200001089
+10.1080/0734578X.2017.1377510
+```
+
+Using the script:
+
+```bash
+uv run insert_new_from_doi.py --input FILEPATH.csv
+```
+
+The script will parse individual DOIs and dry-run insertion. To insert the records into the database, include the `--commit` flag. If `--commit` is set, you will see the output:
+
+```text
+Committing the following citation:
+Emslie, S. D., & Mead, J. I. (2020 , August). The age and vertebrate paleontology of labor-of-love cave, white pine county, nevada. Western North American Naturalist, 80(3). URL: http://dx.doi.org/10.3398/064.080.0301, doi:10.3398/064.080.0301
+```
+
+for each record submitted. Note here that capitalization is often inconsistent. Most publishers on CrossRef do not properly capitalize records, and as such, any automated system will do a poor job of returning properly capitalized records.
+
+## Conclusion
+
+The scripts together support the management of publication records within Neotoma.

Proposals/publications/src/insert_new_from_doi.py

@@ -7,10 +7,10 @@
 
 parser = argparse.ArgumentParser()
 parser.add_argument('--input', '-I', help="A valid output filename.", type= str, default = 'output.csv')
-parser.add_argument('--doi', '-d', help="Which column contains the DOI for upload?", type=str)
-parser.add_argument('--commit', '-c', help="Should we commit the data to the database?", type=bool, default= False)
+parser.add_argument('--commit', '-c', help="Should we commit the data to the database? (exclude flag to avoid committing)", type=bool, default= False)
 
 args = parser.parse_args()
+print(args)
 
 dotenv.load_dotenv()
 
@@ -29,14 +29,13 @@
     citation = add_citation(bibtex)
     if bibtex is not None and citation is not None:
         with conn.cursor() as cur:
-                cur.execute(QUERY, {'doi': i,
-                                    'bibtex': bibtex,
-                                    'citation': citation})
-                if args.commit:
-                    print(f'Committing the following citation:\n{citation}.')
-                    conn.commit()
-                cur.close()
-
+            cur.execute(QUERY, {'doi': i,
+                                'bibtex': bibtex,
+                                'citation': citation})
+            if args.commit is True:
+                print(f'Committing the following citation:\n{citation}.')
+                conn.commit()
+            cur.close()
 if args.commit is True:
      conn.commit()
 else:

Proposals/publications/src/publications/add_citation.py

@@ -29,4 +29,5 @@ def add_citation(bibtex:str) -> str:
         else:
             return [entry.text.render_as('text') for entry in formattedBib][0]
     except Exception as e:
-        
+        print(e)
+        return ''