NB make for both postgres, dev and master sqlite a user ‘visitor@visitor.org’, visitor
Postdoctoral research fellow University Centre for Rural Health, Sydney Medical School, University of Sydney, Sydney, NSW, Australia
And
Data Scientist Spatial Epidemiology Group, Centre for Research and Action in Public Health, Health Research Institute, University of Canberra, Canberra, ACT, Australia
instruction to go into project folder and double click the w2p.cmd you can go to events page, at bottom export to csv
- 2016-03-17: Events contributed by Dr Salimi, UTAS, using Satellite-Only protocol and a variation of APHEA imputation procedures.
fix postgres tables. Try this with backup tables, create shell then insert data.? or can this happen without insert record?
DONE source is compulsory, authors is optional DONE title compulsory DONE volume optional DONE general location means geographical location? optional? remove this. DONE Desirable. This can be URL, doi or weblink of any kind (and date accessed). Also you might want to add the folder location on your computer and the file name. DONE optional summary (this is what you think of it) DONE optional abstract (this is the author thought of it) DONE spell it out somewhere that you can sort descending by clicking ID (twice) DONE then the ‘biomass smoke events’ link is where to go to Add an Event.
Adding events TODO how to add a new place? DONE - place add an Other option DONE - then under that is a free text place.
WONTDO min date, start date? WONTDO max date, end date? no this is ok
make it clear what is compulsory
DONE final step is to email Ivan. and if not that is fine but the expectation is to cite the data (the github)
this becomes dot 5, dot 4 becomes manuscript flesh out dot say step 1 register step 2 add a reference [at this link] step 3 add dates [at this link]
make changes to the protocols page, bring in sync with manuscript, and the manuscript link is to an old pdf in /static
a) Johnston, b) Farhad, c) bare minimun and d) suggest their own bare min is just step 1 get any reference step 2 get as much bibliographical as possible e.g. author, title, publisher, year, URL, date Accessed step 3 go to web2py data entry form
---------- Forwarded message --------- From: Murray.Turner <Murray.Turner@canberra.edu.au> Date: Tue, Jun 26, 2018 at 4:13 PM Subject: RE: Biomass smoke events database To: Ivan.Hanigan <Ivan.Hanigan@canberra.edu.au> Cc: ivan.hanigan@sydney.edu.au <ivan.hanigan@sydney.edu.au>
Hi Ivan,
I’ve finished editing the manuscript using the “Fire” journal template.
Copy attached.
Can you cast your eye over it and suggest any changes?
I’ve highlighted your contact details, which you may/may not wish to update.
I will also email you an updated EndNote Library that could be sent with the manuscript as an attachment.
Don’t forget, if it is accepted and we do need to pay an article processing fee, I can get the money from the ANDS grant.
Cheers,
Murray
From: Ivan.Hanigan Sent: Wednesday, 23 May 2018 9:00 AM To: Murray.Turner <Murray.Turner@canberra.edu.au> Cc: ivan.hanigan@sydney.edu.au Subject: Re: Biomass smoke events database
Hi Murray,
I hope you are well?
I have changed jobs since last time we spoke and now work full time for Uni Sydney with a visiting fellowship at Uni Canberra (so I still live in ACT, but have to do a lot of travel).
I kept meaning to submit the biomass smoke database paper but felt it just does not really fit the Sci Reports mold very well.
I wonder if you have time and are willing to revise the manuscript for the journal “Fire”?
http://www.mdpi.com/journal/fire
It is more suitable for the paper, is open access and has ‘free publication for well-prepared manuscripts submitted in 2017.’
http://www.mdpi.com/journal/fire/instructions
We can submit this as a “Data Descriptor and Technical Notes: containing a description of a data set, including methods used for collecting or producing the data, where the data set may be found, and information about its use. Technical notes can include detailed descriptions of equipment operations, field methods, models, and monitoring method’s.”
Sorry to burden you but I just don’t seem to be able to make the time.
Kind regards,
Ivan
PS my new contact details:
Ivan Hanigan PhD
Data Scientist (Epidemiology)
University Centre for Rural Health
School of Public Health
University of Sydney
Ph: 0428 265 976
Email: ivan.hanigan@sydney.edu.au
From: Murray.Turner Sent: Friday, 4 August 2017 12:44 PM To: Ivan.Hanigan; Fay Johnston; Geoff.Morgan@ucrh.edu.au; david.bowman@utas.edu.au; grant.williamson@utas.edu.au; Farhad Salimi; sarah.henderson@ubc.ca Subject: FW: Biomass smoke events database
Hi Ivan,
I have read through the manuscript and made a few suggestions based on the Nature Scientific Reports submission checklist .
I have also written the opening paragraph for the Discussion section. I hope it’s okay – very happy to change it if I’m off track.
Regards,
Murray
From: Ivan.Hanigan Sent: Tuesday, 1 August 2017 12:07 PM To: Murray.Turner; Fay Johnston; Geoff.Morgan@ucrh.edu.au; david.bowman@utas.edu.au; grant.williamson@utas.edu.au; Farhad Salimi; sarah.henderson@ubc.ca Subject: Biomass smoke events database
Hi all,
What do you think about putting this revised manuscript in to Nature Scientific Reports?
https://www.nature.com/srep/publish/guidelines
Can you please let me know in the next week or so?
Thanks Ivan.
From: Ivan.Hanigan Sent: Saturday, 8 July 2017 11:27 AM To: Fay Johnston; Geoff.Morgan@ucrh.edu.au; david.bowman@utas.edu.au; grant.williamson@utas.edu.au; Murray.Turner; Farhad Salimi; sarah.henderson@ubc.ca Subject: Fw: Decision on your submission to BMC Research Notes -RESN-D-17-00543
FYI, rejected due to inability to find reviewers in the field. I think the suggestions are good and easy enough to make changes.
Send me any thoughts of another Journal?
From: em.resn.0.54602a.866dc6ed@editorialmanager.com <em.resn.0.54602a.866dc6ed@editorialmanager.com> on behalf of BMC Research Notes - Editorial Office <em@editorialmanager.com> Sent: Saturday, 8 July 2017 12:12 AM To: Ivan.Hanigan Subject: Decision on your submission to BMC Research Notes -RESN-D-17-00543
RESN-D-17-00543 Extensible database of validated biomass smoke events for health research Ivan Charles Hanigan, PhD; Geoffrey G Morgan,, PhD; Grant J Williamson, PhD; Farhad Salimi, PhD; Sarah B Henderson,, PhD; Murray R Turner; David M. J. S. Bowman, PhD; Fay H Johnston, PhD BMC Research Notes
Dear Dr Hanigan,
Thank you for considering BMC Research Notes for your manuscript (above). I am sorry to inform you that despite much effort we have been unable to obtain appropriate referees for your manuscript. We are therefore handing it back to you and we are closing your file, so that you may submit it elsewhere. We wish you all the best in finding an alternative venue for your research.
While we regret that we have had to come to this decision, it does seem likely to us that a more specialized journal may have better luck in identifying reviewers.
Below you will find the editorial summary from one of our PhD-level qualified manuscript assessors. While we do not feel able to make a decision on whether your manuscript is publishable based on this report, we hope it will be of use when submitting your manuscript elsewhere.
I wish you every success with your research and hope that you will consider us again in the future.
Best wishes,
Dirk Krüger BMC Research Notes https://bmcresnotes.biomedcentral.com/
BMC Research Notes | Home page
bmcresnotes.biomedcentral.com
BMC Research Notes publishes scientifically valid research outputs that cannot be considered as full research or methodology articles. We support the research …
Reviewer reports: Reviewer 1: “PEER REVIEWER COMMENTS: No academic peer reviewer available.
EDITORIAL STAFF COMMENTS REGARDING REPORTING STANDARDS: Summary: This paper describes the creation of a database that collects evidence linking historical spikes in air pollution with smoke from vegetation fires. Additionally, the authors describe how the database has been extended to be distributed in an open format that allows the research community to add to the history of these events. Research question: There is a clear research question, and the authors have assessed it adequately. The authors thoroughly describe why the data were obtained and provide citations to support the relevance of this paper’s aims. Nonetheless, in the Introduction, the authors should provide more information about where the data (that is included in the database studied) come from. Who enters the data? Who maintains and cleans the database? The authors state that this study describes how the database has been extended to be distributed in an open format; what was the prior format? How wide of an area did it previously cover, and how available was it previously? Methods: The authors provide thorough information on the protocols used to include information in the database. The authors state that the “Bare Minimum Protocol was developed for this paper”, but it is unclear how and why this protocol was created and adapted for use in the current database. They also state that any of the above mentioned protocols can be used, or the database contributor can use his/her own protocol, but it is unclear how consistency in data entry is ensured by allowing variations in contributors’ choice of protocol. Results: Overall, the diagram presented in Figure 1 is helpful in visualizing the process of utilizing the database and system. However, it is unclear why the flow of the diagram is at a diagonal slant. The flow and direction of activities described would be clearer in a horizontal or vertical presentation. Additionally, it is unclear why the majority of arrows are blue, while two arrows are green. Is there significance in the color designation? Conclusions and recommendations: Overall, the authors provide useful information and descriptions of the database used to identify historical spikes in particulate matter concentrations and evaluate whether they are caused by vegetation fire smoke or by other means. Nonetheless, the authors should consider tying the Results and Conclusions back to the citations presented in the Introduction more clearly to illustrate the relevance and need for this paper. Additionally, the authors present a description of the database and its expanded form, but they do not present any discussion of testing this expanded form. Do the authors intend to test it to ensure its effectiveness in capturing necessary data? It is unclear what the future direction or applicability of this paper is, and the authors should make that clearer. In general, this paper seems useful, and with additional information added, it may be suitable for publication. The language quality is sufficient for further consideration or publication.”
If improvements to the English language within your manuscript have been requested we recommend that you address this before submitting to another journal. We recommend that you either get your manuscript reviewed by someone who is fluent in English or, if you would like professional help, you can use any reputable English language editing service. We can recommend our affiliates Nature Research Editing Service (http://bit.ly/NRES_BS) and American Journal Experts (http://bit.ly/AJE_BS) for help with English usage. Please note that use of an editing service is neither a requirement nor a guarantee of publication. Free assistance is available from our English language tutorial (https://www.springer.com/gb/authors-editors/authorandreviewertutorials/writinginenglish) and our Writing resources (http://www.biomedcentral.com/getpublished/writing-resources). These cover common mistakes that occur when writing in English.
Author and Reviewer Tutorials - Writing in English | Springer
AJE - Editing, Formatting, Translation and Illustration Services for scientific researchers by academic experts | AJE | American Journal Experts
bit.ly
AJE helps research break through with top-quality author services from experienced academics. We can help prepare your manuscript, figures, posters, and more.
Writing resources - BioMed Central
Welcome to BioMed Central’s writing resources, a guide on writing and publishing a scientific manuscript. You can use the links to the left or below to find advice …
English Language Editing | Author services from Springer Nature
bit.ly
English Editing from Nature Research ✯ Corrects your language errors ✯ Expert editors in your subject area ✯ All academic text, including research papers
this needs to pick out a lot of stuff from the manuscript on the R package Repo, I am seperating the R pack (tools) from this repo (data and report/manuscript)
- copy current db to alternate folder
- import perth into current database WITHOUT SYNCING TO GITHUB
- rename folders and DO NOT GIT ADD DATABASE_WITH_WA FOLDER
- rewrite script get_events_and_timeseries.R (direct to R, not in orgmode)
- mention sampling effort bias, in metadata
- into data_extracts folder, move old data to versions/. keep metadata and temp
- send to ric up to github without wa
- 13 Sydney
- 9 Illawarra
- 11 Lower Hunter
- 8 Hobart
- 10 Launceston
- 12 Perth
- 1 Albany
- 5 Bunbury
- 6 Busselton
- 7 Geraldton
- 2 Albury
- 3 Armidale
- 4 Bathurst
- 14 Tamworth
- 15 Wagga Wagga
- up to cardat with wa
- up to github without wa
## additional plots for sydney
png(sprintf(“figures_and_tables/%s_bushfires_2002_03.png”, town), width = 1000, height = 700, res = 80) with(extracted, plot(date, pm10_lag0, type = “l”, ylim = c(-10, 150), xlim = c(as.Date(“2002-12-01”), as.Date(“2003-02-28”)))) points( extract[extracted$lfs_pm10_lag0 == 1, “date”] , extract[extracted$lfs_pm10_lag0 == 1, “pm10_lag0”] , col = ‘red’, pch = 16, cex = .7 ) segments( extract[extracted$lfs_pm10_lag0 == 1, “date”] , 10, extract[extracted$lfs_pm10_lag0 == 1, “date”] , extract[extracted$lfs_pm10_lag0 == 1, “pm10_lag0”], lty = 2 )
axis(1, labels = NA) x <- seq(as.Date(“2002-12-01”), as.Date(“2003-02-28”), by = 1) text(x, -10, x, adj = 0, cex = 0.8, srt = 90) title(paste(town, “validated bushfire smoke events (red) in Summer 2002/03”)) dev.off()
png(sprintf(“figures_and_tables/%s_bushfires_2001_02.png”, town), width = 1000, height = 700, res = 80) with(extracted, plot(date, pm10_lag0, type = “l”, ylim = c(-10, 150), xlim = c(as.Date(“2001-12-01”), as.Date(“2002-02-28”)))) points( extract[extracted$lfs_pm10_lag0 == 1, “date”] , extract[extracted$lfs_pm10_lag0 == 1, “pm10_lag0”] , col = ‘red’, pch = 16, cex = .7 ) segments( extract[extracted$lfs_pm10_lag0 == 1, “date”] , 10, extract[extracted$lfs_pm10_lag0 == 1, “date”] , extract[extracted$lfs_pm10_lag0 == 1, “pm10_lag0”], lty = 2 )
axis(1, labels = NA) x <- seq(as.Date(“2001-12-01”), as.Date(“2002-02-28”), by = 1) text(x, -10, x, adj = 0, cex = 0.8, srt = 90) title(paste(town, “validated bushfire smoke events (red) in Summer 2001/02”)) dev.off()
dbSendQuery(ch, “UPDATE pollution_stations_combined_final SET region = ‘Illawarra’ WHERE region = ‘Illawara’”)
## 2020-02-18 there was a difference with Williamson at end of 2013, and this is ## in Horsley too qc <- extracted[extracted$date >= as.Date(“2013-01-01”) & extracted$date <= as.Date(“2013-12-31”), c(“date”, “pm25_lag0”, “lfs_event”, “pm25pct_lag0”)] qc[qc$pm25_lag0 > 25,] qc[qc$pm25pct_lag0 >= 0.95,] ## yep so the 21st is not identified but is obviously the big one. ## I will add this ## TODO need to come back to this month and check the other high days, ## there was also a notable prescribed burn overnight on 21/22 october ## written about in wikipedia
## ok now make the plot again with(extracted, plot(date, pm25_lag0, type = “l”, ylim = c(0, 100))) points( extract[extracted$lfs_pm25_lag0 == 1, “date”] , extract[extracted$lfs_pm25_lag0 == 1, “pm25_lag0”] , col = ‘red’, pch = 16, cex = .7 )
## Horsley found A total of 184 LFS days (2001-2013)
############################################
qc[qc$contributor== “Dr Ivan Hanigan”,] id source title year credentials credentials_other 701 NA modis rapidfire smoke plume visible 2013 modis smoke 702 NA modis rapidfire smoke plume visible 2013 modis smoke authors volume url summary 701 Satellite Data NA 702 Satellite Data NA abstract 701 Ivan submitted this to append data to those identified by Farhad. 702 Ivan added to append to Farhad’s work
dbGetQuery(ch,”select * from biomass_smoke_reference where abstract like ‘Ivan added%’”) dbGetQuery(ch,”select max(id) from biomass_smoke_reference”) 799 dbSendQuery(ch, “UPDATE biomass_smoke_reference SET id = 800 WHERE abstract like ‘Ivan added%’”)
dbGetQuery(ch,”select * from biomass_smoke_reference where abstract like ‘Ivan submitted%’”)
dbSendQuery(ch, “delete from biomass_smoke_reference WHERE abstract like ‘Ivan submitted”)
## and now the event is also a problem dbGetQuery(ch,”select max(id) from biomass_smoke_event”) 1467 names(dbGetQuery(ch,”select * from biomass_smoke_event”)) dbSendQuery(ch, “insert into biomass_smoke_event ( id, biomass_smoke_reference_id, place, event_type, min_date) values (1468, 800, ‘Sydney East’, ‘bushfire’, ‘2013-10-21’) “)
- make empty database folder
- start web2py
it is removed from the with_wa database, but still in the without_wa. check before finalising
events[events$biomass_smoke_reference_id == 656,]
from https://groups.google.com/forum/#!topic/web2py/kCcRMFmZKB8
In the database’s management tool delete/drop the table;
library(RSQLite) drv <- dbDriver(“SQLite”) con <- dbConnect(drv, dbname = “~/tools/web2py/applications/data_inventory_demo/databases/storage.sqlite”) dbListTables(con) dbSendQuery(con, “drop table dataset”)
Trash the table’s .table file in the databases folder;
fi <- dir(pattern = “_dataset.table”) for(f in fi){system(sprintf(“rm %s”, f))}
In db.py set migrate to migrate=’tablename.table’
fake_migrate_all = True, migrate = ‘dataset.table’
Save db.py
Return to any app site, such as admin/default/design/appname Go to the database administration, hit f5
db.py fake_migrate_all = False, migrate = True
go to the webform but DONT insert a record!
Return to db.py to set migrate to migrate=False
fake_migrate_all = True, migrate = False)
in pgadmin rename the orig as today date
> library(RSQLite) > drv <- dbDriver(“SQLite”) > con <- dbConnect(drv, dbname = “~/tools/web2py/applications/biomass_smoke_events_db/databases_20170607/storage.sqlite”) > dbListTables(con) > qc1 <- dbGetQuery(con , “select * from biomass_smoke_event”) > str(qc1) qc2 <- dbGetQuery(con , “select * from biomass_smoke_reference”) str(qc2)
CREATE TABLE public.biomass_smoke_reference ( id integer NOT NULL DEFAULT nextval(‘biomass_smoke_reference_id_seq’::regclass), source character varying(512), title character varying(512), year integer, credentials character varying(512), credentials_other character varying(512), authors character varying(512), volume integer, url character varying(512), summary text, abstract text, protocol_used text, CONSTRAINT biomass_smoke_reference_pk PRIMARY KEY (id) ) WITH ( OIDS=FALSE ); ALTER TABLE public.biomass_smoke_reference OWNER TO w2p_user; GRANT ALL ON TABLE public.biomass_smoke_reference TO w2p_user; GRANT SELECT ON TABLE public.biomass_smoke_reference TO ivan_hanigan;
CREATE TABLE public.biomass_smoke_event ( id integer NOT NULL DEFAULT nextval(‘biomass_smoke_event_id_seq’::regclass), biomass_smoke_reference_id integer, place character varying(512), place_other character varying(512), event_type character varying(512), min_date date, max_date date, burn_area_ha double precision, met_conditions text, CONSTRAINT biomass_smoke_event_pk PRIMARY KEY (id), CONSTRAINT biomass_smoke_event_biomass_smoke_reference_id_fk FOREIGN KEY (biomass_smoke_reference_id) REFERENCES public.biomass_smoke_reference (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE CASCADE ) WITH ( OIDS=FALSE ); ALTER TABLE public.biomass_smoke_event OWNER TO w2p_user; GRANT ALL ON TABLE public.biomass_smoke_event TO w2p_user; GRANT ALL ON TABLE public.biomass_smoke_event TO ivan_hanigan;
Please note that in the database I published to github at ‘databases/storage.sqlite’ there is also a table of each site, lat/lons and pollution concentrations that you could use to do a IDW or kriging surface for each smoke day. That would be really cool. An example query in R is:
- tex
- then
cd ~/tools/web2py/applications/biomass_smoke_events_db/static/manuscript convert -density 300 biosmoke_system_diagram.pdf biosmoke_system_diagram.png
The ability for this database to be extended by other researchers means that new events can be added, and new information for already identified events can be described. These methods provide a systematic framework for retrospective identification of the air quality impacts of biomass smoke. In this paper, we describe the database and data aquisition methods, as well as analytical considerations when validating historical events using a range of reference types.
This is because if other scientists notice an ommision or error in these data they can offer an amendment.
We believe that this will improve the database and benefit the whole biomass smoke health research community.