U/lynnej/add sim notebook

[rhiannonlynne]

Adds prenight/run_sim notebook which will run a default simulation and make some plots.
Running on a night in the past will generate a simulation which matches the time the FBS was on-sky.

[rhiannonlynne]

This doesn't replace in any way the prenight simulations, but adds a place where we could check out default options on the fly.
One thing to consider might be - if the day_obs == today_day_obs, then start the simulation when the FBS went on sky but continue to the end of the night (instead of ending at the current time). .. Then this is like an updateable reference for what might happen in the current night.

[ehneilsen]

github doesn't seem to let me comment on individual lines of notebooks anymore, so I guess I put it all here.

Overall, I think this really should be reworked to avoid duplication with the prenight stuff, primarily by extracting code all the code with overlapping functionality and putting it somewhere it could be called by either: the stuff useful here is likely also stuff that would be useful to add to the prenight briefing, and some things here are near duplicates of things already in the prenights. For example, the code for the downtime, DDF availability, and seeing plots could be moved into schedview, and just called in this notebook, so that these plots could be added to the night summary and/or prenight report, and the airmass plot duplicates the altitude & airmass plot in the prenight (but without the hovertool).

The maps at the end are quite close to some of those I was working on last week for a different issue (SP-2518), and will be making a PR for probably next week. (I have them running in a scratch notebook, and am working on getting them into schedview and schedview_notebooks.)

It would be really good if the simulations created this way were added to the simulation database. This is easy if the notebook has write access to the prenight postgresql database, which unfortunately it won't if it's run with TS. It's always been "on my radar" to make a phalanx service to do on-the-fly new prenight simulations, and if all of the simulation creation code in this notebook was in a python function instead of a notebook, that probably wouldn't be that hard: we could inject the credentials to add to the database using the phalanx secrets mechanism.

[rhiannonlynne]

I don't really think you want these simulations added to the prenight database though. Although a lot of them will be simple one-night for the upcoming night outputs (which would just be identical to what you create in the prenight sims, but not as good because it's only one night), someone could make one for five nights .. or start one five nights into the future just because they want to see what might happen on Friday (but it wouldn't be a really good sim, because you'd be missing the four nights of observations between now and Friday).
Personally, I view this as just a convenient way to let people generate a last-minute simulation for the upcoming night (if we updated the ts_config_scheduler setup, for example or if we get the ToO stuff working, to see what the ToO would result in) or run something for the future -- while also maybe having access to some of the things that aren't making it into the prenight briefing reports yet.

Note that you can also choose a date in the past -- on purpose. This lets you see how close we would have come to what we did, if we knew when we would have been on-sky.

The airmass plot is different than the one in schedview, because the grouping and color-coding is quite different. Also note that it is different once you choose a date in the past.
I will look at what plots you get from schedview for the maps - I saw the PR regarding the mcbryde maps.. the ones I saw in Times Square for the nightsum were not as easy to read.

We can move some of these ad-hoc plots into schedview .. they really were just testing things that turned out to be kind of useful. I can put in a PR for this kind of crappy downtime plot and the DDF visibility .. I suppose what we can do after that is rework the prenight briefing to include more of this too.
The seeing plot .. I'm less sold on. It's kind of interesting when you're looking at the past, but I don't think we know how to interpret the dimm(s) and our seeing model yet so I don't think I'd be in a rush to add it anywhere else.

[ehneilsen]

I don't really think you want these simulations added to the prenight database though. Although a lot of them will be simple one-night for the upcoming night outputs (which would just be identical to what you create in the prenight sims, but not as good because it's only one night), someone could make one for five nights .. or start one five nights into the future just because they want to see what might happen on Friday (but it wouldn't be a really good sim, because you'd be missing the four nights of observations between now and Friday). Personally, I view this as just a convenient way to let people generate a last-minute simulation for the upcoming night (if we updated the ts_config_scheduler setup, for example or if we get the ToO stuff working, to see what the ToO would result in) or run something for the future -- while also maybe having access to some of the things that aren't making it into the prenight briefing reports yet.

One of the major problems with the current prenight simulation code is dealing with last minute updates: we need an easier mechanism for someone add new simulations based on updated ts_config_scheduler setups, new ToO stuff, etc. This code could either accomplish that directly (by including code to add a sim to the archive). Or, if it gets extracted from the notebook into real importable python code (probably in lsst_survey_sim), it can be called from a new phalanx service that we can give write access to the database. Limitations in access to the postgresql database mean that the former approach would limit the use of it to the scheduler team, while the second option would support anyone with USDF access updating the simulations.

[ehneilsen]

Note that you can also choose a date in the past -- on purpose. This lets you see how close we would have come to what we did, if we knew when we would have been on-sky.

Plots to compare actual to simulated visits were the intention of the comparenight notebook in schedview_notebooks. This was written early on, before we really knew much about what would be useful in practice. It its current state, it isn't all that useful. Any plots that you can provide are actually useful could greatly improve comparenight if they were made available to it in functions in schedview. If this is the same code as used for useful prenight plots but with options to add additional elements turned on, that seems like a good thing for avoiding duplication of code and making it easy interpret when moving between reports.
(Feeding comparenight data would be another reason to add simulations to the archive. Having a more comparable simulation might even make the comparison plots that are already there more useful.)

[rhiannonlynne]

I am somewhat confused.
What part of creating the simulation is not already easily importable and re-creatable? What part are you NOT using in the command line scripts already? (maybe this is the problem).
Every part of the actual simulation creation is just calling a function from lsst-survey-sim.

[ehneilsen]

The airmass plot is different than the one in schedview, because the grouping and color-coding is quite different. Also note that it is different once you choose a date in the past.

The color coding in schedview.plot.nightly.plot_alt_vs_time does need to be fixed (things have changed quite a lot since I wrote it, and it's much messier now that it used to be). If there are useful things to add to compare with the past, these could be added too.

[rhiannonlynne]

The ToO reading from a pickle doesn't count -- everything in the notebook is a hack to get around the fact that nothing at USDF can access the lsst.scimma.too_alerts topic which only exists at base and summit. The solution to this is just to get this topic into the usdf copy of the EFD. And then the function from lsst-survey-sim works.

[ehneilsen]

The ToO reading from a pickle doesn't count -- everything in the notebook is a hack to get around the fact that nothing at USDF can access the lsst.scimma.too_alerts topic which only exists at base and summit. The solution to this is just to get this topic into the usdf copy of the EFD. And then the function from lsst-survey-sim works.

Yeah, it was mostle the ToO stuff I was thinking of.

[ehneilsen]

I will look at what plots you get from schedview for the maps - I saw the PR regarding the mcbryde maps.. the ones I saw in Times Square for the nightsum were not as easy to read.

I was referring more to SP-2518 (adding maps of more stuff to nightsum) than SP-2517 (adding the McBryde projection maps).

[ehneilsen]

I can put in a PR for this kind of crappy downtime plot and the DDF visibility .. I suppose what we can do after that is rework the prenight briefing to include more of this too.

I've had ambitious thoughts on a DDF plot for a while, something that kind of replicates what the old iObserve macos observing planning app used to do (it doesn't seem to exist anymore) and that I found useful for other projects. It's basically an altitude plots of pointings of interest, much like the DDF visibility plot you have here, but which also represents the proximity to the moon. It would also be useful to mark the times the scheduler has it pre-scheduled, and maybe other stuff. But, a plot with only a little information (e.g. alt) that actually exists is way better than any plot that only exists in my head!