Including report date / avoiding repeated notification

[neuhoffm]

Is there a way to include the report date in the message that gets sent out? It looks like the report isn't updated very consistently and I'd like to include the report date in the message. There doesn't seem to be a specific css class associated with the report date span but it is the only occurrence of the "Report Date:" string in the file. Python isn't my strong suit or I'd add a PR for this feature.

[dhimmel]

Sorry for the slow response. I think my notifications were turned off somehow.

Anyways I think you are talking about this report date:

While it is posted above the "Donohue Exit Quota and Trailhead Space Available" section, I presume it refers to the date the remaining data on the webpage was updated on?

I'm happy to make this addition... shouldn't be too hard.

[dhimmel]

Currently, the date shown in the message is from the HTTP header:

hackjohn/hackjohn.py

Line 74 in 4f7ae38

text = f'''Spaces available as of {yose_response.headers['Date']}

You can see the headers that get returned using

curl --head https://www.nps.gov/yose/planyourvisit/fulltrailheads.htm

I believe this Date is set by the server that responds to the request.

I like that idea that we should default to using the date from report date, and only provide this date if that field cannot be found.

[dhimmel]

@neuhoffm see if 8ac9f7d implements your suggestion?

Hackjohn now uses "Report Date" for the date in the message. There is also a min_report_date setting that allows you to suppress notifications until the report date increases to the be >= to min_report_date. This could come in handy if Hackjohn identifies a spot, which is taken by time you call, and therefore you want to continue the search, but suppress notifications until the report updates.

[neuhoffm]

All good! I was mainly thinking it could be cool to check if the report has been updated since the last time the script ran and then either do or don't send a notification based off of that if people wanted. I talked to the reservation office and they're updating the report very sporadically now so unfortunately the 11am M-F isn't a guarantee anymore.

[dhimmel]

do or don't send a notification based off of that if people wanted

It would be great if we could send only one notification per report date. This would require some sort of memory (persistence data between runs), which currently does not exist. Something to keep in mind, because it would prevent annoying over-notification.

I talked to the reservation office and they're updating the report very sporadically now so unfortunately the 11am M-F isn't a guarantee anymore.

Wow so the report hasn't been updated in 15 days. That is a real bummer. When you call, do they look through their internal systems to see the latest reservation info? Basically, I'm curious whether this change makes it so you just need to call them periodically to get any up to date information. Or is their internal information now only updated infrequently?

[dhimmel]

I see the website says:

This report does not reflect live numbers, so the trailhead may have filled after the last update. Please note the date and time of updates at the top right of the data.

I think "top right of the data" must refer to "Report Date", although this is on the left.

[dhimmel]

I have thought of another solution. We could have a setting to only notify about vacancies in reports less than days_until_stale days. For example, when days_until_stale=1, we would only send notifications when the Report Date was within one day of the current date. This would prevent getting notifications for a report that was already outdated, and does not require persistent data storage.

@neuhoffm would this address your issue?

[neuhoffm]

That could work, are you worried about it persisting the date the last run pulled to a text file? Although the page only provides a date and not a time so our need for precision is much less. With the page being updated sporadically, I've set the script to run every couple of minutes, I'm trying to figure out if the days_until_stale property would work with that frequency. Thanks for looking into this!

[dhimmel]

That could work, are you worried about it persisting the date the last run pulled to a text file?

I wanted to avoid writing files, but I think that is the simplest (and only robust solution). @neuhoffm check out the changes in fbc511e. See if this works for you?

The program now defaults to writing a file to hackjohn-output.txt in the same directory that the script is located. If hackjohn-output.txt is unchanged, then Telegram notification is suppressed. This should ensure that you only get notifications when something has actually changed in the report or in your processing.

With the page being updated sporadically, I've set the script to run every couple of minutes

Got it. From my recent memory, the report was updated on April 3, April 19, and April 22. That two-week gap is scary, but hopefully they stick to the every-other-day frequency or so. Note that it is possible for sites to rate limit a specific IP address or completely block it for making too many requests. Probably not a big risk here, but who knows.

[neuhoffm]

Looks great! I'll pull down latest and add a PR to support IFTTT if people want to use that instead of middle man

[dhimmel]

add a PR to support IFTTT if people want to use that instead of middle man

That would be a useful additional, I imagine (have never used IFTTT myself).