Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WINTER ingestion into kowalski #174

Merged
merged 56 commits into from
Oct 4, 2022
Merged

WINTER ingestion into kowalski #174

merged 56 commits into from
Oct 4, 2022

Conversation

virajkaram
Copy link
Collaborator

This PR implements the ingestion of the WINTER alert stream into Kowalski.

  • Added alert_broker_winter.py mirroring alert_broker_ztf.py
  • Added winter sections to config.defaults.yaml, and expose different dask ports for the WIINTER dask clients
  • Add 5 fake alerts for testing, and tests for winter mirroring PGIR and ZTF tests
  • Update Dockerfile with WINTER specific entries
  • Add kowalski/check_db_entries.py for easier debugging.

@virajkaram
Copy link
Collaborator Author

@guynir42 @mcoughlin I think the PR is ready for review, thanks!

Copy link
Collaborator

@mcoughlin mcoughlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments. Would be good to add some more info somewhere (docs/README) about what you changed to make this work.

version.txt Outdated Show resolved Hide resolved
@@ -0,0 +1,404 @@
from confluent_kafka import Producer
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some info to the README or similar about what is different about this ingester than for ZTF? I.e. it's all the same but XYZ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a summary of the changes to README. here

@@ -0,0 +1,627 @@
from abc import ABC
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think some summary of what is different than ZTF here would be very useful for future additions.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated README with the details - here

# fixme: PGIR uses 2massj, which is not in sncosmo as of 20210803
# cspjs seems to be close/good enough as an approximation
# 20220818: added WNTR
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added bandpasses, also fixed a small bug so that the filter is not hardcoded for WINTER and any of the WINTER filters can be processed.

ingester.Dockerfile Outdated Show resolved Hide resolved
phot_bp_mean_mag: 1
phot_rp_mean_mag: 1

Gaia_EDR3:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need DR2 and eDR3?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got rid of DR2, we probably want to keep EDR3. Also reduced the xmatch radius to 2 arcsec

Copy link
Collaborator

@guynir42 guynir42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have minor comments but other than that it looks ok to me.

kowalski/alert_broker.py Outdated Show resolved Hide resolved
kowalski/alert_broker.py Outdated Show resolved Hide resolved
kowalski/alert_broker.py Outdated Show resolved Hide resolved
kowalski/dask_cluster_winter.py Show resolved Hide resolved
readme.md Outdated Show resolved Hide resolved
readme.md Outdated Show resolved Hide resolved
tests/test_alert_broker_wntr.py Outdated Show resolved Hide resolved
tests/test_ingester_wntr.py Outdated Show resolved Hide resolved
@mcoughlin
Copy link
Collaborator

@virajkaram Looks like you need to remove version.txt from your commit (thanks for the .gitignore), otherwise should be good to go in.

@virajkaram
Copy link
Collaborator Author

@virajkaram Looks like you need to remove version.txt from your commit (thanks for the .gitignore), otherwise should be good to go in.

Done!

@mcoughlin mcoughlin merged commit 5c0dd02 into skyportal:master Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants