Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ingest byw #435

Merged
merged 37 commits into from
Dec 22, 2023
Merged

Ingest byw #435

merged 37 commits into from
Dec 22, 2023

Conversation

LishaRamon
Copy link
Collaborator

@LishaRamon LishaRamon commented Dec 13, 2023 •
edited by kelle
Loading

Ingesting the sources from Rothermich 2024

Link to relevant issue: Closes # 428

For data ingests:

  • includes script used for ingest
  • includes modified JSON files
  • Add new tests
  • Update the Versions table

@LishaRamon LishaRamon self-assigned this Dec 13, 2023
@LishaRamon
Copy link
Collaborator Author

Line 17 Error: "astropy.io.ascii.core.InconsistentTableError: Number of header columns (9) inconsistent with data columns (7) at data line 0"

Header values: ['Source', 'RA', 'Dec', 'Epoch', 'Equinox', 'Shortname', 'Reference', 'Other_ref', 'Comments']
Data values: ['CWISE J000021.45-481314.9', '0.0893808', '-48.2208077', '2015.4041', 'ICRS', 'CWISE 0000-4813', 'Rothermich et al. Submitted']

I understand how it's seeing there's missing data under two of the header columns but not sure how to go about debugging for it to recognize that having a missing value is okay.

@kelle
Copy link
Collaborator

kelle commented Dec 14, 2023

I think you should use astropy.io.ascii instead of astropy.Table. It looks like ascii.read does a better job of handling empty cells:

https://docs.astropy.org/en/stable/io/ascii/read.html

@LishaRamon
Copy link
Collaborator Author

Line 17 Error Update:
ValueError: ASCII format 'ascii' not in allowed list ['aastex', 'basic', 'cds', 'commented_header', 'csv', 'daophot', 'ecsv', 'fast_basic', 'fast_commented_header', 'fast_csv', 'fast_no_header', 'fast_rdb', 'fast_tab', 'fixed_width', 'fixed_width_no_header', 'fixed_width_two_line', 'html', 'ipac', 'latex', 'mrt', 'no_header', 'qdp', 'rdb', 'rst', 'sextractor', 'tab']

How would I find/recognize the correct 'format'? I assumed csv since it's an excel file, but it breaks the code further.

kelle
kelle reviewed Dec 15, 2023
View reviewed changes
scripts/ingests/ingest_BYW_2024.py Outdated Show resolved Hide resolved
scripts/ingests/ingest_BYW_2024.py Outdated Show resolved Hide resolved
@kelle kelle linked an issue Dec 15, 2023 that may be closed by this pull request
Ingest the Backyard Worlds Benchmarks #428
Closed
@kelle
Copy link
Collaborator

kelle commented Dec 18, 2023

Here's a video to help you format the link to the Google sheet. You don't need to use pandas, just need to get the URL correct:

https://youtu.be/t6WSY9D_ORQ?feature=shared

kelle
kelle reviewed Dec 18, 2023
View reviewed changes
scripts/ingests/ingest_BYW_2024.py Outdated
data_columns = ['Source', 'RA', 'Dec', 'Epoch', 'Equinox', 'Shortname', 'Reference', 'Other_ref', 'Comments'] # columns with wanted data values


def ingest_source(db):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Careful. A function already exists with this name. You imported it from utils. This function should be named something different.

scripts/ingests/ingest_BYW_2024.py Outdated
Comment on lines 25 to 30
ingest_source(db, source = ["CWISE J000021.45-481314.9"],
reference = "Rothermich",
ra = [0.0893808],
dec = [-48.2208077],
epoch = [2015.4041],
equinox = "ICRS")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to try this! Does it work?

@kelle
Copy link
Collaborator

kelle commented Dec 18, 2023

If you can't figure out how to read the Google sheet, just use the CSV file from the issue.

@LishaRamon
Copy link
Collaborator Author

Since ascii.read attempted to read data from local files rather from URLs, by using a library like 'requests' it helped get data and create an object that can be passed to ascii.read to understand

@kelle
Copy link
Collaborator

kelle commented Dec 18, 2023

Great work!

@LishaRamon
Copy link
Collaborator Author

Not sure why the json files didnt update when I added the changes, but they didn't show up for me to push new ones

kelle
kelle approved these changes Dec 22, 2023
View reviewed changes
Copy link
Collaborator

@kelle kelle left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great to me. I think we can merge this tomorrow.

kelle
kelle reviewed Dec 22, 2023
View reviewed changes
data/cwise_j000021.45-481314.9.json Show resolved Hide resolved
scripts/ingests/ingest_backyard_worlds_benchmarks.py Outdated
@@ -0,0 +1,3 @@
test

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file shouldn't be part of the ingest. I would just delete the entire file.

@LishaRamon LishaRamon marked this pull request as ready for review December 22, 2023 18:41
@kelle kelle merged commit f117cc2 into SIMPLE-AstroDB:main Dec 22, 2023
1 check passed
@LishaRamon LishaRamon deleted the ingest-byw branch December 22, 2023 19:50
@LishaRamon LishaRamon restored the ingest-byw branch December 22, 2023 19:52
@LishaRamon LishaRamon deleted the ingest-byw branch December 22, 2023 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kelle kelle kelle approved these changes

Assignees

@LishaRamon LishaRamon

Labels
data ingestion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Ingest the Backyard Worlds Benchmarks
2 participants
@LishaRamon @kelle