Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest byw #435

Merged
merged 37 commits into from
Dec 22, 2023
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
1ffae07
test
LishaRamon Dec 13, 2023
a9b0951
first commit
LishaRamon Dec 13, 2023
03b9ffd
update gitignore
LishaRamon Dec 13, 2023
1b048b0
modular source ingest script
kelle Dec 13, 2023
55402ce
update .gitignore
kelle Dec 13, 2023
4ff5a7f
First ingest-bwy-2024
LishaRamon Dec 14, 2023
11d78f8
Removed lines +added source
LishaRamon Dec 15, 2023
b5b8bef
ascii.read change
LishaRamon Dec 15, 2023
83d99b3
Changed incorrect import name
LishaRamon Dec 15, 2023
aaa7a40
Import error: cannot import 'ascii' from astropy.io.ascii (Line 3)
LishaRamon Dec 16, 2023
720a41d
Update csv viewing
LishaRamon Dec 18, 2023
7ded761
update google sheet file not found error
LishaRamon Dec 18, 2023
055feb3
update google sheet table prints. Accidental json file changes
LishaRamon Dec 18, 2023
ca4cfbf
Updated import modules
LishaRamon Dec 18, 2023
a912394
Source ingest layout attempt
LishaRamon Dec 18, 2023
0bc26b4
Ingest_source: added docstring
kelle Dec 18, 2023
f0de236
ingest source linting
kelle Dec 18, 2023
aaff895
New changes
LishaRamon Dec 18, 2023
f7dd534
black formatting
kelle Dec 18, 2023
d38f92a
Cleaned script + json files
LishaRamon Dec 20, 2023
7450f6a
Updated suggestions
LishaRamon Dec 20, 2023
6f4f05b
New Roth test
LishaRamon Dec 20, 2023
27346f6
mostly black and pylance fixes
kelle Dec 20, 2023
4e43af0
fixed raise_error bug
kelle Dec 20, 2023
6a48f3e
changes blobs to None
kelle Dec 20, 2023
1b31552
fixed JSON files
kelle Dec 20, 2023
91c4967
add _all_
kelle Dec 20, 2023
6d77346
add ingest_source test
kelle Dec 20, 2023
be92726
clean up .gitignore
kelle Dec 20, 2023
cc83b20
Corrected 2mass_j12560183-1257276.json file
LishaRamon Dec 20, 2023
12e1dfd
improved ingest
kelle Dec 20, 2023
7741f6b
Merge branch 'ingest-byw' of github.com:LishaRamon/SIMPLE-db into ing…
kelle Dec 20, 2023
04caf21
Added shortnames in names table
LishaRamon Dec 22, 2023
b21d984
clean up of tests and utils
kelle Dec 22, 2023
e012871
Merge branch 'ingest-byw' of github.com:LishaRamon/SIMPLE-db into ing…
kelle Dec 22, 2023
78f5611
Corrected 2mass-j1256 json again
LishaRamon Dec 22, 2023
5f42f96
Ingest json shortname + deleted extra file
LishaRamon Dec 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
# Ignore the database file and misc
*.db
*.sqlite
*.pyc
.idea
.vscode
.pytest_cache
.ds_store
bdnyc/
bdnyc/
*.code-workspace

38 changes: 38 additions & 0 deletions scripts/ingests/ingest_BYW_2024.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
from scripts.ingests.ingest_utils import *
from scripts.ingests.utils import *
from astropy.io.ascii import ascii
import astropy.units as u
from astropy.coordinates import Angle

SAVE_DB = True # save the data files in addition to modifying the .db file
RECREATE_DB = True # recreates the .db file from the data files
# LOAD THE DATABASE
db = load_simpledb('SIMPLE.db', recreatedb=RECREATE_DB)


# live google sheet
link = "https://docs.google.com/spreadsheets/d/1JFa8F4Ngzp3qAW8NOBurkz4bMKo9zXYeF6N1vMtqDZs/edit#gid=0"
LishaRamon marked this conversation as resolved.
Show resolved Hide resolved

columns = ['source', 'ra', 'dec', 'epoch', 'equinox', 'shortname', 'reference', 'other_ref', 'comments']
byw_table = ascii.read(link, format='ascii', data_start=2, data_end=90, header_start=1, names=columns, guess=False,
LishaRamon marked this conversation as resolved.
Show resolved Hide resolved
fast_reader=False, delimiter=',')

data_columns = ['Source', 'RA', 'Dec', 'Epoch', 'Equinox', 'Shortname', 'Reference', 'Other_ref', 'Comments'] # columns with wanted data values

# replacing empty values ('cdots') with None
for column in data_columns:
byw_table[column][np.where(byw_table[column] == 'cdots')] = None

LishaRamon marked this conversation as resolved.
Show resolved Hide resolved
def ingest_source(db):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Careful. A function already exists with this name. You imported it from utils. This function should be named something different.


ingest_source(db, source = ["CWISE J000021.45-481314.9"],
reference = "Rothermich",
ra = [0.0893808],
dec = [-48.2208077],
epoch = [2015.4041],
equinox = "ICRS")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to try this! Does it work?



# WRITE THE JSON FILES
if SAVE_DB:
db.save_database(directory='data/')
3 changes: 3 additions & 0 deletions scripts/ingests/ingest_backyard_worlds_benchmarks.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
test

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file shouldn't be part of the ingest. I would just delete the entire file.


Loading