Bug: Astroquery version for Gaia

[troyraen]

The ISP has astroquery version 0.4.7.dev8948 installed which causes the error described in #173. Running pip install -r requirements.txt does not update astroquery to a newer version. So astroquery has to be manually uninstalled and reinstalled, otherwise it throws an error. Further, when light curves are fetched in parallel the error gets lost in the background and does not get display -- so Gaia data does not get loaded and the user does not get any notification that something went wrong.

(Yes, the ISP should be updated. But I think we should prevent the notebook from allowing this to happen, regardless of ISP.)

[troyraen]

I wonder if we should add a try/except at the top of the notebook that does something like this:

try:
    from astroquery.gaia import Gaia
    Gaia.launch_job_async(args)  # launch a trivial job
except TypeError:  # this is the error that gets thrown
    pip uninstall astroquery
    pip install astroquery

[bsipocz]

Minor comment: pip install -U --pre astroquery should be the way to go rather than uninstall and install.

I'm close enough to releasing 0.4.7, so this may naturally get resolved soon. After that I plan to switch to a more frequent, date-based release model, so tagged releases will be more frequent.

[troyraen]

In that case I think it's probably fine to wait for v0.4.7. I'll leave this open to remind us to update requirements.txt at that time.

[troyraen]

Using astroquery v0.4.7, the Gaia call is throwing a new error. Pasted below. I have not looked into it and don't know whether our code needs to change or if there is a bug in somewhere else in the chain.

KeyError                                  Traceback (most recent call last)
Cell In[3], line 4
      1 gaiastarttime = time.time()
      3 # get Gaia light curves
----> 4 df_lc_gaia = gaia_get_lightcurves(sample_table, search_radius=1/3600, verbose=0)
      6 # add the resulting dataframe to all other archives
      7 df_lc.append(df_lc_gaia)

File ~/raen/fdn-tmp/light_curves/code_src/gaia_functions.py:45, in gaia_get_lightcurves(sample_table, search_radius, verbose)
     40     return MultiIndexDFObject()
     42 # Extract Light curves ===============
     43 # request the EPOCH_PHOTOMETRY from the Gaia DataLink Service
---> 45 gaia_df = Gaia_retrieve_epoch_photometry(gaia_table)
     47 #if the epochal photometry is empty, return an empty dataframe
     48 if len(gaia_df) == 0:

File ~/raen/fdn-tmp/light_curves/code_src/gaia_functions.py:138, in Gaia_retrieve_epoch_photometry(gaia_table)
    118 """
    119 Function to retrieve EPOCH_PHOTOMETRY catalog product for Gaia
    120 entries using the DataLink. Note that the IDs need to be DR3 source_id and needs to be a list.
   (...)
    132     
    133 """
    135 # gaia datalink server has a threshold of max 5000 requests,
    136 # so we break the input datasets into chunks of size <=5000 sources
    137 # and then send each chunk into the datalink server
--> 138 ids = list(gaia_table["source_id"])
    139 dl_threshold = 5000  # Datalink server threshold
    140 ids_chunks = list(Gaia_chunks(ids, dl_threshold))

File /opt/conda/envs/science_demo/lib/python3.11/site-packages/astropy/table/table.py:2061, in Table.__getitem__(self, item)
   2059 def __getitem__(self, item):
   2060     if isinstance(item, str):
-> 2061         return self.columns[item]
   2062     elif isinstance(item, (int, np.integer)):
   2063         return self.Row(self, item)

File /opt/conda/envs/science_demo/lib/python3.11/site-packages/astropy/table/table.py:264, in TableColumns.__getitem__(self, item)
    253 """Get items from a TableColumns object.
    254 
    255 ::
   (...)
    261   tc[1:3] # <TableColumns names=('b', 'c')>
    262 """
    263 if isinstance(item, str):
--> 264     return OrderedDict.__getitem__(self, item)
    265 elif isinstance(item, (int, np.integer)):
    266     return list(self.values())[item]

KeyError: 'source_id'

[bsipocz]

Dang. Can you open a new issue for the new error, as it would be cleaner? I suspect what the error is, but it might be upstream to the server and I may need to move the issue to astroquery, or at least have a standalone failing example that can be referred to.