TESScut timeouts for larger cuts

[TeHanHunter]

I have been using TESScut to cut 90*90 FFIs and it worked fine until recently. Now it is stuck at downloading and then gives a timeout error.

from astroquery.mast import Tesscut
ra = '60' 
dec = '60'
coord = SkyCoord(ra, dec, unit="deg")
sector_table = Tesscut.get_sectors(coordinates=coord)
hdulist = Tesscut.get_cutouts(coordinates=coord, size=90)

Any help is appreciated!

[bsipocz]

cc @jaymedina

[TeHanHunter]

It seems that my pc does not work. The same script works on a cluster. I will find out why.

[StuartLittlefair]

@TeHanHunter did you ever find out why? I am finding the same issue; I can download the cutouts fine if I use the astrocut API directly using requests...

[TeHanHunter]

Hi, I think the problem persists for me. Especially when the number of sectors is large. I was able to download sectors individually to avoid the issue.

…

On Thu, Mar 23, 2023 at 03:08 StuartLittlefair ***@***.***> wrote: @TeHanHunter <https://github.com/TeHanHunter> did you ever find out why? I am finding the same issue; I can download the cutouts fine if I use the astrocut API directly using requests... — Reply to this email directly, view it on GitHub <#2447 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL4U5CIU3M7J4ESDKNLV7VTW5QOJFANCNFSM5ZG46STA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jaymedina]

Hi all, what is the error you're receiving? Is it a 503 Service Temporarily Unavailable error?

[StuartLittlefair]

No, it's a timeout error "timeout of 600 exceeded".

The thing is, using the API with requests takes only about 1 minute...

[jaymedina]

OK can you send me the code that was successful for you?

[StuartLittlefair]

Sure the snippet below works without timing out:

import warnings
import zipfile
from io import BytesIO

import requests
from astropy.io import fits


def download_tesscut(coords, size=90):
    """
    Download all sectors of a target from TESSCut.

    Parameters
    ----------
    coords : astropy.coordinates.SkyCoord
        Coordinates of the target.
    size: int
        Size of the cutout in pixels.
    """
    URL = "https://mast.stsci.edu/tesscut/api/v0.1/astrocut?ra={}&dec={}&y={}&x={}&units=px&product=SPOC&sector=All"
    response = requests.get(URL.format(coords.ra.deg, coords.dec.deg, size, size))
    response.raise_for_status()

    try:
        ZIPFILE = zipfile.ZipFile(BytesIO(response.content), "r")
    except zipfile.BadZipFile:
        message = response.json()
        warnings.warn(message["msg"])
        return []

    cutout_hdus_list = []
    for name in ZIPFILE.namelist():
        CUTOUT = BytesIO(ZIPFILE.open(name).read())
        cutout_hdus_list.append(fits.open(CUTOUT))

        # preserve the original filename in the fits object
        cutout_hdus_list[-1].filename = name

    return cutout_hdus_list

as opposed to Tesscut.get_cutouts(coordinates=coords, size=90) which times out. For an example which is very likely to timeout (many sectors), coordinates of 63.195208, 75.828417 are a suitable test.

[jaymedina]

OK I'll look into this, thanks!

[jaymedina]

Any astroquery.mast requests that hit the upper limit of 600 seconds get hit with a timeout error. You can see the upper limit by calling the TIMEOUT attribute like so:

Tesscut._service_api_connection.TIMEOUT

>>> 600

You can technically override this by reassigning the TIMEOUT parameter to whatever you want, like so:

Tesscut._service_api_connection.TIMEOUT = 5000

After overriding this parameter I was able to get the 90x90 cutouts HDU list for a single sector for coordinates (0, 89) after a runtime of ~689 seconds:

Tesscut._service_api_connection.TIMEOUT = 5000

coords = '0 89'
start = time.time()
cutouts = Tesscut.get_cutouts(coordinates=coords, size=90, sector=19)

print(f'Time lapsed: {str(time.time()-start)}')

>>> Time lapsed: 689.3902568817139

So if you do it this way, keep in mind it's going to take a really long time if you want to retrieve multiple sectors. As a side note, the download_tesscut function you wrote took about 702 seconds to retrieve all 10 sectors that this pair of coords lands in. The way that the HDU lists are unpacked and read is the same between download_tesscut and cutouts.py, so it must have something to do with the way the requests are made through services.py , and will need more looking into.

Either way, this issue tells me we should add a Warning message for cutout sizes that exceed a certain amount of pixels, and maybe add an extra argument to override the default TIMEOUT arg from the conf.py for cases where people need to request for cutouts this large, and I'll do some investigating of why the astroquery.mast service requests to TESSCut take so long. Let me know your thoughts on moving forward this way @bsipocz

---

Edit: Just did some quick plotting to see what the ~maximum recommended cutout size would be that will let astroquery.mast.Tesscut.get_cutouts run without a timeout error after 600s, and it looks like ~30x30 pixel cutouts is around where the max is (this is a conservative calculation). One thing I wanted to ask: was the 90x90 cutout size selected arbitrarily, or what was the reason for choosing a cutout of that size? 1 pixel spans ~21 arcseconds for TESS, so I imagined the cutout sizes to be more localized (10x15, etc). If there is a scientific reason for selecting anything above 30x30, we can move this issue higher up in priority to work on optimizing the request calls to the TESSCut API.