Fast time-series look-up from a 45 GB Zarr file

[justurbo]

I want to look-up 8760 times for a single lat/lon combo in less than a second from 43.82 GB file of wind data containing:

• 8760 times (every hour in a year)
• 721 latitudes (every 0.25° from -90.0° to 90.0°)
• 1440 longitude (every 0.25° from -180.0° to 179.75°)

The best time we achieved for a single-year look-up was 16 seconds for both u100 and v100 wind speed at 100m vectors. We want to have a sub-second look-up for the whole year as such file read will need to happen on every user request in our API.

if __name__ == '__main__':
    start_time = time.time()

    ds = xr.open_dataset("2021.zarr", engine="zarr", chunks={"time": 50})

    print(f"Took {round((time.time() - start_time) * 1000, 2)}ms")

    location = ds.sel(indexers={"latitude": 53.494, "longitude": 9.979}, method='nearest')

    wind_speed = (location.u100.values ** 2 + location.v100.values ** 2) ** 0.5

    print(f"Wind Speed: {wind_speed} m/s")
    print(f"Took {round((time.time() - start_time) * 1000, 2)}ms")

Output:

Took 94.28ms
Wind Speed: [5.8021994 5.504477  5.4270387 ... 9.563195  8.701231  9.133655 ] m/s
Took 16299.59ms

I would be very thankful for any help!

[rabernat]

The performance in this use case is extremely sensitive to chunking. Can you share how your data are chunked on disk?