Pickled items / ItemCollections are large

[TomAugspurger]

When you make a search, select and item, and pickle it, the bytestream will be (surprisingly) large.

import pystac_client
import pystac

catalog = pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1/")
search = catalog.search(collections=["sentinel-2-l2a"], datetime="2022-09-01/2022-09-12", bbox=(68.1766451353734, 7.965534776232332, 97.40256147663614, 35.494009507787766))
items = search.get_all_items()

If we look at the first item:

>>> len(pickle.dumps(items[0]))
51999495

Comare that with PySTAC's size when constructed from a dictionary

>>> len(pickle.dumps(pystac.Item.from_dict(items[0].to_dict())))
15070

I think this is because the items created by pystac-client set the pystac_client.ItemSearch as the root. I haven't figured out the root cause (sorry) yet for why the client objects are so large, but will look more later.

[jsignell]

I think you are right that the client is massive when you pickle it. One solution might be to change how links are serialized for pickle. I messed around with creating __setstate__ and __getstate__ methods on the Link class:

diff --git a/pystac/link.py b/pystac/link.py
index 47bd630..1e17e51 100644
--- a/pystac/link.py
+++ b/pystac/link.py
@@ -114,6 +114,14 @@ class Link(PathLike):
         self.extra_fields = extra_fields or {}
         self.owner = None
 
+    def __getstate__(self) -> dict[str, Any]:
+        """Ensures that pystac does not encode too much information in links when pickling""" 
+        return self.to_dict()
+    
+    def __setstate__(self, state: dict[str, Any]) -> Link:
+        """Ensures that pystac knows how to decode the pickled ``Link``""" 
+        return self.from_dict(state)
+
     def set_owner(self, owner: STACObject | None) -> Link:
         """Sets the owner of this link.

and I was able to go from:

In [1]: import pystac_client
   ...: import pickle
   ...: import pystac
   ...: 
   ...: catalog = pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1/")
   ...: search = catalog.search(collections=["sentinel-2-l2a"], datetime="2022-09-01/2022-09-12", bbox=(68.1766451353734, 7.965534776232332,
   ...:  97.40256147663614, 35.494009507787766))
   ...: items = search.item_collection()

In [2]: %timeit pickle.dumps(items[0])
361 ms ± 1.01 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: len(pickle.dumps(items[0]))
Out[3]: 51916983

In [4]: item = pystac.Item.from_dict(items[0].to_dict())

In [5]: %timeit pickle.dumps(item)
36.3 µs ± 111 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [6]: len(pickle.dumps(item))
Out[6]: 14882

to:

In [1]: import pystac_client
   ...: import pickle
   ...: import pystac
   ...: 
   ...: catalog = pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1/")
   ...: search = catalog.search(collections=["sentinel-2-l2a"], datetime="2022-09-01/2022-09-12", bbox=(68.1766451353734, 7.965534776232332,
   ...:  97.40256147663614, 35.494009507787766))
   ...: items = search.item_collection()

In [2]: %timeit pickle.dumps(items[0])
63 µs ± 172 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [3]: len(pickle.dumps(items[0]))
Out[3]: 14771

In [4]: item = pystac.Item.from_dict(items[0].to_dict())

In [5]: %timeit pickle.dumps(item)
66.3 µs ± 103 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [6]: len(pickle.dumps(item))
Out[6]: 14771

[m-mohr]

What's the usecase behind pickling them? :-D

[TomAugspurger]

Some systems, like Dask, can use pickle to serialize data between workers.

…

On Mon, Dec 4, 2023 at 10:10 AM Matthias Mohr ***@***.***> wrote: What's the usecase behing pickling them? :-D — Reply to this email directly, view it on GitHub <#315 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIXN6GZIZ5M2EV4FEN3YHXYWBBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVEZTIMZZHEYDKMJZQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRRGM3TCMZRGMYDCONHORZGSZ3HMVZKMY3SMVQXIZI> . You are receiving this email because you authored the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[TomAugspurger]

I think Julia fixed this in stac-utils/pystac#1285.

In [4]: len(pickle.dumps(items[0]))
Out[4]: 14714