Proposed ScreenShot API changes

[jholveck]

I’m proposing some API changes to ScreenShot that break compatibility in some ways, although common usages will still be fine. This is to help us with our overall release planning, as being detailed in #471.

Change 1: Buffers

Goal

The goal here is to prevent unnecessary copies. Currently, the API design requires us copy the pixel data in many conditions.

Many of these changes refer to the Buffer type. This is in collections.abc as of Python 3.12, and also in typing_extensions if we want to use that for earlier versions. We can alternatively alias it to bytes | bytearray | memoryview on earlier versions.

Proposed API Changes

• _raw: Rename raw to _raw. I’m unclear on whether this was ever intended to be exposed. In the proposed new API, it’s the same to the user as bgra anyway, but making it private gives us more flexibility.
• __init__, from_size: Change the data argument from bytearray to Buffer. This lets _grab_impl pass us a memoryview rather than having to make a new copy of the data.
• bgra, rgb: Return a read-only Buffer instead of a bytes object, avoiding a copy.

Also of note:

• __array_interface__: Not a type change, but I’m thinking we might make the provided array read-only.
• buffer, __buffer__: New method. __buffer__ is a Python 3.12-only feature that lets the ScreenShot object act as a Buffer itself, while the buffer method can be used on any version of Python.

Impact

With just these simple API changes, and no backend-specific changes, the cat-detector and video-capture demos are 12% and 15% faster, respectively. That’s for the whole program, not just the screen capture part. The MSS code I used for this test is at https://github.com/jholveck/python-mss/tree/d38af8a36abe2d7ce9f87f95bca78c7502f2a7ca. The proposed API further allows backends to avoid yet another copy, improving performance even more.

Question: read-write or read-only?

One topic to discuss: currently, the user could modify the pixel data. They had access to the read-write bytearray as sct.raw, and also could modify the raw data via NumPy using __array_interface__.

I’m thinking we may not want to make any guaranteed read-write mechanisms available, since that would mean we can’t cache pixels or rgb. In my current implementation, the .buffer() method can be used to request a read-write buffer, since PyTorch ignores the read-only flag (with a warning). But in general, I don’t see much sense in giving users a way to edit the buffer in-place.

Change 2: ScreenShot types

In anticipation of us offering both CPU- and GPU-side screenshot buffers, I propose we make preparations. I propose we create a new ScreenShot parent class for all possible screenshot types, and a ScreenShotCPU for the CPU-side screenshot objects we have today, which derives from ScreenShot. Some methods, like size, would be on ScreenShot itself. Methods that are only applicable to CPU-side screenshots, like bgra, would be defined on ScreenShotCPU. (A future ScreenShotCUDA or whatever would need to define its own way of giving the user access to the GPU buffer containing pixel data. While it could be named bgra as well, ScreenShotCUDA.bgra method would be a different method interface with a different signature, unrelated to ScreenShotCPU.bgra.)

While there’s not yet a GPU-side capture API available, we may want to make ScreenShot a more generic parent for the future class, so we don’t have to break the API when we do add those.

Python typing generics would allow us to give type checkers the information they need to infer the correct return type from grab. For users who use explicit types in their code, though, this class organization will let them declare the correct types today. That would save us from needing a major release to change the API when we do add GPU capabilities.

[BoboTiG]

I think going read-only is the good way. No need to be hard on preventing users to mess with the data though, lets use read-onlyish objects and go with that. I do not recall seeing someone altering pixel data from a MSS object, I am not worry about such a change.

About the ScreenShot class, this is a good idea to create a parent class + derived.
At this end, can we have a unified API for all subclasses? You're talking about the bgra attribute that can be set only for the ScreenShotCPU, what would it be for the ScreenShotGPU then? Since we will break stuff, I would rather have a unified API whatever the subclass so that people can rely on public attributes without regarding the exact class type.

I mean, lets imagine a user code that I would like to prevent spreading:

sc = mss.grab(...)
if isinstance(sc, ScreenShotCPU):
    data = sc.bgra
else:
    data = sc.xxx

(Is this even possible to prevent such case?)

[disgression] As I wrote this code, I was thinking maybe should we ship helpers (like ScreenShot.to_pil() to convert raw BGRA to a PIL Image object, and that would save us from publishing the bgra attribute). Lets not pollute the conversation here haha.

[jholveck]

I think the key point here is that the user will already know what they want to do with the screenshot, and they will tell MSS that up front.

If someone is capturing for PIL, NumPy, OpenCV, etc., they want CPU memory. If someone is capturing for PyTorch or another GPU-accelerated library, they want the pixels to land directly in GPU memory.

That choice happens when they construct MSS, not after they call grab().

So instead of this:

sc = mss.grab(...)
if isinstance(sc, ScreenShotCPU):
    ...
else:
    ...

The expected usage would look like this:

# CPU capture (default)
with MSS(device="cpu") as sct:
    sc: ScreenShotCPU = sct.grab(mon)
    img = Image.frombytes("RGB", sc.size, sc.bgra, "raw", "BGRX")

# GPU capture
with MSS(device="cuda") as sct:
    sc: ScreenShotCUDA = sct.grab(mon)
    tensor = torch.from_dlpack(sc)  # via __dlpack__()

In the second example, the screenshot object would expose the standard __dlpack__ protocol. DLPack is a small interoperability standard that lets GPU libraries (PyTorch, CuPy, etc.) share GPU memory without copying it back to the CPU. It’s essentially the “buffer protocol for GPU memory.”

The important part is: the user chooses "cpu" or "cuda" explicitly. So they already know which type grab() will return. There’s no need for runtime branching.

---

On the API surface itself:

I don’t think we can realistically give ScreenShotCPU and ScreenShotCUDA identical data-access APIs, because CPU and GPU memory are fundamentally different.

A ScreenShotCPU can safely expose:

• .bgra
• .rgb
• .pixels
• .pixel(x, y)
• possibly .to_pil() (I like that idea; feel free to assign an issue to me! But I think that providing .bgra is still useful for cases we didn't anticipate.)

All of these rely on CPU-addressable memory.

A ScreenShotCUDA, however, cannot expose .bgra as a CPU buffer without performing a device-to-host transfer over PCIe. That would silently defeat the entire point of GPU capture. We should avoid APIs that look identical but hide radically different performance costs.

So my proposal is:

• ScreenShot (base class): common metadata only (size, pos, etc.)
• ScreenShotCPU: CPU-specific data accessors (as today)
• ScreenShotCUDA: GPU-specific interop methods (e.g., __dlpack__)

This keeps the API honest. It avoids surprising implicit downloads. And it allows us to introduce GPU capture later without another breaking reorganization.

If we tried to force a completely unified surface (e.g., .bgra exists everywhere), we would either:

• introduce hidden transfers, or
• return objects that aren’t actually bytes-like, which would break existing assumptions.

I think it’s better to make the device choice explicit and let the returned object reflect that choice.

---

That keeps the model simple:

• MSS places pixels where you ask.
• The screenshot object exposes access methods appropriate to where the pixels live.
• No runtime type-branching is needed because the device decision was already made architecturally.

[BoboTiG]

You are right, this is clearer now. And I am aligned with you about specialized attributes.

I am more and more in fond of helpers to simplify the code: the CPU class could have to_pil(), and to_numpy(); while the GPU could have to_pytorch() (and potentially more with time).