Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
c9f4860
Remove byteorder stubs
MatrixEditor Jan 12, 2026
af16bd1
Merge options.py stub file
MatrixEditor Jan 12, 2026
4d32f23
Merge registry.py stubs
MatrixEditor Jan 12, 2026
6c7d160
Merge shared.py
MatrixEditor Jan 12, 2026
d6bc431
Merge exception.py stubs
MatrixEditor Jan 12, 2026
3e008d8
Merge _common.py stubs
MatrixEditor Jan 12, 2026
516af5d
Merge context.py stubs
MatrixEditor Jan 12, 2026
43e3cc4
Merge fields._base.py stubs
MatrixEditor Jan 12, 2026
8ca421f
Merge fields._mixin stubs
MatrixEditor Jan 12, 2026
d9572b7
Merge fields.common stubs
MatrixEditor Jan 12, 2026
9925969
Updated caterpillar.abc code documentation
MatrixEditor Jan 14, 2026
49417a7
Update getstruct typing annotations
MatrixEditor Jan 14, 2026
5fbc291
Remove support for call epressions in ContextPath
MatrixEditor Jan 14, 2026
c74da64
Field: allow _ContainsStruct objects
MatrixEditor Jan 14, 2026
25163ad
Renamed ssize_t and size_t to allow new type syntax
MatrixEditor Jan 14, 2026
7a40570
ConditionalChain accepts boolean as condition
MatrixEditor Jan 14, 2026
3b349bb
Merge crypto.py stubs
MatrixEditor Jan 14, 2026
1c5c26a
Fix Sha1 Digest length
MatrixEditor Jan 14, 2026
d585dc7
Merge hook.py and net.py stubs
MatrixEditor Jan 14, 2026
b877b3f
Merge varint.py and pointer.py stubs
MatrixEditor Jan 14, 2026
62ddb22
Add new extended syntax support
MatrixEditor Jan 14, 2026
49b9a67
Merge bitfield and struct stubs
MatrixEditor Jan 14, 2026
e5b2d8e
Update public exports and shortcuts
MatrixEditor Jan 14, 2026
002b4c0
Rewrite examples and testcases (typing)
MatrixEditor Jan 14, 2026
de47241
Update struct and bitfield decorators to respect kwonly
MatrixEditor Jan 14, 2026
549a1b6
Make byteorder dynamic by default
MatrixEditor Jan 23, 2026
d0e291f
Update examples and tests to use the new syntax
MatrixEditor Jan 23, 2026
7635ae6
Fix actionlike annotations
MatrixEditor Jan 23, 2026
e9a6dfc
Refactor padding struct
MatrixEditor Jan 23, 2026
fdb2bd8
Start updating documentation to v2.8.0
MatrixEditor Jan 23, 2026
20c627d
Add new testcases
MatrixEditor Jan 30, 2026
9a7fac7
Add bitfield_factory along with a mixin type
MatrixEditor Jan 30, 2026
79d1438
Add docs for extended syntax
MatrixEditor Jan 31, 2026
6a02754
Update documentation to include recent changes
MatrixEditor Jan 31, 2026
3c31d83
Add new types module to docs
MatrixEditor Jan 31, 2026
718354f
Backport fix for 3.10
MatrixEditor Feb 1, 2026
e08fd0f
Add changelog for 2.8.0 release
MatrixEditor Feb 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,40 @@
# Changelog

## [2.8.0] - Extended Syntax

### Added

- New concept: *extended* syntax featuring typing compliance; introduces shortcut `f` to annotate fields and `Invisible()` to hide fields from the constructor
- `struct_factory.mixin` and `bitfield_factory.mixin` both provide default function wrappers for packing and unpacking data as well as typing fixes for operating on types directly.
- New `parentctx` context path
- Dynamic endian is now used by default (i. e. can be changed via order parameter in pack() or unpack())
- New generic `Timestamp` class
- A number of new test cases
- New module: `caterpillar.types`: defines default types that can be used as annotations within struct definitions
- `O_DEFAULT_ENDIAN` as a global option to set a global default byteorder
- `O_DEFAULT_ARCH` same concept for arch-like objects
- new 'strict' option to `Enum` struct
- `ATTR_PACK` and `ATTR_UNPACK` to *caterpillar.shared*
- `'order'` and `'arch'` options to pack() and unpack(), which temporarily change the global endianess or arch (compatible with Dynamic byteorder)

### Changes

- Merge all stub files with their corresponding Python files. All typing is now inline.
- `padding` struct now has its own dedicated class
- sizeof() now always returns an integer
- ContextPath: dropped support for the call action
- Field.get_name() now always returns a string
- Rename `ssize` and `size` to `pssize` and `psize` in caterpillar.fields exports
- options.get_flags() always returns a list

### Fixes

- IntFlag support for `Enum` structs
- Fix incorrect Sha1 Digest length
- Add missing global exports in `caterpillar.py`
- Bitfield, Struct and Sequence now respect fields with already configured byteorder
- Prefixed struct now does not require `as_field=True` when calling pack() or unpack()

## [2.7.0] - Dynamic Byteorder

### Added
Expand Down
104 changes: 78 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,41 +23,83 @@ options will be added in the future. Documentation is [here >](https://matrixedi
* insert proper types into the class definition to support documentation and
* it helps you to create cleaner and more compact code.
* There is also a feature that lets you dynamically change the endian within a struct!
* You can even extend Caterpillar and write your parsing logic in C or C++!!
* You can even extend Caterpillar and write your parsing logic in C or C++
* All struct definitions can be typing compliant!!! (tested with pyright)

> [!NOTE]
> Python 3.14 breaks `with` statements in class definitions since `__annotations__` are added at the end
> of a class definition. Therefore, `Digest` and conditional statements **ARE NOT SUPPORTED** using the `with` syntax in Python 3.14+.
> As of version `2.4.5` the `Digest` class has a counterpart (`DigestField`), which can be used to manually specify a digest without
> the need of a `ẁith` statement.
## Give me some code!

*The following code is typing compliant, meaning your static type checker won't*
*scream at you when developing with this code*.

## Give me some code!
<details>
<summary><i>If you want to check out the default syntax, open this block.</i></summary>

```python
from caterpillar.py import *
from caterpillar.types import *

@bitfield(order=LittleEndian)
class Header:
version : 4 # 4bit integer
valid : 1 # 1bit flag (boolean)
ident : (8, CharFactory) # 8bit char
version : 4 # 4bit integer
valid : 1 # 1bit flag (boolean)
ident : (8, CharFactory) # 8bit char
# automatic alignment to 16bits

@struct(order=LittleEndian)
THE_KEY = b"ITS MAGIC"

@struct(order=LittleEndian, kw_only=True)
class Format:
magic : b"ITS MAGIC" # Supports string and byte constants directly
magic : THE_KEY # Supports string and byte constants directly
header : Header
a : uint8 # Primitive data types
b : Dynamic + int32 # dynamic endian based on global config
length : uint8 # String fields with computed lengths
name : String(this.length) # -> you can also use Prefixed(uint8)
a : uint8 # Primitive data types
b : Dynamic + int32 # dynamic endian based on global config
length : uint8 # String fields with computed lengths
name : String(this.length) # -> you can also use Prefixed(uint8)

# custom actions, e.g. for hashes
_hash_begin : DigestField.begin("hash", Md5_Algo)
# Sequences with prefixed, computed lengths -+ part of the MD5 hash
names : CString[uint8::] # |
# -+
# automatic hash creation and verification + default value
hash : Md5_Field("hash", verify=True)

# Creation, packing and unpacking remains the same
```

_hash_begin : DigestField.begin("hash", Md5_Algo) # custom actions, e.g. for hashes
names : CString[uint8::] # Sequences with prefixed, computed lengths
hash : Md5_Field("hash", verify=True) = None # automatic hash creation and verification + default value
</details>

# Instantiation (keyword-only arguments, magic is auto-inferred):
```python
from caterpillar.py import *
from caterpillar.types import *

@bitfield(order=LittleEndian)
class Header:
version : int4_t # 4bit integer
valid : int1_t # 1bit flag (boolean)
ident : f[str, (8, CharFactory)] # 8bit char
# automatic alignment to 16bits

THE_KEY = b"ITS MAGIC"

@struct(order=LittleEndian, kw_only=True)
class Format:
magic : f[bytes, THE_KEY] = THE_KEY # Supports string and byte constants directly
header : Header
a : uint8_t # Primitive data types
b : f[int, Dynamic + int32] # dynamic endian based on global config
length : uint8_t # String fields with computed lengths
name : f[str, String(this.length)] # -> you can also use Prefixed(uint8)

# custom actions, e.g. for hashes
_hash_begin : f[None, DigestField.begin("hash", Md5_Algo)] = None
# Sequences with prefixed, computed lengths -+ part of the MD5 hash
names : f[list[str], CString[uint8::]] # |
# -+
# automatic hash creation and verification + default value
hash : f[bytes, Md5_Field("hash", verify=True)] = b""

# Creation (keyword-only arguments, magic is auto-inferred):
obj = Format(
header=Header(version=2, valid=True, ident="F"),
a=1,
Expand All @@ -66,22 +108,32 @@ obj = Format(
name="foo",
names=["a", "b"]
)
# Packing the object, reads as 'PACK obj FROM Format'

# Packing the object; reads as 'PACK obj FROM Format'
# objects of struct classes can be packed right away
blob = pack(obj, Format)
data_le = pack(obj, Format)
# results in: b'ITS MAGIC0*\x01\x02\x00\x00\x00\x03foo\x02a\x00b\x00)\x9a...'

# Unpacking the binary data, reads as 'UNPACK Format FROM blob'
obj2 = unpack(Format, blob)
obj2 = unpack(Format, data_le)
assert obj2.names == obj.names

# to pack with a different endian for field 'b', use _order
data = pack(obj, Format, _order=BigEndian)
# to pack with a different endian for fields 'a' and 'b', use 'order'
data_be = pack(obj, Format, order=BigEndian)
assert data_le != data_be
```

This library offers extensive functionality beyond basic struct handling. For further details
> [!NOTE]
> Python 3.14 breaks `with` statements in class definitions since `__annotations__` are added at the end
> of a class definition. Therefore, `Digest` and conditional statements **ARE NOT SUPPORTED** using the `with` syntax in Python 3.14+.
> As of version `2.4.5` the `Digest` class has a counterpart (`DigestField`), which can be used to manually specify a digest without
> the need of a `ẁith` statement.

This library offers extensive functionality beyond basic struct definitions. For further details
on its powerful features, explore the official [documentation](https://matrixeditor.github.io/caterpillar/),
[examples](./examples/), and [test cases](./test/).


## Installation

> [!NOTE]
Expand Down
42 changes: 42 additions & 0 deletions docs/sphinx/source/development/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,48 @@ Changelog

*More entries will be added in the future.*

.. _changelog_2.8.0:

[2.8.0] - Extended Syntax
=========================

Added
-----

- New concept: *extended* syntax featuring typing compliance; introduces shortcut :data:`f` to annotate fields and :func:`~caterpillar.model.Invisible` to hide fields from the constructor
- ``struct_factory.mixin`` and ``bitfield_factory.mixin`` both provide default function wrappers for packing and unpacking data as well as typing fixes for operating on types directly.
- New :data:`~caterpillar.context.parentctx` context path
- Dynamic endian is now used by default (i. e. can be changed via order parameter in pack() or unpack())
- New generic :class:`~caterpillar.fields.Timestamp` class
- A number of new test cases
- New module: ``caterpillar.types``: defines default types that can be used as annotations within struct definitions
- :data:`~caterpillar.byteorder.O_DEFAULT_ENDIAN` as a global option to set a global default byteorder
- :data:`~caterpillar.byteorder.O_DEFAULT_ARCH` same concept for arch-like objects
- new 'strict' option to :class:`~caterpillar.fields.Enum` struct
- :data:`~caterpillar.shared.ATTR_PACK` and :data:`~caterpillar.shared.ATTR_UNPACK` to *caterpillar.shared*
- `'order'` and `'arch'` options to pack() and unpack(), which temporarily change the global endianess or arch (compatible with Dynamic byteorder)

Changes
-------

- Merge all stub files with their corresponding Python files. All typing is now inline.
- :class:`~caterpillar.fields.Padding` struct now has its own dedicated class
- sizeof() now always returns an integer
- ContextPath: dropped support for the call action
- Field.get_name() now always returns a string
- Rename `ssize` and `size` to :data:`~caterpillar.fields.pssize` and :data:`~caterpillar.fields.psize` in caterpillar.fields exports
- options.get_flags() always returns a list

Fixes
-----

- IntFlag support for :class:`~caterpillar.fields.Enum` structs
- Fix incorrect Sha1 Digest length
- Add missing global exports in *caterpillar.py*
- Bitfield, Struct and Sequence now respect fields with already configured byteorder
- Prefixed struct now does not require ``as_field=True`` when calling pack() or unpack()


.. _changelog_2.7.0:

[2.7.0] - Dynamic Byteorder
Expand Down
36 changes: 28 additions & 8 deletions docs/sphinx/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,33 @@ efficient.
structs that adjust their size based on the current context. This framework enables you
to write complex structures in a compact and readable manner.

.. tab-set::

.. code-block::
:caption: Simple example of a custom struct
.. tab-item:: Default Syntax

.. code-block::
:caption: Simple example of a custom struct

@struct
class Format:
magic : b"Foo" # constant values
name : CString(...) # \x00-terminated String without a fixed length
value : le + uint16 # little endian encoding
entries: be + CString[uint32::] # arrays with big-endian prefixed length


.. tab-item:: Extended Syntax (>=2.8.0)

.. code-block::
:caption: Simple example of a custom struct with type annotations in-place

@struct(kw_only=True) # keyword-only constructor arguments
class Format:
magic : f[bytes, b"Foo"] = Invisible() # constant values
name : cstr_t # \x00-terminated String without a fixed length
value : f[int, le + uint16] # little endian encoding
entries: f[list[str], be + CString[uint32::]] # arrays with big-endian prefixed length

@struct
class Format:
magic: b"Foo" # constant values
name: CString(...) # \x00-terminated String without a fixed length
value: le + uint16 # little endian encoding
entries: be + CString[uint32::] # arrays with big-endian prefixed length


.. admonition:: Hold up, wait a minute!
Expand Down Expand Up @@ -58,6 +75,9 @@ Format(magic=b'Foo', name='Hello, World!', value=10, entries=['Bar', 'Baz'])
to 4 times. More information are provided when discussing available configuration
:ref:`options`.

- And *YES*, this library tries to enforce strong typing wherever appliacable and supported.
More on that topic later in the tutorial: :ref:`first_steps-extended-syntax`


Where to start?
---------------
Expand Down
17 changes: 10 additions & 7 deletions docs/sphinx/source/library/byteorder.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,24 +15,27 @@ Byteorder
.. autoclass:: DynByteOrder
:members:

.. autofunction:: byteorder(obj, default: Optional[ByteOrder] = None) -> ByteOrder
.. autofunction:: byteorder
.. autofunction:: byteorder_is_little

Standard Byteorder Instances
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoattribute:: caterpillar.byteorder.Native
.. autodata:: caterpillar.byteorder.Native

.. autoattribute:: caterpillar.byteorder.BigEndian
.. autodata:: caterpillar.byteorder.BigEndian

.. autoattribute:: caterpillar.byteorder.LittleEndian
.. autodata:: caterpillar.byteorder.LittleEndian

.. autoattribute:: caterpillar.byteorder.NetEndian
.. autodata:: caterpillar.byteorder.NetEndian

.. autoattribute:: caterpillar.byteorder.SysNative
.. autodata:: caterpillar.byteorder.SysNative

.. autoattribute:: caterpillar.byteorder.Dynamic
.. autodata:: caterpillar.byteorder.Dynamic

.. autodata:: caterpillar.byteorder.LITTLE_ENDIAN_FMT

.. autodata:: caterpillar.byteorder.O_DEFAULT_ENDIAN


Architecture
Expand Down
30 changes: 22 additions & 8 deletions docs/sphinx/source/library/context.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Context classes


.. autoclass:: caterpillar.context.ContextPath
:members:
:members: __call__, __getattribute__, __repr__, parent, parentctx

.. autoclass:: caterpillar.context.ContextLength
:members:
Expand All @@ -29,6 +29,10 @@ Context classes
.. autoclass:: caterpillar.context.ConditionContext


.. autoclass:: caterpillar.context.SetContextVar
:members: __action_pack__, __action_unpack__


Extra options
-------------

Expand All @@ -52,17 +56,15 @@ Extra options
Special paths
-------------

.. autoattribute:: caterpillar.context.this
.. autodata:: caterpillar.context.this

.. autoattribute:: caterpillar.context.ctx
.. autodata:: caterpillar.context.ctx

.. autoattribute:: caterpillar.context.parent
.. autodata:: caterpillar.context.parent

.. autoattribute:: caterpillar.context.root
.. autodata:: caterpillar.context.parentctx

Provides access to the root-level context object.

.. versionadded:: 2.6.0
.. autodata:: caterpillar.context.root


Special Attributes
Expand Down Expand Up @@ -112,6 +114,18 @@ Special Attributes

Points to the root of the entire context hierarchy.

.. autoattribute:: caterpillar.context.CTX_ORDER

Stores the currently used endianess (only in root context).

.. versionadded:: 2.7.0

.. autoattribute:: caterpillar.context.CTX_ARCH

Stores the currently used architecture (only in root context).

.. versionadded:: 2.7.0


Expressions
-----------
Expand Down
7 changes: 7 additions & 0 deletions docs/sphinx/source/library/ext_types.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
.. _library-extended-types:

Types (Extended Syntax)
=======================

.. automodule:: caterpillar.types
:members:
Loading