[ENH]: chroma-metering-macros #4746

c-gamble · 2025-06-03T23:43:58Z

Description of changes

This PR introduces a metering library that provides a procedural macro for defining attributes and events. Please view the relevant ADR (not linked) for additional context.

Todo

First cut
Change disk registry to in-memory
Move registry into macros crate
Realize that first cut actually doesn't work, put everything into one macro
Clean up variable names/functions (readability)
Formalize and expand test suite (robustness)
Add documentation via comments (maintainability)

Test plan

Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

WIP

propel-code-bot · 2025-06-05T21:59:23Z

Introduce Procedural Macro-Based Metering Library (chroma-metering)

This PR introduces a new procedural macro-based metering library (chroma-metering) for the Chroma Rust workspace. The library enables users to define custom 'capabilities' and 'contexts' for structured, reliable, and scalable multi-threaded metering. It supplies a single macro, initialize_metering, which generates all metering infrastructure (traits, forwarding, trait-object downcasting, context management, and async-friendly APIs). Comprehensive documentation, test coverage, and a new workspace member are included.

Key Changes:
• Adds new crate chroma-metering providing a procedural macro system for metering.
• Implements macro-processing logic supporting custom capability/context/handler triples; outputs all boilerplate and safe trait object forwarding.
• Defines runtime (Enter/Exit stack, thread-local handling, custom Receiver integration, safe context downcasting) through generated code.
• Includes detailed in-crate documentation, README, and an extensive async/concurrency-aware test suite.
• Updates Cargo workspace to register and integrate the new crate.

Affected Areas:
• Cargo workspace membership and dependencies
• New codebase in rust/metering/
• Build/test pipeline due to new crate and procedural macro
• Metering-related logic and integration touchpoints

This summary was automatically generated by @propel-code-bot

rust/metering/src/annotations.rs

rust/metering/src/mutators.rs

rust/metering/src/lib.rs

…actoring with deletion

Cargo.toml

rust/metering/Cargo.toml

rust/metering/tests/tests.rs

…per/chroma-metering-lib

rust/metering/Cargo.toml

HammadB

My comments are questions I will ask live

rust/metering/src/capabilities.rs

rust/metering/src/contexts.rs

rust/metering/src/lib.rs

rust/metering/tests/tests.rs

…per/chroma-metering-lib

) ## Description of changes This PR migrates all of the existing metering functionality of Chroma to use the metering library created in #4746. It does not introduce any new metering functionality on top of what was present before. ## Test plan - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes N/A

## Description of changes This PR introduces a metering library that provides a procedural macro for defining attributes and events. Please view the relevant ADR (not linked) for additional context. ## Todo - [x] First cut - [x] Change disk registry to in-memory - [x] Move registry into macros crate - [x] Realize that first cut actually doesn't work, put everything into one macro - [x] Clean up variable names/functions (readability) - [x] Formalize and expand test suite (robustness) - [x] Add documentation via comments (maintainability) ## Test plan - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes WIP

…roma-core#4868) ## Description of changes This PR migrates all of the existing metering functionality of Chroma to use the metering library created in chroma-core#4746. It does not introduce any new metering functionality on top of what was present before. ## Test plan - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes N/A

c-gamble requested review from drewkim and HammadB June 3, 2025 23:43

c-gamble changed the title ~~[ENH]: metering library first cut~~ [ENH]: chroma-metering Jun 3, 2025

c-gamble marked this pull request as ready for review June 5, 2025 21:58

c-gamble requested review from codetheweb, sanketkedia and Sicheng-Pan June 6, 2025 00:06

drewkim reviewed Jun 6, 2025

View reviewed changes

rust/metering/src/annotations.rs Outdated Show resolved Hide resolved

rust/metering/src/mutators.rs Outdated Show resolved Hide resolved

rust/metering/src/lib.rs Outdated Show resolved Hide resolved

c-gamble added 14 commits June 9, 2025 10:14

[ENH]: metering library first cut

f07628d

[CLN]: remove dev comments

b436bc5

[ENH]: consolidate registry crate into macros crate

b857299

[CKPT]: add documentation and clean up code

54a8a4a

[CKPT]: checkpoint for all of the documentation i wrote since i'm ref…

9fff90d

…actoring with deletion

[ENH]: merge into single library using initialize_metering! macro

cf75f92

[ENH]: clean up code

94b3d71

[DOC]: expand on readme

392f355

ckpt

5588fc5

[CLN]: resolve linter errors

022223a

checkpoint

366004e

[ENH]: support foreign macros on unannotated fields

492fe22

[TST]: add test for nested thread closing event

b5455d4

[CLN]: remove mod tests from tests/tests.rs

ed6e4a3

c-gamble force-pushed the cooper/chroma-metering-lib branch from 49241a6 to ed6e4a3 Compare June 9, 2025 17:17