Skip to content

Conversation

@jhelwig
Copy link
Contributor

@jhelwig jhelwig commented Nov 17, 2025

Problem

The Luminork API currently requires separate HTTP requests for each component operation (create, update, delete, erase). When users need to perform operations on multiple components—such as provisioning infrastructure with dozens of resources, bulk updating configuration, or cleaning up test environments—this creates significant overhead:

  • Network inefficiency: While clients can parallelize requests, they must manage concurrency limits and batch serial groups of parallel requests, adding complexity
  • Transaction coordination: Multiple requests cannot be atomic without complex client-side transaction management
  • Error handling complexity: Partial failures require clients to track which operations succeeded and implement rollback logic
  • API request overhead: Each request carries full HTTP overhead (headers, auth, connection setup)

This becomes particularly problematic for:

  • Infrastructure-as-code scenarios provisioning complete environments (10-100+ components)
  • Template instantiation creating interconnected component graphs
  • Test cleanup requiring deletion of many components
  • Bulk configuration updates across component sets

Impact

Users working with multiple components face:

  • Complex parallelization: Must implement batching logic to respect concurrency limits while maximizing throughput
  • Difficult error recovery: Partial failures in parallel batches require complex client-side rollback logic
  • No transaction atomicity: Related operations across multiple requests cannot guarantee all-or-nothing semantics
  • Increased API load: High request volume even with parallelization

This particularly affects:

  • Automated tooling (CI/CD, infrastructure provisioning scripts)
  • Large-scale testing scenarios
  • Users working with templates that instantiate many components
  • Operations on component hierarchies with managed relationships

Solution

This PR adds four new bulk operation endpoints to the Luminork V1 API:

  1. `POST /components/create_many`: Create multiple components in a single transaction
  2. `PUT /components/update_many`: Update multiple components atomically
  3. `DELETE /components/delete_many`: Soft-delete multiple components (marks for deletion, queues actions)
  4. `POST /components/erase_many`: Hard-delete multiple components (immediate removal, no actions)

Implementation approach:

  • Extracted core operations (`operations.rs`): Shared business logic for component operations used by both single and bulk handlers
  • Refactored existing handlers: `create_component` and `update_component` now use the shared core logic
  • Fail-fast error handling: Bulk operations stop on first error, returning the index of the failing operation
  • Transactional semantics: All operations within a request succeed or fail atomically (no partial application)
  • Performance optimizations: Shared data structures (component lists for managed_by resolution, head component sets) fetched once and reused

After this change, users can perform bulk operations with:

  • Single HTTP request for N operations
  • Atomic transaction guarantees
  • Clear error reporting (failed operation index + underlying error)
  • Identical semantics to single-operation equivalents

Design Rationale

Why fail-fast vs. partial success?

  • Matches SDF API semantics: Bulk endpoints in SDF API use fail-fast behavior; Luminork maintains consistency
  • Data consistency: Partial failures leave system in undefined state
  • Simpler error handling: Clients know operation either fully succeeded or fully failed
  • Transaction semantics: Matches database transaction expectations (all-or-nothing)
  • Clear debugging: Index-based error reporting pinpoints exact failure point

Why lazy-fetch component list?

  • Component list only needed when `managed_by` specified
  • Bulk create_many with no managed relationships avoids expensive list operation
  • Fetch once if needed, reuse across all operations in batch
  • Reduces overhead for common case (no management relationships)

Why same route prefix (`/components/*_many`)?

  • Natural API discoverability (bulk variants adjacent to single operations)
  • Consistent with existing Luminork API patterns
  • Clear semantic relationship (create → create_many, update → update_many)
  • Each bulk endpoint mirrors the semantics of its single-operation counterpart

Deployment Notes

API changes:

  • Four new endpoints under `/v1/w/{workspace_id}/change-sets/{change_set_id}/components/`
  • No changes to existing endpoints (backward compatible)
  • All bulk operations require non-HEAD change set (returns 400 BAD_REQUEST on HEAD)

Monitoring:

  • PostHog events: `api_create_many_components`, `api_update_many_components`, `api_delete_many_components`, `api_erase_many_components`
  • Each event includes `count` field with number of operations in batch
  • Existing single-operation audit logs maintained (queued per-component, published on commit)

Performance characteristics:

  • Single transaction for entire batch (all-or-nothing commit)
  • Shared data structures reused across operations (component lists, head component sets)
  • Linear processing (sequential operations within batch)
  • Network overhead reduced from O(N requests) to O(1 request)

Compatibility

  • No breaking changes: Existing single-operation endpoints unchanged
  • API additions only: New bulk endpoints are additive
  • Semantic consistency: Bulk operations behave identically to N individual operations within a transaction

@github-actions
Copy link

github-actions bot commented Nov 17, 2025

Dependency Review

✅ No vulnerabilities or OpenSSF Scorecard issues found.

Scanned Files

None

@jhelwig jhelwig force-pushed the jhelwig/luminork-bulk-components branch from 3bc0055 to d176963 Compare November 17, 2025 19:29
Add error variant for wrapping component operation errors with index
context during bulk operations. The variant delegates HTTP status to the
underlying error while enhancing the message with the failing item index.

This enables agents to identify which item in a bulk request caused a
failure while maintaining proper error semantics.
Create operations.rs module with create_component_core and
update_component_core functions that contain all business logic and
transactional operations (audit logs, WsEvents).

These functions deliberately exclude HTTP layer concerns (commits,
PostHog tracking) to enable reuse across single and bulk handlers.

The core functions handle:
- Schema resolution and variant selection
- Component creation/updating with proper audit logging
- Attribute and secret management
- Management relationship setup
- WsEvent emission for updates

This separation enables shared logic between single-item and bulk
component operations while maintaining a single source of truth for
business logic.

Also made resolve_secret_id pub(super) in update_component.rs to
support reuse in operations module.
Refactor create_component handler to delegate business logic to
operations::create_component_core while keeping HTTP layer concerns
(validation, PostHog tracking, commit) in the handler.

This establishes the pattern for sharing logic between single-item and
bulk operations while maintaining clear separation of concerns.
Refactor update_component handler to delegate business logic to
operations::update_component_core while keeping HTTP layer concerns
(validation, PostHog tracking, commit) in the handler.

Completes the refactoring pattern for sharing logic between single-item
and bulk operations.
Implement POST /components/create_many endpoint for efficient batch
component creation via MCP tools.

Features:
- Processes multiple creation requests in single transaction
- Lazy caching of component list (only fetches if managed_by used)
- Stop-on-first-error with index context for agent debugging
- Returns results in same order as request for correlation
- All-or-nothing commit semantics

Enables AI agents to minimize API round trips when creating multiple
components.
Implement PUT /components/update_many endpoint for efficient batch
component updates via MCP tools.

Features:
- Processes multiple update requests in single transaction
- Stop-on-first-error with index context for agent debugging
- Returns results in same order as request for correlation
- All-or-nothing commit semantics
- Supports all single-update features (name, resource_id, secrets, attributes)

Enables AI agents to efficiently update multiple components in one
request.
Implement DELETE /components/delete_many endpoint for efficient batch
component soft-deletion via MCP tools.

Features:
- Soft-deletes (marks for deletion) multiple components
- Fetches shared data once (head_components, socket maps, base context)
- Maintains socket maps across deletions for efficiency
- Returns status per component (marked_for_deletion, still_exists_on_head, deleted)
- Stop-on-first-error with index context
- All-or-nothing commit semantics

Enables AI agents to efficiently delete multiple components in one
request.
Implement POST /components/erase_many endpoint for efficient batch
component hard-deletion via MCP tools.

Features:
- Hard-deletes (immediate removal) multiple components
- Fetches shared data once (head_components, socket maps, base context)
- Writes audit log per component erasure (transactional)
- Returns erased component IDs in request order
- Stop-on-first-error with index context
- All-or-nothing commit semantics

Enables AI agents to efficiently erase multiple components in one
request without queueing delete actions.
Add routes for all four bulk component operations:
- POST /create_many - Bulk component creation
- PUT /update_many - Bulk component updates
- DELETE /delete_many - Bulk component soft-deletion
- POST /erase_many - Bulk component hard-deletion

Completes the routing layer for bulk component operations, making all
endpoints accessible via the API.
@jhelwig jhelwig force-pushed the jhelwig/luminork-bulk-components branch from d176963 to 4b15071 Compare November 18, 2025 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants