feat(kno-9713): add reusable step marshalling support to CLI #589

MikeCarbone · 2025-09-09T16:44:01Z

Description

Added support for reusable steps in the CLI. This implementation includes:

Core functionality for reading, writing, and processing reusable step data
Helpers for validating and managing reusable step directories
Support for extracting and inlining content from reusable steps
Integration with the existing marshal system

The implementation follows the same pattern as other resource types (workflows, partials, etc.) with specialized handlers for the reusable step structure. It includes the necessary types, processors, and utilities to manage reusable steps both individually and in bulk.

Tasks

KNO-1234

linear · 2025-09-09T16:44:05Z

KNO-9713 [CLI] add reusable step marhsal logic

MikeCarbone · 2025-09-09T16:44:17Z

feat(kno-9713): add reusable step marshalling support to CLI #589 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

MikeCarbone · 2025-09-09T17:19:19Z

package.json

@@ -91,7 +91,8 @@
    "postpack": "shx rm -f oclif.manifest.json",
    "prepack": "yarn build && oclif manifest && oclif readme",
    "test": "mocha --forbid-only \"test/**/*.test.ts\"",
-    "version": "oclif readme && git add README.md"
+    "version": "oclif readme && git add README.md",
+    "check": "yarn run lint && yarn run format.check && yarn run type.check"


tests kept failing in the PR for various reasons here, I prefer them to fail locally so I can quick fix. Running the individual commands was cumbersome, added this to check everything before I submit the PR

MikeCarbone · 2025-09-09T17:19:58Z

test/lib/marshal/message-type/processor.test.ts

@@ -53,7 +53,7 @@ const remoteMessageType: MessageTypeData<WithAnnotation> = {
  },
 };

-describe("lib/marshal/message-typel/processor", () => {
+describe("lib/marshal/message-type/processor", () => {


noticed this, small fix

MikeCarbone · 2025-09-09T17:20:23Z

src/lib/resources.ts

@@ -3,7 +3,7 @@ import type { ResourceType } from "./run-context";
 // TODO Remove this once hidden option is removed from message types / guides
 export type NonHiddenResourceType = Exclude<
  ResourceType,
-  "message_type" | "guide"
+  "message_type" | "guide" | "reusable_step"


we'll hide it for now before release

MikeCarbone · 2025-09-09T17:21:08Z

src/lib/marshal/reusable-step/types.ts

+// TODO: update with actual shape from mAPI
+export type ReusableStepData<A extends MaybeWithAnnotation = unknown> = A & {


wasn't sure which keys belonged here — need to dig into this when I add the resource to mAPI

Check Control.Marshal.Schemas.ReusableStep.ReusableStepSchema, in particular the dump/2 function.

MikeCarbone · 2025-09-09T17:22:49Z

src/lib/marshal/reusable-step/reader.ts

@@ -0,0 +1,220 @@
+import path from "node:path";


a lot of these files are copy / pasted from other resources. I'm not entirely sure about the whats & whys of what's going on, but I think I included the bare minimum, just what we needed before adding to mAPI. Please lmk if there is something here that does not belong here yet

thomaswhyyou · 2025-09-10T19:55:55Z

src/lib/marshal/reusable-step/helpers.ts

+  | ReusableStepDirTarget
+  | ReusableStepsIndexDirTarget;
+
+export const ensureValidCommandTarget = async (


Not used anywhere?

thomaswhyyou · 2025-09-10T20:05:35Z

src/lib/marshal/reusable-step/index.ts

+export * from "./processor.isomorphic";
+export * from "./reader";
+export * from "./types";
+export * from "./writer";


The reader and writer modules are not really used anywhere? They are typically needed for the pull and push commands, and I don't think we have any commands for reusable steps.

If this PR is for dashboard to show commit diffs, you just need the buildReusableStepDirBundle helper, and here's an example of only those changes: #498

thomaswhyyou · 2025-09-10T20:14:34Z

src/lib/marshal/reusable-step/types.ts

+// TODO: update with actual shape from mAPI
+export type ReusableStepData<A extends MaybeWithAnnotation = unknown> = A & {


Check Control.Marshal.Schemas.ReusableStep.ReusableStepSchema, in particular the dump/2 function.

thomaswhyyou · 2025-09-10T20:17:25Z

src/lib/marshal/reusable-step/types.ts

+  id: string;
+  key: string;
+  /** The step content for this reusable step's current version. Currently only HttpFetch is supported. */
+  step: HttpFetchStepData;


Continuing from above -

For example, I don't think there is a step field, because we flatten the object in dump/2, so the data payload you get for a reusable step is:

step object - ref + key, type, environment, created_at, updated_at, sha

https://github.com/knocklabs/control/blob/75aba3b0d50b2ec37e35608d2a0d2659f3c984eb/backend/lib/control/marshal/schemas/reusable_step/reusable_step_schema.ex#L30-L50

thomaswhyyou · 2025-09-10T20:28:20Z

src/lib/marshal/reusable-step/processor.isomorphic.ts

+import { prepareResourceJson } from "../shared/helpers.isomorphic";
+import { ReusableStepData } from "./types";
+
+export const REUSABLE_STEP_JSON = "reusable-step.json";


Suggested change

export const REUSABLE_STEP_JSON = "reusable-step.json";

export const REUSABLE_STEP_JSON = "reusable_step.json";

thomaswhyyou · 2025-09-10T20:40:34Z

src/lib/marshal/shared/helpers.isomorphic.ts

@@ -19,7 +20,8 @@ type ResourceData<A extends WithAnnotation> =
  | PartialData<A>
  | WorkflowData<A>
  | MessageTypeData<A>
-  | GuideData<A>;
+  | GuideData<A>
+  | ReusableStepData<A>;


I think we will need to specify readonly fields for the reusable step schema here: https://github.com/knocklabs/control/blob/b4c72f5a52f42ddf75265c11d21fafe2ee50b2fc/backend/lib/control/marshal/schemas/reusable_step/reusable_step_schema.ex#L9

Basically all the fields in @dump_required_fields.

thomaswhyyou · 2025-09-10T21:02:24Z

src/lib/marshal/reusable-step/processor.isomorphic.ts

+ * its file content. It includes the extractable fields, which are extracted out
+ * and added to the bundle as separate files.
+ */
+export const buildReusableStepDirBundle = (


This probably works for a http fetch step which I believe is the only step type we support with reusable steps, but
one caveat to flag here is that this will likely break once you start supporting other step types.

The proper way to do this would be to refactor the buildWorkflowDirBundle helper, and extract out the part that handles a single step. In ReusableStepSchema, the step field is technically WorkflowStepSchema which can be any step type. But that's more work to refactor.

I'm okay duplicating code and shipping what can work for http steps only - but it will be easy to forget this if/when we ever support additional step types for resuable steps.

MikeCarbone changed the title ~~mike-kno-9713-cli-add-reusable-step-marhsal-logic~~ Add reusable step support to CLI Sep 9, 2025

MikeCarbone changed the title ~~Add reusable step support to CLI~~ feat(kno-9713): add reusable step marshalling support to CLI Sep 9, 2025

MikeCarbone marked this pull request as ready for review September 9, 2025 16:45

MikeCarbone force-pushed the 09-09-mike-kno-9713-cli-add-reusable-step-marhsal-logic branch 3 times, most recently from 9267071 to eec0f18 Compare September 9, 2025 17:05

mike-kno-9713-cli-add-reusable-step-marhsal-logic

08e117d

MikeCarbone force-pushed the 09-09-mike-kno-9713-cli-add-reusable-step-marhsal-logic branch from eec0f18 to 08e117d Compare September 9, 2025 17:13

MikeCarbone commented Sep 9, 2025

View reviewed changes

MikeCarbone requested review from cjbell, meryldakin and thomaswhyyou September 9, 2025 17:23

thomaswhyyou reviewed Sep 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(kno-9713): add reusable step marshalling support to CLI #589

feat(kno-9713): add reusable step marshalling support to CLI #589

Uh oh!

MikeCarbone commented Sep 9, 2025 •

edited

Loading

Uh oh!

linear bot commented Sep 9, 2025

Uh oh!

MikeCarbone commented Sep 9, 2025

Uh oh!

MikeCarbone Sep 9, 2025

Uh oh!

MikeCarbone Sep 9, 2025

Uh oh!

MikeCarbone Sep 9, 2025

Uh oh!

MikeCarbone Sep 9, 2025

Uh oh!

thomaswhyyou Sep 10, 2025

Uh oh!

MikeCarbone Sep 9, 2025

Uh oh!

thomaswhyyou Sep 10, 2025

Uh oh!

thomaswhyyou Sep 10, 2025

Uh oh!

thomaswhyyou Sep 10, 2025

Uh oh!

thomaswhyyou Sep 10, 2025

Uh oh!

thomaswhyyou Sep 10, 2025

Uh oh!

thomaswhyyou Sep 10, 2025

Uh oh!

thomaswhyyou Sep 10, 2025

Uh oh!

Uh oh!

		// TODO: update with actual shape from mAPI
		export type ReusableStepData<A extends MaybeWithAnnotation = unknown> = A & {

	export const REUSABLE_STEP_JSON = "reusable-step.json";
	export const REUSABLE_STEP_JSON = "reusable_step.json";

feat(kno-9713): add reusable step marshalling support to CLI #589

Are you sure you want to change the base?

feat(kno-9713): add reusable step marshalling support to CLI #589

Uh oh!

Conversation

MikeCarbone commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tasks

Uh oh!

linear bot commented Sep 9, 2025

Uh oh!

MikeCarbone commented Sep 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MikeCarbone commented Sep 9, 2025 •

edited

Loading