Skip to content

feat: add CLI tool execution with dynamic schema prompting#22

Merged
gabe-l-hart merged 8 commits intocontextforge-org:mainfrom
MatthewGrigsby:feat/tools-execute-cli
Feb 26, 2026
Merged

feat: add CLI tool execution with dynamic schema prompting#22
gabe-l-hart merged 8 commits intocontextforge-org:mainfrom
MatthewGrigsby:feat/tools-execute-cli

Conversation

@MatthewGrigsby
Copy link
Contributor

@MatthewGrigsby MatthewGrigsby commented Feb 12, 2026

Description

Add CLI support for executing tools with dynamic JSON Schema prompting, so users can run tool calls from cforge without switching to the desktop app or admin UI.

Closes #5

Changes

  • Add cforge tools execute <tool-id> command and wire it into cforge main command registration.
  • Support optional --data <file.json> for prefilled arguments from a JSON object file.
  • Add prompt_for_json_schema(...) in cforge/common.py to drive interactive prompts from dynamic JSON Schemas.
  • Resolve local $ref / $defs references (including array/object references), detect cycles, and reject unsupported external references.
  • Prompt only missing required fields when --data is provided.
  • Improve plugin CLI UX:
    • Move case-insensitive enum helper to shared CaseInsensitiveEnum in cforge/common.py.
    • Parse plugins list --mode case-insensitively (e.g. --mode EnFoRcE).
    • Improve plugin 404 hinting to distinguish missing plugin (plugins get) from admin API unavailability.
  • Add README examples for tools execute.

Testing

  • uv run pytest --no-cov tests/commands/resources/test_tools.py tests/commands/resources/test_plugins.py tests/test_common.py
  • uv run pytest -o addopts='' --cov=cforge.commands.resources.tools --cov=cforge.commands.resources.plugins --cov=cforge.common --cov-report=term-missing:skip-covered tests/commands/resources/test_tools.py tests/commands/resources/test_plugins.py tests/test_common.py
  • Manual validation scenarios documented and exercised in test-cfg-cli/MANUAL_TEST_CASES.md for:
    • tools execute (interactive + --data)
    • $ref object/array schema handling
    • external $ref rejection
    • plugin mode case-insensitive parsing
    • plugin 404 hint behavior

Discussion

  • prompt_for_json_schema currently supports local JSON Pointer references (#/...) and intentionally rejects external refs.
  • Full-repo coverage remains governed by the repository’s global pytest coverage gate; this PR adds targeted coverage for the new command paths and helper logic.

Definition Of Done

  • This PR contains documentation
    • Yes
    • NO (explain)
    • N/A
  • This PR contains unit tests
    • Yes
    • NO (explain)
    • N/A
  • This PR has been tested for backwards compatibility
    • Yes
    • NO (explain)
    • N/A

Follow-up update for a5aacb4

  • Added targeted tests/test_common.py cases to fully cover prompt_for_json_schema edge paths ($ref, enum parsing, optional/required branches, scalar/array/object prompts,
    and additionalProperties behavior).
  • Added minimal pragma annotations in cforge/common.py for defensive/unreachable branches only.
  • Re-ran full suite: uv run pytest passes, and total coverage remains 100%.

Signed-off-by: Matthew Grigsby <38010437+MatthewGrigsby@users.noreply.github.com>
Signed-off-by: Matthew Grigsby <38010437+MatthewGrigsby@users.noreply.github.com>
Copy link
Collaborator

@gabe-l-hart gabe-l-hart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great new feature! I've tested it directly and it works nicely for simple tool calls.

start mcp server

python3 -m mcpgateway.translate      --stdio "docker run --rm -i ghcr.io/ibm/fast-time-server:latest -transport=stdio"      --expose-sse      --port 9005

boot CF

rm tmp/*; CONTEXTFORGE_HOME=$PWD/tmp cforge serve --port 5555 --no-auth
# Register in the UI

test with prompting

PORT=5555 CONTEXTFORGE_HOME=$PWD/tmp cforge tools execute 7a43e085175545f58d214c329889d2a9

test with data

PORT=5555 CONTEXTFORGE_HOME=$PWD/tmp cforge tools execute 7a43e085175545f58d214c329889d2a9 --data <(echo '{"timezone": "EST"}')

I've got a few NITs and a large-ish request to refactor the new prompt_for_json_schema to avoid code duplication with prompt_for_schema.

Signed-off-by: Matthew Grigsby <38010437+MatthewGrigsby@users.noreply.github.com>
Signed-off-by: Matthew Grigsby <38010437+MatthewGrigsby@users.noreply.github.com>
- Skip final jsonschema validation for prompt_for_schema (prompt schema is lossy)

- Add schema_validation helper and regression coverage

- Bump Typer minimum version

Signed-off-by: Matthew Grigsby <38010437+MatthewGrigsby@users.noreply.github.com>
- Add docstrings for nested prompting helpers

- Exclude generated cforge/_version.py from interrogate hook

Signed-off-by: Matthew Grigsby <38010437+MatthewGrigsby@users.noreply.github.com>
@MatthewGrigsby
Copy link
Contributor Author

@gabe-l-hart

I incorporated the refactor + NITs in the latest commits on this branch:

  • Split the former monolithic cforge/common.py into cforge/common/ modules (console, errors, http, prompting, render, schema_validation) and mirrored the same split for tests by moving tests/test_common.py into tests/common/.
  • prompt_for_schema and prompt_for_json_schema now share a single core implementation (_prompt_from_json_schema) in cforge/common/prompting.py; only schema construction differs.
  • tools execute flow cleanup:
    • Validate --data (exists + JSON object) before fetching the tool (fail-fast).
    • Derive prefilled_data + prompt_optional once, and call prompt_for_json_schema(...) exactly once.
    • Removed defensive parsing for stringified inputSchema; it must be a JSON object (or we fall back to an empty object schema when missing).
  • Plugins --mode case-insensitivity: switched mode to str + _parse_plugin_mode() so mixed-case values (e.g. EnFoRcE) don’t fail early in Click/Typer enum validation; added coverage.
  • Prompt schema validation regression: skip final jsonschema validation for prompt_for_schema (Pydantic-derived prompt schema is intentionally lossy for some types), while keeping validation for real JSON Schemas; added regression coverage.
  • Deps: added jsonschema to pyproject.toml because prompt_for_json_schema now validates prompted payloads against JSON Schema (tool inputSchema) before execution; also bumped the Typer minimum version.

Tests: python -m pytest -q is green. Manual test cases also pass.

Copy link
Collaborator

@gabe-l-hart gabe-l-hart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love the blowup of common.py. Much more understandable now. I do have some questions about whether we can use some more standardized jsonschema and pydantic functions to reduce the type/schema munging a bit.

Refactors prompting to be JSON-Schema-first. prompt_for_schema() now uses Pydantic model_json_schema() and tool schemas and Pydantic models go through the same prompting path. Removes the custom x-* hint fields and replaces them with schema-based prompting heuristics (including array-of-strings CSV input and optional object include gating). Fixes default rendering to distinguish “default key present” vs absent and renders defaults via json.dumps.

Adds validate_schema() for early JSON Schema validation, updates tools execute to fail fast on empty IDs and to accept string-encoded schemas when they decode to an object, and restores recursive dict key-type validation for nested Pydantic models. Updates tests accordingly and adjusts interrogate exclusions for generated cforge/_version.py. Cleans up .gitignore duplicates.
Copy link
Collaborator

@gabe-l-hart gabe-l-hart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there! A few small change requests left:

  • Remove the dead function in prompting.py
  • Comment for why we retain the local-only $ref resolution vs using third party
  • Revert changes to .gitignore (unless you want to discuss this one further)
  • Open question on dependency lower-bound bump

Signed-off-by: Matthew Grigsby <38010437+MatthewGrigsby@users.noreply.github.com>
Copy link
Collaborator

@gabe-l-hart gabe-l-hart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ship it! Thanks for this great progress

@gabe-l-hart gabe-l-hart merged commit 33b6e50 into contextforge-org:main Feb 26, 2026
4 checks passed
@MatthewGrigsby MatthewGrigsby deleted the feat/tools-execute-cli branch February 26, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Support running tools

2 participants