External plugins and middleware support #1455

sandys · 2025-11-26T10:39:20Z

portkey is a brilliant project. however, those of us who are experimenting with portkey via npmjs, find it hard to use plugins and middlwares. Because you need to change the source code of portkey to get plugins to work.

i have added an incremental patch to portkey that allows inclusion of external middlwares/plugins...while maintaining the sanctity of the bundled plugins/middlewares. the readme has been updated with the usage with external middlwares.

additionally there is a sample folder with external middlewares/plugins - which is a copy of the bundled plugins/middlewares (just to demonstrate).

sandys · 2025-11-26T20:14:20Z

external middleware can dynamically register custom routes to Portkey without any core modifications (using external plugins/middlewares)

Key Benefits

Standard Hono Pattern - Uses documented Hono plugin architecture
Zero Core Modifications - Portkey code unchanged except for middleware loader enhancement
Automatic Loading - Works via standard --middlewares-dir flag
Production Ready - Real HMAC verification, proper error handling
Composable - Multiple external middlewares can coexist
Compatible - Works with any npm-installed Portkey version

Why It's Good

✅ Uses Hono's standard patterns
✅ No invasive modifications
✅ Minimal code changes
✅ Proven with working example
✅ Documented with tests
✅ Enables ecosystem of external extensions

Working Middleware Example

File: external-examples/middlewares/webhookSignatureVerifier.js (4.9 KB)
Features:
- Demonstrates plugin-style middleware pattern
- Implements real HMAC-SHA256 signature verification
- Registers two routes: POST /webhooks/verify and GET /webhooks/health
- Proper error handling (401, 403, 400)
- Production-ready code with logging

Copilot

Pull request overview

This PR adds support for loading external plugins and middlewares from custom directories, enabling users to extend Portkey Gateway functionality without modifying its core codebase. The implementation includes CLI flags for specifying plugin/middleware directories, loader utilities for dynamic module loading, and comprehensive documentation with working examples.

Key Changes:

Added --plugins-dir and --middlewares-dir CLI flags to load external plugins and middlewares at runtime
Implemented loader utilities (pluginLoader.ts and middlewareLoader.ts) to dynamically import and register external modules
Updated documentation with detailed guides and working examples of external plugins and middlewares

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/start-server.ts	Added CLI argument parsing and loading logic for external plugins/middlewares with error handling
src/loaders/pluginLoader.ts	New loader utility that scans directories, validates manifests, and dynamically imports plugin handlers
src/loaders/middlewareLoader.ts	New loader utility that imports middleware modules and determines if they're plugin-style or standard middlewares
src/handlers/handlerUtils.ts	Fixed hook conversion to handle already-formatted hooks and corrected camelCase property naming for guardrails
plugins/README.md	Added comprehensive documentation section on creating and using external plugins/middlewares
package.json	Added new dependencies for PostgreSQL support and updated package description
external-examples/test-webhook-verifier.sh	Added test script demonstrating webhook signature verification middleware functionality
external-examples/plugins/default-external/wordCountExternal.js	Example external plugin implementing word count guardrail validation
external-examples/plugins/default-external/regexMatchExternal.js	Example external plugin implementing regex pattern matching validation
external-examples/plugins/default-external/manifest.json	Manifest file defining external plugin metadata and function configurations
external-examples/middlewares/webhookSignatureVerifier.js	Example plugin-style middleware that registers custom webhook verification routes
external-examples/middlewares/loggerExternal.ts	Example standard middleware for logging incoming requests and responses
README.md	Updated main README with extensibility section and CLI options documentation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-29T01:27:51Z

src/start-server.ts

+if (pluginsDir) {
+  console.log('🔌 Loading external plugins from:', pluginsDir);
+  try {
+    const externalPlugins = await loadExternalPlugins([pluginsDir]);


The loadExternalPlugins function accepts an array but only a single directory is passed. Consider either passing pluginsDir directly without wrapping it in an array, or support multiple plugin directories by allowing comma-separated values in the CLI argument.

Copilot · 2025-11-29T01:27:51Z

src/start-server.ts

+if (middlewaresDir) {
+  console.log('⚙️  Loading external middlewares from:', middlewaresDir);
+  try {
+    const externalMiddlewares = await loadExternalMiddlewares([middlewaresDir]);


Similar to the plugin loader, loadExternalMiddlewares accepts an array but only a single directory is passed. Consider passing middlewaresDir directly or supporting multiple directories.

Copilot · 2025-11-29T01:27:51Z

src/loaders/middlewareLoader.ts

+          // If middleware is a function that returns a function, it's a plugin
+          if (middleware.length === 0) {
+            // No parameters - likely middleware() that returns a function
+            try {
+              const result = middleware();
+              if (typeof result === 'function') {
+                // It's a plugin - the result is the app handler
+                isPlugin = true;
+                handler = result;
+              } else {
+                // middleware() returned non-function, use original middleware
+                isPlugin = false;
+                handler = middleware;
+              }
+            } catch (e) {
+              // If calling middleware() throws, treat as standard middleware
+              isPlugin = false;
+              handler = middleware;
+            }


Using middleware.length === 0 to detect plugin-style middleware is unreliable. Arrow functions with no parameters still have length === 0, but so do functions with default parameters or rest parameters. This heuristic could misidentify standard middleware that uses default parameters. Consider requiring explicit metadata or checking if the result is a function that accepts an app parameter.

Suggested change

// If middleware is a function that returns a function, it's a plugin

if (middleware.length === 0) {

// No parameters - likely middleware() that returns a function

try {

const result = middleware();

if (typeof result === 'function') {

// It's a plugin - the result is the app handler

isPlugin = true;

handler = result;

} else {

// middleware() returned non-function, use original middleware

isPlugin = false;

handler = middleware;

}

} catch (e) {

// If calling middleware() throws, treat as standard middleware

isPlugin = false;

handler = middleware;

}

// Prefer explicit metadata to detect plugin-style middleware

if (metadata.isPlugin === true) {

try {

const result = middleware();

if (typeof result === 'function') {

isPlugin = true;

handler = result;

} else {

isPlugin = false;

handler = middleware;

}

} catch (e) {

isPlugin = false;

handler = middleware;

}

} else if (metadata.isPlugin === undefined) {

// Fallback: try calling middleware() and see if it returns a function that takes an app parameter

try {

const result = middleware();

if (typeof result === 'function' && (result.length === 1 || result.name === 'appHandler')) {

isPlugin = true;

handler = result;

} else {

isPlugin = false;

handler = middleware;

}

} catch (e) {

isPlugin = false;

handler = middleware;

}

Copilot · 2025-11-29T01:27:51Z

src/loaders/middlewareLoader.ts

+          // Plugin: middleware() returns (app) => void
+          // Standard: middleware is (c, next) => Promise<any>
+          let isPlugin = false;
+          let handler = middleware;
+
+          // If middleware is a function that returns a function, it's a plugin
+          if (middleware.length === 0) {
+            // No parameters - likely middleware() that returns a function
+            try {
+              const result = middleware();
+              if (typeof result === 'function') {
+                // It's a plugin - the result is the app handler
+                isPlugin = true;
+                handler = result;
+              } else {
+                // middleware() returned non-function, use original middleware
+                isPlugin = false;
+                handler = middleware;
+              }
+            } catch (e) {
+              // If calling middleware() throws, treat as standard middleware
+              isPlugin = false;
+              handler = middleware;
+            }
+          }
+


Calling middleware() without any context or parameters could have unintended side effects if the middleware function performs operations during initialization. Consider documenting this behavior or adding safeguards to prevent potentially harmful operations during detection.

Suggested change

// Plugin: middleware() returns (app) => void

// Standard: middleware is (c, next) => Promise<any>

let isPlugin = false;

let handler = middleware;

// If middleware is a function that returns a function, it's a plugin

if (middleware.length === 0) {

// No parameters - likely middleware() that returns a function

try {

const result = middleware();

if (typeof result === 'function') {

// It's a plugin - the result is the app handler

isPlugin = true;

handler = result;

} else {

// middleware() returned non-function, use original middleware

isPlugin = false;

handler = middleware;

}

} catch (e) {

// If calling middleware() throws, treat as standard middleware

isPlugin = false;

handler = middleware;

}

}

// Plugin: middleware() returns (app) => void, but we avoid calling it to prevent side effects.

// Standard: middleware is (c, next) => Promise<any>

// Detection is now based on metadata.isPlugin to avoid calling middleware() during loading.

let isPlugin = !!metadata.isPlugin;

let handler = middleware;

// If plugin, handler should be the factory function; actual invocation happens at runtime.

Copilot · 2025-11-29T01:27:52Z

package.json

    "@portkey-ai/mustache": "^2.1.3",
    "@smithy/signature-v4": "^2.1.1",
    "@types/mustache": "^4.2.5",
+    "@types/pg": "^8.15.6",


PostgreSQL dependencies (@types/pg, pg, pg-boss) are added but not used in any of the changed files. These dependencies should either be removed if not needed for this PR, or their usage should be documented.

Copilot · 2025-11-29T01:27:52Z

package.json

+    "pg": "^8.16.3",
+    "pg-boss": "^12.3.1",


PostgreSQL dependencies (@types/pg, pg, pg-boss) are added but not used in any of the changed files. These dependencies should either be removed if not needed for this PR, or their usage should be documented.

Copilot · 2025-11-29T01:27:52Z

external-examples/test-webhook-verifier.sh

+  # Build curl command
+  local curl_cmd="curl -s -w '\n%{http_code}' -X $method"
+
+  if [ -n "$headers" ]; then
+    curl_cmd="$curl_cmd $headers"
+  fi
+
+  if [ -n "$data" ]; then
+    curl_cmd="$curl_cmd -d '$data'"
+  fi
+
+  curl_cmd="$curl_cmd '$BASE_URL$url'"
+
+  # Run the test
+  local response=$(eval $curl_cmd)


Using eval with $curl_cmd is a security risk if the command contains untrusted input. While the inputs here are controlled, consider using arrays to build the command and executing it directly without eval.

Suggested change

# Build curl command

local curl_cmd="curl -s -w '\n%{http_code}' -X $method"

if [ -n "$headers" ]; then

curl_cmd="$curl_cmd $headers"

fi

if [ -n "$data" ]; then

curl_cmd="$curl_cmd -d '$data'"

fi

curl_cmd="$curl_cmd '$BASE_URL$url'"

# Run the test

local response=$(eval $curl_cmd)

# Build curl command as an array

local curl_cmd=(curl -s -w '\n%{http_code}' -X "$method")

if [ -n "$headers" ]; then

# Split headers into array elements (assumes headers are space-separated, e.g. "-H 'X: Y' -H 'A: B'")

# Use 'read -a' to split, but this is tricky if headers contain spaces inside quotes.

# Instead, use 'eval set -- $headers' to parse quoted headers safely.

eval set -- $headers

while [ $# -gt 0 ]; do

curl_cmd+=("$1")

shift

done

fi

if [ -n "$data" ]; then

curl_cmd+=(-d "$data")

fi

curl_cmd+=("$BASE_URL$url")

# Run the test

local response

response=$("${curl_cmd[@]}")

Copilot · 2025-11-29T01:27:53Z

external-examples/middlewares/webhookSignatureVerifier.js

+        const isValid =
+          providedSignature.length === expectedSignature.length &&
+          Buffer.from(providedSignature).equals(Buffer.from(expectedSignature));


While using Buffer.equals() provides constant-time comparison, the length check before it (providedSignature.length === expectedSignature.length) is not constant-time and could leak information through timing attacks. Use a dedicated constant-time comparison function that checks length internally, or use crypto.timingSafeEqual().

roh26it · 2025-11-29T01:32:18Z

@sandys thanks for this, it is valuable.

Can you take a look at Copilot's comments?

Removed three unused dependencies from gateway/package.json: - @types/pg (^8.15.6) - pg (^8.16.3) - pg-boss (^12.3.1) Gateway now aligns with upstream Portkey's dependency list. ✅ Task 2: Create External Dependency Installer - Directory scanning - Finds package.json files in external middleware/plugin directories - Peer dependency validation - Checks peer dependencies against gateway's available dependencies - npm install execution - Runs npm install --no-save in each directory - Comprehensive error reporting - Returns detailed installation status with failures and mismatches - Version compatibility checking - Validates semver ranges (^, ~, >=, =) for peer dependencies ✅ Task 3: Integrate Into Startup Flow - Call the installer before loading external plugins/middlewares - Find gateway's package.json from multiple fallback paths (handles both src and built code) - Report installation status and fail fast on peer dependency mismatches - Clear error messages for troubleshooting

sandys · 2025-12-02T16:57:18Z

also added support for external providers now !

npm run start:node --
--providers-dir=./external-examples/providers
--plugins-dir=./external-examples/plugins
--middlewares-dir=./external-examples/middlewares
--port=8888

Output shows everything loaded:
✓ Dependencies installed for external packages

🔗 Loading external providers from: ./external-examples/providers
↳ Registered external provider: ext-anthropic
↳ Registered external provider: ext-gemini
↳ Registered external provider: ext-openai
✓ External providers loaded

🔌 Loading external plugins from: ./external-examples/plugins
✓ External plugins loaded

⚙️ Loading external middlewares from: ./external-examples/middlewares
↳ Registering middleware: webhookSignatureVerifier
[WebhookVerifier] Registering route: POST /webhooks/verify
✓ External middlewares loaded

🚀 Your AI Gateway is running at: http://localhost:8888

Test commands (after starting the server):

Test ext-openai provider

curl -X POST http://localhost:8888/v1/chat/completions
-H "x-portkey-provider: ext-openai"
-H "Authorization: Bearer $OPENAI_API_KEY"
-H "Content-Type: application/json"
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hello"}]}'

Test ext-anthropic provider

curl -X POST http://localhost:8888/v1/chat/completions
-H "x-portkey-provider: ext-anthropic"
-H "x-api-key: $ANTHROPIC_API_KEY"
-H "Content-Type: application/json"
-d '{"model":"claude-3-haiku-20240307","messages":[{"role":"user","content":"Say hello"}]}'

Test ext-gemini provider

curl -X POST http://localhost:8888/v1/chat/completions
-H "x-portkey-provider: ext-gemini"
-H "Authorization: Bearer $GOOGLE_API_KEY"
-H "Content-Type: application/json"
-d '{"model":"gemini-1.5-flash","messages":[{"role":"user","content":"Say hello"}]}'

Test webhook verifier middleware

curl -X POST http://localhost:8888/webhooks/verify
-H "Content-Type: application/json"
-H "X-Signature: test-sig"
-d '{"event":"test"}'

sandys · 2025-12-03T17:33:58Z

hi @roh26it could you trigger a re-review please ? im not able to do it.

Copilot

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-08T10:14:34Z

package.json

    "avsc": "^5.7.7",
-    "ioredis": "^5.8.0",
    "hono": "^4.9.7",
+    "ioredis": "^5.8.0",


[nitpick] The reordering of 'hono' and 'ioredis' dependencies appears to be unintentional formatting that doesn't add value to this PR. Consider keeping dependencies in their original order to minimize diff noise.

Suggested change

"hono": "^4.9.7",

"ioredis": "^5.8.0",

"ioredis": "^5.8.0",

"hono": "^4.9.7",

Copilot · 2025-12-08T10:14:35Z

src/handlers/handlerUtils.ts

+    parsedConfigJson.defaultInputGuardrails = defaultsConfig.input_guardrails;
+    parsedConfigJson.defaultOutputGuardrails = defaultsConfig.output_guardrails;


The property names have been changed from snake_case to camelCase (default_input_guardrails → defaultInputGuardrails), but the source properties remain in snake_case (defaultsConfig.input_guardrails). This inconsistency suggests either the source properties should also be camelCase, or these target properties should remain snake_case for consistency.

Copilot · 2025-12-08T10:14:35Z

src/utils/externalDependencyInstaller.ts

+  // Remove 'v' prefix and operators if present
+  gatewayVersion = gatewayVersion.replace(/^[v~^>=]+/, '');


The regex /^[v~^>=]+/ removes all leading version operators from gatewayVersion, but it should only remove the 'v' prefix. Operators like ^, ~, >= are valid in gatewayVersion and removing them will break version comparison. Change to /^v/ to only remove the 'v' prefix.

Suggested change

// Remove 'v' prefix and operators if present

gatewayVersion = gatewayVersion.replace(/^[v~^>=]+/, '');

// Remove 'v' prefix if present

gatewayVersion = gatewayVersion.replace(/^v/, '');

Copilot · 2025-12-08T10:14:35Z

external-examples/test-webhook-verifier.sh

+  echo -e "${YELLOW}Test: $test_name${NC}"
+
+  # Build curl arguments array to avoid eval
+  local curl_args=('-s' '-w' $'\n%{http_code}' '-X' "$method")


Using $'\\n%{http_code}' with ANSI-C quoting in an array assignment may not work correctly in all shells. Consider using a regular string with explicit newline: \"\\n%{http_code}\"

Suggested change

local curl_args=('-s' '-w' $'\n%{http_code}' '-X' "$method")

local curl_args=('-s' '-w' "\n%{http_code}" '-X' "$method")

narengogi · 2025-12-08T11:18:22Z

I quite like this approach too @roh26it and @sandys
supporting external plugins this way is much needed now that portkey plugins are used extensively
but I see some comments in your PR saying this is a stripped version for proof of concept etc, but the PR is quite big, would be nice to push the changes out incrementally, let me check the PR and if we can strip it down

narengogi · 2025-12-08T11:39:33Z

external-examples/providers/ext-anthropic/index.js

+// Export the provider config
+export default {
+  api,
+  chatComplete,


I see you are trying to use porktey as an npm package without having to spin up the gateway, this is nice to have but this approach to export each provider separately as a package kinda defeats the purpose of having a unified signature, people should just import one chatComplete function and the final provider should be determined by passing either provider in initialization or any of the other ways that are supported

I'd also add that if the focus of this PR is to allow people to install external plugins, these changes to export provider transforms is an off track, would recommend having one set of feature changes in the PR to keep it clean and make it easier for reviewers to understand what is happening

narengogi · 2025-12-08T12:56:36Z

src/loaders/providerLoader.ts

@@ -0,0 +1,132 @@
+import fs from 'fs';


fs wouldnt work on workers

sandys · 2025-12-08T14:32:09Z

I quite like this approach too @roh26it and @sandys supporting external plugins this way is much needed now that portkey plugins are used extensively but I see some comments in your PR saying this is a stripped version for proof of concept etc, but the PR is quite big, would be nice to push the changes out incrementally, let me check the PR and if we can strip it down

hi @narengogi
thanks for replying.

so two things:

the PR actually has an external-plugins example directory which is really the copied default plugins. just put here to demonstrate it works. so that is a big blob of code im just pulling in for testing. the actual code changes are much lesser.
the PR does middleware, plugins and providers. i think they were very essential cos my biggest struggle was to get some new providers that you dont support out-of-the-box. the PR is common because the code changes account for all 3 situations - i didnt want to provide a minimal pr for only plugins and then have to change that code again when i bring in providers.

the providers are a package because they need their own dependencies. this was actually the hardest thing to do :)
this keeps portkey itself UNTOUCHED. any and all dependencies come from the individual providers themselves.

as server maintainers, i didnt want you to deal with infinite bugs of "i need mysql package too. please add that to portkey"

sandys added 4 commits November 26, 2025 15:28

external plugins and middlewares supported

772b2a7

updated README

4afd1cd

new architecture for middleware and plugins

6a14aa7

uses hono standard patterns to add/update routes via middleware/plugins

254432d

sandys and others added 2 commits November 27, 2025 01:45

Merge branch 'main' into main

dd37957

dont reformat hooks array if it is already formatted

5438760

roh26it requested review from Copilot and narengogi November 29, 2025 01:26

Copilot AI reviewed Nov 29, 2025

View reviewed changes

sandys added 3 commits November 29, 2025 16:00

fixed issues

ccd0f8e

bug fixes and support for external providers as well

0c2b7c4

sandys added 3 commits December 3, 2025 00:50

add debug hooks

1a0c35a

skip-if-installed support for bun

a940d55

bun support fixed

f39b345

fixed issues in external middleware

fd46fa0

narengogi requested a review from Copilot December 8, 2025 10:13

Copilot AI reviewed Dec 8, 2025

View reviewed changes

narengogi reviewed Dec 8, 2025

View reviewed changes

narengogi mentioned this pull request Dec 24, 2025

[Feature] Enable importing this package's contents #738

Open

Merge branch 'main' into main

6e3f2e8

		parsedConfigJson.defaultInputGuardrails = defaultsConfig.input_guardrails;
		parsedConfigJson.defaultOutputGuardrails = defaultsConfig.output_guardrails;

		// Remove 'v' prefix and operators if present
		gatewayVersion = gatewayVersion.replace(/^[v~^>=]+/, '');

	local curl_args=('-s' '-w' $'\n%{http_code}' '-X' "$method")
	local curl_args=('-s' '-w' "\n%{http_code}" '-X' "$method")

External plugins and middleware support #1455

Are you sure you want to change the base?

External plugins and middleware support #1455

Uh oh!

Conversation

sandys commented Nov 26, 2025

Uh oh!

sandys commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Benefits

Why It's Good

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 29, 2025

Choose a reason for hiding this comment

Uh oh!

roh26it commented Nov 29, 2025

Uh oh!

sandys commented Dec 2, 2025

Test ext-openai provider

Test ext-anthropic provider

Test ext-gemini provider

Test webhook verifier middleware

Uh oh!

sandys commented Dec 3, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

narengogi commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

narengogi Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

narengogi Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

narengogi Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

sandys commented Dec 8, 2025

sandys commented Nov 26, 2025 •

edited

Loading

narengogi commented Dec 8, 2025 •

edited

Loading