Draft
Conversation
Adds a new GenerateTemplate RPC endpoint that serves as the single source of truth for YAML format generation. Replaces frontend imperative YAML builders with a centralized backend handler. - Add EnvVarName field to PropertySpec for custom env var name mapping - Populate EnvVarName values across 14 driver specs (S3, GCS, Azure, ClickHouse, Postgres, BigQuery, Snowflake, Redshift, MotherDuck, Athena, MySQL, Druid, Pinot, Salesforce) - Define GenerateTemplate proto messages with validation annotations - Implement handler: auth check, driver validation, property validation, DuckDB rewrite for object/file stores and sqlite - Add comprehensive YAML renderers for connectors and models with precise formatting via yaml.Node tree construction - Implement env var conflict resolution with _1, _2 suffix appending - Add 54 passing test cases covering all drivers, error scenarios, and edge cases Frontend migration to use this RPC will follow as a separate step. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace client-side YAML builders with backend RPC for single source of truth. - New helper: `generateTemplate()` + `mergeEnvVars()` in generate-template.ts - Updated submission paths: submitAddConnectorForm, submitAddSourceForm, saveConnectorAnyway - Updated preview: computeYamlPreview now calls RPC (async, 150ms debounced) - HTTPS connector path preserved (backend doesn't support headers array yet) - Removed dead code: compileSourceYAML, prepareSourceFormData, maybeRewriteToDuckDb - Backend now handles DuckDB rewrite and env var naming (conflict resolution) - Tests: added mergeEnvVars (9 tests) and normalizeConnectorError (11 tests) - Net: 234 lines removed (~27% reduction in submission/preview code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ector headers
Add HTTPS headers support to the backend GenerateTemplate RPC, enabling removal
of all client-side YAML generation code. The HTTPS connector now uses the same
RPC path as all other connectors.
Backend changes:
- Add `headers` property to HTTPS driver ConfigProperties
- Implement header rendering with sensitive value extraction to .env
- Support Authorization header scheme prefixes (Bearer, Basic, Token, Bot)
- Use `connector.{driver}.{key}` naming convention for header env vars
- Add 4 comprehensive test cases for header handling
Frontend changes:
- Convert key-value UI arrays to maps before RPC submission
- Remove HTTPS-specific YAML preview and submission code
- All connectors now use GenerateTemplate RPC exclusively
- Remove 13 dead functions (compileConnectorYAML, updateDotEnvWithSecrets, etc)
- Consolidate env var naming to backend-only responsibility
Schema cleanup:
- Remove 30 dead `x-env-var-name` annotations (frontend no longer reads them)
- Single source of truth for env var names: backend PropertySpec.EnvVarName
Results:
- -1,317 lines of dead code removed
- +221 lines of backend implementation
- 2,055 frontend tests pass, 62 backend tests pass (4 new HTTPS tests)
The mapping from property Key to env var name doesn't follow a mechanical pattern. Well-known names are shared across drivers (AWS_ACCESS_KEY_ID), some add infixes (AZURE_STORAGE_*), and others diverge entirely (GCS key_id -> GCP_ACCESS_KEY_ID). Each driver must specify explicitly. Also remove personal docs from .gitignore.
- Fix gofmt alignment in connectors.go PropertySpec struct - Remove unused connectorName param from renderDuckDBModelYAML - Use const instead of let for newEnvBlob and connectorYamlBlob
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a GenerateTemplate RPC that moves connector/model YAML generation from the frontend to the backend.
Checklist: