schemads

Caution

This repository is experimental and in progress. Do not use this module.

Note: as of the next release, response-level caching is on by default when you call NewSchemaDatasource. See the Caching section for the default TTLs, scope behaviour, and how to disable or tune.

A wrapper for Grafana data source plugins that adds schema discovery via CallResource. Plugins implement handler interfaces (SchemaHandler, TablesHandler, ColumnsHandler, etc.) and wire them into NewSchemaDatasource; consumers use the Client to fetch table, column, table parameter, and value metadata.

Overview

• Resource base path: abstractionSchema (BaseResourcePath), with sub-paths for each endpoint (e.g. abstractionSchema/tables).
• Handler configured: the wrapper calls the relevant handler and returns the response as JSON.
• Handler not configured: returns 501 Not Implemented with an endpoint-specific sentinel error.
• Other paths: forwarded to the wrapped CallResourceHandler, or 404 if none is set.

Plugin integration

Handler interfaces

Each endpoint has its own handler interface. Implement the ones your plugin supports:

type SchemaHandler interface {
    Schema(ctx context.Context, req *schemas.SchemaRequest) (*schemas.SchemaResponse, error)
}

type TablesHandler interface {
    Tables(ctx context.Context, req *schemas.TablesRequest) (*schemas.TablesResponse, error)
}

type ColumnsHandler interface {
    Columns(ctx context.Context, req *schemas.ColumnsRequest) (*schemas.ColumnsResponse, error)
}

type TableParameterValuesHandler interface {
    TableParameterValues(ctx context.Context, req *schemas.TableParameterValuesRequest) (*schemas.TableParametersValuesResponse, error)
}

type ColumnValuesHandler interface {
    ColumnValues(ctx context.Context, req *schemas.ColumnValuesRequest) (*schemas.ColumnValuesResponse, error)
}

Wiring into the plugin

NewSchemaDatasource returns a *SchemaDatasource that implements backend.CallResourceHandler. Pass nil for any handler your plugin does not support (those endpoints will return 501):

func NewInstance(ctx context.Context, settings backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
    ds := &MyDatasource{}

    schemaDs := schemas.NewSchemaDatasource(
        schemaHandler,
        tablesHandler,
        columnsHandler,
        tableParameterValuesHandler,
        columnValuesHandler
        nil,  // next CallResourceHandler (or an existing handler)
    )
    return &MyInstance{SchemaDatasource: schemaDs}, nil
}

Adding schema support to an existing data source

If your plugin already implements backend.CallResourceHandler (e.g. for health checks, autocomplete, or custom endpoints), pass it as the last argument. Schema requests are intercepted; everything else is forwarded to your existing handler unchanged:

type MyDatasource struct {
    schemas.SchemaDatasource
}

func NewInstance(ctx context.Context, settings backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
    ds := &MyDatasource{}

    schemaDs := schemas.NewSchemaDatasource(
        ds,   // SchemaHandler
        ds,   // TablesHandler
        ds,   // ColumnsHandler
        ds,   // TableParameterValuesHandler
        ds,   // ColumnValuesHandler
        backend.CallResourceHandlerFunc(ds.handleCustomResource),
    )
    ds.SchemaDatasource = *schemaDs

    return ds, nil
}

func (ds *MyDatasource) handleCustomResource(ctx context.Context, req *backend.CallResourceRequest, sender backend.CallResourceResponseSender) error {
    switch req.Path {
    case "custom-endpoint":
        return sender.Send(&backend.CallResourceResponse{
            Status: 200,
            Body:   []byte(`{"ok": true}`),
        })
    default:
        return sender.Send(&backend.CallResourceResponse{
            Status: 404,
            Body:   []byte("not found"),
        })
    }
}

With this setup:

• POST /abstractionSchema/tables (and other schema sub-paths) are handled by the schema handlers.
• All other paths (e.g. /custom-endpoint) are forwarded to handleCustomResource.

To opt out of schema support entirely, pass nil for all handlers — schema requests return 501 and everything else is forwarded:

schemaDs := schemas.NewSchemaDatasource(nil, nil, nil, nil, nil, existingHandler)

Client usage

Create a Client with NewClient, then call the method for each endpoint. Both httpClient and baseURL are required. Headers on each request type are forwarded as HTTP headers.

client, err := schemas.NewClient(httpClient, "https://host/api/ds/uid/resources/abstractionSchema")

Fetching tables:

resp, err := client.FetchTables(ctx, &schemas.TablesRequest{})
// resp.Tables    – []string
// resp.TableParameters – map[string][]TableParameter

Fetching columns for specific tables:

resp, err := client.FetchColumns(ctx, &schemas.ColumnsRequest{
    Tables: []string{"issues"},
})
// resp.Columns – map[string][]Column

Fetching columns scoped by table parameters:

resp, err := client.FetchColumns(ctx, &schemas.ColumnsRequest{
    Tables:          []string{"issues"},
    TableParameters: map[string]string{"organization": "grafana", "repository": "grafana"},
})

Fetching table parameter values:

resp, err := client.FetchTableParameterValues(ctx, &schemas.TableParameterValuesRequest{
    Table:            "issues",
    TableParameter:   "repository",
    DependencyValues: map[string]string{"organization": "grafana"},
})
// resp.TableParameterValues – map[string][]string

Fetching column values:

resp, err := client.FetchColumnValues(ctx, &schemas.ColumnValuesRequest{
    Table:   "issues",
    Columns: []string{"status"},
})
// resp.ColumnValues – map[string][]string

Fetching the full schema:

resp, err := client.FetchSchema(ctx, &schemas.SchemaRequest{})
// resp.FullSchema – *Schema

Table parameters

Table parameters model hierarchical scoping parameters that consumers resolve before querying a table (e.g. a GitHub data source requires selecting an organization and then a repository before querying issues).

Definition

Each TableParameter has:

Field	Type	Description
Name	string	Unique name within the parent table.
DependsOn	[]string	Sibling table parameters whose values must be selected first.
Root	bool	Entry point with no dependencies. At least one is required.
Required	bool	Must be resolved before the table can be queried.

Validation

ValidateSchema is called automatically for fullSchema responses and enforces:

1. Unique names – no two table parameters in a table share a name.
2. Valid references – every DependsOn entry names a sibling table parameter.
3. Acyclic graph – the dependency graph has no cycles.
4. Root rules – at least one table parameter is Root and root table parameters have no dependencies.
5. Required chain – a required table parameter may only depend on other required table parameters.
6. TableParameterValues integrity – Schema.TableParameterValues keys reference existing tables and root table parameters only. Non-root table parameter values depend on ancestor selections and must be fetched incrementally via tableParameterValues with DependencyValues.

Call ValidateSchema directly to validate schemas you construct manually.

Table references (tables)

Consumers that need to embed a parameterised table in a single human-writable string (for example, an internal query language that says FROM <ref>) can use the tables subpackage.

A reference is an undelimited identifier with a non-empty table name:

<table>(<param1>=<value1>,<param2>=<value2>,...)

Examples:

events                                   // no parameters
events(env=prod)                         // one parameter
events(env=prod,service=tempo)           // multiple parameters
tags(name=Promo \(2024\))                // value with escaped reserved chars

The format is purely syntactic — tables does not produce or consume SQL. It exists so that a reference can be parsed unambiguously back into (table, map[paramName]paramValue) even when parameter values are user-supplied free text.

The encoded form has no outer delimiters. If the surrounding system wraps references in delimiters of its own (for example, backticks in a query language), callers must add them on the way out and strip them before calling Parse. See Embedding in a wider grammar below for the standard backtick recipe.

Reserved characters. Inside a table name, parameter key, or parameter value the five characters (, ), ,, =, and \ are reserved and must be backslash-escaped (\(, \,, \\, etc.). Any other backslash sequence is a parse error. Backticks are not reserved and are passed through verbatim.

Whitespace. Outer whitespace (any Unicode whitespace before or after the entire reference) is trimmed by Parse. Inside the reference, decoding tolerates optional whitespace around (, ), =, and ,, so events(env=prod, service=tempo) and events(env=prod,service=tempo) parse identically. Whitespace inside a value is preserved verbatim, but leading and trailing ASCII whitespace in a value is treated as separator padding and is not preserved across round-trips.

Empty values and empty parameter lists. events(env=) decodes to {"env": ""} (an empty string). A key absent from the parameter list is unset, which is distinct from an empty string. An empty parameter list is normalised: both events and events() decode to a TableRef with TableParams == nil, and a TableRef with no parameters always renders as just the table name (no trailing parens).

API.

import "github.com/grafana/schemads/tables"

ref, err := tables.Parse("events(env=prod,service=tempo)")
// ref.Table       -> "events"
// ref.TableParams -> map[string]string{"env": "prod", "service": "tempo"}

s := tables.TableRef{
    Table:       "events",
    TableParams: map[string]string{"env": "prod", "service": "tempo"},
}.String()
// s -> "events(env=prod,service=tempo)"   (params sorted, escaped)

if err := tables.Validate(ref, schema); err != nil {
    // joined error: ErrUnknownTable, ErrUnknownParameter,
    // ErrMissingRequired, and/or ErrMissingDependency.
}

Parse performs only syntactic validation. It requires a non-empty table name, but does not check whether that table exists. Use Validate to check a decoded reference against a Schema: that the table exists, every key is a declared parameter, every required parameter is present, and every present parameter's declared dependencies are also present. Validate aggregates all issues into a single errors.Join error so callers see every problem in one pass. See the package documentation for the full grammar.

Canonical form

The decoder is intentionally lenient (whitespace around separators, events() accepted for a no-parameter reference, etc.) while the encoder is strict (parameters sorted, no whitespace, empty parameter lists collapsed). Two raw inputs that mean the same thing are not necessarily byte-equal. Use tables.Canonicalize to normalise a reference before using it as a map key, cache key, or identity comparison value:

canon, err := tables.Canonicalize(rawInput) // == Parse(rawInput).String()

Embedding in a wider grammar

References produced by this package have no outer delimiters. When a consumer wraps a reference in its own delimiters — for example a query language that uses backticks — that consumer is responsible for escaping its delimiter inside the wrapped content, because the inner format does not reserve any characters for that purpose. In particular, backticks may appear unescaped inside a parameter value.

tables.WrapInBackticks and tables.UnwrapFromBackticks implement the standard double-the-delimiter recipe used by SQL identifier quoting (` → ``):

stored := tables.WrapInBackticks(ref.String())     // "`events(env=prod)`"
inner, err := tables.UnwrapFromBackticks(stored)   // "events(env=prod)"
ref, err   := tables.Parse(inner)

Any backtick inside ref.String() is doubled on the way out and un-doubled on the way in, so the round-trip preserves arbitrary inner content (including parameter values that legitimately contain backticks).

Endpoints

Path	Constant	Request type	Response type
fullSchema	RequestTypeFullSchema	SchemaRequest	SchemaResponse
tables	RequestTypeTables	TablesRequest	TablesResponse
columns	RequestTypeColumns	ColumnsRequest	ColumnsResponse
columnValues	RequestTypeColumnValues	ColumnValuesRequest	ColumnValuesResponse
tableParameterValues	RequestTypeTableParameterValues	TableParameterValuesRequest	TableParametersValuesResponse

Caching

schemads includes a tenant-safe in-memory response cache that is on by default the moment a plugin calls NewSchemaDatasource. It is backed by patrickmn/go-cache with TTL cleanup and a bounded response value size. Plugins can also reuse the same cache instance for in-handler sub-fetches (per-subscription workspaces, per-index field caps, etc.) via cache.Typed.

Default behaviour

Calling the standard constructor enables caching with sensible per-endpoint defaults:

Endpoint	Default TTL	Default scope
fullSchema	5 min	cache.ScopeUser
tables	5 min	cache.ScopeUser
columns	2 min	cache.ScopeUser
tableParameterValues	1 min	cache.ScopeUser
columnValues	disabled (TTL = 0)	cache.ScopeUser

columnValues is intentionally excluded from the default cache because:

1. Time-range dependent. ColumnValuesRequest carries a TimeRange, and values legitimately change with it (e.g. distinct status codes seen in the last 5m vs the last hour).
2. Freshness-sensitive. Column values back autocomplete dropdowns; stale values produce user-visible bugs.
3. Higher PII risk. Values often contain user-identifying data; shorter exposure is the conservative default.

Cache keys mirror the SDK instancemgmt convention so reconfiguration auto-invalidates entries:

namespace#dsUID#updated#proxyHash[#userHash]#sha256(endpoint|minimalTypedFields)

The user component is hashed before it reaches the key string, so keys are safe to log. This means rotating credentials, swapping the PDC tunnel, or editing datasource settings all invalidate cached entries automatically — there is no need to flush manually.

Tuning via Options

Use NewSchemaDatasourceWithOptions to override defaults:

import (
    schemas "github.com/grafana/schemads"
    "github.com/grafana/schemads/cache"
)

opts := schemas.DefaultOptions
// Re-enable ColumnValues with a short TTL — autocomplete results need to feel fresh.
opts.TTL.ColumnValues = 30 * time.Second
// Tune the shared in-memory cache.
opts.Cache.MaxValueBytes = 512 * 1024

ds := schemas.NewSchemaDatasourceWithOptions(
    schemaHandler, tablesHandler, columnsHandler,
    tableParameterValuesHandler, columnValuesHandler,
    next,
    opts,
)

Note: EndpointTTLs is a flat struct (not an overlay map). When you set any TTL, copy DefaultOptions first so you don't accidentally zero out the others.

Disabling caching entirely

ds := schemas.NewSchemaDatasourceWithOptions(..., schemas.Options{
    DisableCache: true,
})

Manually invalidating an entry

Send the request again with the refresh header configured in RefreshPolicy.Header (default X-Schemads-Refresh). The wrapper deletes the cached entry, runs the handler, and re-caches the result. Bypass is rate-limited per (tenant, endpoint) (default 5s) to prevent CPU DoS via constant invalidation.

Choosing a scope

The default ScopeUser is the safe choice. You can relax to ScopeDatasource per endpoint via Options.PerEndpointScope, only after auditing that the endpoint's results do not depend on the calling user's permissions.

Use this checklist:

Question	If yes...
Does the upstream filter results by the calling user's identity, role, or group membership?	Keep ScopeUser (default)
Does the upstream apply RBAC or row-level security tied to the impersonated user?	Keep ScopeUser (default)
Does the response contain PII that should not be visible to all users with read access to the datasource?	Keep ScopeUser (default)
Is the response a function purely of (datasource, request body) — same input, same output, regardless of caller?	Relax to ScopeDatasource

Example — Tables/Columns shared across users on a cluster that does not enforce per-user index visibility:

opts := schemas.DefaultOptions
opts.PerEndpointScope = map[string]cache.Scope{
    schemas.RequestTypeTables:  cache.ScopeDatasource,
    schemas.RequestTypeColumns: cache.ScopeDatasource,
}

When ScopeUser is configured but the request has no user (alerting, public dashboards, recorded queries), the wrapper bypasses the cache and logs a warning rather than serving anonymous results to authenticated users.

In-handler sub-fetch caching

Plugins frequently call upstream APIs from inside their handlers (Azure's per-subscription workspaces, Elasticsearch's per-index _field_caps). Use cache.Typed[T] to memoize these without paying JSON marshal cost:

type Plugin struct {
    schemaDS    *schemas.SchemaDatasource
    workspaces  *cache.Typed[[]Workspace]
}

func NewPlugin(...) *Plugin {
    schemaDS := schemas.NewSchemaDatasource(...)
    return &Plugin{
        schemaDS:   schemaDS,
        // Tag with a label for Prometheus metrics.
        workspaces: cache.NewTyped[[]Workspace](schemaDS.Cache(), "subfetch:workspaces"),
    }
}

func (p *Plugin) listWorkspaces(ctx context.Context, pc backend.PluginContext, sub string) ([]Workspace, error) {
    // Build a tenant-safe key. ScopeUser is recommended for any subscription-
    // scoped lookup since the user's role determines visible workspaces.
    key, err := cache.KeyFromPluginContext(pc, cache.ScopeUser, "workspaces", sub)
    if err != nil {
        // Fall back to direct fetch (e.g. nil User in alerting context).
        return fetchWorkspaces(ctx, sub)
    }
    return p.workspaces.GetOrFetch(ctx, key, 5*time.Minute, func(ctx context.Context) ([]Workspace, error) {
        return fetchWorkspaces(ctx, sub)
    })
}

Typed[T] deduplicates concurrent misses for the same key with singleflight, never caches errors, and increments schemads_cache_*_total{endpoint="subfetch:workspaces"} Prometheus counters. Scope is controlled by the cache.Key you pass in, so the same typed cache can safely hold ScopeUser and ScopeDatasource entries when keys are built correctly.

For small JSON-serializable values, plugins can also use the byte-oriented cache.GetOrFetch helper against ds.Cache():

v, err := cache.GetOrFetch(ctx, ds.Cache(), key, "subfetch:workspaces", 5*time.Minute, fetchFn)

This is appropriate when the cached value is small/serializable and JSON marshal/unmarshal cost is negligible; Typed[T] is preferred for hot paths where marshal/unmarshal would dominate CPU.

Metrics

The cache exports the following Prometheus counters:

Metric	Labels	Meaning
schemads_cache_hits_total	endpoint	Cache hits per endpoint (or sub-fetch label)
schemads_cache_misses_total	endpoint	Cache misses
schemads_cache_evictions_total	endpoint	TTL evictions

Register them with your Prometheus registry once at startup:

cache.MustRegisterMetrics(prometheus.DefaultRegisterer)

If never registered, metrics still record but are not exposed (matches other SDK packages).

Eviction model

• TTL. Entries expire after their per-endpoint TTL. The in-memory cache (cache.MemoryCache) is built on patrickmn/go-cache and runs a background sweep every CleanupInterval (default 5 minutes) so unaccessed entries do not linger.
• Manual. The refresh header (default X-Schemads-Refresh) deletes the specific key for the requesting tenant only — never the whole cache.
• Response size. Byte-oriented response entries larger than MaxValueBytes (default 5 MiB) are not cached.

Column types

Types are aligned with go-mysql-server's type system:

Category	Constants
Boolean	boolean
Integers	int8, int16, int32, int64, uint8, uint16, uint32, uint64
Float	float32, float64
Decimal	decimal (use Column.Precision / Column.Scale)
String	string
Date/Time	date, datetime, timestamp, time, year
Structured	json, enum (use Column.Values), set (use Column.Values)
Binary	blob, bit (use Column.Size, 1-64)

Operators

Columns declare which filter operations they support via Column.Operators:

Constant	Value
OperatorEquals	=
OperatorNotEquals	!=
OperatorGreaterThan	>
OperatorGreaterThanOrEqual	>=
OperatorLessThan	<
OperatorLessThanOrEqual	<=
OperatorLike	like
OperatorIn	in

Error handling

Errors are returned as JSON {"error": "..."} with an HTTP status:

Status	Meaning
500	Handler error or schema validation failure
501	Handler not configured for the requested endpoint

Each endpoint has its own sentinel error for 501 responses:

Sentinel	Endpoint
ErrSchemaNotImplemented	fullSchema
ErrTablesNotImplemented	tables
ErrColumnsNotImplemented	columns
ErrTableParameterValuesNotImplemented	tableParameterValues
ErrColumnValuesNotImplemented	columnValues

The Client returns these sentinels on 501 so callers can use errors.Is:

resp, err := client.FetchColumns(ctx, &schemas.ColumnsRequest{Tables: []string{"t"}})
if errors.Is(err, schemas.ErrColumnsNotImplemented) {
    // plugin does not support this endpoint
}

Partial failures can be reported via the Errors field on each response type (keyed by table or column name).