Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 28 additions & 30 deletions crates/bin/ampd/src/main.rs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should update the feature docs for ampd too. To reflect the new env variable names.

Original file line number Diff line number Diff line change
Expand Up @@ -35,24 +35,24 @@ enum Command {
/// If `--config` is not provided, searches for `config.toml` within this directory.
#[arg(long, env = "AMP_DIR")]
amp_dir: Option<PathBuf>,
/// Enable Arrow Flight RPC Server.
#[arg(long, env = "FLIGHT_SERVER")]
flight_server: bool,
/// Enable JSON Lines Server.
#[arg(long, env = "JSONL_SERVER")]
jsonl_server: bool,
/// Enable Admin API Server.
#[arg(long, env = "ADMIN_SERVER")]
admin_server: bool,
/// Enable Arrow Flight RPC Server (env: AMP_FLIGHT_SERVER).
#[arg(long, num_args = 0..=1, default_missing_value = "true")]
flight_server: Option<bool>,
/// Enable JSON Lines Server (env: AMP_JSONL_SERVER).
#[arg(long, num_args = 0..=1, default_missing_value = "true")]
jsonl_server: Option<bool>,
/// Enable Admin API Server (env: AMP_ADMIN_SERVER).
#[arg(long, num_args = 0..=1, default_missing_value = "true")]
admin_server: Option<bool>,
},
/// Run query server (Arrow Flight, JSON Lines)
Server {
/// Enable Arrow Flight RPC Server.
#[arg(long, env = "FLIGHT_SERVER")]
flight_server: bool,
/// Enable JSON Lines Server.
#[arg(long, env = "JSONL_SERVER")]
jsonl_server: bool,
/// Enable Arrow Flight RPC Server (env: AMP_FLIGHT_SERVER).
#[arg(long, num_args = 0..=1, default_missing_value = "true")]
flight_server: Option<bool>,
/// Enable JSON Lines Server (env: AMP_JSONL_SERVER).
#[arg(long, num_args = 0..=1, default_missing_value = "true")]
jsonl_server: Option<bool>,
},
/// Run a distributed worker node
Worker {
Expand Down Expand Up @@ -107,16 +107,10 @@ async fn main_inner() -> Result<(), Error> {
match command {
Command::Solo {
amp_dir,
mut flight_server,
mut jsonl_server,
mut admin_server,
flight_server,
jsonl_server,
admin_server,
} => {
if !flight_server && !jsonl_server && !admin_server {
flight_server = true;
jsonl_server = true;
admin_server = true;
}

// Resolve the amp state directory, creating it if needed.
let amp_dir = match amp_dir {
Some(dir) => dir,
Expand Down Expand Up @@ -150,17 +144,21 @@ async fn main_inner() -> Result<(), Error> {
Ok(())
}
Command::Server {
mut flight_server,
mut jsonl_server,
flight_server: cli_flight_server,
jsonl_server: cli_jsonl_server,
} => {
// If neither of the flags are set, enable both servers
let config_path = config_path.as_ref().ok_or(Error::MissingConfigPath)?;
let config = config::load(config_path).map_err(Error::LoadConfig)?;

// Merge CLI flags with config values (CLI takes precedence when provided)
let mut flight_server = cli_flight_server.unwrap_or(config.flight_server);
let mut jsonl_server = cli_jsonl_server.unwrap_or(config.jsonl_server);

// If neither flag is set (CLI or config), enable both servers
if !flight_server && !jsonl_server {
flight_server = true;
jsonl_server = true;
}

let config_path = config_path.as_ref().ok_or(Error::MissingConfigPath)?;
let config = config::load(config_path).map_err(Error::LoadConfig)?;
let metadata_db_config = amp_config::metadb::load(config_path, None)
.ok_or(Error::MissingMetadataDbConfig)?;
let addrs = config.server_addrs.clone();
Expand Down
18 changes: 15 additions & 3 deletions crates/bin/ampd/src/solo_cmd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ type ServerFuture = Pin<Box<dyn Future<Output = Result<(), server::service::Serv
pub async fn run(
amp_dir: PathBuf,
config_path: Option<PathBuf>,
flight_server: bool,
jsonl_server: bool,
admin_server: bool,
flight_server: Option<bool>,
jsonl_server: Option<bool>,
admin_server: Option<bool>,
) -> Result<(), Error> {
// 1. Check if user provided metadata_db.url before starting managed PostgreSQL
let effective_config_path = config_path
Expand Down Expand Up @@ -81,6 +81,18 @@ pub async fn run(
)
.map_err(Error::ConfigLoad)?;

// Merge CLI flags with config values (CLI takes precedence when provided)
let mut flight_server = flight_server.unwrap_or(config.flight_server);
let mut jsonl_server = jsonl_server.unwrap_or(config.jsonl_server);
let mut admin_server = admin_server.unwrap_or(config.admin_server);

// If no servers are explicitly enabled (CLI or config), enable all
if !flight_server && !jsonl_server && !admin_server {
flight_server = true;
jsonl_server = true;
admin_server = true;
}

// 4. Init monitoring
let (_providers, meter) =
monitoring::init(config.opentelemetry.as_ref()).map_err(Error::MonitoringInit)?;
Expand Down
54 changes: 52 additions & 2 deletions crates/config/src/config_file.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,20 @@
//! | 1 (highest) | `AMP_CONFIG_*` env vars | `merge` — always wins |
//! | 2 | TOML file values | `merge` — base configuration |
//! | 3 | Deprecated flat keys (`metadata_db_url`, `dataset_defs_dir`) | `join` — fallback only |
//! | 4 (lowest) | Caller-provided `ConfigDefaultsOverride`s | `join` — fallback only |
//! | 4 | Server flag env vars (`AMP_FLIGHT_SERVER`, `AMP_JSONL_SERVER`, `AMP_ADMIN_SERVER`) | `join` — fallback only |
//! | 5 | Deprecated env vars (`FLIGHT_SERVER`, `JSONL_SERVER`, `ADMIN_SERVER`) | `join` — fallback only |
//! | 6 (lowest) | Caller-provided `ConfigDefaultsOverride`s | `join` — fallback only |
//!
//! ## Environment variables
//!
//! All env vars are prefixed with `AMP_CONFIG_` and use double underscores to
//! separate nested keys. For example, `AMP_CONFIG_METADATA_DB__URL` maps to
//! `metadata_db.url` in the config file.

use std::path::{Path, PathBuf};
use std::{
env,
path::{Path, PathBuf},
};

use dump::{ConfigDuration, ParquetConfig};
use figment::{
Expand All @@ -34,6 +39,16 @@ use figment::{
};
use monitoring::config::OpenTelemetryConfig;

/// Environment variable migrations for server enable flags.
///
/// Each tuple contains (new_name, deprecated_name, config_key). The new env var names
/// use the `AMP_*` prefix for consistency with other CLI env vars (AMP_CONFIG, AMP_DIR, AMP_NODE_ID).
const SERVER_FLAG_ENV_VARS: &[(&str, &str, &str)] = &[
("AMP_FLIGHT_SERVER", "FLIGHT_SERVER", "flight_server"),
("AMP_JSONL_SERVER", "JSONL_SERVER", "jsonl_server"),
("AMP_ADMIN_SERVER", "ADMIN_SERVER", "admin_server"),
];

/// Default data directory name - stores Parquet files
pub const DEFAULT_DATA_DIRNAME: &str = "data";

Expand Down Expand Up @@ -128,6 +143,25 @@ pub fn load(
figment = figment.join(Serialized::default("manifests_dir", value));
}

// Handle server flag env vars (AMP_FLIGHT_SERVER, etc.) and their deprecated names
// Priority: TOML/AMP_CONFIG_* (already merged above) > AMP_*_SERVER > FLIGHT_SERVER (deprecated)
for (new_env, deprecated_env, config_key) in SERVER_FLAG_ENV_VARS {
// Check new env var first (AMP_FLIGHT_SERVER)
if let Ok(value) = env::var(new_env) {
let bool_value = parse_bool_env(&value);
figment = figment.join(Serialized::default(config_key, bool_value));
}
// Fall back to deprecated env var (FLIGHT_SERVER) with warning
else if let Ok(value) = env::var(deprecated_env) {
tracing::warn!(
"environment variable `{deprecated_env}` is deprecated; \
use `{new_env}` instead"
);
let bool_value = parse_bool_env(&value);
figment = figment.join(Serialized::default(config_key, bool_value));
}
}

// Caller-provided ConfigDefaultsOverride (lowest priority — fills gaps only)
for ConfigDefaultsOverride(provider) in defaults {
figment = figment.join(provider);
Expand Down Expand Up @@ -187,6 +221,17 @@ pub struct ConfigFile {
/// Admin API server address (default: "0.0.0.0:1610")
pub admin_api_addr: Option<String>,

// Service enable flags (for solo/server modes)
/// Enable Arrow Flight RPC server (default: false, env: AMP_FLIGHT_SERVER)
#[serde(default)]
pub flight_server: bool,
/// Enable JSON Lines server (default: false, env: AMP_JSONL_SERVER)
#[serde(default)]
pub jsonl_server: bool,
/// Enable Admin API server (default: false, env: AMP_ADMIN_SERVER)
#[serde(default)]
pub admin_server: bool,

// Observability
pub opentelemetry: Option<OpenTelemetryConfig>,

Expand Down Expand Up @@ -254,3 +299,8 @@ fn default_keep_alive_interval() -> u64 {
#[derive(Debug, thiserror::Error)]
#[error("Failed to load configuration file")]
pub struct LoadConfigFileError(#[source] pub Box<figment::Error>);

/// Parses a string as a boolean. Truthy values: "1", "true", "yes", "on" (case-insensitive).
fn parse_bool_env(value: &str) -> bool {
matches!(value.to_lowercase().as_str(), "1" | "true" | "yes" | "on")
}
9 changes: 9 additions & 0 deletions crates/config/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@ fn resolve_config(
config_path,
poll_interval: config_file.poll_interval_secs.into(),
keep_alive_interval: config_file.keep_alive_interval,
flight_server: config_file.flight_server,
jsonl_server: config_file.jsonl_server,
admin_server: config_file.admin_server,
worker_events,
})
}
Expand Down Expand Up @@ -223,6 +226,12 @@ pub struct Config {
pub poll_interval: Duration,
/// Keep-alive interval for streaming server (in seconds).
pub keep_alive_interval: u64,
/// Enable Arrow Flight RPC server (env: AMP_FLIGHT_SERVER).
pub flight_server: bool,
/// Enable JSON Lines server (env: AMP_JSONL_SERVER).
pub jsonl_server: bool,
/// Enable Admin API server (env: AMP_ADMIN_SERVER).
pub admin_server: bool,
/// Worker event streaming configuration.
pub worker_events: WorkerEventsConfig,
}
Expand Down
28 changes: 28 additions & 0 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,34 @@ The following optional configuration keys allow you to control the hostname and
- `jsonl_addr`: JSON Lines server address (default: `0.0.0.0:1603`)
- `admin_api_addr`: Admin API server address (default: `0.0.0.0:1610`)

# Service enable flags

These boolean flags control which services are started. They can be set via config file, environment variables, or CLI flags. CLI flags take precedence over config/env values.

| Config Key | Environment Variable | CLI Flag | Description |
|------------|---------------------|----------|-------------|
| `flight_server` | `AMP_FLIGHT_SERVER` | `--flight-server [bool]` | Enable Arrow Flight RPC server |
| `jsonl_server` | `AMP_JSONL_SERVER` | `--jsonl-server [bool]` | Enable JSON Lines server |
| `admin_server` | `AMP_ADMIN_SERVER` | `--admin-server [bool]` | Enable Admin API server (solo mode only) |

**Default behavior:** If none of these flags are explicitly set (via any method), all applicable services are enabled by default. Setting any flag explicitly means only the flagged services will start.

**Precedence:** CLI flag > Config file / Environment variable

**CLI flag values:** `--flight-server` alone enables the server. Use `--flight-server false` to explicitly disable a server that is enabled in the config file.

**Environment variable values:** Accepts `1`, `true`, `yes`, `on` (case-insensitive) as truthy values.

### Deprecated environment variables

The following environment variable names are deprecated and will emit warnings. They continue to work for backward compatibility:

| Deprecated | Replacement |
|------------|-------------|
| `FLIGHT_SERVER` | `AMP_FLIGHT_SERVER` |
| `JSONL_SERVER` | `AMP_JSONL_SERVER` |
| `ADMIN_SERVER` | `AMP_ADMIN_SERVER` |

## Logging

Simplified control of the logging verbosity level is offered by the `AMP_LOG` env var. It accepts
Expand Down
10 changes: 10 additions & 0 deletions docs/config.sample.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,16 @@ manifests_dir = "manifests"
# jsonl_addr = "0.0.0.0:1603" # JSON Lines server address (default: "0.0.0.0:1603")
# admin_api_addr = "0.0.0.0:1610" # Admin API server address (default: "0.0.0.0:1610")

# Service enable flags (for solo/server modes)
# These can also be set via environment variables (AMP_FLIGHT_SERVER, etc.) or CLI flags
# (--flight-server, --flight-server false).
# CLI flags take precedence over config file and environment variable values.
# NOTE: When none of these are explicitly set, all applicable services start by default.
# Setting any flag to true means only the explicitly enabled services will start.
# flight_server = true # Enable Arrow Flight server (env: AMP_FLIGHT_SERVER)
# jsonl_server = true # Enable JSON Lines server (env: AMP_JSONL_SERVER)
# admin_server = true # Enable Admin API server (env: AMP_ADMIN_SERVER, solo mode only)

# ================================
# Table sections
# ================================
Expand Down
15 changes: 15 additions & 0 deletions docs/features/app-ampd-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,21 @@ ampd server
| `flight_addr` | `0.0.0.0:1602` | Arrow Flight server binding |
| `jsonl_addr` | `0.0.0.0:1603` | JSON Lines server binding |

### Service Enable Flags

Control which servers are started via CLI flags, config file, or environment variables. CLI flags take precedence.

| Config Key | Environment Variable | CLI Flag | Description |
|------------|---------------------|----------|-------------|
| `flight_server` | `AMP_FLIGHT_SERVER` | `--flight-server [bool]` | Enable Arrow Flight server |
| `jsonl_server` | `AMP_JSONL_SERVER` | `--jsonl-server [bool]` | Enable JSON Lines server |

**Default behavior:** If no flags are explicitly set, both servers are enabled.

**CLI flag values:** `--flight-server` alone enables the server. Use `--flight-server false` to explicitly disable.

**Environment variable values:** Accepts `1`, `true`, `yes`, `on` (case-insensitive).

### Streaming Settings

| Setting | Default | Description |
Expand Down
18 changes: 18 additions & 0 deletions docs/features/app-ampd-solo.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,24 @@ Solo mode runs all three ampd services in a single process: query server (Flight

Solo mode supports an optional `--config` (or `AMP_CONFIG`) file. If no config is provided, it auto-discovers `<amp_dir>/config.toml` and falls back to zero-config defaults. See [app-ampd-solo-ampdir](app-ampd-solo-ampdir.md) for amp dir and config discovery details.

### Service Enable Flags

Control which services are started via CLI flags, config file, or environment variables. CLI flags take precedence.

| Config Key | Environment Variable | CLI Flag | Description |
|------------|---------------------|----------|-------------|
| `flight_server` | `AMP_FLIGHT_SERVER` | `--flight-server [bool]` | Enable Arrow Flight server |
| `jsonl_server` | `AMP_JSONL_SERVER` | `--jsonl-server [bool]` | Enable JSON Lines server |
| `admin_server` | `AMP_ADMIN_SERVER` | `--admin-server [bool]` | Enable Admin API server |

**Default behavior:** If no flags are explicitly set (via any method), all services are enabled.

**CLI flag values:** `--flight-server` alone enables the server. Use `--flight-server false` to explicitly disable.

**Environment variable values:** Accepts `1`, `true`, `yes`, `on` (case-insensitive).

**Deprecated environment variables:** `FLIGHT_SERVER`, `JSONL_SERVER`, `ADMIN_SERVER` still work but emit warnings. Use the `AMP_*` prefixed versions instead.

## Usage

### Default Behavior
Expand Down
Loading
Loading