Add docs for component metrics & new mysql connection parameters

website/docs/components/data-connectors/mysql.md

@@ -6,6 +6,7 @@ tags:
   - data-connectors
   - mysql
   - relational
+  - component-metrics
 ---
 
 MySQL is an open-source relational database management system that uses structured query language (SQL) for managing and manipulating databases.
@@ -22,6 +23,8 @@ datasets:
       mysql_db: my_database
       mysql_user: my_user
       mysql_pass: ${secrets:mysql_pass}
+      mysql_pool_min: 10
+      mysql_pool_max: 100
 ```
 
 ## Configuration
@@ -89,6 +92,51 @@ The MySQL data connector can be configured by providing the following `params`.
 | `mysql_pass`              | The password to connect with.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | `mysql_sslmode`           | Optional. Specifies the SSL/TLS behavior for the connection, supported values:<br /> <ul><li>`required`: (default) This mode requires an SSL connection. If a secure connection cannot be established, server will not connect.</li><li>`preferred`: This mode will try to establish a secure SSL connection if possible, but will connect insecurely if the server does not support SSL.</li><li>`disabled`: This mode will not attempt to use an SSL connection, even if the server supports it.</li></ul> |
 | `mysql_sslrootcert`       | Optional parameter specifying the path to a custom PEM certificate that the connector will trust.                                                                                                                                                                                                                                                                                                                                                                                                            |
+| `mysql_pool_min`          | The minimum number of connections to keep open in the pool, lazily created when requested.  Default: `10`                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `mysql_pool_max`          | The maximum number of connections to allow in the pool. Default: `100`                                                                                                                                                                                                                                                                                                                                                                                                       |
+
+### `metrics`
+
+The MySQL data connector supports the following optional [component metrics](/docs/features/observability/component_metrics):
+
+| Metric Name | Type | Description |
+| ----------- | ---- | ----------- |
+| `connection_count` | Gauge | Gauge of active connections to the database server |
+| `connections_in_pool` | Gauge | Gauge of active connections that are idling in the pool |
+| `active_wait_requests` | Gauge | Gauge of requests that are waiting for a connection to be returned to the pool |
+| `create_failed` | Counter | Counter of connections that failed to be created |
+| `discarded_superfluous_connection` | Counter | Counter of connections that were closed because there were already enough idle connections in the pool |
+| `discarded_unestablished_connection` | Counter | Counter of connections that were closed because they could not be established |
+| `dirty_connection_return` | Counter | Counter of connections that were returned to the pool but were dirty (ie. open transactions, pending queries, etc) |
+| `discarded_expired_connection` | Counter | Counter of connections that were discarded because they were expired by the pool constraints (i.e. TTL expired) |
+| `resetting_connection` | Counter | Counter of connections that were reset |
+| `discarded_error_during_cleanup` | Counter | Counter of connections that were discarded because they returned an error during cleanup |
+| `connection_returned_to_pool` | Counter | Counter of connections that were returned to the pool |
+
+These metrics are not enabled by default, enable them by setting the `metrics` parameter:
+
+```yaml
+datasets:
+  - from: mysql:mytable
+    name: my_dataset
+    metrics:
+      - name: connection_count
+      - name: connections_in_pool
+      - name: active_wait_requests
+      - name: create_failed
+      - name: discarded_superfluous_connection
+      - name: discarded_unestablished_connection
+      - name: dirty_connection_return
+      - name: discarded_expired_connection
+      - name: resetting_connection
+      - name: discarded_error_during_cleanup
+      - name: connection_returned_to_pool
+    params: &params
+      mysql_host: localhost
+      mysql_tcp_port: 3306
+      mysql_user: my_user
+      mysql_pass: ${secrets:mysql_pass}
+```
 
 ## Types
 
@@ -186,6 +234,22 @@ datasets:
       mysql_pass: ${secrets:mysql_pass}
 ```
 
+### With custom connection pool settings
+
+```yaml
+datasets:
+  - from: mysql:path.to.my_dataset
+    name: my_dataset
+    params:
+      mysql_host: localhost
+      mysql_tcp_port: 3306
+      mysql_db: my_database
+      mysql_user: my_user
+      mysql_pass: ${secrets:mysql_pass}
+      mysql_pool_min: 5
+      mysql_pool_max: 10
+```
+
 ## Secrets
 
 Spice integrates with multiple secret stores to help manage sensitive data securely. For detailed information on supported secret stores, refer to the [secret stores documentation](/docs/components/secret-stores). Additionally, learn how to use referenced secrets in component parameters by visiting the [using referenced secrets guide](/docs/components/secret-stores#using-secrets).

website/docs/features/observability/component_metrics.md

@@ -0,0 +1,60 @@
+---
+title: 'Component Metrics'
+sidebar_label: 'Component Metrics'
+description: 'Learn how to enable optional component metrics.'
+sidebar_position: 1
+pagination_prev: null
+pagination_next: null
+tags:
+  - component-metrics
+---
+
+Component metrics provide detailed insights into the internal state and performance of individual components in Spice. Each component can expose its own set of metrics that can be enabled selectively to monitor specific aspects of its operation.
+
+## Enabling Component Metrics
+
+Component metrics are disabled by default and can be enabled by adding a `metrics` section to the component configuration. Each metric can be enabled individually by specifying its name in the metrics list.
+
+### Example Configuration
+
+```yaml
+datasets:
+  - from: some_component:my_resource
+    name: my_resource
+    metrics:
+      - name: metric_one
+        enabled: true
+      - name: metric_two
+        enabled: true
+      - name: metric_three
+        enabled: false
+    params:
+      param_one: value_one
+      param_two: value_two
+```
+
+## Available Metrics
+
+Each component defines its own set of available metrics. These metrics are exposed in the Prometheus format with the following naming convention:
+
+```text
+{component_type}_{component_name}_{metric_name}
+```
+
+For example, a MySQL dataset component's metrics would be prefixed with `dataset_mysql_`.
+
+## Monitoring Component Metrics
+
+Component metrics are exposed through the same Prometheus-compatible metrics endpoint as other Spice metrics. These metrics can be accessed using standard Prometheus tools or any monitoring system that supports Prometheus metrics.
+
+To view the metrics, make a GET request to the metrics endpoint:
+
+```bash
+curl http://localhost:9090/metrics
+```
+
+The response will include all enabled component metrics in Prometheus format, with proper HELP and TYPE annotations.
+
+## Component-Specific Metrics
+
+For detailed information about metrics available for specific components, view all [components that expose metrics](/docs/tags/component-metrics).

website/docs/features/observability/index.md

@@ -110,3 +110,9 @@ dataset_active_count{engine="duckdb"} 1
 | `tool_load_state`<br/>*(gauge)*                    | Status of the LLM tools. 1=Initializing, 2=Ready, 3=Disabled, 4=Error, 5=Refreshing.             |
 | `view_load_errors`<br/>*(count)*                   | Number of errors loading the view.                                                              |
 | `view_load_state`<br/>*(gauge)*                    | Status of the views. 1=Initializing, 2=Ready, 3=Disabled, 4=Error, 5=Refreshing.                 |
+
+:::note Component Metrics
+
+In addition to these core metrics, individual components can expose their own metrics. For example, the MySQL data connector exposes [connection pool metrics](/docs/components/data-connectors/mysql/#metrics). See [Component Metrics](/docs/features/observability/component_metrics) for more information.
+
+:::

website/docs/reference/spicepod/datasets.md

@@ -334,6 +334,34 @@ Optional. Enable or disable refresh jitter, defaults to `false`. The refresh jit
 
 Optional. The maximum amount of jitter to add to the refresh interval. The jitter is a random value between 0 and `refresh_jitter_max`. Defaults to 10% of `refresh_check_interval`.
 
+## `metrics`
+
+Optional. Enable component-specific metrics for the dataset. Each component can expose its own set of metrics that can be enabled selectively to monitor specific aspects of its operation.
+
+Component metrics are disabled by default and can be enabled by adding a `metrics` section to the dataset configuration. Each metric can be enabled individually by specifying its name in the metrics list.
+
+### Example Configuration
+
+```yaml
+datasets:
+  - from: mysql:my_table
+    name: my_dataset
+    metrics:
+      - name: connection_count
+        enabled: true
+      - name: connections_in_pool
+        enabled: true
+      - name: active_wait_requests
+        enabled: true
+    params:
+      mysql_host: localhost
+      mysql_tcp_port: 3306
+      mysql_user: root
+      mysql_pass: ${secrets:MYSQL_PASS}
+```
+
+For detailed information about metrics available for specific components, see the [component metrics documentation](/docs/features/observability/component_metrics).
+
 ## `acceleration.indexes`
 
 Optional. Specify which indexes should be applied to the locally accelerated table. Not supported for in-memory Arrow acceleration engine.