diff --git a/azure-sql/database-watcher-data.md b/azure-sql/database-watcher-data.md index 90dd9630021..26a0de37f7e 100644 --- a/azure-sql/database-watcher-data.md +++ b/azure-sql/database-watcher-data.md @@ -4,7 +4,7 @@ titleSuffix: Azure SQL Database & SQL Managed Instance description: A detailed description of SQL monitoring data collected by database watcher author: dimitri-furman ms.author: dfurman -ms.date: 10/07/2024 +ms.date: 11/11/2024 ms.service: azure-sql ms.subservice: monitoring ms.topic: conceptual @@ -27,7 +27,7 @@ Database watcher takes advantage of [streaming ingestion](/azure/data-explorer/i ### Interaction between database watcher and application workloads -Enabling database watcher is not likely to have an observable impact on the application workloads. More frequent monitoring queries typically execute in the sub-second range, whereas queries that might require more time, for example to return large datasets, run at infrequent intervals. +Enabling database watcher is not likely to have an observable impact on the application workload performance. More frequent monitoring queries typically execute in the sub-second range, whereas queries that might require more time, for example to return large datasets, run at infrequent intervals. # [SQL database](#tab/sqldb) @@ -43,6 +43,10 @@ If there is resource contention between your application workloads and database The following example configures Resource Governor on a SQL managed instance. It limits CPU consumption by database watcher queries to 30% when there is no CPU contention. When there is CPU contention, this configuration reserves 5% of CPU for the monitoring queries and limits their CPU usage to 10%. It also limits the memory grant size for each monitoring query to 10% of the available memory. +> [!NOTE] +> +> If you make Resource Governor configuration too restrictive, for example by using low `MAX_CPU_PERCENT` or `CAP_CPU_PERCENT` values, database watcher might not be able to collect data reliably or at all because of insufficient compute resources. + ```sql USE master; GO @@ -94,7 +98,9 @@ END CATCH; --- -To avoid concurrency conflicts such as blocking and deadlocks between data collection and database workloads running on your Azure SQL resources, the monitoring queries use short [lock timeouts](/sql/relational-databases/sql-server-transaction-locking-and-row-versioning-guide#customizing-the-lock-time-out) and low [deadlock priority](/sql/t-sql/statements/set-deadlock-priority-transact-sql). If there is a concurrency conflict, priority is given to the application workload queries. Depending on the application workload patterns, this might cause occasional gaps in the collected data for some datasets. +To avoid concurrency conflicts such as blocking and deadlocks between data collection and database workloads running on your Azure SQL resources, the monitoring queries use short [lock timeouts](/sql/relational-databases/sql-server-transaction-locking-and-row-versioning-guide#customizing-the-lock-time-out) and low [deadlock priority](/sql/t-sql/statements/set-deadlock-priority-transact-sql). If there is a concurrency conflict, priority is given to the application workload queries. + +You might observe gaps in the collected data if the overall resource utilization is high, or if concurrency conflicts occur. In these cases, monitoring queries might be deprioritized in favor of application workloads. ### Data collection in elastic pools @@ -103,11 +109,11 @@ To monitor an elastic pool, you must designate one database in the pool as the * > [!TIP] > You can add an empty database to each elastic pool you want to monitor, and designate it as the anchor database. This way, you can move other databases in and out of the pool, or between pools, without interrupting elastic pool monitoring. -Data collected from the anchor database contains pool-level metrics, and certain database-level performance metrics for every database in the pool. For example, this includes resource utilization and request rate metrics for each database. For some scenarios, monitoring an elastic pool as a whole makes it unnecessary to monitor each individual database in the pool. +Data collected from the anchor database contains pool-level metrics, and certain database-level performance metrics for every database in the pool, such as resource utilization and request rate metrics for each database. For some scenarios, adding an elastic pool SQL target to monitor an elastic pool as a whole can make it unnecessary to monitor each individual database in the pool. -Certain monitoring data such as pool-level CPU, memory, storage utilization, and wait statistics is only collected at the elastic pool level because it cannot be attributed to an individual database in a pool. Conversely, certain other data such as query runtime statistics, database properties, table and index metadata is available only at the database level. +Certain monitoring data such as pool-level CPU, memory, storage utilization, and wait statistics is only collected at the elastic pool level because it cannot be attributed to an individual database in a pool. Conversely, certain other data such as query runtime statistics, database properties, table and index metadata is only available if you add individual databases as SQL targets. -If you add individual databases from an elastic pool as database watcher targets, you should add the elastic pool as a target as well. This way, you get a more complete view of the database and pool performance. +If you add individual databases from an elastic pool as SQL targets, you should add the elastic pool as a SQL target as well. This way, you get a more complete view of the database and pool performance. #### Monitor dense elastic pools @@ -116,7 +122,7 @@ A [dense elastic pool](./database/elastic-pool-resource-management.md) contains Compute resources available to database watcher queries in a dense elastic pool are further limited to avoid affecting application queries. Because of this, database watcher might not be able to collect monitoring data from every database in a dense elastic pool. > [!TIP] -> To monitor a dense elastic pool, enable monitoring at the pool level by adding the elastic pool as a target. +> To monitor a dense elastic pool, enable monitoring at the pool level by adding the elastic pool as a SQL target. > > It is not recommended to monitor more than a few individual databases in a dense elastic pool. You might see gaps in the collected data or larger than expected intervals between data samples due to insufficient compute resources available to database watcher queries. @@ -138,11 +144,11 @@ Customers can choose to store collected SQL monitoring data in one of three data To fully control data residency for collected SQL monitoring data, customers must choose a database on an Azure Data Explorer cluster as the data store. -Customers can also align the geography and region of their Azure Data Explorer cluster to the geography and region of the Azure SQL resources being monitored. When the Azure SQL resources are located in multiple regions, customers might need to create multiple watchers and multiple Azure Data Explorer clusters to satisfy their data residency requirements. +Customers can align the geography and region of their Azure Data Explorer cluster to the geography and region of the Azure SQL resources being monitored. When the Azure SQL resources are located in multiple regions, customers might need to create multiple watchers and multiple Azure Data Explorer clusters to satisfy their data residency requirements. ## Datasets -This section describes the datasets available for each target type, including collection frequencies and table names in the data store. +This section describes the datasets available for each SQL target type, including collection frequencies and table names in the data store. > [!NOTE] > During preview, datasets might be added and removed. Dataset properties such as name, description, collection frequency, and available columns are subject to change. @@ -230,7 +236,7 @@ This section describes the datasets available for each target type, including co ### Common columns -For each target type, datasets have common columns, as described in the following tables. +For each SQL target type, datasets have common columns, as described in the following tables. # [SQL database](#tab/sqldb) diff --git a/azure-sql/database-watcher-faq.yml b/azure-sql/database-watcher-faq.yml index 21023ce2b0d..6ff7c85c9d0 100644 --- a/azure-sql/database-watcher-faq.yml +++ b/azure-sql/database-watcher-faq.yml @@ -5,7 +5,7 @@ metadata: description: Frequently asked questions about database watcher for Azure SQL author: dimitri-furman ms.author: dfurman - ms.date: 10/07/2024 + ms.date: 11/11/2024 ms.reviewer: wiassaf ms.service: azure-sql ms.subservice: monitoring @@ -78,7 +78,7 @@ sections: - question: | Does a watcher have an identity I can use to grant it access to my Azure resources? - answer: Yes. You can use either a system assigned or a user assigned managed identity for a watcher. Grant access to this identity to allow a watcher to [collect](database-watcher-manage.md#grant-access-to-sql-targets) and [ingest](database-watcher-manage.md#grant-access-to-data-store) data. Revoke access at any time to stop collection of monitoring data. + answer: Yes. You can use either a system assigned or a user assigned managed identity. Grant access to this identity to allow a watcher to [collect](database-watcher-manage.md#grant-access-to-sql-targets) and [ingest](database-watcher-manage.md#grant-access-to-data-store) data. Revoke access at any time to stop collection of monitoring data. - question: | Are there any built-in RBAC roles or actions specific to database watcher? @@ -86,7 +86,7 @@ sections: - question: | What permissions are required to access database watcher dashboards? - answer: To access dashboards, users require the [assignment](/azure/role-based-access-control/role-assignments-portal) of the RBAC **Reader** role on the watcher resource or on a higher scope such as resource group, subscription, or management group. They also require the [assignment](database-watcher-manage.md#grant-users-and-groups-access-to-the-data-store) of the **Viewer** RBAC role on the Azure Data Explorer or Real-Time Analytics database. These assignments can be made directly or via Microsoft Entra ID group membership. + answer: To access dashboards, users require the [assignment](/azure/role-based-access-control/role-assignments-portal) of the **Reader** RBAC role on the watcher resource or on a higher scope such as resource group, subscription, or management group. They also require the [assignment](database-watcher-manage.md#grant-users-and-groups-access-to-the-data-store) of the **Viewer** RBAC role on the Azure Data Explorer or Real-Time Analytics database. These assignments can be made directly or via Microsoft Entra ID group membership. - name: Data store questions: @@ -130,7 +130,7 @@ sections: - question: | Does it monitor secondary replicas? - answer: Yes. All types of secondary replicas, including readable high availability replicas, geo-replicas, and Hyperscale named replicas are supported. If a Hyperscale database has more than one high availability replica, only one of these replicas can be monitored at a given point in time. + answer: Yes. All types of secondary replicas, including readable high availability replicas, geo-replicas, and Hyperscale named replicas are supported. If a Hyperscale database has more than one high availability replica, only one of these replicas is monitored at a given point in time. - question: | How does it connect to monitoring targets? diff --git a/azure-sql/database-watcher-manage.md b/azure-sql/database-watcher-manage.md index 24bebf89181..9a49a7ea482 100644 --- a/azure-sql/database-watcher-manage.md +++ b/azure-sql/database-watcher-manage.md @@ -5,7 +5,7 @@ description: Setup and configuration details for database watcher author: dimitri-furman ms.author: dfurman ms.reviewer: wiassaf -ms.date: 10/23/2024 +ms.date: 11/06/2024 ms.service: azure-sql ms.subservice: monitoring ms.topic: how-to @@ -93,12 +93,17 @@ To use database watcher, the following prerequisites are required. 1. On the **Review + create** tab, review watcher configuration, and select **Create**. If you select the default option to create a new Azure Data Explorer cluster, the deployment typically takes 15-20 minutes. If you select a database on an existing Azure Data Explorer cluster, on a free Azure Data Explorer cluster, or in Real-Time Analytics, the deployment typically takes up to five minutes. 1. Once the deployment completes, grant the watcher [access to SQL targets](#grant-access-to-sql-targets). + +1. You might also need to grant the watcher [access to the data store](#grant-access-to-data-store). + - Access to a database on a new or existing Azure Data Explorer cluster is granted automatically when the watcher is created if the user creating the watcher is a member of the **Owner** RBAC role for the cluster. - However, you must [grant access to data store using a KQL command](#grant-access-to-data-store) if you select a database in: - - Real-Time Analytics in Microsoft Fabric - - A free Azure Data Explorer cluster + - Real-Time Analytics in Microsoft Fabric. + - A free Azure Data Explorer cluster. + +1. [Create](#create-a-managed-private-endpoint) and approve managed private endpoints if you want to use [private connectivity](database-watcher-overview.md#private-connectivity). -1. [Create managed private endpoints](#create-a-managed-private-endpoint) if you want to use [private connectivity](database-watcher-overview.md#private-connectivity). + - If public access on your SQL targets, the data store, and key vault is enabled and you want to use public connectivity, make sure that all [public connectivity](database-watcher-overview.md#public-connectivity) prerequisites are satisfied. ## Start and stop a watcher @@ -132,9 +137,9 @@ To enable database watcher monitoring for an Azure SQL database, elastic pool, o 1. To add a target, on the **SQL targets** page, select **Add**. 1. Find the Azure SQL resource you want to monitor. Select the resource type and subscription, and then select the SQL target from the list of resources. The SQL target can be in any subscription within the same Microsoft Entra ID tenant as the watcher. -1. To monitor the primary replica and a high availability [secondary replica](./database/read-scale-out.md) of a database, elastic pool, or SQL managed instance, add *two separate targets* for the same resource, and check the **Read intent** box for *one of them*. +1. To monitor the primary replica and a high availability [secondary replica](./database/read-scale-out.md) of a database, elastic pool, or SQL managed instance, add *two separate SQL targets* for the same resource, and check the **Read intent** box for *one of them*. Similarly, create two separate SQL targets for a geo-replica and its high availability secondary replica, if any. - Checking the **Read intent** box configures the SQL target for the high availability secondary replica only. - - Do not check the **Read intent** box if you want to monitor only the primary replica, or if a high availability secondary replica does not exist for this resource, or if the [read scale-out](./database/read-scale-out.md) feature is disabled. + - Do not check the **Read intent** box if you want to monitor only the primary replica or only the geo-replica, or if a high availability secondary replica does not exist for this resource, or if the [read scale-out](./database/read-scale-out.md) feature is disabled. By default, database watcher uses Microsoft Entra authentication when connecting to SQL targets. If you want the watcher to use SQL authentication, check the **Use SQL authentication** box and enter the required details. For more information, see [Additional configuration to use SQL authentication](#additional-configuration-to-use-sql-authentication). @@ -173,13 +178,13 @@ To create a managed private endpoint for a watcher: | Azure Data Explorer cluster | `Microsoft.Kusto/clusters` | `cluster` | | Key vault | `Microsoft.KeyVault/vaults` | `vault` | -1. Select the resource for which you want to create a private endpoint. This can be an Azure SQL logical server or SQL managed instance, an Azure Data Explorer cluster, or a key vault. +1. Select the resource for which you want to create a private endpoint. This can be an Azure SQL logical server, a SQL managed instance, an Azure Data Explorer cluster, or a key vault. - Creating a private endpoint for an Azure SQL Database logical server enables database watcher private connectivity for all database and elastic pool targets on that server. 1. Optionally, enter the description for the private endpoint. This can help the resource owner approve the request. -1. Select **Create**. It can take one or two minutes to create a private endpoint. A private endpoint is created once its provisioning state changes from **Accepted** or **Running** to **Succeeded**. Refresh the view to see the current provisioning state. +1. Select **Create**. It can take a few minutes to create a private endpoint. A private endpoint is created once its provisioning state changes from **Accepted** or **Running** to **Succeeded**. Refresh the view to see the current provisioning state. > [!IMPORTANT] > The private endpoint is created in the **Pending** state. It must be approved by the resource owner before database watcher can use it to connect to the resource. @@ -194,7 +199,7 @@ If a watcher is already running when a private endpoint is approved, it must be > [!TIP] > -> You need to create an additional private endpoint for your Azure Data Explorer cluster if public connectivity to the cluster is disabled. For more information, see [Private connectivity to the data store](#private-connectivity-to-the-data-store). +> You need to create an additional private endpoint for your Azure Data Explorer cluster if cluster public connectivity is disabled. For more information, see [Private connectivity to the data store](#private-connectivity-to-the-data-store). ### Delete a managed private endpoint @@ -231,11 +236,11 @@ The following considerations help you choose the type of managed identity for a - Enabled by default when you create a watcher. - Always associated with a single watcher. - Created and deleted with the watcher. - - If you disable a system assigned identity for a watcher, any access granted to that identity is lost. Re-enabling the system assigned identity for the same watcher creates a new identity with a different object (principal) ID. You need to grant access to [SQL targets](#grant-access-to-sql-targets), [key vault](#additional-configuration-to-use-sql-authentication), and the [data store](#grant-access-to-data-store) to this new identity. + - If you disable a system assigned identity for a watcher, any access granted to that identity is lost. Re-enabling the system assigned identity for the same watcher creates a new, different identity with a different object (principal) ID. You need to grant access to [SQL targets](#grant-access-to-sql-targets), [key vault](#additional-configuration-to-use-sql-authentication), and the [data store](#grant-access-to-data-store) to this new identity. - **User assigned** - Is in effect only if the system assigned identity is disabled for the watcher. - - The same user assigned identity can be assigned to multiple watchers to simplify access management when [monitoring large Azure SQL estates](#monitor-large-estates). Instead of granting access to multiple system assigned identities, access can be granted to a single user assigned identity. + - The same user assigned identity can be assigned to multiple watchers to simplify access management, for example when [monitoring large Azure SQL estates](#monitor-large-estates). Instead of granting access to the system assigned identities of multiple watchers, access can be granted to a single user assigned identity. - To support separation of duties, identity management can be separate from watcher management. A user assigned identity can be created and granted access by a different user, before or after the watcher is created. - Conversely, when a watcher is deleted, the user assigned identity and its access remain unchanged. The same identity can be then used for a new watcher. - Specifying more than one user assigned identity for a watcher is not supported. @@ -274,7 +279,7 @@ When you delete a watcher that has its system assigned managed identity enabled, You must grant access to a recreated watcher, even if you use the same watcher name. -When you delete a watcher, the Azure resources referenced as its targets and data store are not deleted. You retain collected SQL monitoring data in the data store, and you can use the same Azure Data Explorer or Real-Time Analytics database as the data store if you create a new watcher later. +When you delete a watcher, the Azure resources referenced as its SQL targets and the data store are not deleted. You retain collected SQL monitoring data in the data store, and you can use the same Azure Data Explorer or Real-Time Analytics database as the data store if you create a new watcher later. ## Grant access to SQL targets diff --git a/azure-sql/database-watcher-overview.md b/azure-sql/database-watcher-overview.md index 6bfc42f6990..de341f06748 100644 --- a/azure-sql/database-watcher-overview.md +++ b/azure-sql/database-watcher-overview.md @@ -5,7 +5,7 @@ description: An overview of database watcher for Azure SQL, a managed monitoring author: dimitri-furman ms.author: dfurman ms.reviewer: wiassaf -ms.date: 11/04/2024 +ms.date: 11/11/2024 ms.service: azure-sql ms.subservice: monitoring ms.topic: conceptual @@ -200,7 +200,7 @@ This section describes recent database watcher fixes, changes, and improvements. | Time period | Changes | |:--|:--| | November 2024 | - Increase the limit on the number of SQL targets per watcher from 50 to 100. | -| October 2024 | - Fix a bug where the **Table metadata** dataset was not collected if there are any views with invalid table references, or any tables with multiple column check constraints.
- Add support for user assigned identity. For more information, see [Modify watcher identity](database-watcher-manage.md#modify-watcher-identity).
- Automatically grant the watcher access to key vault secrets when adding a SQL target that uses SQL authentication.
- Automatically grant the watcher access to an Azure Data Explorer database when adding a data store to an existing watcher.
- Add the feedback button on the **Overview** page and other pages. | +| October 2024 | - Fix bugs where the **Table metadata** dataset was not collected if there were any views with invalid table references, or any tables with multiple column check constraints.
- Add support for using a user assigned identity as the watcher identity. For more information, see [Modify watcher identity](database-watcher-manage.md#modify-watcher-identity).
- Automatically grant the watcher access to key vault secrets when adding a SQL target that uses SQL authentication.
- Automatically grant the watcher access to an Azure Data Explorer database when adding a data store to an existing watcher.
- Add the feedback button on the **Overview** page and other pages. | | September 2024 | - Fix a bug where the number of user logical sessions in the **Session statistics** dataset was always the same as the number of user sessions, even if [MARS](/sql/relational-databases/native-client/features/using-multiple-active-result-sets-mars) logical sessions were used.
- Fix a bug where elastic pool storage utilization wasn't reported correctly for Hyperscale elastic pools.
- Resolve an issue where for certain datasets, the first sample collected after a watcher restart might contain data that has already been collected before restart.
- Improve collection query performance to avoid timeouts for the **Table metadata** dataset.
- Improve collection reliability for the **Query runtime statistics** and **Query wait statistics** datasets on SQL Managed Instance.
- Add failover-related columns to the **Database replicas** dataset for SQL Managed Instance.
- Add index operational statistics columns to the **Index metadata** datasets.
- Add support for selecting multiple Azure SQL databases in the **Add SQL target** blade.| | August 2024 | - Enable database watcher in the **Central US**, **East US 2**, **North Europe**, and **Sweden Central** Azure regions.
- Add subscription and resource group filters in estate [dashboards](#dashboards). | | July 2024 | - Fix a bug where the **Performance counters** datasets were not collected from databases with a case-sensitive catalog collation, or managed instances with a case-sensitive database collation.
- Fix a bug where data was not collected if the database name in the SQL metadata had a different case than the database name in the Azure Resource Manager (ARM) metadata.
- Fix a bug where the **Query runtime statistics** and **Query wait statistics** datasets were not collected in databases with a large volume of new queries and query plans inserted into Query Store tables.
- Resolve an issue where the **Geo-replicas** and **Replicas** datasets were not collected from Hyperscale databases.
- Add the `subscription_id` and `resource_group_name` [common columns](database-watcher-data.md#common-columns) to all datasets. Requires a one-time [restart](database-watcher-manage.md#start-and-stop-a-watcher) of a watcher.
- Add the `resource_id` [common column](database-watcher-data.md#common-columns) to all datasets. The data appears for SQL targets added in July 2024 or later. To make data appear for an existing SQL target, [remove](database-watcher-manage.md#remove-sql-targets-from-a-watcher) and [re-add](database-watcher-manage.md#add-sql-targets-to-a-watcher) the target, and [restart](database-watcher-manage.md#start-and-stop-a-watcher) the watcher. | diff --git a/azure-sql/database/hyperscale-performance-diagnostics.md b/azure-sql/database/hyperscale-performance-diagnostics.md index 27b990d1ed8..42ee0155092 100644 --- a/azure-sql/database/hyperscale-performance-diagnostics.md +++ b/azure-sql/database/hyperscale-performance-diagnostics.md @@ -35,7 +35,7 @@ The following wait types (in [sys.dm_os_wait_stats](/sql/relational-databases/sy ## Page server reads -The compute replicas do not cache a full copy of the database locally. The data local to the compute replica is stored in the buffer pool (in memory) and in the local resilient buffer pool extension (RBPEX) cache that is a partial (non-covering) cache of data pages. This local RBPEX cache is sized proportionally to the compute size and is three times the memory of the compute tier. RBPEX is similar to the buffer pool in that it has the most frequently accessed data. Each page server, on the other hand, has a covering RBPEX cache for the portion of the database it maintains. +The compute replicas do not cache a full copy of the database locally. The data local to the compute replica is stored in the buffer pool (in memory) and in the local resilient buffer pool extension (RBPEX) cache that is a partial (non-covering) cache of data pages. This local RBPEX cache is sized proportionally to the compute size. RBPEX is similar to the buffer pool in that it has the most frequently accessed data. Each page server, on the other hand, has a covering RBPEX cache for the portion of the database it maintains. When a read is issued on a compute replica, if the data doesn't exist in the buffer pool or local RBPEX cache, a getPage(pageId, LSN) function call is issued, and the page is fetched from the corresponding page server. Reads from page servers are remote reads and are thus slower than reads from the local RBPEX. When troubleshooting IO-related performance problems, we need to be able to tell how many IOs were done via relatively slower remote page server reads. diff --git a/docs/relational-databases/performance/query-store-hints.md b/docs/relational-databases/performance/query-store-hints.md index 24779718fdd..818efa26a4b 100644 --- a/docs/relational-databases/performance/query-store-hints.md +++ b/docs/relational-databases/performance/query-store-hints.md @@ -3,7 +3,7 @@ title: "Query Store hints" description: "Learn about the Query Store hints feature, which can be used to shape query plans without changing application code." author: MikeRayMSFT ms.author: mikeray -ms.date: 01/08/2024 +ms.date: 11/11/2024 ms.service: sql ms.subservice: performance ms.topic: conceptual @@ -111,7 +111,7 @@ When hints are applied, the following result set appears in the `StmtSimple` ele ### Query Store hints and availability groups -For more information, see [Query Store for secondary replicas](query-store-for-secondary-replicas.md). +Query Store hints have no effect on secondary replicas unless Query Store for secondary replicas is enabled. For more information, see [Query Store for secondary replicas](query-store-for-secondary-replicas.md). - Prior to [!INCLUDE [sssql22-md](../../includes/sssql22-md.md)], Query Store hints can be applied against the primary replica of an availability group. - Starting with [!INCLUDE [sssql22-md](../../includes/sssql22-md.md)], when Query Store for secondary replicas is enabled, Query Store hints are also replica-aware for secondary replicas in availability groups.