Add docs for updated deletion api

docs/sources/api/_index.md

@@ -69,6 +69,9 @@ These endpoints are exposed by the ruler:
 
 These endpoints are exposed by the compactor:
 - [`GET /compactor/ring`](#get-compactorring)
+- [`POST /loki/api/v1/delete`](#post-lokiapiv1delete)
+- [`GET /loki/api/v1/delete`](#get-lokiapiv1delete)
+- [`DELETE /loki/api/v1/delete`](#delete-lokiapiv1delete)
 
 A [list of clients](../clients) can be found in the clients documentation.
 
@@ -797,10 +800,6 @@ In microservices mode, the `/ingester/flush_shutdown` endpoint is exposed by the
 
 Displays a web page with the distributor hash ring status, including the state, healthy and last heartbeat time of each distributor.
 
-### `GET /compactor/ring`
-
-Displays a web page with the compactor hash ring status, including the state, healthy and last heartbeat time of each compactor.
-
 ## `GET /metrics`
 
 `/metrics` exposes Prometheus metrics. See
@@ -1094,3 +1093,21 @@ GET /prometheus/api/v1/alerts
 Prometheus-compatible rules endpoint to list all active alerts.
 
 For more information, please check out the Prometheus [alerts](https://prometheus.io/docs/prometheus/latest/querying/api/#alerts) documentation.
+
+## Compactor
+
+### `GET /compactor/ring`
+
+Displays a web page with the compactor hash ring status, including the state, healthy and last heartbeat time of each compactor.
+
+### `POST /loki/api/v1/delete`
+
+Create a new delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-log-entry-deletion).
+
+### `GET /loki/api/v1/delete`
+
+List the existing delete requests for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#list-delete-requests).
+
+### `DELETE /loki/api/v1/delete`
+
+Remove a delete request for the authenticated tenant. More details can be found in the [logs deletion documentation](../operations/storage/logs-deletion.md#request-cancellation-of-a-delete-request).

docs/sources/configuration/_index.md

@@ -2026,6 +2026,13 @@ compacts index shards to more performant forms.
 # CLI flag: -boltdb.shipper.compactor.max-compaction-parallelism
 [max_compaction_parallelism: <int> | default = 1]
 
+# Deletion mode.
+# Can be one of "disabled", "whole-stream-deletion", "filter-only" or "filter-and-delete".
+# When set to (the default) value of "whole-stream-deletion" and if
+# retention_enabled is true then the Log Entry Deletion API is available
+# CLI flag: -boltdb.shipper.compactor.deletion-mode
+[deletion_mode: <string> | default = "whole-stream-deletion"]
+
 # The hash ring configuration used by compactors to elect a single instance for running compactions
 # The CLI flags prefix for this block config is: boltdb.shipper.compactor.ring
 [compactor_ring: <ring>]

docs/sources/operations/storage/logs-deletion.md

@@ -4,10 +4,11 @@ weight: 60
 ---
 # Log Entry Deletion
 
-<span style="background-color:#f3f973;">Log entry deletion is experimental. It is only supported for the BoltDB Shipper index store.</span>
+<span style="background-color:#f3f973;">Log entry deletion is only supported for the BoltDB Shipper index store.</span>
+
+Grafana Loki supports the deletion of log entries from a specified stream.
+Log entries that fall within a specified time window and and match an optional line filter are those that will be deleted.
 
-Grafana Loki supports the deletion of log entries from specified streams.
-Log entries that fall within a specified time window are those that will be deleted.
 
 The Compactor component exposes REST endpoints that process delete requests.
 Hitting the endpoint specifies the streams and the time window.
@@ -17,7 +18,12 @@ Log entry deletion relies on configuration of the custom logs retention workflow
 
 ## Configuration
 
-Enable log entry deletion by setting `retention_enabled` to true in the Compactor's configuration. See the example in [Retention Configuration](../retention#retention-configuration).
+Enable log entry deletion by setting `retention_enabled` to true and `deletion_enabled` to `whole-stream-deletion`, `filter-only` or `filter-and-delete` in the Compactor's configuration. See the example in [Retention Configuration](../retention#retention-configuration).
+
+With `whole-stream-deletion` all the log entries matching the query given in the delete request are removed.
+With `filter-only` log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from the on-disk chunks.
+With `filter-and-delete` log lines matching the query in the delete request are filtered out when querying Loki and they are also removed from the on-disk chunks.
+
 
 A delete request may be canceled within a configurable cancellation period. Set the `delete_request_cancel_period` in the Compactor's YAML configuration or on the command line when invoking Loki. Its default value is 24h.
 
@@ -28,39 +34,41 @@ The Compactor exposes endpoints to allow for the deletion of log entries from sp
 ### Request log entry deletion
 
 ```
-POST /loki/api/admin/delete
-PUT /loki/api/admin/delete
+POST /loki/api/v1/delete
+PUT /loki/api/v1/delete
 ```
 
 Query parameters:
 
-* `match[]=<series_selector>`: Repeated label matcher argument that identifies the streams from which to delete. At least one `match[]` argument must be provided.
+* `query=<series_selector>`: query argument that identifies the streams from which to delete with optional line filters.
 * `start=<rfc3339 | unix_timestamp>`: A timestamp that identifies the start of the time window within which entries will be deleted. If not specified, defaults to 0, the Unix Epoch time.
 * `end=<rfc3339 | unix_timestamp>`: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time.
 
 A 204 response indicates success.
 
-URL encode the `match[]` parameter. This sample form of a cURL command URL encodes `match[]={foo="bar"}`:
+URL encode the `query` parameter. This sample form of a cURL command URL encodes `query={foo="bar"}`:
 
 ```
-curl -g -X POST \ 
-  'http://127.0.0.1:3100/loki/api/admin/delete?match[]={foo="bar"}&start=1591616227&end=1591619692' \ 
+curl -g -X POST \
+  'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}&start=1591616227&end=1591619692' \
   -H 'x-scope-orgid: 1'
+
+  The query parameter can also include filter operations. For example `query={foo="bar"} |= "other"` will filter out lines that contain the string "other" for the streams matching the stream selector `{foo="bar"}`.
 ```
 
 ### List delete requests
 
 List the existing delete requests using the following API:
 
 ```
-GET /loki/api/admin/delete
+GET /loki/api/v1/delete
 ```
 
 Sample form of a cURL command:
 
 ```
 curl -X GET \
-  <compactor_addr>/loki/api/admin/delete \
+  <compactor_addr>/loki/api/v1/delete \
   -H 'x-scope-orgid: <orgid>'
 ```
 
@@ -73,8 +81,7 @@ Loki allows cancellation of delete requests until the requests are picked up for
 Cancel a delete request using this Compactor endpoint:
 
 ```
-POST /loki/api/admin/cancel_delete_request
-PUT /loki/api/admin/cancel_delete_request
+DELETE /loki/api/v1/delete
 ```
 
 Query parameters:
@@ -86,7 +93,7 @@ A 204 response indicates success.
 Sample form of a cURL command:
 
 ```
-curl -X POST \
-  '<compactor_addr>/loki/api/admin/cancel_delete_request?request_id=<request_id>' \
+curl -X DELETE \
+  '<compactor_addr>/loki/api/v1/delete?request_id=<request_id>' \
   -H 'x-scope-orgid: <tenant-id>'
 ```