Merge pull request #82058 from openshift-cherrypick-robot/cherry-pick…

…-80869-to-enterprise-4.17 [enterprise-4.17] OADP-4655 Added section for kopia env variables
openshift · Sep 18, 2024 · 8606aeb · 8606aeb
2 parents 3e0d59e + 539a68f
commit 8606aeb
Show file tree

Hide file tree

Showing 5 changed files with 431 additions and 1 deletion.
diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml
@@ -3378,13 +3378,15 @@ Topics:
       File: oadp-using-data-mover-for-csi-snapshots-doc
     - Name: Using OADP 1.2 Data Mover with Ceph storage
       File: oadp-12-data-mover-ceph-doc
-  - Name: OADP 1.3 Data Mover
+  - Name: OADP 1.3 Data Mover and later
     Dir: installing
     Topics:
     - Name: About the OADP 1.3 Data Mover
       File: about-oadp-1-3-data-mover
     - Name: Backing up and restoring volumes by using CSI snapshots data movement
       File: oadp-backup-restore-csi-snapshots
+    - Name: Overriding Kopia algorithms
+      File: overriding-kopia-algorithms
   - Name: Troubleshooting
     File: troubleshooting
   - Name: OADP API

diff --git a/...tore/application_backup_and_restore/installing/overriding-kopia-algorithms.adoc b/...tore/application_backup_and_restore/installing/overriding-kopia-algorithms.adoc
@@ -0,0 +1,18 @@
+:_mod-docs-content-type: ASSEMBLY
+[id="overriding-kopia-algorithms"]
+= Overriding Kopia hashing, encryption, and splitter algorithms
+include::_attributes/common-attributes.adoc[]
+:context: overriding-kopia-algorithms
+
+
+toc::[]
+
+You can override the default values of Kopia hashing, encryption, and splitter algorithms by using specific environment variables in the Data Protection Application (DPA).
+
+// configuring the DPA with Kopia environment variables
+include::modules/oadp-kopia-configuring-algorithms.adoc[leveloffset=+1]
+// use case to demonstrate the usage of Kopia environment variables
+include::modules/oadp-usecase-kopia-override-algorithms.adoc[leveloffset=+1]
+// benchmarking Kopia algorithms
+include::modules/oadp-kopia-algorithms-benchmarking.adoc[leveloffset=+1]
+
diff --git a/modules/oadp-kopia-algorithms-benchmarking.adoc b/modules/oadp-kopia-algorithms-benchmarking.adoc
@@ -0,0 +1,178 @@
+// Module included in the following assemblies:
+//
+// * backup_and_restore/application_backup_and_restore/installing/overriding-kopia-algorithms.adoc          
+
+:_mod-docs-content-type: PROCEDURE
+[id="oadp-kopia-algorithms-benchmarking_{context}"]
+= Benchmarking Kopia hashing, encryption, and splitter algorithms
+
+You can run Kopia commands to benchmark the hashing, encryption, and splitter algorithms. Based on the benchmarking results, you can select the most suitable algorithm for your workload. In this procedure, you run the Kopia benchmarking commands from a pod on the cluster. The benchmarking results can vary depending on CPU speed, available RAM, disk speed, current I/O load, and so on.
+
+.Prerequisites
+
+* You have installed the {oadp-short} Operator.
+* You have an application with persistent volumes running in a separate namespace.
+* You have run a backup of the application with Container Storage Interface (CSI) snapshots.
+
+.Procedure
+
+. Configure a pod as shown in the following example. Make sure you are using the `oadp-mustgather` image for {oadp-short} version 1.3 and later.
++
+.Example pod configuration
++
+[source,yaml]
+----
+apiVersion: v1
+kind: Pod
+metadata:
+  name: oadp-mustgather-pod
+  labels:
+    purpose: user-interaction
+spec:
+  containers:
+  - name: oadp-mustgather-container
+    image: registry.redhat.io/oadp/oadp-mustgather-rhel9:v1.3
+    command: ["sleep"]
+    args: ["infinity"]
+----
++
+[NOTE]
+=====
+The Kopia client is available in the `oadp-mustgather` image.
+=====
++
+. Create the pod by running the following command:
++
+[source,terminal]
+----
+$ oc apply -f <pod_config_file_name> # <1>
+----
+<1> Specify the name of the YAML file for the pod configuration.
++
+. Verify that the Security Context Constraints (SCC) on the pod is `anyuid`, so that Kopia can connect to the repository.
++ 
+[source,terminal]
+----
+$ oc describe pod/oadp-mustgather-pod | grep scc
+----
++
+.Example output
++
+[source,terminal]
+----
+openshift.io/scc: anyuid
+----
++
+. Connect to the pod via SSH by running the following command:
++ 
+[source,terminal]
+----
+$ oc -n openshift-adp rsh pod/oadp-mustgather-pod
+----
++
+. Connect to the Kopia repository by running the following command:
++
+[source,terminal]
+----
+sh-5.1# kopia repository connect s3 \
+  --bucket=<bucket_name> \ # <1>
+  --prefix=velero/kopia/<application_namespace> \ # <2>
+  --password=static-passw0rd \ # <3>
+  --access-key="<access_key>" \ # <4>
+  --secret-access-key="<secret_access_key>" \ # <5>
+  --endpoint=<bucket_endpoint> \ # <6>
+----
+<1> Specify the object storage provider bucket name.
+<2> Specify the namespace for the application.
+<3> This is the Kopia password to connect to the repository.
+<4> Specify the object storage provider access key.
+<5> Specify the object storage provider secret access key.
+<6> Specify the bucket endpoint. You do not need to specify the bucket endpoint, if you are using {aws-short} S3 as the storage provider.
++
+[NOTE]
+====
+This is an example command. The command can vary based on the object storage provider.
+====
++ 
+. To benchmark the hashing algorithm, run the following command:
++
+[source,terminal]
+----
+sh-5.1# kopia benchmark hashing
+----
++
+.Example output
+[source,terminal]
+----
+Benchmarking hash 'BLAKE2B-256' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'BLAKE2B-256-128' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'BLAKE2S-128' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'BLAKE2S-256' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'BLAKE3-256' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'BLAKE3-256-128' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'HMAC-SHA224' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'HMAC-SHA256' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'HMAC-SHA256-128' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'HMAC-SHA3-224' (100 x 1048576 bytes, parallelism 1)
+Benchmarking hash 'HMAC-SHA3-256' (100 x 1048576 bytes, parallelism 1)
+     Hash                 Throughput
+-----------------------------------------------------------------
+  0. BLAKE3-256           15.3 GB / second
+  1. BLAKE3-256-128       15.2 GB / second
+  2. HMAC-SHA256-128      6.4 GB / second
+  3. HMAC-SHA256          6.4 GB / second
+  4. HMAC-SHA224          6.4 GB / second
+  5. BLAKE2B-256-128      4.2 GB / second
+  6. BLAKE2B-256          4.1 GB / second
+  7. BLAKE2S-256          2.9 GB / second
+  8. BLAKE2S-128          2.9 GB / second
+  9. HMAC-SHA3-224        1.6 GB / second
+ 10. HMAC-SHA3-256        1.5 GB / second
+-----------------------------------------------------------------
+Fastest option for this machine is: --block-hash=BLAKE3-256
+----
++
+. To benchmark the encryption algorithm, run the following command:
++
+[source,terminal]
+----
+sh-5.1# kopia benchmark encryption
+----
++
+.Example output
+[source,terminal]
+----
+Benchmarking encryption 'AES256-GCM-HMAC-SHA256'... (1000 x 1048576 bytes, parallelism 1)
+Benchmarking encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (1000 x 1048576 bytes, parallelism 1)
+     Encryption                     Throughput
+-----------------------------------------------------------------
+  0. AES256-GCM-HMAC-SHA256         2.2 GB / second
+  1. CHACHA20-POLY1305-HMAC-SHA256  1.8 GB / second
+-----------------------------------------------------------------
+Fastest option for this machine is: --encryption=AES256-GCM-HMAC-SHA256
+----
++
+. To benchmark the splitter algorithm, run the following command:
++
+[source,terminal]
+----
+sh-5.1# kopia benchmark splitter
+----
++
+.Example output
+[source,terminal]
+----
+splitting 16 blocks of 32MiB each, parallelism 1
+DYNAMIC                     747.6 MB/s count:107 min:9467 10th:2277562 25th:2971794 50th:4747177 75th:7603998 90th:8388608 max:8388608
+DYNAMIC-128K-BUZHASH        718.5 MB/s count:3183 min:3076 10th:80896 25th:104312 50th:157621 75th:249115 90th:262144 max:262144
+DYNAMIC-128K-RABINKARP      164.4 MB/s count:3160 min:9667 10th:80098 25th:106626 50th:162269 75th:250655 90th:262144 max:262144
+# ...
+FIXED-512K                  102.9 TB/s count:1024 min:524288 10th:524288 25th:524288 50th:524288 75th:524288 90th:524288 max:524288
+FIXED-8M                    566.3 TB/s count:64 min:8388608 10th:8388608 25th:8388608 50th:8388608 75th:8388608 90th:8388608 max:8388608
+-----------------------------------------------------------------
+  0. FIXED-8M                  566.3 TB/s   count:64 min:8388608 10th:8388608 25th:8388608 50th:8388608 75th:8388608 90th:8388608 max:8388608
+  1. FIXED-4M                  425.8 TB/s   count:128 min:4194304 10th:4194304 25th:4194304 50th:4194304 75th:4194304 90th:4194304 max:4194304
+  # ...
+ 22. DYNAMIC-128K-RABINKARP    164.4 MB/s   count:3160 min:9667 10th:80098 25th:106626 50th:162269 75th:250655 90th:262144 max:262144
+----
+
diff --git a/modules/oadp-kopia-configuring-algorithms.adoc b/modules/oadp-kopia-configuring-algorithms.adoc
@@ -0,0 +1,54 @@
+// Module included in the following assemblies:
+//
+// * backup_and_restore/application_backup_and_restore/installing/overriding-kopia-algorithms.adoc          
+
+:_mod-docs-content-type: PROCEDURE
+[id="oadp-kopia-configuring-algorithms_{context}"]
+= Configuring the DPA to override Kopia hashing, encryption, and splitter algorithms
+
+You can use an {oadp-first} option to override the default Kopia algorithms for hashing, encryption, and splitter to improve Kopia performance or to compare performance metrics. You can set the following environment variables in the `spec.configuration.velero.podConfig.env` section of the DPA:
+
+* `KOPIA_HASHING_ALGORITHM`
+* `KOPIA_ENCRYPTION_ALGORITHM`
+* `KOPIA_SPLITTER_ALGORITHM`
+
+.Prerequisites
+
+* You have installed the {oadp-short} Operator.
+* You have created the secret by using the credentials provided by the cloud provider.
+
+.Procedure
+
+* Configure the DPA with the environment variables for hashing, encryption, and splitter as shown in the following example.
++
+.Example DPA
+[source,yaml]
+----
+apiVersion: oadp.openshift.io/v1alpha1
+kind: DataProtectionApplication
+#...
+configuration:
+  nodeAgent:
+    enable: true # <1>
+    uploaderType: kopia # <2>
+  velero:
+    defaultPlugins:
+    - openshift
+    - aws
+    - csi #<3>
+    defaultSnapshotMoveData: true
+    podConfig:
+      env:
+        - name: KOPIA_HASHING_ALGORITHM
+          value: <hashing_algorithm_name> # <4>
+        - name: KOPIA_ENCRYPTION_ALGORITHM
+          value: <encryption_algorithm_name> # <5>
+        - name: KOPIA_SPLITTER_ALGORITHM
+          value: <splitter_algorithm_name> # <6>
+----
+<1> Enable the `nodeAgent`.
+<2> Specify the `uploaderType` as `kopia`.
+<3> Include the `csi` plugin.
+<4> Specify a hashing algorithm. For example, `BLAKE3-256`.
+<5> Specify an encryption algorithm. For example, `CHACHA20-POLY1305-HMAC-SHA256`.
+<6> Specify a splitter algorithm. For example, `DYNAMIC-8M-RABINKARP`.