[Remote Translog] Add support for downloading files from remote translog #5649

sachinpkale · 2022-12-28T10:46:37Z

Description

Add support to download .tlg and .ckp files from remote translog.
Also integrated the download flow with restore API.

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

sachinpkale · 2022-12-28T10:48:42Z

This Draft PR is built on #5638. Once the parent PR is merged, I will rebase the branch and will change the PR state to Ready for review.

github-actions · 2022-12-28T11:08:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8537/
CommitID: 4e136fe
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-12-28T11:26:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8538/
CommitID: 422f993
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-12-28T15:42:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8549/
CommitID: 80da036
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

ashking94 · 2022-12-28T17:27:41Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

@@ -3082,7 +3082,14 @@ public void startRecovery(
                executeRecovery("from store", recoveryState, recoveryListener, this::recoverFromStore);
                break;
            case REMOTE_STORE:
-                executeRecovery("from remote store", recoveryState, recoveryListener, this::restoreFromRemoteStore);
+                final Repository remoteTranslogRepo;


we could set remoteTranslogRepo as null here and then get rid of else block?

is it possible to push down some of this logic to restoreFromRemoteStore method?

remoteTranslogRepo needs to be final as it is provided as argument to a lambda, so we can't initialize it twice.

restoreFromRemoteStore does not have reference to repositoriesService to fetch the repository.

Can we please use Optional instead?

ashking94 · 2022-12-28T18:13:13Z

server/src/main/java/org/opensearch/index/shard/StoreRecovery.java

+            if (repository != null) {
+                FileTransferTracker fileTransferTracker = new FileTransferTracker(shardId);
+                assert repository instanceof BlobStoreRepository : "repository should be instance of BlobStoreRepository";
+                BlobStoreRepository blobStoreRepository = (BlobStoreRepository) repository;
+                TranslogTransferManager translogTransferManager = new TranslogTransferManager(
+                    new BlobStoreTransferService(
+                        blobStoreRepository.blobStore(),
+                        indexShard.getThreadPool().executor(ThreadPool.Names.TRANSLOG_TRANSFER)
+                    ),
+                    blobStoreRepository.basePath().add(shardId.getIndex().getUUID()).add(String.valueOf(shardId.id())),
+                    fileTransferTracker,
+                    fileTransferTracker::exclusionFilter
+                );
+                RemoteFsTranslog.download(translogTransferManager, indexShard.shardPath().resolveTranslog());
+            }
+


shall we move this to a method syncTranslogFromRemoteTranslogStore for ease of read?

ashking94 · 2022-12-28T18:16:57Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

+                Files.delete(file);
+            }
+            Map<String, String> generationToPrimaryTermMapper = translogMetadata.getGenerationToPrimaryTermMapper();
+            for (long i = translogMetadata.getGeneration(); i >= translogMetadata.getMinTranslogGeneration(); i--) {


if segments upload is lagging, then just downloading the most recent translog might not be enough. Can we create a tracking issue? If it exists already, pls do share.

Already tracking it here: #3754

More details are present in #5567 .

ashking94 · 2022-12-28T18:19:54Z

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

+    public boolean downloadTranslog(String primaryTerm, String generation, Path location, boolean latest) throws IOException {
+        logger.info("Downloading translog files with: Primry Term = {}, Generation = {}, Location = {}", primaryTerm, generation, location);
+        String checkpointFilename = "translog-" + generation + ".ckp";
+        if (latest) {
+            checkpointFilename = "translog.ckp";
+        }
+        if (Files.exists(location.resolve(checkpointFilename)) == false) {
+            try (
+                InputStream checkpointFileInputStream = transferService.downloadBlob(
+                    remoteBaseTransferPath.add(primaryTerm),
+                    "translog-" + generation + ".ckp"
+                )
+            ) {
+                Files.copy(checkpointFileInputStream, location.resolve(checkpointFilename));
+            }
+        }
+        String translogFilename = "translog-" + generation + ".tlog";
+        if (Files.exists(location.resolve(translogFilename)) == false) {
+            try (
+                InputStream translogFileInputStream = transferService.downloadBlob(
+                    remoteBaseTransferPath.add(primaryTerm),
+                    "translog-" + generation + ".tlog"
+                )
+            ) {
+                Files.copy(translogFileInputStream, location.resolve(translogFilename));
+            }
+        }
+        return true;
+    }


can we do something like this -

public boolean downloadTranslog(String primaryTerm, String generation, Path location, boolean latest) throws IOException { logger.info("Downloading translog files with: Primry Term = {}, Generation = {}, Location = {}", primaryTerm, generation, location); String ckpFileName = "translog-" + generation + ".ckp"; if (latest) { String ckpWithoutGenerationFileName = "translog.ckp"; downloadToFS(ckpFileName, ckpWithoutGenerationFileName, location, primaryTerm); } // Download Checkpoint file from remote and store on FS downloadToFS(ckpFileName, location, primaryTerm); // Download translog file from remote and store on FS String translogFilename = "translog-" + generation + ".tlog"; downloadToFS(translogFilename, location, primaryTerm); return true; } private void downloadToFS(String fileName, Path location, String primaryTerm) throws IOException { downloadToFS(fileName, fileName, location, primaryTerm); } private void downloadToFS(String remoteFileName, String localFileName, Path location, String primaryTerm) throws IOException { if (Files.exists(location.resolve(localFileName)) == false) { try (InputStream inputStream = transferService.downloadBlob(remoteBaseTransferPath.add(primaryTerm), remoteFileName)) { Files.copy(inputStream, location.resolve(localFileName)); } } }

ashking94 · 2022-12-28T18:25:45Z

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferMetadata.java

+    private static class MetadataFilenameComparator implements Comparator<String> {
+        @Override
+        public int compare(String metadaFilename1, String metadaFilename2) {
+            // Format of metadata filename is <Primary Term>__<Generation>__<Timestamp>
+            String[] filenameTokens1 = metadaFilename1.split(METADATA_SEPARATOR);
+            String[] filenameTokens2 = metadaFilename2.split(METADATA_SEPARATOR);
+            for (int i = 0; i < filenameTokens1.length; i++) {
+                if (filenameTokens1[i].equals(filenameTokens2[i]) == false) {
+                    return (int) (Long.parseLong(filenameTokens1[i]) - Long.parseLong(filenameTokens2[i]));
+                }
+            }
+            return 0;
+        }


can we add a WARN log here - we should not come across such situation where 2 metadata file comparison yield 0. Also, could we add log when we have to resolve to timestamp for comparing? I dont think using timestamp is fair as clocks are not synchronised across the nodes in the distributed system setup.

Totally agree with @ashking94's concerns here. However, putting WARN statements in a comparator could lead to craziness in the log. We don't really have control over how many times and against which pairs of files the sort algorithm will invoke the comparison. Is there somewhere else we can add these statements?

I was also not inclined to add the log in the compare method as it may not provide good debug insights. But as I think more, both the cases: comparing timestamp as well as returning 0 should happen in exceptional cases (we think it should not happen at all). Also, it is not feasible to add this logic in upload flow as it means reading the last uploaded file each time. IMO, we should add these logs in this method. Thoughts?

2 metadata file comparison yield 0. : This will never happen , as there can't be two files with same name in a remote directory .

Timestamp comparison is a tricky thing here - We can check this in upload flow as it has FileTransferTracker . However , we don't add metadata files in that . Alternate is we just throw RuntimeException from readMetadata() ? This is better than logs (we can miss it ) and silent failures

The expectation is that two files with the same primary term and generation should never exist, right? If so, then can we just not include the timestamp in the filename? Then it would fail at upload time, when the file was generated that violated the expected invariant, which seems better than failing hard here (because I think the system would end up stuck because it would keep hitting this same error until something changed).

@sachinpkale in case of a primary-primary relocation, the _ can be same or will be different for the xlog upload? If yes, then failing upload might as well be not correct.

The expectation is that two files with the same primary term and generation should never exist, right?

This should be extremely rare but should still technically possible for isolated writers during primary-primary relocation like what @sachinpkale mentioned. The timestamp serves as a discriminator in all those cases, based on a last writer wins policy.

ashking94

Looks fine on high level, pls make the necessary changes.

gbbafna · 2022-12-29T08:48:25Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

+                Files.delete(file);
+            }
+            Map<String, String> generationToPrimaryTermMapper = translogMetadata.getGenerationToPrimaryTermMapper();
+            for (long i = translogMetadata.getGeneration(); i >= translogMetadata.getMinTranslogGeneration(); i--) {


More details are present in #5567 .

gbbafna · 2022-12-29T08:50:57Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

+            Map<String, String> generationToPrimaryTermMapper = translogMetadata.getGenerationToPrimaryTermMapper();
+            for (long i = translogMetadata.getGeneration(); i >= translogMetadata.getMinTranslogGeneration(); i--) {
+                String generation = Long.toString(i);
+                translogTransferManager.downloadTranslog(


We can download tlog files concurrently like uploads . Since this is not in critical write path and exercised only in failver, its okay to take that as a follow up. We can create a ToDo for now.

Created tracking issue: #5660

gbbafna · 2022-12-29T09:02:37Z

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

@@ -128,6 +132,51 @@ public boolean transferSnapshot(TransferSnapshot transferSnapshot, TranslogTrans
        }
    }

+    public boolean downloadTranslog(String primaryTerm, String generation, Path location, boolean latest) throws IOException {
+        logger.info("Downloading translog files with: Primry Term = {}, Generation = {}, Location = {}", primaryTerm, generation, location);
+        String checkpointFilename = "translog-" + generation + ".ckp";


replace translog- with TRANSLOG_FILE_PREFIX , .ckp with CHECKPOINT_SUFFIX, .tlog with TRANSLOG_FILE_SUFFIX .

We can also use Translog#getFilename , Translog#getCommitCheckpointFileName to generate tlog filename .

gbbafna · 2022-12-29T09:08:04Z

server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

+        if (latest) {
+            checkpointFilename = "translog.ckp";
+        }


We should not put the latest logic here, but handle it inRemoteFsTranslog .

gbbafna · 2022-12-29T10:10:16Z

server/src/main/java/org/opensearch/index/shard/StoreRecovery.java

+                FileTransferTracker fileTransferTracker = new FileTransferTracker(shardId);
+                assert repository instanceof BlobStoreRepository : "repository should be instance of BlobStoreRepository";
+                BlobStoreRepository blobStoreRepository = (BlobStoreRepository) repository;
+                TranslogTransferManager translogTransferManager = new TranslogTransferManager(
+                    new BlobStoreTransferService(
+                        blobStoreRepository.blobStore(),
+                        indexShard.getThreadPool().executor(ThreadPool.Names.TRANSLOG_TRANSFER)
+                    ),
+                    blobStoreRepository.basePath().add(shardId.getIndex().getUUID()).add(String.valueOf(shardId.id())),
+                    fileTransferTracker,
+                    fileTransferTracker::exclusionFilter
+                );


should we move this to RemoteFSTranslog#download ? Creating translogTransferManager, fileTransferTracker etc is responsibility of RemoteFSTranslog only .

We can then try to move the common parts of the ctor and download into separate functions.

The problem is RemoteFsTranslog#download is static method. Even if we create the instances of tracker in download method, we will not be able to re-use it in RemoteFsTranslog.

I agree on that , but at least the logic of creation of fileTransferTracker, translogTransferManager will reside in RemoteFSTranslog only . We will not be reuse the instances, but reuse the code .

In this case, do we assume that FileTransferTracker and TranslogTransferManager should not be initialized outside RemoteFsTranslog? Currently, they are public classes so can be instantiated outside.

sachinpkale · 2023-01-03T09:55:16Z

When we perform a translog metadata download, we perform a remote store LIST, we need to ensure that we are paginating if needed(we should avoid any pagination by ensuring the LIST starts with the most recent blob) but not paginating deep to ensure performance. I guess this needs additional handling if not already addressed elsewhere

As I understand, pagination is handled in the repository specific implementation (I have verified this in repository-S3).
The concern regarding too many entries and impact on performance is valid. This can be handled by very aggressive purging of translog metadata files (a background thread runs every X secs and clean up all the entries but last Y)

gbbafna · 2023-01-03T10:06:05Z

When we perform a translog metadata download, we perform a remote store LIST, we need to ensure that we are paginating if needed(we should avoid any pagination by ensuring the LIST starts with the most recent blob) but not paginating deep to ensure performance. I guess this needs additional handling if not already addressed elsewhere

As I understand, pagination is handled in the repository specific implementation (I have verified this in repository-S3). The concern regarding too many entries and impact on performance is valid. This can be handled by very aggressive purging of translog metadata files (a background thread runs every X secs and clean up all the entries but last Y)

With #5662 and its associated ToDo in #5677 , we will be cleaning up the metadata files with every flush . So it should get taken care of automatically.

sachinpkale · 2023-01-03T10:26:05Z

With #5662 and its associated ToDo in #5677 , we will be cleaning up the metadata files with every flush . So it should get taken care of automatically.

This would definitely help, but still we won't be able to have any control on the number of files created since the last commit.

Signed-off-by: Sachin Kale <kalsac@amazon.com>

github-actions · 2023-01-04T09:50:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8790/
CommitID: d2b2611

Bukhtawar · 2023-01-04T12:18:34Z

When we perform a translog metadata download, we perform a remote store LIST, we need to ensure that we are paginating if needed(we should avoid any pagination by ensuring the LIST starts with the most recent blob) but not paginating deep to ensure performance. I guess this needs additional handling if not already addressed elsewhere

As I understand, pagination is handled in the repository specific implementation (I have verified this in repository-S3). The concern regarding too many entries and impact on performance is valid. This can be handled by very aggressive purging of translog metadata files (a background thread runs every X secs and clean up all the entries but last Y)

With #5662 and its associated ToDo in #5677 , we will be cleaning up the metadata files with every flush . So it should get taken care of automatically.

This approach might not be deterministic and download latencies unpredictable depending on what rate we ingest and how many pages we end up in. We need a mechanism to always have the most latest entries in the first page to guarantee high predictability.

sachinpkale · 2023-01-04T13:03:40Z

This approach might not be deterministic and download latencies unpredictable depending on what rate we ingest and how many pages we end up in. We need a mechanism to always have the most latest entries in the first page to guarantee high predictability.

Created a tracking issue: #5696

…log (opensearch-project#5649) * Add support to download translog from remote store during recovery Signed-off-by: Sachin Kale <kalsac@amazon.com>

…hanges (#5757) * Introduce TranslogFactory for Local/Remote Translog support (#4172) * Introduce TranslogFactory for Local/Remote Translog support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * [Remote Translog] Introduce remote translog with upload functionality (#5392) * Introduce remote translog with upload functionality Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com> * Enable creation of indices using Remote Translog (#5638) * Enable creation of indices using Remote Translog behind a setting and feature flag Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> * [Remote Translog] Add support for downloading files from remote translog (#5649) * Add support to download translog from remote store during recovery Signed-off-by: Sachin Kale <kalsac@amazon.com> * Integrate remote translog download on failover (#5699) * Integrate remote translog download on failover Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>

…hanges (opensearch-project#5757) * Introduce TranslogFactory for Local/Remote Translog support (opensearch-project#4172) * Introduce TranslogFactory for Local/Remote Translog support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * [Remote Translog] Introduce remote translog with upload functionality (opensearch-project#5392) * Introduce remote translog with upload functionality Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com> * Enable creation of indices using Remote Translog (opensearch-project#5638) * Enable creation of indices using Remote Translog behind a setting and feature flag Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> * [Remote Translog] Add support for downloading files from remote translog (opensearch-project#5649) * Add support to download translog from remote store during recovery Signed-off-by: Sachin Kale <kalsac@amazon.com> * Integrate remote translog download on failover (opensearch-project#5699) * Integrate remote translog download on failover Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Introduce TranslogFactory for Local/Remote Translog support (#4172) * Introduce TranslogFactory for Local/Remote Translog support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * [Remote Translog] Introduce remote translog with upload functionality (#5392) * Introduce remote translog with upload functionality Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com> * Enable creation of indices using Remote Translog (#5638) * Enable creation of indices using Remote Translog behind a setting and feature flag Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> * [Remote Translog] Add support for downloading files from remote translog (#5649) * Add support to download translog from remote store during recovery Signed-off-by: Sachin Kale <kalsac@amazon.com> * Integrate remote translog download on failover (#5699) * Integrate remote translog download on failover Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Gaurav Bafna <85113518+gbbafna@users.noreply.github.com>

…hanges (#5757) * Introduce TranslogFactory for Local/Remote Translog support (#4172) * Introduce TranslogFactory for Local/Remote Translog support Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * [Remote Translog] Introduce remote translog with upload functionality (#5392) * Introduce remote translog with upload functionality Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com> * Enable creation of indices using Remote Translog (#5638) * Enable creation of indices using Remote Translog behind a setting and feature flag Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> * [Remote Translog] Add support for downloading files from remote translog (#5649) * Add support to download translog from remote store during recovery Signed-off-by: Sachin Kale <kalsac@amazon.com> * Integrate remote translog download on failover (#5699) * Integrate remote translog download on failover Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com>

sachinpkale force-pushed the remote-txlog-download-flow branch from 4e136fe to 422f993 Compare December 28, 2022 11:01

sachinpkale added the skip-changelog label Dec 28, 2022

sachinpkale force-pushed the remote-txlog-download-flow branch from 422f993 to 80da036 Compare December 28, 2022 15:13

ashking94 reviewed Dec 28, 2022

View reviewed changes

ashking94 requested changes Dec 28, 2022

View reviewed changes

gbbafna reviewed Dec 29, 2022

View reviewed changes

sachinpkale marked this pull request as ready for review December 29, 2022 11:12

sachinpkale requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159 and ryanbogan as code owners December 29, 2022 11:12

Sachin Kale added 12 commits January 4, 2023 14:10

Add support to download translog

3d8d0e8

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Handle absence of remote translog repo

c7681bb

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Address PR comments

075d500

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Address PR comments

7d4e454

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Address PR comments

5c4970b

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Address PR comments

385e859

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Fix spotless errors

4c633d4

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Address PR comments

f9316a8

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Address PR comments

1be30a6

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Address PR comments

7a9c32e

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Fix spotless errors

032b022

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Checksum entire metadata file

d2b2611

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale force-pushed the remote-txlog-download-flow branch from eaec42a to d2b2611 Compare January 4, 2023 09:16

sachinpkale requested a review from Bukhtawar January 4, 2023 09:52

sachinpkale mentioned this pull request Jan 4, 2023

[Remote Translog] Metadata file naming convention to get the latest metadata file in first fetch #5696

Closed

Bukhtawar approved these changes Jan 4, 2023

View reviewed changes

Bukhtawar merged commit 28e9b11 into opensearch-project:main Jan 4, 2023

gbbafna mentioned this pull request Jan 9, 2023

[Backport 2.x] Adding Remote translog upload, download and failover changes #5757

Merged

6 tasks

[Remote Translog] Add support for downloading files from remote translog #5649

[Remote Translog] Add support for downloading files from remote translog #5649

Conversation

sachinpkale commented Dec 28, 2022 • edited Loading

Description

Issues Resolved

Check List

sachinpkale commented Dec 28, 2022

github-actions bot commented Dec 28, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 28, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 28, 2022

Gradle Check (Jenkins) Run Completed with:

ashking94 Dec 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bukhtawar Jan 2, 2023 • edited Loading

Choose a reason for hiding this comment

ashking94 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sachinpkale commented Jan 3, 2023

gbbafna commented Jan 3, 2023

sachinpkale commented Jan 3, 2023

github-actions bot commented Jan 4, 2023

Gradle Check (Jenkins) Run Completed with:

Bukhtawar commented Jan 4, 2023

sachinpkale commented Jan 4, 2023

sachinpkale commented Dec 28, 2022 •

edited

Loading

ashking94 Dec 28, 2022 •

edited

Loading

Bukhtawar Jan 2, 2023 •

edited

Loading