HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters #2975

sumangala-patki · 2021-05-04T12:08:39Z

ABFS open methods require certain information (contentLength, eTag, etc) to create an InputStream for the file at the given path. This information is retrieved via a GetFileStatus request to backend.

However, client applications may often have access to the FileStatus prior to invoking the open API. Providing this FileStatus to the driver through the OpenFileParameters argument of openFileWithOptions() can help avoid the call to Store for FileStatus.

This PR adds handling for the FileStatus instance (if any) provided via the OpenFileParameters argument.

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

sumangala-patki · 2021-05-05T19:57:25Z

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

HNS OAuth

[INFO] Tests run: 98, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 98
[ERROR] Errors: 
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] Tests run: 261, Failures: 0, Errors: 1, Skipped: 50

HNS SharedKey

[INFO] Tests run: 98, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 54
[ERROR] Errors: 
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:635 » TestTimedOut
[ERROR] Tests run: 261, Failures: 0, Errors:1, Skipped: 40

Non-HNS SharedKey

[INFO] Tests run: 98, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 276
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40

Existing JIRA to track timeout errors: AbstractContractDistCp

sumangala-patki · 2021-05-07T08:45:42Z

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

HNS OAuth

[INFO] Tests run: 98, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 98
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 50

HNS SharedKey

[INFO] Tests run: 98, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 54
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40

Non-HNS SharedKey

[INFO] Tests run: 98, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 556, Failures: 0, Errors: 0, Skipped: 276
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40

Note about Yetus error: The asflicense warning is for a file from an older commit that has now been deleted.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

...s/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

anoopsjohn · 2021-07-22T16:44:24Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

@@ -204,38 +204,51 @@ public void registerListener(Listener listener1) {
  public FSDataInputStream open(final Path path, final int bufferSize) throws IOException {
    LOG.debug("AzureBlobFileSystem.open path: {} bufferSize: {}", path, bufferSize);
    // bufferSize is unused.
-    return open(path, Optional.empty());
+    return open(path, new OpenFileParameters());


Concern on using this way and passing OpenFileParameters param.
When FS#openFile() API is been used, you can see how system create the OpenFileParameters(). There may be some stuff coming in as default there.
eg: bufferSize = fileSystem.getConf().getInt(IO_FILE_BUFFER_SIZE_KEY,
IO_FILE_BUFFER_SIZE_DEFAULT);

So correct buffer size as per conf will be there in OpenFileParameters also
We might not be using buffer size from the param now. But later its possible to use some new stuff like this.
So better dont create OpenFileParameters () here and pass.
Instead we can have the private method accept Optional.
In open(Path), we can Optional.empty() anyways.

steveloughran

Hi,

only just seen this...feel free to let me know when you are doing big bits work, especially with unstable bits of the API that I've worked on.

FWIW, I'll be able to use this in the manifest committer #2971; as it well let me go straight from the listing of the directory of manifests to opening each manifest without the HEAD requests. I also have an internal PoC parquet lib which does the same -and showed up the bugs in s3a FS there.

I have some concerns with the patch as is, which I'd like addressed in a "stabilize abfs openFile.withStatus" JIRA, which, once is in, can go in to -3.3 back to back with this

In particular, the openFile code on L705 warns if the FileStatus is of the wrong type.
This turns out to cause problems in Hive or any other FS which wraps the inner FS, and
creates its own FileStatus objects. Why does hive do this? Because FileStatus.getPath()
returns the full path of the reference; hive has to replace this with a new FileStatus
with its mapped path.

printing WARN if the type is wrong is going to fill up Hive logs and
remove all speedup options

In #3289 I soften the requirement for the path to == the object; I'm only checking the final elements are equal. This allows parquet through Hive to use the API.

https://github.com/apache/hadoop/pull/3289/files#diff-4050f95b7e3912145415b6e2f9cd3b0760fcf2ce96bf0980c6c30a6edad2d0fbR5402

In #2584, which is my real "stabilize openfile patch" I do it properly

https://github.com/apache/hadoop/blob/700f99835f21571d9f29f2194327894776abe13f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/PrepareToOpenFile.java

only check name
reject directories
handle LocatedFileStatus
and any other FS subclass, only picking up the length value
and as this has got complex, factored out into its own class

Really, it's only the length which is important, other than that it is just a declaration of "this is a file, skip the HEAD"

So for ABFS to work, I'm going to require the equivalent of that PrepareToOpenFile: support for more FileStatus types. Doesn't need to be factored out, though that turned out to work well for testing.

This makes clear that I need to get #2584 in; it's been neglected.

That patch allows for a seek policy to be specified; all the whole file reads ask for sequential IO, so when hive clusters are deployed with the default seek policy set to random, things like distcp and hadoop fs put don't fail. It critically defines a standard set of seek policies, which Azure can implement too.
I've realized we need one more "whole-file", which says "no need for lazy seek, just do the GET".

Does anyone want to take that PR up? I keep getting distracted by other high-importance tasks.

Meanwhile the manifest committer can be the first user of this; I will verify that it delivers some speed up. It should: job commit with 1000 tasks will go from one list, 1000 HEAD, 1000 GET to 1 list and the 1000 GETs.

FWIW, this is the code in callthe magic committer patch which becomes safe to use with the filename fix there:
https://github.com/apache/hadoop/pull/3289/files#diff-3da795bf4618a7e504e21d3c53f9f7a42c52a75fe232906feb4e66f98794d41eR94

...I'll do the same with the manifest committer; if I do it this week its new tests will help stress the feature.

steveloughran · 2021-08-25T19:47:31Z

...s/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java

+    FutureDataInputStreamBuilder builder = fs.openFile(path);
+    builder.withFileStatus(fileStatus);
+    FSDataInputStream in = builder.build().get();
+    assertEquals(String.format(


use AssertJ

steveloughran · 2021-08-25T19:48:05Z

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

+        eTag = ((VersionedFileStatus) fileStatus).getVersion();
+      } else {
+        if (fileStatus != null) {
+          LOG.warn(


no, just use length & type, don't worry about etag. Or at least just log @ debug

will modify in new JIRA

steveloughran · 2021-08-25T19:49:02Z

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

+      long contentLength;
+      if (fileStatus instanceof VersionedFileStatus) {
+        path = path.makeQualified(this.uri, path);
+        Preconditions.checkArgument(fileStatus.getPath().equals(path),


only check getPath().getName() for equivalence.

Will make changes in HADOOP-17896
I believe getName() will verify the file name only. Would it allow paths with parent directories that don't match?

steveloughran · 2021-08-25T19:59:14Z

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java

+        contentLength = Long.parseLong(
+            op.getResponseHeader(HttpHeaderConfigurations.CONTENT_LENGTH));
+        eTag = op.getResponseHeader(HttpHeaderConfigurations.ETAG);
+      }

      if (parseIsDirectory(resourceType)) {
        throw new AbfsRestOperationException(


is there a test for this

yes, there is a contract test to verify that a FileNotFound exception is thrown on attempt to open a directory
org.apache.hadoop.fs.contract.AbstractContractOpenTest#testOpenReadDir

snvijaya · 2021-08-27T09:12:18Z

Hi,

only just seen this...feel free to let me know when you are doing big bits work, especially with unstable bits of the API that I've worked on.

FWIW, I'll be able to use this in the manifest committer #2971; as it well let me go straight from the listing of the directory of manifests to opening each manifest without the HEAD requests. I also have an internal PoC parquet lib which does the same -and showed up the bugs in s3a FS there.

I have some concerns with the patch as is, which I'd like addressed in a "stabilize abfs openFile.withStatus" JIRA, which, once is in, can go in to -3.3 back to back with this

In particular, the openFile code on L705 warns if the FileStatus is of the wrong type.
This turns out to cause problems in Hive or any other FS which wraps the inner FS, and
creates its own FileStatus objects. Why does hive do this? Because FileStatus.getPath()
returns the full path of the reference; hive has to replace this with a new FileStatus
with its mapped path.

printing WARN if the type is wrong is going to fill up Hive logs and remove all speedup options

In #3289 I soften the requirement for the path to == the object; I'm only checking the final elements are equal. This allows parquet through Hive to use the API.

https://github.com/apache/hadoop/pull/3289/files#diff-4050f95b7e3912145415b6e2f9cd3b0760fcf2ce96bf0980c6c30a6edad2d0fbR5402

In #2584, which is my real "stabilize openfile patch" I do it properly

https://github.com/apache/hadoop/blob/700f99835f21571d9f29f2194327894776abe13f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/PrepareToOpenFile.java

only check name

reject directories

handle LocatedFileStatus

and any other FS subclass, only picking up the length value

and as this has got complex, factored out into its own class

Really, it's only the length which is important, other than that it is just a declaration of "this is a file, skip the HEAD"

So for ABFS to work, I'm going to require the equivalent of that PrepareToOpenFile: support for more FileStatus types. Doesn't need to be factored out, though that turned out to work well for testing.

This makes clear that I need to get #2584 in; it's been neglected.

That patch allows for a seek policy to be specified; all the whole file reads ask for sequential IO, so when hive clusters are deployed with the default seek policy set to random, things like distcp and hadoop fs put don't fail. It critically defines a standard set of seek policies, which Azure can implement too.
I've realized we need one more "whole-file", which says "no need for lazy seek, just do the GET".

Does anyone want to take that PR up? I keep getting distracted by other high-importance tasks.

Meanwhile the manifest committer can be the first user of this; I will verify that it delivers some speed up. It should: job commit with 1000 tasks will go from one list, 1000 HEAD, 1000 GET to 1 list and the 1000 GETs.

FWIW, this is the code in callthe magic committer patch which becomes safe to use with the filename fix there:
https://github.com/apache/hadoop/pull/3289/files#diff-3da795bf4618a7e504e21d3c53f9f7a42c52a75fe232906feb4e66f98794d41eR94

...I'll do the same with the manifest committer; if I do it this week its new tests will help stress the feature.

Hi @steveloughran,

Thanks a lot for taking a look at this change. We will make updates to address the comments provided.

There is one comment on the eTag that I would like to check with you.

eTag is a parameter that is sent with every read request to server today. Server ensures that the file is indeed of the expected eTag before proceeding with the read operation. If the file was recreated during the timespan inputStream was created and read is issued, eTag check will fail and so will the read, else read will end up wrongly getting data from the new file. This is the reason we are mandating eTag to be present and in else case fall back to triggering a GetFileStatus. We will changing the log type to debug as per your comment in the else case.

I did have a look at #2584 which provides the means to pass mandatory and optional keys through OpenFileParameters. Noticed an example where S3 eTag is passed as an optional key (referring to link , for the Hive case where FileStatus is wrapped, is this how eTag gets passed ? And was also wondering how we can get various Hadoop workloads to send it without being aware of which is the current active FileSystem considering the key name is specific to it.

Also wanted to check if it would be right to add eTag as a field into the base FileStatus class. Probably as a sub structure within FileStatus so that it can be expanded in future to hold properties that Cloud storage systems commonly depend on.

OpenFile() change is one of the key performance improvement that we are looking at adopting to. The aspect of FileStatus being passed down in itself reduces HEAD request count by half and look forward to adopting to the new set of read policies too. We will work on understanding how to map various read policies to the current optimizations that the driver has for different read patterns. I think that would translate to a change equivalent to PrepareToOpenFile for ABFS driver. Would it be ok if we make this change once #2584 checks in ?
Currently we are in a bit of tight schedule and short staffed as we aim to complete on the feature work tracked in HADOOP-17853 and another customer requirement that we are in feasibility analysis stage.

steveloughran · 2021-08-27T15:57:31Z

eTag is a parameter that is sent with every read request to server today. Server ensures that the file is indeed of the expected eTag before proceeding with the read operation. If the file was recreated during the timespan inputStream was created and read is issued, eTag check will fail and so will the read, else read will end up wrongly getting data from the new file. This is the reason we are mandating eTag to be present and in else case fall back to triggering a GetFileStatus. We will changing the log type to debug as per your comment in the else case.

I do know about etags. The S3A connector picks it up on the HEAD or first GET and caches it, to at least ensure that things are unchanged for the duration of the stream.

Noticed an example where S3 eTag is passed as an optional key (referring to link , for the Hive case where FileStatus is wrapped, is this how eTag gets passed ? And was also wondering how we can get various Hadoop workloads to send it without being aware of which is the current active FileSystem considering the key name is specific to it.

not in use at all by hive, even in PoCs. I'm not worrying about changes between the list and the open, just about consistency once a file is opened. If a file was changed between hive query planning and work execution then as it is invariably of a different length the workers will notice soon enough. Anyway, HDFS doesn't worry about this so why should the other stores?

When you skip the HEAD, the first GET will return the etag. All you have to do is skip setting the If-Match header on the first request, pick the etag from the response and use it after.

I did have a look at #2584 which provides the means to pass mandatory and optional keys through OpenFileParameters.

This was already in the openFile() API call from the outset. Indeed, abfs already supports one, fs.azure.buffered.pread.disable.

What the new PR adds is

standard options for read policies, split start/end, all of which can be used to optimize reads.
tuning of the withFileStatus() based on initial use in format libraries, attempts to use in hive, etc.
tests for this stuff

No API change, other than withFileStatus(null) required to be a no-op, and requirements in the API spec relaxed to say "only check path.getName()" for a match.

Also wanted to check if it would be right to add eTag as a field into the base FileStatus class. Probably as a sub structure within FileStatus so that it can be expanded in future to hold properties that Cloud storage systems commonly depend o

way too late for that. used everywhere, gets serialized etc. What you could do is add some interface EtagSource with a getEtag() method, and implement that interface for the ABFS and s3a connectors. It's still going to be lost when things like hive replace the custom FileStatus with the standard one.

Where this could be useful is in distcp applications where the etag of an uploaded file could be cached and compared when rescanning a directory. IF a store implements the getFileChecksum() method you can serve the etag that way but (a) distcp can't handle it and (b) if you could get it on every file from a list() call, you save a lot of IO.

So please, have someone implement this in hadoop-common, with specification, tests etc.

OpenFile() change is one of the key performance improvement that we are looking at adopting to. The aspect of FileStatus being passed down in itself reduces HEAD request count by half and look forward to adopting to the new set of read policies too. We will work on understanding how to map various read policies to the current optimizations that the driver has for different read patterns. I think that would translate to a change equivalent to PrepareToOpenFile for ABFS driver.

I've just tried to let apps declare how they want to read things; there's generally a straightforward map to sequential vs random vs adaptive...but I hope that the orc and parquet options would provide even more opportunity to tweak behaviour, e.g .by knowing that there will be an initial footer read sequence, then stripes will be read.

Would it be ok if we make this change once #2584 checks in ?

yes, but I do want to at least have the azure impl not reimplement the same brittleness around withFileStatus() as s3a before it goes into 3.3. That way, consistency.

Currently we are in a bit of tight schedule and short staffed as we aim to complete on the feature work tracked in HADOOP-17853 and another customer requirement that we are in feasibility analysis stage.

I really want to get a hadoop 3.3.2 out the door with this API in it. It's going to happen before '17853 as that is a big piece of work, and which is going to need lots of review time by others, myself included.

… via OpenFileParameters (apache#2975)

… via OpenFileParameters (#2975) Change-Id: I039a0c3cb1c9b603f7dd1be0df03f795525d92bc

… via OpenFileParameters (apache#2975) commit: dcddc6a

…tion Rename resume optimizations: Changes include: 1. Resume the rename only if srcDirectory is present and it's etag matches to the one that was there at the time of rename. Else, just delete the pendingJson. 2. PendingJson will now have Etag information of the srcDir at the time of rename. 3. There is listStatus done after its found that renamePendingJson is there. But since there can be scenario where rename resume shouldnt be done (like srcDir not there, or original srcDir is changed(etag change)), the second listStatus shall be redundant. In case of not requirement, it will not be called. 4. For the resume, fileStatus is already there with listStatus or getFileStatus for renamePendingJson file. On ABFS_3,3.2_dev, for reading the file, we have to open which needs getFileStatus. This is a redundant call and needs optimization. This optimization is taken. For this backported HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters apache#2975 at commit: dcddc6a

sumangala17 added 2 commits May 4, 2021 17:33

use filestatus for open + prelim tests

a690a0c

fix import

6f12319

vinaysbadami suggested changes May 4, 2021

View reviewed changes

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java Outdated Show resolved Hide resolved

sumangala17 added 2 commits May 5, 2021 23:34

move tests, spy gfs count

b0834b0

checkstyle

e1b79e8

minor changes

018bfe2

vinaysbadami approved these changes May 6, 2021

View reviewed changes

sumangala17 added 5 commits May 6, 2021 18:16

one file > readBufferSize

2752efb

speed++

02bfa1e

Merge branch 'trunk' into HADOOP-17682

a212587

minor change

75c5961

import

31ac9a2

merge with tc

94f6884

sumangala-patki marked this pull request as ready for review July 5, 2021 16:35

anoopsjohn reviewed Jul 10, 2021

View reviewed changes

sumangala17 added 5 commits July 13, 2021 11:28

revw

de9e6ea

call client fn + test spy

dd77751

checkstyle

2e23baf

Merge branch 'trunk' into HADOOP-17682

a2c390f

yetus + chkstyle

ed8e034

anoopsjohn reviewed Jul 22, 2021

View reviewed changes

sumangala17 added 7 commits July 23, 2021 12:50

optional

a883995

opt conf

469671d

support no etag + test

1c50b7b

fallback to getfilestatus

e58dcdc

log warn

79e8f8e

fix test

7e70d42

warn freq

7ccc6b0

apache deleted a comment from hadoop-yetus Aug 25, 2021

steveloughran reviewed Aug 25, 2021

View reviewed changes

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions()…

2bc7d0b

… via OpenFileParameters (apache#2975)

steveloughran mentioned this pull request Apr 6, 2022

HADOOP-16202. Enhance openFile() for better read performance against object stores #2584

Closed

steveloughran pushed a commit to steveloughran/hadoop that referenced this pull request Apr 27, 2022

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions()…

97735de

… via OpenFileParameters (apache#2975)

asfgit pushed a commit that referenced this pull request Apr 27, 2022

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions()…

77eea7a

… via OpenFileParameters (#2975) Change-Id: I039a0c3cb1c9b603f7dd1be0df03f795525d92bc

saxenapranav pushed a commit to saxenapranav/hadoop that referenced this pull request Jul 26, 2023

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions()…

a660122

… via OpenFileParameters (apache#2975) commit: dcddc6a

saxenapranav mentioned this pull request Jul 27, 2023

Rename Resume optimization: resume only if srcDir etag matches to the one with which rename was started. ABFSDriver/AbfsHadoop#85

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters #2975

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters #2975

sumangala-patki commented May 4, 2021 •

edited

Loading

Uh oh!

Uh oh!

sumangala-patki commented May 5, 2021

Uh oh!

sumangala-patki commented May 7, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoopsjohn Jul 22, 2021

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran Aug 25, 2021

Uh oh!

steveloughran Aug 25, 2021

Uh oh!

sumangala-patki Sep 8, 2021

Uh oh!

steveloughran Aug 25, 2021

Uh oh!

sumangala-patki Sep 8, 2021

Uh oh!

steveloughran Aug 25, 2021

Uh oh!

sumangala-patki Sep 8, 2021 •

edited

Loading

Uh oh!

snvijaya commented Aug 27, 2021

Uh oh!

steveloughran commented Aug 27, 2021

Uh oh!

Uh oh!

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters #2975

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters #2975

Conversation

sumangala-patki commented May 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sumangala-patki commented May 5, 2021

Uh oh!

sumangala-patki commented May 7, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anoopsjohn Jul 22, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Aug 25, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran Aug 25, 2021

Choose a reason for hiding this comment

Uh oh!

sumangala-patki Sep 8, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran Aug 25, 2021

Choose a reason for hiding this comment

Uh oh!

sumangala-patki Sep 8, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran Aug 25, 2021

Choose a reason for hiding this comment

Uh oh!

sumangala-patki Sep 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snvijaya commented Aug 27, 2021

Uh oh!

steveloughran commented Aug 27, 2021

Uh oh!

Uh oh!

sumangala-patki commented May 4, 2021 •

edited

Loading

sumangala-patki Sep 8, 2021 •

edited

Loading