sync #13
Merged
sync #13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes typo in hadoop-aws/assumed_roles.md Contributed by Joey Krabacher
#3930) Adds a new map type WeakReferenceMap, which stores weak references to values, and a WeakReferenceThreadMap subclass to more closely resemble a thread local type, as it is a map of threadId to value. Construct it with a factory method and optional callback for notification on loss and regeneration. WeakReferenceThreadMap<WrappingAuditSpan> activeSpan = new WeakReferenceThreadMap<>( (k) -> getUnbondedSpan(), this::noteSpanReferenceLost); This is used in ActiveAuditManagerS3A for span tracking. Relates to * HADOOP-17511. Add an Audit plugin point for S3A * HADOOP-18094. Disable S3A auditing by default. Contributed by Steve Loughran.
…er than deleting. (#3940). Contributed by Ayush Saxena. Reviewed-by: Steve Loughran <stevel@apache.org> Reviewed-by: stack <stack@apache.org>
…able (#3967) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Create unix domain socket in java.io.tmpdir instead of test.build.dir to avoid 'File name too long' error. Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
…ternal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953)
…ributed by JiangHua Zhu.
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
…buted by Tamas Domok
…ervicesCapacitySched. Contributed by Tamas Domok
…onfig. Contributed by Tamas Domok
…buted by Tamas Domok
…pacityOverTimePolicy. Contributed by Benjamin Teke
… Contributed by Benjamin Teke
…. Contributed by Andras Gyori
…d by Andras Gyori
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Contributed by Steve Loughran Change-Id: I73af19d2e2e41f4ba686c470726a80c3903a1950
…pancies for RC preparation (#3991) Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
…iner path on yarn applications (#3908) Co-authored-by: Monthon Klongklaew <monthonk@amazon.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
* Centos 8 has reached its End-of-Life and thus its packages are no longer accessible from mirror.centos.org. * This PR switches the baseurl to vault.centos.org where the packages are archived.
…of class name. Contributed by Szilard Nemeth
Contributed by PJ Fanning
…#4000) Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
…apshottable error (#4051)
…4027) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
…d by Andras Gyori
…. Contributed by Andras Gyori
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
…ributed by Szilard Nemeth
…ibuted by Tamas Domok
…sation logical order. Contributed by Andras Gyori
Multi object delete of size more than 1000 is not supported by S3 and fails with MalformedXML error. So implementing paging of requests to reduce the number of keys in a single request. Page size can be configured using "fs.s3a.bulk.delete.page.size" Contributed By: Mukund Thakur
…3941). Contributed by limingxiang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: litao <tomleescut@gmail.com>
Fixes #4054 Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
…ViewFS. To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true. If the trash root for path p is in the same mount point as path p, and one of: * The mount point isn't at the top of the target fs. * The resolved path of path is root (eg it is the fallback FS). * The trash root isn't in user's target fs home directory. get the corresponding viewFS path for the trash root and return it. Otherwise, use <mnt>/.Trash/<user>. Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
…cks(). (#4048). Contributed by JiangHua Zhu.
…ocess is running (#4058)
…tter * New statistic names in StoreStatisticNames (for joint use with s3a committers) * Improvements to IOStatistics implementation classes * RateLimiting wrapper to guava RateLimiter * S3A committer Tasks moved over as TaskPool and added support for RemoteIterator * JsonSerialization.load() to fail fast if source does not exist + tests. This commit is a prerequisite for the main MAPREDUCE-7341 Manifest Committer patch. Contributed by Steve Loughran Change-Id: Ia92e2ab5083ac3d8d3d713a4d9cb3e9e0278f654
This is a mapreduce/spark output committer optimized for performance and correctness on Azure ADLS Gen 2 storage (via the abfs connector) and Google Cloud Storage (via the external gcs connector library). * It is safe to use with HDFS, however it has not been optimized for that use. * It is *not* safe for use with S3, and will fail if an attempt is made to do so. Contributed by Steve Loughran Change-Id: I6f3502e79c578b9fd1a8c1485f826784b5421fca
…EDUCE-7341 Follow-on patch to MAPREDUCE-7341, adding ABFS support and tests * resilient rename * tests for job commit through the manifest committer. contains - HADOOP-17976. ABFS etag extraction inconsistent between LIST and HEAD calls - HADOOP-16204. ABFS tests to include terasort Contributed by Steve Loughran. Change-Id: I0a7d4043bdf19bcb00c033fc389730109b93b77f
…print of ViewFileSystem Fixes #3996
* The source files for hdfs_find uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. * Thus, we need to replace getopt with boost::program_options to make hdfs_find cross platform.
…4068). Contributed by Max Xie. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
…FS (#4067). Contributed by tomscut. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
sync