Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-16338. Correct fsimage error configuration message #3684

Merged
merged 19 commits into from
Dec 6, 2021
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -587,7 +587,7 @@ void doImportCheckpoint(FSNamesystem target) throws IOException {

if (checkpointEditsDirs == null || checkpointEditsDirs.isEmpty()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw checkpointDirs and checkpointEditsDirs will not be null as they are assigned values returned by FSImage.getCheckpointDirs and FSImage.getCheckpointEditsDirs respectively.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, my IntelliJ was giving that warning as well, but that was something not touched here....
I thought Todd added it just for future safety, Didn't research why there was a null check considering it harmless.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it seems fine, just an additional null check, not a big deal. Also good to keep it for future safety as you predicted correctly, just in case if the methods themselves start returning null (less likely but still doable).

throw new IOException("Cannot import image from a checkpoint. "
+ "\"dfs.namenode.checkpoint.dir\" is not set." );
+ "\"dfs.namenode.checkpoint.edits.dir\" is not set." );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there are some checktyles that need fixing. Maybe you can configure checkstyle.xml in your IDE. See ./hadoop-github/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml. The other changes look good to me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great! Thanks @tomscut for your advice,

}

FSImage realImage = target.getFSImage();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@
import org.junit.Test;

import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.fail;

public class TestFSImage {

Expand Down Expand Up @@ -275,6 +276,29 @@ public void testSaveAndLoadStripedINodeFile() throws IOException{
}
}

@Test
public void testImportCheckpoint() throws IOException{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little advice. You can consider using try with resources for MiniDFSCluster cluster = new MiniDFSCluster.Builder(config).build().

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tomscut for your advice , it seems simpler code.

Configuration conf = new Configuration();
conf.set(DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_EDITS_DIR_KEY,"");
MiniDFSCluster cluster = null;
try {
GuoPhilipse marked this conversation as resolved.
Show resolved Hide resolved
cluster = new MiniDFSCluster.Builder(conf).build();
cluster.waitActive();
FSNamesystem fsn = cluster.getNamesystem();
FSImage fsImage= new FSImage(conf);
fsImage.doImportCheckpoint(fsn);
fail("Expect to throw IOException.");
} catch (IOException e) {
GenericTestUtils.assertExceptionContains(
"Cannot import image from a checkpoint. "
+ "\"dfs.namenode.checkpoint.edits.dir\" is not set.", e);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use LambdaTestUtils instead of try-catch-assert

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ayushtkn for your advice, just updated.

} finally {
if (cluster != null) {
cluster.shutdown();
}
}
}

/**
* Test if a INodeFileUnderConstruction with BlockInfoStriped can be
* saved and loaded by FSImageSerialization
Expand Down