Prevent buildtest from hanging due to directory traversal

[shahzebsiddiqui]

At the moment, buildtest has support for directory traversal when searching for buildspecs that can lead to serious issues if one specifies a directory path with lots of files. This can lead to a full directory traversal of filesystem which can impact some of the features in buildtest that may include the following options

• buildtest build -b
• buildtest buildspec validate -b
• buildtest buildspec find -d

buildtest uses the walk_tree method using the file_traversal_limit to stop buildtest from searching for files beyond the limit

buildtest/buildtest/utils/file.py

Lines 228 to 229 in 26ed347

	if files_traversed >= file_traverse_limit:
	return [os.path.abspath(fname) for fname in files_list]

.

The default value is set to 999999 however this may need to be configurable such that user can alter behavior for directory traversal.

We have a few options we can try to customize this value

1. Add support in configuration file and/or command line that can be overridden. Perhaps a key file_traversal_limit in buildspec cache and option --file-traversal-limit.

In configuration file it would be under the system definition it could be something like this

system:
  generic:
      file_traversal_limit: 5000

The objective is to prevent user input to lead into long directory traversal where one inputs the following. The following walk_tree methods should be updated with value of file_traversla

1. buildtest build -b /tmp

buildtest/buildtest/cli/build.py

Line 550 in 0804a13

buildspecs = walk_tree(buildspec, ".yml")

2. buildtest buildspec validate -b /tmp and buildtest bc find -d /

buildtest/buildtest/cli/buildspec.py

Line 239 in 0804a13

buildspec = walk_tree(path, ".yml")

For the option we should have it as a global option.

We should not update the walk_tree method with file_traversal_limit in the following invocation

1. This invocation is for buildtest schema

buildtest/buildtest/cli/schema.py

Line 48 in 0804a13

schema_examples = walk_tree(examples, ".yml")

4. This invocation is for buildtest history
   

   buildtest/buildtest/cli/history.py

   Line 113 in 0804a13

   history_files = walk_tree(BUILD_HISTORY_DIR, ".json")

TODO

• Update the main buildtest configuration file https://github.com/buildtesters/buildtest/blob/devel/buildtest/settings/config.yml and comment it out as default

*Use these in the configuration

# This configuration is used to limit the number of files that can be traversed when searching for buildspecs
# file_traversal_limit: 1000

• Update json schema https://github.com/buildtesters/buildtest/blob/devel/buildtest/schemas/settings.schema.json

• Update all example configuration files to be consistent everywhere https://github.com/buildtesters/buildtest/tree/devel/buildtest/settings, https://github.com/buildtesters/buildtest/tree/devel/tests/cli/configuration, https://github.com/buildtesters/buildtest/tree/devel/tests/settings, https://github.com/buildtesters/buildtest/tree/devel/tests/cli/cdash_examples https://github.com/buildtesters/buildtest/tree/devel/buildtest/schemas/examples/settings.schema.json/valid

• Add tests for buildtest build in https://github.com/buildtesters/buildtest/blob/devel/tests/cli/test_build.py

• Add tests for buildtest buildspec find in https://github.com/buildtesters/buildtest/blob/devel/tests/cli/test_buildspec.py

• Add implementation for by passing the value this should work for buildtest build, buildtest buildspec find and buildtest buildspec validate , see issue above

*For testing we should try testing the following commands as examples and use file_traversal_limit: 1. You would need to create an example configuration file for testing purposes, put it in https://github.com/buildtesters/buildtest/tree/devel/tests/cli/configuration and use it when testing this feature *

1. buildtest build -b tutorials/
2. buildtest buildspec find -s general_tests
3. buildtest buildspec validate -b tutorials
4. Should try edge cases such as 0 or negative value or a floating point value