New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New plugin architecture #885

Merged

pombredanne merged 122 commits into develop from 787-new-plugin-architecture

Feb 7, 2018

Member

pombredanne commented Jan 17, 2018

This is for #787 and a few more tickets.

The core processing is centered now on a Codebase and Resource abstraction that represents the scanned filesystem in memory Efficiently navigate up and down the results "tree" of files without loading scanned results into memory #717 Rework cli.py to use a Resource class instead of iterating paths #736
Everything is a plugin including the scans Scan plugins for plugin architecture #698 .
The CLI becomes therefore much simpler.
A plugin can provide any option it needs and this support Allow a plugin to provide more than one option #787 and Post-scan plugins should be able to take in parameters #748
We support multiple output for Support multiple outputs from a single scan #789 and scans are also plugins for Scan plugins for plugin architecture #698
The interface for output plugins is the same as other plugins Refactor output plugins interface #715

There are a few changes on what is reported:

scanning a single file or empty dir will always return a value even with strtip-root
the root directory has a proper entry with its file info
the counts are correct and do not need to load the whole set of scan results in memory, using instead the lighter weight Codebase and Resources tree
the scan cache has been folded in the Codebase and Resource code. This will enable a simpler implementation for Allow cache location configuration #357 and Allow cache location configuration #878

I want to close #856 in favor of this heavily updated plugin and CLI processing architecture.

@yashdsaraf and @haikoschol ping!

yashdsaraf and others added 30 commits

January 17, 2018 22:59


          Support multiple options for pre-scan plugins #787

065d0a3

Signed-off-by: Yash D. Saraf <yashdsaraf@gmail.com>


          Added MakeHuman Exception License

6095fb1

Signed-off-by: Saravanan G <sgoffl1603@gmail.com>


          formatted LICENSE file and removed rules

4f0475c

Signed-off-by: Saravanan G <sgoffl1603@gmail.com>


          Refactor post scan plugins so they can have command line options

5a304fd

Signed-off-by: Haiko Schol <ext-haiko.schol@here.com>


          Improve documentation of PostScanPlugin.process_results()

f41bca6

Signed-off-by: Haiko Schol <ext-haiko.schol@here.com>


          Rename get_click_options() to get_options()

403b1b2

Signed-off-by: Haiko Schol <ext-haiko.schol@here.com>


          Allow post-scan plugins to take options other than boolean flags

5f4c626

Signed-off-by: Haiko Schol <ext-haiko.schol@here.com>


          Use a common BasePlugin class for all plugins #787

612cf68

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Inline fileutils imports #787

b765432

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Use proper import in prep for Python 3 #787

7f9b74c

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Update help text test to match latest code #787

cff89ad

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Use class method for get_plugin_options #787

75908e7

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Use scans_cache_class not scan_cache_class #787

586d392

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Extract function to get_cache_dir #787

66ec6dc

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Ensure all tests pass #787

81780ea

 * more ScanOption to scancode.__init__.py
 * add active_scans to BasePlugin initializer
 * ensure plugins work consistently
 * try to not make ignores a special case

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Remove junk print statements

d9f8ec1

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Streamline core CLI and plugins processing

4f143b7

 * use structured namedtuple to track Scanners and CommandOptions
 * rework how default scan and selected options are handled using these
   tuples to streamline the corresponding code and the way selected
   options are reported in the scan results (at least for JSON)
 * add is_enabled method to Plugin
 * ensure the help of options is used for each option, and not a global
   help for all the options of a given plugin
 * add new plural option for each scan (such as --licenses) that maps
   exactly to the key used in the scan results. Use plural variables
   throughout to simplify the code
 * use kwargs for the main cli.scancode function for all all options and
   no longer use named arguments: args can come from any plugins and
   are not known until runtime.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Use resource_iter everywhere #787

7ef1cba

 * fileutils.py: remove file_iter and dir_iter as they were not really
   used/useful

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Do not use on-disk file log. Improve pre-scan plugins #787

a6446c9

 * refactor pre-scan plugins to use the same approach as all other
   plugins and not make ignoring a special case
 * cli.py and cache.py: remove the on-disk file log and use an in-memory
   list instead

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Add new extract_zip_raw test function

0c3625b

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Bump attrs and add typing library #787

9884c9f

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Make info a regular scan and other CLI loop reorg #787

f8eabb3

 * move and other file info code such as get_file_infos and Resource to
   new resource.py module. Make infos a proper, regular scan
 * Do use content checksum for scan cache, instead use plain path
   checksum. This enabled making info a proper scan, since content
   checksums are no longer needed for cache handling.
 * remove info cache and use only one cache file per file for its
   details
 * Reorganize scan steps to make them co-located and explicit: inventory
   pre-scan, scan and post-scan are not a set of clean loops
 * move most of the CLI output in the main scancode function
 * Add name class attribute to plugins to support UI reporting

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Remove diag arg from get_resources() #787

40835fd

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Add function to skip first or last iterable item #787

3be4577

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Change interruptible returned values #787

b5e627c

 * now return (error string or None, value or None)
 * this is cleaner than (success, value) where the value could be
   either the callble result or the error string 

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Ensure api function return a list #787

e9cd4e5

 * and not a generator, so they can serialize OK without wrapping

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Use latest attrs and typing #787

59bef89

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Add file base_name to info scans #787

683ca26

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Replace path_to_bytes/unicode by fsen/decode #787

 * This was a useless wrapper
 * Also improve imports locality and other minor refactorings

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>


          Improve base plugins design #787

4ba3b77

 * use generic get_plugins for all plugin types rather than
   a function with a unique name
 * use Resource objects throughout (except for format plugins)
 * enhance BasePlugin base class: the methods are now setup() and
   teardown() to be executed at init/exit time, process_resource() to
   process a single Resource and process_codebase() to process a whole
   Codebase

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Member Author

pombredanne commented Feb 7, 2018

ok, so I am merging now. We can take it from develop for any other quirks.
@haikoschol @yashdsaraf thanks you for starting this in the first place. It took a while but I feel we are in a better place now!

pombredanne merged commit d5c583a into develop

This was referenced Feb 7, 2018

Improved logging and error/exception processing and reporting #11

Closed

Transparent extraction of archives #14

Closed

Document all scancode toolkit json output keys #112

Closed

Simplify the scancode command line options #209

Closed

Provide header level details for a scan: Enhance scancode to include a log or history with useful statistics #211

Closed

Scan Benchmarks #378

Open

Allow cache location configuration #357

Closed

Closed

The root node is not included as part of the scan results #543

Closed

Clean up command line handling code #489

Closed

Dir: Copyright Summary irregular count #607

Closed

Scan stats are misleading: they count files + dirs as files and do not match find #598

Closed

Efficiently navigate up and down the results "tree" of files without loading scanned results into memory #717

Closed

Refactor output plugins interface #715

Closed

Rework the --strip-root and --full-root CLI options as plugins #714

Closed

Post scan plugins for plugin architecture #699

Closed

Scan plugins for plugin architecture #698

Closed

Add tests for the core of plugincode scan output writers #696

Closed

Allow a plugin to provide more than one option #787

Closed

Always collect full stack trace for scan errors #784

Closed

Post-scan plugins should be able to take in parameters #748

Closed

Rework cli.py to use a Resource class instead of iterating paths #736

Closed

Make info collection part of the default #728

Closed

Support multiple outputs from a single scan #789

Closed

sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request


          Upgrade ScanCode to the latest development version

762d5aa

This includes a massive refactoring with lots of performance
improvements. For details see

aboutcode-org/scancode-toolkit#885

sschuberth mentioned this pull request

Upgrade ScanCode to the latest development version oss-review-toolkit/ort#340

Merged

sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request


          Upgrade ScanCode to the latest development version

4519ccf

This includes a massive refactoring with lots of performance
improvements. For details see

aboutcode-org/scancode-toolkit#885

pombredanne deleted the 787-new-plugin-architecture branch

February 19, 2018 21:45

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet