Upgrading scancode to v32.4.1 and adding dev notes

[jamesiri]

this updates the version of scancode in use in production because the v32.1.0 version will incorrectly identify packages with and LGPL license when it isn't present.

v32.4.1 functions as expected and passes all existing tests.

also added some notes about the version of node you have to use to get npm install and the tests to function correctly when running locally as well as guidance for people to use the docker env for dev.

[jamesiri]

this shoudl fix #636, #618, #583, and #502

[JustinGOSSES]

Looks good

[jeffwilcox]

thanks - let's def. get sign-off from @elrayle too

[elrayle]

@qtomlinson @jeffmendoza Did you see this PR?

[elrayle]

Did you try running in the context of https://github.com/clearlydefined/docker_dev_env_experiment. That is the easiest way to run any of the three major apps, i.e. service, website, and crawler, on localhost.

[jamesiri]

yep yep, I did and it was the only way I could get the whole system working for me which is why I added the note.

[qtomlinson]

To simulate Azure queue/blob store, see Azurite setup at https://github.com/clearlydefined/docker_dev_env_experiment#additional-setup-to-simulate-deployment-optional.

[qtomlinson]

Is it possible for mark Alternative Setup as Legacy?

[qtomlinson]

Thanks for the awesome contribution! Glad to see that it addresses the existing issues! Is it possible to run a suite of sanity tests on your local crawler to verify results for various supported components? This will help us to detect any changes in harvest results.

Steps for Sanity/Integration Test to Verify Harvest Results:

1. Clone the operations repository:

   git clone https://github.com/clearlydefined/operations/
   

2. Navigate to the tools/integration directory:

   cd operations/tools/integration
   

3. Set the process.env.DEV_API_BASE_URL environment variable to point to the local test API service.

4. Run the end-to-end harvest test:

   npm run e2e-test-harvest
   

   This command will trigger the harvest of a set of components (as defined in componentsStatic in testConfig.js) using the service pointed by DEV_API_BASE_URL. Wait until the harvest process is complete.

5. Run the end-to-end definition test:

   npm run e2e-test-definition
   

   This command compares the definitions computed on the test API service with those on the production deployment.