Skip to content

Commit 42737c6

Browse files
TG1999TG1999
committed
Address review comments
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
1 parent 8843ab3 commit 42737c6

File tree

2 files changed

+95
-89
lines changed

2 files changed

+95
-89
lines changed

‎etc/scripts/d2d/README.rst‎

Lines changed: 87 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -1,86 +1,103 @@
1-
==============================================================================
2-
Run ScanCode.io Pipelines in Docker (D2D Runner)
3-
==============================================================================
1+
Run ScanCode.io Mapping Script
2+
=============================
43

5-
This script helps execute **ScanCode.io** pipelines in isolated Docker containers,
6-
using a local PostgreSQL database and a working directory named ``./d2d``.
4+
This script executes the ``map_deploy_to_develop`` mapping workflow from
5+
ScanCode.io inside a Docker container. It optionally spins up a temporary
6+
PostgreSQL instance when needed. The script copies the specified input files to
7+
a working directory, runs the mapping, writes the output to a file, and cleans
8+
up afterward.
79

8-
-------------------------------------------------------------------------------
9-
Prerequisites
10-
-------------------------------------------------------------------------------
10+
.. warning::
11+
This script uses ``--network host`` and runs containers locally. Review
12+
security and network implications before using in production systems.
1113

12-
1. **Python 3.8+** must be installed
13-
2. **Docker** must be installed and accessible via ``sudo`` or user group
14+
Usage
15+
-----
1416

15-
-------------------------------------------------------------------------------
16-
Environment Variables
17-
-------------------------------------------------------------------------------
17+
.. code-block:: bash
1818
19-
.. list-table::
20-
:widths: 25 75
21-
:header-rows: 1
19+
./run_mapping.sh <from-path> <to-path> [options] <output-file> <spin-db> [db-port]
2220
23-
* - Variable
24-
- Description
25-
* - ``SCANCODE_DB_PASS``
26-
- Database password (default: ``scancode``)
27-
* - ``SCANCODE_DB_USER``
28-
- Database user (default: ``scancode``)
21+
Arguments
22+
---------
2923

30-
-------------------------------------------------------------------------------
31-
Usage Example
32-
-------------------------------------------------------------------------------
24+
+----------------+------------------------------------------------------------+
25+
| Argument | Description |
26+
+================+============================================================+
27+
| ``from-path`` | Path to the base deployment/scan file |
28+
+----------------+------------------------------------------------------------+
29+
| ``to-path`` | Path to the target deployment/scan file |
30+
+----------------+------------------------------------------------------------+
31+
| ``options`` | D2D pipeline parameters (can be an empty string ``""``) |
32+
+----------------+------------------------------------------------------------+
33+
| ``output-file``| File to write ScanCode.io output into |
34+
+----------------+------------------------------------------------------------+
35+
| ``spin-db`` | ``true`` to spin a temporary DB container, ``false`` to skip|
36+
+----------------+------------------------------------------------------------+
37+
| ``db-port`` | Optional. Port to bind Postgres. Default: ``5432`` |
38+
+----------------+------------------------------------------------------------+
3339

34-
.. code-block:: bash
40+
Example
41+
-------
3542

36-
sudo su -
37-
python3 etc/scripts/run_d2d_scio.py \
38-
--input-file ./path/from/from-intbitset.tar.gz:from \
39-
--input-file ./path/to/to-intbitset.whl:to \
40-
--option Python \
41-
--output res1.json
42-
43-
-------------------------------------------------------------------------------
44-
Parameters
45-
-------------------------------------------------------------------------------
46-
47-
.. list-table::
48-
:widths: 25 75
49-
:header-rows: 1
50-
51-
* - Parameter
52-
- Description
53-
* - ``--input-file <path:tag>``
54-
- Required twice: one tagged ``:from``, one tagged ``:to``
55-
* - ``--option <name>``
56-
- Optional; e.g., ``Python``, ``Java``, ``Javascript``, ``Scala``, ``Kotlin``
57-
* - ``--output <file.json>``
58-
- Required; JSON output file for results
59-
60-
-------------------------------------------------------------------------------
61-
Internal Steps
62-
-------------------------------------------------------------------------------
63-
64-
1. Creates or uses the ``./d2d`` directory
65-
2. Copies ``from`` and ``to`` files into it
66-
3. Spins up a temporary **Postgres 13** container
67-
4. Waits for database readiness
68-
5. Runs **ScanCode.io** pipeline (``map_deploy_to_develop``)
69-
6. Saves pipeline output to the specified JSON file
70-
7. Cleans up containers automatically
71-
72-
-------------------------------------------------------------------------------
73-
Cleanup
74-
-------------------------------------------------------------------------------
75-
76-
Containers are auto-removed, but you can verify active containers with:
43+
Run mapping without database:
7744

7845
.. code-block:: bash
7946
80-
docker ps -a | grep scancode
47+
./run_mapping.sh ./deploy.json ./dev.json "" results.txt false
8148
82-
If manual cleanup is needed:
49+
Run mapping with database on a custom port:
8350

8451
.. code-block:: bash
8552
86-
docker rm -f <container_id>
53+
./run_mapping.sh ./deploy.json ./dev.json "--option=fast" output.txt true 5555
54+
55+
Script Actions
56+
--------------
57+
58+
1. Validates required arguments
59+
2. Starts PostgreSQL in Docker (if ``spin-db=true``)
60+
3. Creates a temporary working directory: ``./d2d``
61+
4. Copies input files into working directory
62+
5. Runs ScanCode.io mapping step:
63+
64+
.. code-block:: text
65+
66+
run map_deploy_to_develop:<D2D_OPTIONS> \
67+
"/code/<from-file>:from,/code/<to-file>:to"
68+
69+
6. Writes mapping output into ``output-file``
70+
7. Cleans up temp directory
71+
8. Stops DB container if it was started
72+
73+
Dependencies
74+
------------
75+
76+
* Bash
77+
* Docker
78+
* Local filesystem permissions for creating ``./d2d`` and writing output
79+
80+
Return Codes
81+
------------
82+
83+
* ``0`` - Success
84+
* ``1`` - Incorrect usage or failure to start DB container
85+
* Non-zero - Any error inside Docker/ScanCode.io runtime
86+
87+
Caveats & Operational Notes
88+
---------------------------
89+
90+
* If the script crashes, the DB container may remain running. Manually remove:
91+
92+
.. code-block:: bash
93+
94+
docker rm -f scancodeio-run-db
95+
96+
* Pipeline options are not validated by the script — invalid options will cause
97+
ScanCode.io to exit non-zero.
98+
* ``--network host`` means container DB access must bind directly to ``localhost``.
99+
100+
License
101+
-------
102+
103+
Same license terms as the containing repository.

‎etc/scripts/d2d/run_d2d.sh‎

Lines changed: 8 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -60,25 +60,14 @@ TO_FILENAME=$(basename "$TO_PATH")
6060

6161
echo "Running ScanCode.io mapping..."
6262

63-
if [ -z "$D2D_OPTIONS" ]; then
64-
docker run --rm \
65-
-v "$(pwd)/$WORKDIR":/code \
66-
--network host \
67-
-e SCANCODEIO_NO_AUTO_DB=1 \
68-
ghcr.io/aboutcode-org/scancode.io:latest \
69-
run map_deploy_to_develop \
70-
"/code/${FROM_FILENAME}:from,/code/${TO_FILENAME}:to" \
71-
> "$OUTPUT_FILE"
72-
else
73-
docker run --rm \
74-
-v "$(pwd)/$WORKDIR":/code \
75-
--network host \
76-
-e SCANCODEIO_NO_AUTO_DB=1 \
77-
ghcr.io/aboutcode-org/scancode.io:latest \
78-
run map_deploy_to_develop:"$D2D_OPTIONS" \
79-
"/code/${FROM_FILENAME}:from,/code/${TO_FILENAME}:to" \
80-
> "$OUTPUT_FILE"
81-
fi
63+
docker run --rm \
64+
-v "$(pwd)/$WORKDIR":/code \
65+
--network host \
66+
-e SCANCODEIO_NO_AUTO_DB=1 \
67+
ghcr.io/aboutcode-org/scancode.io:latest \
68+
run map_deploy_to_develop:"$D2D_OPTIONS" \
69+
"/code/${FROM_FILENAME}:from,/code/${TO_FILENAME}:to" \
70+
> "$OUTPUT_FILE"
8271

8372
echo "Output saved to $OUTPUT_FILE"
8473

0 commit comments

Comments
 (0)