Sends processed data to the database
python3.8 -m venv ingest
Note: You may have to install venv tools on your system. For linux, the command is
sudo apt install python3.8-venv
To activate the virtual environment:
Windows:
ingest\Scripts\activate
MacOS/Linux:
source ingest/bin/activate
You are now in a virtual environment where you can install the requirements and run the main script.
To deactivate the environment, run
deactivate
pip install -r requirements.txt
The probe matcher takes in a path leading to a directory of Unity Simulator data. For each json file found, the script attempts to find matches within the actionList for each of the adept and soartech yaml files. d To run the probe matcher, execute the following command:
python3 probe_matcher.py -i [path_to_directory]
Ensure that the path leads to the Unity Simulator data. The format should be: top-level -> id -> id -> json, with csvs directly under top-level. As the Unity file output organization format changes, the way of accessing this data may also change. For the metrics evaluation data, use the folder in this repo 'metrics-data'.
The probe matcher will output two json files for each input json found: one for soartech and one for adept. If SEND_TO_MONGO is true, these json files (and the raw jsons found) will be sent to the mongo database
| Directory | Explanation |
|---|---|
metrics-data |
Contains all of the valid simulator outputs in the same organizational structure unity outputs. |
adept-evals |
The yaml files for each adept evaluation scenario |
soartech-evals |
The yaml files for each soartech evaluation scenario |
To utilize this function, you will need to manually run the deployment script.
We will eventually have a Jenkins job that will run the script whenever ingest is updated.
To run a new script, add it to the /scripts folder, following the naming convention. The deployment script will check the current database version in mongo to determine which scripts need to be run.
Important: the script must have a main function (def main(mongo_db):)
The script will only run if the version is a newer version of the db, the script also create the collection for the version if it doesn't exist yet. Set your .env file to point to the instances of the TA1 servers you have running locally.