The Orbis Eval frontend provides an easy-to-use web interface to compare the outputs of a Named Entity Recognition (NER) and Named Entity Linking (NEL) model with a Gold Standard.
- Install and use Orbis Eval
- Developer information on the Orbis Eval Frontend
- Developer information on the Orbis Eval Backend
The Orbis 2 Frontend project hosts the Orbis Eval vuejs application, that must be deployed together with the [Orbis 2 backend].
The Orbis2 Docker Container bundles all software necessary to run Orbis Eval.
Copy the .env.example
to .env
and adjust the orbisApiBaseURL
value.
cp .env.example .env
Start and run the container with the following command.
docker run --net host --name orbis2-frontend -p 8090:8090 \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v orbis-data:/var/lib/postgresql/data \
ghcr.io/orbis-eval/orbis2-frontend:latest
The Postgresql database password (required for database backups) will be automatically generated during the container's first startup and printed to the console. All databases and data is stored in the orbis-data
volume. If you prefer to save the data in a local directory, replace orbis-data
with its full path (e.g., /home/albert/orbis-data
).
Once startup has completed you can access Orbis Eval under the URL http://localhost:8090/.
You can use the Orbis importer tool for importing corpora into Orbis Eval.
Orbis currently supports importing NIF turtle files. Its import command always takes two arguments
- the name of the corpus to import and
- the run name (i.e., the internal name of the imported Annotated Corpus) which is used for distinguishing different versions of the annotations provided for a particular corpus (e.g., gold standard annotations, annotations provided by a human or a machine learning algorithm).
The command below, for example, imports the file KORE50.ttl
under the run name KORE50-version1.0
docker exec orbis2-frontend /orbis-backend/scripts/importer.py local KORE50.ttl KORE50-version1.0
Run the following command to obtain a list of all available remote corpora:
docker exec orbis2-frontend /orbis-backend/scripts/importer.py list-remote
To import a remote corpus use the remote
subcommand with the corpus name as provided by list-remote
(e.g., corpus N3-RSS-500 with run name N3-RSS-500-version1). The run name refers to the name of the AnnotatedCorpus (i.e., the corpus and the corresponding annotations)
docker exec orbis2-frontend /orbis-backend/scripts/importer.py remote N3-RSS-500 N3-RSS-500-version1
Please refer to the Orbis2 Backend Documentation for a more detailed documentation on the Orbis 2 import tool.
Orbis Eval currently supports exporting annotations to the NIF turtle format.
The example below exports the annotated corpus (i.e., run) N3-RSS500-version1
to the file N3-RSS500-version1.ttl
using the NIF format.
docker exec orbis2-frontend /orbis-backend/scripts/exporter.py N3-Reuters-128-version1 N3-Reuters-128-version1.ttl --export-format NIF
docker cp orbis2-frontend:/N3-Reuters-128-version1.ttl .
The first command creates the export file and the second copies it to the user's working directory.
Orbis supports computing the following inter-rater agreement (IRR) metrics, which compute the Fleiss Kappa without correction for random agreement and the average macro and micro F1 score.
- 'el_pIRR' (Entity Linking: Inter Rater Agreement; perfect matching)
- 'ec_pIRR' (Entity Classification: Inter Rater Agreement; perfect matching)
- 'er_pIRR' (Entity Recognition: Inter Rater Agreement; perfect matching)
The example below computes the inter-rater agreement between the careercoach2022.v1
, careercoach2022.v2
and careercoach2022.v3
corpus annotations.
docker exec orbis2-frontend /orbis-backend/scripts/orbis-eval.py --metrics er_pIRR -- careercoach2022.v1 careercoach2022.v2 careercoach2022.v3
Metric | kappa_micro | kappa_macro | average_macro_f1 | average_micro_f1 |
---|---|---|---|---|
Entity Recognition: Inter Rater Agreement; perfect matching. | 0.54 | 0.53 | 0.68 | 0.65 |
Orbis Eval supports the following metrics
- 'content_extraction_f1' (Content Extraction: Precision, Recall and F1)
- 'content_classification_f1' (Content Classification: Precision, Recall and F1)
- 'er_pF1' (Entity Recognition: Precision, Recall and F1; perfect matching)
- 'er_oF1' (Entity Recognition: Precision, Recall and F1; overlapping matching)
- 'ec_pF1' (Entity Classification: Precision, Recall and F1; perfect matching)
- 'ec_oF1' (Entity Recognition: Precision, Recall and F1; overlapping matching)
- 'el_pF1' (Entity Linking: Precision, Recall and F1; perfect matching)
- 'el_oF1' (Entity Linking: Precision, Recall and F1; overlapping matching)
- 'slot_filling_pF1' (Slot filling: Precision, Recall and F1; perfect matching)
- 'slot_filling_oF1' (Slot filling: Precision, Recall and F1; overlapping matching)
The following example computes precision, recall and F1 metric comparing the corpus annotations in run careercoach2022-entities.v0
against the careercoach2022-entities.v1
corpus annotations using the entity linking metric with perfect match (el_pF1
, i.e., entities must perfectly match to be considered correct), and overlapping match (el_oF1
, i.e., entities may overlap to be considered correctly).
docker exec orbis2-frontend /orbis-backend/scripts/orbis-eval.py --reference careercoach2022-entities.v1 careercoach2022-entities.v0 --metrics el_pF1 el_oF1
The --reference
parameter specifies the run considered to hold the gold standard annotations.
You can create a backup of the Orbis Postgres database by running the following command, replacing [password]
with the Postgres password printed at first container startup.
docker exec -i -e PGPASSWORD=[password] orbis2-frontend /bin/bash -c "pg_dump --username postgres orbis" > backup.sql
The following information is intended for developers who want to run the Orbis 2 frontend component on their development platforms.
Make sure to install the dependencies:
# yarn
yarn
We use yarn instead of npm, its faster and has better security features. Further, the pinia framework could not be installed properly with npm. IMPORTANT: don't mix npm and yarn, since this could lead to wrong dependencies.
Start the development server on http://localhost:3000
yarn run dev
run the unit tests:
jest
Build the application for production:
yarn run build
Locally preview production build:
yarn run preview
Check out the deployment documentation for more information.
Installed with
yarn add @heroicons/vue