Orbis Eval / Orbis 2 Frontend

The Orbis Eval frontend provides an easy-to-use web interface to compare the outputs of a Named Entity Recognition (NER) and Named Entity Linking (NEL) model with a Gold Standard.

• Install and use Orbis Eval
  • Importing corpora
  • Exporting corpora and annotations
  • Backup all Orbis data
  • Evaluations
• Developer information on the Orbis Eval Frontend
• Developer information on the Orbis Eval Backend

The Orbis 2 Frontend project hosts the Orbis Eval vuejs application, that must be deployed together with the [Orbis 2 backend].

Install and use Orbis Eval

The Orbis2 Docker Container bundles all software necessary to run Orbis Eval.

Inject .env file

Copy the .env.example to .env and adjust the orbisApiBaseURL value.

cp .env.example .env

Start the container

Start and run the container with the following command.

docker run --net host --name orbis2-frontend -p 8090:8090 \
   -e PGDATA=/var/lib/postgresql/data/pgdata \
   -v orbis-data:/var/lib/postgresql/data \
    ghcr.io/orbis-eval/orbis2-frontend:latest

The Postgresql database password (required for database backups) will be automatically generated during the container's first startup and printed to the console. All databases and data is stored in the orbis-data volume. If you prefer to save the data in a local directory, replace orbis-data with its full path (e.g., /home/albert/orbis-data).

Once startup has completed you can access Orbis Eval under the URL http://localhost:8090/.

Import existing corpora into Orbis Eval

You can use the Orbis importer tool for importing corpora into Orbis Eval.

Importing a local corpus

Orbis currently supports importing NIF turtle files. Its import command always takes two arguments

1. the name of the corpus to import and
2. the run name (i.e., the internal name of the imported Annotated Corpus) which is used for distinguishing different versions of the annotations provided for a particular corpus (e.g., gold standard annotations, annotations provided by a human or a machine learning algorithm).

The command below, for example, imports the file KORE50.ttl under the run name KORE50-version1.0

docker exec orbis2-frontend /orbis-backend/scripts/importer.py local KORE50.ttl KORE50-version1.0

Importing a remote corpus

Run the following command to obtain a list of all available remote corpora:

docker exec orbis2-frontend /orbis-backend/scripts/importer.py list-remote

To import a remote corpus use the remote subcommand with the corpus name as provided by list-remote (e.g., corpus N3-RSS-500 with run name N3-RSS-500-version1). The run name refers to the name of the AnnotatedCorpus (i.e., the corpus and the corresponding annotations)

docker exec orbis2-frontend /orbis-backend/scripts/importer.py remote N3-RSS-500 N3-RSS-500-version1

Please refer to the Orbis2 Backend Documentation for a more detailed documentation on the Orbis 2 import tool.

Exporting annotations

Orbis Eval currently supports exporting annotations to the NIF turtle format.

The example below exports the annotated corpus (i.e., run) N3-RSS500-version1 to the file N3-RSS500-version1.ttl using the NIF format.

docker exec orbis2-frontend /orbis-backend/scripts/exporter.py N3-Reuters-128-version1 N3-Reuters-128-version1.ttl --export-format NIF
docker cp orbis2-frontend:/N3-Reuters-128-version1.ttl .

The first command creates the export file and the second copies it to the user's working directory.

Evaluations

Comparing runs - Inter-rater-agreement

Orbis supports computing the following inter-rater agreement (IRR) metrics, which compute the Fleiss Kappa without correction for random agreement and the average macro and micro F1 score.

• 'el_pIRR' (Entity Linking: Inter Rater Agreement; perfect matching)
• 'ec_pIRR' (Entity Classification: Inter Rater Agreement; perfect matching)
• 'er_pIRR' (Entity Recognition: Inter Rater Agreement; perfect matching)

The example below computes the inter-rater agreement between the careercoach2022.v1, careercoach2022.v2 and careercoach2022.v3 corpus annotations.

docker exec orbis2-frontend /orbis-backend/scripts/orbis-eval.py --metrics er_pIRR -- careercoach2022.v1 careercoach2022.v2 careercoach2022.v3

Metric	kappa_micro	kappa_macro	average_macro_f1	average_micro_f1
Entity Recognition: Inter Rater Agreement; perfect matching.	0.54	0.53	0.68	0.65

Comparing a run to a gold standard

Orbis Eval supports the following metrics

• 'content_extraction_f1' (Content Extraction: Precision, Recall and F1)
• 'content_classification_f1' (Content Classification: Precision, Recall and F1)
• 'er_pF1' (Entity Recognition: Precision, Recall and F1; perfect matching)
• 'er_oF1' (Entity Recognition: Precision, Recall and F1; overlapping matching)
• 'ec_pF1' (Entity Classification: Precision, Recall and F1; perfect matching)
• 'ec_oF1' (Entity Recognition: Precision, Recall and F1; overlapping matching)
• 'el_pF1' (Entity Linking: Precision, Recall and F1; perfect matching)
• 'el_oF1' (Entity Linking: Precision, Recall and F1; overlapping matching)
• 'slot_filling_pF1' (Slot filling: Precision, Recall and F1; perfect matching)
• 'slot_filling_oF1' (Slot filling: Precision, Recall and F1; overlapping matching)

The following example computes precision, recall and F1 metric comparing the corpus annotations in run careercoach2022-entities.v0 against the careercoach2022-entities.v1 corpus annotations using the entity linking metric with perfect match (el_pF1, i.e., entities must perfectly match to be considered correct), and overlapping match (el_oF1, i.e., entities may overlap to be considered correctly).

docker exec orbis2-frontend /orbis-backend/scripts/orbis-eval.py --reference careercoach2022-entities.v1 careercoach2022-entities.v0 --metrics el_pF1 el_oF1

The --reference parameter specifies the run considered to hold the gold standard annotations.

Backup all Orbis data

You can create a backup of the Orbis Postgres database by running the following command, replacing [password] with the Postgres password printed at first container startup.

docker exec -i -e PGPASSWORD=[password] orbis2-frontend /bin/bash -c "pg_dump --username postgres orbis" > backup.sql

Orbis Eval Frontend

The following information is intended for developers who want to run the Orbis 2 frontend component on their development platforms.

Setup

Make sure to install the dependencies:

# yarn
yarn

We use yarn instead of npm, its faster and has better security features. Further, the pinia framework could not be installed properly with npm. IMPORTANT: don't mix npm and yarn, since this could lead to wrong dependencies.

Development

Start the development server on http://localhost:3000

yarn run dev

run the unit tests:

jest

Production

Build the application for production:

yarn run build

Locally preview production build:

yarn run preview

Check out the deployment documentation for more information.

Vue Hero Icons

Installed with

yarn add @heroicons/vue