Duplicated prediction results when open Auto-Annotation in NER template

[aisensiy]

Describe the bug
A clear and concise description of what the bug is.

I am trying to use label-studio with the feature of Auto-Annotation in a template of NER. But I found that the same prediction from my ML backend will be triggered again and again.

I am not sure if this is the LabelStudio's responsibility or my ML backend's responsibility to avoid this behavior.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

I think LabelStudio show also know that the current task is already get annoated by the ML backend so that the same annotation will not be added twice.

Screenshots
If applicable, add screenshots to help explain your problem.

Here is a screen record to show the behavior (no voice).

duplicated_annotattion_in_label_studio.mov

In the video you can see no matter how many times I annotated the text, the same prediction result from ML backend will be added to the text.

Environment (please complete the following information):

• OS: Mac OS 12.4
• Label Studio Version 1.5.0

Additional context
Add any other context about the problem here.

[KonstantinKorotaev]

Hi @aisensiy
Could you please check what your ML service returned in last calls?

[aisensiy]

Could you please check what your ML service returned in last calls?

Do you mean the predict result? The result looks like this:

[{'result': [{'from_name': 'label',
    'to_name': 'text',
    'type': 'labels',
    'value': {'start': 0, 'end': 3, 'text': '中金在', 'labels': ['MISC']},
    'score': 0.7986028842714906}]}]

And here is the model.py in my label-studio-ml-backend:

import random

from label_studio_ml.model import LabelStudioMLBase


class DummyModel(LabelStudioMLBase):

    def __init__(self, **kwargs):
        super(DummyModel, self).__init__(**kwargs)
        
        # pre-initialize your variables here
        from_name, schema = list(self.parsed_label_config.items())[0]
        self.from_name = from_name
        self.to_name = schema['to_name'][0]
        self.labels = schema['labels']

    def predict(self, tasks, **kwargs):
        print(self.labels)
        print(tasks)
        results = []
        for task in tasks:
            text = task['data']['text']
            
            results.append({
                'result': [{
                    'from_name': self.from_name,
                    'to_name': self.to_name,
                    'type': 'labels',
                    'value': {
                        'start': 0, 'end': 3, 
                        'text': text[:3],
                        'labels': ['MISC']
                    },
                    'score': random.uniform(0, 1)
                }]
            })
        print(results)
        return results

    def fit(self, completions, workdir=None, **kwargs):
        return {'random': random.randint(1, 10)}

It is based on the dummy_model. I just use the first three letters to generate a dummy result and return.

[KonstantinKorotaev]

Do you mean the predict result? The result looks like this:

Yes, do you have a sequence of calls that lead to duplicated results?

[aisensiy]

Yes. I do have a sequence of actions to manually add some more annotations not covered by the ml backend. But I think it is not necessary to return the prediction multi times. Every time when I do some actions (even when I delete some annotations) the same prediction result will be added.

[aisensiy]

Any process about this issue?

[KonstantinKorotaev]

Hi @aisensiy
I have created a new feature request for it.