Support non-extension mode

[bendichter]

Hi @neuromechanist thanks for your work on this. I'm testing it out for our applications.

Trying this out a few times on our data: reward_time: Reward delivery. The animal received a drop of juice or food for successful completion of the task (seconds)

I think the ideal annotation would be: "Sensory-event, Experiment-stimulus, Gustatory-presentation, Feedback, Reward, Time-value/# s"

The bot gets close sometimes, but it often uses custom extensions, which I don't like. Would it be possible to provide a mode where you force the agent to only use defined terms?

(base) bdichter@MacBook-Pro-133 dev % hedit annotate "reward_time: Reward delivery. The animal received a drop of juice or food for successful completion of the task (seconds)"
Generating HED annotation (api mode)...
╭──────────────────────────────────────── HED Annotation - SUCCESS ────────────────────────────────────────╮
│ Annotation:                                                                                              │
│   Sensory-event, (Task-stimulus-role, Reward), (Animal-agent, Animal), Gustatory-presentation            │
│                                                                                                          │
│ Status: [green][x] Valid[/] [green][x] Faithful[/] [green][x] Complete[/] [dim](4 validation             │
│ attempts)[/]                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯
(base) bdichter@MacBook-Pro-133 dev % hedit annotate "reward_time: Reward delivery. The animal received a drop of juice or food for successful completion of the task (seconds)"
Generating HED annotation (api mode)...
╭──────────────────────────────────────── HED Annotation - SUCCESS ────────────────────────────────────────╮
│ Annotation:                                                                                              │
│   Agent-action, Sensory-event, Experimental-stimulus, Feedback, Positive, (Animal/Receiver),             │
│ ((Human-agent, Experimenter), (Reward, (Gustatory-presentation, (Item/Juice))))                          │
│                                                                                                          │
│ Status: [green][x] Valid[/] [green][x] Faithful[/] [green][x] Complete[/] [dim](4 validation             │
│ attempts)[/]                                                                                             │
│                                                                                                          │
│ Warnings:                                                                                                │
│   - [TAG_EXTENDED] WARNING: [TAG_EXTENDED] Tag extensions found for [Animal/Receiver, Item/Juice] in     │
│ "Agent-action, Sensory-event, Experimental-stimulus, Feedback, Positive, (Animal/Receiver),              │
│ ((Human-agent, Experimenter), (Reward, (Gustatory-presentation, (Item/Juice))))".  (For more information │
│ on this HED warning, see                                                                                 │
│ https://hed-specification.readthedocs.io/en/latest/Appendix_B.html#tag-extended.)                        │
│                                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯
(base) bdichter@MacBook-Pro-133 dev % hedit annotate "reward_time: Reward delivery. The animal received a drop of juice or food for successful completion of the task (seconds)"
Generating HED annotation (api mode)...
╭──────────────────────────────────────── HED Annotation - SUCCESS ────────────────────────────────────────╮
│ Annotation:                                                                                              │
│   Event, Feedback, Task-event-role/Task-success, (Animal), (Item/Juice, Or, Item/Food)                   │
│                                                                                                          │
│ Status: [green][x] Valid[/] [green][x] Faithful[/] [green][x] Complete[/] [dim](3 validation             │
│ attempts)[/]                                                                                             │
│                                                                                                          │
│ Warnings:                                                                                                │
│   - [TAG_EXTENDED] WARNING: [TAG_EXTENDED] Tag extensions found for [Task-event-role/Task-success,       │
│ Item/Juice, Item/Food] in "Event, Feedback, Task-event-role/Task-success, (Animal), (Item/Juice, Or,     │
│ Item/Food)".  (For more information on this HED warning, see                                             │
│ https://hed-specification.readthedocs.io/en/latest/Appendix_B.html#tag-extended.)                        │
│                                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[neuromechanist]

Hi Ben,

Thanks a lot for testing HEDit. Yes, it is a great idea to have multiple prompt scenarios. I'll make an issue for this.

Please note that handling placeholders is planned in #12 [0.7.0], so we are close. Currently, the prompts sets are not well gear toward that.

In addition, I think the short begging sentence in reward_time: Reward delivery. The animal received a drop of juice or food for successful completion of the task (seconds) trips the model (also it does not know how to use Delay/# s, after just moving the first sentence into the second sentence it tried to use sometimes, but failed to position it correctly):

$ hedit annotate "The animal received a drop of juice or food for successful completion of the task after some delay"        1 ↵  
Generating HED annotation (api mode)...
╭─────────────────────────────────────────────────────────────────── HED Annotation - FAILED ───────────────────────────────────────────────────────────────────╮
│ Annotation:                                                                                                                                                   │
│   Event, Task/Successful-completion, Delay/1 s, (Animal-agent, Animal), (Reward, (Item/Juice, Item/Food))                                                     │
│                                                                                                                                                               │
│ Status: [red][ ] Valid[/] [yellow][ ] Faithful[/] [yellow][ ] Complete[/] [dim](5 validation attempts)[/]                                                     │
│                                                                                                                                                               │
│ Errors:                                                                                                                                                       │
│   - [TAG_GROUP_ERROR] ERROR: [TAG_GROUP_ERROR] Tag "Delay/1 s" is only allowed inside of a top-level tag group, but is not at top level in "Event,            │
│ Task/Successful-completion, Delay/1 s, (Animal-agent, Animal), (Reward, (Item/Juice, Item/Food))".  (For more information on this HED error, see              │
│ https://hed-specification.readthedocs.io/en/latest/Appendix_B.html#tag-group-error.)                                                                          │
│                                                                                                                                                               │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Then, I removed the delay: "The animal received a drop of juice or food for successful completion of the task"; the situation is much better, with two more interesting observations:

1. On the default gpt-oss-120, moving temperature all the way to 0 does not make it to reproduce the same annotation.
2. Using a "more affordable" premium model (anthropic/claude-haiku-4.5) consistently worked at the default temperature:

$ hedit annotate "The animal received a drop of juice or food for successful completion of the task" -t 0.1 -m anthropic/claude-haiku-4.5
Generating HED annotation (api mode)...
╭────────────────────────────────────────────────────────────────── HED Annotation - SUCCESS ───────────────────────────────────────────────────────────────────╮
│ Annotation:                                                                                                                                                   │
│   Sensory-event, Gustatory-presentation, Feedback, Positive, (Animal-agent, Ingestible-object), Task-property, Reward                                         │
│                                                                                                                                                               │
│ Status: [green][x] Valid[/] [green][x] Faithful[/] [green][x] Complete[/] [dim](1 validation attempt)[/]                                                      │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Surprisingly, the cost (~$0.015) for anthropic/claude-haiku-4.5 is on par or sometimes better than the default model, as it settles with just a few iterations, and far less token use.

I should also add some instructions to to the prompt to deal with the abstract sentence such as at the beginning of this example.

[bendichter]

OK, I'll try haiku from now on. I have had a lot of success with Anthropic's model's so it doesn't surprise me that haiku works well here.

Sensory-event, Gustatory-presentation, Feedback, Positive, (Animal-agent, Ingestible-object), Task-property, Reward

It seems like the grouping is off here, right? Wouldn't this imply that the animal is an ingestible object?

[neuromechanist]

yes, it is off despite knowing what grouping is. Let’s make another issue to translate reflect the grouping.

I probably should update the faithfulness agent to interpret the parenthesis.

I have not tried other models (notably Gemini Flash and GPT models) #64

[neuromechanist]

This issue is being addressed as part of #88 (Semantic Schema Search Epic). The embeddings-based approach will enable a --no-extend mode that finds the closest existing tags instead of creating extensions.