Implement generic match algorithm definition config for OCL Mapper

[paynejd]

Core OCL API (and the open-source distro) supports at least 2 algos out of the box. OCL Online may host several more algos. This is to define the algorithms that are available to an instance of OCL Mapper -- independent of its deployment environment. Default config should point to a bundled OCL API (i.e. ocl-search, ocl-semantic). OCL Mapper will need a simple data model for this, but expect that the configuration will live in a file managed by sysadmin at this stage.

@snyaggarwal Please review and add your thoughts -- especially the approach of this living in OCL Mapper not OCL API. Since TBv3 will also expose match options, implementing as OCL Mapper config may not be sufficient. What about a single Match Algo config file that is used by any of the services that you have deployed? That way it works with OCL Mapper and TBv3 (and API?)

OCL Online will start with the following options:

• ocl-search (or ocl-standard?)
• ocl-semantic
• ciel-bridge
• scispacy-loinc

field	description	example
id	Unique identifier used in OCL Mapper, saving projects, downloading spreadsheet, etc.	"ocl-semantic"
name	User-friendly name of algorithm	"OCL Semantic Search"
description	User-friendly short description of an algorithm	"semantic (cosine similarity) search and lexical/keyword search on names using cross-lingual vector embeddings with keyword-based property matching"
llm-context	LLM-facing context for interpretation of results	BM25 raw score, etc.
request_url	Match API request URL	https://api.openconceptlab.org/concepts/$match/

Also consider:

• Applicable target repository(ies) and version range(s)
• Supported field mappings
• Authentication Approach

[snyaggarwal]

current definition example:

{'id': 'ocl-search', 'type': 'ocl-search', 'provider': 'ocl', 'order': 1}
{'id': 'ocl-semantic', 'type': 'ocl-semantic', 'provider': 'ocl', 'order': 1}
{'id': 'ocl-ciel-bridge', 'type': 'ocl-ciel-bridge', 'provider': 'ocl', 'order': 2}
{'id': 'ocl-scispacy-loinc', 'type': 'ocl-scispacy', 'provider': 'ocl', 'order': 3}

[paynejd]

• In the end- Nothing about algorithms is hard-coded in OCL Mapper, it's all in config
• Enables us to track algorithm attributes and capabilities
• Document what each returns (full candidate details vs. codes only)
• Define algorithm_type (semantic_search, llm, exact_match, fuzzy_match, bridge_terminology, etc.)
• Input requirements…
  • requires_source_context - Boolean: needs source terminology info
  • requires_target_context - Boolean: needs target terminology metadata
  • max_input_length - Character/token limit
  • supported_languages - ISO language codes
  • batch_capable - Supports batch processing
  • context_fields_used - What input metadata is leveraged (descriptions, synonyms, etc.)
• Output characteristics…
  • returns_full_concepts - Boolean: provides complete concept details
  • returns_scores - Boolean: includes confidence/similarity scores
  • score_range - Min/max score values (e.g., [0, 1])
  • returns_highlights - Boolean: indicates matched text fragments
  • returns_rationale - Boolean: provides explanation for matches
  • max_candidates - Maximum results returned
  • output_format - Structure/schema of results
• Rate limiting, batch limits, authentication, cost_model, can_replace_lookup, ...

[paynejd]

First pass at bringing this together into a specification:

• For OCL hosted algorithms, do not expose configuration to users that are explicitly defined in the algorithm definition, e.g. key, batch size, requires lookup
• key is a customizable field for custom match algorithms and when more than one of the same algo is included in the project config (based on the multi-use-capable attribute
• order should not be part of an individual algorithm's definition -- that should be handled separately (TBD)

Required algorithm attributes:

• Algorithm Core Attributes
  • key - ocl-semantic
  • name - OCL Semantic Match
  • description -
  • algorithm-type - Ideally this is a coded field, e.g. semantic_search, llm, exact_match, fuzzy_match, bridge_terminology
  • provider - ocl
  • request-url -
  • algorithm-type -
  • multi-use-capable (boolean) -
  • supported-target-repositories -
• Request Characteristics
  • batch-capable (boolean) -
  • max-batch-size (int) - Maximum number of rows supported per batch
  • max-concurrent-requests (int) - 1 (default)
  • rate-limit-behavior - ...
  • auth-type -
• Input Characteristics (content)
  • supported_fields - name, etc.
• Output Characteristics
  • candidate-type - standard, bridge -- this impacts many parts of the app, such as display in the candidates tab
  • can_replace_lookup (boolean)
  • score-type - "BM25", etc.