fcs-development-guide.adoc

Guide to Endpoint Development

• Important preliminary questions

• Existing implementations, resources for new development

• Prerequisites

Development Decisions

❓ Can I host the endpoint myself?

❗ No → HelpDesk: CLARIN, Text+

❓ What type of data do I have?

❗ Raw text, Vertical/CONLL, TEI, …

❓ Which search engine do I use / can I use?

❗ KorAP, Korp/CWB, Lucene/Solr/ElasticSearch, BlackLab, (No)SketchEngine, …

❓ Customization or new development?

❗ List of existing endpoint implementations (Awesome List)

❓ Programming language?

❗ Java, Python, (PHP, XQuery)

❓ In-house development: Use of the reference libraries (Java, Python)

❗ Maven Archetype, Korp

❗ SRU + FCS specifications …

Endpoint Implementations

• Korp FCS 2.0 - reference implementation, Korp corpus search

• CQP/SRU bridge - Corpus Workbench (CWB)

• KonText, fcs-noske-endpoint - (No)SketchEngine (CONLL/Vertical)

• oclcsrw - SRW/SRU server for DSpace, Lucene and/or Pears/Newton

• corpus_shell, SADE - MySQL PHP/DDC Perl, eXist/XQuery

• arche-fcs - ARCHE Suite, php

• Blacklab / MTAS - corpus search engines using Lucene/Solr

• KorapSRU - KorAP (IDS)

Sources: clarin, awesome-fcs

New Endpoint Development

• Customization of reference implementation (Korp)

  • Java: github.com/clarin-eric/fcs-korp-endpoint

  • Python: github.com/Querela/fcs-korp-endpoint-python

• Development using CLARIN SRU/FCS libraries

  • Java: github.com/clarin-eric/fcs-endpoint-archetype

  • Docs:

    • Java SRU: clarin-eric.github.io/fcs-sru-server/apidocs/index.html

    • Java FCS: clarin-eric.github.io/fcs-simple-endpoint/apidocs/index.html

    • Python SRU: fcs-sru-server-python.readthedocs.io

    • Python FCS: fcs-simple-endpoint-python.readthedocs.io

• “New” development specifications (for other languages)

  • SRU: OASIS SRU Overview, Library of Congress SRU

  • FCS: github.com/clarin-eric/fcs-misc → “FCS Core 2.0”

• Awesome List: github.com/clarin-eric/awesome-fcs

Prerequisite for local search engine

❗ Full-text search

❓ With Hit markers

❓ Corpus search (segmented text with annotations)

❕ Pagination (total number of hits)

❗ Resource PID

❓ Linking to result pages