Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 140 additions & 0 deletions api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,18 @@ info:
The `Accept-Language` / `edition`-based 303 redirect on
`/id/ref/{uuid}` to a resolver-target manifestation is planned but not
yet live.

A client that does not yet know a key starts at `/reg/works.json` or
`/reg/systems.json`. `/dump/` carries the whole registry as
newline-delimited JSON for bulk consumers.

The site is static, and the host derives every `Content-Type` from the
file extension. This contract therefore declares the media type that
the host actually sends, not the media type that best describes the
payload. A `.json` body is JSON-LD by content and arrives as
`application/json`. A `.jsonl` body is newline-delimited JSON by
content and arrives as `application/octet-stream`. Parse by
documented shape, not by response header.
license:
name: AGPL-3.0-or-later
url: https://www.gnu.org/licenses/agpl-3.0.html
Expand All @@ -33,6 +45,15 @@ servers:
tags:
- name: Records
description: Canonical record URLs under `/id/`.
- name: Collections
description:
Registry-wide JSON-LD collections under `/reg/`. Entry points for a
client that does not yet know a key.
- name: Bulk
description:
Whole-registry artifacts under `/dump/`, rebuilt on every deploy. The
host serves them as `application/octet-stream`; the bodies are
newline-delimited JSON.
paths:
/id/work/{key}/:
get:
Expand Down Expand Up @@ -146,6 +167,105 @@ paths:
application/json:
schema: { $ref: '#/components/schemas/MappingAssertion' }
'404': { $ref: '#/components/responses/NotFound' }
/reg/works.json:
get:
tags: [Collections]
summary: Work collection — JSON-LD
operationId: getWorkCollection
description:
Each item in `@graph` is the same record that `/id/work/{key}.json`
serves, without its own `@context`. The collection carries one
`@context` for the whole graph.
responses:
'200':
description:
Every Work record in the registry, in every status, sorted by
key.
content:
application/json:
schema: { $ref: '#/components/schemas/WorkCollection' }
Comment on lines +184 to +186

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the inconsistency — fixed in 35def8e, but in the opposite direction to the one suggested.

The site is static. GitHub Pages derives every Content-Type from the file extension and discards the header the Astro APIRoute sets, so that header never reaches a client. The live site confirms it:

$ curl -sI https://textrefs.org/id/work/plato.republic.json | grep -i content-type
content-type: application/json; charset=utf-8

$ curl -sI https://textrefs.org/dump/works.jsonl | grep -i content-type
content-type: application/octet-stream

So /id/…json declaring application/json was already accurate to the wire. Switching those four paths to application/ld+json would have made the contract describe a response nobody receives. The inaccurate declarations were the two this branch introduced.

Changed instead:

  • both /reg/ collections → application/json
  • the four /dump/ JSONL paths → application/octet-stream (they were declared application/x-ndjson)

Every .json path in the contract is now application/json and every .jsonl path is application/octet-stream, which is both internally consistent and true.

Since the declared types no longer describe the payloads, info.description, the Bulk tag, and url-layout.md now state the rule explicitly: a .json body is JSON-LD by content, a .jsonl body is newline-delimited JSON, and a client should parse by documented shape rather than by response header.

One related thing left alone: the Content-Type: application/ld+json; charset=utf-8 header in the APIRoute handlers is effectively dead code under a static build. It would matter if the site ever moved to SSR, so it stays — out of scope for this PR.

/reg/systems.json:
get:
tags: [Collections]
summary: CitationSystem collection — JSON-LD
operationId: getCitationSystemCollection
description: Each item in `@graph` is the same record that
`/id/system/{key}.json` serves, without its own `@context`. The
collection carries one `@context` for the whole graph.
responses:
'200':
description:
Every CitationSystem record in the registry, in every status,
sorted by key.
content:
application/json:
schema: { $ref: '#/components/schemas/SystemCollection' }
/dump/works.jsonl:
get:
tags: [Bulk]
summary: Work dump — NDJSON
operationId: getWorksDump
responses:
'200':
description:
Newline-delimited JSON, one Work record object per line, with
no `@context`.
content:
application/octet-stream:
schema: { type: string }
/dump/citation-systems.jsonl:
get:
tags: [Bulk]
summary: CitationSystem dump — NDJSON
operationId: getCitationSystemsDump
responses:
'200':
description:
Newline-delimited JSON, one CitationSystem record object per
line, with no `@context`.
content:
application/octet-stream:
schema: { type: string }
/dump/references.jsonl:
get:
tags: [Bulk]
summary: CanonicalReference dump — NDJSON
operationId: getReferencesDump
responses:
'200':
description:
Newline-delimited JSON, one CanonicalReference record object
per line, with no `@context`.
content:
application/octet-stream:
schema: { type: string }
/dump/mappings.jsonl:
get:
tags: [Bulk]
summary: MappingAssertion dump — NDJSON
operationId: getMappingsDump
responses:
'200':
description:
Newline-delimited JSON, one MappingAssertion record object per
line, with no `@context`.
content:
application/octet-stream:
schema: { type: string }
/dump/datapackage.json:
get:
tags: [Bulk]
summary: Frictionless data-package descriptor
operationId: getDataPackage
responses:
'200':
description:
The Frictionless data-package descriptor that lists the four
JSONL resources with, for each, its byte count and its
`sha256:` hash, plus the registry version.
content:
application/json:
schema: { type: object }
components:
parameters:
FlatKey:
Expand Down Expand Up @@ -341,6 +461,26 @@ components:
items: { $ref: '#/components/schemas/Iri' }
minItems: 1
source: { type: string, minLength: 1 }
WorkCollection:
description: The `/reg/works.json` registry-wide collection of Work records.
type: object
required: ['@context', '@graph']
properties:
'@context': { type: string, format: uri }
'@graph':
type: array
items: { $ref: '#/components/schemas/Work' }
SystemCollection:
description:
The `/reg/systems.json` registry-wide collection of CitationSystem
records.
type: object
required: ['@context', '@graph']
properties:
'@context': { type: string, format: uri }
'@graph':
type: array
items: { $ref: '#/components/schemas/CitationSystem' }
RegistryObject:
oneOf:
- { $ref: '#/components/schemas/Work' }
Expand Down
43 changes: 42 additions & 1 deletion src/content/docs/get-started/url-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ TextRefs uses four URL prefixes, each with one job. Together they make every reg
| Prefix | Role | What lives there |
| -------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/id/` | **Identifier** | The canonical, persistent URL of every record. Each record is published twice: `/id/.../` (HTML) and a sibling `/id/....json` (JSON-LD). |
| `/reg/` | **Browse** | The human registry browser: filter works and citation systems, then browse paginated reference lists from work pages. Links into `/id/`. |
| `/reg/` | **Browse** | The human registry browser: filter works and citation systems, then browse paginated reference lists from work pages. Links into `/id/`. Also serves two JSON-LD collections, so a client can discover records without a key. |
| `/cite/` | **Cite** | Short, memorable URLs (`/cite/{work}/{system}/{locator}` always, and bare `/cite/{work}/{locator}` for the work's preferred system) that redirect to the canonical `/id/` URL. Convenience only. Bare aliases MAY be retargeted. `/id/` is permanent. |
| `/api/` | **API docs** | The OpenAPI document that describes the `/id/` URL contract, plus the JSON-LD `@context` at `/contexts/`. |

Expand Down Expand Up @@ -49,6 +49,47 @@ A client either reads that `<link>` tag, or appends `.json` to the canonical URL

This mirrors how arxiv.org publishes each paper at `/abs/{id}` and `/pdf/{id}` — two static URLs, two representations, no negotiation needed.

## Collections and bulk data

Every `/id/` URL needs a key you already know. The `/reg/` collections give a client a starting point when it does not have one yet.

`/reg/` serves two static JSON-LD collections:

- `https://textrefs.org/reg/works.json` — every work in the registry
- `https://textrefs.org/reg/systems.json` — every citation system in the registry

Each collection wraps its records in one `@context` and one `@graph` array:

```json
{
"@context": "https://textrefs.org/contexts/v1.jsonld",
"@graph": [
{
"id": "https://textrefs.org/id/work/plato.republic",
"key": "plato.republic",
"type": "Work",
"…": "…"
}
]
}
```

Each item carries no `@context` of its own. The collection lists records of every status — active, draft, and retired — sorted by `key`.

For bulk use, five artifacts live under `/dump/`:

- `https://textrefs.org/dump/works.jsonl`
- `https://textrefs.org/dump/citation-systems.jsonl`
- `https://textrefs.org/dump/references.jsonl`
- `https://textrefs.org/dump/mappings.jsonl`
- `https://textrefs.org/dump/datapackage.json` — a Frictionless data-package descriptor. It lists the four JSONL resources with a byte count and a `sha256:` hash for each.

Each `.jsonl` file holds one record per line, with no `@context`. This is the bulk-archive form of the registry.

The site is static, and the host sets each `Content-Type` from the file extension. A `.json` body is JSON-LD by content, but it arrives as `application/json`. A `.jsonl` body arrives as `application/octet-stream`. Parse each body by its documented shape. Do not parse it by the response header.

Use the small JSON-LD collection for a browser client. Use the `/dump/` files for a bulk consumer. See [`/api/`](/api/) for the full contract.

## Why four prefixes, not one

Persistent-identifier systems separate concerns. DOI and ORCID each have a canonical resolver URL that _is_ the API. Their documentation lives at a stable but distinct path. W3ID and PURL add short-alias redirects on top. TextRefs follows the same pattern:
Expand Down
38 changes: 38 additions & 0 deletions src/lib/collection.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { collectionBody, JSONLD_CONTEXT } from './collection.js';
import { fixtureRegistry } from './registry.fixture.js';

test('collectionBody sets the shared JSON-LD context', () => {
const body = collectionBody(fixtureRegistry.systems);
assert.equal(body['@context'], 'https://textrefs.org/contexts/v1.jsonld');
assert.equal(body['@context'], JSONLD_CONTEXT);
});

test('collectionBody sorts @graph by key', () => {
const unsorted = [...fixtureRegistry.systems].reverse();
assert.ok(unsorted.length > 1);
const body = collectionBody(unsorted);
const keys = body['@graph'].map((r) => r.key);
const expected = [...keys].sort((a, b) => a.localeCompare(b));
assert.deepEqual(keys, expected);
});

test('collectionBody does not mutate the input array', () => {
const unsorted = [...fixtureRegistry.systems].reverse();
const before = unsorted.map((r) => r.key);
collectionBody(unsorted);
assert.deepEqual(
unsorted.map((r) => r.key),
before,
);
});

test('collectionBody passes items through unchanged with no per-item @context', () => {
const body = collectionBody(fixtureRegistry.systems);
const item = body['@graph'].find(
(r) => r.key === fixtureRegistry.systems[0].key,
);
assert.deepEqual(item, fixtureRegistry.systems[0]);
assert.equal(item && '@context' in item, false);
});
15 changes: 15 additions & 0 deletions src/lib/collection.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// Registry-wide JSON-LD collections served under `/reg/` (works and citation
// systems). The sort order here is shared with the HTML registry browser
// (`src/pages/reg/index.astro`), so the two views cannot drift apart.

export const JSONLD_CONTEXT = 'https://textrefs.org/contexts/v1.jsonld';

export function byKey(a: { key: string }, b: { key: string }): number {
return a.key.localeCompare(b.key);
}

export function collectionBody<T extends { key: string }>(
records: readonly T[],
) {
return { '@context': JSONLD_CONTEXT, '@graph': [...records].sort(byKey) };
}
25 changes: 23 additions & 2 deletions src/pages/reg/index.astro
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@
import StarlightPage from '@astrojs/starlight/components/StarlightPage.astro';
import { loadWorks, loadSystems } from '../../lib/registry.ts';
import { UNSTABLE_BANNER } from '../../lib/banner.ts';
import { byKey } from '../../lib/collection.ts';

const works = [...loadWorks()].sort((a, b) => a.key.localeCompare(b.key));
const systems = [...loadSystems()].sort((a, b) => a.key.localeCompare(b.key));
const works = [...loadWorks()].sort(byKey);
const systems = [...loadSystems()].sort(byKey);
---

<StarlightPage
Expand All @@ -15,6 +16,26 @@ const systems = [...loadSystems()].sort((a, b) => a.key.localeCompare(b.key));
template: 'doc',
tableOfContents: false,
banner: UNSTABLE_BANNER,
head: [
{
tag: 'link',
attrs: {
rel: 'alternate',
type: 'application/ld+json',
href: '/reg/works.json',
title: 'Works',
},
},
{
tag: 'link',
attrs: {
rel: 'alternate',
type: 'application/ld+json',
href: '/reg/systems.json',
title: 'Citation systems',
},
},
],
}}
>
<p>
Expand Down
8 changes: 8 additions & 0 deletions src/pages/reg/systems.json.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import type { APIRoute } from 'astro';
import { loadSystems } from '../../lib/registry.ts';
import { collectionBody } from '../../lib/collection.ts';

export const GET: APIRoute = () =>
new Response(JSON.stringify(collectionBody(loadSystems()), null, 2), {
headers: { 'Content-Type': 'application/ld+json; charset=utf-8' },
});
8 changes: 8 additions & 0 deletions src/pages/reg/works.json.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import type { APIRoute } from 'astro';
import { loadWorks } from '../../lib/registry.ts';
import { collectionBody } from '../../lib/collection.ts';

export const GET: APIRoute = () =>
new Response(JSON.stringify(collectionBody(loadWorks()), null, 2), {
headers: { 'Content-Type': 'application/ld+json; charset=utf-8' },
});
Loading