Port GeoNames RDF to LDE #782

Activity

1. ddeboer

   added the

   Task

   issue type on Aug 31, 2026
2. ddeboer commented on Aug 31, 2026

   

   ddeboer

   on Aug 31, 2026

   MemberAuthor

   More actions

   Musts 3, 4 and 5 are folded into #511 and are green there:

   • 3 – adminCodesFile is now an optional load. One correction to this issue: it stays a single string, not string | string[]. SPARQL Anything's -l takes one path, and a file (loaded into the default graph) is not interchangeable with a directory (one named graph per RDF file), so an array would have been both unsupported by the CLI and semantically different from what it suggests.
   • 4 – each chunk's --output is asserted non-empty before it is concatenated.
   • 5 – convert() awaits the concatenated output being flushed and closed.

   Must 3 moved forward deliberately: the package is not published yet, so removing the domain-specific name is free now and a breaking bump later. It also prompted a general rule in AGENTS.md (#783) – a package's public API must stay domain-agnostic.

   Remaining here: musts 1–2 (concurrency, JVM heap and a CLI passthrough) and shoulds 6–9.
3. ddeboer commented on Aug 31, 2026

   

   ddeboer

   on Aug 31, 2026

   MemberAuthor

   More actions

   Status: every must and should is done or in review

   #		Where
   Must 1	Parallelism	#795 (open)
   Must 2	JVM heap, CLI escape hatch	#789
   Must 3	Generic, optional load	#511
   Must 4	Guard against an empty chunk output	#511
   Must 5	convert() resolving before the flush	#511
   Should 6	One worker pool, not two phases	#791
   Should 7	convert() returning the chunk outputs	#791, by removing the need
   Should 8	Chunk outputs beside the inputs, never cleaned	#511
   Should 9	wait() race under concurrency	#787

   Once #795 merges, what is left in this issue is the Scope beyond #511 section: chunking, the ontology step, and publication.

   Where the plan changed

   Four items did not land the way this issue described them, and the reasons are worth recording.

   Must 3 – load is a string, not string | string[]. SPARQL Anything's -l takes one path, and the CLI usage documents repetition only for -v and -c. A file is also not interchangeable with a directory: a file loads into the default graph, a directory loads each RDF file into its own named graph. An array would have been unsupported and semantically different from what it suggests.

   Must 1 – no memory-aware default. This issue asks for a default sized from nproc and the cgroup limit. The converter spawns through a TaskRunner, so with Docker or a remote runner those describe the wrong machine: concurrency × heap is spent where the runner puts the processes, which the converter cannot see. The default is therefore 1, and the arithmetic map.sh does belongs to the caller, who knows their deployment. Same reasoning made heap a plain option with a 2g default rather than something derived.

   Should 7 – settled by removing the need. Returning the chunk outputs was for a caller that had to concatenate places + alternate names + ontology itself. Since #511 those outputs live in a per-run directory the converter deletes, so returning them would hand back paths that no longer exist – and since #791 a single convert() call takes all the jobs and concatenates them in order, which is what the returning was for.

   Should 8 – closed by the workDir change in #511, not separately. Per-chunk outputs no longer land beside the inputs: they go into a fresh directory under workDir, removed when the run ends. That also fixed a hazard this issue did not name – a stale .nt from an earlier run could satisfy must 4's non-empty check with the previous run's triples.

   Things found on the way that were not in this issue

   • wait() could hang in three ways, not one (fix(task-runner-native): settle wait() for a process that already closed #787). Beyond the race described here, wait() removed every close listener, so it stranded a stop() in flight and any concurrent wait(); and waiting twice answered the second caller with an empty string. The class now keeps one record per task, created at spawn, which removed the pid-keyed output buffers as well – those were unbounded and unsound, since the OS recycles pids.
   • Shell interpolation is quoted (feat(sparql-anything): add SparqlAnythingConverter for chunked non-RDF to RDF conversion #511), using a shellQuote promoted from sparql-qlever into @lde/task-runner so both consumers share one implementation.
   • Docker never actually worked (feat(sparql-anything): add SparqlAnythingConverter for chunked non-RDF to RDF conversion #511). The generated query file was written to the host temp directory, which a container cannot see, while the docs claimed otherwise. Query files and chunk outputs now live under workDir and are named relative to it.
   • A job's chunks cannot be silently ignored (feat(sparql-anything)!: convert a list of jobs, not a list of chunks #791). A query naming {SOURCE} without chunks, or chunks whose query never names it, is rejected: SPARQL Anything would report the first as a parse error and the second not at all.
   • cliArgs cannot repeat the converter's own flags (feat(sparql-anything)!: convert a list of jobs, not a list of chunks #791). --format=TTL slipping through would concatenate Turtle as if it were N-Triples, and the run would stay green.

   Package

   @lde/sparql-anything is on npm, bootstrapped manually after #511 (0.0.1), with a Trusted Publisher attached; CI has published since.
4. ddeboer commented on Aug 31, 2026

   

   ddeboer

   on Aug 31, 2026

   MemberAuthor

   More actions

   #795 is merged, so every must and should in this issue is done. @lde/sparql-anything is at 0.1.0, @lde/task-runner-native at 0.2.18.

   What changed in #795 after the status comment above, all from review:

   • Stopping the other chunks is best effort. A task runner cannot stop what has already exited – DockerTaskRunner.stop() reads logs and stops the container, both of which reject for one that is gone – which is the likely state when a sibling has just failed. Unguarded, that replaced the conversion failure with a "no such container" error and rejected the join, leaving the other workers unawaited while the run directory was deleted underneath them.
   • A chunk could be started after the run had failed. The abort guard sits before two awaits, so a worker could pass it and then start a process while another chunk was failing – a process not in the set that failure stopped, whose whole conversion the run then waited out. Each worker now checks once its process exists, and stops it.

   New follow-up: DockerTaskRunner cannot run tasks in parallel

   DockerTaskRunner.run() force-removes any container with the configured containerName before starting a task, so with concurrency above one each chunk destroys the container of the chunk before it. @lde/sparql-anything documents this as a warning, but the fix belongs in @lde/task-runner-docker: a per-task container name, with containerName remaining the stable name only where it is the addressable endpoint (as sparql-qlever needs for network).

   Worth doing before the port runs conversions in Docker, which is the deployment the concurrency docs point at.

   What is left in this issue

   Only the Scope beyond #511 section:

   • Chunking – download.sh's chunk_with_header, which splits a TSV into CHUNK_SIZE-row chunks each carrying the header row. Generic, and the converter deliberately takes chunks it does not create, so this is the natural next LDE piece.
   • The ontology step – fetch a pinned ontology_v3.3.rdf and convert RDF/XML to N-Triples. Expressible as a job today: a query with no chunks and load pointing at the ontology.
   • Publication – gzip -9, zip -9, upload to Spaces. Probably stays in the consumer's CI.

   And test.sh's golden-file test, which whatever replaces map.sh has to keep: it runs the real conversion over fixtures and diffs against test/expected/geonames.nt.
5. ddeboer commented on Sep 1, 2026

   

   ddeboer

   on Sep 1, 2026

   MemberAuthor

   More actions

   Correction: the conversion takes ~17 minutes, not ~24 hours

   Must 1 above claimed the job "already takes ~24 h in CI". That is wrong by a factor of about 35, and I have corrected it in the description. The real numbers, from geonames-rdf's Transform GeoNames data to RDF workflow:

   The last seven successful runs took 36, 37, 38, 39, 39, 40 and 45 minutes. Broken down, for the run of 2026-08-30:

   step	duration
   download.sh – fetch, unzip, chunk	2 min 09 s
   map.sh – the conversion itself, already an xargs -P pool	17 min 29 s
   upload artifact	2 min 25 s
   compression (gzip -9, zip -9)	15 min 24 s
   uploads to Spaces	2 min 40 s

   Two things follow that are worth carrying into the port.

   Parallelism is still worth having, for a different reason than stated. map.sh is already parallel, so the 17 minutes is the pooled figure; the risk was never a day-long job getting longer, it was that a sequential convert() would have made a 17-minute step into a much longer one. The measurement does not change must 1, only the size of what it protects.

   Chunking is a smaller share than any of us assumed, and compression is a bigger one. All of download.sh – downloading several hundred megabytes, unzipping, and chunking – is 2 minutes, so chunking itself is seconds. Meanwhile compression takes nearly as long as the conversion. If anything in this pipeline deserves attention on wall-clock grounds, it is the 15 minutes of gzip -9/zip -9 in the publication step, not the chunker.

   I raise it because I have been arguing about chunker implementations on a "it is nothing against a day-long job" basis. Against a 40-minute workflow the argument still holds – a chunker difference of ~50 s is ~2 % – but it holds by a much smaller margin than I claimed, and the honest reason to choose is correctness rather than speed.