Error: failed to create engine: how to create /home/teto/.local/share/wordbase

[teto]

I got directed here from jintindex Jitendex/Jitendex#138.

I would like to create a yomitan-like experience on my text editor www.neovim.io, i.e, have popups with the translated texts appear in some way.

I can do the tokenization well with https://github.com/WorksApplications/sudachi.rs but asking an offline japanese dictionary seems unexpectedly hard ?! tagainjisho works well but can't be asked from cli/api.

This repo seems the more promising so far but I have a few questions/remarks. I know it's WIP so I thought some user feedback could help.
https://github.com/WorksApplications/sudachi.rs

First a reminder for myself on how to compile the binary:

$ cargo install --path ./crates/wordbase-cli/
➜ cargo install
error: found a virtual manifest at `/home/teto/wordbase/Cargo.toml` instead of a package manifest

$ cargo install --path ./crates/wordbase-cli/
...

$ /home/teto/.cargo/bin/wordbase-cli lookup 左
 INFO wordbase: Creating engine using "/home/teto/.local/share/wordbase" as data directory
Error: failed to create engine

Caused by:
    0: failed to connect to database
    1: error returned from database: (code: 14) unable to open database file
    2: (code: 14) unable to open database file

How do I create this engine ?
I tried with the dict subcommand :

$ /home/teto/.cargo/bin/wordbase-cli dict import jitindex
 INFO wordbase: Creating engine using "/home/teto/.local/share/wordbase" as data directory
Error: failed to create engine

.

NB: As a nix user, I like that you provide a nix develop environment, do you think the flake could expose wordbase-cli as well ? this way I could reference this project from my flake directly without having to install it via cargo ?

[aecsocket]

Hi, thanks for the interest in the project!

Sorry if any of the setup instructions were wrong, I've done a big refactor over the last few weeks to focus more on the mobile application instead of the desktop app. This means renaming wordbase -> wordbase-api, wordbase-engine -> wordbase, and some other misc changes. I haven't updated the readme yet to reflect this.

About deinflection and tokenization: if you use Wordbase, you shouldn't need to also use Sudachi or another tokenizer, since Wordbase internally handles deinflecting text (in the case of Japanese, this is done using Lindera).

The wordbase-cli error sounds like something isn't letting SQLite create the file /home/teto/.local/share/wordbase/wordbase.db. Can you check:

• does /home/teto/.local/share/wordbase/ exist?
• does the dir have the right permissions to let you create files there? touch wordbase.db

It may be a NixOS specific issue, maybe if you're using home-manager and have an immutable /home dir? Although usually .local/share shouldn't be immutable, so unfortunately I'm not too sure there.

If you can, try to also get any other tool to create that file /home/teto/.local/share/wordbase/wordbase.db. If that tool fails, then it's likely something wrong with your env not letting you write into that file.

Re the nix develop: I do provide a nix flake for dev, but I actually use this from within a Docker dev container, not on NixOS directly, since I don't use NixOS. I'm not sure how to expose the wordbase-cli binary from the flake (I assume as an outputs somehow), but if you would like to open a PR for this, I'd be happy to merge it!

[teto]

thanks for the kind answer. It's good to know you are ready to help users even though it's wip.
I have a pretty standard setup so nixos shouldn't be in the way here. Creating /home/teto/.local/share/wordbase/ was enough to go through the first hurdle, seems like the program just would not create the folder.

I now have Error: profile ID not specified and I am not sure what that means ?

wordbase via 🦀 v1.87.0 via ❄️  IMPURE (nix-shell-env) on  main 
➜ /home/teto/.cargo/bin/wordbase-cli dict ls
 INFO wordbase: Creating engine using "/home/teto/.local/share/wordbase" as data directory
Error: profile ID not specified
wordbase via 🦀 v1.87.0 via ❄️  IMPURE (nix-shell-env) on  main 
❌1 ❯ /home/teto/.cargo/bin/wordbase-cli dict ls --help
List all dictionaries

Usage: wordbase-cli dict ls

Options:
  -h, --help  Print help

I would like to add a japanese dictionary: how can I do that ? jitindex seems to be a new format so probably wordbase would just support the yomitan one ?

It could help if the --help was more verbose, for instance as to what kind of files to import or where to find those (give examples).

➜  /home/teto/.cargo/bin/wordbase-cli dict import --help
Import a dictionary file from the filesystem

Usage: wordbase-cli dict import <PATH>

Arguments:
  <PATH>  Path to the dictionary file

Options:
  -h, --help  Print help

[aecsocket]

This looks like an oversight, the engine should create the wordbase/ data dir if it doesn't already exist. Easy fix though.

The --profile error is admittedly very vague. When you create the database, it automatically creates a profile with ID 1, which is the default profile. When running a command, you must specify what profile it uses. In this case, run your command like wordbase-cli --profile 1 dict import yomitan.zip, and it should work. I will make this default to the only profile if only 1 exists.

The issue with --help being not descriptive is due to clap IIUC. If you run wordbase-cli --help, it will display the --profile option. But if you run wordbase-cli dict import --help, it won't show that flag, since it's only on the root-level command. I'm not sure how to fix this without an upstream change in clap.

     Running `target/release/wordbase-cli --help`
Usage: wordbase-cli [OPTIONS] <COMMAND>

Commands:
  lookup        Deinflect some text and fetch records for its lemmas
  lookup-lemma  Fetch records for a lemma directly
  deinflect     Deinflect some text and return its lemmas
  render        Fetch records for some text and render the results as HTML
  profile       Manage profiles
  dict          Manage dictionaries
  help          Print this message or the help of the given subcommand(s)

Options:
      --data-dir <DATA_DIR>
  -p, --profile <PROFILE>
  -h, --help                 Print help

Info on the Jitendex format: Jitendex is a Yomitan (not Yomichan) dictionary. But Wordbase supports only Yomitan dictionaries (the newer format), so Jitendex should import mostly ok. But I still haven't implemented imports for everything:

• images/assets
• custom dictionary CSS
• deinflection rules (I'm not even sure how these work, or if I need these)

[aecsocket]

Fixed in f9c000a, you can just run wordbase-cli dict import yomitan.zip and it will work fine (as long as you only have 1 profile).

[teto]

awesome. I was able to downlaod the dict from https://github.com/yomidevs/jmdict-yomitan?tab=readme-ov-file and import it with
/home/teto/.cargo/bin/wordbase-cli --profile 1 dict import ~/Downloads/JMdict_english_with_examples.zip

I could then get results with:

➜ /home/teto/.cargo/bin/wordbase-cli --profile 1 lookup 出る
 INFO wordbase: Creating engine using "/home/teto/.local/share/wordbase" as data directory
RecordEntry {
....
}
Fetched records in 1.012238ms

Now Ideally I would like those results in json ? I suspect the current output is WIP since it dumps rust datastructures.
Maybe you will tell me to leverage the "api" part of wordbase but at least for now where Iam in the prototyping phase it would be easier I think to just have my neovim plugin call a CLI. All other tools failed at emitting a structured output so I think it's a good opportunity for wordbase to avoid that pitfall.

If I can make this work, I will probably submit a PR to package the rust program (I need to investigate how to do that since rust+nix evolves quite fast).

Also I am not sure what are the advantes of profiles but I would think that assuming a default would simplify the UX. (EDIT: maybe fixed on master as your last comment mentions. I will update)

[aecsocket]

The output format of wordbase-cli commands is unspecified, and is designed to be human-readable first. I think the best option would be:

• wordbase-cli lookup 読む outputs a simplified version of all records, maybe with a bit of summary info for each term
  • Other commands like wordbase-cli dict ls will continue to output in a human-readable, tabular format
• wordbase-cli --output json lookup 読む outputs the records in a JSON format
  • Can also extend this to wordbase-cli --output json dict ls, wordbase-cli --output json profile ls etc.
  • This will just be serde_json::to_str(lookups: Vec<RecordLookup>), so it's dead simple to implement

Do be warned that the output of lookup might be very large! Also, the order of lookups DOES matter. The more relevant records come first.

[teto]

It makes sense to be human-readable first I agree. --output json / -ojson can be quite helpful though, especially if it's a simple serde call. I dont care if you break the output I can pin wordbase's version and update on my terms.

Do be warned that the output of lookup might be very large!

Maybe the cli can accept a --limit parameters but I doubt it would be that large for a computer to process. The whole dict could be printed that it would take milliseconds to process the output I believe.

[aecsocket]

Implemented -o / --output json flag to output in JSON in cd5aabe. Normal messages will also be printed, but they will be put in stderr, not stdout. So you can parse JSON directly from stdout for your purposes.

[teto]

ty I've just updated to cd5aabe. I think this breaks the human readable output since now:

/home/teto/.cargo/bin/wordbase-cli --profile 1 lookup 出る 
 INFO wordbase: Creating engine using "/home/teto/.local/share/wordbase" as data directory
 INFO wordbase_cli::lookup: Fetched records in 1.144588ms

with --output json it works fine ! Now I have to understand the returned format which seems to be a straight mapping from yomitan dictionary. The translation seems available in the content part. I always find it odd that the main resource (from what I couild find is yomitan) kinda embeds html directly.

[aecsocket]

I've left the lookup stderr output as a TODO since I don't want to just dump the record info, I'd like to display it in a slightly nicer format. But that's for the future.

The JSON is documented in wordbase-api. Have a look at RecordLookup and any of the types under dict to understand the schema.

Jitendex is a bit weird in relation to other dictionaries. It stores a lot of the actual definition data deeply nested inside the structured content, which makes it annoying to read out the actual contents. Unfortunately I can't do much on my end, and I would suggest that it you have access to a web view, to just render it as html into that web view. Otherwise you can take a look at what Memento does.

[teto]

closing this since I overcame the initial hurdle thanks to your help.