🔣 Words not tokens

[hejny]

|Language| models| generate| one| token| at| a| time|.| But tokens are such a low-level concept. Each model has its own tokenization and they are not humanly comprehensible.

You don't need to know x86 instructions to build a great web page, and with Promptbook you don't need to struggle with tokens to write a great LM application.

Promptbook abstracts from tokens and uses human-understandable units like characters, words, sentences, paragraphs and pages. max-tokens and other technical stuff is done automatically on the background.

---

See also 🧪 Expectations #30

[hejny]

🔎 Other existing solutions

(Do some research / discussion)

• Has anyone else had the same problem?
• Has a project come up with the solution
• Is there a research paper about it?
• Is there an article, video, podcast about it?
• Is this even a good idea?