Skip to content

Commit 5f279b9

Browse files
zuiderkwastbjosvmadolson
authored
Generate man pages (#92)
A Makefile provides 'make' and 'make install' to generate and install man pages. Man pages are built for the binaries (valkey-cli, etc. prefixed by "valkey-", section 1), Config (valkey.conf, section 4), Valkey commands (section 3valkey), documentation overview page ("valkey", section 7) and tutorials and everything else (prefixed by "valkey-", section 7). Scripts in Python are used for preprocessing and pandoc is used to convert markdown to man pages, stored under utils/. A target 'make html' is also provided, to build HTML pages to use locally. This is also using pandoc. The documentation pages for `valkey-cli`, `valkey-benchmark` and `valkey-sentinel` are expanded with usage and options to match what's expected from a man page for a program. A page for `valkey-server` is added. (We didn't have one.) README.md is update with info on how to build and install man pages. Other info regarding the doc repo is slightly updated. Additionally some unrelated smaller changes. Fixes #69. --------- Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech> Co-authored-by: Björn Svensson <bjorn.a.svensson@est.tech> Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
1 parent 94f8706 commit 5f279b9

16 files changed

+1967
-136
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
.idea
22
tmp
33
.DS_Store
4+
_build

‎Makefile‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
# Copyright (C) 2024, The Valkey contributors
2+
# All rights reserved.
3+
# SPDX-License-Identifier: BSD-3-Clause
4+
5+
# Path to the code repo.
6+
VALKEY_ROOT ?= ../valkey
7+
8+
# Where to install man pages
9+
INSTALL_MAN_DIR ?= /usr/local/share/man
10+
11+
# Where to put the generated pages.
12+
BUILD_DIR ?= _build
13+
MD_DIR ?= $(BUILD_DIR)/md
14+
MAN_DIR ?= $(BUILD_DIR)/man
15+
HTML_DIR ?= $(BUILD_DIR)/html
16+
17+
# -----------------------------------
18+
19+
.PHONY: all man html md clean distclean install uninstall
20+
21+
all: man
22+
23+
# ---- Sanity check ----
24+
25+
ifeq ("$(wildcard $(VALKEY_ROOT))","")
26+
$(error Please provide the VALKEY_ROOT variable pointing to the Valkey source code)
27+
endif
28+
29+
ifeq ("$(shell which pandoc)","")
30+
$(error Please install pandoc)
31+
endif
32+
33+
# ---- Source files ----
34+
35+
topics = $(wildcard topics/*)
36+
commands = $(wildcard commands/*.md)
37+
38+
topics_md = $(filter %.md,$(topics))
39+
topics_pics = $(filter-out %.md,$(topics))
40+
41+
# ---- Temp files ----
42+
43+
# JSON files for the commands that have a .md file (excluding undocumented commands).
44+
json_for_documented_commands = $(commands:commands/%.md=$(VALKEY_ROOT)/src/commands/%.json)
45+
46+
$(BUILD_DIR)/.commands-per-group.json: $(VALKEY_ROOT)/src/commands/. utils/build-command-groups.py | $(BUILD_DIR)
47+
utils/build-command-groups.py $(json_for_documented_commands) > $@
48+
$(BUILD_DIR):
49+
mkdir -p $@
50+
51+
# ---- Pre-processed markdown files (make md) ----
52+
53+
md_topics_dir = $(MD_DIR)/topics
54+
md_topics = $(topics:topics/%=$(md_topics_dir)/%)
55+
$(md_topics): | $(md_topics_dir)
56+
57+
md_commands_dir = $(MD_DIR)/commands
58+
md_commands = $(commands:commands/%=$(md_commands_dir)/%)
59+
$(md_commands): | $(md_commands_dir)
60+
61+
md_targets = $(md_topics) $(md_commands) $(md_commands_dir)/index.md
62+
63+
$(md_topics_dir) $(md_commands_dir):
64+
mkdir -p $@
65+
66+
$(md_topics_dir)/%.md: topics/%.md utils/preprocess-markdown.py | $(md_topics_dir)
67+
utils/preprocess-markdown.py $< > $@
68+
$(md_topics_dir)/%.png: topics/%.png | $(md_topics_dir)
69+
cp $< $@
70+
$(md_topics_dir)/%.gif: topics/%.gif | $(md_topics_dir)
71+
cp $< $@
72+
73+
$(md_commands_dir)/index.md: $(BUILD_DIR)/.commands-per-group.json groups.json utils/build-command-index.py | $(md_commands_dir)
74+
utils/build-command-index.py --suffix .md \
75+
--groups-json groups.json \
76+
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json > $@
77+
78+
$(md_commands_dir)/%.md: commands/%.md $(VALKEY_ROOT)/src/commands/%.json $(BUILD_DIR)/.commands-per-group.json \
79+
utils/preprocess-markdown.py utils/command_syntax.py
80+
utils/preprocess-markdown.py --page-type command \
81+
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
82+
--valkey-root $(VALKEY_ROOT) $< > $@
83+
84+
# ---- HTML (make html) ----
85+
86+
html_topics_dir = $(HTML_DIR)/topics
87+
html_topics = $(topics_md:topics/%.md=$(html_topics_dir)/%.html)
88+
html_topics_pics = $(topics_pics:topics/%=$(html_topics_dir)/%)
89+
$(html_topics): | $(html_topics_dir)
90+
91+
html_commands_dir = $(HTML_DIR)/commands
92+
html_commands = $(commands:commands/%.md=$(html_commands_dir)/%.html)
93+
$(html_commands): | $(html_commands_dir)
94+
95+
html_targets = $(html_topics) $(html_topics_pics) $(html_commands) $(html_commands_dir)/index.html
96+
97+
$(html_topics_dir) $(html_commands_dir):
98+
mkdir -p $@
99+
100+
$(html_topics_dir)/%.html: topics/%.md utils/preprocess-markdown.py | $(html_topics_dir)
101+
utils/preprocess-markdown.py --suffix .html $< | pandoc -s --to html -o $@ -
102+
$(html_topics_dir)/%.png: topics/%.png | $(html_topics_dir)
103+
cp $< $@
104+
$(html_topics_dir)/%.gif: topics/%.gif | $(html_topics_dir)
105+
cp $< $@
106+
107+
$(html_commands_dir)/index.html: $(BUILD_DIR)/.commands-per-group.json groups.json utils/build-command-index.py | $(html_commands_dir)
108+
utils/build-command-index.py --suffix .html \
109+
--groups-json groups.json \
110+
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
111+
| pandoc -s --to html -o $@ -
112+
$(html_commands_dir)/%.html: commands/%.md $(VALKEY_ROOT)/src/commands/%.json $(BUILD_DIR)/.commands-per-group.json \
113+
utils/preprocess-markdown.py utils/command_syntax.py
114+
utils/preprocess-markdown.py --suffix .html --page-type command \
115+
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
116+
--valkey-root $(VALKEY_ROOT) $< \
117+
| pandoc -s --to html -o $@ -
118+
119+
# ---- Man pages (make man) ----
120+
121+
# Split topics into configs, programs and topics
122+
progs = valkey-cli valkey-server valkey-benchmark valkey-sentinel valkey-check-rdb valkey-check-aof
123+
programs = $(progs:valkey-%=topics/%.md)
124+
configs = topics/valkey.conf.md
125+
126+
man1_src = $(filter $(programs),$(topics_md))
127+
man3_src = $(commands)
128+
man4_src = $(filter $(configs),$(topics_md))
129+
man7_src = $(filter-out $(programs) $(configs) topics/index.md,$(topics_md))
130+
131+
# Target man pages
132+
man1 = $(man1_src:topics/%.md=$(MAN_DIR)/man1/valkey-%.1)
133+
man3 = $(man3_src:commands/%.md=$(MAN_DIR)/man3/%.3valkey)
134+
man4 = $(man4_src:topics/%.md=$(MAN_DIR)/man4/%.4)
135+
man7 = $(man7_src:topics/%.md=$(MAN_DIR)/man7/valkey-%.7) $(MAN_DIR)/man7/valkey-commands.7 $(MAN_DIR)/man7/valkey.7
136+
137+
man_targets = $(man1) $(man3) $(man4) $(man7)
138+
139+
$(man1): | $(MAN_DIR)/man1
140+
$(man3): | $(MAN_DIR)/man3
141+
$(man4): | $(MAN_DIR)/man4
142+
$(man7): | $(MAN_DIR)/man7
143+
$(MAN_DIR)/man1 $(MAN_DIR)/man3 $(MAN_DIR)/man4 $(MAN_DIR)/man7:
144+
mkdir -p $@
145+
146+
man_scripts = utils/preprocess-markdown.py utils/command_syntax.py utils/links-to-man.py
147+
148+
$(MAN_DIR)/man1/valkey-%.1: topics/%.md $(man_scripts)
149+
utils/preprocess-markdown.py --man --page-type program $< \
150+
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
151+
$(MAN_DIR)/man3/%.3valkey: commands/%.md $(VALKEY_ROOT)/src/commands/%.json $(BUILD_DIR)/.commands-per-group.json $(man_scripts)
152+
utils/preprocess-markdown.py --man --page-type command \
153+
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
154+
--valkey-root $(VALKEY_ROOT) $< \
155+
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
156+
$(MAN_DIR)/man4/%.4: topics/%.md $(man_scripts)
157+
utils/preprocess-markdown.py --man --page-type config $< \
158+
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
159+
$(MAN_DIR)/man7/valkey-%.7: topics/%.md $(man_scripts)
160+
utils/preprocess-markdown.py --man --page-type topic $< \
161+
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
162+
$(MAN_DIR)/man7/valkey.7: topics/index.md $(man_scripts)
163+
utils/preprocess-markdown.py --man --page-type topic $< \
164+
| utils/links-to-man.py - | pandoc -s --to man -o $@ -
165+
$(MAN_DIR)/man7/valkey-commands.7: $(BUILD_DIR)/.commands-per-group.json groups.json utils/build-command-index.py
166+
utils/build-command-index.py --man \
167+
--groups-json groups.json \
168+
--commands-per-group-json $(BUILD_DIR)/.commands-per-group.json \
169+
| pandoc -s --to man -o $@ -
170+
171+
.PHONY: install-man uninstall-man
172+
install-man: man | $(INSTALL_MAN_DIR)/man1 $(INSTALL_MAN_DIR)/man3 $(INSTALL_MAN_DIR)/man4 $(INSTALL_MAN_DIR)/man7
173+
cp $(MAN_DIR)/man1/valkey*.1 $(INSTALL_MAN_DIR)/man1/
174+
cp $(MAN_DIR)/man3/*.3valkey $(INSTALL_MAN_DIR)/man3/
175+
cp $(MAN_DIR)/man4/valkey*.4 $(INSTALL_MAN_DIR)/man4/
176+
cp $(MAN_DIR)/man7/valkey*.7 $(INSTALL_MAN_DIR)/man7/
177+
178+
$(INSTALL_MAN_DIR)/man1 $(INSTALL_MAN_DIR)/man3 $(INSTALL_MAN_DIR)/man4 $(INSTALL_MAN_DIR)/man7:
179+
mkdir -p $@
180+
181+
uninstall-man:
182+
rm $(INSTALL_MAN_DIR)/man1/valkey*.1
183+
rm $(INSTALL_MAN_DIR)/man3/*.3valkey
184+
rm $(INSTALL_MAN_DIR)/man4/valkey*.4
185+
rm $(INSTALL_MAN_DIR)/man7/valkey*.7
186+
187+
# -------
188+
189+
# All targets
190+
targets = $(man_targets) $(html_targets) $(md_targets)
191+
192+
md: $(md_targets)
193+
194+
html: $(html_targets)
195+
196+
man: $(man_targets)
197+
198+
clean:
199+
rm -f $(targets) $(BUILD_DIR)/.commands-per-group.json
200+
distclean:
201+
rm -rf $(BUILD_DIR)
202+
203+
install: install-man
204+
205+
uninstall: uninstall-man

‎README.md‎

Lines changed: 81 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,62 @@
11
# Valkey documentation
22

3-
## Clients
3+
This repo contains the Valkey documentation in Markdown format, which is used
4+
for generating content for the website and man pages.
45

5-
**Important**: Clients listed here, while fully compatible with Valkey, are not official clients for Valkey.
6+
## Installing man pages
7+
8+
This repo comes with a Makefile to build and install man pages.
9+
10+
make VALKEY_ROOT=path/to/valkey
11+
sudo make install INSTALL_MAN_DIR=/usr/local/share/man
12+
13+
Prerequisites: GNU Make, Python 3, Python 3 YAML (pyyaml), Pandoc.
14+
Additionally, the scripts need access to the valkey code repo,
15+
where metadata files about the commands are stored.
16+
17+
The pages are generated under `_build/man/` by default. The default install
18+
location is `/usr/local/share/man` (in the appropriate subdirectories).
19+
20+
To uninstall the man pages, run as root `make uninstall INSTALL_MAN_DIR=/usr/local/share/man`.
21+
22+
It's also possible to build local HTML files for local usage, using `make html`.
23+
These HTML files are generated under `_build/html/` by default. The starting
24+
point of the documentation is `topics/index.html`.
25+
26+
## Writing docs
27+
28+
The content of this doc repo is backing the documentation on the website, man
29+
pages and potentially other formats. Links between pages are *relative* and
30+
point directly to the `.md` files as they are stored in this repo. Don't start
31+
links with `/`. This makes sure the links point to existing files regardless of
32+
where in the file system the docs are located, which makes it easier to find
33+
broken links. In text editors and in the GitHub user inteface, it's possible to
34+
click on the links to open the corresponding Markdown page.
35+
36+
Examples: `../commands/get.md` or `replication.md`.
37+
38+
A few exceptions are links to the `topics/`, `commands/`, `clients/` and
39+
`modules/` directories, which end with a slash. These pages are generated (with
40+
the exception of `topics/` which is in `topics/index.md`).
41+
42+
Examples: `../commands/#sorted-set`, `../topics/`, `./`.
43+
44+
### Topics
45+
46+
The files under `topics/` are generic documentation pages. The `index.md` page is a starting point.
47+
48+
In the top of these files, there's a frontmatter metadata section, between two
49+
lines of three dashes (`---`). These are YAML fields of which we use only the
50+
`title` field (and possibly `linkTitle`). The title field is used instead of an
51+
H1 heading in each of the pages.
52+
53+
### Clients, modules, libraries, tools
54+
55+
We maintain links to clients, modules, libraries and tools in various langauges in
56+
JSON files stored under `clients/`, `modules/`, `libraries/` and `tools/`
57+
respectively.
58+
59+
**Note**: Clients listed here, while fully compatible with Valkey, are not all official clients for Valkey.
660
They are maintained by their original developers.
761

862
All clients are listed under language specific sub-folders of [clients](./clients)
@@ -11,40 +65,41 @@ The path follows the pattern: ``clients/{language}/github.com/{owner}/{repositor
1165
The ``{language}`` component of the path is the path-safe representation
1266
of the full language name which is mapped in [languages.json](./languages.json).
1367

14-
Each client's JSON object represents the details displayed on the [clients documentation page](https://valkey.io/docs/clients).
68+
Each client's JSON object represents the details displayed on the [clients documentation page](https://valkey.io/clients/).
1569

16-
For example [clients/python/github.com/placeholderkv](./clients/python/github.com/placeholderkv/placeholderkv-py.json):
70+
For example [clients/python/github.com/valkey-io/valkey-go.json](./clients/python/github.com/valkey-io/valkey-go.json):
1771

18-
```
72+
```json
1973
{
20-
"name": "placeholderkv-py",
21-
"description": "Mature and supported. Currently the way to go for Python.",
74+
"name": "valkey-go",
75+
"description": "A fast Golang Valkey client that supports Client Side Caching and Auto Pipelining.",
2276
"recommended": true
2377
}
2478
```
2579

26-
## Commands
80+
Modules, libraries and tools follow a similar structure under their respective directories.
2781

28-
Valkey commands are described in the `commands.json` file that is auto generated
29-
from the Valkey repo based on the JSON files in the commands folder.
30-
See: https://github.com/valkey-io/valkey/tree/unstable/src/commands
31-
See: https://github.com/valkey-io/valkey/tree/unstable/utils/generate-commands-json.py
82+
### Commands
83+
84+
The command pages under `commands/` in this repo are not complete without some
85+
metadata from the Valkey repo, namely the JSON files in the `src/commands/`
86+
folder. The content of these JSON files is combined with the Markdown files in
87+
this repo when the documentation is rendered.
88+
89+
See: https://github.com/valkey-io/valkey/tree/unstable/src/commands/
3290

3391
For each command there's a Markdown file with a complete, human-readable
3492
description.
35-
We process this Markdown to provide a better experience, so some things to take
36-
into account:
93+
We process these files to provide a better experience.
3794

3895
* Inside text, all commands should be written in all caps, in between
3996
backticks.
4097
For example: `INCR`.
4198

42-
* You can use some magic keywords to name common elements in Valkey.
43-
For example: `@multi-bulk-reply`.
44-
These keywords will get expanded and auto-linked to relevant parts of the
45-
documentation.
46-
99+
The reply types and descriptions are stored in separate JSON files in this doc repo.
47100
Each command will have a description and both RESP2 and RESP3 return values.
101+
When adding or editing return values, be sure to edit both files. Use the following
102+
links for the reply type.
48103
Regarding the return values, these are contained in the files:
49104

50105
* `resp2_replies.json`
@@ -65,41 +120,18 @@ when processed, produce Markdown content. Here's an example:
65120
}
66121
```
67122

68-
**Important**: when adding or editing return values, be sure to edit both files. Use the following
69-
links for the reply type. Note: do not use `@reply-type` specifiers; use only the Markdown link.
70-
71-
**Important**: all links below are under construction.
72-
73-
```md
74-
@simple-string-reply: [Simple string reply](https://valkey.io/topics/protocol#simple-strings)
75-
@simple-error-reply: [Simple error reply](https://valkey.io/topics/protocol#simple-errors)
76-
@integer-reply: [Integer reply](https://valkey.io/topics/protocol#integers)
77-
@bulk-string-reply: [Bulk string reply](https://valkey.io/topics/protocol#bulk-strings)
78-
@array-reply: [Array reply](https://valkey.io/topics/protocol#arrays)
79-
@nil-reply: [Nil reply](https://valkey.io/topics/protocol#bulk-strings)
80-
@null-reply: [Null reply](https://valkey.io/topics/protocol#nulls)
81-
@boolean-reply: [Boolean reply](https://valkey.io/topics/protocol#booleans)
82-
@double-reply: [Double reply](https://valkey.io/topics/protocol#doubles)
83-
@big-number-reply: [Big number reply](https://valkey.io/topics/protocol#big-numbers)
84-
@bulk-error-reply: [Bulk error reply](https://valkey.io/topics/protocol#bulk-errors)
85-
@verbatim-string-reply: [Verbatim string reply](https://valkey.io/topics/protocol#verbatim-strings)
86-
@map-reply: [Map reply](https://valkey.io/topics/protocol#maps)
87-
@set-reply: [Set reply](https://valkey.io/topics/protocol#sets)
88-
@push-reply: [Push reply](https://valkey.io/topics/protocol#pushes)
89-
```
90-
91-
## Styling guidelines
123+
### Styling guidelines
92124

93125
Please use the following formatting rules (aiming for smaller diffs that are easier to review):
94126

95-
* No need for manual lines wrapping at any specific length,
96-
doing so usually means that adding a word creates a cascade effect and changes other lines.
97-
* Please avoid writing lines that are too long,
98-
this makes the diff harder to review when only one word is changed.
127+
* Please avoid writing lines that are too long.
128+
That makes the diff harder to review when only one word is changed.
129+
* Single linebreaks are not significant in Markdown, so when editing an existing
130+
sentence or paragraph, don't change the existing linebreaks. That just makes
131+
reviewing harder.
99132
* Start every sentence on a new line.
100133

101-
102-
## Checking your work
134+
### Checking your work
103135

104136
After making changes to the documentation, you can use the [spellchecker-cli package](https://www.npmjs.com/package/spellchecker-cli)
105137
to validate your spelling as well as some minor grammatical errors. You can install the spellchecker locally by running:

0 commit comments

Comments
 (0)