Skip to content

Commit 160f12e

Browse files
jeffhostetlerdscho
authored andcommitted
status: add status serialization mechanism
Teach STATUS to optionally serialize the results of a status computation to a file. Teach STATUS to optionally read an existing serialization file and simply print the results, rather than actually scanning. This is intended for immediate status results on extremely large repos and assumes the use of a service/daemon to maintain a fresh current status snapshot. 2021-10-30: packet_read() changed its prototype in ec9a37d (pkt-line.[ch]: remove unused packet_read_line_buf(), 2021-10-14). 2021-10-30: sscanf() now does an extra check that "%d" goes into an "int" and complains about "uint32_t". Replacing with "%u" fixes the compile-time error. 2021-10-30: string_list_init() was removed by abf897b (string-list.[ch]: remove string_list_init() compatibility function, 2021-09-28), so we need to initialize manually. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
1 parent 17284f2 commit 160f12e

14 files changed

+1337
-4
lines changed

Documentation/config/status.txt

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,9 @@ status.submoduleSummary::
7575
the --ignore-submodules=dirty command-line option or the 'git
7676
submodule summary' command, which shows a similar output but does
7777
not honor these settings.
78+
79+
status.deserializePath::
80+
EXPERIMENTAL, Pathname to a file containing cached status results
81+
generated by `--serialize`. This will be overridden by
82+
`--deserialize=<path>` on the command line. If the cache file is
83+
invalid or stale, git will fall-back and compute status normally.

Documentation/git-status.txt

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,19 @@ ignored, then the directory is not shown, but all contents are shown.
149149
threshold.
150150
See also linkgit:git-diff[1] `--find-renames`.
151151

152+
--serialize[=<version>]::
153+
(EXPERIMENTAL) Serialize raw status results to stdout in a
154+
format suitable for use by `--deserialize`. Valid values for
155+
`<version>` are "1" and "v1".
156+
157+
--deserialize[=<path>]::
158+
(EXPERIMENTAL) Deserialize raw status results from a file or
159+
stdin rather than scanning the worktree. If `<path>` is omitted
160+
and `status.deserializePath` is unset, input is read from stdin.
161+
--no-deserialize::
162+
(EXPERIMENTAL) Disable implicit deserialization of status results
163+
from the value of `status.deserializePath`.
164+
152165
<pathspec>...::
153166
See the 'pathspec' entry in linkgit:gitglossary[7].
154167

@@ -421,6 +434,26 @@ quoted as explained for the configuration variable `core.quotePath`
421434
(see linkgit:git-config[1]).
422435

423436

437+
SERIALIZATION and DESERIALIZATION (EXPERIMENTAL)
438+
------------------------------------------------
439+
440+
The `--serialize` option allows git to cache the result of a
441+
possibly time-consuming status scan to a binary file. A local
442+
service/daemon watching file system events could use this to
443+
periodically pre-compute a fresh status result.
444+
445+
Interactive users could then use `--deserialize` to simply
446+
(and immediately) print the last-known-good result without
447+
waiting for the status scan.
448+
449+
The binary serialization file format includes some worktree state
450+
information allowing `--deserialize` to reject the cached data
451+
and force a normal status scan if, for example, the commit, branch,
452+
or status modes/options change. The format cannot, however, indicate
453+
when the cached data is otherwise stale -- that coordination belongs
454+
to the task driving the serializations.
455+
456+
424457
CONFIGURATION
425458
-------------
426459

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
Git status serialization format
2+
===============================
3+
4+
Git status serialization enables git to dump the results of a status scan
5+
to a binary file. This file can then be loaded by later status invocations
6+
to print the cached status results.
7+
8+
The file contains the essential fields from:
9+
() the index
10+
() the "struct wt_status" for the overall results
11+
() the contents of "struct wt_status_change_data" for tracked changed files
12+
() the list of untracked and ignored files
13+
14+
Version 1 Format:
15+
=================
16+
17+
The V1 file begins with a required header section followed by optional
18+
sections for each type of item (changed, untracked, ignored). Individual
19+
item sections are only present if necessary. Each item section begins
20+
with an item-type header with the number of items in the section.
21+
22+
Each "line" in the format is encoded using pkt-line with a final LF.
23+
Flush packets are used to terminate sections.
24+
25+
-----------------
26+
PKT-LINE("version" SP "1")
27+
<v1-header-section>
28+
[<v1-changed-item-section>]
29+
[<v1-untracked-item-section>]
30+
[<v1-ignored-item-section>]
31+
-----------------
32+
33+
34+
V1 Header
35+
---------
36+
37+
The v1-header-section fields are taken directly from "struct wt_status".
38+
Each field is printed on a separate pkt-line. Lines for NULL string
39+
values are omitted. All integers are printed with "%d". OIDs are
40+
printed in hex.
41+
42+
v1-header-section = <v1-index-headers>
43+
<v1-wt-status-headers>
44+
PKT-LINE(<flush>)
45+
46+
v1-index-headers = PKT-LINE("index_mtime" SP <sec> SP <nsec> LF)
47+
48+
v1-wt-status-headers = PKT-LINE("is_initial" SP <integer> LF)
49+
[ PKT-LINE("branch" SP <branch-name> LF) ]
50+
[ PKT-LINE("reference" SP <reference-name> LF) ]
51+
PKT-LINE("show_ignored_files" SP <integer> LF)
52+
PKT-LINE("show_untracked_files" SP <integer> LF)
53+
PKT-LINE("show_ignored_directory" SP <integer> LF)
54+
[ PKT-LINE("ignore_submodule_arg" SP <string> LF) ]
55+
PKT-LINE("detect_rename" SP <integer> LF)
56+
PKT-LINE("rename_score" SP <integer> LF)
57+
PKT-LINE("rename_limit" SP <integer> LF)
58+
PKT-LINE("detect_break" SP <integer> LF)
59+
PKT-LINE("sha1_commit" SP <oid> LF)
60+
PKT-LINE("committable" SP <integer> LF)
61+
PKT-LINE("workdir_dirty" SP <integer> LF)
62+
63+
64+
V1 Changed Items
65+
----------------
66+
67+
The v1-changed-item-section lists all of the changed items with one
68+
item per pkt-line. Each pkt-line contains: a binary block of data
69+
from "struct wt_status_serialize_data_fixed" in a fixed header where
70+
integers are in network byte order and OIDs are in raw (non-hex) form.
71+
This is followed by one or two raw pathnames (not c-quoted) with NUL
72+
terminators (both NULs are always present even if there is no rename).
73+
74+
v1-changed-item-section = PKT-LINE("changed" SP <count> LF)
75+
[ PKT-LINE(<changed_item> LF) ]+
76+
PKT-LINE(<flush>)
77+
78+
changed_item = <byte[4] worktree_status>
79+
<byte[4] index_status>
80+
<byte[4] stagemask>
81+
<byte[4] score>
82+
<byte[4] mode_head>
83+
<byte[4] mode_index>
84+
<byte[4] mode_worktree>
85+
<byte[4] dirty_submodule>
86+
<byte[4] new_submodule_commits>
87+
<byte[20] oid_head>
88+
<byte[20] oid_index>
89+
<byte[*] path>
90+
NUL
91+
[ <byte[*] src_path> ]
92+
NUL
93+
94+
95+
V1 Untracked and Ignored Items
96+
------------------------------
97+
98+
These sections are simple lists of pathnames. They ARE NOT
99+
c-quoted.
100+
101+
v1-untracked-item-section = PKT-LINE("untracked" SP <count> LF)
102+
[ PKT-LINE(<pathname> LF) ]+
103+
PKT-LINE(<flush>)
104+
105+
v1-ignored-item-section = PKT-LINE("ignored" SP <count> LF)
106+
[ PKT-LINE(<pathname> LF) ]+
107+
PKT-LINE(<flush>)

Makefile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1201,6 +1201,8 @@ LIB_OBJS += wrapper.o
12011201
LIB_OBJS += write-or-die.o
12021202
LIB_OBJS += ws.o
12031203
LIB_OBJS += wt-status.o
1204+
LIB_OBJS += wt-status-deserialize.o
1205+
LIB_OBJS += wt-status-serialize.o
12041206
LIB_OBJS += xdiff-interface.o
12051207

12061208
BUILTIN_OBJS += builtin/add.o

builtin/commit.c

Lines changed: 122 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,70 @@ static int opt_parse_porcelain(const struct option *opt, const char *arg, int un
175175
return 0;
176176
}
177177

178+
static int do_serialize = 0;
179+
static int do_implicit_deserialize = 0;
180+
static int do_explicit_deserialize = 0;
181+
static char *deserialize_path = NULL;
182+
183+
/*
184+
* --serialize | --serialize=1 | --serialize=v1
185+
*
186+
* Request that we serialize our output rather than printing in
187+
* any of the established formats. Optionally specify serialization
188+
* version.
189+
*/
190+
static int opt_parse_serialize(const struct option *opt, const char *arg, int unset)
191+
{
192+
enum wt_status_format *value = (enum wt_status_format *)opt->value;
193+
if (unset || !arg)
194+
*value = STATUS_FORMAT_SERIALIZE_V1;
195+
else if (!strcmp(arg, "v1") || !strcmp(arg, "1"))
196+
*value = STATUS_FORMAT_SERIALIZE_V1;
197+
else
198+
die("unsupported serialize version '%s'", arg);
199+
200+
if (do_explicit_deserialize)
201+
die("cannot mix --serialize and --deserialize");
202+
do_implicit_deserialize = 0;
203+
204+
do_serialize = 1;
205+
return 0;
206+
}
207+
208+
/*
209+
* --deserialize | --deserialize=<path> |
210+
* --no-deserialize
211+
*
212+
* Request that we deserialize status data from some existing resource
213+
* rather than performing a status scan.
214+
*
215+
* The input source can come from stdin or a path given here -- or be
216+
* inherited from the config settings.
217+
*/
218+
static int opt_parse_deserialize(const struct option *opt, const char *arg, int unset)
219+
{
220+
if (unset) {
221+
do_implicit_deserialize = 0;
222+
do_explicit_deserialize = 0;
223+
} else {
224+
if (do_serialize)
225+
die("cannot mix --serialize and --deserialize");
226+
if (arg) {
227+
/* override config or stdin */
228+
free(deserialize_path);
229+
deserialize_path = xstrdup(arg);
230+
}
231+
if (deserialize_path && *deserialize_path
232+
&& (access(deserialize_path, R_OK) != 0))
233+
die("cannot find serialization file '%s'",
234+
deserialize_path);
235+
236+
do_explicit_deserialize = 1;
237+
}
238+
239+
return 0;
240+
}
241+
178242
static int opt_parse_m(const struct option *opt, const char *arg, int unset)
179243
{
180244
struct strbuf *buf = opt->value;
@@ -1176,6 +1240,8 @@ static void handle_untracked_files_arg(struct wt_status *s)
11761240
s->show_untracked_files = SHOW_NORMAL_UNTRACKED_FILES;
11771241
else if (!strcmp(untracked_files_arg, "all"))
11781242
s->show_untracked_files = SHOW_ALL_UNTRACKED_FILES;
1243+
else if (!strcmp(untracked_files_arg,"complete"))
1244+
s->show_untracked_files = SHOW_COMPLETE_UNTRACKED_FILES;
11791245
/*
11801246
* Please update $__git_untracked_file_modes in
11811247
* git-completion.bash when you add new options
@@ -1463,6 +1529,19 @@ static int git_status_config(const char *k, const char *v,
14631529
s->relative_paths = git_config_bool(k, v);
14641530
return 0;
14651531
}
1532+
if (!strcmp(k, "status.deserializepath")) {
1533+
/*
1534+
* Automatically assume deserialization if this is
1535+
* set in the config and the file exists. Do not
1536+
* complain if the file does not exist, because we
1537+
* silently fall back to normal mode.
1538+
*/
1539+
if (v && *v && access(v, R_OK) == 0) {
1540+
do_implicit_deserialize = 1;
1541+
deserialize_path = xstrdup(v);
1542+
}
1543+
return 0;
1544+
}
14661545
if (!strcmp(k, "status.showuntrackedfiles")) {
14671546
if (!v)
14681547
return config_error_nonbool(k);
@@ -1503,7 +1582,8 @@ int cmd_status(int argc, const char **argv, const char *prefix)
15031582
static const char *rename_score_arg = (const char *)-1;
15041583
static struct wt_status s;
15051584
unsigned int progress_flag = 0;
1506-
int fd;
1585+
int try_deserialize;
1586+
int fd = -1;
15071587
struct object_id oid;
15081588
static struct option builtin_status_options[] = {
15091589
OPT__VERBOSE(&verbose, N_("be verbose")),
@@ -1518,6 +1598,12 @@ int cmd_status(int argc, const char **argv, const char *prefix)
15181598
OPT_CALLBACK_F(0, "porcelain", &status_format,
15191599
N_("version"), N_("machine-readable output"),
15201600
PARSE_OPT_OPTARG, opt_parse_porcelain),
1601+
{ OPTION_CALLBACK, 0, "serialize", &status_format,
1602+
N_("version"), N_("serialize raw status data to stdout"),
1603+
PARSE_OPT_OPTARG | PARSE_OPT_NONEG, opt_parse_serialize },
1604+
{ OPTION_CALLBACK, 0, "deserialize", NULL,
1605+
N_("path"), N_("deserialize raw status data from file"),
1606+
PARSE_OPT_OPTARG, opt_parse_deserialize },
15211607
OPT_SET_INT(0, "long", &status_format,
15221608
N_("show status in long format (default)"),
15231609
STATUS_FORMAT_LONG),
@@ -1562,10 +1648,26 @@ int cmd_status(int argc, const char **argv, const char *prefix)
15621648
s.show_untracked_files == SHOW_NO_UNTRACKED_FILES)
15631649
die(_("Unsupported combination of ignored and untracked-files arguments"));
15641650

1651+
if (s.show_untracked_files == SHOW_COMPLETE_UNTRACKED_FILES &&
1652+
s.show_ignored_mode == SHOW_NO_IGNORED)
1653+
die(_("Complete Untracked only supported with ignored files"));
1654+
15651655
parse_pathspec(&s.pathspec, 0,
15661656
PATHSPEC_PREFER_FULL,
15671657
prefix, argv);
15681658

1659+
/*
1660+
* If we want to try to deserialize status data from a cache file,
1661+
* we need to re-order the initialization code. The problem is that
1662+
* this makes for a very nasty diff and causes merge conflicts as we
1663+
* carry it forward. And it easy to mess up the merge, so we
1664+
* duplicate some code here to hopefully reduce conflicts.
1665+
*/
1666+
try_deserialize = (!do_serialize &&
1667+
(do_implicit_deserialize || do_explicit_deserialize));
1668+
if (try_deserialize)
1669+
goto skip_init;
1670+
15691671
enable_fscache(0);
15701672
if (status_format != STATUS_FORMAT_PORCELAIN &&
15711673
status_format != STATUS_FORMAT_PORCELAIN_V2)
@@ -1580,6 +1682,7 @@ int cmd_status(int argc, const char **argv, const char *prefix)
15801682
else
15811683
fd = -1;
15821684

1685+
skip_init:
15831686
s.is_initial = repo_get_oid(the_repository, s.reference, &oid) ? 1 : 0;
15841687
if (!s.is_initial)
15851688
oidcpy(&s.oid_commit, &oid);
@@ -1596,6 +1699,24 @@ int cmd_status(int argc, const char **argv, const char *prefix)
15961699
s.rename_score = parse_rename_score(&rename_score_arg);
15971700
}
15981701

1702+
if (try_deserialize) {
1703+
if (s.relative_paths)
1704+
s.prefix = prefix;
1705+
1706+
if (wt_status_deserialize(&s, deserialize_path) == DESERIALIZE_OK)
1707+
return 0;
1708+
1709+
/* deserialize failed, so force the initialization we skipped above. */
1710+
enable_fscache(1);
1711+
repo_read_index_preload(the_repository, &s.pathspec, 0);
1712+
refresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &s.pathspec, NULL, NULL);
1713+
1714+
if (use_optional_locks())
1715+
fd = repo_hold_locked_index(the_repository, &index_lock, 0);
1716+
else
1717+
fd = -1;
1718+
}
1719+
15991720
wt_status_collect(&s);
16001721

16011722
if (0 <= fd)

contrib/completion/git-completion.bash

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1675,7 +1675,7 @@ _git_clone ()
16751675
esac
16761676
}
16771677

1678-
__git_untracked_file_modes="all no normal"
1678+
__git_untracked_file_modes="all no normal complete"
16791679

16801680
_git_commit ()
16811681
{

pkt-line.c

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -230,7 +230,7 @@ static int do_packet_write(const int fd_out, const char *buf, size_t size,
230230
return 0;
231231
}
232232

233-
static int packet_write_gently(const int fd_out, const char *buf, size_t size)
233+
int packet_write_gently(const int fd_out, const char *buf, size_t size)
234234
{
235235
struct strbuf err = STRBUF_INIT;
236236
if (do_packet_write(fd_out, buf, size, &err)) {

pkt-line.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ void packet_write(int fd_out, const char *buf, size_t size);
3030
void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
3131
int packet_flush_gently(int fd);
3232
int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
33+
int packet_write_gently(const int fd_out, const char *buf, size_t size);
3334
int write_packetized_from_fd_no_flush(int fd_in, int fd_out);
3435
int write_packetized_from_buf_no_flush_count(const char *src_in, size_t len,
3536
int fd_out, int *packet_counter);

0 commit comments

Comments
 (0)