"Widdle-to" questions, and related feature suggestions

[bwbug]

Very curious about your new --widdle-to option. First of all, as a point of nomenclature, did you perhaps mean to call this "whittle" rather than "widdle"?

Second, even after reading the help text, I'm unclear on what exactly this option does, what some typical use case would be, and how the iteration process works. Perhaps you can explain by way of example?

Third, even before seeing this new option, I had planned to suggest something that is probably not directly related, but may share some similarities, insofar as it is another way of paring down the input list. What I have in mind is simply stripping headers and footers (of known length) from the input file. Some word list files (especially those that contain metadata) may have one ore more rows with header information (e.g., column titles). Other word list files may include a footer (for example, for a PGP signature). To allow tidy to use such lists without first preprocessing the files with a separate app that can strip out the rows that do not contain words, I would suggest that you implement an option with a syntax like --strip <HEADER_ROWS>,<FOOTER_ROWS> (or if you prefer, two complementary options --strip-first <HEADER_ROWS> and --strip-last <FOOTER_ROWS>) to remove the specified number of header rows and/or footer rows from the file when it is first read in.

To keep things simple, if the <Inputted Word Lists> argument includes more than one file, then the stripping of header and footer rows would be done identically to each of the input files.

[sts10]

did you perhaps mean to call this "whittle" rather than "widdle"?

haha omg how embarrassing. Fixed!

Second, even after reading the help text, I'm unclear on what exactly this option does, what some typical use case would be, and how the iteration process works. Perhaps you can explain by way of example?

This is definitely a niche thing, but I'll go over my particular use-case. Maybe you can help me re-write the help text (and/or add a paragraph to the readme).

Let's say we want to make a diceware list from the most frequently used words in (English) Wikipedia. First, let's download this enwiki-20210820-words-frequency.txt file. Note that this file is sorted by word frequency, and contains word frequency metadata after a space.

davis 129087
publications 128959
pair 128899
churches 128784
wins 128738
applications 128639

Next, let's write a Tidy command. Since we want this to be a diceware list for 5 6-sided dice, we need the list to be exactly 7,776 words. Let's say we also want to remove suffix words, set a minimum and max word length, exclude non-Latin words, lowercase all words, and filter out some profane words.

A naive approach

tidy -AA --cut-to 7776 -SlL -d s -m 3 -M 11 -r ~/lists/profane-words.txt --dice 6 -o wiki-diceware.txt ~/code/wikipedia-word-frequency/results/enwiki-20210820-words-frequency.txt

While this will produce a 7,776-word list from the Wikipedia data, it does not take advantage of the fact that the Wikipedia list is sorted by word frequency. To see why, let's summarize what this command does. It takes all 2,676,542 words on the Wikipedia list, does its various filtering and word modifications, then, thanks to --cut-to 7776, it randomly takes 7,776 words from the list to make its outputted list.

This is no good because it's just as likely to take a word from the way bottom of the list (a very infrequently used word, like amsanctus) as it is from the top. (And actually, since it has to operate on 2 million words, our Tidy command, as composed, takes over 9 minutes on my laptop.)

A workable approach

You might be saying to yourself "That's what the --take-first option is for!" And yes, it is! Let's see how we can use it.

tidy -AA --take-first 13000 --cut-to 7776 -SlL -d s -m 3 -M 11 -r ~/lists/profane-words.txt --dice 6 -o wiki-diceware.txt ~/code/wikipedia-word-frequency/results/enwiki-20210820-words-frequency.txt

Better! We get a word list with 7,776 words.

But... what if we could take just the first 12,000 words? That'd be better, since we wouldn't be including some less frequently words like "minsk" and "boone". Could we get away with 11,900? 11,800?

Optimizing

To find out, we could remove the --cut-to option and keep lowering and raising the --take-first value until we get 7,776 words exactly. But that's a lot of manual guess-and-check. And if you made any changes to your options or you reject/approve lists, you'd have to start over.

Whittling

Instead, we could use Tidy's brand-new, totally correctly spelled --whittle-to feature! Whittle does this guess work for us. We just run:

tidy -AA --whittle-to 7776 -SlL -d s -m 3 -M 11 -r ~/lists/profane-words.txt --dice 6 --force -o wiki-diceware.txt ~/code/wikipedia-word-frequency/results/enwiki-20210820-words-frequency.txt

Tidy will make an initial "guess" of a --take-first value (arbitrarily defaults to 7776*1.4), create the resulting list in memory, then compare this resulting list's length to the desired amount. If it's too long (over 7,776, in this case), it takes fewer words and runs again. If it's too short, it takes more and runs again. It'll run over and over again until it finds an initial --take-first amount that produces the exact number of words the user specified¹. Then that list is outputted to the specified output location.

I argue there are other cases when you'd want a resulting list to be exactly a specified length, but diceware lists is the clearest and/or most common I can think of. Another would be if user wanted just barely clear the brute force or Shannon line.

¹ Whether there are cases in which this will never happen is an open question...

[sts10]

What I have in mind is simply stripping headers and footers (of known length) from the input file. Some word list files (especially those that contain metadata) may have one ore more rows with header information (e.g., column titles). Other word list files may include a footer (for example, for a PGP signature). To allow tidy to use such lists without first preprocessing the files with a separate app that can strip out the rows that do not contain words...

Re: the stripping header/footer lines of metadata: I'm tempted to say this is out-of-scope. Unless you're getting an updated version of this source file every few seconds/minutes/hours, in this case I don't think it's a huge deal to pre-process header and/or footer lines. I'm proud of the --ignore-after/--ignore-before options, since that's handling per line metadata, which is more of a pain to re-apply accurately after processing.

By contrast, header and footer information is trivial to re-add after processing. Plus, you'd likely want to remove PGP header/footer information permanently, since the signature would no longer be valid after any change to the list.

(I'll note here that if we ever #12 working, users could do this header-footing stripping themselves -- probably with some nifty cat command -- and we wouldn't need a new Tidy option.)

[bwbug]

Thank you for the explanation. But I might still be a bit confused. Let me explain what I think I understand, and you can correct me where I'm wrong:

I believe that --take-first and --take-first [ed.: oops, meant -take-rand for the second one] operate directly on the unmodified input word list. In contrast, --cut-to is the equivalent of --take-rand, except that it operates on the final, modified word list. Now you're adding --whittle-to, which (correct me if I'm wrong!) is the equivalent of --take-first, except that it operates on the final, modified word list.

Is this right?* If so, you may want to name the --take-first, --take-first, --cut-to, and --whittle-to options in a way that will highlight the similarities between the last two, and the differences between "take" vs. "cut"/"whittle" (similar to the symmetric naming scheme for --delete-after, --delete-before, --ignore-after, and --ignore-before).

[*and if my understanding of --whittle-to is not correct, would my concept be a faster approach to addressing the use-case you described above??]

[sts10]

I believe that --take-first and [--take-rand] operate directly on the unmodified input word list.

Yes, correct. (Think you meant "rand".) We can think of these as "pre-process" options.

In contrast, --cut-to is the equivalent of --take-rand, except that it operates on the final, modified word list.

Also correct. We can think of --cut-to as a "post-process" option.

Now you're adding --whittle-to, which (correct me if I'm wrong!) is the equivalent of --take-first, except that it operates on the final, modified word list.

Not quite. --whittle-to effectively operates on both the pre-process side and the post-process side. It is essentially a guess-and-check loop. It guesses a --take-first then runs through what we'll call "the tidying process" and sees how long the resulting list would be. If this list is not the desired length, Whittle makes a different --take-first guess and runs the tidying process all over again.

I like that both the pre-process options are called "take". They can be thought of as siblings. In contrast, given how Whittle works on both sides and employs a loop, it's unique, and thus I don't think it's a "sibling" of the --cut-to option (which is strictly a post-process option). As tempting as it it to want the symmetry between these 4 options, it's just not quite there.

I chose "whittle" because it's a process that takes small chunks out until it reaches the desired destination. "Cut" is more abrupt and non-discerning. If we wanted to emphasize the random nature of --cut-to, I suppose we could rename it --cut-randomly-to or --randomly-cut-to, but that seems unnecessary, especially since we have "random" in the help text. So I like the naming scheme as is.

Why can't we just keep the first 7,776 words from the post-processed list?

I think this is the question you're asking here:

[*and if my understanding of --whittle-to is not correct, would my concept be a faster approach to addressing the use-case you described above??]

Or in other words:

Why can't we do all the processing on the entire input list (without ever sorting it alphabetically), then, right before printing the list, take the first 7,776 words to make a good diceware list? Wouldn't that be easier and faster?

The answer to this has to do with how prefix words, suffix words, and Schlinkert pruning work. If we removed all prefix words from the entire inputted list, then made cuts, we'd lose "good" (high word frequency) words unnecessarily. I believe that in order to do the cuts optimally, we need to make all the unrelated cuts first (like reject words, non-Latin words, etc.), then cut prefix/suffix words. That's why I went to all the trouble of making whittle-to do the looping.

You might then say: That's all fine and good, but to the end-user, whittle-to is basically a post-process option, and should be named as such.

I think I disagree with this. Whittle to is more complex than a post-process "cut" option, and I think it should have a unique name to signify that.

[bwbug]

The answer to this has to do with how prefix words, suffix words, and Schlinkert pruning work. If we removed all prefix words from the entire inputted list, then made cuts, we'd lose "good" (high word frequency) words unnecessarily. I believe that in order to do the cuts optimally, we need to make all the unrelated cuts first (like reject words, non-Latin words, etc.), then cut prefix/suffix words. That's why I went to all the trouble of making whittle-to do the looping.

OK, now I understand what you're doing and why! And in this case, I agree that your whittle operation is not a direct relative to the --cut-to option.

Now, what about a use-case in which a user is creating word lists for passphrases that will be required to use separators, and therefore will not use any of the options that prune based on prefixes, suffixes, or unique decodability? For those users, could you implement a non-random --cut-to "sibling" option (say, --cut-to-first), which takes the first 7,776 (or whatever specified value) words just prior to sorting and final output? This would be faster way of getting the benefits of a rank-sorted list as the input if one is not concerned with word overlaps. And if a user simultaneously specifies --cut-to-first with -P, -K, or -S, they could get a warning message telling them to use --whittle-to instead of --cut-to-first.

What do you think?

[sts10]

Slept on this and came up with: renaming cut-to as print-rand, and creating a brand new print-first. I like "print" to help denote it happens at the end. And adding print-first gives us the symmetry with take-first.

I left whittle-to as is. It's the odd cousin!

And if a user simultaneously specifies --cut-to-first with -P, -K, or -S, they could get a warning message telling them to use --whittle-to instead of --cut-to-first.

Neat idea. I've added a "Recommendation" for this niche case.

[bwbug]

creating a brand new print-first

Great! Just making sure that the cut is made prior to sorting, if the alphabetic sort option is left on (--no-sort false). This behavior isn't quite captured in the word "print" (which is not a major issue, unless you're bothered by it).

[sts10]

Just making sure that the cut is made prior to sorting, if the alphabetic sort option is left on (--no-sort false).

Whoops, will fix in next push. Great question/catch. I should write a test for this at some point...

[sts10]

What I have in mind is simply stripping headers and footers (of known length) from the input file. Some word list files (especially those that contain metadata) may have one ore more rows with header information (e.g., column titles). Other word list files may include a footer (for example, for a PGP signature). To allow tidy to use such lists without first preprocessing the files with a separate app that can strip out the rows that do not contain words...

I reconsidered this. Just pushed --read-line-start and --read-line-end options. Might have some questions about how they should be indexed...

[bwbug]

I reconsidered this. Working on --read-line-start and --read-line-end options. Might have some question about how they should be indexed...

I would highly recommend you change this to --skip-rows-start and -skip-rows-end (or --strip-header and --strip-footer, etc.).

I'm not arguing nomenclature here, my point is that it will be much easier to use this feature if the specified arguments represent the number of rows to trim from the top or bottom of the file, instead of (as I think you are planning to do with your --read-line-start and --read-line-end options) having the arguments represent an index of the first and/or last row to read in.

I'll expound on the reasoning for my preference once you confirm your intent for the --read-line-start and --read-line-end options (i.e., that the arguments would represent row-indices), and that you still prefer this over specifying the number of rows to omit.

[sts10]

Haha, ok, let me get --skip-rows-start and -skip-rows-end working and I'll go from there.

[bwbug]

Great, glad to hear it! If you still want to include some functionality more akin to --read-line-start and --read-line-end, perhaps the easiest thing to do would be to allow for an optional two-argument form of the --take-first option. For example, you could let --take-first 10000,5 be interpreted as "read the first 10,000 rows, starting with the 5th row."

[sts10]

If you still want to include some functionality more akin to --read-line-start and --read-line-end, perhaps the easiest thing to do would be to allow for an optional two-argument form of the --take-first option. For example, you could let --take-first 10000,5 be interpreted as "read the first 10,000 rows, starting with the 5th row."

Nah, think you're write that "skip" is the better verb and functionality, since it's easier to count the number of lines in a footer than to get the line number of the file where the footer starts.

I've just pushed v 0.2.65 to the repo, which includes

  --skip-rows-end <SKIP_ROWS_END>
            Skip last number of lines from inputted files. Useful for dealing with footers like from
            PGP signatures

  --skip-rows-start <SKIP_ROWS_START>
            Skip first number of lines from inputted files. Useful for dealing with headers like
            from PGP signatures

As well as print-rand, print-first.