Skip to content

Latest commit

 

History

History
962 lines (754 loc) · 29.9 KB

README.template.org

File metadata and controls

962 lines (754 loc) · 29.9 KB
Raw

Talk

I just gave a talk about this at SCaLE 17x. Here are the video of the talk and the “slides”.

Summary

Vnlog (pronounced “vanillog”) is a toolkit for manipulating tabular ASCII data with labelled fields using normal UNIX tools. The log format is trivially simple:

  • A whitespace-separated table of ASCII human-readable text
  • Lines beginning with # are comments
  • The first line that begins with a single # (not ## or #!) is a legend, naming each column. This is required, and the field names that appear here are referenced by all the tools.
  • Empty fields reported as -

Example:

#!/usr/bin/whatever
# a b c
1 2 3
## comment
4 5 6

Such data can be processed directly with almost any existing tool, and this toolkit allows the user to manipulate this data in a nicer way by relying on standard UNIX tools. The core philosophy is to avoid creating new knowledge as much as possible. Consequently, the vnlog toolkit relies heavily on existing (and familiar!) tools and workflows. As such, the toolkit is small, light, and has a very friendly learning curve.

Synopsis

I have two sets of historical stock data, from the start of 2018 until now (2018/11):

< dji.vnl head -n 4
# Date Open High Low Close AdjClose Volume
2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840
2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000
2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000

And

< tsla.vnl head -n 4
# Date Open High Low Close AdjClose Volume
2018-11-15 342.33 348.58 339.04 348.44 348.44 4486339
2018-11-14 342.70 347.11 337.15 344.00 344.00 5036300
2018-11-13 333.16 344.70 332.20 338.73 338.73 5448600

I can pull out the closing prices:

< dji.vnl vnl-filter -p Close | head -n4
# Close
25289.27
25080.50
25286.49

vnl-filter is primarily a wrapper around awk or perl, allowing the user to reference columns by name. I can then plot the closing prices:

< dji.vnl vnl-filter -p Close |
  feedgnuplot --lines --unset grid

guide-1.svg

Here I kept only the closing price column, so the x-axis is just the row index. The data was in reverse chronological order, so this plot is also in reverse chronological order. Let’s fix that:

< dji.vnl vnl-sort -k Date |
  vnl-filter -p Close |
  feedgnuplot --lines --unset grid

guide-2.svg

The vnl-sort tool (and most of the other vnl-xxx tools) are wrappers around the core tools already available on the system (such as sort, in this case). With the primary difference being reading/writing vnlog, and referring to columns by name.

We now have the data in the correct order, but it’d be nice to see the actual dates on the x-axis. While we’re at it, let’s label the axes too:

< dji.vnl vnl-filter -p Date,Close | head -n4
# Date Close
2018-11-15 25289.27
2018-11-14 25080.50
2018-11-13 25286.49

< dji.vnl vnl-sort -k Date |
  vnl-filter -p Date,Close |
  feedgnuplot --lines --unset grid --timefmt %Y-%m-%d --domain \
              --xlabel 'Date' --ylabel 'Price ($)'

guide-3.svg

What was the highest value of the Dow-Jones index, and when did it happen?

< dji.vnl vnl-sort -rgk Close |
  head -n2 |
  vnl-align
#  Date      Open     High      Low     Close  AdjClose   Volume 
2018-10-03 26833.47 26951.81 26789.08 26828.39 26828.39 280130000

Alrighty. Looks like the high was in October. Let’s zoom in on that month:

< dji.vnl vnl-sort -k Date |
  vnl-filter 'Date ~ /2018-10/' -p Date,Close |
  feedgnuplot --lines --unset grid --timefmt %Y-%m-%d --domain \
              --xlabel 'Date' --ylabel 'Price ($)'

guide-4.svg

OK. Is this thing volatile? What was the largest single-day gain?

< dji.vnl vnl-filter -p '.,d=diff(Close)' |
  head -n4 |
  vnl-align
#  Date      Open     High      Low     Close  AdjClose   Volume     d   
2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 -      
2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 -208.77
2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000  205.99

< dji.vnl vnl-filter -p '.,d=diff(Close)' |
  vnl-sort -rgk d |
  head -n2 |
  vnl-align
#  Date      Open     High      Low     Close  AdjClose   Volume     d   
2018-02-02 26061.79 26061.79 25490.66 25520.96 25520.96 522880000 1175.21

Whoa. So the best single-gain day was 2018-02-02: the dow gained 1175.21 points between closing on Feb 1 and Feb 2. But it actually lost ground that day! What if I looked at the difference between the opening and closing in a single day?

< dji.vnl vnl-filter -p '.,d=Close-Open' |
  vnl-sort -rgk d |
  head -n2 |
  vnl-align
#  Date      Open     High      Low     Close  AdjClose   Volume    d  
2018-02-06 24085.17 24946.23 23778.74 24912.77 24912.77 823940000 827.6

I guess by that metric 2018-02-06 was better. Let’s join the Dow-jones index data and the TSLA data, and let’s look at them together:

vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  head -n4 |
  vnl-align
#  Date    Open_dji High_dji  Low_dji Close_dji AdjClose_dji Volume_dji Open_tsla High_tsla Low_tsla Close_tsla AdjClose_tsla Volume_tsla
2018-11-15 25061.48 25354.56 24787.79 25289.27  25289.27     383292840  342.33    348.58    339.04   348.44     348.44        4486339    
2018-11-14 25388.08 25501.29 24935.82 25080.50  25080.50     384240000  342.70    347.11    337.15   344.00     344.00        5036300    
2018-11-13 25321.21 25511.03 25193.78 25286.49  25286.49     339690000  333.16    344.70    332.20   338.73     338.73        5448600    

vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  vnl-filter -p '^Close' |
  head -n4 |
  vnl-align
# Close_dji Close_tsla
25289.27    348.44    
25080.50    344.00    
25286.49    338.73    

vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  vnl-filter -p '^Close' |
  feedgnuplot --domain --points --unset grid \
              --xlabel 'DJI price ($)' --ylabel 'TSLA price ($)'

guide-5.svg

Huh. Apparently there’s no obvious, strong correlation between TSLA and Dow-Jones closing prices. And we saw that with just a few shell commands, without dropping down into a dedicated analysis system.

Description

Vnlog data is nicely readable by both humans and machines. Any time your application invokes printf() for either diagnostics or logging, consider writing out vnlog-formatted data. You retain human readability, but gain the power all the vnl-... tools provide.

Vnlog tools are designed to be very simple and light. There’s an ever-growing list of other tools that do vaguely the same thing. Some of these:

Many of these provide facilities to run various analyses, and others focus on data types that aren’t just a table (json for instance). Vnlog by contrast doesn’t analyze anything, and targets the most trivial possible data format. This makes it very easy to run any analysis you like in any tool you like. The main envisioned use case is one-liners, and the tools are geared for that purpose. The above mentioned tools are much more powerful than vnlog, so they could be a better fit for some use cases. I claim that

  • 90% of the time you want to do simple things, and vnlog is a great fit for the task
  • If you really do need to do something complex, you shouldn’t be in the shell writing oneliners anymore, and a fully-fledged analysis system (numpy, etc) is more appropriate

In the spirit of doing as little as possible, the provided tools are wrappers around tools you already have and are familiar with. The provided tools are:

  • vnl-filter is a tool to select a subset of the rows/columns in a vnlog and/or to manipulate the contents. This is an awk wrapper where the fields can be referenced by name instead of index. 20-second tutorial:
vnl-filter -p col1,col2,colx=col3+col4 'col5 > 10' --has col6

will read the input, and produce a vnlog with 3 columns: col1 and col2 from the input, and a column colx that’s the sum of col3 and col4 in the input. Only those rows for which both col5 > 10 is true and that have a non-null value for col6 will be output. A null entry is signified by a single - character.

vnl-filter --eval '{s += x} END {print s}'

will evaluate the given awk program on the input, but the column names work as you would hope they do: if the input has a column named x, this would produce the sum of all values in this column.

  • vnl-sort, vnl-uniq, vnl-join, vnl-tail, vnl-ts are wrappers around the corresponding commandline tools. These work exactly as you would expect also: the columns can be referenced by name, and the legend comment is handled properly. These are wrappers, so all the commandline options those tools have “just work” (except options that don’t make sense in the context of vnlog). As an example, vnl-tail -f will follow a log: data will be read by vnl-tail as it is written into the log (just like tail -f, but handling the legend properly). And you already know how to use these tools without even reading the manpages! Note: I use the Linux kernel and the tools from GNU Coreutils exclusively, but this all has been successfully tested on FreeBSD and OSX also. Please let me know if something doesn’t work.
  • vnl-align aligns vnlog columns for easy interpretation by humans. The meaning is unaffected
  • Vnlog::Parser is a simple perl library to read a vnlog
  • vnlog is a simple python library to read a vnlog. Both python2 and python3 are supported
  • libvnlog is a C library to simplify writing a vnlog. Clearly all you really need is printf(), but this is useful if we have lots of columns, many containing null values in any given row, and/or if we have parallel threads writing to a log. In my usage I have hundreds of columns of sparse data, so this is handy
  • vnl-make-matrix converts a one-point-per-line vnlog to a matrix of data. I.e.
$ cat dat.vnl
# i j x
0 0 1
0 1 2
0 2 3
1 0 4
1 1 5
1 2 6
2 0 7
2 1 8
2 2 9
3 0 10
3 1 11
3 2 12

$ < dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp
Writing to '/tmp/x.matrix'

$ cat /tmp/x.matrix
1 2 3
4 5 6
7 8 9
10 11 12

All the tools have manpages that contain more detail. And more tools will probably be added with time.

Workflows and recipes

Storing disjoint data

A common use case is a complex application that produces several semi-related subsets of data at once. Example: a moving vehicle is reporting both its own position and the observed positions of other vehicles; at any given time any number of other vehicles may be observed. Two equivalent workflows are possible:

  • a single unified vnlog stream for all the data
  • several discrete vnlog streams for each data subset

Both are valid approaches

One unified vnlog stream

Here the application produces a single vnlog that contains all the columns, from all the data subsets. In any given row, many of the columns will be empty (i.e. contain only - ). For instance, a row describing a vehicle own position will not have data about any observations, and vice versa. It is inefficient to store all the extra - but it makes many things much nicer, so it’s often worth it. vnl-filter can be used to pull out the different subsets. Sample joint.vnl:

# time x_self x_observation
1      10     -
2      20     -
2      -      100
3      30     -
3      -      200
3      -      300

Here we have 3 instances in time. We have no observations at time 1, one observation at time 2, and two observations at time 3. We can use vnl-filter to pull out the data we want:

$ < joint.vnl vnl-filter -p time,self

# time x_self
1 10
2 20
2 -
3 30
3 -
3 -

If we only care about our own positions, the + modifier in picked columns in vnl-filter is very useful here:

$ < joint.vnl vnl-filter -p time,+self

# time x_self
1 10
2 20
3 30


$ < joint.vnl vnl-filter -p time,+observation

# time x_observation
2 100
3 200
3 300

Note that the default is --skipempty, so if we’re only looking at x_self for instance, then we don’t even need to + modifier:

$ < joint.vnl vnl-filter -p self

# x_self
10
20
30

Also, note that the vnlog C interface works very nicely to produce these datafiles:

  • You can define lots and lots of columns, but only fill some of them before calling vnlog_emit_record(). The rest will be set to -.
  • You can create multiple contexts for each type of data, and you can populate them with data independently. And when calling vnlog_emit_record_ctx(), you’ll get a record with data for just that context.

Several discrete vnlog streams

Conversely, the application can produce separate vnlog streams for each subset of data. Depending on what is desired, exactly, vnl-join can be used to re-join them:

$ cat self.vnl

# time x_self
1 10
2 20
3 30


$ cat observations.vnl

# time x_observation
2 100
3 200
3 300


$ vnl-join -j time -a- self.vnl observations.vnl

# time x_self x_observation
1 10 -
2 20 100
3 30 200
3 30 300

Data statistics

A common need is to compute basic statistics from your data. Many of the alternative toolkits listed above provide built-in facilities to do this, but vnlog does not: it’s meant to be unixy, where each tool has very limited scope. Thus you can either do this with awk like you would normally, or you can use other standalone tools to perform the needed computations. For instance, I can generate some data:

$ seq 2 100 | awk 'BEGIN {print "# x"} {print log($1)}' > /tmp/log.vnl

Then I can compute the mean with awk:

$ < /tmp/log.vnl vnl-filter --eval '{sum += x} END {print sum/NR}'
3.67414

Or I can compute the mean (and other stuff) with a separate standalone tool:

$ < /tmp/log.vnl ministat
x <stdin>
+----------------------------------------------------------------------------+
|                                                                         xx |
|                                                                  x xxxxxxx |
|                                                             xx xxxxxxxxxxxx|
|                                                x  x xxxxxxxxxxxxxxxxxxxxxxx|
|x       x    x    x  x  x  x x x xx xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|                                         |_______________A____M___________| |
+----------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  99      0.693147       4.60517       3.93183     3.6741353    0.85656382

ministat is not a part of the vnlog toolkit, but the vnlog format is generic so it works just fine.

Powershell-style filtering of common shell commands

Everything about vnlog is generic and simple, so it’s easy to use it to process data that wasn’t originally meant to be used this way. For instance filtering the output of ls -l to report only file names and sizes, skipping directories, and sorting by file sizes:

$ ls -l

total 320
-rw-r--r-- 1 dima dima  5044 Aug 25 15:04 Changes
-rw-r--r-- 1 dima dima 12749 Aug 25 15:04 Makefile
-rw-r--r-- 1 dima dima 69789 Aug 25 15:04 README.org
-rw-r--r-- 1 dima dima 33781 Aug 25 15:04 README.template.org
-rw-r--r-- 1 dima dima  5359 Aug 25 15:04 b64_cencode.c
drwxr-xr-x 4 dima dima  4096 Aug 25 15:04 completions
drwxr-xr-x 3 dima dima  4096 Aug 25 15:04 lib
drwxr-xr-x 3 dima dima  4096 Aug 25 15:04 packaging
drwxr-xr-x 2 dima dima  4096 Aug 25 15:04 test
-rwxr-xr-x 1 dima dima  5008 Aug 25 15:04 vnl-align
-rwxr-xr-x 1 dima dima 56637 Aug 25 15:04 vnl-filter
-rwxr-xr-x 1 dima dima  5678 Aug 25 15:04 vnl-gen-header
-rwxr-xr-x 1 dima dima 29815 Aug 25 15:04 vnl-join
-rwxr-xr-x 1 dima dima  3631 Aug 25 15:04 vnl-make-matrix
-rwxr-xr-x 1 dima dima  8372 Aug 25 15:04 vnl-sort
-rwxr-xr-x 1 dima dima  5822 Aug 25 15:04 vnl-tail
-rwxr-xr-x 1 dima dima  4439 Aug 25 15:04 vnl-ts
-rw-r--r-- 1 dima dima   559 Aug 25 15:04 vnlog-base64.h
-rw-r--r-- 1 dima dima  8169 Aug 25 15:04 vnlog.c
-rw-r--r-- 1 dima dima 12677 Aug 25 15:04 vnlog.h


$ (echo '# permissions num_links user group size month day time name';
   ls -l | tail -n +2) |
  vnl-filter 'permissions !~ "^d"' -p name,size |
  vnl-sort -gk size |
  vnl-align

#       name         size
vnlog-base64.h        559
vnl-make-matrix      3631
vnl-ts               4439
vnl-align            5008
Changes              5044
b64_cencode.c        5359
vnl-gen-header       5678
vnl-tail             5822
vnlog.c              8169
vnl-sort             8372
vnlog.h             12677
Makefile            12749
vnl-join            29815
README.template.org 33781
vnl-filter          56637
README.org          69789

With a bit of shell manipulation, these tools can be applied to a whole lot of different data streams that know nothing of vnlog.

C interface

Basic usage

For most uses, vnlog files are simple enough to be generated with plain prints. But then each print statement has to know which numeric column we’re populating, which becomes effortful with many columns. In my usage it’s common to have a large parallelized C program that’s writing logs with hundreds of columns where any one record would contain only a subset of the columns. In such a case, it’s helpful to have a library that can output the log files. This is available. Basic usage looks like this:

In a shell:

vnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' 'void* binary' > vnlog_fields_generated.h

In a C program test.c:

#include "vnlog_fields_generated.h"

int main()
{
    vnlog_emit_legend();

    vnlog_set_field_value__w(-10);
    vnlog_set_field_value__x(40);
    vnlog_set_field_value__y("asdf");
    vnlog_emit_record();

    vnlog_set_field_value__z(0.3);
    vnlog_set_field_value__x(50);
    vnlog_set_field_value__w(-20);
    vnlog_set_field_value__binary("\x01\x02\x03", 3);
    vnlog_emit_record();

    vnlog_set_field_value__w(-30);
    vnlog_set_field_value__x(10);
    vnlog_set_field_value__y("whoa");
    vnlog_set_field_value__z(0.5);
    vnlog_emit_record();

    return 0;
}

Then we build and run, and we get

$ cc -o test test.c -lvnlog

$ ./test

# w x y z binary
-10 40 asdf - -
-20 50 - 0.2999999999999999889 AQID
-30 10 whoa 0.5 -

The binary field in base64-encoded. This is a rarely-used feature, but sometimes you really need to log binary data for later processing, and this makes it possible.

So you

  1. Generate the header to define your columns
  2. Call vnlog_emit_legend()
  3. Call vnlog_set_field_value__...() for each field you want to set in that row.
  4. Call vnlog_emit_record() to write the row and to reset all fields for the next row. Any fields unset with a vnlog_set_field_value__...() call are written as null: -

This is enough for 99% of the use cases. Things get a bit more complex if we have have threading or if we have multiple vnlog ouput streams in the same program. For both of these we use vnlog contexts.

Contexts

To support independent writing into the same vnlog (possibly by multiple threads; this is reentrant), each log-writer should create a context, and use it when talking to vnlog. The context functions will make sure that the fields in each context are independent and that the output records won’t clobber each other:

void child_writer( // the parent context also writes to this vnlog. Pass NULL to
                   // use the global one
                   struct vnlog_context_t* ctx_parent )
{
    struct vnlog_context_t ctx;
    vnlog_init_child_ctx(&ctx, ctx_parent);

    while(records)
    {
        vnlog_set_field_value_ctx__xxx(&ctx, ...);
        vnlog_set_field_value_ctx__yyy(&ctx, ...);
        vnlog_set_field_value_ctx__zzz(&ctx, ...);
        vnlog_emit_record_ctx(&ctx);
    }

    vnlog_free_ctx(&ctx); // required only if we have any binary fields
}

If we want to have multiple independent vnlog writers to different streams (with different columns and legends), we do this instead:

file1.c:

#include "vnlog_fields_generated1.h"

void f(void)
{
    // Write some data out to the default context and default output (STDOUT)
    vnlog_emit_legend();
    ...
    vnlog_set_field_value__xxx(...);
    vnlog_set_field_value__yyy(...);
    ...
    vnlog_emit_record();
}

file2.c:

#include "vnlog_fields_generated2.h"

void g(void)
{
    // Make a new session context, send output to a different file, write
    // out legend, and send out the data
    struct vnlog_context_t ctx;
    vnlog_init_session_ctx(&ctx);
    FILE* fp = fopen(...);
    vnlog_set_output_FILE(&ctx, fp);
    vnlog_emit_legend_ctx(&ctx);
    ...
    vnlog_set_field_value__a(...);
    vnlog_set_field_value__b(...);
    ...
    vnlog_free_ctx(&ctx); // required only if we have any binary fields
    vnlog_emit_record();
}

Note that it’s the user’s responsibility to make sure the new sessions go to a different FILE by invoking vnlog_set_output_FILE(). Furthermore, note that the included vnlog_fields_....h file defines the fields we’re writing to; and if we have multiple different vnlog field definitions in the same program (as in this example), then the different writers must live in different source files. The compiler will barf if you try to #include two different vnlog_fields_....h files in the same source.

Remaining APIs

  • vnlog_printf(...) and vnlog_printf_ctx(ctx, ...) write to a pipe like

printf() does. This exists primarily for comments.

  • vnlog_clear_fields_ctx(ctx, do_free_binary) clears out the data in a context

and makes it ready to be used for the next record. It is rare for the user to have to call this manually. The most common case is handled automatically (clearing out a context after emitting a record). One area where this is useful is when making a copy of a context:

struct vnlog_context_t ctx1;
// .... do stuff with ctx1 ... add data to it ...

struct vnlog_context_t ctx2 = ctx1;
// ctx1 and ctx2 now both have the same data, and the same pointers to
// binary data. I need to get rid of the pointer references in ctx1

vnlog_clear_fields_ctx(&ctx1, false);
  • vnlog_free_ctx(ctx) frees memory for an vnlog context. Do this before

throwing the context away. Currently this is only needed for context that have binary fields, but this should be called for all contexts anyway, in case this changes in a later revision

Base64 interface

The C interface supports writing base64-encoded binary data using Chris Venter’s libb64. The base64-encoder used here was slightly modified: the output appears all on one line, making is suitable to appear in a vnlog field. If we’re writing a vnlog with printf() directly without using the vnlog.h interface described above, we allow this modified base64 encoder to be invoked by itself. Usage:

void* binary_buffer     = ...;
int   binary_buffer_len = ...;

char base64_buffer[vnlog_base64_dstlen_to_encode(binary_buffer_len)];
vnlog_base64_encode( base64_buffer, sizeof(base64_buffer),
                     binary_buffer, binary_buffer_len );

Clearly the above example allocates the base64 buffer on the stack, so it’s only suitable for small-ish data chunks. But if you have lots and lots of data, probably writing it as base64 into a vnlog isn’t the best thing to do.

Python interface

Reading vnlog data into a python program is simple. The vnlog Python module provides three different ways to do that:

  1. slurp the whole thing into a numpy array using the slurp() function. Basic usage: 
    import vnlog
log_numpy_array,list_keys,dict_key_index = \
    vnlog.slurp(filename_or_fileobject)

    This parses out the legend, and then calls numpy.loadtxt(). Null data values (-) are not supported

  2. Iterate through the records: vnlog class, used as an iterator. Basic usage: 
    import vnlog
for d in vnlog.vnlog(filename_or_fileobject):
    print(d['time'],d['height'])

    Null data values are represented as None

  3. Parse incoming lines individually: vnlog class, using the parse() method. Basic usage: 
    import vnlog
parser = vnlog.vnlog()
for l in file:
    parser.parse(l)
    d = parser.values_dict()
    if not d:
        continue
    print(d['time'],d['height'])

Most of the time you’d use options 1 or 2 above. Option 3 is the most general, but also the most verbose

numpy interface

If we need to read data into numpy specifically, nicer tools are available than the generic vnlog Python module. The built-in numpy.loadtxt numpy.savetxt functions work well. For example to write to standard output a vnlog with fields a, b and c:

numpy.savetxt(sys.stdout, array, fmt="%g", header="a b c")

Note that numpy automatically adds the # to the header. To read a vnlog from a file on disk, do something like

array = numpy.loadtxt('data.vnl')

These functions know that # lines are comments, but don’t interpret anything as field headers. That’s easy to do, so I’m not providing any helper libraries. I might do that at some point, but in the meantime, patches are welcome.

Compatibility

I use GNU/Linux-based systems exclusively, but everything has been tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and CentOS. I can imagine there’s something I missed when testing on non-Linux systems, so please let me know if you find any issues.

Caveats and bugs

These tools are meant to be simple, so some things are hard requirements. A big one is that columns are whitespace-separated. There is no mechanism for escaping or quoting whitespace into a single field. I think supporting something like that is more trouble than it’s worth.

Build and installation

Most of this is written in an interpreted language, so there’s nothing to build or install, and you can run the tools directly from the source tree:

$ git clone https://github.com/dkogan/vnlog.git
$ cd vnlog
$ ./vnl-filter .....

If you do want to install to some arbitrary location, do this:

$ make
$ PREFIX=/usr/local make install

The C API is the one component that does require compilation, which can be done by running make. Note: this requires GNU Make and the chrpath tool, which are available in most package repositories.

Installation on Debian-based boxes

vnlog is a part of Debian/buster and Ubuntu/cosmic (18.10) and later. On those boxes you can simply

$ sudo apt install vnlog libvnlog-dev libvnlog-perl python3-vnlog

to get the binary tools, the C API, the perl and python3 interfaces respectively.

Manpages

vnl-filter

xxx-manpage-vnl-filter-xxx

vnl-align

xxx-manpage-vnl-align-xxx

vnl-sort

xxx-manpage-vnl-sort-xxx

vnl-join

xxx-manpage-vnl-join-xxx

vnl-tail

xxx-manpage-vnl-tail-xxx

vnl-ts

xxx-manpage-vnl-ts-xxx

vnl-uniq

xxx-manpage-vnl-uniq-xxx

vnl-gen-header

xxx-manpage-vnl-gen-header-xxx

vnl-make-matrix

xxx-manpage-vnl-make-matrix-xxx

Repository

https://github.com/dkogan/vnlog/

Authors

Dima Kogan (dima@secretsauce.net) wrote this toolkit for his work at the Jet Propulsion Laboratory, and is delighted to have been able to release it publically

Chris Venter (chris.venter@gmail.com) wrote the base64 encoder

License and copyright

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

Copyright 2016-2017 California Institute of Technology

Copyright 2017-2018 Dima Kogan (dima@secretsauce.net)

b64_cencode.c comes from cencode.c in the libb64 project. It is written by Chris Venter (chris.venter@gmail.com) who placed it in the public domain. The full text of the license is in that file.