Skip to content

Latest commit

 

History

History
233 lines (152 loc) · 10.1 KB

README.md

File metadata and controls

233 lines (152 loc) · 10.1 KB
Raw

ffjson: faster JSON for Go

Build Status Fuzzit Status

ffjson generates static MarshalJSON and UnmarshalJSON functions for structures in Go. The generated functions reduce the reliance upon runtime reflection to do serialization and are generally 2 to 3 times faster. In cases where ffjson doesn't understand a Type involved, it falls back to encoding/json, meaning it is a safe drop in replacement. By using ffjson your JSON serialization just gets faster with no additional code changes.

When you change your struct, you will need to run ffjson again (or make it part of your build tools).

Blog Posts

Getting Started

If myfile.go contains the struct types you would like to be faster, and assuming GOPATH is set to a reasonable value for an existing project (meaning that in this particular example if myfile.go is in the myproject directory, the project should be under $GOPATH/src/myproject), you can just run:

go get -u github.com/pquerna/ffjson
ffjson myfile.go
git add myfile_ffjson.go

Performance Status:

  • MarshalJSON is 2x to 3x faster than encoding/json.
  • UnmarshalJSON is 2x to 3x faster than encoding/json.

Features

  • Unmarshal Support: Since v0.9, ffjson supports Unmarshaling of structures.
  • Drop in Replacement: Because ffjson implements the interfaces already defined by encoding/json the performance enhancements are transparent to users of your structures.
  • Supports all types: ffjson has native support for most of Go's types -- for any type it doesn't support with fast paths, it falls back to using encoding/json. This means all structures should work out of the box. If they don't, open a issue!
  • ffjson: skip: If you have a structure you want ffjson to ignore, add ffjson: skip to the doc string for this structure.
  • Extensive Tests: ffjson contains an extensive test suite including fuzz'ing against the JSON parser.

Using ffjson

ffjson generates code based upon existing struct types. For example, ffjson foo.go will by default create a new file foo_ffjson.go that contains serialization functions for all structs found in foo.go.

Usage of ffjson:

        ffjson [options] [input_file]

ffjson generates Go code for optimized JSON serialization.

  -go-cmd="": Path to go command; Useful for `goapp` support.
  -import-name="": Override import name in case it cannot be detected.
  -nodecoder: Do not generate decoder functions
  -noencoder: Do not generate encoder functions
  -w="": Write generate code to this path instead of ${input}_ffjson.go.

Your code must be in a compilable state for ffjson to work. If you code doesn't compile ffjson will most likely exit with an error.

Disabling code generation for structs

You might not want all your structs to have JSON code generated. To completely disable generation for a struct, add ffjson: skip to the struct comment. For example:

// ffjson: skip
type Foo struct {
   Bar string
}

You can also choose not to have either the decoder or encoder generated by including ffjson: nodecoder or ffjson: noencoder in your comment. For instance, this will only generate the encoder (marshal) part for this struct:

// ffjson: nodecoder
type Foo struct {
   Bar string
}

You can also disable encoders/decoders entirely for a file by using the -noencoder/-nodecoder commandline flags.

Using ffjson with go generate

ffjson is a great fit with go generate. It allows you to specify the ffjson command inside your individual go files and run them all at once. This way you don't have to maintain a separate build file with the files you need to generate.

Add this comment anywhere inside your go files:

//go:generate ffjson $GOFILE

To re-generate ffjson for all files with the tag in a folder, simply execute:

go generate

To generate for the current package and all sub-packages, use:

go generate ./...

This is most of what you need to know about go generate, but you can sese more about go generate on the golang blog.

Should I include ffjson files in VCS?

That question is really up to you. If you don't, you will have a more complex build process. If you do, you have to keep the generated files updated if you change the content of your structs.

That said, ffjson operates deterministically, so it will generate the same code every time it run, so unless your code changes, the generated content should not change. Note however that this is only true if you are using the same ffjson version, so if you have several people working on a project, you might need to synchronize your ffjson version.

Performance pitfalls

ffjson has a few cases where it will fall back to using the runtime encoder/decoder. Notable cases are:

  • Interface struct members. Since it isn't possible to know the type of these types before runtime, ffjson has to use the reflect based coder.
  • Structs with custom marshal/unmarshal.
  • Map with a complex value. Simple types like map[string]int is fine though.
  • Inline struct definitions type A struct{B struct{ X int} } are handled by the encoder, but currently has fallback in the decoder.
  • Slices of slices / slices of maps are currently falling back when generating the decoder.

Reducing Garbage Collection

ffjson already does a lot to help garbage generation. However whenever you go through the json.Marshal you get a new byte slice back. On very high throughput servers this can lead to increased GC pressure.

Tip 1: Use ffjson.Marshal() / ffjson.Unmarshal()

This is probably the easiest optimization for you. Instead of going through encoding/json, you can call ffjson. This will disable the checks that encoding/json does to the json when it receives it from struct functions.

	import "github.com/pquerna/ffjson/ffjson"

	// BEFORE:
	buf, err := json.Marshal(&item)

	// AFTER:
	buf, err := ffjson.Marshal(&item)

This simple change is likely to double the speed of your encoding/decoding.

[![GoDoc][1]][2] [1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg [2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Marshal

Tip 2: Pooling the buffer

On servers where you have a lot of concurrent encoding going on, you can hand back the byte buffer you get from json.Marshal once you are done using it. An example could look like this:

import "github.com/pquerna/ffjson/ffjson"

func Encode(item interface{}, out io.Writer) {
	// Encode
	buf, err := ffjson.Marshal(&item)
	
	// Write the buffer
	_,_ = out.Write(buf)
	
	// We are now no longer need the buffer so we pool it. 
	ffjson.Pool(buf)
}

Note that the buffers you put back in the pool can still be reclaimed by the garbage collector, so you wont risk your program building up a big memory use by pooling the buffers.

[![GoDoc][1]][2] [1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg [2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Pool

Tip 3: Creating an Encoder

There might be cases where you need to encode many objects at once. This could be a server backing up, writing a lot of entries to files, etc.

To do this, there is an interface similar to encoding/json, that allow you to create a re-usable encoder. Here is an example where we want to encode an array of the Item type, with a comma between entries:

import "github.com/pquerna/ffjson/ffjson"

func EncodeItems(items []Item, out io.Writer) {
        // We create an encoder.
	enc := ffjson.NewEncoder(out)
	
	for i, item := range items {
		// Encode into the buffer
		err := enc.Encode(&item)
		
		// If err is nil, the content is written to out, so we can write to it as well.
		if i != len(items) -1 {
			_,_ = out.Write([]byte{','})
		}
	}
}

Documentation: [![GoDoc][1]][2] [1]: https://godoc.org/github.com/pquerna/ffjson/ffjson?status.svg [2]: https://godoc.org/github.com/pquerna/ffjson/ffjson#Encoder

Tip 4: Avoid interfaces

We don't want to dictate how you structure your data, but having interfaces in your code will make ffjson use the golang encoder for these. When ffjson has to do this, it may even become slower than using json.Marshal directly.

To see where that happens, search the generated _ffjson.go file for the text Falling back, which will indicate where ffjson is unable to generate code for your data structure.

Tip 5: ffjson all the things!

You should not only create ffjson code for your main struct, but also any structs that is included/used in your json code.

So if your struct looks like this:

type Foo struct {
  V Bar
}

You should also make sure that code is generated for Bar if it is placed in another file. Also note that currently it requires you to do this in order, since generating code for Foo will check if code for Bar exists. This is only an issue if Foo and Bar are placed in different files. We are currently working on allowing simultaneous generation of an entire package.

Improvements, bugs, adding features, and taking ffjson new directions!

Please open issues in Github for ideas, bugs, and general thoughts. Pull requests are of course preferred :)

Similar projects

  • go-codec. Very good project, that also allows streaming en/decoding, but requires you to call the library to use.
  • megajson. This has limited support, and development seems to have almost stopped at the time of writing.

Credits

ffjson has recieved significant contributions from:

License

ffjson is licensed under the Apache License, Version 2.0