[CCF] Design and publish first draft of Cadence Compact Format (CCF) #3593
Description
Problem
Currently, Cadence external values (e.g. transaction arguments and events) are being encoded using a human-readable JSON-Cadence Data Interchange Format, which is inefficient, verbose, and doesn't define deterministic encoding (canonical form).
A need exists to provide a more compact, efficient, and deterministic encoding of Cadence external values.
Closes onflow/cadence#1651
Updates onflow/cadence#2157
The Proposed Solution
Design and publish a more compact, efficient, and deterministic data format that can be used as an alternative to JSON-Cadence Data Interchange Format.
Status
- @fxamacker - Review requirements on notion documenting the need to have an alternative to JSON-based format.
- @fxamacker - Evaluate using RFC 8949 (CBOR) to design and define Cadence Compact Format (CCF) to replace JSON and Cadence Binary Format (CBF) #3448
- @fxamacker - Evaluate using CDDL notation to specify the new data format.
- @fxamacker - Evaluate using EDN notation to describe example encodings of the new format.
- @fxamacker - Design CCF format using an iterative process using a proof-of-concept CCF codec written in Go.
- @fxamacker - Publish abridged first draft of CCF specification for team to review.
The first abridged draft passed initial reviews. Our first meeting about CCF specs on Nov 22, 2022 discussed each CCF encoding example in the draft CCF specs (revision 20221122b) and no problems were found (so far).
For the current status of the draft, see:
- Epic: Cadence Compact Format (CCF) Specification RC and CCF Codec (fully self-describing mode) cadence#2157
- CCF Specs: https://github.com/fxamacker/ccf_draft
- CCF Codec: Add CCF codec implementing CCF Specification (RC1) cadence#2364
Preliminary Results
Size Comparisons
| Encoding | Event Count | Encoded size | Comments |
|---|---|---|---|
| JSON | 48,309 | 13,858,836 | JSON-Cadence Data Interchange Format |
| CCF | 48,309 | 6,159,931 | CCF in fully self-describing and deterministic mode |
CCF's partially self-describing mode would be even smaller (roughly 1/4 the size of JSON) in some use cases.
Benchmark Comparisons
These informal and preliminary benchmarks used CCF in fully self-describing mode (see size comparisons above).
$ benchstat bench_json_events_48k.log bench_ccf_events_48k.log
goos: linux
goarch: amd64
pkg: github.com/onflow/cadence/encoding/ccf
cpu: 13th Gen Intel(R) Core(TM) i5-13600K
│ bench_json_events_48k.log │ bench_ccf_events_48k.log │
│ sec/op │ sec/op vs base │
EncodeBatchEvents-20 96.61m ± 4% 70.73m ± 3% -26.79% (p=0.000 n=10)
DecodeBatchEvents-20 647.7m ± 3% 157.5m ± 3% -75.68% (p=0.000 n=10)
geomean 250.1m 105.5m -57.81%
│ bench_json_events_48k.log │ bench_ccf_events_48k.log │
│ B/op │ B/op vs base │
EncodeBatchEvents-20 32.45Mi ± 0% 25.82Mi ± 0% -20.45% (p=0.000 n=10)
DecodeBatchEvents-20 234.97Mi ± 0% 56.16Mi ± 0% -76.10% (p=0.000 n=10)
geomean 87.32Mi 38.08Mi -56.39%
│ bench_json_events_48k.log │ bench_ccf_events_48k.log │
│ allocs/op │ allocs/op vs base │
EncodeBatchEvents-20 756.6k ± 0% 370.4k ± 0% -51.05% (p=0.000 n=10)
DecodeBatchEvents-20 4.746M ± 0% 1.288M ± 0% -72.86% (p=0.000 n=10)
geomean 1.895M 690.7k -63.55%
These results are still preliminary and subject to change.
UPDATES
- 2023-03-03 updated preliminary results using latest CCF codec benchmarked using 48,000+ events from a mainnet transaction.