v0.5.1

A release that adds support for retaining the format of floating-point numbers from JSON logs, support for CLP’s UI to work with Presto, and some other improvements and bug fixes.

This release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

New features

• clp-json: Support for retaining the format of floating-point numbers. (#1176, #1321)
• webui: Support for CLP’s UI to work with Presto. (#1095, #1108, #1127, #1150, #1167, #1168, #1175, #1179, #1187, #1188, #1191, #1198, #1206, #1209, #1224, #1236, #1244, #1248, #1250, #1251, #1265, #1280, #1285, #1306, #1310)
  • NOTE: This adds the configuration key presto to etc/clp-config.yml.
• clp-json: Support for compressing Zstandard-compressed JSONL files as well as CLP KV-IR files. (#1312)

Bug fixes & improvements

• clp-json/clp-text: Add support for gracefully shutting down the compression scheduler and compression workers (resolves #1037). (#1169, #1323)
• core: Unescape variable strings before dictionary lookup in EncodedVariableInterpreter::encode_and_search_dictionary (fixes #590). (#1270)
• clp-s: Handle pure wildcards and unexpected literal types correctly in EvaluateTimestampIndex (fixes #1096). (#1277)
• kv-ir: Add support for getting the number of log events read from the deserializer. (#1282)
• kv-ir: Add support for duplicate columns in projections. (#1245)
• clp-json/clp-text: Consider all non-loopback IPv4s when selecting listen address to receive search results (fixes #1316). (#1317)
• clp-json/clp-text: Fail job and report failure to user for compression jobs that encounter at least one invalid input path (fixes #308). (#1125)
• clp-json: Warn the user if they do not use the --timestamp-key flag when compressing with the clp-s storage engine. (#1283)
• webui: Expose rate limit configuration (fixes #1019); Increase rate limit to 1000 req/min (fixes #1020). (#1234)
  • NOTE: This adds the configuration key webui.rate_limit to etc/clp-config.yml.

View the full changelog for more details.

Thanks to @anlowee, @AVMatthews, @Bill-hbrhbr, @davemarco, @gibber9809, @haiqi96, @hoophalab, @junhaoliao, @kirkrodrigues, @LinZhihao-723, and @quinntaylormitchell for their contributions.

v0.5.0

A release that includes support for configuring retention periods for archives, deleting datasets entirely, and some other features and bug fixes.

Note

This release includes some changes that are incompatible with previous releases. If this affects you, reach out and we may be able to help with version migration. These changes are marked with "Breaking".

This release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

New features

• clp-json/clp-text: Add support for configuring retention periods for archives and search results. (#1035, #1181, #1205, #1231)
• clp-json: Add dataset-manager tool to support listing datasets, and deleting them entirely. (#1144, #1215, #1225)

Bug fixes & improvements

• clp-json/clp-text: Fix scheduler freezes by updating celery to 5.5.3 with the redis extra (replacing the direct redis dependency) (fixes #1059). (#1213)
• clp-json/clp-text: Remove dependency on native libraries for scripts run on the host (fixes #895, #1185). (#1105, #1197)
• core-clp: Preserve escaped ?-wildcards in queries (fixes #243). (#1070)
• core: Unify clp-s and clp’s unstructured text parsing and search code. (#1101, #1103, #1112, #1138, #1143, #1163)
  • Breaking: This is a change to the archive format used by clp-json (and the core clp-s binary).
• webui: Delete old search results after 60min by default. (#1231)
  • Breaking: Previously, the results of the last query executed before refreshing the page would be retained indefinitely.
• webui: Update query time range on any change in the UI to prevent query submission with out-of-date time range. (#1171)
• webui: Focus/refocus query input box for ease of use. (#1160)
• clp-json/clp-text: Only mount stream_output_dir when stream storage type is FS, to allow running the webui on a different node when using S3 storage. (#1129)
• clp-json/clp-text: Explicitly convert enum to integer to ensure accurate conversions when using the mysql Python library. (#1133)
• webui: Cast BIGINT values as UNSIGNED to fix MySQL-specific type errors in dashboard stats queries (fixes #1137). (#1136)
• clp-json/clp-text: Mark incomplete jobs as failed when schedulers restart. (#1208)
• clp-json/clp-text: Add verbose logging option to archive_manager and simplify output on errors. (#1173)
• clp-json/clp-text: Add query_engine option to clp-config.yml to support starting only compression and UI components when using the Presto query engine. (#1095)
  • Breaking: This is a change to clp-json/clp-text’s config file format.
• core: Replace YAML config with CLI args and env vars for metadata DB. (#1148)
  • Breaking: This is a change to how the core clp/clp-s binaries can be configured to use a MySQL-based database (the change is transparent to clp-json/clp-text users).
• core: Address CVE-2024-3094 and CVE-2025-31115 for xz/lzma dependency (fixes #1093). (#1094)

---

View the full changelog for more details.

Thanks to @anlowee, @Bill-hbrhbr, @davemarco, @davidlion, @gibber9809, @haiqi96, @hoophalab, @jackluo923, @junhaoliao, @kirkrodrigues, @LinZhihao-723, @quinntaylormitchell, @SharafMohamed, and @wraymo for their contributions.

v0.4.0

A release that includes support for organizing compressed logs into datasets, a revamped UI, support for a new IR format designed for structured log events, and more advanced metadata storage within clp-json archives. This release also includes some other features and bug fixes.

Warning

This release contains bug #1059 which can cause compression or search to freeze. This has been fixed in v0.5.0, so we recommend using v0.5.0 or a later release.

Note

We are bumping the minor version due to several breaking changes including format changes in clp-text/clp-json's archive formats, changes to clp-json/clp-s' query syntax, changes to clp-json's config file format, changes to the command line interface, and changes to CLP's runtime requirements. If this affects you, reach out and we may be able to help with version migration.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

New features

• clp-json: Add support for organizing compressed logs into datasets, where each dataset can be searched separately. (#831, #839, #864, #868, #923, #1023, #1029, #1036, #1050, #1091)
• clp-json/clp-text: Replace the UI with a revamped version that has better performance while being more extensible and maintainable. (#645, #647, #791, #797, #804, #807, #824, #830, #848, #866, #875, #880, #891, #892, #899, #910, #912, #913, #915, #931, #933, #936, #937, #939, #941, #949, #950, #954, #956, #957, #958, #959, #962, #967, #972, #973, #974, #988, #990, #991, #993, #995, #997, #999, #1000, #1001, #1003, #1007, #1013, #1015, #1016, #1042, #1053, #1061, #1063, #1064, #1074, #1078, #1079, #1083)
• core: Add support for a new CLP intermediate representation (IR) stream format which is designed for compressing and searching structured (e.g., JSON) log events. (#411, #446, #502, #507, #511, #549, #653, #677, #695, #706, #726, #728, #731, #733, #769, #833, #856, #859, #860, #863, #873, #882, #888, #890, #909, #914)
• clp-s: Add support for storing metadata about ranges of log events (e.g., the name of the log file to which they were printed) in an archive. (#847, #889, #908, #929, #1005, #1026, #1043)

Bug fixes & improvements

• clp-s: Improve compression ratio by delta-encoding the log-order column. (#1021)
• clp-json/clp-text: Add support for configuring the compression level. (#774)
• clp-json: Add support for more AWS authentication methods (e.g., EC2 instance metadata, environment variables, AWS profiles, etc.). (#743, #788, #852)
• clp-json/clp-text: Don't remap output paths when mounting them into CLP's execution container (fixes #960). (#998)
• core: Try to exhaust Zstd's internal buffers when they might contain unconsumed data. (fixes #976). (#977)
• clp-json/clp-text: Use job duration for final compression speed summary. (#823)
• core: Update DictionaryReader::get_entry_matching_value to handle case-insensitive searches (fixes #648). (#690)
• webui: Improve tracking of compression jobs submitted in quick succession (fixes #667). (#679)

View the full changelog for more details.

Thanks to @aestriplex, @anlowee, @AVMatthews, @Bill-hbrhbr, @davemarco, @davidlion, @Eden-D-Zhang, @gibber9809, @haiqi96, @Henry8192, @hoophalab, @jackluo923, @junhaoliao, @kirkrodrigues, @LinZhihao-723, @quinntaylormitchell, @SharafMohamed, @sitaowang1998, and @wraymo for their contributions.

v0.3.0

A release that adds support for using clp-json to both compress logs from object storage and store archives on object storage (docs). This release also includes some other features and bug fixes.

NOTE: We are bumping the minor version due to a breaking format change in clp-text/clp-json’s jobs table format and in clp-json/clp-s’ archive format. If this affects you, reach out and we can help with version migration.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

New features

• clp-json: Add support for compressing logs from S3. (#651)
• clp-json: Add support for storing, decompressing, and searching archives from S3. (#634, #674, #683)
• clp-json: Add support for viewing logs from S3. (#662, #673, #678)
• clp-s: Add support for compressing logs from S3. (#639)
• clp-s: Add support for writing single-file archives. (#563)
• clp-s: Add support for reading and searching single-file archives, including from S3. (#656)
• Add BoundedReader to prevent out-of-bound reads in segmented input streams. (#624)

Bug fixes & improvements

• clp-json: Add option to output search results as raw logs. (#641)
• clp-package: Unify the metadata schema for JSON and IR streams. (#620)
• clp-package: Enable replica set for the MongoDB results cache and configure it when starting the package. (#632)
• clp: Advance to the next message when a message has an out-of-range timestamp when searching archives (fixes #659). (#660)
• clp-s: Unescape string values during ingestion and fix support for search using escape sequences. (#622)
• clp-s: Improve error reporting for directory-creation failure during compression. (#671, #684)
• clp-s: Rename tables section to use segment numbering scheme. (#666)
• Handle 0-byte reads when BufferReader's underlying buffer is fully consumed. (#687)
• Add missing libcurl4 dependency to clp-core and package execution containers. (#670)
• Disable file system translation in checkinstall during dependency installation (fixes #642). (#644)

View the full changelog for more details.

Thanks to @AVMatthews, @Bill-hbrhbr, @Eden-D-Zhang, @LinZhihao-723, @gibber9809, @haiqi96, @jackluo923, @junhaoliao, and @kirkrodrigues for their contributions.

v0.2.1

A release that adds support for viewing JSON search results in context by opening the archive that contains them using the log viewer. This release also includes some features that improve usability.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

New features

• Support for viewing JSON search results in context by opening the archive that contains them using the log viewer. (#569, #584, #596, #600, #615)
  • Note that the log viewer doesn’t open the archive directly but rather CLP decompresses the archive into chunks of JSONL files that the log viewer opens. In a future release, these chunks will be IR files to lower resource usage.
• clp-json: Support for querying fields whose keys contain periods by escaping them with a backslash. (#560, #617)
• Support for deleting archives that are entirely within a time range. (#594)

Bug fixes & improvements

• Homebrew path detection for mariadb-connector-c to fix macOS build failures. (#582)

View the full changelog for more details.

Thanks to @AVMatthews, @Bill-hbrhbr, @LinZhihao-723, @anlowee, @gibber9809, @haiqi96, @junhaoliao, @kirkrodrigues, and @wraymo for their contributions.

v0.2.0

A release that uses the rewritten log viewer and includes new features for clp-s, bug fixes, and improvements.

NOTE: We are bumping the minor version due to a breaking format change in clp-json/clp-s’ archive format. If this affects you, reach out and we can help with version migration.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

New features

• An upgrade to the rewritten log viewer. (#562, #565)
  • The rewrite is based around the clp-ffi-js library (which uses WebAssembly) and improves performance for switching pages by 2.77-4.67x, and log-level filtering by 1.64-4.84x.
  • The rewrite also includes a UI refresh, more fine-grained log-level filtering, support for exporting the decompressed logs, and support for opening larger log files.
  • NOTE: The rewrite also includes support for formatting and viewing JSON logs which will be integrated with CLP in the next release.
• Support for building on CentOS Stream 9. (#521)
  • NOTE: We have dropped support for building on CentOS 7.4.
• clp-s:
  • Support for projecting a subset of columns during search. (#510)
    • Support for projection from the package will be integrated in a future release.
  • Packing of small tables into a single compression stream which can improve compression ratio as well as compression and search performance. (#466)

Bug fixes & improvements

• Fixes to allow CLP’s core to build with shared linking again. (#526)
• Upgrades for vulnerable dependencies. (#536)

View the full changelog for more details.

Thanks to @anlowee, @Henry8192, @LinZhihao-723, @diy1, @gibber9809, @junhaoliao, @kirkrodrigues, and @zechenericduan for their contributions.

v0.1.3

A bug-fix release to improve error reporting, improve the UI, and fix some macOS-specific issues.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Bug fixes & improvements

• clp-s: Report exactly where parsing error occurs when parsing JSON (fixes #514). (#503)
• webui: Improve UI of links for viewing search results in context. (#515)
• core: Add support for retrieving CURL error messages, handle unexpected CURL return code on macOS, and log such codes in tests (fixes #519). (#517)
• core: Mark spdlog specialization functions as const (fixes #524). (#525)

View the full changelog for more details.

Thanks to @AVMatthews, @Bill-hbrhbr, @gibber9809, @haiqi96, @junhaoliao, @kirkrodrigues, and @LinZhihao-723 for their contributions.

v0.1.2

A release with support for viewing text search results in context by opening the file that contains them using using the log viewer. In a future release, we will also add support for viewing JSON search results in context. This release also includes a handful of bug fixes.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Features/Changes

Support for viewing text search results in context by opening the file that contains them using the log viewer. (#420, #424, #435, #437, #440, #442, #449, #451, #452, #458, #460, #468, #470, #472, #474, #476, #481, #485, #489, #490)

Bug-fixes

• clp-package: Add support for running package as root (fixes #500). (#464)
• clp-s: Ensure the option to print archive stats remains set when splitting archives. (#462)
• clp-s: Correctly report uncompressed size of archives when splitting archives (fixes #469). (#463)
• core-clp: Add class to encapsulate libcurl's global resource management to fix a race condition. (#461)
• core-clp: Add a more complete UTF-8 validation function to filter out binary data. (#477)
• core-clp: Fix ArchiveMetadata deserialization by deserializing previously missed field. (#488)
• webui: Optimize Meteor bundle size (fixes #498). (#499)

View the full changelog for more details.

Thanks to @Bill-hbrhbr, @gibber9809, @haiqi96, @Henry8192, @junhaoliao, @kirkrodrigues, @LinZhihao-723, and @wraymo for their contributions.

CLP v0.1.1

A release with a few new features and bug fixes.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Features/Changes

• A new way of storing arrays found in JSON log events. (#355, #401, #408, #413, #429)
  • Try it using clp-s's --structurize-arrays option.
• clp-json no longer adds implicit wildcards (’*’) at the beginning or end of a query. (#406)
• More frequent progress updates (on the command line) for compression jobs. (#388)

Bug fixes

• clp-json: Better handling for wildcards in the middle of a token. (#407)
• Improved robustness when cancelling search jobs. (#425)
• Reduced CPU usage when queries from the webui complete. (#418)

View the full changelog for more details.

Thanks to @davemarco, @gibber9809, @haiqi96, @Henry8192, @kirkrodrigues, @LinZhihao-723, and @wraymo for their contributions.

CLP v0.1.0

A release containing several new features including a new subsystem for compressing JSON logs, a web interface, a server for aggregations, and more!

Note that the release now includes two tarballs:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Highlights

• Added a subsystem for compressing and searching JSON logs. (#217)
• Added a web interface for searching logs and viewing compression statistics. (#250)
• Added support for ingesting CLP IR streams generated by our logging library plugins. (#152)
• Added a reducer worker to support aggregation jobs like count. (#221)
• Added support for tagging archives and filtering archives by tags and/or timestamps. (#138, #285)
• Replaced the scheduler with separate compression and search schedulers for improved robustness and performance. (#238, #240)

This release also includes many other improvements and fixes. View the full changelog for more details.

Thanks to @abvarun226, @davemarco, @davidlion, @diy1, @gibber9809, @haiqi96, @junhaoliao, @kirkrodrigues, @LinZhihao-723, @OliverSM95, @SharafMohamed, and @wraymo for their contributions!