v0.15.1

Changes

This release includes the following changes from v0.15.0:

• Un-deprecated project support (see #357)
  • This support is still marked as "experimental" since it will likely be moved to an external project and may evolve further, but it is no longer considered deprecated
• Fixed JSON decoding support for IgnoreVCSMode (#358)
  • Thanks to @fideloper for reporting this
• Updated to Go 1.18.5
• Switched LookPath calls to use golang.org/x/sys/execabs
  • For more information on the motivation behind this change, see https://go.dev/blog/path-security

v0.15.0

Overview

Mutagen v0.15 is focused on improved session statistics and templated list/monitor output, including support for JSON-based output.

Improved session statistics

Synchronization sessions now include detailed transfer progress information, such as the file currently being transferred and its progress, as well as the overall number of files and the total progress. Synchronization sessions also now include information on the total number of directories, files, and symbolic links on each endpoint, as well as their total size.

Forwarding sessions now include information on the amount of data that has been forwarded in each direction.

All of this information is available via the mutagen sync list, mutagen sync monitor, mutagen forward list, and mutagen forward monitor commands.

Templated output and JSON support

All of the aforementioned list and monitor command also now support --template and --template-file flags that can be used to specify Go templates to format listing and monitoring output. A built-in JSON rendering function has been included. To get an idea of the information available, you can use:

mutagen sync list --template='{{ json . }}'

(HINT: If you pipe the output of a command like this to a tool like jq, it will be easier to read)

The data structures used to render this output are contained in the pkg/api/models package.

This output should not yet be considered 100% stable - it may evolve slightly for the next few minor releases, taking advantage of features such as golang/go#45669 as they become available, and iterating as feedback is provided from real-world integration.

Projects deprecated

Mutagen's project functionality has been deprecated. There are no immediate plans to remove this functionality, though there are currently no plans to evolve it further. If anyone is interested in maintaining and/or evolving this functionality as an external project (similar to Mutagen Compose), please ping jacob@mutagen.io or join the discussion on issue #357.

Changes

A full accounting of changes since v0.14.0 can be found here. Notable changes include:

• All list and monitor commands have had their formatting improved
• All list and monitor commands now include --template and --template-file flags to format output
• JSON output is now supported
• Session statistics have been improved
• Other minor bug fixes and improvements

v0.15.0-beta3

NOTE: This is a pre-release version of Mutagen. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

Mutagen maintains backward compatibility between releases, but sessions created with pre-release versions are not guaranteed to be compatible with final releases. Please wait for the final release if you want to be sure that you can avoid recreating sessions.

Changes

Final release notes and documentation will be issued when v0.15.0 is released.

The focus of this release is custom list/monitor output formatting and improved synchronization/forward statistics.

This release includes the following changes from v0.15.0-beta2:

• Adjusted formatting for list and monitor commands (#354)
• Added support for using Windows' built-in OpenSSH implementation (#355)

v0.15.0-beta2

NOTE: This is a pre-release version of Mutagen. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

Mutagen maintains backward compatibility between releases, but sessions created with pre-release versions are not guaranteed to be compatible with final releases. Please wait for the final release if you want to be sure that you can avoid recreating sessions.

Changes

Final release notes and documentation will be issued when v0.15.0 is released.

The focus of this release is custom list/monitor output formatting and improved synchronization/forward statistics.

This release includes the following changes from v0.15.0-beta1:

• Formatting for list and monitor commands has been improved (#352)
• Error logging for agent handshake failures has been improved (#353)

v0.15.0-beta1

NOTE: This is a pre-release version of Mutagen. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

Mutagen maintains backward compatibility between releases, but sessions created with pre-release versions are not guaranteed to be compatible with final releases. Please wait for the final release if you want to be sure that you can avoid recreating sessions.

Changes

Final release notes and documentation will be issued when v0.15.0 is released.

The focus of this release is custom list/monitor output formatting and improved synchronization/forward statistics.

This release includes the following changes from v0.14.0:

• The list and monitor commands all now include --template and --template-file flags that can be used to specify Go formatting templates. JSON output can be generated using the template {{ json . }}. The data models used for these outputs (which can also be used by client programs decode JSON output) are available in the pkg/api/models packages. Top-level output for all commands is a slice of Session types (forwarding.Session for forwarding sessions and synchronization.Session for synchronization sessions).
• Detailed progress information is now available for synchronization staging, include the file name being transferred, file-level progress information, and total progress information with better size information
• Forwarding data volume is now available for forwarding sessions

v0.14.0

Overview

Mutagen v0.14 is focused on observability and performance. Logging has been vastly expanded, made more readable, and is now able to provide more granular information on individual filesystem operations. On the optimization front, a number of changes have been made to improve bulk file staging on remote endpoints and a few minor performance tweaks have been made to scanning. Finally, filesystem watching code has been slightly refactored and an initial implementation of fanotify-based watching is available.

Logging

While most observation in Mutagen is made using the list and monitor commands, the Mutagen daemon itself also outputs logs. These can be seen by running the daemon in the foreground using mutagen daemon run. This is the exact same command that's spun off as a background process by mutagen daemon start, so you'll need to do mutagen daemon stop first if the daemon is already running. The granularity of these logs can be controlled by setting the MUTAGEN_LOG_LEVEL environment variable to one of disabled, error, warn, info, debug, or trace. The default log level is info.

While this logging has always existed, logs at the debug and trace level haven't always provided sufficient information to debug issues. This release vastly increases the information output by these logs, which now includes information on filesystem watcher observations, change reconciliation, and file staging. Moreover, the local log level is now propagated to agent processes (whose logs are merged into the daemon log).

Initial fanotify support

Filesystem watching is an area where Linux unfortunately lags behind macOS and Windows (at least in practical terms). In particular, it has never had a usable, general purpose, user space accessible, natively recursive filesystem watching API (i.e. an API that can watch an entire filesystem directory hierarchy with only a single watch). This is despite having three user space watching APIs: inotify, dnotify, and fanotify (plus an underlying kernel framework to support these called fsnotify).

Most tools use inotify to perform filesystem watching on Linux. inotify is a non-recursive Linux filesystem watching API that requires a single "watch descriptor" (similar to a file descriptor) for every file and directory being monitored. While many libraries attempt to emulate recursive watching by combining inotify with logic that automatically creates and destroys watches, this approach is subject to race conditions and the relatively low default watch descriptor limit.

Mutagen has always taken a hybrid approach on Linux systems: use readdir/fstatat-based polling for accuracy, combined with a limited number of inotify watches on most-recently-updated contents in order to reduce latency. This has meant that every sync session that Mutagen creates that includes a Linux endpoint (i.e. most of them) is subject to O(n) filesystem scanning behavior (either synchronously or (in the accelerated case) asynchronously) on that endpoint. This manifests as periodic CPU spikes, especially for very large synchronization roots.

The fanotify API does provide recursive watching facilities, and it has existed for a long time, but it didn't previously provide the granularity of events necessary for Mutagen to avoid full filesystem re-scans. However, fanotify recently received an overhaul in Linux 5.1, making it just workable enough for Mutagen to use. Unfortunately, it still has some caveats, namely the fact that the calling process needs CAP_SYS_ADMIN and CAP_DAC_READ_SEARCH capabilities (i.e. it basically needs to be run as root). However, this is fairly workable in the context of containers, and thus Mutagen has shipped an alternative sidecar container image that (when run on a 5.1+ kernel with the relevant capabilities enabled) is able to use fanotify. Since this requires some non-trivial setup, the support is currently only turned on automatically when using Mutagen Compose. However, it is very possible to use it in other containerized setups, so please reach out on the Mutagen Community Slack Workspace if you're interested in trying it.

Enabling fanotify support allows for two things: significantly faster filesystem re-scans (meaning lower-latency change propagation) and avoidance of poll-based scanning (meaning no CPU spikes). This can make an enormous difference in the context of multi-GB codebases or a large number of sessions.

Support for fanotify outside of containers is also on the horizon, likely enabled via some sort of sudo-based mechanism or system-level service, but feedback from the container use case will go a long way in validating its viability.

It's also worth mentioning that the fanotify watching implementation is licensed under the Server Side Public License. Certain new features (in particular those that are only relevant to SaaS embedders of Mutagen) will likely start falling under the SSPL umbrella, though there are no plans to change the license for any of Mutagen's MIT-licensed code, and most new code will continue to fall under the MIT license. If you are interested in licensing these features for SaaS-based usage under an alternative license, please reach out to hello@mutagen.io.

Changes

A full accounting of changes since v0.13.1 can be found here. Notable changes include:

• Logging has been vastly expanded and is now more readable
• Log levels are now propagated to agent processes
• Agent installation failures yield better feedback
• Mutagen commands now include a -v/--version flag to print a human-readable version (thanks @rfay)
  • The version command still exists - its functionality will be expanded in Mutagen v0.15
• Filesystem watching interfaces have been refactored
  • Thanks to @djs55 for fixes to the Windows watching code
• Accelerated scanning is no longer temporarily disabled by Transition
• Initial support for fanotify-based filesystem watching has been added in the sidecar 🚀
• Synchronization file transfers have been further optimized:
  • Path filtering now has a shorthand for cases where all files are required
  • Compression flush boundaries are better optimized
• Filesystem scanning has been further optimized:
  • Accelerated scanning uses a more optimal cache propagation strategy
  • Directory traversal has removed some unnecessary validation code
• Cache saves are now event-driven
• Ancestor saves are now contingent on changes occurring
• Sidecar staging is now in-volume for all volume paths (not just volume roots)
• Updated to Go 1.18
• Other minor bug fixes

v0.14.0-beta1

NOTE: This is a pre-release version of Mutagen. It should not be used on production or mission-critical systems. Use on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

Mutagen maintains backward compatibility between releases, but sessions created with pre-release versions are not guaranteed to be compatible with final releases. Please wait for the final release if you want to be sure that you can avoid recreating sessions.

Changes

Final release notes and documentation will be issued when v0.14.0 is released.

The focus of this release is observability and performance.

This release includes the following changes from v0.13.1:

• Logging has been vastly expanded and is now more readable
• Log levels are now propagated to agent processes
• Agent installation failures yield better feedback
• Mutagen commands now include a -v/--version flag to print a human-readable version (thanks @rfay)
  • The version command still exists - its functionality will be expanded in Mutagen v0.15
• Filesystem watching interfaces have been refactored
  • Thanks to @djs55 for fixes to the Windows watching code
• Accelerated scanning is no longer temporarily disabled by Transition
• Initial support for fanotify-based filesystem watching has been added in the sidecar 🚀
• Synchronization file transfers have been further optimized:
  • Path filtering now has a shorthand for cases where all files are required
  • Compression flush boundaries are better optimized
• Filesystem scanning has been further optimized:
  • Accelerated scanning uses a more optimal cache propagation strategy
  • Directory traversal has removed some unnecessary validation code
• Cache saves are now event-driven
• Ancestor saves are now contingent on changes occurring
• Sidecar staging is now in-volume for all volume paths (not just volume roots)
• Updated to Go 1.18
• Other minor bug fixes

v0.13.1

Changes

This release includes the following changes from v0.13.0:

• Fixed an issue with agent termination on Docker for Windows (mutagen-io/mutagen-compose#11 et al.)
  • Major thanks to @darrylkuhn for his assistance in debugging and testing!
• Modified the sidecar build to pre-embed an agent binary for faster Mutagen Compose startup

v0.13.0

Overview

Mutagen v0.13 is a relatively minor release, with few user-visible changes from v0.12. Most changes were small and internal, designed to support the release of Mutagen Compose.

Mutagen Compose

Mutagen Compose is now the default and only supported Compose/Mutagen integration mechanism. It is a separate, self-contained Compose implementation (accessed via the mutagen-compose command) based on Compose V2.

The aging Compose V1 integration that shipped in the v0.12 and v0.13 beta releases is now removed, with v0.13.0-beta4 containing the most recent version.

Changes

This release also includes the following notable changes from the v0.12.x release series:

• Mutagen now knows about the DOCKER_TLS and DOCKER_CONFIG environment variables
• Mutagen now prints additional Docker transport parameters in session listings
• Mutagen has added a new internal staging mode that stages files in a temporary directory inside the synchronization root
• Mutagen supports a new MUTAGEN_SSH_CONNECT_TIMEOUT environment variable for controlling its default OpenSSH ConnectTimeout value

v0.13.0-beta4

NOTE: This is a pre-release version of Mutagen. It should not be used on production or mission-critical systems. Usage on any system is at your own risk (please see the license). For a better experience, you may wish to use the latest stable release.

Mutagen maintains backward compatibility between releases, but sessions created with pre-release versions are not guaranteed to be compatible with final releases. Please wait for the final release if you want to be sure that you can avoid recreating sessions.

Changes

This release includes the following changes from v0.13.0-beta3:

• Updated dependencies