Tsurugi 1.0.0-BETA3 - Release Notes

[akirakw]

1.0.0-BETA3

Overview

1.0.0-BETA3 resolves some major issues found since 1.0.0-BETA2.

• Failed operations with INSERT or REPLACE will cause the table to be inaccessible afterwards.
• Serializability violations may be not detected in some cases involving race conditions using INSERT or REPLACE.
• Server crashes rarely under some heavy workloads with unique constraint violations by INSERT operations
• LEFT OUTER JOIN may produce incorrect results if its WHERE condition includes right-hand table columns.
• FULL OUTER JOIN with WHERE conditions causes server crash
• Read-only transactions (RTX) may not refer the expected data while running other long transactions (LTXs) are running

In addition, we added a command line tool tgdump to the installation configuration.
It can export Tsurugi table data as Parquet or Apache Arrow formatted files.

• https://github.com/project-tsurugi/tanzawa/blob/master/modules/tgdump/README.md

Other features of this release include overall stability improvements, improved usability of operational management functions, and reduced CPU resource utilization during idle periods.

Important Upgrade Notes

For common upgrade steps from previous versions, please refer to the following documents

• Tsurugi Upgrade Guide

The following sections describe the upgrade notes specific to this version and the required upgrade procedures.

Compatibility with client library (Tsubakuro)

Tsubakuro version 1.2.0 or later must be used to communicate with this version of Tsurugi server.

This release introduces incompatibilities with previous versions in the internal communication protocol.
It includes a handshake mechanism to confirm the details of the client upon establishing a connection with the server.

Please also upgrade the libraries and tools using Tsubakuro to the respective versions. These include Iceaxe and Tsurugi SQL console (a.k.a. tgsql).

It's important to note that Tsubakuro version 1.2.0 or newer is not compatible with Tsurugi 1.0.0-BETA2 or earlier. Attempting to connect with such combinations of versions will result in an exception being sent to the client's connection request. Be aware that the exception message may be confusing, as indicated below.

Example of exception output when connecting from Tsubakuro version 1.2.0 to Tsurugi 1.0.0-BETA2 via TCP connection:

java.io.IOException: Server crashed
            at com.tsurugidb.tsubakuro.channel.common.connection.wire.impl.Link.pullMessage(Link.java:57)
            at com.tsurugidb.tsubakuro.channel.common.connection.wire.impl.ChannelResponse.waitForMainResponse(ChannelResponse.java:73)
            at com.tsurugidb.tsubakuro.channel.common.connection.ForegroundFutureResponse.getInternal(ForegroundFutureResponse.java:107)
            ...

Example of exception output when connecting from Tsubakuro version 1.2.0 to Tsurugi 1.0.0-BETA2 via IPC connection:

com.tsurugidb.tsubakuro.exception.ResponseTimeoutException
            at app//com.tsurugidb.tsubakuro.channel.common.connection.ForegroundFutureResponse.getInternal(ForegroundFutureResponse.java:110)
            at app//com.tsurugidb.tsubakuro.channel.common.connection.ForegroundFutureResponse.get(ForegroundFutureResponse.java:84)
            ...

If you encounter other issues arising from version incompatibility between the server and the client, please consult the following documents for resolution.

• https://github.com/project-tsurugi/tsurugidb/blob/master/docs/upgrade-guide.md#trouble-shooting

Compatibility of Apache Arrow Version

The version of Apache Arrow required by Tsurugi has been changed from 9.0.0 to 14.0.1, by introducing tgdump command and some other reasons.

You can upgrade Apache Arrow by running apt-install.sh in the installer of this version.

If you have already installed the previous version, run apt-install.sh as described in Tsurugi Upgrade Guide and Tsurugi Getting Started to upgrade the versions of the dependent libraries.

For more detailed information about Apache Arrow versions, please refer to the below documentation.

• https://github.com/project-tsurugi/jogasaki/blob/master/README.md

Changelog

⚠ BREAKING CHANGES

• tsubakuro, tateyama, tateyama-bootstrap, and ogawayama: old client can not establish session with new server.

Features

• extend protocol buffer definition to support dump Arrow files (a2559a8)
• implement database and session info handling in request object with compatibility retained (28bd4ec)
• implement database and session info handling in request object with compatibility retained (fc8eb79)
• jogasaki,tsubakuro: protobuf msg. change to support various arrow/parquet file format options (0252f2b)
• jogasaki,tsubakuro: update sql SMV minor version to support dump file formats (bb2d10e)
• jogasaki: support dump arrow files (ddd8e88)
• shirakami: add feature to switch the waiting-bypass mode (006e86d)
• tanzawa: add tgdump command (3c1a8ed)
• tateyama, tateyama-bootstrap: implement altimeter log setup and output for session begin/end (1d95fae)
• tateyama, tateyama-bootstrap: revise error message to show shared memory error and glog warnings (ade5129)
• tateyama: returns SERVICE_UNAVAILABLE diagnostics for requests with unknown service id (dbf8911)
• tateyama: revise error messages (c807551)
• tsubakuro, tateyama, tateyama-bootstrap, and ogawayama: implement handshake prior to session establishment (bc475d1)

Bug Fixes

• add config. param. dev_omit_task_when_idle to avoid scheduling tasks on durability callback (b5e8080)
• add maintenance mode (f67cc96)
• add missing conf logbuflevel (355ab26)
• consider null version (7dbbb42), closes project-tsurugi/tsurugi-issues#440
• correct exception handling in the db recovery process (db63c92)
• delete useless release_message call (916c16d)
• fix about optimistic read logic (16f04bc)
• fix about upsert logic and improve tests (3da90b3), closes project-tsurugi/tsurugi-issues#528
• fix bug about computing cc safe epoch and improve debug tool (5ccea80), closes project-tsurugi/tsurugi-issues#467
• fix bugs about upsert (4dc1868)
• fix SQL optimizer to correct outer join with where .. is null. (35015d4)
• fix SQL optimizer to keep join conditions. (b85831e)
• for ubuntu-20.04 (056031b)
• improve about waiting resolver (790cc7d)
• improve arounde waiting bypass and ci (dc6a65c)
• improve ci, log message (03ea30c)
• jogasaki: avoid extra data copy with INSERT (bd3230e)
• jogasaki: avoid server crash with full outer join with conditions (d2f2d79)
• jogasaki: avoid warning when query returns unknown type (ab03ffd)
• jogasaki: correct handling of decimal types in DDL without precision (5c3881e)
• jogasaki: create index to raise error if table has records (693ef56)
• jogasaki: fix arrow dump to handle multiple record batches correctly (cb14a4d)
• jogasaki: log message to show memory pool statistics (9ddc064)
• jogasaki: split large data into multiple arrow record barches (1d64588)
• jogasaki: stop scheduling durability callback tasks if there is no transcation (a091447)
• jogasaki: validate arrow version strictly (7727fd6)
• ogawayama: fix decimal encoding from takatori::decimal::triple to ::jogasaki::proto::sql::common::Decimal (1573fdf)
• refactor: test: fix version order logic. refactor. improve test (c905cfa), closes project-tsurugi/tsurugi-issues#528
• remove extern from public api (b593b63)
• remove unnecessary logging code (0375964)
• sharksfin: notify shirakami to execute restore correctly (c301e9b)
• tateyama-bootstrap: remove tgundertaker again (b062658)
• tateyama-bootstrap: without the undertaker, tsurugidb's CI will fail, so revive it (7127054)
• tateyama-bootstrap: fix a message string output by tgctl kill (c03132c)
• tateyama-bootstrap: fix problems with tgctl restore (01d4ca3)
• tateyama-bootstrap: if tgctl backup create is performed while tsurugidb is not running, it will start tsurugidb (47489ff)
• tateyama, tateyama-bootstrap: change exit code of tgctl to non zero when the error occurs with duplicate startup (fb2b30c)
• tateyama, tsubakuro: fix regression on streamLink that occurred in commit bc475d1 (1c11b1f)
• tateyama: add more scheduler metrics in log message (a6d8f0b)
• tateyama: alter error message used in handshake error case (9aac9d5)
• tateyama: alter std::shared_ptr handling in stream endpoint (4389cda)
• tateyama: fix an issue about DbMultiSessionTest in iceaxe-testing (cf040d2)
• tateyama: fix regression that occurred in commit bc475d1 (18d381a)
• tateyama: notify maintenance mode to sharksfin (a67d2cc)
• tateyama: preserve compatibility to properly return errors to streamLink clients that do not implement handshaking (b26a632)
• tateyama: prevent starting another tsurugidb with the same database_name as a tsurugidb already started (036ff51)
• tateyama: prevent tsurugidb from being invoked when another tsurugidb process is running (ad5bea0)
• tateyama: revised lambda expression for worker launch (e23ea63)
• tateyama: stop using boost::rational in scheduler busy loop (b048a3c)
• tsubakuro: always dispose resultSetImpl from ServerResourceHolder at resultSetImpl.close() (877df53)
• tsubakuro: fix javadoc for StreamLink.closeWithoutGet() (7775941)
• tsubakuro: fix regression on ipcLink that occurred in commit bc475d1 (d4ff999)
• tsubakuro: fix regression on ipcLink that occurred in commit bc475d1 (8294234)
• tsubakuro: fix stream socket leak at handshake error (314260c)
• update dist/install/conf/tsurugi.ini at glog section (a36bbd4)
• update dist/install/conf/tsurugi.ini at glog section (2nd) (18b30c8)

Documentation

• jogasaki: add markdown for dump format options (bbbb7a7)
• update sql-features.md with IF [NOT] EXISTS for DDLs (7ceafa7)

Distribution

• Add -V option to apt-get in apt-install.sh (43adea9)
• bump apache-arrow version to 14.0.1 in apt-install.sh (3c37f91)
• change distribuion archive name of tanzawa cli (46010d3)
• fix fail to install apache-arrow (df5f4f9)

Styles

• jogasaki: modify coding style around assert (78bcd05)

Miscellaneous Chores

• jogasaki: remove legacy code (8cb0add)
• tateyama: add missing include (e685a13)

Code Refactoring

• tateyama, tateyama-bootstrap: revise build system in regard to ALTIMETER (e134176)
• tsubakuro: refactoring of StreamWire related parts (8c8b37b)

Tests

• add/revise communication tests using MockWire (14a3752)
• fix: add test and fix ltx blind write logic (e5871d9)
• jogasaki: add more testcases for dump arrow (386bc61)
• jogasaki: verify dumping char(n) data type (cb22a22)
• recover testcase for upsert abort (06631ec)

Build System

• enable to rerun install.sh when mpdecimal build fails (c7f1edb)
• reorganize gradle project structure as hierarchical with multiple tools (ec9d3de)
• some fixes for FTBFS with clang++ (0b64b2a)
• tateyama: switch disabling debug service. (15449ba)

Continuous Integration

• add Shirakami test patterns via test lables (5c971df)
• bump actions/upload-artifact to v4 (cf31184)
• fix inputs.targets description in tsurugidb-CI-Test-Modules (6712900)

Reverts

• Revert "fix(tsubakuro): fix regression on ipcLink that occurred in commit bc475d1" due to its impact on tanzawa (0b791c6)
• Revert "feat: implement database and session info handling in request object with compatibility retained", because jogasaki is not ready (d7ae6e5)
• reverting previous reverting commit (a77cda8)

In addition, minor changes include.

---

This discussion was created from the release 1.0.0-BETA3.