1.51.3

Merged PRs

dolt

• 9100: Consistently use pointer receivers for TextStorage methods.
  For structs that are exclusively used in interfaces, there's no reason to not use pointer receivers instead of value receivers. Using value receivers can cause the struct to be copied every time the method is invoked.
  Pointer receivers are also more correct here: copying the struct means that changes to the ImmutableValue field (such as setting its Buf field in ImmutableValue.GetBytes) won't be saved when the method exits, defeating the caching of values loaded from storage.
• 9098: Add support for SSL connections to a source MySQL server
  Allows a replica Dolt sql-server to connect to a MySQL source via SSL when the SOURCE_SSL configuration parameter is supplied.
  Depends on: dolthub/go-mysql-server#2930
• 9097: Prevent unnecessary bytes <-> string conversions and unnecessary allocations when converying to bytes.
• 9095: Add the archive --purge flag to clean up table files
  The dolt archive command is extra safe, and doesn't cleanup table files is converts. The --purge flag will delete table files after the conversion is done.
• 9085: Adding new copy-tags command

go-mysql-server

• 2932: Optimize ConvertToBytes by avoiding unnecessary string <-> bytes conversions and copies.
  ConvertToBytes is a commonly called function to get the string representation of a value. However, it has some unnecessary allocations where a child function allocates a byte buffer, only for the result to be copied into the buffer provided by the parent function. In other places we needlessly round-trip between string and []byte.
  This PR improves performance by removing some of these unneeded copies. In places where ConvertToBytes calls a function that allocates a buffer (instead of using the buffer that ConvertToBytes can provide), we can optimize by using the returned value without copying it again.
  Dolt shows a 7% improvement in the types_table_scan benchmark.
  We can potentially do even better by allowing these child functions to take a buffer, removing the need for an extra allocation.
• 2930: Add support for SSL connections to a source MySQL server
  Adds GMS support for the SOURCE_SSL MySQL replication configuration param.
• 2928: implement stddev_pop(), stddev_samp(), var_pop(), var_samp(), and aliases
  This PR implements STD() and VARIANCE() and its aliases.
  docs: https://dev.mysql.com/doc/refman/8.4/en/aggregate-functions.html#function_std
  syntax: dolthub/vitess#409
  fixes: #9088
• 2926: Provide a SchemaFormatter interface to complement the SqlParser interface
  Dolt uses functions in the sql package to create and then parse CREATE TABLE statements as part of the merge process. Therefore, we need the ability to create these statements for other dialects, just as we do for MySQL.
  This PR also exposes a Noop auth method for the same use case.

vitess

• 409: add var_samp to keywords
• 408: Add support for SOURCE_SSL replication option
• 407: remove incorrect continue statement which skips handling connection
  Caught in PR.

Closed Issues

• 9088: Request for STDEV_POP() (and aliases) and VAR_POP() (and alias)

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.86	0.67	0.36
groupby_scan	13.46	17.63	1.31
index_join	1.47	2.39	1.63
index_join_scan	1.44	1.47	1.02
index_scan	34.95	30.26	0.87
oltp_point_select	0.18	0.26	1.44
oltp_read_only	3.43	5.18	1.51
select_random_points	0.33	0.6	1.82
select_random_ranges	0.37	0.62	1.68
table_scan	34.95	32.53	0.93
types_table_scan	75.82	125.52	1.66
reads_mean_multiplier			1.29

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.32	0.71
oltp_insert	4.03	3.07	0.76
oltp_read_write	8.9	11.45	1.29
oltp_update_index	4.1	3.19	0.78
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.67	6.32	1.11
types_delete_insert	8.28	6.55	0.79
writes_mean_multiplier			0.88

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	97.23	39.13	2.48
tpcc_tps_multiplier			2.48

Overall Mean Multiple	1.55

1.51.2

Merged PRs

dolt

• 9089: refactor: use the built-in max/min to simplify the code
  Contribution: #9044
• 9084: [no release notes] Handle NULL adaptive values
  NULL Adaptive values weren't being properly handled in all codepaths. Some operations during tuple building (such as trying to move them out of band) caused a panic. This fixes that and adds tests.
• 9083: Support for non-MySQL query elements during merge
  During the merge operation, Dolt creates a CREATE TABLE statement for the table being merged, parses it, and uses that information during the merge. This PR allows this process to work with a non-MySQL dialect.
• 9075: Don't alter branch head outside of transactions when amending a commit
  Fixes: #9072
• 9071: Fix typo mistake of commmitted
  Thanks to @jbampton
• 9067: go: sqle/dprocedures: Implement operational inspection hook, dolt_thread_dump stored procedure.
• 9066: go/store/nbs: Add info level logging for a store conjoin.
• 9044: refactor: use the built-in max/min to simplify the code
  Use the built-in max/min from the standard library in Go 1.21 to simplify the code. https://pkg.go.dev/builtin@go1.21.0#max
• 9013: Add Adaptive Inline encoding
  Adaptive Inline encoding refers to values that can be stored either inline in a table, or out-of-band in external storage. Postgres calls them TOAST (The Oversized-Attribute Storage Technique) types.
  This PR adds two new encoding types: one for toasted TEXT columns and one for toasted BLOB columns.
  The important parts of this PR are:
  • The new encoding values StringToastEnc and BytesToastEnc
  • The ToastValue type: this type has methods for creating and inspecting TOAST values.
  • The changes to prolly.PutField and prolly.GetField to handle writing reading and writing these encodings to tuples
  • The changes to TupleBuilder, which choose an inline or outlined representation for each value with a TOAST encoding. It prefers to store these values inline, but selectively outlines columns in a deterministic way if the length of the tuple would otherwise exceed a threshold. (Currently 2KB, not currently configurable.)
    It is not currently possible to use these encodings in Dolt proper. For testing purposes, a variable schema.UseToastTypes has been added. Setting this variable to true will cause Dolt to use these toast encodings for TEXT and BLOB columns instead of the normal address encodings.

go-mysql-server

• 2927: Converting a nil value should return nil, not panic.
  Most uses of convertValue checked if the input was nil, but Binary.Eval didn't.
  NULL should always be converted to NULL, so we can just put the check there.
• 2926: Provide a SchemaFormatter interface to complement the SqlParser interface
  Dolt uses functions in the sql package to create and then parse CREATE TABLE statements as part of the merge process. Therefore, we need the ability to create these statements for other dialects, just as we do for MySQL.
  This PR also exposes a Noop auth method for the same use case.
• 2924: fix ifnull typing
  fixes #9076
• 2923: fix AND optimization
  This PR fixes AND optimizations where TRUE & x -> x.
  Probably should've seen this coming given our bug fix with OR here,
  fixes: #9074
• 2921: fix not equal conversions
  fixes: #9052
• 2913: insert trigger bugs
  fixes: #9039
  Inserts nested inside triggers is a special case that we have minimal testing for. Some of the things that were broken before:
  • we did not prepend nodes inside trigger->insert blocks, so new columns were not accessible / the index offsets estimated based on prepending trigger rows were invalid
  • subqueries inside trigger->insert that have no outer visibility did not make exceptions for trigger rows, which act like lateral rows
  • prepending rules are flaky and mistakenly prepended in-between project->project nodes
• 2893: Add Wrapper values
  This PR adds support for Wrapper values, a new type of value that the storage layer can provide to the engine. This is a generalization of the existing JsonWrapper interface, but is not limited to JSON values.
  Wrapper types are useful because they can represent a value in storage without needing to fully deserialize that value for the engine. The primary use case is for "out-of-band" storage values, such as large BLOB or TEXT values that aren't stored directly in tables.
  The Wrapper interface has the following methods:
  Unwrap/UnwrapAny - These methods both deserialize the wrapped value and return it. UnwrapAny returns an interface{}, while Unwrap has a parameterized return type. The implementation is encouraged to cache this value.
  IsExactLength - Returns true if the size of the value can be known without deserializing it.
  MaxByteLength - The length of the value in bytes. If IsExactLength is true, this is the exact length of the value. Otherwise, it's an upper bound on the length of the value, determined by the schema of the table the value was read from.
  Compare - Some Wrapper implementations may be able to compare wrapped values without needing to deserialize them. This method returns a boolean indicating whether or not this "short-circuit" comparison is possible, and an int indicating the result of the comparison.

Closed Issues

• 9072: Potential concurrency Issue with call dolt_commit('--amend')
• 9076: Unexpected type conversion in IFNULL
• 9074: Incorrect optimization of AND operation in expression in WHERE clause
• 9052: Incorrect optimization of OR operation in expression in WHERE clause
• 9039: Trigger LEFT_JOIN indexing error

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.89	0.65	0.34
groupby_scan	13.46	17.63	1.31
index_join	1.47	2.39	1.63
index_join_scan	1.44	1.44	1.0
index_scan	34.33	30.81	0.9
oltp_point_select	0.18	0.26	1.44
oltp_read_only	3.43	5.18	1.51
select_random_points	0.33	0.59	1.79
select_random_ranges	0.37	0.62	1.68
table_scan	34.95	32.53	0.93
types_table_scan	75.82	134.9	1.78
reads_mean_multiplier			1.3

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.32	0.71
oltp_insert	4.1	3.07	0.75
oltp_read_write	8.9	11.45	1.29
oltp_update_index	4.18	3.19	0.76
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.67	6.32	1.11
types_delete_insert	8.28	6.67	0.81
writes_mean_multiplier			0.88

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	97.8	40.1	2.44
tpcc_tps_multiplier			2.44

Overall Mean Multiple	1.54

1.51.1

Merged PRs

dolt

• 9063: Normalize AUTO_INCREMENT values in tables.
  When the AUTO_INCREMENT field is unset, the value of the next AUTO_INCREMENT column defaults to 1. Thus, setting it to 0 and 1 produce identical behavior.
  By normalizing the value as its written, we ensure that both operations produce the same table hash as if the value was unset.
• 9061: Bug fix for doltgres range iter
• 9059: go: cmd/dolt: commands/sql: Fix the displayed query time in interactive mode after a query completes.
  The migration to subcontexts of the Queryist sql.Context meant that we lost the default setting of the query start time. Restore it in the sql command implementation itself.
• 9050: go,integration-tests/{bats,go-sql-server-driver}: Implement GC sesssion lifecycle validation at the storage layer. Enable checks for bats and go sql server integration tests.
  This makes *NomsBlockStore check the incoming Context object to ensure that it itself has been invovled in the appropriate GC lifecycle callbacks.
  It fixes a problem with statspro.AnalyzeTable, where the GC lifecycle callbacks happened more than once for a single session.
  It fixes some callsites to appropriately make the GC lifecycle callbacks, including LateBindingQueryist opening one session command for the whole lifetime of the returned sql.Context.
• 9047: go: utils/publishrelease/buildpgobinaries.sh: Bump optcross to pick up new dolthub_toolchains changes.
• 9041: go: store/datas/pull: Error handling improvements. Target an approximate file size instead of a number of chunks when uploading files.
  Our old heuristic was to cut a file and start uploading it when it reached a fixed number of chunks. We chose that number of chunks as just (1<<30 / 4096), which is the average chunk size the chunker targets. But the chunk targets pre-compressed sizes. So file uploads which were hitting the target could vary a lot in size. It's better to just track how many bytes we've written so far and cut it when (approximately) appropriate. This is still best effort and only approximately.
  Also improves error handling and structured concurrency a bit. Moves a number of goroutines up to the top-level errgroup in Pull(). Avoids creating – and thus potentially leaking – goroutines in NewPuller(), deferring them until Pull actually gets called. Gets rid of an unnecessary request-response thread structure in the implementation of PullChunkFetcher.
• 9032: integration-tests/go-sql-server-driver: Move max-connections testing to go-sql-server-driver.
• 9029: go: Add some GC lifecycle callbacks to commands which use a local SqlEngine.
• 9023: go: mvdata: Improve dump, import and export so that these commands create a single SqlEngine over their lifetime.
  Pass the SqlEngine along and reuse it in the operations where we dump table contents, parse import schemas, etc.
• 8965: Root Objects
  Counterpart to Doltgres PR:
  • dolthub/doltgresql#1265

go-mysql-server

• 2921: fix not equal conversions
  fixes: #9052
• 2920: fix negative integer limits
  In golang, the negation of the minimum int8, int16, int32, int64 is the same.
  fix: #9053
• 2919: fix double negatives
  We can't apply NOT(NOT(expr)) -> expr optimization if there will be a type conversion involved.
  This PR also fixes bad test involving current timestamps.
  fixes: #9054
• 2918: more precision fixes for unix_timestamp
  Changes:
  • have datetime_type equality to also compare precision
  • retain precision for date and datetime types in a table
    fixes: #9025
• 2916: prevent panics in replacePkSort
  Not sure how to reproduce this, but this should make the code more safe.
  fixes: dolthub/go-mysql-server#2915
• 2914: bump: dolthub/go-icu-regex
• 2912: Support for negative bool values
  Fixes #9036
• 2911: check group by ordinal range
  This PR adds an error check when attempting to group by a column index that is out of range.
  fixes: #9037

Closed Issues

• 9052: Incorrect optimization of operation in expression in WHERE clause
• 9053: Incorrect negation of minimum signed integer
• 9035: NULL in GROUP BY column is interpreted as 1
• 9054: Double negation is treated as original value in WHERE clause
• 9025: unix_timestamp's precision should keep with parameter
• 9051: Dolthub commit changes history loads forever
• 9037: Unexpected crash when using GROUP BY with non-column position
• 2915: Panic when calling min/max (without a Primary Key?)

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.93	0.67	0.35
groupby_scan	13.46	17.95	1.33
index_join	1.47	2.43	1.65
index_join_scan	1.44	1.44	1.0
index_scan	34.95	30.81	0.88
oltp_point_select	0.18	0.27	1.5
oltp_read_only	3.49	5.28	1.51
select_random_points	0.34	0.61	1.79
select_random_ranges	0.37	0.62	1.68
table_scan	34.95	31.37	0.9
types_table_scan	75.82	116.8	1.54
reads_mean_multiplier			1.28

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.32	0.71
oltp_insert	4.1	3.13	0.76
oltp_read_write	9.06	11.65	1.29
oltp_update_index	4.18	3.19	0.76
oltp_update_non_index	4.18	3.13	0.75
oltp_write_only	5.77	6.32	1.1
types_delete_insert	8.43	6.67	0.79
writes_mean_multiplier			0.88

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	96.16	39.18	2.45
tpcc_tps_multiplier			2.45

Overall Mean Multiple	1.54

1.51.0

Merged PRs

Minor version bump here reflects changes in background statistics management. The new statistics system tracks updates for all branches with a common backing cache that reduces memory consumption when branches have overlapping table data. See docs for the expanded interface.

dolt

• 9034: Be more flexible for missing working sets when loading db
  Fixes: #9031
• 9028: Test new connection configuration in vitess
  This PR is both a dependency bump and enabling the configuration args in the dependency.
• 9021: go: gcctx,dsess: Move GCSafepointController into an independent package. Make dsess.AutoIncrementTracker background database access participate in GC lifecycle callbacks.
• 9020: go: branch_control,sqle/dsess: Pass the *sql.Context to the GetBranch accessor. Wrap it for the branch_control.Context conversion.
• 9016: Remove duplicate github.com/vbauerster/mpb dependency
• 9006: Replace jpillora/backoff with cenkalti/backoff/v4
  This is a drop-in replacement.
  Both github.com/jpillora/backoff and github.com/cenkalti/backoff/v4 provide exponential backoff implementations. Since we already use github.com/cenkalti/backoff/v4 elsewhere in the codebase, this change standardizes our dependency by using a single implementation consistently.
• 9002: Replace golang.org/x/exp/slices with stdlib slices
  The experimental functions in golang.org/x/exp/slices are now available in the standard library in Go 1.21.
  Reference: https://go.dev/doc/go1.21#slices
• 9000: Fix spelling
• 8991: Remove duplicate github.com/vbauerster/mpb dependency
  https://github.com/dolthub/dolt/blob/b7c0ddb4aac3702114c4e060651007d590a8be8c/go/cmd/dolt/cli/stdio.go#L28
  https://github.com/dolthub/dolt/blob/b7c0ddb4aac3702114c4e060651007d590a8be8c/go/cmd/dolt/cli/stdio_test.go#L23
  It's weird that we use the tagged v8 version in the source code but non-tagged version in the test.
• 8815: [stats] Rewrite stat management to use single threaded event loop
  Reference statspro/doc.go for more detailed overview. Replaced stats management with a worker->scheduler->executer system. Worker does the default background job organizing, scheduler serializes background and async requests, executor calls function callbacks with a ticker delay. GC performed within a loop.
  Early perf testing seems to have little/no impact on TPC-C when the job ticker is 100-500ms.
  fixes: #8844

go-mysql-server

• 2907: special case for unix_timestamp
  When the provided timestamp is in string format, we preserve the scale of the original string (assuming it is valid).
  Additionally, we should round precision instead of truncating.
  #9025
• 2902: Support ALTER TABLE ... ADD COLUMN with an inline check constraint
  Adds support for adding a column to a table with an inline constraint declared. Previously, the inline constraint was parsing, but ignored.
  Depends on: dolthub/vitess#405
  Originally discovered as part of testing DoltHub's Postgres schema with Doltgres.
• 2901: disable_merge_join corner case
  Disable merge join previously could only kick in if other hints were applied. We also want it to work in the absence of hints.

vitess

• 407: remove incorrect continue statement which skips handling connection
  Caught in PR.
• 406: Avoid spin waits and dead connections in mysql server
  Related (partial fix): #8943
• 405: Set ConstraintAction if an added column was specified with an inline constraint
  Currently Dolt/GMS ignore an inline check constraint definition in an ALTER TABLE ... ADD COLUMN statement. This change detects if a constraint is present, and sets ConstraintAction so that GMS will add the constraint.
• 403: Log a warning when max connections is hit
  Fixes: #8942
  Tested by configuring a server with 2 max conns, then connecting to it 3 times:

  lcl:~/Documents/data_dir_1/db3$ dolt sql-server --config ./config.yaml
  Starting server with Config HP="localhost:3306"|T="28800000"|R="false"|L="info"|S="/tmp/mysql.sock"
  WARN[0000] unix socket set up failed: file already in use: /tmp/mysql.sock
  INFO[0000] Server ready. Accepting connections.
  WARN[0000] secure_file_priv is set to "", which is insecure.
  WARN[0000] Any user with GRANT FILE privileges will be able to read any file which the sql-server process can read.
  WARN[0000] Please consider restarting the server with secure_file_priv set to a safe (or non-existent) directory.
  INFO[0004] NewConnection                                 DisableClientMultiStatements=false connectionID=1
  INFO[0006] NewConnection                                 DisableClientMultiStatements=false connectionID=2
  WARN[0009] max connections reached. Clients waiting. Increase server max connections
  

• 401: Revert moving call to recycleReadPacket()
  Dolt binlog tests started failing, due to the change in location of calling recycleReadPacket(). This change moves those two calls back to their original location.

Closed Issues

• 9036: Unexpected invalid type error on boolean
• 9037: Unexpected crash when using GROUP BY with non-column position
• 9031: Pushing a new branch to a running dolt sql-server remote port causes ERRO[0536] working set not found error spewed to logs
• 9025: unix_timestamp's precision should keep with parameter
• 6724: dolt merge doesn't produce deterministic hashes
• 8844: Panic from vector index (possibly from stats)

1.50.9

Merged PRs

dolt

go-mysql-server

• 2901: disable_merge_join corner case
  Disable merge join previously could only kick in if other hints were applied. We also want it to work in the absence of hints.
• 2899: [memo] hash join right-side cardinality can be filtered
  Previously assumed HASH_JOIN would load whole right table into memory. This overestimates the cost, we only read and cache the indexed fraction.
  perf benchmarks: #9007

Closed Issues

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.89	0.67	0.35
groupby_scan	13.22	17.95	1.36
index_join	1.47	2.43	1.65
index_join_scan	1.44	1.42	0.99
index_scan	34.95	29.72	0.85
oltp_point_select	0.18	0.27	1.5
oltp_read_only	3.49	5.28	1.51
select_random_points	0.34	0.6	1.76
select_random_ranges	0.37	0.62	1.68
table_scan	34.95	31.37	0.9
types_table_scan	75.82	114.72	1.51
reads_mean_multiplier			1.28

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.21	0.7
oltp_insert	4.1	3.07	0.75
oltp_read_write	8.9	11.45	1.29
oltp_update_index	4.18	3.13	0.75
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.67	6.32	1.11
types_delete_insert	8.43	6.55	0.78
writes_mean_multiplier			0.87

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	96.85	40.13	2.41
tpcc_tps_multiplier			2.41

Overall Mean Multiple	1.52

1.50.8

Merged PRs

dolt

go-mysql-server

• 2900: Clean up regex.Regex disposal
  Preserves Regex instances when creating new nodes from WithChildren() for RegexpInstr , RegexpLike, and RegexpSubstr instances. Disables panics when detecting a leaked Regex and installs a custom function to log an error.
• 2899: [memo] hash join right-side cardinality can be filtered
  Previously assumed HASH_JOIN would load whole right table into memory. This overestimates the cost, we only read and cache the indexed fraction.
  perf benchmarks: #9007
• 2898: [memo] variable to disable merge join
  Use SET @@GLOBAL.disable_merge_join = 1 to prevent join planning from every selecting a merge join. There will always be HASH_JOIN and INNER_JOIN alternatives.

Closed Issues

1.50.7

Merged PRs

dolt

go-mysql-server

• 2898: [memo] variable to disable merge join
  Use SET @@GLOBAL.disable_merge_join = 1 to prevent join planning from every selecting a merge join. There will always be HASH_JOIN and INNER_JOIN alternatives.
• 2896: support for INSERT .. RETURNING, a postgres extension
• 2895: Bug fix: resolve column default expressions for ALTER COLUMN nodes
  When building plan nodes to handle altering a column's nullability or type, without respecifying the full column definition, the built node should ensure any column default expressions are resolved.
  This originally showed up as a panic in Doltgres, because Doltgres' TypeSanitzer found the UnresolvedColumnDefault instance and tried to invoke operations on it.
  Note that it's not currently possible to trigger this issue with Dolt or GMS because MySQL has a more limited syntax than Postgres for altering a column without respecfiying its full column definition, and Dolt/GMS does not support that syntax currently (e.g. ALTER TABLE t ALTER COLUMN SET VISIBLE).

Closed Issues

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.89	0.67	0.35
groupby_scan	13.7	17.95	1.31
index_join	1.5	2.43	1.62
index_join_scan	1.44	1.42	0.99
index_scan	34.95	29.72	0.85
oltp_point_select	0.18	0.26	1.44
oltp_read_only	3.49	5.28	1.51
select_random_points	0.34	0.6	1.76
select_random_ranges	0.37	0.63	1.7
table_scan	34.95	31.37	0.9
types_table_scan	75.82	114.72	1.51
reads_mean_multiplier			1.27

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.32	0.71
oltp_insert	4.1	3.13	0.76
oltp_read_write	9.06	11.45	1.26
oltp_update_index	4.18	3.19	0.76
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.77	6.32	1.1
types_delete_insert	8.43	6.67	0.79
writes_mean_multiplier			0.87

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	96.45	40.01	2.41
tpcc_tps_multiplier			2.41

Overall Mean Multiple	1.52

1.50.6

Merged PRs

dolt

• 9005: go/store/nbs: Fix perf regression in taking backups.
  1.50.0 introduced a perf regression which mainly effects taking backups to AWS or GCS. This is a partial fix which makes backups perform as before when they are going to a completely new destination. Incremental backups to an existing store are still slower. We continue to work on addressing the regression.
• 9003: Up server config default max conns from 100 to 1000
• 8994: Bug fix for TypeCompatibilityChecker to ignore ExtenedTypes
  The following query caused a panic in Doltgres, because DoltTable was using TypeCompatibilityChecker to test if the column type change was compatible. TypeCompatibilityChecker wasn't able to handle the ExtendedType instance and panic'ed. This change makes TypeCompatibilityChecker ignore extended types, since it doesn't know enough about them to determine if a type change is compatible or not.
• 8993: Bump golang.org/x/net in /go/gen/proto/dolt/services/eventsapi
  Bumps golang.org/x/net from 0.23.0 to 0.36.0.
  updated-dependencies:
  • dependency-name: golang.org/x/net dependency-type: indirect ...
• 8992: Remove unnecessary hashicorp/go-uuid dependency
  We are already using github.com/google/uuid for UUIDs. This dependency was accidentally introduced.
• 8990: Remove unnecessary hashicorp/go-uuid dependency
  We are already using github.com/google/uuid for UUIDs. I think this dependency was accidentally introduced.
  https://github.com/dolthub/dolt/blob/b7c0ddb4aac3702114c4e060651007d590a8be8c/go/go.mod#L24
  https://github.com/dolthub/dolt/blob/b7c0ddb4aac3702114c4e060651007d590a8be8c/go/go.mod#L71
• 8988: Test for INSERT ... ON DUPLICATE UPDATE with dolt_docs table
  Adds a test for the bug fix in dolthub/go-mysql-server#2892
• 8979: Bump golang.org/x/net from 0.23.0 to 0.36.0 in /go/gen/proto/dolt/services/eventsapi
• 8850: refactor: using clear to simplify the code
  This is a new built-in function added in the go1.21 standard library, which can make the code more concise and easy to read.

go-mysql-server

• 2895: Bug fix: resolve column default expressions for ALTER COLUMN nodes
  When building plan nodes to handle altering a column's nullability or type, without respecifying the full column definition, the built node should ensure any column default expressions are resolved.
  This originally showed up as a panic in Doltgres, because Doltgres' TypeSanitzer found the UnresolvedColumnDefault instance and tried to invoke operations on it.
  Note that it's not currently possible to trigger this issue with Dolt or GMS because MySQL has a more limited syntax than Postgres for altering a column without respecfiying its full column definition, and Dolt/GMS does not support that syntax currently (e.g. ALTER TABLE t ALTER COLUMN SET VISIBLE).
• 2892: Bug fix: Call StatementBegin for all TableEditors used in insertIter
insertIter can be given a sql.RowInserter and a sql.RowUpdater, and when the ON DUPLICATE UPDATE clause is present, it can use both to update a table. StatementBegin was only being called on the sql.RowInserter and not for the sql.RowUpdater, which caused a problem for some implementations (e.g. docsWriter) that depend on StatementBegin being called.
  Test for this added to the Dolt package (#8988), using the docWriter implementation where this bug was originally discovered.
• 2891: look through table aliases for table names when applying triggers
  Table aliases on update statements break triggers, because we are unable to find any trigger matches on the alias.
  The fix is to find the real table name when searching for triggers.
  fixes: #8926
• 2890: Sort tables by FK dependencies for DROP TABLES
  When multiple tables are supplied to DROP TABLES, MySQL (and also Postgres) will sort the tables by foreign key dependencies, so that the tables can be dropped cleanly.
  Resolves dolthub/doltgresql#1272
• 2889: don't erase dual table projection
  The rule eraseProjection erases the projection from select '', which causes validateSchemaSource to error.
  Adding a case in validateSchemaSource resulted in dummy rows being outputted, so I think this is the best solution for now.
  fixes: #8977
• 2888: Allow type conversions for foreign key checks
  This PR introduces a new method for the ExtendedType interface and uses it to allow converting between compatible column types during a foreign key check.
• 2887: Correct JSON_LENGTH with an object parameter
  When JSON_LENGTH is passed a JSON object, it should always return the number of fields in the object. However, due to a typo, it currently only returns 1 for objects. This PR corrects that behavior.
• 2886: reintroduce last_insert_id discrepancy
  Apparently the last_insert_id in ok result, is not the same as select last_insert_id().
  When there is an auto_increment column, the insert_id in the OKResult is the first value that increments the auto_increment value.
  Reverts some changes from: dolthub/go-mysql-server#2616
  fixes: #8914

Closed Issues

• 8942: Log when max_connections is reached
• 7428: Feature request for supporting health and readiness checks in Dolt DB
• 5997: Pro-active dolt gc warning
• 5785: dolt table import -u inserts and updates duplicate keys instead of failing. Unclear what the desired behavior is.
• 5535: dolt pull from DoltHub does not complete (gets stuck) in hours when a clone takes minutes
• 5371: dolt clone loads all dolt databases in working directory into memory
• 8926: BUG: BEFORE UPDATE Trigger Not Working in Dolt
• 3629: dolt cli commands take a long time for large databases
• 2322: dolt persisted globals --global are not initialized during sql engine creation
• 1987: Automatically GC during SQL server execution
• 864: Postgres compatible mode for dolt diff -r sql and dolt patch
• 8975: Cannot merge branches when two indexes covers same column
• 2847: BUG: Element types don't match MySQL: query-results have inconsistent types

1.50.5

Merged PRs

dolt

• 8984: Fix schema merge bug when schema is identical with multiple indexes on the same columns
  FIxes: #8975

Closed Issues

• 8975: Cannot merge branches when two indexes covers same column
• 8907: Why DOLT_DIFF_SUMMARY shows record without data change and schema change?
• 8977: Can't select ''

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.93	0.67	0.35
groupby_scan	13.22	17.95	1.36
index_join	1.47	2.43	1.65
index_join_scan	1.47	1.44	0.98
index_scan	34.33	30.26	0.88
oltp_point_select	0.18	0.26	1.44
oltp_read_only	3.49	5.28	1.51
select_random_points	0.33	0.6	1.82
select_random_ranges	0.37	0.62	1.68
table_scan	34.95	31.37	0.9
types_table_scan	75.82	114.72	1.51
reads_mean_multiplier			1.28

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.32	0.71
oltp_insert	4.1	3.07	0.75
oltp_read_write	8.9	11.45	1.29
oltp_update_index	4.18	3.13	0.75
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.67	6.32	1.11
types_delete_insert	8.43	6.67	0.79
writes_mean_multiplier			0.88

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	95.47	39.83	2.4
tpcc_tps_multiplier			2.4

Overall Mean Multiple	1.52

1.50.4

Merged PRs

dolt

• 8976: go: sqle/dsess: session.go: Fix transaction savepoint creation when cluster replication is enabled.
  Cluster replication creates a system database, dolt_cluster, which does not support transactions. There are also UserSpaceDatabases which do not support CreateTransaction. CreateTransaction has special code to avoid visiting them when recording the start transaction state. CreateSavepoint needed to be updated with that logic as well.
• 8974: Add support for setting notices in the session
  Adds the ability to queue notice messages in the session, for them to be sent to the client during the execution phase. Related to dolthub/doltgresql#1255.
• 8970: Fix InsertID discrepancy tests
  Fixes tests involving OKPacket's InsertID not aligning with last_insert_id.
  related: #8914
  bump: dolthub/go-mysql-server#2886

go-mysql-server

• 2886: reintroduce last_insert_id discrepancy
  Apparently the last_insert_id in ok result, is not the same as select last_insert_id().
  When there is an auto_increment column, the insert_id in the OKResult is the first value that increments the auto_increment value.
  Reverts some changes from: dolthub/go-mysql-server#2616
  fixes: #8914
• 2885: [planbuilder] sort expression aliases always referenced
  This fixes a bug where a sort expression alias computed in the lower projection fails to index the nested expression.
  Below, the first plan's sort searches for c5:6 in the child, but only finds a:7. The second plan fixes the correctness issue. Obviously there are more desirable projection organizations that version two, but this is small enough of an edge case that I think rewriting projection management with proper expression interning would be overkill right now. The rest of the plan tests look OK/improvements.

  select distinct abs(c5) as a from one_pk where c2 in (1,11,31) order by a
  before:
  Sort(abs(one_pk.c5:6)->a:7 ASC nullsFirst)
  └─ Distinct
  └─ Project
  ├─ columns: [abs(one_pk.c5:1)->a:0]
  └─ Filter
  ├─ HashIn
  │   ├─ one_pk.c2:0
  │   └─ TUPLE(1 (tinyint), 11 (tinyint), 31 (tinyint))
  └─ ProcessTable
  └─ Table
  ├─ name: one_pk
  └─ columns: [c2 c5]
  after:
  Distinct
  └─ Project
  ├─ columns: [abs(one_pk.c5:5)->a:0]
  └─ Sort(a:6 ASC nullsFirst)
  └─ Project
  ├─ columns: [one_pk.pk:0!null, one_pk.c1:1, one_pk.c2:2, one_pk.c3:3, one_pk.c4:4, one_pk.c5:5, abs(one_pk.c5:5)->a:0]
  └─ Filter
  ├─ HashIn
  │   ├─ one_pk.c2:2
  │   └─ TUPLE(1 (tinyint), 11 (tinyint), 31 (tinyint))
  └─ ProcessTable
  └─ Table
  ├─ name: one_pk
  └─ columns: [pk c1 c2 c3 c4 c5]

Closed Issues

• 8964: Recurrent Panic with querying geometry types after some queries
• 8926: BUG: BEFORE UPDATE Trigger Not Working in Dolt
• 8914: MySQL is not consistent regarding last_insert_id; go-mysql-server ceased to emulate this inconsistency in v0.19.0

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.96	0.67	0.34
groupby_scan	13.46	17.63	1.31
index_join	1.5	2.43	1.62
index_join_scan	1.44	1.42	0.99
index_scan	36.89	29.72	0.81
oltp_point_select	0.18	0.26	1.44
oltp_read_only	3.49	5.09	1.46
select_random_points	0.34	0.59	1.74
select_random_ranges	0.37	0.62	1.68
table_scan	36.89	30.81	0.84
types_table_scan	80.03	112.67	1.41
reads_mean_multiplier			1.24

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.21	0.7
oltp_insert	4.1	3.07	0.75
oltp_read_write	9.06	11.24	1.24
oltp_update_index	4.18	3.13	0.75
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.77	6.21	1.08
types_delete_insert	8.43	6.55	0.78
writes_mean_multiplier			0.86

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	96.5	40.13	2.4
tpcc_tps_multiplier			2.4

Overall Mean Multiple	1.50