Skip to content

Large row sizes panic on insert #7524

@max-hoffman

Description

@max-hoffman

MySQL prevents creating tables when column sizes are too big, for example:

mysql>  create table t (x varchar(30) primary key, y varchar(65535), z varchar(65535));
ERROR 1074 (42000): Column length too big for column 'y' (max = 16383); use BLOB or TEXT instead

We permit this, and allow writes for small string sizes, but panic in different ways approaching the max row size. A setup script is below, with some of the panics I see depending on the specific y column length.

setup:

	longField := ""
	for i := 0; i < 65535; i++ {
		longField += "a"
	}
	var scripts = []queries.ScriptTest{
		{
			Name: "physical columns added after virtual one",
			SetUpScript: []string{
				"create table xy (x varchar(1) primary key, y varchar(65535));",
				fmt.Sprintf("insert into xy values ('%s', '%s')", "a", "a"),
			},
			Assertions: []queries.ScriptTestAssertion{
				{
					Query: "select count(*) from xy",
					Expected: []sql.Row{
						{1},
					},
				},
			},
		},
	}

length 65529:

panic: assertion failed: in-progress chunk must be non-empty to create chunk boundary

goroutine 144 [running]:
github.com/dolthub/dolt/go/store/prolly/tree.assertTrue(...)
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/prolly/tree/node_cursor.go:622
github.com/dolthub/dolt/go/store/prolly/tree.(*chunker[...]).handleChunkBoundary(0x1?, {0x1057ab2f0?, 0x140007d15e0?})
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/prolly/tree/chunker.go:331 +0x158
github.com/dolthub/dolt/go/store/prolly/tree.(*chunker[...]).append(0x10584d020, {0x1057ab2f0, 0x140007d15e0}, {0x14000010108, 0x4, 0x4}, {0x14001b24000, 0xfffc, 0xfffc}, 0x1)
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/prolly/tree/chunker.go:293 +0xa8

length 65535:

panic: runtime error: index out of range [65535] with length 0 [recovered]
	panic: runtime error: index out of range [65535] with length 0

goroutine 163 [running]:
testing.tRunner.func1.2({0x107b5b700, 0x140001b2288})
	/usr/local/go/src/testing/testing.go:1545 +0x1c4
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1548 +0x360
panic({0x107b5b700?, 0x140001b2288?})
	/usr/local/go/src/runtime/panic.go:914 +0x218
github.com/dolthub/dolt/go/store/val.writeByteString(...)
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/val/codec.go:591
github.com/dolthub/dolt/go/store/val.writeString(...)
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/val/codec.go:576
github.com/dolthub/dolt/go/store/val.(*TupleBuilder).PutString(0x14000ed5290, 0x14000060250?, {0x14001542000?, 0xffff})
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/val/tuple_builder.go:300 +0x20c
github.com/dolthub/dolt/go/store/prolly/tree.PutField({0x107cfb2f0, 0x1400099bb80}, {0x107d02290, 0x1400027bb80}, 0x14000ed5290, 0x0, {0x1078280a0?, 0x140000603e0?})
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/prolly/tree/prolly_fields.go:224 +0x1778
github.com/dolthub/dolt/go/libraries/doltcore/sqle/writer.prollyIndexWriter.Insert({0x14000702900, 0x14000ed4f30, {0x140001b18e8, 0x1, 0x1}, 0x14000ed5290, {0x140001b18f0, 0x1, 0x1}}, {0x107cfb2f0, ...}, ...)
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/libraries/doltcore/sqle/writer/prolly_index_writer.go:144 +0x130

length 65533:

panic: runtime error: slice bounds out of range [-2:] [recovered]
	panic: runtime error: slice bounds out of range [-2:]

goroutine 127 [running]:
testing.tRunner.func1.2({0x107193700, 0x140001b38f0})
	/usr/local/go/src/testing/testing.go:1545 +0x1c4
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1548 +0x360
panic({0x107193700?, 0x140001b38f0?})
	/usr/local/go/src/runtime/panic.go:914 +0x218
github.com/dolthub/dolt/go/store/val.writeFieldCount(...)
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/val/tuple.go:194
github.com/dolthub/dolt/go/store/val.allocateTuple({0x107324e18?, 0x10a57a233?}, 0xfffe, 0x1)
	/Users/maxhoffman/go/github.com/dolthub/dolt/go/store/val/tuple.go:119 +0xdc

length 65551:

Received unexpected error:
        	            	length is 65551 but max allowed is 16383
        	            	
        	            	github.com/dolthub/go-mysql-server/sql/types.CreateString
        	            		/Users/maxhoffman/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.17.1-0.20240220224127-651f4cc10f14/sql/types/strings.go:138
        	            	github.com/dolthub/go-mysql-server/sql/types.ColumnTypeToType

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions