Cant merge and because dolt does not let me resolve conflicts

[adagrad]

I try to merge changes into my main branch:

mnt/findb$ dolt merge schema
Updating d1lhsk9v9m9c0947467q7gs1nroo1q6h..n5crct6h4rh5u1qhduf9j55t9ehs65s4
Unable to stage changes: add and commit to finish merge
error: Failed to commit changes.
cause: schema conflicts for table yfinance_symbol_info:
       	two columns with the name 'start_date'

Ok we got some conflict (which we really should not) but let's try to resolve them:

/mnt/findb$ dolt status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean

Looks like we just can't.

dolt clone adagrad/findb
dolt merge origin/schema

[adagrad]

I have branched of 2 new branches with the current state

• schema/issues/3950
• main/issues/3950

Meanwhile, on the original branches, I try my best to get them fixed. Eventually, a drop and recreation will fix the problem.

[timsehn]

If you get conflicts, you should be able to resolve on the command line with dolt conflicts resolve --ours or dolt conflicts resolve --theirs.

It's weird that dolt status is not showing a conflict. Let me see if I can repro.

[timsehn]

I was able to repro the dolt status problem:

adagrad_bug $ dolt init
Successfully initialized dolt data repository.
adagrad_bug $ dolt sql -q "create table bug (pk int, primary key(pk))";
adagrad_bug $ dolt commit -am "Created table"
commit gnnhjn8bhc57b902ir5uvaba3cokerhc (HEAD -> main) 
Author: Tim Sehn <tim@dolthub.com>
Date:  Thu Jul 28 09:06:29 -0700 2022

        Created table

adagrad_bug $ dolt checkout -b branch
Switched to branch 'branch'
adagrad_bug $ dolt sql -q "alter table bug add column start_date int" 
adagrad_bug $ dolt commit -am "Added start_date column as int"
commit 6mhssfctb9bhcvl6qenpver5u8a457ml (HEAD -> branch) 
Author: Tim Sehn <tim@dolthub.com>
Date:  Thu Jul 28 09:07:40 -0700 2022

        Added start_date column as int

adagrad_bug $ dolt checkout main
Switched to branch 'main'
adagrad_bug $ dolt sql -q "alter table bug add column start_date date" 
adagrad_bug $ dolt commit -am "Added start_date column as date"
commit 3fpgd72s4eped8jao4e4mvdl1rue2341 (HEAD -> main) 
Author: Tim Sehn <tim@dolthub.com>
Date:  Thu Jul 28 09:08:11 -0700 2022

        Added start_date column as date

adagrad_bug $ dolt merge branch
'Updating 3fpgd72s4eped8jao4e4mvdl1rue2341..6mhssfctb9bhcvl6qenpver5u8a457ml
Unable to stage changes: add and commit to finish merge
error: Failed to commit changes.
cause: schema conflicts for table bug:
       	two columns with the name 'start_date'
       
adagrad_bug $ dolt status
On branch main
nothing to commit, working tree clean

And it also looks like we've broken schema conflict resolution on the command line.

adagrad_bug $ dolt conflicts cat bug
adagrad_bug $ adagrad_bug $ dolt conflicts resolve bug --theirs
adagrad_bug $ dolt sql -q "show create table bug"
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                         |
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| bug   | CREATE TABLE `bug` (
  `pk` int NOT NULL,
  `start_date` date,
  PRIMARY KEY (`pk`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_bin |
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------+
adagrad_bug $

We recently changed this logic because in SQL there is no way to resolve schema conflicts. We either need to:

1. Give a different message like "Merge aborted. Schema conflicts cannot be resolved."
2. Make schema conflicts resolveable on the command line.

[timsehn]

@druvv will get to this today and we'll get a release out for it.

[adagrad]

I cant even merge after I have dropped the table in main

/mnt/findb$ dolt sql -q "show tables;"
+---------------------+
| Tables_in_findb     |
+---------------------+
| tzinfo_exchange     |
| yfinance_quote      |
| yfinance_quote_meta |
| yfinance_symbol     |
+---------------------+
/mnt/findb$ dolt merge schema
Updating 48vrdn3cg4edsup4e1upjjqulvnsfvtc..5imnbbn9di7pv2igjfpbhk8o2vugubjr
Unable to stage changes: add and commit to finish merge
error: Failed to commit changes.
cause: conflict: table with same name deleted and modified

[timsehn]

I cant even merge after I have dropped the table in main

It's better to give us reproduction steps. I'm not sure I would expect a branch with a merge base of table exists with branch a dropping the table and branch b modifying the table to merge.

[timsehn]

If you cut a branch off the tip where the table does not exist and then do the modifications on a branch, that will merge.

[adagrad]

Interestingly I never had problems and I did similar things quite often as long as the tables looked identical they always merged ... but now I am basically stuck.

reproduction steps are:

dolt clone adagrad/findb
dolt checkout main/issues/3950
dolt merge origin/schema/issues/3950

[timsehn]

I'm trying to help with this in the next couple hours.

[timsehn]

It looks like from the diff, all you diff was rename 'fullTimeEmployees' to 'full_time_employees'. We're not sure why this is throwing a conflict.

[adagrad]

aha, meanwhile i was able to just recreate the table and the merge went through ... puh

[timsehn]

Sorry for the poor experience. We're looking into what is going on. Glad it is not urgent any more :-)

[adagrad]

No worries, I am happy that you respond so quickly.

[druvv]

Hi @adagrad, there is definitely a bug here. We use an internal tag system (soon to be deprecated) to identify a column. We expect that if you changed a column from one type to another (say INT to VARCHAR) and then change it back (VARCHAR to INT) that you would see the same tag that you started with.

As you can see here that is not the case:

$ dolt sql -q "CREATE TABLE t (pk INT PRIMARY KEY, col1 int);"
$ dolt schema tags
+-------+--------+-------+
| table | column | tag   |
+-------+--------+-------+
| t     | pk     | 15476 |
| t     | col1   | 10878 |
+-------+--------+-------+
$ dolt sql -q "ALTER TABLE t MODIFY COLUMN col1 VARCHAR(100);"
$ dolt schema tags
+-------+--------+-------+
| table | column | tag   |
+-------+--------+-------+
| t     | pk     | 15476 |
| t     | col1   | 11127 |
+-------+--------+-------+
$ dolt sql -q "ALTER TABLE t MODIFY COLUMN col1 int;"
$ dolt schema tags
+-------+--------+-------+
| table | column | tag   |
+-------+--------+-------+
| t     | pk     | 15476 |
| t     | col1   | 9427  | <- does not equal the original tag
+-------+--------+-------+

This alone wouldn't create a conflict, but because there were other modifications between when the column was first created and when the MODIFY statements were issued, the tags misaligned between the branches. And when two columns have the same name but a different tag, a schema conflict is thrown.

We are going to eliminate the tag system eventually, but I wanted to provide some insight into what happened here.