Skip to content

Revert "[SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework #27327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

brkyvz
Copy link
Contributor

@brkyvz brkyvz commented Jan 22, 2020

What changes were proposed in this pull request?

This reverts commit b5cb9ab.

Why are the changes needed?

The merged commit (#27243) was too risky for several reasons:

  1. It doesn't fix a bug
  2. It makes the resolution of the table that's going to be altered a child. We had avoided this on purpose as having an arbitrary rule change the child of AlterTable seemed risky. This change alone is a big -1 for me for this change.
  3. While the code may look cleaner, I think this approach makes certain things harder, e.g. differentiating between the Hive based Alter table CHANGE COLUMN and ALTER COLUMN syntax. Resolving and normalizing columns for ALTER COLUMN also becomes a bit harder, as we now have to check every single AlterTable command instead of just a single ALTER TABLE ALTER COLUMN statement

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing unit tests

This closes #27315

@SparkQA
Copy link

SparkQA commented Jan 23, 2020

Test build #117260 has finished for PR 27327 at commit 15868a5.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnresolvedV2Relation(
  • case class ResolvedView(identifier: Identifier) extends LeafNode
  • case class AlterTableAddColumnsStatement(
  • case class AlterTableAlterColumnStatement(
  • case class AlterTableRenameColumnStatement(
  • case class AlterTableDropColumnsStatement(
  • case class AlterTableSetPropertiesStatement(
  • case class AlterTableUnsetPropertiesStatement(
  • case class AlterTableSetLocationStatement(
  • case class AlterTable(

@brkyvz
Copy link
Contributor Author

brkyvz commented Jan 23, 2020

retest this please

@dongjoon-hyun
Copy link
Member

cc @imback82 and @cloud-fan

@cloud-fan
Copy link
Contributor

  1. I think it's pretty natural to have the table as a child of the command. We can't do it before as all tables must be resolved to v2 relation and we may have rules to catch v2 relation to do scan related optimization and break the commands. Now we have dedicated table plans for commands (ResolvedTable, ResolvedView, etc.)
  2. I don't think we can differentiate Hive based Alter table CHANGE COLUMN and ALTER COLUMN syntax. Even if revert this commit, we still create the same statement plan for these 2 syntaxes.
  3. Normalizing columns is a valid concern. It's easier to do normalization if we have a single logical plan for all ALTER TABLE commands. We can bring back the statement plans, or create AlterTable in the parser directly.

In general I don't think the original PR has a serious problem. But ALTER TABLE is a very important command and I'm fine to revert if we don't have enough confidence in it.

@SparkQA
Copy link

SparkQA commented Jan 23, 2020

Test build #117267 has finished for PR 27327 at commit 15868a5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnresolvedV2Relation(
  • case class ResolvedView(identifier: Identifier) extends LeafNode
  • case class AlterTableAddColumnsStatement(
  • case class AlterTableAlterColumnStatement(
  • case class AlterTableRenameColumnStatement(
  • case class AlterTableDropColumnsStatement(
  • case class AlterTableSetPropertiesStatement(
  • case class AlterTableUnsetPropertiesStatement(
  • case class AlterTableSetLocationStatement(
  • case class AlterTable(

Copy link
Member

@gatorsmile gatorsmile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gatorsmile
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants