Skip to content

Commit

Permalink
Sqoop import all tables
Browse files Browse the repository at this point in the history
  • Loading branch information
RajasekarSribalan authored Jul 7, 2018
1 parent 8218ad2 commit eb65fc0
Showing 1 changed file with 58 additions and 0 deletions.
58 changes: 58 additions & 0 deletions Command
Original file line number Diff line number Diff line change
Expand Up @@ -886,3 +886,61 @@ so we need to manually delete it.



***** Import all table *****

use Warehouse-dir ,since it will create sub directory with table names.

sqoop import-all-tables \
--connect "jdbc:mysql://localhost:3306/mytrainingdb" \
--username root \
--password root \
--warehouse-dir /sqoop/alltables/ \
--delete-target-dir

Note: If there are any tables with no primary key,then we cannot have more than one mappers or we cannot use the deafult splits.It will fail in mid way.
Ley us test it that way first.

Each table must have a single-column primary key.
You must intend to import all columns of each table.
You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.
No incremental load is possible

We get error as below,

18/07/07 19:30:56 ERROR tool.BaseSqoopTool: Error parsing arguments for import-all-tables:
18/07/07 19:30:56 ERROR tool.BaseSqoopTool: Unrecognized argument: --delete-target-dir

So for --import-all-tables,we canno use --delete-target-dir and few others listed below,
--table, --split-by, --columns, and --where

So, let us run now,

sqoop import-all-tables \
--connect "jdbc:mysql://localhost:3306/mytrainingdb" \
--username root \
--password root \
--warehouse-dir /sqoop/alltables/


Note: Recompile with -Xlint:deprecation for details.
18/07/07 19:36:17 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hduser/compile/20a4ac617bb7f2680c617173c408ea19/COMPLETED_TXN_COMPONENTS.jar
18/07/07 19:36:17 ERROR tool.ImportAllTablesTool: Error during import: No primary key could be found for table COMPLETED_TXN_COMPONENTS. Please specify one with --split-by or perform a sequential import with '-m 1'.



Since one of the table doesnt have primary key, we are getting above error.

Hence we have to set --auto-reset-to-one-mapper
So,

sqoop import-all-tables \
--connect "jdbc:mysql://localhost:3306/mytrainingdb" \
--username root \
--password root \
--warehouse-dir /sqoop/alltables/ \
--autoreset-to-one-mapper

sqoop list-tables \
--connect "jdbc:mysql://localhost:3306/mytrainingdb" \
--username root \
--password root

0 comments on commit eb65fc0

Please sign in to comment.