[Bug report] spark can't access table created by spark client #6056
Closed
Description
Version
main branch
Describe what's wrong
- configure hive_catalog stored in gravitino in spark_default.conf
- use hive_catalog which under the metalake
- use {database}
- create a hive table
- we got a error although we can access the table though web of gravitino.
spark-sql> create table union_hive(id int, name string, dt string) stored as parquet;
24/12/31 20:21:22 WARN ObjectStore: Failed to get database hive_db, returning NoSuchObjectException
Error in query: Database 'hive_db' not found
Error message and/or stacktrace
spark-sql> create table union_hive(id int, name string, dt string) stored as parquet;
24/12/31 20:21:22 WARN ObjectStore: Failed to get database hive_db, returning NoSuchObjectException
Error in query: Database 'hive_db' not found
How to reproduce
- create a hive_catalog in one metalake
- configure the metalake
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri=http://gravitino-host:8090/
spark.sql.gravitino.metalake=hive_catalog
spark.sql.gravitino.enableIcebergSupport=true
spark.sql.warehouse.dir=hdfs://nn-1:8020/user/warehouse
spark.sql.gravitino.authType=simple
- use spark to access the catalog and create a hive table, we got an error althouth we can list the table
spark-sql> use hive_catalog;
24/12/31 20:19:38 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Time taken: 0.086 seconds
spark-sql> use hive_db;
Time taken: 3.332 seconds
spark-sql> create table union_hive(id int, name string, dt string) stored as parquet;
24/12/31 20:21:22 WARN ObjectStore: Failed to get database hive_db, returning NoSuchObjectException
Error in query: Database 'hive_db' not found
# we can query the table
spark-sql> show tables;
union_hive
Time taken: 1.015 seconds, Fetched 4 row(s)
4 check whether table is created on gravitino thought web ui, found it does exist.
Additional context
No response