Skip to content

[Bug report] spark can't access table created by spark client #6056

Closed
@Aireed

Description

Version

main branch

Describe what's wrong

  1. configure hive_catalog stored in gravitino in spark_default.conf
  2. use hive_catalog which under the metalake
  3. use {database}
  4. create a hive table
  5. we got a error although we can access the table though web of gravitino.
spark-sql> create table union_hive(id int, name string, dt string) stored as parquet;
24/12/31 20:21:22 WARN ObjectStore: Failed to get database hive_db, returning NoSuchObjectException
Error in query: Database 'hive_db' not found

image

Error message and/or stacktrace

spark-sql> create table union_hive(id int, name string, dt string) stored as parquet;
24/12/31 20:21:22 WARN ObjectStore: Failed to get database hive_db, returning NoSuchObjectException
Error in query: Database 'hive_db' not found

How to reproduce

  1. create a hive_catalog in one metalake
  2. configure the metalake
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri=http://gravitino-host:8090/
spark.sql.gravitino.metalake=hive_catalog
spark.sql.gravitino.enableIcebergSupport=true
spark.sql.warehouse.dir=hdfs://nn-1:8020/user/warehouse
spark.sql.gravitino.authType=simple
  1. use spark to access the catalog and create a hive table, we got an error althouth we can list the table
spark-sql> use hive_catalog;
24/12/31 20:19:38 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Time taken: 0.086 seconds
spark-sql> use hive_db;
Time taken: 3.332 seconds
spark-sql> create table union_hive(id int, name string, dt string) stored as parquet;
24/12/31 20:21:22 WARN ObjectStore: Failed to get database hive_db, returning NoSuchObjectException
Error in query: Database 'hive_db' not found
# we can query the table 
spark-sql> show tables;
union_hive
Time taken: 1.015 seconds, Fetched 4 row(s)

4 check whether table is created on gravitino thought web ui, found it does exist.

Additional context

No response

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions