You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-50855][CONNECT][TESTS][FOLLOWUP] Refactor TransformWithStateConnectSuite to run DROP TABLE IF EXISTS my_sink in beforeAll/afterEach
### What changes were proposed in this pull request?
This PR refactors the `TransformWithStateConnectSuite`:
1. Overrides the `beforeAll` method to execute `spark.sql("DROP TABLE IF EXISTS my_sink")`, ensuring that no table named `my_sink` exists before the test cases in `TransformWithStateConnectSuite` are executed.
2. Overrides the `afterEach` method to execute `spark.sql("DROP TABLE IF EXISTS my_sink")`, ensuring that any potential `my_sink` table is cleaned up after each test case in `TransformWithStateConnectSuite` is executed.
3. Removes the calls to `spark.sql("DROP TABLE IF EXISTS my_sink")` within the test cases.
### Why are the changes needed?
The PR at apache#49488 introduced the `TransformWithStateConnectSuite`, in which the test case `transformWithState - batch query` did not delete the table `my_sink` after execution. Additionally, due to inheriting from `RemoteSparkSession`, the test cases shared a connect server, which caused the test case `Table APIs` in the `CatalogSuite` to fail during the Maven daily test:
- https://github.com/apache/spark/actions/runs/13654375212/job/38169921062

Therefore, this PR refactors the `TransformWithStateConnectSuite` to ensure that the table named `my_sink` does not exist before or after the execution of the tests in `TransformWithStateConnectSuite`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual check:
```
build/mvn -DskipTests -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl clean install
build/mvn test -pl sql/connect/client/jvm -fae
```
Before
```
CatalogSuite:
- Database APIs
- CatalogMetadata APIs
- Table APIs *** FAILED ***
Array(Table[name='my_sink', catalog='spark_catalog', database='default', tableType='MANAGED', isTemporary='false']) was not empty (CatalogSuite.scala:91)
Run completed in 3 minutes, 20 seconds.
Total number of tests run: 1474
Suites: completed 36, aborted 0
Tests: succeeded 1473, failed 1, canceled 0, ignored 6, pending 0
*** 1 TEST FAILED ***
```
After
```
Run completed in 3 minutes, 22 seconds.
Total number of tests run: 1474
Suites: completed 36, aborted 0
Tests: succeeded 1474, failed 0, canceled 0, ignored 6, pending 0
All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closesapache#50155 from LuciferYang/SPARK-50855-FOLLOWUP.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Copy file name to clipboardExpand all lines: sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/streaming/TransformWithStateConnectSuite.scala
0 commit comments