[#1193] improvement(docs): Add the document about how to debug trino connector locally. #2446

yuqi1129 · 2024-03-07T01:59:55Z

What changes were proposed in this pull request?

Add a document about how to debug develop and debug trino-connector locally.

Why are the changes needed?

To make users more knowledgeable about Gravitino trino connector.

Fix: #1193

Does this PR introduce any user-facing change?

N/A.

How was this patch tested?

N/A.

xiacongling · 2024-03-07T14:46:22Z

docs/assets/trino/pom.xml

+        <dependency>
+            <groupId>com.datastrato.gravitino</groupId>
+            <artifactId>gravitino-client-java-runtime</artifactId>
+            <version>0.4.0-SNAPSHOT</version>
+        </dependency>
+
+        <dependency>
+            <groupId>com.datastrato.gravitino</groupId>
+            <artifactId>bundled-catalog</artifactId>
+            <version>0.4.0-SNAPSHOT</version>
+        </dependency>
+
+        <dependency>
+            <groupId>com.datastrato.gravitino</groupId>
+            <artifactId>catalog-common</artifactId>
+            <version>0.4.0-SNAPSHOT</version>
+        </dependency>


these packages are not in maven central repository. can you publish them? or can we build and publish them locally from source code?

shall we use current snapshots (0.5.0-SNAPSHOT)?

I believe you can install them with the latest version 0.5.0-SNAPSHOT locally.

yuqi1129 · 2024-03-08T07:39:40Z

@diqiu50 ,
Could you kindly review it for me?

xiacongling · 2024-03-09T03:16:26Z

docs/assets/trino/pom.xml

+
+        <dependency>
+            <groupId>com.datastrato.gravitino</groupId>
+            <artifactId>gravitino-client-java-runtime</artifactId>


there is no such artifact after a local publish. do you mean client-java-runtime?

I suggest providing a script to install Graviton-related dependencies into the local Maven repository.

there is no such artifact after a local publish. do you mean client-java-runtime?

Exactly.

I suggest providing a script to install Graviton-related dependencies into the local Maven repository.

I will convert the artifacts to the standard release versions and the user can download them from Maven central repo.

Using the Maven Central Repository is a good choice. We need to consider situations where a user modifies a class in the client-java-runtime. In such cases, the user might not be able to install the new package.

OK, a local repo is necessary.

docs/assets/trino/pom.xml

xiacongling · 2024-03-09T03:20:48Z

docs/trino-connector/development.md

+2. Create a catalog in the Gravitino server, for more information, please refer to the [Gravitino metadata management](../manage-metadata-using-gravitino.md). Assuming we have just created a MySQL catalog using the following command:
+
+```curl
+curl -X POST -H "Content-Type: application/json" -d '{"name":"test","comment":"comment","properties":{}}' http://localhost:8090/api/metalakes
+
+curl -X POST -H "Content-Type: application/json" -d '{"name":"mysql_catalog3","type":"RELATIONAL","comment":"comment","provider":"jdbc-mysql", "properties":{
+  "jdbc-url": "jdbc:mysql://127.0.0.1:3306?useSSL=false&allowPublicKeyRetrieval=true",
+  "jdbc-user": "root",
+  "jdbc-password": "123456",
+  "jdbc-driver": "com.mysql.cj.jdbc.Driver"
+}}' http://localhost:8090/api/metalakes/test/catalogs
+```


catalogs are innecessary for setting up dev environment.

Can you use the Trino command line to display the Gravition catalog if you do not create catalogs here?

yes. i just start gravitino server built from source, and add a test empty metalake. on trino side, there is only gravitino.properties under the catalog dir, and Trino dev server started up normally.

I mean if we could list the catalogs in the format like "test.mysql_catalog" by the Trino client.

docs/trino-connector/development.md

xiacongling · 2024-03-09T03:57:01Z

Setting up with Trino 421 successfully. I've encountered some problems during the setup:

recommanded to use Trino ≥426 (unfortunately, we are using Trino 421). in my case, ConnectorMetadata.dropSchema has only 2 arguments (cascade dropping was not supportted yet and this interface method was removed in trino 426), so i need to update gravitino's implementation, rebuild the project and publish artifacts to local maven repository;
struggled with signing for some time when publishing my own artifactories to local repository, can we make signing an optional task and skip it by default? we only need to apply signing in public releases, right?

yuqi1129 · 2024-03-09T06:23:29Z

@diqiu50

recommanded to use Trino ≥426 (unfortunately, we are using Trino 421). in my case, ConnectorMetadata.dropSchema has only 2 arguments (cascade dropping was not supportted yet and this interface method was removed in trino 426), so i need to update gravitino's implementation, rebuild the project and publish artifacts to local maven repository;

About this point, please take a look if we need to modify the supported versions, as we only test and verify versions newer than 426.

struggled with signing for some time when publishing my own artifactories to local repository, can we make signing an optional task and skip it by default? we only need to apply signing in public releases, right?

Yeah, it would be great help if we can skip the signing stage. currently, you can use maven to install to your local repo, or can you work on it?

diqiu50 · 2024-03-11T02:02:10Z

@diqiu50

recommanded to use Trino ≥426 (unfortunately, we are using Trino 421). in my case, ConnectorMetadata.dropSchema has only 2 arguments (cascade dropping was not supportted yet and this interface method was removed in trino 426), so i need to update gravitino's implementation, rebuild the project and publish artifacts to local maven repository;

About this point, please take a look if we need to modify the supported versions, as we only test and verify versions newer than 426.

struggled with signing for some time when publishing my own artifactories to local repository, can we make signing an optional task and skip it by default? we only need to apply signing in public releases, right?

Yeah, it would be great help if we can skip the signing stage. currently, you can use maven to install to your local repo, or can you work on it?

I have verified it; indeed, the dropSchema interface in Trino-421 does not include 'cascade'. We need to update the documentation.

I think we need to conduct a test later to check the compatibility with different Trino versions.

diqiu50 · 2024-03-11T02:39:30Z

docs/trino-connector/development.md

+5. Change the `pom.xml` file in the `trino-gravitino` module accordingly. This is a [example](../assets/trino/pom.xml) of the `pom.xml` file in the `trino-gravitino` module.
+6. Try to compile module `trino-gravitino` to see if there are any errors. 
+```shell
+mvn clean -pl 'plugin/trino-gravitino' package -DskipTests -Dcheckstyle.skip -Dair.check.skip-checkstyle=true -DskipTests -Dair.check.skip-all=true


We need to build the Trino as well

I have an error:

[INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal io.trino:trino-maven-plugin:12:generate-service-descriptor (default-generate-service-descriptor) on project trino-graviton: [ERROR] [ERROR] Existing service descriptor for io.trino.spi.Plugin found in output directory. [ERROR] -> [Help 1]

The solution is remove the file trino-connector/src/main/resources/META-INF/services/io.trino.spi.Plugin
Do you have this problem

you can install trino modules (exluding gravitino) at first, using:
mvn -pl '!plugin/trino-gravitino' clean install -DskipTests

Ok I understand that. My question is whether anyone else has encountered this error.
@yuqi1129 The steps on how to build Trino are missing.

@diqiu50 I have added it.

yuqi1129 · 2024-03-11T08:29:45Z

@diqiu50
Please help to look again.

diqiu50 · 2024-03-13T03:04:51Z

docs/trino-connector/development.md

+
+1. Clone the Trino repository from the [GitHub](https://github.com/trinodb/trino) repository. We advise you to use the release version 426 or 435. 
+2. Open the Trino project in your IDEA.
+3. Create a new module for the Gravitino connector in the Trino project as the following picture (we will use the name `trino-gravitino` as the module name in the following steps). ![trino-gravitino](../assets/trino/create-gravitino-connector.jpg)


Do users know how to create a module?

This should be tiny things compared to developing The Trino connector. If users lack this knowledge, I don't think they can develop the connector effectively.

diqiu50 · 2024-03-13T03:10:16Z

docs/trino-connector/development.md

+java -jar trino-cli-429-executable.jar --server localhost:8180
+```
+:::note
+The `trino-cli-429-executable.jar` is the Trino CLI jar file, you can download it from the [Trino release page](https://trino.io/download.html).


Why do we use version 429?

It's just an example. It works well for me with version 429, and another version might be just as good, I will add more details and let users choose the package version.

yuqi1129 · 2024-03-13T09:34:46Z

@diqiu50
I have updated the PR, please help take a look if you are free.

diqiu50

LGTM

…trino connector locally. (apache#2446) ### What changes were proposed in this pull request? Add a document about how to debug develop and debug `trino-connector` locally. ### Why are the changes needed? To make users more knowledgeable about Gravitino trino connector. Fix: apache#1193 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? N/A.

Add the document about how to debug trino connector locally.

2a535aa

yuqi1129 self-assigned this Mar 7, 2024

yuqi1129 mentioned this pull request Mar 7, 2024

[Improvement] Add how to develop Gravitino trino connector for new developers #1193

Closed

Optimize code.

0c16422

xiacongling reviewed Mar 7, 2024

View reviewed changes

Add more picture.

6e40abe

yuqi1129 requested a review from xiacongling March 8, 2024 07:39

xiacongling suggested changes Mar 9, 2024

View reviewed changes

diqiu50 reviewed Mar 11, 2024

View reviewed changes

yuqi1129 added 2 commits March 11, 2024 15:45

Fix again

eeb2d25

Fixed again

86a1ff4

xiacongling mentioned this pull request Mar 11, 2024

[Improvement] skip signing for non-public publishing #2504

Open

Refresh the PR

6e6d3a1

diqiu50 reviewed Mar 13, 2024

View reviewed changes

yuqi1129 added 2 commits March 13, 2024 17:29

Merge branch 'main' of github.com:datastrato/gravitino into issue_1193

13b6c39

Rebase main and fix changes.

56180b9

diqiu50 reviewed Mar 14, 2024

View reviewed changes

diqiu50 approved these changes Mar 14, 2024

View reviewed changes

yuqi1129 merged commit b154519 into apache:main Mar 14, 2024
5 checks passed

yuqi1129 deleted the issue_1193 branch March 14, 2024 03:20

yuqi1129 mentioned this pull request Mar 16, 2024

[Improvement] Add document about how to develop Gravitino Trino connector in IDE #2087

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#1193] improvement(docs): Add the document about how to debug trino connector locally. #2446

[#1193] improvement(docs): Add the document about how to debug trino connector locally. #2446

yuqi1129 commented Mar 7, 2024

xiacongling Mar 7, 2024 •

edited

Loading

xiacongling Mar 7, 2024

yuqi1129 Mar 7, 2024

yuqi1129 commented Mar 8, 2024

xiacongling Mar 9, 2024

diqiu50 Mar 11, 2024

yuqi1129 Mar 11, 2024

yuqi1129 Mar 11, 2024

diqiu50 Mar 12, 2024

yuqi1129 Mar 12, 2024

xiacongling Mar 9, 2024

yuqi1129 Mar 11, 2024

xiacongling Mar 11, 2024

yuqi1129 Mar 11, 2024

yuqi1129 Mar 11, 2024

xiacongling commented Mar 9, 2024

yuqi1129 commented Mar 9, 2024

diqiu50 commented Mar 11, 2024

diqiu50 Mar 11, 2024

diqiu50 Mar 11, 2024

xiacongling Mar 11, 2024

diqiu50 Mar 12, 2024

yuqi1129 Mar 12, 2024

yuqi1129 commented Mar 11, 2024

diqiu50 Mar 13, 2024 •

edited

Loading

yuqi1129 Mar 13, 2024

diqiu50 Mar 13, 2024

yuqi1129 Mar 13, 2024

yuqi1129 commented Mar 13, 2024

diqiu50 left a comment

[#1193] improvement(docs): Add the document about how to debug trino connector locally. #2446

[#1193] improvement(docs): Add the document about how to debug trino connector locally. #2446

Conversation

yuqi1129 commented Mar 7, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

xiacongling Mar 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuqi1129 commented Mar 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiacongling commented Mar 9, 2024

yuqi1129 commented Mar 9, 2024

diqiu50 commented Mar 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuqi1129 commented Mar 11, 2024

diqiu50 Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuqi1129 commented Mar 13, 2024

diqiu50 left a comment

Choose a reason for hiding this comment

xiacongling Mar 7, 2024 •

edited

Loading

diqiu50 Mar 13, 2024 •

edited

Loading