Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#3460] feat(api): Add API design for Tag system #3486

Merged
merged 6 commits into from
May 23, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions api/src/main/java/com/datastrato/gravitino/MetadataObject.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
/*
* Copyright 2024 Datastrato Pvt Ltd.
* This software is licensed under the Apache License version 2.
*/

package com.datastrato.gravitino;

import com.datastrato.gravitino.annotation.Unstable;
import javax.annotation.Nullable;

/**
* The MetadataObject is the basic unit of the Gravitino system. It represents the metadata object
* in the Gravitino system. The object can be a metalake, catalog, schema, table, topic, etc.
*/
@Unstable
public interface MetadataObject {
/**
* The type of object in the Gravitino system. Every type will map one kind of the entity of the
* underlying system.
*/
enum Type {
/**
* A metalake is a concept of tenant. It means an organization. A metalake contains many data
* sources.
*/
METALAKE,
/**
* A catalog is a collection of metadata from a specific metadata source, like Apache Hive
* catalog, Apache Iceberg catalog, JDBC catalog, etc.
*/
CATALOG,
/**
* A schema is a sub collection of the catalog. The schema can contain filesets, tables, topics,
* etc.
*/
SCHEMA,
/** A fileset is mapped to a directory on a file system like HDFS, S3, ADLS, GCS, etc. */
FILESET,
/** A table is mapped the table of relational data sources like Apache Hive, MySQL, etc. */
TABLE,
/**
* A topic is mapped the topic of messaging data sources like Apache Kafka, Apache Pulsar, etc.
*/
TOPIC,
/** A column is a sub-collection of the table that represents a group of same type data. */
COLUMN
}

/**
* The parent full name of the object. If the object doesn't have parent, this method will return
* null.
*
* @return The parent full name of the object.
*/
@Nullable
String parent();

/**
* The name of th object.
*
* @return The name of the object.
*/
String name();

/**
* The full name of th object. Full name will be separated by "." to represent a string identifier
* of the object, like catalog, catalog.table, etc.
*
* @return The name of the object.
*/
String fullName();

/**
* The type of the object.
*
* @return The type of the object.
*/
Type type();
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
*/
package com.datastrato.gravitino.authorization;

import com.datastrato.gravitino.MetadataObject;
import com.datastrato.gravitino.annotation.Unstable;
import java.util.List;
import javax.annotation.Nullable;

/**
* The securable object is the entity which access can be granted. Unless allowed by a grant, access
Expand Down Expand Up @@ -41,38 +41,7 @@
* can use add `read table` privilege for `catalog1.schema1` directly
*/
@Unstable
public interface SecurableObject {

/**
* The parent full name of securable object. If the securable object doesn't have parent, this
* method will return null.
*
* @return The parent full name of securable object.
*/
@Nullable
String parent();

/**
* The name of th securable object.
*
* @return The name of the securable object.
*/
String name();

/**
* The full name of th securable object. If the parent isn't null, the full name will join the
* parent full name and the name with `.`, otherwise will return the name.
*
* @return The name of the securable object.
*/
String fullName();

/**
* The type of securable object
*
* @return The type of securable object.
*/
Type type();
public interface SecurableObject extends MetadataObject {

/**
* The privileges of the securable object. For example: If the securable object is a table, the
Expand All @@ -82,34 +51,4 @@ public interface SecurableObject {
* @return The privileges of the role.
*/
List<Privilege> privileges();

/**
* The type of securable object in the Gravitino system. Every type will map one kind of the
* entity of the underlying system.
*/
enum Type {
/**
* A catalog is a collection of metadata from a specific metadata source, like Apache Hive
* catalog, Apache Iceberg catalog, JDBC catalog, etc.
*/
CATALOG,
/**
* A schema is a sub collection of the catalog. The schema can contain filesets, tables, topics,
* etc.
*/
SCHEMA,
/** A fileset is mapped to a directory on a file system like HDFS, S3, ADLS, GCS, etc. */
FILESET,
/** A table is mapped the table of relational data sources like Apache Hive, MySQL, etc. */
TABLE,
/**
* A topic is mapped the topic of messaging data sources like Apache Kafka, Apache Pulsar, etc.
*/
TOPIC,
/**
* A metalake is a concept of tenant. It means an organization. A metalake contains many data
* sources.
*/
METALAKE
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@
*/
package com.datastrato.gravitino.authorization;

import com.datastrato.gravitino.MetadataObject;
import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;

/** The helper class for {@link SecurableObject}. */
Expand Down Expand Up @@ -139,7 +141,11 @@ public String name() {

@Override
public String fullName() {
return toString();
if (parent != null) {
return parent + "." + name;
} else {
return name;
}
}

@Override
Expand All @@ -159,11 +165,18 @@ public int hashCode() {

@Override
public String toString() {
if (parent != null) {
return parent + "." + name;
} else {
return name;
}
String privilegesStr =
privileges.stream()
.map(p -> "[" + p.simpleString() + "]")
.collect(Collectors.joining(","));

return "SecurableObject: [fullName="
+ fullName()
+ "], [type="
+ type
+ "], [privileges="
+ privilegesStr
+ "]";
}

@Override
Expand All @@ -189,9 +202,9 @@ public boolean equals(Object other) {
* @return The created {@link SecurableObject}
*/
public static SecurableObject parse(
String fullName, SecurableObject.Type type, List<Privilege> privileges) {
String fullName, MetadataObject.Type type, List<Privilege> privileges) {
if ("*".equals(fullName)) {
if (type != SecurableObject.Type.METALAKE) {
if (type != MetadataObject.Type.METALAKE) {
throw new IllegalArgumentException("If securable object isn't metalake, it can't be `*`");
}
return SecurableObjects.ofAllMetalakes(privileges);
Expand All @@ -215,7 +228,7 @@ public static SecurableObject parse(
* @return The created {@link SecurableObject}
*/
static SecurableObject of(
SecurableObject.Type type, List<String> names, List<Privilege> privileges) {
MetadataObject.Type type, List<String> names, List<Privilege> privileges) {
if (names == null) {
throw new IllegalArgumentException("Cannot create a securable object with null names");
}
Expand All @@ -234,8 +247,8 @@ static SecurableObject of(
}

if (names.size() == 1
&& type != SecurableObject.Type.CATALOG
&& type != SecurableObject.Type.METALAKE) {
&& type != MetadataObject.Type.CATALOG
&& type != MetadataObject.Type.METALAKE) {
throw new IllegalArgumentException(
"If the length of names is 1, it must be the CATALOG or METALAKE type");
}
Expand All @@ -245,9 +258,9 @@ static SecurableObject of(
}

if (names.size() == 3
&& type != SecurableObject.Type.FILESET
&& type != SecurableObject.Type.TABLE
&& type != SecurableObject.Type.TOPIC) {
&& type != MetadataObject.Type.FILESET
&& type != MetadataObject.Type.TABLE
&& type != MetadataObject.Type.TOPIC) {
throw new IllegalArgumentException(
"If the length of names is 3, it must be FILESET, TABLE or TOPIC");
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
/*
* Copyright 2024 Datastrato Pvt Ltd.
* This software is licensed under the Apache License version 2.
*/

package com.datastrato.gravitino.exceptions;

import com.google.errorprone.annotations.FormatMethod;

/** Exception thrown when a tag with specified name is not existed. */
public class NoSuchTagException extends NotFoundException {

/**
* Constructs a new exception with the specified detail message.
*
* @param message the detail message.
* @param args the arguments to the message.
*/
@FormatMethod
public NoSuchTagException(String message, Object... args) {
super(message, args);
}

/**
* Constructs a new exception with the specified detail message and cause.
*
* @param cause the cause.
* @param message the detail message.
* @param args the arguments to the message.
*/
@FormatMethod
public NoSuchTagException(Throwable cause, String message, Object... args) {
super(cause, message, args);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
/*
* Copyright 2024 Datastrato Pvt Ltd.
* This software is licensed under the Apache License version 2.
*/

package com.datastrato.gravitino.exceptions;

import com.google.errorprone.annotations.FormatMethod;

/** Exception thrown when a tag with specified name already exists. */
public class TagAlreadyExistsException extends AlreadyExistsException {

/**
* Constructs a new exception with the specified detail message.
*
* @param message the detail message.
* @param args the arguments to the message.
*/
@FormatMethod
public TagAlreadyExistsException(String message, Object... args) {
super(message, args);
}

/**
* Constructs a new exception with the specified detail message and cause.
*
* @param cause the cause.
* @param message the detail message.
* @param args the arguments to the message.
*/
@FormatMethod
public TagAlreadyExistsException(Throwable cause, String message, Object... args) {
super(cause, message, args);
}
}
49 changes: 49 additions & 0 deletions api/src/main/java/com/datastrato/gravitino/tag/SupportsTags.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
/*
* Copyright 2024 Datastrato Pvt Ltd.
* This software is licensed under the Apache License version 2.
*/

package com.datastrato.gravitino.tag;

import com.datastrato.gravitino.annotation.Evolving;
import com.datastrato.gravitino.exceptions.NoSuchTagException;

/**
* Interface for supporting getting or associate tags to objects. This interface will be mixed with
* metadata objects to provide tag operations.
*/
@Evolving
public interface SupportsTags {

/**
* List all the tag names for the specific object.
*
* @return The list of tag names.
*/
String[] listTags();

/**
* List all the tags with details for the specific object.
*
* @return The list of tags.
*/
Tag[] listTagsInfo();

/**
* Get a tag by its name for the specific object.
*
* @param name The name of the tag.
* @return The tag.
*/
Tag getTag(String name) throws NoSuchTagException;

/**
* Associate tags to the specific object. The tagsToAdd will be added to the object, and the
* tagsToRemove will be removed from the object.
*
* @param tagsToAdd The arrays of tag name to be added to the object.
* @param tagsToRemove The array of tag name to be removed from the object.
* @return The array of tag names that are associated with the object.
*/
String[] associateTags(String[] tagsToAdd, String[] tagsToRemove);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for this method, the requestor need to know which tags are new, which are existed but to be removed, this might be a little complicated for the requestor; how about change it to "applyTags(String[] tags)", so that each time the requestor pass in a full set of the tags, no need to distinguish which are new and which are to be remove;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the current API, users still don't need to distinguish whether the tag is new or not. They can add all the tags they want to tagsToAdd, I will handle this in the backend.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, the backend will compare and recognize that, which is good for the client.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a tag appears in two arrays at the same time, what will be the result?

Copy link
Contributor Author

@jerryshao jerryshao May 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be no-op. Same as if we both enable setProperty and removeProperty on the same key for alterTable.

}
Loading
Loading