REST Endpoints

Contents

• Data Models
  • Datastream
  • DatastreamSource
  • DatastreamDestination
  • ServerHealth
  • ServerComponentHealth
  • ConnectorHealth
  • TaskHealth
• Endpoints
  • Datastream Management Service (DMS)
    • Get a datastream
    • Get all datastreams
    • Create a datastream
    • Update one or more datastreams
    • Delete a datastream
    • Pause a datastream
    • Resume a datastream
    • Pause source partitions
    • Resume source partitions
  • Diagnostics
    • Get server component health (single instance)
    • Get server component health (all instances)
    • Get server health
    • Get consumption status for a specific connector

Data Models

Datastream

Reference schema file: com.linkedin.datastream.common.Datastream

Field Name	Type	Description
name	string	Datastream name
connectorName	string	Connector to use for consuming data from source Name must match a connector in brooklin.server.connectorNames
transportProviderName	string	TransportProvider to use for producing data to destination Name must match a transport provider in brooklin.server.transportProviderNames
Status	DatastreamStatus (enum)	Datastream status Can be one of "INITIALIZING" "READY" "PAUSED" "DELETING"
source	DatastreamSource	Source from which the Connector consumes
destination	DatastreamDestination	Destination to which the Transport Provider produces
metadata	map	Generic datastream metadata User-defined key/value pair Example usages: ownership, expiration …​ etc.

DatastreamSource

Reference schema file: com.linkedin.datastream.common.Datastream

Field Name	Type	Description
connectionString	string	Connection string representing source to consume from
partitions	int	Number of partitions of source data

DatastreamDestination

Reference schema file: com.linkedin.datastream.common.Datastream

Field Name	Type	Description
connectionString	string	Connection string representing destination to produce to
partitions	int	Number of partitions of destination data
keySerDe	string	Fully-qualified class name of serializer/deserializer Used with the key of destination data Must implement com.linkedin.datastream.serde.SerDe
payloadSerDe	string	Same as keySerDe Used with the payload of destination data
envelopeSerDe	string	Same as keySerDe Used with the envelope of destination data

ServerHealth

Reference schema file: com.linkedin.datastream.diagnostics.ServerHealth

Field Name	Type	Description
clusterName	string	Brooklin cluster name
instanceName	string	Brooklin instance name
connectors	array of ConnectorHealth	Connectors loaded in the Brooklin instance

ServerComponentHealth

Reference schema file: com.linkedin.datastream.diagnostics.ServerComponentHealth

Field Name	Type	Description
succeeded	boolean	A flag indicating whether the health query succeeded
status	string	Brooklin server component status
instanceName	string	Hostname of the Brooklin instance that handled the request
errorMessages	string	Error messages encountered (if any)

ConnectorHealth

Reference schema file: com.linkedin.datastream.diagnostics.ConnectorHealth

Field Name	Type	Description
connectorName	string	Connector name
strategy	string	Name of AssignmentStrategy impl used by the Connector
tasks	array of TaskHealth	Tasks assigned to the Connector instance

TaskHealth

Reference schema file: com.linkedin.datastream.diagnostics.TaskHealth

Field Name	Type	Description
name	string	Datastream task name
datastreams	string	Comma-separated list of datastream names associated with the task
partitions	string	Array literal of partitions associated with the task (e.g. [0, 1, 5])
source	string	Datastream source connection string
destination	string	Datastream destination connection string
statusCode	string (enum)	Datastream task status code Can be one of "OK" "ERROR" "COMPLETE" "PAUSED"
statusMessage	string	Datastream task status message
sourceCheckpoint	string	Datastream source checkpoint

Endpoints

Datastream Management Service (DMS)

Reference source files:

• com.linkedin.datastream.DatastreamRestClient

• com.linkedin.datastream.server.dms.DatastreamResources

Get a datastream

URL	GET /datastream/:datastreamName
URL Params	Required datastreamName=[string]
Response	Schema Datastream Content-Type application/json

Get all datastreams

URL	GET /datastream/
Response	Schema Array of Datastreams Content-Type application/json

Create a datastream

URL	POST /datastream/
Request Body	Content-Type application/json Schema Datastream Required Fields name connectorName source.connectionString metadata.owner Constraints New datastream name cannot match that of an existing datastream
Success Response	201 Created

Update one or more datastreams

URL	PUT /datastream?ids=List(:datastreamName1,:datastreamName2,…​)
URL Params	Constraints At least one datastream has to be specified Specified datastreams must all exist All specified datastreams must belong to the same group (i.e. have identical metadata.system.taskPrefix values) All datastreams in the same group must be updated together (no partial group updates). In the case of updating multiple datastreams, they must all have identical source and destination properties. Updating any of the following fields of a datastream is prohibited: connectorName transportProviderName destination Status Updating any datastream is subject to the approval of its Connector (and is not enabled by default).
Request Body	Content-Type application/json Schema Map of datastream name to Datastream as the value of a literal entities key { "entities": { } }
Success Response	200 OK

Delete a datastream

URL	DELETE /datastream/:datastreamName
URL Params	Required datastreamName=[string] Constraints Datastream must exist
Success Response	200 OK

Pause a datastream

URL	POST /datastream/:datastreamName?action=pause
URL Params	Required datastreamName=[string] Constraints Datastream must exist
Request Body	Optional force Description A flag indicating whether the entire datastream group (i.e. all datastreams with the same connector and system.taskPrefix metadata property value) should be paused Content-Type application/json Schema { "force" : "[boolean]" }
Success Response	200 OK

Resume a datastream

URL	POST /datastream/:datastreamName?action=resume
URL Params	Required datastreamName=[string] Constraints Datastream must exist
Request Body	Optional force Description A flag indicating whether the entire datastream group (i.e. all datastreams with the same connector and system.taskPrefix metadata property value) should be resumed Content-Type application/json Schema { "force" : "[boolean]" }
Success Response	200 OK

Pause source partitions

URL	POST datastream/:datastreamName?action=pauseSourcePartitions
URL Params	Required datastreamName=[string] Constraints Datastream must exist Datastream Status must be READY Datastream must not be part of any datastream group Datastream Connector must support this type of update
Request Body	Content-Type application/json Schema Keys and values under sourcePartitions are expected to be of type string Key should be datastream source names Value can either be: a comma-separated list of partitions (e.g. "0,4,6") an asterisk "*" (indicating all partitions) { "sourcePartitions" : { "key1": "value1", "key2": "value2" } }
Success Response	200 OK

Resume source partitions

URL	POST datastream/:datastreamName?action=resumeSourcePartitions
URL Params	Required datastreamName=[string] Constraints Datastream must exist Datastream Status must be READY Datastream must not be part of any datastream group Datastream Connector must support this type of update
Request Body	Content-Type application/json Schema Keys and values under sourcePartitions are expected to be of type string Key should be datastream source names Value can either be: a comma-separated list of partitions (e.g. "0,4,6") an asterisk "*" (indicating all partitions) { "sourcePartitions" : { "key1": "value1", "key2": "value2" } }
Success Response	200 OK

Diagnostics

Reference source files:

• com.linkedin.datastream.server.diagnostics.ServerComponentHealthResources

• com.linkedin.diagnostics.ServerComponentHealthRestClient

• com.linkedin.datastream.server.diagnostics.ServerHealthResources

• com.linkedin.datastream.server.diagnostics.ConnectorPositionReportResource

• com.linkedin.diagnostics.ConnectorPositionReportRestClient

Get server component health (single instance)

This API gets the health status of any sever component in a single Brooklin instance.

URL	GET /diag?q=status&type=:componentType&scope=:componentScope&content=:componentInput/
URL Params	Required componentType=[string] componentScope=[string] Optional componentInput=[string] Notes Only currently supported componentType is: "connector" componentScope should be set to Connector name componentInput is passed to the Connector in question. It can be utilized to pass connector-specific queries or filters. The Connector in question must implement com.linkedin.datastream.common.DiagnosticsAware to be able to respond to such queries
Response	Content-Type application/json Schema Array of ServerComponentHealth

Get server component health (all instances)

This API gets the health status of any sever component in all the instances in the same Brooklin cluster.

URL	GET /diag?q=allStatus&type=:componentType&scope=:componentScope&content=:componentInput/
URL Params	Same as Get server component health (single instance)
Response	Same as Get server component health (single instance)

Get server health

URL	GET /health/
Response	Content-Type application/json Schema ServerHealth

Get consumption status for a specific connector

URL	GET /diag?q=allStatus&type=connector&scope=:connectorName&content=position
URL Params	Required connectorName=[String] Notes The value for connectorName should be the the same as the connectorName field in the Datastream schema.
Response	Content-Type application/json Schema Array of ServerComponentHealth Notes The Connector must implement the com.linkedin.datastream.common.DiagnosticsAware interface. The field status will be populated with a JSON-serialized string containing position data as defined in the way that the Connector implements com.linkedin.datastream.common.DiagnosticsAware.