Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update json-jq-transform-task.md #261

Merged
merged 5 commits into from
Jan 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/reference-docs/system-tasks/event-task.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Conductor supports the following eventing models:

### Use Cases
Consider a use case where at some point in the execution, an event is published to an external eventing system such as SQS.
Event tasks are useful for creating event based dependencies for workflows and tasks.
Event tasks are useful for creating event-based dependencies for workflows and tasks.

Consider an example where we want to publish an event into SQS to notify an external system.

Expand All @@ -28,7 +28,7 @@ Consider an example where we want to publish an event into SQS to notify an exte
}
```

An example where we want to publish a message to conductor's internal queuing system.
An example is when we want to publish a message to Conductor's internal queuing system.
```json
{
"type": "EVENT",
Expand All @@ -50,7 +50,7 @@ An example where we want to publish a message to conductor's internal queuing sy
| sink | External event queue in the format of `prefix:location`. The Prefix is either `sqs` or `conductor`, and `location` specifies the actual queue name. e.g. "sqs:send_email_queue" |

#### Output Configuration
The Tasks' output is sent as a payload to the external event. In case of SQS, the task's output is sent as a payload to the SQS message.
The Tasks' output is sent as a payload to the external event. In the case of SQS, the task's output is sent as a payload to the SQS message.

### Supported Queuing Systems
Conductor has support for the following external event queueing systems as part of the OSS build
Expand Down
45 changes: 22 additions & 23 deletions docs/reference-docs/system-tasks/http-task.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ sidebar_position: 1

## Introduction

An HTTP task is useful when you have a requirements such as
An HTTP task is useful when you have requirements such as

1. Making calls to another service that exposes an API via HTTP
2. Fetch any resource or data present on an endpoint
Expand All @@ -19,10 +19,9 @@ An HTTP task is useful when you have a requirements such as

## Use Cases

If we have a scenario where we need to make an HTTP call into another service, we can make use of HTTP tasks. You can
use the data returned from the HTTP call in your subsequent tasks as inputs. Using HTTP tasks you can avoid having to
write the code that talks to these services and instead let Conductor manage it directly. This can reduce the code you
have to maintain and allows for a lot of flexibility.
If we have a scenario where we need to make an HTTP call to another service, we can use HTTP tasks. You can
use the data returned from the HTTP call in your subsequent tasks as inputs. Using HTTP tasks, you can avoid having to
write the code that talks to these services and instead let the Conductor manage it directly. This can reduce the code you must maintain and allow for much flexibility.

## Configuration

Expand All @@ -36,14 +35,14 @@ HTTP task is defined directly inside the workflow with the task type `HTTP`.

|Name|Type|Example|Description|
|---|---|---|---|
| uri | String || URI for the service. Can be a partial when using vipAddress or includes the server address.|
| uri | String || URI for the service. It can be a partial value when using vipAddress or includes the server address.|
| method | String || HTTP method. One of the GET, PUT, POST, DELETE, OPTIONS, HEAD|
| accept | String || Accept header as required by server. Defaults to ```application/json``` |
| accept | String || Accept header as required by the server. Defaults to ```application/json``` |
| contentType | String || Content Type - supported types are ```text/plain```, ```text/html```, and ```application/json``` (Default)|
| headers| Map[String, Any] || A map of additional http headers to be sent along with the request.|
| body| Map[] || Request body |
| vipAddress | String || When using discovery based service URLs.|
| asyncComplete | Boolean |TODO: Link to details| ```false``` to mark status COMPLETED upon execution ; ```true``` to keep it IN_PROGRESS, wait for an external event (via Conductor or SQS or EventHandler) to complete it.
| vipAddress | String || When using discovery-based service URLs.|
| asyncComplete | Boolean |TODO: Link to details| ```false``` to mark status COMPLETED upon execution; ```true``` to keep it IN_PROGRESS, wait for an external event (via Conductor or SQS or EventHandler) to complete it.
| oauthConsumerKey | String || [OAuth](https://oauth.net/core/1.0/) client consumer key |
| oauthConsumerSecret | String || [OAuth](https://oauth.net/core/1.0/) client consumer secret |
| connectionTimeOut | Integer || Connection Time Out in milliseconds. If set to 0, equivalent to infinity. Default: 100. |
Expand All @@ -55,14 +54,14 @@ HTTP task is defined directly inside the workflow with the task type `HTTP`.
|---|---|---|
| response | Map | JSON body containing the response if one is present |
| headers | Map[String, Any] | Response Headers |
| statusCode | Integer | [Http Status Code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) |
| reasonPhrase | String | Http Status Code's reason phrase |
| statusCode | Integer | [HTTP Status Code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) |
| reasonPhrase | String | HTTP Status Code's reason phrase |

## Examples

Following is the example of HTTP task with `GET` method.
Following is an example of an HTTP task with the `GET` method.

We can use variables in our URI as show in the example below. For details on how to use inputs, refer to
We can use variables in our URI, as show in the example below. For details on how to use inputs, refer to
the [Task Inputs page](/content/docs/how-tos/Tasks/task-inputs).

```json
Expand All @@ -79,7 +78,7 @@ the [Task Inputs page](/content/docs/how-tos/Tasks/task-inputs).
}
```

Following is the example of HTTP task with `POST` method.
Following is an example of an HTTP task with the `POST` method.

> Here we are using variables for our POST body which happens to be data from a previous task. This is an example of how you can **chain** HTTP calls to make complex flows happen without writing any additional code.

Expand All @@ -102,7 +101,7 @@ Following is the example of HTTP task with `POST` method.
}
```

Following is the example of HTTP task with `PUT` method.
Following is an example of an HTTP task with the `PUT` method.

```json
{
Expand All @@ -123,7 +122,7 @@ Following is the example of HTTP task with `PUT` method.
}
```

Following is the example of HTTP task with `DELETE` method.
Following is an example of an HTTP task with the `DELETE` method.

```json
{
Expand All @@ -139,26 +138,26 @@ Following is the example of HTTP task with `DELETE` method.
}
```
### Codelabs with HTTP tasks
* [Running first workflow](/content/docs/getting-started/run/running-first-workflow) uses HTTP Task to call an API.
* [Hello World Pt.2](/content/docs/codelab/helloworld2#the-http-task) uses the HTTP task to send an IP Address and receive a location.
* [Order Fulfillment Pt 4](/content/docs/codelab/orderfulfillment4) calls an API to reorder widgets.
* [Sequential HTTP Tasks](/content/docs/codelab/sequentialHTTPtasks) Makes 2 HTTP tasks. Data from the first response is used as input in the 2nd task.
* [Running the first workflow](/content/docs/getting-started/run/running-first-workflow) uses HTTP Task to call an API.
* [Hello World Part 2](/content/docs/codelab/helloworld2#the-http-task) uses the HTTP task to send an IP Address and receive a location.
* [Order Fulfillment Part4](/content/docs/codelab/orderfulfillment4) calls an API to reorder widgets.
* [Sequential HTTP Tasks](/content/docs/codelab/sequentialHTTPtasks) makes two HTTP tasks. Data from the first response is used as input in the 2nd task.


## Best Practices

1. Why are my HTTP tasks not getting picked up?
1. We might have too many HTTP tasks in the queue. There is a concept called Isolation Groups that you can rely on
for prioritizing certain HTTP tasks over others. Read more here: [Isolation Groups](https://conductor.netflix.com/configuration/isolationgroups.html)
for prioritizing certain HTTP tasks over others. Read more here: [Isolation Groups](https://conductor.netflix.com/configuration/isolationgroups.html).
2. Why is my HTTP Task timing out with ```Failed to invoke HTTP task due to: java.lang.Exception: I/O error on GET request for "<url>": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out```?
1. The default timeout for an HTTP request is 150ms. If your API takes longer than this, you will need to increase the timeout parameters. In your ```inputParameters``` under ```http_request``` add the two following parameters (the timeouts are in milliseconds):
1. The default timeout for an HTTP request is 150ms. If your API takes longer than this, you will need to increase the timeout parameters. In your ```inputParameters``` under ```http_request```, add the two following parameters (the timeouts are in milliseconds):
```json
"connectionTimeOut": 1000,
"readTimeOut": 1000
```

3. Can I retry my HTTP Tasks?
1. Yes. You can add retries, and retry parameters to your HTTP Task.
1. Yes. You can add retries and retry parameters to your HTTP Task.

4. I'm getting rate limited. Can I slow down my HTTP Task?
1. Yes! By extending system tasks and adding the following parameters:
Expand Down
4 changes: 2 additions & 2 deletions docs/reference-docs/system-tasks/inline-task.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The inline task helps execute necessary logic at the workflow run-time using an

## Use Cases

Consider a scenario we have to run simple evaluations in the Conductor server while creating Workers. An inline task can be used to run these evaluations using an evaluator engine.
Consider a scenario where we have to run simple evaluations in the Conductor server while creating Workers. An inline task can be used to run these evaluations using an evaluator engine.

## Configuration
```json
Expand All @@ -34,7 +34,7 @@ Consider a scenario we have to run simple evaluations in the Conductor server wh
}
```

Following are the parameters in the above example:
The following are the parameters in the above example:

1. `"evaluatorType"` - Type of the evaluator. Supported evaluators: graaljs, javascript, which evaluates the javascript expression.

Expand Down
40 changes: 19 additions & 21 deletions docs/reference-docs/system-tasks/json-jq-transform-task.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,35 +9,33 @@ sidebar_position: 1
```
## Introduction

The JSON_JQ_TRANSFORM task is a System task that allows processing of JSON data that is supplied to the task, by using the
The JSON_JQ_TRANSFORM task is a System task that allows the processing of JSON data that is supplied to the task by using the
popular JQ processing tool’s query expression language.

<p align="center"><iframe width="560" height="315" src="https://www.youtube.com/embed/amUiNTMgTkE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></p>

Check the [JQ Manual](https://stedolan.github.io/jq/manual/v1.5/), and the
[JQ Playground](https://jqplay.org/) for more information on JQ
Check the [JQ Manual](https://stedolan.github.io/jq/manual/v1.5/) and the [JQ Playground](https://jqplay.org/) for more information on JQ.

## Use Cases

JSON is a popular format of choice for data-interchange. It is widely used in web and server applications, document
storage, API I/O etc. It’s also used within Conductor to define workflow and task definitions and passing data and state
between tasks and workflows. This makes a tool like JQ a natural fit for processing task related data. Some common
usages within Conductor includes, working with HTTP task, JOIN tasks or standalone tasks that try to transform data from
the output of one task to the input of another.
JSON is a popular format of choice for data interchange. It is widely used in web and server applications, document
storage, API I/O, etc. It’s also used within Conductor to define workflow, task definitions and pass data and state
between tasks and workflows. This makes a tool like JQ a natural fit for processing task-related data. Some common
usages within Conductor include, working with HTTP tasks, JOIN tasks or standalone tasks that try to transform data from one task's output to another's input.

## Configuration

Here is an example of a _`JSON_JQ_TRANSFORM`_ task. The `inputParameters` attribute is expected to have a value object
that has the following
with the following.

1. A list of key value pair objects denoted key1/value1, key2/value2 in the example below. Note the key1/value1 are
1. A list of key-value pair objects denoted key1/value1, key2/value2 in the example below. Note the key1/value1 are
arbitrary names used in this example.

2. A key with the name `queryExpression`, whose value is a JQ expression. The expression will operate on the value of
the `inputParameters` attribute. In the example below, the `inputParameters` has 2 inner objects named by attributes
the `inputParameters` attribute. In the example below, the `inputParameters` have two inner objects named by attributes
`key1` and `key2`, each of which has an object that is named `value1` and `value2`. They have an associated array of
strings as values, `"a", "b"` and `"c", "d"`. The expression `key3: (.key1.value1 + .key2.value2)` concat's the 2
string arrays into a single array against an attribute named `key3`
strings as values, `"a", "b"` and `"c", "d"`. The expression `key3: (.key1.value1 + .key2.value2)` concatenates the 2
string arrays into a single array against an attribute named `key3`.

```json
{
Expand Down Expand Up @@ -65,7 +63,7 @@ that has the following
The execution of this example task above will provide the following output. The `resultList` attribute stores the full
list of the `queryExpression` result. The `result` attribute stores the first element of the resultList. An
optional `error`
attribute along with a string message will be returned if there was an error processing the query expression.
attribute along with a string message will be returned if there is an error processing the query expression.

```json
{
Expand Down Expand Up @@ -97,8 +95,8 @@ attribute along with a string message will be returned if there was an error pro
| name | Task Name. A unique name that is descriptive of the task function |
| taskReferenceName | Task Reference Name. A unique reference to this task. There can be multiple references of a task within the same workflow definition |
| type | Task Type. In this case, JSON_JQ_TRANSFORM |
| inputParameters | The input parameters that will be supplied to this task. The parameters will be a JSON object of at least 2 attributes, one of which will be called queryExpression. The others are user named attributes. These attributes will be accessible by the JQ query processor |
| inputParameters/user-defined-key(s) | User defined key(s) along with values. |
| inputParameters | The input parameters that will be supplied to this task. The parameters will be a JSON object of at least 2 attributes, one of which will be called queryExpression. The others are user-named attributes. These attributes will be accessible by the JQ query processor |
| inputParameters/user-defined-key(s) | User-defined key(s) along with values. |
| inputParameters/queryExpression | A JQ query expression |

### Output Configuration
Expand All @@ -107,7 +105,7 @@ attribute along with a string message will be returned if there was an error pro
| ----------- | ----------- |
| result | The first results returned by the JQ expression |
| resultList | A List of results returned by the JQ expression |
| error | An optional error message, indicating that the JQ query failed processing |
| error | An optional error message indicating that the JQ query failed to process |


## Example JQ transforms
Expand All @@ -116,10 +114,10 @@ attribute along with a string message will be returned if there was an error pro

This excerpt is from the [Simple ETL Codelab](/content/docs/usecases/Simple_ETL).

A HTTP Task makes an API call to GitHub to request a list of "stargazers" (users who have starred a repository). The API response (for just one user) looks like:
An HTTP Task makes an API call to GitHub to request a list of "stargazers" (users who have starred a repository). The API response (for just one user) looks like this:


Snippet of ```${hundred_stargazers_ref.output}```
The snippet of ```${hundred_stargazers_ref.output}```

``` JSON

Expand Down Expand Up @@ -151,7 +149,7 @@ Snippet of ```${hundred_stargazers_ref.output}```

```

We only need the ```starred_at``` and ```login``` parameters for users who starred the repository AFTER a given date (provided as an input to the workflow ```${workflow.input.cutoff_date}```). We'll use the JQ Transform to simplify the output:
We only need the ```starred_at``` and ```login``` parameters for users who starred the repository after a given date (provided as an input to the workflow ```${workflow.input.cutoff_date}```). We'll use the JQ Transform to simplify the output:

```JSON
{
Expand Down Expand Up @@ -189,4 +187,4 @@ The entire expression is wrapped in [] to indicate that the response should be a

## Creating Dynamic Fork Inputs

[Dynamic Forks](/content/docs/reference-docs/dynamic-fork-task) require multiple JSON inputs that are created on demand. JQ Transforms are ideal for this, and are used in the [Order Fulfillment Codelab](/content/docs/codelab/orderfulfillment7).
[Dynamic Forks](/content/docs/reference-docs/dynamic-fork-task) require multiple JSON inputs that are created on demand. JQ Transforms are ideal for this and are used in the [Order Fulfillment Codelab](/content/docs/codelab/orderfulfillment7).
Loading