Merge pull request #144 from orkes-io/staging

publishing secrets to docs

blog/2022-08-09-conductor-has-secrets.md

@@ -0,0 +1,103 @@
+---
+slug: conductor-has-secrets
+title: SHH! Conductor has secrets!
+authors: doug 
+tags: [Netflix Conductor, orchestration, security, 2022]
+image: https://orkes.io/content/img/Top-Secret.jpeg
+---
+
+
+We are really excited to announce the latest feature to Orkes' cloud hosted version of Netflix Conductor.  It is now no longer a secret - we support the use of secrets in your workflow definitions! Now you can be certain that your secret keys, tokens and values that you use in your workflows are secure!
+
+
+<!--truncate-->
+
+## What do you mean by secrets?
+
+In many of applications today, interaction with third party applications is common. Typically this will require some form of authentication to gain access.  When you are coding, there is a concept of a local secure store where sensitive values are kept (and thus not shared to GitHub etc.)  This prevents accidental disclosure of your secrets when posting code to GitHub or when sharing your code to other teams.
+
+Until now, there has been no way to securely use any sensitive value in a Conductor workflow.  Just about every developer has a story of accidentally posting a sensitive value on GitHub. Here's my story of accidentally sharing a sensitive value with a COnductor workflow:
+
+In the [`order fulfillment` codelab](https://orkes.io/content/docs/codelab/orderfulfillment5#adding-a-error-flow), the failure workflow has a Slack token that is unique, and if publicly accessible could be used to SPAM a Slack channel.  When writing the tutorial, I shared the task definition in the docs - *with* the Slack token.
+
+Slack caught this:
+
+<p align="center"><img src="/content/img/slack_oops.jpg" alt="aaccidently shared a hardcoded slack token" width="600" style={{paddingBottom: 40, paddingTop: 40}} /></p>
+
+Super embarrassing, but no serious consequences (in this instance).
+
+## Don't let this happen to you!
+
+In Orkes hosted instances of Netflix Conductor, we now feature secrets.  You can save your secret on the Conductor server, and Conductor will *use* the value when required, but will not expose the value in any outputs from the workflow.
+
+It is a very easy setup - simply login to your instance of Netflix Conductor at Orkes (or try our [Playground](https://play.orkes.io) for free!).  In the left navigation, click `Secrets`.  This will lead to a table of your secrets (which is probably empty). 
+
+<p align="center"><img src="/content/img/secrets_dashboard.jpg" alt="the Orkes Cloud Secrets dashboard" width="700" style={{paddingBottom: 40, paddingTop: 40}} /></p>
+
+Click `Add Secret`, give it a name, paste in your value, and press save. That's all there is to it.
+
+## Using your secret
+
+In Conductor workflows, secrets use a similar format to other variables.  For example, to reference an input variable called `address` you'd use `${workflow.input.address}`.
+
+If you had a secret called `Stripe_api_key`, you reference this value with the variable `${workflow.secrets.Stripe_api_key}`.
+
+## An example
+
+Accessing GitHubs APIs requires an API token.  In the following HTTP task, I call a GitHub API, and can reference the secret `Doug_github` for the authorization header.
+
+```json
+{
+      "name": "Get_repo_details",
+      "taskReferenceName": "Get_repo_details_ref",
+      "inputParameters": {
+        "http_request": {
+                 // highlight-next-line
+          "uri": "https://api.github.com/repos/${workflow.input.gh_account}/${workflow.input.gh_repo}",
+          "method": "GET",
+          "headers": {
+                 // highlight-next-line
+            "Authorization": "token ${workflow.secrets.Doug_github}",
+            "Accept": "application/vnd.github.v3.star+json"
+          },
+          "readTimeOut": 2000,
+          "connectionTimeOut": 2000
+        }
+      },
+      "type": "HTTP",
+      "decisionCases": {},
+      "defaultCase": [],
+      "forkTasks": [],
+      "startDelay": 0,
+      "joinOn": [],
+      "optional": false,
+      "defaultExclusiveJoinTask": [],
+      "asyncComplete": false,
+      "loopOver": []
+    }
+```
+
+When this workflow is run, other variables are replaced, but the value of the secret remains a secret.  Note that in the uri, `${workflow.input.gh_account}/${workflow.input.gh_repo}` is replaced with `netflix/conductor`, but the authorization header remains obfuscated.
+
+```json
+{
+  "headers": {
+         // highlight-next-line
+    "Authorization": "token ${workflow.secrets.Doug_github}",
+    "Accept": "application/vnd.github.v3.star+json"
+  },
+  "method": "GET",
+  "readTimeOut": 2000,
+       // highlight-next-line
+  "uri": "https://api.github.com/repos/netflix/conductor",
+  "connectionTimeOut": 2000
+}
+
+```
+
+
+## Conclusion
+
+Secrets have been one of the most requested features for Netflix Conductor when we speak to developers, and for that reason we're excited to announce this launch.  We cannot wait to hear about how this release is making workflow development more secure and opening new avenues of development - now that these values can be securely stored.
+
+Give them a try in the [Orkes Playground](https://play.orkes.io), and we w ould love to hear what you think in our [Slack](https://join.slack.com/t/orkes-conductor/shared_invite/zt-xyxqyseb-YZ3hwwAgHJH97bsrYRnSZg) or [Discord](https://discord.com/invite/P6vVt9xKSQ) communities

docs/getting-started/playground/first-playground-application.md

@@ -178,6 +178,12 @@ We've now created everything in the Playground that our Worker will need to conn
 
 ## Worker
 
+:::caution
+Workers can be deployed in multiple locations.  This very simple workflow requires that all workers are deployed on the same machine - as the first worker saves the image locally, and the second worker uploads the local file to S3. 
+
+If your workers are distributed, the 2nd worker may not be on the same server, and will be unable to find the local file.
+:::
+
 With your key and secret, we can run the OrkesWorkers application.  In the Orkesworkers repository, ```resources/application.properties``` ensure you have these two lines:
 
 

docs/how-tos/Workers/write-workers.md

@@ -18,10 +18,12 @@ Workers poll a Conductor task, and the parameters of each invocation is defined
 
 ### Worker Callbacks
 
+When a worker picks up a task, it can send a callback to Conductor:
+`callBackAfterSeconds` with a value in seconds.
 
+For example, if a task sends `callBackAfterSeconds=60` to the Conductor server, the queue expects a response from the worker within 60 seconds.
+
+If no response is received by the Conductor server within those 60s, the task is placed back into the queue, and can be picked up by another worker.
+
+This has been used by developers as a [heartbeat mechanism](https://github.com/Netflix/conductor/issues/896) for tasks that can run for a long period, but also sometimes crash.
 
-### Control Poll timing
-### Long running tasks
-### adding Execution logs
-### Adding Output
-### Failing Tasks

docs/how-tos/Workflows/create-secrets.md

@@ -2,21 +2,60 @@ import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 
 
-# Secrets
+# Workflow Secrets
 
 Many applications require the use of sensitive values that should be protected from exposure.  Items like usernames, passwords, API keys etc. are all sensitive values that should not be kept in a workflow (that might end up on GitHub or another public site.)
 
-Just as GitHub has the concept of `secrets`, so does Orkes Conductor.Now you can define your variables in a secure and safe way, knowing that they will not be exposed in the workflow, or shared with other teammates.
+Just as programming languages and GitHub have the concept of `secrets`, so does Orkes Conductor. You can define your variables in a secure and safe way knowing that they will not be exposed in the workflow, or shared with other teammates.
+
+
+## Using a Secret
+
+To use a secret in a workflow, you must first [create a secret](#creating-a-secret).  
+
+Let's assume your secret is called `GitHub_Token` To reference that secret, use the variable `${workflow.secrets.GitHub_Token}`.
+
+<p align="center"><iframe width="560" height="315" src="https://www.youtube.com/embed/yC4kOEHFfqE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></p>
+
+## Example
+
+The US Postal Service offers APIs to help automate the shipping process with the post office.  Each API call requires a UserId to be submitted.  This UserId can be used to buy postage, so it needs to be kept secure. WE've created a secret called `post_office_username` that we can now use in all API calls. (This workflow can be found is the [Conductor Examples](https://github.com/conductor-sdk/conductor-examples) github repository.):
+
+```shell
+https://production.shippingapis.com/ShippingAPI.dll?API=RateV4&XML=<RateV4Request \
+// highlight-next-line
+USERID=${workflow.secrets.post_office_username}> \
+<Revision>2</Revision> \
+<Package ID="0"><Service>priority</Service> \
+<ZipOrigination>04046</ZipOrigination> \
+<ZipDestination>98260</ZipDestination> \
+<Pounds>20</Pounds> \
+<Ounces>0</Ounces> \
+<Container>variable</Container> \
+<Width>12</Width> \
+<Length>12</Length> \
+<Height>12</Height> \
+<Girth></Girth> \
+<Machinable>TRUE</Machinable> \
+</Package></RateV4Request>
+```
+By using `${workflow.secrets.post_office_username}`, we obfuscate this sensitive value, and it never appears in the workflow execution, or in any output files of Conductor, yet we are able to connect with the USPS, and obtain the postage price for our package ($82.10, in case you're wondering).
 
 ## Creating a secret
 
 We'll walk through a few approaches to create a secret, and then examples of implementing a secret.  To create a secret, you can use the API or you can use the Orkes Dashboard.  
 
+<p align="center"><iframe width="560" height="315" src="https://www.youtube.com/embed/O_Ngo1Gg2Co" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></p>
+
+
 <Tabs groupId="code" values={[
         {label: 'API', value: 'api'},
         {label: 'Orkes Dashboard', value: 'orkes'},
     ]}>
-
+
+
+
+
 <TabItem value="api">
 
  Each secret must have a unique key for your Conductor instance.  The first step in creating a secret key:value pair is to ensure that the key is not in use.
@@ -59,8 +98,8 @@ This creates the secret key:value pair of ```pinetree:needle```.  The only respo
 <TabItem value="orkes">
 The Orkes Dashboard allows you to create secret key:value pairs via the UI.
 
-1. Login to your Orkes Cloud dashboard
-2. Click "Workflow Definitions."
+1. Login to your Orkes Cloud dashboard.
+2. Click "Workflow Definitions." or "Secrets" in the left navigation. 
 3. Click the "Manage Secrets" button at the top of the page.
 
 Or you can simply navigate to the url `secrets` in your Orkes Cloud dashboard. 
@@ -70,48 +109,39 @@ Or you can simply navigate to the url `secrets` in your Orkes Cloud dashboard.
 The Secrets dashboard lists all of the secrets connected to your account:
 <p align="center"><img src="/content/img/secrets_dashboard.jpg" alt="the Orkes Cloud Secrets dashboard" width="700" style={{paddingBottom: 40, paddingTop: 40}} /></p>
 
-Clicking the "eye" icon next the secret will expose the secret's value and also gives you the options to change the secret value, or delete the secret altogether.
+Clicking the "eye" icon next the secret will expose the secret's value and also give you the options to change the secret value, or delete the secret altogether.
 
 To add a secret click the `Add Secret` button at the top.  Add your Key:value and click the `Add` button.  If you receive an error, it is likely because the key is already in use, so simply select a different key.
 
 </TabItem>
 
 </Tabs>
 
-## Example
 
-The US Postal Service offers APIs to help automate the shipping process with the post office.  Each API call requires a UserId to be submitted.  This UserId can be used to buy postage, so it needs to be kept secure.
+## Permissions
 
-In the following API call, we're not buying postage, but just getting the shipping rate for a package. (This workflow can be found is the [Conductor Examples](https://github.com/conductor-sdk/conductor-examples) github repository.  The API call to the post office is a bit messy, but in the request, we must include the username (line two in the example below):
+If a workflow uses a Secret, we need to add access control permissions for the application to access the secret (just as an [application gives access to a workflow or task](/content/docs/getting-started/concepts/access-control-applications)).
 
-```shell
-https://production.shippingapis.com/ShippingAPI.dll?API=RateV4&XML=<RateV4Request \
-// highlight-next-line
-USERID=${workflow.secrets.post_office_username}> \
-<Revision>2</Revision> \
-<Package ID="0"><Service>priority</Service> \
-<ZipOrigination>04046</ZipOrigination> \
-<ZipDestination>98260</ZipDestination> \
-<Pounds>20</Pounds> \
-<Ounces>0</Ounces> \
-<Container>variable</Container> \
-<Width>12</Width> \
-<Length>12</Length> \
-<Height>12</Height> \
-<Girth></Girth> \
-<Machinable>TRUE</Machinable> \
-</Package></RateV4Request>
-```
-By using `${workflow.secrets.post_office_username}`, we obfuscate this sensitive value, and it never appears in the workflow execution, or in any output files of Conductor, yet we are able to connect with the USPS, and obtain the postage price for our package ($82.10, in case you're wondering).
 
+There are 3 access parameters used with Secrets:
 
-## Permissions
+* `READ`: Allows the application to read (and therefore use) the secret.
+* `UPDATE`: Provides access to share the secret with others.
+* `CREATE`: Create or overwrite an existing secret.
+
+In general, `READ` is the permission to be shared in most cases. 
 
-If the task using the Secret utilizes a worker, we need to add permissions for the worker to access the secret (just as an [application gives access to a workflow or task](/content/docs/getting-started/concepts/access-control-applications)).
+
+<Tabs groupId="code" values={[
+        {label: 'API', value: 'api'},
+        {label: 'Orkes Dashboard', value: 'orkes'},
+    ]}>
+
+<TabItem value="api">
 
-There is not UI to add a secret permission, but the API does work.  The endpoint is `/api/auth/authorization`.
+The endpoint is `/api/auth/authorization`.
 
-Using the Playground and Curl, the command looks like 
+Using the Playground and Curl, the command looks like:
 
 ```shell
 curl -X POST "https://play.orkes.io/api/auth/authorization" \
@@ -127,5 +157,18 @@ curl -X POST "https://play.orkes.io/api/auth/authorization" \
     "access":["READ"]}'
 ```
 
+Note that the type is `SECRET_NAME`.
+
 In this call, we are give the application `orkes-workers` `READ` access to our `post_office_username` secret.
+</TabItem>
+<TabItem value="orkes">
+
+The Orkes Dashboard, navigate to the "Applications" menu.  Choose the application you wish to add the secret to, and choose `add permission`: 
+
+<p align="center"><img src="/content/img/add_secret.jpg" alt="adding a secret via UI" width="400" style={{paddingBottom: 40, paddingTop: 40}} /></p>
+
+In the image above, we area adding the secret `GH_KEY` to our application (with `READ` access.). Click the `Add Permission` button, and the secret will be added.
+
+</TabItem>
+</Tabs>
 

sidebars.js

@@ -191,6 +191,7 @@
                     items: [
                         'how-tos/Workflows/create-workflow',
                         'how-tos/Workflows/updating-workflows',
+                        'how-tos/Workflows/create-secrets',
                         'how-tos/Workflows/versioning-workflows',
                         'how-tos/Workflows/starting-workflows',
                         'how-tos/Workflows/schedule-workflow',