AWS Batch - Submit Job (#6)

* Update role to allow lambda function to submit AWS Batch jobs

* Add support to submit aws batch jobs

* Update diagrams for batch

* Update README for supporting AWS Batch

* Update json schema for AWS Batch

Author: javibravo

README.md

@@ -13,6 +13,7 @@ Tasks are scheduled using the same syntax for expressions as linux
 
 * **Queue task**: send message to AWS SQS queue.
 * **Lambda task**: invoke AWS lambda function.
+* **Batch task**: submit AWS Batch job.
 * **HTTP task**: send HTTP requests (GET & POST).
 
 Tasks are defined in YAML files and are stored in a S3 bucket.
@@ -283,7 +284,7 @@ All parameters of the function will be supported soon.
 ### Lambda task
 
 It invokes an AWS lambda functions.
-The task definition must contains following keys
+The task definition must contains following keys:
 
 * **type**: *lambda*
 * **FunctionName**: Name of the lambda function to invoke (string)
@@ -302,6 +303,29 @@ task:
 
 Function is invoked using [boto3 Lambda.Client.invoke_async](http://boto3.readthedocs.io/en/latest/reference/services/lambda.html#Lambda.Client.invoke_async)
 
+### Batch task
+
+It submits AWS Batch Jobs.
+The task definition must contains following keys:
+
+* **type**: *batch*
+* **jobName**: name to assign to the job (string)
+* **jobQueue**: name of the queue in AWS Batch (string)
+* **jobDefinition**: name of the job definition in AWS Batch (string)
+
+``` yaml
+name: 'Enrich new stats every hour'
+expression: '0 * * * *'
+task:
+  type: 'bath'
+  jobName: 'enrich-stats'
+  jobDefinition: 'enrich-stats-definition:1'
+  jobQueue: 'jobs_high_priority'
+```
+
+It is a wrapper for [boto3 Batch.Client.submit_job](http://boto3.readthedocs.io/en/latest/reference/services/batch.html#Batch.Client.submit_job).
+It means all parameters for the method can be set in the task definition.
+
 ### HTTP task
 
 It send and HTTP request (GET or POST).

lambda-cron-diagram.png

@@ -1,2 +1,2 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<mxfile userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" version="6.1.0.2" editor="www.draw.io" type="device"><diagram name="Page-1">7VpLk9o4EP41HHH5gW04DpNk97Cpmqockj1Rwha2amTLkeUB8utXsi1AfoAINqS2gANW69nf12qp20yc12T3FwVZ/JWEEE9sM9xNnE8T27Yc1+M/QrKvJP5sUQkiisK60VHwDf2CtdCspQUKYa40ZIRghjJVGJA0hQFTZIBSslWbbQhWZ81ABFuCbwHAbel3FLK4ks5d8yj/G6IoljNbZl2zBsF7REmR1vNNbGdTfqrqBMix6vZ5DEKyPRE5nyfOKyWEVU/J7hViga2Erer3paf2sG4KU6bTwa46fABcQLnicl1sL7EIQR5D0dycOMuYJZg/WvyRLz0TTZJdJIzAANvcNnJGKAdnBdJwxclhfBkrbhzoA9K9kTurdRG8QyZ6M0re4SvBhPIxUpLyoZb1YiBlcNerkHWAiZsfJAlkdM+byA5zv+pSW54jOdseeXRnlSg+oVBaLKgtJzqMfESPP9QAdoPpDAxmQJKsYBWYKWRbQt9RGhkYJOsQjIrhXMHQNjswXLQx9O3bMZz9TzCcScwegKE3MIYgy1Y5pB8ogLmR/8xXPwsoRr4beJbjtcDz3DZ4swHA8wcGD3EnSLnlrchmxWJue7kRM5atMkoYCQi+pw0u3BaMvtnhCxe3wzi/DCNMwxdxVvNSgEGeo0CFs+oAw9ZRfVH/0z3WpV8toxADxo8m9frQoXQ9wxvhZB7htc2FetTMTHWInBQ0gHWv00P4wkD2vDEQAzSCrDVQycFBbS1aFo+kRapr9airSdsVDFlDMdS35OEZklfgx1Jk340ieyiK+pY8AkUat+ZLFHEi6P5HfYaUhX9FwXBl8Q1SxNcEqWyzQ+yH7MyfT5rzUrO1rgXwNZf4KhfYCqmT+9ijPKy1UA+wWZNTbeOYNQZyxjMOjSjgTsZhW2Nbh3fei3gq6o49mhdxGzedJr/ahtK35BEMRSPUuZOh+IuxDcU/byh+A3VnNENpzrT4XUPpW/LwhuJqGIqIE1AA8D9gDfEbyRFDJOVVa8IYSVSjkW1fMIpEG0ayVhAysR2v/PCaDcL4RL4sv13hzq+CQiMPeGxUYGFCg+QiXPVYt3yrZRoyOhw8FNS4iolkX9arZ52fBGvZ3Lxaf6mc9GzevKW/17E15s7t+nt30d83zwIwbYf+TpcrGEBfVyNo7dhHvXsOi4rlISN8sofqnPD5vSgzz/Y1u3Pjim+1O6vMsvDjSsaiuW8RyX0DcapyIyRBkZTM6O1d7zx37XSDHOSUuqbn/C3qNAJbqTdKyuz/ZZc4BMHlZC95Vr2lEPADWdigneBkWa/nk0gLcQhehNL2lyBMrZKVDUpDSEUik0tDwAD/KdnivzHY1k+OwMo3TdO3p6J26syn3DTgdJ/gqWXPjSyNbqL0cLPS2o5DcOpZf4D7mamnT/teMpYz8jSCzKczUiz3/El6P2fkaYSAT2d0C6UPcEYal/DRnZGleqOOcHo0b+Q+vdGV3qgnc3Z/b6TxOvDpjW6h9AHeSOMt5ejeSCqyV4v38EbPQO1ab3Q+ATdtJ1VGc0fPSG0od9TD6f3dka8RqY3/zrL5b4fx3lk2ZrrhxX/Pkm9OIvPi8U+EVfPjPzWdz/8B</diagram></mxfile>
+<mxfile userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" version="6.2.4" editor="www.draw.io" type="device"><diagram name="Page-1">7VrLcqM6EP0aL02BeHqZOJOZxdxbqcpi5q4oGWRbFUCMkGM7X38lQLZlwJZjcKZmkiwMjV59TqulbmlkT9PNVwrz5T8kRskImPFmZD+MALCsicl/hGRbSTynFiwojutCe8EzfkO1UBZb4RgVSkFGSMJwrgojkmUoYooMUkrWarE5SdRec7hADcFzBJOm9AeO2bKSBq65l39DeLGUPVtm/WUGo5cFJaus7m8E7Hn5V31OoWyrLl8sYUzWByL7y8ieUkJY9ZRupigR2ErYqnqPHV9346YoYzoVQFXhFSYrJEdcjottJRYxLJZIFDdH9v2SpQl/tPgjH3ouiqSbhTACA64LYBSMUA5OCLM45OQwPoyQGwd+RXRrFHY4W0UviInajJIXNCUJobyNjGS8qfvm6GuFeHWGNgeiWpuviKSI0S0vIr8GflWltjxbcrbe8+g6lWh5QKHr1dZTW85i1/IePf5QA9gOpt0zmBFJ8xWrwMwQWxP6grOFkcB0FsNBMQwUDIHZguGkiaEPrsfQ+UMwdExbwdCSBnaIod/E0HOux9DrGUOY52GB6CuOUGEUv4rw1wqJlm8G3sTRws7uYQ77PWOHuQ+k3PBCMg/ZkpteYSwZy8OcEkYiktwQRVs2cQgjaIFxcj2MwXkYURbfiaWav0UJLAocqXB2IoBiZfFu6n+on9ni6msZRQlkfGVSdw8tStc9PBFO5h5eYE5UeOUeRzZRkBWNUF3rcA0+0xAIjhpikC4QazRUcrBTW4uWyUfSItW1OtTVpO0Chqy+GOoacv8MyR3wx1IEbkYR6IuiriEPQJHGpvkcRZwZuv1ZryHly3/ixXDl6xOimI8JUVlmg9lPWZk/HxTnb8elOy2gglPZrlbAKLuv38TD7sLImlPnmFNt43COGrKHMw6NIOBGxgGsnq3Du9CLeCrqNhjMi7hqT+CYX21D6RryAIaiEencyFD8Sc+G4l9oKP4R6vZghnLc0+S9htI15P4NxdUwFBEU4Agm3+EMJU+kwAyTjH+aEcZIqhqNLHuX4IUow0jeCEJGPJYq//iXOU6SA/l9+d8W7rytKDKKiMdGq0SYUC+pCFdd1i2/GcPIsO/QNBzQbRraoaDGVkzk+nJ9PXcZSziTLZin9ZfKSc/mBQ39vZapEdjX6++9X/8ddFcDMG6mTew2V9CDvq5G0NoyjzrnXCI+3O8SwgdzqE4Jn56LMvEMLpmdc1f8V7OzSiwLP65kLI7nLSaFb2BOS2HEJFqlJYktc9fr4rSTO7dBnTQCJd1wwotrU6cR2Eq9cVom/8+7xD4ILju7K/LqkELAD+XLHG8EJ/f1eB5EWohDcCeUBo9RnFklK3OcxYiKPCaXxpBB/lOyxX+XcF0/2QIr3zRNH4zF17EdjLlpoPE2TcYWCIw8W1xF6W5npTUd++DUs85zOrj7cdTVp7kvGcoZeRpB5t/sjC7eSdzOGXkaIeCnM7qG0g9wRhqb8MGdkaV6o5ZwejBv5H56o1PeyOmg+HfwRhqngZ/e6BpKP8AbaZxSDu6NpCJb9fUW3ugzUDvpjfwOiruD7GZSZTB39Bmpvcsd6XN6e3fka0Rqw59Zmqo/GvDM8qinKw7+O4bcfxJZXs36o6YdODvtYArf+BSD62JcMJRFOBHS8rbm47S+GRb+u7sWFkYk34Z3P57DB0z5IMJptSTu5mV3NrxzVW2YdPeCGhzlekEz9xm0XLuTsqumcB/HlhcdUe/PrgwrAJrnV60u4V0nVOC0M3HVW6TWpJl27+voUu0JOEeX2HSdiWPZ7UO+2pnw1/2F5Kr4/ta3/eV/</diagram></mxfile>

lambda-cron-diagram.xml

@@ -42,6 +42,9 @@ def get_lambda_task_runner(self, task):
     def get_http_task_runner(self, task):
         return HttpTask(task)
 
+    def get_batch_task_runner(self, task):
+        return BatchJobTask(task)
+
 
 class Task:
 
@@ -84,3 +87,13 @@ def run(self):
             FunctionName=self.task['FunctionName'],
             InvokeArgs=json.dumps(self.task['InvokeArgs'])
         )
+
+
+class BatchJobTask(Task):
+
+    def get_batch_client(self):
+        return boto3.client('batch')
+
+    def run(self):
+        self.task.pop('type')
+        self.get_batch_client().submit_job(**self.task)

lambda_cron/aws/lib/task_runner.py

@@ -28,6 +28,18 @@
           },
           "required": ["type", "FunctionName", "InvokeArgs"]
         },
+        {
+          "properties": {
+            "type": { "type": "string", "enum": ["batch"] },
+            "jobName": { "type": "string" },
+            "jobQueue": { "type": "string" },
+            "jobDefinition": { "type": "string" },
+            "dependsOn": { "type": "array" },
+            "parameters": { "type": "object" },
+            "containerOverrides": { "type": "object" }
+          },
+          "required": ["type", "jobName", "jobQueue", "jobDefinition"]
+        },
         {
           "properties": {
             "type": { "type": "string", "enum": ["http"] },

lambda_cron/schema.json

@@ -87,6 +87,10 @@ Resources:
             Action:
               - lambda:InvokeFunction
             Resource: ['*']
+          - Effect: Allow
+            Action:
+              - batch:SubmitJob
+            Resource: ['*']
 
   LambdaCronHourlyEvent:
     Type: AWS::Events::Rule

lambda_cron/template.cfn.yml

@@ -0,0 +1,10 @@
+name: 'test batch task'
+expression: '0 11 * * *'
+task:
+  type: 'batch'
+  jobName: 'my-batch-job'
+  jobQueue: 'job-queue-hihg-priority'
+  jobDefinition: 'my-batch-job-definition'
+  parameters:
+    argument_1: 'value_argument_1'
+    argument_2: 2

tests/resources/tasks/valid/batch_task.yml

@@ -15,7 +15,7 @@
 import pytest
 import json
 from mock import patch
-from lambda_cron.aws.lib.task_runner import TaskRunner, QueueTask, InvokeLambdaTask, HttpTask
+from lambda_cron.aws.lib.task_runner import TaskRunner, QueueTask, InvokeLambdaTask, HttpTask, BatchJobTask
 from lambda_cron.aws.lib.cron_checker import CronChecker
 
 
@@ -323,3 +323,57 @@ def test_http_not_supported_method(get_requests_client_mock, http_client_spy, cr
     with pytest.raises(Exception) as exception_info:
         task_runner.run(http_get_task_definition)
     assert "Http method not supported: put" in str(exception_info.value)
+
+
+class BatchClientSpy:
+    def __init__(self):
+        self.parameters = None
+        self.calls = 0
+
+    def submit_job(self, **kwargs):
+        self.parameters = kwargs
+        self.calls += 1
+
+
+@pytest.fixture(scope="function")
+def batch_client_spy():
+    return BatchClientSpy()
+
+
+BATCH_TASK_BODY =\
+    {
+        'type': 'batch',
+        'jobName': 'testing-batch-job',
+        'jobQueue': 'testing-batch-job-queue',
+        'jobDefinition': 'testing-batch-job-definition',
+        'parameters':
+            {
+                'param_1': 'value_1',
+                'param_2': 'value_2'
+            }
+    }
+
+
+@pytest.fixture(scope="function")
+def batch_task_definition():
+    return {
+        'name': 'Test task',
+        'expression': '0 11 * * *',
+        'task': dict(BATCH_TASK_BODY)
+    }
+
+
+@patch.object(BatchJobTask, 'get_batch_client')
+def test_batch_should_run_basic(get_batch_client_mock, batch_client_spy, cron_checker, batch_task_definition):
+    get_batch_client_mock.return_value = batch_client_spy
+
+    task_runner = TaskRunner(cron_checker)
+    task_runner.run(batch_task_definition)
+
+    assert batch_client_spy.calls == 1
+    assert 'jobName' in batch_client_spy.parameters
+    assert batch_client_spy.parameters['jobName'] == 'testing-batch-job'
+    assert 'jobQueue' in batch_client_spy.parameters
+    assert batch_client_spy.parameters['jobQueue'] == 'testing-batch-job-queue'
+    assert 'parameters' in batch_client_spy.parameters
+    assert batch_client_spy.parameters['parameters'] == BATCH_TASK_BODY['parameters']