Simplify configuration. Environments opt-in processing of jobs

Instead of promoting the approach of passing AWS credentials through
environment variables, advertise using instance profiles and make it
default configuration.

README.md

@@ -29,51 +29,27 @@ You have your Rails application deployed on the [Amazon Elastic Beanstalk](http:
   end
   ```
   * Choose a visibility timeout that exceeds the maximum amount of time a single job will take.
-3. Create a new user with SQS permissions:
+3. Give your EC2 instances permission to send messages to SQS queues:
   * Stay logged in and select the _IAM_ service from the services menu.
-  * Create a new user and store the credentials.
-  * Attach the **AmazonSQSFullAccess** policy to this user.
-4. Give your web application permissions to send messages to the SQS queue by adding:
-
+  * Select the _Roles_ submenu.
+  * Find the role that you select as the instance profile when creating the Elastic Beanstalk web environment.
+  ![Instance Profile](/docs/instance_profile.png?raw=true "Architecture Diagram" =20x20)
+  * Attach the **AmazonSQSFullAccess** policy to this role.
+  * Make yourself familiar with [AWS Service Roles, Instance Profiles, and User Policies](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts-roles.html).
+4. Tell the gem the region of your SQS queue that you created in step 2:
   * Select the web environment that is currently hosting your application and open the _Software Configuration_ settings.
-    * Add **AWS_ACCESS_KEY_ID** and set it to _access key id_ of the newly created user (from Step 3).
-    * Add **AWS_SECRET_ACCESS_KEY** and set it to the _secret access key_ of the newly created user (from Step 3).
-    * Add **AWS_REGION** and set it to the _region_ of the SQS queue, created in Step 2.
-    * Add **DISABLE_SQS_CONSUMER** and set it to `true`.
-
-  * Alternatively, instead of passing the credentials through these specific environment variables, you can change the configuration in order to
-  use different variables.
-
-  ```Ruby
-  # config/application.rb
-  module YourApp
-    class Application < Rails::Application
-      config.active_elastic_job.aws_region =  # defaults to ENV['AWS_REGION']
-      config.active_elastic_job.aws_access_key_id = # defaults to ENV['AWS_ACCESS_KEY_ID']
-      config.active_elastic_job.aws_secret_access_key = # defaults to ENV['AWS_SECRET_ACCESS_KEY'] || ENV['AWS_SECRET_KEY'] || ENV['AMAZON_SECRET_ACCESS_KEY']
-      config.active_elastic_job.disable_sqs_confumer = # defaults to ENV['DISABLE_SQS_CONSUMER']
-    end
-  end
-  ```
-
-  * Or , if your web environment is runinng EC2 instances with sufficient permissions, you can tell this gem to use the EC2 credentials:
-
-  ```Ruby
-  # config/application.rb
-  module YourApp
-    class Application < Rails::Application
-      # ..
-      config.active_elastic_job.aws_credentials = Aws::InstanceProfileCredentials.new
-    end
-  end
-  ```
+  * Add **AWS_REGION** and set it to the _region_ of the SQS queue, created in Step 2.
 
 5. Create a worker environment:
   * Stay logged in and select the _Elastic Beanstalk_ option from the services menu.
   * Select your application, click the _Actions_ button and select **Launch New Environment**.
   * Click the **create worker** button and select the identical platform that you had chosen for your web environment.
   * In the _Worker Details_ form, select the queue, that you created in Step 2, as the worker queue, and leave the MIME type to `application/json`. The visibility timeout setting should exceed the maximum time that you expect a single background job will take. The HTTP path setting can be left as it is (it will be ignored).
-6. Configure Active Elastic Job as the queue adapter.
+
+6. Configure the worker environment for processing jobs:
+  * Select the worker environment that you just have created and open the _Software Configuration_ settings.
+  * Add **PROCESS_ACTIVE_ELASTIC_JOBS** and set it to `true`.
+7. Configure Active Elastic Job as the queue adapter.
 
   ```Ruby
   # config/application.rb
@@ -83,11 +59,11 @@ You have your Rails application deployed on the [Amazon Elastic Beanstalk](http:
     end
   end
   ```
-7. Verify that both environments—web and worker—have the same secret base key:
+8. Verify that both environments—web and worker—have the same secret base key:
   * In the _Software Configuration_ settings of the web environment, copy the value of the **SECRET_KEY_BASE** variable.
   * Open the _Software Configuration_ settings of the worker environment and add the **SECRET_KEY_BASE** variable. Paste the value from the web environment, so that both environments have the same secret key base.
 
-8. Deploy the application to both environments (web and worker).
+9. Deploy the application to both environments (web and worker).
 
 ## FAQ
 A summary of frequently asked questions:

lib/active_elastic_job/rack/sqs_message_consumer.rb

@@ -57,8 +57,7 @@ def call(env) #:nodoc:
       private
 
       def enabled?
-        var = ENV['DISABLE_SQS_CONSUMER'.freeze]
-        var == nil || var == 'false'.freeze
+        Rails.application.config.active_elastic_job.process_jobs == true
       end
 
       def verify!(request)

lib/active_elastic_job/railtie.rb

@@ -1,14 +1,11 @@
 module ActiveElasticJob
   class Railtie < Rails::Railtie
     config.active_elastic_job = ActiveSupport::OrderedOptions.new
-    config.active_elastic_job.aws_region = ENV['AWS_REGION']
-    config.active_elastic_job.aws_access_key_id = ENV['AWS_ACCESS_KEY_ID']
-    config.active_elastic_job.aws_secret_access_key = ENV['AWS_SECRET_ACCESS_KEY'] || ENV['AWS_SECRET_KEY'] || ENV['AMAZON_SECRET_ACCESS_KEY']
-    config.active_elastic_job.disable_sqs_confumer = ENV['DISABLE_SQS_CONSUMER']
+    config.active_elastic_job.process_jobs = ENV['PROCESS_ACTIVE_ELASTIC_JOBS'] == 'true' || false
+    config.active_elastic_job.aws_credentials = Aws::InstanceProfileCredentials.new
 
     initializer "active_elastic_job.insert_middleware" do |app|
-      disabled = app.config.active_elastic_job.disable_sqs_confumer
-      if disabled == 'false' || disabled.nil?
+      if app.config.active_elastic_job.process_jobs == true
         if app.config.force_ssl
           app.config.middleware.insert_before(ActionDispatch::SSL,ActiveElasticJob::Rack::SqsMessageConsumer)
         else

lib/active_job/queue_adapters/active_elastic_job_adapter.rb

@@ -50,14 +50,14 @@ def initialize(serialized_job)
       #    #..
       #  end
       class NonExistentQueue < Error
-        def initialize(queue_name)
+        def initialize(queue_name, aws_region)
 
           super(<<-MSG)
             The job is bound to queue at #{queue_name}.
             Unfortunately a queue with this name does not exist in this
             region. Either create an Amazon SQS queue named #{queue_name} -
             you can do this in AWS console, make sure to select region
-            '#{Rails.application.config.active_elastic_job.aws_region}' - or you
+            '#{aws_region}' - or you
             select another queue for your jobs.
           MSG
         end
@@ -106,7 +106,7 @@ def enqueue_at(job, timestamp) #:nodoc:
             @queue_urls[job.queue_name.to_s] = nil
             retry
           end
-          raise NonExistentQueue, job
+          raise NonExistentQueue.new(job, aws_region)
         rescue Aws::Errors::ServiceError => e
           raise Error, "Could not enqueue job, #{e.message}"
         end
@@ -166,17 +166,15 @@ def check_job_size!(serialized_job)
         end
 
         def aws_sqs_client
-          c =  Rails.application.config.active_elastic_job
-          @aws_sqs_client ||= if c.aws_credentials.present?
-                                Aws::SQS::Client.new(
-                                  credentials: c.aws_credentials,
-                                  region: c.aws_region)
-                              else
-                                Aws::SQS::Client.new(
-                                  access_key_id: c.aws_access_key_id,
-                                  secret_access_key: c.aws_secret_access_key,
-                                  region: c.aws_region)
-                              end
+          @aws_sqs_client ||= Aws::SQS::Client.new(credentials: aws_sqs_client_credentials )
+        end
+
+        def aws_sqs_client_credentials
+          Rails.application.config.active_elastic_job.aws_credentials
+        end
+
+        def aws_region
+          Rails.application.config.active_elastic_job.aws_region
         end
 
         def message_digest(messsage_body)

spec/active_elastic_job/rack/sqs_message_consumer_spec.rb

@@ -9,17 +9,18 @@
   let(:secret_key_base) { 's3krit' }
   let(:rails_app) { double("rails_app") }
 
+  subject(:sqs_message_consumer) {
+    ActiveElasticJob::Rack::SqsMessageConsumer.new(app)
+  }
+
   before do
     allow(Rails).to receive(:application) { rails_app }
     allow(rails_app).to receive(:secrets) {
       { secret_key_base: secret_key_base }
     }
+    allow(sqs_message_consumer).to receive(:enabled?) { true }
   end
 
-  subject(:sqs_message_consumer) {
-    ActiveElasticJob::Rack::SqsMessageConsumer.new(app)
-  }
-
   it "passes an ordinary request through" do
     expect(app).to receive(:call).with(env).and_return(original_response)
     expect(sqs_message_consumer.call(env)).to eq(original_response)
@@ -100,18 +101,6 @@
           expect(sqs_message_consumer.call(env)[0]).to eq('200')
         end
 
-        context "when DISABLE_SQS_CONSUMER is true" do
-          around(:example) do |example|
-            with_modified_env DISABLE_SQS_CONSUMER: 'true', &example
-          end
-
-          it "passes request through" do
-            expect(app).to receive(:call).with(env).
-              and_return(original_response)
-            expect(sqs_message_consumer.call(env)).to eq(original_response)
-          end
-        end
-
         context "when digest is ommited" do
           before do
             env['HTTP_X_AWS_SQSD_ATTR_MESSAGE_DIGEST'] = nil

spec/active_job/queue_adapters/active_elastic_job_adapter_spec.rb

@@ -10,38 +10,16 @@ def initialize; end;
 
   let(:job) { Helpers::TestJob.new }
   let(:secret_key_base) { "s3krit" }
-
-  let(:rails_app) { double("rails_app") }
-
-  class StubbedConfig
-
-    def aws_access_key_id
-      ENV['AWS_ACCESS_KEY_ID']
-    end
-
-    def aws_secret_access_key
-      ENV['AWS_SECRET_ACCESS_KEY']
-    end
-
-    def aws_region
-      ENV['AWS_REGION']
-    end
-
-    def aws_credentials
-      nil
-    end
-  end
-
-  let(:config) { double('config') }
+  let(:aws_credentials) { 
+    Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
+  }
 
   before do
-    allow(Rails).to receive(:application) { rails_app }
-    allow(rails_app).to receive(:config) { config }
-    allow(config).to receive(:active_elastic_job) { StubbedConfig.new }
     allow(adapter).to receive(:secret_key_base) { secret_key_base }
+    allow(adapter).to receive(:aws_sqs_client_credentials) { aws_credentials }
+    allow(adapter).to receive(:aws_region) { ENV['AWS_REGION'] }
   end
 
-
   describe ".enqueue" do
     it "selects the correct queue" do
       expect(adapter).to receive(:queue_url).with(job.queue_name).and_return(

spec/integration/rails-app-5.0/.elasticbeanstalk/config.yml

@@ -3,7 +3,7 @@ branch-defaults:
     environment: web-env
     group_suffix: null
 global:
-  application_name: rails-eb-job-integration-testing
+  application_name: active_elastic_job
   default_ec2_keyname: null
   default_platform: Ruby 2.2 (Passenger Standalone)
   default_region: eu-central-1

spec/integration/rails-app-5.0/config/environments/production.rb

@@ -76,6 +76,4 @@
 
   # Do not dump schema after migrations.
   config.active_record.dump_schema_after_migration = false
-
-  config.active_elastic_job.aws_credentials = Aws::InstanceProfileCredentials.new
 end