Few update

.github/workflows/python-package.yml

@@ -1,21 +1,18 @@
-# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
-# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
-
 name: Python package
 
 on:
   push:
-    branches: [ develop ]
+    branches: [ master ]
   pull_request:
-    branches: [ develop ]
+    branches: [ master ]
 
 jobs:
   build:
 
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.5, 3.6, 3.7, 3.8]
+        python-version: [3.6, 3.7, 3.8, 3.9]
 
     steps:
     - uses: actions/checkout@v2
@@ -26,8 +23,8 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install pytest
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+        if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt; fi
     - name: Test with pytest
       run: |
         pytest

.github/workflows/python-validate.yml

@@ -2,17 +2,17 @@ name: Python validate releasability
 
 on:
   push:
-    branches: [ develop ]
+    branches: [ master ]
   pull_request:
-    branches: [ develop ]
+    branches: [ master ]
 
 jobs:
   build:
 
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.6, 3.7, 3.8]
+        python-version: [3.6, 3.7, 3.8, 3.9]
 
     steps:
     - uses: actions/checkout@v2
@@ -23,8 +23,8 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install pytest
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+        if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt; fi
     - name: Validate packaging / publishability
       run: |
         pip install setuptools wheel twine readme_renderer[md]

.gitignore

@@ -54,3 +54,6 @@ docs/_build/
 
 # Emacs backup files
 *~
+
+datalake_auditor
+data

.travis.yml

@@ -1,13 +1,13 @@
 language: python
 dist: xenial
 python:
-  - "2.7"
-  - "3.4"
-  - "3.5"
   - "3.6"
   - "3.7"
+  - "3.8"
+  - "3.9"
 install:
   - pip install -r requirements.txt
+  - pip install -r requirements-dev.txt
   - pip install coverage python-coveralls
 script: nosetests tests/unit --cover-erase --with-coverage --cover-package skew
 after_success: coveralls

README.md

@@ -24,6 +24,12 @@ This pattern identifies a specific EC2 instance running in the `us-west-2`
 region under the account ID `123456789012`.  The account ID is the 12-digit
 unique identifier for a specific AWS account as described
 [here](http://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html).
+
+## Configuration
+
+Without any configuration file, `skew` use the account ID of the caller,
+and credentials defined on your system (aws environment variable, iam role of your instance, ...)
+
 To allow `skew` to find your account number, you need to create a `skew`
 YAML config file.  By default, `skew` will look for your config file in
 `~/.skew` but you can use the `SKEW_CONFIG` environment variable to tell `skew`
@@ -46,15 +52,19 @@ file.
 
 The main purpose of skew is to identify resources or sets of resources
 across services, regions, and accounts and to quickly and easily return the
-data associated with those resources. For example, if you wanted to return
-the data associated with the example ARN above:
+data associated with those resources.
+
+## Usage
+
+For example, if you wanted to return the data associated with the example ARN above:
 
 ```python
 from skew import scan
 
 arn = scan('arn:aws:ec2:us-west-2:123456789012:instance/i-12345678')
 for resource in arn:
-    print(resource.data)
+    print(resource.data) # return a dict
+    print(resource.json_dump()) # dump resource in json
 ```
 
 The call to `scan` returns an ARN object which implements the
@@ -81,8 +91,28 @@ you would use:
 arn = scan('arn:aws:dynamodb:us-.*:234567890123:table/*')
 ```
 
-CloudWatch Metrics
-------------------
+## Command line Usage
+
+```bash
+python -m "skew" --uri "arn:aws:events:eu-west-1:*:rule/*" --output-path "./data"
+```
+
+In order to retreive all options:
+
+```bash
+python -m "skew" -h
+usage: __main__.py [-h] --uri URI --output-path OUTPUT_PATH [--normalize]
+
+SKEW alias Stock Keeping Unit
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --uri URI             scan uri (arn:aws:*:*:1235678910:*/*)
+  --output-path OUTPUT_PATH
+                        output directory
+```
+
+## CloudWatch Metrics
 
 In addition to making the metadata about a particular AWS resource available
 to you, `skew` also tries to make it easy to access the available CloudWatch
@@ -154,8 +184,7 @@ You can also customize the data returned rather than using the default settings:
 >>>
 ```
 
-Filtering Data
---------------
+## Filtering Data
 
 Each resource that is retrieved is a Python dictionary.  Some of these (e.g.
 an EC2 Instance) can be quite large and complex.  Skew allows you to filter
@@ -182,8 +211,7 @@ filtered data is available as the `filtered_data` attribute of the
 Resource object.  The full, unfiltered data is still available as the
 `data` attribute.
 
-Multithreaded Usage
--------------------
+## Multithreaded Usage
 
 Skew is single-threaded by default, like most Python libraries. In order to
 speed up the enumeration of matching resources, you can use multiple threads:
@@ -211,11 +239,121 @@ for service in arn.service.choices():
 
 (thanks to @alFReD-NSH for the snippet)
 
-More Examples
--------------
+## More Examples
 
 [Find Unattached Volumes](https://gist.github.com/garnaat/73804a6b0bd506ee6075)
 
 [Audit Security Groups](https://gist.github.com/garnaat/4123f1aefe7d65df9b48)
 
 [Find Untagged Instances](https://gist.github.com/garnaat/11004f5661b4798d27c7)
+
+## Supported Service
+
+| Name             |
+| ---------------- |
+| route53          |
+| cloudfront       |
+| elasticbeanstalk |
+| ecs              |
+| kms              |
+| redshift         |
+| efs              |
+| sns              |
+| cloudwatch       |
+| cloudtrail       |
+| acm              |
+| sqs              |
+| elasticache      |
+| ecr              |
+| lambda           |
+| elb              |
+| stepfunctions    |
+| events           |
+| iam              |
+| rds              |
+| cloudsearch      |
+| logs             |
+| firehose         |
+| autoscaling      |
+| s3               |
+| support          |
+| ec2              |
+| cloudformation   |
+| opsworks         |
+| es               |
+| elbv2            |
+| kinesis          |
+| ses              |
+| dynamodb         |
+| apigateway       |
+
+## Supported Resource
+
+| Name                                    |
+| --------------------------------------- |
+| aws.acm.certificate                     |
+| aws.apigateway.restapis                 |
+| aws.autoscaling.autoScalingGroup        |
+| aws.autoscaling.launchConfigurationName |
+| aws.cloudfront.distribution             |
+| aws.cloudformation.stack                |
+| aws.cloudsearch.domain                  |
+| aws.cloudwatch.alarm                    |
+| aws.logs.log-group                      |
+| aws.cloudtrail.trail                    |
+| aws.dynamodb.table                      |
+| aws.ec2.address                         |
+| aws.ec2.customer-gateway                |
+| aws.ec2.key-pair                        |
+| aws.ec2.image                           |
+| aws.ec2.instance                        |
+| aws.ec2.natgateway                      |
+| aws.ec2.network-acl                     |
+| aws.ec2.route-table                     |
+| aws.ec2.internet-gateway                |
+| aws.ec2.security-group                  |
+| aws.ec2.snapshot                        |
+| aws.ec2.volume                          |
+| aws.ec2.vpc                             |
+| aws.ec2.flow-log                        |
+| aws.ec2.vpc-peering-connection          |
+| aws.ec2.subnet                          |
+| aws.ec2.launch-template                 |
+| aws.ecs.cluster                         |
+| aws.ecs.task-definition                 |
+| aws.ecr.registery                       |
+| aws.ecr.repository                      |
+| aws.efs.filesystem                      |
+| aws.elasticache.cluster                 |
+| aws.elasticache.subnet-group            |
+| aws.elasticache.snapshot                |
+| aws.elasticbeanstalk.application        |
+| aws.elasticbeanstalk.environment        |
+| aws.elb.loadbalancer                    |
+| aws.elbv2.loadbalancer                  |
+| aws.elbv2.targetgroup                   |
+| aws.es.domain                           |
+| aws.events.rule                         |
+| aws.firehose.deliverystream             |
+| aws.iam.group                           |
+| aws.iam.instance-profile                |
+| aws.iam.role                            |
+| aws.iam.policy                          |
+| aws.iam.user                            |
+| aws.iam.server-certificate              |
+| aws.kinesis.stream                      |
+| aws.kms.key                             |
+| aws.lambda.function                     |
+| aws.opsworks.stack                      |
+| aws.rds.db                              |
+| aws.rds.secgrp                          |
+| aws.redshift.cluster                    |
+| aws.route53.hostedzone                  |
+| aws.route53.healthcheck                 |
+| aws.s3.bucket                           |
+| aws.stepfunctions.statemachine          |
+| aws.sqs.queue                           |
+| aws.ses.identity                        |
+| aws.sns.subscription                    |
+| aws.sns.topic                           |
+| aws.support.check                       |

changelog.md

@@ -0,0 +1,49 @@
+# Change log
+
+## 1.0.0 (coming soon)
+
+- Python 3 and dependencies:
+  - Fix yam constructor DeprecationWarning and update PyYaml
+  - Remove python 2, python 3.4 and 3.5 support, add 3.8, 3.9 test unit configuration
+  - Update and separate dev dependencies from module dependencies
+  - Align python syntax on version 3
+- Integrate Change from Christophe Morio (https://github.com/lbncmorio/skew/tree/more-resources)
+- Configuration:
+  - Remove mandatory needs for skew.yaml (using iam metadata associated and default boto3 credentials initialization)
+- aws client:
+  - Fix Error and termination BUG with awsclient
+  - Add boto3 config default with retries={"max_attempts": 20, "mode": "adaptive"}
+- Resource Enumeration:
+  - Fix resource enumeration when no resource type is define
+  - Rewrote filtering resource and add a warning if filter operation is missing when needed
+  - Fix bad component matchs operation if similar component share a common prefix (like elb and elbv2)
+- Resource Loading
+  - Change enumerate to avoir loading all resources loaded in memory
+  - Add lazy loading of full data with method _load_extra_attribute on Resource
+  - Add lazy load per item on Log group for log_streams, metric_filters, queries, subscriptions
+- Additional Ressource and details:
+  - Group users, policy inline and attached
+  - kinesis description
+  - S3 bucket properties (acl, encryption, logging, cors, policy, notifications, ...)
+  - elbV2 and target group
+  - Cloud front Domain
+  - cloud search and region list update
+  - opsworks availaible on 9 regions
+  - api gateway
+  - Name EC2 with Instane Id or Tag Name value if exists
+  - Cloudtrail: fix enumeration and tags, add trail detail and trail status
+  - Add json_dump with optional normalisation
+  - Add ECR Registery
+  - Add ECR Repository
+  - Add Kms Key
+  - Add service definition on ecs cluster
+  - Add StepFunction (alias states)
+  - Add Event rule
+- Github:
+  - set master branch as base branch
+  - update workflow
+- Add Command line utility
+
+## 0.19.0
+
+- no change log

requirements-dev.txt

@@ -0,0 +1,7 @@
+pytest
+mock==4.0.3
+nose==1.3.7
+tox==3.20.1
+placebo==0.9.0
+coverage
+python-coveralls

requirements.txt

@@ -1,8 +1,2 @@
-boto3>=1.2.3
-placebo==0.4.3
-six>=1.9.0
-PyYAML==3.13
-python-dateutil>=2.1,<3.0.0
-mock==1.0.1
-nose==1.3.4
-tox==1.8.1
+boto3==1.16.35
+PyYAML==5.3.1

setup.py

@@ -36,7 +36,6 @@
         'Intended Audience :: System Administrators',
         'Natural Language :: English',
         'License :: OSI Approved :: Apache Software License',
-        'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 3'
     ],
 )

skew/__init__.py

@@ -14,8 +14,16 @@
 import os
 
 from skew.arn import ARN
+from skew.awsdefaults import get_all_activated_regions, get_caller_identity_account_id
 
-__version__ = open(os.path.join(os.path.dirname(__file__), '_version')).read()
+__all__ = [
+    "__version__",
+    "get_all_activated_regions",
+    "get_caller_identity_account_id",
+    "scan",
+]
+
+__version__ = open(os.path.join(os.path.dirname(__file__), "_version")).read()
 
 
 def scan(sku, **kwargs):