Design a way to more quickly write AWS sync plugins #349
Description
Design a way to more quickly write AWS sync plugins
Description:
Describe your idea. Please be detailed. If a feature request, please
describe the desired behavior, what scenario it enables, and how it
would be used.
Background
An intel module needs to perform a common set of steps:
-
Get data with a certain set of fields from an external API
-
Perform a set of data transformations such as generating unique IDs if vendor-provided ones (e.g. AWS ARNs) are not available.
-
Load new nodes and relationships to the graph and connect them to all related nodes such as AWS accounts and others
-
Make indexes for the new node types
-
Create integration tests to ensure the nodes and connections are loaded as expected
-
Update our schema documents
This is a lot of boilerplate code to write and there are many repeated sections across intel modules.
The idea
To improve our developer experience, we could create an abstraction layer that accepts as input what data types the developer wants to retrieve and their expected forms in the graph (steps 1-4). This will greatly improve our velocity on adding new plugins.
This issue will initially be scoped to AWS syncs and we will revisit this approach for the other modules.
The task
Write a specification document on what this abstraction interface could look like and submit requests for comment.