Project Restructure: Service Level Packages #20431
Description
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Closely related #20000
Description
The structure of the Terraform AWS provider codebase is not scaling well as it grows. This explosive growth is fuelled by the AWS provider’s popularity and AWS’s ever increasing services. As a result, maintainers and community contributors face an unnecessary layer of difficulty maintaining the code base such as non-responsive IDEs and the inability to see all project files on GitHub.
In order to better follow Go best practices and address these growing pains, we are planning significant code refactoring of the Terraform AWS Provider. The refactoring will position the provider for better maintainability now and into the future as AWS ever expands its offerings.
Code for every AWS service we support, in 2,000+ Go files, lives in one massive package. A single line change requires the entire package to be recompiled, which at best, can take several minutes. Contributors, maintainers, IDEs, Go compilers, and security scanning products, have unnecessary overhead dealing with such a massive package. At the same time, many tasks require jumping back and forth between different directory structures.
Proposal
The primary goal is simply stated: move everything related to a single AWS service, such as ec2, into one package. This includes resources, data sources, associated tests, and support code. During the refactor, we will also undertake housekeeping to simplify things like naming and imports. Further details are given below.
This solution provides several benefits:
- A reduction in defect risks since service packages are independent. Unintended consequences of changes in one service are less likely to affect other services.
- Having everything you need in one compact package for new development and maintenance tasks reduces the mental burden of scrolling and jumping around in the code.
- Not having unrelated code in a package makes it easier to grasp what is in the package.
- Go is optimized to compile the least number of packages necessary. Go will often be able to compile a single service package rather than the entire provider.
- Service Level Package structure will require a better-defined interface and division of responsibilities between specific service packages and general support packages. Clear divisions lead to uncoupling and help ensure proper testing.
- Simplicity in the structure makes the project more readable and reduces the challenge of submitting contributions. A clearer understanding of the project leads to better quality contributions.
Cutover Process
This refactoring will occur as a "big-bang" change, internally these changes will be automated and tested against our acceptance test suite before rollout. The consequence of this is that most if not all existing Pull Requests will become unmergeable "as-is" due to the large amount of structural changes.
The date of the rollout will be communicated publicly, and guides on how to update your Pull Request will also be made available closer to the confirmed date.
We realize that many Pull Requests have been sitting for some time without being addressed. We want to make it very clear that the mergability of a Pull Request due to structural changes like this do not affect its prioritization for review. Once a Pull Request is prioritized for review, the maintainers will either make those changes directly or work with the PR author to do so. There is no expectation from the maintainers that the community updates their PR's en-masse, but we realize that some authors will choose to do so in order to explore the new package structure. We will make these requirements explicit both in the contributing guide, our linter suite and from this issue.