Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create AWS architectural guidance #6489

Merged
merged 92 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
ebaa429
Add first draft
kbaley Mar 12, 2024
b6e20d8
Removing warnings
NChaganlal Mar 13, 2024
dfd7e21
Improvements on the queue worker architecture style intro
NChaganlal Mar 13, 2024
05c7c98
Improved SQS information on AWS architecture guidance
NChaganlal Mar 13, 2024
5fbc290
Add AWS data stores and microservices docs
kbaley Mar 13, 2024
957952b
Move well-architected to index
WilliamBZA Mar 14, 2024
f91ed1e
Move compute to own file
WilliamBZA Mar 14, 2024
97f072c
Move event driven to own file
WilliamBZA Mar 14, 2024
af99698
AWS: Extracted queue based architecture style from base document
NChaganlal Mar 14, 2024
927f2f1
AWS: Extract multi tier architecture style from the base document
NChaganlal Mar 14, 2024
d197085
AWS: Update multi tier architecture title
NChaganlal Mar 14, 2024
e591d1f
Move well-archicted to it's own file
WilliamBZA Mar 14, 2024
57f085f
Merge branches 'master' and 'tf-cloudxp-aws-arch-guidance-258' of git…
WilliamBZA Mar 14, 2024
82e63e2
AWS: Improve queue-based-architecture - additional resources
NChaganlal Mar 14, 2024
e1bbb1a
AWS: Extracted Messaging from base document and standardized titles
NChaganlal Mar 14, 2024
1381401
AWS: Extract observability from base document
NChaganlal Mar 14, 2024
6c0a4c1
AWS: Moving content from base-document.md to index.md and add links t…
NChaganlal Mar 14, 2024
1f7ab9a
AWS: Delete base-document.md
NChaganlal Mar 14, 2024
73c41fa
AWS: Add metadata to well architected page
NChaganlal Mar 14, 2024
df0eac0
AWS: Replacing "you" from well-architected.md
NChaganlal Mar 15, 2024
bca371e
AWS: Replacing reference to "your" on well-architected.md
NChaganlal Mar 15, 2024
721c8d9
AWS: Replace reference to "you" and "your"
NChaganlal Mar 15, 2024
d5a0bb6
AWS: Replacing "you" and "your" references in the observability.md
NChaganlal Mar 15, 2024
0a7db62
AWS: Remove word "Please" from queue-based-architecture.md
NChaganlal Mar 15, 2024
5e204fa
AWS: Replacing "we" on observability.md
NChaganlal Mar 15, 2024
0a68b0a
AWS: Replace "you" and "your" from messaging.md
NChaganlal Mar 15, 2024
50eed22
AWS: Remove "?" from headers
NChaganlal Mar 15, 2024
922e90c
AWS: Removing url prefix from index.md
NChaganlal Mar 15, 2024
3dc3d2f
AWS: Replacing "you" and "your" in compute.md
NChaganlal Mar 15, 2024
c7946f8
AWS: Replaced word "your" in data-stores.md
NChaganlal Mar 15, 2024
e88f34c
AWS: Fixed data-stores.md urls
NChaganlal Mar 15, 2024
bb126ed
AWS: Fix `pay-as-you-go` in messaging.md
NChaganlal Mar 15, 2024
99b53b7
AWS: Created images folder and image on queue-based-architecture.md
NChaganlal Mar 15, 2024
c156476
AWS: Fix event-driven-architecture links
NChaganlal Mar 15, 2024
b12852d
AWS: Lowecase AWS in event-driven-architecture.md
NChaganlal Mar 15, 2024
399f8be
AWS: Fix urls in compute.md
NChaganlal Mar 15, 2024
50f20c2
AWS: Fix URL fix in event-driven-architecture.md
NChaganlal Mar 15, 2024
4ecd855
AWS: Replacing word "our" in observability.md
NChaganlal Mar 15, 2024
a132280
AWS: Fix urls in messaging.md
NChaganlal Mar 15, 2024
cb15c27
AWS: Fix urls in microservices.md
NChaganlal Mar 15, 2024
3872b44
AWS: Replace "us" in observability.md
NChaganlal Mar 15, 2024
ed74b70
AWS: Fix urls in observability.md
NChaganlal Mar 15, 2024
2835a7a
AWS: Fix URLs in well-architected.md
NChaganlal Mar 15, 2024
25e9369
AWS: Fix urls in multi-tier-architecture.md
NChaganlal Mar 15, 2024
3c92892
AWS: Added data-stores link to multi-tier-architecture.md
NChaganlal Mar 15, 2024
c1f8194
AWS: Fix urls in observability.md
NChaganlal Mar 15, 2024
c6159e2
AWS: Fix opentelemetry link in observability.md
NChaganlal Mar 15, 2024
c9bfe3f
Fix content errors
kbaley Mar 15, 2024
9c58eaf
remove concept from aws section
lailabougria Mar 20, 2024
40444e1
add observability as a concept
lailabougria Mar 20, 2024
b0de64a
add azure section for observability
lailabougria Mar 20, 2024
f133a40
add menu items
lailabougria Mar 20, 2024
c5cea01
fix wording
lailabougria Mar 20, 2024
3f4e39b
fix link
lailabougria Mar 20, 2024
da591ce
Edit aws observability content
WilliamBZA Mar 20, 2024
c77762e
Update architecture/aws/observability.md
WilliamBZA Mar 20, 2024
1c4e12a
AWS: Fix observability link
NChaganlal Mar 21, 2024
5f58e54
Update architecture/aws/observability.md
NChaganlal Mar 21, 2024
a9e0aa2
AWS: Improve architecture/aws/observability.md
NChaganlal Mar 22, 2024
c8c7775
Update architecture/aws/observability.md
WilliamBZA Mar 25, 2024
3725c2b
Update architecture/aws/observability.md
WilliamBZA Mar 25, 2024
5d06877
Apply suggestions from code review
WilliamBZA Mar 25, 2024
663759f
Update observability.md
WilliamBZA Mar 25, 2024
da788c1
Add missing words
WilliamBZA Mar 25, 2024
a642a46
Edit first paragraph
WilliamBZA Mar 26, 2024
afd8b5c
Review other sections and tweak azure content
WilliamBZA Mar 26, 2024
57e9ded
Update architecture/azure/web-queue-worker.md
kbaley Mar 27, 2024
f7d3705
Review of AWS Compute section (#6521)
kbaley Mar 27, 2024
6ae1fd7
Update queue-based-architecture diagram and replace visio document wi…
NChaganlal Mar 28, 2024
56ab2ba
Merge pull request #6529 from Particular/queue-based-review
WilliamBZA Mar 28, 2024
8592c04
Update architecture/aws/observability.md
lailabougria Mar 28, 2024
64397d0
Review of AWS Well-Architected article (#6532)
kbaley Mar 28, 2024
069250a
Update observability.md to remove sample TODOs
NChaganlal Mar 28, 2024
424c796
Update architecture/aws/observability.md
kbaley Mar 28, 2024
5e4510c
Merge pull request #6519 from Particular/observability-edit
WilliamBZA Apr 2, 2024
b9a6f7c
Edit messaging page (#6535)
WilliamBZA Apr 4, 2024
ea3fa4c
AWS: Update messaging.md to fix the anchor
NChaganlal Apr 4, 2024
4bff322
Editorial Review event-driven
NChaganlal Apr 4, 2024
7345b5d
Edit the multi-tier content
WilliamBZA Apr 4, 2024
7ee7226
Reflect improvements done on AWS on the Azure docs
NChaganlal Apr 5, 2024
11ebb1a
Merge pull request #6551 from Particular/event-driven
WilliamBZA Apr 9, 2024
d8a09ee
AWS summaries (#6556)
kbaley Apr 9, 2024
72cb073
Summaries for Azure architecture articles (#6563)
kbaley Apr 9, 2024
d06531e
Fix heading indentation on architecture articles
kbaley Apr 9, 2024
38a48d4
Edit data store (#6562)
WilliamBZA Apr 9, 2024
58c52a6
Add CTAs to AWS articles
kbaley Apr 9, 2024
2a957a8
Merge pull request #6564 from Particular/aws-ctas
WilliamBZA Apr 10, 2024
38a951f
AWS: Add diagrams, references and miro board backup
NChaganlal Apr 10, 2024
847be51
Apply suggestions from code review
WilliamBZA Apr 11, 2024
3aa228a
Merge pull request #6566 from Particular/aws-add-diagrams
WilliamBZA Apr 11, 2024
2036219
Review of AWS microservices (#6555)
kbaley Apr 11, 2024
a348b9b
Flesh out AWS architecture intro (#6568)
kbaley Apr 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Edit messaging page (#6535)
* Edit aws sqs page

* Update architecture/aws/messaging.md

* Apply suggestions from code review

Co-authored-by: Kyle Baley <kyle.baley@particular.net>

* Update architecture/aws/messaging.md

---------

Co-authored-by: Nisha Chaganlal <nisha.chaganlal@gmail.com>
Co-authored-by: Kyle Baley <kyle.baley@particular.net>
  • Loading branch information
3 people authored Apr 4, 2024
commit b9a6f7c61b270543094508aa928bfd5285725183
87 changes: 33 additions & 54 deletions architecture/aws/messaging.md
Original file line number Diff line number Diff line change
@@ -1,83 +1,62 @@
---
title: AWS messaging services
summary:
reviewed: 2024-03-14
reviewed: 2024-03-28
callsToAction: ['solution-architect', 'poc-help']
---

AWS provides a few messaging options that one can take advantage of when using NServiceBus.
AWS offers [multiple messaging services](https://aws.amazon.com/messaging/). The Particular Service Platform offers messaging support within AWS through [Amazon SQS](https://aws.amazon.com/sqs/) or a [SQL Server-based transport](#sql-transport).

The Particular Service Platform already offers in NServiceBus the Amazon SQS transport that leverages the technologies highlighted below.

### Amazon SNS

[https://aws.amazon.com/sns/](https://aws.amazon.com/sns/)
### Amazon SQS

[Amazon Simple Notification Service](https://aws.amazon.com/sns/) (Amazon SNS) is a fully managed web service that enables publishers to send messages to subscribers using various endpoint types. Publishers can communicate asynchronously with subscribers by sending messages to a topic, which is a logical access point and communication channel. Subscribers can receive messages using a supported endpoint type, such as Amazon Data Firehose, Amazon SQS, AWS Lambda, HTTP, email, mobile push notifications, and mobile text messages (SMS). Amazon SNS can be used for application-to-application (A2A) messaging, where messages are delivered from one application to another, or for application-to-person (A2P) messaging, where messages are delivered to customers with SMS texts, push notifications, and email. Amazon SNS provides features such as message filtering, batching, ordering, deduplication, encryption, and delivery retries to help developers build reliable, scalable, and secure applications.
[Amazon Simple Queue Service](https://aws.amazon.com/sqs/) (Amazon SQS) is a scalable and managed message queuing service provided by AWS that enables the decoupling of application components. This service is designed to help developers build robust distributed applications, making it easier to manage traffic, system failures, and complex workflows. Amazon SQS provides a reliable and secure platform for sending, storing, and receiving messages at any volume, thereby streamlining the process of building and scaling microservices, distributed systems, and serverless applications.

:heavy_plus_sign: Pros:

- It provides instantaneous, push-based delivery of messages, which eliminates the need to poll for new information and updates.
- It has simple APIs and easy integration with applications, which reduces the development effort and complexity.
- It supports flexible message delivery over multiple transport protocols, such as HTTP, email, SMS, and mobile push notifications.
- It has an inexpensive, pay-as-&#121;ou-go model with no up-front costs, which lowers the operational expenses.
- Highly scalable
- Highly reliable
- Easy integration with other AWS services
- Sensitive data is secured through server-side encryption (SSE)
- Cost-effective: charges are based on usage reducing the need for capacity planning and pre-provisioning

:heavy_minus_sign: Cons:

- It is not suitable for ordered message processing, as it does not guarantee the order of delivery or the number of deliveries for each message.
- It has limitations in fine-grained control over retry policies, as it only allows configuring the number of retries and the delay between retries for each endpoint type.
- It may incur additional costs for using other AWS services, such as SQS, Lambda, or S3, to process or store the messages delivered by SNS.
- It may have compatibility issues with some third-party services or applications, as it only supports a subset of protocols and format.
- Limited message size (256Kb per message).
- Limited control over retry policies which forces delegation of handling retries to consumers increasing the overall complexity of the system
- SQS supports a subset of protocols and formats which can cause compatibility issues with third-party applications
- No local store-and-forward mechanism available

---
Call to action??
---
NServiceBus addresses some of these limitations:
- Limited message size: NServiceBus allows the use of S3 to work with larger payloads. For more information, review the documentation for the [Amazon SQS transport topology](/transports/sqs/topology.md#s3) and [Amazon SQS configuration options](/transports/sqs/configuration-options.md).
- Limited control over retry policies: NServiceBus provides customizable [immediate](/nservicebus/recoverability/configure-immediate-retries.md) and [delayed](/nservicebus/recoverability/configure-delayed-retries.md) retry policies.

#### When to use Amazon SNS
[**Try the SQS transport sample →**](/samples/aws/sqs-simple/)

Amazon SNS should be used when the organization is looking for a solution to achieve higher decoupling between the publishers and the topic subscribers.
#### When to use the Amazon SQS transport

### Amazon SQS
The Amazon SQS transport should be considered the default choice for AWS-based systems. Alternatives should be considered only if SQS cannot be used for organizational reasons. The Amazon SQS transport uses Amazon SNS and S3 under the hood.

[https://aws.amazon.com/sqs/](https://aws.amazon.com/sqs/)
### SQL Server transport

[Amazon Simple Queue Service](https://aws.amazon.com/sqs/) (Amazon SQS) is a scalable and managed message queuing service provided by AWS that enables the decoupling of application components. This service is designed to help developers build robust, distributed applications with decoupled components, making it easier to manage traffic, system failures, and complex workflows. SQS provides a reliable and secure platform for sending, storing, and receiving messages at any volume, thereby streamlining the process of building and scaling microservices, distributed systems, and serverless applications.
SQL Server transport is an NServiceBus feature that can use existing SQL Server databases as feature-complete message queues.

:heavy_plus_sign: Pros:

- Highly scalable with an ability to handle large volumes of messages automatically
- Highly reliable as messages are locked during processing to prevent loss and enable concurrency.
- Easy integration with other AWS services
- Enables decoupling and scalability of microservices, distributed systems and serverless applications
- Cost effective as charges are based on usage with no upfront costs easing up the need to do capacity planning and pre-provisioning
- Secure as it allows components to send sensitive data between applications either by managing keys using AWS Key management (AWS KMS) and by using Amazon SQS server side encryption (SSE)
- Durable as messages are stored on multiple servers
- Supports message deduplication
- Queues can be fully customizable
- Runs on infrastructure which often already exists
- Strong transaction integration with business data operations
- Runs on cloud-hosted and on-premises SQL Server-compatible data stores (including Amazon RDS)
- Arbitrary message sizes
- Allows for exactly-once processing if business data and message data are in the same database
- Ease of backup and recovery as business data and messages are backed up in the same database

:heavy_minus_sign: Cons:

- Limited message size (256Kb per message). NServiceBus mitigates this by allowing one to take advantage, in a seamless way, of S3 to work with larger payloads. For more information review the documentation for the [Amazon SQS transport topology](/transports/sqs/topology.md#s3) and [Amazon SQS configuration options](/transports/sqs/configuration-options.md).
- Limited control over retry policies which forces delegation of handling retries to consumers increasing the overall complexity of the system
- Messages are only visible for a configurable period of time which can lead to challenges when failures occur
- As the system grows in complexity, managing a large number of queues can be challenging
- Even with FIFO (First-In-First-Out) queues, strict message ordering can be a challenge increasing complexity and impact system performance
- SQS supports a subset of protocols and formats which can originate compatibility issues with third party applications

---
Call to action

- [Simple AmazonSQS Transport usage • Amazon SQS Transport Samples • Particular Docs](/samples/aws/sqs-simple/) (include DataBus properties info)
- [https://docs.particular.net/samples/aws/sagas-lambda-aurora/](/samples/aws/sagas-lambda-aurora/)
---
- More expensive and laborious to scale
- Impacts overall database performance
- Lower message throughput compared to specialized message queuing technologies

#### When to use Amazon SQS
[**Try the SQL transport sample →**](/samples/sqltransport/simple/)

Use Amazon SQS when there is a need for a solution:
#### When to use SQL transport

- To decouple microservices and facilitate asynchronous communication between them
- Where there isn't a need replay events or commands to understand current state of entities/processes
- When there is a need to manage workloads that require data processing in batches
- When there is a need to send notifications or alerts within an application
- When there is a need to do data ingestion and use it as a buffer for incoming requests
- That requires a fanout to send identical copies of a message to multiple queues in parallel - combined with Amazon SNS.
Consider SQL transport if an existing application already uses a SQL Server-compatible data store and limited amount of messaging is being introduced. SQL transport can be a good stepping-stone when introducing messaging into an existing system without the introduction of new infrastructure.
7 changes: 5 additions & 2 deletions architecture/azure/messaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,16 +66,19 @@ SQL transport is an NServiceBus feature that can use existing SQL Server databas

- Runs on infrastructure which often already exists
- Strong transaction integration with business data operations
- Runs on cloud hosted and on-premises SQL Server-compatible data stores (including SQL Server Express for development, testing, and CI)
- Runs on cloud-hosted and on-premises SQL Server-compatible data stores (including SQL Server Express for development, testing, and CI)
- Arbitrary message sizes
- Can cater for exactly-once processing if business data and message data are in the same database
- Ease of backup and recovery as business data and messages are backed up in the same database

:heavy_minus_sign: Cons:

- Not an actual message queue
- More expensive and laborious to scale
- Impacts overall database performance
- Lower message throughput compared to specialized message queuing technologies

[**Try the SQL transport sample →**](/samples/azure/storage-queues/)
[**Try the SQL transport sample →**](/samples/sqltransport/simple/)

#### When to use SQL transport

Expand Down
Loading