Updates Microservice book on Resilient communication features with HttpClientFactory in .NET Core 2.1 with Polly #5900

CESARDELATORRE · 2018-06-11T03:43:43Z

Summary

Updates Microservice book on Resilient communication features with HttpClientFactory in .NET Core 2.1 with Polly

…/dotnet-architecture/docs into features/resilient-communication

mairaw

Didn't review everything yet but wanted to give you some feedback already.

mairaw · 2018-06-11T06:23:15Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

 ---
-title: Implementing custom HTTP call retries with exponential backoff
-description: .NET Microservices Architecture for Containerized .NET Applications | Implementing custom HTTP call retries with exponential backoff
+title: Explore custom HTTP call retries with exponential backoff


do we need to rename this?

Perhaps change from "Explore" to "Implement"? Then, it would still match the file name (which is better SEO)

This section is meant to be named "Explore" because it is not how you should implement it as we have better API. This is an initial exploration if you had to do it from scratch, so you can compare.
I'm renaming the files to explore, ok? 👍

mairaw · 2018-06-11T06:27:36Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

-title: Implementing custom HTTP call retries with exponential backoff
-description: .NET Microservices Architecture for Containerized .NET Applications | Implementing custom HTTP call retries with exponential backoff
+title: Explore custom HTTP call retries with exponential backoff
+description: .NET Microservices Architecture for Containerized .NET Applications | Explore custom HTTP call retries with exponential backoff. The purpose of this secction is just to explain and explore the problem.


typo: secction

The description could be something like "Learn how you could handle possible HTTP failure scenarios using your own implementation..."

It needs to help people pick your topic when seeing it on search engine results. For example, this is how it looks right now:

mairaw · 2018-06-11T06:29:22Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

+**Important note:** The purpose of this section is just to show you how you could try to create your own custom code to implement HTTP call retries. However, it is not recommended to do it by your own but to use more powerful and reliable while simpler to use mechanisms, such as HttpClientFactory with Polly, available since .NET Core 2.1. Those recommended approaches are explained in the next sections. 

-As an initial exploration, you could implement your own code with a utility class for exponential backoff as in [RetryWithExponentialBackoff.cs](https://gist.github.com/CESARDELATORRE/6d7f647b29e55fdc219ee1fd2babb260), plus code like the following (which is also available on a [GitHub repo](https://gist.github.com/CESARDELATORRE/d80c6423a1aebaffaf387469f5194f5b)).
+As an initial exploration, you could implement your own code with a utility class for exponential backoff as in [RetryWithExponentialBackoff.cs](https://gist.github.com/CESARDELATORRE/6d7f647b29e55fdc219ee1fd2babb260), plus code like the following (which is also available at this [GitHub repo](https://gist.github.com/CESARDELATORRE/d80c6423a1aebaffaf387469f5194f5b)).


can we add these to the dotnet/samples repo instead? That way, you could just refer to the file here and it would get imported. We don't usually use gists for source code sharing in Docs.

Also, we verify them using a CI build. You can still refer to the location on GitHub for ease of access to the raw code.

mairaw · 2018-06-11T06:31:26Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

+In order to create resilient microservices, you would need to handle possible HTTP failure scenarios. For that purpose, you could create your own implementation of retries with exponential backoff.

-In addition to handling temporal resource unavailability, the exponential backoff also needs to take into account that the cloud provider might throttle availability of resources to prevent usage overload. For example, creating too many connection requests very quickly might be viewed as a Denial of Service ([DoS](https://en.wikipedia.org/wiki/Denial-of-service_attack)) attack by the cloud provider. As a result, you need to provide a mechanism to scale back connection requests when a capacity threshold has been encountered.
+**Important note:** The purpose of this section is just to show you how you could try to create your own custom code to implement HTTP call retries. However, it is not recommended to do it by your own but to use more powerful and reliable while simpler to use mechanisms, such as HttpClientFactory with Polly, available since .NET Core 2.1. Those recommended approaches are explained in the next sections. 


section -> article?

Second sentence is a bit hard to parse. Suggestion:
However, we recommend that you use for your microservices more powerful and reliable yet simpler to use mechanisms, such as HttpClientFactory with Polly. The HttpClientFactory API is available since .NET Core 2.1.

use the traditional important notes for docs?

> [!IMPORTANT] > Text

I'd consider simplifying the first sentence. Try "This section shows you how you could create your own custom code to ..."

mairaw · 2018-06-11T06:36:02Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

+# Explore custom HTTP call retries with exponential backoff

-In order to create resilient microservices, you need to handle possible HTTP failure scenarios. For that purpose, you could create your own implementation of retries with exponential backoff.
+In order to create resilient microservices, you would need to handle possible HTTP failure scenarios. For that purpose, you could create your own implementation of retries with exponential backoff.


In order to create -> To create
I'd not include would here

For that purpose, you could create your own implementation of retries with exponential backoff. ->
perhaps you should make it more clear here that is one of the options.
For example: One way of handling those failures, although not recommended, is to create your own implementation of retries...

mairaw · 2018-06-11T06:48:15Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

 ```

-However, this code is suitable only as a proof of concept. The next topic explains how to use more sophisticated and proven libraries.
+However, and as mentioned, this code is suitable only as a proof of concept. The next topic explains how to use more sophisticated approaches while simpler to use with HttpClientFactory, available since .NET Core 2.1, plus proven libraries like Polly.


+However, and as mentioned, -> Remember that

next topic -> next article

Last sentence is too long. Break it down similar to what I showed earlier.

mairaw · 2018-06-11T06:51:28Z

...re/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly.md

@@ -1,167 +1,83 @@
 ---
 title: Implementing HTTP call retries with exponential backoff with Polly
 description: .NET Microservices Architecture for Containerized .NET Applications | Implementing HTTP call retries with exponential backoff with Polly


update title too and description

mairaw · 2018-06-11T06:52:17Z

...re/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly.md

-    // Other methods ...
-}
-```
+Your project has to be using the ASPNET Core 2.1 packages from NuGet. You typically need the AspNetCore metapackage, and the extension package Microsoft.Extensions.Http.Polly.


codefence the non-translatable parts

ASPNET -> ASP.NET

mairaw · 2018-06-11T06:52:49Z

...re/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly.md

+**Configure a client with Polly’s Retry policy, in Startup**

-You can use the standard implementation if you do not want to use a resilient mechanism, as when you are developing or testing simpler approaches. The following code shows the standard HttpClient implementation allowing requests with authentication tokens as an optional case.
+As previously shown in previous sections, you need to define a named or typed client HttpClient configuration in your standard Startup.ConfigureServices(...) method, but now, you add incremental code specifying the policy for the Http retries with exponential backoff, as below:


As previously shown in previous sections -> As shown in previous sections

as below -> as follows, as in the following example

when talking about APIs that exist in docs, you can link to them. for example:
<xref:System.Net.Http.HttpClient>

mairaw · 2018-06-11T06:57:03Z

...re/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly.md

-        // Rest of the code and other Http methods ...
+//ConfigureServices()  - Startup.cs
+services.AddHttpClient<IBasketService, BasketService>()
+        .SetHandlerLifetime(TimeSpan.FromMinutes(5))  //Set lifetime to 5 minutes


style guide asks to spell out numbers from zero to nine
https://docs.microsoft.com/en-us/style-guide/numbers#numerals-vs-words

mairaw · 2018-06-11T06:58:26Z

...oservices-architecture/implement-resilient-applications/implement-circuit-breaker-pattern.md

 ```

-Whenever the \_apiClient member object is used, it internally uses the wrapper class with Polly policiesؙ—the Retry policy, the Circuit Breaker policy, and any other policy that you might want to apply from the Polly policies collection.
+In the code example above, the circuit breaker policy is configured so it break or open the circuit when there have been five exceptions when retrying the Http requests. Then, 30 seconds will be the duration or the break.


in the code example above -> in the previous example

mairaw · 2018-06-11T06:59:59Z

...mplement-resilient-applications/implement-resilient-entity-framework-core-sql-connections.md


 The first DbContext is \_catalogContext and the second DbContext is within the \_integrationEventLogService object. The Commit action is performed across multiple DbContexts using an EF execution strategy.

+


don't add extra empty lines

mairaw · 2018-06-11T07:04:18Z

...plement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests.md

+
+The original and well-know [HttpClient](https://docs.microsoft.com/dotnet/api/system.net.http.httpclient?view=netstandard-2.0) class can be easily used, but in some cases, it is not being properly used by many developers. 
+
+As a first issue, while this class is disposable, using it with the using statement is not the best choice because even when you dispose HttpClient object, the underlying socket is not immediately released and can cause a serious issue named ‘sockets exhaustion’ as explained in this [blog post by Simon Timms](https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/). 


for blog posts, we usually refer to them by their titles, not by who wrote. Using title also helps with SEO. For example:
as explained in this blog post by Simon Timms ->
For more information about X, see You're using HttpClient wrong and it is destabilizing your software blog post.

mairaw · 2018-06-11T07:05:00Z

...plement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests.md

+- Use Typed Clients
+- Use Generated Clients
+
+For the sake of brevity, this guidance shows the most structured way to use HttpClientFactory that is to use Typed Clients (Service Agent pattern), but all options are documented and are currently listed in this [article covering HttpClientFactory usage](https://docs.microsoft.com/aspnet/core/fundamentals/http-requests?view=aspnetcore-2.1#consumption-patterns).  


don't add the view query parameter. otherwise, you'll need to change once a new version comes.

BillWagner

Overall this looks good. (I can tell you worked hard to get the Acrolinx score higher. It shows).

I left a few comments, some kudos, and one general suggestion, which you can consider as you keep updating content.

Let me know if you have any questions.

BillWagner · 2018-06-12T20:47:56Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

 ---
-title: Implementing custom HTTP call retries with exponential backoff
-description: .NET Microservices Architecture for Containerized .NET Applications | Implementing custom HTTP call retries with exponential backoff
+title: Explore custom HTTP call retries with exponential backoff


Perhaps change from "Explore" to "Implement"? Then, it would still match the file name (which is better SEO)

BillWagner · 2018-06-12T20:57:19Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

+In order to create resilient microservices, you would need to handle possible HTTP failure scenarios. For that purpose, you could create your own implementation of retries with exponential backoff.

-In addition to handling temporal resource unavailability, the exponential backoff also needs to take into account that the cloud provider might throttle availability of resources to prevent usage overload. For example, creating too many connection requests very quickly might be viewed as a Denial of Service ([DoS](https://en.wikipedia.org/wiki/Denial-of-service_attack)) attack by the cloud provider. As a result, you need to provide a mechanism to scale back connection requests when a capacity threshold has been encountered.
+**Important note:** The purpose of this section is just to show you how you could try to create your own custom code to implement HTTP call retries. However, it is not recommended to do it by your own but to use more powerful and reliable while simpler to use mechanisms, such as HttpClientFactory with Polly, available since .NET Core 2.1. Those recommended approaches are explained in the next sections. 


I'd consider simplifying the first sentence. Try "This section shows you how you could create your own custom code to ..."

BillWagner · 2018-06-12T21:00:23Z

...e/implement-resilient-applications/implement-custom-http-call-retries-exponential-backoff.md

+**Important note:** The purpose of this section is just to show you how you could try to create your own custom code to implement HTTP call retries. However, it is not recommended to do it by your own but to use more powerful and reliable while simpler to use mechanisms, such as HttpClientFactory with Polly, available since .NET Core 2.1. Those recommended approaches are explained in the next sections. 

-As an initial exploration, you could implement your own code with a utility class for exponential backoff as in [RetryWithExponentialBackoff.cs](https://gist.github.com/CESARDELATORRE/6d7f647b29e55fdc219ee1fd2babb260), plus code like the following (which is also available on a [GitHub repo](https://gist.github.com/CESARDELATORRE/d80c6423a1aebaffaf387469f5194f5b)).
+As an initial exploration, you could implement your own code with a utility class for exponential backoff as in [RetryWithExponentialBackoff.cs](https://gist.github.com/CESARDELATORRE/6d7f647b29e55fdc219ee1fd2babb260), plus code like the following (which is also available at this [GitHub repo](https://gist.github.com/CESARDELATORRE/d80c6423a1aebaffaf387469f5194f5b)).


Also, we verify them using a CI build. You can still refer to the location on GitHub for ease of access to the raw code.

BillWagner · 2018-06-12T21:12:14Z

...re/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly.md

 ```

-The interesting implementation is to code another, similar class, but using Polly to implement the resilient mechanisms you want to use—in the following example, retries with exponential backoff.
+The **AddPolicyHandler()** method is what adds policies to the HttpClient objects you will use. In this case, it is adding a Polly’s policy for Http Retries with exponential backoff.


Code fence HttpClient (to avoid translation.)

That section was originally named "implement" but since it is just an initial approximation to explore but not recommended to implement, that's why I prefer to call it "Explore".
I didn't change the filename since we might not need to..
👍 to the rest of suggestions.

BillWagner · 2018-06-13T02:05:19Z

...re/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly.md

 ```

-With Polly, you define a Retry policy with the number of retries, the exponential backoff configuration, and the actions to take when there is an HTTP exception, such as logging the error. In this case, the policy is configured so it will try the number of times specified when registering the types in the IoC container. Because of the exponential backoff configuration, whenever the code detects an HttpRequest exception, it retries the Http request after waiting an amount of time that increases exponentially depending on how the policy was configured.
+With Polly, you can define a Retry policy with the number of retries, the exponential backoff configuration, and the actions to take when there is an HTTP exception, such as logging the error. In this case, the policy is configured so it will try six times and the seconds between each retry will be exponential, starting on two seconds.


This would be more clear for the last sentence:

"In this case, the policy is configured to try six times with an exponential retry, starting at two seconds."

BillWagner · 2018-06-13T13:55:26Z

...oservices-architecture/implement-resilient-applications/implement-circuit-breaker-pattern.md


-## Implementing a Circuit Breaker pattern with Polly
+Instead, the application should be coded to accept that the operation has failed and handle the failure accordingly.
+In addition, a dangerous risk you can have when using Http retries is that when any microservice is failing or simply performing very slowly, the other client apps or services might start retrying http requests. If that failure persists for a long time, because of so many Http retries you could be causing a Denial of Service ([DoS](https://en.wikipedia.org/wiki/Denial-of-service_attack)) attack to yourself. 


This reads awkwardly. Your main point is at the end of the paragraph. Consider:

Using Http retries carelessly could result in creating a Denial of Service (DoS) attack within your own software. As a microservice fails or performs slowly, multiple clients might repeatedly retry failed requests. That creates a dangerous risk of exponentially increasing traffic targeted at the failing service.

BillWagner · 2018-06-13T13:56:21Z

...oservices-architecture/implement-resilient-applications/implement-circuit-breaker-pattern.md

+Therefore, you need some kind of defense barrier so the retries stop requests when it is not worth to keep trying. That defense barrier is precisely the circuit breaker.

-The eShopOnContainers application uses the Polly Circuit Breaker policy when implementing HTTP retries. In fact, the application applies both policies to the ResilientHttpClient utility class. Whenever you use an object of type ResilientHttpClient for HTTP requests (from eShopOnContainers), you will be applying both those policies, but you could add additional policies, too.
+The Circuit Breaker pattern has a different purpose than the Retry pattern. The Retry pattern enables an application to retry an operation in the expectation that the operation will eventually succeed. The Circuit Breaker pattern prevents an application from performing an operation that is likely to fail. An application can combine these two patterns. However, the retry logic should be sensitive to any exception returned by the circuit breaker, and it should abandon retry attempts if the circuit breaker indicates that a fault is not transient.


💯 This is a very clear description, and explains the distinctions very well.

BillWagner · 2018-06-13T14:01:16Z

...oservices-architecture/implement-resilient-applications/implement-circuit-breaker-pattern.md

+The *AddPolicyHandler()* method is what adds policies to the HttpClient objects you will use. In this case, it is adding a Polly’s policy for a circuit breaker.

-You use the ResilientHttpClient utility class in a way similar to how you use the .NET HttpClient class. In the following example from the eShopOnContainers MVC web application (the OrderingService agent class used by OrderController), the ResilientHttpClient object is injected through the httpClient parameter of the constructor. Then the object is used to perform HTTP requests.
+In order to have a more modular approach, the Http Retry Policy can be defined in a separate method named GetCircuitBreakerPolicy(), as the following code.


I think this is confusing. You make a great distinction between "retry" and "circuit breaker" in other text. Now, the retry policy is defined in a method called GetCircuitBreakerPolicy(). I think the text should change.

BillWagner · 2018-06-13T14:01:29Z

...oservices-architecture/implement-resilient-applications/implement-circuit-breaker-pattern.md

+```

-## Using the ResilientHttpClient utility class from eShopOnContainers
+The *AddPolicyHandler()* method is what adds policies to the HttpClient objects you will use. In this case, it is adding a Polly’s policy for a circuit breaker.


Code fence AddPolicyHandler().

Right, that was a typo. I was meaning the Circuit Breaker policy, not the Retry policy. Good Catch!
Added also the Code fence AddPolicyHandler() 👍

BillWagner · 2018-06-13T14:16:18Z

...oservices-architecture/implement-resilient-applications/implement-circuit-breaker-pattern.md

+You can implement different logic for when to open/break the circuit. Or you can try an HTTP request against a different back-end microservice if there is a fallback datacenter or redundant back-end system. 

-Finally, another possibility for the CircuitBreakerPolicy is to use Isolate (which forces open and holds open the circuit) and Reset (which closes it again). These could be used to build a utility HTTP endpoint that invokes Isolate and Reset directly on the policy. Such an HTTP endpoint could also be used, suitably secured, in production for temporarily isolating a downstream system, such as when you want to upgrade it. Or it could trip the circuit manually to protect a downstream system you suspect to be faulting.
+Finally, another possibility for the CircuitBreakerPolicy is to use Isolate (which forces open and holds open the circuit) and Reset (which closes it again). These could be used to build a utility HTTP endpoint that invokes Isolate and Reset directly on the policy.  Such an HTTP endpoint could also be used, suitably secured, in production for temporarily isolating a downstream system, such as when you want to upgrade it. Or it could trip the circuit manually to protect a downstream system you suspect to be faulting.


Code fence CircuitBreakerPolicy, Isolate and Reset.

CESARDELATORRE · 2018-06-18T18:54:02Z

Updates based on Bill's and Maira's feedback are already pushed into the branch.
Please, merge when possible. 👍

CESARDELATORRE · 2018-07-03T21:04:15Z

I have pushed additional updates based on Bill's and Maira's feedback. Please, review and merge when possible. Thanks! 👍

CESARDELATORRE added 3 commits June 10, 2018 20:37

Uodates HttpClientFactory netcore2.1 with Polly Retries & Circuit-Breaks

89ae426

Uodates HttpClientFactory netcore2.1 with Polly Retries & Circuit-Breaks

42b7934

Merge branch 'features/resilient-communication' of https://github.com…

66febc4

…/dotnet-architecture/docs into features/resilient-communication

mairaw reviewed Jun 11, 2018

View reviewed changes

BillWagner requested changes Jun 13, 2018

View reviewed changes

Updates based on Bill Wagner's feedback

caaf450

mairaw and others added 2 commits July 3, 2018 01:44

Merge branch 'master' into features/resilient-communication

85e8dca

Updates based on Feedback from Maira and Bill

978c095

BillWagner approved these changes Jul 5, 2018

View reviewed changes

BillWagner merged commit 1b772c4 into dotnet:master Jul 5, 2018


		The first DbContext is \_catalogContext and the second DbContext is within the \_integrationEventLogService object. The Commit action is performed across multiple DbContexts using an EF execution strategy.


		The original and well-know [HttpClient](https://docs.microsoft.com/dotnet/api/system.net.http.httpclient?view=netstandard-2.0) class can be easily used, but in some cases, it is not being properly used by many developers.

		As a first issue, while this class is disposable, using it with the using statement is not the best choice because even when you dispose HttpClient object, the underlying socket is not immediately released and can cause a serious issue named ‘sockets exhaustion’ as explained in this [blog post by Simon Timms](https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/).

Updates Microservice book on Resilient communication features with HttpClientFactory in .NET Core 2.1 with Polly #5900

Updates Microservice book on Resilient communication features with HttpClientFactory in .NET Core 2.1 with Polly #5900

Uh oh!

Conversation

CESARDELATORRE commented Jun 11, 2018

Summary

Uh oh!

mairaw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CESARDELATORRE Jul 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BillWagner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

CESARDELATORRE Jul 3, 2018 •

edited

Loading