Using Polly In .NET For Resilient HTTP Connections
A network call is a boundary between your application and a system you do not control. The remote API may be restarting, a load balancer may close an idle connection, DNS may briefly fail, or a service may return a temporary overload response. In a local development environment, these issues can be easy to miss because everything runs quickly on the same machine. In production, they become ordinary operating conditions.
Polly is a .NET resilience library for handling these transient failures in a deliberate way. It provides policies and resilience pipelines for retries, timeouts, circuit breakers, rate limiting, fallbacks and related behaviours. When it is combined with IHttpClientFactory, a .NET application can send HTTP requests with consistent rules while avoiding the connection-management problems associated with creating a new HttpClient for every call.
This matters for Australian applications as much as anywhere else. An API hosted in Sydney might depend on a payment provider in Melbourne, a CRM platform in Singapore or an identity service in the United States. A brief interruption between those locations can affect a customer checking out in Brisbane or a field worker using a tablet outside Perth. Resilience should account for real network conditions rather than assuming every request will complete immediately.
The goal is not to repeat a failed request until it eventually succeeds. Poorly configured retries can increase load during an outage, duplicate non-idempotent operations and make an already slow user experience worse. A good Polly configuration distinguishes temporary faults from permanent ones, limits the amount of work performed and exposes enough telemetry to explain what happened.
Understanding Transient HTTP Failures
A transient failure is an error that may disappear if the operation is attempted again after a short delay. Examples include HTTP 408 Request Timeout, 429 Too Many Requests and many 5xx responses. A connection reset, temporary DNS problem or gateway timeout can also be transient, depending on the service and the operation being performed.
Permanent failures need different treatment. A 400 response caused by invalid JSON will not be fixed by sending the same JSON three more times. A 401 response may require a refreshed access token, while a 404 response could indicate an incorrect resource identifier. Polly can technically retry many kinds of exceptions, but the policy should reflect the meaning of the request and the contract of the remote API.
Retries also need an idempotency decision. Repeating a GET is usually safe, while repeating a POST that creates an order may create duplicate records. An API that supports idempotency keys gives the client a safer way to retry writes. Without that protection, a timeout after the server accepted a payment request leaves the client unable to know whether a second attempt is safe.
Timeouts are an important part of this model. A retry without a timeout can leave several requests waiting for an unresponsive server, consuming threads and sockets. Set a reasonable overall deadline, then ensure individual attempts cannot run indefinitely. In Australia, where an application may serve users across long regional links and mobile networks, the timeout should allow for normal latency without allowing a dead dependency to hold the request open forever.
Registering Resilience With HttpClientFactory
IHttpClientFactory centralises the creation and configuration of named or typed clients. It manages handler lifetimes, supports dependency injection and makes it easier to apply the same resilience rules to every call made to a particular API. This is preferable to constructing new HttpClient() inside a controller or service method.
Modern Polly integrations use the resilience pipeline APIs available through packages such as Microsoft.Extensions.Http.Resilience. A typical registration can look like this:
using Microsoft.Extensions.Http.Resilience;
builder.Services.AddHttpClient<WeatherClient>(client =>
{
client.BaseAddress = new Uri("https://api.example.com/");
client.Timeout = Timeout.InfiniteTimeSpan;
})
.AddStandardResilienceHandler();
The standard handler supplies a sensible combination of rate limiting, total request timeout, retry and circuit breaker behaviour. Setting the HttpClient timeout to infinite in this example allows the resilience handler to own the timeout boundary. If two unrelated timeout mechanisms are configured, the shorter one may cancel the request before the intended pipeline can record or handle the failure.
For a production API, explicit settings are often easier to reason about:
builder.Services.AddHttpClient<OrdersClient>(client =>
{
client.BaseAddress = new Uri("https://orders.example.com/");
client.Timeout = Timeout.InfiniteTimeSpan;
})
.AddResilienceHandler("orders-pipeline", pipeline =>
{
pipeline.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3,
Delay = TimeSpan.FromMilliseconds(500),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true
});
pipeline.AddTimeout(TimeSpan.FromSeconds(10));
pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
SamplingDuration = TimeSpan.FromSeconds(30),
MinimumThroughput = 10,
BreakDuration = TimeSpan.FromSeconds(15)
});
});
The exact package versions and option names should match the .NET and Polly packages used by the project. Polly v8 introduced a pipeline-oriented model, while older tutorials often show PolicyWrap and WaitAndRetryAsync from Polly v7. Those examples can still appear in existing applications, but new projects should avoid mixing old and new registration styles without understanding how the handlers interact.
A typed client keeps the HTTP boundary in a focused class:
public sealed class OrdersClient(HttpClient httpClient)
{
public async Task<Order?> GetOrderAsync(
string orderId,
CancellationToken cancellationToken)
{
using var response = await httpClient.GetAsync(
$"orders/{orderId}",
cancellationToken);
response.EnsureSuccessStatusCode();
return await response.Content.ReadFromJsonAsync<Order>(
cancellationToken);
}
}
The cancellation token should come from the incoming request, background job or message-processing operation. Resilience must stop when the caller has gone away. A user in Adelaide should not leave a server-side retry pipeline running after closing the browser tab, and a cancelled queued job should not continue consuming capacity.
Choosing Retry And Backoff Rules
A retry policy needs three decisions: which outcomes are retryable, how many attempts are allowed and how long to wait between them. Exponential backoff increases the delay after each failed attempt, reducing pressure on a struggling dependency. Jitter adds a small random variation so thousands of clients do not retry at the same instant.
The server’s response should influence the delay. A Retry-After header on a 429 response may tell the client when to try again. Ignoring that instruction can turn a rate limit into a longer outage. A maximum delay is still useful because an external service might provide an impractically large value or a malformed one.
Use the following checks when defining an HTTP retry strategy:
- Retry temporary transport failures and carefully selected 408, 429 and 5xx responses.
- Respect
Retry-Afterwhen the remote service supplies it. - Add exponential backoff with jitter to spread competing clients over time.
- Keep the attempt count low and cap the total elapsed time.
- Exclude validation, authentication and other failures that need a different fix.
Retries should happen at the correct level. Retrying a whole business operation can repeat several downstream calls and create unexpected side effects. Retrying the individual safe HTTP request is usually easier to analyse. If a workflow needs several calls, consider whether it has a durable job model, idempotency key or compensation mechanism rather than wrapping the entire workflow in a broad retry.
A retry callback is a useful place for structured logging, but it should not log sensitive request bodies or tokens. Record the client name, target operation, attempt number, delay and outcome. In a busy Melbourne e-commerce service, those fields help distinguish an isolated upstream timeout from a retry storm affecting every checkout request.
Protecting Dependencies With Circuit Breakers
A circuit breaker prevents an application from repeatedly calling a dependency that is already failing. In the closed state, calls flow normally. After enough failures within a sampling period, the breaker opens and rejects calls immediately for a defined duration. A later half-open state allows a limited test request; success closes the circuit, while failure opens it again.
The breaker does not repair the remote service. Its value is protecting local resources and making failure quicker and more predictable. Without one, every web request may wait through several retries against an unavailable API. Thread pools, connection pools and queues can fill up, causing a failure in one dependency to spread through the application.
A breaker should be tuned from traffic levels and dependency behaviour. MinimumThroughput prevents a tiny number of requests from opening the circuit too easily, while FailureRatio determines how many calls must fail. A high-volume public API may need a shorter sampling window than a low-volume internal administrative tool. Measure before selecting values rather than copying a configuration from an unrelated service.
Fallbacks can provide a controlled alternative when a circuit is open. A product catalogue might return recently cached data, while a flight-booking operation should clearly report that it cannot proceed. A fallback must never pretend a critical write succeeded. The response should preserve the distinction between unavailable, declined and completed states so that users and support staff can act on it.
When the dependency is part of a locally reproducible development stack, Docker can make resilience tests easier to run. For example, Docker Compose setup can provide a .NET service and SQL Server together, allowing developers to stop a container, introduce latency or restart a dependency while observing the client’s behaviour.
Testing And Observing Resilience
A resilience pipeline is production code and should be tested as deliberately as authentication or database access. Unit tests can use a custom HttpMessageHandler that returns a sequence such as 503, 503 and then 200. The test can verify the number of attempts, the final response and whether cancellation stops further work.
Integration tests should cover timeout and cancellation paths as well. A fake server can delay its response, close the connection or return a 429 with Retry-After. These tests reveal whether the timeout is owned by Polly, the HTTP client or a hosting layer, and whether the application incorrectly retries a request that should be rejected immediately.
Useful resilience tests include:
- A safe
GETsucceeds after two temporary 503 responses. - A 400 or 401 response is returned without unnecessary retries.
- A 429 response honours a bounded server-provided delay.
- An open circuit rejects calls quickly and later permits a probe.
- A cancelled request ends the pipeline without further attempts.
Observability completes the design. Emit metrics for attempts, retry delays, timeout counts, circuit state changes, rate-limit responses and final outcomes. Use distributed tracing so an outgoing request and its retries can be connected to the incoming request. A dashboard should show both the dependency’s failure rate and the additional latency introduced by resilience.
Pay attention to Australian operating patterns when setting alerts. Traffic may peak around Australian Eastern Time business hours, end-of-month invoicing or major retail events such as Boxing Day sales. A service used by regional customers may have a different latency profile from one serving Sydney data-centre traffic. Capture location, client version and dependency name without collecting unnecessary personal information.
Resilience settings should be reviewed alongside capacity and deployment changes. A new retry rule can multiply traffic during an incident, and a circuit breaker can hide a dependency problem if its open state is never alerted. Treat policy changes as observable operational changes, with documented reasons and rollback options.
A reliable HTTP integration combines bounded retries, timeouts, circuit breaking, cancellation, idempotency and clear telemetry. Polly supplies the building blocks, while the application owner must decide which failures are safe to repeat and which require human or business intervention. Apply those rules through IHttpClientFactory, test them against realistic failure modes and keep the policy close to the client that understands the remote API.
Add a small failure-injection test to the next .NET service you maintain, then inspect its logs and metrics during a controlled outage. That practical exercise will expose unsafe retries and unclear timeouts before a real dependency failure reaches customers.