Using Polly with F# async workflows by Mark Seemann

How to use Polly as a Circuit Breaker in F# async workflows.

Release It! describes a stability design pattern called Circuit Breaker, which is used to fail fast if a downstream service is experiencing problems.

I recently had to add a Circuit Breaker to an F# async workflow, and although Circuit Breaker isn't that difficult to implement (my book contains an example in C#), I found it most prudent to use an existing implementation. Polly seemed a good choice.

In my F# code base, I was already working with an 'abstraction' of the type HttpRequestMessage -> Async<HttpResponseMessage>: given an HttpClient called client, the implementation is as simple as client.SendAsync >> Async.AwaitTask. Since SendAsync can throw HttpRequestException or WebException, I wanted to define a Circuit Breaker policy for these two exception types.

While Polly supports policies for Task-based APIs, it doesn't automatically work with F# async workflows. The problem is that whenever you convert an async workflow into a Task (using Async.AwaitTask), or a Task into an async workflow (using Async.StartAsTask), any exceptions thrown will end up buried within an AggregateException. In order to dig them out again, I first had to write this function:

// Exception -> bool
let rec private isNestedHttpException (e : Exception) =
    match e with
    | :? AggregateException as ae ->
        ae.InnerException :: Seq.toList ae.InnerExceptions
        |> Seq.exists isNestedHttpException
    | :? HttpRequestException -> true
    | :? WebException -> true
    | _ -> false

This function recursively searches through all inner exceptions of an AggregateException and returns true if it finds one of the exception types I'm interested in handling; otherwise, it returns false.

This predicate enabled me to write the Polly policy I needed:

open Polly
 
// int -> TimeSpan -> CircuitBreaker.CircuitBreakerPolicy
let createPolicy exceptionsAllowedBeforeBreaking durationOfBreak =
    Policy
        .Handle<AggregateException>(fun ae -> isNestedHttpException ae)
        .CircuitBreakerAsync(exceptionsAllowedBeforeBreaking, durationOfBreak)

Since Polly exposes an object-oriented API, I wanted a curried alternative, so I also wrote this curried helper function:

// Policy -> ('a -> Async<'b>) -> 'a -> Async<'b>
let private execute (policy : Policy) f  req =
    policy.ExecuteAsync(fun () -> f req |> Async.StartAsTask) |> Async.AwaitTask

The execute function executes any function of the type 'a -> Async<'b> with a Polly policy. As you can see, there's some back-and-forth between Tasks and async workflows, so this is probably not the most efficient Circuit Breaker ever configured, but I wagered that since the underlying operation was going to involve an HTTP request and response, the overhead would be insignificant. No one has complained yet.

When Polly opens the Circuit Breaker, it throws an exception of the type BrokenCircuitException. Again because of all the marshalling, this also gets wrapped within an AggregateException, so I had to write another function to unwrap it:

// Exception -> bool
let rec private isNestedCircuitBreakerException (e : Exception) =
    match e with
    | :? AggregateException as ae ->
        ae.InnerException :: Seq.toList ae.InnerExceptions
        |> Seq.exists isNestedCircuitBreakerException
    | :? CircuitBreaker.BrokenCircuitException -> true
    | _ -> false

The isNestedCircuitBreakerException is similar to isNestedHttpException, so it'd be tempting to refactor. I decided, however, to rely on the rule of three and leave both functions as they were.

In my F# code I prefer to handle application errors using Either values instead of relying on exceptions, so I wanted to translate any BrokenCircuitException to a particular application error. With the above isNestedCircuitBreakerException predicate, this was now possible with a try/with expression:

// Policy -> ('a -> Async<'b>) -> 'a -> Async<Result<'b, BoundaryFailure>>
let sendUsingPolicy policy send req =
    async {
        try
            let! resp = req |> execute policy send
            return Result.succeed resp
        with e when isNestedCircuitBreakerException e ->
                return Result.fail Controllers.StabilityFailure }

This function takes a policy, a send function that actually sends the request, and the request to send. If all goes well, the response is lifted into a Success case and returned. If the Circuit Breaker is open, a StabilityFailure value is returned instead.

Since the with expression uses an exception filter, all other exceptions will still be thrown from this function.

It might still be worthwhile to look into options for a more F#-friendly Circuit Breaker. One option would be to work with the Polly maintainers to add such an API to Polly itself. Another option would be to write a separate F# implementation.

Comments

• ← Previous
• Archive
• Next →