Feign Outbound metrics 

With dropwizard microservices, you can easily add inbound metrics on your jax-rs http resource classes via annotations:

@Path("/example")
@Produces(MediaType.TEXT_PLAIN)
public class ExampleResource {
@GET
@Timed
@Metered
@ExceptionMetered
public String show() {
return "yay";
}
}
view raw ExampleResource.java hosted with ❤ by GitHub

The metrics can easily been reported to graphite database and visualized via Kibana.

WYIIWYG – What you instrument, is what you get!

On the other hand, a microservice often contains client libraries to access other services via http. Feign is client library, which provides a wrapper and simplifies the api to communicate to the target services.

In contrast to the inbound metrics from the example above, it is also desirable to monitor the outbound metrics of each of the targeted operations.

Looking at the third-party libraries of http://metrics.dropwizard.io/3.2.3/manual/third-party.html there is already something to retrieve metrics on http level. So in case you are using okhttp as http client implementation you can use https://github.com/raskasa/metrics-okhttp and you will receive information about request durations and connection pools.  Same holds good for Apache httpclient instrumentation.

okhttp example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrumenting feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
// instrumenting ok http
.client(new OkHttpClient(InstrumentedOkHttpClients.create(metricRegistry)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();
view raw OkHttpExample.java hosted with ❤ by GitHub

Metric output:

-- Gauges ----------------------------------------------------------------------
okhttp3.OkHttpClient.connection-pool-idle-count
value = 1
okhttp3.OkHttpClient.connection-pool-total-count
value = 1
-- Counters --------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-running
count = 0
-- Meters ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-completed
count = 1
mean rate = 0,84 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
okhttp3.OkHttpClient.network-requests-submitted
count = 1
mean rate = 0,83 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
-- Timers ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-duration
count = 1
mean rate = 0,84 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 215,41 milliseconds
max = 215,41 milliseconds
mean = 215,41 milliseconds
stddev = 0,00 milliseconds
median = 215,41 milliseconds
75% <= 215,41 milliseconds
95% <= 215,41 milliseconds
98% <= 215,41 milliseconds
99% <= 215,41 milliseconds
99.9% <= 215,41 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

httpclient example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrument feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry)).client(
// setting an instrumented httpclient
new ApacheHttpClient(InstrumentedHttpClients
.createDefault(metricRegistry, HttpClientMetricNameStrategies.HOST_AND_METHOD)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();

Metric output:

-- Gauges ----------------------------------------------------------------------
org.apache.http.conn.HttpClientConnectionManager.available-connections
value = 1
org.apache.http.conn.HttpClientConnectionManager.leased-connections
value = 0
org.apache.http.conn.HttpClientConnectionManager.max-connections
value = 20
org.apache.http.conn.HttpClientConnectionManager.pending-connections
value = 0
-- Meters ----------------------------------------------------------------------
-- Timers ----------------------------------------------------------------------
org.apache.http.client.HttpClient.api.github.com.get-requests
count = 1
mean rate = 4,19 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 174,59 milliseconds
max = 174,59 milliseconds
mean = 174,59 milliseconds
stddev = 0,00 milliseconds
median = 174,59 milliseconds
75% <= 174,59 milliseconds
95% <= 174,59 milliseconds
98% <= 174,59 milliseconds
99% <= 174,59 milliseconds
99.9% <= 174,59 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

As you can see, the provided metrics only provid information on http level, not really showing differences between different service endpoints. The only differentation is available on the httpclient metrics, which shows metrics based on host and http methods.

Closing the gap

What was missing in my eyes was a way to instrument metrics on the interface level, which is provided from the Feign builder. In my example below I am calling the github API on two different resource endpoints, contributors and repositorySearch. With the instrumentation on http, one is not able to see and monitor those one by one.

Therefore I created a library, which makes it possible to instrument metrics on method or interface level by using annotations like you do it in jersey resource classes.

Using this instrumentation you are able to retrieve metrics based on the interface and methods the client is calling. So for example when you start reporting via JMX, you are able to see the metrics in jconsole.

Usage of the library

To instrument the feign interfaces you basically have to do three things:

  1. add the maven dependency to the pom.xml of your project.
    <dependency>
      <groupId>com.github.mwiede</groupId>
      <artifactId>metrics-feign</artifactId>
      <version>1.0</version>
    </dependency>
  2. add FeignOutboundMetricsDecorator as invocationHandlerFactory in Feign.builder
  3. add the metric annotations @Timed, @Metered and @ExceptionMetered to the interface you are using with feign.
@Timed
@Metered
@ExceptionMetered
interface GitHub {
@RequestLine("GET /repos/{owner}/{repo}/contributors")
List<Contributor> contributors(@Param("owner") String owner, @Param("repo") String repo);
}
static class Contributor {
String login;
int contributions;
}
public static void main(String... args) {
MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
// Fetch and print a list of the contributors to this library.
List<Contributor> contributors = github.contributors("mwiede", "metrics-feign");
for (Contributor contributor : contributors) {
System.out.println(contributor.login + " (" + contributor.contributions + ")");
}
reporter.report();
}
view raw Example.java hosted with ❤ by GitHub

The library is available from maven central and the source is hosted at github, so please checkout https://github.com/mwiede/metrics-feign

Please follow and like us:
0

Exploring Feign – Retrying

Feign is a library, which makes it easier to implement a http client. Recently more and more people start writing http clients, because they are creating microservices which communicate with http protocol. So there are all sorts of libraries supporting this task like Jersey, Resteasy and others – and there is Feign.

Today I do not want to explain the basic functionality, this is all done on the Readme page itself. Today I want to get into the details of a feature, which becomes more and more important, because in modern distributed systems, you want to have resilient behaviour, which means that you want to design your service in the way, that it can handle unexpected situations without noticing on user’s site. For example an API you are calling is not reachable at the moment, the request times out or the requested resource is not yet available. To solve this issue, you need to apply a retry pattern, so that you increase the chance that the service request is successfull after the first, the second or the nth attempt.

What most developers don’t know, Feign has a default retryer built-in.

Now I show a few code examples, what you can expect from this feature. What I am showing are junit tests with a client mock, so that we are able to stub certain errors and verify, how many retries have been made.

Case 1) Success

no retry needed.

@Test
public void testSuccess() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(1)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 2) Destination never reachable.

In this case, we can see the Default Retryer working, which ends up doing 5 attempts, but finally the client invocation throws an exception.

@Test
public void testDefaultRetryerGivingUp() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenThrow(
new UnknownHostException());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(5)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 3) Configure maximal number of attempts

Taking the same error scenario from case 2, this example shows how to configure the retryer to stop trying after the 3rd attempt.

@Test
public void testRetryerAttempts() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenThrow(
new UnknownHostException());
final int maxAttempts = 3;
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.retryer(new Retryer.Default(1, 100, maxAttempts))
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(maxAttempts)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 4) trigger retrying by error code decoding

For some (restful) services, http status code 409 (conflict) is used to express a wrong state of the target resource, that might change after resubmitting the request. We simulate, that the first retry will lead to a successfull response.

@Test
public void testCustomRetryConfigByErrorDecoder() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(409).headers(Collections.<String, Collection<String>>emptyMap())
.build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
class RetryOn409ConflictStatus extends ErrorDecoder.Default {
@Override
public Exception decode(final String methodKey, final Response response) {
if (409 == response.status()) {
return new RetryableException("getting conflict and retry", null);
} else
return super.decode(methodKey, response);
}
}
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.errorDecoder(new RetryOn409ConflictStatus())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(2)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 4a) Behavior without error decoder

If no error decoder is configured, no retry is executed by Feign.

@Test
public void test409Error() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(409).headers(Collections.<String, Collection<String>>emptyMap())
.build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(1)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 5) Evaluation of Retry-After header

In contrast to the cases 4 and 4a, any response having a Retry-After header, which is a standard header defined in http protocol, the default Feign behavior is to honor this and trigger a retry at the date given.

@Test
public void test400ErrorWithRetryAfterHeader() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response
.builder()
.status(400)
.headers(
Collections.singletonMap(Util.RETRY_AFTER,
(Collection<String>) Collections.singletonList("1"))).build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(2)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

You can download my example on Github.

Please follow and like us:
0