Jax-RS + Codahale/dropwizard metrics + CDI + Prometheus

I used to use dropwizard built-in metrics annotations and graphite, but now I wanted to integrate these into my javaee project exposing prometheus metrics format. The main difference between graphite and prometheus is the push or pull mentality. So instead of pushing the metrics data to the sink, they are provided via http servlet and the prometheus server is scraping them from there.

There are two registry classes which we have to bring together. One is the com.codahale.metrics.MetricRegistry which holds all codahale metrics and the other one is the io.prometheus.client.CollectorRegistry which holds all metrics being published in the prometheus. So in our case, we will receive all metrics from our Jax-RS resource classes annotated with com.codahale.metrics.annotation.Timed or com.codahale.metrics.annotation.ExceptionMetered.

The prometheus library contains some default Jvm metrics (DefaultExports), but I could not use them because of some sun jdk classes which do not exist in OpenJDK for example. But it is good to add these in addition to the metrics coming from our annotated Jax-RS resource classes. So this is the class

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.dropwizard.DropwizardExports;
import io.prometheus.client.hotspot.ClassLoadingExports;
import io.prometheus.client.hotspot.GarbageCollectorExports;
import io.prometheus.client.hotspot.MemoryPoolsExports;
import io.prometheus.client.hotspot.ThreadExports;
import io.prometheus.client.hotspot.VersionInfoExports;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.context.Destroyed;
import javax.enterprise.context.Initialized;
import javax.enterprise.event.Observes;
import javax.inject.Inject;
import com.codahale.metrics.MetricRegistry;
/**
* A bean, which is starting the metrics reporter during startup and its is shutting down the
* reporter when application is destroyed.
*
*
*/
@ApplicationScoped
public class MetricsBean {
@Inject
private MetricRegistry registry;
public void init(@Observes @Initialized(ApplicationScoped.class) final Object init) {
// DefaultExports.initialize();
new MemoryPoolsExports().register();
new GarbageCollectorExports().register();
new ThreadExports().register();
new ClassLoadingExports().register();
new VersionInfoExports().register();
CollectorRegistry.defaultRegistry.register(new DropwizardExports(registry));
}
public void destroy(@Observes @Destroyed(ApplicationScoped.class) final Object init) {
CollectorRegistry.defaultRegistry.clear();
}
}
view raw MetricsBean.java hosted with ❤ by GitHub

and these are the dependencies for the project:

<dependency>
<groupId>io.astefanutti.metrics.cdi</groupId>
<artifactId>metrics-cdi</artifactId>
<version>1.3.6</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_servlet</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_dropwizard</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.0.23</version>
</dependency>
view raw pom.xml hosted with ❤ by GitHub

As you can see, by the help of CDI we are able to bind everything together during the startup phase of the application inside of the JEE container.

Then we add the  servlet to the web.xml like this and we are done:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" version="3.0">
<welcome-file-list>
<welcome-file>/index.html</welcome-file>
</welcome-file-list>
<!-- -->
<servlet>
<servlet-name>prometheusMetrics</servlet-name>
<servlet-class>io.prometheus.client.exporter.MetricsServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>prometheusMetrics</servlet-name>
<url-pattern>/prometheusMetrics/*</url-pattern>
</servlet-mapping>
</web-app>
view raw Web.xml hosted with ❤ by GitHub

In your prometheus server create a scrape config

scrape_configs:
- job_name: 'jaxrs'
metrics_path: /prometheusMetrics/
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['www.example.com:80']
labels:
group: 'jaxrs'
view raw prometheus.yml hosted with ❤ by GitHub

and you should be able to see you metrics in prometheus expression browser or Grafana.

Please follow and like us:
0

Exploring Feign – Retrying

Feign is a library, which makes it easier to implement a http client. Recently more and more people start writing http clients, because they are creating microservices which communicate with http protocol. So there are all sorts of libraries supporting this task like Jersey, Resteasy and others – and there is Feign.

Today I do not want to explain the basic functionality, this is all done on the Readme page itself. Today I want to get into the details of a feature, which becomes more and more important, because in modern distributed systems, you want to have resilient behaviour, which means that you want to design your service in the way, that it can handle unexpected situations without noticing on user’s site. For example an API you are calling is not reachable at the moment, the request times out or the requested resource is not yet available. To solve this issue, you need to apply a retry pattern, so that you increase the chance that the service request is successfull after the first, the second or the nth attempt.

What most developers don’t know, Feign has a default retryer built-in.

Now I show a few code examples, what you can expect from this feature. What I am showing are junit tests with a client mock, so that we are able to stub certain errors and verify, how many retries have been made.

Case 1) Success

no retry needed.

@Test
public void testSuccess() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(1)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 2) Destination never reachable.

In this case, we can see the Default Retryer working, which ends up doing 5 attempts, but finally the client invocation throws an exception.

@Test
public void testDefaultRetryerGivingUp() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenThrow(
new UnknownHostException());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(5)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 3) Configure maximal number of attempts

Taking the same error scenario from case 2, this example shows how to configure the retryer to stop trying after the 3rd attempt.

@Test
public void testRetryerAttempts() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenThrow(
new UnknownHostException());
final int maxAttempts = 3;
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.retryer(new Retryer.Default(1, 100, maxAttempts))
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(maxAttempts)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 4) trigger retrying by error code decoding

For some (restful) services, http status code 409 (conflict) is used to express a wrong state of the target resource, that might change after resubmitting the request. We simulate, that the first retry will lead to a successfull response.

@Test
public void testCustomRetryConfigByErrorDecoder() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(409).headers(Collections.<String, Collection<String>>emptyMap())
.build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
class RetryOn409ConflictStatus extends ErrorDecoder.Default {
@Override
public Exception decode(final String methodKey, final Response response) {
if (409 == response.status()) {
return new RetryableException("getting conflict and retry", null);
} else
return super.decode(methodKey, response);
}
}
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.errorDecoder(new RetryOn409ConflictStatus())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(2)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 4a) Behavior without error decoder

If no error decoder is configured, no retry is executed by Feign.

@Test
public void test409Error() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(409).headers(Collections.<String, Collection<String>>emptyMap())
.build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(1)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 5) Evaluation of Retry-After header

In contrast to the cases 4 and 4a, any response having a Retry-After header, which is a standard header defined in http protocol, the default Feign behavior is to honor this and trigger a retry at the date given.

@Test
public void test400ErrorWithRetryAfterHeader() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response
.builder()
.status(400)
.headers(
Collections.singletonMap(Util.RETRY_AFTER,
(Collection<String>) Collections.singletonList("1"))).build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(2)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

You can download my example on Github.

Please follow and like us:
0