Feign Outbound metrics 

With dropwizard microservices, you can easily add inbound metrics on your jax-rs http resource classes via annotations:

@Path("/example")
@Produces(MediaType.TEXT_PLAIN)
public class ExampleResource {
@GET
@Timed
@Metered
@ExceptionMetered
public String show() {
return "yay";
}
}
view raw ExampleResource.java hosted with ❤ by GitHub

The metrics can easily been reported to graphite database and visualized via Kibana.

WYIIWYG – What you instrument, is what you get!

On the other hand, a microservice often contains client libraries to access other services via http. Feign is client library, which provides a wrapper and simplifies the api to communicate to the target services.

In contrast to the inbound metrics from the example above, it is also desirable to monitor the outbound metrics of each of the targeted operations.

Looking at the third-party libraries of http://metrics.dropwizard.io/3.2.3/manual/third-party.html there is already something to retrieve metrics on http level. So in case you are using okhttp as http client implementation you can use https://github.com/raskasa/metrics-okhttp and you will receive information about request durations and connection pools.  Same holds good for Apache httpclient instrumentation.

okhttp example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrumenting feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
// instrumenting ok http
.client(new OkHttpClient(InstrumentedOkHttpClients.create(metricRegistry)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();
view raw OkHttpExample.java hosted with ❤ by GitHub

Metric output:

-- Gauges ----------------------------------------------------------------------
okhttp3.OkHttpClient.connection-pool-idle-count
value = 1
okhttp3.OkHttpClient.connection-pool-total-count
value = 1
-- Counters --------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-running
count = 0
-- Meters ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-completed
count = 1
mean rate = 0,84 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
okhttp3.OkHttpClient.network-requests-submitted
count = 1
mean rate = 0,83 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
-- Timers ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-duration
count = 1
mean rate = 0,84 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 215,41 milliseconds
max = 215,41 milliseconds
mean = 215,41 milliseconds
stddev = 0,00 milliseconds
median = 215,41 milliseconds
75% <= 215,41 milliseconds
95% <= 215,41 milliseconds
98% <= 215,41 milliseconds
99% <= 215,41 milliseconds
99.9% <= 215,41 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

httpclient example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrument feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry)).client(
// setting an instrumented httpclient
new ApacheHttpClient(InstrumentedHttpClients
.createDefault(metricRegistry, HttpClientMetricNameStrategies.HOST_AND_METHOD)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();

Metric output:

-- Gauges ----------------------------------------------------------------------
org.apache.http.conn.HttpClientConnectionManager.available-connections
value = 1
org.apache.http.conn.HttpClientConnectionManager.leased-connections
value = 0
org.apache.http.conn.HttpClientConnectionManager.max-connections
value = 20
org.apache.http.conn.HttpClientConnectionManager.pending-connections
value = 0
-- Meters ----------------------------------------------------------------------
-- Timers ----------------------------------------------------------------------
org.apache.http.client.HttpClient.api.github.com.get-requests
count = 1
mean rate = 4,19 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 174,59 milliseconds
max = 174,59 milliseconds
mean = 174,59 milliseconds
stddev = 0,00 milliseconds
median = 174,59 milliseconds
75% <= 174,59 milliseconds
95% <= 174,59 milliseconds
98% <= 174,59 milliseconds
99% <= 174,59 milliseconds
99.9% <= 174,59 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

As you can see, the provided metrics only provid information on http level, not really showing differences between different service endpoints. The only differentation is available on the httpclient metrics, which shows metrics based on host and http methods.

Closing the gap

What was missing in my eyes was a way to instrument metrics on the interface level, which is provided from the Feign builder. In my example below I am calling the github API on two different resource endpoints, contributors and repositorySearch. With the instrumentation on http, one is not able to see and monitor those one by one.

Therefore I created a library, which makes it possible to instrument metrics on method or interface level by using annotations like you do it in jersey resource classes.

Using this instrumentation you are able to retrieve metrics based on the interface and methods the client is calling. So for example when you start reporting via JMX, you are able to see the metrics in jconsole.

Usage of the library

To instrument the feign interfaces you basically have to do three things:

  1. add the maven dependency to the pom.xml of your project.
    <dependency>
      <groupId>com.github.mwiede</groupId>
      <artifactId>metrics-feign</artifactId>
      <version>1.0</version>
    </dependency>
  2. add FeignOutboundMetricsDecorator as invocationHandlerFactory in Feign.builder
  3. add the metric annotations @Timed, @Metered and @ExceptionMetered to the interface you are using with feign.
@Timed
@Metered
@ExceptionMetered
interface GitHub {
@RequestLine("GET /repos/{owner}/{repo}/contributors")
List<Contributor> contributors(@Param("owner") String owner, @Param("repo") String repo);
}
static class Contributor {
String login;
int contributions;
}
public static void main(String... args) {
MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
// Fetch and print a list of the contributors to this library.
List<Contributor> contributors = github.contributors("mwiede", "metrics-feign");
for (Contributor contributor : contributors) {
System.out.println(contributor.login + " (" + contributor.contributions + ")");
}
reporter.report();
}
view raw Example.java hosted with ❤ by GitHub

The library is available from maven central and the source is hosted at github, so please checkout https://github.com/mwiede/metrics-feign

Please follow and like us:
0

Jax-RS + Codahale/dropwizard metrics + CDI + Prometheus

I used to use dropwizard built-in metrics annotations and graphite, but now I wanted to integrate these into my javaee project exposing prometheus metrics format. The main difference between graphite and prometheus is the push or pull mentality. So instead of pushing the metrics data to the sink, they are provided via http servlet and the prometheus server is scraping them from there.

There are two registry classes which we have to bring together. One is the com.codahale.metrics.MetricRegistry which holds all codahale metrics and the other one is the io.prometheus.client.CollectorRegistry which holds all metrics being published in the prometheus. So in our case, we will receive all metrics from our Jax-RS resource classes annotated with com.codahale.metrics.annotation.Timed or com.codahale.metrics.annotation.ExceptionMetered.

The prometheus library contains some default Jvm metrics (DefaultExports), but I could not use them because of some sun jdk classes which do not exist in OpenJDK for example. But it is good to add these in addition to the metrics coming from our annotated Jax-RS resource classes. So this is the class

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.dropwizard.DropwizardExports;
import io.prometheus.client.hotspot.ClassLoadingExports;
import io.prometheus.client.hotspot.GarbageCollectorExports;
import io.prometheus.client.hotspot.MemoryPoolsExports;
import io.prometheus.client.hotspot.ThreadExports;
import io.prometheus.client.hotspot.VersionInfoExports;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.context.Destroyed;
import javax.enterprise.context.Initialized;
import javax.enterprise.event.Observes;
import javax.inject.Inject;
import com.codahale.metrics.MetricRegistry;
/**
* A bean, which is starting the metrics reporter during startup and its is shutting down the
* reporter when application is destroyed.
*
*
*/
@ApplicationScoped
public class MetricsBean {
@Inject
private MetricRegistry registry;
public void init(@Observes @Initialized(ApplicationScoped.class) final Object init) {
// DefaultExports.initialize();
new MemoryPoolsExports().register();
new GarbageCollectorExports().register();
new ThreadExports().register();
new ClassLoadingExports().register();
new VersionInfoExports().register();
CollectorRegistry.defaultRegistry.register(new DropwizardExports(registry));
}
public void destroy(@Observes @Destroyed(ApplicationScoped.class) final Object init) {
CollectorRegistry.defaultRegistry.clear();
}
}
view raw MetricsBean.java hosted with ❤ by GitHub

and these are the dependencies for the project:

<dependency>
<groupId>io.astefanutti.metrics.cdi</groupId>
<artifactId>metrics-cdi</artifactId>
<version>1.3.6</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_servlet</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_dropwizard</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.0.23</version>
</dependency>
view raw pom.xml hosted with ❤ by GitHub

As you can see, by the help of CDI we are able to bind everything together during the startup phase of the application inside of the JEE container.

Then we add the  servlet to the web.xml like this and we are done:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" version="3.0">
<welcome-file-list>
<welcome-file>/index.html</welcome-file>
</welcome-file-list>
<!-- -->
<servlet>
<servlet-name>prometheusMetrics</servlet-name>
<servlet-class>io.prometheus.client.exporter.MetricsServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>prometheusMetrics</servlet-name>
<url-pattern>/prometheusMetrics/*</url-pattern>
</servlet-mapping>
</web-app>
view raw Web.xml hosted with ❤ by GitHub

In your prometheus server create a scrape config

scrape_configs:
- job_name: 'jaxrs'
metrics_path: /prometheusMetrics/
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['www.example.com:80']
labels:
group: 'jaxrs'
view raw prometheus.yml hosted with ❤ by GitHub

and you should be able to see you metrics in prometheus expression browser or Grafana.

Please follow and like us:
0