Free tier hunt – howto combine heroku and openshift

Background

Recently, RedHat shutdown its openshift platform v2. Now only openshift 3 is available and it is based on kubernetes and docker.

So why do I care?

I had some JEE projects running on openshift 2 and now I was forced to migrate them to the new infrastructure. Basically my applications consist of a tomcat or wildfly server running next to a mysql database. This setup was pretty easy and I could run this at no cost as long as I could tolerate, that the cartridges would be shutdown if they idle longer then 24 hours (meaning no http request was coming to the server during this time). But ok…

So now I had to migrate my projects following the migration guide provided by Redhat. But in parallel I was interested whether I have other options or if there are any other Paas providers giving out a free tier for personal small projects. And yes, there are plenty of them, but as I found, all have their limitations.

AWS is for 1 year, then pay. Google cloud … cant remember what was holding me back…Oracle cloud gives a certain amount of money, but evaluation phase is 3 month max. Microsoft…really?

Going with Heroku, but…

Then I found heroku offering “free dyno”, which also idles after 30min inactivity, but I wanted to give it a try. Later I found, that if you want to use a database with heroku, the limitations on the “free” database are like 5MB or 100 rows, so even if I have only a few playaround datasets, that was too small.

Then I had the idea of connecting two Paas providers. One giving me the application tier for free, the other one is giving me the database for free.

I ended up looking on how to connect to a database running on openshift from outside of the docker environment. What I found is the same way an admin would connect to it, via a tunnel and/or port forwarding.

Oh my god, the latency between application and database will be huge!

Yes, that is possible, but as I do not want to propose this setup for an enterprise application running in production, I am fine.

So here is my solution on how to connect from an application running in a heroku dyno to a mysql database running in openshift.

Heroku provides a mechanism which allows you to pack anything into your dyno, which you need to run your application. So if you need java, then the buildpack “heroku/java” is for you. If you need node, there is a buildpack for node and so on. The nice thing about the buildbacks is, that you can also create them on your own, using the buildback API

A buildpack can contain a shell script (profile.d), which is executed during startup of the container. The perfect way to create a tunnel and provide access to my remote database. You can find the buildpack at https://github.com/mwiede/heroku-buildpack-oc

So here is how you can create a heroku application having access to a remote database:

  1. install Heroku CLI
  2. create an app
  3. add my buildpack
    heroku buildpacks:add https://github.com/mwiede/heroku-buildpack-oc
  4. configure environment variables
    $ heroku config:set OC_LOGIN_ENDPOINT=https://api.starter-ca-central-1.openshift.com 
    $ heroku config:set OC_LOGIN_TOKEN=askdjalskdj 
    $ heroku config:set OC_POD_NAME=mysql-1-weuoi 
    $ heroku config:set OC_LOCAL_PORT=3306 
    $ heroku config:set OC_REMOTE_PORT=3306
  5. deploy the app
  6. look for the logs, whether connection works properly.

Advanced usage

The profile.d script contains a loop so whenever the connection of the tunnel shuts down, it tries to open it up again.

From the perspective of openshift, the database runs in a so called pods and unfortunely it’s name can change.

I tried to make this as robust as possible, so the name in OC_POD_NAME should only contain a prefix of how the pod is named, for instance “mysql” is enough to detect the right one.

Feign Outbound metrics 

With dropwizard microservices, you can easily add inbound metrics on your jax-rs http resource classes via annotations:

@Path("/example")
@Produces(MediaType.TEXT_PLAIN)
public class ExampleResource {
@GET
@Timed
@Metered
@ExceptionMetered
public String show() {
return "yay";
}
}
view raw ExampleResource.java hosted with ❤ by GitHub

The metrics can easily been reported to graphite database and visualized via Kibana.

WYIIWYG – What you instrument, is what you get!

On the other hand, a microservice often contains client libraries to access other services via http. Feign is client library, which provides a wrapper and simplifies the api to communicate to the target services.

In contrast to the inbound metrics from the example above, it is also desirable to monitor the outbound metrics of each of the targeted operations.

Looking at the third-party libraries of http://metrics.dropwizard.io/3.2.3/manual/third-party.html there is already something to retrieve metrics on http level. So in case you are using okhttp as http client implementation you can use https://github.com/raskasa/metrics-okhttp and you will receive information about request durations and connection pools.  Same holds good for Apache httpclient instrumentation.

okhttp example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrumenting feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
// instrumenting ok http
.client(new OkHttpClient(InstrumentedOkHttpClients.create(metricRegistry)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();
view raw OkHttpExample.java hosted with ❤ by GitHub

Metric output:

-- Gauges ----------------------------------------------------------------------
okhttp3.OkHttpClient.connection-pool-idle-count
value = 1
okhttp3.OkHttpClient.connection-pool-total-count
value = 1
-- Counters --------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-running
count = 0
-- Meters ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-completed
count = 1
mean rate = 0,84 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
okhttp3.OkHttpClient.network-requests-submitted
count = 1
mean rate = 0,83 events/second
1-minute rate = 0,00 events/second
5-minute rate = 0,00 events/second
15-minute rate = 0,00 events/second
-- Timers ----------------------------------------------------------------------
okhttp3.OkHttpClient.network-requests-duration
count = 1
mean rate = 0,84 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 215,41 milliseconds
max = 215,41 milliseconds
mean = 215,41 milliseconds
stddev = 0,00 milliseconds
median = 215,41 milliseconds
75% <= 215,41 milliseconds
95% <= 215,41 milliseconds
98% <= 215,41 milliseconds
99% <= 215,41 milliseconds
99.9% <= 215,41 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

httpclient example

MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
// instrument feign
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry)).client(
// setting an instrumented httpclient
new ApacheHttpClient(InstrumentedHttpClients
.createDefault(metricRegistry, HttpClientMetricNameStrategies.HOST_AND_METHOD)))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
execute...
reporter.report();

Metric output:

-- Gauges ----------------------------------------------------------------------
org.apache.http.conn.HttpClientConnectionManager.available-connections
value = 1
org.apache.http.conn.HttpClientConnectionManager.leased-connections
value = 0
org.apache.http.conn.HttpClientConnectionManager.max-connections
value = 20
org.apache.http.conn.HttpClientConnectionManager.pending-connections
value = 0
-- Meters ----------------------------------------------------------------------
-- Timers ----------------------------------------------------------------------
org.apache.http.client.HttpClient.api.github.com.get-requests
count = 1
mean rate = 4,19 calls/second
1-minute rate = 0,00 calls/second
5-minute rate = 0,00 calls/second
15-minute rate = 0,00 calls/second
min = 174,59 milliseconds
max = 174,59 milliseconds
mean = 174,59 milliseconds
stddev = 0,00 milliseconds
median = 174,59 milliseconds
75% <= 174,59 milliseconds
95% <= 174,59 milliseconds
98% <= 174,59 milliseconds
99% <= 174,59 milliseconds
99.9% <= 174,59 milliseconds
view raw gistfile1.txt hosted with ❤ by GitHub

As you can see, the provided metrics only provid information on http level, not really showing differences between different service endpoints. The only differentation is available on the httpclient metrics, which shows metrics based on host and http methods.

Closing the gap

What was missing in my eyes was a way to instrument metrics on the interface level, which is provided from the Feign builder. In my example below I am calling the github API on two different resource endpoints, contributors and repositorySearch. With the instrumentation on http, one is not able to see and monitor those one by one.

Therefore I created a library, which makes it possible to instrument metrics on method or interface level by using annotations like you do it in jersey resource classes.

Using this instrumentation you are able to retrieve metrics based on the interface and methods the client is calling. So for example when you start reporting via JMX, you are able to see the metrics in jconsole.

Usage of the library

To instrument the feign interfaces you basically have to do three things:

  1. add the maven dependency to the pom.xml of your project.
    <dependency>
      <groupId>com.github.mwiede</groupId>
      <artifactId>metrics-feign</artifactId>
      <version>1.0</version>
    </dependency>
  2. add FeignOutboundMetricsDecorator as invocationHandlerFactory in Feign.builder
  3. add the metric annotations @Timed, @Metered and @ExceptionMetered to the interface you are using with feign.
@Timed
@Metered
@ExceptionMetered
interface GitHub {
@RequestLine("GET /repos/{owner}/{repo}/contributors")
List<Contributor> contributors(@Param("owner") String owner, @Param("repo") String repo);
}
static class Contributor {
String login;
int contributions;
}
public static void main(String... args) {
MetricRegistry metricRegistry = new MetricRegistry();
final ConsoleReporter reporter = ConsoleReporter.forRegistry(metricRegistry).convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS).build();
GitHub github = Feign.builder().invocationHandlerFactory(
new FeignOutboundMetricsDecorator(new InvocationHandlerFactory.Default(), metricRegistry))
.decoder(new GsonDecoder()).target(GitHub.class, "https://api.github.com");
// Fetch and print a list of the contributors to this library.
List<Contributor> contributors = github.contributors("mwiede", "metrics-feign");
for (Contributor contributor : contributors) {
System.out.println(contributor.login + " (" + contributor.contributions + ")");
}
reporter.report();
}
view raw Example.java hosted with ❤ by GitHub

The library is available from maven central and the source is hosted at github, so please checkout https://github.com/mwiede/metrics-feign

Jax-RS + Codahale/dropwizard metrics + CDI + Prometheus

I used to use dropwizard built-in metrics annotations and graphite, but now I wanted to integrate these into my javaee project exposing prometheus metrics format. The main difference between graphite and prometheus is the push or pull mentality. So instead of pushing the metrics data to the sink, they are provided via http servlet and the prometheus server is scraping them from there.

There are two registry classes which we have to bring together. One is the com.codahale.metrics.MetricRegistry which holds all codahale metrics and the other one is the io.prometheus.client.CollectorRegistry which holds all metrics being published in the prometheus. So in our case, we will receive all metrics from our Jax-RS resource classes annotated with com.codahale.metrics.annotation.Timed or com.codahale.metrics.annotation.ExceptionMetered.

The prometheus library contains some default Jvm metrics (DefaultExports), but I could not use them because of some sun jdk classes which do not exist in OpenJDK for example. But it is good to add these in addition to the metrics coming from our annotated Jax-RS resource classes. So this is the class

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.dropwizard.DropwizardExports;
import io.prometheus.client.hotspot.ClassLoadingExports;
import io.prometheus.client.hotspot.GarbageCollectorExports;
import io.prometheus.client.hotspot.MemoryPoolsExports;
import io.prometheus.client.hotspot.ThreadExports;
import io.prometheus.client.hotspot.VersionInfoExports;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.context.Destroyed;
import javax.enterprise.context.Initialized;
import javax.enterprise.event.Observes;
import javax.inject.Inject;
import com.codahale.metrics.MetricRegistry;
/**
* A bean, which is starting the metrics reporter during startup and its is shutting down the
* reporter when application is destroyed.
*
*
*/
@ApplicationScoped
public class MetricsBean {
@Inject
private MetricRegistry registry;
public void init(@Observes @Initialized(ApplicationScoped.class) final Object init) {
// DefaultExports.initialize();
new MemoryPoolsExports().register();
new GarbageCollectorExports().register();
new ThreadExports().register();
new ClassLoadingExports().register();
new VersionInfoExports().register();
CollectorRegistry.defaultRegistry.register(new DropwizardExports(registry));
}
public void destroy(@Observes @Destroyed(ApplicationScoped.class) final Object init) {
CollectorRegistry.defaultRegistry.clear();
}
}
view raw MetricsBean.java hosted with ❤ by GitHub

and these are the dependencies for the project:

<dependency>
<groupId>io.astefanutti.metrics.cdi</groupId>
<artifactId>metrics-cdi</artifactId>
<version>1.3.6</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_servlet</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_dropwizard</artifactId>
<version>0.0.23</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.0.23</version>
</dependency>
view raw pom.xml hosted with ❤ by GitHub

As you can see, by the help of CDI we are able to bind everything together during the startup phase of the application inside of the JEE container.

Then we add the  servlet to the web.xml like this and we are done:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" version="3.0">
<welcome-file-list>
<welcome-file>/index.html</welcome-file>
</welcome-file-list>
<!-- -->
<servlet>
<servlet-name>prometheusMetrics</servlet-name>
<servlet-class>io.prometheus.client.exporter.MetricsServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>prometheusMetrics</servlet-name>
<url-pattern>/prometheusMetrics/*</url-pattern>
</servlet-mapping>
</web-app>
view raw Web.xml hosted with ❤ by GitHub

In your prometheus server create a scrape config

scrape_configs:
- job_name: 'jaxrs'
metrics_path: /prometheusMetrics/
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
static_configs:
- targets: ['www.example.com:80']
labels:
group: 'jaxrs'
view raw prometheus.yml hosted with ❤ by GitHub

and you should be able to see you metrics in prometheus expression browser or Grafana.

Exploring Feign – Retrying

Feign is a library, which makes it easier to implement a http client. Recently more and more people start writing http clients, because they are creating microservices which communicate with http protocol. So there are all sorts of libraries supporting this task like Jersey, Resteasy and others – and there is Feign.

Today I do not want to explain the basic functionality, this is all done on the Readme page itself. Today I want to get into the details of a feature, which becomes more and more important, because in modern distributed systems, you want to have resilient behaviour, which means that you want to design your service in the way, that it can handle unexpected situations without noticing on user’s site. For example an API you are calling is not reachable at the moment, the request times out or the requested resource is not yet available. To solve this issue, you need to apply a retry pattern, so that you increase the chance that the service request is successfull after the first, the second or the nth attempt.

What most developers don’t know, Feign has a default retryer built-in.

Now I show a few code examples, what you can expect from this feature. What I am showing are junit tests with a client mock, so that we are able to stub certain errors and verify, how many retries have been made.

Case 1) Success

no retry needed.

@Test
public void testSuccess() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(1)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 2) Destination never reachable.

In this case, we can see the Default Retryer working, which ends up doing 5 attempts, but finally the client invocation throws an exception.

@Test
public void testDefaultRetryerGivingUp() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenThrow(
new UnknownHostException());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(5)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 3) Configure maximal number of attempts

Taking the same error scenario from case 2, this example shows how to configure the retryer to stop trying after the 3rd attempt.

@Test
public void testRetryerAttempts() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenThrow(
new UnknownHostException());
final int maxAttempts = 3;
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.retryer(new Retryer.Default(1, 100, maxAttempts))
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(maxAttempts)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 4) trigger retrying by error code decoding

For some (restful) services, http status code 409 (conflict) is used to express a wrong state of the target resource, that might change after resubmitting the request. We simulate, that the first retry will lead to a successfull response.

@Test
public void testCustomRetryConfigByErrorDecoder() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(409).headers(Collections.<String, Collection<String>>emptyMap())
.build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
class RetryOn409ConflictStatus extends ErrorDecoder.Default {
@Override
public Exception decode(final String methodKey, final Response response) {
if (409 == response.status()) {
return new RetryableException("getting conflict and retry", null);
} else
return super.decode(methodKey, response);
}
}
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.errorDecoder(new RetryOn409ConflictStatus())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(2)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 4a) Behavior without error decoder

If no error decoder is configured, no retry is executed by Feign.

@Test
public void test409Error() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response.builder().status(409).headers(Collections.<String, Collection<String>>emptyMap())
.build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
try {
github.contributors("OpenFeign", "feign");
fail("not failing");
} catch (final Exception e) {
} finally {
verify(clientMock, times(1)).execute(any(Request.class), any(Options.class));
}
}
view raw FeignTest.java hosted with ❤ by GitHub

Case 5) Evaluation of Retry-After header

In contrast to the cases 4 and 4a, any response having a Retry-After header, which is a standard header defined in http protocol, the default Feign behavior is to honor this and trigger a retry at the date given.

@Test
public void test400ErrorWithRetryAfterHeader() throws IOException {
when(clientMock.execute(any(Request.class), any(Options.class))).thenReturn(
Response
.builder()
.status(400)
.headers(
Collections.singletonMap(Util.RETRY_AFTER,
(Collection<String>) Collections.singletonList("1"))).build(),
Response.builder().status(200).headers(Collections.<String, Collection<String>>emptyMap())
.build());
final GitHub github =
Feign.builder().client(clientMock).decoder(new GsonDecoder())
.target(GitHub.class, "https://api.github.com");
github.contributors("OpenFeign", "feign");
verify(clientMock, times(2)).execute(any(Request.class), any(Options.class));
}
view raw FeignTest.java hosted with ❤ by GitHub

You can download my example on Github.