Monday, January 4, 2016

Avoid Vertical Limits in Microservice Architectures

The microservice architecture allows us to partition an application into tiny sub-applications, which are easy to maintain and to deploy. This pattern is already widely adopted to implement backend systems. But the frontend is usually still one large application, at least when it comes to deployment. This article describes some thoughts on how to address this problem.
The microservice architecture is en vogue, everybody seems to know all about it, and feels like having to spread the truth. Including me ;-) But honestly, we are just about to learn how to cope with this kind of architecture. I guess we are like a kid that just managed to make some first steps, when we suddenly try to start running… that’s usually the moment when you fall over your own feet. Microservices are no free lunch, it definitely has its price (that’s the part we have already learned). For developers it feels uncomfortable, since instead of developing one application, they have to deal with dozens or hundreds. Usually the microservice-believers are the people who drive and maintain an application over a long period of time; those poor chaps that know the pain of change. And that is were their trust – or better: hope – in microservices comes from.

So what is this article about? I’m currently working in a web project for a customer, where we are using Spring Boot to create a microservice-based backend. We heavily use the Job DSL to drive our CI pipeline, and also all kind of configuration and tooling. And the front end is a single page application built with AngularJS. So far, so good, but there is a catch. One goal of this microservice idea is to have independent development teams, that drive features within their own development cycle. In the backend this is feasible due to the microservice architecture. But the front end is still one large application. Angular allows to partition the application into modules, but after all it is assembled to one application, so this is a single point of integration. Let me explain that point using a practical example. It is a stupid-simple web shop, stripped down to the bones. It is just a catalog of items, which we can put into a shopping cart:



I have prepared the example as a Spring Boot based application in the backed, with some AngularJS in the frontend. You will find the code on GitHub, the version we start with, is on the branch one-ui. All projects have maven and gradle builds, the readme will provide you enough information to run the application.
Just some note before we start: I’m not a web developer. In the last 20 years I have build – beside backend systems – various UIs in Motif, QT, Java AWT, Swing and SWT. But I have never done web frontends, there has never been the chance or need for that. So please be merciful if my angular and javascript looks rather… er, surprising to you ;-)
The architecture of our stupid simple web shop looks like this:

initial2

The browser accesses the application via a router, which limits access to the backend, helps with the CORS problem, and may also perform some security checks. The web-server hosts the AngularJS application and the assets. The AngularJS application itself communicates with the API-Gateway, which encapsulates the microservices in the backend, and provides services and data optimized for the UI. The API-Gateway pattern is often used in order to not to bloat the microservices with functionality where it not belongs to. We will see its advantages later on. The gateway talks to the backend microservice, which then perform their dedicated functionality using their own isolated databases (the databases has been omitted for simplicity in the example code).

So far nothing new, what’s the point? Well, what is that microservice idea all about? Ease of change. Make small changes easy to apply and deploy. Minimize the risk of change by having to deploy only a small part of the application instead of a large one-size-fits-all application. In the backend this is now widely adopted. The microservices are tiny, and easy to maintain. As long as their interfaces to other services remain (compatible), we may replace them with a new version in a running production environment, without affecting the others. And – if done right – without downtime. But what about the frontend? The frontend is mostly still one large application. If we make a small change to the UI, we have to build and redeploy the whole thing. Even worse, since it is one application, bug fixes and features are often developed in parallel by large teams, which makes it difficult to release minor changes separately. At last, that’s what the microservice story is all about: Dividing your application into small sub-applications which can be developed and deployed separately, giving each its own lifecycle. And consequently, this pattern should be applied to the complete application, from database via services to the frontend. But currently, our microservice architecture thinking ends at the UI, and that’s what I call a vertical limit.

We investigated how other people are dealing with this situation, and – no wonder – there were a lot of complaints about the same problem. But surprisingly, most advices to address this were to use the API-Gateway… er, but this does not solve the problem?!? So let’s think about it: we are dividing our backend into fine grained microservices in order to cope with changes. Consequently, we should exactly the same in UI. We could partition our UI into components, where features make up the logical boundaries. Let’s take our silly vertical shop example to exercise this. We separate this UI at least into two logical components: the catalog showing all available items, and the shopping cart. Angular provides a mechanism to build components: directives! The cart and the catalog are already encapsulated in angular directives (what a coincidence ;-). So what if we put those parts each on its own web-server? Let’s see how our architecture would look like:

multipleUI_oneGateway2

Hmm, obviously the API gateway is still a common denominator. The gateway’s job is to abstract from the backend microservice and to serve the UI in the best suited way. So consequently, we should divide the gateway also. But since it so closely related to the UI, we are packaging ‘em both into one service in our example. Let’s do so, you will find the code for this version on the branch ui-components. Now the architecture looks like this:

 componentsDashed

Hey wait! I’ve had look at your UI code. It’s not just the directives, there are also angular services. And the add-button in the catalog directive is effectively calling addItem() on the cart-service. That’s quite true. But is that a problem? In our microservice backend, services are calling other services also. This is not a problem, as long as the service’ API doesn’t change, resp. remains compatible. The same holds on the javascript side. As long as our API doesn’t change (remains compatible), new service rolllouts and internal changes are not a problem. But we have to design these interfaces between components wisely: (angular) services may be used by other components, so we have to be careful with changes. Another way of communication between components is broadcasting events. Actually the cart component is firing an event every time the cart changes, in order to give other components a chance to react on that. So we have to be cautious with changes to events also. New resp. more data is not a problem, removing or renaming existing properties is. So we simply put all functionality dealing with the cart on the cart web-server, and the catalog stuff on the catalog web-server… including their dedicated directives, services, assets a whatsoever. Our communication looks like this:

communication

Big words and bloated architecture. It ain’t worth it! You think so? Let’s make a small change to our application and examine its impact: Our customer does not like our shopping cart: “It just shows each items article number, and its amount. That’s not quite helpful. The user needs the article’s name…its price…and the sum of all items in the cart.”


Well, yeah, she’s right. So let’s fix that. But the cart microservice does not provide more information. Article names and prices are the catalog service’ domain. So the cart service could call the catalog service and enrich its data? But that’s not the cart’s domain, it should not have to know about that. No, providing the data in a way appropriate for the UI is the job of the API gateway (finally it pays ;-). So instead of delegating all calls to the cart service, the cart’s API gateway merges the data from the cart with the catalog data to provide cart items with article names and prices:

cartGateway

Now we simply adapt the UI code of the cart in order to use and visualize this additional data. You will find the source code of this final version on the master. All changes we made are isolated in the cart UI resp. its API gateway, so we only have to redeploy this single server. Let’s do so. And if we now reload our application, tada:


So, we made a change to the shopping cart without affecting the remaining application. Without redeploying the complete UI. All the advantages we take the microservice-burden on our shoulders for, now pays in the front end also. Just treat your front end as a union of distinct features. In agile development it is all about features resp. stories, and now we we are able develop and release features in its own lifecycle. Hey wait, the index.html still assembles the cart and the catalog to a single page. So there is still a single point of integration! Yep, you’re right. This is still one UI, but we have componentized it. Once a component is referenced in a page, that’s it. Any further changes to internals of that component does not affect the hosting page. Our microservices are also referenced by other parties, so this is quite similar.

The point is to avoid vertical limits in this kind of architecture. We have to separate our application into small sub-applications in all layers, namely persistence, service backend and frontend. In the backend this cut is often domain-driven, and in the front end we can use features to partition our application. For sure there are better ways and technologies to implement that, but I guess this is a step into the right direction.
We learn from failure, not from success!
Bram Stoker

1) This was poor attempt to avoid the word monolith, which is – at the time of writing – mistakenly flavored with a bad smack.

Wednesday, December 9, 2015

Java and JSON ain't friends

Most application frameworks provide some REST support, which is – depending on the language you are using – either dirt cheap, or quite complex. In the Java world part of these frameworks is some kind of mapping from JSON to Java and vice versa, most of them using the Jackson mapping framework. It feels quite natural: you model your domain objects directly in Java. If you don’t have any constraints, the JSON might even follow your model. If the JSON is predefined (as part of the API), you can either design your Java classes so they fit the generated JSON, or provide Jackson with some mapping hints to do so. But you know all that, right? So what am I talking about here?

The point is: domain models may vary in size from a few properties to x-nested sky high giants… and so are the resulting Java model classes. What makes things even worse, is that domain models change over time. Often you don’t know all the requirements front of, also requirements change over time. So domain models are subject to change. All that is still not a big problem, as long as you don’t need to communicate those domain models to others. If other parties are involved, they have adapt to the changes. Let’s take the following JSON describing a person:
{
 "id":"32740748234",
 "firstName":"Herbert",
 "lastName":"Birdsfoot",
}
We can write a simple Java class that will de-/serialize from/to this JSON:
public class Person {

 private String id;
 private String firstName;
 private String lastName;

 public String getId() {
  return id;
 }

 public void setId(String id) {
  this.id = id;
 }

 public String getFirstName() {
  return firstName;
 }

 public void setFirstName(String firstName) {
  this.firstName = firstName;
 }

 public String getLastName() {
  return lastName;
 }

 public void setLastName(String lastName) {
  this.lastName = lastName;
 }
}
We can write a simple test to verify proper mapping to JSON:
public class PersonTest {

 private final ObjectMapper mapper = new ObjectMapper();
 private String personJsonString;

 @Before
 public void setUp() throws Exception {
  personJsonString = IOUtils.toString(this.getClass()
    .getResourceAsStream("person.json"));
 }

 @Test
 public void testMapJsonToPerson() throws Exception {
  final Person person = mapper.readValue(personJsonString, Person.class);
  checkPerson(person);
 }

 protected void checkPerson(final Person person) {
  assertNotNull(person);
  assertEquals("32740748234", person.getId());
  assertEquals("Herbert", person.getFirstName());
  assertEquals("Birdsfoot", person.getLastName());
 }
If we run the test, everything is nicely green:

PersonTestGreen

That was easy. But now we have new requirements: we need to extend our person with some address data:
{
 "id":"32740748234",
 "firstName":"Herbert",
 "lastName":"Birdsfoot",
 "address":{
  "street":"Sesamestreet",
  "number":"123",
  "zip":"10123",
  "city":"New York",
  "country":"USA"
 }
}
If we run our test against that JSON we will get red

PersonTestRed

If you have a look at the StackTrace, Jackson is complaining about unknown properties
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: 
Unrecognized field "address" (class rst.sample.Person), not marked as 
ignorable (3 known properties: "lastName", "id", "firstName"])
 at [Source: {
 "id":"32740748234",
 "firstName":"Herbert",
 "lastName":"Birdsfoot",
 "address":{
  "street":"Sesamestreet",
  "number":"123",
  "zip":"10123",
  "city":"New York",
  "country":"USA"
 }
}; line: 5, column: 13] (through reference chain: rst.sample.Person["address"])
 at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.
    from(UnrecognizedPropertyException.java:51)
    ...
Now we have two choices. We can extend our Java model by the missing properties. That’s quite easy. And if the producer of that JSON is using Java either, we might just copy their model, can’t we? Well….you may. I have seen this in a current microservice project. People have been passing model classes around on every change, often asking for some common domain model lib. Don’t do that. Never ever. First of all ask yourself: are you really interested in the data introduced by the change, or are you only adapting to satisfy the serialization? If you need all the data, you have to adapt. If you don’t need the data, don’t adapt, there are mechanisms to prevent serialization complaints. There is e.g. an easy way to tell Jackson to ignore superfluous properties:
@JsonIgnoreProperties(ignoreUnknown=true)
public class Person {
...
If you run your test again, everything is green again. In fact, application frameworks utilizing Jackson like e.g. Spring are configuring Jackson to always ignore unknown properties, since this makes sense in most situations. But be aware of it since it is not explicit to you.

PersonTestGreen

So what about that anger on common domain model libs? I have seen this quite often in customer projects: developers start to create some common domain model libs, so everyone out in the project may use it. The point is: over time people are extending the domain models with their specific needs… whether anyone else needs it or not. And this leads to models bloated with any kind of niche domain functionality depending on a hell lot of other bloated domain objects. Don’t do it. Duplicate the models and let every party evolve its own view to the domain.
You have to choose where to pay the price of complexity. Because DDD is about reducing complexity in the software, the outcome is that you pay a price with respect to maintaining duplicate models and possibly duplicate data.
Eric EvansDomain Driven Design
But – as always – things might not be that easy. What if you do not care about that superfluous data, but you have to pass it to another party. Hey, we give them the person ID, so they can retrieve all the data they want on demand. If this is the case, you are safe. But sometimes you don’t want to pass data by handing a (foreign) key, which actually means: by reference. Depending on the business case you may have to pass a snapshot of the current data, means: by value. So what about that case, do I have to copy the model classes again in order to specify all possible properties?!? Damn, in dynamic languages like Javascript or Clojure the JSON is generically “mapped” to the object, and I do not have to care for any class schema. Couldn’t we do that in Java also, at least in some – well, less comfortable – way? If you search online for solutions on that problem, you will often find this one:
public class Person {
 
 private String id;
 private String firstName;
 private String lastName;
 private final Map<String, Object> map = Maps.newLinkedHashMap();
...
 @JsonAnySetter
 public void add(final String key, final Object>  value) {
  map.put(key, value);
 }

 @JsonAnyGetter
 public Map<String, Object> getMap() {
  return map;
 }
}
Let’s give this solution a try. We will extend our test in order to check if the JSON output created by serialization equals the original input:
public class PersonTest {
...
 @Test
 public void testMapJsonToPersonToJson() throws Exception {
  final Person person = mapper.readValue(personJsonString, Person.class);
  final String newJson = mapper.writeValueAsString(person);
  JSONAssert.assertEquals(personJsonString, newJson, true);
 }
}
Let it run and, tada:

PersonTestRed2

… it fails, er?!? Yep, the solution described above works for simple properties, but not for nested ones. So we got to do better. Instead of Object, use Jackson’s JsonNode:
public class Person {
...
 private final Map<String, JsonNode> map = Maps.newLinkedHashMap();
...
 @JsonAnySetter
 public void add(final String key, final JsonNode value) {
  map.put(key, value);
 }

 @JsonAnyGetter
 public Map<String, JsonNode> getMap() {
  return map;
 }
}
Run the test again, and <drumroll>:

PersonTestGreen

Phew, green :-)

So this is the solution that solves our problem. It is both compatible to changes – as long as the properties we are actually using are not subject to change – and reconstructs the original JSON as we received it. Work done.

That’s it for today
Ralf
The only way to have a friend is to be one.
Ralph Waldo Emerson

Monday, September 14, 2015

Use MTOM to Efficiently Transmit Binary Content in SOAP

Currently JSON-based REST services are en vogue, but when it comes to integrating enterprise services, SOAP is still widely used. In a recent project I had to deal with binary content sent to a third party SOAP-Service. Thanks to great tooling, calling a SOAP service is not a big deal. But the binary data varied in size from a few kB to many MB, and this brought up some issues in transmission size and memory usage. That's where MTOM comes to the rescue, a standard for efficiently tranmitting binary data in a SOAP request. This article published on DZone describes, what MTOM can do for you by converting a tiny spring client-server project from default SOAP to MTOM.

Wednesday, June 10, 2015

Spring Integration Tests with MongoDB rulez


While unit testing is always preferable, integration tests are a good and necessary supplement to either perform end to end tests, or tests involving (third party) backends. Databases are such a candidate where integrations might make sense: usually we encapsulate persistence with some kind of repository service layer, which we can mock in tests running against the repository. But when it comes to testing the repository itself, integration tests are quite useful. Spring integration tests allow you to test functionality against a running Spring application, and thereby allows to test against a running database instance. But as you do in unit tests, you have to perform a proper set up of test data, and clean up the database afterwards. That's what this article published on DZone is about: proper database set- and clean-up in Spring integration tests with MongoDB.

Monday, April 13, 2015

Job DSL Part III

The previous part of this little series on the Job DSL gave you some examples on maintenance, automating the job creation itself and creating views. This last installment will complete the little round trip through the Job DSL with some hints on documentation, tooling and pitfalls.

Documentation

If you search the internet for the Job DSL one of the first hits will be the corresponding wiki. This is the most valuable source of information. It is well structured and maintained, so new features and missing pieces are filled in regularly. If you are looking for any details on jobs, the job reference is your target. If you like to generate a view, there is a corresponding view reference.

job-dsl-wiki-png

Job DSL Source

The documentation on the Job DSL is quite extensive, but so is the Job DSL itself. They are steadily closing the gaps, but sometimes a piece of information is missing. A prominent example: enumeration values. There are some attributes, that only accept a definite set of values, so you have to know them. Let’s take the CloneWorkspace Publisher as an example. There are two attributes with enumeration values: String criteria = 'Any', String archiveMethod = 'TAR'

cloneWorkspace-wiki

But what about all other values that are acceptable for criteria and archiveMethod? The documentation (currently) says nothing about that. In cases like this it is the easiest thing to have a look at the source code of the Job DSL:

cloneWorkspace-source

Ah, there you go: criteria accepts the values Any, Not Failed and Successful. And archiveMethod has TAR and ZIP. But how can I find the appropriate source for the Job DSL? If you have a look at the Job DSL repository, you will find three major packages:helpers, jobs and views. As the name implies, jobs contains all the job types, and views the different view types. All other stuff like publishers, scm, triggers and the like are located in helpers, so that’s usually the place to start your search. Our CloneWorkspace Publisher is – good naming is priceless - a publisher, so if we step down from the helper to the publisher package: ta-da, here it is :-)

See you at the playground

Sometimes it’s not easy to get your DSL straight. Examples are outdated, you do not get the point, or you just have a typo. Anyway, you type your DSL into the Jenkins editor, save your change and retry again and again, until you fix it.But all this is quite time consuming and developers are impatient creatures: we are used to syntax highlighting and incremental compilation while-u-write. This kind of typing feels a bit historical, so there should be something more adequate, and here it is: The Job DSL Playground is a web application, that let’s type in some DSL (with syntax highlighting) in the left editor side, and shows the corresponding Jenkins config.xml on the other side:

playground

Using the playground has two major benefits. First: no edit-save cycles, so you are much faster. Second: you see the generated configuration XML, which can be useful when you set up a DSL by reverse engineering; means: you have an existing configuration and you want to create a DSL generating exactly that one. I highly recommend you to give it a try, it’s pretty cool.

Nuts and Bolts

The Job DSL is a mature tool and bugs are seldom, but sometimes the devil is in the detail. I’d like you to introduce to some pitfalls I fell into when working with the Job DSL… and how to work around ‘em.

ConfigSlurper

The ConfigSlurper is a generic Groovy DSL parser, which we have used in our examples to parse the microservice.dsl.
def microservices = '''
microservices {
  ad {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'ad'
  }
  billing {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'billing'
  }
  // ... more microservices
}
'''

def slurper = new ConfigSlurper()
def config = slurper.parse(microservices)

// create job for every microservice
config.microservices.each { name, data ->
  createBuildJob(name,data)
}


If you try to use the ConfigSlurper like this in Jenkins you will get an error message:

Processing provided DSL script
ERROR: Build step failed with exception
groovy.lang.MissingMethodException: No signature of method: groovy.util.ConfigSlurper.parse() is applicable for argument types: (script14284650953961421329905) values: [script14284650953961421329905@1563f9f]
Possible solutions: parse(groovy.lang.Script), parse(java.lang.Class), parse(java.lang.String), parse(java.net.URL), parse(java.util.Properties), parse(groovy.lang.Script, java.net.URL)

Possible solution is parse(String)?!? Well, that’s what we do, isn’t it? After searching for a while a stumbled over a post which explained that there is a problem with the ConfigSlurper in the Job DSL, and the workaround is to fix the class loader:

def slurper = new ConfigSlurper()
// fix classloader problem using ConfigSlurper in job dsl
slurper.classLoader = this.class.classLoader
def config = slurper.parse(microservices)

Ah, now it works :-)  This problem may be fixed by the time you are reading this, but just in case you experience this bug, you now have a workaround.

Loosing the DSL context

When I tried out nesting nested views for the post Brining in the herd I stumbled over the following problem, that you sometimes loose the context of the Job DSL when nesting closures. My first attempt was to just nest some nested views. I invented a new attribute group in the microservice.dsl, so I can assign a microservice to one of the (fictional) groups backend, base or frontend. For each of these groups a nested view is created. These group views are supposed to contain a nested view for each microservice in that group, which in turn contains the build pipeline view. Say what?!? The following pictures will show the target situation:

nested-overview

nested-base

nested-base-help

That’s what I wanted to build, so I started straight ahead. I used the groupBy() Groovy method to create a map with the group attribute as keys, and the corresponding microservices as values. Then iterate over theses groups and create a nested view for these. In each group, iterate over the contained microservices, and created a nested Build Pipeline View:

// create nested build pipeline view
def microservicesByGroup = config.microservices.groupBy { name,data -> data.group } 
nestedView('Build Pipeline') { 
   description('Shows the service build pipelines')
   columns {
      status()
      weather()
   }
   views {
      microservicesByGroup.each { group, services ->
          nestedView("${group}") {
            description('Shows the service build pipelines')
            columns {
               status()
               weather()
            }
            views {
               services.each { name,data ->
                  view("${name}", type: BuildPipelineView) {
                     selectedJob("${name}-build")
                     triggerOnlyLatestJob(true)
                     alwaysAllowManualTrigger(true)
                     showPipelineParameters(true)
                     showPipelineParametersInHeaders(true)
                     showPipelineDefinitionHeader(true)
                     startsWithParameters(true)
                  }
               }
            }
         }
      }
   }   
}

Makes sense, doesn’t it? But what came out, is that:

nested-bad

Ooookay. The (nested) Build Pipeline views are on the same nest-level as our intermediate group views backend, base and frontend. If you have a look at the generated config.xml you will see that there is only one <views> element, and all <view> elements are actually children of that element…what happened? Obviously creating the Build Pipeline view has been applied to the outer NestedViewsContext. I don’t know too much about Groovy, but closure code is applied to the delegate, so the delegate seems to be wrong here. Let’s see if can fix that by applying the view creation to the correct delegate:

def microservicesByGroup = config.microservices.groupBy { name,data -> data.group } 
nestedView('Build Pipeline') { 
   description('Shows the service build pipelines')
   columns {
      status()
      weather()
   }
   views {
      microservicesByGroup.each { group, services ->
         view("${group}", type: NestedView) {
            description('Shows the service build pipelines')
            columns {
               status()
               weather()
            }
            views {
               def viewsDelegate = delegate
               services.each { name,data ->
                  // Use the delegate of the 'views' closure 
                  // to create the view.
                    viewsDelegate.buildPipelineView("${name}") {
                     selectedJob("${name}-build")
                     triggerOnlyLatestJob(true)
                     alwaysAllowManualTrigger(true)
                     showPipelineParameters(true)
                     showPipelineParametersInHeaders(true)
                     showPipelineDefinitionHeader(true)
                     startsWithParameters(true)
                  }
               }
            }
         }
      }
   }   
}

So now we explicitly use the surrounding views closure’s delegate to create the view, and…yep, now it works:

nested-overview

If you now inspect the config.xml you will actually find an outer and three inner <views> representing the groups, where each group contains the <view> elements for the Build Pipelines. Fixing the delegate is not a cure for cancer, but it will save your day in situations like these.

Done

That’s all I’ve got to say about the Job DSL :-)

Have a nice day 
Ralf
Sure it's a big job; but I don't know anyone who can do it better than I can.
John F. Kennedy
Update 08/17/2015: Added fixes by rhinoceros in order to adapt to Job DSL API changes

Tuesday, March 31, 2015

Job DSL Part II

In the first part of this little series I was talking about some of the difficulties you have to tackle when dealing with microservices, and how the Job DSL Plugin can help you to automate the creation of Jenkins jobs. In today’s installment I will show you some of the benefits in maintenance. Also we will automate the job creation itself, and create some views.

Let’s recap what we got so far. We have created our own DSL to describe the microservices. Our build Groovy script iterates over the microservices, and creates a build job for each using the Job DSL. So what if we want to alter our existing jobs? Just give it a try: we’d like to have JUnit test reports in our jobs. All we have to do, is to extend our job DSL a little bit by adding a JUnit publisher:
  freeStyleJob("${name}-build") {
  
    ...
    steps {
      maven {
        mavenInstallation('3.1.1')
        goals('clean install')
      }
    }
  
    publishers {
      archiveJunit('/target/surefire-reports/*.xml')
    }
  
  }

Run the seed job again. All existing jobs has been extended by the JUnit report. The great thing about the Job DSL is, that it alters only the config. The job’s history and all other data remains, just like you edited the job using the UI. So maintenance of all our jobs is a breeze using the Job DSL Note: Be aware that the report does not show until you run the tests twice.

test-report

Automating the job generation itself

Wouldn’t it be cool, if the jobs would be automatically re-generated, if we change our job description or add another microservice? Quite easy. Currently our microservice- and job-DSL are hardcoded into the seed job. But we can move that into a (separate) repository, watch and check it out in our seed job, and use this instead of the hardcoded DSL. So at first we put our microservice- and job-DSL into two files called microservice.dsl and job.dsl.

microservice.dsl:
microservices {
  ad {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'ad'
  }
  billing {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'billing'
  }
  cart {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'cart'
  }
  config {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'config'
  }
  controlling {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'controlling'
  }
  customer {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'customer'
  }
  datastore {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'datastore'
  }
  help {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'help'
  }
  logon {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'logon'
  }
  order {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'order'
  }
  preview {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'preview'
  }
  security {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'security'
  }
  shipping {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'shipping'
  }
  shop {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'shop'
  }
  statistics {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'statistics'
  }
  warrenty {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'warrenty'
  }
}

job.dsl
def slurper = new ConfigSlurper()
// fix classloader problem using ConfigSlurper in job dsl
slurper.classLoader = this.class.classLoader
def config = slurper.parse(readFileFromWorkspace('microservices.dsl'))

// create job for every microservice
config.microservices.each { name, data ->
  createBuildJob(name,data)
}


def createBuildJob(name,data) {
  
  freeStyleJob("${name}-build") {
  
    scm {
      git {
        remote {
          url(data.url)
        }
        branch(data.branch)
        createTag(false)
      }
    }
  
    triggers {
       scm('H/15 * * * *')
    }

    steps {
      maven {
        mavenInstallation('3.1.1')
        goals('clean install')
      }
    }

    publishers {
      archiveJunit('/target/surefire-reports/*.xml')
    }
  
  }

}

We now check it into a repository so we can reference it in our seed build (you don’t have to this, I already prepared that for you in the master branch of our jobdsl-sample repository at GitHub).

Finally we have to adapt our seed build to watch and check out the jobdsl-sample repository

dsl-scm-section

… and use the checked out job.dsl instead of the hardcoded one:

dsl-groovy-section

That’s it. Now the seed job polls for changes on our sample respository, so if somebody adds a new microservice or alters our job.dsl, all jobs will be (re-)created automatically without any manual intervention.

Note: We could have put both the microservice.dsl and job.dsl in one file, as we had it in the first place. But now you can use your microservice.dsl independently of the job.dsl to automate all kinds of stuff. In our current project we use it e.g. for deployment and tooling like monitoring etc.


Creating Views

In the post Brining in the herd I described how helpful views can be to get a overview to all your jobs, or even aggregate information. The Job DSL allows you to generate views just like jobs, so it’s a perfect fit for that need. In order to have some examples to play with, we will increase our set of jobs by adding an integration-test and deploy-job for every microservice. These jobs don’t do anything at all (means: they are worthless), we just use them to set up a build pipeline.

Note: in order to use the build pipeline, you have to install the Build Pipeline Plugin.

We set up a new DSL file called pipeline.dsl and push it to our sample repository (again you don’t have to do this, it’s already there). We add two additional jobs per microservice, and set up a downstream cascade. Means: at the end of each build (-pipeline-step) the next one is triggered:

pipeline.dsl
def slurper = new ConfigSlurper()
// fix classloader problem using ConfigSlurper in job dsl
slurper.classLoader = this.class.classLoader
def config = slurper.parse(readFileFromWorkspace('microservices.dsl'))

// create job for every microservice
config.microservices.each { name, data ->
  createBuildJob(name,data)
  createITestJob(name,data)
  createDeployJob(name,data)
}


def createBuildJob(name,data) {
  
  freeStyleJob("${name}-build") {
  
    scm {
      git {
        remote {
          url(data.url)
        }
        branch(data.branch)
        createTag(false)
      }
    }
  
    triggers {
       scm('H/15 * * * *')
    }

    steps {
      maven {
        mavenInstallation('3.1.1')
        goals('clean install')
      }
    }

    publishers {
      archiveJunit('/target/surefire-reports/*.xml')
      downstream("${name}-itest", 'SUCCESS')
    }
  }

}

def createITestJob(name,data) {
  freeStyleJob("${name}-itest") {
    publishers {
      downstream("${name}-deploy", 'SUCCESS')
    }
  }
}

def createDeployJob(name,data) {
  freeStyleJob("${name}-deploy") {}
}

Now change your seed job to use use the pipeline.dsl instead of the job.dsl and let it run. Now we have three jobs for each microservice cascaded as a build pipeline.

all-build

The Build Pipeline Plugin comes with its own view, the build pipeline view. If you set up this view and provide a build job, the view will render all cascading jobs as a pipeline. So now we are gonna generate a pipeline view for each microservice. As before I already provided the DSL for you in the repository.

pipeline-view.dsl
...

// create build pipeline view for every service
config.microservices.each { name, data ->
   buildPipelineView(name) {
     selectedJob("${name}-build")
   }
}

...

Not that complicated, eh? We just iterate over the microservices, and create a build pipeline view for each. All we got to specify, is the name of the first job in the pipeline. The others are found by following the downstream cascade. Ok, so configure your seed job to use the pipeline-view.dsl and let it run. Now we have created a pipeline view for every microservice:

all-pipelines

If you select one view, you will see the state of all steps in the pipeline:

one-pipeline

Having a single view for each microservice will soon become confusing, but as described in Brining in the herd, nested views will help you by aggregating information. So we are gonna group all our pipeline views together by nesting them in one view. We our going to generate a nested view containing the build pipeline views of all our microservices. i will list only the difference to the previous example. The complete script is provided for you on GitHub ;-)

pipeline-nested-view.dsl
...
// create nested build pipeline view
nestedView('Build Pipeline') { 
   description('Shows the service build pipelines')
   columns {
      status()
      weather()
   }
   views {
      config.microservices.each { name,data ->
         println "creating build pipeline subview for ${name}"
          buildPipelineView("${name}") {
            selectedJob("${name}-build")
            triggerOnlyLatestJob(true)
            alwaysAllowManualTrigger(true)
            showPipelineParameters(true)
            showPipelineParametersInHeaders(true)
            showPipelineDefinitionHeader(true)
            startsWithParameters(true)
         }
      }
   }
}
...

So what do we do here? We create a nested view with the columns status and weather, and create a view of type BuildPipelineView for each microservice. A difference you might notice compared to our previous example, is that we are setting some additional properties in the build pipeline view. The point is how we create the view. Before we used the dedicated DSL for the build pipeline view which sets some property values by default. Here we are using the generic view, so in order to get the same result, we have to set these values explicitly. Enough of the big words, configure your seed job to use the pipeline-nested-view.dsl, and let it run.

Note: You need to install the Nested View Plugin into your Jenkins in order to run this example.

pipeline-overview 

Cool. This gives us a nice overview to the state of all our build pipelines. And you can still watch every single pipeline by selecting one of the nested views:

pipeline-overview-one

So what have we got so far? Instead of using a hardcoded DSL in the job, we moved it to a dedicated repository. Our seed job watches this repository, and automatically runs on any change. Means if we alter our job configuration or add a new microservices, the corresponding build jobs are automatically (re-)created. We also created some views to get more insight into the health of our build system.

That’s it for today. In the next and last installment I’d like to give you some hints on how to dig deeper into the Job DSL: Where you will find some more information, where to look if the documentation is missing something, faster turnaround using the playground, and some pitfalls I’ve already fallen into.

Regards
Ralf
I don't know that there are any short cuts to doing a good job.
Sandra Day O'Connor
Update 08/17/2015: Added fixes by rhinoceros in order to adapt to Job DSL API changes

Sunday, March 29, 2015

Job DSL Part I

Jenkins CI is a great tool for automating your build and deployment pipeline. You set up jobs for build, test, deployment and whatever, and let Jenkins do the work. But there’s a catch. In the recent blog post Bringing in the herd I already talked a bit about the difficulties you have to tackle if you are dealing with microservices: they are like rabbits! When you start with a project, there may be only a couple of microservices, but soon there will be a few dozens or even hundreds.Setting up jobs for these herds is a growing pain you have to the master, and that’s where the Job DSL comes to the rescue. This post is the start of a small series on the Job DSL.

One lesson we already learned about microservices is that you have to automate everything. Even – or especially – the configuration of the tooling used to build, deploy, and monitor your application. Not to mention the things you have to do to run the application like distributing, load balancing etc. But let’s start with the build pipeline for the moment. Setting up a Jenkins job is an easy task. When you create a job using the UI, you just have to select the things Jenkins is supposed to do, like check out source code from GIT, run the maven or gradle build, and publish the test results. Once the job does what you want, it is easy to set up this job for another project: Jenkins allows to make copies. Just adapt some data like names and paths, and that’s it. So there’s no challenge in creating jobs for new microservices. If you have to create multiple jobs for each microservice – let’s say integration and acceptance tests, release builds, deployment to various environments – things start to get annoying. But one day you recognize that you – just for example - forgot to publish the checkstyle results, and you will have to change all your existing jobs… manually :-0

Don’t do it. Not even once! What does developers do in order to avoid repetitive, boring, annoying, error-prone tasks? They write a script, yep.We are lazy bones, so instead of doing stuff, we’re telling the machine what to do, and have cup of coffee while the work is being done. Jenkins job definitions are nothing but a little XML, so we could easily write a little script - Groovy has great built-in support for processing XML - and generate that. We could even invent a DSL using Groovy, so our script will be more readable.And since all that is so obvious, somebody already had this idea: The Jenkins Job DSL Plugin.

The development of this plugin was driven by one of the protagonists of microservices: Netflix. They currently have about 600 microservices, so they really need to automate everything. And that’s why they invented the Job DSL: it allows you to describe your Jenkins job using a predefined DSL. It is implemented as a Jenkins plugin, so the creation of the Jenkins job is performed as a Jenkins job itself: Let’s start with a basic example. At first we create a seed job. That’s the common lingual for the job that generates other jobs using the Job DSL:

seed-create

We will need only a single build step: the Job DSL:

seed-create-jobdsl

Now copy the following sample DSL to the editor field:

freeStyleJob('order-build') { 
  scm { 
    git { 
      remote { 
        url('https://github.com/ralfstuckert/jobdsl-sample.git') 
      } 
      branch('order')
      createTag(false) 
    } 
  } 
  triggers { 
     scm('H/15 * * * *') 
  } 

  steps { 
    maven { 
      mavenInstallation('3.1.1') 
      goals('clean install') 
    } 
  } 
}

Let’s go through it step by step. We define a freestyle build job named order-build. Next is a source control block with a GIT repository. I don’t wanted to set up a dozen repositories for the projects used in this example, so I used different branches. So to check out the order project, select the branch named order. We don’t want Jenkins to create a tag (with the build-number) after the checkout, so we set this property to false. In the trigger block we watch the source control system for changes every 15 minutes. In the following (build-) steps block, we define just one step: maven. A maven installation is selected (as predefined in Jenkins) and the goals clean and install are executed. Save and run. Now we have generated a new job order-build:

seed-and-order

Looks good, so let run the order-build: Yep, it builds :-)

order-run

Ok, so we generated a build job. But we could have done the same thing using the Jenkins UI, so where is the big deal? The benefit of generating jobs pays off when generate the same class of job for multiple projects. Let’s say we have a some projects named customer, order, datastore etc. Now we will extend our DSL with a little Groovy code that iterates over these projects, and create a build job for each. So (re-)configure your seed build, and replace the DSL part with the following stuff:

def microservices = '''
microservices {
  ad {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'ad'
  }
  billing {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'billing'
  }
  cart {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'cart'
  }
  config {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'config'
  }
  controlling {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'controlling'
  }
  customer {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'customer'
  }
  datastore {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'datastore'
  }
  help {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'help'
  }
  logon {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'logon'
  }
  order {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'order'
  }
  preview {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'preview'
  }
  security {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'security'
  }
  shipping {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'shipping'
  }
  shop {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'shop'
  }
  statistics {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'statistics'
  }
  warrenty {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'warrenty'
  }
}
'''

def slurper = new ConfigSlurper()
// fix classloader problem using ConfigSlurper in job dsl
slurper.classLoader = this.class.classLoader
def config = slurper.parse(microservices)

// create job for every microservice
config.microservices.each { name, data ->
  createBuildJob(name,data)
}


def createBuildJob(name,data) {
  
  freeStyleJob("${name}-build") {
  
    scm {
      git {
        remote {
          url(data.url)
        }
        branch(data.branch)
        createTag(false)
      }
    }
  
    triggers {
       scm('H/15 * * * *')
    }

    steps {
      maven {
        mavenInstallation('3.1.1')
        goals('clean install')
      }
    }

  }

}

Ok, let’s go through this again step by step. At firs, we define al little DSL describing our microservices. After that, we use the Groovy ConfigSlurper to parse the DSL (ignore this class loader stuff for the moment, that’s a bug). Than we iterate over the microservices and pass the name and the data of each service to the method createBuildJob(). This method contains the Job DSL we used in the first example. Well, almost. We parameterized some things like the name, Git URL and branch, so we can reuse the DSL for creating all the build jobs.

Let the seed job run again and watch the output:

all-projects-seed-console

Looks good. Now let’s see the dashboard:

all-projects-overview


Ta-da. It’s done. We generated a dozen build jobs using a single script. That’s it for the first installment. In the next part, we will alter our DSL, automate the Job creation itself, and create some views.
When you got a job to do, 
you gotta do it well.
Paul McCartney – Live and let die