Monday, April 13, 2015

Job DSL Part III

The previous part of this little series on the Job DSL gave you some examples on maintenance, automating the job creation itself and creating views. This last installment will complete the little round trip through the Job DSL with some hints on documentation, tooling and pitfalls.

Documentation

If you search the internet for the Job DSL one of the first hits will be the corresponding wiki. This is the most valuable source of information. It is well structured and maintained, so new features and missing pieces are filled in regularly. If you are looking for any details on jobs, the job reference is your target. If you like to generate a view, there is a corresponding view reference.

job-dsl-wiki-png

Job DSL Source

The documentation on the Job DSL is quite extensive, but so is the Job DSL itself. They are steadily closing the gaps, but sometimes a piece of information is missing. A prominent example: enumeration values. There are some attributes, that only accept a definite set of values, so you have to know them. Let’s take the CloneWorkspace Publisher as an example. There are two attributes with enumeration values: String criteria = 'Any', String archiveMethod = 'TAR'

cloneWorkspace-wiki

But what about all other values that are acceptable for criteria and archiveMethod? The documentation (currently) says nothing about that. In cases like this it is the easiest thing to have a look at the source code of the Job DSL:

cloneWorkspace-source

Ah, there you go: criteria accepts the values Any, Not Failed and Successful. And archiveMethod has TAR and ZIP. But how can I find the appropriate source for the Job DSL? If you have a look at the Job DSL repository, you will find three major packages:helpers, jobs and views. As the name implies, jobs contains all the job types, and views the different view types. All other stuff like publishers, scm, triggers and the like are located in helpers, so that’s usually the place to start your search. Our CloneWorkspace Publisher is – good naming is priceless - a publisher, so if we step down from the helper to the publisher package: ta-da, here it is :-)

See you at the playground

Sometimes it’s not easy to get your DSL straight. Examples are outdated, you do not get the point, or you just have a typo. Anyway, you type your DSL into the Jenkins editor, save your change and retry again and again, until you fix it.But all this is quite time consuming and developers are impatient creatures: we are used to syntax highlighting and incremental compilation while-u-write. This kind of typing feels a bit historical, so there should be something more adequate, and here it is: The Job DSL Playground is a web application, that let’s type in some DSL (with syntax highlighting) in the left editor side, and shows the corresponding Jenkins config.xml on the other side:

playground

Using the playground has two major benefits. First: no edit-save cycles, so you are much faster. Second: you see the generated configuration XML, which can be useful when you set up a DSL by reverse engineering; means: you have an existing configuration and you want to create a DSL generating exactly that one. I highly recommend you to give it a try, it’s pretty cool.

Nuts and Bolts

The Job DSL is a mature tool and bugs are seldom, but sometimes the devil is in the detail. I’d like you to introduce to some pitfalls I fell into when working with the Job DSL… and how to work around ‘em.

ConfigSlurper

The ConfigSlurper is a generic Groovy DSL parser, which we have used in our examples to parse the microservice.dsl.
def microservices = '''
microservices {
  ad {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'ad'
  }
  billing {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'billing'
  }
  // ... more microservices
}
'''

def slurper = new ConfigSlurper()
def config = slurper.parse(microservices)

// create job for every microservice
config.microservices.each { name, data ->
  createBuildJob(name,data)
}


If you try to use the ConfigSlurper like this in Jenkins you will get an error message:

Processing provided DSL script
ERROR: Build step failed with exception
groovy.lang.MissingMethodException: No signature of method: groovy.util.ConfigSlurper.parse() is applicable for argument types: (script14284650953961421329905) values: [script14284650953961421329905@1563f9f]
Possible solutions: parse(groovy.lang.Script), parse(java.lang.Class), parse(java.lang.String), parse(java.net.URL), parse(java.util.Properties), parse(groovy.lang.Script, java.net.URL)

Possible solution is parse(String)?!? Well, that’s what we do, isn’t it? After searching for a while a stumbled over a post which explained that there is a problem with the ConfigSlurper in the Job DSL, and the workaround is to fix the class loader:

def slurper = new ConfigSlurper()
// fix classloader problem using ConfigSlurper in job dsl
slurper.classLoader = this.class.classLoader
def config = slurper.parse(microservices)

Ah, now it works :-)  This problem may be fixed by the time you are reading this, but just in case you experience this bug, you now have a workaround.

Loosing the DSL context

When I tried out nesting nested views for the post Brining in the herd I stumbled over the following problem, that you sometimes loose the context of the Job DSL when nesting closures. My first attempt was to just nest some nested views. I invented a new attribute group in the microservice.dsl, so I can assign a microservice to one of the (fictional) groups backend, base or frontend. For each of these groups a nested view is created. These group views are supposed to contain a nested view for each microservice in that group, which in turn contains the build pipeline view. Say what?!? The following pictures will show the target situation:

nested-overview

nested-base

nested-base-help

That’s what I wanted to build, so I started straight ahead. I used the groupBy() Groovy method to create a map with the group attribute as keys, and the corresponding microservices as values. Then iterate over theses groups and create a nested view for these. In each group, iterate over the contained microservices, and created a nested Build Pipeline View:

// create nested build pipeline view
def microservicesByGroup = config.microservices.groupBy { name,data -> data.group } 
nestedView('Build Pipeline') { 
   description('Shows the service build pipelines')
   columns {
      status()
      weather()
   }
   views {
      microservicesByGroup.each { group, services ->
          nestedView("${group}") {
            description('Shows the service build pipelines')
            columns {
               status()
               weather()
            }
            views {
               services.each { name,data ->
                  view("${name}", type: BuildPipelineView) {
                     selectedJob("${name}-build")
                     triggerOnlyLatestJob(true)
                     alwaysAllowManualTrigger(true)
                     showPipelineParameters(true)
                     showPipelineParametersInHeaders(true)
                     showPipelineDefinitionHeader(true)
                     startsWithParameters(true)
                  }
               }
            }
         }
      }
   }   
}

Makes sense, doesn’t it? But what came out, is that:

nested-bad

Ooookay. The (nested) Build Pipeline views are on the same nest-level as our intermediate group views backend, base and frontend. If you have a look at the generated config.xml you will see that there is only one <views> element, and all <view> elements are actually children of that element…what happened? Obviously creating the Build Pipeline view has been applied to the outer NestedViewsContext. I don’t know too much about Groovy, but closure code is applied to the delegate, so the delegate seems to be wrong here. Let’s see if can fix that by applying the view creation to the correct delegate:

def microservicesByGroup = config.microservices.groupBy { name,data -> data.group } 
nestedView('Build Pipeline') { 
   description('Shows the service build pipelines')
   columns {
      status()
      weather()
   }
   views {
      microservicesByGroup.each { group, services ->
         view("${group}", type: NestedView) {
            description('Shows the service build pipelines')
            columns {
               status()
               weather()
            }
            views {
               def viewsDelegate = delegate
               services.each { name,data ->
                  // Use the delegate of the 'views' closure 
                  // to create the view.
                    viewsDelegate.buildPipelineView("${name}") {
                     selectedJob("${name}-build")
                     triggerOnlyLatestJob(true)
                     alwaysAllowManualTrigger(true)
                     showPipelineParameters(true)
                     showPipelineParametersInHeaders(true)
                     showPipelineDefinitionHeader(true)
                     startsWithParameters(true)
                  }
               }
            }
         }
      }
   }   
}

So now we explicitly use the surrounding views closure’s delegate to create the view, and…yep, now it works:

nested-overview

If you now inspect the config.xml you will actually find an outer and three inner <views> representing the groups, where each group contains the <view> elements for the Build Pipelines. Fixing the delegate is not a cure for cancer, but it will save your day in situations like these.

Done

That’s all I’ve got to say about the Job DSL :-)

Have a nice day 
Ralf
Sure it's a big job; but I don't know anyone who can do it better than I can.
John F. Kennedy
Update 08/17/2015: Added fixes by rhinoceros in order to adapt to Job DSL API changes