How continuous integration with Jenkins serves verification flows
A technique built for software development is now helping hardware engineers master increasingly complex verification flows.
Increasing time spent on verification poses practical design management problems, especially at the intersection between design iterations and data flows. In response, many hardware engineering teams are adopting an established concept from the software world, ‘continuous integration’ (CI). Most use the open-source Jenkins automation server to implement the technique.
The main problem CI addresses in verification (and, for that matter generally) is easily described.
More ‘time’ spent in verification tends to translate not so much into lengthier design cycles as a need to assign more people to the team. Further complications occur when, for example, team members are based at multiple locations (nationally and/or internationally) and some specialized verification tasks have to be outsourced to contractors.
At some point, all these users will check-out the main project to undertake their work within it and, when each has finished, check the project back in for harmonization with their colleagues’ efforts. This is a necessary process but one that, if not properly managed, can quickly drag a project down to ‘integration hell’, where there are multiple ‘versions’ floating around in confusion.
Continuous integration establishes process automation and standard practices that aim to ensure the frequent and appropriate harmonization of a project’s main build. CI proceeds from the assumption that the longer a core project is checked out, the harder it becomes to eventually reintegrate the work done on it.
Therefore within a CI flow for a software project, a daily reintegration point is typically set. However, these can be more or less frequent depending on the nature of the work in hand. Some projects automatically undertake CI whenever a piece of code is checked back in.
Continuous integration for hardware verification
Before going further, we must distinguish between what a typical EDA vendor’s verification management software does and what CI software and processes do.
We will use the example of the Questa VRM (Verification Run Manager) tool from Mentor Graphics. This is proprietary software and, within the context of the broader Questa suite, automatically analyzes the critical hardware-design-specific aspects of a project run, once it has been triggered.
For the CI, we will then use the example of Jenkins, the most popular CI build currently available. Its role here can be roughly split into two parts. Jenkins triggers a Questa VRM regression, and after that run, it extracts metrics from VRM that are displayed for the user over a web-based interface.
This is a simplification – good CI can entail multiple tasks and regressions – but it does illustrate the different strengths of the two types of software tool and how they complement one another.
Jenkins, as the CI, understands scheduling, the processes of checking in and out a main project build, and the need for reports after each harmonization as work progresses.
VRM, as a hardware design tool triggered by the CI, has the ability to understand what needs to be verified and harmonized for that specific type of project and can generate metrics for a tool like Jenkins to restate in a standard form.
There are a number of CI builds available, but Jenkins – an open-source spin-off from the Hudson project at Sun Microsystems/Oracle – is by far the most common, accounting for more than 75% of the market.
As a popular open-source format (released under an MIT license), Jenkins today has a large number of contributors who maintain its core stability and efficiency, and who have also authored almost 1,400 plugins to make CI easier to use with other software. They also provide valuable resources for users who are coming fresh to both the CI concept and the Jenkins software specifically.
Given Jenkins’ heritage, most plugins are still aimed at software development (although a good many also perform useful generic tasks). However, as the hardware community is becoming more interested in CI processes, more are being added to address that market’s needs. These ‘hardware’ contributions now include a plugin for the Questa VRM.
Thomas Ellis has written extensively about the specific use of Jenkins with Questa at Mentor’s Verification Academy, and you can see his description of its set-up, use and features here.
One specific point he makes about the compatibility between Jenkins and VRM in terms of regressions reports is worth restating here also. It illustrates the greater granularity VRM users can achieve with a full CI strategy.
“The plug-in leverages the vast amount of data VRM collects from the regression runs, allowing for all sorts of data to be analyzed that would otherwise need to be collected and reported manually,” Ellis says. “Otherwise difficult questions become easy to answer. Has this test with this seed ever failed before? What is the host utilization like during a nightly regression? When did coverage drop off?”
The important step being taken here is that in addition to using regression to harmonize the efforts of a potentially scattered team, CI can also be leveraged – via a plugin – to harvest data and flag up potential bugs in a design at an early stage.
You can find out more about Jenkins, its history and the features in its recent 2.0 release – as well as sign up for its forums and consult other users – at http://jenkins.io.