Verification IP: the questions you should ask

Neill Mullinger is group marketing manager for VIP at Synopsys.

Drop-in verification IP is a mythical creature: Unlike design IP, VIP users don’t have the luxury of ‘drag and drop’ implementations that require relatively little protocol expertise on the user’s part. Verification engineers need protocol knowledge to check that coverage is complete, properly interpret results and debug unexpected behavior. Verification engineers require fast access to information and knowledge of the protocol to quickly ramp on new protocols and new versions of existing protocols.

These requirements won’t go away. Protocols are becoming still-more complex and numerous, and the days of writing your own VIP with the attendant risk of spec misinterpretations and ongoing support load are long gone. Development teams recognize the need to spend engineering cycles on value-added tasks, not re-inventing the wheel. So the onus falls on the third-party VIP vendors. Their VIP must have the capabilities to reduce the protocol expertise needed by users. Choosing the right VIP vendor is a critical choice for project success.

The challenge lies in getting VIP that has the right combination of qualities. With that goal in mind, here is my take on 10 broad ‘need-to-knows’ and, within them, some specific questions to ask vendors. They should help you find VIP that maximizes the effectiveness of your verification.

One quick note: these themes are not ranked in a particular order. Your own projects are likely to determine that, and priorities will vary for different markets. However, you will probably want to pull on all of them to some extent.

Productivity

Minimize the time-sinks in protocol verification.

Getting up to speed on a new protocol is challenging, especially when working with today’s complex protocols like USB 3.0, ARM AMBA 4, AXI4 and AMBA 4 ACE. The challenge is often compounded by the need to ramp on a VIP and its APIs, configuration settings and use-model. It is important to understand what features VIP contains to assist engineers who are not protocol experts in getting to first test as quickly as possible. Some specific questions to ask here are:

Is there fast-track documentation?

• What is provided to help quickly configure the VIP for the target DUT?
• How many example use-cases are there that you can leverage for your project?
• How complete and usable is the documentation?
• Does the VIP come packaged in a ready to use environment?
• How complex is the VIP to instantiate and compile (multi-language VIP often requires multiple compile steps).
• Does the VIP include a verification plan?

The planning stage in particular defines the coverage and requires a great deal of work based around the specification to map out relevant tests that will sufficiently exercise the protocol. The third-party protocol experts who wrote the VIP should be able to provide this.

Methodology and language

Get VIP that is the best fit with your environment.

If the VIP uses a different language and/or methodology than your existing flow, you will need to rely on translation wrappers. These can translate the methodology and language for your testbench. But be warned: Unwieldy language translations can hamper ease-of-use and reduce performance.

Some key alignment questions here are:

• Is the base VIP in a different language and methodology?
• Is every aspect of the methodology and language that you want to utilize in your environment supported by the VIP and available at the API of the VIP?
• How complete is the methodology support for channels, messages, callbacks, error injection, coverage reporting and so on.

Performance

The VIP should run quickly in your simulator.

Growing protocol complexity means that VIP takes up an increasing portion of testbench code. Performance is critical and can be a determining factor in overall verification throughput. So you don’t want your VIP to be a bottleneck.

Key checklist items here include:

• Does the VIP run natively in the simulator?
• Does it have multiple translation wrappers or use a PLI?
• Does it run faster on one simulator than another?

Remember, using VIP that runs natively and does not rely on wrappers can be two, three, even four times better in terms of performance than VIP that does not run natively and needs wrappers.

Coverage

Your VIP should help you know when you are done.

Built-in protocol coverage enables you to measure your progress towards hitting coverage goals without having to find a protocol expert to write a large amount of testbench code.

There are two sides to coverage:

1. A verification plan documents the coverage points needed to verify the protocol.
2. Built-in coverage bins report the coverage points that have been hit in simulation.

Having this kind of capability built into the VIP saves weeks of development time and instills confidence as you make progress toward the fullest viable coverage.

Also, you need some flexibility. So, to recap, determine here:

• How complete is the built-in coverage reporting?
• Is it SystemVerilog?
• Can I easily edit and add to the plan and the coverage?

Testbench

Your VIP should give you a jump-start toward reaching coverage.

Look at what features come with the VIP that save on testbench development and help you reach high coverage goals rapidly. There are some fundamental indicators:

• Is there a library of built-in sequences to jump-start testbench development?
• Does the library provide a good base-line of high coverage from which you can quickly identify the coverage holes in the design (using the built-in coverage reporting of the VIP)?
• Can you then modify the sequences to hit more coverage points?

Typically, a few directed tests will be needed to round-out simulation.

Debug

Your VIP must tell you what went wrong and what caused it.

Today’s protocols can be very hard to debug using traditional simulation. Unraveling the interleaved transfers, transactions, packets and handshaking can be very time-consuming. Most VIP only provides simple log files to show the traffic. You should push for more:

• What visualization aids does your VIP vendor provide?
• How will it help you efficiently locate and debug errors?
• How easily can it help you see relationships between the transactions?
• Can you quickly filter out irrelevant traffic and just see the transactions with certain attributes?
• Can you correlate between different protocols in a straightforward way?
• Does the VIP link what is shown in the simulation log directly with the protocol activity?

Architecture

Know what’s going on under the hood

Alignment between the architectures of a VIP and the protocol matters. For example, a consistent language and methodology from the testbench through the API and then through to the wires makes it simpler for users to inject errors, monitor traffic and create appropriate scoreboards. So, think about these questions:

• Does the architecture of the VIP match the architecture of the protocol?
• Does the VIP enable access to all layers of a multilayered protocol (e.g., PCI Express or USB 3.0)?
• How easy is it to use, monitor and augment the VIP accordingly?

Quality

You must be able to trust the VIP and its vendor.

Quality is essential for productivity. VIP is supposed to provide an ideal model against which to test your design. To be able to trust your supplier, determine:

• What is the VIP vendor’s track record on quality?
• Do they have a history of success with many customers?
• Do they have a robust regression environment?
• Do they test for your methodology and simulator before the release of each new version?
• Is the VIP architecture straightforward and therefore less prone to bugs?

Support

When errors occur, you need to be able to get outside help.

As important as it is for verification engineers to have some knowledge of IPs, they almost certainly can’t reach expert-level for every IP in use – even across an entire team. Today’s designs incorporate so many specifications, and new ones are being added at such a rate that the resources needed to achieve that in-house are a tremendous challenge.

However, when hard-to-trace errors occur, you probably will need to speak to someone who does have that intimate protocol knowledge. As obvious as it sounds, making sure that you can get strong vendor support is vital. So, extending our list of key questions now adds:

• What is the vendor’s reputation for support in the industry?
• How well positioned is the company to provide the support you need for the VIP you are using?
• Can your supplier give access to people with deep experience of the protocol, methodology and simulation features?
• Are those specialists available in your region so you can get quick answers to urgent questions?

Simulator support

VIP is a long-term commitment.

Verification environments evolve for technical and business reasons. Many companies have a mix of simulation environments and need VIP that works across them. It is important that VIP does not trap you into a certain methodology or simulator. Many vendors claim simulator independence but you need to confirm that as well. So drill down and ask:

• Does your supplier fully test its VIP on all methodologies and simulators?
• Which features work only on one simulator and/or in one methodology?
• What would happen if you decided to change either methodology or simulator?

Ready to go

It is a fair list of things to consider, but, as we noted at the start, you are making a critical choice. If you get it wrong, you’ll have more to worry about than just buyer’s remorse.

As you’d suspect, we’d also be happy to give you our answers on these and questions of your own that hopefully they have provoked. Synopsys offers a broad portfolio of Verification IP, for more information visit: www.synopsys.com/VIP

Neill Mullinger is Group Marketing Manager, VIP, Synopsys.