Verification defined(zz)

source::  

http://www.chipdesignmag.com/bailey/2008/05/15/verification-defined/trackback/

By: Brian Bailey

No blog on verification would be complete without a discussion about what verification actual is – and this is a subject that is near and dear to my heart. In discussions with EDA vendors and users, it is clear that they sometimes forget the fundamentals – and so I always include a small section about this in any talk that I give. It amazes me that people do not get tired of hearing me say this, and yet so far, nobody has thrown any rotten eggs (but that could be because I rarely do talks over meals).

A great starting point are the IEEE definitions. While these definitions are long and wordy, I have yet to find any other definitions that are as complete and include everything that should be there. I wish I knew who to thank for these definitions, so if you know – please let me know so that I can send them my thanks.

So without further ado, here is the definition for verification:

Confirmation by examination and provisions of objective evidence that specified requirements have been fulfilled.

In short, verification is about ensuring that an implementation matches its specification. We will come back and dissect this definition in a minute, but for purposes of contrast, here is the definition for validation:

Confirmation by examination and provisions of objective evidence that the particular requirements for a specific intended use are fulfilled.

In short, validation is about making sure that the specification is correct, or in other words is fit for the intended purpose.

So lets go back to verification – which is what we spend 99% of our time doing, and dissect the definition.

-Confirmation by examination. If you don’t look at something, then you have no hope of telling if it is right or not. While this may sound obvious, it is one of the most inefficient aspects of verification today. As an example, if a verification scenario is produced to verify a particular aspect of functionality, and it is expected that correct behavior. can be ascertained by observing a particular value in memory at a certain time, then checking only for that value while telling you that your test example passed fails to verify if any bad actions were produced on the other outputs of the design. They simply were not observed even though a large amount of simulation time may have been spent producing that information.

-provisions of objective evidence. If we continue with our example above, we could claim to have solved the problem by looking at the output produced by the verification scenario and by capturing the results of the other data produced by the simulator. Comparing this against subsequent runs may tell you if a change made in the design affected other aspects of the output. But that evidence is not objective. It is not known if it was right to start with and, as so often happens, a team under pressure will just verify again the primary means of telling that the test passed and will just capture the new outputs on the non-essential signals. In other words, if we do not know why a design produced certain signals, values or events, then we cannot use them to inform. us about the validity of those results.

-specified requirements. The best way to interpret this is to consider it as the definition of coverage. Functional coverage points are one way in which these can be derived from the specification and even ordered in terms of priority of the functionality that must be verified. However, don’t dismiss the structural coverage metrics, which include code, expression, path, FSM and many other metrics that can be automatically extracted from the implementation source. A properly prepared verification plan can state that a certain requirement has been satisfied if one of these coverage metrics has been satisfied. This is particularly powerful if we are talking about path or FSM coverage and for some forms of verification, toggle coverage is also a highly valuable indicator. Let’s remember that all of these are nothing but indicators. They are isolated observations about the state of the design at a specific moment in time, and tell us nothing about what happened before or will happen in the future. To find out why that is important, read on to the next clause.

-have been fulfilled. Last but by no means least, is that we have to ensure that when we meet the conditions for the specified requirements, that an actual act of verification has taken place. Sorry for the recursion there. The problem is that the coverage metrics in use today are what are called stimulus metrics, in that they tell how effective the stimulus is at activating certain things in the design – be they lines of code, or functional observations. Coverage does not equate to verification. In fact all of these coverage metrics assume that full verification is happening by another mechanism – by the checkers. But how can we tell if that is actually happening? Today, there are two mechanisms. The first is to only put functional coverage points in the checkers. This will ensure that all effects are propagated to a place where a validity check is being made. This still does not ensure that the right check is being performed, but it is a lot better. The second is to use a tool, such as Certitude from Certess. This actually measures which coverage points are actually propagated and checked by using a technique called mutation testing. What this does is to make a small change in the design and actually check to see that the testbench fails. If it doesn’t then you know the coverage point may have been satisfied, but is worthless. I will save further discussion about mutation analysis to another blog entry.

So with all of that as a background, how do some of the verification tools out there actually stack up? I am not going to provide my answer to that just yet as I would like you to think about it, but as a hint the answer is very few tools actually do everything necessary to be called verification according to my definition. Let me know which ones you think do and don’t stack up?

Brian
I’ve got you covered.