Introduction to System-on-Chip Functional Verification

Taking On The Verification Challenge With Better Performance, Capacity, and Productivity Avery Design Systems
2 Atwood Lane Andover, Ma 01810 978 689 7286 www.avery-design.com

Seminar Agenda

The state of functional verification today

Scope and scale Current best practices

Technology Methodology System-On-Chip Large-scale systems designers

Current problems and challenges

The future of functional verification

Industry Trendlines Averys perspective and vision Avery solutions and products

What is Driving Functional Verification?

Verification requirements grows at a multiple of Moore's Law

10X for ASICs 100X for ASIC-based systems and SOCs which include embedded software

Verification Complexity = (Architectural Complexity, Amount of Reuse, Clock Frequency, System Software Worst-Case Execution Time, Engineer Experience)

10B Verification 100M cycles 1M

2002 1996 1990

100K 1M 10M Effective gate count

Verification productivity level increases lags all other aspects of design!

Growing Design Verification Investment...

Design Complexit y

Time to Market

Increasing design complexity imposes a thorough functional verification process Verification needed at all levels : Block - Module - System

How SOCs and Embedded Systems Exponentially Expand the Scope and Scale Embedded System Host System of Verification ? /SOC
System Applications Operating System Device Drivers Hardware Devices Embedded System Application Embedded OS/Firmware Device Drivers Hardware Devices System Verification Target

Host System

ASB-PCI Bridge

1. 2. 3. 4.

Multiple core functions On-chip interfaces protocols Multiple off-chip interface protocols Embedded processors, firmware, OS

What Kinds of Verification Are Being Used?

Architectural verification models (MatLab), Formal Model Checking

Custom performance modeling,

Algorithm

Implementation verification

Circuit-level

Spice, Fast Spice, Symbolic simulation,

Formal Equivalency

Gate/Structure HW Emulation

Logic Simulation, Formal Equivalency,

Timing RTL

Static Timing Analysis Logic Simulation, Formal Model Checking,

HW Emulation

Manufacturing verification

At-speed

Logic Simulation ATPG, BIST

Manufacturing defects

The SOC Verification Process

Architecture Spec
Performance Analysis Algorithm/Architecture Validation

ASIC Functional Spec

Verification Test Plan

If you start verification at the same time as design with 2X the number of engineers, you may have tests ready when the ASIC RTL model is done
Test Assertions Test Streams Functional Coverage Monitors Result Checkers Protocol Verifiers Reference Models/Golden Results

RTL Modelling

Verification Target Configuration

A Closer Look at Implementation Verification

System verification:

Does the overall design work? - Run application : check end result Does the communication between blocks work? - Check blocks synchronization, latency Does an input stream generate the expected output behavior? - Check block functionality and corner cases.

Module verification:

Block verification:

Finding/fixing bugs costs in the verification process

Time to fix a bug
System

Module

Block

Design integration stage

Chip NREs increasing making respins an unaffordable proposition RESPIN Average ASIC NRE ~$122,000 SOC NREs range from $300,000 to $1,000,000

Current verification practice

High quality verification is only done at system level

Self-checking methodology is standard At the end of the system verification a high confidence level is reached

Module level verification is typically NOT done

It is felt that too much work is involved and little pay-back Nobody is really responsible for module verification

10

Current verification practice (2)

Block level verification is done ad-hock

Often stimuli-only and visual inspection of simulation results Structural coding style instead of behavioral Testbench derived from RTL implementation and not specification

Testing what is implemented and not what is specified

Testbench typically not well structured / everything in a flat process

Too much verification / Debug needs to be done at system level Resulting in a long verification process (70% of the design cycle time)

difficult reuse, hard to read and maintain; hard to document

11

System verification

Should be used as an overall application validation

Not suited for finding RTL implementation bugs Do I get the correct end state results? The sequence of operation is not relevant Signal activity is not relevant Run application / test case and check that at the end I get the expected memory / register image It is too hard to define the sequence of operation; a cycle accurate circuit model is typically not available. Application specialist; not necessarily RTL design / VHDL literate.

Simulation end-state validation

Typical approach

Engineering type

12

System verification prerequisites

Test plan

Application requirements Overall performance tests Finding / debugging RTL bugs at system level is extremely time consuming RTL designers might not be available to support debug process Long iteration cycles

Extensive block and module verification COMPLETED

Re-verify block and module functionality before restarting system verification Implies new synthesis runs

Finding Bugs late in the design process can substantially delays tape-out. Bugs will/should be found but their number should be minimized by extensively verifying Blocks and Modules

13

Module Verification

Required to verify inter-block communication / sub-system functionality

M 1 M 3

M 2
B1 B5 B3 B2 B4

...

M n

Module-level testbench characteristics

Bus protocol generator and checkers Reuse of module-level generators

14

Module verification goals

Verify module operation Verify inter-block communication Verify module performance

sufficient cache-memory and FIFOs size sufficient inter-block bus bandwidth optimal processing power partitioning

Pre-requisite : Block-level verification COMPLETED Engineering type

Senior designer / Lead designer Verification engineer

15

Block verification
Data flow2 Data flow1 Rese t Wai t Acces s Rea d Writ e

Block

Checker 1 Checker 2

16

Block verification goals

Verification goal

Debug RTL implementation Verify implementation is according to spec Verify block post-synthesis netlist RTL designer Possibly not the same person who did the RTL implementation simple data flow one process for each data stream define test cases before RTL implementation develop test cases before or during RTL implementation

Engineering type

Testbench architecture

Process

17

Best practice for a Block and Module testbench

Easy and fast to write for RTL designer use HDL language at behavioral level Self-checking : mandatory for efficient verification Visual inspection is error-prone especially in debug cycle Hierarchically structured allows abstraction: dataflow - operation - signal Reusable direct reuse of (protocol) generators and checkers from block to modules reuse from project to project when same basic protocol is used : ARM AMBA Bus, ATM-UTOPIA, PCI, SONET,...

18

Whats Considered Average Today?

Applications
Total Regression Cycles Cycles Single Test Length (Cycles)

Voice over Packet (VoP) chip-set embedded solution

1010

107

105

G.7XX, G.168, DSL, IP 1-2M gates 40 Gbps per wavelength 3.2 Tbps fiber capacity 622 MHz (SONET/SDH 192), 322K bits/frame (STS-192) 25M gates / wavelength OC-3 to OC-48, ATM 30 M gates 45 M transistors

Long-haul optical transport system

1013

1010

107

Terabit router

1012 1016

109 1012

106 106

Intel P4 microprocessor

19

Functional Testing Methods

20

Functional Testing Methods Contd

Simulation-based

Directed testing

Generate testcases for each test assertion and related test parameters Interfaces exercised independently Block, module, and system-level Generate testcases for one or more test assertions concurrently Test parameters selected randomly from enumerated list Multiple interfaces exercised concurrently Block, module, system-level Automatic functional test generation Deterministic process using SAT solver engine(s) Guided by functional coverage targets

Pseudo-random testing

Auto-directed testing

Formal methods

Formal model checkers

Prove design complies with design properties Or not, generate counter examples Verifies specific assertions/properties in the context of actual simulation Plus, searches limited state space around seed state (assertion triggers) for noncompliance and generates counter examples

Hybrid, Semi-formal checkers

21

Measuring Progress of Verification

How do you know youre getting close?

Test pass rate Bug occurrence rate HDL code coverage (line, state, state arcs, expression) Micro-architectural features exercised (normal, corner cases) IO and System Protocols Exercised

How do you know youve testing everything

22

Putting together the SOC Verification Environment Puzzle

23

So Whats The Answer?

Industry Trendlines (and Hype)

Design productivity

C++ based design will supercede HDL-based design

Faster simulation through abstraction Using the same language for HW and SW must make HWSW co-design and verification easier Software engineers can lend a hand designing the hardware

Behavioral synthesis FPGAs will replace ASICs

Ill finish the design and verify it in the lab!

Verification productivity

High-level Verification Languages (HVLs) are the only way to implement transaction-level verification Formal model checking is on the verge of becoming practical

25

What is the Industry Doing?

Use new language for verification other than design

Synopsys Vera, Verisitys e-language, Accelleras SUGAR

Use C or C++ for design and verification

Draft standard by June 2003

Synopsys SystemC, Cadences TestBuilder

Next generation Verilog HDL and simulator

Withering?

Draft standard Co-designs Superlog combines C constructs and Verilog together by June 2003
SystemVerilog Language Standard builds on IEEE-1364-2001 Averys TestWizard adds verification features to Verilog IEEE-1364-1995, VHDL 1076 (VCS, NC-Verilog, MTI)

Extend current Verilog simulators

26

Pros and Cons for New Language Approach

Language is specific to address verification

But 80% of verification features are the same as design features

Verification engineers have their own language

Divides design and verification into 2 separate groups

Performance penalties to pass data between verification and design

Difficult for verification to get design data

27

Pros and Cons of C and C++ Approach

Everyone knows C, a lot know C++

Multi-bit operators can be easily be overloaded as C++ operators You dont care about timing, X, and design contains no bus

Fast simulation, some people claim that simulation is 100X faster than Verilog

But the design description level was different between the two There is no inherent speed advantage if models are described at the same level Difficult to handle large amount of parallelism

Pay nobody

Difficult to correlate design bugs to C or C++ model Difficult to do incremental refinement of the design Synthesis and other tools are not mature Cannot accept legacy code

28

Pros and Cons of Next Generation Approach

Its not Verilog, its not C

No backwards compatibility

Long standards approval process Requires widespread compliance by EDA vendors

Verification-centric enhancements do not have same priority as design and synthesis-centric enhancements

29

Verification Crisis is Not Solvable by Raising Abstraction Level Alone

Verilog and C are at the same abstraction level with only data structure limitation

Designs today only use a very limited subset, restricted by Synopsys synthesis subset

Systematic exhaustive testing of designs is against human nature

Verification crisis is only solved with breakthroughs in methods, technologies, process

30

The Avery Vision

Tackle the 100 M gate systems verification challenge through better

Productivity Performance + Capacity Economics

Innovate new verification technologies and flows addressing RTL, HW-SW systems, and IP Build on customers investments in tools, language standards, and engineering know how

31

The Avery Strategy

Defining Principles for Effective Functional Verification

Transaction-oriented verification

Raise test generation abstraction and tools to match rise in architectural complexity Standards-based design increases design and interoperability and designer productivity System-level simulation performance/capacity Early HW-SW integration Design management Use only existing standard implementation languages

Reusable verification IP

System integration issues

Build on existing tools and methods

32

The Avery Solution

Required Elements for Effective Functional Verification

Verilog/VHDL/C-based testbench automation

Up to 3 X better efficiency using native Verilog and C/C++

Test development Coverage analysis Assertions and protocol verification

Distributed parallel simulation

5-10 X increase in performance and capacity vs. stand-alone HDL simulators through parallel simulation processing

33

The Avery Solution Contd

Required Elements for Effective Functional Verification

Hardware-Software co-verification

Earlier system SW integration and compliance validation Integrated HDL and C/C++ verification flow for IP

C/C++, CycleC, SystemC, Cynlib, and SpecC

Leverage all the best EDA tools and solutions

Access to best-in-class simulators

34

TestWizard Is

35

TestWizard is a Complete Transaction-level Verification Solution

Transaction-based Verification

Decouple testcase content and signal-level interfaces Apply pseudo-random test case generation to generate more thorough set of testcases Eliminate manual result checking and use only dynamic selfchecking tests and transaction database logging

Assertion-based Verification

Use automatic protocol assertion checking to ensure system compliance during every simulation

Coverage-based Verification

Formally measure functional test coverage vs. design architecture Check all properties & scenarios have been tested

36

Key Benefits of TestWizard

Compared to conventional HDL methods

High-level structured methods and psuedo-random generation generates comprehensive workloads to test design 3X more productivity Up 10X better code density Closed-loop method achieves 100% functional coverage Assertions provide always on validation of protocols

37

38

TestWizard Verification is

Transaction modeling using complex data structures Test generators generate random transaction sequences based on test parameters Transactors stream transactions in/out of DUT Result checking is dynamic self-checking using transaction database queries Assertion-based verification checks protocol compliance on the fly Closed-loop functional & assertions coverage analysis adapts traffic shape for more effective & efficient testing

39

#1: Think Transactions

Container of basic units of data and control that a system works on Requires complex datatypes to implement Decouple tests/transactions and signal-level interfaces

40

TestWizard Supports Complex Datastructures

Records

Hierarchical (i.e. Records of records) Record Pointers Dynamically allocated/deallocated Dynamic field sizing Unions support composite types Extend base records Randomization

Supports reuse strategies

Integrated with other TestWizard functions

Default and instance overriding of fields

Record streaming functions simplify BFM/transactors Transaction database logging and SQL-like queries to simplify transactionlevel verification Mutable and hierarchical provides unstructured data aggregation Supports string, numeric constants and numeric ranges

List types and list access functions

41

Examples of Complex System Transactions: Internet Protocols

42

#2: Random Traffic Shaping Parameters

Protocols, #packets, interval time, IP src/dst addresses, MPLS labels, Time-to-live, hop-limit, MPLS label stack depth, LDP messages

43

TestWizard Supports Intelligent Random Testing in Verilog/VHDL

Automated stream generation modes

Directed Constrained Random

Boundary case pseudo-random Weighted Distributions Related case pseudo-random Cyclic pseudo-random Filter functions Enumerated value ranges

Uses list types for value set definitions

44

45

#3 Transaction Database Logging

Stores transaction records

Supports multiple transaction record types Timestamps

Automatically manages database by size and age

46

#4: Dynamic Result Checking

Transaction Database required to verify complex transaction processing like:

Out-of-order transactions Multi-protocol processing Modified headers QoS and SLA

Supports complex searches

Record Type, Field Type, Field Value, Time

47

#4a: Dynamic Checking Examples

Does the router generate ICMP Type 11 (timeout messages) back to IP source address when TTL and HLIM reaches zero

Predictor stores IP pkt in Port #0 egress expect xlog db and starts timeout assertion thread Router egress emits ICMP Type 11 packet Dynamic checker examines actual egress pkt and compares to expected egress pkt; eventual match or timeout assertion violation

Is IPv6 QoS met for certain specified flow labels Does router perform correct label stack operations Does router correctly respond to LDP label map requests for good and bad labels

48

#5 Assertions

On the Fly protocol verification of System-level and Interface protocols Powerful & concise temporal property semantics to Verilog/VHDL

Repeating sequences Temporal ranges Conditional sequences Logical or/and

Highly reusable Comparable to Sugar (Accellera FPSL WG) Sugar-to-TestWizard translator available soon

49

#5 Assertions contd

roperty: For all read or write ansactions in the address range: hAA:-hBB, data transfers of 1-16 bytes re allowed. Data transfers can complete ut-of-order. On transaction completion TestWizard assertions support complex etection, byte counter must match Overlapping sequences using threads and nitial transaction header byte count

tags

50

#5 Assertions contd
Assertion:
Define requestor address range and byte count range lists Define assertion thread tags for requestor header address and byte count Define assertion tags for split completion address and byte count Define sequence: On new transaction request (start), save address and byte count headers to new assertion thread( h1); wait for end2 to signal split completion (tag3), then match on any threads using completion address; if match exists, on next clock cycle (h4), capture completion byte count (h2); compare completion byte count to requestor byte count

51

#6 Feature-Set Coverage

Answers the Qs

Am I done yet Have I tested everything Are these tests efficient Coverage tasks monitor system input and responses Coverage equations determine coverage level

Coverage models

Feature Set Test Coverage

# Verification Cycles

52

TestWizard Automates Adaptive Testing

Constrained random transaction generator Functional Coverage Monitor Result Checker Design Model Under Test

Set overall and incremental coverage criteria to DYNAMICALLY select random modes and control total run length

53

Coverage Example
Profile router workload by profiling output check coverage
// Setup transaction packet verifier profiler initial begin $tb_list_append_values(verify_packet_LH, "verify_ip_icmp_timeout_msg_latency", "verify_ip6_icmp_timeout_msg_latency", "verify_ip6_flow_latency", "verify_mpls_stack_forwarding", "verify_ldp_map_request"); $tb_list_append_values(verify_packet_H, `verify_ip_icmp, `verify_ip6_icmp, `verify_ip6_flow, `verify_mpls, `verify_ldp); $tb_profile_create(verify_packet_PH, verify_packet, verify_packet_H, verify_packet_LH); end /* Update transaction packet verifier profiler */ always @(update_profile) begin $tb_update_profile(top.mon0.verify_packet_PH, top.mon1.verify_packet_PH, top.mon2.verify_packet_PH, top.mon3.verify_packet_PH); test_coverage0 = 25 * $tb_profile_coverage(top.mon0.verify_packet_PH); test_coverage1 = 25 * $tb_profile_coverage(top.mon1.verify_packet_PH); test_coverage2 = 25 * $tb_profile_coverage(top.mon2.verify_packet_PH); test_coverage3 = 25 * $tb_profile_coverage(top.mon3.verify_packet_PH); test_coverage = test_coverage0 + test_coverage1 + test_coverage2 + test_coverage3; if (verbose) $display("test_coverage=%3.3f", test_coverage); end

54

Coverage Example Contd

top.mon0.verify_packet, top.mon0.verify_packet_PH ("verify_ip_icmp_timeout_msg_latency": 32'sh00000001) = 0 : 0.00 ("verify_ip6_icmp_timeout_msg_latency": 32'sh00000002) = 0 : 0.00 ("verify_ip6_flow_latency": 32'sh00000003) = 4 : 57.14 ("verify_mpls_stack_forwarding": 32'sh00000004) = 3 : 42.86 ("verify_ldp_map_request": 32'sh00000005) = 0 : 0.00 (default) = 13 : 185.71

Question: Have I tested every router workload scenario? Measured Answer: Egress packets on port 0 checked for 2 out of 5 workload scenarios

55

#7 TestWizard Enables C/C++ HW-SW Co-verification

VCI API - The Transport Layer

Simulator independent write code that runs on all simualtors regardless of (PLI/VPI, FLI/VHPI) Supports both client-server and PLI-based programming models

Verilog and sequential C processes run concurrency

Supports both sockets and shared memory IPC

Leverage diagnostics programs and protocol stacks to generate test content VCI API features

Remote Verilog and C/Perl task calls Remote C/Perl routine call backs Synchronization primitives Verilog-C/Perl variable and record accesses Shared Verilog-C/Perl semaphores

Fully deterministic simulation

56

Product Roadmap
Core Technology
Guided Search Insight - Automatic functional test generation Intelligent Random TestWizard - VHDL, Perl - Advanced Assertions - Transaction db - ModelSim TestWizard - Sugar

TestWizard - Verilog/C - VCS, NC-Sim

Directed

June01

02

03

57

Insight - Next Generation of Testing

When directed and random testing runs out of gas

Random test runaway Overlooked/missed tests and test parameters

% Effort

System-level testcases become exponentially more complex to develop and debug

% Bugs Found

58

Insight - Auto-Directed Test Generation DAC 2003 Introduction!

Industrys 1st Guided Search testbench tool

Uses the same random test generator to deterministically solve for test parameter values that satisfy coverage requirements using

Multi-variable filter constraints Enumerated test parameter value sets Temporal properties (assertions and assumptions)

Generates incremental feature-set coverage reports Outputs test/transaction sequencing to drive regression on all the simulation platforms
Fusion Simulation Technology
Verilog Logic Simulation Symbolic Simulation Temporal SAT Solver

59

Insight Example

Symbolic Simulation

Random variable Assertion/Property Coverage Monitor

SAT Target

60

Insight Example

Insight solves for missed coverage points Egress result checker (one per port) Ingress packet traffic monitor (one per port) For egress monitor (port 0), generates 3 directed tests using existing testbench and assertions (design properties) Outputs directed test sequencer for regression testing

top.mon0.verify_packet, top.mon0.verify_packet_PH ("verify_ip_icmp_timeout_msg_latency": 32'sh00000001) = 0 : 0.00 ("verify_ip6_icmp_timeout_msg_latency": 32'sh00000002) = 0 : 0.00 ("verify_ip6_flow_latency": 32'sh00000003) = 4 : 57.14 ("verify_mpls_stack_forwarding": 32'sh00000004) = 3 : 42.86 ("verify_ldp_map_request": 32'sh00000005) = 0 : 0.00 (default) = 13 : 185.71

61

Insight Example

Symbolic simulation

Seed environment

Symbolic simulation starts at time 113,000 ns About 2X packet time + interval time (i.e. 1000 clock cycles)

Symbolic trace period

SAT

SAT targets

Egress ports result checker coverage monitors Ingress packet traffic monitors Generate packet from ingress port with route to egress port 0 with following parameters

Target #1, Result checker, port 0 (verify_ip_icmp)

Protocol: IPV4 TTL field = 1 or 0 Other random parameters are selected randomly

62

Conclusion

TestWizard is a robust transaction-level verification solution

Models complex transactions Generates directed and random transaction-level tests Supports powerful transaction database for result validation of complex transaction processing Assertions provide powerful and reusable protocol verification Validates feature set coverage of system operation and port traffic

Insight using SAT solver is 1st coveragedriven, auto-directed functional test generator Delivers 3X better productivity with up to 10X less verification code Improves design integrity with advanced and measurable methods Builds on existing simulators, HDLs, and methods you already use today!

63