The Importance of Re-creating In-the-

Wild Infection Conditions for Testing

Multi-Layered Security Products
Mark Kennedy
May 15th, 2007
Overview

1 Current Trends

2 Traditional Static Analysis

3 Proactive Static Analysis

4 Dynamic Analysis

5 Lab Bias

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
2
Problem Statement

• Current testing methods only exercise a portion of

security suites
– Heavily geared toward static file scanning
• Signatures
• Packers
• Emulators

• New types of Security Suites require new types of testing

– Multiple layers protection
– Existing testing methods test only a portion of these solutions

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
3
Current Trends

Types and Techniques Motivations and Payloads

• Obfuscation Techniques • Yesterday’s Threats
– Polymorphism – Spreading
– Metamorphism – Fame (infamy)
• Making the news
– Packed Variant
– Vandalism
– In Memory only Threats (no on disk
• Current Threats
footprint)
– Monetary gain
• Yesterday’s Threats • Bancos
– File Infectors • Identity theft

– Mass Mailing Worms – Long lasting control of the machine

– High value assets of specific machines
• VB Script
• SMTP Mass Mailers
• Current Threats
– Non Self Replicating
– Targeted Attacks
• Threats created for a specific target
– File Infectors and Worms decline
The Importance of Re-creating In-the-wild Infection Conditions for
Testing Multi-Layered Security Products
4
Traditional Testing Method

• Primarily Static Analysis

• Large directory of Zoo and ITW samples
• Extensions modified to prevent accidental execution
• Names changed to indicate threat or family

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
5
Traditional Testing Method

• Pros for Traditional Static Analysis

– Fast
• Helps meet tight deadlines
– Well understood
– Large existing collections
• Cons for Traditional Static Analysis
– Highly dependent on signatures
– Limited heuristics due to threat not actually executing on a live
system
– Vulnerable to obfuscation
– Limited effectiveness to truly new threats

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
6
Proactive Static Analysis

• Tested using Traditional Testing Method

– Freeze Virus signatures
– Rollback Virus signatures
• Windows emulators
– NOD32
• Sand Box Emulators
– BitDefender
– Norman Sandbox

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
7
Proactive Static Analysis

• Pros
– Detect threats prior to execution
– Detect threats without signatures
– Can bypass some obfuscation techniques
• Cons
– Performance intensive
– Vulnerable to sophisticated obfuscation techniques
• Obfuscators which make use of obscure APIs cannot be emulated
• Obfuscators which make use of obscure instructions can fool them
• Malcode can detect the emulator and change its behavior
• Threat could require a minimum number of executions or time prior to
becoming active

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
8
Results:

• Current testing methods are becoming less meaningful

– Only testing a portion of the Security Suite
– Individual results are accurate, but do not fully reflect the true
customer experience
• Reliability
– Static testing has become unreliable due to the increased
dynamic nature of malware
• Bottom line: Current tests are not producing as
Customer-relevant results as they could

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
9
Multi-Layered Security Products

• Defense in Depth
– Firewall
– Host based Intrusion Prevention
– Buffer Overflow Protection / Browser Exploit Protection
– Real-time file scanning
– Shields
– Behavior Blocking

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
10
My Only Marketing Slide (I promise)
Symantec Client Layered Protection Architecture

Zero Day Threats Malware & Spyware

Network Filtering
“Block threats before they impact the client”

Behavior Blocking
“Police execution activity”

Storage Filtering
“Don’t let threats persist!”

OS & Application Vulnerabilities Targeted Attacks & Insider Threats

The Importance of Re-creating In-the-wild Infection Conditions for

Page 11
Testing Multi-Layered Security Products
A Word about Success

• Correct Decision making

– Blocks threat at earliest possible point
– Low False Positive rate
• Automatic decision making
– No prompting/asking for permission
– Most users are not qualified to answer correctly
– May become fatigued
– Turn solution off

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
12
You All Remember This

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
13
Defense in Depth: Firewall

• First line of defense

• Inbound
– Prevents threats from getting onto the machine by:
• Blocking known C&C ports
• Blocking ports used by non-essential services e.g. RPC

• Outbound:
– If threats cannot communicate their damage can be limited
– Application control. Only allow known, authorized applications.

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
14
Defense in Depth: Host Based IPS

• Analysis of network blocks

– Blocks malicious behavior
– Lets good behavior through
• Detect and block known Command and Control
sequences
– Outbound
– Inbound
• Detect incoming vulnerability exploit attacks
– Known signatures
– Generic exploit signatures
• A generic signature can block an entire family

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
15
Defense in Depth: Buffer Overflow
Protection / Browser Exploit Protection
• Protect against Drive-by Downloads
– One of the most popular vectors for malware to get on the
machine.
– Any website is vulnerable, even trusted ones! Therefore any user
can be infected, even if they only visit trusted websites.
• Prevents exploits in malicious HTML, VML etc.
• Detect buffer overflows in Browser script
• Detect abuse of Browser ActiveX objects
• BID 22680 (http://www.securityfocus.com/bid/22680)
– Microsoft Internet Explorer OnUnload Javascript Browser
Entrapment Vulnerability

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
16
Defense in Depth: Real-time File
Scanning
• Scans files when created or accessed
• Known signature detection
• Static Heuristic analysis
• Can analyze file prior to any access

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
17
Defense in Depth: Shields

• Monitor known hook points in OS

– Can look for suspicious hook points
– Can detect “over” hooking
• Monitor interactions with other processes on the system
– Detect injection, both direct and through Windows Hooks
– Detect attempts to terminate security processes
• Monitor tampering with security settings
– Attempts to disable firewall
– Attempts to add self to firewall exceptions
• Monitor tampering with HOSTS file

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
18
Defense in Depth: Behavior Blocking

• Closely related to Shields

• Can monitor how executables arrive on system
• Can correlate actions across numerous shield points
• Can detect collaboration between multiple processes
• Have a holistic view of system and interactions
• Has the context necessary to make correct decisions

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
19
An Analogy: Automobile Safety

• Past
– Safety was defined by seat belts
– Tests checked seat belts in isolation
• Current
– Auto safety is a system
• Anti-lock brakes (ABS) Is it fair to say one car
• Steering stabilization is safer than another
• Crumple zones based only on seat
• Airbags (driver, passenger, side) belts?
• Seat belts

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
20
Scoring Gradient: File Based Threat
Content never
Never impact
reaches box

Impact, but no
damage (bumper)

Impact, but no
Never executes
injuries

Executes but
Minor injuries,
cannot
victims walk away
communicate

Communicates but
Major injuries, but
is automatically
survive
removed

Communicates but
Some Fatalities is removed by
definitions

Fatalities, car Communicates and

explodes, kills is never detected /
bystanders cannot be removed
The Importance of Re-creating In-the-wild Infection Conditions for
Testing Multi-Layered Security Products
21
Detractions

• Blocks which require user interaction should score lower

– Asking the user to make decisions is problematic
• Blocks which require updates should score lower
– Effectiveness subject to delays
• False positives should score lower
– User will lose confidence
– May impact productivity

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
22
This All Leads To…

Dynamic Testing: Testing real threats on

real machines

Other Industries have adopted

• Auto industry stages real crashes with real cars
• Airline industry stages real crashes with real
airplanes

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
23
Dynamic Testing

• Running real threats on real machines

– This is the acid test
– This is what matters to customers
• Running on real internet
– Many new threats need to phone home, or make contact in some
way
– Many of today’s threats are primarily a threat to the machine they
are running on, not to others (at least initially)
• Retrieving information off the test machine does no harm
• Only threats like spam bots which become active would be an issue, and that
can be mitigated
– Some threats are dangerous, so you must know

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
24
Dynamic Testing (continued)

• Introduction vector and mode of execution important

– If a threat arrives from email and expects to be launched as an
attachment, launching it another way may change its behavioral
profile
– If a threat arrives via a browser exploit, then it should be created
and launched by the browser
– The firewall must be configured just like the customer would for
their environment
• In a home network environment, most customers put machines on their home
network into the trusted zone.
• This would automatically open up ports that are normally closed by the firewall.
• Any machine that is infected on that network could infect this machine.

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
25
Discreet Dynamic Testing

• Isolate proactive portions of a product

• Prevent signature update
– Side effect: This may prevent product update
• Detections likely to have generic names
– Bloodhound
– Variant
– Exploit
– Newmalware
– Unknown

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
26
Dynamic Testing: Benefits

• Lab results better match real world

– Understand Lab Bias
– Take steps to limit it
• Greater Credibility
– Static testing is not as accurate a reflection of user experience
• Customer relevant results
• System testing methodology
– Legacy testing methods have inherent bias towards signatures
that leads to skewed results
– As the threat landscape has evolved, and the security suites have
evolved, so too must the testing methodology

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
27
Lab Biases

• Platform
• Method of introduction
• Method of invocation
• Internet connectivity
• Definition Rollback or freeze

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
28
Lab Biases: Platform

• VMWare and Virtual PC

– Threats may detect that they are executing in a virtual
environment
– Once detected, they may modify their behavior
– Sufficient Resources required to run
• If threat cannot perform escalation, or exceeds resources then the threat may
not function
• OS Revision and Patch Level
– Some threats may rely on unpatched vulnerabilities to operate
– Threat may not run, or may not exhibit malicious behavior under
certain circumstances
• Open ports
• Installed components

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
29
Lab Bias: Method of Introduction

• Circumstances by which a threat is introduced to a

system may be important
• Some Portals may be more trusted than others
– A Portal is way to introduce software
• Email
• Browser
• CD
• USB key
– Some are more trusted
• CD
– Than others
• Email
• Browser

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
30
Lab Bias: Method of Invocation

• Automatic vs. manual vs. very manual

– Automatic
• Drive-by download
• Downloader
– Manual
• Email attachment
• Double-click
– Very manual
• Command prompt, navigate, run

• These influence the behavioral score

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
31
Lab Bias: Internet Connectivity

• Many threats need to phone home

• Establish connection for Command and Control
• Establish connection for content delivery

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
32
Lab Bias: Definition Rollback or Freeze

• Tests some aspect of heuristic/behavior detection

• Artificial state that does not match customer experience
• Can inadvertently roll back heuristic/behavioral
componentry
• Can create mismatch errors should components presume
minimum version of definitions

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
33
Dynamic Testing: “Do”s

• Configure machines to natural conditions

– Test with unpatched OS
– Test with default security features of suite enabled
• Pay attention to threat injection vector
– Email borne threats should be tested from email
– Browser borne threats should be tested using the browser
• If arrive from exploit, construct an exploit

• Pay attention to invocation

– If a threat needs to run twice, once to “install” and once to act,
test it that way
• Use as much “real” internet as is safe
– If a threat does not affect other machines, give it freer reign

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
34
Dynamic Testing: “Don’t”s

• Just scan the file and conclude effectiveness

– Many other layers may provide detection
• Launch the threats manually
– Particularly from the desktop
• Publish tests without publishing criteria
– Important to understand what the data means
• Publish tests without publishing methodology
– Important to understand how the data was calculated

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
35
Summary

• Threats have changed

• Testing methodology must also change
– Better simulate real world conditions
– Actively execute threats
• Need objective method for comparing
• Not an easy problem to solve
– However, it is an important problem that must be solved

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
36
Questions?

The Importance of Re-creating In-the-wild Infection Conditions for

Testing Multi-Layered Security Products
37
 Thank You!

Mark Kennedy
Mark_Kennedy@Symantec.com
310-449-4263

Copyright © 2007 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered
trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their
respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information
in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is
subject to change without notice. The Importance of Re-creating In-the-wild Infection Conditions for
Testing Multi-Layered Security Products
38