Troubleshooting Firewalls

BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved.
14470_04_2008_c3
Cisco Public
Troubleshooting Firewalls
BRKSEC-3020
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Note: Cisco IOS firewall is covered in SEC-3000 (Troubleshooting Cisco IOS Security Features) and will not be covered in this presentation
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
Packet Flow
Cisco Public
Understanding the Packet Flow

To effectively troubleshoot a problem, one must first understand the packet path through the network Attempt to isolate the problem down to a single device Then perform a systematic walk of the packet path through the device to determine where the problem could be For problems relating to the Cisco ASA/PIX/FWSM, always
Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol Determine the interfaces through which the flow passes
Note: All firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both
Example Flow
Flow
SRC IP: 10.1.1.9 DST IP: 198.133.219.25 SRC Port: 11030 DST Port: 80 Protocol: TCP
Interfaces
Source: Inside Destination: Outside
Client: 10.1.1.9
In
Servers
Packet Flow
Eng
er rtn Pa
si de
DM
Accounting
Ho st in g
With the Flow Defined, Examination of Configuration Issues Boils Down to Just the Two Interfaces: Inside and Outside
Server: 198.133.219.25
Cisco Public
Outside
Understanding the Packet Flow

Once the device and flow have been identified, walk the path of the packet through the device The packet path through the firewall is illustrated in the next several slides For troubleshooting, pay careful attention to where the packet can be dropped in the decision-making process
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
Packet Processing Flow Diagram

The diagram below will be referenced on the following slides; it is shown here enlarged for reference
Cisco Public
Packet Processing: Ingress Interface
Packet arrives on ingress interface Input counters incremented Software input queue is an indicator of load No buffers indicates packet drops, typically due to bursty traffic
ASA-5540# show interface gb-ethernet1 interface gb-ethernet1 "inside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 IP address 10.1.1.1, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 5912749 packets input, 377701207 bytes, 0 no buffer Received 29519 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0)
Packet Processing: Locate Connection
Check first for existing connection If connection exists, flow is matched; bypass ACL check If no existing connection
TCP non-SYN packet, drop and log TCP SYN or UDP packet, pass to ACL checks Established Connection:
ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
Syslog Because of No Connection, and Non-SYN Packet:

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
10
Packet Processing: ACL Check
First packet in flow is processed through interface ACLs ACLs are first match First packet in flow matches ACE, incrementing hit count by one Denied packets are dropped and logged
Packet Permitted by ACL:
ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
Syslog When Packet Is Denied by ACL:

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"
11
Packet Processing: Match Translation
First packet in flow must match a translation rule* A quick route lookup is done only to determine egress interface Translation rule can be to NAT, or not to NAT NAT order of operations dictates what happens with overlapping translation rules Once translation rule is matched, connection is created Translation Exists:
ASA-5540# show xlate debug NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00
Syslogs When No Translation Rule Found: (305005No NAT; 305006No Global)

ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst outside:198.133.219.25/80 ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst outside:198.133.219.25/80
12
Translation and NAT Order of Operations
1. nat 0 access-list (nat-exempt) 2. Match existing xlates 3. Match static commands (Cisco ASA/PIX first match; FWSM best match)
Static NAT with and without access-list Static PAT with and without access-list
First Match
4. Match nat commands

nat <id> access-list (first match) nat <id> <address> <mask> (best match) If the ID is 0, create an identity xlate Use global pool for dynamic NAT Use global pool for dynamic PAT
Cisco Public
13
Packet Processing: Inspections/Sec Checks
Inspections are applied to ensure protocol compliance (Optional) Customized AIC inspections NAT embedded IPs in payload Additional security checks are applied to the packet (Optional) Packets passed to Content Security and Control (CSC) Module
Syslog from Packets Denied by Security Check:
ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP
14
Packet Processing: NAT IP Header
Translate the IP address in the IP header Translate the port if performing PAT Update checksums (Optional) Following the above, pass packet to IPS (AIP) module
Cisco Public
15
Packet Processing: Egress Interface
Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet) Egress interface is determined first by translation rules If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface Example:
Inside
172.16.0.0/16
Outside DMZ
172.16.12.0/24 172.16.12.4
Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

static (inside,outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0 static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0
Cisco Public
16
Packet Processing: L3 Route Lookup
Once on egress interface, an interface route lookup is performed Only routes pointing out the egress interface are eligible Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface
Syslog from Packet on Egress Interface with No Route Pointing Out Interface:
ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9
Cisco Public
17
Packet Processing: L2 Address Lookup
Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed Layer 2 rewrite of MAC header If Layer 2 resolution failsno syslog show arp will not display an entry for the L3 next hop debug arp will indicate if we are not receiving an ARP reply
Cisco Public
18
Packet Processing: Transmit Packet
Packet is transmitted on wire Interface counters will increment on interface Output hardware and software queues indicate buffering at driver level, interface is busy
ASA-5540# show interface gb-ethernet0 interface gb-ethernet0 "outside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c IP address 172.18.124.64, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 3529518 packets input, 337798466 bytes, 0 no buffer Received 32277 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 5585431 packets output, 359059032 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/2) software (0/0)
19
Agenda
Cisco Public
20
10
Cisco ASA/PIXUnderstanding the Architecture

Cisco ASA/PIX platforms process all packets in software (via the central CPU)
All packets are processed first inusually also first out
No software limits on the number of ACEs (rules) that can be configured.

Each ACE takes a minimum of 212 bytes of RAM.
Cisco ASA platforms have software imposed connection limits; Cisco PIX platforms do not (bound by RAM)
Cisco Public
21
FWSMUnderstanding the Architecture

FWSM Process Most Packets in Hardware, with Some Packets Needing to be Processed in Softwarevia the Control Point (CP) Packets processed in hardware have zero impact on CPU Similarly, if the CPU is pegged at 100%, this has zero impact on packets processed in hardware
Cisco Public
22
11
FWSM Architectural Overview

Control Point (CP) Central CPU
Software Hardware
Session Manager NP 3
Control Point ACL Compilation, Fixups, Syslog, AAA in Software Session Manager Session Establishment and Teardown, AAA Cache, ACLs
Fast Path NP 1
Fast Path NP 2
FWSM
Fast Path Flow Identification, Security Checks and NAT in Hardware
C6K Backplane Interface
Cisco Public
23
FWSMACL Rule Limits

ACL rule limits are about the only hardware limit users run into In multimode, ACL resources are divided in 13 equal partitions (12 active, one backup)
If you have less than 12 contexts, wasted reserved space
FWSM 2.3 introduced

resource acl-partitionset the number of ACL partitions allocate-acl-partitionassigns a context to a specific partition
FWSM 3.2 introduced

resource-ruleallows further customization of a partition
FWSM 4.0 introduced

resource partition customize the size of individual partitions
* See the Appendix for examples of each of these commands

Cisco Public
24
12
FWSMHardware Limits
FWSM has several hardware limits that should be considered in your network design Limits are hard set, but vary based on single or multimode Some limits include:
2.3 (Multimode) ACEs AAA Rules Global Statements Static NAT Statements Policy NAT ACEs NAT Translations Connections Route table entries Fixup/Inspect Rules 56,627 (9,704) 3,942 (606) 1K (1K) 2K (2K) 3,942 (606) 256K (256K) 999,990 (999,990) 32K (32K) 32 (32 per)
Increase over 2.3 Increase over 3.1
3.1 (Multimode) 72,806 (11,200) 6,451 (992) 4K (4K) 2K (2K) 1,843 (283) 256K (256K) 999,990 (999,990) 32K (32K) 4147 (1,417)
4.0 (Multimode) 100,567 (14,801) 8,744 (1,345) 4K (4K) 2K (2K) 2,498 (384) 256K (256K) 999,990 (999,990) 32K (32K) 5621 (1,537) 3747 (576)
3.2 / 4.0 Configurable X X
X X
25
Filter Statements 3942 (606) 2764 (425) *Complete list in FWSM docs, Appendix A (Specifications)
Classifier in Multimode
FWSM has a single MAC address for all interfaces Cisco ASA/PIX has single MAC for shared interfaces (physical interfaces have unique MACs)
Cisco ASA/PIX 7.2 introduces an option to change this
When the firewall receives a packet, it must classify it to determine where to send the packet Packets are classified based on the following
Unique ingress interface/VLAN Packets destination IP matches a global IP
Cisco Public
26
13
Example Inbound traffic is classified to context CTX3, based on the global IP in the static
FWSM Inside 10.1.1.2 Inside VLAN 5 10.1.2.2 Inside VLAN 6 10.1.3.2 static (inside,outside) 10.14.3.89 10.1.3.2
DST IP VLAN 310.14.3.x CTX1 .1 10.14.3.89
SRC IP 192.168.5.4
VLAN 4
Inbound Packet Outside MSFC
CTX2
.2
CTX3
.3
Shared Interface
27
If the firewall is unable to classify a packet, the following syslog message is generated in the Admin context*
%FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 10.14.3.25/80
*Added to FWSM 3.1

28
14
Agenda
Cisco Public
29
Failover Basics
Active/standby vs. primary/secondary Serial vs. LAN failover Stateful failover (optional) A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall Both firewalls swap MAC and IP addresses when a failover occurs Level 1 syslogs will give reason of failover
Internet
Stateful LAN/Serial Secondary (Standby) Primary (Active)
Corp
Cisco Public
30
15
Verifying Failover Configuration

PIX(config)# show failover Failover On Cable status: N/A - LAN-based failover enabled Failover unit Primary Failover LAN Interface: Failover Ethernet5 (up) Unit Poll frequency 1 seconds, holdtime 3 seconds Interface Poll frequency 5 seconds, holdtime 25 seconds Monitored Interfaces 2 of 250 maximum Version: Ours 7.2(3), Mate 7.2(2) Last Failover at: 18:30:43 UTC Apr 12 2007 This host: Primary - Active Active time: 5371 (sec) Interface outside (10.36.8.36): Normal Interface inside (10.5.5.144): Normal Other host: Secondary - Standby Ready Active time: 0 (sec) Interface outside (10.36.8.37): Normal Interface inside (10.5.5.145): Normal Stateful Failover Logical Update Statistics Link : Failover Ethernet5 (up) Stateful Obj xmit xerr General 86 0 sys cmd 74 0
Interface Monitoring
rcv 73 73
rerr 0 0
Cisco Public
31
What Triggers a Failover?

Power loss/reload (this includes crashes) on the active firewall SSM interface/module failure The standby becoming healthier than the active firewall
Cisco Public
32
16
What Triggers a Failover?

Two consecutive hello messages missed on any monitored interface forces the interface into testing mode Both units first verify the link status on the interface Next, both units execute the following tests
Network activity test ARP test Broadcast ping test
The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failed
33
What to Do After a Failover

Always check the syslogs to determine root cause Example: switch port failed on inside interface of active firewall
Syslogs from Primary (Active) Firewall
ASA-4-411002: Line protocol on Interface inside, changed state to down ASA-1-105007: (Primary) Link status Down on interface 1 ASA-1-104002: (Primary) Switching to STNDBYinterface check, mate is healthier
Syslogs from Secondary (Standby) Firewall

ASA-1-104001: (Secondary) Switching to ACTIVEmate want me Active
Cisco Public
34
17

Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover state This information is not saved across reboots
ASA# show failover state This host State Primary Failed Secondary Active Last Failure Reason Ifc Failure Inside: Failed None Date/Time 12:56:00 UTC May 6 2007
Other host -
====Configuration State=== Sync Done ====Communication State=== Mac set
Cisco Public
35
Agenda
Cisco Public
36
18
Troubleshooting Tools
Syslogs Debug commands Show commands Packet capture Packet tracer
Cisco Public
37
Uses of Syslogs
Primary mechanism to record traffic to and through the firewall The best troubleshooting tool available
Archival Purposes Debugging Purposes
Console
Syslog Server Internet
SSH Client
Trap SNMP Server Syslog . Buffered
Cisco Public
38
19
ASA Syslog Level vs. Number of Messages

Number of Messages (SUM) Description Ver. 6.3 Emergencies Alerts Critical Errors Warnings Notifications Informational Debugging 0 41 (41) 21 (62) 74 (136) 56 (192) 21 (213) 95 (308) 15 (323) Ver. 7.0 0 62 (62) 29 (91) 274 (365) 179 (544) 161 (705) 234 (939) 217 (1156) Ver. 7.2 0 77 (77) 35 (112) 334 (446) 267 (713) 206 (919) 302 (1221) 258 (1479) Ver. 8.0 0 78 (78) 49 (127) 361 (488) 280 (768) 216 (984) 335 (1319) 266 (1585) Ver. 8.1 0 87 (87) 50 (137) 363 (500) 281 (781) 218 (999) 337 (1336) 267 (1603)
Log Level 0 1 2 3 4 5 6 7
Cisco Public
39
FWSM Syslog Level vs. Number of Messages

Number of Messages (SUM) Description Ver. 2.3 Emergencies Alerts Critical Errors Warnings Notifications Informational Debugging 0 58 (58) 21 (79) 94 (173) 131 (304) 26 (330) 116 (446) 23 (469) Ver. 3.1 0 67 (67) 29 (96) 305 (401) 194 (595) 167 (762) 245 (1007) 225 (1232) Ver. 3.2 0 67 (67) 29 (96) 306 (402) 196 (598) 169 (767) 248 (1015) 225 (1240) Ver. 4.0 0 67 (67) 29 (96) 318 (414) 199 (613) 178 (791) 255 (1046) 226 (1272)
Log Level 0 1 2 3 4 5 6 7
Cisco Public
40
20
What Are Modifiable Syslog Levels?

[no] logging message <syslog_id> level <level>
Modifiable syslog levels

Allows one to move any syslog message to any level Levels
0Emergency 1Alert 2Critical 3Errors 4Warnings 5Notifications 6Informational 7Debugging
Problem
You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level 7 (debug)
%PIX-7-111009: User johndoe executed cmd: show run
The problem is we dont want to log all 1602 other syslogs that are generated at debug level
41
How to Create Modifiable Syslog Levels

Solution
[no] logging message <syslog_id> level <level>
Lower syslog message 111009 to level 3 (error)

ASA(config)# logging message 111009 level 3
Or
ASA(config)# logging message 111009 level error
Now our syslog looks as follows

%ASA-3-111009: User johndoe executed cmd: show run
To restore the default syslog level

ASA(config)# no logging message 111009 level error
Or
ASA(config)# logging message 111009 level 7
Cisco Public
42
21
Debug Commands
1. Debugs should not be the first choice to troubleshoot a problem 2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution 3. Debugs are not conditional* 4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug
* Crypto Conditional Debugging was added to Cisco ASA/PIX 8.0

43
Debug ICMP Trace

Internet
http://www.cisco.com
Valuable tool used to troubleshoot connectivity issues Provides interface and translation information to quickly determine flow Echo-replys must be explicitly permitted through ACL, or ICMP inspection must be enabled
Example debug icmp trace output
ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22 ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2
44
22
Logging Debugs to Syslog

Problem
Log only debug output to syslog
Solution
Create a logging list with only syslog ID 711001 Enable debug output to syslogs Log on the logging list
ASA(config)# logging list Networkers message 711001 ASA(config)# logging debug-trace ASA(config)# logging trap Networkers
Cisco Public
45
Show Output Filters

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Use output filters to filter the output of show command to only the information you want to see To use them, at the end of show <Command>, use the pipe character | followed by
begin include exclude grep grep v
Start displaying the output beginning at the first match of the RegEx, and continue to display the remaining output Display any line that matches the RegEx Display any line that does not match the RegEx Same as include Same as exclude
Cisco Public
46
23
Example: Show Output Filters

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Examples
Display the interface stats starting with the inside interface
show interface | begin inside
Display the access-list entries that contain address 10.1.1.5

show access-list | grep 10.1.1.5
Display the config, except for the access-lists

show run | exclude access-list
Display only access-list entries that have non-zero hitcounts

show access-list | grep v hitcnt=0
Display a count of the number of connections each host has

show local-host | include host|count/limit Note: You must include a space on either side of the pipe for the command to be accepted; also, trailing spaces are counted
47
Show CPU Usage

Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled The show cpu usage command displays the CPU over time as a running average
pixfirewall# show cpu usage CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%
*First introduced in Cisco PIX OS version 6.0(1)/FWSM 1.1(1)

48
24
Show Traffic
The show traffic command displays the traffic received and transmitted out each interface of the firewall
ASA# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec
49
Show Xlate and Show Xlate Debug

The show xlate command displays information about the translations through the firewall You can limit the output to just the local or global IP
ASA# show xlate 2 in use, 2381 most used Global 172.18.124.68 Local 10.1.1.9 PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101) debug Adds interface names, idle and xlate ASA# show xlate debug timeouts 2 in use, 2381 most used Flags: D - DNS, d - dump, I - identity, i - inside, n - no random, o - outside, r - portmap, s - static NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:02:03 timeout 3:00:00 TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024 flags r idle 0:00:08 timeout 0:00:30
50
25
Show Conn and Show Conn Detail

real Interface names added in 7.2(4), 8.0(4)
ASA# show conn 2 in use, 64511 most used TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags
Idle Time, Bytes Transferred
Connection Flags
ASA# show conn detail 2 in use, 64511 most used

Flags: A B E G i k O R R s X -
detail Adds uptime and timeout in 7.2(4), 8.0(4)
awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, outside back connection, F - outside FIN, f - inside FIN, group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, incomplete, J - GTP, j - GTP data, K - GTP t3-response Skinny media, M - SMTP data, m - SIP media, n - GUP outbound data, P - inside back connection, q - SQL*Net data, outside acknowledged FIN, UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, inspected by service module
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101, flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123, flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431
51
ExampleConnection Build Up
1. Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags saA 2. The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show A 3. The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered up (U flag) 4. The outside device sends the first data packet; the connection is updated and an I is added to the flags to indicate the firewall received Inbound data on that connection 5. Finally, the inside device has sent a data packet and the connection is updated to include the O flag
1 5 3 SYN+ACK Data SYN ACK Connection Flags 42
UI UIO s A UaA
Inside Client Outside Server
52
Cisco Public
26
ExampleConnection Teardown
1. Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface 2. The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show UfFR 3. The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists
3 1
FIN+ACK ACK FIN
Connection Flags 2
UfFRr UfUfFR
Inside Client Outside Server
53
Cisco Public
Connection FlagsQuick Reference

Outbound Connection Inbound Connection
Cisco Public
54
27
TCP Connection Termination Reasons

If a TCP connection is built through the firewall, it will always have a teardown reason The TCP teardown syslog is logged at level 6 If you are having problems with connections abnormally closing, temporally increase your logging level (or move the syslog down), and check the teardown reason
ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_port duration time bytes number [reason] [(user)]
Cisco Public
55
TCP Connection Termination Reasons Quick Reference

Reason Conn-Timeout Deny Terminate Failover Primary Closed FIN Timeout Flow Closed by Inspection Flow Terminated by IPS Flow Reset by IPS Flow Terminated by TCP Intercept Invalid SYN Idle Timeout IPS Fail-Close SYN Control
Description Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout Flow Was Terminated by Application Inspection The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout Flow Was Terminated by Inspection Feature Flow Was Terminated by IPS Flow Was Reset by IPS Flow Was Terminated by TCP Intercept SYN Packet Not Valid Connection Timed Out Because It Was Idle Longer Than the Timeout Value Flow Was Terminated Due to IPS Card Down Back Channel Initiation from Wrong Side
Cisco Public
56
28
TCP Connection Termination Reasons Quick Reference (Cont.)

Reason SYN Timeout TCP Bad Retransmission TCP Fins TCP Invalid SYN TCP Reset-I TCP Reset-O Description Force Termination After Two Minutes Awaiting Three-Way Handshake Completion Connection Terminated Because of Bad TCP Retransmission Normal Close Down Sequence Invalid TCP SYN Packet TCP Reset Was Sent From the Inside Host TCP Reset Was Sent From the Outside Host
TCP Segment Partial Overlap Detected a Partially Overlapping Segment TCP Unexpected Window Size Variation Tunnel Has Been Torn Down Uauth Deny Unknown Xlate Clear
Connection Terminated Due to a Variation in the TCP Window Size Flow Terminated Because Tunnel Is Down Connection Denied by URL Filtering Server Catch-All Error User Executed the Clear Xlate Command
Cisco Public
57
show local-host
A local-host entry is created for any IP tracked through the firewall It groups the xlates, connections, and AAA information Very useful for seeing the connections terminating on servers
ASA# show local-host Interface inside: 1131 active, 2042 maximum active, 0 denied local host: <10.1.1.9>, TCP connection count/limit = 1/unlimited TCP embryonic count = 0 TCP intercept watermark = 50 UDP connection count/limit = 0/unlimited AAA: user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10) absolute timeout: 0:05:00 inactivity timeout: 0:00:00 Xlate(s): Global 172.18.124.69 Local 10.1.1.9 Conn(s): TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO
58
29
show service-policy
The show service-policy command is used to quickly see what inspection policies are applied and the packets matching them
ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0 Inspect: ftp, packet 43, drop 0, reset-drop 0 Inspect: h323 h225, packet 0, drop 0, reset-drop 0 Inspect: h323 ras, packet 0, drop 0, reset-drop 0 Inspect: http, packet 562, drop 0, reset-drop 0 Inspect: netbios, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, drop 0, reset-drop 0 Inspect: rtsp, packet 0, drop 0, reset-drop 0 Inspect: skinny, packet 349, drop 0, reset-drop 0 Inspect: esmtp, packet 0, drop 0, reset-drop 0 ... Interface outside: Service-policy: VoIP Class-map: voice_marked Priority: Interface outside: aggregate drop 0, aggregate transmit 349
59
show service-policy flow

Use to determine what policies a given flow will match in the Modular Policy Framework (MPF) Eventually all policies will be in MPF
ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23 Global policy: Service-policy: global_policy Interface outside: Service-policy: outside_policy Class-map: inbound_class Match: access-list telnet_inbound Access rule: permit tcp host 10.1.1.2 host 10.0.0.2 eq telnet Action: Output flow: set connection timeout tcp 0:05:00
Cisco Public
60
30
show asp drop

Packets dropped in the Accelerated Security Path (ASP) will increment a counter Frame drop counters are per packet, flow drops are per flow Some counters have corresponding syslogs
ASA# show asp drop Frame drop: Invalid encapsulation (invalid-encap) Invalid tcp length (invalid-tcp-hdr-length) Invalid udp length (invalid-udp-length) No valid adjacency (no-adjacency) No route to host (no-route) Reverse-path verify failed (rpf-violated) Flow is denied by access rule (acl-drop) First TCP packet not SYN (tcp-not-syn) Bad TCP flags (bad-tcp-flags) TCP option list invalid (tcp-bad-option-list) TCP MSS was too large (tcp-mss-exceeded) Bad TCP Checksum (bad-tcp-cksum) *Drop counters are documented in the CMD Ref, under show asp drop
10897 9382 10 5594 1009 15 25247101 36888 67148 731 10942 893
61
Packet Capture
capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>] [circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>] [match <prot> {host <sip> | <sip> <mask> | any} [eq | lt |gt <port>] {host <dip> | <dip> <mask> | any} [eq | lt | gt <port>]] [real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]
Capture command first introduced in Cisco PIX 6.2; FWSM 2.3; it deprecates the debug packet command 7.2(3) and 8.0(3) added a real-time option ASDM 6.0 adds a capture wizard Capture sniffs packets on an interface that match an ACL, or match line Key steps
Create an ACL that will match interesting traffic Define the capture and bind it to an access-list and interface View the capture on the firewall, or copy it off in .pcap format
62
31
Packet Capture (Cont.)

Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface Capture buffer saved in RAM (default size 512KB) Default is to stop capturing when buffer is full Default packet length is 1518 bytes Copy captures off via TFTP or HTTPS
Capture In Inside Capture Out Outside
Cisco Public
63
Where Packets Are Captured in Packet Flow
Ingress Packets Captured
Egress Packets Captured
Packets are captured at the first and last points they can be in the flow Ingress packets are captured before any packet processing has been done on them Egress packets are captured after all processing (excluding L2 source MAC rewrite)
64
32
Capture Command: Example

Problem: User on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (198.133.219.25); the user is getting PATed to 192.168.2.2
Capture In Inside 10.1.3.2 Capture Out www.cisco.com Outside
Internet
198.133.219.25
10.1.3.2
192.168.2.2
Step 1: Create ACL for Both Inside and Outside Interface Step 2: Create Captures on Both Inside and Outside Interface Step 3: Have Inside User Access www.cisco.com Step 4: Copy the Captures Off to a TFTP Server Step 5: Analyze Captures with Sniffer Program
65
Capture Command: Example

Step 1: Create ACL for both inside and outside interface
! Outside Capture ACL Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80 Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2 ! Inside Capture ACL Access-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80 Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2
Step 2: Create captures on both inside and outside interface

capture out access-list 100 interface outside packet-length 1518 capture in access-list 101 interface inside packet-length 1518
Step 3: Have inside user access www.cisco.com Step 4: Copy the captures off to a TFTP server
! ASA ver 7.0+ / FWSM 3.0+ copy capture copy /pcap capture:out tftp://10.1.3.5/out.pcap copy /pcap capture:in tftp://10.1.3.5/in.pcap ! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcap copy capture:in tftp://10.1.3.5/in.pcap pcap
Or copy using https:

https://<FW_IP>/capture/out/pcap
66
33
Packet Capture: Example

Step 5: Analyze captures with sniffer program
Outside CAP
Inside CAP
Outbound SYN, No SYN+ACK
Cisco Public
67
Capturing Packets Dropped by the ASP

Capture all packets dropped by the ASP
ASA# capture drops type asp-drop all
Capture on a specific drop reason

ASA# capture drops type asp-drop invalid-tcp-hdr-length
ASA# capture drop type asp-drop ? acl-drop all bad-crypto bad-ipsec-natt bad-ipsec-prot bad-ipsec-udp bad-tcp-cksum bad-tcp-flags Flow is denied by configured rule All packet drop reasons Bad crypto return in packet Bad IPSEC NATT packet IPSEC not AH or ESP Bad IPSEC UDP packet Bad TCP checksum Bad TCP flags
Cisco Public
68
34
Packet Capture: Limitations on FWSM

Capture functionality is available on the FWSM starting in 2.3
However, only packets processed by the control point could be captured
FWSM 3.1(1) added support to capture packets in hardware

Only ingress packets were captured
Session Manager NP 3
Capture requires an ACL to be applied Capture copies the matched packets in hardware to the control point where they are captured; be careful not to flood the control point with too much traffic
FWSM
FWSM 3.1(5) both ingress and egress transient packets can be captured which flow through hardware
Fast Path Fast Path NP 1 NP 1
69
Packet Tracer
Packet tracer is the future of troubleshooting configuration issues (and many other issues) Introduced in version 7.2 and ASDM 5.2 A packet can be traced by:
Defining the packet characteristics via the CLI Capturing the packets using the trace option
Cisco Public
70
35
Packet Tracer: Overview

A packet tagged with the trace option is injected into the interface, and processed in the data-plane Each action taken on the packet is recorded in the packet itself When the packet reaches the egress interface, or is dropped, it is punted to the control-plane The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration
Cisco Public
71
Packet Tracer: Creating Packet via CLI

From the CLI, define the input interface along with source and destination IPs and ports
packet-tracer input <intf> <proto> <src_ip> <src_port> <dst_ip> <dst_port>
ExampleTrace the flow from inside host 10.1.1.2 to http://www.cisco.com (198.133.219.25)

ASA# packet-tracer input inside tcp 10.1.1.2 1025 198.133.219.25 80
Cisco Public
72
36
Packet Tracer: Example Output

ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80 Phase: 1 Type: FLOW-LOOKUP Subtype: Result: ALLOW Config: Additional Information: Found no matching flow, creating a new flow Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group in in interface inside access-list in extended permit tcp any any eq www Additional Information: Phase: 3 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map match-all inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect http service-policy global_policy global Additional Information:
73
Packet Tracer: Example Output (Cont.)

... Phase: 10 Type: NAT Subtype: Result: ALLOW Config: nat (inside) 1 10.1.1.0 255.255.255.0 Additional Information: Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255 ... Phase: 15 Type: ROUTE-LOOKUP Subtype: output and adjacency Result: ALLOW Config: Additional Information: found next-hop 209.165.201.1 using egress ifc outside adjacency Active next-hop mac address 000a.f331.83c0 hits 0 >>>>Packet successfully forwarded to fast path<<<<
Cisco Public
74
37
Packet Tracer: Tracing Captured Packet

Create a capture using the trace option
ASA# capture inside access-list web interface inside trace .
Find the packet in the capture you want traced

ASA# 68 1: 2: 3: 4: 5: show capture inside packets captured 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: ... S S . P .
ack ack ack Ack
Then select that packet to be traced

ASA# show capture inside trace packet-number 4 .
Cisco Public
75
Packet Tracer: ASDM

ASDM includes a nice GUI front-end to the packet tracer tool It is located off the Tools menu Input the packets characteristics in the top half Actions taken on the packet are shown in the bottom half, along with associated config and links back to modify that config entry in ASDM
Cisco Public
76
38
Packet Tracer: ASDM (Screen Shot)
Define Packet
Action Matching Config Link Back to Edit Rule
Final Result
Cisco Public
77
Agenda
Cisco Public
78
39
Case Study
Intermittent Access to Web Server
Cisco Public
79
Case Study: Intermittent Access to Web Server

Problem Most external clients are not able to load companys web page
NATed to 10.1.1.50
HTTP Requests to 192.168.1.50
Internet Web Server

10.1.1.50 ASA-5510
Clients
Cisco Public
80
40
Traffic Spike
Cisco Public
81

show perfmon indicates high number of embryonic connections
ASA-5510# show perfmon PERFMON STATS: Xlates Connections TCP Conns UDP Conns URL Access URL Server Req TCP Fixup TCP Intercept Established Conns TCP Intercept Attempts TCP Embryonic Conns Timeout HTTP Fixup FTP Fixup AAA Authen AAA Author AAA Account VALID CONNS RATE in TCP INTERCEPT: ASA-5510# Current 0/s 2059/s 2059/s 0/s 0/s 0/s 0/s 0/s 0/s 1092/s 0/s 0/s 0/s 0/s 0/s Current N/A Average 0/s 299/s 299/s 0/s 0/s 0/s 0/s 0/s 0/s 4/s 0/s 0/s 0/s 0/s 0/s Average 95.00%
Cisco Public
82
41

Issue show conn to see who is creating the connections
Random Sources Embryonic Conns
ASA-5510# show conn 54764 in use, 54764 most used TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB . . .
Cisco Public
83
Traffic Permitted Connection Count Jumps
SYN Flood Detected
Cisco Public
84
42

Apply TCP Intercept to stop the SYN flood attack
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 ! service-policy interface_policy interface outside
Cisco Public
85
TCP Intercept applied
Few clients represent 50+ % of traffic
Cisco Public
86
43

Apply per-client-max option to limit the number of connections any single client can establish
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 per-client-max 25 ! service-policy interface_policy interface outside
Cisco Public
87
per-client-max
TCP Intercept
Cisco Public
88
44
Attacks Being Mitigated by ASA
Attacks Still Occurring

Cisco Public
89
Case Study
Poor Voice Quality
Cisco Public
90
45
Case Study: Poor Voice Quality

Problem Poor Outbound Voice Quality at SOHO sites
Outbound RTP Stream
100 Mbps ASA-5505
100 Mbps
Cable Modem
2 Mbps
WAN
Cisco Public
91

Solution: Traffic Shaping What is Traffic Shaping, and why is it needed here? Why wont Policing work? Why wont Priority Queuing alone work?
Shape to 2 Mbps
Cable Modem 100 Mbps ASA-5505 100 Mbps
WAN
2 Mbps
Cisco Public
92
46

Configuration Example (Traffic Shaping) Solution Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface.
class-map voice-traffic match dscp af13 ef ! policy-map qos_class_policy class voice-traffic priority ! policy-map qos_outside_policy class class-default shape average 2000000 service-policy qos_class_policy ! service-policy qos_outside_policy interface outside
To view statistics on the operation of the shaper, use the command show service-policy shape
93

Things to keep in mind: Shaping can only be applied to the class class-default Shaping only works in the outbound direction on an interface The shaping value is in bits per second, and must be a multiple of 8000 The shaping policy is applied to all sub-interfaces on a physical interface Not supported on the ASA-5580 platform Not supported in Transparent or Multi-context mode
Cisco Public
94
47
Agenda
Cisco Public
95
Tools
ASDM Output interpreter Online learning modules
Cisco Public
96
48
ASDM
Run as a standalone application using the ADSM Launcher This allows for one-stop access to multiple firewalls ASDM 6.0 adds Upgrade Wizard to upgrade ASA and ASDM software direct from cisco.com ASDM 6.1 works with both ASA 8.1 and 8.0 releases ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases
Cisco Public
97
ASDM Home Page
Device Information
CPU, Memory, Conns/Sec, Interface Traffic
Real-Time Syslogs
Cisco Public
98
49
Using ASDM for Monitoring

Up to Four Different Graphs Can Be Displayed
Great for Monitoring Trends
Cisco Public
99
ASDM: Editing Rules from the Log Viewer
Select Log Entry from Viewer
Right-Click on Message to View or Edit Associated Rule
Cisco Public
100
50
ASDM: Syslogs Explained
Cisco Public
101
ASDM 6.0
Drag-and-drop and inplace editing for simplified policy editing User interface customization with dockable windows and toolbars New Firewall Dashboard that provides at-a-glance status of firewall services Live ACL hitcount in firewall rule table for easy policy auditing
Cisco Public
102
51
Output Interpreter
Linked Off the Technical Support and Documentation Tools and Resources Section on CCO
Great Tool for Catching Configuration Errors
Paste in the show run Output and Hit Submit
Cisco Public
103
Output Interpreter: Example Output
Warning: Unused Statics
Warning: Unapplied Crypto Map
Warning: Invalid Crypto Map
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
104
52
Online Learning Modules on CCO

Great way to learn about new features in the ASA Located on CCO From http://www.cisco.com select: Products and Solutions
Security ASA 5500 Series Adaptive Security Appliances Training resources Online learning modules
Direct link
http://www.cisco.com/en/US/partner/products/ps6120/tsd_ products_support_online_learning_modules_list.html
105
Agenda
Cisco Public
106
53
Cisco ASA/PIX/FWSM Best Practices

Enable ip verify reverse-path on all interfaces Set embryonic and maximum connection counts on static and nat statements; for 7.2.1+ use per-client-max Configure logging to syslog server Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you dont want to see Disable telnet access! Use SSH for management access Enable authentication for management access (console/SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback
Cisco Public
107
Cisco ASA/PIX/FWSM Best Practices

Restrict DMZ access inbound to your internal networks Baseline CPU load, connection counts, xlate counts, and traffic (per interface) Run the latest maintenance release in your train Upgrade major feature trains only when you need new features, or after train has matured
Cisco Public
108
54
Cisco ASA/PIX Release Process

Interim Images 7.0(2.1) 7.0(2.2) 7.0(2.3)
GD
GD
Future 7.0(9) EoL
7.0(2)
7.0(4) 7.1(2)
7.0(5)
7.0(6)
7.0(7)
7.0(8)
Maintenance Trains
Bug Fixes Waterfall Down
7.2(2)
7.2(3)
7.2(4) 8.0(3)
7.2(5) 8.0(4) 8.1(2)
7.0(1)
7.1(1)
7.2(1)
8.0(2)
8.1(1)
8.2(1)
Major Feature Releases Time

109
Cisco PIX - End of Sale / End of Life Milestones
Cisco has Announced the End-of-Sale and End-of-Life Dates for Cisco PIX Security Appliances
End of Sale: July 28, 2008 Last day of sale for software, accessories, and licenses: January 28, 2009 End of Software Maintenance Releases: July 28, 2009 End of Support / End of Life: July 27, 2013
Cisco Public
110
55
Q and A
Cisco Public
111
Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested books I recommend: Cisco ASA: All-in-One Firewall, IPS, and VPN Adaptive Security Appliance Available Onsite at the Cisco Company Store
112
56
Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Dont forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008. Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.
Cisco Public
113
Cisco Public
114
57
Appendix
Lucky You This appendix contains some extra information which you may find useful, but I just didnt have enough time to cover in the lecture Enjoy :-)
Cisco Public
115
Appendix
Release trains Gotchas upgrading Cisco PIX to 7.0 Cisco PIX password recovery Case study
Out of Order Packet Buffering TCP MSS issue Out of memory High CPU
Online Tools (Network Professionals Connection, Bug Toolkit) Information to include when opening a TAC case
116
58
Cisco PIX Release Process

Interim Images 6.0(2.100) 6.0(2.101) 6.0(2.122) Bug Fixes Committed to Multiple Trains CSCdy54228
6.0(2)
6.0(3)
6.0(4) 6.1(2) 6.1(3) 6.1(4) 6.2(2) 6.1(5) 6.2(3) 6.2(4) 6.3(2)
Maintenance Trains
6.3(3)
6.3(4)
6.0(1)
6.1(1)
6.2(1)
6.3(1)
Major Feature Releases Time

117
Cisco PIX/ASA/FWSM Code Base History

In Sync
6.3(1)
Port Features Bug Fixes
Cisco PIX
6.0(1)
Feature Releases 6.1(1) 6.2(1)
Cisco PIX/ASA
7.1(1) 7.2(1)
7.0(1)
FWSM
1.1(1)
Feature Releases
2.2(1)
2.3(1)
3.1(1)
3.2(1)
4.0(1)
SafeHarbor
1.1(2)
1.1(3)
2.3(2)
Maintenance Releases 3.1(2)
3.2(2)
SafeHarbor
3.2(4)
GD
3.1(6)
3.1(10)
Time
118
59
Gotchas in Upgrading to 7.0

Cisco PIX-515s (non-E) and Cisco PIX-535 should be upgraded from monitor mode Upgrading from monitor mode requires you to copy the 7.0 image over twice
Once from monitor mode (to boot 7.0 and format flash) Once after 7.0 is up and flash has been formatted (to save image in flash)
The upgrade process automatically converts your pre-7.0 config to the new 7.0 CLI If there were any errors during the config conversion process, view them by issuing
show startup-config errors
119
Cisco PIX Password Recovery

Password recovery can only be performed by uploading the password recovery utility to the Cisco PIX from monitor mode via TFTP Password recovery will also remove any AAA commands A password recovery utility is created for each major Cisco PIX release (6.1, 6.2, 6.3); however, the utilities are backwards compatible
Example: np63.bin for Cisco PIX 6.3 and prior releases
http://www.cisco.com/warp/public/110/34.shtml
Cisco Public
120
60
Example: Cisco PIX Password Recovery

Example
TFTP Inside monitor> interface 1 monitor> address 14.36.1.88 monitor> file np63.bin 172.18.108.26 14.36.1.88 monitor> gateway 14.36.1.1 monitor> server 172.18.108.26 monitor> tftp tftp np63.bin@172.18.108.26 via 14.36.1.1............... Received 92160 bytes Do you wish to erase the passwords? [yn] y The following lines will be removed from the configuration: enable password 8Ry2YjIyt7RRXU24 encrypted passwd 2KFQnbNIdI.2KYOU encrypted aaa authentication serial console LOCAL aaa authentication telnet console LOCAL aaa authentication ssh console LOCAL aaa authentication enable console LOCAL Do you want to remove the commands listed above from the configuration? [yn] y Passwords and aaa commands have been erased. Rebooting..
121
Case Study
Out-of-order packet buffering
Cisco Public
122
61
Case Study: Out-of-Order Packets

Inspections require ordered packets Packets sent to the SSM (AIP and CSC) require ordered packets Cisco ASA/PIX will buffer up to three packets by default Buffering can be increased on ASA by using the queue-limit option under the tcp-map
Cisco Public
123
Case Study: Out-of-Order Packets

Problem Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows If the out-of-order packet buffer isnt large enough, traffic is dropped and packets must be retransmitted
192.168.1.30 Client Inside Outside Server Packet 10 Packet 11 Packet 12 Packet 13 Packet 14 Packet 15 10.16.9.2
Dropped on Network Buffer Dropped by Firewall
Cisco Public
124
62
Case Study: Out-of-Order Packet Buffering Example

How to detect?
ASA# show asp drop Frame drop: ... TCP packet SEQ past window TCP packet buffer full ... 46331 90943
How to fix?
access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0 ! tcp-map OOO-Buffer queue-limit 6 ! class-map tcp-options match access-list OOB-nets ! policy-map global_policy class tcp-options set connection advanced-options OOO-Buffer ! service-policy global_policy global
125
Case Study: Out-of-Order Packet Buffering Example

How to verify?
ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default ... Class-map: tcp-options Set connection policy: Set connection advanced-options: OOB-Buffer Retransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Out-of-order packets: 2340 No buffer drops : 0
Cisco Public
126
63
Case Study
TCP MSS (Maximum Segment Size)
Cisco Public
127
Case Study: TCP MSS

MSS is the Maximum Segment Sizeor the maximum amount of data that can be sent in a single packet The MSS is set in the SYN packets The device that receives the MSS advertisement cannot send more data in a single packet to the peer than specified by the MSS
Cisco Public
128
64
Case Study: TCP MSS

Problem Some servers have broken TCP stacks and ignore the MSS advertised by the Client The firewall will drop packets that exceed the advertised MSS
192.168.1.30 Client SYN MSS=1380 Inside Outside Server 10.16.9.2
SYN+ACK MSS=1400
DATA=1390
Cisco Public
129
Case Study: TCP MSS Example

How to detect?
ASA# show asp drop Frame drop: TCP MSS was too large 943
%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390
How to fix?
access-list MSS-hosts permit tcp any host 10.16.9.2 ! tcp-map mss-map exceed-mss allow ! class-map mss match access-list MSS-hosts ! policy-map global_policy class mss set connection advanced-options mss-map ! service-policy global_policy global
130
65
Case Study: TCP MSS Example

How to verify?
ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518 ASA# show capture mss-capture 0 packets captured 0 packets shown
How else could you verify?
Cisco Public
131
Case Study
Out of Memory
Cisco Public
132
66
Case Study: Out of Memory

Problem Users are unable to access the Internet No new connections are working All old (long lived) connections continue to work
Step 1: Check the Syslogs
%PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error
Step 2: Check the Amount of Free Memory Available

Hardware: PIX-515E, 64 MB RAM pixfirewall# show memory Free memory: 714696 bytes Used memory: 66394168 bytes ---------------------------Total memory: 67108864 bytes
133

Step 3: What Eats Up Memory (RAM) on the Cisco PIX?
Cisco PIX image (run from RAM) Configuration IPSec database Xlates (translations) Connections A Small Global Pool Is Used, Overloading to a PAT Address What Can Eat Up 64 MB on a Cisco PIX-515E?
Step 4: Lets Check the Translations
pixfirewall# show xlate 251 in use, 258 most used PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276) Global 209.165.201.25 Local 10.1.1.102 PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)
Varied Source IPs

134
67

Step 5: Check the Connections
pixfirewall# show conn 147456 in use, 147456 most used TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA . . .
Q: Why is the connection count so high?

135

Take a Look at the Traffic Load
pixfirewall# show traffic outside: received (in 25.000 secs): 1475 packets 469050 bytes 59 pkts/sec 18762 bytes/sec transmitted (in 25.000 secs): 167619 packets 9654480 bytes 6704 pkts/sec 386179 bytes/sec inside: received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 25.000 secs): 1050 packets 118650 bytes 42 pkts/sec 4746 bytes/sec
Traffic Flow
Vast majority of traffic is coming in the inside interface and going out the outside interface
Inside
Outside
136
68

Step 6: Review What We Know and Take Action
pixfirewall# show conn count 147456 in use, 147456 most used pixfirewall# show xlate count 251 in use, 258 most used
Conn Count Is Very High, but xlate Count Is Low

Many connections per xlate Probably one, or a few hosts, are generating the vast majority of connections Most likely due to a virus on the host(s)
Cisco Public
137

Step 7: Find the Host(s) Generating All the Connections
pixfirewall# show local-host | local host: <10.1.1.131>, TCP connection count/limit UDP connection count/limit local host: <10.1.1.51>, TCP connection count/limit UDP connection count/limit local host: <10.1.1.236>, TCP connection count/limit UDP connection count/limit . . . local host: <10.1.1.99>, TCP connection count/limit UDP connection count/limit include host|count/limit = 0/unlimited = 0/unlimited = 2/unlimited = 0/unlimited = 0/unlimited = 0/unlimited
Only Show Lines That Have the Word host or count/limit in Them
= 146608/unlimited = 0/unlimited
Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections
138
69

Step 8: Now That We Found the Host, Lets Look at the Connections It Is Generating
pixfirewall# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited TCP embryonic count = 146606 Note: All Connections UDP connection count/limit = 0/unlimited Are Embryonic Xlate(s): Global 209.165.201.21 Local 10.1.1.99 Conn(s): TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags TCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags Connections to Random TCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags Destinations on TCP/135 TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags MS Blaster TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags . . .
Cisco Public
saA saA saA saA saA saA saA saA saA saA
139

Cisco PIX provides two methods to limit the number of connections per host
TCP intercept Max connections
Question: Which One Can Be Used Here?
TCP intercept wont help because the source address is valid Limiting the maximum number of connections each internal host can have is the only option
Cisco Public
140
70

Step 9: Limit Infected Host(s) Impact on Network Configure the MAX TCP connections for NATed hosts to be 50
pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0
Note: The local-host must be cleared before the new connection limits are applied
pixfirewall(config)# clear local-host 10.1.1.99 pixfirewall(config)# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, The Infected Host Is TCP connection count/limit = 50/50 TCP embryonic count = 50 Limited to 50 TCP TCP intercept watermark = unlimited Connections UDP connection count/limit = 0/unlimited . . .
Cisco Public
141

Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit
pixfirewall# show conn count 126 in use, 147456 most used pixfirewall# show memory Free memory: 47716152 bytes Used memory: 19392712 bytes ---------------------------Total memory: 67108864 bytes
Things look much better now Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus?
nat (inside) 1 10.1.1.99 255.255.255.255 50 0 nat (inside) 1 0.0.0.0 0.0.0.0 0 0
Cisco Public
142
71
Case Study
High CPU Usage
Cisco Public
143
High CPU Usage on the Cisco PIX

Problem: Cisco PIX CPU Running Very High A quick overview of the show processes command
Number of msec This Process Has Been on the CPU pixfirewall(config)# show processes PC 001eab19 001f00bd 00119abf 003e4425 001e26fb 00300a29 SP 008a5a74 00a28dbc 02d280dc 02d2a26c 0533940c 04c0f504 STATE 00557910 00557910 0055b070 00557dd8 00557d88 00557910 Runtime 0 0 0 74440 6070290 0 SBASE 008a4aec 00a27e44 02d27274 02d28324 05338484 04c0e57c Stack 3628/4096 3832/4096 3688/4096 6936/8192 3684/4096 3944/4096 Process arp_timer FragDBGC dbgtrace Logger 557poll xlate clean The Name of the Process
Hsi Lsi Lwe Lwe Crd Lsi
For more information on the output of the show processes command, see
http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml
144
72

Step 1: Determine What Process Is Eating the CPU Take the difference in output of two show processes over a period of time The following output was a diff of the processes taken one minute apart
Process_Name Logger pix/intf3 557poll i82543_timer i82542_timer
Runtime (msec) 25940 18410 9250 4180 2230
In One Minute, These Processes Account for 44 Seconds of CPU Time ~ 73% The Interface Polling Processes Always Run, and Are Not Counted in the CPU Usage
Cisco Public
145

Step 2: Focus on the Processes with High CPU Time Logging is taking up much of the CPU; lets review what we have configured to log
This Is Cumulative pixfirewall(config)# show log Syslog logging: enabled Since the Cisco PIX Standby logging: disabled Was Last Rebooted Console logging: disabled Monitor logging: disabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 5919412 messages logged Logging to lab 172.18.173.123 History logging: disabled Notice the Change . . . pixfirewall(config)# show log Over a Few Minutes Syslog logging: enabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 6172472 messages logged Logging to lab 172.18.173.123
146
73

Syslog Server Is Controlled by a Different Group
Enable buffered logging to same level as syslog server, and examine the buffered messages
pixfirewall(config)# show log Buffer logging: level warnings, 31527 messages logged Trap logging: level warnings, 6453127 messages logged Logging to lab 172.18.173.123
. . . 400011: 400011: 400011: 400011: 400011: 400011:
Cisco PIXs Interface Address
IDS:2001 IDS:2001 IDS:2001 IDS:2001 IDS:2001 IDS:2001
ICMP ICMP ICMP ICMP ICMP ICMP
unreachable unreachable unreachable unreachable unreachable unreachable
from from from from from from
172.18.173.123 172.18.173.123 172.18.173.123 172.18.173.123 172.18.173.123 172.18.173.123
to to to to to to
14.36.1.88 14.36.1.88 14.36.1.88 14.36.1.88 14.36.1.88 14.36.1.88
on on on on on on
interface interface interface interface interface interface
lab lab lab lab lab lab
Cisco Public
147

Examine IDS Configuration
pixfirewall(config)# show run | grep audit ip audit name IDS info action alarm ip audit interface lab IDS
Syslog service was down on the syslog server ICMP unreachable was generated by syslog server for each syslog message the Cisco PIX sent it Cisco PIXs IDS configuration also logged every ICMP unreachable message, creating the exponentially increasing problem
Syslog Server Lab Outside Syslog Message ICMP Unreachable IDS Syslog Message
148
74

Solution
Bring back up syslog service on server Take server offline Configure Cisco PIX to not log IDS ICMP unreachable messages
ip audit signature 2001 disable or no logging message 400011

pixfirewall# show run | grep signature ip audit signature 2001 disable pixfirewall# show cpu usage CPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes: 99%
Cisco Public
149

Summary Examine the DIFF of two show processes taken over a one minute interval Find the process taking up the highest amount of CPU (excluding the polling processes) Take actions to lower that processs CPU time Reexamine the CPU output, and repeat as necessary
Cisco Public
150
75
FWSM
Additional Architecture Information
Cisco Public
151
FWSM and ACLs

ACLs on the FWSM are compiled on the control point and pushed down into hardware (NP 3) During compile time, CPU should stay at ~ 99%
ACL compile uses all free CPU cycles Allows compile to complete in shortest time possible
Once compile is complete, rules are attempted to be pushed into hardware

Successful download
Access Rules Download Complete: Memory Utilization: 49%
Failed download (exceeded HW memory)

ERROR: Unable to add, access-list config limit reached
Cisco Public
152
76
FWSM and ACLs (Multimode)

Use show np 3 acl stats to see the current ACL resource utilization in that context
FWSM/admin(config)# show np 3 acl stats ---------------------------ACL Tree Statistics ---------------------------Rule count : 9584 Bit nodes (PSCB's): 8760 Leaf nodes : 8761 Total nodes : 17521 (max 24260) Leaf chains : 6912 Total stored rules: 15673 Max rules in leaf : 3 Node depth : 32 ---------------------------Note: One ACE does not equal one node
Total Number of ACEs This Is the Hardware Limit
153

Use show np 3 acl tree to see which ACL tree a context is mapped to
ACL Tree Number
FWSM# show np 3 acl tree -------------------------------------------ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 3 Context (004) Accounting -------------------------------------------Context Name
Cisco Public
154
77

Use the command resource acl-partition <num-ofpartitions> to reduce the number of active partitions created; default is 12 Use the command allocate-acl-partition <num> to assign a context to a specific ACL tree
FWSM(config)# context Accounting FWSM(config-context)# allocate-acl-partition 0 FWSM(config-context)# show np 3 acl tree -------------------------------------------ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------Both Use Tree 0 Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 0 Context (004) Accounting -------------------------------------------BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
155
FWSMResource Rule
FWSM 3.2 introduced
resource-ruleallows further customization of a partition
resource rule nat 10000 acl 2200 filter 400 fixup 595 est 70 aaa 555 console 283
show resource-ruledisplays information about the current rule allocation

FWSM# show resource rule Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 1843 1843 10000 ACL 74188 74188 74188 Filter 2764 2764 5528 Fixup 4147 4147 10000 Est Ctl 460 460 460 Est Data 460 460 460 AAA 6451 6451 10000 Console 1843 1843 3686 -----------+---------+----------+--------Total 92156 92156 Partition Limit - Configured Limit = Available to allocate 92156 92156 = 0
156
78
FWSMResource Partition
FWSM 4.0 introduced
resource partitionallows customization of the size of individual partitions (multi-context mode)
FWSM(config)# resource partition 10 FWSM(config-partition)# size 1000 WARNING: The rule max has been reset based on partition size 1000. The <size> command leads to re-partitioning of ACL Memory. It will not take effect until you save the configuration and reboot.
Before
FWSM# show resource rule partition 10 Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 384 384 833 ACL 14801 14801 14801 Filter 576 576 1152 Fixup 1537 1537 3074 Est Ctl 96 96 96 Est Data 96 96 96 AAA 1345 1345 2690 Console 384 384 768 -----------+---------+----------+--------Total 19219 19219 Partition Limit - Configured Limit = Available to allocate 19219 19219 = 0
After
FWSM# show resource rule partition 10 Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 20 20 43 ACL 770 770 770 Filter 30 30 60 Fixup 80 80 160 Est Ctl 5 5 5 Est Data 5 5 5 AAA 70 70 140 Console 20 20 40 -----------+---------+----------+--------Total 1000 1000 Partition Limit - Configured Limit = Available to allocate 1000 1000 = 0 157
FWSM and Control Point

The traffic that makes it to the control point is traffic that requires layer 7 fixup (embedded NAT, or cmd inspection)
FTP VoIP (SIP/SKINNY/H.323/RTSP) DNS XDMCP etc. Session Manager NP 3
Traffic sourced from, or destined to, the FWSM also goes through the control point
Syslogs URL filtering (WebSense/N2H2) Management traffic (telnet/SSH/HTTPS/SNMP) Failover communications Routing protocols (OSPF/ RIP) etc.
Fast Path Fast Path NP 1 NP 2 FWSM
AAA (RADIUS/TACACS+)
158
79
FWSM and Network Processors

The Session ManagerNP 3

Processes first packet in a flow ACL checks Translation creation Embryonic/established connection counts TCP/UDP Checksums Sequence number randomization TCP intercept etc.
Control Point (CP) Central CPU FWSM C6K Backplane Interface Fast Path Fast Path NP 1 NP 2 Session Manager NP 3
The fast pathNP 1 and 2

Performs per packet session lookup Maintains connection table Performs NAT/PAT TCP checks
FWSM Fast Path Fast Path NP 1 NP 2 Session Manager NP 3
Fragmentation reassembly etc.

159
FWSM - Enabling the Completion Unit

Due to the FWSMs NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the firewall.
4 3 2 1
4 2 3 1
This issue might be encountered when performing TCP throughput testing, or passing high speed TCP flows through the FWSM Examples: CIFS, FTP, AFP, backups FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order (by enabling a hardware knob on the NPs called the Completion Unit). In multiple mode enter this command in the admin context configuration; It will then be enabled for all contexts on the firewall.
160
80
Online Tools
Networking Professionals Connection Bug Toolkit
Cisco Public
161
Networking Professionals Connection

Online Open Forum to Ask Questions
Anyone Can Ask a Question, and Anyone Can Answer
Regular Ask the Expert Events on Certain Topics
Cisco Public
162
81
Networking Professionals Connection

Online Open Forum to Ask Questions
Anyone Can Ask a Question, and Anyone Can Answer
Regular Ask the Expert Events on Certain Topics
http://www.cisco.com/go/netpro
163
Bug Toolkit
On the Support Tools and Resources Page
Cisco Public
164
82
Bug ToolkitProduct Selection
Select Security, Then Cisco ASA 5500 Series
Cisco Public
165
Bug ToolkitAdvanced Search
Version
Search Keywords
Severity
Status
Cisco Public
166
83
Bug ToolkitSearch Results
Select Link to View Details of Bug
Cisco Public
167
Bug ToolkitBug Details
First Fixed-In Releases
Cisco Public
168
84
Opening a TAC Case

If after using all your troubleshooting tools you still cannot resolve the problem, please open a TAC case
http://www.cisco.com/techsupport/servicerequest/
At a minimum include:
Detailed problem description Output from show tech
Optionally include:
Syslogs captured during time of problem Sniffer traces from both interfaces using the capture command (capturing only the relevant packets, and saved in pcap format)
Cisco Public
169
Failover Prompt Display Configuration

The firewalls prompt maybe changed to display certain keyword Usage
prompt <keyword> [<keyword> ...]
Syntax
keywords: Hostname Configures the prompt to display the hostname Domain Context only) Priority State Slot Configures the prompt to display the domain Configures the prompt to display the current context (multi-mode Configures the prompt to display the 'failover lan unit' setting Configures the prompt to display the current traffic handling state Configures the prompt to display the slot location (when applicable)
Example
FWSM(config)# prompt hostname domain priority state slot FWSM/cisco.com/sec/actNoFailover/4(config)#
170
85

Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover history This information is not saved across reboots
ASA# show failover history ========================================================================== From State To State Reason ========================================================================== Disabled Negotiation Set by the CI config cmd Negotiation Just Active No Active unit found Just Active Active Drain No Active unit found Active Drain Active Applying Config No Active unit found Active Applying Config Active Config Applied No Active unit found Active Config Applied Active No Active unit found Active Failed Interface check ==========================================================================
Cisco Public
171
Disabling Debug Commands

undebug all
Starting with Cisco PIX 6.3/FWSM 2.3, all debugs can be disabled simultaneously by issuing no debug all or undebug all (un all for short)
ASA# show debug debug icmp trace debug sip ASA# un all ASA# show debug ASA#
. .
Cisco Public
172
86
Syslog Enhancements in 7.0 and 3.1

Logging via e-mail to an SMTP server Event classes to group syslogs Logging to an FTP server Logging to Flash Logging debug messages encapsulated in syslogs
Cisco Public
173
87

Troubleshooting Firewalls

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Troubleshooting Firewalls

Uploaded by

Copyright:

Available Formats

BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved.

Understanding the Packet Flow

Understanding the Packet Flow

Packet Processing Flow Diagram

Packet Processing: Ingress Interface

Packet Processing: Locate Connection

Syslog Because of No Connection, and Non-SYN Packet:

Packet Processing: ACL Check

Syslog When Packet Is Denied by ACL:

Packet Processing: Match Translation

Syslogs When No Translation Rule Found: (305005No NAT; 305006No Global)

Translation and NAT Order of Operations

4. Match nat commands

Packet Processing: Inspections/Sec Checks

Packet Processing: NAT IP Header

Packet Processing: Egress Interface

Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

Packet Processing: L3 Route Lookup

Packet Processing: L2 Address Lookup

Packet Processing: Transmit Packet

Cisco ASA/PIXUnderstanding the Architecture

No software limits on the number of ACEs (rules) that can be configured.

FWSMUnderstanding the Architecture

FWSM Architectural Overview

Fast Path Flow Identification, Security Checks and NAT in Hardware

C6K Backplane Interface

FWSMACL Rule Limits

FWSM 2.3 introduced

FWSM 3.2 introduced

FWSM 4.0 introduced

* See the Appendix for examples of each of these commands

3.2 / 4.0 Configurable X X

DST IP VLAN 310.14.3.x CTX1 .1 10.14.3.89

Inbound Packet Outside MSFC

*Added to FWSM 3.1

Stateful LAN/Serial Secondary (Standby) Primary (Active)

Verifying Failover Configuration

What Triggers a Failover?

What Triggers a Failover?

What to Do After a Failover

Syslogs from Secondary (Standby) Firewall

What to Do After a Failover

====Configuration State=== Sync Done ====Communication State=== Mac set

ASA Syslog Level vs. Number of Messages

FWSM Syslog Level vs. Number of Messages

What Are Modifiable Syslog Levels?

Modifiable syslog levels

How to Create Modifiable Syslog Levels

Lower syslog message 111009 to level 3 (error)

Now our syslog looks as follows

To restore the default syslog level

* Crypto Conditional Debugging was added to Cisco ASA/PIX 8.0

Debug ICMP Trace

Logging Debugs to Syslog

Show Output Filters

Example: Show Output Filters

Display the access-list entries that contain address 10.1.1.5

Display the config, except for the access-lists

Display only access-list entries that have non-zero hitcounts

Display a count of the number of connections each host has

Show CPU Usage

*First introduced in Cisco PIX OS version 6.0(1)/FWSM 1.1(1)

Show Xlate and Show Xlate Debug