ID114 Wrestling the Snake: Performance Tuning 101

Wes Morgan | Senior Software Engineer

2012 IBM Corporation

Agenda

Why Are We Here? Setting A Reasonable Goal It's a Cycle, Folks Understand Your Environment A word about add on products Take a Picture of Everyday Life Common Red Flags Thinking about SANs Dealing with the Virtual World A Quick Word About the Operating System
2 | 2012 IBM Corporation

Why Are We Here?

Supply and Demand Throwing hardware at it no longer an option Stretching existing systems Making upgrades tactical Need numbers to justify actions Know what to look for

3 |

2012 IBM Corporation

Setting A Reasonable Goal

We aren't going to Unicorn-and-Rainbow Land

Even the best designs can be stressed

Changing demand New initiatives, new software, new users Most reasonable goal: Meet peak demand with (some) room to spare

4 |

2012 IBM Corporation

It's a Cycle, Folks

You can't do this once and call it finished As system load changes, we must adjust and adapt
Remember that load can change in volume, footprint, complexity or all of the above

Revisit performance on a regular basis

Know what numbers are important for your enterprise Know who supports what in your organization Automate reports (monthly, bi-monthly) Catch trends before they become problems

Revisit performance on any significant change

New software releases Changing OS (or OS version) Adding/moving/removing virtual machines Networking changes in your enterprise
5 | 2012 IBM Corporation

Understand Your Environment!

Know your dependencies

Directories (Active Directory, LDAP) Third-party products (Domino add-ins, Portal adjuncts) Back-end components (WCM, DB2, etc.)

Know your data flow

What's handling your data, where, and when?

Know your networking

Are you traversing firewalls? Must you use proxy servers?

Know where your load lies

What do clients do? What do servers handle?

6 |

2012 IBM Corporation

Understand Your Everyday Life

Gather baseline performance statistics

perfpmr (AIX), platform statistics (Domino), perfmon (Windows)

First Question: What does routine work look like? Make sure that your baseline includes at least one peak use time
Monday morning, 9 am?

Use this baseline to understand where your system pushes Identify potential pressure points
e.g. Peak use of particular facilities, but not yet in the danger zone

Track back to application-layer activities

System backups Domino agents Third-party scripts

7 |

2012 IBM Corporation

Environment, Data Flow and Performance

Authentication

Application

Storage

Firewall

Third-Party Component

8 |

2012 IBM Corporation

A word about add on products

What is an add on product?

Any IBM or third party product or component designed to link into a Domino server. Can be a silent killer of resources

Off the shelf application

Know the impact before you begin Know what the best practices are

API Tool Kit

Provide multiple ways to do the same task With great flexibility comes great responsibility

9 |

2012 IBM Corporation

A word about add on products Blackberry Enterprise Server

Research In Motion's answer to mobile connectivity Does not run on your mail server but can have a huge impact NBES thread count can starve the hand held or overload the mail server

10 |

2012 IBM Corporation

A word about add on products Applications

IBM Lotus Enterprise Integrator (LEI) / Domino Enterprise Connection Services (DECS)
Provide the interface to back-end database systems Can drive a high disk and network load

Extension Manager / NSFHooks

Test, Test, Test Use LotusScript when available

11 |

2012 IBM Corporation

Common Red Flags - Disk

Disk I/O
Huge contributor to performance problems Disk Queue Length

# of pending requests (regardless of their size) Every OS vendor agrees disk queues longer than 2.0 indicate a bottleneck Look for both peak queue length and average queue length Usually expressed as a percentage Effects of high utilization are obvious High utilization can contribute to high queue lengths

Disk utilization

12 |

2012 IBM Corporation

Common Red Flags - Memory

Paging
High paging can indicate RAM starvation Also increases disk activity can send you 'over the edge' with disk latency Set paging files to at least 1.5x RAM as baseline

Swapping
High swapping can indicate process thrashing Can be triggered by high volume of short-term activity (e.g. scripts, agents)

NOTE: No real consensus on bad numbers for these

Examine these in comparison to your healthy baseline Look for spikes or high plateaus

Virtual servers add to the complexity here!

13 |

2012 IBM Corporation

Common Red Flags - CPU

Looking for repeated spikes or high plateaus Easy for one process to effectively hang your system May need to go to per-process views to find culprit First cut match against known actions (agents, scripts, backups)

14 |

2012 IBM Corporation

Thinking About SANs

They aren't infinite

Know what shares your SAN (or SAN slice) Know general SAN operations (e.g. caching, backups, etc.) Remember that another system's I/O can affect your performance

They aren't magic

Applications/OS still see them as a single disk Disk queueing and utilization concerns still apply Demand can differ drastically among applications

15 |

2012 IBM Corporation

Dealing With The Virtual World

Regardless of vendor, you're still sharing one set of physical resources Still subject to interference from other virtual machines Still need to provision virtual machines as fully as real hardware Compare your performance to other virtual machines on the same host Be careful with moving virtual machines
Pay attention to overall application architecture Can lose fault-tolerance/failover

Remember, you now have TWO potential bottleneck points

Within your virtual machine The virtual host itself

Example: Output queue length on VM network interface

ASK ABOUT OVERCOMMITTING THE VIRTUAL HOST!!!

16 |

2012 IBM Corporation

A Word About OS Patches

Be very, very careful Some patches can introduce problems all their own Use a test environment whenever possible Keep abreast of new patches Be sure to spot-check performance numbers after patch installation

17 |

2012 IBM Corporation

What About BIOS and Drivers?

Keep system BIOS as close to latest and greatest as possible

Some vendors release new BIOS versions almost monthly May address/optimize important subsystems May resolve particular performance issues

Watch for new device drivers as dictated by your performance

If you're disk-heavy, keep an eye out for SAN/controller drivers If you're network-heavy, watch for advanced network drivers

Partner with your hardware team!

18 |

2012 IBM Corporation

In Conclusion...

No one right answer across systems or applications Performance tuning driven by your users, your data, your network It's a repeating cycle get used to it Partner with network/hardware teams for best effect

19 |

2012 IBM Corporation

Questions & Answers

I'll be in the Meet the Developers Lab all week... Don't forget the Session Evaluation... THANKS FOR BEING HERE! Twitter: @wesmorgan1

20 |

2012 IBM Corporation

Legal disclaimer
IBM Corporation 2012. All Rights Reserved. The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBMs current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBMs sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

21 |

2012 IBM Corporation