Data Center Infrastructure Management

(DCIM)
(CFRT Peer Review Workshop)
11/20/08
BROADCOM PROPRIETARY & CONFIDENTIAL
1
 Schedule
– 9:30am to 10:00am Registration and CFRT Networking
– 10:00am to 10:30am CFRT Membership Business - Membership Status and
Planning
– 10:30am to 11:00am Broadcom’s Data Center Journey                   
(speaker: Dave Dalesandro)
– 11:00am to 11:30am  Broadcom’s Review of Data Center Management Software
                        (speaker: Dave Holub)
– 11:30am to 12:15pm Interactive Demo - Broadcom’s DCIM application 
(presenter: Dave Eastman)
– 12:15pm to 1:30pm  Lunch and Round Table Interaction  Working
Lunch (served in meeting room)
– 1:35pm to 2:10pm Panel Q&A (Sorena, Holub, Eastman, Dalesandro, Vippin)
– 2:15pm to 2:45pm CFRT Wrap-up Networking and Informal Q&A
– 2:50 pm to 3:15pm        Broadcom Data Center tour (for those interested)

BROADCOM PROPRIETARY & CONFIDENTIAL

2
 Presenters
• Dave Dalesandro
– Director, IT X-Functional Architecture
– Data Center Design and Management
– (ddales@broadcom.com)

• Dave Holub
– Principal Project Manager, - IT
– Data Center infrastructure design architect
– (holub@broadcom.com)

• Dave Eastman
– Sr. Principal Project Manager, - IT
– DCIM Prototype Developer
– (deastman@broadcom.com)

BROADCOM PROPRIETARY & CONFIDENTIAL

3
 Goal

Provide data and build processes to operationalize

the management of our data centers

BROADCOM PROPRIETARY & CONFIDENTIAL

4
 Why?
Data Center Infrastructure Optimization
• Broadcom data centers have been self service from inception
– This only works ‘if’ you have excess capacity
– With excess capacity ‘Optimization’ is not as critical of an issue

• Data centers cannot be self service when you near capacity

– Finding available infrastructure is ‘not’ a trivial task

• We need to get the most out of our “large $” investments in data centers

• We need to protect the data center from overload or unbalanced situations

BROADCOM PROPRIETARY & CONFIDENTIAL

5
 Compute Growth Over Time
compute power

300000

250000

200000

150000 compute power

100000

50000

0
2000 2001 2002 2003 2004 2005 2006 2007 2008

BROADCOM PROPRIETARY & CONFIDENTIAL

6
 Compute Power
• Processor Per-core EDA Points
• Xeon 2.8GHz 32-bit 6
• Xeon 3.0GHz 32-bit 6.4
• Opteron 2.2GHz 11
• Opteron 2.4GHz 12
• Opteron 2.6GHz 13
• Opteron 2.8GHz 13.9
• Intel 2.66GHz 17.4
• Intel 3.16GHz 19.1

BROADCOM PROPRIETARY & CONFIDENTIAL

7
 Global Disk Trends
Site (All) StorageType (All)
Doubling storage capacity every 2 years!!!

2500000

2000000

1500000
Data
GB

1000000

500000

0
20060329

20060628

20061227

20070328

20070627

20071226

20080326

20080723
20060927

20070926

20080625
Date

BROADCOM PROPRIETARY & CONFIDENTIAL

8
 Data Center Infrastructure
• Data Centers are valuable resources as they get close to capacity
those resources must be carefully managed
• Infrastructure includes
– Racks
– Switches and switch ports
– VLANs
– Patch panels and cables (of all types)
– Power utilization and monitoring
– Generators
– High voltage power components BMS
– HVAC components
• By accurately tracking the usage of systems and their placement in the
data center we can ensure that overload conditions do not occur

BROADCOM PROPRIETARY & CONFIDENTIAL

9
 How did we get here?
• Company growth and technology changes means growth in storage and
compute
• Evaluated several commercial applications
• We did not know what we needed until we took the journey
• We had rules (3840W for example) but did not know the cascading impact
• We were always doing fire drills to meet the demands of the business
• Lead times for data center expansion are long
• Our largest data centers were designed 10+ years ago
• Finding all the components to install just a single system in the data
center was becoming increasingly time consuming

BROADCOM PROPRIETARY & CONFIDENTIAL

10
BROADCOM PROPRIETARY & CONFIDENTIAL
11
Data Center Management

A Software Schlep

David Holub
Principal Project Manager
IS Infrastructure
11/20/08

BROADCOM PROPRIETARY & CONFIDENTIAL

12
 Software Smorgasbord
Various Buckets - Where have we looked, What have we Implemented?
Building Management System
Controls Integration – Outside the Scope of this Meeting/Discussion
Mechanical Electrical Plant Monitoring and Alerts
Propriety – Not Implemented Everywhere / Consistently Across Facilities
Multi-Protocol – Modbus, SNMP
Costly Integration and Customization
Various Scales – UPS’s from 50-500kVA modules
Critical Information for Management – Must Have Data
Commercial Application Vendors
Large – Expensive/Comprehensive Packages
“I Need A Bus Load of Consultants to get by”
Other Extreme - Open Source – Best at Monitoring & Alerts
OS Airlines.. Unix Airline - Each passenger brings a piece of the airplane and a box of tools to the airport. They gather on
the tarmac…

Heat and Power modeling packages

Instrumentation – Tie dye
CFD Analysis
Visualization Packages
Drawing Pictures
Flat Databases – hard to populate and keep current..
Cable Plant Management
Telecom-Centric
Proprietary Patch – Panel Systems
Designers

BROADCOM PROPRIETARY & CONFIDENTIAL

13
 Customized Views
• Data Center Floor – Not in all Places

BROADCOM PROPRIETARY & CONFIDENTIAL

14
 PDU Panel Example
• BCM
– Yes/No
Probably not
in all places

BROADCOM PROPRIETARY & CONFIDENTIAL

15
 Rack Power Over Time
• 5 Months .5 Amps Variation @ 110 Volts (Lost in the Sauce)

BROADCOM PROPRIETARY & CONFIDENTIAL

16
 BRCM over Time
• 5 Months - 1.8 Amps @ 110 Volts (Wrong Granularity…)

BROADCOM PROPRIETARY & CONFIDENTIAL

17
 Larger Electrical Scale
• Static Transfer Switch

BROADCOM PROPRIETARY & CONFIDENTIAL

18
 6 Months in the life of a ASTS
• Power Growth – Refresh Cycles – Great Data

BROADCOM PROPRIETARY & CONFIDENTIAL

19
 Open Source Monitoring & Alerts
• Nice Graphs – Alerts, Thresholds, Notification lots of work…

BROADCOM PROPRIETARY & CONFIDENTIAL

20
 Poor Man’s CFD
Instrument the Data Center
Understand Change
w/o Blanking
w/Blanking

BROADCOM PROPRIETARY & CONFIDENTIAL

21
 Uber Data Center Application
• Expensive in $ and Resources
– To Buy
– To Implement

• Simple Tasks Become:

• Cumbersome
• Require Great Expertise
• Canonical Issues
– Training
– Support
– Waiting for Godot

BROADCOM PROPRIETARY & CONFIDENTIAL

22
 Cable Management Icky Problems…
• Who Cares?

• Some People Do..

BROADCOM PROPRIETARY & CONFIDENTIAL

23
Cable Management Applications
• Highly Generalized
– Not Data Center Specific
– Carriers
– Contractors
– Vendors Specific

BROADCOM PROPRIETARY & CONFIDENTIAL

24
Visualization Packages

BROADCOM PROPRIETARY & CONFIDENTIAL

25
Theoretical Models

BROADCOM PROPRIETARY & CONFIDENTIAL

26
 Cool But I Need…
• Need to Roll Up the Data - Pull Together Various Pieces on a Dashboard
– Live Within Your Constraints
• Thresholds
• Deployment Rules
• Remediation – “Houston We Have a Problem” -Find All These
– Planning
• Build Consensus – Team Understanding of Data Center Ecology
• Capacity - Relationship to Redundancy and Resiliency
– Prediction
• ‘What if..?’ Various scenarios for: Refresh / Deployment / Growth
– Hierarchical Relationships
• Achieving Balance Across Infrastructure UPS’s, PDU’s, Floor Layouts/Cooling
• Efficiently using resources and capital – Getting the most out of Data Centers
– Change management
• Run a Project – We need to Deploy “X” (Space, Power, Cables Ports)
• Day –to- Day tasks – Translation into Work Orders

BROADCOM PROPRIETARY & CONFIDENTIAL

27
 Data Center Infrastructure
Management
The Application Story

Due to the proprietary nature of the application and the fact that a
decision to commercialize the product is pending, the application slides
have been removed from this copy of the presentation. However, as
discussed in the workshop if there is interest to learn more and
possibly partner with Broadcom in the further development of the
application contact should be made directly with Dave Eastman at
Broadcom.
Dave Eastman; Sr. Principal Project Manager’ IS Infrastructure; deastman@broadcom.com

BROADCOM PROPRIETARY & CONFIDENTIAL

28
 Direct Benefits
• Thresholds are understood and tracked
• Efficient use of data center Infrastructure
• Predictable consumption enables non-reactive procurement
• Change Management process is key to success
• Issues that we had no idea of before the application are now identified
issues can be rectified during upgrades or planned outages
• What if scenario provides alternatives (stretches the $DC)
– Adjust thresholds to understand the impact
– Refresh opportunities
– Some redundancy can be reduced if the impact is understood

BROADCOM PROPRIETARY & CONFIDENTIAL

29
 Indirect Benefits
• Unbalanced or over load conditions rectified before they become issues
• The journey has allowed us to get commitment from all involved
• Next generation design improvements
– Cabling
– Power distribution
– Density
– Efficiency
– Instrumentation is critical and there are options
– Modular/POD designs allow latest technology to be implemented by only building
what you need when you need it
• Rebates for refresh from Power companies

BROADCOM PROPRIETARY & CONFIDENTIAL

30
 What Next?
• This is a Peer Review Workshop so . . .

• Tell us what you’re thinking . . .

– Fill out the 1-page participant response survey

BROADCOM PROPRIETARY & CONFIDENTIAL

31
 Lunch
Round Table Discussion

BROADCOM PROPRIETARY & CONFIDENTIAL

32
 Panel Q & A
• Dave Dalesandro • Al Sorena
– Director, IT X-Functional Architecture – Manager, Critical Facilities
– Data Center Design and Management – (asorena@broadcom.com)
– (ddales@broadcom.com)
• Vippin Chadha
• Dave Holub • Regional IT Manager, North America
– Principal Project Manager, - IT • (vchadha@broadcom.com)
– Data Center infrastructure design
architect
– (holub@broadcom.com)

• Dave Eastman
– Sr. Principal Project Manager, - IT
– DCIM Prototype Developer
– (deastman@broadcom.com)

BROADCOM PROPRIETARY & CONFIDENTIAL

33
CFRT Wrap-up Networking and
Informal Q&A

BROADCOM PROPRIETARY & CONFIDENTIAL

34
Broadcom Data Center Tour

(for those interested)

THE END

BROADCOM PROPRIETARY & CONFIDENTIAL

35