Data Center Power and Cooling

Information about Data Center Power and Cooling

Published on November 15, 2007

Author: Estelle

Source: authorstream.com

Content

Slide1:  Data Center Power & Cooling Keeping up with Change 2003 IT Infrastructure Trends and Challenges Power Infrastructure Basics Planning and Deployment Cooling Best Practices Ken Baker [email protected] ISS Rack & Power May 9, 2003 Notice: The information contained herein is copyrighted property and may not be reproduced without the written permission of the Hewlett Packard Company Lone Star Chapter why are we talking today?:  why are we talking today? due to server and storage densification, yesterday’s data center infrastructure and design practices cannot keep pace with the growing demands for power and cooling the high density architectures and products we talked about for the last two years are here today 3 Key Metrics to Remember:  3 Key Metrics to Remember Supply Load Distribution on the customer front:  on the customer front what are the right questions to be asking? have I engaged facility engineering in planning for growth and hardware deployments? have I planned for growth in power and heat density? is my data center configured to provide sufficient airflow to the new servers & storage products? you own and manage the environment ! Slide5:  power trends power supplies trends:  power supplies trends power supply capacities have not grown substantially and still range from 200 -1200 watts the physical size is decreasing high voltage operation (200+VAC) as a requirement is becoming more common power supply efficiency is starting to improve power trends:  power trends two years ago average servers per rack = 4 - 6 average watts per rack = 1500 - 3000 today average servers per rack = 8 - 12 average watts per rack = 5000 - 6000 not unusual to see 7- 12 - 18KW implementations Slide8:  processor trends Xeon MP 400Mhz 667 Mhz >90W 91W per CPU >100W 62-90W Xeon DP 533Mhz 667Mhz 800Mhz 95W 38W per CPU >100W 400Mhz 75W 88W 133Mhz 125W What Key Metric is Changing Here?:  What Key Metric is Changing Here? Load historical server comparison:  historical server comparison the original SystemPro first ProLiant PC Server 33MHz 386 processor 8 Mb RAM, 210 Mb HDD $36,000 400W BL 10e blade first ProLiant blade server 900 MHz processor 512 Mb RAM, 40 Gb HDD $1,800 20-25W(x24) server trends:  server trends generation over generation density growth rates 2p, 4gb, 2hdd, 1pci 2p, 4gb, 6hdd, 2pci 4p, 8gb, 4hdd, 3pci 1U 2U 4U combining all of the trends:  combining all of the trends Watts Per U Overall power density per Rack Unit “U” is becoming the key metric, capturing all of the key contributors to the densification. platform, form factor power in watts coolest hottest Server Power Density Per Rack Unit:  Server Power Density Per Rack Unit Storage, PL and ML Class Watts per U Server Power Density Per Rack Unit:  Server Power Density Per Rack Unit BL and DL Class Watts per U Slide15:  delivering power for hyper dense data centers: Delivering More Power:  Delivering More Power overcoming the “desktop mentality” standard 120V, 10A PC power cord is all you need open power receptacles indicates available unlimited power powering the infrastructure. Must Break the 15A and 20A branch barrier Options include multiple 30A and 3 Phase branches more power requires larger power distribution hardware beyond the desktop:  beyond the desktop larger copper cabling SJ/SO 3 x 10Awg SJ/SO 4 x 10Awg NEMA L6-30P, 208V, 30A, 3w NEMA L15-30P 208V, 30A 3ph, 4w beyond the desktop:  beyond the desktop IEC 309 Pin/Sleeve Plugs 16A 1Ø 32A 1Ø 32A 3Ø larger copper cabling HO5V - 3x4.0mm HO5V – 5x4.0mm power density issues:  power density issues on the electrical side, why is deploying full racks of servers a problem? total rack load not the problem issue lies with how power is distributed line cord/distribution outlet relationship and restrictions commercial branch circuits in North America:  commercial branch circuits in North America today’s power densities dictate leaving the 120V infrastructure, using multiple 208V 30A feeds and looking forward to using 3 phase power to meet the density demands. power limitations associated with each common branch circuit type What Key Metric is Changing Here?:  What Key Metric is Changing Here? Distribution influence of the panel:  influence of the panel 84 pole panel, 42 2-pole locations 208VAC breakers requires 2 poles 30 amp breaker is limited by NEC to 24A continuous duty 24A x 42 2-pole breakers 150kVA available! Appears to be plenty of overall power! typical panel with 84 pole positions PDU/panel infrastructure:  PDU/panel infrastructure x8 NEMA 5-15P (for Low Volt use ONLY) – 15A each/12A total x8 IEC 320 C13 outlets 10A each/12A total x4 IEC320 C19 Outlets 16A each/12A total Each IEC C19 Outlet is limited to 12A 24A continuous duty L6-30R limited to 24A continuous 2 pole 30A breaker typical panel with 84 pole positions power requirements:  power requirements commercial branch circuits in North America:  commercial branch circuits in North America example of distribution limitations:  example of distribution limitations 24A PDU 208V single/bi-phase PDU limited to 4992 VA subject cabinet of 21 DL380G2 would require 8560 VA total power or two 24A PDUs redundancy is required, that doubles the PDU count to 4 4 breakers (8 poles) per cabinet (4 PDUs) = 80 poles 10 cabinets per 84 pole panel 210 servers provided with power increasing power capacity:  increasing power capacity why not install larger PDU? any larger 1/2 phase PDU eliminates cost effective pluggable solutions (hardwiring) no cost-effective standard connectors > 30A only potential solution move to 3 phase power distribution cost effective pluggable solution > NEMA L15-30P 30A 3 phase PDU power :  30A 3 phase PDU power most efficient way to distribute power overall available power in a 30A circuit rises from 4995 VA to 8650VA total available current rises from 24 A to 42 A fewer overall panel positions used per rack of high density loads panel utilization for racks of DL380G2:  panel utilization for racks of DL380G2 Slide30:  single-phase to 3-phase density comparison What have we changed?:  What have we changed? copper cabling SJ/SO 3 x 10Awg SJ/SO 4 x 10Awg NEMA L6-30P, 208V, 30A, 3w NEMA L15-30P 208V, 30A 3ph, 4w Before After summary/power issues:  summary/power issues server densification is toping out conventional power infrastructure and methods we have exceeded the ability of a plug and play solution to be deployed in a high density manner due to power distribution limitations without developing new approaches moving to 3 phase power to the rack enclosure is the future Slide33:  power planning power planning methods:  power planning methods use “name-plate” ratings worked in yesterdays environment costly method resulting in wasted infrastructure $ use actual power measurements most accurate approach numbers are difficult to generate and collect use “ProLiant Power Calculators” best practice for advanced planning numbers are more realistic Factoring for future growth Rate of change in today’s market continues at 25%-30% ISS power sizing calculators:  ISS power sizing calculators calculator public website:  calculator public website http://h18001.www1.hp.com/partners/microsoft/utilities/power.html about power calculators:  about power calculators based on actual system measurements taken on systems running NT and exercise utilities all major system components (CPU, memory and drives) are exercised at 100% duty cycle power results may be higher than your actual configuration leaving you extra headroom calculators can be found on each servers Active Answers page under configuration tools new calculator public website Slide38:  Issues in cooling hyper dense data centers cooling trends:  cooling trends as power consumption grows, so does the thermal demands cooling needs must be expressed in tons of AC today largest issue is not finding the cooling media, but finding a way to get it where it belongs driving new technologies to deliver cooled media cooling requirements:  cooling requirements cooling trends:  cooling trends 2 years ago average BTU load was 10-15K BTU’s average U size was 5-7U today average BTU load is approaching 28K BTU’s average U size is moving down to 3U air-conditioning:  air-conditioning issues we never thought we would have to tackle how many servers can a ton of A/C cool? how do we maximize efficiency? how much airflow does a densely populated cabinet require? how does the warmed air get back to the CRAC? The mechanics of server airflow:  The mechanics of server airflow Cool air in, warm air out All servers today are designed for front to back airflow airflow data for servers:  airflow data for servers hot isle – cold isle data center layout:  hot isle – cold isle data center layout Datacenter air-conditioning:  Datacenter air-conditioning one ton of air-conditioning is 12,000BTU’s how big is a ton of air-conditioning? one ton DX air conditioner takes roughly the cubic space of a 42U cabinet problem is delivery and return, not production issues of poor CRAC placement:  issues of poor CRAC placement If CRAC is placed to close to cabinets negative air pressure is created, robbing system of cooling combined with end of row recirculation, this can create some big hot spots The negative affects of mixed air:  The negative affects of mixed air Contributors: low ceiling - low return air volume blanking panels not used internal rack recirculation end of row recirculation low volume perf tiles low supply plenum pressure blanking panels, not for looks any more:  blanking panels, not for looks any more any ISS product any ISS product any ISS product any ISS product dual supply plenum configuration for supporting high-density solutions :  dual supply plenum configuration for supporting high-density solutions data center research:  data center research hp “cool labs” research on analysis of data center with CFD theoretical model constructed physical model of to prove theory physical model analysis proved theoretical model assumptions analyzed accuracy proven to be within 7% hp data center validation services ”Smart Cooling”:  hp data center validation services ”Smart Cooling” Current setup, determine need for detailed validation services engage the data center services team front end information from the customer is compared against criteria document if the data center exceeds the key criteria the services team suggests a detailed 3D CFD analysis. rule of thumb, data centers exceeding 100 W per sq ft are primary candidates for detailed analysis. (gross load over gross area) data center before “Static Smart Cooling”:  data center before “Static Smart Cooling” Vertical Recirculation Highest temperature in room = 57 C (Goal is 45 C) computational fluid dynamics:  computational fluid dynamics detailed CFD analysis first models the customer data center’s physical layout air conditioning resources enterprise infrastructure equipment then provisions air conditioning resources CRAC unit settings, perforated floor tile layout return air vents / supply vents (if applicable) heat load distribution to provide the best possible support for the customer’s enterprise data center after “Static Smart Cooling”:  data center after “Static Smart Cooling” Highest temperature in room = 47 C (HPL goal was 45 C) Vertical Recirculation Removed Before After What Key Metric is Changing Here?:  What Key Metric is Changing Here? Distribution Recommendations:  Recommendations When arranging cabinets in a data center, arrange front to front, back to back Use “Blanking Panels” to fill all empty space in racks. This prevents the short circuiting of cold air to the hot aisle. Calculate air conditioning based on “Sensible Capacity” not rated tonnage. Map out a maximum load on the facility and keep to it May involve empty slots in racks Recommendations (cont):  Recommendations (cont) Avoid creating hot spots Work to balance load in your data center to maximize HVAC capabilities Biggest mistake in laying out equipment in data centers is arranging by component type Racks and rows full of servers (high heat loads) Racks and rows full of Storage Slide59:  [email protected] – Ken Baker [email protected] – Bob Pereira Questions? References:  References 1. Patel, C.D., Bash, C.E., Belady, C., Stahl, L., Sullivan, D., July 2001, “Computational fluid dynamics modeling of high compute density data centers to assure system inlet air specifications”, Proceedings of IPACK’01 – The PacificRim/ASME International Electronics Packaging Technical Conference and Exhibition, Kauai, Hawaii. [Modeling Validation] 2. Stahl, L., Belady, C. L., Oct 2001, “Designing an Alternative to Conventional Room Cooling”, Proceedings of the INTELEC’01 International Telecommunications Energy Conference, Edinburgh, Scotland ……………………………[Infrastructure] 3. Friedrich, R., Patel, C.D., Jan 2002, “Towards planetary scale computing - technical challenges for next generation Internet computing”, THERMES 2002, Santa Fe, New Mexico……………………………………………..[Pervasive Computing and Data Centers] 4. Patel, C.D., Sharma, R.K, Bash, C.E., Beitelmal, A, May 2002, “Thermal Considerations in Cooling Large Scale High Compute Density Data Centers”, ITherm 2002 - Eighth Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems, San Diego, California……………………. [Static Provisioning] 5. Sharma, R.K, Bash, C.E., Patel, C.D., Dimensionless Parameters for Evaluation of Thermal Design and Performance in Large Scale Data Centers, 8th AIAA/ASME Joint Thermophysics and Heat Transfer Conference, St. Louis, June 2002. [Data Center Figure of Merit, non-dimensional numbers for data centers] 6. DeLorenzo, D., Thermal Trends and Inflection, 7X24 Exchange 2002, Orlando, Florida [Trends]

Related presentations


Other presentations created by Estelle

personality
19. 11. 2007
0 views

personality

budd hind rev
13. 12. 2007
0 views

budd hind rev

A Mother in Mannville
03. 10. 2007
0 views

A Mother in Mannville

ANGEL pervasive presentation
01. 10. 2007
0 views

ANGEL pervasive presentation

Liderazgo Estrategico
05. 11. 2007
0 views

Liderazgo Estrategico

NewGenLib
02. 10. 2007
0 views

NewGenLib

The Underground Railroad
05. 12. 2007
0 views

The Underground Railroad

Rajaram edited
10. 12. 2007
0 views

Rajaram edited

Chapter 06
05. 11. 2007
0 views

Chapter 06

viajestiempo
12. 11. 2007
0 views

viajestiempo

constellations
13. 11. 2007
0 views

constellations

Celestial Navigation
14. 11. 2007
0 views

Celestial Navigation

Telstra
26. 11. 2007
0 views

Telstra

DOPE WWW
06. 12. 2007
0 views

DOPE WWW

Presentation TerryHicks
29. 12. 2007
0 views

Presentation TerryHicks

Ecology 10
01. 01. 2008
0 views

Ecology 10

Biomass SouthCoast
04. 01. 2008
0 views

Biomass SouthCoast

IntlPepperCAW06 wyenandt
07. 01. 2008
0 views

IntlPepperCAW06 wyenandt

Rhodes
28. 12. 2007
0 views

Rhodes

FMF Sec of Res pics 06
05. 11. 2007
0 views

FMF Sec of Res pics 06

WilliamRiley
04. 01. 2008
0 views

WilliamRiley

ca sos hearing presentation
29. 10. 2007
0 views

ca sos hearing presentation

EPCglobal update
15. 11. 2007
0 views

EPCglobal update

Auteur Theory and Robert Altman
19. 02. 2008
0 views

Auteur Theory and Robert Altman

Evaluating Diversified Companies
24. 02. 2008
0 views

Evaluating Diversified Companies

20040225 Peter Overview of h264
27. 02. 2008
0 views

20040225 Peter Overview of h264

russ rev
29. 02. 2008
0 views

russ rev

VBWG05 ASCOT
05. 03. 2008
0 views

VBWG05 ASCOT

Discover seminar2
11. 03. 2008
0 views

Discover seminar2

LT1001N Lecture 3 2006 7
14. 03. 2008
0 views

LT1001N Lecture 3 2006 7

China Outbound Tourism free
27. 03. 2008
0 views

China Outbound Tourism free

050623 Kaj je CIVITAS
01. 11. 2007
0 views

050623 Kaj je CIVITAS

Gene Edwards
30. 03. 2008
0 views

Gene Edwards

CUHistory
13. 04. 2008
0 views

CUHistory

assembly CU final 2 Oct
05. 11. 2007
0 views

assembly CU final 2 Oct

Cooking With Cranberries
05. 11. 2007
0 views

Cooking With Cranberries

6th US History SOL Review
24. 12. 2007
0 views

6th US History SOL Review

CS455 Introduction
28. 11. 2007
0 views

CS455 Introduction

Thorstensen
27. 12. 2007
0 views

Thorstensen

cdr
07. 11. 2007
0 views

cdr

Freak the Mighty
30. 12. 2007
0 views

Freak the Mighty

03 using objects
05. 11. 2007
0 views

03 using objects

AI Lecture72006
16. 11. 2007
0 views

AI Lecture72006

GSA 2006 Heart Disease
01. 12. 2007
0 views

GSA 2006 Heart Disease