Recommended background reading from the APC white paper library APC white papers:

• APC white paper 57 [Evans, 2014]
• APC white paper 59 [Evans, 2012]
• APC white paper 68 [Rasmussen and Standley, 2020]

Heat removal is necessary for all data centre environments, from tiny closets to large multi-acre dedicated facilities. Almost all the electrical power input to IT equipment is converted to heat. Without any means to remove heat, the temperature in the closed space will rise quickly.

1 Temperature

Temperature quantifies how hot an environment is.

In Europe, temperature is usually expressed in Celsius (formerly Centigrade) unit. The Fahrenheit scale (°F) is sometimes encountered in US-centric publications about cooling systems.

---

Table 1: Temperature Scales (Ahrens 1994)

---

1.1 Temperature conversions

The most common conversion is to/from Fahrenheit.

The Kelvin scale is rarely encountered in applied cooling work but is directly related to the Celsius scale:

2 Thermal envelope

Equipment and humans can only tolerate a certain range of temperatures, or thermal envelope. If IT equipment is operated outside of its thermal envelope it can lead to CPU throttling, thermal shutdown and reduced long-term lifespan.

The American Society of Heating, Refrigeration and Air conditioning Engineers (ASHRAE) have published a number of standards for thermal envelopes of data centre environments. The most recent guidelines published in 2011 are given in section 2.

---

Table 2: ASHRAE 2011 Guidelines

---

Classes A1 to A4 represent normal data centre environments, where environmental controls (e.g. cooling) are used. The ASHRAE recommended thermal envelope is between 18 and 27 degrees.

3 Cooling methods

In order to keep the temperature and relative humidity within permitted limits, we must rely on some method of cooling.

Conduction

uses the room’s surfaces to remove heat to the surrounding building.

Passive ventilation

involves vents placed appropriately within the room to permit hot air to flow naturally out, to be replaced by cooler incoming air.

Fan-assist ventilation

works similarly to passive ventilation, but the air movement is assisted by a fan.

Dedicated cooling

is where the room air is not ventilated, but instead heat is removed from it.

3.1 Cooling method selection

subsection 1 shows the available cooling methods for smaller data centres and server rooms. As the IT Load increases in size, we often require dedicated cooling solutions.

Figure 1: Cooling methods (APC)

3.2 Fan-assisted ventilation

Fans can be used on in smaller data centre environments to keep the temperature under control by exchanging the air with the ambient / outside environment, subsection 2.

Fans can be thermostatically controlled. They can be powered from a UPS to ensure the fans run even when mains power fails.

In practical terms, anything more than a simple closet with a few devices will need refrigerated cooling. Conduction, passive ventilation and fan-assisted cooling schemes are ultimately limited by the outdoor temperature. Consider trying to keep a room at 22C when the outdoor temperature is 34C!

4 Refrigeration cycle

The only way to make heat “flow uphill” from cold to hot is to assist it. From our point of view, we want to remove heat from our computer room and reject it to the atmosphere, to keep the room temperature under control. section 3 shows the basic refrigeration cycle. For a more thorough treatment, see Evans [2014].

There are four stages evident in section 3:

1. Cold liquid refrigerant in the evaporator is warmed by air passing over it, and boils at roughly 7.8 °C. The air passing over the evaporator gives up some of its heat energy. It leaves at a cooler temperature than it entered at.
2. The compressor increases the pressure of the gaseous refrigerant, greatly increasing its temperature to over 50 °C. In doing so, it also acts as a pump for the refrigerant around the loop, which is carrying the heat energy to reject.
3. Hot gaseous refrigerant enters the condenser coil, across which is circulated outside air. As the refrigerant is hotter than the outside air, it gives up its heat to the outside air. The air passing over the condenser receives heat energy from the hot refrigerant, leaving at a warmer temperature than it entered at. The refrigerant is cooled below its boiling point and changes phase to a liquid. It will still be quite hot to the touch!
4. The warm liquid flows through the expansion valve, which limits the flow of refrigerant such that it is boiled off in the evaporator. When the refrigerant emerges from the expansion valve, it expands since the flow is limited, and is ready for another cycle in the evaporator.

Watch the Refrigeration Cycle 101 video on YouTube

5 Types of cooling

Comfort cooling

is what we are familiar with as air conditioning. Warm humid air in an occupied space is cooled and dehumidified. Usage varies with season and personal preferences.

Precision cooling

is designed to work 24/7 365 days a year to keep temperature (and often humidity) within a set band for IT equipment to work effectively.

There are two main categories of cooling system seen in data centre environments, direct expansion (DX) and chilled water.

6 Computer Room Air Conditioners

Refrigerated cooling is provided in a data centre environments by two main families of equipment: Direct eXpansion (DX) and Chilled Water. DX systems where the evaporator directly meets air in the data centre environment are the simplest to understand and most common in small and medium environments.

DX systems normally take the form of a Computer Room Air Conditioner, or CRAC. A CRAC is normally a large floor-standing unit containing the evaporator coil, blower fan, compressor and other components. For in-depth information refer to Evans [2012].

CRACs are most usually seen in so-called downflow configuration as in section 5.

• Warm return air enters the CRAC via an opening on the top of the unit. Sometimes the return air is ducted to the unit from above the ceiling.
• Cold supply air leaves the CRAC via an opening in the bottom:
  • If there is a raised floor, the cold air is blown out the bottom of the unit under the raised floor. It flows under the raised floor and exits through perforated tiles.
  • In the case of a solid floor, the cold air normally leaves via a large grille on the bottom of the unit.

Other configurations include upflow (reverse of the above, but only on a solid floor), horizontal and other configurations. Cooling units are sometimes located in other areas and ducted to the data centre environment.

For the cooling system to work properly, we must use a hot aisle / cold aisle arrangement. Where a raised floor is used, the perforated tiles must be in the cold aisle, and not in the hot aisle.

CRAC evaporator fans normally run continuously. The refrigeration system cycles on and off under thermostatic control to maintain the selected temperature. Control is often based on the return, supply or space air temperature. More sophisticated control strategies are becoming common, but the basic idea outlined above is sufficient.

6.1 Self-contained air-cooled DX

Self-contained units have all refrigeration components within the CRAC’s casing. The condenser supply and exhaust are ducted from the outdoors. Limited in cooling capacity to approx 15 kW due to unit size and ducting, subsection 6.

Figure 6: DX self-contained CRAC (APC)

Candidate for up to 15 kW. Often seen in small on-site server rooms.

6.2 Air-Cooled DX

Direct expansion, often abbreviated DX, cooling systems house the evaporator, compressor and expansion valve within the CRAC unit. The condenser is sited externally, with wiring and refrigerant connections from the CRAC unit, subsection 7.

Figure 7: DX air-cooled CRAC with condenser (APC)

Candidate for 7 kW to 200 kW. Multiple units often used for larger capacities and to provide redundancy (see later).

An alternative layout of the Air-Cooled DX system has the compressor is located in the outdoor unit, called a condensing unit. This arrangement is commonly called the split system.

6.3 Glycol-cooled DX

The Glycol-cooled DX replaces the air-cooled condenser with a refrigerant-to-fluid condenser inside the CRAC casing, subsection 9. Liquid coolant consisting of water mixed with ethylene glycol is circulated between the condenser and a dry cooler (coil with a fan) by a pump package.

Figure 9: DX Glycol-cooled CRAC with Dry Cooler (APC)

Some advantages of this system are:

• Refrigeration circuit is sealed inside the CRAC.
• Glycol can be pumped much longer distances.
• Can bypass the refrigeration system if the ambient air is cool enough. Glycol pumped through an economizer coil instead. See later on!

Candidate for 30 kW to 1000 kW, and where distance exceeds ?? metres. Systems can either be:

Unitary

where a single CRAC is linked to a single dry cooler.

Shared

where a number of CRACs share one or more dry coolers.

6.4 Water-cooled DX

The Water-cooled DX is similar to the glycol-cooled version, except that a water loop is used instead of a glycol loop. The dry cooler is replaced with a cooling tower, often on the facility’s roof, where jets of water are cooled by spraying them into a moving air stream.

Figure 11: DX water-cooled CRAC with cooling tower (APC)

It is more efficient that Glycol-based systems in hot dry climates, where evaporation in the cooling tower assists with its operation. However, the water treatment must be carefully considered. There is also a strong risk of legionella in these systems if not properly looked after.

Water cooled DX is rarely installed in a unitary fashion. One or more shared cooling towers servicing multiple CRACs with cooled water is by far the most common configuration.

Cooling towers are often operated by facilities management or by the landlord in larger buildings, and water is supplied metered or unmetered “as a service”. The same towers often serve not only technical cooling, but also comfort cooling, process cooling and refrigerated storage (for food and other products) systems.

7 Chilled Water Cooling Systems

Chilled water cooling systems involve the supply of chilled water to Computer Room Air Handlers (CRAH) in the data centre environment. CRAHs are similar to CRACs in appearance, and are available in the same wide range of airflow configurations. They are simpler than CRACs, containing only a blower fan, finned water coil and controls.

The chilled water is produced in a separate chiller. The chiller rejects heat from the chilled water loop to the atmosphere in similar ways to DX CRACs. Air-cooled, glycol-cooled and water cooled chillers are all very common.

Although section 13 depicts a unitary system, chilled water systems are almost always installed such that a large number of CRACs or other cooling loads are served by a small number of chillers and cooling towers / fluid coolers. Chilled water systems also can incorporate large buffer tanks for load balancing, energy price optimisation and to provide ride through in case of power failure.

Many buildings, particularly city skyscrapers, use centralised water-cooled chilled water system. A small number of very large cooling towers serve a number of chillers which supply chilled water for data centre, comfort cooling and other cooling purposes. These are generally managed by facilities personnel.

In a multi-tenant building, the landlord will often supply chilled water as a metered chargeable service to tenants. Condenser loop water is often also available at a much cheaper unit rate (which can be used for a DX water-cooled CRAC or local water-cooled chiller for IT purposes).

8 Sizing

Generally, we total up the IT loads and pad the result by 30%.

9 Efficiency metrics

9.1 Coefficient of Performance (COP)

The coefficient of performance indicates how many kW of heat is removed per kW of electrical power input to the air conditioning system. It is calculated as:

The COP has no units since the numerator and denominator have the same units.

9.2 British Thermal Units

Heat is a form of energy, the SI unit of which is the Joule, J. When a given amount of energy is transferred per second (e.g. 1 J s⁻¹), it is given the unit of the Watt, W.

Many ideas around refrigeration and air conditioning originated from the United States, where imperial units are still common. In the imperial system, energy is measured in the British Thermal Unit, or BTU. Power is measured in BTU h⁻¹.

To convert from kW to BTU h⁻¹, we use the relation:

9.3 Energy Efficiency Ratio (EER)

The EER is normally calculated as:

The EER has units of BTU h⁻¹ W⁻¹, but the unit is often omitted.

Note that the EER and COP are directly related by the factor:

9.4 Power Usage Effectiveness

The power usage effectiveness ($\text{PUE}$) of a data centre environment is:

The $\text{PUE}$is a dimensionless number. An ideal $\text{PUE}$would be 1.

References