Page 1 HP Cluster Platform InfiniBand Interconnect Installation and User's Guide HP Part Number: A-CPIBI- 1 E Published: October 2006...
Page 2 © Copyright 2008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.
Page 3: Table Of Contents
Table of Contents About This Manual......................11 Organization............................11 Audience...............................12 Related Documentation........................12 Voltaire Documentation........................12 HP Cluster Platform Documentation....................12 Cluster Component Documentation....................13 HP Encourages Your Comments......................13 Text Annotations...........................13 Conventions............................14 Safety Considerations.....................15 Information on Ergonomics........................16 Recycling...............................17 1 InfiniBand Technology Overview................19 1.1 Identifying InfiniBand Fabric Components..................20 1.2 ISR 9xxxx-series Interconnects.......................21 1.3 Identifying the ISR 9024 Interconnect (SDR)...................22 1.4 Identifying the ISR 9024 S/D Interconnect (RoHS Compliant) ............22...
Page 4 3.6 Operating the ISR 9024 S/D Interconnect..................47 3.7 Troubleshooting the ISR 9024 S/D....................47 4 Installing and Maintaining the ISR 9096 and ISR 9288 Interconnects....49 4.1 Overview of the ISR 9XXX Interconnects..................49 4.1.1 ISR ISR 9XXX Interconnect Features..................49 4.1.2 ISR ISR 9XXX Interconnect Architecture.................50 4.1.3 ISR ISR 9XXX Building Blocks....................50 4.1.3.1 Field Replaceable Units....................51 4.1.4 ISR ISR 9XXX System Firmware....................51...
Page 5 7.1.7 Cable Routing Procedure for the ISR 9288 ................88 7.2 Connecting the ISR 9288 Router Blades..................89 7.2.1 Connecting to IPR GbE Ports....................89 7.2.2 Connecting to FCR FC Ports....................90 7.3 Chassis Administrative Connections....................90 7.3.1 Cabling the ISR 9024 Management Connections..............90 7.3.2 Cabling the ISR 9096 and the ISR 9288 Management Connections........90 7.3.3 Connecting the Management Port Serial Interface..............91 8 Installing and Maintaining HCA Cards and Mezzanines........93 8.1 Overview of the Voltaire HCA 400 PCI–X, and HCA 400 PCI-Express Cards.......93...
Page 6 F.5 Mellanox PCI-Express HCA Specifications...................128 F.6 Mellanox Memory Free PCI-Express HCA (SDR) Specifications..........129 F.7 Mellanox Memory Free PCI-Express HCA (DDR) Specifications..........129 F.8 Mellanox PCI-Express DDR HCA Specifications................130 F.9 HPC 4x DDR IB Mezzanine HCA Specifications................130 Index..........................133 Table of Contents...
Page 7 List of Figures Typical InfiniBand Network......................21 ISR 9024 Interconnect Front Panel View..................22 ISR 9024 S/D (RoHS Compliant) Interconnect Front Panel View..........23 ISR9096 Interconnect Front Panel View..................23 ISR 9096 Interconnect Rear Panel View..................24 ISR 9288 Interconnect Front Panel View..................25 Internally Managed ISR 9024, Front Panel..................30 ISR 9024 Management Card and DB-9 Adapter (Mini USB)............31 Externally Managed ISR 9024, Front Panel...................32 ISR 9024 Rear Panel (InfiniBand Ports)..................32...
Page 8 Topspin/Mellanox SDR PCI–X HCA.....................98 Topspin/Mellanox SDR PCI–Express HCA...................98 Mellanox PCI–X HCA........................99 Mellanox PCI–Express HCA.......................100 Mellanox Memory Free PCI–Express HCA (SDR)..............101 Mellanox Memory Free PCI–Express HCA (DDR)..............102 Mellanox PCI–Express HCA (DDR)....................104 8-10 4x DDR IB Mezzanine HCA......................105 List of Figures...
Page 9 List of Tables ISR 9XXX Standard Configurations....................64 sFB-12 Fabric Board LED Status....................73 Line Board Status LEDs.........................74 sMB Board LEDs..........................75 Power Supply Unit LEDs......................77 IPR Blade LEDs..........................78 FCR Blade LEDs..........................79 sCTRL Board LEDs........................81 sFU-8 Fan Module LEDs ......................84 InfiniBand Cable Characteristics....................85 HCA-400 Status LEDs........................97...
Page 11: About This Manual
About This Manual This manual describes how to install and operate an InfiniBand interconnect as part of an HP Cluster Platform. This manual limits detailed information and procedures for cluster components that have their own documentation, such as the cluster nodes, network switches, and system racks. This manual contains references to the component documentation, which should be readily available during the installation procedure.
Page 12: Audience
Chapter Description Appendix E: 4x DDR IB Defines the operational and technical specifications for the 4x DDR InfiniBand Switch Switch Module Module for the c-Class Bladesystem enclosure. Specifications Appendix F: HCA PCI Defines the operational and technical specifications for the HCA-400-series and other Card Specifications HCA cards installed in cluster nodes.
Page 13: Cluster Component Documentation
• Blades InfiniBand Interconnect Cabling Tables – Provides the point-to-point InfiniBand cabling tables for c-Class BladeSystems. • c-Class Blade Cable Management Bracket Installation Guide – Provides the installation procedure for the c-Class Blade Cable Management Bracket. • HP Cluster Platform InfiniBand Fabric Management and Diagnostic Guide – Provides test and diagnostic procedures that are specific to the application of Voltaire InfiniBand interconnects.
Page 14: Conventions
Warning ! Text set off in this manner indicates that failure to follow directions could result in bodily harm or loss of life. Caution: Text set off in this manner indicates that failure to follow directions could result in equipment damage or data loss.
Page 15: Safety Considerations
Safety Considerations To avoid bodily harm and damage to electronic components, read the following safety and comfort guidelines before unpacking and configuring the cluster components. Heed the following additional warnings and refer to the HP Cluster Platform Overview and the HP Cluster Platform Site Preparation Guide to obtain specific information on safety issues.
Page 16: Information On Ergonomics
Refer to the specifications section of the component documentation to find the weight of a component. Removing and Replacing For your safety, never remove the cover from a cluster Component Covers component without first disconnecting the power cord from the power outlet and removing any connection to a telecommunications network.
Page 17: Recycling
Recycling Shipping an integrated cluster generates far less packaging than the individual components that it contains. However, large clusters use a substantial amount of packaging material that is not reusable. The bulk of the packaging material is recyclable, and is labeled as such. You should plan on providing a number of dumpsters into which this packaging can be sorted and recycled.
Page 19: Infiniband Technology Overview
1 InfiniBand Technology Overview InfiniBand™ is a specification of the InfiniBand® Trade Association of which Hewlett-Packard is a member. The trade association has generated specifications for a 100 Gb/s communications protocol for high-bandwidth, low latency server clusters. The same communications protocol can operate across all system components for computing, communications, and storage as a distributed fabric.
Page 20: Identifying Infiniband Fabric Components
— Security and partitioning — Integrated management tools • A strong technical future, with support for: — PCI cards: ◦ Voltaire SDR PCI-X ◦ Topspin/Mellanox SDR PCI-X Rev B; Mellanox SDR PCI-X Rev C ◦ Topspin/Mellanox SDR PCI-Express Rev B; Mellanox SDR PCI-Express Rev C ◦...
Page 21: Isr 9Xxxx-Series Interconnects
— A PCI bus adapter that is installed in a host, such as a server. — Any networked device that includes an InfiniBand-compliant HCA chip on its motherboard. • An interconnect (switch) connecting all devices (such as servers) in a switched point-to-point network.
Page 22: Identifying The Isr 9024 Interconnect (Sdr)
• Support only single data rate links (SDR). • Are not compliant with Restriction of Hazardous Substances. RoHS, European Union Directive 2002/95/EC, which restricts the use of specific hazardous materials found in electrical and electronic products. All applicable products in the EU market after July 1, 2006 must pass RoHS compliance.
Page 23: Identifying The Isr 9096 Interconnect
Figure 1-3 ISR 9024 S/D (RoHS Compliant) Interconnect Front Panel View The ISR 9024 S/D (RoHS Compliant) interconnect is a fully non-blocking interconnect with a theoretical throughput of 960 Gb/s. This device has the following physical and operational features: • A 1U chassis, designed for industry-standard racks.
Page 24: Identifying The Isr 9288 Interconnect
Important: Do not mix SDR and DDR components in an ISR 9096 chassis. Figure 1-5 ISR 9096 Interconnect Rear Panel View The ISR 9096 enables high performance applications to run on distributed nodes, share common storage and share network resources. This device has the following physical and operational features: •...
Page 25: Isr 9288 Interconnect Front Panel View
Figure 1-6 ISR 9288 Interconnect Front Panel View Important: Do not mix SDR and DDR components in an ISR 9288 chassis. The ISR 9288 enables high performance applications to run on distributed nodes, share common storage and share network resources. This device has the following physical and operational features: •...
Page 26: Understanding The Interconnect Management Capabilities
• Up to 11 router blade drawers, model sRBD that each support up to three router blades, which are either of the following models: — TCP/IP internet protocol router blade, model IPR that provides four Gigabit Ethernet ports, and an RJ-45 management port that includes both 10/100 Base-T Ethernet and serial ports.
Page 27 See the Voltaire InfiniBand Fabric Management and Diagnostic Guide and the HP Cluster Platform InfiniBand Fabric Management and Diagnostic Guide for information on how to launch and use the management interfaces. 1.7 Understanding the Interconnect Management Capabilities...
Page 29: Installing And Maintaining The Isr 9024 Interconnect
2 Installing and Maintaining the ISR 9024 Interconnect The hardware maintenance operations described in this chapter require appropriate training and knowledge of safety procedures. The safety procedures for Cluster Platform are described in the HP Cluster Platform Site Preparation Guide. The following interconnect maintenance topics are described: •...
Page 30: Internally Managed Isr 9024 Front Panel
— DC to DC conversion. — Temperature sensing mechanism. • The management card, which is a processor PCI mezzanine card (PrPMC), is installed only in an internally managed ISR 9024. It provides the following features: — A CPU, operating at 400 MHz. —...
Page 31: Isr 9024 Management Card
Power supply module that converts AC to DC supply. One or two power supplies are installed. Serial port connector that provides access to the serial CLI console interface. Ethernet RJ-45 port that provides management interface over a local network (The management data traffic is called out-of-band, because it does not pass over the InfiniBand high-speed network).
Page 32: Externally Managed Isr 9024 Front Panel
2.1.3 Externally Managed ISR 9024 Front Panel In this configuration, the I²C port panel replaces the management card panel. The I²C port provides access to channel management functions. Figure 2-3 shows the front panel of the externally managed configuration. Figure 2-3 Externally Managed ISR 9024, Front Panel Figure 2-3 shows the following front panel features: Captive fasteners retaining the power supply unit.
Page 33: Cooling And Airflow
The following features are shown: Indicator (green LED ) showing the physical status of each InfiniBand link. Indicator (amber LED ) showing the logical status of each InfiniBand link. Reset button: • Press and hold this button for more than 2 seconds, but less than 6 seconds to reset the management card function without affecting data traffic through the high-speed network.
Page 34: Mounting The Isr 9024 In The Rack
Open the small box and verify its contents against the packing list as follows. Some parts are optional, and might not be included in all shipments: Verify that the chassis is the correct configuration; internally managed or externally managed. Identify the rail kit consisting of: Four angle brackets.
Page 35 Determine the correct mounting location for the interconnect in your model of cluster platform. The position is specified as a U-location, which relates to the marked locations on the vertical columns an a HP 10000-series rack. Each U-location has three holes; top, middle, and bottom.
Page 36: Installing The Isr 9024 Power Supply Unit
2.4 Installing the ISR 9024 Power Supply Unit An ISR 9024 contains either one or two power supplies in the side bays of the unit chassis, as shown in Figure 2-3. Warning: To maintain the correct cooling air flow, you must fill an unused power supply bay by using the supplied blank filler panel Use the following procedure to install or remove a power supply.
Page 37 The PS LED is flashing. The PSU is not properly plugged in or not completely seated: • Check the power cord and its connections. Ensure that the cord is connected to the correct inlet in single-PSU configurations. • Check the installation of the PSU module. The SYS LED does not illuminate.
Page 39: Installing And Maintaining The Isr 9024 S/D Interconnect (Rohs Compliant)
3 Installing and Maintaining the ISR 9024 S/D Interconnect (RoHS Compliant) The hardware maintenance operations described in this chapter require appropriate training and knowledge of safety procedures. The safety procedures for HP Cluster Platform are described in the HP Cluster Platform Site Preparation Guide. The following interconnect maintenance topics are described: •...
Page 40: Internally Managed Isr 9024 S/D-M Front Panel
The ISR 9024 S/D chassis is a 1U-high 19-inch wide rack mounted ANSI, ETSI and RoHS-compliant chassis that contains the following modules: • The base board, which hosts the InfiniBand 4X SDR or DDR switch chip and provides the following: —...
Page 41: Isr 9024S/D-M Management Card
Figure 3-1 Internally Managed ISR 9024 D–M, Front Panel Figure 3-1 shows the following front panel features: Power supply indicator. Hot-swappable power supply. Fan unit indicator. Hot-swappable fan unit (contains two fans for high-availability) with auto-heat sensing to allow for silent fan operation. Front-to-rear cooling. Reset button.
Page 42: Isr 9024 S/D Rear Panel
Figure 3-2 Externally Managed ISR 9024 S/D, Front Panel Figure 2-3 shows the following front panel features: The following list corresponds to the callouts shown in Figure 3-2. Power supply indicator. Power supply module. Fan unit indicator. Fan unit. Reset button. System status LEDs include: •...
Page 43: Unpacking The Isr 9024 S/D Interconnect
• Info • Power IEC power receptacle. 3.2 Unpacking the ISR 9024 S/D Interconnect The chassis is shipped in a double wall carton, surrounded by shock absorption material as shown in Figure 3-4. Figure 3-4 ISR 9024 S/D Packaging Use the following procedure to unpack the ISR 9024 S/D (callout 3 in Figure 3-4): Place the carton on a secure work platform at a safe working height.
Page 44: Mounting The Isr 9024 S/D In The Rack
Open the small box (callout 4 in Figure 3-4) and verify its contents against the packing list as follows. Some parts are optional, and might not be included in all shipments: Verify that the chassis is the correct configuration; internally managed or externally managed.
Page 45: The Isr 9024 S/D Fan Unit
Attach the front L-brackets to either side of the chassis, using the three 8/32-in screws provided, as shown by callout 1 in Figure 3-5. Attach the rear L-brackets loosely to each rail, as shown by callout 2 in Figure 3-5. The screws should be just loose enough that you can easily adjust the position of the bracket on the rail.
Page 46: Replacing The Fan Unit
3.4.1 Replacing the Fan Unit In normal operation, the two fans work at 50% utilization. In case of fan failure or high temperature detection, the fans go into Turbo mode. In case of fan failure, the fan drawer LED and the PS/FAN LED on the rear panel blink.
Page 47: Operating The Isr 9024 S/D Interconnect
Position the rear (connector end) of the power supply in the empty bay. Holding the power supply level with the chassis, slide it into the slot until it meets resistance at the chassis connector. Do not force the unit if you feel resistance, ensure that it is level and square with the bay.
Page 48 The SM LED does not illuminate. The SM LED does not illuminate if the ISR 9024 S/D is externally managed (there is no management card present). If the SM indicator does not illuminate on an internally managed ISR 9024 S/D-M, the management card is not running the subnet management software.
Page 49: Installing And Maintaining The Isr 9096 And Isr 9288 Interconnects
4 Installing and Maintaining the ISR 9096 and ISR 9288 Interconnects This chapter describes the installation and maintenance of the Voltaire ISR 9096 and the ISR 9288 interconnects. The actual product name will be used when referring to a specific feature or procedure.
Page 50: Isr Isr 9Xxx Interconnect Architecture
— ISR 9096: up to 96 InfiniBand 4X (10 Gb/s, 20 Gb/s DDR) ports or 32 12X (30 Gb/s, 3.84 Tb/s DDR) ports. — ISR 9288: up to 288 InfiniBand 4X (10 Gb/s, 20 Gb/s DDR) ports or 96 12X (30 Gb/s, 3.84 Tb/s DDR) ports.
Page 51: Field Replaceable Units
from the front of the chassis to the Midplane. The sLB and Router Module signals are routed from the rear of the chassis to the midplane. The ISR 9096 has one non-swappable board, and the ISR 9288 has three non-swappable boards.
Page 52: Isr Isr 9Xxx Control Panels And Reset Buttons
• Low level link and PHY function. • Subnet management agent (SMA). • Performance management agent (PMA). You upgrade the firmware by using the Ethernet, RS232, or I²C port to connect to the system on the sMB board. You can also upgrade the firmware in-band via any of the InfiniBand line board ports.
Page 53: Unpacking The Isr 9288
Caution: Due to the weight of the chassis, two people are required to remove it from the shipping container. Before unpacking, ensure that the shipping containers are not damaged. Damage to shipping containers (other than the normal wear-and-tear of shipping) indicates that the contents might have been subjected to excessive shock.
Page 54: Documentation And Cd Location
Remove all of the above components and the packing foam from the top of the crate. Remove the documentation and the CD wrapped in an antistatic bag on top of the chassis. Figure 4-2 Documentation and CD Location Item Description Getting Started Short Guide ISR 9288 Product CD and other CDs, according to system configuration Open up the front door of the wooden crate by releasing the clamps...
Page 55: Unpacking The Isr 9096
Remove the chassis from the box and place it on a flat surface. Open any additional boxes and remove the items inside. Verify the contents, which depend on the configuration of your cluster. 4.2.2 Unpacking the ISR 9096 Use the following procedure to unpack the ISR 9096 container: Remove the top cover of the box.
Page 56: Verifying The Shipment
Figure 4-5 ISR 9096 Accessories Box Item Description Ground stud kit Console cable ISR 9096 Product CD and other CDs, depending on system configuration Power cable Tray screws kit Configurations that include cable management will also have the following items in the accessories box: •...
Page 57: Isr 9288 Front View
Note: Depending on the reason for the chassis replacement, the shipment might contain preinstalled modules. In other instances (such as a failed mid-plane in the original chassis) you might need to depopulate the failed chassis, and transfer the modules to the new chassis after you install the chassis in the rack.
Page 58: Repacking The Chassis
Figure 4-7 ISR 9288 Rear View Item Description 24-port line board Line board latch sRBD router blade drawer Router blade, installed in the sRBD router blade drawer One of 5 redundant power supplies sCTRL control module 4.3 Repacking the Chassis If you need to return a defective chassis for servicing or replacement, it must be packed for shipping in its original container.
Page 59: Resources Required To Perform An Installation
14U in the rack, and the ISR 9096 chassis occupies 6U in the rack. Under normal circumstances, the only reason for performing an installation is when you remove a complete chassis from its location in an existing HP Cluster Platform. Note the following constraints when replacing a chassis: •...
Page 60: Isr 9Xxx Rack Mount Procedure
The two L-brackets that are pre-assembled to the front sides of the interconnect chassis are also used to secure the chassis to the rack columns, as shown in Figure 4-9. Figure 4-9 ISR 9XXX Rack Mount Procedure Figure 4-10 shows a view of the correct assembly of each telescoping rail unit from the rear of the rack.
Page 61: Assembling 9Xxx Rails
Assemble each pair of rails, as shown in Figure 4-11. Figure 4-1 1 Assembling 9XXX Rails Place the slotted rail (callout 1 in Figure 4-11) on the outside of the larger rail (callout 3 in Figure 4-11) Secure the slotted rail loosely, using eight of the machine screws provided. Two of these screws are shown by (callout 2 in Figure 4-11)
Page 62: Attaching The Cable Management Hooks
Align the left cable management hook assembly with the top of the chassis as shown in Figure 4-12. Figure 4-12 Attaching the Cable Management Hooks For each of the 5 mounting holes in the hook assembly, clip an M6 cage nut into the back of the rack column.
Page 63: Installing The Cabling Clamps
4.4.3 Installing the Cabling Clamps If line boards are installed in slots 1 and 2 of the chassis (the topmost slots) or the number of cables connected to the interconnect exceeds 128 and the interconnect is installed in a standard 600mm wide cabinet, additional cabling clamps are required, as shown in Figure 4-13, which...
Page 64: Isr 9288 Chassis Front Panel Configuration
Table 4-1 ISR 9XXX Standard Configurations Description ISR 9288 ISR 9096 Fabric boards sFB-12 (up to four) sFB-4 (up to four) Fan units Horizontal: sFU-8 Vertical: sFU-4 Vertical: N/A Control (rear) sCTRL Power supply unit sPSU (up to five) sPSU (up to four) High-memory sMB for extended sMB-hi-mem fabric management...
Page 65: Maximum Infiniband Ports, No Srbd Installed
The front panel contains the following modules and features: A master sMB module. The sFU-8 eight-fan horizontal cooling module. The sFU-4 four-fan vertical cooling module. Up to 4 sFB-12 fabric boards,. A redundant slave sMB module. L-brackets that you use to attach the chassis to the rack columns. In an HP Cluster Platform, the ISR 9288 rear panel is configured as shown in Figure 4-15 Figure...
Page 66: Populated I/O Drawer
Figure 4-16 Populated I/O Drawer The following features are shown in Figure 4-16: sRBD router blade drawer. Blade installed in the sRBD, Location of the locking levers for inserting and removing sRBD router blade drawers sLB-24 InfiniBand line board. In an HP Cluster Platform, the ISR 9096 front panel is configured as shown in Figure 4-17: Figure 4-17 ISR 9096 Chassis Front Panel Configuration...
Page 67: Post-Installation Tasks
Figure 4-18 ISR 9096 InfiniBand Ports The following features are shown in Figure 4-18: Up to 4 sLB-24 InfiniBand line boards Locking levers for inserting and removing the line boards Blade installed in the sRBD sRBD router blade drawer A single sCTRL management module Ethernet slots 4.6 Post-installation Tasks Following the installation of the interconnect, your next tasks are as follows:...
Page 69: Hp 4X Ddr Ib Switch Module For C-Class Bladesystems Overview
5 HP 4x DDR IB Switch Module for c-Class BladeSystems Overview The 4x DDR IB switch module is a double-wide switch module for the HP BladeSystem c-Class enclosure. It is based on the Mellanox 24-port InfiniScale III 4x DDR InfiniBand switch chip. When an IB mezzanine HCA is plugged into the c-Class server blade, the mezzanine HCA is connected to the IB switch through the mid-plane in the c-Class enclosure.
Page 70: Subnet Manager
Note: You must remove the c-Class blade cable management bracket to remove or replace a 4x DDR IB switch module installed in the back of a c-Class BladeSystem enclosure. To remove the bracket, reverse the installation procedure described in the c-Class Blade Cable Management Bracket Installation Guide.
Page 71: Installing Field-Replaceable Units In The Isr 9Xxx Chassis
6 Installing Field-Replaceable Units in the ISR 9XXX Chassis The ISR 9288 chassis is pre-assembled with the fan units (sFU-4 and sFU-8) and the rear panel sCTRL board. You must install all boards in their designated slot to enable their full functionality. All boards are hot swap enabled (for some boards, only when the hot swap indicator LED is illuminated) and you can insert and extract these boards without turning off electrical power.
Page 72: Installing Or Replacing The Sfb-12 Fabric Board
Figure 6-1 Board Ejectors Figure 6-1 shows the top ejector latches on the fabric boards, and includes the following information: Security screws on each board complete the board seating, and lock of the board in place. The latch release button, which also electronically signals a board extraction request when you press it.
Page 73: Installing Or Replacing Slb-24 Line Boards
Table 6-1 sFB-12 Fabric Board LED Status Indication Description Link Each pair of LEDs indicates the status of six ports of each line board. The two LEDs in each pair are: • Green – The physical link is up when this LED is illuminated. •...
Page 74: Line Board Front Panel
You can also connect and disconnect InfiniBand cables while the ISR 9288 is powered on. If the node at the other end of a link is powered up, you will see the link status LEDs illuminate to indicate the link status. The rear of the ISR 9288 chassis provides slots for up to 12 line boards.
Page 75: Installing Or Replacing The Smb (Management) Board
Top left. Bottom right. Verify the status of the following LEDs: The power LED is illuminated. The info LED is not illuminated. Use the following procedure to extract a line board from the chassis: Unsecure and disconnect all cables. Release the security screws. Press the red buttons to unlock the ejectors.
Page 76: Installing And Replacing Power Supply Units
Table 6-3 sMB Board LEDs (continued) Indication Description Power (Green) Illuminated: The voltage levels of the board are within nominal values Off: There is a power problem, if the chassis is powered up. Otherwise, no power is applied to the chassis. Info This is a general purpose green LED for management use.
Page 77: Populating The Router Blade Drawer (Srbd)
Figure 6-5 Power Supply Unit Figure 6-5 shows the following features of the PSU: IEC Power Inlet. Fan vents. Power status signal LEDs. Locking screw. Table 6-4 lists and describes the power supply module LEDs. Table 6-4 Power Supply Unit LEDs Indication Description DC ON (Green)
Page 78: Overview Of The Fcr (Fibre Channel Router) Blade
• Four small form-factor SFP GBIC GbE ports, providing a fast link to an IP network for devices on the InfiniBand fabric • An RJ-45 management port. Figure 6-6 shows the front panel of the IPR blade. Figure 6-6 IPR Blade Front Panel Figure 6-6 shows the following features of the IPR blade: The link/activity LEDs, labelled 1 –...
Page 79: Installing A Router Blade
Figure 6-7 FCR Blade Front Panel Figure 6-7 shows the following features of the FCR blade: The link/activity LEDs, labelled 1 – 4, one for each labelled FC port. Gigabit Ethernet ports, labelled FC1 – FC4. 1, 2, and Sys LEDs that indicate the following status: •...
Page 80: Installing Or Replacing The Router Blade Drawer (Srbd)
Ensure that the lock handle is disengaged. Holding the module by the corners of its bezel, align it so that it is level and square with the front panel of the chassis. Slide in the module until most of the unit is in the slot and you feel the resistance of the midplane connector, and the lock handle begins to engage in its locking slot.
Page 81: Installing And Replacing The Sctrl Board
6.8 Installing and Replacing the sCTRL Board The sCTRL board provides the chassis management ports (Ethernet and RS-232 DB9 serial) and includes a reset button for the chassis. You install the sCTRL board on the rear panel of the ISR 9288 chassis.
Page 82: Installing The Sfu-4 Fan Module
Table 6-7 sCTRL Board LEDs (continued) LED label Description CM Active 2 (Green) If illuminated, the chassis manager is running on management card #2 Eth 1 and Eth 2 Each Ethernet port (which provides access to the management interface) has two LEDs, which provide the following status information: •...
Page 83: Installing The Sfu-8 Fan Module
Warning! Never remove the sFU-4 and sFU-8 at the same time; at least one fan module must be installed at all times. Use the following procedure to replace a defective fan module by hot-swap: Unpack and prepare all components on a convenient work surface adjacent to the chassis before you begin the swap procedure.
Page 84: Sfu-8 Fan Module Leds
Table 6-8 sFU-8 Fan Module LEDs LED Label Description Temp (Amber) This LED indicates an over-temperature fault on the chassis. When present, this usually indicates a problem with one of the fan modules. • Illuminated: An over-temperature fault is detected •...
Page 85: Cabling The Interconnect
7 Cabling the Interconnect Each model of HP cluster platform has a set of specific port-to-port cabling tables that describe the origin and destination of every link between the interconnect and the cluster nodes. When the cluster is integrated at the factory, the cables are installed and tested. Certain cables are then removed and the cluster racks are packaged for shipping.
Page 86: Cabling Tables
Caution: Do not bend the InfiniBand cables too sharply. The minimum bend radius is 4” (10 cm). • Follow the cabling procedure to avoid creating airflow dams, particularly for the ISR 9024, which employs passive cooling. • Consider the requirements and constraints of the installation location. For example, if cooling air is supplied through floor vents, avoid covering the vents.
Page 87: Cable Routing Procedure For The Isr 9096
Note: Cluster Platform Express (CPE) configurations use a cable routing procedure that is specific to CPE models. Follow the cabling instructions in the Cluster Platform Express Installation Guide. The cabling tables are specific to: • The total number of nodes in the cluster. •...
Page 88: Connecting The Isr 9096 Router Blades
— Whether the cluster is a bounded or federated design. — Whether the cluster is a full or constrained bandwidth design. — The presence of utility racks (UXR). Use the following procedure to route and connect cables: During factory integration, all cables are labelled with their origin and destination ports. Begin at the bottom of the chassis with line board 4.
Page 89: Connecting The Isr 9288 Router Blades
Figure 7-2 Routing Model for a Fully Configured ISR9288 Use the following procedure to route and connect cables: During factory integration, all cables are labelled with their origin and destination ports. Begin at the bottom with the lowest line board in the chassis, as shown in Figure 7-2.
Page 90: Connecting To Fcr Fc Ports
7.2.2 Connecting to FCR FC Ports Use the following procedure to connect a cable to the Fibre Channel port: Connect the small form-factor FC 1G/2G 850nm LC transceiver connector to the FC port on the FCR module. Connect the other end of the cable to the appropriate device. 7.3 Chassis Administrative Connections You make the initial CLI connection to a chassis through a PC connected to a serial port.
Page 91: Connecting The Management Port Serial Interface
Use the following procedure to connect to the management port Ethernet Interface: See the Ethernet Network Cabling Tables to determine where the management port connects to an in-rack ProCurve switch. Connect the end of the management cable with the single RJ-45 connector to the management port on the interconnect.
Page 93: Installing And Maintaining Hca Cards And Mezzanines
8 Installing and Maintaining HCA Cards and Mezzanines A feature of InfiniBand fabrics is that individual components can contain an InfiniBand host channel adapter (HCA), or as in the case of an HP BladeSystem server blade, a mezzanine HCA. Every device containing an HCA is a visible and manageable part of the InfiniBand fabric. The HCA can be an integrated circuit that is embedded on the circuit board of a device, (such as a storage array controller) or it can be in the form of a PCI card that you install in a server.
Page 94: Installing The Hca
Figure 8-1 HCA-400 PCI Card The following features of the HCA-400 PCI card are identified by the numbered callouts in Figure 8-1. An InfiniBand port, one of two. The indicator LEDs for physical (green) and logical (amber) link status. The card's metal bracket. Always handle the card by holding only the bracket. Do not touch the printed circuit board, card components or the gold-plated connector.
Page 95: Typical Hca-400 Pci Card Installation
server's maintenance documentation and understand the procedures for installing a PCI card before you begin installing the HCA. Caution: The HCA-400 is PCI-X 2.2 and PCI 2.1 compliant (32/64 bit, 66/100/133 MHz). All server models that are qualified by HP are tested and configured to meet the requirements of this HCA card. You cannot use these installation instructions with alternate server models, or with alternate configurations of supported servers.
Page 96 Configure the node out of the cluster, as described in the operating environment documentation. Shut down the node, as described in the User Guide for the server. Disconnect an existing InfiniBand cable link. This step might require the removal of cable management components that are specific to the model of server and the InfiniBand interconnect.
Page 97: Topspin/Mellanox Pci-X Hca
11. Attach the metal bracket to the bulkhead with the retaining screw that you removed in step 12. Close the server chassis and slide it back into the rack. 13. Reinstall any cable management components that you removed in step 4. 14.
Page 98: Topspin/Mellanox Pci-Express Hca
Figure 8-3 Topspin/Mellanox SDR PCI–X HCA Features of the Topspin/Mellanox PCI-X HCA include: • Two 4x InfiniBand 10 Gb/s ports (40 Gb/s InfiniBand bandwidth full duplex – 2 ports x 10 Gb/s/port x 2 for full duplex). • 64-bit PCI-X v2.2 133Mhz •...
Page 99: Mellanox Pci-X Hca (Sdr)
The LED ports on the Topspin/Mellanox PCI-Express HCA blink to indicate traffic on the InfiniBand link on initial setup. The installation and LEDs of the Topspin cards are similar to that of the Voltaire HCAs described previously in this chapter. 8.5 Mellanox PCI-X HCA (SDR) The Mellanox PCI-X HCA supports InfiniBand protocols.
Page 100: Mellanox Pci-Express Hca (Sdr)
Port 2 LED Name Physical Link - Green Data Activity - Yellow 8.6 Mellanox PCI-Express HCA (SDR) The Mellanox PCI-Express HCA supports InfiniBand protocols. It is single data rate (SDR) card with two 4x InfiniBand 10 Gb/s ports and 128 MB memory. Figure 8-6 shows the Mellanox PCI-Express HCA.
Page 101: Mellanox Memory Free Pci-Express Hca (Sdr)
no data is being passed. The activity link blinks when data is being passed. If the LEDs are not active, either the physical or the logical (or both) connections have not been established. Port 1 LED Name Physical Link - Green Data Activity - Yellow Port 2 LED Name...
Page 102: Leds
• EEPROM used for storing Vital Product Data (VPD) • Embedded InfiniRISC™ Processors for Management & Subnet Management Agent (SMA) • Integrated Physical Layer SerDes • Integrated GSA (General Service Agents) • Low-Latency Communication Technology • Flexible Completion Mechanism Support (Completion Queue, Event, or Polled operation) •...
Page 103: Leds
• Hardware support for up to 16 million QPs, EEs and CQs • Memory Protection and Translation Tables fully implemented in hardware • IB Native layer 4 DMA hardware acceleration • Multicast support • Programmable MTU size from 256 to 2K bytes •...
Page 104: Mellanox Pci-Express Ddr Hca Leds
Figure 8-9 Mellanox PCI–Express HCA (DDR) Features of the Mellanox PCI-Express DDR HCA include: • Two 4X InfiniBand copper ports for connecting InfiniBand traffic (4X IB connectors) • 4X port supports 20 Gb/s • Third-Generation HCA core • On board DDR SDRAM memory (memory configurations vary) •...
Page 105: Hp Hpc 4X Ddr Ib Mezzanine Hca
8.9.2 HP HPC 4x DDR IB Mezzanine HCA HP Cluster Platform supports a server blade mezzanine HCA card that has the following features: • 4x DDR • Single port • PCI–Express • 20 Gb/s • Memory-free The HPC 4x DDR IB Mezzanine HCA ships with the Voltaire GridStack software. Voltaire GridStack is a comprehensive set of drivers and protocols that enable applications to utilize the RDMA features in the hardware.
Page 107: Postinstallation Troubleshooting And Diagnostics
9 Postinstallation Troubleshooting and Diagnostics This chapter provides information on the interconnect firmware's logging and monitoring functions. Use these functions to perform initial fabric debugging and confirm device failures if you encounter a problem that is indicated by the interconnect or HCA LED status arrays. This is not a definitive list of cluster diagnostics, which are described in the Voltaire InfiniBand Fabric Management and Diagnostic Guide and the HP Cluster Platform InfiniBand Fabric Management and Diagnostic Guide.
Page 108: Debugging A Fabric Failure By Using Performance Management (Pm)
Check the Link LEDs on the line boards front panel for the various ports. In the event of a system lock-up in the ISR 9XXX, use the reset button to reset and reboot the system. The system has two reset buttons: one is located on the front panel of the sFU-8 fan assembly module and one of the front panel of the sCTRL board.
Page 109 The following status values are associated with port failures: • BROKEN_LINK One end of the physical link is up, but the other end is unreachable. • INVALID_LINK The port shows different connectivity on both sides of the link. • DUPLICATE_HCA_GUID Another HCA (or interconnect) exists with the same GUID.
Page 111: A Isr 9024 Interconnect Specifications
A ISR 9024 Interconnect Specifications The IBTA 1.1 compliant InfiniBand technical specifications of the ISR 9024 are as follows: Feature Technical Specification InfiniBand Ports 4X and/or 12X factory configurable copper ports Up to 24 4x (10 Gb/s) InfiniBand non-blocking full-wire speed ports 140-nanosecond single-hop port to port switching latency Overall throughput of up to 480 Gb/s.
Page 112 Feature Technical Specification Externally managed over 400,000 hours Environmental Operating Humidity 15% to 80%, non-condensing Altitude 0 to 9843 ft (3000m) Environmental Storage Temperature 13°F to 185°F (-25°C to +85°C) Humidity 5% to 90% non-condensing Altitude 0 ft to 15,000 ft (4570 m) ISR 9024 Interconnect Specifications...
Page 113: B Isr 9024 S/D Interconnect Specifications (Rohs Compliant)
B ISR 9024 S/D Interconnect Specifications (RoHS Compliant) The technical specifications of the ISR 9024 S/D interconnect (RoHS Compliant) are as follows: Feature Technical Specification InfiniBand Ports 24 4X Single Data Rate (SDR – 10 Gb/s) ports, or 24 4X Dual Data Rate (DDR – 20 or 10 Gb/s auto-negotiate) Interconnect options: copper, with optional support for optical adaptors on top row 12 ports Indicators: physical and logical status...
Page 114 Feature Technical Specification Power Rating Power consumption*: ISR 9024S — 51W, max. ISR 9024S-M — 63W, max. ISR 9024D — 58W, max. ISR 9024D-M — 69W, max. BTU/hour = Watts x 3.413 Power Factor: 90 Vac/60 Hz/Max Load = 0.998 120 Vac/60 Hz/Max Load = 0.997 230 Vac/60 Hz/Max Load = 0.973 264 Vac/60 Hz/Max Load = 0.957...
Page 115 Feature Technical Specification EN55022:98/EN55024:98/EN61000-3-2:00/ EN61000-3-3:95 This device complies with par 15 of FCC rules. Operation is subject to the following two conditions: (1) This device may not cause harmful interference (2) this device must accept any interference received. VCCI-A ICES-003 C-Tick RoHS RoHS5...
Page 117: C Isr 9096 Interconnect Specifications
C ISR 9096 Interconnect Specifications The technical specifications of the ISR 9096 are as follows: Feature Specification Chassis Architecture • Four slots, each of which can accommodate one line board or one router blade drawer (hosting up to 3 router blades). •...
Page 118 Feature Specification Router Blade Drawer You can install up to three of the following InfiniBand form factor router blades in each router blade drawer: • Voltaire IPR (IP Router): — IPR InfiniBand Form Factor for integration with ISR 9096, providing high-speed layer 4-7 switching and TCP Termination —...
Page 119: D Isr 9288 Interconnect Specifications
D ISR 9288 Interconnect Specifications The technical specifications of the ISR 9288 are as follows: Feature Specification Chassis Architecture Based on passive Midplane boards which allow the IB differential signals, control signals and power connectivity between the different boards in the chassis. Cables for InfiniBand, management, and power all located at the rear.
Page 120 Feature Specification Router Blades You can install up to three of the following InfiniBand form factor router blades in each router blade drawer (sRBD): • FCR InfiniBand form factor for integration with ISR 9288 and OEM system: — Four 1G/2G Fiber Channel ports —...
Page 121 Feature Specification Weight 110 to 187 lbs (50 to 85 kg), depending on configuration Environmental • Ambient Operating Temperature 32° to 113°F (0° to 45°C). • Operating Humidity 15 to 80%, non-condensing. • Operating Altitude 0 to 9843 ft (3000m). •...
Page 123: E 4X Ddr Ib Switch Module Specifications
E 4x DDR IB Switch Module Specifications The technical specifications of the 4x DDR IB Switch Module are as follows: 4x DDR IB Switch Module Specifications Compliance IBTA version 1.1 compatible ROHS-R5 General Specifications Communications Processor MT25204A0-FCC-D, InfiniHost III Lx On-chip memory 4 KB MTU Data path...
Page 125: F Hca Pci Card Specifications
F HCA PCI Card Specifications Appendix F includes specifications for the following HCA PCI cards and mezzanine HCAs: • Voltaire HCA–400 PCI Card Specifications (Section F.1). • Topspin/Mellanox PCI-X HCA Specifications (Section F.2). • Topspin/Mellanox PCI-Express HCA Specifications (Section F.3). •...
Page 126: Topspin/Mellanox Pci-X Hca Specifications
Feature Specification Physical Characteristics Low-profile MD2. Dimensions(H x D) 2.5 in. (64 mm) x 6.7 in. ( 170 mm). Environmental, Operating Temperature: 32 Deg to 122 Deg F (0 Deg to 50Deg C) Humidity: 15 to 80%, non-condensing. Altitude: 0 to 9843 ft (3000m). Environmental, Storage Temperature: -13 Deg to 158 Deg F (-25 Deg to 70 Deg C).
Page 127: Topspin/Mellanox Pci-Express Hca Specifications
64–Bit Linux: Red Hat Enterprise Linux AS 3, 4 SuSE LINUX Enterprise Server 8, 9 F.3 Topspin/Mellanox PCI-Express HCA Specifications This section includes the specifications for the Topspin/Mellanox PCI-Express HCA. The HCA offers two 4x InfiniBand 10 Gb/s ports providing an aggregate 40 Gb/s InfiniBand bandwidth full duplex (2 ports x 10 Gb/s/port x 2 for full duplex).
Page 128: Mellanox Pci-X Hca Specifications
F.4 Mellanox PCI-X HCA Specifications The specifications listed below cover the Mellanox PCI-X SDR HCA. Physical Size: 2.5 in. x 6.6 in. Air Flow: 200 LFM @55°C 10 Gb/s Connector: InfiniBand Dual Copper MicroGigaCN Power and Environmental Voltage: 12V, 3.3V Maximum Power: 9.2W Temperature:...
Page 129: Mellanox Memory Free Pci-Express Hca (Sdr) Specifications
Safety: IEC/EN 60950-1:2001 ETSI EN 300 019-2-2 Environmental: IEC 60068-2- 64, 29, 32 F.6 Mellanox Memory Free PCI-Express HCA (SDR) Specifications The specifications listed below cover the Mellanox Mem-Free PCI-Express HCA (SDR). Physical Size: 54mm x 102mm (2.13 in. x 4 in.) Air Flow: 200 LFM @55°C 10 Gb/s Connector:...
Page 130: Mellanox Pci-Express Ddr Hca Specifications
F.8 Mellanox PCI-Express DDR HCA Specifications The specifications listed below cover the Mellanox PCI-Express DDR HCA. Physical Size: 2.5in. x 6.6in. Air Flow: 200 LFM @55°C 4X 20 Gb/s Connector: InfiniBand (Copper) Power and Environmental Voltage: 12V, 3.3V Maximum Power: Temperature: 0°...
Page 131 4x DDR IB Mezzanine HCA Specifications Non-operating Temperature 40 to 700 C. Non-operating Humidity 5% to 95% (non-condensing) Power requirement 1.35 A at 3.3V max (4.5W) Emissions Classifications FCC CFR 47 Part 15 Class A CISPR 22 Class A ICES-003 Class A VCCI Class A ACA CISPR 22 Class A F.9 HPC 4x DDR IB Mezzanine HCA Specifications...
Page 133: Index
Index administrative cabling , 90 Symbols Ethernet tables, 85 12x ports ISR 9024 administrative, 90 ISR 9024, 30 ISR 9096 administrative, 90 24-port 12x, 29 ISR 9288 administrative, 90 24-port 4x, 29 sCTRL, 90 4x DDR IB mezzanine HCA, 105 sMB, 90 specifications, 130 tables, 85...
Page 134 debugging, 108 fabric topology , 29 DB-9 adapter fan unit, SFU-2, 24 ISR 9024, 30, 31 (see also sFU-2) ISR 9024 S/D, 40 fan unit, SFU-4, 26 DB-9 connector, 30 (see also sFU-4) DB9 cable, 26 fan unit, SFU-8, 24, 26 device (see also sFU-8) management, 26...
Page 135 HCA 4x DDR IB mezzanine, 105 HCA, 94 HCA mellanox memory free PCI-Express DDR, 102 inter-integrated circuits, 30 HCA mellanox memory free PCI-Express SDR, 101 (see also I²C) HCA mellanox PCI-Express SDR, 100 interconnect HCA Mellanox PCI-X (SDR), 99 cabling, 85 HCA port interconnect administration, 26 GUID, 109...
Page 136 ISR 9024 S/D architecture, 50 cooling, 45 backplane, 51 DB-9 adapter, 40 building blocks, 50 externally managed, 39 cable management hooks, 62 front panel, externally managed, 41 cabling clamps, 63 installing PSU, 46 cabling procedure, 88 internally managed, 39 control panels, 52 master mode, 40 fabric board, 50 (see also internally managed)
Page 137 ISR 9xxxx-series interconnect, 21 VoltaireVision fabric manager, 26 Web browser, 26 I²C, 30 Port on ISR 9024 S/D, 41 management board, 24, 25, 50 Port on ISR9024, 32 (see also sMB) management card ISR 9024, 31 Java Web client, 26 ISR 9024S/D-M, 41 management information base, 26 (see also MIB)
Page 138 ISR 9288, 49 PSU, 24, 26, 50 Installing in ISR 9024 S/D, 46 Installing in ISR9024, 36 packaging ISR 9024, 30 recycling, 17 ISR 9096, 51 packing ISR 9288, 51, 76 ISR 9096, 58 LEDs, 77 ISR 9288, 58 PCI card, 93 PCI cards, 20 queue pairs, 20 PCI mezzanine card, 30...
Page 139 installing, 79 sMB management board, 24, 25 router blade drawer, 24, 26, 50 SNMP, 26 (see also sRBD) specifications 4x DDR IB mezzanine HCA, 130 4x DDR IB switch module, 123 safety HCA, 125 battery, 16 ISR 9024, 111 burn injuries, 16 ISR 9024 S/D, 113 caution, 14 ISR 9096, 117...
Page 140 specifications, 127 Topspin/Mellanox PCI-X HCA specifications, 126 torque setting, 35, 45 transport layer, 20 troubleshooting clear bad ports, 109 fabric failure, 108 failed ports, 108 HCA port GUID, 109 ISR 9024, 36, 47 NodeInfo, 108 port GUID, 108 port status values, 109 ports, 108 U-location, 35, 44 unpacking...

This manual is also suitable for:

Cluster platform

HP Cluster Platform Interconnects v2010 User Manual

1 Infiniband Technology Overview

2 Installing and Maintaining the ISR 9024 Interconnect

3 Installing and Maintaining the ISR 9024 S/D Interconnect (Rohs Compliant)

4 Installing and Maintaining the ISR 9096 and ISR 9288 Interconnects

5 HP 4X DDR IB Switch Module for C-Class Bladesystems Overview

6 Installing Field-Replaceable Units in the ISR 9XXX Chassis

7 Cabling the Interconnect

8 Installing and Maintaining HCA Cards and Mezzanines

9 Postinstallation Troubleshooting and Diagnostics

Quick Links

Troubleshooting

Related Manuals for HP Cluster Platform Interconnects v2010

Summary of Contents for HP Cluster Platform Interconnects v2010

This manual is also suitable for:

Table of Contents