If everyone is thinking the same, someone isn't thinking

Lori MacVittie

Subscribe to Lori MacVittie: eMailAlertsEmail Alerts
Get Lori MacVittie via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Cloud Computing, Virtualization Magazine, Infrastructure On Demand, PC Security Journal, Cloudonomics Journal, Infrastructure 2.0 Journal, Security Journal, Datacenter Automation, CIO/CTO Update, Telecom Innovation

Blog Feed Post

The Rise of the Out-of-Band Management Network

Cloud and virtualization share a common attribute: dynamism. That dynamism comes at a price…

Cloud and virtualization share a common attribute: dynamism. That dynamism comes at a price...

Let’s talk about management. Specifically, let’s talk about how management of infrastructure impacts the network and vice-versa, because there is a tendency to ignore that the more devices and solutions you have in an infrastructure the more chatty they necessarily become.

In most organizations management of the infrastructure is accomplished via a management network. This is usually separate from the core network in that it is segmented out by VLANs, but it is still using the core physical network to transport data between devices and management solutions. In some organizations an “overlay management network” or “out-of-band” network is used. This network is isolated – physically – from the core network and essentially requires a second network implementation over which devices and management solutions communicate. This is obviously an expensive proposition, and not one that is implemented unless it’s absolutely necessary. Andrew Bach, senior vice president of network services for NYSE Euronext (New York Stock Exchange) had this to say about an “overlay management network” in “Out-of-band network management ensures data center network uptime

Bach said out-of-band network management requires not only a separate network infrastructure but a second networking vendor. NYSE Euronext couldn't simply use its production network vendor, Juniper, to build the overlay network. He described this approach as providing his data center network with genetic diversity.

"This is a generalized comment on network design philosophy and not reflective on any one vendor. Once you buy into a vendor, there is always a possibility that their fundamental operating system could have a very bad day," Bach said. "If you have systemic failure in that code, and if your management platform is of the same breed and generation, then there is a very good chance that you will not only lose the core network but you will also lose your management network. You will wind up with absolutely no way to see what's going on in that network, with no way to effect repairs because everything is dead and everything is suffering from the same failure."

"Traditionally, in more conventional data centers, what you do is you buy a vendor's network management tool, you attach it to the network and you manage the network in-band – that is, the management traffic flows over the same pipes as the production traffic," Bach said.

Most enterprises will manage their data center network in-band by setting up a VLAN for management traffic across the infrastructure and dedicating a certain level of quality of service (QoS) to that management traffic so that it can get through when the production traffic is having a problem, said Joe Skorupa, research vice president at Gartner.

Right now most enterprises manage their infrastructure via a management network that’s logically separate but not physically isolated from the core network. A kind of hybrid solution. But with the growing interest in implementing private cloud computing that will certainly increase the collaboration amongst infrastructure components and a true out-of-band management implementation may become a necessity for more organizations – both horizontally across industries and vertically down the “size” stack.


In our discussions around Infrastructure 2.0 one of the focal points is always collaboration; that is, infrastructure needs to collaborate in order to automatically respond to changes in the architecture. For example, when a virtual machine is launched there needs to be communication with the IPAM (IP Address Management) solution, which needs to share the  new IP address with, well, every other component in the infrastructure. Each component needs to know about the new instance because it might need to apply a policy or take action based on the networking component or application that is running inside the VM. The Load balancer needs to know whether to add the application to an existing resource pool, and whether or not there are security, acceleration, or access policies that must be applied to it. The network firewall may need to know about the virtual machine or application so it can allow or deny access appropriately, and other applications may need to know it’s available, as well.

All this “knowing” about an application being launched – particularly its network location - results in a flood of traffic on the network. In a cloud computing, a.k.a. shared resource, environments, this traffic must compete with other traffic and vice versa.

You might be thinking at this point that a physically separate, out-of-band management network would solve the problem. Not entirely. Some of the traffic created from dynamic environments is going to require the use of the “core” network. For example, the load balancer must be able to (1) discover newly launched application instances on the network via IP address and (2) continually monitor the health of the application to ensure availability and properly distribute requests to enable scalability. The choice of health monitoring techniques can significantly impact the amount of traffic and load on the network. For example, many implementations end up using an ICMP “ping” to determine the availability of a given server (not the best approach in general to ensure application layer health, but it does ensure that the server, at least, is available and reachable).

Now, ICMP has some interesting properties that result in a heavier load on the network than one might expect:

ICMP messages are constructed at the IP layer, usually from a normal IP datagram that has generated an ICMP response. IP encapsulates the appropriate ICMP message with a new IP header (to get the ICMP message back to the original sending host) and transmits the resulting datagram in the usual manner.

For example, every machine (such as an intermediate router) that forwards an IP datagram has to decrement the time to live (TTL) field of the IP header by one; if the TTL reaches 0, an ICMP Time to live exceeded in transit message is sent to the source of the datagram.

Internet Control Message Protocol (ICMP) via Wikipedia

So every device – virtual or physical - between the load balancer and its destination server must “touch” the ICMP packet to decrement the TTL field. Not a big deal usually, but multiply that requirement to modify a single packet times hundreds and thousands of servers and remember that health checks must be configured to occur on a fairly short time interval to enable the load balancer to react to the volatility inherent in dynamic environments. Remember, too, that ARP (Address Resolution Protocol) requests and replies are constantly flying around the same network as the load balancer must keep track of both IP address and MAC address, which means using ARP to gather that data. That’s a lot of processing and packets on the wire at any given second. Which is going to impose a heavier burden on the network and, in turn, potentially degrade performance for end-users because their requests have to traverse the same network.


That’s just health checks and user-initiated requests. Consider what happens to the network when other collaborative (infrastructure 2.0) messages are being exchanged, and then remember that we have to multiply the amount of traffic by hundreds to account for resources being shared by more than one application. The amount of traffic that could be generated is staggering and will certainly have an impact on the overall efficiency of the network and its performance. Both the performance of management-related functions and the application need to be considered of equal importance. Degradation of management functions can inhibit scalability by slowing down the reaction to an increase in demand or an attack and render the application unavailable or compromised.

The result is that out-of-band management networks are likely to become a “must have” instead of a “nice to have” for a much broader spectrum of organizations. In the past only the largest of organizations would implement such a network primarily because of the costs – both CAPEX and OPEX – involved but as the adoption of automation and orchestration of virtualized resources internal to the organization’s data center continues to occur it will drive the need down into smaller organizations.

The alternative, of course, is to upgrade the network. The bad news is that option may be necessary anyway depending on the processing capabilities of your core network to handle the increased burden on the network that comes from a dynamic data center.

Read the original blog entry...

More Stories By Lori MacVittie

Lori MacVittie is responsible for education and evangelism of application services available across F5’s entire product suite. Her role includes authorship of technical materials and participation in a number of community-based forums and industry standards organizations, among other efforts. MacVittie has extensive programming experience as an application architect, as well as network and systems development and administration expertise. Prior to joining F5, MacVittie was an award-winning Senior Technology Editor at Network Computing Magazine, where she conducted product research and evaluation focused on integration with application and network architectures, and authored articles on a variety of topics aimed at IT professionals. Her most recent area of focus included SOA-related products and architectures. She holds a B.S. in Information and Computing Science from the University of Wisconsin at Green Bay, and an M.S. in Computer Science from Nova Southeastern University.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.