THE CURIOUS WANDERING GUY by Bob Brown

It seems as if everyone I talk to is trying to get a handle on monitoring. I also have found monitoring to be a complex topic, with a lot of misconceptions. Last time I blogged about monitoring, it was to dispel the myth that monitoring is a plug and play solution. The other big misconception I hear is that monitoring makes your facility more efficient. This also is a myth. Monitoring is a tool in your tool chest to help you accomplish your goals. The challenge is to clearly define your monitoring goals, so you can know which tools you should be using. As a form of discussion I like to divide monitoring into four levels:

Reactive Monitoring (Alarm & Notification) – In the most basic sense these are merely sensors and probes designed to send alert status via trouble lights, audible sirens or pager notification. These monitoring systems have minimum intelligence and usually work independently. Most modern equipment in technology rooms have notification functions as standard or optional features. Legacy equipment may require hardware upgrades but these too generally have alarm notification capabilities. But who gets the alarm? How do you ensure that the alarms reach the right personnel? Just imagine what would happen if you have a power interruption or mechanical failure and no one gets the heads up. Will an audible siren be heard? Is the call out contact information up-to-date?

Proactive Monitoring (Automated Alarms & Intelligent Response) – At this level of monitoring, alarms and notifications are standardized through a single monitoring management system. Bringing the concept of “A Single Pane of Glass”, it pulls together your isolated monitoring systems at local and remote locations. The alarm codes give you summaries or theories of the alert status, making people better able to react to certain events and in some cases respond. Now, this does not advocate conducting remote diagnoses from a NOC, nor the replacement of qualified experts. All alarm codes and responses must be outlined in a comprehensive set of Policies and Procedures, and somebody has to set this up and be responsible for its implementation.

Predictive Monitoring (Trending, Analytics and Visualization) – These are the most sophisticated real time monitoring systems with data tracking capabilities. They offer customized reports down to the most granular level. These reports serve as the baseline to measure efficiencies inside your facility. It is here where historical data, trending analytics and modeling come together for the optimization of space, power and cooling. While these systems can give you a snapshot of your PUE today, it cannot tell you what specific changes need to be performed to improve efficiency. It takes someone with knowledge to interpret these reports and enact an action plan.

Management Tools (Capacity Planning) – These are not necessarily monitoring systems, but lately I have seen some monitoring components added to these tools, just enough to make it confusing. These tools are useful and have a place next to monitoring systems. They range from asset management & location tracking systems, change control & service request systems, space planning software, capacity forecasting systems and computer fluid dynamic modeling systems. When used in conjunction with monitoring systems, they become an effective aide in the management of technology facilities.

I have yet to see a single monitoring solution able to perform all these roles. The reality is you will end up with a mix of the monitoring solutions I have outlined above. It makes sense to start at the beginning. First, set up Reactive Monitoring (alarms and notification), then add a module to allow Predictive Monitoring (intelligent response). Now you are ready for Predictive Monitoring (trending and analytics). With the money you are saving on reduced power usage, splurge on some Management Tools (capacity planning).

Regardless of what your monitoring goals may be, it is also important to align your policies and procedures with each individual strategy. Getting it right will require a big investment in time and training.

For anybody who has been around long enough, you may recall the first time you sent a fax. It was like nothing we had ever seen before; we could actually send documents over telephone lines. If you were not around then maybe you remember back in the 90’s, the birth of the internet and the first time you were able to surf online. The ability to send or receive emails, explore websites, and emerging online communities caused widespread addiction to information. These were cultural shifts that totally altered our daily lives both in the workplace and at home.

Widespread access to video conferencing systems has been limited for two main reasons: availability of bandwidth and costs. Legacy video standards depend upon huge amounts of bandwidth to deliver high quality video. The only way to ensure a robust network connection requires costly dedicated circuits and Quality of Service (QoS) agreements. The video conference room(s), the video conferencing equipment, service level agreements and the special environmental needs of the encoding/decoding appliances drive the total cost of ownership into the tens to hundreds of thousands of dollars.

However, new innovations are turning this industry on its head promising to make it available to everyone. Having tested an IP based video conferencing system on my laptop for about three months I can now say that I am a believer.

Next generation video conferencing systems eliminate the need for dedicated network lines, QoS agreements, and proprietary hardware. H.264-SVC is a new set of video compression standards that allows video scaling, which greatly reduces packet loss, for high quality video over low bandwidth connections. The quality is stunning when compared to legacy encoding/decoding standards. This leads to the possibility for video conferencing on an open network with near lossless results, and all at a reasonable cost.

For the first time, high quality multipoint video conferencing is accessible to the desktop level. Now you don’t need to reserve that big conference room. Your desk, home offices, remotely located employees and the mobile workforce can all have the same video conferencing experience you find in a large legacy system. Imagine what this can do for how your employees interact with one another. It opens up the possibility for people to remotely communicate face to face anytime, anywhere. It could be the end of the 9 to 5 workday as virtual offices extend across time zones and continents. What could this mean to your company culture?

What a time to be in technology. There are changes everywhere: virtualization, cloud computing, Power over Ethernet, innovations in efficiency for both power & cooling and the list goes on. Out of this changing landscape one trend stands out as the golden hammer, Monitoring.  While the concept seems so simple there is more confusion, false expectations, and unfulfilled promises about monitoring than any other data center topic today.

Some confusion can be contributed to the industry using “Monitoring” as a catch phrase to describe multiple ideas from alarm & notification systems, capacity planning systems, and trending analysis systems. These all fall in the realm of monitoring but taken in context they are very different from one another. Some software systems perform one function really well while others can perform any combination.

Let’s face it adopting new technology is difficult and overall I would have to say the monitoring industry hasn’t done enough to help rollout this new technology. Many companies that produce monitoring solutions view themselves simply as software houses or manufacturers. They don’t quote the total cost of installation. What about implementation or setup with all the separate components that may need bits and pieces added to communicate with the software?  Unfortunately, this leaves their customers in the lurch and trying to figure out for themselves how to grapple with setting up their new monitoring system. They often claim it is an easy install. It is far from easy.  Promises of increased efficiency, increased availability, optimization of IT assets & productivity in staff are empty claims without substantial professional services.

The physical installation of a monitoring system is only the first step. I hear story after story on companies that never use their monitoring system because the installation and programming process became much more complicated then what they expected. Or, companies gave up on their monitoring systems because they didn’t get the appropriate training or they never changed their internal policies and procedures on how to deal with the alarms. Or, they get the trending reports but don’t know what to do with the information. And the most common scenario, how the monitoring system will impact other departments or groups.

So here is my advice to you if you want to reap the rewards of an efficient data center, get real time data about your facility, or implement an automatic alarm & notification system. You are not buying an off the shelf piece of technology that continuously runs in the background. You are subscribing to change the very way your department and every group that touches your data center interacts, communicates and operates.

Monitoring can deliver on all its promises and improve the way your data center operates but you need to be prepared. It takes planning, coordination, communication and above all it takes a commitment of time and resources from all departments. It’s not plug n play.

I picked up the paper today and saw an article that struck me. In this downturned economy some people are packing up their station wagons and heading into California’s foothills hoping to strike it rich by panning for gold. Which got me to thinking about our current gold rush here in Silicon Valley. Now, we may not be panning for gold in the streets, but people across the industry are desperately trying to mine efficiency out their facilities in an attempt to save money.

The Efficient Data Center Summit hosted by Google on April 1st 2009 gave us a rare glimpse into the inner workings of truly state-of-the-art data center facilities and the results of unconventional engineering & design. I believe their designs and concepts to achieve an average PUE (Power Usage Effectiveness) factor of 1.21 are to be commendable and I stand & cheer.  But what does Google’s ability to achieve a PUE factor of 1.21 really tell the rest of us, and better yet, how can we benefit from it?  

It’s now common knowledge that a lower PUE can cut down on electricity costs, but how is this accomplished? First you need to benchmark your PUE as a starting point. Using trending and analytics you can make smart incremental adjustments to lower your PUE number. The incentives to be more efficient are numerous and the payoffs can be huge.

From an operational expenditure perspective the lure of having a low PUE speaks for itself.  But, transforming your data center efficiency rating is no easy task. The key is to know where to find inefficiencies and what to do once you locate them.

It is interesting to me that in spite of the potential operational savings there are still data center managers who feel they just don’t have the means or budget to spend the capital on upgrades, or they don’t know if they can justify the ROI to make the investments.

The cost of improvements that may be required will come back in a short amount of time and save you significantly more moving forward. Let’s say you manage a .5MW data center with a PUE factor of 2.00 and spend $569,400 annually on electricity to support your IT load. Reducing your PUE factor to 1.9 would potentially mean a 10% reduction in your support load’s electrical demands. Over ten years that is $569,400. Imagine what you could do with your budget if you didn’t have to pay for cooling or lights for a full year.

Which brings me to another thing that I encounter as a consultant all the time; people unwilling to spend capital money now that can save them tons of money down the road. Often this is because there are two different people in charge of capital expenditures and operation costs. The solution is to have better communication. If you can align your teams to see the common goal, you can easily lower your PUE, have a better, more efficient data center, and save money down the road. It really is a win, win.

So as those people head into the hills in search of the mother load, I’m looking around at Facilities and IT managers and asking, what kind of gold can you collect from your data center mine?

Historically it seems like there has always been a communication problem between IT & Facilities. This is because they don’t speak to each other often enough; and because they speak different languages.   I keep thinking back to that old George Carlin routine “Football vs. Baseball”

• In Facilities they measure kilowatts – In IT  they measure bandwidth and bytes
• In Facilities they deploy chillers  –  IT  deploys fans
• Facilities is responsible for the power and cooling systems that support your data centers – IT  racks and stacks hundreds or thousands of computers that support the virtual world, it is not visible to the users, but it can’t be down.

High density computing has likely sucked up all the available expansion capacity and we are left with a precarious balance between IT and Facilities that requires accurate capacity planning in an ever changing environment.

So who are these new superheroes that are able to align IT & Facilities? And, how are they able to drive them towards common corporate goals and business objectives?

Dave Morrison of Adobe Systems’ Critical Operation Manager Global
Dave had a pretty comfortable job as the Chief Engineer for the Adobe headquarters complex and he has accumulated a lot of Data Center experience over his career prior to his latest position. Dave is a likeable guy and does not carry around a big ego (probably a requirement for this role). Over the years he always had a good relationship with IT. In fact, when Adobe was planning for data center space somewhere around the globe, they would turn to Dave for facilities advice. Eventually Adobe asked Dave if he would accept a new role to plan and maintain Critical Operations globally. It was a leap out of his comfort zone, but Dave reports he is excited with the challenge every day, and looks forward to what is ahead.

Jeremy Rodriguez of VMware, Chief Efficiency Officer
Jeremy has to have one of the coolest job titles ever.  His job is specifically designed to be that liaison between the IT needs and the energy efficiency required by Facilities.  At a previous employer he worked in IT and was involved in a server virtualization project that sold on the benefits of virtualization and opened his eyes to the opportunity.  He managed to steer his career to VmWare and landed one of the more interesting job descriptions along the way. Jeremy has a lot of talent but he is especially good at illustrating the benefits of energy efficiency efforts. This is an important trait, the ability to communicate with management and coax the authorization and short term funding to realize the long term benefits. Much of Jeremy’s focus is on Data Centers since they are such energy hogs. VmWare is in an elite league. They have a model to build and operate Data Center space in a very efficient manner. 

Dean Nelson of Sun Microsystems’ Global Lab & Datacenter Design Services (GDS), Inc.
Dean has managed to move within Sun Microsystems in a variety of interesting roles. His latest gig has him leading a team that has danced between IT and Facilities to consolidate equipment and resources. As a result of their ability to reach across the aisles, this team is greatly reducing the amount of Data Center space, while also slashing electricity usage. Dean Nelson's Blog

Looking forward, Facilities and IT must work together. Not every organization is big enough to justify a new position, but take a cue from these guys, better communication results in better projects. You can improve your communication. I challenge you to look at your departmental relationships and start thinking about ways that you can improve the collaboration.