Over a million developers have joined DZone.

Using benchmarking vendors (Part 1 of 2)

DZone 's Guide to

Using benchmarking vendors (Part 1 of 2)

· Performance Zone ·
Free Resource

This installment of the benchmarking series focuses on using benchmarking vendors and will be in two parts. This part will focus on when you might want to use a benchmarking vendor, and the second part will focus on what to do once you find a vendor.

Benchmarking vendors, like all other vendor, should be held to a high standard. There are many vendors out there, and depending on what you are trying to accomplish, there are some really, really good options available. Regardless of what they say, though, there isn’t a single vendor out there specializing in every type of benchmarking need you may have. They might be able to execute all your benchmarking projects for you, but my first piece of advice is to do your homework to see if there is a specialist in your specific area of need.

When to use a vendorbenchmarking, vendor, partner, metrics

When in doubt, seek advice. Any benchmarking vendor worth the standard deviation of their square root will be happy to lend you advice on your project. The good ones know what they do well and when to refer you elsewhere, and they also care enough about the discipline to want to help you. Don’t expect them to spend significant hours with you, though, outside of an engagement, though. They have to stay in business. Nobody (vendor or client) wants a project to go poorly, but everybody also has to stay in business. Here are some instances when I think you should use a specialist for your benchmarking project.

  • You’re new to benchmarking – If this is your first benchmarking project, and there isn’t a benchmarking group within your organization. You want to make sure your first project is a learning experience for you, and you don’t have to learn everything the hard way. You also want to increase the likelihood of success of your first project; who wants to start with a false start or failure.
  • It is a complex project – When you are benchmarking a wicked problem or a process that spans the entire organization a vendor can help you navigate many of the pitfalls like ensuring you pick the right team and stakeholders, develop your communication plan, and (what many times is the most important role vendors play) being the “bad guy” to deliver news you might not be able to deliver. Vendors can take the heat because they get to leave the next day, you still have to work with the folks in your organization the next day.
  • It’s a high profile project – If the project is being sponsored at a very high level and the outcomes will be guiding organizational strategies. When all eyes are on you, you might want a partner to increase the chances of success.
  • The project involves very sensitive information – Whenever you are going to collect very sensitive information (i.e., cost-based, metric data), you might want a third-party to be an aggregator to blind the data for all participants.
  • There might be legal implications – This is somewhat tied to the previous bullet. Most benchmarking projects focus on processes that are not trade secrets or involve cost or pricing data to the degree that antitrust concerns become an issue to address. But whenever there is ANY question, engage your legal team and you might want to use a third-party. If your legal team will be involved, there will probably legal counsel involved from the organizations in your benchmarking group. A moderator is always a good idea in these instances.

When there is any doubt, I always recommend at least finding someone to talk with regarding your project, but be prepared and clear and don’t waste their time. A little advice, especially early on, will go a long way in helping ensure the success of your project.


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}