With the recent uproar of a specific magic quadrant (MQ) report being released, I just couldn't help myself.
In the technology space, some decision makers are believed to put a lot of weight on analyst reports. These reports are made to cover different areas of technology, and aim to advise what may or may not be a good product to meet specific solution criteria. This is well and good, but analysts, like humans are fallible. Being unbiased is hard, and comparing different solutions on their actual merits requires very in-depth knowledge of each product.
I'd postulate that research, no matter the analyst firm involved, has to be taken with an appropriate grain of salt. This is why people evaluate products before buying them and do not just trust external sources. Analyst reports are a small part of the picture, often a literal picture, that positions different companies on a grid to stack rank their available features and their completeness of vision.
Most people can understand that boiling this down to a single point on a chart, or bean if you will, oversimplifies things. But analyst reports appear to matter. If done properly every vendor in the specific area of research gets to contribute to these reports. So how does this work? Let me introduce you to a magic triangle, and no, not the one from math class.
To place your bean on the triangle, there are only a few variables that matter:
1. Vendor Spend
The position of your bean goes up the more you spend. This is an unfortunate side-effect of how vendor spend affects analysts. The amount spent with an analyst firm dramatically improves their ability to understand how the vendor's technology stack works. This does not imply bias, just better understanding.
2. Features Advertised
The position of your bean moves up based on the amount of features you say you have. And the reference customers that will back this up. This does not account for how stuff works in the wild. 'Check box only' features are allowed.
One of the funny things about enterprise software, is that this is often purchased by people other than those using it day-to-day. The more the analyst understands, the more the report represents reality, and whether they are able to see the difference between vapor-ware and actual functionality. This can have unfortunate consequences.
A side note here, as a technology vendor, you would be well advised to be careful with your reference customer list, as analysts, like anyone else, have a tendency to switch jobs, at which point they may end up working for one of your competitors.
3. Market Presence
Your bean's position is also affected by how established you are. This is simply determined by the market share you command. More customers is better. The thought here is that if other customers are using your solution, it must be good. Again, this is completely unrelated to how stuff actually works in the wild. If everyone jumps off a cliff, you naturally follow.
Have your beans and eat them too.
Take your beans, your magic shapes and I'll raise you a random number generator. Would I use my subscription for analyst report data to make everyday decisions as a buyer? Would I read 'strategic guides for leaders in enterprise'? Probably not. But having a list of names is a good starting point. Where they are on the chart matters far less.
Analyst reports are best used when recognizing them for what they are. Pure opinion. Then analyst's business model depends on taking money from the very people they are evaluating. Instead of trusting the position of the bean on the chart you would do well to do your own research, your own proof of concept and then encourage management to choose the solution you vetted out.
If only. Sometimes it boils down to budget and management will end up choosing the solution that still fits on paper, but perhaps costs less, or meets other organizational needs. This is where unbiased reports do matter. Like any other source of information, if we don't question these magic shaped unicorn bean positioning systems -- at a least a little, we're probably deceiving ourselves.