The survey has been held since 2001 and the results are being read by an ever-increasing audience. This is your guide to understanding how we create the AVS and its deliverables. It outlines the methodology that the Analyst Observatory uses.
Why We Conduct the AVS
The AVS shows which industry analysts firms are most valued, and by whom. Since 2001, we've asked thousands of professionals about the analyst services they use. The results help you to understand which analysts firms have the most impact, and which will give you the most value. Each year, around 1,000 professionals complete the survey, with around 15% from the Asia-Pacific region, and the rest evenly shared between the Americas and EMEA.
The AVS offers three big benefits to clients: our detailed report, typically between 100 and 150 pages, a recorded webinar to give you the overview, and a face-to-face video workshop where we help you use the AVS to develop your strategy.
The AVS is a unique evaluation of the IT Research marketplace based on two high-level criteria, use, and value.
We ask professionals which analysts firms they use. Analyst research has changed a lot over the last 20 years, when analysts insight was consumed by reading expensive reports. Now analysts have a wide range of services, and have a substantial impact on companies that don’t have formal subscription agreements. The AVS prompts users to comment on the 100 largest analyst firms, and also allows people to write in the names of other firms they work with.
It’s easy to compare the revenue of analyst firms, but the ability of those firms to capture value is not the same as their ability to generate value for users. Thousands of people tell us which services they use, from which analysts’ firms, and which services they value the most.
The AVS prompts users to comment on over 100 notable analyst firms, and also allows people to write in the names of other firms they work with. The firms are presented in two lists. The main list includes the 60 analyst firms most commented on in previous years' AVS, including every firm with the full-time equivalent of 10 or more analysts.
Each year we also have a candidate list: we ask participants which of 40 to 50 smaller firms they would have commented on if they have if they had been in the main list. In the following year's survey we relegate firms on the main list which were used less than firms on the candidate list. In the 2016 survey, for example, Avasant moved up to the main list, and Greyhound Research moved down to the candidate list.
The final report aims to show the data we collect as directly as possible. Most charts in the detailed report show actual percentages, rather than weighted scores. When we aggregate the data, the only weights come directly from use and value scores given by respondents to the survey.
Responses are kept anonymous from both clients and the staff who produce the report. Staff use base data for report production that is free from personally identifying variables (such as personal or company names, email addresses, IP and MAC addresses). In the reports, numerical responses are presented only as aggregate averages. Free text comments are described only by segment (for example, 'Enterprise user, North America'). We do not publish any comments by respondents we feel might be personally identifiable.
While the survey does not prevent multiple responses from one firm, it does prevent multiple responses from the same machine.
AVS Process Road Map: How Does the AVS Work?
The Analyst Value Survey process generally spans four key milestones, each of which has specific deliverables:
- The participants' webinar
- The detailed annual report
- Debate in our forums and workshops
- Analyst Firm Awards launch lunch and segment reports
How Do I Use the AVS?
The AVS is the first tool you need to understand which analyst firms to use, to hire, or to pitch to.
Buyers of analyst services use the results of the AVS to see which firms are perceived as delivering the highest impact on their IT decision making.
Analyst relations and marketing professionals at technology providers use the AVS to improve and justify their prioritisation of relationships with analysts who impact their business.
Analyst firms use the AVS to understand how they can deliver more value and overtake their competition. As the M&A market grows, many analyst firms use the AVS to consider partnerships and investment opportunities.
Keep in mind that high-value analysts might not be the only option for you. Smaller firms without a full portfolio of slick services can be cheaper, and sometimes just as close to growing market sectors. A small analyst firm might have a lot to offer and, indeed, many upstart firms are now delivering more value than the major analyst stalwarts.
Analyst Observatory Objectivity
For many professionals, the AVS is the compass used to help themselves and their stakeholders to navigate through the complex global market. That is why we aim to maximise the participation in the survey, in order to better ensure its objectivity and resilience. We are committed to avoiding bias and potential conflicts of interest. The design, execution, and analysis of the survey is separate from its commercialisation. The Analyst Observatory's researchers are committed to the highest standards of ethics in the research process.
An Analyst Firm Mentioned in AVS Encouraged Me to Participate. Is This Allowed?
Because there are no commercially available databases of analysts' users, the AVS relies on the Observatory's extensive mailing list, and invitations which are shared by our staff, sponsors, and partners. Several analyst firms also share links to the survey: firms are asked to use a custom URL which allows us to compare responses from different sources.
On the plus side, when firms share the link, the survey gets much more data about them and their competitors. On the negative side, it produces some concerns that the results could become imbalanced. Our researchers have been running the survey since 2001 and take these concerns seriously. When we compare the responses, we see that respondents who come from invites from analyst firms are not notably different in their thoughts about those firms than those who do not.
What it does do, however, is push up the share of voice of those firms and their competitors. For example, respondents who use the firms that have shared links to the survey are more likely to use IDC than are those who are not. So those firms are also boosting IDC's profile, although there's no notably difference in how those respondents feel about IDC.
After consulting with many people, including members of the Market Research Society, we have designed our survey accordingly. We focus on average scores, so that changing the share of voice doesn't impact on the outcomes. Because of this design we believe that it is, on balance, a good thing that analyst firms help us gather more data.