Phil Hearn: Blogger, Writer & Founder of MRDC Software Ltd.

A Comparison Of Crosstabulation Software For Market Research Using 10 Criteria


If you want to produce crosstabs (or tables) from surveys, the software choices include QPSMR, Q-Research, Wincross, Tabx, mTab, SPSS, MarketSight, SurveyMonkey, Qualtrics and many more. This guide focuses on menu-driven/graphical user interface (GUI) products and evaluates what these nine products offer, exploring their focus and strengths.

The Survey Tabulation Market

One thing that immediately made this task quite difficult was each product covers different parts of the whole survey questionnaire process. By this, I include:

  • Collecting data on paper questionnaires (often called PAPI – pen and paper interviewing)
  • Collecting data online (often called CAWI – computer-assisted web interviewing)
  • Collecting data by telephone (often called CATI – computer-assisted telephone interviewing)
  • Collecting data by interviewers using a device (often called CAPI) – computer-assisted personal interviewing
  • Data editing/cleaning
  • Tabulations
  • Charting/reporting to Excel/PowerPoint
  • Producing online dashboards

No product covered all these disciplines, although some online data collection platforms may claim that you can use their platform for PAPI, CAPI, or CATI. While this may be true, generally, online survey platforms are not suitable for other data collection forms. For example, telephone surveys need telephone management systems for callbacks, appointments etc. CAPI needs a stable internet connection at all times.

This Evaluation Guide

I have produced this evaluation guide from information on each product’s website or in the public domain. I have excluded QPSMR from this evaluation guide as I am seeking to be as impartial and objective as possible. I have produced the guide on a ‘best endeavours’ basis.


>> If you want to see how QPSMR compares with these products, click here >>>


If you notice any inaccuracy, please email me at phil.hearn@mrdcsoftware.com. I will amend as appropriate and note any errata publicly. I will be reviewing each product on ten criteria. I will be making some comments about QPSMR separately at the end of this article.


So, let’s get started. The ten criteria are as follows:

1. Website Impression

[expand title=”Read about this topic”]

It might just be my opinion, but I think you can sometimes learn a lot from the company’s website. However, as a general point about this review, you may make wrong assumptions if any information is difficult to find. Some websites were informative, demonstrated a focus on market research (as opposed to, say, customer feedback) and had a clear message. Sadly, others gave me the impression of either trying to cover every possible potential customer, being experts or ‘the best’ in everything or too vague.


Informative websites

The websites with the most useful information to assist an evaluation were undoubtedly Q-Research and MarketSight. Both had accessible details on features, what the interface looks like, and the output types you can generate. Finding the information you need for an initial assessment was easy and of a high standard. The mTab website pushed the message of ‘empowering researchers’ and ‘producing insights’ but did not get into the messy details of producing tabulations in enough depth for me. To be honest, I felt that mTab might be underselling itself. Wincross was clear when it came to talking about tables – in fairness, that was the website’s focus. There are other Wincross products; it was less clear how they fitted together when the main tables engine. The other four fared less favourably.

Less informative websites

Qualtrics gave me the impression that there was a lot of power in the software, but no more than that. The website focuses on selling to different markets with emphasis on Experience Management and Experience Feedback. It produced some doubts, possibly unfairly. This leaves Tabx, SPSS and Surveymonkey. Tabx gives you grids of tickboxes showing that Tabx does ‘everything’. I’m a bit cynical about tickboxes – the list, of course, excludes things that Tabx doesn’t do. There is a lack of depth, which caused to dig a bit harder than I maybe did with the others. I found that significance tests are available at 95% and 99%, for example, which presumably means significance tests are not available at any other level. I saw some red flags here. The SPSS website was plain unfriendly. You felt that SPSS was just a small part of the business and didn’t warrant much information. Maybe, SPSS has greater general awareness, so there is no need to be as informative as the others. SurveyMonkey left me cold as I felt that the aim was to explain to beginners what a survey was. Further, I discovered almost no detail about what tabulations SurveyMonkey offers.

[/expand]

2. Tabulations & Ease Of Use

[expand title=”Read about this topic”]

I decided on three considerations when looking at producing tabulations, the central focus of this review article. These were a) the look and feel of the software and its ease of use b) how fully-featured the software was c) the type of data files that you can use.

Is there enough for research agencies?

Wincross and Q-Research came out best in this category, partly because they provided the most relevant information and left me feeling that they would be robust enough for most survey analysis carried out by research agencies. Neither product mentioned any ability to handle hierarchical data or data from multiple sources, which is growing in importance. Wincross did look a bit old-fashioned but had a good range of necessary features.

Doing the basics well

Tabx, MarketSight, Qualtrics and SurveyMonkey all cover the basics well but may start to struggle if you have fussy or complex needs. You get the feeling that MarketSight and Qualtrics offer more features if it’s features you need. The claim by Tabx to be ‘complete crosstab software’ is ambiguous and may be misleading. These four products all looked fairly easy to use, although some features may be hard to find, so documentation, support and videos can help. SPSS covers the basics and gives you routes through scripting to endless possibilities. It does not feel like good software for running batches of tables, a common and basic requirement. The user interface of SPSS is also quite daunting in places unless you know what you want. This leaves mTab, which looks as though it has more than it gives away. I want more detail on the website to know for sure. Qualtrics and MarketSight offered some degrees of confidence in merging other data or multiple data sources easily.

Extensibility

The important question to me is what you do when you hit a limitation in a software product. I noticed limitations like 255 columns for a banner, fixed significance tests, maximum sample sizes, basic weighting capabilities only etc. – whether these are a problem would vary from company to company but need consideration before purchasing. Some of the products allow you to use scripting to extend the power (see section 9 below), but you need someone skilled enough to use these more advanced tools.

[/expand]

3. Data Collection

[expand title=”Read about this topic”]

For data collection, there is a big divergence. Some focus on data collection (mainly online), some have a data collection part to their offering, others have add-on products, and some expect you to import the data from elsewhere. Tabx, mTab, MarketSight and Q-Research,  all assume you will import or read your data from any other platform. SurveyMonkey and Qualtrics both have plenty of tools for online data collection. Although you could use an online data collection tool for paper surveys or even CATI, they are, in my opinion, the wrong tools for those methodologies. SurveyMonkey’s focus is on collecting data online, while Qualtrics seeks to offer more of a full service for the ‘Feedback’ market. SPSS has a data entry capability, but it’s rather like typing into a spreadsheet. SPSS may have a separate product for data collection, but the information is not readily available unless it is hidden! Wincross has individual products, which are specifically for CATI and online surveys.

[/expand]

4. Data Editing

[expand title=”Read about this topic”]

I decided to make data editing one of the criteria as I consider it to be important. By this, I mean that having collected or imported data, you can make changes and, preferably, check data for logical consistency and accuracy. Although it may be important, it is not an especially interesting topic, and I suspect that some products have editing tools but do not bother to mention them. However, I can only report what I find.

An unspoken subject

Arguably, SPSS wins in this category. Its spreadsheet-style allows editing of data, and you can build checks on data even if the interface is clunky to use. Wincross specifically mentions data editing, although there are no further details – I suspect it has enough tools – while Qualtrics and SurveyMonkey as online data collection platforms almost certainly give access to the data to make changes. Q-Research, Tabx, mTab and MarketSight make no mention of editing data. These products rely on importing data. Therefore, you may have to get your data right at the source before importation. Alternatively, they may not mention their data editing tools.

[/expand]

5. Reporting

[expand title=”Read about this topic”]

Reporting data, usually as PowerPoint slides or dashboards, produced some interesting findings. On the one hand, it showed a common trend, while on the other hand, it indicated a big difference – I would go as far as saying it was a split in the market. Except for SPSS, data visualisation figured prominently for all the other seven products. The outputs included in all cases both dashboards and charts, mostly in PowerPoint and Excel. So, where is the split in the market, you might wonder? This split is important as data visualisation has become a hot topic in market research. But hot topics do not mean the market is well served.

Flexibility in data visualisation

The split comes in flexibility. There is a whole ocean of difference between a dashboard that you can generate semi-automatically and one that you can customise to meet the precise requirements of a client. They are two different things. Being able to produce a quick, basic dashboard with little effort will serve many market researchers well. However, creating a dashboard that allows you (more or less) complete control of the look and feel, has different views for different users requires a much higher level of flexibility and sophistication in the software. The issue is complex as the software products do not fall into the two extremes – flexible and inflexible – there is a whole spectrum of flexibility.

Enough flexibility is great, but it’s not as easy as that

It would seem that ‘enough flexibility’ is the key in this market. This is mainly true, but if you are servicing a client who wants something that is not possible – a particular infographic, for example – there may be no way to meet a client’s needs. It’s not like analysis where you can usually find another, perhaps more cumbersome, to provide what a client wants.

The product comparisons

Comparing the products is difficult even if you take a month’s trial. It is no wonder that companies that can afford it use Tableau, which many see as the gold standard. The products I compared have some good looking dashboards or reports, but how flexible they are and whether they meet your needs is difficult to assess. mTab impressed me most – I particularly liked that it acknowledged the importance of pooling data sources and offering flexible data visualisation. Qualtrics looks flexible and powerful in this area, while MarketSight seems to have plenty of features. It’s harder to judge SurveyMonkey as its website focuses on teaching you about surveys and how easy the whole business is. Q-Research hooks into another separately priced product of its own, while Tabx and Wincross make it easy to do but may have limitations. SPSS stands alone, but, then again, its core strength is its multivariate analysis tools, in my view. On the subject of multivariate analysis, I was surprised to find that Q-Research, Wincross and mTab all have some multivariate analysis tools.

[/expand]

6. Market Research Focus

[expand title=”Read about this topic”]

You could carry out a survey by entering data and analysing it in Microsoft Excel. Would I recommend Excel for some very simple research? Yes, assuming you have a licence already. Would I recommend it for 99% or more of market research surveys? Never. This article is for people who want to produce tabulations from market research surveys, not counting the number of cars in car parks for each hour of a day. Therefore, I think it is important that a software product’s focus is market research to get the best tools for tabulations.

Customer feedback and typical market research surveys are very different

Wincross and Q-Research are undoubtedly about producing tables from market research surveys. They have plenty of features, options, and their purpose is to generate batches of tables – not just one or two crosstabulations. Tabx would claim to be the same with its ‘complete crosstab software’ strapline, but I suspect the former two have been put to the test far more rigorously. The other five score poorly. mTab presents itself as market research being one of nine industries for its users. MarketSight doesn’t even list market research – and I don’t include market research under marketing, I am afraid. SurveyMonkey is too busy talking about surveys and teaching you what they are to have any real focus.

Meanwhile, Qualtrics fixes on its experience management theme – the nearest it gets is marketing & brand research, amongst other sectors that it favours. Frankly, I am suspicious of products that do not overtly mention market research, as questionnaires used by research agencies are usually far more complex than those used for customer feedback, for example. SPSS stands alone as it covers a lot of different things. I will always consider it a leading tool for multivariate analysis and limited for market research tabulations of even moderate complexity.

[/expand]

7. Compatibility

[expand title=”Read about this topic”]

I strongly believe that it is important for software to be able to read in data from relevant sources and to be able to transfer data where necessary. Some software vendors may believe their solution is all you need, but I think this is a fallacy. It is increasingly important to be able to move data from system to system; clients expect data to be mobile nowadays and readily accessible wherever it is needed. Further, unless your needs are limited and clearly defined, you need to have flexibility. You need to be able to choose which products suit your needs best for each part of the research process.


Market research software and external systems

Market research has two standards for transferring data between software products. Triple-S (LINK) is an established form for transferring metadata (texts, definitions and data) from one market research system to another. Most market research software vendors have made the standard available. Alternatively, SPSS has become a proprietary standard. Whilst I wouldn’t say I like proprietary standards, SPSS does allow open access to its file format. There is a new standard in the pipeline, called TSAPI, an API (LINK). This standard is still in development but is an exciting prospect if it progresses.

External systems generally use APIs to connect from one system to another. Few market research systems have reached this level yet. Most systems have low-level connections through Excel or CSV file formats, but these only connect data and mean that texts and definitions are lost.


Importing data

Tabx, MarketSight and Q-Research all read Triple-S and SPSS data files. Wincross and mTab read SPSS data files, although mTab also reads data from other ‘leading software tools’. Some also read Excel data, but this would be data only with no questionnaire texts. SPSS and SurveyMonkey seem to work in isolation, although MarketSight, SPSS and Qualtrics rely on APIs to connect software systems. At this stage, APIs may be fine to connect to external business systems but are less useful within the market research sphere.


Exporting data

Maybe some software vendors are less willing to see data exported from their software. None of the eight products mentioned exports to both Triple-S and SPSS, which I found surprising. Tabx and mTab make no mention of exports. Maybe, they see their products as ‘destination’ products. I understand this to some extent, but if it is true, I find it worrying. APIs may be the way in the future, but there is slow adoption of this route for the more market research-focused products.

[/expand]

8. Help Facilities/Videos

[expand title=”Read about this topic”]

This category surprised me. I imagined that most software products would have a good selection of videos to help users get started and cover any common topic areas. I also expected informative blog articles, documentation and more. However, few of the products did well here.


The content was only available at a high level

Some products had videos, blogs, white papers, and documentation, but it was only at a high level – more of a selling tool than for users’ benefit. This finding applied to mTab, Tabx, Q-Research and Qualtrics. Wincross had a handful of videos, half of which were seven years old. MarketSight performed better than the others with five fairly detailed videos and ten shorter ones. SurveyMonkey and SPSS have a wider reach than the other products, I suspect, and rely on content you can find by Google searches. In my opinion, this is a frustrating and time-consuming way to find help, as you often find videos that are not particularly helpful.

[/expand]

9. Scripting

[expand title=”Read about this topic”]

Many software products offer a scripting language for more complex tasks. Where this is a general-purpose programming language, like Javascript, this can mean having to write a lot of code for some operations. Something that may be fairly simple in other software or a market research-focused scripting language may be difficult or clumsy to do and need expertise.

Menu-driven only systems

Tabx, mTab, Wincross and SurveyMonkey make no mention of being able to fall back on a scripting language. Wincross is well-featured for tabulations and mTab is well-featured for data visualisation, so the need for a programming language is much reduced. SPSS has its own language as well as giving you access to Python and other languages. Q-Research allows you to use R and Javascript, while Qualtrics enables the use of Javascript.

Scripting languages for market research

The programs that allow access to a scripting language still struggle to match highly developed scripting languages like our own MRDCL and others such as Quantum, Merlin and Uncle. These products all have languages dedicated to making variable definitions, data handling and tabulations as easy as possible.

[/expand]

10. Pricing

[expand title=”Read about this topic”]

Pricing models varied substantially from product to product. Most offer annual licences. Some have add-ons which can inflate costs quickly. Others look inexpensive until you make higher usage or add a team of users. Two products, mTab and Qualtrics, did not wish to share their pricing models on their websites.


Varying prices

SurveyMonkey starts at £900 per user per year, with a minimum of three users. If you are looking for an online platform that does basic tabulations, it is not bad value. If your focus is on flexibility in tabulations mainly, it is expensive. SPSS and MarketSight both get costly if you have a team of users. SPSS costs GBP1775 per user per year (assuming you want the tables add-on module), while MarketSight is similar at USD2160 per user per year. Tabx comes in a GBP1080 per user per year, which is reasonable, but ten users cost GBP8316 per year, which looks overpriced to me. Q-Research offers two pricing models at GBP1219 per user per year or GBP3657 per concurrent user per year. Wincross has a completely different pricing model in that you buy a perpetual licence and pay a lower annual fee for maintenance and upgrades. With the high initial price for WIncross, you need to be sure that Wincross is right for you, though. A team of ten users costs US$15000 upfront, but annual costs are only US$2200 per year. There are, however, add-ons for the Executive distributable licences and the CATI module costs tens of thousands of dollars upfront. Best to check before you start!

[/expand]


QPSMR vs The Competition


One of the things that I learnt from this review is that, in most cases, each product had its focus and areas where it excelled. I realised that QPSMR, the product we sell for menu-driven tabulations, is unique in its strengths and market focus. I noticed these major differences about QPSMR when compared to all (or, at least, most of) the other products:

  • QPSMR has a big engine. For example, it can:

                                  – Handle millions of records

                                  – Have questions/variables with 30000 codes for each variable

                                  – Produce tables with 30000 rows

                                  – Produce tables with 5000 columns

                                  – It has over 250 options for tables

  • QPSMR has a smooth upgrade path to a market research-focused scripting language in MRDCL. QPSMR and MRDCL work together using the same engine to process tables.
  • QPSMR is ideal for paper surveys, CAPI and CATI, although it does not offer CAWI
  • It imports and exports data from/to Triple-S and SPSS
  • The qpsmr-learning.com website has over 60 videos to help users get started and covers almost every topic.
  • Its focus is entirely on the market research industry
  • It is more competitively priced than alternatives
  • It does not have a dashboard component, although you can easily connect to several dashboard products

Summary

So, what did I learn from this research and review? Products have different strengths and often a specific focus. The quality of the actual software was high generally, although no product excelled in every area. Initially, I was surprised that QPSMR appears to stand alone with no direct competitor, but, on reflection, I realise few of these products are direct competitors. The nearest matches were probably MarketSight vs Qualtrics and Wincross vs Q-Research, but these had their differences. Products like mTab and SurveyMonkey serve different markets, it would seem, to QPSMR. mTab looks to be strong in managing data sources and visualising data, whereas SurveyMonkey is more about data collection.

The Right Software Mix

My opinion that no single product rarely does everything that a research agency needs, I believe, strengthens my argument that it is important to have software that connects well with other products. The days when one or even two products covered all your needs are gone. I hear the term ‘agile research’ used widely at the moment. I’m not fond of the term, but if you want to be agile, you need to have the right mix of tools available for all the projects you undertake. That’s why you need software products that overlap with each other and work well together.

The Research Industry Still Needs Better Software

One of my views of market research is that the industry is getting good and doing simpler research very well. There is a gap in dealing with complexity well. This gap includes complex data structures, complex data analysis, complex reporting, and complex dashboarding – or, just, tasks that are more complex than average, perhaps. Our link from QPSMR to MRDCL for scripted tabulations is good but not ideal, but automation, improved reporting and dashboarding form the core of our developments for the 2020s.