The Future of Scripting in Market Research Data Processing
I remember in 1995 or thereabouts reading a prediction that proved to be wrong. The prediction was that, within five years, scripting market research analysis would be dead. Here we are in 2020, and complex or high volume market research analysis still uses scripting languages. What’s more, many people specify questionnaires for various data collection methodologies using a scripting language. So, what is the future of scripting languages in market research? Let me try and make some predictions that are better than that one in 1995.
What are the benefits of scripting languages
I’ve written about this topic before, but scripting languages have advantages and disadvantages over menu-driven software that uses a graphical user interface (GUI). The most common advantages are:
- Efficient ways to write repetitive requirements
- Ability to specify complex requirements
- Ability to re-use complex script in other projects
- Easier to update more complex scripts
- Fewer limits to what you can offer clients
- Efficient for companies with high volume
- Where potential growth is important
Of course, there are disadvantages. You are likely to need more skilled staff, handing over or sharing projects can be more difficult and learning to get the most from the software can take months or even years rather than days or weeks. However, in my view, scripting languages are here to stay, but with a difference from those dark days of 1995.
The changes in market research data processing
Market research has changed. On the one hand, surveys have got simpler. On the other hand, they have become more complex. Or, perhaps, more accurately, what we want to do with survey data has become more complex. Generally, online surveys mean that surveys are more straightforward and contain simpler data. To overuse the word ‘simple’, surveys are simpler because more surveys are self-completion online surveys. Therefore, they have to be intuitive to answer, which often leads to the data being simpler. However, online surveys open up opportunities to be more creative and present a questionnaire in more convenient or particular ways.
Data goals have got more complex
Of course, this is not to say that research agencies no longer need to use complex survey instruments to collect the data they wish to deliver. For this reason alone, scripting languages are likely to survive anyway, but that’s only half the story. There is a growing need to deliver more from the data – and quickly. This may be something basic like an automated PowerPoint report or something more modern like an online dashboard. Indeed, it might be delivering data automatically to a customer’s own databases.
The more you want to do, the more automation you need
So, let’s get to the nub of why I think scripting languages are not dead. The more market research expands, and the more that technology offers in possibilities, the more that scripting languages are applicable and will consequently grow. To be clear, I am not forecasting an end to GUI software. Many tasks are much better to do with the use of mouse and menu selections, but scripting for the reasons cited above offers an edge in many cases.
More hybrids of scripting and GUI software
I do foresee a greater mix of scripting with GUI software. Indeed, that has already happened. Taking our MRDCL tabulation software, that has for years been first and foremost a scripting language, but more and we are adding more tools to augment the power and flexibility of the scripting language with menu options, ‘point and clicks’ and ‘drag and drops’. Scripting remains at the heart of the system, but the tools to optimise its benefits may be through a graphical user interface.
The spread of scripting
So, what of the future? Well, scripting has been in use for tabulations for over 40 years and for questionnaire specifications for 25 years or more. The next steps are the use of a script to manage data, particularly where there is a benefit in merging data sources or delivering that data in reports or dashboards.
Programming vs scripting
Some will argue that they are already scripting. Some would argue that programming is the same as scripting – and, in some ways, it is. I differentiate between programming and scripting. Programming, to my mind, is the use of an open language that can do ‘anything’ – not just market research tasks. Scripting is where you are working, usually within a software product, either using a proprietary language or tools specifically for market research. Scripting languages for market research are growing and will continue to grow.
Skills are growing
Another barrier that is diminishing is the availability of a skilled workforce. More and more college leavers have good programming and analytical skills, it, therefore, means that having a team of people, who can script to embrace the advantages cited previously, is becoming easier. Not only is the gap between the cost of employing skilled scripting staff and less skilled GUI software users decreasing, but the costs of training are reducing as the aptitude to learn is far greater. I would estimate that the time to train someone to use MRDCL has more than halved in the last 20 years.
No barriers with scripting
The exciting part for me is that scripting brings fewer barriers to what is possible. This benefit allows research companies to provide better insights to their customers and deliver more. The market research industry is much stronger if it can deliver what the client wants rather than what it is possible to provide with least effort.
MRDC Software’s direction
MRDC’s direction is to offer improved automation and hybrid solutions where possible. In other words, giving access to scripting languages, but not forgetting that a friendly user interface reduces errors and makes some things easier to carry out. This approach is the direction of MRDCL with the goals of automating as much as possible and making access to its powerful scripting language as easy as possible. We have also brought this to our new secondary analysis tool, Resolve, which is mainly a GUI software product, but with an interface for more powerful or efficient scripted tasks. Arguably, this is a hybrid coming from the opposite direction to MRDCL. MRDCL is very much a scripting language at its heart with GUI tools to make it better and easier to use; Resolve is the opposite.
MRDCL – what’s next?
We will keep MRDCL on course to grow over the coming years. Over the last 10-15 years, we have focused on making MRDCL the most productive tool for tabulations. For the last 2-3 years and the foreseeable future, our focus is on automation and expanding the scope of what users can script using MRDCL.
Scripting is not dead and will not be in the foreseeable future. As market research expands, the need for more powerful solutions will grow. That growth will need both script and GUI tools, but will also need smart ways of giving clients what they want. And, that must be the goal. Surely.