Ways to organise market research data analysis and data processing

One of the things I like to do once a year is to question the way we do everything. This covers everything that I can think of – how new sales are handled, how purchase invoices are handled, how we answer the phone etc. You name it, I try to question it. It’s a good exercise and can involve everyone too.

Getting the right people doing the right work

A client recently had gone through a similar process regarding the data analysis and data processing system in their market research agency. The manager observed that whilst he had a really good team of data analysts, the team was over-skilled for the many of the projects they handled. This had arisen, to a great extent, because the company had taken on an increasing number of smaller, fast online projects. When I suggested that the overqualified staff probably did excellent or good work on difficult projects but poor or average quality work on simple projects, my client hesitated for a second and then nodded in agreement. Most people do better when they are challenged.

The risk of using lower level software

The problem here was that the data processing team had been set up to use MRDCL and Quantum, the two leading scripting languages for survey analysis. Whilst both are powerful software products, they need skilled staff to use them. If the requirements are easy, a simpler product would suffice. However, the risk of using a lower level product with lower grade staff is that client requests may escalate their requirements and go beyond the scope of the lower level product.

Getting best value from staff

One of the problems with survey analysis is that a particular project becomes one person’s responsibility. This has three adverse effects. Firstly, it means that staff cannot share projects by helping out or covering during illness and holidays easily. Secondly, skilled staff spend too much of their time doing simple work. Thirdly, costs for data processing are higher than they need to be.

Solution #1 – Use software where projects can be transferred to skilled staff when necessary

The main benefits of using a scripting language are that complex or repetitive requirements can be generated in a fraction of the time it takes with less advanced systems. You don’t want to lose that benefit, particularly if you have a high or heavy workload. The solution we offer is that complex projects are handled in MRDCL, whilst simpler projects are handled using QPSMR. As QPSMR and MRDCL are compatible and use the same engine, an expert can take over a QPSMR project if the going unexpectedly gets tougher. Everything you do in QPSMRcan be taken and used by MRDCL users as QPSMR simply generates MRDCL script and runs it for you.

Solution #2 – Separate the difficult parts of the work from the easy parts

There are two main phases to producing survey analysis – defining the variables needed for analysis and specifying the tables you want from those variables. Typically, the variable definition task is the more complex part of the work. Therefore, it can make sense for many projects to script the variables in MRDCL and then move the project into Resolve the free tabulation analysis tool, so that less skilled data processing staff or researchers produce their own tables. If there’s a couple of difficult tables, the MRDCL team can write some script that can be included in the Resolve table run or run them separately.

Solution #3 – Use Excel Productivity Scripting (EPS)

The third solution is to use the powerful EPS tools within MRDCL. EPS is a technique within MRDCL where you can build templates in Excel to store instructions that are processed by MRDCL. The technique needs some up front work to handle all of the things you need, but once in place it means you have a customised system that handles requirements through an easy Excel interface. And, what’s more, it’s easy to follow what someone does as Excel keeps work organised. If some special MRDCL script is needed to specify some variables or tables, the MRDCL analyst can easily include these into the program. It means that less skilled staff can do most of the work, even complex or repetitive requirements, while skilled staff spend more of their time doing what they are good at.

Solution #4 – Working in partnership with MRDC

In some cases, learning the skills to handle complex projects can outweigh the value of those complex projects. In such cases, we recommend that a partnership might work well. You can handle most of the work yourselves using MRDCL and/or QPSMR, but buy expertise from MRDC. If you just have the occasional complex project or one where expertise would benefit, we often work with customers so that you only pay for the expertise you need.

Reappraising how work is handled

What I am advocating is being in a position where on a project by project basis, you can choose the right approach. Sometimes, having one system for all projects might look attractive and easier to manage, but whether you are getting the best from your team, giving your clients the best service as well as fulfilling your staff’s own aims is another matter.