Phil Hearn: Blogger, Writer & Founder of MRDC Software Ltd.

Effective sample size calculator – with free Excel working model

Target weighting is a technique used in market research to scale a sample of respondents to a known or desired target or population. For example, you may have 40% males and 60% females but want 50% of both males and females. MRDC Software can provide you with a free working model in Excel to calculate the weights you need to apply and check your resultant effective sample size. You can download it here. You will need to run under Windows as it uses Excel macros.

Scope

This article aims to explain how target weighting works and to understand the effects of the weighting on your data. After you have applied weighting factors to your data, the effective sample size will usually be reduced.

The article only explains how target weighting works and some basic checks you should apply. You should seek expert advice if you are unsure about the effects of applying weighting. The article also does not explain how to carry out weighting in software products, such as SPSS.

To use the calculator, you will need to follow the steps correctly. These are explained in a video, which you can view here.

What is target weighting?

Target weighting allows you to scale survey respondents to a targeted number of respondents. You will usually weight respondents to one or more questions or variables to do this. For example, you may wish to adjust your sample to represent 50% males and 50% females. If your survey sample had 40% males and 60% females, you would apply a weighting factor to the male sample of 1.25 (50 divided by 40) and to the female sample of 0.83333 (50 divided by 60). If these weighting factors are applied to the data, you will get a weighted sample of 50% males and 50% females.

Target weighting is often carried out using a two-dimensional matrix. For example, gender within age group. For example, you may have the following targets:

Males 18-34 – 20%

Males 35-54 – 15%

Males 55+ – 15%

Females 18-34 – 15%

Females 35-54 – 25%

Females 55+ – 10%

You should note that weighting targets can be figures or percentages. The examples here use percentages, but the working model allows either.

In this example, if males aged 18-34 were 15% of the sample, they would get a weight of 1.33333 (20% divided by 15%). The weights for the other five cells would be calculated in the same way.

The targets do not need to be a matrix – you could have four age groups for females and three for males, for example. However, the percentages must add up to 100%, and every respondent should fall into one (and only one) of the categories.

Three (or more) dimensional weighting matrices are also possible, but care should be taken not to spread the sample too thinly as your effective sample size may reduce too much (see next section).

You will note that a factor is calculated for each cell of your weighting matrix by dividing the target by the actual number/percentage of respondents.

What is the effective sample size?

The effective sample size is the sample which could have been achieved by an unweighted controlled sample. In other words, if your sample size of 200 has an effective sample size of 100, you could have achieved results as statistically accurate as a random sample of 100 that met your target criteria.

The effective sample size is an important figure that should always be checked when weighting survey data. An extreme example shows this. Let’s say your targets are 50% males and 50% females in a sample of 200 people, but you have 198 males and 2 females in your sample. When you apply weighting as described above, your effective sample would be 7.92. In other words, your sample of 200 people has reduced to the validity of a random sample of approximately 8 respondents comprised of 4 males and 4 females.

What does this mean? It is easy to think that target weighting can be used to solve sampling defects. To some extent, it can, yet it comes with a penalty. It is good practice to observe carefully the weighting factors being applied to each cell of your weighting target matrix, but, as an absolute minimum, you should always check (and preferably show) the effective sample size in any analysis you do.

Other things to consider

Each respondent must fall into one (and only one) of the categories in your weighting target matrix. If a respondent has missing data, such as no age, where you are scaling data to age groups, there is no magic solution; you will need to remove the record or apply a weight to this record manually.

Target weighting often use interlocking cells for its targets. For example, where you have two genders and three ages, you would need targets for each of the six cells in the target weighting matrix. If you only have targets for the two genders and the three age groups, you must use rim weighting. We have another free Excel macro that can help you with this and information to explain rim weighting in more detail

It is sometimes tempting to weight the data for too many questions/variables when striving to correct sample deficiencies. Try to avoid using too many cells, particularly when you have a small sample. For example, it is possible to weight to 2 gender categories by 4 age groups by 5 regions by 4 social grades, but this will result in each respondent falling into one of 160 cells (2 * 4 * 5 * 4). If your sample size is only 300, you are almost certain to get some high or low weights.

Where no one is in your sample for a particular cell, you cannot weight no one to a target! This may sound obvious, but it can be an undiscovered problem, particularly where a software package is making the calculations automatically. It is important to check this. The usual solution is to collapse some cells together – maybe 3 age groups rather than 6 age groups.

Need further help?

We can offer software that can automatically integrate simple or complex weighting procedures into your analysis and also provide a consultancy service if you want more help. Contact nikki.sunga@mrdcsoftware.com for more information.