Welcome to our Blog

Theoretical relevance is more important than statistical significance

July 29, 2024

Over the years, I have assisted many university students on their research projects, particularly in relation to data analysis were bivariate and multivariate statistical analyses are required. In the process, I have observed a tendency among some of the researchers of focusing primarily on finding statistically significant relationships between some variables in their studies as opposed to focusing on specific hypotheses testing.

Hypothesis testing, by definition, implies that either a statistically significant relationship will be found between some variables of interest; in which case the null hypothesis is rejected, or a non-statistically significant relationship will be indicated between those variables; in which case we fail to reject the null hypothesis.

So, instead of being excited just because some statistically significant relationships have been found or, conversely, being disappointed just because the analysis has not found any statistically significant relationships, the important thing is to check if those results have some theoretical grounding or support in existing literature.

Not every statistically significant or non-significant relationship between some variables in a study will make theoretical sense. For example, in a study done within a highly patriarchal community it would be rather odd to find that there was a significant association between gender and formal employment but in favour of females. The opposite would be expected instead. It would arguably also be odd and cast doubt on the study’s external validity if no association at all was found between employment and gender in such a case.

Although not as common, there are also instances where, due to the desire to have some statistically significant results to talk about in their research reports, some researchers will perform some rather redundant analyses under the guise of “secondary objectives”. An example of (completely made up) would be in a study where some height and weight data were collected for the purpose of computing the study participants’ body mass index (BMI), which in turn would be used to examine if there was an association between BMI and susceptibility to certain medical conditions X and Y. Now, imagine the subsequent analysis found that there was no association between BMI and medical condition X or BMI and medical condition Y. And then driven by the “thirst” to at least have some results with some p-value that is less than 0.05 somewhere in their paper, the researcher then runs some linear correlational analysis between height and weight. They could even also want some test performed to examine if the two medical conditions X and Y are associated, even if this would be somewhat redundant analysis in relation to the main study aim.

In summary, theoretical relevance or logic matters more than mere statistical significance. Put differently, statistical significance will not count for much if it lacks theoretical significance.    

Workshops Discourse Analysis

May 05, 2023

Business or organisational workshops are a valuable platform for information gathering and sharing. Whether a workshop audience comprises of internal stakeholders (e.g. staff) or external stakeholders (e.g. customers), the value of such sessions is enhanced when the discourse is properly captured, analysed, and packaged. Workshop reports stand a better chance of being read, assimilated, and implemented by key role players and decision makers when they are well-packaged.

Our Workshoper™ package is a discourse processing service wherein we record, transcribe, and conduct thematic analysis of speeches, comments, and other forms of discourse from business workshops, conferences, and other such gatherings. We thus use the term “workshop” broadly to describe different types of organised meetings or gatherings.

This service offering entails a four-phase process of 1)- data collection 2)- data processing 3)- data analysis and 4)- report writing. To reduce costs and/or to fit in with client’s organisational protocol, the data collection phase can be done internally and independently by the workshop organisers and facilitators, with the data sent to us afterwards for processing, analysis, and reporting. The information can be collected in various ways, depending on the nature of the workshop and client preferences. The common data formats that we work with are audio, video, transcripts, notes, papers, and questionnaires.

The final output are informative and analytical reports that go beyond just raw and unstructured verbatim transcripts of the workshop proceedings and discourse.

Effect of mandatory open-ended questions on survey completion rate

April 07, 2023

The collection of data through surveys is a critical component of most market and social research projects. This is because even where secondary or archival data is available, it often needs to be analysed and triangulated against some primary data collected from a representative sample. As research consultants we therefore create and administer surveys, mostly online, on a regular basis.

Each time we create an online survey, a key consideration is whether we will get a good response rate. Given the traditionally low response rates to online surveys in general, you always hope that everyone who starts the survey finishes it. Survey dropouts do not only leave you with an insufficient sample but with potentially biased results too as those who dropout may be unique members of your target population.

There are several factors that influence the survey completion rate, primary of which is survey length. Apart from surveys that are too lengthy, other factors that have been shown to drive non-completion is having too many open-ended questions and making answering the questions mandatory.

We recently scripted a questionnaire that was pre-designed by a client into an online survey. In the process, we noticed that, apart from a couple of standard and easy to answer demographic questions, the substantive section of the survey started off with an open-ended question which was also mandatory to answer. Having discussed the potential negative effect of this question on survey completion, we decided to proceed without any amendments while we monitored the response trends. We subsequently noticed that a mere 53% of those who were starting the survey were completing it fully. Moreover, almost all the dropouts were occurring at the point where respondents had to answer the mandatory open-ended question referred to above. We also noticed that although there were many more survey questions after compared to before this specific question, those who went past this question tended to complete the remainder of the survey. As such, survey length did not appear to be the main impediment in this case. We thus duly removed the mandatory answer instruction from the question and although the completion rate was still not ideal, it did improve significantly by 10 percentage points to 63%.

As the popular Survey Monkey online survey platform advises, surveys are a form of conversation and should, among other things, start with a “friendly, easy to answer question”. As we have also highlighted in some previous articles, one of the key questionnaire design principles is to minimise the burden on respondents. Our observation from this survey lends further support to this proposition.

Things to avoid when designing survey questionnaires

April 27, 2021

Research findings can only be as good as the data upon which the findings are based. In turn, the data can only be as good as the data collection tools and methods. Over the years, we have had the privilege to review and work with survey questionnaires from diverse sectors and covering a variety of research topics and research objectives. In the process, we have observed some pitfalls to avoid if survey instruments are to meet the basic requirements of reliability, validity, and user-friendliness. Below, we highlight some of the pitfalls that we have observed in questionnaire design.

Double barreled questions

This where two or more constructs or phenomena are included for measurement in the same question. An example is a “yes/no” type question that is phrased as follows: “Did you report the matter to the police and have they provided updates on their investigations?”. The problem here is that some respondents may have reported the matter to the police, but the police may not have given them any progress updates. They are therefore left in limbo on how to answer the question. Depending on how the survey is programmed, some may then randomly choose either a “yes” or a “no” just to get past the question. The simple solution is to break it down into two separate questions.

Questions that require computations

Respondents must not be burdened with questions that require some form of mental or arithmetic calculation before they can provide an answer. So, instead of asking “What percentage of your monthly income are you left with after paying for basic expenses?” rather break it down into two parts. a)- “What is your monthly income before expenses?” b)- “On average, how much do you spend on basic expenses per month?”. The percentage of monthly income left after basic percentages can then be calculated at data analysis. In any case, how confident can one be that respondents are calculating the percentages accurately?

Skewed Likert scale options

Likert scales should ideally be balanced in terms of providing respondents an equal chance of choosing answer options that lie on the positive and negative ends of the scale. For example, in a question: “How worried are you about the possible impact of the COVID-19 pandemic on your business?”, the following answer options are biased towards capturing responses that show participants are “worried” about the impact of the pandemic.

• Extremely worried
• Very worried
• Somewhat worried
• Not worried at all

The scale could therefore be balanced out by using the following answer options instead:

• Very worried
• Somewhat worried
• Somewhat not worried
• Not worried at all

Mixing positive and negative statements

A commonly used approach in surveys is to measure certain factors based on a list of subconstructs that are theoretically known or assumed to constitute that factor. For example, Customer Service at a bank could be assessed by asking customers to rate the bank on the following sub-elements (statements).

1. The bank’s services are easily accessible.
2. The bank understands my needs.
3. My queries are resolved quickly.
4. The bank treats me as a valued customer.

Let us say the above statements are rated on a 5point Likert scale where 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4= agree and 5 = strongly agree. In such a case, the overall customer service score can be computed as the average (mean) score out of 5 where the closer to 5 the mean score, the higher the customer service rating.

Now, imagine statement c) above was instead phrased as “My queries are NOT resolved quickly”. While respondents can still correctly choose the extent to which they agree or disagree with the statement, computing the overall customer service score becomes a bit challenging. This is because the ideal answer for this statement becomes 1/5 (i.e. strongly disagree) while it is 5/5 (i.e. strongly agree) for the other three statements. Mixing positively and negatively phrased statements also compounds scale reliability assessment as measured by tests such as Cronbach’s Alpha.

The list of pitfalls above is clearly not exhaustive. Best practice is therefore to pilot test survey questionnaires prior to rolling out the survey for main data collection. This is because no amount of post-data collection analysis will fix errors associated with data that was collected using a faulty tool.

Sampling methods are either probabilistic or nonprobabilistic – not quantitative or qualitative

October 27, 2020

Reference to “qualitative” versus “quantitative” sampling methods abounds in everyday parlance as well as in some anecdotal articles on research sampling methods. But, strictly speaking, sampling methods are not quantitative or qualitative, they are either probabilistic or non-probabilistic. The fact that qualitative research usually uses non-probability sampling methods does not justify referring to those methods (e.g. purposive sampling and snowballing) as “qualitative sampling” methods. It is similarly misleading to refer to probability sampling methods as “quantitative sampling” methods. Indeed, most quantitative research studies also employ non-probability sampling techniques such as convenience and quota sampling.

A study does not become qualitative or quantitative solely because of the sampling method used. Rather, the main difference between qualitative and quantitative research is that the former is largely theory building while the latter is largely theory testing. This major difference is manifested in the form of data that is collected (i.e. words versus numbers) and in the data analysis methods used (i.e. statistical versus thematic). So, the research methodology is decided first based on the study’s analytical objectives and the appropriate sampling method (i.e. the way to go about drawing and studying a fraction of the target population) is decided thereafter, depending on the profile of the target population and other practical considerations. In my view, if the target population is large and easy to access (e.g. people who buy bread) and a probability sampling technique is easy to implement, this should be done irrespective of research methodology. By the same rule, non-probability sampling (e.g. convenience sampling) is more appropriate for finite and hard to find sub-populations and can be used regardless of research methodology.

There shouldn’t be a blanket rule that precludes the use of probability sampling in qualitative research. For example, if I was conducting focus group discussions with people who shop at a given shopping mall, I could identify and recruit one or two people who shop at the mall and thereafter ask them to refer me to other people they know who also shop at the same mall (i.e. use the snowballing technique). But I could also post recruiters at the entrance to the shopping mall and instruct them to intercept every 3rd person coming through and invite them to take part in the focus group discussion (i.e. use systematic random sampling).

It would appear one of the reasons for the conceptual exclusion of probability sampling in qualitative research is that qualitative research findings are not meant for generalization to the population from which the sample was drawn. Admittedly, that is the case in most qualitative studies. But the question is – if the population of interest is finite enough (e.g. the shopping mall example given above) and a probability sampling technique can be employed to select the study participants, why can’t qualitative research findings be made generalizable too, within certain analytical parameters? Put differently, given some known advantages of probability sampling, what harm is there in adopting this sampling methodology in instances where it lends itself to qualitative studies too?

We are scoring almost 10 out of 10 on customer satisfaction!

October 23, 2020

Customer satisfaction surveys are arguably the most common type of market research undertaken by brands, products, and service providers. This makes perfect sense given that customers are the most important asset for any business and are critical for business sustainability and success.

In line with the foregoing, and as market research consultants, a message that we constantly communicate to current and potential business clients is the importance of tracking customer experience on a regular basis. One morning, it dawned on us that we were a business too and yet we had not conducted a formal customer experience study in the 5 years of our existence. Surely, we had to practice what we preached! Admittedly, we did have a good sense of what our clients felt about our services through various solicited and unsolicited feedback, but we still needed to get feedback in a more systematic and inclusive manner. Thus, in October 2020, we invited previous clients to complete an online survey to rate our services.

A total of 30 clients who had used our services in their individual capacity (i.e. as opposed to organisational representatives) completed the survey which had just three questions as below:

1) – How did you get to know about our services?

One of the most reliable indicators of service quality and customer satisfaction is when clients who have received a service go on to recommend the same service provider to others. It is therefore pleasing that as much as 60% of the clients who completed the survey had been referred to us by previous clients. The other 40% had found information about us on the Internet.

2) – How satisfied or dissatisfied were you with our services?

Almost all (97%) of the clients were satisfied with the service they had received from us. Notably, this included 2 in 3 clients (67%) who were “extremely satisfied”. Altogether, none of the 30 clients in the survey expressed any level of dissatisfaction with our services.

3) – How likely are you to recommend our services to others?

The Net Promoter Score (NPS) measures customer loyalty by asking customers how likely they are to recommend a brand or service to others. The NPS is the percentage of customers who, on scale of 0 to 10, rate their likelihood to recommend a brand or service to a friend or colleague as 9 or 10 (i.e. promoters) minus the percentage rating this at 6 or below (i.e. detractors). Customers who give a rating score of 7 or 8 are referred to as “passives”. The NPS correlates highly to customer satisfaction and customer loyalty and has also been shown to correlate with revenue growth relative to competitors. As a general guideline, an NPS of between 0 and 30 is good, 30 to 70 is excellent and 70 to 100 is simply amazing. Our NPS from the survey was indicated at 63 and is quite pleasing. This score is indeed consistent with the earlier reported prevalence of positive word-of-mouth referrals by our clients.

Conclusion

Although a cross-sectional design such as the one we used in this survey does yield valid results, it is largely retrospective. For example, some of the clients in the survey had last received a service from us 4 years prior to the survey and, as such, their feedback was susceptible to time lapse biasing factors. We therefore decided not to close the survey whose results are reported above but to convert it into an ongoing real-time customer feedback system. Through this system, we are sending new and repeat clients a digital link on which they get to rate our services immediately after a service interaction. Among the advantages of real-time client satisfaction measurement is that we get the opportunity to correct any shortcomings before we get to serve the next client and thus improve our clients’ overall service experience.

You can check real-time feedback results from our clients here.

The importance of post COVID-19 usage and attitudes research

August 24, 2020

In market research, Usage and Attitudes (U&A) studies are a common tool that provides brands with a holistic understanding of the market in which they operate. Among other things, the purpose of U&A studies is to determine the size and value of the market for a specific product or service. The idea is to continually innovate product and service offerings, as informed by market needs and expectations, and thus sustain a competitive edge in the market.

Brands usually do not conduct U&A studies frequently. This is because consumer needs generally do not change too often, but rather evolve over time. However, there are instances when market dynamics change suddenly and drastically due to some unforeseen social, economic or such other developments. An example is the onset of the COVID-19 pandemic. The pandemic did not only disrupt health systems and economies, but also impacted consumers financially, psychologically, and behaviourally. For example, in trying to enhance their resilience against the virus, families and individuals are known to have changed their diet regimes. People changed product types, purchase frequencies, purchase channels, consumption occasions, and monthly budgets etc.

It is not known if or when the COVID-19 threat will be contained. But what is likely is that even after the epidemic has been fully contained, some of the consumer behaviours adopted during the pandemic will be retained going into the future. This could be because consumers may have accidentally found some relevance and benefits in some of these behaviours. Brands are therefore more-or-less obligated to conduct post COVID-19 U&A studies to investigate the nature and extent of consumer behaviour and other market changes occasioned by the pandemic. Such studies are important in answering the following critical questions, inter alia:

• How has product/service consumption changed?
• What products/services are consumers now using?
• Why are they using those products/services?
• How frequently are they using the products/services?
• Which channels are they using to buy the products/services?
• How much are they spending on the products/services?
• Are the products/services performing to consumer expectations?

The research methodology for the post COVID-19 U&A studies will depend on such things as the target market profiles, product/service type and specific business objectives, but would typically comprise of both qualitative and quantitative research. As we have argued elsewhere, the most insightful research is where the statistics from quantitative research are given context and meaning by the “lived” experiences and perspectives of the consumers as gathered through qualitative techniques.

Impact of data collection method on Net Promoter Scores (NPS)

June 21, 2019

Research has consistently shown that the method of data collection affects the responses given by the study participants. Among other factors, this is due to differences in the extent to which the participants feel under pressure to satisfice or to provide socially desirable responses (Holbrook et al, 2003).

We examined responses from a customer satisfaction study in which different data collection approaches were used to administer the same survey. The dual method had been adopted due to some sample accessibility challenges. The study had a sample of 1023 customers and of these 580 (57%) were interviewed using the computer aided telephonic method (CATI) and 443 (43%) were interviewed face to face.

One of the metrics in the study was the Net Promoter Score (NPS) in which customers who gave scores of 9 or 10 on their likelihood to recommend the service provider to others were categorized as Promoters. We found that 87% of the face to face respondents were in the Promoter category compared to 61% among the telephonic participants, and this was a statistically significant difference (p=0.01).

A follow up logistic regression model controlled for gender, age, race, and region. The results showed that face to face respondents were 4 times more likely to be Promoters than those interviewed over the phone (OR = 4.2, 95% CI=3.0-5.9). This raised reliability and validity issues. We concluded that the face to face respondents had probably felt under more pressure to be “nice” on the service provider and therefore recommended that responses obtained using this method be discarded altogether.

In any case, it’s better to assume happy customers are unhappy than to assume unhappy customers are happy.

Ranking responses from focus group discussions

January 27, 2019

In qualitative research, responses and viewpoints are identified and reported regardless of how often they were mentioned and irrespective of the point in the discussion or interview process they were mentioned. In other words, both spontaneous and probed responses are important. Although there is scientific merit in attaching certain interpretations to responses that are given first or spontaneously, it ought to be borne in mind that some important points or insights often only emerge after encouragement and probing by the researcher.

Another important point here is that whereas determining “order of mention” is quite feasible in unstructured qualitative interviews with just one respondent (i.e. one-to-one depth interviews), it would be a methodological flaw to try and do so in focus group discussions. First, and by instruction, focus group participants speak one at a time. Therefore “first mention” in a group discussion is really only “first mention” by the participant who happened to speak first. It does not follow that a different participant speaking first would have said the same things and/or in the same order. Secondly, participants in focus groups tend not to repeat points already covered by others. It can therefore be misleading trying to infer the importance of certain ideas or viewpoints based just on the number of times something was said in a focus group (e.g. through a basic word count). Indeed, once a point has been made and noted, moderators tend to probe for additional, or even opposing viewpoints, and not for confirmation of points already made.

In addition to the above within-group challenges, researchers often conduct the analysis across several focus groups and there is usually no uniformity across the groups in relation to what is mentioned first or what is mentioned the most. The analysis therefore focuses on identifying ideas that came up within and across the groups or ideas that came up in some groups but not in others, but without attaching too much importance on identifying the order and frequency of the opinions. That is the function of quantitative surveys.

There are several dynamics at play in focus groups that have biasing effects too. One such bias is dominant characters who tend to speak first and/or the most. There is thus a danger of ending up taking their individual opinions and projecting them as overall group opinions. That is why, where some form of ranking or quantification is required, some structured response forms are handed out for self-completion by the individual participants, or the facilitator asks for a show of hands and performs a count.

Ultimately, the real value of qualitative research lies in identifying and listing all the relevant factors in relation to the phenomenon being investigated. The importance and/or prevalence of those factors within the target audience can then be measured and reported more definitively through a follow-up quantitative survey.