WIXLIB

TOWARDS MOREAGILE AND EFFICIENTPRODUCT TESTINGOpportunities and limitations for small sample sizesAuthor: Nikolai Reynolds, Josef Zach, Jinho ChoContributor: Colin HoMay 2021ABSTRACTThe historic assumption that larger sample sizes are neededwhen testing prototypes in the early stage of the productfor product tests derives from the hypothesis that there isdevelopment and (for cost rationalization studies) in the latera risk of unreliable and varying consumer responses, i.e.,stage of product development. Our findings suggest smallvariance. Today, large consumer panels and databasing ofsample sizes can be considered when the objective is toconsumer responses allow us to revisit historic variancecreate differentiating products such as for pre-screeningassumptions for product development. In this paper weof prototypes. Such pre-screening allows us to save costinvestigate the variance of products tested across regionsand time by reducing the number of products required forand categories using different scales from 36,500 consumersfurther testing. However, for other types of testing such asin our database. We assess how reliable a smaller samplecost rationalization studies, or when subgroups need to besize of n 50 is versus larger samples of n 150 or moreanalyzed, larger sample sizes are recommended.IPSOS TOWARDS MORE AGILE AND EFFICIENT PRODUCT TESTING1

SAMPLE SIZE CONSIDERATIONSProduct testing measures properties or performance ofTherefore, a good planning of the right sample sizeproducts. It covers any process in which a researcherconsidering the data collection is crucial for the quality ofmeasures a product’s compliance, performance, safety, andany intended research (Maxwell et al, 2008).quality. To assess the product properties or performance,product testing survey data can be collected in many waysThe sample size also has implications for the investmentfrom face-to-face, telephone, video-based interviewing overneeded to run the research. Prototypes or in-marketto online and mixed mode designs.products need to be provided, shipped, or placed and theempty packaging shipped back or destroyed. Large sampleThe type of data collection needs to be chosen in such asizes drive these logistical investments and, in many cases,way that a sufficient sample size can be collected from thealso the research timing. To reduce the need for largerrelevant target population (Wilkinson and McTiernan, 2020).sample sizes and thus more test products, it is importantThe relevant target group sizes in the total population arethat the sample structure and quality are paid attention to inoften, depending on the category and market, in millions.the planning (Ramsey and Wagner, 2015).LARGE SAMPLE SIZES ARE NOT NEEDED IN EVERY AREADepending on the scope and subject of research, the levelof statistical means (Gosset and Fisher, 1925). His sampleof variance between a sample and the total population cansizes were below 50.be different. For instance, if we were to assess how anindigenous population raise their children, variance could beAs products may differ even when factory settings have beenhigh. But if we were to examine how the indigenous languagemaintained, it is important to select products from differentin its grammar is constructed, the variance would be lower,production batches. But typically, in product research, thereand a small sample size could be considered.is more variance coming from consumer perceptions thanfrom the variance within the same products from a factory,In the medical industry smaller sample sizes are commonlyi.e. batch variability.selected, such as n 15. If the measurements in thesetests do not vary much between people, it is possibleThe sources of such variance can be manifold: if theto handle n 15 with statistical procedures (significancesampling frame is incorrectly defined, question items uncleartest, estimating types of error). Similarly, in the productor questionnaires too long, response behavior can deviatedevelopment cycle, products need to run through clinicalstrongly. Such variance can lead to the situation where ittrials too before being tested by consumers. In these clinicalcan be observed that two identical products are statisticallystudies smaller sample sizes are also often chosen. But evensignificantly different in performance.after clinical approval, we find product tests with smallersample sizes with sensory expert panels ranging fromAt Ipsos, we put an extra effort into applying strict surveyn 10-50 (Lawless and Heymann, 2010) as well as within theresearch rules to avoid such situations by reducing thefactory where batch variability is assessed.margins of error (qualification of respondents, short enoughquestionnaire, avoiding response directed questions, clearTesting the batch variability is basically one of the startingdescription of question items, unbiased interview dynamicspoints in history for significance testing in product testing.etc.). As we put all these quality measures in place in ourWilliam Sealy Gosset, who was a master brewer at Guinness,sampling, the hypothesis is derived that data quality shouldassessed differences between batches of beer, i.e. batchprovide the same robustness, independent of its sample size.variability, by comparing arithmetic means with each other,later using his pen name “student” t-distribution and testIPSOS TOWARDS MORE AGILE AND EFFICIENT PRODUCT TESTING2

Figure 1 Ipsos Product Development Life CycleExplorationIdentify the next generationproduct features and benefitsCost Savings/QIGuidanceEnsure profitablelongevity in-marketPrototype Screeningand OptimisationRenovationBenchmarkingMonitor product’sin-market performanceInnovationValidationRefine products forin-market successSource: Ipsos 2021WITH GOOD DATA QUALITY, DO SAMPLES WITH N 50 PROVIDENEARLY IDENTICAL RESULTS AS SAMPLES WITH N 150 ?As the world’s largest product tester, Ipsos has theFor each study we considered first the full sampleadvantage to build upon massive data sets from consumer-(depending on the study: n 150-450) and estimated the gapbased product testing. To answer the key question ofbetween the best and worst performing products in overallwhether we would come to the same conclusion with aliking, i.e., “best-worst gap”. Using a Monte Carlo Simulationsmaller sample size than with a larger, we randomly selectedwith 10,000 iterations, we estimated the best-worst gaps ofa subset of studies from the Ipsos Product Database.smaller sample size of n 50 per study. To allow a relativeTo answer the second question, at what stage of productcomparison between different Overall Liking scales, wedevelopment smaller sample sizes can be considered, wedivided the best-worst gap by the number of scale points.selected studies related to guidance testing related prototypeWe then calculated a Pearson correlation coefficient betweenscreening and cost saving measures (see Figure 1).average consumer acceptance scores of all tested productsof a small sample size (n 50) and the average consumerIn total, the analyses encompassed 36,779 consumeracceptance scores from the original sample size of eachresponses to how much they liked the product testedstudy. We ran the analyses separately for early-stagerated either a 9-point, 7-point or 5-point Overall Likingprototype tests and for cost rationalization studies.scale across Africa, Asia, Europe, Latin America, NorthAmerica for 185 consumer goods products. To considercategory effects, we covered beverages, food, non-humanfood, personal and home care studies. These studies wereconducted between 2015 to 2021.IPSOS TOWARDS MORE AGILE AND EFFICIENT PRODUCT TESTING3

SMALL SAMPLE SIZES FOR EARLY-STAGE PROTOTYPE SCREENINGFigure 2 depicts the relationship between the correlationsdifferences between the prototypes were 20% or larger,and the relative best-worst gaps of each study. From thesemeasured by consumer acceptance, were stable with arelationships we can derive a very strong correlation of 0.9small sample size of n 50. This means that for early-stageif the relative best-worst gap is 20% of the scale range orproduct tests, we should consider prototypes which aremore. Regarding category or regional effects, no specificdifferentiating. Sensory panels can make sure differentiatedpatterns can be identified. Studies in which the performanceproducts are selected for consumer-based screening.Figure 2 Similarities of results using smaller sample sizes in prototype screening studiesSouth America,Food/Beverage1.000.95North America,Pet Food0.90Asia, Food/BeverageNorth America,Food/BeverageEurope, Pet FoodSouth America,Food/BeverageAfrica,Personal Care0.85Africa,Food/BeverageEurope,Pet FoodAsiaPersonal CareEurope,Food/Beverage0.800.75Asia, Food/Beverage0.700.650.600%10%20%30%40%50%60%Note: Y-axis: Pearson Correlation of n 50 vs n 150-n 300, X-axis: relative difference of best and worst performing products withineach study, i.e., “best-worst gap”. To statistically assess whether the gap of the mean scores explain the similarities between smaller andlarger samples, we applied a logit transformed regression model. The logit transformed regression model presented as Figure 2 reveals thatdifferences and correlations are not constant, i.e., are heteroscedastic. To counter heteroscedastic effects, a common method in statisticsis to conduct a logit transformation (Greene, 2002). The logit transformed regression model is statistically significant. The best-worst gapexplains the similarities between the larger and smaller sample sizes with 99% confidence.Source: Analysis of selective studies in early-stage product tests from Ipsos Product Testing Database.SMALL SAMPLE SIZES FOR LATE-STAGE COST SAVING STUDIESCost saving studies have an opposite objective to early-stagesuch studies it is often about ensuring product performanceprototype development. While in prototype development it isis maintained when removing/modifying ingredients to makemore about maximizing differentiation to achieve superiority,a more profitable, for example because there are regulatoryin a cost rationalization study it is about minimizingchanges, or changes in availability of specific ingredients.differentiation, i.e., creating nearly identical products. InIPSOS TOWARDS MORE AGILE AND EFFICIENT PRODUCT TESTING4