The Abstraction Penalty in Research
Every concept test contains a hidden variable that most researchers never control for: the cognitive load of engaging with something that does not yet exist.
When you show a participant a description of a future product -- even a well-written one -- you are asking them to simultaneously imagine the experience, evaluate its value, compare it to current solutions, and articulate their reaction. That is four distinct cognitive tasks layered on top of social performance pressure.
The result? Participants default to one of three coping strategies:
- Polite agreement -- "Yeah, that sounds useful" (minimal cognitive investment)
- Literal interpretation -- Fixating on specific words in your description rather than the underlying concept
- Anchoring to the familiar -- "So it is like [existing product] but with [one feature]?"
None of these responses tell you what you actually need to know: whether this concept solves a real problem in a way that would change behavior.
Why Traditional Concept Descriptions Fail
The standard concept test format -- a paragraph describing the product followed by Likert-scale purchase intent questions -- was designed for physical goods in the 1970s. It worked reasonably well when the "concept" was a new flavor of toothpaste or a slightly different razor configuration.
For digital products, services, and platform experiences, this format catastrophically underperforms. Here is why:
Digital experiences are contextual. A feature that sounds useless in a survey becomes indispensable at 11pm when you are trying to finish a report. Without context, participants evaluate concepts in a vacuum that bears no resemblance to their actual decision environment.
Value is emergent. The value of most modern products comes from repeated use patterns, network effects, or workflow integration -- none of which can be communicated in a description. Asking someone to evaluate Slack from a paragraph description in 2012 would have produced meaningless data.
Abstraction triggers different evaluation criteria. When something is concrete (a prototype you can click), people evaluate it experientially -- does this feel right? When something is abstract (a description), people evaluate it analytically -- does this make logical sense? These are fundamentally different cognitive processes that produce different conclusions.
This is the scaffolding problem: without concrete anchors, you are not testing the concept. You are testing your participant's ability to imagine it.
The Four Levels of Concept Scaffolding
Effective concept testing requires matching your scaffolding level to the maturity of your idea and the sophistication of your research question.
Level 1: Scenario Anchoring
Instead of describing the product, describe the situation where someone would need it. This is the lowest-investment scaffolding that still dramatically improves data quality.
Without scaffolding: "We are building a tool that automatically summarizes meeting transcripts and extracts action items. How useful would this be to you?"
With scenario anchoring: "Think about last Tuesday. You had back-to-back meetings from 10am to 3pm. At 3:30, your manager asks for an update on what was decided in the product review meeting. You cannot remember the specifics because it was three meetings ago. In that moment, what would you actually do?"
The scenario approach works because it activates episodic memory rather than hypothetical reasoning. Participants are now evaluating from a position of felt experience, not abstract logic. Their responses connect to actual behaviors rather than post-hoc rationalizations of what they think they would do.
Level 2: Analogy Bridges
For more complex concepts, use analogies to existing products or experiences that participants already understand. This reduces the cognitive load of imagination while still leaving room for genuine evaluation.
Without scaffolding: "An AI-powered research assistant that automatically identifies themes across multiple interviews and surfaces contradictions in participant responses."
With analogy bridge: "Imagine if your research repository worked like a really sharp junior analyst -- someone who had read every transcript, remembered every quote, and could immediately say 'Wait, that contradicts what participant 7 said last week.' You still make the analytical decisions, but the pattern-spotting happens automatically."
Analogy bridges work because they give participants a mental model to reason from. They can now think about gaps in the analogy ("But a junior analyst would understand context that AI might miss") rather than struggling to form any mental model at all.
The key is choosing analogies that are close enough to be useful but imperfect enough to invite critique. If your analogy perfectly captures the concept, you do not need the concept -- the existing product already satisfies the need. As practitioners of collaborative analysis know, the best insights emerge from productive disagreement with a framework.
Level 3: Concrete Artifacts
Prototypes, mockups, storyboards, and wizard-of-oz demonstrations. This is where most teams think concept testing starts, but it is actually the third level -- and jumping here without the lower levels often produces misleading data.
The risk of concrete artifacts is premature specificity. Participants react to the specific implementation rather than the underlying concept. They tell you the button should be blue instead of green, when you need to know whether the entire feature category matters to them.
The solution is to use artifacts that are deliberately incomplete -- sketchy enough to signal "this is not final" while concrete enough to engage experiential evaluation. The research on visual elicitation techniques demonstrates that imperfect artifacts consistently produce richer participant responses than polished ones.
Level 4: Experiential Simulation
The gold standard: letting participants actually experience a version of the concept in their real context. Wizard-of-oz testing, concierge MVPs, or time-bounded trials.
This level produces the highest-quality data but requires the most investment. Use it when the concept is high-stakes and the lower levels have produced ambiguous results.
Common Scaffolding Mistakes
Over-scaffolding: Providing so much context and explanation that you lead participants to the "right" answer. If your scaffolding takes three minutes to deliver, you have built a sales pitch, not a research stimulus.
Inconsistent scaffolding across participants: If participant 3 gets a detailed scenario and participant 7 gets a one-sentence description, your data is incomparable. Standardize your scaffolding while leaving response space open -- a principle well-established in progressive disclosure interview design.
Testing the scaffold instead of the concept: If participants love your prototype but the underlying need does not exist, you have validated your design skills, not product-market fit. Always include questions that probe the underlying problem independent of your specific solution.
Single-format dependency: Using only one type of scaffolding across all participants. Different people engage differently with scenarios vs. prototypes vs. analogies. The most robust concept tests use at least two scaffolding levels and triangulate across them -- applying the same multi-source evidence principles that drive reliable decision-making in any domain.
Measuring Scaffolding Effectiveness
How do you know if your scaffolding is working? Track these signals:
Response specificity: Are participants giving concrete, detailed reactions ("I would use this on Mondays when I prep for my weekly report") or vague generalities ("Yeah, that could be useful sometimes")? Higher specificity indicates better scaffolding.
Unprompted elaboration: Do participants spontaneously extend the concept ("Oh, and could it also...") or do they wait for your next question? Spontaneous elaboration means they have internalized the concept enough to reason about it independently.
Critique quality: Are participants offering substantive critiques ("This would not work for my team because we use asynchronous communication") or surface objections ("I do not like subscriptions")? Substantive critique requires genuine engagement with the concept.
Behavioral indicators: Do participants lean forward, ask clarifying questions, or relate the concept to specific situations in their life? Physical engagement signals cognitive engagement.
The best concept tests are not the ones that produce the highest purchase intent scores. They are the ones that produce the richest data about how the concept connects (or fails to connect) to people's actual lives. Scaffolding is what makes that richness possible.
The Scaffolding Stack in Practice
For your next concept test, try this structure:
- Open with scenario anchoring (2-3 minutes) -- Ground participants in a real situation where the problem exists
- Introduce with analogy bridge (1-2 minutes) -- Give them a mental model to reason from
- Show a concrete artifact (3-5 minutes) -- Let them engage experientially
- Probe beneath the surface (10-15 minutes) -- Ask about the problem, not just the solution
This layered approach builds comprehension incrementally. By the time participants see your artifact, they already understand the problem space and have a mental model to evaluate against. Their responses will be grounded, specific, and actionable -- which is all research ever needed to be.
The difference between a concept test that produces "7 out of 10 participants expressed interest" and one that produces "here is exactly how this concept fits (and does not fit) into our users' workflows" is not sample size or statistical rigor. It is scaffolding.
