Article by ICG member Matt Gibbs
A practical guide for insights leaders evaluating agentic AI solutions
We’re increasingly hearing the question: can’t I just drop survey and insights data into ChatGPT, Claude, or Copilot? The short answer: Yes, you can. But you’ll encounter serious problems with data security, numerical accuracy, and analytical consistency. Having spent the past few years building agentic systems specifically for market research, I’ve learned what works and what doesn’t. Here are six principles that should guide any implementation.
1. Keep Your Data Out of Public Endpoints
As soon as you paste data into a public model endpoint, you lose control. You cannot govern what it may be used to train, who may access it, where it’s stored, or for how long. This is likely to breach data processing agreements and industry codes of conduct. Quant or Qual, the answer is the same.
The right approach: Any implementation must use isolated model instances within a controlled cloud environment, ideally inheriting enterprise security architecture from one of the big providers like Microsoft or AWS. Data should remain within defined boundaries, with demonstrable compliance.
2. Never Let the Model “See” Raw Numbers
When you paste data into a chat model, everything becomes tokens. The model doesn’t compute statistics. It predicts what plausible-looking numbers might be. This leads to percentages that don’t sum correctly, fabricated base sizes, “insights” that contradict the actual data, and no provenance. You simply cannot verify where a number came from.
The right approach: The model should never consume raw data as tokens. Instead, give it access to tabulation engines and insights tools tested in the real-world of market research for decades. That way the model can compute exact figures on demand. When an agent reports “32% of prospects are aware of Brand X,” that number should come from a validated statistical computation, not a language model estimate. Every insight must trace back to a reproducible run, auditable by design.
3. Preserve Your Curated Table Libraries
Research teams invest significant effort building validated table specifications: the correct variables, consistent filtering, proper weighting, meaningful labels. Dropping raw data into a generic chat model discards all that intellectual property.
The right approach: Build systems where the agent can search and run from a library of pre-built, validated table specs. Any data point should be traceable to source. Better yet, enable the system to discover what analysis has been done before through conversation history at the organisational level. This institutional knowledge compounds over time.
4. Demand Statistical Rigour
Survey analysis requires demographic weighting, subgroup filters, and significance testing. Chat models have no native capacity for applying weighting schemes consistently, filtering to target populations, calculating statistical significance between groups, or warning when base sizes are too small to report. These are language models, not statistical engines. They cannot reliably execute the same analysis twice.
The right approach: Combine dynamic spec generation by models with a proper cross-tabulation engine producing the numbers. Users should be able to instruct in conversational English, with models generating the coding syntax and passing it to a validated engine. The statistical computation must be deterministic and reproducible.
5. Embed Sector-Specific Expertise
Generic chat models know nothing about your organisation, your sector, or your projects. Every conversation starts from zero.
The right approach: Build contextual understanding into the system. The agent should know your organisation, your projects, your data structure, and your previous analyses. Embed sector-specific expertise that creates organisational “common sense.” Pre-built workflows should exist for common research tasks: segmentation, TURF analysis, correspondence mapping, brand tracking, verbatim coding, strategic reports. These encode lessons from years of practice. Multiple team members should receive consistent analysis approaches, not whatever the model happens to generate that day.
6. Optimise for Speed to Insight
Using a raw model for survey analysis requires extensive prompt engineering, data formatting, and manual verification. Not every stakeholder who needs insight has time to coach a model into usefulness, repeatedly, across every conversation.
The right approach: Eliminate ramp-up time. Analysts shouldn’t need to re-explain data structures, variable names, or project context. Pre-built table libraries should mean common questions resolve in seconds, not hours. Outputs should be client-ready: branded presentations, interactive dashboards, and web visualisations, not chat text that needs reformatting. Most importantly, implement MR-grade guardrails. The system should refuse to fabricate quotes, validate numbers against source tables, use hedged language appropriately, and separate technical statistics from executive summaries.
Beyond Pre-Built Workflows
The most powerful implementations give the model access to a full coding environment within a secure, isolated workspace. Users can request bespoke analyses, custom visualisations, or entirely new tools simply by describing what they need in plain English. The agent writes and executes code on the user’s behalf. This means an insights director can say “build me an interactive dashboard showing brand health over time with drill-down by region” and receive a working application, not a description of how one might be built. The same applies to statistical analyses that don’t fit pre-packaged workflows, custom data transformations, or one-off client deliverables.
This is the emerging capability of agentic software: code generation orchestrated by conversation, constrained by guardrails, and delivered within a controlled environment. The opportunity is to provide this capability to research teams without requiring them to become programmers.
Summary
Raw AI gives you a conversation. A properly architected agentic system gives you a research capability. The model is the easy part. The tooling around it is where value is created. The questions that serious insights directors should be asking of any solution are:
- Where does my data go, and who controls it?
- How are numbers computed, and can I trace them to source?
- Does the system understand my existing analysis, or do I start fresh each time?
- What safeguards prevent fabricated insights reaching stakeholders?
Any solution worth considering should answer these questions with confidence.
Market research has built its reputation on transparent methodology and data governance over decades. If a significant portion of our sector begins using public AI models without safeguards, we don’t just risk individual credibility, we risk the trust that underpins the entire industry. There are tools that can help. Tools that are designed to serve our professions. Please use them.