Designing Perceived Competence in AI Tutors
Problem & Target Users
An experimental HCI study exploring how an AI tutor's communication style shapes perceived competence, learning performance, and user experience.
My Role
UX Research & AI Interaction Design
Collaborative research study involving experimental design, Gemini Gems customisation, prompt engineering, pilot testing, participant data collection, data analysis, and iterative refinement.
Project Type & Context
Academic Research · Human–AI Interaction · 35 Participants
Methods & Participants
01 The Challenge
Conversational AI systems are increasingly used as learning companions and tutors. While two AI models may draw from the exact same factual information, the way an AI presents knowledge—its tone, hesitation, structure, and phrasing—can profoundly shape how capable it appears to learners.
This study investigated how an AI tutor's communication style shapes human learning outcomes and subjective perceptions without altering underlying knowledge. Our research compared two customised Gemini Gems with different communication styles while maintaining the same educational content and overall interaction structure. Neither tutor was designed to be objectively more knowledgeable than the other; instead, the study isolated communication style as the experimental manipulation to observe its direct effects on learners.
Core Research Question
“How does an AI tutor's perceived competence influence learning performance, confidence, and user experience?”
02 Research & Exploration
To explore this question rigorously, we examined how linguistic markers such as hesitation, hedges, and structural clarity impact perceived competence in human–AI dialogue.
We translated these insights into an experimental methodology that preserved educational invariants while methodically contrasting communication styles across learning sessions.
Two customised Gemini Gems: the same learning material and interaction structure, but contrasting communication styles.
35 participants experienced both conditions in a counterbalanced, within-subjects study, completing learning tasks about AI categories and cognitive biases.
Measure immediate task accuracy, answer confidence, trust and interaction experience, followed by a learning assessment 24 hours later.
Figure 01: Behavioural Design Matrix
Contrasting conversational dimensions between the Competent and Less Competent Gemini Gems across Explanations, Structure, Feedback, Interaction style, and Tone.
03 Design & Iteration
Prompt engineering served as the interaction design method for constructing both Gemini Gems. The design decisions were structured around shared educational constraints and contrasting conversational personas:
Shared Constraints
Same knowledge sources, educational content, overall task structure, and interaction rules across both conditions.
Higher-Competence Persona
Clear, structured, confident and factual explanations and feedback, authoritative summaries, and unambiguous guidance.
Lower-Competence Persona
Tentative language, expressed uncertainty, hedges, self-questioning, and less confident feedback.
Figure 02: Prompt Refinement & Shared Constraints
Iterative 3-stage prompt evolution (Initial exploration → Refined prompt → Final prompt based on pilot testing) with identical educational invariants.
What the pilot changed
A two-person pilot highlighted difficulties switching between tutors, possible effects of prior topic knowledge, and confusion around one trust-scale dimension. The team responded by introducing supervised sessions, refining participant eligibility, and removing the confusing security item from the TIAS questionnaire.
04 Outcomes & Learnings
The study evaluated responses from 35 participants who experienced both conditions. The data revealed three core findings regarding task performance, subjective trust, and delayed knowledge retention:
+0.89 correct answers
On average, participants answered 0.89 more immediate assessment questions correctly after using the higher-competence tutor. The difference in immediate accuracy was statistically significant.
Similar perceived trust
Despite the difference in task accuracy, trust and several user-experience measures remained broadly similar between the two conditions.
An open question about retention
Both conditions showed reduced performance after 24 hours. Although the higher-competence condition showed a larger observed decline, the difference in retention was not statistically significant.
05 What this study taught us
Designing an AI learning experience involves more than making its responses sound convincing. This study showed that communication style can influence immediate learning performance, while perceived trust does not necessarily reflect those differences. It also raised an important question for future research: how might AI tutors support learners' understanding and agency, rather than simply delivering confident answers?
Exploratory study with 35 convenience-sampled participants, short learning tasks, and a 24-hour follow-up. Results should not be generalised to long-term educational outcomes.