Exploring motivations,response quality,and fit in ChatGPT-assisted information tasks

Using large language model tools for information tasks

Information tasks are goal-directed activities that involve seeking, acquiring, evaluating, and using information to meet specific objectives (Byström and Järvelin, 1995; Vakkari, 1999). They are fundamental to both everyday life and professional domains. Broder (2002) categorizes web-oriented information tasks as informational, navigational, and transactional, highlighting distinct user goals that shape search behavior. Prominent theoretical models, including Dervin’s sense-making (Dervin, 1992), Bates’s berry-picking (Bates, 1989), and Kuhlthau’s Information Search Process (Kuhlthau, 1991), describe the behavioral patterns and cognitive stages users pass through when performing information tasks. The introduction and rapid integration of large language model tools, such as ChatGPT, Copilot, and Gemini, have reshaped how people perform information tasks. By offering generative, conversational support within a single interface, LLM tools enable a range of activities that previously required multiple tools or greater effort to coordinate, ranging from text generation, summarization, coding assistance, ideation, and more (Noy and Zhang, 2023; Wang et al., 2024b).

Recent empirical work has begun to map the diverse user intents associated with LLM tools. More recent large-scale descriptive analyses conducted after our study (e.g. Chatterji et al., 2025) likewise find that everyday LLM tool use clusters around information-oriented activities such as writing, learning, and practical problem-solving. Wang et al. (2024b) derive a multi-intent taxonomy from usage logs and user surveys, while Bodonhelyi et al. (2024) examine intent recognition and user satisfaction across informational, problem-solving, creative, educational, technical, and professional contexts. In educational settings, students most commonly use LLM tools for explanation, feedback, and writing refinement (Wang and Fan, 2025). Researchers further report that people turn to AI tools for information tasks when they want fast synthesis or a one-stop answer, especially when traditional search feels overwhelming, yields noisy or poorly aligned results, or when users have limited domain knowledge or face ill-defined tasks (Yen et al., 2024; Zhou and Li, 2026).

A key factor shaping how people experience information tasks, with or without LLM tools, is task complexity. In information science, task complexity has been studied extensively as a determinant of search behavior, information use, and user experience. Byström and Järvelin (1995) proposed an influential framework in which task complexity is defined by a priori determinability: the degree to which the task process, information requirements, and expected outcomes can be predicted in advance. Campbell (1988) identified multiple paths, uncertainty, and novelty as core drivers of complexity. Empirical work has shown that users’ familiarity with the task topic and domain knowledge significantly shape their search strategies, perceived difficulty, and information outcomes (Bell and Ruthven, 2004; Wildemuth, 2004). More recently, studies of search task complexity have explored how task structure, user knowledge, and system affordances interact to determine task experience (Choi et al., 2020; White, 2016; Wildemuth et al., 2014). As LLM tools become common for information tasks, understanding how task demands and users’ familiarity with the topic, required skills, and procedures shape interaction and outcomes becomes increasingly important. Familiarity is one important dimension of perceived task complexity, but it does not exhaust the broader construct.

These findings increasingly link usage patterns to perceived utility, creativity support, and cognitive assistance. These intents continually encourage reliance on LLM tools as responsive, context-aware assistants for diverse information tasks.

User experience with LLM tools for information tasks

User interaction with LLM tools involves dynamic, iterative processes where users typically pose an initial query, inspect the response, and then reformulate or provide feedback (Trippas et al., 2024). Researchers have identified significant behavioral patterns that differ from the use of traditional search engines. In a study with 1526 participants performing a product comparison task, Kaiser et al. (2025) found that participants using ChatGPT wrote longer, more conversational prompts and visited fewer web pages compared to participants using Google. Another study observed that Bing Chat (ChatGPT) users asked fewer but longer question-style prompts and tended to rephrase questions, whereas classic Bing searchers issued more, shorter queries and refined searches by adding terms (Wazzan et al., 2024). In a study of STEM students solving physics problems, researchers found that ChatGPT interaction commonly involved copy/pasting the full text of the questions to ChatGPT and asking for explanations, translations, summaries, or corrections; in contrast, 96% of the Google queries were “systematic” in that they involved “extracting keywords or dividing the question” (Krupp et al., 2023).

Effective interaction with LLM tools depends on the model’s capacity to adapt to context and users’ knowledge of prompt engineering to create a collaborative environment for accomplishing complex informational goals (Bodonhelyi et al., 2024; Wang et al., 2024a). Bodonhelyi et al. (2024) found that GPT-4 outperformed GPT-3.5 on the recognition of common intents, but is conversely often outperformed by GPT-3.5 on the recognition of less frequent intents. Moreover, whenever the user intent is correctly recognized, users are more satisfied with the answers to intent-based reformulations of GPT-4 compared to GPT-3.5, but they tend to be more satisfied with the answers of the models to their original prompts compared to the reformulated ones. Another study by Kim et al. (2024) found that users often lack effective strategies to mitigate their dissatisfaction when LLM tools fail to understand their intentions. They also noted that users with a limited understanding of LLM tools tend to experience greater dissatisfaction and are less proactive in addressing the challenges they experience.

Surveys and empirical studies show that clearer, more structured prompts tend to yield higher-quality outputs and reduce error/ambiguity, making prompt engineering an important user competency in information tasks (Trad et al., 2025).

Output quality evaluation and challenges of using LLM tools

Comparisons of users’ performance with GenAI tools among different tasks have found mixed outcomes. Evaluations show that while LLM tools excel in generating coherent and fluent text and provide useful explanations, they face challenges in maintaining factual accuracy, complex reasoning, and handling nuanced ethical considerations. In a study involving product comparison, ChatGPT users finished faster and were more accurate, but Google users reported higher reuse intentions (Kaiser et al., 2025). For image-geolocation tasks, traditional search yielded higher accuracy and better supported exploratory refinement than chat alone (Wazzan et al., 2024). In physics problem-solving, ChatGPT users scored lower and over-trusted its outputs, highlighting the risk of unverified acceptance (Krupp et al., 2023). In a study involving socio-scientific reasoning, participants who used ChatGPT-3.5 felt the task was easier (lower cognitive load) but produced weaker justifications; in contrast, participants who used traditional web search yielded stronger, better-supported conclusions (Stadler et al., 2024).

Challenges and concerns remain: Hallucination remains a persistent reliability issue that can mislead users and erode trust in AI-assisted information systems (Ji et al., 2023). Biases embedded within LLM tools, stemming from imbalanced training data, raise ethical concerns by perpetuating stereotypes or discriminatory behaviors (Gallegos et al., 2024). Privacy risks include the possibility of training-data extraction from deployed models (Yao et al., 2024). Finally, there is growing recognition of risks related to overreliance on LLM tool outputs without adequate verification and a decrease in critical thinking, underlining the need for improved user education and interface design (Krupp et al., 2023).

Taken together, prior work shows clear utility of LLM tools for drafting, explanation, and synthesis, but uneven reliability for tasks demanding verifiability, multi-step reasoning, or domain rigor. Outcomes consistently depend not on the model alone but on task properties, technology affordances, and user practices. Our work offers a descriptive baseline of how tasks, motivations, perceived response quality, and perceived fit co-occur in real-world ChatGPT use, and we suggest directions for future confirmatory research.

Survey design

To ensure that participants could report on recent experiences, we used a screening question at the start of the survey asking whether they had used ChatGPT for assistance or to complete tasks on at least two separate occasions since January 1, 2024, with survey data collected in March 2024. Only those who answered yes proceeded to the main questionnaire after providing informed consent.

The questionnaire comprised three parts: (1) demographics (age, gender, occupation, and education); (2) general experience with ChatGPT (version access, frequency of use, breadth of use contexts, and self-assessed understanding of functionality and prompt-crafting skill); and (3) two recent tasks completed with ChatGPT’s help. To elicit variation in tasks, we asked participants to describe two recent ChatGPT-supported tasks: first one they felt was relatively “easy” and then one they felt was more “challenging.” The task order was fixed for all participants; we discuss the implications of this design choice in the Limitations section. Participants then indicated when they completed each task, explained why they used ChatGPT, and rated their familiarity with the task topic, required skills, and procedures on separate 1–5 Likert scales.

These familiarity ratings were significantly higher for the self-identified easy task than for the challenging task on all three items (Table 3). For readability, we sometimes refer to these as higher-familiarity and lower-familiarity tasks. This terminology should be understood as a descriptive shorthand based on participants’ self-reports, not as evidence that the survey experimentally manipulated familiarity or task complexity.

The survey deliberately used everyday language (“easy” and “challenging”) rather than the technical term “task complexity” to elicit natural variation in participants’ real-world experiences. Prior work shows that familiarity with the topic, required skills, and procedures is one important dimension of perceived task complexity (Bell and Ruthven, 2004; Byström and Järvelin, 1995; Campbell, 1988; Wildemuth, 2004). Accordingly, we interpret the easy/challenging contrast as related to, but not equivalent to, differences in familiarity and perceived task complexity.

Next, participants were asked to rate ChatGPT’s response to each task on five dimensions using a common stem (“Please indicate your rating of the answer provided by ChatGPT for this specific task”) followed by five statements: the answer was specific, relevant, accurate, understandable, and creative (each on a 1–5 Likert scale). These five facets were chosen to capture complementary aspects of how well ChatGPT supported task completion: specificity and relevance are core criteria in information-retrieval and question-answering evaluations, accuracy reflects the extent to which content is factually or procedurally correct, understandability speaks to clarity and ease of following responses, and creativity captures the novelty and generative value of outputs that is central to LLM tools (Guzik et al., 2023; Scquizzato et al., 2024). We treat these as distinct, face-valid indicators of perceived response quality rather than as a single latent scale, and therefore report them separately instead of aggregating them into one composite score. Finally, participants were asked to reflect on what made them think ChatGPT was or was not a good fit for the task, and whether they had any concerns or challenges during the task process. The complete survey instrument is provided in Appendix 1 (Table 1).

Data collection and preparation

We fielded a Qualtrics online survey on Amazon Mechanical Turk (MTurk) during March 2024, using platform checks to discourage multiple submissions (Hauser and Schwarz, 2016; Mason and Suri, 2012). Eligible workers were located in the United States, had a minimum approval rating of 95%, and at least 100 approved HITs, criteria associated with more attentive and reliable responses.

Participants received $5 as compensation once their responses were accepted. Responses with substantial missing or obviously irrelevant answers (e.g. blank task descriptions, copied or repeated text) were rejected, and the participant did not receive compensation. The study was approved by the Institutional Review Board of the authors’ institution.

We received 188 submissions. Following an initial quality screen, we excluded 62 cases for one or more of the following reasons: (a) missing or blank task descriptions; (b) evidence of automated/LLM-generated content (e.g. highly templated phrasing, repeated long-form responses across different respondents, or internal inconsistencies); and (c) duplicate entries. We then reviewed the remaining responses to ensure that each task had a specific goal and that participants had actually used ChatGPT to complete it, removing 16 additional cases that did not meet these criteria. The final analytic sample comprised 110 participants, each contributing two tasks (220 tasks in total: 110 familiar and 110 unfamiliar).

Data analysis

Our qualitative coding drew on four open-ended questions per task (see Table 1 (Appendix 1)). Participants’ task descriptions were coded to produce the task-type taxonomies reported in RQ1 (Tables 1 and 2). Their reasons for using ChatGPT were coded to identify motivations (RQ1, Table 4). Their retrospective fit reflections and concerns/challenges were coded to identify perceived fit reasons and concerns (RQ3).

OPEN IN VIEWER

Table 1.

Taxonomy of participant-selected relatively easy tasks ( N = 110 ).

Task type	Definition	Prevalence	Example
Short creative projects	Generate a short poem, story, slogan, or programming for a novel project.	30.9% (n = 34)	“Write a fun poem about a dog.” (P_19_easy)
Recommendation	Suggest movies, gifts, diet (recipe) or vacation spots with light criteria.	28.2% (n = 31)	“I asked for a gift idea for a political event. . . under 100 dollars” (P_40_easy)
Factual lookup	Retrieve a single fact, date, list, or statistic with no analysis	24.5% (n = 27)	“I needed a list of the last 10 leaders of Russia and the years. . .” (P_1_easy)
Text editing/proofreading	Rephrase paragraphs for tone, clarity, or professionalism.	8.2% (n = 9)	“I needed to edit a very simple paragraph. . .” (P_2_easy)
Quick explanation	Explain a single concept in plain language.	8.2% (n = 9)	“I asked ChatGPT to define off scale in photography. . .” (P_28_easy)

The table reports task types, brief definitions, their prevalence in the sample, and illustrative user prompts.

OPEN IN VIEWER

Table 2.

Taxonomy of participant-selected more challenging tasks ( N = 110 ).

Task Type	Definition	Prevalence	Example
Research and information synthesis	Focuses on compiling, analyzing, or explaining in-depth information on topics such as history, science, law, or data, often requiring synthesis from multiple sources or perspectives.	21.8% (n = 24)	“I wanted to get an overview of the arguments made by the anti-abolitionists of the civil war era” (P_11_challenging)
Creative writing and content creation	Generating original narrative, artistic, or expressive content, often requiring imagination, detail, and iteration to achieve the desired output.	15.5% (n = 17)	“I asked it to write a children’s story book with alot more details and presented in certain order and manner” (P_27_challenging)
Technical tutorials and troubleshooting	Covers step-by-step instructions for repairs, installations, or resolving technical issues with devices or systems, or problems that require logical reasoning.	14.5% (n = 16)	“I was troubleshooting a firewall-it runs on a Unix shell so I was trying to find relevant commands to find information” (P_72_challenging)
Professional and business writing	Drafting or refining business-oriented documents, strategies, or communications, emphasizing professionalism, structure, and audience appropriateness.	13.6% (n = 15)	“I asked ChatGPT to help me write a business plan for a weekly farmers’ market in our neighborhood.” (P_31_challenging)
Personal advice and decision making	Seeking guidance on individual life issues, such as relationships, health, finances, or emotional challenges.	13.6% (n = 15)	“I asked about how to help my friend fight their drug addiction.” (P_61_challenging)
Planning and organization	Entails structuring events, budgets, schedules, or tools for organization, often with constraints like budget or space.	10.9% (n = 12)	“I wanted to plan a surprise party for someone. . .To me this is tough as it is a layered situation with many aspects.” (P_55_challenging)
Programming and code related	Requests for writing, modifying, or debugging code, including scripts, extensions, or software structures.	10% (n = 11)	“Help me code a Chrome extension that will refresh a website and look for specific words in a certain number of second intervals.” (P_39_challenging)

The table reports task types, brief definitions, their prevalence in the sample, and illustrative user prompts.

To analyze task types, both authors independently read all 220 task descriptions and each generated their own list of task-type categories. The two authors then met to compare their category lists, discussed differences in naming and scope, resolved discrepancies, and merged similar categories until full agreement was reached. This negotiated-agreement approach (Campbell et al., 2013) produced the final taxonomies of five familiar-task types and seven unfamiliar-task types. Because most task descriptions mapped unambiguously to a single category (e.g. a coding request vs a recipe recommendation), disagreements arose primarily around category naming rather than fundamental assignment differences. We acknowledge that no separate inter-rater reliability coefficient was computed for the task-type classification; future work should include such a check to further validate the taxonomy.

For the remaining open-ended questions (motivations, fit reflections, and concerns), coding followed a four-round inductive process.

In Round 1 (exploratory code extraction), both authors independently read all open-ended responses and extracted in-vivo codes (i.e. codes using participants’ own words). This produced 328 motivation codes (156 for easy tasks + 172 for challenging tasks), 329 fit codes (120 + 209), and 205 concern codes (81 + 124), including repetitive and similar codes from different participants. Because this round was an exploratory extraction phase aimed at generating an initial code pool rather than applying a fixed scheme, formal inter-rater reliability was not computed at this stage. Codes were allowed to emerge from the data rather than from any pre-specified theoretical categories.

In Round 2 (codebook consolidation and split coding), the two authors met to compare their independently generated code lists, negotiated agreements on code names, clarified conceptual differences, and merged repetitive codes so that each code had a single, unique label. Using this consolidated codebook, each author then independently re-coded half of the responses (55 participants each). After coding, both authors cross-reviewed each other’s coded responses. For motivation codes, 36 required reconciliation (19 for easy tasks, 17 for challenging tasks); for fit codes, 32 required reconciliation (26 for easy tasks, 6 for challenging tasks); and for concern codes, 10 required reconciliation (three for easy tasks, seven for challenging tasks). All discrepancies were discussed and resolved in a follow-up meeting. Overall, the low proportion of disagreements (e.g. 36 of 220 for motivations, or 16.4%) suggests that the shared codebook produced consistent application across the split data.

In Round 3 (independent categorization), each author independently organized the finalized codes into higher-level categories for each question (motivations, fit, and concerns) and drafted category descriptions. When the two authors compared their independently developed category structures, they arrived at largely overlapping groupings—for example, both authors identified the same core categories for motivations (e.g. technology capabilities, task demands, individual preferences) and for concerns (e.g. accuracy, specificity, AI literacy), differing mainly in how they labeled or subdivided boundary cases.

In Round 4 (consensus finalization), the two authors met to reconcile any remaining differences in category names and boundaries, jointly refined category descriptions by reviewing original participant quotes, and conducted a final pass through all categories and their descriptions to ensure clarity and completeness. This iterative negotiated-agreement process (Campbell et al., 2013) produced the final coding scheme.

We conducted non-parametric statistical tests using the Wilcoxon signed-rank test for within-subjects comparisons of participants’ task-familiarity ratings and their evaluations of ChatGPT’s responses across the two tasks, given the potential order confound. For families of related tests (e.g. the three familiarity dimensions; the five RQ2 response-quality facets), we controlled the family-wise error rate using Holm-adjusted p -values and report these adjusted values where applicable. Effect sizes for our primary RQ2 analyses are matched-pairs rank-biserial correlations r_rb; for the preliminary familiarity checks, which function as manipulation checks rather than focal hypotheses, we report only test statistics and p -values. Because this is an exploratory survey study constrained by MTurk recruitment, we did not conduct an a priori power analysis; with 110 participants (220 tasks), the design offers good sensitivity to medium or larger within-subject effects but may be underpowered for subtle subgroup differences. We note that because each participant contributes two tasks with five ratings each, there is a within-person dependency structure that separate Wilcoxon signed-rank tests do not fully capture; while the paired design controls for between-participant differences within each dimension, it does not model the correlation across the five dimensions jointly. Future work with larger samples could employ multilevel ordinal models to account for both participant-level and item-level variability.

Participants and their experiences with ChatGPT

Among the 110 participants, 60% (n = 66) identified as male, 38% (n = 42) as female, and 2% (n = 2) did not report gender. Ages ranged from 24 to 70 years (mean ≈ 40 ; median 39). Education levels ranged from high school or equivalent to graduate degrees, with 47% (n = 52) holding a bachelor’s degree, and occupations spanned business/management/sales (18%), construction/manufacturing (15%), tech/IT (15%), and a variety of other sectors.

Most respondents (90%, n = 99) reported using the free version of ChatGPT for the tasks, 8% (n = 9) used the paid version, and 2% (n = 2) had previously held a paid account that had expired. Usage frequency varied: 81% (n = 90) used ChatGPT several times a week, 10% (n = 11) several times daily, and 8% (n = 9) less than once a month. Overall, participants’ self-reported familiarity with LLM tools ranged from 3.17 to 3.97 across four 5-point dimensions, with means of 3.41 for understanding, 3.48 for prompting skill, 3.97 for ethics awareness, and 3.17 for describing novel applications.

Task types and participants’ task familiarity

Among the 220 reported tasks, 34 (15.5%) were completed within the past 24 hours, 108 (49.1%) within the past week, 58 (26.4%) within the past month, and 20 (9.1%) more than a month earlier. Thus, about two-thirds of the tasks occurred within a week of the survey, suggesting that participants’ memories of their experiences were relatively fresh.

Our analysis (Tables 1 and 2) shows that the tasks participants identified as relatively easy were predominantly short, single-step requests that emphasized execution and light polishing for text. Among the five relatively easy task types, short creative projects were most common, followed by recommendations, factual lookup, text editing/proofreading, and quick explanations. By contrast, tasks participants identified as more challenging often required multiple steps and integration: research and information synthesis and creative writing and content creation were the two most prevalent types, followed by technical tutorials and troubleshooting, professional and business writing, personal advice and decision making, planning and organization, and programming and code-related tasks. Overall, the tasks participants identified as relatively easy tended to be short and loosely constrained, whereas the tasks they identified as more challenging more often combined multiple requirements and constraints, such as budgets, timelines, or technical artifacts, and were sometimes associated with broader or higher-stakes goals.

In terms of participants’ level of familiarity with their tasks, our quantitative analysis indicates statistical significance among all three dimensions. Table 3 shows the differences in users’ familiarity between the two tasks: all three comparisons were statistically significant between the familiar and unfamiliar task after Holm correction across the three tests: (1) topic familiarity (Z = 2.374, p_Holm = 0.018), (2) skill familiarity (Z = 3.140, p_Holm = 0.006), and (3) procedure familiarity (Z = 3.157, p_Holm = 0.004). Together, these scores indicate that the two participant-selected tasks differed significantly in self-reported topic, skill, and procedure familiarity. We use these ratings descriptively to characterize the easy/challenging contrast, rather than as evidence of an experimentally controlled familiarity or complexity manipulation.

OPEN IN VIEWER

Table 3.

Participants’ median self-reported topic, skill, and procedure familiarity for the relatively easy and more challenging tasks (Wilcoxon signed-rank test results, Holm-adjusted p -values).

measure	Median (familiar)	Median (unfamiliar)	positive Ranks (n)	negative Ranks (n)	Ties	Z	p Holm
Topic familiarity	5	4	33	15	50	2.374	0.018*
Skill familiarity	5	4	33	14	51	3.140	0.006*
Procedure familiarity	5	4	36	15	46	3.157	0.004*

Note. Measure refers to the familiarity dimension (topic, skill, or procedure). Each row reports a separate Wilcoxon signed-rank test for that dimension. Median (familiar) and Median (unfamiliar) are the sample medians of participants’ 1–5 self-ratings for the two types of tasks, respectively. For each participant and dimension, we computed the signed difference (familiar-task rating minus unfamiliar-task rating). Positive Ranks is the count of participants with a positive difference (familiar-task rating higher than unfamiliar-task rating); Negative Ranks is the count with a negative difference (unfamiliar-task rating higher); Ties is the count with zero difference. The test assigns ranks to the absolute values of the non-zero differences and compares the sums of ranks associated with positive versus negative differences. Z is the resulting standardized test statistic (two-sided). p_Holm is the Holm-adjusted p -value for that row’s test, controlling family-wise error across the three familiarity dimensions.

*

indicates p_Holm < 0.05.

Motivation: Technology, task, and individual characteristics (RQ1)

We identified 12 reasons motivating people to use ChatGPT (Table 4) and organized them into three groups based on whether they related to the technology, the task, or the individual user.

OPEN IN VIEWER

Table 4.

Reasons that motivated participants to use ChatGPT, grouped by whether they relate to the technology, the task, or the individual user.

Motivation	Description
Technology characteristics
Comparative advantages (general)	Perceived strengths/advantages of using ChatGPT for the task or over other tools (e.g., search engines).
Language & content production	ChatGPT’s advanced skills for written content; drafting, rewriting, or formatting.
Creativity & ideation	ChatGPT is a creative and generative tool that can support brainstorming, conceptual thinking, and inspiration.
Information quality	ChatGPT provides information that meets a reasonably high standard of quality.
Time and efficiency	ChatGPT could complete the task faster and reduce effort.
ChatGPT’s novelty	ChatGPT as a new kind of tool with capabilities users had not seen in other systems (e.g., conversational generation, flexible rewriting)
Task characteristics
Nature of task	The simplicity or challenge inherent in the task (e.g., task itself is straightforward or contains multi-step, technical, or in-depth processes).
Information availability	Whether the needed information for tasks is publicly available and searchable or scattered and requires synthesis and interpretation.
Specific task outcome needs	Task goals/outputs have specific requirements about what must be produced and how it should be formatted.
Individual characteristics
User preference/context	Personal preference to avoid a specific type of work, bypass a skill deficit, or the desire to focus attention elsewhere.
Previous experience with ChatGPT	Participants’ previous experience using ChatGPT as a reason in their decision to use it again
Curiosity/experimentation	Participant’s desire to experiment with ChatGPT for fun or to explore different solutions.

Perceived response quality of ChatGPT’s output (RQ2)

RQ2 examined how participants perceived ChatGPT’s response quality for each task. In the following analyses, “higher-familiarity” and “lower-familiarity” refer to participants’ self-reported familiarity ratings for the tasks they had selected as relatively easy and more challenging, respectively. These labels are used descriptively and should not be read as experimentally assigned task conditions. Participants rated responses on a 1–5 Likert scale across five dimensions: Specificity, Relevance, Accuracy, Understandability, and Creativity. Because ratings were paired within participants, we used two-sided Wilcoxon signed-rank tests, and p -values were Holm-adjusted across the five tests. Effect sizes are matched-pairs rank-biserial correlations r_rb, where negative values indicate higher scores for responses to the participant-selected relatively easy/higher-familiarity tasks. Because the Wilcoxon signed-rank test requires complete familiar-unfamiliar pairs, sample sizes for each aspect ( N = 98 − 100 ) reflect the number of participants with non-missing ratings for both tasks on that dimension (out of 110 participants and 220 tasks overall).

Across four aspects, participants rated ChatGPT’s responses to the relatively easy/higher-familiarity tasks higher than responses to the more challenging/lower-familiarity tasks: Specificity (N = 100, W = 192.0, p_Holm = 0.001, r_rb = −0.594), Relevance (N = 98, W = 57.5, p_Holm < 0.001, r_rb = −0.753), Accuracy (N = 100, W = 286.0, p_Holm < 0.001, r_rb = −0.585), and Understandability (N = 100, W = 64.5, p_Holm = 0.001, r_rb = −0.703). Creativity showed no reliable difference (N = 99, W = 754.0, p_Holm = 0.311, r_rb = −0.148).

Although we analyze the five evaluation dimensions (specificity, relevance, accuracy, understandability, and creativity) separately rather than as a composite score, we conducted a post-hoc psychometric check to examine whether these conceptually distinct facets showed reasonable coherence as a set of perceived-quality indicators. Using all tasks with complete data ( N = 206 ), Cronbach’s alpha for the five-item set was α = . 80 ; when computed separately by task type, alpha was α = . 72 for familiar tasks ( N = 107 ) and α = . 82 for unfamiliar tasks ( N = 99 ). Inter-item correlations were all positive: specificity, relevance, accuracy, and understandability were moderately to strongly correlated with one another ( r = . 44 – . 69 ), while creativity showed more modest but still positive associations with the other items ( r = . 29 – . 40 ). This pattern supports our decision to report the five dimensions separately: the first four facets form a coherent cluster of perceived output quality, while creativity taps a related but somewhat distinct aspect of the response.

Although medians were similar (reflecting ties common in Likert data), the signed-rank analyses reveal systematic within-participant shifts favoring responses to the relatively easy/higher-familiarity tasks on four of five aspects (Figure 1).

OPEN IN VIEWER

Figure 1.

Participant ratings for relatively easy/higher-familiarity versus more challenging/lower-familiarity tasks across five aspects: (a) Specificity, (b) Relevance, (c) Accuracy, (d) Understandability, and (e) Creativity.

Participants’ retrospective reasons for fit and misfit (RQ3)

RQ3 focuses on participants’ retrospective reasons for why ChatGPT was or was not a good fit for the information tasks they performed. From their open-ended explanations, we identified six categories of reasons participants felt ChatGPT worked well for their tasks (Table 5) and six categories of reasons they felt it did not (Table 6).

OPEN IN VIEWER

Table 5.

Participants’ reasons for perceiving ChatGPT as a good fit for their tasks.

Category	Description
UI/UX of ChatGPT	Refers to the ease, enjoyment, and intuitive interaction with ChatGPT, featuring a user-friendly and engaging interface with clear, accessible responses.
Response quality	Refers to ChatGPT as a reliable and knowledgeable source that provides relevant answers and, at times, creative responses.
Efficiency	Refers to ChatGPT as helpful for saving time, simplifying tasks, and streamlining workflows by providing fast and relevant content.
Affective experience	Refers to users feeling comfortable, supported, and understood during friendly, empathetic interactions with ChatGPT, where they can express themselves without judgment.
Guidance and support (unfamiliar)	Refers to ChatGPT providing clear guidance, relevant feedback, and consistent support, addressing entire prompts and reducing user stress with minimal input.
Engagement and learning (unfamiliar)	Refers to the way users engage with ChatGPT to generate ideas, gain insights, and receive solutions while finding the interaction informative and easy to learn from.

OPEN IN VIEWER

Table 6.

Participants’ concerns and challenges when using ChatGPT for their tasks.

Category	Description
Lack of accuracy and reliability	ChatGPT provides incorrect or made-up information, ignores rules, or loses track of references, causing users to question its accuracy and trustworthiness.
Lack of specificity and originality	ChatGPT’s responses fail to add real value because they lack concrete details or examples and are generic, repetitive, or difficult to follow.
Privacy and biases	Users are concerned about the handling of personal information and potential bias in ChatGPT’s suggestions, for example when it appears to favor paid over free options.
ChatGPT’s limitations of understanding (unfamiliar)	ChatGPT struggles with understanding nuanced or complex requests and lacks the adaptive and interactive qualities found in human communication.
Task Preference Misalignment (unfamiliar)	ChatGPT’s output is burdensome and the style doesn’t fit the user; the guidance misses the user’s level, and the user must rewrite, trim, or complete key parts themselves.
User AI literacy (unfamiliar)	Users found it challenging to craft effective prompts, guide ChatGPT, and ensure accurate responses, including difficulties with prompt specificity, communication, and understanding what the model needs for clarity.

Among the six categories of why ChatGPT fit, UI/UX was the most frequently mentioned reason why participants liked ChatGPT. They described it as user-friendly, has a straightforward format, and has “a modern layout/user interface” (P_105_challenging). One participant commented: “I liked how it was smooth and not complicated. I always want to have these types of interactions when using technology” (P_26_challenging).

In the Response Quality category, specificity, accuracy, and creativity stood out as valuable to participants. One participant noted, “I like the specific information. . .the details” (P_40_challenging), another participant said, “ChatGPT was very creative. I got very different recipes from the two slightly different prompts I gave it” (P_90_easy).

In terms of efficiency, participants focused on ChatGPT itself being fast, and the interaction being low-effort for different tasks. For example, one participant said, “I can just talk to the AI very casual like. My questions don’t have to be very specific to get the answers I want. It always seems to understand exactly what I’m trying to ask it.” (P_7_easy).

Participants also commented on the affective experience of ChatGPT for smoothing their task process. One participant wrote, “I liked that I could talk to someone. I liked that the chat did not judge me” (P_42_challenging). Another participant said, “I liked feeling like someone cared about my finances. I liked that it gave me some good advice and told me about some apps that would help me save money on groceries.” (P_77_challenging).

Two categories emerged specifically in relation to what participants liked about ChatGPT for the unfamiliar task: Guidance and Support, and User Engagement & Learning. Participants reported liking ChatGPT for the unfamiliar task because they felt that the tool broke down processes clearly. One participant said, “It doesn’t just simply give you the answers. Math problems are complicated and it is nice to see it walk through it.” (P_5_challenging). Another reported, “It was very easy to prompt ChatGPT for more information. I could easily apply new prompts to the situation and get more specific information after looking at what it already provided me with which made the process fun and engaging” (P_12_challenging). These responses suggest that, in the more challenging/lower-familiarity tasks, participants valued ChatGPT’s ability to provide structured guidance, break down multi-step processes, and support iterative exploration of the problem space.

On the contrary, participants reported a range of concerns and challenges that arose during or after using ChatGPT, particularly for their unfamiliar tasks. These concerns, drawn from participants’ retrospective reflections and their reports of challenges encountered during the task, fell into six categories (Table 6). Among the six categories, lack of accuracy and reliability is a frequently mentioned reason why participants felt ChatGPT did not work well for their tasks. They described that the responses it provided were not always accurate, and hallucinated information at times, or ignored task rules. A participant wrote, “I like the specific information, although it was wrong. I also liked the details given, again it was inaccurate.” (P_40_challenging), and another person wrote, “I have no way of knowing if it is correct without spending an hour in search engines. Which is what I was attempting to avoid.” (P_65_easy).

Participants were concerned that responses were repetitive and lacked originality. For example, a participant wrote “It gave me the same solutions I’ve already tried to no avail.” (P_36_challenging). A participant explained, “It just isn’t very good at going beyond the obvious, the cliched. I wanted a fresh angle and it failed.” (P_15_challenging).

Several participants mentioned their concerns related to personal data leaks and biased information. For instance, one participant said: “I was also concerned about providing personal information and how that would be used or shared when developing the final summary.” (P_75_challenging). Another participant described: “I was also concerned that it would provide more paid services than free services, giving me little to no options on how to solve the issue at hand.” (P_56_challenging).

Participants also felt that ChatGPT lacked the natural tone of a human, and it did not measure up to human creativity and nuance in its responses, especially for their unfamiliar tasks. One participant wrote, “AI did not understand me and I ended up seeking human help and perspective to resolve my problem” (P_2_challenging). Other participants also expressed their preference for human help for the unfamiliar task, noting the limitations ChatGPT has in understanding creative, subjective, or expertise-oriented tasks like law or medicine.

In some cases, participants expressed conviction that ChatGPT is fundamentally ill-suited to the demands and preferences of their unfamiliar tasks. They felt that ChatGPT is incompatible with their workflow, standards, or goals, thereby justifying partial or complete rejection of the tool. For instance, one participant wrote: “ChatGPT suggested solutions that just would not fit well within the space without causing difficulties.” (P_12_challenging). Another participant said: “it just refused to answer and was useless.” (P_11_challenging). Another person mentioned, “the code never worked. I kept asking for a different code and every time it failed.” (P_84_challenging).

Participants also reported uncertainty about the best way to frame their tasks as prompts when using ChatGPT for unfamiliar tasks. There was a learning curve for participants as they realized they needed to structure prompts to get ChatGPT to respond accurately. Comments are made like: “I struggled to ask the question. . .ended up having to excise all unrelated code.” (P_57_challenging). And another person said, “I was concerned if I did not prompt the system well enough that there may have been inaccurate details.” (P_109_challenging).

Reasons motivating the use of ChatGPT for information tasks (RQ1)

RQ1 examined what motivates people to use ChatGPT for information tasks. Motivations clustered around three dimensions: technology capabilities, task demands, and individual preferences.

Technology-related motivations were prominent across both task types. Participants valued ChatGPT’s fluent language generation, rephrasing capabilities, ideation support, and time savings. For familiar tasks, these capabilities were often the primary reason for choosing ChatGPT over other tools; for unfamiliar tasks, participants relied on ChatGPT to obtain starting points, structure problems, and reduce effort. These patterns are consistent with studies showing that when LLM tools’ capabilities match task activities, the intention to use increases (Al-Mamary et al., 2024).

Task-related motivations reflected the contrast between the two task types. Participants turned to ChatGPT when a task was either straightforward but tedious or perceived as demanding because it required synthesizing information from many sources, producing text in specific formats, or dealing with loosely defined needs. Participants appear to recruit ChatGPT where they expect a match between task properties and the system’s strengths: rapid retrieval and micro-editing for more bounded tasks versus structured guidance and synthesis for tasks they experienced as more demanding or less familiar (Oliński and Sieciński, 2025).

Individual-level motivations included compensating for skill gaps in writing, coding, or language; prior positive experience with ChatGPT; and curiosity. Several participants described shifting from content creators to evaluators and editors of AI-generated text. Overall, motivations reflect perceived alignment between what the technology can do, what the task requires, and the user’s own skills and preferences, rather than undifferentiated enthusiasm for generative AI.

Perceived response quality of ChatGPT’s output (RQ2)

RQ2 examined how participants perceived ChatGPT’s response quality across familiar and unfamiliar tasks, as reflected in their ratings on specificity, relevance, accuracy, understandability, and creativity. Overall, ratings were positive for both task types, with median scores of 4 or 5 and means above 3.5 across all five dimensions. Within-participant comparisons, however, revealed a clear pattern: familiar-task responses were rated significantly higher than unfamiliar-task responses on specificity, relevance, accuracy, and understandability, while creativity did not differ reliably between the two. Thus, even when overall ratings were high, participants reported lower response quality on four key dimensions for the tasks they had identified as more challenging and rated as less familiar.

The qualitative analysis (RQ3) clarifies why. For familiar tasks, participants praised ChatGPT’s simplicity, clarity, comprehensiveness, and well-organized answers, and mainly criticized outputs that were too broad, long, or repetitive. For unfamiliar tasks, they reported that responses were less useful when ChatGPT did not fully address the query, suggested impractical solutions, or required substantial prompt refinement; they also noted that answers sometimes lacked specificity, originality, or accuracy for complex problems. Taken together, these patterns are consistent with a descriptive picture in which users rely on ChatGPT to generate content while allocating some of their own effort to evaluating and verifying its outputs, especially in tasks they experienced as more challenging or less familiar; however, our retrospective self-report data do not allow us to make strong claims about underlying cognitive processes or objective task complexity. In line with this descriptive pattern, participants’ expectations and critiques appeared sharper in the more challenging/lower-familiarity tasks, and they explicitly expressed dissatisfaction with incomplete responses on these tasks. This contrasts with Suri et al. (2024), who report higher satisfaction for more complex tasks even with partial responses; one plausible reason is that their study relies on LLM-based evaluations of conversation histories, whereas ours uses human participants’ ratings of ChatGPT’s response quality on their own familiar and unfamiliar tasks, combined with qualitative reflections. By focusing on human judgments of response quality in a within-participant design, our study offers a distinct, task-sensitive view of how people assess ChatGPT across different kinds of information tasks.

Participants’ retrospective reasons for ChatGPT-task fit and concerns (RQ3)

RQ3 examined participants’ retrospective accounts of why ChatGPT was or was not a good fit, and the concerns they encountered. Six reasons for perceived fit and six categories of concerns emerged (see Tables 5 and 6). A clear pattern distinguished the two task types: for familiar tasks, participants were largely satisfied and reported few concerns; for unfamiliar tasks, they relied more heavily on ChatGPT for guidance, and this reliance amplified dissatisfaction when responses were inaccurate, vague, or difficult to follow. While concerns such as hallucinations, unreliable outputs, and limited contextual understanding have been documented as general issues with GenAI tools (Carlini et al., 2021; Ji et al., 2023), our findings suggest that the severity and nature of these concerns differed across the participant-selected tasks and appeared to be related to users’ self-reported familiarity, perceived task demands, and specific needs. This suggests that future LLM tools should take task context, users’ self-reported familiarity, and perceived support needs into account when calibrating response strategies and verification support. Participants sometimes described ChatGPT as a “supportive scaffold” that provided guidance and reduced cognitive strain, while their own effort shifted toward interpreting, adjusting, and verifying AI-generated content. This pattern suggests that LLM tools may redistribute cognitive effort during information tasks: whereas traditional search requires users to locate, evaluate, and synthesize individual sources, ChatGPT-assisted work appears to shift the focus toward evaluating and refining consolidated outputs. Future research could investigate how this reallocation of cognitive effort varies across task types and user expertise levels.

Rather than operationalizing perceived fit as a numeric construct, our study asks participants to describe in their own words why ChatGPT was experienced as a good or poor fit for real-world information tasks. This approach surfaces concrete reasons for perceived fit and concerns about perceived limitations, including UI/UX, response properties, efficiency, affective experience, guidance, reliability issues, and AI literacy challenges, which shape users’ sense of fitness across different tasks.

Implications

Our findings have implications for both theory and the design of LLM tools. Theoretically, they provide a descriptive picture of how task demands, technology perceptions, and individual factors co-occur in everyday ChatGPT use and how users narrate perceived “fit” in their own terms. By examining participant-selected easy and challenging tasks alongside self-reported topic, skill, and procedure familiarity, we connect our descriptive findings to classic notions of task complexity in interactive information retrieval (Byström and Järvelin, 1995; Choi et al., 2020; White, 2016; Wildemuth et al., 2014) and suggest that users’ experiences of task difficulty in AI-supported work may be shaped not only by task properties and user knowledge, but also by evolving expectations of what the system can and cannot do. Notably, the three groups of motivations we identified (technology-related, task-related, and individual-related) and participants’ retrospective accounts of fit and concerns suggest that the Task–Technology Fit (TTF) framework (Goodhue and Thompson, 1995) could serve as a productive lens for future confirmatory research. Studies could operationalize TTF constructs using validated scales, employ counterbalanced experimental designs with standardized tasks, and test causal pathways from task and technology characteristics to perceived fit and outcomes across different LLM tools and task contexts.

On the design side, the results argue against a one-size-fits-all interaction style. For familiar tasks such as factual lookups or light editing, participants preferred concise, direct answers without unnecessary elaboration. For unfamiliar tasks, however, participants wanted step-by-step explanations, clarification of assumptions, and structured guidance. For example, participants troubleshooting technical systems valued responses that broke down the process into manageable steps, and those writing business plans wanted ChatGPT to surface relevant constraints (e.g. budgets, timelines) rather than produce generic templates. This points to the need for adaptive response strategies and user-controllable interaction modes (e.g. efficient vs explanatory) that can be adjusted according to users’ goals, self-reported familiarity, and the level of guidance they need.

Finally, participants’ concerns about hallucinations, incomplete answers, and opaque reasoning underscore the importance of transparency and verification support, especially for unfamiliar or consequential tasks. Features such as lightweight uncertainty cues, optional source attribution, short explanations of reasoning for key steps, and contextual tips about when and how to double-check outputs could help users calibrate their reliance on LLM tools to the demands of the task and better support their emerging role as evaluators and editors of AI-generated content.