ARTIFICIAL INTELLIGENCE (AI) – PROLONGED USE LIKELY TO REDUCE CREATIVITY AND IMPAIR COGNITION

Figure 5 a shows the trend in the individual-level diversity of ideas produced by participants. In the Exposure phase, the Kruskal-Wallis H test revealed a significant difference between conditions (H(2) = 9.95, p = 0.007). Post-hoc pairwise comparisons indicated that participants who received the List of Strategies produced significantly more similar ideas within each round compared to those in the List of Ideas condition (p = 0.009) and the No LLM Response condition (p = 0.041). In contrast, there was no significant difference between the List of Ideas and No LLM Response conditions (p = 1.000). The Kruskal-Wallis test for the Test phase also showed a significant difference between conditions (H(2) = 7.71, p = 0.021). Participants in the List of Strategies condition continued to generate more similar ideas compared to those in the No LLM Response condition (p = 0.017), although no significant difference was found between the List of Strategies and List of Ideas conditions (p = 0.348), or between No LLM Response and List of Ideas (p = 0.747).

Interestingly, while both LLM-assisted conditions showed a decline in idea diversity across rounds, participants in the No LLM Response condition maintained or even slightly improved their diversity. This result is somewhat unexpected, as it might have been assumed that unassisted participants would experience fatigue, leading to less varied ideas over time, whereas those who had support during the exposure rounds would be better equipped to produce diverse ideas.

We simulated group-level diversity by randomly sampling ideas across participants based on their condition, item, and phase. Figure 5 b illustrates the trends in group-level diversity. In the Exposure rounds, the average median cosine distance between ideas was similar across all conditions (p > 0.05), which is unexpected. Participants in the List of Ideas condition, who had access to the same LLM-generated ideas, were expected to produce more similar ideas as a group. However, this may be due to participants not fully adopting the AI suggestions, as indicated by earlier findings. Additionally, the random sampling of 7 ideas from a pool of 20 for each object may have contributed to the lack of overlap.

Interestingly, participants exposed to LLM-generated strategies during the exposure phase continued to generate more similar ideas in the test phase, even without LLM assistance, a phenomenon known as homogenization. This raises concerns that LLMs, which provide ubiquitous frameworks for thinking, could lead to reduced diversity in collective thinking, with people continuing to produce similar ideas even after they stop using the LLM.

Figure 6 summarizes participants’ self-reported measures of creativity, feelings toward AI use, and the difficulty of generating uses for the Test object. Across all conditions, participants reported a decline in their perceived creativity after the experiment, a consistent drop that is somewhat unexpected. One might assume that receiving a list of AI-generated ideas would either further diminish creativity, as participants might feel they cannot match the AI, or conversely, seeing strategies during exposure rounds could boost their self-assessed creativity by providing structured guidance.

The change in participants’ feelings toward AI remained relatively stable across conditions, with those in the List of Ideas condition reporting more than double the magnitude of change compared to the other groups. Interestingly, participants who received a list of ideas during the exposure rounds also found it easier to generate their own ideas during the test phase. This is surprising, as one might expect that exposure to AI-generated ideas would make it harder for participants to think independently in the test round, and that AI strategies in exposure would make it easier to independently come up with ideas.

As shown in Figure 9, participants in the LLM Answer condition did better in the Exposure rounds in comparison to the other conditions. However, this trend changed in the Test rounds. Following our pre-registered analysis plan, the ANCOVA revealed a significant effect of verbal fluency on performance (F(1, 1256) = 16.715, p < 0.001) as well as a significant main effect of condition (F(2, 1256) = 5.134, p = 0.006). Post-hoc comparisons using Tukey’s HSD test indicated that participants in the LLM Guidance condition performed significantly worse than those who did not receive any LLM responses in exposure rounds (p = 0.005), with a mean difference of − 10.6 [95% CI: − 18.6, −2.7]. While the LLM Answer condition did not differ significantly from the No LLM Assistance condition (p = 0.682), there was a trend suggesting that the LLM Guidance condition performed worse than the LLM Answer condition (p = 0.064).

This suggests that, although the difference between the LLM Answer and LLM Guidance conditions did not reach statistical significance levels, the LLM Answer condition may lie somewhere in between the other two conditions. However, this observation should be interpreted cautiously, given the lack of statistical significance in this comparison.

The positive performance of participants in the LLM Answer condition during exposure rounds underscores the effectiveness of LLMs in providing accurate solutions to the task. This suggests that LLMs are indeed proficient at generating correct answers and that participants are capable of recognizing when to leverage LLM advice effectively. Exposure to LLM assistance—whether in the form of direct answers or strategic guidance—did not translate into enhanced unaided performance and may have even been counterproductive. This phenomenon can be understood through the lens of convergent thinking, which often relies on achieving an ’aha’ moment or insight. LLM Guidance might have disrupted this process by providing too much additional information to take in, thereby hindering the natural cognitive processes necessary for independent problem-solving. Similarly, LLM Answer could have impeded participants’ engagement in their own creative thinking, as they might have become overly reliant on external solutions rather than developing their own problem-solving strategies.

Figure 10 shows participants’ change in self-reported levels of creativity, change in feelings towards AI use perceived difficulty of test rounds, and perceived helpfulness of exposure rounds. In plot (a), we see that across all experimental conditions, participants exhibited a reduction in self-reported creativity ratings from pre-experiment to post-experiment surveys. Both LLM Answer and LLM Guidance conditions saw a greater average reduction in creativity ratings, with LLM Guidance showing an average reduction more than twice as substantial as that observed in the No LLM Assistance condition. This suggests that reliance on LLMs, particularly in the LLM Guidance condition, may have undermined participants’ confidence in their own creative abilities. The more substantial reduction in creativity ratings for LLM Guidance could indicate that receiving a wide range of possible word connections, rather than necessarily coming up with their own, diminished participants’ sense of ownership over their creative process. Futrthermore, the less significant decrease observed in the LLM Answer condition could be from participants recognizing they were receiving direct answers which were completely detached from their own creative abilities. Plot (b) shows that the change in attitudes toward AI use in daily life stayed roughly constant between pre-experiment to post-experiment surveys. with LLM Answer seeing slight attitude increase. Feelings towards AI use was measured using a 3-point Likert scale (with values being “More concerned than excited”, “Equally excited and concerned” “More excited than concerned”). This slight increase in positive attitudes toward AI in the LLM Answer condition could be attributed to participants recognizing the effectiveness of LLMs in completing tasks accurately, reinforcing their confidence in AI’s capabilities. The reliable performance of the LLM may have led to a more favorable view of its potential for everyday use. In plot (c), perceived difficulty of the test rounds was highest for the LLM Guidance condition, and slightly lower in the No LLM Assistance and LLM Answer conditions. That is to say, test rounds felt more difficult when the participant had completed previous tasks with guiding AI assistance, rather than just a straight answer, or no assistance at all. On average, perceived difficulty was slightly above 2.5 on a 4-point Likert scale (ranging from “very easy” to “very difficult”). meaning that participants found test rounds a little on the difficult side. The higher perceived difficulty in the LLM Guidance condition may be due to participants relying on the given word associations during the exposure rounds, which might have made it harder for them to independently generate associations in the test rounds where no assistance was given. Finally, plot (d) measured the percentage of participants who found the exposure helpful for their completion of the test rounds. Note that this was not explicitly measuring whether participants found the LLM assistance helpful, but rather whether completing the exposure rounds aided them in performing the test rounds. Participants who received no LLM assistance during the exposure rounds were the most likely to find these rounds helpful. In comparison, the proportion of participants who found the exposure rounds helpful was roughly 5% lower in the LLM Answer condition and 7% lower in the LLM Guidance condition. This result may suggest that participants without LLM assistance had to rely more on their own problem-solving strategies during the exposure rounds, which could have enhanced their perceived usefulness of these rounds in preparing for the test phase. In contrast, those receiving LLM assistance might have become more dependent on the AI, leading to a lower perception of the exposure rounds’ value for their independent performance during the test rounds.

Across both experiments, we observed that LLM assistance was helpful during the exposure phase, aligning with previous research that highlights LLMs’ ability to enhance creative task performance when users have access to AI-generated ideas or strategies [2, 14]. However, in the test phase, participants who had no prior exposure to LLMs consistently performed better (this was not statistically significant in all cases). In the divergent thinking task, participants who worked without LLM assistance generated more original ideas on average, and in the convergent thinking task, those without LLM exposure were better able to identify the correct connecting word compared to those who had LLM exposure. These findings suggest that while LLMs may provide short-term boosts in creativity during assisted tasks, they might inadvertently hinder independent creative performance when users are asked to perform without assistance. This raises important questions about the long-term impact of repeated LLM use on human creativity and cognition.

From a design perspective, it is critical to consider not just human-AI performance during exposure phases but also human performance in unassisted tasks after using AI. Systems should be designed with long-term human flourishing in mind, ensuring that the benefits of AI assistance do not come at the cost of diminished independent creative abilities. Ensuring that users can effectively transition from AI-supported creativity to autonomous creative work will help mitigate potential long-term harms associated with over-reliance on LLMs, as highlighted by concerns in the literature about cognitive decline with repeated AI use [27].

Our experiments reveal that the effects of LLMs vary significantly depending on the aspect of creativity being measured. In divergent thinking tasks, where participants were asked to generate a wide range of ideas, we observed more skepticism toward LLM assistance. Participants seemed less inclined to adopt AI-generated suggestions, which may be due to the nature of divergent thinking itself—encouraging exploration and unconventional approaches [4, 25, 51, 53]. In contrast, for the convergent thinking task, where participants were tasked with narrowing down the ideas to a single solution, LLM assistance during the exposure phase appeared more beneficial. This is consistent with the idea that convergent thinking is more structured and goal-oriented, making it easier for participants to recognize when LLMs are effectively guiding them toward the correct solution [17, 42, 56].

LLMs, however, may also hinder the creative process. In divergent thinking, the introduction of AI-generated ideas may distract participants, consume valuable cognitive resources, or prevent full engagement with the task. This aligns with theories of creativity such as Boden’s concept of conceptual spaces, which emphasize the importance of users exploring and navigating creative possibilities independently [9]. Over-reliance on LLMs during divergent thinking may disrupt this exploration, leading to less original ideas, as seen in the originality results (Figure 4). Similarly, in convergent thinking tasks, LLMs might steer participants toward specific solutions too quickly, reducing the need for deep engagement with the problem space [25].

These findings underscore the importance of carefully calibrating trust and reliance in human-LLM systems. Designers should incorporate measures that help users recognize when to trust and rely on LLMs, and when to prioritize their own cognitive processes. This approach can help balance the benefits of AI support with the risks of undermining the human creative process, ensuring that LLMs enhance rather than detract from creativity in the long term.

Existing research has shown that LLM usage can lead to the homogenization of ideas within groups, where participants tend to converge on similar outcomes when using AI-generated suggestions [2, 19, 45]. Our findings extend this concern, showing that even when people stop using LLMs that provide strategic frameworks for thinking, the homogenization effect can persist. In our divergent thinking experiment, participants who received LLM-generated strategies exhibited reduced diversity in their idea sets both during and after LLM use, suggesting that such frameworks may have a lasting impact on creative processes, potentially stifling the generation of more varied or unconventional ideas.

Interestingly, we did not find the same effect for participants who received direct ideas from a standard LLM. The List of Ideas condition did not lead to the same lasting homogenization once the LLM was no longer present. This indicates that while LLM-generated strategies can have long-term effects on creative diversity, providing ideas without a guiding framework may allow for more cognitive flexibility once participants stop using AI. Further, participants with no LLM interactions in the study collectively generated the most diverse idea sets (Figure 5). In our convergent thinking experiment, participants who received LLM guidance during exposure performed worse in the test round compared to those who received direct answers. This highlights the complexity of designing LLMs as coaches or guides. While direct answers may not always seem ideal, in some cases, they can be more effective than offering frameworks or strategies. Designers of Human-AI systems must take these findings into account, ensuring that LLM interactions are structured to avoid unintentional long-term effects on cognitive diversity. Controlled experiments, such as those conducted here, can be useful methods in refining and optimizing the design of coach-like LLM systems [35, 57].