New Paper in Science Advances

Paper co-authored with Oliver Hauser titled "Generative AI enhances individual creativity but reduces the collective diversity of novel content"

Source: ChatGPT 4o

Oliver Hauser and I have published our paper titled Generative AI enhances individual creativity but reduces the collective diversity of novel content in Science Advances. In January 2023—the early weeks of ChatGPT—Oliver and I spoke about launching a research project around generative AI. We played around with a few topics and we settled on looking at how generative AI might help a core human activity—creativity. It was important for us to focus on how generative AI might help human output. We were not interested in running a horse race between humans and AI. So, in our research project we asked: how can generative AI help humans produce more creative work?

The experiment

Source: Doshi and Hauser (2024)

With that idea in mind, we set out to conduct an experimental study. We asked about 300 writers to write an 8-sentence fictional story (about, say, an adventure on another planet) for a young adult audience. We randomly assign writers to one of three conditions: (1) a human only condition where writers received no AI assistance; (2) a 1-AI idea condition where writers could receive a single 3-sentence idea; and (3) a 5-AI idea condition where writers could get up to 5 ideas.

Then we passed on those stories to a separate set of 600 evaluators, who provided their assessments on stories without knowing whether writers were offered AI assistance. We asked the evaluators to asses the story on two dimensions of creativity and on other emotional and quality characteristics. We also used natural language processing techniques on the stories to arrive at an object measure for how similar stories were to one another.

The results

We find that getting AI ideas helps humans write more creative stories, in terms of two measures, novelty and usefulness. We understand that it is difficult to quantify the notion of creativity, but these two dimensions capture different (and helpful) aspects of creativity. The first dimension, novelty, captures how different, or original, a story is from others. The second dimension, usefulness, captures how "fit to task" a story is. In other words, could the story be published and be of interest to a young adult audience?

We also find that stories that receive 5 AI ideas are evaluated as better written, more enjoyable, having more of a plot twist, and less boring. Those stories are also more likely to change what an evaluator expects from future stories they read.

The increase in creativity that we captured was not even across all the writers. It was most pronounced for writers who were the least inherently creative in our sample. In other words, writers who were not inherently creative received a big boost in how creative their stories were. In fact, when getting 5 AI ideas, the boost made their stories level to the most inherently creative writers in our sample. We did not see a boost for those writers who were among the most inherently creative, though. So, we did not see AI could "supercharge" that the most creative (yet).

Despite these improvements in creativity for individual stories, we did find an important downside from using generative AI. Stories that received generative AI ideas looked more like another than did stories who did not receive any AI assistance. So, even though stories by people (without AI) were less creative, according to the independent evaluators, they were more diverse. And, as a society, we might want to encourage that diversity. We want these "outliers" in literature, and in art, and in innovation of new ideas and technologies. So this compression of stories that received AI assistance was a source of concern for us.

The paper has a number of additional results, such as evaluator attitudes toward AI and how much they believe “ownership” should be discounted if an author uses AI.

The takeaway

So, the main punchline of the paper is that the use of generative AI to produce ideas for a creative task leads to a social delimma: individuals have all the reason in the world to use generative AI to write stories that will be percevied as being more creative, but as a society, we might lose out on the outlier ideas, because all the stories look increasingly like one another.

If you would like to hear about the article, we spoke with the Naked Scientists and NPR. Both conversations are very interesting and accessible to understand the general points of the study. If you prefer to read about the study, please check out fantastic write ups at The Guardian, MIT Technology Review, or TechCrunch.

Did you enjoy reading this article? Please subscribe if you found it informative. More to come about generative AI, research, and how generative AI may play a role in organizations and in society…