New working paper on how experience plays a big role in how researchers take on research ideas from generative AI
The years of experience that researchers have is a major factor in how they take on board new research ideas from generative AI
My colleagues, Sen Chai (at McGill University) and Matthias Tröbinger (at ESSEC), and I have posted a new working paper (read: not yet published in a peer-reviewed journal) titled, “Generative AI in Scientific Ideation: How Research Experience Moderates Perception and Integration.” We have seen a lot of academic and industry research that considers how generative AI helps in various knowledge tasks. We wanted to push that question to research, which is where some of the boundaries of human knowledge are being pushed outwards. If generative AI is going to help push knowledge at the frontier, we reasoned that it was important to assess how researchers might respond to getting those ideas, because (at least for now), those new ideas are ultimately coming from those researchers.
The experiment
The design of the experiment took inspiration from my prior paper with Oliver Hauser (at the University of Exeter) on how generative AI helps creativity. We recruit over 300 academic researchers from over 30 disciplines, including medicine, business, history, physics, law, and literary studies. Each researcher was asked to submit the title and abstract of a recently published paper. We randomly assigned some researchers to receive either one or three ideas produced by generative AI based on their prior work. Then we asked each person to write a new research proposal.
After writing their research proposal, we asked them to evaluate the proposal. We asked a number of questions to capture the following dimensions of their perceptions about their submitted proposal about their research agenda overall:
Novelty. How new and original did they think their proposal was?
Feasibility. How likely is it that the proposal could be a full-blown study?
Impact. If the proposal was developed, how much impact would it have?
Research agenda. How excited and energized are they about their overall research agenda?
Attitudes. What is their emotional state?
The results
We expected generative AI ideas to have an effect on how researchers perceived their own proposals’ novelty and feasibility, but we did not see any effect in our data.
What we did find, though, is that how experienced the researcher is has a big impact on the effect of generative AI ideas. First, less experienced researchers tended to incorporate the ideas from AI much more so than did more experienced researchers.
Second, more experienced researchers who got AI ideas found the exercise of coming up with research proposals in our study to be much less enjoyable.
Among researchers with less experience: researchers who received AI ideas found their own research proposals as being more novel and impactful, and they were more positive about their research agenda. The result among more experienced researchers was the opposite: those who received AI ideas thought of their own research proposals were less novel, less impactful, and they less enthusiastic about their research agenda.
Why the difference?
Based on comments provided by the participants, we found some explanations for these dramatic differences. Less experienced researchers considered generative AI as:
Providing them with new research directions
Validating their own ideas
When provided with generative AI ideas, more experienced researchers disproportionately:
Discounted ideas as coming from an outsider
Demonstrated hesitancy toward new technology
Perceived the technology as a challenge to their identity
How this matters for organizations?
As organizations continue their efforts to deploy the technology, it is important to understand how different employees (in different roles) respond differently to it. Some may not accept or test its capabilities, feeling more comfortable or skilled in existing processes. I could imagine a hypothetical response among some people might be: “That is not the way we do things around here.” Employees like this may need to adjust how they view their identity and their role in performing different tasks. Change in the organization to adapt to AI will also require change in people.
For others, the opportunity to kickstart new directions and new lines of thinking might prove to be a boon to productivity and innovation. Here the caution is whether there is any cost to this boost. Specifically, does a less experienced person forgo some investment in themselves and their own generativeness and creativity if they lean on AI ideas to steer their thinking?
Ultimately, the question of how generative AI gets integrated into research and business and government and other parts of our economy are going to depend as much on the people using it as it will on the technology. Our research illustrates this idea in a very stark fashion.