US’ #NIH Australia’s #ARC : 'NO’ to ChatGPT for peer-review
US' #NSF and Europe's #ERC mulling it over with working groups.
Concerns:
*Privacy/Piracy “the information becomes part of its training data. ” (why I don’t chatGPT though I really hate writing)
*Error ”AI-written reviews will be error-prone"
*Bias ”against non-mainstream views”
*Boring "lack … creativity “
@atthenius Oh lord it hadn't even occurred to me science organizations might be considering this, I guess I can't fault them for considering the question but thank goodness the answer is NO. For two of them at least -_-
@Lennvor In the article its more like someone asked to do peer review for a proposal. Depending on your familiarity with the topic, that can take half a day or so… but someone noted with ChatGPT, they could get it done quick (my estimate, half an HOUR).
Peer review for proposals is generally compensated, but peer review of research papers is NOT (its community service, you review me, I review you for free so the publisher can cash in our labor).
There is thus an incentive to expedite reviews.
@atthenius What does "Peer review for a proposal" mean though ? Summarizing the proposal ? Reading it and giving your estimation of whether it's good enough, or follows guidelines, or some other required metric ? Writing a document that extracts and highlights some specific bits of information from the proposal ? Something else ?
For most of those cases I'm sure you can use ChatGPT to generate a document that makes it look like this was done but I wouldn't call it "getting it done".
Most proposals go through ‘panel review’ where ALL the proposals submitted to a call are reviewed by ~30 of their peers. Usually, there is one primary & 2-3 tertiary reviewers who evaluate the proposal before the panel review. During the review, each primary reviewer presents opinions on study feasibility, proposers qualifications, potential significance. Then the tertiary reviewers chime in.
At the end the whole panel grades each proposal & only the ‘best’ (<10%) get funded.
@atthenius That sure sounds to me like something you can fake, but not do, with ChatGPT. Looking at the article it seems it was about individual participants replacing their contributions, not the whole process being replaced. The latter would be dumb but the former strikes me as straight-up fraud. The reason they're asking scientists to do this review and not random people off the street capable of forming a sentence is that they're relying on the scientist's opinion based on their expertise - neither of which ChatGPT has, and if it did it still wouldn't be that scientist's.
