4 Comments
User's avatar
Nathan Lambert's avatar

Nice post. I’d like to see more thought on what modern scientific institutions look like given the new pace of science. Working on a piece thinking about this, will reference this.

Expand full comment
Nathan Lambert's avatar

IMO AI oversight in peer review is inevitable.

Expand full comment
Stephen Malina's avatar

Looking forward to your piece on this! I agree AI oversight in peer review is inevitable. Thinking out loud, it feels like there's two components that right now are conflated in human reviews: "correctness/validity checks" and "taste" (interestingness/novelty/etc.). It feels like the first part is lower hanging fruit right now and an area I'd love to see more work in. The initial tests (https://www.owlposting.com/p/can-o1-preview-find-major-mistakes, https://amistrongeryet.substack.com/p/the-black-spatula-project) using o1 to find mistakes are a good start but would love to see an industrial scale effort here to add all the bells and whistles (citation validity checks, coding environments for repros, etc.).

Expand full comment
Nathan Lambert's avatar

I have a long reading list at this point but plugging away

Expand full comment