Nice post. I’d like to see more thought on what modern scientific institutions look like given the new pace of science. Working on a piece thinking about this, will reference this.
Looking forward to your piece on this! I agree AI oversight in peer review is inevitable. Thinking out loud, it feels like there's two components that right now are conflated in human reviews: "correctness/validity checks" and "taste" (interestingness/novelty/etc.). It feels like the first part is lower hanging fruit right now and an area I'd love to see more work in. The initial tests (https://www.owlposting.com/p/can-o1-preview-find-major-mistakes, https://amistrongeryet.substack.com/p/the-black-spatula-project) using o1 to find mistakes are a good start but would love to see an industrial scale effort here to add all the bells and whistles (citation validity checks, coding environments for repros, etc.).
Nice post. I’d like to see more thought on what modern scientific institutions look like given the new pace of science. Working on a piece thinking about this, will reference this.
IMO AI oversight in peer review is inevitable.
Looking forward to your piece on this! I agree AI oversight in peer review is inevitable. Thinking out loud, it feels like there's two components that right now are conflated in human reviews: "correctness/validity checks" and "taste" (interestingness/novelty/etc.). It feels like the first part is lower hanging fruit right now and an area I'd love to see more work in. The initial tests (https://www.owlposting.com/p/can-o1-preview-find-major-mistakes, https://amistrongeryet.substack.com/p/the-black-spatula-project) using o1 to find mistakes are a good start but would love to see an industrial scale effort here to add all the bells and whistles (citation validity checks, coding environments for repros, etc.).
I have a long reading list at this point but plugging away