SWE-agent is our new system for autonomously solving issues in GitHub repos. It gets similar accuracy to Devin on SWE-bench, takes 93 seconds on avg + it's open source!

We designed a new agent-computer interface to make it easy for GPT-4 to edit+run code
github.com/princeton-nlp/…

1/n Are AI Forecasters the Next Big Thing? How A.I. Learned to Make Freaky Accurate Forecasts

New research unveils an ingenious system that unlocks the untapped potential of artificial intelligence to predict the future. Drawing on the burgeoning prowess of language models to…

Great paper, showing impressive performance from LLMs for forecasting using an elegant retrieval, reasoning + finetuning approach. I expect further improvements in forecasting are possible with current LLMs with more iteration.

Beating prediction markets with chatbots sounds cool. In a recent work arxiv.org/abs/2402.18563, we get somewhat close to that.

As another perspective, forecasting is a great capability domain to benchmark LM reasoning, calibration, pre-training knowledge, and more. 🧵1/n

Can we build an LLM system to forecast geo-political events at the level of human forecasters?

Introducing our work Approaching Human-Level Forecasting with Language Models!

Arxiv: arxiv.org/abs/2402.18563
Joint work with Danny Halawi, Fred Zhang, and John(Yueh-Han) Chen

GPT-4 with simple engineering can predict the future around as well as crowds:
arxiv.org/abs/2402.18563
On hard questions, it can do better than crowds.

If these systems become extremely good at seeing the future, they could serve as an objective, accurate third-party. This would…