Academics have the most to gain from using AI, but they also have the highest requirements for truthfulness and reliability. So far even the best ChatGPT model was not capable of reasoning complex academic thoughts. This might change with the o1 update in September 2024. Here, we compare the improvements and how academics can leverage them.
Join the Effortless Newsletter. Learn how AI, digital note-taking and lit review tools can make your academic work more productive, creative, and relaxed.
What is OpenAI’s new o1 AI?
OpenAI’s newest family of models are called o1 and available for preview since September 13th 2024. They have significantly better math and reasoning capabilities by using longer chains of thought before replying to produce more truthful results with fewer hallucinations. At the moment, however, there are also a number of drawbacks. Compared to the previous model, GPT-4o, the new o1 model is slower, more expensive, and lacks multi-media capabilities.
For researchers this new model is very promising as it is much better at logic and can help not only with text and coding tasks but as a conversation partner to bounce off ideas regarding novel and speculative research ideas.
How to use the new o1 release?
The new release is rolled out to most premium customers, you may have access to it already if you have been paying for ChatGPT. To access the new release go to ChatGPT, open a new chat and click on the model name at the top left corner. Select “o1-preview”.
Keep in mind that currently o1-preview is limited to just 30 messages a week and o1-mini to 50.
What Is The Difference Between o1-preview And o1-mini?
OpenAI does not provide explicit information on the difference between the two models other than o1-mini being 80% cheaper to use, which will not concern you unless you are accessing the models programmatically through an API (i.e. not through the normal chat interface). The o1-mini model is planned to become free for all ChatGPT users. Likely the limitations will be along the lines of how much information can be processed simultaneously, which can also strongly influence the final results.
OpenAI’s o1 vs GPT-4o?
The o1-preview is an early release of the OpenAI o1 model, which is trained using reinforcement learning to perform complex reasoning. The o1 model can produce long internal chains of thought before responding to the user leading to significant gains in use-cases involving logic. Take a look at this comparison from the official o1 release post:
The state-of-the-art GPT-4o model is seriously outperformed in three main areas:
- Mathematical questions
- Coding questions
- Science Questions
While coding was one of the first and most impactful use-cases for AI, PhD-Level Science questions are for the first time (allegedly) at a human level. We will explore this in a few examples in the post.
When should I use GPT-4o instead of o1?
According to OpenAI’s release notes tasks involving textual understanding and creative writing (i.e. less formal logic) are better done with GPT-4o. Not only is it much faster, but it can browse the internet and generate and analyze images. Take a look at this comparison (Image credit OpenAI):
In this comparison, humans were asked to judge which replies they liked more, GPT-4os or the new o1 model. When it came to writing and editing text, the models were on par. As mentioned above the more logic something involved the more the strengths of the new o1 models became visible. These are the exact use-cases o1 was designed for. At the beginning it is worth experimenting with both to find when it is worth using o1. In this tutorial on using AI for academic writing you can learn how to best leverage AI for this purpose.
Note that the use of the o1-preview model is currently limited to 30 messages a week. This cap is likely to change soon.
How to effectively use ChatGPT?
While ChatGPT’s capabilities are advancing, one thing remains important: The way you talk to ChatGPT. We might think of ChatGPT as a research assistant or co-pilot, but that is not the case. It is a generic intelligence that has no background information about you and your use-cases and is subject to strange and almost emotional effects in its reply. In my course on AI, you can learn the art of prompting, how to use AI and how to build assistants for your academic work. According to reviewers, it is one of the best courses fir academic AI available:
Case Studies: Using o1 for Academic Research
Given the current limitations for o1’s use, I was able to test only a few use cases that nonetheless were very promising. Dealing with academic information, o1 gave much longer, deeper and more “thoughtful” replies, allowing me to follow its thought chain and build my own knowledge.
New o1 Model Gives 3x More Detailed Replies
In this example I asked the model a fairly complicated question that you can’t google:
“Describe what Species Distribution models do in ecology and explain to me whether or not they’re suitable for vulnerability assessments.”
The question is kept a little bit vague, as “vulnerability assessments” can mean many things. What o1 did correctly here is that it first defined what this means and then analyzed the suitability of species distribution models to do them. The result is an answer that is more than 3x longer and far more nuanced.
Notice how much more detailed the comparison is. In this section, I highlighted the critical part of the question: suitability for vulnerability assessment.
While GPT-4 gives me just two sentences, o1’s reply is a detailed list of four bullet points.
Summarizing research papers with Open AI’s New o1 Model
Next, I copied and pasted the text of a fairly dry and theoretical paper on ecological models to compare the insights the two models would give me. The results of the initial summary are almost identical, with o1 being slightly better structured and longer. No clear winner here.
Logical reasoning using the o1 model
The paper above dealt with the distinction of process-based vs correlative models. The former explicitly model a process (e.g. how something works), the latter model how patterns of inputs match patterns of outputs – they are not concerned with how something works, but only with the results. Machine learning models are almost always correlative. I decided to ask this “trick question” to the two models. The results are wildly different. While both models declined to answer this question directly (due to lack of information), GPT-4o’s reply was quite short and not very clear (143 words), while o1 gave me a summary of points to consider, defined alternatives, and pros and cons of process-based models (759 words).
This example illustrates how the new o1 model thinks. First it understand the question and breaks it down into parts. For example, I am looking for a definition, so it pulls out the definition and writes it down. Then my question demands it to apply it to a certain problem, so it creates two questions and tries to answer them. In a way it “thinks out loud” compared to the old GPT-4o model.
Using The OpeanAI’s o1-preview for Math and Reasoning
In this example from OpenAI, the models are asked to calculate the acidity of a solution based on the acidity of the different constituents. Here is the question:
Both models engage in a chain of thought that tries to get to the answer step by step. Here is the answer that only the o1 model got right:
In addition to the correct answer, the reasoning leading to it is about two times longer than in GPT-4o. This illustrates o1’s capabilities of not only solving problems but also teaching students to solve problems on their own.
Some more examples of o1 vs GPT-4o comparisons can be found in the original release article.
Conclusion
OpenAI’s new o1 model is very promising not only because it has better reasoning capabilities but also because it allows it to see the way it “thinks.” The chain-of-thought approach might, in itself, be a big help when wrangling complex academic topics or big code bases. If you work in the theoretical field, you might benefit the most from this release, and it might have just crossed the threshold to become a true research assistant.
