As more and more researchers use AI for academic writing, it becomes essential to have reliable methods of evaluating AI-assisted writing. AI detector tools, like Paperpal’s, provide a novel solution to evaluating our writing, so we can better automate tedious writing tasks without losing our voice and flow to generative AI. AI detectors are dedicated tools designed to evaluate how human or AI-generated a text appears, and can be used to help make our AI-assisted writing better and more human. Let’s dive into how best to use AI and AI detectors to improve our academic writing.

Introduction: Academic Writing with AI

AI isn’t just transforming the research process, it’s redefining every step, from ideation to publishing. Nowhere is this more evident than in the explosion of AI-assisted writing tools in manuscripts submitted to journals and conferences. It’s likely that at least 50% of researchers today are using AI to assist in writing, whether that’s using AI-generated text, using AI to evaluate their work or some other use case. That’s regardless of position, with one study finding that over 20% of senior professors use AI writing tools daily. Here is a comparison of a few currently available tools:

But how are researchers checking their writing? How do you know when AI has helped your writing versus when it has started to take over?

This is where AI detectors come into play. These tools may be more often viewed as ways to “test” a piece of writing, perhaps in a classroom or assignment setting. But, those use cases are less reliable, since AI detection is still far from 100% accurate, and such assessments can lead to errors.

Where AI detectors really shine is in helping us to evaluate our own writing, especially if you used AI in the process. These detectors aim to break down papers based on what text is likely human-written vs AI-written, allowing you to get granular look at your paper’s composition, as well as a general overview of how natural the writing is.

How to Write with AI: Responsible Frameworks

After extensive experimentation, I’ve found that the best way to use AI for academic writing is as an aid that keeps that researcher at the centre of the process. That generally means using AI as a way to automate tedious writing steps, while using source information you are already familiar with (i.e. notes, resources, papers, etc.). You can also ensure that any AI-generated text accurately references your input sources.

One framework for responsible AI-writing is the Atomic Statement Method. This is something I discuss at length in my courses, but it boils down to this: using AI to transform hundreds of notes and sources you’ve curated into an initial paper draft. For example, here’s how to write your next Introduction with AI using this method.

But, this is often easier said than done. After all, you can maintain responsibility for all the source information, but if the AI aggregates and summarises the text, how AI-like will the final paper sound? That’s exactly when AI detectors come in handy. We’ll look at using these tools as a means to evaluate your own work and to maintain the highest-level of quality and human touch in your writing.

How much is too much?

“Writing with AI” means very different things to different people. To some, it may mean using source knowledge, notes or even full-text inputs to improve exisiting manuscripts or combine different ideas. For others, it may mean providing ChatGPT a simple prompt, and copying-and-pasting the provided text as-is.

For me, it means follow responsible and ethical guidelines that keep the researcher at the centre of the process. Over-reliance on AI not only is unethical, but jeopardises our cognitive sovereignty as researchers and limits our potential. Check out my video on ethical AI use in writing for a longer discussion (or to share what you think!).

What are AI Detectors?

Various AI detectors exist today and function in a similar manner. To use an AI detector, simply upload a piece of text as plain text. The detector will provide a report of how human vs AI the text appears. Input limits depend on which tool you use, and whether you are on a free or premium tier. You can then edit your writing based on the report, and pass the text in again, seeing if your changes helped push your score towards the right direction.

We’ll look in detail at using the Paperpal AI Detector to evaluate and improve academic writing below.

How do AI Detectors Work?

AI detectors are trained on large corpuses of text to identify patterns that indicates AI-generated writing. It’s not just about the repetition of certain words or phrases that AI bots tend to go for (although this is an important variable). AI-like writing can also be inferred based on how the text reads, or how a phrase sounds based on the context. In this way, the general overall flow and how a narrative is delivered tends to trump more granular indications of AI-generated text.

To truly understand how “AI” a document sounds, these detectors need to consider both a high-level document-wide analysis, as well as a sentence-by-sentence breakdown. When AI detectors first came out, they mainly focused on document-wide review. Today, the best tools provide both, allowing us to more efficiently improve text that was written with the assistance of AI. We’ll be looking at detail at one of these detectors, Paperpal, below.

When Not to Use AI Detectors

No AI detector is yet 100% accurate at distinguishing human vs AI-generated text. You’ll quickly discover this after running a few experiments of your own with these tools. For this reason, it may be best exercise caution when using these tools as plagiarism checkers, cheating detectors or as a form of evidence for disciplinary reasons.

For example, the following text is 100% AI-generated from ChatGPT, but QuillBot’s AI Detector tool estimated it as only 64% AI-generated, with specific sections being “human-written”.

The same can happen in reverse, that 100% human-generated text may score as having some AI influence, even where none exists.

AI detectors and generative AI tools will continue to evolve as any cat-and-mouse game, with each one trying to outpace the other. This is especially true as AI-generated text becomes more advanced at utilising different tones of voice, styles and vocabulary to better blend with human-generated text.

However, for the time being, AI detectors are still a handy way to get a general sense of how text was written, and to check your own work.

Using Paperpal AI Detector to Improve Academic Writing

Let’s look in detail at one of the newest AI detectors available, built by Paperpal. This AI detector can be used for free up to 5 times per day, with a 1,200 word input limit per attempt. To use it, simply sign up for a free account with Paperpal, and navigate to the AI Detector tool in the sidebar.

Next, let’s dive into using this AI detector with the express purpose of improving an existing manuscript draft. We’ll use the Paperpal AI detector to evaluate the human vs AI makeup of the text, edit the manuscript, and re-run our work.

Paperpal AI Detector Use Case 1: Improving 100% AI-Generated Text

As an initial example, we’ll use text from a 100% AI-generated manuscript that was submitted to a top conference in machine learning (ICLR) last year. This is the most extreme example of using an AI detector, because the text was 100% AI-generated. If you use something like the Atomic Statement Method (discussed above), then your writing will be a blend of AI-generated and human-written text. We’ll look at that (far more common!) use case in the next section.

We use Paperpal to evaluate the Introduction text, and Paperpal rightly identifies the text as largely AI-written (93%).

Based on this report, we can now work to improve the manuscript by reviewing the AI-heavy writing patterns that Paperpal discovered. We’ll start by focusing on the section highlighted in dark pink, denoted as “AI-written” and which makes up 36% of the total input text.

We reviewed the “AI-written” section in detail, re-writing it with a more natural and conversational tone, while retaining the key facts. The full text was again reviewed by Paperpal, with a satisfying result. The tool identified that now only 79% of the text is AI-written, and this section in particular has a blend of human and AI-generated text. Success!

It’s worth noting that we had to go through two rounds of edits in order to significantly improve the score for this section. After one round of edits, the entire section was still marked as being “AI-written”.

The challenge stems from trying to de-AI a text that is already fully generated by AI. This can be especially difficult, since the AI has opted for a particular format and flow, which can be hard to improve without taking a very big step back. It’s not so different from editing the writing of a co-author or collaborator. We need to see the entire story of a paragraph or section, and consider the best way the entire passage can flow in order to truly improve the writing. With AI-generated text, this can be tricky since the AI nearly always sounds accurate, but is not necessarily the best way to express something. That’s the difference between an AI or mechanical voice versus the real human touch researchers have when writing academically.

Paperpal AI Detector Use Case 2: Improving a Hybrid Manuscript

Many researchers today are using some kind of AI-assisted hybrid approach to their writing. The Atomic Statement Method is one way to do so, allowing us to automate writing processes while maintaining control and access to all the content, inputs and sources. You may also be incorporating AI in your writing in various other ways, like:

• Editing writing with AI (Grammarly, Jenni AI, Paperpal, Scite)
• Generating outlines and first draft or discussing ideas (ChatGPT, Gemini, Claude, Perplexity)
• Transforming notes & sources into drafts (NotebookLM, ChatGPT, Custom GPTs)

Whether you’re using AI as a “light touch” to check for grammar, or as an early step to process ideas and overall flow, that incorporation of AI may come off in the final writing. For all these cases, it can be helpful to run your draft through an AI detector to get a pulse on how “AI-like” the final writing sounds.

Here, I’ve uploaded a real paper draft into Paperpal’s AI Detector tool. This was a draft I wrote with some assistance from AI. For this draft, I used AI to help transform my notes into outlines, improve grammar, and to help with overall structure. Since I rarely use AI-generated text exactly as-is, the result is quite good: only 25% of text was identified as having AI writing patterns.

Above is just a snippet of a full paper I passed into Paperpal, with the specific “AI-heavy” passage displayed. Most of my paper was in the green or yellow zones (either human-written or a very light touch of AI). By seeing which sentences and passages it deems are the most “AI-heavy” in voice (highlighted in red), and I can start with those to update my draft.

Compare this to a pre-AI-era paper, which is 100% written by humans:

While this paper is also identified as having some AI-writing, the Human-AI blend is much lower, notably the “AI-heavy” part of writing is entirely absent. This makes sense, as AI was trained on pre-AI-era papers like this one, and the writing it produces will be similar to those papers. That, of course, goes both ways, making the pre-AI-human writing similar to post-AI AI writing. However, the Human-AI Blend bars can give you a more fine-grained indication of the truth.

With tools like Paperpal’s AI Detector, I wouldn’t aim for a perfect 0% score, because the nature of detecting AI in writing is (and always will be) imperfect. However, an overall score of 25% gives a strong pulse-check that the article reads largely as human-written, but still needs a bit of improvement. Especially the heavy-AI mix (orange highlights) should be eliminated. The line-specific suggestions give me a clear place to start with my edits.

Paperpal vs ZeroGPT AI Detectors

ZeroGPT, another popular AI detector tool, provides an overall percentage for how much AI-generated text is in a given input. It also highlights the text it is most certain was AI-generated. However, it lacks the nuanced analysis that Paperpal provides. Paperpal highlights text in different colours based on how intense the AI-impact is (high, medium, or low). This makes it easier to prioritise which text to focus on in order to ensure a high-quality, human-readable manuscript.

Paperpal vs Grammarly AI Detectors

Grammarly also offers an AI detector tool, which provides a general score of “Resembles AI text” vs “No AI text patterns found”. However, the Grammarly AI detector doesn’t provide the sentence-by-sentence breakdown that we see in Paperpal and other detectors. It provides only an overall document analysis.

Summary: Using AI Detectors to Improve Papers

AI detectors are a powerful way to improve academic writing and ensure your manuscripts sound human-written. This isn’t just important to avoid “AI-sounding”, but to ensure the overall narrative and flow is as natural as possible.

AI detectors and AI generative tools are each advancing at unprecedented rates, making it difficult to truly detect when or how AI is used. For this reason, be careful not to take scores at face-value, and avoid using them as evidence for disciplinary action or to prove AI usage in text. These detectors do not claim to be 100% reliable, and such a strict interpretation isn’t intended.

AI detectors are a useful tool for the writing process to evaluate how human versus AI your text sounds. Try AI detectors like Paperpal’s which provide line-by-line analysis and give you a clear starting point for editing your work, as well as ability to re-test after you made changes.

If you find yourself getting routinely high AI-sounding scores on these detectors, you may want to revisit how you’re approaching your writing. There is a way to use AI to enhance writing without losing cognitive sovereignty or your unique voice. Check out my dedicated Writing with AI Course to learn more.