Authored by Yixin Qian
Before I started this project, I thought Python was just a friendly snake emoji. Turns out it’s also a programming language that follows rules as strictly as a grammar teacher who has had too much coffee.
My Digital Research Incubator project is simple to explain but surprisingly chaotic to do:
I want to test the quality of ChatGPT’s feedback on student essays. That’s it. But, of course, I had to choose the complicated way—by using Python to develop tools to evaluate the AI model’s enhancement of the text.
Why This Matters (A Serious Reason Hidden in a Funny Adventure)
Students use AI every day now. Some ask ChatGPT to fix their essays. Some ask it to rewrite whole paragraphs. Some ask it to make their writing “sound more professional,” and ChatGPT turns it into something that looks like a legal contract from 1876.
Because I’m a language teacher, a language learner, and now—whether I’m ready or not—a researcher, this project feels strangely personal. I really want to know: Can we trust AI feedback? Or should we treat it like a weather forecast: helpful, but sometimes just wrong? Also, when I write my own research papers and ask ChatGPT to check them, I always wonder: “Did this robot really help me, or did it just confidently lie in a polite way?”
Learning Python: Like Learning a New Language, but with More Error Messages
Python is basically a new language for me. Except instead of vocabulary words like “apple” or “library,” I get things like
- len()
- append ()
- str.split()
Python has its own rules, and when I break them, it doesn’t just underline my mistakes—it refuses to work, throws red errors at me, and acts like I betrayed it.
Whenever I don’t understand something (which is often), I ask ChatGPT:
“Explain this like I’m a five-year-old. With funny examples, please.”
And then ChatGPT says things like:
“Imagine len() is your friend who counts how many dumplings you have.”
Yes. This is the level of explanation I currently need.
What Python Will Help Me Do (Once I Stop Fighting With It)
Even though I’m still learning, I will use Python to study and interpret that data from my tests with ChatGPT. For example, I plan to write a simple script to count the types of feedback and how many times ChatGPT discusses issues concerning grammar, vocabulary, organization, content, or other things. I will also use a Python script to look at the consistency of the AI tool. If ChatGPT comments on different mistakes every time it reads the same essay, that says something important. It means the robot is unpredictable—like that friend who keeps giving different restaurant recommendations every time you ask. From this data, I will also use Python to support the visualization of the collected data, highlighting which mistakes ChatGPT points out, which mistakes it usually ignores, and how feedback changes over time.
What I’m Actually Doing (In Real-Person Words)
To start the project, I downloaded a giant data package with 1,000+ student essays written by real language learners. These essays demonstrate the range of mistakes common to student essays: grammar, vocabulary, organization, punctuation, etc. and sometimes a combination of all of these issues at the same time.
And then came my first challenge, which hit me like a flying Python error message: I had absolutely no clue how to get Python to read my data. I opened the folder full of essays. I opened Python. And Python basically responded with, “Wow, that’s a lot of files. Good luck! Not my problem!” I stared at the screen thinking, “Okay Python, buddy, you’re supposed to HELP me. Why are you acting like a lazy coworker who refuses to do anything unless I say the magic password?”
However, I realized that Python wasn’t going to complete all the steps of my project. First, I needed to gather data on how ChatGPT provides feedback in order to work with Python scripts to analyze that feedback. So, next I chose one essay and gave it to ChatGPT like a teacher handing papers to a very fast but slightly unpredictable teaching assistant. But instead of grading each essay once, I make ChatGPT grade it three times. Why?
Because I want to see if ChatGPT is
- consistent
- inconsistent
- secretly moody
- or maybe just bored
So far, I’ve noticed that sometimes it points out ten grammar mistakes the first time, seven the second time, and twelve the third time. It’s like the robot had different amounts of coffee each round.
The raw data I collect from these ChatGPT sessions, will form the basis for the next steps of analysis using Python scripts.
Challenges: Me vs. Python vs. My Own Confusion
Here are my top challenges so far:
1. Python syntax
Python says things like “IndentationError: unexpected indent,” which basically means:“You put one extra space. You ruined everything.”
2. Designing my rubrics
How do I decide what counts as “good feedback”?
Does ChatGPT need to catch every mistake?
Is it enough if the feedback is clear?
Is it “good” if it sounds friendly?
It is hard to quantify what counts as “good feedback.” Even humans disagree on the definition.
3. Understanding the data
I’m still trying to decode the giant document of student essays, clean the text, and get everything into a format Python can work with. Python is picky. It wants everything tidy and organized—basically the opposite of real student writing.
What I’m Learning (That Surprised Me)
As a researcher and someone learning to code, I’ve noticed a number of things through this process:
- Python really is like learning a language.
- ChatGPT explains code to me like a patient tutor babysitting a confused toddler.
- Sometimes ChatGPT gives very accurate feedback—and sometimes it misses mistakes even my grandma could find.
- Consistency is not AI’s strongest quality.
And honestly? I’m realizing that studying AI requires as much caution as using AI.
If I Had Unlimited Resources … Oh, the Things I Would Do
If I suddenly won the research lottery, I would
- Collect thousands of essays
- Compare every major AI tool out there
- Work with linguists, teachers, and programmers
- Build a website that gives students AI feedback AND shows how reliable it is
- Maybe even train my own AI model
Basically, I want to do everything. But for now, it’s just me, Python, ChatGPT, and an Excel file slowly turning into a monster.
Conclusion: Learning to Talk to Robots (and Python)
In the end, this project is one big adventure. I’m learning Python. I’m studying AI feedback.
I’m testing consistency, accuracy, and patterns. And I’m laughing at myself almost every day.
My main goal is simple: Help students understand how much they can trust AI—and how careful they should be when using it. And honestly? By the time I finish this project, I might become fluent in two new languages: Python and robot logic. If that’s not research progress, I don’t know what is.