It's been a week since I posted Holt Online Essay Scoring: Can Computers Do That? I find the whole idea that a computer can score subjective elements of essay writing (like logical argument and word choice) fascinating. I've gotten some great feedback, but I'd love more. So please share your thoughts and experiences on this topic. All I could find were sites by Holt or sites describing the program, and very little objective discussion on the pros and cons of computerized scoring.
I talked to Melissa's 9th grade English teacher and she was very nice but very convinced that computer scoring was worthwhile. She said the online essay scoring is just a pilot project, she always looks at the essays herself, and whatever score the child gets is not part of their grade (that's good news, but the kids still have to get a score of 4 before they graduate).
Whenever I wanted to talk about the shortcomings of the program, she'd refer to the disclaimer on Holt's website:
Our system works best for students who make a good faith effort to respond appropriately to the prompt. If you stay focused on the topic and do your best to write a well-developed essay, you will receive a score and a description of the scoring criteria.
For her, this disclaimer was enough to dismiss Peter's nonsense essay. But, as I pointed out, he actually got a score and feedback AS IF he had written an appropriate response that stayed on topic. Apparently, the computer thinks Peter's essay is fine, giving it a 4, that magic "passing" score. This fact alone should convince anyone that the program is unreliable.
To help me understand how the program works and why she's using it, Melissa's teacher sent me an article from The Irish Times by Carol Power called "Computerised Essay Evaluation Replacing Human Markers," July 2, 2004. Here's an excerpt:
These companies gather sets of essays written by hundreds of students of varying levels and have their papers graded by trained teachers. The companies then train their software engines to mimic the standards - content, grammar, spelling - the hundreds of expert human scorers use.
"We take the training papers, run them through the computer and assemble algorithms particular to each question," said Mr Richard Swartz, executive director of technology products and services at ETS, the largest of the three companies. "The computer learns the linguistic features used by the readers and so we develop our scoring model," he added. "It only takes a couple of days to train software engines to score tests."
and a little further down the article:
"Indiana is the first state to use a computer-scored English essay test in a statewide assessment and its experience could influence testing decisions in other states. The Indiana Department of Education has provided all ninth-grade English classrooms in the state with access to Criterion Online Writing Evaluation, an automated essay reading service developed by ETS. Criterion evaluates a student's writing skills and provides instant score reporting and diagnostic feedback to both the instructor and the student."
Our school system here in Illinois feels the pressure to perform based on the No Child Left Behind Act, and the fact that Indiana performs better than we do while using computerized essay scoring seems reason enough to implement computer scoring around here.
This is just another example of good intentions in an impossible situation. The teachers want to teach, the government wants measurable results, and the children are caught in the middle. I've always thought our current educational system acted like a machine. I guess I shouldn't be surprised when they gravitate toward mechanical methods.
7 comments:
Grr, I have issues with that on SO MANY levels. People seem to think that computers can do it all but they don't get that we still cannot program decent AI, and a list of objectives still doesn't work when the material is subjective. My husband is a programmer who often deals with the subject of game AI and predicting human behavior and it is amazing how clueless people are about what is and what is not possible.
Then there is the whole issue of negative feedback and training children to work the system to avoid negative feedback--what portion of her class is actually willing to take a lower points and do it right if they know that wording something another way will get them higher points? Sounds like someone has unrealistic expectations or just plain doesn't understand her students.
Jena, I wish I had sources or references or something to help you out, but I don't. I am appalled by this, really, but not surprised.
My son is talking about going to high school here in IL because of sports, so I have personal incentive to want this to not be used in IL.
And I'm sure that IN does absolutely everything else the same as IL in education so using computerized essay scoring is the reason why their test scores are higher, right? Gee whiz.
Someone is making money out of this. And the government is saving money. Or their money is going to big companies instead of hiring more teachers. They probably think they'll save money. I would almost guarantee that this is about money and efficiency NOT about high quality education. gRRR.
I have to agree with the other commenters. There is a push for the kids to do well but there is no push to improve the actual system of teaching. And yes...someone has to be making money from this!
Yes, people are making money on this -- big money! This is ridiculous. I hate to sound like a dinosaur, but the idea of a computer scoring essays is nuts. So much of writing is about feeling, which computers can't do. And can computers really understand all the various connotations of words? As a college instructor and a professional writer, I find this offensive in so many ways.
I was just grading essays this morning, and sometimes you read something that is grammatically flawless, but it just doesn't make sense or is missing the mark. As I read one essay, I was at first stunned at how perfectly it was written, but when I got to the end, I realized he had never really carried through on his thesis statement. He got sidetracked and never got back on topic, although the writing was flawless. The student is clearly an excellent writer, but he needs to work on organization and critical thinking skills. I can't imagine that a computer would have found anything wrong with his essay.
Writing well is like painting or sculpting. It's an art. I wonder how a Picasso would score if a computer graded it.
I agree with all of you. The idea that a machine can truly "read" is just beyond our technology right now, but it's convenient and the textbook companies make it all sound so scientific and true. Thanks for your thoughts on this!
BTW--blogger is having trouble with the new follow thing--I will try again later.
Post a Comment