Holt Online Essay Scoring: Can Computers Do That?

Holt, Rinehart, and Winston, a part of Harcourt, is a major textbook company in the US. Their website says, "Since 1866, we have been in the business of helping teachers teach and students learn." Our public school uses these textbooks and their internet programs.

Today Melissa, my ninth grader, wrote an essay in school. She typed it into the Holt Online Essay Scoring program, pushed a button, and in less than five seconds she had a score and an assessment page that told her she had "limited ability in word choice," "demonstrates minimal unity," "frequently loses focus," and "uses routine, predictable ideas" (among other things).

I didn't know a computer could grade an essay. I thought humans did that. I know the ACT and SAT have human essay scorers--I applied to be one. Either technology has taken a giant leap forward, or this machine was unnecessarily giving generic, negative comments, tearing down my 14 year-old's confidence in her ability to write.

I told Peter about this and he immediately wanted to look at this program to see what it did. Melissa knew the url and off he went. He answered the prompt and got a score of three out of six.

Here's the prompt:

"Your principal is considering a new grading policy that replaces number or letter grades on report cards with pass or fail. What is your position concerning this issue? Write a letter to your principal stating your position and supporting it with convincing reasons. Be sure to explain your reasons in detail."

Here's what Peter wrote as a test, to see what would happen:

The grading system currently used in the vast majority of schools is counterproductive. Students learn that the important part of education is impressing a teacher, rather than truly becoming proficient in the subject matter. Replacing this with a Pass/Fail system would solve this problem.

Under a Pass/Fail system, students could still be penalized if they don't do the work. Teachers would require a good faith effort in order to pass the class. However, the focus would no longer be on acing quizzes and other trivial factors. Students would be able to focus on learning the material. Students who are interested in becoming educated will still be motivated without grades, and students who are slackers already surely can't get any worse.

His automatic assessment said he had "limited word choice" with "noticeable lapses in the logical flow of ideas," etc. and gave him a 3 out of 6. Melissa was told the goal of the program is to get a four by the time you are a junior in high school. Not sure what that means exactly. I'll have to ask her English teacher.

Now Peter wanted to see what he could do to this first essay to get a higher score, so he added "Dear Mr. Principal" and "Sincerely, Stud Studentman." He still got a 3, but now the computer said he "competent ability in word choice." Wow! Way to go Peter!

Next he began to add nonsense words to see if the computer would react. Nope. No change--still a 3 with the same comments. The essay got increasingly bizarre but no change in score. So he decided to add a few paragraphs he copied off of Kem's Utterly Merciless Guide to Essay Writing. Now he had the traditional five paragraphs. He put in a couple transition words, pushed the button, and sure enough, he got the magic score of four! Here's the essay and the computer's assessment:

"On a 6-point scale, here's your score: 4

This response demonstrates competent success with the persuasive writing task. For the most part, the essay:

* focuses on a reasonable position, with minor distractions

* shows effective organization, with minor lapses

* offers mostly thoughtful ideas and reasons

* elaborates reasons and evidence with a mixture of the general and the specific

* exhibits general control of written language

(All essays that score a 4 get the above generic response. You can see it here.)

The Essay that Earned a Score of 4:

Dear Mr. Principal,

The grading system always used in the vast majority of cheeses is counterproductive. This is because students learn that the frankly important part of fly fishing is impressing a classroom, rather than truly becoming an eyesore in the subject matter. The three problems are too much cheese, acrobatic flying monkeys, and the electoral college system. Replacing this monstrosity of a teacher's pet with a Pass/Fail system would solve every problem in the world.

First, too much cheese leads to bad grades. Under a Pass/Fail system, students could still be drawn and quartered if they don't do the bull-baiting. Factory workers would require a good love effort in order to pass the cheese. However, the focus would no longer be on acing quizzes and other lunatic factors. Students would be able to focus on learning the website. Students who are interested in becoming housewives will still be motivated without grades, and students who are drug addicts already surely can't get any more smelly.

In addition, flying acrobatic monkeys will attack this student, whose name may as well be Siegfried. He is baffled at the assignment. Journey motifs? What does that mean? He takes a look at the two texts on which he wants to write, the General Prologue to Geoffrey Chaucer's Canterbury Tales and Book I of Edmund Spenser's Faerie Queene. Both texts involve physical journeys, but he can't really think of anything to say about that. They're just journeys, right? What's to say?

Finally, the electoral college system eats brains and takes names. Siegfried gets to work and busily lifts ideas (and sometimes even full paragraphs, since they fit so well) out of the online papers. He adds a generic introduction and a conclusion that repeats it almost word for word. Voilà: an essay...and he's hardly had to think about it at all.

In conclusion, if you haven't seen the comments on the last post, check them out. Eliminate the cheese, the monkeys, and colleges. Two Instigators of Filthy Plagiarism have infiltrated the site, and I and a reader named Kyle have declared war on them. Will you join in our crusade? Who will be strong and stand with me? Somewhere beyond the barricade is there a world you long to see? I think so.

Sincerely,

Stud Studentman

----
From the Intelligent Essay Assessor:

Analytic Feedback for Your Essay

Our system has analyzed your essay for five important writing traits:

* Content and Development

* Focus and Organization

* Effective Sentences

* Word Choice

* Grammar, Usage, and Mechanics

Study the statements that describe each trait to help you improve your writing.

Content and Development Your essay shows competent ability for this trait. For the most part, the essay:

* uses some meaningful and thoughtful ideas

* elaborates and supports some ideas with a mixture of general and specific details, reasons, explanations, and/or examples

Focus and Organization Your essay shows competent ability for this trait. For the most part, the essay:

* addresses the prompt, but may include minor digressions

* shows some awareness of audience

* displays effective organization and transitions, with minor lapses

* demonstrates general unity and completeness

Effective Sentences Your essay shows competent ability for this trait. For the most part, the essay:

* generally forms sentences correctly but with occasional errors

* demonstrates sentence quality with few, if any, awkward sentences

* displays some variety in sentence types, lengths, structures, and beginnings

* displays some fluency

Word Choice Your essay shows competent ability for this trait. For the most part, the essay:

* uses words that are generally appropriate to audience and purpose

* uses some words that are precise and accurate

* may use figurative language and imagery somewhat effectively

Grammar, Usage, and Mechanics Your essay shows competent ability for this trait. For the most part, the essay:

* demonstrates general command of language conventions

* exhibits general command of spelling, punctuation, and capitalization"

Needless to say, I'll be showing this to her English teacher.

update: I did talk to her teacher. Here's the response.