My school district is hosting one of the in-state MAP (Missouri Assessment Program) grading sites. Teacher’s from all over Missouri would come to grade the third grade science test from all over Missouri. Being a Missouri third grade teacher, I have a natural curiosity into how the MAP is graded. One of my fellow teachers applied and was hired to spend a week grading those tests. She called to let me know that there would be a day when Missouri educators could come and visit the site to see how all this MAP grading stuff works out. I naturally decided to make an appearance.
I went prepared to learn how I can better enable my students to take a stupid the MAP test. I found the room that I was told to go to and walked in. There were about 30 to 40 teachers sitting in front of computers (some very nice computers I might add) reading and quietly tapping in keys on the keyboard. I walked, unnoticed behind the teacher that invited me and said hi. It was at that moment that the Grand Poobah of the testing site looked up and made a bee line for me.
“Excuse me sir!” The Grand Poobah was almost at a full sprint across the rather tiny room. “Excuse me! I need to speak to you out here.”
“Oh, I’m sorry, I thought that visitors were allowed today.” I started making my retreat, evidently I had breeched some kind of protocol that I wasn’t aware of.
“Yes, that is today, I just need you to come in this other room and view a little presentation and sign a non-discolsure agreement.”
“Non-disclosure agreement?”
“Yes, basically saying that you won’t talk about the test questions.”
“Well okay.” I didn’t mention that I had already seen all the third grade science questions, that probably wouldn’t have been greeted with great enthusiasm.
I was ushered into a small room where a PowerPoint presentation was made going over how the test is developed and then scored. The presentation also covered the non-disclosure agreement, I could discuss how the test is graded but I could not discuss any test questions that I would see, they are confidential. Again I refrained from mentioning that I had already seen the test and had a pretty good track record of not letting test secrets slip to the general public. (Pssst! You can however, view some released items from the 1999 science test in PDF here, but don’t tell them I told you.) The whole presentation made me feel a little uncomfortable because I was the only person who was listening to the presentation. I had never been in the position to have a PowerPoint presentation being made to just me, I didn’t know where to look, the presenter or the PowerPoint. After the awkward presentation, the signing of the NDA, and the receiving of the correct credentials, I was allowed to enter the inner sanctum.
After students finish the MAP test, they are packed up and shipped off to California where each test is cataloged and disassembled. The test is then scanned into a computer. The evidently complex grading software then distributes the tests to the correct grading centers, some in Missouri but most in California.
Each grader goes through about a week’s training period to grade a specific section of the MAP test. They then my grade a normed example test to make sure that they are correctly grading the test according to the sometimes convoluted grading rubrics. After being certified on one section of the test (it seemed that each section consisted of about 15 to 20 questions) the teacher is then allowed to begin grading. Each grade is set in front of a top of the line computer (I think I was told that the computers were used for three years) and an enormous 20 inch LCD screen. A test would be distributed to them from California that they would then grade by inputting the point value that the student earned. The program had several tools to help graders if they weren’t able to read a response, they could invert colors, enlarge specific sections of the test, they could even remove the “template” or the actual test print leaving only what the student had written. If a grader still was unable to decipher the response, the grader would then ask the graders on either side and then the “table leader,” or the person that had received even more training in California on one of the specific sections of questions, would have to come over and agree that it was illegible. (I was actually able to help two of the graders make out the child’s response, however, unfortunately for the student, it didn’t help with the score.)
The table leaders didn’t actually grade any tests. They would answer questions from the graders and then they would grade behind. When the table leader would grade behind, they would get test that had already been graded by the graders and they would input what they thought the score should be and then the computer would reveal what the grader gave. If the scores matched then the table leader would move on, if they didn’t then the table leader would either change their score or change the score of the grader. If a grader was often not agreeing with the table leader, the grader would be retrained and then have to be re-certified before they were allowed to continue grading. The table leader would grade behind about 10 test from each of the graders. Each grader would grade hundreds of test in one day.
Another “safeguard” to check up on the graders is an already graded test would be sent to their terminal, unknown to the grader, and their scores would then be compared to what the correct score was. Again, if the grader failed to grade correctly they would be retrained and re-certified before they could continue grading.
Only about 10% of all Missouri tests are actually graded by Missouri teachers. The rest are graded in California by people who have been certified to grade questions, and that is about the only qualifications required. I have heard stories from other teachers who have been graders in past years about people who were hardly able to speak English being certified to grade these tests. They could make out and understand most written english, but when it came to reading third grader hand writing… forget about it. That scares me.
So what did I learn?
- In my humble opinion, there is a lot of money being spent on these tests. Too much. I wonder if it is worth it. Of course I have an opinion on that too.
- As I went around to the graders, I asked them, so what have you learned from grading these tests? All of the response had nothing to do with content and very few had to do with methodology. Most of the teachers said something about how to have students take the test. Tell your kids to write in print. Make sure they erase really good, the scanner doesn’t like any smudges, make sure they write in large print… My biggest complaint about the MAP test has always been the large amount of time that I spend on “test skills” rather than on content or thinking skills. These are test skills aren’t really useful on other tests even, just this one. Preparation for the MAP test certainly doesn’t get older students ready for the ACT or SAT tests.
- As the day goes on (I visited later in the afternoon) the graders begin to loose their sanity. Well, maybe they get a little punch happy. I can’t hardly blame them, sitting in front of a computer for 6 or 7 hours in a day grading these has to be strenuous. To break the monotony, when a teacher would come across a funny response, and there were plenty, they would share it with all the graders around them.
- One of the graders told me, after carefully looking around to who was listening, that even with all the safeguards in place, the grading still varies widely and doesn’t really show an accurate picture of what the student knows. The Grand Poobah then quickly came over and cut off her head for such heresy. (I made that last bit up.)
The experience was very interesting. I’m glad that I went but it certainly didn’t improve my thoughts of the MAP test.
Class dismissed!
With the new movie Batman Begins, the Batman movie franchise gets a much needed rebirth. As the title suggests, the movie shows how Batman becomes Batman. It is basically the same story that you will find in the comics and the first Batman movie. This movie takes it one step further and shows Bruce Wayne’s struggle to deal with his problems back in Gotham City and how he finally decides to deal with it. Of course this part of the movie is Batman-less, and thankfully, I wasn’t sitting there thinking “When are we going to see Batman?” That section of the movie holds its own. As soon as Wayne returns to Gotham, we witness how the physical presence of Batman comes about. For the rest of the movie Batman fights the baddies, saving Gotham and his loved ones.
