Evaluating curricula: #1—Jamie's committee hears from the curriculum company representative
What do you want to know about curricula?
This a little fictitious story about Jamie, a special education teacher in a local education agency named Monterrey and a member of Monterrey’s advisory committee on curricula for the LEA. But Jamie could be a parent, a general education teacher, or an administrator who’s been appointed to this committee. Imagine that you are Jamie.
You not only like the other committee members, but you respect them. You’re convinced that they want to recommend curricula that will benefit students and school faculty members. Of course, you have differences—some of you tilt more toward phonics but some lean toward literature-based lessons—but you’re pretty sure that everyone cares about outcomes and is willing to listen.
Here’s a vignette about one of Jamie’s experience.—JohnL
Act 1, Scene 1
You are Jamie. An educational advocate comes to your LEA (and your committee) and essentially says, “Yo, I got the best method of [reading | math | behavior management | whatever], and I want y’all to buy it. It’s really groovy. It rings all the bells: It’s based on evidence; it has supplements for individualization; other LEAs tell us it’s their go-to resource….”
“Should I continue,” the consultant said, and then, without waiting for confirmation, she did: “It has embedded professional development and support. There’re student-friendly materials with illustrations by a Caldecott Award winner. The lessons and materials are designed to be considerate of kids’ backgrounds according to guidelines recommended by the DCAS [Diversity Council of American Schools}. It’s easy for teachers; it tells exactly what to do in convenient checklists, but it’s not scripted. It adjusts to students’ performance so that each student gets what he or she needs. We do this by using a tiered system with primary prevention, secondary adaptations, and tertiary individualization....”
At the same time that you’re already starting to tune out the advocate’s presentation, you begin leafing through her handout (a fine and slick booklet!), looking for anything that mentions evidence of effectiveness. “This stuff rings a lot of good bells,” you’re thinking, “But does it actually work?”
Eventually, after a beautiful slide show using many painterly images from the Caldecott Award winner and gushing quotes from ecstatic customers (“We had no idea our kids would do so well!” “Last year we had so many failing students, but once we started using XYZ, we hardly have any!”), you wonder how long the presentation will last. You are resisting letting your eyes close.
When the presenter says, “Any questions?” you look around at your admired colleagues on the committee, bolster your courage, and say, “Yes. I’m Jamie Ireland from San Julian Middle School, special ed. Thank you for your presentation. I’m just wondering, do you have any actual research evidence about XYZ?”
“Yes,” the consultant replies. “Let me go back and find that slide about Smithtown Elementary. Remember...just a minute...yes, here it is. [The consultant shows an earlier slide with a beautiful graph…perhaps one you missed while your thoughts wandered.] Now I know that you’re middle school, but middle isn’t that different from elementary…but remember that Smithtown’s scores went from 67% passing the state test to 81% passing in only three years of using XYZ. That’s a pretty great improvement! It’s almost 5% a year of improvement! Oh, yes…we don’t have break-out data about special ed kids, but I can assure you that they did really well, too. You know what a rising tide does to all the little boats, right?”
“Nice, indeed....” You hesitate, glance around again, and then decide to ask your next question. “Uhm, how did the control group do in that study?”
“Oh!” the consultant replies. “Well, this was what’s called a ‘quasi-experiment’; there wasn’t any control group. We didn’t want to cut kids out from benefitting from the program.”
You politely half-listen as other members of the committee ask questions.
Act 1, Scene 2
Later, after the meeting has ended, you find that the assistant superintendent for instruction is calling you in the hallway as you are leaving. “Jamie! Hold on. I’ve got a question for you,” she says. She catches up and as you walk together to the parking lot, you talk together. You notice the slightly brisk air of the early-December evening and how it seems to be getting dark sooner than it was, just a few months before when school was just starting.
After some small talk, she asks, “So, Jamie, were you happy with that answer about research?”
“Not really,” you reply. “I’m not some big scientist, but I think it’s important to have a control or a comparison group...so you know what would have happened if kids didn’t get the special program. Like, you know, I’m just comparing tonight’s temperature to the temp in September?”
“Well,” Anna the assistant superintendent says, “I agree. And I’m not a big scientist either, but I remember that important research in all my classes—my biology classes, my psychology classes, shoot, even my archeology classes—that research had control groups...with all that talk about random stuff, too!”
“Right!” you reply. “And I don’t get how they can say it works better than everything if they haven’t even compared it to something...one other thing, at least.”
“I agree,” Anna replies. “We need some hard evidence. I know a professor at the university who likes this research stuff. Brenda Bebopp is her name. I think we—well you and the committee—could use some help understanding how to judge a program’s evidence of how effective they are. Do you think I should call her?”
“Well, sure.” you say, while walking across the cool, dark parking lot. “But it’d be great to get advice from someone else, too. I mean, what if Professor Bebee (?) is biased...maybe not in favor of the XYZ program, but about some way of looking at research? Do you know what I mean? Do you know anyone else, too?”
“Oh, of course I do,” Anna replies. “I don’t think Brenda—it’s ‘BeBopp,’ by the way—is biased. But you’re right about how people having their own views, even researchers. So, I’ll shop around for another research expert, too. Good idea!”
As you open your car door, you catch Anna’s eye and, looking at her intently, you say, “Thanks, Anna. I’m really glad we’re going this direction. You have the clout. Maybe XYZ is really good...but it would be nice to know it is instead of just believing it is.”
“You’re right, Jamie.” Anna replies. “Monterrey shouldn’t be spending hundreds of thousands a year—maybe millions!—on this if it doesn’t help kids and teachers.”
“Right, Anna!” you reply. “Talk soon! And drive safely!”