The school district has released an updated report from UC Davis researchers Scott Carrell, et al, who conducted a study analyzing the impact of the AIM program on students in and outside of the program. Based on requests following the presentation on June 4, they have released an update which includes a discussion of the subgroup analyses that they presented at the June 4 board meeting.
The main finding of that report was a “no benefit, no-harm” finding. The researchers found “no evidence that the program positively affects achievement scores of participating students and no evidence that the program negatively affects nonparticipating students, on average.”
They add, “This ‘no benefit, no-harm’ finding should be considered within the context of the program’s cost: the DJUSD spends considerable resources on universal testing and retesting (as do private citizens for additional testing) and given the financial costs and capacity constraints associated with this program, we should expect these costs to be balanced by some measurable benefit.”
They add that the analysis does “not allow us to make explicit recommendations about what an ideal gifted and talented program in the DJUSD should look like. We can conclude, however, that clearer identification of ‘gifted’ children will increase program transparency.”
While the main bulk of their analysis found that there was “no benefit, no-harm” – presented at the June 4 meeting was a subgroup analysis that found one big exception being with Hispanic students.
On page 8 of the revised report, they write:
We also estimate the effects of the AIM program on students not in AIM, for the different subgroups described in Section 4c. Using the same estimation strategy that we use for the full sample in Section 4d, and across different outcomes and specifications, we find no consistent evidence that the AIM program affects any of these subgroups, save for Hispanic students. For Hispanic students, we consistently find that AIM negatively impacts both math and ELA scores. Specifically, we find that AIM reduces the math CST scores of Hispanic students who are not in the AIM program by 21 points and the ELA CST scores of Hispanic students not in the AIM program by 15 points.
That leads to a big question as to why would Hispanic students be negatively impacted by AIM, and why only them?
The Vanguard is hoping to sit down with the researchers to clarify a number of things. However, we have a number of methodological questions that need to be resolved.
As we noted in a previous analysis (Sunday Commentary II: Is the GATE Program In Need of Change?, July 12, 2015), the researchers had to grapple with some methodological issues in order to attempt to create a natural control experiment in order “to disentangle the actual effect of a program from the effect of different types of individuals choosing to participate or not participate in a program.”
They ultimately settle on a regression discontinuity research design: “However, because of how students qualify for AIM in the DJUSD, certain parts of the qualification process are effectively random. It is this randomization that we exploit to estimate the effect of the AIM program on students in the program. This approach is called a regression discontinuity research design (RDD).”
They measure those who just qualified for the AIM program against those who just missed qualification based on their OLSAT score. Given questions about the OLSAT, that test may lack a lot of validity. We brought up a number of concerns about this approach back in July, which should be considered.
However, in the end they find, “Students just to the right of the qualification threshold score, on average, 3 points higher on their ELA CST [English-language Arts California Standards Tests] and 6 points lower on their math CST than students just to the left of the qualification threshold. These effects are small and statistically indistinguishable from zero. As such, there is no evidence that the AIM program has an effect on students in the AIM program.”
The dependent variable in this study was the score on the California Standards Test (CST) or STAR (Standardized Testing and Reporting) Test. In our discussions with a number of experts, there were serious questions raised as to whether enrollment in the GATE/AIM program is going to have any effect on the student’s score on the CST.
Whether that objection is valid or not, that would seem to be something that the researchers need to establish. If their dependent variable is not impacted by the program, then of course we would expect no effect. The fact that they in fact find very small statistical effects further casts doubt on the validity of their measure – but this problem is never scrutinized in their report.
It is important to note that the state has in fact stopped using the STAR as of July 1, 2013. Part of the reason for that change was the belief the STAR test was no longer a proper measure of student growth. The STAR test, which was for the most part multiple choice questions, was scrapped in favor of more rigorous, thought-provoking exams.
“Multiple-choice, fill-in-the-bubble tests alone simply cannot do the job anymore, and it’s time for California to move forward with assessments that measure the real-world skills our students need to be ready for a career and for college,” said State Superintendent of Public Instruction Tom Torlakson in January 2013.
In addition to questions about the use of CST scores for the dependent variable measuring student performance and the effectiveness of the AIM program, we have questions as to why Hispanics and Hispanics alone are negatively impacted by AIM. We have only the data analysis presented by the researchers with no operationalized theory to guide us.
This addition prompts more questions – questions that the Vanguard hopes to answer in the coming weeks.
—David M. Greenwald reporting