On Monday, based on conversations that the Vanguard had with key members of the community and a review of the reports by Tobin White and Scott Carrell, the Vanguard noted among other things that 331 of 492 retested through TONI (Test of Nonverbal Intelligence) had no risk factors at all.
Since then, the Vanguard has been told that those numbers are inaccurate. But the district has thus far withheld those data, citing confidentiality law regarding access to student-level data.
The Vanguard was told that a number of citizens filed a Public Records Act request asking for, among other things, access to the data provided to Tobin White and Scott Carrell. The district apparently reached some sort of understanding with the researchers to provide student data that is otherwise private to the university.
However, there are questions as to what data the researchers actually acquired. The Vanguard has been told that the researchers never talked to the former director of the AIM program, Deanne Quinn, and never had access to her data.
While the district is certainly within its legal rights to withhold private student records from the public, there are clearly ways that the district could get around such problems. The district maintained that the data was released to the researchers pursuant to a written agreement for limited research purposes.
There appears nothing to prevent the district from releasing the data more generally by simply redacting the names. It is not as though the records – detached from the names – have significant identifying information.
The issue of GATE identification is among the key issues facing the district. Critics maintain that the integrity of the program is undermined by the loose and capricious way in which retesting is administered.
The OLSAT (Otis-Lennon School Ability Test) test is used as a general means to screen GATE-identified students. About 25 percent of those who qualify do so on the OLSAT alone. However, critics point to the 27 percent identified through private testing and 49 percent through the TONI as evidence that the identification process is fatally flawed.
Tobin White noted that just three percent of students score at the 99th-percentile through the OLSAT, while 28 percent do so through TONI. He found that students administered the TONI were six times more likely to qualify than those taking only the OLSAT and nine times more likely to score in the 99th percentile.
However, defenders of the program, point out that a key part of the analysis was missed by critics – the search and serve process.
The main instrument that the district uses is the OLSAT, which attempts to assess “examinees’ ability to cope with school learning tasks, to suggest their possible placement for school learning functions, and to evaluate their achievement in relation to the talents they bring to school learning situations.”
If the student does not qualify based on the minimum OLSAT scores at the 96th percentile, there are a set of specific criteria for triggering the search and serve process for re-screening. Yesterday, we wrote that the district is not “enforcing the rule that retesting for the TONI occur with students within five standard errors of the 96th percentile.”
But if you read the Gate Master Plan, you see that this is only one criterion for retesting. As was pointed out to me, if you believe that the OLSAT is problematic for low-SES (socioeconomic status) kids and kids with other learning impairments, then you wouldn’t want simply to rely on those who are near-misses to fill the program.
Instead, there are several assessments that trigger the search and serve process. These include risk factors such as: socioeconomic status, language, health, designated special education, etc. There are also work sample assessments and parent or teacher indicators of gifted characteristics.
While Tobin White found 331 of 492 retested through TONI had no risk factors at all, defenders point out that Professor White is making conclusions based on information that was not available to him.
The district did not keep records on the kids who do not qualify for the AIM program due to legal prohibitions. Moreover, the state cannot keep records of those who do qualify without permission of the parents.
People familiar with the search and serve process believe that the number of those who retest with no risk factors is far closer to 15 to 20 percent. There are a variety reasons why the students may retest. Some may be ethnic minorities who do not have identified risk factors, but otherwise meet the criteria.
But there are other reasons as well. Sometimes the students fill out the information incorrectly on the scantron and it does not record the information. Sometimes they may skip sections of the test for no apparent reason. Finally, sometimes they are given incorrect instructions.
One of the reasons that the TONI test takers end up qualifying at a higher rate than the general population is that the retesting is based on a full assessment of the student’s abilities – look at the gap between math and verbal, their scoring on the STAR tests, teacher and parent evaluations, and entire profiles.
The argument that has been made to the Vanguard by both sides of this debate is that we need to move from the realm of anecdote to the realm of data analysis. However, that data analysis cannot take place without the access to good and reliable data.
The Vanguard is going to pursue getting the kind of data that can verify or refute the claims by the researchers who have written reports. If we wish to assess the AIM program, we should do so based on the best possible data and there are ways to protect the privacy of children while allowing the public a full range of information to scrutinize claims made on both sides.
—David M. Greenwald reporting