While there were a number of procedural issues asked about the presentations on Tuesday night regarding the district AIM program, much of the hard questions focused particularly on the identification and testing issue.
Board Member Madhavi Sunder noted that the importance of process is not only to ensure equal access, but to make sure that the board receives information that is vetted.
However, Board Member Susan Lovenburg said that they saw nothing unusual about the process and Board President Alan Fernandes said that the agenda item followed board policies that allowed for items to be placed on the agenda by board members.
The board heard presentations from two UC Davis professors, presenting their findings regarding AIM assessment and identification data.
Madhavi Sunder pointed out that the two tests serve very different populations, with the OLSAT given to the general population and thus obtaining a smaller percentage of qualifying scores. The TONI serves two populations – the first are those who scored within five points on the OLSAT and the second is the “search and serve” group ‒ whether they have a huge discrepancy in scores or they needed more time.
Tobin White, an Associate Professor at the UC Davis School of Education, will discuss his assessment of current testing practices at DJUSD. Currently they use two exams, the OLSAT (Otis-Lennon School Ability Test) and the TONI (Test of Nonverbal Intelligence) to identify students eligible for the AIM program.
According to Professor White, the OLSAT identifies about 24% of the AIM students, the TONI 49% of AIM students, and 27% are identified through some other test.
The TONI is by design more inclusive, as it is used to retest students who come within five points of qualifying for enrollment on the OLSAT test. Therefore, it is not completely surprising that students administered the TONI were “six times more likely to qualify than those taking only the OLSAT.” They were also nine times more likely, according to Professor White, to score in the 99th percentile.
He writes, “These are radically different measures, yet they are being treated as equivalent in program placement decisions.”
He adds, citing research, “The TONI was not designed to replace broad-based intelligence tests but rather to provide an alternative method of assessment when a subject’s cognitive, language, or motor impairments rendered traditional tests of intelligence inappropriate and ineffectual.”
The TONI was designed specifically for populations of deaf and hearing impaired, those with aphasia, dyslexia and other disorders related to spoken and written language, and those not proficient with written and spoken English.
The TONI is also the more ethnically diverse testing protocol. Currently the District’s AIM program is 60% White, 19% Hispanic, 16% Asian, and 3% Black. The OLSAT generates 48% White qualifiers, 44% Asian, 6% Latino and 1% Black. The TONI is more ethnically diverse with 43% white, 31% Latino, 12% Asian, and 8% Black.
“Generally there is some reason to believe that these particular students might have a false negative on the OLSAT,” she said. “They may actually have some gifted potential that was not apparent on the OLSAT.”
Professor Tobin White noted that it’s really one-third of the population that is affected. However, the purpose of the TONI is to identify students where language is an issue and where a verbal test might not be the best measure. “To get up to a third seems like a pretty big jump,” he said.
Professor White concludes that “the current re-screening process is fundamentally flawed.” The “TONI and OLSAT have dramatically different psychometric properties, one score should not simply replace the other.”
He noted, “The TONI is designed to address only a limited range of the search and serve criteria, yet is being applied for all of them.”
He concludes, “The TONI is clearly identifying a more diverse group of students than the OLSAT, but as a function of student selection rather than test bias.”
Madhavi Sunder noted that the TONI doesn’t just look at language learners, but all categories of disadvantage, like students who didn’t go to preschool and other such disadvantages like family history and student background.
“When you come from a low socio-economic background and you’re not necessarily going to have all of the those enrichment opportunities around language, and it’s precisely those kids who we think are at a disadvantage on the (OLSAT), there’s a lot of language that’s required to do that test,” she said. “That’s really the heart, are we testing the wrong kids with the TONI?” She argues, “We should be using these non-verbal tests for a larger population.”
Ms. Sunder argued, “Fairness isn’t how many tests each person gets to take, but is every kid getting the opportunity to take the appropriate test for them.” She argued we need to be using the practices for the method of test that we are offering.
Board Member Susan Lovenburg said, “The big conclusion is there is no perfect test.” She said, “What is concerning here is that one test is being thrown out in favor of a second test for some kids. That’s the kind of problematic element of what’s happening here.”
She wants data and evidence to look at as we make our decisions. She said, “Particularly in the world of education research, you can find research to support almost any decision as long as your going to ignore the research that conflicts with it.”
Ms. Lovenburg added, “Ultimately the question that is being asked is are we appropriately identifying the students that need these special services.” She said, “I’m not in favor of dismantling the self-contained GATE program. I don’t speak for any of my colleagues, they can speak for themselves. But I do see that value in the program for students who truly need the services.”
However, she stated, “I do have some significant concerns that we’re over-identifying students into the program… We need to address that as a district.”
Board Member Barbara Archer said, “When I look at this data, I see that 50 percent of the students in this dataset qualified through the TONI for the AIM program. That caused me to wonder, well what were the reasons?”
She said she wanted to understand the reasons why students were being re-screened through the TONI. “What I was told is that we don’t keep track of the reason why 50 percent of our students qualify using the TONI, and that’s concerning to me.”
“The other point of concern is that it seems to me statistically impossible that 28 percent of students could score in the 99th percentile on a test – that doesn’t make sense to me,” she said.
Ms. Archer said, “When you see that 50 percent of students were being qualified with the TONI, and you realize it is possible – although we don’t know because we don’t have the data – that you’re lumping together margin of error kids, ELL kids, low SES, and the various other reasons, that seems to me a suspect metric because you’re lumping so many different groups into a non-verbal category.”
She added, “It seems like the TONI is advantaging everyone.”
Board Member Tom Adams said, “The overall test results would raise questions.” He noted that issues of reliability and validity depend on what the test is trying to measure. In this case the question is whether you are GATE, and in need of GATE services. “The question is, can you really swap out – as has the district done – the identification that occurs under OLSAT with TONI?”
“Do these tests really measure the same thing?” he asked. “That’s a test that I really don’t think has been addressed, because I don’t think they measure the same thing.”
He acknowledged the need for more than one test, particularly with people who lack the language skills for the OLSAT. “In this sense, it’s not that anybody’s against GATE identification, it’s not that anybody is trying to prevent anybody from trying to get into the program, the real question is… what is the best measure of those kids in identifying them in terms of them getting these services that they need.”
Susan Lovenburg would note that we lack the resources to throw a full range of tests at every student. Tom Adams asked whether we using this in a manner that reflects best practices.
Finally, Madhavi Sunder asked, what is the alternative to the current way of evaluating AIM identifiers?
The challenge that the district faces is that, while it is easy to argue that the TONI perhaps sets too low a bar, with the OLSAT alone, GATE would be 92% White and Asian with only 8% Black, Latino, and American Indian.
These are challenges not easily resolved and an issue that tends to create sharply held beliefs and divisions.
—David M. Greenwald reporting