Background...

All students enter school with a combination of "headwinds" and "tailwinds". Tailwinds are the things that make school easier for students. Tailwinds may include factors such as coming from a home with parents of high education levels and economic stability, being a native English speaker, not having a disability, or being a member of the cultural majority. Each of those characteristics plays a role in helping a student experience success in school.

Headwinds on the other hand make school more difficult. Headwinds can include having economic instability at home, parents with lower levels of education, having a disability, or still learning English. The more headwinds a student has, the more difficulty they will have in maximizing their academic potential and the more “tailwinds” they will need. Tailwinds come in the form of high-quality instruction, support, and intervention.

The Academic Support Index, or ASI, quantifies these headwinds. A student’s ASI is the sum of their headwinds. Their ASI can also be considered a measure of the amount of support that they will need in order to mitigate the impact of those educational headwinds. Students with a low ASI will likely need very little additional support outside of Tier 1 instruction. Higher ASI students will likely need proportionally higher amounts of Tier 2 and sometimes Tier 3 supports.

There is a strong relationship between the ASI and academic outcomes including assessments such as the SAT, Smarter Balanced Assessments, AP and IB tests, kindergarten screeners, grade point averages, rates of college eligibility, matriculation, and degree attainment. We have studied these effects over seven years of data as well as across urban, suburban, and rural schools. To date over 400,000 students have been scored on the ASI. (See the featured post below for a list of papers and presentations on the ASI).

Because the ASI is able to reliably predict student outcomes you have to opportunity to interrupt that predictability by using the ASI to make sure that you are identifying the right students for early intervention and support. With effective intervention, predictive analytics can become preventive analytics.

Wednesday, March 2, 2016

LCAP: Which is better at identifying potentially underperforming students: ASI 3+ or "Unduplicated"?

I was wondering: How different are the categories of ASI 3 and higher and the LCAP's "Unduplicated students" and how well do they predict academic difficulties?



Before discussing any analysis I think it is important to point out that the category of "Unduplicated" is a binary grouping: either you are at-risk for academic underperformance or you aren't. The ASI is fundamentally different in that it provides a score indicating the degree to which a student is likely to struggle.  In that sense the ASI can be much more useful to schools in designing and assigning students to appropriate supports and interventions.

In order to compare the two I used the category of ASI 3+ vs. Unduplicated.  In prior analysis students with "headwinds" scoring 3+ consistently differentiate in their performance from those scoring two or less.


To begin with I look at how many students each group identified. ASI 3+ consisted of 44% of the student population. Unduplicated made up 39% of the population.  33% of the students were identified in both screens.  Clearly, the ASI 3+ screen is identifying more students but is it doing a more efficient job of identifying those students who might underperform?

The four questions that I hoped to answer were:

  1. Which screen was better for identifying students who scored "Not Meeting Standards" on their English and Math Smarter Balanced Assessments? (I chose the SBA because even with its flaws it is an objective measure of student performance.)
  2. Which screen had lower levels of false positives?  For this I looked at students in both groups who not only did they not underperform academically, but actually met the "Exceeded Standards" level on the SBA.
  3. Which screen had the lower level of false negatives?  In other words, which screening method failed to identify students who ultimately did score in the "Not Meeting Standards" range on the SBA.
  4. Bottom line:  Of those students who scored "Not Meeting Standards" on the SBA, which did better at identifying them in advance:  ASI 3+ or Unduplicated?
Here are my conclusions: 
  1. The ASI 3+ screen was better able to identify students who would underperform on both Math and English Language Arts across almost all grade levels.
    • Advantage ASI
  2. The ASI 3+ screen was less likely to identify students as at-risk as it had lower rates of false positives, i.e. students who were identified who scored Exceeding Standards.
    • Advantage ASI
  3. The ASI 3+ screen was less likely to omit students who would score "Not Meeting Standards" than the Unduplicated group.
    • Advantage ASI
  4. For those students who scored "Not Meeting Standards" on math the ASI 3+ group identified 87% of them in advance. The Unduplicated identified 73%.  Win for ASI 3+.  For the English Language Arts assessment, ASI 3+ identified 88% vs. 76% for Unduplicated. 
    • Advantage ASI

-------------------------------------------------------------------------------------------------------------------------
For those that really like to get into the weeds I have included some data below.  Green indicates a better performance, dusty rose means a worse performance, and orange means a tie. Keep in mind when looking at these that not all categories of SBA performance are included so the numbers will not add up to 100%.  Both ASI 3+ and Unduplicated identified many students who performed above the Not Meeting Standards threshold. I've only addressed how well they identified (and misidentified) student at the highest and lowest performing levels.



Looking at 95% Confidence Intervals for student performance on the 8th grade assessment you will see that the performance between ASI 3+ and Unduplicated is statistically the same for both math and ELA.  








Posted by