Analysis: SAT Assessment

RECOMMENDATION: File legislation calling for a moratorium on the use of standardized assessment until further research is done assessing its effect on the neurology of the developing human brain.

 

 

 

I was asked by the president of the District 214 Education Association to pull out information from my research on standardized assessments specific to SAT.  Below is all relevant information I have on this topic to date for anyone interested.

 

SAT ANALYSIS

SAT	ACT
·   140 questions and required essay ·   Guessing no longer hurts overall score ·   3 hours and 45 minutes ·   Total score out of 2400 ·   Math accounts for ¼ of overall score ·   10 Sections: 3 Critical Reading, 3 Math, 3 Writing, 1 Experimental	·   215 questions and optional essay (most colleges require it) ·   No guessing penalty ·   3 hours and 25 minutes ·   Total composite score of 1-36 ·   Math accounts for 1/3 of overall score ·   4 Sections: English, Math, Reading Science

*In effect fall 2014, spring 2015, fall 2015

COST

State officials claim, “The SAT scored higher than its competitor, the ACT, on a number of criteria, including cost. State officials estimate the test would cost $4.7 million a year during the three-year contract, with tests costing $33.30 per student. That’s almost $1.4 million less per year than the ACT would cost at $39.50 to $56.50 per exam, depending on whether students take a writing component.” (www.dailyherald.com/article/20151124/news/151129331/)

The above claim made by state officials indicating SAT is cheaper to implement does not delineate costs. Was this financial analysis based on a per test fee alone, or does it take into consideration the average of ALL related costs including bundled portions of this assessment such as home access for students to SAT software? In actuality, the SAT could, in fact, be a more expensive assessment to implement when all related administrative fees are specifically defined and averaged into actual cost-per-student for comparison.

GUESSING

Like the ACT, SAT tests no longer feature a penalty for guessing the wrong answer. Coming free with the SAT would be four free college applications, using the Common Application, for low-income students.

ESSAY SECTION

According to a survey of 300 colleges and universities, only a handful of them will expect applicants to submit their score from the new SAT’s essay section.

When the College Board introduced a major update to the SAT college entrance exam, it changed up the approach to the essay question and made it optional.

The essay task will ask students to read a passage between 650 and 750 words and prepare a “facts-based essay” that examines how the author built his or her argument. The essay will be scored separately. The current version of the test “simply asks students to develop a persuasive essay about an issue,” he noted in a statement. The essay score currently is combined with the multiple-choice writing section.

Kaplan questioned college admissions officers to understand how they would approach the introduction of the new generation of SAT and found that many of the top tier institutions would no longer require the essay submission. Those include, according to reporting by Bloomberg Business, Columbia University, Cornell and the University of Pennsylvania. Among the ones that will continue requiring the optional essay section are Harvard University, Yale, Princeton and Dartmouth College.

According to the survey, two-thirds of schools will remain neutral, neither requiring nor recommending it. Nineteen percent will recommend but not require it. Another 13 percent will require it; and 2 percent will only require it for specific programs.

When U Penn announced in July that it would no longer require the essay portion of the SAT or ACT, Yvonne Romero Da Silva, Penn’s director of admissions, said the decision was based on careful consideration. “Our internal analysis as well as a review of the extensive research provided by the College Board showed that the essay component of the SAT was the least predictive element of the overall writing section of the SAT. Given the impending redesign of the SAT and PSAT/NMSQT, which will make the essay portion of the assessment optional, we could no longer support requiring the essay portion of either exam given its weaker predictive power.”

Of course, Boothroyd pointed out, “One thing to consider is that an optional but more challenging section provides an opportunity for students who are good writers and analysts to distinguish themselves. Schools appreciate applicants who challenge themselves, so earning a high score on an optional section can factor favorably on an application.”
(campustechnology.com/articles/2015/10/05/sat-essay-losing-steam-among-admissions-officers.aspx)

ANALYSIS

A Lexile analyzer is available at www.lexile.com to confirm the findings below. I am using the Lexile score needed for students to read independently since Lexile scores reflect only 75% comprehension.  Students should ideally independently comprehend 100% of text in order to accurately respond to assessment questions.

SAT

 I analyzed the text of SAT Practice Tests 
(Administration Time:  Reading – 65 Minutes)

1 Test 4 Passage 1 (9-12) – 1050L (Independent Reading Level – 1150L)

1 Test 4 Passage 2 (9-12) – 950L (Independent Reading Level – 1050L)

2 Test 4 Passage 1 (13-24) – 1310L (Independent Reading Level – 1410L)

2 Test 4 Passage 2 (13-24) – 1170L (Independent Reading Level – 1270L)

3 Test 7 Questions 6-7 – 1130L Independent Reading Level – 1230L)

3 Test 7 Questions 8-9 – 1120L (Independent Reading Level – 1220L)

3 Test 7 Questions 10-15 – 1410L (Independent Reading Level – 1510L)

3 Test 7 Questions 16-24 – 1140L (Independent Reading Level – 1240L)

Practice Test 2 QUEST 1-10 Reading 1290L (Independent Reading Level – 1390L)

Practice Test 2 QUEST 22-32 Reading 1240L (Independent Reading Level – 1340L)

QUEST 1-5 Reading 1170L (Independent Reading Level – 1270L)

QUEST 6-8 Reading 1170L Independent Reading Level – 1250L)

QUEST 9-14 Reading 1120L Independent Reading Level – 1320L)

QUEST 15-19 Reading 1040L (Independent Reading Level – 1140L)

QUEST 20-24 Reading 1080L Independent Reading Level – 1180L)

Career readiness information from MetaMetrix shows the following:

LEXILES AND LIFELONG READING: Federal Tax Form – 1260L Aetna Health Care Discount Form – 1360L GM Protection Plan – 1150L Medical Insurance Benefit Package – 1280L Application for Student Loan – 1270L CD-DVD Player Instructions – 1080L Installing Child Safety Seat – 1170L Microsoft Windows User Manual – 1150L Drivers’ Manual – 1220L

READING IN THE WORKPLACE: Labor – 1000L Service – 1050L Construction – 1080L Craftsman – 1100L Clerk – 1110L Foreman – 1200L Secretary – 1250L Sales – 1270L Supervisor – 1270L Nurse – 1310L Executive – 1320L Teacher – 1340L Accountant – 1400L Scientist – 1450L

LEXILE SCORES NEEDED FOR: Education (11–12)  – 1130L Work – 1260L Community College – 1295L University – 1395L

PRIMARY SOURCES:        Bill of Rights – 1540L Declaration of Independence – 1450L Gettysburg Address – 1490L Preamble to the Constitution – 1930L Emancipation Proclamation – 2040L Magna Carta – 1740L

FINDINGS
(See Explanations and Examples)

1.  Use of excessively high text complexity with no research identifying average reading Lexile of junior level high school student
2. No research supporting SAT adequately addresses cognitive development and individual abilities of adolescents
3.  Analysis of subtle shifts in focus throughout course of entire passage, use of unfamiliar or multi-definition vocabulary (i.e. reservations)
4.  Use of figurative language is difficult for ELL students
5.  SAT claims use of context clues, but none actually provided in text
6. Dependence upon vocabulary and background knowledge rather than critical thinking skills in order to answer correctly
7. Lack of content appealing to visual and kinesthetic modalities
   The examples below contain visual details but no pictures are provided to help those students who process information better visually
8. Reliance on answer from previous question in order to answer another question (SAT representatives claimed they weren’t doing this)
9. Multi-text compare/contrast (SAT representatives claimed they weren’t doing this)
10. Reliance on information processing rather than introspective, original, creative and conceptual thought processes, which are sensory rather than memory-based, and thereby, rewiring the neural-activity of student brain through repetition of processing activities in limited and specific parts of the brain
11. Use of quantitative assessment to measure qualitative data
12. Use of facts-based essay rather than essay requiring student perspective

SAT Assessment Analysis
(Explanations and Examples)

1. Use of excessively high text complexity with no research identifying average reading Lexile of junior level high school student.

2.  No research supporting SAT adequately addresses cognitive development and individual abilities of adolescents.
There is no research stating, “fewer instances of finitely detailed, multi-process, evaluative questions will support changing cognitive needs for high school students. Scaffolding begins at simplistic level then increases in complexity. This is very different than “fewer complicated questions” versus “increased number of complicated questions.”

3.  Analysis of subtle shifts in focus throughout course of entire passage, use of unfamiliar or multi-definition vocabulary (i.e. reservations).

a.  reservations a character has about a person he has just met to a growing appreciation that character has of the person’s worth.
b.  ambivalence a character feels about his sensitive nature to the character’s recognition of the advantages of having profound emotions. 
c.  intensity of feeling a character has for another person to the character’s concern that that intensity is not reciprocated.
d.  value a character attaches to the wonders of the natural world to a rejection of that sort of beauty in favor of human artistry.

4.  Use of figurative language is difficult for ELL students.
In the context of the passage, the author’s use of the phrase “her light step flying to keep time with his long stride” (line 6) is primarily meant to convey the idea that

a.  Ethan and Mattie share a powerful enthusiasm.
b.  Mattie strives to match the speed at which Ethan works.
c.  Mattie and Ethan playfully compete with each other.
d.  Ethan walks at a pace that frustrates Mattie

5.  SAT claims use of context clues, but none are actually provided in text.

EXAMPLE 1 PASSAGE: The coming decades will likely see more intense clustering of jobs, innovation, and productivity in a smaller number of bigger cities and city-regions.

As used in line 64, “intense” most nearly means

a.  emotional
b.  concentrated
c.  brilliant
d.  determined

In order to provide context, the sentence provided in the passage should look like one of the following four types of context clues:

DEFINITION: The coming decades will likely see more intense clustering of jobs, innovation, and productivity in a smaller number of bigger cities and city-regions since there will be a more development but only in certain locations.

(This particular question is ridiculously confusing to students due to the juxtaposition of terms, “smaller” and “bigger.”)

SYNONYM/ANTONYM: The coming decades will likely see more intense clustering of jobs, innovation, and productivity in a smaller number of bigger cities and city- regions, which is no longer balanced in number as in past years.

EXAMPLE: The coming decades will likely see more intense clustering of jobs, innovation, and productivity in a smaller number of bigger cities and city-regions; for instance, they will grow on the east coast but will decrease in cities and city-regions in the west.

RESTATEMENT: The coming decades will likely see more intense, or extreme, clustering of jobs, innovation, and productivity in a smaller number of bigger cities and city-regions.

EXAMPLE 2 PASSAGE (again, no context provided): Wallace J. Nichols tracked this epic journey with a satellite tag.

As used in line 4, “tracked” most nearly means

a.  searched for
b.  traveled over
c.  followed
d.  hunted

6.  Dependence upon vocabulary and background knowledge rather than critical thinking skills in order to answer correctly.

The passage most strongly suggests that Adelita used which of the following to navigate her 9,000-mile journey?

a.  The current of the North Atlantic gyre
b.  Cues from electromagnetic coils designed by Putman and Lohmann
c.  The inclination and intensity of Earth’s magnetic field
d.  A simulated “magnetic signature” configured by Lohmann

The stance Jordan takes in the passage is best described as that of

a.  an idealist setting forth principles.
b.  an advocate seeking a compromise position.
c.  an observer striving for neutrality.
d.  a scholar researching a historical controversy.
(If I replaced words in red with familiar vocabulary, my students could answer this question.)

The main rhetorical effect of the series of three phrases beginning in line 6 (“the diminution, the subversion, the destruction”) is to

a.  convey with increasing intensity the seriousness of the threat Jordan sees to the Constitution.
b.  clarify that Jordan believes the Constitution was first weakened, then sabotaged, then broken.
c.  indicate that Jordan thinks the Constitution is prone to failure in three distinct ways.
d.  propose a three-part agenda for rescuing the Constitution from the current crisis.

7.  Lack of content appealing to visual and kinesthetic modalities.
The examples below contain visual details but no pictures are provided to help those students who process information better visually.

A map showing N, S, E, W would have been helpful!

Putman thinks that the turtles work out their position using two features of the Earth’s magnetic field that change over its surface. They can sense the field’s inclination, or the angle at which it dips towards the surface. At the poles, this angle is roughly 90 degrees and at the equator, it’s roughly zero degrees. They can also sense its intensity, which is strongest near the poles and weakest near the Equator. Different parts of the world have unique combinations of these two variables. Neither corresponds directly to either latitude or longitude, but together, they provide a “magnetic signature” that tells the turtle where it is.

It can reasonably be inferred from the passage and graphic (above) that if scientists adjusted the coils to reverse the magnetic field simulating that in the East Atlantic (Cape Verde Islands), the hatchlings would most likely swim in which direction?

8.  Reliance on answer from previous question in order to answer another question (SAT representatives claimed they weren’t doing this).

Which choice provides the best evidence for the answer to the previous question?

9.  Multi-text compare/contrast (SAT representatives claimed they weren’t doing this).

See passage for questions 20-24. This isn’t necessarily bad; it should just be noted this is a more complicated skill and text should not be overly complex since what should be measured is a student’s ability to compare varying points of view, not weed through frustrating text with which students have no familiarity.

10.  Reliance on information processing rather than introspective, original, creative and conceptual thought processes, which are sensory rather than memory-based, and thereby, rewiring the neural-activity of student brain through repetition of processing activities in limited and specific parts of the brain.

Ironically, I found the following passage in a practice SAT:

In a Science article published in early 2009, prominent developmental psychologist Patricia Greenfield reviewed more than 40 studies of the effects of various types of media on intelligence and learning ability. She concluded that, “every medium develops some cognitive skills at the expense of others.” Our growing use of the Net and other screen- based technologies, she wrote, has led to the “widespread and sophisticated development of visual-spatial skills.” But those gains go hand in hand with a weakening of our capacity for the kind of “deep processing” that underpins “mindful knowledge acquisition, inductive analysis, critical thinking, imagination, and reflection.”

11.  Use of quantitative assessment to measure qualitative data.
Standardized assessments do not measure what they claim to measure. Genius is achieved through visual, creative, conceptual-based thinking. This would be the difference between receptive processing and expressive processing. Standardized assessments do not measure expressive processing; therefore, they cannot assess student potential. What goes in is important – but what is SIGNIFICANTLY more important is what comes out.

12.  Use of facts-based essay rather than essay requiring a student perspective.

Eliminating student perspective places the onus of importance on what an author thinks rather than what a student thinks.   While it is important to understand multiple perspectives, it is more important for students to come to their own conclusions and support those conclusions based on relevant facts and examples. Since educators will teach to the test, removing the essence of student thought from standardized assessment is a dangerous path to travel. What is it to an individual what others think? What matters to the individual is what the individual thinks. Yes, thought should be based on multiple perspectives, but the most essential aspect of learning – is thinking (not regurgitating)!

REMAINING QUESTIONS
Information still needed from SAT reps:

1. Representatives said there was a validity study – I’d like to see the study.

2. Research supporting how assessment meets changing cognitive development of high school students.

3. There is no research stating, “fewer instances of finitely detailed, multi-process, evaluative questions will support changing and not yet fully developed cognitive abilities of high school students” as stated by representatives.  Scaffolding begins at simplistic level then increases in complexity.  This is very different than “fewer complicated questions” versus “increased number of complicated questions” as a means to compensate for adolescent brains that are not yet fully developed.

4. I would like to see research supporting representatives’ claim that removing a student’s point of view on written expression portion of assessment promotes increased skill in critical thinking – this is simply nonsense. Critical thinking lies in the synthesis and application of knowledge, NOT in the identification and comparison of knowledge.

5. I’d like to see a copy of the contract, and I’d like to do a cost analysis between SAT and ACT.

6. I would like to see research supporting receptive processing outweighs expressive processing when determining college/career success. What portion of SAT and ACT assessments are receptive vs. expressive?

7. SAT needs to develop score ranges and to scaffold in complexity each skill measured similar to ACT.

8. A study needs to be performed identifying the average Lexile ranges of students enrolled in advanced, average, and preparatory level English courses.

RECOMMENDATION:

A research study published in 2008 indicates the possibility that repeated and targeted brain activity to specific parts of the human brain may weaken, or eliminate the use of, other areas of the brain. The brain only has so much neural support. If the brain is trained through repetition to narrow this neural support to a specific region of the brain, then neural activity will supply less support, or perhaps no longer support, other very important areas of the brain, specifically those areas enabling students to think conceptually and creatively.

Based on these findings, my first recommendation would be to file legislation calling for a moratorium on the use of standardized assessment until this possibility is further researched.

My second recommendation, if implementation cannot be avoided, is the implementation of ACT over SAT for the following reasons.

1. I do not recommend amplifying the amount of time students are exposed to repetitious and narrow utilization of the brain’s neuroactivity. Time spent practicing skills that rely heavily on memory, organization, and planning MUST be balanced with time spent utilizing parts of the brain that foster creative and problem-solving processes, which are sensory in nature. I do not recommend students use the Home-Based package offered by SAT. Teachers should offer support in the classroom balanced with opportunities for students to employ creativity and problem-solving skills, which educators have always identified as best practice.
2. I do not recommend excluding a student’s point of view from written expression assessments. Teachers will teach to the test because it makes no sense not to do so. Again, it is not the regurgitation of information that produces original and inspirational thought. It is the synthesis and application of a student’s unique point of view that should always be a priority in education.
3. ACT offers scaffolding and score ranges and SAT does not at this point in time. Teachers do not have to reinvent the wheel if they are allowed to continue using ACT. Master teachers become master teachers because they have opportunity to perfect their work. This can never take place if they continue to start from scratch each year. Teachers were told PARCC would not cause them to reinvent what they were already doing; but in practice, this really was not the case.

FINAL NOTE:

My experience has been that students at the preparatory level in reading comprehension can score anywhere in the range of beginning reader (decoding) – 800L. Students enrolled in average level courses score between 800-1000L. Students in advanced courses have not been assessed, but they should be in order to determine the accessibility of standardized assessment for a wide range of student ability levels.

Consideration should be given to the intended purpose of standardized assessment in contrast to what its implementation actually achieves in practice.