Fund a Review of, and Update, the National Reporting System

Date: July 27, 2010

. . . to ensure that required outcomes measures are appropriate for all levels of adult education students.

—An NCL WIA Reauthorization Priority

Article by Peggy McGuire

How might the National Reporting System (NRS) be reformed in order to best serve the needs of adult learners? Whether or not WIA is reauthorized this year, we should continue to move NRS reform forward. Thoughtful dialogue and recommendations are needed and welcomed from YOU! Just enter your comments and recommendations for NRS reform in the Comment box below. Your contributions will also show policy makers that there is significant interest in this issue.

NCL will share results of these guest blog conversations on WIA reauthorization with legislative staffs during National Adult Education and Family Literacy Week.

To help the conversation along, we have framed some of the key issues regarding the functionality and fairness of the NRS as it is currently being implemented:

Issue: Educational Functioning Levels

Using current educational functioning levels to quantifiably measure educational gains is too rigid and does not allow the system to accurately count statistically significant learning gains, particularly at the lower end of the basic skill levels. This makes it difficult for programs to demonstrate progress with lower level learners specifically, who most often cannot make two years progress in one year in order to be counted as completers, and may provide a disincentive to serve them. We also know from experience that middle level learners tend to plateau, making it difficult to show gains using the current EFLs. The system needs to be overhauled in order to be fair, in particular, with respect to low and mid-level learners.

Issue: Currently Approved Standardized Tests

The approved standardized tests do not very effectively assess at the lowest end of the lowest skills levels. Learners demonstrate more variation within categories than the test actually reveals. Progress is being made by these learners but their progress is not being measured; the categories are too broad to accurately reflect their progress. Additionally, the difficulty with test administration, the need for improved pre/post test protocols and guidance, and the limited number of approved tests are affecting the validity of assessment results at all levels.

Issue: Performance Assessment and Project Learners

Adults come to adult education for assistance with a variety of goals, including the need to accomplish specific learning projects. The immigrant worker in the mall wants to read the mall map placard so he can direct shoppers. Parents want to support their children’s learning. A forklift driver wants to be able to read her manifest. A senior adult wants help with personal finance mathematics. It is not practical to capture their learning using the pre/post standardized tests currently approved in the NRS.

The NRS can discourage local programs from serving these important adult learning goals. Yet in spite of this NRS limitation, local programs do continue to serve these adult learners and accept that the reporting system will not give them credit for this work.

In 2002, the National Academies of Science, Board on Testing and Assessment (BOTA) published their report entitled Performance Assessments for Adult Education: Exploring the Measurement Issues.

The report was the culmination of BOTA’s discussions with researchers and practitioners regarding adult education assessment issues including alignment of performance assessment tasks to specific learning goals. A recurring comment warned that the individual states did not have the funds to develop such performance assessments.

Issue: Tracking Adult Learners Over Longer Periods of Time and Collecting Longitudinal Data In order to demonstrate the impact of adult basic education and adult English language acquisition, we need a better understanding of what happens to the adult learner in the longer term. Using existing data, we do not know what happens to adult learners when they leave or complete adult education. In practice, it is challenging for local programs to track adult learners both while they are in the program and after they leave; the burden for collecting longitudinal data on adult learners cannot be placed on the provider, but instead should be placed on the state and federal agencies. Many states are also prohibited by State law or State policy from using Social Security numbers to track education, training, and work outcomes. A means for collecting these data needs to be identified and used.

Issue: Tracking Educational Gains and Outcomes of All Students
Currently, the NRS only requires providers to report on a student’s progress toward their individually-determined goal and to report academic learning gains, as this is the default measure for students who have not specified a goal. Therefore, all other core measures are tracked only for students who report a goal related to the measure. Transitions from adult education into employment or postsecondary education, for example, are only measured for those who report having such goals when they enter an adult education program.

The current way of reporting limits the ability of states to demonstrate the real impact of their adult education system, since students may make educational gains or achieve outcomes that they did not originally set as goals. Since few students enter adult education seeing themselves as college material, the number of students for whom transitions are measured is quite small and does not reflect the positive impact that adult education programs can have on student aspirations. In addition, the current reporting method ignores the fact that many adult education students enter programs with multiple goals, some of which will take longer to realize. The current way of measuring outcomes also makes it difficult to compare state performance since some states may be less ambitious than others in working with students on setting higher goals.

So—what do YOU have to say about any or all of these key issues related to NRS reform? What issues do you see that may not be described above? We are eager to hear your thoughts, experiences, and recommendations over the next few weeks. Please share them with your colleagues by entering them in the Comments box below, and let the conversation begin! All the best!

Resources:

Educational Functioning Level Descriptors—Outcome Measures Definitions

NRS Test Benchmarks for Educational Functioning Levels

» Why the NCL Priorities for Reauthorizing WIA Are Important: Views From the Field

» Tips for Using Blog Discussions to Advance WIA Reauthorization Issues

» What are YOUR Priorities for Reauthorizing the Workforce Investment Act?

» Use A Consistent Definition of Adult Education and Allowable Local Adult Education Activities

Tags: NRS reform, reauthorization, WIA

Comments RSS TrackBack 9 comments

Regina Suitt

in July 28th, 2010 @ 18:00

Programs are being held accountable by NRS, in the same way as “No Child Left Behind” is holding K-12 accountable. But I think there are at least a few ways to improve the system and truly track the gains and goals adult ed students make:
1.) Allow “passing the Official GED exam” to count for anyone, but especially ABE III’s and high level ESOL students. I’ve seen countless times where a student who isn’t an ASE II gets their GED, but it doesn’t help our program achieve its gain outcomes – it’s too bad and very demoralizing to teachers.
2.) Many students come in to labs or shorter length classes for one or two specific skills. It’s hard to explain to them why they have to take a lengthy standardized test that may take several hours when all they really want to do is brush up on the essay for the GED or take a driver license test. The arbitrary 60 hours between tests inhibits gains for these students.
3.) Give teachers the flexibility to decide upon the length of time between tests. I understand a guideline or recommendation is helpful, but I’ve seen self paced lab students zoom through lessons in 10 or 12 hours… do a truck load of homework at home…. And be ready to make a significant gain on a test. There should be the ability to test before 60 hours without documenting why so much.
4.) Tracking students after they’ve left is too much work for a program that is trying to keep up with the current needs of students. There has to be a way for a state or national agency to handle that task.
5.) The 2+ year level increase is too much to ask for in the short time that most adult education students step in and out of the program. With jobs, families, transportation and other real challenges, why do they have to do so much more than in K-12 and for far less money.
6.) Nearly 90% of our 5,000+ students are below the 9th grade level in one, two, or all three areas. They need more than one year to catch up, stay caught up, and then plan their occupational goals. They should not hurt a program because they can’t make that 2+ level gain in less than a year.

[Reply]
Adam Hostetter

in July 28th, 2010 @ 18:13

Peggy McGuire says, “[Our current measure for an educational gain] makes it difficult for programs to demonstrate progress with lower level learners specifically, who most often cannot make two years progress in one year in order to be counted as completers, and may provide a disincentive [for programs] to serve them.” I absolutely agree. To expect adults (at any level) to leap over 2-grade levels is quite a challenge. Considering what the research tells us about persistence and student perceptions of persistence, success on a standardized test within certain date sensitive time limits is very ill-suited to the reality of adult learners. When we look at our data for test scores increasing but NOT meeting the gains threshold, our students accomplishments are outstanding–often, over 80% show increases, while about 50% show “gains.” The true adult literacy learner does not tend to show educational gains by the NRS definition. Other measures are critical to truly gauge student success.

Regarding “Performance Assessment and Project Learners,” This is another critically important issue in Adult Education. We worked with students to develop a comprehensive definition of “student achievement” from their view and found exactly what Peggy McGuire reports here–goals are diverse. We found that students most often measure their own achievement based on these categories: Persistence, Ownership of Learning, Acquisition of Relevant Knowledge and Skills, Increased Confidence and Self-Esteem, Self and Family Advocacy, Leadership and Responsible Civic Engagement. It is important for our programs to be able to measure student achievement in a broader way, which more suits what adults aim to achieve by coming to our programs.

[Reply]
Peggy McGuire Reply:
July 30th, 2010 at 9:08 am
Thank you, Regina and Adam, for your thoughtful comments. Whatever the issues with the practicality, transparency and fairness of the current National Reporting System, it sounds like you share my fundamental belief that accountability is indeed a critical aspect of the work we do. Once we are provided with adequate tools, resources and support to do our jobs, I think we ought to then be accountable for the results. And the best way to be accountable to all of our constituents — students, funders, community stakeholders, etc. — is to align our reporting system to the authentic needs and goals of the adult learners who seek our services. In my mind, a reformed and effective NRS would fully achieve that alignment.

I see in your postings some concrete ideas for moving in that direction: counting all significant outcomes for all learners; allowing learners and the teachers who work with them to decide when/how soon/how often testing is appropriate; and figure out a process by which some entity other than the local program can take on the responsibility of tracking longer-term outcomes after students leave the program.

Further, you both discuss the difficulties posed by the current required level gains — how unrealistic the requirements are for some students, and how frustrating it is especially when programs can see and document increases that don’t meet the NRS-established thresholds for “gain”. It makes sense to me to try to imagine other measures that would provide a fuller and more authentic picture of what student achievement looks like in our programs — either replacing, or at least explicating and enriching, the information we currently provide about level gains (even those that don’t meet the current thresholds). You have provided some intriguing proposals fo new categories of “achievement” that might be measured and reported in a reformed NRS. They bear some consideration.

So thanks again for sharing your thoughts. All the best!
Jackie Taylor

in August 3rd, 2010 @ 13:49

Hello Everyone,

The National Coalition for Literacy just completed a new set of broad recommendations for NRS reform and it can be found by visiting: http://blog.ncladvocacy.org/wp-content/uploads/2010/08/NRS_Recommendations_FINAL.pdf

These recommendations are a first step in advocating for NRS reform and will be useful as we continue to specify recommendations that could be implemented within the next year and beyond.

Regina, I don’t think your suggestion for changing the 60 hour requirement is on this list — we were too far along in our adoption process with this document when we learned of it. However, I will present it at our next NRS reform meeting where we will discuss further recommendations, strategies, and tactics for advancing NRS reform.

Please keep your ideas coming; let’s brainstorm how we can effectively help move NRS reform forward.

Thanks, Jackie Taylor

[Reply]
Rick Burgin

in August 3rd, 2010 @ 18:35

First of all, thank you for allowing us to have these discussions through this format…it’s interesting to see how other centers operate, and how they perceive things.

I am writing to voice my concerns over the required minimum number of instructional hours a student must put in before being allowed to posttest. We use the TABE, therefore our students are required to put in 60 hours.

Many of our students are in situations where obtaining a GED is a priority, and to hold them back just so they can obtain the number of hours required by the federal government is ludicrous. Consider a few of the examples of students we are currently serving at our ABE/GED Center:
● Unemployed, seeking a job which requires a GED
● Homeless
● Court-ordered
● Entering military

Our students don’t all fit into the square box that the folks who are making these rules would like to put them. There are reasons why students left school to begin with; replicating a situation they were previously in (i.e., traditional high schools where they failed in the first place) does no good for anyone. Requiring someone to put in seat time does no good for anyone.

Educational gain has little to do with the number of hours a student spends in a program. Students should be evaluated on a case-by-case basis to determine whether or not they are ready to posttest. Instructors who have worked with students know when their students are ready. Our own ABE/GED Center instructors have a combined total of 92 years of teaching experience in ABE. Teacher-made tests and chapter reviews, not seat time, tell instructors when a student has mastered the material and is ready to move on and posttest.

Many factors determine student progression rates: one-to-one or small groups vs. classroom instruction; quality and intensity of instruction; intellectual capability – some students learn more quickly than others; and motivation, just to name a few.

Some students can move from one Educational Functioning Level (EFL) to the next in 15 hours, while others might take 1,500 hours. A student who pretests with the TABE at an NRS Low Adult Secondary Education level (NRS Level 5) with a scaled score of 594 in math (10.9 grade equivalent using the TABE) only needs to increase their scaled score by 1 point in order to be in the next level (595 = 11.0 grade equivalent). Do we really need to hold this student back for 60 hours before we posttest him? Chances are pretty good that he can increase his score to a 12.9 (much less an 11.0 grade equivalent) in much less than 60 hours. A student who pretests (TABE) at an NRS Beginning Basic Education level with a scaled score of 368 in reading could take hundreds of hours to bump to the next level. In my experience, students with difficulties in math can progress very quickly from one level to the next. Students with difficulties in reading tend to take much longer. The policy of 60 instructional hours before being allowed to posttest takes none of these factors into consideration.

What curriculum is used to determine that 60 (or 50 or 30) hours is the number at which students can move from one NRS level to the next? We would like to see the research that proves a student must have 60 hours of instruction before showing educational gains; our center’s federal tables on educational gain show otherwise! We have been operating for years with no minimum hours required between a pre- and a posttest. We also exceed the state’s federal Negotiated Performance Levels for both ABE and Adult Secondary Education (ASE).

A research article, “Federal Accountability System Fuels the Decline of the Adult Education & Literacy System of the United States” written in 2003 by Thomas G. Sticht, an International Consultant in Adult Education, compares attendance and progress of Wyoming ABE programs with Florida ABE programs. After reading the article, we certainly question why OVAE would mandate all students to put in the same number of instructional hours before being allowed to post-test

Some centers offer very limited hours of instruction during a program year. Many centers are unable to offer 60 hours of classroom instruction, even during a full program year. As you may know, Wyoming is a very rural state; many, if not most, sites instruct on a part-time basis. ABE/GED instruction is offered at detention and treatment centers, again on a very part-time basis. How will students be able to access 60 hours of instruction when it isn’t even available to them during a year’s time?

Every center in Wyoming has experienced federal and state budget cuts this year. How can we offer any more instruction while we are already understaffed? Our facilities are bursting at the seams with students. Where are we supposed to put students who need to attend all those extra hours?

OVAE states that centers must follow test publisher guidelines. CTB McGraw-Hill, the publisher of TABE, states on its website (under Frequently Asked Questions on the TABE): “There is no recommended minimum number of hours of instruction that must occur if a different form or different level of the TABE pre-test is administered as a post-test.” We have operated under these guidelines for years, and have yet to see a revised TABE publisher’s manual that states post-testing should occur only after 60 hours. OVAE, however, insists that CTB McGraw-Hill requires 60 hours before students be allowed to posttest.

It should be noted that Wyoming’s assessment policy allows for a limited number of students to posttest before 60 hours of instruction; however, these exceptions are only allowable for 20% of the students whose pretest places them at one of the 2 highest Educational Functioning Levels (i.e., ASE Low and ASE High). ABE students (EFL levels 1 – 4) are never allowed to posttest before 60 hours (according to the policy).

Our ABE/GED Center compared statistics for Table 4b (which tallies educational gain for students who have been both pre-and post-tested), on students who had more than 12 hours of attendance with students who had more than 60 hours of attendance. Students who were in the program for 60 or more hours made less educational gain than students who had less than 60 hours. Generally, students who spend more than 60 hours in the program are the students who take longer to progress (i.e., students with learning disabilities, slower learners, students with ADHD, etc.). In 2008-2009, our center had 328 ABE/GED students who were both pre- and post-tested. There were only 33 students who had greater than 60 hours. A total of 173 of our ABE/GED students earned their GED in 2008-2009. Of the 173 GED graduates: 165 (95%) earned a GED with under 60 hours of instruction; only 8 (5%) earned a GED with 60 or more hours of instruction.

We set high standards for our students, but there are times when it is not realistic for students to spend 60 hours in a program. Seat time does not put food on the table! The policy requiring all students to put in the same number of hours before being post-tested needs to be eliminated.

Thanks again for listening.

[Reply]
Peggy McGuire Reply:
August 4th, 2010 at 9:21 am
Thanks so much, Rick, for your cogent and well-supported arguments. I hear you echoing the sentiments of earlier “commenters” regarding mandated instructional hours before testing, and I appreciate how you frame those sentiments with a lot of solid information.

As I considered your response I found myself asking what I’m fairly sure the framers of any improved NRS will be asking: how do we establish a policy 1) that makes intuitive good sense in terms of the real lives, needs and learning purposes of the adults we serve, 2) that (apparently) depends in large measure on students and their teachers being able to make well-informed decisions about the students’ readiness to take tests, and 3) can achieve (1) and (2) while still promoting fair, consistent and accurate reporting across states and programs? What would such a policy look like “on the ground”?

I suppose my “bigger picture” question would be this: Is it possible for an accountability reporting system to be this flexible and “national” at the same time? I want to believe that the answer is a resounding “yes!”. But I also think we need to provide some fairly concrete answers to those 3 questions I posed above (and maybe other important questions as well?) in order to support our contention. I think our field has plenty of smart, creative and committed practitioners who could bring their grounded experience and innovative thinking to bear on the task! Perhaps this period of activity related to improving the current NRS is the opportunity to get that process started.

Many thanks for your insights. All the best!
Tara Waln-Lewellyn

in August 6th, 2010 @ 13:05

I agree with the argument regarding the first issue, which concerns Educational Functioning Levels. In addition to the aforementioned limitations by Ms. Suitt, I believe we also need to recognize that there are often organic deficits that students may have that will influence their initial EFL and subsequent movement. These deficits may include learning differences, learning difficulties, learning disabilities, mental health issues, developmental disabilities and substance abuse or addiction.
As is typical of Adult Education, my program’s largest challenges are time and funding. The program in which I am employed is in the state of Nebraska. The State of Nebraska Department of Education requires the following minimum number of hours of instruction per K-12 school year:
Kindergarten: minimum 400 hrs
1st-8th Grades: minimum 1032 hrs
9th-12th Grades: minimum 1080 hrs
Our program offers six hours of class time per week for 44 weeks during the fiscal year. Totaled, we offer only 264 hours of class time and direct instruction to our students, noting that students may enter the program at various educational levels during the year.

http://blog.ncladvocacy.org/wp-content/uploads/2010/08/NRS_LEVELvsK12Chart.pdf

Although this is simply a snapshot of one program, I am confident that other programs in the nation face similar challenges.

[Reply]
Jeff Carter

in August 20th, 2010 @ 09:50

A few weeks ago, the Office of Vocational and Adult Education (OVAE) at the U.S. Department of Education proposed several potential reforms to the NRS and met with state directors of adult education to discuss them. I was invited to attend the meeting as an observer, representing ProLiteracy, and I thought it might be useful to share my notes from that meeting, which I’ve posted here:

http://blog.ncladvocacy.org/wp-content/uploads/2010/08/NotesonAug2010NRSmtg.pdf

In these notes I review the six issues and recommendations first, including my take on the reaction to these recommendations, and then add some general observations about the proceedings.

[Reply]
Jackie Taylor

in November 11th, 2010 @ 12:01

Hello All:

Here is an update on NRS reform:

In August 2010, the National Coalition for Literacy developed general policy positions for NRS reform that will guide NCL’s advocacy efforts and it is here:

http://ncladvocacy.org/NRS_Recommendations_FINAL.pdf

Since that time, NCL developed a response to OVAE’s recommendations for NRS reform. This response can be found here:

http://ncladvocacy.org/NCL_Response_to_OVAE_NRS_Recommendations/NCL_Response_to_OVAE_NRS_Recommendations.pdf

Because NCL’s positions are developed through consensus of its Members and affirmed via Member vote, all 24 national organizations that are Members of NCL (http://www.national-coalition-literacy.org/membership.html) support these positions.

Thanks to everyone who worked on them! Lets keep pushing the issue forward.

Jackie Taylor

[Reply]

National Coalition for Literacy

Get Alerts and Updates

Polls

Facebook

Categories

Links

Meta

Fund a Review of, and Update, the National Reporting System

Leave a comment

Comments RSS TrackBack 9 comments

Regina Suitt

Adam Hostetter

Jackie Taylor

Rick Burgin

Tara Waln-Lewellyn

Jeff Carter

Jackie Taylor