Dispelling the Myth of “New Is Better”
In an era where data-driven instruction is more critical than ever, the quality of the assessments informing that data must be undeniably sound. For K-12 school district leaders seeking to support instructional improvement and equitable student outcomes, the choice between using field-tested items versus newly written items on benchmark assessments can have significant consequences.
At first glance, the promise of “fresh” test items each year may sound appealing. It suggests improved test security and reduced risk of item overexposure. Some assessment providers emphasize the benefit of using newly developed items each year, claiming that new items allow teachers to retain access to completed assessments after student results are delivered. However, behind the scenes of effective assessment lies a rigorous process designed to ensure accuracy, fairness, and instructional utility—starting with item field-testing.
This article examines why field-tested items consistently lead to more accurate, reliable, and equitable benchmark assessments than newly created, untested items. It also examines common misconceptions about new item development and offers evidence-backed reasons for prioritizing field-tested content in benchmark systems.
What Is a Field-Tested Item?
Field-tested items are questions that have been previously administered to students without contributing to their official score, and whose performance has been analyzed for quality and alignment. These items undergo statistical scrutiny, including evaluations of difficulty, discrimination, and fairness. Only after meeting performance standards are they considered suitable for operational use in benchmark or summative assessments.
In contrast, newly-created items, no matter how well-written, are essentially unproven. Without empirical evidence of how students respond to them, educators and district leaders are taking a risk each time these unvetted items are used to drive instructional decisions or measure student growth.
The Case for Field-Tested Items in Benchmark Assessments
1. Improved Accuracy and Validity
The primary goal of any assessment is to accurately measure what it intends to measure. Without field testing, item writers and content reviewers are left relying solely on expert judgment, which, while valuable, does not guarantee that the item will perform well in a real-world testing environment.
Field testing provides empirical data on how an item functions. It reveals whether an item is too easy or too difficult, whether it discriminates effectively between high- and low-performing students, and whether it introduces unintended distractions or factors that don’t relate to the skill being measured (known as construct-irrelevant variance).1
Field-tested and statistically validated items provide educators with greater confidence that student scores accurately reflect genuine understanding, rather than flaws or inconsistencies in the test itself. This is especially important in high-stakes environments where data from benchmark assessments drive resource allocation, intervention strategies, and instructional shifts.
2. Detection and Reduction of Bias
Fairness is a cornerstone of modern assessment practice. Without data from diverse populations, it is difficult to know whether an item may inadvertently favor one group over another.
Field-testing enables test developers to conduct differential item functioning (DIF) analyses, which detect potential biases based on factors such as gender, ethnicity, socioeconomic status, or geographic region. Items that display unfair performance across subgroups can be revised or removed entirely before being entered into operational use.
The U.S. Department of Education, in its publication “The Use of Tests as Part of High-Stakes Decision-Making for Students,” asserts that field-testing is a best practice in the development of equitable assessments and enables test developers to identify and eliminate items that may introduce construct-irrelevant variance or bias for particular groups of students.2
By contrast, relying on newly written items without prior data makes it impossible to screen for these problems, potentially compromising the fairness of the assessment and eroding trust in the data.
3. More Reliable Results for Instructional Use
Benchmark assessments are intended to inform teaching and learning. But if the items on an assessment have not been field-tested, the resulting scores may be based on flawed or inconsistent questions.
Unvetted items may:
- Confuse students due to ambiguous language
- Fail to align with intended standards despite best efforts
- Measure unintended skills or knowledge
When districts use assessments filled with such items, they risk misidentifying student needs or falsely interpreting patterns in performance.
“Assessment instruments built from field-tested items result in greater score reliability and decision consistency,” notes the Handbook of Test Development (2nd ed., Lane et al., 2016). “This is especially critical when assessments are used for instructional guidance.”3
Field-tested items have been carefully reviewed and refined to avoid issues that can cloud assessment results. By removing confusing language, ensuring alignment with standards, and verifying that each item measures the intended skill, these items help produce clearer and more dependable results. This gives educators more accurate information about what students know and can do—without the interference of poorly performing questions.
4. Support for Longitudinal Comparisons
Districts increasingly rely on benchmark assessments not just for point-in-time diagnostics, but to track student growth over time. For these comparisons to be meaningful, test quality must be stable.
Field-tested items provide that stability. Because their performance characteristics are well understood, they can be strategically reused, scaled, or benchmarked, allowing for more accurate year-over-year comparisons of student performance.
In contrast, newly written items inserted each year—especially without sufficient overlap with previous tests—introduce uncertainty. Even slight changes in difficulty, phrasing, or structure can confound comparisons across time. Without calibration, a district may appear to have gains or declines in student performance that are merely artifacts of item quality, not actual learning.
5. Replenishment and Sustainability of Item Banks
Field-testing is also essential for building and maintaining robust item banks. High-quality item banks serve as the foundation for flexible assessment systems that can deliver tailored, standards-aligned assessments across schools and years.
By continuously field-testing new items, assessment developers can retire outdated or overexposed questions while ensuring a steady pipeline of usable content. This proactive approach is endorsed by Smarter Balanced, a multi-state assessment consortium, which notes that “items on interim assessments [should be] field tested as part of the general pool . . . and meet the same measurement criteria as items on the summative test.”4
A strategy that relies exclusively on new items each year, without time to gather and analyze data, places undue strain on item developers and increases the likelihood of relying on unproven content to meet volume targets.
The Myth of “Freshness”: Why New ≠ Better
Advocates of newly created test items often cite concerns about test security, overexposure, and alignment with updated blueprints as reasons to forgo reuse. While these concerns are valid, they do not necessitate abandoning field-tested content.
In fact, well-managed assessment systems can maintain test security while leveraging field-tested items, especially when supported by large item pools, randomized test construction, and secure administration protocols. Security concerns should be addressed through assessment design and delivery safeguards—not by sacrificing item quality.
Likewise, field-tested items can (and often do) reflect updated standards and blueprints. Field testing is a process, not a timestamp; items aligned to current standards can and should be field-tested before they are used in official assessments. The “freshness” argument loses its power when freshness comes at the expense of reliability and validity.
Moreover, prior-year test versions can remain accessible for record-keeping and analysis without requiring 100% newly built, untested content each year. A hybrid approach—combining recently field-tested items with a foundation of validated content—is both practical and preferable.
A Call to Action for District Leaders
As district administrators evaluate assessment vendors and design benchmark programs, the emphasis should be on evidence of quality, not novelty. Before adopting benchmark tests that rely solely on “fresh” items, ask:
- What data supports the quality and fairness of these items?
- Have the items been field-tested with representative student populations?
- How are item statistics used to inform revisions or retirements?
- What safeguards are in place to ensure test results are valid, reliable, and relevant?
If the answers to these questions are unclear or absent, districts risk basing instructional decisions on shaky foundations.
Conclusion: Field-Tested Items as the Gold Standard
In the high-stakes environment of K–12 education, benchmark assessments must do more than generate data—they must generate trustworthy data. Field-tested items offer the best path to valid, reliable, and equitable assessments that support teachers, inform instruction, and promote student growth.
The allure of “new every year” may seem strong, but district leaders must resist the idea that new automatically means better. Just as educators pilot new curricula before full adoption, assessments should undergo real-world field testing before being used to measure learning.
The future of effective benchmark assessment lies not in reinvention, but in rigor, refinement, and responsible item reuse.
…………………………………………….
Notes
- American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. ↩︎
- U.S. Department of Education, Office for Civil Rights. The Use of Tests as Part of High-Stakes Decision-Making for Students: A Resource Guide for Educators and Policy-Makers. Washington, DC: U.S. Department of Education; December 2000. Available at: https://www.ed.gov/media/document/testingresourcepdf-54486.pdf. ↩︎
- Haladyna, T. M. (2016). Roles and importance of validity studies. In S. Lane, M. R. Raymond, & T. M. Haladyna (Eds.), Handbook of test development (2nd ed., pp. 1–18). Routledge. ↩︎
- Smarter Balanced Assessment Consortium. (2024, April 22). Chapter 1: Validity. In 2023-24 Interim Assessment Technical Report. SmarterBalanced. https://technicalreports.smarterbalanced.org/2023-24_interim-report/_book/validity.html ↩︎
