5 Tips from Smarter Item Writing
I just spent the last week writing Smarter Balanced Assessment Claim 1 Reading items through OSPI. It was a great experience. I made friends with several other teachers. I learned there are many teachers out there with similar vision, goals, and willingness to work hard for the betterment of our kids. I strengthened connections with the OSPI office as well as other teachers from around the state. Most importantly, I learned things I can bring back to my classroom, my team, and my district.
So here they are – five tips from my experience with “Smarter” item writing.
1 – Variety of Texts
I know we have all heard it, and maybe we have all said it, but students need experience with a variety of texts. Not only do students need to read the classics, they also need to be reading challenging texts in history, science, and modern literature. They need to grapple with poems as well as complex Tier 3 content-specific vocabulary. Our students need experience with archaic language as well as scientific terms. However, we cannot leave literary texts behind in the face of informational texts.
Here is a chart of possible stimuli from SBAC’s website and stimulus specifications document:
2 – Complex Texts
Having students read texts that can challenge them does not mean they have to be full of advanced vocabulary and complicated sentence structure. There do need to be words that challenge them, which can be figured out based on context, but having text that is accessible is important too. What we are looking for in good, rigorous text is depth. We want to find texts that have layers of meaning, figurative language, and advanced structures. Texts should challenge students’ thinking, not just their vocabulary.
You can find more information and the stimulus specifications here. http://www.smarterbalanced.org/wp-content/uploads/2015/08/ELA-Stimulus-Specifications.pdf
3 – Item Formatting
Something interested that I learned, which I believe will be important and useful to tell me students, is the rules for answer formatting. Have you even been the student who thinks that an answer can’t be the right answer because the teacher wouldn’t put it in a certain spot? Surely they wouldn’t have 3 C’s in a row? That can’t be right?
Well, question and answer formatting is specific. If answers are all one word, they are in alphabetical order (numerical value small to large or large to small if one number). If they are paraphrases or provided possible responses (not quotes), they are in order by length. If they are quotes, they are in the order of how they appear in the text. This seems amazing to me. So intentional, but I would not have thought of it and it really takes the thought out of how to organize the answers.
Another tip for my kids has to do with the vague questions. Have you seen any in the practice test that say select all that apply? Well, there are questions where students need to pick two or three responses, but from now on they will be written with the specific number in the question. That’s the good news. The bad news is that there are not enough new items yet to replace all of those vague stems. Here comes the tip. The 50% rule. Did you know about this rule? Generally, there are 5-8 answers when the students need to pick multiple responses (my team leaned toward 5). No more than 50% of those answers can be right answers. So, if there are only five choices, there should be 2 right. If there are 6, there will be 2 or 3. This might help some students stress out less over those vague multiple select questions.
You can find more information about the style guide and formatting here. http://www.smarterbalanced.org/wp-content/uploads/2016/04/Smarter_Balanced_Style_Guide.pdf
4 – Specifications and Item Stems
Did you have a booklet of item stems for the previous test? Maybe you were one of the people who lived by your stems on a ring? Well, they still exist, but they are a bit more complicated. Smarter Balanced has created item stems for the item writers. This provides consistency, which I will talk about later, but it also helps you make sure that the question is asking for the right evidence and targeting the right standard. That is something we had to do as item writers. We have to identify the standard targeted by our question, what depth of knowledge is required, and what evidence the question is asking the students for. Look at this example from literary language use:
The item specifications can be found under the Items and Specifications section and then in the appropriate content tab here. http://www.smarterbalanced.org/assessments/development/
5 – Consistent and Rigorous
Smarter Balanced, the cooperating states, and the teachers who work on this test are trying their best to make it as consistent and as fair as it can be. Every part of our process had us think about bias and sensitivity, specific word choice/item targeting, and whether our distractor answers were reasonable selections given students’ thinking. I felt like this was a rigorous process for me as an item writer. Some of my co-writers commented on the difference between WASL writing and this experience. Gone are the days where you can whip out some test questions and call it good. If you want your questions in class to have the same consistency and rigor as the state test, then you need to think about spending an hour (or more sometimes) hammering out the word choice of one item (like we did) to make it as clean and specific as possible.
You can find the process and math that goes into checking items for reliability here. (You can also apply to do data analysis for SBAC) http://www.smarterbalanced.org/wp-content/uploads/2015/08/2014-15_Technical_Report.pdf
Again, the overall test development page is here. http://www.smarterbalanced.org/assessments/development/
Happy assessment writing to you. I hope it is as challenging and stimulating for you as it was for me. I’m off to item review next, maybe I’ll see some of the ones we wrote last week! @ejohnstonteach
In my non-teacher consumed hours I love to spend time with my husband and son, play board games, sew/craft/quilt, and read (I DO teach ELA).I aspire to be more into fitness and outdoors more often, though I find a comfy chair and a good book/movie mightily appealing.