WIXLIB

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note87-d. Readability formulas alone can’t tell you whether written materialsare clear and effective.Source: Created for this Toolkit. This figure summarizes many of the topics that areaddressed in the Toolkit Guidelines for Writing and Design. For details, see ToolkitParts 3, 4, and 5.We’ve put a reader at the center of Figure 7-d to emphasize that it’s the reader who decides what’s worthreading (Redish, 1993). It’s also the reader—not a grade level score—who decides whether material iseasy to understand and use (see Toolkit Part 2, Using a reader-centered approach to develop and testwritten material).Besides emphasizing that the reader is the ultimate judge, Figure 7-d reminds us that difficulty of thewords and sentences is only one of many, many factors that contribute to making materials clear andeffective. This means that using a grade level score as a sole criterion or summary indicator can

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note9mislead you into thinking that your materials are suitable and effective when they are not. Keepingindividual words and sentences easy enough for your readers is necessary but not sufficient for ensuringthat they can understand and use the material.The guidelines for writing in this book do include one guideline on reading grade level (Guideline 3.8 inToolkit Part 4, Chapter 3, Guidelines for writing style). But they also include 25 other guidelines onwriting as well as 46 guidelines on graphic design (for the full list, see Toolkit Part 3, Summary List of the“Toolkit Guidelines for Writing and Design”). Collectively, these guidelines address other factors thataffect ease of reading and use, including all of the issues listed above in Figure 7-d.Grade level scores tend to be unreliableVarious studies have identified technical weaknesses of readability formulas (see the literature review byRedish, 2000) and experts have referred to the “inherent unreliability” of the formulas (Schriver, 2000).One issue is the wide range of variation in scores for the same text (Hochhauser, 1999). There are manyreadability formulas and they take different approaches to estimating the difficulty of text. Formulas varyin which attributes of words and sentences they take into account and how they measure them. Giventhese differences among formulas, it’s reasonable to expect that using different formulas on the same textmight produce slightly different results. But sometimes the differences are large enough to bedisconcerting. Depending on which method you use and how you use it, scores for the same text candiffer by two, three, or more grade levels.There are some systematic differences among formulas. For example, literacy specialists warn thatFlesch-Kincaid scores tend to underestimate actual reading grade level because they are often severalgrade levels below results obtained using other measures (Root & Stableford, 1998; Audrey Riffenburgh,Personal communication, 1999).[NOTE: The Flesch-Kincaid is included in many versions of well-known word-processing packages suchas Microsoft Word and WordPerfect, and it produces a grade level score. Be careful not to confuse the

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note10Flesch-Kincaid with the other Flesch test, which is called the Flesch Reading Ease formula. The FleschReading Ease Scale (also known as the Flesch Index or Readability Score) is a score from 0 to 100 thatyou interpret by referring to a chart. For this formula, higher scores mean easier to read. For example, ascore of 70 to 80 means “fairly easy” and approximately seventh grade reading level.]In addition to the wide range of grade levels for the same text, there are problems of unreliablemeasurement related to scoring by computer: Some computerized assessments using the same formulaproduce different scores for the same text (Riffenburgh, 2005). In addition, as we explain below, if youuse a computerized formula, you need to prepare your text first to avoid misleading results.Preparing text for more accurate machine scoringIt’s important to prepare documents before you score them with a computerized readability formula.Otherwise, your results could be off by a considerable margin.To score a document, a computerized readability formula will typically analyze (1) word length asmeasured by the average number of syllables per word and (2) sentence length as measured by theaverage number of words per sentence. The program specifies how to do this and the computer followsinstructions in a totally mechanical way:

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note11 As shown in the picture above, embedded punctuation confuses the computer when it is countingthe number of sentences. Most readability programs tell the computer to sense the end of asentence by looking for the type of punctuation that normally marks the end of a sentence, such asa period, question mark, or exclamation point. Sometimes this punctuation falls within a sentence,rather than at the end, but the computer can’t distinguish this. Titles, headings, and bulleted lists also mislead the computer. There is usually no punctuationto help computers distinguish ordinary sentences from titles, headings, and bulleted lists. If thecomputer keeps searching for punctuation such as a period or question mark or exclamation point,it will include the text from headings as part of the first sentence that follows the heading.Obviously, the counts of sentence length can be miscalculated.To help the computer do its calculations correctly, it’s essential to prepare the text first by removingthings that will confuse and mislead the computer: Since the computer interprets any period as the end of a sentence, you need to remove embeddedpunctuation such as periods that are used for abbreviations. You also need to remove text that is not in full sentences, such as titles, headings, and bulletedpoints that are not full sentences.Check your program documentation for information and specific instructions. If you score a documentboth by hand and by computer, be sure to use the same sample of text for both methods in order to makemeaningful comparisons of the results.

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note12Grade level scores are less precise than they soundand prone to misinterpretationMeasurement is impreciseMost readability formulas produce a grade level score, such as “5th grade.” Some add a decimal, such as“5.3 grade.” Scores of this type are not nearly as precise as they sound, for several reasons: Scores can vary greatly for the same text. As we saw in the previous section, grade level scoresfor the same piece can differ by formula, and computer programs that use the same formula canyield different results. Scores have a margin of error. Grade level scores are approximations. Scores can differ depending on which part of a document is scored. Sometimes readabilityscores are based on scoring the entire document. Other times, especially for long documents, thescores are based on scoring samples drawn from the document. Usually, scores are based on theaverage score across three samples. You could get different scores for the same document if theyare based on different samples, especially if there are big variations in the text within thedocument. (Later on, we give instructions for using the Fry method and the SMOG to scorewritten material by hand. The instructions for drawing samples suggest ways to select passagesthat are representative of the material as a whole.)The name “readability” implies more than is actually being measuredBesides measurement imprecision, another concern is that grade level scores are prone tomisinterpretation. It’s easy to interpret a grade level score in a way that goes well beyond what hasactually been measured by the readability formula (that is, the average length of its words and sentences).For example, you might think from the name, “readability formulas,” that the formulas measure reading

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary noteease or comprehension – but they do not. As shown in Figure 7-e below, readability scores are notmeasures of comprehension, even though they are often interpreted that way.7-e.What does a readability score actually mean?Source: Compiled and formatted for this Toolkit, based on common themes in the literatureand the writer’s personal communication with experts in the field.13

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note14Imposing a grade level requirement has thepotential to do harmIt’s easy to infer from the name, “readability formulas,” that the formulas measure reading ease orcomprehension – which, as we’ve just discussed, they do not. Nonetheless, the formulas are often used asstandards to be met, such as “write it at the sixth grade level.”When readability scores are used as a standard, it’s with good intentions: the purpose is to help ensurethat the material is not too difficult for its intended readers. While it’s crucial to have a good matchbetween the reading skills of your intended readers and the difficulty of the material (see Guideline 3.8 inToolkit Part 4, Chapter 3, Guidelines for Writing Style), an appropriate reading grade level does not byitself ensure this good match. And, in fact, applying a grade level standard has the potential to do harm,because using grade level standards based on readability formulas can lead writers to produce text that isactually less readable even though it scores at a lower grade level.If you must meet a standard of “sixth grade level,” and your material scores at 8th grade level, what canyou do? It can be tempting to “write to the test” by dividing sentences and substituting shorter words.Edits of this type will improve a grade level score, but at the possible price of making the text choppy andharder to read. As shown below, substituting shorter words and chopping sentences into shorter ones isnot necessarily a service to your readers.

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note15We’ve just seen that heavy reliance on grade level standards can encourage “writing to the test,” whichends up making the material harder to read. Heavy reliance on grade level standards can also beproblematic in other ways: Using a grade level score as an overall indicator may give you a false sense of confidenceand cause you to miss problems with the material that make it hard for readers to understandand use. As shown earlier in Figure 7-d, even when a reading grade level is appropriate for theintended readers, there are still many ways in which written material can miss the mark.Using grade level scores as a standard may encourage people to treat grade level scores atface value, rather than as rough approximations. As already noted, there are manymeasurement issues associated with use of readability formulas. Given that scores can varygreatly and be unreliable, it makes sense to interpret all readability scores with caution.

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note16Recommendations for using readability formulasResponding to the concerns raised in the preceding sections, this section offers the recommendations forusing readability formulas shown below in Figure 7-f.7-f. Recommendations for using readability formulas.Use readability formulas only as tools for occasional limited use -not as ways to measure overall suitability of documents.Use scores from readability formulas as a check on difficulty of words andsentences – not as indicators of comprehension, not as summaryassessments of reading ease or usability, and not as a guide to writing. Ifmaterial is too difficult for the intended readers, a readability score mighthelp you convince others that revisions are essential. In general, makewriting clearly and cohesively in “plain language” your general goal forany written material for any audience. Rely on feedback from yourintended readers as the ultimate test of whether materials are clear andeffective (see Toolkit Part 6, How to collect and use feedback fromreaders).Pick your readability formula and method carefully (this Toolkitrecommends scoring written material by hand using the Fry methodor the SMOG).Scoring by hand tends to be more reliable than computer scoring. Also,working directly with the text makes you more aware of your writinghabits and helps you spot ways to improve. If you use a computerizedreadability formula, prepare the text first to avoid misleading results. Thisincludes removing embedded punctuation and text that is not in fullsentences.Interpret a score from a readability formula as indicating a generalrange of difficulty rather than a specific grade level.

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note17Report scores from readability formulas in ways that acknowledgethe narrow scope and limitations of readability formulas.When reporting a readability score, tell which formula and method youused, what it measures, and include other information to help people makea meaningful interpretation of the score. Tell whether the material hasbeen tested with the intended readers. Consider listing the words that werecounted as “difficult” by the formula (those with 3 syllables) to helpothers judge whether they are likely to be familiar to the intended readers.Source: Created for this Toolkit. These recommendations and the discussion of eachrecommendation reflect themes in the references and resources listed at the end of this document, aswell as suggestions from subject matter experts listed in the acknowledgments. They also draw ondiscussion in the precursor to this Toolkit titled Writing and Designing Print Materials for Beneficiaries:A Guide for State Medicaid Agencies, which was published in 1999 by the Health Care FinancingAdministration (HCFA, known today as CMS); now out of print.Do not use readability formulas to assess overall suitability

TOOLKIT for Making Written Material Clear and EffectiveSECTION 4 Special topics for writing and designPart 7: Using readability formulas: A cautionary note18Readability formulas can be a tool for occasional limited use if you keep in mind what they measure andtake care to avoid over-interpreting the results. You can use them as a quick screen for difficulty. Ifmaterial is difficult, readability scores can help convince people that revisions are needed.Based on the cautions we have discussed, here is what this Toolkit recommends: Use readability formulas as a tool for identifying long words and long sentences that may betoo difficult for your intended readers. Later in this chapter, we show you how to score text byhand using the Fry method or the SMOG. When you score text by hand, you will be marking it inways that highlight the longer words and sentences. These markings will make you moreconscious of your word choices and sentence structures and may help you see new ways tosimplify the material.Don’t use a score for reading grade level as your only indicator of difficulty or as a measureof comprehension. Keeping in mind that readability formulas only measure the length ofindividual words and sentences, don’t use them as a summary indicator or as your sole or finalstandard for judging suitability of materials.Don’t try to make written material easier to read simply by shortening sentences andsubstituting short words for long ones. “Writing to the test” in this way will improve the scorebut may make the material choppy and harder to read. When you need to reduce the reading difficulty of your materials, use readability formulasin combination with all the other Toolkit guidelines. Treat readability formulas as only onetool among many that can help you see ways to make

TOOLKIT for Making Written Material Clear and Effective SECTION 4 Special topics for writing and design Part 7: Using readability formulas: A cautionary note . 4. Why be cautious about using readability formulas? Sections that follow discuss the cautions that are listed below in Figure 7-b