[This article was first published on The 20% Statistician, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
< !--[if !mso]>< ![endif]-->Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
< !--[if gte mso 9]>< xml> < o:OfficeDocumentSettings> < o:AllowPNG/> < ![endif]-->< !--[if gte mso 9]>< xml> < w:WordDocument> < w:View>Normal < w:Zoom>0 < w:TrackMoves>false < w:TrackFormatting/> < w:DoNotShowPropertyChanges/> < w:PunctuationKerning/> < w:ValidateAgainstSchemas/> < w:SaveIfXMLInvalid>false < w:IgnoreMixedContent>false < w:AlwaysShowPlaceholderText>false < w:DoNotPromoteQF/> < w:LidThemeOther>EN-US < w:LidThemeAsian>X-NONE < w:LidThemeComplexScript>X-NONE < w:Compatibility> < w:BreakWrappedTables/> < w:SnapToGridInCell/> < w:WrapTextWithPunct/> < w:UseAsianBreakRules/> < w:DontGrowAutofit/> < w:SplitPgBreakAndParaMark/> < w:EnableOpenTypeKerning/> < w:DontFlipMirrorIndents/> < w:OverrideTableStyleHps/> < m:mathPr> < m:mathFont m:val="Cambria Math"/> < m:brkBin m:val="before"/> < m:brkBinSub m:val="--"/> < m:smallFrac m:val="off"/> < m:dispDef/> < m:lMargin m:val="0"/> < m:rMargin m:val="0"/> < m:defJc m:val="centerGroup"/> < m:wrapIndent m:val="1440"/> < m:intLim m:val="subSup"/> < m:naryLim m:val="undOvr"/> < ![endif]-->< !--[if gte mso 9]>< xml> < w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false" DefSemiHidden="false" DefQFormat="false" DefPriority="99" LatentStyleCount="371"> < w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/> < w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/> < w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 2"/> < w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 3"/> < w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 4"/> < w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 5"/> < w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 6"/> < w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 7"/> < w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 8"/> < w:LsdException Locked="false" Priority="9" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="heading 9"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 5"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 6"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 7"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 8"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index 9"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 1"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 2"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 3"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 4"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 5"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 6"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 7"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 8"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" Name="toc 9"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Normal Indent"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="footnote text"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="annotation text"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="header"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="footer"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="index heading"/> < w:LsdException Locked="false" Priority="35" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="caption"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="table of figures"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="envelope address"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="envelope return"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="footnote reference"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="annotation reference"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="line number"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="page number"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="endnote reference"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="endnote text"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="table of authorities"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="macro"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="toa heading"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List 5"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Bullet 5"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Number 5"/> < w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Closing"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Signature"/> < w:LsdException Locked="false" Priority="1" SemiHidden="true" UnhideWhenUsed="true" Name="Default Paragraph Font"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text Indent"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="List Continue 5"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Message Header"/> < w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Salutation"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Date"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text First Indent"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text First Indent 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Note Heading"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text Indent 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Body Text Indent 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Block Text"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Hyperlink"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="FollowedHyperlink"/> < w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/> < w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Document Map"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Plain Text"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="E-mail Signature"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Top of Form"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Bottom of Form"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Normal (Web)"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Acronym"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Address"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Cite"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Code"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Definition"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Keyboard"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Preformatted"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Sample"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Typewriter"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="HTML Variable"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Normal Table"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="annotation subject"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="No List"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Outline List 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Outline List 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Outline List 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Simple 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Simple 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Simple 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Classic 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Classic 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Classic 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Classic 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Colorful 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Colorful 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Colorful 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Columns 5"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 5"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 6"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 7"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Grid 8"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 4"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 5"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 6"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 7"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table List 8"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table 3D effects 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table 3D effects 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table 3D effects 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Contemporary"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Elegant"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Professional"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Subtle 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Subtle 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Web 1"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Web 2"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Web 3"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Balloon Text"/> < w:LsdException Locked="false" Priority="39" Name="Table Grid"/> < w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true" Name="Table Theme"/> < w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/> < w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/> < w:LsdException Locked="false" Priority="60" Name="Light Shading"/> < w:LsdException Locked="false" Priority="61" Name="Light List"/> < w:LsdException Locked="false" Priority="62" Name="Light Grid"/> < w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/> < w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/> < w:LsdException Locked="false" Priority="65" Name="Medium List 1"/> < w:LsdException Locked="false" Priority="66" Name="Medium List 2"/> < w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/> < w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/> < w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/> < w:LsdException Locked="false" Priority="70" Name="Dark List"/> < w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/> < w:LsdException Locked="false" Priority="72" Name="Colorful List"/> < w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/> < w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/> < w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/> < w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/> < w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/> < w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/> < w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/> < w:LsdException Locked="false" SemiHidden="true" Name="Revision"/> < w:LsdException Locked="false" Priority="34" QFormat="true" Name="List Paragraph"/> < w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/> < w:LsdException Locked="false" Priority="30" QFormat="true" Name="Intense Quote"/> < w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/> < w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/> < w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/> < w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/> < w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/> < w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/> < w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/> < w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/> < w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/> < w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/> < w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/> < w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/> < w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/> < w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/> < w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/> < w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/> < w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/> < w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/> < w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/> < w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/> < w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/> < w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/> < w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/> < w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/> < w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/> < w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/> < w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/> < w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/> < w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/> < w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/> < w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/> < w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/> < w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/> < w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/> < w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/> < w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/> < w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/> < w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/> < w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/> < w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/> < w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/> < w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/> < w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/> < w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/> < w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/> < w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/> < w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/> < w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/> < w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/> < w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/> < w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/> < w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/> < w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/> < w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/> < w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/> < w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/> < w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/> < w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/> < w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/> < w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/> < w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/> < w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/> < w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/> < w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/> < w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/> < w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/> < w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/> < w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/> < w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/> < w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/> < w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/> < w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/> < w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/> < w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/> < w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/> < w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/> < w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/> < w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/> < w:LsdException Locked="false" Priority="19" QFormat="true" Name="Subtle Emphasis"/> < w:LsdException Locked="false" Priority="21" QFormat="true" Name="Intense Emphasis"/> < w:LsdException Locked="false" Priority="31" QFormat="true" Name="Subtle Reference"/> < w:LsdException Locked="false" Priority="32" QFormat="true" Name="Intense Reference"/> < w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/> < w:LsdException Locked="false" Priority="37" SemiHidden="true" UnhideWhenUsed="true" Name="Bibliography"/> < w:LsdException Locked="false" Priority="39" SemiHidden="true" UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/> < w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/> < w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/> < w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/> < w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/> < w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/> < w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/> < w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/> < w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/> < w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/> < w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/> < w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/> < w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/> < w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/> < w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 1"/> < w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/> < w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/> < w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/> < w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/> < w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 1"/> < w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 1"/> < w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 2"/> < w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/> < w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/> < w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/> < w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/> < w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 2"/> < w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 2"/> < w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 3"/> < w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/> < w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/> < w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/> < w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/> < w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 3"/> < w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 3"/> < w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 4"/> < w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/> < w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/> < w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/> < w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/> < w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 4"/> < w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 4"/> < w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 5"/> < w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/> < w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/> < w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/> < w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/> < w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 5"/> < w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 5"/> < w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light Accent 6"/> < w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/> < w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/> < w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/> < w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/> < w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful Accent 6"/> < w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful Accent 6"/> < w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/> < w:LsdException Locked="false" Priority="47" Name="List Table 2"/> < w:LsdException Locked="false" Priority="48" Name="List Table 3"/> < w:LsdException Locked="false" Priority="49" Name="List Table 4"/> < w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/> < w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/> < w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/> < w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 1"/> < w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/> < w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/> < w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/> < w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/> < w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 1"/> < w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 1"/> < w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 2"/> < w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/> < w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/> < w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/> < w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/> < w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 2"/> < w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 2"/> < w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 3"/> < w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/> < w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/> < w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/> < w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/> < w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 3"/> < w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 3"/> < w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 4"/> < w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/> < w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/> < w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/> < w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/> < w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 4"/> < w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 4"/> < w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 5"/> < w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/> < w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/> < w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/> < w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/> < w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 5"/> < w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 5"/> < w:LsdException Locked="false" Priority="46" Name="List Table 1 Light Accent 6"/> < w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/> < w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/> < w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/> < w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/> < w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful Accent 6"/> < w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful Accent 6"/> < ![endif]-->< !--[if gte mso 10]>< ![endif]-->After performing a study, you can correctly conclude there is an effect or not, but you can also incorrectly conclude there is an effect (a false positive, alpha, or Type 1 error) or incorrectly conclude there is no effect (a false negative, beta, or Type 2 error).
The goal of collecting data is to provide evidence for or against a hypothesis. Take a moment to think about what ‘evidence’ is – most researchers I ask can’t come up with a good answer. For example, researchers sometimes think p-values are evidence, but p-values are only correlated with evidence.
Evidence in science is necessarily relative. When data is more likely assuming one model is true (e.g., a null model) compared to another model (e.g., the alternative model), we can say the model provides evidence for the null compared to the alternative hypothesis. P-values only give you the probability of the data under one model – what you need for evidence is the relative likelihood of two models.
Bayesian and likelihood approaches should be used when you want to talk about evidence, and here I’ll use a very simplistic likelihood model where we compare the relative likelihood of a significant result when the null hypothesis is true (i.e., making a Type 1 error) with the relative likelihood of a significant result when the alternative hypothesis is true (i.e., *not* making a Type 2 error).
Let’s assume we have a ‘methodological fetishist’ (Ellemers, 2013) who is adamant about controlling their alpha level at 5%, and who observes a significant result. Let’s further assume this person performed a study with 80% power, and that the null hypothesis and alternative hypothesis are equally (50%) likely. The outcome of the study has a 2.5% probability of being a false positive (a 50% probability that the null hypothesis is true, multiplied by a 5% probability of a Type 1 error), and a 40% probability of being a true positive (a 50% probability that the alternative hypothesis is true, multiplied by an 80% probability of finding a significant effect).
The relative evidence for H1 versus H0 is 0.40/0.025 = 16. In other words, based on the observed data, and a model for the null and a model for the alternative hypothesis, it is 16 times more likely that the alternative hypothesis is true than that the null hypothesis is true. For educational purposes, this is fine – for statistical analyses, you would use formal likelihood or Bayesian analyses.
Now let’s assume you agree that providing evidence is a very important reason for collecting data in an empirical science (another goal of data collection is estimation – but I’ll focus on hypothesis testing here). We can now ask ourselves what the effect of changing the Type 1 error or the Type 2 error (1-power) is on the strength of our evidence. And let’s agree that we will conclude that whichever error impacts the strength of our evidence the most, is the most important error to control. Deal?
We can plot the relative likelihood (the probability a significant result is a true positive, compared to a false positive) assuming H0 and H1 are equally likely, for all levels of power, and for all alpha levels. If we do this, we get the plot below:
Or for a rotating version (yeah, I know, I am an R nerd):
So when is the evidence in our data the strongest? Not surprisingly, this happens when both types of errors are low: the alpha level is low, and the power is high (or the Type 2 error rate is low). That is why statisticians recommend low alpha levels and high power. Note that the shape of the plot remains the same regardless of the relative likelihood H1 or H0 is true, but when H1 and H0 are not equally likely (e.g., H0 is 90% likely to be true, and H1 is 10% likely to be true) the scale on the likelihood ratio axis increases or decreases.
Now for the main point in this blog post: we can see that an increase in the Type 2 error rate (or a reduction in power) reduces the evidence in our data, but it does so relatively slowly. However, we can also see that an increase in the Type 1 error rate (e.g., as a consequence of multiple comparisons without controlling for the Type 1 error rate) quickly reduces the evidence in our data. Royall (1997)recommends that likelihood ratios of 8 or higher provide moderate evidence, and likelihood ratios of 32 or higher provide strong evidence. Below 8, the evidence is weak and not very convincing.
If we calculate the likelihood ratio for alpha = 0.05, and power from 1 to 0.1 in steps of 0.1, we get the following likelihood ratios: 20, 18, 16, 14, 12, 10, 8, 6, 4, 2. With 80% power, we get the likelihood ratio of 16 we calculated above, but even 40% power leaves us with a likelihood ratio of 8, or moderate evidence (see the figure above). If we calculate the likelihood ratio for power = 0.8 and alpha levels from 0.05 to 0.5 in steps of 0.05, we get the following likelihood ratios: 16, 8, 5.3, 4, 3.2, 2.67, 2.29, ,2, 1.78, 1.6. An alpha level of 0.1 still yields moderate evidence (assuming power is high enough!) but further inflation makes the evidence in the study very weak.
To conclude: Type 1 error rate inflation quickly destroys the evidence in your data, whereas Type 2 error inflation does so less severely.
Type 1 error control is important if we care about evidence. Although I agree with Fiedler, Kutzner, and Kreuger (2012) that a Type 2 error is also very important to prevent, you simply can not ignore Type 1 error control if you care about evidence. Type 1 error control is more important than Type 2 error control, because inflating Type 1 errors will very quickly leave you with evidence that is too weak to be convincing support for your hypothesis, while inflating Type 2 errors will do so more slowly. By all means, control Type 2 errors – but not at the expense of Type 1 errors.
I want to end by pointing out that Type 1 and Type 2 error control is not a matter of ‘either-or’. Mediocre statistics textbooks like to point out that controlling the alpha level (or Type 1 error rate) comes at the expense of the beta (Type 2) error, and vice-versa, sometimes using the horrible seesaw metaphor below:
Image from: http://www.statisticsfromatoz.com/blog/statistics-tip-of-the-week-the-alpha-and-beta-error-seesaw
But this is only true if the sample size is fixed. If you want to reduce both errors, you simply need to increase your sample size, and you can make Type 1 errors and Type 2 errors are small as you want, and contribute extremely strong evidence when you collect data.
Ellemers, N. (2013). Connecting the dots: Mobilizing theory to reveal the big picture in social psychology (and why we should do this): The big picture in social psychology. European Journal of Social Psychology, 43(1), 1–8. https://doi.org/10.1002/ejsp.1932
Fiedler, K., Kutzner, F., & Krueger, J. I. (2012). The Long Way From -Error Control to Validity Proper: Problems With a Short-Sighted False-Positive Debate. Perspectives on Psychological Science, 7(6), 661–669. https://doi.org/10.1177/1745691612462587
Royall, R. (1997). Statistical Evidence: A Likelihood Paradigm. London ; New York: Chapman and Hall/CRC.To leave a comment for the author, please follow the link and comment on their blog: The 20% Statistician.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.