ANOVA Intraclass Correlation Calculator Guide: ICC(A,1), ICC(A,k), ICC(C,1), and ICC(C,k)
The intraclass correlation coefficient, or ICC, is a family of reliability coefficients used when measurements are grouped within the same subjects, targets, or observational units. It is widely used for interrater reliability, test-retest reliability, measurement reproducibility, and related designs.
There is no single universal ICC formula. Shrout and Fleiss described multiple ICC forms, and later frameworks expanded and clarified the distinction among statistical model, measurement type, and agreement definition.
Koo and Li emphasize that researchers must specify which ICC form they calculate because different forms have different assumptions and interpretations. Their reporting guidance recommends stating the ICC model, type, and definition rather than reporting only a bare coefficient.
This calculator implements a specific family of two-way ANOVA coefficients from a balanced, fully crossed subjects-by-raters design. Every subject must be measured by every rater or measurement occasion, and the input consists of the ANOVA mean square for subjects, mean square for raters, residual mean square, number of subjects, and number of raters.
The calculator returns four coefficients because two separate conceptual choices remain after the two-way design is established.
The first choice is absolute agreement versus consistency. Absolute agreement treats systematic differences among raters as disagreement. If one rater consistently scores every subject five points higher than another rater, absolute agreement decreases because the numerical measurements do not match.
Consistency asks a different question. If raters preserve the relative ordering of subjects but one rater systematically scores higher than another, consistency can remain high because systematic rater-level offsets are not counted in the same way as disagreement.
The second choice is single measurement versus average of k measurements. A single-measure ICC estimates the reliability of one rater or one measurement. An average-measure ICC estimates the reliability of the mean of exactly k ratings or measurements.
This distinction is critical. A reliability coefficient for the average of three raters cannot be reported as though it described the reliability of one randomly selected rater.
The absolute-agreement single-measure coefficient is denoted ICC(A,1) in the McGraw-Wong framework and is commonly matched to Shrout-Fleiss ICC(2,1). Its denominator contains subject variation, residual error, and a term representing systematic rater variation.
The consistency single-measure coefficient ICC(C,1), commonly corresponding to ICC(3,1), excludes the systematic rater mean-square component from the denominator. For this reason, consistency ICC is often higher than absolute agreement when raters differ systematically in level.
Average-measure versions ICC(A,k) and ICC(C,k) describe the reliability of the mean of the k entered measurements. Averaging can substantially increase reliability because independent or residual measurement noise is reduced when multiple observations are combined.
ICC selection must be determined from the research design and inferential target before looking at the numerical results. Choosing whichever ICC happens to be largest after the calculation undermines the meaning of the reliability analysis.
Recent methodological work also cautions that the conventional fixed-versus-random rater distinction can be more nuanced than older simplified decision trees suggest, particularly in generalizability and incomplete-rater designs. This calculator therefore remains intentionally limited to the classical balanced two-way ANOVA setting for which its formulas are defined.
The result should also be interpreted with confidence intervals and context. A point estimate such as 0.82 does not reveal how uncertain the reliability estimate is, and a high ICC can coexist with substantial absolute measurement error when the sampled subjects are highly heterogeneous.
This calculator therefore provides transparent ANOVA-based ICC estimates while clearly identifying the estimand behind each one rather than presenting a single unlabeled reliability number.
How to Calculate Intraclass Correlation From a Two-Way ANOVA Table
- Confirm that the design is fully crossed: Every subject should have been measured by every rater or measurement occasion.
- Confirm that the design is balanced: The classical formulas assume the same number of rater observations for every subject.
- Enter subject mean square: Copy MSR from the subject or row effect in the appropriate two-way ANOVA table.
- Enter rater mean square: Copy MSC from the rater or column effect.
- Enter residual mean square: Use the subject-by-rater residual/error mean square from the corresponding ANOVA.
- Enter n and k: n is the number of subjects or targets and k is the number of raters or repeated measurements.
- Choose agreement or consistency conceptually: Use absolute agreement when raters must produce similar numerical values; use consistency only when systematic differences in level are acceptable for the intended application.
- Choose single or average measurement conceptually: Use single-measure ICC when decisions rely on one rating and average-measure ICC when the operational score is the mean of exactly k ratings.
- Report the full ICC form: Report the model, measurement type, agreement definition, estimate, and preferably confidence interval rather than only “ICC = 0.82.”
Formula and variables
MSR captures variation among subjects, MSC captures systematic variation among raters or measurement occasions, and MSE captures residual subject-by-rater error. Absolute-agreement coefficients include systematic rater differences in the reliability denominator, while consistency coefficients do not. Average-measure coefficients estimate reliability of the mean of k ratings and therefore have smaller effective error contributions than single-measure coefficients.
ICC(A,1) = (MSR − MSE) / [MSR + (k−1)MSE + k(MSC−MSE)/n]; ICC(C,1) = (MSR − MSE) / [MSR + (k−1)MSE]; ICC(A,k) = (MSR − MSE) / [MSR + (MSC−MSE)/n]; ICC(C,k) = (MSR − MSE) / MSR- MSR — Mean square for rows / subjects
- The two-way ANOVA mean square associated with differences among subjects or targets.
- MSC — Mean square for columns / raters
- The two-way ANOVA mean square associated with systematic differences among raters or measurement occasions.
- MSE — Residual mean square
- The two-way ANOVA error mean square representing residual subject-by-rater variation.
- n — Number of subjects
- The number of independent subjects or targets rated by every rater.
- k — Number of raters
- The number of raters or repeated measurements contributing to each subject.
- ICC(A,1) — Absolute agreement, single measure
- Reliability of one measurement when systematic rater-level differences count as disagreement.
- ICC(A,k) — Absolute agreement, average measures
- Reliability of the mean of k measurements when numerical agreement among raters matters.
- ICC(C,1) — Consistency, single measure
- Reliability of one measurement when systematic rater-level offsets are not treated as disagreement.
- ICC(C,k) — Consistency, average measures
- Reliability of the mean of k measurements under the consistency definition.
Scenario 1: Ten Subjects Rated by Three Raters
A balanced two-way subjects-by-raters ANOVA produces MSR = 100, MSC = 50, and MSE = 10 for 10 subjects rated by 3 raters.
- MSR
- 100
- MSC
- 50
- MSE
- 10
- Subjects
- 10
- Raters
- 3
- Common numerator = MSR − MSE = 100 − 10 = 90.
- ICC(A,1) denominator = 100 + (3−1)(10) + 3(50−10)/10.
- ICC(A,1) denominator = 100 + 20 + 12 = 132.
- ICC(A,1) = 90/132 ≈ 0.6818.
- ICC(C,1) denominator = 100 + (3−1)(10) = 120.
- ICC(C,1) = 90/120 = 0.7500.
- ICC(A,3) denominator = 100 + (50−10)/10 = 104.
- ICC(A,3) = 90/104 ≈ 0.8654.
- ICC(C,3) = 90/100 = 0.9000.
Result: ICC(A,1) ≈ 0.6818; ICC(A,3) ≈ 0.8654; ICC(C,1) = 0.7500; ICC(C,3) = 0.9000.
Consistency is higher than absolute agreement because systematic rater variation is ignored under the consistency definition but treated as disagreement under absolute agreement. Average-measure reliability is also higher because the mean of three ratings is more reliable than one individual rating under this model.
Understanding your results
ICC(A,1)
This estimates the reliability of one measurement under an absolute-agreement definition.
Systematic rater offsets count against reliability.
ICC(A,k)
This estimates reliability of the mean of exactly k measurements under absolute agreement.
It should not be reported as reliability of one individual measurement.
ICC(C,1)
This estimates the consistency reliability of one measurement.
Systematic rater-level differences are removed from the disagreement component represented in the denominator.
ICC(C,k)
This estimates consistency reliability of the average of k measurements.
It is usually larger than ICC(C,1) when averaging reduces residual measurement noise.
Negative ICC estimate
ANOVA-based sample estimates can become negative when residual variability exceeds the estimated between-subject component.
Do not silently clamp the result to zero because the negative estimate contains information about the sample variance decomposition.
Assumptions
- Every subject is measured by every rater or measurement occasion.
- The design is balanced.
- Subjects are independent of one another.
- Measurements are quantitative.
- The entered MSR, MSC, and MSE come from the correct two-way ANOVA decomposition.
- The selected ICC estimand matches the intended use of the measurements.
- Average-measure coefficients refer to the mean of exactly k measurements.
- The classical ANOVA variance-component assumptions are sufficiently appropriate for the study design.
Limitations
- ICC is a family of coefficients rather than one universal statistic. Shrout and Fleiss originally described six forms for common reliability designs, and later frameworks distinguish additional model, type, and definition combinations. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/18839484/?utm_source=chatgpt.com))
- Koo and Li emphasize that different ICC forms rely on distinct assumptions and can produce materially different interpretations. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/27330520/?utm_source=chatgpt.com))
- This calculator is limited to balanced fully crossed two-way ANOVA mean squares.
- It does not derive MSR, MSC, and MSE from raw observations.
- It is not designed for missing ratings, partially crossed raters, nested raters, unbalanced designs, or complex multilevel structures.
- Modern mixed-effects approaches can be more appropriate than classical ANOVA formulas when data are incomplete, unbalanced, hierarchical, or otherwise structurally complex. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/29218829/?utm_source=chatgpt.com))
- The calculator does not currently produce confidence intervals.
- A point estimate without a confidence interval can overstate precision, especially with small numbers of subjects or raters.
- A high ICC does not necessarily mean measurements have small absolute error.
- ICC depends partly on between-subject heterogeneity. A highly diverse sample can produce a large ICC even when measurement differences are clinically substantial.
- Conversely, a homogeneous sample can produce a modest ICC even when absolute measurement differences are small.
- Pearson correlation is not a substitute for ICC when agreement is the target because two raters can correlate perfectly while differing systematically in level.
- The common poor/moderate/good/excellent interpretation thresholds are heuristic guidelines rather than universal scientific laws.
- Koo and Li propose categories based on the 95% confidence interval, not merely the point estimate, and application-specific consequences remain important. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/27330520/?utm_source=chatgpt.com))
- Negative sample ICC estimates can occur and should not automatically be replaced with zero.
- Reliability does not establish validity. A measurement system can be highly reliable while consistently measuring the wrong construct or producing biased values.
- The calculator does not determine whether rater effects should scientifically be regarded as fixed, random, or generalizable beyond the sampled raters.
- Recent methodological literature notes that fixed-versus-random rater interpretation can be more nuanced than traditional simplified ICC decision trees imply. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/36048052/?utm_source=chatgpt.com))
Common mistakes
- Reporting “ICC” without specifying the ICC form.
- Calling ICC(2,1) a consistency coefficient.
- Calling ICC(3,1) an absolute-agreement coefficient.
- Using ICC(A,k) when the real-world decision is based on one measurement.
- Using ICC(C,1) when actual numerical agreement among raters is required.
- Choosing whichever ICC is numerically largest after seeing the results.
- Using Pearson correlation as a measure of absolute agreement.
- Ignoring systematic rater bias.
- Interpreting high reliability as evidence of validity.
- Ignoring the confidence interval.
- Clamping negative ICC estimates to zero without disclosure.
- Using balanced ANOVA formulas for missing or unbalanced rating data.
- Using average-measure ICC without stating the number of ratings being averaged.
- Applying universal reliability labels without considering the intended application.
Practical use cases
Scenario 2: Interrater absolute agreement
Three radiologists independently measure the same lesion diameter and the measurements are intended to be numerically interchangeable.
Absolute-agreement ICC is the relevant concept because systematic differences in measured diameter should count as disagreement.
Scenario 3: Interrater consistency
Several raters score performances and only relative ranking among subjects is important.
Consistency ICC can be relevant when a constant systematic scoring offset among raters is considered acceptable.
Scenario 4: One rater used operationally
The reliability study uses three raters, but future decisions will be based on one individual rating.
A single-measure ICC should be reported rather than the more favorable average-of-three result.
Scenario 5: Mean of three measurements used operationally
The final reported score is always the average of three independent ratings.
An average-measure ICC based on k = 3 directly reflects the reliability of that composite score under the specified model.
Scenario 6: Systematic rater offset
Rater B scores every subject approximately five points higher than Rater A while preserving subject ordering.
Consistency can remain high, but absolute agreement decreases because the raters do not produce matching values.
Planning and decision guide
ICC measures relative reliability within clustered measurements
Measurements from the same subject are expected to resemble each other more than measurements from different subjects when reliability is high.
ICC formalizes that relationship through variance components or equivalent ANOVA mean-square expressions.
ICC differs from ordinary Pearson correlation
Pearson correlation measures linear association between two variables.
ICC can incorporate both association and measurement-model structure, including whether systematic level differences count as disagreement.
Scenario 7: Perfect correlation but poor agreement
Rater B always records exactly Rater A + 10.
Pearson correlation can equal 1, yet absolute agreement is imperfect because every measured value differs by ten units.
The two-way ANOVA decomposes subjects, raters, and residual variation
MSR summarizes differences among subjects.
MSC summarizes systematic rater differences.
MSE contains residual subject-by-rater variation and measurement error.
Between-subject variability is essential to ICC
Reliability asks how much observed variability reflects stable differences among subjects versus measurement-related variation.
When subjects differ greatly relative to error, ICC tends to increase.
Scenario 8: Heterogeneous subjects
Subjects span a very wide range of true scores.
Even moderately noisy raters can preserve those large between-subject differences and produce a relatively high ICC.
Homogeneous samples can reduce ICC
When subjects have very similar true values, measurement noise occupies a larger share of observed variance.
The reliability coefficient can therefore decrease even if absolute measurement error itself has not worsened.
This is why ICC is sample-dependent
Reliability estimates should be interpreted in relation to the population sampled.
An ICC from a heterogeneous research sample may not transfer directly to a narrow clinical population.
Absolute agreement counts rater differences as error
ICC(A,1) includes the rater mean-square term in its denominator.
Systematic differences among raters therefore reduce the coefficient.
Consistency removes systematic rater-level offset
ICC(C,1) excludes the MSC-based rater-level term from its denominator.
The coefficient therefore focuses more strongly on whether subjects are ranked or differentiated consistently.
Scenario 9: Why ICC(C,1) can exceed ICC(A,1)
MSR = 100, MSC = 50, MSE = 10.
The agreement denominator includes systematic rater variation, whereas the consistency denominator does not.
This produces ICC(A,1) ≈ 0.6818 and ICC(C,1) = 0.75.
Agreement and consistency answer different scientific questions
Neither is universally superior.
The correct choice depends on whether absolute numerical interchangeability or relative consistency is required.
Use absolute agreement for interchangeable measurement
If one thermometer reads 37.0°C and another systematically reads 38.0°C, they are not in absolute agreement.
A high consistency ranking alone would not make the devices interchangeable.
Use consistency when systematic offsets are explicitly acceptable
Some rating systems care primarily about whether subjects are ordered similarly.
In those settings, a stable rater offset may be considered irrelevant to the reliability estimand.
Single-measure ICC answers the one-rating question
The “1” in ICC(A,1) or ICC(C,1) means reliability of a single measurement.
It does not mean there was only one rater in the study.
Scenario 10: Five-rater reliability study
Five raters provide data for estimating ICC(C,1).
The resulting coefficient still estimates reliability of one rating, not reliability of all five ratings combined.
Average-measure ICC answers the composite-score question
The “k” refers to the mean of the k ratings entered in the reliability design.
It is appropriate when the operational score is actually constructed by averaging those measurements.
Scenario 11: Average of four judges
If every competition score is the mean of four judges, ICC(A,4) or ICC(C,4) can describe reliability of that averaged result under the chosen agreement definition.
It should not be labeled ICC(A,k) without also reporting k = 4.
Averaging usually increases reliability
Independent or residual errors partially cancel when several measurements are averaged.
This is why average-measure ICC often exceeds the corresponding single-measure ICC.
Average reliability is not free information
Higher reliability depends on actually obtaining all k measurements.
If future practice uses only one rater, the average-measure coefficient is operationally misleading.
ICC(A,1) corresponds to common Shrout-Fleiss ICC(2,1) terminology
In the standard balanced two-way random interpretation, ICC(2,1) is the single-measure absolute-agreement coefficient.
Koo and Li emphasize the need to report model, type, and definition rather than relying only on shorthand labels. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/27330520/?utm_source=chatgpt.com))
ICC(C,1) corresponds to common ICC(3,1) consistency terminology
The two-way mixed consistency form is commonly labeled ICC(3,1) in the Shrout-Fleiss convention.
It excludes systematic rater mean differences from the disagreement component.
Do not interchange notation systems carelessly
McGraw-Wong A/C notation emphasizes agreement definition, while Shrout-Fleiss numbering emphasizes classical model forms.
Report both when useful to reduce ambiguity.
Recommended result labels
ICC(A,1) / ICC(2,1): absolute agreement, single measurement.
ICC(A,k) / ICC(2,k): absolute agreement, average measurements.
ICC(C,1) / ICC(3,1): consistency, single measurement.
ICC(C,k) / ICC(3,k): consistency, average measurements.
The study must be fully crossed for this calculator
Every rater must evaluate every subject.
The resulting two-way ANOVA separates subject, rater, and residual terms under the classical balanced design.
Scenario 12: Missing rating
One rater failed to score three subjects.
The classical balanced formulas no longer correspond exactly to the incomplete dataset and a mixed-model or other appropriate approach should be considered.
Balanced means the same rating structure applies to every subject
Each subject contributes the same number k of measurements.
Unbalanced observational designs require more general variance-component methods.
Modern reliability research extends beyond classical balanced ANOVA
Recent methodological work provides ICC approaches for incomplete, unbalanced, and multilevel designs.
The calculator should remain explicit about its classical scope rather than pretending the simple formulas cover every reliability study. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/36048052/?utm_source=chatgpt.com))
The rater model must match the intended generalization
Traditional ICC frameworks distinguish whether raters are conceptualized as a random sample from a broader population or as the particular raters of interest.
The design question should be considered before selecting an ICC label.
Fixed versus random is more subtle than a dropdown label
Recent generalizability-theory work argues that raters are seldom truly fixed in the simplistic sense commonly assumed.
The calculator can explain classical terminology while warning that advanced study designs may require more nuanced modeling. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/36048052/?utm_source=chatgpt.com))
Test-retest reliability can sometimes use the same two-way logic
Repeated occasions can occupy the rater/measurement dimension when every subject is measured on every occasion and the corresponding model is scientifically appropriate.
The same agreement-versus-consistency distinction still requires interpretation.
Scenario 13: Device repeatability
Every participant is measured on three fixed sessions.
The research question must determine whether exact agreement across sessions or merely consistency in subject ordering is important.
A negative ICC estimate can occur
When MSR is less than MSE, the common numerator MSR − MSE becomes negative.
This reflects a sample in which estimated between-subject signal is small relative to residual variation.
Scenario 14: Negative estimate
MSR = 8 and MSE = 10.
The numerator is −2, so an ANOVA-based ICC estimate can be negative.
Do not automatically truncate negative estimates
A population reliability parameter is often conceptualized within a nonnegative range under variance-component models.
But the unconstrained sample ANOVA estimator can be negative, and silently replacing it with zero hides information about the observed data.
ICC can theoretically exceed conventional interpretation bounds in degenerate sample formulas
Finite-sample ANOVA estimates can behave unexpectedly when variance-component assumptions fail or the data are degenerate.
The interface should avoid forcing every computed value into a cosmetic 0–1 progress bar without explanation.
Confidence intervals are essential for reliability reporting
The point estimate alone cannot show statistical precision.
Koo and Li recommend reporting the ICC with its 95% confidence interval and selecting interpretive categories based on the interval when such heuristics are used. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/27330520/?utm_source=chatgpt.com))
The current calculator does not compute confidence intervals
The result page should state this prominently rather than presenting the point estimate as complete reliability evidence.
A future enhancement can add model-specific F-based or mixed-model confidence intervals.
Sample size affects confidence precision
Reliability estimated from ten subjects is generally much less precise than the same point estimate from hundreds of subjects.
The coefficient value alone does not reveal that difference.
Number of raters also affects average-measure reliability
Average-measure coefficients depend directly on k.
Changing from two to five ratings changes the reliability of the mean even if the underlying single-rating reliability is unchanged.
Scenario 15: More raters
A measurement system with moderate single-rater reliability can produce a substantially more reliable composite when several independent ratings are averaged.
That result applies only if those multiple ratings are actually collected.
ICC resembles a variance-component proportion
In random-effects interpretations, ICC can be viewed conceptually as between-subject variance divided by total relevant variance.
NIST describes one-way ICC in precisely this variance-component form. ([itl.nist.gov](https://itl.nist.gov/div898/software/dataplot/refman1/auxillar/onewayan.htm?utm_source=chatgpt.com))
The two-way agreement model adds rater-related variance to total disagreement
This is why systematic rater differences affect ICC(A,1).
Consistency excludes that systematic level effect from the denominator of the classical coefficient.
High ICC does not mean small measurement differences
If subjects vary enormously, raters can preserve subject ranking and produce a large ICC despite meaningful absolute error.
Reliability and absolute measurement error should therefore be reported separately when both matter.
Scenario 16: Wide subject range
True scores range from 10 to 200 while typical rater disagreement is ±10.
Relative reliability can still be high even though a ten-unit error may be clinically unacceptable.
Agreement studies often need complementary error metrics
Standard error of measurement, limits of agreement, coefficient of variation, or related metrics can answer questions that ICC alone cannot.
The appropriate companion depends on the measurement scale and study purpose.
Bland-Altman agreement and ICC answer different questions
ICC summarizes relative reliability through variance structure.
Limits-of-agreement analysis examines the magnitude and distribution of pairwise measurement differences.
Reliability and validity are separate
A scale can give nearly identical scores every time and still systematically measure the wrong quantity.
High ICC therefore establishes neither construct validity nor criterion validity.
Scenario 17: Miscalibrated instrument
A device consistently reads five units too high for every measurement.
Repeated reliability can be excellent while measurement validity or calibration is poor.
Reliability depends on the intended measurement unit
If clinical decisions use one rater, single-measure reliability matters.
If decisions use the average of a panel, average-measure reliability matters.
Do not publish only the largest of four coefficients
The four calculator outputs are not competing estimates of the same question.
They answer different reliability questions.
Scenario 18: Choosing after seeing results
ICC(A,1) = 0.62 and ICC(C,4) = 0.91.
Reporting 0.91 simply because it looks better is invalid if operational decisions use one absolute measurement.
Interpretation thresholds are heuristics
Koo and Li suggest values below 0.5 as poor, 0.5–0.75 moderate, 0.75–0.90 good, and above 0.90 excellent based on the 95% confidence interval.
These cutoffs should not replace application-specific requirements. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/27330520/?utm_source=chatgpt.com))
Scenario 19: ICC 0.82
A point estimate of 0.82 may appear “good” under one common heuristic.
A wide confidence interval from 0.52 to 0.94 would indicate substantially more uncertainty than the point estimate alone suggests.
Reliability requirements should be established prospectively
Clinical measurement, research screening, individual decision-making, and group-level research can require different reliability standards.
The calculator should not declare one coefficient universally acceptable.
MSR represents subject discrimination
Large MSR relative to residual error tends to increase ICC.
It reflects the degree to which subjects differ systematically in the observed data.
MSC represents systematic rater-level differences
Large MSC can reduce agreement ICC while leaving consistency ICC relatively unaffected.
This is one of the most informative contrasts produced by the calculator.
Scenario 20: Large MSC
Raters differ substantially in their average score level.
ICC(A,1) falls because those offsets count as disagreement, while ICC(C,1) can remain comparatively high.
MSE represents residual interaction and measurement noise
Larger MSE reduces both agreement and consistency reliability because measurements become less stable within subjects.
This residual variation cannot be removed merely by redefining agreement as consistency.
Scenario 21: Large MSE
Raters disagree unpredictably from subject to subject.
Both agreement and consistency ICCs decline.
Average-measure consistency simplifies elegantly
ICC(C,k) = (MSR − MSE)/MSR in the balanced two-way formulation used here.
The residual contribution is reduced by averaging across k measurements.
Average-measure agreement still retains systematic rater effects
ICC(A,k) retains the MSC-based agreement penalty.
Averaging reduces random error but does not magically erase systematic disagreement among raters.
Scenario 22: Systematic bias survives averaging
Every rater gives internally stable measurements but one rater is always substantially higher.
Averaging can improve random precision while the absolute-agreement definition still recognizes the rater-level discrepancy.
The ANOVA F-value calculator answers a different question
The ANOVA F-value Calculator tests mean differences among independent groups.
This ICC calculator uses ANOVA mean-square components to quantify reliability within clustered repeated measurements.
Do not interpret ICC as another ANOVA significance test
The ICC point estimate describes reliability or variance partitioning.
It is not equivalent to the omnibus group-mean F statistic from ordinary one-way ANOVA.
The calculator should show the design assumptions beside the results
Balanced.
Fully crossed.
Quantitative measurements.
Independent subjects.
Correct two-way ANOVA mean squares.
The strongest result labels each coefficient in plain language
Do not display only ICC(2,1), ICC(2,k), ICC(3,1), and ICC(3,k).
Add “absolute agreement / consistency” and “single measurement / average of k measurements” directly beside each result.
The strongest reporting output includes the estimand
Example: “ICC(A,1) = 0.68, two-way absolute agreement, single measurement.”
A complete research report should additionally include confidence interval, study design, subjects, raters, and software or estimation method.
Frequently asked questions
What is an intraclass correlation coefficient?
ICC is a family of reliability coefficients that quantify similarity of measurements clustered within the same subjects or targets.
Is there only one ICC formula?
No. Shrout and Fleiss described multiple forms, and later frameworks distinguish model, measurement type, and agreement definition. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/18839484/?utm_source=chatgpt.com))
What ICCs does this calculator compute?
ICC(A,1), ICC(A,k), ICC(C,1), and ICC(C,k) from a balanced fully crossed two-way ANOVA.
What is ICC(A,1)?
It is a single-measure absolute-agreement ICC, commonly corresponding to Shrout-Fleiss ICC(2,1).
What is ICC(A,k)?
It estimates absolute-agreement reliability of the mean of k measurements.
What is ICC(C,1)?
It is a single-measure consistency ICC, commonly corresponding to ICC(3,1).
What is ICC(C,k)?
It estimates consistency reliability of the average of k measurements.
What does A mean in ICC(A,1)?
A denotes absolute agreement.
What does C mean in ICC(C,1)?
C denotes consistency.
What does 1 mean in ICC(A,1)?
It means reliability of one individual measurement.
What does k mean in ICC(A,k)?
It means reliability of the average of the k measurements used in the design.
What is the difference between agreement and consistency ICC?
Absolute agreement treats systematic rater differences as disagreement. Consistency allows systematic offsets when relative subject ordering is preserved.
Which is better, agreement or consistency?
Neither is universally better. The appropriate definition depends on whether numerical interchangeability or relative consistency is required.
Why is consistency ICC often higher?
Systematic differences among raters are not included in the same way in the consistency denominator, so stable rater offsets can reduce agreement more than consistency.
What is MSR?
MSR is the subject or row mean square from the balanced two-way ANOVA.
What is MSC?
MSC is the rater or column mean square and reflects systematic average differences among raters.
What is MSE?
MSE is the residual mean square representing subject-by-rater error or unexplained measurement variation.
What does n mean?
n is the number of subjects or targets.
What does k mean?
k is the number of raters or repeated measurements for each subject.
Can I calculate ICC from raw data with this calculator?
Not directly. This calculator expects the appropriate two-way ANOVA mean squares.
Does every subject need to be rated by every rater?
Yes for the classical balanced fully crossed formulas implemented here.
Can I use this calculator with missing ratings?
Not reliably. Missing or incomplete rating designs often require mixed-effects or other generalized reliability models.
Can I use this calculator with unequal numbers of ratings?
The implemented formulas assume a balanced design, so unbalanced structures require a more general method.
Can ICC be negative?
Yes as a sample ANOVA estimate. This can occur when residual variation is larger than the estimated between-subject component.
Should negative ICC be changed to zero?
Not silently. Reporting the negative estimate preserves information about the observed variance structure.
Can ICC be greater than 1?
Conventional population ICC parameters are generally interpreted within the reliability range, but unconstrained sample estimators can behave unusually in degenerate or assumption-violating data. The raw estimate should be investigated rather than cosmetically forced into a range.
What is a good ICC?
Koo and Li suggest heuristic categories based on the 95% confidence interval: below 0.5 poor, 0.5–0.75 moderate, 0.75–0.90 good, and above 0.90 excellent, but application-specific standards remain important. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/27330520/?utm_source=chatgpt.com))
Should I interpret the point estimate alone?
Preferably not. Reliability should be reported with a confidence interval when possible.
Does this calculator calculate confidence intervals?
No. The current implementation calculates point estimates only.
Does a high ICC mean raters agree exactly?
Not necessarily. High consistency can coexist with systematic numerical differences, and high ICC generally can coexist with meaningful absolute measurement error.
Is ICC the same as Pearson correlation?
No. Pearson correlation measures linear association, while ICC incorporates clustering and the chosen reliability model.
Can Pearson correlation be 1 while agreement is poor?
Yes. If one rater is always ten units above another, the correlation can be perfect even though absolute agreement is not.
Is ICC the same as Cronbach alpha?
No. They can be mathematically related in some designs, particularly average-measure consistency settings, but they are defined and interpreted within different measurement frameworks.
Is ICC the same as Cohen kappa?
No. ICC is primarily used for quantitative measurements; kappa is commonly used for categorical agreement.
Can I use ICC for test-retest reliability?
Yes when the repeated-measure design and selected ICC model appropriately represent the measurement occasions and intended inference.
Can I use ICC for interrater reliability?
Yes. Interrater reliability is one of the most common uses of ICC.
Should raters be random or fixed?
That depends on the study’s intended generalization. Recent methodological work notes that this distinction is more nuanced than simplified traditional rules often suggest. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/36048052/?utm_source=chatgpt.com))
Can I choose the ICC with the highest value?
No. The ICC form should be selected from the study design and intended reliability question before inspecting the result.
Why is average-measure ICC higher than single-measure ICC?
Averaging multiple measurements reduces the effect of residual random measurement error.
Can I report ICC(A,k) if future decisions use one rater?
No. That would report reliability of an averaged score rather than the actual single-measure decision process.
Does a high ICC prove the measurement is valid?
No. Reliability describes reproducibility or consistency, while validity concerns whether the intended construct or quantity is measured correctly.
Why does sample heterogeneity affect ICC?
Greater between-subject variability increases the signal component relative to measurement error and can raise ICC.
What should I report with ICC?
Report the ICC model/form, agreement definition, single or average measurement type, estimate, confidence interval when available, study design, number of subjects and raters, and estimation software or method. Koo and Li explicitly recommend reporting model, type, and definition. ([pubmed.ncbi.nlm.nih.gov](https://pubmed.ncbi.nlm.nih.gov/27330520/?utm_source=chatgpt.com))
How does ICC relate to ANOVA?
Classical ICC estimators use ANOVA mean squares to estimate how much variability comes from subjects, raters, and residual error.
Is this the same as the ANOVA F-value calculator?
No. The ANOVA F-value Calculator tests mean differences among independent groups, while this calculator uses two-way ANOVA variance structure to estimate reliability.
How accurate is an ICC calculator?
The formulas can be calculated exactly from valid mean squares and design inputs. The scientific validity of the coefficient depends on choosing the correct ICC model and satisfying the assumptions of the underlying reliability design.
Sources and review
- Intraclass Correlations: Uses in Assessing Rater Reliability — Psychological Bulletin / PubMed. Accessed 2026-09-01.
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research — Journal of Chiropractic Medicine / PubMed. Accessed 2026-09-01.
- Updated Guidelines on Selecting an Intraclass Correlation Coefficient for Interrater Reliability — Psychological Methods / PubMed. Accessed 2026-09-01.
- Intraclass Correlation: Improved Modeling Approaches and Applications — Human Brain Mapping / PubMed. Accessed 2026-09-01.
- One-Way ANOVA — Variance Components and Intraclass Correlation — NIST/SEMATECH Dataplot. Accessed 2026-09-01.
- Intraclass Correlation / Correlation Ratio — NIST/SEMATECH Dataplot. Accessed 2026-09-01.
Reviewed 2026-09-01 by Dr Akawak Ejigu, DBA.