College rankings combine a dozen or so measures, such as graduation rates, graduates' salaries and surveys of what college leaders think of other colleges, into one score. Each publisher decides how much every measure counts. A 2024 review by NORC at the University of Chicago found that the major rankings can't show that this single number measures "quality," and none of them report a margin of error.
How college rankings are calculated
Every ranking follows the same basic recipe: pick some measures, give each one a weight, add them up and sort the colleges from top to bottom. The ingredients and the amounts differ a lot from one publisher to the next.
| Ranking | What counted most in 2024 |
|---|---|
| U.S. News & World Report | Survey of college leaders rating other colleges (20%), graduation rates (16%), graduation rates compared with a prediction (10%), spending per student (8%), plus about 15 smaller measures |
| Wall Street Journal | Student outcomes such as salaries and graduation (70%), learning environment (20%), diversity (10%) |
| Forbes | Graduates' salaries (20%), debt (15%), graduation rate (15%), a list of notable alumni leaders (15%), return on investment (15%), retention (10%), academic success (10%) |
| QS World University Rankings | Survey of academics' opinions (30%), research citations (20%), survey of employers' opinions (15%), faculty-to-student ratio (10%), plus five smaller measures |
Source: NORC at the University of Chicago, "College Ranking Systems: A Methodological Review" (2024), Table A3.
Same colleges, different recipes. That's why one school can sit in very different places on different lists in the same year.
Five things a ranking can't tell you
NORC's researchers reviewed five major rankings using the standards scientists apply to any measurement, from IQ tests to opinion polls. Here's what they found, in plain terms.
1. There's no agreed definition of "best"
Each publisher decides what "best" means, and many change their definition from year to year. Without a stable definition, it's hard to know what a rank is actually measuring.
2. The weights are judgment calls
Why should graduation rates count 16% and not 12%? The publishers rarely say. NORC concluded that "there is little rationale for the precise percentages assigned to each category." Change the weights and the order changes.
3. A rank hides how close colleges really are
Ranks make every step look the same size, but they aren't. In the Wall Street Journal's 2024 list, the scores of the #2 and #3 colleges were 0.2 points apart, while #10 and #11 were 2.0 points apart: ten times the gap, shown as the same one-step difference.
4. There's no margin of error
A weather forecast or an election poll tells you how sure it is. College rankings don't. NORC found that none of the major rankings report how much uncertainty sits behind each position, so there's no way to tell whether #8 and #15 are truly different.
5. The data aren't always right
Some of the numbers come from colleges themselves and aren't independently checked. In 2024, U.S. News corrected the scores of 213 colleges after publishing its list because of an error in how it collected data.
When the numbers were wrong: Columbia
In 2021, Columbia University was ranked #2 among national universities by U.S. News. In February 2022, Michael Thaddeus, a Columbia mathematics professor, published an analysis showing that some figures Columbia had reported, including how many classes had fewer than 20 students, didn't match what he found in the university's own course listings. That September, Columbia acknowledged that some of its data had been inaccurate, and it was then placed 18th.
Columbia's classrooms and professors hadn't changed. Only the numbers had. That's worth remembering whenever a college moves up or down a list.
Rankings shape what colleges do, too
Rankings don't just describe colleges; they also push them. Research summarized by NORC found that moving up in the rankings tends to bring a college more applications, and that some colleges have shifted money and attention toward the specific measures rankings reward. Families feel the pull as well: one survey found that 58% of college-bound seniors considered rankings to some degree.
What the researchers suggest instead
NORC's recommendations point in a clear direction:
- Show the parts, not just the total. Report each measure separately so families can compare what they care about.
- Use broad ratings instead of exact ranks. Grouping colleges into a few bands avoids pretending that #11 and #12 are meaningfully different.
- Let each family set its own weights. The researchers suggested that personal weightings, like The New York Times' "Build Your Own College Rankings" tool, could be more meaningful than one list built on an average family's priorities.
That last idea is another way of describing fit. The question isn't which college is "best," but which one is best for you. Research on outcomes points the same way: what students do and experience in college predicts later success and wellbeing better than how selective the college is.
FAQ
How are college rankings calculated? Publishers choose a set of measures, such as graduation rates, salaries and reputation surveys, assign each a weight, add them into one score and sort colleges by that score.
Are U.S. News rankings accurate? A 2024 University of Chicago review of U.S. News and four other major rankings found that none could show their single number reliably measures educational quality, and none report a margin of error. Their underlying data, like graduation rates, can still be useful on their own.
Do college rankings matter? They influence applications, but research finds that a student's experiences in college, such as mentors, engaged professors and real projects, matter more for later success and wellbeing than a college's rank or selectivity.
What's the difference between a ranking and a rating? A ranking puts colleges in a single order, #1 to #400. A rating places them in broad groups. Researchers suggest ratings mislead less, because small score differences don't turn into big-looking gaps.
What would your own ranking look like?
Lens's assessments show what matters to you in a college, from size and setting to how you like to learn, so you can build a list around your own priorities. Start with Lens →
Related
- How to use college rankings without letting them choose for you
- Does college selectivity matter? What the research says about fit
Sources
- Barari, S., Newsom, E., Park, J. E., & Paddock, S. M. (2024). College Ranking Systems: A Methodological Review. NORC at the University of Chicago (funded by Vanderbilt University).
- Thaddeus, M. (2022). An Investigation of the Facts Behind Columbia's U.S. News Ranking.
- Challenge Success (2018). A "Fit" Over Rankings: Why College Engagement Matters More Than Selectivity. Stanford Graduate School of Education.