( Mark Lennihan, File / AP Photo )
The influential college rankings from U.S. News & World Report were released earlier this week and Columbia University dropped from number 2 to 18. Francie Diep, senior reporter covering money in higher education for The Chronicle of Higher Education, reports on the incident, which involved one of Columbia's own professors questioning the data, and what it reveals about the college ranking system as a whole.
[music]
Brian Lehrer: It's The Brian Lehrer Show on WNYC. Good morning, everyone. Today on the show, NPR legal affairs correspondent Nina Totenberg joins us with her new book Dinners with Ruth. Can you imagine being a fly on the wall as Nina Totenberg and Ruth Bader Ginsburg chatted and dined? We'll find out what was on the culinary and intellectual menu. We'll get to some current legal affairs too. There are one or two things going on, right? It's a rare chance for you to call in and ask Nina Totenberg a question.
That's coming up in about an hour at around eleven o'clock. Also, today, why New Jersey might bring back paper shopping bags. New Jerseyans, should they do that? Are you one of those people who now have piles of reusable bags in your home? Masha Gessen today on the possible turning point right now for the war in Ukraine and maybe, just maybe, for Vladimir Putin's power. Masha Gessen coming up second today.
We begin today with the question, what the heck is happening at Columbia University? What does it tell us about how colleges are ranked? As you may have heard, the influential US News & World Report college rankings list now has dropped Columbia from number 2 overall down to number 18 because of what the magazine calls data accuracy problems. That means it goes from being number 2 right behind Princeton and right ahead of MIT, Harvard, Stanford, and Yale down to 18 trailing behind Vanderbilt, Rice, Washington University, and Cornell, and just ahead of Notre Dame.
You may be asking yourself, "What did Columbia University do to deserve this demotion? It must be pretty bad," or you might be asking yourself, "Those all sound like good schools. Why do so many people care what US News & World Report thinks and how do they really know anyway?" Let's try to answer as many of those questions as we can. We will do this with your input in the phones as we go at 212-433-WNYC. With us to help do this is Francie Diep, senior reporter for The Chronicle of Higher Education. Hi, Francie, thanks for coming on WNYC. Welcome.
Francie Diep: Hello.
Brian Lehrer: Let's start with this week's news about Columbia and we'll try to go deeper from there. What were these data accuracy problems?
Francie Diep: Yes, so this story starts actually in February. At that time, a math professor at Columbia posted on his faculty webpage this whole big analysis where he claimed that Columbia was actually submitting incorrect data to US News for its ranking. We should say here that US News really relies on colleges to self-report data for their rankings.
Brian Lehrer: Wait, wait. Before you even go there, it was a Columbia professor who blew the whistle on his school, not giving good information to US News & World Report? I hope he's a tenured professor.
Francie Diep: He certainly is and, yes, the call came from within the house. That's true.
Brian Lehrer: Okay.
Francie Diep: Anyway, so yes. He posted this analysis. Then I think from there, probably US News officials started looking into it, although they didn't see anything publicly at first. Columbia initially denied that there was any problem with their data. A little bit later in the year, US News actually totally removed Columbia's numerical ranking, saying that Columbia officials had not substiten-- well, I can't say the word, had answered their questions about their data. Then Columbia said they were going to do this whole review of their data and make sure that it's okay.
Because this review was going to take a long time, they were not going to submit data for this year's coming rankings or the rankings that just came out earlier this month. US News, even if you don't send them data, they still rank you. Basically, this year, when US News came out with their rankings, they didn't have self-report data from Columbia. They figured out their ranking on their own and that's what gave Columbia a much lower ranking. Usually, if you don't send in data, you get maybe a lower ranking than you would have otherwise.
Brian Lehrer: We'll get to what some of the specific data categories were that Columbia did not report. Some of them are pretty interesting. In this initial assessment by the Columbia math professor, is there an assertion here that Columbia tried to deceive US News & World Report to get a higher ranking?
Francie Diep: I hate to speak for anyone, but I have interviewed him a few times. My understanding is that he, as we all, cannot go into the minds of others and figure out their motivations. Possibly, it was an honest or unintentional mistake. Either way, he's saying that the data that went in was incorrect. I will say that Columbia did put out corrected data a little bit before US News rankings came out. One of their data points was pretty different. Their corrected data point versus what they had originally sent in. This data point was the number of classes on campus that are for undergraduates that have 20 or fewer students in it. Originally, they said it was more than 80% of classes were that small. Now, they say 57% of classes are that small.
Brian Lehrer: How could it change that much? How could they not know? That seems like a pretty knowable fact.
Francie Diep: I don't know. I don't know how they couldn't know.
Brian Lehrer: Right, so that's an open question. Another interesting one was the percentage of Pell Grant recipients who graduate from the school, right? That would be an indication of whether lower-income students who get admitted to this elite university, Columbia, actually make it through the four years. That's why they measure that?
Francie Diep: Yes, so that apparently is something that US News depends on colleges to report to them because, as a reporter, I got these spreadsheets of the different colleges' data before the rankings came out. When I looked through the spreadsheets, I saw that Pell Grant recipients' graduation rates was something that Columbia didn't have where other national universities did have.
Like I said before, Columbia did put out corrected data a little bit before the rankings came out. They just weren't able to turn that data in time to be part of the rankings, but they posted it publicly on their own website. I think it's likely they have that graduation rate in their public data set. I haven't looked, but they just weren't able to submit it in time to US News, so it isn't a part of US News ranking.
Brian Lehrer: Now, pulling out to the bigger context for this, you report that US News relies heavily on self-reported data to calculate college rankings. What kinds of data most importantly besides those two categories we just mentioned?
Francie Diep: Oh, that's a good question. Let me think. Because US News doesn't-- they're like fairly open about what goes into their rankings, but they don't want to give out the exact formula. Sometimes it's a little unclear, what they get from the schools versus what they get from public databases such as federal databases. My guess would be that class sizes are a big thing that they depend on the schools to give them. Alumni giving, I think, is probably another big one.
Brian Lehrer: Those kinds of things, a mix of things, the colleges themselves report and things that are available in public databases, you say?
Francie Diep: Yes.
Brian Lehrer: In general, on those self-reported pieces, how can anyone trust that any college is self-reporting accurately? US News & World Report, that's a news organization. They wouldn't let politicians or probably any of their other sources self-report their stories on the news pages of the magazine, right?
Francie Diep: Yes, so this is a big point of contentions, the big thing that a lot of critics of their rankings point to. It's something of a bigger issue that, I think, that Columbia event points to is that there is no real third-party vetting the data that colleges sent into US News. We would have had no idea about all of Columbia's data problems if this professor hadn't just decided he was interested and looked into this more closely. Yes, that's it. I think it's probably just too many colleges and too much data for the size of the US News team for them to really--
Brian Lehrer: Yes, vet everything.
Francie Diep: Yes, vet everything.
Brian Lehrer: Let's take a phone call. Jessica in Jersey City, you're on WNYC. Hi, Jessica.
Jessica: Oh, hi. Can you hear me?
Brian Lehrer: Can hear just fine.
Jessica: Oh, wonderful. I was just wondering. Number one, I think it's problematic that everything is self-reporting, right? That's always the problem. Then I think without also including outcome data, I feel like there's so much emphasis on input-type data like how small the class sizes are, how many professors have advanced degrees, how much money they have. What about outcome data? How many of their graduates actually get jobs in the fields that they want? Just the success of the students, I think that that's a real-world-type data point that is not included at all in the ranking system-
Brian Lehrer: Go ahead.
Jessica: -that I know of, and I think especially because higher education is so expensive that that would help. You know what I mean? It would help maybe people make decisions based on what maybe they want to pursue.
Brian Lehrer: Right. Well, stay there for a second, Jessica, because I want to ask you a follow-up question. Francie, they do report some outcomes data at least. The category we were just discussing, graduation rate among Pell Grant recipients, that's an outcome. The person graduated or the person didn't graduate. Are there more like that as far as you know?
Francie Diep: Yes, there are some outcomes. Yes, there's overall graduation rates. There's also how well colleges retain their students from their first year to second year. That said, data on where students go on afterwards, whether they're happy with their jobs, that is not at all included. It would be pretty hard to track, I think. There are, I think, some of the law or business schools that may have a little bit of that included in their rankings. I'm not totally sure. That also, again, is hard to track and also probably easy to fudge because you really have to rely on the school to track that for you and send that in to you.
Brian Lehrer: Right, and they probably have to rely on the graduates themselves to say, "Did I get a job in the field that I majored in," and things like that. Jessica, did you also tell our screener that you think the whole system of college rankings doesn't translate to modern times for some reason?
Jessica: Yes, I guess I just think with educating a population, how much money that costs, I just feel like-- I don't know. I guess it feels a little elitist in these times to really be parsing through like, "Oh, who's better? Stanford or MIT? Am I going to go to Columbia or to Harvard?" I don't know. I just think for how much time and energy and probably money that gets put into it, is it really of that much value for real for the average person really thinking about the decision to go to college and where to go?
Brian Lehrer: Thank you for that question. Francie, is there an answer? There are really a few questions in there. One is, isn't this for a relatively small group of applicants to a relatively small number of elite colleges and universities? What's the answer to that as far as you know?
Francie Diep: Yes, I think, absolutely, that's true. There is survey data out there to suggest that a large minority, something like one in five college students really use rankings to make their decision about where to go to college. One important thing to keep in mind is that most Americans who get some higher education go to community colleges and US News does not rank those at all. If that's on your path, rankings don't matter to you.
Brian Lehrer: Well, how long is the full list? If Columbia is 18 and I see right below it, Notre Dame, 19; UC Berkeley, 20; UCLA, 21, how long does that list go on?
Francie Diep: I think they rank every four-year college, so it's pretty long. It's like 200-some. At some point, they stop giving people numbers because it seems a little rude to be like, "Well, you're number 305." It's pretty long.
Brian Lehrer: How do you know something is number 305 instead of 304?
Francie Diep: Sure. Yes, exactly.
Brian Lehrer: How influential are the US News rankings? To whom do they matter and with what impact on the schools or their students?
Francie Diep: That's a great question. There are a few different answers to that. One is, obviously, the students who care about them most likely tend to be students who are going to go to private colleges. A narrow slice of the college-going population cares. The schools themselves care a lot. You can just see this if you look in colleges' strategic plans oftentimes, especially if they're mid-list. It's part of their overall university goal to move up in the rankings. Officials care, whenever they move up, even one or two steps.
They put out press releases and send them to people like me. The third kind of audience for this is maybe a little more niche. In my reporting, I've found, for example, the state of Florida actually wrote into their laws about how the public universities are funded that they can get some bonus funding for doing well on US News like metrics. That's a big deal. If you're a public university, you can get more money for doing better on this. You're going to go after it.
Brian Lehrer: I guess there are different rankings. In addition to US News these days, you got me curious saying how long the list was to look up my own undergraduate college, which was SUNY at Albany. What came up is that it's number 182 and that's not US News though. Oh yes, it is. No, I take it back. That is the US News. I thought I was looking at a different ranking. I may have to withdraw that question, but okay. Yes, Albany is number 182 on this year's list of best colleges in the category of-- This is a little unclear to me, but somehow, the category of national universities. There you go. It goes to at least 182. You said it goes beyond 200. Are there competitors to US News & World Report?
Francie Diep: Yes, there are a number of publications that rank colleges. Some of them rank general undergrad programs the way that US News national universities this does. Some of them rank business schools or other more specific programs. They're a big deal for US colleges that are trying to attract international students, which can be a big deal for public colleges. International students pay out-of-state tuition. It's really helpful for them. US News is the best-known in the US and one of the oldest, but there are certainly many lists.
Brian Lehrer: I will say that Albany when I went there, I think, was a pretty darn good school. From what I hear, it's a pretty darn good school today. Don't drag 182 necessarily, but maybe we shouldn't be dragging the whole idea of the rankings too much. I could see where, at least in theory, maybe you have a sense of whether it comes out this way in practice. At least in theory, if the rankings mean something to the success of the schools, then it gives them an incentive to do well at some of these metrics, to have small class sizes, to help make sure that their Pell Grant students get the support they need to graduate and whatever else.
Francie Diep: Yes, so I think that's one big argument for the rankings is that if you choose the metrics right, you're going to incentivize schools to do things to improve those metrics. The Florida case that I was talking about in the years that the state legislature has had metrics in their funding formula for public universities, all the public have improved their graduation rates, lowered their tuition. Those are great things for college-going kids and their families.
I think the criticism that people leverage against this is that it's US News, a bunch of journalists who, I suppose, they're well-meaning, but it's them choosing the metrics that should matter versus-- I don't know. I suppose you could have the colleges do it themselves or that there's some conflict of interest there too, but it's not. It's not experts saying this is what should matter. This is what colleges should be aiming for. It's this group that doesn't really have a whole-- I don't want to say they don't have expertise in higher ed because they've been doing this for a long time. They have some kind of expertise, but you could imagine a better group of people.
Brian Lehrer: Those kinds of incentives helping make places better. We're talking, if you're just joining us, about Columbia University being dropped in the US News college rankings from number 2 to number 18 this week for the present academic year because of missing data about things like the rate of graduation among Pell Grant students and the percentage of classes that have 20 students or fewer. Janet in Manhattan, you're on WNYC with Francie Diep from The Chronicle of Higher Education. Hi, Janet.
Janet: Hi there. I just wanted to give another perspective on the thing about self-reporting, which does seem strange, right? We are having another problem at NYU, and that's that we've been self-reporting information to try to get them to report our numbers correctly. We have not been able to get them to change. We have incorrect data about tuition and they have merged us with another department at NYU that we're not a part of. The numbers are all different and we can't get them to change it.
Brian Lehrer: Francie, is that indicative of something that goes on with respect to a tug of war between schools and the rankings?
Francie Diep: I haven't heard that specific complaint before, but I'm sure it happens. There are hundreds of schools that US News deals with. I'm sure that it can be hard sometimes to get customer service.
Brian Lehrer: Can you go into that a little more? Go ahead, Janet.
Janet: Yes, or just to get them to read the information correctly and change according to how you submit each year. It seems simple if you have to submit that they take into account what you're submitting.
Brian Lehrer: Are you close enough to this, Janet, to give a really good example of what they're getting wrong?
Janet: Well, I can give you a general thing on they are merging us with a department that has 1,400 students and we have 48. In merging it, the tuition is different for each student in the class size and the reputation of each school is completely different because of the size. Those are the specifics I can know. To tell you the truth, someone might know better than me.
Brian Lehrer: Janet, thank you very much for calling in. We'll continue on this in a minute. Hayam in South Slope, parent of a recent Columbia grad, we see you. You'll be next. Dominique in Suffolk County, who has a college counseling company and has information for all of you on where else to find data about graduation rates and how many of the graduates get good jobs in their fields, she'll pass that along. Stay with us. Brian Lehrer on WNYC.
[music]
Brian Lehrer: Brian Lehrer on WNYC as we talk about college rankings inspired by the story this week of Columbia University being dropped in the US News & World Report rankings from number 2 to number 18 for the current academic year because of missing data about things like class size and graduation rates. Hayam in South Slope, you're on-- Oh, I said you were going to be second. Hang on. Who was I saying was going to be next? What? I guess it was Hayam. Dominique in Suffolk, you'll be next. Hang on. Hayam, parent of a recent Columbia grad, I see. Welcome to WNYC today.
Hayam: Hello, Brian. How are you doing? I would say this is a comeuppance for Columbia. I see Columbia in contrast to my alma mater, City College, which was across the valley there in Hamilton Heights. Columbia is one of the schools that although it has some wonderful, wonderful programs and professors, it's kind of a corporate school. They're unique. Maybe not unique, but they're distinguished, and that they failed to give any discount to their students once the pandemic hit.
As the dorms emptied out and everything went onto virtual instruction, they didn't return any kind of a discount or a reimbursement to their students even as they built their modernist jewel box of a campus down there in West Harlem. I would say, "Hey, 18 isn't so bad." Your tenured math professor did the right thing. This is just another case of patting your resume and getting caught. Maybe like Donald Trump.
Brian Lehrer: [chuckles] Hayam, thank you very much. That's a first-time comparison, Columbia University to Donald Trump. Francie, Hayam mentions a few other things that have been in the news about Columbia, how some of the students wanted partial tuition reimbursements for going to school remotely rather than having the full campus experience when that was in effect. There's also tension with TAs, graduate students who get paid to do things. There's labor issues there. Do those kinds of things play at all into the rankings or are they completely separate realms?
Francie Diep: They're pretty separate. That said, a big part of the rankings is how other university presidents rank your university. That's like 20%, so a pretty good chunk. I don't know--
Brian Lehrer: Wait, a big chunk at how other university presidents, your competitors, rank you?
Francie Diep: Yes, you get this survey from US News as a president and you fill out what you think of all these other universities. That's yet another criticism that's been leveraged against the rankings is that like, "I'm the president of Columbia. Do I really know anything about a university across the country?" For example, it's still on my list of universities that I'm supposed to say something about, so yes.
Brian Lehrer: Interesting. I could see all kinds of potential gamesmanship. I don't know if it goes on. If Yale wants to outbox Princeton for number one, then maybe they'll rank Princeton a little lower and Vanderbilt a little higher. [chuckles] I don't know. Yale doesn't stack up quite as well?
Francie Diep: Yes, this actually has happened. Quite a while ago, the University of Florida's president was found to have ranked University of Florida number one above Harvard, Yale, et cetera on a bunch of these surveys in an attempt to get Florida further up.
Brian Lehrer: I'm sure the president of the University of Florida has nothing good to say about the University of Miami, but that's probably more of a sports story. Okay, Dominique in Bohemia in Suffolk County, you're on WNYC. Hi, Dominique.
Dominique: Hi, Brian. Thanks for taking my call. I'm a longtime listener, third-time caller. [chuckles] I provide college counseling to families through my company, Crimson Coaching. I just wanted to tell families in response to a caller's previous lament that US News & World Report doesn't provide things like job placement and graduation rates. There actually are a lot of places where families can go to get this information for free and also in a much less-biased fashion.
I think US News & World Report is a pretty biased publication for a few reasons. The common data set is our federally-reported statistics, everything from the percentage of students who get Pell Grants to the graduation rates. These, by law, have to be on college websites. They can be difficult to find on a college website. I recommend families googling the name of the college plus common data set. You will find all these statistics there. Another great place to find--
Brian Lehrer: That's a great tip. People take notes. Keep going, Dominique.
Dominique: [laughs] I'm a teacher at heart. [laughs]
Brian Lehrer: Great.
Dominique: Also, under the Obama administration, the Obama administration really tried to make this process much more transparent. They started a website called College Navigator. It is just a wealth of information for every single college in the United States. Families can just google College Navigator and find that information there. I will also give a shout-out to your fellow journalists, New York journalists who had great books recently.
Jeff Selingo and Ron Lieber, both at The New York Times. Anybody who's applying to college right now should really read those books. Selingo's is Who Gets In and Why. Lieber's is The Price You Pay for College. Then if it's okay, I also recommend people go to my website, crimsoncoaching.com, and read my blog, where I talk about all these kinds of issues as well.
Brian Lehrer: After this call, I think people will and it sounds like well-deserved. Crimsoncoaching.com?
Dominique: Crimsoncoaching.com, exactly.
Brian Lehrer: Thank you very, very much. The previous caller brought up campus culture, Francie. I wonder if the rankings even try to get at anything about the culture of a school through their methodology. Columbia, just to take our local Ivy League example that we've been talking about, sometimes gets criticized from the left as not respectful enough of difference on campus, or from the right as to politically correct and going overboard to protect people from racist and sexist speech, depends who the critics are.
I imagine the balance of how that all actually comes out on campus is different. Columbia being Columbia than, let's say, the number 19 school, Notre Dame, given its campus culture in history. Does US News make any attempt to assess any of that and rank schools or describe schools on those value judgments?
Francie Diep: Those kinds of things don't go into the numerical ranking. That said, US News just asks for a lot of information from schools that don't go into the ranking. This is unfortunate. You can, as a student, purchase a subscription like a special subscription to extra data, and then you can, in the profile of the school, look up stuff about what percentage of students are in Greek houses.
More campus culture stuff is available. It doesn't go into the numerical rankings. It does exist somewhere on the US News website if you pay for it. Generally, those kinds of things are hard to find without deeply looking into the school itself, visiting if you have the means to do so, and looking up news articles and stuff about the school you're interested in.
Brian Lehrer: I did see in one of your articles that people say there's more of a social justice bent to the criticism of the rankings recently. How so?
Francie Diep: I think more and more of higher ed, in general, has come to believe that what makes a good college includes more social justice aims like taking a student from a lower socioeconomic class and being able to put them up higher through this education. There's been more criticism of like it just seems unfair like the caller said before that being a top school in this list means you have a lot of money. The students that you accept generally have a lot of money as well. There's not as much reward in the rankings for doing a lot with the budget you have or making a big difference to students who have a tougher time making it through college. Those are the kinds of things that people talk about nowadays.
Brian Lehrer: James in Queens, you're on WNYC. Hi, James.
James: Thank you, Brian, for taking my call. I'm actually concerned that colleges are self-ranking. There got to be a better system. We are talking about the education of our children, the future of our children. This is not satisfactory to me. I think more than one parent will be concerned. If colleges are costing so much money because they already have established a reputation, we need to vet those colleges.
We need to actually see that it's a new issue for us to contemplate before we can make those decisions. I just want to take my answer off the air, but please let us know apart from the lady that said that we have to go to her site, which I'm going to be doing right away. I think there's more than one issue for parents to be concerned about. My son is a straight student. He's going to college. I don't even know [unintelligible 00:33:56] just because the name is sounding good.
Brian Lehrer: [chuckles] What do you think the name of the school might provide, the big marquee name of a school that your son might get into, might provide him in his career and the rest of his life?
James: That is true. That is true, but we all look for a better outcome for our children. If you go among peers and you talk and say my son went to Harvard, he went to Princeton, in the prestigious things, but those name over time is taken for granted because we don't know if they're actually performing to the task that they've been known for. You go in there bragging, "My son is near."
All the students, they don't necessarily say anything about their school because they want to get good employment. "Oh, I come from Harvard," but what is the culture out there? What is the school dispensing as education? Are you actually up to demonstrate your ability to do something because you came from Princeton? Those are questions that are really concerning.
Brian Lehrer: James, thank you very much. Certainly, the reputations of the schools and I guess somewhat based on these US News & World Report rankings, Francie, carry a lot of weight, right? Do you have a degree from an Ivy League school, let's say, any of them, whether it's the number 1 school or the number 18 school that happens to be Ivy League and happens to be located in Upper Manhattan? Employers are probably going to give a lot of people extra points just for having that degree without looking into details of the quality of the education.
Francie Diep: Yes, for sure. It's hard to measure what kind of boost you get just from the name and if that boost is worth the price you pay in tuition. Those are all really hard questions to get at.
Brian Lehrer: All right, we are going to leave it there. We haven't answered all the hard questions, but maybe we've answered some and good information as well from some of our callers about how to look further if you are in college shopping mode right now for yourself or for a loved one. There you go. We thank Francie Diep from The Chronicle of Higher Education. Thank you so much, Francie.
Francie Diep: Thank you.
Copyright © 2022 New York Public Radio. All rights reserved. Visit our website terms of use at www.wnyc.org for further information.
New York Public Radio transcripts are created on a rush deadline, often by contractors. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of New York Public Radio’s programming is the audio record.