Georgia State University uses “big data” to tackle the problem of college completion rates: The ‘Moneyball’ solution for higher education.
The ‘Moneyball’ solution for higher education
It’s a lot easier to start college than to finish. Can ‘big data’ help?
Full Article Published in POLITICO - 01/16/2019
ATLANTA—On a blistering hot June day, Georgia State University incoming freshman Jaila Heathman found herself feted inside the football stadium by cheerleaders, football coach Shawn Elliott, grilled chicken sandwiches and the booming sound of “Uptown Funk.” The academic year wouldn’t start for two months, and Heathman wasn’t a handpicked athletic recruit or an academic star. In fact, she was someone who Georgia State’s computer system decided was just a bit underprepared for college.
Heathman had been invited to the stadium party by an algorithm. For the past seven years, Georgia State has been feeding student data into a “Moneyball”-style predictive analytics system, a custom piece of software designed to figure out who is ready to succeed in college, and how to keep them enrolled. By crunching numbers such as the student’s grades in critical classes like 10th-grade English, Georgia State identified Heathman as among the university’s weaker incoming freshmen, meaning she’s at risk of joining the 31 million Americans who started college and never finished. The stadium event was a highlight of its Summer Success Academy, a pre-college academic prep program that coaches participants selected by the system on skills like managing time, reading textbooks and talking to professors.
“I feel like that shows how much effort they are willing to put into their students,” Heathman said. “A lot of schools, and even professors, they’ll just let you fail and not even care because you’re paying for the class.”
Georgia State’s leaders turned to predictive analytics to fix a long, perplexing problem in higher education: The 4 out of 10 students who start college and don’t finish in at least six years—some never. Education is supposed to be a ladder to prosperity, but in the decades-long push to open college doors wide, an unintended casualty has been the college dropout. The percentage of Americans who started college but didn’t earn a four-year degree has doubled since 1960, with an estimated 20 percent of working adults having some college credits, but no degree—a sign of dashed hopes and, often, the crush of student loan debt without the economic boost of a diploma.
Georgia State University students walk between classes on the campus in downtown Atlanta. The state university has improved its graduation rate by monitoring students using a predictive analytics system. | Raymond McCrea Jones for Politico
In years past, it was just accepted at Georgia State—as it is in much of higher education—that this attrition was inevitable, and a large chunk of students like Heathman would simply fizzle out, to be replaced with a new crop of students. But pressure is growing to change this. The nation’s $1.5 trillion student loan burden weighs heavily on students and schools. The Great Recession reshaped the conversation in higher education to focus more on workplace skills and value. And better data make it easier to track graduation statistics; several states now have performance-based higher education funding systems—all factors pressing on universities to address the issue.
Georgia State isn’t the only university using “big data” to tackle the problem of college completion, but it stands out for two reasons: First, the program has been operating longer than most of the others, and second, because it has improved outcomes with low-income, first-generation and minority students—groups that have the hardest time succeeding in higher education.
For the past seven years, the university has monitored its 52,000 students using 800 different academic risk factors, from how well they're doing in algebra to whether they've started to miss class regularly. It also tracks 14 financial indicators, like unpaid student debt. The goal is to catch students before they reach a crisis point and provide the advising and instructional help to get them through it.
Georgia State’s six-year graduation rate in 2018 was 55 percent, up from 48 percent in 2011 before the program rolled out. That might not sound earth-shattering, but when it comes to college completion, every percentage point is hard-won. The fact that its student body is disproportionately disadvantaged economically and socially also makes this progress notable. These days, Georgia State graduates more black students than any other nonprofit educational institution in the United States, according to an analysis by Diverse Issues in Higher Education, and minority students make up 71 percent of its student body. Nearly 60 percent of Georgia State’s students are eligible for Pell Grants, meaning many come from households making $40,000 or less.
For all these reasons, Georgia State has become a celebrity of sorts as universities desperately look for the secret sauce to improve college completion rates. Microsoft founder Bill Gates has visited the campus to see the data in action, as have Education Secretary Betsy DeVos and John B. King Jr., the Education secretary before her. College administrators from as far away as South Africa have come to observe first hand.
The question they are all trying to answer is whether Georgia State’s model is a one-off, or a recipe that can be duplicated elsewhere. Georgia State’s analytics program is hard to separate from the university's broader investment in academic counseling, which has seen the school vastly expand its advising staff over the past seven years, from 13 professionals to more than 70—a big salary commitment for a cash-strapped state university.
Timothy Renick, vice president and vice provost at Georgia State University, who oversees the campus’ predictive analytics program. “It’s about time we delivered on promises that we made,” he said. | Raymond McCrea Jones for POLITICO
Timothy Renick, a Princeton- and Dartmouth-trained religious scholar who is the Georgia State administrator overseeing the effort, says by adding the additional advisers, the university brought the student-to-adviser ratio down to 320 to 1, which is near the recommended standard by the National Association of Academic Advisers. He notes that while many universities have the same proportion of counselors, Georgia State is one of the rare large public universities whose black, Hispanic and Pell-eligible students have graduated at or above the overall rate of the student body the past four years. The university’s predictive analytics system has allowed it to move the needle, he says, in ways the others haven’t.
Still, even with that improvement, nearly half of Georgia State students still fail to graduate within six years. And skeptics of data-analytics programs have raised worries that their intensive student tracking borders on Big Brother-style surveillance—and on substitute parenting, at a stage in life when students have a right to privacy and should be rising and falling on their own decisions. For his part, Renick sees the ethical equation differently, calling the system a “moral imperative.”
“For generations, well-resourced institutions have been very willing to sell the promise of a college degree to low-income families. And these families have often trusted us their sons and daughters, and also their finances, taking out loans and putting great hope in the promises that we offer,’’ Renick said in an interview. “It’s about time we begin to deliver on promises that we’ve made.”
WITH THE EDGE of campus just a few blocks from Martin Luther King Jr.’s Ebenezer Baptist Church, Georgia State has helped bring life to a gloomy section of downtown Atlanta as its ambitions grew over the past two decades from a commuter school to an urban residential campus, constructing new student housing and classrooms, taking over and renovating old office towers. The nerve center of the advising system is located on the fourth and fifth floors of a 26-story white stone tower that used to be a bank; Coca-Cola used to store its secret formula in a basement vault.
Georgia State University graduates more African American students than any other non-profit educational institution in the United States. | Raymond McCrea Jones for Politico
Eric Cuevas, the director of student success at Georgia State, said many universities put the student advising center in a basement office. “We get priority placement,” Cuevas said.
When academic counselor Emily Buis arrives at work each morning, she opens her computer to a “dashboard” and a spreadsheet she uses to track each of the 250-plus students she advises, and every interaction she has with each one. Thanks to the school’s prediction system, each student on her screen is identified with a “risk level” of green, yellow or red. She can also see the data behind why the student is coded this way; a STEM student underperforming in a math class, for example, might need extra attention.
The goal for Buis is to communicate by email, phone or in person at least once with each student assigned to her within the first four to five weeks of the semester. A few months later, Buis individually reviews each student’s registration for the next semester to ensure they are in the right classes to keep them on the path to graduation; those who haven’t registered get a phone call.
In between, an “early alert” from a professor to Buis with concerns about any aspect of a student’s performance will prompt her to reach out to the student. Sudden drops in GPA or low early quiz scores are cause for concern. She also reaches out to students assigned to her through various “campaigns”; one week, for example, she’ll check in with all her pre-journalism students. Buis said it sounds “cheesy,” but “all of my students are on my radar.”
Kiara Collins, an adviser at Georgia State University, offers guidance to a student using the university’s predictive analytics computer system. | Raymond McCrea Jones for POLITICO
The heart of Georgia State’s system is its predictive analytics engine, an algorithm built in 2012 with 10 years of data that included 144,000 student records and 2.5 million grades. It’s updated nightly, and the data is scoured to look for patterns in student performance.
Georgia State’s partner in developing the system was EAB, a Washington-based firm. Renick said the system—groundbreaking when it was developed—was built on ideas from the health care industry, where hospitals would scour large data sets for early signs a patient was at risk.
For all the details collected about each student, the main emphasis of Georgia State’s program is ensuring students get a few big things right: that they're in a major well-suited to their talents, taking classes in the correct sequence, and not loading up their schedule with classes they don’t need. In 2018 alone, university officials said they did nearly 3,000 “course corrections” to make sure students were enrolled in the right classes for their major. Before the system was rolled out, Renick said the average Georgia State student was taking more than 20 “wasted” credit hours that did not apply to any graduation requirement; now, extraneous courses have been reduced so much that students are graduating on average a semester sooner.
Each adviser has an extra screen on his or her desk that shows students what the system predicts about their chance of success in a major based on the academic records of previous Georgia State students. A student wanting to go to law school who has gotten Cs in political science classes but As in English might be encouraged to switch to an English major but keep their dream of going to law school. The system signals that those who get a C in math pass chemistry at just a 40 percent rate, so weaker math students are encouraged to get more support before they attempt chemistry.
It might sound impersonal, but counselors have found that the predictive analytics system is sometimes more effective as a tool, since it gives students data to work with, rather than advice from a counselor they may or may not know or trust. “You turn the screen around and they make their own decision,” said Elisha Jarrett, associate director of the university advisement center.
For Jarrett, the real benefits of the system have been its ability to flag students who aren’t failing but might be quietly struggling, and for whom the right support in time would make a huge difference. “The student that got the B-, the C, or maybe the C-, we were failing those students,” Jarrett said. “We didn’t have a way to really reach out and capture those students and make sure those students were coming in.”
For many Georgia State students, their first encounter with the predictive analytics system is an email from an adviser asking the student to come see them. Casey Prout, a sophomore who wants to be a nurse, got one of those emails her first week of freshman year after a low score on a chemistry quiz. She met with an academic coach who discussed study strategies with her. Ultimately, she said, she got a high ‘C’ in the class—enough to keep her on track in her major—and was glad she stuck it out. “I was like, oh my gosh, how did they already know I struggled in chemistry?” Prout said. At first, she said, she felt singled out. “But then, once you go in there, they reassure you: ‘We caught it early, and we wanted to make sure you keep going from here’.”
At first the system felt novel for the students and counselors. But today, the emails and other supports built around the predictive analytics system are so second nature for Georgia State students that some don’t even know the program is unusual.
“They don’t do that?” said student Cabria de Chabert when told other universities don’t send out similar alerts. “There’s only so much a person can do coming out of high school without the experience. You’re going to need a little push, so I appreciate it.”
Cabria de Chabert, a junior at Georgia State University, says she appreciates the nudges she gets from her adviser and the university’s data analytics system. “There’s only so much a person can do coming out of high school without the experience,” she says. | Raymond McCrea Jones for Politico
The data have identified some unexpected patterns that signal trouble; the system also identified “toxic combinations” of classes, and now encourages students not to take these classes in the same semester. The university determined, for example, that students might pass physics and organic chemistry at good rates if taken separately, but struggle if they take them at the same time.
Both the College of Business and the School of Nursing revamped their academic prerequisites after the number crunching. In the business school, students used to need a 2.5 GPA to take upper-level classes—leading about 1,000 students hovering right below that cutoff to take “junk” classes for multiple semesters as they attempted to improve their GPA, said Allison Calhoun-Brown, the associate vice president for student success. Today, the requirement is more focused: business students need a 2.8 GPA in five select classes deemed more critical for them, which addressed the problem, she said. In the nursing program, it was long believed that a two-course sequence in Physiology and Anatomy was the most important predictor of success. But Georgia State’s data found a student’s grade in his or her first math class was more predictive than either one, so the math score is now used to determine whether a student is admitted into the program, Renick said.
The university is constantly looking for other ways to put the data to work. After it learned that 20 percent of its incoming freshmen were dropping out the summer before they even started, it developed an artificial intelligence “chatbot” to answer common questions about starting college. The bot handled 200,000 queries—and immediately delivered results. About 320 additional students attended that fall compared with the patterns of previous years.
The university also pays close attention to students’ finances. Georgia State wants to know if a student has financial holds or an unpaid balance, or has exhausted eligibility for a Pell Grant. Since even small-scale financial stresses cause dropouts, the university pays out about $2 million annually in microgrants to students identified as needing financial help. A $300 grant is enough to keep some students in school.
Looking ahead, the university is exploring the possibility of monitoring each student’s “electronic footprint” systematically to see if it can be used as an early warning sign that a student is in distress. The hope is that if a student who had been regularly signing onto campus Wi-Fi and electronic course platforms were suddenly to stop, the university would be able to find a way to intervene early.
Students don’t always welcome being singled out, such as the incoming freshmen told they need to participate in the Summer Success Academy. But Calhoun-Brown said the school tries to approach students and parents with a truthful, but respectful tone, emphasizing that the program has identified the student as someone who could benefit from extra assistance—similar to what a Division I athlete would receive.
“We are trying to communicate that we believe in the students,” Calhoun-Brown said. “We are trying to build our reputation at Georgia State on not who we exclude, but who we include, and who we succeed with.”
Elliott, the football coach, says the university’s rising graduation rates have even become a recruiting tool to attract athletes. Elliott said his players are in the advising center almost daily, and the analytics system has become a selling point with prospective players’ parents, who want to make sure their children graduate.
“A lot of universities talk about eligibility but … when they step foot on campus we’re talking about graduation,” Elliott said. “That’s the ultimate goal.”
IN THE YEARS since Georgia State rolled out its system, multiple vendors have begun offering similar services, and the use of predictive analytics as a student success tool has become much more prevalent. Demographic data, grades, test scores and attendance records are all scoured for clues to student success or failure.
As with many appealing technologies, few of those adopters are using the tool to its full potential, says Bridget Burns, an expert on higher-education innovation. Burns, executive director of the University Innovation Alliance, sees Georgia State as an example of one of the few universities truly harvesting the potential of predictive analytics. “There is not a light switch. You do not plug a predictive analytics system into your campus and all of a sudden it’s a miracle. It doesn’t work like that. It is complicated. It is difficult,” Burns said.
Bruce Vandal, a senior vice president at Complete College America, a nonprofit focused on raising college graduation rates, said systems that use data to track and then intervene to help students are the wave of the future, but are only one part of the solution. Getting universities to focus on student support services, including advisers, can be a daunting task. This often isn’t where colleges direct their attention, he said. In the race to build flashy student facilities and hire top faculty, “getting institutions to redistribute their resources to these types of activities is a daunting one,” Vandal said.
At Georgia State, the technology is a tiny fraction of the university’s advising costs.The university spends about $150,000 annually on the technology, and the additional advisers cost about $2.5 million. It might seem hard to justify at universities that are increasingly run like businesses. But Renick says the cost-benefit is more than clear: Dropouts mean lost revenue for the school, and the university generates more than $3 million in tuition and fees for every 1 percentage point increase in students who stay enrolled. He estimates that the university saves millions with the program.
Georgia State’s model isn’t the only way to use an analytics program. Arizona State University, one of the nation’s largest universities, with more than 100,000 traditional and online students, offers an advanced online adviser system that requires students to take courses early on deemed diagnostic for success in the major they’re pursuing. It also aggressively tracks attendance and reaches out to students who miss class. The University of Texas has a comprehensive initiative focused on getting students graduated in four years that uses predictive analytics to rate each student’s chance of graduation before they step onto the campus and builds freshman housing groups using the data. Both schools have seen increases in graduation rates.
Some university leaders envision a time when predictive analytics systems in higher education could become even more sophisticated, drawing a portrait of the whole student that incorporates social activities such as Greek life membership, study abroad participation and swipe-card data that track the time students are spending in a campus gym or computer lab—all activities that universities use to help group students or track them in other ways to help them succeed.
Georgia State University students walk between classes on the campus in downtown Atlanta. | Raymond McCrea Jones for Politico
Amid the excitement about these types of systems there are also some ethical concerns, including that they are a little too Big Brother. Alan Rubel, director of the Center for Law, Society and Justice at the University of Wisconsin, said it may be true, for example, that a student who is political or religious might be more successful as a student, but that doesn’t mean the university should track whether students go to campaign rallies or church services. Ethics concerns have also been raised about building students’ health and criminal records into the systems.
“I don’t think universities ought to be in the business of collecting that information in the first place, regardless of whether it’s relevant to our mission,” Rubel said. “There are some things, surely, that universities ought not to be in the business of collecting or corresponding to academic concerns.”
Beyond what universities are doing, Rubel said there are too few safeguards restricting what third-party vendors and researchers can do with the data. Georgia State says it follows federal privacy laws and that student data are passcode protected, housed behind firewalls and accessible only to those authorized and trained to use them. Renick emphasized under Georgia State’s approach, it’s not collecting much information that it didn’t before it rolled out the system—the difference is that the insights are shared with students.
Others worry that while the universities’ motivation might be good, the systems might be pushing low-income and minority students away from science, engineering or math fields by reinforcing biases and inequities in America’s education system.
E. Gordon Gee, a longtime college administrator who is the president of West Virginia University, said he views predictive analytics as an important safety net but worries the systems can’t account for serendipity, such as the students he’s seen over the years who discover an academic passion and quickly change academically for the better.
“We don’t want to become mechanical in how we treat people,” Gee said.
Iris Palmer, a senior education policy analyst at the New America think tank who tracks the use of predictive analytics in higher education, said it’s a “really hard line” for universities to walk. “On one hand, if you know that a student doesn’t have the academic record to be successful in a major like nursing or engineering, don’t you have the obligation to tell them that ahead of time before they waste their time and energy? And shouldn’t you be able to say, ‘Hey, there’s this alternative you might want to try to be better suited to your academics’?” Palmer said.
On the other hand, Palmer said, the systems have to be rolled out carefully because “we have historical inequalities and structural and racial issues that can be embedded into their systems if we’re not careful in how to build and how we implement the systems.”
At Georgia State, Renick said the system doesn’t use factors such as race, ethnicity or income when it comes to making predictions. He said the university has 100 percent more students graduating in STEM majors since 2011, in part because the university is able to offer additional supports to help get STEM majors to graduation.
For all its successes, the system isn’t perfect. After the end of her first semester at Georgia State, Heathman said she didn’t feel the same enthusiasm for the university that she had months earlier. While she enjoyed the stadium celebration and appreciated the support she received, Heathman said it was too “sporadic.” She said she failed algebra after getting confused about the start time for her final, so she’ll have to retake the class.
But importantly, she’s still enrolled in school.
“Basically, I just have to talk to my counselor to figure out what’s going on before it ends with me having a super-low GPA,” Heathman said.
Kimberly Hefling is an education reporter for POLITICO Pro.
Authors:
Kimberly Hefling khefling@politico.com @@khefling