Don’t show me the money!
Or at least show me something more indicative of technology transfer success at universities and non-profit research institutes: such is the growing sentiment within the tech-transfer community as it attempts to paint a better picture of how universities are transforming basic research into commercial products.
As such, a number of professional, government, and academic organizations, such as the Association of University Technology Managers, the US Department of Commerce, and various academic research teams are currently examining alternative metrics for evaluating the success of university tech-transfer activities.
The idea of adopting alternative metrics for tech transfer is particularly salient in the wake of AUTM’s US Licensing Activity Survey for fiscal year 2006, released last week (see related story, this issue).
The AUTM survey includes information on licensing income for participating universities and non-profit research institutes, but it does not specifically rank schools by those numbers. According to some experts, however, the media and the general public tend to seize upon those numbers as the best measure of success, which can portray university tech-transfer offices as being concerned only about the money.
“The difficulty is that [funding sources] outside of the university — whether they be governors or senators or others in Washington — are continuing to look for those productivity measures,” Lesa Mitchell, vice president of advancing innovations at the Kauffman Foundation, told BTW last week. “Unfortunately that usually looks back and focuses on licensing, which causes the circular problem.”
To address this problem, AUTM, with the help of the Kauffman Foundation, is attempting to change the type of data it collects from universities and they way it presents that data in its annual licensing activity reports.
According to Mitchell, officials from Kauffman and AUTM met two years ago to discuss different ways the organizations could better communicate the impact of tech transfer on the economy and the greater good. Kauffman ended up providing AUTM an undisclosed amount of funding to support two projects.
The first was the Better World project, a series of publications by AUTM intended to promote public understanding of academic research, technology transfer, and its impact on the general public and the economy. The publications provide mostly qualitative case studies of tech-transfer successes — that is, real products that are currently in the marketplace that were spawned from university research, and the stories of how they got there.
“The beauty of that is that it draws attention to innovations coming out of the university that really don’t have anything to do with licensing income, but are showing the real value of innovation in the commercial marketplace and how they are changing lives,” Mitchell said. “Our idea was to distract the attention from licensing income, and place it on outcomes.”
The second project was the formation of a task group to broaden licensing metrics. In collaboration with organizations such as the National Governors’ Association and the American Association of Universities, AUTM has surveyed stakeholders such as policy makers, legislators, funders, and university leadership to identify what information about tech transfer is important to them.
According to Dana Bostrom, vice president of metrics and surveys for AUTM, the organization plans to begin implementing these changes in the 2007 AUTM survey, which should be distributed to participants by summer of 2008.
“First, they wanted more information on transactions,” Bostrom said. “The licensing survey talks about licensing, which is a big activity in our offices, but by no means all of it. So we’ll probably start asking more about inter-institutional agreements; research agreements; or materials transfer agreements.”
Bostrom said that task force also identified an interest among stakeholders in the scope of resources and the structure at tech-transfer offices. As such, Bostrom said that AUTM plans to include new licensing activity questions for the 2007 survey such as: “What is the responsibility of your office? Are you responsible for insignia licensing, MTAs, licenses, [and/or] collaborations? How much do you have to do?”
Lastly, Bostrom said that AUTM plans to adopt a “goals” survey to better determine what goals tech-transfer offices are trying to achieve, and how they relate to the overall goals of their universities. “Hopefully they’re very congruent,” Bostrom said.
There is some uncertainty surrounding this last task, Bostrom added. “I’m not sure how we’re going to proceed with a goals survey,” she said. “We know there is some interest, and that folks have done some related work, but I think AUTM needs to study that a little bit more before we can deploy it in a way that will be helpful.”
Varying Measures of Success
Bostrom, who is also director of innovation and industry alliances at Portland State University, said that one of the biggest challenges in identifying performance metrics is that different universities measure tech-transfer success in different ways.
“When I started this metrics project almost two years ago, I said that I wanted to measure goals, and put universities in groups of what they are most trying to achieve,” Bostrom said.
|
“I think anything would be a better metric than licensing income, simply because it sends the wrong message.” |
“I had just come from UC-Berkeley, and they would tell you that their goal was to move technology out into use, but also to generate research income. Generating licensing revenue was very low in priority,” she added. “If you talk to other universities, they will tell you that generating licensing income is their number one priority. There are just different goals that different universities have, but they’re not well-stated on an aggregate basis.”
Still, Bostrom said that she “gets the sense” from her colleagues that “they don’t look so much at income but at [invention] disclosure numbers, staffing numbers, or active agreements.”
Some have suggested that a better way to gauge tech-transfer success at universities would be to compare the amount of research funding a university receives with the number of licenses it executes, inventions it discloses, or patents it files or is awarded. The problem with this approach, according to Kauffman’s Mitchell, is that people can fall into the same trap examining some of those metrics as they would examining licensing income.
“One would assume the right answer is to measure the volume of innovations moving out of the university,” Mitchell said. “Of course the difficulty then is if you’re simply looking at numbers of licenses, you can’t measure that in the same way, because it could be 100 licenses on one technology.
“But if in fact we could understand volume, and what it means in terms of impact outside of the university and into the commercial marketplace, one would assume that would be a better metric — comparative to R&D funding — than licensing income,” she added. “I think anything would be a better metric than licensing income, simply because it sends the wrong message.”
Alternative Approaches
Other organizations and research groups are also attempting to address the metrics problem. For example, a team led by Ashley Stevens, director of Boston University’s Office of Technology Development; and including Mark Rohrbaugh, director of the National Institutes of Health’s Office of Technology Transfer, is putting the finishing touches on a study that will attempt to quantitatively measure the impact of university technology transfer on currently marketed pharmaceutical products (see BTW, 6/25/2007).
The group released preliminary data from the study at last year’s annual AUTM meeting in San Francisco, and is planning to unveil an updated poster at the 2008 annual meeting in San Diego, according to Bostrom.
Mitchell applauded the work, but added that it is “not too different from case studies in that it’s going to give us a slice of life for a specific period of time, but it doesn’t necessarily help us on an ongoing basis.”
In addition, last year the US Department of Commerce formed an advisory committee entitled, “Measuring Innovation in the 21st Century Economy.”
The group of 10 CEOs and five academics, led by Kauffman president and CEO Carl Schramm, first met in February to outline its goals, and held a second meeting on Sept. 12 to finalize a set of recommendations that are due for publication before the end of the year, although an exact date has yet to be released.
Bostrom said that the committee has developed a list of 12 recommendations for new metrics to measure innovation at both US companies and universities.
“One of the things they emphasize in their report is that while you can do a lot of things with companies financially in terms of discussion on a greater economic analysis, transfer pricing, and tax-related stuff, it’s not necessarily relevant to universities,” Bostrom said.
“But one of the things they have said is that a lot of times with universities you’re just going to have to be satisfied with stories,” she added. “It’s unfortunate, but I don’t know that anyone has come up with a better mechanism at this point.”