Deep learning system prioritizes tasks and selectively forgets
Deep learning system prioritizes tasks and selectively forgets Heading link
Assistant Professor Amit R. Trivedi is developing a next-generation deep learning accelerator — a system that can understand which tasks are most important and prioritize the data it’s processing.
The accelerator, Neuroplane, is being developed with Mark Hersam at Northwestern University and Anand Raghunathan at Purdue University. The trio received a $1.2 million grant from the National Science Foundation (NSF) in support of the project.
Hersam’s group is developing new computing materials such as molybdenum disulfide, a flexible, two-dimensional material enabling single atomic layer channel transistors. Trivedi is creating dynamic architectures for deep learning that automatically reconfigure themselves depending on the complexity of the task. And Raghunathan’s team is working on the end-stage, connecting these accelerators to applications.
Conventional computing technologies have been pushed to their limits with the increasing complexity of deep learning systems. To improve the efficiency of deep learning, designers have introduced layers such as hypernetworks — which can weigh the input against the main network — and attention networks, which embed inference, allowing the technology to directly draw conclusions from the data it receives.
But these layers add to the amount of high-order processing, which uses space and power. To minimize this, Neuroplane will limit the number of operations. A complex task can leverage the entire computing engine, but for a simpler task, the architecture will automatically prune out its depth to minimize energy consumption for computing.
“We are not only doing this processing, but these new parameters and new architecture are trying to understand: How difficult is this task, and should I really do all the computations? And if I do all the processing, then I can do things very lightly,” Trivedi said. “The system can then say, ‘I can make the same predictions, but I don’t have to use the same amount of energy.’”
This technology will also selectively ‘forget’ some of what it’s learned.
“In our human brain, unless we forget, you can imagine how terrible life can become. Forgetting allows us to learn new things,” he said. “The technology can understand what tasks are more important and focus on them today. Tomorrow, they become outdated and obsolete. It can then selectively forget.”
This platform will allow devices, including cell phones, to have more power than what is possible today. For example, current cell phones can perform data processing, but their limited hardware, size, and battery constraints prevent some high-performance operations such as artificial intelligence operations and optimization, which can be conducted on PCs.
These new materials and circuits will allow the team to build the power of a PC into a phone.
This project is one of Trivedi‘s sixth concurrent grants. He has been able to attract funding from a wide variety of sources, including an NSF CAREER Award, a U.S. Department of Defense equipment grant, a National Institutes of Health award, and an industry award from Intel Labs. In the last two years alone, Trivedi has brought in more than $1.25 million in outside funding to UIC.