Wednesday, June 25, 2014

For those interested, I highly recommend looking at the slides and recorded presentations form the Neuromorphic-Inspired Computational Elements Conference - http://nice.sandia.gov/videos.html

It started last year and was expanded greatly for this past February's conference to a 3-day event.

There were many outstanding talks there on many levels and describing many variations of the approaches all aimed at developing 'brain-inspired' alternatives to our Von Neumann/Turing architecture in place now for almost 70 years.

In my mind, the best individual approaches they and several others have described over the past few years cover most of the topics brought up at RC2 but in a much more comprehensive and cohesive manner. That is, these approaches contain the key elements of 'approximate computing', 'augmented CMOS', and of course neuromorphic.

There are other even more exciting and capable research projects going on which weren't discussed at this event, but I collectively there has been tremendous progress over just the past couple of years which has generated a strong new momentum which in turn will further accelerate the work to develop the next computing platform. Conferences, journals, workshops, and new funded programs have been developing to fuel this momentum. Despite all of this, most people it seems aren't yet fully aware of all of this activity, but that issue too is starting to be addressed as the main stream media has begun to pick up on this emerging world.

It's an exciting time to be involved in, and although similar in feeling to the way we felt in the 70's and 80's when both computing and communications went 'personal', the possibilities being opened by these new approaches are many times greater.

Monday, May 12, 2014

Background reading of value


"For those attending the IEEE Rebooting Computing Workshop this week with an interest in neuromorphic computing, we recommend:
 
'Finding a roadmap to achieve large neuromorphic hardware systems' by Jennifer Hansler and Bo Marr (http://journal.frontiersin.org/Journal/10.3389/fnins.2013.00118/full).  This paper provides interesting and relevant material for the workshop that is expected to help guide our discussion."
 
Dave Mountain, Facilitator for the Neuromorphic Computing discussion at RC2

Tuesday, April 29, 2014

Approximate Computing: A Path beyond the Tyranny of Digital Computing


Over the course of past four decades, consistent and exponential improvement in transistor scaling coupled with continuous advances in general-purpose processor design has exponentially reduced the cost of raw material for computing, i.e., performance; making it a pervasive commodity. While in 1971, at the dawn of microprocessors, 1 MIPS (Million Instruction Per Second) cost $5000, today the same level of performance costs less than 5¢. This exponential reduction in the cost has made the computing industry, the industry of new capabilities. We are not an industry of replacement whose economic cycle relies on consumers replacing inventory when they run out of products. It is even hard to perceive running out of a software app such as Microsoft office. Instead, the computing industry’s economic ecosystem relies on continuously providing new capabilities both at the device and at the service level.

Before the effective end of Dennard scaling, we consistently improved performance and efficiency while maintaining generality in general-purpose computing. As the benefits from scaling diminish and the current paradigm ofthe microprocessor design, multicore processors, significantly falls short ofthe traditional cadence of performance, we are facing an “iron triangle”; we can only choose any two of performance, efficiency, and generality at the expense of the third. Energy efficiency now fundamentally limits microprocessor performance gains. These shortcomings may drastically curtail computing industry from continuously delivering new capabilities, the backbone of its economic ecosystem. Hence, developing solutions that improve performance and efficiency, while retaining as much generality as possible, are of outmost importance.

Radical departures from conventional approaches are necessary to provide energy-efficacy and large performance gains for a wide range of applications and domains. One such departure is general-purpose approximate computing, where error in computation is acceptable and the traditional robust digital abstraction of near-perfect accuracy is relaxed. Conventional techniques in energy-efficient computing navigate a design space defined by the two dimensions of performance and energy, and traditionally trade one for the other. General-purpose approximate computing explores a third dimension, error, and trades the accuracy of computation for gains in both energy and performance.

This radical approach in general-purpose computing will only be beneficial if a large body of applications can tolerate error during execution. Fortunately, as the landscape of computing is changing toward providing more personalize and more targeted experience for the users, a vast number of emerging applications are inherently approximate and error-resilient. These applications can be categorized to four classes:
(1) Applications with analog inputs (wearable electronics, voice recognition, scene reconstructions).
(2) Applications with analog output (multimedia).
(3) Applications with multiple possible answers (machine learning, web search, heuristics).
(4) Convergent applications (big data analytics, optimizations).

More importantly, in this realm of computing, the rate of data generation and collection is growing overwhelmingly beyond what conventional computing platforms can process. By trading off computation accuracy for gains in performance and efficiency, general-purpose approximate computing aims to exploit this emerging opportunity in the application level to tackle the aforementioned fundamental challenges in the transistor and architecture level. One may visualize these trade-offs as finding the Pareto-optimal points in the processor design space, as shown below. Traditionally, for any set of workloads, the set of possible processor implementations may be plotted, with energy efficiency on one axis and performance on the other, and the best implementations residing on the two-dimensional frontier. When approximation is supported, the degree of permissible error represents a third axis. The Pareto surface in this three-dimensional space represents the best points of performance, efficiency, and error. However, this surface is not yet well understood. Navigating this three dimensional space provides many opportunities for innovation across the entire system stack.



As an instance, analog circuits inherently trade accuracy for significant gains in energy-efficiency. However, it is challenging to utilize them in a way that is both programmable and generally useful. In our most recent work that will be presented in International Symposium on Computer Architecture (ISCA) on June 2014, we propose a solution—from circuit to compiler—that enables general-purpose use of limited-precision, analog hardware to accelerate “approximable” code—code that can tolerate imprecise execution. We utilize an algorithmic transformation that automatically converts approximable regions of code from a von Neumann model to an “analog” neural model.   The core idea is to learn how aregion of approximable code behaves and automatically replace the original code with an efficient computation of the learned model. The neural transformation of general-purpose approximable code provides an avenue for realizing the benefits of analog computation while targeting code written in conventional languages. The insights from this work show that it is crucial to expose analog circuit characteristics to the compilation and neural network training phases. At run time, while the processor executes the program, it invokes a reconfigurable accelerator, which we named Neural Processing Unit (NPU), instead of running the original region of code. Our most recent work reports on the design and integration of a mixed-signal NPU for general-purpose code execution. The NPU model offers a way to exploit analog efficiencies, despite their challenges, for a wider range of applications than is typically possible. Further, mixed-signal execution delivers much larger savings for NPUs than digital. Analog neural acceleration provides whole application speedup of 3.7× and energy savings of 6.3× with quality loss less around 10%. Even though the results are very encouraging, there are still several challenges that need to be overcome. The full range of applications that can exploit mixed-signal NPUs is still unknown, as is whether it will be sufficiently large to drive adoption in high-volume microprocessors. It is still an open question how developers might reason about the acceptable level of error when an application undergoes an approximate execution including analog acceleration. Finally, in a noisy, high-performance microprocessor environment, it is unclear that an analog NPU would not be adversely affected. However, the significant gains from A-NPU acceleration and the diversity of the studied applications suggest a potentially promising path forward. This work also shows how relaxing the abstraction of near-perfect accuracy can provides a bridge between two disjoint models of computing, neuromorphic and von Neumann.

Despite its great potential, practical and prevalent use of general-purpose approximate computing requires techniques that seamlessly integrate with the current well-established practices of programming and system design and provide a smooth and evolutionary adaptation path for this revolutionary paradigm in computing.

In general, when conventional approaches run out of steam, it is time for extreme creativity. In fact, we may be living the most exciting era of computing!

Hadi Esmaeilzadeh is the Catherine M. and James E. Allchin Early Career Professor of Computer Science at Georgia Institute of Technology. His dissertation received the 2013 William Chan Memorial Dissertation Award from University of Washington. He founded the Alternative Computing Technologies (ACT) Lab, where he works with his students on developing new technologies and cross-stack solutions to develop the next generation computing systems for emerging applications. Hadi received his Ph.D. in Computer Science and Engineering from University of Washington in 2013. He has a Master’s degree in Computer Science from The University of Texas at Austin (2010), and a Master’s degree in Electrical and Computer Engineering from University of Tehran (2005). Hadi received the Google Research Faculty Award in 2013.
Hadi’s research is recognized by three Communications of the ACM Research Highlights and three IEEE Micro Top Picks. His work on dark silicon has been profiled in New York Times.




Wednesday, March 26, 2014

Approximate computing

In an effort to set up a community around approximate computing, we are asking participants in Rebooting Computing that are knowledgeable about this topic to post references to explanatory material.

Approximate computing is an approach to  low-energy computing. There are general approaches toward lowering the energy consumption of a computer that would have the side effect of reducing the reliability of the computation and results. This effect is mitigated in approximate computing through algorithms that tolerate uncertainty where it is unimportant to the answer and bolster uncertain information through codes and other algorithmic methods.

We request that people posting technical materials make themselves available to explain the material or to participate in a dialog.

Neuromorphic processing

In an effort to set up a community around adiabatic and reversible computing, we are asking participants in Rebooting Computing that are knowledgeable about this topic to post references to explanatory material.

The scope of this thread can include both analog and digital brain-inspired or neuromorphic approaches.

Living brains use chemical and electrical analog behavior in synapses and soma (nerve cell bodies) to compute in the ways needed for survival.The analog scope of this thread is artificial systems with a human-engineered device or circuit taking the place of a synapse neuron. These systems are sometimes called "neuromorphic."

However, Artificial Neural Networks (ANNs) can be implemented digitally using fixed or floating point values in lieu of analog signals. The digital scope of this thread would be devices and architectures that would execute the neural algorithms with speed and energy efficiency much higher than possible with a von Neumann computer.

We request that people posting technical materials make themselves available to explain the material or to participate in a dialog.

Extending CMOS technology direction

In an effort to set up a community around extending CMOS, we are asking participants in Rebooting Computing that are knowledgeable about this topic to post references to explanatory material.

We are defining this topic area to include topics at or beyond the current frontier of technology. There appears to be a consensus that CMOS and Moore's Law will continue only for a finite period of time, with this blog thread focused on what comes beyond that time. To be true to the phrase "rebooting computing," the systems envisioned by this thread should include a new active device that has circuit behavior distinctly different from a transistor, a new circuit distinctly different from CMOS, or a new device that serves a special role in a non-von Neumann architecture.



We request that people posting technical materials make themselves available to explain the material or to participate in a dialog.

Saturday, March 22, 2014

update of low-power image recognition competition

Committee Members:

These are the current members in the committee.  If you want to join (or you want to recommend someone), please send email to me (yunglu@purdue.edu). Thank you.

  • Alex Berg (UNC)
  • David Kirk (Nvidia)
  • Yung-Hsiang Lu (Purdue)
  • Gi-Joon Nam (IBM, representative of ACM SIGDA)

We are inclined to remove the idea of using a display for giving the test data.  This will introduce too much uncertainty in data acquisition through cameras. Instead, we plan to give the data to participants through networks directly.

The competition will be held in mid 2015.  The rules will be announced in summer 2014.

Adiabatic and reversible computing

In an effort to set up a community around adiabatic and reversible computing, we are asking participants in Rebooting Computing that are knowledgeable about this topic to post references to explanatory material.

We request that people posting technical materials make themselves available to explain the material or to participate in a dialog.

Thursday, March 20, 2014

RCS 2 organizational comments


The following points were discussed by the organizing committee.

Rebooting Computing participants are asked to find future technology options in an expansive way. There is a lot of precedent for identifying technology options that are subjectively or quantitatively better than current technology. However, Rebooting Computing encourages participants to look further. First, the current technology (CMOS, or Moore's Law) will continue to advance for some time. Technology options should be compared against the endpoint of road maps rather than current products. Furthermore, there are multiple radical technologies under consideration. It will be important to see how the forward-looking technologies stack up against each other.


RCS 2 is viewing applications as the driver for technology requirements. The committee is proposing three applications classes:
  • Mobile computing
  • Learning and reasoning, based on the "executive assistant" from RCS 1
  • Servers and supercomputers
The four technology classes being proposed are (in no particular order):
  1. Neuromorphic, or brain inspired
  2. Incremental improvements on CMOS
  3. Adiabatic and Reversible
  4. Approximate computing
Many of the combinations of the applications and a technology are currently topic of research:

 

(*NLP = Natural Language Processing; *CNT = Carbon Nanotube; *JJ = Josephson Junction)

The organizing committee discussed the following timeline:

Pre-RCS 2: The blog and website will be seeded with information about Rebooting Computing. Also, some participants will be encouraged to create summary information of technology areas. These will provide context for the program at the RCS 2 event.

At RCS 2: Participants will be encouraged to form groups around self-consistent technology ideas. For example, people advocating a hardware technology in conjunction with people interested in a specific aspect of the applications.

Over summer 2014: Participants will be encouraged to maintain a persistent involvement through the website, blog, e-mail, and personal interactions.

At RCS 3, around October 2014: The goal of this event will be to enable participants to offer "rebooted computing" ideas in forum of people that can appreciate the ideas, including contrasting multiple approaches to computing.


Wednesday, January 29, 2014

competition for energy-efficient computing (image recognition)

This figure shows the idea of the competition. The inputs will be a set of images. The score is the ratio of the accuracy and the amount of energy consumed. The participants have the freedom to offload computation.














The competition will be held in 2015 with a major conference (one candidate is IEEE International Symposium on Circuits and Systems in June). The winner must be able to recognize or classify at least half of the images (i.e. the accuracy must exceed 50%).
Prizes:
  • first prize is a cash award of USD$5,000
  • second prize $3,000
  • third $2,000. 
The participants will receive a set of training data at least one month in advance.

Tuesday, January 28, 2014

questions for energy efficiency in Rebooting Computing 2

Hello, Everyone,

The RC (rebooting computing) committee gave me the tasks of writing a
set of questions to discuss for the blogger as well as the next RC
Summit.  Here is my first attempt.  Please feel free to comment and
change the list.  Thank you.

What technologies have the potential replacing CMOS?  Where are they
standing now?  What are the likely timelines?

Will Boolean logic be replaced?  What are the candidates?

Will future computers be hybrid, including some parts that can produce
precise answers (number crunching) and some parts that produce
approximate answers (such as recognition)? 

Will it be possible to power most of the "edge devices" (sensors,
mobile phones, tablets) by ambient energy sources?  If so, how?  If
not, why? 

What is the theoretical minimum energy for computing, communication,
and storage? 

Let me further define the units of computing, communication,
and storage.

One unit for computing is one double-precision floating-point
division.

One unit of communication is sending 1MB and receiving 1MB data
(excluding headers) for wireless: 10 meters away; for wires or fiber:
100 meters away.

One unit of storage is saving 1MB data (without metadata).

What is the state of the art in the energy consumption for computing,
communication, and storage?

What is the distance between the theoretically minimum and the current
state of the art?

What are the barriers reducing the distances?

How are different power management solutions evaluated? Do you think
future electronics should have "energy counters" that can report their
energy consumption?  Would this accelerate the research and innovation
improving energy efficiency?

Is it possible to develop "energy benchmarks" (similar to performance
benchmarks)?  As most computing devices are connected to networks,
would it be necessary to develop energy benchmarks that consider
network connections?  How should the energy consumed by shared
resources (routers and servers) be counted?

Thursday, January 16, 2014

Action Items for energy efficiency

From the summit, I think there are two main actions items for the energy efficiency group.
  1. Organize a competition of low-power designs.
  2. Organize workshops discussing how to design systems that can migrate computation to more energy efficient locations or locations with renewable energy