Jeff Dean

Regarding energy efficiency, real brains are definitely much more energy efficient and have much more computational ability than current machines. However, the gap is perhaps not as bad as it might seem, for the reason that real brains take ~20 years to "train", whereas, since we are impatient machine learning researchers, we want to do experiments in a week. If we were willing to have our experimental cycle time be 20 years instead of 1 week, we could clearly get much better energy efficiency, but we prefer the faster cycle time for experiments, even if it costs us in energy efficiency.

I had to write a thesis to graduate with honors at the University of Minnesota, so I worked with my advisor, Vipin Kumar, and settled on the problem of exploring parallel training of neural nets on a 64 processor hypercube machine that the department had. Since neural nets were very computationally intensive, parallel training even back then was an attractive notion to allow scaling the approach to more realistic problems. For the thesis, I was able to get interesting speedups on training models for relatively moderately-sized problems. The computational model of neural nets, of many layers of abstraction, each building on each other, really appealed to me at the time, and I went into grad school with the intention of studying parallel computing, but ended up getting seduced by the appeal of writing compilers for high-level object-oriented languages, and did my Ph.D. research work in that area. However, that little notion that neural nets were interesting never really went away, and about five years ago, it seemed like it was worth exploring again, now that both computational power and interesting datasets had grown so much in the ~2 decades since I last revisited them. This led to the genesis of the Google Brain project (initially started by myself, Andrew Ng, and Greg Corrado).