Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Then perhaps we should try to understand that learning mechanism. Currently there's almost no communication between DL and neuroscience communities.


The learning mechanism[1] is actually relatively well understood (at least as far as neuroscience goes) but it isn't used for ANNs because Spiking Neural Networks(SNNs) don't map well onto conventional computing hardware whereas the matrix multiplication used by conventional ANNs does.

I agree that it's worth researching but currently it is quite difficult to compete with conventional ANNs due to the commonality of processors well suited for ANNs.

[1]https://en.wikipedia.org/wiki/Spike_timing_dependent_plastic...


STDP is one of the early steps towards understanding the learning mechanism(s) in the brain, but we are a long way from enough understanding to actually reproduce it.

Not only we don't know how to train spiking networks, we don't even know how the information is encoded: pulse frequency or pulse timings or ...? No one knows. How can you compete with anything if you have no idea how it works?

Also, this has nothing to do with computing hardware. You can easily simulate anything your want on conventional processors. Huge computing clusters have been built for spiking model simulations, and nothing interesting came out of it. Invent the algorithm first, then we will build the hardware for it.


STDP based SNNs don't take much secret sauce to function reasonably well for basic tasks[1]. Some people have even started to apply reward modulated STDP to the problem which is nice to see since it's a more complete version of STDP[2]. Even so, the amount of research in the area is tiny. I agree that you need to figure out the whole working algorithm before trying to scale things up but to do that you need a research base and currently that is difficult because running large SNNs is very slow for an average researcher without access to specialized hardware. The primary factor which kicked off the deep learning revolution was the idea that suddenly you could train a DNN 10x faster using a GPU rather than a CPU. Without this revelation, most researchers just wouldn't bother because DL research took too long. GPUs are common and lots of people had access to them. SNNs obviously can be simulated on any CPU but for large models it's incredibly slow. SNNs have an incredible amount of potential paralellizeability but all of it is in the form of a massively asynchronous system where the order of events is critically important. CPUs and GPUs are unable to take advantage of most of the potential speedups inherent to this structure outside of specific cases where you have groups of non-interacting neurons. Due to the event based rather than signal based nature of SNNs, batched training also seems to be substantially less effective than in ANNs further complicating training.

[1]https://arxiv.org/pdf/1611.01421.pdf

[2]https://arxiv.org/pdf/1705.09132.pdf


The main problem with SNNs is not that they don't work well. It's not that they are slow to run on CPUs/GPUs. Those would indeed be temporary problems. The fundamental problem is we don't know the right abstraction level to imitate brain operations. It's far from clear that simulating spikes is necessary to implement brain's "learning algorithms". There are two main arguments for using spikes: "biological realism", and "energy efficiency". Neither is convincing: if you want to simulate a conventional CPU you don't want to simulate IV curves for transistors. You don't want to simulate CMOS gates. No, because the lowest abstraction level necessary to understand the computer operation is boolean logic. Anything below that is completely irrelevant. I strongly suspect that simulating spikes is below the lowest relevant abstraction level. By the way, Jeff Hawkins of Numenta agrees with me, and he's pretty strict about "biological plausibility" of his neocortex algorithms. As for energy efficiency - sure, spikes might be the most efficient way to compute and encode information given the constraints of biological brains. But why should we care about those constraints? We are building computing machinery in silicon, using vastly different technology, circuits, and tricks from what's available in "wetware". It does not make any sense to copy biological evolution tricks to improve efficiency when building things in silicon. Neuromorphic hardware papers always mention energy efficiency and then proceed to compare their analog spiking chips to digital (!) chips. That's ridiculous! I can't think of a single good argument why would spiking analog circuits be more energy efficient than non-spiking analog circuits, if we are talking about any existing computing hardware technology (or one that's likely to be developed in the foreseeable future).

Deep learning took off in 2012 not because faster hardware allowed us to develop good algorithms. The algorithms (gradient descent optimization, backpropagation, convolutional and recurrent layers) have been developed -- using slow hardware -- long before (Lecun demonstrated state of the art on MNIST back in 1998). The fast hardware allowed us to scale up the existing good algorithms. I don't see any good algorithms developed for SNNs. Perhaps these algorithms can be developed, perhaps faster hardware is indeed necessary, but as I argue above, the motivation to pursue this research is just not obvious to me.

Note that we shouldn't confuse this SNN research (such as the papers you cited), to efforts like Human Brain Project, where they're actually trying to derive higher level brain algorithms from accurate simulations of low level mechanics. Emphasis on accurate, because as any neuroscientist will tell you (e.g. [1]), these SNNs have very little to do with what's actually going on in a brain on any abstraction level.

[1] https://spectrum.ieee.org/tech-talk/semiconductors/devices/b...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: