Machine learning implementation by memristive circuits

Head investigator prof. Ing. Martin Klimo, PhD.

In September 2018, the European Council adopted the Regulation establishing the European High-Performance Computing Joint Undertaking (EuroHPC Joint Undertaking). It reflects the current state of computer technology and calls for exascale high-performance computers. Until now, the leading platform to implement cognitive systems was a classical computer, and all its limitations are mirrored as limits of cognitive systems themselves. Quantum [1], biomolecular [2] and extended analog computing platforms [3] were recognised as promising ideas. However, to this day none of these concepts provided sufficiently mature hardware implementation, which would allow the systematic approach to solving problems from formulation to the application. Their computational ability has so far been demonstrated only in ad hoc examples. For bridging this gap between current computers and one of the future visionary paradigms, we need a concept of cognitive systems that will allow implementation using programmable massively parallel hardware with extremely low consumption, which can be created using technologies presently available (see [4] for a survey). The proposed project has the ambition to achieve a merge of neuromorphic computing [5,6] with fuzzy logic, based on memristive circuits [7]. Programming of such a cognitive system does not consist of assembling a sequence of instructions, but rather by the creation of the structure of memristor circuits and applying machine learning methods.

The concept of the fuzzy logic system has been successfully applied not only for pattern recognition [8] but also for process control [9], forecasting of time series [10], etc. Therefore, the cognition systems area is fully prepared for the cognition-system based on fuzzy logic implementation. It is known that fuzzy inference can be arranged in a variety of fuzzy logic [11]. The high performance of the system will be achieved at the level of the fuzzy logic gate, which integrates computation and memory. [12] shows that Min-Max fuzzy logic [13] based on bidirectionally connected memristors [7] can play this role.

Low energy consumption will be accomplished by using the ability to store memory without any power consumption, the fact that they are passive elements and due to reversibility at the level of the gate. Integration of memory and computing will eliminate energy consumption due to signal transfer primarily caused by parasitic capacitances. Production of memristor memories was announced for years 2014-15; however, the advancement of 3D flash postponed their introduction to the market. Thus, reliable manufacturing is expected to be available very shortly. The neuromorphic platform also benefits from the development of nanotechnology [14]. The proposed concept combines the advantages of reversible computing with the processing of spiking inputs. That is achieved mainly by the new design of reversible fuzzy logic gates. At present Boolean reversible gates are studied [15], but their complement – spiking reversible gates are not yet known and will be explored in the project. Since 2008, the focus has been on the practical implementation of memristor [16,17], although it theoretical prediction has been published in 1971 [18] and designed as a memory element in 1960 [19]. Memristors using nanotechnology provide for new ways of overcoming limits of current computers (we use the term “memristor” in the sense of bipolar resistive non-volatile switch). These include already mentioned energy savings, but also the high density of integration of memories. Moreover, the memristor integrates memory and computation. At present we see three main directions for using memristor in information processing. The first is based on the fact that Boolean implication can be implemented by a pair of memristors [20]. That allows creating Boolean logic functions using memristor switch. The second is as a weighting coefficient in the artificial neural network or fuzzy neural network [21]. Lastly, neuromorphic computing [22], uses memristor to emulate a synapse in an artificial  piking system. Differently, from the Boolean approach, the latter two methods are based on the creation of analog circuits, in which inputs, outputs, and states of memristors are analog.

At the University of Zilina, we work on using fuzzy logic for pattern recognition since 2007, when we found that using multiplicative fuzzy logic allows one to create fuzzy flip-flops with memory [23]. Networks of fuzzy flip flops were studied as a tool for pattern recognition. After H-P paper [16] our team recognised the potential of memristors for implementing Zadeh fuzzy logic [8]. We developed three concepts for using memristors:

1. implementation of fuzzy logic circuits with memristors with low threshold voltage [24]
2. implementation of switching functions using memristors with high threshold voltage [25],
3. implementation of deep neural networks with binary weighted coefficients implemented by the memristive crossbar [26].

These were tested on vowel recognition task, especially for the critical pair aa/ao (TIMIT notation [27]), and on hand-written digits (MNIST database [28]). The concept of fuzzy cognitive computing using memristor circuits was presented in 2012 [29]. The first measurements of basic memristor circuits were published in 2013 [30] and confirm the validity of our simulator of memristor circuits on GPU. Papers [31,32] confirm the possibility to sort the analog values by the memristive circuit. The main goal of our research is the development of the concept of memristive implementation of a spiking fuzzy logic system for pattern recognition from the architecture point of view, as well as design approach and evaluation. The particular goals from the architecture point of view are to design a concept suitable for memristor implementation:

– circuits with fuzzy logic gates in the spiking mode,

– a general reversible spiking fuzzy logic system for pattern recognition,

– proposal of a specific fuzzy logic system application for pattern recognition without a need for external power.

The particular goals from the technological point of view are:

– evaluating the possibility of memristor implementation of simple fuzzy logic functions,

– a deeper understanding of technological limits for reproducibility, accuracy, and speed of recognition.

The particular goals from the design and evaluation point of view:

– creation of modelling tools,

– a benchmark of spiking fuzzy systems for pattern recognition with memristors implementation for acoustic and visual cognitive functions.

To conclude, the way of the machine learning method implementation seems to be a crucial point for the successful applications. We expect that the implementation has to offer naturally massive parallel way of low energy computing and the memristive circuits offer it until quantum computing will be available.

References

[1] B. By John II, Future Quantum Computers Already Threatening Today’s Data, Nextgov.Com (Online). (2018).

[2] M. Swami, Computing in a cell, Nat. Rev. Genet. (2008). doi:10.1038/nrg2486.

[3] L.A. Rubel, The extended analog computer, Adv. Appl. Math. (1993). doi:10.1006/aama.1993.1003.

[4] C.D. Schuman, T.E. Potok, R.M. Patton, J.D. Birdwell, M.E. Dean, G.S. Rose, J.S. Plank, A Survey of Neuromorphic Computing and Neural Networks in Hardware, (2017) 1–88. http://arxiv.org/abs/1705.06963.

[5] Q. Yu, H. Tang, J. Hu, K.C. Tan, Introduction, Intell. Syst. Ref. Libr. (2017). doi:10.1007/978-3-319-55310-8_1.

[6] G.W. Burr, R.M. Shelby, A. Sebastian, S. Kim, S. Kim, S. Sidler, K. Virwani, M. Ishii, P. Narayanan, A. Fumarola, L.L. Sanches, I. Boybat, M. Le Gallo, K. Moon, J. Woo, H. Hwang, Y. Leblebici, Neuromorphic computing using non-volatile memory, Adv. Phys. X. (2017). doi:10.1080/23746149.2016.1259585.

[7] K. Fröhlich, I. Kundrata, M. Blaho, M. Precner, M. Tapajna, M. Klimo, O. Šuch, O. Škvarek, Hafnium oxide and tantalum oxide-based resistive switching structures for realization of minimum and maximum functions, J. Appl. Phys. (2018). doi:10.1063/1.5025802.

[8] L.A. Zadeh, Soft computing, fuzzy logic and recognition technology, in: 2002. doi:10.1109/fuzzy.1998.686373.

[9] K.M. Passino, S. Yurkovich, Fuzzy control, in: Control Syst. Handb. Control Syst. Adv. Methods, Second Ed., 2010. doi:10.1201/b10384.

[10] L. Zhao, A new method to predict enrollments based on fuzzy time series, in: Proc. World Congr. Intell. Control Autom., 2010. doi:10.1109/WCICA.2010.5553945.

[11] V. Novák, Which logic is the real fuzzy logic?, in: Fuzzy Sets Syst., 2006. doi:10.1016/j.fss.2005.10.010.

[12] K. Fröhlich, I. Kundrata, M. Blaho, M. Precner, M. Ťapajna, M. Klimo, O. Šuch, O. Škvarek, Performance of HfOx- and TaOx-based Resistive Switching Structures for Realization of Minimum and Maximum Functions, MRS Adv. (2018). doi:10.1557/adv.2018.377.

[13] L.A. Zadeh, 1965 J(Zadeh) Fuzzy Sets.pdf, Inf. Control. (1965). doi:10.1109/2.53.

[14] G.S. Rose, R. Pino, Q. Wu, Exploiting memristance for low-energy neuromorphic computing hardware, in: Proc. – IEEE Int. Symp. Circuits Syst., 2011. doi:10.1109/ISCAS.2011.5938208.

[15] R. Wille, R. Drechsler, Towards a design flow for reversible logic, 2010. doi:10.1007/978-90-481-9579-4.

[16] D.B. Strukov, G.S. Snider, D.R. Stewart, R.S. Williams, The missing memristor found, Nature. (2008). doi:10.1038/nature06932.

[17] D.B. Strukov, G.S. Snider, D.R. Stewart, R.S. Williams, Erratum: The missing memristor found, Nature. (2009). doi:10.1038/nature08166.

[18] L.O. Chua, Memristor—The Missing Circuit Element, IEEE Trans. Circuit Theory. (1971). doi:10.1109/TCT.1971.1083337.

[19] B. Widrow, An Adaptive Adaline Neuron Using Chemical Memistors, 1960.

[20] J. Borghetti, G.S. Snider, P.J. Kuekes, J.J. Yang, D.R. Stewart, R.S. Williams, Memristive switches enable stateful logic operations via material implication, Nature. (2010). doi:10.1038/nature08940.

[21] H. Kim, M.P. Sah, C. Yang, T. Roska, L.O. Chua, Memristor bridge synapses, in: Proc. IEEE, 2012. doi:10.1109/JPROC.2011.2166749.

[22] T. Serrano-Gotarredona, T. Masquelier, T. Prodromakis, G. Indiveri, B. Linares-Barranco, STDP and sTDP variations with memristors for spiking neuromorphic learning systems, Front. Neurosci. (2013). doi:10.3389/fnins.2013.00002.

[23] M. Klimo, J. Boroň, Temporary properties of RS fuzzy flip-flops, Acta Tech. Jaurinensis. 4 (2011) 127–133.

[24] M. Klimo, Implementation of Fuzzy Logic Systems by Nanotechnology, in: NIMS Hot Top. Work. Fuzzy Theory Appl., National Institute for Mathematical Sciences, Daejeon, Korea, 2012: pp. 7–8.

[25] M. Klimo, O. Such, O. Skvarek, M. Fratrik, Memristor-based pattern matching, Semicond. Sci. Technol. 29 (2014). doi:10.1088/0268-1242/29/10/104007.

[26] M. Klimo, P.T. Ábek, O. Šuch, J. Smieško, O. Škvarek, Implementation of a deep ReLU neuron network with a memristive circuit, Int. J. Unconv. Comput. 12 (2016).

[27] Z. Victor, S. Seneff, J. Glass, TIMIT acoustic-phonetic continuous speech corpus, Speech Commun. (1990).

[28] Y. Lecun, THE MNIST DATABASE of handwritten digits, Html(MNIST Dataset). (2010).

[29] O. Šuch, M. Klimo, S. Foltán, G. Karol, Computational Concept Based on Complementary Resistive Switches, in: J. Heber, D. Schlom, Y. Tokura, R. Waser, M. Wuttig (Eds.), Nat. Conf. Front. Eectronic Mater., WILEY-VCH, Aachen, Germany, 2012: pp. 190–191. doi:10.1016/s1359-0286(98)80058-2.

[30] M. Fratrik, S. Badura, M. Klimo, O. Skvarek, Memristor measurements and simulations, in: Proc. Int. Semicond. Conf. CAS, 2013. doi:10.1109/SMICND.2013.6688668.

[31] M. Frátrik, M. Klimo, O. Šuch, O. Škvarek, Memristive sorting networks, Phys. Status Solidi Curr. Top. Solid State Phys. 12 (2015). doi:10.1002/pssc.201100000.

[32] L. Nielen, S. Ohm, O. Šuch, M. Klimo, R. Waser, E. Linn, Memristive sorting networks enabled by electrochemical metallization cells, Int. J. Unconv. Comput. 12 (2016).

[33] O. Šuch, M. Klimo, N.T. Kemp, O. Škvarek, Passive memristor synaptic circuits with multiple timing dependent plasticity mechanisms, AEU – Int. J. Electron. Commun. (2018). doi:10.1016/j.aeue.2018.09.025.