Volume 17, Issue 2 (Journal of Control, V.17, N.2 Summer 2023)
JoC 2023, 17(2): 1-23 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Mohammadzadeh Ayooghi V, Aliyari-Shoorehdeli M. Deep Learning based Models for Nonlinear System Identification. JoC 2023; 17 (2) :1-23
URL: http://joc.kntu.ac.ir/article-1-1008-en.html
URL: http://joc.kntu.ac.ir/article-1-1008-en.html
1- K. N. Toosi University of Technology
Abstract: (974 Views)
Deep learning-based models appropriately perform in modeling complex problems in computer vision and natural language processing (NLP). With this in mind, nonlinear system identification methods can benefit from tools developed in deep learning, leading to enriched frameworks to choose from. For this purpose, we are going to review some potential structures and methods of deep learning that can be used in nonlinear system identification. Although we comprehensively review the applicable tools of deep learning to system identification, this paper mainly focuses on using latent variable models (LVM) for identifying nonlinear state space models. LVMs are powerful tools for extending generative models primarily developed for only generating static data, yet by a combination of recurrent neural network (RNN) and variational auto-encoders (VAE), they can also generate sequential data. A structured version of introduced models compatible with control systems will also be given. The study shows that the deep learning models have a comparative performance to traditional and classic ones.
Type of Article: Research paper |
Subject:
New approaches in control engineering
Received: 2023/07/27 | Accepted: 2023/09/11 | ePublished ahead of print: 2023/09/19 | Published: 2023/09/21
Received: 2023/07/27 | Accepted: 2023/09/11 | ePublished ahead of print: 2023/09/19 | Published: 2023/09/21
References
1. [1] L. Ljung, "System identification, Theory for the user." System science series, Prentice Hall, Upper Saddle River, NJ, USA, Second edition, 1999.
2. [2] Zadeh, L. "On the identification problem." IRE Transactions on Circuit Theory 3.4 (1956): 277-281. [DOI:10.1109/TCT.1956.1086328]
3. [3] Nelles, Oliver. "Nonlinear system identification, from classical approach to neural networks, fuzzy systems, and Gaussian process". Springer, Berlin, Heidelberg, 2020. [DOI:10.1007/978-3-030-47439-3]
4. [4] T. Soderstrom, P. Stocia, "System identification", Prentice-Hall, Inc, 1988.
5. [5] Schoukens, Johan, and Lennart Ljung. "Nonlinear system identification: A user-oriented road map." IEEE Control Systems Magazine 39.6 (2019): 28-99. [DOI:10.1109/MCS.2019.2938121]
6. [6] Billings, Stephen A. "Identification of nonlinear systems-a survey." IEE Proceedings D (Control Theory and Applications). Vol. 127. No. 6. IET Digital Library, 1980. [DOI:10.1049/ip-d.1980.0047]
7. [7] Zheng, Qingsheng, and Evanghelos Zafiriou. "Nonlinear system identification for control using Volterra-Laguerre expansion." Proceedings of 1995 American Control Conference-ACC'95. Vol. 3. IEEE, 1995.
8. [8] Korenberg, Michael J., and Ian W. Hunter. "The identification of nonlinear biological systems: Wiener kernel approaches." Annals of Biomedical Engineering 18 (1990): 629-654. [DOI:10.1007/BF02368452]
9. [9] Schoukens, Maarten, and Koen Tiels. "Identification of block-oriented nonlinear systems starting from linear approximations: A survey." Automatica 85 (2017): 272-292. [DOI:10.1016/j.automatica.2017.06.044]
10. [10] Chiuso, Alessandro, and Gianluigi Pillonetto. "System identification: A machine learning perspective." Annual Review of Control, Robotics, and Autonomous Systems 2 (2019): 281-304. [DOI:10.1146/annurev-control-053018-023744]
11. [11] Pillonetto, Gianluigi, et al. "Kernel methods in system identification, machine learning and function estimation: A survey." Automatica 50.3 (2014): 657-682. [DOI:10.1016/j.automatica.2014.01.001]
12. [12] Bishop, Christopher M., and Nasser M. Nasrabadi. Pattern recognition and machine learning. Vol. 4. No. 4. New York: springer, 2006.
13. [13] Heaton, Jeff. Ian goodfellow, yoshua bengio, and aaron courville: "Deep learning." (2018): 305-307. [DOI:10.1007/s10710-017-9314-z]
14. [14] Bengio, Yoshua, Aaron Courville, and Pascal Vincent. "Representation learning: A review and new perspectives." IEEE transactions on pattern analysis and machine intelligence 35.8 (2013): 1798-1828. [DOI:10.1109/TPAMI.2013.50]
15. [15] Noroozi, Mehdi, and Paolo Favaro. "Unsupervised learning of visual representations by solving jigsaw puzzles." European conference on computer vision. Cham: Springer International Publishing, 2016. [DOI:10.1007/978-3-319-46466-4_5]
16. [16] Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv: 1312.6114 (2013).
17. [17] Fraccaro, Marco. "Deep latent variable models for sequential data." English. PhD thesis DTU University (2018).
18. [18] Barber, David. Bayesian reasoning and machine learning. Cambridge University Press, 2012. [DOI:10.1017/CBO9780511804779]
19. [19] Koller, Daphne, and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
20. [20] Hinton, Geoffrey E. "A practical guide to training restricted Boltzmann machines." Neural Networks: Tricks of the Trade: Second Edition. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012. 599-619. [DOI:10.1007/978-3-642-35289-8_32]
21. [21] Fischer, Asja, and Christian Igel. "An introduction to restricted Boltzmann machines." Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 17th Iberoamerican Congress, CIARP 2012, Buenos Aires, Argentina, September 3-6, 2012. Proceedings 17. Springer Berlin Heidelberg, 2012.
22. [22] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
23. [23] Gianluigi Pillonetto, Aleksandr Aravkin, Daniel Gedon, Lennart Ljung, Antônio H. Ribeiro, Thomas B. Schön, Deep network for system identification, arXiv, 2023.
24. [24] Ljung, Lennart, et al. "Deep learning and system identification." IFAC-PapersOnLine 53.2 (2020): 1175-1181. [DOI:10.1016/j.ifacol.2020.12.1329]
25. [25] M. Forgione and D. Piga. Dynonet: A neural network architecture for learning dynamical systems. Int. J. Adapt. Control Signal Process., 35(4):612-626, 2021. [DOI:10.1002/acs.3216]
26. [26] J. Hendriks, F.K. Gustafsson, A.H. Ribeiro, A. Wills, and T.B. Schon. Deep energy-based NARX models. In Proceedings of the 19th IFAC Symposium on System Identification (SYSID), 2021. [DOI:10.1016/j.ifacol.2021.08.410]
27. [27] LeCun, Yann, et al. "Energy-based models in document recognition and computer vision." Ninth International Conference on Document Analysis and Recognition (ICDAR 2007). Vol. 1. IEEE, 2007. [DOI:10.1109/ICDAR.2007.4378728]
28. [28] Andersson, Carl, et al. "Deep convolutional networks in system identification." 2019 IEEE 58th conference on decision and control (CDC). IEEE, 2019. [DOI:10.1109/CDC40024.2019.9030219]
29. [29] Lea, Colin, et al. "Temporal convolutional networks: A unified approach to action segmentation." Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14. Springer International Publishing, 2016.
30. [30] R. Calandra, J. Peters, C. Rasmussen, and M.P. Deisenroth. Manifold gaussian processes for regression. In 2016 International Joint Conference on Neural Networks (IJCNN), pages 3338-3345,2016. [DOI:10.1109/IJCNN.2016.7727626]
31. [31] Y. Cho and L. Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems, volume 22, 2009.
32. [32] Nagel, Tobias, and Marco F. Huber. "Autoencoder-inspired Identification of LTI systems." 2021 European Control Conference (ECC). IEEE, 2021. [DOI:10.23919/ECC54610.2021.9655185]
33. [33] D. Gedon, N. Wahlstr¨om, T.B. Sch¨on, and L. Ljung. Deep state space models for nonlinear system identification. In Proceedings of the 19th IFAC Symposium on System Identification (SYSID), 2021. [DOI:10.1016/j.ifacol.2021.08.406]
34. [34] M. Karl, M. Soelch, J. Bayer, and P. van der Smagt. Deep variational Bayes filters: Unsupervised learning of state space models from raw data, 2017.
35. [35] M. Watter, J. Springenberg, J. Tobias, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Volume 2, pages 2746-2754, Cambridge, MA, USA, 2015. MIT Press.
36. [36] Rangapuram, Syama Sundar, et al. "Deep state space models for time series forecasting." Advances in neural information processing systems 31 (2018).
37. [37] Courts, Jarrad, et al. "Variational state and parameter estimation." IFAC-PapersOnLine 54.7 (2021): 732-737. [DOI:10.1016/j.ifacol.2021.08.448]
38. [38] Courts, Jarrad, et al. "Variational System Identification for Nonlinear State-Space Models." arXiv preprint arXiv: 2012.05072 (2020).
39. [39] Menghani, Gaurav. "Efficient deep learning: A survey on making deep learning models smaller, faster, and better." ACM Computing Surveys 55.12 (2023): 1-37. [DOI:10.1145/3578938]
40. [40] Xu, Canwen, and Julian McAuley. "A survey on model compression and acceleration for pretrained language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 9. 2023. [DOI:10.1609/aaai.v37i9.26255]
41. [41] Choudhary, Tejalal, et al. "A comprehensive survey on model compression and acceleration." Artificial Intelligence Review 53 (2020): 5113-5155. [DOI:10.1007/s10462-020-09816-7]
42. [42] Cheng, Yu, et al. "A survey of model compression and acceleration for deep neural networks." arXiv preprint arXiv:1710.09282 (2017).
43. [43] Cheng, Yu, et al. "Model compression and acceleration for deep neural networks: The principles, progress, and challenges." IEEE Signal Processing Magazine 35.1 (2018): 126-136. [DOI:10.1109/MSP.2017.2765695]
44. [44] Liang, Tailin, et al. "Pruning and quantization for deep neural network acceleration: A survey." Neurocomputing 461 (2021): 370-403. [DOI:10.1016/j.neucom.2021.07.045]
45. [45] Rokh, Babak, Ali Azarpeyvand, and Alireza Khanteymoori. "A comprehensive survey on model quantization for deep neural networks." arXiv preprint arXiv: 2205.07877 (2022).
46. [46] Gholami, Amir, et al. "A survey of quantization methods for efficient neural network inference." Low-Power Computer Vision. Chapman and Hall/CRC, 2022. 291-326. [DOI:10.1201/9781003162810-13]
47. [47] Liang, Tailin, et al. "Pruning and quantization for deep neural network acceleration: A survey." Neurocomputing 461 (2021): 370-403. [DOI:10.1016/j.neucom.2021.07.045]
48. [48] Frankle, Jonathan, and Michael Carbin. "The lottery ticket hypothesis: Finding sparse, trainable neural networks." arXiv preprint arXiv: 1803.03635 (2018).
49. [49] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv: 1503.02531 (2015).
50. [50] Gou, Jianping, et al. "Knowledge distillation: A survey." International Journal of Computer Vision 129 (2021): 1789-1819. [DOI:10.1007/s11263-021-01453-z]
51. [51] Hastie, Trevor, et al. The elements of statistical learning: data mining, inference, and prediction. Vol. 2. New York: springer, 2009. [DOI:10.1007/978-0-387-84858-7]
52. [52] Doersch, Carl. "Tutorial on variational autoencoders." arXiv preprint arXiv: 1606.05908 (2016).
53. [53] Burda, Yuri, Roger Grosse, and Ruslan Salakhutdinov. "Importance weighted autoencoders." arXiv preprint arXiv: 1509.00519 (2015) [40] Barber, David, A. Taylan Cemgil, and Silvia Chiappa, eds. Bayesian time series models. Cambridge University Press, 2011.
54. [54] Makhzani, Alireza, et al. "Adversarial autoencoders." arXiv preprint arXiv: 1511.05644 (2015). [39] Burda, Yuri, Roger Grosse, and Ruslan Salakhutdinov. "Importance weighted autoencoders." arXiv preprint arXiv: 1509.00519 (2015)
55. [55] Girin, Laurent, et al. "Dynamical variational autoencoders: A comprehensive review." arXiv preprint arXiv:2008.12595 (2020). [DOI:10.1561/9781680839135]
56. [56] Beal, Matthew James. Variational algorithms for approximate Bayesian inference. University of London, University College London (United Kingdom), 2003.
57. [57] Hochreiter, Sepp, and Jürgen Schmidhuber. "Long short-term memory." Neural computation 9.8 (1997): 1735-1780. [DOI:10.1162/neco.1997.9.8.1735]
58. [58] Chung, Junyoung, et al. "Empirical evaluation of gated recurrent neural networks on sequence modeling." arXiv preprint arXiv:1412.3555 (2014).
59. [59] Su, Yuanhang, and C-C. Jay Kuo. "Recurrent neural networks and their memory behavior: a survey." APSIPA Transactions on Signal and Information Processing 11.1 (2022). [DOI:10.1561/116.00000123]
60. [60] Graves, Alex. "Generating sequences with recurrent neural networks." arXiv preprint arXiv:1308.0850 (2013).
61. [61] Krishnan, Rahul G., Uri Shalit, and David Sontag. "Deep kalman filters." arXiv preprint arXiv: 1511.05121 (2015).
62. [62] Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv: 1409.0473 (2014).
63. [63] Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).
64. [64] Han, Kai, et al. "A survey on vision transformer." IEEE transactions on pattern analysis and machine intelligence 45.1 (2022): 87-110. [DOI:10.1109/TPAMI.2022.3152247]
65. [65] Khan, Salman, et al. "Transformers in vision: A survey." ACM computing surveys (CSUR) 54.10s (2022): 1-41. [DOI:10.1145/3505244]
66. [66] Liu, Yang, et al. "A survey of visual transformers." IEEE Transactions on Neural Networks and Learning Systems (2023). [DOI:10.1109/TNNLS.2022.3227717]
67. [67] Lin, Tianyang, et al. "A survey of transformers." AI Open (2022). [DOI:10.1016/j.aiopen.2022.10.001]
68. [68] Rosendahl, Jan, et al. "Analysis of positional encodings for neural machine translation." Proceedings of the 16th International Conference on Spoken Language Translation. 2019.
69. [69] Beintema, Gerben I., Maarten Schoukens, and Roland Tóth. "Deep subspace encoders for nonlinear system identification." Automatica 156 (2023): 111210. [DOI:10.1016/j.automatica.2023.111210]
70. [70] Masti, Daniele, and Alberto Bemporad. "Learning nonlinear state-space models using autoencoders." Automatica 129 (2021): 109666. [DOI:10.1016/j.automatica.2021.109666]
71. [71] Lopez, Ryan, and Paul J. Atzberger. "Variational autoencoders for learning nonlinear dynamics of physical systems." arXiv preprint arXiv:2012.03448 (2020).
72. [72] Chung, Junyoung, et al. "A recurrent latent variable model for sequential data." Advances in neural information processing systems 28 (2015).
73. [73] Bayer, Justin, and Christian Osendorfer. "Learning stochastic recurrent networks." arXiv preprint arXiv: 1411.7610 (2014).
74. [74] Fraccaro, Marco, et al. "Sequential neural models with stochastic layers." Advances in neural information processing systems 29 (2016).
75. [75] Fraccaro, Marco, et al. "A disentangled recognition and nonlinear dynamics model for unsupervised learning." Advances in neural information processing systems 30 (2017).
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
