Universal Approximation Results for the Temporal Restricted Boltzmann Machine and the Recurrent Temporal Restricted Boltzmann Machine
Date
2016
Authors
Odense, Simon
Edwards, Roderick
Journal Title
Journal ISSN
Volume Title
Publisher
Journal of Machine Learning Research
Abstract
The Restricted Boltzmann Machine (RBM) has proved to be a powerful tool in machine
learning, both on its own and as the building block for Deep Belief Networks (multi-layer
generative graphical models). The RBM and Deep Belief Network have been shown to be
universal approximators for probability distributions on binary vectors. In this paper we
prove several similar universal approximation results for two variations of the Restricted
Boltzmann Machine with time dependence, the Temporal Restricted Boltzmann Machine
(TRBM) and the Recurrent Temporal Restricted Boltzmann Machine (RTRBM). We show
that the TRBM is a universal approximator for Markov chains and generalize the theorem
to sequences with longer time dependence. We then prove that the RTRBM is a universal
approximator for stochastic processes with nite time dependence. We conclude with a
discussion on e ciency and how the constructions developed could explain some previous
experimental results.
Description
Keywords
TRBM, RTRBM, machine learning, universal approximation
Citation
Odense, S. & Edwards, R. (2016). Universal approximation results for the temporal restricted Boltzmann machine the recurrent temporal restricted Boltzmann machine. Journal of Machine Learning Research, 17(158), 1-21. http://jmlr.org/papers/v17/15-478.html