Universal Approximation Results for the Temporal Restricted Boltzmann Machine and the Recurrent Temporal Restricted Boltzmann Machine

Date

2016

Authors

Odense, Simon
Edwards, Roderick

Journal Title

Journal ISSN

Volume Title

Publisher

Journal of Machine Learning Research

Abstract

The Restricted Boltzmann Machine (RBM) has proved to be a powerful tool in machine learning, both on its own and as the building block for Deep Belief Networks (multi-layer generative graphical models). The RBM and Deep Belief Network have been shown to be universal approximators for probability distributions on binary vectors. In this paper we prove several similar universal approximation results for two variations of the Restricted Boltzmann Machine with time dependence, the Temporal Restricted Boltzmann Machine (TRBM) and the Recurrent Temporal Restricted Boltzmann Machine (RTRBM). We show that the TRBM is a universal approximator for Markov chains and generalize the theorem to sequences with longer time dependence. We then prove that the RTRBM is a universal approximator for stochastic processes with nite time dependence. We conclude with a discussion on e ciency and how the constructions developed could explain some previous experimental results.

Description

Keywords

TRBM, RTRBM, machine learning, universal approximation

Citation

Odense, S. & Edwards, R. (2016). Universal approximation results for the temporal restricted Boltzmann machine the recurrent temporal restricted Boltzmann machine. Journal of Machine Learning Research, 17(158), 1-21. http://jmlr.org/papers/v17/15-478.html