Training Kanter’s Bit Generator by Natural Gradient Descent
MSc Thesis, Dept. of Mathematics, King's College London, 1998. [Full text (pdf)]
Natural gradient descent (NGD) learning is compared with ordinary gradient descent (OGD) for Kanter’s bit generator. Analytic results for one or two input bits and one output bit show that the generalization error decreases exponentially with time for NGD learning, while OGD atudents attain an error that only decreases inversely proportional to learning time. Similar results are found numerically for two input bits and more output bits. In some cases students end up in local minima. In this study NGD suffered slightly more from this problem than OGD, but we suspect that in other models it could just as easily work out the other way round. For more than two input bits no analytic results are obtained, and various options for future research with numerical methods are suggested.