Gated Linear Networks (GLNs) are a family of backpropation-free neural networks. Each neuron in a GLN predicts the target density (or probability mass) based on the outputs of the previous layer and is trained under a logarthmic loss.
Neurons have probabilistic "activation functions". Implementations are provided for the following distributions:
-
Gaussian, for regression.
-
Bernoulli, for binary classification and multi-class classification using a one-vs-all scheme.
Usage examples are provided in examples.
Because each neuron implements a probability density/mass function we need to ensure that they are well defined. For example, the scale parameter for a Gaussian density needs to be positive. We implement these constraints using linear projections and clipping.
Because each neuron predicts the target, we can use any neuron output as the "network output", and are not bound to the last layer. Typically last layer neuron(s) are the best predictors, but they might take longer to converge in theory. In this implementation, we use a single neuron at the last layer, which then forms the network output.
There are alternative ways of aggregating, e.g. see Switching Aggregation in Appendix D of Gaussian Gated Linear Networks (link: https://arxiv.org/pdf/2006.05964.pdf).
Coming soon.