What is an Artificial Neural Network?

July 11, 2018

In short, an Artificial Neural Network, or ANN, is a deep-learning method that imitates the way a human brain works. Neural in ANN refers to the fact that the algorithm is modelling the neuron in a human brain. A neuron takes an input and if the the input(s) reaches a threshold an outgoing event is triggered.

Similarly, an ANN has inputs. These inputs are mapped to what are called hidden layers. The hidden layers are where this "thresholding" occurs through what is called an activation function. For example, in a classification problem, the activation function is called relu or the rectifier function. It is simply: max(x, 0). In other words, if x is less-than or equal zero, return 0, else return x.

This output can then go to another hidden layer with (usually) the same activation function, or to the output layer. Once at the output layer, a different activation function is used. For classification, the sigmoid activation function defined below is used.

What is really interesting about ANN is that we use functions we are already familiar with to do the same work in a different way. In fact, probably the best-known classification algorithm--logarithmic regression--uses a very similar function to what we are calling the sigmoid activation function. However, sigmoids are an entire class of s-shaped functions and are not unique to ANNs. The output layer then sends the result from the second activation function to the cost function. Each of the inputs in the input layer have weights attributed to them. These weights are updated by a cost function. The cost function compares the actual value to the predicted value and updates the weights accordingly to make the model more predictive. This is really the power of the algorithm. This is exactly how the algorithm learns. Below is a visual representation of this process I just described.