Artificial Neural Networks - Intro for beginners

The perceptron

Single node perceptron

Perceptrons form the basis for ANNs. Perceptron takes input and produces output as below:

Input ➡️ Activation Function ➡️ Output

Input

If the weight is 0- input remains unaltered coming into the perceptron. Below is what happens to the input

∑ wizi  ≽ t then y = 1. Else y = 0.

 i

t is the threshold which is set by the outgoing part. So the key to output is based on the weighted sum and the threshold.

Activation function

This is the processing part of the neuron and this determines output. So most commonly the ones used are the Sigmoid function and the hyperbolic tangent.

QUESTION: Which activation function should I use?

I am going to talk of three key activation functions

a) Sigmoid

The Sigmoid returns 0 or 1 and in code can be written as

return 1.0/(1.0+Math.exp(-x)

A Sigmoid is a mathematical curve having the characteristic S shape.

DISADVANTAGE: Descending Gradient

b) Tanh

DISADVANTAGE: Descending Gradient

c) RELU - This is the one you should use (especially in the hidden layer)

#Look up Leaky Relu and Maxout

Output

This looks at the Error and this introduces the concept of BackPropagation. This is basically going back, adjusting the weight getting the output and ultimately reducing the error as much as possible.

Backpropagate to update weights!

 

Sample ANN 

aj = wx +b
aj = sigmoid(hj)          -> Or whichever AF you desire
E = y - ŷ (Actual - Prediction)

#Back Propagate as necessary