🧠 Fundamentals of Neural Networks

• Introduction to Neural Networks

  • Bunch of Neurons

• Types of Neural Networks

  • Multi-Layer Perceptron (MLP): Simple feedforward implementation
  • Convolutional Neural Network (CNN): Uses convolution operation, good for image recognition

• Activation Functions

  • Sigmoid: More complicated to compute, not used nowadays.

  $$ \frac{1}{1+e^{-x}} $$

  

  • Rectified Linear Unit (ReLU): Easier to compute, similar performance to Sigmoid, widely used.

  $$ max(0,a) $$

  

• Training vs Inference

• Forward Propagation Basics

• Matrix Operations in NN

🧩 Neuron Architecture

What is a Neuron (Mathematical Model)

• In simple notation, the input of the next layer is given by this function:

$$ a^{(1)}=\sigma(Wa^{(0)}+b) $$

• Here:
  • $a^{(0)}$ = current layer inputs
  • $a^{(1)}$ = next layer inputs
  • W = weights
  • b = bias
  • $\sigma$ = activation function
• Here is the equation in matrix form

$$ \begin{bmatrix} a_0 \\ a_1 \\ \vdots \\ a_{n-1} \end{bmatrix}^{(1)}=\sigma\left( \begin{bmatrix} w_{00} & w_{01} & \cdots & w_{0,n-1} \\ w_{10} & w_{11} & \cdots & w_{1,n-1} \\ \vdots & \vdots & \ddots & \vdots \\ w_{m-1,0} & w_{m-1,1} & \cdots & w_{m-1,n-1} \end{bmatrix} \begin{bmatrix} a_0 \\ a_1 \\ \vdots \\ a_{n-1} \end{bmatrix}^{(0)} + \begin{bmatrix} b_0 \\ b_1 \\ \vdots \\ b_{m-1} \end{bmatrix} \right)

$$

Block Diagram of a Neuron

Fixed Point Representation and Arithmetic

For the arithmetic in the neuron, we will use fixed point representation, which allows us to represent real numbers using integers by allocating a fixed number of bits to the integer and fractional parts. Unlike floating point, fixed point is hardware-efficient and better suited for FPGA implementations where resources and power are limited.