import numpy as np\n\ndef AND(x1, x2):\n x = np.array([x1,x2])\n w = np.array([0.5, 0.5])\n b = -0.7\n temp = np.sum(w*x )+ b\n if temp<= 0:\n return 0\n else:\n return 1\n \ndef NAND(x1,x2):\n x = np.array([x1, x2])\n w = np.array([-0.5, -0.5])\n b = 0.7\n temp = np.sum(w*x)+b\n if temp <= 0:\n return 0\n else: \n return 1\n\ndef Nand(x1, x2): # same as above\n return int(not bool(AND(x1,x2)))\n \n \ndef OR(x1,x2):\n x = np.array([x1, x2])\n w = np.array([0.5, 0.5])\n b = -0.2\n temp = np.sum(w*x) + b\n if temp <=0:\n return 0\n else:\n return 1<\/code><\/pre>\n\n\n\n1.2 XOR Gate – Apply More than Two Perceptrons to Achieve Non-Linear Classification<\/strong><\/h4>\n\n\n\n![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-1024x161.png\")
<\/figure><\/div>\n\n\n![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-1.png\")
<\/figure><\/div>\n\n\n![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-2.png\")
<\/figure><\/div>\n\n\ndef XOR(x1, x2):\n s1 = NAND(x1, x2)\n s2 = OR(x1,x2)\n y = AND(s1, s2)\n return y<\/code><\/pre>\n\n\n\n- It has been proved that any functions could be represented by a combination of 2 perceptrons with sigmoid as the activation function.<\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-3.png\")
<\/figure><\/div>\n\n\nApply Multi – Layers of Perceptrons, we can achieve the simulation of any non-linear transformation. An important part is that Activation Function is necessary to be inserted into different layers. It can be easily proved that: <\/p>\n\n\n\n
$$ W_2\\Bigg( \\big(W_1 X + b_1 \\big) \\Bigg) + b_2 \\equiv W^* x + b^*$$<\/p>\n\n\n\n
Multi-layer of Linear Transformation is still Linear Transformation. So, multi-layer of perceptrons becomes useless without Activation Function.<\/p>\n\n\n\n
However, if activation function, in other word a non-linear transformation, is added between layers, then the neural network could mimic any non-linear function.<\/p>\n\n\n\n
- Therefore, what the Deep Learning is doing is to apply multi-layer perceptrons<\/strong> to mimic the pattern<\/strong> of a certain thing. Some key points are
- (1) multi-layer perceptrons<\/strong> are named the Neural Network<\/strong>. Each adjacent layers are doing linear transformation WX + b<\/span> and then apply a activation function,<\/strong> such as Sigmoid(x) = \\frac{1}{1+e^{-x}}<\/span>; <\/li>
- (2) Update the weight matrix W<\/span> and b<\/span>, until the neural network could output an “optimal” result.<\/li><\/ul><\/li><\/ul>\n\n\n\n
- (1) How to define the layers and network structure, (2) how to evaluate the result is “Optimal” or not, and (3) how to find the weights <\/em><\/strong><\/em><\/strong>are the main problems in Deep Learning.<\/li><\/ul>\n\n\n\n
2. Network Structure – Forward Propagation<\/h2>\n\n\n\n
Let’s start with the first Question. How to define the layers and network structure. Through the Network, <\/p>\n\n\n\n
- we input Data, X<\/span>, initially. <\/li>
- Data are transformed by ( Perceptrons – “Affine”, Activation Functions ) multi-times, which are the multi-layers<\/strong>. <\/li>
- Then, the results are passed through a Softmax<\/strong> function to reform results into percentages. <\/li>
- Finally, a Loss<\/strong> is calculate.<\/li><\/ol>\n\n\n\n
$$ x \\to f(.) \\to a(.) \\to \\text{\u2026} \\to Softmax(.) \\to Loss(.)$$<\/p>\n\n\n\n
, where f(x) = xw + b<\/span><\/p>\n\n\n\n, where a(x)<\/span> is the activation function.<\/p>\n\n\n\n, where Loss(x)<\/span> is the loss function.<\/p>\n\n\n\n2.1 Affine – Linear Transformation<\/h4>\n\n\n\n![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-4.png\")
<\/figure><\/div>\n\n\n$$ f(X) = X \\cdot W +b $$<\/p>\n\n\n\n
2.2 Activation Function<\/h4>\n\n\n\n- ReLU, f(x) = max(x, 0<\/span><\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-5.png\")
<\/figure><\/div>\n\n\n- Sigmoid, f(x) = \\frac{1}{1+e^{-x}}<\/span><\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-6.png\")
<\/figure><\/div>\n\n\n- tanh<\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-7.png\")
<\/figure><\/div>\n\n\nThere are some properties of activation function, such as symmetric to the zero point, differentiable, etc. Those details are not discuss here.<\/p>\n\n\n\n
2.3 Softmax<\/h4>\n\n\n\n
$$ \\vec{X} = (x_1, \u2026, x_i, ..) $$<\/p>\n\n\n\n
$$ Softmax(\\vec{X}) = \\vec{Y} = (\u2026, \\frac{e^{x_i}}{\\sum e^{x_i}} ,\u2026)^T $$<\/p>\n\n\n\n
Inputs are reform to be percentages, and those percentages are sum to be one. <\/p>\n\n\n\n
P.S.<\/p>\n\n\n\n
$$ y_k = \\frac{e^{a_k}}{\\sum_{i=1}^{n}e^{a_i} } = \\frac{C e^{a_k}}{C \\sum_{i=1}^{n}e^{a_i} } $$<\/p>\n\n\n\n
$$ = \\frac{e^{a_k+ lnC} }{\\sum_{i=1}^{n}e^{a_i + lnC} } $$<\/p>\n\n\n\n
$$ = \\frac{e^{a_k – C’} }{\\sum_{i=1}^{n}e^{a_i – C’} } $$<\/p>\n\n\n\n
\u4e3a\u4e86\u9632\u6b62\u6ea2\u51fa\uff0c\u4e8b\u5148\u628ax\u51cf\u53bb\u6700\u5927\u503c\u3002\u6700\u5927\u503c\u662f\u6709\u6548\u6570\u636e\uff0c\u5176\u4ed6\u503c\u6ea2\u4e0d\u6ea2\u51fa\u53ef\u7ba1\u4e0d\u4e86\uff0c\u4e5f\u4e0d\u5173\u5fc3\u3002<\/p>\n\n\n\n
2.4 Loss Function<\/h4>\n\n\n\n- Cross Entropy, L = -\\sum t_i \\ ln(y_i) = – ln(\\vec{Y}) \\cdot \\mathbf{t}^T<\/span><\/li>
- MSE, See “Linear Regression Paddle” note. E = \\frac{1}{2} \\sum_k (y_k – t_k)^2<\/span><\/li><\/ul>\n\n\n\n
3. Minimise Loss & Update Weights<\/h2>\n\n\n\n
We consider the “Optimal” weights are such that they can result in minimum loss. So, in other words, we aim to find w<\/span> and b<\/span> that can minimise the loss function.<\/p>\n\n\n\nHow to Do It ?<\/h4>\n\n\n\n
$$ arg\\min_{w, b} Loss $$<\/p>\n\n\n\n
$$ \\hat{w}: \\frac{\\partial L}{\\partial w} = 0 ,\\quad \\hat{b}: \\frac{\\partial L}{\\partial b} = 0$$<\/p>\n\n\n\n
Remember, the pathway:<\/p>\n\n\n\n
$$ x \\to f(.) \\to a(.) \\to \\text{\u2026} \\to Softmax(.) \\to Loss(.)$$<\/p>\n\n\n\n
To solve the `F.O.C, we need to apply the Chain rule backward, which is called the Backward Propagation.<\/strong><\/p>\n\n\n\nBy Chain Rule,<\/strong><\/p>\n\n\n\n$$ \\frac{\\partial Loss(w)}{\\partial w} = \\frac{\\partial Loss(.)}{\\partial Softmax} \\cdot \\frac{\\partial Softmax(.)}{\\partial a} \\cdot \\frac{\\partial a}{\\partial f} \\cdot \\frac{\\partial f}{\\partial a} \u2026 \\cdot \\frac{\\partial f}{\\partial w} $$<\/p>\n\n\n\n
Let’s Decompose each Part in the Following.<\/p>\n\n\n\n
3.1 Loss<\/h4>\n\n\n\n
$$ \\frac{\\partial Loss(\\vec{Y}, \\vec{t})}{\\partial \\vec{Y}} $$<\/p>\n\n\n\n
- Cross Entropy<\/li><\/ul>\n\n\n\n
$$ L = -\\sum t_i \\ ln(y_i) = – ln(\\vec{Y}) \\cdot \\mathbf{t}^T$$<\/p>\n\n\n\n
$$\\frac{\\partial L}{\\partial y_i} = – \\frac{t_i}{y_i}$$<\/p>\n\n\n\n
$$\\frac{\\partial L}{\\partial \\vec{Y}} = – \\big(\u2026,\\frac{t_i}{y_i},\u2026\\big)^T$$<\/p>\n\n\n\n
3.2 Softmax<\/h4>\n\n\n\n
$$ \\vec{X} = (x_1, \u2026, x_i, ..) $$<\/p>\n\n\n\n
$$ Softmax(\\vec{X}) = \\vec{Y} = (\u2026, \\frac{e^{x_i}}{\\sum e^{x_i}} ,\u2026)^T $$<\/p>\n\n\n\n
$$ \\frac{\\partial Softmax(\\vec{X})}{\\partial \\vec{X}} = Diag(\\vec{Y}) – Y\\cdot Y^T $$<\/p>\n\n\n\n
\n\n\n\nDeviation is as the Following,<\/strong><\/p>\n\n\n\nhttps:\/\/blog.csdn.net\/Wild_Young\/article\/details\/121912675<\/a><\/p>\n\n\n\n$\\frac{\\partial Y}{\\partial X}$:<\/p>\n\n\n\n
$$Softmax(x) = \\frac{1}{1+e^{-x}}$$<\/p>\n\n\n\n
Let X<\/span> be a vector with shape = (n,1), and
$Y = Softmax(X) $.<\/p>\n\n\n\nThat means, for each element of y_i<\/span> in Y<\/p>\n\n\n\n$$ y_k = \\frac{-e^{x_k}}{\\sum_i^N e^{x_i}} $$<\/p>\n\n\n\n
So,<\/p>\n\n\n\n
$$ \\frac{\\partial Y}{\\partial x_k} $$<\/p>\n\n\n\n
If j = i<\/span>,<\/p>\n\n\n\n$$ \\frac{\\partial y_j}{\\partial x_i} = \\frac{\\partial }{\\partial x_i} \\Bigg( \\frac{e^{x_i}}{\\sum e^{x_i}} \\Bigg) $$<\/p>\n\n\n\n
$$ = \\frac{ (e^{x_i})’\\sum e^{x_i} – e^{x_i} (\\sum e^{x_i})’ }{\\big( \\sum e^{x_i} \\big)^2} $$<\/p>\n\n\n\n
$$ =\\frac{e^{x_i}}{\\sum e^{x_i}} – \\frac{e^{x_i}}{\\sum e^{x_i}} \\cdot \\frac{e^{x_i}}{\\sum e^{x_i}}$$<\/p>\n\n\n\n
$$= y_i – y_i^2 =y_i(1-y_i)$$<\/p>\n\n\n\n
If j \\neq i<\/span>,<\/p>\n\n\n\n$$ \\frac{\\partial y_j}{\\partial x_i} = \\frac{\\partial }{\\partial x_i} \\Bigg( \\frac{e^{x_j}}{\\sum e^{x_i}} \\Bigg) $$<\/p>\n\n\n\n
$$ = \\frac{-e^{x_i} e^{x_j}}{(\\sum e^{x_i})^2} $$<\/p>\n\n\n\n
$$ =- \\frac{e^{x_j}}{\\sum e^{x_i}} \\cdot \\frac{e^{x_i}}{\\sum e^{x_i}} $$<\/p>\n\n\n\n
$$ = -y_j y_i $$<\/p>\n\n\n\n
$$ \\frac{\\partial y_j}{\\partial x_i}=y_i – y_i^2 \\quad \\text{,if $i = j$} $$<\/p>\n\n\n\n
$$\\frac{\\partial y_j}{\\partial x_i} = -y_j y_i \\quad \\text{,if $i \\neq j$} $$<\/p>\n\n\n\n
Therefore, for \\frac{\\partial Y}{\\partial X}<\/span>, we got the Jacobian matrix,<\/p>\n\n\n\n$$\\frac{\\partial \\vec{Y}}{\\partial \\vec{X}} = Diag(Y) – Y^T Y$$<\/p>\n\n\n\n
However, in the backward propagation of Softmax, we only need the diagonal, which is in the case of i = j.<\/p>\n\n\n\n
$$ \\frac{\\partial y_i}{\\partial x_i} = y_i-y_i^2 =y_i(1-y_i)$$<\/p>\n\n\n\n
3.3 Loss (Cross-Entropy) and Softmax<\/h4>\n\n\n\n
There is a trick with the combination between cross-entropy-error and softmax.<\/p>\n\n\n\n
The Cross-Entropy Loss function is, <\/p>\n\n\n
\n![\"\"](\"https:\/\/fanyuzhao.com\/wp-content\/uploads\/2023\/01\/image-8.png\")
<\/figure><\/div>\n\n\n, where \\vec{Y_{log}}^T<\/span> is apply log to each element of \\vec{Y}<\/span>, and then take transformation.<\/p>\n\n\n\nhttps:\/\/www.matrixcalculus.org<\/a><\/p>\n\n\n\n$$\\frac{\\partial L}{\\partial \\vec{Y}} = \\vec{\\mathbf{t}}^T \\cdot Diag(Y_{1\/Y}) $$<\/p>\n\n\n\n
, where Y_{1\/Y}<\/span> is 1\/y_i<\/span> for each element of Y<\/span>.<\/p>\n\n\n\n$$ \\frac{\\partial L}{\\partial x_k } = \\frac{\\partial L}{\\partial y_k}\\cdot \\frac{\\partial y_k}{\\partial x_k} $$<\/p>\n\n\n\n
$$\\frac{\\partial L}{\\partial x_k } =- \\sum_i \\frac{\\partial }{\\partial {y}_k} \\bigg( t_i \\ ln(y_i) \\bigg) \\cdot \\frac{\\partial y_k}{\\partial x_k} $$<\/p>\n\n\n\n
$$\\frac{\\partial L}{\\partial x_k } =- \\sum_i \\frac{t_i }{{ y}_i} \\cdot \\frac{\\partial {y}_i}{\\partial x_k} $$<\/p>\n\n\n\n
Plug in the derivatives of Softmax<\/strong>, \\frac{\\partial y}{\\partial x}<\/span><\/p>\n\n\n\n$$\\frac{\\partial L}{\\partial x_k } = – \\frac{t_k }{{ y}_k}
\\cdot \\frac{\\partial {y}_k}{\\partial x_k} \\sum_{i\\neq k} \\frac{t_i }{{ y}_i} \\cdot \\frac{\\partial {y}_i}{\\partial x_k} $$<\/p>\n\n\n\n
$$ = -\\frac{t_k }{{ y}_k}
\\cdot {y}_k (1-{y}_k) \\sum_{i\\neq k} \\frac{t_i }{{ y}_i} \\cdot (- {y}_i {y}_k)
$$<\/p>\n\n\n\n
$$ = – t_k (1-{y}_k) \\sum_{i\\neq k} t_i {y}_k
$$<\/p>\n\n\n\n
$$ = – t_k \\sum_{i} t_i \\ {y}_k
$$<\/p>\n\n\n\n
$$ = – t_k {y}k \\sum{i} t_i \\quad\\text{by the tag, $t$, is sum to be 1}
$$<\/p>\n\n\n\n
$$ \\frac{\\partial L}{\\partial x_k } ={y}_k – t_k$$<\/p>\n\n\n\n
So, we get \\frac{\\partial L}{\\partial x_k} ={y}_k – t_k<\/span>.<\/p>\n\n\n\n3.4 Affine<\/h4>\n\n\n\n
$$ \\frac{\\partial f}{\\partial w} $$<\/p>\n\n\n\n
$$ \\frac{\\partial f}{\\partial b} $$<\/p>\n\n\n\n
- Backward<\/li><\/ul>\n\n\n\n
$$ \\frac{\\partial f}{\\partial X} =\\cdot \\ W^T \\quad \\text{, Right Multiplied}$$<\/p>\n\n\n\n
$$ \\frac{\\partial f}{\\partial W} = X^T \\cdot \\ \\quad \\text{, Left Multiplied}$$<\/p>\n\n\n\n
$$ \\frac{\\partial f}{\\partial b} = \\mathbf{1}^T \\cdot \\quad \\text{, Left Multiplied} $$<\/p>\n\n\n\n
4. Other Theoretical Details<\/h2>\n\n\n\n- Batch<\/strong>: mini-batch and epoch for improving training accuracy.<\/li>
- Weights Initialisation<\/strong>: Xavier – Sigmoid and tank, He – ReLU<\/li>
- Weights Updating<\/strong>: SGD, Momentum, Adam, etc.<\/li>
- Overfitting<\/strong>: Batches, Regularisation, Dropout.<\/li>
- Hyperparameters<\/strong>: Bayes<\/li><\/ol>\n\n\n\n
See Chapter 6 of the book.<\/p>\n\n\n\n
5. Code<\/h2>\n\n\n\n
See Attachment for the Code Realisation by purely Numpy.<\/p>\n\n\n\n
Collection_of_Classes<\/a>Download<\/a><\/div>\n\n\n\n<\/p>\n\n\n\n
Code and Books: http:\/\/82.157.170.111:1011\/s\/9D9BCBbCop6ERaD<\/p>\n","protected":false},"excerpt":{"rendered":"
1. Perceptron 1.1 AND & OR & ‘N’ Gate Linear Classification might be applied by AND & OR. 1.2 XOR Gate – Apply More than Two Perceptrons to Achieve Non-Linear Classification It has been proved that any functions could be represented by a combination of 2 perceptrons with sigmoid as the activation function. Apply Multi … Continue reading Deep Learning from Scratch<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,8],"tags":[],"_links":{"self":[{"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=\/wp\/v2\/posts\/4880"}],"collection":[{"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4880"}],"version-history":[{"count":115,"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=\/wp\/v2\/posts\/4880\/revisions"}],"predecessor-version":[{"id":5048,"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=\/wp\/v2\/posts\/4880\/revisions\/5048"}],"wp:attachment":[{"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4880"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4880"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fanyuzhao.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4880"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}