Ch04.Neural_network_learning.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 신경망 학습\n",
    "---\n",
    "**학습**이란 훈련 데이터로부터 가중치 매개변수의 최적값을 자동으로 획득하는 것을 뜻한다.  \n",
    "신경망이 학습할 수 있도록 해주는 **지표**인 손실 함수를 소개한다. 이 손실 함수의 결괏값을 가장 작게 만드는 가중치 매개변수를 찾는 것이 학습의 목표이다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 데이터에서 학습\n",
    "데이터에서 학습한다는 것은 가중치 매개변수의 값을 데이터를 보고 자동으로 결정한다는 뜻이다.\n",
    "\n",
    "### 4.1.1 데이터 주도 학습\n",
    "기계학습은 데이터가 생명이다. 데이터에서 답을 찾고 데이터에서 패턴을 발견하고 데이터로 이야기를 만드는, 그것이 바로 기계학습이다.  \n",
    "그런 방법의 하나로, 이미지에서 **특징**<sup>feature</sup>을 추출하고 그 특징의 패턴을 기계학습 기술로 학습하는 방법이 있다.  \n",
    "이미지의 특징은 보통 벡터로 기술하고, 컴퓨터 비전 분야에서는 SIFT, SURF, HOG 등의 특징을 많이 사용한다.  \n",
    "이런 특징을 사용하여 이미지 데이터를 벡터로 변환하고, 변환된 벡터를 가지고 지도 학습 방식의 대표 분류 기법인 SVM, KNN 등으로 학습할 수 있다.\n",
    "\n",
    "하지만 신경망은 이미지를 '있는 그대로' 학습한다.\n",
    "\n",
    "**※ NOTE**  \n",
    "딥러닝을 **종단간 기계학습**<sup>end-to-end machine learning</sup>이라고 한다.  \n",
    "종단간은 '처음부터 끝까지' 라는 의미로, 데이터(입력)에서 목표한 결과(출력)을 사람의 개입 없이 얻는다는 뜻이다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1.2 훈련 데이터와 시험 데이터\n",
    "기계학습 문제는 데이터를 **훈련 데이터**<sup>training data</sup>와 **시험 데이터**<sup>test data</sup>로 나눠 학습과 실험을 수행하는 것이 일반적이다.  \n",
    "우선 훈련 데이터만 사용하여 학습하면서 최적의 매개변수를 찾는다. 그런 다음 시험 데이터를 사용하여 앞서 훈련한 모델의 실력을 평가하는 것이다.  \n",
    "이를 구분하는 이유는 범용적으로 사용할 수 있는 모델을 원하기 때문이다.  \n",
    "그래서 데이터셋 하나로만 매개변수의 학습과 평가를 수행하면 올바른 평가가 될 수 없다. 한 데이터셋에만 지나치게 최적화된 상태를 **오버피팅**<sup>overfitting</sup> 이라고 한다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 손실 함수\n",
    "신경망 학습에서는 현재의 상태를 '하나의 지표'로 표현한다. 그리고 그 지표를 가장 좋게 만들어주는 가중치 매개변수의 값을 탐색하는 것이다.  \n",
    "최적의 매개변수를 탐색하는데 사용되는 지표를 **손실 함수**<sup>loss function</sup>라고 한다.  \n",
    "일반적으로 손실 함수는 평균 제곱 오차와 교차 엔트로피 오차를 사용한다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.1 평균 제곱 오차\n",
    "가장 많이 쓰이는 손실 함수는 **평균 제곱 오차**<sup>mean squared error, MSE</sup>이다.\n",
    "\n",
    "**[식 4.1]**  \n",
    "$\n",
    "\\begin{align}\n",
    "E = \n",
    "\\frac{1}{n} \\sum_{i=1}^n(y_i - t_i)^2\n",
    "\\end{align}\n",
    "$\n",
    "\n",
    "- $y_i$는 신경망의 출력(신경망이 추정한 값)\n",
    "- $t_i$는 정답 레이블\n",
    "- $n$은 데이터의 차원 수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = [0.1, 0.05, 0.6, 0.0, 0.05, 0.1, 0.0, 0.1, 0.0, 0.0]\n",
    "t = [0, 0, 1, 0, 0, 0, 0, 0, 0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "신경망의 출력 y 는 소프트맥스 함수의 출력이다.  \n",
    "정답 레이블인 t는 정답을 가리키는 위치의 원소는 1로, 그 외에는 0으로 표기한다. 이는 **원-핫 인코딩** 이라 한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_squared_error(y, t):\n",
    "    return np.sum((y-t)**2) / t.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.019500000000000007"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "0.11950000000000001"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t = [0, 0, 1, 0, 0, 0, 0, 0, 0, 0]\n",
    "\n",
    "y = [0.1, 0.05, 0.6, 0.0, 0.05, 0.1, 0.0, 0.1, 0.0, 0.0]\n",
    "\n",
    "# 첫 번째 결과\n",
    "display(mean_squared_error(np.array(y),np.array(t)))\n",
    "\n",
    "y = [0.1, 0.05, 0.1, 0.0, 0.05, 0.1, 0.0, 0.6, 0.0, 0.0]\n",
    "\n",
    "# 두 번째 결과\n",
    "display(mean_squared_error(np.array(y),np.array(t)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "위의 평균 제곱 오차의 결과는 첫 번째 추정 결과가 (오차가 더 작으므로) 정답에 더 가깝다는 것을 알 수 있다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2.2 교차 엔트로피 오차\n",
    "또 다른 손실 함수로서 **교차 엔트로피 오차**<sup>cross entropy error, CEE</sup>도 자주 이용한다.\n",
    "\n",
    "**[식 4.2]**  \n",
    "$\n",
    "\\begin{align}\n",
    "E = \n",
    "-\\sum_{i=1}^n t_i \\log y_i\n",
    "\\end{align}\n",
    "$\n",
    "\n",
    "- $\\log$는 밑이 $e$인 자연로그 $\\log_e$\n",
    "- $y_i$는 신경망의 출력\n",
    "- $t_i$는 정답 레이블"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHrdJREFUeJzt3Xt81PWd7/HXNxcIuZM7JIQkkEACCGIAXRWrIsXWa9V6abdusbV2j/U8ejn2orbr9tjt0d263V52pd1aj2fVVo9WWy/1igiKAspNIFxCAgFym5A7k9t8948ZFDGQwUzmN7+Z9/PxmEdnmF9/8/ky4e0339/39/0aay0iIhI94pwuQEREQkvBLiISZRTsIiJRRsEuIhJlFOwiIlFGwS4iEmUU7CIiUUbBLiISZRTsIiJRJsGJD83JybElJSVOfLSIiGtt2LCh1VqbO9JxjgR7SUkJ69evd+KjRURcyxhTH8xxGooREYkyCnYRkSijYBcRiTKOjLEPZ2BggIaGBrxer9OljEpSUhJFRUUkJiY6XYqIxKiQBLsxZhnwcyAe+K219qeneo6GhgbS0tIoKSnBGBOKssLOWovH46GhoYHS0lKnyxGRGDXqoRhjTDzwK+BioAq43hhTdarn8Xq9ZGdnuzbUAYwxZGdnu/63DhFxt1CMsS8Edltra621/cBjwOWf5ERuDvWjoqENIuJuoRiKKQT2H/O6AVgUgvOKiLiatZbW7n7qPT3UeXqp9/Tw+eopTMlKHtPPDUWwD9dF/dhGqsaYm4GbAYqLi0PwsaGXmppKd3e302WIiIscDe86Tw97W3uoa+2h3tNLncf/v919gx8cG2dgfvFEVwR7AzDlmNdFwMHjD7LWrgBWAFRXV2sHbRFxlY7eAWpbu/0B3tLDXk8vda3+MD82vBPiDEUTJ1CSk8KCkiymZidTkpNCSXYKhZkTGJcw9rPMQxHs64ByY0wpcAC4DrghBOd1jLWW22+/neeffx5jDHfeeSfXXnstPp+PW2+9lddff53S0lJ8Ph/Lly/n6quvdrpkEQkB78AQ+9p6qW3pprbVH+C1gfBu6+n/4Lg4A4UTJ1CSncJV8wspzUlhak4KpdkpFE2cQEK8s7cIjTrYrbWDxphbgb/in+74O2vt+6M5591/fp9tBztHW9pHVE1O50eXzgrq2CeffJKNGzeyadMmWltbWbBgAYsXL2bNmjXU1dWxZcsWmpubqaysZPny5SGtU0TGlrWWlq4+drd0U9vSQ21LD3tauqlt7ebA4SP4jhlPyEsbT2lOCkur8inLTaE0J5XSnGSmZCUzPiHeuUaMICTz2K21zwHPheJckWD16tVcf/31xMfHk5+fz3nnnce6detYvXo111xzDXFxcRQUFHD++ec7XaqInMDAkI96Ty97WrrZ3dzNnpZu9rT0UNvcTdcxQycTEuMpzUlhblEmV55exLTcFEpz/I+0JHfeaBgxd54eK9ie9VixdvhLACf6cxFxzpH+oQ/Ce1dzF7ub/c/rPb0MHtP9LkhPYlpeClecXsi03BSm5aVSlpvKpPQk4uKia5pyRAa70xYvXswDDzzAjTfeSFtbG6tWreK+++6jr6+Phx56iBtvvJGWlhZWrlzJDTe4+nKCiGt09w2yq6mLXYHgPvr8QPsRjva54uMMU7OTmZ6byqdnFTA9L5VpuamU5bq39/1JKNiHceWVV/LWW28xd+5cjDHce++9FBQUcNVVV/HKK68we/ZsKioqWLRoERkZGU6XKxJVevsH2dXUzc5AcNc0drGrqYuDHR/e0T0uIY6ynBROL57INWdMoTw/lfK8VKZmp4Rl1kmkM04ML1RXV9vjN9rYvn07lZWVYa/lVHV3d5OamorH42HhwoWsWbOGgoKCjxzjlraIOKl/0Edtqz+4dzZ1UdPoD/N9bb0fHDMuIY5pualU5KdSkZ9GeV4q5flpFGclEx9lwyfBMMZssNZWj3Sceuyn6JJLLqG9vZ3+/n7uuuuuj4W6iHyUtZYD7UeoaexiR+BR09hJbUvPB2PgCXGG0pwU5hRmcPUZRVTkp1GR7++Bx2KAj5aC/RStXLnS6RJEIlZP3yA1TV1sP9TJjkNd7GjsZEdjF13eD2ehFGZOYGZBGksq85lRkMaMgjTKclI1hBJCERXs1lrXL6KlmTMSC6y1HOzwsv1gJ9sOdbI98Khv6/3gQmba+ARmFKRxxbxCZhSkMTMQ4rF0EdMpERPsSUlJeDweVy/de3Q99qSkJKdLEQmZwSEfe1p6eP9gB9sCQb7tUCftvQMfHDM1O5nKgnSuPL2IyklpVE5Kp2jiBNf+W3a7iAn2oqIiGhoaaGlpcbqUUTm6g5KIG3kHhqhp7GLrwQ7eP9jJ+wc62NHYRd+gD4DxCXHMnJTOxbMnUTUpjarJ6cwoSCd1fMREiRBBwZ6YmKhdh0TC6Ej/ENsOdbL1QAdbDnSw9UAHu5q7GQpc0ExPSmDW5Az+9sypzCpMZ9bkDMpyUhxfB0VGFjHBLiJjxzswxI7GLrY0tLO5wR/kx4Z4dso4ZhdmcGFlHrMnZzC7MENDKS6mYBeJMoNDPnY1d7O5oZ1NDR1sbminprGLgaEPQ3xOUQYXVeUzpzCDOUUZFKQnKcSjiIJdxMWOzhHftL+DTQ3tbNzXzpYDHRwZGAIgLSmB04oy+Mq5ZcwtymBOUSaTMxTi0U7BLuIiPX2D/gDf3857+/yP1u4+wH+X5qzJ6Vy7YApzp2QwtyiTkuyUqFvgSkamYBeJUNZa6jy9vFt/mHf3Hebdfe3UNHZ+sF54aU4Ki8tzmFecybwpmcwsSNdNPgIo2EUihndgiM0NHWyoP8yGQJgf3bUnLSmBeVMyueiCcuYHgjwzeZzDFUukUrCLOMTT3cf6QIivq2tj64GODy5wluWkcOHMPOZPncj84omU56VqSEWCpmAXCZOGw72sq2vjnb3+x56WHgDGxcdxWlEGy88ppXpqFmdMnUhWinrj8skp2EXGgLWWva09vLO3jbcDQX6g/Qjgv/GnuiSLq84oYkFJFnMKM0hKjNz9M8V9FOwiIWCtpba1h7W1HtbWtvF2rYfmLv9slZzUcSwszeLmxWUsLM1iRn6ahlVkTCnYRT6h/W29vLmnlTf3eHhrz4dBnpc2nrOmZbOoNJtFZVmU5aRo3riElYJdJEgtXX3+IN/t4c3aVva3+YdWclL9QX5WWTZnlmVRqiAXhynYRU6gt3+Qd/a2sXpXK6t3t7KjsQvwj5GfNS2br5xTxt9My2Z6XqqCXCKKgl0kwOezbDvUyRu7WnljVwvr6w7TP+RjXEIc1VMncvuyGZwzPYdZkzO0XZtENAW7xDRPdx9v7Gpl1c4WVu1qobXbf0NQ5aR0/u7sEs6ZnsOCkiwmjNOsFXEPBbvEFJ/PsqmhnddqWni9ppnNBzqwFrJSxnFueQ6Ly3M5tzyHvHTtgiXupWCXqNfpHWDVzhZe3d7M6ztb8PT0YwzMm5LJN5dUcF5FLnMKMzQFUaKGgl2iUr2nh5e2NfHK9mbW1bUx6LNkJidyXkUuF8zMY3F5LhN1d6dEKQW7RIWjQywvbmvi5W1N7GruBqAiP5WvLi7jwpl5zJuSqW3dJCYo2MW1+gd9rK318Nf3G3lpWxPNXX3ExxkWlmRx/cJillTmU5yd7HSZImGnYBdX8Q4M8frOFl7Y2sjL25vo8g6SPC6eT83IZWlVAefPyCMjOdHpMkUcpWCXiHekf4jXapp5dsshXtvRTG//EJnJiXx6VgHLZhVwTnmOFtESOYaCXSKSd2CIlTXN/HnzIV7d3syRgSFyUsdx5emFXDx7EovKskjUeLnIsBTsEjEGhny8sauFP286xIvvN9LT7w/zq84o5DNzJrGoNFt3fIoEYVTBboy5BvgHoBJYaK1dH4qiJHb4fJYN+w7zp/cO8NyWQxzuHSBjQiKXzp3MpXMns6g0SzNZRE7RaHvsW4HPAQ+EoBaJIXtaunnq3QP8aeMBGg4fYUJiPBdV5XPZ3MksrsjVpswiozCqYLfWbge0sp0Epb23nz9vOsgT7x5g0/524gycU57Lt5dWsLSqgJTxGhkUCQX9S5IxNeSzvLGrhcfXN/DStib6h3zMLEjjjs9Ucvm8yVqTRWQMjBjsxpiXgYJh3rrDWvt0sB9kjLkZuBmguLg46ALFnfa39fLH9ft5fH0DjZ1eJiYncsOiYq6pLmLW5AynyxOJaiMGu7V2SSg+yFq7AlgBUF1dbUNxToks/YM+XtrWxCPv1LNmt4c4A+dV5PKjS6u4sDJf4+YiYaKhGBm1/W29PPLOPh5fv5/W7n4KMyfwrYsquPqMIiZnTnC6PJGYM9rpjlcCvwBygWeNMRuttZ8OSWUS0Xw+y+s7W3h4bT2v1TRjgAsr87lhUTGLy3M131zEQaOdFfMU8FSIahEX6DgywOPr9/Pw2nrqPb3kpo3nG+dP57qFxeqdi0QIDcVIUPa29vDgmr08saGB3v4hqqdO5NtLZ7BsVoHGzkUijIJdTshay9raNn77Ri2v1jSTGBfHpXMn8+WzS5hdqJktIpFKwS4fMzjk47mtjfxmVS1bDnSQnTKOb1xQzhfPLCYvTfPORSKdgl0+4B0Y4vH1+3lgVS0Nh49QlpPCT66cw+fmF2pZXBEXUbALXd4BHl5bz+9W76W1u5/TizP54SVVLKnM1wbPIi6kYI9hHb0DPPjmXh5cU0fHkQEWV+Ty95+axqLSLK3/I+JiCvYY1HFkgP9cvZcHV++lq2+QpVX53HrBdE4rynS6NBEJAQV7DOnyDvC71XX8dnUtXd5BLp5dwG0XllM5Kd3p0kQkhBTsMcA7MMTDb9Xz65W7Odw7wEVV+XxzSQVVkxXoItFIwR7FhnyWJzbs5/6XdtHY6eXc8hy+s3QGc6doyEUkminYo5C1lld3NPPT53ewq7mbeVMy+dfr5nFmWbbTpYlIGCjYo8y2g53872e38eYeD6U5Kfz7F+azbHaBZrmIxBAFe5Ro7e7jX16s4bF1+8mYkMjdl83ihkXFJGojaJGYo2B3uYEhHw+/Vc/9L+/kSP8Qy88u5bYLyslITnS6NBFxiILdxd6u9XDX01vZ2dTNueU5/OjSWUzPS3W6LBFxmILdhTzdffzT8zt4YkMDhZkTeOBvz2BpVb7G0UUEULC7irWWJzY0cM9z2+n2DvL1T03jtgvKmTBOC3SJyIcU7C6xz9PLD57awurdrSwomcg9V86hIj/N6bJEJAIp2COcz2f5/Zt13PfXGuLjDD++YjZfWFisVRdF5IQU7BFsn6eX//XEJt7e28b5M3K558o52ldUREakYI9A1loeW7efH/9lG3HGcO/Vp3HNGUW6OCoiQVGwR5i2nn6+9/838+K2Js6ens29V8+lUL10ETkFCvYIsnpXK9/640baewe487OVLD+7VGPpInLKFOwRYHDIx89f2cUvX9vNtNxUHvzyAmZNznC6LBFxKQW7w5o6vdz26Hu8vbeNa84o4u7LZ5E8Tl+LiHxyShAHvV3r4X888i49fUP87PNz+dz8IqdLEpEooGB3gLX+uen3PLud4qxkHv3qmZTrZiMRCREFe5h5B4b4wZNbePK9AyypzOdn184lPUkrMYpI6CjYw6ilq4+vPbyed/e1880lFXzjguma9SIiIadgD5Pthzr5ykPr8fT08e9fmM/FcyY5XZKIRCkFexi8sauFWx7eQGpSAo9/7W+YU6SpjCIydhTsY+xP7x3gO49vYnpeKr//8kIKMpKcLklEopyCfQytWLWHnzy3gzPLsljxpWpdJBWRsFCwjwFrLff9tYZfr9zDZ0+bxM8+P5fxCdoMQ0TCY1Rb2Btj7jPG7DDGbDbGPGWMyQxVYW5lreXuP2/j1yv3cMOiYn5x3ekKdREJq1EFO/ASMNtaexqwE/j+6EtyryGf5ftPbuH3b9Zx0zml3HPFbE1nFJGwG1WwW2tftNYOBl6uBWL2nnifz/L9Jzfz2Lr9fOOC6dz52Uqtny4ijhhtj/1Yy4HnQ3g+17DW8sNntvLH9Q3cdsF0vr10hkJdRBwz4sVTY8zLQMEwb91hrX06cMwdwCDwXyc5z83AzQDFxcWfqNhIZK3lH/+yjf+3dh+3nDeNb15U4XRJIhLjRgx2a+2Sk71vjLkRuAS40FprT3KeFcAKgOrq6hMe5zb3v7STB9fUsfzsUr67TD11EXHeqKY7GmOWAd8FzrPW9oamJPd4eG09//bqbj5fXcRdl2hMXUQiw2jH2H8JpAEvGWM2GmP+IwQ1ucILWw/xw6e3cuHMPH5y5RyFuohEjFH12K2100NViJusq2vjtsc2cvqUTH55w3wS4kN5DVpEZHSUSKdof1svX3t4A0WZE/jPGxcwYZxuPhKRyKJgPwXdfYN85aH1DA75+O2N1UxMGed0SSIiH6O1YoI05LP8z0ffY3dLNw99eSFlualOlyQiMiz12IN0/0s7eWVHMz+6tIpzynOcLkdE5IQU7EF4raaZX762m2urp/Cls0qcLkdE5KQU7CM42H6Eb/1hIzML0rj78llOlyMiMiIF+0kMDPm49ZF36R/08esvzCcpUTNgRCTy6eLpSfzzizW8u6+dX1x/ui6WiohrqMd+Au/sbWPFqlquX1jMpXMnO12OiEjQFOzD6O4b5NuPb2TKxGTu/Gyl0+WIiJwSDcUM455nt9Nw+Ah//NpZpIzXX5GIuIt67Md5bUczj76zj5vPLWNBSZbT5YiInDIF+zF6+gb5wVNbqMhP1YYZIuJaCvZj/OvLOznU4eWfPjdHUxtFxLUU7AHbD3XyuzV1XLdgCmdM1RCMiLiXgh3w+Sx3/mkrGRMS+e6ymU6XIyIyKgp24IkNDWyoP8z3Lp6ppXhFxPViPti7vAP8nxd2sKBkIlfPL3K6HBGRUYv5YP/Nqlo8Pf3cdUkVcXHat1RE3C+mg725y8tv3tjLJadN4rSiTKfLEREJiZgO9p+/vIuBIR/fWTrD6VJEREImZoO9tqWbx9bt54ZFxZTkpDhdjohIyMRssP/zizUkJcRx24XlTpciIhJSMRnsNY1dPLelkZvOKSUndbzT5YiIhFRMBvsDr+8heVw8Xz671OlSRERCLuaCveFwL09vOsj1C4t1M5KIRKWYC/bfvrEXA9x0jnrrIhKdYirYPd19PLZuH1ecXsjkzAlOlyMiMiZiKtgfeqse74CPW84rc7oUEZExEzPBfqR/iIferGNpVT7T89KcLkdEZMzETLD/ZfNBOo4MsFxj6yIS5WIm2B9bt5+y3BQWlWoTDRGJbjER7DubuthQf5jrFxRjjFZwFJHoFhPB/ug7+0iMN3xufqHTpYiIjLlRBbsx5sfGmM3GmI3GmBeNMZNDVVioeAeGeOq9A3x6VgHZWj5ARGLAaHvs91lrT7PWzgP+AvwwBDWF1F/fb6S9d4DrFxY7XYqISFiMKtittZ3HvEwB7OjKCb1H3t7H1OxkzirLdroUEZGwSBjtCYwx9wBfAjqA80ddUQjVe3p4e28bty+boW3vRCRmjNhjN8a8bIzZOszjcgBr7R3W2inAfwG3nuQ8Nxtj1htj1re0tISuBSfx3JZGAC6fp4umIhI7RuyxW2uXBHmuR4BngR+d4DwrgBUA1dXVYRmyeeH9RuYWZVCodWFEJIaMdlbMsdsPXQbsGF05oXOw/Qib9rezbPYkp0sREQmr0Y6x/9QYMwPwAfXALaMvKTRe2Oofhlk2u8DhSkREwmtUwW6tvSpUhYTaC1sbmVmQRqk2qhaRGBOVd542d3lZV9+m3rqIxKSoDPYX32/CWrhY4+siEoOiMthf2NpIWU4KFfmpTpciIhJ2URfs7b39vFXr4dOzC7SSo4jEpKgL9lW7WhnyWZZW5TtdioiII6Iu2NfWekgbn8CcwgynSxERcUT0BfseDwtLs0iIj7qmiYgEJarSr6nTS21rD2dqJUcRiWFRFexraz0ACnYRiWlRF+xpSQlUTU53uhQREcdEWbC3sag0i3itvS4iMSxqgr2xw8teja+LiERPsL+9V+PrIiIQRcH+1h4P6UkJVE7S+LqIxLaoCfa1tR4WlmZrfF1EYl5UBPuhjiPUeXo5syzL6VJERBwXFcH+zt42QOPrIiIQJcG+7VAn4+LjmFGQ5nQpIiKOi4pg33Goi2l5qSRqfRgRkegI9prGLirVWxcRAaIg2A/39NPY6dUwjIhIgOuDfUdjFwAzNX9dRASIgmCvaewE0FCMiEiA64N9R2MXE5MTyU0b73QpIiIRISqCfWZBujauFhEJcHWw+3yWnU1dunAqInIMVwf7/sO99PYPUTlJwS4icpSrg337ocCMmALNiBEROcrVwV7T2IUxUJGvHruIyFGuDvYdjZ2UZKcwYVy806WIiEQMlwd7FzPUWxcR+QjXBvuR/iHqPD3M1IVTEZGPcG2w72zqwlpdOBUROZ5rg73m6BoxmsMuIvIRIQl2Y8x3jDHWGJMTivMFo87TQ0KcYUpWcrg+UkTEFUYd7MaYKcBFwL7RlxO8xk4v+elJ2rxaROQ4oeix3w/cDtgQnCtoTZ1e8tK18JeIyPFGFezGmMuAA9baTSGqJ2hNnX0UpCeF+2NFRCJewkgHGGNeBgqGeesO4AfA0mA+yBhzM3AzQHFx8SmUOLymDi/nTA/bkL6IiGuMGOzW2iXD/bkxZg5QCmwKLJlbBLxrjFlorW0c5jwrgBUA1dXVoxq26ekbpKtvkHz12EVEPmbEYD8Ra+0WIO/oa2NMHVBtrW0NQV0n1dTpBaAgQ2PsIiLHc+U89sZAsKvHLiLycZ+4x348a21JqM41kiYFu4jICbmyx97U2QegWTEiIsNwZbA3dnhJG59AyviQ/cIhIhI1XBnsujlJROTEXBvsBRkahhERGY5Lg71PF05FRE7AdcHu81maAguAiYjIx7ku2Nt6+xn0Wc2IERE5AdcFe2OH5rCLiJyM64L9w5uTNCtGRGQ4Lgz2wM1JmhUjIjIs1wV7Y6cXYyA3VT12EZHhuC7Ymzq85KSOJyHedaWLiISF69KxqcurGTEiIifhumBv7NAcdhGRk3FdsPtvTtL4uojIibgq2L0DQxzuHdBQjIjISbgq2Fu6/FMd8zXVUUTkhFwV7NoST0RkZO4K9sByAhqKERE5MVcF+9HlBBTsIiIn5rpgH58QR/oEbYknInIirgr2abmpXDGvEGOM06WIiEQsV3V9r1tYzHULi50uQ0Qkormqxy4iIiNTsIuIRBkFu4hIlFGwi4hEGQW7iEiUUbCLiEQZBbuISJRRsIuIRBljrQ3/hxrTAtSfwv8lB2gdo3IimdodW2K13RC7bT/Vdk+11uaOdJAjwX6qjDHrrbXVTtcRbmp3bInVdkPstn2s2q2hGBGRKKNgFxGJMm4J9hVOF+AQtTu2xGq7IXbbPibtdsUYu4iIBM8tPXYREQlSRAW7MWaZMabGGLPbGPO9Yd4fb4z5Q+D9t40xJeGvMvSCaPe3jDHbjDGbjTGvGGOmOlFnqI3U7mOOu9oYY40xUTFrIph2G2M+H/jO3zfGPBLuGsdCED/nxcaY14wx7wV+1j/jRJ2hZoz5nTGm2Riz9QTvG2PMvwX+XjYbY+aP+kOttRHxAOKBPUAZMA7YBFQdd8zfA/8ReH4d8Aen6w5Tu88HkgPPvx4r7Q4clwasAtYC1U7XHabvuxx4D5gYeJ3ndN1havcK4OuB51VAndN1h6jti4H5wNYTvP8Z4HnAAGcCb4/2MyOpx74Q2G2trbXW9gOPAZcfd8zlwEOB508AFxr375M3Yrutta9Za3sDL9cCRWGucSwE830D/Bi4F/CGs7gxFEy7vwr8ylp7GMBa2xzmGsdCMO22QHrgeQZwMIz1jRlr7Sqg7SSHXA78X+u3Fsg0xkwazWdGUrAXAvuPed0Q+LNhj7HWDgIdQHZYqhs7wbT7WDfh/6+7243YbmPM6cAUa+1fwlnYGAvm+64AKowxa4wxa40xy8JW3dgJpt3/AHzRGNMAPAd8IzylOe5UM2BEkbTn6XA97+On7ARzjNsE3SZjzBeBauC8Ma0oPE7abmNMHHA/8HfhKihMgvm+E/APx3wK/29nbxhjZltr28e4trEUTLuvB35vrf0XY8xZwMOBdvvGvjxHhTzXIqnH3gBMOeZ1ER//VeyDY4wxCfh/XTvZrzhuEEy7McYsAe4ALrPW9oWptrE0UrvTgNnASmNMHf6xx2ei4AJqsD/nT1trB6y1e4Ea/EHvZsG0+ybgjwDW2reAJPxrqUS7oDLgVERSsK8Dyo0xpcaYcfgvjj5z3DHPADcGnl8NvGoDVx9cbMR2B4YkHsAf6tEw3gojtNta22GtzbHWllhrS/BfW7jMWrvemXJDJpif8z/hv2COMSYH/9BMbVirDL1g2r0PuBDAGFOJP9hbwlqlM54BvhSYHXMm0GGtPTSqMzp9xXiYq8M78V89vyPwZ/+I/x80+L/ox4HdwDtAmdM1h6ndLwNNwMbA4xmnaw5Hu487diVRMCsmyO/bAD8DtgFbgOucrjlM7a4C1uCfMbMRWOp0zSFq96PAIWAAf+/8JuAW4JZjvu9fBf5etoTi51x3noqIRJlIGooREZEQULCLiEQZBbuISJRRsIuIRBkFu4hIlFGwi4hEGQW7iEiUUbCLiESZ/wYpzVpT2qGjowAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(0.01, 1.0, 0.01)\n",
    "y = np.log(x)\n",
    "\n",
    "plt.plot(x, y, label=\"log\")\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_entropy_error(y, t):\n",
    "    # np.log() 함수에 0을 입력하면 마이너스 무한대가 되므로 보정한다.\n",
    "    delta = 1e-7\n",
    "    return -np.sum(t * np.log(y + delta))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.510825457099338"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "2.302584092994546"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t = [0, 0, 1, 0, 0, 0, 0, 0, 0, 0]\n",
    "\n",
    "y = [0.1, 0.05, 0.6, 0.0, 0.05, 0.1, 0.0, 0.1, 0.0, 0.0]\n",
    "\n",
    "# 첫 번째 결과\n",
    "display(cross_entropy_error(np.array(y),np.array(t)))\n",
    "\n",
    "y = [0.1, 0.05, 0.1, 0.0, 0.05, 0.1, 0.0, 0.6, 0.0, 0.0]\n",
    "\n",
    "# 두 번째 결과\n",
    "display(cross_entropy_error(np.array(y),np.array(t)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3.3 미니배치 학습\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}