update 5-5损失函数losses

Author: Amberlan1001

五、TensorFlow的中阶API/5-5损失函数losses.ipynb

@@ -1,5 +1,190 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 5-5损失函数losses\n",
+    "\n",
+    "一般来说，监督学习的目标函数由损失函数和正则化项组成。（Objective = Loss + Regularization）\n",
+    "\n",
+    "对于keras模型，目标函数中的正则化项一般在各层中指定，例如使用Dense的 kernel_regularizer 和 bias_regularizer等参数指定权重使用l1或者l2正则化项，此外还可以用kernel_constraint 和 bias_constraint等参数约束权重的取值范围，这也是一种正则化手段。\n",
+    "\n",
+    "损失函数在模型编译时候指定。对于回归模型，通常使用的损失函数是平方损失函数 mean_squared_error。\n",
+    "\n",
+    "对于二分类模型，通常使用的是二元交叉熵损失函数 binary_crossentropy。\n",
+    "\n",
+    "对于多分类模型，如果label是one-hot编码的，则使用类别交叉熵损失函数 categorical_crossentropy。如果label是类别序号编码的，则需要使用稀疏类别交叉熵损失函数 sparse_categorical_crossentropy。\n",
+    "\n",
+    "如果有需要，也可以自定义损失函数，自定义损失函数需要接收两个张量y_true,y_pred作为输入参数，并输出一个标量作为损失函数值。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import tensorflow as tf\n",
+    "from tensorflow.keras import layers,models,losses,regularizers,constraints"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 一、损失函数和正则化项"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Model: \"sequential\"\n",
+      "_________________________________________________________________\n",
+      "Layer (type)                 Output Shape              Param #   \n",
+      "=================================================================\n",
+      "dense (Dense)                (None, 64)                4160      \n",
+      "_________________________________________________________________\n",
+      "dense_1 (Dense)              (None, 10)                650       \n",
+      "=================================================================\n",
+      "Total params: 4,810\n",
+      "Trainable params: 4,810\n",
+      "Non-trainable params: 0\n",
+      "_________________________________________________________________\n"
+     ]
+    }
+   ],
+   "source": [
+    "tf.keras.backend.clear_session()\n",
+    "\n",
+    "model = models.Sequential()\n",
+    "model.add(layers.Dense(64, input_dim=64,\n",
+    "                kernel_regularizer=regularizers.l2(0.01), \n",
+    "                activity_regularizer=regularizers.l1(0.01),\n",
+    "                kernel_constraint = constraints.MaxNorm(max_value=2, axis=0))) \n",
+    "model.add(layers.Dense(10,\n",
+    "        kernel_regularizer=regularizers.l1_l2(0.01,0.01),activation = \"sigmoid\"))\n",
+    "model.compile(optimizer = \"rmsprop\",\n",
+    "        loss = \"sparse_categorical_crossentropy\",metrics = [\"AUC\"])\n",
+    "model.summary()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 二、内置损失函数"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "内置的损失函数一般有类的实现和函数的实现两种形式。\n",
+    "\n",
+    "如：CategoricalCrossentropy 和 categorical_crossentropy 都是类别交叉熵损失函数，前者是类的实现形式，后者是函数的实现形式。\n",
+    "\n",
+    "常用的一些内置损失函数说明如下。\n",
+    "\n",
+    "* mean_squared_error（平方差误差损失，用于回归，简写为 mse, 类实现形式为 MeanSquaredError 和 MSE）\n",
+    "\n",
+    "* mean_absolute_error (绝对值误差损失，用于回归，简写为 mae, 类实现形式为 MeanAbsoluteError 和 MAE)\n",
+    "\n",
+    "* mean_absolute_percentage_error (平均百分比误差损失，用于回归，简写为 mape, 类实现形式为 MeanAbsolutePercentageError 和 MAPE)\n",
+    "\n",
+    "* Huber(Huber损失，只有类实现形式，用于回归，介于mse和mae之间，对异常值比较鲁棒，相对mse有一定的优势)\n",
+    "\n",
+    "* binary_crossentropy(二元交叉熵，用于二分类，类实现形式为 BinaryCrossentropy)\n",
+    "\n",
+    "* categorical_crossentropy(类别交叉熵，用于多分类，要求label为onehot编码，类实现形式为 CategoricalCrossentropy)\n",
+    "\n",
+    "* sparse_categorical_crossentropy(稀疏类别交叉熵，用于多分类，要求label为序号编码形式，类实现形式为 SparseCategoricalCrossentropy)\n",
+    "\n",
+    "* hinge(合页损失函数，用于二分类，最著名的应用是作为支持向量机SVM的损失函数，类实现形式为 Hinge)\n",
+    "\n",
+    "* kld(相对熵损失，也叫KL散度，常用于最大期望算法EM的损失函数，两个概率分布差异的一种信息度量。类实现形式为 KLDivergence 或 KLD)\n",
+    "\n",
+    "* cosine_similarity(余弦相似度，可用于多分类，类实现形式为 CosineSimilarity)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 三、自定义损失函数"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "自定义损失函数接收两个张量y_true,y_pred作为输入参数，并输出一个标量作为损失函数值。\n",
+    "\n",
+    "也可以对tf.keras.losses.Loss进行子类化，重写call方法实现损失的计算逻辑，从而得到损失函数的类的实现。\n",
+    "\n",
+    "下面是一个Focal Loss的自定义实现示范。Focal Loss是一种对binary_crossentropy的改进损失函数形式。\n",
+    "\n",
+    "在类别不平衡和存在难以训练样本的情形下相对于二元交叉熵能够取得更好的效果。\n",
+    "\n",
+    "详见《如何评价Kaiming的Focal Loss for Dense Object Detection？》\n",
+    "\n",
+    "https://www.zhihu.com/question/63581984"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def focal_loss(gamma=2., alpha=0.25):\n",
+    "    \n",
+    "    def focal_loss_fixed(y_true, y_pred):\n",
+    "        pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n",
+    "        pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n",
+    "        loss = -tf.sum(alpha * tf.pow(1. - pt_1, gamma) * tf.log(1e-07+pt_1)) \\\n",
+    "           -tf.sum((1-alpha) * tf.pow( pt_0, gamma) * tf.log(1. - pt_0 + 1e-07))\n",
+    "        return loss\n",
+    "    return focal_loss_fixed"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class FocalLoss(losses.Loss):\n",
+    "    \n",
+    "    def __init__(self,gamma=2.0,alpha=0.25):\n",
+    "        self.gamma = gamma\n",
+    "        self.alpha = alpha\n",
+    "\n",
+    "    def call(self,y_true,y_pred):\n",
+    "        \n",
+    "        pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n",
+    "        pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n",
+    "        loss = -tf.sum(self.alpha * tf.pow(1. - pt_1, self.gamma) * tf.log(1e-07+pt_1)) \\\n",
+    "           -tf.sum((1-self.alpha) * tf.pow( pt_0, self.gamma) * tf.log(1. - pt_0 + 1e-07))\n",
+    "        return loss"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,