4 - 7 - Normal Equation Noninvertibility (Optional) (6 min).srt

1
00:00:00,000 --> 00:00:03,162
在这段视频中我想谈谈正规方程 ( normal equation )
(字幕整理：中国海洋大学 黄海广,haiguang2000@qq.com )

2
00:00:00,000 --> 00:00:03,162
In this video, I want to talk about the normal equation

3
00:00:03,162 --> 00:00:05,212
以及它们的不可逆性

4
00:00:03,162 --> 00:00:05,212
and non-invertibility.

5
00:00:05,212 --> 00:00:07,877
尽管是一种较为深入的概念

6
00:00:05,212 --> 00:00:07,877
This is a somewhat more advanced concept,

7
00:00:07,877 --> 00:00:10,289
但总有人问我有关这方面的问题

8
00:00:07,877 --> 00:00:10,289
but it is something that I've often been asked about.

9
00:00:10,289 --> 00:00:12,711
因此 我想在这里来讨论它

10
00:00:10,289 --> 00:00:12,711
And so I wanted to talk about it here.

11
00:00:12,711 --> 00:00:14,752
由于概念较为深入

12
00:00:12,711 --> 00:00:14,752
But this is a somewhat more advanced concept,

13
00:00:14,752 --> 00:00:17,982
所以对这段可选材料大家放轻松吧

14
00:00:14,752 --> 00:00:17,982
so feel free to consider this optional material

15
00:00:17,982 --> 00:00:22,413
下面

16
00:00:17,982 --> 00:00:22,413
There's a phenomenon that you may run into

17
00:00:22,413 --> 00:00:24,416
举一个比较实用的例子

18
00:00:22,413 --> 00:00:24,416
that's maybe for some of you useful to understand.

19
00:00:24,416 --> 00:00:26,619
这是一个关于正规方程和线性回归的例子

20
00:00:24,416 --> 00:00:26,619
But even if you don't understand it,

21
00:00:26,619 --> 00:00:28,450
即使你没能理解

22
00:00:26,619 --> 00:00:28,450
the normal equation and linear regression,

23
00:00:28,450 --> 00:00:30,539
也没有关系

24
00:00:28,450 --> 00:00:30,539
you should really get that to work okay.

25
00:00:30,539 --> 00:00:33,195
问题如下

26
00:00:30,539 --> 00:00:33,195
Here's the issue:

27
00:00:33,195 --> 00:00:35,691
你或许可能对

28
00:00:33,195 --> 00:00:35,691
For those of you that are maybe somewhat

29
00:00:35,691 --> 00:00:37,876
线性代数比较熟悉

30
00:00:35,691 --> 00:00:37,876
more familar with linear algebra,

31
00:00:37,876 --> 00:00:39,884
有些同学曾经问过我

32
00:00:37,876 --> 00:00:39,884
what some students have asked me is,

33
00:00:39,884 --> 00:00:42,542
当计算

34
00:00:39,884 --> 00:00:42,542
when computing this

35
00:00:42,542 --> 00:00:45,130
θ等于inv(X'X ) X'y （注：X的转置翻译为X'，下同）

36
00:00:42,542 --> 00:00:45,130
theta equals ( X<u>transpose X )<u>inverse X<u>transpose y</u></u></u>

37
00:00:45,130 --> 00:00:49,476
那对于矩阵X'X的结果是不可逆的情况呢?

38
00:00:45,130 --> 00:00:49,476
what if the matrix X<u>transpose X is non-invertible?</u>

39
00:00:49,476 --> 00:00:52,336
如果你懂一点线性代数的知识

40
00:00:49,476 --> 00:00:52,336
So, for those of you that know a bit more linear algebra

41
00:00:52,336 --> 00:00:55,171
你或许会知道

42
00:00:52,336 --> 00:00:55,171
you may know that only some matrices

43
00:00:55,171 --> 00:00:58,598
有些矩阵可逆 而有些矩阵不可逆

44
00:00:55,171 --> 00:00:58,598
are invertible and some matrices do not have an inverse

45
00:00:58,598 --> 00:01:00,540
我们称那些不可逆矩阵为

46
00:00:58,598 --> 00:01:00,540
we call those non-invertible matrices,

47
00:01:00,540 --> 00:01:04,737
奇异或退化矩阵

48
00:01:00,540 --> 00:01:04,737
singular or degenerate matrices.

49
00:01:04,737 --> 00:01:08,893
问题的重点在于X'X的不可逆的问题

50
00:01:04,737 --> 00:01:08,893
The issue or the problem of X<u>tranpose X being non-invertible</u>

51
00:01:08,893 --> 00:01:11,287
很少发生

52
00:01:08,893 --> 00:01:11,287
should happen pretty rarely.

53
00:01:11,287 --> 00:01:16,749
在Octave里 如果你用它来实现θ的计算

54
00:01:11,287 --> 00:01:16,749
And in Octave, if you implement this to compute theta,

55
00:01:16,749 --> 00:01:20,636
你将会得到正解

56
00:01:16,749 --> 00:01:20,636
it turns out that this will actually do the right thing.

57
00:01:20,636 --> 00:01:24,629
在这里我不想赘述

58
00:01:20,636 --> 00:01:24,629
I'm getting a little bit technical now and I don't want to go into details,

59
00:01:24,629 --> 00:01:28,207
在Octave里 有两个函数可以求解矩阵的逆

60
00:01:24,629 --> 00:01:28,207
but Octave has two functions for inverting matrices:

61
00:01:28,207 --> 00:01:32,146
一个被称为pinv ( ) 另一个是inv ( )

62
00:01:28,207 --> 00:01:32,146
One is called pinv(), and the other is called inv().

63
00:01:32,146 --> 00:01:36,089
这两者之间的差异是些许计算过程上的

64
00:01:32,146 --> 00:01:36,089
The differences between these two are somewhat technical.

65
00:01:36,089 --> 00:01:38,107
一个是所谓的伪逆 另一个被称为逆

66
00:01:36,089 --> 00:01:38,107
One's called the pseudo-inverse, one's called the inverse.

67
00:01:38,107 --> 00:01:42,658
使用pinv ( ) 函数可以展现数学上的过程

68
00:01:38,107 --> 00:01:42,658
You can show mathemically so as long as you use the pinv() function,

69
00:01:42,658 --> 00:01:47,145
这将计算出θ的值

70
00:01:42,658 --> 00:01:47,145
then this will actually compute the value of theta that you want,

71
00:01:47,145 --> 00:01:51,227
即便矩阵X'X是不可逆的

72
00:01:47,145 --> 00:01:51,227
even if X<u>transpose X is non-invertible.</u>

73
00:01:51,227 --> 00:01:54,095
在pinv ( ) 和 inv ( ) 之间

74
00:01:51,227 --> 00:01:54,095
The specific details between what is the difference between

75
00:01:54,095 --> 00:01:55,959
又有哪些具体区别 ?

76
00:01:54,095 --> 00:01:55,959
pinv() and what is inv()

77
00:01:55,959 --> 00:01:58,562
其中inv ( ) 引入了先进的数值计算的概念

78
00:01:55,959 --> 00:01:58,562
that is somewhat advanced numerical computing concepts,

79
00:01:58,562 --> 00:02:00,907
我真的不希望讲那些

80
00:01:58,562 --> 00:02:00,907
that I don't really want to get into.

81
00:02:00,907 --> 00:02:02,993
因此 我认为

82
00:02:00,907 --> 00:02:02,993
But I thought in this optional

83
00:02:02,993 --> 00:02:04,672
可以试着给你一点点直观的参考

84
00:02:02,993 --> 00:02:04,672
video I try to give you a little bit of intuition

85
00:02:04,672 --> 00:02:08,823
关于矩阵X'X的不可逆的问题

86
00:02:04,672 --> 00:02:08,823
about what it means that X<u>transpose X to be non-invertible.</u>

87
00:02:08,823 --> 00:02:12,108
如果你懂一点线性代数

88
00:02:08,823 --> 00:02:12,108
For those of you that know a bit more linear algebra

89
00:02:12,108 --> 00:02:13,556
或许你可能会感兴趣

90
00:02:12,108 --> 00:02:13,556
and might be interested.

91
00:02:13,556 --> 00:02:15,948
我不会从数学的角度来证明它

92
00:02:13,556 --> 00:02:15,948
I'm not going to proove this mathematically,

93
00:02:15,948 --> 00:02:18,684
但如果矩阵X'X结果是不可逆的

94
00:02:15,948 --> 00:02:18,684
but if X<u>transpose X is non-invertible,</u>

95
00:02:18,684 --> 00:02:22,596
通常有两种最常见的原因

96
00:02:18,684 --> 00:02:22,596
there are usually two most common causes:

97
00:02:22,596 --> 00:02:26,238
第一个原因是 如果不知何故 在你的学习问题

98
00:02:22,596 --> 00:02:26,238
The first cause is if somehow, in your learning problem,

99
00:02:26,238 --> 00:02:28,461
你有多余的功能

100
00:02:26,238 --> 00:02:28,461
you have redundant features,

101
00:02:28,461 --> 00:02:30,844
例如 在预测住房价格时

102
00:02:28,461 --> 00:02:30,844
concretely, if you try to predict housing prices

103
00:02:30,844 --> 00:02:34,877
如果x1是以英尺为尺寸规格计算的房子

104
00:02:30,844 --> 00:02:34,877
and if x<u>1 is the size of a house in square-feet,</u>

105
00:02:34,877 --> 00:02:37,792
x2是以平方米为尺寸规格计算的房子

106
00:02:34,877 --> 00:02:37,792
and x<u>2 is the size of the house in square-meters,</u>

107
00:02:37,792 --> 00:02:46,071
同时 你也知道1米等于3 28英尺 ( 四舍五入到两位小数 )

108
00:02:37,792 --> 00:02:46,071
then, you know, 1 meter is equal to 3.28 feet, rounded to two decimals,

109
00:02:46,071 --> 00:02:48,947
这样 你的这两个特征值将始终满足约束

110
00:02:46,071 --> 00:02:48,947
and so your two features will always satisfy the constraint

111
00:02:48,947 --> 00:02:55,378
x1等于x2倍的3.28平方

112
00:02:48,947 --> 00:02:55,378
that x<u>1 equals 3(.28)^2 times x<u>2.</u></u>

113
00:02:55,378 --> 00:02:59,107
并且你可以将这过程显示出来 讲到这里 可能 或许对你来说有点难了

114
00:02:55,378 --> 00:02:59,107
And you can show, for those of you - this is somehwat advanced linear algebra now,

115
00:02:59,107 --> 00:03:01,169
但如果你在线性代数上非常熟练

116
00:02:59,107 --> 00:03:01,169
but if you're an expert in linear algebra,

117
00:03:01,169 --> 00:03:05,275
实际上 你可以用这样的一个线性方程 来展示那两个相关联的特征值

118
00:03:01,169 --> 00:03:05,275
you can actually show that if your two features are related via a linear equation like this,

119
00:03:05,275 --> 00:03:09,095
矩阵X'X将是不可逆的

120
00:03:05,275 --> 00:03:09,095
then matrix X<u>transpose X will be non-invertible.</u>

121
00:03:09,095 --> 00:03:13,320
第二个原因是 在你想用大量的特征值

122
00:03:09,095 --> 00:03:13,320
The second thing that can cause X<u>transpose X to be non-invertible</u>

123
00:03:13,320 --> 00:03:17,043
尝试实践你的学习算法的时候

124
00:03:13,320 --> 00:03:17,043
is if you're trying to run a learning algorithm

125
00:03:17,043 --> 00:03:18,850
可能会导致矩阵X'X的结果是不可逆的

126
00:03:17,043 --> 00:03:18,850
with a <i>lot</i> of a features.

127
00:03:18,850 --> 00:03:23,035
具体地说 在m小于或等于n的时候

128
00:03:18,850 --> 00:03:23,035
Concretely, if m is less than or equal to n.

129
00:03:23,035 --> 00:03:27,723
例如 有m等于10个的训练实例

130
00:03:23,035 --> 00:03:27,723
For example, if you imagine that you have m equals 10 training examples

131
00:03:27,723 --> 00:03:31,192
也有n等于100的特征数量

132
00:03:27,723 --> 00:03:31,192
and that you have n equals 100 features, then you're trying

133
00:03:31,192 --> 00:03:36,829
要找到适合的 ( n +1 ) 维参数矢量θ 这是第

134
00:03:31,192 --> 00:03:36,829
to fit a parameter vector theta, which is (n+1)-dimensional,

135
00:03:36,829 --> 00:03:39,308
这将会变成一个101维的矢量

136
00:03:36,829 --> 00:03:39,308
so it's a 101-dimensional

137
00:03:39,308 --> 00:03:43,602
尝试从10个训练实例中找到满足101个参数的值

138
00:03:39,308 --> 00:03:43,602
you're trying to fit a 101 parameters from just 10 training examples.

139
00:03:43,602 --> 00:03:46,899
这工作可能会让你花上一阵子时间

140
00:03:43,602 --> 00:03:46,899
And this turns out to sometimes work,

141
00:03:46,899 --> 00:03:49,078
但这并不总是一个好主意

142
00:03:46,899 --> 00:03:49,078
but to not always be a good idea.

143
00:03:49,078 --> 00:03:52,212
因为 正如我们所看到 你只有10个例子 以适应这100或101个参数

144
00:03:49,078 --> 00:03:52,212
Because, as we see later, you might not have enough data

145
00:03:52,212 --> 00:03:58,432
数据还是有些少

146
00:03:52,212 --> 00:03:58,432
if you only have 10 examples to fit 100 or 101 parameters.

147
00:03:58,432 --> 00:04:01,924
稍后我们将看到

148
00:03:58,432 --> 00:04:01,924
We'll see later in this course, why this might be too little data

149
00:04:01,924 --> 00:04:04,418
如何使用小数据样本以得到这100或101个参数

150
00:04:01,924 --> 00:04:04,418
to fit this many parameters.

151
00:04:04,418 --> 00:04:07,544
通常 我们会使用一种叫做正则化的线性代数方法

152
00:04:04,418 --> 00:04:07,544
But commonly, what we do then if m is less than n,

153
00:04:07,544 --> 00:04:12,513
通过删除某些特征或者是使用某些技术

154
00:04:07,544 --> 00:04:12,513
is to see if we can either delete some features or to use a technique

155
00:04:12,513 --> 00:04:14,689
来解决当m比n小的时候的问题

156
00:04:12,513 --> 00:04:14,689
called regularization,

157
00:04:14,689 --> 00:04:17,477
这也是在本节课后面要讲到的内容

158
00:04:14,689 --> 00:04:17,477
which is something that we will talk about a bit later in this course as well,

159
00:04:17,477 --> 00:04:21,905
即使你有一个相对较小的训练集

160
00:04:17,477 --> 00:04:21,905
that will kind of let you fit a <i>lot</i> of parameters using a <i>lot</i> of features

161
00:04:21,905 --> 00:04:24,117
也可使用很多的特征来找到很多合适的参数

162
00:04:21,905 --> 00:04:24,117
even if you have a relatively small training set.

163
00:04:24,117 --> 00:04:27,698
有关正规化的内容将是本节之后课程的话题

164
00:04:24,117 --> 00:04:27,698
But this regularization will be a later topic in this course.

165
00:04:27,698 --> 00:04:32,628
总之当你发现的矩阵X'X的结果是奇异矩阵

166
00:04:27,698 --> 00:04:32,628
But to summarize, if ever you find that X<u>transpose X is singular</u>

167
00:04:32,628 --> 00:04:35,877
或者找到的其它矩阵是不可逆的

168
00:04:32,628 --> 00:04:35,877
or alternatively find is non-invertible,

169
00:04:35,877 --> 00:04:38,380
我会建议你这么做

170
00:04:35,877 --> 00:04:38,380
what I would recommend you do is

171
00:04:38,380 --> 00:04:42,016
首先 看特征值里是否有一些多余的特征

172
00:04:38,380 --> 00:04:42,016
first: look at your features and see if you have redundant features

173
00:04:42,016 --> 00:04:45,304
像这些x1和x2是线性相关的

174
00:04:42,016 --> 00:04:45,304
like these x<u>1 and x<u>2 being linearly dependent,</u></u>

175
00:04:45,304 --> 00:04:48,017
或像这样 互为线性函数

176
00:04:45,304 --> 00:04:48,017
or being a linear function of each other, like so

177
00:04:48,017 --> 00:04:49,841
同时 当有一些多余的特征时

178
00:04:48,017 --> 00:04:49,841
and if you do have redundant features and

179
00:04:49,841 --> 00:04:51,493
可以删除这两个重复特征里的其中一个

180
00:04:49,841 --> 00:04:51,493
if you just delete one of these features -

181
00:04:51,493 --> 00:04:53,724
无须两个特征同时保留

182
00:04:51,493 --> 00:04:53,724
you really don't need both of these features,

183
00:04:53,724 --> 00:04:55,601
所以 发现多余的特征删除二者其一

184
00:04:53,724 --> 00:04:55,601
so if you just delete one of these features

185
00:04:55,601 --> 00:04:58,586
将解决不可逆性的问题

186
00:04:55,601 --> 00:04:58,586
that will solve your non-invertibility problem

187
00:04:58,586 --> 00:05:02,655
因此 首先应该通过观察所有特征检查是否有多余的特征

188
00:04:58,586 --> 00:05:02,655
and, so first think through my features and check if any are redundant

189
00:05:02,655 --> 00:05:05,481
如果有多余的就删除掉

190
00:05:02,655 --> 00:05:05,481
and if so, then, you know, keep deleting the redundant features

191
00:05:05,481 --> 00:05:07,659
直到他们不再是多余的为止

192
00:05:05,481 --> 00:05:07,659
until they are no longer redundant.

193
00:05:07,659 --> 00:05:09,799
如果特征里没有多余的

194
00:05:07,659 --> 00:05:09,799
And if your features are non redundant,

195
00:05:09,799 --> 00:05:11,939
我会检查是否有过多的特征

196
00:05:09,799 --> 00:05:11,939
I would check if I might have too many features,

197
00:05:11,939 --> 00:05:13,638
如果特征数量实在太多

198
00:05:11,939 --> 00:05:13,638
and if that's the case I would either

199
00:05:13,638 --> 00:05:16,140
我会删除些 用较少的特征来反映尽可能多内容

200
00:05:13,638 --> 00:05:16,140
delete some features if I can bare to use fewer features,

201
00:05:16,140 --> 00:05:20,708
否则我会考虑使用正规化方法

202
00:05:16,140 --> 00:05:20,708
or else I would consider using regularization,

203
00:05:20,708 --> 00:05:22,821
这也是我们将要谈论的话题

204
00:05:20,708 --> 00:05:22,821
which is this topic that we will talk about later.

205
00:05:22,821 --> 00:05:27,877
同时 这也是有关标准方程的内容

206
00:05:22,821 --> 00:05:27,877
So, that's it for the normal equation and what it means

207
00:05:27,877 --> 00:05:31,885
如果矩阵X'X是不可逆的

208
00:05:27,877 --> 00:05:31,885
if the matrix X<u>transpose X is non-invertible.</u>

209
00:05:31,885 --> 00:05:35,710
通常来说 不会出现这种情况

210
00:05:31,885 --> 00:05:35,710
But this is a problem that hopefully you run into pretty rarely.

211
00:05:35,710 --> 00:05:40,554
如果在Octave里

212
00:05:35,710 --> 00:05:40,554
And if you just implement it in Octave using the pinv() function

213
00:05:40,554 --> 00:05:42,853
可以用伪逆函数pinv ( ) 来实现

214
00:05:40,554 --> 00:05:42,853
which is called the pseudo-inverse function

215
00:05:42,853 --> 00:05:46,700
这种使用不同的线性代数库的方法被称为伪逆

216
00:05:42,853 --> 00:05:46,700
so you use a different linear algebra library, that is called pseudo-inverse

217
00:05:46,700 --> 00:05:50,071
即使X'X的结果是不可逆的

218
00:05:46,700 --> 00:05:50,071
but that implementation should just do the right thing

219
00:05:50,071 --> 00:05:52,582
但算法执行的流程是正确的

220
00:05:50,071 --> 00:05:52,582
even if X<u>transpose X is non-invertible</u>

221
00:05:52,582 --> 00:05:55,198
总之 出现不可逆矩阵的情况极少发生

222
00:05:52,582 --> 00:05:55,198
which should happen pretty rarily anyway

223
00:05:55,198 --> 99:59:59,000
所以在大多数实现线性回归中 出现不可逆的问题不应该过多的关注

224
00:05:55,198 --> 99:59:59,000
so this should not be a problem for most implementations of linear regression.