- #1
In the previous video, we gave a
mathematical definition of gradient
- #2
descent. Let's delve deeper, and in this
video, get better intuition about what the
- #3
algorithm is doing, and why the steps of
the gradient descent algorithm might make
- #4
sense. Here's the gradient descent
algorithm that we saw last time. And, just
- #5
to remind you, this parameter, or this
term, alpha, is called the learning rate.
- #6
And it controls how big a step we take
when updating my parameter theta J. And
- #7
this second term here is the derivative
term. And what I want to do in this video
- #8
is give you better intuition about what each of
these two terms is doing and why, when put
- #9
together, this entire update makes sense.
In order to convey these intuitions, what
- #10
I want to do is use a slightly simpler
example where we want to minimize the
- #11
function of just one parameter. So, so we
have a, say we have a cost function J of
- #12
just one parameter, theta one, like we
did, you know, a few videos back. Where
- #13
theta one is a real number, okay? Just so we can have 1D plots, which
- #14
are a little bit simpler to look at. And
let's try to understand why gradient
- #15
descent would do on this function.
[sound]. So, let's say here's my function.
- #16
J of theta one, and so that's my, and
where theta one is a real number. Right,
- #17
now let's say I've initialized gradient
descent with theta one at this location.
- #18
So image that we start off at that point
on my function. What gradient descent will
- #19
do, is it will update. Theta one gets
updated as theta one minus Alpha times DD
- #20
theta one J of theta one, right? and oh an
just as an aside you know this, this
- #21
derivative term, right? If you're
wondering why I changed the notation from
- #22
these partial derivative symbols. If you
don't know what the difference is between
- #23
these partial derivative symbols and the
dd theta don't worry about it. Technically
- #24
in mathematics we call this a partial
derivative, we call this a derivative,
- #25
depending on the number of, of parameters
in the function J, but that's a
- #26
mathematical technicality, so, you know
For the purpose of this lecture, think of
- #27
these partial symbols, and DD theta one as
exactly the same thing. And, don't worry
- #28
about whether there are any differences.
I'm gonna try to use the mathematically
- #29
precise notation. But for our purposes,
these notations are really the same thing.
- #30
So, let's see what this, this equation
will do. And so we're going to compute
- #31
this derivative of, I'm not sure if you've
seen derivatives in calculus before. But
- #32
what a derivative, at this point, does, is
basically saying, you know, let's. Take
- #33
the tangent to that point, like that
straight line, the red line, just,
- #34
just touching this function and
let's look at the slope of this red line. That's
- #35
where the derivative is. It says
what's the slope of the line that is just
- #36
tangent to the function, okay, and the
slope of the line is of course is just
- #37
right, you know just the height divided by
this horizontal thing. Now. This line has
- #38
a positive slope, so it has a positive
derivative. And so, my update to theta is
- #39
going to be, theta one gives the update that
theta one minus alpha times some positive
- #40
number. Okay? Alpha, the learning
rate is always a positive number. And so
- #41
I'm gonna to take theta one, this update
as theta one minus something. So I'm gonna
- #42
end up moving theta one to the left. I'm
gonna decrease theta one and we can see
- #43
this is the right thing to do because I
actually went ahead in this direction you
- #44
know to get me closer to the minimum over
there. So, gradient descent so far seems
- #45
to be doing the right thing. Let's look at
another example. So let's take my same
- #46
function j. Just trying to draw the same
function j of theta one. And now let's say
- #47
I had instead initialized my parameter
over there on the left. So theta one is
- #48
here. I'm gonna add that point on the
surface. Now, my derivative term, d, d
- #49
theta one j of theta one, when evaluated
at this point, gonna look at right. The
- #50
slope of that line. So this derivative
term is a slope of this line. But this