- #1
We've previously defined the cost function
J. In this video I want to tell you about
- #2
an algorithm called gradient descent for
minimizing the cost function J. It turns
- #3
out gradient descent is a more general
algorithm and is used not only in linear
- #4
regression. It's actually used all over
the place in machine learning. And later
- #5
in the class we'll use gradient descent to
minimize other functions as well, not just
- #6
the cost function J, for linear regression.
So in this video, I'm going to
- #7
talk about gradient descent for minimizing
some arbitrary function J. And then in
- #8
later videos, we'll take those algorithm
and apply it specifically to the cost
- #9
function J that we had to find for linear
regression. So here's the problem
- #10
setup. We're going to see that we have
some function J of (theta0, theta1).
- #11
Maybe it's a cost function from linear
regression. Maybe it's some other function
- #12
we want to minimize. And we want
to come up with an algorithm for
- #13
minimizing that as a function of J of
(theta0, theta1). Just as an aside,
- #14
it turns out that gradient descent
actually applies to more general
- #15
functions. So imagine if you have
a function that's a function of
- #16
J of (theta0, theta1, theta2, up to
some theta n). And you want to
- #17
minimize over (theta0 up to theta n)
of this J of (theta0 up to theta n).
- #18
It turns out gradient descent
is an algorithm for solving
- #19
this more general problem, but for the
sake of brevity, for the sake of
- #20
your succinctness of notation, I'm just
going to present only two parameters
- #21
throughout the rest of this video. Here's
the idea for gradient descent. What we're
- #22
going to do is we're going to start off
with some initial guesses for theta0 and
- #23
theta1. Doesn't really matter what they
are, but a common choice would be if we
- #24
set theta0 to 0, and
set theta1 to 0. Just initialize
- #25
them to 0. What we're going to do in
gradient descent is we'll keep changing
- #26
theta0 and theta1 a little bit to
try to reduce J of (theta0, theta1)
- #27
until hopefully we wind up at a minimum or
maybe a local minimum. So, let's see
- #28
see pictures of what gradient descent
does. Let's say I try to minimize this
- #29
function. So notice the axes. This is,
(theta0, theta1) on the horizontal
- #30
axes, and J is a vertical axis. And so the
height of the surface shows J, and we
- #31
want to minimize this function. So, we're
going to start off with (theta0, theta1)
- #32
at some point. So imagine picking some
value for (theta0, theta1), and that
- #33
corresponds to starting at some point on
the surface of this function. Okay? So
- #34
whatever value of (theta0, theta1)
gives you some point here. I did
- #35
initialize them to 0, but
sometimes you initialize it to other
- #36
values as well. Now. I want us to imagine
that this figure shows a hill. Imagine
- #37
this is like a landscape of some grassy
park with two hills like so.
- #38
And I want you to imagine that you are
physically standing at that point on the
- #39
hill on this little red hill in your park.
In gradient descent, what we're
- #40
going to do is spin 360 degrees around and
just look all around us and ask, "If I were
- #41
to take a little baby step in some
direction, and I want to go downhill as
- #42
quickly as possible, what direction do I
take that little baby step in if I want to
- #43
go down, if I sort of want to physically
walk down this hill as rapidly as
- #44
possible?" Turns out that if we're standing
at that point on the hill, you look all
- #45
around, you find that the best direction
to take a little step downhill
- #46
is roughly that direction. Okay. And now
you're at this new point on your hill.
- #47
You're going to, again, look all around, and then
say, "What direction should I step in order
- #48
to take a little baby step downhill?" And
if you do that and take another step, you
- #49
take a step in that direction, and then
you keep going. You know, from this new
- #50
point, you look around, decide what
direction will take you downhill most