🎮 CS234 – Reinforcement Learning

Stanford CS234 Reinforcement Learning I Introduction to Reinforcement Learning I 2024 I Lecture 1

Source: https://www.youtube.com/watch?v=WsvFL-LjA6U

---

Transcript

[00:00:05] hi everyone we're going to go ahead and
[00:00:06] hi everyone we're going to go ahead and get started I'm Emma Brill um I'm
[00:00:08] get started I'm Emma Brill um I'm delighted to welcome you to
[00:00:10] delighted to welcome you to reinforcement learning
[00:00:11] reinforcement learning cs234 um this is a brief overview of the
[00:00:14] cs234 um this is a brief overview of the class uh and what we're going to be
[00:00:16] class uh and what we're going to be covering today and just want to start
[00:00:20] covering today and just want to start that probably everyone's heard of
[00:00:21] that probably everyone's heard of reinforcement learning these days and
[00:00:23] reinforcement learning these days and that wasn't true about 10 15 years ago
[00:00:25] that wasn't true about 10 15 years ago but you can describe what is happening
[00:00:27] but you can describe what is happening in reinforcement learning by a pretty
[00:00:30] in reinforcement learning by a pretty simple statement which is the idea of an
[00:00:32] simple statement which is the idea of an automated agent learning through
[00:00:34] automated agent learning through experience to make good decisions now
[00:00:38] experience to make good decisions now that's a pretty simple statement to say
[00:00:41] that's a pretty simple statement to say um it sort of encapsulates a lot of what
[00:00:42] um it sort of encapsulates a lot of what me and my lab and many many others have
[00:00:44] me and my lab and many many others have been trying to work on for the last 10
[00:00:46] been trying to work on for the last 10 to 15 years um but it's sort of
[00:00:48] to 15 years um but it's sort of deceptively simple because it involves a
[00:00:50] deceptively simple because it involves a lot of different really challenging and
[00:00:52] lot of different really challenging and important
[00:00:54] things so the first is that any sort of
[00:00:57] things so the first is that any sort of General agenda to try to achieve General
[00:01:00] General agenda to try to achieve General artificial intelligence has to include
[00:01:01] artificial intelligence has to include the ability to make decisions there's
[00:01:04] the ability to make decisions there's been absolutely enormous progress in
[00:01:06] been absolutely enormous progress in what we would call sort of perceptual um
[00:01:08] what we would call sort of perceptual um machine learning things like being able
[00:01:10] machine learning things like being able to perceive faces or cats or like
[00:01:13] to perceive faces or cats or like identify cars and we call that often uh
[00:01:16] identify cars and we call that often uh perceptual machine learning because it
[00:01:18] perceptual machine learning because it focuses on trying to say identify
[00:01:20] focuses on trying to say identify something but of course in reality what
[00:01:22] something but of course in reality what we're all trying to do all the time is
[00:01:24] we're all trying to do all the time is also to make decisions based on our
[00:01:26] also to make decisions based on our perception and based on our information
[00:01:28] perception and based on our information we're receiving and and so it's critical
[00:01:30] we're receiving and and so it's critical if we think about what it means to be
[00:01:32] if we think about what it means to be intelligent to consider how to make
[00:01:34] intelligent to consider how to make decisions and not just any decisions but
[00:01:37] decisions and not just any decisions but what it means to have good
[00:01:39] what it means to have good decisions this sort of question over how
[00:01:42] decisions this sort of question over how can we learn to make decisions
[00:01:43] can we learn to make decisions particularly faced uh by uncertainty and
[00:01:46] particularly faced uh by uncertainty and limited data has been a central question
[00:01:49] limited data has been a central question that people have been thinking about at
[00:01:50] that people have been thinking about at least since the
[00:01:51] least since the 1950s um particularly pioneered by the
[00:01:54] 1950s um particularly pioneered by the ideas of Richard bman and we'll hear a
[00:01:57] ideas of Richard bman and we'll hear a lot more about belman's equation which
[00:01:58] lot more about belman's equation which many of you might have seen before for
[00:02:00] many of you might have seen before for later even um in this lecture or next
[00:02:02] later even um in this lecture or next lecture so there's one sort of argument
[00:02:05] lecture so there's one sort of argument for studying reinforcement learning um
[00:02:07] for studying reinforcement learning um which is it's an essential part of
[00:02:08] which is it's an essential part of intelligence it has to be part of a
[00:02:10] intelligence it has to be part of a general agenda of artificial
[00:02:12] general agenda of artificial intelligence and so we should study it
[00:02:14] intelligence and so we should study it to try to understand what it means to be
[00:02:16] to try to understand what it means to be intelligent and that certainly for me is
[00:02:18] intelligent and that certainly for me is one of the really big motivations is to
[00:02:20] one of the really big motivations is to I think there's just a lot of
[00:02:21] I think there's just a lot of fundamental questions about what is the
[00:02:23] fundamental questions about what is the data needed to learn to make good
[00:02:25] data needed to learn to make good decisions but there's another really
[00:02:27] decisions but there's another really good motivation to study reinforcement
[00:02:29] good motivation to study reinforcement learning which is but it's practical and
[00:02:30] learning which is but it's practical and it allows us to solve problems we'd like
[00:02:32] it allows us to solve problems we'd like to
[00:02:33] to solve so in particular over the last
[00:02:36] solve so in particular over the last roughly decade there started to be a lot
[00:02:38] roughly decade there started to be a lot of really impressive successes of using
[00:02:40] of really impressive successes of using reinforcement learning to tackle
[00:02:42] reinforcement learning to tackle problems or to get unprecedented
[00:02:44] problems or to get unprecedented performance in a lot of really important
[00:02:47] performance in a lot of really important domains so the first one is the board
[00:02:49] domains so the first one is the board game go so who here plays
[00:02:52] game go so who here plays go okay a few people maybe maybe not you
[00:02:55] go okay a few people maybe maybe not you can talk to the people that raise their
[00:02:56] can talk to the people that raise their hands so it's an incredibly popular
[00:02:58] hands so it's an incredibly popular board game it's also an incredi
[00:03:00] board game it's also an incredi hardboard game it's far harder than
[00:03:02] hardboard game it's far harder than chess um and it was considered a really
[00:03:04] chess um and it was considered a really long outstanding question of artificial
[00:03:06] long outstanding question of artificial intelligence but roughly like I guess
[00:03:09] intelligence but roughly like I guess about eight years ago now eight to nine
[00:03:11] about eight years ago now eight to nine years ago um there was a team at Deep
[00:03:13] years ago um there was a team at Deep Mind which was still a fairly small
[00:03:14] Mind which was still a fairly small organization at that point that thought
[00:03:17] organization at that point that thought that they could make significant Headway
[00:03:19] that they could make significant Headway at teaching AI agents to be able to play
[00:03:22] at teaching AI agents to be able to play go and the idea in this case is that
[00:03:25] go and the idea in this case is that we're going to combine between the ideas
[00:03:27] we're going to combine between the ideas of reinforcement learning and Monte
[00:03:29] of reinforcement learning and Monte Carlo research which is something we're
[00:03:31] Carlo research which is something we're going to hear about later in this class
[00:03:33] going to hear about later in this class um to create a system that played go
[00:03:35] um to create a system that played go better than any humans in the world and
[00:03:38] better than any humans in the world and so there's even a movie now about sort
[00:03:40] so there's even a movie now about sort of one of the seminal um games in that
[00:03:43] of one of the seminal um games in that sort of endeavor and how humans felt
[00:03:45] sort of endeavor and how humans felt about that and and how the creators of
[00:03:47] about that and and how the creators of the AI systems felt about that but this
[00:03:50] the AI systems felt about that but this uh feat was achieved far earlier than
[00:03:52] uh feat was achieved far earlier than what people expected and one of the key
[00:03:54] what people expected and one of the key reasons for that was the was using
[00:03:56] reasons for that was the was using reinforcement learning
[00:04:00] another really interesting place that
[00:04:01] another really interesting place that we've seen progress of um using
[00:04:03] we've seen progress of um using reinforcement learning to tackle
[00:04:04] reinforcement learning to tackle incredible challenges is in the idea of
[00:04:06] incredible challenges is in the idea of Fusion science fusion is a potential
[00:04:10] Fusion science fusion is a potential approach for trying to tackle the huge
[00:04:11] approach for trying to tackle the huge energy issues that we have and trying to
[00:04:13] energy issues that we have and trying to transition to more sustainable options
[00:04:15] transition to more sustainable options for that and one of the challenges here
[00:04:17] for that and one of the challenges here and I'm not a fusion um expert is to
[00:04:20] and I'm not a fusion um expert is to manipulate and sort of control um things
[00:04:23] manipulate and sort of control um things within um a vessel and so what the
[00:04:26] within um a vessel and so what the reinforcement learning question in this
[00:04:28] reinforcement learning question in this case is how do you command the
[00:04:30] case is how do you command the controllers the coil controllers in
[00:04:32] controllers the coil controllers in order to manipulate this into different
[00:04:34] order to manipulate this into different types of shapes and so this was a nature
[00:04:36] types of shapes and so this was a nature paper from two years ago where they
[00:04:38] paper from two years ago where they showed you could use deep reinforcement
[00:04:39] showed you could use deep reinforcement learning techniques to accomplish this
[00:04:42] learning techniques to accomplish this in a way that was far more flexible than
[00:04:43] in a way that was far more flexible than had previously been
[00:04:46] had previously been imagined one of my favorite examples of
[00:04:48] imagined one of my favorite examples of the applications of reinforcement
[00:04:49] the applications of reinforcement learning um comes from a pretty recent
[00:04:51] learning um comes from a pretty recent important case which is covid um testing
[00:04:55] important case which is covid um testing so this was a system that was deployed
[00:04:57] so this was a system that was deployed in Greece they had limited resources and
[00:04:59] in Greece they had limited resources and they were trying to understand who you
[00:05:01] they were trying to understand who you should test in order to help control the
[00:05:03] should test in order to help control the epidemic because as many of you may know
[00:05:05] epidemic because as many of you may know there's a lot of sort of free movement
[00:05:07] there's a lot of sort of free movement within Europe um and there was a lot of
[00:05:09] within Europe um and there was a lot of Transitions and they were trying to
[00:05:10] Transitions and they were trying to think about how to leverage their
[00:05:11] think about how to leverage their resources in a data- driven way because
[00:05:14] resources in a data- driven way because of course the epidemic was changing um
[00:05:16] of course the epidemic was changing um and so this was a beautiful paper by a
[00:05:18] and so this was a beautiful paper by a Stanford Graduate um HMA Basi and her
[00:05:21] Stanford Graduate um HMA Basi and her colleague she's a professor over at Penn
[00:05:22] colleague she's a professor over at Penn now that use reinforcement learning to
[00:05:25] now that use reinforcement learning to really quickly do this and it was
[00:05:27] really quickly do this and it was deployed so Greece used this um for for
[00:05:30] deployed so Greece used this um for for their um testing at the
[00:05:32] their um testing at the border but perhaps the most famous
[00:05:34] border but perhaps the most famous example recently is chat
[00:05:36] example recently is chat GPT so I think that um as many of you
[00:05:39] GPT so I think that um as many of you might know natural language processing
[00:05:41] might know natural language processing has had incredible successes over the
[00:05:43] has had incredible successes over the last decade and there worlds a lot of
[00:05:46] last decade and there worlds a lot of work trying to use Transformers um to
[00:05:48] work trying to use Transformers um to make really really capable natural
[00:05:50] make really really capable natural language systems but up to around you
[00:05:53] language systems but up to around you know I guess like a year and a half ago
[00:05:56] know I guess like a year and a half ago um most of that work was not known to
[00:05:57] um most of that work was not known to the broader public so even even though
[00:05:59] the broader public so even even though we were getting these amazing advances
[00:06:01] we were getting these amazing advances in natural language processing it wasn't
[00:06:03] in natural language processing it wasn't at the state yet where like everybody
[00:06:04] at the state yet where like everybody was using it and so the key idea of chat
[00:06:08] was using it and so the key idea of chat GPT was to use reinforcement learning to
[00:06:10] GPT was to use reinforcement learning to create vastly more capable
[00:06:13] create vastly more capable systems and I like to talk about chat
[00:06:15] systems and I like to talk about chat GPT not because it's perhaps the most
[00:06:17] GPT not because it's perhaps the most well-known success for reinforcement
[00:06:18] well-known success for reinforcement learning but also because it exhibits a
[00:06:21] learning but also because it exhibits a lot of the different technical
[00:06:22] lot of the different technical challenges and questions that we're
[00:06:23] challenges and questions that we're going to be covering in this class so
[00:06:25] going to be covering in this class so let's just walk through sort of how at a
[00:06:28] let's just walk through sort of how at a very high level sort of this from chat
[00:06:30] very high level sort of this from chat GPT of how the chat GPT system works in
[00:06:33] GPT of how the chat GPT system works in terms of training so the first thing it
[00:06:35] terms of training so the first thing it does is it does what we would probably
[00:06:37] does is it does what we would probably call Behavior
[00:06:40] cloning or imitation learning and we'll
[00:06:43] cloning or imitation learning and we'll be covering that in this
[00:06:45] be covering that in this class and we'll be talking more about it
[00:06:48] class and we'll be talking more about it even in this lecture so what did it do
[00:06:50] even in this lecture so what did it do so again just to remind I suspect
[00:06:52] so again just to remind I suspect everybody in this class probably uses
[00:06:54] everybody in this class probably uses chat gbt probably multiple times a day
[00:06:56] chat gbt probably multiple times a day or Claude or gemini or one of the other
[00:06:59] or Claude or gemini or one of the other um large language model systems but just
[00:07:01] um large language model systems but just in case you have not um the idea in this
[00:07:04] in case you have not um the idea in this case is you might have some sort of
[00:07:05] case is you might have some sort of prompt or task you want your language
[00:07:06] prompt or task you want your language system to do like explain reinforcement
[00:07:09] system to do like explain reinforcement learning to a
[00:07:10] learning to a six-year-old and then someone gives a
[00:07:13] six-year-old and then someone gives a response like we give treats and
[00:07:15] response like we give treats and punishments to teach Etc you can think
[00:07:17] punishments to teach Etc you can think you can try this out with chat GPT and
[00:07:19] you can try this out with chat GPT and see how well you think it explains it um
[00:07:22] see how well you think it explains it um and then what that was treated as is
[00:07:23] and then what that was treated as is sort of a direct supervised learning
[00:07:25] sort of a direct supervised learning problem so just trying to take that
[00:07:27] problem so just trying to take that input and then to produce that output
[00:07:30] input and then to produce that output and we will call that also imitation
[00:07:32] and we will call that also imitation learning or behavior cloning in this
[00:07:33] learning or behavior cloning in this class and we'll talk about why so that
[00:07:36] class and we'll talk about why so that was the first step and this is sort of
[00:07:37] was the first step and this is sort of what people have been doing in uh
[00:07:39] what people have been doing in uh natural language processing and the
[00:07:41] natural language processing and the systems were good but they weren't that
[00:07:42] systems were good but they weren't that good so the next idea was to try to
[00:07:45] good so the next idea was to try to explicitly think about utility or
[00:07:47] explicitly think about utility or rewards like how good were these um
[00:07:50] rewards like how good were these um particular labels or these particular
[00:07:52] particular labels or these particular outputs so here we're going to actually
[00:07:54] outputs so here we're going to actually build a model we're going to build a
[00:07:56] build a model we're going to build a model of a reward
[00:07:59] model of a reward which relates to model-based
[00:08:01] which relates to model-based reinforcement
[00:08:03] reinforcement learning and the way we're going to do
[00:08:05] learning and the way we're going to do this is um or the way they did this is
[00:08:06] this is um or the way they did this is we collect preference data we ask people
[00:08:08] we collect preference data we ask people to compare or rank ACR across different
[00:08:11] to compare or rank ACR across different forms of outputs and then we use that to
[00:08:13] forms of outputs and then we use that to learn a preference model and we're going
[00:08:15] learn a preference model and we're going to cover that in this class that's going
[00:08:17] to cover that in this class that's going to be one of the differences to this
[00:08:18] to be one of the differences to this class compared to a couple years ago
[00:08:20] class compared to a couple years ago that um I think preference-based reward
[00:08:22] that um I think preference-based reward signals are really important and very
[00:08:24] signals are really important and very powerful and so we're going to be
[00:08:25] powerful and so we're going to be covering that in this class this term so
[00:08:28] covering that in this class this term so in this case they would learn a reward
[00:08:30] in this case they would learn a reward and again don't worry if you haven't if
[00:08:32] and again don't worry if you haven't if you're not familiar with what rewards
[00:08:33] you're not familiar with what rewards are and stuff we'll go through all of
[00:08:34] are and stuff we'll go through all of that I just want to give you a high
[00:08:35] that I just want to give you a high level sort of sense of how chat GPT is
[00:08:37] level sort of sense of how chat GPT is related to some of the things we're
[00:08:38] related to some of the things we're going to cover in the class so they
[00:08:40] going to cover in the class so they learned a reward signal and then they
[00:08:42] learned a reward signal and then they did reinforcement learning using that
[00:08:44] did reinforcement learning using that learned reward system signal so now
[00:08:47] learned reward system signal so now they're going to do reinforcement
[00:08:51] learning and this is called
[00:08:55] learning and this is called rhf because it is reinforcement learning
[00:08:57] rhf because it is reinforcement learning from Human feedback and I'll just note
[00:09:00] from Human feedback and I'll just note here that this was not the first time
[00:09:01] here that this was not the first time this idea was um uh introduced it had
[00:09:04] this idea was um uh introduced it had been introduced maybe about four to 5
[00:09:07] been introduced maybe about four to 5 years before this for sort of Robotics
[00:09:09] years before this for sort of Robotics simulated robotics tasks um but chat GPT
[00:09:13] simulated robotics tasks um but chat GPT demonstrated that this really made a
[00:09:15] demonstrated that this really made a huge difference in
[00:09:16] huge difference in performance and so I think that's a it's
[00:09:19] performance and so I think that's a it's a really nice example of the types of
[00:09:20] a really nice example of the types of ideas that we're going to be covering as
[00:09:22] ideas that we're going to be covering as well asort of the incredible successes
[00:09:23] well asort of the incredible successes that are
[00:09:26] possible now for even before chat GPT
[00:09:29] possible now for even before chat GPT came along there was starting to be a
[00:09:31] came along there was starting to be a huge interest in reinforcement
[00:09:33] huge interest in reinforcement learning so some of you we have an
[00:09:35] learning so some of you we have an optional textbook for the class um it's
[00:09:36] optional textbook for the class um it's by Sutton Narto Richard Sutton is um
[00:09:40] by Sutton Narto Richard Sutton is um from Canada and is one of the sort of
[00:09:42] from Canada and is one of the sort of founders of the field and when I started
[00:09:44] founders of the field and when I started in reinforcement learning and I would go
[00:09:46] in reinforcement learning and I would go give my talks and conferences it used to
[00:09:47] give my talks and conferences it used to be like me and Rich and 30 other people
[00:09:50] be like me and Rich and 30 other people nobody cared about reinforcement
[00:09:51] nobody cared about reinforcement learning um I mean except for you know a
[00:09:53] learning um I mean except for you know a few of us did because we thought it was
[00:09:54] few of us did because we thought it was really amazing but as you can see like
[00:09:56] really amazing but as you can see like sort of through like the 2000s which is
[00:09:59] sort of through like the 2000s which is when I I was getting my training around
[00:10:00] when I I was getting my training around here um there just weren't that many
[00:10:03] here um there just weren't that many papers and the community wasn't nearly
[00:10:04] papers and the community wasn't nearly as large but this nice paper by Peter
[00:10:07] as large but this nice paper by Peter Henderson the the y- axis's uh
[00:10:11] Henderson the the y- axis's uh papers shows that there's been this sort
[00:10:13] papers shows that there's been this sort of enormous increase in interest and I
[00:10:16] of enormous increase in interest and I think a lot of this was really due to
[00:10:18] think a lot of this was really due to the fact that kind of around here there
[00:10:21] the fact that kind of around here there was some amazing excess um successes on
[00:10:23] was some amazing excess um successes on the Atari video games where people
[00:10:25] the Atari video games where people showed that you could learn directly
[00:10:27] showed that you could learn directly from Pixel input to make decisions and
[00:10:30] from Pixel input to make decisions and then there started to be the successes
[00:10:31] then there started to be the successes in alphago and then there became more
[00:10:33] in alphago and then there became more and more
[00:10:34] and more successes so it is an incredible time
[00:10:37] successes so it is an incredible time for reinforcement learning this curve
[00:10:38] for reinforcement learning this curve has continued to go up um however I
[00:10:41] has continued to go up um however I think it's also important to notice that
[00:10:43] think it's also important to notice that there's also a number of
[00:10:45] there's also a number of Skeptics so there was a pretty famous
[00:10:48] Skeptics so there was a pretty famous talk by Yan Lun um in 2016 at one of the
[00:10:52] talk by Yan Lun um in 2016 at one of the major machine learning conferences nurs
[00:10:55] major machine learning conferences nurs so Yan theun for those of you who don't
[00:10:56] so Yan theun for those of you who don't know him is one of the sort of seminal
[00:10:58] know him is one of the sort of seminal figures in neural network research he
[00:11:01] figures in neural network research he has won the Turing award he is um an
[00:11:04] has won the Turing award he is um an amazing amazing researcher so he gave a
[00:11:06] amazing amazing researcher so he gave a keynote at nerps I believe it was a
[00:11:09] keynote at nerps I believe it was a keynote he certainly gave a very famous
[00:11:10] keynote he certainly gave a very famous talk there where he was talking about
[00:11:12] talk there where he was talking about the role of different types of machine
[00:11:14] the role of different types of machine learning questions and at sub areas in
[00:11:17] learning questions and at sub areas in terms of making progress on machine
[00:11:19] terms of making progress on machine learning and he very famously talked
[00:11:21] learning and he very famously talked about machine learning as a cake and so
[00:11:23] about machine learning as a cake and so he said that the main cake is really
[00:11:25] he said that the main cake is really unsupervised learning and that's really
[00:11:27] unsupervised learning and that's really going to be the body the most important
[00:11:29] going to be the body the most important aspect respective machine learning
[00:11:30] aspect respective machine learning things like representation learning from
[00:11:32] things like representation learning from unlabeled data that's really going to be
[00:11:34] unlabeled data that's really going to be the core that's where we're going to
[00:11:35] the core that's where we're going to have huge amounts of data and we're
[00:11:37] have huge amounts of data and we're going to make a lot of progress and then
[00:11:39] going to make a lot of progress and then supervised learning was the icing so
[00:11:41] supervised learning was the icing so that's still pretty important it's like
[00:11:42] that's still pretty important it's like very important part of a cake at least
[00:11:44] very important part of a cake at least in my opinion um it doesn't have as much
[00:11:46] in my opinion um it doesn't have as much we don't have as much supervised
[00:11:48] we don't have as much supervised learning um and it's sort of this
[00:11:50] learning um and it's sort of this additional one and then he argued that
[00:11:52] additional one and then he argued that reinforcement learning was just the
[00:11:54] reinforcement learning was just the Cherry now you know cherries are
[00:11:56] Cherry now you know cherries are important um but uh but not nearly as
[00:11:59] important um but uh but not nearly as much perhaps as the rest of the cake and
[00:12:01] much perhaps as the rest of the cake and he went on and talked about some places
[00:12:02] he went on and talked about some places where he thought that RL still might
[00:12:04] where he thought that RL still might have a role but it was considered a
[00:12:06] have a role but it was considered a really important talk because what he
[00:12:09] really important talk because what he was sort of demonstrating is that
[00:12:11] was sort of demonstrating is that reinforcement learning was having a part
[00:12:13] reinforcement learning was having a part to play in machine learning but maybe
[00:12:15] to play in machine learning but maybe only a very minor part now I think it'd
[00:12:18] only a very minor part now I think it'd be interesting to talk to him today I
[00:12:19] be interesting to talk to him today I haven't talked to him recently so I
[00:12:20] haven't talked to him recently so I don't know what his current opinion is
[00:12:22] don't know what his current opinion is um but I think it's a really important
[00:12:24] um but I think it's a really important thing to think about like where are all
[00:12:26] thing to think about like where are all of these different techniques important
[00:12:28] of these different techniques important and where will we be able to to make the
[00:12:29] and where will we be able to to make the most progress in terms of advancing
[00:12:34] Ai and so with that we're going to try
[00:12:36] Ai and so with that we're going to try and do our first poll which is about why
[00:12:38] and do our first poll which is about why you guys want to take this class um so
[00:12:40] you guys want to take this class um so so we'll look through these you'll have
[00:12:42] so we'll look through these you'll have to bear with us a little bit with um we
[00:12:43] to bear with us a little bit with um we had a few technical difficulties that
[00:12:45] had a few technical difficulties that we're working with CTL on but it should
[00:12:47] we're working with CTL on but it should work out so if you go to either the
[00:12:48] work out so if you go to either the first link and Ed or you go to this HTTP
[00:12:51] first link and Ed or you go to this HTTP you can if you have any issues like do
[00:12:53] you can if you have any issues like do you want to be registered if it's
[00:12:55] you want to be registered if it's hanging just skip the registration
[00:12:57] hanging just skip the registration refresh that should all sort it out and
[00:12:59] refresh that should all sort it out and then didn't just Ed in your sun ID as
[00:13:01] then didn't just Ed in your sun ID as your screen and just take a second and
[00:13:03] your screen and just take a second and write down a bit about why you want to
[00:13:05] write down a bit about why you want to take this
[00:13:06] take this class and it could be anything it could
[00:13:08] class and it could be anything it could be that you're really curious about
[00:13:10] be that you're really curious about something it could be because you're
[00:13:11] something it could be because you're doing an internship and they told you
[00:13:13] doing an internship and they told you you had to take something about
[00:13:14] you had to take something about reinforcement learning any of the any of
[00:13:16] reinforcement learning any of the any of the things are
[00:13:17] the things are fine just take a minute or
[00:13:28] two
[00:13:58] for
[00:15:27] for for
[00:16:15] thanks for all the great reasons um I
[00:16:17] thanks for all the great reasons um I will talk about some of those when I
[00:16:19] will talk about some of those when I talk about also what we're going to
[00:16:20] talk about also what we're going to cover today and try to address why I
[00:16:22] cover today and try to address why I think that a lot of the things people
[00:16:23] think that a lot of the things people are bringing up are things that we're
[00:16:25] are bringing up are things that we're going to be touching upon so I think if
[00:16:28] going to be touching upon so I think if we want to think about um I think it's
[00:16:31] we want to think about um I think it's really important to start thinking about
[00:16:33] really important to start thinking about what is what is reinforcement learning
[00:16:34] what is what is reinforcement learning about because if we understand what it's
[00:16:36] about because if we understand what it's about then we know what sort of uh types
[00:16:38] about then we know what sort of uh types of questions we're interested in this
[00:16:39] of questions we're interested in this space and we also understand what sort
[00:16:41] space and we also understand what sort of applications it might be helpful for
[00:16:43] of applications it might be helpful for um though of course your creativity is
[00:16:45] um though of course your creativity is unlimited so you can see what you might
[00:16:46] unlimited so you can see what you might come up with other ideas that people may
[00:16:48] come up with other ideas that people may not have thought of for applying RL but
[00:16:51] not have thought of for applying RL but the the the four things that people top
[00:16:53] the the the four things that people top typically think about when they think
[00:16:54] typically think about when they think about reinforcement learning as a
[00:16:56] about reinforcement learning as a discipline and as the um sort of what
[00:16:57] discipline and as the um sort of what reinforcement learning involves is
[00:17:00] reinforcement learning involves is optimization delayed consequences
[00:17:02] optimization delayed consequences exploration and
[00:17:04] exploration and generalization so the first is
[00:17:07] generalization so the first is optimization um and the optimization
[00:17:09] optimization um and the optimization aspect is really just saying that we're
[00:17:11] aspect is really just saying that we're thinking about the best way to make
[00:17:14] thinking about the best way to make decisions which means that we explicitly
[00:17:16] decisions which means that we explicitly have to have some notion of
[00:17:19] have to have some notion of utility this um would an example of this
[00:17:23] utility this um would an example of this would be something like finding the
[00:17:24] would be something like finding the minimum distance route between two
[00:17:25] minimum distance route between two cities given a network of roads this
[00:17:27] cities given a network of roads this means you can directly compare different
[00:17:29] means you can directly compare different solutions because if one solution has a
[00:17:31] solutions because if one solution has a smaller distance than the other it is
[00:17:32] smaller distance than the other it is strictly preferred so there are many
[00:17:35] strictly preferred so there are many many important optimization questions
[00:17:37] many important optimization questions and reinforcement learning because it is
[00:17:38] and reinforcement learning because it is concerned with making good decisions
[00:17:40] concerned with making good decisions cares about us being able to rank or you
[00:17:42] cares about us being able to rank or you know decide across those different
[00:17:45] know decide across those different ones the second one is delayed
[00:17:48] ones the second one is delayed consequences the idea being that the
[00:17:50] consequences the idea being that the decisions that we make now can affect
[00:17:51] decisions that we make now can affect things far later so maybe saving for
[00:17:54] things far later so maybe saving for retirement now has some uh immediate
[00:17:57] retirement now has some uh immediate cost but leads to some significant
[00:17:59] cost but leads to some significant benefit later or maybe there's something
[00:18:00] benefit later or maybe there's something you can do early in a video game that
[00:18:02] you can do early in a video game that later has a lot of benefit there are two
[00:18:05] later has a lot of benefit there are two reasons why delayed consequences is
[00:18:07] reasons why delayed consequences is challenging one is for the reason of
[00:18:10] challenging one is for the reason of planning um many of you might have
[00:18:12] planning um many of you might have actually raised your hand if you've
[00:18:13] actually raised your hand if you've taken AI at
[00:18:15] taken AI at Stanford okay so about half of you so
[00:18:17] Stanford okay so about half of you so you probably saw planning in Ai and and
[00:18:19] you probably saw planning in Ai and and planning is the idea that even when we
[00:18:22] planning is the idea that even when we understand how the world works it might
[00:18:23] understand how the world works it might be really complicated to try to decide
[00:18:25] be really complicated to try to decide what the optimal thing is to do so you
[00:18:27] what the optimal thing is to do so you could think of this like chess the rules
[00:18:29] could think of this like chess the rules are known um it's still really
[00:18:30] are known um it's still really complicated to think about what's the
[00:18:32] complicated to think about what's the right thing to do so when the decisions
[00:18:34] right thing to do so when the decisions you make involve reasoning not just
[00:18:36] you make involve reasoning not just about the immediate outcomes but the
[00:18:37] about the immediate outcomes but the longer term ramifications these sort of
[00:18:39] longer term ramifications these sort of planning problems are even
[00:18:41] planning problems are even harder but the other reason this is
[00:18:43] harder but the other reason this is really hard is when we're learning
[00:18:44] really hard is when we're learning meaning that like we don't know how the
[00:18:46] meaning that like we don't know how the world works and we're trying to
[00:18:47] world works and we're trying to understand how um through direct
[00:18:49] understand how um through direct experience so when we're learning um
[00:18:52] experience so when we're learning um temporal credit assignment is hard
[00:18:54] temporal credit assignment is hard meaning that if you take some action now
[00:18:56] meaning that if you take some action now and later on you receive a good outcome
[00:18:58] and later on you receive a good outcome or a bad outcome how do you figure out
[00:19:00] or a bad outcome how do you figure out which of your outcomes caused that good
[00:19:02] which of your outcomes caused that good or bad later later
[00:19:04] or bad later later result this happens all the time to us
[00:19:06] result this happens all the time to us as humans right like how do you know why
[00:19:08] as humans right like how do you know why you got into Stamford well I don't know
[00:19:10] you got into Stamford well I don't know was it because you colored your you know
[00:19:11] was it because you colored your you know you did you wanted coloring contest when
[00:19:13] you did you wanted coloring contest when you were a six because you scored well
[00:19:15] you were a six because you scored well in the SAT because you went to a good
[00:19:16] in the SAT because you went to a good high school or you wrote a really good
[00:19:18] high school or you wrote a really good essay it's really hard to understand
[00:19:20] essay it's really hard to understand this in some cases it may be impossible
[00:19:22] this in some cases it may be impossible but when we're getting to make repeated
[00:19:24] but when we're getting to make repeated decisions it's really important that we
[00:19:26] decisions it's really important that we can start to use the prior experience to
[00:19:28] can start to use the prior experience to figure out out which decisions were
[00:19:30] figure out out which decisions were important or LED to good outcomes so
[00:19:32] important or LED to good outcomes so that we can repeat them so that's one of
[00:19:34] that we can repeat them so that's one of the reasons why this is
[00:19:36] the reasons why this is hard exploration is one of my favorite
[00:19:38] hard exploration is one of my favorite things um in terms of reinforcement
[00:19:40] things um in terms of reinforcement learning and the idea of this is that
[00:19:43] learning and the idea of this is that the agent can only learn about the world
[00:19:45] the agent can only learn about the world through direct experience so it's like
[00:19:48] through direct experience so it's like trying to learn to ride a bike by trying
[00:19:50] trying to learn to ride a bike by trying and failing and trying again um and
[00:19:53] and failing and trying again um and through that direct experiencing
[00:19:54] through that direct experiencing learning the right way to ride a bike
[00:19:58] learning the right way to ride a bike and the key idea about this is that um
[00:20:02] and the key idea about this is that um information is censored in that you only
[00:20:04] information is censored in that you only get to learn about what you
[00:20:06] get to learn about what you try so for example right now you don't
[00:20:08] try so for example right now you don't know how much worse your life would be
[00:20:09] know how much worse your life would be if you were MIT I went to MIT for grad
[00:20:12] if you were MIT I went to MIT for grad school MIT is also a great place um but
[00:20:15] school MIT is also a great place um but you generally can't ever understand what
[00:20:17] you generally can't ever understand what that
[00:20:17] that counterfactual um life would have been
[00:20:19] counterfactual um life would have been like right it's one of the the central
[00:20:22] like right it's one of the the central challenges um it's also a huge challenge
[00:20:24] challenges um it's also a huge challenge in causal inference um which is another
[00:20:26] in causal inference um which is another big interest of mine and something my
[00:20:27] big interest of mine and something my lab works on so so this's this General
[00:20:29] lab works on so so this's this General challenge that you only get to learn
[00:20:30] challenge that you only get to learn about the actual things that you do as
[00:20:33] about the actual things that you do as an agent or as a human as an agent Etc
[00:20:36] an agent or as a human as an agent Etc um and so the question is how do you use
[00:20:38] um and so the question is how do you use that experience to figure out how to
[00:20:40] that experience to figure out how to make good decisions so as a concrete
[00:20:42] make good decisions so as a concrete example of this you can imagine you're a
[00:20:43] example of this you can imagine you're a company and you give some promotion to
[00:20:45] company and you give some promotion to all your customers you can't know what
[00:20:47] all your customers you can't know what it would have been like if you didn't
[00:20:49] it would have been like if you didn't give the promotion to those customers
[00:20:51] give the promotion to those customers and even if you can give it to One
[00:20:52] and even if you can give it to One customer and not another they are not
[00:20:54] customer and not another they are not the same people so I can't rewind and
[00:20:56] the same people so I can't rewind and say dilip who is our head ta this time
[00:20:59] say dilip who is our head ta this time I'm not going to give you the promotion
[00:21:00] I'm not going to give you the promotion let's see how that world would have
[00:21:01] let's see how that world would have worked out that's one of the central
[00:21:04] worked out that's one of the central challenges so we'll talk a lot about
[00:21:06] challenges so we'll talk a lot about exploration later because it's one of
[00:21:07] exploration later because it's one of the key things that is different
[00:21:09] the key things that is different compared to many prior
[00:21:11] compared to many prior approaches and generalization um has to
[00:21:14] approaches and generalization um has to do with this question of really wanting
[00:21:16] do with this question of really wanting to solve really big challenging problems
[00:21:19] to solve really big challenging problems so we'll talk a lot about what decision
[00:21:21] so we'll talk a lot about what decision policies are but in general you can just
[00:21:22] policies are but in general you can just think of them as a mapping from
[00:21:24] think of them as a mapping from experience to to a decision and you
[00:21:26] experience to to a decision and you might think in those cases you could
[00:21:27] might think in those cases you could just pre-program it so like if your
[00:21:29] just pre-program it so like if your robot goes down the hallway if it hits
[00:21:31] robot goes down the hallway if it hits the end of the hallway turn
[00:21:33] the end of the hallway turn left but let's think about um a video
[00:21:36] left but let's think about um a video game which we can think of as just sort
[00:21:37] game which we can think of as just sort of generally having some input image so
[00:21:40] of generally having some input image so let's imagine that it's something like
[00:21:41] let's imagine that it's something like 300 by
[00:21:43] 300 by 400 and let's say we have at least 256
[00:21:46] 400 and let's say we have at least 256 different colors so now we have an image
[00:21:50] different colors so now we have an image set of images that we could see that is
[00:21:51] set of images that we could see that is at least 256 to the 300 CR 400 so those
[00:21:55] at least 256 to the 300 CR 400 so those are at least the space of images and
[00:21:57] are at least the space of images and that's probably an underestimate
[00:21:59] that's probably an underestimate and now we get to think about what we
[00:22:01] and now we get to think about what we would do in each of those different
[00:22:03] would do in each of those different scenarios so the combinatorics are
[00:22:06] scenarios so the combinatorics are completely mind-blowing and we can't
[00:22:07] completely mind-blowing and we can't write these down in a table so this is
[00:22:09] write these down in a table so this is why we would need something like a deep
[00:22:10] why we would need something like a deep neural network or something else in
[00:22:12] neural network or something else in order for us to try to make decisions in
[00:22:15] order for us to try to make decisions in these real like realistic settings which
[00:22:16] these real like realistic settings which are extremely large in terms of the the
[00:22:19] are extremely large in terms of the the type of scenarios the number of
[00:22:21] type of scenarios the number of scenarios we want to make decisions
[00:22:24] scenarios we want to make decisions on so you've probably seen all of these
[00:22:26] on so you've probably seen all of these ideas or at least most of them in other
[00:22:28] ideas or at least most of them in other classes for other types of AI or machine
[00:22:31] classes for other types of AI or machine learning so I think it's useful just to
[00:22:33] learning so I think it's useful just to contrast what is reinforcement learning
[00:22:35] contrast what is reinforcement learning doing um compared to these other ones so
[00:22:38] doing um compared to these other ones so the first is AI planning so in AI
[00:22:41] the first is AI planning so in AI planning generally we're doing some form
[00:22:42] planning generally we're doing some form of optimization Trying to minimize a
[00:22:45] of optimization Trying to minimize a distance or something like that we are
[00:22:47] distance or something like that we are often trying to handle delayed
[00:22:49] often trying to handle delayed consequences and um and those are the
[00:22:52] consequences and um and those are the two main things so we we might also have
[00:22:53] two main things so we we might also have to do generalization if the the size of
[00:22:56] to do generalization if the the size of the Space is really large
[00:22:58] the Space is really large okay so that is how so RL in general
[00:23:02] okay so that is how so RL in general will involve all of these so this is how
[00:23:04] will involve all of these so this is how those would
[00:23:05] those would compare if we think about something like
[00:23:07] compare if we think about something like supervised learning supervised learning
[00:23:09] supervised learning supervised learning does involve learning so we learn from
[00:23:13] does involve learning so we learn from data you know whether something's a cat
[00:23:14] data you know whether something's a cat or
[00:23:15] or not and we have to do
[00:23:20] generalization so we have those two
[00:23:22] generalization so we have those two things and again this is going to be
[00:23:23] things and again this is going to be compared to reinforcement learning which
[00:23:25] compared to reinforcement learning which has all of those
[00:23:29] in contrast to supervised learning where
[00:23:30] in contrast to supervised learning where you get the correct labels and
[00:23:32] you get the correct labels and unsupervised learning we don't get any
[00:23:33] unsupervised learning we don't get any labels but we're still learning from
[00:23:35] labels but we're still learning from experience and we're still trying to do
[00:23:41] generalization now the next thing and
[00:23:43] generalization now the next thing and this has become a really poopular thing
[00:23:45] this has become a really poopular thing is to think about whether we can map um
[00:23:48] is to think about whether we can map um reinforcement learning to imitation
[00:23:49] reinforcement learning to imitation learning we talked about this really
[00:23:51] learning we talked about this really briefly um about chat GPT and we'll talk
[00:23:54] briefly um about chat GPT and we'll talk about a lot more in the
[00:23:56] about a lot more in the course so in imitation learning or
[00:23:59] course so in imitation learning or behavior cloning or reducing
[00:24:01] behavior cloning or reducing reinforcement learning to supervised
[00:24:02] reinforcement learning to supervised learning we generally assume that we get
[00:24:04] learning we generally assume that we get access to expert
[00:24:06] access to expert trajectories so this could be like
[00:24:08] trajectories so this could be like someone saying what they would do in
[00:24:09] someone saying what they would do in response to those prompts it could be
[00:24:11] response to those prompts it could be someone driving a car and then you want
[00:24:13] someone driving a car and then you want to mimic their behavior um or or some
[00:24:16] to mimic their behavior um or or some other similar example so then these
[00:24:18] other similar example so then these ideas is that we get input
[00:24:19] ideas is that we get input demonstrations of good policies and that
[00:24:22] demonstrations of good policies and that allows us to reduce reinforcement
[00:24:23] allows us to reduce reinforcement learning back to supervised
[00:24:25] learning back to supervised learning so we're s of taking this we're
[00:24:28] learning so we're s of taking this we're reducing it back to
[00:24:31] reducing it back to here now I think in general the idea of
[00:24:33] here now I think in general the idea of reductions is incredibly powerful um for
[00:24:36] reductions is incredibly powerful um for those of you that have taken CS Theory
[00:24:38] those of you that have taken CS Theory classes that's what we do all the time
[00:24:39] classes that's what we do all the time we reduce things to sat or other things
[00:24:41] we reduce things to sat or other things like that and in general I think in
[00:24:43] like that and in general I think in computer science it's one of the
[00:24:44] computer science it's one of the strengths of it that they think of how
[00:24:46] strengths of it that they think of how can we reduce one problem to another and
[00:24:48] can we reduce one problem to another and then inherit all the progress that's
[00:24:49] then inherit all the progress that's been made on that
[00:24:50] been made on that problem so in this way reinforcement
[00:24:53] problem so in this way reinforcement learning is similar to other aspects of
[00:24:55] learning is similar to other aspects of computer science in that we will try
[00:24:57] computer science in that we will try often to reduce
[00:24:59] often to reduce reinforcement learning to other problems
[00:25:01] reinforcement learning to other problems this is particularly done in the
[00:25:02] this is particularly done in the theoretical aspects of reinforcement
[00:25:05] theoretical aspects of reinforcement learning yeah yeah just oh whenever you
[00:25:08] learning yeah yeah just oh whenever you um ask just because I'm going to try and
[00:25:09] um ask just because I'm going to try and learn names could you say your name
[00:25:10] learn names could you say your name please yeah my name is yeah so just to
[00:25:13] please yeah my name is yeah so just to be clear um imitation learning then
[00:25:15] be clear um imitation learning then isn't like a separate technique it's
[00:25:17] isn't like a separate technique it's just an application of supervised
[00:25:18] just an application of supervised learning to like the specific
[00:25:20] learning to like the specific reinforcement learning context it's good
[00:25:22] reinforcement learning context it's good question so I think some of you I mean
[00:25:24] question so I think some of you I mean there's a lot of techniques that think
[00:25:25] there's a lot of techniques that think about when you're doing imitation
[00:25:27] about when you're doing imitation learning specifically for kind of
[00:25:28] learning specifically for kind of decision data you can just think of it
[00:25:31] decision data you can just think of it just reducing it back um if you want to
[00:25:35] just reducing it back um if you want to do imitation learning where you might
[00:25:36] do imitation learning where you might recover like the reward function we'll
[00:25:38] recover like the reward function we'll talk more about that soon and others
[00:25:40] talk more about that soon and others then you may need to use other types of
[00:25:41] then you may need to use other types of techniques as well but like the most
[00:25:44] techniques as well but like the most straightforward aspect is just to say
[00:25:46] straightforward aspect is just to say I've got demonstrations I'm going to
[00:25:48] I've got demonstrations I'm going to ignore um sort of like this uh delayed
[00:25:51] ignore um sort of like this uh delayed consequences aspect and exploration and
[00:25:53] consequences aspect and exploration and I'm just going to reduce it
[00:25:55] I'm just going to reduce it back yeah and name first please
[00:25:59] back yeah and name first please wait what do you mean by input
[00:26:00] wait what do you mean by input demonstrations of good calls what does
[00:26:02] demonstrations of good calls what does that mean great question so let me give
[00:26:04] that mean great question so let me give you an example so people have thought a
[00:26:05] you an example so people have thought a lot about this maybe maybe one of the
[00:26:07] lot about this maybe maybe one of the first examples of this or one of the
[00:26:09] first examples of this or one of the first really public examples of this was
[00:26:10] first really public examples of this was for driving like at least what you could
[00:26:12] for driving like at least what you could do is you could I could drive a car it
[00:26:14] do is you could I could drive a car it could record everything that I do in
[00:26:16] could record everything that I do in terms of controlling the steering wheel
[00:26:18] terms of controlling the steering wheel and then we could if I'm a good driver
[00:26:20] and then we could if I'm a good driver they could say that's a good
[00:26:21] they could say that's a good demonstration so instead of the car
[00:26:23] demonstration so instead of the car trying to learn from itself how to steer
[00:26:25] trying to learn from itself how to steer the wheel in order to say successfully
[00:26:27] the wheel in order to say successfully Drive you could have humans drive it and
[00:26:29] Drive you could have humans drive it and it could try to figure out at each point
[00:26:32] it could try to figure out at each point what like how should I steer the wheel
[00:26:34] what like how should I steer the wheel in order to um have good behavior so the
[00:26:38] in order to um have good behavior so the idea is that you actually have access
[00:26:40] idea is that you actually have access already to good demonstrations of what
[00:26:42] already to good demonstrations of what is a good
[00:26:44] is a good policy yeah name CH
[00:26:48] please what do you exactly mean by
[00:26:50] please what do you exactly mean by optimization
[00:26:53] and optimization and what what do you
[00:26:56] and optimization and what what do you mean by optimization
[00:26:59] mean by optimization uh okay good question so what I mean is
[00:27:00] uh okay good question so what I mean is that when we do imitation learning from
[00:27:03] that when we do imitation learning from good trajectories we are assuming that
[00:27:05] good trajectories we are assuming that like we want to do well so we want to
[00:27:07] like we want to do well so we want to actually get a good policy so imitation
[00:27:09] actually get a good policy so imitation learning normally we're not normally
[00:27:11] learning normally we're not normally trying to imitate bad performance um but
[00:27:14] trying to imitate bad performance um but you could think of this as sort of
[00:27:16] you could think of this as sort of reinforcement learning but without the
[00:27:18] reinforcement learning but without the the exploration part because it's not
[00:27:19] the exploration part because it's not trying to pick its own
[00:27:25] data have the optimization
[00:27:28] data have the optimization yeah so I think um because we normally
[00:27:30] yeah so I think um because we normally don't have the notion of uh utility in
[00:27:32] don't have the notion of uh utility in those so you might say this is a cat or
[00:27:34] those so you might say this is a cat or it's not a cat it's not like a good
[00:27:37] it's not a cat it's not like a good picture of a cat or not whereas in
[00:27:38] picture of a cat or not whereas in decisions we often have a real valued
[00:27:40] decisions we often have a real valued scalar value of like it was like you
[00:27:42] scalar value of like it was like you know A7 good
[00:27:45] know A7 good decision yeah name first
[00:27:51] please optimize yes so we do often we
[00:27:55] please optimize yes so we do often we always have loss functions um and that's
[00:27:57] always have loss functions um and that's a great but where in those cases there's
[00:27:59] a great but where in those cases there's not normally a utility that goes so if
[00:28:01] not normally a utility that goes so if you could
[00:28:02] you could get you could maybe have some smooth
[00:28:04] get you could maybe have some smooth notion there of how well do you match
[00:28:06] notion there of how well do you match like a stochastic policy a stochastic
[00:28:08] like a stochastic policy a stochastic output there um but for many of those it
[00:28:11] output there um but for many of those it would be more if it's like did you say
[00:28:13] would be more if it's like did you say it was a cat or not you would have a
[00:28:15] it was a cat or not you would have a binary 01
[00:28:17] binary 01 loss
[00:28:19] loss yeah so does that mean if you have the
[00:28:21] yeah so does that mean if you have the data for imitation learning it's like
[00:28:24] data for imitation learning it's like almost always better than reinforcement
[00:28:26] almost always better than reinforcement learning you're kind of like avoiding
[00:28:27] learning you're kind of like avoiding the you can like directly learn what is
[00:28:31] the you can like directly learn what is good say that again so like if you have
[00:28:34] good say that again so like if you have the data for imitation learning like you
[00:28:36] the data for imitation learning like you have someone actually driving the car
[00:28:38] have someone actually driving the car does that mean that um you'll probably
[00:28:41] does that mean that um you'll probably learn a better policy than reinforcement
[00:28:43] learn a better policy than reinforcement learning or that's almost always great
[00:28:46] learning or that's almost always great question so we'll get into that so the
[00:28:47] question so we'll get into that so the question was if you can hear that is if
[00:28:48] question was if you can hear that is if you have good demonstrations say of
[00:28:49] you have good demonstrations say of driving behavior um that you're using
[00:28:51] driving behavior um that you're using imitation learning can that be better
[00:28:53] imitation learning can that be better than reinforcement learning it will
[00:28:54] than reinforcement learning it will depend on your reinforcement learning
[00:28:55] depend on your reinforcement learning algorithm in general reinforcement
[00:28:57] algorithm in general reinforcement learning should always always be able to
[00:28:59] learning should always always be able to exceed equal or ex um exceed the
[00:29:02] exceed equal or ex um exceed the performance of imitation learning yeah
[00:29:05] performance of imitation learning yeah so can you explain the difference
[00:29:07] so can you explain the difference between ilil and
[00:29:10] between ilil and rlf yes great question so in imitation
[00:29:12] rlf yes great question so in imitation learning um what we would uh have and
[00:29:15] learning um what we would uh have and this is what this was the first part you
[00:29:17] this is what this was the first part you would say people give me um given a
[00:29:20] would say people give me um given a prompt I look on the internet and I
[00:29:21] prompt I look on the internet and I assume that those were good so um in the
[00:29:24] assume that those were good so um in the internet I see if someone said like how
[00:29:26] internet I see if someone said like how to explain uh reinforcement learning to
[00:29:28] to explain uh reinforcement learning to six-year-old this is what they said back
[00:29:29] six-year-old this is what they said back and so I just train on those what rhf
[00:29:33] and so I just train on those what rhf said is that well you know the
[00:29:35] said is that well you know the internet's a big place probably not all
[00:29:36] internet's a big place probably not all of it is good answers so now let's
[00:29:38] of it is good answers so now let's actually ask people which of these two
[00:29:40] actually ask people which of these two responses they prefer and now we're
[00:29:41] responses they prefer and now we're going to try to do reinforcement
[00:29:43] going to try to do reinforcement learning on that to actually get to a
[00:29:44] learning on that to actually get to a better
[00:29:45] better policy
[00:29:48] policy yeah I'd like to ask so alab actually
[00:29:51] yeah I'd like to ask so alab actually discover some go strategies that are not
[00:29:54] discover some go strategies that are not invented by humans that we have never
[00:29:56] invented by humans that we have never experienced before so doesn't mean that
[00:29:59] experienced before so doesn't mean that if we apply imitation learning too much
[00:30:01] if we apply imitation learning too much it might actually hinder the model's
[00:30:02] it might actually hinder the model's capabilities to explore like what is
[00:30:05] capabilities to explore like what is actually good in like what humans have
[00:30:07] actually good in like what humans have thought of which is probably absolutely
[00:30:09] thought of which is probably absolutely and actually I think this is on the next
[00:30:11] and actually I think this is on the next side let's go it good okay perfect so
[00:30:13] side let's go it good okay perfect so this turn to where are some of the
[00:30:15] this turn to where are some of the places that you might hope that
[00:30:16] places that you might hope that reinforcement learning would be better
[00:30:17] reinforcement learning would be better than these other strategies so one of
[00:30:20] than these other strategies so one of them is where you don't have examples of
[00:30:21] them is where you don't have examples of desired Behavior this is exactly like
[00:30:23] desired Behavior this is exactly like the example that was just brought up if
[00:30:25] the example that was just brought up if you want to go beyond Human Performance
[00:30:27] you want to go beyond Human Performance you cannot rely on Human Performance
[00:30:29] you cannot rely on Human Performance just to do imitation learning because
[00:30:30] just to do imitation learning because you're not going to be able to go get
[00:30:32] you're not going to be able to go get better than it so there are a lot of
[00:30:34] better than it so there are a lot of application areas I think particularly
[00:30:36] application areas I think particularly in areas like healthcare or education
[00:30:38] in areas like healthcare or education and others where we think we can go
[00:30:40] and others where we think we can go beyond Human Performance um and so in
[00:30:43] beyond Human Performance um and so in those cases reinforcement learning
[00:30:44] those cases reinforcement learning because it's trying to optimize
[00:30:45] because it's trying to optimize performance um could go beyond it could
[00:30:48] performance um could go beyond it could be a particularly useful technique and
[00:30:50] be a particularly useful technique and others where you don't have any existing
[00:30:52] others where you don't have any existing data for a task so there might be
[00:30:54] data for a task so there might be something where you think of it as um a
[00:30:55] something where you think of it as um a decision-making problem but you don't
[00:30:57] decision-making problem but you don't have prior data you need to learn from
[00:30:58] have prior data you need to learn from scratch um and you want to directly
[00:31:01] scratch um and you want to directly optimize so that's another place where
[00:31:02] optimize so that's another place where reinforcement learning can be very
[00:31:05] reinforcement learning can be very powerful another category is interesting
[00:31:08] powerful another category is interesting because in some ways it's also kind of a
[00:31:10] because in some ways it's also kind of a reduction technique and this is the
[00:31:12] reduction technique and this is the question to place where you have an
[00:31:13] question to place where you have an enormous search or optimization problem
[00:31:15] enormous search or optimization problem with delayed outcomes so there's been a
[00:31:18] with delayed outcomes so there's been a number of examples of the work of doing
[00:31:20] number of examples of the work of doing this from Deep Mind which have been
[00:31:22] this from Deep Mind which have been really extremely elegant so what I I put
[00:31:24] really extremely elegant so what I I put up here is Alpha tensor um if you
[00:31:26] up here is Alpha tensor um if you haven't heard of it it's a faster way to
[00:31:28] haven't heard of it it's a faster way to do matrix multiplication which is kind
[00:31:30] do matrix multiplication which is kind of mind-blowing so what they did is they
[00:31:32] of mind-blowing so what they did is they said all right there's standard ways to
[00:31:34] said all right there's standard ways to do matrix multiplication this comes up
[00:31:36] do matrix multiplication this comes up all the time could we learn an algorithm
[00:31:39] all the time could we learn an algorithm that would be better at matrix
[00:31:40] that would be better at matrix multiplication not me as like a
[00:31:42] multiplication not me as like a scientist try to write down an algorithm
[00:31:44] scientist try to write down an algorithm have a an agent learn one and they they
[00:31:48] have a an agent learn one and they they showed yes and the way they did that was
[00:31:49] showed yes and the way they did that was with reinforcement learning um and
[00:31:51] with reinforcement learning um and they've done this in other cases too
[00:31:53] they've done this in other cases too like learning faster sorting
[00:31:55] like learning faster sorting algorithms so I think this is a pretty
[00:31:56] algorithms so I think this is a pretty incredible Frontier the idea is saying
[00:31:58] incredible Frontier the idea is saying could we have ai actually be inventing
[00:32:00] could we have ai actually be inventing new algorithms um and one of the ways
[00:32:03] new algorithms um and one of the ways that they they framed it here and you
[00:32:05] that they they framed it here and you can think of alphao is similar is that
[00:32:07] can think of alphao is similar is that it was a really really really large
[00:32:09] it was a really really really large search
[00:32:10] search problem and the challenge with really
[00:32:12] problem and the challenge with really really large search problems is that
[00:32:14] really large search problems is that even there we may not have great
[00:32:16] even there we may not have great techniques for solving them and so it's
[00:32:18] techniques for solving them and so it's sort of a reduction you can think of
[00:32:20] sort of a reduction you can think of people taking a planning problem and
[00:32:21] people taking a planning problem and trying to reduce it to a reinforcement
[00:32:23] trying to reduce it to a reinforcement learning problem to make it more
[00:32:25] learning problem to make it more tractable so that's pretty wild most of
[00:32:27] tractable so that's pretty wild most of the time we think of sort of ourl go
[00:32:29] the time we think of sort of ourl go being reduced in the other direction or
[00:32:30] being reduced in the other direction or involving planning or about that but
[00:32:32] involving planning or about that but here in some ways you can think of these
[00:32:34] here in some ways you can think of these as like um either adversarial planning
[00:32:36] as like um either adversarial planning problems or expecting Max problems that
[00:32:38] problems or expecting Max problems that are being reduced back to to learning as
[00:32:40] are being reduced back to to learning as a way to just more efficiently go
[00:32:42] a way to just more efficiently go through the search space so those are
[00:32:44] through the search space so those are two of the areas that I think are
[00:32:45] two of the areas that I think are particularly promising in terms of why
[00:32:47] particularly promising in terms of why reinforcement learning is still a really
[00:32:50] reinforcement learning is still a really um practical and really important area
[00:32:52] um practical and really important area to think
[00:32:53] to think about I think I saw a question back but
[00:32:55] about I think I saw a question back but maybe you yeah oh what was your name
[00:32:58] maybe you yeah oh what was your name for Alpha tenser is that like it's FAS
[00:33:01] for Alpha tenser is that like it's FAS but within some error of the correct
[00:33:02] but within some error of the correct matrix product it's faster but with some
[00:33:05] matrix product it's faster but with some error or do you actually get the corre
[00:33:07] error or do you actually get the corre value no you get the correct value which
[00:33:08] value no you get the correct value which is wild yeah yeah so no it's just better
[00:33:11] is wild yeah yeah so no it's just better yeah and one of the really clever things
[00:33:13] yeah and one of the really clever things they had to think of in this case was
[00:33:14] they had to think of in this case was how do you know that the answer is
[00:33:16] how do you know that the answer is correct how could you like provably
[00:33:17] correct how could you like provably verify that so yeah incredibly
[00:33:21] verify that so yeah incredibly elegant all right now we're going to go
[00:33:24] elegant all right now we're going to go quickly through some course Logistics um
[00:33:26] quickly through some course Logistics um before starting to dive into some
[00:33:27] before starting to dive into some content and feel free to interrupt me
[00:33:29] content and feel free to interrupt me throughout this or anything else if you
[00:33:30] throughout this or anything else if you have other
[00:33:32] have other questions so in terms of the content um
[00:33:35] questions so in terms of the content um we're going to start off by talking
[00:33:36] we're going to start off by talking about markup decision processes and
[00:33:38] about markup decision processes and planning and then we're going to talk
[00:33:40] planning and then we're going to talk about model free um policy evaluation
[00:33:43] about model free um policy evaluation and model free control don't worry if
[00:33:44] and model free control don't worry if you don't know what I mean by model I
[00:33:46] you don't know what I mean by model I I'll specify it then we're going to jump
[00:33:48] I'll specify it then we're going to jump into policy search policy search is
[00:33:50] into policy search policy search is things like um uh proximal policy
[00:33:53] things like um uh proximal policy optimization and other uh and reinforc
[00:33:56] optimization and other uh and reinforc and other approaches some of you guys
[00:33:57] and other approaches some of you guys might have already seen related ideas
[00:34:00] might have already seen related ideas say in robotics if you've taken them um
[00:34:03] say in robotics if you've taken them um and then I'm highlighting here that this
[00:34:04] and then I'm highlighting here that this is one of the important differences
[00:34:06] is one of the important differences compared to Prior years so we're going
[00:34:10] compared to Prior years so we're going to do um a a deep dive into offline
[00:34:12] to do um a a deep dive into offline reinforcement learning offline here
[00:34:14] reinforcement learning offline here meaning that we have a fixed amount of
[00:34:16] meaning that we have a fixed amount of data and we want to learn from it to get
[00:34:18] data and we want to learn from it to get a good decision policy and during this
[00:34:21] a good decision policy and during this we're going to talk about reinforcement
[00:34:22] we're going to talk about reinforcement learning from Human feedback and direct
[00:34:24] learning from Human feedback and direct preference
[00:34:25] preference optimization so that's um going to be a
[00:34:28] optimization so that's um going to be a new that's going to be a new sort of
[00:34:30] new that's going to be a new sort of third part of the course that we haven't
[00:34:31] third part of the course that we haven't done assignments on before so I think
[00:34:33] done assignments on before so I think that'll be pretty exciting and we'll
[00:34:35] that'll be pretty exciting and we'll also talk about exploration and do
[00:34:36] also talk about exploration and do Advanced
[00:34:40] topics so the high level learning goals
[00:34:43] topics so the high level learning goals of the class is that by the end of class
[00:34:44] of the class is that by the end of class you should be able to define the key
[00:34:45] you should be able to define the key features of reinforcement learning you
[00:34:48] features of reinforcement learning you should be able to given an application
[00:34:51] should be able to given an application specify how you would write that down as
[00:34:52] specify how you would write that down as a reinforcement learning problem as well
[00:34:54] a reinforcement learning problem as well as well as whether or not you think it
[00:34:56] as well as whether or not you think it would be good to use RL for it
[00:34:58] would be good to use RL for it that you can Implement and Code common
[00:34:59] that you can Implement and Code common RL algorithms and that you understand
[00:35:01] RL algorithms and that you understand the theoretical and empirical approaches
[00:35:04] the theoretical and empirical approaches for evaluating the quality of an RL
[00:35:06] for evaluating the quality of an RL algorithm so as you could probably
[00:35:08] algorithm so as you could probably imagine from those papers going up
[00:35:09] imagine from those papers going up there's going to be there's going to be
[00:35:10] there's going to be there's going to be continued progress in this field there's
[00:35:12] continued progress in this field there's going to be a huge number of different
[00:35:14] going to be a huge number of different RL algorithms and so one of the key
[00:35:15] RL algorithms and so one of the key things that I hope to talk about is sort
[00:35:17] things that I hope to talk about is sort of how do you evaluate and compare them
[00:35:19] of how do you evaluate and compare them um which might vary depending on the
[00:35:21] um which might vary depending on the application area you care
[00:35:23] application area you care about so the way that the course is
[00:35:26] about so the way that the course is structured is that we'll have live
[00:35:27] structured is that we'll have live lectures we'll have three homeworks
[00:35:29] lectures we'll have three homeworks we'll have a midterm we'll have a
[00:35:31] we'll have a midterm we'll have a multiple choice
[00:35:32] multiple choice quiz we'll do a final project um and
[00:35:36] quiz we'll do a final project um and then we'll have what I call check or
[00:35:37] then we'll have what I call check or refresh your understanding exercises
[00:35:39] refresh your understanding exercises which will be going through um the poll
[00:35:42] which will be going through um the poll anywhere and we'll have problem sessions
[00:35:44] anywhere and we'll have problem sessions which are optional problem sessions are
[00:35:47] which are optional problem sessions are a great chance to think more about the
[00:35:49] a great chance to think more about the conceptual and the theoretical aspects
[00:35:50] conceptual and the theoretical aspects of the class um and they'll be held
[00:35:53] of the class um and they'll be held starting next
[00:35:56] week so one of the main application
[00:35:59] week so one of the main application areas I think about a lot is education I
[00:36:01] areas I think about a lot is education I think education is one of the greatest
[00:36:02] think education is one of the greatest tools we have to try to address poverty
[00:36:05] tools we have to try to address poverty um and inequality and so I'm really
[00:36:07] um and inequality and so I'm really interested in evidence uh to think about
[00:36:10] interested in evidence uh to think about how do we educate effectively so with
[00:36:12] how do we educate effectively so with respect to that I wanted to share this
[00:36:14] respect to that I wanted to share this paper that came out I guess almost a
[00:36:16] paper that came out I guess almost a decade ago to now where they did a study
[00:36:18] decade ago to now where they did a study to look at how people who are taking
[00:36:20] to look at how people who are taking massive open online courses how they
[00:36:23] massive open online courses how they spent their time and how that related to
[00:36:24] spent their time and how that related to their learning outcomes and what they
[00:36:26] their learning outcomes and what they found is that um if you do more
[00:36:28] found is that um if you do more activities there seem to be a six times
[00:36:30] activities there seem to be a six times L larger learning benefit compared to
[00:36:32] L larger learning benefit compared to watching videos or
[00:36:34] watching videos or reading and um you might think this is
[00:36:36] reading and um you might think this is just based on time but it wasn't in fact
[00:36:38] just based on time but it wasn't in fact it seemed like students spent less time
[00:36:40] it seemed like students spent less time per activity than reading a page and I
[00:36:43] per activity than reading a page and I bring this up because sometimes I have
[00:36:45] bring this up because sometimes I have people who come talk to me right before
[00:36:46] people who come talk to me right before the midterm and they say I've rewatched
[00:36:48] the midterm and they say I've rewatched your lectures like three times you know
[00:36:49] your lectures like three times you know what else can I do and while I am
[00:36:51] what else can I do and while I am flattered if that they want to watch the
[00:36:53] flattered if that they want to watch the the lectures time I really highly
[00:36:55] the lectures time I really highly recommend you don't do that that instead
[00:36:57] recommend you don't do that that instead you spend time um doing problems uh or
[00:37:01] you spend time um doing problems uh or going through problems from the sessions
[00:37:02] going through problems from the sessions going through the homework going through
[00:37:03] going through the homework going through the check your understandings it's far
[00:37:05] the check your understandings it's far more effective and efficient in
[00:37:07] more effective and efficient in general so in general engage practice um
[00:37:10] general so in general engage practice um particularly forced recall where you
[00:37:12] particularly forced recall where you have to sort of think about things um uh
[00:37:15] have to sort of think about things um uh without checking the answers is shown to
[00:37:17] without checking the answers is shown to be very effective for learning um and so
[00:37:20] be very effective for learning um and so to achieve the class learning goals I
[00:37:22] to achieve the class learning goals I encourage you to spend as much time as
[00:37:24] encourage you to spend as much time as you can or the time you have available
[00:37:25] you can or the time you have available for the course on those typ type of um
[00:37:28] for the course on those typ type of um sort of directly engaging activities
[00:37:30] sort of directly engaging activities rather than more passive ones like
[00:37:32] rather than more passive ones like reading or watching yeah name
[00:37:37] reading or watching yeah name first um do you have a time frame for
[00:37:40] first um do you have a time frame for when the um problem sessions will be
[00:37:43] when the um problem sessions will be held great question we will announce
[00:37:45] held great question we will announce those um by the end of
[00:37:46] those um by the end of tomorrow and for those ones we know it's
[00:37:49] tomorrow and for those ones we know it's like impossible to coordinate schedules
[00:37:50] like impossible to coordinate schedules so um if you can't make it we encourage
[00:37:52] so um if you can't make it we encourage you to come in person but if you can't
[00:37:54] you to come in person but if you can't make it we also release all the
[00:37:55] make it we also release all the materials and the videos afterwards
[00:38:01] okay um I will highlight I guess just
[00:38:03] okay um I will highlight I guess just also on this too and I saw several
[00:38:05] also on this too and I saw several people asking about this well um just go
[00:38:08] people asking about this well um just go back to this part allow me to
[00:38:11] back to this part allow me to cover so several people mentioned that
[00:38:14] cover so several people mentioned that they were excited about having some more
[00:38:15] they were excited about having some more theoretical aspects um this class does
[00:38:18] theoretical aspects um this class does involve Theory uh it is um perhaps
[00:38:22] involve Theory uh it is um perhaps there's probably more Theory I think
[00:38:23] there's probably more Theory I think probably than the normal um machine
[00:38:25] probably than the normal um machine learning in AI classes probably a little
[00:38:26] learning in AI classes probably a little bit more and not as much as like an
[00:38:28] bit more and not as much as like an advanced seminar on Theory so normally
[00:38:31] advanced seminar on Theory so normally most problem problem sets will have like
[00:38:33] most problem problem sets will have like one Theory question um and if you're not
[00:38:36] one Theory question um and if you're not familiar with some of the sort of
[00:38:38] familiar with some of the sort of theoretical you know theoretical
[00:38:39] theoretical you know theoretical techniques um totally fine you can come
[00:38:41] techniques um totally fine you can come to problem sessions you don't have to
[00:38:42] to problem sessions you don't have to have any prior background um in doing
[00:38:45] have any prior background um in doing proofs to be able to succeed another
[00:38:47] proofs to be able to succeed another thing people asked about were um Monte
[00:38:50] thing people asked about were um Monte Carlo treer several people brought up
[00:38:52] Carlo treer several people brought up reinforcement learning from Human
[00:38:53] reinforcement learning from Human feedback we will be talking about that
[00:38:55] feedback we will be talking about that some people asked about uh multi agents
[00:38:58] some people asked about uh multi agents we're going to be thinking about Monte
[00:38:59] we're going to be thinking about Monte Carlo treesearch and other ways to have
[00:39:01] Carlo treesearch and other ways to have multiple agents that are making
[00:39:02] multiple agents that are making decisions um and a number of people said
[00:39:04] decisions um and a number of people said they wanted to get up to speed on sort
[00:39:06] they wanted to get up to speed on sort of the latest ideas in reinforcement
[00:39:07] of the latest ideas in reinforcement learning so they could read papers or do
[00:39:09] learning so they could read papers or do things in their applications and I think
[00:39:12] things in their applications and I think this was all very relevant to
[00:39:14] this was all very relevant to that um the final thing is just uh we
[00:39:18] that um the final thing is just uh we have five wonderful Tas who will be
[00:39:19] have five wonderful Tas who will be supporting the main ways to get
[00:39:21] supporting the main ways to get information about the class is to go to
[00:39:23] information about the class is to go to the website or go to Ed um we'll be
[00:39:25] the website or go to Ed um we'll be releasing our office hours by theend of
[00:39:27] releasing our office hours by theend of tomorrow and we'll start them for the
[00:39:29] tomorrow and we'll start them for the rest of the week and all of you guys are
[00:39:31] rest of the week and all of you guys are completely capable of succeeding in the
[00:39:33] completely capable of succeeding in the course and we're here to
[00:39:35] course and we're here to help
[00:39:37] help yeah yeah thank you going back to the
[00:39:39] yeah yeah thank you going back to the course topic slide yeah do some of those
[00:39:42] course topic slide yeah do some of those topics include model based approaches as
[00:39:44] topics include model based approaches as well yeah so the first part great
[00:39:46] well yeah so the first part great question so um when we first start
[00:39:49] question so um when we first start talking about here we'll talk about
[00:39:50] talking about here we'll talk about models um at the beginning and
[00:39:52] models um at the beginning and particularly when we're defining Markoff
[00:39:53] particularly when we're defining Markoff decision processes and then we will
[00:39:56] decision processes and then we will likely be talking again about them more
[00:39:57] likely be talking again about them more when we get into the offline
[00:39:59] when we get into the offline approach there's a lot of really
[00:40:00] approach there's a lot of really interesting questions about um when
[00:40:02] interesting questions about um when we're picking different there's a lot of
[00:40:04] we're picking different there's a lot of we we'll get into the fact that there's
[00:40:05] we we'll get into the fact that there's a lot of different representations you
[00:40:06] a lot of different representations you can use for reinforcement learning and
[00:40:09] can use for reinforcement learning and there's a lot of questions over which to
[00:40:10] there's a lot of questions over which to use when or when you combine them and in
[00:40:12] use when or when you combine them and in particular where do errors propagate in
[00:40:14] particular where do errors propagate in the different types of representations
[00:40:16] the different types of representations in terms of leading to error in the
[00:40:19] in terms of leading to error in the final decisions you
[00:40:20] final decisions you make but modelbased reinforcement
[00:40:22] make but modelbased reinforcement learning can certainly be a really
[00:40:23] learning can certainly be a really powerful
[00:40:25] powerful tool any other questions on the
[00:40:26] tool any other questions on the logistics
[00:40:31] all right so let's start to dive into
[00:40:33] all right so let's start to dive into the
[00:40:35] material all right we're going to start
[00:40:37] material all right we're going to start with a refresher exercise so um raise
[00:40:40] with a refresher exercise so um raise your hand if you've seen reinforcement
[00:40:41] your hand if you've seen reinforcement learning at least a little bit in the
[00:40:43] learning at least a little bit in the past okay so most people not all um if
[00:40:46] past okay so most people not all um if you haven't if everything I am about to
[00:40:48] you haven't if everything I am about to say doesn't make sense don't worry we're
[00:40:49] say doesn't make sense don't worry we're going to cover it um but I like to kind
[00:40:51] going to cover it um but I like to kind of get a a gauge in case people are like
[00:40:53] of get a a gauge in case people are like I've seen all of this before for the
[00:40:55] I've seen all of this before for the very beginning of the course um so this
[00:40:57] very beginning of the course um so this is going to be a refresh exercise we're
[00:40:59] is going to be a refresh exercise we're going to do it on Ed um I'll put the
[00:41:00] going to do it on Ed um I'll put the link up again or you can go to Ed it'll
[00:41:02] link up again or you can go to Ed it'll be the second link so here's the
[00:41:04] be the second link so here's the question we're going to think about how
[00:41:05] question we're going to think about how would we formulate a particular problem
[00:41:07] would we formulate a particular problem as a reinforcement learning problem or
[00:41:09] as a reinforcement learning problem or as a Markoff decision process so one of
[00:41:12] as a Markoff decision process so one of the first application areas to use
[00:41:14] the first application areas to use reinforcement learning for Education
[00:41:16] reinforcement learning for Education used it in roughly the following way not
[00:41:19] used it in roughly the following way not exactly the idea was that you would have
[00:41:21] exactly the idea was that you would have a student that didn't know a set of
[00:41:23] a student that didn't know a set of topics let's here just consider addition
[00:41:26] topics let's here just consider addition which we'll assume is an easier topic
[00:41:27] which we'll assume is an easier topic for people to learn and subtraction
[00:41:29] for people to learn and subtraction which we're going to assume is harder
[00:41:32] which we're going to assume is harder imagine the beginning the student
[00:41:33] imagine the beginning the student doesn't know either of these things and
[00:41:35] doesn't know either of these things and what the AI tutor agent can do is they
[00:41:37] what the AI tutor agent can do is they can provide practice problems they can
[00:41:39] can provide practice problems they can provide subtraction uh practice problems
[00:41:42] provide subtraction uh practice problems or they can provide addition practice
[00:41:44] or they can provide addition practice problems and what happens is the AI
[00:41:46] problems and what happens is the AI agent gets a reward of plus one if the
[00:41:49] agent gets a reward of plus one if the agent if the student gets the problem
[00:41:50] agent if the student gets the problem right and they get a minus one if the
[00:41:52] right and they get a minus one if the student gets the problem
[00:41:54] student gets the problem wrong and so what I'd like you to think
[00:41:56] wrong and so what I'd like you to think about here is to model is a decision
[00:41:58] about here is to model is a decision process what would like the state space
[00:42:00] process what would like the state space be the action space the reward model if
[00:42:03] be the action space the reward model if you've taken classes with markraft Sy
[00:42:05] you've taken classes with markraft Sy process before you don't remember it's
[00:42:06] process before you don't remember it's totally fine to look up you know and
[00:42:08] totally fine to look up you know and refresh your memory this is not a test
[00:42:10] refresh your memory this is not a test um I'd like you to write down sort of
[00:42:12] um I'd like you to write down sort of you know what would a Dynamics model
[00:42:13] you know what would a Dynamics model represent in this case and then in
[00:42:16] represent in this case and then in particular what would a policy to
[00:42:18] particular what would a policy to optimize the expected discounted sum of
[00:42:20] optimize the expected discounted sum of rewards do in this case for how I've set
[00:42:23] rewards do in this case for how I've set up this
[00:42:24] up this scenario so I'd like you to write down
[00:42:26] scenario so I'd like you to write down your enter them in Ed and then we're
[00:42:28] your enter them in Ed and then we're going to do some small group discussion
[00:42:31] going to do some small group discussion um in about 5
[00:42:56] minutes for
[00:43:49] and if you're not familiar with these
[00:43:50] and if you're not familiar with these particular words like State space Etc
[00:43:52] particular words like State space Etc it's still fine just to think about you
[00:43:54] it's still fine just to think about you know given what I've told you about the
[00:43:56] know given what I've told you about the reward for an agent what might happen in
[00:43:58] reward for an agent what might happen in this
[00:44:26] case e
[00:45:21] ah okay sorry you might have to switch
[00:45:23] ah okay sorry you might have to switch to
[00:45:25] bed
[00:46:25] for for
[00:47:12] all right try to enter in something it's
[00:47:14] all right try to enter in something it's okay if you're not sure and then turn to
[00:47:16] okay if you're not sure and then turn to someone near you and compare what you
[00:47:25] did
[00:47:55] e
[00:49:25] e e
[00:50:14] all right we're going to come back um
[00:50:15] all right we're going to come back um hopefully I heard a lot of really
[00:50:17] hopefully I heard a lot of really fruitful discussions um
[00:50:20] fruitful discussions um so let's see I know at least one group I
[00:50:23] so let's see I know at least one group I talked to um I had a great idea for what
[00:50:26] talked to um I had a great idea for what the St space could be do you guys want
[00:50:28] the St space could be do you guys want to share what your state space was and
[00:50:29] to share what your state space was and maybe tell your name as well oh
[00:50:34] sure the state space could be just like
[00:50:36] sure the state space could be just like a set of word pairs of like to like
[00:50:39] a set of word pairs of like to like natural numbers or any kind of numbers
[00:50:40] natural numbers or any kind of numbers of like how good the student is at
[00:50:43] of like how good the student is at addition then how good student is um at
[00:50:46] addition then how good student is um at subtraction yeah so you could imagine
[00:50:48] subtraction yeah so you could imagine something which is is at
[00:50:51] something which is is at addition and subtraction so you could
[00:50:53] addition and subtraction so you could imagine something like this where you
[00:50:55] imagine something like this where you just have like a vector pair where it's
[00:50:56] just have like a vector pair where it's like maybe their 0.9 close to Mastery
[00:50:59] like maybe their 0.9 close to Mastery for addition and like 04 close to
[00:51:02] for addition and like 04 close to Mastery for subtraction this is not the
[00:51:04] Mastery for subtraction this is not the only way you could write down there's
[00:51:06] only way you could write down there's lots of choices for the state space but
[00:51:07] lots of choices for the state space but that would certainly be one reasonable
[00:51:08] that would certainly be one reasonable one those are challenging in some ways
[00:51:12] one those are challenging in some ways because you can't directly observe them
[00:51:13] because you can't directly observe them but it's a pretty natural way to write
[00:51:15] but it's a pretty natural way to write it down um and in fact there are
[00:51:17] it down um and in fact there are commercial systems that essentially do
[00:51:18] commercial systems that essentially do that where they have like um for those
[00:51:20] that where they have like um for those of you familiar with hidden Markoff
[00:51:22] of you familiar with hidden Markoff models it's basically a hidden Markoff
[00:51:23] models it's basically a hidden Markoff model over whether someone has mastered
[00:51:25] model over whether someone has mastered something or not can we have a different
[00:51:27] something or not can we have a different type of State space that they wrote
[00:51:30] type of State space that they wrote down yeah may I talked about we
[00:51:33] down yeah may I talked about we basically wanted oh could you say your
[00:51:35] basically wanted oh could you say your name first please um the knowledge that
[00:51:37] name first please um the knowledge that the student has and also maybe the
[00:51:39] the student has and also maybe the questions that have already been asked
[00:51:40] questions that have already been asked to like capture the environment the
[00:51:42] to like capture the environment the current environment that were at um so
[00:51:44] current environment that were at um so we like I guess this is a better
[00:51:46] we like I guess this is a better representation of capturing the
[00:51:47] representation of capturing the knowledge the student has we were
[00:51:48] knowledge the student has we were thinking of also just like the history
[00:51:51] thinking of also just like the history of questions and students answers
[00:51:52] of questions and students answers whether they got it right or not um I
[00:51:55] whether they got it right or not um I guess that's harder to represent
[00:51:57] guess that's harder to represent no that's beautiful so exactly what was
[00:51:59] no that's beautiful so exactly what was that right yeah that's exactly so that
[00:52:00] that right yeah that's exactly so that was the other one I was hoping people
[00:52:01] was the other one I was hoping people might come up with which is the idea of
[00:52:03] might come up with which is the idea of this just being a history like a history
[00:52:05] this just being a history like a history of all the previous questions you've
[00:52:07] of all the previous questions you've asked given or all the questions the
[00:52:08] asked given or all the questions the robot has given the person and what
[00:52:10] robot has given the person and what they've responded um so you could
[00:52:12] they've responded um so you could imagine it's like uh
[00:52:15] imagine it's like uh observation
[00:52:17] observation question
[00:52:19] question reward dot dot dot and in fact those two
[00:52:23] reward dot dot dot and in fact those two representations here the history and how
[00:52:25] representations here the history and how the student um how good the student is
[00:52:27] the student um how good the student is can be depending on your representation
[00:52:28] can be depending on your representation be exactly isomorphic so sometimes this
[00:52:31] be exactly isomorphic so sometimes this can be um a sufficient statistic to
[00:52:33] can be um a sufficient statistic to capture that history and as was pointing
[00:52:35] capture that history and as was pointing out one of the challenges with histories
[00:52:37] out one of the challenges with histories is that they grow unboundedly so if you
[00:52:39] is that they grow unboundedly so if you want to have like your neural network be
[00:52:41] want to have like your neural network be predicting something you might be able
[00:52:42] predicting something you might be able to use something like an lstm or you
[00:52:44] to use something like an lstm or you might want to summarize the state so
[00:52:46] might want to summarize the state so those are both great ideas for what the
[00:52:48] those are both great ideas for what the um States could be there's not a right
[00:52:49] um States could be there's not a right answer both of them would be great but
[00:52:51] answer both of them would be great but there's also other ones the actions I
[00:52:53] there's also other ones the actions I heard many people share what the actions
[00:52:54] heard many people share what the actions are someone want to tell me what they in
[00:52:56] are someone want to tell me what they in there I know you guys mentioned what the
[00:52:57] there I know you guys mentioned what the action space was sure just whether you
[00:53:00] action space was sure just whether you go an addition or subtraction exactly so
[00:53:02] go an addition or subtraction exactly so these are just like what the agent can
[00:53:04] these are just like what the agent can actually do the teaching agent addition
[00:53:07] actually do the teaching agent addition question or subtraction and the reward
[00:53:11] question or subtraction and the reward model is plus one if the student gets it
[00:53:16] right I saw some questions um about what
[00:53:20] right I saw some questions um about what a Dynamics model is and um inside of the
[00:53:24] a Dynamics model is and um inside of the responses people were putting on the
[00:53:25] responses people were putting on the form what I mean by a Dynamics model
[00:53:27] form what I mean by a Dynamics model here and we'll talk a lot more about
[00:53:29] here and we'll talk a lot more about this is what happens to the state of the
[00:53:31] this is what happens to the state of the student after a question is given so in
[00:53:35] student after a question is given so in this case and I talked to some people
[00:53:37] this case and I talked to some people about this who had you know had a great
[00:53:39] about this who had you know had a great understanding of this already the idea
[00:53:41] understanding of this already the idea would be sort of how does either that
[00:53:42] would be sort of how does either that history change after you give um an a
[00:53:45] history change after you give um an a question to the student or how does this
[00:53:47] question to the student or how does this the sort of internal knowledge of the
[00:53:49] the sort of internal knowledge of the student change so the hope would be as
[00:53:51] student change so the hope would be as long as this sort of you know curriculum
[00:53:53] long as this sort of you know curriculum is vaguely reasonable that after you
[00:53:54] is vaguely reasonable that after you give the student an addition question
[00:53:56] give the student an addition question they now know more about addition you
[00:53:59] they now know more about addition you know or they're more likely to have
[00:54:00] know or they're more likely to have mastered addition so that would be sort
[00:54:03] mastered addition so that would be sort of the this idea of there being a
[00:54:04] of the this idea of there being a Dynamics process that where you know you
[00:54:06] Dynamics process that where you know you start in one state you get an action and
[00:54:09] start in one state you get an action and now you transition to a new state
[00:54:10] now you transition to a new state afterwards and we'll talk a lot more
[00:54:12] afterwards and we'll talk a lot more about that now what is the challenge
[00:54:15] about that now what is the challenge with this particular
[00:54:17] with this particular representation yeah can you say your
[00:54:18] representation yeah can you say your name first
[00:54:20] name first please um depending on your
[00:54:22] please um depending on your implementation there's a risk that the
[00:54:24] implementation there's a risk that the the agent just gives really easy
[00:54:26] the agent just gives really easy problems yeah in fact that's exactly
[00:54:29] problems yeah in fact that's exactly exactly right and in fact that's exactly
[00:54:31] exactly right and in fact that's exactly what we think will happen so we think
[00:54:32] what we think will happen so we think that um an agent that is
[00:54:35] that um an agent that is maximizing its
[00:54:38] reward should
[00:54:42] reward should only give easy
[00:54:45] only give easy questions so in this paper which I took
[00:54:47] questions so in this paper which I took the inspiration from for this example it
[00:54:50] the inspiration from for this example it was very close to this where they tried
[00:54:51] was very close to this where they tried to pick not um correctness but how long
[00:54:53] to pick not um correctness but how long it took people to do problems and so if
[00:54:56] it took people to do problems and so if the student took less time to do
[00:54:57] the student took less time to do problems um which isn't necessarily bad
[00:55:00] problems um which isn't necessarily bad in itself it might indicate some notion
[00:55:02] in itself it might indicate some notion of fluency the agent got more reward but
[00:55:04] of fluency the agent got more reward but of course what that means is that you
[00:55:06] of course what that means is that you should just give really easy questions
[00:55:07] should just give really easy questions that will take the student no time to do
[00:55:08] that will take the student no time to do because then the agent can get lots and
[00:55:10] because then the agent can get lots and lots and lots of reward um and this is
[00:55:13] lots and lots of reward um and this is probably not what the intend like the
[00:55:16] probably not what the intend like the designers of this system to try to help
[00:55:17] designers of this system to try to help students learn things intended um they
[00:55:20] students learn things intended um they probably actually wanted the students to
[00:55:22] probably actually wanted the students to learn both addition and subtraction um
[00:55:25] learn both addition and subtraction um but I bring this up because this is an
[00:55:26] but I bring this up because this is an example of what is often called reward
[00:55:28] example of what is often called reward hacking where the reward that we specify
[00:55:31] hacking where the reward that we specify does not necessarily um provide the
[00:55:34] does not necessarily um provide the behavior that we really hope to achieve
[00:55:36] behavior that we really hope to achieve and we will talk a lot more about this
[00:55:38] and we will talk a lot more about this in this case it's a fairly simple
[00:55:40] in this case it's a fairly simple example where we can see it fairly
[00:55:41] example where we can see it fairly quickly but there are a lot of cases
[00:55:44] quickly but there are a lot of cases where it's a lot more subtle to
[00:55:45] where it's a lot more subtle to understand whether or not the system
[00:55:46] understand whether or not the system really will do what you hope it will do
[00:55:49] really will do what you hope it will do um and we'll talk about that more
[00:55:50] um and we'll talk about that more throughout the
[00:55:52] throughout the course all right great so we're going to
[00:55:55] course all right great so we're going to now just start to talk about sort of
[00:55:56] now just start to talk about sort of sequential decision- making more broadly
[00:55:58] sequential decision- making more broadly and some of this will be reviewed for
[00:55:59] and some of this will be reviewed for some of you but I think it's useful to
[00:56:01] some of you but I think it's useful to go through and refresh our memories um
[00:56:03] go through and refresh our memories um so the idea in sequential decision-
[00:56:05] so the idea in sequential decision- making under uncertainty is that we're
[00:56:07] making under uncertainty is that we're going to have an agent that is taking
[00:56:10] going to have an agent that is taking decisions or actions so I'm going to use
[00:56:12] decisions or actions so I'm going to use actions and decisions
[00:56:15] actions and decisions interchangeably which are going to
[00:56:17] interchangeably which are going to interact in the world and then they're
[00:56:19] interact in the world and then they're going to get back some sort of
[00:56:20] going to get back some sort of observation and reward signal so in in
[00:56:23] observation and reward signal so in in the first example I just gave you it's
[00:56:25] the first example I just gave you it's like the agent um
[00:56:26] like the agent um you know provides a problem to the
[00:56:28] you know provides a problem to the student and then they see whether the
[00:56:30] student and then they see whether the student gets that correctly or incorrect
[00:56:32] student gets that correctly or incorrect and then they also use that information
[00:56:34] and then they also use that information to get a reward so s of getting reward
[00:56:36] to get a reward so s of getting reward and feedback and the goal in this case
[00:56:39] and feedback and the goal in this case is for the agent to select actions to
[00:56:41] is for the agent to select actions to maximize the total expected future
[00:56:43] maximize the total expected future reward meaning both the immediate reward
[00:56:45] reward meaning both the immediate reward they get now as well as the rewards
[00:56:47] they get now as well as the rewards they're going to get over time and this
[00:56:49] they're going to get over time and this generally is often going to involve
[00:56:51] generally is often going to involve balancing long-term and short-term
[00:56:53] balancing long-term and short-term rewards so there are lots and lots of
[00:56:55] rewards so there are lots and lots of examples I'll just go through a couple
[00:56:57] examples I'll just go through a couple of them just to give you a sense so one
[00:56:59] of them just to give you a sense so one is something like web advertising um in
[00:57:01] is something like web advertising um in this case you know Amazon for example
[00:57:04] this case you know Amazon for example might choose like a web ad ad to show
[00:57:06] might choose like a web ad ad to show you or a product to suggest to you they
[00:57:08] you or a product to suggest to you they might observe things like view time and
[00:57:10] might observe things like view time and whether or not you click on the ad
[00:57:12] whether or not you click on the ad whether or not you make a purchase and
[00:57:14] whether or not you make a purchase and the goal in this case could probably be
[00:57:15] the goal in this case could probably be for them to optimize either click time
[00:57:17] for them to optimize either click time or view time or
[00:57:20] or view time or Revenue in the context of something like
[00:57:22] Revenue in the context of something like robotics the control space or the
[00:57:25] robotics the control space or the decision space might be something like
[00:57:26] decision space might be something like how to move a
[00:57:28] how to move a joint um and then the feedback that the
[00:57:31] joint um and then the feedback that the the robot might get back might be
[00:57:32] the robot might get back might be something like a camera image of a
[00:57:33] something like a camera image of a kitchen and perhaps they just get a plus
[00:57:35] kitchen and perhaps they just get a plus one if there are no more dishes on the
[00:57:38] one if there are no more dishes on the counter now just quick question could
[00:57:40] counter now just quick question could this potentially be a reward hacked
[00:57:42] this potentially be a reward hacked specification I see some Smiles what
[00:57:44] specification I see some Smiles what could
[00:57:46] could happen yeah sorry robot could just push
[00:57:51] happen yeah sorry robot could just push everything off the car which I will say
[00:57:54] everything off the car which I will say um with you know it's tempting right
[00:57:56] um with you know it's tempting right you're like I'm just going to make it
[00:57:57] you're like I'm just going to make it all go away but in fact it does not
[00:57:59] all go away but in fact it does not solve the problem and now you just have
[00:58:00] solve the problem and now you just have broken dishes and food on the floor um
[00:58:02] broken dishes and food on the floor um so that would not be a good thing to do
[00:58:03] so that would not be a good thing to do so yeah this would be probably not a
[00:58:05] so yeah this would be probably not a great reward to put you probably want a
[00:58:07] great reward to put you probably want a reward more like that the dishes are
[00:58:08] reward more like that the dishes are inside of the dishwasher and finally
[00:58:10] inside of the dishwasher and finally clean so not just that they were put in
[00:58:12] clean so not just that they were put in there but actually that you ran the
[00:58:13] there but actually that you ran the dishwasher so this would be a second
[00:58:15] dishwasher so this would be a second example of of a setting another would be
[00:58:18] example of of a setting another would be something like blood pressure control
[00:58:20] something like blood pressure control where you could imagine that the agent
[00:58:22] where you could imagine that the agent gives recommendations like exercise or
[00:58:25] gives recommendations like exercise or medication the feedback is things like
[00:58:27] medication the feedback is things like blood pressure and then you would Define
[00:58:29] blood pressure and then you would Define some reward like maybe plus one if
[00:58:31] some reward like maybe plus one if you're in a healthy range else some sort
[00:58:33] you're in a healthy range else some sort of you know sloping penalty for being
[00:58:34] of you know sloping penalty for being outside of the healthy
[00:58:38] range all right so all of these are nice
[00:58:40] range all right so all of these are nice examples of the numerous ways where we
[00:58:42] examples of the numerous ways where we often try to make sequences of decisions
[00:58:44] often try to make sequences of decisions under uncertainty in general um we're
[00:58:47] under uncertainty in general um we're going to assume that we have a finite
[00:58:49] going to assume that we have a finite series of time steps so we're not going
[00:58:51] series of time steps so we're not going to be thinking about continuous time in
[00:58:52] to be thinking about continuous time in this class lots of interesting things
[00:58:54] this class lots of interesting things there we're not going to cover it what
[00:58:56] there we're not going to cover it what we're going to assume is that the agent
[00:58:57] we're going to assume is that the agent is making a series of decisions so we're
[00:58:59] is making a series of decisions so we're going to think of there being a series
[00:59:00] going to think of there being a series of time steps like you know 1 minute 2
[00:59:03] of time steps like you know 1 minute 2 minute 3 minute 4 minute the agent will
[00:59:05] minute 3 minute 4 minute the agent will take an action the world will update
[00:59:07] take an action the world will update given that action and admit an
[00:59:08] given that action and admit an observation and reward and then the
[00:59:11] observation and reward and then the agent receives that updates and then
[00:59:13] agent receives that updates and then makes another decision and we just close
[00:59:15] makes another decision and we just close this loop it's a feedback
[00:59:17] this loop it's a feedback cycle in this case as we sort of just
[00:59:19] cycle in this case as we sort of just talked about at a high level we can
[00:59:21] talked about at a high level we can think of there being histories which is
[00:59:23] think of there being histories which is sequences of past actions rewards and
[00:59:26] sequences of past actions rewards and out observations up to the present time
[00:59:29] out observations up to the present time point so the history HT would consist of
[00:59:32] point so the history HT would consist of all the previous actions of the agent
[00:59:34] all the previous actions of the agent the observations it receives and the
[00:59:36] the observations it receives and the reward it's
[00:59:37] reward it's got in general this is something you
[00:59:40] got in general this is something you could use to make decisions you could
[00:59:41] could use to make decisions you could just keep track of everything you've
[00:59:43] just keep track of everything you've experienced so far and then condition on
[00:59:45] experienced so far and then condition on that to try to make your next
[00:59:47] that to try to make your next decision but we often are going to
[00:59:49] decision but we often are going to assume that there's some sort of
[00:59:51] assume that there's some sort of sufficient statistic that we can use to
[00:59:53] sufficient statistic that we can use to summarize the history it will be much
[00:59:55] summarize the history it will be much more practic iCal in many cases yeah oh
[00:59:58] more practic iCal in many cases yeah oh sorry just make observation basically
[01:00:00] sorry just make observation basically like the
[01:00:02] hisory and what's your
[01:00:05] hisory and what's your name um uh so the observation in this
[01:00:07] name um uh so the observation in this case would be something like the
[01:00:09] case would be something like the immediate information you get back after
[01:00:10] immediate information you get back after the Last Action so in the case of the
[01:00:12] the Last Action so in the case of the student it would have been whether they
[01:00:13] student it would have been whether they get the last problem correct or not so
[01:00:15] get the last problem correct or not so just like a single time step and then
[01:00:18] just like a single time step and then the history would be everything like up
[01:00:20] the history would be everything like up to this time Point good
[01:00:24] question so in particular often to make
[01:00:26] question so in particular often to make things tractable and because often in
[01:00:28] things tractable and because often in reality it's not a terrible assumption
[01:00:30] reality it's not a terrible assumption we're going to normally make the Markoff
[01:00:32] we're going to normally make the Markoff assumption um and the idea is that we're
[01:00:34] assumption um and the idea is that we're going to try to come up with some sort
[01:00:36] going to try to come up with some sort of informative information state that is
[01:00:38] of informative information state that is a sufficient statistic of the history so
[01:00:40] a sufficient statistic of the history so we don't have to keep around all of the
[01:00:42] we don't have to keep around all of the prior history of everything the agents
[01:00:44] prior history of everything the agents ever done or seen or gotten reward
[01:00:46] ever done or seen or gotten reward for and what we say is a state St is
[01:00:49] for and what we say is a state St is Mark off if and only if the probability
[01:00:52] Mark off if and only if the probability of going to the next state given the
[01:00:55] of going to the next state given the current state in Act
[01:00:56] current state in Act is the same as if youd conditioned on
[01:00:58] is the same as if youd conditioned on the whole entire
[01:01:00] the whole entire history so another way to say this which
[01:01:03] history so another way to say this which I think is kind of a nice evocative idea
[01:01:05] I think is kind of a nice evocative idea um this is not from me it's from others
[01:01:07] um this is not from me it's from others is that the future is independent of the
[01:01:09] is that the future is independent of the past given the
[01:01:11] past given the present that means if you have a rich
[01:01:13] present that means if you have a rich representation of your current state you
[01:01:15] representation of your current state you don't have to think about the previous
[01:01:18] don't have to think about the previous history and of course in general this
[01:01:20] history and of course in general this will be true if you make S equal to
[01:01:24] will be true if you make S equal to HT but in general we're going to be
[01:01:26] HT but in general we're going to be thinking often of sort of projecting
[01:01:28] thinking often of sort of projecting down to a much smaller State space so
[01:01:30] down to a much smaller State space so for example you might say well I could
[01:01:31] for example you might say well I could think about someone's blood pressure
[01:01:33] think about someone's blood pressure from all of time but maybe it's
[01:01:35] from all of time but maybe it's sufficient just to think of their blood
[01:01:36] sufficient just to think of their blood pressure over the last like two hours in
[01:01:38] pressure over the last like two hours in order to make my next decision
[01:01:41] order to make my next decision yeah uhuh um is there a difference
[01:01:45] yeah uhuh um is there a difference between State and observation and dis
[01:01:49] between State and observation and dis great question yes in general so I'll
[01:01:50] great question yes in general so I'll give you a particular example um Atari
[01:01:52] give you a particular example um Atari um which is these video games that uh
[01:01:54] um which is these video games that uh deine learned an agent to play what
[01:01:56] deine learned an agent to play what their stayed in that case was the last
[01:01:58] their stayed in that case was the last four frames so not just the last frame
[01:02:01] four frames so not just the last frame the last four frames does may have an
[01:02:03] the last four frames does may have an idea why you might want four frames
[01:02:04] idea why you might want four frames instead of one yeah um maybe like uh so
[01:02:08] instead of one yeah um maybe like uh so you can see like if there's momentum to
[01:02:10] you can see like if there's momentum to an object already moving exactly it
[01:02:11] an object already moving exactly it gives you velocity and acceleration yeah
[01:02:14] gives you velocity and acceleration yeah so there are a number of cases where you
[01:02:16] so there are a number of cases where you might think that there are parts of the
[01:02:17] might think that there are parts of the state that really depend on temporal
[01:02:18] state that really depend on temporal differences and then in those cases
[01:02:20] differences and then in those cases you're going to want more than just the
[01:02:21] you're going to want more than just the immediate
[01:02:22] immediate State great questions
[01:02:26] State great questions all right so why is this popular um it's
[01:02:28] all right so why is this popular um it's used all the time uh it's simple it can
[01:02:31] used all the time uh it's simple it can often be satisfied as we were just
[01:02:32] often be satisfied as we were just discussing if you use some history as
[01:02:34] discussing if you use some history as part of the state um generally there are
[01:02:36] part of the state um generally there are many cases where you can just use the
[01:02:38] many cases where you can just use the most recent State not always but but
[01:02:40] most recent State not always but but many cases and it has huge implications
[01:02:43] many cases and it has huge implications for computational complexity data
[01:02:44] for computational complexity data required and resulting
[01:02:46] required and resulting performance what I mean by the resulting
[01:02:48] performance what I mean by the resulting performance is is that in many of these
[01:02:50] performance is is that in many of these cases just like in a lot of statistics
[01:02:52] cases just like in a lot of statistics and machine learning there will be
[01:02:53] and machine learning there will be trade-offs between bias and variance and
[01:02:56] trade-offs between bias and variance and so there'll be a trade-off to between
[01:02:57] so there'll be a trade-off to between using states that are really small and
[01:02:59] using states that are really small and easy for us to work with but aren't
[01:03:02] easy for us to work with but aren't really able to capture the complexity of
[01:03:03] really able to capture the complexity of the world and the applications we care
[01:03:05] the world and the applications we care about so that it might be fast to learn
[01:03:08] about so that it might be fast to learn with those sort of representations But
[01:03:09] with those sort of representations But ultimately performance is poor so
[01:03:12] ultimately performance is poor so there'll often be trade-offs with how we
[01:03:13] there'll often be trade-offs with how we actually the expressive power of our
[01:03:15] actually the expressive power of our representations versus how long it takes
[01:03:17] representations versus how long it takes us to
[01:03:20] learn right so one of the big questions
[01:03:23] learn right so one of the big questions when we talk about sequential
[01:03:24] when we talk about sequential decision-making processes is is um is
[01:03:27] decision-making processes is is um is the state Markoff and is the world
[01:03:29] the state Markoff and is the world partially
[01:03:30] partially observable so partial oh
[01:03:34] observable so partial oh yeah uh my question is that doesn't the
[01:03:36] yeah uh my question is that doesn't the mark of assumption make this reward
[01:03:39] mark of assumption make this reward attribution problem somehow like
[01:03:42] attribution problem somehow like harder why all right good question um
[01:03:45] harder why all right good question um well I don't know I guess you could
[01:03:46] well I don't know I guess you could imagine it might make it easier or
[01:03:47] imagine it might make it easier or harder there's still the question of you
[01:03:50] harder there's still the question of you might only get periodic rewards and you
[01:03:53] might only get periodic rewards and you still would have to figure out which
[01:03:54] still would have to figure out which decisions caused you to get to a state
[01:03:56] decisions caused you to get to a state where you got those
[01:03:59] rewards yeah so let me think of it so
[01:04:01] rewards yeah so let me think of it so you might have a case where the reward
[01:04:03] you might have a case where the reward might be a function of your current
[01:04:05] might be a function of your current state um yeah uh let me think if I can
[01:04:09] state um yeah uh let me think if I can think of a um a good example okay so
[01:04:11] think of a um a good example okay so let's say maybe um you get a you want to
[01:04:16] let's say maybe um you get a you want to run a marathon and you get a plus 100 if
[01:04:18] run a marathon and you get a plus 100 if you make it Boston Marathon is a
[01:04:20] you make it Boston Marathon is a competitive Marathon to get into so you
[01:04:22] competitive Marathon to get into so you you get a plus 100 if you can qualify
[01:04:24] you get a plus 100 if you can qualify for Boston and you do a lot of different
[01:04:26] for Boston and you do a lot of different things in your training regime you like
[01:04:28] things in your training regime you like eat healthy and you sleep and you train
[01:04:30] eat healthy and you sleep and you train and then um and you get zero reward for
[01:04:32] and then um and you get zero reward for any of that and then on like the day of
[01:04:35] any of that and then on like the day of your race you see if you qualify for
[01:04:37] your race you see if you qualify for Boston so your state only like your your
[01:04:40] Boston so your state only like your your reward for getting into Boston only
[01:04:42] reward for getting into Boston only depends on that current state but you
[01:04:43] depends on that current state but you don't know which of those decisions was
[01:04:45] don't know which of those decisions was it that you ate well was it that you
[01:04:46] it that you ate well was it that you slept was it that you trained every week
[01:04:48] slept was it that you trained every week for 17 weeks caused you to get to the
[01:04:51] for 17 weeks caused you to get to the state in which you qualified for Boston
[01:04:54] state in which you qualified for Boston um and so that's independent of the
[01:04:55] um and so that's independent of the market assumption in that case because
[01:04:57] market assumption in that case because you still have the question of what
[01:04:59] you still have the question of what series of decisions allowed you to get
[01:05:00] series of decisions allowed you to get to a state that achieved High reward
[01:05:04] to a state that achieved High reward great
[01:05:05] great question so another thing is whether the
[01:05:07] question so another thing is whether the world is partially observable we will
[01:05:09] world is partially observable we will mostly not be talking about this in this
[01:05:10] mostly not be talking about this in this class Michael coocher has a great class
[01:05:13] class Michael coocher has a great class where he talks about this a lot but this
[01:05:15] where he talks about this a lot but this does relate to the case we talked about
[01:05:17] does relate to the case we talked about with students so for students one way
[01:05:20] with students so for students one way you could think about that is that
[01:05:21] you could think about that is that there's some latent state that you can't
[01:05:22] there's some latent state that you can't directly access which is whether or not
[01:05:24] directly access which is whether or not they know addition or they know
[01:05:25] they know addition or they know subtraction but you get noisy
[01:05:27] subtraction but you get noisy observations when they do problems where
[01:05:29] observations when they do problems where they get it right or get it wrong and
[01:05:32] they get it right or get it wrong and the reason it's noisy is because you
[01:05:33] the reason it's noisy is because you know all of us make mistakes on addition
[01:05:35] know all of us make mistakes on addition sometimes whereas I have complete faith
[01:05:37] sometimes whereas I have complete faith that everyone here actually knows how to
[01:05:38] that everyone here actually knows how to do addition um and sometimes you might
[01:05:40] do addition um and sometimes you might guess right even if you don't know it so
[01:05:42] guess right even if you don't know it so the idea is that it's latent you don't
[01:05:43] the idea is that it's latent you don't directly get to observe it um this comes
[01:05:46] directly get to observe it um this comes up uh in a lot of Robotics problems too
[01:05:49] up uh in a lot of Robotics problems too so I'll just give a quick example here
[01:05:51] so I'll just give a quick example here if you have a robot that uses a Laser
[01:05:54] if you have a robot that uses a Laser Rangefinder to figure out out these
[01:05:56] Rangefinder to figure out out these little arrows or
[01:05:57] little arrows or lasers to figure out its environment so
[01:06:02] lasers to figure out its environment so it could have like 180° of laser range
[01:06:04] it could have like 180° of laser range finders and what it's getting back is
[01:06:05] finders and what it's getting back is just the distance in all these different
[01:06:07] just the distance in all these different angles till where it hits a wall so as
[01:06:10] angles till where it hits a wall so as you can imagine many rooms would look
[01:06:12] you can imagine many rooms would look identical so any room that has like kind
[01:06:14] identical so any room that has like kind of the same dimensions would look
[01:06:15] of the same dimensions would look identical to that robot and it wouldn't
[01:06:17] identical to that robot and it wouldn't be able to tell is it on the third floor
[01:06:19] be able to tell is it on the third floor or the second floor so that would be a
[01:06:21] or the second floor so that would be a partially observable case where it can't
[01:06:23] partially observable case where it can't uniquely identify its state based on
[01:06:24] uniquely identify its state based on this observation
[01:06:26] this observation so we won't talk too much about about
[01:06:28] so we won't talk too much about about that but it's important to know about
[01:06:29] that but it's important to know about another thing is whether the Dynamics
[01:06:30] another thing is whether the Dynamics are deterministic or
[01:06:32] are deterministic or stochastic so there are many cases where
[01:06:34] stochastic so there are many cases where things are close to deterministic like
[01:06:36] things are close to deterministic like if I um put down a piece on a a go board
[01:06:40] if I um put down a piece on a a go board it goes there but there are other things
[01:06:42] it goes there but there are other things that we often treat as stochastic like
[01:06:44] that we often treat as stochastic like when I flip a coin I don't know whether
[01:06:45] when I flip a coin I don't know whether it's going to be heads or tails so
[01:06:47] it's going to be heads or tails so that'll be an important decision and
[01:06:49] that'll be an important decision and then the final thing is whether the
[01:06:51] then the final thing is whether the actions influence only immediate reward
[01:06:53] actions influence only immediate reward um or reward in next state
[01:06:57] so as an example of this you might
[01:06:59] so as an example of this you might imagine if you were making a policy for
[01:07:01] imagine if you were making a policy for what ad to show to people and you just
[01:07:03] what ad to show to people and you just imagine for each person coming onto the
[01:07:05] imagine for each person coming onto the web you just show them like onto your
[01:07:07] web you just show them like onto your website you show an ad and then they go
[01:07:08] website you show an ad and then they go away and they either buy something or
[01:07:09] away and they either buy something or they don't um a bandit would be a case
[01:07:12] they don't um a bandit would be a case where like you just have bless you you
[01:07:14] where like you just have bless you you have a series of customers coming in and
[01:07:16] have a series of customers coming in and so whether or not um I showed a
[01:07:20] so whether or not um I showed a particular ad and he clicks on it or not
[01:07:22] particular ad and he clicks on it or not does not impact whether or not Ellen
[01:07:24] does not impact whether or not Ellen comes along and and leges an ad so
[01:07:27] comes along and and leges an ad so that's a case where you it impacts your
[01:07:29] that's a case where you it impacts your immediate reward but not the next state
[01:07:32] immediate reward but not the next state we can talk more about that all right
[01:07:34] we can talk more about that all right let's think about a particular sort of
[01:07:36] let's think about a particular sort of running example we'll think of a Mars
[01:07:38] running example we'll think of a Mars Rover so Mars Rover is a markof decision
[01:07:40] Rover so Mars Rover is a markof decision process we imagine that Mars is really
[01:07:42] process we imagine that Mars is really small we only have seven places in Mars
[01:07:45] small we only have seven places in Mars so in this case we would have the state
[01:07:48] so in this case we would have the state is the location of the Rover which is
[01:07:50] is the location of the Rover which is one of seven discrete locations we could
[01:07:53] one of seven discrete locations we could have actions called try left and try
[01:07:54] have actions called try left and try right meaning that our Rover is not
[01:07:56] right meaning that our Rover is not perfect so sometimes it tries go a
[01:07:58] perfect so sometimes it tries go a direction and it doesn't succeed and
[01:08:00] direction and it doesn't succeed and let's imagine that we have rewards which
[01:08:02] let's imagine that we have rewards which is there's some interesting field sites
[01:08:05] is there's some interesting field sites and so if you spend time over here you
[01:08:07] and so if you spend time over here you get a plus one and you have spend over
[01:08:09] get a plus one and you have spend over here you get a plus 10 and else you get
[01:08:11] here you get a plus 10 and else you get zero
[01:08:12] zero reward so this would be a particular
[01:08:14] reward so this would be a particular case where we could think of there being
[01:08:15] case where we could think of there being these dates and these actions um and
[01:08:21] rewards so when we think of um a markof
[01:08:23] rewards so when we think of um a markof decision process we think of there being
[01:08:26] decision process we think of there being a Dynamics um and a reward model so in
[01:08:30] a Dynamics um and a reward model so in particular the Dynamics model is going
[01:08:32] particular the Dynamics model is going to tell us how the state evolves as we
[01:08:35] to tell us how the state evolves as we make decisions we will not always have
[01:08:37] make decisions we will not always have direct access to this but the idea is
[01:08:39] direct access to this but the idea is that in the world there is some Dynamics
[01:08:41] that in the world there is some Dynamics process and things are changing as we
[01:08:43] process and things are changing as we make
[01:08:44] make decisions so in particular we generally
[01:08:47] decisions so in particular we generally want to allow for stochastic systems
[01:08:49] want to allow for stochastic systems meaning that given we're currently in a
[01:08:51] meaning that given we're currently in a state and we take a particular action
[01:08:53] state and we take a particular action what is the distribution over our next
[01:08:55] what is the distribution over our next states that we might
[01:08:56] states that we might reach so for example I'm that Mars Rover
[01:08:59] reach so for example I'm that Mars Rover and um I'm going to try to go to the
[01:09:01] and um I'm going to try to go to the right it might be that I can go to the
[01:09:04] right it might be that I can go to the right with like 50% probability but I'm
[01:09:07] right with like 50% probability but I'm I'm not a very accurate Rover and so 50%
[01:09:09] I'm not a very accurate Rover and so 50% of the time I go to the left or maybe I
[01:09:11] of the time I go to the left or maybe I stay in the same
[01:09:12] stay in the same location so this Dynamics model just
[01:09:15] location so this Dynamics model just specifies what actually the distribution
[01:09:17] specifies what actually the distribution of outcomes that can happen in the world
[01:09:18] of outcomes that can happen in the world when I make a
[01:09:19] when I make a decision the reward model predicts the
[01:09:22] decision the reward model predicts the immediate reward which is if I'm in this
[01:09:24] immediate reward which is if I'm in this state and I take the action what is my
[01:09:26] state and I take the action what is my expected
[01:09:27] expected reward I want to highlight here that
[01:09:29] reward I want to highlight here that there are different conventions you
[01:09:31] there are different conventions you could have the reward be a function only
[01:09:34] could have the reward be a function only of the current state excuse me it could
[01:09:36] of the current state excuse me it could be a function of the state and the
[01:09:37] be a function of the state and the action you take or it could be a
[01:09:39] action you take or it could be a function of the state the action you
[01:09:40] function of the state the action you take and the next day you
[01:09:42] take and the next day you reach you'll see all of these
[01:09:45] reach you'll see all of these conventions in reinforcement learning
[01:09:46] conventions in reinforcement learning papers probably the most common one is
[01:09:49] papers probably the most common one is this um but we'll try just to be
[01:09:51] this um but we'll try just to be specific whenever we're using it so it's
[01:09:53] specific whenever we're using it so it's clear and you can always ask me or ask
[01:09:55] clear and you can always ask me or ask any the Tas if it's not
[01:09:57] any the Tas if it's not clear bless you so let's think about
[01:10:00] clear bless you so let's think about sort of what a stochastic Mars Rover
[01:10:02] sort of what a stochastic Mars Rover model would be so I've written down a
[01:10:06] model would be so I've written down a particular choice for the reward and
[01:10:09] particular choice for the reward and let's imagine that part of the Dynamics
[01:10:11] let's imagine that part of the Dynamics model is the following which is if I
[01:10:13] model is the following which is if I start in state S1 and I try to go to the
[01:10:16] start in state S1 and I try to go to the right then I have some probability of
[01:10:18] right then I have some probability of going to S2 else I have some probability
[01:10:21] going to S2 else I have some probability of staying
[01:10:23] of staying here what I want to clear about here and
[01:10:26] here what I want to clear about here and this relates to the question before
[01:10:27] this relates to the question before about U models is that this is like the
[01:10:30] about U models is that this is like the agent's idea of how the world works it
[01:10:33] agent's idea of how the world works it doesn't have to be how the world
[01:10:34] doesn't have to be how the world actually works so what I told you in the
[01:10:37] actually works so what I told you in the previous slides is that imagine that in
[01:10:39] previous slides is that imagine that in this world in reality this gives you
[01:10:42] this world in reality this gives you plus one and this gives you plus 10 in
[01:10:44] plus one and this gives you plus 10 in terms of the reward that's how the world
[01:10:46] terms of the reward that's how the world actually works but the agent might have
[01:10:48] actually works but the agent might have the wrong model of how the world works
[01:10:50] the wrong model of how the world works because it only learns about the world
[01:10:52] because it only learns about the world through its experiences or it just might
[01:10:54] through its experiences or it just might have a bad model
[01:10:56] have a bad model so this is an example of sort of like a
[01:10:58] so this is an example of sort of like a model-based Markov system where we where
[01:11:00] model-based Markov system where we where the agent would have a particular
[01:11:02] the agent would have a particular representation of the Dynamics model and
[01:11:04] representation of the Dynamics model and a particular assumption over the how the
[01:11:06] a particular assumption over the how the rewards
[01:11:10] work in these settings we have a policy
[01:11:13] work in these settings we have a policy a decision policy is just going to be a
[01:11:14] a decision policy is just going to be a mapping from states to actions it's like
[01:11:17] mapping from states to actions it's like an if then
[01:11:19] an if then table if it's deterministic we just have
[01:11:21] table if it's deterministic we just have a single action that we would take in a
[01:11:23] a single action that we would take in a particular state like maybe we always
[01:11:26] particular state like maybe we always show this one ad to a particular
[01:11:28] show this one ad to a particular customer or we could have a stochastic
[01:11:31] customer or we could have a stochastic policy where we randomize this so this
[01:11:33] policy where we randomize this so this would be something like oh when this
[01:11:35] would be something like oh when this customer shows up I show you know a
[01:11:37] customer shows up I show you know a vacation ad or I show a board game ad
[01:11:39] vacation ad or I show a board game ad with you know 90% probability versus
[01:11:42] with you know 90% probability versus 10% both types of policies are really
[01:11:45] 10% both types of policies are really common and it can depend in part what
[01:11:46] common and it can depend in part what sort of domain you're in and whether
[01:11:48] sort of domain you're in and whether you're trying to learn from that
[01:11:51] you're trying to learn from that experience okay so let's see what that
[01:11:53] experience okay so let's see what that would look like in this case so for Mars
[01:11:56] would look like in this case so for Mars rer you could say that no matter where
[01:11:58] rer you could say that no matter where it is it always just tries to go
[01:12:03] right so that would just be one example
[01:12:05] right so that would just be one example of a policy you could have and it just
[01:12:08] of a policy you could have and it just requires you to specify for every single
[01:12:10] requires you to specify for every single state what is the action you would take
[01:12:11] state what is the action you would take or what is the distribution over actions
[01:12:13] or what is the distribution over actions you would
[01:12:16] take so in this sort of setting we're
[01:12:18] take so in this sort of setting we're normally interested in two M oh yeah
[01:12:21] normally interested in two M oh yeah yeah question so it's like making like
[01:12:24] yeah question so it's like making like decisions based on the state that it's
[01:12:25] decisions based on the state that it's in can it like learn to like switch from
[01:12:28] in can it like learn to like switch from like different types of policy so not
[01:12:31] like different types of policy so not just like different actions ver on the
[01:12:33] just like different actions ver on the state but also switch to like checking
[01:12:35] state but also switch to like checking the past State the future state in the
[01:12:37] the past State the future state in the same way like in deep learning it like
[01:12:39] same way like in deep learning it like tries a bunch of different functions can
[01:12:41] tries a bunch of different functions can it do that or can it not do that remind
[01:12:44] it do that or can it not do that remind me your
[01:12:46] me your name um yeah so great question it will
[01:12:49] name um yeah so great question it will in general in general when we're
[01:12:50] in general in general when we're learning we're it'll change its policy a
[01:12:52] learning we're it'll change its policy a lot over time so it might start with a
[01:12:54] lot over time so it might start with a particular policy and then over time it
[01:12:56] particular policy and then over time it will explore lots of different policies
[01:12:57] will explore lots of different policies in trying to search for something that's
[01:12:59] in trying to search for something that's good that's a great question and that
[01:13:01] good that's a great question and that relates to what I was just putting here
[01:13:03] relates to what I was just putting here which is two of the central questions
[01:13:05] which is two of the central questions we're going to talk a lot about
[01:13:06] we're going to talk a lot about particularly at the beginning is
[01:13:07] particularly at the beginning is evaluation and control evaluation says
[01:13:10] evaluation and control evaluation says someone gives you a fixed policy and you
[01:13:12] someone gives you a fixed policy and you want to know how good it is like maybe
[01:13:14] want to know how good it is like maybe your boss says hey I think this is the
[01:13:16] your boss says hey I think this is the right way to advertise to customers and
[01:13:18] right way to advertise to customers and we're going to make a lot of money and
[01:13:19] we're going to make a lot of money and you go out and you just deploy that
[01:13:21] you go out and you just deploy that particular decision policy and you see
[01:13:23] particular decision policy and you see how much money you make so that would be
[01:13:25] how much money you make so that would be evaluation control is you actually want
[01:13:27] evaluation control is you actually want to find the best policy and so in
[01:13:30] to find the best policy and so in general to actually find the best policy
[01:13:32] general to actually find the best policy we're going to have to do a lot of trial
[01:13:34] we're going to have to do a lot of trial and error and we want to do that in a
[01:13:35] and error and we want to do that in a strategic efficient way so we can
[01:13:37] strategic efficient way so we can quickly learn what that good policy
[01:13:41] is so in general we're going to be
[01:13:43] is so in general we're going to be talking about things I just want to
[01:13:44] talking about things I just want to highlight we're going to sort of build
[01:13:45] highlight we're going to sort of build up in complexity in terms of the type of
[01:13:47] up in complexity in terms of the type of problems we're talking about so we're
[01:13:50] problems we're talking about so we're going to be thinking about both like
[01:13:52] going to be thinking about both like planning and control and sort of
[01:13:54] planning and control and sort of thinking about
[01:13:56] thinking about uh how complicated these spaces
[01:13:59] uh how complicated these spaces are okay so we're going to think about
[01:14:03] are okay so we're going to think about evaluation and control because
[01:14:05] evaluation and control because evaluation is often a subpart of doing
[01:14:07] evaluation is often a subpart of doing control if you know how good a policy is
[01:14:09] control if you know how good a policy is you may be able to improve it and then
[01:14:12] you may be able to improve it and then we're going to talk about tabular and
[01:14:14] we're going to talk about tabular and function approximation
[01:14:15] function approximation methods because we're going to want to
[01:14:17] methods because we're going to want to be able to solve really large problems
[01:14:19] be able to solve really large problems and then we're going to talk about both
[01:14:24] planning and
[01:14:25] planning and learning in planning we're going to
[01:14:27] learning in planning we're going to assume someone gives us that Dynamics
[01:14:30] assume someone gives us that Dynamics model and that reward model and the
[01:14:32] model and that reward model and the state in action space and we're just
[01:14:34] state in action space and we're just going to try to find a really good
[01:14:35] going to try to find a really good policy and in learning we're going to
[01:14:37] policy and in learning we're going to actually have to control the decisions
[01:14:38] actually have to control the decisions we make to give us information that
[01:14:40] we make to give us information that allows us to identify an optimal
[01:14:44] policy all right so what we're going to
[01:14:47] policy all right so what we're going to start with is sort of the simplest of
[01:14:48] start with is sort of the simplest of this settings which we're going to
[01:14:49] this settings which we're going to assume that we have a finite set of
[01:14:51] assume that we have a finite set of states and actions we're given models of
[01:14:53] states and actions we're given models of the world meaning someone like writes
[01:14:54] the world meaning someone like writes down down for us what those look like
[01:14:57] down down for us what those look like and we want to evaluate the performance
[01:14:59] and we want to evaluate the performance of the best decision
[01:15:00] of the best decision policy and then compute the optimal
[01:15:02] policy and then compute the optimal policy and we can think of this really
[01:15:04] policy and we can think of this really as AI
[01:15:06] as AI planning okay so to think about how this
[01:15:09] planning okay so to think about how this works we're going to start with Markoff
[01:15:11] works we're going to start with Markoff processes and then build up to mdps and
[01:15:13] processes and then build up to mdps and this is relevant because it turns out
[01:15:15] this is relevant because it turns out you can think of evaluation as basically
[01:15:17] you can think of evaluation as basically being a Markoff reward
[01:15:20] being a Markoff reward process okay so how does a markof chain
[01:15:22] process okay so how does a markof chain work and just raise your hand if you've
[01:15:23] work and just raise your hand if you've seen markof chains before
[01:15:25] seen markof chains before awesome okay so most people have which
[01:15:27] awesome okay so most people have which is great so this is a memor random
[01:15:29] is great so this is a memor random process there's no rewards yet um
[01:15:32] process there's no rewards yet um there's a finite set of states in this
[01:15:33] there's a finite set of states in this case and we have a Dynamics model and if
[01:15:36] case and we have a Dynamics model and if it's just a finite set of States we can
[01:15:37] it's just a finite set of States we can just write this down as a
[01:15:39] just write this down as a matrix okay just says what's the
[01:15:42] matrix okay just says what's the probability going to the next state
[01:15:43] probability going to the next state given the previous state and so you
[01:15:46] given the previous state and so you could just have this say in our this
[01:15:48] could just have this say in our this would be a Markoff chain transition
[01:15:49] would be a Markoff chain transition Matrix for our Mars rer
[01:15:53] Matrix for our Mars rer case and if you want to to get an
[01:15:55] case and if you want to to get an episode you just sample so let's say
[01:15:56] episode you just sample so let's say you're you always touchdown as dat S4
[01:15:58] you're you always touchdown as dat S4 you just sample episodes from that
[01:16:00] you just sample episodes from that particular chain yeah um all rows and
[01:16:03] particular chain yeah um all rows and columns you add to one all of the um
[01:16:07] columns you add to one all of the um what's your name yeah so all of the rows
[01:16:11] what's your name yeah so all of the rows have to sum to
[01:16:16] one then is it coincidence that column
[01:16:19] one then is it coincidence that column yeah okay yeah I was thinking just now
[01:16:22] yeah okay yeah I was thinking just now that I should have changed that it's a
[01:16:23] that I should have changed that it's a good question yeah cuz and we'll see
[01:16:25] good question yeah cuz and we'll see also why why that's important later okay
[01:16:28] also why why that's important later okay in a Markoff reward process it's a
[01:16:29] in a Markoff reward process it's a Markoff chain plus rewards so same as
[01:16:33] Markoff chain plus rewards so same as before but now we have a reward function
[01:16:35] before but now we have a reward function that tells us how good each of those
[01:16:36] that tells us how good each of those states are okay and we're also going to
[01:16:38] states are okay and we're also going to have a discount factor and I'll talk
[01:16:40] have a discount factor and I'll talk about that in a
[01:16:41] about that in a second we still have no actions um and
[01:16:44] second we still have no actions um and we can express r as a
[01:16:47] we can express r as a vector so in this we could imagine our
[01:16:50] vector so in this we could imagine our markof for word process where we have a
[01:16:52] markof for word process where we have a plus one and S1 10 in S7 so+ one plus
[01:16:57] plus one and S1 10 in S7 so+ one plus and zero in all other
[01:16:59] and zero in all other states okay in this case this is where
[01:17:02] states okay in this case this is where we start to see the ideas that are going
[01:17:04] we start to see the ideas that are going to be really useful for decision
[01:17:05] to be really useful for decision processes which is we can start to think
[01:17:06] processes which is we can start to think about how good particular trajectories
[01:17:08] about how good particular trajectories are so we're going to have a horizon and
[01:17:11] are so we're going to have a horizon and you're going to see this in your
[01:17:12] you're going to see this in your homework too which is the number of time
[01:17:13] homework too which is the number of time steps in each episode it could be
[01:17:15] steps in each episode it could be infinite or it could be finite it's like
[01:17:18] infinite or it could be finite it's like basically how many time steps do you get
[01:17:19] basically how many time steps do you get to make
[01:17:20] to make decisions and the return which we're
[01:17:23] decisions and the return which we're going to call GT is just going to be the
[01:17:26] going to call GT is just going to be the discounted sum of rewards from the time
[01:17:28] discounted sum of rewards from the time step current time step to the end of The
[01:17:30] step current time step to the end of The Horizon and a value function in this
[01:17:33] Horizon and a value function in this case is just going to be the expected
[01:17:36] case is just going to be the expected return in general this is not going to
[01:17:38] return in general this is not going to be the same as the actual return unless
[01:17:40] be the same as the actual return unless you just have a deterministic process
[01:17:43] you just have a deterministic process because the idea is that you're going to
[01:17:44] because the idea is that you're going to have stochasticity in the trajectories
[01:17:46] have stochasticity in the trajectories you reach and because of that you're
[01:17:48] you reach and because of that you're going to get different
[01:17:51] rewards so you might wonder if you
[01:17:54] rewards so you might wonder if you haven't seen it for why do we have this
[01:17:55] haven't seen it for why do we have this discount Factor thing so we're of
[01:17:58] discount Factor thing so we're of weighing um earlier rewards more than
[01:18:00] weighing um earlier rewards more than later rewards one is that it's just
[01:18:02] later rewards one is that it's just mathematically really convenient it's
[01:18:04] mathematically really convenient it's going to help us not sum to Infinity
[01:18:05] going to help us not sum to Infinity particularly if we have infinite number
[01:18:07] particularly if we have infinite number of time steps we can make decisions um
[01:18:10] of time steps we can make decisions um and it turns out humans often act as if
[01:18:12] and it turns out humans often act as if there is a discount Factor like often we
[01:18:14] there is a discount Factor like often we sort of implicitly weigh future rewards
[01:18:17] sort of implicitly weigh future rewards less than immediate rewards um and this
[01:18:20] less than immediate rewards um and this is true for organizations
[01:18:21] is true for organizations too and if the epsod lengths are always
[01:18:24] too and if the epsod lengths are always finite you can always bless you use
[01:18:26] finite you can always bless you use gamma equal one meaning you don't have
[01:18:28] gamma equal one meaning you don't have to make a large
[01:18:30] to make a large discount but when you have infinite
[01:18:32] discount but when you have infinite Horizons it's generally important to
[01:18:34] Horizons it's generally important to make this less than one so your rewards
[01:18:35] make this less than one so your rewards don't blow up part of that is because
[01:18:38] don't blow up part of that is because it's really hard to compare Infinities
[01:18:40] it's really hard to compare Infinities so it's hard to say like this reward
[01:18:41] so it's hard to say like this reward that this policy that has infinite
[01:18:43] that this policy that has infinite reward is better than this other policy
[01:18:44] reward is better than this other policy that has infinite reward whereas you can
[01:18:46] that has infinite reward whereas you can keep everything bounded if you have a
[01:18:48] keep everything bounded if you have a gamma less than
[01:18:49] gamma less than one all right next time we will start to
[01:18:53] one all right next time we will start to talk about how we actually can comp the
[01:18:55] talk about how we actually can comp the value of these types of markof reward
[01:18:57] value of these types of markof reward processes and then start to connect it
[01:18:58] processes and then start to connect it to decision processes I'll see you on
[01:19:00] to decision processes I'll see you on Wednesday thanks

Stanford CS234 Reinforcement Learning I Tabular MDP Planning I 2024 I Lecture 2

Source: https://www.youtube.com/watch?v=gHdsUUGcBC0

---

Transcript

[00:00:05] hi everybody welcome back this is a
[00:00:07] hi everybody welcome back this is a lecture two from reinforcement learning
[00:00:09] lecture two from reinforcement learning um we're going to start with a refresh
[00:00:11] um we're going to start with a refresh your understanding again these are just
[00:00:13] your understanding again these are just a sort of a quick way to check your
[00:00:15] a sort of a quick way to check your conceptual understanding from the most
[00:00:17] conceptual understanding from the most recent lectures or occasionally we'll go
[00:00:19] recent lectures or occasionally we'll go back a little bit to do this you just
[00:00:21] back a little bit to do this you just need to log in to Ed everybody should be
[00:00:23] need to log in to Ed everybody should be added to Ed if you're not just send us
[00:00:25] added to Ed if you're not just send us an email to our mailing list um so if
[00:00:28] an email to our mailing list um so if you go to Ed please follow the steps
[00:00:29] you go to Ed please follow the steps given to log in first before you click
[00:00:31] given to log in first before you click the links so if you follow those steps
[00:00:33] the links so if you follow those steps and then you're logged in with your sun
[00:00:34] and then you're logged in with your sun ID then when you click on the poll links
[00:00:37] ID then when you click on the poll links it should just take you right there and
[00:00:38] it should just take you right there and it'll just log all your responses if
[00:00:41] it'll just log all your responses if you're curious about how we use these
[00:00:42] you're curious about how we use these for participation points um you can just
[00:00:44] for participation points um you can just go to the website to see how we
[00:00:45] go to the website to see how we calculate it I think it's um we use just
[00:00:48] calculate it I think it's um we use just a percentage of these if you do a
[00:00:50] a percentage of these if you do a sufficient percentage then you get full
[00:00:52] sufficient percentage then you get full participation points it's
[00:00:56] optional all right so we're going to
[00:00:57] optional all right so we're going to start with this today the question is in
[00:00:59] start with this today the question is in markup decision process says a large
[00:01:00] markup decision process says a large discount Factor gamma means that
[00:01:02] discount Factor gamma means that short-term rewards are much more
[00:01:03] short-term rewards are much more influential than long-term rewards um
[00:01:06] influential than long-term rewards um and then a second question to start
[00:01:07] and then a second question to start thinking about is in general so last
[00:01:10] thinking about is in general so last time we started talking about sequential
[00:01:12] time we started talking about sequential decision- making under uncertainty and
[00:01:14] decision- making under uncertainty and one of the things we often would like in
[00:01:16] one of the things we often would like in Real World Systems is monotonic
[00:01:18] Real World Systems is monotonic Improvement meaning that if we get more
[00:01:20] Improvement meaning that if we get more data or we get more computation we know
[00:01:23] data or we get more computation we know that the system is going to be better
[00:01:25] that the system is going to be better make in our case better decisions than
[00:01:27] make in our case better decisions than it could if it had less computation or
[00:01:29] it could if it had less computation or less data
[00:01:30] less data and so the question um that I'm posing
[00:01:33] and so the question um that I'm posing to you now and that we're going to
[00:01:34] to you now and that we're going to discuss today is is it possible to
[00:01:36] discuss today is is it possible to construct algorithms for computing
[00:01:38] construct algorithms for computing decision policies so that we can
[00:01:40] decision policies so that we can guarantee with additional computation
[00:01:42] guarantee with additional computation which we could also think of often as
[00:01:43] which we could also think of often as iteration um that we're going to
[00:01:45] iteration um that we're going to monotonically improve the decision
[00:01:48] monotonically improve the decision policy and you can start to think about
[00:01:50] policy and you can start to think about if you're already aware of any
[00:01:51] if you're already aware of any algorithms that might have that property
[00:01:53] algorithms that might have that property if you think it's impossible or if you
[00:01:54] if you think it's impossible or if you think if it's true do you think that all
[00:01:56] think if it's true do you think that all algorithms would satisfy that okay
[00:01:59] algorithms would satisfy that okay that's not for the poll that's just to
[00:02:00] that's not for the poll that's just to start thinking about and we'll come back
[00:02:02] start thinking about and we'll come back to it
[00:02:04] to it later all right so I'll just give you
[00:02:06] later all right so I'll just give you another minute or two to do this refresh
[00:02:08] another minute or two to do this refresh your understanding it's just a quick one
[00:02:10] your understanding it's just a quick one and then we'll
[00:02:24] go and again these are not assessment
[00:02:26] go and again these are not assessment questions so you're welcome to look back
[00:02:27] questions so you're welcome to look back on lecture slides from last time you're
[00:02:30] on lecture slides from last time you're also welcome to talk to anybody right
[00:02:31] also welcome to talk to anybody right next to you
[00:03:08] all right it looks like we actually have
[00:03:09] all right it looks like we actually have um maybe a 23 1/3 split on this question
[00:03:14] um maybe a 23 1/3 split on this question um the correct answer is false does
[00:03:16] um the correct answer is false does somebody want to say why it's
[00:03:20] false yeah and remind me your
[00:03:23] false yeah and remind me your name yeah I think because you multiply
[00:03:25] name yeah I think because you multiply the longer term rewards by uh the gamma
[00:03:28] the longer term rewards by uh the gamma so a large gamma means that the long
[00:03:30] so a large gamma means that the long rewards are we get decently that's right
[00:03:32] rewards are we get decently that's right so if you have exactly said so if gamma
[00:03:35] so if you have exactly said so if gamma was one you would care about short-term
[00:03:36] was one you would care about short-term rewards exactly the same as long-term
[00:03:38] rewards exactly the same as long-term rewards in general if gamma was Zero you
[00:03:40] rewards in general if gamma was Zero you wouldn't care about long-term rewards at
[00:03:42] wouldn't care about long-term rewards at all you'd be entirely myopic but as
[00:03:44] all you'd be entirely myopic but as gamma gets closer to one it's sort of a
[00:03:46] gamma gets closer to one it's sort of a relatively waiting more of longer
[00:03:48] relatively waiting more of longer rewards than you would
[00:03:51] otherwise
[00:03:54] otherwise great all right and yes as I said H well
[00:03:56] great all right and yes as I said H well we'll get more into the conceptual
[00:03:58] we'll get more into the conceptual question later um the other thing that I
[00:04:00] question later um the other thing that I wanted to clarify I saw there was some
[00:04:02] wanted to clarify I saw there was some questions on this last time as well as
[00:04:04] questions on this last time as well as after class as well as on Ed is sort of
[00:04:06] after class as well as on Ed is sort of I had mentioned when I was uh making
[00:04:08] I had mentioned when I was uh making distinguishment between reinforcement
[00:04:10] distinguishment between reinforcement learning and other forms of AI machine
[00:04:12] learning and other forms of AI machine learning this notion of optimization but
[00:04:15] learning this notion of optimization but I think that that was a little bit un
[00:04:17] I think that that was a little bit un like it was more confusing than it was
[00:04:19] like it was more confusing than it was helpful um and because depending on how
[00:04:21] helpful um and because depending on how you think of it in machine learning II
[00:04:24] you think of it in machine learning II we always have some form of metric or
[00:04:26] we always have some form of metric or optimization so you can think of a loss
[00:04:28] optimization so you can think of a loss as also being we're trying to to
[00:04:29] as also being we're trying to to minimize the loss and so that also
[00:04:31] minimize the loss and so that also sounds like an optimization problem so
[00:04:33] sounds like an optimization problem so you can just ignore that distinction for
[00:04:35] you can just ignore that distinction for now I do think in general when we're
[00:04:37] now I do think in general when we're thinking about decision making it's
[00:04:39] thinking about decision making it's going to be very important what we think
[00:04:41] going to be very important what we think of as that metric and so it won't
[00:04:43] of as that metric and so it won't necessarily just be loss functions we
[00:04:44] necessarily just be loss functions we can have lots of different scalar values
[00:04:46] can have lots of different scalar values or even multiple objectives but the
[00:04:49] or even multiple objectives but the distinction of whether or not supervised
[00:04:50] distinction of whether or not supervised learning is using optimization is
[00:04:53] learning is using optimization is perhaps not so
[00:04:56] helpful okay great so let's go ahead and
[00:04:58] helpful okay great so let's go ahead and get started um so I do also just want to
[00:05:01] get started um so I do also just want to highlight that for some of you um and I
[00:05:04] highlight that for some of you um and I got a question about this we've also got
[00:05:05] got a question about this we've also got a couple questions about this this this
[00:05:07] a couple questions about this this this first week or two will overlap a little
[00:05:09] first week or two will overlap a little bit with some of the other classes you
[00:05:10] bit with some of the other classes you might have taken so particularly if
[00:05:12] might have taken so particularly if you've taken 238 with Michael Cocker the
[00:05:14] you've taken 238 with Michael Cocker the beginning May overlap the things that
[00:05:16] beginning May overlap the things that will probably still be different in the
[00:05:18] will probably still be different in the first couple weeks is I expect there's
[00:05:20] first couple weeks is I expect there's going to be um a higher level of theory
[00:05:22] going to be um a higher level of theory in the first week or two about the
[00:05:24] in the first week or two about the properties of some of these algorithms
[00:05:26] properties of some of these algorithms and what sort of guarantees we have and
[00:05:29] and what sort of guarantees we have and then after I suspect after that most of
[00:05:31] then after I suspect after that most of the content in the rest of the class
[00:05:32] the content in the rest of the class will be quite different if you have any
[00:05:34] will be quite different if you have any questions about how this compares to a
[00:05:36] questions about how this compares to a lot of the other decision-making classes
[00:05:38] lot of the other decision-making classes that are offered at Stanford don't
[00:05:39] that are offered at Stanford don't hesitate to reach out to me in office
[00:05:41] hesitate to reach out to me in office hours on Ed or after
[00:05:43] hours on Ed or after class all right now why do we do this
[00:05:46] class all right now why do we do this because also you might be thinking we
[00:05:47] because also you might be thinking we want to get to alphago or I want to get
[00:05:49] want to get to alphago or I want to get to controlling robots or I want to get
[00:05:51] to controlling robots or I want to get to optimizing llms why are we starting
[00:05:53] to optimizing llms why are we starting with systems like the seven State Mars
[00:05:55] with systems like the seven State Mars rover that we're going to look at and
[00:05:57] rover that we're going to look at and the reason is because actually a lot of
[00:05:59] the reason is because actually a lot of the ideas um that enabled people to
[00:06:02] the ideas um that enabled people to solve alphago um and do things like rhf
[00:06:05] solve alphago um and do things like rhf or reinforcement learning from Human
[00:06:07] or reinforcement learning from Human feedback really Builds on this
[00:06:09] feedback really Builds on this fundamental notion of decision processes
[00:06:11] fundamental notion of decision processes and I think it's much easier to really
[00:06:13] and I think it's much easier to really cleanly see how these ideas come up
[00:06:15] cleanly see how these ideas come up bless you when you can actually see
[00:06:17] bless you when you can actually see these in the the world is tabular you
[00:06:19] these in the the world is tabular you can just write down all the states so
[00:06:21] can just write down all the states so that's why I think it's helpful but even
[00:06:23] that's why I think it's helpful but even today we're going to start to see where
[00:06:24] today we're going to start to see where those ideas might be applied so we're
[00:06:26] those ideas might be applied so we're going to start to see things like policy
[00:06:27] going to start to see things like policy search which is sort of the foundations
[00:06:29] search which is sort of the foundations towards things like policy gradients
[00:06:31] towards things like policy gradients which are extremely widely used so you
[00:06:33] which are extremely widely used so you can think of all of these as just being
[00:06:35] can think of all of these as just being building blocks that we're going to use
[00:06:36] building blocks that we're going to use to build up to get to the point we're
[00:06:37] to build up to get to the point we're later going to be and very soon within a
[00:06:39] later going to be and very soon within a couple weeks tackling things that are
[00:06:41] couple weeks tackling things that are state-of-the-art
[00:06:42] state-of-the-art algorithms all right so what we're going
[00:06:44] algorithms all right so what we're going to be doing today is really focusing on
[00:06:46] to be doing today is really focusing on making good decisions given a markof
[00:06:48] making good decisions given a markof decision process and so that means both
[00:06:50] decision process and so that means both being enough to understand how good a
[00:06:52] being enough to understand how good a particular decision policy is as well as
[00:06:54] particular decision policy is as well as what is an optimal decision
[00:06:57] what is an optimal decision policy and
[00:07:00] policy and when I say we're given a model of the
[00:07:01] when I say we're given a model of the world what I mean is that we are given
[00:07:03] world what I mean is that we are given sort of that Dynamics model which tells
[00:07:06] sort of that Dynamics model which tells us how the world evolves when we make
[00:07:11] decisions and we are given a reward
[00:07:13] decisions and we are given a reward model which tells us how good decisions
[00:07:16] model which tells us how good decisions are and last time we talked about
[00:07:19] are and last time we talked about Markoff processes and we were starting
[00:07:21] Markoff processes and we were starting to talk about Markoff reward processes
[00:07:23] to talk about Markoff reward processes because they can end up being really
[00:07:25] because they can end up being really useful when we're trying to evaluate how
[00:07:27] useful when we're trying to evaluate how good a particular decision uh policy is
[00:07:30] good a particular decision uh policy is and we'll see a lot of the same ideas
[00:07:32] and we'll see a lot of the same ideas from markof for word processes to mdps
[00:07:36] from markof for word processes to mdps okay all right so let's just refresh our
[00:07:38] okay all right so let's just refresh our memory so as this is the question that
[00:07:40] memory so as this is the question that we had before of like you know how do we
[00:07:42] we had before of like you know how do we think of uh the influence of discount
[00:07:44] think of uh the influence of discount factors as was said what happens is we
[00:07:47] factors as was said what happens is we multiply the next reward by the discount
[00:07:49] multiply the next reward by the discount Factor two rewards Away by the discount
[00:07:52] Factor two rewards Away by the discount Factor squared Etc and so as you can see
[00:07:55] Factor squared Etc and so as you can see there if the Horizon is really long or
[00:07:57] there if the Horizon is really long or as it goes to Infinity rewards will have
[00:07:59] as it goes to Infinity rewards will have zer value eventually because gamma is
[00:08:01] zer value eventually because gamma is less than
[00:08:02] less than one so the idea of the value function
[00:08:05] one so the idea of the value function was to say remember this is a markof
[00:08:07] was to say remember this is a markof reward process we don't have decisions
[00:08:08] reward process we don't have decisions yet just says how much um is the
[00:08:11] yet just says how much um is the expected discounted summer rewards we
[00:08:13] expected discounted summer rewards we will get starting this state and acting
[00:08:16] will get starting this state and acting most of the time today forever so most
[00:08:18] most of the time today forever so most of the time we'll think of today of just
[00:08:20] of the time we'll think of today of just like getting to act forever and how much
[00:08:21] like getting to act forever and how much reward would you get and because this
[00:08:23] reward would you get and because this gamma if as long as the gamma is less
[00:08:25] gamma if as long as the gamma is less than one here that will be a finite
[00:08:27] than one here that will be a finite number
[00:08:29] number so we're just starting to talk about how
[00:08:31] so we're just starting to talk about how could we compute this so again remember
[00:08:33] could we compute this so again remember the return is going to be a particular
[00:08:36] the return is going to be a particular series of rewards you might get if you
[00:08:37] series of rewards you might get if you start in this state in act forever and V
[00:08:40] start in this state in act forever and V is going to be on average how much
[00:08:41] is going to be on average how much reward would you get if you start in
[00:08:43] reward would you get if you start in this state in act
[00:08:45] this state in act forever all right so one of the key
[00:08:48] forever all right so one of the key ideas here is that Computing the value
[00:08:50] ideas here is that Computing the value of an infinite Horizon Markoff reward
[00:08:53] of an infinite Horizon Markoff reward process leverages the Markoff property
[00:08:56] process leverages the Markoff property which was this idea that the future is
[00:08:58] which was this idea that the future is independent of the past past given the
[00:09:00] independent of the past past given the present so given your current state you
[00:09:02] present so given your current state you don't have to think more about the the
[00:09:04] don't have to think more about the the history so what that applies when we try
[00:09:06] history so what that applies when we try to compute what the expected reward is
[00:09:08] to compute what the expected reward is future reward is from a state is we can
[00:09:10] future reward is from a state is we can think of well what is the immediate
[00:09:11] think of well what is the immediate reward we get in that state plus all the
[00:09:14] reward we get in that state plus all the different states we could get to next
[00:09:16] different states we could get to next under our Dynamics model and then the
[00:09:18] under our Dynamics model and then the value of their
[00:09:20] value of their reward how much do we weigh each of
[00:09:22] reward how much do we weigh each of those well we weigh each of those just
[00:09:23] those well we weigh each of those just according to what is the probability I
[00:09:25] according to what is the probability I could get to each of those next
[00:09:27] could get to each of those next States and if you're familiar things
[00:09:29] States and if you're familiar things like tree search you can think of it as
[00:09:31] like tree search you can think of it as just I'm in my starting State I think of
[00:09:32] just I'm in my starting State I think of all the next States I could go to each
[00:09:34] all the next States I could go to each of them have some weight depending on
[00:09:35] of them have some weight depending on the probability i' get there and then I
[00:09:38] the probability i' get there and then I sum all of those up according to their
[00:09:42] values okay and this is going to be the
[00:09:44] values okay and this is going to be the basis of the Bellman equation which
[00:09:45] basis of the Bellman equation which we're going to see lots
[00:09:47] we're going to see lots about okay so if we wanted to think
[00:09:50] about okay so if we wanted to think about how we could solve this one way we
[00:09:53] about how we could solve this one way we could think of it is if we have a
[00:09:55] could think of it is if we have a tabular world meaning that we can write
[00:09:57] tabular world meaning that we can write we can maintain a uh scale value for
[00:10:00] we can maintain a uh scale value for every single state separately so this is
[00:10:02] every single state separately so this is like our Mars Rover case um then we
[00:10:04] like our Mars Rover case um then we could just Express the value function in
[00:10:06] could just Express the value function in a matrix equation so we say the value of
[00:10:09] a matrix equation so we say the value of each of the states is exactly equal to
[00:10:10] each of the states is exactly equal to the immediate reward plus gamma times
[00:10:13] the immediate reward plus gamma times the um transition probability to all the
[00:10:15] the um transition probability to all the next
[00:10:18] next States and so that's nice because now we
[00:10:20] States and so that's nice because now we can just directly solve for what the
[00:10:22] can just directly solve for what the value is so we know that this has to
[00:10:25] value is so we know that this has to hold so now what we're going to do is
[00:10:27] hold so now what we're going to do is just invert that to solve for V
[00:10:30] just invert that to solve for V so what we would say in this case is we
[00:10:32] so what we would say in this case is we would say V minus gamma * P of V this is
[00:10:37] would say V minus gamma * P of V this is p and again I'll apologize that in the
[00:10:40] p and again I'll apologize that in the different uh things you see online or
[00:10:43] different uh things you see online or the textbook Etc people sometimes use T
[00:10:45] the textbook Etc people sometimes use T for transition Matrix they sometimes use
[00:10:47] for transition Matrix they sometimes use P for like probabilities going to the
[00:10:49] P for like probabilities going to the next state um if it's ever confusing
[00:10:51] next state um if it's ever confusing what notation being used don't hesitate
[00:10:52] what notation being used don't hesitate to reach out okay so we just rewrite it
[00:10:55] to reach out okay so we just rewrite it like this is equal to
[00:10:57] like this is equal to R and then we move this so we have B of
[00:11:01] R and then we move this so we have B of IUS gamma p is equal to r i equals the
[00:11:05] IUS gamma p is equal to r i equals the identity
[00:11:09] Matrix which means V is equal to IUS
[00:11:13] Matrix which means V is equal to IUS gamma
[00:11:14] gamma P inverse time
[00:11:18] P inverse time R so why do I show this I show this
[00:11:21] R so why do I show this I show this because if you know how the world works
[00:11:22] because if you know how the world works you have the Dynamics model you know
[00:11:24] you have the Dynamics model you know what the reward function is and the
[00:11:26] what the reward function is and the world is small enough you can just
[00:11:28] world is small enough you can just directly solve for this
[00:11:30] directly solve for this this isn't for decision yet this is just
[00:11:31] this isn't for decision yet this is just showing us blessy like what the value
[00:11:34] showing us blessy like what the value would be of each of the states so this
[00:11:37] would be of each of the states so this is one way to solve it we would call
[00:11:38] is one way to solve it we would call this like sort of the analytics
[00:11:41] this like sort of the analytics solution and this require one thing to
[00:11:44] solution and this require one thing to note here is this requires a matrix
[00:11:45] note here is this requires a matrix inverse and so there are faster
[00:11:47] inverse and so there are faster algorithms than n cubed n being the
[00:11:49] algorithms than n cubed n being the number of states but you know in general
[00:11:51] number of states but you know in general Matrix inverses are fairly expensive so
[00:11:53] Matrix inverses are fairly expensive so this has been done once but this is a
[00:11:55] this has been done once but this is a fairly expens if your state space the
[00:11:58] fairly expens if your state space the number of states you have is large this
[00:11:59] number of states you have is large this can be expensive and it also requires
[00:12:02] can be expensive and it also requires that um the identity Matrix minus gamma
[00:12:05] that um the identity Matrix minus gamma times uh the dam the Dynamics model is
[00:12:09] times uh the dam the Dynamics model is invertible okay so this is one way we
[00:12:11] invertible okay so this is one way we can solve this yeah and remind me your
[00:12:12] can solve this yeah and remind me your name um in practice what usually happens
[00:12:16] name um in practice what usually happens like do people just go ahead and take
[00:12:19] like do people just go ahead and take the Matrix
[00:12:20] the Matrix inverse let me the questions in practice
[00:12:23] inverse let me the questions in practice you usually find that this these kinds
[00:12:25] you usually find that this these kinds of matrices are invertible and if yes do
[00:12:28] of matrices are invertible and if yes do people just go ahead and The Matrix or
[00:12:31] people just go ahead and The Matrix or it's a good question so in practice is
[00:12:33] it's a good question so in practice is it invertible and what do people do in
[00:12:35] it invertible and what do people do in practice normally we're dealing with
[00:12:36] practice normally we're dealing with State spaces that are far too large so
[00:12:38] State spaces that are far too large so we can't do this yeah good question
[00:12:42] we can't do this yeah good question there might be cases where it's small
[00:12:43] there might be cases where it's small enough but in general no so that's a
[00:12:46] enough but in general no so that's a great motivation for a second approach
[00:12:48] great motivation for a second approach which is instead of doing it directly
[00:12:49] which is instead of doing it directly analytically we're going to use dynamic
[00:12:51] analytically we're going to use dynamic programming and we're going to design an
[00:12:52] programming and we're going to design an iterative algorithm okay and this is
[00:12:55] iterative algorithm okay and this is going to be very very similar to what
[00:12:56] going to be very very similar to what we're going to see for decision
[00:12:57] we're going to see for decision processes so the idea in this case is
[00:12:59] processes so the idea in this case is we're not going to do this in one step
[00:13:01] we're not going to do this in one step but we're going to avoid that Matrix
[00:13:02] but we're going to avoid that Matrix inverse which might be pretty expensive
[00:13:04] inverse which might be pretty expensive so we're going to initialize the value
[00:13:06] so we're going to initialize the value of a state to zero for all s and you can
[00:13:09] of a state to zero for all s and you can think about whether or not it actually M
[00:13:11] think about whether or not it actually M matters what we initialize to but just
[00:13:12] matters what we initialize to but just imagine we do that and then for a series
[00:13:15] imagine we do that and then for a series of iterations K is our iteration
[00:13:17] of iterations K is our iteration variable for all the states in s what we
[00:13:19] variable for all the states in s what we do is we say we're going to sort of make
[00:13:21] do is we say we're going to sort of make a new copy of our value function and we
[00:13:24] a new copy of our value function and we say VK of s is equal to R of S Plus
[00:13:27] say VK of s is equal to R of S Plus gamma sum over S Prime probability of
[00:13:29] gamma sum over S Prime probability of going to S Prime given s times the value
[00:13:32] going to S Prime given s times the value that we already have for Kus one of S
[00:13:35] that we already have for Kus one of S Prime and we just do this over and over
[00:13:38] Prime and we just do this over and over and over again until our value function
[00:13:39] and over again until our value function stops conver stops
[00:13:41] stops conver stops changing and we'll talk soon about
[00:13:43] changing and we'll talk soon about whether it will stop changing the nice
[00:13:45] whether it will stop changing the nice thing about this is that it's only s s
[00:13:48] thing about this is that it's only s s for each iteration so this would be an
[00:13:52] for each iteration so this would be an iteration instead of a matrix inverse
[00:14:00] all right so this is how you could
[00:14:01] all right so this is how you could compute the value of an MRP now we're
[00:14:03] compute the value of an MRP now we're going to see how we could do that for an
[00:14:05] going to see how we could do that for an mdp so markup decision process is very
[00:14:08] mdp so markup decision process is very similar to a Markoff reward process but
[00:14:10] similar to a Markoff reward process but now we get to add in actions so now
[00:14:11] now we get to add in actions so now we're actually going to be starting to
[00:14:13] we're actually going to be starting to make decisions and the idea now is that
[00:14:16] make decisions and the idea now is that the Dynamics transition model um will
[00:14:18] the Dynamics transition model um will probably depend on the action you take
[00:14:20] probably depend on the action you take so you're going to get to different
[00:14:22] so you're going to get to different distributions of next States plus you
[00:14:24] distributions of next States plus you and so it could be something like you
[00:14:25] and so it could be something like you know you think of um depending on the ad
[00:14:27] know you think of um depending on the ad you show a customer they might different
[00:14:29] you show a customer they might different things depending on the controls of your
[00:14:31] things depending on the controls of your robot it's going to move or manipulate
[00:14:33] robot it's going to move or manipulate its hand in a different way General
[00:14:34] its hand in a different way General these Dynamics are going to be a
[00:14:35] these Dynamics are going to be a function of the action um and we are
[00:14:38] function of the action um and we are going to for right now uh talk assume
[00:14:40] going to for right now uh talk assume the reward is a function of the state
[00:14:42] the reward is a function of the state and the action you
[00:14:43] and the action you take so you often say that an mdp is
[00:14:45] take so you often say that an mdp is defined by a tuple sa Dynamics model
[00:14:48] defined by a tuple sa Dynamics model reward model and
[00:14:50] reward model and Gamma okay so we could think of that for
[00:14:53] Gamma okay so we could think of that for here so now we have our same little Mars
[00:14:55] here so now we have our same little Mars Rover but now we actually have two
[00:14:56] Rover but now we actually have two different Dynamics models one one for if
[00:14:59] different Dynamics models one one for if we take A1 and one if we take A2 this is
[00:15:02] we take A1 and one if we take A2 this is just an example one these in these cases
[00:15:04] just an example one these in these cases these are deterministic in general we
[00:15:06] these are deterministic in general we can have them be stochastic and we would
[00:15:08] can have them be stochastic and we would also need to specify what the reward is
[00:15:10] also need to specify what the reward is so maybe we have zero reward in all of
[00:15:11] so maybe we have zero reward in all of these states and plus one here and plus
[00:15:15] these states and plus one here and plus 10 at the end and this would just Define
[00:15:17] 10 at the end and this would just Define so once you've defined the state space
[00:15:19] so once you've defined the state space the action space the reward function the
[00:15:21] the action space the reward function the Dynamics model and Gamma then you've
[00:15:23] Dynamics model and Gamma then you've defined your
[00:15:25] defined your mdp okay all right so now we have get to
[00:15:29] mdp okay all right so now we have get to start to think about policies which is
[00:15:31] start to think about policies which is what we'll be talking about throughout
[00:15:32] what we'll be talking about throughout the course which is how do we make
[00:15:35] the course which is how do we make decisions depending on the state we're
[00:15:37] decisions depending on the state we're in and the policy is going to specify
[00:15:41] in and the policy is going to specify the action take which can be
[00:15:42] the action take which can be deterministic or
[00:15:44] deterministic or stochastic and often we're going to
[00:15:46] stochastic and often we're going to think of it as being
[00:15:48] think of it as being stochastic and we'll talk about the
[00:15:50] stochastic and we'll talk about the properties of stochastic versus
[00:15:52] properties of stochastic versus deterministic ones and why you might
[00:15:53] deterministic ones and why you might want one or the other um quite a bit in
[00:15:55] want one or the other um quite a bit in the class but we can generally do
[00:15:57] the class but we can generally do everything we're doing in each
[00:16:01] all right so an mdp plus a policy is
[00:16:03] all right so an mdp plus a policy is just a Markoff reward
[00:16:06] just a Markoff reward process why is that because once you
[00:16:08] process why is that because once you specify how you're going to act you've
[00:16:11] specify how you're going to act you've sort of removed the policy part and so
[00:16:12] sort of removed the policy part and so if you want to know how good that policy
[00:16:14] if you want to know how good that policy is so let's say someone says again your
[00:16:17] is so let's say someone says again your boss says hey how good is this thing at
[00:16:18] boss says hey how good is this thing at you know like advertising to customers
[00:16:20] you know like advertising to customers for example then once you've decided
[00:16:23] for example then once you've decided what the policy is we can think of the
[00:16:25] what the policy is we can think of the reward as just being a weighted sum over
[00:16:27] reward as just being a weighted sum over what the probability is is taking that
[00:16:29] what the probability is is taking that action in that state times the reward
[00:16:31] action in that state times the reward for that state in action and then your
[00:16:33] for that state in action and then your Dynamics model which is a little more
[00:16:35] Dynamics model which is a little more subtle which is now you're taking a
[00:16:37] subtle which is now you're taking a weighted sum over all of the transition
[00:16:39] weighted sum over all of the transition Dynamics according to the action you
[00:16:41] Dynamics according to the action you take weighted by the probability you
[00:16:43] take weighted by the probability you take that
[00:16:46] action so it just defines um a markof
[00:16:50] action so it just defines um a markof process because now you just have this
[00:16:52] process because now you just have this sort of transform Dynamics model where
[00:16:54] sort of transform Dynamics model where we've merged in the policy Okay so why
[00:16:59] we've merged in the policy Okay so why is this helpful and this is something
[00:17:00] is this helpful and this is something that you may or may not have seen in
[00:17:01] that you may or may not have seen in previous classes one of the reasons why
[00:17:02] previous classes one of the reasons why this is helpful is because now we can
[00:17:04] this is helpful is because now we can just say oh any techniques we have for
[00:17:05] just say oh any techniques we have for markof reward processes we could also
[00:17:07] markof reward processes we could also apply to evaluating the value of a
[00:17:10] apply to evaluating the value of a particular policy in a Markoff decision
[00:17:12] particular policy in a Markoff decision process because we've just reduced an
[00:17:14] process because we've just reduced an mdp and policy evaluation back to an
[00:17:21] MRP all right
[00:17:23] MRP all right so if we think about doing policy
[00:17:25] so if we think about doing policy evaluation with an
[00:17:27] evaluation with an mdp um we can just plug in the actual
[00:17:31] mdp um we can just plug in the actual policy that we would be T be using okay
[00:17:35] policy that we would be T be using okay so what we have in this case is that
[00:17:38] so what we have in this case is that instead of now we actually get to make
[00:17:40] instead of now we actually get to make decisions and so then we get to say what
[00:17:42] decisions and so then we get to say what is the probability of taking the action
[00:17:43] is the probability of taking the action in this state times the expected
[00:17:46] in this state times the expected discounted sum of rewards at that
[00:17:49] discounted sum of rewards at that point so this looks very similar to an
[00:17:52] point so this looks very similar to an MRP except for we're saying based on the
[00:17:54] MRP except for we're saying based on the probability for each action what would
[00:17:56] probability for each action what would we get
[00:17:57] we get next and we call this a Bellman backup
[00:18:00] next and we call this a Bellman backup for a particular
[00:18:02] for a particular policy because this is um going to
[00:18:04] policy because this is um going to specify what is our expected discounted
[00:18:06] specify what is our expected discounted sum of future Awards if we take start in
[00:18:09] sum of future Awards if we take start in this um State and follow the
[00:18:12] this um State and follow the policy and just notice that if the
[00:18:14] policy and just notice that if the policy is actually deterministic we can
[00:18:16] policy is actually deterministic we can reduce it back to a case where we've
[00:18:18] reduce it back to a case where we've sort of averaged
[00:18:20] sort of averaged over these rewards so remember this was
[00:18:23] over these rewards so remember this was just going to be
[00:18:26] just going to be um if you have a particular action and
[00:18:28] um if you have a particular action and then you're just going to index into
[00:18:30] then you're just going to index into what the reward is for that particular
[00:18:31] what the reward is for that particular action so we can see that
[00:18:33] action so we can see that here okay and just raise your hand if
[00:18:36] here okay and just raise your hand if you've seen this before like if you've
[00:18:38] you've seen this before like if you've seen the okay good so like probably at
[00:18:40] seen the okay good so like probably at least two-third of
[00:18:42] least two-third of people all
[00:18:46] right okay so if you want to check your
[00:18:48] right okay so if you want to check your answers if you something this is new for
[00:18:50] answers if you something this is new for you then um one thing to do is to try to
[00:18:53] you then um one thing to do is to try to check that you can um do this sort of
[00:18:55] check that you can um do this sort of value iteration um or this policy
[00:18:57] value iteration um or this policy evaluation for the Mars where for
[00:18:58] evaluation for the Mars where for example we won't go through it in class
[00:19:00] example we won't go through it in class but can check the answers I'll release
[00:19:02] but can check the answers I'll release them at the end of the slide just to
[00:19:03] them at the end of the slide just to check that you know how to apply
[00:19:05] check that you know how to apply this all right so of course shortly
[00:19:08] this all right so of course shortly we're going to be interested in not just
[00:19:09] we're going to be interested in not just EV valuing the value of a single policy
[00:19:11] EV valuing the value of a single policy but finding an optimal
[00:19:12] but finding an optimal policy so one question is how many
[00:19:14] policy so one question is how many policies are there and is the optimal
[00:19:17] policies are there and is the optimal policy value
[00:19:20] policy value unique so we'll just take a second you
[00:19:22] unique so we'll just take a second you can go to the polls and enter in your
[00:19:24] can go to the polls and enter in your answer
[00:19:35] okay great so it looks like most people
[00:19:37] okay great so it looks like most people um got the vasary appeal got the right
[00:19:39] um got the vasary appeal got the right answer for the first one which is it's 2
[00:19:41] answer for the first one which is it's 2 to the 7 in general the number of
[00:19:43] to the 7 in general the number of policies we have is going to be a to the
[00:19:48] policies we have is going to be a to the S because for every single state we
[00:19:50] S because for every single state we could choose any of the
[00:19:52] could choose any of the actions and also most people got the
[00:19:54] actions and also most people got the next one right which is great which is
[00:19:56] next one right which is great which is the optimal policy um the one with the
[00:19:58] the optimal policy um the one with the SI is not always
[00:20:00] SI is not always unique it can be unique it depends on
[00:20:02] unique it can be unique it depends on the problem um but it's not going to be
[00:20:04] the problem um but it's not going to be unique whenever more than one action has
[00:20:07] unique whenever more than one action has the same identical value so when you
[00:20:10] the same identical value so when you have ties
[00:20:11] have ties yeah um how do we generally deal with
[00:20:14] yeah um how do we generally deal with invalid actions because like for example
[00:20:18] invalid actions because like for example if we're in S1 and we choose left I
[00:20:21] if we're in S1 and we choose left I would imagine to me that's an invalid
[00:20:23] would imagine to me that's an invalid action not sure we really do with that
[00:20:27] action not sure we really do with that yeah so the question was um if we have
[00:20:28] yeah so the question was um if we have invalid action so in general you can
[00:20:30] invalid action so in general you can have a different state a different
[00:20:32] have a different state a different action space be possible in every state
[00:20:34] action space be possible in every state that's also very common in like
[00:20:35] that's also very common in like recommendation engines that you know
[00:20:37] recommendation engines that you know you'd only it's only a subset of
[00:20:38] you'd only it's only a subset of Articles you might show to some people
[00:20:40] Articles you might show to some people based on their state um in this
[00:20:42] based on their state um in this particular example we're going to assume
[00:20:43] particular example we're going to assume that it's always it's not actually go
[00:20:45] that it's always it's not actually go left it's try left and so if you try to
[00:20:46] left it's try left and so if you try to go left and there's nothing rest of the
[00:20:48] go left and there's nothing rest of the world you just fail and you stay in the
[00:20:49] world you just fail and you stay in the same place but in general most of the
[00:20:52] same place but in general most of the time in the class we're going to assume
[00:20:53] time in the class we're going to assume the action space is the same for all
[00:20:54] the action space is the same for all states but in some cases it might be
[00:20:56] states but in some cases it might be different good question
[00:21:00] right okay so in mdp control we're going
[00:21:03] right okay so in mdp control we're going to want to not just have the policy like
[00:21:06] to want to not just have the policy like evaluate a particular policy but we're
[00:21:08] evaluate a particular policy but we're going to compute the optimal policy so
[00:21:09] going to compute the optimal policy so we want to take the argmax over the
[00:21:11] we want to take the argmax over the policy space which in general is that a
[00:21:13] policy space which in general is that a to the s
[00:21:15] to the s space and there is going to exist a
[00:21:17] space and there is going to exist a unique optimal value
[00:21:19] unique optimal value function and the optimal policy inside
[00:21:22] function and the optimal policy inside of a tabular m DP in an infinite Horizon
[00:21:25] of a tabular m DP in an infinite Horizon is unique and deterministic so
[00:21:29] is unique and deterministic so um those are two properties that are
[00:21:30] um those are two properties that are good to be familiar
[00:21:32] good to be familiar with so now we're going to think about
[00:21:34] with so now we're going to think about how do we actually compute this and what
[00:21:36] how do we actually compute this and what its other properties are so one is that
[00:21:38] its other properties are so one is that it's stationary um what I mean by that
[00:21:41] it's stationary um what I mean by that here is that in infinite Horizon problem
[00:21:44] here is that in infinite Horizon problem you always have an infinite number of
[00:21:46] you always have an infinite number of additional time steps and so the optimal
[00:21:49] additional time steps and so the optimal thing to do just depends on your state
[00:21:52] thing to do just depends on your state it doesn't depend on the time step we'll
[00:21:55] it doesn't depend on the time step we'll think more about what happens when you
[00:21:56] think more about what happens when you only have a finite number ler your H is
[00:21:59] only have a finite number ler your H is finite and what might happen there but
[00:22:01] finite and what might happen there but for most of today we're just going to
[00:22:02] for most of today we're just going to focus on the infinite Horizon problem
[00:22:05] focus on the infinite Horizon problem and as I said and most of you guys
[00:22:07] and as I said and most of you guys already knew that we um this in general
[00:22:09] already knew that we um this in general is not
[00:22:11] unique okay so one option is policy
[00:22:14] unique okay so one option is policy search and this is where we are going to
[00:22:17] search and this is where we are going to get into oh yeah and remember me your
[00:22:19] get into oh yeah and remember me your name is the optimality condition on the
[00:22:22] name is the optimality condition on the initial St is the optimality conditional
[00:22:25] initial St is the optimality conditional on the what do you mean by that
[00:22:29] on the what do you mean by that uh oh yes the optimality yes it'll be
[00:22:33] uh oh yes the optimality yes it'll be per state yeah so you the optimal policy
[00:22:36] per state yeah so you the optimal policy will be defined per state the idea is
[00:22:38] will be defined per state the idea is that you can take a different action in
[00:22:39] that you can take a different action in every state and you want to know what
[00:22:41] every state and you want to know what the optimal thing is to do to maximize
[00:22:42] the optimal thing is to do to maximize your expected discounted summer rewards
[00:22:44] your expected discounted summer rewards from every state individually like Point
[00:22:46] from every state individually like Point wise good question yeah yeah two
[00:22:48] wise good question yeah yeah two interconnected questions um why is there
[00:22:51] interconnected questions um why is there a unique optimal value function and
[00:22:53] a unique optimal value function and second is um can you remind me again of
[00:22:57] second is um can you remind me again of what was the reason why
[00:22:59] what was the reason why may not necessarily be unique you
[00:23:00] may not necessarily be unique you mentioned a specific
[00:23:02] mentioned a specific case ah so the policy optimal policy is
[00:23:04] case ah so the policy optimal policy is not necessarily unique because there
[00:23:06] not necessarily unique because there could be more than one action with the
[00:23:07] could be more than one action with the same value and the optimal value
[00:23:09] same value and the optimal value function is unique for reasons we'll see
[00:23:11] function is unique for reasons we'll see later in this class like later today
[00:23:14] later in this class like later today we'll prove it um okay so the one of the
[00:23:18] we'll prove it um okay so the one of the things and this is going to go back to
[00:23:19] things and this is going to go back to the conceptual question I put at the
[00:23:21] the conceptual question I put at the beginning of class is we would like to
[00:23:23] beginning of class is we would like to ideally have methods and algorithms that
[00:23:26] ideally have methods and algorithms that have monotonic Improvement uh cap
[00:23:28] have monotonic Improvement uh cap capabilities and so policy search is
[00:23:30] capabilities and so policy search is going to be one of those so what we're
[00:23:32] going to be one of those so what we're going to do here is we're going to try
[00:23:34] going to do here is we're going to try to search to compute the optimal policy
[00:23:36] to search to compute the optimal policy there's a to the S deterministic
[00:23:38] there's a to the S deterministic policies in general you could imagine
[00:23:40] policies in general you could imagine just you know enumerating all of them
[00:23:41] just you know enumerating all of them and evaluating them all but we can often
[00:23:44] and evaluating them all but we can often do better than that and when I say
[00:23:45] do better than that and when I say better what I mean here is we can reduce
[00:23:48] better what I mean here is we can reduce the computation needed to try to
[00:23:49] the computation needed to try to identify the optimal policy so we
[00:23:51] identify the optimal policy so we shouldn't have to iterate through all a
[00:23:53] shouldn't have to iterate through all a to the S
[00:23:56] policies so how does policy iterate a
[00:23:58] policies so how does policy iterate a work the idea is that we're going to
[00:24:00] work the idea is that we're going to alternate between having a candidate
[00:24:02] alternate between having a candidate decision policy that might be optimal
[00:24:04] decision policy that might be optimal we're going to evaluate it and then
[00:24:05] we're going to evaluate it and then we're going to see if we can improve it
[00:24:08] we're going to see if we can improve it and then if we can improve it we will
[00:24:09] and then if we can improve it we will and otherwise we're going to
[00:24:11] and otherwise we're going to Halt so what we do how we do this is
[00:24:13] Halt so what we do how we do this is we're just going to initialize it
[00:24:14] we're just going to initialize it randomly which just means we're going to
[00:24:16] randomly which just means we're going to start off and we're going to say for
[00:24:17] start off and we're going to say for every single state we're going to pick
[00:24:18] every single state we're going to pick an
[00:24:19] an action and then while our policy is
[00:24:22] action and then while our policy is still changing so this is the L1 Norm um
[00:24:25] still changing so this is the L1 Norm um it measures if the policy changed for
[00:24:26] it measures if the policy changed for any state just as a refresher
[00:24:29] any state just as a refresher what we're going to first do is we're
[00:24:30] what we're going to first do is we're going to um evaluate the policy and then
[00:24:32] going to um evaluate the policy and then we're going to try to improve it
[00:24:38] so so in order to do it to do that sort
[00:24:40] so so in order to do it to do that sort of policy Improvement step it's going to
[00:24:42] of policy Improvement step it's going to be helpful to define the Q function
[00:24:44] be helpful to define the Q function again I know for many of you this is
[00:24:45] again I know for many of you this is probably a review the Q function of a
[00:24:47] probably a review the Q function of a particular policy is just what is the
[00:24:49] particular policy is just what is the reward of the immediate state in action
[00:24:52] reward of the immediate state in action plus the discounted sum of future
[00:24:54] plus the discounted sum of future rewards if we were to after that action
[00:24:57] rewards if we were to after that action act according to the policy
[00:24:59] act according to the policy so it's sort of like saying okay first
[00:25:01] so it's sort of like saying okay first when you're in this state you're going
[00:25:02] when you're in this state you're going to take this action and then from then
[00:25:03] to take this action and then from then on you're going to follow whatever your
[00:25:04] on you're going to follow whatever your policy tells you to
[00:25:05] policy tells you to do so and for any of you who've seen Q
[00:25:09] do so and for any of you who've seen Q learning you you've seen this sort of
[00:25:11] learning you you've seen this sort of idea a lot okay so what we're going to
[00:25:14] idea a lot okay so what we're going to try to do in this case why would we want
[00:25:16] try to do in this case why would we want a q function it turns out it's going to
[00:25:17] a q function it turns out it's going to make the policy Improvement step really
[00:25:19] make the policy Improvement step really easy so what we're going to first do is
[00:25:21] easy so what we're going to first do is we're going to say I'm going to take my
[00:25:22] we're going to say I'm going to take my particular policy I'm going to compute
[00:25:24] particular policy I'm going to compute the Q value for that particular policy
[00:25:27] the Q value for that particular policy Pi I CU we're going to be iterating and
[00:25:29] Pi I CU we're going to be iterating and then after that we're going to compute a
[00:25:31] then after that we're going to compute a new policy Pi I + 1 by just taking the
[00:25:33] new policy Pi I + 1 by just taking the ARX of
[00:25:34] ARX of Q so for our Q function we're just going
[00:25:37] Q so for our Q function we're just going to say according to this Q function
[00:25:39] to say according to this Q function which says what are the is the expected
[00:25:41] which says what are the is the expected discounted sum of rewards if I start in
[00:25:43] discounted sum of rewards if I start in this state take this action and follow
[00:25:45] this state take this action and follow Supply um which of those actions is the
[00:25:48] Supply um which of those actions is the best and we can Define that per
[00:25:52] best and we can Define that per state yeah is there any relationship
[00:25:54] state yeah is there any relationship between the Q function and the value
[00:25:56] between the Q function and the value function because it kind of looks
[00:25:57] function because it kind of looks similar
[00:25:58] similar yeah yeah so we often call it a q
[00:26:00] yeah yeah so we often call it a q function the the state action value
[00:26:04] function okay all right
[00:26:08] function okay all right so this is sort of just what we do now
[00:26:10] so this is sort of just what we do now now we're going to have this Q function
[00:26:11] now we're going to have this Q function we're generally going to do this by
[00:26:13] we're generally going to do this by having this q and then we will do PI I +
[00:26:16] having this q and then we will do PI I + 1 of s is equal to
[00:26:20] 1 of s is equal to arax over a of Q of sa per state and
[00:26:26] arax over a of Q of sa per state and then we just repeat this over and over
[00:26:28] then we just repeat this over and over again
[00:26:30] again okay so there's um a number of questions
[00:26:32] okay so there's um a number of questions you might have about this you might say
[00:26:33] you might have about this you might say okay this seems like a vaguely
[00:26:34] okay this seems like a vaguely reasonable thing to do but does it have
[00:26:36] reasonable thing to do but does it have any formal properties are we guaranteed
[00:26:38] any formal properties are we guaranteed to improve you know what can we say
[00:26:40] to improve you know what can we say about
[00:26:40] about this so to do that I think it's useful
[00:26:43] this so to do that I think it's useful to delve into what the policy
[00:26:45] to delve into what the policy Improvement step is actually doing okay
[00:26:48] Improvement step is actually doing okay so what the policy Improvement uh what
[00:26:51] so what the policy Improvement uh what when we compute the Q function this is
[00:26:53] when we compute the Q function this is the equation for the Q function so we s
[00:26:56] the equation for the Q function so we s of take our old policy pii okay and then
[00:26:58] of take our old policy pii okay and then we compute the Q function of this and we
[00:27:01] we compute the Q function of this and we can do this
[00:27:04] can do this iteratively and now what we want to do
[00:27:06] iteratively and now what we want to do in this case is think about what is the
[00:27:09] in this case is think about what is the performance going to be of the new
[00:27:10] performance going to be of the new policy we extract
[00:27:13] policy we extract okay all right so what the Q function
[00:27:18] okay all right so what the Q function says is we're going to be able to show
[00:27:20] says is we're going to be able to show that the Q function the the best thing
[00:27:23] that the Q function the the best thing of the Q function is better than the
[00:27:25] of the Q function is better than the value of the old policy okay
[00:27:28] value of the old policy okay so what does this say so the first thing
[00:27:30] so what does this say so the first thing is just how we've computed this is just
[00:27:32] is just how we've computed this is just the policy evaluation
[00:27:37] step and we know that if we have a Q
[00:27:41] step and we know that if we have a Q function over s and a for a particular s
[00:27:44] function over s and a for a particular s Max over a of Q Pi of sa has to be at
[00:27:47] Max over a of Q Pi of sa has to be at least as good as the Q function for any
[00:27:51] least as good as the Q function for any of this any of the actions okay so we
[00:27:53] of this any of the actions okay so we know that this has to be this thing is
[00:27:56] know that this has to be this thing is always greater than equal to Q Pi I
[00:27:59] always greater than equal to Q Pi I of sa a for all
[00:28:03] a and then this is just that equation
[00:28:06] a and then this is just that equation this is just whatever what exactly is Q
[00:28:08] this is just whatever what exactly is Q pii of
[00:28:09] pii of sa is the
[00:28:12] sa is the definition almost except for it's
[00:28:15] definition almost except for it's particularly for the actions this is
[00:28:17] particularly for the actions this is first
[00:28:23] specifically if we were to follow the
[00:28:25] specifically if we were to follow the previous
[00:28:26] previous policy so remember this is the equation
[00:28:29] policy so remember this is the equation for Q Pi I of sa think about one of
[00:28:32] for Q Pi I of sa think about one of those actions that you could have done
[00:28:34] those actions that you could have done is exactly what the old policy would
[00:28:35] is exactly what the old policy would have told you to do that is what this
[00:28:38] have told you to do that is what this equation is if you just take a here and
[00:28:40] equation is if you just take a here and you plug in pi I of s
[00:28:51] a so that's just exactly what this is
[00:28:54] a so that's just exactly what this is and that is just the definition of v Pi
[00:28:56] and that is just the definition of v Pi I of s
[00:28:59] okay so what is this saying what this is
[00:29:02] okay so what is this saying what this is saying
[00:29:03] saying is if you were to take your new policy
[00:29:07] is if you were to take your new policy Pi I + 1 so remember Pi I + 1 is defined
[00:29:09] Pi I + 1 so remember Pi I + 1 is defined as the argmax of this Q function is
[00:29:12] as the argmax of this Q function is whatever maximizes your new Q function
[00:29:15] whatever maximizes your new Q function so what this says is if you were to take
[00:29:16] so what this says is if you were to take Pi I + one of s for one action and then
[00:29:19] Pi I + one of s for one action and then follow Pi I
[00:29:21] follow Pi I forever so that's what the Q function
[00:29:23] forever so that's what the Q function represents then our expected sum of
[00:29:25] represents then our expected sum of rewards is at least as good as if we'd
[00:29:27] rewards is at least as good as if we'd always followed
[00:29:29] always followed pii so that's what this equation is
[00:29:31] pii so that's what this equation is telling us it's like if I get to make
[00:29:33] telling us it's like if I get to make one decision differently and then from
[00:29:35] one decision differently and then from then on I follow my old policy the value
[00:29:37] then on I follow my old policy the value I can expect is at least as good as the
[00:29:39] I can expect is at least as good as the value I could expect if I had always
[00:29:41] value I could expect if I had always followed the old
[00:29:42] followed the old policy
[00:29:44] policy okay does anybody have any questions
[00:29:46] okay does anybody have any questions about that because then the next step is
[00:29:48] about that because then the next step is going to build on
[00:29:52] that okay can you go back to
[00:29:55] that okay can you go back to that sure for the policy Improvement
[00:29:57] that sure for the policy Improvement yeah
[00:30:07] so the step that we talking about is
[00:30:09] so the step that we talking about is this one right like the policy improv
[00:30:11] this one right like the policy improv yeah we're trying to see like when we do
[00:30:12] yeah we're trying to see like when we do the policy Improvement step and we
[00:30:15] the policy Improvement step and we extract out instead of Max here argmax
[00:30:17] extract out instead of Max here argmax to get out the new policy how does the
[00:30:19] to get out the new policy how does the value of that relate to the value of the
[00:30:22] value of that relate to the value of the thing you could have done before in that
[00:30:25] thing you could have done before in that state and so this is just trying to say
[00:30:27] state and so this is just trying to say like what really is is Q Pi I of sa it
[00:30:30] like what really is is Q Pi I of sa it is the value you get if you first take a
[00:30:32] is the value you get if you first take a and then you follow Pi I from then
[00:30:34] and then you follow Pi I from then onwards okay so it's saying if you were
[00:30:36] onwards okay so it's saying if you were to do that then this new action you've
[00:30:39] to do that then this new action you've computed this argmax policy is actually
[00:30:42] computed this argmax policy is actually better than what you would have gotten
[00:30:44] better than what you would have gotten before or at least as good but the thing
[00:30:47] before or at least as good but the thing that should seem slightly strange to you
[00:30:49] that should seem slightly strange to you is I am not creating this sort of hybrid
[00:30:52] is I am not creating this sort of hybrid policy where like I take one new action
[00:30:54] policy where like I take one new action and then I follow pii forever I'm
[00:30:56] and then I follow pii forever I'm creating an entirely new policy where
[00:30:58] creating an entirely new policy where I'm not just going to follow Pi I + 1
[00:31:01] I'm not just going to follow Pi I + 1 for one step I'm going to follow it for
[00:31:02] for one step I'm going to follow it for all remaining steps okay so this should
[00:31:06] all remaining steps okay so this should not yet convince you that doing that is
[00:31:08] not yet convince you that doing that is actually any better than my old policy
[00:31:11] actually any better than my old policy this would say if you take one new
[00:31:12] this would say if you take one new action and then follow your old policy
[00:31:14] action and then follow your old policy it's going to be better than your old
[00:31:15] it's going to be better than your old policy so that's why we have to do
[00:31:17] policy so that's why we have to do additional work to show that we're
[00:31:19] additional work to show that we're actually going to get a monotonic
[00:31:20] actually going to get a monotonic Improvement if we just follow this new
[00:31:22] Improvement if we just follow this new policy always okay all right so let's go
[00:31:25] policy always okay all right so let's go through that all right so what we're
[00:31:27] through that all right so what we're going to prove is we're going to say
[00:31:29] going to prove is we're going to say actually that's true the new policy we
[00:31:31] actually that's true the new policy we construct through this policy
[00:31:33] construct through this policy Improvement step is somewhat remarkably
[00:31:35] Improvement step is somewhat remarkably going to be strictly a monotonic
[00:31:38] going to be strictly a monotonic Improvement compared to the old policy
[00:31:40] Improvement compared to the old policy unless it's
[00:31:42] unless it's identical that means that every step of
[00:31:44] identical that means that every step of policy Improvement we're going to get a
[00:31:46] policy Improvement we're going to get a better and better policy for every
[00:31:50] better and better policy for every state okay so um and the only time we
[00:31:54] state okay so um and the only time we not is if if we've already converged
[00:31:56] not is if if we've already converged okay so let's go through that
[00:31:59] okay so let's go through that okay so this is going to prove to us
[00:32:01] okay so this is going to prove to us that the value of the old policy is less
[00:32:04] that the value of the old policy is less than or equal to the value of the new
[00:32:05] than or equal to the value of the new policy meaning we're going to get like
[00:32:07] policy meaning we're going to get like this monotonic
[00:32:09] this monotonic Improvement so what we're going to do in
[00:32:11] Improvement so what we're going to do in this case is we are going to first write
[00:32:14] this case is we are going to first write out so this is just the definition this
[00:32:16] out so this is just the definition this is the
[00:32:20] definition of Max over a of our Q pii
[00:32:25] definition of Max over a of our Q pii okay all right so let's just write out
[00:32:27] okay all right so let's just write out what this is okay this is going to be
[00:32:29] what this is okay this is going to be equal
[00:32:30] equal to and it'll be written out more neatly
[00:32:33] to and it'll be written out more neatly on the next page
[00:32:50] too okay so what did I do here I noticed
[00:32:54] too okay so what did I do here I noticed that the definition of pi I + 1 is
[00:32:58] that the definition of pi I + 1 is exactly the ARG Max of this expression
[00:33:00] exactly the ARG Max of this expression instead of Max so when we did the policy
[00:33:03] instead of Max so when we did the policy Improvement the way we did the policy
[00:33:04] Improvement the way we did the policy Improvement was we took the argmax of
[00:33:06] Improvement was we took the argmax of the Q function so instead of having this
[00:33:09] the Q function so instead of having this max out here I'm just going to plug in
[00:33:11] max out here I'm just going to plug in pi I + 1 because that's going to give me
[00:33:13] pi I + 1 because that's going to give me something that's exactly equal to the
[00:33:15] something that's exactly equal to the max a for that whole
[00:33:17] max a for that whole expression okay right and so this is
[00:33:21] expression okay right and so this is exactly equal to that but what we can
[00:33:23] exactly equal to that but what we can show next or what we can do next is that
[00:33:26] show next or what we can do next is that we can just add the same terms and
[00:33:29] we can just add the same terms and notice that this is the same this is
[00:33:31] notice that this is the same this is less than or equal to Q Pi I of S Prime
[00:33:35] less than or equal to Q Pi I of S Prime a
[00:33:36] a prime because the value of pi I um for S
[00:33:41] prime because the value of pi I um for S Prime so following a particular policy
[00:33:44] Prime so following a particular policy always has to be less than or equal to
[00:33:46] always has to be less than or equal to taking the max over the Q function for
[00:33:48] taking the max over the Q function for that
[00:33:49] that policy why is that true because either
[00:33:52] policy why is that true because either either the max is the same as the pi I
[00:33:54] either the max is the same as the pi I action or there's a better action
[00:34:01] okay all right so that's the less than
[00:34:03] okay all right so that's the less than or equals and then we can just expand
[00:34:06] or equals and then we can just expand this expression out and this is going to
[00:34:07] this expression out and this is going to start to get a little bit messy which is
[00:34:09] start to get a little bit messy which is why it'll be nice to have it on the next
[00:34:10] why it'll be nice to have it on the next slide too but what will happen here is
[00:34:14] slide too but what will happen here is you can see how the expansion
[00:34:17] you can see how the expansion works
[00:34:20] works okay and why is this important this is
[00:34:22] okay and why is this important this is important because this is going to allow
[00:34:23] important because this is going to allow us to think about not if we just take
[00:34:25] us to think about not if we just take this new action on the first step but on
[00:34:27] this new action on the first step but on all all future steps okay so what we had
[00:34:29] all all future steps okay so what we had here is we had this thing which was Max
[00:34:32] here is we had this thing which was Max a over Q Pi I of s Prim a prime we're
[00:34:34] a over Q Pi I of s Prim a prime we're going to expand out what that expression
[00:34:36] going to expand out what that expression is okay because
[00:34:39] is okay because notice this thing here is exactly equal
[00:34:43] notice this thing here is exactly equal to this thing which we know is here okay
[00:34:46] to this thing which we know is here okay so we're just going to substitute it
[00:34:48] so we're just going to substitute it okay so we can put that in here so this
[00:34:51] okay so we can put that in here so this is R of S Prime Pi I + 1 of s prime plus
[00:34:57] is R of S Prime Pi I + 1 of s prime plus gamma sum over S Prime meaning S Prime
[00:35:01] gamma sum over S Prime meaning S Prime here I'm just using to speed two time
[00:35:03] here I'm just using to speed two time steps
[00:35:11] away okay why was that useful well what
[00:35:14] away okay why was that useful well what we've just said is that the value of pi
[00:35:17] we've just said is that the value of pi of s is less than or equal to taking
[00:35:19] of s is less than or equal to taking this new better action for one time step
[00:35:22] this new better action for one time step and then following the old policy I've
[00:35:24] and then following the old policy I've now done that recursively so I've said
[00:35:26] now done that recursively so I've said well now that's also less than or equal
[00:35:27] well now that's also less than or equal to if I take that new action once and
[00:35:30] to if I take that new action once and then I take it again and then I follow
[00:35:32] then I take it again and then I follow the old
[00:35:33] the old policy and then you just repeat this
[00:35:36] policy and then you just repeat this okay so like you just keep nesting this
[00:35:38] okay so like you just keep nesting this and what you can see here is that you
[00:35:39] and what you can see here is that you have these less than or equal that
[00:35:41] have these less than or equal that happen when instead of plugging in the
[00:35:43] happen when instead of plugging in the value of the old policy you allow
[00:35:45] value of the old policy you allow yourself to take a Max over that Q
[00:35:47] yourself to take a Max over that Q function okay and what happens if you do
[00:35:50] function okay and what happens if you do this all the way out it's will exactly
[00:35:52] this all the way out it's will exactly become equal to V of Pi I + 1 of s so
[00:35:57] become equal to V of Pi I + 1 of s so dot dot
[00:35:59] dot dot okay so I have that here so what this is
[00:36:02] okay so I have that here so what this is shown here is that the value you had
[00:36:05] shown here is that the value you had under the old policy of the state is
[00:36:07] under the old policy of the state is less than or equal to the value of that
[00:36:09] less than or equal to the value of that State under the new policy so this
[00:36:12] State under the new policy so this proves the monotonic Improvement which
[00:36:14] proves the monotonic Improvement which is super cool so this now says if we do
[00:36:17] is super cool so this now says if we do policy evaluation where you just keep
[00:36:18] policy evaluation where you just keep Computing the Q function and taking a
[00:36:20] Computing the Q function and taking a Max you will always monotonically
[00:36:22] Max you will always monotonically improve unless you stay the same
[00:36:30] all right so now let's do our Le next
[00:36:32] all right so now let's do our Le next check your understanding which is given
[00:36:34] check your understanding which is given everything I've just said if the policy
[00:36:36] everything I've just said if the policy doesn't change can it ever change again
[00:36:39] doesn't change can it ever change again and is there a maximum number of
[00:36:41] and is there a maximum number of iterations of policy
[00:36:44] iteration yeah it yeah um on the
[00:36:47] iteration yeah it yeah um on the previous slide is the dot dot dot
[00:36:48] previous slide is the dot dot dot supposed to represent like just like ALB
[00:36:50] supposed to represent like just like ALB manipulation yeah yeah you just keep
[00:36:53] manipulation yeah yeah you just keep expanding this all the way yeah good
[00:36:57] expanding this all the way yeah good question
[00:37:01] all right let's take a second and do the
[00:37:26] p for
[00:38:07] yeah what's your
[00:38:09] yeah what's your name um where at what point did we feel
[00:38:13] name um where at what point did we feel that this is actually leading to an
[00:38:14] that this is actually leading to an improvement like can we just like stay
[00:38:18] improvement like can we just like stay in the same like value level and like
[00:38:21] in the same like value level and like cuz the inequality was greater than or
[00:38:24] cuz the inequality was greater than or equal to so is it possible that you're
[00:38:26] equal to so is it possible that you're always equal to where you started
[00:38:28] always equal to where you started yeah it's a great question so um and in
[00:38:30] yeah it's a great question so um and in fact that really it so right that like
[00:38:32] fact that really it so right that like I've just shown less than or equal
[00:38:36] I've just shown less than or equal to what what we can um well I guess we
[00:38:39] to what what we can um well I guess we can discuss this in a second but um it
[00:38:41] can discuss this in a second but um it will be a monotonic Improvement unless
[00:38:43] will be a monotonic Improvement unless you're already the optimal
[00:38:45] you're already the optimal policy so if there's any state at which
[00:38:47] policy so if there's any state at which you can improve you will and if you stay
[00:38:49] you can improve you will and if you stay the same well actually we'll talk about
[00:38:51] the same well actually we'll talk about this now so um because the it's nicely
[00:38:54] this now so um because the it's nicely split between the answers for both of
[00:38:56] split between the answers for both of these questions so may maybe everybody
[00:38:57] these questions so may maybe everybody turn to somebody nearby you and discuss
[00:39:01] turn to somebody nearby you and discuss whether you think the policy can ever
[00:39:03] whether you think the policy can ever change if it didn't change initially and
[00:39:05] change if it didn't change initially and is there a maximum number of iterations
[00:39:08] is there a maximum number of iterations okay because it's pretty evenly split
[00:39:09] okay because it's pretty evenly split amongst people who
[00:39:26] voted e
[00:40:04] that's
[00:40:55] I want to make sure to clarify something
[00:40:56] I want to make sure to clarify something cuz I that came up in a good
[00:40:58] cuz I that came up in a good conversation which is let's assume for
[00:41:00] conversation which is let's assume for the moment there are no ties I know I
[00:41:02] the moment there are no ties I know I said that in general optimistic policies
[00:41:04] said that in general optimistic policies can have ties and that's true but for
[00:41:06] can have ties and that's true but for the point of view of this question it is
[00:41:07] the point of view of this question it is easiest to think about if there is only
[00:41:09] easiest to think about if there is only a single unique optimal policy so why
[00:41:13] a single unique optimal policy so why don't we do that again none of these are
[00:41:15] don't we do that again none of these are for um assessment they're only for your
[00:41:16] for um assessment they're only for your own learning but um but just in terms of
[00:41:19] own learning but um but just in terms of what you're thinking through my
[00:41:20] what you're thinking through my intention was to think about the simpler
[00:41:22] intention was to think about the simpler case where there is a single optimal
[00:41:24] case where there is a single optimal policy and then under that case we the
[00:41:27] policy and then under that case we the policy can ever change once it once it
[00:41:30] policy can ever change once it once it hasn't changed once what I mean by the
[00:41:31] hasn't changed once what I mean by the policy doesn't change meaning when we
[00:41:33] policy doesn't change meaning when we have had a policy and we do policy
[00:41:35] have had a policy and we do policy Improvement and our new improved policy
[00:41:37] Improvement and our new improved policy is the same as the old
[00:41:39] is the same as the old policy um so under the case that I just
[00:41:42] policy um so under the case that I just said which is that it's deterministic um
[00:41:45] said which is that it's deterministic um and that there is a single optimal
[00:41:47] and that there is a single optimal policy uh raise your hand if you said
[00:41:50] policy uh raise your hand if you said the policy once it doesn't change it can
[00:41:52] the policy once it doesn't change it can never change again that's the correct
[00:41:54] never change again that's the correct answer okay does somebody want to
[00:41:55] answer okay does somebody want to explain why
[00:42:01] you're all
[00:42:02] you're all correct yeah what's your remind me your
[00:42:05] correct yeah what's your remind me your name um it kind of intuitively made
[00:42:08] name um it kind of intuitively made sense in the sense of like you're doing
[00:42:10] sense in the sense of like you're doing the expected value so like you're
[00:42:12] the expected value so like you're summing over all or you're over all the
[00:42:15] summing over all or you're over all the actions even if there's like St cacity
[00:42:17] actions even if there's like St cacity in the system you're still taking the
[00:42:19] in the system you're still taking the average value so like didn't change
[00:42:23] average value so like didn't change before won change again yeah you are
[00:42:25] before won change again yeah you are taking those and so definitely along
[00:42:26] taking those and so definitely along those line so if we look at sort of what
[00:42:28] those line so if we look at sort of what was the definition of the policy
[00:42:31] was the definition of the policy Improvement step let me just go a couple
[00:42:33] Improvement step let me just go a couple slides back okay so what we said is we
[00:42:35] slides back okay so what we said is we computed the Q function and then we
[00:42:38] computed the Q function and then we extracted a new
[00:42:40] extracted a new policy if Pi I + 1 is the same as Pi I
[00:42:45] policy if Pi I + 1 is the same as Pi I is Q of Pi I + 1 equal to Q of I Pi I +
[00:42:49] is Q of Pi I + 1 equal to Q of I Pi I + 1 or I probably said that wrong to many
[00:42:51] 1 or I probably said that wrong to many eyes let me just so the question is
[00:42:55] eyes let me just so the question is if if pi I is equal to Pi I + 1 is Q Pi
[00:43:01] if if pi I is equal to Pi I + 1 is Q Pi I = to Q Pi I +
[00:43:07] 1 so if it's the same policy do they
[00:43:09] 1 so if it's the same policy do they have the same Q function yeah is he men
[00:43:12] have the same Q function yeah is he men Nots okay so if your policy hasn't
[00:43:14] Nots okay so if your policy hasn't changed meaning that your old policy is
[00:43:16] changed meaning that your old policy is the same as your new policy then Q pii
[00:43:19] the same as your new policy then Q pii is equal to Q Pi I + 1 which means that
[00:43:22] is equal to Q Pi I + 1 which means that when you do this for Q Pi I + 1 and then
[00:43:24] when you do this for Q Pi I + 1 and then try to extract a policy it'll be exactly
[00:43:26] try to extract a policy it'll be exactly the same
[00:43:27] the same so once you're stuck there it'll be
[00:43:29] so once you're stuck there it'll be you'll stuck forever now if you have
[00:43:32] you'll stuck forever now if you have ties it's more complicated so if you
[00:43:34] ties it's more complicated so if you have multiple actions that can achieve
[00:43:36] have multiple actions that can achieve the same Q function depends how you
[00:43:38] the same Q function depends how you break them if you break them
[00:43:39] break them if you break them deterministically you'll stay in the
[00:43:40] deterministically you'll stay in the same place if not you may sort of
[00:43:42] same place if not you may sort of oscillate between all the policies which
[00:43:43] oscillate between all the policies which are optimal otherwise known as all the
[00:43:46] are optimal otherwise known as all the policies for which they have the same Q
[00:43:48] policies for which they have the same Q Pi I okay but in the simpler case that I
[00:43:52] Pi I okay but in the simpler case that I mentioned once you've got to that single
[00:43:53] mentioned once you've got to that single policy you won't ever change
[00:43:57] policy you won't ever change and what that means is given that we
[00:44:00] and what that means is given that we also only have a finite number of
[00:44:01] also only have a finite number of policies if it's deterministic so
[00:44:04] policies if it's deterministic so assuming if we stick to determinance so
[00:44:06] assuming if we stick to determinance so I'll just say
[00:44:08] I'll just say no if Pi star is unique okay for all s
[00:44:15] no if Pi star is unique okay for all s so that means for every state there's a
[00:44:16] so that means for every state there's a unique optimal action um is there
[00:44:19] unique optimal action um is there maximum number of iterations for policy
[00:44:20] maximum number of iterations for policy iteration if you have deterministic
[00:44:22] iteration if you have deterministic policies there's only a of the S
[00:44:24] policies there's only a of the S policies as everyone was saying before
[00:44:26] policies as everyone was saying before which is great um and so since the
[00:44:28] which is great um and so since the policy Improvement step either improves
[00:44:31] policy Improvement step either improves the value of your policy or halts that
[00:44:33] the value of your policy or halts that means you only go through each policy
[00:44:36] means you only go through each policy once and at most once right like there
[00:44:39] once and at most once right like there might be some you never bother to go
[00:44:41] might be some you never bother to go through and so that means that policy
[00:44:43] through and so that means that policy iteration will Halt and it will take at
[00:44:45] iteration will Halt and it will take at most a to the S policies if it takes a
[00:44:47] most a to the S policies if it takes a to the s that means that you evaluated
[00:44:49] to the s that means that you evaluated every single policy in generally you
[00:44:51] every single policy in generally you won't
[00:44:53] won't okay so this is um what shows that we
[00:44:56] okay so this is um what shows that we actually they do get this monotonic
[00:44:57] actually they do get this monotonic Improvement this is really nice with
[00:44:59] Improvement this is really nice with every single because you could imagine
[00:45:00] every single because you could imagine in cases where like there there's oh
[00:45:03] in cases where like there there's oh question yeah
[00:45:06] sure yet that we're going to do better
[00:45:09] sure yet that we're going to do better than random right like there's no we
[00:45:11] than random right like there's no we haven't guaranteed that whatever we
[00:45:13] haven't guaranteed that whatever we converge to is better than
[00:45:15] converge to is better than random oh we we've show we've proven
[00:45:17] random oh we we've show we've proven that we're going to get get to the
[00:45:19] that we're going to get get to the optimal policy and the optimal policy
[00:45:22] optimal policy and the optimal policy may be just random right because
[00:45:24] may be just random right because depending on the the environment you
[00:45:27] depending on the the environment you might just like there there is no you
[00:45:29] might just like there there is no you can't do better than random is that you
[00:45:31] can't do better than random is that you mean in terms of like how you design
[00:45:32] mean in terms of like how you design actions yeah so for example if it is the
[00:45:34] actions yeah so for example if it is the case that all of your actions have
[00:45:36] case that all of your actions have exactly the same reward doesn't matter
[00:45:38] exactly the same reward doesn't matter whether you act randomly or you follow a
[00:45:40] whether you act randomly or you follow a policy the value you get would be
[00:45:42] policy the value you get would be exactly the same as random whether or
[00:45:44] exactly the same as random whether or not you can do better than the random
[00:45:45] not you can do better than the random will depend on the domain the hope is in
[00:45:48] will depend on the domain the hope is in general we can do a lot
[00:45:51] better Okay so we've shown now that
[00:45:54] better Okay so we've shown now that here's an algorithm where as we do a
[00:45:55] here's an algorithm where as we do a more comp more and more computation we
[00:45:57] more comp more and more computation we get better and better policies and this
[00:45:59] get better and better policies and this is great because you may not actually
[00:46:00] is great because you may not actually want to go particularly if like the
[00:46:02] want to go particularly if like the state space is you know very large you
[00:46:03] state space is you know very large you may not want to go till where your
[00:46:05] may not want to go till where your policy entirely stops changing so if you
[00:46:07] policy entirely stops changing so if you have like an ending time time
[00:46:09] have like an ending time time requirement you can still guarantee that
[00:46:11] requirement you can still guarantee that like hey I'm getting better and better
[00:46:12] like hey I'm getting better and better and maybe I stop after 100 iterations or
[00:46:14] and maybe I stop after 100 iterations or a thousand iterations um and just use
[00:46:17] a thousand iterations um and just use that policy okay so this is one which
[00:46:19] that policy okay so this is one which has that nice monotony
[00:46:21] has that nice monotony guarante can you say what that is oh
[00:46:25] guarante can you say what that is oh sure yes and what's your name uh
[00:46:28] sure yes and what's your name uh yeah so a here is the number of actions
[00:46:30] yeah so a here is the number of actions and S here is the number of states so
[00:46:32] and S here is the number of states so the decision policy space is for every
[00:46:34] the decision policy space is for every state you could to pick one of the
[00:46:36] state you could to pick one of the actions so you multiply all of
[00:46:41] those
[00:46:42] those okay so so this also shows here about
[00:46:46] okay so so this also shows here about how yeah exactly what I said on the
[00:46:48] how yeah exactly what I said on the previous one that if your policy doesn't
[00:46:50] previous one that if your policy doesn't change it'll never change again
[00:46:53] change it'll never change again again me
[00:46:57] again me five star is
[00:47:00] five star is unique
[00:47:03] per okay so that's one way to go one way
[00:47:06] per okay so that's one way to go one way is that we can do policy iteration and
[00:47:08] is that we can do policy iteration and the interesting thing about policy
[00:47:09] the interesting thing about policy iteration is that at every time point
[00:47:11] iteration is that at every time point you have an explicit policy and what
[00:47:14] you have an explicit policy and what that policy tells you is how to act
[00:47:16] that policy tells you is how to act forever and when using that policy and
[00:47:19] forever and when using that policy and when you compute the Q value it says how
[00:47:22] when you compute the Q value it says how much rewards will you get if you take
[00:47:23] much rewards will you get if you take this action in this date and then follow
[00:47:24] this action in this date and then follow that other policy forever so again
[00:47:27] that other policy forever so again remember today we're in the infinite
[00:47:28] remember today we're in the infinite Horizon case for you know unless I
[00:47:31] Horizon case for you know unless I specify
[00:47:32] specify otherwise value iteration but but you
[00:47:34] otherwise value iteration but but you know along the way a lot of those
[00:47:36] know along the way a lot of those actions and the decisions we make may
[00:47:37] actions and the decisions we make may not be very good so your early policies
[00:47:39] not be very good so your early policies might be pretty bad we know we're
[00:47:41] might be pretty bad we know we're monotonically improving but the early
[00:47:42] monotonically improving but the early policies might be bad value iteration is
[00:47:45] policies might be bad value iteration is different the idea is that at every time
[00:47:48] different the idea is that at every time every iteration we're going to maintain
[00:47:49] every iteration we're going to maintain the optimal value of starting in a state
[00:47:52] the optimal value of starting in a state but as if we only get to make a finite
[00:47:54] but as if we only get to make a finite number of
[00:47:55] number of decisions so remember in in policy
[00:47:57] decisions so remember in in policy iteration we always have a policy and we
[00:48:00] iteration we always have a policy and we have the value of acting in it forever
[00:48:01] have the value of acting in it forever it just might not be very good value
[00:48:03] it just might not be very good value iteration is say is how what is the
[00:48:06] iteration is say is how what is the optimal thing for me to do if I can just
[00:48:07] optimal thing for me to do if I can just make one decision okay like I can take
[00:48:09] make one decision okay like I can take one step okay I'm going to figure out
[00:48:11] one step okay I'm going to figure out what the optimal thing is to do for one
[00:48:13] what the optimal thing is to do for one step now I'm going to imagine I get to
[00:48:14] step now I'm going to imagine I get to take two steps and I'm going to build on
[00:48:16] take two steps and I'm going to build on what I know I can do for one step and so
[00:48:18] what I know I can do for one step and so now build like the optimal thing to do
[00:48:20] now build like the optimal thing to do for two steps so the interesting thing
[00:48:22] for two steps so the interesting thing with value iteration is you always have
[00:48:24] with value iteration is you always have an optimal value but for the wrong
[00:48:26] an optimal value but for the wrong Horizon
[00:48:28] Horizon so one has a value for the full horiz
[00:48:30] so one has a value for the full horiz the you know the infinite Horizon it
[00:48:32] the you know the infinite Horizon it might be a bad policy the other one has
[00:48:35] might be a bad policy the other one has the optimal value and thing to do but
[00:48:37] the optimal value and thing to do but for the wrong
[00:48:38] for the wrong Horizon okay so and the idea in value
[00:48:41] Horizon okay so and the idea in value iteration is you just kind of keep going
[00:48:42] iteration is you just kind of keep going to longer and longer and longer episodes
[00:48:44] to longer and longer and longer episodes thinking of getting to do like H+ one
[00:48:46] thinking of getting to do like H+ one steps or h plus two steps and then you
[00:48:48] steps or h plus two steps and then you build upon your previous Solutions using
[00:48:50] build upon your previous Solutions using dynamic programming so let's see how to
[00:48:52] dynamic programming so let's see how to do that okay so this is where we get
[00:48:55] do that okay so this is where we get into the Bellman equation is is the sort
[00:48:57] into the Bellman equation is is the sort of the seminal work of um Richard belman
[00:49:00] of the seminal work of um Richard belman and the idea here is as we've said is
[00:49:02] and the idea here is as we've said is that for a particular policy we satisfy
[00:49:04] that for a particular policy we satisfy the Bellman
[00:49:05] the Bellman equation and we can for turn that into
[00:49:08] equation and we can for turn that into an
[00:49:10] an algorithm so in particular there's a
[00:49:12] algorithm so in particular there's a thing called the Bellman backup operator
[00:49:14] thing called the Bellman backup operator and what it says is if you give me a
[00:49:16] and what it says is if you give me a value function which right now we can
[00:49:18] value function which right now we can think of it just being a vector later
[00:49:19] think of it just being a vector later we'll get into function
[00:49:21] we'll get into function approximation and we do a Bellman backup
[00:49:23] approximation and we do a Bellman backup essentially it's like saying I knew you
[00:49:25] essentially it's like saying I knew you know I had a value fun function and I
[00:49:27] know I had a value fun function and I want to think about what should I do if
[00:49:28] want to think about what should I do if I get to do the best thing that
[00:49:29] I get to do the best thing that maximizes my immediate reward plus my
[00:49:32] maximizes my immediate reward plus my expected future reward given that value
[00:49:34] expected future reward given that value function okay so it says I'm going to
[00:49:37] function okay so it says I'm going to figure out if I take a Max over all the
[00:49:39] figure out if I take a Max over all the actions what's the reward at that State
[00:49:41] actions what's the reward at that State in the action plus the discounted sum of
[00:49:43] in the action plus the discounted sum of rewards using the value function you've
[00:49:45] rewards using the value function you've given
[00:49:47] given me and what that does is it need yields
[00:49:50] me and what that does is it need yields a new Vector of of values over all your
[00:49:53] a new Vector of of values over all your States so this is being done person
[00:49:56] States so this is being done person State okay and this is called the
[00:49:58] State okay and this is called the Bellman operator comes up all the time
[00:50:02] Bellman operator comes up all the time okay all right so how do we do value
[00:50:05] okay all right so how do we do value iteration well we're just going to do
[00:50:07] iteration well we're just going to do this
[00:50:09] this recursively so we're just going to Loop
[00:50:11] recursively so we're just going to Loop until we hit convergence um uh just as a
[00:50:14] until we hit convergence um uh just as a refresher this is the um L Infinity norm
[00:50:18] refresher this is the um L Infinity norm and what that means is that it is equal
[00:50:20] and what that means is that it is equal to the max over S V of
[00:50:25] to the max over S V of s right out more
[00:50:31] carefully equal to Max over s v k + 1 of
[00:50:37] carefully equal to Max over s v k + 1 of s minus VK of
[00:50:40] s minus VK of s what that just means is that if you
[00:50:42] s what that just means is that if you have two vectors you look for every
[00:50:44] have two vectors you look for every single entry and you find the entry in
[00:50:46] single entry and you find the entry in which those two vectors are most
[00:50:48] which those two vectors are most different and that's the L Infinity Norm
[00:50:50] different and that's the L Infinity Norm just as a refresher for some of you who
[00:50:52] just as a refresher for some of you who might not have seen it or not seen it
[00:50:53] might not have seen it or not seen it recently so what value iteration does is
[00:50:56] recently so what value iteration does is is we're just going to have a loop it's
[00:50:59] is we're just going to have a loop it's going to look very similar to what we
[00:51:00] going to look very similar to what we saw for the markof reward process we're
[00:51:02] saw for the markof reward process we're going to initialize our value function
[00:51:04] going to initialize our value function and then for each date we're just going
[00:51:05] and then for each date we're just going to do this Bellman
[00:51:07] to do this Bellman backup and so it's like we took our
[00:51:09] backup and so it's like we took our previous value function we do our
[00:51:10] previous value function we do our Bellman backup and we get a new value
[00:51:13] Bellman backup and we get a new value function we do this over and over and
[00:51:15] function we do this over and over and over again until our value function
[00:51:16] over again until our value function stops
[00:51:18] stops changing so for policy iteration we kept
[00:51:20] changing so for policy iteration we kept going until our policy stops changing
[00:51:22] going until our policy stops changing here we keep going until our value
[00:51:24] here we keep going until our value function stops changing
[00:51:26] function stops changing and what that that condition means is it
[00:51:29] and what that that condition means is it says I keep going until the difference
[00:51:31] says I keep going until the difference between my old value of estate and my
[00:51:33] between my old value of estate and my new value of estate is really small for
[00:51:34] new value of estate is really small for all the
[00:51:35] all the states yeah and REM me your
[00:51:38] states yeah and REM me your name I have a question how to conduct
[00:51:42] name I have a question how to conduct this value iteration to you just said it
[00:51:45] this value iteration to you just said it works with finite Horizon and why policy
[00:51:49] works with finite Horizon and why policy iteration works with yeah good question
[00:51:51] iteration works with yeah good question so what you could think of this as so
[00:51:53] so what you could think of this as so great question the beginning if you
[00:51:55] great question the beginning if you don't get to make any decisions the
[00:51:57] don't get to make any decisions the expected discount of sum rewards you get
[00:51:59] expected discount of sum rewards you get from many states is zero you didn't get
[00:52:01] from many states is zero you didn't get to make any decisions you never got any
[00:52:03] to make any decisions you never got any reward the first round of this it's like
[00:52:05] reward the first round of this it's like you're saying okay I get I get zero
[00:52:08] you're saying okay I get I get zero reward if I don't make any
[00:52:09] reward if I don't make any decisions this would K would be you know
[00:52:12] decisions this would K would be you know K would be one here for the next round
[00:52:15] K would be one here for the next round so we'd say like if I get to make one
[00:52:16] so we'd say like if I get to make one decision then I would take a Max over a
[00:52:19] decision then I would take a Max over a my reward times discount Factor time
[00:52:22] my reward times discount Factor time zero so it's now saying what is the best
[00:52:26] zero so it's now saying what is the best thing I get to do if I get to make one
[00:52:29] thing I get to do if I get to make one decision okay so what this will be is on
[00:52:31] decision okay so what this will be is on the on the first round this will just be
[00:52:34] the on the first round this will just be equal to so if V if V is equal to zero
[00:52:37] equal to so if V if V is equal to zero for all s then what we would get when we
[00:52:39] for all s then what we would get when we do this backup we get VK + 1 is just
[00:52:43] do this backup we get VK + 1 is just equal to of let me put over s is equal
[00:52:47] equal to of let me put over s is equal to Max over a r of sa
[00:52:50] to Max over a r of sa a
[00:52:51] a because this part will be zero if your
[00:52:54] because this part will be zero if your value is zero so so now this is like
[00:52:57] value is zero so so now this is like saying okay before I get no reward
[00:52:59] saying okay before I get no reward because I make no decisions now what's
[00:53:00] because I make no decisions now what's the best thing I should do if I get to
[00:53:02] the best thing I should do if I get to make one decision the next round I'll
[00:53:04] make one decision the next round I'll say what if I get to make one decision
[00:53:06] say what if I get to make one decision now and then plus this will get plugged
[00:53:09] now and then plus this will get plugged into your
[00:53:10] into your value
[00:53:13] value yeah um so so the expression that we're
[00:53:15] yeah um so so the expression that we're plugging into the max function is that
[00:53:18] plugging into the max function is that is that the same as like Q of sa like a
[00:53:21] is that the same as like Q of sa like a good question so in general that's going
[00:53:22] good question so in general that's going to be um max over a q of
[00:53:26] to be um max over a q of because yeah because here we're we're
[00:53:29] because yeah because here we're we're requiring ourselves to be take the max
[00:53:31] requiring ourselves to be take the max over the actions we take great question
[00:53:34] over the actions we take great question yeah uh in policy iteration we were also
[00:53:37] yeah uh in policy iteration we were also initializing the values of we were in
[00:53:40] initializing the values of we were in the policy iteration we were just
[00:53:42] the policy iteration we were just randomly initializing our policy so
[00:53:44] randomly initializing our policy so we're saying like in this state you go
[00:53:46] we're saying like in this state you go left in this state you go right Etc and
[00:53:48] left in this state you go right Etc and then we were evaluating the value of
[00:53:49] then we were evaluating the value of that we could when we do that evaluation
[00:53:52] that we could when we do that evaluation yes in that part we were setting b equal
[00:53:53] yes in that part we were setting b equal to zero and then doing this iteratively
[00:54:06] if we had done this in well we can do
[00:54:08] if we had done this in well we can do that we can do this as part of the
[00:54:09] that we can do this as part of the policy evaluation um for policy
[00:54:11] policy evaluation um for policy iteration but what do you mean we would
[00:54:13] iteration but what do you mean we would be able to detect
[00:54:18] Cy comp successive policies
[00:54:27] that's right so it's a good point say if
[00:54:28] that's right so it's a good point say if inside of the policy iteration one
[00:54:30] inside of the policy iteration one instead of just um halting when your uh
[00:54:33] instead of just um halting when your uh policy has stopped changing you could
[00:54:34] policy has stopped changing you could also hold your value function and stop
[00:54:36] also hold your value function and stop changing very cute
[00:54:38] changing very cute function yeah you say yeah and what's
[00:54:41] function yeah you say yeah and what's your
[00:54:49] [Music]
[00:54:51] [Music] name yeah it's a great question um I
[00:54:55] name yeah it's a great question um I don't I'll look that up for next week to
[00:54:57] don't I'll look that up for next week to my knowledge Al there isn't over that in
[00:55:00] my knowledge Al there isn't over that in one that would not be instance dependent
[00:55:03] one that would not be instance dependent um impr practice policy search is very
[00:55:06] um impr practice policy search is very very
[00:55:14] popular I think part of it maybe I
[00:55:17] popular I think part of it maybe I think part of it is probably that um it
[00:55:21] think part of it is probably that um it also often has this nice monotonic
[00:55:24] also often has this nice monotonic Improvement value iteration does does
[00:55:26] Improvement value iteration does does not necessarily have the monotonic
[00:55:27] not necessarily have the monotonic Improvement requirement here so it is
[00:55:31] Improvement requirement here so it is always the optimal thing to do for the
[00:55:32] always the optimal thing to do for the wrong Horizon whereas the other one says
[00:55:35] wrong Horizon whereas the other one says it may not be optimal for ages but it
[00:55:37] it may not be optimal for ages but it will always be monotonically
[00:55:41] improving great questions okay let's see
[00:55:44] improving great questions okay let's see what the properties are for Value
[00:55:45] what the properties are for Value iteration because these are a really
[00:55:47] iteration because these are a really useful great questions and we'll see why
[00:55:49] useful great questions and we'll see why this whole thing ends up working so just
[00:55:52] this whole thing ends up working so just I want to highlight here you could think
[00:55:53] I want to highlight here you could think of belt policy iteration also as Bellman
[00:55:55] of belt policy iteration also as Bellman operations and I think this good to what
[00:55:56] operations and I think this good to what your question just out too so the bman
[00:55:59] your question just out too so the bman back operator for a particular policy Pi
[00:56:01] back operator for a particular policy Pi is defined as follows you don't you see
[00:56:03] is defined as follows you don't you see you don't see the max anymore you just
[00:56:04] you don't see the max anymore you just are committing to doing a particular
[00:56:06] are committing to doing a particular policy and then policy evaluations
[00:56:08] policy and then policy evaluations amounts to Computing the fix point and
[00:56:10] amounts to Computing the fix point and I'll Define that more formally in a
[00:56:11] I'll Define that more formally in a second um and so to do policy evaluation
[00:56:14] second um and so to do policy evaluation you just repeatedly apply this operator
[00:56:17] you just repeatedly apply this operator until V stops changing that was the
[00:56:19] until V stops changing that was the iterative algorithm we saw before just
[00:56:21] iterative algorithm we saw before just with different
[00:56:23] with different notation all right let's start to talk
[00:56:25] notation all right let's start to talk soon second about fixed point so that
[00:56:29] soon second about fixed point so that and then what we would say here is when
[00:56:30] and then what we would say here is when we do policy another way to do policy
[00:56:32] we do policy another way to do policy Improvement is to explicitly do another
[00:56:34] Improvement is to explicitly do another backup but take the arc Max instead of
[00:56:36] backup but take the arc Max instead of the max okay so it's the only difference
[00:56:38] the max okay so it's the only difference this is the same as what you're doing
[00:56:39] this is the same as what you're doing for the Q function so this is
[00:56:43] for the Q function so this is Q Pi K of
[00:56:48] sa I'm just showing you different
[00:56:49] sa I'm just showing you different notations but the same thing and also
[00:56:51] notations but the same thing and also help you will talk about the Bellman
[00:56:52] help you will talk about the Bellman backup for a particular
[00:56:54] backup for a particular policy but normally when people say bman
[00:56:57] policy but normally when people say bman backups they mean for the optimal all
[00:56:58] backups they mean for the optimal all right so let's just go back to Value
[00:57:00] right so let's just go back to Value iteration because while I've told you
[00:57:01] iteration because while I've told you how to compute a value function I
[00:57:03] how to compute a value function I haven't told you how to get out a policy
[00:57:05] haven't told you how to get out a policy from it so the standard way to do this
[00:57:08] from it so the standard way to do this would be you would go through this
[00:57:09] would be you would go through this process and then you would do it say one
[00:57:11] process and then you would do it say one more time and extract the argmax instead
[00:57:13] more time and extract the argmax instead of the max to actually get your
[00:57:15] of the max to actually get your policy so normally in this case you
[00:57:17] policy so normally in this case you don't bother to compute a policy along
[00:57:19] don't bother to compute a policy along the way you just do value iteration a
[00:57:21] the way you just do value iteration a bunch of times then at some point you
[00:57:22] bunch of times then at some point you extract a policy
[00:57:26] all right let's see about some
[00:57:28] all right let's see about some properties of this so why this is a good
[00:57:30] properties of this so why this is a good thing I've already told we've already
[00:57:31] thing I've already told we've already seen that policy iteration is guaranteed
[00:57:33] seen that policy iteration is guaranteed to converge because there's only a
[00:57:34] to converge because there's only a finite number of policies you never
[00:57:36] finite number of policies you never repeat a policy and so either you're at
[00:57:38] repeat a policy and so either you're at the optimal policy already or you keep
[00:57:40] the optimal policy already or you keep improving for Value iteration it may not
[00:57:43] improving for Value iteration it may not be clear yet that this should
[00:57:45] be clear yet that this should converge so I'm first going to just
[00:57:47] converge so I'm first going to just Define a contraction operator so let's
[00:57:49] Define a contraction operator so let's let OB be an operator like the Bellman
[00:57:51] let OB be an operator like the Bellman backup so if you can just think of as
[00:57:53] backup so if you can just think of as like an algebraic equation if this is
[00:57:54] like an algebraic equation if this is something you know if you haven't seen
[00:57:56] something you know if you haven't seen before which is totally fine um and then
[00:57:58] before which is totally fine um and then this is just going to denote any Norm so
[00:58:00] this is just going to denote any Norm so like the L Infinity Norm or
[00:58:03] like the L Infinity Norm or others if when you apply the operator to
[00:58:06] others if when you apply the operator to two different say value functions we can
[00:58:09] two different say value functions we can just think of these here as
[00:58:11] just think of these here as vectors and that distance after you
[00:58:13] vectors and that distance after you apply the operator is smaller than the
[00:58:15] apply the operator is smaller than the distance before then it's a contraction
[00:58:18] distance before then it's a contraction operator so just give it a bit of
[00:58:19] operator so just give it a bit of intuition for this in case contraction
[00:58:21] intuition for this in case contraction operators isn't something you've seen
[00:58:22] operators isn't something you've seen before if you think about having two
[00:58:25] before if you think about having two value functions and then and there is
[00:58:26] value functions and then and there is some states on which they really differ
[00:58:28] some states on which they really differ in their value what this says is that if
[00:58:30] in their value what this says is that if you then apply an operator to them and
[00:58:32] you then apply an operator to them and we're going to prove that the Bellman
[00:58:33] we're going to prove that the Bellman operator is one of them that afterwards
[00:58:35] operator is one of them that afterwards they get closer together so that Max
[00:58:37] they get closer together so that Max difference between the states is smaller
[00:58:39] difference between the states is smaller afterwards yeah is
[00:58:41] afterwards yeah is it uh yeah um the is this like a if or
[00:58:46] it uh yeah um the is this like a if or just if like can there be contraction
[00:58:49] just if like can there be contraction operators that don't satisfy this we're
[00:58:52] operators that don't satisfy this we're going to look at this um as specifically
[00:58:54] going to look at this um as specifically Uh Oh you mean are there so my um I'd
[00:58:58] Uh Oh you mean are there so my um I'd have to check I'm not an expert in all
[00:58:59] have to check I'm not an expert in all of contraction operators so I'll say
[00:59:01] of contraction operators so I'll say what I will show is that the Bellman
[00:59:02] what I will show is that the Bellman operator satisfies this statement and
[00:59:04] operator satisfies this statement and therefore then we can show that we're
[00:59:06] therefore then we can show that we're going to converge to a fixed
[00:59:10] Point all right so particularly under a
[00:59:13] Point all right so particularly under a couple minor assumptions your discount
[00:59:16] couple minor assumptions your discount Factor being less than one or you end up
[00:59:18] Factor being less than one or you end up in a terminal state with probability one
[00:59:20] in a terminal state with probability one essentially both of these make sure that
[00:59:21] essentially both of these make sure that your your expected sum of discounted
[00:59:23] your your expected sum of discounted rewards is
[00:59:24] rewards is bounded then the Bellman backup where
[00:59:27] bounded then the Bellman backup where you do this Max is a
[00:59:29] you do this Max is a contraction which means that your
[00:59:31] contraction which means that your distance is going to shrink and Shrink
[00:59:33] distance is going to shrink and Shrink between two value functions so your v k
[00:59:36] between two value functions so your v k + 1 versus its difference to V K that
[00:59:40] + 1 versus its difference to V K that distance in terms of the max difference
[00:59:41] distance in terms of the max difference in any of the states is going to get
[00:59:43] in any of the states is going to get smaller and we we'll go through that now
[00:59:46] smaller and we we'll go through that now okay so this is proving that we end up
[00:59:50] okay so this is proving that we end up getting the um this is a contraction
[00:59:53] getting the um this is a contraction operator so the bell bone backup is a
[00:59:54] operator so the bell bone backup is a contraction operator on V for gamma less
[00:59:56] contraction operator on V for gamma less than one let me just make this large
[00:59:59] than one let me just make this large okay so we're going to use the infinity
[01:00:01] okay so we're going to use the infinity Norm which again is just same for where
[01:00:03] Norm which again is just same for where is the max difference in the values for
[01:00:05] is the max difference in the values for any two states and what I'm defining
[01:00:07] any two states and what I'm defining here is two different value functions so
[01:00:10] here is two different value functions so this could be anything what I'm going to
[01:00:12] this could be anything what I'm going to try to show is that after you do the
[01:00:13] try to show is that after you do the Bellman operator that can be no larger
[01:00:15] Bellman operator that can be no larger than the max difference before you did
[01:00:17] than the max difference before you did the Bellman backup
[01:00:19] the Bellman backup okay all right so what we have here this
[01:00:23] okay all right so what we have here this is the first inequality so this is
[01:00:24] is the first inequality so this is important what I'm going to say is right
[01:00:26] important what I'm going to say is right now so this is just the definition of
[01:00:28] now so this is just the definition of the Bellman backup operator what you can
[01:00:31] the Bellman backup operator what you can see here is I have two different
[01:00:32] see here is I have two different Maxes because I'm going to do the max
[01:00:35] Maxes because I'm going to do the max over a for the first value function and
[01:00:37] over a for the first value function and a Max over a prime for the other value
[01:00:40] a Max over a prime for the other value function what I'm going to say now is if
[01:00:42] function what I'm going to say now is if you do that instead this has to be less
[01:00:45] you do that instead this has to be less than or equal to if you pulled the max a
[01:00:48] than or equal to if you pulled the max a out and you required both of them to use
[01:00:52] out and you required both of them to use the same action
[01:01:00] and why is this true because essentially
[01:01:02] and why is this true because essentially what we're allowing here is we're
[01:01:04] what we're allowing here is we're allowing us before we could pick
[01:01:07] allowing us before we could pick different actions for both bman backups
[01:01:12] different actions for both bman backups and now we can pick
[01:01:21] one okay so that means that instead of
[01:01:24] one okay so that means that instead of getting to maximize the second thing
[01:01:26] getting to maximize the second thing separately we're just going to try to
[01:01:26] separately we're just going to try to maximize the difference so that's the
[01:01:28] maximize the difference so that's the first place this less than or equal to
[01:01:29] first place this less than or equal to is going to come in okay once we do that
[01:01:32] is going to come in okay once we do that now everything's taking the same action
[01:01:33] now everything's taking the same action on the same time step so we can um get
[01:01:35] on the same time step so we can um get rid of these because they're
[01:01:39] rid of these because they're identical
[01:01:41] identical okay so we can just say this is just
[01:01:43] okay so we can just say this is just exactly equal to Max a I'm going to pull
[01:01:46] exactly equal to Max a I'm going to pull out the discount factor of sum over S
[01:01:49] out the discount factor of sum over S Prime probability of S Prime given
[01:01:52] Prime probability of S Prime given sa of VK of S Prime VJ of
[01:01:59] sa of VK of S Prime VJ of s okay now again what I'm going to do is
[01:02:02] s okay now again what I'm going to do is I'm going to bound this and I'm going to
[01:02:04] I'm going to bound this and I'm going to say the difference between the two value
[01:02:06] say the difference between the two value functions that any state is always less
[01:02:08] functions that any state is always less than or equal to their Max difference
[01:02:11] than or equal to their Max difference across all the states okay so this is
[01:02:13] across all the states okay so this is less than or equal to Max
[01:02:15] less than or equal to Max a gamma Su over S Prime probability of S
[01:02:20] a gamma Su over S Prime probability of S Prime given an
[01:02:22] Prime given an sa VK minus VJ
[01:02:27] okay because the difference between any
[01:02:30] okay because the difference between any between particular States is always less
[01:02:31] between particular States is always less than the max difference between any of
[01:02:33] than the max difference between any of the states so I upper bounded it using
[01:02:36] the states so I upper bounded it using this expression okay but now this term
[01:02:39] this expression okay but now this term Now does not depend on States and so I
[01:02:43] Now does not depend on States and so I can take it out of the sum okay this is
[01:02:45] can take it out of the sum okay this is just some constant it's like seven so
[01:02:47] just some constant it's like seven so this is equal to Max over a gamma
[01:03:03] but this is just a transition model and
[01:03:05] but this is just a transition model and the probability that we go to Su state
[01:03:07] the probability that we go to Su state has to sum up to one if we sum over all
[01:03:10] has to sum up to one if we sum over all next States because for any state in
[01:03:12] next States because for any state in action you're in you always have to go
[01:03:13] action you're in you always have to go to Su next state so this is just equal
[01:03:15] to Su next state so this is just equal to
[01:03:16] to one so we get that this is just equal to
[01:03:20] one so we get that this is just equal to Max over
[01:03:21] Max over a and now there's no more dependence on
[01:03:24] a and now there's no more dependence on a so it just disappears
[01:03:28] okay so what that said is that the
[01:03:30] okay so what that said is that the distance the max difference between the
[01:03:33] distance the max difference between the Bellman backed
[01:03:35] Bellman backed up value functions we get from starting
[01:03:38] up value functions we get from starting with two different value functions has
[01:03:39] with two different value functions has to be no
[01:03:41] to be no larger than the max difference between
[01:03:43] larger than the max difference between the value functions before times
[01:03:49] gamma and if your gamma's less than one
[01:03:51] gamma and if your gamma's less than one that means you're strictly
[01:03:53] that means you're strictly Contracting because it means that that
[01:03:55] Contracting because it means that that Max difference has to be smaller than it
[01:03:57] Max difference has to be smaller than it was before it would be like 0.9 *
[01:03:59] was before it would be like 0.9 * whatever the at most 09 times whatever
[01:04:01] whatever the at most 09 times whatever the distance was
[01:04:03] the distance was before okay so that's really really cool
[01:04:06] before okay so that's really really cool because that means now that if we apply
[01:04:08] because that means now that if we apply value iteration we're repeatedly doing
[01:04:09] value iteration we're repeatedly doing the bman backup we're shrinking the
[01:04:12] the bman backup we're shrinking the distance so if you think of having a
[01:04:13] distance so if you think of having a series of value functions so you've got
[01:04:15] series of value functions so you've got like v0 and V1 and V2 and V3 V4 dot dot
[01:04:20] like v0 and V1 and V2 and V3 V4 dot dot dot can think of what this distance
[01:04:23] dot can think of what this distance is okay and what this is saying saying
[01:04:26] is okay and what this is saying saying is that these
[01:04:27] is that these distances are going to be shrinking over
[01:04:31] distances are going to be shrinking over time and I've told you before that the
[01:04:34] time and I've told you before that the value function is unique so that means
[01:04:36] value function is unique so that means as you shrink and shrink and shrink and
[01:04:37] as you shrink and shrink and shrink and Shrink this is eventually going to
[01:04:40] Shrink this is eventually going to become a unique value
[01:04:43] become a unique value function because if there were you can
[01:04:46] function because if there were you can think about it too if there were two
[01:04:47] think about it too if there were two different value functions you can think
[01:04:48] different value functions you can think about what would happen after you do a
[01:04:50] about what would happen after you do a BMA backup
[01:04:52] BMA backup operator they different
[01:04:56] operator they different okay so this proves it's a
[01:04:59] okay so this proves it's a contraction yeah and just to note this
[01:05:01] contraction yeah and just to note this here even if all the inequalities are
[01:05:03] here even if all the inequalities are equalities is still a contraction if
[01:05:04] equalities is still a contraction if gamma is less than one still kind of
[01:05:06] gamma is less than one still kind of making
[01:05:07] making progress all right so here's some
[01:05:09] progress all right so here's some thoughts in case you want to think about
[01:05:11] thoughts in case you want to think about this more to prove the valuator
[01:05:13] this more to prove the valuator convergence to Unique solution for
[01:05:14] convergence to Unique solution for discret and state action spaces whether
[01:05:16] discret and state action spaces whether initialization matters at all and is the
[01:05:20] initialization matters at all and is the value of the policy extracted from value
[01:05:21] value of the policy extracted from value iteration at each round guaranteed to
[01:05:23] iteration at each round guaranteed to monotonically improve
[01:05:26] monotonically improve so these are all great things to think
[01:05:29] so these are all great things to think about okay so let's go back to sort of
[01:05:31] about okay so let's go back to sort of more practically this is then value
[01:05:34] more practically this is then value iteration for finite well actually I'll
[01:05:35] iteration for finite well actually I'll pause here in case anybody has a
[01:05:37] pause here in case anybody has a question about the proof yes under my
[01:05:39] question about the proof yes under my mirr
[01:05:39] mirr name can you go back to yeah um I I
[01:05:44] name can you go back to yeah um I I understand except the first one where we
[01:05:47] understand except the first one where we like take the max out of the norm Max
[01:05:50] like take the max out of the norm Max action why is that greater than having
[01:05:53] action why is that greater than having separate inside yeah good question so um
[01:05:57] separate inside yeah good question so um this uh what this is saying here is you
[01:06:00] this uh what this is saying here is you know we have you could think of this a q
[01:06:01] know we have you could think of this a q function so we get to pick a Max for
[01:06:03] function so we get to pick a Max for that Q function and then we subtract off
[01:06:06] that Q function and then we subtract off the max over another q function and you
[01:06:09] the max over another q function and you could imagine that you could think of
[01:06:11] could imagine that you could think of there being lots of different pairs of
[01:06:13] there being lots of different pairs of actions in this case and either this Max
[01:06:16] actions in this case and either this Max is the same as this one so it's actually
[01:06:19] is the same as this one so it's actually let say in particular concretely that
[01:06:21] let say in particular concretely that this is like gives you a one that's
[01:06:24] this is like gives you a one that's those one so this one is either A1 or
[01:06:27] those one so this one is either A1 or another
[01:06:29] another a if it's another a that's just because
[01:06:32] a if it's another a that's just because this is actually larger than what Qs of
[01:06:36] this is actually larger than what Qs of like so let me just write this out just
[01:06:37] like so let me just write this out just in case so we can think of there as
[01:06:39] in case so we can think of there as either being qf A1
[01:06:43] either being qf A1 under this or QJ of s a where a is not
[01:06:50] under this or QJ of s a where a is not equal to A1 okay so either this Max is
[01:06:54] equal to A1 okay so either this Max is exactly the same as this one in which
[01:06:56] exactly the same as this one in which case this is equal or this is different
[01:06:59] case this is equal or this is different and the only time it would be different
[01:07:00] and the only time it would be different is if that value was actually larger
[01:07:02] is if that value was actually larger than the value of Q sa1 and if this was
[01:07:05] than the value of Q sa1 and if this was larger this difference would be smaller
[01:07:08] larger this difference would be smaller because you'd be subtracting off a
[01:07:09] because you'd be subtracting off a larger
[01:07:12] value so that's why we can turn this
[01:07:15] value so that's why we can turn this into an inequality it's either the same
[01:07:17] into an inequality it's either the same if they happened to have both picked the
[01:07:18] if they happened to have both picked the same action or it would have picked
[01:07:19] same action or it would have picked another action for which that whole
[01:07:21] another action for which that whole difference would have been
[01:07:22] difference would have been smaller good question
[01:07:27] okay yeah can you go back to the like
[01:07:29] okay yeah can you go back to the like the questions you were posing yep yeah
[01:07:32] the questions you were posing yep yeah um is the value of the third question
[01:07:35] um is the value of the third question out there is that is the value of the
[01:07:37] out there is that is the value of the policy extracted from value
[01:07:39] policy extracted from value iteration aren't we isn't isn't that
[01:07:44] iteration aren't we isn't isn't that like implicit within value iteration
[01:07:45] like implicit within value iteration that it like with each uh like each new
[01:07:49] that it like with each uh like each new value function is better than the
[01:07:50] value function is better than the previous one and therefore the policy
[01:07:52] previous one and therefore the policy will also be better or that's a question
[01:07:54] will also be better or that's a question yeah
[01:07:56] yeah the question whether it is guaranteeing
[01:07:57] the question whether it is guaranteeing that we have not proven anything about
[01:07:59] that we have not proven anything about that yet we Prov that for policy
[01:08:01] that yet we Prov that for policy duration but this is just to think about
[01:08:03] duration but this is just to think about it in this
[01:08:05] it in this case Okay so let's go now anybody else
[01:08:07] case Okay so let's go now anybody else have a question on the
[01:08:11] proof okay all right so one thing I just
[01:08:13] proof okay all right so one thing I just want to mention briefly and it'll come
[01:08:15] want to mention briefly and it'll come up on the homework is thinking about
[01:08:16] up on the homework is thinking about this for finite Horizons so most of
[01:08:18] this for finite Horizons so most of today almost all of today we've assumed
[01:08:20] today almost all of today we've assumed that we get to act forever so we have an
[01:08:22] that we get to act forever so we have an infinite Horizon but there will
[01:08:23] infinite Horizon but there will certainly be cases where there's a
[01:08:25] certainly be cases where there's a finite
[01:08:26] finite Horizon and in this case this goes back
[01:08:29] Horizon and in this case this goes back to the sort of thinking of value
[01:08:30] to the sort of thinking of value iteration is just Computing the optimal
[01:08:32] iteration is just Computing the optimal value for each
[01:08:35] value for each Horizon so if we have K 1 to H H being
[01:08:39] Horizon so if we have K 1 to H H being like the max Horizon you want to compute
[01:08:40] like the max Horizon you want to compute for then for each date at each round you
[01:08:43] for then for each date at each round you would have a a value function k+ 1 which
[01:08:46] would have a a value function k+ 1 which tells you sort of how many decisions you
[01:08:47] tells you sort of how many decisions you make what your horizon is um and we
[01:08:51] make what your horizon is um and we would just do this back up so this looks
[01:08:52] would just do this back up so this looks exactly the same as what we saw before
[01:08:54] exactly the same as what we saw before but now you could also get a policy here
[01:08:57] but now you could also get a policy here so this would be the policy associated
[01:08:58] so this would be the policy associated with that value function of what is the
[01:09:01] with that value function of what is the argmax
[01:09:02] argmax action so it compute a series of
[01:09:06] action so it compute a series of policies one for each
[01:09:08] policies one for each step
[01:09:12] okay one other thing I want to mention
[01:09:14] okay one other thing I want to mention here and there'll be um we'll talk about
[01:09:16] here and there'll be um we'll talk about this on the homework too um one other
[01:09:19] this on the homework too um one other thing I I want to mention here is that
[01:09:21] thing I I want to mention here is that you can also just simulate the value um
[01:09:24] you can also just simulate the value um and then uh of a particular policy so
[01:09:27] and then uh of a particular policy so this is also really popular once we
[01:09:28] this is also really popular once we start to get into really big state
[01:09:30] start to get into really big state spaces so if we think of the fact that
[01:09:32] spaces so if we think of the fact that in a lot of these algorithms we're doing
[01:09:33] in a lot of these algorithms we're doing some sort of policy evaluation one thing
[01:09:36] some sort of policy evaluation one thing you could do is you just take your
[01:09:38] you could do is you just take your policy and you know what your Dynamics
[01:09:39] policy and you know what your Dynamics model is you know what your reward is
[01:09:41] model is you know what your reward is and you just roll it out so if you're in
[01:09:43] and you just roll it out so if you're in a state you simulate what the next state
[01:09:45] a state you simulate what the next state might be then you get a reward um so you
[01:09:49] might be then you get a reward um so you can just generate a really large number
[01:09:50] can just generate a really large number of episodes and then just average them
[01:09:53] of episodes and then just average them so like I'm just like oh how you know
[01:09:54] so like I'm just like oh how you know how good is this policy if your boss
[01:09:56] how good is this policy if your boss asking you you just like run it on 100
[01:09:57] asking you you just like run it on 100 people you average their rewards and
[01:09:59] people you average their rewards and then you're
[01:10:00] then you're done and this is something that becomes
[01:10:02] done and this is something that becomes really popular when it starts to be say
[01:10:04] really popular when it starts to be say hard to write down what that Dynamics
[01:10:06] hard to write down what that Dynamics model is explicitly or do that kind of
[01:10:08] model is explicitly or do that kind of sum over S Prime but it's really easy to
[01:10:11] sum over S Prime but it's really easy to sample and I'll note for that that um
[01:10:13] sample and I'll note for that that um you could use concentration inequalities
[01:10:15] you could use concentration inequalities like hting inequality um and uh
[01:10:17] like hting inequality um and uh Bernstein for those of you familiar with
[01:10:19] Bernstein for those of you familiar with them to bound how many how much data do
[01:10:22] them to bound how many how much data do you need for this estimate to be close
[01:10:24] you need for this estimate to be close to the true
[01:10:26] to the true one and the great thing is that it's not
[01:10:28] one and the great thing is that it's not that many so like if you have an
[01:10:30] that many so like if you have an enormous State space like I don't know
[01:10:32] enormous State space like I don't know your Amazon or something like that or
[01:10:33] your Amazon or something like that or you've got patient data and it's
[01:10:35] you've got patient data and it's incredibly High dimensional um you don't
[01:10:37] incredibly High dimensional um you don't have to do that huge sum over S Prime
[01:10:39] have to do that huge sum over S Prime you can just sample and you're your
[01:10:41] you can just sample and you're your accuracy generally improves by one over
[01:10:43] accuracy generally improves by one over square root n the number of samples
[01:10:45] square root n the number of samples you're doing and the nice thing is that
[01:10:47] you're doing and the nice thing is that this also requires no assumption about
[01:10:49] this also requires no assumption about the Markoff structure so you might have
[01:10:51] the Markoff structure so you might have a partially observable scenario which
[01:10:52] a partially observable scenario which also comes up a lot in things like
[01:10:54] also comes up a lot in things like healthcare and then you can just roll
[01:10:55] healthcare and then you can just roll out your policy and just see how well it
[01:10:58] out your policy and just see how well it works okay well in healthcare you
[01:10:59] works okay well in healthcare you probably wouldn't just randomly roll out
[01:11:01] probably wouldn't just randomly roll out any policy but um You probably see my
[01:11:03] any policy but um You probably see my mean yeah remember your
[01:11:15] name that's right so here this is just
[01:11:17] name that's right so here this is just this is just the policy evaluation stage
[01:11:19] this is just the policy evaluation stage exactly and you could either do it to
[01:11:21] exactly and you could either do it to compute the value of a policy or as you
[01:11:22] compute the value of a policy or as you just just suggested do it for the Q
[01:11:25] just just suggested do it for the Q value so you like start off in a state
[01:11:27] value so you like start off in a state for each of the different actions then
[01:11:28] for each of the different actions then you roll out the policy till the end and
[01:11:31] you roll out the policy till the end and this is just a really popular other
[01:11:33] this is just a really popular other technique and it'll come up in other
[01:11:34] technique and it'll come up in other places so I wanted to start saying we
[01:11:36] places so I wanted to start saying we can do that here
[01:11:37] can do that here too okay so you can also think about
[01:11:41] too okay so you can also think about doing all of these in the case of the
[01:11:42] doing all of these in the case of the Mars Rober and and I I won't go through
[01:11:44] Mars Rober and and I I won't go through it now but I'll you can use these as
[01:11:46] it now but I'll you can use these as sort of examples to step through these
[01:11:48] sort of examples to step through these different algorithms okay and kind of
[01:11:50] different algorithms okay and kind of think about how you would compute these
[01:11:52] think about how you would compute these type of
[01:11:53] type of policies all right
[01:11:57] policies all right so I will maybe I'll I'll get to the end
[01:12:00] so I will maybe I'll I'll get to the end of this but I'll leave you with sort of
[01:12:01] of this but I'll leave you with sort of two things one is um sort of a thought
[01:12:04] two things one is um sort of a thought question which is is the optimal policy
[01:12:05] question which is is the optimal policy stationary what that means is
[01:12:07] stationary what that means is independent of time steps and the finite
[01:12:09] independent of time steps and the finite Horizon tasks and we'll we'll explore
[01:12:11] Horizon tasks and we'll we'll explore this issue to on the
[01:12:14] this issue to on the homework
[01:12:15] homework and I also just want to S of refresh
[01:12:18] and I also just want to S of refresh some terminology so in the context of
[01:12:20] some terminology so in the context of markof decision processes and
[01:12:21] markof decision processes and reinforcement learning when we say
[01:12:23] reinforcement learning when we say models what we normally mean is a
[01:12:24] models what we normally mean is a mathematical model of the Dynamics and
[01:12:26] mathematical model of the Dynamics and reward policy is a function mapping from
[01:12:29] reward policy is a function mapping from states to actions that can be
[01:12:30] states to actions that can be deterministic or stochastic and the
[01:12:32] deterministic or stochastic and the value function is this expected
[01:12:33] value function is this expected discounted sum of rewards from starting
[01:12:35] discounted sum of rewards from starting in a state and following a
[01:12:38] in a state and following a policy the things that should be clear
[01:12:40] policy the things that should be clear to you is you should be a to understand
[01:12:41] to you is you should be a to understand what a Markoff process is a Markoff
[01:12:43] what a Markoff process is a Markoff reward process is an mdp the Bellman
[01:12:46] reward process is an mdp the Bellman operator contraction model Q value and
[01:12:48] operator contraction model Q value and policy and you should be able to
[01:12:49] policy and you should be able to implement both value iteration and
[01:12:51] implement both value iteration and policy iteration and you'll have
[01:12:52] policy iteration and you'll have practice doing that on the homework you
[01:12:54] practice doing that on the homework you should also understand some of this
[01:12:55] should also understand some of this strengths and weaknesses in terms of
[01:12:56] strengths and weaknesses in terms of what we've discussed in terms of the
[01:12:57] what we've discussed in terms of the computational complexity of some of
[01:12:59] computational complexity of some of these different operations and be able
[01:13:01] these different operations and be able to prove contraction properties about
[01:13:03] to prove contraction properties about these as well as sort of understand like
[01:13:06] these as well as sort of understand like which of these are really leveraging the
[01:13:07] which of these are really leveraging the Markoff assumption versus which of them
[01:13:09] Markoff assumption versus which of them don't require that so next week we'll
[01:13:12] don't require that so next week we'll continue to talk about this and we'll
[01:13:14] continue to talk about this and we'll start to talk about uh function
[01:13:15] start to talk about uh function approximation and how we can also learn
[01:13:17] approximation and how we can also learn when we don't know what these models are
[01:13:19] when we don't know what these models are I'll see you then thanks

Stanford CS234 Reinforcement Learning I Policy Evaluation I 2024 I Lecture 3

Source: https://www.youtube.com/watch?v=jjq51TRNVvk

---

Transcript

[00:00:05] hey everybody welcome back we're going
[00:00:07] hey everybody welcome back we're going to get started with a refresh your
[00:00:08] to get started with a refresh your understanding poll you can go to Ed and
[00:00:11] understanding poll you can go to Ed and to see all the polls for today um just
[00:00:14] to see all the polls for today um just remember to log in first so that we can
[00:00:15] remember to log in first so that we can log it for participation points the two
[00:00:18] log it for participation points the two questions ask you to think about what we
[00:00:20] questions ask you to think about what we talked about last time in terms of
[00:00:21] talked about last time in terms of Markoff decision processes um and what
[00:00:24] Markoff decision processes um and what sort of guarantees or type of properties
[00:00:26] sort of guarantees or type of properties they have
[00:00:47] yeah you you what
[00:00:51] yeah you you what the again yeah great question tabular
[00:00:54] the again yeah great question tabular mdp is where we can write down what the
[00:00:56] mdp is where we can write down what the value is of a state as a table so like
[00:00:58] value is of a state as a table so like you can just have one end for what the
[00:01:00] you can just have one end for what the value is for each
[00:01:02] value is for each state this is in contract to like neural
[00:01:04] state this is in contract to like neural networks or things like that where you
[00:01:06] networks or things like that where you don't have one parameter per state
[00:01:43] okay we'll take like another one or two
[00:01:45] okay we'll take like another one or two minutes we have a good amount of
[00:01:46] minutes we have a good amount of controversy on these questions so I'll
[00:01:49] controversy on these questions so I'll see how this converges
[00:02:24] I remember seeing this a to the power s
[00:02:26] I remember seeing this a to the power s number in in a previous context was it
[00:02:28] number in in a previous context was it in the context of policy
[00:02:32] yeah does someone want to remember um
[00:02:34] yeah does someone want to remember um why is a to the S important I remember
[00:02:37] why is a to the S important I remember why it is yeah remember your
[00:02:39] why it is yeah remember your name is it like the number of total
[00:02:42] name is it like the number of total possible policies exactly right exactly
[00:02:45] possible policies exactly right exactly right so there's at most a to the S
[00:02:46] right so there's at most a to the S potential um uh deterministic policies
[00:03:00] all right so most people selected the
[00:03:02] all right so most people selected the correct answer for the first one um
[00:03:04] correct answer for the first one um which is this is true so ASM totically
[00:03:07] which is this is true so ASM totically this
[00:03:08] this will value iteration and policy
[00:03:10] will value iteration and policy iteration are correct in the tabular
[00:03:14] iteration are correct in the tabular discreet markup decision process case
[00:03:17] discreet markup decision process case and they will ASM totically both
[00:03:18] and they will ASM totically both converge and compute the right value
[00:03:20] converge and compute the right value function the second one looks like it's
[00:03:22] function the second one looks like it's pretty evenly split so I'd like you to
[00:03:25] pretty evenly split so I'd like you to turn to someone near you and um and
[00:03:27] turn to someone near you and um and argue for what you said for the second
[00:03:29] argue for what you said for the second one
[00:03:59] Li
[00:04:56] all right so maybe your has Chang
[00:05:11] [Music]
[00:05:16] all so the answer is true um and about
[00:05:19] all so the answer is true um and about half of you said that do someone get
[00:05:20] half of you said that do someone get want to tell me why this is true yeah is
[00:05:22] want to tell me why this is true yeah is it yeah um I'm not sure if this is
[00:05:25] it yeah um I'm not sure if this is correct but I just I think the value
[00:05:27] correct but I just I think the value iteration might not be um um guaranteed
[00:05:30] iteration might not be um um guaranteed to converge so that's if it's not
[00:05:32] to converge so that's if it's not guaranteed to converge it could just be
[00:05:33] guaranteed to converge it could just be unbounded so that certainly would be the
[00:05:35] unbounded so that certainly would be the case but fortunately it is guaranteed to
[00:05:37] case but fortunately it is guaranteed to converge if gamma is less than one um so
[00:05:40] converge if gamma is less than one um so but you're correct that it that it um
[00:05:42] but you're correct that it that it um Can require more iterations does anybody
[00:05:44] Can require more iterations does anybody have an and remind me your
[00:05:46] have an and remind me your name so my thinking was that uh in uh
[00:05:51] name so my thinking was that uh in uh like in policy itation we know that in
[00:05:52] like in policy itation we know that in each step we're going to improve to a
[00:05:54] each step we're going to improve to a new policy but um in each value step you
[00:05:57] new policy but um in each value step you might not reach a new policy you might
[00:05:59] might not reach a new policy you might take m multiple steps to reach a new
[00:06:02] take m multiple steps to reach a new policy that's right so um in policy
[00:06:05] policy that's right so um in policy iteration um and I talked to some people
[00:06:07] iteration um and I talked to some people about this too there can only be a to
[00:06:09] about this too there can only be a to the S um so in policy iteration there's
[00:06:11] the S um so in policy iteration there's at most a to the S because you only go
[00:06:13] at most a to the S because you only go through each policy once but for Value
[00:06:15] through each policy once but for Value iteration it can be more and I'll give
[00:06:17] iteration it can be more and I'll give you an example um so and one way I would
[00:06:21] you an example um so and one way I would think in general if you see this sort of
[00:06:22] think in general if you see this sort of question is to think about well can I
[00:06:23] question is to think about well can I can come up with a counter example to
[00:06:25] can come up with a counter example to say where this would be different so
[00:06:28] say where this would be different so consider a really silly mark off
[00:06:30] consider a really silly mark off decision process where there's just one
[00:06:31] decision process where there's just one state in one action so if there's one
[00:06:33] state in one action so if there's one state and one action there is literally
[00:06:35] state and one action there is literally one policy you can only do one thing and
[00:06:37] one policy you can only do one thing and there's only one state to do it in um so
[00:06:39] there's only one state to do it in um so that means that policy iteration is
[00:06:40] that means that policy iteration is going to take one round but for what
[00:06:43] going to take one round but for what value um iteration is going to do is
[00:06:45] value um iteration is going to do is it's going to keep going until the value
[00:06:46] it's going to keep going until the value function stops changing or stops
[00:06:48] function stops changing or stops changing within like a very small amount
[00:06:52] changing within like a very small amount and so what would happen in value
[00:06:53] and so what would happen in value iteration and feel free to go back to
[00:06:54] iteration and feel free to go back to your notes from last time is we would
[00:06:56] your notes from last time is we would start off and let's say the reward is 1
[00:07:00] start off and let's say the reward is 1 and Gamma is9 and we initialize the
[00:07:02] and Gamma is9 and we initialize the value function to zero so then if you
[00:07:05] value function to zero so then if you use a geometric series and if you
[00:07:07] use a geometric series and if you haven't seen that before just come chat
[00:07:08] haven't seen that before just come chat with me about it I'm happy to say um the
[00:07:11] with me about it I'm happy to say um the varar of this date is 1 over 1 minus
[00:07:13] varar of this date is 1 over 1 minus gamma because you get 1 plus gamma * 1
[00:07:17] gamma because you get 1 plus gamma 1 plus gamma s 1 dot dot dot so because
[00:07:20] plus gamma s * 1 dot dot dot so because it's like you're always staying in that
[00:07:22] it's like you're always staying in that state and you're always taking that
[00:07:23] state and you're always taking that action and you just day in and day out
[00:07:25] action and you just day in and day out that's what you get forever so so the
[00:07:27] that's what you get forever so so the actual value function it get the you get
[00:07:29] actual value function it get the you get eventually is 1 over 1 minus gamma um or
[00:07:32] eventually is 1 over 1 minus gamma um or you know about 10 um but after the first
[00:07:35] you know about 10 um but after the first iteration of value iteration V1 of s is
[00:07:38] iteration of value iteration V1 of s is just
[00:07:39] just one so that means you know one is not
[00:07:42] one so that means you know one is not close to 10 or not that close to 10 so
[00:07:44] close to 10 or not that close to 10 so we haven't converged yet so we'll have
[00:07:46] we haven't converged yet so we'll have to continue to do a bunch of iterations
[00:07:48] to continue to do a bunch of iterations of value iteration whereas at that point
[00:07:51] of value iteration whereas at that point um policy iteration would stop because
[00:07:53] um policy iteration would stop because it would just evaluate the value of the
[00:07:55] it would just evaluate the value of the one policy we have which is to take that
[00:07:56] one policy we have which is to take that one action and you'd be
[00:07:58] one action and you'd be done and I bring this up just to
[00:08:00] done and I bring this up just to illustrate that like both even though
[00:08:02] illustrate that like both even though both of those algorithms are guaranteed
[00:08:03] both of those algorithms are guaranteed to converge the right thing eventually
[00:08:05] to converge the right thing eventually they can have quite different behavior
[00:08:06] they can have quite different behavior in the short
[00:08:07] in the short term did you have a question I was just
[00:08:10] term did you have a question I was just going to ask is it only converging in
[00:08:12] going to ask is it only converging in the limit or all of them are only
[00:08:15] the limit or all of them are only converging limit yeah so all of them are
[00:08:17] converging limit yeah so all of them are converging um like ASM totically as you
[00:08:20] converging um like ASM totically as you do this over and over again good
[00:08:24] question great well welcome back um uh
[00:08:26] question great well welcome back um uh if you just came in feel free to go
[00:08:28] if you just came in feel free to go through the questions later what we're
[00:08:29] through the questions later what we're going to be doing today is to kind of
[00:08:31] going to be doing today is to kind of continue in this more simple setting
[00:08:33] continue in this more simple setting where we don't do any function
[00:08:34] where we don't do any function approximation um but we are now going to
[00:08:37] approximation um but we are now going to think about the fact where we don't have
[00:08:38] think about the fact where we don't have models of the world and what I mean by
[00:08:41] models of the world and what I mean by that is that we're not given a Dynamics
[00:08:42] that is that we're not given a Dynamics model and we're not given a reward model
[00:08:44] model and we're not given a reward model our agent just has to try things out in
[00:08:46] our agent just has to try things out in the world to learn how good they are and
[00:08:48] the world to learn how good they are and we're going to start with model free
[00:08:50] we're going to start with model free policy evaluation um in the case where
[00:08:53] policy evaluation um in the case where we still have a small enough number of
[00:08:55] we still have a small enough number of contexts or states that we could write
[00:08:57] contexts or states that we could write down a value for every single one of of
[00:08:59] down a value for every single one of of them separately so that's why we call it
[00:09:03] them separately so that's why we call it tabular um I'll just say one thing in
[00:09:05] tabular um I'll just say one thing in terms of logistics so office hours are
[00:09:07] terms of logistics so office hours are on the website we'll try to keep that
[00:09:09] on the website we'll try to keep that calendar updated it's just a Google
[00:09:10] calendar updated it's just a Google Calendar it'll include the location um
[00:09:13] Calendar it'll include the location um uh and if you go to Q status for the
[00:09:15] uh and if you go to Q status for the one1 office hours we'll make sure that
[00:09:17] one1 office hours we'll make sure that the zoom is either on that or on the web
[00:09:19] the zoom is either on that or on the web or on the website um I'll have office
[00:09:22] or on the website um I'll have office hours starting this week on Thursdays uh
[00:09:24] hours starting this week on Thursdays uh and mine are for project and conceptual
[00:09:26] and mine are for project and conceptual questions so feel free to come ask me
[00:09:28] questions so feel free to come ask me about anything in class um I I won't be
[00:09:30] about anything in class um I I won't be going through code but you can go talk
[00:09:32] going through code but you can go talk to the Tas about that um or you can come
[00:09:34] to the Tas about that um or you can come in and brainstorm about projects or or
[00:09:36] in and brainstorm about projects or or general questions about reinforcement
[00:09:38] general questions about reinforcement learning um I'm sure some of you are
[00:09:40] learning um I'm sure some of you are starting to think about projects uh just
[00:09:42] starting to think about projects uh just in general some people are asking so
[00:09:44] in general some people are asking so what's kind of in scope um I put
[00:09:46] what's kind of in scope um I put something on Ed about that but also in
[00:09:48] something on Ed about that but also in general it can be like a new idea for
[00:09:49] general it can be like a new idea for reinforcement learning a new application
[00:09:51] reinforcement learning a new application um it can be something you're doing in
[00:09:54] um it can be something you're doing in if you're already doing research and
[00:09:55] if you're already doing research and reinforcement learning it can also be
[00:09:57] reinforcement learning it can also be replicating part of an existing paper
[00:10:00] replicating part of an existing paper and this is really helpful actually it's
[00:10:02] and this is really helpful actually it's helpful for the whole Community because
[00:10:04] helpful for the whole Community because there's a lot of work going on um and
[00:10:06] there's a lot of work going on um and people are making different choices
[00:10:08] people are making different choices about hyperparameters and seeds and
[00:10:10] about hyperparameters and seeds and things like that so it really is very
[00:10:12] things like that so it really is very valuable too to see what things we can
[00:10:15] valuable too to see what things we can replicate does anybody have any
[00:10:16] replicate does anybody have any questions about that or other logistics
[00:10:18] questions about that or other logistics of the class before we get
[00:10:21] going all right so let's get into policy
[00:10:24] going all right so let's get into policy evaluation um so as I said what we're
[00:10:27] evaluation um so as I said what we're going to be doing today is to think
[00:10:29] going to be doing today is to think about how do we learn through direct
[00:10:31] about how do we learn through direct experience how good decisions are and
[00:10:33] experience how good decisions are and we're going to assume that we have a
[00:10:34] we're going to assume that we have a fixed policy so again like our boss says
[00:10:37] fixed policy so again like our boss says how good is this way of you know
[00:10:38] how good is this way of you know advertising to customers or U maybe
[00:10:41] advertising to customers or U maybe you're in a setting where you're trying
[00:10:42] you're in a setting where you're trying to see how good is the patient outcomes
[00:10:44] to see how good is the patient outcomes from the current
[00:10:45] from the current protocol and the idea is that we're only
[00:10:47] protocol and the idea is that we're only going to be using data from the
[00:10:49] going to be using data from the environment so um and today this
[00:10:52] environment so um and today this experience is going to come from
[00:10:52] experience is going to come from executing that policy let me just move
[00:10:54] executing that policy let me just move this up so you can see a bit
[00:10:57] this up so you can see a bit better okay today we're we going to
[00:10:59] better okay today we're we going to assume that when we get this data it's
[00:11:01] assume that when we get this data it's from directly executing a particular
[00:11:03] from directly executing a particular policy later we'll think we'll talk
[00:11:05] policy later we'll think we'll talk about other sort of relaxations to
[00:11:08] about other sort of relaxations to this and so I'm going to try to motivate
[00:11:10] this and so I'm going to try to motivate today why this is a useful thing to do
[00:11:13] today why this is a useful thing to do um and what sort of properties we would
[00:11:15] um and what sort of properties we would want to try to compare different
[00:11:17] want to try to compare different algorithms so it will turn out that this
[00:11:19] algorithms so it will turn out that this type of thing comes up all the time it
[00:11:21] type of thing comes up all the time it comes up when we actually want to make
[00:11:22] comes up when we actually want to make decisions and learn better policies um
[00:11:25] decisions and learn better policies um and it's going to be an important part
[00:11:27] and it's going to be an important part of much more complicated algorithm
[00:11:29] of much more complicated algorithm like deep Q learning and policy gradient
[00:11:32] like deep Q learning and policy gradient and other things where we want to sort
[00:11:33] and other things where we want to sort of be able to see how good are current
[00:11:35] of be able to see how good are current things so that then we can take gradient
[00:11:37] things so that then we can take gradient steps or improve our
[00:11:41] policy okay so this is what we're going
[00:11:43] policy okay so this is what we're going to try to cover today we're going to
[00:11:45] to try to cover today we're going to cover multicolor policy evaluation
[00:11:47] cover multicolor policy evaluation temporal difference learning certainty
[00:11:49] temporal difference learning certainty equivalence and batch policy evaluation
[00:11:52] equivalence and batch policy evaluation and maybe raise your hand if you've seen
[00:11:53] and maybe raise your hand if you've seen temporal difference learning
[00:11:56] temporal difference learning before okay raise your hand if you've
[00:11:58] before okay raise your hand if you've seen Q learning
[00:12:00] seen Q learning okay great Q learning is the control
[00:12:02] okay great Q learning is the control version basically of temporal difference
[00:12:03] version basically of temporal difference learning so you'll see a lot of
[00:12:05] learning so you'll see a lot of similarities there okay all right before
[00:12:08] similarities there okay all right before we dive into this I just want to recall
[00:12:10] we dive into this I just want to recall a couple definitions we'll have we're
[00:12:12] a couple definitions we'll have we're going to use G for the return which
[00:12:14] going to use G for the return which means that from this particular State
[00:12:16] means that from this particular State what is the discounted sum of rewards we
[00:12:18] what is the discounted sum of rewards we get for a particular
[00:12:20] get for a particular episode the state value function is
[00:12:22] episode the state value function is saying on average what is that reward we
[00:12:25] saying on average what is that reward we get and the state action value says if I
[00:12:28] get and the state action value says if I start in this state take this action and
[00:12:30] start in this state take this action and then follow this policy what is the
[00:12:32] then follow this policy what is the expected discounted sum of
[00:12:35] rewards so we saw last week that we
[00:12:39] rewards so we saw last week that we might want to do dynamic programming for
[00:12:41] might want to do dynamic programming for policy evaluation when we do have access
[00:12:43] policy evaluation when we do have access to the models so again what I mean by
[00:12:46] to the models so again what I mean by that is that where someone gives you a
[00:12:47] that is that where someone gives you a function for the reward and a function
[00:12:50] function for the reward and a function for the Dynamics model and so we saw we
[00:12:53] for the Dynamics model and so we saw we could do this sort of bellman like
[00:12:54] could do this sort of bellman like backup for a particular policy this was
[00:12:57] backup for a particular policy this was different than the Bellman equation
[00:12:58] different than the Bellman equation because there's no Max
[00:12:59] because there's no Max not trying to take a Max over different
[00:13:01] not trying to take a Max over different actions we're just taking whatever
[00:13:03] actions we're just taking whatever action is specified by the policy and in
[00:13:07] action is specified by the policy and in this equation here this is for like a
[00:13:09] this equation here this is for like a deterministic
[00:13:12] policy otherwise we need like some
[00:13:14] policy otherwise we need like some additional averaging over all the
[00:13:16] additional averaging over all the actions that could be taken by that
[00:13:18] actions that could be taken by that policy and just to remind ourselves here
[00:13:21] policy and just to remind ourselves here um before we converge this policy this V
[00:13:25] um before we converge this policy this V Pi K is like an estimate of the value of
[00:13:27] Pi K is like an estimate of the value of the policy it's not the actual value yet
[00:13:30] the policy it's not the actual value yet it's just like an estimate and it's
[00:13:31] it's just like an estimate and it's hopefully improving over as we do more
[00:13:36] iterations and what you can and another
[00:13:38] iterations and what you can and another just good thing to remind ourselves of
[00:13:40] just good thing to remind ourselves of is that this is here this sort of expect
[00:13:44] is that this is here this sort of expect expected discounted sum of rewards what
[00:13:46] expected discounted sum of rewards what we're doing in this equation is we are
[00:13:48] we're doing in this equation is we are plugging in this term as an estimate of
[00:13:53] plugging in this term as an estimate of the expected discounted rewards for the
[00:13:56] the expected discounted rewards for the future so we're saying like oh we've got
[00:13:58] future so we're saying like oh we've got this estimate of the value function
[00:14:00] this estimate of the value function we're going to plug in and say my reward
[00:14:02] we're going to plug in and say my reward is my immediate reward plus my
[00:14:03] is my immediate reward plus my discounted sum of future rewards and
[00:14:06] discounted sum of future rewards and this is what I'm using for my discounted
[00:14:07] this is what I'm using for my discounted sum of future
[00:14:08] sum of future rewards and so this is known as
[00:14:11] rewards and so this is known as bootstrapping because we're plugging in
[00:14:13] bootstrapping because we're plugging in one estimate in order to help us do
[00:14:15] one estimate in order to help us do another
[00:14:16] another estimate and we'll see a picture of this
[00:14:18] estimate and we'll see a picture of this graphically
[00:14:19] graphically later right Monte Carlo policy
[00:14:21] later right Monte Carlo policy evaluation is a really simple idea but
[00:14:24] evaluation is a really simple idea but it's very useful um and it is commonly
[00:14:27] it's very useful um and it is commonly done so as essentially the idea with POC
[00:14:30] done so as essentially the idea with POC multicol policy evaluation is we are
[00:14:32] multicol policy evaluation is we are just going to simulate or act in the
[00:14:35] just going to simulate or act in the real world so this is just saying you've
[00:14:37] real world so this is just saying you've got a policy which means you know what
[00:14:39] got a policy which means you know what action to take in every state today
[00:14:41] action to take in every state today we'll mostly focus on deterministic
[00:14:42] we'll mostly focus on deterministic policies just to make it easier um so
[00:14:46] policies just to make it easier um so I'll just say that
[00:14:47] I'll just say that for for
[00:14:50] for for most of
[00:14:52] most of today
[00:14:54] today assume Pi is deterministic
[00:15:01] but all these ideas can easily be
[00:15:02] but all these ideas can easily be extended just easy to write that down
[00:15:04] extended just easy to write that down without having to do an expectation over
[00:15:05] without having to do an expectation over actions everywhere okay so what's the
[00:15:08] actions everywhere okay so what's the idea in this case well the value
[00:15:11] idea in this case well the value function is just an average over the
[00:15:14] function is just an average over the returns it's like an expectation over
[00:15:16] returns it's like an expectation over the trajectories or the returns you
[00:15:18] the trajectories or the returns you could get by following the
[00:15:20] could get by following the policy and therefore the value is just
[00:15:23] policy and therefore the value is just the mean of returns and we know how to
[00:15:26] the mean of returns and we know how to approximate means we just do things a
[00:15:28] approximate means we just do things a bunch of times than we
[00:15:31] bunch of times than we average and so as an example of this it
[00:15:33] average and so as an example of this it might be something there someone says
[00:15:35] might be something there someone says okay when I want to know if we um say
[00:15:38] okay when I want to know if we um say give a particular set of patient
[00:15:39] give a particular set of patient treatments um and maybe those treatments
[00:15:42] treatments um and maybe those treatments take a year for example and someone
[00:15:44] take a year for example and someone wants to know on average how good is
[00:15:46] wants to know on average how good is that well what you could do is you could
[00:15:47] that well what you could do is you could have a 100 patients I'll go through that
[00:15:50] have a 100 patients I'll go through that particular protocol for a year and then
[00:15:52] particular protocol for a year and then average their outcomes and that would be
[00:15:55] average their outcomes and that would be an example of Monte Carlo policy
[00:15:57] an example of Monte Carlo policy evaluation is you just execute the
[00:15:59] evaluation is you just execute the policy for many different episodes and
[00:16:02] policy for many different episodes and then you average okay and one thing just
[00:16:05] then you average okay and one thing just to note here is that you can have cases
[00:16:07] to note here is that you can have cases here where not all the trajectories of
[00:16:09] here where not all the trajectories of the same
[00:16:10] the same length so imagine in the patient case I
[00:16:13] length so imagine in the patient case I just gave you might have that some
[00:16:14] just gave you might have that some people drop out of a trial during the
[00:16:16] people drop out of a trial during the year or maybe they finish their
[00:16:18] year or maybe they finish their treatment successfully and and so then
[00:16:20] treatment successfully and and so then they're also done so all the
[00:16:22] they're also done so all the trajectories may not be the same length
[00:16:24] trajectories may not be the same length but essentially you can just think of it
[00:16:25] but essentially you can just think of it as just have many many trajectories you
[00:16:28] as just have many many trajectories you know maybe this one has a gal 10 this
[00:16:31] know maybe this one has a gal 10 this had a gal 5 this has a gal 10 you just
[00:16:35] had a gal 5 this has a gal 10 you just average over all of them and that is
[00:16:38] average over all of them and that is your value
[00:16:40] your value function
[00:16:42] function okay now one of the benefits of this is
[00:16:44] okay now one of the benefits of this is that when we do this we're not actually
[00:16:47] that when we do this we're not actually benefits or drawbacks and we'll talk
[00:16:48] benefits or drawbacks and we'll talk about this this is making no assumption
[00:16:51] about this this is making no assumption that the system is a Markoff decision
[00:16:53] that the system is a Markoff decision process it's just averaging so your
[00:16:57] process it's just averaging so your system might not be Markoff
[00:17:00] system might not be Markoff and what I mean by that is that you in
[00:17:02] and what I mean by that is that you in general will have a finite set of
[00:17:04] general will have a finite set of features to describe the state so if we
[00:17:06] features to describe the state so if we think about that patient example I just
[00:17:08] think about that patient example I just had maybe you have different vitals of
[00:17:10] had maybe you have different vitals of the patient maybe you have static
[00:17:12] the patient maybe you have static demographic variables but we do not have
[00:17:15] demographic variables but we do not have all the features that are probably going
[00:17:16] all the features that are probably going to describe how someone's going to react
[00:17:18] to describe how someone's going to react to a set of
[00:17:19] to a set of treatments and so because of that you
[00:17:22] treatments and so because of that you may or may not think that in the
[00:17:23] may or may not think that in the features you have access to the system
[00:17:25] features you have access to the system is
[00:17:26] is Markoff but this doesn't require this
[00:17:28] Markoff but this doesn't require this state to be mark off it's just averaging
[00:17:30] state to be mark off it's just averaging you just roll out your policy many times
[00:17:32] you just roll out your policy many times and you
[00:17:33] and you average now a really important thing
[00:17:35] average now a really important thing here is that it can only be applied to
[00:17:37] here is that it can only be applied to episodic
[00:17:39] episodic mdps so what do I mean by that I mean
[00:17:41] mdps so what do I mean by that I mean your episode has to end in order for you
[00:17:44] your episode has to end in order for you to see what the total return
[00:17:47] to see what the total return was so if you have sort of horizon
[00:17:50] was so if you have sort of horizon lengths or episodes that last for a year
[00:17:52] lengths or episodes that last for a year that's okay it's a little bit slow but
[00:17:54] that's okay it's a little bit slow but you could do that but if you want to
[00:17:56] you could do that but if you want to just think of how good a policy is if
[00:17:58] just think of how good a policy is if you just are going to act forever and
[00:17:59] you just are going to act forever and never stop this wouldn't work we're not
[00:18:03] never stop this wouldn't work we're not without some additional
[00:18:06] approximations somebody have any
[00:18:07] approximations somebody have any questions about either of these two
[00:18:09] questions about either of these two things about it not assuming the state
[00:18:10] things about it not assuming the state is marov yeah um you're talking about
[00:18:13] is marov yeah um you're talking about like Medical Treatments so um I mean
[00:18:17] like Medical Treatments so um I mean does this only work if like the
[00:18:18] does this only work if like the treatment only lasts like I know a same
[00:18:21] treatment only lasts like I know a same amount of time for every patient like
[00:18:22] amount of time for every patient like six months um because if they have
[00:18:25] six months um because if they have different Lanes right like how can that
[00:18:26] different Lanes right like how can that be episodic great question yeah so what
[00:18:30] be episodic great question yeah so what is like well if it can it be episodic if
[00:18:32] is like well if it can it be episodic if the episodes are different length it
[00:18:33] the episodes are different length it could be so it could be that you have
[00:18:35] could be so it could be that you have like a fixed policy and maybe that
[00:18:36] like a fixed policy and maybe that policy says if someone doesn't respond
[00:18:38] policy says if someone doesn't respond to this type of treatment we do this
[00:18:40] to this type of treatment we do this additional type of treatment and in fact
[00:18:42] additional type of treatment and in fact that's very common um as long as the
[00:18:45] that's very common um as long as the episode is guaranteed to end like you
[00:18:47] episode is guaranteed to end like you know that treatment could only last day
[00:18:48] know that treatment could only last day for a year total you can still average
[00:18:50] for a year total you can still average overall those outcomes you just sum over
[00:18:53] overall those outcomes you just sum over the the return for those different
[00:18:55] the the return for those different ones yeah for each of these um
[00:18:59] ones yeah for each of these um trajectories are we supposed to begin
[00:19:01] trajectories are we supposed to begin with this like different state or we can
[00:19:04] with this like different state or we can actually start with the same state great
[00:19:06] actually start with the same state great question so um if we want to get a value
[00:19:08] question so um if we want to get a value function for all states we need to see
[00:19:10] function for all states we need to see all the states inside of these
[00:19:11] all the states inside of these trajectories and we'll talk about how we
[00:19:12] trajectories and we'll talk about how we estimate these in a second so I think
[00:19:14] estimate these in a second so I think this will be
[00:19:16] this will be answered okay so for example how might
[00:19:18] answered okay so for example how might we compute this so we would like to get
[00:19:21] we compute this so we would like to get the value for all the states that are
[00:19:23] the value for all the states that are reachable inside of your policy so what
[00:19:26] reachable inside of your policy so what you could do is we can initialize two
[00:19:28] you could do is we can initialize two different variables One n of s is just
[00:19:30] different variables One n of s is just going to be the counts the number of
[00:19:32] going to be the counts the number of times that we've updated our estimate
[00:19:34] times that we've updated our estimate for State s g ofs here is going to start
[00:19:38] for State s g ofs here is going to start with zero which is we've never seen any
[00:19:40] with zero which is we've never seen any returns from this
[00:19:41] returns from this state so what every visit Monte Carlo
[00:19:44] state so what every visit Monte Carlo does is it samples an episode so this
[00:19:47] does is it samples an episode so this goes up to sometime step T TI this has
[00:19:51] goes up to sometime step T TI this has to be finite but it could you know be
[00:19:53] to be finite but it could you know be different on different episodes and then
[00:19:56] different on different episodes and then we compute the discounted sum of Rewards
[00:19:58] we compute the discounted sum of Rewards for that episode okay so starting at
[00:20:02] for that episode okay so starting at time step T excuse me how much reward do
[00:20:05] time step T excuse me how much reward do we get till the end of the
[00:20:07] we get till the end of the episode and then for every time step
[00:20:09] episode and then for every time step until the end we see if this is the
[00:20:12] until the end we see if this is the first time that states been visited then
[00:20:14] first time that states been visited then we update the total number of times we
[00:20:16] we update the total number of times we visited this state for the first time
[00:20:18] visited this state for the first time per EP you know in each episode we
[00:20:20] per EP you know in each episode we increment the total return and we
[00:20:24] average so that just like Steps along
[00:20:27] average so that just like Steps along and it just says okay you know maybe the
[00:20:29] and it just says okay you know maybe the first time I reached if you think of the
[00:20:30] first time I reached if you think of the Mars rover example see I have it in the
[00:20:32] Mars rover example see I have it in the next one no I think I put it so for a
[00:20:34] next one no I think I put it so for a lot of today I've moved a lot of the
[00:20:36] lot of today I've moved a lot of the worked examples till the end of the
[00:20:38] worked examples till the end of the slides but if you want to go through
[00:20:39] slides but if you want to go through them later I encourage you to um so for
[00:20:42] them later I encourage you to um so for example in the case of the Mars rover
[00:20:44] example in the case of the Mars rover you might imagine you start in state
[00:20:45] you might imagine you start in state like S3 and then on that particular one
[00:20:48] like S3 and then on that particular one you get a reward of one for that episode
[00:20:51] you get a reward of one for that episode and so then you would average in one
[00:20:54] and so then you would average in one starting in that state as three till the
[00:20:56] starting in that state as three till the end
[00:21:00] so this is first visit Monte Carlo
[00:21:02] so this is first visit Monte Carlo evaluation which means you only update a
[00:21:04] evaluation which means you only update a state at most once in each episode so if
[00:21:07] state at most once in each episode so if you had something like this S1 went to
[00:21:11] you had something like this S1 went to S2 went to S3 went to S2 went to S3 so
[00:21:16] S2 went to S3 went to S2 went to S3 so in this case dot dot
[00:21:18] in this case dot dot dot you would update S1 once in this
[00:21:22] dot you would update S1 once in this trajectory and you would update S2 once
[00:21:24] trajectory and you would update S2 once and S3 once even though you in fact
[00:21:27] and S3 once even though you in fact visit those States multi times you only
[00:21:29] visit those States multi times you only update them for the first time you visit
[00:21:32] update them for the first time you visit yeah so could this have the problem
[00:21:35] yeah so could this have the problem where if the state is really rare
[00:21:38] where if the state is really rare uncommon then we get a really bad ass or
[00:21:42] uncommon then we get a really bad ass or we just never visit it at all so if you
[00:21:44] we just never visit it at all so if you don't ever visit a state under the
[00:21:46] don't ever visit a state under the policy that's okay because then you
[00:21:48] policy that's okay because then you don't have a value for it but then it's
[00:21:49] don't have a value for it but then it's kind of undefined because you never
[00:21:51] kind of undefined because you never would reach it um uh in this case as
[00:21:55] would reach it um uh in this case as says if there's a state that's really
[00:21:57] says if there's a state that's really rare for you to reach inside of your
[00:21:59] rare for you to reach inside of your policy it might take a lot of
[00:22:01] policy it might take a lot of trajectories in order to get a good
[00:22:02] trajectories in order to get a good estimate of it so maybe there's some
[00:22:04] estimate of it so maybe there's some rare side effect um of a treatment plan
[00:22:07] rare side effect um of a treatment plan and it's going to take a lot of
[00:22:09] and it's going to take a lot of trajectories um to to observe that that
[00:22:12] trajectories um to to observe that that was one of the challenges with the covid
[00:22:13] was one of the challenges with the covid vaccine is that you know of course it
[00:22:14] vaccine is that you know of course it was a finite number of people is pretty
[00:22:16] was a finite number of people is pretty large number but pretty but finite and
[00:22:18] large number but pretty but finite and some side effects don't show up until
[00:22:20] some side effects don't show up until you get much many many more it's true
[00:22:23] you get much many many more it's true generally for treatments even if on
[00:22:24] generally for treatments even if on average they're totally fine that you
[00:22:25] average they're totally fine that you won't see some of those rare side
[00:22:26] won't see some of those rare side effects until you get an enormous number
[00:22:28] effects until you get an enormous number of
[00:22:29] of trajectories now covid vaccine certainly
[00:22:31] trajectories now covid vaccine certainly had you know the benefits there way out
[00:22:33] had you know the benefits there way out let way um side effects but my point is
[00:22:36] let way um side effects but my point is just to highlight that you know
[00:22:37] just to highlight that you know depending on how frequently you see the
[00:22:39] depending on how frequently you see the states it will take you more or less um
[00:22:41] states it will take you more or less um number of total episodes in order to
[00:22:43] number of total episodes in order to observe yeah so kind of it doesn't
[00:22:46] observe yeah so kind of it doesn't matter that I saw like S2 again or S3
[00:22:48] matter that I saw like S2 again or S3 again in this trajectory for this
[00:22:50] again in this trajectory for this algorithm no um I mean it probably
[00:22:52] algorithm no um I mean it probably affects the reward still it's just that
[00:22:54] affects the reward still it's just that you don't use that data so an
[00:22:55] you don't use that data so an alternative is called every visit where
[00:22:58] alternative is called every visit where every time you see this dat in that
[00:22:59] every time you see this dat in that trajectory you update
[00:23:03] it and as you might imagine there let's
[00:23:05] it and as you might imagine there let's say you see a it's a really long
[00:23:07] say you see a it's a really long trajectory and you see s to many times
[00:23:10] trajectory and you see s to many times then you would update for all of
[00:23:13] those so I'm just going to show you of
[00:23:15] those so I'm just going to show you of three common different ones so this is
[00:23:17] three common different ones so this is sort of the this is a work example you
[00:23:20] sort of the this is a work example you can go through later if you want which
[00:23:21] can go through later if you want which is if you imagine this is the Mars rover
[00:23:24] is if you imagine this is the Mars rover the rewards are on either side this is
[00:23:26] the rewards are on either side this is the particular trajectory you can
[00:23:27] the particular trajectory you can compute the first visit and the every
[00:23:29] compute the first visit and the every visit Monte Carlo
[00:23:31] visit Monte Carlo estimates so both of those are totally
[00:23:34] estimates so both of those are totally reasonable things to do perhaps more
[00:23:35] reasonable things to do perhaps more common is what's known as incremental
[00:23:37] common is what's known as incremental Monte Carlo okay and this does kind of
[00:23:40] Monte Carlo okay and this does kind of what you would expect it to do which is
[00:23:41] what you would expect it to do which is you maintain a running estimate for what
[00:23:44] you maintain a running estimate for what is the value under a policy for a
[00:23:46] is the value under a policy for a particular State and you smoothly update
[00:23:49] particular State and you smoothly update that as you get more
[00:23:50] that as you get more data so what we would do in this case is
[00:23:53] data so what we would do in this case is you keep track of the number of times
[00:23:55] you keep track of the number of times you visited that
[00:23:56] you visited that state and then you weigh your old
[00:23:59] state and then you weigh your old estimate by the number of times you
[00:24:02] estimate by the number of times you visitate minus 1 NS plus your new return
[00:24:05] visitate minus 1 NS plus your new return you just observed div by NS so that's
[00:24:08] you just observed div by NS so that's just sort of your way you know you're
[00:24:09] just sort of your way you know you're kind of constantly updating your value
[00:24:12] kind of constantly updating your value function for this state as you get more
[00:24:15] function for this state as you get more data and for those of you who have done
[00:24:18] data and for those of you who have done machine learning which is probably most
[00:24:19] machine learning which is probably most of you this should look pretty familiar
[00:24:21] of you this should look pretty familiar this is kind of like a learning
[00:24:24] this is kind of like a learning rate this is your updated value and this
[00:24:26] rate this is your updated value and this is your old value
[00:24:30] is your old value and in fact that's what we're going to
[00:24:31] and in fact that's what we're going to see here okay so you can think of this
[00:24:34] see here okay so you can think of this in general it doesn't have to be 1/ n it
[00:24:37] in general it doesn't have to be 1/ n it can just be any Alpha here so any sort
[00:24:40] can just be any Alpha here so any sort of like Alpha here is um just a learning
[00:24:44] of like Alpha here is um just a learning rate and we're just smoothly updating
[00:24:47] rate and we're just smoothly updating our estimate of what is the value
[00:24:49] our estimate of what is the value function for a particular
[00:24:53] State and we'll see lots lots and lots
[00:24:56] State and we'll see lots lots and lots of algorithms like that similar to part
[00:24:57] of algorithms like that similar to part what you saw in machine
[00:25:00] what you saw in machine learning the key thing here is the the
[00:25:02] learning the key thing here is the the estimates that we're using which we
[00:25:04] estimates that we're using which we often call sort of you know we also
[00:25:06] often call sort of you know we also might use the word targets often is we
[00:25:08] might use the word targets often is we have this estimate
[00:25:11] have this estimate here and then we have our old estimate
[00:25:13] here and then we have our old estimate so this is you know one sample of what
[00:25:15] so this is you know one sample of what is the return starting in this state
[00:25:17] is the return starting in this state till the end of the
[00:25:21] episode so I think it's helpful to think
[00:25:23] episode so I think it's helpful to think a little bit about what this looks like
[00:25:25] a little bit about what this looks like pictorially and this also relates a lot
[00:25:27] pictorially and this also relates a lot to we'll see this we talk about things
[00:25:29] to we'll see this we talk about things like
[00:25:31] alphago so I think it's helpful today
[00:25:34] alphago so I think it's helpful today and it'll look somewhat familiar to you
[00:25:35] and it'll look somewhat familiar to you if you've seen things like Mini Max
[00:25:37] if you've seen things like Mini Max trees or expecting Max trees who here
[00:25:39] trees or expecting Max trees who here has seen expecting Max trees
[00:25:42] has seen expecting Max trees before okay so maybe one so most people
[00:25:45] before okay so maybe one so most people not so this might be a useful sort of um
[00:25:47] not so this might be a useful sort of um uh representation so I think one way to
[00:25:49] uh representation so I think one way to think about this is what we're trying to
[00:25:51] think about this is what we're trying to do is we're trying to um think of what
[00:25:53] do is we're trying to um think of what the value is starting in a certain State
[00:25:56] the value is starting in a certain State and we want to take um we know what the
[00:25:58] and we want to take um we know what the action is we're going to take cuz we've
[00:25:59] action is we're going to take cuz we've got a fixed policy so that says we start
[00:26:01] got a fixed policy so that says we start in this date s we take this action a
[00:26:04] in this date s we take this action a this is prescribed by our policy so this
[00:26:06] this is prescribed by our policy so this is pi of Pi of s is going to equal
[00:26:11] is pi of Pi of s is going to equal a and then after we do that because the
[00:26:13] a and then after we do that because the world might be stochastic we're going to
[00:26:15] world might be stochastic we're going to have a bunch of next States we could
[00:26:17] have a bunch of next States we could reach so we have probability of S Prime
[00:26:20] reach so we have probability of S Prime given s and
[00:26:23] given s and a and what we're trying to do when we do
[00:26:26] a and what we're trying to do when we do um policy evaluation is we're trying to
[00:26:28] um policy evaluation is we're trying to get expectation over all the potential
[00:26:30] get expectation over all the potential Futures we might end up in by following
[00:26:32] Futures we might end up in by following this policy you know and maybe in some
[00:26:34] this policy you know and maybe in some cases there's really good patient
[00:26:35] cases there's really good patient outcomes hopefully most of the time and
[00:26:37] outcomes hopefully most of the time and maybe sometimes there's less good
[00:26:38] maybe sometimes there's less good patient outcomes and we want to do an
[00:26:40] patient outcomes and we want to do an expectation over all of this so we can
[00:26:42] expectation over all of this so we can think of that as a tree as just we start
[00:26:44] think of that as a tree as just we start in a state we take an action we look at
[00:26:46] in a state we take an action we look at the branching factor of all possible
[00:26:48] the branching factor of all possible next States and then we
[00:26:50] next States and then we repeat excuse
[00:26:52] repeat excuse me so if we think of what the policy
[00:26:54] me so if we think of what the policy evaluation diagram is doing for each
[00:26:56] evaluation diagram is doing for each state we know what the next action is we
[00:26:57] state we know what the next action is we take
[00:26:59] take and then we Branch again in terms of
[00:27:02] and then we Branch again in terms of States so this is like S Prime and this
[00:27:05] States so this is like S Prime and this is S
[00:27:12] Prime and so we can just think of like
[00:27:14] Prime and so we can just think of like kind of this tree of possibilities going
[00:27:16] kind of this tree of possibilities going out but we don't have any there's no
[00:27:18] out but we don't have any there's no branching on the actions because the
[00:27:19] branching on the actions because the actions are fixed by our
[00:27:22] actions are fixed by our policy so then if we go all the way out
[00:27:24] policy so then if we go all the way out and then what we want to do is we want
[00:27:25] and then what we want to do is we want to figure out what the value function is
[00:27:27] to figure out what the value function is can think of this is you know an
[00:27:29] can think of this is you know an expensive way to do dynamic programming
[00:27:31] expensive way to do dynamic programming um what you would do is you would take
[00:27:33] um what you would do is you would take an expectation over the states and then
[00:27:35] an expectation over the states and then you would propagate those values back up
[00:27:37] you would propagate those values back up to the
[00:27:38] to the root and if you don't find this a useful
[00:27:40] root and if you don't find this a useful conceptual way to think about it it's
[00:27:42] conceptual way to think about it it's fine it just I think it can be helpful
[00:27:44] fine it just I think it can be helpful to then think about what these different
[00:27:45] to then think about what these different algorithms are doing in terms of
[00:27:47] algorithms are doing in terms of approximations so this is sort of what
[00:27:49] approximations so this is sort of what we would like to do in order to get V Pi
[00:27:52] we would like to do in order to get V Pi of s but we want to do this a much more
[00:27:55] of s but we want to do this a much more computational umly efficient way and
[00:27:58] computational umly efficient way and also sample efficient way so what Monte
[00:28:01] also sample efficient way so what Monte Carlo policy evaluation is going to do
[00:28:04] Carlo policy evaluation is going to do is it's going to look like that
[00:28:05] is it's going to look like that particular equation here and what it is
[00:28:08] particular equation here and what it is doing here is it is going to
[00:28:11] doing here is it is going to approximate of these full Expectations
[00:28:14] approximate of these full Expectations by a
[00:28:17] by a sample so in particular what it's doing
[00:28:19] sample so in particular what it's doing here is it's updating the Value Estimate
[00:28:21] here is it's updating the Value Estimate by using a sample of the return to
[00:28:24] by using a sample of the return to approximate an expectation
[00:28:29] and we do this many times you know we
[00:28:30] and we do this many times you know we average over many such returns so it's
[00:28:32] average over many such returns so it's kind of like saying you have this
[00:28:33] kind of like saying you have this enormous branching tree you could do an
[00:28:35] enormous branching tree you could do an expectation over all of that sort of
[00:28:37] expectation over all of that sort of explicitly up from the roots or you
[00:28:39] explicitly up from the roots or you could just sample many times and that's
[00:28:41] could just sample many times and that's also going to approximate the
[00:28:42] also going to approximate the tree and the more samples you get the
[00:28:45] tree and the more samples you get the better that's going to be as an
[00:28:46] better that's going to be as an approximation of the
[00:28:48] approximation of the tree and this type of ideas has been um
[00:28:51] tree and this type of ideas has been um used in many different types of
[00:28:52] used in many different types of algorithms there's um some really nice
[00:28:54] algorithms there's um some really nice work in the mid 2000s like by Michael
[00:28:56] work in the mid 2000s like by Michael Kern and others and then similar ideas
[00:28:59] Kern and others and then similar ideas were really the foundation that then led
[00:29:00] were really the foundation that then led to some of the advances of like Monte
[00:29:02] to some of the advances of like Monte Carlo tree search and then that went
[00:29:04] Carlo tree search and then that went into
[00:29:05] into alphago so this what this is what Monte
[00:29:09] alphago so this what this is what Monte Carlo tree search is doing so notice
[00:29:11] Carlo tree search is doing so notice it's not doing any form of
[00:29:12] it's not doing any form of bootstrapping there's no dynamic
[00:29:14] bootstrapping there's no dynamic programming that's going on here it's
[00:29:16] programming that's going on here it's just rolling out and then what we have
[00:29:18] just rolling out and then what we have here is it is
[00:29:22] using this here as sort of U this sample
[00:29:26] using this here as sort of U this sample as an exp as an AO
[00:29:30] okay all right so that's how Monti Carlo
[00:29:32] okay all right so that's how Monti Carlo policy evaluation works one natural
[00:29:35] policy evaluation works one natural question in this case is how good is
[00:29:36] question in this case is how good is that estimate so we're going to see lots
[00:29:38] that estimate so we're going to see lots of different ways and lots of algorithms
[00:29:40] of different ways and lots of algorithms for trying to do policy evaluation and
[00:29:42] for trying to do policy evaluation and so you might know ask like well how do I
[00:29:44] so you might know ask like well how do I pick them on them like what are the
[00:29:45] pick them on them like what are the properties I should think about so one
[00:29:48] properties I should think about so one pretty basic property that you might
[00:29:50] pretty basic property that you might want is consistency which means that as
[00:29:53] want is consistency which means that as you get more and more data does your
[00:29:55] you get more and more data does your estimate actually converge to the true
[00:29:59] estimate actually converge to the true value of the policy for all the
[00:30:02] value of the policy for all the states and this is something you
[00:30:04] states and this is something you probably want in many cases at least
[00:30:06] probably want in many cases at least because otherwise it means that even if
[00:30:08] because otherwise it means that even if you had infinite data your estimate
[00:30:10] you had infinite data your estimate still going to be wrong now as we start
[00:30:12] still going to be wrong now as we start to think about more complicated settings
[00:30:13] to think about more complicated settings we might have to St you know um sort of
[00:30:16] we might have to St you know um sort of be satisfied with this less good
[00:30:19] be satisfied with this less good objective but here for right now we're
[00:30:20] objective but here for right now we're going to we're hoping we can just write
[00:30:23] going to we're hoping we can just write down the value of every state as an
[00:30:24] down the value of every state as an entry and a table that we should be able
[00:30:26] entry and a table that we should be able to get consistency
[00:30:28] to get consistency a second thing we might want is
[00:30:29] a second thing we might want is computational efficiency we'd like this
[00:30:32] computational efficiency we'd like this not to be too expensive for us to
[00:30:34] not to be too expensive for us to compute we'd like us not to require too
[00:30:36] compute we'd like us not to require too much
[00:30:37] much memory and we'd like it to have
[00:30:40] memory and we'd like it to have statistical efficiency which is sort of
[00:30:43] statistical efficiency which is sort of essentially how does the accuracy of the
[00:30:45] essentially how does the accuracy of the estimate change with the amount of
[00:30:47] estimate change with the amount of data and what that means here is like
[00:30:49] data and what that means here is like sort of more formally we'd like to know
[00:30:51] sort of more formally we'd like to know how quickly do these things converge um
[00:30:53] how quickly do these things converge um as you get more and more
[00:30:55] as you get more and more data and then in reality we often care
[00:30:57] data and then in reality we often care about empirical accuracy just how what
[00:31:00] about empirical accuracy just how what is our mean squared error for our types
[00:31:01] is our mean squared error for our types of our
[00:31:02] of our estimators so how good is Monte Carlo
[00:31:05] estimators so how good is Monte Carlo well let's just first quickly remind
[00:31:08] well let's just first quickly remind ourselves that the bias of an estimator
[00:31:10] ourselves that the bias of an estimator is the so if we have an estimat Theta
[00:31:12] is the so if we have an estimat Theta which we're going to be thinking of is
[00:31:14] which we're going to be thinking of is like our value function approximate
[00:31:16] like our value function approximate approximation it's going to be the
[00:31:18] approximation it's going to be the difference between on average what our
[00:31:21] difference between on average what our estimator is versus the True Value
[00:31:23] estimator is versus the True Value that's our
[00:31:25] that's our bias and the variance of an estimator is
[00:31:27] bias and the variance of an estimator is the difference between this and its
[00:31:29] the difference between this and its expectation squared and the expectation
[00:31:31] expectation squared and the expectation of that and the mean squared error is
[00:31:33] of that and the mean squared error is going to be variance plus bias
[00:31:36] going to be variance plus bias squ
[00:31:37] squ okay all right so generally we would
[00:31:39] okay all right so generally we would like an estimator that has low mean squ
[00:31:41] like an estimator that has low mean squ error which means we want it to have
[00:31:42] error which means we want it to have like lower zero bias and low
[00:31:47] variance something to think about if
[00:31:48] variance something to think about if you're less familiar with these is um
[00:31:51] you're less familiar with these is um whether or not if an estimator is
[00:31:52] whether or not if an estimator is unbiased is it consistent it is not
[00:31:55] unbiased is it consistent it is not necessarily consistent just so you know
[00:31:58] necessarily consistent just so you know right so what we would like here is that
[00:32:02] right so what we would like here is that ASM totically the probability that our
[00:32:04] ASM totically the probability that our estimator so n here is the amount of
[00:32:06] estimator so n here is the amount of data we're using to construct that
[00:32:07] data we're using to construct that estimator the probability that as we get
[00:32:10] estimator the probability that as we get an infinite amount of data that our
[00:32:12] an infinite amount of data that our estimate is different than the true
[00:32:14] estimate is different than the true value by more than Epsilon it has to go
[00:32:16] value by more than Epsilon it has to go to
[00:32:19] to zero okay so we would like it to be
[00:32:21] zero okay so we would like it to be consistent so how does Monte Carlo fa on
[00:32:24] consistent so how does Monte Carlo fa on these sort of um
[00:32:26] these sort of um properties well um first visit is
[00:32:30] properties well um first visit is unbiased so it's an unbiased estimator
[00:32:33] unbiased so it's an unbiased estimator of the DU policy and by the law of large
[00:32:35] of the DU policy and by the law of large Lumar as the amount of data you have
[00:32:37] Lumar as the amount of data you have goes to Infinity per state so you know
[00:32:40] goes to Infinity per state so you know if you have really rare States you're
[00:32:41] if you have really rare States you're still going to need a number of samples
[00:32:43] still going to need a number of samples to you know to estimate them but as the
[00:32:45] to you know to estimate them but as the amount of data you have goes to Infinity
[00:32:47] amount of data you have goes to Infinity you'll converge so it's consistent and
[00:32:48] you'll converge so it's consistent and it's
[00:32:50] it's unbiased okay every visit Monte Carlo is
[00:32:54] unbiased okay every visit Monte Carlo is biased one way to think about that is um
[00:32:57] biased one way to think about that is um in the first case all your data is IID
[00:33:00] in the first case all your data is IID independent and identically distributed
[00:33:02] independent and identically distributed in the every visit case imagine that you
[00:33:05] in the every visit case imagine that you visit State S2 and then like you know
[00:33:07] visit State S2 and then like you know four steps later you visit
[00:33:09] four steps later you visit S2 well their returns are going to be
[00:33:12] S2 well their returns are going to be correlated because they're both in the
[00:33:13] correlated because they're both in the same trajectory so they're not IID
[00:33:16] same trajectory so they're not IID anymore that's just some intuition for
[00:33:18] anymore that's just some intuition for why it might be biased but it's also
[00:33:20] why it might be biased but it's also consistent and it often has better means
[00:33:21] consistent and it often has better means squared error because you get to use
[00:33:23] squared error because you get to use more of your data inside of a single
[00:33:26] more of your data inside of a single trajectory to do more updates
[00:33:28] trajectory to do more updates and then incremental Monte Carlo methods
[00:33:30] and then incremental Monte Carlo methods depend on the learning baate as you
[00:33:33] depend on the learning baate as you might
[00:33:34] might expect
[00:33:36] expect so see that here so um let's imagine
[00:33:40] so see that here so um let's imagine that we are going to have our Alpha
[00:33:42] that we are going to have our Alpha parameter which is our learning rate
[00:33:43] parameter which is our learning rate which is trading off between our new
[00:33:45] which is trading off between our new estimate and our old
[00:33:47] estimate and our old estimate it can actually change per time
[00:33:49] estimate it can actually change per time step so just like how you can generally
[00:33:51] step so just like how you can generally Decay your learning rate you can change
[00:33:53] Decay your learning rate you can change your learning rate here and if your
[00:33:55] your learning rate here and if your learning rate is such that if you sum up
[00:33:57] learning rate is such that if you sum up all of its values for a particular state
[00:33:59] all of its values for a particular state it goes to Infinity but the square is
[00:34:01] it goes to Infinity but the square is less than infinity then you will
[00:34:03] less than infinity then you will converge to the True
[00:34:07] Value so and again this is a Prett the
[00:34:10] Value so and again this is a Prett the these are um a pretty common types of
[00:34:12] these are um a pretty common types of criteria we'll see for some of the
[00:34:14] criteria we'll see for some of the algorithms we have that sort of under
[00:34:16] algorithms we have that sort of under some sort of smoothness guarantees for
[00:34:18] some sort of smoothness guarantees for the um uh learning rates we'll have some
[00:34:20] the um uh learning rates we'll have some decent properties yeah for my
[00:34:23] decent properties yeah for my Ming those conditions aren't met like do
[00:34:26] Ming those conditions aren't met like do you definitely not have a guarantee or
[00:34:28] you definitely not have a guarantee or are there other conditions that can give
[00:34:30] are there other conditions that can give you a guarantee and those are just like
[00:34:31] you a guarantee and those are just like some of the great question so asking you
[00:34:34] some of the great question so asking you know um is it is it required to have
[00:34:38] know um is it is it required to have these conditions these are sufficient um
[00:34:41] these conditions these are sufficient um they aren't necessary always a lot of
[00:34:43] they aren't necessary always a lot of that will depend on the particular
[00:34:44] that will depend on the particular problem domain too and like what the
[00:34:46] problem domain too and like what the Dynamics and the reward is um to my
[00:34:48] Dynamics and the reward is um to my knowledge I'm not sure if there are
[00:34:50] knowledge I'm not sure if there are other really General conditions like
[00:34:51] other really General conditions like that but there might be for specific
[00:34:53] that but there might be for specific problem classes it's a good question
[00:34:59] now one of the problems with this is
[00:35:01] now one of the problems with this is that in generally it's a pretty high
[00:35:02] that in generally it's a pretty high variance estimator so you're kind of
[00:35:05] variance estimator so you're kind of getting certainly like with every visit
[00:35:07] getting certainly like with every visit or sorry certainly for um uh first visit
[00:35:11] or sorry certainly for um uh first visit Monte Carlo you're only updating the
[00:35:13] Monte Carlo you're only updating the state at most once per episode so it can
[00:35:15] state at most once per episode so it can take a long time so you can imagine that
[00:35:17] take a long time so you can imagine that if you have very different outcomes from
[00:35:20] if you have very different outcomes from the same starting state so maybe you
[00:35:22] the same starting state so maybe you know most of the time you have pretty
[00:35:24] know most of the time you have pretty average outcomes but maybe one in 100
[00:35:26] average outcomes but maybe one in 100 times you have a really bad outcome it's
[00:35:28] times you have a really bad outcome it's going to take a long time for that
[00:35:30] going to take a long time for that estimator to converge so in general this
[00:35:32] estimator to converge so in general this is a pretty high variance estimator even
[00:35:35] is a pretty high variance estimator even though it is often unbiased and it is
[00:35:38] though it is often unbiased and it is consistent and then the other big
[00:35:40] consistent and then the other big requirement is that it requires episodic
[00:35:42] requirement is that it requires episodic settings so you have to wait till the
[00:35:44] settings so you have to wait till the end of the episode to update your
[00:35:46] end of the episode to update your estimate and for here right now that
[00:35:49] estimate and for here right now that might not seem that bad but when we
[00:35:50] might not seem that bad but when we start getting into control de and
[00:35:52] start getting into control de and decision- making you might want to use
[00:35:54] decision- making you might want to use the data you have already in that
[00:35:55] the data you have already in that episode to change the behavior of the
[00:35:57] episode to change the behavior of the agent
[00:35:59] agent so you can imagine something like if
[00:36:00] so you can imagine something like if you're um doing like uh self-driving
[00:36:04] you're um doing like uh self-driving cars or something you're already getting
[00:36:05] cars or something you're already getting some evidence that um the car is not
[00:36:07] some evidence that um the car is not working as expected within a single
[00:36:09] working as expected within a single episode that might be really long you
[00:36:11] episode that might be really long you might want to use that information to
[00:36:12] might want to use that information to change how your steering for
[00:36:15] change how your steering for example okay all right so just to
[00:36:18] example okay all right so just to summarize here what it does is it's um
[00:36:20] summarize here what it does is it's um it's not using the Markoff process it's
[00:36:22] it's not using the Markoff process it's updating your value function estimate
[00:36:24] updating your value function estimate using a sample of the return to
[00:36:26] using a sample of the return to approximate the expectation and under
[00:36:28] approximate the expectation and under some pretty mild conditions it converges
[00:36:30] some pretty mild conditions it converges to the true value of the
[00:36:33] to the true value of the state and in some cases it will turn out
[00:36:36] state and in some cases it will turn out that even if you actually know the true
[00:36:38] that even if you actually know the true Dynamics model and reward you might
[00:36:39] Dynamics model and reward you might still want to do
[00:36:42] this and I think one thing that's useful
[00:36:44] this and I think one thing that's useful to think about here is um yeah systems
[00:36:46] to think about here is um yeah systems which you think the markof property
[00:36:48] which you think the markof property might be violated at least with the
[00:36:50] might be violated at least with the features that you'd be using to
[00:36:52] features that you'd be using to represent the
[00:36:55] state all right now let's go on to
[00:36:58] state all right now let's go on to temporal difference learning and this is
[00:37:00] temporal difference learning and this is again sort of related to Q learning
[00:37:01] again sort of related to Q learning which we'll get to in the next
[00:37:04] which we'll get to in the next lecture so set and Berto which is a
[00:37:07] lecture so set and Berto which is a textbook that um is an optional one for
[00:37:09] textbook that um is an optional one for oh yeah and is
[00:37:12] oh yeah and is it um that a good question so if we
[00:37:14] it um that a good question so if we don't know the rewards model how do we
[00:37:17] don't know the rewards model how do we calculate the rewards for the trajectory
[00:37:19] calculate the rewards for the trajectory ah great question so the Assumption here
[00:37:21] ah great question so the Assumption here is that um it's kind of like you either
[00:37:24] is that um it's kind of like you either are in a real setting where you can
[00:37:27] are in a real setting where you can sample these from an article or
[00:37:28] sample these from an article or something in the real world is giving
[00:37:29] something in the real world is giving you these so you may not have an
[00:37:31] you these so you may not have an explicit representation for the word
[00:37:32] explicit representation for the word model but you can get them so like your
[00:37:34] model but you can get them so like your customer buys something or they don't or
[00:37:36] customer buys something or they don't or you have um you know a side effect so
[00:37:39] you have um you know a side effect so you don't NE have a parametric model but
[00:37:40] you don't NE have a parametric model but you are getting real rewards it's a good
[00:37:44] you are getting real rewards it's a good question anybody else have any other
[00:37:46] question anybody else have any other questions about Monte Carlo before we go
[00:37:48] questions about Monte Carlo before we go on to temporal difference
[00:37:50] on to temporal difference learning and I'm going to call it just
[00:37:52] learning and I'm going to call it just temporal difference learning now um and
[00:37:54] temporal difference learning now um and then I'll specify that it's actually td0
[00:37:57] then I'll specify that it's actually td0 for most of what I'm going to talk about
[00:38:00] for most of what I'm going to talk about so just specify
[00:38:03] so just specify mostly discuss tt0 and I I'll specify
[00:38:07] mostly discuss tt0 and I I'll specify what I mean by the zero
[00:38:08] what I mean by the zero shortly so certain abto which is one of
[00:38:11] shortly so certain abto which is one of the optional textbooks for the class um
[00:38:13] the optional textbooks for the class um says if one had to identify one idea as
[00:38:15] says if one had to identify one idea as Central and Noble to RL it would
[00:38:17] Central and Noble to RL it would undoubtedly be temporal difference
[00:38:19] undoubtedly be temporal difference learning and what their point is is that
[00:38:23] learning and what their point is is that it really is sort of it's a way still to
[00:38:24] it really is sort of it's a way still to construct estimators both for control um
[00:38:27] construct estimators both for control um and for policy evaluation and the idea
[00:38:30] and for policy evaluation and the idea is if we think back to that tree I
[00:38:32] is if we think back to that tree I showed you and I'll show you some more
[00:38:33] showed you and I'll show you some more there's going to be a way to sort of
[00:38:35] there's going to be a way to sort of combine between the idea of sampling to
[00:38:37] combine between the idea of sampling to approximate expectations and
[00:38:39] approximate expectations and bootstrapping to approximate future
[00:38:42] bootstrapping to approximate future returns and we'll see that in a second
[00:38:45] returns and we'll see that in a second it is model free meaning you don't need
[00:38:46] it is model free meaning you don't need to have like a parametric representation
[00:38:48] to have like a parametric representation of the reward function or the Dynamics
[00:38:50] of the reward function or the Dynamics model and the nice thing is you can use
[00:38:52] model and the nice thing is you can use it in episodic settings or in infinite
[00:38:55] it in episodic settings or in infinite discounted Horizon settings you just
[00:38:57] discounted Horizon settings you just like set up off your robot and then it's
[00:38:58] like set up off your robot and then it's just going to have to learn to act
[00:39:01] just going to have to learn to act forever okay and one of the key ideas is
[00:39:04] forever okay and one of the key ideas is that we're going to update our estimates
[00:39:05] that we're going to update our estimates of the value of a state immediately so
[00:39:07] of the value of a state immediately so I'll put Pi here because we're still
[00:39:09] I'll put Pi here because we're still talking about a policy after every
[00:39:12] talking about a policy after every single Tuple of State action reward next
[00:39:15] single Tuple of State action reward next state all right so let's see how that
[00:39:18] state all right so let's see how that works so again remember our goal is just
[00:39:21] works so again remember our goal is just to compute the expected discounted sum
[00:39:23] to compute the expected discounted sum of rewards for a particular
[00:39:25] of rewards for a particular policy now let's think back to the bman
[00:39:27] policy now let's think back to the bman operator so if we know the mdp models
[00:39:30] operator so if we know the mdp models and we have a particular policy we could
[00:39:32] and we have a particular policy we could write the Bellman operator like
[00:39:34] write the Bellman operator like that and what we were doing in
[00:39:36] that and what we were doing in incremental every visit Monte Carlo is
[00:39:39] incremental every visit Monte Carlo is we were updating the estimate using one
[00:39:41] we were updating the estimate using one sample of the
[00:39:43] sample of the return and the idea now is to say well
[00:39:47] return and the idea now is to say well this was one
[00:39:48] this was one sample but maybe we could just maintain
[00:39:50] sample but maybe we could just maintain we have access to a value function why
[00:39:53] we have access to a value function why couldn't we look up and instead of
[00:39:54] couldn't we look up and instead of having you know what the rewards were in
[00:39:57] having you know what the rewards were in the state till the end of the
[00:39:59] the state till the end of the trajectory we have we observed a
[00:40:02] trajectory we have we observed a particular reward we got to a particular
[00:40:04] particular reward we got to a particular next
[00:40:05] next state why don't we use the value
[00:40:07] state why don't we use the value function for that
[00:40:11] state so what we're doing in this case
[00:40:14] state so what we're doing in this case is instead of using G we're plugging in
[00:40:16] is instead of using G we're plugging in the immediate reward plus gamma times
[00:40:18] the immediate reward plus gamma times the discounted sum of future rewards
[00:40:20] the discounted sum of future rewards using our current estimate of the value
[00:40:22] using our current estimate of the value function for that next state we reached
[00:40:29] and here the reason one of the things is
[00:40:31] and here the reason one of the things is that we don't need we don't have to wait
[00:40:33] that we don't need we don't have to wait do this
[00:40:35] do this immediately as soon as we reach S Prime
[00:40:42] immediately as soon as we reach S Prime so as soon as we reach S Prime we can
[00:40:44] so as soon as we reach S Prime we can you know as soon as we see the next
[00:40:45] you know as soon as we see the next state we can immediately update the
[00:40:46] state we can immediately update the value of our current state so we have to
[00:40:48] value of our current state so we have to wait till the end of the episode we can
[00:40:50] wait till the end of the episode we can use this for infinite Horizon
[00:40:53] problems
[00:40:56] problems okay so this is what that looks like and
[00:41:00] okay so this is what that looks like and we're of going to call that the TD
[00:41:03] we're of going to call that the TD Target and again that should look like
[00:41:05] Target and again that should look like machine learning it should look like um
[00:41:07] machine learning it should look like um uh what we just did with Monte Carlo
[00:41:09] uh what we just did with Monte Carlo that what we're plugging in here is
[00:41:11] that what we're plugging in here is we're saying we're taking our old
[00:41:13] we're saying we're taking our old estimate and we are moving it we are
[00:41:15] estimate and we are moving it we are shifting a little bit by our learning
[00:41:17] shifting a little bit by our learning rate
[00:41:19] rate towards our Target which is our reward
[00:41:22] towards our Target which is our reward plus our discount some of future rewards
[00:41:24] plus our discount some of future rewards using that plugin estimate
[00:41:28] and when we think of sort of how much
[00:41:30] and when we think of sort of how much our estimate is changing we often call
[00:41:33] our estimate is changing we often call that the td0
[00:41:35] that the td0 error which looks at excuse me how
[00:41:38] error which looks at excuse me how different is my current estimate at the
[00:41:40] different is my current estimate at the value of a state versus the estimate
[00:41:43] value of a state versus the estimate that I'm plugging
[00:41:45] that I'm plugging in and again if you've seen Q learning
[00:41:47] in and again if you've seen Q learning before this is going to look really
[00:41:49] before this is going to look really similar to what we had but there's no
[00:41:51] similar to what we had but there's no Max or things like that you'll see those
[00:41:54] Max or things like that you'll see those soon okay
[00:41:58] soon okay so the td0 learning algorithm just looks
[00:42:01] so the td0 learning algorithm just looks like the following you sample a tuple of
[00:42:04] like the following you sample a tuple of a state action rewards uh next state you
[00:42:08] a state action rewards uh next state you update the value for that starting State
[00:42:11] update the value for that starting State and you repeat and so your T goes to t +
[00:42:14] and you repeat and so your T goes to t + 1 and then you get the next coule just
[00:42:17] 1 and then you get the next coule just do this over and over and over again so
[00:42:19] do this over and over and over again so like in our Mars Rover example you have
[00:42:21] like in our Mars Rover example you have sort of State action reward next state
[00:42:23] sort of State action reward next state you update and then you just shift along
[00:42:31] let's see what that might look like
[00:42:33] let's see what that might look like here
[00:42:34] here so in this case let's imagine we have a
[00:42:37] so in this case let's imagine we have a policy where we always take action A1
[00:42:40] policy where we always take action A1 we're going to make our discount Factor
[00:42:42] we're going to make our discount Factor one to make the math easy um and we're
[00:42:45] one to make the math easy um and we're going to assume that any action from
[00:42:47] going to assume that any action from State one or S7 terminates the
[00:42:50] State one or S7 terminates the episode and then what we see in this
[00:42:52] episode and then what we see in this case is we have the following trajectory
[00:42:53] case is we have the following trajectory we start in state S3 we take action A1
[00:42:57] we start in state S3 we take action A1 we get a reward of zero so this is the
[00:42:59] we get a reward of zero so this is the reward we transition to State S2 and so
[00:43:02] reward we transition to State S2 and so so forth till the end of the
[00:43:04] so forth till the end of the episode so what we would have in this
[00:43:07] episode so what we would have in this case is that we would make it um so that
[00:43:11] case is that we would make it um so that the first up we update we would do be V
[00:43:13] the first up we update we would do be V of
[00:43:14] of S3 and what we would say is that's my
[00:43:16] S3 and what we would say is that's my old estimate of V of
[00:43:18] old estimate of V of S3 * 1 - Alpha I've just Rewritten the
[00:43:22] S3 * 1 - Alpha I've just Rewritten the above equation here because there's this
[00:43:24] above equation here because there's this was basically one and this is Alpha *
[00:43:27] was basically one and this is Alpha minus B plus Alpha the immediate re
[00:43:31] minus B plus Alpha the immediate re plus gamma V of
[00:43:36] S2 so that's what that would look like
[00:43:39] S2 so that's what that would look like and
[00:43:40] and here my imagine that I'm I I've
[00:43:43] here my imagine that I'm I I've initialized all of them to be zero to
[00:43:45] initialized all of them to be zero to start so this would still just look like
[00:43:48] start so this would still just look like zero okay and in fact the only what
[00:43:52] zero okay and in fact the only what would be the only state I would update
[00:43:53] would be the only state I would update to not be zero in this episode
[00:44:13] for it to be updated not to be zero
[00:44:15] for it to be updated not to be zero either its immediate reward has to be
[00:44:17] either its immediate reward has to be one or it has to be transitioning to a
[00:44:19] one or it has to be transitioning to a state whose value is not zero
[00:44:30] yeah so what we're seeing here is that
[00:44:33] yeah so what we're seeing here is that um in this case we have state action
[00:44:35] um in this case we have state action reward next date and so this is the TD
[00:44:38] reward next date and so this is the TD update and what I was saying here is
[00:44:41] update and what I was saying here is that we've initialized all of them to be
[00:44:43] that we've initialized all of them to be zero which means that in order for their
[00:44:45] zero which means that in order for their value to change from being zero either
[00:44:49] value to change from being zero either their immediate reward has to be non Zer
[00:44:52] their immediate reward has to be non Zer or we have to transition to a state
[00:44:53] or we have to transition to a state whose value is not zero because all of
[00:44:57] whose value is not zero because all of them their current value is
[00:45:00] zero
[00:45:04] yeah were you going to guess which state
[00:45:07] yeah were you going to guess which state it's upd yeah which one it was well when
[00:45:10] it's upd yeah which one it was well when you're in
[00:45:12] you're in state state one you have a reward one
[00:45:14] state state one you have a reward one that's right yeah so this is so none you
[00:45:16] that's right yeah so this is so none you don't see any reward here until you get
[00:45:18] don't see any reward here until you get to State S1 I'll just highlight
[00:45:22] to State S1 I'll just highlight here so at that point is when you update
[00:45:25] here so at that point is when you update your value function that's the first
[00:45:27] your value function that's the first first time that you get to anything
[00:45:28] first time that you get to anything that's that any reward becomes non zero
[00:45:31] that's that any reward becomes non zero so in that case what you get is S1 is
[00:45:33] so in that case what you get is S1 is equal to V of S1 1 - Alpha + Alpha * 1 +
[00:45:39] equal to V of S1 1 - Alpha + Alpha * 1 + gamma B of s Terminal S terminal is
[00:45:43] gamma B of s Terminal S terminal is always zero so it just becomes Alpha *
[00:45:50] 1 so why am I make I do a lot of algebra
[00:45:53] 1 so why am I make I do a lot of algebra here I want to do it because um if you
[00:45:56] here I want to do it because um if you work this out and I won't go through it
[00:45:58] work this out and I won't go through it here but I think it's a useful
[00:46:00] here but I think it's a useful exercise the TD episode TD estimate you
[00:46:04] exercise the TD episode TD estimate you would get for your whole value function
[00:46:06] would get for your whole value function at the end of this episode is quite
[00:46:08] at the end of this episode is quite different than what you get with Monte
[00:46:10] different than what you get with Monte Carlo so TD updates after every single
[00:46:13] Carlo so TD updates after every single tupple every single state action reward
[00:46:15] tupple every single state action reward next state Tuple and so that means when
[00:46:17] next state Tuple and so that means when you reach the end of the episode if you
[00:46:19] you reach the end of the episode if you look at what your value function would
[00:46:21] look at what your value function would be and I've written the value function
[00:46:22] be and I've written the value function here as um just as a vector but this is
[00:46:25] here as um just as a vector but this is the value of S1 this is the value of s 7
[00:46:27] the value of S1 this is the value of s 7 so I've just written it as a vector this
[00:46:30] so I've just written it as a vector this is what your value function would be it
[00:46:32] is what your value function would be it would say oh my current estimate for S1
[00:46:34] would say oh my current estimate for S1 is one and everything else is zero but
[00:46:37] is one and everything else is zero but if you look at first visit Monte Carlo
[00:46:39] if you look at first visit Monte Carlo it's quite different okay and if we make
[00:46:42] it's quite different okay and if we make gamma equal to one here which I said it
[00:46:44] gamma equal to one here which I said it would be it would be 1 one 1 0 0 0
[00:46:47] would be it would be 1 one 1 0 0 0 0 why is this because Monte Carlo waits
[00:46:50] 0 why is this because Monte Carlo waits till the end of the episode and then it
[00:46:52] till the end of the episode and then it uses the returns to update any state
[00:46:56] uses the returns to update any state that was visited once in that
[00:46:59] that was visited once in that episode and the reason that's important
[00:47:01] episode and the reason that's important is that now actually we have we filled
[00:47:03] is that now actually we have we filled in a lot more things because we knew we
[00:47:05] in a lot more things because we knew we observed in that case that not just did
[00:47:07] observed in that case that not just did we get a reward here but then we saw
[00:47:09] we get a reward here but then we saw what S2 got which was also a reward of
[00:47:11] what S2 got which was also a reward of one and what S3 got which was also a
[00:47:13] one and what S3 got which was also a reward of
[00:47:14] reward of one and the reason I bring this up is
[00:47:16] one and the reason I bring this up is that there's going to be different
[00:47:17] that there's going to be different choices about how these behave
[00:47:18] choices about how these behave particularly when you don't have a lot
[00:47:19] particularly when you don't have a lot of data at the beginning which may be
[00:47:21] of data at the beginning which may be more or less data efficient or sample
[00:47:24] more or less data efficient or sample efficient and ideas of sample efficiency
[00:47:26] efficient and ideas of sample efficiency will come up a lot we'll see that a lot
[00:47:28] will come up a lot we'll see that a lot later on um but we'll see it on uh
[00:47:30] later on um but we'll see it on uh Thursday as
[00:47:32] Thursday as well all right so what does this look
[00:47:34] well all right so what does this look like in terms of the
[00:47:35] like in terms of the tree so we go back to our tree which is
[00:47:38] tree so we go back to our tree which is like expanding out potential Futures
[00:47:40] like expanding out potential Futures what we can see here is that TD is
[00:47:43] what we can see here is that TD is updating the value estimate using a
[00:47:44] updating the value estimate using a sample of St + one to approximate an
[00:47:48] sample of St + one to approximate an expectation so in reality if you're
[00:47:50] expectation so in reality if you're doing dynamic programming you would want
[00:47:52] doing dynamic programming you would want to do a weighted expectation over all
[00:47:54] to do a weighted expectation over all the next States you could reach weight
[00:47:57] the next States you could reach weight by the probability of getting there what
[00:47:59] by the probability of getting there what TD is doing is it's just sampling one of
[00:48:02] TD is doing is it's just sampling one of those and that sample is you know an
[00:48:05] those and that sample is you know an approximation of that expectation so
[00:48:08] approximation of that expectation so we're going from this to sampling the
[00:48:11] we're going from this to sampling the next
[00:48:13] next state but similar to dynamic programming
[00:48:15] state but similar to dynamic programming it is then bootstrapping so unlike Monte
[00:48:18] it is then bootstrapping so unlike Monte Carlo which goes all the way out to get
[00:48:20] Carlo which goes all the way out to get a sample of that value function here
[00:48:23] a sample of that value function here we're just plugging in V so this part
[00:48:25] we're just plugging in V so this part looks the same
[00:48:29] so TD does both sampling to approximate
[00:48:33] so TD does both sampling to approximate expectations and it boot straps by using
[00:48:35] expectations and it boot straps by using your existing estim of the value
[00:48:44] function all right so let's just do a
[00:48:46] function all right so let's just do a check your understanding um so this is a
[00:48:50] check your understanding um so this is a poll so what I'd like you to think about
[00:48:52] poll so what I'd like you to think about is how this learning rate might affect
[00:48:53] is how this learning rate might affect things so whether different choices of
[00:48:57] things so whether different choices of this is going to weigh the TD Target or
[00:48:59] this is going to weigh the TD Target or more uh more or less than the pass V
[00:49:02] more uh more or less than the pass V estimate um and what might happen when
[00:49:06] estimate um and what might happen when your state space is stochastic meaning
[00:49:09] your state space is stochastic meaning that when you start in one state you
[00:49:10] that when you start in one state you might end up in multiple next States
[00:49:13] might end up in multiple next States what does that mean about convergence
[00:49:15] what does that mean about convergence and the implication for learning
[00:49:18] and the implication for learning rates as well as thinking about
[00:49:20] rates as well as thinking about deterministic markof decision processes
[00:49:22] deterministic markof decision processes deterministic Market decision processes
[00:49:25] deterministic Market decision processes what I mean by that is that
[00:49:29] oops P of S Prime given sa is equal to 1
[00:49:34] oops P of S Prime given sa is equal to 1 for
[00:49:37] for exactly 1 s prime meaning that there's
[00:49:40] exactly 1 s prime meaning that there's no stochasticity when you're in a state
[00:49:42] no stochasticity when you're in a state in action you always go to one
[00:49:43] in action you always go to one particular next
[00:49:45] particular next state so that's a deterministic barup
[00:49:47] state so that's a deterministic barup decision
[00:49:48] decision process so just take a few minutes now
[00:49:50] process so just take a few minutes now and um look into this
[00:50:04] and you should be able to select all
[00:50:06] and you should be able to select all that are true but if you can't let me
[00:50:09] that are true but if you can't let me know you cannot
[00:50:12] know you cannot okay right well then I mean again these
[00:50:15] okay right well then I mean again these are only for your thoughts so just try
[00:50:16] are only for your thoughts so just try to like write down for yourself which of
[00:50:18] to like write down for yourself which of these you think are true and then we'll
[00:50:20] these you think are true and then we'll talk about them the second I'll check
[00:50:22] talk about them the second I'll check into these for next time
[00:51:12] yes that's what I just heard sorry about
[00:51:14] yes that's what I just heard sorry about that so I'll try to fix that for next
[00:51:15] that so I'll try to fix that for next time just try to have in your head of
[00:51:17] time just try to have in your head of which ones you think are correct and
[00:51:18] which ones you think are correct and I'll ask you to compare with someone in
[00:51:19] I'll ask you to compare with someone in a second thanks for letting me
[00:51:25] know for
[00:51:58] all right turn to your neighbor um and
[00:52:00] all right turn to your neighbor um and check and uh particularly focus on the
[00:52:04] check and uh particularly focus on the last
[00:52:05] last two and see if you agree on your answers
[00:52:08] two and see if you agree on your answers for those
[00:52:38] like let's say MVP a single
[00:53:54] no
[00:54:02] I'm GNA make
[00:54:12] [Music]
[00:54:30] you like construct pry
[00:55:21] yeah I think people
[00:55:28] no have any
[00:55:57] all right great had some great
[00:55:59] all right great had some great discussions um okay so for the first one
[00:56:03] discussions um okay so for the first one this is going to be false because if we
[00:56:07] this is going to be false because if we have Alpha equals z then we don't care
[00:56:09] have Alpha equals z then we don't care about the TD Target at all just just
[00:56:13] about the TD Target at all just just totally drops out so we're always we
[00:56:15] totally drops out so we're always we never
[00:56:16] never update um in the second case this is
[00:56:21] update um in the second case this is true because this means if Alpha is
[00:56:23] true because this means if Alpha is equal to one then this part and this
[00:56:26] equal to one then this part and this part cancels out we just have
[00:56:28] part cancels out we just have this so that means whenever we see an
[00:56:31] this so that means whenever we see an update we always update uh we totally
[00:56:34] update we always update uh we totally change your estimate
[00:56:35] change your estimate potentially the third one is a little
[00:56:37] potentially the third one is a little bit subtle um this is true does somebody
[00:56:40] bit subtle um this is true does somebody want to give me an example where this
[00:56:41] want to give me an example where this might
[00:56:45] occur yeah yeah um if you have like two
[00:56:48] occur yeah yeah um if you have like two states where they just keep pointing at
[00:56:50] states where they just keep pointing at each other is that the cas yes and in
[00:56:53] each other is that the cas yes and in particular if there's um if you could
[00:56:56] particular if there's um if you could would go to either of those state with
[00:56:57] would go to either of those state with some
[00:56:58] some probability yeah so so if you're in
[00:57:01] probability yeah so so if you're in these are cases so imagine that like I I
[00:57:03] these are cases so imagine that like I I to think of it is like a coin flip so
[00:57:05] to think of it is like a coin flip so imagine that you have one state where
[00:57:07] imagine that you have one state where after this you either go to a
[00:57:09] after this you either go to a state maybe it's 50% probability you get
[00:57:12] state maybe it's 50% probability you get plus one and 50% probability you get
[00:57:14] plus one and 50% probability you get minus
[00:57:16] minus one and then your problem just resets so
[00:57:18] one and then your problem just resets so imagine it's like a really short problem
[00:57:20] imagine it's like a really short problem you start off you get zero re W you
[00:57:22] you start off you get zero re W you transition to a state and then your
[00:57:23] transition to a state and then your episode resets so in this case either on
[00:57:26] episode resets so in this case either on that um round you're gonna get plus one
[00:57:28] that um round you're gonna get plus one or you're GNA get minus one so you're
[00:57:31] or you're GNA get minus one so you're either get plus one or minus one here
[00:57:32] either get plus one or minus one here and you'll just flip back and forth
[00:57:34] and you'll just flip back and forth between the plus one minus one plus
[00:57:35] between the plus one minus one plus oneus one plus one one so that's just to
[00:57:38] oneus one plus one one so that's just to highlight that if you do have systems
[00:57:39] highlight that if you do have systems which are stochastic the fact that in
[00:57:42] which are stochastic the fact that in your target you are using a single
[00:57:44] your target you are using a single sample of that stochasticity to
[00:57:46] sample of that stochasticity to approximate the expectation um can be
[00:57:49] approximate the expectation um can be bad but that does not mean and I guess
[00:57:52] bad but that does not mean and I guess this gets to I think was it like that
[00:57:54] this gets to I think was it like that was asking before that like in many of
[00:57:56] was asking before that like in many of these cases it it's sort of the ne um
[00:57:59] these cases it it's sort of the ne um there's the sort of cases where like it
[00:58:02] there's the sort of cases where like it might be possible that this would happen
[00:58:04] might be possible that this would happen but it won't always so in this case
[00:58:06] but it won't always so in this case there do exist deterministic systems um
[00:58:08] there do exist deterministic systems um where um even if Alpha is equal to one
[00:58:11] where um even if Alpha is equal to one you can converge so again think of
[00:58:13] you can converge so again think of something that I like often to think
[00:58:15] something that I like often to think about really small mdps to get some
[00:58:16] about really small mdps to get some intuition for this if you have a case
[00:58:18] intuition for this if you have a case where there's just a terminal State and
[00:58:22] where there's just a terminal State and there's no more transitions so like you
[00:58:23] there's no more transitions so like you get to some point where you always go to
[00:58:25] get to some point where you always go to some terminal state dat and then it's
[00:58:27] some terminal state dat and then it's plus 10 there there's no more updates
[00:58:30] plus 10 there there's no more updates then just plus 10 there's no more
[00:58:33] then just plus 10 there's no more expectation and in general any case
[00:58:35] expectation and in general any case where you know if there's no
[00:58:36] where you know if there's no stochasticity and you're near the end
[00:58:38] stochasticity and you're near the end and there's no more stochasticity in
[00:58:39] and there's no more stochasticity in that episode um those can be cases real
[00:58:42] that episode um those can be cases real St
[00:58:43] St converge okay great so this is just and
[00:58:46] converge okay great so this is just and I I encourage you um to go through some
[00:58:48] I I encourage you um to go through some of the worked examples if you want to
[00:58:50] of the worked examples if you want to just to see some more comparisons over
[00:58:52] just to see some more comparisons over uh you know the difference between Monte
[00:58:54] uh you know the difference between Monte Carlo and TD methods in this case
[00:58:57] Carlo and TD methods in this case just to summarize what we're doing in TD
[00:58:59] just to summarize what we're doing in TD learning we're bootstrapping and
[00:59:00] learning we're bootstrapping and sampling we're sampling to approximate
[00:59:02] sampling we're sampling to approximate our expectation overall the
[00:59:04] our expectation overall the stochasticity we're bootstrapping
[00:59:06] stochasticity we're bootstrapping because we don't want to use a full
[00:59:07] because we don't want to use a full return um we are in taking V to
[00:59:11] return um we are in taking V to approximate that can be use an episodic
[00:59:13] approximate that can be use an episodic or infinite Horizon settings it is
[00:59:15] or infinite Horizon settings it is generally lower variance we're doing
[00:59:17] generally lower variance we're doing lots and lots and lots of updates um it
[00:59:20] lots and lots and lots of updates um it is a consistent estimator if your
[00:59:21] is a consistent estimator if your learning rate Alpha satisfies the same
[00:59:23] learning rate Alpha satisfies the same conditions specified for incremental
[00:59:25] conditions specified for incremental Monte Carlo policy
[00:59:28] Monte Carlo policy valuation I here today only introduce
[00:59:31] valuation I here today only introduce td0 What td0 refers to is you take the
[00:59:34] td0 What td0 refers to is you take the immediate reward then you immediately
[00:59:36] immediate reward then you immediately bootstrap and plug in the value of um
[00:59:40] bootstrap and plug in the value of um the next state so we did
[00:59:48] r+
[00:59:51] r+ okay we did r + gamma V of S Prime
[00:59:57] okay we did r + gamma V of S Prime versus summing up all your discounted
[00:59:59] versus summing up all your discounted rewards for the whole episode in general
[01:00:01] rewards for the whole episode in general you could have something kind of in
[01:00:02] you could have something kind of in between so you could
[01:00:04] between so you could have plus RT + 1+ gamma
[01:00:09] have plus RT + 1+ gamma squ so you could have something like
[01:00:11] squ so you could have something like this so in general you could say like do
[01:00:14] this so in general you could say like do some sort of combination of like using
[01:00:17] some sort of combination of like using partial returns and then bootstrapping
[01:00:19] partial returns and then bootstrapping um there's a lot of different TD methods
[01:00:22] um there's a lot of different TD methods that kind of interpolate between taking
[01:00:24] that kind of interpolate between taking one step and then plugging in the value
[01:00:26] one step and then plugging in the value versus only plugging not using any value
[01:00:29] versus only plugging not using any value function
[01:00:30] function approximation and if you want to think
[01:00:31] approximation and if you want to think about this graphically it's kind of like
[01:00:33] about this graphically it's kind of like thinking
[01:00:34] thinking about do you plug in V of S Prime here
[01:00:38] about do you plug in V of S Prime here or do you plug it in
[01:00:40] or do you plug it in no way
[01:00:42] no way lower yeah is there
[01:00:46] lower yeah is there like is there empal like um estimate of
[01:00:50] like is there empal like um estimate of what like a good trade of for for like
[01:00:53] what like a good trade of for for like computational complexity versus uh
[01:00:56] computational complexity versus uh performance like have they found like a
[01:00:58] performance like have they found like a good number for it good question
[01:01:01] good number for it good question unfortunately in many cases it will be
[01:01:03] unfortunately in many cases it will be depending on the domain um one thing I
[01:01:06] depending on the domain um one thing I think to think about here too is
[01:01:09] think to think about here too is that you can think of this part doesn't
[01:01:11] that you can think of this part doesn't require the Markoff assumption right so
[01:01:14] require the Markoff assumption right so if you have a system where you're not
[01:01:16] if you have a system where you're not confident but maybe you're like well
[01:01:18] confident but maybe you're like well maybe it's
[01:01:19] maybe it's like I'm willing to say that I'll plug
[01:01:22] like I'm willing to say that I'll plug in a markup assumption eventually
[01:01:23] in a markup assumption eventually because it's going to be lower variance
[01:01:25] because it's going to be lower variance um but want to preserve the fact that
[01:01:27] um but want to preserve the fact that maybe it's not markof and sort of have a
[01:01:28] maybe it's not markof and sort of have a short Horizon um often people do use
[01:01:31] short Horizon um often people do use something in between the two so they
[01:01:33] something in between the two so they often do consider this in between for
[01:01:35] often do consider this in between for for multiple reasons but it gives you
[01:01:36] for multiple reasons but it gives you some of this flexibility it often is um
[01:01:39] some of this flexibility it often is um a lower bias it's a great
[01:01:43] a lower bias it's a great question all right what we're going to
[01:01:45] question all right what we're going to do now is think about also how some of
[01:01:46] do now is think about also how some of these ideas relate to dynamic
[01:01:48] these ideas relate to dynamic programming which is what we saw in an
[01:01:50] programming which is what we saw in an earlier lecture because we could use
[01:01:51] earlier lecture because we could use this also for policy evaluation we know
[01:01:54] this also for policy evaluation we know how to use it for policy evaluation
[01:01:56] how to use it for policy evaluation if we are given the models but some of
[01:01:59] if we are given the models but some of you guys might have been thinking well
[01:02:00] you guys might have been thinking well we have data now if we have data because
[01:02:03] we have data now if we have data because we're taking the policy in the
[01:02:04] we're taking the policy in the environment couldn't we use that to
[01:02:06] environment couldn't we use that to estimate a reward model or couldn't we
[01:02:07] estimate a reward model or couldn't we use that to estimate the Dynamics model
[01:02:10] use that to estimate the Dynamics model and that's what's known as certainty
[01:02:12] and that's what's known as certainty equivalence um approaches so the idea
[01:02:15] equivalence um approaches so the idea here is that you're going to be getting
[01:02:16] here is that you're going to be getting data as you execute this policy and you
[01:02:19] data as you execute this policy and you can compute a Dynamics model from that
[01:02:21] can compute a Dynamics model from that data so you could use like a maximum
[01:02:23] data so you could use like a maximum likelihood mdp model remember right now
[01:02:26] likelihood mdp model remember right now we're in the tabular setting so we can
[01:02:28] we're in the tabular setting so we can have a parameter for every single state
[01:02:30] have a parameter for every single state in action so we can just count we can
[01:02:32] in action so we can just count we can just say how many times was I in this
[01:02:34] just say how many times was I in this state took this action and transitioned
[01:02:36] state took this action and transitioned to this next state divided by the number
[01:02:38] to this next state divided by the number of times I was in that state in action
[01:02:41] of times I was in that state in action so this just gives you um a maximum
[01:02:43] so this just gives you um a maximum likelihood estimate of the Dynamics
[01:02:44] likelihood estimate of the Dynamics model and you can do the same thing for
[01:02:46] model and you can do the same thing for the reward model and of course as you
[01:02:48] the reward model and of course as you might imagine you can do this with much
[01:02:50] might imagine you can do this with much more complicated function approximators
[01:02:52] more complicated function approximators like deep neural networks too but the
[01:02:54] like deep neural networks too but the idea is that once you have this model
[01:02:57] idea is that once you have this model and it's called a certainty equivalence
[01:02:59] and it's called a certainty equivalence model because we're now going to ignore
[01:03:02] model because we're now going to ignore any error in these models so we have
[01:03:05] any error in these models so we have finite data these models will definitely
[01:03:06] finite data these models will definitely be wrong but let's ignore that for now
[01:03:09] be wrong but let's ignore that for now so once you have this maximum likelihood
[01:03:11] so once you have this maximum likelihood mdp model you can just compute the value
[01:03:14] mdp model you can just compute the value of a policy using the same methods we
[01:03:16] of a policy using the same methods we saw last week because you have Dynamics
[01:03:19] saw last week because you have Dynamics model now and a reward model and you can
[01:03:22] model now and a reward model and you can see some examples about this at the end
[01:03:24] see some examples about this at the end of the lecture slides
[01:03:30] so one of the benefits of this and this
[01:03:32] so one of the benefits of this and this gets to question is this is really data
[01:03:34] gets to question is this is really data efficient so I showed you an example for
[01:03:36] efficient so I showed you an example for the Mars rover before where we only
[01:03:39] the Mars rover before where we only updated one of the states with TD
[01:03:41] updated one of the states with TD learning we updated three of the states
[01:03:44] learning we updated three of the states with um uh Monte Carlo what this does
[01:03:47] with um uh Monte Carlo what this does here is it tries to update it computes
[01:03:49] here is it tries to update it computes the Dynamics mod reward model for all
[01:03:51] the Dynamics mod reward model for all states and actions and then it tries to
[01:03:53] states and actions and then it tries to update all of them okay so it's going to
[01:03:56] update all of them okay so it's going to compute a value for every single
[01:03:59] compute a value for every single state now the downside of that is that
[01:04:02] state now the downside of that is that now we're doing policy evaluation with a
[01:04:03] now we're doing policy evaluation with a full model which is either going to be
[01:04:05] full model which is either going to be something like s s a for iterative
[01:04:07] something like s s a for iterative methods or maybe even worse so it's
[01:04:10] methods or maybe even worse so it's computationally expensive but it's
[01:04:11] computationally expensive but it's really data
[01:04:14] really data efficient because as soon as you reach
[01:04:16] efficient because as soon as you reach any state for which um you get say a
[01:04:18] any state for which um you get say a positive reward you can kind of
[01:04:20] positive reward you can kind of propagate that to any other state that
[01:04:22] propagate that to any other state that you know is possible to reach from there
[01:04:26] you know is possible to reach from there it's still consistent it's going to
[01:04:28] it's still consistent it's going to converge to the right right thing for
[01:04:30] converge to the right right thing for Markoff models um and I it can easily
[01:04:34] Markoff models um and I it can easily generally be used for all policy
[01:04:35] generally be used for all policy evaluation which we're going to get into
[01:04:37] evaluation which we're going to get into yeah sorry what is NSA in this equation
[01:04:40] yeah sorry what is NSA in this equation sorry great question NSA here is the
[01:04:42] sorry great question NSA here is the number of times we've been in that state
[01:04:43] number of times we've been in that state and taken that
[01:04:45] and taken that action yeah so this is counts
[01:04:59] okay okay yeah this seems like pretty
[01:05:01] okay okay yeah this seems like pretty similar to Carlo I think so um is just
[01:05:05] similar to Carlo I think so um is just the difference that you're Genera like
[01:05:07] the difference that you're Genera like probability as opposed to like
[01:05:09] probability as opposed to like calculating G great question um we're
[01:05:11] calculating G great question um we're going to hold that thought for asking
[01:05:13] going to hold that thought for asking how similar is this to mtic car not it's
[01:05:15] how similar is this to mtic car not it's actually going to be pretty different
[01:05:16] actually going to be pretty different we're going to see this in a second um
[01:05:18] we're going to see this in a second um we are going to be we're going to be
[01:05:20] we are going to be we're going to be using we are using our data similar to
[01:05:22] using we are using our data similar to mon we're using the data to we're going
[01:05:24] mon we're using the data to we're going to use the data here to compute models
[01:05:26] to use the data here to compute models and then propagate information but
[01:05:27] and then propagate information but they're going to end up making some
[01:05:29] they're going to end up making some interesting different decisions so let's
[01:05:30] interesting different decisions so let's see that now it's a great
[01:05:33] see that now it's a great PR okay so now let's get into batch
[01:05:35] PR okay so now let's get into batch policy evaluation so I've said like
[01:05:37] policy evaluation so I've said like there are these different methods they
[01:05:38] there are these different methods they might have different computational
[01:05:40] might have different computational complexity they might be more or less
[01:05:41] complexity they might be more or less data
[01:05:42] data efficient so one thing that you might
[01:05:45] efficient so one thing that you might imagine doing and we'll see this a lot
[01:05:47] imagine doing and we'll see this a lot shortly a lot more next time is well if
[01:05:50] shortly a lot more next time is well if I have some data how could I best use
[01:05:52] I have some data how could I best use the data that I have and this comes a
[01:05:54] the data that I have and this comes a lot up a lot in the research that me and
[01:05:56] lot up a lot in the research that me and my lab do because we're often dealing
[01:05:57] my lab do because we're often dealing with patient data or student data or
[01:06:00] with patient data or student data or legal data or others where like it's
[01:06:01] legal data or others where like it's really expensive to get the data or it's
[01:06:03] really expensive to get the data or it's costly or could be harmful and we want
[01:06:05] costly or could be harmful and we want to get as much information as we can out
[01:06:07] to get as much information as we can out of the data we have so when I say batch
[01:06:10] of the data we have so when I say batch what I mean is imagine that you have a
[01:06:11] what I mean is imagine that you have a set of K episodes and now what you want
[01:06:13] set of K episodes and now what you want to do is you want to do value you want
[01:06:16] to do is you want to do value you want to do um policy evaluation just with
[01:06:18] to do um policy evaluation just with that data so what we're going to do is
[01:06:20] that data so what we're going to do is repeatedly sample one of the episodes
[01:06:22] repeatedly sample one of the episodes that we have of those K and we're going
[01:06:24] that we have of those K and we're going to apply Monte Carlo or TD Z to that
[01:06:26] to apply Monte Carlo or TD Z to that episode we're just going to do that over
[01:06:28] episode we're just going to do that over and over and over and over again and
[01:06:30] and over and over and over again and we'll see this a lot more next time as
[01:06:31] we'll see this a lot more next time as well and so the idea is to just
[01:06:34] well and so the idea is to just understand um if we do this given that
[01:06:37] understand um if we do this given that finite amount of data what will Monte
[01:06:39] finite amount of data what will Monte Carlo and td0 converge to in terms of
[01:06:41] Carlo and td0 converge to in terms of the um evaluation of the policy so let's
[01:06:44] the um evaluation of the policy so let's go through that this this really nice
[01:06:46] go through that this this really nice example from sardo to kind of illustrate
[01:06:48] example from sardo to kind of illustrate this we have a really small domain we
[01:06:51] this we have a really small domain we have two states we're going to say gamma
[01:06:53] have two states we're going to say gamma is equal to one there's no discounting
[01:06:55] is equal to one there's no discounting and we have eight episodes of experience
[01:06:58] and we have eight episodes of experience so in one episode we started in um State
[01:07:02] so in one episode we started in um State uh a we got a reward of zero we
[01:07:05] uh a we got a reward of zero we transitioned to B and we got another
[01:07:07] transitioned to B and we got another reward of zero okay so in this case you
[01:07:10] reward of zero okay so in this case you can think of it as like this had a
[01:07:12] can think of it as like this had a trajectory like
[01:07:14] trajectory like that in some episodes we started in
[01:07:17] that in some episodes we started in state B and we just got an immediate
[01:07:20] state B and we just got an immediate reward of one and we observe that six
[01:07:23] reward of one and we observe that six times so in six trajectories we just
[01:07:25] times so in six trajectories we just happened to to start in state B and we
[01:07:26] happened to to start in state B and we got a reward of one and then in one
[01:07:29] got a reward of one and then in one trajectory we started in state B and we
[01:07:31] trajectory we started in state B and we got a reward of
[01:07:33] got a reward of zero so first imagine if you ran TD
[01:07:36] zero so first imagine if you ran TD updates over this data in infinite
[01:07:38] updates over this data in infinite amount of time what do you think the
[01:07:40] amount of time what do you think the estimative VB would
[01:07:44] be remember that what it works for there
[01:07:47] be remember that what it works for there is we have 1us Alpha times our old
[01:07:51] is we have 1us Alpha times our old estimate so I'll just put V plus Alpha *
[01:07:55] estimate so I'll just put V plus Alpha * our immediate reward plus gamma times
[01:07:58] our immediate reward plus gamma times the next state but here it's just
[01:08:01] the next state but here it's just terminal because we always
[01:08:04] terminal because we always terminate after B we always terminate so
[01:08:07] terminate after B we always terminate so you never get any future discounted
[01:08:08] you never get any future discounted rewards in this case so what the updates
[01:08:11] rewards in this case so what the updates look like for TD learning is that you
[01:08:13] look like for TD learning is that you would have 1- Alpha time your old
[01:08:14] would have 1- Alpha time your old estimate plus Alpha time whatever reward
[01:08:16] estimate plus Alpha time whatever reward you get in
[01:08:18] you get in B and imagine you just iterate over
[01:08:21] B and imagine you just iterate over these over and over and over
[01:08:23] these over and over and over again somebody have any guesses of what
[01:08:26] again somebody have any guesses of what the reward would be for
[01:08:36] v um would it be 75 yeah somebody else
[01:08:42] v um would it be 75 yeah somebody else want to explain how why it's
[01:08:51] 75 is see some
[01:08:54] 75 is see some nods yeah cuz in this case we had eight
[01:08:58] nods yeah cuz in this case we had eight episodes in two of
[01:09:00] episodes in two of them when we started in B we got zero
[01:09:03] them when we started in B we got zero and in six of them we got
[01:09:05] and in six of them we got one so we just average those rewards and
[01:09:09] one so we just average those rewards and imagine you know we're just doing this
[01:09:11] imagine you know we're just doing this many many many many many many
[01:09:13] many many many many many many times So eventually you would just
[01:09:15] times So eventually you would just converge to this estimate being
[01:09:21] 75 what about for Monte
[01:09:24] 75 what about for Monte Carlo so let's do Monte Carlo for v
[01:09:27] Carlo so let's do Monte Carlo for v b what would that look like so remember
[01:09:31] b what would that look like so remember for Monte Carlo it would be 1 Alpha * B
[01:09:35] for Monte Carlo it would be 1 Alpha * B of
[01:09:36] of B plus Alpha * G where you start in
[01:09:42] B plus Alpha * G where you start in state
[01:09:48] B is it going to be the same thing is it
[01:09:50] B is it going to be the same thing is it going to be different
[01:10:00] so Monty Carlo we're just going to we're
[01:10:03] so Monty Carlo we're just going to we're averaging over all the returns we get
[01:10:06] averaging over all the returns we get starting in that
[01:10:18] state is it like so when we start at B
[01:10:21] state is it like so when we start at B so then wouldn't it be 6 over s uh 6
[01:10:24] so then wouldn't it be 6 over s uh 6 over 8 oh 68 again yeah um but don't we
[01:10:27] over 8 oh 68 again yeah um but don't we start at a in the first
[01:10:29] start at a in the first one uh in one episode we start an A but
[01:10:31] one uh in one episode we start an A but we're just trying to compute the value
[01:10:32] we're just trying to compute the value of you right now we'll get to in a
[01:10:34] of you right now we'll get to in a second yeah think ahead yeah so the
[01:10:39] second yeah think ahead yeah so the Monte Carlo estimate so we're just
[01:10:40] Monte Carlo estimate so we're just trying to contract so just to what we're
[01:10:42] trying to contract so just to what we're trying to do here is we're trying to see
[01:10:43] trying to do here is we're trying to see will these two algorithms converge to
[01:10:45] will these two algorithms converge to the same thing or not we're going to
[01:10:47] the same thing or not we're going to start off and look at what the value of
[01:10:48] start off and look at what the value of B of State B in Monte in TD we would
[01:10:51] B of State B in Monte in TD we would converge to
[01:10:53] converge to 75 because the immediate reward
[01:10:55] 75 because the immediate reward is either one or zero and the discount
[01:10:57] is either one or zero and the discount sum of future words is zero because we
[01:11:00] sum of future words is zero because we terminate and in Monte Carlo we just
[01:11:02] terminate and in Monte Carlo we just average over all the returns we get when
[01:11:04] average over all the returns we get when we've started in B and that is also 6id
[01:11:08] we've started in B and that is also 6id 8 all right so now this the hard
[01:11:11] 8 all right so now this the hard one okay what about V of
[01:11:14] one okay what about V of a what will we converge to in these two
[01:11:18] a what will we converge to in these two cases so let's do check your
[01:11:20] cases so let's do check your understanding and you can respond in the
[01:11:22] understanding and you can respond in the poll and feel free to talk to someone
[01:11:23] poll and feel free to talk to someone next to you
[01:11:27] and again the intent of this is to think
[01:11:28] and again the intent of this is to think about are these actually Computing the
[01:11:30] about are these actually Computing the same thing or not in this
[01:11:34] same thing or not in this case and remember this is a different
[01:11:36] case and remember this is a different setting than I told you before that both
[01:11:38] setting than I told you before that both of these things can be consistent but
[01:11:40] of these things can be consistent but that was if you get infinite data what
[01:11:43] that was if you get infinite data what this is looking at is if you only have a
[01:11:44] this is looking at is if you only have a finite amount of data you just go over
[01:11:46] finite amount of data you just go over it over and over again either with Monte
[01:11:49] it over and over again either with Monte Carlo updates or with TD will you
[01:11:51] Carlo updates or with TD will you converge to the same thing
[01:11:56] and if you're not sure or you're
[01:11:57] and if you're not sure or you're confused feel free to put that in the
[01:11:58] confused feel free to put that in the poll
[01:12:24] too for
[01:13:19] okay there's lots of different answers
[01:13:21] okay there's lots of different answers here um so this is a great one to talk
[01:13:23] here um so this is a great one to talk to somebody nearby you so I you see if
[01:13:25] to somebody nearby you so I you see if you're getting the same things and can
[01:13:27] you're getting the same things and can use our collective intelligence
[01:13:57] think
[01:14:25] get old
[01:14:56] [Music]
[01:15:05] I'm hearing a lot of good discussion so
[01:15:06] I'm hearing a lot of good discussion so I'm fir to interrupt but this is kind of
[01:15:08] I'm fir to interrupt but this is kind of a fun one so why don't we um let's start
[01:15:11] a fun one so why don't we um let's start with uh
[01:15:12] with uh TD so raise uh someone want to explain
[01:15:16] TD so raise uh someone want to explain why it's 75 for TD there are multiple
[01:15:19] why it's 75 for TD there are multiple people that got
[01:15:22] that yeah would you want explain what
[01:15:24] that yeah would you want explain what you and your partner I think yeah so I
[01:15:26] you and your partner I think yeah so I think if you just like um look so uh
[01:15:29] think if you just like um look so uh we're just looking at like
[01:15:31] we're just looking at like this your name sorry yeah there's like
[01:15:35] this your name sorry yeah there's like just one episode where the a where we're
[01:15:37] just one episode where the a where we're at a um and so in that episode um the
[01:15:40] at a um and so in that episode um the immediate reward is zero but then we
[01:15:42] immediate reward is zero but then we have to do plus gamma time um the reward
[01:15:45] have to do plus gamma time um the reward of the next state B um and then V Pi V
[01:15:48] of the next state B um and then V Pi V is 75 we gotten the previous part so
[01:15:52] is 75 we gotten the previous part so Valu to 75 that's right
[01:15:55] Valu to 75 that's right so Monte Carlo is not that estimate so
[01:15:58] so Monte Carlo is not that estimate so TD gives you 75 what does Monte Carlo
[01:16:01] TD gives you 75 what does Monte Carlo give you it's not
[01:16:08] 75 and again multiple of you guys got it
[01:16:11] 75 and again multiple of you guys got it correct oh sorry I I have a question is
[01:16:13] correct oh sorry I I have a question is gamma the same as Alpha here oh good
[01:16:16] gamma the same as Alpha here oh good question um uh G no here I'm assuming
[01:16:18] question um uh G no here I'm assuming that gamma is one and it's a great
[01:16:20] that gamma is one and it's a great question so I someone else was asking
[01:16:22] question so I someone else was asking this too so um I'm assuming that Al is
[01:16:26] this too so um I'm assuming that Al is set
[01:16:28] set correctly for these to converge oh yeah
[01:16:33] correctly for these to converge oh yeah it's a good question someone else had
[01:16:34] it's a good question someone else had that too so I'm assuming that we're like
[01:16:36] that too so I'm assuming that we're like we're going over our data in infinite
[01:16:37] we're going over our data in infinite amount of time but we're decaying Alpha
[01:16:39] amount of time but we're decaying Alpha correctly as we do that a great question
[01:16:41] correctly as we do that a great question somebody want to explain what Monte
[01:16:43] somebody want to explain what Monte Carlo is it's not
[01:16:45] Carlo is it's not 75 is
[01:16:47] 75 is it zero yes it is great someone want to
[01:16:51] it zero yes it is great someone want to explain why it's zero
[01:16:55] so B to Carlo is
[01:16:59] zero yeah we've only seen one trory a
[01:17:04] zero yeah we've only seen one trory a shows up
[01:17:05] shows up and that's right remind me your name I'm
[01:17:08] and that's right remind me your name I'm yeah what said is exactly right so we've
[01:17:10] yeah what said is exactly right so we've only seen one trajectory and I know some
[01:17:12] only seen one trajectory and I know some other people made the same observation
[01:17:14] other people made the same observation so we've only seen one trajectory where
[01:17:16] so we've only seen one trajectory where there was a at
[01:17:17] there was a at all we for mon Carlo we just average
[01:17:20] all we for mon Carlo we just average over all the returns we've seen we that
[01:17:22] over all the returns we've seen we that so that's only zero so I bring that
[01:17:25] so that's only zero so I bring that because even though um ASM totically all
[01:17:28] because even though um ASM totically all of these things converge to the right
[01:17:29] of these things converge to the right thing under some mild assumptions um
[01:17:32] thing under some mild assumptions um with finite data which is what we're
[01:17:33] with finite data which is what we're almost always going to have in reality
[01:17:35] almost always going to have in reality even if you go over it multiple times
[01:17:36] even if you go over it multiple times they are converging to sometimes totally
[01:17:38] they are converging to sometimes totally different things and here is what they
[01:17:40] different things and here is what they are converging to in general Monte Carlo
[01:17:43] are converging to in general Monte Carlo is converging to the minimum mean
[01:17:45] is converging to the minimum mean squared error with respect to the
[01:17:47] squared error with respect to the observed returns so it's just going to
[01:17:49] observed returns so it's just going to set it so it minimizes the error between
[01:17:51] set it so it minimizes the error between the observe turns it's seen and um H its
[01:17:55] the observe turns it's seen and um H its value so in this case that would be B of
[01:17:58] value so in this case that would be B of a equals
[01:17:59] a equals z so that is the minimum means squ error
[01:18:02] z so that is the minimum means squ error td0 converges to the dynamic programming
[01:18:05] td0 converges to the dynamic programming policy for the mdp with a maximum
[01:18:08] policy for the mdp with a maximum likelihood model
[01:18:09] likelihood model estimates so you guys remember how we
[01:18:11] estimates so you guys remember how we just talked about certainty equivalence
[01:18:13] just talked about certainty equivalence what we were doing here is we're taking
[01:18:15] what we were doing here is we're taking all our
[01:18:16] all our data it's a it's a the answer you get
[01:18:19] data it's a it's a the answer you get from td0 if you do this batch process is
[01:18:21] from td0 if you do this batch process is the same as if you had computed your
[01:18:23] the same as if you had computed your maximum likely could markof decision
[01:18:26] maximum likely could markof decision process from the data you have and then
[01:18:28] process from the data you have and then you did dynamic programming with it okay
[01:18:31] you did dynamic programming with it okay so this that will be exactly the same as
[01:18:35] so this that will be exactly the same as this and so in particular it is
[01:18:37] this and so in particular it is leveraging and using the markof
[01:18:39] leveraging and using the markof Assumption and that's why it could
[01:18:42] Assumption and that's why it could actually chain these things together so
[01:18:44] actually chain these things together so you can see here for Monte Carlo it
[01:18:47] you can see here for Monte Carlo it doesn't doesn't know that the value of a
[01:18:51] doesn't doesn't know that the value of a has to be related to the value of B in
[01:18:53] has to be related to the value of B in terms of this boot tring
[01:18:56] terms of this boot tring relationship but TD is making That
[01:18:59] relationship but TD is making That explicit it's using the Markoff decision
[01:19:02] explicit it's using the Markoff decision process to say the value of a has to
[01:19:04] process to say the value of a has to exactly be equal to the immediate reward
[01:19:06] exactly be equal to the immediate reward you get in a plus gamma times the states
[01:19:08] you get in a plus gamma times the states that I could get into which is always B
[01:19:10] that I could get into which is always B so the value of
[01:19:11] so the value of B so TD learning is explicitly baking
[01:19:15] B so TD learning is explicitly baking that into the the solution you get
[01:19:16] that into the the solution you get whereas Monte Carlo is not Monte Carlo
[01:19:18] whereas Monte Carlo is not Monte Carlo is just trying to minimize the mean
[01:19:19] is just trying to minimize the mean squared error for the returns you see so
[01:19:22] squared error for the returns you see so they can end up giving you very
[01:19:24] they can end up giving you very different solutions and depending on
[01:19:26] different solutions and depending on whether your prop markup property is
[01:19:27] whether your prop markup property is really satisfied or not you might want
[01:19:29] really satisfied or not you might want one or the
[01:19:32] other awesome so this just summarized
[01:19:34] other awesome so this just summarized quickly sort of some of the different
[01:19:36] quickly sort of some of the different properties and approaches and just
[01:19:37] properties and approaches and just highlights here that temporal difference
[01:19:39] highlights here that temporal difference really does explo exploit this Markoff
[01:19:41] really does explo exploit this Markoff structure and that could be really
[01:19:42] structure and that could be really helpful if you want to leverage that to
[01:19:44] helpful if you want to leverage that to get better estimates of earlier States
[01:19:46] get better estimates of earlier States like in the case we just
[01:19:47] like in the case we just saw so just to summarize we finished
[01:19:50] saw so just to summarize we finished going through policy evaluation with um
[01:19:52] going through policy evaluation with um uh tabular settings and then on uh th uh
[01:19:55] uh tabular settings and then on uh th uh Wednesday what we're going to do is uh
[01:19:56] Wednesday what we're going to do is uh talk about control and we'll start to
[01:19:58] talk about control and we'll start to talk about function approximation as
[01:19:59] talk about function approximation as well all right thanks see you then

Stanford CS234 Reinforcement Learning I Q learning and Function Approximation I 2024 I Lecture 4

Source: https://www.youtube.com/watch?v=b_wvosA70f8

---

Transcript

[00:00:06] all right welcome back um we're going to
[00:00:07] all right welcome back um we're going to start lecture four in reinforcement
[00:00:09] start lecture four in reinforcement learning so we're going to be covering
[00:00:11] learning so we're going to be covering today Q learning we're going to cover
[00:00:13] today Q learning we're going to cover deep Q learning this result came out in
[00:00:16] deep Q learning this result came out in roughly 2014 um and I remember it being
[00:00:18] roughly 2014 um and I remember it being a really big deal because one of the big
[00:00:20] a really big deal because one of the big conferences neural information
[00:00:21] conferences neural information processing systems Deep Mind came and
[00:00:24] processing systems Deep Mind came and had this like amazing demonstration that
[00:00:26] had this like amazing demonstration that they were able to now have an agent that
[00:00:29] they were able to now have an agent that could learn to play video video games
[00:00:30] could learn to play video video games really well and an important thing to
[00:00:32] really well and an important thing to note here is like they're doing video
[00:00:34] note here is like they're doing video games from Pixel input so like they're
[00:00:36] games from Pixel input so like they're just getting the same input is what we
[00:00:38] just getting the same input is what we do and what the agent was learning to do
[00:00:40] do and what the agent was learning to do is to control the game um through this
[00:00:43] is to control the game um through this and through reinforcement learning and
[00:00:45] and through reinforcement learning and so we'll talk today about the algorithm
[00:00:47] so we'll talk today about the algorithm that they did to do that um and we'll
[00:00:49] that they did to do that um and we'll build up to that point and this is a
[00:00:51] build up to that point and this is a short video they show to just illustrate
[00:00:54] short video they show to just illustrate how the agent is learning through direct
[00:00:57] how the agent is learning through direct experience to try to optimize the score
[00:01:00] experience to try to optimize the score and so what it learns in this case is it
[00:01:01] and so what it learns in this case is it starts to learn particular strategies
[00:01:03] starts to learn particular strategies that allow it to do really
[00:01:05] that allow it to do really well which may or may not be the same
[00:01:07] well which may or may not be the same ones is what humans would
[00:01:09] ones is what humans would use and so it was pretty incredible this
[00:01:12] use and so it was pretty incredible this is one of the sort of most impressive
[00:01:14] is one of the sort of most impressive successes of reinforcement learning at
[00:01:16] successes of reinforcement learning at this point um particularly at trying to
[00:01:18] this point um particularly at trying to do tasks that humans can do as well um
[00:01:21] do tasks that humans can do as well um and from Pixel inputs and so we're going
[00:01:23] and from Pixel inputs and so we're going to see today sort of how that algorithm
[00:01:27] works all right so but before we do that
[00:01:30] works all right so but before we do that let's start with a quick check your
[00:01:31] let's start with a quick check your understanding um these are posted inside
[00:01:33] understanding um these are posted inside of Ed and this asks you to think about
[00:01:36] of Ed and this asks you to think about the policy Improvement stage so we're
[00:01:38] the policy Improvement stage so we're going to be talking today a lot about
[00:01:40] going to be talking today a lot about learning through direct experience um
[00:01:43] learning through direct experience um and scaling up towards function
[00:01:44] and scaling up towards function approximation with doing that but first
[00:01:46] approximation with doing that but first let's think about uh when we're doing
[00:01:49] let's think about uh when we're doing this what sort of form the policy has um
[00:01:53] this what sort of form the policy has um and then as we do this evaluation we do
[00:01:55] and then as we do this evaluation we do this repeated evaluation and policy
[00:01:57] this repeated evaluation and policy Improvement um what happens in this case
[00:02:07] these are the first two questions on the
[00:02:08] these are the first two questions on the polls
[00:02:35] sorry I just joined
[00:02:36] sorry I just joined CL it's on Ed yeah what's your name
[00:02:42] CL it's on Ed yeah what's your name thanks yeah so if anybody's not as you
[00:02:44] thanks yeah so if anybody's not as you new the class you can go to Ed you
[00:02:46] new the class you can go to Ed you should be able to get to that through
[00:02:58] canvas
[00:03:28] for all right we have good agreement on
[00:03:30] for all right we have good agreement on the first one this policy is stochastic
[00:03:33] the first one this policy is stochastic under the assumption that um for each
[00:03:35] under the assumption that um for each state there's a unique Max and it means
[00:03:37] state there's a unique Max and it means that the new policy will be
[00:03:39] that the new policy will be deterministic so almost every I think
[00:03:40] deterministic so almost every I think everybody said that correctly which is
[00:03:42] everybody said that correctly which is great um so now so this is
[00:03:46] great um so now so this is the the answer for this but there's some
[00:03:49] the the answer for this but there's some disagreement about the second one so why
[00:03:51] disagreement about the second one so why don't you turn to a neighbor and compare
[00:03:53] don't you turn to a neighbor and compare what you got for whether you can compute
[00:03:55] what you got for whether you can compute Q Pi I +1 um by using this to generate
[00:03:59] Q Pi I +1 um by using this to generate new
[00:04:24] trajectories and remember what I mean by
[00:04:26] trajectories and remember what I mean by this is I want to know whether or not
[00:04:27] this is I want to know whether or not you can get the state action value for
[00:04:30] you can get the state action value for every state and action pair under this
[00:04:32] every state and action pair under this new policy so I want to know if you can
[00:04:35] new policy so I want to know if you can compute Q of sa under this um new policy
[00:05:09] so I'll give you a hint if a policy is
[00:05:14] so I'll give you a hint if a policy is deterministic how many actions does it
[00:05:16] deterministic how many actions does it take in the same
[00:05:21] state one right so are you going to get
[00:05:24] state one right so are you going to get any data about any other actions in that
[00:05:27] any data about any other actions in that state so can we compute the Q value of
[00:05:31] state so can we compute the Q value of all actions in that state
[00:05:34] all actions in that state no that's right yeah so this is
[00:05:39] no that's right yeah so this is false we can't compute it because if we
[00:05:42] false we can't compute it because if we have a deterministic policy then we only
[00:05:45] have a deterministic policy then we only ever take Pi of s so we would only take
[00:05:49] ever take Pi of s so we would only take Pi of I + 1 of s that would be the only
[00:05:53] Pi of I + 1 of s that would be the only action we' ever take in that state
[00:05:55] action we' ever take in that state because the policy is deterministic it
[00:05:57] because the policy is deterministic it only takes that one that one action and
[00:05:59] only takes that one that one action and so that means you're just not going to
[00:06:00] so that means you're just not going to get any data about what it would be like
[00:06:02] get any data about what it would be like to take other actions in that
[00:06:05] to take other actions in that state and so that's useful to know
[00:06:07] state and so that's useful to know because it means that if we had models
[00:06:08] because it means that if we had models of the Dynamics or if we had if we um if
[00:06:11] of the Dynamics or if we had if we um if we had models of the reward and we could
[00:06:12] we had models of the reward and we could do some other things then we might be
[00:06:14] do some other things then we might be able to compute these Q values but here
[00:06:16] able to compute these Q values but here if we're going to start thinking about
[00:06:17] if we're going to start thinking about just learning this from data and from
[00:06:19] just learning this from data and from direct experience that if we have a
[00:06:21] direct experience that if we have a deterministic policy it's not going to
[00:06:23] deterministic policy it's not going to give us any data about trying different
[00:06:25] give us any data about trying different actions in the same state and so that's
[00:06:28] actions in the same state and so that's going to introduce some challenges that
[00:06:30] going to introduce some challenges that we have to tackle when we're trying to
[00:06:32] we have to tackle when we're trying to get data about the world um in order to
[00:06:34] get data about the world um in order to learn an optimal qy
[00:06:41] function great so what we're going to be
[00:06:43] function great so what we're going to be doing today then is try to think about
[00:06:45] doing today then is try to think about building on what we learned last time
[00:06:47] building on what we learned last time about policy evaluation um where we're
[00:06:49] about policy evaluation um where we're trying to learn directly from experience
[00:06:51] trying to learn directly from experience to be able to evaluate how good a
[00:06:52] to be able to evaluate how good a particular decision policy is how do we
[00:06:54] particular decision policy is how do we leverage that information to then
[00:06:56] leverage that information to then actually learn an optimal policy to
[00:06:57] actually learn an optimal policy to actually learn a good decision um you
[00:07:00] actually learn a good decision um you know a good policy uh without having to
[00:07:02] know a good policy uh without having to model of how the world works so we don't
[00:07:03] model of how the world works so we don't have access to an explicit parametric
[00:07:05] have access to an explicit parametric representation of the Dynamics model or
[00:07:07] representation of the Dynamics model or the reward model and then we're also
[00:07:09] the reward model and then we're also going to talk about uh value function
[00:07:12] going to talk about uh value function approximation and in particular we're
[00:07:13] approximation and in particular we're going to talk about Q
[00:07:16] going to talk about Q learning with deep neural
[00:07:19] learning with deep neural networks AK DQ which led to this really
[00:07:23] networks AK DQ which led to this really seminal result in like having machines
[00:07:26] seminal result in like having machines that can just play directly from Vision
[00:07:28] that can just play directly from Vision to learn how to play games like like
[00:07:29] to learn how to play games like like Atari um but I'll just pause here in
[00:07:31] Atari um but I'll just pause here in case anybody had any questions or
[00:07:33] case anybody had any questions or logistic questions before we dive into
[00:07:38] this all right and we're going to cover
[00:07:40] this all right and we're going to cover a lot today um because next week we're
[00:07:42] a lot today um because next week we're going to start policy gradient methods
[00:07:44] going to start policy gradient methods um and we're doing that because we think
[00:07:46] um and we're doing that because we think that that's a really important thing to
[00:07:48] that that's a really important thing to focus on um so but there will be quite a
[00:07:51] focus on um so but there will be quite a lot today uh and you're welcome to reach
[00:07:53] lot today uh and you're welcome to reach out I've put a bunch of worked examples
[00:07:54] out I've put a bunch of worked examples at the end in case people want to step
[00:07:56] at the end in case people want to step through some of those with Mars Rober
[00:07:57] through some of those with Mars Rober and others
[00:08:00] and others all right so these are we're going to
[00:08:01] all right so these are we're going to we're going to discuss a bunch of things
[00:08:02] we're going to discuss a bunch of things and we're going to start by thinking
[00:08:04] and we're going to start by thinking about staying in the tabular land so
[00:08:06] about staying in the tabular land so staying where we can write down the
[00:08:07] staying where we can write down the value function as a vector and then
[00:08:09] value function as a vector and then trying to learn how to make optimal
[00:08:11] trying to learn how to make optimal decisions in that
[00:08:13] decisions in that case so let's first just talk about the
[00:08:15] case so let's first just talk about the idea of generalized policy Improvement
[00:08:18] idea of generalized policy Improvement so we've s seen before this idea of
[00:08:20] so we've s seen before this idea of alternating between policy valuation and
[00:08:23] alternating between policy valuation and policy Improvement and now we're going
[00:08:25] policy Improvement and now we're going to think about that for slightly more
[00:08:26] to think about that for slightly more General cases of policies
[00:08:30] so what we just said here is that if the
[00:08:33] so what we just said here is that if the policy is deterministic we can't compute
[00:08:35] policy is deterministic we can't compute the state action value for any action
[00:08:37] the state action value for any action that's not the policy and so um what
[00:08:40] that's not the policy and so um what we'd like to be able to do now is to
[00:08:42] we'd like to be able to do now is to have kind of more coverage and to do
[00:08:44] have kind of more coverage and to do that we're going to have stochastic
[00:08:45] that we're going to have stochastic policies because if the policy is
[00:08:47] policies because if the policy is stochastic then we'll try multiple
[00:08:49] stochastic then we'll try multiple actions in the same state and we can use
[00:08:51] actions in the same state and we can use that data to estimate the Q
[00:08:53] that data to estimate the Q function so we're staying in what we're
[00:08:55] function so we're staying in what we're calling model-free policy iteration
[00:08:57] calling model-free policy iteration meaning we're not trying to explicitly
[00:08:59] meaning we're not trying to explicitly build build a Dynamics or reward model
[00:09:01] build build a Dynamics or reward model we're just trying to directly estimate a
[00:09:03] we're just trying to directly estimate a q function and once we have a Q function
[00:09:05] q function and once we have a Q function then we can extract from it an argx
[00:09:08] then we can extract from it an argx policy or something else okay and we're
[00:09:11] policy or something else okay and we're now going to be using an estimated Q
[00:09:16] because we will be
[00:09:22] estimating Q from data directly from
[00:09:26] estimating Q from data directly from experience
[00:09:30] all right so this is going to introduce
[00:09:31] all right so this is going to introduce this General challenge of exploration
[00:09:33] this General challenge of exploration which is we can only learn about the
[00:09:35] which is we can only learn about the things we try in the world this is just
[00:09:37] things we try in the world this is just like the you can't know how much better
[00:09:39] like the you can't know how much better or worse your life would be right now if
[00:09:40] or worse your life would be right now if you're drinking coffee at Koopa same
[00:09:43] you're drinking coffee at Koopa same thing like we we can only learn about
[00:09:44] thing like we we can only learn about the actions that we take um and the and
[00:09:47] the actions that we take um and the and so we need to learn about actions by
[00:09:49] so we need to learn about actions by trying them so we need to
[00:09:51] trying them so we need to explore but the downside in general is
[00:09:53] explore but the downside in general is if we try new actions we are spending
[00:09:55] if we try new actions we are spending less time using our knowledge to make
[00:09:58] less time using our knowledge to make good decisions
[00:10:00] good decisions so you might imagine that you can act
[00:10:02] so you might imagine that you can act random randomly always and that would
[00:10:04] random randomly always and that would work for like learning a lot about the
[00:10:06] work for like learning a lot about the world and learning a lot about Q
[00:10:08] world and learning a lot about Q functions but you wouldn't be finding
[00:10:10] functions but you wouldn't be finding you wouldn't be acting using that
[00:10:12] you wouldn't be acting using that knowledge to try to gain High reward so
[00:10:14] knowledge to try to gain High reward so this is known as the general challenge
[00:10:16] this is known as the general challenge between like exploration and
[00:10:18] between like exploration and exploitation um how much time do we
[00:10:21] exploitation um how much time do we spend exploring and getting new data
[00:10:23] spend exploring and getting new data about things that might be good versus
[00:10:25] about things that might be good versus how many time how much of the time do we
[00:10:27] how many time how much of the time do we exploit our knowledge of how the world
[00:10:28] exploit our knowledge of how the world works according to the data we have so
[00:10:30] works according to the data we have so far to try to make good
[00:10:32] far to try to make good decisions and this will come up a lot
[00:10:34] decisions and this will come up a lot this is there's a really deep questions
[00:10:36] this is there's a really deep questions around here about thinking of um how do
[00:10:39] around here about thinking of um how do we quantify our uncertainty in our
[00:10:40] we quantify our uncertainty in our knowledge and then how do we propagate
[00:10:42] knowledge and then how do we propagate that uncertainty into the value of that
[00:10:44] that uncertainty into the value of that uncertainty for Downstream decision-
[00:10:46] uncertainty for Downstream decision- making so we'll see a lot more about
[00:10:47] making so we'll see a lot more about that later in the course and this is
[00:10:50] that later in the course and this is continues to be a really active area of
[00:10:51] continues to be a really active area of research this is not at all solved um
[00:10:54] research this is not at all solved um but here we're just going to start to
[00:10:55] but here we're just going to start to see some simple methods to try to tackle
[00:10:57] see some simple methods to try to tackle this um challenges sort of balancing
[00:11:00] this um challenges sort of balancing between these two
[00:11:02] between these two things so one of the simplest things you
[00:11:04] things so one of the simplest things you could imagine doing is what's called
[00:11:06] could imagine doing is what's called Epsilon greedy and the idea with Epsilon
[00:11:09] Epsilon greedy and the idea with Epsilon greedy is you're going to just spend
[00:11:10] greedy is you're going to just spend some of the time doing things randomly
[00:11:12] some of the time doing things randomly and some of the times doing things the
[00:11:14] and some of the times doing things the best way you know how because um you're
[00:11:16] best way you know how because um you're kind of exploiting that knowledge so if
[00:11:18] kind of exploiting that knowledge so if we just have a finite number of actions
[00:11:20] we just have a finite number of actions because right now we're still in the
[00:11:21] because right now we're still in the tabular case so we just have a finite
[00:11:24] tabular case so we just have a finite number of states and a finite number of
[00:11:25] number of states and a finite number of actions then Epsilon greedy policy says
[00:11:29] actions then Epsilon greedy policy says um
[00:11:30] um with high probability so we have some
[00:11:32] with high probability so we have some Epsilon here epsilon's going to be less
[00:11:34] Epsilon here epsilon's going to be less than one could be like probability could
[00:11:36] than one could be like probability could be 0.1 for example so with high
[00:11:38] be 0.1 for example so with high probability you're going to take
[00:11:39] probability you're going to take whatever action maximizes your Q value
[00:11:42] whatever action maximizes your Q value in your current
[00:11:44] in your current state so you're going to kind of exploit
[00:11:46] state so you're going to kind of exploit your knowledge for whatever your state
[00:11:47] your knowledge for whatever your state action value says and you're going to do
[00:11:49] action value says and you're going to do that with probability 1 minus Epsilon
[00:11:52] that with probability 1 minus Epsilon and then otherwise you're going to take
[00:11:53] and then otherwise you're going to take an action at
[00:11:55] an action at random and so when you pick an action
[00:11:57] random and so when you pick an action Rand uniformly at random it might be one
[00:11:59] Rand uniformly at random it might be one of the same one as the argmax or it
[00:12:01] of the same one as the argmax or it might be a different one but either way
[00:12:03] might be a different one but either way the main idea is that essentially you
[00:12:04] the main idea is that essentially you spend 1 minus Epsilon percentage of the
[00:12:07] spend 1 minus Epsilon percentage of the time being greedy with respect to your
[00:12:10] time being greedy with respect to your knowledge and Epsilon per of time acting
[00:12:15] randomly so it's like you know maybe you
[00:12:18] randomly so it's like you know maybe you say like okay I'm committed to trying
[00:12:19] say like okay I'm committed to trying out new things at my restaurant so once
[00:12:21] out new things at my restaurant so once a week I will try a random dish and the
[00:12:22] a week I will try a random dish and the other six days I'll pick whatever I like
[00:12:25] other six days I'll pick whatever I like like whatever I've liked in the past and
[00:12:26] like whatever I've liked in the past and has always been good
[00:12:29] has always been good so this is a pretty simple strategy this
[00:12:31] so this is a pretty simple strategy this is not trying to have a deep notion of
[00:12:33] is not trying to have a deep notion of uncertainty or trying to quantify that
[00:12:35] uncertainty or trying to quantify that but um but neveress this can be pretty
[00:12:39] but um but neveress this can be pretty effective so in particular we can prove
[00:12:42] effective so in particular we can prove things about policy improvement with
[00:12:44] things about policy improvement with Epsilon greedy policies so what we
[00:12:46] Epsilon greedy policies so what we proved in the past is that if you do
[00:12:48] proved in the past is that if you do policy iteration when you know the
[00:12:50] policy iteration when you know the Dynamics and reward models you are
[00:12:52] Dynamics and reward models you are guaranteed to monotonically improve so
[00:12:55] guaranteed to monotonically improve so each round of policy iteration either
[00:12:57] each round of policy iteration either you would stay the same in which case
[00:12:58] you would stay the same in which case you found the optimal policy or you
[00:13:01] you found the optimal policy or you wouldn't change it and in that case um
[00:13:04] wouldn't change it and in that case um or or you would
[00:13:06] or or you would improve but when we did that proof we
[00:13:08] improve but when we did that proof we assumed policy Improvement using um a
[00:13:10] assumed policy Improvement using um a deterministic policy and it turns out
[00:13:12] deterministic policy and it turns out the same property holds with Epsilon
[00:13:14] the same property holds with Epsilon greedy
[00:13:16] greedy policies so if your policy is always
[00:13:18] policies so if your policy is always like an Epsilon greedy policy you can
[00:13:20] like an Epsilon greedy policy you can also get this kind of monotonic
[00:13:25] improvement so in particular and I'm not
[00:13:27] improvement so in particular and I'm not going to do the full proof today but
[00:13:29] going to do the full proof today but I'll leave it in just for time but um
[00:13:32] I'll leave it in just for time but um what this shows here is imagine that you
[00:13:33] what this shows here is imagine that you have a Q function like you have some
[00:13:35] have a Q function like you have some policy Pi I and you have a Q function
[00:13:37] policy Pi I and you have a Q function which tells you the state action value
[00:13:39] which tells you the state action value for that policy
[00:13:40] for that policy pii and Pi I is eg greedy which means
[00:13:44] pii and Pi I is eg greedy which means some of the time it acts greedy with
[00:13:46] some of the time it acts greedy with respect to that Q function and some of
[00:13:48] respect to that Q function and some of the time it selects an action at random
[00:13:51] the time it selects an action at random so that's what it means to be um an EG
[00:13:53] so that's what it means to be um an EG greedy policy with respect to that Q is
[00:13:56] greedy policy with respect to that Q is it it's making those decisions when it's
[00:13:57] it it's making those decisions when it's being greedy it's with respect to that Q
[00:14:00] being greedy it's with respect to that Q function so what this says is that Pi I
[00:14:02] function so what this says is that Pi I + 1 is a monotonic Improvement um so
[00:14:06] + 1 is a monotonic Improvement um so that V pii + 1 is greater than uh V Pi
[00:14:09] that V pii + 1 is greater than uh V Pi I and we can prove this here so
[00:14:12] I and we can prove this here so essentially we're trying to prove in
[00:14:13] essentially we're trying to prove in this case that the new policy that you
[00:14:15] this case that the new policy that you extract through doing policy Improvement
[00:14:18] extract through doing policy Improvement which is still an egedy policy is going
[00:14:20] which is still an egedy policy is going to be better than your old egedy
[00:14:24] to be better than your old egedy policy and the main idea is just to say
[00:14:26] policy and the main idea is just to say you can kind of also do policy
[00:14:27] you can kind of also do policy Improvement when you don't have
[00:14:29] Improvement when you don't have deterministic policies but you have
[00:14:31] deterministic policies but you have these kind of egedy policies and you
[00:14:32] these kind of egedy policies and you could still get monotonic
[00:14:36] Improvement and now I'll leave that um
[00:14:39] Improvement and now I'll leave that um I'll put that at the end uh for for
[00:14:41] I'll put that at the end uh for for later post
[00:14:42] later post proof okay so this is just to highlight
[00:14:45] proof okay so this is just to highlight like here's one thing we could do and
[00:14:46] like here's one thing we could do and we're going to see that this is actually
[00:14:47] we're going to see that this is actually going to be a pretty helpful thing to do
[00:14:49] going to be a pretty helpful thing to do this is one thing we could do um to try
[00:14:51] this is one thing we could do um to try to get data about other actions so so
[00:14:54] to get data about other actions so so we're not just taking a single action in
[00:14:55] we're not just taking a single action in a single state but we actually have some
[00:14:58] a single state but we actually have some probability of trying out multiple
[00:15:00] probability of trying out multiple actions and just to make that concrete
[00:15:02] actions and just to make that concrete if you think back to our Mars Rover
[00:15:03] if you think back to our Mars Rover example there are only seven states so
[00:15:05] example there are only seven states so if you act in it for a long time you'd
[00:15:07] if you act in it for a long time you'd repeatedly reach the same States what
[00:15:09] repeatedly reach the same States what this EG greedy policy is doing is saying
[00:15:11] this EG greedy policy is doing is saying like even when you get to the same state
[00:15:13] like even when you get to the same state you might take different actions and so
[00:15:14] you might take different actions and so over time you're going to get data that
[00:15:16] over time you're going to get data that allows you to estimate the Q value of
[00:15:18] allows you to estimate the Q value of that whole
[00:15:20] that whole policy so now we're going to see is how
[00:15:22] policy so now we're going to see is how we can use these ideas of kind of EDD
[00:15:24] we can use these ideas of kind of EDD policies to actually do control so what
[00:15:27] policies to actually do control so what I mean by that is that we're going to
[00:15:28] I mean by that is that we're going to try to learn optimal ways of acting in
[00:15:30] try to learn optimal ways of acting in the environment and we're going to start
[00:15:32] the environment and we're going to start we're going to have the same scenario as
[00:15:34] we're going to have the same scenario as last time so we're going to either have
[00:15:35] last time so we're going to either have Monte Carlo approaches where we simulate
[00:15:38] Monte Carlo approaches where we simulate in the world and then we use that to try
[00:15:40] in the world and then we use that to try to improve or temporal difference
[00:15:42] to improve or temporal difference approaches which more directly try to
[00:15:44] approaches which more directly try to use the Bellman and Markoff
[00:15:47] use the Bellman and Markoff structure okay so let's start with Monte
[00:15:50] structure okay so let's start with Monte Carlo so remember what we had before we
[00:15:53] Carlo so remember what we had before we used to have this Monte Carlo policy
[00:15:55] used to have this Monte Carlo policy evaluation algorithm where on we would
[00:15:58] evaluation algorithm where on we would repeatedly Loop we would sample the Ki
[00:16:00] repeatedly Loop we would sample the Ki episode so we just like sample a series
[00:16:03] episode so we just like sample a series of states and actions under a particular
[00:16:06] of states and actions under a particular policy okay and then you could compute
[00:16:10] policy okay and then you could compute the return from each step till the end
[00:16:13] the return from each step till the end of the
[00:16:14] of the episode and then what you would do is
[00:16:16] episode and then what you would do is you would bless you you would update for
[00:16:18] you would bless you you would update for the first time you visited a particular
[00:16:19] the first time you visited a particular State action Tuple you would update the
[00:16:22] State action Tuple you would update the Q value by a weighted average between
[00:16:24] Q value by a weighted average between your old
[00:16:25] your old estimate and then your new Target which
[00:16:29] estimate and then your new Target which was just the sum of rewards you got
[00:16:31] was just the sum of rewards you got starting in that state in action till
[00:16:33] starting in that state in action till the end of the
[00:16:34] the end of the episode okay so this is cod we often
[00:16:37] episode okay so this is cod we often call as like our
[00:16:40] Target and we were using that because um
[00:16:43] Target and we were using that because um we knew from Monte Carlo that uh a sing
[00:16:46] we knew from Monte Carlo that uh a sing that that what we want to do is really
[00:16:47] that that what we want to do is really estimate the value of starting in this
[00:16:49] estimate the value of starting in this state taking this action and following
[00:16:51] state taking this action and following this policy to the end um to the end of
[00:16:54] this policy to the end um to the end of the episode that uh we can get a sample
[00:16:56] the episode that uh we can get a sample of that by doing this and that's sort of
[00:16:59] of that by doing this and that's sort of this sample is an unbiased um
[00:17:01] this sample is an unbiased um approximation to the true expected sum
[00:17:04] approximation to the true expected sum of rewards you would get starting in
[00:17:05] of rewards you would get starting in this state in action and going till the
[00:17:07] this state in action and going till the end of the
[00:17:08] end of the episode yeah we can
[00:17:12] apply we're going to see that yes
[00:17:14] apply we're going to see that yes exactly yeah so when we thought about
[00:17:16] exactly yeah so when we thought about this before we thought the policy was
[00:17:18] this before we thought the policy was like a deterministic policy or that was
[00:17:19] like a deterministic policy or that was the easiest way to think that but now
[00:17:21] the easiest way to think that but now the policy could be stochastic and so it
[00:17:23] the policy could be stochastic and so it could be
[00:17:23] could be e yeah great question okay so now policy
[00:17:27] e yeah great question okay so now policy this policy
[00:17:29] this policy good well here we'll we'll go on to the
[00:17:31] good well here we'll we'll go on to the next
[00:17:32] next one okay so this was my Monte Carlo
[00:17:37] one okay so this was my Monte Carlo policy
[00:17:38] policy evaluation now what we could try to do
[00:17:40] evaluation now what we could try to do is Monte Carlo online control so what
[00:17:44] is Monte Carlo online control so what I'm going to do here is I'm going to
[00:17:45] I'm going to do here is I'm going to introduce a different an additional line
[00:17:47] introduce a different an additional line here at the bottom which says after I do
[00:17:50] here at the bottom which says after I do an episode I'm going to potentially
[00:17:52] an episode I'm going to potentially change my policy so you can think of
[00:17:54] change my policy so you can think of this is like my policy evaluation part
[00:17:57] this is like my policy evaluation part and this is my policy Improvement
[00:17:59] and this is my policy Improvement and again I'll just write out what this
[00:18:01] and again I'll just write out what this means so what this means is that for for
[00:18:05] means so what this means is that for for each state each
[00:18:08] each state each s um the policy for S is going to be
[00:18:12] s um the policy for S is going to be equal to ARG
[00:18:15] equal to ARG Max Q of
[00:18:20] sa with probability 1us Epsilon else
[00:18:26] sa with probability 1us Epsilon else random
[00:18:29] random so that's what I mean by I say we're
[00:18:30] so that's what I mean by I say we're doing the policy Improvement step is we
[00:18:32] doing the policy Improvement step is we take our Q function we say either you
[00:18:35] take our Q function we say either you would take the argmax action or you
[00:18:38] would take the argmax action or you would act
[00:18:39] would act randomly sorry what are we looping over
[00:18:41] randomly sorry what are we looping over in the outermost
[00:18:43] in the outermost Loop is
[00:18:44] Loop is it yeah this would be yes this would be
[00:18:47] it yeah this would be yes this would be K yeah yeah so this is just you can
[00:18:49] K yeah yeah so this is just you can think of the loop here and I'll write
[00:18:50] think of the loop here and I'll write that down
[00:18:52] that down um Loop over the
[00:18:54] um Loop over the episodes so it's like I play one game of
[00:18:57] episodes so it's like I play one game of Atari and then I update my policy
[00:18:59] Atari and then I update my policy evaluation and maybe I change my policy
[00:19:01] evaluation and maybe I change my policy then I do another round of Atari so I
[00:19:03] then I do another round of Atari so I like Play Break Out you know a million
[00:19:06] like Play Break Out you know a million times sometimes more than that in some
[00:19:08] times sometimes more than that in some of these
[00:19:09] of these cases yeah and right is it yeah it's
[00:19:12] cases yeah and right is it yeah it's still yeah I'm going to confused by this
[00:19:15] still yeah I'm going to confused by this last line that you have out there so I
[00:19:18] last line that you have out there so I mean isn't it implicit that you are
[00:19:21] mean isn't it implicit that you are using the a new you're using a new Que
[00:19:24] using the a new you're using a new Que in the on let's say you're done with
[00:19:27] in the on let's say you're done with iteration number k
[00:19:31] when you're sampling the next episode
[00:19:33] when you're sampling the next episode you're using the
[00:19:35] you're using the updated ah great question okay so maybe
[00:19:38] updated ah great question okay so maybe I should so what this says here is that
[00:19:41] I should so what this says here is that um initially you construct so your Q
[00:19:43] um initially you construct so your Q initially is zero everywhere you could
[00:19:45] initially is zero everywhere you could initialize in some ways but your Q is
[00:19:46] initialize in some ways but your Q is zero everywhere and you're going to slap
[00:19:48] zero everywhere and you're going to slap something that's egy with respect to
[00:19:49] something that's egy with respect to that now if your Q value is zero
[00:19:52] that now if your Q value is zero everywhere it means that all of your
[00:19:53] everywhere it means that all of your actions are tied you have no information
[00:19:55] actions are tied you have no information you basically are just acting randomly
[00:19:57] you basically are just acting randomly what this says is that the way we Act is
[00:19:59] what this says is that the way we Act is always with respect to our current
[00:20:01] always with respect to our current policy so the first time are you can
[00:20:04] policy so the first time are you can think of as like motor babbling right
[00:20:05] think of as like motor babbling right like your agent will just like randomly
[00:20:06] like your agent will just like randomly Press buttons it'll move over the screen
[00:20:09] Press buttons it'll move over the screen it'll do that till it wins or loses the
[00:20:10] it'll do that till it wins or loses the game and then it will update its Q value
[00:20:15] game and then it will update its Q value and what this is saying is that the next
[00:20:17] and what this is saying is that the next time you're going to change what that
[00:20:19] time you're going to change what that policy is that you're using to act so
[00:20:22] policy is that you're using to act so hopefully it won't Babble quite as much
[00:20:24] hopefully it won't Babble quite as much it's like oh well sometimes I hit
[00:20:25] it's like oh well sometimes I hit something and then I got an increase in
[00:20:26] something and then I got an increase in the points so maybe I'll try to do that
[00:20:28] the points so maybe I'll try to do that action
[00:20:36] again great question okay so I've have
[00:20:38] again great question okay so I've have not said anything about that yet I
[00:20:39] not said anything about that yet I haven't said anything about what the
[00:20:40] haven't said anything about what the properties are of this yeah in the back
[00:20:42] properties are of this yeah in the back and remember your name is it required to
[00:20:45] and remember your name is it required to do this on policy or could you do this
[00:20:46] do this on policy or could you do this off policy like collect a number of
[00:20:48] off policy like collect a number of demonstrations and then update later
[00:20:50] demonstrations and then update later great question yes you can definitely do
[00:20:51] great question yes you can definitely do off policy and we'll see that in a
[00:20:52] off policy and we'll see that in a couple slides yeah okay any questions
[00:20:55] couple slides yeah okay any questions these are great okay so you should be be
[00:20:58] these are great okay so you should be be skeptical that this is naturally going
[00:21:00] skeptical that this is naturally going to do anything reasonable but it's
[00:21:01] to do anything reasonable but it's certainly something you could run right
[00:21:03] certainly something you could run right like something that you could write down
[00:21:04] like something that you could write down in a computer so this is a process so
[00:21:07] in a computer so this is a process so then a question would be and I I put
[00:21:11] then a question would be and I I put some this is an optional worked example
[00:21:13] some this is an optional worked example you can go through it just to think
[00:21:14] you can go through it just to think about like how it would actually update
[00:21:16] about like how it would actually update these so some important properties are
[00:21:19] these so some important properties are how expensive is this does it converge
[00:21:21] how expensive is this does it converge to the optimal qar as well as what is is
[00:21:24] to the optimal qar as well as what is is it is its empirical performance
[00:21:29] it is its empirical performance let's think first whether or not we
[00:21:30] let's think first whether or not we think this is a good idea um and whether
[00:21:32] think this is a good idea um and whether or not we think that this procedure here
[00:21:35] or not we think that this procedure here is guaranteed to become a good estimate
[00:21:39] is guaranteed to become a good estimate of um the optimal qar so this is another
[00:21:43] of um the optimal qar so this is another check your understanding it's on Ed but
[00:21:46] check your understanding it's on Ed but what I would like you to think about
[00:21:49] what I would like you to think about here is that given the process I've just
[00:21:51] here is that given the process I've just shown you here do you think that the Q
[00:21:54] shown you here do you think that the Q value we're Computing is an estimate of
[00:21:57] value we're Computing is an estimate of the current policy and do you think it
[00:21:59] the current policy and do you think it will ultimately become
[00:22:02] will ultimately become qar and if you think it might or might
[00:22:05] qar and if you think it might or might not under some conditions that's fine
[00:22:06] not under some conditions that's fine too you can put that in
[00:22:08] too you can put that in there
[00:22:11] there yeah like K changes with that's right
[00:22:37] what is q k QP K is the true State
[00:22:41] what is q k QP K is the true State action value function for the py K
[00:22:45] action value function for the py K policy so what is expected discounted
[00:22:47] policy so what is expected discounted sum of rewards if you start in state s
[00:22:49] sum of rewards if you start in state s take action a and then follow p
[00:22:51] take action a and then follow p k yeah thanks for the clarification
[00:23:10] you my
[00:23:13] you my meting why
[00:23:16] you I have not said anything about
[00:23:18] you I have not said anything about whether I
[00:23:19] whether I very oh oh
[00:23:22] very oh oh um oh yeah yeah here I am yes I don't
[00:23:25] um oh yeah yeah here I am yes I don't yep why are we doing that we'll talk
[00:23:26] yep why are we doing that we'll talk about that
[00:23:30] forgot that I had already put that in
[00:23:31] forgot that I had already put that in there yeah we can talk about
[00:23:41] it okay and one thing just to note here
[00:23:44] it okay and one thing just to note here and I think this is question to so just
[00:23:46] and I think this is question to so just to be clear here as you're thinking
[00:23:48] to be clear here as you're thinking about this so this is like an
[00:23:51] about this so this is like an approximation to policy iteration so
[00:23:53] approximation to policy iteration so we're kind of doing policy evaluation
[00:23:55] we're kind of doing policy evaluation and then policy
[00:23:56] and then policy Improvement but it's it's helpful to
[00:23:59] Improvement but it's it's helpful to think about kind of how much time we're
[00:24:01] think about kind of how much time we're spending doing policy Improvement versus
[00:24:03] spending doing policy Improvement versus policy evaluation so what this is saying
[00:24:05] policy evaluation so what this is saying here is that you're going to sample one
[00:24:09] here is that you're going to sample one episode and then you're going to do
[00:24:11] episode and then you're going to do policy
[00:24:12] policy evaluation okay this is all just one
[00:24:16] evaluation okay this is all just one episode so it's like I'm going to play
[00:24:18] episode so it's like I'm going to play one like I'm going to play until I win
[00:24:20] one like I'm going to play until I win or lose at breakout once under a
[00:24:24] or lose at breakout once under a particular policy and then I'm going to
[00:24:26] particular policy and then I'm going to change my policy then I'm going to play
[00:24:28] change my policy then I'm going to play with my new policy once and I'm going to
[00:24:32] with my new policy once and I'm going to change my
[00:24:34] change my policy and some of those games might be
[00:24:36] policy and some of those games might be really long or some of them might be
[00:24:38] really long or some of them might be really short yeah so this is like just
[00:24:42] really short yeah so this is like just this is like representing
[00:24:45] this is like representing after so then I think I might just be
[00:24:51] confused when they're just like if I'm
[00:24:53] confused when they're just like if I'm playing a
[00:24:54] playing a game the epis just follow one after the
[00:24:57] game the epis just follow one after the other there just there's just one K
[00:25:00] other there just there's just one K episode there's one kith episode yeah so
[00:25:03] episode there's one kith episode yeah so like K is like like I play so if K is
[00:25:06] like K is like like I play so if K is one I'm going to play my first game and
[00:25:08] one I'm going to play my first game and I'm going to play it until I win or lose
[00:25:09] I'm going to play it until I win or lose until the you know the game so maybe
[00:25:11] until the you know the game so maybe breakout finishes or maybe I'm playing
[00:25:13] breakout finishes or maybe I'm playing Tetris and like I fail and I died and
[00:25:15] Tetris and like I fail and I died and that and that is one episode and I'm
[00:25:17] that and that is one episode and I'm going to use that to then update my Q
[00:25:19] going to use that to then update my Q function then I'm going to change it and
[00:25:20] function then I'm going to change it and say okay my next round I'm going to play
[00:25:22] say okay my next round I'm going to play differently then I play Tetris again
[00:25:24] differently then I play Tetris again until I fail and then um I see what the
[00:25:27] until I fail and then um I see what the total point points are I update my Q
[00:25:29] total point points are I update my Q function and I repeat and some of those
[00:25:31] function and I repeat and some of those episodes might be really short so maybe
[00:25:33] episodes might be really short so maybe the first time particularly for these
[00:25:34] the first time particularly for these agents the first time they play Tetris
[00:25:36] agents the first time they play Tetris maybe they lose in like 10 steps might
[00:25:39] maybe they lose in like 10 steps might be a really short step um later maybe
[00:25:41] be a really short step um later maybe they play for a long time but in general
[00:25:43] they play for a long time but in general I've not told you anything about how
[00:25:45] I've not told you anything about how long these episodes are they might be
[00:25:47] long these episodes are they might be really short but they might be really
[00:25:51] long
[00:25:53] long okay and I think one useful way I find
[00:25:56] okay and I think one useful way I find to think about this is that think about
[00:25:58] to think about this is that think about a they're really short like really
[00:26:00] a they're really short like really really short like I take two steps and I
[00:26:01] really short like I take two steps and I just fail I did something really dumb so
[00:26:04] just fail I did something really dumb so in that
[00:26:05] in that case think about whether Q would be a
[00:26:08] case think about whether Q would be a good estimate of Q Pi
[00:26:11] good estimate of Q Pi K like would it be good if you've only
[00:26:13] K like would it be good if you've only seen two states or would it be pretty
[00:26:20] bad so why you turn to someone near you
[00:26:22] bad so why you turn to someone near you I think most people have voted if you
[00:26:23] I think most people have voted if you are have written something you haven't
[00:26:25] are have written something you haven't but why don't you check and see what you
[00:26:27] but why don't you check and see what you think
[00:26:57] for
[00:27:27] e e
[00:28:05] one
[00:28:37] time
[00:28:58] so
[00:29:54] don't C
[00:30:10] okay awesome I'm hearing a lot of really
[00:30:11] okay awesome I'm hearing a lot of really good discussion but I'm going to
[00:30:12] good discussion but I'm going to interrupt you because I want to make
[00:30:13] interrupt you because I want to make sure we get to dqn um so this is where
[00:30:17] sure we get to dqn um so this is where so one of the reasons that I bring up
[00:30:19] so one of the reasons that I bring up this particular example is that you know
[00:30:21] this particular example is that you know here it's tabular things are a little
[00:30:23] here it's tabular things are a little bit smaller so it's a bit easier to see
[00:30:25] bit smaller so it's a bit easier to see but essentially what I kind of want you
[00:30:26] but essentially what I kind of want you guys to get out of today is that it
[00:30:28] guys to get out of today is that it should be sort of shocking that
[00:30:30] should be sort of shocking that reinforcement learning works um and
[00:30:32] reinforcement learning works um and we're not going to have time to go
[00:30:33] we're not going to have time to go through all the Deep mathematical
[00:30:35] through all the Deep mathematical reasons for why it does work sometimes
[00:30:36] reasons for why it does work sometimes um in this class but I'm happy to give
[00:30:38] um in this class but I'm happy to give people pointers but so there's several
[00:30:40] people pointers but so there's several things that are really kind of odd if
[00:30:42] things that are really kind of odd if you start to think about this when when
[00:30:43] you start to think about this when when you go through this so first of all Q is
[00:30:45] you go through this so first of all Q is not an estimate of Q Pi K it is not
[00:30:49] not an estimate of Q Pi K it is not because it is averaging over policies
[00:30:51] because it is averaging over policies that are changing every episode or
[00:30:54] that are changing every episode or potentially changing every episode right
[00:30:56] potentially changing every episode right because um in in fact in general it will
[00:30:59] because um in in fact in general it will be right because we're decaying Epsilon
[00:31:01] be right because we're decaying Epsilon so we're changing Epsilon each round
[00:31:03] so we're changing Epsilon each round which means we're making things more and
[00:31:04] which means we're making things more and more deterministic but in addition to
[00:31:06] more deterministic but in addition to that our our Q might be changing so
[00:31:08] that our our Q might be changing so essentially like I'm just trying a
[00:31:10] essentially like I'm just trying a policy one round um and then I update my
[00:31:14] policy one round um and then I update my queue and then I try something again and
[00:31:17] queue and then I try something again and it's sort of like you know extreme
[00:31:18] it's sort of like you know extreme example of this would be like flipping a
[00:31:19] example of this would be like flipping a coin once and deciding whether what it's
[00:31:21] coin once and deciding whether what it's bias is or something like that like
[00:31:23] bias is or something like that like that's just not very much data to do
[00:31:25] that's just not very much data to do this evaluation and you're also you're
[00:31:27] this evaluation and you're also you're averaging this over
[00:31:28] averaging this over many many different policies so Q is not
[00:31:31] many many different policies so Q is not an estimate of Q Pi K it's this weird
[00:31:34] an estimate of Q Pi K it's this weird weighted average of all the previous
[00:31:36] weighted average of all the previous data and all the policies you you've
[00:31:37] data and all the policies you you've done
[00:31:40] before like is the last latest k k minus
[00:31:45] before like is the last latest k k minus one well but not really right because
[00:31:47] one well but not really right because it's I mean you've averaged in that part
[00:31:50] it's I mean you've averaged in that part that part is from PI K plus one but this
[00:31:52] that part is from PI K plus one but this old thing was over all of your is like
[00:31:53] old thing was over all of your is like this weird weighted average of all the
[00:31:55] this weird weighted average of all the other policies you've tried so yes it is
[00:31:58] other policies you've tried so yes it is that but also like that plus all the
[00:32:00] that but also like that plus all the other policies so it's this weird thing
[00:32:03] other policies so it's this weird thing right um the second thing is that we're
[00:32:06] right um the second thing is that we're only doing and I was talking to some
[00:32:07] only doing and I was talking to some people about this we're only doing one
[00:32:10] people about this we're only doing one roll out to try to evaluate a policy and
[00:32:13] roll out to try to evaluate a policy and you might imagine there's a lot of
[00:32:14] you might imagine there's a lot of stochasticity like even in something
[00:32:16] stochasticity like even in something like you know um some games there's like
[00:32:18] like you know um some games there's like random roles of the dice and stuff like
[00:32:19] random roles of the dice and stuff like that which means even with the same
[00:32:21] that which means even with the same strategy you might get different
[00:32:22] strategy you might get different outcomes each time um so it' be like if
[00:32:24] outcomes each time um so it' be like if you know you drove to SF and you did it
[00:32:26] you know you drove to SF and you did it once and there was no traffic and so
[00:32:28] once and there was no traffic and so you're like oh I can always get to SF in
[00:32:30] you're like oh I can always get to SF in like you know I don't know 20 minutes on
[00:32:32] like you know I don't know 20 minutes on the highway but for those of you that
[00:32:34] the highway but for those of you that tried to SF you would know that often
[00:32:35] tried to SF you would know that often there's lots of traffic and so you would
[00:32:37] there's lots of traffic and so you would need to average over many rounds of
[00:32:39] need to average over many rounds of doing this to see how good a particular
[00:32:41] doing this to see how good a particular rout is so the weird thing here is that
[00:32:44] rout is so the weird thing here is that we're just doing like kind of one roll
[00:32:46] we're just doing like kind of one roll out we're averaging into this weird Q
[00:32:48] out we're averaging into this weird Q thing which is now going to be this
[00:32:50] thing which is now going to be this weighted average over all the policies
[00:32:51] weighted average over all the policies we've done and we have this weird
[00:32:54] we've done and we have this weird Epsilon
[00:32:55] Epsilon thing and it is should not be clear yet
[00:32:58] thing and it is should not be clear yet that we will necessarily converge to qar
[00:33:01] that we will necessarily converge to qar like we are getting more and more
[00:33:02] like we are getting more and more deterministic over time because we're
[00:33:04] deterministic over time because we're reducing Epsilon so reducing Epsilon
[00:33:07] reducing Epsilon so reducing Epsilon here towards zero eventually we're going
[00:33:08] here towards zero eventually we're going to converge towards something
[00:33:11] to converge towards something deterministic but you may or may not be
[00:33:13] deterministic but you may or may not be convinced yet that the thing we're going
[00:33:14] convinced yet that the thing we're going to converge to is actually
[00:33:18] qar so
[00:33:20] qar so fortunately there are some sufficient
[00:33:23] fortunately there are some sufficient conditions under which we can guarantee
[00:33:25] conditions under which we can guarantee that this sort of thing will converge to
[00:33:26] that this sort of thing will converge to qar and it's really it's quite beautiful
[00:33:30] qar and it's really it's quite beautiful that this works okay so one is what's
[00:33:32] that this works okay so one is what's called greedy in the limit of infinite
[00:33:34] called greedy in the limit of infinite exploration or
[00:33:36] exploration or Glee so the idea in this case is that if
[00:33:39] Glee so the idea in this case is that if you can ensure that all state action
[00:33:41] you can ensure that all state action pairs are visited an infinite number of
[00:33:43] pairs are visited an infinite number of times meaning the number of counts that
[00:33:46] times meaning the number of counts that you have for a particular State and
[00:33:48] you have for a particular State and action pair goes to Infinity for all
[00:33:50] action pair goes to Infinity for all states and
[00:33:52] states and actions this is for all
[00:33:59] and the behavior policy and what I mean
[00:34:01] and the behavior policy and what I mean by the behavior policy is is this is the
[00:34:03] by the behavior policy is is this is the policy you're actually using to make
[00:34:04] policy you're actually using to make decisions in the world um and it will be
[00:34:07] decisions in the world um and it will be important there'll be distinctions
[00:34:08] important there'll be distinctions between this and other policies soon
[00:34:10] between this and other policies soon which is why we call this Behavior
[00:34:12] which is why we call this Behavior policy if the behavior so if you s Paul
[00:34:15] policy if the behavior so if you s Paul State action pairs an infinite number of
[00:34:16] State action pairs an infinite number of times and your behavior policy converges
[00:34:19] times and your behavior policy converges to the greedy policy which means that
[00:34:21] to the greedy policy which means that ASM totically the action you select in a
[00:34:24] ASM totically the action you select in a state is exactly equal to the ARG Max of
[00:34:27] state is exactly equal to the ARG Max of your Q function
[00:34:28] your Q function with probability one so you're just
[00:34:30] with probability one so you're just getting more and more deterministic so
[00:34:32] getting more and more deterministic so then then you were being greedy in the
[00:34:34] then then you were being greedy in the limit of infinite exploration that says
[00:34:37] limit of infinite exploration that says that you're exploring everything an
[00:34:40] that you're exploring everything an infinite number of times you're always
[00:34:41] infinite number of times you're always continuing to try all actions in all
[00:34:43] continuing to try all actions in all states but you're getting more and more
[00:34:45] states but you're getting more and more deterministic so this is what it means
[00:34:47] deterministic so this is what it means to be
[00:34:49] to be Glee if you have a Glee
[00:34:53] Glee if you have a Glee algorithm um and I'll just note here
[00:34:55] algorithm um and I'll just note here like a simple way to do this is to do EG
[00:34:58] like a simple way to do this is to do EG greedy where Epsilon Is Res reduced to
[00:35:01] greedy where Epsilon Is Res reduced to zero at the following
[00:35:06] rate so we'd have this so that's sort of
[00:35:09] rate so we'd have this so that's sort of a simple one you know
[00:35:11] a simple one you know and
[00:35:20] visit and that should hold as as long as
[00:35:22] visit and that should hold as as long as you have an egy strategy then you will
[00:35:25] you have an egy strategy then you will be able to visit all states and actions
[00:35:27] be able to visit all states and actions so so you're you're going to be visiting
[00:35:28] so so you're you're going to be visiting all states and actions under this GLE
[00:35:32] all states and actions under this GLE strategy then under that the Monte Carlo
[00:35:34] strategy then under that the Monte Carlo algorithm I just showed you four tabular
[00:35:37] algorithm I just showed you four tabular representations will converge to
[00:35:40] representations will converge to qstar which means as long as you Decay
[00:35:43] qstar which means as long as you Decay Epsilon at this rate you are actually
[00:35:45] Epsilon at this rate you are actually converging to qar you're getting more
[00:35:46] converging to qar you're getting more and more deterministic you're still
[00:35:47] and more deterministic you're still visiting all states and actions an
[00:35:49] visiting all states and actions an infinite number of times and this
[00:35:51] infinite number of times and this procedure is guaranteed to ASM totically
[00:35:53] procedure is guaranteed to ASM totically get you to the optimal Q function which
[00:35:55] get you to the optimal Q function which is pretty cool and it should be somewhat
[00:35:56] is pretty cool and it should be somewhat surprising
[00:36:00] all right so that is Glee and that is
[00:36:02] all right so that is Glee and that is one one of the reasons why we like to
[00:36:04] one one of the reasons why we like to think about EG greedy algorithms because
[00:36:06] think about EG greedy algorithms because they have this nice property that we can
[00:36:08] they have this nice property that we can prove that we are going to get an
[00:36:09] prove that we are going to get an optimal policy even though all we're
[00:36:11] optimal policy even though all we're doing is we're acting in the world and
[00:36:13] doing is we're acting in the world and we're getting this
[00:36:14] we're getting this data now what you should be thinking
[00:36:16] data now what you should be thinking about at this point is that all right
[00:36:17] about at this point is that all right here's the Monte Carlo approach to doing
[00:36:19] here's the Monte Carlo approach to doing this there's probably going to be a
[00:36:20] this there's probably going to be a temporal difference uh approach to doing
[00:36:22] temporal difference uh approach to doing this and that's what we're going to see
[00:36:23] this and that's what we're going to see now so now we're going to look into
[00:36:26] now so now we're going to look into temporal difference methods for
[00:36:29] temporal difference methods for control okay so one of the interesting
[00:36:32] control okay so one of the interesting things is that um there's going to be
[00:36:35] things is that um there's going to be two different types of algorithms that
[00:36:37] two different types of algorithms that we're going to focus on for temporal
[00:36:39] we're going to focus on for temporal difference for
[00:36:40] difference for control and the idea in these settings
[00:36:43] control and the idea in these settings is that we're going to alternate between
[00:36:44] is that we're going to alternate between two steps again kind of this policy
[00:36:46] two steps again kind of this policy evaluation versus policy Improvement and
[00:36:49] evaluation versus policy Improvement and one of the key things to think about in
[00:36:50] one of the key things to think about in this case is sort of how much time are
[00:36:52] this case is sort of how much time are we spending doing evaluation versus
[00:36:53] we spending doing evaluation versus Improvement and what are we trying to
[00:36:55] Improvement and what are we trying to evaluate and what are we improving with
[00:36:57] evaluate and what are we improving with respect to
[00:36:58] respect to so the idea now is that we're going to
[00:37:00] so the idea now is that we're going to compute qy using temporal difference
[00:37:02] compute qy using temporal difference updating with an EG greedy policy and
[00:37:05] updating with an EG greedy policy and then we're going to do policy
[00:37:06] then we're going to do policy Improvement in the same way that we saw
[00:37:08] Improvement in the same way that we saw before for Monti Carlo methods so we can
[00:37:11] before for Monti Carlo methods so we can do this EG greedy thing where we are
[00:37:13] do this EG greedy thing where we are greedy with respect to our current Q
[00:37:15] greedy with respect to our current Q value and the first algorithm we're
[00:37:17] value and the first algorithm we're going to see is called
[00:37:18] going to see is called sarsa and the reason it is called sarsa
[00:37:21] sarsa and the reason it is called sarsa is it is
[00:37:23] is it is State
[00:37:25] State action reward
[00:37:28] action reward next
[00:37:29] next state next action is short for
[00:37:33] state next action is short for that s r s sarsa that's an easy way to
[00:37:38] that s r s sarsa that's an easy way to think to to remember why this method
[00:37:40] think to to remember why this method would be called Sara because those are
[00:37:43] would be called Sara because those are the um tles we need in order to do
[00:37:45] the um tles we need in order to do updates we need s a r s prime a prime to
[00:37:48] updates we need s a r s prime a prime to do an
[00:37:49] do an update and this is going to be an on
[00:37:51] update and this is going to be an on policy algorithm and this is related to
[00:37:54] policy algorithm and this is related to um what was suggested in the back remind
[00:37:55] um what was suggested in the back remind me your name
[00:37:58] me your name yeah exactly what said so can we also
[00:38:00] yeah exactly what said so can we also use sort of off policy data and we'll
[00:38:02] use sort of off policy data and we'll see that really shortly but um Sara is
[00:38:05] see that really shortly but um Sara is going to be on policy and what we mean
[00:38:06] going to be on policy and what we mean by that is that it's going to be
[00:38:08] by that is that it's going to be Computing an estimate of the Q value of
[00:38:10] Computing an estimate of the Q value of the policy we're using to act or the
[00:38:12] the policy we're using to act or the what policy we're using to make
[00:38:14] what policy we're using to make decisions in the
[00:38:15] decisions in the world so let's see how it works so in
[00:38:18] world so let's see how it works so in general the the form of Sara is the
[00:38:21] general the the form of Sara is the following um we are going to iterate our
[00:38:25] following um we are going to iterate our Loop is going to be such that
[00:38:28] Loop is going to be such that we start off so this is the a the we
[00:38:30] we start off so this is the a the we start in some State this is the S we
[00:38:32] start in some State this is the S we take an action a um we observe reward in
[00:38:35] take an action a um we observe reward in the next state and then we Loop and we
[00:38:39] the next state and then we Loop and we take the next action still according to
[00:38:41] take the next action still according to the same
[00:38:43] the same policy and then what we're going to do
[00:38:45] policy and then what we're going to do is we're going to update our Q function
[00:38:48] is we're going to update our Q function given this topple of sorsa essentially
[00:38:51] given this topple of sorsa essentially and what we're going to do in this case
[00:38:53] and what we're going to do in this case is it's going to look similar to what we
[00:38:55] is it's going to look similar to what we saw before so we're going to have
[00:38:58] saw before so we're going to have our updated one is our old
[00:39:04] our updated one is our old Value Plus Alpha so this is like our
[00:39:07] Value Plus Alpha so this is like our learning rate our
[00:39:10] learning rate our Target St + 1 A+
[00:39:14] Target St + 1 A+ one minus Q of
[00:39:20] st8 so this is the
[00:39:26] Target and it's it's going to look
[00:39:27] Target and it's it's going to look similar to what we saw for td0 where we
[00:39:30] similar to what we saw for td0 where we plug in our immediate reward plus our
[00:39:32] plug in our immediate reward plus our estimate of the expected discounted sum
[00:39:34] estimate of the expected discounted sum of rewards starting in that next state
[00:39:37] of rewards starting in that next state and one of the important things to
[00:39:38] and one of the important things to notice in this case is we are plugging
[00:39:42] notice in this case is we are plugging in the actual action we took in the next
[00:39:46] in the actual action we took in the next state so we're saying what is the
[00:39:48] state so we're saying what is the expected discounted sum of reward
[00:39:49] expected discounted sum of reward starting in this state and taking this
[00:39:51] starting in this state and taking this action well one one uh estimate of it is
[00:39:54] action well one one uh estimate of it is the immediate reward I got plus gamma
[00:39:56] the immediate reward I got plus gamma times the Q value for the state I
[00:39:58] times the Q value for the state I reached plus the action I would take
[00:40:00] reached plus the action I would take under this policy
[00:40:01] under this policy next and that's one of the reasons why
[00:40:03] next and that's one of the reasons why it's called on policy because it's being
[00:40:05] it's called on policy because it's being specific to the action you would
[00:40:07] specific to the action you would actually take under this
[00:40:10] actually take under this policy all right and then after the the
[00:40:12] policy all right and then after the the then the next thing we do is we do
[00:40:14] then the next thing we do is we do policy Improvement and what we would do
[00:40:16] policy Improvement and what we would do in this case is again similar to what we
[00:40:18] in this case is again similar to what we saw in the other one so for all s this
[00:40:20] saw in the other one so for all s this just means for all for anybody who's
[00:40:22] just means for all for anybody who's hasn't seen this notation Pi of s is
[00:40:25] hasn't seen this notation Pi of s is equal to R Max over
[00:40:28] equal to R Max over a q of
[00:40:31] a q of sa with probability almost
[00:40:45] abs and then what we do is we update our
[00:40:48] abs and then what we do is we update our time step we update our
[00:40:51] time step we update our Epsilon and then what we're going to do
[00:40:53] Epsilon and then what we're going to do is just repeat so then we're going to um
[00:40:56] is just repeat so then we're going to um go then we're going to go to the next
[00:40:58] go then we're going to go to the next we're going to take our next state take
[00:41:00] we're going to take our next state take an action and repeat this
[00:41:02] an action and repeat this updating so this is called Gap um quick
[00:41:06] updating so this is called Gap um quick question uh like do we I I have bit
[00:41:09] question uh like do we I I have bit confused about setting Pi of s do we say
[00:41:13] confused about setting Pi of s do we say Pi is a deterministic policy that is one
[00:41:16] Pi is a deterministic policy that is one of this with this probability and the
[00:41:18] of this with this probability and the other one with the other probability or
[00:41:19] other one with the other probability or are we saying it's a stochastic policy
[00:41:22] are we saying it's a stochastic policy that can it's a stochastic policy yeah
[00:41:25] that can it's a stochastic policy yeah so it's a um it's a stochastic policy at
[00:41:27] so it's a um it's a stochastic policy at the very beginning it's totally random
[00:41:29] the very beginning it's totally random you just take any action in any state
[00:41:31] you just take any action in any state later you're defining it with respect to
[00:41:32] later you're defining it with respect to your current Q value and you're either
[00:41:34] your current Q value and you're either Being Greedy with respect to that Q
[00:41:36] Being Greedy with respect to that Q value selecting action at random yeah so
[00:41:39] value selecting action at random yeah so um one concern that I had was like What
[00:41:41] um one concern that I had was like What if we reach a Dom State
[00:41:44] if we reach a Dom State and ah good question okay and um this
[00:41:46] and ah good question okay and um this actually came up in another conversation
[00:41:48] actually came up in another conversation earlier this morning yes so if it is if
[00:41:51] earlier this morning yes so if it is if you reach a terminal State then you just
[00:41:52] you reach a terminal State then you just reset so if t t + 2 is
[00:42:01] terminal
[00:42:03] terminal reset
[00:42:05] reset episode and
[00:42:08] episode and S so if you ever reach a state where
[00:42:10] S so if you ever reach a state where it's Turnal what would happen next is
[00:42:12] it's Turnal what would happen next is then your whole whole episode just
[00:42:14] then your whole whole episode just resets you sample as initial state from
[00:42:16] resets you sample as initial state from the world and then you repeat so just
[00:42:17] the world and then you repeat so just like if I like finish my game I failed
[00:42:19] like if I like finish my game I failed at Tesla it reinitializes the world so
[00:42:22] at Tesla it reinitializes the world so these are still sort of assumed to be
[00:42:23] these are still sort of assumed to be continuing processes
[00:42:25] continuing processes yeah um I'm wondering
[00:42:31] what great question so um and what we're
[00:42:35] what great question so um and what we're going to see in just like a slide or two
[00:42:36] going to see in just like a slide or two and you guys are probably half of you at
[00:42:38] and you guys are probably half of you at least has probably seen this before
[00:42:39] least has probably seen this before we're going to see Q learning and that's
[00:42:41] we're going to see Q learning and that's where it's going to be off
[00:42:42] where it's going to be off policy okay very quick question uh like
[00:42:46] policy okay very quick question uh like the uh when it's s t plus2 uh do we do
[00:42:49] the uh when it's s t plus2 uh do we do the step seven or do we skip it for one
[00:42:51] the step seven or do we skip it for one step and do it in the next one cuz like
[00:42:54] step and do it in the next one cuz like if it's terminal or in general like if
[00:42:56] if it's terminal or in general like if it's terminal if it's terminal you would
[00:42:59] it's terminal if it's terminal you would halt here and then you would reset the
[00:43:01] halt here and then you would reset the whole thing then you would need to take
[00:43:03] whole thing then you would need to take an action observe or or next state and
[00:43:05] an action observe or or next state and then jump into five so You' sort of have
[00:43:06] then jump into five so You' sort of have to reset to
[00:43:07] to reset to two great
[00:43:09] two great question all right so let's see um well
[00:43:12] question all right so let's see um well first let's talk about whether this is
[00:43:13] first let's talk about whether this is guaranteed to do anything reasonable um
[00:43:15] guaranteed to do anything reasonable um and then we'll going so I've written
[00:43:18] and then we'll going so I've written this out neatly here and then there's a
[00:43:20] this out neatly here and then there's a worked example for the Mars Rob at the
[00:43:22] worked example for the Mars Rob at the end of the
[00:43:23] end of the slides okay so one thing to note here to
[00:43:27] slides okay so one thing to note here to is that now we've defined a general
[00:43:28] is that now we've defined a general learning rate so that we have a general
[00:43:30] learning rate so that we have a general learning rate here okay and we also have
[00:43:34] learning rate here okay and we also have make sure I keep this in
[00:43:37] here we're going to keep updating our
[00:43:40] here we're going to keep updating our Epsilon okay so is this a good
[00:43:45] Epsilon okay so is this a good approach so we can think of a couple
[00:43:47] approach so we can think of a couple different things here we can think of
[00:43:48] different things here we can think of the computational complexity so here
[00:43:51] the computational complexity so here after each tle we're doing an
[00:43:53] after each tle we're doing an update and in fact we know that that's
[00:43:56] update and in fact we know that that's in general only going to um change the Q
[00:43:58] in general only going to um change the Q value for the states and the actions
[00:44:00] value for the states and the actions that we're updating so we just deserve
[00:44:01] that we're updating so we just deserve doing that that small update each time
[00:44:04] doing that that small update each time we don't have to sum over all the states
[00:44:06] we don't have to sum over all the states so there's nothing that depends on the
[00:44:08] so there's nothing that depends on the state space siid per update but of
[00:44:10] state space siid per update but of course we're doing this many many many
[00:44:12] course we're doing this many many many times does this converge to the optimal
[00:44:14] times does this converge to the optimal Q function so what we have here in this
[00:44:17] Q function so what we have here in this case is we have this weighted um
[00:44:19] case is we have this weighted um combination between our last Q function
[00:44:21] combination between our last Q function and like this new
[00:44:23] and like this new Target and again Q is an estimate of the
[00:44:27] Target and again Q is an estimate of the performance of a policy that might be
[00:44:28] performance of a policy that might be changing each time
[00:44:30] changing each time point so it's similar to Monte Carlo
[00:44:32] point so it's similar to Monte Carlo like we're just like we're constantly
[00:44:33] like we're just like we're constantly changing the policy in this case and so
[00:44:36] changing the policy in this case and so that should feel a little bit
[00:44:37] that should feel a little bit concerning and empirically it often does
[00:44:39] concerning and empirically it often does quite well but Q learning is more
[00:44:42] quite well but Q learning is more popular so what are the convergence
[00:44:44] popular so what are the convergence properties so it turns out that in terms
[00:44:46] properties so it turns out that in terms of some of the mathematical formulations
[00:44:48] of some of the mathematical formulations this relates really strongly to
[00:44:49] this relates really strongly to stochastic
[00:44:51] stochastic approximation and this is a deep
[00:44:53] approximation and this is a deep literature with lots of really amazing
[00:44:54] literature with lots of really amazing results kind of the the in the finite
[00:44:57] results kind of the the in the finite State and finite action case it's going
[00:44:59] State and finite action case it's going to converge to the optimal qar for sarsa
[00:45:03] to converge to the optimal qar for sarsa if your policy sequence satisfies the
[00:45:06] if your policy sequence satisfies the condition of Glee so we're going to
[00:45:07] condition of Glee so we're going to visit all states and actions an infinite
[00:45:08] visit all states and actions an infinite number of times and we're getting
[00:45:10] number of times and we're getting greedier and greedier over time and we
[00:45:13] greedier and greedier over time and we have to put in a condition about the
[00:45:14] have to put in a condition about the learning rates the step sizes so in
[00:45:18] learning rates the step sizes so in particular they have to satisfy the
[00:45:19] particular they have to satisfy the Ruben's Monroe
[00:45:21] Ruben's Monroe sequence so they have to satisfy these
[00:45:23] sequence so they have to satisfy these two things which is their sum goes to
[00:45:25] two things which is their sum goes to infinity and their squared this an
[00:45:27] infinity and their squared this an infinity and we've seen this
[00:45:30] infinity and we've seen this before and an example of this would be a
[00:45:32] before and an example of this would be a = 1/t satisfies these
[00:45:37] conditions
[00:45:40] conditions so these results really sort of rely on
[00:45:43] so these results really sort of rely on these really nice results from
[00:45:44] these really nice results from stochastic approximation because it
[00:45:46] stochastic approximation because it should be a little bit surprising you
[00:45:47] should be a little bit surprising you can think of this as like kind of
[00:45:48] can think of this as like kind of there's these different there's these
[00:45:51] there's these different there's these mixing processes that are going on
[00:45:52] mixing processes that are going on because our policy is changing our
[00:45:54] because our policy is changing our estimates are changing how can we be
[00:45:56] estimates are changing how can we be sure that it's is essentially going to
[00:45:57] sure that it's is essentially going to be stable enough that over time we're
[00:45:59] be stable enough that over time we're actually going to converge to something
[00:46:01] actually going to converge to something that's both fixed like we're not just
[00:46:03] that's both fixed like we're not just going to oscillate forever and that it
[00:46:06] going to oscillate forever and that it is optimal so it should not be at all
[00:46:09] is optimal so it should not be at all clear why this would necessarily work um
[00:46:12] clear why this would necessarily work um and uh the this is where we sort of rely
[00:46:15] and uh the this is where we sort of rely on those results from stochastic
[00:46:17] on those results from stochastic approximation that also had to be
[00:46:18] approximation that also had to be extended to think about these particular
[00:46:20] extended to think about these particular cases during a number of really
[00:46:22] cases during a number of really beautiful papers from the
[00:46:24] beautiful papers from the 1990s so there's those the 1992 and 1994
[00:46:31] 1990s so there's those the 1992 and 1994 papers that show
[00:46:34] papers that show this so some really cool results that
[00:46:36] this so some really cool results that illustrate why this is possible okay so
[00:46:40] illustrate why this is possible okay so sarsa for tabular settings um under some
[00:46:43] sarsa for tabular settings um under some mild conditions is guaranteed to
[00:46:44] mild conditions is guaranteed to converge to qar so now let's see if we
[00:46:47] converge to qar so now let's see if we can do off policy learning so off policy
[00:46:50] can do off policy learning so off policy learning is the idea that now we're
[00:46:51] learning is the idea that now we're going to be trying to estimate and
[00:46:53] going to be trying to estimate and evaluate a policy using experience
[00:46:55] evaluate a policy using experience gathered from following a different
[00:46:56] gathered from following a different policy
[00:46:58] policy so so far we've been thinking about like
[00:46:59] so so far we've been thinking about like Monte Carlo methods and starsa where
[00:47:01] Monte Carlo methods and starsa where we're at least sort of kind of trying to
[00:47:03] we're at least sort of kind of trying to always approximate the value of the most
[00:47:05] always approximate the value of the most recent policy or averaged over all those
[00:47:07] recent policy or averaged over all those policies but now we're going to
[00:47:09] policies but now we're going to explicitly be trying to estimate qar at
[00:47:12] explicitly be trying to estimate qar at all time
[00:47:13] all time points okay so in Q
[00:47:17] points okay so in Q learning we are going to try to directly
[00:47:20] learning we are going to try to directly estimate the value of pi star which
[00:47:22] estimate the value of pi star which remember we don't know because if we
[00:47:23] remember we don't know because if we knew what pi star was then we wouldn't
[00:47:25] knew what pi star was then we wouldn't have to do any of this learning with
[00:47:27] have to do any of this learning with another Behavior policy pi b so we're
[00:47:29] another Behavior policy pi b so we're going to be acting in one way and we're
[00:47:31] going to be acting in one way and we're going to be trying to use that data to
[00:47:33] going to be trying to use that data to estimate the value of an alternative
[00:47:35] estimate the value of an alternative policy and that's what how what Q
[00:47:37] policy and that's what how what Q learning
[00:47:38] learning does so in Q learning the key difference
[00:47:42] does so in Q learning the key difference is that instead of trying to think about
[00:47:44] is that instead of trying to think about what is the action we actually took on
[00:47:46] what is the action we actually took on the next time step we're just going to
[00:47:48] the next time step we're just going to figure out what is the best action I
[00:47:50] figure out what is the best action I could have taken because we know for the
[00:47:53] could have taken because we know for the qstar value it is the estimate of the
[00:47:56] qstar value it is the estimate of the optimal
[00:47:57] optimal expected reward you could get if you
[00:47:59] expected reward you could get if you take the the current action and then act
[00:48:01] take the the current action and then act optimally from now
[00:48:02] optimally from now on so really you would normally like to
[00:48:05] on so really you would normally like to have something like this so sum over S
[00:48:07] have something like this so sum over S Prime probability of S Prime given sa a
[00:48:10] Prime probability of S Prime given sa a Time B star of S Prime that's what you'd
[00:48:13] Time B star of S Prime that's what you'd have in like the Bellman equation and
[00:48:16] have in like the Bellman equation and what we're going to do here what Q
[00:48:17] what we're going to do here what Q learning does is it
[00:48:20] learning does is it approximates that by this
[00:48:23] approximates that by this Max and that is different than what
[00:48:25] Max and that is different than what sersa does because sari used the actual
[00:48:30] sersa does because sari used the actual action and CER says I don't really care
[00:48:32] action and CER says I don't really care what actual action you took I care about
[00:48:34] what actual action you took I care about what is the best thing you could have
[00:48:35] what is the best thing you could have done there because that's giving me a
[00:48:36] done there because that's giving me a better estimate of the maximum expected
[00:48:38] better estimate of the maximum expected discounted sum of rewards I'd get from
[00:48:41] discounted sum of rewards I'd get from that state till the end of
[00:48:43] that state till the end of time so that is what Q learning is doing
[00:48:46] time so that is what Q learning is doing so it looks really similar to the sarsa
[00:48:48] so it looks really similar to the sarsa update but our Target is going to be the
[00:48:50] update but our Target is going to be the reward I got plus the best reward that I
[00:48:53] reward I got plus the best reward that I think I could have achieved from that
[00:48:54] think I could have achieved from that next state
[00:48:58] okay all right so then we get an
[00:49:00] okay all right so then we get an algorithm that looks extremely similar
[00:49:02] algorithm that looks extremely similar to what we saw before but we have this
[00:49:05] to what we saw before but we have this Max over the next action and then I'll
[00:49:08] Max over the next action and then I'll just make sure I think I forgot to write
[00:49:13] that so whether we're doing Monte Carlo
[00:49:16] that so whether we're doing Monte Carlo or Sara or Q learning in all of these
[00:49:18] or Sara or Q learning in all of these cases we're interleaving Gathering some
[00:49:20] cases we're interleaving Gathering some data under our current Epsilon greedy
[00:49:22] data under our current Epsilon greedy policy and then using it to update a q
[00:49:24] policy and then using it to update a q value and because we don't know what the
[00:49:27] value and because we don't know what the actual Q function is we're sort of doing
[00:49:29] actual Q function is we're sort of doing this weighted approximation between um
[00:49:33] this weighted approximation between um our current estim of the Q function and
[00:49:35] our current estim of the Q function and the Target that we just put in and we do
[00:49:37] the Target that we just put in and we do this over and over and over
[00:49:42] again so similar to sarsa the conditions
[00:49:45] again so similar to sarsa the conditions to make sure that Q learning in the
[00:49:47] to make sure that Q learning in the tabular case so things get a lot more
[00:49:49] tabular case so things get a lot more complicated once we go
[00:49:52] complicated once we go into into the function approximation
[00:49:55] into into the function approximation case but in order for Q learning um with
[00:49:58] case but in order for Q learning um with EEG exploration to converge the optimal
[00:50:00] EEG exploration to converge the optimal qar you again need to visit everything
[00:50:02] qar you again need to visit everything infinitely often your step sizes has to
[00:50:04] infinitely often your step sizes has to satisfy the robins Monroe sequence um
[00:50:08] satisfy the robins Monroe sequence um and one important thing to notice here
[00:50:12] and one important thing to notice here is that you can estimate qar without
[00:50:16] is that you can estimate qar without being
[00:50:19] being Glee which is different than sarsa
[00:50:22] Glee which is different than sarsa because you're always doing this
[00:50:25] because you're always doing this Max so so even if you act completely
[00:50:28] Max so so even if you act completely randomly so just like infinite
[00:50:30] randomly so just like infinite exploration not being greedy you can
[00:50:33] exploration not being greedy you can learn qar because in your qar estimate
[00:50:38] learn qar because in your qar estimate here you're always doing this Max
[00:50:41] here you're always doing this Max a that's an important difference
[00:50:43] a that's an important difference compared to
[00:50:45] sersa but if you actually want to use
[00:50:47] sersa but if you actually want to use that information to make good decisions
[00:50:49] that information to make good decisions in the world you need to become greedy
[00:50:50] in the world you need to become greedy over time and be using that information
[00:50:53] over time and be using that information to actually select the best action
[00:50:54] to actually select the best action according to your Q function
[00:50:57] according to your Q function so in for EG algorithms with Q learning
[00:50:59] so in for EG algorithms with Q learning you normally Decay your Epsilon over
[00:51:01] you normally Decay your Epsilon over time so you're getting more and more
[00:51:02] time so you're getting more and more deterministic and you're taking your
[00:51:04] deterministic and you're taking your estimate of what qar is and using it to
[00:51:06] estimate of what qar is and using it to make
[00:51:10] decisions okay we're now going to go
[00:51:12] decisions okay we're now going to go into function approximation I'm just
[00:51:14] into function approximation I'm just going to pause there in case people had
[00:51:15] going to pause there in case people had any
[00:51:22] questions um so foring
[00:51:32] great great question so um if there are
[00:51:35] great great question so um if there are no ties in your Q function like as in
[00:51:38] no ties in your Q function like as in like for any action there is or any
[00:51:39] like for any action there is or any state there is a uniquely best action
[00:51:41] state there is a uniquely best action it'll converge to a deterministic
[00:51:44] it'll converge to a deterministic policy um if there are ties it'll
[00:51:46] policy um if there are ties it'll generally you know pick between those
[00:51:47] generally you know pick between those arbitrarily there'll be like an infinite
[00:51:49] arbitrarily there'll be like an infinite number of optimal policies if there are
[00:51:51] number of optimal policies if there are ties in your Q function great question
[00:51:58] all right so now what we're going to do
[00:51:59] all right so now what we're going to do is we're going to layer on function
[00:52:00] is we're going to layer on function approximation on top so this was all
[00:52:02] approximation on top so this was all assuming that we just had this table
[00:52:04] assuming that we just had this table where we could write down the value for
[00:52:06] where we could write down the value for every state and action separately and
[00:52:08] every state and action separately and now we want to use function
[00:52:09] now we want to use function approximation so we can start to do
[00:52:11] approximation so we can start to do problems like
[00:52:13] problems like Atari so the motivation for doing this
[00:52:16] Atari so the motivation for doing this and I know for those of you who've taken
[00:52:17] and I know for those of you who've taken machine learning this is probably clear
[00:52:20] machine learning this is probably clear but it's nice to think about what this
[00:52:21] but it's nice to think about what this means in the context of reinforcement
[00:52:23] means in the context of reinforcement learning so what are the things that we
[00:52:24] learning so what are the things that we might be storing or trying to to
[00:52:26] might be storing or trying to to manipulate that might be the Dynamics or
[00:52:28] manipulate that might be the Dynamics or reward model the value function the
[00:52:30] reward model the value function the state action value function or the
[00:52:32] state action value function or the policy and if you were thinking about
[00:52:34] policy and if you were thinking about pixel space you do not want to write
[00:52:36] pixel space you do not want to write that down as like one different value
[00:52:39] that down as like one different value for every bless you for every different
[00:52:41] for every bless you for every different um possible image in the world so we're
[00:52:44] um possible image in the world so we're going to want compact representations
[00:52:46] going to want compact representations like what we can do with neural networks
[00:52:48] like what we can do with neural networks so that we reduce the memory we need to
[00:52:50] so that we reduce the memory we need to like write down those Dynamics models
[00:52:51] like write down those Dynamics models the value function or Q or the policy we
[00:52:54] the value function or Q or the policy we reduce the computation and ideally we
[00:52:56] reduce the computation and ideally we might even be able to reduce the
[00:52:59] might even be able to reduce the experience and I think this last point
[00:53:01] experience and I think this last point maybe is a particularly interesting one
[00:53:02] maybe is a particularly interesting one to think about so you can imagine like
[00:53:04] to think about so you can imagine like if you're agent is learning to play an
[00:53:06] if you're agent is learning to play an Atari game or play Breakout it um it
[00:53:09] Atari game or play Breakout it um it might want to know that like oh well if
[00:53:11] might want to know that like oh well if these pixels are slightly different here
[00:53:13] these pixels are slightly different here most of the time you might still take
[00:53:14] most of the time you might still take the same decision and so then instead of
[00:53:17] the same decision and so then instead of having to learn from scratch what to do
[00:53:18] having to learn from scratch what to do in each state you can get this sort of
[00:53:20] in each state you can get this sort of generalization and that could be really
[00:53:22] generalization and that could be really important in terms of reducing the
[00:53:24] important in terms of reducing the amount of data we need to learn to make
[00:53:25] amount of data we need to learn to make good decisions
[00:53:29] all right so how do we do this what
[00:53:32] all right so how do we do this what we're going to try to do is we're going
[00:53:33] we're going to try to do is we're going to essentially do the same thing as what
[00:53:34] to essentially do the same thing as what we did before but we're also going to
[00:53:36] we did before but we're also going to have to incorporate a function
[00:53:37] have to incorporate a function approximation step so let's just think
[00:53:40] approximation step so let's just think about how we would do this if we had an
[00:53:41] about how we would do this if we had an oracle so what I mean by this is we're
[00:53:44] oracle so what I mean by this is we're not thinking yet right now about all the
[00:53:46] not thinking yet right now about all the learning and like Gathering data we're
[00:53:48] learning and like Gathering data we're just assuming how do we fit a function
[00:53:49] just assuming how do we fit a function to represent our Q
[00:53:51] to represent our Q function so let's imagine that you had
[00:53:53] function so let's imagine that you had an oracle that for any state in action
[00:53:55] an oracle that for any state in action it would give you the true value for a
[00:53:57] it would give you the true value for a particular policy and that state in
[00:53:59] particular policy and that state in action so it would tell you like that's
[00:54:01] action so it would tell you like that's three or that's
[00:54:03] three or that's seven so then you could say okay now
[00:54:05] seven so then you could say okay now I've just got a supervised learning
[00:54:06] I've just got a supervised learning problem I've got input tles of states
[00:54:09] problem I've got input tles of states and actions and I have output values of
[00:54:11] and actions and I have output values of my Q
[00:54:12] my Q function and what I want to do now is
[00:54:14] function and what I want to do now is just learn a function to like a
[00:54:16] just learn a function to like a regression function to say given the
[00:54:17] regression function to say given the state in action what is the
[00:54:19] state in action what is the output and so imagine that you're in a
[00:54:22] output and so imagine that you're in a case where like you know we have a
[00:54:24] case where like you know we have a continuous set of states and we only
[00:54:25] continuous set of states and we only have one
[00:54:27] have one action then you might just have all
[00:54:29] action then you might just have all these different
[00:54:31] these different points and maybe you just want to learn
[00:54:33] points and maybe you just want to learn a function that predicts the
[00:54:37] a function that predicts the Q for every single state you just learn
[00:54:40] Q for every single state you just learn like a parametric function or it could
[00:54:41] like a parametric function or it could be a deep neural
[00:54:43] be a deep neural network and in general just like in
[00:54:45] network and in general just like in supervised learning the objective is
[00:54:46] supervised learning the objective is going to be to find the best approximate
[00:54:48] going to be to find the best approximate representation of Q given some weights
[00:54:50] representation of Q given some weights or given some neural network
[00:54:53] or given some neural network architecture so we've got like you know
[00:54:55] architecture so we've got like you know some neural net
[00:54:58] and we're just going to fit this to try
[00:55:01] and we're just going to fit this to try to if we had these
[00:55:03] to if we had these points but of course we don't have these
[00:55:05] points but of course we don't have these points and we're going to see how we're
[00:55:07] points and we're going to see how we're going to handle it we don't have these
[00:55:08] going to handle it we don't have these points but this is kind of the intuition
[00:55:09] points but this is kind of the intuition is that if you had these then you could
[00:55:11] is that if you had these then you could do the function approximation step by
[00:55:13] do the function approximation step by saying okay well how do I I am going to
[00:55:15] saying okay well how do I I am going to handle generalization by using like a
[00:55:17] handle generalization by using like a linear function or a deep neural network
[00:55:19] linear function or a deep neural network to say for each of these states and
[00:55:20] to say for each of these states and actions what is the
[00:55:24] output so just to highlight here in this
[00:55:27] output so just to highlight here in this class generally we will be focusing on
[00:55:29] class generally we will be focusing on methods that use stochastic gradient
[00:55:30] methods that use stochastic gradient descent to try to fit these functions
[00:55:32] descent to try to fit these functions and again I expect most of this is
[00:55:34] and again I expect most of this is familiar for you guys if you've done
[00:55:35] familiar for you guys if you've done machine learning if you haven't you can
[00:55:37] machine learning if you haven't you can come talk to me or any of the
[00:55:39] come talk to me or any of the Tas um generally we're going to just use
[00:55:42] Tas um generally we're going to just use mean squared error and we're going to
[00:55:44] mean squared error and we're going to try to fit a function that minimizes the
[00:55:47] try to fit a function that minimizes the mean squared error we're going to do
[00:55:48] mean squared error we're going to do gradient descent to find a local minimum
[00:55:50] gradient descent to find a local minimum and we're going to do stochastic
[00:55:52] and we're going to do stochastic gradient descent um just to you to
[00:55:54] gradient descent um just to you to compute an approximate gradient
[00:56:00] so in this
[00:56:03] so in this case I have here is just to write that
[00:56:06] case I have here is just to write that out really quickly you would have
[00:56:07] out really quickly you would have something like this you'd have
[00:56:11] WJ is just the derivative of
[00:56:16] this like
[00:56:20] that so I'm going to take this equation
[00:56:22] that so I'm going to take this equation star and I'm just going to take the
[00:56:24] star and I'm just going to take the derivative of it which is going to be
[00:56:47] two so we're just going to take the
[00:56:49] two so we're just going to take the derivative of this and essentially that
[00:56:50] derivative of this and essentially that just means we're going to have to take
[00:56:51] just means we're going to have to take the derivative through our Q function
[00:56:54] the derivative through our Q function representation like using autoi per deal
[00:56:57] representation like using autoi per deal networks and then we can use this to
[00:56:59] networks and then we can use this to update our our
[00:57:01] update our our weights all right so we'll do stochastic
[00:57:03] weights all right so we'll do stochastic gradient descent to do
[00:57:05] gradient descent to do this and the main thing is that that's
[00:57:09] this and the main thing is that that's what we're going to be doing to plug in
[00:57:11] what we're going to be doing to plug in um in order to do policy evaluation or
[00:57:13] um in order to do policy evaluation or to do
[00:57:15] to do control so of course in general we don't
[00:57:17] control so of course in general we don't have those we don't have for each state
[00:57:19] have those we don't have for each state in action what the Q value was if it was
[00:57:21] in action what the Q value was if it was we wouldn't need to do any learning we
[00:57:23] we wouldn't need to do any learning we need to learn that from data and so the
[00:57:26] need to learn that from data and so the idea is that we're going to do model
[00:57:28] idea is that we're going to do model free state action value function
[00:57:30] free state action value function approximation so just like what we've
[00:57:32] approximation so just like what we've been seeing before we we're doing model
[00:57:34] been seeing before we we're doing model free state action value function um now
[00:57:38] free state action value function um now we're going to actually do that but just
[00:57:39] we're going to actually do that but just do an approximation we're instead of
[00:57:40] do an approximation we're instead of writing it down as a table we're going
[00:57:42] writing it down as a table we're going to write it down with these parameters
[00:57:43] to write it down with these parameters function parameterized
[00:57:45] function parameterized functions okay
[00:57:49] functions okay so the idea now is like similarly we
[00:57:52] so the idea now is like similarly we just saw before all these methods either
[00:57:54] just saw before all these methods either where we use Monte Carlo methods or
[00:57:56] where we use Monte Carlo methods or difference methods to try to do these
[00:57:58] difference methods to try to do these approximations um now what we're going
[00:58:00] approximations um now what we're going to do is that when we do the estimate
[00:58:02] to do is that when we do the estimate update step we're also going to fit the
[00:58:04] update step we're also going to fit the function
[00:58:05] function approximator so just like in the
[00:58:07] approximator so just like in the algorithms we saw before where we do
[00:58:09] algorithms we saw before where we do like policy evaluation and policy
[00:58:10] like policy evaluation and policy Improvement now when we do the policy
[00:58:12] Improvement now when we do the policy evaluation we're also going to just
[00:58:13] evaluation we're also going to just refit like our whole Q function for
[00:58:18] refit like our whole Q function for example okay so let's see how that could
[00:58:20] example okay so let's see how that could work so for Monte Carlo value function
[00:58:23] work so for Monte Carlo value function approximation we're going to remember
[00:58:25] approximation we're going to remember that our return G is an unbiased but
[00:58:27] that our return G is an unbiased but noisy sample of the expected return so
[00:58:30] noisy sample of the expected return so we can think of us having like this
[00:58:32] we can think of us having like this state action return State action return
[00:58:35] state action return State action return Etc and so you can substitute in those
[00:58:39] Etc and so you can substitute in those G's for the True Q Pi when you're doing
[00:58:42] G's for the True Q Pi when you're doing your
[00:58:44] fitting so let's see what that would
[00:58:46] fitting so let's see what that would look like so in this
[00:58:49] look like so in this case remember what we would like here
[00:58:51] case remember what we would like here when we're doing our function
[00:58:52] when we're doing our function approximation is that this is the real Q
[00:58:56] approximation is that this is the real Q of the policy but we don't know what the
[00:58:58] of the policy but we don't know what the real policy real Q value is so we're
[00:59:00] real policy real Q value is so we're just going to plug in our observe
[00:59:03] just going to plug in our observe return so you know we
[00:59:06] return so you know we want
[00:59:07] want would like
[00:59:10] would like Q of
[00:59:13] Q of sa but we don't have that so we're going
[00:59:15] sa but we don't have that so we're going to plug in the return that we just
[00:59:18] to plug in the return that we just observed and then we'll just do the
[00:59:20] observed and then we'll just do the derivative we'll be plugging that in for
[00:59:21] derivative we'll be plugging that in for our derivative and then we update our
[00:59:22] our derivative and then we update our weights using that derivative with
[00:59:24] weights using that derivative with respect to minimizing the me
[00:59:28] squitter so this would just be for
[00:59:31] squitter so this would just be for policy evaluation if you have a fixed
[00:59:32] policy evaluation if you have a fixed policy you would just do this at each
[00:59:35] policy you would just do this at each time point so after you see um after you
[00:59:37] time point so after you see um after you get a return then you would update your
[00:59:40] get a return then you would update your Q function you do this many many
[00:59:42] Q function you do this many many times and for some of you this might
[00:59:44] times and for some of you this might start to look redundant but I think it's
[00:59:45] start to look redundant but I think it's just useful to see that essentially the
[00:59:47] just useful to see that essentially the structure of all of these algorithms
[00:59:49] structure of all of these algorithms whether it is policy evaluation or
[00:59:51] whether it is policy evaluation or tabular or function approximation is
[00:59:53] tabular or function approximation is extremely similar we are just either
[00:59:56] extremely similar we are just either sampling an episode or sampling a tle we
[00:59:59] sampling an episode or sampling a tle we are going to do one step which is like
[01:00:00] are going to do one step which is like policy evaluation where we update our
[01:00:02] policy evaluation where we update our estimate of the Q function maybe
[01:00:04] estimate of the Q function maybe optionally do function approximation
[01:00:06] optionally do function approximation fitting and then we're going to use that
[01:00:08] fitting and then we're going to use that to figure out how to act next if we
[01:00:11] to figure out how to act next if we doing control we'll see an example of
[01:00:13] doing control we'll see an example of that
[01:00:14] that shortly okay so that is Monte
[01:00:17] shortly okay so that is Monte Carlo okay
[01:00:23] oops okay all right for temporal
[01:00:26] oops okay all right for temporal learning it's very similar but now we
[01:00:28] learning it's very similar but now we are going to um have this weighted sum
[01:00:31] are going to um have this weighted sum where we plug
[01:00:32] where we plug in we bootstrap so we plug in our
[01:00:34] in we bootstrap so we plug in our current estimate of the value of S Prime
[01:00:37] current estimate of the value of S Prime so this is the same update we saw before
[01:00:39] so this is the same update we saw before this was for tabular cases and now we're
[01:00:42] this was for tabular cases and now we're going to do it for function
[01:00:45] going to do it for function approximation okay so let's first just
[01:00:48] approximation okay so let's first just see how we do it for function approxim
[01:00:50] see how we do it for function approxim it's just useful I think when we look at
[01:00:51] it's just useful I think when we look at this to think about all the different
[01:00:52] this to think about all the different ways we're doing
[01:00:54] ways we're doing approximations we are sampling to
[01:00:56] approximations we are sampling to approximate the expected value over the
[01:00:58] approximate the expected value over the next state we are bootstrapping to plug
[01:01:00] next state we are bootstrapping to plug in what the value of those states are
[01:01:02] in what the value of those states are and now we're also going to do function
[01:01:03] and now we're also going to do function approximation because we're going to
[01:01:05] approximation because we're going to represent the value of a function with
[01:01:07] represent the value of a function with some weights okay so we're going to have
[01:01:09] some weights okay so we're going to have these
[01:01:11] these weights all
[01:01:13] weights all right and again we can just do
[01:01:15] right and again we can just do stochastic gradient descent to fit our
[01:01:17] stochastic gradient descent to fit our weight function to represent that value
[01:01:22] function okay
[01:01:27] so you'll get something like this where
[01:01:29] so you'll get something like this where as long as you're if you're in a
[01:01:30] as long as you're if you're in a terminal State you'll restart the
[01:01:32] terminal State you'll restart the episode otherwise you'll just be doing
[01:01:34] episode otherwise you'll just be doing these Computing the gradient with
[01:01:36] these Computing the gradient with respect to your minimizing your mean
[01:01:38] respect to your minimizing your mean sored eror and updating your
[01:01:43] weights and then the then we're we'll
[01:01:47] weights and then the then we're we'll see how we do this for
[01:01:49] see how we do this for control so for control it's going to be
[01:01:51] control so for control it's going to be very
[01:01:53] very similar now what we'll make sure to do
[01:01:55] similar now what we'll make sure to do is we're always going to be using the Q
[01:01:56] is we're always going to be using the Q function instead of the value
[01:01:58] function instead of the value function and we're now often going to be
[01:02:01] function and we're now often going to be doing off policy learning again like Q
[01:02:04] doing off policy learning again like Q learning so we'll again do stochastic
[01:02:07] learning so we'll again do stochastic gradient descent with respect to our Q
[01:02:09] gradient descent with respect to our Q function we're going to sample the
[01:02:10] function we're going to sample the gradient and we'll have very you know an
[01:02:12] gradient and we'll have very you know an algorithm is very similar to the one
[01:02:14] algorithm is very similar to the one we've seen
[01:02:15] we've seen before so we can either use sarsa where
[01:02:20] before so we can either use sarsa where we have our Q function where we always
[01:02:22] we have our Q function where we always plug in what is the actual action we
[01:02:24] plug in what is the actual action we took next or we can have q learning
[01:02:27] took next or we can have q learning where we plug in a Max over the next q
[01:02:32] function and raise your hand if you've
[01:02:34] function and raise your hand if you've implemented deep Q learning
[01:02:37] implemented deep Q learning before okay so 1% but most people not
[01:02:40] before okay so 1% but most people not yeah okay so you going to imagine in
[01:02:41] yeah okay so you going to imagine in general this is any form of function
[01:02:43] general this is any form of function approximator but often this is going to
[01:02:45] approximator but often this is going to be like a deep neural
[01:02:47] be like a deep neural network
[01:02:49] network okay now one thing I just want to
[01:02:51] okay now one thing I just want to highlight here is that again just sort
[01:02:53] highlight here is that again just sort of being in terms of being concerned
[01:02:55] of being in terms of being concerned whether all this is going to work
[01:02:57] whether all this is going to work there's a lot of approximations that are
[01:02:59] there's a lot of approximations that are happening here so particularly for Q
[01:03:01] happening here so particularly for Q learning and it's led to what Sarto um
[01:03:04] learning and it's led to what Sarto um the authors of the book that is optional
[01:03:06] the authors of the book that is optional textbook for the class called The Deadly
[01:03:09] textbook for the class called The Deadly Triad and what they say is that if you
[01:03:11] Triad and what they say is that if you are doing bootstrapping meaning that
[01:03:13] are doing bootstrapping meaning that you're plugging in an estimate of what
[01:03:15] you're plugging in an estimate of what is the value of the next state and
[01:03:17] is the value of the next state and you're doing function approximation like
[01:03:19] you're doing function approximation like you're using like a deep neural network
[01:03:20] you're using like a deep neural network or linear function and you're doing off
[01:03:23] or linear function and you're doing off policy learning where you acting in a
[01:03:25] policy learning where you acting in a different way than the the data you're
[01:03:26] different way than the the data you're getting under those cases you may not
[01:03:28] getting under those cases you may not converge at
[01:03:30] converge at all like you just your Q function May
[01:03:33] all like you just your Q function May oscillate you may not converge to um
[01:03:35] oscillate you may not converge to um anything and um you are certainly not
[01:03:38] anything and um you are certainly not guaranteed to converge to
[01:03:40] guaranteed to converge to qar so it's just good to keep in mind
[01:03:44] qar so it's just good to keep in mind that that could occur um I think for
[01:03:47] that that could occur um I think for some intuition for why this can occur
[01:03:49] some intuition for why this can occur the Bellman operator if you think back a
[01:03:51] the Bellman operator if you think back a couple lectures ago we proved as a
[01:03:52] couple lectures ago we proved as a contraction meaning that as we apply it
[01:03:54] contraction meaning that as we apply it repeatedly we went to this fix point in
[01:03:57] repeatedly we went to this fix point in the tabular setting but the problem is
[01:04:00] the tabular setting but the problem is is that like when you do a bman backup
[01:04:01] is that like when you do a bman backup that operator is a contraction meaning
[01:04:03] that operator is a contraction meaning that if you apply the bment operator two
[01:04:05] that if you apply the bment operator two different things their distance gets
[01:04:06] different things their distance gets smaller
[01:04:07] smaller afterwards value function approximation
[01:04:10] afterwards value function approximation fitting can be an expansion which means
[01:04:12] fitting can be an expansion which means if you take two things and then you try
[01:04:14] if you take two things and then you try to do value function approximation like
[01:04:16] to do value function approximation like you f align to this one and align to
[01:04:17] you f align to this one and align to this one the distance between two points
[01:04:20] this one the distance between two points afterwards can actually get bigger than
[01:04:21] afterwards can actually get bigger than before you did the value function
[01:04:23] before you did the value function approximation so there's a really
[01:04:25] approximation so there's a really beautiful example of this in a paper by
[01:04:26] beautiful example of this in a paper by Jeff Gordon um from 1995 I will just
[01:04:30] Jeff Gordon um from 1995 I will just Jeff
[01:04:32] Jeff Gordon 1995 has a really nice example of
[01:04:36] Gordon 1995 has a really nice example of this where you just can kind of visually
[01:04:37] this where you just can kind of visually see when you have these two functions
[01:04:39] see when you have these two functions and these like points that after you do
[01:04:41] and these like points that after you do this value function approximation you've
[01:04:43] this value function approximation you've actually made the distance between them
[01:04:45] actually made the distance between them bigger and so that means that you have
[01:04:47] bigger and so that means that you have this thing where you're kind of
[01:04:48] this thing where you're kind of alternating between something which you
[01:04:50] alternating between something which you know is a contraction and driving you um
[01:04:52] know is a contraction and driving you um driving you towards a fix point and
[01:04:54] driving you towards a fix point and something which might actually Amplified
[01:04:56] something which might actually Amplified distance differences and so because of
[01:04:58] distance differences and so because of that it's not always the case that
[01:04:59] that it's not always the case that you're guaranteed to converge to a fixed
[01:05:01] you're guaranteed to converge to a fixed point so this is something important to
[01:05:04] point so this is something important to know however I think it's also kind of a
[01:05:06] know however I think it's also kind of a um it's an important part of the history
[01:05:09] um it's an important part of the history of the field in that in the 1990s there
[01:05:11] of the field in that in the 1990s there was a bunch of work showing that this
[01:05:13] was a bunch of work showing that this could occur that like even with some
[01:05:15] could occur that like even with some really simple settings like linear value
[01:05:17] really simple settings like linear value function approximators you just
[01:05:19] function approximators you just approximate things with a line um that
[01:05:21] approximate things with a line um that sometimes you could get these kind of
[01:05:22] sometimes you could get these kind of oscillations or lack of convergence and
[01:05:24] oscillations or lack of convergence and so people were really concerned about
[01:05:26] so people were really concerned about using function approximators with
[01:05:27] using function approximators with reinforcement learning but then what
[01:05:30] reinforcement learning but then what happened is that Deep Mind showed well
[01:05:32] happened is that Deep Mind showed well actually there are some ways to tackle
[01:05:34] actually there are some ways to tackle this and we can do really amazing things
[01:05:36] this and we can do really amazing things with it and so I think it's a useful
[01:05:38] with it and so I think it's a useful like a useful lesson from history over
[01:05:40] like a useful lesson from history over the difference between like what can
[01:05:42] the difference between like what can occur in maybe some sort of hard you
[01:05:44] occur in maybe some sort of hard you know not ideal cases versus what
[01:05:47] know not ideal cases versus what actually occurs in practice and so we
[01:05:48] actually occurs in practice and so we shouldn't let like sort of some of the
[01:05:51] shouldn't let like sort of some of the negative examples limit us from
[01:05:53] negative examples limit us from considering what might work in some
[01:05:54] considering what might work in some other scenarios
[01:05:56] other scenarios so let's see that now let's see
[01:05:59] so let's see that now let's see dqn okay
[01:06:01] dqn okay so the idea with dqn is we're going to
[01:06:04] so the idea with dqn is we're going to use these ideas to actually play Atari
[01:06:06] use these ideas to actually play Atari so we're going to take in images of the
[01:06:07] so we're going to take in images of the game we're going to use convolutional
[01:06:09] game we're going to use convolutional neural networks and we're going to um
[01:06:11] neural networks and we're going to um have a really big deep neural network to
[01:06:13] have a really big deep neural network to represent the Q function and do Q
[01:06:15] represent the Q function and do Q learning with
[01:06:16] learning with it okay so the idea was well we knew
[01:06:21] it okay so the idea was well we knew that sometimes like Q learning with
[01:06:22] that sometimes like Q learning with value function approximation can diverge
[01:06:25] value function approximation can diverge and 's a number of different issues but
[01:06:27] and 's a number of different issues but one of them is kind of this stability
[01:06:29] one of them is kind of this stability thing so we know that there's
[01:06:30] thing so we know that there's correlations between samples your data
[01:06:32] correlations between samples your data is not IID which is what you would
[01:06:34] is not IID which is what you would normally want for when you're doing
[01:06:35] normally want for when you're doing function approximation and the other is
[01:06:38] function approximation and the other is that you have this kind of
[01:06:38] that you have this kind of non-stationary Target thing which is
[01:06:41] non-stationary Target thing which is like when you plug in say with TD
[01:06:43] like when you plug in say with TD learning you're plugging in gamma plus
[01:06:45] learning you're plugging in gamma plus sorry R plus gamma times the value of
[01:06:48] sorry R plus gamma times the value of your next state and that value of the
[01:06:50] your next state and that value of the next Tate is constantly changing as you
[01:06:51] next Tate is constantly changing as you get more
[01:06:52] get more data so what dqn did is they said well
[01:06:56] data so what dqn did is they said well what we're going to do is we're going to
[01:06:57] what we're going to do is we're going to use experience Replay in particular
[01:06:59] use experience Replay in particular we're going to reuse tuples over time
[01:07:02] we're going to reuse tuples over time and we're also going to get fixed Q
[01:07:03] and we're also going to get fixed Q targets and both of those things ended
[01:07:05] targets and both of those things ended up making a really big
[01:07:07] up making a really big difference particularly one of them
[01:07:09] difference particularly one of them we'll see in a second so the idea of
[01:07:11] we'll see in a second so the idea of experience replay is to say in general
[01:07:14] experience replay is to say in general if I think about um states that are
[01:07:16] if I think about um states that are nearby their Q function might be pretty
[01:07:19] nearby their Q function might be pretty similar and if I'm doing lots of updates
[01:07:21] similar and if I'm doing lots of updates that's breaking my IID stuff that I want
[01:07:23] that's breaking my IID stuff that I want for my function approximation
[01:07:26] for my function approximation so another thing you could do is just
[01:07:27] so another thing you could do is just have a replay buffer of lots of all the
[01:07:29] have a replay buffer of lots of all the different tuples you've seen in the past
[01:07:31] different tuples you've seen in the past and you just sample from one of those
[01:07:34] and you just sample from one of those and then compute a Target value and then
[01:07:36] and then compute a Target value and then do stochastic gradient
[01:07:38] do stochastic gradient descent and this might be really helpful
[01:07:41] descent and this might be really helpful anyway just in terms of data efficiency
[01:07:43] anyway just in terms of data efficiency because it means that instead of like
[01:07:44] because it means that instead of like taking your data and using it once and
[01:07:46] taking your data and using it once and then throwing it away you keep it and
[01:07:48] then throwing it away you keep it and then you can replay it just like how we
[01:07:50] then you can replay it just like how we talked about sort of batch learning last
[01:07:52] talked about sort of batch learning last time so an experience replay can be
[01:07:55] time so an experience replay can be useful because we're both replaying our
[01:07:56] useful because we're both replaying our data and so we can sort of squeeze more
[01:07:58] data and so we can sort of squeeze more information out of it and also we can
[01:08:00] information out of it and also we can select from very different parts of the
[01:08:02] select from very different parts of the past history which makes those updates
[01:08:04] past history which makes those updates more
[01:08:05] more independent okay so this is um and in
[01:08:08] independent okay so this is um and in general we're not going to keep the
[01:08:09] general we're not going to keep the buffer for all time we might keep like
[01:08:11] buffer for all time we might keep like the last million episodes or things like
[01:08:13] the last million episodes or things like that okay so that's one thing we could
[01:08:16] that okay so that's one thing we could do um now the other thing is that if we
[01:08:21] do um now the other thing is that if we think about what's happening in this
[01:08:22] think about what's happening in this case the way we change the weights is
[01:08:25] case the way we change the weights is going to be um in general the weights
[01:08:30] going to be um in general the weights appear here and here and here so this
[01:08:34] appear here and here and here so this target value is a function of the
[01:08:36] target value is a function of the weights itself because you're using a
[01:08:37] weights itself because you're using a value function approximation to
[01:08:39] value function approximation to represent the value of your next state
[01:08:41] represent the value of your next state and so the problem is that in general
[01:08:43] and so the problem is that in general this is going to change on your next
[01:08:45] this is going to change on your next update because you've just changed your
[01:08:47] update because you've just changed your your
[01:08:47] your weights um and this can also lead to
[01:08:50] weights um and this can also lead to instabilities because if you think of
[01:08:52] instabilities because if you think of sort of supervised learning you know
[01:08:54] sort of supervised learning you know your XY pairs your y is changing for
[01:08:56] your XY pairs your y is changing for even for the same X over time because
[01:08:58] even for the same X over time because you're changing your Q
[01:09:00] you're changing your Q function okay because this is a function
[01:09:02] function okay because this is a function of the weight and so as the weights
[01:09:04] of the weight and so as the weights change this sort of Target value is
[01:09:06] change this sort of Target value is going to change even for the same input
[01:09:09] going to change even for the same input so the second idea is to have fixed Q
[01:09:12] so the second idea is to have fixed Q updates and what the idea here is and so
[01:09:14] updates and what the idea here is and so remember this is like when we say the
[01:09:16] remember this is like when we say the target weight this is it's going to be
[01:09:18] target weight this is it's going to be what's we're using for Target weights is
[01:09:21] what's we're using for Target weights is that the weight the the weights or the
[01:09:23] that the weight the the weights or the parameters we're using to to estimate
[01:09:26] parameters we're using to to estimate the value of the next state we reach we
[01:09:28] the value of the next state we reach we are going to not update those as
[01:09:30] are going to not update those as much so we're going to have our Target
[01:09:33] much so we're going to have our Target network using a different set of Weights
[01:09:34] network using a different set of Weights than the weights that are being updated
[01:09:36] than the weights that are being updated so you can see here that we have a w
[01:09:38] so you can see here that we have a w minus meaning that like we're trying to
[01:09:40] minus meaning that like we're trying to make this more like supervised learning
[01:09:42] make this more like supervised learning where we have a fixed output y that is
[01:09:44] where we have a fixed output y that is not changing um while we're trying to
[01:09:46] not changing um while we're trying to update our
[01:09:48] update our W and so if you think about sort of the
[01:09:50] W and so if you think about sort of the example we want to just sort of draw it
[01:09:52] example we want to just sort of draw it like this here's our states here's our Q
[01:09:55] like this here's our states here's our Q function
[01:09:56] function um right now we'd like to sort of make
[01:09:58] um right now we'd like to sort of make sure that these points when we're like
[01:10:00] sure that these points when we're like trying to fit a line that those y's are
[01:10:02] trying to fit a line that those y's are not changing a lot what we're trying to
[01:10:03] not changing a lot what we're trying to fit the line and in general because
[01:10:06] fit the line and in general because they're a function of the weights
[01:10:07] they're a function of the weights themselves they might be moving and
[01:10:08] themselves they might be moving and perturbing and so what we're saying is
[01:10:10] perturbing and so what we're saying is no we're going to fix these so you can
[01:10:12] no we're going to fix these so you can think of this as just being you know a
[01:10:14] think of this as just being you know a fixed number for a while and then do
[01:10:16] fixed number for a while and then do multiple updates on this W to try to fit
[01:10:18] multiple updates on this W to try to fit that
[01:10:21] function and so what that means is we
[01:10:24] function and so what that means is we just have to uh we we keep around these
[01:10:26] just have to uh we we keep around these Target weights and we keep around um the
[01:10:28] Target weights and we keep around um the other
[01:10:30] other weights and this allows us to do this is
[01:10:32] weights and this allows us to do this is what the is called the fixed Q
[01:10:34] what the is called the fixed Q updating so if you think about what the
[01:10:37] updating so if you think about what the pseudo code would look like in this ta
[01:10:47] case is it's going to look pretty
[01:10:49] case is it's going to look pretty similar to the things we've seen you're
[01:10:50] similar to the things we've seen you're going to sample an action you're going
[01:10:51] going to sample an action you're going to observe order in the next state
[01:10:52] to observe order in the next state you're going to store the transition in
[01:10:54] you're going to store the transition in a replay buffer so you're going to keep
[01:10:56] a replay buffer so you're going to keep track of it then you're going to sample
[01:10:57] track of it then you're going to sample a random mini batch of tles from the
[01:10:59] a random mini batch of tles from the past you're going to do something you
[01:11:01] past you're going to do something you know keep track of if episodes
[01:11:03] know keep track of if episodes terminated otherwise you're going to say
[01:11:05] terminated otherwise you're going to say my target y that I'm going to try to fit
[01:11:07] my target y that I'm going to try to fit in my function approximator is my
[01:11:09] in my function approximator is my immediate reward plus the Maxum
[01:11:11] immediate reward plus the Maxum reactions of my Q function with my
[01:11:13] reactions of my Q function with my target
[01:11:14] target weights so I use my deep neural network
[01:11:16] weights so I use my deep neural network to predict the value of that state
[01:11:18] to predict the value of that state action pair and then I'm going to do
[01:11:20] action pair and then I'm going to do gradient descent on the difference
[01:11:21] gradient descent on the difference between these predicted y's and my
[01:11:24] between these predicted y's and my current estimate my current weights so
[01:11:26] current estimate my current weights so this is just the function fitting
[01:11:28] this is just the function fitting part and then you repeat this and then
[01:11:31] part and then you repeat this and then periodically you update your target
[01:11:33] periodically you update your target weights okay and I just want to
[01:11:36] weights okay and I just want to highlight here there's a bunch of
[01:11:38] highlight here there's a bunch of different choices to be made you have to
[01:11:39] different choices to be made you have to decide like what function approximator
[01:11:41] decide like what function approximator you're using you're using a deep neural
[01:11:42] you're using you're using a deep neural network um what's your learning rate how
[01:11:45] network um what's your learning rate how often to update the target weight uh how
[01:11:48] often to update the target weight uh how big should your replay buffer be there's
[01:11:49] big should your replay buffer be there's a lot of different choices that you have
[01:11:51] a lot of different choices that you have to
[01:11:54] make all right let's just take a quick
[01:11:57] make all right let's just take a quick second here to see if this Parts give
[01:11:59] second here to see if this Parts give chance to think about this
[01:12:01] chance to think about this part you may have just SE the answer but
[01:12:03] part you may have just SE the answer but that's okay um which is okay in dqn
[01:12:07] that's okay um which is okay in dqn we're going to compute the target value
[01:12:08] we're going to compute the target value for the sampled State action reward next
[01:12:09] for the sampled State action reward next States using a separate set of Target
[01:12:12] States using a separate set of Target weights so does that change the
[01:12:14] weights so does that change the computation time does it change the
[01:12:16] computation time does it change the memory requirements um are you not
[01:12:20] memory requirements um are you not sure put that in here
[01:12:26] we're now going to maintain two
[01:12:28] we're now going to maintain two different sets of weights to do our
[01:12:29] different sets of weights to do our function
[01:12:54] approximation e
[01:13:30] right um yep I see almost every converts
[01:13:33] right um yep I see almost every converts the the right answer very quickly it is
[01:13:36] the the right answer very quickly it is doubling the memory requirements so you
[01:13:37] doubling the memory requirements so you have to keep track of a second set of
[01:13:39] have to keep track of a second set of parameters um H it does not change the
[01:13:42] parameters um H it does not change the computation time just changes the the
[01:13:45] computation time just changes the the memory requirements so we just keep
[01:13:47] memory requirements so we just keep around two copies they have your deep
[01:13:48] around two copies they have your deep neural network one with the old weights
[01:13:49] neural network one with the old weights one with the new ones um and then the
[01:13:51] one with the new ones um and then the queue updating with respect to that is
[01:13:53] queue updating with respect to that is the same
[01:13:55] the same all right let's see what that actually
[01:13:56] all right let's see what that actually does so the kind of the key Innovations
[01:13:58] does so the kind of the key Innovations for dqn where we are going to use deep
[01:14:01] for dqn where we are going to use deep neural networks that had been done
[01:14:02] neural networks that had been done before but um not with I think this is
[01:14:04] before but um not with I think this is the first really big example with
[01:14:05] the first really big example with convolution convolutional neural
[01:14:07] convolution convolutional neural networks um it's going to maintain these
[01:14:10] networks um it's going to maintain these really large episodic replays and then
[01:14:13] really large episodic replays and then it is also going to have these fixed
[01:14:15] it is also going to have these fixed targets all right so what they have here
[01:14:18] targets all right so what they have here is they're going to do these series of
[01:14:20] is they're going to do these series of convolutions output and they're going to
[01:14:22] convolutions output and they're going to Output a q value for each action and and
[01:14:25] Output a q value for each action and and they're going to use that to make
[01:14:26] they're going to use that to make decisions and I think one of the things
[01:14:29] decisions and I think one of the things well there's multiple really remarkable
[01:14:30] well there's multiple really remarkable things about this paper one is that they
[01:14:33] things about this paper one is that they got extremely good performance across a
[01:14:34] got extremely good performance across a really wide set of games so instead of
[01:14:36] really wide set of games so instead of only having a few Benchmark tasks they
[01:14:38] only having a few Benchmark tasks they looked at the whole Suite of performance
[01:14:40] looked at the whole Suite of performance they are learning a different policy per
[01:14:42] they are learning a different policy per video game but it is the same neural
[01:14:44] video game but it is the same neural network architecture and I believe all
[01:14:46] network architecture and I believe all the same hyperparameters too so the idea
[01:14:48] the same hyperparameters too so the idea with that is to say like could we
[01:14:50] with that is to say like could we actually have sort of the same type of
[01:14:51] actually have sort of the same type of architecture in the same way that we
[01:14:53] architecture in the same way that we don't swap brains when we do different
[01:14:54] don't swap brains when we do different tasks
[01:14:55] tasks um and but have the same learning
[01:14:57] um and but have the same learning algorithm learn to be able to do many
[01:14:59] algorithm learn to be able to do many different types of tasks and so I think
[01:15:01] different types of tasks and so I think that was pretty impressive that they
[01:15:03] that was pretty impressive that they showed that that was possible so you
[01:15:04] showed that that was possible so you have the same algorithm same hyper
[01:15:06] have the same algorithm same hyper parameters but it could you know learn
[01:15:07] parameters but it could you know learn to do well in many different
[01:15:10] to do well in many different tasks I think one of the interesting
[01:15:12] tasks I think one of the interesting things about the paper is to consider
[01:15:14] things about the paper is to consider what were the aspects that were
[01:15:15] what were the aspects that were important for Success so here's just um
[01:15:18] important for Success so here's just um a subset of algorithms uh sorry subset
[01:15:20] a subset of algorithms uh sorry subset of the domains this is six this is a few
[01:15:22] of the domains this is six this is a few of the games and they also compare to
[01:15:25] of the games and they also compare to using a much more simple function
[01:15:27] using a much more simple function approximator and what you can see here
[01:15:29] approximator and what you can see here is that the Deep neural network is not
[01:15:32] is that the Deep neural network is not actually better right like the Deep
[01:15:33] actually better right like the Deep neural network does not look better than
[01:15:36] neural network does not look better than um uh than the linear case so it's not
[01:15:39] um uh than the linear case so it's not clear that just using a more function
[01:15:41] clear that just using a more function approx like um it wasn't just that they
[01:15:43] approx like um it wasn't just that they used a much more careful function
[01:15:47] approximator and the second thing was
[01:15:49] approximator and the second thing was whether they used this fixed q and that
[01:15:52] whether they used this fixed q and that helped um so you can see now that they
[01:15:55] helped um so you can see now that they are exceeding the performance of using a
[01:15:57] are exceeding the performance of using a more simple function approximator so
[01:15:59] more simple function approximator so this idea of kind of keeping things
[01:16:01] this idea of kind of keeping things stable is helpful um in terms of
[01:16:04] stable is helpful um in terms of oscillations but keeping using the
[01:16:07] oscillations but keeping using the replay was incredibly helpful so they
[01:16:10] replay was incredibly helpful so they went from like you know three or 10 up
[01:16:12] went from like you know three or 10 up to
[01:16:13] to 241 um or in some you know something
[01:16:16] 241 um or in some you know something from either roughly three times as good
[01:16:18] from either roughly three times as good and sometimes even more like a couple
[01:16:19] and sometimes even more like a couple orders of
[01:16:20] orders of magnitude so it was incredibly helpful
[01:16:23] magnitude so it was incredibly helpful to use an experience play buffer and
[01:16:26] to use an experience play buffer and maybe this isn't so surprising because
[01:16:28] maybe this isn't so surprising because it means that they are just reusing
[01:16:29] it means that they are just reusing their data a lot um but it was really
[01:16:33] their data a lot um but it was really you know incredibly important I think
[01:16:35] you know incredibly important I think that's really helpful to motivate why
[01:16:36] that's really helpful to motivate why thinking about sample efficiency and re
[01:16:38] thinking about sample efficiency and re reusing your data is helpful um and then
[01:16:41] reusing your data is helpful um and then combining these ideas led to even bigger
[01:16:43] combining these ideas led to even bigger benefits so it was helpful to have both
[01:16:45] benefits so it was helpful to have both the fixed targets and the replay buffer
[01:16:47] the fixed targets and the replay buffer but if you could only pick one the
[01:16:48] but if you could only pick one the replay buffer was just enormously
[01:16:52] replay buffer was just enormously helpful all right so as you guys know um
[01:16:56] helpful all right so as you guys know um uh there's been an enormous amount of
[01:16:57] uh there's been an enormous amount of interest in reinforcement learning and
[01:16:58] interest in reinforcement learning and deep reinforcement learning since there
[01:17:00] deep reinforcement learning since there was some immediate improvements kind of
[01:17:02] was some immediate improvements kind of within the next year or two um one is
[01:17:05] within the next year or two um one is called double dqn um uh and that also
[01:17:09] called double dqn um uh and that also it's a very simple change just maybe one
[01:17:11] it's a very simple change just maybe one or two lines and it does increase some
[01:17:14] or two lines and it does increase some of the requirements but for uh memory
[01:17:17] of the requirements but for uh memory but it is a really helpful approach so
[01:17:21] but it is a really helpful approach so um it tries to deal with the fact that
[01:17:22] um it tries to deal with the fact that you can get some interesting
[01:17:23] you can get some interesting maximization by it issues and happy to
[01:17:26] maximization by it issues and happy to talk about that offline um but so there
[01:17:28] talk about that offline um but so there was a few different immediate next
[01:17:29] was a few different immediate next algorithms uh but then there's been
[01:17:31] algorithms uh but then there's been enormous amount of work since and I
[01:17:32] enormous amount of work since and I think it really led to huge excitement
[01:17:34] think it really led to huge excitement in how we could couple these with really
[01:17:36] in how we could couple these with really um impressive function
[01:17:38] um impressive function approximators so just to summarize um
[01:17:41] approximators so just to summarize um the things that you should understand is
[01:17:42] the things that you should understand is to be able to implement td0 of Monte
[01:17:44] to be able to implement td0 of Monte Carlo one policy evaluation so things
[01:17:46] Carlo one policy evaluation so things like we talked about last time um you
[01:17:48] like we talked about last time um you should be able to implement Q learning
[01:17:49] should be able to implement Q learning SARS MC control algorithms again in
[01:17:52] SARS MC control algorithms again in tabular
[01:17:53] tabular settings you should understand and what
[01:17:55] settings you should understand and what are the issues that can cause
[01:17:56] are the issues that can cause instability um so things like function
[01:17:59] instability um so things like function approximation bootstrapping and off
[01:18:01] approximation bootstrapping and off policy learning and have an intuitive
[01:18:03] policy learning and have an intuitive sense for why that might be concerning
[01:18:05] sense for why that might be concerning and then also you should know some of
[01:18:06] and then also you should know some of the key features in dqn dqn that were
[01:18:09] the key features in dqn dqn that were critical and then next week we're going
[01:18:11] critical and then next week we're going to start to talk about a very different
[01:18:12] to start to talk about a very different way to do things which is just policy
[01:18:14] way to do things which is just policy gradient methods it is similar again to
[01:18:17] gradient methods it is similar again to this you can see how important policy
[01:18:18] this you can see how important policy iteration is it's going to be similar to
[01:18:20] iteration is it's going to be similar to policy iteration um and it's kind of
[01:18:23] policy iteration um and it's kind of similar to policy iteration of Monte
[01:18:24] similar to policy iteration of Monte Carlo and certain ways and directly
[01:18:25] Carlo and certain ways and directly trying to work with the policy I'll see
[01:18:28] trying to work with the policy I'll see you then

Stanford CS234 Reinforcement Learning I Policy Search 1 I 2024 I Lecture 5

Source: https://www.youtube.com/watch?v=L6OVEmV3NcE

---

Transcript

[00:00:05] hi everybody welcome back we're going to
[00:00:06] hi everybody welcome back we're going to go ahead and get started with a refresh
[00:00:08] go ahead and get started with a refresh your
[00:00:28] understanding okay hopefully everyone
[00:00:30] understanding okay hopefully everyone had a chance to think about this a
[00:00:31] had a chance to think about this a little bit more um so let's go through
[00:00:33] little bit more um so let's go through the answers the first one is
[00:00:37] true so if you are trying to evaluate
[00:00:41] true so if you are trying to evaluate the value of this is in the tabular case
[00:00:43] the value of this is in the tabular case so this is where we're assuming we're
[00:00:45] so this is where we're assuming we're going to sample each coule at random and
[00:00:48] going to sample each coule at random and we do a q learning update we do this an
[00:00:49] we do a q learning update we do this an infinite amount of times um we know for
[00:00:53] infinite amount of times um we know for standard tabular learning we can
[00:00:55] standard tabular learning we can converge to the true value of a policy
[00:00:58] converge to the true value of a policy um uh under as long as our learning rate
[00:01:01] um uh under as long as our learning rate schedule as such so if there's an exists
[00:01:03] schedule as such so if there's an exists a learning rate schedule under if you're
[00:01:06] a learning rate schedule under if you're decaying your learning rate um at the
[00:01:07] decaying your learning rate um at the right level then you will converge to
[00:01:10] right level then you will converge to the True Q value in the tabular case
[00:01:12] the True Q value in the tabular case because there's no function
[00:01:13] because there's no function approximation that's happening there in
[00:01:16] approximation that's happening there in the second case this is also true so we
[00:01:19] the second case this is also true so we talked a bit um about how we could think
[00:01:22] talked a bit um about how we could think about doing these things in a batch way
[00:01:24] about doing these things in a batch way where we do it over and over and over
[00:01:25] where we do it over and over and over again we take our existing data and we
[00:01:27] again we take our existing data and we run it through our either TD learning
[00:01:29] run it through our either TD learning update or our sarer update or other
[00:01:32] update or our sarer update or other things um and we said that the TD
[00:01:35] things um and we said that the TD learning updates if you do it in a batch
[00:01:37] learning updates if you do it in a batch way are equivalent to just taking um a
[00:01:40] way are equivalent to just taking um a certainty equivalent model which means
[00:01:43] certainty equivalent model which means you estimate the Dynamics model and you
[00:01:44] you estimate the Dynamics model and you estimate the reward model excuse me um
[00:01:47] estimate the reward model excuse me um from your existing data and then you do
[00:01:49] from your existing data and then you do dynamic
[00:01:50] dynamic programming so that's what we saw I
[00:01:52] programming so that's what we saw I think we saw that in lecture
[00:01:53] think we saw that in lecture three this one is false does somebody
[00:01:57] three this one is false does somebody want to say why it's false this one is
[00:01:59] want to say why it's false this one is not true
[00:02:03] there's a number of reasons why it could
[00:02:04] there's a number of reasons why it could be false I I want to share why why is
[00:02:07] be false I I want to share why why is dqi not guaranteed to necessarily
[00:02:09] dqi not guaranteed to necessarily convert to the optimal Q
[00:02:16] function
[00:02:19] function yeah I mean would you need to enforce a
[00:02:23] yeah I mean would you need to enforce a certain number of iterations to for it
[00:02:26] certain number of iterations to for it to have any chance of converging at all
[00:02:28] to have any chance of converging at all so good point so it related that so
[00:02:29] so good point so it related that so certainly if you don't do enough
[00:02:30] certainly if you don't do enough iterations but even if you do an
[00:02:32] iterations but even if you do an infinite number of iterations it also
[00:02:34] infinite number of iterations it also might not be guaranteed to converge me
[00:02:36] might not be guaranteed to converge me tell me why you went infinite so right
[00:02:39] tell me why you went infinite so right you certainly need a lot of iterations
[00:02:40] you certainly need a lot of iterations but even if you had a lot of interations
[00:02:42] but even if you had a lot of interations you still still might not be guaranteed
[00:02:44] you still still might not be guaranteed to
[00:02:44] to converge I think here it helps to think
[00:02:47] converge I think here it helps to think about what we often call realizability
[00:02:48] about what we often call realizability which is we don't know what the
[00:02:50] which is we don't know what the functional form is of Q and so you could
[00:02:52] functional form is of Q and so you could think of the fact that I'm GNA draw it
[00:02:55] think of the fact that I'm GNA draw it in as if the state space was
[00:02:57] in as if the state space was onedimensional but in general of course
[00:02:59] onedimensional but in general of course like the state space is like this Vector
[00:03:02] like the state space is like this Vector um or it's like you know images and so
[00:03:03] um or it's like you know images and so it's really high dimensional but imagine
[00:03:06] it's really high dimensional but imagine that it was onedimensional even here you
[00:03:08] that it was onedimensional even here you don't know what what your V function or
[00:03:10] don't know what what your V function or your Q function might look like and so
[00:03:13] your Q function might look like and so if you are using the wrong approximator
[00:03:15] if you are using the wrong approximator like if you are using say a
[00:03:17] like if you are using say a line instead of a multi degree
[00:03:21] line instead of a multi degree polinomial then no matter how much data
[00:03:23] polinomial then no matter how much data you have you're not going to converge to
[00:03:24] you have you're not going to converge to the optimal key function or you what be
[00:03:26] the optimal key function or you what be guaranteed that
[00:03:28] guaranteed that you because just can't even realize
[00:03:31] you because just can't even realize it so in general um and there's all
[00:03:35] it so in general um and there's all sorts of additional instability things
[00:03:36] sorts of additional instability things that mean we can't be guaranteed it's
[00:03:38] that mean we can't be guaranteed it's going to converge so we're not
[00:03:40] going to converge so we're not guaranteed it'll converge but
[00:03:41] guaranteed it'll converge but empirically it often does pretty well so
[00:03:44] empirically it often does pretty well so we'll see we and if you look at the
[00:03:46] we'll see we and if you look at the empirical results it often does really
[00:03:48] empirical results it often does really quite
[00:03:49] quite well great so so far in the class we've
[00:03:53] well great so so far in the class we've talked a lot about these value function
[00:03:55] talked a lot about these value function based methods where we thought about
[00:03:57] based methods where we thought about having an explicit representation of the
[00:03:59] having an explicit representation of the expected sum of discounted rewards
[00:04:02] expected sum of discounted rewards starting in a state or starting in a
[00:04:03] starting in a state or starting in a particular state in action and so we
[00:04:06] particular state in action and so we talked a lot about value functions and Q
[00:04:08] talked a lot about value functions and Q functions um and now we're going to talk
[00:04:10] functions um and now we're going to talk a lot about policy
[00:04:12] a lot about policy search and so we're going to still think
[00:04:15] search and so we're going to still think about there being this policy which is a
[00:04:17] about there being this policy which is a mapping of states to actions or a
[00:04:20] mapping of states to actions or a mapping from States and actions to a
[00:04:24] mapping from States and actions to a number between zero and one such that it
[00:04:26] number between zero and one such that it sums to one we always have to do at
[00:04:28] sums to one we always have to do at least one action in every state but we
[00:04:31] least one action in every state but we don't necessarily have to have an
[00:04:32] don't necessarily have to have an explicit representation of the value
[00:04:34] explicit representation of the value function
[00:04:35] function anymore so these have been very popular
[00:04:40] anymore so these have been very popular and important and if we think back to
[00:04:43] and important and if we think back to what ourl algorithms involve they
[00:04:45] what ourl algorithms involve they involve optimization delayed
[00:04:47] involve optimization delayed consequences exploration and
[00:04:49] consequences exploration and generalization and we've seen examples
[00:04:51] generalization and we've seen examples of the all of these ideas so far and
[00:04:53] of the all of these ideas so far and we'll go a lot more into some of them as
[00:04:55] we'll go a lot more into some of them as we go through the course but one thing
[00:04:57] we go through the course but one thing you might be wondering about is you know
[00:05:00] you might be wondering about is you know could we play the trick that's often
[00:05:01] could we play the trick that's often done in U computer science and try to
[00:05:03] done in U computer science and try to reduce reinforcement learning to another
[00:05:05] reduce reinforcement learning to another problem so could we do something like
[00:05:07] problem so could we do something like just like online optimization so we know
[00:05:10] just like online optimization so we know that we uh don't know how the world
[00:05:12] that we uh don't know how the world works and we're trying to find a good
[00:05:14] works and we're trying to find a good control policy but could we do something
[00:05:16] control policy but could we do something like sort of um online optimization
[00:05:19] like sort of um online optimization where we're trying to search for a good
[00:05:21] where we're trying to search for a good policy and in this way you can give
[00:05:24] policy and in this way you can give policy gradient being related at a high
[00:05:25] policy gradient being related at a high level to this type of idea it's not a
[00:05:27] level to this type of idea it's not a reduction based approach but it's sort
[00:05:29] reduction based approach but it's sort of thinking about sort of well can we
[00:05:31] of thinking about sort of well can we just directly search to find a good
[00:05:33] just directly search to find a good policy and policy gr methods have been
[00:05:35] policy and policy gr methods have been extremely influential particularly over
[00:05:37] extremely influential particularly over the last 5 to 10 years so they're used
[00:05:39] the last 5 to 10 years so they're used for lots of areas they're used for
[00:05:41] for lots of areas they're used for things like you know sequence level
[00:05:42] things like you know sequence level training with recurrent roal networks
[00:05:44] training with recurrent roal networks that was based on reinforce which is an
[00:05:46] that was based on reinforce which is an algorithm we're going to go through
[00:05:47] algorithm we're going to go through today um it has been used for things of
[00:05:50] today um it has been used for things of end to-end training of deep Vis Muto
[00:05:53] end to-end training of deep Vis Muto policies so this was really influential
[00:05:55] policies so this was really influential work in the robotics Community about a
[00:05:57] work in the robotics Community about a decade ago I'm just going to show you a
[00:05:59] decade ago I'm just going to show you a quick quick video of
[00:06:00] quick quick video of it so this is work that was done by
[00:06:03] it so this is work that was done by Professor Chelsea Finn as part of her
[00:06:05] Professor Chelsea Finn as part of her PhD thesis along with Sergey LaVine and
[00:06:07] PhD thesis along with Sergey LaVine and others at
[00:06:11] Berkeley see if this will work with
[00:06:28] audio so what you can see there that
[00:06:30] audio so what you can see there that what they're
[00:06:32] what they're doing is what they're going to be trying
[00:06:34] doing is what they're going to be trying to do is learn from like so they showed
[00:06:36] to do is learn from like so they showed you a really they showed a big Network
[00:06:38] you a really they showed a big Network and what they're trying to go is go
[00:06:40] and what they're trying to go is go directly from pixels to learn what the
[00:06:43] directly from pixels to learn what the robot should do and this is one of the
[00:06:45] robot should do and this is one of the first examples of people trying to do
[00:06:47] first examples of people trying to do this directly from images let's just go
[00:06:49] this directly from images let's just go back to some of the tasks that they're
[00:06:50] back to some of the tasks that they're using
[00:07:18] and so that was part of the motivation
[00:07:20] and so that was part of the motivation too is that they want to be able to
[00:07:22] too is that they want to be able to learn these tasks in way that will
[00:07:23] learn these tasks in way that will generalize
[00:07:39] so this is another example of sort of
[00:07:41] so this is another example of sort of trying to do direct policy gradient
[00:07:43] trying to do direct policy gradient methods um in order to go from like
[00:07:47] methods um in order to go from like really um large complex State spaces
[00:07:50] really um large complex State spaces into direct decisions uh now in homework
[00:07:54] into direct decisions uh now in homework two and we haven't covered po yet but
[00:07:56] two and we haven't covered po yet but you're going to be implementing proximal
[00:07:58] you're going to be implementing proximal policy optimization which is one of the
[00:08:00] policy optimization which is one of the methods that build on the sort of
[00:08:01] methods that build on the sort of methods that we're going to talk about
[00:08:02] methods that we're going to talk about today um and that was used PO was used
[00:08:05] today um and that was used PO was used as part of training chat GPT so as you
[00:08:07] as part of training chat GPT so as you can see all of these algorithms have
[00:08:09] can see all of these algorithms have become incredibly influential in part
[00:08:11] become incredibly influential in part because they can scale really well to
[00:08:13] because they can scale really well to extremely complex inputs whether it be
[00:08:14] extremely complex inputs whether it be images or high dimensional robotic tasks
[00:08:17] images or high dimensional robotic tasks or even things like natural language and
[00:08:19] or even things like natural language and so they're very powerful they're often
[00:08:22] so they're very powerful they're often used in conjunction with things like
[00:08:24] used in conjunction with things like State action values as we'll talk about
[00:08:26] State action values as we'll talk about later but you don't have to use them
[00:08:28] later but you don't have to use them with them so they're really useful sort
[00:08:30] with them so they're really useful sort of class of things to know
[00:08:32] of class of things to know about so in particular just like how
[00:08:35] about so in particular just like how last time we saw that you could
[00:08:37] last time we saw that you could approximate a state action value or a
[00:08:40] approximate a state action value or a value function with a set of parameters
[00:08:42] value function with a set of parameters so we can do function
[00:08:43] so we can do function approximation um in those cases we
[00:08:47] approximation um in those cases we thought of directly learning a value
[00:08:49] thought of directly learning a value function or a state action value
[00:08:50] function or a state action value function and then it generate a policy
[00:08:53] function and then it generate a policy from the state action value so something
[00:08:55] from the state action value so something like EG greedy where we either take what
[00:08:57] like EG greedy where we either take what the Q value suggests as the best action
[00:08:59] the Q value suggests as the best action or weact randomly and what we're going
[00:09:01] or weact randomly and what we're going to do today instead and I'll try to be
[00:09:04] to do today instead and I'll try to be careful about not using the same um we
[00:09:06] careful about not using the same um we used W before to parameterize our state
[00:09:08] used W before to parameterize our state action values and I'm going to try to be
[00:09:10] action values and I'm going to try to be careful about using Theta just to make
[00:09:11] careful about using Theta just to make it clear um we're going to directly
[00:09:13] it clear um we're going to directly parameterize the policy and we're going
[00:09:16] parameterize the policy and we're going to try to learn parameterized policies
[00:09:18] to try to learn parameterized policies so we can think of these as like you
[00:09:19] so we can think of these as like you know deep convolutional neural networks
[00:09:21] know deep convolutional neural networks which at the end will output um either
[00:09:24] which at the end will output um either an action or if we have an action as
[00:09:26] an action or if we have an action as input we'll output a
[00:09:27] input we'll output a probability and the goal in this case as
[00:09:30] probability and the goal in this case as is normal is that we want to find a way
[00:09:32] is normal is that we want to find a way to act in the world that will give us
[00:09:33] to act in the world that will give us High reward so we want to find a policy
[00:09:35] High reward so we want to find a policy with the highest value function V pi and
[00:09:38] with the highest value function V pi and we're again not going to be focusing on
[00:09:40] we're again not going to be focusing on model based learning so we're so going
[00:09:43] model based learning so we're so going to try to directly learn from experience
[00:09:44] to try to directly learn from experience we're not going to assume we have access
[00:09:46] we're not going to assume we have access or that we're explicitly building a
[00:09:50] model and I think one of the things
[00:09:52] model and I think one of the things that's helpful to think about is there's
[00:09:54] that's helpful to think about is there's these sort of different views or lenses
[00:09:55] these sort of different views or lenses into reinforcement learning so this is a
[00:09:57] into reinforcement learning so this is a nice picture from David silver who's an
[00:09:59] nice picture from David silver who's an amazing person in reinforcement learning
[00:10:01] amazing person in reinforcement learning he was you know one of the main leads on
[00:10:03] he was you know one of the main leads on alphao and a number of other incredible
[00:10:05] alphao and a number of other incredible papers so we can think of it is like you
[00:10:08] papers so we can think of it is like you have some methods which are value based
[00:10:09] have some methods which are value based we're explicitly building a value
[00:10:11] we're explicitly building a value function we have other ones that are
[00:10:12] function we have other ones that are policy based and the ones that are in
[00:10:14] policy based and the ones that are in the intersection are often known as
[00:10:16] the intersection are often known as actor critic methods who here has either
[00:10:19] actor critic methods who here has either implemented or heard of actor critic
[00:10:20] implemented or heard of actor critic methods before okay so some people and
[00:10:23] methods before okay so some people and underground they're extremely popular
[00:10:25] underground they're extremely popular and in actor critic methods um you will
[00:10:27] and in actor critic methods um you will often combine between the benefit it's a
[00:10:29] often combine between the benefit it's a value based and policy and so for
[00:10:32] value based and policy and so for example alphago is a actor critic method
[00:10:35] example alphago is a actor critic method in the sense that it is often having
[00:10:37] in the sense that it is often having explicit representation of the policy
[00:10:38] explicit representation of the policy and of a value
[00:10:41] and of a value function okay so we'll get to actor
[00:10:43] function okay so we'll get to actor critic methods later today we're going
[00:10:45] critic methods later today we're going to focus on policy
[00:10:48] based so now that we're going to most of
[00:10:51] based so now that we're going to most of the time we've thought about policies so
[00:10:53] the time we've thought about policies so far we thought about deterministic
[00:10:54] far we thought about deterministic policies or egedy policies and now we're
[00:10:57] policies or egedy policies and now we're going to think much more generally about
[00:10:58] going to think much more generally about stochastic policies um and that's going
[00:11:02] stochastic policies um and that's going to be important because as we saw last
[00:11:04] to be important because as we saw last time if you only have a deterministic
[00:11:06] time if you only have a deterministic policy it's much harder to learn about
[00:11:07] policy it's much harder to learn about actions you don't try whereas now we're
[00:11:10] actions you don't try whereas now we're going to think about having stochastic
[00:11:11] going to think about having stochastic policies where you're going to be
[00:11:12] policies where you're going to be getting information about lots of
[00:11:14] getting information about lots of different
[00:11:15] different actions so let's think about a
[00:11:17] actions so let's think about a particular example also to kind of
[00:11:19] particular example also to kind of illustrate some of the things that
[00:11:21] illustrate some of the things that policy gradiant methods are going to
[00:11:22] policy gradiant methods are going to help us handle so who here has played
[00:11:26] help us handle so who here has played rock paper
[00:11:27] rock paper scissors okay most people I think it's
[00:11:29] scissors okay most people I think it's called Rambo in um Chinese it's a very
[00:11:32] called Rambo in um Chinese it's a very popular game throughout the world it's a
[00:11:34] popular game throughout the world it's a stochastic game where you know or what
[00:11:38] stochastic game where you know or what each side can pick um a particular
[00:11:40] each side can pick um a particular strategy and the state you could think
[00:11:43] strategy and the state you could think of there being a state you could keep
[00:11:45] of there being a state you could keep track of what your opponent has done
[00:11:46] track of what your opponent has done over time so think for a second about
[00:11:49] over time so think for a second about whether the um a deterministic policy
[00:11:51] whether the um a deterministic policy can be optimal if you're playing this
[00:11:53] can be optimal if you're playing this game
[00:11:56] repeatedly so raise your hand if a
[00:11:58] repeatedly so raise your hand if a deterministic policy can be
[00:12:01] deterministic policy can be optimal raise your hand if you think a
[00:12:03] optimal raise your hand if you think a stochastic policy is
[00:12:05] stochastic policy is optimal okay someone who said stochastic
[00:12:08] optimal okay someone who said stochastic explain
[00:12:11] why
[00:12:14] why yes two is like circular like there's no
[00:12:17] yes two is like circular like there's no one one best one that that's right yeah
[00:12:21] one one best one that that's right yeah so there's no best like there's nothing
[00:12:22] so there's no best like there's nothing that strictly dominates all the other
[00:12:24] that strictly dominates all the other strategies and also if you're
[00:12:25] strategies and also if you're deterministic what can your opponent do
[00:12:29] deterministic what can your opponent do like if I say I'm always going to pick
[00:12:31] like if I say I'm always going to pick um paper what does my opponent
[00:12:34] um paper what does my opponent do they're always gonna pick like the
[00:12:38] do they're always gonna pick like the other one like rock to to beat me so
[00:12:40] other one like rock to to beat me so anything you do that's deterministic can
[00:12:42] anything you do that's deterministic can be exploited by your opponent if you are
[00:12:44] be exploited by your opponent if you are playing repeatedly and so the optimal
[00:12:47] playing repeatedly and so the optimal thing to do here is to be sarcastic the
[00:12:49] thing to do here is to be sarcastic the optimal policy has to be sarcastic here
[00:12:51] optimal policy has to be sarcastic here otherwise all deterministic policies are
[00:12:54] otherwise all deterministic policies are strictly dominated by good stochastic
[00:12:57] strictly dominated by good stochastic policy okay and now you might think well
[00:13:00] policy okay and now you might think well all right well that sounds different
[00:13:01] all right well that sounds different than what we've seen so far but one of
[00:13:03] than what we've seen so far but one of the challenges here is the system is not
[00:13:05] the challenges here is the system is not Markoff um so it's not stochastic what
[00:13:09] Markoff um so it's not stochastic what your um adversary will play next they're
[00:13:11] your um adversary will play next they're not random or it might be if they're
[00:13:12] not random or it might be if they're playing stochastic policy excuse me but
[00:13:14] playing stochastic policy excuse me but in general the sto that they can react
[00:13:16] in general the sto that they can react to What You've seen so far um and it's
[00:13:18] to What You've seen so far um and it's not just like you know a random
[00:13:20] not just like you know a random environment like a coin flip on the next
[00:13:21] environment like a coin flip on the next time and in this case actually a uniform
[00:13:24] time and in this case actually a uniform random policy is optimal it's a Nash
[00:13:27] random policy is optimal it's a Nash equilibrium right
[00:13:29] equilibrium right okay so that's one case where having a
[00:13:32] okay so that's one case where having a stochastic policy would be really
[00:13:33] stochastic policy would be really helpful so you could just have a fixed
[00:13:35] helpful so you could just have a fixed stochastic policy and it would be
[00:13:37] stochastic policy and it would be optimal but you couldn't necessar write
[00:13:39] optimal but you couldn't necessar write this down easily as a q function and
[00:13:41] this down easily as a q function and just take the ARX there's not a
[00:13:42] just take the ARX there's not a deterministic policy for this
[00:13:44] deterministic policy for this environment that is optimal and so it's
[00:13:47] environment that is optimal and so it's less clear how you would write that down
[00:13:49] less clear how you would write that down directly in terms of a q function in
[00:13:51] directly in terms of a q function in part because the system do not mark
[00:13:53] part because the system do not mark off so here's another example where we
[00:13:56] off so here's another example where we might want to have stochastic policies
[00:13:57] might want to have stochastic policies and it's where we have alias or partial
[00:14:00] and it's where we have alias or partial observability so imagine this case where
[00:14:02] observability so imagine this case where like you have a robot that's walking
[00:14:03] like you have a robot that's walking along and you know maybe they have
[00:14:05] along and you know maybe they have sensors so they can tell how far they
[00:14:07] sensors so they can tell how far they are from the walls but under Those
[00:14:09] are from the walls but under Those sensors these two gray boxes look
[00:14:12] sensors these two gray boxes look identical because like from the agents
[00:14:15] identical because like from the agents point of view if they have only
[00:14:17] point of view if they have only immediate sensors both of those places
[00:14:18] immediate sensors both of those places will look identical and so they can't
[00:14:21] will look identical and so they can't distinguish those gray States and
[00:14:23] distinguish those gray States and imagine that um you know you just
[00:14:25] imagine that um you know you just because you have a feature
[00:14:26] because you have a feature representation that just tells you about
[00:14:28] representation that just tells you about what your what you have a wall to the
[00:14:30] what your what you have a wall to the north to the east to the South or to the
[00:14:31] north to the east to the South or to the West so those two gr States would look
[00:14:33] West so those two gr States would look identical if that was your feature
[00:14:35] identical if that was your feature representation so you could have a value
[00:14:38] representation so you could have a value based reinforcement learning
[00:14:39] based reinforcement learning representation where use an approximate
[00:14:41] representation where use an approximate value function where you take in this as
[00:14:44] value function where you take in this as the state
[00:14:45] the state representation or you could have a
[00:14:46] representation or you could have a policy based one that takes in
[00:14:50] policy based one that takes in those so the challenge here is that if
[00:14:53] those so the challenge here is that if you're value based you have to do the
[00:14:55] you're value based you have to do the same thing in those two gra States
[00:14:58] same thing in those two gra States because you you can't distinguish them
[00:15:00] because you you can't distinguish them so from your perspective it's like
[00:15:02] so from your perspective it's like you're in the same place no matter which
[00:15:03] you're in the same place no matter which of those two places you're in so if
[00:15:05] of those two places you're in so if you're going to do a value function
[00:15:07] you're going to do a value function based and then extract a deterministic
[00:15:09] based and then extract a deterministic policy you would either always have to
[00:15:12] policy you would either always have to go say to the left in those cases or
[00:15:14] go say to the left in those cases or always go to the
[00:15:16] always go to the right and neither of those would always
[00:15:20] right and neither of those would always be
[00:15:22] be good
[00:15:24] good okay so under alosine meaning that we
[00:15:27] okay so under alosine meaning that we don't know whether which of the two GR
[00:15:29] don't know whether which of the two GR States we're in when we're in one of
[00:15:30] States we're in when we're in one of them ad opal policy will always move
[00:15:32] them ad opal policy will always move west in both States or east in both
[00:15:34] west in both States or east in both States and either way it might get stuck
[00:15:36] States and either way it might get stuck and never be able to reach the money but
[00:15:40] and never be able to reach the money but um and that's what's going to happen if
[00:15:41] um and that's what's going to happen if we do a value based reinforcement
[00:15:42] we do a value based reinforcement learning approach so that's not great
[00:15:45] learning approach so that's not great you're going to you know Traverse this
[00:15:46] you're going to you know Traverse this for a long time you're not going to be
[00:15:47] for a long time you're not going to be getting high reward what could you do if
[00:15:50] getting high reward what could you do if you wanted to have a stochastic
[00:15:56] policy so that allows you to act
[00:15:59] policy so that allows you to act R you know randomly or you know
[00:16:00] R you know randomly or you know stochastically in any state what do you
[00:16:02] stochastically in any state what do you think would be the right thing to do in
[00:16:03] think would be the right thing to do in the gray States if you could have a
[00:16:05] the gray States if you could have a stochastic
[00:16:08] policy uh with just some probability you
[00:16:11] policy uh with just some probability you go either East or yeah exactly so um so
[00:16:15] go either East or yeah exactly so um so you could just randomize it so an
[00:16:17] you could just randomize it so an optimal sastic policy will randomly move
[00:16:20] optimal sastic policy will randomly move East or West in The Gray State because
[00:16:22] East or West in The Gray State because it doesn't know which one it's in and
[00:16:24] it doesn't know which one it's in and half the time that'll be the right thing
[00:16:25] half the time that'll be the right thing to do so um so that means much more the
[00:16:29] to do so um so that means much more the time it'll go into here and it generally
[00:16:31] time it'll go into here and it generally will reach the goal State pretty
[00:16:34] will reach the goal State pretty quickly so this is another case where
[00:16:36] quickly so this is another case where the system is not markof this is notk
[00:16:42] the system is not markof this is notk off and the state
[00:16:44] off and the state features so because we have alosine
[00:16:47] features so because we have alosine meaning the system is partially
[00:16:48] meaning the system is partially observable is not a markup system one
[00:16:51] observable is not a markup system one way to handle that is to treat it as a
[00:16:53] way to handle that is to treat it as a partially observable markof decision
[00:16:54] partially observable markof decision process Michael Cocker talks a lot about
[00:16:56] process Michael Cocker talks a lot about those in his classes um but an
[00:16:59] those in his classes um but an alternative is to use a stochastic
[00:17:01] alternative is to use a stochastic policy and you can also act very well
[00:17:04] policy and you can also act very well here so those are two examples of sort
[00:17:07] here so those are two examples of sort of the type of thing that might be able
[00:17:09] of the type of thing that might be able to EAS be easy to handle with policy
[00:17:11] to EAS be easy to handle with policy gradient methods or stochastic policies
[00:17:13] gradient methods or stochastic policies that might be hard to um tackle with the
[00:17:16] that might be hard to um tackle with the type of methods we've seen so far okay
[00:17:18] type of methods we've seen so far okay so now we have to think about if we have
[00:17:20] so now we have to think about if we have policies and in general we're going to
[00:17:21] policies and in general we're going to want them to be stochastic how are we
[00:17:23] want them to be stochastic how are we going to learn what are good policies
[00:17:25] going to learn what are good policies like we have this you know now we have a
[00:17:27] like we have this you know now we have a function space over policy and we want
[00:17:29] function space over policy and we want to learn which of them have good
[00:17:32] to learn which of them have good values so if we're in an episodic
[00:17:35] values so if we're in an episodic environment we can use the policy value
[00:17:37] environment we can use the policy value at the start state so we can just say
[00:17:39] at the start state so we can just say I'm going to similar to the Monte Carlo
[00:17:41] I'm going to similar to the Monte Carlo methods if I start in this state I run
[00:17:42] methods if I start in this state I run this policy what would be my expected
[00:17:44] this policy what would be my expected reward be until the end of the episode
[00:17:47] reward be until the end of the episode we're going to mostly focus on the
[00:17:48] we're going to mostly focus on the episodic case today um but you can
[00:17:51] episodic case today um but you can extend these to sort of more of an
[00:17:52] extend these to sort of more of an infinite Horizon
[00:17:55] infinite Horizon case all right so once we think of it in
[00:17:57] case all right so once we think of it in this way we can really think okay this
[00:17:59] this way we can really think okay this sounds like an optimization problem so
[00:18:02] sounds like an optimization problem so we really just want to find um the
[00:18:05] we really just want to find um the parameters that maximize the value so
[00:18:07] parameters that maximize the value so you could
[00:18:08] you could say here you can think of this as being
[00:18:10] say here you can think of this as being your thetas so I'm just draw it one
[00:18:12] your thetas so I'm just draw it one dimensional but in general this could be
[00:18:14] dimensional but in general this could be you know all the parameters in a deep
[00:18:15] you know all the parameters in a deep neural network and then this is V of
[00:18:18] neural network and then this is V of theta or of a particular starting state
[00:18:22] theta or of a particular starting state it might look like
[00:18:24] it might look like this and what your goal would be is to
[00:18:27] this and what your goal would be is to find the parameters of your policy that
[00:18:31] find the parameters of your policy that maximize the value
[00:18:33] maximize the value function and so this is an optimization
[00:18:36] function and so this is an optimization problem but it's a hard optimization
[00:18:38] problem but it's a hard optimization problem because we don't have that
[00:18:39] problem because we don't have that function you can only estimate it
[00:18:41] function you can only estimate it through
[00:18:42] through data and so you can imagine like you
[00:18:44] data and so you can imagine like you start off you have no idea how how Theta
[00:18:47] start off you have no idea how how Theta maps to be then you have to learn that
[00:18:48] maps to be then you have to learn that over
[00:18:53] time so once we think of it as an
[00:18:55] time so once we think of it as an optimization problem where we don't know
[00:18:57] optimization problem where we don't know what the function is there are a lot of
[00:18:59] what the function is there are a lot of different methods we could think about
[00:19:00] different methods we could think about to try to solve this problem and what
[00:19:02] to try to solve this problem and what we're going to focus on today mostly is
[00:19:04] we're going to focus on today mostly is ones that are going to exploit something
[00:19:06] ones that are going to exploit something about the structure of sequential
[00:19:08] about the structure of sequential decision
[00:19:09] decision processes but there are methods that
[00:19:11] processes but there are methods that completely ignore kind of all of this um
[00:19:15] completely ignore kind of all of this um and in particular you can even use
[00:19:17] and in particular you can even use things that completely ignore gradients
[00:19:18] things that completely ignore gradients so you can do things like hill climbing
[00:19:20] so you can do things like hill climbing or genetic algorithms or cross entropy
[00:19:23] or genetic algorithms or cross entropy methods where you may not think of sort
[00:19:25] methods where you may not think of sort of any of the type of structure of the
[00:19:26] of any of the type of structure of the parameter space and in some cases that
[00:19:29] parameter space and in some cases that can work really
[00:19:31] can work really well so there's a really nice example um
[00:19:34] well so there's a really nice example um by my colleague uh step Collins who's
[00:19:37] by my colleague uh step Collins who's over in the mechanical engineering
[00:19:38] over in the mechanical engineering department he does some really
[00:19:40] department he does some really interesting work on
[00:19:50] yeah yes but you can it's a you can make
[00:19:53] yeah yes but you can it's a you can make it a distribution over actions so you
[00:19:55] it a distribution over actions so you can you can output something between
[00:19:57] can you can output something between zero and one you can have input action
[00:19:59] zero and one you can have input action and so then you would have a stochastic
[00:20:02] and so then you would have a stochastic policy out the netork you would then
[00:20:05] policy out the netork you would then compute up for all the actions then you
[00:20:06] compute up for all the actions then you would have to have some yeah you'd have
[00:20:07] would have to have some yeah you'd have to pick a random number and then use
[00:20:09] to pick a random number and then use that to select yeah yeah good
[00:20:14] question so my colleagu uh Stephen
[00:20:16] question so my colleagu uh Stephen Collings over in mechanical engineering
[00:20:18] Collings over in mechanical engineering does a lot of work on exoskeletons there
[00:20:20] does a lot of work on exoskeletons there are lots of reasons exoskeletons could
[00:20:21] are lots of reasons exoskeletons could be really helpful particularly for
[00:20:23] be really helpful particularly for people that have motor impairments but
[00:20:25] people that have motor impairments but one of the challenges of them is that uh
[00:20:27] one of the challenges of them is that uh you have to them actually help people
[00:20:29] you have to them actually help people walk so um if you clamp something on to
[00:20:32] walk so um if you clamp something on to say your leg then the way my physical
[00:20:35] say your leg then the way my physical configuration is may not be the same as
[00:20:37] configuration is may not be the same as your physical configuration I'm pretty
[00:20:38] your physical configuration I'm pretty tall and so you'd really like to make
[00:20:40] tall and so you'd really like to make sure that this helps each individual in
[00:20:42] sure that this helps each individual in the best way possible but you don't want
[00:20:44] the best way possible but you don't want it to have to learn over the course of
[00:20:46] it to have to learn over the course of like you know two years how to best
[00:20:48] like you know two years how to best optimize to someone because they're not
[00:20:50] optimize to someone because they're not going to wait that long to use it so
[00:20:52] going to wait that long to use it so what they did is they use policy methods
[00:20:54] what they did is they use policy methods policy search methods to quickly
[00:20:57] policy search methods to quickly personalize the parameters of an
[00:20:59] personalize the parameters of an exoskeleton um and they called this
[00:21:00] exoskeleton um and they called this humanin the loop exoskeleton
[00:21:02] humanin the loop exoskeleton optimization so the idea in this case is
[00:21:04] optimization so the idea in this case is what they're trying to figure out is
[00:21:06] what they're trying to figure out is what is the parameters of their
[00:21:08] what is the parameters of their exoskeleton and what they're looking at
[00:21:10] exoskeleton and what they're looking at is essentially how much it helps you
[00:21:12] is essentially how much it helps you walk so how much it reduces the effort
[00:21:14] walk so how much it reduces the effort needed to walk and so what they could do
[00:21:16] needed to walk and so what they could do in this case they're not using like a
[00:21:18] in this case they're not using like a gradient bath method they're using just
[00:21:20] gradient bath method they're using just CES which is so they train is of
[00:21:23] CES which is so they train is of continuous optimization is they'd have
[00:21:25] continuous optimization is they'd have people walk under sort of a few
[00:21:26] people walk under sort of a few different control parameters
[00:21:29] different control parameters they would see which of those seem to be
[00:21:30] they would see which of those seem to be most effective and then they would move
[00:21:32] most effective and then they would move the policies they' try in that direction
[00:21:34] the policies they' try in that direction with some
[00:21:35] with some stochasticity and I think it was within
[00:21:37] stochasticity and I think it was within maybe two or 3 hours using this they
[00:21:40] maybe two or 3 hours using this they could find substantially better policies
[00:21:42] could find substantially better policies I think it increased metabolic
[00:21:44] I think it increased metabolic efficiency like maybe by 20% or 30% it
[00:21:47] efficiency like maybe by 20% or 30% it was pretty remarkable and so this was
[00:21:49] was pretty remarkable and so this was published in science about um seven
[00:21:50] published in science about um seven eight years ago but that's another
[00:21:52] eight years ago but that's another example of a place where you can do this
[00:21:54] example of a place where you can do this sort of like online optimization you
[00:21:56] sort of like online optimization you don't necess have to think about the
[00:21:57] don't necess have to think about the temporal structure of the
[00:22:01] policy
[00:22:03] policy yeah start with a default policy and
[00:22:07] yeah start with a default policy and then try to improve that before great
[00:22:09] then try to improve that before great question yeah so in all of these cases
[00:22:11] question yeah so in all of these cases we're going to have to assume that we
[00:22:12] we're going to have to assume that we initialize our policy parameterization
[00:22:14] initialize our policy parameterization in some way just like how we initialized
[00:22:16] in some way just like how we initialized our value function to zero to start now
[00:22:18] our value function to zero to start now we're going to or if you had it for deep
[00:22:20] we're going to or if you had it for deep uh for the Deep Q Network it would be
[00:22:22] uh for the Deep Q Network it would be whatever your neur network
[00:22:23] whatever your neur network parameters yeah great question
[00:22:29] now it's just useful to to know about
[00:22:31] now it's just useful to to know about these because um they often work pretty
[00:22:33] these because um they often work pretty well so I think sometimes we like to
[00:22:36] well so I think sometimes we like to leverage the structure specific to say
[00:22:38] leverage the structure specific to say our markof decision process but in some
[00:22:40] our markof decision process but in some cases just leveraging these ones which
[00:22:42] cases just leveraging these ones which may not use very much structure at all
[00:22:44] may not use very much structure at all can actually do really well so it's just
[00:22:47] can actually do really well so it's just good to keep in mind that there are a
[00:22:48] good to keep in mind that there are a lot of ways to do online
[00:22:51] lot of ways to do online optimization all right so this is often
[00:22:53] optimization all right so this is often a great Baseline to try the great thing
[00:22:55] a great Baseline to try the great thing about this is it can work with any
[00:22:56] about this is it can work with any policy parameterizations even if it's
[00:22:58] policy parameterizations even if it's not differentiable because it's not
[00:23:00] not differentiable because it's not using gradients so it doesn't need to be
[00:23:02] using gradients so it doesn't need to be differential and it's also often very
[00:23:05] differential and it's also often very easy to paralyze so CES for those of you
[00:23:08] easy to paralyze so CES for those of you who not who haven't seen it before um
[00:23:11] who not who haven't seen it before um you'll have a number of different
[00:23:12] you'll have a number of different policies you kind of try in parallel and
[00:23:14] policies you kind of try in parallel and then you'll use that to sort of update
[00:23:16] then you'll use that to sort of update and shift to another set of policies and
[00:23:18] and shift to another set of policies and that's what they did uh Professor
[00:23:20] that's what they did uh Professor Collings did and in a lot of cases the
[00:23:23] Collings did and in a lot of cases the problems that we think about are places
[00:23:24] problems that we think about are places where you know you'll have many
[00:23:25] where you know you'll have many customers you'll have many many robot
[00:23:27] customers you'll have many many robot arms and so you can par
[00:23:29] arms and so you can par things one of the limitations is that
[00:23:31] things one of the limitations is that it's often less data efficient because
[00:23:32] it's often less data efficient because it's ignoring the temporal structure so
[00:23:35] it's ignoring the temporal structure so if if you have temporal structure or you
[00:23:36] if if you have temporal structure or you have a gradient gradient information it
[00:23:38] have a gradient gradient information it may be more effective to use
[00:23:41] may be more effective to use that all right so what we're going to
[00:23:44] that all right so what we're going to focus on in this class is differentiable
[00:23:47] focus on in this class is differentiable methods so we're going to focus on
[00:23:48] methods so we're going to focus on places where we can do stochastic
[00:23:51] places where we can do stochastic gradient descent including on the policy
[00:23:54] gradient descent including on the policy parameterization so if we have like our
[00:23:56] parameterization so if we have like our policy parameterized by a deep neural
[00:23:58] policy parameterized by a deep neural network we can propagate through that
[00:24:00] network we can propagate through that and update it update those
[00:24:03] and update it update those parameters so we're going to um we're
[00:24:06] parameters so we're going to um we're going to focus here mostly on methods
[00:24:08] going to focus here mostly on methods that do use gradient descent and that
[00:24:11] that do use gradient descent and that often Leverage The sequential structure
[00:24:12] often Leverage The sequential structure of that we're making a sequence of
[00:24:14] of that we're making a sequence of decisions and we want to optimize to
[00:24:16] decisions and we want to optimize to make those sequence of
[00:24:19] decisions so to do that we're going to
[00:24:21] decisions so to do that we're going to explicitly Define the gradient and we're
[00:24:23] explicitly Define the gradient and we're going to write down the value function
[00:24:25] going to write down the value function in terms of as a function of the policy
[00:24:27] in terms of as a function of the policy parameters so that we can be
[00:24:29] parameters so that we can be clear that this value function um you
[00:24:33] clear that this value function um you know relies on those policies and we're
[00:24:36] know relies on those policies and we're going to focus today on episodic markof
[00:24:38] going to focus today on episodic markof decision processes where we go for a
[00:24:40] decision processes where we go for a single episode stop reset and keep
[00:24:43] single episode stop reset and keep going
[00:24:45] going right so now what we're going to do is
[00:24:48] right so now what we're going to do is we're only going to be trying to get in
[00:24:49] we're only going to be trying to get in general to a local maximum now it's
[00:24:53] general to a local maximum now it's possible you're lucky and you're sort of
[00:24:55] possible you're lucky and you're sort of convex in the in the space of the the
[00:24:58] convex in the in the space of the the value it's bace of the policy parameters
[00:25:00] value it's bace of the policy parameters but in general we're not going to assume
[00:25:02] but in general we're not going to assume convexity so at best we're going to hope
[00:25:04] convexity so at best we're going to hope to just get to some sort of local Maxima
[00:25:06] to just get to some sort of local Maxima in our
[00:25:07] in our space so if we have again if we only had
[00:25:11] space so if we have again if we only had one parameter and we have something like
[00:25:14] one parameter and we have something like this where we might get to here we might
[00:25:17] this where we might get to here we might get to here in general we're not going
[00:25:18] get to here in general we're not going to make Global optimality guarantees
[00:25:21] to make Global optimality guarantees this is in big cont been contrast to the
[00:25:24] this is in big cont been contrast to the tabular cases we saw before we were
[00:25:25] tabular cases we saw before we were guaranteed to get to the optimal Q
[00:25:27] guaranteed to get to the optimal Q function optimal value function now
[00:25:30] function optimal value function now we're just going to hope to given our
[00:25:32] we're just going to hope to given our policy parameterization let's try to get
[00:25:35] policy parameterization let's try to get to What's um a local Optima in that
[00:25:38] to What's um a local Optima in that policy
[00:25:40] policy parameterization so it's sort of a
[00:25:41] parameterization so it's sort of a policy specific policy class specific
[00:25:44] policy specific policy class specific guarantee and it's only a local Optima
[00:25:47] guarantee and it's only a local Optima and what we'll be doing is we're just
[00:25:48] and what we'll be doing is we're just going to be trying to sort of take the
[00:25:49] going to be trying to sort of take the gradient of the policy with respect to
[00:25:51] gradient of the policy with respect to the
[00:25:52] the parameters okay and as usual we're going
[00:25:54] parameters okay and as usual we're going to have a step size parameter so we're
[00:25:56] to have a step size parameter so we're going to take the gradient of the value
[00:25:58] going to take the gradient of the value fun with respect to the parameters and
[00:25:59] fun with respect to the parameters and take a small step and the key thing is
[00:26:02] take a small step and the key thing is going to be thinking about places where
[00:26:03] going to be thinking about places where we can do this All directly using sort
[00:26:06] we can do this All directly using sort of smooth
[00:26:10] functions okay now one way you could do
[00:26:13] functions okay now one way you could do this of course when you see this now you
[00:26:15] this of course when you see this now you probably immediately think of Auto diff
[00:26:16] probably immediately think of Auto diff methods and think about we can just back
[00:26:18] methods and think about we can just back parate Etc but it's worth noting that
[00:26:20] parate Etc but it's worth noting that when these methods began to start to get
[00:26:23] when these methods began to start to get popular um they didn't necessarily have
[00:26:25] popular um they didn't necessarily have Auto diff yet uh I know they there was
[00:26:28] Auto diff yet uh I know they there was research then still um and what one of
[00:26:31] research then still um and what one of the things people started thinking about
[00:26:32] the things people started thinking about for this is how you could use this for
[00:26:34] for this is how you could use this for robotics so this is a nice paper from
[00:26:36] robotics so this is a nice paper from 2004 so 20 years ago by Peter Stones
[00:26:39] 2004 so 20 years ago by Peter Stones group it was right around then I think
[00:26:41] group it was right around then I think maybe RoboCop was maybe 6 years old then
[00:26:43] maybe RoboCop was maybe 6 years old then or something um I think they started
[00:26:45] or something um I think they started back in 1998 or so so they're these
[00:26:47] back in 1998 or so so they're these little quadruped robots and the goal was
[00:26:49] little quadruped robots and the goal was to think about having getting robotics
[00:26:51] to think about having getting robotics to the stage where you could have robots
[00:26:54] to the stage where you could have robots play human players I think that was the
[00:26:56] play human players I think that was the goal by either 2030 or 2050 I forget but
[00:26:59] goal by either 2030 or 2050 I forget but what they was going to start with it was
[00:27:00] what they was going to start with it was quadrupeds so one of the big challenges
[00:27:03] quadrupeds so one of the big challenges at the beginning because you know
[00:27:04] at the beginning because you know everywhere you start with the beginning
[00:27:05] everywhere you start with the beginning challenges and you go from there is just
[00:27:07] challenges and you go from there is just getting them to walk fast enough so if
[00:27:09] getting them to walk fast enough so if they're going to score goals and they're
[00:27:11] they're going to score goals and they're going to compete you need them to walk
[00:27:12] going to compete you need them to walk quickly and so there's this question of
[00:27:14] quickly and so there's this question of just how do you learn fast walks you
[00:27:17] just how do you learn fast walks you know so that they can sort of trying to
[00:27:19] know so that they can sort of trying to teach robots to run and what they found
[00:27:21] teach robots to run and what they found here is that they could use policy
[00:27:23] here is that they could use policy methods and policy search methods just
[00:27:25] methods and policy search methods just to learn a faster way for it to walk and
[00:27:28] to learn a faster way for it to walk and so they parameterized sort of the curve
[00:27:31] so they parameterized sort of the curve of how the the foot moves as a set of
[00:27:33] of how the the foot moves as a set of parameters and that defined the policy
[00:27:35] parameters and that defined the policy for moving those joints and then what
[00:27:37] for moving those joints and then what they did is they just had these walk
[00:27:39] they did is they just had these walk back and forth many many times and what
[00:27:41] back and forth many many times and what they would do is they'd have them walk
[00:27:43] they would do is they'd have them walk um you know with some particular policy
[00:27:46] um you know with some particular policy parameters they would see how fast they
[00:27:48] parameters they would see how fast they walked they would do finite different
[00:27:50] walked they would do finite different methods so they weren't trying to
[00:27:51] methods so they weren't trying to explicitly do auto dip or anything there
[00:27:54] explicitly do auto dip or anything there and then they would slightly change the
[00:27:56] and then they would slightly change the policy parameters and repeat
[00:27:59] policy parameters and repeat and they substantially they learned a
[00:28:01] and they substantially they learned a substantially faster walk during that
[00:28:03] substantially faster walk during that and I think it took maybe around four
[00:28:05] and I think it took maybe around four hours or so but they just had to replace
[00:28:07] hours or so but they just had to replace the batteries a couple
[00:28:09] the batteries a couple time so just an example to say like you
[00:28:11] time so just an example to say like you know it's lovely to have autoi you can
[00:28:13] know it's lovely to have autoi you can do really complicated things now but
[00:28:14] do really complicated things now but these methods can work even in really
[00:28:16] these methods can work even in really basic settings particularly where you
[00:28:18] basic settings particularly where you think you have pretty bad models of like
[00:28:20] think you have pretty bad models of like how the world works and so now you can
[00:28:22] how the world works and so now you can just be directly data driven and why is
[00:28:24] just be directly data driven and why is this as a hard problem for those of you
[00:28:26] this as a hard problem for those of you that haven't done robotics it involves a
[00:28:28] that haven't done robotics it involves a whole bunch of contact forces um you
[00:28:31] whole bunch of contact forces um you know the ground may be uh well there
[00:28:34] know the ground may be uh well there they have to learn on this particular
[00:28:35] they have to learn on this particular ground you may not know because it's
[00:28:37] ground you may not know because it's commercial Hardware you may not know
[00:28:39] commercial Hardware you may not know exactly all the parameters that the
[00:28:40] exactly all the parameters that the designers put in so you can just be data
[00:28:43] designers put in so you can just be data driven okay as opposed to maybe having
[00:28:45] driven okay as opposed to maybe having like a physics
[00:28:48] like a physics simulator all right so just to summarize
[00:28:50] simulator all right so just to summarize so far the benefits of policy based RL
[00:28:52] so far the benefits of policy based RL is that um we're going to have often
[00:28:56] is that um we're going to have often better convergence properties often
[00:28:58] better convergence properties often going to be able to guarantee that we
[00:28:59] going to be able to guarantee that we get to a local Optima whereas we didn't
[00:29:01] get to a local Optima whereas we didn't have that for deep Q learning um they're
[00:29:03] have that for deep Q learning um they're often really effective in high
[00:29:04] often really effective in high dimensional or continuous action spaces
[00:29:07] dimensional or continuous action spaces and you can learn stochastic
[00:29:09] and you can learn stochastic policies but the methods we've seen so
[00:29:11] policies but the methods we've seen so far might be more inefficient and higher
[00:29:13] far might be more inefficient and higher variance and we often only get to
[00:29:15] variance and we often only get to something of a local
[00:29:17] something of a local Optima and we'll see some things to help
[00:29:20] Optima and we'll see some things to help with kind of the inefficiency in a
[00:29:23] with kind of the inefficiency in a second all right so now what we're going
[00:29:25] second all right so now what we're going to dive into is how do we do this when
[00:29:27] to dive into is how do we do this when we are willing to have differentiable
[00:29:33] policies so the hope is that we can
[00:29:36] policies so the hope is that we can actually compute the policy gradient
[00:29:38] actually compute the policy gradient analytically so we don't have to do it
[00:29:40] analytically so we don't have to do it with finite
[00:29:42] with finite differences and we're going to focus on
[00:29:44] differences and we're going to focus on policies where it's differentiable as
[00:29:46] policies where it's differentiable as long as it's non zero so we're going to
[00:29:49] long as it's non zero so we're going to assume that we can always compute the
[00:29:50] assume that we can always compute the gradient of the policy parameters
[00:29:53] gradient of the policy parameters themselves and there are a number of
[00:29:54] themselves and there are a number of different classes we can do this for um
[00:29:56] different classes we can do this for um and there are many popular classes uh
[00:29:58] and there are many popular classes uh including of course deep neural
[00:30:01] including of course deep neural networks so popular ones are often
[00:30:05] networks so popular ones are often softmax softmax is used all the time
[00:30:07] softmax softmax is used all the time I'll explain what is in a second
[00:30:09] I'll explain what is in a second gaussian um and neural networks and
[00:30:12] gaussian um and neural networks and again just to be clear here what I mean
[00:30:13] again just to be clear here what I mean by a policy class is what is the
[00:30:16] by a policy class is what is the functional form we are using to give us
[00:30:18] functional form we are using to give us a
[00:30:21] probability of an action given a
[00:30:24] probability of an action given a state so are we having something like
[00:30:26] state so are we having something like well I guess we can just see on this
[00:30:28] well I guess we can just see on this next slide what what these will look
[00:30:31] next slide what what these will look like okay so we're going to assume I'm
[00:30:34] like okay so we're going to assume I'm going to give you some examples of those
[00:30:35] going to give you some examples of those of what you know what softmax um and G
[00:30:38] of what you know what softmax um and G neural networks look like in a second in
[00:30:40] neural networks look like in a second in terms of how how we differentiate them
[00:30:43] terms of how how we differentiate them but these are just different ways for us
[00:30:44] but these are just different ways for us to parameterize what is the probability
[00:30:46] to parameterize what is the probability of an action given a
[00:30:47] of an action given a state um actually I guess I'll give a
[00:30:49] state um actually I guess I'll give a quick example with Gan for Gan you could
[00:30:52] quick example with Gan for Gan you could imagine let's imagine I have a robot and
[00:30:54] imagine let's imagine I have a robot and I'm trying to figure out say um how much
[00:30:57] I'm trying to figure out say um how much speed to apply then you might have a
[00:31:00] speed to apply then you might have a policy class that
[00:31:01] policy class that says the action I take is equal to a
[00:31:05] says the action I take is equal to a Galan centered around
[00:31:08] Galan centered around 0.5 with some standard deviation so it
[00:31:11] 0.5 with some standard deviation so it would be a stochastic policy um and it
[00:31:15] would be a stochastic policy um and it would say the average amount of speed
[00:31:17] would say the average amount of speed you're going to apply is 0.5 but you're
[00:31:18] you're going to apply is 0.5 but you're going to have some variability around
[00:31:20] going to have some variability around that that would give you some stochastic
[00:31:23] that that would give you some stochastic Behavior sometimes the robot would go
[00:31:25] Behavior sometimes the robot would go really slowly sometimes it would go fast
[00:31:27] really slowly sometimes it would go fast so it would go in a negative
[00:31:30] so it would go in a negative Direction okay so let's keep assuming
[00:31:32] Direction okay so let's keep assuming that the policy is differentiable
[00:31:34] that the policy is differentiable whenever it's non zero we know the
[00:31:35] whenever it's non zero we know the gradient that still doesn't tell us how
[00:31:37] gradient that still doesn't tell us how to solve policy gradient methods yet
[00:31:40] to solve policy gradient methods yet because what we want to do is take
[00:31:41] because what we want to do is take derivatives of the value function so we
[00:31:44] derivatives of the value function so we want to say I want to find the maximum
[00:31:47] want to say I want to find the maximum the policy that has the best value
[00:31:48] the policy that has the best value function which means I'm going to need
[00:31:49] function which means I'm going to need to take the derivative of the value
[00:31:51] to take the derivative of the value function with respect to the policy
[00:31:53] function with respect to the policy parameters okay so remember that the
[00:31:56] parameters okay so remember that the policy value the value of the initial
[00:31:58] policy value the value of the initial starting State under a policy is going
[00:32:00] starting State under a policy is going to be the expected dis expected sum of
[00:32:03] to be the expected dis expected sum of rewards we don't have to use discounting
[00:32:06] rewards we don't have to use discounting for most of today if we assume it's
[00:32:07] for most of today if we assume it's finite let just say we're assume we're
[00:32:11] finite let just say we're assume we're in the episodic
[00:32:13] in the episodic case and so this is finite so no
[00:32:16] case and so this is finite so no discount counting for
[00:32:21] now so we don't need discounting for now
[00:32:23] now so we don't need discounting for now because it's always a finite length so
[00:32:24] because it's always a finite length so we're never going to have infinite
[00:32:26] we're never going to have infinite reward so the policy value is just the
[00:32:28] reward so the policy value is just the expected sum of discounted rewards when
[00:32:30] expected sum of discounted rewards when we follow the policy parameterized by
[00:32:32] we follow the policy parameterized by Theta till the end of the episode
[00:32:35] Theta till the end of the episode starting from the state
[00:32:38] starting from the state s0 and there are lots of different ways
[00:32:40] s0 and there are lots of different ways for us to write this
[00:32:42] for us to write this down so one ways for us to write down B
[00:32:45] down so one ways for us to write down B of Sr is equal to well it's equal to the
[00:32:48] of Sr is equal to well it's equal to the state action value averaged over the
[00:32:51] state action value averaged over the probability of us taking each of those
[00:32:53] probability of us taking each of those actions under our
[00:32:55] actions under our policy so this here just says what is
[00:32:58] policy so this here just says what is the probability of me taking this action
[00:33:00] the probability of me taking this action starting state s0 if I have policy
[00:33:02] starting state s0 if I have policy parameterized by Theta times what is my
[00:33:05] parameterized by Theta times what is my Q value of starting in that state taking
[00:33:08] Q value of starting in that state taking that particular action and then
[00:33:09] that particular action and then following that policy for the rest of
[00:33:12] following that policy for the rest of it okay so this is one way to write it
[00:33:15] it okay so this is one way to write it but we can also think of a quite
[00:33:16] but we can also think of a quite different way which is let's think about
[00:33:19] different way which is let's think about trajectories I'll write down what these
[00:33:21] trajectories I'll write down what these are in a
[00:33:23] are in a second okay so this is a
[00:33:26] second okay so this is a trajectory what's a trajectory that's
[00:33:28] trajectory what's a trajectory that's going to be s0 and then it's going to be
[00:33:30] going to be s0 and then it's going to be an action and then S1 dot dot dot
[00:33:35] dot
[00:33:38] sampled from PI Theta okay so another
[00:33:43] sampled from PI Theta okay so another way we can think of the value and then
[00:33:45] way we can think of the value and then this is going to be the
[00:33:48] this is going to be the reward for that trajectory
[00:33:55] that and we've called RG before so we
[00:33:59] that and we've called RG before so we I'll just write down that in
[00:34:11] case so another way we can think of the
[00:34:13] case so another way we can think of the value is we say well let's just sum over
[00:34:15] value is we say well let's just sum over all possible trajectories we could reach
[00:34:17] all possible trajectories we could reach under this policy and what would be the
[00:34:19] under this policy and what would be the reward for each of those trajectories
[00:34:21] reward for each of those trajectories and I'm just going to take a weighted
[00:34:22] and I'm just going to take a weighted sum now of course you might be thinking
[00:34:24] sum now of course you might be thinking that's totally intractable and yes in
[00:34:26] that's totally intractable and yes in general um if H you have a really long
[00:34:29] general um if H you have a really long trajectory then it's going to be inct
[00:34:32] trajectory then it's going to be inct and you have a really large State space
[00:34:33] and you have a really large State space and you could reach many states in
[00:34:34] and you could reach many states in general it's not going to be possible to
[00:34:35] general it's not going to be possible to actually enumerate this but this is
[00:34:38] actually enumerate this but this is mathematically well defined this is just
[00:34:40] mathematically well defined this is just an expectation over the reward of
[00:34:43] an expectation over the reward of trajectories and we know whenever we see
[00:34:45] trajectories and we know whenever we see expectations that we can approximate
[00:34:47] expectations that we can approximate those with finite samples you can think
[00:34:49] those with finite samples you can think of just taking n samples just like what
[00:34:51] of just taking n samples just like what we saw with Monte Carlo methods and
[00:34:53] we saw with Monte Carlo methods and using that to approximate a
[00:34:55] using that to approximate a trajectory so in general this is inter
[00:34:58] trajectory so in general this is inter able General
[00:35:04] intractable but we can approximate by
[00:35:13] sampling all right so this is one way we
[00:35:17] sampling all right so this is one way we could write down so this is another also
[00:35:19] could write down so this is another also valid way to write down what is the
[00:35:21] valid way to write down what is the value starting the state and following
[00:35:22] value starting the state and following the
[00:35:25] the policy okay so I've written that doubt
[00:35:27] policy okay so I've written that doubt more neatly here
[00:35:28] more neatly here P of T Theta is the probability over
[00:35:31] P of T Theta is the probability over trajectories when you execute that
[00:35:33] trajectories when you execute that policy starting State s0 and that is the
[00:35:35] policy starting State s0 and that is the sum of the words for a
[00:35:36] sum of the words for a trajectory in this class we're going to
[00:35:38] trajectory in this class we're going to focus on this latter definition but
[00:35:41] focus on this latter definition but inside of set and Bardo they have a nice
[00:35:43] inside of set and Bardo they have a nice way to think about policy gradient
[00:35:44] way to think about policy gradient methods that starts from the other
[00:35:46] methods that starts from the other definition so you can always look at
[00:35:47] definition so you can always look at that but both are totally valid
[00:35:53] definitions all right so now we're going
[00:35:55] definitions all right so now we're going to focus on thinking about likely Hood
[00:35:57] to focus on thinking about likely Hood ratio policies so we're going to be
[00:35:59] ratio policies so we're going to be thinking about this case where we have a
[00:36:01] thinking about this case where we have a distribution over trajectories and then
[00:36:04] distribution over trajectories and then what is the sum of rewards for each of
[00:36:05] what is the sum of rewards for each of those
[00:36:06] those trajectories so we have our value
[00:36:08] trajectories so we have our value function and now what we want to do is
[00:36:10] function and now what we want to do is find the
[00:36:12] find the argmax so that we
[00:36:14] argmax so that we maximize us having probability of
[00:36:17] maximize us having probability of getting trajectories with high
[00:36:19] getting trajectories with high reward so that's nice so instead of just
[00:36:21] reward so that's nice so instead of just thinking about the value function we now
[00:36:23] thinking about the value function we now can think of it as okay I want to have
[00:36:25] can think of it as okay I want to have policies that induce trajectories
[00:36:26] policies that induce trajectories through the state space through this
[00:36:28] through the state space through this state in action space they give me high
[00:36:32] reward so what we're going to need to be
[00:36:33] reward so what we're going to need to be able to do is to take a um a gradient
[00:36:37] able to do is to take a um a gradient through the right hand
[00:36:39] through the right hand side okay so that's what we're going to
[00:36:41] side okay so that's what we're going to do
[00:36:44] now just okay so we're going to take the
[00:36:49] now just okay so we're going to take the gradient of this because once we have
[00:36:50] gradient of this because once we have the gradient of the value function with
[00:36:51] the gradient of the value function with respect to the policy parameters we can
[00:36:53] respect to the policy parameters we can update our policy parameters to increase
[00:36:56] update our policy parameters to increase hopefully the value of the policy that
[00:36:58] hopefully the value of the policy that we're at okay so what we're going to do
[00:37:00] we're at okay so what we're going to do is we're going to say we're going to
[00:37:01] is we're going to say we're going to take the gradient with respect to the
[00:37:02] take the gradient with respect to the right hand side we can rewrite this by
[00:37:06] right hand side we can rewrite this by pushing in the
[00:37:13] gradient okay now R of to doesn't depend
[00:37:16] gradient okay now R of to doesn't depend on the policy parameters that's just
[00:37:18] on the policy parameters that's just what is the reward once you've told me
[00:37:20] what is the reward once you've told me what a trajectory is so we can put that
[00:37:22] what a trajectory is so we can put that on the other side
[00:37:27] this is the only part that depends on
[00:37:29] this is the only part that depends on the policy
[00:37:30] the policy parameters and now I'm going to play um
[00:37:33] parameters and now I'm going to play um a trick I'm going to
[00:37:39] note that this is going to be equal to
[00:37:44] note that this is going to be equal to well I'm going to do something that's
[00:37:44] well I'm going to do something that's going to seem not very helpful for a
[00:37:46] going to seem not very helpful for a second and then we'll see why it's
[00:37:48] second and then we'll see why it's helpful just going to multiply and
[00:37:50] helpful just going to multiply and divide by the probability of a
[00:37:55] trajectory I haven't done anything I've
[00:37:57] trajectory I haven't done anything I've just multiplied by one and I've happened
[00:37:59] just multiplied by one and I've happened to multiply by the top and bottom by the
[00:38:02] to multiply by the top and bottom by the probability of that trajectory but then
[00:38:04] probability of that trajectory but then I'm going to note
[00:38:07] I'm going to note that the derivative with respect to log
[00:38:10] that the derivative with respect to log of the trajectory in Theta is just equal
[00:38:14] of the trajectory in Theta is just equal to one over the probability
[00:38:17] OFA time the derivative with respect to
[00:38:22] OFA time the derivative with respect to the trajectory in
[00:38:24] the trajectory in Theta okay because the derivative of log
[00:38:26] Theta okay because the derivative of log is just equal to one over the value
[00:38:28] is just equal to one over the value times the derivative of the thing inside
[00:38:30] times the derivative of the thing inside the log so that looks exactly like this
[00:38:35] the log so that looks exactly like this okay so that's the trick that we're
[00:38:38] okay so that's the trick that we're playing here and so we can rewrite this
[00:38:40] playing here and so we can rewrite this then as the probability I'll tell you
[00:38:43] then as the probability I'll tell you why we do this in a
[00:38:52] second
[00:38:53] second okay all right and let me just rewrite
[00:38:55] okay all right and let me just rewrite it one more time so so it's more easy to
[00:39:09] see why did we do this okay the reason
[00:39:12] see why did we do this okay the reason we did this is that in general it's
[00:39:14] we did this is that in general it's going to be hard for us to think about
[00:39:15] going to be hard for us to think about or it might be tricky for us to think
[00:39:17] or it might be tricky for us to think about how do we propagate our derivative
[00:39:19] about how do we propagate our derivative through something that's an expectation
[00:39:21] through something that's an expectation we had an expectation over all the
[00:39:22] we had an expectation over all the trajectories waited by the reward of
[00:39:24] trajectories waited by the reward of those trajectories we now want to want
[00:39:26] those trajectories we now want to want to take a gradient with with respect to
[00:39:27] to take a gradient with with respect to it we want to end up with something that
[00:39:30] it we want to end up with something that is computable from samples um because
[00:39:32] is computable from samples um because it's easy for us to get samples we can
[00:39:34] it's easy for us to get samples we can actually run our policy in the
[00:39:35] actually run our policy in the environment so by playing this trick
[00:39:38] environment so by playing this trick what we now have is something that we
[00:39:39] what we now have is something that we can also sample because this is now an
[00:39:41] can also sample because this is now an expectation over trajectories of the
[00:39:44] expectation over trajectories of the reward of the trajectory weighted by the
[00:39:46] reward of the trajectory weighted by the gradient of the log of the probability
[00:39:48] gradient of the log of the probability of that trajectory okay and we'll talk
[00:39:50] of that trajectory okay and we'll talk soon about how you compute this part but
[00:39:53] soon about how you compute this part but this expectation can be sampled
[00:39:57] this expectation can be sampled okay because this is just a probability
[00:39:59] okay because this is just a probability over trajectories and so we could sample
[00:40:01] over trajectories and so we could sample say 100 of them and approximate that
[00:40:03] say 100 of them and approximate that outter
[00:40:05] outter expectation so that's one of the right
[00:40:07] expectation so that's one of the right reasons why this is I'm just writing
[00:40:08] reasons why this is I'm just writing this out more neatly here on the next
[00:40:09] this out more neatly here on the next slide this is called the likelihood
[00:40:13] slide this is called the likelihood ratio this term
[00:40:15] ratio this term here and so that's one of the benefits
[00:40:17] here and so that's one of the benefits to doing this is that we want to end up
[00:40:18] to doing this is that we want to end up with something that is computable we
[00:40:20] with something that is computable we want to be able to get this gradient
[00:40:22] want to be able to get this gradient with respect to the value function for
[00:40:23] with respect to the value function for the policy parameters and so this is
[00:40:25] the policy parameters and so this is going to give us something that we can
[00:40:27] going to give us something that we can app approximate with samples and we can
[00:40:29] app approximate with samples and we can compute okay all right now you still
[00:40:33] compute okay all right now you still might be a little bit concerned because
[00:40:35] might be a little bit concerned because all right maybe you think yeah I can
[00:40:38] all right maybe you think yeah I can maybe compute this by like writing
[00:40:40] maybe compute this by like writing things out in the environment but I'm
[00:40:41] things out in the environment but I'm still going to have to take this
[00:40:43] still going to have to take this derivative and how am I going to do that
[00:40:45] derivative and how am I going to do that and what does it end up depending on so
[00:40:48] and what does it end up depending on so let's do a Next
[00:40:50] let's do a Next Step so as I said what we're going to do
[00:40:52] Step so as I said what we're going to do here this is an expectation this is an
[00:40:55] here this is an expectation this is an expectation
[00:40:58] and we're going to approximate that
[00:41:00] and we're going to approximate that expectation with an imperical
[00:41:04] expectation with an imperical estimate okay so we're just going to
[00:41:05] estimate okay so we're just going to instead of actually taking you know all
[00:41:07] instead of actually taking you know all possible trajectories particularly in
[00:41:09] possible trajectories particularly in the case of vision um input you could
[00:41:11] the case of vision um input you could imagine that would be completely insane
[00:41:13] imagine that would be completely insane so we're just going to approximate it by
[00:41:14] so we're just going to approximate it by taking n
[00:41:16] taking n samples but we still have to handle this
[00:41:19] samples but we still have to handle this okay so that's what we're going to do
[00:41:20] okay so that's what we're going to do next so this first part should all seem
[00:41:23] next so this first part should all seem clear the second part should Le
[00:41:25] clear the second part should Le certainly for most of us would not be
[00:41:27] certainly for most of us would not be clear yet about how we do that second
[00:41:30] clear yet about how we do that second part okay so what do we do with that
[00:41:33] part okay so what do we do with that what we're going to do now is we're
[00:41:34] what we're going to do now is we're going to decompose that latter part into
[00:41:37] going to decompose that latter part into States and
[00:41:39] States and actions so remember that what this means
[00:41:42] actions so remember that what this means here is this is going to be a particular
[00:41:45] here is this is going to be a particular trajectory we get by following a policy
[00:41:47] trajectory we get by following a policy for teps or until the end of the episode
[00:41:51] for teps or until the end of the episode okay so let me just write remind
[00:41:52] okay so let me just write remind ourselves what T is going to look like
[00:41:54] ourselves what T is going to look like here
[00:42:00] so this is going to be like time step
[00:42:02] so this is going to be like time step here I'm using the subscript As Time
[00:42:05] here I'm using the subscript As Time step okay so let's just write out what a
[00:42:08] step okay so let's just write out what a trajectory is and what those
[00:42:09] trajectory is and what those probabilities
[00:42:13] are approximating the probability of the
[00:42:15] are approximating the probability of the trajectory just to be one over n no good
[00:42:18] trajectory just to be one over n no good question yeah or oh sorry yes for for
[00:42:21] question yeah or oh sorry yes for for this part yes yes we're assuming that
[00:42:23] this part yes yes we're assuming that we're s for each of the trajectories
[00:42:25] we're s for each of the trajectories we're using like a Monte Carlo estimate
[00:42:27] we're using like a Monte Carlo estimate we're just using 1 over M but you know
[00:42:29] we're just using 1 over M but you know if some trajectories are more likely
[00:42:30] if some trajectories are more likely than others they'll appear more in that
[00:42:32] than others they'll appear more in that set of
[00:42:33] set of M it's good question okay so let's now
[00:42:38] M it's good question okay so let's now try to express what the probability is
[00:42:39] try to express what the probability is of a
[00:42:40] of a trajectory okay so the probability of a
[00:42:44] trajectory okay so the probability of a trajectory we can write out as
[00:42:46] trajectory we can write out as follows so we're going to still have
[00:42:49] follows so we're going to still have that
[00:42:51] outside log we're going to do the
[00:42:54] outside log we're going to do the following okay going to say mu of s not
[00:42:57] following okay going to say mu of s not is equal to the probability of s okay
[00:43:00] is equal to the probability of s okay that's just like what is our probability
[00:43:02] that's just like what is our probability distribution over our starting State
[00:43:04] distribution over our starting State okay so that's me and then what we're
[00:43:06] okay so that's me and then what we're going to have is the
[00:43:08] going to have is the following t = 0 to T minus
[00:43:13] following t = 0 to T minus one we going to have our
[00:43:18] policy so this is going to say what is
[00:43:21] policy so this is going to say what is the probability that I pick the action I
[00:43:23] the probability that I pick the action I picked given the current state I'm
[00:43:26] picked given the current state I'm in times the
[00:43:29] in times the probability of St + 1
[00:43:33] probability of St + 1 given s0 to t a z to
[00:43:40] T so what I've done is I've just written
[00:43:43] T so what I've done is I've just written out what is happening in my trajectory
[00:43:46] out what is happening in my trajectory here is I start I have some distribution
[00:43:48] here is I start I have some distribution over C in this initial State under my
[00:43:51] over C in this initial State under my policy I have some probability of taking
[00:43:53] policy I have some probability of taking a z and that's here then um I'm going to
[00:43:57] a z and that's here then um I'm going to assume for a second that rewards are
[00:43:58] assume for a second that rewards are deterministic but you could add in a
[00:44:00] deterministic but you could add in a reward term here and then I'm going to
[00:44:02] reward term here and then I'm going to say well what's the chance that I get to
[00:44:04] say well what's the chance that I get to State S1 given my history given the
[00:44:07] State S1 given my history given the previous States and the
[00:44:09] previous States and the actions so I've just written this out as
[00:44:12] actions so I've just written this out as um a joint probability okay and now what
[00:44:16] um a joint probability okay and now what I can do is I can use the fact that log
[00:44:17] I can do is I can use the fact that log of a * B is equal to log of a plus log
[00:44:19] of a * B is equal to log of a plus log of B so I'm just going to decompose all
[00:44:21] of B so I'm just going to decompose all these terms okay so I'm not applying my
[00:44:23] these terms okay so I'm not applying my gradient yet I'm just going to have log
[00:44:27] gradient yet I'm just going to have log mu of
[00:44:28] mu of S Plus su/ t = 0 tus
[00:44:38] 1 t = t
[00:44:48] 1 let put the L
[00:44:59] sorry it's a bit messy I'll make sure to
[00:45:01] sorry it's a bit messy I'll make sure to add a clean
[00:45:04] add a clean version um so what I've done is I've
[00:45:07] version um so what I've done is I've just decomposed my log but now this is
[00:45:09] just decomposed my log but now this is really nice
[00:45:11] really nice because this term is not a function of
[00:45:14] because this term is not a function of theta this is just my initial starting
[00:45:16] theta this is just my initial starting State distribution it has nothing to do
[00:45:18] State distribution it has nothing to do with my policy so this drops out does
[00:45:21] with my policy so this drops out does this part depend on my
[00:45:23] this part depend on my policy yes does this part depend on my
[00:45:26] policy yes does this part depend on my policy
[00:45:27] policy no so when we take the d of it it
[00:45:31] no so when we take the d of it it disappears so that is beautiful because
[00:45:33] disappears so that is beautiful because now it means we don't have to know about
[00:45:34] now it means we don't have to know about our Dynamics model okay so the only term
[00:45:38] our Dynamics model okay so the only term that is still around after
[00:45:45] this is this thing
[00:45:58] all
[00:46:00] all right so this is great because now we
[00:46:03] right so this is great because now we don't depend on our Dynamics
[00:46:05] don't depend on our Dynamics model we have written down what this
[00:46:09] model we have written down what this term is as a function of um so we're
[00:46:14] term is as a function of um so we're just doing this term right now as just
[00:46:16] just doing this term right now as just the sum of the derivative of the log of
[00:46:18] the sum of the derivative of the log of the policy at that particular point so
[00:46:21] the policy at that particular point so we're sort of summing up for each of the
[00:46:23] we're sort of summing up for each of the different actions we took along the way
[00:46:26] different actions we took along the way what was the log of their probability
[00:46:27] what was the log of their probability and taking the derivative of that whole
[00:46:33] term all
[00:46:35] term all right so we don't need any Dynamics
[00:46:37] right so we don't need any Dynamics model which is
[00:46:39] model which is great and I'm just going to say here I'm
[00:46:43] great and I'm just going to say here I'm going to make sure that um something is
[00:46:44] going to make sure that um something is consistent here oh
[00:46:49] yeah with all the meth
[00:46:54] uhuh the PSS t+ one for the um part why
[00:47:00] uhuh the PSS t+ one for the um part why do we look at the entire history and not
[00:47:02] do we look at the entire history and not just the Past St in action great
[00:47:05] just the Past St in action great question so what I've written out as so
[00:47:08] question so what I've written out as so this question is a good one I wrote down
[00:47:10] this question is a good one I wrote down here the Dynamics in a really general
[00:47:12] here the Dynamics in a really general form I am writing them down and I'm not
[00:47:14] form I am writing them down and I'm not making the markup assumption we could
[00:47:16] making the markup assumption we could make the markup assumption but what I
[00:47:18] make the markup assumption but what I wanted to point out here is that you
[00:47:19] wanted to point out here is that you don't have to make the markup assumption
[00:47:21] don't have to make the markup assumption does not matter so because the Dynamics
[00:47:26] does not matter so because the Dynamics model are independent of your policy
[00:47:27] model are independent of your policy when you take the derivative they
[00:47:29] when you take the derivative they completely drop out whether they are
[00:47:31] completely drop out whether they are markof whether they are non-markov Etc
[00:47:33] markof whether they are non-markov Etc and so that's really nice it shows that
[00:47:35] and so that's really nice it shows that in this case it's not making the Markoff
[00:47:39] assumption now I did make the Markoff
[00:47:41] assumption now I did make the Markoff assumption somewhere I made it here
[00:47:43] assumption somewhere I made it here because I assume that um I made the
[00:47:44] because I assume that um I made the Markoff assumption in this sense I
[00:47:45] Markoff assumption in this sense I assume my policy was Markoff my policy
[00:47:48] assume my policy was Markoff my policy is only depending on the current state
[00:47:50] is only depending on the current state um but your policy also could depend on
[00:47:52] um but your policy also could depend on a history of States you know you could
[00:47:54] a history of States you know you could have like a current neur uh neural
[00:47:56] have like a current neur uh neural network work or any of the other
[00:47:58] network work or any of the other representations you might want to choose
[00:48:00] representations you might want to choose there um and you would still then this
[00:48:02] there um and you would still then this would just depend on your
[00:48:05] history good question all right so I
[00:48:08] history good question all right so I just want to
[00:48:09] just want to go and I want to make sure that I wrote
[00:48:11] go and I want to make sure that I wrote it down neatly in terms of the most
[00:48:13] it down neatly in terms of the most general form that's why I'm skipping
[00:48:14] general form that's why I'm skipping those right now one of the things to
[00:48:16] those right now one of the things to note here in terms of just notation is
[00:48:18] note here in terms of just notation is that people often call this thing here a
[00:48:20] that people often call this thing here a score function so this derivative with
[00:48:23] score function so this derivative with respect to log of the policy itself we
[00:48:26] respect to log of the policy itself we often call it score
[00:48:29] often call it score function
[00:48:30] function okay so in general
[00:48:34] okay so in general um we um the nice thing is that it's
[00:48:37] um we um the nice thing is that it's generally not very hard to compute the
[00:48:38] generally not very hard to compute the score function so if you have a
[00:48:40] score function so if you have a differentiable function we can compute
[00:48:42] differentiable function we can compute the score function pretty easily in many
[00:48:44] the score function pretty easily in many cases let just make this bit smaller
[00:48:47] cases let just make this bit smaller okay so let's see what that might look
[00:48:49] okay so let's see what that might look like for a couple different policy
[00:48:52] like for a couple different policy classes so one thing we could do which
[00:48:55] classes so one thing we could do which is a pretty popular thing to do is do a
[00:48:56] is a pretty popular thing to do is do a softmax policy so the idea in this case
[00:49:00] softmax policy so the idea in this case is that let's take a linear combination
[00:49:02] is that let's take a linear combination of features so 5
[00:49:05] of features so 5 sa dot product with Theta and then you
[00:49:08] sa dot product with Theta and then you could say the probability of Your Action
[00:49:11] could say the probability of Your Action is proportional to the exponentiated
[00:49:13] is proportional to the exponentiated weight so you take the exponent of that
[00:49:16] weight so you take the exponent of that dot product between the features and
[00:49:18] dot product between the features and then you normalize it and that gives you
[00:49:20] then you normalize it and that gives you generally a stochastic policy um you can
[00:49:23] generally a stochastic policy um you can also have a temperature parameter in
[00:49:24] also have a temperature parameter in there if you want
[00:49:26] there if you want and the nice thing about this is that we
[00:49:28] and the nice thing about this is that we can write it um we can take the
[00:49:30] can write it um we can take the derivative of this very easily okay so
[00:49:32] derivative of this very easily okay so we can just do that quickly here just to
[00:49:34] we can just do that quickly here just to illustrate so what this is this is just
[00:49:37] illustrate so what this is this is just to illustrate that like it is often very
[00:49:38] to illustrate that like it is often very feasible to take the derivative with
[00:49:40] feasible to take the derivative with respect to the policy
[00:49:42] respect to the policy parameterization okay so this is just
[00:49:44] parameterization okay so this is just going to be the derivative of the log of
[00:49:47] going to be the derivative of the log of e to 5 of s t Theta
[00:49:59] so we can do this here and we can
[00:50:01] so we can do this here and we can rewrite this here
[00:50:22] as okay and so this is just going to be
[00:50:26] as okay and so this is just going to be equal to five
[00:50:43] s so I'm just taking the derivative of
[00:50:45] s so I'm just taking the derivative of this for a particular
[00:50:48] this for a particular Theta and so we can just rewrite that 5
[00:51:06] okay so what I've done here so I've
[00:51:08] okay so what I've done here so I've taken the derivative with respect to
[00:51:10] taken the derivative with respect to this function for a particular
[00:51:12] this function for a particular Theta and then what I've said here is
[00:51:14] Theta and then what I've said here is well you could notice that this here is
[00:51:17] well you could notice that this here is exactly just equal to my Pi Theta of
[00:51:20] exactly just equal to my Pi Theta of essay so it's like I'm getting this
[00:51:23] essay so it's like I'm getting this waiting over the
[00:51:24] waiting over the features okay on the next slide nether
[00:51:28] features okay on the next slide nether okay so the score function for the for
[00:51:32] okay so the score function for the for the softmax policy is just going to be
[00:51:34] the softmax policy is just going to be equal to the feature sa FSA minus the
[00:51:38] equal to the feature sa FSA minus the expected value over the policy of the
[00:51:42] expected value over the policy of the features
[00:51:46] yeah oh great question well I could be
[00:51:49] yeah oh great question well I could be for example you could think of it as
[00:51:51] for example you could think of it as like if you have um a large neural
[00:51:53] like if you have um a large neural network that's doing some representation
[00:51:54] network that's doing some representation it could be the last layer like the
[00:51:56] it could be the last layer like the second last there and then you could
[00:51:58] second last there and then you could just do like a linear dot product
[00:52:01] just do like a linear dot product that yeah it's a good question you can
[00:52:04] that yeah it's a good question you can or in case of customers it could be a
[00:52:05] or in case of customers it could be a whole bunch of different features and
[00:52:07] whole bunch of different features and then you have different we
[00:52:10] there all right
[00:52:13] there all right so this is also possible to do for other
[00:52:15] so this is also possible to do for other functions so for Gans um uh we often
[00:52:19] functions so for Gans um uh we often want to think about that for continuous
[00:52:20] want to think about that for continuous action spaces which are really useful
[00:52:21] action spaces which are really useful for robotics where you might have like
[00:52:23] for robotics where you might have like continuous torqus or continuous
[00:52:24] continuous torqus or continuous accelerations Etc you can think of there
[00:52:27] accelerations Etc you can think of there being a mean which is a linear
[00:52:28] being a mean which is a linear combination of some State features your
[00:52:30] combination of some State features your variance might be fixed or could also be
[00:52:33] variance might be fixed or could also be parameterized and then your policy is a
[00:52:35] parameterized and then your policy is a gan so like you maybe you're sampling
[00:52:37] gan so like you maybe you're sampling some particular action dependent on your
[00:52:39] some particular action dependent on your state along with some variance and then
[00:52:42] state along with some variance and then you can again just directly compute what
[00:52:45] you can again just directly compute what the score function would be in this case
[00:52:47] the score function would be in this case in close
[00:52:48] in close form but in general you're often
[00:52:50] form but in general you're often probably going to be using this with
[00:52:51] probably going to be using this with deep neural networks and then you can
[00:52:52] deep neural networks and then you can just use autoi to do this it's just to
[00:52:54] just use autoi to do this it's just to illustrate that there's a number of
[00:52:55] illustrate that there's a number of different functional forms where you can
[00:52:57] different functional forms where you can compute this
[00:53:00] analytically okay all right so just to
[00:53:02] analytically okay all right so just to recap this what we've shown so far is
[00:53:05] recap this what we've shown so far is that we can have policy methods where we
[00:53:07] that we can have policy methods where we have a direct parameterization of the
[00:53:08] have a direct parameterization of the policy we can write down the value
[00:53:10] policy we can write down the value function as being a weighted sum over
[00:53:12] function as being a weighted sum over the trajectories generated by that
[00:53:13] the trajectories generated by that policy times their reward it turns out
[00:53:16] policy times their reward it turns out that when we want to take the derivative
[00:53:17] that when we want to take the derivative of that we can reexpress it so that we
[00:53:19] of that we can reexpress it so that we just think of we only we don't need the
[00:53:21] just think of we only we don't need the Dynamics model and we're weighing these
[00:53:23] Dynamics model and we're weighing these score functions
[00:53:27] so now let's just do um a small check
[00:53:29] so now let's just do um a small check your understanding
[00:53:31] your understanding about uh likelihood ratio score function
[00:53:34] about uh likelihood ratio score function policy gradients and so what I'd like
[00:53:36] policy gradients and so what I'd like you to do is say does it require that
[00:53:38] you to do is say does it require that your reward function is differentiable
[00:53:40] your reward function is differentiable can you only use it with markof decision
[00:53:41] can you only use it with markof decision process is it Ely useful mostly for
[00:53:44] process is it Ely useful mostly for infinite Horizon tasks a and b a b and c
[00:53:47] infinite Horizon tasks a and b a b and c none of the above are not sure let's
[00:53:49] none of the above are not sure let's just take a second to do that
[00:54:00] all right we have a good split at the
[00:54:02] all right we have a good split at the end nobody is not sure um but there is a
[00:54:05] end nobody is not sure um but there is a lot of spread so why don't you talk to
[00:54:07] lot of spread so why don't you talk to your neighbor and see if we can come to
[00:54:09] your neighbor and see if we can come to more consensus
[00:54:28] all right Sor interrup some good
[00:54:30] all right Sor interrup some good discussions but I want to make sure we
[00:54:32] discussions but I want to make sure we get through reinforced today so um so
[00:54:34] get through reinforced today so um so this a little bit of a tricky one in
[00:54:36] this a little bit of a tricky one in fact when I was giving it to one of my
[00:54:37] fact when I was giving it to one of my Tas I forgot to put the none of the
[00:54:39] Tas I forgot to put the none of the above and he was like wait what so um so
[00:54:42] above and he was like wait what so um so it's none of the above um it and so the
[00:54:44] it's none of the above um it and so the first one's part of the actual elegant
[00:54:46] first one's part of the actual elegant aspect of policy gradients so as you can
[00:54:49] aspect of policy gradients so as you can see here you need the policy function to
[00:54:52] see here you need the policy function to be differentiable but the reward
[00:54:54] be differentiable but the reward function does not have to be the reward
[00:54:56] function does not have to be the reward function is not a function of the policy
[00:54:58] function is not a function of the policy in the way that we've written it here so
[00:54:59] in the way that we've written it here so that's pretty elegant so that that has
[00:55:01] that's pretty elegant so that that has motivated people in a really wide range
[00:55:04] motivated people in a really wide range of areas where you really might have
[00:55:05] of areas where you really might have very complicated reward functions um to
[00:55:08] very complicated reward functions um to be interested in using what we're going
[00:55:09] be interested in using what we're going to see soon which is reinforce which is
[00:55:11] to see soon which is reinforce which is based on this idea because you just need
[00:55:13] based on this idea because you just need the policy parameterization um to be
[00:55:15] the policy parameterization um to be differentiable so that's really cool um
[00:55:18] differentiable so that's really cool um B doesn't have to be markof because as
[00:55:21] B doesn't have to be markof because as we saw the Dynamics model drops out and
[00:55:23] we saw the Dynamics model drops out and so was just saying in that case it
[00:55:25] so was just saying in that case it doesn't um doesn't appear at all so
[00:55:27] doesn't um doesn't appear at all so doesn't need to be mark off you don't
[00:55:28] doesn't need to be mark off you don't need differentiability um we are
[00:55:30] need differentiability um we are assuming that it's finded Horizon uh so
[00:55:33] assuming that it's finded Horizon uh so that we can actually episodic so we can
[00:55:35] that we can actually episodic so we can get M more than one if it was infinite
[00:55:37] get M more than one if it was infinite Horizon we'd only get nals 1 so so all
[00:55:40] Horizon we'd only get nals 1 so so all three of these are
[00:55:43] false
[00:55:45] false so let me just make sure I Circle
[00:55:50] that okay so just to give brief
[00:55:52] that okay so just to give brief intuitions because to make sure that we
[00:55:53] intuitions because to make sure that we get to reinforce um you can think of if
[00:55:56] get to reinforce um you can think of if this is a generic way of writing this
[00:55:58] this is a generic way of writing this down we sort of have some function times
[00:56:00] down we sort of have some function times the derivative of log of some other
[00:56:02] the derivative of log of some other probability function and you can think
[00:56:04] probability function and you can think of this first part is measuring how good
[00:56:06] of this first part is measuring how good a sample is and what the idea is is that
[00:56:09] a sample is and what the idea is is that when you have the derivative you're
[00:56:10] when you have the derivative you're trying to sort of move up the log
[00:56:13] trying to sort of move up the log probability of samples that have high
[00:56:16] probability of samples that have high reward because you generally want
[00:56:18] reward because you generally want policies that visit parts of the state
[00:56:19] policies that visit parts of the state in action space where we get high reward
[00:56:21] in action space where we get high reward so that's kind of the intuition and the
[00:56:23] so that's kind of the intuition and the nice thing is that that F doesn't have
[00:56:25] nice thing is that that F doesn't have to be differentiable it could be
[00:56:27] to be differentiable it could be discontinuous could be unknown as long
[00:56:29] discontinuous could be unknown as long as you can get samples from it um so it
[00:56:32] as you can get samples from it um so it could be extremely flexible to what is
[00:56:33] could be extremely flexible to what is that reward or objective
[00:56:36] that reward or objective function so I put a couple slides here
[00:56:40] function so I put a couple slides here um I believe remember if it was John
[00:56:43] um I believe remember if it was John Schulman who originally had these ones
[00:56:45] Schulman who originally had these ones um I put some credits at the front but
[00:56:47] um I put some credits at the front but you can think of sort of com taking a
[00:56:48] you can think of sort of com taking a combination between what the probability
[00:56:50] combination between what the probability is of your input of your X as well as
[00:56:52] is of your input of your X as well as your function so in our case that's
[00:56:53] your function so in our case that's going to be the reward function this is
[00:56:56] going to be the reward function this is is generally going to be the reward
[00:56:57] is generally going to be the reward function over trajectories and this is
[00:56:59] function over trajectories and this is going to be our
[00:57:01] going to be our policy it gives us probabilities of the
[00:57:04] policy it gives us probabilities of the trajectories and so you can think of
[00:57:05] trajectories and so you can think of combining between these two to actually
[00:57:08] combining between these two to actually change your parameter
[00:57:11] change your parameter space just to give a little bit of
[00:57:13] space just to give a little bit of intuition over what the what this sort
[00:57:15] intuition over what the what this sort of um gradient estimation is
[00:57:18] of um gradient estimation is doing
[00:57:20] doing okay so in general we can also write
[00:57:22] okay so in general we can also write down a policy gradient theorem which
[00:57:24] down a policy gradient theorem which says we could either use something like
[00:57:27] says we could either use something like episodic
[00:57:28] episodic reward or we could be trying to look at
[00:57:31] reward or we could be trying to look at average reward per time step or we could
[00:57:33] average reward per time step or we could be trying to look at average value and
[00:57:35] be trying to look at average value and in all of these cases we can end up
[00:57:38] in all of these cases we can end up writing something that looks really
[00:57:39] writing something that looks really similar to the equation I showed you
[00:57:41] similar to the equation I showed you before which is the derivative with
[00:57:43] before which is the derivative with respect to these value functions or
[00:57:46] respect to these value functions or something like a value function is going
[00:57:48] something like a value function is going to look something like the derivative
[00:57:50] to look something like the derivative with this score function the expected
[00:57:52] with this score function the expected value over the trajectories you're going
[00:57:54] value over the trajectories you're going to get of the log of the parameters
[00:57:57] to get of the log of the parameters times the Q function or the um the
[00:58:02] times the Q function or the um the return for that particular State action
[00:58:05] return for that particular State action pair following the
[00:58:06] pair following the policy okay and there's a nice D
[00:58:09] policy okay and there's a nice D derivation in s and Bardo about that at
[00:58:12] derivation in s and Bardo about that at a high level I think the useful thing to
[00:58:13] a high level I think the useful thing to know here is just that um there's many
[00:58:16] know here is just that um there's many we can extend it Beyond just thinking of
[00:58:17] we can extend it Beyond just thinking of kind of like this sample of return we
[00:58:20] kind of like this sample of return we can think of they being Q functions all
[00:58:22] can think of they being Q functions all right now what I've shown you so far uh
[00:58:25] right now what I've shown you so far uh is something that is correct and we can
[00:58:27] is something that is correct and we can turn it into an algorithm but it does
[00:58:29] turn it into an algorithm but it does not leverage much of the temporal
[00:58:31] not leverage much of the temporal structure so what do I mean by that so
[00:58:34] structure so what do I mean by that so what we've written down here is a valid
[00:58:35] what we've written down here is a valid gradient it's unbiased um but it can be
[00:58:38] gradient it's unbiased um but it can be very noisy so we're estimating this by
[00:58:41] very noisy so we're estimating this by Monte Carlo method because we have these
[00:58:43] Monte Carlo method because we have these M samples and as we know from Monte
[00:58:45] M samples and as we know from Monte Carlo methods before they are unbiased
[00:58:47] Carlo methods before they are unbiased but they can be very high variant and so
[00:58:50] but they can be very high variant and so some of the ways to make this more
[00:58:51] some of the ways to make this more practical and what I mean by that is the
[00:58:53] practical and what I mean by that is the better estimate of the gradient um and
[00:58:55] better estimate of the gradient um and hopefully with less data because
[00:58:57] hopefully with less data because ultimately we're going to have to be
[00:58:59] ultimately we're going to have to be using this information to update our
[00:59:00] using this information to update our weights to try to get to a good policy
[00:59:03] weights to try to get to a good policy so we want this to be data efficient is
[00:59:05] so we want this to be data efficient is we can try to leverage the temporal
[00:59:06] we can try to leverage the temporal structure and we can also include
[00:59:08] structure and we can also include baselines right so let's first see the
[00:59:11] baselines right so let's first see the temporal structure so what we've done
[00:59:13] temporal structure so what we've done before is we've summed up all the
[00:59:15] before is we've summed up all the rewards from a whole trajectory and
[00:59:17] rewards from a whole trajectory and we've multiplied it by the sum of this
[00:59:20] we've multiplied it by the sum of this score function for the whole trajectory
[00:59:22] score function for the whole trajectory that's what we've done so far we can
[00:59:25] that's what we've done so far we can instead think of it as what if we have
[00:59:27] instead think of it as what if we have the gradient estimator for a single
[00:59:28] the gradient estimator for a single reward term okay so this is just you
[00:59:32] reward term okay so this is just you know for one time step we can think of
[00:59:35] know for one time step we can think of it for there which is we have that
[00:59:37] it for there which is we have that single time step times the score
[00:59:40] single time step times the score function for the remaining time steps or
[00:59:42] function for the remaining time steps or or for the for the time steps up to that
[00:59:44] or for the for the time steps up to that point
[00:59:46] point okay so it's like we just think of the
[00:59:48] okay so it's like we just think of the partial trajectory until we got that
[00:59:52] reward so we want to think about s of
[00:59:55] reward so we want to think about s of the Dera of this this is the reward we
[00:59:57] the Dera of this this is the reward we got at this time point so instead of
[00:59:58] got at this time point so instead of having this whole sum we just think of
[01:00:00] having this whole sum we just think of well what is sort of the trajectory that
[01:00:02] well what is sort of the trajectory that we got up to that time point and all of
[01:00:04] we got up to that time point and all of their score
[01:00:06] their score functions
[01:00:08] functions okay does that make sense have questions
[01:00:11] okay does that make sense have questions about that
[01:00:14] part okay so this is like for a single
[01:00:17] part okay so this is like for a single time step T Prime and so now what we can
[01:00:20] time step T Prime and so now what we can do is we
[01:00:22] do is we can sum this overall time steps so
[01:00:26] can sum this overall time steps so instead of having the sum of all overall
[01:00:28] instead of having the sum of all overall rewards times this we can say well we
[01:00:31] rewards times this we can say well we know that for one time step it is equal
[01:00:33] know that for one time step it is equal to the expected value of the reward for
[01:00:34] to the expected value of the reward for that time step multiply times these
[01:00:38] that time step multiply times these score functions up to that point so
[01:00:40] score functions up to that point so let's just rewrite it like that now
[01:00:41] let's just rewrite it like that now we're just going to some over the
[01:00:42] we're just going to some over the rewards we got for all time
[01:00:45] steps all
[01:00:47] steps all right so now what we can do is we can do
[01:00:51] right so now what we can do is we can do slight rearrangement okay so what we can
[01:00:54] slight rearrangement okay so what we can notice is that for each of the points so
[01:00:57] notice is that for each of the points so you can think of it as I have so this is
[01:01:01] you can think of it as I have so this is t 0 1 2 3 and you can think of all of
[01:01:06] t 0 1 2 3 and you can think of all of these sort
[01:01:07] these sort of score functions I have at each time
[01:01:10] of score functions I have at each time point so the score function at time Step
[01:01:13] point so the score function at time Step Zero is going to appear for r0 R1 R2 R3
[01:01:19] Zero is going to appear for r0 R1 R2 R3 dot dot dot okay so that's what I've
[01:01:23] dot dot dot okay so that's what I've just done here I've said this first term
[01:01:25] just done here I've said this first term is going to appear here
[01:01:27] is going to appear here in with a reward function of all of the
[01:01:31] in with a reward function of all of the subsequent time
[01:01:33] subsequent time points okay because that decision
[01:01:36] points okay because that decision happened and then we got a reward and
[01:01:37] happened and then we got a reward and then we got a whole bunch of rewards
[01:01:40] then we got a whole bunch of rewards later the on the first time step it can
[01:01:43] later the on the first time step it can affect the reward you get at time step
[01:01:45] affect the reward you get at time step one and all the future after that okay
[01:01:49] one and all the future after that okay so this term here this score function
[01:01:52] so this term here this score function can
[01:01:53] can influence um the one that we get on time
[01:01:55] influence um the one that we get on time step one can influence time step one all
[01:01:58] step one can influence time step one all the way out to the end of the episode
[01:02:00] the way out to the end of the episode the one we get on time step two can
[01:02:01] the one we get on time step two can influence time step two all the way out
[01:02:03] influence time step two all the way out to the end of the
[01:02:06] to the end of the episode so essentially this is like
[01:02:08] episode so essentially this is like saying my reward on time step three
[01:02:10] saying my reward on time step three cannot be impacted by decisions I make
[01:02:12] cannot be impacted by decisions I make on time step four time only flows one
[01:02:15] on time step four time only flows one way so if we think about what those
[01:02:17] way so if we think about what those score functions were and like we think
[01:02:19] score functions were and like we think of like the trajectories that were
[01:02:20] of like the trajectories that were generated they're a temporal structure
[01:02:23] generated they're a temporal structure and so it means that we cannot have
[01:02:26] and so it means that we cannot have um if we change the policy parameters
[01:02:29] um if we change the policy parameters such that decisions in the future change
[01:02:31] such that decisions in the future change that can't affect my reward on earlier
[01:02:33] that can't affect my reward on earlier time steps okay so this is leveraging
[01:02:35] time steps okay so this is leveraging the temporal
[01:02:36] the temporal structure so this just allows us to
[01:02:39] structure so this just allows us to rewrite the equation so that now we have
[01:02:43] rewrite the equation so that now we have for each of the different score
[01:02:44] for each of the different score functions essentially which of the
[01:02:46] functions essentially which of the rewards they
[01:02:48] rewards they influence and the reason this is
[01:02:50] influence and the reason this is important is because here you could see
[01:02:53] important is because here you could see that we are multiplying each of the
[01:02:54] that we are multiplying each of the score functions by all of the
[01:02:57] score functions by all of the rewards and now we're only going to
[01:02:59] rewards and now we're only going to multiply them by the rewards they
[01:03:01] multiply them by the rewards they influence and so in general that's going
[01:03:03] influence and so in general that's going to be way less than having the full set
[01:03:04] to be way less than having the full set of rewards so this is going to reduce
[01:03:07] of rewards so this is going to reduce the variance of our estimator without
[01:03:09] the variance of our estimator without causing any bias just leveraging the
[01:03:11] causing any bias just leveraging the fact that decisions in the future can't
[01:03:13] fact that decisions in the future can't affect your rewards in the
[01:03:17] past all right so that is one of the
[01:03:20] past all right so that is one of the first things that we're going to um do
[01:03:22] first things that we're going to um do in this
[01:03:23] in this case so we're going to write um so
[01:03:26] case so we're going to write um so remember in this case that if we sum up
[01:03:28] remember in this case that if we sum up all the rewards from the current time
[01:03:30] all the rewards from the current time step to the end we've just called that
[01:03:31] step to the end we've just called that the return we've seen that before from
[01:03:33] the return we've seen that before from multi
[01:03:34] multi Carlo so we can just rewrite this
[01:03:36] Carlo so we can just rewrite this expression like
[01:03:38] expression like that and that gives us the reinforce
[01:03:41] that and that gives us the reinforce algorithm so this is the reinforce
[01:03:43] algorithm so this is the reinforce algorithm that has been incredibly
[01:03:44] algorithm that has been incredibly influential um in NLP and Robotics and
[01:03:47] influential um in NLP and Robotics and many many areas okay and so what this
[01:03:50] many many areas okay and so what this says here is that the way we change our
[01:03:53] says here is that the way we change our parameter is just our learning rate
[01:03:55] parameter is just our learning rate times our score function times the
[01:03:58] times our score function times the return we got from that time step till
[01:04:00] return we got from that time step till the end of the episode so we still have
[01:04:02] the end of the episode so we still have to wait till the end of the episode to
[01:04:04] to wait till the end of the episode to update anything but what happens is we
[01:04:06] update anything but what happens is we run a full episode with our current
[01:04:07] run a full episode with our current policy and then for each time step we
[01:04:10] policy and then for each time step we slightly change our policy parameters by
[01:04:12] slightly change our policy parameters by using a learning rate the score function
[01:04:14] using a learning rate the score function for that time step plus the return we
[01:04:17] for that time step plus the return we got from that time step till the end of
[01:04:18] got from that time step till the end of the
[01:04:19] the episode and then we just steep through
[01:04:21] episode and then we just steep through that for the whole episode and that's
[01:04:23] that for the whole episode and that's given us T different updates to our
[01:04:25] given us T different updates to our policy
[01:04:27] policy parameterization and then we just repeat
[01:04:29] parameterization and then we just repeat over and over and over again and what
[01:04:32] over and over and over again and what that guarantees to us is that eventually
[01:04:34] that guarantees to us is that eventually we will land in a local Optima of um the
[01:04:37] we will land in a local Optima of um the value function for the policy
[01:04:43] parameterization so this is called Monte
[01:04:45] parameterization so this is called Monte Carlo policy gradient or known as the
[01:04:46] Carlo policy gradient or known as the reinforce I believe this was in roughly
[01:04:50] reinforce I believe this was in roughly 1992 it's about 30 years ago there's
[01:04:53] 1992 it's about 30 years ago there's been many many many Poli grading
[01:04:56] been many many many Poli grading algorithms are built on this
[01:04:59] idea okay now when you're looking at
[01:05:02] idea okay now when you're looking at this you might still be concerned that
[01:05:04] this you might still be concerned that if from remembering back from the Monte
[01:05:06] if from remembering back from the Monte Carlo methods we've covered that this
[01:05:09] Carlo methods we've covered that this estimate G can often be pretty high
[01:05:11] estimate G can often be pretty high variance so in general if you're just
[01:05:14] variance so in general if you're just directly kind of averaging over sample
[01:05:16] directly kind of averaging over sample returns that might be high
[01:05:17] returns that might be high variance so one of the next fixes we can
[01:05:20] variance so one of the next fixes we can do and we'll get to this more on um
[01:05:22] do and we'll get to this more on um Wednesday is to introduce a Baseline
[01:05:27] and see I'll just say so the the goals
[01:05:30] and see I'll just say so the the goals here is that we're going to hopefully
[01:05:31] here is that we're going to hopefully try to converge as quickly as possible
[01:05:32] try to converge as quickly as possible to local Optima so we want to reduce the
[01:05:36] to local Optima so we want to reduce the um the variance over our gradient
[01:05:38] um the variance over our gradient estimate and so the Baseline is going to
[01:05:41] estimate and so the Baseline is going to allow us to hopefully reduce well in
[01:05:43] allow us to hopefully reduce well in general yes reduce the the variance over
[01:05:45] general yes reduce the the variance over this estimation
[01:05:47] this estimation process and kind of the we'll see two
[01:05:51] process and kind of the we'll see two ideas next which is introducing a
[01:05:54] ideas next which is introducing a Baseline and and then thinking about an
[01:05:56] Baseline and and then thinking about an alternative to the Monte Carlo
[01:05:58] alternative to the Monte Carlo returns so those are the ideas that
[01:06:01] returns so those are the ideas that we're going to go through next I guess
[01:06:02] we're going to go through next I guess I'll just do one more thing and we'll we
[01:06:05] I'll just do one more thing and we'll we we'll go through the proof of it next
[01:06:06] we'll go through the proof of it next time so I'll just introduce the concept
[01:06:07] time so I'll just introduce the concept of Baseline and then we'll we'll prove
[01:06:09] of Baseline and then we'll we'll prove it next time so the idea in this case is
[01:06:11] it next time so the idea in this case is that we're just going to subtract
[01:06:12] that we're just going to subtract something off and we're going to
[01:06:14] something off and we're going to subtract something off that only depends
[01:06:16] subtract something off that only depends on the
[01:06:17] on the state this
[01:06:19] state this is only depends on the
[01:06:24] is only depends on the state okay this is not a function of
[01:06:26] state okay this is not a function of your policy only depends on the state
[01:06:29] your policy only depends on the state and it will turn out and we'll prove
[01:06:31] and it will turn out and we'll prove this next time it's pretty elegant that
[01:06:34] this next time it's pretty elegant that for any choice of something that only
[01:06:35] for any choice of something that only depends on your state the gradient
[01:06:37] depends on your state the gradient estimator is still
[01:06:39] estimator is still unbiased so you could subtract off
[01:06:41] unbiased so you could subtract off anything there that is only a function
[01:06:43] anything there that is only a function of your state and you didn't change the
[01:06:45] of your state and you didn't change the bias of your estimator which is kind of
[01:06:48] bias of your estimator which is kind of wild um and we'll prove that next
[01:06:51] wild um and we'll prove that next time and but the goal is that we can
[01:06:53] time and but the goal is that we can hopefully reduce the variance of our
[01:06:55] hopefully reduce the variance of our estimated gradient by subtracting off
[01:06:57] estimated gradient by subtracting off the right thing okay and just
[01:07:00] the right thing okay and just intuitively the way to think about the
[01:07:02] intuitively the way to think about the Baseline is that you don't necess just
[01:07:05] Baseline is that you don't necess just care about whether or not the gradient
[01:07:07] care about whether or not the gradient is positive or negative and whether
[01:07:09] is positive or negative and whether returns were um good or bad you might
[01:07:12] returns were um good or bad you might care about like well how much better or
[01:07:14] care about like well how much better or worse are these returns compared to
[01:07:16] worse are these returns compared to something else I could have done like I
[01:07:18] something else I could have done like I want to know whether this policy a is
[01:07:19] want to know whether this policy a is better than policy B and maybe both of
[01:07:21] better than policy B and maybe both of them give you positive returns one of
[01:07:23] them give you positive returns one of them gives you 100 and one of them gives
[01:07:24] them gives you 100 and one of them gives you 90
[01:07:26] you 90 but you'd really like the one with 100
[01:07:27] but you'd really like the one with 100 so you'd really like to move your policy
[01:07:29] so you'd really like to move your policy parameters in the direction of stuff
[01:07:31] parameters in the direction of stuff that is better than other Alternatives
[01:07:34] that is better than other Alternatives and that's kind of the idea of a
[01:07:35] and that's kind of the idea of a baseline is to say like well maybe I
[01:07:37] baseline is to say like well maybe I know that I could probably always get
[01:07:38] know that I could probably always get like 90 for this particular State how
[01:07:41] like 90 for this particular State how much better is this policy for this
[01:07:42] much better is this policy for this state compared to something I could do
[01:07:43] state compared to something I could do on
[01:07:44] on average and so we're going to sort of
[01:07:46] average and so we're going to sort of intuitively increase the log probability
[01:07:48] intuitively increase the log probability of an action proportionally to how much
[01:07:50] of an action proportionally to how much its returns were better than expected
[01:07:53] its returns were better than expected where kind of the Baseline is giving you
[01:07:55] where kind of the Baseline is giving you that expected
[01:07:56] that expected value and we'll see formally on
[01:07:59] value and we'll see formally on Wednesday how that by doing this with
[01:08:01] Wednesday how that by doing this with the Baseline it doesn't introduce any
[01:08:02] the Baseline it doesn't introduce any bias so this is going to be one of the
[01:08:04] bias so this is going to be one of the ways that we're going to get better uh
[01:08:06] ways that we're going to get better uh better ingredients the other thing that
[01:08:08] better ingredients the other thing that we're going to do on Wednesday is we're
[01:08:09] we're going to do on Wednesday is we're at least going to start talking about po
[01:08:11] at least going to start talking about po which is the second which is part of
[01:08:13] which is the second which is part of your homework too bless you um which is
[01:08:16] your homework too bless you um which is going to involve more ways to kind of be
[01:08:18] going to involve more ways to kind of be more efficient and effective in the
[01:08:19] more efficient and effective in the policy that we do I'll see you then
[01:08:21] policy that we do I'll see you then thanks