🤖 CS224N – NLP with Deep Learning

Stanford CS224N: NLP with Deep Learning | Spring 2024 | Lecture 1 - Intro and Word Vectors

Source: https://www.youtube.com/watch?v=DzpHeXVSC5I

---

Transcript

[00:00:05] so the thing that seems kind of amazing
[00:00:07] so the thing that seems kind of amazing um to me and us is the fact that well
[00:00:11] um to me and us is the fact that well actually this course was taught um just
[00:00:14] actually this course was taught um just last quarter and here we are with the
[00:00:17] last quarter and here we are with the enormous number of people again taking
[00:00:19] enormous number of people again taking this class um I guess that says
[00:00:22] this class um I guess that says something maybe approximately what it
[00:00:24] something maybe approximately what it says is chat
[00:00:26] says is chat GPT um um
[00:00:31] GPT um um um but anyway it's great to have you all
[00:00:33] um but anyway it's great to have you all lots of exciting content to have and um
[00:00:36] lots of exciting content to have and um hope you'll all enjoy it so let me get
[00:00:39] hope you'll all enjoy it so let me get started and um start telling you a bit
[00:00:42] started and um start telling you a bit about the course um before diving
[00:00:44] about the course um before diving straight into um today's content um for
[00:00:47] straight into um today's content um for people still coming in you know there
[00:00:50] people still coming in you know there are oodles of seats still right on
[00:00:52] are oodles of seats still right on either side especially down near the
[00:00:54] either side especially down near the front there are tons of seats so um do
[00:00:58] front there are tons of seats so um do feel um empowered to go out and seek
[00:01:02] feel um empowered to go out and seek those seats if other if people on the
[00:01:04] those seats if other if people on the corridors are really nice they could
[00:01:06] corridors are really nice they could even move towards the edges to make it
[00:01:08] even move towards the edges to make it easier for people but um One Way or
[00:01:10] easier for people but um One Way or Another feel free to find a seat okay so
[00:01:14] Another feel free to find a seat okay so this is the plan for what I want to get
[00:01:15] this is the plan for what I want to get through today so first of all I'm going
[00:01:18] through today so first of all I'm going to tell you about the course for a few
[00:01:20] to tell you about the course for a few minutes um then um have a few remarks
[00:01:23] minutes um then um have a few remarks about human language and word meaning um
[00:01:27] about human language and word meaning um then the main technical thing we want to
[00:01:29] then the main technical thing we want to get into today today is start learning
[00:01:31] get into today today is start learning about the word Tove algorithm so the
[00:01:34] about the word Tove algorithm so the word Tove algorithm is slightly over a
[00:01:36] word Tove algorithm is slightly over a decade old now it was introduced in um
[00:01:41] decade old now it was introduced in um 2013 but it was a wildly successful
[00:01:45] 2013 but it was a wildly successful simple way of learning Vector
[00:01:47] simple way of learning Vector representations of words so I want to
[00:01:49] representations of words so I want to show you um that as a sort of a first
[00:01:52] show you um that as a sort of a first easy baby system for the kind of neural
[00:01:55] easy baby system for the kind of neural representations that we're going to talk
[00:01:57] representations that we're going to talk about in class um with then going to get
[00:02:00] about in class um with then going to get more concrete with that looking at its
[00:02:02] more concrete with that looking at its objective function gradients and
[00:02:04] objective function gradients and optimization and then hopefully if all
[00:02:07] optimization and then hopefully if all goes I stick to schedule spend a few
[00:02:09] goes I stick to schedule spend a few minutes just um playing around an i
[00:02:12] minutes just um playing around an i python notebook um huh I'm going to have
[00:02:15] python notebook um huh I'm going to have to change computers for that um than um
[00:02:18] to change computers for that um than um sort of um seeing some of the things you
[00:02:20] sort of um seeing some of the things you can do with this okay so this is the um
[00:02:24] can do with this okay so this is the um course Logistics and brief I'm
[00:02:26] course Logistics and brief I'm Christopher Manning hi again everyone um
[00:02:29] Christopher Manning hi again everyone um the head ta is who unfortunately has a
[00:02:33] the head ta is who unfortunately has a bit of a health problem so he's not
[00:02:34] bit of a health problem so he's not actually here today um we've got a
[00:02:36] actually here today um we've got a course manager for the course who
[00:02:39] course manager for the course who is who is up the back there um and then
[00:02:43] is who is up the back there um and then we've got a whole um lot of teas if
[00:02:46] we've got a whole um lot of teas if you're a TA who's here you could stand
[00:02:48] you're a TA who's here you could stand up and wave or something like that so
[00:02:50] up and wave or something like that so people can see a few of the Tas and see
[00:02:54] people can see a few of the Tas and see some friendly faces um okay we've got
[00:02:57] some friendly faces um okay we've got some Tas um and some other ones and so
[00:03:00] some Tas um and some other ones and so you can look at them on the website um
[00:03:02] you can look at them on the website um if you're here you know what time the
[00:03:04] if you're here you know what time the class is um there's an email list but
[00:03:08] class is um there's an email list but preferably don't use it and use the Ed
[00:03:11] preferably don't use it and use the Ed site that you can find on the course
[00:03:13] site that you can find on the course website so the main place to go and look
[00:03:16] website so the main place to go and look for information is the course website
[00:03:18] for information is the course website which we've got up here and that then
[00:03:20] which we've got up here and that then links in to Ed which is what we're going
[00:03:23] links in to Ed which is what we're going to use as the main discussion board
[00:03:25] to use as the main discussion board please use that rather than sending
[00:03:28] please use that rather than sending emails um to um the first assignment for
[00:03:32] emails um to um the first assignment for this class it's a sort of an easy one
[00:03:34] this class it's a sort of an easy one it's the warm-up assignment but we want
[00:03:36] it's the warm-up assignment but we want to get people busy and doing stuff um
[00:03:39] to get people busy and doing stuff um straight away so the first assignment is
[00:03:41] straight away so the first assignment is already live on the web page and it's
[00:03:44] already live on the web page and it's due next Tuesday before class so um
[00:03:48] due next Tuesday before class so um you're slightly less than seven days
[00:03:50] you're slightly less than seven days left to do it um so do get started on
[00:03:53] left to do it um so do get started on that um and to help with that um we're
[00:03:56] that um and to help with that um we're going to be immediately starting office
[00:03:58] going to be immediately starting office hours um tomorrow and they're also
[00:04:00] hours um tomorrow and they're also described on the website um we also do a
[00:04:03] described on the website um we also do a few tutorials on Friday um the first of
[00:04:07] few tutorials on Friday um the first of these tutorials is a tutorial on Python
[00:04:09] these tutorials is a tutorial on Python and numpy um many people don't need that
[00:04:12] and numpy um many people don't need that because they've done other classes and
[00:04:14] because they've done other classes and done this um but for some people we try
[00:04:17] done this um but for some people we try and make this class accessible to
[00:04:19] and make this class accessible to everybody so if you'd like to um brush
[00:04:21] everybody so if you'd like to um brush up a bit on python or how to use numpy
[00:04:24] up a bit on python or how to use numpy it's a great thing to go along to and
[00:04:26] it's a great thing to go along to and who's right over there is going to be
[00:04:28] who's right over there is going to be teaching it on Friday
[00:04:30] teaching it on Friday today okay what do we hope to teach you
[00:04:34] today okay what do we hope to teach you know at the end of the quarter when you
[00:04:36] know at the end of the quarter when you get the evow you'll be asked to rate
[00:04:39] get the evow you'll be asked to rate whether this class met its learning
[00:04:42] whether this class met its learning goals these are my learning goals um
[00:04:47] goals these are my learning goals um what are they so the first one um is to
[00:04:50] what are they so the first one um is to teach you about the foundations and
[00:04:53] teach you about the foundations and current methods for using deep learning
[00:04:56] current methods for using deep learning applied to natural language processing
[00:04:58] applied to natural language processing so this class
[00:05:00] so this class tries to sort of build up from the
[00:05:02] tries to sort of build up from the bottom up so we start off doing simple
[00:05:04] bottom up so we start off doing simple things like word vectors and feed
[00:05:06] things like word vectors and feed forward neural networks recurrent
[00:05:07] forward neural networks recurrent networks and attention we then fairly
[00:05:10] networks and attention we then fairly quickly move into the kind of key
[00:05:12] quickly move into the kind of key methods they Ed for NLP um in
[00:05:16] methods they Ed for NLP um in 2024 I wrote down here Transformers and
[00:05:19] 2024 I wrote down here Transformers and coder decoder models I probably should
[00:05:21] coder decoder models I probably should have written large language models
[00:05:22] have written large language models somewhere in this list as well um but
[00:05:25] somewhere in this list as well um but then pre-training and post-training of
[00:05:27] then pre-training and post-training of large language models adaptation model
[00:05:30] large language models adaptation model interpretability agents Etc but that's
[00:05:33] interpretability agents Etc but that's not the only thing that we want to do so
[00:05:35] not the only thing that we want to do so there are a couple of other things that
[00:05:36] there are a couple of other things that we crucially want to achieve um the
[00:05:39] we crucially want to achieve um the second is to give you
[00:05:42] second is to give you some understanding of human languages
[00:05:45] some understanding of human languages and the difficulties in understanding
[00:05:47] and the difficulties in understanding and producing them on computers now
[00:05:50] and producing them on computers now there are few of you in this class who
[00:05:51] there are few of you in this class who are Linguistics Majors or perhaps the
[00:05:54] are Linguistics Majors or perhaps the symbolic systems Majors yay to the
[00:05:56] symbolic systems Majors yay to the symbolic systems Majors but for quite a
[00:05:59] symbolic systems Majors but for quite a few of the rest of you um you'll never
[00:06:01] few of the rest of you um you'll never see any uh Linguistics in the sense of
[00:06:05] see any uh Linguistics in the sense of understanding how language Works apart
[00:06:07] understanding how language Works apart from this class so we do want to try and
[00:06:10] from this class so we do want to try and convey a little bit of a sense of what
[00:06:12] convey a little bit of a sense of what some of the issues are in language
[00:06:14] some of the issues are in language structure and why it's proven to be
[00:06:18] structure and why it's proven to be quite difficult um to get computers to
[00:06:21] quite difficult um to get computers to understand human languages even though
[00:06:23] understand human languages even though humans seem very good at learning to
[00:06:26] humans seem very good at learning to understand each other and then the final
[00:06:28] understand each other and then the final thing that we want to make it on to um
[00:06:31] thing that we want to make it on to um is actually concretely Building Systems
[00:06:34] is actually concretely Building Systems so that this isn't just a theory class
[00:06:37] so that this isn't just a theory class that we actually want you to leave this
[00:06:39] that we actually want you to leave this class thinking oh yeah in my first job
[00:06:43] class thinking oh yeah in my first job wherever you go whether it's a startup
[00:06:45] wherever you go whether it's a startup or a big Tech or um some nonprofit oh
[00:06:48] or a big Tech or um some nonprofit oh there's something they want to do that
[00:06:51] there's something they want to do that they'd like that would be useful if we
[00:06:52] they'd like that would be useful if we had a text classification system or we
[00:06:55] had a text classification system or we did information extraction to get some
[00:06:57] did information extraction to get some kind of facts out of documents I know to
[00:06:59] kind of facts out of documents I know to build that I can build that system
[00:07:02] build that I can build that system because I did
[00:07:05] cs224n okay um here's how you get graded
[00:07:08] cs224n okay um here's how you get graded um so we have four assignments mainly
[00:07:12] um so we have four assignments mainly one and a half weeks long apart from the
[00:07:14] one and a half weeks long apart from the first one they make up almost half the
[00:07:16] first one they make up almost half the grade the other half of the grade is
[00:07:19] grade the other half of the grade is made up of a final project which there
[00:07:22] made up of a final project which there are two variants of a custom or default
[00:07:25] are two variants of a custom or default final project which we'll get on to in a
[00:07:28] final project which we'll get on to in a minute um and then there's a few percent
[00:07:30] minute um and then there's a few percent that go for
[00:07:32] that go for participation um six late
[00:07:35] participation um six late days um collaboration policy um like all
[00:07:39] days um collaboration policy um like all other CS classes we've had issues with
[00:07:43] other CS classes we've had issues with people not doing their own work we
[00:07:45] people not doing their own work we really do want you to learn things in
[00:07:47] really do want you to learn things in this class and the way you do that is by
[00:07:50] this class and the way you do that is by doing your own work um so make sure you
[00:07:53] doing your own work um so make sure you understand that um and so for the
[00:07:56] understand that um and so for the assignments everyone is expected to do
[00:07:58] assignments everyone is expected to do their own assign assignments um you can
[00:08:01] their own assign assignments um you can talk to your friends but you're expected
[00:08:02] talk to your friends but you're expected to do your own assignment for the final
[00:08:05] to do your own assignment for the final project you can do that as a group um
[00:08:08] project you can do that as a group um then we have the issue of um AI tools
[00:08:12] then we have the issue of um AI tools now of course in this class we love
[00:08:14] now of course in this class we love large language models but nevertheless
[00:08:17] large language models but nevertheless we don't want you to do your assignments
[00:08:19] we don't want you to do your assignments by saying hey chat GPT could you answer
[00:08:22] by saying hey chat GPT could you answer question three for me um that is not the
[00:08:25] question three for me um that is not the way to learn things um if you want to
[00:08:27] way to learn things um if you want to make use of um AI as a tool to assist
[00:08:30] make use of um AI as a tool to assist you such as for coding assistance go for
[00:08:33] you such as for coding assistance go for it um but we're wanting you to be
[00:08:36] it um but we're wanting you to be working out how to answer assignment
[00:08:38] working out how to answer assignment questions by
[00:08:40] questions by yourself okay so this is what the
[00:08:42] yourself okay so this is what the assignments look like so assignment one
[00:08:45] assignments look like so assignment one is meant to be an easy onramp and it's
[00:08:47] is meant to be an easy onramp and it's done as a Jupiter notebook assignment
[00:08:51] done as a Jupiter notebook assignment two um then has people uh you know what
[00:08:57] two um then has people uh you know what can I say here we are at this fine
[00:09:00] can I say here we are at this fine liberal artarts and Engineering
[00:09:03] liberal artarts and Engineering institution we're not at a coding boot
[00:09:06] institution we're not at a coding boot camp so we hope that people have some
[00:09:08] camp so we hope that people have some deep understanding of how things work so
[00:09:11] deep understanding of how things work so in assignment two we actually want you
[00:09:14] in assignment two we actually want you to do some math and understand how
[00:09:18] to do some math and understand how things work in neural networks um so for
[00:09:21] things work in neural networks um so for some people assignment to is the
[00:09:23] some people assignment to is the scariest assignment in the whole class
[00:09:26] scariest assignment in the whole class um but then it's also the place where we
[00:09:28] um but then it's also the place where we introduce py talk which is software
[00:09:30] introduce py talk which is software package we use for building newal
[00:09:32] package we use for building newal networks and we build a dependency paa
[00:09:35] networks and we build a dependency paa which we'll get to later as something
[00:09:37] which we'll get to later as something more linguistic um then for assignment
[00:09:40] more linguistic um then for assignment three and four we move on to larger
[00:09:43] three and four we move on to larger projects using pytorch with gpus and
[00:09:45] projects using pytorch with gpus and we'll be making use of Google cloud and
[00:09:49] we'll be making use of Google cloud and um for those two assignments um we look
[00:09:52] um for those two assignments um we look at doing machine translation and getting
[00:09:55] at doing machine translation and getting information um out with Transformers and
[00:09:58] information um out with Transformers and then these are the two final project
[00:10:00] then these are the two final project options so essentially you know we have
[00:10:03] options so essentially you know we have a default final project where we give
[00:10:05] a default final project where we give you a lot of scaffolding and an outline
[00:10:07] you a lot of scaffolding and an outline of what to do but um it's still an
[00:10:10] of what to do but um it's still an open-ended project there are lots of
[00:10:12] open-ended project there are lots of different things you can try to make
[00:10:14] different things you can try to make this system work better and we encourage
[00:10:17] this system work better and we encourage you to explore um but nevertheless
[00:10:20] you to explore um but nevertheless you're given a leg up from quite a lot
[00:10:22] you're given a leg up from quite a lot of um scaffolding we'll talk about this
[00:10:25] of um scaffolding we'll talk about this more but you can either do that option
[00:10:27] more but you can either do that option or you can just come up with totally
[00:10:29] or you can just come up with totally your own project and do
[00:10:32] your own project and do that okay that's the course any
[00:10:35] that okay that's the course any questions on the
[00:10:37] questions on the course yes for the final project how are
[00:10:40] course yes for the final project how are mentors assigned um so if you if you can
[00:10:45] mentors assigned um so if you if you can find your own Mentor your interest in
[00:10:48] find your own Mentor your interest in something and there's someone that's
[00:10:50] something and there's someone that's happy to Mentor you that person can be
[00:10:52] happy to Mentor you that person can be your Mentor otherwise one of the course
[00:10:55] your Mentor otherwise one of the course Tas will be your mentor and how that
[00:10:58] Tas will be your mentor and how that person is assigned is uh one of the T
[00:11:02] person is assigned is uh one of the T who is in charge of final projects
[00:11:04] who is in charge of final projects assigns people and they do the best they
[00:11:07] assigns people and they do the best they can in terms of finding people with some
[00:11:09] can in terms of finding people with some expertise and having to divide all the
[00:11:11] expertise and having to divide all the students across the mentors roughly
[00:11:14] students across the mentors roughly equally any other
[00:11:19] questions okay I'll power ahead um human
[00:11:23] questions okay I'll power ahead um human language and word
[00:11:25] language and word meaning um so let me just sort of um say
[00:11:29] meaning um so let me just sort of um say a little bit about the big picture here
[00:11:32] a little bit about the big picture here um so we're in the area of artificial
[00:11:35] um so we're in the area of artificial intelligence and we've got this idea
[00:11:38] intelligence and we've got this idea that humans are intelligent and then
[00:11:41] that humans are intelligent and then there's the question of you know how
[00:11:44] there's the question of you know how does language um fit into that and you
[00:11:47] does language um fit into that and you know this is something that there is
[00:11:49] know this is something that there is some argument about and if you want to
[00:11:51] some argument about and if you want to you can run off in onto social media and
[00:11:54] you can run off in onto social media and read some of the arguments about these
[00:11:56] read some of the arguments about these things and contribute to it um if you
[00:11:58] things and contribute to it um if you wish too um but here is my perhaps bias
[00:12:02] wish too um but here is my perhaps bias take as a linguist um well you can
[00:12:06] take as a linguist um well you can compare human beings um to some of our
[00:12:10] compare human beings um to some of our nearest neighbors like chimpanzees
[00:12:13] nearest neighbors like chimpanzees bonobos and things like that and you
[00:12:16] bonobos and things like that and you know well one big distinguishing thing
[00:12:19] know well one big distinguishing thing is we have language and they don't um
[00:12:24] is we have language and they don't um but you know in most other respects um
[00:12:27] but you know in most other respects um chimps are very similar human beings
[00:12:30] chimps are very similar human beings right you know in they they can use
[00:12:33] right you know in they they can use tools they can plan how to solve things
[00:12:37] tools they can plan how to solve things um they've got really good memory um
[00:12:39] um they've got really good memory um chimps have better short-term memory
[00:12:41] chimps have better short-term memory than human beings do right so that in
[00:12:44] than human beings do right so that in most respects it's hard to show an
[00:12:47] most respects it's hard to show an intelligence difference between chimps
[00:12:49] intelligence difference between chimps and people except for the fact that we
[00:12:52] and people except for the fact that we have language but us having language has
[00:12:55] have language but us having language has been this enormous differentiator right
[00:12:59] been this enormous differentiator right that if you look around um what happened
[00:13:02] that if you look around um what happened on the planet you know that there are
[00:13:04] on the planet you know that there are creatures that are stronger than us
[00:13:06] creatures that are stronger than us faster than us more venomous than us
[00:13:09] faster than us more venomous than us have every possible Advantage um but
[00:13:11] have every possible Advantage um but human beings took over the whole place
[00:13:14] human beings took over the whole place and how did that happen we had language
[00:13:17] and how did that happen we had language um so we could communicate and that
[00:13:20] um so we could communicate and that communication allowed us to have human
[00:13:24] communication allowed us to have human ascendency but I'd like to me so one big
[00:13:27] ascendency but I'd like to me so one big role of language is the fact fact that
[00:13:29] role of language is the fact fact that it allows communication but I'd like to
[00:13:32] it allows communication but I'd like to suggest it's actually not the only role
[00:13:34] suggest it's actually not the only role of language that language has also
[00:13:38] of language that language has also allowed humans I would argue to achieve
[00:13:41] allowed humans I would argue to achieve a higher level of thought so there are
[00:13:44] a higher level of thought so there are various kinds of thoughts that you can
[00:13:46] various kinds of thoughts that you can have without any language involved you
[00:13:48] have without any language involved you know you can think about a scene you can
[00:13:51] know you can think about a scene you can move some bits of furniture around in
[00:13:52] move some bits of furniture around in your mind and there's no language and
[00:13:55] your mind and there's no language and obviously emotional responses of feeling
[00:13:58] obviously emotional responses of feeling scared or excited
[00:13:59] scared or excited they happen and there's no language
[00:14:01] they happen and there's no language involved but you know I think most of
[00:14:04] involved but you know I think most of the time when we're doing higher level
[00:14:07] the time when we're doing higher level cognition um if you're thinking to
[00:14:09] cognition um if you're thinking to yourself oh gee my friend seemed upset
[00:14:12] yourself oh gee my friend seemed upset about what I said last night my should
[00:14:15] about what I said last night my should probably work out how to fix that or
[00:14:17] probably work out how to fix that or maybe I could BL BL BL BL I think we
[00:14:20] maybe I could BL BL BL BL I think we think in language and plan out things
[00:14:22] think in language and plan out things and so that it's given us a scaffolding
[00:14:25] and so that it's given us a scaffolding to do much more detailed thought and
[00:14:27] to do much more detailed thought and planning
[00:14:29] planning most recently of all of course human
[00:14:32] most recently of all of course human beings invented ways to write um and
[00:14:36] beings invented ways to write um and that led so writing is really really
[00:14:39] that led so writing is really really recent I mean no one really knows how
[00:14:42] recent I mean no one really knows how old human languages are you know most
[00:14:44] old human languages are you know most people think a few 100,000 years not
[00:14:46] people think a few 100,000 years not very long by evolutionary time scales
[00:14:49] very long by evolutionary time scales but writing we do know writing is really
[00:14:52] but writing we do know writing is really really recent so writing is about 5,000
[00:14:55] really recent so writing is about 5,000 years old um and so but you know writing
[00:15:00] years old um and so but you know writing proved to be this um again this amazing
[00:15:04] proved to be this um again this amazing cognitive tool that just gave Humanity
[00:15:06] cognitive tool that just gave Humanity an enormous leg up because Suddenly It's
[00:15:09] an enormous leg up because Suddenly It's Not only that you could share
[00:15:11] Not only that you could share information and learn from the people
[00:15:13] information and learn from the people that were standing within 50 feet of you
[00:15:15] that were standing within 50 feet of you you could then um share knowledge across
[00:15:19] you could then um share knowledge across time and space so really um having
[00:15:22] time and space so really um having writing was enough to take us from the
[00:15:25] writing was enough to take us from the Bronze Age very simple um metal working
[00:15:28] Bronze Age very simple um metal working to the kind of um you know mobile phones
[00:15:31] to the kind of um you know mobile phones and all the other technology that we
[00:15:34] and all the other technology that we walk around with today in just a very
[00:15:36] walk around with today in just a very short amount of time so language is
[00:15:39] short amount of time so language is pretty
[00:15:41] pretty cool um but it's you know one shouldn't
[00:15:44] cool um but it's you know one shouldn't only fixate on um the sort of knowledge
[00:15:49] only fixate on um the sort of knowledge side of language and how that's made um
[00:15:52] side of language and how that's made um human beings great I mean there's this
[00:15:54] human beings great I mean there's this other side of language where language is
[00:15:57] other side of language where language is this very flexible system which is used
[00:16:01] this very flexible system which is used as a social tool by human beings um so
[00:16:05] as a social tool by human beings um so that we can speak with a lot of
[00:16:09] that we can speak with a lot of imprecision and nuance and emotion in
[00:16:12] imprecision and nuance and emotion in language and we can get people to
[00:16:15] language and we can get people to understand we can set up sort of new
[00:16:17] understand we can set up sort of new ways of thinking about things by using
[00:16:20] ways of thinking about things by using words for them and languages aren't
[00:16:22] words for them and languages aren't static languages change as human beings
[00:16:25] static languages change as human beings use them that languages aren't something
[00:16:28] use them that languages aren't something that would delivered down on Tablets by
[00:16:31] that would delivered down on Tablets by God languages are things that humans
[00:16:33] God languages are things that humans constructed and humans changed them with
[00:16:36] constructed and humans changed them with each success of generation and indeed
[00:16:39] each success of generation and indeed most of the innovation in language
[00:16:41] most of the innovation in language happens among young people you know
[00:16:44] happens among young people you know people that are either a few years
[00:16:45] people that are either a few years younger than you are most of you are now
[00:16:48] younger than you are most of you are now um in their um earlier teens going into
[00:16:51] um in their um earlier teens going into the 20s right that's a big period of
[00:16:53] the 20s right that's a big period of linguistic Innovation where people think
[00:16:55] linguistic Innovation where people think up cool new phrases and ways of saying
[00:16:58] up cool new phrases and ways of saying things and some of those get embedded
[00:17:00] things and some of those get embedded and extended and that then becomes the
[00:17:02] and extended and that then becomes the future of language um so um herb Clark
[00:17:06] future of language um so um herb Clark used to be a um psychologist um at
[00:17:10] used to be a um psychologist um at Stanford he's now retired but he had
[00:17:12] Stanford he's now retired but he had this rather um nice quote the common
[00:17:15] this rather um nice quote the common misconception is that language use has
[00:17:17] misconception is that language use has primarily to do with words and what they
[00:17:19] primarily to do with words and what they mean it doesn't it has primarily to do
[00:17:22] mean it doesn't it has primarily to do with people and what they
[00:17:24] with people and what they mean okay so that's language and two
[00:17:27] mean okay so that's language and two slides for you um so now we'll skip
[00:17:30] slides for you um so now we'll skip ahead to deep learning so in the last
[00:17:33] ahead to deep learning so in the last decade or so we've been able to make um
[00:17:37] decade or so we've been able to make um fantastic progress in doing more with
[00:17:39] fantastic progress in doing more with computers understanding um human
[00:17:42] computers understanding um human languages um in using deep learning
[00:17:45] languages um in using deep learning we'll say a bit more about the history
[00:17:47] we'll say a bit more about the history later on but you know work on trying to
[00:17:50] later on but you know work on trying to do things with human language started in
[00:17:51] do things with human language started in the 1950s so it had been sort of going
[00:17:54] the 1950s so it had been sort of going for 60 years or so and you know there
[00:17:58] for 60 years or so and you know there was some stuff it's not that nobody
[00:18:00] was some stuff it's not that nobody could do anything but you know the
[00:18:03] could do anything but you know the ability to understand and produce
[00:18:06] ability to understand and produce language had always been kind of
[00:18:08] language had always been kind of questionable where it's really in the
[00:18:10] questionable where it's really in the last decade with new networks that just
[00:18:13] last decade with new networks that just enormous strides of progress have been
[00:18:15] enormous strides of progress have been made um and that's led into the world
[00:18:17] made um and that's led into the world that we have today so one of the first
[00:18:20] that we have today so one of the first big breakthroughs came in the area of
[00:18:23] big breakthroughs came in the area of using um neural NLP systems for machine
[00:18:27] using um neural NLP systems for machine translation and so this started about
[00:18:30] translation and so this started about 2014 and was already deployed live on
[00:18:34] 2014 and was already deployed live on services like Google by 2016 it was so
[00:18:38] services like Google by 2016 it was so good that it saw really really rapid um
[00:18:41] good that it saw really really rapid um commercial deployment and I mean overall
[00:18:44] commercial deployment and I mean overall this kind of facility um with machine
[00:18:48] this kind of facility um with machine translation just means that you're
[00:18:51] translation just means that you're growing up in such a different world um
[00:18:54] growing up in such a different world um to people a few Generations back right
[00:18:58] to people a few Generations back right people a few Generations back um that
[00:19:01] people a few Generations back um that unless you actually knew different
[00:19:03] unless you actually knew different languages of different people you sort
[00:19:06] languages of different people you sort of had no chance to communicate with
[00:19:08] of had no chance to communicate with them where um now we're very close to
[00:19:11] them where um now we're very close to having something like the Babel Fish
[00:19:13] having something like the Babel Fish from Hitchhiker's Guide to the Galaxy um
[00:19:16] from Hitchhiker's Guide to the Galaxy um for understanding all languages it's
[00:19:18] for understanding all languages it's just it's not a Babel Fish it's a cell
[00:19:22] just it's not a Babel Fish it's a cell phone but you know you can have it out
[00:19:23] phone but you know you can have it out between two people and have it do
[00:19:26] between two people and have it do simultaneous translation and you know
[00:19:28] simultaneous translation and you know it's not perfect people keep on doing
[00:19:30] it's not perfect people keep on doing research on this but um you know by and
[00:19:33] research on this but um you know by and large it means you can pick anything up
[00:19:35] large it means you can pick anything up from different areas of the world um as
[00:19:38] from different areas of the world um as you can see this example is from a
[00:19:40] you can see this example is from a couple of years ago since it's still
[00:19:41] couple of years ago since it's still from the um Co pandemic era but you know
[00:19:44] from the um Co pandemic era but you know I can um see this um Swahili from Kenya
[00:19:49] I can um see this um Swahili from Kenya and say oh gee I wonder what that means
[00:19:51] and say oh gee I wonder what that means stick it into Google translate and um I
[00:19:55] stick it into Google translate and um I can learn that Malawi um lost two
[00:19:58] can learn that Malawi um lost two ministers due to um Co infections and
[00:20:02] ministers due to um Co infections and they died right so you know we're just
[00:20:04] they died right so you know we're just in this different era of being able to
[00:20:06] in this different era of being able to understand stuff and then there are lots
[00:20:09] understand stuff and then there are lots of other things that we can do with
[00:20:10] of other things that we can do with modern NLP so until a few years ago um
[00:20:15] modern NLP so until a few years ago um we had web search engines and you put in
[00:20:18] we had web search engines and you put in some text you could write it as a
[00:20:20] some text you could write it as a sentence if you wanted to but it didn't
[00:20:22] sentence if you wanted to but it didn't really matter whether you wrote a
[00:20:23] really matter whether you wrote a sentence or not because what you got was
[00:20:25] sentence or not because what you got was some keywords that were then matched
[00:20:28] some keywords that were then matched against index and you were shown some
[00:20:30] against index and you were shown some pages that might have the answers to
[00:20:32] pages that might have the answers to your questions but these days um you can
[00:20:35] your questions but these days um you can put an actual question into a modern
[00:20:38] put an actual question into a modern search engine like when did Kendrick
[00:20:40] search engine like when did Kendrick Lamar's first album come out it can go
[00:20:43] Lamar's first album come out it can go and find documents that have relevant
[00:20:45] and find documents that have relevant information it can read those documents
[00:20:48] information it can read those documents and it can give you an answer so that it
[00:20:51] and it can give you an answer so that it actually can become an answer engine
[00:20:54] actually can become an answer engine rather than just something that finds
[00:20:55] rather than just something that finds documents that might be relevant to what
[00:20:57] documents that might be relevant to what you're interested in and the way that
[00:20:59] you're interested in and the way that that's done is with big neural networks
[00:21:02] that's done is with big neural networks so that you might commonly have um for
[00:21:05] so that you might commonly have um for your query you've got a retrieval neural
[00:21:08] your query you've got a retrieval neural network which can find passages that are
[00:21:11] network which can find passages that are similar to The query they might then be
[00:21:13] similar to The query they might then be reranked by a second neural network and
[00:21:16] reranked by a second neural network and then there'll be a third reading neural
[00:21:18] then there'll be a third reading neural network that'll read those passages um
[00:21:21] network that'll read those passages um and synthesize information from them
[00:21:24] and synthesize information from them which it then returns as the
[00:21:26] which it then returns as the answer okay that gets to about
[00:21:30] answer okay that gets to about 2018 um but then things got more
[00:21:32] 2018 um but then things got more advanced again so it was really around
[00:21:36] advanced again so it was really around 2019 that people started to see the
[00:21:39] 2019 that people started to see the power of large language models and so
[00:21:42] power of large language models and so back in
[00:21:43] back in 2019 those of us in NLP were really
[00:21:46] 2019 those of us in NLP were really excited about
[00:21:48] excited about gpt2 um it didn't make much of an impact
[00:21:50] gpt2 um it didn't make much of an impact on the Nightly News but it was really
[00:21:52] on the Nightly News but it was really exciting an NLP land um because gpt2
[00:21:57] exciting an NLP land um because gpt2 already for the first time time meant
[00:21:59] already for the first time time meant here was a large language model that
[00:22:02] here was a large language model that could just generate fluent text that
[00:22:05] could just generate fluent text that really until then um NLP systems had
[00:22:08] really until then um NLP systems had done sort of a decent job at
[00:22:10] done sort of a decent job at understanding certain facts out of text
[00:22:13] understanding certain facts out of text but we've just never been able to
[00:22:14] but we've just never been able to generate fluent text that was at all
[00:22:17] generate fluent text that was at all good um where here what you could do
[00:22:19] good um where here what you could do with gpt2 is you could um write
[00:22:22] with gpt2 is you could um write something like the start of a story a
[00:22:24] something like the start of a story a train Carriage containing controlled
[00:22:26] train Carriage containing controlled nuclear materials was stolen in cincin
[00:22:28] nuclear materials was stolen in cincin today its whereabouts are unknown and
[00:22:32] today its whereabouts are unknown and then GPT 2 would just write a
[00:22:35] then GPT 2 would just write a continuation the incident occurred on
[00:22:37] continuation the incident occurred on the downtown train line which runs from
[00:22:39] the downtown train line which runs from Covington ashin stations in an email to
[00:22:42] Covington ashin stations in an email to Ohio news outlets the US Department of
[00:22:44] Ohio news outlets the US Department of energy set is working with the Federal
[00:22:46] energy set is working with the Federal Railroad Administration to find the
[00:22:48] Railroad Administration to find the Thief dot dot dot and so the way this is
[00:22:50] Thief dot dot dot and so the way this is working is this conditioning on all the
[00:22:53] working is this conditioning on all the past material and as I show at the very
[00:22:56] past material and as I show at the very bottom line down here it's generating
[00:22:59] bottom line down here it's generating one word at a time as to what word it
[00:23:02] one word at a time as to what word it thinks would be likely to come next
[00:23:04] thinks would be likely to come next after that um and so from that simple
[00:23:07] after that um and so from that simple method of sort of generating words out
[00:23:09] method of sort of generating words out of U one after another it's able to
[00:23:12] of U one after another it's able to produce excellent text and the thing to
[00:23:15] produce excellent text and the thing to notice is I mean this text is not only
[00:23:18] notice is I mean this text is not only kind of you know formally correct you
[00:23:21] kind of you know formally correct you know not the spellings correct and the
[00:23:23] know not the spellings correct and the sentences are real sentences not
[00:23:26] sentences are real sentences not disconnected garbage but you know it
[00:23:28] disconnected garbage but you know it actually understands a lot right so the
[00:23:31] actually understands a lot right so the prompt that was written said there were
[00:23:34] prompt that was written said there were stolen nuclear materials in Cincinnati
[00:23:37] stolen nuclear materials in Cincinnati but you know gpt2 knows a lot of stuff
[00:23:40] but you know gpt2 knows a lot of stuff it knows that Cincinnati is in Ohio it
[00:23:44] it knows that Cincinnati is in Ohio it knows that in the United States it's the
[00:23:46] knows that in the United States it's the department of energy that regulates
[00:23:48] department of energy that regulates nuclear materials um it knows if
[00:23:51] nuclear materials um it knows if something is stolen it's a theft and
[00:23:54] something is stolen it's a theft and that that would um make sense that um
[00:23:58] that that would um make sense that um people are getting involved with that um
[00:24:01] people are getting involved with that um it talks about you know there's train
[00:24:03] it talks about you know there's train Carriage So it's talking about the train
[00:24:05] Carriage So it's talking about the train line where it goes it really knows a lot
[00:24:07] line where it goes it really knows a lot and can write you know coherent
[00:24:10] and can write you know coherent discourse um like a real story so that's
[00:24:13] discourse um like a real story so that's kind of amazing um but you know things
[00:24:17] kind of amazing um but you know things moved on from there and so now we're in
[00:24:20] moved on from there and so now we're in the world of chat GPT and
[00:24:22] the world of chat GPT and gp4 and one of the things that we will
[00:24:25] gp4 and one of the things that we will talk about later is this was a huge huge
[00:24:28] talk about later is this was a huge huge user success because now you could ask
[00:24:32] user success because now you could ask questions or give it commands and it
[00:24:35] questions or give it commands and it would do what you wanted and that was
[00:24:38] would do what you wanted and that was further amazing so here I'm saying hey
[00:24:41] further amazing so here I'm saying hey please draft a polite email to my boss
[00:24:43] please draft a polite email to my boss Jeremy that I would not be able to come
[00:24:45] Jeremy that I would not be able to come into the office for the next two days
[00:24:47] into the office for the next two days because my 9-year-old song that's a
[00:24:50] because my 9-year-old song that's a misspelling for son but the system works
[00:24:52] misspelling for son but the system works fine Des spite it um Peter is angry with
[00:24:55] fine Des spite it um Peter is angry with me that I'm not giving him much time
[00:24:58] me that I'm not giving him much time and it writes a nice email um it
[00:25:01] and it writes a nice email um it corrects the spelling mistake because it
[00:25:04] corrects the spelling mistake because it knows people make spelling mistakes it
[00:25:05] knows people make spelling mistakes it doesn't talk about songs and everything
[00:25:08] doesn't talk about songs and everything works out beautifully um you can get it
[00:25:11] works out beautifully um you can get it to do other things so you can um ask it
[00:25:15] to do other things so you can um ask it what is unusual about this image um so
[00:25:18] what is unusual about this image um so in thinking about meaning one of the
[00:25:20] in thinking about meaning one of the things that's interesting with these
[00:25:22] things that's interesting with these recent models um is that they're
[00:25:24] recent models um is that they're multimodal and can operate across modes
[00:25:28] multimodal and can operate across modes and so um a favorite term that we coined
[00:25:31] and so um a favorite term that we coined at Stanford is the term Foundation
[00:25:33] at Stanford is the term Foundation models which we use as a generalization
[00:25:36] models which we use as a generalization of large language models to have the
[00:25:39] of large language models to have the same kind of technology used across
[00:25:41] same kind of technology used across different modalities images sound um
[00:25:46] different modalities images sound um various kinds of bioinformatic things
[00:25:48] various kinds of bioinformatic things DNA RNA things like that seismic waves
[00:25:52] DNA RNA things like that seismic waves any kind of signal building these same
[00:25:55] any kind of signal building these same kind of large
[00:25:56] kind of large models another place that you can see
[00:25:59] models another place that you can see that um is going from text to images um
[00:26:05] that um is going from text to images um so if I asked for a picture of a train
[00:26:07] so if I asked for a picture of a train going over the Golden Gate Bridge um
[00:26:10] going over the Golden Gate Bridge um this is now um darly 2 um it gives me a
[00:26:15] this is now um darly 2 um it gives me a picture of a train going over the Golden
[00:26:17] picture of a train going over the Golden Gate Bridge um this is a perfect time to
[00:26:20] Gate Bridge um this is a perfect time to welcome anyone who's watching this um on
[00:26:23] welcome anyone who's watching this um on Stanford online um if you're on Stanford
[00:26:25] Stanford online um if you're on Stanford online and are not in the Bay Area the
[00:26:28] online and are not in the Bay Area the important thing to know is no trains go
[00:26:31] important thing to know is no trains go over the Golden Gate Bridge um but you
[00:26:34] over the Golden Gate Bridge um but you might not be completely happy with this
[00:26:36] might not be completely happy with this picture um because you know it shows the
[00:26:39] picture um because you know it shows the Golden Gate Bridge and a train going
[00:26:41] Golden Gate Bridge and a train going over it but it doesn't show the bay so
[00:26:43] over it but it doesn't show the bay so maybe I'd like to um get with the bay in
[00:26:46] maybe I'd like to um get with the bay in the background and if I ask for that
[00:26:48] the background and if I ask for that well look now I've got um a train going
[00:26:51] well look now I've got um a train going over the Golden Gate Bridge with the bay
[00:26:52] over the Golden Gate Bridge with the bay in the background but you still might
[00:26:55] in the background but you still might not be this this might not be exactly
[00:26:58] not be this this might not be exactly what you want like maybe you'd prefer
[00:27:00] what you want like maybe you'd prefer something that's a pencil drawing so I
[00:27:03] something that's a pencil drawing so I can say a train going over the Golden
[00:27:05] can say a train going over the Golden Gate Bridge detailed pencil drawing and
[00:27:07] Gate Bridge detailed pencil drawing and I can get a pencil drawing um or maybe
[00:27:10] I can get a pencil drawing um or maybe it's unrealistic that the Golden Gate
[00:27:12] it's unrealistic that the Golden Gate Bridge only has trains going over it now
[00:27:15] Bridge only has trains going over it now um so maybe it be good to have some cars
[00:27:17] um so maybe it be good to have some cars as well um so I could ask for a train
[00:27:20] as well um so I could ask for a train and cars and we can get a train and cars
[00:27:23] and cars and we can get a train and cars going over it um now I actually made
[00:27:26] going over it um now I actually made these ones all by myself so should be
[00:27:28] these ones all by myself so should be impressed with my generative AI artwork
[00:27:31] impressed with my generative AI artwork um but these examples are actually a bit
[00:27:33] um but these examples are actually a bit old now because they're done with DAR 2
[00:27:35] old now because they're done with DAR 2 and if you keep up with these things
[00:27:37] and if you keep up with these things that's a few years ago there now Dary 3
[00:27:39] that's a few years ago there now Dary 3 and so on so we can now get much fancier
[00:27:41] and so on so we can now get much fancier things again right an illustration from
[00:27:44] things again right an illustration from a graphic novel a bustling City street
[00:27:46] a graphic novel a bustling City street under the shine of a full moon the
[00:27:48] under the shine of a full moon the sidewalks bustling with pedestrians
[00:27:50] sidewalks bustling with pedestrians enjoying the night life at the corner
[00:27:52] enjoying the night life at the corner stall a young woman with fiery red hair
[00:27:54] stall a young woman with fiery red hair dressed in a signature velvet cloak is
[00:27:57] dressed in a signature velvet cloak is haggling with the grumpy old vendor the
[00:27:59] haggling with the grumpy old vendor the grumpy vendor a tall sophisticated man
[00:28:01] grumpy vendor a tall sophisticated man is wearing a sharp suit Sports a
[00:28:03] is wearing a sharp suit Sports a noteworthy mustache is animatedly
[00:28:05] noteworthy mustache is animatedly conversing on his steampunk telephone
[00:28:08] conversing on his steampunk telephone and pretty much um we're getting all of
[00:28:12] and pretty much um we're getting all of that okay um so let's now get on to
[00:28:16] that okay um so let's now get on to starting to think more about meaning so
[00:28:20] starting to think more about meaning so perhaps um what can we do for meaning
[00:28:25] perhaps um what can we do for meaning right so if you think of words and there
[00:28:28] right so if you think of words and there meaning um if you look up a dictionary
[00:28:31] meaning um if you look up a dictionary and say what does meaning mean um
[00:28:33] and say what does meaning mean um meaning is defined as the idea that is
[00:28:35] meaning is defined as the idea that is represented by a word or phrase the idea
[00:28:38] represented by a word or phrase the idea that a person wants to express by using
[00:28:40] that a person wants to express by using words the idea that is expressed um and
[00:28:45] words the idea that is expressed um and in in linguistics you know if you go and
[00:28:48] in in linguistics you know if you go and do a semantics class or something the
[00:28:50] do a semantics class or something the commonest way of thinking of of meaning
[00:28:53] commonest way of thinking of of meaning is somewhat like what's presented up
[00:28:55] is somewhat like what's presented up above there that meaning is thought of
[00:28:58] above there that meaning is thought of as a pairing between what sometimes
[00:29:01] as a pairing between what sometimes called signifier and signified but it's
[00:29:03] called signifier and signified but it's perhaps easy to think of as a symbol a
[00:29:06] perhaps easy to think of as a symbol a word and then an idea or thing and so
[00:29:10] word and then an idea or thing and so this notion is referred to as
[00:29:12] this notion is referred to as denotational semantics so the idea or
[00:29:15] denotational semantics so the idea or thing is the denotation of the symbol
[00:29:18] thing is the denotation of the symbol and so this same idea of denotational
[00:29:20] and so this same idea of denotational semantics has also been used for
[00:29:21] semantics has also been used for programming languages because in
[00:29:23] programming languages because in programming languages you have symbols
[00:29:26] programming languages you have symbols like while and if
[00:29:28] like while and if variables and they have a meaning and
[00:29:31] variables and they have a meaning and that could be their
[00:29:32] that could be their denotation um so we sort of would say
[00:29:35] denotation um so we sort of would say that the meaning of tree is all the
[00:29:38] that the meaning of tree is all the trees you can find out around the world
[00:29:41] trees you can find out around the world um that's sort of a okay notion of
[00:29:44] um that's sort of a okay notion of meaning um it's a popular one it's never
[00:29:47] meaning um it's a popular one it's never been very
[00:29:48] been very obvious or at least traditionally it
[00:29:50] obvious or at least traditionally it wasn't very obvious as of what we could
[00:29:52] wasn't very obvious as of what we could do with that to get it into computers so
[00:29:55] do with that to get it into computers so if you looked in the pre-neural world
[00:29:58] if you looked in the pre-neural world when people tried to look at meanings
[00:30:01] when people tried to look at meanings inside computers they sort of had to do
[00:30:04] inside computers they sort of had to do something much more primitive of looking
[00:30:07] something much more primitive of looking at words and their relationship so a
[00:30:09] at words and their relationship so a very common traditional solution was to
[00:30:12] very common traditional solution was to make use of word net and word net was
[00:30:15] make use of word net and word net was kind of a sort of fancy thesaurus that
[00:30:17] kind of a sort of fancy thesaurus that showed word relations so i' tell you
[00:30:20] showed word relations so i' tell you about synonyms and is a kind of things
[00:30:23] about synonyms and is a kind of things um so a panda is a kind of carnivore
[00:30:26] um so a panda is a kind of carnivore which is a placental which is a m
[00:30:28] which is a placental which is a m and things like that good has various
[00:30:31] and things like that good has various meanings it's a trade good or the sense
[00:30:33] meanings it's a trade good or the sense of goodness and you could explore with
[00:30:35] of goodness and you could explore with that but systems like wordnet um were
[00:30:39] that but systems like wordnet um were never very good for computational
[00:30:42] never very good for computational meaning um they missed a lot of nuance
[00:30:45] meaning um they missed a lot of nuance wordnet would tell you that proficient
[00:30:47] wordnet would tell you that proficient is a synonym for good but if you think
[00:30:50] is a synonym for good but if you think about all the things that you would say
[00:30:52] about all the things that you would say were good you know that was a good shot
[00:30:54] were good you know that was a good shot would you say that was a proficient shot
[00:30:57] would you say that was a proficient shot sounds kind of weird to me um you know
[00:30:59] sounds kind of weird to me um you know there's a lot of color and Nuance on how
[00:31:01] there's a lot of color and Nuance on how words are used um word net is very
[00:31:05] words are used um word net is very incomplete it's missing anything that's
[00:31:07] incomplete it's missing anything that's kind of cooler more modern slang this
[00:31:10] kind of cooler more modern slang this maybe isn't very modern slang now but
[00:31:12] maybe isn't very modern slang now but you won't find more modern slang either
[00:31:14] you won't find more modern slang either in it it's sort of very human-made Etc
[00:31:17] in it it's sort of very human-made Etc it's got a lot of issues so um this led
[00:31:20] it's got a lot of issues so um this led into the idea of can we represent
[00:31:22] into the idea of can we represent meaning differently and this leads us
[00:31:24] meaning differently and this leads us into word
[00:31:26] into word vectors um so when we have words Wicked
[00:31:31] vectors um so when we have words Wicked badass Nifty wizard what do they turn
[00:31:35] badass Nifty wizard what do they turn into when we have
[00:31:38] into when we have computers um well
[00:31:40] computers um well effectively um you know words are these
[00:31:43] effectively um you know words are these discrete symbols um that they're just
[00:31:46] discrete symbols um that they're just kind of some kind of atom or symbol and
[00:31:49] kind of some kind of atom or symbol and if we then turn those into something
[00:31:52] if we then turn those into something that's closer to math um how symbols are
[00:31:57] that's closer to math um how symbols are normally rep presented is you have a
[00:32:00] normally rep presented is you have a vocabulary and your word is some item in
[00:32:04] vocabulary and your word is some item in that vocabulary so Motel is the that
[00:32:06] that vocabulary so Motel is the that word in the vocabulary and hotel is this
[00:32:09] word in the vocabulary and hotel is this word in the vocabulary and commonly this
[00:32:12] word in the vocabulary and commonly this is what computational systems do you
[00:32:14] is what computational systems do you take all your strings and you index them
[00:32:17] take all your strings and you index them to numbers and that's the sort of
[00:32:19] to numbers and that's the sort of position in a vector that they belong in
[00:32:22] position in a vector that they belong in um and well we have huge numbers of
[00:32:25] um and well we have huge numbers of words so we might have a huge vocabulary
[00:32:28] words so we might have a huge vocabulary so we'll have very big and long vectors
[00:32:30] so we'll have very big and long vectors and so these get referred to as one hot
[00:32:33] and so these get referred to as one hot vectors um for representing the meaning
[00:32:36] vectors um for representing the meaning of words um but
[00:32:39] of words um but representing words by one hot vectors
[00:32:42] representing words by one hot vectors turns out to not be a very good way of
[00:32:45] turns out to not be a very good way of computing with them it was used for
[00:32:48] computing with them it was used for decades um but it turns out to be kind
[00:32:50] decades um but it turns out to be kind of problematic and part of why it's
[00:32:52] of problematic and part of why it's problematic is it doesn't have any
[00:32:56] problematic is it doesn't have any natural inherent sense
[00:32:58] natural inherent sense of the meanings of words you just have
[00:33:00] of the meanings of words you just have different words you have hotel and motel
[00:33:03] different words you have hotel and motel and house and chair and so if you think
[00:33:06] and house and chair and so if you think about in terms of these Vector
[00:33:09] about in terms of these Vector representations that if you have motel
[00:33:11] representations that if you have motel and hotel there's no indication that
[00:33:14] and hotel there's no indication that they're kind of similar they're just two
[00:33:16] they're kind of similar they're just two different symbols which have ones in
[00:33:19] different symbols which have ones in different positions in the vector or
[00:33:21] different positions in the vector or formally in math terms um if you think
[00:33:24] formally in math terms um if you think about taking the dotproduct of these two
[00:33:27] about taking the dotproduct of these two vectors zero um the two vectors are
[00:33:30] vectors zero um the two vectors are orthogonal they have nothing to do with
[00:33:33] orthogonal they have nothing to do with each other now there are things that you
[00:33:35] each other now there are things that you can do with that you can start saying oh
[00:33:37] can do with that you can start saying oh let me start building up some other
[00:33:39] let me start building up some other resource of word similarity and I'll
[00:33:42] resource of word similarity and I'll consult that resource of word similarity
[00:33:45] consult that resource of word similarity and it'll tell me that motels and hotels
[00:33:47] and it'll tell me that motels and hotels are similar to each other and people did
[00:33:50] are similar to each other and people did things like that right in web search it
[00:33:52] things like that right in web search it was referred to as query expansion
[00:33:54] was referred to as query expansion techniques um but still the point is
[00:33:57] techniques um but still the point is that there's no natural notion of
[00:34:00] that there's no natural notion of similarity in one hot
[00:34:02] similarity in one hot vectors um and so the the idea was that
[00:34:07] vectors um and so the the idea was that maybe we could do better than that that
[00:34:10] maybe we could do better than that that we could learn to include similarity in
[00:34:13] we could learn to include similarity in the vectors themselves and so that leads
[00:34:16] the vectors themselves and so that leads into the idea of word vectors um but it
[00:34:19] into the idea of word vectors um but it also leads into a different way of
[00:34:21] also leads into a different way of thinking about semantics um I just
[00:34:24] thinking about semantics um I just realized I forgot to say one thing back
[00:34:26] realized I forgot to say one thing back two slides um these kind of
[00:34:30] two slides um these kind of representations are referred to as
[00:34:32] representations are referred to as localist representations meaning that
[00:34:34] localist representations meaning that there's one point in which um something
[00:34:38] there's one point in which um something is represented so that um you've got
[00:34:42] is represented so that um you've got here is the representation of motel and
[00:34:45] here is the representation of motel and here is the representation of Hotel it's
[00:34:48] here is the representation of Hotel it's in one place in the vector that each
[00:34:50] in one place in the vector that each word is represented and they'll be
[00:34:52] word is represented and they'll be different to what we do next um so
[00:34:55] different to what we do next um so there's an alternative idea of semantic
[00:34:58] there's an alternative idea of semantic um which goes back quite a long way
[00:35:01] um which goes back quite a long way people commonly quote this quote of Jr F
[00:35:05] people commonly quote this quote of Jr F who was a British linguist who said in
[00:35:07] who was a British linguist who said in 1957 you shall know a word by the
[00:35:09] 1957 you shall know a word by the company it keeps but also goes back to
[00:35:12] company it keeps but also goes back to philosophical work by binstein and
[00:35:14] philosophical work by binstein and others that what you should do is
[00:35:17] others that what you should do is represent a word's meaning by the
[00:35:20] represent a word's meaning by the context in which it appears um so the
[00:35:23] context in which it appears um so the words that appear around the word give
[00:35:26] words that appear around the word give information
[00:35:28] information um about its meaning and so that's the
[00:35:30] um about its meaning and so that's the idea of what's called distributional
[00:35:32] idea of what's called distributional semantics in contrast to denotational
[00:35:35] semantics in contrast to denotational semantics so if I want to know about the
[00:35:38] semantics so if I want to know about the word banking I say give me some
[00:35:41] word banking I say give me some sentences that use the word banking here
[00:35:43] sentences that use the word banking here are some sentences using the word
[00:35:44] are some sentences using the word banking government debt problems turning
[00:35:47] banking government debt problems turning into banking crises as happened in 2009
[00:35:51] into banking crises as happened in 2009 etc etc and knowing about that context
[00:35:55] etc etc and knowing about that context words that occur around banking
[00:35:58] words that occur around banking those will become the meaning of banking
[00:36:01] those will become the meaning of banking and so we're going to use those
[00:36:04] and so we're going to use those statistics um about words and what other
[00:36:08] statistics um about words and what other words appear around them in order to
[00:36:12] words appear around them in order to learn a new kind of representation of a
[00:36:16] learn a new kind of representation of a word so our new representation of words
[00:36:20] word so our new representation of words is we're going to represent them now as
[00:36:23] is we're going to represent them now as a dense a sort of a shorter dense Vector
[00:36:27] a dense a sort of a shorter dense Vector that giv the meaning of the words now my
[00:36:30] that giv the meaning of the words now my vectors are very short here um these are
[00:36:32] vectors are very short here um these are only eight dimensional if I counted
[00:36:34] only eight dimensional if I counted right so I could fit them on my slide
[00:36:36] right so I could fit them on my slide they're not that short in practice they
[00:36:38] they're not that short in practice they might be 200 2,000 but reasonably short
[00:36:43] might be 200 2,000 but reasonably short they're not going to be like the half a
[00:36:44] they're not going to be like the half a million of the half a million different
[00:36:46] million of the half a million different words in our vocabulary and the idea is
[00:36:50] words in our vocabulary and the idea is if words have stuff to do with each
[00:36:52] if words have stuff to do with each other um they'll have sort of similar
[00:36:55] other um they'll have sort of similar vectors which corresponds to their dot
[00:36:58] vectors which corresponds to their dot product being large so for Banking and
[00:37:01] product being large so for Banking and monetary in my example here both of them
[00:37:03] monetary in my example here both of them are positive in the First Dimension
[00:37:05] are positive in the First Dimension positive in the second dimension
[00:37:07] positive in the second dimension negative on the third on the fourth
[00:37:10] negative on the third on the fourth they've got opposite signs so if we want
[00:37:11] they've got opposite signs so if we want to work out the dot product we're taking
[00:37:14] to work out the dot product we're taking the product of the corresponding terms
[00:37:17] the product of the corresponding terms and it'll get bigger to the extent that
[00:37:19] and it'll get bigger to the extent that both of the corresponding ones have the
[00:37:21] both of the corresponding ones have the same sides and bigger if they have large
[00:37:24] same sides and bigger if they have large um
[00:37:25] um magnitude Okay so these are what we call
[00:37:29] magnitude Okay so these are what we call word vectors which are also known as
[00:37:31] word vectors which are also known as embeddings or newal word representations
[00:37:35] embeddings or newal word representations or phrases like that and so the first
[00:37:37] or phrases like that and so the first thing we want to do is learn good word
[00:37:41] thing we want to do is learn good word vectors for different words and our word
[00:37:44] vectors for different words and our word vectors will be good word vectors if um
[00:37:50] vectors will be good word vectors if um they give us a good sense of the
[00:37:52] they give us a good sense of the meanings of words they know which words
[00:37:55] meanings of words they know which words are similar to other words in meaning
[00:37:58] are similar to other words in meaning um we refer to them as
[00:38:00] um we refer to them as embeddings um because we can think of
[00:38:03] embeddings um because we can think of this as a vector in a high dimensional
[00:38:05] this as a vector in a high dimensional space and so that we're embedding each
[00:38:08] space and so that we're embedding each word as a position in that
[00:38:10] word as a position in that high-dimensional space and the
[00:38:13] high-dimensional space and the dimensionality of the space um will be
[00:38:15] dimensionality of the space um will be the length of the vector so it might be
[00:38:17] the length of the vector so it might be something like a 300 dimensional space
[00:38:20] something like a 300 dimensional space um now that kind of gets problematic
[00:38:24] um now that kind of gets problematic because human beings can't look at 300
[00:38:27] because human beings can't look at 300 dimensional spaces and aren't very good
[00:38:29] dimensional spaces and aren't very good at understanding or visualizing what
[00:38:32] at understanding or visualizing what goes on in them so the only thing that I
[00:38:35] goes on in them so the only thing that I can show you is um two-dimensional
[00:38:38] can show you is um two-dimensional spaces but um a thing that is good to
[00:38:44] spaces but um a thing that is good to have somewhat in your head is that
[00:38:47] have somewhat in your head is that really high-dimensional spaces behave
[00:38:50] really high-dimensional spaces behave extremely differently to two-dimensional
[00:38:53] extremely differently to two-dimensional spaces in high dimensional spaces things
[00:38:58] spaces in high dimensional spaces things can in a two-dimensional space you're
[00:39:01] can in a two-dimensional space you're only near to something else if you got
[00:39:02] only near to something else if you got similar X and Y coordinates in a high
[00:39:05] similar X and Y coordinates in a high dimensional space things can be very
[00:39:07] dimensional space things can be very near to all sorts of things on different
[00:39:10] near to all sorts of things on different dimensions in the space and so we can
[00:39:12] dimensions in the space and so we can capture different senses of words and
[00:39:15] capture different senses of words and ways that words are similar to each
[00:39:17] ways that words are similar to each other um but here's the kind of picture
[00:39:20] other um but here's the kind of picture we end up with so we're what we're going
[00:39:21] we end up with so we're what we're going to do is learn a way to um represent all
[00:39:26] to do is learn a way to um represent all words as vectors based on the other
[00:39:30] words as vectors based on the other words that they within context and we
[00:39:33] words that they within context and we can embed them into this Vector space
[00:39:35] can embed them into this Vector space and of course you can't read anything
[00:39:37] and of course you can't read anything there but you know we can zoom into this
[00:39:39] there but you know we can zoom into this space further and if we zoom into this
[00:39:42] space further and if we zoom into this space and just show a bit of it well
[00:39:44] space and just show a bit of it well here's a part of the space um where it's
[00:39:46] here's a part of the space um where it's showing um country words and some other
[00:39:49] showing um country words and some other location words so we've got so of um
[00:39:52] location words so we've got so of um countries up the top there we've got
[00:39:54] countries up the top there we've got some nationality terms British
[00:39:56] some nationality terms British Australian American European um further
[00:39:59] Australian American European um further down or we can go to another piece of
[00:40:02] down or we can go to another piece of the space and here's a bit of the space
[00:40:04] the space and here's a bit of the space um where we have verbs and not only if
[00:40:06] um where we have verbs and not only if we got verbs but you know there's
[00:40:08] we got verbs but you know there's actually quite quite a lot of fine
[00:40:10] actually quite quite a lot of fine structure here of what's similar that
[00:40:12] structure here of what's similar that represents things about verbs so you've
[00:40:15] represents things about verbs so you've got um sort of verbs of you know
[00:40:19] got um sort of verbs of you know communication statements saying thinking
[00:40:22] communication statements saying thinking expecting grouping together come and go
[00:40:24] expecting grouping together come and go group together down the bottom you've
[00:40:27] group together down the bottom you've got forms of the verb have then you've
[00:40:29] got forms of the verb have then you've got forms of the verb to be above them
[00:40:32] got forms of the verb to be above them you've got become and remain which are
[00:40:34] you've got become and remain which are actually sort of similar to the verb to
[00:40:36] actually sort of similar to the verb to be um because they take these sort of
[00:40:40] be um because they take these sort of complements of state so just that you
[00:40:42] complements of state so just that you can same as you can say I am angry you
[00:40:46] can same as you can say I am angry you can say he remained angry or he became
[00:40:49] can say he remained angry or he became angry right so those verbs are more so
[00:40:53] angry right so those verbs are more so than most verbs sort of similar to the
[00:40:54] than most verbs sort of similar to the verb to be so we get these kind of
[00:40:57] verb to be so we get these kind of interesting semantic spaces where things
[00:41:00] interesting semantic spaces where things that have similar meaning are close by
[00:41:02] that have similar meaning are close by to each other and so the question is how
[00:41:06] to each other and so the question is how do we get to those things and how we get
[00:41:09] do we get to those things and how we get to those things is then um for you know
[00:41:13] to those things is then um for you know that there are various ways of doing it
[00:41:15] that there are various ways of doing it but the one I want to get through um
[00:41:17] but the one I want to get through um today is showing you about um word DEC
[00:41:21] today is showing you about um word DEC okay I'll pause for 30 seconds for bread
[00:41:23] okay I'll pause for 30 seconds for bread breath anyone have a question or
[00:41:25] breath anyone have a question or anything they want to know yes
[00:41:32] so but it doesn't to solve the problem
[00:41:36] so but it doesn't to solve the problem where the similar meanings um might
[00:41:39] where the similar meanings um might depend on context right so let's take
[00:41:42] depend on context right so let's take your example about profession
[00:41:44] your example about profession ver so those two words have their own
[00:41:48] ver so those two words have their own batteries and we understand similarity
[00:41:52] batteries and we understand similarity some spectors but it's contact right
[00:41:56] some spectors but it's contact right because if you have different contact
[00:41:58] because if you have different contact those two similar and this Al does
[00:42:03] those two similar and this Al does not yes correct um so that's a good
[00:42:06] not yes correct um so that's a good thought you can keep it for a few weeks
[00:42:08] thought you can keep it for a few weeks to some extent yeah so for the first
[00:42:10] to some extent yeah so for the first thing we're going to do we're just going
[00:42:12] thing we're going to do we're just going to learn one word Vector for a string so
[00:42:16] to learn one word Vector for a string so we're going to have a word let's say
[00:42:19] we're going to have a word let's say it's star and we're going to learn one
[00:42:21] it's star and we're going to learn one word Vector for it so that absolutely
[00:42:24] word Vector for it so that absolutely doesn't capture the meaning of a word in
[00:42:27] doesn't capture the meaning of a word in context so it won't be saying whether
[00:42:30] context so it won't be saying whether it's meaning a Hollywood star or an
[00:42:33] it's meaning a Hollywood star or an astronomical star or something like that
[00:42:36] astronomical star or something like that and so later on we're going to get on to
[00:42:38] and so later on we're going to get on to contextual meaning representation so
[00:42:40] contextual meaning representation so wait for that but the thing I would like
[00:42:43] wait for that but the thing I would like to going along with what I said about
[00:42:45] to going along with what I said about high dimensional spaces being weird the
[00:42:48] high dimensional spaces being weird the the cool thing that we will already find
[00:42:51] the cool thing that we will already find is our representation for Star will be
[00:42:55] is our representation for Star will be very close to um the representations for
[00:42:59] very close to um the representations for astronomical words like nebula um and
[00:43:03] astronomical words like nebula um and what other every other astronomical
[00:43:05] what other every other astronomical words you know and
[00:43:07] words you know and simultaneously um it'll be um very close
[00:43:11] simultaneously um it'll be um very close to words that mean something like a
[00:43:13] to words that mean something like a Hollywood star
[00:43:17] Hollywood star um help me out no any words that mean
[00:43:19] um help me out no any words that mean something similar um celebrity that's a
[00:43:22] something similar um celebrity that's a good one okay yeah how are youing the
[00:43:27] good one okay yeah how are youing the Bing is to a lower dimensional space
[00:43:29] Bing is to a lower dimensional space visualized um so that pictures I was
[00:43:33] visualized um so that pictures I was showing you used a particular method
[00:43:35] showing you used a particular method called tne um which is a nonliner
[00:43:39] called tne um which is a nonliner dimensionality reduction that tends to
[00:43:41] dimensionality reduction that tends to work better for high dimensional new
[00:43:43] work better for high dimensional new representations um then PCA which you
[00:43:46] representations um then PCA which you might know um but I'm not going to go
[00:43:48] might know um but I'm not going to go into that now yes um how do you know Di
[00:43:57] but not
[00:43:59] but not too I mean that's something that people
[00:44:02] too I mean that's something that people have worked on it depends on how much it
[00:44:05] have worked on it depends on how much it depends on how much data you've got to
[00:44:07] depends on how much data you've got to make your representations over you know
[00:44:10] make your representations over you know so normally you know it's worked out
[00:44:13] so normally you know it's worked out either empirically for what works best
[00:44:16] either empirically for what works best or practically based on how big vectors
[00:44:19] or practically based on how big vectors you want to work on I mean to give you
[00:44:21] you want to work on I mean to give you some idea you know things start to work
[00:44:25] some idea you know things start to work well when you get to 100 dimensional
[00:44:27] well when you get to 100 dimensional space for a long time people used 300
[00:44:30] space for a long time people used 300 Dimensions because that seemed to work
[00:44:32] Dimensions because that seemed to work pretty well but as we people have
[00:44:34] pretty well but as we people have started building Huger and Huger models
[00:44:37] started building Huger and Huger models with way way more data it's now become
[00:44:39] with way way more data it's now become increasingly common to use numbers like
[00:44:42] increasingly common to use numbers like 1,000 or even 2,000 dimensional vectors
[00:44:46] 1,000 or even 2,000 dimensional vectors yeah
[00:44:48] yeah okay so um you mentioned that there sort
[00:44:51] okay so um you mentioned that there sort of hidden structur the small areas as
[00:44:53] of hidden structur the small areas as well as large areas of the embeding and
[00:44:56] well as large areas of the embeding and as in pieces different like different
[00:44:59] as in pieces different like different structures will come up but generally we
[00:45:01] structures will come up but generally we seem to use distance as the single
[00:45:03] seem to use distance as the single metric for closeness which doesn't seem
[00:45:05] metric for closeness which doesn't seem to me that we get it like distance
[00:45:07] to me that we get it like distance between like this and that in space
[00:45:09] between like this and that in space three will be the same right so how
[00:45:11] three will be the same right so how would that we don't only use distance we
[00:45:14] would that we don't only use distance we also use directions in the spaces having
[00:45:17] also use directions in the spaces having semantic meanings and I'll show a
[00:45:18] semantic meanings and I'll show a example that soon
[00:45:21] example that soon yeah the entri they seem to be between1
[00:45:26] yeah the entri they seem to be between1 and one is there a reason for that or do
[00:45:28] and one is there a reason for that or do we have bounds that we them um so good
[00:45:33] we have bounds that we them um so good question I mean you know they don't have
[00:45:36] question I mean you know they don't have to be um and the way we're going to
[00:45:40] to be um and the way we're going to learn them they're not bounded but you
[00:45:43] learn them they're not bounded but you know you can bound things sometimes
[00:45:45] know you can bound things sometimes people length normalize so that um the
[00:45:48] people length normalize so that um the vectors are of length one but at any
[00:45:51] vectors are of length one but at any rate normally in this work we use some
[00:45:53] rate normally in this work we use some method called you know regularization
[00:45:56] method called you know regularization that tries to kind of keep coefficients
[00:45:58] that tries to kind of keep coefficients small so they're generally not getting
[00:46:01] small so they're generally not getting huge yeah a specific word for example
[00:46:05] huge yeah a specific word for example like the bank we use as before in the
[00:46:07] like the bank we use as before in the previous slides um so is there like a
[00:46:11] previous slides um so is there like a like for the word representation is
[00:46:13] like for the word representation is there like a single embedding for each
[00:46:16] there like a single embedding for each word or do we have multiple embeddings
[00:46:18] word or do we have multiple embeddings for each word but what we're doing at
[00:46:20] for each word but what we're doing at the moment each word each string of
[00:46:24] the moment each word each string of letters has a single embed in and what
[00:46:28] letters has a single embed in and what you can think of that embedding as as
[00:46:32] you can think of that embedding as as kind of as an average over all its
[00:46:35] kind of as an average over all its senses um so for example like Bank it
[00:46:38] senses um so for example like Bank it can mean like the financial institution
[00:46:41] can mean like the financial institution or it can also mean like the river B and
[00:46:44] or it can also mean like the river B and then what I said before about star
[00:46:46] then what I said before about star applies the interesting thing is you'll
[00:46:50] applies the interesting thing is you'll find that we're able to come up with a
[00:46:52] find that we're able to come up with a representation where our learned
[00:46:53] representation where our learned representation because it's kind of an
[00:46:55] representation because it's kind of an average of those will end up similar to
[00:46:58] average of those will end up similar to words that are semantically evoked by
[00:47:01] words that are semantically evoked by both senses um I think I probably about
[00:47:04] both senses um I think I probably about about go on at this point um okay word
[00:47:08] about go on at this point um okay word to ve okay so word Tove was this method
[00:47:12] to ve okay so word Tove was this method of learning word vectors um that was
[00:47:15] of learning word vectors um that was thought up by tamas mikolov and
[00:47:17] thought up by tamas mikolov and colleagues at Google um in
[00:47:20] colleagues at Google um in 2013 you know it wasn't the first method
[00:47:23] 2013 you know it wasn't the first method there are other people that did methods
[00:47:25] there are other people that did methods of learning word vectors that go back to
[00:47:28] of learning word vectors that go back to about um the turn of the Millennium um
[00:47:31] about um the turn of the Millennium um it wasn't the last there ones that come
[00:47:33] it wasn't the last there ones that come after it as well but it was a
[00:47:35] after it as well but it was a particularly simple one um and this
[00:47:38] particularly simple one um and this particularly you know fast running one
[00:47:41] particularly you know fast running one and so it really caught people's um
[00:47:43] and so it really caught people's um attention so the idea of it is um that
[00:47:47] attention so the idea of it is um that we start off with a large amount of text
[00:47:52] we start off with a large amount of text so that can just be thought of as a long
[00:47:54] so that can just be thought of as a long list of words and in l p we refer to
[00:47:57] list of words and in l p we refer to that as a corpus Corpus um is just Latin
[00:48:01] that as a corpus Corpus um is just Latin for body um so you know it's exactly the
[00:48:04] for body um so you know it's exactly the same as if you have a dead person on the
[00:48:05] same as if you have a dead person on the floor right that's a corpus no um yeah
[00:48:10] floor right that's a corpus no um yeah so it's just a body but we mean a body
[00:48:11] so it's just a body but we mean a body of text um not a live person oh sorry a
[00:48:16] of text um not a live person oh sorry a dead person um yeah um if you want to
[00:48:19] dead person um yeah um if you want to know more about Latin um since some
[00:48:21] know more about Latin um since some there isn't very good classical
[00:48:23] there isn't very good classical education these days um Corpus despite
[00:48:27] education these days um Corpus despite the US ending is a fourth declen neuter
[00:48:31] the US ending is a fourth declen neuter noun and that means um the plural of
[00:48:35] noun and that means um the plural of Corpus is not corpy the plural of Corpus
[00:48:40] Corpus is not corpy the plural of Corpus is corpora um so I'm sure sometime later
[00:48:45] is corpora um so I'm sure sometime later in this class I will read a projector
[00:48:48] in this class I will read a projector assignment that refers to corpy and I
[00:48:51] assignment that refers to corpy and I will know that that person was not
[00:48:52] will know that that person was not paying attention um in the first lecture
[00:48:55] paying attention um in the first lecture um or else they should have said corpora
[00:48:59] um or else they should have said corpora um c r p o r a is the correct um form
[00:49:02] um c r p o r a is the correct um form for that okay I should move on okay so
[00:49:05] for that okay I should move on okay so we have our text then we know um that
[00:49:10] we have our text then we know um that we're going to represent each word so
[00:49:13] we're going to represent each word so this is each word type so you know star
[00:49:16] this is each word type so you know star or Bank Etc so for wherever it occurs by
[00:49:20] or Bank Etc so for wherever it occurs by a single vector and so what we're going
[00:49:23] a single vector and so what we're going to do in this algorithm is we're going
[00:49:25] to do in this algorithm is we're going to go through each position in the text
[00:49:28] to go through each position in the text and so at each position in the text
[00:49:30] and so at each position in the text which is a list of words we're going to
[00:49:32] which is a list of words we're going to have a center word and words outside it
[00:49:36] have a center word and words outside it um and then what we're going to do is
[00:49:38] um and then what we're going to do is use the similarity of the word vectors
[00:49:42] use the similarity of the word vectors for C and the outside words to calculate
[00:49:45] for C and the outside words to calculate the probability that they should have
[00:49:47] the probability that they should have occurred or not um and then we just keep
[00:49:50] occurred or not um and then we just keep fiddling and we learn ve word vectors
[00:49:53] fiddling and we learn ve word vectors now you know at First Sight I'll show
[00:49:56] now you know at First Sight I'll show this more concretely maybe I'll just
[00:49:57] this more concretely maybe I'll just show up more concretely first so here's
[00:49:59] show up more concretely first so here's the idea we're going to have um a vector
[00:50:03] the idea we're going to have um a vector for each word type so a word type means
[00:50:06] for each word type so a word type means you know the word problems wherever it
[00:50:08] you know the word problems wherever it occurs which is differentiated from a
[00:50:11] occurs which is differentiated from a word token which is this instance of the
[00:50:14] word token which is this instance of the word problems so we're going to have a
[00:50:16] word problems so we're going to have a vector for each word type and so I'm
[00:50:20] vector for each word type and so I'm going to want to know look in this text
[00:50:24] going to want to know look in this text the word turning occurred before
[00:50:26] the word turning occurred before occurred before the word into How likely
[00:50:29] occurred before the word into How likely should that have been to happen and what
[00:50:32] should that have been to happen and what I'm going to do is calculate a
[00:50:34] I'm going to do is calculate a probability of the word turning
[00:50:37] probability of the word turning occurring close to the word into and I'm
[00:50:40] occurring close to the word into and I'm going to do that for each word in a
[00:50:43] going to do that for each word in a narrow context in example here I'm
[00:50:45] narrow context in example here I'm saying I'm using two words to the left
[00:50:48] saying I'm using two words to the left and two words to the right and what I
[00:50:50] and two words to the right and what I want to do is make those probability
[00:50:53] want to do is make those probability estimates as good as possible so in
[00:50:55] estimates as good as possible so in particular I want the probability of
[00:50:57] particular I want the probability of cooccurrence to be high for words that
[00:51:00] cooccurrence to be high for words that actually do occur within the nearby
[00:51:02] actually do occur within the nearby context of each other and so then the
[00:51:05] context of each other and so then the question is how am I going to oh and
[00:51:09] question is how am I going to oh and once I've done it for that word I'm
[00:51:10] once I've done it for that word I'm going to go along and do exactly the
[00:51:13] going to go along and do exactly the same thing for the next
[00:51:15] same thing for the next word and so I'll can continue through
[00:51:18] word and so I'll can continue through the text in that way and so what we want
[00:51:22] the text in that way and so what we want to do is come up with vector
[00:51:26] to do is come up with vector representations of words that will let
[00:51:29] representations of words that will let us predict these probabilities quote
[00:51:32] us predict these probabilities quote unquote well now you know there's a huge
[00:51:35] unquote well now you know there's a huge limit to how well we can do it because
[00:51:37] limit to how well we can do it because you know we've got a simple model
[00:51:39] you know we've got a simple model obviously when you see the word banking
[00:51:42] obviously when you see the word banking I can't tell you that the word into is
[00:51:44] I can't tell you that the word into is going to occur before banking but you
[00:51:47] going to occur before banking but you know I want to do it as well as possible
[00:51:50] know I want to do it as well as possible so what I want my model to say is after
[00:51:53] so what I want my model to say is after the word banking crisis is pretty likely
[00:51:58] the word banking crisis is pretty likely um but the word skillet is not very
[00:52:03] um but the word skillet is not very likely and if I can do that I'm doing a
[00:52:06] likely and if I can do that I'm doing a good job and so we turn that into a
[00:52:09] good job and so we turn that into a piece of math um here's how we do it
[00:52:12] piece of math um here's how we do it turn it into a piece of math so we're
[00:52:15] turn it into a piece of math so we're going to go through our Corpus every
[00:52:17] going to go through our Corpus every position in the Corpus um and we're
[00:52:20] position in the Corpus um and we're going to have a fixed window size M
[00:52:23] going to have a fixed window size M which was two in my example and then
[00:52:26] which was two in my example and then what we you're going to want to do is
[00:52:27] what we you're going to want to do is have the probability of words in the
[00:52:30] have the probability of words in the context um being as high as possible so
[00:52:34] context um being as high as possible so we want to maximize this likelihood
[00:52:37] we want to maximize this likelihood where we're going through every position
[00:52:38] where we're going through every position in the text and then we're going through
[00:52:41] in the text and then we're going through every word in the context and sort of
[00:52:44] every word in the context and sort of wanting to make this
[00:52:47] big okay so conceptually that's what
[00:52:51] big okay so conceptually that's what we're doing but in practice um we never
[00:52:54] we're doing but in practice um we never quite do that um we um use two little
[00:52:59] quite do that um we um use two little tricks here um the first one is you know
[00:53:02] tricks here um the first one is you know for completely arbitrary reasons it
[00:53:05] for completely arbitrary reasons it really makes no difference um everyone
[00:53:08] really makes no difference um everyone got into minimizing things rather than
[00:53:11] got into minimizing things rather than maximizing things and so the algorithms
[00:53:14] maximizing things and so the algorithms that we use get referred to as gradient
[00:53:16] that we use get referred to as gradient descent as you'll see in a moment so the
[00:53:18] descent as you'll see in a moment so the first thing we do um is put a minor sign
[00:53:22] first thing we do um is put a minor sign in front so that we can minimize it
[00:53:24] in front so that we can minimize it rather than maximize it that part's
[00:53:25] rather than maximize it that part's pretty trivial
[00:53:27] pretty trivial but the second part is here we have this
[00:53:29] but the second part is here we have this enormous product and working with
[00:53:32] enormous product and working with enormous products is more difficult for
[00:53:34] enormous products is more difficult for the math so the second thing that we do
[00:53:37] the math so the second thing that we do is introduce a logarithm and so once we
[00:53:40] is introduce a logarithm and so once we take the log of the likelihood um that
[00:53:44] take the log of the likelihood um that then um when we take logs of products
[00:53:48] then um when we take logs of products they turn into sums and so now we can
[00:53:51] they turn into sums and so now we can sum over each word position the text sum
[00:53:56] sum over each word position the text sum over for each word in the context window
[00:53:58] over for each word in the context window and then sum these log probabilities and
[00:54:01] and then sum these log probabilities and then we still got the minus sign in
[00:54:03] then we still got the minus sign in front so we want to minimize the sum of
[00:54:06] front so we want to minimize the sum of log probabilities so what we're doing is
[00:54:10] log probabilities so what we're doing is um then wanting to look at the negative
[00:54:13] um then wanting to look at the negative log
[00:54:15] log likelihood um and then the final thing
[00:54:17] likelihood um and then the final thing that we do is you know to since this
[00:54:20] that we do is you know to since this will get um bigger depending on the
[00:54:23] will get um bigger depending on the number of words in the Corpus um we
[00:54:25] number of words in the Corpus um we divide through by the number of words in
[00:54:27] divide through by the number of words in the Corpus and so our objective function
[00:54:30] the Corpus and so our objective function is the average negative log likelihood
[00:54:33] is the average negative log likelihood um so by minimizing this objective
[00:54:36] um so by minimizing this objective function we're maximizing the
[00:54:39] function we're maximizing the probability of words in the
[00:54:41] probability of words in the context okay um we're almost there
[00:54:46] context okay um we're almost there that's what we want to do um we've got a
[00:54:49] that's what we want to do um we've got a couple more tricks that we want to get
[00:54:52] couple more tricks that we want to get through the next one is well I've said
[00:54:54] through the next one is well I've said we want to maximize the this probability
[00:54:58] we want to maximize the this probability how do we maximize this probability what
[00:55:00] how do we maximize this probability what is this probability we haven't defined
[00:55:03] is this probability we haven't defined how we're going to calculate this
[00:55:05] how we're going to calculate this probability and this is where the word
[00:55:08] probability and this is where the word vectors come in so we're going to Define
[00:55:11] vectors come in so we're going to Define this
[00:55:12] this probability in terms of the word Vector
[00:55:15] probability in terms of the word Vector so we're going to say each word type is
[00:55:18] so we're going to say each word type is represented by a vector of real numbers
[00:55:21] represented by a vector of real numbers these are 100 real numbers and we're
[00:55:23] these are 100 real numbers and we're going to have a formula that works out
[00:55:26] going to have a formula that works out the probability simply in terms of the
[00:55:30] the probability simply in terms of the vectors of for each um word that there
[00:55:33] vectors of for each um word that there are no other parameters in this model so
[00:55:36] are no other parameters in this model so over here I've shown this Theta which
[00:55:38] over here I've shown this Theta which are the parameters of our model and all
[00:55:41] are the parameters of our model and all and only the parameters of our model are
[00:55:45] and only the parameters of our model are these word vectors for each word in the
[00:55:48] these word vectors for each word in the vocabulary that's a lot of parameters
[00:55:50] vocabulary that's a lot of parameters because we have a lot of words and and
[00:55:52] because we have a lot of words and and we've got fairly big word vectors but
[00:55:54] we've got fairly big word vectors but they are the only parameters
[00:55:57] they are the only parameters okay and how we do that is um by using
[00:56:01] okay and how we do that is um by using this little um trick here we're going to
[00:56:03] this little um trick here we're going to say the probability of an outside word
[00:56:06] say the probability of an outside word given a center word is going to be
[00:56:09] given a center word is going to be defined in terms of the dotproduct of
[00:56:12] defined in terms of the dotproduct of the two word vectors so if things have a
[00:56:15] the two word vectors so if things have a high dot product they'll be similar and
[00:56:19] high dot product they'll be similar and therefore they'll have a high
[00:56:20] therefore they'll have a high probability of cooccurrence where I mean
[00:56:23] probability of cooccurrence where I mean similar in a kind of a weird sense right
[00:56:26] similar in a kind of a weird sense right it is the case that we're going to want
[00:56:27] it is the case that we're going to want to say hotel and motel are similar but
[00:56:30] to say hotel and motel are similar but you know it's also the case that we're
[00:56:32] you know it's also the case that we're going to want to have the word the able
[00:56:36] going to want to have the word the able to appear easily before the word student
[00:56:39] to appear easily before the word student so in some weird sense the also has to
[00:56:42] so in some weird sense the also has to be similar to student um that has to be
[00:56:44] be similar to student um that has to be similar to basically any noun right um
[00:56:48] similar to basically any noun right um okay so we're going to work witht
[00:56:50] okay so we're going to work witht products and then we do this funky
[00:56:52] products and then we do this funky little bit of math here and that will
[00:56:54] little bit of math here and that will give us our probability
[00:56:56] give us our probability ities okay so let's just go through the
[00:56:59] ities okay so let's just go through the funky bit of math um so here's our
[00:57:03] funky bit of math um so here's our formula for the
[00:57:04] formula for the probabilities so what we're doing here
[00:57:07] probabilities so what we're doing here is we're starting off with this dot
[00:57:10] is we're starting off with this dot product right so the dot product is you
[00:57:13] product right so the dot product is you take the two vectors you multiply each
[00:57:16] take the two vectors you multiply each component together and you sum them so
[00:57:18] component together and you sum them so if they both the same sign that
[00:57:22] if they both the same sign that increases your dot product and if
[00:57:24] increases your dot product and if they're both big and increases it a lot
[00:57:26] they're both big and increases it a lot okay so that gives us a similarity
[00:57:30] okay so that gives us a similarity between two vectors and that's unbounded
[00:57:33] between two vectors and that's unbounded that's that's just a real number it can
[00:57:35] that's that's just a real number it can be either negative or positive okay but
[00:57:37] be either negative or positive okay but what we'd like to get out is a
[00:57:39] what we'd like to get out is a probability so for our next tricks we
[00:57:42] probability so for our next tricks we first of all exponentiate because if we
[00:57:45] first of all exponentiate because if we take um e to X for any X we now have to
[00:57:50] take um e to X for any X we now have to get something positive out right that's
[00:57:52] get something positive out right that's what exponentiation does okay and then
[00:57:55] what exponentiation does okay and then well since it's meant to be a
[00:57:57] well since it's meant to be a probability we'd like it to be between 0
[00:58:00] probability we'd like it to be between 0 and 1 and so we turn it into numbers
[00:58:03] and 1 and so we turn it into numbers between 0 and one in the dumbest way
[00:58:06] between 0 and one in the dumbest way possible which is we just normalize so
[00:58:09] possible which is we just normalize so that we work out the quantity in the
[00:58:11] that we work out the quantity in the numerator for every possible context
[00:58:14] numerator for every possible context word um and so we get the total of all
[00:58:18] word um and so we get the total of all of those numbers and divide through by
[00:58:20] of those numbers and divide through by it and then we're getting a probability
[00:58:22] it and then we're getting a probability distribution of How likely different
[00:58:24] distribution of How likely different words are in this context
[00:58:27] words are in this context text okay um yeah so that this little
[00:58:31] text okay um yeah so that this little trick that we're doing here is referred
[00:58:34] trick that we're doing here is referred to as the softmax function so for the
[00:58:36] to as the softmax function so for the softmax function you can take um
[00:58:40] softmax function you can take um unbounded real numbers put them through
[00:58:43] unbounded real numbers put them through this little softmax trick that we just
[00:58:45] this little softmax trick that we just went through the steps of and what
[00:58:47] went through the steps of and what you'll get out is a probability
[00:58:50] you'll get out is a probability distribution so I'm now getting in this
[00:58:53] distribution so I'm now getting in this example uh probability distribution over
[00:58:57] example uh probability distribution over context words my probability estimates
[00:59:00] context words my probability estimates over all the context words in my
[00:59:02] over all the context words in my vocabulary will sum up to one by
[00:59:05] vocabulary will sum up to one by definition by the way that I've con
[00:59:07] definition by the way that I've con constructed this um so it's called the
[00:59:10] constructed this um so it's called the softmax function because it amplifies
[00:59:14] softmax function because it amplifies the probabilities of the largest things
[00:59:16] the probabilities of the largest things that's because of the um the X function
[00:59:21] that's because of the um the X function but it's soft because it still assigns
[00:59:24] but it's soft because it still assigns some probability to smaller items but
[00:59:27] some probability to smaller items but you know it's sort of a funny name
[00:59:30] you know it's sort of a funny name because you know when you think about um
[00:59:33] because you know when you think about um max I mean Max normally picks out just
[00:59:36] max I mean Max normally picks out just one thing whereas the soft Max is
[00:59:39] one thing whereas the soft Max is turning a bunch of real numbers into a
[00:59:42] turning a bunch of real numbers into a probability
[00:59:43] probability distribution um so this soft Max is used
[00:59:47] distribution um so this soft Max is used everywhere um in deep learning any time
[00:59:50] everywhere um in deep learning any time that we're wanting to turn things that
[00:59:51] that we're wanting to turn things that are just vectors in RN into
[00:59:54] are just vectors in RN into probabilities we shove them through a
[00:59:56] probabilities we shove them through a soft Max
[00:59:58] soft Max function okay
[01:00:02] function okay um so
[01:00:06] um so so in some sense this part I think still
[01:00:10] so in some sense this part I think still seems very abstract and I mean the
[01:00:14] seems very abstract and I mean the reason it seems very abstract is um
[01:00:19] reason it seems very abstract is um because I've sort of said we have
[01:00:22] because I've sort of said we have vectors for each word and using these
[01:00:25] vectors for each word and using these vectors
[01:00:26] vectors we can then calculate
[01:00:28] we can then calculate probabilities um but where do the
[01:00:31] probabilities um but where do the vectors come from and the answer to
[01:00:34] vectors come from and the answer to where the vectors are going to come from
[01:00:37] where the vectors are going to come from is we're going to turn this into an
[01:00:39] is we're going to turn this into an optimization problem we have a large
[01:00:42] optimization problem we have a large amount of text and so therefore we can
[01:00:46] amount of text and so therefore we can hope to find word vectors that make the
[01:00:50] hope to find word vectors that make the contexts of the words in our observed
[01:00:53] contexts of the words in our observed text as big as possible so literally
[01:00:56] text as big as possible so literally what we're going to do is we're going to
[01:00:59] what we're going to do is we're going to start off with random vectors for every
[01:01:01] start off with random vectors for every word and then we want to fiddle those
[01:01:04] word and then we want to fiddle those vectors so that the calculated
[01:01:08] vectors so that the calculated probabilities of words in a context go
[01:01:11] probabilities of words in a context go up and we're going to keep fiddling
[01:01:13] up and we're going to keep fiddling until they stop going up anymore and
[01:01:15] until they stop going up anymore and we're getting the highest probability
[01:01:17] we're getting the highest probability estimates um that we can and the way
[01:01:20] estimates um that we can and the way that we do that fiddling um is we use
[01:01:23] that we do that fiddling um is we use calculus um so you know what we're going
[01:01:26] calculus um so you know what we're going to do is kind of conceptually exactly
[01:01:30] to do is kind of conceptually exactly what you do if you're in something like
[01:01:31] what you do if you're in something like a two-dimensional space like the picture
[01:01:33] a two-dimensional space like the picture on the right right that if you want to
[01:01:36] on the right right that if you want to find the minimum in this two-dimensional
[01:01:38] find the minimum in this two-dimensional space and you start off at the top left
[01:01:41] space and you start off at the top left what you can do is say let me work out
[01:01:44] what you can do is say let me work out the derivatives of the function um at
[01:01:47] the derivatives of the function um at the top left and they sort of Point sort
[01:01:50] the top left and they sort of Point sort of down and a bit to the right and so
[01:01:52] of down and a bit to the right and so you can walk down and a bit to the right
[01:01:54] you can walk down and a bit to the right and you can say oh G given where I am
[01:01:57] and you can say oh G given where I am now um let me work out the derivatives
[01:02:00] now um let me work out the derivatives what direction do they point and they're
[01:02:02] what direction do they point and they're still pointing down but a bit more to
[01:02:03] still pointing down but a bit more to the right so you can walk a bit further
[01:02:05] the right so you can walk a bit further that way and you can keep on walking and
[01:02:08] that way and you can keep on walking and eventually you'll make it to the minimum
[01:02:11] eventually you'll make it to the minimum of the space um in our case um we've got
[01:02:15] of the space um in our case um we've got a lot more than two Dimensions so our
[01:02:18] a lot more than two Dimensions so our parameters for our model are the
[01:02:21] parameters for our model are the concatenation of all the word vectors
[01:02:25] concatenation of all the word vectors but it's even slightly worse than I've
[01:02:26] but it's even slightly worse than I've explained um up until now because
[01:02:30] explained um up until now because actually for each word we assume two
[01:02:34] actually for each word we assume two vectors we assume one vector when
[01:02:36] vectors we assume one vector when they're the center word and one vector
[01:02:38] they're the center word and one vector when they're the outside word doing that
[01:02:41] when they're the outside word doing that just makes the math a bit simpler which
[01:02:43] just makes the math a bit simpler which I can explain later so if we say had a
[01:02:46] I can explain later so if we say had a 100 dimensional vectors we'll have 100
[01:02:49] 100 dimensional vectors we'll have 100 parameters for ad v as an outside word
[01:02:53] parameters for ad v as an outside word 100 parameters for um ah as an outside
[01:02:56] 100 parameters for um ah as an outside word all the way through to 100
[01:02:58] word all the way through to 100 parameters for zebra as an outside word
[01:03:01] parameters for zebra as an outside word then we'd have 100 parameters for arvar
[01:03:03] then we'd have 100 parameters for arvar as a um a center word continuing down so
[01:03:08] as a um a center word continuing down so you know if we had a vocabulary of
[01:03:10] you know if we had a vocabulary of 400,000 words and 100 um dimensional
[01:03:14] 400,000 words and 100 um dimensional word vectors that means we'd have
[01:03:16] word vectors that means we'd have 400,000 * 2 is
[01:03:19] 400,000 * 2 is 800,000 time 100 we'd have 80 million
[01:03:22] 800,000 time 100 we'd have 80 million parameters so that's a lot of parameters
[01:03:25] parameters so that's a lot of parameters in our space to try and Fiddle to
[01:03:28] in our space to try and Fiddle to optimize things but luckily we have big
[01:03:30] optimize things but luckily we have big computers um and that's the kind of
[01:03:33] computers um and that's the kind of thing that we do so we simply um say G
[01:03:38] thing that we do so we simply um say G this is our optimization problem we're
[01:03:41] this is our optimization problem we're going to compute the gradients of all of
[01:03:46] going to compute the gradients of all of these parameters and that will um give
[01:03:49] these parameters and that will um give us the answer of what we have um and you
[01:03:55] us the answer of what we have um and you know
[01:03:57] know this feels like magic I mean it doesn't
[01:04:00] this feels like magic I mean it doesn't really seem like you know we could just
[01:04:03] really seem like you know we could just start with nothing we could start with
[01:04:05] start with nothing we could start with random word vectors and a pile of text
[01:04:08] random word vectors and a pile of text and say uh do some math and we will get
[01:04:12] and say uh do some math and we will get something useful out um but the miracle
[01:04:15] something useful out um but the miracle of what happens in these deep learning
[01:04:17] of what happens in these deep learning spaces is we do get something useful out
[01:04:20] spaces is we do get something useful out we can just um minimize um all of the
[01:04:23] we can just um minimize um all of the parameters and
[01:04:26] parameters and um then we'll get something useful out
[01:04:29] um then we'll get something useful out um so what I wanted to uh I guess I'm
[01:04:32] um so what I wanted to uh I guess I'm not going to quite get to the end of
[01:04:34] not going to quite get to the end of what I hope to today um but what we I
[01:04:36] what I hope to today um but what we I wanted to do is sort of um get through
[01:04:41] wanted to do is sort of um get through um some of um what we do here but you
[01:04:45] um some of um what we do here but you know I wanted to take a few minutes to
[01:04:47] know I wanted to take a few minutes to sort of go through concretely how we do
[01:04:50] sort of go through concretely how we do um the math of
[01:04:52] um the math of minimization now lots of different
[01:04:55] minimization now lots of different people will um take um
[01:04:59] people will um take um cs224n and some of you know way more
[01:05:02] cs224n and some of you know way more math than I do and so if this next 10
[01:05:06] math than I do and so if this next 10 minutes might be extremely boring and if
[01:05:09] minutes might be extremely boring and if that's the case you can either catch up
[01:05:11] that's the case you can either catch up on Discord or Instagram or something or
[01:05:14] on Discord or Instagram or something or else you can leave but it turns out
[01:05:16] else you can leave but it turns out there are other people that do
[01:05:18] there are other people that do cs224n that can't quite remember um when
[01:05:21] cs224n that can't quite remember um when they lasted a math course and we'd like
[01:05:25] they lasted a math course and we'd like um everybody to be able to learn
[01:05:27] um everybody to be able to learn something about this um so I do actually
[01:05:30] something about this um so I do actually like in the first two weeks to kind of
[01:05:32] like in the first two weeks to kind of go through it a bit concretely so let's
[01:05:35] go through it a bit concretely so let's um try to do this so this was our
[01:05:37] um try to do this so this was our likelihood and then we'd already covered
[01:05:40] likelihood and then we'd already covered the fact that what we were going to do
[01:05:42] the fact that what we were going to do is have an objective function in terms
[01:05:44] is have an objective function in terms of our parameters that was the negative
[01:05:48] of our parameters that was the negative the average negative log likelihood
[01:05:51] the average negative log likelihood across all the
[01:05:52] across all the words
[01:05:54] words um if I remember the notation for this
[01:05:58] um if I remember the notation for this the
[01:05:59] the sum um in this
[01:06:03] sum um in this oops um I'll probably have a hard time
[01:06:05] oops um I'll probably have a hard time writing this um the sum of position M
[01:06:11] writing this um the sum of position M I've got a more neatly written out
[01:06:13] I've got a more neatly written out version of it that appears on the
[01:06:14] version of it that appears on the version of the slides it's on the
[01:06:17] version of the slides it's on the webiz um and then we're going to be
[01:06:19] webiz um and then we're going to be taking this
[01:06:21] taking this log of the probability of the word
[01:06:26] log of the probability of the word at
[01:06:27] at position um t
[01:06:32] plus sorry position J um t +
[01:06:38] plus sorry position J um t + [Music]
[01:06:39] [Music] J
[01:06:41] J okay trying to write this on my iPad is
[01:06:44] okay trying to write this on my iPad is not working super well I'll confess
[01:06:47] not working super well I'll confess we'll see how I get on um
[01:06:50] we'll see how I get on um WT okay
[01:06:53] WT okay um okay and so then we had the form of
[01:06:57] um okay and so then we had the form of what we um wanted to use for the
[01:07:01] what we um wanted to use for the probability and the probability of an
[01:07:04] probability and the probability of an outside word given a context word is was
[01:07:08] outside word given a context word is was then this soft maxed equation where
[01:07:10] then this soft maxed equation where we're taking the x of the outside
[01:07:16] we're taking the x of the outside vector and the center
[01:07:20] vector and the center Vector over the normalization term where
[01:07:24] Vector over the normalization term where we sum over the
[01:07:38] vocabulary okay um so
[01:07:42] vocabulary okay um so to to work out um how to change our
[01:07:46] to to work out um how to change our parameters so our parameters are all of
[01:07:49] parameters so our parameters are all of these word vectors that we summarize um
[01:07:53] these word vectors that we summarize um inside Theta
[01:07:55] inside Theta what we're then going to want to do is
[01:07:58] what we're then going to want to do is work out the partial
[01:08:00] work out the partial derivative of this objective function
[01:08:04] derivative of this objective function with respect to all the parameters Theta
[01:08:08] with respect to all the parameters Theta but you know in particular um I'm going
[01:08:11] but you know in particular um I'm going to cons just start doing here the
[01:08:14] to cons just start doing here the partial derivatives with respect to um
[01:08:17] partial derivatives with respect to um the
[01:08:18] the center
[01:08:20] center word and we can work through the outside
[01:08:23] word and we can work through the outside words separately well this partial
[01:08:26] words separately well this partial derivative is a big a big sum and it's a
[01:08:29] derivative is a big a big sum and it's a big sum of terms like this and so when I
[01:08:34] big sum of terms like this and so when I have a partial derivative of a big sum
[01:08:36] have a partial derivative of a big sum of terms I can work out the partial
[01:08:39] of terms I can work out the partial derivatives of each term independently
[01:08:42] derivatives of each term independently and then sum them so what I want to be
[01:08:44] and then sum them so what I want to be doing is working out um the partial
[01:08:49] doing is working out um the partial derivative
[01:08:51] derivative of the log of this probability which
[01:08:54] of the log of this probability which equals the the log of
[01:08:57] equals the the log of that with respect to the center Vector
[01:09:02] that with respect to the center Vector um and so at this point I have a log of
[01:09:07] um and so at this point I have a log of two things being divided and so that
[01:09:10] two things being divided and so that means I can separate that out of the log
[01:09:13] means I can separate that out of the log of the numerator minus the log of the
[01:09:17] of the numerator minus the log of the denominator and so what I'll be doing is
[01:09:20] denominator and so what I'll be doing is working out the partial derivative with
[01:09:23] working out the partial derivative with respect to the center vector
[01:09:25] respect to the center vector of the log the
[01:09:28] of the log the numerator um log X of
[01:09:33] numerator um log X of utvc
[01:09:34] utvc minus um
[01:09:37] minus um the partial
[01:09:39] the partial derivative um with respect to the
[01:09:42] derivative um with respect to the denominator which is then the log of the
[01:09:48] denominator which is then the log of the sum of w = 1 to V of x
[01:09:58] who okay I'm having real trouble here
[01:10:02] who okay I'm having real trouble here writing I look at the slides where I
[01:10:04] writing I look at the slides where I wrote it neatly at home okay um so I
[01:10:08] wrote it neatly at home okay um so I want to work with these two terms now at
[01:10:13] want to work with these two terms now at this point um part of
[01:10:17] it at this point part of it is easy
[01:10:21] it at this point part of it is easy because here I just have a log of an
[01:10:23] because here I just have a log of an exponential and so so those two
[01:10:26] exponential and so so those two functions just cancel out and go away
[01:10:29] functions just cancel out and go away and so then I want to get the partial
[01:10:32] and so then I want to get the partial derivative of U outside transpose V
[01:10:37] derivative of U outside transpose V Center um with respect to V Center and
[01:10:41] Center um with respect to V Center and so um what you get for the answer to
[01:10:44] so um what you get for the answer to that is that that just comes out um as
[01:10:49] that is that that just comes out um as U and um maybe you remember that but if
[01:10:54] U and um maybe you remember that but if you don't remember that the thing to
[01:10:56] you don't remember that the thing to think about is okay this is a whole
[01:11:00] think about is okay this is a whole Vector right and so we've got a vector
[01:11:03] Vector right and so we've got a vector here and a vector here so what this is
[01:11:05] here and a vector here so what this is going to be looking like is like sort of
[01:11:08] going to be looking like is like sort of U1 V1 plus U2
[01:11:13] U1 V1 plus U2 V2 um plus u3 V3 Etc long and so what
[01:11:19] V2 um plus u3 V3 Etc long and so what we're going to want to do is work out
[01:11:21] we're going to want to do is work out the partial derivative with respect to
[01:11:24] the partial derivative with respect to each element
[01:11:26] each element um VI right and so if you just think of
[01:11:29] um VI right and so if you just think of a sort of a single element derivative
[01:11:32] a sort of a single element derivative with respect to um
[01:11:34] with respect to um V1 well it's going to be just U1 because
[01:11:39] V1 well it's going to be just U1 because every other term would go to zero um and
[01:11:41] every other term would go to zero um and then if you worked it out with respect
[01:11:43] then if you worked it out with respect to V2 then it would be just U2 and every
[01:11:47] to V2 then it would be just U2 and every other term goes to zero and so since you
[01:11:50] other term goes to zero and so since you keep on doing that along the whole
[01:11:52] keep on doing that along the whole Vector that what you're going to get out
[01:11:54] Vector that what you're going to get out is the vector
[01:11:55] is the vector um U1 U2
[01:11:58] um U1 U2 u3 um down the vocab um for the whole
[01:12:02] u3 um down the vocab um for the whole list of vocab
[01:12:03] list of vocab items okay so that part is easy um but
[01:12:08] items okay so that part is easy um but then we also want to um work out the
[01:12:13] then we also want to um work out the partial derivatives of that one and at
[01:12:16] partial derivatives of that one and at that point I maybe have to um go to
[01:12:20] that point I maybe have to um go to another slide so we then want to have
[01:12:25] another slide so we then want to have um the partial
[01:12:29] um the partial derivative
[01:12:30] derivative um with respect to
[01:12:33] um with respect to VC of
[01:12:35] VC of the
[01:12:37] the log of the sum equals W 1 to V of the x
[01:12:44] log of the sum equals W 1 to V of the x u w transvers VC right so at this point
[01:12:50] u w transvers VC right so at this point things aren't quite so easy and we have
[01:12:53] things aren't quite so easy and we have to remember a little bit more calcul
[01:12:55] to remember a little bit more calcul so in particular what we have to
[01:12:57] so in particular what we have to remember is the chain rule so here we
[01:13:01] remember is the chain rule so here we have this inside function so that we've
[01:13:04] have this inside function so that we've got a function um we've got a function G
[01:13:10] got a function um we've got a function G of VC which you know we might say the
[01:13:13] of VC which you know we might say the output of that is Zed and then we put
[01:13:17] output of that is Zed and then we put outside that an extra function f and so
[01:13:22] outside that an extra function f and so when we have something like that what we
[01:13:25] when we have something like that what we get is the derivative of f with respect
[01:13:28] get is the derivative of f with respect to VC we can take the derivative of f
[01:13:33] to VC we can take the derivative of f with respect to Z times the derivative
[01:13:36] with respect to Z times the derivative of Z with respect to um VC right that's
[01:13:40] of Z with respect to um VC right that's the chain rule so we're going to then
[01:13:44] the chain rule so we're going to then apply that here so first of all we're
[01:13:48] apply that here so first of all we're going to take the derivative of log um
[01:13:52] going to take the derivative of log um and so the derivative of log is one X
[01:13:55] and so the derivative of log is one X you have to remember that or look it up
[01:13:57] you have to remember that or look it up or get mathematic to do it for you or
[01:14:00] or get mathematic to do it for you or something like that
[01:14:02] something like that um and so we're going to have one over
[01:14:07] um and so we're going to have one over the inside Z part the sum of w equal 1
[01:14:11] the inside Z part the sum of w equal 1 to V of the X
[01:14:14] to V of the X uwt
[01:14:16] uwt VC um and then that's going to be
[01:14:20] VC um and then that's going to be multiplied by the derivative of the
[01:14:24] multiplied by the derivative of the inside part part
[01:14:26] inside part part um so then we're going to have the
[01:14:30] um so then we're going to have the derivative with respect to VC of the sum
[01:14:34] derivative with respect to VC of the sum of w =
[01:14:37] of w = 1 to
[01:14:40] 1 to V of the X
[01:14:47] of okay um
[01:14:51] of okay um so um so that's um made us a little bit
[01:14:55] so um so that's um made us a little bit of progress um but we've still got
[01:14:58] of progress um but we've still got something to do here um and so well what
[01:15:01] something to do here um and so well what we're going to do here is we're going to
[01:15:03] we're going to do here is we're going to notice Oh wait we're again in the space
[01:15:07] notice Oh wait we're again in the space to run the chain rule again so now we've
[01:15:10] to run the chain rule again so now we've got this function well so first of all
[01:15:13] got this function well so first of all we can move the sum to the outside right
[01:15:16] we can move the sum to the outside right because we've got a sum of terms W = one
[01:15:19] because we've got a sum of terms W = one to V and so we want to work out the
[01:15:21] to V and so we want to work out the derivatives of the inside piece um with
[01:15:25] derivatives of the inside piece um with respect to it sorry I'm doing this kind
[01:15:26] respect to it sorry I'm doing this kind of informally of just doing this piece
[01:15:28] of informally of just doing this piece now um okay so this again gives us an F
[01:15:33] now um okay so this again gives us an F over a function um
[01:15:37] over a function um G and so we're going to again want to
[01:15:40] G and so we're going to again want to split the pieces up and so use the chain
[01:15:43] split the pieces up and so use the chain rule one more time so we're going to
[01:15:45] rule one more time so we're going to have the sum of w = 1 to V and now we
[01:15:49] have the sum of w = 1 to V and now we have to know what the derivative of x is
[01:15:51] have to know what the derivative of x is and the derivative of x is X so that
[01:15:54] and the derivative of x is X so that will be X of
[01:15:58] will be X of uux
[01:16:00] uux tv0 and then we're taking the derivative
[01:16:04] tv0 and then we're taking the derivative of the inside part with respect to VC of
[01:16:09] of the inside part with respect to VC of ux um T
[01:16:11] ux um T VC well luckily this was the bit that we
[01:16:15] VC well luckily this was the bit that we already knew how to do because we worked
[01:16:17] already knew how to do because we worked it out before and so this is going to be
[01:16:19] it out before and so this is going to be the sum of w equal 1 to V of this X
[01:16:27] times
[01:16:30] times ux okay so then at this point um we want
[01:16:35] ux okay so then at this point um we want to combine these two forms together so
[01:16:39] to combine these two forms together so that we want to combine this part that
[01:16:42] that we want to combine this part that we worked out and this piece here that
[01:16:45] we worked out and this piece here that we've worked out and if we canine
[01:16:49] we've worked out and if we canine combine them together with what we
[01:16:52] combine them together with what we worked out on the first slide um for
[01:16:55] worked out on the first slide um for numerator um since this is we have
[01:17:00] numerator um since this is we have the U which was the derivative of the
[01:17:05] the U which was the derivative of the numerator and then for the um derivative
[01:17:09] numerator and then for the um derivative of the denominator we're going to
[01:17:13] of the denominator we're going to have um on top this part and then on the
[01:17:17] have um on top this part and then on the bottom we're going to have that part and
[01:17:19] bottom we're going to have that part and so we can rewrite that as the sum from
[01:17:23] so we can rewrite that as the sum from wal 1 to V of the
[01:17:29] wal 1 to V of the X of
[01:17:31] X of uxt
[01:17:33] uxt v0 *
[01:17:35] v0 * ux over um the
[01:17:38] ux over um the [Music]
[01:17:40] [Music] sum sorry x = 1 to V sum over W = 1 to V
[01:17:48] sum sorry x = 1 to V sum over W = 1 to V of the X this part here of UW
[01:17:56] okay so we can rearrange things in that
[01:18:00] okay so we can rearrange things in that form and then lo and behold we find that
[01:18:04] form and then lo and behold we find that we've recreated here this form of the
[01:18:07] we've recreated here this form of the softn equation so we end up with
[01:18:11] softn equation so we end up with u minus the sum x = 1 to V of the
[01:18:18] u minus the sum x = 1 to V of the probability of x given c times um U of x
[01:18:25] probability of x given c times um U of x so what this is saying is we're wanting
[01:18:29] so what this is saying is we're wanting to have this quantity which takes the
[01:18:32] to have this quantity which takes the actual observed U vector and it's
[01:18:35] actual observed U vector and it's comparing it to the weighted prediction
[01:18:40] comparing it to the weighted prediction so we're taking the weight of some of
[01:18:42] so we're taking the weight of some of the our current ux vectors um based on
[01:18:46] the our current ux vectors um based on How likely we we were they were to occur
[01:18:50] How likely we we were they were to occur and so this is a form that you see quite
[01:18:53] and so this is a form that you see quite a bit in these kind of der
[01:18:55] a bit in these kind of der you get exer observed minus expected the
[01:18:59] you get exer observed minus expected the weighted average and so what you'd like
[01:19:01] weighted average and so what you'd like to have is your expectation the weighted
[01:19:05] to have is your expectation the weighted average be the same as um the what was
[01:19:09] average be the same as um the what was observed because then you'll get a
[01:19:11] observed because then you'll get a derivative of zero which means that
[01:19:13] derivative of zero which means that you've hit a
[01:19:15] you've hit a maximum um and so that gives us you know
[01:19:19] maximum um and so that gives us you know the form of the derivative of the um
[01:19:25] the form of the derivative of the um that we're having with respect to the
[01:19:27] that we're having with respect to the center Vector parameters to finish it
[01:19:30] center Vector parameters to finish it off you'd have to then work it out also
[01:19:32] off you'd have to then work it out also for the um outside Vector parameters but
[01:19:34] for the um outside Vector parameters but hey it's officially the end of class
[01:19:37] hey it's officially the end of class time so I'd better wrap up quickly now
[01:19:39] time so I'd better wrap up quickly now but you know so the deal is we're going
[01:19:42] but you know so the deal is we're going to work out all of these derivatives um
[01:19:45] to work out all of these derivatives um for each parameter and then these
[01:19:48] for each parameter and then these derivatives will give a direction to
[01:19:50] derivatives will give a direction to change numbers which will let us find
[01:19:53] change numbers which will let us find good word vectors
[01:19:55] good word vectors automatically um I do want you to
[01:19:58] automatically um I do want you to understand how this works but
[01:20:00] understand how this works but fortunately you'll find out very quickly
[01:20:02] fortunately you'll find out very quickly that computers will do this for you and
[01:20:04] that computers will do this for you and on a regular basis you don't actually
[01:20:06] on a regular basis you don't actually have to do it yourself um more about
[01:20:08] have to do it yourself um more about that on Thursday okay see you everyone

Stanford CS224N: NLP with Deep Learning | Spring 2024 | Lecture 2 - Word Vectors and Language Models

Source: https://www.youtube.com/watch?v=nBor4jfWetQ

---

Transcript

[00:00:05] okay I should try and get
[00:00:08] okay I should try and get started okay so what we're going to do
[00:00:11] started okay so what we're going to do today is we're going to try and um do
[00:00:14] today is we're going to try and um do everything else that you need to know
[00:00:17] everything else that you need to know about word vectors and start to learn a
[00:00:20] about word vectors and start to learn a teeny bit about new Nets and then we'll
[00:00:22] teeny bit about new Nets and then we'll kind of get much further into sort of
[00:00:24] kind of get much further into sort of doing more with the math of new or Nets
[00:00:27] doing more with the math of new or Nets next week so this is the general plan um
[00:00:30] next week so this is the general plan um so I'm going to sort of uh sort of
[00:00:33] so I'm going to sort of uh sort of finish up from where I was last time
[00:00:35] finish up from where I was last time with optimization Basics then look a
[00:00:38] with optimization Basics then look a little bit more about word to V and word
[00:00:40] little bit more about word to V and word vectors and then some of the variants of
[00:00:43] vectors and then some of the variants of word to v um and then I'm going to
[00:00:45] word to v um and then I'm going to briefly consider alternatives sort of
[00:00:48] briefly consider alternatives sort of like what can you get from just counting
[00:00:50] like what can you get from just counting words in different ways um then we're
[00:00:52] words in different ways um then we're going to go on and talk a little bit
[00:00:54] going to go on and talk a little bit about the evaluation of word vectors um
[00:00:58] about the evaluation of word vectors um the topic of word sensor that already
[00:01:00] the topic of word sensor that already came up a couple of times um last time
[00:01:03] came up a couple of times um last time when people were asking questions and
[00:01:06] when people were asking questions and then towards the end um start to
[00:01:08] then towards the end um start to introduce the idea of classification
[00:01:11] introduce the idea of classification doing new classification and um what new
[00:01:15] doing new classification and um what new networks are about which is something
[00:01:17] networks are about which is something that they'll then expand on more in the
[00:01:19] that they'll then expand on more in the second
[00:01:20] second week um before I get into that just
[00:01:23] week um before I get into that just notes on course um organization um so
[00:01:26] notes on course um organization um so remember the first assignment is already
[00:01:29] remember the first assignment is already out and it's it's due before class next
[00:01:33] out and it's it's due before class next Tuesday um so then um our python review
[00:01:37] Tuesday um so then um our python review session is going to be taught this
[00:01:39] session is going to be taught this Friday um 3:30 to 4:20 it's not going to
[00:01:42] Friday um 3:30 to 4:20 it's not going to be taught here it's going to be taught
[00:01:44] be taught here it's going to be taught in gates bo1 the gates basement um and
[00:01:47] in gates bo1 the gates basement um and encourage everyone again to um come to
[00:01:50] encourage everyone again to um come to office hours and help sessions they've
[00:01:52] office hours and help sessions they've already started they're listed on the
[00:01:54] already started they're listed on the website um we're having these sort of um
[00:01:58] website um we're having these sort of um office our help sessions in classrooms
[00:02:01] office our help sessions in classrooms with multiple Tas so um just turn up if
[00:02:05] with multiple Tas so um just turn up if you're on campus and you can be helped
[00:02:07] you're on campus and you can be helped and if you are on campus we'd like you
[00:02:09] and if you are on campus we'd like you to just turn up that we do also have a
[00:02:11] to just turn up that we do also have a zoom option for Stanford online
[00:02:16] students um finally I have office hours
[00:02:19] students um finally I have office hours which I have not yet open but I will
[00:02:21] which I have not yet open but I will open sometime tonight um they're going
[00:02:24] open sometime tonight um they're going to be on Monday afternoons now obviously
[00:02:27] to be on Monday afternoons now obviously given the number of people not everyone
[00:02:29] given the number of people not everyone can make it into my office hours and I'm
[00:02:31] can make it into my office hours and I'm going to do these by appointment so
[00:02:33] going to do these by appointment so they're by 15minute appointments on
[00:02:36] they're by 15minute appointments on kendley um but you know I'm very happy
[00:02:38] kendley um but you know I'm very happy to talk to some people um and you know
[00:02:43] to talk to some people um and you know uh I put this little note at the end
[00:02:44] uh I put this little note at the end saying don't hog the slots um some
[00:02:47] saying don't hog the slots um some people think it'd be a really good idea
[00:02:49] people think it'd be a really good idea if they really work out how to sign up
[00:02:51] if they really work out how to sign up every week for an office hour session
[00:02:54] every week for an office hour session than me with me and that's sort of a
[00:02:56] than me with me and that's sort of a little bit antisocial um so um
[00:03:00] little bit antisocial um so um think about that um okay so at the end
[00:03:03] think about that um okay so at the end of last time I did a sort of bad job of
[00:03:07] of last time I did a sort of bad job of trying to write on write on slides of
[00:03:09] trying to write on write on slides of working out the derivatives of word to
[00:03:13] working out the derivatives of word to be um and hopefully you could read it
[00:03:15] be um and hopefully you could read it much more clearly in the the version
[00:03:17] much more clearly in the the version that appears on the website where I was
[00:03:19] that appears on the website where I was doing it at home more carefully um so
[00:03:23] doing it at home more carefully um so that was saying that we had this lock
[00:03:25] that was saying that we had this lock function and our job was to work out its
[00:03:28] function and our job was to work out its derivatives which would tell us which
[00:03:31] derivatives which would tell us which direction to go to walk downhill um and
[00:03:35] direction to go to walk downhill um and so I didn't really quite finish the loop
[00:03:37] so I didn't really quite finish the loop here so you know we have some cost
[00:03:40] here so you know we have some cost function that we want to minimize and
[00:03:43] function that we want to minimize and then we work out the gradients of that
[00:03:45] then we work out the gradients of that the gradient of that function to work
[00:03:47] the gradient of that function to work out which direction is downhill and then
[00:03:50] out which direction is downhill and then the simplest algorithm is then that we
[00:03:54] the simplest algorithm is then that we that we work out the direction downhill
[00:03:58] that we work out the direction downhill we walk a little bit in that direction
[00:03:59] we walk a little bit in that direction ction and then we repeat we work out the
[00:04:03] ction and then we repeat we work out the gradient at this point we walk downhill
[00:04:05] gradient at this point we walk downhill a little bit and we keep on going and
[00:04:08] a little bit and we keep on going and we'll get to the minimum and with a sort
[00:04:11] we'll get to the minimum and with a sort of a one-dimensional function like this
[00:04:13] of a one-dimensional function like this it's sort of very simple we're just
[00:04:14] it's sort of very simple we're just walking down hill but when we have a
[00:04:17] walking down hill but when we have a function of many many dimensions when we
[00:04:20] function of many many dimensions when we calculate the gradient at different
[00:04:21] calculate the gradient at different points we might be starting to um walk
[00:04:24] points we might be starting to um walk in different directions and so that's
[00:04:27] in different directions and so that's why we need to do calculus and have
[00:04:28] why we need to do calculus and have gradients and so this gives us the basic
[00:04:31] gradients and so this gives us the basic algorithm of gradient
[00:04:34] algorithm of gradient descent um and so under the gradient
[00:04:37] descent um and so under the gradient descent algorithm um what we're doing is
[00:04:41] descent algorithm um what we're doing is that we've got um our loss function J
[00:04:46] that we've got um our loss function J we're working out its gradient um and
[00:04:49] we're working out its gradient um and then we're taking a little bit of a
[00:04:52] then we're taking a little bit of a little multipli of the gradient so that
[00:04:54] little multipli of the gradient so that Alpha is our step size or learning rate
[00:04:57] Alpha is our step size or learning rate um and that's normally Alpha is a very
[00:04:59] um and that's normally Alpha is a very small number number something like 10- 3
[00:05:01] small number number something like 10- 3 or 10-4 or maybe even 10- 5 so we're
[00:05:05] or 10-4 or maybe even 10- 5 so we're taking a really little bit of the
[00:05:07] taking a really little bit of the gradient and then we're subtracting it
[00:05:10] gradient and then we're subtracting it from our parameters to get new
[00:05:13] from our parameters to get new parameters and as we do that we will
[00:05:16] parameters and as we do that we will walk downhill and the reason why we want
[00:05:19] walk downhill and the reason why we want to have a small learning rate is we
[00:05:21] to have a small learning rate is we don't want to walk too fast so if from
[00:05:23] don't want to walk too fast so if from here we worked out the gradient and said
[00:05:25] here we worked out the gradient and said it's in this direction and we just kept
[00:05:28] it's in this direction and we just kept on walking we sort of might end up way
[00:05:31] on walking we sort of might end up way over here or if we had a really big step
[00:05:33] over here or if we had a really big step size we might even end up at a worse
[00:05:35] size we might even end up at a worse Point than we started with so we want to
[00:05:37] Point than we started with so we want to take little steps to walk downhill and
[00:05:41] take little steps to walk downhill and so that's the very basic gradient
[00:05:42] so that's the very basic gradient descent algorithm now the very basic
[00:05:46] descent algorithm now the very basic gradient descent algorithm we never use
[00:05:50] gradient descent algorithm we never use what we actually use is the next thing
[00:05:53] what we actually use is the next thing up which is called stochastic gradient
[00:05:55] up which is called stochastic gradient descent so the problem is for the basic
[00:05:59] descent so the problem is for the basic gradient Cent algorithm we've worked out
[00:06:03] gradient Cent algorithm we've worked out um for an entire set of data um what the
[00:06:09] um for an entire set of data um what the objective function is and what the um
[00:06:12] objective function is and what the um slope at the point of evaluation is and
[00:06:17] slope at the point of evaluation is and in general um we've got a lot of data in
[00:06:21] in general um we've got a lot of data in which we're Computing models so simply
[00:06:24] which we're Computing models so simply trying to calculate our objective
[00:06:26] trying to calculate our objective function over all of our data for our
[00:06:29] function over all of our data for our model the training data for the model
[00:06:32] model the training data for the model would take us a very very long time um
[00:06:35] would take us a very very long time um and so that's very um very expensive to
[00:06:38] and so that's very um very expensive to compute and so we'd wait a very long
[00:06:40] compute and so we'd wait a very long time before we make even a single step
[00:06:43] time before we make even a single step of gradient update um so for neural Nets
[00:06:46] of gradient update um so for neural Nets what you're always doing is using this
[00:06:48] what you're always doing is using this variant that's called stochastic
[00:06:49] variant that's called stochastic gradient descent and so for stochastic
[00:06:52] gradient descent and so for stochastic gradient descent what that means is we
[00:06:54] gradient descent what that means is we pick a very small subset of our data
[00:06:57] pick a very small subset of our data like maybe we pick 16 or 2 Data items
[00:07:01] like maybe we pick 16 or 2 Data items and we pretend that's all of our data
[00:07:04] and we pretend that's all of our data and we evaluate the function J based on
[00:07:07] and we evaluate the function J based on that small subset and work out the
[00:07:09] that small subset and work out the gradient based on that small subset so
[00:07:12] gradient based on that small subset so it's a noisy inaccurate estimate of the
[00:07:14] it's a noisy inaccurate estimate of the gradient and we use that um to be the
[00:07:18] gradient and we use that um to be the direction in which we walk um so that's
[00:07:21] direction in which we walk um so that's normally referred to also as having mini
[00:07:24] normally referred to also as having mini batches or mini batch gradient
[00:07:27] batches or mini batch gradient descent um and in theory working out the
[00:07:32] descent um and in theory working out the gradient based on this small subset um
[00:07:36] gradient based on this small subset um is an approximation but one of the
[00:07:38] is an approximation but one of the interesting things in the way things
[00:07:41] interesting things in the way things have emerged in new network land is it
[00:07:43] have emerged in new network land is it turns out that new networks actually
[00:07:45] turns out that new networks actually often work better when you throw some
[00:07:48] often work better when you throw some noise into the system that having this
[00:07:50] noise into the system that having this noise in the system gives you jiggle and
[00:07:53] noise in the system gives you jiggle and moves things around and so actually
[00:07:57] moves things around and so actually stochastic gradient descent not only is
[00:07:59] stochastic gradient descent not only is way way faster but actually works better
[00:08:02] way way faster but actually works better as a system for optimization of neural
[00:08:06] as a system for optimization of neural networks okay so if you remember from
[00:08:09] networks okay so if you remember from last time um for um word de the idea was
[00:08:13] last time um for um word de the idea was we started by just saying each word has
[00:08:16] we started by just saying each word has a random um Vector representing it so we
[00:08:20] a random um Vector representing it so we will literally sort of just get random
[00:08:23] will literally sort of just get random small numbers and fill up the vectors
[00:08:25] small numbers and fill up the vectors with those random small numbers um
[00:08:27] with those random small numbers um there's an important point there which
[00:08:29] there's an important point there which is you do have to initialize your
[00:08:32] is you do have to initialize your vectors with random small numbers if you
[00:08:34] vectors with random small numbers if you just leave the all the vectors as zero
[00:08:37] just leave the all the vectors as zero then nothing works um and that's because
[00:08:41] then nothing works um and that's because if everything starts off the same you
[00:08:44] if everything starts off the same you get these sort of false symmetries which
[00:08:46] get these sort of false symmetries which means that you can't learn so you always
[00:08:49] means that you can't learn so you always do want to be initializing your vectors
[00:08:52] do want to be initializing your vectors with random numbers and then we're going
[00:08:54] with random numbers and then we're going to go through each position in the
[00:08:56] to go through each position in the Corpus using our estimates we're going
[00:08:59] Corpus using our estimates we're going to try try and predict the probability
[00:09:01] to try try and predict the probability of words in the context as we talked
[00:09:03] of words in the context as we talked about last time then we so that gives us
[00:09:06] about last time then we so that gives us an objective function from which we can
[00:09:09] an objective function from which we can then look at our errors look at our
[00:09:12] then look at our errors look at our gradient um and update the vectors so
[00:09:15] gradient um and update the vectors so that they learn to predict surrounding
[00:09:17] that they learn to predict surrounding words better and so the incredible thing
[00:09:20] words better and so the incredible thing is that we can do no more than that and
[00:09:24] is that we can do no more than that and we end up learning word vectors which
[00:09:27] we end up learning word vectors which actually capture quite a lot of the
[00:09:30] actually capture quite a lot of the semantics the meaning and relationships
[00:09:32] semantics the meaning and relationships between different words so you know when
[00:09:35] between different words so you know when this was first
[00:09:38] this was first um discovered um for these algorithms I
[00:09:41] um discovered um for these algorithms I mean it really feels like magic that you
[00:09:43] mean it really feels like magic that you can just sort of do this math simple
[00:09:47] can just sort of do this math simple math over a lot of text um and actually
[00:09:51] math over a lot of text um and actually learn about the meanings of words that
[00:09:53] learn about the meanings of words that it's sort of just sort of surprising
[00:09:55] it's sort of just sort of surprising that something so simple could work but
[00:09:58] that something so simple could work but as time has gone on this same recipe has
[00:10:01] as time has gone on this same recipe has then been for all kinds of learning
[00:10:03] then been for all kinds of learning about
[00:10:04] about um um the behavior of language from new
[00:10:07] um um the behavior of language from new networks um so let's just um go through
[00:10:12] networks um so let's just um go through a sense of how that is but before we do
[00:10:14] a sense of how that is but before we do that so
[00:10:16] that so um let me just mention so for our word
[00:10:19] um let me just mention so for our word toac algorithms the only parameters of
[00:10:22] toac algorithms the only parameters of the model are these word vectors they're
[00:10:25] the model are these word vectors they're the outside word vectors and the center
[00:10:27] the outside word vectors and the center word vectors which we actually treat as
[00:10:30] word vectors which we actually treat as disjoint as I mentioned last time and
[00:10:33] disjoint as I mentioned last time and when we do the computations we're
[00:10:35] when we do the computations we're considering the dot product between the
[00:10:38] considering the dot product between the various um possible outside words with
[00:10:43] various um possible outside words with our Center word and we using those to
[00:10:46] our Center word and we using those to get a probability distribution over How
[00:10:48] get a probability distribution over How likely the model thinks that different
[00:10:51] likely the model thinks that different outside words were and then we're
[00:10:53] outside words were and then we're comparing that to the actual outside
[00:10:56] comparing that to the actual outside word in the context and that gives us
[00:10:58] word in the context and that gives us our source of error so as such this is
[00:11:01] our source of error so as such this is what's referred to in NLP as a bag of
[00:11:04] what's referred to in NLP as a bag of words model that it doesn't actually
[00:11:06] words model that it doesn't actually know about the structure of sentences
[00:11:08] know about the structure of sentences and or even what's to the left and
[00:11:10] and or even what's to the left and what's to the right it's predicting
[00:11:12] what's to the right it's predicting exactly the same probabilities at each
[00:11:15] exactly the same probabilities at each position to the left or right um but
[00:11:17] position to the left or right um but it's wanting to know about what kind of
[00:11:19] it's wanting to know about what kind of words appear in the context of the
[00:11:21] words appear in the context of the center word um so I just wanted to uh
[00:11:26] center word um so I just wanted to uh stop this for a minute and um
[00:11:30] stop this for a minute and um let's see not that one
[00:11:38] um so give you some kind of a sense that
[00:11:42] um so give you some kind of a sense that this really um does work um so this is a
[00:11:45] this really um does work um so this is a little Jupiter notebook um that I've got
[00:11:48] little Jupiter notebook um that I've got um for this
[00:11:51] um for this um okay and so this is using and here
[00:11:54] um okay and so this is using and here I'm using a package um gen Sim which we
[00:11:57] I'm using a package um gen Sim which we don't continue to use after that really
[00:12:02] don't continue to use after that really um but it's sort of one package that let
[00:12:04] um but it's sort of one package that let you load and play with word vectors and
[00:12:08] you load and play with word vectors and the word vectors I'm going to use here
[00:12:10] the word vectors I'm going to use here are are glove word vectors and actually
[00:12:13] are are glove word vectors and actually I'm going to um glove was a model we
[00:12:16] I'm going to um glove was a model we built at Stanford and I'm going to
[00:12:18] built at Stanford and I'm going to actually talk about it a little bit
[00:12:19] actually talk about it a little bit later um so strictly speaking um these
[00:12:23] later um so strictly speaking um these aren't exactly word to ve word vectors
[00:12:25] aren't exactly word to ve word vectors but they behave in exactly the same way
[00:12:28] but they behave in exactly the same way um and so okay so now it's loaded up by
[00:12:32] um and so okay so now it's loaded up by word vectors because the word vectors
[00:12:34] word vectors because the word vectors are a big data file and so as we've
[00:12:37] are a big data file and so as we've discussed um for a word right that the
[00:12:41] discussed um for a word right that the representation of any word here is bread
[00:12:44] representation of any word here is bread is just um a vector of real numbers
[00:12:48] is just um a vector of real numbers right so um I'm using a 100 dimensional
[00:12:51] right so um I'm using a 100 dimensional word vectors to keep things quicker um
[00:12:54] word vectors to keep things quicker um for my class demo so this is the word
[00:12:57] for my class demo so this is the word bread um and then I can say well what's
[00:13:00] bread um and then I can say well what's the representation for
[00:13:02] the representation for croissant
[00:13:04] croissant um and um this is croissant and we can
[00:13:09] um and um this is croissant and we can sort of get a visual sense of oh they're
[00:13:11] sort of get a visual sense of oh they're at least a little bit similar right so
[00:13:13] at least a little bit similar right so the first components are both negative
[00:13:16] the first components are both negative the second components are both positive
[00:13:18] the second components are both positive the third components are both negative
[00:13:21] the third components are both negative and large the fourth components are both
[00:13:23] and large the fourth components are both positive right they they seem like
[00:13:26] positive right they they seem like they're kind of similar vectors so that
[00:13:28] they're kind of similar vectors so that seems kind of hopeful because that means
[00:13:31] seems kind of hopeful because that means that it knows that bread and croissant
[00:13:33] that it knows that bread and croissant are a bit um similar to each other um
[00:13:37] are a bit um similar to each other um the this package has a nice simple
[00:13:39] the this package has a nice simple function where rather than doing that by
[00:13:41] function where rather than doing that by hand you can just ask it about all the
[00:13:44] hand you can just ask it about all the word vectors and say which ones are most
[00:13:47] word vectors and say which ones are most similar so I can ask it um what um words
[00:13:51] similar so I can ask it um what um words in its vocabulary most similar to USA
[00:13:54] in its vocabulary most similar to USA and in this model everything's been
[00:13:56] and in this model everything's been lowercased I should mention um and so if
[00:13:59] lowercased I should mention um and so if I do that it has Canada America U.S.A
[00:14:03] I do that it has Canada America U.S.A then United States Australia well those
[00:14:06] then United States Australia well those seem a fairly reasonable list of most
[00:14:08] seem a fairly reasonable list of most similar words though you might think
[00:14:10] similar words though you might think it's a little strange that Canada wins
[00:14:12] it's a little strange that Canada wins out over the USA with dots over it
[00:14:16] out over the USA with dots over it um um similarly I can ask if what's most
[00:14:19] um um similarly I can ask if what's most similar to banana and I get coconut
[00:14:21] similar to banana and I get coconut mango bananas potato pineapple fruit Etc
[00:14:25] mango bananas potato pineapple fruit Etc again pretty sensible you know a little
[00:14:27] again pretty sensible you know a little bit of a bias to more tropical fruits or
[00:14:30] bit of a bias to more tropical fruits or I can go to cron and ask what's most
[00:14:33] I can go to cron and ask what's most similar to cran the most similar things
[00:14:35] similar to cran the most similar things to cron isn't bread but it's things like
[00:14:37] to cron isn't bread but it's things like Brios bagette fatcha um which sort of
[00:14:40] Brios bagette fatcha um which sort of basically makes sense though here's
[00:14:42] basically makes sense though here's pudding here um and I've got wait I'd
[00:14:46] pudding here um and I've got wait I'd already done oh sorry yeah I remember
[00:14:49] already done oh sorry yeah I remember what this is right so with this most
[00:14:52] what this is right so with this most similar you've got a positive word
[00:14:54] similar you've got a positive word vector and you're saying what other
[00:14:56] vector and you're saying what other words are most similar in position to
[00:14:58] words are most similar in position to that that um there's something else you
[00:15:01] that that um there's something else you can do which you can say is this is let
[00:15:04] can do which you can say is this is let me take the negative of that word vector
[00:15:06] me take the negative of that word vector and say what's most similar to the
[00:15:09] and say what's most similar to the negative of it and you could possibly
[00:15:12] negative of it and you could possibly think a that might be useful to find
[00:15:14] think a that might be useful to find antonyms or something like that I mean
[00:15:16] antonyms or something like that I mean the truth is it isn't if you ask for the
[00:15:20] the truth is it isn't if you ask for the things that are most similar to the
[00:15:21] things that are most similar to the negative of the banana Vector um and in
[00:15:24] negative of the banana Vector um and in most other vectors it's the same you get
[00:15:27] most other vectors it's the same you get off out these weirdo things things that
[00:15:29] off out these weirdo things things that you're not really sure if they're words
[00:15:30] you're not really sure if they're words at all or maybe they are in some other
[00:15:33] at all or maybe they are in some other language or some of them are names right
[00:15:35] language or some of them are names right like shichi is a Japanese name um but
[00:15:38] like shichi is a Japanese name um but not very useful stuff don't that um they
[00:15:41] not very useful stuff don't that um they not don't really feel like a negative of
[00:15:43] not don't really feel like a negative of banana but it turns out that from there
[00:15:47] banana but it turns out that from there we get this um powerful ability of um
[00:15:51] we get this um powerful ability of um that was observed for word to ve which
[00:15:54] that was observed for word to ve which is that we could isolate semantic
[00:15:57] is that we could isolate semantic components um and then put them together
[00:16:00] components um and then put them together in interesting ways so um looking at
[00:16:04] in interesting ways so um looking at this picture what we could do is start
[00:16:06] this picture what we could do is start with a positive Vector for King from the
[00:16:10] with a positive Vector for King from the origin up the king then we could use the
[00:16:13] origin up the king then we could use the negation to say subtract out the vector
[00:16:16] negation to say subtract out the vector for man and then we could have another
[00:16:19] for man and then we could have another positive Vector of add on the um Vector
[00:16:22] positive Vector of add on the um Vector for woman and then we can ask the model
[00:16:25] for woman and then we can ask the model is if you're over here in the space um
[00:16:28] is if you're over here in the space um what is the nearest word to you over
[00:16:31] what is the nearest word to you over there and so um that's what this um next
[00:16:35] there and so um that's what this um next thing does right it's sort of says
[00:16:38] thing does right it's sort of says positive Vector for King negative for
[00:16:40] positive Vector for King negative for man also positive for Queen where does
[00:16:43] man also positive for Queen where does that get you to and that gets you to
[00:16:46] that get you to and that gets you to Queen yay um and so this was the most
[00:16:50] Queen yay um and so this was the most celebrated property that was discovered
[00:16:53] celebrated property that was discovered with these word vectors that they
[00:16:55] with these word vectors that they weren't only good for meaning similarity
[00:16:58] weren't only good for meaning similarity but that they were good um for doing
[00:17:02] but that they were good um for doing these kind of meaning components and
[00:17:05] these kind of meaning components and these got referred to as analogies
[00:17:07] these got referred to as analogies because that's you can think of them as
[00:17:09] because that's you can think of them as a is to B as C is to what so it's sort
[00:17:12] a is to B as C is to what so it's sort of like um woman is to King no sorry um
[00:17:16] of like um woman is to King no sorry um man is to King or King is to man as um
[00:17:21] man is to King or King is to man as um I'm saying this the wrong way around man
[00:17:23] I'm saying this the wrong way around man is to King as woman is to what um in the
[00:17:26] is to King as woman is to what um in the analogies and so here I've defined a
[00:17:29] analogies and so here I've defined a little function that is now saying this
[00:17:32] little function that is now saying this little function just automates that and
[00:17:34] little function just automates that and we'll compute
[00:17:35] we'll compute analogies um and so now I can ask it in
[00:17:39] analogies um and so now I can ask it in just this analogy format um man is to
[00:17:42] just this analogy format um man is to King as woman is to queen and um that
[00:17:46] King as woman is to queen and um that one was sort of the canonical example
[00:17:49] one was sort of the canonical example but you know you can actually sort of
[00:17:51] but you know you can actually sort of has have fun with this and I mean you
[00:17:55] has have fun with this and I mean you know uh this is pretty oldfashioned
[00:17:58] know uh this is pretty oldfashioned stuff you know know I I feel like I'm
[00:18:01] stuff you know know I I feel like I'm maybe like now at this point an old guy
[00:18:04] maybe like now at this point an old guy talking about how much fun we used to
[00:18:06] talking about how much fun we used to have sitting around the radio listening
[00:18:08] have sitting around the radio listening to radio plays um because you know
[00:18:11] to radio plays um because you know basically no one uses this stuff anymore
[00:18:13] basically no one uses this stuff anymore and there are much much better and
[00:18:15] and there are much much better and fancier things like chat GPT but you
[00:18:18] fancier things like chat GPT but you know back in the day when I was younger
[00:18:21] know back in the day when I was younger um you know it was really stunning
[00:18:24] um you know it was really stunning already just how this very simple model
[00:18:27] already just how this very simple model built on very simple data um could just
[00:18:32] built on very simple data um could just have quite good semantic understanding
[00:18:35] have quite good semantic understanding and do quite good analogies so you can
[00:18:37] and do quite good analogies so you can actually you know play with this quite a
[00:18:39] actually you know play with this quite a bit and have a bit of fun so you can do
[00:18:42] bit and have a bit of fun so you can do something like analogy Australia comma
[00:18:48] something like analogy Australia comma be
[00:18:50] be France okay what people think the answer
[00:18:53] France okay what people think the answer will
[00:18:54] will be close the answer gives us Champagne
[00:18:58] be close the answer gives us Champagne but that seems a pretty good answer um I
[00:19:01] but that seems a pretty good answer um I could then put in Russia what what
[00:19:04] could then put in Russia what what people think vodka yeah it'll see you
[00:19:07] people think vodka yeah it'll see you can get back vodka you know this is
[00:19:10] can get back vodka you know this is actually works kind of
[00:19:12] actually works kind of interestingly um you know I could do a
[00:19:15] interestingly um you know I could do a different one I can do the test
[00:19:17] different one I can do the test something different I can do something
[00:19:19] something different I can do something like pencil is
[00:19:22] like pencil is to
[00:19:24] to sketching as
[00:19:27] sketching as camera is
[00:19:31] to photographing yeah that works quite
[00:19:34] to photographing yeah that works quite well um so um we built this model in
[00:19:39] well um so um we built this model in 2014 so it's a little bit out of date in
[00:19:43] 2014 so it's a little bit out of date in politics so you know well we we can't do
[00:19:46] politics so you know well we we can't do the last decade of politics which is
[00:19:48] the last decade of politics which is maybe unfortunate but you know we could
[00:19:50] maybe unfortunate but you know we could try out older politics questions so we
[00:19:53] try out older politics questions so we could try um
[00:19:57] could try um Obama is to
[00:20:02] Clinton
[00:20:04] Clinton as
[00:20:05] as Reagan is to if you remember your Us
[00:20:08] Reagan is to if you remember your Us world history class any guesses what
[00:20:11] world history class any guesses what it's going to
[00:20:13] it's going to say there's a bush one any other
[00:20:17] say there's a bush one any other ideas some people have different
[00:20:19] ideas some people have different opinions of Bill Clinton
[00:20:22] opinions of Bill Clinton any uh um what it answers is Nixon which
[00:20:26] any uh um what it answers is Nixon which I think is actually kind of fair
[00:20:29] I think is actually kind of fair um
[00:20:31] um but um you can also get it to do some
[00:20:36] but um you can also get it to do some just sort of language syntactic facts so
[00:20:39] just sort of language syntactic facts so you can do something like tallest to
[00:20:43] you can do something like tallest to tallest as long
[00:20:47] tallest as long oops as long is to this one's
[00:20:53] oops as long is to this one's easy um yeah so you know with this simp
[00:20:58] easy um yeah so you know with this simp simple method of learning with this
[00:21:00] simple method of learning with this simple bag of words model it's enough to
[00:21:03] simple bag of words model it's enough to learn a lot about the semantics of words
[00:21:08] learn a lot about the semantics of words and you know stuff that's beyond
[00:21:10] and you know stuff that's beyond conventional semantics right you know
[00:21:12] conventional semantics right you know like our examples with Australia De Beer
[00:21:15] like our examples with Australia De Beer as Russia as to vodka I mean that sort
[00:21:17] as Russia as to vodka I mean that sort of cultural World Knowledge which goes a
[00:21:20] of cultural World Knowledge which goes a little bit beyond um what people
[00:21:22] little bit beyond um what people normally think of as sort of word
[00:21:23] normally think of as sort of word meaning semantics but it's also in there
[00:21:26] meaning semantics but it's also in there yes if you perhaps subract the distance
[00:21:29] yes if you perhaps subract the distance from let's say like man and King does
[00:21:31] from let's say like man and King does that also capture a concept of
[00:21:33] that also capture a concept of relationship between two words like
[00:21:34] relationship between two words like would that give you back like ruler like
[00:21:36] would that give you back like ruler like something like that we taking the like
[00:21:39] something like that we taking the like the difference between two vectors does
[00:21:40] the difference between two vectors does capture
[00:21:41] capture some
[00:21:43] some right the distance between man so so Man
[00:21:49] right the distance between man so so Man compared to King should be a ruler
[00:21:51] compared to King should be a ruler concept but isn't that what I'm using CU
[00:21:53] concept but isn't that what I'm using CU then I'm taking that I'm taking the
[00:21:57] then I'm taking that I'm taking the distance between man and
[00:22:00] distance between man and King is what I'm adding on to woman to
[00:22:03] King is what I'm adding on to woman to get the queen
[00:22:05] get the queen right right yeah so I can depending on
[00:22:09] right right yeah so I can depending on if you think of these words depending on
[00:22:11] if you think of these words depending on which thing you think of as the analogy
[00:22:14] which thing you think of as the analogy you can think of it you've both got um a
[00:22:19] you can think of it you've both got um a a vector a difference Vector between
[00:22:21] a vector a difference Vector between words that gives you a gender analogy
[00:22:25] words that gives you a gender analogy and one that gives you a ruler analogy
[00:22:27] and one that gives you a ruler analogy yeah absolutely
[00:22:30] yeah absolutely any other
[00:22:32] any other questions yeah um in the watch ve umm we
[00:22:37] questions yeah um in the watch ve umm we get two vectors like for each one a u
[00:22:40] get two vectors like for each one a u and a v but here you only have one
[00:22:42] and a v but here you only have one vector so how do you go from two to one
[00:22:46] vector so how do you go from two to one yeah good question I mean the commonest
[00:22:49] yeah good question I mean the commonest way in practice was you just average the
[00:22:51] way in practice was you just average the two of them and and and really you sort
[00:22:54] two of them and and and really you sort of find out that they um end up very
[00:22:57] of find out that they um end up very close you
[00:22:59] close you know um because if you think of it since
[00:23:02] know um because if you think of it since you're going along every position of the
[00:23:04] you're going along every position of the text you're both going to be the case
[00:23:06] text you're both going to be the case where if the text is sort of you know
[00:23:09] where if the text is sort of you know the octopus has legs you know you're
[00:23:11] the octopus has legs you know you're going to have octopus in the center with
[00:23:13] going to have octopus in the center with legs in the context and a couple of time
[00:23:15] legs in the context and a couple of time steps later it's going to be legs in the
[00:23:16] steps later it's going to be legs in the center with octavus in the context so
[00:23:19] center with octavus in the context so you know although they vary a bit for
[00:23:21] you know although they vary a bit for all the regions of the neuron Nets very
[00:23:23] all the regions of the neuron Nets very basically they end up very similar and
[00:23:25] basically they end up very similar and people normally just average them yeah
[00:23:28] people normally just average them yeah can you
[00:23:29] can you this process so use this uh the answer
[00:23:32] this process so use this uh the answer of one to then be placed into the
[00:23:34] of one to then be placed into the analogy function of the another and see
[00:23:36] analogy function of the another and see how far away you can go before it SS
[00:23:38] how far away you can go before it SS break
[00:23:40] break down I think you can um so you what wait
[00:23:44] down I think you can um so you what wait you're wanting to how far away how
[00:23:46] you're wanting to how far away how distant of a relation between two words
[00:23:48] distant of a relation between two words can you do before it starts providing
[00:23:51] can you do before it starts providing incorrect relationship
[00:23:55] between but are you wanting to sort of
[00:23:57] between but are you wanting to sort of make two steps from somewhere or yeah
[00:24:03] make two steps from somewhere or yeah St many St go away
[00:24:09] froming so it doesn't always work I mean
[00:24:12] froming so it doesn't always work I mean there are so examples that are fail um
[00:24:15] there are so examples that are fail um I'm sort of shy to try that now because
[00:24:17] I'm sort of shy to try that now because I don't have a predefined function that
[00:24:19] I don't have a predefined function that did it and that might take me um too
[00:24:21] did it and that might take me um too long but you could play with it at home
[00:24:24] long but you could play with it at home and see how it works for you
[00:24:29] curious is a clarification why is it
[00:24:31] curious is a clarification why is it that we use two separate sets of vectors
[00:24:33] that we use two separate sets of vectors for word uh is it just to get more
[00:24:35] for word uh is it just to get more parameters or is there um I'll get back
[00:24:38] parameters or is there um I'll get back to that maybe I should go on at this
[00:24:41] to that maybe I should go on at this point um let me move on and kind of just
[00:24:44] point um let me move on and kind of just get through some more details of the
[00:24:47] get through some more details of the word Tove algorithm um so um just a
[00:24:52] word Tove algorithm um so um just a technical point on this class so you
[00:24:54] technical point on this class so you don't make any big mistakes and waste
[00:24:56] don't make any big mistakes and waste your weekend I mean for most instances
[00:24:59] your weekend I mean for most instances of 224n we've actually had people
[00:25:03] of 224n we've actually had people Implement from scratch word to VC as
[00:25:05] Implement from scratch word to VC as assignment to um but you know for this
[00:25:09] assignment to um but you know for this quarter doing it in Spring quarter as
[00:25:12] quarter doing it in Spring quarter as you probably know Spring quarter is
[00:25:13] you probably know Spring quarter is actually a little shorter than the other
[00:25:15] actually a little shorter than the other two quarters um we decided to skip
[00:25:18] two quarters um we decided to skip having people Implement word to VC um so
[00:25:21] having people Implement word to VC um so don't look at the old assignment to that
[00:25:23] don't look at the old assignment to that says Implement word to V or else you'll
[00:25:25] says Implement word to V or else you'll be misspending your time wait for the
[00:25:27] be misspending your time wait for the newer assignment to to come out um but
[00:25:31] newer assignment to to come out um but you know despite that let me just sort
[00:25:32] you know despite that let me just sort of um say a little bit more about some
[00:25:36] of um say a little bit more about some of the details right yeah so why two
[00:25:38] of the details right yeah so why two vectors um so the two vectors is it just
[00:25:43] vectors um so the two vectors is it just makes the math a little bit easy so if
[00:25:46] makes the math a little bit easy so if you think about the math right if you
[00:25:49] you think about the math right if you have the same vectors for the center
[00:25:52] have the same vectors for the center word and for the outside words well for
[00:25:55] word and for the outside words well for whatever the the center word is let's
[00:25:58] whatever the the center word is let's say it's octopus that when you're going
[00:26:00] say it's octopus that when you're going through the trying out every possible
[00:26:04] through the trying out every possible context word for the normalization at
[00:26:07] context word for the normalization at some point you'll hit octopus again and
[00:26:10] some point you'll hit octopus again and so at that point you'll have a quadratic
[00:26:12] so at that point you'll have a quadratic term right you'll have the x squared of
[00:26:14] term right you'll have the x squared of the octopus vector and that kind of
[00:26:17] the octopus vector and that kind of messes up I mean you're clever people
[00:26:20] messes up I mean you're clever people you could work out the math of it but it
[00:26:22] you could work out the math of it but it makes the math more of a mess right CU
[00:26:25] makes the math more of a mess right CU every other term it's something
[00:26:26] every other term it's something different and it's just like a X and
[00:26:29] different and it's just like a X and then at one position you've got an x s
[00:26:31] then at one position you've got an x s um so it just makes the math Messier and
[00:26:34] um so it just makes the math Messier and so they kept it really simple by just
[00:26:37] so they kept it really simple by just having them be disjoint vectors but it
[00:26:40] having them be disjoint vectors but it it doesn't make it better I mean it
[00:26:42] it doesn't make it better I mean it actually turns out it works a fraction
[00:26:44] actually turns out it works a fraction better if you do it right um but in
[00:26:48] better if you do it right um but in practice people have usually just
[00:26:50] practice people have usually just estimated them separately and then
[00:26:53] estimated them separately and then average them at the end um there if if
[00:26:55] average them at the end um there if if you actually look at the paper here's m
[00:26:58] you actually look at the paper here's m of it out you can find it 2013 paper um
[00:27:02] of it out you can find it 2013 paper um there's actually sort of a family of
[00:27:04] there's actually sort of a family of methods that they describe um so they
[00:27:07] methods that they describe um so they described two methods one of which um
[00:27:11] described two methods one of which um was that you have an inside word that's
[00:27:13] was that you have an inside word that's predicting the words around it and then
[00:27:16] predicting the words around it and then the other one tried to predict the
[00:27:18] the other one tried to predict the center word from all the words in the
[00:27:20] center word from all the words in the context which was called continuous bag
[00:27:22] context which was called continuous bag of words in their paper um the one that
[00:27:24] of words in their paper um the one that I've described is Skip gram which is
[00:27:27] I've described is Skip gram which is simpler and works just great um but then
[00:27:31] simpler and works just great um but then the other part of it is um for working
[00:27:35] the other part of it is um for working out um what loss function to be used for
[00:27:38] out um what loss function to be used for training and what I've um presented so
[00:27:42] training and what I've um presented so far um is naive soft Max where we just
[00:27:46] far um is naive soft Max where we just consider every possible choice of a
[00:27:49] consider every possible choice of a context word and just run all the math
[00:27:52] context word and just run all the math um you know that's totally doable and
[00:27:55] um you know that's totally doable and with our modern super fast computers
[00:27:57] with our modern super fast computers it's not even that unreasonable to do we
[00:27:59] it's not even that unreasonable to do we do things like this all the time but at
[00:28:01] do things like this all the time but at least at the time that they um wrote
[00:28:04] least at the time that they um wrote their paper this seemed kind of
[00:28:06] their paper this seemed kind of expensive and they considered other
[00:28:08] expensive and they considered other Alternatives um like a hierarchical
[00:28:10] Alternatives um like a hierarchical softmax which I'm not going to explain
[00:28:12] softmax which I'm not going to explain right now but I do just want to explain
[00:28:15] right now but I do just want to explain negative
[00:28:17] negative sampling okay so this is just to sort of
[00:28:19] sampling okay so this is just to sort of see a bit of a different way of doing
[00:28:21] see a bit of a different way of doing things so for what we did last time we
[00:28:25] things so for what we did last time we had this sort of straightforward soft
[00:28:28] had this sort of straightforward soft Max equation and so in the denominator
[00:28:31] Max equation and so in the denominator you're summing over every word in the
[00:28:34] you're summing over every word in the vocabulary um and so if you might have
[00:28:36] vocabulary um and so if you might have 400,000 words in your vocabulary a lot
[00:28:39] 400,000 words in your vocabulary a lot of words in human languages you know
[00:28:41] of words in human languages you know that's kind of a a big sum especially
[00:28:44] that's kind of a a big sum especially when for each element of the sum you're
[00:28:46] when for each element of the sum you're taking a DOT product between 100
[00:28:49] taking a DOT product between 100 dimensional or 300 dimensional vectors
[00:28:51] dimensional or 300 dimensional vectors and then exponentiating it right a lot
[00:28:54] and then exponentiating it right a lot of math going on somewhere in there um
[00:28:57] of math going on somewhere in there um so
[00:28:59] so maybe we could short circuit that and so
[00:29:02] maybe we could short circuit that and so the idea of the negative sampling was to
[00:29:04] the idea of the negative sampling was to say well rather than evaluating it for
[00:29:07] say well rather than evaluating it for every single possible word maybe we
[00:29:11] every single possible word maybe we could just sort of train some simple
[00:29:14] could just sort of train some simple logistic regressions where they're going
[00:29:17] logistic regressions where they're going to say you should like the true word
[00:29:20] to say you should like the true word that's in the context and if we randomly
[00:29:23] that's in the context and if we randomly pick a few other words you shouldn't
[00:29:25] pick a few other words you shouldn't like them very much um and that's skip
[00:29:28] like them very much um and that's skip gram negative sampling so that's what
[00:29:30] gram negative sampling so that's what this looks like as an equation um so
[00:29:33] this looks like as an equation um so we've got our Center word and our actual
[00:29:37] we've got our Center word and our actual context word and we're um saying well
[00:29:41] context word and we're um saying well let's work out the term for the actual
[00:29:45] let's work out the term for the actual Center word we'd like this to be um high
[00:29:50] Center word we'd like this to be um high probability um so since we're minimizing
[00:29:53] probability um so since we're minimizing we're going to negate that and have it
[00:29:55] we're going to negate that and have it go down and then we're going to sample
[00:29:57] go down and then we're going to sample some other words and we'd like this to
[00:29:59] some other words and we'd like this to be the opposite um but the the other
[00:30:02] be the opposite um but the the other thing that we've changed here is now
[00:30:04] thing that we've changed here is now we're not using the soft Max anymore
[00:30:07] we're not using the soft Max anymore we're using this Sigma which stands for
[00:30:10] we're using this Sigma which stands for the logistic function which is often
[00:30:13] the logistic function which is often called the sigmoid um sigmoid just means
[00:30:16] called the sigmoid um sigmoid just means s-shaped and but you know you could
[00:30:18] s-shaped and but you know you could actually have an Infinity of s-shaped
[00:30:20] actually have an Infinity of s-shaped functions and the one that we actually
[00:30:22] functions and the one that we actually use is the logistic function so the
[00:30:25] use is the logistic function so the logistic function has this form and and
[00:30:28] logistic function has this form and and Maps um from any real number to a
[00:30:32] Maps um from any real number to a probability between zero and one um so
[00:30:36] probability between zero and one um so what we're wanting to say at that point
[00:30:38] what we're wanting to say at that point is for the real outside world we're
[00:30:43] is for the real outside world we're hoping that this dot product is large so
[00:30:46] hoping that this dot product is large so its probability is near one um and so
[00:30:49] its probability is near one um and so that will then sort of help with the
[00:30:52] that will then sort of help with the minimization and for the other words
[00:30:54] minimization and for the other words we'd like their probability to be small
[00:30:58] we'd like their probability to be small um so we what like them to appear sort
[00:31:01] um so we what like them to appear sort of over here um and that's what this is
[00:31:05] of over here um and that's what this is calculating but as written it's sort of
[00:31:07] calculating but as written it's sort of sticking the minus sign on the inside
[00:31:10] sticking the minus sign on the inside there which works because of the this is
[00:31:12] there which works because of the this is symmetric right so you're wanting to be
[00:31:15] symmetric right so you're wanting to be over here which means that if you negate
[00:31:18] over here which means that if you negate it you'll be on this side which will be
[00:31:26] large okay um
[00:31:28] large okay um and so then the final bit of this which
[00:31:30] and so then the final bit of this which is the asterisk is so we're going to
[00:31:33] is the asterisk is so we're going to pick a few words you know it might only
[00:31:35] pick a few words you know it might only be five or 10 that are our negative
[00:31:38] be five or 10 that are our negative samples um but for picking those words
[00:31:42] samples um but for picking those words what works well is not just to sort of
[00:31:46] what works well is not just to sort of pick sort of randomly uniformly from all
[00:31:49] pick sort of randomly uniformly from all the 400,000 words in our vocab what you
[00:31:53] the 400,000 words in our vocab what you basically want to do is sort of be
[00:31:55] basically want to do is sort of be paying attention to how common the words
[00:31:57] paying attention to how common the words are something like that is a really
[00:31:59] are something like that is a really common word and so we refer to that as
[00:32:02] common word and so we refer to that as the unigram distribution that means
[00:32:04] the unigram distribution that means you're s just taking individual words
[00:32:06] you're s just taking individual words independently how commonly they are so
[00:32:09] independently how commonly they are so about 10% of the time you'd be choosing
[00:32:11] about 10% of the time you'd be choosing the um but so that's sort of roughly
[00:32:15] the um but so that's sort of roughly what you want to do for sampling but
[00:32:17] what you want to do for sampling but people have found that you can actually
[00:32:19] people have found that you can actually do even a bit better than that so the
[00:32:21] do even a bit better than that so the standard thing that they presented for
[00:32:24] standard thing that they presented for word DEC is you're taking the unigram
[00:32:27] word DEC is you're taking the unigram probability of the word and raising it
[00:32:29] probability of the word and raising it to the power of
[00:32:31] to the power of 34 uh what does that end up
[00:32:35] doing question for the audience if I
[00:32:39] doing question for the audience if I take probabilities and raise them to the
[00:32:41] take probabilities and raise them to the three
[00:32:43] three quarters some less frequent words just
[00:32:46] quarters some less frequent words just become sing more correct yeah so the
[00:32:51] become sing more correct yeah so the raising to the
[00:32:52] raising to the 34s means that you're sort of somewhat
[00:32:56] 34s means that you're sort of somewhat upping the probability
[00:32:59] upping the probability of the less frequent words so you're
[00:33:02] of the less frequent words so you're sort of in between you know between
[00:33:05] sort of in between you know between having every word uniform and exactly
[00:33:07] having every word uniform and exactly using their relative frequencies in the
[00:33:10] using their relative frequencies in the text you're sort of moving a little bit
[00:33:12] text you're sort of moving a little bit in the direction of uniform and so you
[00:33:15] in the direction of uniform and so you get better results by going somewhat in
[00:33:17] get better results by going somewhat in the distance of sampling more uniformly
[00:33:20] the distance of sampling more uniformly but you don't want to go all the way
[00:33:22] but you don't want to go all the way there which should correspond to I guess
[00:33:25] there which should correspond to I guess putting um a zero in there rather than
[00:33:28] putting um a zero in there rather than three
[00:33:30] three quarters okay um
[00:33:34] quarters okay um yeah okay uh let's see I had a side here
[00:33:38] yeah okay uh let's see I had a side here but Time Rushes along so let's not
[00:33:41] but Time Rushes along so let's not bother with this side it's not that
[00:33:42] bother with this side it's not that important um okay so that's um the word
[00:33:47] important um okay so that's um the word to VC algorithm that we've seen all of
[00:33:50] to VC algorithm that we've seen all of um in its different forms um a
[00:33:55] um in its different forms um a reasonable uh wonder that you have at
[00:33:58] reasonable uh wonder that you have at this point is you
[00:34:01] this point is you know this seems a kind of a weird way of
[00:34:05] know this seems a kind of a weird way of doing what we're wanting to do right the
[00:34:07] doing what we're wanting to do right the idea is look we have this text we have
[00:34:11] idea is look we have this text we have words and we have words in the context
[00:34:14] words and we have words in the context of words it sort of seems like an
[00:34:17] of words it sort of seems like an obvious thing to do would be to say
[00:34:19] obvious thing to do would be to say let's just count some statistics we have
[00:34:22] let's just count some statistics we have words and there are other words that
[00:34:23] words and there are other words that occur in their context so let's just see
[00:34:26] occur in their context so let's just see how often the word word swim occurs next
[00:34:29] how often the word word swim occurs next to octopus and how often the word fish
[00:34:32] to octopus and how often the word fish occurs next to octopus let's get some
[00:34:34] occurs next to octopus let's get some counts um and see how often words occur
[00:34:38] counts um and see how often words occur in the context of other words and maybe
[00:34:40] in the context of other words and maybe we could use that um to calculate some
[00:34:44] we could use that um to calculate some form of word vectors um and so that's
[00:34:48] form of word vectors um and so that's something that people have already also
[00:34:50] something that people have already also considered so if we use the same kind of
[00:34:53] considered so if we use the same kind of idea of a context window we could just
[00:34:55] idea of a context window we could just make a matrix of how often words occur
[00:34:58] make a matrix of how often words occur in the context of other words and so you
[00:35:01] in the context of other words and so you know here's a baby example my Corpus is
[00:35:04] know here's a baby example my Corpus is I like deep learning I like NLP I enjoy
[00:35:07] I like deep learning I like NLP I enjoy flying um and my context window I'm
[00:35:10] flying um and my context window I'm using is just one word to the left and
[00:35:12] using is just one word to the left and the right and then I can make this kind
[00:35:15] the right and then I can make this kind of um co-occurrence count Matrix um
[00:35:19] of um co-occurrence count Matrix um where I'm putting in the counts of
[00:35:21] where I'm putting in the counts of different words in every context and you
[00:35:24] different words in every context and you know because my Corpus is so small um
[00:35:27] know because my Corpus is so small um every thing in the Matrix is a zero or
[00:35:30] every thing in the Matrix is a zero or one except for right here where I've got
[00:35:31] one except for right here where I've got the twos because I have I like twice
[00:35:34] the twos because I have I like twice right but in principle I've got a matrix
[00:35:37] right but in principle I've got a matrix of counts for all the different counts
[00:35:39] of counts for all the different counts here um so maybe you know this gives
[00:35:43] here um so maybe you know this gives this gives me a word Vector right you
[00:35:45] this gives me a word Vector right you know here's a word Vector for deep um is
[00:35:49] know here's a word Vector for deep um is this long Vector here and you know I
[00:35:51] this long Vector here and you know I could just say that is my word vector
[00:35:53] could just say that is my word vector and indeed sometimes people have done
[00:35:56] and indeed sometimes people have done that but they're kind of of ungainly
[00:35:58] that but they're kind of of ungainly word vectors because if we have 400,000
[00:36:02] word vectors because if we have 400,000 words in our vocabulary the size of this
[00:36:05] words in our vocabulary the size of this Matrix is 400,000 by
[00:36:08] Matrix is 400,000 by 400,000 which is a lot worse than our
[00:36:10] 400,000 which is a lot worse than our word ve word vectors because if we're
[00:36:13] word ve word vectors because if we're making them only 100 dimensional um
[00:36:15] making them only 100 dimensional um we've only got 400,000 by 100 which is
[00:36:19] we've only got 400,000 by 100 which is still a big number but it's a lot
[00:36:20] still a big number but it's a lot smaller than 400,000 time 400,000 so
[00:36:24] smaller than 400,000 time 400,000 so that's inconvenient so when people have
[00:36:26] that's inconvenient so when people have started with um these kind of
[00:36:29] started with um these kind of cooccurrence Matrix um the general thing
[00:36:32] cooccurrence Matrix um the general thing that people have done is to say well
[00:36:35] that people have done is to say well somehow we want to reduce the
[00:36:37] somehow we want to reduce the dimensionality of that Matrix so that we
[00:36:40] dimensionality of that Matrix so that we have a smaller Matrix to deal with um
[00:36:44] have a smaller Matrix to deal with um and so then how can we reduce the
[00:36:47] and so then how can we reduce the dimensionality of the Matrix and at this
[00:36:50] dimensionality of the Matrix and at this point if you remember your linear
[00:36:52] point if you remember your linear algebra and stuff like that you should
[00:36:54] algebra and stuff like that you should be thinking of things like PCA and
[00:36:57] be thinking of things like PCA and particular if you want it to work for
[00:36:59] particular if you want it to work for any Matrix of any shape there's the um
[00:37:02] any Matrix of any shape there's the um singular value decomposition so there
[00:37:05] singular value decomposition so there the classic singular value decomposition
[00:37:07] the classic singular value decomposition for any Matrix you can rewrite it as a
[00:37:11] for any Matrix you can rewrite it as a product of three matrices a u and a v
[00:37:15] product of three matrices a u and a v which are both
[00:37:16] which are both orthonormal um which means that um you
[00:37:19] orthonormal um which means that um you get these um
[00:37:22] get these um independent
[00:37:23] independent um vectors um they orthog to each other
[00:37:29] um vectors um they orthog to each other um and then in the middle we have the
[00:37:32] um and then in the middle we have the singular vectors which are ordered in
[00:37:35] singular vectors which are ordered in size is the most important singular
[00:37:37] size is the most important singular vector and these are sort of waiting
[00:37:39] vector and these are sort of waiting terms on the different number of uh the
[00:37:43] terms on the different number of uh the different dimensions and so this is sort
[00:37:45] different dimensions and so this is sort of the full SVD
[00:37:48] of the full SVD decomposition um but you know part of it
[00:37:51] decomposition um but you know part of it is a relevant because if I've got this
[00:37:53] is a relevant because if I've got this picture you know nothing is happening in
[00:37:55] picture you know nothing is happening in the part that's sort of shown in yellow
[00:37:57] the part that's sort of shown in yellow there um but if you want you know at the
[00:38:01] there um but if you want you know at the moment you know this is just an a a good
[00:38:05] moment you know this is just an a a good a full decomposition but if we wanting
[00:38:08] a full decomposition but if we wanting to have sort of smaller low dimensional
[00:38:10] to have sort of smaller low dimensional vectors well the next trick we pull is
[00:38:13] vectors well the next trick we pull is we say well we know where the smallest
[00:38:15] we say well we know where the smallest singular vectors are so we could just
[00:38:18] singular vectors are so we could just set them to zero and if we did that then
[00:38:21] set them to zero and if we did that then more of this goes away and we end up
[00:38:25] more of this goes away and we end up with two-dimensional representation
[00:38:28] with two-dimensional representation of our words and so that gives us
[00:38:31] of our words and so that gives us another way of um forming
[00:38:34] another way of um forming low-dimensional word
[00:38:36] low-dimensional word representations um and this had actually
[00:38:39] representations um and this had actually been explored before modern new word
[00:38:42] been explored before modern new word vectors and using algorithms such as
[00:38:44] vectors and using algorithms such as latent semantic analysis um and it is
[00:38:49] latent semantic analysis um and it is sort of half worked but it never worked
[00:38:53] sort of half worked but it never worked very well but you know some people
[00:38:56] very well but you know some people especially in psychology had kept on
[00:38:58] especially in psychology had kept on working on it um and among other people
[00:39:02] working on it um and among other people in the early 2000s there was this grad
[00:39:04] in the early 2000s there was this grad student um Doug roie who kept on working
[00:39:07] student um Doug roie who kept on working on it and he came up with an algorithm
[00:39:11] on it and he came up with an algorithm um that he called Kohl's and you know he
[00:39:16] um that he called Kohl's and you know he he had known as other people before him
[00:39:18] he had known as other people before him had known that just sort of doing an SVD
[00:39:21] had known that just sort of doing an SVD on Raw counts didn't seem to give you
[00:39:23] on Raw counts didn't seem to give you word vectors that worked very well um
[00:39:26] word vectors that worked very well um but he had some ideas to do better than
[00:39:28] but he had some ideas to do better than that um so one thing that helps a lot is
[00:39:31] that um so one thing that helps a lot is if you log the
[00:39:32] if you log the frequencies so you can put log
[00:39:34] frequencies so you can put log frequencies in the cells um but then um
[00:39:39] frequencies in the cells um but then um he sort of used some other ideas some of
[00:39:41] he sort of used some other ideas some of which were also picked up in word to V
[00:39:44] which were also picked up in word to V one of which is ramping the windows so
[00:39:46] one of which is ramping the windows so that you count closer words more than
[00:39:48] that you count closer words more than further away words um he used piercon
[00:39:52] further away words um he used piercon correlations instead of counts Etc but
[00:39:55] correlations instead of counts Etc but he ended up coming up with a low dial
[00:39:58] he ended up coming up with a low dial version of Word vectors that are sort of
[00:40:00] version of Word vectors that are sort of ultimately still based on an SVD um and
[00:40:04] ultimately still based on an SVD um and he got out these word vectors and you
[00:40:07] he got out these word vectors and you know interestingly sort of no one really
[00:40:10] know interestingly sort of no one really noticed at the time but Doug roie in his
[00:40:13] noticed at the time but Doug roie in his dissertation effectively discovered this
[00:40:17] dissertation effectively discovered this same property of having linear semantic
[00:40:19] same property of having linear semantic components so look here we go here's one
[00:40:22] components so look here we go here's one of so this is actually you know picture
[00:40:24] of so this is actually you know picture from his dissertation and look here you
[00:40:27] from his dissertation and look here you know we've got this meaning component
[00:40:29] know we've got this meaning component which is doer of an event and he's
[00:40:31] which is doer of an event and he's essentially shown with the way he's
[00:40:33] essentially shown with the way he's processed his word vectors that the do
[00:40:36] processed his word vectors that the do of an event is a linear meaning
[00:40:39] of an event is a linear meaning component that you can use to move
[00:40:41] component that you can use to move between a verb and the doer of the verb
[00:40:44] between a verb and the doer of the verb kind of cool but he didn't become famous
[00:40:47] kind of cool but he didn't become famous because no one was paying attention to
[00:40:49] because no one was paying attention to what he had come up with so that so once
[00:40:52] what he had come up with so that so once word DEC um became popular that was
[00:40:57] word DEC um became popular that was something that I was kind of interested
[00:41:00] something that I was kind of interested in and so um working together with a
[00:41:03] in and so um working together with a postto Jeffrey Pennington um we thought
[00:41:07] postto Jeffrey Pennington um we thought that you know you there was interest in
[00:41:11] that you know you there was interest in this sort of space of having doing
[00:41:13] this sort of space of having doing things with matrices of counts and how
[00:41:16] things with matrices of counts and how do you then get them to work well as
[00:41:19] do you then get them to work well as word vectors in the same way that word
[00:41:21] word vectors in the same way that word de V worked well as word vectors and so
[00:41:24] de V worked well as word vectors and so that's what led um into the glove
[00:41:27] that's what led um into the glove algorithm that was what I was actually
[00:41:29] algorithm that was what I was actually showing you and so um what we wanted was
[00:41:34] showing you and so um what we wanted was to say look we want a model in which um
[00:41:39] to say look we want a model in which um linear components sort of adding or
[00:41:41] linear components sort of adding or subtracting a vector in a vector space
[00:41:44] subtracting a vector in a vector space correspond to a meaning difference how
[00:41:47] correspond to a meaning difference how can we do that um and Jeffrey um did
[00:41:52] can we do that um and Jeffrey um did good thinking and uh math and um thought
[00:41:56] good thinking and uh math and um thought about that for a bit and his solution
[00:42:00] about that for a bit and his solution was to say well if we think of um that
[00:42:04] was to say well if we think of um that ratios of cooccurrence probabilities can
[00:42:07] ratios of cooccurrence probabilities can encode meaning components so if we can
[00:42:09] encode meaning components so if we can make a ratio of cooccurrence
[00:42:11] make a ratio of cooccurrence probabilities into something linear in
[00:42:14] probabilities into something linear in the vector space will get the kind of
[00:42:16] the vector space will get the kind of result that word DEC or Doug Ro got um
[00:42:20] result that word DEC or Doug Ro got um so what does that mean well so if you
[00:42:24] so what does that mean well so if you start thinking of words occurring in the
[00:42:26] start thinking of words occurring in the context of I ice you might think that
[00:42:28] context of I ice you might think that sort of solid and water are likely to K
[00:42:31] sort of solid and water are likely to K near ice and gas or a random word like
[00:42:35] near ice and gas or a random word like random aren't likely to occur near ice
[00:42:39] random aren't likely to occur near ice but and similar for steam you'd expect
[00:42:43] but and similar for steam you'd expect um that you know gas and water are
[00:42:46] um that you know gas and water are likely to occur near steam but probably
[00:42:48] likely to occur near steam but probably not solid or random and well if you're
[00:42:51] not solid or random and well if you're just looking at one of these you don't
[00:42:53] just looking at one of these you don't really get meaning components because
[00:42:56] really get meaning components because you get something that's large here or
[00:42:58] you get something that's large here or large here but if you then look at the
[00:43:01] large here but if you then look at the ratio of two of these cooccurrence
[00:43:05] ratio of two of these cooccurrence probabilities then what you get out is
[00:43:08] probabilities then what you get out is that for solid it's going to be large
[00:43:10] that for solid it's going to be large and small is going to be um and for gas
[00:43:14] and small is going to be um and for gas it's going to be small and so you're
[00:43:16] it's going to be small and so you're getting a direction in the space which
[00:43:19] getting a direction in the space which will correspond to the um solid liquid
[00:43:22] will correspond to the um solid liquid gas dimension of physics um whereas for
[00:43:26] gas dimension of physics um whereas for the other words it will be about one um
[00:43:29] the other words it will be about one um this is just uh uh the wave your hands
[00:43:32] this is just uh uh the wave your hands this was the conception of the idea but
[00:43:34] this was the conception of the idea but if you actually do the counts this
[00:43:36] if you actually do the counts this actually works out so using real data
[00:43:39] actually works out so using real data this is what you get for cooccurrence
[00:43:42] this is what you get for cooccurrence and indeed you kind of get these sort of
[00:43:44] and indeed you kind of get these sort of factors of 10 um in both of these
[00:43:47] factors of 10 um in both of these directions of these two and the numbers
[00:43:49] directions of these two and the numbers are over there are approximately one um
[00:43:53] are over there are approximately one um so um Jeffrey's idea was well we going
[00:43:57] so um Jeffrey's idea was well we going to start with a cooccurrence count
[00:44:00] to start with a cooccurrence count Matrix and we want to make this turn
[00:44:05] Matrix and we want to make this turn into a linear component and well how do
[00:44:08] into a linear component and well how do you do that well first of all it sort of
[00:44:10] you do that well first of all it sort of makes sense immediately that you should
[00:44:12] makes sense immediately that you should be putting a login right because once
[00:44:14] be putting a login right because once you put a log in um this ratio will be
[00:44:17] you put a log in um this ratio will be being turned into something that's
[00:44:19] being turned into something that's subtracted um and so simply all you have
[00:44:23] subtracted um and so simply all you have to do is have a log by linear model
[00:44:25] to do is have a log by linear model where the um dotproduct of two word
[00:44:29] where the um dotproduct of two word vectors models this conditional
[00:44:32] vectors models this conditional probability and then the difference
[00:44:34] probability and then the difference between two vectors will be
[00:44:36] between two vectors will be corresponding to this log of the ratio
[00:44:39] corresponding to this log of the ratio of their cooccurrence
[00:44:41] of their cooccurrence probabilities um so that was basically
[00:44:44] probabilities um so that was basically the glove model um so you're wanting to
[00:44:48] the glove model um so you're wanting to you know model this dot product um such
[00:44:53] you know model this dot product um such that it's being close to the log of the
[00:44:56] that it's being close to the log of the cooccur Cur probability but you sort of
[00:44:59] cooccur Cur probability but you sort of do a little bit of extra work to have
[00:45:01] do a little bit of extra work to have some biased terms and and some frequency
[00:45:04] some biased terms and and some frequency thresholds which aren't very important
[00:45:07] thresholds which aren't very important um so I'm going to skip past them but I
[00:45:09] um so I'm going to skip past them but I think that basic intuition as to what's
[00:45:13] think that basic intuition as to what's the important thing to get linear
[00:45:14] the important thing to get linear meaning components is a good one to know
[00:45:19] about okay is everyone good
[00:45:23] about okay is everyone good today
[00:45:25] today cool yes all right noticed the the
[00:45:28] cool yes all right noticed the the original X Matrix you show was like 3x 5
[00:45:31] original X Matrix you show was like 3x 5 or something shouldn't It Be Square so
[00:45:34] or something shouldn't It Be Square so yeah I mean if you're doing sorry yeah I
[00:45:37] yeah I mean if you're doing sorry yeah I maybe should have just shown you a
[00:45:38] maybe should have just shown you a square one if you're just doing
[00:45:40] square one if you're just doing vocabulary to vocabulary yes it should
[00:45:42] vocabulary to vocabulary yes it should be square um but there was a bit in the
[00:45:45] be square um but there was a bit in the slides that I didn't mention that there
[00:45:46] slides that I didn't mention that there was another way you could do it where
[00:45:48] was another way you could do it where you did it words versus documents and
[00:45:50] you did it words versus documents and then it would be non Square um but yeah
[00:45:52] then it would be non Square um but yeah you're right so let we can just just um
[00:45:57] you're right so let we can just just um consider the square case Okay
[00:46:00] consider the square case Okay um so you know hey I showed you that
[00:46:04] um so you know hey I showed you that demo of the glove vectors and they work
[00:46:06] demo of the glove vectors and they work great didn't
[00:46:07] great didn't they um so you know these are good
[00:46:09] they um so you know these are good vectors um but in general in NLP we'd
[00:46:13] vectors um but in general in NLP we'd like to have things that we can evaluate
[00:46:16] like to have things that we can evaluate and know whether things are really good
[00:46:19] and know whether things are really good um and
[00:46:21] um and so everywhere through the course we're
[00:46:23] so everywhere through the course we're going to want to evaluate things and
[00:46:25] going to want to evaluate things and work out how good they are and what's
[00:46:27] work out how good they are and what's better and what's worse um and so one of
[00:46:31] better and what's worse um and so one of the fundamental Notions of evaluation
[00:46:33] the fundamental Notions of evaluation that will come up again and again is
[00:46:35] that will come up again and again is intrinsic and extrinsic evaluations so
[00:46:38] intrinsic and extrinsic evaluations so an intrinsic evaluation is where you are
[00:46:41] an intrinsic evaluation is where you are doing a very specific internal subtask
[00:46:46] doing a very specific internal subtask and you just try and score whether it's
[00:46:48] and you just try and score whether it's good or bad so normally intrinsic
[00:46:50] good or bad so normally intrinsic evaluations are fast to compute help you
[00:46:54] evaluations are fast to compute help you understand the component you're building
[00:46:56] understand the component you're building but they are sort of distant from your
[00:46:59] but they are sort of distant from your Downstream task and improving the
[00:47:01] Downstream task and improving the numbers internally may or may not help
[00:47:04] numbers internally may or may not help you and that's um the contrast with an
[00:47:07] you and that's um the contrast with an extrinsic evaluation um which is that
[00:47:11] extrinsic evaluation um which is that you've got some real task you want to do
[00:47:13] you've got some real task you want to do question answering or document
[00:47:15] question answering or document summarization or machine
[00:47:17] summarization or machine translation um and you want to know
[00:47:20] translation um and you want to know whether some clever bit of um internal
[00:47:23] whether some clever bit of um internal modeling will help you on that task so
[00:47:26] modeling will help you on that task so then you have to sort of run an entire
[00:47:28] then you have to sort of run an entire system and work out Downstream
[00:47:31] system and work out Downstream accuracies um and find out whether it
[00:47:34] accuracies um and find out whether it actually helps you at the end of the day
[00:47:36] actually helps you at the end of the day um but that often means it's kind of you
[00:47:38] um but that often means it's kind of you know indirect so harder to see exactly
[00:47:41] know indirect so harder to see exactly what's happening in your task um so for
[00:47:44] what's happening in your task um so for something like word vectors you know if
[00:47:46] something like word vectors you know if we just sort of measure are they
[00:47:48] we just sort of measure are they modeling word similarity well that's an
[00:47:52] modeling word similarity well that's an intrinsic
[00:47:53] intrinsic evaluation but you know we'd probably
[00:47:56] evaluation but you know we'd probably like to know whether they model word
[00:47:59] like to know whether they model word similarity well for some Downstream task
[00:48:02] similarity well for some Downstream task which might be um doing web search right
[00:48:05] which might be um doing web search right we'd like um when you say you know cell
[00:48:09] we'd like um when you say you know cell phone or mobile phone that it comes out
[00:48:11] phone or mobile phone that it comes out at about the same um so that would then
[00:48:13] at about the same um so that would then be web search might be our extrinsic
[00:48:17] be web search might be our extrinsic evaluation okay so for word factors um
[00:48:22] evaluation okay so for word factors um two two intrinsic evaluations the ones
[00:48:25] two two intrinsic evaluations the ones we've already seen
[00:48:27] we've already seen so there's the word Vector analogies um
[00:48:30] so there's the word Vector analogies um you know I cheated when I showed you the
[00:48:33] you know I cheated when I showed you the glove demo I only showed you ones that
[00:48:36] glove demo I only showed you ones that work but if you play for it yourself you
[00:48:38] work but if you play for it yourself you can find some that don't work um so what
[00:48:41] can find some that don't work um so what we can do is sort of have a set of um
[00:48:45] we can do is sort of have a set of um word analogies and find out which ones
[00:48:48] word analogies and find out which ones work now you know in general glove does
[00:48:51] work now you know in general glove does work you know here's a set of word
[00:48:53] work you know here's a set of word vectors um showing you um the sort of
[00:48:57] vectors um showing you um the sort of um female distinction it's kind of good
[00:48:59] um female distinction it's kind of good and linear but in general for different
[00:49:02] and linear but in general for different ones it's going to work and it's not
[00:49:04] ones it's going to work and it's not work and you're going to be able to
[00:49:05] work and you're going to be able to score what percentage of the time it
[00:49:08] score what percentage of the time it works um or we can do um word similarity
[00:49:12] works um or we can do um word similarity How We Do word similarity is we actually
[00:49:15] How We Do word similarity is we actually use human judgments of similarity so um
[00:49:19] use human judgments of similarity so um psychologists ask undergrads and they
[00:49:23] psychologists ask undergrads and they say here is the word plane and car how
[00:49:26] say here is the word plane and car how similar are they on a scale of 1 to 10
[00:49:29] similar are they on a scale of 1 to 10 or 0 to 10 maybe actually I think it's 0
[00:49:31] or 0 to 10 maybe actually I think it's 0 to 10 here on a scale of 0 to 10 and the
[00:49:35] to 10 here on a scale of 0 to 10 and the person says seven and then they ask
[00:49:38] person says seven and then they ask another person um and uh they average
[00:49:43] another person um and uh they average what the undergrads say and they come
[00:49:45] what the undergrads say and they come out with these numbers so you know Tiger
[00:49:47] out with these numbers so you know Tiger Tiger gets 10 book and paper got a an
[00:49:50] Tiger gets 10 book and paper got a an average of
[00:49:52] average of 7.46 um plane and car got
[00:49:55] 7.46 um plane and car got 5.77 um stock and phone got 1.62 and
[00:49:59] 5.77 um stock and phone got 1.62 and stock and jaguar got
[00:50:01] stock and jaguar got 0.92 um noisy process but you roughly
[00:50:04] 0.92 um noisy process but you roughly get to see how similar people think
[00:50:06] get to see how similar people think words are and so then we ask our models
[00:50:09] words are and so then we ask our models to also score how similar they think
[00:50:12] to also score how similar they think words are and then we get models of how
[00:50:16] words are and then we get models of how well the scores are correlated um
[00:50:20] well the scores are correlated um between um human judgments and our
[00:50:23] between um human judgments and our models judgments um and so here are sort
[00:50:26] models judgments um and so here are sort of a big table of numbers that we um
[00:50:29] of a big table of numbers that we um don't need to go through all of but you
[00:50:31] don't need to go through all of but you know it sort of shows that a plain SVD
[00:50:34] know it sort of shows that a plain SVD Works terribly simply doing SVD over log
[00:50:38] Works terribly simply doing SVD over log counts already starts to work reasonably
[00:50:41] counts already starts to work reasonably um and then you know here's um the two
[00:50:43] um and then you know here's um the two word DEC algorithms sibo and Skip gram
[00:50:47] word DEC algorithms sibo and Skip gram and here are numbers from our glove
[00:50:49] and here are numbers from our glove vectors and so you get these kind of
[00:50:51] vectors and so you get these kind of scores that you can then score different
[00:50:53] scores that you can then score different models as to how good they are
[00:50:57] models as to how good they are and well then you can ALS oh sorry yeah
[00:51:00] and well then you can ALS oh sorry yeah um that's the only thing I have there
[00:51:02] um that's the only thing I have there but you know what can you do for
[00:51:05] but you know what can you do for Downstream evaluation well then you want
[00:51:07] Downstream evaluation well then you want to um pick some Downstream task and so a
[00:51:11] to um pick some Downstream task and so a simple Downstream task that's been used
[00:51:13] simple Downstream task that's been used a lot um in NLP is what's called named
[00:51:17] a lot um in NLP is what's called named entity recognition and so that's
[00:51:20] entity recognition and so that's recognizing names of things and what
[00:51:22] recognizing names of things and what type they are um so if the sentence is
[00:51:25] type they are um so if the sentence is Chris Manning lives in poo Alto you want
[00:51:28] Chris Manning lives in poo Alto you want to say Chris and Manning that's the name
[00:51:30] to say Chris and Manning that's the name of a person and poo and outto that's the
[00:51:34] of a person and poo and outto that's the name of a place um so that can be the
[00:51:37] name of a place um so that can be the task and well that's the kind of task
[00:51:40] task and well that's the kind of task which you might think word vectors would
[00:51:42] which you might think word vectors would help you with um and it's um indeed the
[00:51:45] help you with um and it's um indeed the case right so he what's label was
[00:51:47] case right so he what's label was discreet was a baseline symbolic um
[00:51:51] discreet was a baseline symbolic um probabilistic um name entity recognition
[00:51:55] probabilistic um name entity recognition task and by putting word vectors into it
[00:51:58] task and by putting word vectors into it you can make them the numbers go up so
[00:52:01] you can make them the numbers go up so these numbers for glav are higher than
[00:52:03] these numbers for glav are higher than the ones on the first line and so I'm
[00:52:06] the ones on the first line and so I'm getting substantial improvements from
[00:52:09] getting substantial improvements from adding word vectors to my system yay
[00:52:15] um okay
[00:52:18] um okay um I'll pile ahead into the next thing
[00:52:21] um I'll pile ahead into the next thing this next one I think is interesting we
[00:52:23] this next one I think is interesting we should spend a minute on and it came up
[00:52:25] should spend a minute on and it came up in your questions last time words have
[00:52:29] in your questions last time words have lots of meanings most words um have a
[00:52:34] lots of meanings most words um have a whole bunch of meanings words that don't
[00:52:37] whole bunch of meanings words that don't have a lot of different meanings are
[00:52:39] have a lot of different meanings are only some very specialized scientific
[00:52:41] only some very specialized scientific words okay so my example of of word with
[00:52:45] words okay so my example of of word with multiple meanings is probably not the
[00:52:47] multiple meanings is probably not the first one you think of all the time um
[00:52:49] first one you think of all the time um the most famous example of a word with a
[00:52:51] the most famous example of a word with a lot of meanings is bank which already
[00:52:52] lot of meanings is bank which already came up last time and I use star which
[00:52:55] came up last time and I use star which is another one here's a word that you
[00:52:57] is another one here's a word that you probably don't use that often um but it
[00:52:59] probably don't use that often um but it you know it still has lots of meaning so
[00:53:01] you know it still has lots of meaning so the word Pike what are some things that
[00:53:03] the word Pike what are some things that the word Pike can
[00:53:05] the word Pike can mean fish a fish yes it's a kind of fish
[00:53:08] mean fish a fish yes it's a kind of fish okay we've got one what else can a pike
[00:53:11] okay we've got one what else can a pike be yeah a spear a spear yeah for the
[00:53:14] be yeah a spear a spear yeah for the Dungeons and Dragons crowd yeah there's
[00:53:16] Dungeons and Dragons crowd yeah there's a long arm right yep that's another one
[00:53:19] a long arm right yep that's another one yeah a road right yes so Pike is used as
[00:53:23] yeah a road right yes so Pike is used as a shorthand well a shorthand for a ter
[00:53:26] a shorthand well a shorthand for a ter Turn Pike why it's called a Turnpike
[00:53:28] Turn Pike why it's called a Turnpike where yeah originally you had you know
[00:53:30] where yeah originally you had you know this the spey looking thing um at the
[00:53:33] this the spey looking thing um at the start of it as sort of count people okay
[00:53:35] start of it as sort of count people okay we've got three other thing meanings for
[00:53:37] we've got three other thing meanings for pike yeah is it also a crap like a
[00:53:42] pike yeah is it also a crap like a [Music]
[00:53:43] [Music] fraternity I'll believe you I can't say
[00:53:45] fraternity I'll believe you I can't say I know that one
[00:53:48] I know that one um are
[00:53:51] um are Pikes sharp as like a needle something
[00:53:55] Pikes sharp as like a needle something Sharp
[00:53:57] Sharp maybe I mean I think it's really the
[00:53:59] maybe I mean I think it's really the sort of Pike as the
[00:54:02] weapon other scratch your heads um one
[00:54:06] weapon other scratch your heads um one that I think a lot of you will have seen
[00:54:09] that I think a lot of you will have seen um in diving and swimming you can do a
[00:54:13] um in diving and swimming you can do a pike Olympics if you see olympic diving
[00:54:17] pike Olympics if you see olympic diving there are Pikes anyone seen
[00:54:20] there are Pikes anyone seen those um trust me that's a pike um okay
[00:54:25] those um trust me that's a pike um okay um and we've sort of been doing um the
[00:54:28] um and we've sort of been doing um the noun uses but you know you can also use
[00:54:32] noun uses but you know you can also use Pike as a verb right you know like once
[00:54:35] Pike as a verb right you know like once you've got your medieval weapon you can
[00:54:37] you've got your medieval weapon you can Pike somebody um and that's a usage of
[00:54:41] Pike somebody um and that's a usage of Pike um and you can do other ones right
[00:54:44] Pike um and you can do other ones right so uh here we go here's
[00:54:47] so uh here we go here's um um ones I got from a dictionary we
[00:54:51] um um ones I got from a dictionary we got most of those there are sort of
[00:54:53] got most of those there are sort of weirder usages right like coming down
[00:54:55] weirder usages right like coming down the pike that's kind of a metaphorical
[00:54:57] the pike that's kind of a metaphorical use that comes um from the the road
[00:55:01] use that comes um from the the road sense but it sort of ends up meaning the
[00:55:03] sense but it sort of ends up meaning the future um yeah um in Australia we also
[00:55:07] future um yeah um in Australia we also use Pike to mean um sort of chicken out
[00:55:10] use Pike to mean um sort of chicken out of doing something um but I don't think
[00:55:13] of doing something um but I don't think that usage is really used in the US
[00:55:15] that usage is really used in the US anyway words have lots of meanings so
[00:55:17] anyway words have lots of meanings so how can you deal with that well one way
[00:55:20] how can you deal with that well one way you could deal with it is to say okay
[00:55:23] you could deal with it is to say okay words have several meanings
[00:55:27] words have several meanings and so we're just going to say words
[00:55:29] and so we're just going to say words have several meanings and then we're
[00:55:31] have several meanings and then we're going to take instances of words in text
[00:55:34] going to take instances of words in text we're going to Cluster them based on
[00:55:37] we're going to Cluster them based on their similarity of occurrence to decide
[00:55:39] their similarity of occurrence to decide which sense of the word to regard each
[00:55:43] which sense of the word to regard each token as and then we're going to learn
[00:55:46] token as and then we're going to learn word vectors for those token clusters
[00:55:50] word vectors for those token clusters which are our sensors and you can do
[00:55:52] which are our sensors and you can do that um we did it in 2012
[00:55:56] that um we did it in 2012 before word de came out um so you see
[00:55:59] before word de came out um so you see here we have bank one and somewhere over
[00:56:03] here we have bank one and somewhere over here we have Bank two and here we have
[00:56:06] here we have Bank two and here we have Jaguar one Jaguar 2 Jaguar 3 Jaguar 4
[00:56:12] Jaguar one Jaguar 2 Jaguar 3 Jaguar 4 and you know this really works out great
[00:56:15] and you know this really works out great right so Jaguar one um picks out the
[00:56:19] right so Jaguar one um picks out the sense of the um kind of car right and
[00:56:22] sense of the um kind of car right and it's close to luxury and convertible
[00:56:24] it's close to luxury and convertible Jaguar 2
[00:56:26] Jaguar 2 comes right close to software in
[00:56:28] comes right close to software in Microsoft and this one's a bit of a
[00:56:31] Microsoft and this one's a bit of a historical one but you know it's when
[00:56:34] historical one but you know it's when you were when most of you were five or
[00:56:37] you were when most of you were five or whatever you might remember um Apple
[00:56:39] whatever you might remember um Apple used to use large cats for versions of
[00:56:44] used to use large cats for versions of um Mac OS right so sort of Mac OS 10.3
[00:56:49] um Mac OS right so sort of Mac OS 10.3 or something like that a long time ago
[00:56:51] or something like that a long time ago was called Jaguar right so it's software
[00:56:54] was called Jaguar right so it's software close to Microsoft um Jaguar 3 um okay
[00:56:59] close to Microsoft um Jaguar 3 um okay string keyboard solo musical um drum
[00:57:03] string keyboard solo musical um drum base that's because there's a Jaguar
[00:57:05] base that's because there's a Jaguar keyboard um and then finally um the sort
[00:57:10] keyboard um and then finally um the sort of what we think of is the basic sense
[00:57:12] of what we think of is the basic sense but turns out turns up rather Less in
[00:57:15] but turns out turns up rather Less in text corpor normally um Jaguar next to
[00:57:18] text corpor normally um Jaguar next to Hunter is the animal right so it's done
[00:57:20] Hunter is the animal right so it's done a good job at learning the different
[00:57:22] a good job at learning the different sensors but you know that's not what's
[00:57:26] sensors but you know that's not what's actually usually done these days and
[00:57:28] actually usually done these days and instead you know what's usually done is
[00:57:31] instead you know what's usually done is you do only have one vector for Jaguar
[00:57:35] you do only have one vector for Jaguar and when you do that um or Pike here the
[00:57:39] and when you do that um or Pike here the one vector you learn is a weighted a
[00:57:44] one vector you learn is a weighted a weighted average of the vectors that you
[00:57:47] weighted average of the vectors that you would have learned for the
[00:57:49] would have learned for the senses um it's often referred to as a
[00:57:53] senses um it's often referred to as a superposition because somehow um neet
[00:57:56] superposition because somehow um neet math people like to use physics terms um
[00:58:00] math people like to use physics terms um and
[00:58:00] and so they call it a superposition but it's
[00:58:03] so they call it a superposition but it's a weighted average so you're taking the
[00:58:06] a weighted average so you're taking the relative frequency of the different
[00:58:08] relative frequency of the different senses and multiplying the vectors you
[00:58:10] senses and multiplying the vectors you would have learned if you'd had sense
[00:58:12] would have learned if you'd had sense vectors and that's what you get as the
[00:58:15] vectors and that's what you get as the representation as a whole
[00:58:18] representation as a whole um and um you know I can make a sort of
[00:58:22] um and um you know I can make a sort of a linguistic argument as to why you
[00:58:24] a linguistic argument as to why you might want to do that um which is you
[00:58:27] might want to do that um which is you know although this model of words have
[00:58:31] know although this model of words have senses is you know very longstanding and
[00:58:35] senses is you know very longstanding and common I mean it's essentially the way
[00:58:37] common I mean it's essentially the way dictionaries are built right you look up
[00:58:39] dictionaries are built right you look up a word in the dictionary and it says
[00:58:41] a word in the dictionary and it says sense one sense 2 sense three and you
[00:58:44] sense one sense 2 sense three and you get them for things like bank or Jaguars
[00:58:46] get them for things like bank or Jaguars we're talking about I mean it's sort of
[00:58:50] we're talking about I mean it's sort of really a broken model right that
[00:58:53] really a broken model right that like word meanings have a lot of nuance
[00:58:57] like word meanings have a lot of nuance they're used in a lot of different
[00:58:59] they're used in a lot of different contexts they are extreme examples like
[00:59:02] contexts they are extreme examples like Bank wherever it was where we have
[00:59:05] Bank wherever it was where we have Finance bank and Bank of a river bank
[00:59:07] Finance bank and Bank of a river bank over here where it seems like the senses
[00:59:10] over here where it seems like the senses are this far apart but you know most
[00:59:13] are this far apart but you know most words have sort of different meanings
[00:59:15] words have sort of different meanings but they're not actually that far apart
[00:59:17] but they're not actually that far apart and trying to cut them into senses seems
[00:59:20] and trying to cut them into senses seems actually very artificial and you know
[00:59:24] actually very artificial and you know there you know if you look up different
[00:59:26] there you know if you look up different dictionaries and you say how many senses
[00:59:28] dictionaries and you say how many senses does this word have pretty much everyone
[00:59:31] does this word have pretty much everyone will give you a different answer so the
[00:59:33] will give you a different answer so the kind of situation you have is a word
[00:59:36] kind of situation you have is a word like like field well a field can be used
[00:59:40] like like field well a field can be used for a place where you grow a crop um it
[00:59:43] for a place where you grow a crop um it can be used for sort of natural things
[00:59:46] can be used for sort of natural things like a a rock field or an ice field it
[00:59:50] like a a rock field or an ice field it can be used for a sporting field um
[00:59:54] can be used for a sporting field um there's the mathematical sense of field
[00:59:56] there's the mathematical sense of field now all of these things sort of have
[00:59:58] now all of these things sort of have something to do with each other I mean
[01:00:00] something to do with each other I mean the math one's further away but the
[01:00:01] the math one's further away but the physical ones are sort sort of flat
[01:00:04] physical ones are sort sort of flat spaces um but you know the the the sense
[01:00:08] spaces um but you know the the the sense of it being a sporting field is clearly
[01:00:11] of it being a sporting field is clearly kind of different from the sense of it
[01:00:12] kind of different from the sense of it being an ice field is the ice field and
[01:00:16] being an ice field is the ice field and The Rock field different or am I just
[01:00:19] The Rock field different or am I just modifying they are they different senses
[01:00:22] modifying they are they different senses right so really you sort of have a kind
[01:00:24] right so really you sort of have a kind of a a
[01:00:26] of a a uh what a math person say is sort of
[01:00:29] uh what a math person say is sort of like some probability density
[01:00:31] like some probability density distribution over things that can be
[01:00:33] distribution over things that can be meant by the meaning of a word so it
[01:00:35] meant by the meaning of a word so it sort of maybe makes sense to more use
[01:00:37] sort of maybe makes sense to more use this model where you're just actually
[01:00:39] this model where you're just actually saying we have a vector that's an
[01:00:42] saying we have a vector that's an average of all the contexts and we'll
[01:00:44] average of all the contexts and we'll see more of that when we get to
[01:00:45] see more of that when we get to contextual word vectors um later on um
[01:00:49] contextual word vectors um later on um but one more surprising result on this
[01:00:53] but one more surprising result on this is um since you you have um the vector
[01:00:59] is um since you you have um the vector for pike overall being the sum of these
[01:01:03] for pike overall being the sum of these different um sense vectors you know
[01:01:08] different um sense vectors you know standard math would tell you that if you
[01:01:10] standard math would tell you that if you just have the single Vector there's no
[01:01:13] just have the single Vector there's no way that you can recover the individual
[01:01:15] way that you can recover the individual sense
[01:01:16] sense vectors but um higher maath tells you um
[01:01:21] vectors but um higher maath tells you um that actually these um Vector spaces are
[01:01:26] that actually these um Vector spaces are so high dimensional and sparse that you
[01:01:29] so high dimensional and sparse that you can use ideas from sparse coding Theory
[01:01:33] can use ideas from sparse coding Theory to reconstruct the sense vectors out of
[01:01:36] to reconstruct the sense vectors out of the whole vector and if you actually
[01:01:39] the whole vector and if you actually want to understand this um some of the
[01:01:42] want to understand this um some of the people in statistics David Dono I think
[01:01:44] people in statistics David Dono I think is one of them um teach courses on
[01:01:46] is one of them um teach courses on sparse coding Theory um but um I'm not
[01:01:50] sparse coding Theory um but um I'm not going to try and teach that but you know
[01:01:52] going to try and teach that but you know here's an example um from this paper the
[01:01:55] here's an example um from this paper the San
[01:01:56] San Al where one of the ATS is tar who's now
[01:01:59] Al where one of the ATS is tar who's now faculty in computer science here where
[01:02:03] faculty in computer science here where they uh starting off with the word
[01:02:05] they uh starting off with the word vector and using sparse coding um to
[01:02:09] vector and using sparse coding um to divide out sense vectors from one um
[01:02:12] divide out sense vectors from one um word vector and they work pretty well
[01:02:14] word vector and they work pretty well right so here's one sense of tie which
[01:02:17] right so here's one sense of tie which is piece of clothing another sense of
[01:02:20] is piece of clothing another sense of tie which is ties in the game um this
[01:02:24] tie which is ties in the game um this one is sort of similar to that one and
[01:02:26] one is sort of similar to that one and I'll admit but this sense of tie here is
[01:02:28] I'll admit but this sense of tie here is then a tie as sort of you put on your
[01:02:31] then a tie as sort of you put on your electrical cables um then you have the
[01:02:34] electrical cables um then you have the musical sense of tie right at least four
[01:02:36] musical sense of tie right at least four out of five they've done a pretty good
[01:02:39] out of five they've done a pretty good job of getting senses out of this single
[01:02:42] job of getting senses out of this single um word vector by sparse coding so
[01:02:44] um word vector by sparse coding so sparse coding must be cool if you want
[01:02:46] sparse coding must be cool if you want to go off and learn more about
[01:02:49] to go off and learn more about it
[01:02:53] okay okay so that's everything I was
[01:02:56] okay okay so that's everything I was going to say about words vectors and
[01:02:58] going to say about words vectors and word
[01:02:59] word senses is everyone good to there any
[01:03:05] questions I'll Rush ahead for the last
[01:03:08] questions I'll Rush ahead for the last two pieces okay so I just wanted to
[01:03:11] two pieces okay so I just wanted to start to introdu in the last 15 minutes
[01:03:14] start to introdu in the last 15 minutes um the ideas of how we can build neural
[01:03:20] um the ideas of how we can build neural classifiers um and how we start to build
[01:03:23] classifiers um and how we start to build in general neural network works I mean
[01:03:26] in general neural network works I mean in a sense we've already built a very
[01:03:30] in a sense we've already built a very simple neural classifier our word DEC
[01:03:34] simple neural classifier our word DEC model is predicting what words are
[01:03:37] model is predicting what words are likely to occur in the context of
[01:03:39] likely to occur in the context of another word um and you can think of
[01:03:41] another word um and you can think of that as a classifier but let's look at a
[01:03:43] that as a classifier but let's look at a simple classifier like our named entity
[01:03:45] simple classifier like our named entity recognizers I mentioned before so for
[01:03:48] recognizers I mentioned before so for the named entity recognizer we want to
[01:03:50] the named entity recognizer we want to label words with their class so we want
[01:03:53] label words with their class so we want to say these two words are a person but
[01:03:56] to say these two words are a person but the same words Paris and Hilton are then
[01:04:00] the same words Paris and Hilton are then locations in this second sentence so
[01:04:03] locations in this second sentence so words can be ambiguous as to what their
[01:04:06] words can be ambiguous as to what their class is um and the other state is that
[01:04:11] class is um and the other state is that they're not a named entity at all
[01:04:13] they're not a named entity at all they're just a word um that is some
[01:04:15] they're just a word um that is some other word and this is something that's
[01:04:17] other word and this is something that's used in lots of places um as a a bit of
[01:04:22] used in lots of places um as a a bit of understanding so if you've seen any of
[01:04:24] understanding so if you've seen any of those web pages um where they've sort of
[01:04:26] those web pages um where they've sort of you know tagged company names with a
[01:04:28] you know tagged company names with a stock ticker or um there links on a
[01:04:32] stock ticker or um there links on a Wikipedia page to a Wikipedia page or
[01:04:35] Wikipedia page to a Wikipedia page or something like that right you've got
[01:04:37] something like that right you've got named entities where commonly after
[01:04:40] named entities where commonly after finding the named entities um you're
[01:04:42] finding the named entities um you're doing this second stage of entity
[01:04:44] doing this second stage of entity linking where you're then linking the
[01:04:46] linking where you're then linking the named entity to some canonical form of
[01:04:49] named entity to some canonical form of it like a wikkipedia page but we're not
[01:04:51] it like a wikkipedia page but we're not going to talk about the second part of
[01:04:53] going to talk about the second part of it um for the rest of the day
[01:04:56] it um for the rest of the day um and
[01:04:58] um and so um we could say that building with
[01:05:01] so um we could say that building with our word vectors we've got this simple
[01:05:04] our word vectors we've got this simple task where what we're going to do is
[01:05:07] task where what we're going to do is we're going to look at a word in context
[01:05:09] we're going to look at a word in context because sometimes Paris is a name of a
[01:05:12] because sometimes Paris is a name of a person sometimes it's a location and so
[01:05:15] person sometimes it's a location and so we're going to want to look at this word
[01:05:17] we're going to want to look at this word in its context and say aha this is a
[01:05:20] in its context and say aha this is a name of a location in this instance and
[01:05:24] name of a location in this instance and so the way that we going to do it um is
[01:05:27] so the way that we going to do it um is we're going to form a a window
[01:05:30] we're going to form a a window classifier so we're going to take a word
[01:05:33] classifier so we're going to take a word with a couple of words of context on
[01:05:35] with a couple of words of context on each side and for the words in our
[01:05:38] each side and for the words in our context window we're going to use our
[01:05:40] context window we're going to use our word vectors because we want to show
[01:05:42] word vectors because we want to show they're useful for something and then we
[01:05:43] they're useful for something and then we want to feed this into something that is
[01:05:46] want to feed this into something that is a classifier and our classifier um it's
[01:05:51] a classifier and our classifier um it's actually going to be a really simple loc
[01:05:53] actually going to be a really simple loc classifier we're only here going to do
[01:05:56] classifier we're only here going to do location or not a location so this one
[01:05:59] location or not a location so this one here we're W to say for this um window
[01:06:02] here we're W to say for this um window here yes it's a location um and whereas
[01:06:07] here yes it's a location um and whereas if it had
[01:06:08] if it had been I what I love Paris Hilton greatly
[01:06:13] been I what I love Paris Hilton greatly um then we'd be saying no because Paris
[01:06:16] um then we'd be saying no because Paris the word in the middle of the context
[01:06:19] the word in the middle of the context then isn't um a location so that's sort
[01:06:22] then isn't um a location so that's sort of the idea of a classification or
[01:06:24] of the idea of a classification or classifier we're making assigning some
[01:06:28] classifier we're making assigning some set of classes um to things right so in
[01:06:32] set of classes um to things right so in general for
[01:06:33] general for classifiers um we do supervised learning
[01:06:37] classifiers um we do supervised learning which means we have some labeled
[01:06:39] which means we have some labeled examples our training data set so we
[01:06:42] examples our training data set so we have input items XI and for each one
[01:06:45] have input items XI and for each one we've got a class Yi so I had for my ex
[01:06:50] we've got a class Yi so I had for my ex example training examples ones like I
[01:06:53] example training examples ones like I love Paris Hilton greatly that
[01:06:56] love Paris Hilton greatly that was negative not a location and I visit
[01:07:01] was negative not a location and I visit Paris Every Spring that's positive that
[01:07:04] Paris Every Spring that's positive that is a location where I'm actually
[01:07:05] is a location where I'm actually classifying the middle word okay so
[01:07:08] classifying the middle word okay so inputs labels and in general we've got
[01:07:11] inputs labels and in general we've got um labels are a set of classes so my set
[01:07:14] um labels are a set of classes so my set here is simply location not a location
[01:07:17] here is simply location not a location but I could get fancier and I could say
[01:07:21] but I could get fancier and I could say I've got five classes I've got location
[01:07:23] I've got five classes I've got location person name
[01:07:26] person name uh whatever other ones there are um
[01:07:29] uh whatever other ones there are um company name drug name right I could be
[01:07:32] company name drug name right I could be assigning a bunch of or other not a name
[01:07:36] assigning a bunch of or other not a name a bunch of different classes but I'm
[01:07:37] a bunch of different classes but I'm going to be doing it with only two
[01:07:40] going to be doing it with only two because I'm using this example on next
[01:07:42] because I'm using this example on next Tuesday's lecture as well and I'm
[01:07:44] Tuesday's lecture as well and I'm wanting to keep it simple um so that's
[01:07:47] wanting to keep it simple um so that's what we're going to do and so what we're
[01:07:51] what we're going to do and so what we're going to be using in our class um is
[01:07:54] going to be using in our class um is neural classifiers and so I just wanted
[01:07:57] neural classifiers and so I just wanted to sort of um sort of just sort of go
[01:08:01] to sort of um sort of just sort of go through quickly just the sort of food
[01:08:03] through quickly just the sort of food for thought as we go into it so for a
[01:08:06] for thought as we go into it so for a typical um stats machine learning
[01:08:10] typical um stats machine learning classifier you can build classifiers
[01:08:13] classifier you can build classifiers like logistic regression or softmax
[01:08:15] like logistic regression or softmax classifiers or other ones like support
[01:08:18] classifiers or other ones like support Vector machines or naive Bays or
[01:08:21] Vector machines or naive Bays or whatever else you might have seen the
[01:08:24] whatever else you might have seen the vast majority of these classifiers are
[01:08:27] vast majority of these classifiers are linear classifiers meaning that they
[01:08:29] linear classifiers meaning that they have a linear decision boundary and when
[01:08:32] have a linear decision boundary and when we're learning these classifiers we're
[01:08:34] we're learning these classifiers we're learning parameters here W but our the
[01:08:38] learning parameters here W but our the our inputs are fixed that our inputs are
[01:08:41] our inputs are fixed that our inputs are represented by symbols like um or
[01:08:44] represented by symbols like um or quantities so we have fixed inputs we
[01:08:47] quantities so we have fixed inputs we learn parameters as weights into that
[01:08:50] learn parameters as weights into that are used to multiply the inputs and then
[01:08:53] are used to multiply the inputs and then we use a linear decision boundary
[01:08:56] we use a linear decision boundary so when we have our neural classifier
[01:08:59] so when we have our neural classifier we're kind of getting some more power um
[01:09:02] we're kind of getting some more power um so first of all we're not only learning
[01:09:05] so first of all we're not only learning weights W for our classifier we're also
[01:09:08] weights W for our classifier we're also learning distributed representations for
[01:09:10] learning distributed representations for our words so our words can sort of re
[01:09:15] our words so our words can sort of re represent our word vectors re represent
[01:09:18] represent our word vectors re represent the actual words of symbols and can move
[01:09:20] the actual words of symbols and can move them around in the space so that in
[01:09:23] them around in the space so that in terms of the original space we've got a
[01:09:27] terms of the original space we've got a nonlinear classifier that can represent
[01:09:29] nonlinear classifier that can represent much more complex functions um but we
[01:09:33] much more complex functions um but we will then sort of use the word vectors
[01:09:36] will then sort of use the word vectors to re-represent um those words to do a
[01:09:39] to re-represent um those words to do a final classification so at the end of
[01:09:42] final classification so at the end of our deep Network which we're about to
[01:09:45] our deep Network which we're about to build we will have a linear classifier
[01:09:48] build we will have a linear classifier in terms of our re-represented vectors
[01:09:51] in terms of our re-represented vectors but not in terms of our original space
[01:09:53] but not in terms of our original space let me try and be concrete about that
[01:09:56] let me try and be concrete about that okay so here's what I'm going to use and
[01:09:58] okay so here's what I'm going to use and we'll use again next Tuesday um as my um
[01:10:02] we'll use again next Tuesday um as my um little neural network um and so I start
[01:10:07] little neural network um and so I start with some words museums and Paris are
[01:10:10] with some words museums and Paris are amazing I first of
[01:10:12] amazing I first of all come up with the word embedding of
[01:10:15] all come up with the word embedding of those using my word vectors so now I'm
[01:10:18] those using my word vectors so now I'm got this sort of high dimensional Vector
[01:10:21] got this sort of high dimensional Vector which is just a concatenation of five
[01:10:23] which is just a concatenation of five word vectors so you know if I have 100
[01:10:25] word vectors so you know if I have 100 dimensional word vectors this is 500
[01:10:27] dimensional word vectors this is 500 dimensional and then I'm going to put it
[01:10:29] dimensional and then I'm going to put it through a neural network layer which is
[01:10:32] through a neural network layer which is simply multiplying that vector by a
[01:10:35] simply multiplying that vector by a matrix and adding on a bias um vector
[01:10:40] matrix and adding on a bias um vector and then I'm going to put it through
[01:10:41] and then I'm going to put it through some
[01:10:42] some nonlinearity which might be for example
[01:10:44] nonlinearity which might be for example the logistic function that we've already
[01:10:46] the logistic function that we've already seen so that will give me a new
[01:10:49] seen so that will give me a new representation and in particular um if
[01:10:54] representation and in particular um if the w is say 8 by 500 I'll be reducing
[01:10:58] the w is say 8 by 500 I'll be reducing it to a much or what yeah 8 by 500 I'll
[01:11:02] it to a much or what yeah 8 by 500 I'll be reducing it to a much smaller Vector
[01:11:06] be reducing it to a much smaller Vector right so then I can do after that I can
[01:11:09] right so then I can do after that I can multiply my hidden representation the
[01:11:12] multiply my hidden representation the middle of my neural network by another
[01:11:15] middle of my neural network by another vector and that will give me a score and
[01:11:17] vector and that will give me a score and I'm going to put the score into the
[01:11:19] I'm going to put the score into the logistic um function that we saw earlier
[01:11:21] logistic um function that we saw earlier to say what's the probability this is
[01:11:23] to say what's the probability this is the location so at this point um my
[01:11:28] the location so at this point um my classifier is going to be a linear
[01:11:31] classifier is going to be a linear classifier in terms of this internal
[01:11:34] classifier in terms of this internal representation that's used right at the
[01:11:36] representation that's used right at the end but it's going to be a nonlinear
[01:11:38] end but it's going to be a nonlinear classifier in terms of my word
[01:11:43] vectors
[01:11:45] vectors um
[01:11:46] um okay
[01:11:48] okay uh great um here's one other thing this
[01:11:53] uh great um here's one other thing this is just sort of uh uh a note for learner
[01:11:57] is just sort of uh uh a note for learner ahead um since you want to know this
[01:11:59] ahead um since you want to know this when we start doing the next assignments
[01:12:02] when we start doing the next assignments I mean up until now I've presented
[01:12:04] I mean up until now I've presented everything as you know doing log
[01:12:07] everything as you know doing log likelihood and negative log likelihood
[01:12:10] likelihood and negative log likelihood for building our models um very soon now
[01:12:14] for building our models um very soon now assignment two we're going to be
[01:12:16] assignment two we're going to be starting to do things um with pytorch
[01:12:19] starting to do things um with pytorch and when you um start working out your
[01:12:23] and when you um start working out your um losses with payto watch um what
[01:12:26] um losses with payto watch um what you're going to be wanting to use is
[01:12:28] you're going to be wanting to use is cross entropy loss and so just let me
[01:12:31] cross entropy loss and so just let me quickly say what cross entropy loss is
[01:12:34] quickly say what cross entropy loss is so cross entropy is from information
[01:12:37] so cross entropy is from information Theory so if you have a true probability
[01:12:40] Theory so if you have a true probability distribution p and you're Computing a
[01:12:43] distribution p and you're Computing a probability distribution Q your cross
[01:12:46] probability distribution Q your cross entropy loss is like this um so it's the
[01:12:51] entropy loss is like this um so it's the log of your um model probability
[01:12:56] log of your um model probability um the expectation of that under your um
[01:12:59] um the expectation of that under your um true probability distribution but
[01:13:02] true probability distribution but there's sort of a special case whereas
[01:13:04] there's sort of a special case whereas if you have ground truth or gold or
[01:13:07] if you have ground truth or gold or Target data where things are labeled one
[01:13:12] Target data where things are labeled one zero so like for examples of um I love
[01:13:16] zero so like for examples of um I love Paris um when warm right I'm just
[01:13:21] Paris um when warm right I'm just labeling it one for location probability
[01:13:24] labeling it one for location probability one it's a location probability zero is
[01:13:27] one it's a location probability zero is not a location so if you're just
[01:13:30] not a location so if you're just labeling the right class as probability
[01:13:32] labeling the right class as probability one then in this summation every other
[01:13:36] one then in this summation every other term goes to zero and the only thing
[01:13:40] term goes to zero and the only thing you're left with is what probability is
[01:13:43] you're left with is what probability is my model what log probability is my
[01:13:46] my model what log probability is my model giving to the right class and so
[01:13:50] model giving to the right class and so that then is um your log likelihood
[01:13:54] that then is um your log likelihood which we can use for um the negative log
[01:13:58] which we can use for um the negative log likelihood um little bit of a
[01:14:01] likelihood um little bit of a complication here just remember that you
[01:14:03] complication here just remember that you want to use cross entry loss in py tort
[01:14:06] want to use cross entry loss in py tort when building the model okay um before
[01:14:10] when building the model okay um before we end today um here is um my obligatory
[01:14:13] we end today um here is um my obligatory one picture of human neurons don't miss
[01:14:16] one picture of human neurons don't miss it because I'm not going to show any
[01:14:18] it because I'm not going to show any more of these um okay these are human
[01:14:21] more of these um okay these are human neurons right human neurons um were
[01:14:25] neurons right human neurons um were the um inspiration for neural networks
[01:14:29] the um inspiration for neural networks right so human neurons have a single
[01:14:32] right so human neurons have a single output which comes down this axon um and
[01:14:37] output which comes down this axon um and then um when you have these
[01:14:42] then um when you have these outputs um they then feed into other
[01:14:46] outputs um they then feed into other neurons I guess I don't really have an
[01:14:48] neurons I guess I don't really have an example here but in general one output
[01:14:52] example here but in general one output can feed into multiple different neurons
[01:14:54] can feed into multiple different neurons you can see the different things hanging
[01:14:55] you can see the different things hanging into it so you know you have the output
[01:14:58] into it so you know you have the output connecting to the input and sort of
[01:15:00] connecting to the input and sort of where you make this connection right
[01:15:02] where you make this connection right that's the synapses that people talk
[01:15:05] that's the synapses that people talk about and so one neuron will normally
[01:15:08] about and so one neuron will normally have many many inputs where it picks
[01:15:11] have many many inputs where it picks things up from other neurons and they
[01:15:13] things up from other neurons and they all go into the nucleus of the cell and
[01:15:17] all go into the nucleus of the cell and the nucleus combines together all those
[01:15:20] the nucleus combines together all those inputs and kind of what happens is if
[01:15:23] inputs and kind of what happens is if there's enough positive activation from
[01:15:26] there's enough positive activation from all of these um inputs it then sends
[01:15:31] all of these um inputs it then sends signals down its output now strictly how
[01:15:35] signals down its output now strictly how um neurons work is that they send spikes
[01:15:39] um neurons work is that they send spikes so the level of activation a neuron is
[01:15:41] so the level of activation a neuron is its rate of spiking um but that
[01:15:44] its rate of spiking um but that immediately got turned in artificial
[01:15:46] immediately got turned in artificial neural networks into just a real value
[01:15:50] neural networks into just a real value as to what is its level of activation um
[01:15:53] as to what is its level of activation um and so it does this so this was kind of
[01:15:56] and so it does this so this was kind of the genuine inspiration of all of our
[01:15:59] the genuine inspiration of all of our neural networks right so a binary
[01:16:01] neural networks right so a binary logistic regression is kind of a bit
[01:16:05] logistic regression is kind of a bit similar to a neuron right it has
[01:16:07] similar to a neuron right it has multiple inputs you're working out your
[01:16:11] multiple inputs you're working out your total level of excitation where in
[01:16:14] total level of excitation where in particular you can have inputs that are
[01:16:17] particular you can have inputs that are both exciting positive inputs and inputs
[01:16:21] both exciting positive inputs and inputs that are um negative um which are then
[01:16:25] that are um negative um which are then inhibitory inputs you combine them all
[01:16:27] inhibitory inputs you combine them all together and you get an output that's
[01:16:29] together and you get an output that's your level of exit
[01:16:32] your level of exit excitation and you're then sort of
[01:16:34] excitation and you're then sort of converting that through some
[01:16:36] converting that through some nonlinearity and so this was proposed as
[01:16:39] nonlinearity and so this was proposed as a very simple model of human neurons now
[01:16:43] a very simple model of human neurons now human neurons are way more complex than
[01:16:45] human neurons are way more complex than this um and some people like
[01:16:47] this um and some people like neuroscientists think we maybe should be
[01:16:49] neuroscientists think we maybe should be doing a better model of actual human
[01:16:53] doing a better model of actual human neurons but um
[01:16:55] neurons but um in terms of what's being done in the
[01:16:57] in terms of what's being done in the current neural networks eat the World
[01:16:59] current neural networks eat the World Revolution everyone's forgotten about
[01:17:02] Revolution everyone's forgotten about that and is just sticking with this very
[01:17:04] that and is just sticking with this very very simple model um which conveniently
[01:17:08] very simple model um which conveniently turns into linear algebra in a very
[01:17:11] turns into linear algebra in a very simple way um so this gives us sort of
[01:17:15] simple way um so this gives us sort of like a single neuron but then precise
[01:17:19] like a single neuron but then precise right so this is which this s single
[01:17:22] right so this is which this s single neuron if you use the logistic function
[01:17:24] neuron if you use the logistic function is identical to logistic regression
[01:17:27] is identical to logistic regression which you've probably seen in some stats
[01:17:29] which you've probably seen in some stats class or something but the difference is
[01:17:32] class or something but the difference is that for newal networks we don't just
[01:17:34] that for newal networks we don't just have one logistic regression we have a
[01:17:37] have one logistic regression we have a bunch of logistic regressions at once
[01:17:42] bunch of logistic regressions at once and well that' be tricky if we had to
[01:17:44] and well that' be tricky if we had to Define what each of these logistic
[01:17:46] Define what each of these logistic regressions was calculating but what we
[01:17:50] regressions was calculating but what we don't what we do is we just feed them
[01:17:53] don't what we do is we just feed them into another logistic regression and so
[01:17:57] into another logistic regression and so we have some eventual output that we
[01:18:01] we have some eventual output that we want to be something like we want it to
[01:18:03] want to be something like we want it to say you know this is or isn't a location
[01:18:07] say you know this is or isn't a location but then what will happen is by our
[01:18:10] but then what will happen is by our machine learning these intermediate
[01:18:12] machine learning these intermediate logistic regressions will figure out all
[01:18:15] logistic regressions will figure out all by themselves something useful to do
[01:18:18] by themselves something useful to do that's the magic um right so that you
[01:18:22] that's the magic um right so that you get this sort of self-learning prop
[01:18:25] get this sort of self-learning prop where the model has a lot of parameters
[01:18:28] where the model has a lot of parameters and internally um will work out useful
[01:18:31] and internally um will work out useful things to do so in general we can get
[01:18:33] things to do so in general we can get more Magic by having more layers in the
[01:18:35] more Magic by having more layers in the neural network and that we will um build
[01:18:39] neural network and that we will um build up functions so effectively these
[01:18:41] up functions so effectively these intermediate layers let us learn a model
[01:18:45] intermediate layers let us learn a model that re represents the input data in
[01:18:48] that re represents the input data in ways that will make it easier to
[01:18:51] ways that will make it easier to classify or easier to interpret and do
[01:18:54] classify or easier to interpret and do things with Downstream in our new
[01:18:58] things with Downstream in our new network um and it's time so I should
[01:19:01] network um and it's time so I should stop there thank you

Stanford CS224N: NLP with Deep Learning | Spring 2024 | Lecture 3 - Backpropagation, Neural Network

Source: https://www.youtube.com/watch?v=HnliVHU2g9U

---

Transcript

[00:00:05] okay hi everyone I'll be started okay so
[00:00:09] okay hi everyone I'll be started okay so it's Tuesday of week two so hopefully
[00:00:12] it's Tuesday of week two so hopefully that means everyone has done the
[00:00:15] that means everyone has done the assignment one everyone done assignment
[00:00:17] assignment one everyone done assignment one um you know uh I'm if I'm saying
[00:00:21] one um you know uh I'm if I'm saying this I'm probably saying it to the wrong
[00:00:23] this I'm probably saying it to the wrong people but it it seems like every year
[00:00:27] people but it it seems like every year some people blow some of their um late
[00:00:31] some people blow some of their um late days on assignment one and it's really
[00:00:33] days on assignment one and it's really just the wrong place to use them um so
[00:00:37] just the wrong place to use them um so yeah hopefully you've all done
[00:00:38] yeah hopefully you've all done assignment one um and did not that this
[00:00:41] assignment one um and did not that this is meant to be the easy onramp um and
[00:00:44] is meant to be the easy onramp um and then we go straight on from that so that
[00:00:47] then we go straight on from that so that out today we have assignment two so
[00:00:50] out today we have assignment two so assignment two um has two purposes um
[00:00:55] assignment two um has two purposes um purpose one is to make you do some math
[00:00:59] purpose one is to make you do some math um to um some understanding of what
[00:01:02] um to um some understanding of what newal networks really compute and how
[00:01:05] newal networks really compute and how they compute it and that's what I'm
[00:01:06] they compute it and that's what I'm going to talk about today is also going
[00:01:08] going to talk about today is also going through that math um but then
[00:01:11] through that math um but then simultaneously maybe it does three
[00:01:13] simultaneously maybe it does three things in assignment two um so we're
[00:01:16] things in assignment two um so we're going to be learning something about
[00:01:17] going to be learning something about dependency paing which will be actually
[00:01:20] dependency paing which will be actually something about um language structure
[00:01:22] something about um language structure and Linguistics but then thirdly um for
[00:01:26] and Linguistics but then thirdly um for assignment two we're going to start
[00:01:27] assignment two we're going to start using pytorch so pytorch is is one of
[00:01:30] using pytorch so pytorch is is one of the leading software Frameworks for deep
[00:01:32] the leading software Frameworks for deep learning and the one that we're going to
[00:01:35] learning and the one that we're going to um use for this class um so for p i mean
[00:01:40] um use for this class um so for p i mean the assignment 3 pytorch is
[00:01:45] the assignment 3 pytorch is exceedingly um scaffolded so it's sort
[00:01:48] exceedingly um scaffolded so it's sort of you know here's this thing and you
[00:01:50] of you know here's this thing and you have to write these two lines use these
[00:01:52] have to write these two lines use these two functions um but nevertheless um for
[00:01:56] two functions um but nevertheless um for help people get up to speed and get
[00:01:59] help people get up to speed and get started using pytorch on Friday at 3:30
[00:02:03] started using pytorch on Friday at 3:30 in Gates bo1 or it will again be
[00:02:05] in Gates bo1 or it will again be recorded um we have a tutorial on
[00:02:08] recorded um we have a tutorial on pytorch and so that's a great way to get
[00:02:11] pytorch and so that's a great way to get more of a sense of py torch and how it
[00:02:13] more of a sense of py torch and how it works before doing assignment
[00:02:16] works before doing assignment two um yeah uh the other things yeah so
[00:02:21] two um yeah uh the other things yeah so for nearly all the lectures we've got um
[00:02:24] for nearly all the lectures we've got um further reading of places that you can
[00:02:26] further reading of places that you can look um of all the classes in the attire
[00:02:30] look um of all the classes in the attire um this um for many people might be a
[00:02:35] um this um for many people might be a really good one to look at the suggested
[00:02:37] really good one to look at the suggested readings we have several readings which
[00:02:39] readings we have several readings which are sort of shorter tutorials and
[00:02:42] are sort of shorter tutorials and reviews of the kind of um Matrix
[00:02:45] reviews of the kind of um Matrix calculus um and linear algebra that we
[00:02:48] calculus um and linear algebra that we need for this class um so really
[00:02:51] need for this class um so really encourage you um to look at those um if
[00:02:54] encourage you um to look at those um if you decide that one is your favorite you
[00:02:56] you decide that one is your favorite you can tell us on Ed which one you think is
[00:02:58] can tell us on Ed which one you think is the best one to choose between between
[00:03:00] the best one to choose between between them I kind of like the one that's first
[00:03:02] them I kind of like the one that's first on the list but maybe you'll feel
[00:03:03] on the list but maybe you'll feel differently um yeah um conversely um
[00:03:08] differently um yeah um conversely um yeah so today will be sort of all math
[00:03:11] yeah so today will be sort of all math and then Thursday will be kind of all
[00:03:15] and then Thursday will be kind of all language and Linguistics some people
[00:03:17] language and Linguistics some people find the language and Linguistics hard
[00:03:18] find the language and Linguistics hard as well um so I guess different kinds of
[00:03:21] as well um so I guess different kinds of people um okay so getting straight into
[00:03:25] people um okay so getting straight into it um so where we started last time um
[00:03:30] it um so where we started last time um I'd sort of shown these baby neural
[00:03:32] I'd sort of shown these baby neural networks and sort of said well we can
[00:03:34] networks and sort of said well we can think of each of those orange things as
[00:03:37] think of each of those orange things as basically like a little logistic
[00:03:39] basically like a little logistic regression unit and the crucial
[00:03:42] regression unit and the crucial difference from then the kind of um
[00:03:45] difference from then the kind of um statistics machine learning you see in a
[00:03:47] statistics machine learning you see in a stats class 109 or wherever is that in
[00:03:51] stats class 109 or wherever is that in those you have one logistic regression
[00:03:54] those you have one logistic regression and you're defining the input features
[00:03:56] and you're defining the input features to it and you've got some decision
[00:03:59] to it and you've got some decision variable that you want to un have at the
[00:04:02] variable that you want to un have at the output here you're sort of building
[00:04:04] output here you're sort of building these Cascades of little logistic
[00:04:07] these Cascades of little logistic regressions and so the idea is right at
[00:04:09] regressions and so the idea is right at the end we're going to Define what we
[00:04:11] the end we're going to Define what we want we're going to capture that by our
[00:04:14] want we're going to capture that by our objective function or loss function but
[00:04:16] objective function or loss function but for the stuff in the middle that that
[00:04:19] for the stuff in the middle that that stuff in the middle is going to be a
[00:04:21] stuff in the middle is going to be a chance for the neural network to learn
[00:04:24] chance for the neural network to learn by itself what would be useful inputs to
[00:04:29] by itself what would be useful inputs to further Downstream neurons what kind of
[00:04:34] further Downstream neurons what kind of functions should I come up with in terms
[00:04:36] functions should I come up with in terms of my inputs that will help me provide
[00:04:40] of my inputs that will help me provide um useful outputs to help the final
[00:04:43] um useful outputs to help the final computation down the track um and you
[00:04:47] computation down the track um and you know if you haven't sort of seen and um
[00:04:49] know if you haven't sort of seen and um thought about this much before I mean I
[00:04:52] thought about this much before I mean I think you know it's worth uh sitting
[00:04:55] think you know it's worth uh sitting with that idea for a moment CU this is
[00:04:57] with that idea for a moment CU this is really a super powerful idea which is
[00:05:00] really a super powerful idea which is what's made neural networks more
[00:05:03] what's made neural networks more powerful in most circumstances than
[00:05:06] powerful in most circumstances than other forms of machine learning the fact
[00:05:08] other forms of machine learning the fact that you have this
[00:05:11] that you have this self-organization of intermediate levels
[00:05:13] self-organization of intermediate levels of representation that you use to
[00:05:15] of representation that you use to compute things that will be useful
[00:05:18] compute things that will be useful Downstream for what you eventually want
[00:05:20] Downstream for what you eventually want to
[00:05:21] to do um the other reason I was bring back
[00:05:23] do um the other reason I was bring back up this picture is I've sort of wanted
[00:05:25] up this picture is I've sort of wanted to go straight from here um to matric
[00:05:30] to go straight from here um to matric um so while you could sort of wire
[00:05:34] um so while you could sort of wire together neurons however you wanted to
[00:05:37] together neurons however you wanted to and arguably if you look at human brains
[00:05:40] and arguably if you look at human brains they look more like neurons wired
[00:05:42] they look more like neurons wired together however you wanted to but for
[00:05:44] together however you wanted to but for what's done with neural networks
[00:05:46] what's done with neural networks basically there's always this kind of
[00:05:48] basically there's always this kind of regular structure of layers so once we
[00:05:51] regular structure of layers so once we have this regular structure of layers we
[00:05:54] have this regular structure of layers we are taking um the output of one of our
[00:06:00] are taking um the output of one of our neurons at one layer and we're feeding
[00:06:03] neurons at one layer and we're feeding them together with weights um to produce
[00:06:07] them together with weights um to produce um the inputs to the next layer so we're
[00:06:10] um the inputs to the next layer so we're taking the X1 X2 X3 outputs we're
[00:06:13] taking the X1 X2 X3 outputs we're multiplying them all by weights we're
[00:06:15] multiplying them all by weights we're adding a bias
[00:06:18] adding a bias term um and then we're going to put it
[00:06:20] term um and then we're going to put it through a nonlinearity and that will
[00:06:22] through a nonlinearity and that will give us the value at the next layer so
[00:06:24] give us the value at the next layer so if we then kind of collapse that to a
[00:06:27] if we then kind of collapse that to a vector and this to a vector
[00:06:30] vector and this to a vector that then collapses into a computation
[00:06:32] that then collapses into a computation that first of all we're doing a matrix
[00:06:36] that first of all we're doing a matrix multiplication we're calculating WX of
[00:06:40] multiplication we're calculating WX of the inputs and then we're adding on the
[00:06:42] the inputs and then we're adding on the biases as a a vector of biases which
[00:06:45] biases as a a vector of biases which gives us this intermediate value Z and
[00:06:48] gives us this intermediate value Z and then we have this nonlinearity or
[00:06:52] then we have this nonlinearity or activation function which is applied um
[00:06:55] activation function which is applied um to that which gives us the values in the
[00:06:58] to that which gives us the values in the next layer of the new Network and the
[00:07:01] next layer of the new Network and the activation function is applied to a
[00:07:03] activation function is applied to a vector and produces a vector but it's
[00:07:05] vector and produces a vector but it's operating on each of the individual
[00:07:08] operating on each of the individual components of that Vector one at a time
[00:07:10] components of that Vector one at a time so we've got some Scala function that
[00:07:13] so we've got some Scala function that we're just applying to each element of
[00:07:15] we're just applying to each element of the
[00:07:15] the vector um and so that's the kind of
[00:07:19] vector um and so that's the kind of picture um we saw when I did this
[00:07:23] picture um we saw when I did this example and I'm going to continue to use
[00:07:25] example and I'm going to continue to use this example in today's class remember
[00:07:27] this example in today's class remember we were going to decide whether the word
[00:07:29] we were going to decide whether the word in the middle of the input window was a
[00:07:32] in the middle of the input window was a location or not and so we were doing the
[00:07:35] location or not and so we were doing the matrix multiplication putting it through
[00:07:38] matrix multiplication putting it through the nonlinearity we're then just doing a
[00:07:40] the nonlinearity we're then just doing a DOT product here and then we're going
[00:07:43] DOT product here and then we're going that got stuck into um uh sigmoid um to
[00:07:48] that got stuck into um uh sigmoid um to predict yes or no um and the final thing
[00:07:52] predict yes or no um and the final thing I wanted to say a little bit about is
[00:07:55] I wanted to say a little bit about is these FS the
[00:07:56] these FS the nonlinearity or the activation function
[00:07:59] nonlinearity or the activation function and where did they come in well the
[00:08:02] and where did they come in well the starting point of where they came in in
[00:08:04] starting point of where they came in in the history of newal networks is when
[00:08:07] the history of newal networks is when people came up with this idea that well
[00:08:10] people came up with this idea that well you could represent um the operation of
[00:08:13] you could represent um the operation of a basic neuron by doing a matrix
[00:08:17] a basic neuron by doing a matrix multiplication of the inputs and then
[00:08:21] multiplication of the inputs and then having a bias term or here represents a
[00:08:24] having a bias term or here represents a threshold term to see whether the
[00:08:27] threshold term to see whether the neurons should fire or not that was
[00:08:30] neurons should fire or not that was actually in the very first
[00:08:31] actually in the very first implementation which B dates back to the
[00:08:34] implementation which B dates back to the 1940s as done as a threshold right so
[00:08:38] 1940s as done as a threshold right so that if um the activation was greater
[00:08:42] that if um the activation was greater than Theta you output one otherwise you
[00:08:45] than Theta you output one otherwise you output zero and well if you have a
[00:08:48] output zero and well if you have a threshold um the the two lines are flat
[00:08:52] threshold um the the two lines are flat right so there is no slope there is no
[00:08:55] right so there is no slope there is no gradient um so that's actually makes
[00:08:58] gradient um so that's actually makes learning much harder so the whole secret
[00:09:02] learning much harder so the whole secret of what we build with newal networks and
[00:09:05] of what we build with newal networks and in an alternative name that's um popular
[00:09:08] in an alternative name that's um popular in some circles these days is
[00:09:11] in some circles these days is gradient-based learning and the entire
[00:09:14] gradient-based learning and the entire idea of gradient based learning is if we
[00:09:17] idea of gradient based learning is if we actually have some slopes um then it's
[00:09:19] actually have some slopes um then it's like going skiing during spring break
[00:09:22] like going skiing during spring break you can work out where it's steeper and
[00:09:24] you can work out where it's steeper and you can head down where it's steeper um
[00:09:27] you can head down where it's steeper um and that will allow us to op imiz our
[00:09:30] and that will allow us to op imiz our function to learn much more quickly and
[00:09:34] function to learn much more quickly and so that's one reason that we don't just
[00:09:36] so that's one reason that we don't just want to have threshold units we want to
[00:09:39] want to have threshold units we want to have things with slopes so we um have
[00:09:42] have things with slopes so we um have gradient so in subsequent work people
[00:09:45] gradient so in subsequent work people started using activation functions with
[00:09:49] started using activation functions with slopes um and so the first popular one
[00:09:53] slopes um and so the first popular one um was this sigmoidal logistic that
[00:09:55] um was this sigmoidal logistic that we've seen for mapping to probabilities
[00:09:58] we've seen for mapping to probabilities but you know it's sort of imperfect it
[00:10:01] but you know it's sort of imperfect it seemed because you know the output was
[00:10:03] seemed because you know the output was always non- negative so that sort of
[00:10:06] always non- negative so that sort of tends to push things towards bigger
[00:10:08] tends to push things towards bigger numbers so um there was quite a bit of
[00:10:12] numbers so um there was quite a bit of use then of this tan H function um and
[00:10:17] use then of this tan H function um and you'll actually see tan H when we do
[00:10:19] you'll actually see tan H when we do assignment three we'll be using tan H's
[00:10:21] assignment three we'll be using tan H's in our current new networks um and so um
[00:10:27] in our current new networks um and so um I've written there um the formula
[00:10:29] I've written there um the formula usually give for Tan H in terms of
[00:10:32] usually give for Tan H in terms of exponentials um yeah if your math is
[00:10:35] exponentials um yeah if your math is Rusty it's not obvious that tan H and
[00:10:37] Rusty it's not obvious that tan H and logistic have much to do with each other
[00:10:39] logistic have much to do with each other um but if you want to treat this as a
[00:10:42] um but if you want to treat this as a math problem um that a 10 H is literally
[00:10:46] math problem um that a 10 H is literally just a rescaled logistic you're
[00:10:48] just a rescaled logistic you're stretching it by two and moving it down
[00:10:50] stretching it by two and moving it down by one it's the same function
[00:10:54] by one it's the same function um okay um but you know so that's nice
[00:10:58] um okay um but you know so that's nice but if you're calculating 10 hes you
[00:11:00] but if you're calculating 10 hes you have to do all of these exponentials and
[00:11:03] have to do all of these exponentials and you know exponentials are kind of slow
[00:11:05] you know exponentials are kind of slow on your computer and things like that
[00:11:07] on your computer and things like that you might wonder whether you couldn't
[00:11:09] you might wonder whether you couldn't get away with something much cheaper and
[00:11:12] get away with something much cheaper and so people thought about that and thought
[00:11:14] so people thought about that and thought oh maybe we could just use a so-called
[00:11:16] oh maybe we could just use a so-called hard tan H um where it has a slope of
[00:11:20] hard tan H um where it has a slope of one in the middle and is then just flat
[00:11:22] one in the middle and is then just flat outside that area and you know that
[00:11:25] outside that area and you know that seemed to work in many cases and so that
[00:11:28] seemed to work in many cases and so that then led to the popularity of the
[00:11:31] then led to the popularity of the rectified linear unit um so the
[00:11:34] rectified linear unit um so the rectified linear unit is simply zero on
[00:11:37] rectified linear unit is simply zero on the negative region and then is yal x in
[00:11:40] the negative region and then is yal x in the positive region now this seems kind
[00:11:44] the positive region now this seems kind of wonky and goes against what I was
[00:11:47] of wonky and goes against what I was saying about gradient based learning
[00:11:48] saying about gradient based learning because once you're in the negative
[00:11:50] because once you're in the negative region there's no gradient um you're
[00:11:52] region there's no gradient um you're just dead um but in the positive region
[00:11:56] just dead um but in the positive region there is gradient and the gradient is
[00:11:59] there is gradient and the gradient is particularly simple right the slope is
[00:12:01] particularly simple right the slope is always one um and so you know this still
[00:12:06] always one um and so you know this still feels slightly perverse to me but you
[00:12:09] feels slightly perverse to me but you know this really became the norm of what
[00:12:12] know this really became the norm of what people use for a number of years because
[00:12:15] people use for a number of years because people found that although for an
[00:12:18] people found that although for an individual neuron it was dead half the
[00:12:20] individual neuron it was dead half the time anytime it went negative that
[00:12:22] time anytime it went negative that overall for your newal network some
[00:12:24] overall for your newal network some things would be alive so it kind of gave
[00:12:27] things would be alive so it kind of gave sort of a form of specializ ation and
[00:12:30] sort of a form of specializ ation and the fact that the slope was always one
[00:12:32] the fact that the slope was always one meant that you got um really easy
[00:12:36] meant that you got um really easy productive backward flow of gradients in
[00:12:38] productive backward flow of gradients in a way we'll talk about later and so um
[00:12:42] a way we'll talk about later and so um learning with realu turned to be out to
[00:12:45] learning with realu turned to be out to be very effective and people started
[00:12:48] be very effective and people started using the realu nonlinearity everywhere
[00:12:51] using the realu nonlinearity everywhere and it sort of became the default in the
[00:12:53] and it sort of became the default in the norm and you'll see us using it in the
[00:12:55] norm and you'll see us using it in the assignments in particular we use it in
[00:12:57] assignments in particular we use it in assignment to and so get to see that it
[00:13:00] assignment to and so get to see that it works but nevertheless at some point
[00:13:03] works but nevertheless at some point people sort of had second thoughts and
[00:13:06] people sort of had second thoughts and decided you know having a dead over half
[00:13:08] decided you know having a dead over half of its range maybe isn't such a good
[00:13:11] of its range maybe isn't such a good idea after all even though it seemed to
[00:13:12] idea after all even though it seemed to work great for a few years and so a lot
[00:13:15] work great for a few years and so a lot of what's happened since then is then to
[00:13:18] of what's happened since then is then to come up with other functions which are
[00:13:20] come up with other functions which are in some sense Ru like but not actually
[00:13:24] in some sense Ru like but not actually dead um so
[00:13:27] dead um so um um
[00:13:29] um um okay I don't really yeah uh here we go
[00:13:33] okay I don't really yeah uh here we go enough so one one version of that is the
[00:13:36] enough so one one version of that is the socalled Leaky value so for the Leaky
[00:13:39] socalled Leaky value so for the Leaky value you make the negative half a
[00:13:42] value you make the negative half a straight line as well with a very minor
[00:13:44] straight line as well with a very minor slope but still it's got a little bit of
[00:13:46] slope but still it's got a little bit of slope um there is then a variant of that
[00:13:48] slope um there is then a variant of that called the parametric value where you
[00:13:51] called the parametric value where you have one extra parameter which is
[00:13:53] have one extra parameter which is actually what the slope of the ne the
[00:13:55] actually what the slope of the ne the negative part is and people showed some
[00:13:58] negative part is and people showed some positive result with that um more
[00:14:00] positive result with that um more recently again and this is what you
[00:14:03] recently again and this is what you often see in recent Transformer models
[00:14:06] often see in recent Transformer models um is you see um nonlinearities like
[00:14:10] um is you see um nonlinearities like Swiss swis and Jello so both of these
[00:14:14] Swiss swis and Jello so both of these are sort of fancy functions but kind of
[00:14:17] are sort of fancy functions but kind of what they both look like is basically
[00:14:20] what they both look like is basically this is yal X to all intense and
[00:14:23] this is yal X to all intense and purposes not quite but approximately and
[00:14:25] purposes not quite but approximately and then you got sort of some funky bit of
[00:14:27] then you got sort of some funky bit of curve down here which again gives you a
[00:14:29] curve down here which again gives you a bit of slope um it's sort of the curve
[00:14:31] bit of slope um it's sort of the curve is going the opposite way that's sort of
[00:14:33] is going the opposite way that's sort of a bit funny but they seem to work well
[00:14:35] a bit funny but they seem to work well commonly used um in recent Transformer
[00:14:39] commonly used um in recent Transformer models um so you know that's a bit of a
[00:14:42] models um so you know that's a bit of a dump of all the nonlinearities people
[00:14:45] dump of all the nonlinearities people use I mean the details of that aren't
[00:14:48] use I mean the details of that aren't super important right now but the
[00:14:51] super important right now but the important thing to um have in your head
[00:14:54] important thing to um have in your head is why do we need nonlinearities and the
[00:14:58] is why do we need nonlinearities and the way to think about about that is that
[00:15:01] way to think about about that is that what we're doing with neural networks is
[00:15:03] what we're doing with neural networks is function approximation there's some very
[00:15:06] function approximation there's some very complex function that we want to learn
[00:15:08] complex function that we want to learn you know like maybe we want to go from a
[00:15:10] you know like maybe we want to go from a piece of text to its meaning or we want
[00:15:12] piece of text to its meaning or we want to be interpreting visual scenes or
[00:15:15] to be interpreting visual scenes or something like that um and so we want to
[00:15:18] something like that um and so we want to build really good function
[00:15:20] build really good function approximators and well if you're just
[00:15:23] approximators and well if you're just doing Matrix multiplies a matrix
[00:15:25] doing Matrix multiplies a matrix multiply of a vector is a linear
[00:15:27] multiply of a vector is a linear transform so that doesn't let you
[00:15:30] transform so that doesn't let you multiply complex functions I guess
[00:15:32] multiply complex functions I guess strictly if you put a bias on the end
[00:15:34] strictly if you put a bias on the end it's then an aine transform but let's
[00:15:36] it's then an aine transform but let's keep it simple linear transforms right
[00:15:39] keep it simple linear transforms right so if you're doing multiple if you're
[00:15:41] so if you're doing multiple if you're doing multiple Matrix multiplies you're
[00:15:44] doing multiple Matrix multiplies you're doing multiple linear transforms but
[00:15:46] doing multiple linear transforms but they compose so you could have just
[00:15:48] they compose so you could have just multiplied these two matrices together
[00:15:50] multiplied these two matrices together and you'd have a single linear transform
[00:15:52] and you'd have a single linear transform so you get no power in terms of
[00:15:56] so you get no power in terms of representation by having multi-layer
[00:15:59] representation by having multi-layer networks that are just Matrix multiplies
[00:16:02] networks that are just Matrix multiplies you know as in a little aside in terms
[00:16:05] you know as in a little aside in terms of representational power having
[00:16:08] of representational power having multi-layer Matrix multiplies gives you
[00:16:11] multi-layer Matrix multiplies gives you no power but if you think about in terms
[00:16:13] no power but if you think about in terms of learning actually it does give you
[00:16:15] of learning actually it does give you some power so in the theoretical
[00:16:18] some power so in the theoretical Community looking at newal networks
[00:16:19] Community looking at newal networks there actually quite a few papers that
[00:16:21] there actually quite a few papers that look at linear newal networks meaning
[00:16:23] look at linear newal networks meaning that they're just sequences of the
[00:16:25] that they're just sequences of the multiplies with no nonlinearities
[00:16:28] multiplies with no nonlinearities because they have interesting learning
[00:16:30] because they have interesting learning properties even though they give you no
[00:16:32] properties even though they give you no representational power um okay but we'd
[00:16:35] representational power um okay but we'd like to be able to learn functions like
[00:16:37] like to be able to learn functions like this not only functions like this and to
[00:16:40] this not only functions like this and to be able to learn functions like this we
[00:16:42] be able to learn functions like this we need more than linear transforms and we
[00:16:45] need more than linear transforms and we achieve those by having something that
[00:16:48] achieve those by having something that makes us be calculating a nonlinear
[00:16:51] makes us be calculating a nonlinear function and it's these activation
[00:16:54] function and it's these activation functions that give us nonlinear
[00:16:56] functions that give us nonlinear functions okay cool
[00:17:00] functions okay cool um okay so then getting on to today so
[00:17:04] um okay so then getting on to today so the whole thing we want to do now is
[00:17:06] the whole thing we want to do now is gradient based learning right this is
[00:17:08] gradient based learning right this is our stochastic gradient thecenter
[00:17:10] our stochastic gradient thecenter equation where here you know that upside
[00:17:14] equation where here you know that upside down triangle symbol right that's our
[00:17:16] down triangle symbol right that's our gradient we're wanting to work out the
[00:17:18] gradient we're wanting to work out the slope of our objective function and so
[00:17:23] slope of our objective function and so this is how we're going to learn by
[00:17:25] this is how we're going to learn by calculating gradients so what we want to
[00:17:27] calculating gradients so what we want to know is how do we calculate the
[00:17:29] know is how do we calculate the gradients for an arbitrary function and
[00:17:32] gradients for an arbitrary function and so what I want to do today is first of
[00:17:35] so what I want to do today is first of all do this by hand um for math um and
[00:17:40] all do this by hand um for math um and then discuss you know how do we do it
[00:17:43] then discuss you know how do we do it computationally um which is effectively
[00:17:46] computationally um which is effectively um the famous thing that's taken as
[00:17:48] um the famous thing that's taken as powering um underpowering all of neuron
[00:17:50] powering um underpowering all of neuron Nets which is the back propagation
[00:17:52] Nets which is the back propagation algorithm but the back propagation
[00:17:54] algorithm but the back propagation algorithm is just automating the math
[00:17:57] algorithm is just automating the math okay and so for the math it's Matrix
[00:18:01] okay and so for the math it's Matrix calculus and at this point then there's
[00:18:03] calculus and at this point then there's a huge Spectrum um between people who
[00:18:06] a huge Spectrum um between people who know much more math than me and people
[00:18:08] know much more math than me and people who barely ever learned this um but you
[00:18:11] who barely ever learned this um but you know uh I hope to explain the
[00:18:14] know uh I hope to explain the essentials um or remind people of them
[00:18:18] essentials um or remind people of them um enough that you're at least at a
[00:18:20] um enough that you're at least at a starting point um for reading some other
[00:18:23] starting point um for reading some other stuff and doing homework too so um let's
[00:18:26] stuff and doing homework too so um let's get into that and so I'm going to spend
[00:18:28] get into that and so I'm going to spend about half the time on those two halves
[00:18:31] about half the time on those two halves um and you know the hope is that after
[00:18:33] um and you know the hope is that after this you'll feel like oh I actually
[00:18:35] this you'll feel like oh I actually understand how new networks work under
[00:18:37] understand how new networks work under the hood fingers crossed
[00:18:40] the hood fingers crossed okay so here we go so if you're a
[00:18:43] okay so here we go so if you're a Stanford student um you maybe did math
[00:18:47] Stanford student um you maybe did math 51 um or else you could have done math
[00:18:50] 51 um or else you could have done math 51 which teaches linear algebra
[00:18:53] 51 which teaches linear algebra multivariate calculus and modern
[00:18:55] multivariate calculus and modern applications um math 51 covers every
[00:18:59] applications um math 51 covers every everything I'm going to talk about and
[00:19:00] everything I'm going to talk about and way more stuff so if you actually know
[00:19:02] way more stuff so if you actually know that and remember it you can um look at
[00:19:05] that and remember it you can um look at Instagram for the next 35 minutes um but
[00:19:08] Instagram for the next 35 minutes um but I think the problem is um that you know
[00:19:12] I think the problem is um that you know quite apart from the fact a lot of
[00:19:13] quite apart from the fact a lot of people do it as Frost um you know this
[00:19:16] people do it as Frost um you know this is a lot to get through in 10 weeks and
[00:19:19] is a lot to get through in 10 weeks and I think that a lot of the people who do
[00:19:21] I think that a lot of the people who do this class sort of by two years later
[00:19:23] this class sort of by two years later don't really have much ability to use
[00:19:25] don't really have much ability to use any of it um but you know if you
[00:19:27] any of it um but you know if you actually looked at this book really
[00:19:29] actually looked at this book really Harden for a very really long time you
[00:19:31] Harden for a very really long time you would have discovered that actually
[00:19:34] would have discovered that actually right towards the end of the book in
[00:19:36] right towards the end of the book in appendix G um there's actually an
[00:19:38] appendix G um there's actually an appendix on newal networks and the
[00:19:40] appendix on newal networks and the multivariable chain roll which is
[00:19:43] multivariable chain roll which is precisely what we're going to be using
[00:19:45] precisely what we're going to be using um for um doing our neural networks um
[00:19:49] um for um doing our neural networks um but there are only two problems one
[00:19:51] but there are only two problems one problem is that this is Page 697 of the
[00:19:54] problem is that this is Page 697 of the book and I'm not sure anyone ever gets
[00:19:56] book and I'm not sure anyone ever gets that far and the problem is even if you
[00:20:00] that far and the problem is even if you do get that far you know I don't know I
[00:20:03] do get that far you know I don't know I find these pages that they're really
[00:20:05] find these pages that they're really dense texty Pages it's not even easy to
[00:20:08] dense texty Pages it's not even easy to understand them if you have gone there
[00:20:10] understand them if you have gone there so here's my attempt on that um so the
[00:20:13] so here's my attempt on that um so the Mantra to have in your head is G if I
[00:20:17] Mantra to have in your head is G if I can remember basic single variable
[00:20:20] can remember basic single variable calculus you know that I've got 3x squ
[00:20:23] calculus you know that I've got 3x squ and the derivative of that is 6X that's
[00:20:26] and the derivative of that is 6X that's all you sort of need to know right the
[00:20:28] all you sort of need to know right the ENT is multivariable calculus is just
[00:20:32] ENT is multivariable calculus is just like single variable calculus except
[00:20:35] like single variable calculus except you're using
[00:20:36] you're using matrices okay so that's our article of
[00:20:38] matrices okay so that's our article of faith and we're going to do that and so
[00:20:41] faith and we're going to do that and so what we're wanting to do is to do Matrix
[00:20:44] what we're wanting to do is to do Matrix calculus or the generalization of that
[00:20:46] calculus or the generalization of that tensor calculus sort of using um vectors
[00:20:49] tensor calculus sort of using um vectors matrices and higher order tensors
[00:20:52] matrices and higher order tensors because if we can do things and what's
[00:20:54] because if we can do things and what's referred to as vectorized gradients in
[00:20:56] referred to as vectorized gradients in the neural network world that that will
[00:20:59] the neural network world that that will be sort of the fast efficient way to do
[00:21:02] be sort of the fast efficient way to do our operations you know so if you want
[00:21:05] our operations you know so if you want to think it all through you can do it
[00:21:07] to think it all through you can do it single variable at a time and check that
[00:21:10] single variable at a time and check that you're doing the right thing and I sort
[00:21:12] you're doing the right thing and I sort of tried to indicate that in the first
[00:21:13] of tried to indicate that in the first lecture but if we want to have our
[00:21:15] lecture but if we want to have our networks go room we want to be doing
[00:21:18] networks go room we want to be doing Matrix
[00:21:20] Matrix calculus okay so let's work up to doing
[00:21:23] calculus okay so let's work up to doing that okay so this is the part that I I
[00:21:27] that okay so this is the part that I I trust everyone can remember right so we
[00:21:30] trust everyone can remember right so we have f ofx = x cubed and we can do
[00:21:34] have f ofx = x cubed and we can do single variable um derivative and the
[00:21:40] single variable um derivative and the derivative is
[00:21:42] derivative is 3x² everyone remember that one okay
[00:21:46] 3x² everyone remember that one okay that's something we can all start from
[00:21:48] that's something we can all start from and remember this derivative is saying
[00:21:50] and remember this derivative is saying the slope of things right so the slope
[00:21:53] the slope of things right so the slope of things lets us work out where is
[00:21:56] of things lets us work out where is something steep so we'll be able to go
[00:21:58] something steep so we'll be able to go skiing right that's our goal right and
[00:22:02] skiing right that's our goal right and so you can think of the slope of things
[00:22:05] so you can think of the slope of things as how much the output will change if we
[00:22:08] as how much the output will change if we change the input a bit right that's our
[00:22:10] change the input a bit right that's our measure of um steepness right so um that
[00:22:14] measure of um steepness right so um that so since the derivative is 3x^2 if we're
[00:22:17] so since the derivative is 3x^2 if we're at x = 1 that means the slope is about 3
[00:22:21] at x = 1 that means the slope is about 3 * 1 S 3 so if I work out the value of
[00:22:26] * 1 S 3 so if I work out the value of the function for 1.01 it's gone up by
[00:22:29] the function for 1.01 it's gone up by about three times point I move the X by
[00:22:32] about three times point I move the X by 0.01 and the output moved by 0.03 where
[00:22:36] 0.01 and the output moved by 0.03 where if I go to x = 4 the derivative is 3 *
[00:22:40] if I go to x = 4 the derivative is 3 * 4^2 is 48 and so if I work out the value
[00:22:43] 4^2 is 48 and so if I work out the value of the function at 4.01 I get
[00:22:46] of the function at 4.01 I get approximately
[00:22:47] approximately 64.4 versus 64 right that small
[00:22:51] 64.4 versus 64 right that small difference from 4 to 4.01 has been
[00:22:54] difference from 4 to 4.01 has been magnified 48 times in the output okay
[00:22:58] magnified 48 times in the output okay okay so now we just uh remember the
[00:23:02] okay so now we just uh remember the Mantra it's going to be exactly the same
[00:23:04] Mantra it's going to be exactly the same single value calculus um but with more
[00:23:08] single value calculus um but with more stuff so if we have a function with n
[00:23:11] stuff so if we have a function with n inputs we're then going to work out its
[00:23:14] inputs we're then going to work out its gradient um which is its partial
[00:23:17] gradient um which is its partial derivative with respect to each input so
[00:23:20] derivative with respect to each input so its gradient will now be a vector of the
[00:23:24] its gradient will now be a vector of the same size as the number of inputs um and
[00:23:27] same size as the number of inputs um and there's this funky symb
[00:23:29] there's this funky symb um which people pronounce various ways I
[00:23:32] um which people pronounce various ways I mean you know this kind of originated
[00:23:34] mean you know this kind of originated some kind of someone's weird way of
[00:23:36] some kind of someone's weird way of drawing a calligraphic d right so it is
[00:23:38] drawing a calligraphic d right so it is really a D um so I think I'll mainly
[00:23:41] really a D um so I think I'll mainly just call it D but sometimes people call
[00:23:43] just call it D but sometimes people call it partial or funy D or some some other
[00:23:46] it partial or funy D or some some other name right so you have DF dx1 DF dx2 for
[00:23:50] name right so you have DF dx1 DF dx2 for each of the variables okay so if we go
[00:23:53] each of the variables okay so if we go beyond that um and then have a function
[00:23:57] beyond that um and then have a function with um n inputs and M outputs what we
[00:24:03] with um n inputs and M outputs what we then get for um the gradient is what's
[00:24:07] then get for um the gradient is what's referred to as the Jacobian now actually
[00:24:11] referred to as the Jacobian now actually um the dude this is named after was a
[00:24:13] um the dude this is named after was a German Jew so it should really be yakobi
[00:24:18] German Jew so it should really be yakobi um but no one says that in this country
[00:24:20] um but no one says that in this country um Jacobian um okay so the Jacobian is
[00:24:26] um Jacobian um okay so the Jacobian is then a matrix of partial
[00:24:29] then a matrix of partial derivatives um where you're working out
[00:24:31] derivatives um where you're working out for each output and each input the
[00:24:35] for each output and each input the partial derivative between the component
[00:24:37] partial derivative between the component of the input and the output so this
[00:24:41] of the input and the output so this looks like the kind of thing that we're
[00:24:43] looks like the kind of thing that we're going to have when we have a neural
[00:24:45] going to have when we have a neural network layer because we're going to
[00:24:47] network layer because we're going to have um n inputs and M outputs for the
[00:24:50] have um n inputs and M outputs for the two layers of our neural networks so
[00:24:52] two layers of our neural networks so we'll be using these kind of
[00:24:56] we'll be using these kind of jacobians okay um so then you know the
[00:24:59] jacobians okay um so then you know the whole idea of newal networks is we've
[00:25:01] whole idea of newal networks is we've got these
[00:25:04] got these multi-level
[00:25:05] multi-level computations and they're going to
[00:25:07] computations and they're going to correspond to composition of functions
[00:25:10] correspond to composition of functions so we need to know how to compose things
[00:25:14] so we need to know how to compose things both for calculating functions and for
[00:25:17] both for calculating functions and for calculating their gradients so if we
[00:25:20] calculating their gradients so if we have a one variable function and we want
[00:25:23] have a one variable function and we want to um work out um its derivative in
[00:25:27] to um work out um its derivative in terms of a composition of two functions
[00:25:31] terms of a composition of two functions what we're doing is multiplying um our
[00:25:35] what we're doing is multiplying um our computations okay so um if you compose
[00:25:39] computations okay so um if you compose together um Z of Y um that's the
[00:25:42] together um Z of Y um that's the function that we did at the beginning
[00:25:44] function that we did at the beginning that gives you oh was it no it's not
[00:25:47] that gives you oh was it no it's not sorry it's different okay Z of Y gives
[00:25:50] sorry it's different okay Z of Y gives you
[00:25:51] you 3x2 right and so we know that the
[00:25:53] 3x2 right and so we know that the derivative of that is um
[00:25:56] derivative of that is um 6X okay if we do it in terms of the
[00:26:00] 6X okay if we do it in terms of the pieces we can work out DZ
[00:26:04] pieces we can work out DZ Dy um which is just going to be three
[00:26:08] Dy um which is just going to be three and um Dy DX is 2X and we can work out
[00:26:13] and um Dy DX is 2X and we can work out the total derivative by multiplying
[00:26:15] the total derivative by multiplying these two pieces and we get 6X the same
[00:26:18] these two pieces and we get 6X the same answer right um so um Matrix calculus is
[00:26:23] answer right um so um Matrix calculus is exactly like single variable calculus
[00:26:26] exactly like single variable calculus except we're using um tensors of
[00:26:29] except we're using um tensors of different um so the word tensor is used
[00:26:32] different um so the word tensor is used to mean as you go up that Spectrum in in
[00:26:35] to mean as you go up that Spectrum in in its size so from sort of Scala to Vector
[00:26:37] its size so from sort of Scala to Vector to Matrix to then you know what in
[00:26:41] to Matrix to then you know what in computer science is normally still uh is
[00:26:43] computer science is normally still uh is multidimensional arrays um that spectrum
[00:26:47] multidimensional arrays um that spectrum is then tensors of different um
[00:26:50] is then tensors of different um Dimensions okay so um when we have
[00:26:54] Dimensions okay so um when we have multiple variable functions we're going
[00:26:56] multiple variable functions we're going to multiply jacobians so here we have a
[00:26:59] to multiply jacobians so here we have a function WX + B and then we um compose
[00:27:04] function WX + B and then we um compose um the nonlinearity F to get H and so
[00:27:09] um the nonlinearity F to get H and so we're going to be able to compute that
[00:27:11] we're going to be able to compute that in the same way as a product of these
[00:27:14] in the same way as a product of these partial derivatives which are
[00:27:18] partial derivatives which are jacobians okay so let's start looking at
[00:27:21] jacobians okay so let's start looking at a few examples of what we get so let's
[00:27:24] a few examples of what we get so let's count with start with an element wise
[00:27:26] count with start with an element wise activation function so when we have a a
[00:27:31] activation function so when we have a a vector that's being calculated as the
[00:27:34] vector that's being calculated as the activation function of a previously
[00:27:36] activation function of a previously computed quantity well we're Computing
[00:27:39] computed quantity well we're Computing that component wise as I explained
[00:27:41] that component wise as I explained before so hi equals F of Z and where the
[00:27:46] before so hi equals F of Z and where the sort of f is our activation function
[00:27:48] sort of f is our activation function that actually applies to a scalar but
[00:27:51] that actually applies to a scalar but you know overall this layer is a
[00:27:53] you know overall this layer is a function with n outputs and N inputs and
[00:27:56] function with n outputs and N inputs and so it's going to have an N by n Jacobian
[00:27:59] so it's going to have an N by n Jacobian and well what that's going to so this is
[00:28:01] and well what that's going to so this is our definition of the Jacobian um but in
[00:28:06] our definition of the Jacobian um but in this case this is sort of a special case
[00:28:09] this case this is sort of a special case because if I equals J um then we're
[00:28:14] because if I equals J um then we're going to have um the output um
[00:28:19] going to have um the output um J the HJ depending on Z and otherwise
[00:28:24] J the HJ depending on Z and otherwise it's going to be zero CU for the off
[00:28:26] it's going to be zero CU for the off diagonal entries it doesn't matter how
[00:28:29] diagonal entries it doesn't matter how you change the value it's not changing
[00:28:31] you change the value it's not changing the output because the output only
[00:28:33] the output because the output only depends on the corresponding index and
[00:28:36] depends on the corresponding index and so what we're going to get for this
[00:28:37] so what we're going to get for this Jacobian of activation functions is a
[00:28:40] Jacobian of activation functions is a matrix where everything is zero apart
[00:28:43] matrix where everything is zero apart from the diagonal terms um that
[00:28:46] from the diagonal terms um that correspond to where we're calculating
[00:28:49] correspond to where we're calculating the activation function and for those
[00:28:52] the activation function and for those ones we're going to have to work out how
[00:28:54] ones we're going to have to work out how to compute the derivative of our
[00:28:57] to compute the derivative of our activation function that was on
[00:29:00] activation function that was on assignment one one of the questions on
[00:29:02] assignment one one of the questions on assignment one I do believe or or was it
[00:29:05] assignment one I do believe or or was it on assignment two no no it's assignment
[00:29:07] on assignment two no no it's assignment two one of the questions on assignment
[00:29:09] two one of the questions on assignment two I got that wrong one of the ones on
[00:29:10] two I got that wrong one of the ones on the new assignment is say hey um can you
[00:29:13] the new assignment is say hey um can you work out um the derivative of a logistic
[00:29:16] work out um the derivative of a logistic function well then we'd be a to plug
[00:29:18] function well then we'd be a to plug that straight into um fpre so I'm not
[00:29:21] that straight into um fpre so I'm not going to give that answer away today um
[00:29:24] going to give that answer away today um okay so um other things that we want to
[00:29:27] okay so um other things that we want to do um with uh Jacobian is well we have
[00:29:32] do um with uh Jacobian is well we have this um layer of our neural network
[00:29:35] this um layer of our neural network where we're um calculating WX + B and we
[00:29:39] where we're um calculating WX + B and we can want to work out the partial
[00:29:40] can want to work out the partial derivative of that with respect to X um
[00:29:43] derivative of that with respect to X um you know this is the kind of place where
[00:29:46] you know this is the kind of place where it actually works to remember the Mantra
[00:29:49] it actually works to remember the Mantra and say Matrix calculus is just like
[00:29:52] and say Matrix calculus is just like single value variable calculus but with
[00:29:55] single value variable calculus but with matrices so if you just don't use your
[00:29:57] matrices so if you just don't use your bra too hard and think oh it's just like
[00:30:00] bra too hard and think oh it's just like single variable calculus so what should
[00:30:01] single variable calculus so what should the answer be it's obviously going to be
[00:30:03] the answer be it's obviously going to be W right and indeed it is um similarly if
[00:30:07] W right and indeed it is um similarly if we want to do the same thing for wxb and
[00:30:10] we want to do the same thing for wxb and work out the partial derivative with
[00:30:12] work out the partial derivative with respect to B um well that would be one
[00:30:15] respect to B um well that would be one in terms of single variable calculus and
[00:30:18] in terms of single variable calculus and so in Matrix calculus that becomes an
[00:30:21] so in Matrix calculus that becomes an identity Matrix okay slightly different
[00:30:23] identity Matrix okay slightly different the same idea um but that's reflecting
[00:30:26] the same idea um but that's reflecting the fact that b is actually Vector so we
[00:30:28] the fact that b is actually Vector so we need we need it to be coming out um as
[00:30:31] need we need it to be coming out um as an identity Matrix um okay so um higher
[00:30:37] an identity Matrix um okay so um higher up in my example picture I did this sort
[00:30:40] up in my example picture I did this sort of vector um do product of
[00:30:44] of vector um do product of UT um and well what happens if we work
[00:30:49] UT um and well what happens if we work out the um the
[00:30:53] out the um the um to C in of that what we end up with
[00:30:57] um to C in of that what we end up with strict ly is we come out with
[00:31:00] strict ly is we come out with HT um and you know this is sort of like
[00:31:04] HT um and you know this is sort of like when you're working out um well we did
[00:31:07] when you're working out um well we did this in the first class right when we
[00:31:09] this in the first class right when we did a a DOT product calculation that you
[00:31:12] did a a DOT product calculation that you kind of get for each individual element
[00:31:15] kind of get for each individual element you get the opposite term and so you get
[00:31:18] you get the opposite term and so you get um the other Vector coming out um these
[00:31:21] um the other Vector coming out um these are sort of good ones to comput it at
[00:31:23] are sort of good ones to comput it at home for practice to make sure um you
[00:31:26] home for practice to make sure um you really do know the answers and why they
[00:31:28] really do know the answers and why they work out the way they
[00:31:30] work out the way they do okay um so let's go back to our
[00:31:34] do okay um so let's go back to our little neural net um this was most of
[00:31:37] little neural net um this was most of our neural net up above our neural net
[00:31:40] our neural net up above our neural net um there was the
[00:31:42] um there was the nonlinearity now I'm going to uh leave
[00:31:45] nonlinearity now I'm going to uh leave that out of this time oh see I got it
[00:31:47] that out of this time oh see I got it wrong it's on assignment two um but you
[00:31:50] wrong it's on assignment two um but you know normally you'd be C calculating the
[00:31:53] know normally you'd be C calculating the partials of the output the loss function
[00:31:56] partials of the output the loss function with respect to the inputs but since the
[00:31:59] with respect to the inputs but since the loss function is On Assignment two I'm
[00:32:01] loss function is On Assignment two I'm going to leave that out and I'm just
[00:32:03] going to leave that out and I'm just going to calculate derivatives with
[00:32:05] going to calculate derivatives with respect to this score that feeds into
[00:32:08] respect to this score that feeds into the loss function so we first we got um
[00:32:11] the loss function so we first we got um the newal network layer the nonlinearity
[00:32:15] the newal network layer the nonlinearity and then we're doing this dot product to
[00:32:17] and then we're doing this dot product to work out a score for each position which
[00:32:19] work out a score for each position which feeds into the logistic function so if
[00:32:22] feeds into the logistic function so if you want to work out
[00:32:24] you want to work out ddb um so that's with respect to the
[00:32:27] ddb um so that's with respect to the bias for first so the way we do it is
[00:32:30] bias for first so the way we do it is you know we break up our equations into
[00:32:33] you know we break up our equations into our individual pieces that are composed
[00:32:36] our individual pieces that are composed together and so that means we break this
[00:32:38] together and so that means we break this up so we first calculate the zal WX + B
[00:32:42] up so we first calculate the zal WX + B then we apply the activation function to
[00:32:45] then we apply the activation function to the different
[00:32:46] the different components okay then after that
[00:32:50] components okay then after that um what we remember to do is okay to
[00:32:54] um what we remember to do is okay to work out um our um partial derivatives
[00:33:00] work out um our um partial derivatives of B of s with respect to B that what
[00:33:04] of B of s with respect to B that what we're going to be doing is doing the
[00:33:07] we're going to be doing is doing the product of um the partial derivatives of
[00:33:10] product of um the partial derivatives of the component pieces so we're applying
[00:33:13] the component pieces so we're applying um The Matrix calculus version of the
[00:33:15] um The Matrix calculus version of the chain rule so dsdb equals D sdh * DHD Z
[00:33:22] chain rule so dsdb equals D sdh DHD Z
[00:33:23] * dzb um and which corresponds to these
[00:33:26] dzb um and which corresponds to these three layers that are composed together
[00:33:30] three layers that are composed together and so at that point um we remember our
[00:33:32] and so at that point um we remember our useful jacobians from the previous slide
[00:33:36] useful jacobians from the previous slide and we can just apply them so the top
[00:33:39] and we can just apply them so the top one um DSD is um the you
[00:33:45] one um DSD is um the you transpose or else or maybe it's you
[00:33:48] transpose or else or maybe it's you let's come back to that there's a fine
[00:33:51] let's come back to that there's a fine point on that that I will explain more
[00:33:53] point on that that I will explain more about later um
[00:33:55] about later um um okay um then for the DH DZ that was
[00:34:00] um okay um then for the DH DZ that was the activation function where we got the
[00:34:02] the activation function where we got the diagonal of um the derivative of f of Z
[00:34:07] diagonal of um the derivative of f of Z and then for dzdb that's where we got
[00:34:10] and then for dzdb that's where we got the identity function okay um so we can
[00:34:15] the identity function okay um so we can simplify that down and so what that's
[00:34:18] simplify that down and so what that's going to end up as is the UT transpose
[00:34:23] going to end up as is the UT transpose that funny symbol there times the um the
[00:34:27] that funny symbol there times the um the vector
[00:34:28] vector element y's um derivative of f um this
[00:34:32] element y's um derivative of f um this symbol which doesn't normally turn up um
[00:34:36] symbol which doesn't normally turn up um in your regular math course but turns up
[00:34:38] in your regular math course but turns up all the time in New networks is referred
[00:34:41] all the time in New networks is referred to as the Hadad product and the Hadad
[00:34:44] to as the Hadad product and the Hadad product is meaning element wise
[00:34:46] product is meaning element wise multiplication so it's not like a cross
[00:34:49] multiplication so it's not like a cross product where you put two vectors
[00:34:50] product where you put two vectors together and you get out one number of
[00:34:52] together and you get out one number of Scala you put two vectors together you
[00:34:55] Scala you put two vectors together you element wise multiply them all and
[00:34:57] element wise multiply them all and you're left with another Vector of the
[00:35:00] you're left with another Vector of the same
[00:35:01] same type okay so that so now this gave us
[00:35:05] type okay so that so now this gave us are working out of the partials of dstb
[00:35:08] are working out of the partials of dstb and for a neural network um we want to
[00:35:11] and for a neural network um we want to work out all the other partials as well
[00:35:13] work out all the other partials as well so overall here in the picture right we
[00:35:18] so overall here in the picture right we had the X the W the B um and the U and
[00:35:26] had the X the W the B um and the U and we'd like to work out partials with
[00:35:28] we'd like to work out partials with respect to all of those variables so we
[00:35:31] respect to all of those variables so we can change their values and learn so
[00:35:34] can change their values and learn so that our model predicts better um so um
[00:35:39] that our model predicts better um so um so suppose we now want to um calculate
[00:35:43] so suppose we now want to um calculate ddw so again we can split it up with the
[00:35:46] ddw so again we can split it up with the same chain Rule and say
[00:35:48] same chain Rule and say ddw equals the product of these three
[00:35:51] ddw equals the product of these three things and the important thing to notice
[00:35:54] things and the important thing to notice is that two of those three things were
[00:35:56] is that two of those three things were exactly the same ones that we calculated
[00:35:58] exactly the same ones that we calculated before the only bit that's different is
[00:36:01] before the only bit that's different is that at the end we're now doing DZ DW
[00:36:05] that at the end we're now doing DZ DW rather than
[00:36:06] rather than dzdb and so the first central idea that
[00:36:10] dzdb and so the first central idea that we'll come back to when we do
[00:36:11] we'll come back to when we do computation graphs is oh we really want
[00:36:14] computation graphs is oh we really want to avoid doing repeated work so we want
[00:36:18] to avoid doing repeated work so we want to realize that those two parts of
[00:36:20] to realize that those two parts of things are the same and since we're just
[00:36:22] things are the same and since we're just sort of multiplying these um partial
[00:36:25] sort of multiplying these um partial derivatives together right we can just
[00:36:27] derivatives together right we can just compute what that part is and reuse it
[00:36:30] compute what that part is and reuse it and so if we want to um wait yeah okay
[00:36:36] and so if we want to um wait yeah okay so if we're wanting to calculate
[00:36:39] so if we're wanting to calculate ddw the part that's the same this part
[00:36:43] ddw the part that's the same this part here we can refer to as Delta so Delta
[00:36:47] here we can refer to as Delta so Delta is sort of the Upstream gradient or the
[00:36:49] is sort of the Upstream gradient or the error signal the part that you've got
[00:36:51] error signal the part that you've got from sort of starting at the beginning
[00:36:53] from sort of starting at the beginning DS DH DH DZ this sort of shared Upstream
[00:36:57] DS DH DH DZ this sort of shared Upstream part we can calculate that once and then
[00:37:01] part we can calculate that once and then we can use it um to calculate both of
[00:37:04] we can use it um to calculate both of these two things and for dsdb because
[00:37:09] these two things and for dsdb because the dzdb just comes out as the identity
[00:37:11] the dzdb just comes out as the identity Matrix um the answer is just Delta but
[00:37:16] Matrix um the answer is just Delta but for D
[00:37:17] for D sdw we need to work out the DZ DW before
[00:37:22] sdw we need to work out the DZ DW before we're
[00:37:24] we're finished okay so what um does what do we
[00:37:29] finished okay so what um does what do we get for that last piece so one question
[00:37:32] get for that last piece so one question you might start off with and is normally
[00:37:35] you might start off with and is normally a good thing to think about when you're
[00:37:37] a good thing to think about when you're doing assignment problems on this and
[00:37:39] doing assignment problems on this and other things is the first thing to think
[00:37:41] other things is the first thing to think about is you know what do things look
[00:37:44] about is you know what do things look like like am I should the answer be a
[00:37:47] like like am I should the answer be a vector a matrix uh what size should it
[00:37:50] vector a matrix uh what size should it be and things like that so for
[00:37:53] be and things like that so for ddw um W is an N bym Matrix
[00:37:58] ddw um W is an N bym Matrix um and S is a scalar so therefore since
[00:38:03] um and S is a scalar so therefore since we have one output and M * m inputs the
[00:38:08] we have one output and M * m inputs the answer according to math should be that
[00:38:12] answer according to math should be that we've got a one by n * m Jacobian I a
[00:38:17] we've got a one by n * m Jacobian I a big long row Vector um but here's where
[00:38:22] big long row Vector um but here's where um things get a teeny bit tricky um and
[00:38:26] um things get a teeny bit tricky um and there's sort of we end up with this
[00:38:29] there's sort of we end up with this weird mess of um math and Engineering
[00:38:34] weird mess of um math and Engineering convenience because you know immediately
[00:38:36] convenience because you know immediately what we're wanting to do is we're
[00:38:39] what we're wanting to do is we're wanting to take our old parameters which
[00:38:42] wanting to take our old parameters which will be in stored in the form of
[00:38:44] will be in stored in the form of matrices vectors and so on that we're
[00:38:47] matrices vectors and so on that we're using as coefficients and we're going to
[00:38:49] using as coefficients and we're going to want to subtract from them um our you
[00:38:53] want to subtract from them um our you know a fraction of our calculated
[00:38:55] know a fraction of our calculated gradient so what we'd like to do is have
[00:38:59] gradient so what we'd like to do is have our um calculated gradients in the same
[00:39:03] our um calculated gradients in the same shapes as our parameters because then we
[00:39:06] shapes as our parameters because then we can just do subtraction whereas if
[00:39:08] can just do subtraction whereas if they've turned into a God Almighty row
[00:39:10] they've turned into a God Almighty row Vector um that's not quite so convenient
[00:39:13] Vector um that's not quite so convenient um so it turns out that what we end up
[00:39:17] um so it turns out that what we end up doing um is using something that gets
[00:39:20] doing um is using something that gets referred to as the shape convention that
[00:39:23] referred to as the shape convention that um we
[00:39:25] um we uh uh
[00:39:27] uh uh reshape our um jacobians so they fit
[00:39:33] reshape our um jacobians so they fit into things that are of the same shape
[00:39:36] into things that are of the same shape as the parameters that we are using so
[00:39:40] as the parameters that we are using so we're going to represent um DSD W as an
[00:39:43] we're going to represent um DSD W as an N bym Matrix laid out as follows and
[00:39:48] N bym Matrix laid out as follows and that's a place that one people can get
[00:39:51] that's a place that one people can get confused okay so that's what we want to
[00:39:53] confused okay so that's what we want to calculate that kind of Matrix but um and
[00:39:57] calculate that kind of Matrix but um and so that Matrix is going to be Delta * DZ
[00:40:00] so that Matrix is going to be Delta * DZ DW so Delta is going to be part of the
[00:40:03] DW so Delta is going to be part of the answer and then we want to know um what
[00:40:06] answer and then we want to know um what DZ DW is um and the answer is going to
[00:40:10] DZ DW is um and the answer is going to be it's going to come out like this so D
[00:40:12] be it's going to come out like this so D sdw is going to be um delta T * XT so
[00:40:17] sdw is going to be um delta T * XT so it's going to be the product of the
[00:40:19] it's going to be the product of the Upstream gradient which was the same
[00:40:21] Upstream gradient which was the same thing we calculated before for the other
[00:40:23] thing we calculated before for the other two quantities and then a local inputs
[00:40:27] two quantities and then a local inputs symbol which um is input signal which is
[00:40:32] symbol which um is input signal which is here coming out to
[00:40:34] here coming out to XT okay um and you know so we're taking
[00:40:38] XT okay um and you know so we're taking the transposes of those two vectors
[00:40:41] the transposes of those two vectors which it means that we end up
[00:40:43] which it means that we end up calculating an outer product of those
[00:40:45] calculating an outer product of those two vectors um which gives us our
[00:40:49] two vectors um which gives us our gradient um and so why is that the right
[00:40:53] gradient um and so why is that the right answer well you know it kind of looks
[00:40:55] answer well you know it kind of looks convenient CU that's giving us something
[00:40:57] convenient CU that's giving us something of the right shape um for what I was
[00:40:59] of the right shape um for what I was arguing we want to find out and we have
[00:41:01] arguing we want to find out and we have the right number of terms um now I'm
[00:41:05] the right number of terms um now I'm going to rush through this so I
[00:41:07] going to rush through this so I encourage you to read um the uh lecture
[00:41:10] encourage you to read um the uh lecture notes and do this more carefully but um
[00:41:13] notes and do this more carefully but um let me at least a little bit explain why
[00:41:15] let me at least a little bit explain why it makes sense right so um if you think
[00:41:19] it makes sense right so um if you think of one weight um in so all of these
[00:41:23] of one weight um in so all of these connections are our Matrix right The
[00:41:25] connections are our Matrix right The Matrix is being represented by all these
[00:41:26] Matrix is being represented by all these lines and in your network so if you
[00:41:29] lines and in your network so if you think of one number in The Matrix so
[00:41:31] think of one number in The Matrix so here is
[00:41:32] here is w23 so it's connecting from input 3 or
[00:41:36] w23 so it's connecting from input 3 or it's multiplying input three to give
[00:41:39] it's multiplying input three to give part of the answer of
[00:41:41] part of the answer of H2 right so it's this line here um so
[00:41:45] H2 right so it's this line here um so for this line here um this weight is
[00:41:50] for this line here um this weight is being used only in the calculation of
[00:41:52] being used only in the calculation of H2 and the only thing it's dependent on
[00:41:55] H2 and the only thing it's dependent on is X3 um so that if you're then wanting
[00:41:59] is X3 um so that if you're then wanting to work out the partial
[00:42:03] to work out the partial of um h
[00:42:06] of um h two um or Z2 sorry yeah um sorry yeah
[00:42:13] two um or Z2 sorry yeah um sorry yeah sorry Z2 the partial of Z2 with respect
[00:42:17] sorry Z2 the partial of Z2 with respect to X3 it's sort of depending on these
[00:42:21] to X3 it's sort of depending on these two pieces only and that's what you're
[00:42:23] two pieces only and that's what you're achieving um by working out um the sort
[00:42:28] achieving um by working out um the sort of outer product like
[00:42:31] of outer product like that okay um yeah so let me just come
[00:42:35] that okay um yeah so let me just come back um one more time to um this the
[00:42:39] back um one more time to um this the sort of question of the shape of
[00:42:42] sort of question of the shape of derivatives
[00:42:44] derivatives um you know so I already sort of fudged
[00:42:47] um you know so I already sort of fudged it um when I was sort of um talking
[00:42:51] it um when I was sort of um talking about oh should I put the the transpose
[00:42:54] about oh should I put the the transpose there or should I nod and get a row
[00:42:56] there or should I nod and get a row vector is a column Vector um so there's
[00:43:00] vector is a column Vector um so there's sort of this disagreement between
[00:43:04] sort of this disagreement between whether you kind of have the Jacobian
[00:43:07] whether you kind of have the Jacobian form which is what actually makes the
[00:43:09] form which is what actually makes the chain roll work right in terms of doing
[00:43:13] chain roll work right in terms of doing multiplication um versus the shape
[00:43:16] multiplication um versus the shape convention which is how we store
[00:43:18] convention which is how we store everything for our computations and Mak
[00:43:21] everything for our computations and Mak doing stochastic gradient descent where
[00:43:24] doing stochastic gradient descent where you're um subtracting um whatever kind
[00:43:27] you're um subtracting um whatever kind of tensor you have easy um so
[00:43:32] of tensor you have easy um so um you know this can be a source of
[00:43:34] um you know this can be a source of confusion um since we're doing a
[00:43:37] confusion um since we're doing a computer science course for the answers
[00:43:39] computer science course for the answers in the assignment we expect you to
[00:43:41] in the assignment we expect you to follow the shape convention so you know
[00:43:44] follow the shape convention so you know if you're working out the derivatives
[00:43:46] if you're working out the derivatives with respect to some Matrix it should be
[00:43:49] with respect to some Matrix it should be shaped like a matrix with the same
[00:43:51] shaped like a matrix with the same parameters um but you know you may well
[00:43:55] parameters um but you know you may well want to think about Jacobian forms and
[00:43:57] want to think about Jacobian forms and Computing your answers I mean there are
[00:43:59] Computing your answers I mean there are sort of two ways to go about do doing
[00:44:01] sort of two ways to go about do doing this one way of doing it is to sort of
[00:44:04] this one way of doing it is to sort of work out all the math using Jacobian Al
[00:44:07] work out all the math using Jacobian Al math 51 and at the end just to reshape
[00:44:10] math 51 and at the end just to reshape it so it fits into the same shape as the
[00:44:14] it so it fits into the same shape as the parameters according to our shape
[00:44:16] parameters according to our shape convention I mean the other way is to
[00:44:19] convention I mean the other way is to sort of do each stage following the
[00:44:21] sort of do each stage following the shape convention but then you sort of
[00:44:23] shape convention but then you sort of have to be game to sort of reshape
[00:44:26] have to be game to sort of reshape things as needed by sort of doing um
[00:44:29] things as needed by sort of doing um transposing to have things work out at
[00:44:31] transposing to have things work out at the different
[00:44:33] the different stages okay that was my attempt to
[00:44:36] stages okay that was my attempt to quickly review the
[00:44:39] math most people are still here um I
[00:44:43] math most people are still here um I will now go on to the second half and go
[00:44:47] will now go on to the second half and go on to the um How We Do the computation
[00:44:50] on to the um How We Do the computation right so you know so most of um yeah so
[00:44:55] right so you know so most of um yeah so the famous thing that powers new
[00:44:57] the famous thing that powers new networks is the back propagation
[00:44:59] networks is the back propagation algorithm so the back propagation
[00:45:02] algorithm so the back propagation algorithm is really only two things you
[00:45:06] algorithm is really only two things you know its invention made people famous
[00:45:09] know its invention made people famous because it gave an effective learning
[00:45:11] because it gave an effective learning algorithm but you know at a fundamental
[00:45:13] algorithm but you know at a fundamental level the back propagation algorithm is
[00:45:16] level the back propagation algorithm is only two things thing one is you use the
[00:45:20] only two things thing one is you use the chain rule you do calculus of complex
[00:45:23] chain rule you do calculus of complex functions and Thing Two is
[00:45:27] functions and Thing Two is um you store intermediate results so you
[00:45:30] um you store intermediate results so you never recompute the same stuff again
[00:45:33] never recompute the same stuff again that's all there is to the um back
[00:45:35] that's all there is to the um back propagation
[00:45:37] propagation algorithm and so let's just go through
[00:45:40] algorithm and so let's just go through that so if we're
[00:45:42] that so if we're computationally wanting to deal um with
[00:45:45] computationally wanting to deal um with you know functions and doing back
[00:45:48] you know functions and doing back propagation we can think of them as
[00:45:51] propagation we can think of them as being represented as a graph and in some
[00:45:54] being represented as a graph and in some way or another um this kind of graph is
[00:45:58] way or another um this kind of graph is being used inside your newal network
[00:46:01] being used inside your newal network framework so here is here is a re-
[00:46:03] framework so here is here is a re- representation of my little neural
[00:46:05] representation of my little neural network for finding whether the word at
[00:46:07] network for finding whether the word at the center is a location so I'm taking
[00:46:11] the center is a location so I'm taking the X Vector input I'm multiplying it by
[00:46:14] the X Vector input I'm multiplying it by W I'm adding B to it I'm putting it
[00:46:17] W I'm adding B to it I'm putting it through the
[00:46:18] through the nonlinearity and then I'm um doing the
[00:46:20] nonlinearity and then I'm um doing the dot product with my Vector U right so
[00:46:23] dot product with my Vector U right so that was my computation and so the The
[00:46:27] that was my computation and so the The Source nodes are the inputs in this
[00:46:30] Source nodes are the inputs in this graph the interior nodes then the
[00:46:33] graph the interior nodes then the operations I do um and so then the edges
[00:46:38] operations I do um and so then the edges that connect those together then pass
[00:46:41] that connect those together then pass along the result of each operation so I
[00:46:44] along the result of each operation so I passed along WX to the addition function
[00:46:48] passed along WX to the addition function with B then I that gives me Z that I
[00:46:52] with B then I that gives me Z that I pass through the nonlinearity which
[00:46:53] pass through the nonlinearity which gives me H which I then um dot product
[00:46:56] gives me H which I then um dot product with the U to get S okay so I do
[00:47:00] with the U to get S okay so I do precisely this computation and this is
[00:47:03] precisely this computation and this is referred to as Ford propagation or the
[00:47:06] referred to as Ford propagation or the forward pass of a neural network so um
[00:47:09] forward pass of a neural network so um the forward pass just calculates
[00:47:12] the forward pass just calculates functions okay but then once we've done
[00:47:16] functions okay but then once we've done that what we want to
[00:47:18] that what we want to do is then work out gradients so we can
[00:47:22] do is then work out gradients so we can do gradient based learning and so that
[00:47:25] do gradient based learning and so that part is then referred to as back
[00:47:29] part is then referred to as back propagation or the backward pass and
[00:47:32] propagation or the backward pass and then we run things backward so for
[00:47:35] then we run things backward so for running things backward we're going to
[00:47:37] running things backward we're going to use the same graph and we're going to
[00:47:39] use the same graph and we're going to backwards pass along at gradients and so
[00:47:42] backwards pass along at gradients and so we start at the right hand side and we
[00:47:45] we start at the right hand side and we have dsds so dsds is just one because
[00:47:49] have dsds so dsds is just one because you know um if you change S you've
[00:47:52] you know um if you change S you've changed s and then what we want to do is
[00:47:56] changed s and then what we want to do is sort of then work further back so we can
[00:47:59] sort of then work further back so we can work out DSD ddz dsdb ddw dsdx as we
[00:48:05] work out DSD ddz dsdb ddw dsdx as we work back um and so this is um the what
[00:48:11] work back um and so this is um the what we want to work out with gradients um
[00:48:14] we want to work out with gradients um and so how are we going to do that well
[00:48:17] and so how are we going to do that well if we look at a single node so for
[00:48:19] if we look at a single node so for example our um our nonlinearity node but
[00:48:23] example our um our nonlinearity node but any node where H equal F ofx what we can
[00:48:27] any node where H equal F ofx what we can have is an upstream gradient um DS DH
[00:48:32] have is an upstream gradient um DS DH and what we want to do is calculate the
[00:48:35] and what we want to do is calculate the downstream gradient of the next variable
[00:48:38] downstream gradient of the next variable down the D
[00:48:40] down the D sdz and the way that we're going to do
[00:48:42] sdz and the way that we're going to do that is we're going to say well let's
[00:48:46] that is we're going to say well let's look at F what is F's gradient and
[00:48:50] look at F what is F's gradient and that's going to be our local gradient
[00:48:53] that's going to be our local gradient and then this is immediately what gives
[00:48:55] and then this is immediately what gives us the chain rule
[00:48:57] us the chain rule that D sdz is going to be the product of
[00:49:01] that D sdz is going to be the product of our Upstream gradient D sdh times the
[00:49:05] our Upstream gradient D sdh times the DHD Z the local um gradient that we
[00:49:08] DHD Z the local um gradient that we calculate at that node so Downstream
[00:49:11] calculate at that node so Downstream gradient equals um Upstream gradient
[00:49:15] gradient equals um Upstream gradient times local
[00:49:18] times local gradient
[00:49:21] gradient um oh yeah that's what that's what it
[00:49:23] um oh yeah that's what that's what it says when I press um that again okay so
[00:49:27] says when I press um that again okay so this is the sort of the um single the
[00:49:31] this is the sort of the um single the single input single output case though
[00:49:34] single input single output case though those inputs might be vectors or
[00:49:36] those inputs might be vectors or matrices or something like that um we
[00:49:39] matrices or something like that um we then have sort of more complex graph
[00:49:42] then have sort of more complex graph cases um
[00:49:44] cases um so I think I should have retitled this
[00:49:46] so I think I should have retitled this SL oh yeah so still so sorry so the next
[00:49:50] SL oh yeah so still so sorry so the next case is for our node it might have
[00:49:53] case is for our node it might have multiple inputs so this is where we're
[00:49:55] multiple inputs so this is where we're calculating
[00:49:57] calculating WX so in that case we still have an up
[00:50:01] WX so in that case we still have an up we have a single Upstream gradient and
[00:50:04] we have a single Upstream gradient and then what we're going to do is we want
[00:50:07] then what we're going to do is we want to calculate the downstream gradient
[00:50:10] to calculate the downstream gradient with respect to each input and the way
[00:50:13] with respect to each input and the way we're going to do that is we're going to
[00:50:15] we're going to do that is we're going to work out the local gradient with respect
[00:50:18] work out the local gradient with respect to each input and then we're going to do
[00:50:20] to each input and then we're going to do the same kind of multiplication of
[00:50:23] the same kind of multiplication of Upstream gradient times local gradient
[00:50:27] Upstream gradient times local gradient with respect to each input again um
[00:50:30] with respect to each input again um chain
[00:50:32] chain rule okay um so here's a little example
[00:50:36] rule okay um so here's a little example of this so I'm this isn't really uh the
[00:50:40] of this so I'm this isn't really uh the kind of thing you normally see in a
[00:50:41] kind of thing you normally see in a neural network but it's an easy example
[00:50:44] neural network but it's an easy example so F of XYZ is going to be x + y * the
[00:50:48] so F of XYZ is going to be x + y * the max of y z and we've got current values
[00:50:53] max of y z and we've got current values of X Y and Z of 1 2 and z respectively
[00:50:57] of X Y and Z of 1 2 and z respectively so here's our little computation graph
[00:51:00] so here's our little computation graph um and so for forward propagation you
[00:51:03] um and so for forward propagation you know we're going to do this addition
[00:51:05] know we're going to do this addition we're going to do this Max function and
[00:51:07] we're going to do this Max function and then we're going to multiply the two and
[00:51:10] then we're going to multiply the two and that gives us the value of f um so we
[00:51:13] that gives us the value of f um so we can run that with the current values of
[00:51:15] can run that with the current values of X Y and Z and this is what we get so the
[00:51:19] X Y and Z and this is what we get so the max of two and 0 is two addition is
[00:51:22] max of two and 0 is two addition is three the answer is six okay so then
[00:51:26] three the answer is six okay so then after having done that we run the
[00:51:29] after having done that we run the backward propagation and yeah so this
[00:51:32] backward propagation and yeah so this procedure you know is not actually
[00:51:34] procedure you know is not actually special to new networks right you can
[00:51:36] special to new networks right you can use it for any piece of math if you want
[00:51:38] use it for any piece of math if you want to just run your math um on pytorch
[00:51:41] to just run your math um on pytorch rather than um working it out in your
[00:51:43] rather than um working it out in your head or with Mathematica um okay so now
[00:51:46] head or with Mathematica um okay so now we work out um backwards so we want to
[00:51:50] we work out um backwards so we want to know the local gradient so d a DZ is
[00:51:55] know the local gradient so d a DZ is going to be one
[00:51:58] going to be one sorry I said that wrong d a DX is going
[00:52:00] sorry I said that wrong d a DX is going to be 1 so a = x + y d a d y = 1 um for
[00:52:05] to be 1 so a = x + y d a d y = 1 um for the max function that's going to depend
[00:52:08] the max function that's going to depend on which of the two is larger because
[00:52:10] on which of the two is larger because it's going to have a slope of one for
[00:52:14] it's going to have a slope of one for the one that's the biggest and zero for
[00:52:16] the one that's the biggest and zero for the one that's the smallest um and then
[00:52:19] the one that's the smallest um and then for the product that's like what we saw
[00:52:22] for the product that's like what we saw with vectors that dfda is going to be B
[00:52:26] with vectors that dfda is going to be B and D FDB is going to be a um so those
[00:52:29] and D FDB is going to be a um so those are all our local gradients and so then
[00:52:32] are all our local gradients and so then we can use those to calculate out the
[00:52:35] we can use those to calculate out the derivatives so dfdf is one um we then
[00:52:39] derivatives so dfdf is one um we then multiply that by the two um local
[00:52:44] multiply that by the two um local gradients um that are
[00:52:46] gradients um that are calculated
[00:52:49] calculated um for A and B so that gives us um two
[00:52:54] um for A and B so that gives us um two and three where you're swapping over the
[00:52:57] and three where you're swapping over the numbers then for the max um that we're
[00:53:02] numbers then for the max um that we're having the one that is biggest um we're
[00:53:06] having the one that is biggest um we're taking the Upstream times one so it gets
[00:53:10] taking the Upstream times one so it gets three the other one gets zero and then
[00:53:13] three the other one gets zero and then for the plus we're just sending the
[00:53:15] for the plus we're just sending the gradient down in both directions and so
[00:53:18] gradient down in both directions and so both of them come out as two um and so
[00:53:23] both of them come out as two um and so that gives us DF DX so the final
[00:53:26] that gives us DF DX so the final function value is two D
[00:53:29] function value is two D fdy we're taking the three and adding
[00:53:32] fdy we're taking the three and adding the two I'll mention that again in a
[00:53:34] the two I'll mention that again in a minute which gives us five and then
[00:53:38] minute which gives us five and then DFD Z is zero um and we should be able
[00:53:42] DFD Z is zero um and we should be able to again be able to quickly check that
[00:53:44] to again be able to quickly check that we've got this right right so um if we
[00:53:48] we've got this right right so um if we consider you know the slope around um Z
[00:53:54] consider you know the slope around um Z as you change z a little so Z is Z if we
[00:53:57] as you change z a little so Z is Z if we make Z
[00:53:59] make Z 0.1 that makes absolutely no difference
[00:54:02] 0.1 that makes absolutely no difference to what the computed function value is
[00:54:05] to what the computed function value is um so the gradient there is zero that's
[00:54:08] um so the gradient there is zero that's correct um so if I change up the Top If
[00:54:12] correct um so if I change up the Top If I change x a little bit right if I
[00:54:15] I change x a little bit right if I change X to
[00:54:17] change X to 1.1 then I'll be calculating um 1.1 + 2
[00:54:22] 1.1 then I'll be calculating um 1.1 + 2 is
[00:54:24] is 3.1 um and then I'll be taking the max
[00:54:28] 3.1 um and then I'll be taking the max which is two and I'll be calculating
[00:54:33] which is two and I'll be calculating 5.1 um and so wait no I did that
[00:54:39] 5.1 um and so wait no I did that wrong oh times two
[00:54:42] wrong oh times two wait I didn't do the multiplication
[00:54:45] wait I didn't do the multiplication right um sorry yeah so we get the 3.1
[00:54:50] right um sorry yeah so we get the 3.1 that's multiplied by two that gives us
[00:54:53] that's multiplied by two that gives us 6.2 so a change of 0.1 in the X has
[00:54:57] 6.2 so a change of 0.1 in the X has moved things up by 2 so that corresponds
[00:55:00] moved things up by 2 so that corresponds to the gradient being two and so then
[00:55:03] to the gradient being two and so then the final case is well what if we change
[00:55:06] the final case is well what if we change y to um so y started off as two and made
[00:55:12] y to um so y started off as two and made it 2.1 then we're going to get 2.1
[00:55:17] it 2.1 then we're going to get 2.1 multiplied by 1 is
[00:55:21] 2.1 61 6.5 and right and then we've got
[00:55:26] 2.1 61 6.5 and right and then we've got the 2.1
[00:55:28] the 2.1 here the oh sorry I keep doing this
[00:55:31] here the oh sorry I keep doing this wrong 2.1 + 1 = 3.1 and then we've got
[00:55:35] wrong 2.1 + 1 = 3.1 and then we've got 2.1 as the max so we've got 2.1 *
[00:55:40] 2.1 as the max so we've got 2.1 * 3.1 and that comes out to be
[00:55:45] 3.1 and that comes out to be 6.51 so it's approximately gone up by so
[00:55:48] 6.51 so it's approximately gone up by so our 0.1 difference has gone up to
[00:55:51] our 0.1 difference has gone up to approximately 0.5 this is just an
[00:55:53] approximately 0.5 this is just an estimate um and so that correspond to
[00:55:56] estimate um and so that correspond to the gradient being five right we get
[00:55:59] the gradient being five right we get this five times multiplication of our
[00:56:02] this five times multiplication of our changes okay um and so that that fact
[00:56:06] changes okay um and so that that fact that illustrates the fact that the right
[00:56:08] that illustrates the fact that the right thing to do is when you have outward
[00:56:11] thing to do is when you have outward branches in your um computation graph
[00:56:15] branches in your um computation graph and you're running the um back
[00:56:18] and you're running the um back propagation that what you do is you sum
[00:56:22] propagation that what you do is you sum the gradients right um so that for this
[00:56:26] the gradients right um so that for this case we had y being um the Y is sort of
[00:56:31] case we had y being um the Y is sort of going into these two different things in
[00:56:34] going into these two different things in our previous chart so once we've worked
[00:56:36] our previous chart so once we've worked out the Upstream gradients we sum them
[00:56:39] out the Upstream gradients we sum them to get the total gradient and so that's
[00:56:41] to get the total gradient and so that's what we did back here we had two outward
[00:56:44] what we did back here we had two outward things and we sort of took these
[00:56:46] things and we sort of took these calculated Upstream gradients of two and
[00:56:48] calculated Upstream gradients of two and three and we just summ them to get five
[00:56:52] three and we just summ them to get five and that gave the right answer
[00:56:59] okay um and so you can think about that
[00:57:03] okay um and so you can think about that um for the sort of just generally how
[00:57:07] um for the sort of just generally how the sort of things to think about as
[00:57:09] the sort of things to think about as sort of gradients move around in these
[00:57:12] sort of gradients move around in these pictures so that when we have a plus
[00:57:16] pictures so that when we have a plus operation that um plush just sort of
[00:57:20] operation that um plush just sort of distributes gradient so the same
[00:57:22] distributes gradient so the same gradient that's the Upstream gradient
[00:57:24] gradient that's the Upstream gradient goes to each input
[00:57:26] goes to each input um when you have a Max it's kind of like
[00:57:29] um when you have a Max it's kind of like a router of gradient so the max is going
[00:57:32] a router of gradient so the max is going to send the gradient to one of the
[00:57:35] to send the gradient to one of the inputs and send nothing at all to the
[00:57:38] inputs and send nothing at all to the other inputs um and when you have a
[00:57:41] other inputs um and when you have a multiplication it's a little bit funky
[00:57:43] multiplication it's a little bit funky because you're sort of um doing this
[00:57:46] because you're sort of um doing this sort of switching of the forward
[00:57:48] sort of switching of the forward coefficient so you're taking the
[00:57:50] coefficient so you're taking the Upstream gradient multiplied by the
[00:57:53] Upstream gradient multiplied by the opposite um for coefficient gives you
[00:57:57] opposite um for coefficient gives you your um Downstream
[00:58:00] your um Downstream gradient okay um so we kind of have this
[00:58:04] gradient okay um so we kind of have this systematic way of being able to sort of
[00:58:07] systematic way of being able to sort of forward pass calculate the values of
[00:58:09] forward pass calculate the values of functions then run this backward to work
[00:58:13] functions then run this backward to work out um the um gradients heading down the
[00:58:17] out um the um gradients heading down the network and so the main other thing of
[00:58:21] network and so the main other thing of the back propagation algorithm is just
[00:58:24] the back propagation algorithm is just that we want to do this
[00:58:26] that we want to do this efficiently so the wrong way to do it
[00:58:29] efficiently so the wrong way to do it would be to say well gee I want to
[00:58:31] would be to say well gee I want to calculate dstb
[00:58:33] calculate dstb ddw dsdx ddu so let me start doing those
[00:58:38] ddw dsdx ddu so let me start doing those one at a time and when I've done them
[00:58:40] one at a time and when I've done them all I will stop because that means if
[00:58:43] all I will stop because that means if you first calculated dstb you do all of
[00:58:47] you first calculated dstb you do all of the part that's in blue um but then if
[00:58:50] the part that's in blue um but then if you went on to DS
[00:58:53] you went on to DS DW um you'd be calculating all the part
[00:58:56] DW um you'd be calculating all the part in red and well just as we saw in the
[00:58:59] in red and well just as we saw in the math part when we were doing it as math
[00:59:02] math part when we were doing it as math um these parts are exactly the same
[00:59:05] um these parts are exactly the same you're doing exactly the same
[00:59:07] you're doing exactly the same computations so you only want to do
[00:59:10] computations so you only want to do those that part once and work out this
[00:59:13] those that part once and work out this Upstream gradient or error signal that
[00:59:16] Upstream gradient or error signal that is being then sort of calculated and is
[00:59:19] is being then sort of calculated and is then being shared so the picture that we
[00:59:22] then being shared so the picture that we want to have is you're doing together
[00:59:25] want to have is you're doing together the Shar part and then you're only sort
[00:59:28] the Shar part and then you're only sort of doing separately the little bits um
[00:59:31] of doing separately the little bits um that you need to
[00:59:33] that you need to do okay um boy I seem to have been
[00:59:36] do okay um boy I seem to have been rushing through today and I'm going to
[00:59:38] rushing through today and I'm going to actually end early unless anyone is
[00:59:40] actually end early unless anyone is going to slow me down but I did have uh
[00:59:43] going to slow me down but I did have uh just a few more slides um to go through
[00:59:46] just a few more slides um to go through um yeah so the sort of
[00:59:49] um yeah so the sort of generalization of this as an algorithm
[00:59:52] generalization of this as an algorithm is you know in the general case you know
[00:59:55] is you know in the general case you know so we normally have these sort of neural
[00:59:58] so we normally have these sort of neural network layers and matrices which you
[01:00:01] network layers and matrices which you can represent as vectors and
[01:00:03] can represent as vectors and matrices um and you know it's sort of
[01:00:07] matrices um and you know it's sort of nice and clean and it looks like um
[01:00:10] nice and clean and it looks like um doing that in Calculus class I mean
[01:00:13] doing that in Calculus class I mean strictly speaking that isn't necessary
[01:00:16] strictly speaking that isn't necessary so the algorithm for forward propagation
[01:00:18] so the algorithm for forward propagation and backward propagation that I've
[01:00:21] and backward propagation that I've outlined that you can have it work in a
[01:00:24] outlined that you can have it work in a completely arbitrary comput ation graph
[01:00:27] completely arbitrary comput ation graph providing it's a dag that doesn't have
[01:00:29] providing it's a dag that doesn't have Cycles in it um so the general algorithm
[01:00:32] Cycles in it um so the general algorithm is well you've got a whole bunch of
[01:00:35] is well you've got a whole bunch of variables that depend on other variables
[01:00:38] variables that depend on other variables there's some way in which we can sort
[01:00:41] there's some way in which we can sort them so that each variable only depends
[01:00:44] them so that each variable only depends on variables to the left of it so that's
[01:00:47] on variables to the left of it so that's referred to as a topological sort of the
[01:00:50] referred to as a topological sort of the outputs and so that means there's a way
[01:00:52] outputs and so that means there's a way we can do a forward pass where we're
[01:00:56] we can do a forward pass where we're calculating um variables in terms of
[01:00:59] calculating um variables in terms of ones that have already been calculated
[01:01:02] ones that have already been calculated but you know if we want to have some
[01:01:03] but you know if we want to have some extra wonky AR so it's not like nice
[01:01:06] extra wonky AR so it's not like nice Matrix multiplies or anything we're
[01:01:08] Matrix multiplies or anything we're totally allowed to um do that or we can
[01:01:11] totally allowed to um do that or we can have things not fully connected right so
[01:01:13] have things not fully connected right so there's no connections across here right
[01:01:16] there's no connections across here right we can have an arbitrary computation
[01:01:18] we can have an arbitrary computation graph um and so that gives us our
[01:01:20] graph um and so that gives us our forward propagation and then once we've
[01:01:24] forward propagation and then once we've done the forward propagation we can
[01:01:26] done the forward propagation we can initialize the output gradient as as one
[01:01:31] initialize the output gradient as as one and then we're going to visit the nodes
[01:01:34] and then we're going to visit the nodes in reverse order and at for each node
[01:01:38] in reverse order and at for each node we're going to compute a gradient by
[01:01:41] we're going to compute a gradient by using the Upstream gradient and the
[01:01:43] using the Upstream gradient and the local gradient to compute the downstream
[01:01:46] local gradient to compute the downstream gradient and so then we can head back
[01:01:48] gradient and so then we can head back down the computation graph and work out
[01:01:51] down the computation graph and work out all of the downstream gradients and so
[01:01:54] all of the downstream gradients and so the crucial thing to notice is that if
[01:01:57] the crucial thing to notice is that if you do it correctly um that working out
[01:02:03] you do it correctly um that working out um the the gradients has the same bigo
[01:02:09] um the the gradients has the same bigo complexity as working out the forward
[01:02:12] complexity as working out the forward calculation right so that if you're
[01:02:14] calculation right so that if you're doing more you know in if if in terms of
[01:02:17] doing more you know in if if in terms of Big O terms right you might have
[01:02:19] Big O terms right you might have different functions depending on what
[01:02:20] different functions depending on what the derivatives are but in bigo terms if
[01:02:23] the derivatives are but in bigo terms if you're doing more work in the backward
[01:02:26] you're doing more work in the backward path than you're doing in the forward
[01:02:27] path than you're doing in the forward paths that means that you're somehow
[01:02:30] paths that means that you're somehow failing to do this um efficient
[01:02:33] failing to do this um efficient computation and that you're recomputing
[01:02:35] computation and that you're recomputing some of your
[01:02:37] some of your work okay um so because we have such a
[01:02:41] work okay um so because we have such a good algorithm here you should be able
[01:02:45] good algorithm here you should be able to just work out the backward path
[01:02:48] to just work out the backward path automatically and that that gets
[01:02:50] automatically and that that gets referred to as automatic differentiation
[01:02:53] referred to as automatic differentiation so if you had the symbolic form of what
[01:02:59] so if you had the symbolic form of what you're calculating with your forward
[01:03:01] you're calculating with your forward pass um you should just be able to say
[01:03:04] pass um you should just be able to say yo computer um can you work out the
[01:03:07] yo computer um can you work out the backward pass for me and you know kind
[01:03:10] backward pass for me and you know kind of mathematical like it could look at
[01:03:13] of mathematical like it could look at the symbolic form of all of your
[01:03:15] the symbolic form of all of your functions um work out their derivatives
[01:03:19] functions um work out their derivatives and do the entire thing for you um so
[01:03:23] and do the entire thing for you um so early on there was a
[01:03:26] early on there was a pioneering um deep learning framework
[01:03:29] pioneering um deep learning framework theano principally from the um
[01:03:31] theano principally from the um university of Montreal which attempted
[01:03:35] university of Montreal which attempted to do precisely that that you had the
[01:03:37] to do precisely that that you had the entire forward path computation started
[01:03:40] entire forward path computation started in symbolic form and it just did the
[01:03:43] in symbolic form and it just did the entire thing for you and worked out the
[01:03:47] entire thing for you and worked out the backward pass
[01:03:49] backward pass automatically um but you know somehow
[01:03:52] automatically um but you know somehow that sort of proved to be um too
[01:03:56] that sort of proved to be um too heavyweight or um hard to deal with
[01:03:59] heavyweight or um hard to deal with different things or people just like to
[01:04:01] different things or people just like to write their own python or whatever it is
[01:04:04] write their own python or whatever it is um so that idea did not fully succeed
[01:04:08] um so that idea did not fully succeed and so what in practice all of the
[01:04:11] and so what in practice all of the current main Frameworks have fallen back
[01:04:14] current main Frameworks have fallen back on is something that's actually less
[01:04:18] on is something that's actually less automated than that so it's sort of like
[01:04:20] automated than that so it's sort of like we've gone backwards in time but the
[01:04:22] we've gone backwards in time but the software's g a lot better really it's a
[01:04:24] software's g a lot better really it's a lot staer and faster um so all of the
[01:04:28] lot staer and faster um so all of the modern deep learning Frameworks sort of
[01:04:31] modern deep learning Frameworks sort of say look I will manage the computation
[01:04:34] say look I will manage the computation graph for you and I can run the forward
[01:04:37] graph for you and I can run the forward propagation path and the backward
[01:04:38] propagation path and the backward propagation path but you're going to
[01:04:41] propagation path but you're going to have to work out the local um
[01:04:43] have to work out the local um derivatives yourself um so if you're if
[01:04:47] derivatives yourself um so if you're if you're putting in a layer or putting in
[01:04:51] you're putting in a layer or putting in um you know a function like an
[01:04:53] um you know a function like an activation function in the in a neural
[01:04:56] activation function in the in a neural network your class your python class
[01:05:00] network your class your python class that represents that you're going to
[01:05:02] that represents that you're going to have to tell me what the forward um
[01:05:05] have to tell me what the forward um computation is and what the local
[01:05:08] computation is and what the local gradient is and I'm just going to call
[01:05:10] gradient is and I'm just going to call your local gradient and assume it's
[01:05:13] your local gradient and assume it's correct um so there's a bit more that
[01:05:16] correct um so there's a bit more that has to be done manually so so the part
[01:05:19] has to be done manually so so the part that's automated then um is that you
[01:05:23] that's automated then um is that you know when you know precisely this code
[01:05:27] know when you know precisely this code obviously but roughly you know inside
[01:05:30] obviously but roughly you know inside the Deep learning software um it's
[01:05:33] the Deep learning software um it's Computing with a computation graph and
[01:05:36] Computing with a computation graph and it's got a forward and a backward and
[01:05:37] it's got a forward and a backward and it's doing what I presented on the
[01:05:39] it's doing what I presented on the pictures before so for the forward um
[01:05:43] pictures before so for the forward um pass it's topologically sorting all the
[01:05:47] pass it's topologically sorting all the nodes of the graph and then it's going
[01:05:49] nodes of the graph and then it's going through them and for each node in the
[01:05:52] through them and for each node in the graph it's calling its forward function
[01:05:55] graph it's calling its forward function which will be able to compute its local
[01:05:58] which will be able to compute its local value in terms of its inputs which have
[01:06:01] value in terms of its inputs which have already been calculated because it's
[01:06:03] already been calculated because it's topologically sorted and then it's um
[01:06:06] topologically sorted and then it's um running the backward pass and in the
[01:06:08] running the backward pass and in the backward pass you're reversing your
[01:06:10] backward pass you're reversing your topological sort and then you're working
[01:06:13] topological sort and then you're working out um the gradient which is going to be
[01:06:17] out um the gradient which is going to be the multiplication of the Upstream error
[01:06:19] the multiplication of the Upstream error signal times your local gradient and so
[01:06:22] signal times your local gradient and so what a human being has to implement um
[01:06:27] what a human being has to implement um is that for anything whether it's a
[01:06:29] is that for anything whether it's a single gate here's a multiply gate or a
[01:06:32] single gate here's a multiply gate or a newal network layer you have to
[01:06:34] newal network layer you have to implement a forward pass and a backward
[01:06:37] implement a forward pass and a backward pass so here for my baby example since
[01:06:41] pass so here for my baby example since we're just doing multiplication my
[01:06:43] we're just doing multiplication my forward pass is that I just multiply the
[01:06:46] forward pass is that I just multiply the two numbers and um return it so I'm
[01:06:49] two numbers and um return it so I'm specifying that for the local node and
[01:06:52] specifying that for the local node and then the other part is that I have to
[01:06:55] then the other part is that I have to work out those gradients and well we
[01:06:59] work out those gradients and well we sort of know how to do that because
[01:07:00] sort of know how to do that because that's the examples that we've been
[01:07:02] that's the examples that we've been doing here um but notice that there's
[01:07:05] doing here um but notice that there's sort of a trick right for what I've got
[01:07:07] sort of a trick right for what I've got Now you kind of can't write down what
[01:07:11] Now you kind of can't write down what the gradients are cuz you know what
[01:07:16] the gradients are cuz you know what these cuz you know backward is just
[01:07:18] these cuz you know backward is just taking as an input the Upstream gradient
[01:07:21] taking as an input the Upstream gradient and you can't work out what the
[01:07:24] and you can't work out what the downstream gradient are going to be
[01:07:26] downstream gradient are going to be unless you know what function values
[01:07:29] unless you know what function values you're calculating it at um so the
[01:07:32] you're calculating it at um so the standard trick that all which is how
[01:07:35] standard trick that all which is how everyone writes this code is you're
[01:07:37] everyone writes this code is you're relying on the fact that the forward is
[01:07:39] relying on the fact that the forward is being calculated before the backward and
[01:07:42] being calculated before the backward and so your forward method um shoves into
[01:07:46] so your forward method um shoves into some local variables of the class what
[01:07:49] some local variables of the class what the values of the inputs are and then
[01:07:52] the values of the inputs are and then you have them available um so when you
[01:07:54] you have them available um so when you get to the back backward pass you can do
[01:07:57] get to the back backward pass you can do what we did before um that um the DX is
[01:08:02] what we did before um that um the DX is going to be the Upstream error signal
[01:08:05] going to be the Upstream error signal times the opposite input and um and
[01:08:08] times the opposite input and um and similarly for Dy and that's going to
[01:08:11] similarly for Dy and that's going to give us the
[01:08:12] give us the answer okay um just two last things then
[01:08:17] answer okay um just two last things then to mention yeah so doing this um
[01:08:22] to mention yeah so doing this um your you need to write um you need to
[01:08:25] your you need to write um you need to get the math right for what's the
[01:08:28] get the math right for what's the derivative of your function so you get
[01:08:31] derivative of your function so you get the right backward calculation so the
[01:08:34] the right backward calculation so the standard way to check that you've got
[01:08:37] standard way to check that you've got the right backward calculation is to do
[01:08:40] the right backward calculation is to do manual gradient checking um with numeric
[01:08:44] manual gradient checking um with numeric gradients so the way you do that um is
[01:08:49] gradients so the way you do that um is you sort of like for the couple of
[01:08:51] you sort of like for the couple of examples I did when I said oh let's
[01:08:53] examples I did when I said oh let's check it by for
[01:08:55] check it by for going from 1 to 1.1 what should the
[01:08:58] going from 1 to 1.1 what should the slope be approximately we're going to do
[01:09:00] slope be approximately we're going to do that in an automated way and so we're
[01:09:03] that in an automated way and so we're going to say at the value X let's
[01:09:06] going to say at the value X let's estimate what the gradient should be and
[01:09:08] estimate what the gradient should be and the way to do that is to pick a small H
[01:09:12] the way to do that is to pick a small H there isn't a magical number because it
[01:09:14] there isn't a magical number because it depends on the function but typically
[01:09:16] depends on the function but typically you know for neural networks around 10us
[01:09:18] you know for neural networks around 10us 4 is good um a small H and work out the
[01:09:23] 4 is good um a small H and work out the function value I forward part
[01:09:26] function value I forward part at x + H and x - H divided by the run
[01:09:31] at x + H and x - H divided by the run which is 2 H and that should give you an
[01:09:34] which is 2 H and that should give you an estimate of the slope what the backward
[01:09:37] estimate of the slope what the backward pass is calculating and you want those
[01:09:40] pass is calculating and you want those two numbers to be approximately equal
[01:09:43] two numbers to be approximately equal you know within some 10us 2 of each
[01:09:46] you know within some 10us 2 of each other and then probably you're
[01:09:47] other and then probably you're calculating the gradient right and if
[01:09:50] calculating the gradient right and if they aren't equal um that um you
[01:09:56] they aren't equal um that um you probably have made a mistake um yeah so
[01:09:59] probably have made a mistake um yeah so um note that this formul for the version
[01:10:02] um note that this formul for the version I did for my examples I just compared to
[01:10:06] I did for my examples I just compared to x with x + H right I did a one-sided
[01:10:10] x with x + H right I did a one-sided estimate which is normally what you get
[01:10:12] estimate which is normally what you get taught in a math class if you're doing
[01:10:15] taught in a math class if you're doing this to check your gradients numerically
[01:10:18] this to check your gradients numerically you're far far better off doing this
[01:10:21] you're far far better off doing this two-sided estimate because it's much
[01:10:23] two-sided estimate because it's much more accurate and stable when you're
[01:10:24] more accurate and stable when you're doing it equally around both sides of
[01:10:27] doing it equally around both sides of your H um yeah so this looks easy to do
[01:10:32] your H um yeah so this looks easy to do um if if this was just so good why
[01:10:36] um if if this was just so good why doesn't everyone do this all the time
[01:10:37] doesn't everyone do this all the time and forget about calculus um you know
[01:10:40] and forget about calculus um you know the reason you don't want to do this is
[01:10:43] the reason you don't want to do this is that doing this is incredibly slow right
[01:10:46] that doing this is incredibly slow right because you have to repeat this
[01:10:48] because you have to repeat this computation for every parameter of your
[01:10:51] computation for every parameter of your model that you're not getting the kind
[01:10:53] model that you're not getting the kind of speed UPS your getting from the um
[01:10:57] of speed UPS your getting from the um back propagation algorithm but you know
[01:10:59] back propagation algorithm but you know it's useful for checking your
[01:11:00] it's useful for checking your implementation is correct you know in
[01:11:02] implementation is correct you know in the old days before Frameworks like py
[01:11:05] the old days before Frameworks like py torch um you know we used to write
[01:11:08] torch um you know we used to write everything by hand and people often got
[01:11:10] everything by hand and people often got things wrong um but nowadays you know
[01:11:13] things wrong um but nowadays you know it's less needed but it's good to check
[01:11:14] it's less needed but it's good to check that if you've implemented your own new
[01:11:16] that if you've implemented your own new layer that it's doing the right
[01:11:19] layer that it's doing the right thing okay um yeah so that's everything
[01:11:23] thing okay um yeah so that's everything that we need to know about new Nets
[01:11:25] that we need to know about new Nets propagation is the chain rule applied
[01:11:28] propagation is the chain rule applied efficiently forward pass is just
[01:11:30] efficiently forward pass is just function application backward pass is
[01:11:33] function application backward pass is chain rule applied inefficiently um so
[01:11:38] chain rule applied inefficiently um so you know uh we're going to inflict pain
[01:11:41] you know uh we're going to inflict pain on our students by making them do some
[01:11:44] on our students by making them do some math and calculate some of these things
[01:11:47] math and calculate some of these things and um do the homework and I know
[01:11:50] and um do the homework and I know that'll be harder for some of you than
[01:11:52] that'll be harder for some of you than others um you know that in some sense
[01:11:57] others um you know that in some sense you don't actually need to know how to
[01:11:58] you don't actually need to know how to do this the beauty of these modern deep
[01:12:00] do this the beauty of these modern deep learning Frameworks is they'll do it all
[01:12:02] learning Frameworks is they'll do it all for you they predefine common layer
[01:12:05] for you they predefine common layer types and you can just plug them
[01:12:06] types and you can just plug them together like pieces of Lego and they'll
[01:12:08] together like pieces of Lego and they'll be computed right and this is precisely
[01:12:11] be computed right and this is precisely the reason that high school students
[01:12:13] the reason that high school students across the country and the world can now
[01:12:15] across the country and the world can now do deep learning projects for their
[01:12:18] do deep learning projects for their science fairs because you don't actually
[01:12:20] science fairs because you don't actually have to understand any of this math um
[01:12:22] have to understand any of this math um you can just use what's given to you um
[01:12:25] you can just use what's given to you um but you know um we kind of uh want to
[01:12:28] but you know um we kind of uh want to hope that you actually do understand
[01:12:31] hope that you actually do understand something about what's going on under
[01:12:33] something about what's going on under the hood and how new networks work so
[01:12:36] the hood and how new networks work so therefore we make you suffer a little
[01:12:37] therefore we make you suffer a little bit and of course you know if you sort
[01:12:41] bit and of course you know if you sort of wanting to look at and understand
[01:12:44] of wanting to look at and understand more complex things you need to have
[01:12:46] more complex things you need to have some sense of what's going on so later
[01:12:48] some sense of what's going on so later on when we get on to a current new
[01:12:50] on when we get on to a current new networks we'll talk a bit about things
[01:12:53] networks we'll talk a bit about things like exploding and Vanishing great
[01:12:55] like exploding and Vanishing great and if you want to have some
[01:12:57] and if you want to have some understanding about well why things
[01:12:58] understanding about well why things aren't working and things are going
[01:13:00] aren't working and things are going wrong um then you sort of want to know
[01:13:03] wrong um then you sort of want to know what it's actually calculating rather
[01:13:04] what it's actually calculating rather than just thinking it's all a black box
[01:13:07] than just thinking it's all a black box magic and so that's why we hope to have
[01:13:09] magic and so that's why we hope to have uh taught something about that okay I
[01:13:13] uh taught something about that okay I think I'm done if the audience is
[01:13:15] think I'm done if the audience is sufficiently stunned um and we can stop
[01:13:18] sufficiently stunned um and we can stop for today okay thank you

Stanford CS224N: NLP with Deep Learning | Spring 2024 | Lecture 4 - Dependency Parsing

Source: https://www.youtube.com/watch?v=KVKvde-_MYc

---

Transcript

[00:00:05] okay hi
[00:00:06] okay hi everyone okay so for today we're going
[00:00:09] everyone okay so for today we're going to um you know I guess do a 180 from
[00:00:13] to um you know I guess do a 180 from where we were on Tuesday and so today um
[00:00:18] where we were on Tuesday and so today um I'm going to talk about um syntactic
[00:00:22] I'm going to talk about um syntactic structure linguistic structure of human
[00:00:24] structure linguistic structure of human language sentences dependency passing
[00:00:28] language sentences dependency passing and how well dependency and then how you
[00:00:30] and how well dependency and then how you go about building dependency paes so um
[00:00:34] go about building dependency paes so um we're um solidly inter linguistic Zone
[00:00:38] we're um solidly inter linguistic Zone today um how many people in the audience
[00:00:41] today um how many people in the audience have done a Linguistics class yay okay
[00:00:45] have done a Linguistics class yay okay there's some people have done
[00:00:46] there's some people have done Linguistics classes okay great um and
[00:00:50] Linguistics classes okay great um and for the rest of you well you know this
[00:00:51] for the rest of you well you know this is your chance to see a little bit of
[00:00:53] is your chance to see a little bit of human language structure and if you like
[00:00:55] human language structure and if you like it you can enroll in a Linguistics class
[00:00:57] it you can enroll in a Linguistics class later on um yeah so
[00:01:02] oops so um you know so assignment two we
[00:01:06] oops so um you know so assignment two we handed out on Tuesday so in the second
[00:01:09] handed out on Tuesday so in the second half of assignment two um what your job
[00:01:12] half of assignment two um what your job to do is to build a neural dependency
[00:01:14] to do is to build a neural dependency paard using p torch um as you'll s we'll
[00:01:18] paard using p torch um as you'll s we'll sort of come to later on really the bit
[00:01:21] sort of come to later on really the bit that you have to build is just the
[00:01:22] that you have to build is just the machine learning bit of making decisions
[00:01:25] machine learning bit of making decisions and really we give you most of the rest
[00:01:27] and really we give you most of the rest of the new dependency paa but this is
[00:01:30] of the new dependency paa but this is also then a chance to remind you that
[00:01:32] also then a chance to remind you that assignment two in that second half uses
[00:01:35] assignment two in that second half uses py torch one of the leading deep
[00:01:37] py torch one of the leading deep learning Frameworks um so if you're not
[00:01:40] learning Frameworks um so if you're not familiar with that it'd be a really good
[00:01:41] familiar with that it'd be a really good idea to also go along to um the Friday
[00:01:45] idea to also go along to um the Friday pie torch tutorial um though we have
[00:01:48] pie torch tutorial um though we have tried to make assignment too so it's a
[00:01:50] tried to make assignment too so it's a fairly good place for learning pie torch
[00:01:53] fairly good place for learning pie torch as you go along um we'll say more soon
[00:01:56] as you go along um we'll say more soon about um final projects but you're
[00:01:58] about um final projects but you're certainly already in encourage to come
[00:02:00] certainly already in encourage to come and meet with Tas or me about final
[00:02:03] and meet with Tas or me about final projects and we're putting up
[00:02:04] projects and we're putting up information about the T so you can know
[00:02:06] information about the T so you can know more about them on um the office hours
[00:02:10] more about them on um the office hours page okay so let's get straight into it
[00:02:13] page okay so let's get straight into it and start looking at linguistic
[00:02:15] and start looking at linguistic structure so um in in um thinking about
[00:02:21] structure so um in in um thinking about linguistic structure of human languages
[00:02:24] linguistic structure of human languages there are two primary ways um that
[00:02:28] there are two primary ways um that people have thought about it so one way
[00:02:31] people have thought about it so one way is um using the idea that linguists
[00:02:34] is um using the idea that linguists normally call phrase structure which is
[00:02:37] normally call phrase structure which is then represented in terms of what um
[00:02:40] then represented in terms of what um computer scientists normally know as
[00:02:43] computer scientists normally know as context free grammars so I'm going to
[00:02:45] context free grammars so I'm going to spend a couple of minutes um going over
[00:02:48] spend a couple of minutes um going over that view of things but you know
[00:02:50] that view of things but you know actually it's not the main one that I'm
[00:02:52] actually it's not the main one that I'm going to talk about in this class I'm
[00:02:54] going to talk about in this class I'm going to spend most of this class
[00:02:56] going to spend most of this class talking about an alternative way of
[00:02:58] talking about an alternative way of thinking about things called dependency
[00:03:00] thinking about things called dependency grammars um there are actually some
[00:03:02] grammars um there are actually some correspondences you can make between the
[00:03:04] correspondences you can make between the two ways of thinking about things but
[00:03:06] two ways of thinking about things but I'm not going to go through into those
[00:03:08] I'm not going to go through into those um here today so for the constituency
[00:03:11] um here today so for the constituency grammar or phrase structure um version
[00:03:14] grammar or phrase structure um version of things the way that you go about
[00:03:17] of things the way that you go about thinking about the structure of human
[00:03:19] thinking about the structure of human languages is well there are words
[00:03:21] languages is well there are words languages have lots of words hundreds of
[00:03:24] languages have lots of words hundreds of thousands of words but it seems like a
[00:03:27] thousands of words but it seems like a lot of the words nearly all the words in
[00:03:30] lot of the words nearly all the words in fact fall into a few basic classes that
[00:03:34] fact fall into a few basic classes that represent their nature and how they
[00:03:37] represent their nature and how they behave in sentences so for words like
[00:03:41] behave in sentences so for words like the examples here we have nouns so cat
[00:03:44] the examples here we have nouns so cat is a noun door is a noun but you know
[00:03:48] is a noun door is a noun but you know something like Linguistics is also a
[00:03:50] something like Linguistics is also a noun so we have nouns um and then we
[00:03:54] noun so we have nouns um and then we have other kinds of words so something
[00:03:56] have other kinds of words so something like cuddly is an adjective a word that
[00:03:58] like cuddly is an adjective a word that can modify nouns um and then we have
[00:04:02] can modify nouns um and then we have theth for the the cuddly cat um the is
[00:04:07] theth for the the cuddly cat um the is sort of a slightly more complex one as
[00:04:10] sort of a slightly more complex one as to how to name um normally in modern
[00:04:13] to how to name um normally in modern Linguistics refer to words like that as
[00:04:15] Linguistics refer to words like that as a determiner you might also see the name
[00:04:18] a determiner you might also see the name article and when sometimes when people
[00:04:21] article and when sometimes when people try to shoehorn um human language into
[00:04:25] try to shoehorn um human language into eight part of speech categories that
[00:04:27] eight part of speech categories that they say it's an adjective that doesn't
[00:04:28] they say it's an adjective that doesn't really behave like regular adjectives
[00:04:31] really behave like regular adjectives and then we have words like by or
[00:04:32] and then we have words like by or through or on and to and ones like that
[00:04:35] through or on and to and ones like that and so they're then prepositions right
[00:04:37] and so they're then prepositions right so we have these classes and with lots
[00:04:40] so we have these classes and with lots of words fitting into each class and so
[00:04:42] of words fitting into each class and so they're referred to conventionally as
[00:04:44] they're referred to conventionally as parts of speech but then once we've got
[00:04:47] parts of speech but then once we've got words we start putting them into bigger
[00:04:50] words we start putting them into bigger units so the cuddly cat is some kind of
[00:04:53] units so the cuddly cat is some kind of unit and so it seems like this is a um
[00:04:57] unit and so it seems like this is a um explication of a noun cat and so this
[00:05:01] explication of a noun cat and so this gets referred to as a noun phrase um and
[00:05:04] gets referred to as a noun phrase um and then by the door well this is a phrase
[00:05:08] then by the door well this is a phrase um but actually it has inside it this
[00:05:10] um but actually it has inside it this the door and that's a noun phrase but
[00:05:14] the door and that's a noun phrase but this bigger unit here of by the door is
[00:05:17] this bigger unit here of by the door is then a prepositional
[00:05:19] then a prepositional phrase and we can continue to build
[00:05:22] phrase and we can continue to build bigger units so inside you know this we
[00:05:27] bigger units so inside you know this we have this phrase that we've already
[00:05:28] have this phrase that we've already looked at with the noun phrase and a
[00:05:30] looked at with the noun phrase and a prepositional phrase but then we can
[00:05:32] prepositional phrase but then we can have another noun phrase the cuddly cat
[00:05:36] have another noun phrase the cuddly cat and we can put them together and build a
[00:05:39] and we can put them together and build a bigger noun phrase the cuddly Cat by the
[00:05:43] bigger noun phrase the cuddly Cat by the door and so to represent this you can
[00:05:46] door and so to represent this you can start to write um a phrase structure
[00:05:49] start to write um a phrase structure grammar or a context free grammar that
[00:05:52] grammar or a context free grammar that represents what are the possibilities
[00:05:54] represents what are the possibilities for building up sentences here in
[00:05:56] for building up sentences here in English those similar kinds of phrase
[00:05:58] English those similar kinds of phrase structure grammars can be written for
[00:06:00] structure grammars can be written for other languages so this is sort of
[00:06:03] other languages so this is sort of starting to give you possible structures
[00:06:05] starting to give you possible structures for a noun phrase so you can have a noun
[00:06:09] for a noun phrase so you can have a noun phrase just goes to a determiner
[00:06:12] phrase just goes to a determiner followed by a noun but then as well as
[00:06:15] followed by a noun but then as well as the cat and a dog you can have the large
[00:06:19] the cat and a dog you can have the large cats so you might say that okay rather
[00:06:21] cats so you might say that okay rather than that I might want to have as a
[00:06:23] than that I might want to have as a better rule that a noun phrase goes to a
[00:06:27] better rule that a noun phrase goes to a determiner and optional adjective
[00:06:30] determiner and optional adjective and then a noun if you think about it
[00:06:33] and then a noun if you think about it you can sort of have multiple adjectives
[00:06:36] you can sort of have multiple adjectives so you can have the the large um green
[00:06:41] so you can have the the large um green cat or something like that um so you can
[00:06:45] cat or something like that um so you can really get multiple adjectives that are
[00:06:46] really get multiple adjectives that are heing and that sort of star the clean
[00:06:49] heing and that sort of star the clean star says you can have lots of them um
[00:06:52] star says you can have lots of them um the large cuddly green cat um but then
[00:06:56] the large cuddly green cat um but then you can stick things after the now
[00:06:59] you can stick things after the now phrase so you can put these
[00:07:01] phrase so you can put these prepositional phrases like in a crate so
[00:07:04] prepositional phrases like in a crate so we might also want to say that a noun
[00:07:05] we might also want to say that a noun phrase can be Rewritten as a noun phrase
[00:07:09] phrase can be Rewritten as a noun phrase followed by a prepositional phrase where
[00:07:12] followed by a prepositional phrase where a prepositional phrase um can be
[00:07:15] a prepositional phrase um can be represented by preposition followed by a
[00:07:17] represented by preposition followed by a noun phrase and somewhere we're also
[00:07:19] noun phrase and somewhere we're also going to want to represent our parts of
[00:07:22] going to want to represent our parts of speech membership so a
[00:07:25] speech membership so a determiner um can go to words like a or
[00:07:30] determiner um can go to words like a or and an adjective can go to words like
[00:07:35] and an adjective can go to words like large cuddly or many other words that
[00:07:39] large cuddly or many other words that I'm not going to write
[00:07:40] I'm not going to write down and a preposition can go to words
[00:07:45] down and a preposition can go to words like in on
[00:07:48] like in on under Etc after that okay so now I've
[00:07:52] under Etc after that okay so now I've got a a little grammar here and this
[00:07:55] got a a little grammar here and this little grammar here could sort of make
[00:07:58] little grammar here could sort of make everything
[00:08:00] everything I've gotten these sentences well
[00:08:02] I've gotten these sentences well actually can do this one too it can do
[00:08:03] actually can do this one too it can do the large barking one where there are
[00:08:05] the large barking one where there are multiple ones um but then if I start
[00:08:08] multiple ones um but then if I start going Beyond these noun phrases and say
[00:08:13] going Beyond these noun phrases and say think of sentences like talk to the cat
[00:08:16] think of sentences like talk to the cat or talk to the large cuddly dog by the
[00:08:21] or talk to the large cuddly dog by the door well now I've got here a verb talk
[00:08:25] door well now I've got here a verb talk and I've again got a
[00:08:27] and I've again got a preposition so I might um then have more
[00:08:30] preposition so I might um then have more rules that says I can have a verb phrase
[00:08:33] rules that says I can have a verb phrase and the verb phrase can go to a verb and
[00:08:37] and the verb phrase can go to a verb and then a prepositional phrase and then
[00:08:40] then a prepositional phrase and then that could explain these two sentences
[00:08:42] that could explain these two sentences as well and in this kind of way I can
[00:08:45] as well and in this kind of way I can start to build up a grammar of the
[00:08:48] start to build up a grammar of the structure of English sentences as a
[00:08:51] structure of English sentences as a context free grammar make sense yeah
[00:08:55] context free grammar make sense yeah okay um and so that's
[00:09:00] okay um and so that's um what is being quite commonly done in
[00:09:04] um what is being quite commonly done in linguistics and elsewhere
[00:09:07] linguistics and elsewhere um okay
[00:09:10] um okay uh yeah uh so let me just do that once
[00:09:14] uh yeah uh so let me just do that once more but behind in this one so one thing
[00:09:18] more but behind in this one so one thing I can do here is say oh I have I'm going
[00:09:23] I can do here is say oh I have I'm going to look at this with its phrase
[00:09:26] to look at this with its phrase structure and if I write it upside down
[00:09:28] structure and if I write it upside down to give myself some space for later you
[00:09:31] to give myself some space for later you know I could start
[00:09:33] know I could start um making a phrase structure that is of
[00:09:38] um making a phrase structure that is of this
[00:09:38] this sentence um and I'll start to run out of
[00:09:41] sentence um and I'll start to run out of space but I can sort of start to make
[00:09:46] space but I can sort of start to make this phrase structure of the sentence so
[00:09:49] this phrase structure of the sentence so that's um phrase structure um but
[00:09:52] that's um phrase structure um but there's another form of
[00:09:54] there's another form of representation that has been fairly
[00:09:56] representation that has been fairly widely used in linguistics and we're
[00:09:59] widely used in linguistics and we're going and has been commonly used in NLP
[00:10:01] going and has been commonly used in NLP and we're going to use for the paes we
[00:10:03] and we're going to use for the paes we built and that's dependency structure so
[00:10:06] built and that's dependency structure so dependency structure represents things
[00:10:08] dependency structure represents things in a slightly different way it thinks
[00:10:11] in a slightly different way it thinks about words that are the main word or
[00:10:14] about words that are the main word or head or something and then which words
[00:10:17] head or something and then which words they take as modifiers or arguments so
[00:10:20] they take as modifiers or arguments so for look in the large crate in the
[00:10:22] for look in the large crate in the kitchen by the door well this is um
[00:10:26] kitchen by the door well this is um describing a looking command so that the
[00:10:29] describing a looking command so that the head of the whole thing is looking um
[00:10:32] head of the whole thing is looking um and then looking is taking one or more
[00:10:36] and then looking is taking one or more arguments or modifiers and well what the
[00:10:40] arguments or modifiers and well what the looking is saying here is well what you
[00:10:43] looking is saying here is well what you want to do is look in the large crate so
[00:10:48] want to do is look in the large crate so we are looking in something and then
[00:10:51] we are looking in something and then when what we're looking in is a crate
[00:10:54] when what we're looking in is a crate and then the crate has some modifiers
[00:10:57] and then the crate has some modifiers it's a large crate it's the large crate
[00:11:01] it's a large crate it's the large crate and then the crate um is also um placed
[00:11:06] and then the crate um is also um placed somewhere it's placed in the kitchen so
[00:11:11] somewhere it's placed in the kitchen so that in the kitchen is also modifying
[00:11:14] that in the kitchen is also modifying crate um and then we've got over here
[00:11:17] crate um and then we've got over here the by the door um
[00:11:20] the by the door um well the by the door um is also
[00:11:24] well the by the door um is also modifying crate so we've also got a link
[00:11:27] modifying crate so we've also got a link down over to here and that gives us our
[00:11:30] down over to here and that gives us our piece of structure here which having
[00:11:32] piece of structure here which having filled that in makes me realize I
[00:11:34] filled that in makes me realize I actually got it wrong in when I was
[00:11:36] actually got it wrong in when I was doing the constituency representation
[00:11:40] doing the constituency representation whoopsie so I should get my CU in the
[00:11:43] whoopsie so I should get my CU in the constituency representation I made the
[00:11:46] constituency representation I made the kitchen by the door into a phrase that
[00:11:48] kitchen by the door into a phrase that was actually wrong whoops bad me um so
[00:11:51] was actually wrong whoops bad me um so what I should have actually had um was
[00:11:56] what I should have actually had um was we had another prepositional phrase that
[00:11:59] we had another prepositional phrase that went to a noun phrase of in the kitchen
[00:12:03] went to a noun phrase of in the kitchen and then both of
[00:12:07] those were coming off a bigger noun
[00:12:11] those were coming off a bigger noun phrase like that whoopsie okay um I get
[00:12:15] phrase like that whoopsie okay um I get it right most of the time okay but so
[00:12:17] it right most of the time okay but so this idea of um dependency structure is
[00:12:20] this idea of um dependency structure is that we're sort of finding what is the
[00:12:23] that we're sort of finding what is the head word and then we're saying which
[00:12:25] head word and then we're saying which things modify the headword and either of
[00:12:28] things modify the headword and either of these represent ations we can be using
[00:12:31] these represent ations we can be using um to sort of work out what the
[00:12:34] um to sort of work out what the structure of sentences is in terms of
[00:12:37] structure of sentences is in terms of which words go together and which words
[00:12:39] which words go together and which words modify other words and so the basic idea
[00:12:43] modify other words and so the basic idea is so when humans communicate we
[00:12:46] is so when humans communicate we communicate in a linear stream right so
[00:12:49] communicate in a linear stream right so that if it's um conventional writing
[00:12:52] that if it's um conventional writing systems it's a linear stream of words
[00:12:55] systems it's a linear stream of words that you're reading if it's spoken
[00:12:58] that you're reading if it's spoken language like you're understanding me
[00:13:00] language like you're understanding me speaking right now it's not a linear
[00:13:03] speaking right now it's not a linear stream of words it's a linear Sound
[00:13:05] stream of words it's a linear Sound Stream and you know like when people
[00:13:09] Stream and you know like when people speak there aren't any you know there
[00:13:10] speak there aren't any you know there isn't white space between words when
[00:13:13] isn't white space between words when people speak you know occasionally
[00:13:14] people speak you know occasionally people pause at the end of a clause or
[00:13:17] people pause at the end of a clause or sentence or something but in general I'm
[00:13:19] sentence or something but in general I'm just sort of speaking continuous words
[00:13:21] just sort of speaking continuous words that run one into each other so that
[00:13:23] that run one into each other so that there's a a a linear sequence of sounds
[00:13:26] there's a a a linear sequence of sounds coming out my mouth um and you have to
[00:13:28] coming out my mouth um and you have to do all bit like that but the but if
[00:13:31] do all bit like that but the but if you're then thinking oh gee
[00:13:34] you're then thinking oh gee um I can actually understand Chris
[00:13:38] um I can actually understand Chris talking then somehow you're taking that
[00:13:41] talking then somehow you're taking that linear stream and you're turning it into
[00:13:44] linear stream and you're turning it into a meaning where certain words are
[00:13:47] a meaning where certain words are modifying other words and you have these
[00:13:50] modifying other words and you have these bigger units um like constituents that
[00:13:52] bigger units um like constituents that are understanding the meaning of the
[00:13:54] are understanding the meaning of the sentence and so human listeners need to
[00:13:58] sentence and so human listeners need to work out what modifies what to be able
[00:14:01] work out what modifies what to be able to understand sentences
[00:14:04] to understand sentences correctly and so similarly our models
[00:14:08] correctly and so similarly our models need to be able to understand sentence
[00:14:10] need to be able to understand sentence structure in order to be able to
[00:14:12] structure in order to be able to interpret language correctly and so for
[00:14:15] interpret language correctly and so for what we're going to be doing for
[00:14:16] what we're going to be doing for building dependency paes is we're going
[00:14:19] building dependency paes is we're going to be
[00:14:20] to be explicitly building a neural network
[00:14:23] explicitly building a neural network model that says let's find the structure
[00:14:26] model that says let's find the structure of these sentences um in way we actually
[00:14:29] of these sentences um in way we actually move away from that later on because
[00:14:32] move away from that later on because when we move into um Transformer
[00:14:34] when we move into um Transformer language models they just take in the um
[00:14:38] language models they just take in the um sequence of words but actually inside
[00:14:41] sequence of words but actually inside the parameters of the newal network they
[00:14:43] the parameters of the newal network they recognizing and building the same kind
[00:14:45] recognizing and building the same kind of structural units and we'll talk about
[00:14:47] of structural units and we'll talk about that later in the class um to give you
[00:14:50] that later in the class um to give you more of a sense of how um these you know
[00:14:53] more of a sense of how um these you know understanding what modifies what is
[00:14:56] understanding what modifies what is important for interpretation um here are
[00:14:58] important for interpretation um here are a few funny examples from newspaper
[00:15:01] a few funny examples from newspaper headlines and they're funny examples
[00:15:03] headlines and they're funny examples because you get um there sentences don't
[00:15:07] because you get um there sentences don't just have one way of interpreting them
[00:15:10] just have one way of interpreting them um when you have a sequence of words
[00:15:13] um when you have a sequence of words commonly in human languages sequence of
[00:15:16] commonly in human languages sequence of words are ambiguous and it's relying on
[00:15:19] words are ambiguous and it's relying on human interpretation of what makes sense
[00:15:22] human interpretation of what makes sense and what goes together to work out how
[00:15:24] and what goes together to work out how to read them so here's a first example
[00:15:29] to read them so here's a first example um scientists count whales from space um
[00:15:33] um scientists count whales from space um now that's um ambiguous and you can give
[00:15:36] now that's um ambiguous and you can give this two possible readings so how can
[00:15:39] this two possible readings so how can you give this um headline two possible
[00:15:43] you give this um headline two possible readings yeah one is that they scien in
[00:15:47] readings yeah one is that they scien in space counting whales and the other one
[00:15:49] space counting whales and the other one is that they're whales from in space
[00:15:51] is that they're whales from in space yeah so one possibility is so we've got
[00:15:54] yeah so one possibility is so we've got this prepositional phrase this is a
[00:15:56] this prepositional phrase this is a prepositional phrase here one
[00:15:59] prepositional phrase here one possibility is that this prepositional
[00:16:02] possibility is that this prepositional phrase is
[00:16:04] phrase is modifying or is the OB yeah it's
[00:16:07] modifying or is the OB yeah it's modifying
[00:16:09] modifying whales um so they're whales from space
[00:16:13] whales um so they're whales from space and the other possibility is that it's
[00:16:16] and the other possibility is that it's the counting that's happening from space
[00:16:18] the counting that's happening from space so the scientists are counting it from
[00:16:20] so the scientists are counting it from space okay so that corresponds to my two
[00:16:23] space okay so that corresponds to my two pictures here um so in one picture it's
[00:16:27] pictures here um so in one picture it's the counting that is happening from
[00:16:31] the counting that is happening from space um which is actually the right
[00:16:34] space um which is actually the right interpretation of what the article is
[00:16:35] interpretation of what the article is about um but in the other interpretation
[00:16:38] about um but in the other interpretation we have space whales um and the and the
[00:16:41] we have space whales um and the and the scientists are counting the space whales
[00:16:43] scientists are counting the space whales that are arriving or something like that
[00:16:46] that are arriving or something like that and so then we have um the from space
[00:16:50] and so then we have um the from space that are modifying the
[00:16:51] that are modifying the whales okay um so what we have here
[00:16:55] whales okay um so what we have here right is a prepositional phrase which
[00:16:58] right is a prepositional phrase which comes after a noun phrase it's just a
[00:17:00] comes after a noun phrase it's just a onew noun phrase here whales that's fine
[00:17:03] onew noun phrase here whales that's fine and then before that is a verb and so
[00:17:06] and then before that is a verb and so the thing so one place in English where
[00:17:09] the thing so one place in English where you get a lot of ambiguities is from
[00:17:12] you get a lot of ambiguities is from these prepositional phrases because
[00:17:14] these prepositional phrases because whenever you get prepositional phrases
[00:17:16] whenever you get prepositional phrases and prepositional phrases are really
[00:17:18] and prepositional phrases are really common in English if you think about it
[00:17:20] common in English if you think about it whenever you get them like this it's
[00:17:23] whenever you get them like this it's always ambiguous as
[00:17:26] always ambiguous as to oops it's always amb uous as to what
[00:17:30] to oops it's always amb uous as to what earlier thing in the sentence they
[00:17:32] earlier thing in the sentence they dependent of um and so um you know you
[00:17:39] dependent of um and so um you know you can sort of put in another prepositional
[00:17:41] can sort of put in another prepositional phrase in the morning or something like
[00:17:44] phrase in the morning or something like that and so then the ambiguities just
[00:17:48] that and so then the ambiguities just multiply and so the important thing to
[00:17:51] multiply and so the important thing to notice here about human language is
[00:17:56] notice here about human language is human language is in synta I terms
[00:17:59] human language is in synta I terms globally ambiguous right so in
[00:18:02] globally ambiguous right so in programming languages you have local
[00:18:06] programming languages you have local ambiguities interpretation how many
[00:18:08] ambiguities interpretation how many people have done a compilers class I
[00:18:10] people have done a compilers class I think very few these days anyone done a
[00:18:12] think very few these days anyone done a compilers class okay it looks like less
[00:18:15] compilers class okay it looks like less people have done a compilers class than
[00:18:16] people have done a compilers class than a Linguistics class that's
[00:18:19] a Linguistics class that's interesting okay well I won't make too
[00:18:22] interesting okay well I won't make too many analogies to compilers classes in
[00:18:24] many analogies to compilers classes in you know when I was young you know that
[00:18:27] you know when I was young you know that was still the old days kind CS
[00:18:29] was still the old days kind CS curriculum where writing interpreters
[00:18:31] curriculum where writing interpreters and compilers were seen as the main sta
[00:18:34] and compilers were seen as the main sta of Computer Science Education but no
[00:18:36] of Computer Science Education but no more I guess um yeah so um in
[00:18:40] more I guess um yeah so um in programming languages you can have a
[00:18:42] programming languages you can have a local ambiguity but ambiguities are
[00:18:45] local ambiguity but ambiguities are always resolved right so we have simple
[00:18:47] always resolved right so we have simple rules and programming languages that
[00:18:50] rules and programming languages that else is um construed with the nearest if
[00:18:54] else is um construed with the nearest if um you know it's a bit different in
[00:18:56] um you know it's a bit different in Python because it's indentation but you
[00:18:58] Python because it's indentation but you know There are rules that so there's
[00:19:00] know There are rules that so there's never Global ambiguity in a programming
[00:19:02] never Global ambiguity in a programming language um and but human languages just
[00:19:07] language um and but human languages just aren't like that right that there's
[00:19:09] aren't like that right that there's nothing that resolves which of these two
[00:19:12] nothing that resolves which of these two readings is correct if I made it a
[00:19:14] readings is correct if I made it a bigger sentence that'd still be
[00:19:16] bigger sentence that'd still be ambiguous you're just sort of meant to
[00:19:18] ambiguous you're just sort of meant to read it and use context in your
[00:19:20] read it and use context in your intelligence to decide um what's going
[00:19:23] intelligence to decide um what's going on and so to take a a bigger but real
[00:19:27] on and so to take a a bigger but real example um this is the kind of boring
[00:19:31] example um this is the kind of boring sentence that you can read in the Wall
[00:19:33] sentence that you can read in the Wall Street Journal most mornings um the
[00:19:36] Street Journal most mornings um the board approved its acquisition by Royal
[00:19:38] board approved its acquisition by Royal Trustco limited of Toronto for $27 a
[00:19:41] Trustco limited of Toronto for $27 a share at its monthly meeting um so um
[00:19:45] share at its monthly meeting um so um what you can see in this sentence is
[00:19:47] what you can see in this sentence is we've got a verb and then we've got a
[00:19:50] we've got a verb and then we've got a noun phrase and then what are what after
[00:19:53] noun phrase and then what are what after that we have four prepositional phrases
[00:19:56] that we have four prepositional phrases in a row okay so what do these
[00:19:58] in a row okay so what do these prepositional phrases modify so what
[00:20:01] prepositional phrases modify so what does by Royal Trustco limited
[00:20:06] modify the acquisition right so it's the
[00:20:09] modify the acquisition right so it's the acquisition by Royal Trust Co then of
[00:20:13] acquisition by Royal Trust Co then of Toronto
[00:20:15] Toronto modifies so it's Royal Trustco limited
[00:20:18] modifies so it's Royal Trustco limited of
[00:20:19] of Toronto um so yeah later on
[00:20:22] Toronto um so yeah later on prepositional phrases can also modify
[00:20:25] prepositional phrases can also modify earlier prepositional phrases or at
[00:20:27] earlier prepositional phrases or at least the noun phrase inside them World
[00:20:29] least the noun phrase inside them World Trustco limited okay for $27 a
[00:20:34] Trustco limited okay for $27 a share is back to modifying the
[00:20:38] share is back to modifying the acquisition okay at its monthly
[00:20:41] acquisition okay at its monthly meeting is is yeah it's the approval so
[00:20:45] meeting is is yeah it's the approval so it's gone way back up to there right um
[00:20:49] it's gone way back up to there right um but you know um yeah so you know so if
[00:20:53] but you know um yeah so you know so if you start having sentences um with a
[00:20:56] you start having sentences um with a whole bunch of prepositional phrases
[00:20:58] whole bunch of prepositional phrases like this you can start getting more and
[00:21:01] like this you can start getting more and more ambiguities of attachment I mean
[00:21:04] more ambiguities of attachment I mean you don't get the
[00:21:06] you don't get the full you don't get the sort of full free
[00:21:09] full you don't get the sort of full free choice factorial number of attachment
[00:21:12] choice factorial number of attachment points because there is a restriction
[00:21:15] points because there is a restriction that these dependencies don't cross um
[00:21:19] that these dependencies don't cross um so once you've gone back further you
[00:21:21] so once you've gone back further you have to stay equally far back or go even
[00:21:23] have to stay equally far back or go even back further back again but nevertheless
[00:21:26] back further back again but nevertheless so the number of readings you get is
[00:21:28] so the number of readings you get is Catalan series which is a series you see
[00:21:31] Catalan series which is a series you see in a whole bunch of other places if
[00:21:32] in a whole bunch of other places if you've done any graph Theory or anything
[00:21:34] you've done any graph Theory or anything like that you know if you're doing
[00:21:36] like that you know if you're doing triangulations you get um cat alarms
[00:21:39] triangulations you get um cat alarms because you get the same property um
[00:21:41] because you get the same property um that things don't cross so it's an
[00:21:43] that things don't cross so it's an exponentially growing um sequence of
[00:21:46] exponentially growing um sequence of possible readings and so it quickly gets
[00:21:49] possible readings and so it quickly gets very big so I think when you you've got
[00:21:53] very big so I think when you you've got four prepositional phrases you get 13
[00:21:56] four prepositional phrases you get 13 readings and if you have five you 27 and
[00:22:00] readings and if you have five you 27 and you know grows up from there so you get
[00:22:02] you know grows up from there so you get a lot of ambiguities but the crucial
[00:22:04] a lot of ambiguities but the crucial thing to notice is you know human beings
[00:22:07] thing to notice is you know human beings read sentences like this every morning
[00:22:10] read sentences like this every morning or at least people who work in banking
[00:22:12] or at least people who work in banking do while you know having their Corn
[00:22:14] do while you know having their Corn Flakes and you know they their brain
[00:22:17] Flakes and you know they their brain doesn't explode trying to think about
[00:22:19] doesn't explode trying to think about the 13 different readings and which one
[00:22:21] the 13 different readings and which one is correct right we just sort of do this
[00:22:24] is correct right we just sort of do this as we go along and it's sort of obvious
[00:22:27] as we go along and it's sort of obvious um okay let's just do a couple more
[00:22:29] um okay let's just do a couple more examples of where we get ambiguities in
[00:22:33] examples of where we get ambiguities in um in human language so a different one
[00:22:36] um in human language so a different one you get is coordination scope ambiguity
[00:22:40] you get is coordination scope ambiguity so shuttle veteran and longtime Nas
[00:22:42] so shuttle veteran and longtime Nas executive Fred Gregory appointed the
[00:22:44] executive Fred Gregory appointed the board how is this sentence
[00:22:47] board how is this sentence ambiguous it mean two people or one
[00:22:50] ambiguous it mean two people or one person yeah so there can either be one
[00:22:54] person yeah so there can either be one person Fred Gregory and they're both a
[00:22:57] person Fred Gregory and they're both a shuttle veteran and a NASA
[00:23:00] shuttle veteran and a NASA executive or it can be that there are
[00:23:03] executive or it can be that there are two people there's a shuttle veteran and
[00:23:07] two people there's a shuttle veteran and um there's a longtime Nessa executive
[00:23:10] um there's a longtime Nessa executive Fred
[00:23:11] Fred Gregory okay yeah so and we'd be kind of
[00:23:14] Gregory okay yeah so and we'd be kind of capturing those by having extra grammar
[00:23:17] capturing those by having extra grammar rules where a noun phrase can go to a
[00:23:19] rules where a noun phrase can go to a noun phrase a conjunction and a noun
[00:23:22] noun phrase a conjunction and a noun phrase um but then another another thing
[00:23:26] phrase um but then another another thing that you get in English is um apposition
[00:23:29] that you get in English is um apposition so you can have a noun phrase that's a
[00:23:32] so you can have a noun phrase that's a descriptive noun phrase of another noun
[00:23:34] descriptive noun phrase of another noun phrase like a name um you know the
[00:23:37] phrase like a name um you know the author Fred Gregory or something like
[00:23:39] author Fred Gregory or something like that um um saying the word English again
[00:23:44] that um um saying the word English again I I I meant to comment um so you know
[00:23:48] I I I meant to comment um so you know I'm I'm only going to give English
[00:23:49] I'm I'm only going to give English examples here um in different languages
[00:23:53] examples here um in different languages you don't get all the same ambiguities
[00:23:56] you don't get all the same ambiguities um so if you're familiar with same
[00:24:00] um so if you're familiar with same Chinese um you might have thought about
[00:24:02] Chinese um you might have thought about the prepositional phrase example of wait
[00:24:05] the prepositional phrase example of wait a minute we don't have that one because
[00:24:08] a minute we don't have that one because the prepositional phrase modifying the
[00:24:10] the prepositional phrase modifying the verb would appear before the verb and
[00:24:12] verb would appear before the verb and the object noun would be afterward so it
[00:24:14] the object noun would be afterward so it would be completely unambiguous and
[00:24:16] would be completely unambiguous and that's true um but you know that doesn't
[00:24:19] that's true um but you know that doesn't mean that Chinese is unambiguous Chinese
[00:24:22] mean that Chinese is unambiguous Chinese has lots of very bad
[00:24:25] has lots of very bad ambiguities and um yeah it's just that
[00:24:29] ambiguities and um yeah it's just that you know different languages have
[00:24:30] you know different languages have different syntactic structures okay um
[00:24:33] different syntactic structures okay um here's so sometimes um in English
[00:24:38] here's so sometimes um in English especially when you're sort of in a more
[00:24:39] especially when you're sort of in a more written form rather than having an
[00:24:41] written form rather than having an explicit con coordination word you can
[00:24:45] explicit con coordination word you can just sort of use ju to position with a
[00:24:47] just sort of use ju to position with a comma um to have the idea of
[00:24:51] comma um to have the idea of coordination so here's a um fun example
[00:24:55] coordination so here's a um fun example um from the first Trump Administration
[00:24:58] um from the first Trump Administration of how we can have a coordination scope
[00:25:01] of how we can have a coordination scope ambiguity um doctor no heart cognitive
[00:25:05] ambiguity um doctor no heart cognitive issues um right so again this is the
[00:25:08] issues um right so again this is the same kind of coordination scope
[00:25:11] same kind of coordination scope ambiguity that it can either be kind of
[00:25:13] ambiguity that it can either be kind of no hard and cognitive issues being
[00:25:16] no hard and cognitive issues being conjoined together like that or else it
[00:25:19] conjoined together like that or else it could be that it's no heart or cognitive
[00:25:23] could be that it's no heart or cognitive issues being conjoined um together like
[00:25:26] issues being conjoined um together like that you make the choice
[00:25:29] that you make the choice um okay uh let's
[00:25:33] um okay uh let's see oh this this is this is my risque
[00:25:36] see oh this this is this is my risque one for a different kind of ambiguity um
[00:25:39] one for a different kind of ambiguity um trigger warning um students get
[00:25:42] trigger warning um students get firsthand job
[00:25:45] firsthand job experience so this one is also um an
[00:25:49] experience so this one is also um an ambiguity um you know as to whether
[00:25:51] ambiguity um you know as to whether you're having the um the
[00:25:55] you're having the um the firsthand and then both the job and the
[00:25:59] firsthand and then both the job and the firsthand a modifying experience or
[00:26:03] firsthand a modifying experience or there's this other reading if you have a
[00:26:04] there's this other reading if you have a smutty mind that might come to you
[00:26:08] smutty mind that might come to you um okay one more fun one okay mutilated
[00:26:12] um okay one more fun one okay mutilated body washes up on Rio Beach to be used
[00:26:15] body washes up on Rio Beach to be used for Olympics beach
[00:26:17] for Olympics beach volleyball okay so what are what are the
[00:26:20] volleyball okay so what are what are the two possible readings of this sentence
[00:26:23] two possible readings of this sentence you know these are real examples from
[00:26:24] you know these are real examples from quality newspapers
[00:26:27] quality newspapers um okay what are the two readings of
[00:26:29] um okay what are the two readings of this sentence
[00:26:35] yeah so we've got so so the here we have
[00:26:39] yeah so we've got so so the here we have one of these
[00:26:41] one of these infinitival um so infinitival verb
[00:26:45] infinitival um so infinitival verb phrase to be used for Olympic beach
[00:26:48] phrase to be used for Olympic beach volleyball and for
[00:26:50] volleyball and for these as well you know they kind of have
[00:26:53] these as well you know they kind of have the same effect as prepositional phrases
[00:26:57] the same effect as prepositional phrases um that they can can modify um different
[00:27:00] um that they can can modify um different things um so it can either be the Rio
[00:27:03] things um so it can either be the Rio Beach that's going to be used for the
[00:27:06] Beach that's going to be used for the Olympic beach volleyball or it's going
[00:27:08] Olympic beach volleyball or it's going to be the mutilated body that gets used
[00:27:10] to be the mutilated body that gets used for the um beach
[00:27:13] for the um beach volleyball okay um yeah so the so these
[00:27:16] volleyball okay um yeah so the so these are the kind of ways in which we sort of
[00:27:18] are the kind of ways in which we sort of want to use um the structure of the
[00:27:21] want to use um the structure of the sentence to understand what they're
[00:27:23] sentence to understand what they're meaning we also use it in lots of sort
[00:27:26] meaning we also use it in lots of sort of just sort of M
[00:27:28] of just sort of M practical ways um when we're building
[00:27:31] practical ways um when we're building various kinds of natural language
[00:27:33] various kinds of natural language processing systems so you know a kind of
[00:27:36] processing systems so you know a kind of thing that people often in Practical
[00:27:39] thing that people often in Practical systems do is that they want to get out
[00:27:41] systems do is that they want to get out facts of various kinds so for people who
[00:27:44] facts of various kinds so for people who um do stuff with bioinformatics that
[00:27:47] um do stuff with bioinformatics that they commonly want to get out things
[00:27:49] they commonly want to get out things like protein protein interaction facts
[00:27:51] like protein protein interaction facts and so commonly you can get those kind
[00:27:54] and so commonly you can get those kind of facts out by looking for patterns so
[00:27:58] of facts out by looking for patterns so you know have a verb of interacts that's
[00:28:00] you know have a verb of interacts that's going to be indicating um an interaction
[00:28:03] going to be indicating um an interaction pattern and well it's going to be taking
[00:28:05] pattern and well it's going to be taking arguments so it's going to be taking a
[00:28:08] arguments so it's going to be taking a subject and interacts with the
[00:28:10] subject and interacts with the prepositional argument and so that will
[00:28:13] prepositional argument and so that will be um an interaction that kisy whatever
[00:28:16] be um an interaction that kisy whatever that is interacts with sass a but in
[00:28:19] that is interacts with sass a but in this case the Sass a is coordinated with
[00:28:21] this case the Sass a is coordinated with the kai a and the kai B so it's also
[00:28:24] the kai a and the kai B so it's also going to end up interacting with those
[00:28:27] going to end up interacting with those two other things as well and so you can
[00:28:29] two other things as well and so you can use the sort of sentence structure
[00:28:31] use the sort of sentence structure patterns of a dependency PA to be
[00:28:33] patterns of a dependency PA to be getting out the kind of um facts and
[00:28:36] getting out the kind of um facts and events that you're interested in for
[00:28:38] events that you're interested in for something like an um event understanding
[00:28:40] something like an um event understanding system and people you know do these kind
[00:28:43] system and people you know do these kind of anal analyses over biomedical texts
[00:28:47] of anal analyses over biomedical texts to build up the kind of structured
[00:28:49] to build up the kind of structured databases of known protein protein
[00:28:51] databases of known protein protein interactions and things of that
[00:28:54] interactions and things of that sort okay so linguistic structure is
[00:28:58] sort okay so linguistic structure is useful um and it's syntactically very
[00:29:03] useful um and it's syntactically very ambiguous and so you should think of
[00:29:06] ambiguous and so you should think of humans as active interpreters that are
[00:29:09] humans as active interpreters that are using their contextual knowledge both of
[00:29:12] using their contextual knowledge both of earlier stuff in the text knowledge of
[00:29:14] earlier stuff in the text knowledge of the world around them how the world
[00:29:16] the world around them how the world Works to work out the right um structure
[00:29:19] Works to work out the right um structure yeah so now I want to go on um and show
[00:29:22] yeah so now I want to go on um and show you a bit more about sort of dependency
[00:29:25] you a bit more about sort of dependency grammars which is what we're going to be
[00:29:27] grammars which is what we're going to be using so for dependency syntax that it
[00:29:31] using so for dependency syntax that it postulates that you can capture the
[00:29:34] postulates that you can capture the structure of a sentence by having these
[00:29:38] structure of a sentence by having these sort of asymmetric um dependent
[00:29:41] sort of asymmetric um dependent relations which we might just call
[00:29:43] relations which we might just call arrows which are going from heads to
[00:29:46] arrows which are going from heads to dependence so here the sentence is um
[00:29:49] dependence so here the sentence is um bills on ports and immigration were
[00:29:51] bills on ports and immigration were submitted by Senator Brownback
[00:29:54] submitted by Senator Brownback Republican of Kansas and we sort of
[00:29:56] Republican of Kansas and we sort of picking out heads um and then we got um
[00:30:00] picking out heads um and then we got um things that depend on them that modify
[00:30:03] things that depend on them that modify them um yeah so if you're on the um
[00:30:08] them um yeah so if you're on the um video audience and you are educated in
[00:30:11] video audience and you are educated in the United States and you're over the
[00:30:13] the United States and you're over the age of 50 um or if you happen to go to
[00:30:17] age of 50 um or if you happen to go to one of those kind of private schools
[00:30:19] one of those kind of private schools where they also teach Latin um you might
[00:30:23] where they also teach Latin um you might have seen sentence diagramming um so
[00:30:26] have seen sentence diagramming um so read Kellogg um sentence diagramming was
[00:30:30] read Kellogg um sentence diagramming was something that was actually very
[00:30:31] something that was actually very widespread um in American Education um
[00:30:35] widespread um in American Education um which really was a it was a some it was
[00:30:38] which really was a it was a some it was really dependency grammar was a sort of
[00:30:40] really dependency grammar was a sort of a somewhat quirky form of dependency
[00:30:42] a somewhat quirky form of dependency grammar where you had to write lines at
[00:30:44] grammar where you had to write lines at different angles and stuff like that but
[00:30:47] different angles and stuff like that but basically you're writing sort of heads
[00:30:49] basically you're writing sort of heads and their dependence underneath them
[00:30:51] and their dependence underneath them with different funny shaped lines um it
[00:30:54] with different funny shaped lines um it also was dependency
[00:30:56] also was dependency grammar okay um so this is the start of
[00:30:59] grammar okay um so this is the start of a dependency grammar but just like the
[00:31:02] a dependency grammar but just like the the funny angled lines of sentence
[00:31:04] the funny angled lines of sentence diagramming normally people want to add
[00:31:07] diagramming normally people want to add some more information than that um and
[00:31:10] some more information than that um and so most commonly um that the arrows are
[00:31:14] so most commonly um that the arrows are then typed by giving the name of some
[00:31:17] then typed by giving the name of some grammatical relation so something can be
[00:31:19] grammatical relation so something can be the noun subject or an oblique or an
[00:31:25] the noun subject or an oblique or an appositional modifier or a case mark or
[00:31:28] appositional modifier or a case mark or things like that um and um I I'm just
[00:31:33] things like that um and um I I'm just trying to give you the idea of
[00:31:35] trying to give you the idea of dependency grammars I'm not expecting
[00:31:37] dependency grammars I'm not expecting you to master all of these names and
[00:31:40] you to master all of these names and ways of doing things um and you know
[00:31:44] ways of doing things um and you know there are different systems of deciding
[00:31:46] there are different systems of deciding what's heads and dependence and not all
[00:31:48] what's heads and dependence and not all the details are important what you
[00:31:51] the details are important what you should get into your head is just sort
[00:31:53] should get into your head is just sort of the basic idea of what one of these
[00:31:55] of the basic idea of what one of these does and some sense of oh it should be
[00:31:58] does and some sense of oh it should be at the phrase level it should be
[00:32:00] at the phrase level it should be representing what's modifying what so we
[00:32:03] representing what's modifying what so we do actually ask some questions um on the
[00:32:07] do actually ask some questions um on the assignment and so for the cases like the
[00:32:10] assignment and so for the cases like the prepositional phrase what is it
[00:32:12] prepositional phrase what is it modifying you should be able to give the
[00:32:14] modifying you should be able to give the right answer to
[00:32:16] right answer to that okay
[00:32:19] that okay um yeah um okay so uh this is just a
[00:32:24] um yeah um okay so uh this is just a little bit more um vocabulary so yeah we
[00:32:27] little bit more um vocabulary so yeah we have these or dependencies and so I'm
[00:32:30] have these or dependencies and so I'm going to say that they connect between a
[00:32:32] going to say that they connect between a head and a dependent but sometimes
[00:32:34] head and a dependent but sometimes people use other words like governor and
[00:32:36] people use other words like governor and modifier and things like that um and so
[00:32:40] modifier and things like that um and so dependencies are generally taken and
[00:32:43] dependencies are generally taken and will be taking them as forming a tree so
[00:32:46] will be taking them as forming a tree so you've got something that's connected as
[00:32:49] you've got something that's connected as cyclic and has a single root to it so
[00:32:52] cyclic and has a single root to it so our single root is the top of the
[00:32:54] our single root is the top of the sentence here
[00:32:56] sentence here um so
[00:32:58] um so um dependency so although what you see
[00:33:02] um dependency so although what you see most often these days either in a
[00:33:04] most often these days either in a Linguistics class or when you get taught
[00:33:07] Linguistics class or when you get taught CS
[00:33:08] CS 103 at Stanford or computer science what
[00:33:12] 103 at Stanford or computer science what you see there is normally context free
[00:33:15] you see there is normally context free grammars or phase structure grammars I
[00:33:17] grammars or phase structure grammars I mean really you know it is dependency
[00:33:20] mean really you know it is dependency grammars that have the really long
[00:33:23] grammars that have the really long history so really the predominant way of
[00:33:26] history so really the predominant way of representing the structure of human
[00:33:28] representing the structure of human languages throughout human history is
[00:33:31] languages throughout human history is dependency grammar um so who linguist um
[00:33:34] dependency grammar um so who linguist um Herald as the first dependency
[00:33:37] Herald as the first dependency grammarian or really the first person
[00:33:39] grammarian or really the first person who tried to write the grammar of a
[00:33:42] who tried to write the grammar of a human language period was panani so
[00:33:45] human language period was panani so panani was working with Sanskrit um
[00:33:48] panani was working with Sanskrit um panani lived so long ago that actually
[00:33:50] panani lived so long ago that actually people don't really know when he lived I
[00:33:53] people don't really know when he lived I mean he lived somewhere between about
[00:33:55] mean he lived somewhere between about the 4th and 8th Century before the
[00:33:57] the 4th and 8th Century before the common ER
[00:33:58] common ER but really no one knows when um but you
[00:34:01] but really no one knows when um but you know he lived um sort of up in part of
[00:34:04] know he lived um sort of up in part of actually what's now Afghanistan um and
[00:34:07] actually what's now Afghanistan um and um for Motivated for largely religious
[00:34:11] um for Motivated for largely religious reasons um he said about developing a
[00:34:14] reasons um he said about developing a grammar of Sanskrit and the way he
[00:34:16] grammar of Sanskrit and the way he represented the syntax of Sanskrit was
[00:34:19] represented the syntax of Sanskrit was using a dependency grammar um so there
[00:34:21] using a dependency grammar um so there was a lot of work on grammar in Arabic
[00:34:24] was a lot of work on grammar in Arabic in the first Millennium they used
[00:34:26] in the first Millennium they used dependency grammars um in contrast um
[00:34:30] dependency grammars um in contrast um the idea of sort of having context free
[00:34:33] the idea of sort of having context free grammars that's really really recent so
[00:34:35] grammars that's really really recent so the first work on um phrase structure
[00:34:38] the first work on um phrase structure grammars dates to the 40s and then was
[00:34:41] grammars dates to the 40s and then was sort of um canonicalized by the work of
[00:34:44] sort of um canonicalized by the work of Chomsky in the
[00:34:46] Chomsky in the 1950s yeah so um a fact for the computer
[00:34:51] 1950s yeah so um a fact for the computer science part of people in the audience
[00:34:53] science part of people in the audience so computer dear computer scientists if
[00:34:56] so computer dear computer scientists if you know about Chomsky computer
[00:34:58] you know about Chomsky computer scientists normally know two things
[00:34:59] scientists normally know two things about Chomsky one is they hate on the
[00:35:02] about Chomsky one is they hate on the Chomsky hierarchy that they were forced
[00:35:04] Chomsky hierarchy that they were forced to learn um in CS 103 or equivalent
[00:35:07] to learn um in CS 103 or equivalent classes and the second one is he's a
[00:35:10] classes and the second one is he's a very left politician um but um if I only
[00:35:13] very left politician um but um if I only deal with the first one of the two now
[00:35:16] deal with the first one of the two now um the Chomsky hierarchy was not
[00:35:19] um the Chomsky hierarchy was not invented either to torture Elementary
[00:35:22] invented either to torture Elementary computer scientists um or um to explain
[00:35:26] computer scientists um or um to explain fundamental facts about formal language
[00:35:28] fundamental facts about formal language Theory the Chomsky hierarchy was
[00:35:30] Theory the Chomsky hierarchy was actually invented in thinking about
[00:35:33] actually invented in thinking about human languages because at that time and
[00:35:37] human languages because at that time and in stuff that's come more often it was
[00:35:40] in stuff that's come more often it was commonly the case that um people were
[00:35:44] commonly the case that um people were modeling human languages with um Regular
[00:35:48] modeling human languages with um Regular finite so finite State grammar
[00:35:51] finite so finite State grammar equivalent mechanisms and Chomsky wanted
[00:35:54] equivalent mechanisms and Chomsky wanted to argue that that was a completely
[00:35:56] to argue that that was a completely inadequate um formalism to represent um
[00:36:00] inadequate um formalism to represent um the complexity of human language and so
[00:36:03] the complexity of human language and so it was in the context of arguments about
[00:36:05] it was in the context of arguments about human language was why he developed um
[00:36:07] human language was why he developed um the Chomsky
[00:36:09] the Chomsky hierarchy okay um yeah so anyway uh
[00:36:12] hierarchy okay um yeah so anyway uh that's enough of the history of that um
[00:36:15] that's enough of the history of that um here's my uh picture of part of panan
[00:36:18] here's my uh picture of part of panan grammar but actually or a version of it
[00:36:21] grammar but actually or a version of it actually um this is really really
[00:36:24] actually um this is really really misleading and because one of the
[00:36:26] misleading and because one of the astounding facts about P's grammar and
[00:36:29] astounding facts about P's grammar and part of why no one knows what century he
[00:36:31] part of why no one knows what century he lived in was Pan's grammar was composed
[00:36:35] lived in was Pan's grammar was composed orally um so this sort of kind of blows
[00:36:38] orally um so this sort of kind of blows my mind you know it's it seems you know
[00:36:42] my mind you know it's it seems you know um some of um the famous things in the
[00:36:45] um some of um the famous things in the west like Homer's works right the
[00:36:48] west like Homer's works right the Odyssey and The Iliad right they were
[00:36:50] Odyssey and The Iliad right they were originally oral works that were passed
[00:36:53] originally oral works that were passed down um in oral form you know you can
[00:36:57] down um in oral form you know you can that seems hard to do but you can kind
[00:36:59] that seems hard to do but you can kind of believe if you did plays in high
[00:37:01] of believe if you did plays in high school or something that someone could
[00:37:04] school or something that someone could um memorize the Odyssey perhaps um but
[00:37:07] um memorize the Odyssey perhaps um but the idea that people could memorize a
[00:37:10] the idea that people could memorize a grammar of a
[00:37:12] grammar of a language passing it down for hundreds of
[00:37:15] language passing it down for hundreds of years um kind of blows my mind um but
[00:37:18] years um kind of blows my mind um but that's exactly what happened um yeah
[00:37:21] that's exactly what happened um yeah with pines grammar um so you know really
[00:37:25] with pines grammar um so you know really although this is sort of an old
[00:37:26] although this is sort of an old birchbark manuscript you know that
[00:37:28] birchbark manuscript you know that really it probably dates from about a
[00:37:31] really it probably dates from about a millennium after panan um composed um
[00:37:34] millennium after panan um composed um his grammar okay getting back to the um
[00:37:37] his grammar okay getting back to the um modern days um yeah so um for things to
[00:37:41] modern days um yeah so um for things to know yeah so I mean we don't want you to
[00:37:45] know yeah so I mean we don't want you to fixate on the sort of details of
[00:37:47] fixate on the sort of details of dependency grammar structure providing
[00:37:49] dependency grammar structure providing you have the rough idea but just one
[00:37:51] you have the rough idea but just one thing um that you can possibly be
[00:37:53] thing um that you can possibly be confused about is you know there people
[00:37:57] confused about is you know there people do things in different ways one way in
[00:38:00] do things in different ways one way in which they don't agree is even which way
[00:38:02] which they don't agree is even which way to draw the arrows so some people draw
[00:38:06] to draw the arrows so some people draw arrows um from the head pointing at the
[00:38:09] arrows um from the head pointing at the dependence and there are other people
[00:38:11] dependence and there are other people who draw the arrows starting at the
[00:38:12] who draw the arrows starting at the dependent and pointing back at the heads
[00:38:15] dependent and pointing back at the heads um so um for modern dependency grammar
[00:38:19] um so um for modern dependency grammar um largely follows the work of um Lucien
[00:38:22] um largely follows the work of um Lucien tener a French um linguist um he did the
[00:38:28] tener a French um linguist um he did the um arrows pointing from the head to the
[00:38:30] um arrows pointing from the head to the dependent and so that's what I'm doing
[00:38:32] dependent and so that's what I'm doing today but um you'll see both um we sort
[00:38:36] today but um you'll see both um we sort of said that you know normally you
[00:38:38] of said that you know normally you assume that you have a tree with a
[00:38:40] assume that you have a tree with a single root it's kind of common and it
[00:38:43] single root it's kind of common and it works out more easily for the paing if
[00:38:46] works out more easily for the paing if you sort of add to a sentence a sort of
[00:38:48] you sort of add to a sentence a sort of a fake root node so you know that that's
[00:38:51] a fake root node so you know that that's going to be the starting point and it's
[00:38:53] going to be the starting point and it's going to take one dependent which is the
[00:38:56] going to take one dependent which is the word that's the head of the sentence
[00:38:58] word that's the head of the sentence and then you're going to work down from
[00:39:00] and then you're going to work down from there okay um so um before getting more
[00:39:05] there okay um so um before getting more into doing dependency paing I just
[00:39:07] into doing dependency paing I just wanted to sort of take a little detour
[00:39:11] wanted to sort of take a little detour um to tell you about the you know the
[00:39:14] um to tell you about the you know the importance um that happened with sort of
[00:39:17] importance um that happened with sort of the rise of annotated data um in natural
[00:39:21] the rise of annotated data um in natural language processing so and you know this
[00:39:25] language processing so and you know this is sort of an interesting flip slop
[00:39:27] is sort of an interesting flip slop that's occurred but we're going to sort
[00:39:29] that's occurred but we're going to sort of today go in One Direction and later
[00:39:31] of today go in One Direction and later class we'll go in the other direction um
[00:39:34] class we'll go in the other direction um so in early natural language processing
[00:39:37] so in early natural language processing um people started to see oh human
[00:39:41] um people started to see oh human languages have structure so what we
[00:39:45] languages have structure so what we should do is start writing rules for the
[00:39:48] should do is start writing rules for the structure of human languages and you
[00:39:50] structure of human languages and you know I start writing a few context free
[00:39:53] know I start writing a few context free grammar rules for the structure of
[00:39:54] grammar rules for the structure of English on that early slide and you
[00:39:56] English on that early slide and you could also write dependency grammar
[00:39:58] could also write dependency grammar structure rules so um people tried to do
[00:40:01] structure rules so um people tried to do natural language processing by having
[00:40:04] natural language processing by having rules grammar rules dictionaries of
[00:40:07] rules grammar rules dictionaries of parts of speech and things like that and
[00:40:09] parts of speech and things like that and that gave you paes um that in retrospect
[00:40:15] that gave you paes um that in retrospect worked out pretty badly and it worked
[00:40:19] worked out pretty badly and it worked out pretty badly for a number of reasons
[00:40:21] out pretty badly for a number of reasons one reason is that although there are
[00:40:25] one reason is that although there are these sort of very canonical clear
[00:40:27] these sort of very canonical clear structures in human languages there's a
[00:40:29] structures in human languages there's a very long tale of messy stuff where all
[00:40:32] very long tale of messy stuff where all kinds of weird um usages start to emerge
[00:40:37] kinds of weird um usages start to emerge in human languages which sort of means
[00:40:40] in human languages which sort of means you just got this it's just really hard
[00:40:42] you just got this it's just really hard to get coverage for handwritten language
[00:40:45] to get coverage for handwritten language um and that's because people you humans
[00:40:48] um and that's because people you humans use language creatively right so you
[00:40:51] use language creatively right so you know you can start thinking of some of
[00:40:53] know you can start thinking of some of the things um that you've probably come
[00:40:57] the things um that you've probably come I'm probably not very good at Young
[00:40:59] I'm probably not very good at Young persons slang usages of grammar these
[00:41:02] persons slang usages of grammar these days but you know um the kind of ones
[00:41:05] days but you know um the kind of ones that you might be still familiar with
[00:41:07] that you might be still familiar with right Star Wars you have Yoda talk where
[00:41:09] right Star Wars you have Yoda talk where you rearrange the sentences but people
[00:41:11] you rearrange the sentences but people still understand them right so that's
[00:41:13] still understand them right so that's changing the word order and earlier on
[00:41:16] changing the word order and earlier on than that there was sort of a uh little
[00:41:19] than that there was sort of a uh little a bit of a f to putting not at the end
[00:41:21] a bit of a f to putting not at the end of the sentences of that's a really
[00:41:24] of the sentences of that's a really great idea not um and well you know
[00:41:28] great idea not um and well you know people um learn to understand that but
[00:41:31] people um learn to understand that but it's different to regular grammar right
[00:41:33] it's different to regular grammar right so it it's really hard to write a full
[00:41:36] so it it's really hard to write a full grammar but the bigger reason actually
[00:41:38] grammar but the bigger reason actually is the problem of ambiguity I talked
[00:41:40] is the problem of ambiguity I talked about right that if you just write a
[00:41:43] about right that if you just write a grammar well you know my sentence with
[00:41:46] grammar well you know my sentence with the prepositional phrases had 13
[00:41:49] the prepositional phrases had 13 different paes and you didn't have much
[00:41:51] different paes and you didn't have much reason to choose between them but you
[00:41:53] reason to choose between them but you know if you had information about how
[00:41:56] know if you had information about how often words modify other words then you
[00:41:59] often words modify other words then you could get some statistics and start to
[00:42:02] could get some statistics and start to predict in which order which things
[00:42:05] predict in which order which things modify other things and so people wanted
[00:42:08] modify other things and so people wanted to start to be able to do that
[00:42:09] to start to be able to do that prediction that underlies probalistic or
[00:42:12] prediction that underlies probalistic or machine learning models and so to be
[00:42:15] machine learning models and so to be able to do that that led you know sort
[00:42:18] able to do that that led you know sort of earliest antecedence in the 6s but
[00:42:21] of earliest antecedence in the 6s but really starting in the late 80s and into
[00:42:24] really starting in the late 80s and into the '90s that people decide you the way
[00:42:27] the '90s that people decide you the way make progress in um natural language
[00:42:30] make progress in um natural language processing natural language
[00:42:32] processing natural language understanding is to build annotated data
[00:42:35] understanding is to build annotated data resources and so all through the '90s
[00:42:38] resources and so all through the '90s and the 2000s decades the name of the
[00:42:42] and the 2000s decades the name of the game for a lot of natural language
[00:42:43] game for a lot of natural language processing was people building annotated
[00:42:46] processing was people building annotated data resources and then building machine
[00:42:49] data resources and then building machine learning systems on top using those
[00:42:51] learning systems on top using those resources now that's kind of gone into
[00:42:53] resources now that's kind of gone into reverse and gone away again with large
[00:42:55] reverse and gone away again with large language models which will to another
[00:42:57] language models which will to another week or so but here's an example so this
[00:43:01] week or so but here's an example so this is the universal dependencies tree Banks
[00:43:04] is the universal dependencies tree Banks which I'm actually been heavily involved
[00:43:06] which I'm actually been heavily involved with and it's a cool resource for all
[00:43:08] with and it's a cool resource for all kinds of purposes because it's actually
[00:43:11] kinds of purposes because it's actually a wide
[00:43:12] a wide crosslinguistic um database where
[00:43:14] crosslinguistic um database where there's over a 100 different languages
[00:43:17] there's over a 100 different languages um with sentences passed with a uniform
[00:43:20] um with sentences passed with a uniform um dependency formalism so it's actually
[00:43:23] um dependency formalism so it's actually really good for things like cross
[00:43:24] really good for things like cross linguistic work and psychol linguistic
[00:43:26] linguistic work and psychol linguistic work work but you know what these are is
[00:43:30] work work but you know what these are is taking sentences I think mamar was a
[00:43:33] taking sentences I think mamar was a famous goat trainer or
[00:43:34] famous goat trainer or something um and putting a dependency
[00:43:38] something um and putting a dependency structure on it it's sort of all written
[00:43:40] structure on it it's sort of all written there sort of very squished down and
[00:43:42] there sort of very squished down and human beings are producing these
[00:43:44] human beings are producing these dependency structures and then this is
[00:43:46] dependency structures and then this is giving us data that we can learn things
[00:43:49] giving us data that we can learn things from dependency like dependency paes
[00:43:51] from dependency like dependency paes from and indeed um for what you do on
[00:43:54] from and indeed um for what you do on homework 2 this is precisely what you'll
[00:43:56] homework 2 this is precisely what you'll be using is data of this sort um to
[00:44:00] be using is data of this sort um to build a dependency paa and it's going to
[00:44:03] build a dependency paa and it's going to learn that you know that you have goat
[00:44:05] learn that you know that you have goat trainers um and you have famous trainers
[00:44:09] trainers um and you have famous trainers and so it'll build up sort of statistics
[00:44:11] and so it'll build up sort of statistics and information to predict what kinds of
[00:44:13] and information to predict what kinds of things are
[00:44:15] things are likely um yeah so starting off building
[00:44:20] likely um yeah so starting off building a tree Bank like that
[00:44:22] a tree Bank like that um feels kind of like oh this is going
[00:44:25] um feels kind of like oh this is going to be slow hard work work and it is
[00:44:28] to be slow hard work work and it is actually slow hard work um but it proved
[00:44:31] actually slow hard work um but it proved to be a very effective strategy because
[00:44:34] to be a very effective strategy because it gave wonderful reusable resources
[00:44:38] it gave wonderful reusable resources that once people had done it once all
[00:44:41] that once people had done it once all sorts of people could use it to build
[00:44:43] sorts of people could use it to build paes part of speech taggers um to do
[00:44:47] paes part of speech taggers um to do psycholinguistic models and all kinds of
[00:44:49] psycholinguistic models and all kinds of things you'd get the sort of
[00:44:51] things you'd get the sort of distributional frequency information
[00:44:53] distributional frequency information that's good for machine learning it also
[00:44:55] that's good for machine learning it also provided one other thing that's crucial
[00:44:58] provided one other thing that's crucial is it gave a method to evaluate systems
[00:45:02] is it gave a method to evaluate systems to say how good they are at um producing
[00:45:06] to say how good they are at um producing paes I mean this may seem kind of
[00:45:10] paes I mean this may seem kind of comical to you in the modern era of
[00:45:12] comical to you in the modern era of machine learning but the fact of the
[00:45:14] machine learning but the fact of the matter is when people did natural
[00:45:16] matter is when people did natural language processing in the 50s 60s 7s
[00:45:21] language processing in the 50s 60s 7s nobody had evaluation methods the way
[00:45:25] nobody had evaluation methods the way you showed people you had good paa is
[00:45:28] you showed people you had good paa is you ran the program you said type in a
[00:45:31] you ran the program you said type in a sentence look at what it look look it's
[00:45:33] sentence look at what it look look it's worked it's a really good paa um you
[00:45:36] worked it's a really good paa um you know there was no systematic valuation
[00:45:39] know there was no systematic valuation of NLP systems whatsoever um so actually
[00:45:43] of NLP systems whatsoever um so actually saying look here's uh thousand hand
[00:45:46] saying look here's uh thousand hand passas sentences let's evaluate how well
[00:45:49] passas sentences let's evaluate how well your paa does on those you know that was
[00:45:52] your paa does on those you know that was actually a revolutionary new development
[00:45:55] actually a revolutionary new development um that happened in the um
[00:45:57] um that happened in the um well end of the 80s but especially in
[00:45:59] well end of the 80s but especially in the
[00:46:00] the 90s okay um so now we going have it now
[00:46:06] 90s okay um so now we going have it now that we have all of those knowledge um
[00:46:08] that we have all of those knowledge um we're going to um want to start building
[00:46:12] we're going to um want to start building dependency paes and so I'm going to um
[00:46:16] dependency paes and so I'm going to um show a particular way of dependency
[00:46:18] show a particular way of dependency passing which is the one you're going to
[00:46:20] passing which is the one you're going to use in the assignment um but you know
[00:46:22] use in the assignment um but you know just first off it's sort of worth just
[00:46:24] just first off it's sort of worth just thinking for a moment you know what kind
[00:46:27] thinking for a moment you know what kind of information should a dependency paer
[00:46:30] of information should a dependency paer have to make decisions so these are kind
[00:46:32] have to make decisions so these are kind of the four factors the sort of the
[00:46:35] of the four factors the sort of the obvious things that are useful for
[00:46:37] obvious things that are useful for dependency passing I mean the first one
[00:46:40] dependency passing I mean the first one is sort of thinking of the two words at
[00:46:43] is sort of thinking of the two words at the ends of the arrow as to whether they
[00:46:45] the ends of the arrow as to whether they are plausible right so that for um
[00:46:49] are plausible right so that for um discussion of the outstanding issues was
[00:46:52] discussion of the outstanding issues was completed um so to have um discussion of
[00:46:57] completed um so to have um discussion of issues right that's that's a plausible
[00:47:01] issues right that's that's a plausible dependency um to have um you know what's
[00:47:05] dependency um to have um you know what's the silly one to have something like um
[00:47:08] the silly one to have something like um the being a dependent of completed that
[00:47:11] the being a dependent of completed that makes no sense at all so you know what
[00:47:14] makes no sense at all so you know what words there are involved um the second
[00:47:18] words there are involved um the second one is dependency distance so you can
[00:47:21] one is dependency distance so you can have longdistance dependencies that go a
[00:47:24] have longdistance dependencies that go a long way but most dependenc sees are
[00:47:28] long way but most dependenc sees are short distance you know a lot of words
[00:47:30] short distance you know a lot of words are depending on their neighboring words
[00:47:32] are depending on their neighboring words at a very short distance so that's a
[00:47:35] at a very short distance so that's a good preference to have um as well as
[00:47:38] good preference to have um as well as just the distance it's somewhat
[00:47:40] just the distance it's somewhat informative knowing what's in between so
[00:47:44] informative knowing what's in between so it's rare for dependencies to span verbs
[00:47:47] it's rare for dependencies to span verbs or
[00:47:48] or punctuation um and then there's a final
[00:47:52] punctuation um and then there's a final one which is to think of the veency of
[00:47:55] one which is to think of the veency of heads and that's how many
[00:47:57] heads and that's how many arguments um they take so that if you
[00:48:01] arguments um they take so that if you have sort of something like a verb broke
[00:48:04] have sort of something like a verb broke um well it probably has something um to
[00:48:09] um well it probably has something um to the left because it probably has who did
[00:48:12] the left because it probably has who did the breaking and it probably has
[00:48:15] the breaking and it probably has something to the right um cuz there
[00:48:18] something to the right um cuz there might be the cup or something like that
[00:48:21] might be the cup or something like that um but you know it doesn't have to be
[00:48:23] um but you know it doesn't have to be that cuz it could be the cup broke um so
[00:48:27] that cuz it could be the cup broke um so you can have something to the left but
[00:48:30] you can have something to the left but nothing to the right but you sort of
[00:48:32] nothing to the right but you sort of have to have something to the left and
[00:48:34] have to have something to the left and conversly you can't have any number of
[00:48:37] conversly you can't have any number of things you can't sort of just say he
[00:48:39] things you can't sort of just say he broke the cup the sorcer um the dish um
[00:48:43] broke the cup the sorcer um the dish um right so it doesn't take just lots of
[00:48:45] right so it doesn't take just lots of arguments to the left so we've got a
[00:48:46] arguments to the left so we've got a notion of veency like
[00:48:49] notion of veency like that um yeah there's one other tricky
[00:48:53] that um yeah there's one other tricky little notion on dependency paring which
[00:48:56] little notion on dependency paring which is
[00:48:57] is normally um normally dependencies kind
[00:49:01] normally um normally dependencies kind of Nest like this um and nesting
[00:49:04] of Nest like this um and nesting dependencies corresponds to a tree
[00:49:07] dependencies corresponds to a tree structure as you'd have in a context
[00:49:09] structure as you'd have in a context free grammar
[00:49:15] [Music]
[00:49:17] [Music] yeah because in a sense when I read the
[00:49:19] yeah because in a sense when I read the sentence
[00:49:21] sentence which I thought that the most important
[00:49:24] which I thought that the most important discussion
[00:49:27] discussion [Music]
[00:49:31] so so I will fair enough I will assert
[00:49:36] so so I will fair enough I will assert that this
[00:49:38] that this is uh
[00:49:40] is uh sentence and discussion is the subject
[00:49:45] sentence and discussion is the subject of the
[00:49:46] of the verb completed and you know normally for
[00:49:51] verb completed and you know normally for a sentence we say the main thing in the
[00:49:54] a sentence we say the main thing in the sentence is its verb and so yeah so
[00:49:57] sentence is its verb and so yeah so that's why the root is heading to
[00:49:59] that's why the root is heading to completed and the subject of the verb is
[00:50:01] completed and the subject of the verb is also an important thing but the
[00:50:04] also an important thing but the arguments of the verb like the subject
[00:50:06] arguments of the verb like the subject of the verb the object of the verb if
[00:50:08] of the verb the object of the verb if there is one prepositional phrase
[00:50:09] there is one prepositional phrase modifiers they're all taken as
[00:50:11] modifiers they're all taken as dependence of the
[00:50:14] dependence of the verb
[00:50:17] verb yeah up on that is
[00:50:21] yeah up on that is it is it not the that you start
[00:50:24] it is it not the that you start with so um if you have a sentence with a
[00:50:29] with so um if you have a sentence with a verb like this um it's always that is
[00:50:34] verb like this um it's always that is always the answer I mean some of the
[00:50:37] always the answer I mean some of the details here depend on languages but
[00:50:40] details here depend on languages but there are languages in which you don't
[00:50:41] there are languages in which you don't have to have a verb in a sentence um and
[00:50:44] have to have a verb in a sentence um and you can get things like
[00:50:47] you can get things like um um just I mean you can do it in sort
[00:50:51] um um just I mean you can do it in sort of very restricted ways in in in English
[00:50:54] of very restricted ways in in in English right so if you just sort of say easy as
[00:50:57] right so if you just sort of say easy as pie um there's no verb and so then
[00:51:00] pie um there's no verb and so then you're saying easy um the adjective
[00:51:03] you're saying easy um the adjective which is sort of the predicate adjective
[00:51:05] which is sort of the predicate adjective is then the head of the
[00:51:08] is then the head of the sentence
[00:51:10] sentence sorry like a question like what is the
[00:51:13] sorry like a question like what is the story is the is like we would still look
[00:51:16] story is the is like we would still look at that as the that is complicated some
[00:51:20] at that as the that is complicated some people would say it is um and some
[00:51:23] people would say it is um and some people would say it isn't um and
[00:51:26] people would say it isn't um and particular in Universal dependencies we
[00:51:29] particular in Universal dependencies we don't actually say that is is the head
[00:51:30] don't actually say that is is the head of the sentence but I don't want to get
[00:51:32] of the sentence but I don't want to get too far into this if you want you could
[00:51:34] too far into this if you want you could sort of look more at how things are done
[00:51:37] sort of look more at how things are done but you know I want to fully admit that
[00:51:39] but you know I want to fully admit that you know dependency grammar isn't sort
[00:51:42] you know dependency grammar isn't sort of one uniquely defined Theory people
[00:51:44] of one uniquely defined Theory people have had different ideas of which things
[00:51:46] have had different ideas of which things to take as the head in various
[00:51:49] to take as the head in various circumstances and they argue about it
[00:51:51] circumstances and they argue about it linguists argue about what the right
[00:51:53] linguists argue about what the right structure is to put over all sorts of
[00:51:55] structure is to put over all sorts of sentences but that the fact that people
[00:51:58] sentences but that the fact that people do things different ways doesn't mean
[00:52:00] do things different ways doesn't mean that everybody doesn't agree that there
[00:52:02] that everybody doesn't agree that there are units there are phrases and
[00:52:05] are units there are phrases and modifiers and ambiguities and so on
[00:52:08] modifiers and ambiguities and so on between them okay um yeah so normally we
[00:52:12] between them okay um yeah so normally we get this sort of nesting that
[00:52:13] get this sort of nesting that corresponds to what you can build with
[00:52:15] corresponds to what you can build with context free grammar structure but
[00:52:18] context free grammar structure but sometimes in human languages you get
[00:52:21] sometimes in human languages you get dependencies that don't Nest so you get
[00:52:25] dependencies that don't Nest so you get sentences like I'll give a talk tomorrow
[00:52:27] sentences like I'll give a talk tomorrow on new networks where actually the on
[00:52:31] on new networks where actually the on new networks is modifying the talk where
[00:52:35] new networks is modifying the talk where the
[00:52:36] the yesterday um is an argument of sorry the
[00:52:40] yesterday um is an argument of sorry the tomorrow is an argument of give and so
[00:52:42] tomorrow is an argument of give and so you get these Crossing dependencies
[00:52:45] you get these Crossing dependencies which are referred to as non-projective
[00:52:48] which are referred to as non-projective dependencies um you also get them when
[00:52:51] dependencies um you also get them when you form questions so who did Bill buy
[00:52:55] you form questions so who did Bill buy the coffee from yesterday that the who
[00:52:58] the coffee from yesterday that the who is the object of the preposition from
[00:53:02] is the object of the preposition from but it's been moved out the front and so
[00:53:04] but it's been moved out the front and so that again um gives us
[00:53:08] that again um gives us non-projective um if you think about
[00:53:12] non-projective um if you think about it um yeah you can still say um that you
[00:53:16] it um yeah you can still say um that you have a dependency tree but it's got the
[00:53:20] have a dependency tree but it's got the words in different orders and so one of
[00:53:22] words in different orders and so one of the things that you have to cope with
[00:53:24] the things that you have to cope with for full dependency PA
[00:53:27] for full dependency PA is dealing with this non projectivity
[00:53:29] is dealing with this non projectivity but I mean actually we're not going to
[00:53:30] but I mean actually we're not going to deal with it in our paes we're only
[00:53:32] deal with it in our paes we're only going to do um projective dependency
[00:53:35] going to do um projective dependency paing okay so there are various ways
[00:53:39] paing okay so there are various ways that people do dependency paring people
[00:53:42] that people do dependency paring people have done it by dynamic programming
[00:53:45] have done it by dynamic programming people who've done it using graph
[00:53:47] people who've done it using graph algorithms if I have enough time at the
[00:53:49] algorithms if I have enough time at the end I might mention that again people
[00:53:52] end I might mention that again people have done it with constraint
[00:53:53] have done it with constraint satisfaction methods if you saw those in
[00:53:55] satisfaction methods if you saw those in cs220 21 um but the most common way in
[00:54:00] cs220 21 um but the most common way in practice that's emerged um has been this
[00:54:03] practice that's emerged um has been this transition based paing um which is kind
[00:54:08] transition based paing um which is kind of sort of interesting as well and gives
[00:54:11] of sort of interesting as well and gives the sort of a very simple machine
[00:54:13] the sort of a very simple machine learning mechanism um so it makes it
[00:54:16] learning mechanism um so it makes it good for um assignment two and so that's
[00:54:19] good for um assignment two and so that's what we're going to explore
[00:54:22] what we're going to explore here okay so um what we do
[00:54:27] here okay so um what we do um in Greedy decision based paing in
[00:54:30] um in Greedy decision based paing in transition based paing is you know this
[00:54:33] transition based paing is you know this is where it's unfortunate that only two
[00:54:35] is where it's unfortunate that only two people in the class have done a
[00:54:36] people in the class have done a compiler's class um so um so a simple
[00:54:41] compiler's class um so um so a simple form of passing that's also used in
[00:54:43] form of passing that's also used in compilers classes something called shift
[00:54:46] compilers classes something called shift reduced paing where you start sort of
[00:54:49] reduced paing where you start sort of bottom up and you start putting little
[00:54:51] bottom up and you start putting little units together and build bigger
[00:54:53] units together and build bigger constituents but um if most people have
[00:54:56] constituents but um if most people have hav't seen it that's not going to be um
[00:54:59] hav't seen it that's not going to be um very much um help so I'm going to give
[00:55:01] very much um help so I'm going to give you a concrete example so the things to
[00:55:04] you a concrete example so the things to know is we' have two data structure well
[00:55:07] know is we' have two data structure well we have more than two I guess um for
[00:55:09] we have more than two I guess um for dealing with the sentence we have two
[00:55:11] dealing with the sentence we have two data structures we have a buffer which
[00:55:14] data structures we have a buffer which has the words of our input sentence and
[00:55:17] has the words of our input sentence and then we start building pieces of um
[00:55:19] then we start building pieces of um sentence structure which we put on a
[00:55:22] sentence structure which we put on a stack and a little trick to know is for
[00:55:25] stack and a little trick to know is for the buffer the the top is written to the
[00:55:27] the buffer the the top is written to the left and for the stack the top is
[00:55:29] left and for the stack the top is written to the right um and so we take
[00:55:32] written to the right um and so we take actions which are like shift and reduce
[00:55:35] actions which are like shift and reduce actions and when we take ARC building
[00:55:39] actions and when we take ARC building actions we build up a set of dependency
[00:55:41] actions we build up a set of dependency arcs which are going to be the
[00:55:43] arcs which are going to be the dependency structure of our sentence and
[00:55:46] dependency structure of our sentence and that's all incredibly abstract um and so
[00:55:49] that's all incredibly abstract um and so I'm going to show an example um which
[00:55:52] I'm going to show an example um which hopefully will a bit give the idea um
[00:55:57] hopefully will a bit give the idea um so here's an example so I want to do
[00:55:59] so here's an example so I want to do this very um simple example um of
[00:56:04] this very um simple example um of passing up the sentence I ate fish um so
[00:56:10] passing up the sentence I ate fish um so I so the way I do this is I have my
[00:56:13] I so the way I do this is I have my stack and so I start by putting the root
[00:56:17] stack and so I start by putting the root symbol on my stack and then I have in my
[00:56:20] symbol on my stack and then I have in my buffer all the words of the sentence and
[00:56:23] buffer all the words of the sentence and so that's the sort of start condition
[00:56:25] so that's the sort of start condition I've written and very small print there
[00:56:28] I've written and very small print there then for each step of processing I have
[00:56:31] then for each step of processing I have a choice of three operations I can
[00:56:34] a choice of three operations I can either shift which moves the top word on
[00:56:38] either shift which moves the top word on the buffer onto the stack or I can do
[00:56:42] the buffer onto the stack or I can do left Arc or right Arc and these are my
[00:56:45] left Arc or right Arc and these are my two reduce operations that build a
[00:56:47] two reduce operations that build a little bit of syntactic structure by
[00:56:50] little bit of syntactic structure by saying that one word is a dependent of
[00:56:52] saying that one word is a dependent of another word in either a left or a right
[00:56:55] another word in either a left or a right direction so here a sequence of
[00:56:57] direction so here a sequence of operations I can take and um so starting
[00:57:02] operations I can take and um so starting off the first thing I can do is shift so
[00:57:05] off the first thing I can do is shift so then I've moved I onto the stack um I
[00:57:09] then I've moved I onto the stack um I can decide that I want to shift again
[00:57:12] can decide that I want to shift again and so then I'd take eight and also move
[00:57:16] and so then I'd take eight and also move it onto the stack and so I've now got
[00:57:18] it onto the stack and so I've now got three things on my
[00:57:21] three things on my stack so at this point you know I can do
[00:57:26] stack so at this point you know I can do other things I mean in particular a left
[00:57:29] other things I mean in particular a left Arc is going to say well I can take the
[00:57:32] Arc is going to say well I can take the top two things on the stack and make the
[00:57:37] top two things on the stack and make the uh the thing on the top The Head and the
[00:57:40] uh the thing on the top The Head and the thing one down on the stack a dependent
[00:57:43] thing one down on the stack a dependent of it so if I do a left Arc operation
[00:57:47] of it so if I do a left Arc operation I'm effectively saying that the I is a
[00:57:49] I'm effectively saying that the I is a dependent of eight and then I pop both
[00:57:52] dependent of eight and then I pop both of then I pop the dependent off the
[00:57:55] of then I pop the dependent off the stack back but I add on that I've built
[00:57:59] stack back but I add on that I've built um a dependency that I made I a
[00:58:01] um a dependency that I made I a dependent of eight I could then do
[00:58:04] dependent of eight I could then do another shift operation so I shift fish
[00:58:08] another shift operation so I shift fish um from the buffer onto the stack and
[00:58:11] um from the buffer onto the stack and then I can do a right Arc which says um
[00:58:15] then I can do a right Arc which says um okay I'm going to have fish as a
[00:58:17] okay I'm going to have fish as a dependent of eight so then fish
[00:58:19] dependent of eight so then fish disappears from the stack and I add in
[00:58:22] disappears from the stack and I add in this new dependency saying fishes
[00:58:26] this new dependency saying fishes dependent of eight um I then do right
[00:58:29] dependent of eight um I then do right Arc again um which is then saying that
[00:58:34] Arc again um which is then saying that um eight is a dependent of root so I'm
[00:58:37] um eight is a dependent of root so I'm left with just root on my stack and I've
[00:58:39] left with just root on my stack and I've built a new dependent saying eight is a
[00:58:41] built a new dependent saying eight is a dependent of root and at this point I've
[00:58:44] dependent of root and at this point I've gone to the finishing condition my
[00:58:46] gone to the finishing condition my finishing condition is that my buffer is
[00:58:48] finishing condition is that my buffer is empty and my um stack contains just the
[00:58:52] empty and my um stack contains just the word root um and so this gives me a
[00:58:56] word root um and so this gives me a little step set of operations referred
[00:59:00] little step set of operations referred to as the transitions of
[00:59:02] to as the transitions of transition-based passing and by making a
[00:59:04] transition-based passing and by making a sequence of these different transitions
[00:59:07] sequence of these different transitions I can build sentence structure and I've
[00:59:10] I can build sentence structure and I've got choices of when to shift and when to
[00:59:14] got choices of when to shift and when to reduce and whether to reduce left or
[00:59:17] reduce and whether to reduce left or reduce right the arc left Arc right and
[00:59:20] reduce right the arc left Arc right and so by making different ones of those
[00:59:22] so by making different ones of those choices I could make any structure for
[00:59:25] choices I could make any structure for the sentence that I wanted to so you
[00:59:28] the sentence that I wanted to so you know if I somehow thought that this
[00:59:31] know if I somehow thought that this sentence should have a different
[00:59:33] sentence should have a different structure and that I should be the head
[00:59:36] structure and that I should be the head and eights are dependent of that and
[00:59:38] and eights are dependent of that and fishes are dependent of that well I
[00:59:41] fishes are dependent of that well I could achieve this by making some
[00:59:43] could achieve this by making some different choices as to I'd now be
[00:59:47] different choices as to I'd now be saying I was doing a right Arc operation
[00:59:50] saying I was doing a right Arc operation so that eight would become a dependent
[00:59:52] so that eight would become a dependent of I rather than the other way around so
[00:59:54] of I rather than the other way around so the choices of which oper operations I
[00:59:56] the choices of which oper operations I take determine the syntactic structure
[00:59:59] take determine the syntactic structure the set of dependencies that I have
[01:00:02] the set of dependencies that I have built um which are my set of
[01:00:04] built um which are my set of dependencies down here now the center of
[01:00:07] dependencies down here now the center of dependencies I built were exactly the
[01:00:09] dependencies I built were exactly the right ones because at each step I took
[01:00:12] right ones because at each step I took the right
[01:00:13] the right operation um and so the essential um
[01:00:18] operation um and so the essential um idea of um transition based paing and
[01:00:23] idea of um transition based paing and where it came to the four was
[01:00:26] where it came to the four was um there was a particular guy who's I've
[01:00:30] um there was a particular guy who's I've got a photo of him in somewhere in a bit
[01:00:32] got a photo of him in somewhere in a bit I thought um so yoram Nea um is a
[01:00:35] I thought um so yoram Nea um is a Swedish NLP person um and in the early
[01:00:40] Swedish NLP person um and in the early 2000s um he came up with the idea of um
[01:00:45] 2000s um he came up with the idea of um rather than doing the kind of dynamic
[01:00:47] rather than doing the kind of dynamic programming and chart paing and things
[01:00:50] programming and chart paing and things that people commonly used to do with
[01:00:53] that people commonly used to do with PES these days we have machine Lear
[01:00:56] PES these days we have machine Lear learning so maybe we could build a fast
[01:00:59] learning so maybe we could build a fast efficient paa and the way we're going to
[01:01:02] efficient paa and the way we're going to build it is with this making a sequence
[01:01:06] build it is with this making a sequence of Transitions and it'll be the job of
[01:01:09] of Transitions and it'll be the job of the machine learning to predict what is
[01:01:12] the machine learning to predict what is the right transition at each point in
[01:01:14] the right transition at each point in time so if you do that you know at each
[01:01:19] time so if you do that you know at each point you're dealing with one thing and
[01:01:23] point you're dealing with one thing and so the number of operations you're doing
[01:01:25] so the number of operations you're doing to pass a sentence is linear so this
[01:01:28] to pass a sentence is linear so this gives a linear time passing algorithm
[01:01:32] gives a linear time passing algorithm whereas if um you've seen context free
[01:01:35] whereas if um you've seen context free grammars and stuff like that in CS 103
[01:01:39] grammars and stuff like that in CS 103 and you want to do anything where you're
[01:01:41] and you want to do anything where you're fully considering the paes and
[01:01:43] fully considering the paes and structures of context free grammars
[01:01:45] structures of context free grammars you've then got a cubic time algorithm
[01:01:48] you've then got a cubic time algorithm which is much less Pleasant to be
[01:01:50] which is much less Pleasant to be dealing with um so for the simplest form
[01:01:54] dealing with um so for the simplest form of transition-based paing you do no
[01:01:56] of transition-based paing you do no search whatsoever at each step you're
[01:01:59] search whatsoever at each step you're just predicting the next transition and
[01:02:02] just predicting the next transition and so you're doing this sort of sequence of
[01:02:04] so you're doing this sort of sequence of transition predictions as machine
[01:02:07] transition predictions as machine learning operations and that sequence
[01:02:09] learning operations and that sequence gives you the path structure of the
[01:02:11] gives you the path structure of the sentence and the essential result that
[01:02:14] sentence and the essential result that Neo is able to show is that machine
[01:02:17] Neo is able to show is that machine learning um is good enough that you can
[01:02:20] learning um is good enough that you can do this and get a very accurate paraa
[01:02:24] do this and get a very accurate paraa despite the fact that it no search
[01:02:26] despite the fact that it no search whatsoever is just doing predictions in
[01:02:29] whatsoever is just doing predictions in this
[01:02:31] this way okay um so how did so when he did in
[01:02:38] way okay um so how did so when he did in 2005 that was before neural networks
[01:02:40] 2005 that was before neural networks came to the four and so the way he was
[01:02:43] came to the four and so the way he was doing it was by using a sort of an older
[01:02:47] doing it was by using a sort of an older style um symbolic feature based um
[01:02:50] style um symbolic feature based um machine Learning System so he had a big
[01:02:53] machine Learning System so he had a big classifier which might have been a list
[01:02:55] classifier which might have been a list istic regression classifier or something
[01:02:58] istic regression classifier or something else like a support Vector machine and
[01:03:00] else like a support Vector machine and so to power that he was using indicator
[01:03:04] so to power that he was using indicator features so the kind of features you'd
[01:03:06] features so the kind of features you'd use is that the word on the top of the
[01:03:09] use is that the word on the top of the stack is the word good and it's part of
[01:03:11] stack is the word good and it's part of speech as an adjective or um the word um
[01:03:17] speech as an adjective or um the word um on the top of the stack is good but the
[01:03:21] on the top of the stack is good but the word that's sort of second on the stack
[01:03:24] word that's sort of second on the stack is the verb had
[01:03:26] is the verb had right you'd get these sort of
[01:03:27] right you'd get these sort of combinations of matching functions and
[01:03:30] combinations of matching functions and they would be used as features in a
[01:03:32] they would be used as features in a machine Learning System to predict the
[01:03:34] machine Learning System to predict the pars um but the problem is that once you
[01:03:37] pars um but the problem is that once you started building these features that
[01:03:41] started building these features that were conjunctions of multiple terms you
[01:03:43] were conjunctions of multiple terms you ended up with millions and millions of
[01:03:45] ended up with millions and millions of features right because you putting
[01:03:47] features right because you putting particular words and features um and
[01:03:50] particular words and features um and then you're combining choices of
[01:03:51] then you're combining choices of multiple words so they're just millions
[01:03:53] multiple words so they're just millions and millions of features so had to deal
[01:03:56] and millions of features so had to deal with millions and millions of features
[01:03:58] with millions and millions of features and furthermore individual features were
[01:04:01] and furthermore individual features were exceedingly sparse that you barely ever
[01:04:03] exceedingly sparse that you barely ever saw them right that you'd have a feature
[01:04:06] saw them right that you'd have a feature that only turned up you know 10 times in
[01:04:08] that only turned up you know 10 times in a million sentences because they are
[01:04:10] a million sentences because they are matching these very precise systems so
[01:04:13] matching these very precise systems so on the one hand um by making these
[01:04:16] on the one hand um by making these feature conjunctions paing got more
[01:04:19] feature conjunctions paing got more accurate and indeed people produced
[01:04:21] accurate and indeed people produced pretty accurate paes in those days but
[01:04:24] pretty accurate paes in those days but they had sort of these unappealing
[01:04:26] they had sort of these unappealing characteristics of this
[01:04:28] characteristics of this sort um yeah so uh before going on
[01:04:33] sort um yeah so uh before going on further I should just explain how we
[01:04:35] further I should just explain how we evaluate dependency paes um so um to
[01:04:41] evaluate dependency paes um so um to evaluate dependency paes we're basically
[01:04:44] evaluate dependency paes we're basically assessing are you getting the
[01:04:47] assessing are you getting the dependency arcs arrows your proposing
[01:04:50] dependency arcs arrows your proposing right um so here is someone's dependency
[01:04:53] right um so here is someone's dependency PA she saw the video
[01:04:57] PA she saw the video lecture
[01:04:59] lecture um well actually sorry that's the gold
[01:05:01] um well actually sorry that's the gold pa okay that's a correct pa okay she saw
[01:05:05] pa okay that's a correct pa okay she saw the video lecture that's a correct PA so
[01:05:08] the video lecture that's a correct PA so you can write out what are um the the
[01:05:11] you can write out what are um the the different
[01:05:12] different dependencies right so one's head is two
[01:05:16] dependencies right so one's head is two two's head is zero word three's head is
[01:05:19] two's head is zero word three's head is five words Four's head is five word
[01:05:22] five words Four's head is five word five's head is two so these pairs of
[01:05:25] five's head is two so these pairs of numbers represent our dependencies then
[01:05:29] numbers represent our dependencies then if someone proposes a PA of the sentence
[01:05:32] if someone proposes a PA of the sentence you can literally say okay which of
[01:05:35] you can literally say okay which of these did they get right so they didn't
[01:05:36] these did they get right so they didn't get this one right um they got the rest
[01:05:40] get this one right um they got the rest of them right so they accuracy is 80%
[01:05:43] of them right so they accuracy is 80% and so sometimes people just assess the
[01:05:46] and so sometimes people just assess the arcs unlabeled and so that's referred to
[01:05:49] arcs unlabeled and so that's referred to as unlabeled dependency accuracy but
[01:05:53] as unlabeled dependency accuracy but sometimes people also want to label them
[01:05:55] sometimes people also want to label them with with um subject determiner object
[01:05:59] with with um subject determiner object Etc and say also are you getting the
[01:06:01] Etc and say also are you getting the labels right so in this case only two of
[01:06:04] labels right so in this case only two of the five labels are right so the labeled
[01:06:07] the five labels are right so the labeled accuracy of the dependency passes is
[01:06:12] 40% okay um
[01:06:17] 40% okay um so so that was sort of what people did
[01:06:21] so so that was sort of what people did um until the mid
[01:06:24] um until the mid 2010s um and and I sort of already
[01:06:26] 2010s um and and I sort of already started saying this the problems with
[01:06:29] started saying this the problems with indicator features were they were
[01:06:33] indicator features were they were sparse you didn't see them often they
[01:06:36] sparse you didn't see them often they were incomplete because there's some
[01:06:38] were incomplete because there's some words and combinations you've seen and
[01:06:40] words and combinations you've seen and some you just didn't see in the training
[01:06:41] some you just didn't see in the training data so you're missing features um but
[01:06:45] data so you're missing features um but the final problem is actually just
[01:06:47] the final problem is actually just Computing all those um symbolic features
[01:06:50] Computing all those um symbolic features was just expensive it turns out that if
[01:06:53] was just expensive it turns out that if you did runtime analysis most of time in
[01:06:56] you did runtime analysis most of time in the paing wasn't in doing the machine
[01:06:58] the paing wasn't in doing the machine learning decisions it was just simply in
[01:07:01] learning decisions it was just simply in Computing the features that you put into
[01:07:03] Computing the features that you put into this dependency paa so as new net
[01:07:06] this dependency paa so as new net started to um show that they were
[01:07:08] started to um show that they were successful for things that suggested
[01:07:11] successful for things that suggested that maybe you could build um a better
[01:07:14] that maybe you could build um a better dependency paa by using a newal net um
[01:07:18] dependency paa by using a newal net um transition-based dependency paa which
[01:07:20] transition-based dependency paa which would benefit from the kind of dens and
[01:07:23] would benefit from the kind of dens and compact um feature Vector
[01:07:26] compact um feature Vector representations that we've already
[01:07:28] representations that we've already started to see um and so that's um what
[01:07:32] started to see um and so that's um what started to be explored and in
[01:07:36] particular who was then a PhD student of
[01:07:39] particular who was then a PhD student of mine um and was head TA of 224n twice
[01:07:43] mine um and was head TA of 224n twice actually in the earlier days um so um
[01:07:47] actually in the earlier days um so um she built um uh neural transition-based
[01:07:50] she built um uh neural transition-based dependency paa um and showed the success
[01:07:54] dependency paa um and showed the success of this method
[01:07:56] of this method um so this was um nea's transition-based
[01:07:59] um so this was um nea's transition-based dependency paa um people had also
[01:08:03] dependency paa um people had also explored other methods of dependency
[01:08:05] explored other methods of dependency passing so these were two graph-based
[01:08:07] passing so these were two graph-based dependency paes and essentially um for
[01:08:11] dependency paes and essentially um for the kind of um symbolic feature machine
[01:08:15] the kind of um symbolic feature machine learning methods um NE paa was really
[01:08:19] learning methods um NE paa was really fast because I was using this linear
[01:08:21] fast because I was using this linear trans um transition based paing idea
[01:08:25] trans um transition based paing idea that depend the graph-based dependency
[01:08:28] that depend the graph-based dependency paes were way way slower right you know
[01:08:31] paes were way way slower right you know they're about what 50 times slower um
[01:08:33] they're about what 50 times slower um but they were slightly more accurate you
[01:08:35] but they were slightly more accurate you can see here that you're getting a bit
[01:08:37] can see here that you're getting a bit better numbers so essentially what was
[01:08:40] better numbers so essentially what was able to show was um you could build
[01:08:44] able to show was um you could build something that was basically as accurate
[01:08:46] something that was basically as accurate as the best known um graph-based
[01:08:48] as the best known um graph-based dependency pases but it was fast like
[01:08:52] dependency pases but it was fast like other um transition based fases indeed
[01:08:56] other um transition based fases indeed despite the fact that it's you might
[01:08:58] despite the fact that it's you might think that oh now I've got real numbers
[01:09:00] think that oh now I've got real numbers and matrices and stuff surely that
[01:09:02] and matrices and stuff surely that should be slowing me down the reality
[01:09:05] should be slowing me down the reality was that the symbolic models spent so
[01:09:08] was that the symbolic models spent so much time in feature um computation that
[01:09:12] much time in feature um computation that actually you could make it faster at the
[01:09:14] actually you could make it faster at the same time by using a newal
[01:09:16] same time by using a newal network okay so how did that work um
[01:09:20] network okay so how did that work um well so we've already seen word
[01:09:22] well so we've already seen word embedding so it's going to exploit word
[01:09:25] embedding so it's going to exploit word embed so it can use word representations
[01:09:29] embed so it can use word representations and that has the advantage that even if
[01:09:31] and that has the advantage that even if you haven't seen particular words and
[01:09:32] you haven't seen particular words and particular configurations you've seen
[01:09:35] particular configurations you've seen similar words and so it can exploit
[01:09:38] similar words and so it can exploit what's likely in terms of word
[01:09:40] what's likely in terms of word similarity but it went a bit further
[01:09:42] similarity but it went a bit further than that because why only have
[01:09:44] than that because why only have distributed representations of words we
[01:09:47] distributed representations of words we also have parts of speech and although I
[01:09:49] also have parts of speech and although I sort of said just noun verb adjective
[01:09:53] sort of said just noun verb adjective most actual systems in NLP of parts of
[01:09:57] most actual systems in NLP of parts of speech are much more fine grained so
[01:09:59] speech are much more fine grained so they have different parts of speech for
[01:10:01] they have different parts of speech for plural nouns versus singular nouns so
[01:10:04] plural nouns versus singular nouns so they're sort of different symbols but
[01:10:06] they're sort of different symbols but they're very similar to each other so we
[01:10:08] they're very similar to each other so we might give them distributed
[01:10:10] might give them distributed representation so they're also close to
[01:10:12] representation so they're also close to each other and the same for the types of
[01:10:16] each other and the same for the types of our labels for dependencies some of them
[01:10:18] our labels for dependencies some of them are pretty closely related as well so
[01:10:20] are pretty closely related as well so all of these were being given
[01:10:22] all of these were being given distributed
[01:10:24] distributed representations and so then to represent
[01:10:27] representations and so then to represent the state of the dependency paa for
[01:10:30] the state of the dependency paa for predicting transitions what you were
[01:10:32] predicting transitions what you were doing is you had the same kind of stack
[01:10:35] doing is you had the same kind of stack and buffer and you are taking the key
[01:10:39] and buffer and you are taking the key elements of the stack and the buffer
[01:10:41] elements of the stack and the buffer which are essentially the first thing on
[01:10:43] which are essentially the first thing on the buffer the word that you would be
[01:10:45] the buffer the word that you would be shifting if you're going to do a shift
[01:10:47] shifting if you're going to do a shift and the two things at the top of the
[01:10:49] and the two things at the top of the stack so these are the things that if
[01:10:51] stack so these are the things that if you're either doing a left Arc or a
[01:10:53] you're either doing a left Arc or a right Arc they're things that you're
[01:10:55] right Arc they're things that you're considering combining so for those
[01:10:58] considering combining so for those you're going to be taking the
[01:11:00] you're going to be taking the distributed representations of the word
[01:11:03] distributed representations of the word and their parts of speech and also with
[01:11:06] and their parts of speech and also with a bit more complexity um for
[01:11:08] a bit more complexity um for dependencies you've already constructed
[01:11:10] dependencies you've already constructed if maybe something on the stack is
[01:11:12] if maybe something on the stack is already involved in a dependency each of
[01:11:15] already involved in a dependency each of those we're going to take their
[01:11:16] those we're going to take their distributed representations and we're
[01:11:18] distributed representations and we're going to just concatenate them together
[01:11:21] going to just concatenate them together to produce a big um Vector in the same
[01:11:24] to produce a big um Vector in the same way were concatenating together the five
[01:11:27] way were concatenating together the five words in the last class for predicting
[01:11:30] words in the last class for predicting whether something was the location or
[01:11:32] whether something was the location or not and then we're going to feed that um
[01:11:36] not and then we're going to feed that um into our neural network so um we our
[01:11:41] into our neural network so um we our input layer is our concatenate
[01:11:43] input layer is our concatenate distributed
[01:11:44] distributed representations we're going to put that
[01:11:46] representations we're going to put that through a hidden layer which is like we
[01:11:48] through a hidden layer which is like we were talking about last time WX plus b
[01:11:52] were talking about last time WX plus b then put through a relue
[01:11:54] then put through a relue nonlinearity and then we're going to put
[01:11:57] nonlinearity and then we're going to put above that the same kind of um another
[01:12:01] above that the same kind of um another multiply by a matri so we've got a
[01:12:03] multiply by a matri so we've got a second layer of neural network uh plus
[01:12:06] second layer of neural network uh plus B2 and we're going to take the output of
[01:12:09] B2 and we're going to take the output of that and then we're going to put that
[01:12:10] that and then we're going to put that through a soft Max that gives a
[01:12:12] through a soft Max that gives a probability distribution over whether to
[01:12:16] probability distribution over whether to um shift or do a left Arc or a right Arc
[01:12:20] um shift or do a left Arc or a right Arc operation and so the other way that this
[01:12:24] operation and so the other way that this crucially gave us more power is that
[01:12:27] crucially gave us more power is that other people's dependency paes were
[01:12:30] other people's dependency paes were still using linear classifiers things
[01:12:33] still using linear classifiers things like support Vector machines or logistic
[01:12:36] like support Vector machines or logistic regressions where we had a deep neural
[01:12:38] regressions where we had a deep neural network that gave us a nonlinear
[01:12:41] network that gave us a nonlinear classifier but and so that's why we
[01:12:43] classifier but and so that's why we could be more accurate than other um
[01:12:46] could be more accurate than other um previous um transition based paes and so
[01:12:49] previous um transition based paes and so this um essentially um showed that you
[01:12:53] this um essentially um showed that you could build this very accurate neural
[01:12:56] could build this very accurate neural dependency paa and that it
[01:12:59] dependency paa and that it outperformed um symbolic um probalistic
[01:13:03] outperformed um symbolic um probalistic representations and basically was as
[01:13:05] representations and basically was as good as any other dependency paa that
[01:13:08] good as any other dependency paa that was known so um back a decade to go um
[01:13:12] was known so um back a decade to go um this was a big hit um people got very
[01:13:14] this was a big hit um people got very excited about it um the people at Google
[01:13:17] excited about it um the people at Google got very excited about it because this
[01:13:19] got very excited about it because this gave a um a scalable way remember it's
[01:13:22] gave a um a scalable way remember it's linear time in which you could
[01:13:24] linear time in which you could efficiently go of and pass um the entire
[01:13:27] efficiently go of and pass um the entire web um so they did um some further work
[01:13:31] web um so they did um some further work on taking that model and proving it so
[01:13:34] on taking that model and proving it so that they um made an a deeper neural
[01:13:37] that they um made an a deeper neural network version with bigger vectors and
[01:13:41] network version with bigger vectors and better tuned hyperparameters and they
[01:13:43] better tuned hyperparameters and they added on to a beam search I've just
[01:13:46] added on to a beam search I've just presented the greedy version where you
[01:13:48] presented the greedy version where you always CH just immediately make the best
[01:13:50] always CH just immediately make the best choice but you can improve these paas by
[01:13:53] choice but you can improve these paas by doing some amount of search that does
[01:13:56] doing some amount of search that does help um and so um they pushed that up
[01:14:00] help um and so um they pushed that up and so rather than our kind of 92 uas
[01:14:04] and so rather than our kind of 92 uas here they got it to you know 94
[01:14:08] here they got it to you know 94 94.6 and I mean
[01:14:11] 94.6 and I mean um you're uh you're probably um all too
[01:14:17] um you're uh you're probably um all too young to remember this but um you know
[01:14:20] young to remember this but um you know really at the time of
[01:14:22] really at the time of 2016 you know Google did their kind of
[01:14:25] 2016 you know Google did their kind of typical big PR PR Splash for dependency
[01:14:29] typical big PR PR Splash for dependency parza which kind of blew my mind since I
[01:14:31] parza which kind of blew my mind since I didn't ever think that anyone was really
[01:14:34] didn't ever think that anyone was really going to be writing articles in W and
[01:14:36] going to be writing articles in W and Venture beat and those kind of tech um
[01:14:39] Venture beat and those kind of tech um blogs but you know Google had it all
[01:14:42] blogs but you know Google had it all over the place of the world's most
[01:14:44] over the place of the world's most accurate paa and they gave it a silly
[01:14:47] accurate paa and they gave it a silly name pzy mpas face which really worked
[01:14:50] name pzy mpas face which really worked well um for getting lots of media pickup
[01:14:53] well um for getting lots of media pickup um and so that was then um a very
[01:14:56] um and so that was then um a very successful
[01:14:58] successful paa um I've still got a couple of
[01:15:01] paa um I've still got a couple of minutes left so let me just um do the
[01:15:04] minutes left so let me just um do the last um three slides to show you sort of
[01:15:08] last um three slides to show you sort of another way of doing things which is
[01:15:10] another way of doing things which is also actually also a powerful paing
[01:15:12] also actually also a powerful paing method that is commonly used so um that
[01:15:16] method that is commonly used so um that was transition-based paing and that's
[01:15:18] was transition-based paing and that's what you'll use an assignment to another
[01:15:21] what you'll use an assignment to another way of doing things with dependencies
[01:15:23] way of doing things with dependencies and paing can be done neural is what's
[01:15:26] and paing can be done neural is what's referred to as graph-based dependency
[01:15:28] referred to as graph-based dependency paes and in graph-based dependency paes
[01:15:32] paes and in graph-based dependency paes what you do is um for each word you sort
[01:15:38] what you do is um for each word you sort of ask for each word what am I a
[01:15:41] of ask for each word what am I a dependent of right so if the sentence is
[01:15:45] dependent of right so if the sentence is the big cat sat each word for example
[01:15:48] the big cat sat each word for example big has to be a dependent of one of the
[01:15:51] big has to be a dependent of one of the other four words in this sentence
[01:15:53] other four words in this sentence including this possibility of root so we
[01:15:56] including this possibility of root so we ask am I dependent of that am I
[01:15:58] ask am I dependent of that am I dependent of root am I dependent of cat
[01:16:01] dependent of root am I dependent of cat am I dependent of sat and we want to
[01:16:03] am I dependent of sat and we want to score each of those possibilities and so
[01:16:06] score each of those possibilities and so hopefully we decide the most likely one
[01:16:09] hopefully we decide the most likely one is the Bigg as a dependent of cat and
[01:16:12] is the Bigg as a dependent of cat and then we're going to do the same for
[01:16:14] then we're going to do the same for every other word so you know sat could
[01:16:16] every other word so you know sat could be a dependent of any of these words and
[01:16:19] be a dependent of any of these words and so we could start asking okay which of
[01:16:22] so we could start asking okay which of these words is it most likely a
[01:16:25] these words is it most likely a dependent
[01:16:27] dependent of uh beat to sat cat to sat um sorry
[01:16:31] of uh beat to sat cat to sat um sorry that's unreadable now but hopefully we
[01:16:34] that's unreadable now but hopefully we decide um that sat most likely as the
[01:16:37] decide um that sat most likely as the verb is a dependent of root so we sort
[01:16:41] verb is a dependent of root so we sort of scoring the N squared possible you
[01:16:44] of scoring the N squared possible you know dependencies of the sentence and
[01:16:47] know dependencies of the sentence and each one is given a score and then once
[01:16:50] each one is given a score and then once we've done that our job is let me go to
[01:16:54] we've done that our job is let me go to this one cleaner okay we've decided the
[01:16:56] this one cleaner okay we've decided the good one there and so we we're going to
[01:16:58] good one there and so we we're going to do this using some of the same features
[01:17:01] do this using some of the same features we talked about looking at the words at
[01:17:03] we talked about looking at the words at each end looking at what occurs between
[01:17:05] each end looking at what occurs between them looking at what occurs around them
[01:17:08] them looking at what occurs around them um thinking about um things um and then
[01:17:12] um thinking about um things um and then once we've done that the only other
[01:17:14] once we've done that the only other thing that's a constraint is well we
[01:17:16] thing that's a constraint is well we want the dependencies to form a tree um
[01:17:20] want the dependencies to form a tree um so that we need to do um something like
[01:17:23] so that we need to do um something like a minimum spanning tree algorithm to
[01:17:26] a minimum spanning tree algorithm to sort of find the minimum cost tree
[01:17:28] sort of find the minimum cost tree because we don't want to find a solution
[01:17:31] because we don't want to find a solution where there are Cycles or the parts of
[01:17:34] where there are Cycles or the parts of the sentence end up disconnected with
[01:17:35] the sentence end up disconnected with each other um and so that's graph-based
[01:17:38] each other um and so that's graph-based dependency paes and so just as in the
[01:17:42] dependency paes and so just as in the older symbolic paing days where the
[01:17:44] older symbolic paing days where the graph based dependency paes were more
[01:17:47] graph based dependency paes were more accurate than the transition based paes
[01:17:50] accurate than the transition based paes um that we then started doing some work
[01:17:53] um that we then started doing some work on neural graph based depend dependency
[01:17:55] on neural graph based depend dependency paring and so here's our neurog graph
[01:17:57] paring and so here's our neurog graph based dependency paring um which was
[01:18:00] based dependency paring um which was then a bit over a percent more accurate
[01:18:04] then a bit over a percent more accurate than pzy mpaz face the world's best um
[01:18:07] than pzy mpaz face the world's best um dependency paer um so um so that got us
[01:18:11] dependency paer um so um so that got us to 2017 I mean obviously this is still a
[01:18:14] to 2017 I mean obviously this is still a few years ago but to get further into
[01:18:17] few years ago but to get further into the latest um paring Stories We then
[01:18:20] the latest um paring Stories We then need to sort of get into the ER of large
[01:18:22] need to sort of get into the ER of large language models which I'm not doing
[01:18:24] language models which I'm not doing today um but it's this neural graph
[01:18:26] today um but it's this neural graph based dependency paa um that's in um
[01:18:30] based dependency paa um that's in um stanza our open- source um paing
[01:18:33] stanza our open- source um paing software that's available and that you
[01:18:35] software that's available and that you can see it's using this algorithm as the
[01:18:37] can see it's using this algorithm as the more accurate one okay so now you
[01:18:40] more accurate one okay so now you hopefully know everything about
[01:18:41] hopefully know everything about syntactic structure constituency
[01:18:44] syntactic structure constituency independency paing and are fully
[01:18:46] independency paing and are fully qualified to do assignment to so good
[01:18:48] qualified to do assignment to so good luck with that thanks

Stanford CS224N: NLP with Deep Learning | Spring 2024 | Lecture 5 - Recurrent Neural Networks

Source: https://www.youtube.com/watch?v=fyc0Jzr74y4

---

Transcript

[00:00:06] okay um let me get started for today so
[00:00:09] okay um let me get started for today so for today um first of all I'm going to
[00:00:13] for today um first of all I'm going to spend a few minutes talking about a
[00:00:16] spend a few minutes talking about a couple more new net Concepts including
[00:00:19] couple more new net Concepts including actually a couple of the concepts that
[00:00:21] actually a couple of the concepts that turn up in assignment two um um then the
[00:00:26] turn up in assignment two um um then the bulk of today is then going to be moving
[00:00:29] bulk of today is then going to be moving on to introducing what are language
[00:00:32] on to introducing what are language models um and then after introducing
[00:00:36] models um and then after introducing language models we're going to um
[00:00:39] language models we're going to um introduce a new kind of newal network
[00:00:42] introduce a new kind of newal network which is one way to build language
[00:00:43] which is one way to build language models which is recurrent newal networks
[00:00:47] models which is recurrent newal networks um they're important thing to know about
[00:00:49] um they're important thing to know about and we use them in assignment three um
[00:00:52] and we use them in assignment three um but they're certainly not the only way
[00:00:54] but they're certainly not the only way to build language models in fact
[00:00:56] to build language models in fact probably a lot of you already know that
[00:00:58] probably a lot of you already know that there's this other kind of new Network
[00:01:00] there's this other kind of new Network called Transformers and we'll get on to
[00:01:02] called Transformers and we'll get on to those after we've done recurrent new
[00:01:05] those after we've done recurrent new Nets um Talk a bit about problems with
[00:01:08] Nets um Talk a bit about problems with um Rec current new networks and well if
[00:01:10] um Rec current new networks and well if I have time I'll get onto the um recap
[00:01:13] I have time I'll get onto the um recap um before getting into the content of
[00:01:15] um before getting into the content of the class I thought I could just spend a
[00:01:17] the class I thought I could just spend a minute on giving you the stats of who is
[00:01:19] minute on giving you the stats of who is in um
[00:01:21] in um cs224n um who's in
[00:01:24] cs224n um who's in cs224n kind of looks like the pie charts
[00:01:27] cs224n kind of looks like the pie charts they show in CS 106a these days um
[00:01:30] they show in CS 106a these days um except more grad students I guess um so
[00:01:33] except more grad students I guess um so the four big groups are the computer
[00:01:36] the four big groups are the computer science undergrads the computer science
[00:01:38] science undergrads the computer science grads um the Undeclared
[00:01:41] grads um the Undeclared undergraduates and the um ndo grads so
[00:01:44] undergraduates and the um ndo grads so this is a large portion of the scpd
[00:01:46] this is a large portion of the scpd students though um some of them are
[00:01:48] students though um some of them are Under Computer Science grads so that
[00:01:51] Under Computer Science grads so that makes up about 60% of the audience um
[00:01:55] makes up about 60% of the audience um and if you're not in one of those four
[00:01:56] and if you're not in one of those four big groups um you're in the other 40%
[00:02:00] big groups um you're in the other 40% and everybody is somewhere so there are
[00:02:01] and everybody is somewhere so there are lots of other interesting groups down
[00:02:04] lots of other interesting groups down here so you know the the bright orange
[00:02:06] here so you know the the bright orange down here that's where the math and
[00:02:09] down here that's where the math and physics phds are um up here I mean
[00:02:13] physics phds are um up here I mean interestingly we now have more
[00:02:15] interestingly we now have more statistics grad students and there are
[00:02:17] statistics grad students and there are ssis undergrads it didn't used to be
[00:02:20] ssis undergrads it didn't used to be that way around in NLP classes um and
[00:02:23] that way around in NLP classes um and you know one of my favorite groups um
[00:02:26] you know one of my favorite groups um the little um magenta group down here um
[00:02:30] the little um magenta group down here um these are the humanity undergrads yay
[00:02:33] these are the humanity undergrads yay Humanity's undergrads um in terms of
[00:02:37] Humanity's undergrads um in terms of years it breaks down like this um first
[00:02:40] years it breaks down like this um first year grad students are the biggest group
[00:02:42] year grad students are the biggest group tons of Juniors and seniors and a couple
[00:02:46] tons of Juniors and seniors and a couple of Brave FR are any Brave FR here today
[00:02:50] of Brave FR are any Brave FR here today [Laughter]
[00:02:52] [Laughter] yeah okay um welcome yeah
[00:02:58] yeah okay um welcome yeah so um modern newal networks especially
[00:03:02] so um modern newal networks especially language models are
[00:03:05] language models are enormous um this chart's sort of out of
[00:03:07] enormous um this chart's sort of out of date because it only goes up to
[00:03:10] date because it only goes up to 2022 but it's sort of actually hard to
[00:03:13] 2022 but it's sort of actually hard to make an accurate chart for 2024 because
[00:03:16] make an accurate chart for 2024 because in the last couple of years um the
[00:03:18] in the last couple of years um the biggest language model makers have in
[00:03:20] biggest language model makers have in general stopped saying how large their
[00:03:22] general stopped saying how large their language models are in terms of
[00:03:23] language models are in terms of parameters but at any rate they're
[00:03:26] parameters but at any rate they're clearly um huge models which um
[00:03:30] clearly um huge models which um have over a 100 billion parameters um
[00:03:34] have over a 100 billion parameters um and so large and then deep in terms of
[00:03:38] and so large and then deep in terms of very many layers new Nets are a
[00:03:41] very many layers new Nets are a Cornerstone of modern NLP systems we're
[00:03:44] Cornerstone of modern NLP systems we're going to be um pretty quickly working
[00:03:47] going to be um pretty quickly working our way up to look at those kind of deep
[00:03:50] our way up to look at those kind of deep models but I just sort of for starting
[00:03:53] models but I just sort of for starting off with something simpler you know I
[00:03:55] off with something simpler you know I did just want to kind of um key you in
[00:03:59] did just want to kind of um key you in for a few minutes into a little bit of
[00:04:01] for a few minutes into a little bit of History right um so um the last time
[00:04:06] History right um so um the last time neural Nets were popular was in the 80s
[00:04:09] neural Nets were popular was in the 80s and 90s and that was when people worked
[00:04:12] and 90s and that was when people worked out the back propagation algorithm Jeff
[00:04:14] out the back propagation algorithm Jeff Hinton and colleagues um made famous the
[00:04:16] Hinton and colleagues um made famous the back propagation algorithm that we've
[00:04:18] back propagation algorithm that we've looked at and that allowed the training
[00:04:21] looked at and that allowed the training of neural Nets with hidden
[00:04:24] of neural Nets with hidden layers um and so but in those days
[00:04:28] layers um and so but in those days pretty much all the new du Nets with
[00:04:30] pretty much all the new du Nets with hidden layers that were trained were
[00:04:32] hidden layers that were trained were trained with one hidden layer you had
[00:04:34] trained with one hidden layer you had the input the hidden layer and the
[00:04:36] the input the hidden layer and the output and that's all that there was and
[00:04:40] output and that's all that there was and the reason for that was for a very very
[00:04:44] the reason for that was for a very very long time people couldn't really get
[00:04:47] long time people couldn't really get things to work um with more hidden
[00:04:50] things to work um with more hidden layers so that only started to change in
[00:04:53] layers so that only started to change in the Resurgence of what often got called
[00:04:56] the Resurgence of what often got called Deep learning but anyway back to neural
[00:04:58] Deep learning but anyway back to neural Nets um started around 2006 and this was
[00:05:04] Nets um started around 2006 and this was um one of the influential papers at the
[00:05:06] um one of the influential papers at the time um greedy layerwise training of
[00:05:08] time um greedy layerwise training of deep neural networks by Yoshua Benjo and
[00:05:11] deep neural networks by Yoshua Benjo and colleagues and so right at the beginning
[00:05:13] colleagues and so right at the beginning of that paper they observed this the the
[00:05:17] of that paper they observed this the the problem however until recently it was
[00:05:19] problem however until recently it was believed too difficult to train deep
[00:05:21] believed too difficult to train deep multi-layer newal networks empirically
[00:05:24] multi-layer newal networks empirically deep networks were generally found to be
[00:05:27] deep networks were generally found to be not better and often worse than new
[00:05:29] not better and often worse than new networks with one or two hidden layers
[00:05:32] networks with one or two hidden layers um Jerry Torro um was actually a faculty
[00:05:35] um Jerry Torro um was actually a faculty member who worked very early on
[00:05:37] member who worked very early on autonomous driving with new networks um
[00:05:40] autonomous driving with new networks um as this is a negative result there's not
[00:05:42] as this is a negative result there's not been much report in the machine learning
[00:05:44] been much report in the machine learning literature um so that really you know
[00:05:49] literature um so that really you know although people had newal networks and
[00:05:51] although people had newal networks and back propagation and current newal
[00:05:53] back propagation and current newal networks we're going to talk about today
[00:05:55] networks we're going to talk about today that for a very long period of time you
[00:06:00] that for a very long period of time you know 15 years or so things seemed
[00:06:03] know 15 years or so things seemed completely stuck in that you couldn't
[00:06:06] completely stuck in that you couldn't although in theory it seemed like deep
[00:06:08] although in theory it seemed like deep neural network should be promising in
[00:06:10] neural network should be promising in practice um they didn't work and so it
[00:06:15] practice um they didn't work and so it really then took some new developments
[00:06:17] really then took some new developments that happened in the late 2000s decade
[00:06:20] that happened in the late 2000s decade and then more profoundly in the 2010s
[00:06:23] and then more profoundly in the 2010s decade to actually figure out how we
[00:06:26] decade to actually figure out how we could have deep neural networks that
[00:06:29] could have deep neural networks that actually worked working far better than
[00:06:31] actually worked working far better than the shallow neural networks and leading
[00:06:33] the shallow neural networks and leading into the networks that um we have today
[00:06:37] into the networks that um we have today and you know we're going to be starting
[00:06:39] and you know we're going to be starting to talk about some of those things in
[00:06:42] to talk about some of those things in this class and come um coming up with
[00:06:46] this class and come um coming up with classes and I mean I I think you know
[00:06:50] classes and I mean I I think you know the tendency when you see the things
[00:06:52] the tendency when you see the things that got new networks to work much
[00:06:55] that got new networks to work much better like the the the natural action
[00:07:00] better like the the the natural action is to sort of shrug and be underwhelmed
[00:07:04] is to sort of shrug and be underwhelmed and think oh is this all there is to it
[00:07:06] and think oh is this all there is to it this doesn't exactly seem like difficult
[00:07:09] this doesn't exactly seem like difficult science um and in some sense that's true
[00:07:13] science um and in some sense that's true they're fairly little introductions of
[00:07:16] they're fairly little introductions of new ideas and tweaks of things but
[00:07:20] new ideas and tweaks of things but nevertheless a handful of little ideas
[00:07:23] nevertheless a handful of little ideas and tweaks of things turn things around
[00:07:27] and tweaks of things turn things around from a field that was sort of stuck for
[00:07:29] from a field that was sort of stuck for 15 years going nowhere and which nearly
[00:07:32] 15 years going nowhere and which nearly everyone had abandoned because of that
[00:07:35] everyone had abandoned because of that to suddenly turning around and there
[00:07:38] to suddenly turning around and there being the ability to train these deeper
[00:07:41] being the ability to train these deeper neural networks which then behaved
[00:07:43] neural networks which then behaved amazingly better as machine learning
[00:07:45] amazingly better as machine learning systems than other things that had
[00:07:48] systems than other things that had preceded them and dominated for the for
[00:07:51] preceded them and dominated for the for the intervening time so that took a lot
[00:07:54] the intervening time so that took a lot of time so what are these things um one
[00:07:58] of time so what are these things um one of them which you can greet with a bit
[00:08:00] of them which you can greet with a bit of a yawn in some sense is doing better
[00:08:04] of a yawn in some sense is doing better regularization of neural Nets so
[00:08:08] regularization of neural Nets so regularization is the idea that Beyond
[00:08:11] regularization is the idea that Beyond just having a loss that we want to
[00:08:14] just having a loss that we want to minimize in terms of describing the data
[00:08:18] minimize in terms of describing the data um we want to in some other ways
[00:08:21] um we want to in some other ways manipulate what parameters we learn so
[00:08:24] manipulate what parameters we learn so that our models work better and so
[00:08:28] that our models work better and so normally we have some more complex loss
[00:08:32] normally we have some more complex loss function that does some regularization
[00:08:35] function that does some regularization the most common way of doing this is
[00:08:37] the most common way of doing this is what's called L2 loss where you add on
[00:08:40] what's called L2 loss where you add on this um parameter squared term at the
[00:08:43] this um parameter squared term at the end and this regularization says you
[00:08:48] end and this regularization says you know it would be kind of good to find a
[00:08:52] know it would be kind of good to find a model with small parameter weights so
[00:08:54] model with small parameter weights so you should be finding the smallest
[00:08:56] you should be finding the smallest parameter weights um that will explain
[00:08:58] parameter weights um that will explain your data well and there's a lot you can
[00:09:01] your data well and there's a lot you can say about um regularization these kind
[00:09:05] say about um regularization these kind of losses they get talked about a lot
[00:09:08] of losses they get talked about a lot more in other classes um like cs229
[00:09:12] more in other classes um like cs229 machine learning and so um I'm not going
[00:09:15] machine learning and so um I'm not going to say very much about it this is in the
[00:09:17] to say very much about it this is in the machine learning theory class um but I
[00:09:20] machine learning theory class um but I do just want to sort of um put in one
[00:09:23] do just want to sort of um put in one note that's sort of very relevant um to
[00:09:29] note that's sort of very relevant um to um what's happened in recent new
[00:09:31] um what's happened in recent new networks work um so the classic view of
[00:09:34] networks work um so the classic view of regularization was we needed this kind
[00:09:37] regularization was we needed this kind of regularization to prevent our
[00:09:39] of regularization to prevent our networks from
[00:09:41] networks from overfitting meaning that they would do a
[00:09:44] overfitting meaning that they would do a very good job at modeling the training
[00:09:47] very good job at modeling the training data but then they would generalize
[00:09:51] data but then they would generalize badly to new data that was shown and so
[00:09:54] badly to new data that was shown and so the picture that you got shown was this
[00:09:56] the picture that you got shown was this that as you train on some training data
[00:10:00] that as you train on some training data your error necessarily goes
[00:10:04] your error necessarily goes down however after some point you start
[00:10:08] down however after some point you start learning specific properties of things
[00:10:11] learning specific properties of things that happen to turn up in those training
[00:10:14] that happen to turn up in those training examples and that you're learning things
[00:10:16] examples and that you're learning things that are only good for the training
[00:10:18] that are only good for the training examples and so they won't generalize
[00:10:21] examples and so they won't generalize well to different pieces of data you see
[00:10:23] well to different pieces of data you see at test time so if you have a separate
[00:10:26] at test time so if you have a separate validation set or a final test test set
[00:10:30] validation set or a final test test set you would and you traced out the error
[00:10:33] you would and you traced out the error or loss on that um validation or test
[00:10:37] or loss on that um validation or test set that after some point it would start
[00:10:39] set that after some point it would start to go up again this is a quirk in my bad
[00:10:43] to go up again this is a quirk in my bad PowerPoint it's just meant to go up um
[00:10:46] PowerPoint it's just meant to go up um and the fact that it goes up um is then
[00:10:49] and the fact that it goes up um is then you have overfit your training data and
[00:10:52] you have overfit your training data and have making the parameters numerically
[00:10:54] have making the parameters numerically small is meant to lessen the extent to
[00:10:57] small is meant to lessen the extent to which you overfit on your training data
[00:10:59] which you overfit on your training data um this is not a picture um that modern
[00:11:04] um this is not a picture um that modern newal Network people believe at all
[00:11:08] newal Network people believe at all instead the picture has changed like
[00:11:10] instead the picture has changed like this um we don't believe that
[00:11:14] this um we don't believe that overfitting exists anymore but what we
[00:11:17] overfitting exists anymore but what we are concerned about is models that will
[00:11:22] are concerned about is models that will generalize well to different data um so
[00:11:26] generalize well to different data um so that when we train you know So In
[00:11:29] that when we train you know So In classical statistics the idea that you
[00:11:32] classical statistics the idea that you could train billions of parameters like
[00:11:36] could train billions of parameters like large neuron Nets now have um would be
[00:11:39] large neuron Nets now have um would be seen as ridiculous because you could not
[00:11:41] seen as ridiculous because you could not possibly estimate those parameters well
[00:11:44] possibly estimate those parameters well um and so you just have all of this
[00:11:47] um and so you just have all of this noisy mess um but what's actually been
[00:11:50] noisy mess um but what's actually been found is that yeah it's true you can't
[00:11:53] found is that yeah it's true you can't estimate the numbers well but what you
[00:11:55] estimate the numbers well but what you get is a kind of an interesting
[00:11:57] get is a kind of an interesting averaging function from all these Myriad
[00:12:00] averaging function from all these Myriad numbers and if you do it right what
[00:12:03] numbers and if you do it right what happens is as you go on
[00:12:07] happens is as you go on training that for a while it might look
[00:12:10] training that for a while it might look like you're starting to overfit but if
[00:12:12] like you're starting to overfit but if you keep on training in a huge Network
[00:12:15] you keep on training in a huge Network um not only will your training loss
[00:12:18] um not only will your training loss continue to go down very infinitesimally
[00:12:21] continue to go down very infinitesimally but your validation loss will go down as
[00:12:24] but your validation loss will go down as well and so that huge on huge networks
[00:12:28] well and so that huge on huge networks these days is um we train our models so
[00:12:33] these days is um we train our models so that they overfit to the training data
[00:12:36] that they overfit to the training data almost completely right so that if you
[00:12:38] almost completely right so that if you train a huge Network now on a training
[00:12:41] train a huge Network now on a training set you can essentially train them to
[00:12:44] set you can essentially train them to get zero loss you know maybe it's
[00:12:48] get zero loss you know maybe it's 0.007 loss or something but you can
[00:12:51] 0.007 loss or something but you can train them to get zero loss because
[00:12:52] train them to get zero loss because you've got such Rich models you can
[00:12:54] you've got such Rich models you can perfectly fit memorize the entire
[00:12:57] perfectly fit memorize the entire training set now classically that would
[00:13:00] training set now classically that would have been seen as a disaster because
[00:13:01] have been seen as a disaster because you've overfit the training data with
[00:13:04] you've overfit the training data with modern large neural networks it's not
[00:13:05] modern large neural networks it's not seen as a disaster because providing
[00:13:08] seen as a disaster because providing you've done regularization well that
[00:13:11] you've done regularization well that your model will also generalize well to
[00:13:14] your model will also generalize well to different
[00:13:15] different data however the flip side of that is
[00:13:19] data however the flip side of that is normally this kind of L2 regularization
[00:13:22] normally this kind of L2 regularization or similar ones like L1 regularization
[00:13:25] or similar ones like L1 regularization aren't strong enough regularization to
[00:13:27] aren't strong enough regularization to achieve that effect and so neural
[00:13:29] achieve that effect and so neural network people have turned to other
[00:13:32] network people have turned to other methods of regularization of which
[00:13:34] methods of regularization of which everyone's favorite is Dropout so this
[00:13:37] everyone's favorite is Dropout so this is one of the things that's on the
[00:13:39] is one of the things that's on the assignment and um at this point I should
[00:13:43] assignment and um at this point I should uh um apologize or something because the
[00:13:48] uh um apologize or something because the way the way Dropout is um done the way
[00:13:53] way the way Dropout is um done the way Dropout is presented here is sort of the
[00:13:55] Dropout is presented here is sort of the original formulation the way Dropout is
[00:13:57] original formulation the way Dropout is presented on the assignment is the way
[00:13:59] presented on the assignment is the way now normally done in deep learning
[00:14:01] now normally done in deep learning packages so um the there are a couple of
[00:14:04] packages so um the there are a couple of details that vary a bit and let me just
[00:14:07] details that vary a bit and let me just present the main idea here and um not
[00:14:10] present the main idea here and um not worry too much about the details of the
[00:14:12] worry too much about the details of the math so the idea of Dropout is at
[00:14:15] math so the idea of Dropout is at training time um every time you are
[00:14:20] training time um every time you are doing a piece of training with an
[00:14:22] doing a piece of training with an example what you're going to do is
[00:14:25] example what you're going to do is inside the middle layers of the neural
[00:14:27] inside the middle layers of the neural network you're just going to throw away
[00:14:30] network you're just going to throw away some of the inputs and so technically
[00:14:32] some of the inputs and so technically the way you do this is you have a random
[00:14:36] the way you do this is you have a random mask that you sample each time of zeros
[00:14:38] mask that you sample each time of zeros and ones you do a hadam mod product of
[00:14:41] and ones you do a hadam mod product of that with the data so some of the data
[00:14:44] that with the data so some of the data items go to zero and you have different
[00:14:48] items go to zero and you have different Mass each time so for the next um thing
[00:14:52] Mass each time so for the next um thing you know I've now masked out um
[00:14:54] you know I've now masked out um something different this time and so
[00:14:57] something different this time and so you're just sort of Random ly throwing
[00:14:59] you're just sort of Random ly throwing away the inputs and the effect of this
[00:15:03] away the inputs and the effect of this is that you're training the model that
[00:15:06] is that you're training the model that it has to be robust and work well and
[00:15:10] it has to be robust and work well and make as much use of every input as it
[00:15:13] make as much use of every input as it can it can't decide that can be
[00:15:15] can it can't decide that can be extremely reliant on you know component
[00:15:18] extremely reliant on you know component 17 of the vector because sometimes it's
[00:15:21] 17 of the vector because sometimes it's just going to randomly disappear so if
[00:15:23] just going to randomly disappear so if there are other features that you could
[00:15:25] there are other features that you could use instead that would let you work out
[00:15:27] use instead that would let you work out what to do next you should also know how
[00:15:30] what to do next you should also know how to make use of those features so at
[00:15:33] to make use of those features so at training time you randomly delete things
[00:15:36] training time you randomly delete things at test time sort of for efficiency but
[00:15:38] at test time sort of for efficiency but also quality of the answer um you don't
[00:15:41] also quality of the answer um you don't delete anything you keep all of your
[00:15:43] delete anything you keep all of your weights but you just rescale things to
[00:15:46] weights but you just rescale things to make up for the fact that you used to be
[00:15:48] make up for the fact that you used to be dropping
[00:15:49] dropping things okay um so what there are several
[00:15:52] things okay um so what there are several ways that you can think of explaining
[00:15:55] ways that you can think of explaining this one motivation that's often given
[00:15:58] this one motivation that's often given is is that this prevents feature co-
[00:16:01] is is that this prevents feature co- adaptation so rather than a model being
[00:16:05] adaptation so rather than a model being able to learn complex functions of
[00:16:07] able to learn complex functions of feature 7 8 and 11 can help me predict
[00:16:11] feature 7 8 and 11 can help me predict this it knows that some of the features
[00:16:13] this it knows that some of the features might be missing so it has to sort of
[00:16:16] might be missing so it has to sort of make use of things in a more flexible
[00:16:18] make use of things in a more flexible way another way of thinking of it is
[00:16:21] way another way of thinking of it is that there's been a lot of work on model
[00:16:23] that there's been a lot of work on model ensembles where you can sort of mix
[00:16:26] ensembles where you can sort of mix together um different models and improve
[00:16:28] together um different models and improve your your results if you're training
[00:16:30] your your results if you're training with Dropout it's kind of like you're
[00:16:33] with Dropout it's kind of like you're training with a huge model Ensemble
[00:16:35] training with a huge model Ensemble because you're training with the
[00:16:37] because you're training with the Ensemble of the power set the
[00:16:39] Ensemble of the power set the exponential number of every possible
[00:16:42] exponential number of every possible Dropout of features all at once and that
[00:16:45] Dropout of features all at once and that gives you a a very good model um so
[00:16:48] gives you a a very good model um so there are different ways of thinking
[00:16:50] there are different ways of thinking about it I mean if you've seen na bays
[00:16:53] about it I mean if you've seen na bays and logistic regression models before
[00:16:56] and logistic regression models before you know I kind of think a nice way to
[00:16:58] you know I kind of think a nice way to think of it is that it gives a sort of a
[00:17:00] think of it is that it gives a sort of a middle ground between the two because
[00:17:02] middle ground between the two because for naive based models you're waiting
[00:17:04] for naive based models you're waiting each feature independently just based on
[00:17:07] each feature independently just based on the data statistics doesn't matter what
[00:17:09] the data statistics doesn't matter what other features are there in a logistic
[00:17:11] other features are there in a logistic regression weights are set in the
[00:17:13] regression weights are set in the context of all the other features and
[00:17:17] context of all the other features and with Dropout you're somewhere in between
[00:17:19] with Dropout you're somewhere in between you're seeing the weights in the context
[00:17:20] you're seeing the weights in the context of some of the other features but
[00:17:22] of some of the other features but different ones will disappear at
[00:17:23] different ones will disappear at different times um but you know
[00:17:26] different times um but you know following work that was done um at
[00:17:29] following work that was done um at Stanford by Stefan varer and others that
[00:17:32] Stanford by Stefan varer and others that generally these PE days people regard
[00:17:35] generally these PE days people regard Dropout as a form of feature dependent
[00:17:38] Dropout as a form of feature dependent regularization and he shows some
[00:17:40] regularization and he shows some theoretical results as to why to think
[00:17:42] theoretical results as to why to think of it that
[00:17:43] of it that way okay I think we've implicitly seen
[00:17:46] way okay I think we've implicitly seen this one um but vectorization is the
[00:17:51] this one um but vectorization is the idea no for Loops always use vectors
[00:17:55] idea no for Loops always use vectors matrices and tensors right the entire
[00:17:59] matrices and tensors right the entire success and speed of deep learning works
[00:18:03] success and speed of deep learning works from the fact that we can do things with
[00:18:05] from the fact that we can do things with vectors matrices and tensors so you know
[00:18:08] vectors matrices and tensors so you know if you're writing for Loops in any
[00:18:11] if you're writing for Loops in any language but especially in Python things
[00:18:13] language but especially in Python things run really slowly if you can do things
[00:18:17] run really slowly if you can do things with vectors and matrices even on CPU
[00:18:20] with vectors and matrices even on CPU things run at least an order of
[00:18:22] things run at least an order of magnitude faster and well what everyone
[00:18:25] magnitude faster and well what everyone really wants to move to doing in deep
[00:18:27] really wants to move to doing in deep learning is running things on gpus or
[00:18:31] learning is running things on gpus or sometimes now newal processing units and
[00:18:33] sometimes now newal processing units and then you're getting you know two three
[00:18:35] then you're getting you know two three orders of magnitude of speed up so um do
[00:18:38] orders of magnitude of speed up so um do always think about I should be doing
[00:18:40] always think about I should be doing things with vectors and matrices if I'm
[00:18:44] things with vectors and matrices if I'm writing a for Loop for anything that
[00:18:46] writing a for Loop for anything that isn't some very superficial bit of input
[00:18:48] isn't some very superficial bit of input processing I've almost certainly made a
[00:18:50] processing I've almost certainly made a mistake and I should be working out um
[00:18:53] mistake and I should be working out um how to do things um with vectors and
[00:18:56] how to do things um with vectors and metries and you know that's kind of
[00:18:58] metries and you know that's kind of thing like Dropout you don't want to
[00:19:00] thing like Dropout you don't want to write a for Loop um that goes through
[00:19:02] write a for Loop um that goes through all the positions and sets some of them
[00:19:04] all the positions and sets some of them to zero you want to be sort of using a
[00:19:07] to zero you want to be sort of using a vector operation with your
[00:19:09] vector operation with your mask um two more I think um parameter
[00:19:14] mask um two more I think um parameter initialization I mean this one might not
[00:19:16] initialization I mean this one might not be obvious but when we start training
[00:19:21] be obvious but when we start training our neural networks in almost all cases
[00:19:26] our neural networks in almost all cases it's vital that we
[00:19:29] it's vital that we initialize the parameters of our
[00:19:32] initialize the parameters of our matrices to some random numbers and the
[00:19:36] matrices to some random numbers and the reason for this is if we just start with
[00:19:39] reason for this is if we just start with the um if we just start with our
[00:19:42] the um if we just start with our matrices all zero or some other constant
[00:19:46] matrices all zero or some other constant normally the case is that we have um
[00:19:51] normally the case is that we have um symmetry so it's sort of like in this
[00:19:54] symmetry so it's sort of like in this picture when you're starting on this
[00:19:56] picture when you're starting on this Saddle Point um that you know it's
[00:19:58] Saddle Point um that you know it's symmetric to the left and the right and
[00:20:01] symmetric to the left and the right and um or whatever forward and backwards and
[00:20:03] um or whatever forward and backwards and left and right and so is you sort of
[00:20:06] left and right and so is you sort of don't know where to go and you might be
[00:20:09] don't know where to go and you might be sort of stuck and stay in the one place
[00:20:11] sort of stuck and stay in the one place I mean normally a way to think about is
[00:20:15] I mean normally a way to think about is the operations that you're doing to all
[00:20:17] the operations that you're doing to all the elements in The Matrix are sort of
[00:20:19] the elements in The Matrix are sort of the same so rather than having you know
[00:20:22] the same so rather than having you know a whole Vector of features if all of
[00:20:25] a whole Vector of features if all of them have the same value initially often
[00:20:27] them have the same value initially often it's sort of like you only have one
[00:20:28] it's sort of like you only have one feature and you've just got a lot of
[00:20:30] feature and you've just got a lot of copies of it so to initialize learning
[00:20:34] copies of it so to initialize learning and have things work well we almost
[00:20:37] and have things work well we almost always want to set all the weights to
[00:20:39] always want to set all the weights to very small random numbers um and so at
[00:20:45] very small random numbers um and so at that point you know CL when I say very
[00:20:49] that point you know CL when I say very small we sort of want to make them an A
[00:20:51] small we sort of want to make them an A Range so that they don't disappear to
[00:20:53] Range so that they don't disappear to zero if we make them a bit smaller and
[00:20:57] zero if we make them a bit smaller and um they don't sort of start blowing up
[00:20:59] um they don't sort of start blowing up into huge numbers when we multipli them
[00:21:01] into huge numbers when we multipli them by things and doing this initialization
[00:21:04] by things and doing this initialization at the right scale was used to be seen
[00:21:07] at the right scale was used to be seen as something pretty important and there
[00:21:09] as something pretty important and there were particular methods that had a basis
[00:21:11] were particular methods that had a basis of sort of thinking of what happens once
[00:21:13] of sort of thinking of what happens once you do Matrix multiplies that people had
[00:21:16] you do Matrix multiplies that people had worked out and often used one of these
[00:21:18] worked out and often used one of these was this harier
[00:21:20] was this harier initialization which was sort of working
[00:21:22] initialization which was sort of working out um what variance of your uniform
[00:21:25] out um what variance of your uniform distribution to be um variant of your
[00:21:29] distribution to be um variant of your distribution to be using based on the
[00:21:31] distribution to be using based on the sort of number of inputs and outputs of
[00:21:34] sort of number of inputs and outputs of a layer and things like that the
[00:21:36] a layer and things like that the specifics of that um you know I think we
[00:21:39] specifics of that um you know I think we still use to initialize things in
[00:21:41] still use to initialize things in assignment two but we'll see later that
[00:21:44] assignment two but we'll see later that they go away because people have come up
[00:21:46] they go away because people have come up with clever methods um in particular
[00:21:49] with clever methods um in particular doing layer normalization which sort of
[00:21:51] doing layer normalization which sort of obviates the need to be so careful on
[00:21:53] obviates the need to be so careful on the initialization but you still need to
[00:21:55] the initialization but you still need to initialize things to something
[00:21:59] initialize things to something okay then the fast the final one which
[00:22:02] okay then the fast the final one which I'll is also something that appears in
[00:22:05] I'll is also something that appears in the second assignment that I just want
[00:22:06] the second assignment that I just want to say a word about um was optimizers so
[00:22:10] to say a word about um was optimizers so we talked about in class um stochastic
[00:22:14] we talked about in class um stochastic gradient descent and did the basic um
[00:22:16] gradient descent and did the basic um equations for stochastic gradient
[00:22:18] equations for stochastic gradient descent and you know to a first
[00:22:21] descent and you know to a first approximation there's nothing wrong with
[00:22:23] approximation there's nothing wrong with stochastic gradient descent and if you
[00:22:26] stochastic gradient descent and if you fiddle around enough you can usually get
[00:22:28] fiddle around enough you can usually get stochastic gradients send actually to
[00:22:29] stochastic gradients send actually to work well for almost any problem but um
[00:22:33] work well for almost any problem but um getting it to work well is very
[00:22:36] getting it to work well is very dependent on getting the scales of
[00:22:38] dependent on getting the scales of things right of sort of having the right
[00:22:40] things right of sort of having the right step size and often you have to have a
[00:22:42] step size and often you have to have a learning rate schedule with decreasing
[00:22:44] learning rate schedule with decreasing step sizes and various other
[00:22:46] step sizes and various other complications so people have come up
[00:22:49] complications so people have come up with um more sophisticated optimizers
[00:22:53] with um more sophisticated optimizers for newal networks and for complex Nets
[00:22:57] for newal networks and for complex Nets sometimes these seem kind of necessary
[00:22:59] sometimes these seem kind of necessary to get them to learn well and at any
[00:23:02] to get them to learn well and at any rate they give you sort of lots of
[00:23:04] rate they give you sort of lots of margins of safety since they're much
[00:23:06] margins of safety since they're much less dependent on you setting different
[00:23:09] less dependent on you setting different hyperparameters right and the idea of
[00:23:12] hyperparameters right and the idea of all of the well all the methods I
[00:23:14] all of the well all the methods I mentioned and the most commonly used
[00:23:16] mentioned and the most commonly used methods is that for each parameter
[00:23:20] methods is that for each parameter they're accumulating a measure of what
[00:23:24] they're accumulating a measure of what the gradient has been in the past and
[00:23:26] the gradient has been in the past and they've got some idea of the scale of
[00:23:28] they've got some idea of the scale of the gradient the slope for a particular
[00:23:31] the gradient the slope for a particular parameter and then they're using that to
[00:23:34] parameter and then they're using that to decide how much you move the learning
[00:23:36] decide how much you move the learning rate at each time step so the simplest
[00:23:39] rate at each time step so the simplest method that was come up with this one
[00:23:41] method that was come up with this one called adrad if you know John duci and E
[00:23:44] called adrad if you know John duci and E he was one of the co-inventors of this
[00:23:47] he was one of the co-inventors of this um you know it's simple and nice enough
[00:23:49] um you know it's simple and nice enough but it tends to stall early then people
[00:23:52] but it tends to stall early then people came up with different methods Adam's
[00:23:54] came up with different methods Adam's the one that's on assignment two it's a
[00:23:56] the one that's on assignment two it's a really good safe place to start art um
[00:23:59] really good safe place to start art um but in a way um sort of our word vectors
[00:24:03] but in a way um sort of our word vectors have a special property because of their
[00:24:05] have a special property because of their spareness that you know you're very
[00:24:07] spareness that you know you're very sparely updating them because particular
[00:24:10] sparely updating them because particular words only turn up occasionally so
[00:24:12] words only turn up occasionally so people have actually come up with
[00:24:14] people have actually come up with particular um optimizers that sort of
[00:24:17] particular um optimizers that sort of have special properties for things like
[00:24:19] have special properties for things like word vectors and so these ones with the
[00:24:22] word vectors and so these ones with the w at the end can sometimes be good to
[00:24:24] w at the end can sometimes be good to try and then you know again there's a
[00:24:26] try and then you know again there's a whole family of extra ideas that people
[00:24:30] whole family of extra ideas that people have used to improve optimizers and if
[00:24:32] have used to improve optimizers and if you want to learn about that you can go
[00:24:33] you want to learn about that you can go off and do an optimization class like
[00:24:36] off and do an optimization class like Conex optimization but there ideas like
[00:24:39] Conex optimization but there ideas like momentum and nesterov acceleration and
[00:24:41] momentum and nesterov acceleration and things like that and all of those things
[00:24:43] things like that and all of those things people also variously try to use um but
[00:24:46] people also variously try to use um but Adam is a good name to remember um if
[00:24:48] Adam is a good name to remember um if you remember nothing
[00:24:50] you remember nothing else okay that took longer than I hoped
[00:24:53] else okay that took longer than I hoped but I'll get on now to language models
[00:24:56] but I'll get on now to language models okay language models so you know um in
[00:25:01] okay language models so you know um in some sense language model is just two
[00:25:04] some sense language model is just two English words but when in NLP we say
[00:25:07] English words but when in NLP we say language models we mean it as a
[00:25:09] language models we mean it as a technical term that has a particular
[00:25:12] technical term that has a particular meaning um so the idea of a language
[00:25:16] meaning um so the idea of a language model is something that can
[00:25:19] model is something that can predict well what word is going to come
[00:25:22] predict well what word is going to come next or more precisely it's going to put
[00:25:25] next or more precisely it's going to put a probability distribution over what
[00:25:27] a probability distribution over what words come next so the students open
[00:25:30] words come next so the students open there what words are likely to come
[00:25:35] next
[00:25:37] next bags laptops laptops notbook notebooks
[00:25:41] bags laptops laptops notbook notebooks notebooks yeah um I have some of those
[00:25:44] notebooks yeah um I have some of those at least okay um yeah I mean so right so
[00:25:50] at least okay um yeah I mean so right so so these are kind of likely words and if
[00:25:53] so these are kind of likely words and if on top of those we put a probability on
[00:25:57] on top of those we put a probability on each one then we have a language model
[00:26:00] each one then we have a language model so formally we've got a context of
[00:26:04] so formally we've got a context of proceeding items we're putting a
[00:26:06] proceeding items we're putting a probability distribution over the next
[00:26:09] probability distribution over the next item which means that the sum of the
[00:26:12] item which means that the sum of the estimates of this for the um items in
[00:26:14] estimates of this for the um items in the vocabulary will sum to one and if
[00:26:18] the vocabulary will sum to one and if we've defined a p like this that
[00:26:20] we've defined a p like this that predicts probabilities of next words
[00:26:23] predicts probabilities of next words that is called a language
[00:26:26] that is called a language model as it says here um an alternative
[00:26:29] model as it says here um an alternative way um that you can think of a language
[00:26:32] way um that you can think of a language model is that a language model is a
[00:26:35] model is that a language model is a system that assigns a probability to a
[00:26:38] system that assigns a probability to a piece of text and um so we can say that
[00:26:43] piece of text and um so we can say that a language model can can take any piece
[00:26:45] a language model can can take any piece of text and give it a probability and
[00:26:48] of text and give it a probability and the reason we can do that is we can use
[00:26:50] the reason we can do that is we can use the chain rule so I want to know the
[00:26:53] the chain rule so I want to know the probability of any stretch of text I say
[00:26:57] probability of any stretch of text I say given my previous definition of language
[00:26:59] given my previous definition of language model easy I can do that probability of
[00:27:02] model easy I can do that probability of X1 with a null preceding context times
[00:27:06] X1 with a null preceding context times the probability of X2 given X1 Etc along
[00:27:10] the probability of X2 given X1 Etc along I can do this chain rule decomposition
[00:27:13] I can do this chain rule decomposition and then the terms of that decomposition
[00:27:16] and then the terms of that decomposition are precisely what the language model as
[00:27:19] are precisely what the language model as I defined it previously provides okay so
[00:27:22] I defined it previously provides okay so language models are this essential
[00:27:25] language models are this essential technology for NLP just just about
[00:27:29] technology for NLP just just about everything from the simplest places
[00:27:31] everything from the simplest places forward um where people do things with
[00:27:34] forward um where people do things with human language and computers people use
[00:27:37] human language and computers people use language models in particular you know
[00:27:40] language models in particular you know they weren't something that got invented
[00:27:42] they weren't something that got invented in 2022 with chat GPT language models
[00:27:46] in 2022 with chat GPT language models have been Central to NLP at least since
[00:27:50] have been Central to NLP at least since the 80s the idea of them goes back to at
[00:27:52] the 80s the idea of them goes back to at least the 50s um so anytime you're
[00:27:56] least the 50s um so anytime you're typing on your phone and it's making
[00:27:59] typing on your phone and it's making suggestions of next words regardless of
[00:28:01] suggestions of next words regardless of whether you like those suggestions or
[00:28:03] whether you like those suggestions or not um those suggestions are being
[00:28:06] not um those suggestions are being generated by a language Model A Very uh
[00:28:09] generated by a language Model A Very uh traditionally a compact not very good
[00:28:12] traditionally a compact not very good language model so it can run sort of
[00:28:15] language model so it can run sort of quickly and very little memory in your
[00:28:17] quickly and very little memory in your keyboard application um if you go on
[00:28:20] keyboard application um if you go on Google and you start typing some stuff
[00:28:23] Google and you start typing some stuff and it's telling you stuff that could
[00:28:25] and it's telling you stuff that could come after it um to complete your query
[00:28:28] come after it um to complete your query well again that's being generated by a
[00:28:31] well again that's being generated by a language model so how can you build a
[00:28:34] language model so how can you build a language model so before getting into
[00:28:36] language model so before getting into new language models I've got just a few
[00:28:39] new language models I've got just a few slides to tell you about the old days of
[00:28:42] slides to tell you about the old days of language modeling so this is sort of how
[00:28:45] language modeling so this is sort of how language models were built um from
[00:28:49] language models were built um from 1975 until you know effectively around
[00:28:53] 1975 until you know effectively around about
[00:28:55] about 2012 um so
[00:28:58] 2012 um so we want to put probabilities on these
[00:29:01] we want to put probabilities on these sequences um and the way we're going to
[00:29:05] sequences um and the way we're going to do it is we're going to build what's
[00:29:07] do it is we're going to build what's called an engram language model um and
[00:29:11] called an engram language model um and so this is meaning we're going to look
[00:29:12] so this is meaning we're going to look at Short word
[00:29:14] at Short word subsequences and use them to predict so
[00:29:17] subsequences and use them to predict so N is a variable describing how short are
[00:29:21] N is a variable describing how short are the word sequences that we're going to
[00:29:23] the word sequences that we're going to use to predict um so if we just look at
[00:29:26] use to predict um so if we just look at the probabilities of individual words we
[00:29:29] the probabilities of individual words we have a unigram language model if we look
[00:29:32] have a unigram language model if we look at probabilities of pairs of words Byram
[00:29:35] at probabilities of pairs of words Byram language model um uh probabilities of
[00:29:38] language model um uh probabilities of three words trigram language models
[00:29:40] three words trigram language models probabilities of more than three words
[00:29:43] probabilities of more than three words they get called four gr language models
[00:29:45] they get called four gr language models five gr language models six gr language
[00:29:48] five gr language models six gr language models um so for people with a Classics
[00:29:50] models um so for people with a Classics education this is horrific of course um
[00:29:54] education this is horrific of course um in particular not even these ones are
[00:29:56] in particular not even these ones are correct um because Graham is a Greek
[00:30:00] correct um because Graham is a Greek root so it should really have Greek
[00:30:02] root so it should really have Greek numbers in front here um so you should
[00:30:04] numbers in front here um so you should have Monograms and diagrams um and you
[00:30:08] have Monograms and diagrams um and you know actually so the first person who
[00:30:09] know actually so the first person who introduced the idea of engram models was
[00:30:11] introduced the idea of engram models was actually Claude Shannon when he was
[00:30:13] actually Claude Shannon when he was working out information Theory the same
[00:30:16] working out information Theory the same guy that did cross entropy and all of
[00:30:18] guy that did cross entropy and all of that and if you look at his 1951 paper
[00:30:21] that and if you look at his 1951 paper he uses diagrams um but the idea died
[00:30:24] he uses diagrams um but the idea died about there and everyone else this is
[00:30:27] about there and everyone else this is what people say in practice um It's Kind
[00:30:30] what people say in practice um It's Kind kind of cute I like it a nice you know
[00:30:33] kind of cute I like it a nice you know practical notation um so to build these
[00:30:36] practical notation um so to build these models the idea is look we're just going
[00:30:39] models the idea is look we're just going to count how often different engrs
[00:30:43] to count how often different engrs appear in text and use those to build
[00:30:46] appear in text and use those to build our probability estimates um and in
[00:30:50] our probability estimates um and in particular our trick is that we make a
[00:30:53] particular our trick is that we make a mark of assumption so that if we're
[00:30:56] mark of assumption so that if we're predicting the next word based on a long
[00:30:59] predicting the next word based on a long context we say ah tell you what we're
[00:31:02] context we say ah tell you what we're not going to use all of it we're only
[00:31:04] not going to use all of it we're only going to use the most recent n minus one
[00:31:08] going to use the most recent n minus one words so we have this big context and we
[00:31:11] words so we have this big context and we throw most of it away and so if we're
[00:31:14] throw most of it away and so if we're predicting word XT + one based on simply
[00:31:18] predicting word XT + one based on simply the preceding n minus one words well
[00:31:21] the preceding n minus one words well then we can make the
[00:31:22] then we can make the prediction using
[00:31:25] prediction using NRS um for
[00:31:28] NRS um for let's whatever it is if we use n is
[00:31:32] let's whatever it is if we use n is three would' have a triam here and
[00:31:36] three would' have a triam here and normalized by a Bagram down here and
[00:31:39] normalized by a Bagram down here and that that would give us relative um
[00:31:42] that that would give us relative um frequencies of the different
[00:31:45] frequencies of the different terms um so we can do that simply by
[00:31:49] terms um so we can do that simply by counting how often NRS occur in a large
[00:31:55] counting how often NRS occur in a large amount of text and simply divid through
[00:31:58] amount of text and simply divid through by the counts and that gives us a
[00:32:00] by the counts and that gives us a relative frequency estimate of the
[00:32:03] relative frequency estimate of the probability of different
[00:32:05] probability of different continuations does that make sense yeah
[00:32:08] continuations does that make sense yeah that's a way to do it okay um so suppose
[00:32:12] that's a way to do it okay um so suppose we're um learning a a forr language
[00:32:15] we're um learning a a forr language model right and we've got a piece of
[00:32:17] model right and we've got a piece of text as the Proctor started the clock
[00:32:20] text as the Proctor started the clock the students open there so well to
[00:32:23] the students open there so well to estimate things we are going to throw
[00:32:27] estimate things we are going to throw away all but the preceding three words
[00:32:30] away all but the preceding three words so we're going to estimate based on
[00:32:32] so we're going to estimate based on students open there and so we're going
[00:32:34] students open there and so we're going to work out the probabilities by looking
[00:32:37] to work out the probabilities by looking for counts of students open their W and
[00:32:41] for counts of students open their W and counts of students open there um so we
[00:32:45] counts of students open there um so we might have in a corpus that students
[00:32:47] might have in a corpus that students open there occurred a thousand times
[00:32:49] open there occurred a thousand times students open their books occurred 400
[00:32:52] students open their books occurred 400 times and so we'd say the probability
[00:32:54] times and so we'd say the probability estimate is simply 0.4 for B
[00:32:58] estimate is simply 0.4 for B if exams occurred 100 times the
[00:33:01] if exams occurred 100 times the probability estimate is
[00:33:03] probability estimate is 0.1 for
[00:33:05] 0.1 for exams
[00:33:08] exams um and well you can sort of see that
[00:33:11] um and well you can sort of see that this is bad it's not terrible because if
[00:33:14] this is bad it's not terrible because if you are going to try and predict the
[00:33:15] you are going to try and predict the next word in a simple way looking at the
[00:33:18] next word in a simple way looking at the immediately prior words is are the most
[00:33:21] immediately prior words is are the most helpful words to look at but it's it's
[00:33:23] helpful words to look at but it's it's clearly sort of primitive because you
[00:33:26] clearly sort of primitive because you know if you known the prior text was as
[00:33:29] know if you known the prior text was as the Proctor started the clock that makes
[00:33:31] the Proctor started the clock that makes it sound likely that the words should
[00:33:32] it sound likely that the words should have been exams where since you're
[00:33:35] have been exams where since you're estimating just based on students open
[00:33:37] estimating just based on students open theirs well you'd be more likely to
[00:33:40] theirs well you'd be more likely to choose books because it's more common so
[00:33:43] choose books because it's more common so it's a kind of a crude estimate but it's
[00:33:46] it's a kind of a crude estimate but it's a decent enough place to start um it's a
[00:33:50] a decent enough place to start um it's a crude estimate that could be problematic
[00:33:52] crude estimate that could be problematic in other ways I mean why why else might
[00:33:56] in other ways I mean why why else might we kind of get in
[00:33:58] we kind of get in Troubles by using this our probability
[00:34:02] estimate
[00:34:05] estimate yeah so there are a lot of engrams yeah
[00:34:08] yeah so there are a lot of engrams yeah so there are a lot of words and
[00:34:10] so there are a lot of words and therefore there are a lot of lot of
[00:34:12] therefore there are a lot of lot of engrams yeah so that's a problem we'll
[00:34:14] engrams yeah so that's a problem we'll come to it later uh anything else maybe
[00:34:17] come to it later uh anything else maybe up the back um like the word w might not
[00:34:19] up the back um like the word w might not even show up in the training data so you
[00:34:21] even show up in the training data so you might just have a count zero for that
[00:34:23] might just have a count zero for that yeah yeah so um so if we're counting
[00:34:28] yeah yeah so um so if we're counting over any reasonable size Corpus there
[00:34:31] over any reasonable size Corpus there are lots of words that we just are not
[00:34:35] are lots of words that we just are not going to have seen right that they never
[00:34:38] going to have seen right that they never happen to occur in the text that we
[00:34:40] happen to occur in the text that we counted over you know so if you start
[00:34:44] counted over you know so if you start thinking students open there you know
[00:34:46] thinking students open there you know there are lots of things that you could
[00:34:48] there are lots of things that you could put there you know students open their
[00:34:50] put there you know students open their accounts or if the students are doing
[00:34:53] accounts or if the students are doing dissections in a biology class maybe
[00:34:56] dissections in a biology class maybe students open their frogs I don't know
[00:34:58] students open their frogs I don't know um you know that there are lots of words
[00:35:01] um you know that there are lots of words that in some context you know would
[00:35:03] that in some context you know would actually be possible and lots of them
[00:35:06] actually be possible and lots of them that we won't have seen and so it give
[00:35:08] that we won't have seen and so it give them a probability estimate of zero and
[00:35:11] them a probability estimate of zero and that tends to be an especially bad thing
[00:35:13] that tends to be an especially bad thing to do with probabilities because once we
[00:35:15] to do with probabilities because once we have a probability estimate of zero any
[00:35:17] have a probability estimate of zero any computations that we do that involve
[00:35:19] computations that we do that involve that will instantly go to zero so we
[00:35:22] that will instantly go to zero so we have to deal with some of these problems
[00:35:24] have to deal with some of these problems so for that sparity problem right yeah
[00:35:28] so for that sparity problem right yeah that we could have the word never
[00:35:31] that we could have the word never occurred in the numerator and so simply
[00:35:35] occurred in the numerator and so simply done we get a probability estimate of
[00:35:38] done we get a probability estimate of zero the way that was dealt with was
[00:35:41] zero the way that was dealt with was that people just hacked the counts a
[00:35:43] that people just hacked the counts a little to make it non zero so there are
[00:35:45] little to make it non zero so there are lots of ways that are explored but the
[00:35:47] lots of ways that are explored but the easiest way is you just sort of added a
[00:35:50] easiest way is you just sort of added a little Delta like you know 0.25 to
[00:35:54] little Delta like you know 0.25 to counts so things that you never saw got
[00:35:56] counts so things that you never saw got a count of 0 .25 in total and things you
[00:36:00] a count of 0 .25 in total and things you saw once got to count of 1.25 and then
[00:36:03] saw once got to count of 1.25 and then there are no zeros anymore everything is
[00:36:05] there are no zeros anymore everything is possible um you could think then there's
[00:36:08] possible um you could think then there's a second problem that wait you might
[00:36:11] a second problem that wait you might never have seen stupid students open
[00:36:13] never have seen stupid students open there before and so that means your
[00:36:16] there before and so that means your denominator is just undefined and you
[00:36:20] denominator is just undefined and you don't have any counts in the numerator
[00:36:22] don't have any counts in the numerator either so you sort of need to do
[00:36:24] either so you sort of need to do something different there and the
[00:36:26] something different there and the standard trick was used then was that
[00:36:29] standard trick was used then was that you um did back off so if you couldn't
[00:36:32] you um did back off so if you couldn't estimate words coming after students
[00:36:35] estimate words coming after students open there you just worked out the
[00:36:37] open there you just worked out the estimates for come words coming after
[00:36:40] estimates for come words coming after open there and if you couldn't estimate
[00:36:43] open there and if you couldn't estimate that you just use the estimate of words
[00:36:45] that you just use the estimate of words coming after there so you used less and
[00:36:48] coming after there so you used less and less context until you could get an
[00:36:50] less context until you could get an estimate that you could
[00:36:52] estimate that you could use um but you know something to note is
[00:36:55] use um but you know something to note is that we''ve got these conflicting
[00:36:57] that we''ve got these conflicting pressures now so that on the one hand
[00:37:00] pressures now so that on the one hand you know if you want to come up with a
[00:37:02] you know if you want to come up with a better estimate that you would like to
[00:37:05] better estimate that you would like to use more context I to have a larger engr
[00:37:09] use more context I to have a larger engr but on the other hand as you make use
[00:37:14] but on the other hand as you make use more more conditioning words well the
[00:37:19] more more conditioning words well the storage size problem someone mentioned
[00:37:22] storage size problem someone mentioned gets worse and worse because the number
[00:37:24] gets worse and worse because the number of NRS that you have to know about is
[00:37:26] of NRS that you have to know about is going up exponen eventally with the size
[00:37:28] going up exponen eventally with the size of the context but also your spareness
[00:37:32] of the context but also your spareness problems are getting way way worse and
[00:37:34] problems are getting way way worse and you're almost necessarily going to be
[00:37:36] you're almost necessarily going to be ending up seeing zeros and so because of
[00:37:39] ending up seeing zeros and so because of that you know in practice where things
[00:37:42] that you know in practice where things tended to um sort of max out was five
[00:37:46] tended to um sort of max out was five and occasionally people use six gr and
[00:37:49] and occasionally people use six gr and seven G but most of the time you know
[00:37:52] seven G but most of the time you know between the sort of spareness and the
[00:37:54] between the sort of spareness and the cost of storage 5 G was the large thing
[00:37:57] cost of storage 5 G was the large thing people dealt with um so
[00:38:00] people dealt with um so um a famous resource from back in the
[00:38:04] um a famous resource from back in the 2000s decade that Google released um was
[00:38:07] 2000s decade that Google released um was Google engrs which was built on a a
[00:38:10] Google engrs which was built on a a trillion word web Corpus and had counts
[00:38:13] trillion word web Corpus and had counts of n g and it gave counts of n g up to
[00:38:17] of n g and it gave counts of n g up to nals 5 and that is where they
[00:38:20] nals 5 and that is where they stopped okay well we've sort of said the
[00:38:23] stopped okay well we've sort of said the storage problem the storage problem is
[00:38:25] storage problem the storage problem is well to do this you need to store the
[00:38:27] well to do this you need to store the these counts the number of counts is
[00:38:30] these counts the number of counts is going up exponentially in the amount of
[00:38:32] going up exponentially in the amount of Contex size um okay um but you know
[00:38:37] Contex size um okay um but you know what's good about engram language models
[00:38:39] what's good about engram language models they're really easy to build you can um
[00:38:43] they're really easy to build you can um build one yourself in a few minutes when
[00:38:45] build one yourself in a few minutes when you've got want to have a bit of fun on
[00:38:47] you've got want to have a bit of fun on the weekend um you know all you have to
[00:38:50] the weekend um you know all you have to do is start sort of storing um these
[00:38:53] do is start sort of storing um these counts for engrams and you can use them
[00:38:56] counts for engrams and you can use them to predict things so you know for if at
[00:38:58] to predict things so you know for if at least if you do it over a small Corpus
[00:39:01] least if you do it over a small Corpus like a couple of million words of text
[00:39:04] like a couple of million words of text um you know you can build an engram
[00:39:06] um you know you can build an engram language model in seconds on your laptop
[00:39:08] language model in seconds on your laptop or you have to buil write the software
[00:39:11] or you have to buil write the software okay a few minutes to write the software
[00:39:13] okay a few minutes to write the software but building the model takes seconds
[00:39:15] but building the model takes seconds because you know there's no training in
[00:39:17] because you know there's no training in your network all you do is count how
[00:39:18] your network all you do is count how often um engrams occur and so once
[00:39:23] often um engrams occur and so once you've done that you can then run an
[00:39:25] you've done that you can then run an engram language model to generate text
[00:39:28] engram language model to generate text you know we could do text generation
[00:39:30] you know we could do text generation before chat GPT right so if I have a
[00:39:33] before chat GPT right so if I have a trigram language model I can start off
[00:39:36] trigram language model I can start off with some words today the and I could
[00:39:39] with some words today the and I could look at my stored engrams and get a
[00:39:43] look at my stored engrams and get a probability distribution over next words
[00:39:46] probability distribution over next words and here they are you know note the
[00:39:49] and here they are you know note the strong patterning of these um these um
[00:39:55] strong patterning of these um these um probabilities cuz remember they're all
[00:39:57] probabilities cuz remember they're all der from counts right that are being
[00:39:59] der from counts right that are being normalized so really these are words
[00:40:02] normalized so really these are words that occurred once these are words that
[00:40:04] that occurred once these are words that occurred twice these are words that
[00:40:06] occurred twice these are words that occurred four times in this context
[00:40:08] occurred four times in this context right so they're sort of in some sense
[00:40:10] right so they're sort of in some sense crude when you look at them more
[00:40:12] crude when you look at them more carefully but so what we could do is
[00:40:14] carefully but so what we could do is then at this point you know we roll a a
[00:40:18] then at this point you know we roll a a die and get a random number from 0er to
[00:40:21] die and get a random number from 0er to one and we can use that sample from this
[00:40:24] one and we can use that sample from this distribution
[00:40:26] distribution um sorry yeah
[00:40:29] um sorry yeah um so we sample from this distribution
[00:40:33] um so we sample from this distribution and so that if we sort of um generate so
[00:40:37] and so that if we sort of um generate so of as our random number something
[00:40:40] of as our random number something like
[00:40:42] like 35 if we go down from the top we'd say
[00:40:46] 35 if we go down from the top we'd say okay we've sampled the word price today
[00:40:49] okay we've sampled the word price today the price and then we repeat over we
[00:40:51] the price and then we repeat over we condition on that we probability
[00:40:53] condition on that we probability distribution of the next word um we
[00:40:56] distribution of the next word um we generate a random numbered and use it to
[00:40:58] generate a random numbered and use it to sample from the distribution um we say
[00:41:01] sample from the distribution um we say generate 2 and so we choose of um we now
[00:41:05] generate 2 and so we choose of um we now condition on that we get a probability
[00:41:08] condition on that we get a probability distribution we generate a random number
[00:41:11] distribution we generate a random number which is 0.5 or something and so we get
[00:41:14] which is 0.5 or something and so we get gold coming out and we can say today the
[00:41:17] gold coming out and we can say today the price of gold and we can keep on doing
[00:41:19] price of gold and we can keep on doing this and generate some text and so
[00:41:22] this and generate some text and so here's some text generated um from 2
[00:41:26] here's some text generated um from 2 million words training data using a
[00:41:29] million words training data using a trigram language model today the price
[00:41:32] trigram language model today the price of gold per ton while production of shoe
[00:41:35] of gold per ton while production of shoe lasts and shoe industry the bank
[00:41:38] lasts and shoe industry the bank intervened just after it considered and
[00:41:39] intervened just after it considered and rejected an IMF demand to rebuild
[00:41:42] rejected an IMF demand to rebuild depleted European stocks September 3rd
[00:41:45] depleted European stocks September 3rd in primary 76 cents a share um
[00:41:50] in primary 76 cents a share um now okay that text isn't great um but
[00:41:54] now okay that text isn't great um but you know I actually want people to you
[00:41:56] you know I actually want people to you know be in a positive of mood today um
[00:41:59] know be in a positive of mood today um and you know actually it's not so bad
[00:42:03] and you know actually it's not so bad right it's sort of surprisingly
[00:42:06] right it's sort of surprisingly grammatical I mean in particular like I
[00:42:09] grammatical I mean in particular like I lowercased everything so this is the IMF
[00:42:11] lowercased everything so this is the IMF that should be capitalized of the
[00:42:13] that should be capitalized of the international monetary fund right you
[00:42:16] international monetary fund right you know there are big pieces of this that
[00:42:18] know there are big pieces of this that even make sense right the bank
[00:42:20] even make sense right the bank intervened just after it considered and
[00:42:22] intervened just after it considered and rejected an if IMF demand you know
[00:42:25] rejected an if IMF demand you know that's pretty much making sense as a
[00:42:27] that's pretty much making sense as a piece of text um right so
[00:42:31] piece of text um right so it's mostly grammatical it looks like
[00:42:34] it's mostly grammatical it looks like you know English text I mean it it makes
[00:42:37] you know English text I mean it it makes no sense right it's sort of really
[00:42:39] no sense right it's sort of really incoherent so there there's work to do
[00:42:42] incoherent so there there's work to do but you know what was already you could
[00:42:45] but you know what was already you could see that even these simple engram models
[00:42:48] see that even these simple engram models you could from a very low level you
[00:42:52] you could from a very low level you could kind of approach what text and
[00:42:55] could kind of approach what text and human language worked like in from below
[00:42:59] human language worked like in from below and you know I could easily make this
[00:43:01] and you know I could easily make this better even with the engram language
[00:43:02] better even with the engram language model because you know rather than two
[00:43:04] model because you know rather than two million words of text if I trained on 10
[00:43:06] million words of text if I trained on 10 million words of text would be better if
[00:43:08] million words of text would be better if I then rather than a trigram model could
[00:43:10] I then rather than a trigram model could go to a forr model get better and You'
[00:43:13] go to a forr model get better and You' sort of start getting better and better
[00:43:16] sort of start getting better and better um approximations of text um and so this
[00:43:20] um approximations of text um and so this is essentially what people um did until
[00:43:25] is essentially what people um did until about
[00:43:25] about 2012 and and you know really uh the same
[00:43:30] 2012 and and you know really uh the same story that people um tell today that
[00:43:33] story that people um tell today that scale will solve everything is exactly
[00:43:36] scale will solve everything is exactly the same story that people used to tell
[00:43:39] the same story that people used to tell in the early
[00:43:40] in the early 2010s with these engram language models
[00:43:43] 2010s with these engram language models if you weren't getting a good enough
[00:43:44] if you weren't getting a good enough results with your 10 million words of
[00:43:47] results with your 10 million words of text and a trigram language model the
[00:43:49] text and a trigram language model the answer was that if you had a 100 million
[00:43:52] answer was that if you had a 100 million words of text and a for gram language
[00:43:54] words of text and a for gram language model you'd do better and then if you
[00:43:56] model you'd do better and then if you had a trillion words of text in a five
[00:43:59] had a trillion words of text in a five gr language model You' do better and gee
[00:44:01] gr language model You' do better and gee wouldn't it be good if we could collect
[00:44:02] wouldn't it be good if we could collect 10 trillion words of text so we could
[00:44:04] 10 trillion words of text so we could train an even better engram language
[00:44:07] train an even better engram language model same strategy um but it turns out
[00:44:10] model same strategy um but it turns out that sometimes you can do better with
[00:44:13] that sometimes you can do better with better models as well as simply scale um
[00:44:16] better models as well as simply scale um and so things got reinvented and started
[00:44:20] and so things got reinvented and started again with building neural language
[00:44:23] again with building neural language models um so how can we build a neural
[00:44:26] models um so how can we build a neural language model um so you know we've got
[00:44:29] language model um so you know we've got the same task of having a sequence of
[00:44:33] the same task of having a sequence of words and we want to put a probability
[00:44:35] words and we want to put a probability estimate over what word comes next um
[00:44:39] estimate over what word comes next um and so the simplest way you could do
[00:44:41] and so the simplest way you could do that which you'll hopefully all have
[00:44:43] that which you'll hopefully all have thought of because it connects what we
[00:44:46] thought of because it connects what we did in earlier classes look we already
[00:44:49] did in earlier classes look we already had this idea that we could have
[00:44:52] had this idea that we could have represented context by the concatenation
[00:44:55] represented context by the concatenation of some word vectors and we could put
[00:44:58] of some word vectors and we could put that into a neural network and we could
[00:45:02] that into a neural network and we could use that to predict something and in the
[00:45:05] use that to predict something and in the example I did in the last couple of
[00:45:07] example I did in the last couple of classes what we used it to predict was
[00:45:10] classes what we used it to predict was is the center word a location or not a
[00:45:13] is the center word a location or not a location just a binary choice but that's
[00:45:16] location just a binary choice but that's not the only thing we could predict we
[00:45:18] not the only thing we could predict we could have predicted lots of things with
[00:45:20] could have predicted lots of things with this new network we could have predicted
[00:45:22] this new network we could have predicted whether the piece of text was positive
[00:45:24] whether the piece of text was positive or negative we could have predicted
[00:45:26] or negative we could have predicted whether it was written in English or
[00:45:28] whether it was written in English or Japanese you know we can predict lots of
[00:45:30] Japanese you know we can predict lots of things so one thing we could choose to
[00:45:33] things so one thing we could choose to predict is we could choose to predict
[00:45:35] predict is we could choose to predict what word is going to come next after
[00:45:37] what word is going to come next after this window of text we'd have a model
[00:45:39] this window of text we'd have a model just like this one apart from up the top
[00:45:42] just like this one apart from up the top instead of doing this binary
[00:45:44] instead of doing this binary classification we' do a many many way
[00:45:47] classification we' do a many many way classification over what is the next
[00:45:50] classification over what is the next word that is going to appear in the
[00:45:52] word that is going to appear in the piece of text and that would then give
[00:45:54] piece of text and that would then give us a neural language model in particular
[00:45:58] us a neural language model in particular it give us a fixed window neural
[00:46:01] it give us a fixed window neural language model so that we'd do the same
[00:46:05] language model so that we'd do the same Markoff assumption trick of throwing
[00:46:07] Markoff assumption trick of throwing away the further back context and so for
[00:46:10] away the further back context and so for the fixed window um we'll you know um
[00:46:15] the fixed window um we'll you know um use word embeddings which you can
[00:46:18] use word embeddings which you can concatenate we'll put it through a
[00:46:20] concatenate we'll put it through a hidden layer and then we'll take the
[00:46:22] hidden layer and then we'll take the output of that hidden layer um multiply
[00:46:26] output of that hidden layer um multiply it by by another layer say and then put
[00:46:29] it by by another layer say and then put that through a soft Max and get an
[00:46:31] that through a soft Max and get an output distribution and so this gives us
[00:46:35] output distribution and so this gives us a sort of a fixed window neural language
[00:46:38] a sort of a fixed window neural language model and you know apart from the fact
[00:46:40] model and you know apart from the fact that we're now doing a classification
[00:46:43] that we're now doing a classification over many many many classes this is
[00:46:46] over many many many classes this is exactly like what we did um last week so
[00:46:49] exactly like what we did um last week so it should look kind of familiar it's
[00:46:52] it should look kind of familiar it's also kind of like what you're doing for
[00:46:54] also kind of like what you're doing for assignment two um and so this is
[00:46:57] assignment two um and so this is essentially the first kind of new
[00:47:00] essentially the first kind of new language model that was
[00:47:03] language model that was proposed um so in particular um yosua
[00:47:06] proposed um so in particular um yosua Benjo um really sort of right at the
[00:47:09] Benjo um really sort of right at the beginning of the 21st century suggested
[00:47:12] beginning of the 21st century suggested that you could do this that rather than
[00:47:14] that you could do this that rather than using an engram language model you could
[00:47:16] using an engram language model you could use a fixed window neural language model
[00:47:20] use a fixed window neural language model and you know even at that point um he
[00:47:23] and you know even at that point um he and colleagues were able to get some
[00:47:26] and colleagues were able to get some positive results from this model but you
[00:47:29] positive results from this model but you know at the time it wasn't widely
[00:47:31] know at the time it wasn't widely noticed it didn't really take off that
[00:47:34] noticed it didn't really take off that much and you know it was sort of for a
[00:47:36] much and you know it was sort of for a combination of reasons when it was only
[00:47:38] combination of reasons when it was only a fixed window it was sort of not that
[00:47:41] a fixed window it was sort of not that different to engrs in some sense and
[00:47:44] different to engrs in some sense and although the new network could give
[00:47:46] although the new network could give better
[00:47:47] better generalization it could be argued rather
[00:47:49] generalization it could be argued rather than using counts I mean in practice you
[00:47:53] than using counts I mean in practice you know new Nets were still hard to run
[00:47:56] know new Nets were still hard to run without gpus and people felt and I think
[00:48:00] without gpus and people felt and I think in general this was the case that you
[00:48:02] in general this was the case that you could get more oomph by doing the scale
[00:48:06] could get more oomph by doing the scale story and um collecting your engram
[00:48:09] story and um collecting your engram counts on hundreds of billions of words
[00:48:12] counts on hundreds of billions of words of text um rather than trying to make a
[00:48:14] of text um rather than trying to make a new network out of it and so it didn't
[00:48:17] new network out of it and so it didn't really sort of especially take off at
[00:48:19] really sort of especially take off at that time but you know in principle it
[00:48:21] that time but you know in principle it seemed a nice thing it you know got rid
[00:48:24] seemed a nice thing it you know got rid of the spasy problem um it got rid of
[00:48:27] of the spasy problem um it got rid of the storage costs you no longer have to
[00:48:29] the storage costs you no longer have to store all observed engrs you just have
[00:48:32] store all observed engrs you just have to store the parameters of your newal
[00:48:34] to store the parameters of your newal network but it didn't solve all the
[00:48:37] network but it didn't solve all the problems that we'd like to solve so in
[00:48:40] problems that we'd like to solve so in particular we still have this problem of
[00:48:42] particular we still have this problem of the Markoff assumption that we're just
[00:48:44] the Markoff assumption that we're just using a small fixed context beforehand
[00:48:48] using a small fixed context beforehand to predict
[00:48:49] to predict from um and there are some disadvantages
[00:48:53] from um and there are some disadvantages to enlarging that window and you know
[00:48:56] to enlarging that window and you know there's no fixed window that's ever big
[00:48:58] there's no fixed window that's ever big enough um there's another there's
[00:49:01] enough um there's another there's another thing that if you look
[00:49:03] another thing that if you look technically at this model that might
[00:49:06] technically at this model that might sort of make you suspicious of it which
[00:49:09] sort of make you suspicious of it which is you know when we have words in
[00:49:13] is you know when we have words in different positions that those words and
[00:49:17] different positions that those words and different positions will be treated by
[00:49:20] different positions will be treated by completely different subp parts of this
[00:49:22] completely different subp parts of this Matrix W so you might think that you
[00:49:26] Matrix W so you might think that you know know okay for predicting that books
[00:49:30] know know okay for predicting that books comes next you know the fact that this
[00:49:32] comes next you know the fact that this is a student um is important but it
[00:49:36] is a student um is important but it doesn't matter so much exactly where the
[00:49:39] doesn't matter so much exactly where the word student occurs right you know the
[00:49:42] word student occurs right you know the context could have been the students
[00:49:44] context could have been the students slowly open there um and it's still the
[00:49:48] slowly open there um and it's still the same students we've just got a bit
[00:49:50] same students we've just got a bit different linguistic structure where
[00:49:52] different linguistic structure where this W Matrix would be using completely
[00:49:55] this W Matrix would be using completely separate parameters to be learning stuff
[00:49:57] separate parameters to be learning stuff about student here versus student in
[00:50:00] about student here versus student in this position so that seems kind of
[00:50:02] this position so that seems kind of inefficient and wrong um and so that
[00:50:06] inefficient and wrong um and so that suggested that we kind of need a
[00:50:08] suggested that we kind of need a different kind of neural architecture
[00:50:11] different kind of neural architecture that can process any length of input and
[00:50:15] that can process any length of input and can use the same parameters to say hey I
[00:50:19] can use the same parameters to say hey I saw the word student that's evidence
[00:50:21] saw the word student that's evidence that things like books exams homework
[00:50:24] that things like books exams homework will be turning up regardless of where
[00:50:26] will be turning up regardless of where it occurs and so that then led to
[00:50:30] it occurs and so that then led to exploration of this different neural
[00:50:32] exploration of this different neural network architecture um called recurrent
[00:50:35] network architecture um called recurrent neural networks which is what I'll go on
[00:50:37] neural networks which is what I'll go on to next but before I do is everyone
[00:50:40] to next but before I do is everyone basically okay with what a language
[00:50:42] basically okay with what a language model is yeah no
[00:50:47] questions okay um we're current newal
[00:50:51] questions okay um we're current newal networks
[00:50:54] um so
[00:50:57] um so recurrent newal networks is a different
[00:51:00] recurrent newal networks is a different family of newal networks so effectively
[00:51:03] family of newal networks so effectively in this class we see several neural
[00:51:06] in this class we see several neural network
[00:51:07] network architectures um so in some sense the
[00:51:11] architectures um so in some sense the first architecture we saw was word to V
[00:51:14] first architecture we saw was word to V it's a sort of a very simple um encoder
[00:51:18] it's a sort of a very simple um encoder decoder architecture um the second
[00:51:21] decoder architecture um the second family we saw was feed forward Network
[00:51:26] family we saw was feed forward Network or fully connected layer classic neural
[00:51:29] or fully connected layer classic neural networks and the third family we're
[00:51:31] networks and the third family we're going to see is recurrent neural
[00:51:33] going to see is recurrent neural networks which have different kinds and
[00:51:35] networks which have different kinds and then we'll go on and go on to
[00:51:37] then we'll go on and go on to Transformer models okay so the idea of a
[00:51:41] Transformer models okay so the idea of a recurrent newal network is that you've
[00:51:44] recurrent newal network is that you've got one set of Weights that are going to
[00:51:48] got one set of Weights that are going to be applied through successive moments in
[00:51:52] be applied through successive moments in time I successive positions in the text
[00:51:56] time I successive positions in the text and as you do that you're going to
[00:51:58] and as you do that you're going to update the parameters as you go um we'll
[00:52:01] update the parameters as you go um we'll go through this in quite a bit of detail
[00:52:04] go through this in quite a bit of detail but you know here's the idea of it so
[00:52:06] but you know here's the idea of it so we've got the students open there and we
[00:52:09] we've got the students open there and we want to predict with that and the way
[00:52:12] want to predict with that and the way that we're going to do
[00:52:14] that we're going to do it okay I've still got four words in my
[00:52:17] it okay I've still got four words in my example so I can put stuff down the left
[00:52:19] example so I can put stuff down the left side of the slide but there could have
[00:52:22] side of the slide but there could have been 24 words with recurrent new
[00:52:24] been 24 words with recurrent new networks because they can deal with any
[00:52:26] networks because they can deal with any length of context okay so as before our
[00:52:30] length of context okay so as before our words start off as just words or one hot
[00:52:34] words start off as just words or one hot vectors and we can look up their word
[00:52:36] vectors and we can look up their word embeddings just like
[00:52:38] embeddings just like before okay but now to compute
[00:52:42] before okay but now to compute probabilities for the next word we're
[00:52:45] probabilities for the next word we're going to do something different so our
[00:52:47] going to do something different so our hidden layer is going to be recurrent
[00:52:51] hidden layer is going to be recurrent and by recurrent it means we're going to
[00:52:54] and by recurrent it means we're going to sort of change a hidden State at each
[00:52:57] sort of change a hidden State at each time step as we proceed through the text
[00:53:00] time step as we proceed through the text from left to right um so we're going to
[00:53:03] from left to right um so we're going to start off with an h0 which is the
[00:53:05] start off with an h0 which is the initial hidden state which can actually
[00:53:08] initial hidden state which can actually just be all zeros um and then at each
[00:53:12] just be all zeros um and then at each time step what we're going to do is
[00:53:15] time step what we're going to do is we're going to multiply the previous
[00:53:18] we're going to multiply the previous hidden state by a weight M
[00:53:20] hidden state by a weight M Matrix we're going to take a word
[00:53:23] Matrix we're going to take a word embedding and multiply it by a weight
[00:53:25] embedding and multiply it by a weight Matrix
[00:53:26] Matrix and then we're going to sum the results
[00:53:28] and then we're going to sum the results of those two things and that's going to
[00:53:30] of those two things and that's going to give us a new hidden state so that
[00:53:33] give us a new hidden state so that hidden state will then sort of store a
[00:53:37] hidden state will then sort of store a memory of everything that's been seen so
[00:53:39] memory of everything that's been seen so far so we'll do that and then we'll
[00:53:43] far so we'll do that and then we'll continue along so we'll multiply the
[00:53:45] continue along so we'll multiply the next word vector by the same weight
[00:53:48] next word vector by the same weight Matrix we we store the previous multiply
[00:53:52] Matrix we we store the previous multiply the previous hidden state by the same
[00:53:54] the previous hidden state by the same weight M Matrix wa each and we add them
[00:53:58] weight M Matrix wa each and we add them together and get a new
[00:54:00] together and get a new representation um I've only sort of said
[00:54:04] representation um I've only sort of said this bit so I've left out a bit commonly
[00:54:06] this bit so I've left out a bit commonly there are two other things you're doing
[00:54:08] there are two other things you're doing you're adding on a biased term because
[00:54:10] you're adding on a biased term because we usually separate out a bias term and
[00:54:13] we usually separate out a bias term and you're putting things through a
[00:54:14] you're putting things through a nonlinearity so I should make sure I
[00:54:16] nonlinearity so I should make sure I mention that and for recurrent neural
[00:54:19] mention that and for recurrent neural networks most commonly this nonlinearity
[00:54:22] networks most commonly this nonlinearity has actually been the tan H function so
[00:54:24] has actually been the tan H function so it's sort of balanced on the positive
[00:54:26] it's sort of balanced on the positive negative side and so you keep on doing
[00:54:29] negative side and so you keep on doing that through each step and so the idea
[00:54:32] that through each step and so the idea is once we've got to here this H4 hidden
[00:54:36] is once we've got to here this H4 hidden state is a hidden state that in some
[00:54:38] state is a hidden state that in some sense has read the text up until now
[00:54:41] sense has read the text up until now it's seen all of the students open there
[00:54:44] it's seen all of the students open there and if the word students occurred in any
[00:54:47] and if the word students occurred in any of these positions it will have been
[00:54:49] of these positions it will have been multiplied by the same we Matrix and
[00:54:53] multiplied by the same we Matrix and added into the hidden state so it's kind
[00:54:55] added into the hidden state so it's kind of got a cleaner
[00:54:56] of got a cleaner low parameter way of incorporating in
[00:54:59] low parameter way of incorporating in the information that seen so now I want
[00:55:02] the information that seen so now I want to predict the next word and to predict
[00:55:05] to predict the next word and to predict the next word I'm then going to do based
[00:55:08] the next word I'm then going to do based on the final hidden State the same thing
[00:55:11] on the final hidden State the same thing I did kind of thing I did before so I'm
[00:55:14] I did kind of thing I did before so I'm going to multiply that hidden state by
[00:55:17] going to multiply that hidden state by matrix and add another bias and stick
[00:55:19] matrix and add another bias and stick that through a soft Max and use that to
[00:55:24] that through a soft Max and use that to um sample from that soft Max well the
[00:55:26] um sample from that soft Max well the softmax will give me a language model of
[00:55:28] softmax will give me a language model of probability over all next words and I
[00:55:31] probability over all next words and I can sample from it to generate the next
[00:55:36] word that make
[00:55:38] word that make sense okay recurrent new um
[00:55:42] sense okay recurrent new um networks
[00:55:48] um okay um so for current newal networks
[00:55:52] um okay um so for current newal networks we can now process any length of
[00:55:55] we can now process any length of preceding content next and we'll just
[00:55:57] preceding content next and we'll just put more and more stuff in our hidden
[00:56:00] put more and more stuff in our hidden State um the so our computation is using
[00:56:05] State um the so our computation is using information from many steps back um our
[00:56:10] information from many steps back um our model Size Doesn't increase for having a
[00:56:13] model Size Doesn't increase for having a long context right you know we have to
[00:56:16] long context right you know we have to do more computation for a long context
[00:56:19] do more computation for a long context but our representation of that long
[00:56:21] but our representation of that long context just remains this fixed size
[00:56:23] context just remains this fixed size hidden Vector h of whatever dimension it
[00:56:26] hidden Vector h of whatever dimension it is so there's no exponential blowout
[00:56:29] is so there's no exponential blowout anymore um there's the same weights
[00:56:31] anymore um there's the same weights appli in every time step so there's a
[00:56:33] appli in every time step so there's a symmetry and how inputs are processed um
[00:56:36] symmetry and how inputs are processed um there are some
[00:56:38] there are some catches um the biggest catch in practice
[00:56:42] catches um the biggest catch in practice is that recurrent computation is slow so
[00:56:45] is that recurrent computation is slow so for the feed forward layer we just had
[00:56:48] for the feed forward layer we just had you know our input Vector we multiply it
[00:56:51] you know our input Vector we multiply it by matrix we multiply it by matrix
[00:56:53] by matrix we multiply it by matrix however many times and then at the end
[00:56:55] however many times and then at the end we're done whereas here we've sort of
[00:56:58] we're done whereas here we've sort of stuck with this sequentiality that you
[00:57:00] stuck with this sequentiality that you have to be doing one hidden Vector at a
[00:57:03] have to be doing one hidden Vector at a time in fact this is going against what
[00:57:06] time in fact this is going against what I said at the beginning of class because
[00:57:08] I said at the beginning of class because essentially here you're doing a for Loop
[00:57:11] essentially here you're doing a for Loop um you're going through for time equals
[00:57:13] um you're going through for time equals 1 to T and then you're generating and
[00:57:16] 1 to T and then you're generating and term each hidden vector and that's one
[00:57:18] term each hidden vector and that's one of the big problems with rnns that have
[00:57:21] of the big problems with rnns that have led them to fall out of favor um there's
[00:57:25] led them to fall out of favor um there's another
[00:57:26] another problem that we'll look at more is that
[00:57:30] problem that we'll look at more is that in theory this is perfect you're just
[00:57:33] in theory this is perfect you're just incorporating all of the past context in
[00:57:36] incorporating all of the past context in in your hidden Vector in practice it
[00:57:39] in your hidden Vector in practice it tends not to work perfectly because you
[00:57:42] tends not to work perfectly because you know although stuff you saw back here is
[00:57:46] know although stuff you saw back here is in some sense still alive in the hidden
[00:57:48] in some sense still alive in the hidden Vector as you come across here that your
[00:57:52] Vector as you come across here that your memory of it gets more and more distant
[00:57:55] memory of it gets more and more distant and it's the words that you saw recently
[00:57:57] and it's the words that you saw recently that dominate The Hidden State now in
[00:58:00] that dominate The Hidden State now in some sense that's right because the
[00:58:01] some sense that's right because the recent stuff is the most important stuff
[00:58:03] recent stuff is the most important stuff that's freshest in your mind you know
[00:58:05] that's freshest in your mind you know it's the same with human beings um they
[00:58:08] it's the same with human beings um they tend to forget stuff from further back
[00:58:10] tend to forget stuff from further back as well um but rnns especially in the
[00:58:13] as well um but rnns especially in the simple form that I've just explained
[00:58:15] simple form that I've just explained forget stuff from further back um rather
[00:58:19] forget stuff from further back um rather too quickly and we'll come back to that
[00:58:22] too quickly and we'll come back to that again um into in Thursday's class
[00:58:27] again um into in Thursday's class okay so for training an RNN language
[00:58:31] okay so for training an RNN language model um the starting off point is we
[00:58:33] model um the starting off point is we get a big Corpus of text again um and
[00:58:37] get a big Corpus of text again um and then we're going to compute um for each
[00:58:42] then we're going to compute um for each time step a prediction of the
[00:58:45] time step a prediction of the probability of next words and then
[00:58:48] probability of next words and then there's going to be an actual next word
[00:58:51] there's going to be an actual next word and we're going to use you know that as
[00:58:54] and we're going to use you know that as the basis of our loss
[00:58:56] the basis of our loss um so our loss function is the cross
[00:58:59] um so our loss function is the cross entropy between the predicted
[00:59:01] entropy between the predicted probability and what the actual next
[00:59:03] probability and what the actual next word that we saw is which again as in
[00:59:06] word that we saw is which again as in the example I showed before is just the
[00:59:09] the example I showed before is just the the negative log likelihood of the
[00:59:11] the negative log likelihood of the actual next word ideally you'd like to
[00:59:15] actual next word ideally you'd like to predict the actual next word with
[00:59:17] predict the actual next word with probability one which means the negative
[00:59:20] probability one which means the negative log of one would be zero and there'd be
[00:59:23] log of one would be zero and there'd be no loss but in practice if you give it
[00:59:26] no loss but in practice if you give it an estimate of 0 five there's only a
[00:59:28] an estimate of 0 five there's only a little bit of loss and so on and so um
[00:59:32] little bit of loss and so on and so um to get our overall objective function we
[00:59:35] to get our overall objective function we work out the average loss the average
[00:59:39] work out the average loss the average negative log likelihood of predicting
[00:59:41] negative log likelihood of predicting each word in turn so showing that as
[00:59:44] each word in turn so showing that as pictures if our Corpus is the students
[00:59:47] pictures if our Corpus is the students open their exams we're first of all
[00:59:50] open their exams we're first of all going to be trying to predict um you
[00:59:53] going to be trying to predict um you know what
[00:59:54] know what comes after the and we will predict some
[01:00:00] comes after the and we will predict some word with um different probabilities and
[01:00:03] word with um different probabilities and then we'll say oh the actual next word
[01:00:05] then we'll say oh the actual next word is students okay you gave that a
[01:00:07] is students okay you gave that a probability of 0.05 say because all you
[01:00:11] probability of 0.05 say because all you know was the first word was the okay
[01:00:13] know was the first word was the okay there's a loss for that um the negative
[01:00:16] there's a loss for that um the negative log prob given to students we then go on
[01:00:20] log prob given to students we then go on and generate the probability estimate
[01:00:23] and generate the probability estimate over the next words and then we say well
[01:00:26] over the next words and then we say well the actual word is opened what
[01:00:28] the actual word is opened what probability estimate did you give to
[01:00:30] probability estimate did you give to that we get a negative probability loss
[01:00:33] that we get a negative probability loss keep on running this along and then we
[01:00:36] keep on running this along and then we sum all of those losses and we average
[01:00:40] sum all of those losses and we average them per word and that's our sort of
[01:00:43] them per word and that's our sort of average per word loss and we want to
[01:00:46] average per word loss and we want to make that as small as possible and so
[01:00:50] make that as small as possible and so that's our training mechanism and it's
[01:00:53] that's our training mechanism and it's important to to know no that you know
[01:00:57] important to to know no that you know for generating this loss we're not just
[01:01:00] for generating this loss we're not just doing free generation we're not just
[01:01:02] doing free generation we're not just saying to the model go off and generate
[01:01:04] saying to the model go off and generate a sentence um what we're actually doing
[01:01:07] a sentence um what we're actually doing is at each step we're effectively saying
[01:01:10] is at each step we're effectively saying okay the prefix is the students open
[01:01:13] okay the prefix is the students open what probability distribution do you put
[01:01:15] what probability distribution do you put on next words after that um generate it
[01:01:19] on next words after that um generate it with our current new network and then
[01:01:21] with our current new network and then say ask for the actual next word what
[01:01:24] say ask for the actual next word what probability estimate did you give to
[01:01:26] probability estimate did you give to there and that's our loss but then what
[01:01:29] there and that's our loss but then what we do is stick there into our current
[01:01:32] we do is stick there into our current new network the right answer so we
[01:01:35] new network the right answer so we always go back to the right answer
[01:01:38] always go back to the right answer generate probability distribution for
[01:01:41] generate probability distribution for next words and then ask okay what
[01:01:43] next words and then ask okay what probability did you give to the actual
[01:01:45] probability did you give to the actual next word exams and then again we use
[01:01:49] next word exams and then again we use the actual next word so we do one step
[01:01:51] the actual next word so we do one step of generation then we pull it back to
[01:01:54] of generation then we pull it back to what was actually gener ated what was
[01:01:57] what was actually gener ated what was what was actually in the text and then
[01:01:59] what was actually in the text and then we ask it for guesses over the next word
[01:02:02] we ask it for guesses over the next word and repeat forever and so the fact that
[01:02:04] and repeat forever and so the fact that we don't do free generation but we pull
[01:02:07] we don't do free generation but we pull it back to the actual piece of text each
[01:02:10] it back to the actual piece of text each time um makes things simple because we
[01:02:13] time um makes things simple because we sort of know what an actual author used
[01:02:17] sort of know what an actual author used for the next word um and that process is
[01:02:21] for the next word um and that process is called teacher forcing and so the most
[01:02:23] called teacher forcing and so the most common way to
[01:02:26] common way to language models is using this kind of
[01:02:28] language models is using this kind of teacher forcing method I mean it's not
[01:02:30] teacher forcing method I mean it's not perfect in all respects cuz you know
[01:02:33] perfect in all respects cuz you know we're not actually exploring different
[01:02:35] we're not actually exploring different things the model might want to generate
[01:02:37] things the model might want to generate on its own and seeing what comes after
[01:02:39] on its own and seeing what comes after them we're only doing the tell me the
[01:02:42] them we're only doing the tell me the next word from some human generated
[01:02:44] next word from some human generated piece of
[01:02:51] text okay um so that's how we get losses
[01:02:56] text okay um so that's how we get losses um and then after that um we want to as
[01:03:00] um and then after that um we want to as before use these losses to update the
[01:03:03] before use these losses to update the parameters of a newal
[01:03:06] parameters of a newal network okay um and how do we do that um
[01:03:12] network okay um and how do we do that um well in principle you know we just have
[01:03:14] well in principle you know we just have all of the texts that we've collected
[01:03:17] all of the texts that we've collected which you could think of as just a
[01:03:18] which you could think of as just a really long sequence of okay we've got a
[01:03:21] really long sequence of okay we've got a billion words of text here it is right
[01:03:24] billion words of text here it is right so in theory you could just run your um
[01:03:28] so in theory you could just run your um recurrent newal network over your
[01:03:29] recurrent newal network over your billion words of text updating the
[01:03:32] billion words of text updating the context as you go um but that would make
[01:03:36] context as you go um but that would make it very difficult to train a model
[01:03:40] it very difficult to train a model because you'd be accumulating these
[01:03:42] because you'd be accumulating these losses for a billion steps and you'd
[01:03:45] losses for a billion steps and you'd have to store them um and then You' be
[01:03:49] have to store them um and then You' be you'd have to store hidden States so you
[01:03:51] you'd have to store hidden States so you could update parameters and it just
[01:03:53] could update parameters and it just wouldn't work so what we actually do is
[01:03:56] wouldn't work so what we actually do is we cut our training data into segments
[01:04:00] we cut our training data into segments of a reasonable length and then we're
[01:04:03] of a reasonable length and then we're going to sort of run our recurrent newal
[01:04:06] going to sort of run our recurrent newal Network on those segments and then we're
[01:04:09] Network on those segments and then we're going to compute a loss for each segment
[01:04:12] going to compute a loss for each segment and then we're going to update the
[01:04:15] and then we're going to update the parameters of the recurrent new network
[01:04:18] parameters of the recurrent new network based on the losses that we found for
[01:04:20] based on the losses that we found for that segment um I I describe it here as
[01:04:24] that segment um I I describe it here as the segments being sentences or
[01:04:26] the segments being sentences or documents which seems a linguistically
[01:04:29] documents which seems a linguistically nice thing it turns out that um in
[01:04:33] nice thing it turns out that um in recent practice when you're wanting to
[01:04:35] recent practice when you're wanting to scale most efficiently on gpus people
[01:04:39] scale most efficiently on gpus people don't bother with those linguistic
[01:04:41] don't bother with those linguistic niceties they just say a segment is 100
[01:04:44] niceties they just say a segment is 100 words just cut every 100 Words and the
[01:04:48] words just cut every 100 Words and the reason why that's really convenient is
[01:04:50] reason why that's really convenient is you can then create a batch of segments
[01:04:53] you can then create a batch of segments all of which are 100 words long and
[01:04:55] all of which are 100 words long and stick those in a matrix and do um
[01:04:59] stick those in a matrix and do um vectorized training more efficiently um
[01:05:02] vectorized training more efficiently um and things go great for you okay but
[01:05:06] and things go great for you okay but there's still a few more things that we
[01:05:07] there's still a few more things that we need to know um to get things to work
[01:05:09] need to know um to get things to work great for you I was try and get a bit
[01:05:11] great for you I was try and get a bit bit more through this before um today
[01:05:14] bit more through this before um today ends so we sort of need to know about
[01:05:17] ends so we sort of need to know about how to work out the derivative of our
[01:05:21] how to work out the derivative of our loss with respect to um the
[01:05:26] loss with respect to um the parameters of our recurrent newal
[01:05:29] parameters of our recurrent newal Network and the interesting case here is
[01:05:33] Network and the interesting case here is you know these wh parameters are sort of
[01:05:36] you know these wh parameters are sort of being used everywhere through the neural
[01:05:39] being used everywhere through the neural network at each stage as are the we ones
[01:05:43] network at each stage as are the we ones so they appear at many places in the
[01:05:45] so they appear at many places in the network so how do we work out the
[01:05:48] network so how do we work out the partial derivatives of the loss with
[01:05:52] partial derivatives of the loss with respect to the repeated weight
[01:05:54] respect to the repeated weight matrices and and the answer to that is
[01:05:57] matrices and and the answer to that is oh it's really simple um you can just
[01:06:01] oh it's really simple um you can just sort of pretend that those wh's in each
[01:06:05] sort of pretend that those wh's in each position are different and work out the
[01:06:09] position are different and work out the partials with respect to them at one
[01:06:11] partials with respect to them at one position and then to get the partials
[01:06:14] position and then to get the partials with respect to wh you just sum whatever
[01:06:17] with respect to wh you just sum whatever you found in the different
[01:06:20] you found in the different positions and so um that is sort of okay
[01:06:26] positions and so um that is sort of okay the gradient with respect to repeated
[01:06:28] the gradient with respect to repeated weight is the sum of the gradient with
[01:06:30] weight is the sum of the gradient with respect to each time it appears and the
[01:06:33] respect to each time it appears and the reason why that is it sort of follows
[01:06:36] reason why that is it sort of follows what I talked about in lecture three um
[01:06:40] what I talked about in lecture three um that we talk or you know you can also
[01:06:44] that we talk or you know you can also think about it in terms of what you
[01:06:46] think about it in terms of what you might remember um from you know
[01:06:49] might remember um from you know multivariable chain roles but you know
[01:06:51] multivariable chain roles but you know the way I introduced in lecture three is
[01:06:54] the way I introduced in lecture three is that gradient at outward branches and so
[01:06:58] that gradient at outward branches and so what you can think about it in a case
[01:07:01] what you can think about it in a case like this is that you've got uh wh
[01:07:05] like this is that you've got uh wh Matrix which is being copied by identity
[01:07:08] Matrix which is being copied by identity to wh1 wh2 wh3 wh4 Etc at each time step
[01:07:15] to wh1 wh2 wh3 wh4 Etc at each time step and so since those are identity copies
[01:07:19] and so since those are identity copies they have um a partial derivative with
[01:07:23] they have um a partial derivative with respect to each other of one
[01:07:26] respect to each other of one and so then we apply the multivariable
[01:07:29] and so then we apply the multivariable chain roll to these copies um and so
[01:07:34] chain roll to these copies um and so we've then got an outward branching node
[01:07:37] we've then got an outward branching node and you're just summing the gradients um
[01:07:40] and you're just summing the gradients um to get the total gradient of each time
[01:07:43] to get the total gradient of each time for The
[01:07:48] Matrix okay
[01:07:52] Matrix okay um yeah I mean there's one other trick
[01:07:55] um yeah I mean there's one other trick that's perhaps worth knowing I mean if
[01:07:58] that's perhaps worth knowing I mean if you've got sort of segments that are 100
[01:08:00] you've got sort of segments that are 100 long um a common speed up is to say oh
[01:08:04] long um a common speed up is to say oh maybe we don't actually have to run back
[01:08:07] maybe we don't actually have to run back propagation for 100 time steps maybe we
[01:08:09] propagation for 100 time steps maybe we could just run it for 20 times steps and
[01:08:12] could just run it for 20 times steps and stop which is referred to as truncated
[01:08:14] stop which is referred to as truncated back propagation through time I mean in
[01:08:17] back propagation through time I mean in practice that tends to be sufficient
[01:08:20] practice that tends to be sufficient note in particular you're still on the
[01:08:22] note in particular you're still on the forward path updating your hidden State
[01:08:25] forward path updating your hidden State using your full context but in the back
[01:08:28] using your full context but in the back propagation you're just sort of cutting
[01:08:31] propagation you're just sort of cutting it short um to speed up
[01:08:34] it short um to speed up training okay um so just as I did before
[01:08:38] training okay um so just as I did before with an engram language model we can use
[01:08:42] with an engram language model we can use uh RNN language model to generate text
[01:08:46] uh RNN language model to generate text and it's pretty much the same idea
[01:08:48] and it's pretty much the same idea except now we're sort of um rather than
[01:08:51] except now we're sort of um rather than just using counts of engrs we're using
[01:08:54] just using counts of engrs we're using the hidden state of our neural network
[01:08:57] the hidden state of our neural network to give us the input to a a probability
[01:09:01] to give us the input to a a probability distribution that we can then sample
[01:09:03] distribution that we can then sample from so I can start with the initial
[01:09:05] from so I can start with the initial hidden State um I can use the start of
[01:09:09] hidden State um I can use the start of sentence symbol I mean the example I had
[01:09:11] sentence symbol I mean the example I had before I started immediately with the um
[01:09:15] before I started immediately with the um hoping that that was less confusing the
[01:09:17] hoping that that was less confusing the first time but what you should have
[01:09:19] first time but what you should have asked is wait a minute where did the the
[01:09:21] asked is wait a minute where did the the come from um so normally what we
[01:09:24] come from um so normally what we actually do is is use a special start of
[01:09:28] actually do is is use a special start of sequence symbol like this angle
[01:09:31] sequence symbol like this angle bracketed s and so we sort of feed it in
[01:09:34] bracketed s and so we sort of feed it in as a pseudo word which has a word
[01:09:37] as a pseudo word which has a word embedding and then we based on this will
[01:09:40] embedding and then we based on this will be generating first words of the text um
[01:09:43] be generating first words of the text um so we end up with some
[01:09:46] so we end up with some representation from which we can sample
[01:09:49] representation from which we can sample and get the first word so now we don't
[01:09:52] and get the first word so now we don't have any actual text so what we're going
[01:09:54] have any actual text so what we're going to do
[01:09:55] to do is take that generated word that we
[01:09:58] is take that generated word that we generated and copy it down as the next
[01:10:01] generated and copy it down as the next input and then we're going to run the
[01:10:04] input and then we're going to run the next stage of newal network um sample
[01:10:08] next stage of newal network um sample from the probability distribution and
[01:10:10] from the probability distribution and next word favorite copy it down as the
[01:10:13] next word favorite copy it down as the next word of the input and keep on
[01:10:16] next word of the input and keep on generating and so this is referred to as
[01:10:18] generating and so this is referred to as a roll out that you're kind of
[01:10:21] a roll out that you're kind of continuing to roll the dice and generate
[01:10:23] continuing to roll the dice and generate forward and generate a piece of text and
[01:10:26] forward and generate a piece of text and so um and normally you want to stop at
[01:10:31] so um and normally you want to stop at some point and the way we can stop it
[01:10:33] some point and the way we can stop it some point is we can have a second
[01:10:35] some point is we can have a second special symbol um the angle bracket SLS
[01:10:40] special symbol um the angle bracket SLS which um says end of um your sequence so
[01:10:44] which um says end of um your sequence so we can generate an end of sequence
[01:10:46] we can generate an end of sequence symbol and then we can um stop and so
[01:10:50] symbol and then we can um stop and so using this we can sort of generate
[01:10:52] using this we can sort of generate pieces of text and essentially you know
[01:10:55] pieces of text and essentially you know this is exactly what's happening if you
[01:10:57] this is exactly what's happening if you use something like chat GPT right that
[01:11:00] use something like chat GPT right that the model is a more complicated model
[01:11:02] the model is a more complicated model that we've haven't yet gotten to but
[01:11:04] that we've haven't yet gotten to but it's generating the response to you by
[01:11:07] it's generating the response to you by doing this kind of process of generating
[01:11:09] doing this kind of process of generating a word at the time treating it as an
[01:11:12] a word at the time treating it as an input and generating the next word and
[01:11:15] input and generating the next word and generating this sort of roll out and
[01:11:17] generating this sort of roll out and that's why and it's done
[01:11:19] that's why and it's done probabilistically so if you do it
[01:11:21] probabilistically so if you do it multiple times um you can get different
[01:11:24] multiple times um you can get different answers we haven't yet gone to chat GPT
[01:11:27] answers we haven't yet gone to chat GPT but we can have a little bit of fun um
[01:11:29] but we can have a little bit of fun um so you can take this simple recurrent
[01:11:32] so you can take this simple recurrent newal Network that we've just built here
[01:11:34] newal Network that we've just built here and you can train it on any piece of
[01:11:37] and you can train it on any piece of text and get it to generate stuff so for
[01:11:41] text and get it to generate stuff so for example I can train it on Barack Obama's
[01:11:44] example I can train it on Barack Obama's speeches so that's a small Corpus right
[01:11:46] speeches so that's a small Corpus right you know he didn't talk that much right
[01:11:48] you know he didn't talk that much right I've only got a few hundred thousand
[01:11:50] I've only got a few hundred thousand words of text it's not a huge Corpus
[01:11:53] words of text it's not a huge Corpus I'll just show this and then I can
[01:11:54] I'll just show this and then I can answer the question um but you know I
[01:11:56] answer the question um but you know I can generate from it and I get something
[01:11:59] can generate from it and I get something like the United States will step up to
[01:12:01] like the United States will step up to the cost of a new challenges of the
[01:12:03] the cost of a new challenges of the American people that will share the fact
[01:12:05] American people that will share the fact that we created the problem they were
[01:12:08] that we created the problem they were attacked and so that they have to say
[01:12:09] attacked and so that they have to say that all the task of the final days of
[01:12:11] that all the task of the final days of war that I will not be able to get this
[01:12:14] war that I will not be able to get this done um yeah well maybe that's slightly
[01:12:17] done um yeah well maybe that's slightly better than my engram language model
[01:12:19] better than my engram language model still not perfect you might say but
[01:12:21] still not perfect you might say but somewhat better maybe did you have a
[01:12:24] somewhat better maybe did you have a question uh yeah so since we're like
[01:12:28] question uh yeah so since we're like training the mod like truncated set of
[01:12:30] training the mod like truncated set of the Corpus that impose some kind of like
[01:12:33] the Corpus that impose some kind of like limitation on like how much we can like
[01:12:36] limitation on like how much we can like produce and like still have some cency
[01:12:38] produce and like still have some cency like meaning like
[01:12:42] like meaning like foring um so yeah so I suggested we're
[01:12:45] foring um so yeah so I suggested we're going to chunk the S chunk the text into
[01:12:48] going to chunk the S chunk the text into 100w units so you know that's the limit
[01:12:52] 100w units so you know that's the limit of the amount of Prior context that
[01:12:53] of the amount of Prior context that we're going to use so I mean that's a
[01:12:56] we're going to use so I mean that's a fair amount 100 words that's typically
[01:12:58] fair amount 100 words that's typically several sentences but to the extent that
[01:13:01] several sentences but to the extent that you wanted to know even more about the
[01:13:04] you wanted to know even more about the further back context you wouldn't be
[01:13:06] further back context you wouldn't be able to and you know certainly that's
[01:13:09] able to and you know certainly that's one of the ways in which modern large
[01:13:11] one of the ways in which modern large language models are using far bigger
[01:13:14] language models are using far bigger context than that they're now using
[01:13:15] context than that they're now using thousands of words of Prior context yeah
[01:13:19] thousands of words of Prior context yeah absolutely it's a limit on how much far
[01:13:22] absolutely it's a limit on how much far back context so in some sense actually
[01:13:25] back context so in some sense actually even though in theory a current newal
[01:13:28] even though in theory a current newal Network can feed in an arbitrary length
[01:13:30] Network can feed in an arbitrary length context as soon as I say oh practically
[01:13:33] context as soon as I say oh practically we cut it into segments you know
[01:13:35] we cut it into segments you know actually that means we are making a
[01:13:37] actually that means we are making a Markoff assumption again and we're
[01:13:39] Markoff assumption again and we're saying the further back context doesn't
[01:13:42] saying the further back context doesn't matter yeah okay uh couple more examples
[01:13:47] matter yeah okay uh couple more examples um so instead of Barack Obama I can feed
[01:13:50] um so instead of Barack Obama I can feed in Harry Potter which is a somewhat
[01:13:52] in Harry Potter which is a somewhat bigger Corpus of text actually and
[01:13:55] bigger Corpus of text actually and generate from that and so I can get um
[01:13:58] generate from that and so I can get um sorry Harry shouted panicking I'll leave
[01:14:01] sorry Harry shouted panicking I'll leave those brooms in London are they no idea
[01:14:03] those brooms in London are they no idea said nearly headless Nick casting low
[01:14:05] said nearly headless Nick casting low close by Cedric carrying the last bit of
[01:14:08] close by Cedric carrying the last bit of trial charms from Harry's shoulder and
[01:14:10] trial charms from Harry's shoulder and to answer him the common room perched
[01:14:13] to answer him the common room perched upon it forearms held a shining knob
[01:14:16] upon it forearms held a shining knob from when the spider hadn't felt it
[01:14:17] from when the spider hadn't felt it seamed he reached the teams
[01:14:20] seamed he reached the teams too well there you are um you can do
[01:14:23] too well there you are um you can do other things as well um
[01:14:26] other things as well um so you can train it on recipes and
[01:14:28] so you can train it on recipes and generate a recipe um this one's a recipe
[01:14:32] generate a recipe um this one's a recipe I don't suggest you try and cook um but
[01:14:36] I don't suggest you try and cook um but it looks sort of like a recipe if you
[01:14:38] it looks sort of like a recipe if you don't look very hard um chocolate Ranch
[01:14:42] don't look very hard um chocolate Ranch Barbecue categories game casseroles
[01:14:45] Barbecue categories game casseroles cookies cookies yield six servings two P
[01:14:50] cookies cookies yield six servings two P two tablespoons of Parmesan cheese
[01:14:52] two tablespoons of Parmesan cheese chopped um one cup of Co coconut milk
[01:14:55] chopped um one cup of Co coconut milk and three eggs beaten um Place each
[01:14:58] and three eggs beaten um Place each pasture over layers of lumps shape
[01:15:01] pasture over layers of lumps shape mixture into the moderate oven and
[01:15:05] mixture into the moderate oven and simmer until firm serve hot and bodied
[01:15:09] simmer until firm serve hot and bodied fresh mustard orange and cheese combine
[01:15:13] fresh mustard orange and cheese combine the cheese and salt together the dough
[01:15:16] the cheese and salt together the dough in a large Skillet and the ingredients
[01:15:19] in a large Skillet and the ingredients and stir in the chocolate and pepper H
[01:15:22] and stir in the chocolate and pepper H um yeah it's not exactly very consistent
[01:15:25] um yeah it's not exactly very consistent recipe when it comes down to it it sort
[01:15:27] recipe when it comes down to it it sort of has a langage of a recipe but it's
[01:15:31] of has a langage of a recipe but it's Absolut maybe if I had scaled it more
[01:15:33] Absolut maybe if I had scaled it more and had a bigger Corpus it would have
[01:15:35] and had a bigger Corpus it would have done a bit better um but it's definitely
[01:15:37] done a bit better um but it's definitely not using the ingredients there are um
[01:15:41] not using the ingredients there are um let's see it's almost um time today so
[01:15:44] let's see it's almost um time today so maybe about all I can do um is uh do H I
[01:15:50] maybe about all I can do um is uh do H I can do one more fun example and then
[01:15:52] can do one more fun example and then after that oh yeah I probably should
[01:15:55] after that oh yeah I probably should that bit at the start next time um so as
[01:15:57] that bit at the start next time um so as a variant of building RNN language
[01:16:00] a variant of building RNN language models I mean so far we've been building
[01:16:03] models I mean so far we've been building them over words so our you know token
[01:16:09] them over words so our you know token time steps over which you build it as
[01:16:12] time steps over which you build it as words I mean actually you can use the
[01:16:14] words I mean actually you can use the idea of recurrent new networks over any
[01:16:16] idea of recurrent new networks over any other size unit and people have used
[01:16:19] other size unit and people have used them for other things so people have
[01:16:21] them for other things so people have used them in bioinformatics for things
[01:16:23] used them in bioinformatics for things like um DNA for sort of having Gene
[01:16:27] like um DNA for sort of having Gene sequencing or protein sequencing or
[01:16:29] sequencing or protein sequencing or anything like that but even staying with
[01:16:32] anything like that but even staying with language um instead of building them
[01:16:34] language um instead of building them over words you can build them over
[01:16:37] over words you can build them over characters so that my I'm generating at
[01:16:41] characters so that my I'm generating at a a letter at a time rather than a word
[01:16:44] a a letter at a time rather than a word at a time and so that can sometimes be
[01:16:47] at a time and so that can sometimes be useful because it allows us to sort of
[01:16:50] useful because it allows us to sort of generate things um that sort of look
[01:16:53] generate things um that sort of look like Words um and perhaps have the
[01:16:55] like Words um and perhaps have the structure of English words um and so and
[01:16:59] structure of English words um and so and so similarly there are other things that
[01:17:01] so similarly there are other things that you can do so before I
[01:17:04] you can do so before I initialized the hidden State I said oh
[01:17:08] initialized the hidden State I said oh you just have an initial hidden State
[01:17:10] you just have an initial hidden State you can make it zeros if you want well
[01:17:13] you can make it zeros if you want well sometimes we're going to build a
[01:17:15] sometimes we're going to build a contextual RNN where we can initialize
[01:17:18] contextual RNN where we can initialize the hidden State um with something else
[01:17:20] the hidden State um with something else so in particular I can initialize the
[01:17:23] so in particular I can initialize the hidden state with the RGB values of a
[01:17:27] hidden state with the RGB values of a color and so I can have initialized the
[01:17:30] color and so I can have initialized the hidden state with the color and generate
[01:17:33] hidden state with the color and generate character at a time the name of paint
[01:17:36] character at a time the name of paint colors and I can train a model based on
[01:17:40] colors and I can train a model based on um a paint company's catalog of names of
[01:17:44] um a paint company's catalog of names of colors and their um RGB of their colors
[01:17:48] colors and their um RGB of their colors and then I can give it different
[01:17:50] and then I can give it different different paint colors and it'll come up
[01:17:52] different paint colors and it'll come up with names for them and it actually does
[01:17:55] with names for them and it actually does an excellent job this one worked really
[01:17:57] an excellent job this one worked really well look at this um this one here is
[01:18:00] well look at this um this one here is gasty pink Power gray Naval tan bco
[01:18:07] gasty pink Power gray Naval tan bco white hble gray Home Star Brown now
[01:18:11] white hble gray Home Star Brown now couldn't you just imagine finding all of
[01:18:13] couldn't you just imagine finding all of these in a paint catalog I mean some of
[01:18:15] these in a paint catalog I mean some of them are there's some really good ones
[01:18:18] them are there's some really good ones over here in the bottom right this color
[01:18:20] over here in the bottom right this color here is
[01:18:22] here is dope and then um this Stoner blue purple
[01:18:28] dope and then um this Stoner blue purple s stinky
[01:18:30] s stinky bean and Turley now I think I've got a a
[01:18:34] bean and Turley now I think I've got a a real business opportunity here in the
[01:18:37] real business opportunity here in the Paint Company Market um for my recurrent
[01:18:40] Paint Company Market um for my recurrent new network okay I'll stop there for
[01:18:42] new network okay I'll stop there for today and do more of the science of new
[01:18:44] today and do more of the science of new networks next time