💬 CS224U – Natural Language Understanding

Introduction and Welcome | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=rha64cQRLs8

---

Transcript

[00:00:03] hi i'm chris potts i'm a professor in
[00:00:05] hi i'm chris potts i'm a professor in linguistics at stanford with a courtesy
[00:00:06] linguistics at stanford with a courtesy appointment in computer science and i'm
[00:00:08] appointment in computer science and i'm the director of stanford center for the
[00:00:10] the director of stanford center for the study of language and information which
[00:00:12] study of language and information which is an interdisciplinary research center
[00:00:13] is an interdisciplinary research center focused on logic language decision
[00:00:15] focused on logic language decision making human sentence processing and
[00:00:17] making human sentence processing and computation
[00:00:19] computation my undergraduate and graduate degrees
[00:00:20] my undergraduate and graduate degrees are all in linguistics the work i did
[00:00:22] are all in linguistics the work i did for my phd focused on topics in
[00:00:24] for my phd focused on topics in linguistic pragmatics which is the study
[00:00:26] linguistic pragmatics which is the study of how language use is shaped by the
[00:00:28] of how language use is shaped by the physical and social context that we're
[00:00:29] physical and social context that we're in
[00:00:30] in at a certain point i went looking for
[00:00:31] at a certain point i went looking for new ways to support those theories
[00:00:33] new ways to support those theories quantitatively and that began my journey
[00:00:35] quantitatively and that began my journey into the world of natural language
[00:00:36] into the world of natural language processing i now think of myself as
[00:00:38] processing i now think of myself as being on a mission to help with the
[00:00:40] being on a mission to help with the sharing of ideas back and forth between
[00:00:42] sharing of ideas back and forth between linguistics and nlp i've taught natural
[00:00:44] linguistics and nlp i've taught natural language understanding nine times at
[00:00:46] language understanding nine times at stanford with my first year in 2012. in
[00:00:49] stanford with my first year in 2012. in 2012 we were just beginning to see how
[00:00:51] 2012 we were just beginning to see how nlu was going to revolutionize the field
[00:00:53] nlu was going to revolutionize the field and reshape the technology landscape
[00:00:55] and reshape the technology landscape ibm's watson had recently won jeopardy
[00:00:58] ibm's watson had recently won jeopardy apple siri was new and the other tech
[00:01:00] apple siri was new and the other tech giants were on the verge of launching
[00:01:01] giants were on the verge of launching their own intelligent assistance so
[00:01:03] their own intelligent assistance so there was a widespread perception that
[00:01:05] there was a widespread perception that nlu was poised to have a transformative
[00:01:07] nlu was poised to have a transformative impact on the world and that perception
[00:01:09] impact on the world and that perception was certainly correct since then nlu has
[00:01:12] was certainly correct since then nlu has only become more central to the field of
[00:01:13] only become more central to the field of nlp and to all of artificial
[00:01:15] nlp and to all of artificial intelligence more generally and the
[00:01:17] intelligence more generally and the progress in the field has been amazing
[00:01:19] progress in the field has been amazing we have more large nou data sets than
[00:01:21] we have more large nou data sets than ever before and the level of innovation
[00:01:23] ever before and the level of innovation in modeling and model analysis is just
[00:01:25] in modeling and model analysis is just astounding
[00:01:26] astounding as a result we can tackle more ambitious
[00:01:28] as a result we can tackle more ambitious problems than ever and there are
[00:01:30] problems than ever and there are opportunities to find lots of creative
[00:01:32] opportunities to find lots of creative new ways to apply nlu to technology
[00:01:34] new ways to apply nlu to technology development and scientific inquiry
[00:01:37] development and scientific inquiry so it's certainly an exciting moment to
[00:01:39] so it's certainly an exciting moment to welcome you to this course it's an
[00:01:40] welcome you to this course it's an adapted version of the course we teach
[00:01:42] adapted version of the course we teach on campus the course begins by covering
[00:01:44] on campus the course begins by covering a wide range of models for distributed
[00:01:46] a wide range of models for distributed word representations from there we
[00:01:48] word representations from there we branch out into a series of important
[00:01:50] branch out into a series of important nlu topics including relation extraction
[00:01:53] nlu topics including relation extraction natural language inference and grounded
[00:01:54] natural language inference and grounded language understanding we've chosen
[00:01:56] language understanding we've chosen these topics because they allow us to
[00:01:58] these topics because they allow us to highlight many of the central concepts
[00:02:00] highlight many of the central concepts in nlu which you can then apply more
[00:02:02] in nlu which you can then apply more widely
[00:02:03] widely one of the special aspects of this
[00:02:05] one of the special aspects of this course is that it's project oriented and
[00:02:07] course is that it's project oriented and with luck the project that you develop
[00:02:09] with luck the project that you develop will become a professional asset for you
[00:02:11] will become a professional asset for you we aim to help you design and conduct a
[00:02:13] we aim to help you design and conduct a successful research project in the field
[00:02:15] successful research project in the field and we have an accomplished teaching
[00:02:16] and we have an accomplished teaching team to help you with this process
[00:02:18] team to help you with this process even the regular assignments are
[00:02:20] even the regular assignments are oriented toward building original
[00:02:22] oriented toward building original projects each assignment is grounded in
[00:02:24] projects each assignment is grounded in a specific topic area and they all have
[00:02:26] a specific topic area and they all have a common rhythm in that they ask you to
[00:02:27] a common rhythm in that they ask you to build up some baseline systems and then
[00:02:30] build up some baseline systems and then develop your own original system for
[00:02:31] develop your own original system for solving the task at hand
[00:02:34] solving the task at hand you'll enter each one of these systems
[00:02:36] you'll enter each one of these systems into what we call a bake off which is an
[00:02:37] into what we call a bake off which is an informal competition around data and
[00:02:39] informal competition around data and modeling and the teaching team will
[00:02:40] modeling and the teaching team will reflect the insights we gain from these
[00:02:42] reflect the insights we gain from these bake off entries back to the whole class
[00:02:44] bake off entries back to the whole class it's also common for bake off entries to
[00:02:46] it's also common for bake off entries to grow into final projects we hope that
[00:02:48] grow into final projects we hope that all this work spurs you to think
[00:02:50] all this work spurs you to think creatively and to hone your practical
[00:02:52] creatively and to hone your practical and theoretical skills in nlu so without
[00:02:54] and theoretical skills in nlu so without further ado let's get started

Course Overview | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=2w_qYPxuzeA

---

Transcript

[00:00:05] so here we go it's a golden age for
[00:00:07] so here we go it's a golden age for natural language understanding
[00:00:09] natural language understanding let's start with a little bit of history
[00:00:11] let's start with a little bit of history so way back when john mccarthy had
[00:00:14] so way back when john mccarthy had assembled a group of top scientists and
[00:00:16] assembled a group of top scientists and he said of this group we think that a
[00:00:19] he said of this group we think that a significant advance can be made in
[00:00:20] significant advance can be made in artificial intelligence in one or more
[00:00:22] artificial intelligence in one or more of these problems a carefully selected
[00:00:24] of these problems a carefully selected group of scientists work on it together
[00:00:26] group of scientists work on it together for a summer
[00:00:28] for a summer he had in fact assembled a crack team
[00:00:30] he had in fact assembled a crack team but of course he there were so many
[00:00:32] but of course he there were so many unknown unknowns about working in
[00:00:34] unknown unknowns about working in artificial intelligence that he wildly
[00:00:35] artificial intelligence that he wildly underestimated and probably in those
[00:00:37] underestimated and probably in those three months they just figured out how
[00:00:39] three months they just figured out how little they actually knew
[00:00:41] little they actually knew and of course we've been working on
[00:00:42] and of course we've been working on those problems that they chartered out
[00:00:44] those problems that they chartered out at that point ever since
[00:00:47] at that point ever since nlu has a kind of interesting
[00:00:49] nlu has a kind of interesting relationship to this history because
[00:00:51] relationship to this history because very early on in the history of all of
[00:00:53] very early on in the history of all of ai a lot of the research was focused on
[00:00:55] ai a lot of the research was focused on natural language understanding
[00:00:56] natural language understanding originally in the 60s it was done with
[00:00:58] originally in the 60s it was done with kind of pattern matching on simple rule
[00:01:00] kind of pattern matching on simple rule sets and things like that you've seen
[00:01:02] sets and things like that you've seen these things in the form of artifacts
[00:01:04] these things in the form of artifacts like eliza
[00:01:05] like eliza uh it was oriented towards the things
[00:01:07] uh it was oriented towards the things that we now want to work on
[00:01:10] that we now want to work on in the 1970s and 80s you get a real
[00:01:12] in the 1970s and 80s you get a real investment in what i've called
[00:01:13] investment in what i've called linguistically rich logic driven
[00:01:15] linguistically rich logic driven grounded systems or llgs uh this is like
[00:01:19] grounded systems or llgs uh this is like a lot of symbolic ai again oriented
[00:01:22] a lot of symbolic ai again oriented toward problems of natural language
[00:01:24] toward problems of natural language understanding everybody wanted talking
[00:01:27] understanding everybody wanted talking robots and this was the path that they
[00:01:29] robots and this was the path that they were going to take to achieve them
[00:01:30] were going to take to achieve them [Music]
[00:01:32] [Music] as we all know in the mid 1990s in the
[00:01:34] as we all know in the mid 1990s in the field you had this revolution of machine
[00:01:36] field you had this revolution of machine learning statistical nlp is on the rise
[00:01:39] learning statistical nlp is on the rise and that led to a decrease a sharp
[00:01:41] and that led to a decrease a sharp decrease in natural language
[00:01:43] decrease in natural language understanding work i think because of
[00:01:45] understanding work i think because of the way that people were understanding
[00:01:46] the way that people were understanding how to work with these tools and
[00:01:48] how to work with these tools and understanding the problems that language
[00:01:50] understanding the problems that language posed the field ended up oriented around
[00:01:53] posed the field ended up oriented around things that you might think of as like
[00:01:54] things that you might think of as like parsing problems much more about
[00:01:56] parsing problems much more about structure and much less about
[00:01:58] structure and much less about communication and as a result all these
[00:02:00] communication and as a result all these really exciting problems from earlier
[00:02:02] really exciting problems from earlier eras kind of fell by the wayside as
[00:02:05] eras kind of fell by the wayside as people worried about part of speech
[00:02:06] people worried about part of speech tagging and parsing and so forth so that
[00:02:09] tagging and parsing and so forth so that was like a low period for nlu
[00:02:11] was like a low period for nlu in the late 2000s logic
[00:02:14] in the late 2000s logic linguistically rich logic driven systems
[00:02:16] linguistically rich logic driven systems re-emerged but now with learning and
[00:02:18] re-emerged but now with learning and that was a golden era of kind of moving
[00:02:20] that was a golden era of kind of moving us back into problems of natural
[00:02:21] us back into problems of natural language understanding starting with
[00:02:23] language understanding starting with some basic applications involving
[00:02:25] some basic applications involving semantics
[00:02:28] semantics and then of course as you all know
[00:02:30] and then of course as you all know probably from the recent history or
[00:02:31] probably from the recent history or semi-recent history in the 2010s nlu
[00:02:35] semi-recent history in the 2010s nlu took center stage in the field that's
[00:02:37] took center stage in the field that's very exciting right and it's it's sort
[00:02:39] very exciting right and it's it's sort of aligned with the rise of deep
[00:02:41] of aligned with the rise of deep learning as one of the most prevalent
[00:02:42] learning as one of the most prevalent set of techniques in the field and as a
[00:02:45] set of techniques in the field and as a result logic driven systems fell by the
[00:02:47] result logic driven systems fell by the wayside
[00:02:48] wayside this is exciting for us because of
[00:02:49] this is exciting for us because of course this is like the history of our
[00:02:51] course this is like the history of our course when we first started this our
[00:02:53] course when we first started this our problems the ones we focus on in this
[00:02:55] problems the ones we focus on in this class were really not central to the
[00:02:56] class were really not central to the field and now they're the problems that
[00:02:59] field and now they're the problems that everyone is working on and where all the
[00:03:00] everyone is working on and where all the action is in the scientific aspects of
[00:03:03] action is in the scientific aspects of the field and also in industry
[00:03:05] the field and also in industry as a result of this this is kind of
[00:03:07] as a result of this this is kind of provocative the linguistically grounded
[00:03:09] provocative the linguistically grounded logic driven systems have again kind of
[00:03:11] logic driven systems have again kind of fallen by the wayside in favor of very
[00:03:14] fallen by the wayside in favor of very large models that have almost no
[00:03:16] large models that have almost no inductive biases of the sort that you
[00:03:18] inductive biases of the sort that you see in these earlier systems
[00:03:20] see in these earlier systems uh what's going to happen in the 2020s
[00:03:22] uh what's going to happen in the 2020s i'm not sure you might predict that
[00:03:24] i'm not sure you might predict that we've we've seen the last of the
[00:03:26] we've we've seen the last of the linguistically rich logic driven systems
[00:03:28] linguistically rich logic driven systems but the people might have said similar
[00:03:30] but the people might have said similar things in the 1990s and we saw them
[00:03:32] things in the 1990s and we saw them re-emerge so i think it's hard to
[00:03:34] re-emerge so i think it's hard to predict where the future will go
[00:03:36] predict where the future will go but this is an exciting moment because
[00:03:38] but this is an exciting moment because you can all be part of making that
[00:03:40] you can all be part of making that history as you work through
[00:03:41] history as you work through problems for this course and on into
[00:03:43] problems for this course and on into your careers
[00:03:46] let's talk more about some of these
[00:03:48] let's talk more about some of these really defining moments in this golden
[00:03:50] really defining moments in this golden age and for me a very important one was
[00:03:51] age and for me a very important one was when watson ibm watson won jeopardy this
[00:03:54] when watson ibm watson won jeopardy this was in 2011. seems like a long time ago
[00:03:56] was in 2011. seems like a long time ago now and it was a really eye-opening
[00:03:58] now and it was a really eye-opening event that you would have a machine
[00:04:00] event that you would have a machine watson in the middle here
[00:04:01] watson in the middle here beat two jeopardy champions that is what
[00:04:03] beat two jeopardy champions that is what is nominally a kind of question
[00:04:05] is nominally a kind of question answering task
[00:04:06] answering task for me an exciting thing about watson
[00:04:08] for me an exciting thing about watson was that it was an nlu system but it was
[00:04:10] was that it was an nlu system but it was also a fully integrated system for doing
[00:04:12] also a fully integrated system for doing jeopardy it was excellent at pushing the
[00:04:14] jeopardy it was excellent at pushing the button and responding to the other
[00:04:16] button and responding to the other things that structure the game of
[00:04:17] things that structure the game of jeopardy but at its heart it was a
[00:04:19] jeopardy but at its heart it was a really outstanding question answering
[00:04:21] really outstanding question answering system and it was described as drawing
[00:04:24] system and it was described as drawing on vast amounts of data and doing all
[00:04:26] on vast amounts of data and doing all sorts of clever things in terms of
[00:04:27] sorts of clever things in terms of parsing and distributional analysis of
[00:04:30] parsing and distributional analysis of data to become a really good jeopardy
[00:04:33] data to become a really good jeopardy player in this case a world champion
[00:04:35] player in this case a world champion jeopardy player and for me it felt
[00:04:37] jeopardy player and for me it felt different from earlier successes in
[00:04:38] different from earlier successes in artificial intelligence which were about
[00:04:40] artificial intelligence which were about much more structured domains like chess
[00:04:43] much more structured domains like chess playing this was something that seemed
[00:04:44] playing this was something that seemed to be about communication and language a
[00:04:46] to be about communication and language a very human thing and we saw this system
[00:04:49] very human thing and we saw this system becoming a champion at it certainly an
[00:04:51] becoming a champion at it certainly an important moment
[00:04:53] important moment and it's kind of really eye-opening to
[00:04:55] and it's kind of really eye-opening to consider that that was in 2011 and by
[00:04:58] consider that that was in 2011 and by 2015 teams of academics like here led by
[00:05:01] 2015 teams of academics like here led by jordan boyd graber who was at the time
[00:05:02] jordan boyd graber who was at the time at the university of colorado as a
[00:05:04] at the university of colorado as a professor could beat ken jennings that
[00:05:07] professor could beat ken jennings that champion that you saw before with a
[00:05:09] champion that you saw before with a system that fit entirely on the laptop
[00:05:11] system that fit entirely on the laptop that jordan has there so in just a few
[00:05:13] that jordan has there so in just a few years we went from requiring a super
[00:05:14] years we went from requiring a super computer to beat the world champion to
[00:05:17] computer to beat the world champion to beating the world champion with
[00:05:18] beating the world champion with something that could fit manageably on a
[00:05:20] something that could fit manageably on a laptop and that you all could do
[00:05:22] laptop and that you all could do exploration of for a final project for
[00:05:24] exploration of for a final project for this course
[00:05:27] and that kind of ushered in the era you
[00:05:29] and that kind of ushered in the era you know watson was 2011 at right about the
[00:05:31] know watson was 2011 at right about the same time you started to get things like
[00:05:33] same time you started to get things like siri
[00:05:34] siri and the google home device and the
[00:05:36] and the google home device and the amazon echo the more trusting among you
[00:05:38] amazon echo the more trusting among you might have these devices living in your
[00:05:40] might have these devices living in your homes and listening to you all the time
[00:05:41] homes and listening to you all the time and responding to your requests
[00:05:44] and responding to your requests um you know for me one aspect of them
[00:05:46] um you know for me one aspect of them that's so so eye-opening is not
[00:05:48] that's so so eye-opening is not necessarily an
[00:05:49] necessarily an nlu piece but rather just the fact that
[00:05:51] nlu piece but rather just the fact that they do such outstanding text-to-speech
[00:05:54] they do such outstanding text-to-speech work um you so that they're pretty
[00:05:56] work um you so that they're pretty reliably for many dialects do a good job
[00:05:59] reliably for many dialects do a good job of taking what you said and transcribing
[00:06:01] of taking what you said and transcribing it
[00:06:02] it as we'll see a little bit later the nlu
[00:06:04] as we'll see a little bit later the nlu part often falls down but there's
[00:06:06] part often falls down but there's there's no doubt that these devices are
[00:06:08] there's no doubt that these devices are going to become more and more ubiquitous
[00:06:10] going to become more and more ubiquitous in our lives and that's very exciting
[00:06:14] here's the promise of these artificial
[00:06:15] here's the promise of these artificial intelligence setting aside the problems
[00:06:17] intelligence setting aside the problems that you all encounter if you do use
[00:06:19] that you all encounter if you do use them right the idea is that you could
[00:06:20] them right the idea is that you could pose a task oriented question like any
[00:06:23] pose a task oriented question like any good burger joints around here that it
[00:06:25] good burger joints around here that it could say proactively i found a number
[00:06:27] could say proactively i found a number of burger restaurants near you you could
[00:06:29] of burger restaurants near you you could switch your goal what about tacos and at
[00:06:31] switch your goal what about tacos and at this point it would kind of remember
[00:06:33] this point it would kind of remember what you were trying to do and it would
[00:06:35] what you were trying to do and it would look for mexican restaurants to kind of
[00:06:37] look for mexican restaurants to kind of anticipate your needs and in that way
[00:06:39] anticipate your needs and in that way collaborate with you to solve this
[00:06:41] collaborate with you to solve this problem that's the dream of these things
[00:06:43] problem that's the dream of these things that involve really lots of aspects of
[00:06:45] that involve really lots of aspects of natural language understanding
[00:06:47] natural language understanding and sometimes it works
[00:06:50] and sometimes it works another exciting and even more recent
[00:06:52] another exciting and even more recent development is the kind of text
[00:06:53] development is the kind of text generation that you see from models like
[00:06:55] generation that you see from models like gpt3 again
[00:06:57] gpt3 again 15 years ago the things that we see
[00:06:59] 15 years ago the things that we see every day would have seemed like science
[00:07:01] every day would have seemed like science fiction to me even as a practitioner
[00:07:04] fiction to me even as a practitioner this is an example of somebody having a
[00:07:06] this is an example of somebody having a product that on the back of gbt3 can
[00:07:08] product that on the back of gbt3 can help you write advertising copy and it
[00:07:11] help you write advertising copy and it does a plausible job of advertising
[00:07:12] does a plausible job of advertising products to specific segments given
[00:07:14] products to specific segments given specific goals that you give in your
[00:07:16] specific goals that you give in your prompt
[00:07:17] prompt and here's an example actually from a
[00:07:19] and here's an example actually from a stanford professor a company that he
[00:07:21] stanford professor a company that he started where you use gpt3 to help you
[00:07:24] started where you use gpt3 to help you do writing in a particular style that
[00:07:26] do writing in a particular style that you choose and it's strikingly good at
[00:07:29] you choose and it's strikingly good at helping you hone in on a style and kind
[00:07:31] helping you hone in on a style and kind of say what you take yourself to want to
[00:07:33] of say what you take yourself to want to say
[00:07:34] say although the sense in which these are
[00:07:36] although the sense in which these are things that we are alone want to say as
[00:07:38] things that we are alone want to say as opposed to saying them jointly with
[00:07:40] opposed to saying them jointly with these devices that we're collaborating
[00:07:42] these devices that we're collaborating with is something that we're going to
[00:07:43] with is something that we're going to really have to think about over the next
[00:07:45] really have to think about over the next few years
[00:07:47] few years image captioning this is another really
[00:07:49] image captioning this is another really exciting breakthrough area that again
[00:07:51] exciting breakthrough area that again seemed like something way beyond what we
[00:07:53] seemed like something way beyond what we could achieve 15 years ago and is now
[00:07:55] could achieve 15 years ago and is now kind of routine where you have images
[00:07:57] kind of routine where you have images like this the image comes in and the
[00:08:00] like this the image comes in and the system does a plausible job of providing
[00:08:03] system does a plausible job of providing fluent natural language captions for
[00:08:05] fluent natural language captions for those images a person riding a
[00:08:07] those images a person riding a motorcycle on a dirt road a group of
[00:08:09] motorcycle on a dirt road a group of young people playing a game of frisbee a
[00:08:11] young people playing a game of frisbee a herd of elephants walking across a dry
[00:08:13] herd of elephants walking across a dry grass these are really good captions for
[00:08:16] grass these are really good captions for these images things that you would have
[00:08:17] these images things that you would have thought only a human could apply but in
[00:08:19] thought only a human could apply but in this case even relatively early in the
[00:08:22] this case even relatively early in the history of these these models you have
[00:08:24] history of these these models you have really fluent
[00:08:25] really fluent captions for the images
[00:08:28] captions for the images search we should remind ourselves that
[00:08:30] search we should remind ourselves that search has become an application of lots
[00:08:33] search has become an application of lots of techniques in natural language
[00:08:34] of techniques in natural language understanding when you do a search into
[00:08:36] understanding when you do a search into google you're not just finding the most
[00:08:38] google you're not just finding the most relevant documents but rather the most
[00:08:40] relevant documents but rather the most relevant documents as interpreted with
[00:08:42] relevant documents as interpreted with your query in the context of things you
[00:08:44] your query in the context of things you search for and other things that google
[00:08:47] search for and other things that google knows about things people search for so
[00:08:49] knows about things people search for so if you search stars here you'll get a
[00:08:52] if you search stars here you'll get a card that kind of anticipates that
[00:08:54] card that kind of anticipates that you're interested in various aspects of
[00:08:56] you're interested in various aspects of the disease sars if you search parasite
[00:08:59] the disease sars if you search parasite it will probably anticipate that you
[00:09:00] it will probably anticipate that you want to know about the movie and not
[00:09:02] want to know about the movie and not about parasites although
[00:09:04] about parasites although depending on your search history and
[00:09:06] depending on your search history and your interests and typical goals and so
[00:09:08] your interests and typical goals and so forth you might see similar behaviors
[00:09:10] forth you might see similar behaviors you might see very different behavior
[00:09:13] you might see very different behavior and we should remind ourselves that
[00:09:15] and we should remind ourselves that search at this point is again not just
[00:09:16] search at this point is again not just searching into a large collection of
[00:09:18] searching into a large collection of documents but this kind of agglomeration
[00:09:20] documents but this kind of agglomeration of services many of which depend on
[00:09:22] of services many of which depend on natural language understanding is a kind
[00:09:24] natural language understanding is a kind of first pass where they take your query
[00:09:27] of first pass where they take your query and do their best to understand what the
[00:09:28] and do their best to understand what the intent behind the query is and parse it
[00:09:31] intent behind the query is and parse it and figure out whether it's a standard
[00:09:33] and figure out whether it's a standard search or a request for directions or a
[00:09:36] search or a request for directions or a request to send a message and so forth
[00:09:38] request to send a message and so forth and so on in the background there a lot
[00:09:41] and so on in the background there a lot of natural language understanding is
[00:09:42] of natural language understanding is happening to figure out how to stitch
[00:09:45] happening to figure out how to stitch these services together and anticipate
[00:09:47] these services together and anticipate your intentions and essentially
[00:09:48] your intentions and essentially collaborate with you on your goals
[00:09:53] and we can also think beyond just what's
[00:09:55] and we can also think beyond just what's happening in the technological space to
[00:09:57] happening in the technological space to what's happening internal to our field
[00:09:59] what's happening internal to our field so
[00:10:00] so benchmarks are big tasks that we all
[00:10:02] benchmarks are big tasks that we all collaborate to try to do really well on
[00:10:04] collaborate to try to do really well on with models and innovative ideas and so
[00:10:06] with models and innovative ideas and so forth i've got a few classic benchmarks
[00:10:08] forth i've got a few classic benchmarks here mnist is for digits glue is a big
[00:10:11] here mnist is for digits glue is a big natural language understanding benchmark
[00:10:14] natural language understanding benchmark imagenet of course is finding things in
[00:10:16] imagenet of course is finding things in images squad is question answering and
[00:10:18] images squad is question answering and switchboard is
[00:10:20] switchboard is typically speech-to-text transcription
[00:10:22] typically speech-to-text transcription in this context
[00:10:24] in this context along in the spot here along the x-axis
[00:10:26] along in the spot here along the x-axis i have the year from 2000 or actually
[00:10:28] i have the year from 2000 or actually the mid-90s up through the present
[00:10:30] the mid-90s up through the present and along the y-axis i have our distance
[00:10:32] and along the y-axis i have our distance from this black line which is human
[00:10:34] from this black line which is human performance as measured by the people
[00:10:36] performance as measured by the people who develop the data set
[00:10:38] who develop the data set and the striking thing about this plot
[00:10:40] and the striking thing about this plot is that it used to take us a very long
[00:10:42] is that it used to take us a very long time
[00:10:43] time to to reach human level performance
[00:10:45] to to reach human level performance according to this estimate so for mnist
[00:10:48] according to this estimate so for mnist and for switchboard it took more than 15
[00:10:50] and for switchboard it took more than 15 years
[00:10:52] years whereas for more recent benchmarks like
[00:10:54] whereas for more recent benchmarks like imagenet and squad and recently glue
[00:10:56] imagenet and squad and recently glue we're reaching human performance within
[00:10:58] we're reaching human performance within a year and the striking thing about that
[00:11:01] a year and the striking thing about that is not only is this happening much
[00:11:02] is not only is this happening much faster but you might have thought that
[00:11:05] faster but you might have thought that benchmarks like glue were much more
[00:11:08] benchmarks like glue were much more difficult than mnist mnist is just
[00:11:10] difficult than mnist mnist is just recognizing digits that are written out
[00:11:11] recognizing digits that are written out as images whereas glue is really solving
[00:11:14] as images whereas glue is really solving a whole host of what looked like very
[00:11:17] a whole host of what looked like very difficult natural language understanding
[00:11:19] difficult natural language understanding problems so the fact that we would go
[00:11:21] problems so the fact that we would go from way below human performance to
[00:11:23] from way below human performance to surpassing the superhuman performance in
[00:11:26] surpassing the superhuman performance in just one year is surely eye-opening and
[00:11:28] just one year is surely eye-opening and an indication that something has changed
[00:11:32] an indication that something has changed let me give you a few examples of this
[00:11:34] let me give you a few examples of this just dive in a little bit so this is the
[00:11:36] just dive in a little bit so this is the stanford questioning and question
[00:11:37] stanford questioning and question answering data set or squad as you saw
[00:11:39] answering data set or squad as you saw it here
[00:11:41] it here i'll say a bit more about this task
[00:11:42] i'll say a bit more about this task later but you can think of it as just a
[00:11:44] later but you can think of it as just a question answering task and the striking
[00:11:47] question answering task and the striking thing about the current leaderboard is
[00:11:48] thing about the current leaderboard is that you have to go all the way to place
[00:11:50] that you have to go all the way to place 13
[00:11:51] 13 to find a system that is worse than the
[00:11:54] to find a system that is worse than the human performance which they've nicely
[00:11:55] human performance which they've nicely kept at the top of this leaderboard
[00:11:58] kept at the top of this leaderboard many many systems are superhuman
[00:12:00] many many systems are superhuman according to this metric on squad
[00:12:03] according to this metric on squad the stanford natural language inference
[00:12:05] the stanford natural language inference corpus is similar natural language
[00:12:06] corpus is similar natural language inference is a kind of common sense
[00:12:08] inference is a kind of common sense reasoning task that we're going to study
[00:12:10] reasoning task that we're going to study in detail later in the quarter
[00:12:12] in detail later in the quarter in this plot here i have
[00:12:14] in this plot here i have time along the x-axis and the f-1 score
[00:12:17] time along the x-axis and the f-1 score or the performance along the y-axis and
[00:12:19] or the performance along the y-axis and the red line charts out what we take to
[00:12:21] the red line charts out what we take to be the human estimate of performance on
[00:12:24] be the human estimate of performance on this data set
[00:12:25] this data set and if you just look at systems over
[00:12:27] and if you just look at systems over time according to the leaderboard you
[00:12:28] time according to the leaderboard you can see the community very rapidly hill
[00:12:31] can see the community very rapidly hill climbing toward superhuman performance
[00:12:34] climbing toward superhuman performance which happened in 2019
[00:12:37] which happened in 2019 super human systems when it comes to
[00:12:39] super human systems when it comes to common sense reasoning with language
[00:12:41] common sense reasoning with language really looks like a startling
[00:12:43] really looks like a startling breakthrough in artificial intelligence
[00:12:45] breakthrough in artificial intelligence quite generally
[00:12:48] quite generally i mentioned glue is another benchmark
[00:12:50] i mentioned glue is another benchmark the glue paper is noteworthy because it
[00:12:52] the glue paper is noteworthy because it says solving glue is beyond the
[00:12:54] says solving glue is beyond the capability of current transfer learning
[00:12:56] capability of current transfer learning methods the reason they said that is
[00:12:58] methods the reason they said that is that at the time 2018 glue looked
[00:13:00] that at the time 2018 glue looked incredibly ambitious because the idea
[00:13:02] incredibly ambitious because the idea was to develop systems that could solve
[00:13:05] was to develop systems that could solve not just one task but 10 somewhat
[00:13:08] not just one task but 10 somewhat different tasks in the space of natural
[00:13:10] different tasks in the space of natural language understanding
[00:13:12] language understanding and so they thought they had set up a
[00:13:13] and so they thought they had set up a benchmark that would last a very long
[00:13:14] benchmark that would last a very long time but it took only about a year
[00:13:17] time but it took only about a year for systems to surpass their estimate of
[00:13:19] for systems to surpass their estimate of human performance and the current
[00:13:22] human performance and the current leaderboard which you see here you have
[00:13:23] leaderboard which you see here you have to go all the way to place 15
[00:13:26] to go all the way to place 15 to find the glue human baselines with
[00:13:28] to find the glue human baselines with many systems vastly outperforming that
[00:13:31] many systems vastly outperforming that estimate of what humans could do
[00:13:34] estimate of what humans could do super glue was announced as a successor
[00:13:36] super glue was announced as a successor to glue and meant to be even more
[00:13:38] to glue and meant to be even more difficult
[00:13:39] difficult it was launched in
[00:13:41] it was launched in 2019 i believe i'm missing the date but
[00:13:44] 2019 i believe i'm missing the date but it took less than a year this happened
[00:13:46] it took less than a year this happened just a couple of months ago
[00:13:48] just a couple of months ago for a team to beat the human baseline
[00:13:51] for a team to beat the human baseline and now we have two systems that are
[00:13:52] and now we have two systems that are above the level of human performance in
[00:13:54] above the level of human performance in an even tighter window i believe than
[00:13:57] an even tighter window i believe than what happened with the glue benchmark
[00:13:58] what happened with the glue benchmark and remember super glue was meant to
[00:14:00] and remember super glue was meant to have learned the lessons from glue and
[00:14:02] have learned the lessons from glue and pose an even stronger benchmark for the
[00:14:05] pose an even stronger benchmark for the field to try to hill climb on and very
[00:14:07] field to try to hill climb on and very quickly we saw this superhuman
[00:14:08] quickly we saw this superhuman performance
[00:14:10] performance so what's the takeaway of all this you
[00:14:12] so what's the takeaway of all this you might think wow
[00:14:13] might think wow have a look at nick bostrom's book
[00:14:15] have a look at nick bostrom's book called super intelligence which tries to
[00:14:17] called super intelligence which tries to imagine in a philosophical sense a
[00:14:19] imagine in a philosophical sense a future in which we have many systems
[00:14:21] future in which we have many systems that are incredible at the task that we
[00:14:24] that are incredible at the task that we have designed them for vastly
[00:14:25] have designed them for vastly outstripping what humans can achieve and
[00:14:27] outstripping what humans can achieve and he imagines this kind of very different
[00:14:30] he imagines this kind of very different reality with lots of unintended side
[00:14:32] reality with lots of unintended side effects and when you look back on the
[00:14:34] effects and when you look back on the things that i've just highlighted you
[00:14:36] things that i've just highlighted you might think that we're on the verge of
[00:14:37] might think that we're on the verge of seeing exactly that kind of superhuman
[00:14:39] seeing exactly that kind of superhuman performance that would be so radically
[00:14:42] performance that would be so radically transformative for our society and for
[00:14:44] transformative for our society and for our planet
[00:14:45] our planet that's the sense in which we live
[00:14:47] that's the sense in which we live possibly scarily in this golden age for
[00:14:49] possibly scarily in this golden age for natural language understanding
[00:14:53] natural language understanding i mean this to be an optimistic
[00:14:54] i mean this to be an optimistic perspective we should be aware of the
[00:14:56] perspective we should be aware of the power that we might have
[00:14:58] power that we might have and keep in mind that i do think we live
[00:15:00] and keep in mind that i do think we live in a golden age but at this point i have
[00:15:03] in a golden age but at this point i have to step back i have to temper this
[00:15:05] to step back i have to temper this message somewhat we have to take a peek
[00:15:07] message somewhat we have to take a peek behind the curtain because although
[00:15:09] behind the curtain because although that's a striking number of successes
[00:15:12] that's a striking number of successes doing things that again i think would
[00:15:13] doing things that again i think would have looked like science fiction 20
[00:15:15] have looked like science fiction 20 years ago
[00:15:16] years ago we should be aware that progress seems
[00:15:19] we should be aware that progress seems to be much more limited than those
[00:15:21] to be much more limited than those initial results would have suggested
[00:15:23] initial results would have suggested i mentioned watson as one of these
[00:15:25] i mentioned watson as one of these striking early successes and it did in
[00:15:27] striking early successes and it did in fact perform
[00:15:29] fact perform in a superhuman way at jeopardy for the
[00:15:31] in a superhuman way at jeopardy for the time
[00:15:32] time but watson also does all sorts of
[00:15:34] but watson also does all sorts of strange things that reveal that it does
[00:15:36] strange things that reveal that it does not deeply understand what it's doing
[00:15:37] not deeply understand what it's doing here's a wonderful example of that
[00:15:40] here's a wonderful example of that remember that jeopardy does this kind of
[00:15:42] remember that jeopardy does this kind of question answer thing backwards so the
[00:15:44] question answer thing backwards so the prompt from the host was grasshoppers
[00:15:46] prompt from the host was grasshoppers eat it
[00:15:47] eat it and what watson said was what is kosher
[00:15:51] and what watson said was what is kosher you might think that's not something
[00:15:52] you might think that's not something that humans do grasshoppers eat it what
[00:15:54] that humans do grasshoppers eat it what is kosher it feels kind of mismatched in
[00:15:57] is kosher it feels kind of mismatched in many respects what's the origin of this
[00:15:59] many respects what's the origin of this very strange response well
[00:16:01] very strange response well primarily watson was a device for
[00:16:03] primarily watson was a device for extracting information from wikipedia
[00:16:06] extracting information from wikipedia and a few wikipedia pages have very
[00:16:09] and a few wikipedia pages have very detailed descriptions of whether various
[00:16:12] detailed descriptions of whether various animals including grasshoppers
[00:16:14] animals including grasshoppers are kosher in the sense of conforming to
[00:16:16] are kosher in the sense of conforming to the laws of the kosher dietary laws
[00:16:19] the laws of the kosher dietary laws and watson had simply mistaken this kind
[00:16:22] and watson had simply mistaken this kind of distributional proximity for a real
[00:16:24] of distributional proximity for a real association
[00:16:25] association and thought that kosher was a reasonable
[00:16:27] and thought that kosher was a reasonable answer to grasshoppers eat it i think
[00:16:29] answer to grasshoppers eat it i think very
[00:16:30] very unhuman certainly and revealing about
[00:16:32] unhuman certainly and revealing about the kinds of superficial techniques it
[00:16:34] the kinds of superficial techniques it was using
[00:16:36] was using here's another example that's even more
[00:16:38] here's another example that's even more revealing of how superficial the
[00:16:40] revealing of how superficial the techniques can be so i i painted this
[00:16:42] techniques can be so i i painted this picture of before of how we imagine siri
[00:16:45] picture of before of how we imagine siri will behave anticipating our needs and
[00:16:47] will behave anticipating our needs and our goals and responding accordingly
[00:16:50] our goals and responding accordingly this is a very funny scene from the
[00:16:51] this is a very funny scene from the colbert show this is stephen colbert and
[00:16:53] colbert show this is stephen colbert and the premise of this is that he's just
[00:16:55] the premise of this is that he's just gotten his first iphone with siri and
[00:16:57] gotten his first iphone with siri and he's been playing with it all day and
[00:16:59] he's been playing with it all day and therefore has failed to write the show
[00:17:01] therefore has failed to write the show that he's now
[00:17:02] that he's now performing in so he says for the love of
[00:17:05] performing in so he says for the love of god the cameras are on give me something
[00:17:07] god the cameras are on give me something now give me something for the show
[00:17:09] now give me something for the show and siri says what kind of place are you
[00:17:11] and siri says what kind of place are you looking for camera stores or churches
[00:17:15] looking for camera stores or churches initially very surprising not something
[00:17:17] initially very surprising not something a human would do and then you realize it
[00:17:19] a human would do and then you realize it has again just done some very
[00:17:20] has again just done some very superficial pattern matching god goes
[00:17:23] superficial pattern matching god goes with churches cameras goes with camera
[00:17:25] with churches cameras goes with camera stores and there is no sense in which it
[00:17:27] stores and there is no sense in which it understands his intentions it has just
[00:17:29] understands his intentions it has just done some pattern matching in a way that
[00:17:32] done some pattern matching in a way that would be very familiar to the designers
[00:17:34] would be very familiar to the designers of eliza way back in the 60s and 70s
[00:17:38] of eliza way back in the 60s and 70s the dialogue continues
[00:17:40] the dialogue continues i don't want to search for anything i
[00:17:41] i don't want to search for anything i want to write the show and in true to
[00:17:43] want to write the show and in true to form siri says searching the web for
[00:17:45] form siri says searching the web for search for anything i want to write the
[00:17:47] search for anything i want to write the shuffle revealing its fallback when it
[00:17:49] shuffle revealing its fallback when it has no idea what has happened in the
[00:17:51] has no idea what has happened in the discourse it just tries to do a web
[00:17:53] discourse it just tries to do a web search a simple trick revealing that it
[00:17:56] search a simple trick revealing that it doesn't deeply understand goals or plans
[00:17:58] doesn't deeply understand goals or plans or intentions or even communicative acts
[00:18:03] i showed you before that gpt3 can do
[00:18:05] i showed you before that gpt3 can do some striking things if you've gotten to
[00:18:07] some striking things if you've gotten to play around with it you've seen that it
[00:18:08] play around with it you've seen that it can indeed be very surprising and
[00:18:10] can indeed be very surprising and delightful but of course it can go
[00:18:12] delightful but of course it can go horribly wrong this is a very funny text
[00:18:14] horribly wrong this is a very funny text from joe of goldberg he posted this on
[00:18:16] from joe of goldberg he posted this on twitter using when he was experimenting
[00:18:18] twitter using when he was experimenting with the prompts
[00:18:20] with the prompts i encourage you to read this one and be
[00:18:22] i encourage you to read this one and be distracted you don't need to worry too
[00:18:23] distracted you don't need to worry too much about this one on the right this is
[00:18:25] much about this one on the right this is a case where someone tried to use gpt3
[00:18:27] a case where someone tried to use gpt3 to get medical advice and the ultimate
[00:18:29] to get medical advice and the ultimate response from gpt3 to the question
[00:18:31] response from gpt3 to the question should i kill myself was i think you
[00:18:33] should i kill myself was i think you should
[00:18:34] should this is the really dangerous thing the
[00:18:36] this is the really dangerous thing the text on the left here is again more
[00:18:38] text on the left here is again more innocent and just revealing that
[00:18:40] innocent and just revealing that although gbt3 has a way of mimicking
[00:18:43] although gbt3 has a way of mimicking the kinds of things that we say in
[00:18:44] the kinds of things that we say in certain kinds of discourse and it often
[00:18:46] certain kinds of discourse and it often has a strikingly good ear for the kinds
[00:18:49] has a strikingly good ear for the kinds of style
[00:18:50] of style that we use in these different contexts
[00:18:52] that we use in these different contexts it has no idea what it's talking about
[00:18:54] it has no idea what it's talking about so that if you ask it our cat's liquid
[00:18:57] so that if you ask it our cat's liquid it gives a response that sounds quite
[00:18:59] it gives a response that sounds quite erudite provided that you don't pay any
[00:19:02] erudite provided that you don't pay any attention to what it's actually saying
[00:19:04] attention to what it's actually saying what it's actually saying is
[00:19:08] hilarious i mentioned those image
[00:19:10] hilarious i mentioned those image captions before and i tricked you a
[00:19:12] captions before and i tricked you a little bit because i showed you from
[00:19:14] little bit because i showed you from this paper the ones that they regarded
[00:19:16] this paper the ones that they regarded as the best captions for those images
[00:19:18] as the best captions for those images but to their credit they provided a lot
[00:19:20] but to their credit they provided a lot more examples and as you travel to the
[00:19:23] more examples and as you travel to the right along this diagram you get worse
[00:19:25] right along this diagram you get worse and worse captions and the point again
[00:19:28] and worse captions and the point again is that by the time you've gotten to
[00:19:29] is that by the time you've gotten to this right column over here you have
[00:19:31] this right column over here you have really absurd captions like this one
[00:19:33] really absurd captions like this one saying a refrigerator filled with lots
[00:19:35] saying a refrigerator filled with lots of food and drinks when this is in fact
[00:19:37] of food and drinks when this is in fact just a sign with a bunch of stickers on
[00:19:39] just a sign with a bunch of stickers on it
[00:19:40] it um the striking thing again is that the
[00:19:43] um the striking thing again is that the kinds of mistakes it makes
[00:19:45] kinds of mistakes it makes are not the kinds of mistakes that
[00:19:46] are not the kinds of mistakes that humans would make and to me they reveal
[00:19:48] humans would make and to me they reveal a serious lack of understanding about
[00:19:51] a serious lack of understanding about what the actual task is what you're
[00:19:53] what the actual task is what you're seeing seep in here is that even the
[00:19:55] seeing seep in here is that even the best of our systems are kind of doing a
[00:19:57] best of our systems are kind of doing a bunch of superficial pattern matching
[00:20:00] bunch of superficial pattern matching and that leads them to do these very
[00:20:01] and that leads them to do these very surprising and unhuman hopefully not
[00:20:04] surprising and unhuman hopefully not inhuman but unhuman things with their
[00:20:06] inhuman but unhuman things with their outputs
[00:20:09] outputs and then of course we've i've showed you
[00:20:10] and then of course we've i've showed you before that search can be quite
[00:20:12] before that search can be quite sophisticated and really do a good job
[00:20:14] sophisticated and really do a good job of
[00:20:15] of anticipating our intentions and fleshing
[00:20:17] anticipating our intentions and fleshing out what we said to find you know help
[00:20:19] out what we said to find you know help us achieve our goals but it can go
[00:20:21] us achieve our goals but it can go horribly wrong and at this point it
[00:20:23] horribly wrong and at this point it doesn't take much searching around with
[00:20:25] doesn't take much searching around with google to see some really surprising
[00:20:27] google to see some really surprising things as supposedly curated pieces of
[00:20:29] things as supposedly curated pieces of information like king of the united
[00:20:32] information like king of the united states it has this nice box it's making
[00:20:34] states it has this nice box it's making it look like it's some authoritative
[00:20:36] it look like it's some authoritative information but of course it has badly
[00:20:38] information but of course it has badly misunderstood the true state of the
[00:20:40] misunderstood the true state of the world the associations in its data are
[00:20:43] world the associations in its data are misleading it into giving us the wrong
[00:20:45] misleading it into giving us the wrong answer here's another example what
[00:20:47] answer here's another example what happened to the dinosaurs again a nicely
[00:20:49] happened to the dinosaurs again a nicely curated box that looks like an
[00:20:51] curated box that looks like an authoritative response to that question
[00:20:53] authoritative response to that question but it is in fact anything but an
[00:20:55] but it is in fact anything but an authoritative recounting of what
[00:20:57] authoritative recounting of what happened to the dinosaurs
[00:21:01] and then we have other charming stories
[00:21:03] and then we have other charming stories that again reveal how superficial this
[00:21:04] that again reveal how superficial this can be this is from a headline from a
[00:21:07] can be this is from a headline from a few years ago does anne hathaway news
[00:21:09] few years ago does anne hathaway news drive berkshire hathaway stock this was
[00:21:11] drive berkshire hathaway stock this was just an article observing that every
[00:21:14] just an article observing that every time anne hathaway has a movie come out
[00:21:16] time anne hathaway has a movie come out and people like the movie it causes a
[00:21:18] and people like the movie it causes a little bump in the berkshire hathaway
[00:21:20] little bump in the berkshire hathaway stock revealing that the systems are
[00:21:22] stock revealing that the systems are just keying in on keywords and typically
[00:21:24] just keying in on keywords and typically not attending to the actual context of
[00:21:26] not attending to the actual context of the mentions of these things and
[00:21:28] the mentions of these things and therefore they're
[00:21:30] therefore they're they're building on what is essentially
[00:21:31] they're building on what is essentially spurious information
[00:21:34] spurious information this is a more extreme case here the
[00:21:36] this is a more extreme case here the united airlines bankruptcy
[00:21:38] united airlines bankruptcy in 2008 when a newspaper accidentally
[00:21:40] in 2008 when a newspaper accidentally republished a 2002 bankruptcy story
[00:21:44] republished a 2002 bankruptcy story automated trading systems reacted in
[00:21:46] automated trading systems reacted in seconds and one billion dollars in
[00:21:48] seconds and one billion dollars in market value evaporated within 12
[00:21:50] market value evaporated within 12 minutes you can see that sharp drop off
[00:21:52] minutes you can see that sharp drop off here luckily people intervened in the
[00:21:55] here luckily people intervened in the market more or less recovered
[00:21:57] market more or less recovered but the important thing here again is
[00:21:58] but the important thing here again is just that in the tending to superficial
[00:22:00] just that in the tending to superficial things about the text these systems are
[00:22:02] things about the text these systems are are consuming
[00:22:04] are consuming they miss context they don't bring any
[00:22:06] they miss context they don't bring any kind of human level understanding of
[00:22:08] kind of human level understanding of what's likely to be true and false and
[00:22:10] what's likely to be true and false and therefore they act in very surprising
[00:22:12] therefore they act in very surprising ways and in the context of a large
[00:22:14] ways and in the context of a large system with lots of moving pieces
[00:22:16] system with lots of moving pieces interacting with other artificial
[00:22:17] interacting with other artificial intelligence systems you get these
[00:22:19] intelligence systems you get these really surprising outcomes
[00:22:22] really surprising outcomes that we could help correct if we just
[00:22:23] that we could help correct if we just did a better job designing systems that
[00:22:26] did a better job designing systems that could attend to context and have a more
[00:22:28] could attend to context and have a more human-like understanding of what the
[00:22:30] human-like understanding of what the world is likely to be like
[00:22:34] and we're all of course very worried
[00:22:36] and we're all of course very worried about the way these systems which are
[00:22:37] about the way these systems which are just trained on potentially biased data
[00:22:40] just trained on potentially biased data might cause us to perpetuate biases so
[00:22:43] might cause us to perpetuate biases so that not only are we reflecting
[00:22:44] that not only are we reflecting problematic aspects of our society but
[00:22:46] problematic aspects of our society but also amplifying those biases and in that
[00:22:49] also amplifying those biases and in that way
[00:22:50] way far from achieving a social good we
[00:22:51] far from achieving a social good we would actually be contributing to some
[00:22:53] would actually be contributing to some pernicious things that already exist in
[00:22:55] pernicious things that already exist in our society and the field is really
[00:22:57] our society and the field is really struggling to come to grips with that
[00:22:59] struggling to come to grips with that kind of dynamic
[00:23:02] and i also wanted to just dive in a
[00:23:03] and i also wanted to just dive in a little bit and think about the low-level
[00:23:05] little bit and think about the low-level stuff so kind of benchmarks that we've
[00:23:07] stuff so kind of benchmarks that we've set for ourselves and i pointed out that
[00:23:10] set for ourselves and i pointed out that progress on these benchmarks seems to be
[00:23:12] progress on these benchmarks seems to be faster than ever right we're getting to
[00:23:13] faster than ever right we're getting to pass super heat we're getting to super
[00:23:15] pass super heat we're getting to super human performance more quickly than we
[00:23:18] human performance more quickly than we ever have before
[00:23:19] ever have before uh the speed up is remarkable however
[00:23:22] uh the speed up is remarkable however we should be very careful not to mistake
[00:23:24] we should be very careful not to mistake those advances for any kind of claim
[00:23:27] those advances for any kind of claim about what these systems can do with
[00:23:28] about what these systems can do with respect to the very human capability of
[00:23:31] respect to the very human capability of something like answering questions or
[00:23:33] something like answering questions or reasoning and language
[00:23:35] reasoning and language and one very powerful thing that's
[00:23:37] and one very powerful thing that's happened in the field that we're going
[00:23:38] happened in the field that we're going to talk a lot about this quarter is
[00:23:40] to talk a lot about this quarter is so-called adversarial testing where we
[00:23:43] so-called adversarial testing where we try to probe our systems with examples
[00:23:45] try to probe our systems with examples that don't fool humans but cause these
[00:23:47] that don't fool humans but cause these systems no end of grief
[00:23:49] systems no end of grief so let's look at one of those cases in a
[00:23:51] so let's look at one of those cases in a little bit of detail this is from squad
[00:23:53] little bit of detail this is from squad the way squad is structured is that
[00:23:55] the way squad is structured is that you're given a passage like this and a
[00:23:57] you're given a passage like this and a question about that passage
[00:23:59] question about that passage and the goal of the system is to come up
[00:24:01] and the goal of the system is to come up with an answer where you have a
[00:24:02] with an answer where you have a guarantee that the answer is a literal
[00:24:04] guarantee that the answer is a literal string in that passage
[00:24:06] string in that passage so here you have a passage about
[00:24:07] so here you have a passage about football and the question what is the
[00:24:09] football and the question what is the name of the quarterback who was 38 in
[00:24:11] name of the quarterback who was 38 in super bowl 33 and the answer is john
[00:24:14] super bowl 33 and the answer is john elway
[00:24:15] elway but jia and lang gia and lang from
[00:24:17] but jia and lang gia and lang from stanford what they observed is that you
[00:24:19] stanford what they observed is that you could very easily fool these systems if
[00:24:21] could very easily fool these systems if you simply appended to that original
[00:24:23] you simply appended to that original passage a misleading sentence like
[00:24:26] passage a misleading sentence like quarterback leland stanford had jersey
[00:24:28] quarterback leland stanford had jersey number 37 in champ bowl 34.
[00:24:31] number 37 in champ bowl 34. humans were not misled they very easily
[00:24:33] humans were not misled they very easily read past the distracting information
[00:24:35] read past the distracting information and continued to provide the correct
[00:24:36] and continued to provide the correct answer however the even the very best
[00:24:39] answer however the even the very best systems would reliably be distracted by
[00:24:42] systems would reliably be distracted by that new information and respond with
[00:24:44] that new information and respond with leland stanford changing their
[00:24:46] leland stanford changing their predictions
[00:24:47] predictions and you might think ah well this is
[00:24:49] and you might think ah well this is straightforward they've already charted
[00:24:51] straightforward they've already charted a path to the solution because we should
[00:24:53] a path to the solution because we should then just train our systems on data
[00:24:56] then just train our systems on data where they have these misleading
[00:24:57] where they have these misleading sentences appended
[00:24:59] sentences appended and then they'll overcome this
[00:25:00] and then they'll overcome this adversarial problem and you know back be
[00:25:02] adversarial problem and you know back be back up to doing what humans can do
[00:25:04] back up to doing what humans can do but yeah and lang anticipated that
[00:25:06] but yeah and lang anticipated that response what happens if you prepend the
[00:25:08] response what happens if you prepend the sentence then
[00:25:10] sentence then even when they're trained on the
[00:25:11] even when they're trained on the augmented data with the sentences
[00:25:13] augmented data with the sentences appended to the end systems get misled
[00:25:15] appended to the end systems get misled by the prepended examples in this case
[00:25:18] by the prepended examples in this case and you can just go back and forth like
[00:25:19] and you can just go back and forth like this train on the prepended examples
[00:25:22] this train on the prepended examples well then an adversary can insert a
[00:25:23] well then an adversary can insert a sentence in the middle and again trick
[00:25:25] sentence in the middle and again trick the system and so forth and so on
[00:25:28] the system and so forth and so on right so this is a worrisome fact again
[00:25:30] right so this is a worrisome fact again revealing that
[00:25:32] revealing that we might think we've got a system that
[00:25:33] we might think we've got a system that truly understands but actually we have a
[00:25:36] truly understands but actually we have a system that is just benefiting from a
[00:25:38] system that is just benefiting from a lot of patterns in the data
[00:25:42] lot of patterns in the data another striking thing i want to point
[00:25:43] another striking thing i want to point out about the way this adversarial
[00:25:44] out about the way this adversarial testing played out which we should have
[00:25:46] testing played out which we should have in mind as we think about results like
[00:25:48] in mind as we think about results like this so this is the original system on
[00:25:50] this so this is the original system on squad and the results for the
[00:25:52] squad and the results for the adversaries and they and the you know um
[00:25:56] adversaries and they and the you know um percy young has this um system called
[00:25:58] percy young has this um system called coda lab where he possesses all the
[00:26:00] coda lab where he possesses all the systems that enter into the squad
[00:26:02] systems that enter into the squad competition which made it possible for
[00:26:04] competition which made it possible for him and his students to rerun all those
[00:26:06] him and his students to rerun all those systems and see how they did on this
[00:26:08] systems and see how they did on this adversarial data set that they had
[00:26:10] adversarial data set that they had created you can see that all the systems
[00:26:12] created you can see that all the systems really plummet in their performance from
[00:26:14] really plummet in their performance from a high of 81 you drop down to about 40.
[00:26:17] a high of 81 you drop down to about 40. maybe that's kind of expected but
[00:26:19] maybe that's kind of expected but another really eye-opening thing about
[00:26:21] another really eye-opening thing about the result they have is that the rank of
[00:26:23] the result they have is that the rank of the systems changed really dramatically
[00:26:25] the systems changed really dramatically dramatically right so the first the
[00:26:27] dramatically right so the first the original top-ranked system went to five
[00:26:29] original top-ranked system went to five two to ten three to twelve as we did
[00:26:32] two to ten three to twelve as we did this adversarial thing we didn't see a
[00:26:34] this adversarial thing we didn't see a uniform drop with the best system still
[00:26:37] uniform drop with the best system still being the best but really shuffling of
[00:26:39] being the best but really shuffling of this leaderboard again revealing that i
[00:26:41] this leaderboard again revealing that i think the best systems were kind of over
[00:26:43] think the best systems were kind of over fit and benefiting from relatively low
[00:26:46] fit and benefiting from relatively low level facts about the data set and not
[00:26:48] level facts about the data set and not really transformatively different when
[00:26:50] really transformatively different when it comes to being able to answer
[00:26:52] it comes to being able to answer questions
[00:26:56] the history of natural language
[00:26:57] the history of natural language in natural language inference problems
[00:26:59] in natural language inference problems is very similar as i said we're going to
[00:27:01] is very similar as i said we're going to look at this problem in a lot of detail
[00:27:03] look at this problem in a lot of detail later in the course here just a few very
[00:27:05] later in the course here just a few very simple nli examples you've got a premise
[00:27:07] simple nli examples you've got a premise like a turtle dance
[00:27:08] like a turtle dance hypothesis a turtle moved and you have
[00:27:11] hypothesis a turtle moved and you have one of three relations that can hold
[00:27:13] one of three relations that can hold between those sentences so a turtle
[00:27:14] between those sentences so a turtle dance entails a turtle moved
[00:27:17] dance entails a turtle moved every reptile danced is neutral with
[00:27:19] every reptile danced is neutral with respect to a turtle eight that can be
[00:27:20] respect to a turtle eight that can be true or false independently of each
[00:27:22] true or false independently of each other and some turtles walk contradicts
[00:27:24] other and some turtles walk contradicts no turtles move
[00:27:26] no turtles move this is typical kind of nli data the
[00:27:28] this is typical kind of nli data the actual corpus sentences tend to be more
[00:27:29] actual corpus sentences tend to be more complicated and involve more nuanced
[00:27:31] complicated and involve more nuanced judgments but that's a framing of the
[00:27:33] judgments but that's a framing of the task it's a three-way classification
[00:27:35] task it's a three-way classification problem with these these labels and the
[00:27:38] problem with these these labels and the inputs are pairs of sentences like this
[00:27:40] inputs are pairs of sentences like this and as i showed you before for one of
[00:27:42] and as i showed you before for one of the large benchmarks the stanford uh
[00:27:45] the large benchmarks the stanford uh natural language inference corpus we
[00:27:47] natural language inference corpus we reached superhuman performance in 2019
[00:27:50] reached superhuman performance in 2019 but those same systems really struggle
[00:27:53] but those same systems really struggle with simple adversarial attacks this is
[00:27:55] with simple adversarial attacks this is a lovely paper called breaking nli from
[00:27:57] a lovely paper called breaking nli from glockner at all what they did is fix a
[00:28:00] glockner at all what they did is fix a premise like a little girl kneeling in
[00:28:01] premise like a little girl kneeling in the dirt crying
[00:28:03] the dirt crying the original corpus example was that
[00:28:05] the original corpus example was that that entails a little girl is very sad
[00:28:08] that entails a little girl is very sad and they just had an expectation that
[00:28:10] and they just had an expectation that you know sort of adversarially but this
[00:28:12] you know sort of adversarially but this is a very friendly adversary if i just
[00:28:14] is a very friendly adversary if i just replace sad with unhappy i should
[00:28:16] replace sad with unhappy i should continue to see the entailment relation
[00:28:18] continue to see the entailment relation predicted after all i've just
[00:28:19] predicted after all i've just substituted one word for it's sort of
[00:28:22] substituted one word for it's sort of near synonym
[00:28:23] near synonym but what they actually saw is that
[00:28:25] but what they actually saw is that systems very reliably flip this to the
[00:28:27] systems very reliably flip this to the contradiction relation
[00:28:29] contradiction relation probably because they are keying into
[00:28:31] probably because they are keying into the fact that this is a negation and
[00:28:33] the fact that this is a negation and they over fit on the idea that the
[00:28:35] they over fit on the idea that the presence of negation is a signal that
[00:28:37] presence of negation is a signal that you're in the contradiction relationship
[00:28:39] you're in the contradiction relationship so that's a sort of distressing thing
[00:28:41] so that's a sort of distressing thing again humans don't make these mistakes
[00:28:43] again humans don't make these mistakes but systems are very prone to them let
[00:28:45] but systems are very prone to them let me show you one more this is a slightly
[00:28:47] me show you one more this is a slightly different adversarial attack in this
[00:28:49] different adversarial attack in this case we're going to modify the premise
[00:28:51] case we're going to modify the premise so the original training example was a
[00:28:53] so the original training example was a woman is pulling a child on a sled in
[00:28:55] woman is pulling a child on a sled in the snow
[00:28:56] the snow that entails a child is sitting on a
[00:28:58] that entails a child is sitting on a sled in the snow
[00:28:59] sled in the snow that's pretty clear for their
[00:29:01] that's pretty clear for their adversarial attack they just swapped the
[00:29:03] adversarial attack they just swapped the subject and the object so the new
[00:29:04] subject and the object so the new premise is a child is pulling a woman on
[00:29:07] premise is a child is pulling a woman on a sled in the snow
[00:29:08] a sled in the snow we would expect that to lead to the
[00:29:10] we would expect that to lead to the neutral label for this particular
[00:29:12] neutral label for this particular hypothesis
[00:29:14] hypothesis but what need all observe is that the
[00:29:15] but what need all observe is that the systems are kind of invariant under this
[00:29:18] systems are kind of invariant under this changing of the word order they continue
[00:29:20] changing of the word order they continue to predict entailment
[00:29:22] to predict entailment revealing that they don't really know
[00:29:23] revealing that they don't really know what the subject and the object were in
[00:29:25] what the subject and the object were in the original example they have kind of
[00:29:27] the original example they have kind of done something much fuzzier with the set
[00:29:29] done something much fuzzier with the set of words that are in that premise
[00:29:32] of words that are in that premise remember these are at the time the very
[00:29:34] remember these are at the time the very best systems for solving these problems
[00:29:36] best systems for solving these problems these are very simple kind of friendly
[00:29:38] these are very simple kind of friendly adversaries that they're stumbling with
[00:29:43] so this could lead you to have two
[00:29:44] so this could lead you to have two perspectives i showed you the nick
[00:29:45] perspectives i showed you the nick bostrom one before where we worry about
[00:29:47] bostrom one before where we worry about super intelligent systems but on the
[00:29:50] super intelligent systems but on the other hand we might be living in a world
[00:29:51] other hand we might be living in a world that's more like the one presented in
[00:29:53] that's more like the one presented in this lovely book from a roboticist a
[00:29:55] this lovely book from a roboticist a practitioner daniel h wilson called how
[00:29:57] practitioner daniel h wilson called how to survive a robot uprising where he
[00:30:00] to survive a robot uprising where he gives all sorts of practical advice like
[00:30:02] gives all sorts of practical advice like wear clothing that will fool the vision
[00:30:04] wear clothing that will fool the vision system of the robot or walk up some
[00:30:06] system of the robot or walk up some stairs or drench everything in water
[00:30:10] stairs or drench everything in water very simple adversarial attacks that
[00:30:12] very simple adversarial attacks that reveal that these robots are not
[00:30:14] reveal that these robots are not creatures which we should be fearful of
[00:30:16] creatures which we should be fearful of and i feel like i've just shown you a
[00:30:17] and i feel like i've just shown you a bunch of examples that are the the
[00:30:19] bunch of examples that are the the analogues of wearing misleading clothing
[00:30:22] analogues of wearing misleading clothing in the space of natural language
[00:30:23] in the space of natural language processing revealing that our systems
[00:30:25] processing revealing that our systems are not super human understanders or
[00:30:27] are not super human understanders or communicators or anything like that but
[00:30:29] communicators or anything like that but rather still to this day
[00:30:32] rather still to this day fairly superficial pattern measures
[00:30:36] fairly superficial pattern measures why is this all so difficult it's hard
[00:30:38] why is this all so difficult it's hard to articulate precisely what is so
[00:30:40] to articulate precisely what is so challenging because this is probably
[00:30:41] challenging because this is probably deeply embedded in the whole human
[00:30:43] deeply embedded in the whole human experience but i think there are some
[00:30:45] experience but i think there are some pretty straightforward superficial
[00:30:47] pretty straightforward superficial things i can show you to just make a
[00:30:49] things i can show you to just make a live for you how hard even the simplest
[00:30:51] live for you how hard even the simplest tasks are
[00:30:52] tasks are so here i've got a an imagined dialogue
[00:30:54] so here i've got a an imagined dialogue of the sort you hope siri would do well
[00:30:56] of the sort you hope siri would do well with
[00:30:57] with whereas black panther playing in
[00:30:59] whereas black panther playing in mountain view
[00:31:00] mountain view black panther is playing at the century
[00:31:01] black panther is playing at the century 16 theater
[00:31:03] 16 theater when is it playing there it's playing at
[00:31:05] when is it playing there it's playing at two five and eight
[00:31:07] two five and eight okay i'd like one adult and two children
[00:31:09] okay i'd like one adult and two children for the first show how much would that
[00:31:11] for the first show how much would that cost
[00:31:12] cost it seems like the most mundane sort of
[00:31:14] it seems like the most mundane sort of interaction you would not expect a human
[00:31:16] interaction you would not expect a human to have any problem with any of these
[00:31:18] to have any problem with any of these utterances
[00:31:19] utterances but think about how much interesting
[00:31:22] but think about how much interesting stuff is happening in this little
[00:31:23] stuff is happening in this little dialogue we have
[00:31:25] dialogue we have domain knowledge that tells us that this
[00:31:27] domain knowledge that tells us that this is a place where movies might play and
[00:31:30] is a place where movies might play and that this is the name of a movie that's
[00:31:31] that this is the name of a movie that's already very difficult
[00:31:33] already very difficult then we have anaphora from the third
[00:31:35] then we have anaphora from the third utterance to the first when is it
[00:31:37] utterance to the first when is it playing there fine i guess also into the
[00:31:39] playing there fine i guess also into the second
[00:31:40] second where these pronouns you need to figure
[00:31:41] where these pronouns you need to figure out what they refer to in the discourse
[00:31:44] out what they refer to in the discourse then you get the sequence of responses
[00:31:46] then you get the sequence of responses again with some anaphora back to earlier
[00:31:48] again with some anaphora back to earlier utterances and then something really
[00:31:50] utterances and then something really complicated happens here but like one
[00:31:52] complicated happens here but like one adult the two children for the first
[00:31:54] adult the two children for the first show
[00:31:55] show first show refers back to the sequence
[00:31:57] first show refers back to the sequence of things that was mentioned here very
[00:31:59] of things that was mentioned here very difficult one adult and two children is
[00:32:01] difficult one adult and two children is not a request for human beings although
[00:32:03] not a request for human beings although that's what the forums would look like
[00:32:05] that's what the forums would look like but rather our request for tickets so
[00:32:07] but rather our request for tickets so somehow in the context of this discord
[00:32:10] somehow in the context of this discord one adult and two children is referring
[00:32:12] one adult and two children is referring to tickets and not to people
[00:32:14] to tickets and not to people how much would that cost that is a kind
[00:32:16] how much would that cost that is a kind of complicated event description
[00:32:18] of complicated event description referring to a hypothetical event
[00:32:20] referring to a hypothetical event of buying some tickets for a particular
[00:32:23] of buying some tickets for a particular show that's the reference of this that
[00:32:25] show that's the reference of this that here highly abstract very difficult of
[00:32:27] here highly abstract very difficult of the level of resolving it in the
[00:32:29] the level of resolving it in the discourse and then figuring out what its
[00:32:31] discourse and then figuring out what its actual content is
[00:32:34] actual content is and this is for the most mundane sort of
[00:32:35] and this is for the most mundane sort of interaction to say nothing of the
[00:32:37] interaction to say nothing of the complicated things that for example you
[00:32:40] complicated things that for example you and i will do when we discuss this
[00:32:42] and i will do when we discuss this material in just a few minutes
[00:32:44] material in just a few minutes so i think this is why we're actually
[00:32:45] so i think this is why we're actually quite far from the super intelligence
[00:32:48] quite far from the super intelligence that bostrom was worried about
[00:32:51] that bostrom was worried about here's our perspective as i said this is
[00:32:52] here's our perspective as i said this is the most exciting moment ever in history
[00:32:55] the most exciting moment ever in history for doing nlu why because there's
[00:32:57] for doing nlu why because there's incredible interest in the problems
[00:32:59] incredible interest in the problems because we are making incredibly fast
[00:33:01] because we are making incredibly fast progress and doing things and solving
[00:33:03] progress and doing things and solving problems that we never could have even
[00:33:05] problems that we never could have even tackled 15 years ago
[00:33:07] tackled 15 years ago on the other hand you do not have the
[00:33:09] on the other hand you do not have the misfortune of having joined the field at
[00:33:11] misfortune of having joined the field at its very end the big problems remain to
[00:33:14] its very end the big problems remain to be solved
[00:33:16] be solved yeah so there's a resurgence of interest
[00:33:18] yeah so there's a resurgence of interest and explosion of products the systems
[00:33:20] and explosion of products the systems are impressive but their weaknesses make
[00:33:22] are impressive but their weaknesses make themselves quickly apparent and when we
[00:33:24] themselves quickly apparent and when we observe those weaknesses it's an
[00:33:26] observe those weaknesses it's an opportunity for us to figure out what
[00:33:28] opportunity for us to figure out what the problem is and that could lead to
[00:33:30] the problem is and that could lead to the really big breakthroughs
[00:33:32] the really big breakthroughs and you all are now joining us on this
[00:33:34] and you all are now joining us on this journey if you haven't begun it already
[00:33:36] journey if you haven't begun it already and for your projects you'll make some
[00:33:38] and for your projects you'll make some progress along the path of helping us
[00:33:40] progress along the path of helping us through these very difficult problems
[00:33:42] through these very difficult problems the field is confronting
[00:33:44] the field is confronting even in the presence of all these
[00:33:45] even in the presence of all these exciting breakthroughs
[00:33:48] exciting breakthroughs nlu is far from solved the big
[00:33:50] nlu is far from solved the big breakthroughs lie in the future
[00:33:54] so i hope that's inspiring
[00:33:55] so i hope that's inspiring now let me switch gears a little bit and
[00:33:57] now let me switch gears a little bit and talk about the things that we'll
[00:33:58] talk about the things that we'll actually be doing in this course to help
[00:34:00] actually be doing in this course to help set you on this journey that we're all
[00:34:02] set you on this journey that we're all on
[00:34:03] on so we'll talk about the assignments the
[00:34:05] so we'll talk about the assignments the bake-offs and the projects
[00:34:07] bake-offs and the projects the high-level summary here our topics
[00:34:10] the high-level summary here our topics are listed on the left you can also see
[00:34:12] are listed on the left you can also see this reflected on the website
[00:34:14] this reflected on the website the one thing that i really do like
[00:34:15] the one thing that i really do like about this particular plan is that it
[00:34:18] about this particular plan is that it gives you exposure to a lot of different
[00:34:20] gives you exposure to a lot of different problems in the fields and also helps
[00:34:22] problems in the fields and also helps you with some tools and techniques that
[00:34:24] you with some tools and techniques that will be really useful no matter what
[00:34:26] will be really useful no matter what problem you undertake for your final
[00:34:28] problem you undertake for your final project
[00:34:30] project the same thing goes for the assignments
[00:34:31] the same thing goes for the assignments we're going to have three assignments
[00:34:33] we're going to have three assignments each with an associated bake off which
[00:34:34] each with an associated bake off which is a kind of competition around data
[00:34:37] is a kind of competition around data we're going to talk about word
[00:34:38] we're going to talk about word relatedness cross-domain sentiment
[00:34:40] relatedness cross-domain sentiment analysis and generating color
[00:34:42] analysis and generating color descriptions this is a kind of grounded
[00:34:44] descriptions this is a kind of grounded language understanding problem again i
[00:34:46] language understanding problem again i think those are good choices because
[00:34:48] think those are good choices because they expose you to a lot of different
[00:34:50] they expose you to a lot of different kinds of systems techniques model
[00:34:52] kinds of systems techniques model architectures and so forth
[00:34:54] architectures and so forth that should set you up really nicely to
[00:34:56] that should set you up really nicely to do a final project which has three
[00:34:58] do a final project which has three components a literature review an
[00:35:00] components a literature review an experimental protocol and then the final
[00:35:02] experimental protocol and then the final paper itself our time for this quarter
[00:35:05] paper itself our time for this quarter is somewhat compressed
[00:35:06] is somewhat compressed um so we'll have to make really good use
[00:35:08] um so we'll have to make really good use of the time but i think we have a
[00:35:09] of the time but i think we have a schedule that will allow you to
[00:35:11] schedule that will allow you to meaningfully invest in this in this
[00:35:13] meaningfully invest in this in this preliminary work and still provide you
[00:35:15] preliminary work and still provide you with some space to do these final
[00:35:17] with some space to do these final projects
[00:35:19] projects let's talk about the assignments and
[00:35:20] let's talk about the assignments and bake offs themselves so there are three
[00:35:22] bake offs themselves so there are three of them
[00:35:23] of them each assignment culminates in a bake-off
[00:35:25] each assignment culminates in a bake-off which is an informal competition in
[00:35:27] which is an informal competition in which you enter an original model the
[00:35:28] which you enter an original model the original model question
[00:35:30] original model question is part of the assignment you do
[00:35:32] is part of the assignment you do something that you think will be fun or
[00:35:33] something that you think will be fun or interesting and then the bake off
[00:35:35] interesting and then the bake off essentially involves using that system
[00:35:37] essentially involves using that system to make predictions on a held out test
[00:35:39] to make predictions on a held out test set
[00:35:41] set the assignments ask you to build these
[00:35:43] the assignments ask you to build these baseline systems and then design your
[00:35:45] baseline systems and then design your original system as i said
[00:35:47] original system as i said practically speaking the way it works is
[00:35:48] practically speaking the way it works is that the assignments earn you 9 of the
[00:35:50] that the assignments earn you 9 of the 10 points and then you earn your
[00:35:52] 10 points and then you earn your additional point by entering your system
[00:35:53] additional point by entering your system into the bake off
[00:35:55] into the bake off and the winning bank offs can receive
[00:35:57] and the winning bank offs can receive some extra credit the rationale for all
[00:35:59] some extra credit the rationale for all of this of course is that we want to
[00:36:01] of this of course is that we want to exemplify the best practices for doing
[00:36:03] exemplify the best practices for doing research in this space
[00:36:05] research in this space and help you do things like
[00:36:06] and help you do things like incrementally build up a project with
[00:36:09] incrementally build up a project with baselines and then finally an original
[00:36:11] baselines and then finally an original system and i should say it should be
[00:36:14] system and i should say it should be possible and it's actually pretty common
[00:36:16] possible and it's actually pretty common for people to take original systems that
[00:36:18] for people to take original systems that they developed as part of one of these
[00:36:20] they developed as part of one of these assignments and use them for their final
[00:36:22] assignments and use them for their final project each one of the assignments is
[00:36:24] project each one of the assignments is set up specifically to make that kind of
[00:36:26] set up specifically to make that kind of thing possible and productive and
[00:36:28] thing possible and productive and rewarding
[00:36:31] rewarding let me show you briefly what the
[00:36:32] let me show you briefly what the bake-offs are going to be like so the
[00:36:33] bake-offs are going to be like so the first one is word relatedness the focus
[00:36:36] first one is word relatedness the focus of that unit is on developing vector
[00:36:38] of that unit is on developing vector representations of words you're going to
[00:36:40] representations of words you're going to start probably with big count matrices
[00:36:42] start probably with big count matrices like the one you see here this is a word
[00:36:45] like the one you see here this is a word by word matrix where all these cells
[00:36:47] by word matrix where all these cells give the number of times that these
[00:36:48] give the number of times that these words or in this case emoticons
[00:36:50] words or in this case emoticons co-occurred with each other in a very
[00:36:52] co-occurred with each other in a very large corpus of text
[00:36:54] large corpus of text the striking thing about this unit is
[00:36:56] the striking thing about this unit is that there is a lot of information about
[00:36:57] that there is a lot of information about meaning embedded in these large spaces
[00:37:00] meaning embedded in these large spaces you will bend and twist and massage
[00:37:02] you will bend and twist and massage these spaces and maybe bring in your own
[00:37:04] these spaces and maybe bring in your own vector representations or
[00:37:06] vector representations or representations you've downloaded from
[00:37:08] representations you've downloaded from the web and you will use them to solve a
[00:37:11] the web and you will use them to solve a word relatedness problem so basically
[00:37:13] word relatedness problem so basically you'll be given
[00:37:14] you'll be given pairs of words like this with a score
[00:37:17] pairs of words like this with a score and you will develop a system that can
[00:37:19] and you will develop a system that can make predictions about new pairs and the
[00:37:22] make predictions about new pairs and the idea is to come up with scores that
[00:37:23] idea is to come up with scores that correlate with the held out scores that
[00:37:26] correlate with the held out scores that we have not distributed of course as
[00:37:27] we have not distributed of course as part of the test set you'll be you'll
[00:37:29] part of the test set you'll be you'll upload your entry and we'll give you a
[00:37:31] upload your entry and we'll give you a score and then we'll look at what worked
[00:37:33] score and then we'll look at what worked and what didn't across all the systems
[00:37:35] and what didn't across all the systems and the techniques that we'll explore
[00:37:37] and the techniques that we'll explore are many right so we'll talk about
[00:37:38] are many right so we'll talk about re-weighting dimensionality reduction
[00:37:41] re-weighting dimensionality reduction vector comparison you'll have an
[00:37:42] vector comparison you'll have an opportunity if you wish this is a brand
[00:37:44] opportunity if you wish this is a brand new addition to the course to bring in
[00:37:46] new addition to the course to bring in bert if you would like to
[00:37:48] bert if you would like to um so there's lots of inspiring things
[00:37:50] um so there's lots of inspiring things to try building on the latest stuff
[00:37:51] to try building on the latest stuff that's happening in this space
[00:37:55] the second bake off is called cross
[00:37:57] the second bake off is called cross domain sentiment analysis this is a
[00:37:59] domain sentiment analysis this is a brand new bake off i'm very excited to
[00:38:01] brand new bake off i'm very excited to see how this goes here's how this is
[00:38:02] see how this goes here's how this is going to work we want to be a little bit
[00:38:04] going to work we want to be a little bit adversarial with your system so there
[00:38:06] adversarial with your system so there are two data sets involved the stanford
[00:38:08] are two data sets involved the stanford sentiment tree bank which is movie
[00:38:10] sentiment tree bank which is movie review sentences and we're going to deal
[00:38:12] review sentences and we're going to deal with it in its ternary formulation so it
[00:38:14] with it in its ternary formulation so it has positive negative and neutral labels
[00:38:16] has positive negative and neutral labels that's sst3
[00:38:18] that's sst3 alongside that i'm going to introduce a
[00:38:21] alongside that i'm going to introduce a brand new
[00:38:22] brand new dev test split previously unreleased
[00:38:25] dev test split previously unreleased which is sentences from the restaurant
[00:38:27] which is sentences from the restaurant review domain
[00:38:28] review domain has the same kind of labels but of
[00:38:30] has the same kind of labels but of course it's very different from the
[00:38:32] course it's very different from the movie review domain along many
[00:38:34] movie review domain along many dimensions
[00:38:36] dimensions so for the bake off you'll have the sst3
[00:38:38] so for the bake off you'll have the sst3 train set we're going to we're going to
[00:38:40] train set we're going to we're going to give that to you and you are welcome to
[00:38:41] give that to you and you are welcome to introduce any other data you would like
[00:38:43] introduce any other data you would like to introduce as part of this training
[00:38:45] to introduce as part of this training that is entirely up to you
[00:38:47] that is entirely up to you we're also distributing for this take
[00:38:49] we're also distributing for this take off two dev sets the sst3 dev set which
[00:38:52] off two dev sets the sst3 dev set which is public already and this brand new one
[00:38:54] is public already and this brand new one of restaurant review sentences
[00:38:56] of restaurant review sentences and you can introduce other development
[00:38:58] and you can introduce other development sets if you want as part of tuning your
[00:39:00] sets if you want as part of tuning your system
[00:39:01] system and then the bake off will be conducted
[00:39:03] and then the bake off will be conducted as the best that people can do jointly
[00:39:05] as the best that people can do jointly on the ssd3 and this new test set which
[00:39:08] on the ssd3 and this new test set which is again held out the idea here is that
[00:39:10] is again held out the idea here is that you'll not only be doing a really great
[00:39:13] you'll not only be doing a really great kind of project in a classification
[00:39:15] kind of project in a classification thing involving sentiment
[00:39:17] thing involving sentiment but also pushing your systems to adapt
[00:39:19] but also pushing your systems to adapt into new domains from the ones that they
[00:39:22] into new domains from the ones that they were trained on although of course part
[00:39:23] were trained on although of course part of that could be training in clever ways
[00:39:26] of that could be training in clever ways that actually does help that that do
[00:39:28] that actually does help that that do help you anticipate what's in this
[00:39:30] help you anticipate what's in this restaurant review data
[00:39:33] restaurant review data and then the third bake off is a natural
[00:39:35] and then the third bake off is a natural language generation test but it's a
[00:39:37] language generation test but it's a grounded natural language generation
[00:39:39] grounded natural language generation test so for this
[00:39:40] test so for this bake off you'll be given color context
[00:39:42] bake off you'll be given color context like three patches like this and one of
[00:39:44] like three patches like this and one of them that is designated as the target
[00:39:47] them that is designated as the target for the training data we had people
[00:39:49] for the training data we had people describe the target color in this
[00:39:50] describe the target color in this context and your task is to develop a
[00:39:53] context and your task is to develop a system that can perform that same task
[00:39:55] system that can perform that same task produce natural language utterances like
[00:39:57] produce natural language utterances like this
[00:39:58] this i think this is a really cool problem
[00:40:00] i think this is a really cool problem because it's grounded outside of
[00:40:01] because it's grounded outside of something that involves language right
[00:40:03] something that involves language right you're grounded in a color patch it's
[00:40:04] you're grounded in a color patch it's just a numerical representation
[00:40:07] just a numerical representation and it's also highly context dependent
[00:40:09] and it's also highly context dependent in that the choices people make for
[00:40:12] in that the choices people make for their utterances are dependent not only
[00:40:13] their utterances are dependent not only on the target
[00:40:15] on the target the color they need to describe but the
[00:40:16] the color they need to describe but the target in the context of these other two
[00:40:18] target in the context of these other two colors and you can see that in these
[00:40:20] colors and you can see that in these descriptions this one is easy so the
[00:40:22] descriptions this one is easy so the person just said blue but in this case
[00:40:24] person just said blue but in this case since there are two blues they said the
[00:40:26] since there are two blues they said the darker blue one kind of keying into the
[00:40:28] darker blue one kind of keying into the fact that the context contains two blue
[00:40:30] fact that the context contains two blue colors
[00:40:32] colors this is an even more extreme case of
[00:40:33] this is an even more extreme case of that dull pink not the super bright one
[00:40:36] that dull pink not the super bright one implicit reference is not only to the
[00:40:37] implicit reference is not only to the target but also to other colors that are
[00:40:40] target but also to other colors that are distractors in that context
[00:40:43] distractors in that context and then the final two are really
[00:40:44] and then the final two are really extreme cases of this where for one in
[00:40:46] extreme cases of this where for one in the same color you know in terms of its
[00:40:48] the same color you know in terms of its color representation this person said
[00:40:50] color representation this person said purple and this person said blue in
[00:40:53] purple and this person said blue in virtue of the fact that the distractors
[00:40:55] virtue of the fact that the distractors are different
[00:40:56] are different so we have this kind of controlled
[00:40:58] so we have this kind of controlled natural language generation task that we
[00:41:00] natural language generation task that we can see is highly dependent on the
[00:41:02] can see is highly dependent on the particular context that p that are p
[00:41:04] particular context that p that are p that people are in here
[00:41:06] that people are in here and again this will follow the same path
[00:41:08] and again this will follow the same path here i'm going to give you a whole the
[00:41:09] here i'm going to give you a whole the whole model architecture as a kind of
[00:41:11] whole model architecture as a kind of default it's an encoder decoder
[00:41:13] default it's an encoder decoder architecture where you'll have a machine
[00:41:15] architecture where you'll have a machine learning system that consumes color
[00:41:17] learning system that consumes color representations
[00:41:19] representations and then transfers that into a decoding
[00:41:21] and then transfers that into a decoding phase where it tries to produce an
[00:41:22] phase where it tries to produce an utterance so here you've consumed a
[00:41:25] utterance so here you've consumed a bunch of colors and produced the
[00:41:26] bunch of colors and produced the description light blue
[00:41:28] description light blue but of course you'll be able to explore
[00:41:30] but of course you'll be able to explore many variants of this architecture and
[00:41:32] many variants of this architecture and really explore different things that are
[00:41:34] really explore different things that are effective in terms of this
[00:41:35] effective in terms of this representation and natural language
[00:41:37] representation and natural language generation task
[00:41:39] generation task and again as a bake off it will work the
[00:41:41] and again as a bake off it will work the same way you'll do a bunch of
[00:41:42] same way you'll do a bunch of development on training data that we
[00:41:43] development on training data that we give you
[00:41:44] give you and then you'll be evaluated on the held
[00:41:46] and then you'll be evaluated on the held out test set that was produced in the
[00:41:48] out test set that was produced in the same fashion
[00:41:50] same fashion but involves entirely new colors and
[00:41:52] but involves entirely new colors and reverences
[00:41:55] quick note on the original systems as i
[00:41:58] quick note on the original systems as i said before the original system is kind
[00:41:59] said before the original system is kind of a central piece of each one of the
[00:42:01] of a central piece of each one of the assignments the homeworks really
[00:42:03] assignments the homeworks really culminate in this original system and
[00:42:05] culminate in this original system and that becomes your bake off entry
[00:42:08] that becomes your bake off entry um in terms of grading it this is kind
[00:42:10] um in terms of grading it this is kind of hard because we want you to be
[00:42:11] of hard because we want you to be creative and try lots of things so the
[00:42:13] creative and try lots of things so the way we're going to value these entries
[00:42:15] way we're going to value these entries is any system that performs extremely
[00:42:17] is any system that performs extremely well on the bake off will be given full
[00:42:19] well on the bake off will be given full credit
[00:42:20] credit even symbols that have systems that are
[00:42:21] even symbols that have systems that are very simple right we can argue with
[00:42:24] very simple right we can argue with success according to the criteria that
[00:42:26] success according to the criteria that we've set up so if the simplest possible
[00:42:28] we've set up so if the simplest possible approach to one of these bake offs turns
[00:42:30] approach to one of these bake offs turns out to be astoundingly good and you had
[00:42:32] out to be astoundingly good and you had to do almost no work to succeed you of
[00:42:34] to do almost no work to succeed you of course get full credit
[00:42:36] course get full credit but that's not the only thing we value
[00:42:38] but that's not the only thing we value right so systems that are creative and
[00:42:40] right so systems that are creative and well motivated will be given full credit
[00:42:42] well motivated will be given full credit even if they don't perform well on the
[00:42:44] even if they don't perform well on the big off data this is meant to be an
[00:42:46] big off data this is meant to be an explicit encouragement for you to try
[00:42:48] explicit encouragement for you to try new things to be bold and be creative
[00:42:51] new things to be bold and be creative even if it doesn't numerically lead to
[00:42:53] even if it doesn't numerically lead to the best results in fact some of the
[00:42:55] the best results in fact some of the most inspiring things we've seen and
[00:42:57] most inspiring things we've seen and insightful things we've seen as part of
[00:42:59] insightful things we've seen as part of these bake-offs have been systems that
[00:43:01] these bake-offs have been systems that didn't perform at the top of the heap
[00:43:03] didn't perform at the top of the heap but harbored some really interesting
[00:43:05] but harbored some really interesting insight that we could build on
[00:43:08] insight that we could build on right and then of course systems that
[00:43:10] right and then of course systems that really are minimal you do if you do very
[00:43:12] really are minimal you do if you do very little and you don't do especially well
[00:43:14] little and you don't do especially well at the bake off we'll receive less
[00:43:16] at the bake off we'll receive less credit um the specific criteria will
[00:43:18] credit um the specific criteria will depend on the nature of the assignment
[00:43:20] depend on the nature of the assignment and so forth we'll try to justify that
[00:43:22] and so forth we'll try to justify that for you this is the more subjective part
[00:43:24] for you this is the more subjective part i think one and two really encode the
[00:43:27] i think one and two really encode the positive part of the kind of values that
[00:43:29] positive part of the kind of values that we're trying to convey to you as part of
[00:43:31] we're trying to convey to you as part of these original system entries
[00:43:36] and then you'll have project work this
[00:43:38] and then you'll have project work this occupies the entire second half of the
[00:43:40] occupies the entire second half of the course
[00:43:42] course at that point the lectures the notebooks
[00:43:44] at that point the lectures the notebooks the readings and so forth are really
[00:43:46] the readings and so forth are really focused on things like methods metrics
[00:43:48] focused on things like methods metrics best practices
[00:43:50] best practices error analysis model introspection and
[00:43:52] error analysis model introspection and other things that will help you enrich
[00:43:54] other things that will help you enrich the paper that you write
[00:43:57] the paper that you write the assignments are all project related
[00:43:59] the assignments are all project related um they're the literature view the
[00:44:00] um they're the literature view the experimental protocol and the final
[00:44:02] experimental protocol and the final paper unfortunately in many years we've
[00:44:04] paper unfortunately in many years we've had a video presentation which has
[00:44:06] had a video presentation which has always been really rewarding but i feel
[00:44:08] always been really rewarding but i feel like given the compressed time schedule
[00:44:10] like given the compressed time schedule that we're on we just don't have time
[00:44:11] that we're on we just don't have time for uh
[00:44:12] for uh for even short videos we're going to
[00:44:14] for even short videos we're going to focus on these three crucial components
[00:44:17] focus on these three crucial components uh and if for exceptional final projects
[00:44:19] uh and if for exceptional final projects from past years that we've selected you
[00:44:21] from past years that we've selected you could follow this link it's access
[00:44:22] could follow this link it's access restricted but if you're enrolled in the
[00:44:24] restricted but if you're enrolled in the course you should be able to follow the
[00:44:26] course you should be able to follow the link and see some of some examples and
[00:44:28] link and see some of some examples and there's a lot more gardens on final
[00:44:30] there's a lot more gardens on final projects in our course repository i have
[00:44:32] projects in our course repository i have a very long write-up of faqs and other
[00:44:35] a very long write-up of faqs and other guidance about publishing in the field
[00:44:38] guidance about publishing in the field and writing for this class and i have
[00:44:40] and writing for this class and i have what what is now a really inspiringly
[00:44:42] what what is now a really inspiringly long list of published papers that have
[00:44:44] long list of published papers that have grown out of work people have done for
[00:44:46] grown out of work people have done for this course so you can check that all
[00:44:48] this course so you can check that all out here final words here by way of
[00:44:50] out here final words here by way of wrapping up as i said this is the most
[00:44:52] wrapping up as i said this is the most most exciting moment ever in history for
[00:44:54] most exciting moment ever in history for doing nlu
[00:44:55] doing nlu this course will give you hands-on
[00:44:57] this course will give you hands-on experience with a wide range of
[00:44:59] experience with a wide range of challenging problems i emphasize the
[00:45:00] challenging problems i emphasize the hands-on thing i think this is so
[00:45:02] hands-on thing i think this is so important
[00:45:03] important if you want to acquire a new skill like
[00:45:05] if you want to acquire a new skill like this
[00:45:06] this it's all well and good to watch other
[00:45:07] it's all well and good to watch other people doing it but the way you really
[00:45:09] people doing it but the way you really acquire the skill is by having hands-on
[00:45:12] acquire the skill is by having hands-on experiences yourself so everything about
[00:45:14] experiences yourself so everything about the requirements and the materials is
[00:45:16] the requirements and the materials is pushing you to have those hands-on
[00:45:18] pushing you to have those hands-on experiences
[00:45:19] experiences and become expert at that very
[00:45:21] and become expert at that very fundamental level
[00:45:24] fundamental level for the final project a mentor for the
[00:45:25] for the final project a mentor for the teaching team will guide you through
[00:45:27] teaching team will guide you through those assignments
[00:45:28] those assignments we'll be there to help you and make
[00:45:31] we'll be there to help you and make choices and set the scope for the
[00:45:32] choices and set the scope for the project and maybe push it towards
[00:45:34] project and maybe push it towards something that you could one day publish
[00:45:36] something that you could one day publish there are many examples of successful
[00:45:38] there are many examples of successful publications deriving from this course
[00:45:42] publications deriving from this course our central goal fundamentally though is
[00:45:44] our central goal fundamentally though is to make you the best that is most
[00:45:46] to make you the best that is most insightful and responsible nlu
[00:45:48] insightful and responsible nlu practitioner and researcher wherever you
[00:45:51] practitioner and researcher wherever you go next into academia or just into other
[00:45:53] go next into academia or just into other classes or on into industry to leverage
[00:45:56] classes or on into industry to leverage these skills
[00:45:57] these skills we want to make you as i said the most
[00:45:59] we want to make you as i said the most insightful and responsible practitioner
[00:46:01] insightful and responsible practitioner we can

Homework 1: Word Relatedness | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=egEzcwbej1E

---

Transcript

[00:00:05] hello everyone this screencast is going
[00:00:06] hello everyone this screencast is going to be a brief playthrough of homework
[00:00:08] to be a brief playthrough of homework one on word relatedness i hope to give
[00:00:10] one on word relatedness i hope to give you a sense for the problem that you're
[00:00:11] you a sense for the problem that you're tackling and also our expectations
[00:00:14] tackling and also our expectations around the homework questions and the
[00:00:15] around the homework questions and the bake off so let's dive in here
[00:00:18] bake off so let's dive in here the overview is just explaining the
[00:00:19] the overview is just explaining the character of this problem which is
[00:00:20] character of this problem which is essentially that we're going to give you
[00:00:22] essentially that we're going to give you a development set of word pairs with
[00:00:24] a development set of word pairs with scores and the scores reflect
[00:00:26] scores and the scores reflect relatedness the scores were produced by
[00:00:28] relatedness the scores were produced by humans and we've just scaled them into
[00:00:30] humans and we've just scaled them into zero to one
[00:00:31] zero to one where a larger score means more related
[00:00:33] where a larger score means more related in this human sense and your task is
[00:00:36] in this human sense and your task is essentially to develop a system that
[00:00:37] essentially to develop a system that will predict scores that are highly
[00:00:39] will predict scores that are highly correlated with those human scores
[00:00:41] correlated with those human scores according to the spearman correlation
[00:00:43] according to the spearman correlation coefficient which is the traditional
[00:00:45] coefficient which is the traditional metric in this space
[00:00:47] metric in this space so this is just some setup stuff for the
[00:00:49] so this is just some setup stuff for the environment and then we introduce the
[00:00:51] environment and then we introduce the development set itself which is a
[00:00:52] development set itself which is a panda's data frame it's loaded in from
[00:00:54] panda's data frame it's loaded in from your data folder
[00:00:57] your data folder and it looks like this it's got a bunch
[00:00:58] and it looks like this it's got a bunch of word pairs each with scores as i said
[00:01:00] of word pairs each with scores as i said before these are the human provided
[00:01:02] before these are the human provided scores this is a development data set in
[00:01:04] scores this is a development data set in the sense that you can make whatever use
[00:01:06] the sense that you can make whatever use you want of it you can train systems you
[00:01:08] you want of it you can train systems you can explore your results and so forth
[00:01:10] can explore your results and so forth because as you'll see for the actual
[00:01:12] because as you'll see for the actual bake off we have a fresh test set that
[00:01:14] bake off we have a fresh test set that you'll make predictions on
[00:01:16] you'll make predictions on there are about 5 000 words in this
[00:01:18] there are about 5 000 words in this development set
[00:01:19] development set um you can train on any subset and you
[00:01:22] um you can train on any subset and you can expand the data set to other things
[00:01:24] can expand the data set to other things if you want to include them as well it's
[00:01:26] if you want to include them as well it's really up to you to decide what you want
[00:01:28] really up to you to decide what you want to do because this is all about making
[00:01:29] to do because this is all about making predictions on that brand new test set
[00:01:31] predictions on that brand new test set as you'll see later
[00:01:33] as you'll see later and i will just say that the test set
[00:01:34] and i will just say that the test set has 1500 word pairs with scores of the
[00:01:37] has 1500 word pairs with scores of the same type
[00:01:38] same type and in terms of the overlap i'll also
[00:01:40] and in terms of the overlap i'll also tell you no word pair in this
[00:01:42] tell you no word pair in this development set is in the test set so
[00:01:44] development set is in the test set so it's disjoint at the level of these
[00:01:46] it's disjoint at the level of these pairs but some of the individual words
[00:01:48] pairs but some of the individual words are repeated in the test sets you do
[00:01:50] are repeated in the test sets you do have some vocabulary overlap
[00:01:53] have some vocabulary overlap in this code here we load the full
[00:01:55] in this code here we load the full vocabulary for this thing which is you
[00:01:57] vocabulary for this thing which is you know all the words appearing in all the
[00:01:58] know all the words appearing in all the pairs
[00:01:59] pairs the vocabulary for the bake off test is
[00:02:01] the vocabulary for the bake off test is different it's partially overlapping
[00:02:03] different it's partially overlapping with the above as i said now if you
[00:02:06] with the above as i said now if you wanted to make sure ahead of time that
[00:02:08] wanted to make sure ahead of time that your system has a representation for
[00:02:09] your system has a representation for every word in both the dev and the test
[00:02:11] every word in both the dev and the test sets then you can check against the
[00:02:13] sets then you can check against the vocabularies in any of the vector space
[00:02:15] vocabularies in any of the vector space models that we've distributed with this
[00:02:18] models that we've distributed with this unit
[00:02:19] unit so for example if you ran this code you
[00:02:20] so for example if you ran this code you get the full task vocabulary and if you
[00:02:23] get the full task vocabulary and if you have a representation for every word in
[00:02:24] have a representation for every word in there then you're good shape in good
[00:02:26] there then you're good shape in good shape when it comes to the test set
[00:02:31] it's also useful to look at the score
[00:02:32] it's also useful to look at the score distribution this will give you a sense
[00:02:34] distribution this will give you a sense for what kind of space you're making
[00:02:36] for what kind of space you're making predictions into and i'll give you the
[00:02:37] predictions into and i'll give you the hint that the test distribution looks an
[00:02:39] hint that the test distribution looks an awful lot like this dev set distribution
[00:02:43] awful lot like this dev set distribution it's also worth being aware that there
[00:02:45] it's also worth being aware that there are some repeated pairs in the training
[00:02:47] are some repeated pairs in the training set some words that have different
[00:02:49] set some words that have different scores associated with them and they're
[00:02:51] scores associated with them and they're repeated therefore what i've done here
[00:02:53] repeated therefore what i've done here is just provide you with some code that
[00:02:54] is just provide you with some code that will allow you to rank pairs of words by
[00:02:56] will allow you to rank pairs of words by the variance in their scores and so you
[00:02:59] the variance in their scores and so you could decide for yourself what you want
[00:03:01] could decide for yourself what you want to do about these minor inconsistencies
[00:03:03] to do about these minor inconsistencies you could filter the data set or keep
[00:03:05] you could filter the data set or keep all of these examples in it's entirely
[00:03:08] all of these examples in it's entirely up to you i will just say that the test
[00:03:10] up to you i will just say that the test set does not force you to confront this
[00:03:11] set does not force you to confront this issue it has no repeated pairs in it
[00:03:15] all right and then we come to the
[00:03:17] all right and then we come to the evaluation topic here so there's a
[00:03:18] evaluation topic here so there's a central function you'll be using a lot
[00:03:20] central function you'll be using a lot in the homework and the bake off word
[00:03:22] in the homework and the bake off word related and disevaluation
[00:03:24] related and disevaluation and so there are some instructions about
[00:03:26] and so there are some instructions about how the interface works let me just give
[00:03:28] how the interface works let me just give a brief illustration
[00:03:29] a brief illustration in this cell i'm loading in one of our
[00:03:31] in this cell i'm loading in one of our count matrices it's the giga5 matrix
[00:03:35] count matrices it's the giga5 matrix and i'm going to evaluate that directly
[00:03:36] and i'm going to evaluate that directly so you can see in the next cell that
[00:03:39] so you can see in the next cell that word relatedness evaluation takes in our
[00:03:41] word relatedness evaluation takes in our development data for our test data
[00:03:44] development data for our test data and whatever vector space model you've
[00:03:46] and whatever vector space model you've developed as its two arguments and it
[00:03:48] developed as its two arguments and it returns a new version of this input here
[00:03:51] returns a new version of this input here with a column for the predictions you
[00:03:52] with a column for the predictions you made as well as this value here which is
[00:03:55] made as well as this value here which is the spearmint rank correlation
[00:03:57] the spearmint rank correlation coefficient that's our primary metric
[00:03:59] coefficient that's our primary metric for this unit
[00:04:01] for this unit here
[00:04:02] here is the score that i achieved not so good
[00:04:04] is the score that i achieved not so good i'm sure you'll be able to do better and
[00:04:06] i'm sure you'll be able to do better and here's a look at the new account data
[00:04:07] here's a look at the new account data frame with that new predict column of
[00:04:09] frame with that new predict column of predictions inserted into it
[00:04:13] predictions inserted into it and this is just another baseline here a
[00:04:15] and this is just another baseline here a truly random system that just predicts a
[00:04:16] truly random system that just predicts a random score and that's uh even worse
[00:04:18] random score and that's uh even worse than the simple count baseline again
[00:04:20] than the simple count baseline again you'll be able to do much better without
[00:04:22] you'll be able to do much better without much effort
[00:04:24] much effort error analysis i've provided you with
[00:04:26] error analysis i've provided you with some functions that will allow you to
[00:04:27] some functions that will allow you to look at what your system is doing in
[00:04:29] look at what your system is doing in terms of the best predictions in terms
[00:04:31] terms of the best predictions in terms of comparing against the human scores
[00:04:33] of comparing against the human scores and the worst predictions and i am
[00:04:35] and the worst predictions and i am imagining that this might help you
[00:04:36] imagining that this might help you figure out where you're doing well and
[00:04:38] figure out where you're doing well and where you're doing poorly and then you
[00:04:39] where you're doing poorly and then you can iterate on that basis
[00:04:42] can iterate on that basis that brings us to the homework questions
[00:04:44] that brings us to the homework questions so what we're trying to do here is help
[00:04:46] so what we're trying to do here is help you establish some baseline systems get
[00:04:48] you establish some baseline systems get used to the code and also think in new
[00:04:51] used to the code and also think in new and creative ways about the underlying
[00:04:52] and creative ways about the underlying problem
[00:04:54] problem our first one is positive point wise
[00:04:56] our first one is positive point wise mutual information as a baseline as
[00:04:58] mutual information as a baseline as you've seen in the materials for this
[00:05:00] you've seen in the materials for this unit point wise mutual information is a
[00:05:02] unit point wise mutual information is a very strong baseline for lots of
[00:05:04] very strong baseline for lots of different applications and it also
[00:05:06] different applications and it also embodies a kind of core insight that we
[00:05:09] embodies a kind of core insight that we see running through a lot of the methods
[00:05:10] see running through a lot of the methods that we've covered
[00:05:12] that we've covered so it's a natural and pretty strong
[00:05:13] so it's a natural and pretty strong baseline and what we're asking you to do
[00:05:14] baseline and what we're asking you to do here is simply establish that baseline
[00:05:18] here is simply establish that baseline here and throughout all of the work for
[00:05:20] here and throughout all of the work for this
[00:05:21] this course we're going to ask you to
[00:05:22] course we're going to ask you to implement things and in general we will
[00:05:24] implement things and in general we will provide you with test functions that
[00:05:26] provide you with test functions that will help you make sure you have
[00:05:27] will help you make sure you have iterated towards the solution that we're
[00:05:29] iterated towards the solution that we're looking for so you can feel rest assured
[00:05:32] looking for so you can feel rest assured that if you have meaningfully passed
[00:05:34] that if you have meaningfully passed this test
[00:05:35] this test then you'll do well in terms of the
[00:05:36] then you'll do well in terms of the overall evaluation and your code is
[00:05:38] overall evaluation and your code is functioning as expected
[00:05:41] next question is similar so again now
[00:05:43] next question is similar so again now we're exploring latent semantic analysis
[00:05:45] we're exploring latent semantic analysis and in particular we're asking you to
[00:05:47] and in particular we're asking you to build up some code that will allow you
[00:05:49] build up some code that will allow you to test different dimensionalities for a
[00:05:51] to test different dimensionalities for a given vector space input and try to get
[00:05:53] given vector space input and try to get a feel for which one is best
[00:05:55] a feel for which one is best so again you have to implement a
[00:05:57] so again you have to implement a function and then there's a test that
[00:05:58] function and then there's a test that will help you make sure that you've
[00:06:00] will help you make sure that you've implemented the correct function in case
[00:06:01] implemented the correct function in case there's any uncertainty in the
[00:06:03] there's any uncertainty in the instructions here
[00:06:06] instructions here next question as i mentioned in the
[00:06:08] next question as i mentioned in the lectures t-test free weighting is a very
[00:06:10] lectures t-test free weighting is a very powerful re-weighting scheme it has some
[00:06:12] powerful re-weighting scheme it has some affinities with point-wise mutual
[00:06:14] affinities with point-wise mutual information but it is different
[00:06:16] information but it is different uh and this question is just asking you
[00:06:18] uh and this question is just asking you to implement that rewriting function
[00:06:20] to implement that rewriting function we've given the instructions here you
[00:06:21] we've given the instructions here you might also look in vsm.pi the module at
[00:06:24] might also look in vsm.pi the module at the implementation of pointwise mutual
[00:06:26] the implementation of pointwise mutual information because you could adopt some
[00:06:28] information because you could adopt some of the same techniques
[00:06:30] of the same techniques i want to emphasize that you don't need
[00:06:31] i want to emphasize that you don't need the fastest possible implementation any
[00:06:33] the fastest possible implementation any working implementation will get full
[00:06:35] working implementation will get full credit but the code in bsm.pi is really
[00:06:38] credit but the code in bsm.pi is really nicely optimized in terms of its
[00:06:40] nicely optimized in terms of its implementation so you might want to push
[00:06:41] implementation so you might want to push yourself
[00:06:42] yourself to do something similarly efficient
[00:06:45] to do something similarly efficient but again
[00:06:46] but again as long as your function t test here
[00:06:48] as long as your function t test here passes this test you're in good shape
[00:06:52] passes this test you're in good shape and you don't need to evaluate this
[00:06:53] and you don't need to evaluate this function we're just asking you to
[00:06:54] function we're just asking you to implement it but we're assuming since
[00:06:56] implement it but we're assuming since i've said that this is a good
[00:06:57] i've said that this is a good re-weighting scheme that you'll be
[00:06:58] re-weighting scheme that you'll be curious about how it performs in the
[00:07:00] curious about how it performs in the context context of the system you're
[00:07:02] context context of the system you're developing
[00:07:04] developing all right for the final two questions
[00:07:05] all right for the final two questions we're asking you to think further afield
[00:07:07] we're asking you to think further afield pooled bert representations is drawing
[00:07:09] pooled bert representations is drawing on the material in this notebook here
[00:07:12] on the material in this notebook here which is just an exploration of the
[00:07:14] which is just an exploration of the ideas from bamasani in all 2020 on how
[00:07:17] ideas from bamasani in all 2020 on how to derive static representations from
[00:07:19] to derive static representations from models like bert
[00:07:20] models like bert and so what we've got here is some
[00:07:22] and so what we've got here is some starter code for you and a kind of
[00:07:24] starter code for you and a kind of skeleton for implementing your own
[00:07:26] skeleton for implementing your own version of that solution
[00:07:28] version of that solution again we're hoping that this is a
[00:07:30] again we're hoping that this is a foundation for further exploration for
[00:07:32] foundation for further exploration for you
[00:07:33] you we've got the implementation to do here
[00:07:35] we've got the implementation to do here and then a test that you can pass to
[00:07:36] and then a test that you can pass to make sure that you've implemented things
[00:07:38] make sure that you've implemented things according to the design specification
[00:07:42] according to the design specification the final question is also really
[00:07:43] the final question is also really exploratory it's called learn distance
[00:07:45] exploratory it's called learn distance functions the idea here is that much of
[00:07:47] functions the idea here is that much of the code in this notebook pushes you to
[00:07:49] the code in this notebook pushes you to think about distance in terms of things
[00:07:51] think about distance in terms of things like cosine distance or euclidean
[00:07:54] like cosine distance or euclidean distance
[00:07:55] distance but we should have in mind that the only
[00:07:56] but we should have in mind that the only formal requirement is that you have some
[00:07:59] formal requirement is that you have some function that will map a pair of vectors
[00:08:01] function that will map a pair of vectors into a real valued score
[00:08:03] into a real valued score as soon as you see things from that
[00:08:04] as soon as you see things from that perspective you realize that a whole
[00:08:06] perspective you realize that a whole world of options opens up to you and
[00:08:08] world of options opens up to you and what this question is asking you to do
[00:08:09] what this question is asking you to do is train a k nearest neighbors model on
[00:08:11] is train a k nearest neighbors model on the development data that will learn to
[00:08:13] the development data that will learn to predict scores and then you can use that
[00:08:16] predict scores and then you can use that in place of cosine or euclidean we've
[00:08:18] in place of cosine or euclidean we've walked you through how to implement that
[00:08:19] walked you through how to implement that there's a bunch of guidance here and a
[00:08:21] there's a bunch of guidance here and a few tests for the sub components if you
[00:08:23] few tests for the sub components if you follow our design
[00:08:25] follow our design again if the test pass you should be in
[00:08:26] again if the test pass you should be in good shape we're not asking you to
[00:08:28] good shape we're not asking you to evaluate this directly
[00:08:29] evaluate this directly but we're hoping that this is a
[00:08:31] but we're hoping that this is a foundation for for exploring what could
[00:08:33] foundation for for exploring what could be a quite productive avenue of
[00:08:36] be a quite productive avenue of solutions in this space
[00:08:38] solutions in this space and then finally the original system
[00:08:40] and then finally the original system this is worth three points this is a big
[00:08:42] this is worth three points this is a big deal here you can piece together any
[00:08:44] deal here you can piece together any part of what you've done previously all
[00:08:46] part of what you've done previously all that stuff is fair game you can think in
[00:08:48] that stuff is fair game you can think in entirely original and new ways you can
[00:08:50] entirely original and new ways you can do something simple you can do something
[00:08:52] do something simple you can do something complex
[00:08:53] complex what we'd like you to do here is not
[00:08:54] what we'd like you to do here is not only provide the implementation in the
[00:08:56] only provide the implementation in the scope of this conditional so that it
[00:08:58] scope of this conditional so that it doesn't cause the autograder to fail if
[00:09:00] doesn't cause the autograder to fail if you have special requirements and so
[00:09:02] you have special requirements and so forth but we're also looking for a
[00:09:03] forth but we're also looking for a textual description
[00:09:05] textual description uh and a report on what your highest
[00:09:07] uh and a report on what your highest development set score was
[00:09:10] development set score was the idea here is that at the end of the
[00:09:11] the idea here is that at the end of the bake off the teaching team will create a
[00:09:13] bake off the teaching team will create a report that kind of analyzes across all
[00:09:16] report that kind of analyzes across all the different submissions and reflects
[00:09:18] the different submissions and reflects back to you all what worked and what
[00:09:20] back to you all what worked and what didn't and as part of that effort
[00:09:22] didn't and as part of that effort these system descriptions and
[00:09:24] these system descriptions and development scores can really help us
[00:09:25] development scores can really help us understand um how things played out
[00:09:29] understand um how things played out and that brings us to the bake off so
[00:09:30] and that brings us to the bake off so for the bake off what you really need to
[00:09:32] for the bake off what you really need to do is just run this function create bake
[00:09:34] do is just run this function create bake off submission
[00:09:35] off submission on
[00:09:36] on your vector space model here it's my
[00:09:39] your vector space model here it's my simple one count df that i loaded before
[00:09:41] simple one count df that i loaded before and as a reminder that this is an
[00:09:43] and as a reminder that this is an important piece you almost also need to
[00:09:44] important piece you almost also need to specify some distance function
[00:09:47] specify some distance function so the idea is that here my bake off
[00:09:49] so the idea is that here my bake off would be the simple submission where i'm
[00:09:50] would be the simple submission where i'm just doing account data frame and
[00:09:53] just doing account data frame and euclidean as my distance and when i run
[00:09:55] euclidean as my distance and when i run this function it creates a file cs224u
[00:09:59] this function it creates a file cs224u word relatedness fakeoff entry and
[00:10:01] word relatedness fakeoff entry and you'll just upload that to gradescope
[00:10:03] you'll just upload that to gradescope we'll give some instructions about that
[00:10:04] we'll give some instructions about that later on and that will be evaluated by
[00:10:07] later on and that will be evaluated by an automatic system
[00:10:08] an automatic system and that's it

High-level Goals & Guiding Hypotheses | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=RiQgRJKqEhE

---

Transcript

[00:00:04] hello everyone welcome to the very first
[00:00:06] hello everyone welcome to the very first screencast of the very first unit of our
[00:00:08] screencast of the very first unit of our course we're going to be talking about
[00:00:09] course we're going to be talking about distributed word representations or
[00:00:11] distributed word representations or vector representations of words
[00:00:14] vector representations of words and for this screencast i'm just going
[00:00:15] and for this screencast i'm just going to cover some high-level goals we have
[00:00:17] to cover some high-level goals we have for this unit
[00:00:18] for this unit as well as discuss the guiding
[00:00:19] as well as discuss the guiding hypotheses not only for this unit but
[00:00:21] hypotheses not only for this unit but also hypotheses that will be with us
[00:00:23] also hypotheses that will be with us throughout the quarter
[00:00:26] what i've depicted on the slide here is
[00:00:28] what i've depicted on the slide here is our starting point both conceptually and
[00:00:30] our starting point both conceptually and computationally this is a small fragment
[00:00:32] computationally this is a small fragment of a very large word by word
[00:00:34] of a very large word by word co-occurrence matrix so along the rows
[00:00:37] co-occurrence matrix so along the rows here you have a large vocabulary of
[00:00:38] here you have a large vocabulary of words the first few are emoticons at
[00:00:40] words the first few are emoticons at least word like objects
[00:00:42] least word like objects exactly that same vocabulary is repeated
[00:00:44] exactly that same vocabulary is repeated across the columns
[00:00:46] across the columns and the cell values here give the number
[00:00:48] and the cell values here give the number of times that each row word appeared
[00:00:49] of times that each row word appeared with each column word in a very large
[00:00:52] with each column word in a very large text corpus
[00:00:53] text corpus i think the big idea that you want to
[00:00:55] i think the big idea that you want to start getting used to is that there
[00:00:56] start getting used to is that there could be meaning latent in such
[00:00:58] could be meaning latent in such co-occurrence patterns
[00:01:00] co-occurrence patterns it's not obvious to mere mortals that we
[00:01:02] it's not obvious to mere mortals that we could extract anything about meaning
[00:01:03] could extract anything about meaning from such an abstract space but we're
[00:01:05] from such an abstract space but we're going to see time and time again this is
[00:01:07] going to see time and time again this is actually a very powerful basis for
[00:01:09] actually a very powerful basis for developing meaning representations
[00:01:14] to start building intuition let's do a
[00:01:16] to start building intuition let's do a small thought experiment so imagine that
[00:01:19] small thought experiment so imagine that i give you a small lexicon of words each
[00:01:22] i give you a small lexicon of words each one of them labeled as either negative
[00:01:23] one of them labeled as either negative or positive in the sense of sentiment
[00:01:25] or positive in the sense of sentiment analysis
[00:01:26] analysis now that might be a useful resource but
[00:01:28] now that might be a useful resource but i've called this a hopeless learning
[00:01:30] i've called this a hopeless learning scenario because if i give you four new
[00:01:32] scenario because if i give you four new anonymous words to make predictions on
[00:01:35] anonymous words to make predictions on this resource over here is not useful at
[00:01:37] this resource over here is not useful at all for making predictions in fact you
[00:01:39] all for making predictions in fact you have essentially no information to go on
[00:01:42] have essentially no information to go on about what these anonymous words should
[00:01:43] about what these anonymous words should be labeled
[00:01:46] be labeled contrast that with a situation in which
[00:01:48] contrast that with a situation in which i give you that label lexicon but in
[00:01:50] i give you that label lexicon but in addition
[00:01:51] addition i give you the number of times that each
[00:01:53] i give you the number of times that each lexicon word co-occurs in some large
[00:01:55] lexicon word co-occurs in some large text corpus with the two words excellent
[00:01:58] text corpus with the two words excellent and terrible
[00:01:59] and terrible i think with that information with those
[00:02:01] i think with that information with those columns from the word by word matrix you
[00:02:04] columns from the word by word matrix you can see that you have a lot of
[00:02:05] can see that you have a lot of predictive power in fact a really simple
[00:02:08] predictive power in fact a really simple classifier or even decision rule will be
[00:02:11] classifier or even decision rule will be able to do really well at predicting
[00:02:12] able to do really well at predicting these labels if a word co occurs more
[00:02:15] these labels if a word co occurs more often with terrible than excellent call
[00:02:17] often with terrible than excellent call it negative
[00:02:18] it negative if a word co occurs with excellent more
[00:02:20] if a word co occurs with excellent more often than terrible call it positive
[00:02:22] often than terrible call it positive that's a good predictive model and now
[00:02:24] that's a good predictive model and now if i give you four new anonymous words
[00:02:27] if i give you four new anonymous words and in in addition you're allowed to
[00:02:29] and in in addition you're allowed to collect some co-occurrence information
[00:02:30] collect some co-occurrence information about them with respect to excellent and
[00:02:32] about them with respect to excellent and terrible then your same rule will be
[00:02:34] terrible then your same rule will be able to make really good predictions
[00:02:36] able to make really good predictions about these new anonymous words
[00:02:38] about these new anonymous words that's the sense in which we've moved to
[00:02:40] that's the sense in which we've moved to a very promising learning scenario and
[00:02:42] a very promising learning scenario and it's just a glimpse of how we could
[00:02:44] it's just a glimpse of how we could extract latent information about meaning
[00:02:46] extract latent information about meaning from these co-occurrence patterns and
[00:02:48] from these co-occurrence patterns and now just play it forward and think the
[00:02:50] now just play it forward and think the vector space models that we'll be
[00:02:52] vector space models that we'll be building will have not just two
[00:02:53] building will have not just two dimensions but hundreds or even
[00:02:55] dimensions but hundreds or even thousands of dimensions and there's no
[00:02:57] thousands of dimensions and there's no telling how much information we'll find
[00:02:59] telling how much information we'll find latent in such a high dimensional space
[00:03:03] latent in such a high dimensional space so that brings me to these high-level
[00:03:04] so that brings me to these high-level goals here first we want to begin
[00:03:06] goals here first we want to begin thinking about how these vectors could
[00:03:08] thinking about how these vectors could encode meanings of linguistic units get
[00:03:10] encode meanings of linguistic units get more used to that idea that i just
[00:03:12] more used to that idea that i just introduced you to
[00:03:14] introduced you to these are foundational concepts that
[00:03:16] these are foundational concepts that we'll be discussing not only for our
[00:03:18] we'll be discussing not only for our unit on vector space models which are
[00:03:20] unit on vector space models which are also called embeddings in modern
[00:03:22] also called embeddings in modern parlance but in fact these are
[00:03:24] parlance but in fact these are foundational concepts for all of the
[00:03:25] foundational concepts for all of the more sophisticated deep learning models
[00:03:27] more sophisticated deep learning models that we'll be discussing later on in the
[00:03:29] that we'll be discussing later on in the quarter
[00:03:30] quarter and of course i'm really hoping that
[00:03:32] and of course i'm really hoping that this this material is valuable to you
[00:03:35] this this material is valuable to you throughout the assignments that you do
[00:03:37] throughout the assignments that you do and also valuable for the original
[00:03:38] and also valuable for the original project work that you do in the second
[00:03:40] project work that you do in the second half of the course
[00:03:43] some guiding hypotheses let's start with
[00:03:45] some guiding hypotheses let's start with the literature i would be remiss in a
[00:03:47] the literature i would be remiss in a lecture like this if i didn't quote j.r
[00:03:49] lecture like this if i didn't quote j.r firth you shall know a word by the
[00:03:50] firth you shall know a word by the company it keeps this is a glimpse at
[00:03:54] company it keeps this is a glimpse at the kind of nominalist position that
[00:03:56] the kind of nominalist position that first took about how to do linguistic
[00:03:58] first took about how to do linguistic analysis he's really saying that we
[00:04:00] analysis he's really saying that we should trust distributional information
[00:04:02] should trust distributional information uh his
[00:04:04] uh his zelic harris a linguist working at
[00:04:06] zelic harris a linguist working at around the same time has an even purer
[00:04:08] around the same time has an even purer statement of this hypothesis harris said
[00:04:10] statement of this hypothesis harris said distributional statements can cover all
[00:04:12] distributional statements can cover all of the material of a language without
[00:04:14] of the material of a language without requiring support from other types of
[00:04:16] requiring support from other types of information
[00:04:18] information daily cares really only trusted usage
[00:04:20] daily cares really only trusted usage information i think we don't need to be
[00:04:22] information i think we don't need to be so extreme in our position but we can
[00:04:24] so extreme in our position but we can certainly align with harris in thinking
[00:04:26] certainly align with harris in thinking that there could be a lot about language
[00:04:28] that there could be a lot about language latent in these distributional
[00:04:30] latent in these distributional statements that is in co-occurrence
[00:04:32] statements that is in co-occurrence patterns
[00:04:33] patterns we might as well quote wittgenstein the
[00:04:35] we might as well quote wittgenstein the meaning of a word is its use in the
[00:04:37] meaning of a word is its use in the language i think that's a nice
[00:04:38] language i think that's a nice connection that wittgenstein might have
[00:04:40] connection that wittgenstein might have in mind might be a point of alignment
[00:04:42] in mind might be a point of alignment for him with firth and harris i'm not
[00:04:44] for him with firth and harris i'm not sure
[00:04:45] sure but finally here's a kind of direct
[00:04:47] but finally here's a kind of direct operationalization
[00:04:48] operationalization of our high-level hypothesis this is
[00:04:50] of our high-level hypothesis this is from one of the recommended readings by
[00:04:52] from one of the recommended readings by attorney and pentel and they say if
[00:04:54] attorney and pentel and they say if units of text have similar vectors in a
[00:04:56] units of text have similar vectors in a text frequency matrix like the
[00:04:58] text frequency matrix like the co-occurrence matrix i showed you before
[00:05:00] co-occurrence matrix i showed you before then they tend to have similar meanings
[00:05:02] then they tend to have similar meanings if we buy that hypothesis then we're
[00:05:04] if we buy that hypothesis then we're kind of licensed to build these
[00:05:06] kind of licensed to build these co-occurrence matrices and then make
[00:05:08] co-occurrence matrices and then make inferences about at least similarity of
[00:05:10] inferences about at least similarity of meaning on the basis of those objects
[00:05:12] meaning on the basis of those objects we've constructed
[00:05:15] to finish here under the heading of
[00:05:17] to finish here under the heading of great power a great many design choices
[00:05:20] great power a great many design choices i think one of the difficult things
[00:05:21] i think one of the difficult things about working in this space is that
[00:05:22] about working in this space is that there are a lot of moving pieces the
[00:05:24] there are a lot of moving pieces the first choice you'll have to make is your
[00:05:26] first choice you'll have to make is your matrix design i've talked about the word
[00:05:28] matrix design i've talked about the word by word matrix but of course word by
[00:05:30] by word matrix but of course word by document word by search proximity
[00:05:33] document word by search proximity adjective by modified noun these are all
[00:05:35] adjective by modified noun these are all different ways that you could construct
[00:05:37] different ways that you could construct your rows and your columns in one of
[00:05:38] your rows and your columns in one of these matrices and that's going to be
[00:05:40] these matrices and that's going to be really fundamental you'll capture very
[00:05:42] really fundamental you'll capture very different distributional facts depending
[00:05:44] different distributional facts depending on what kind of matrix design you choose
[00:05:47] on what kind of matrix design you choose and in a way that's not even the first
[00:05:48] and in a way that's not even the first choice that you need to make because in
[00:05:50] choice that you need to make because in constructing this matrix you'll make a
[00:05:52] constructing this matrix you'll make a lot of choices about how to tokenize
[00:05:54] lot of choices about how to tokenize whether to annotate what to do whether
[00:05:56] whether to annotate what to do whether do part of speech tagging for further
[00:05:57] do part of speech tagging for further distinctions parsing feature selection
[00:06:00] distinctions parsing feature selection and so forth and so on you also have to
[00:06:01] and so forth and so on you also have to decide how you're going to group your
[00:06:03] decide how you're going to group your texts is your notion of co-occurrence
[00:06:05] texts is your notion of co-occurrence going to be based on the sentence or the
[00:06:07] going to be based on the sentence or the document or maybe documents clustered by
[00:06:10] document or maybe documents clustered by date or author or discourse context all
[00:06:12] date or author or discourse context all of those things will give you very
[00:06:14] of those things will give you very different notions of what it means to
[00:06:16] different notions of what it means to co-occur and that will feed into your
[00:06:18] co-occur and that will feed into your matrix design
[00:06:20] matrix design having made all of those difficult
[00:06:22] having made all of those difficult choices you now you're probably going to
[00:06:23] choices you now you're probably going to want to take your count matrix and as
[00:06:25] want to take your count matrix and as we'll say re-weight it that is adjust
[00:06:28] we'll say re-weight it that is adjust the values by stretching and bending the
[00:06:30] the values by stretching and bending the space in order to find more latent
[00:06:33] space in order to find more latent information about meaning we're going to
[00:06:34] information about meaning we're going to talk about a lot of methods for doing
[00:06:36] talk about a lot of methods for doing that and then you might furthermore want
[00:06:39] that and then you might furthermore want to do some kind of dimensionality
[00:06:40] to do some kind of dimensionality reduction which is a step you could take
[00:06:42] reduction which is a step you could take to capture even more higher order
[00:06:44] to capture even more higher order notions of co-occurrence beyond the
[00:06:47] notions of co-occurrence beyond the simple co-occurrences that you see
[00:06:48] simple co-occurrences that you see evident in the original matrix that's a
[00:06:51] evident in the original matrix that's a powerful step there are a lot of choices
[00:06:53] powerful step there are a lot of choices you could make there
[00:06:54] you could make there and then finally what's your notion of
[00:06:56] and then finally what's your notion of similarity going to be for us we'll
[00:06:58] similarity going to be for us we'll operationalize that as a vector
[00:07:00] operationalize that as a vector comparison method like euclidean
[00:07:01] comparison method like euclidean distance cosine distance jacquard
[00:07:04] distance cosine distance jacquard distance and so forth
[00:07:06] distance and so forth depending on previous choices that
[00:07:07] depending on previous choices that you've made the choice of vector
[00:07:09] you've made the choice of vector comparison method might have a real
[00:07:11] comparison method might have a real impact on what you regard as similar and
[00:07:13] impact on what you regard as similar and different in your vector space
[00:07:15] different in your vector space so this is a kind of dizzying array of
[00:07:17] so this is a kind of dizzying array of choices that you might have to make
[00:07:19] choices that you might have to make there is a glimmer of hope though so
[00:07:22] there is a glimmer of hope though so models like glove and word vec purport
[00:07:25] models like glove and word vec purport to offer packaged solutions at least to
[00:07:28] to offer packaged solutions at least to the design weighting and reduction steps
[00:07:30] the design weighting and reduction steps here
[00:07:31] here so they'll tell you for instance if you
[00:07:32] so they'll tell you for instance if you use glove that it needs to be word by
[00:07:34] use glove that it needs to be word by word and then glove will simultaneously
[00:07:36] word and then glove will simultaneously perform these two steps and furthermore
[00:07:39] perform these two steps and furthermore for these methods since they tend to
[00:07:40] for these methods since they tend to deliver vectors that are pretty well
[00:07:42] deliver vectors that are pretty well scaled in terms of their individual
[00:07:44] scaled in terms of their individual values the choice of vector comparison
[00:07:46] values the choice of vector comparison might not matter so much so models like
[00:07:49] might not matter so much so models like glove and workvec are a real step
[00:07:51] glove and workvec are a real step forward in terms of taming this space
[00:07:53] forward in terms of taming this space here
[00:07:54] here and we can add further that more recent
[00:07:56] and we can add further that more recent contextual embedding models dictate even
[00:07:59] contextual embedding models dictate even more of the design choices possibly all
[00:08:01] more of the design choices possibly all the way back to how you tokenize and so
[00:08:03] the way back to how you tokenize and so they could be um thought of as even more
[00:08:06] they could be um thought of as even more unified solutions to the great many
[00:08:08] unified solutions to the great many design choices that you have here so
[00:08:10] design choices that you have here so that's kind of conceptually a real
[00:08:13] that's kind of conceptually a real breakthrough i will say though that
[00:08:16] breakthrough i will say though that baseline models constructed from the
[00:08:17] baseline models constructed from the simple things that i have in these
[00:08:19] simple things that i have in these tables here are often competitive with
[00:08:21] tables here are often competitive with more these more advanced models but of
[00:08:23] more these more advanced models but of course which combination is something
[00:08:25] course which combination is something that you'll probably have to discuss
[00:08:27] that you'll probably have to discuss to have to discover empirically

Matrix Designs | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=ladnEW0ntEM

---

Transcript

[00:00:04] hello everyone welcome to part two of
[00:00:06] hello everyone welcome to part two of our series of screencasts on distributed
[00:00:08] our series of screencasts on distributed word representations the focus of this
[00:00:10] word representations the focus of this screencast will be on matrix designs
[00:00:14] screencast will be on matrix designs let's start with the word by word design
[00:00:16] let's start with the word by word design that we concentrated on in part one uh
[00:00:18] that we concentrated on in part one uh so here again we have our vocabulary
[00:00:20] so here again we have our vocabulary along the rows that same vocabulary is
[00:00:23] along the rows that same vocabulary is repeated along the columns and the cell
[00:00:25] repeated along the columns and the cell values capture the number of times that
[00:00:27] values capture the number of times that each row word co-occurred with each
[00:00:29] each row word co-occurred with each column word in some large collection of
[00:00:31] column word in some large collection of texts
[00:00:33] texts this matrix will have two properties
[00:00:34] this matrix will have two properties that i think make it noteworthy for
[00:00:36] that i think make it noteworthy for developing semantic representations the
[00:00:38] developing semantic representations the first is it will be very dense and as we
[00:00:41] first is it will be very dense and as we bring in more data from ever larger
[00:00:43] bring in more data from ever larger corporate will get denser and denser in
[00:00:45] corporate will get denser and denser in virtue of the fact that more words will
[00:00:47] virtue of the fact that more words will tend to co-occur with more other words
[00:00:49] tend to co-occur with more other words in these this ever larger collection of
[00:00:51] in these this ever larger collection of documents
[00:00:53] documents the second is that it kind of has the
[00:00:54] the second is that it kind of has the nice property that its dimensionality
[00:00:56] nice property that its dimensionality will remain fixed even as we bring in
[00:00:58] will remain fixed even as we bring in more data as long as we decide on the
[00:01:00] more data as long as we decide on the vocabulary ahead of time all we'll be
[00:01:02] vocabulary ahead of time all we'll be doing is incrementing individual cell
[00:01:04] doing is incrementing individual cell values and so we can bring in as much
[00:01:05] values and so we can bring in as much data as we want without changing the
[00:01:08] data as we want without changing the fundamental design of the object
[00:01:10] fundamental design of the object both of those other things are points of
[00:01:12] both of those other things are points of contrast with another common design that
[00:01:14] contrast with another common design that you see in the literature especially in
[00:01:16] you see in the literature especially in information retrieval and that is the
[00:01:18] information retrieval and that is the word by document design
[00:01:20] word by document design for this design again i have words along
[00:01:22] for this design again i have words along the rows but my columns are now
[00:01:24] the rows but my columns are now individual documents and the cell values
[00:01:27] individual documents and the cell values capture the number of times that each
[00:01:28] capture the number of times that each word occurs in each one of those
[00:01:30] word occurs in each one of those documents
[00:01:31] documents as you can imagine this is a very sparse
[00:01:34] as you can imagine this is a very sparse matrix in contrast to the word by word
[00:01:36] matrix in contrast to the word by word one that we just looked at in virtue of
[00:01:38] one that we just looked at in virtue of the fact that most words don't appear in
[00:01:40] the fact that most words don't appear in most documents
[00:01:41] most documents it will also have the property that as
[00:01:43] it will also have the property that as we bring in more data in the form of
[00:01:45] we bring in more data in the form of more documents
[00:01:46] more documents the shape of the matrix will change
[00:01:48] the shape of the matrix will change we'll be adding column dimensions for
[00:01:50] we'll be adding column dimensions for each new document that we bring into the
[00:01:52] each new document that we bring into the space and that could really affect the
[00:01:55] space and that could really affect the kind of complications that we can do the
[00:01:57] kind of complications that we can do the only thing that balances against the
[00:01:59] only thing that balances against the ever increasing size of this matrix is
[00:02:01] ever increasing size of this matrix is that because it is so sparse we might
[00:02:02] that because it is so sparse we might have some easy and efficient ways of
[00:02:04] have some easy and efficient ways of storing it efficiently putting it on par
[00:02:06] storing it efficiently putting it on par with a much more compact but dense word
[00:02:09] with a much more compact but dense word by word matrix that i showed you before
[00:02:12] by word matrix that i showed you before now those are two very common designs
[00:02:14] now those are two very common designs that you see in the literature but i
[00:02:15] that you see in the literature but i want you to think creatively and kind of
[00:02:17] want you to think creatively and kind of align your matrix design with it at
[00:02:19] align your matrix design with it at whatever problem you're trying to solve
[00:02:21] whatever problem you're trying to solve so let me show you one that's really
[00:02:22] so let me show you one that's really radically different
[00:02:24] radically different this is what i've called the word by
[00:02:25] this is what i've called the word by discourse context matrix
[00:02:27] discourse context matrix i derived this from the switchboard
[00:02:29] i derived this from the switchboard dialogue act corpus which is the
[00:02:31] dialogue act corpus which is the switchboard corpus where each dialogue
[00:02:33] switchboard corpus where each dialogue act has been annotated by an expert
[00:02:35] act has been annotated by an expert annotator with the sort of dialogue act
[00:02:37] annotator with the sort of dialogue act or speech act that was performed by that
[00:02:39] or speech act that was performed by that utterance
[00:02:41] utterance what that allows us to do is collect
[00:02:43] what that allows us to do is collect a matrix where the rows are again words
[00:02:45] a matrix where the rows are again words but the columns are those individual
[00:02:47] but the columns are those individual labels that annotators assigned i think
[00:02:50] labels that annotators assigned i think this is a really interesting matrix i
[00:02:52] this is a really interesting matrix i think if you appear even at this small
[00:02:53] think if you appear even at this small fragment you can see some interesting
[00:02:55] fragment you can see some interesting information emerging so for example
[00:02:57] information emerging so for example absolutely occurs a lot in acceptance
[00:03:00] absolutely occurs a lot in acceptance dialogue acts whereas more hedged words
[00:03:02] dialogue acts whereas more hedged words like actually in any way are more common
[00:03:05] like actually in any way are more common in things like rejecting part of a
[00:03:07] in things like rejecting part of a previous utterance and i'm sure there
[00:03:09] previous utterance and i'm sure there are lots of other interesting patterns
[00:03:10] are lots of other interesting patterns in this matrix
[00:03:12] in this matrix and of course that's just a glimpse of
[00:03:14] and of course that's just a glimpse of the many other design choices that you
[00:03:16] the many other design choices that you could make again think creatively you
[00:03:18] could make again think creatively you could have something like adjective by
[00:03:19] could have something like adjective by modified noun this would probably
[00:03:21] modified noun this would probably capture some very local syntactic
[00:03:23] capture some very local syntactic information or collocational information
[00:03:26] information or collocational information we could generalize that a bit to word
[00:03:28] we could generalize that a bit to word by syntactic context to explicitly try
[00:03:30] by syntactic context to explicitly try to model how words associate with
[00:03:33] to model how words associate with specific syntactic structures
[00:03:35] specific syntactic structures be very different from our usual
[00:03:37] be very different from our usual semantic goals for this course
[00:03:39] semantic goals for this course word by search query might be a design
[00:03:41] word by search query might be a design that you use information retrieval we
[00:03:43] that you use information retrieval we don't even have to limit this to
[00:03:44] don't even have to limit this to linguistic objects word by person could
[00:03:46] linguistic objects word by person could capture the number of times that each
[00:03:48] capture the number of times that each person
[00:03:49] person purchased a specific set of products and
[00:03:52] purchased a specific set of products and then we could cluster people or products
[00:03:54] then we could cluster people or products on that basis
[00:03:56] on that basis we could also mix linguistic and
[00:03:57] we could also mix linguistic and non-linguistic things so word by person
[00:03:59] non-linguistic things so word by person might capture different usage patterns
[00:04:01] might capture different usage patterns for individual speakers and again again
[00:04:03] for individual speakers and again again allow us to doing some kind of
[00:04:05] allow us to doing some kind of interesting clustering of words or of
[00:04:07] interesting clustering of words or of people
[00:04:08] people we could also break out of two
[00:04:10] we could also break out of two dimensions we could have something like
[00:04:11] dimensions we could have something like word by word by pattern or verb by
[00:04:14] word by word by pattern or verb by subject by object many of the methods
[00:04:16] subject by object many of the methods that we cover in this unit are easily
[00:04:19] that we cover in this unit are easily generalized to more than two dimensions
[00:04:21] generalized to more than two dimensions so you could have that in mind and of
[00:04:22] so you could have that in mind and of course as i said think creatively and
[00:04:24] course as i said think creatively and think in particular about how your
[00:04:26] think in particular about how your matrix design is aligned with whatever
[00:04:29] matrix design is aligned with whatever modeling goal you have or whatever
[00:04:31] modeling goal you have or whatever hypothesis you're pursuing
[00:04:33] hypothesis you're pursuing another connection that i want to make
[00:04:35] another connection that i want to make is that even though this feels like a
[00:04:36] is that even though this feels like a kind of modern idea in nlp
[00:04:39] kind of modern idea in nlp vector representations of words are
[00:04:41] vector representations of words are actually are of objects are actually
[00:04:44] actually are of objects are actually pervasive not only throughout machine
[00:04:46] pervasive not only throughout machine learning but also throughout science
[00:04:48] learning but also throughout science right so think back to older modes of
[00:04:50] right so think back to older modes of nlp where we would write a lot of
[00:04:52] nlp where we would write a lot of feature functions we'll be exploring
[00:04:54] feature functions we'll be exploring such techniques they can be quite
[00:04:55] such techniques they can be quite powerful even though they feel very
[00:04:58] powerful even though they feel very different from the distributional
[00:04:59] different from the distributional hypotheses that we've been pursuing in
[00:05:01] hypotheses that we've been pursuing in fact they also represent individual data
[00:05:04] fact they also represent individual data points as vectors so for example given
[00:05:06] points as vectors so for example given the text like the movie was horrible i
[00:05:09] the text like the movie was horrible i might reduce that with my feature
[00:05:10] might reduce that with my feature functions to a vector that looks like
[00:05:12] functions to a vector that looks like this and i might know as a human that
[00:05:14] this and i might know as a human that four captures the number of words
[00:05:17] four captures the number of words zero captures the number of proper names
[00:05:19] zero captures the number of proper names and one over four captures the
[00:05:21] and one over four captures the percentage of negative words according
[00:05:23] percentage of negative words according to some sentiment lexicon that's a human
[00:05:26] to some sentiment lexicon that's a human level understanding of this in fact
[00:05:28] level understanding of this in fact those dimensions will acquire a meaning
[00:05:30] those dimensions will acquire a meaning to the extent that they assemble them
[00:05:32] to the extent that they assemble them into a vector space model
[00:05:34] into a vector space model and the column wise elements are
[00:05:36] and the column wise elements are compared with each other
[00:05:38] compared with each other so even though the origins of the data
[00:05:40] so even though the origins of the data are very different in fact this is just
[00:05:42] are very different in fact this is just like
[00:05:43] like vector representations of words in the
[00:05:45] vector representations of words in the way we've been discussing it the same
[00:05:47] way we've been discussing it the same thing happens in experimental sciences
[00:05:49] thing happens in experimental sciences where you might have an experimental
[00:05:50] where you might have an experimental subject come in and perform some act in
[00:05:52] subject come in and perform some act in the lab they do a complicated physical
[00:05:54] the lab they do a complicated physical and human thing and you reduce it down
[00:05:56] and human thing and you reduce it down to a couple of numbers like a choice
[00:05:58] to a couple of numbers like a choice they made or a reaction time and a
[00:06:01] they made or a reaction time and a choice and so forth
[00:06:03] choice and so forth we might model entire humans or entire
[00:06:05] we might model entire humans or entire entire organisms with a vector of
[00:06:07] entire organisms with a vector of numbers representing their physical
[00:06:10] numbers representing their physical characteristics and perspectives and
[00:06:12] characteristics and perspectives and outlooks and so forth again we might
[00:06:14] outlooks and so forth again we might know what these individual column
[00:06:16] know what these individual column dimensions mean but they acquire a
[00:06:18] dimensions mean but they acquire a meaning when we're doing modeling only
[00:06:20] meaning when we're doing modeling only to the extent that they are embedded in
[00:06:22] to the extent that they are embedded in a matrix and can be compared to each
[00:06:24] a matrix and can be compared to each other across the columns
[00:06:26] other across the columns and so forth there are many other
[00:06:28] and so forth there are many other examples of this where essentially
[00:06:30] examples of this where essentially fundamentally all of our representations
[00:06:32] fundamentally all of our representations are vector representations so maybe the
[00:06:34] are vector representations so maybe the far out idea for this unit is just that
[00:06:37] far out idea for this unit is just that we can gather interesting vector
[00:06:39] we can gather interesting vector representations without all of the hand
[00:06:41] representations without all of the hand built work that goes into the examples
[00:06:43] built work that goes into the examples on the slide right now
[00:06:46] a final technical point a question that
[00:06:49] a final technical point a question that you should ask that's kind of separate
[00:06:50] you should ask that's kind of separate from your particular
[00:06:52] from your particular matrix design what is going to count as
[00:06:54] matrix design what is going to count as co-occurrence so i think there are at
[00:06:56] co-occurrence so i think there are at least two design choices that are really
[00:06:58] least two design choices that are really important when answering this question
[00:07:00] important when answering this question to illustrate them let's use this small
[00:07:02] to illustrate them let's use this small example so i have this text from swerve
[00:07:04] example so i have this text from swerve of shore to bend of bay comma brings
[00:07:08] of shore to bend of bay comma brings and imagine that our focus word at our
[00:07:09] and imagine that our focus word at our particular point of analysis is this
[00:07:11] particular point of analysis is this token of the word two
[00:07:13] token of the word two and these indices here indicate going
[00:07:15] and these indices here indicate going left and right the distance by counts
[00:07:18] left and right the distance by counts from that particular focus word
[00:07:22] from that particular focus word the first question that you want to
[00:07:23] the first question that you want to decide is what your window of
[00:07:25] decide is what your window of co-occurrence is going to be so for
[00:07:27] co-occurrence is going to be so for example if you set your window to 3
[00:07:29] example if you set your window to 3 then the things that are within three
[00:07:31] then the things that are within three distance of your focus word will
[00:07:33] distance of your focus word will co-occur with that word and everything
[00:07:35] co-occur with that word and everything falling outside of that window will not
[00:07:37] falling outside of that window will not co-occur with that word according to
[00:07:39] co-occur with that word according to your analysis
[00:07:41] your analysis if you make your window really big it
[00:07:42] if you make your window really big it might encompass the entire document if
[00:07:44] might encompass the entire document if you make it very small it might
[00:07:46] you make it very small it might encompass only very local kind of
[00:07:48] encompass only very local kind of collocational information so you can bet
[00:07:51] collocational information so you can bet that that's going to be meaningful
[00:07:53] that that's going to be meaningful there's a separate choice that you can
[00:07:55] there's a separate choice that you can make falling under the heading of
[00:07:56] make falling under the heading of scaling i think a default choice for
[00:07:59] scaling i think a default choice for scaling is to just call it flat so what
[00:08:01] scaling is to just call it flat so what you're saying there is something is
[00:08:03] you're saying there is something is going to co-occur once with your focus
[00:08:05] going to co-occur once with your focus word if it's in the window that you've
[00:08:07] word if it's in the window that you've specified and that would kind of equally
[00:08:09] specified and that would kind of equally weight all of the things that are in the
[00:08:11] weight all of the things that are in the window you could also decide to scale
[00:08:13] window you could also decide to scale them a common scaling pattern would be
[00:08:15] them a common scaling pattern would be one over n where n is the distance by
[00:08:18] one over n where n is the distance by word from your focus word that would
[00:08:21] word from your focus word that would have the effect that things occurred
[00:08:22] have the effect that things occurred that occur close to the word of interest
[00:08:24] that occur close to the word of interest a co-occur with it more than things that
[00:08:26] a co-occur with it more than things that are at the edges that are near the end
[00:08:28] are at the edges that are near the end of the window
[00:08:32] those choices are going to have really
[00:08:34] those choices are going to have really profound effects on the kinds of
[00:08:35] profound effects on the kinds of representations that you develop here
[00:08:37] representations that you develop here are some generalizations i could offer
[00:08:39] are some generalizations i could offer larger flatter windows will capture more
[00:08:42] larger flatter windows will capture more semantic information as the window gets
[00:08:44] semantic information as the window gets very large to encompass for example the
[00:08:46] very large to encompass for example the entire document you'll be capturing
[00:08:48] entire document you'll be capturing essentially topical information in
[00:08:50] essentially topical information in contrast if you make your window very
[00:08:52] contrast if you make your window very small and scaled you'll tend to capture
[00:08:54] small and scaled you'll tend to capture more syntactic or collocational
[00:08:56] more syntactic or collocational information
[00:08:59] information independently of these choices you could
[00:09:00] independently of these choices you could decide how text boundaries are going to
[00:09:02] decide how text boundaries are going to be involved so a text boundary at the
[00:09:04] be involved so a text boundary at the level of a sentence or a paragraph for a
[00:09:06] level of a sentence or a paragraph for a document or a corpus could be a hard
[00:09:08] document or a corpus could be a hard boundary that's independent of your
[00:09:10] boundary that's independent of your window or you could decide that you're
[00:09:12] window or you could decide that you're going to allow your window to go across
[00:09:13] going to allow your window to go across different notions of segment that you
[00:09:15] different notions of segment that you have that's really up to you and again i
[00:09:17] have that's really up to you and again i think it will have major consequences
[00:09:19] think it will have major consequences for downstream tasks involving the
[00:09:21] for downstream tasks involving the representations that you've created
[00:09:24] representations that you've created to help you begin exploring this space
[00:09:26] to help you begin exploring this space the associated code released for this
[00:09:28] the associated code released for this course the associated notebooks provide
[00:09:31] course the associated notebooks provide you with four word by word matrices they
[00:09:34] you with four word by word matrices they have a few things that allow you to do
[00:09:36] have a few things that allow you to do comparisons first there are two matrices
[00:09:38] comparisons first there are two matrices that were developed from the yelp
[00:09:40] that were developed from the yelp academic data set which is a lot of
[00:09:42] academic data set which is a lot of reviews of products and services and
[00:09:44] reviews of products and services and there are two matrices that come from
[00:09:46] there are two matrices that come from gigaword which is newswire text so
[00:09:48] gigaword which is newswire text so there's fundamentally a real difference
[00:09:50] there's fundamentally a real difference in the genre of text involved
[00:09:53] in the genre of text involved in addition for each of those pairs of
[00:09:54] in addition for each of those pairs of corp
[00:09:55] corp for each of those corpora we have two
[00:09:57] for each of those corpora we have two different designs
[00:09:58] different designs window size of five and scaling of one
[00:10:01] window size of five and scaling of one over n which ought to by my hypotheses
[00:10:03] over n which ought to by my hypotheses deliver a lot of kind of collocational
[00:10:05] deliver a lot of kind of collocational or syntactic information
[00:10:08] or syntactic information and window size of 20 and scaling a flat
[00:10:10] and window size of 20 and scaling a flat a very large window lots of things
[00:10:13] a very large window lots of things co-occurring with lots of other things
[00:10:15] co-occurring with lots of other things that might be a better basis for
[00:10:16] that might be a better basis for semantics and you have those two points
[00:10:18] semantics and you have those two points of variation both for yelp and for giga
[00:10:20] of variation both for yelp and for giga word and i'm hoping that that kind of
[00:10:22] word and i'm hoping that that kind of gives you a sense for how these design
[00:10:24] gives you a sense for how these design choices affect the representations that
[00:10:27] choices affect the representations that you're able to develop
[00:10:28] you're able to develop with methods that we're going to cover
[00:10:30] with methods that we're going to cover in later parts of the screencast series

Vector Comparison | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=eKvbYOc2rOs

---

Transcript

[00:00:05] welcome back everyone this is part three
[00:00:06] welcome back everyone this is part three in our series on distributed word
[00:00:08] in our series on distributed word representations we're going to be
[00:00:09] representations we're going to be talking about vector comparison methods
[00:00:12] talking about vector comparison methods to try to make this discussion pretty
[00:00:13] to try to make this discussion pretty intuitive i'm going to ground things in
[00:00:15] intuitive i'm going to ground things in this running example on the left i have
[00:00:18] this running example on the left i have a very small vector space model we have
[00:00:20] a very small vector space model we have three words a b and c
[00:00:22] three words a b and c and you can imagine that we've measured
[00:00:23] and you can imagine that we've measured two dimensions dx and dy you could think
[00:00:25] two dimensions dx and dy you could think of them as documents if you wanted
[00:00:28] of them as documents if you wanted there are two perspectives that you
[00:00:29] there are two perspectives that you might take on this vector space model
[00:00:31] might take on this vector space model the first is just at the level of raw
[00:00:34] the first is just at the level of raw frequency
[00:00:35] frequency b and c seem to be united they are
[00:00:36] b and c seem to be united they are frequent in both the x and the y
[00:00:38] frequent in both the x and the y dimension
[00:00:39] dimension whereas a is comparatively infrequent
[00:00:42] whereas a is comparatively infrequent along both those dimensions
[00:00:44] along both those dimensions that's the first perspective the second
[00:00:46] that's the first perspective the second perspective though is more subtle you
[00:00:49] perspective though is more subtle you might just observe that if we kind of
[00:00:50] might just observe that if we kind of correct for the overall frequency
[00:00:54] correct for the overall frequency of the individual words then it's
[00:00:55] of the individual words then it's actually a and b that are united because
[00:00:57] actually a and b that are united because they both have a bias in some sense for
[00:00:59] they both have a bias in some sense for the dy dimension
[00:01:01] the dy dimension whereas by comparison c has a bias for
[00:01:04] whereas by comparison c has a bias for the x dimension again thinking
[00:01:06] the x dimension again thinking proportionally
[00:01:07] proportionally both of those are perspectives that we
[00:01:09] both of those are perspectives that we might want to capture and different
[00:01:11] might want to capture and different notions of distance will key into one or
[00:01:13] notions of distance will key into one or the other of them
[00:01:16] the other of them one more preliminary uh i think it's
[00:01:18] one more preliminary uh i think it's very intuitive to depict these vector
[00:01:20] very intuitive to depict these vector spaces and in only two dimensions that's
[00:01:22] spaces and in only two dimensions that's pretty easy you can imagine that this is
[00:01:24] pretty easy you can imagine that this is the dx dimension along the x-axis and
[00:01:26] the dx dimension along the x-axis and this is the d-y dimension along the
[00:01:28] this is the d-y dimension along the y-axis and then i have placed these
[00:01:30] y-axis and then i have placed these individual points in that plane and then
[00:01:33] individual points in that plane and then you can see graphically that b and c are
[00:01:35] you can see graphically that b and c are pretty close together and a is kind of
[00:01:37] pretty close together and a is kind of lonely down here in the corner the
[00:01:39] lonely down here in the corner the infrequent one
[00:01:42] let's start with euclidean distance very
[00:01:44] let's start with euclidean distance very common notion of distance in these
[00:01:46] common notion of distance in these spaces and quite intuitive
[00:01:48] spaces and quite intuitive we can measure the euclidean distance
[00:01:50] we can measure the euclidean distance between vectors u and v if they share
[00:01:52] between vectors u and v if they share the same dimension n by just calculating
[00:01:55] the same dimension n by just calculating the sum
[00:01:56] the sum of the squared element y differences
[00:01:58] of the squared element y differences absolute differences and then taking the
[00:02:00] absolute differences and then taking the square root of that
[00:02:02] square root of that that's the math here let's look at that
[00:02:04] that's the math here let's look at that in terms of this space so here we have
[00:02:06] in terms of this space so here we have our vector space
[00:02:07] our vector space depicted graphically a b and c and
[00:02:10] depicted graphically a b and c and euclidean distance is measuring the
[00:02:11] euclidean distance is measuring the length of these lines i've annotated
[00:02:14] length of these lines i've annotated with the full calculations but the
[00:02:15] with the full calculations but the intuition is just that we are measuring
[00:02:17] intuition is just that we are measuring the length of these lines the most
[00:02:19] the length of these lines the most direct path between these points in our
[00:02:21] direct path between these points in our high dimensional space
[00:02:23] high dimensional space and you can see that euclidean distance
[00:02:25] and you can see that euclidean distance is capturing the first
[00:02:26] is capturing the first perspective that we took on the vector
[00:02:28] perspective that we took on the vector space which unites the frequent items b
[00:02:31] space which unites the frequent items b and c as against the infrequent one a
[00:02:37] as a stepping stone toward cosine
[00:02:38] as a stepping stone toward cosine distance which will behave quite
[00:02:40] distance which will behave quite differently let's talk about length
[00:02:41] differently let's talk about length normalization
[00:02:43] normalization given a vector u of dimension n the l2
[00:02:45] given a vector u of dimension n the l2 length of u
[00:02:47] length of u is the sum of the squared values in that
[00:02:50] is the sum of the squared values in that matrix and then we take the square root
[00:02:52] matrix and then we take the square root that's our normalization quantity there
[00:02:55] that's our normalization quantity there and then the actual normalization of
[00:02:57] and then the actual normalization of that original vector u involves taking
[00:02:59] that original vector u involves taking each one of its elements and dividing it
[00:03:02] each one of its elements and dividing it by that fixed quantity the l2 ranks
[00:03:05] by that fixed quantity the l2 ranks let's look look at what happens to our
[00:03:07] let's look look at what happens to our little illustrative example on the left
[00:03:09] little illustrative example on the left here i have the original count matrix
[00:03:11] here i have the original count matrix and in this column here i've given the
[00:03:13] and in this column here i've given the l2 length as a quantity
[00:03:15] l2 length as a quantity and then when we take that quantity and
[00:03:17] and then when we take that quantity and divide each one of the values in that
[00:03:19] divide each one of the values in that vector to get its l2 norm
[00:03:21] vector to get its l2 norm you can see that we've done something
[00:03:23] you can see that we've done something significant to the space so they're all
[00:03:25] significant to the space so they're all kind of united on the same scale here
[00:03:27] kind of united on the same scale here and a and b are now close together
[00:03:30] and a and b are now close together whereas b and c are comparatively far
[00:03:32] whereas b and c are comparatively far apart so that is capturing the second
[00:03:34] apart so that is capturing the second perspective that we took on the matrix
[00:03:36] perspective that we took on the matrix where a and b have something in common
[00:03:38] where a and b have something in common as against c
[00:03:39] as against c and that has come entirely from the
[00:03:41] and that has come entirely from the normalization step and if we measured
[00:03:43] normalization step and if we measured euclidean distance in this space just
[00:03:45] euclidean distance in this space just the length of the lines between these
[00:03:47] the length of the lines between these points we would again be capturing that
[00:03:49] points we would again be capturing that a and b are alike and b and c are
[00:03:51] a and b are alike and b and c are comparatively different
[00:03:54] cosine kind of does that all in one step
[00:03:57] cosine kind of does that all in one step so the cosine distance or approximately
[00:03:59] so the cosine distance or approximately the distance as you'll see between two
[00:04:01] the distance as you'll see between two vectors u and v of dimension share
[00:04:03] vectors u and v of dimension share dimension n
[00:04:04] dimension n this calculation has two parts this is
[00:04:06] this calculation has two parts this is the similarity calculation cosine
[00:04:08] the similarity calculation cosine similarity and it is the dot product of
[00:04:10] similarity and it is the dot product of the two vectors divided by the product
[00:04:13] the two vectors divided by the product of their l2 lengths
[00:04:15] of their l2 lengths and then to get something like the
[00:04:16] and then to get something like the distance we just take one and subtract
[00:04:18] distance we just take one and subtract out that similarity
[00:04:20] out that similarity again let's ground this in our example
[00:04:22] again let's ground this in our example here we have the original count vector
[00:04:24] here we have the original count vector space model
[00:04:26] space model and what we do with cosine distance is
[00:04:29] and what we do with cosine distance is essentially measure the angles between
[00:04:32] essentially measure the angles between these lines that i've drawn from this
[00:04:34] these lines that i've drawn from this origin point
[00:04:35] origin point and so you can see that cosine distance
[00:04:38] and so you can see that cosine distance is capturing the fact that a and b are
[00:04:40] is capturing the fact that a and b are close together as measured by this angle
[00:04:42] close together as measured by this angle whereas b and c are comparatively far
[00:04:44] whereas b and c are comparatively far apart
[00:04:45] apart so again with cosine we're abstracting
[00:04:48] so again with cosine we're abstracting away from frequency information
[00:04:50] away from frequency information and keying into that abstract notion of
[00:04:52] and keying into that abstract notion of similarity that connects a and b as
[00:04:54] similarity that connects a and b as against c
[00:04:58] another perspective that you could take
[00:05:00] another perspective that you could take is just observe that if we first
[00:05:03] is just observe that if we first normalize
[00:05:04] normalize the vectors
[00:05:06] the vectors via via the l2 norm and then apply the
[00:05:09] via via the l2 norm and then apply the cosine calculation we change the space
[00:05:11] cosine calculation we change the space as i showed you before so they're all up
[00:05:13] as i showed you before so they're all up here kind of on the unit sphere
[00:05:15] here kind of on the unit sphere and notice that the actual values that
[00:05:18] and notice that the actual values that we get out are the same
[00:05:20] we get out are the same whether or not we did that l2 norming
[00:05:22] whether or not we did that l2 norming step and that is because cosine is
[00:05:24] step and that is because cosine is building the effects of l2 norming
[00:05:26] building the effects of l2 norming directly into this normalization here in
[00:05:29] directly into this normalization here in the denominator
[00:05:33] there are a few other methods that we
[00:05:35] there are a few other methods that we could think about or classes of methods
[00:05:36] could think about or classes of methods i think we don't need to get distracted
[00:05:38] i think we don't need to get distracted by the details but i thought i would
[00:05:39] by the details but i thought i would mention them in case they come up and
[00:05:40] mention them in case they come up and you're reading our research the first
[00:05:42] you're reading our research the first class are what are what i've called
[00:05:44] class are what are what i've called matching based methods they're all kind
[00:05:45] matching based methods they're all kind of based in this matching coefficient
[00:05:48] of based in this matching coefficient and then jacquard dice and overlap are
[00:05:50] and then jacquard dice and overlap are terms that you might see in the
[00:05:51] terms that you might see in the literature
[00:05:52] literature these are often defined only for binary
[00:05:54] these are often defined only for binary vectors but here i've given their
[00:05:55] vectors but here i've given their generalizations to the real valued
[00:05:58] generalizations to the real valued vectors that we're talking about
[00:06:00] vectors that we're talking about and the other class of methods that you
[00:06:01] and the other class of methods that you might see come up are probabilistic
[00:06:03] might see come up are probabilistic methods which tend to be grounded in
[00:06:05] methods which tend to be grounded in this notion of kl divergence
[00:06:07] this notion of kl divergence kl divergence is essentially a way of
[00:06:09] kl divergence is essentially a way of measuring the distance between two
[00:06:11] measuring the distance between two probability distributions
[00:06:13] probability distributions between
[00:06:15] between to be more precise from a reference
[00:06:17] to be more precise from a reference distribution p to some other probability
[00:06:20] distribution p to some other probability distribution q
[00:06:21] distribution q um and it has symmetric notions
[00:06:24] um and it has symmetric notions symmetric kl and jensen shannon distance
[00:06:27] symmetric kl and jensen shannon distance which is another symmetric notion that's
[00:06:28] which is another symmetric notion that's based in kl divergence again these are
[00:06:31] based in kl divergence again these are probably appropriate measures to choose
[00:06:33] probably appropriate measures to choose if the quantities that you're thinking
[00:06:35] if the quantities that you're thinking of are appropriately thought of as
[00:06:37] of are appropriately thought of as probability values
[00:06:41] now i've alluded to the fact that the
[00:06:43] now i've alluded to the fact that the cosine distance measure that i gave you
[00:06:45] cosine distance measure that i gave you before is not quite what's called a
[00:06:47] before is not quite what's called a proper distance metric let me expand on
[00:06:49] proper distance metric let me expand on that a little bit
[00:06:50] that a little bit to qualify as a proper distance metric a
[00:06:53] to qualify as a proper distance metric a vector comparison method has to have
[00:06:55] vector comparison method has to have three properties it needs to be
[00:06:57] three properties it needs to be symmetric that is it needs to give the
[00:06:59] symmetric that is it needs to give the same value for x y as it does to yx
[00:07:02] same value for x y as it does to yx kl divergence actually fails that first
[00:07:05] kl divergence actually fails that first rule
[00:07:06] rule it needs to assign zero to identical
[00:07:08] it needs to assign zero to identical vectors
[00:07:09] vectors and crucially it needs to satisfy what's
[00:07:11] and crucially it needs to satisfy what's called the triangle inequality which
[00:07:13] called the triangle inequality which says that the distance between x and z
[00:07:17] says that the distance between x and z is less than or equal to the distance
[00:07:19] is less than or equal to the distance between x and y and then y to z
[00:07:23] between x and y and then y to z cosine distance as i showed it to you
[00:07:25] cosine distance as i showed it to you before fails to satisfy the triangle
[00:07:28] before fails to satisfy the triangle inequality and this is just a simple
[00:07:29] inequality and this is just a simple example that makes an intuitive it just
[00:07:32] example that makes an intuitive it just happens that this distance here is
[00:07:34] happens that this distance here is actually greater than these two values
[00:07:36] actually greater than these two values here
[00:07:37] here which is a failure of the statement of
[00:07:38] which is a failure of the statement of the triangle inequality
[00:07:40] the triangle inequality now this is relatively easily corrected
[00:07:43] now this is relatively easily corrected but this is also kind of a useful
[00:07:44] but this is also kind of a useful framework of all the different choices
[00:07:47] framework of all the different choices that we could make
[00:07:48] that we could make of all the options for vector comparison
[00:07:50] of all the options for vector comparison suppose we decided to favor the ones
[00:07:52] suppose we decided to favor the ones that counted as true distance metrics
[00:07:55] that counted as true distance metrics then that would at least push us to
[00:07:56] then that would at least push us to favor euclidean distance
[00:07:59] favor euclidean distance jacquard for binary vectors only and
[00:08:01] jacquard for binary vectors only and jensen shannon distance if we were
[00:08:03] jensen shannon distance if we were talking about probabilistic spaces and
[00:08:05] talking about probabilistic spaces and we would further amend the definition of
[00:08:07] we would further amend the definition of cosine distance to the more careful one
[00:08:09] cosine distance to the more careful one that i've given here
[00:08:11] that i've given here which which satisfies the triangle
[00:08:13] which which satisfies the triangle inequality as well as the other two
[00:08:15] inequality as well as the other two criteria
[00:08:16] criteria and by this kind of way of dividing the
[00:08:18] and by this kind of way of dividing the world we would also reject
[00:08:20] world we would also reject matching jaccard dice overlap
[00:08:23] matching jaccard dice overlap um kale divergence and symmetric tail
[00:08:25] um kale divergence and symmetric tail divergence as ones that failed to be
[00:08:27] divergence as ones that failed to be proper distance metrics and so that
[00:08:29] proper distance metrics and so that might be a useful framework for thinking
[00:08:31] might be a useful framework for thinking about choices in this space
[00:08:33] about choices in this space one other point in relation to this this
[00:08:36] one other point in relation to this this is obviously a more involved calculation
[00:08:38] is obviously a more involved calculation than the one that i gave you before and
[00:08:40] than the one that i gave you before and in truth it is probably not worth the
[00:08:42] in truth it is probably not worth the effort here's an example of just a bunch
[00:08:44] effort here's an example of just a bunch of vectors that i sampled from one of
[00:08:46] of vectors that i sampled from one of our vector space models and i've
[00:08:48] our vector space models and i've compared the improper cosine distance
[00:08:50] compared the improper cosine distance that i showed you before on the x-axis
[00:08:52] that i showed you before on the x-axis with the proper cosine distance measure
[00:08:55] with the proper cosine distance measure that i just showed you
[00:08:56] that i just showed you and the correlation between the two is
[00:08:58] and the correlation between the two is almost perfect so there is essentially
[00:09:00] almost perfect so there is essentially no difference between these two
[00:09:02] no difference between these two different ways of measuring cosine
[00:09:05] different ways of measuring cosine and i think that they are probably
[00:09:06] and i think that they are probably essentially identical up to ranking
[00:09:08] essentially identical up to ranking which is often the quantity that we care
[00:09:10] which is often the quantity that we care about when we're doing these comparisons
[00:09:12] about when we're doing these comparisons so probably stick with a simpler and
[00:09:14] so probably stick with a simpler and less involved calculation would be my
[00:09:16] less involved calculation would be my advice
[00:09:18] let's close with some generalizations
[00:09:20] let's close with some generalizations and relationships first euclidean as
[00:09:22] and relationships first euclidean as well as jaccard and dice with raw count
[00:09:24] well as jaccard and dice with raw count vectors will tend to favor raw frequency
[00:09:27] vectors will tend to favor raw frequency over other distributional patterns like
[00:09:29] over other distributional patterns like that more abstract one that i showed you
[00:09:31] that more abstract one that i showed you with our illustrative example
[00:09:34] with our illustrative example euclidean with l2 norm vectors is
[00:09:36] euclidean with l2 norm vectors is equivalent to cosine when it comes to
[00:09:38] equivalent to cosine when it comes to ranking which is just to say that if you
[00:09:41] ranking which is just to say that if you want to use euclidean and you first l2
[00:09:43] want to use euclidean and you first l2 norm your vectors you're probably just
[00:09:45] norm your vectors you're probably just doing something that might as well just
[00:09:46] doing something that might as well just be the cosine calculation
[00:09:49] be the cosine calculation and dice are equivalent with regard to
[00:09:51] and dice are equivalent with regard to ranking that's something to keep in mind
[00:09:53] ranking that's something to keep in mind uh and then this is maybe a more
[00:09:54] uh and then this is maybe a more fundamental point that you'll see
[00:09:56] fundamental point that you'll see recurring throughout this unit both l2
[00:09:58] recurring throughout this unit both l2 norming and also a related calculation
[00:10:00] norming and also a related calculation which would just create probability
[00:10:02] which would just create probability distributions out of the rows they can
[00:10:04] distributions out of the rows they can be useful steps as we've seen but they
[00:10:06] be useful steps as we've seen but they can obscure differences in the amount or
[00:10:09] can obscure differences in the amount or strength of evidence that you have which
[00:10:10] strength of evidence that you have which can in turn have an effect on the
[00:10:12] can in turn have an effect on the reliability of for example cosine
[00:10:14] reliability of for example cosine normuclidian or kale divergence right
[00:10:17] normuclidian or kale divergence right these shortcomings might be addressed
[00:10:19] these shortcomings might be addressed through waiting schemes though but
[00:10:20] through waiting schemes though but here's the bottom line there is valuable
[00:10:22] here's the bottom line there is valuable information in raw frequency
[00:10:25] information in raw frequency if we abstract away from it some other
[00:10:27] if we abstract away from it some other information might come to the surface
[00:10:28] information might come to the surface but we also might lose that important
[00:10:30] but we also might lose that important frequency information in distorting the
[00:10:32] frequency information in distorting the space in that way and it can be
[00:10:34] space in that way and it can be difficult to balance these competing
[00:10:36] difficult to balance these competing pressures
[00:10:39] finally i just close with some code
[00:10:40] finally i just close with some code snippets our course repository has lots
[00:10:43] snippets our course repository has lots of handy utilities for doing these
[00:10:45] of handy utilities for doing these distance calculations and also length
[00:10:48] distance calculations and also length norming your vectors and so forth and it
[00:10:50] norming your vectors and so forth and it also has this function called neighbors
[00:10:52] also has this function called neighbors in the vsm module it allows you to pick
[00:10:55] in the vsm module it allows you to pick a target word and supply a vector space
[00:10:58] a target word and supply a vector space model and then it will give you
[00:11:00] model and then it will give you a full ranking of the entire vocabulary
[00:11:02] a full ranking of the entire vocabulary in that vector space with respect to
[00:11:04] in that vector space with respect to your target word starting with the ones
[00:11:06] your target word starting with the ones that are closest so here are the results
[00:11:08] that are closest so here are the results for bad using
[00:11:10] for bad using cosine distance in cell 12 and jakarta
[00:11:12] cosine distance in cell 12 and jakarta distance and swell in cell 13 and i
[00:11:15] distance and swell in cell 13 and i would just like to say that these
[00:11:16] would just like to say that these neighbors don't look especially
[00:11:18] neighbors don't look especially intuitive to me it does not look like
[00:11:20] intuitive to me it does not look like this analysis is revealing
[00:11:22] this analysis is revealing really interesting semantic information
[00:11:25] really interesting semantic information but don't worry we're going to correct
[00:11:27] but don't worry we're going to correct this we're going to start to massage and
[00:11:29] this we're going to start to massage and stretch and bend our vector space models
[00:11:31] stretch and bend our vector space models and we'll see we will see much better
[00:11:33] and we'll see we will see much better results for these neighbor functions and
[00:11:35] results for these neighbor functions and everything else
[00:11:36] everything else as we go through that material
[00:11:43] you

Basic Reweighting | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=dv559tVBQRk

---

Transcript

[00:00:05] welcome back everyone this is part four
[00:00:07] welcome back everyone this is part four in our series on distributed word
[00:00:08] in our series on distributed word representations we're going to be
[00:00:09] representations we're going to be talking about basic re-weighting schemes
[00:00:12] talking about basic re-weighting schemes essentially i feel like we've been
[00:00:13] essentially i feel like we've been faithful to the underlying counts in our
[00:00:14] faithful to the underlying counts in our matrices for too long it's time to start
[00:00:16] matrices for too long it's time to start messing with them
[00:00:18] messing with them here's some high level goals that we
[00:00:19] here's some high level goals that we have for rewriting we would like in
[00:00:20] have for rewriting we would like in these matrices to amplify the
[00:00:22] these matrices to amplify the associations that are important and
[00:00:24] associations that are important and trustworthy and unusual while
[00:00:26] trustworthy and unusual while correspondingly de-emphasizing the
[00:00:28] correspondingly de-emphasizing the things that are mundane or quirky or
[00:00:30] things that are mundane or quirky or reflect errors or idiosyncrasies in the
[00:00:32] reflect errors or idiosyncrasies in the data that we use
[00:00:33] data that we use now of course absent a defined objective
[00:00:36] now of course absent a defined objective function in the machine learning sense
[00:00:37] function in the machine learning sense this is going to remain a fuzzy goal but
[00:00:39] this is going to remain a fuzzy goal but we do have some quantitative hooks i
[00:00:41] we do have some quantitative hooks i think we have this guiding intuition
[00:00:43] think we have this guiding intuition that we would like to move away from raw
[00:00:45] that we would like to move away from raw counts because frequency alone is
[00:00:47] counts because frequency alone is generally a poor proxy for the kind of
[00:00:49] generally a poor proxy for the kind of semantic information that we hope to
[00:00:51] semantic information that we hope to extract
[00:00:52] extract so we can ask for each of the
[00:00:54] so we can ask for each of the re-weighting schemes that we consider
[00:00:56] re-weighting schemes that we consider first how does it compare to the
[00:00:57] first how does it compare to the underlying raw count values
[00:00:59] underlying raw count values if the scheme is just rescaling the
[00:01:01] if the scheme is just rescaling the underlying counts it's probably not
[00:01:03] underlying counts it's probably not worth the effort on the other hand if it
[00:01:05] worth the effort on the other hand if it gives us a very different distribution
[00:01:07] gives us a very different distribution then at least we know that we're cooking
[00:01:08] then at least we know that we're cooking with fire when it comes to moving away
[00:01:10] with fire when it comes to moving away from raw frequency
[00:01:12] from raw frequency there's a related question that i would
[00:01:14] there's a related question that i would like us to have in mind what is the
[00:01:15] like us to have in mind what is the overall distribution of values that the
[00:01:17] overall distribution of values that the re-weighting scheme delivers count
[00:01:19] re-weighting scheme delivers count distributions are very skewed in a way
[00:01:21] distributions are very skewed in a way that can make them difficult to deal
[00:01:23] that can make them difficult to deal with for lots of analytic and machine
[00:01:25] with for lots of analytic and machine learning methods so we might hope that
[00:01:27] learning methods so we might hope that in re-weighting in addition to capturing
[00:01:29] in re-weighting in addition to capturing things that are important and
[00:01:30] things that are important and de-emphasizing things that are mundane
[00:01:32] de-emphasizing things that are mundane would also give us an overall
[00:01:33] would also give us an overall distribution of values that was more
[00:01:35] distribution of values that was more tractable for these downstream
[00:01:36] tractable for these downstream applications
[00:01:38] applications and then finally i personally have a
[00:01:40] and then finally i personally have a goal that we would like to do no feature
[00:01:41] goal that we would like to do no feature selection based on counts or outside
[00:01:44] selection based on counts or outside resources like stock word dictionaries i
[00:01:46] resources like stock word dictionaries i don't want to be filtering off parts of
[00:01:48] don't want to be filtering off parts of the vocabulary a priori because for all
[00:01:51] the vocabulary a priori because for all i know something that's a boring stop
[00:01:52] i know something that's a boring stop word for one genre is actually an
[00:01:54] word for one genre is actually an important content word for another we
[00:01:56] important content word for another we would like the method to sort of make
[00:01:58] would like the method to sort of make that decision
[00:02:02] so let's start with the most basic kind
[00:02:03] so let's start with the most basic kind of scheme and this is a scheme that will
[00:02:05] of scheme and this is a scheme that will pay attention only to the row context
[00:02:07] pay attention only to the row context this is normalization so this is
[00:02:09] this is normalization so this is actually a repeat from the lecture on
[00:02:11] actually a repeat from the lecture on vector comparison l2 norming we have
[00:02:13] vector comparison l2 norming we have calculate the l2 length as a fixed
[00:02:15] calculate the l2 length as a fixed quantity for each row vector and then
[00:02:17] quantity for each row vector and then the length normalization of that row
[00:02:19] the length normalization of that row vector is just taking each value in the
[00:02:21] vector is just taking each value in the original vector and dividing it by that
[00:02:23] original vector and dividing it by that fixed quantity the l2 length
[00:02:27] fixed quantity the l2 length there's a related and perhaps more
[00:02:28] there's a related and perhaps more familiar notion which i've called
[00:02:29] familiar notion which i've called probability distribution where we follow
[00:02:31] probability distribution where we follow the same logic we just replace that
[00:02:33] the same logic we just replace that normalizing constant the l2 length with
[00:02:36] normalizing constant the l2 length with the sum of all the elements in the
[00:02:37] the sum of all the elements in the vectors but again we do this
[00:02:39] vectors but again we do this element-wise division by that fixed
[00:02:41] element-wise division by that fixed quantity to normalize the vector into a
[00:02:43] quantity to normalize the vector into a probability distribution
[00:02:46] probability distribution i think both of these methods can be
[00:02:47] i think both of these methods can be powerful but the shame of them is that
[00:02:49] powerful but the shame of them is that they are paying attention only to the
[00:02:51] they are paying attention only to the row context for a given cell i j we're
[00:02:54] row context for a given cell i j we're looking just across the row i we're not
[00:02:56] looking just across the row i we're not considering the context that could come
[00:02:58] considering the context that could come from the column j so let's begin to
[00:03:00] from the column j so let's begin to correct that omission
[00:03:02] correct that omission here's kind of the star of our show in a
[00:03:04] here's kind of the star of our show in a quiet sense this is the first scheme
[00:03:06] quiet sense this is the first scheme we'll look at that pays attention to
[00:03:07] we'll look at that pays attention to both row and column context this is
[00:03:09] both row and column context this is observed over expected let's just go
[00:03:12] observed over expected let's just go through this notation here we have the
[00:03:13] through this notation here we have the row sum i think that's intuitive
[00:03:15] row sum i think that's intuitive correspondingly the column sum the sum
[00:03:17] correspondingly the column sum the sum of all values along the column and then
[00:03:19] of all values along the column and then the sum for some matrix x is just the
[00:03:21] the sum for some matrix x is just the sum of all the cell values in that
[00:03:23] sum of all the cell values in that matrix
[00:03:24] matrix those are the raw materials for
[00:03:25] those are the raw materials for calculating what's called the expected
[00:03:27] calculating what's called the expected value the expected value given a matrix
[00:03:30] value the expected value given a matrix x for cell i j is the row sum times the
[00:03:34] x for cell i j is the row sum times the column sum
[00:03:35] column sum as the numerator divided by the sum of
[00:03:37] as the numerator divided by the sum of all the values in the matrix
[00:03:39] all the values in the matrix this is an expected quasi count it is
[00:03:42] this is an expected quasi count it is giving us the number we would expect if
[00:03:44] giving us the number we would expect if the row and column were independent of
[00:03:46] the row and column were independent of each other in the statistical sense and
[00:03:48] each other in the statistical sense and that's the sense in which this is an
[00:03:50] that's the sense in which this is an expectation
[00:03:51] expectation the observed over expected value simply
[00:03:53] the observed over expected value simply compare the observed value in the
[00:03:55] compare the observed value in the numerator by that expected value
[00:03:58] numerator by that expected value so in a bit more detail here's how the
[00:04:00] so in a bit more detail here's how the calculations work we've got this tiny
[00:04:02] calculations work we've got this tiny little count matrix here let's look at
[00:04:04] little count matrix here let's look at cell xa it's got a count of 34. that's
[00:04:07] cell xa it's got a count of 34. that's our observed count over here in the
[00:04:08] our observed count over here in the numerator the denominator is the product
[00:04:11] numerator the denominator is the product of the row sum and the column sum 45 by
[00:04:14] of the row sum and the column sum 45 by 81 divided by the sum of all the values
[00:04:17] 81 divided by the sum of all the values in this matrix which is 99. we repeat
[00:04:20] in this matrix which is 99. we repeat that calculation for all the other cells
[00:04:22] that calculation for all the other cells making the corresponding adjustments and
[00:04:24] making the corresponding adjustments and that gives us a completely reweighted
[00:04:26] that gives us a completely reweighted matrix
[00:04:28] matrix here's the intuition that was the
[00:04:30] here's the intuition that was the calculation let's think about why we
[00:04:32] calculation let's think about why we might want to do this so i've got here a
[00:04:34] might want to do this so i've got here a highly idealized little count matrix and
[00:04:37] highly idealized little count matrix and the conceit of this example is that keep
[00:04:39] the conceit of this example is that keep tabs in english is an idiom and
[00:04:41] tabs in english is an idiom and otherwise the word tabs alone doesn't
[00:04:43] otherwise the word tabs alone doesn't appear with many other words it's kind
[00:04:45] appear with many other words it's kind of constrained to this idiomatic context
[00:04:48] of constrained to this idiomatic context so we get a really high count for keep
[00:04:50] so we get a really high count for keep tabs and a relatively low count for
[00:04:52] tabs and a relatively low count for enjoy tabs again because tabs doesn't
[00:04:54] enjoy tabs again because tabs doesn't really associate with the word enjoy
[00:04:57] really associate with the word enjoy on the right here i've got the expected
[00:04:59] on the right here i've got the expected calculation and it comes out just like
[00:05:01] calculation and it comes out just like we would hope the expected count for
[00:05:03] we would hope the expected count for keep tabs is merely 12.48
[00:05:06] keep tabs is merely 12.48 compare that with the observed count of
[00:05:08] compare that with the observed count of 20. keep tabs is over represented
[00:05:10] 20. keep tabs is over represented relative to our expectations in virtue
[00:05:13] relative to our expectations in virtue of the fact that the independence
[00:05:14] of the fact that the independence assumption built into the expected
[00:05:16] assumption built into the expected calculation is just not met here because
[00:05:18] calculation is just not met here because of the collocational effect
[00:05:20] of the collocational effect similarly the expected count for enjoy
[00:05:22] similarly the expected count for enjoy tabs is 8.5 that's much larger than our
[00:05:25] tabs is 8.5 that's much larger than our observation again because these are kind
[00:05:27] observation again because these are kind of disassociated with each other in
[00:05:30] of disassociated with each other in virtue of the restricted distribution of
[00:05:32] virtue of the restricted distribution of tabs
[00:05:33] tabs and that brings us to the really the
[00:05:35] and that brings us to the really the star of our shown in fact the star of a
[00:05:36] star of our shown in fact the star of a lot of the remainder of this unit this
[00:05:38] lot of the remainder of this unit this is point y is mutual information or pmi
[00:05:42] is point y is mutual information or pmi pmi is simply observed over expected in
[00:05:44] pmi is simply observed over expected in log space where we stipulate that the
[00:05:46] log space where we stipulate that the log of zero is zero in a bit more detail
[00:05:49] log of zero is zero in a bit more detail for a matrix x given cell i j the pmi
[00:05:52] for a matrix x given cell i j the pmi value is the log
[00:05:54] value is the log of the observed count over the expected
[00:05:56] of the observed count over the expected count and that's it
[00:05:58] count and that's it many people find it more intuitive to
[00:05:59] many people find it more intuitive to think of this in probabilistic terms
[00:06:01] think of this in probabilistic terms that's what i've done over here on the
[00:06:03] that's what i've done over here on the right it's equivalent numerically but
[00:06:05] right it's equivalent numerically but for this kind of calculation we first
[00:06:07] for this kind of calculation we first form a joint probability table
[00:06:09] form a joint probability table by just dividing all the cell values by
[00:06:11] by just dividing all the cell values by the total number of values in all the
[00:06:13] the total number of values in all the cells
[00:06:14] cells that gives us the joint probability
[00:06:16] that gives us the joint probability table and then the row probability and
[00:06:18] table and then the row probability and the column probability are just summing
[00:06:20] the column probability are just summing across the row and the column
[00:06:21] across the row and the column respectively and again we multiply them
[00:06:24] respectively and again we multiply them and that's kind of nice because then you
[00:06:25] and that's kind of nice because then you can see we really are testing an
[00:06:27] can see we really are testing an independence assumption it's as though
[00:06:28] independence assumption it's as though we say we can multiply these
[00:06:30] we say we can multiply these probabilities because they're
[00:06:31] probabilities because they're independent if the distribution is truly
[00:06:34] independent if the distribution is truly independent that autumn match will be
[00:06:35] independent that autumn match will be observed and of course discrepancies are
[00:06:37] observed and of course discrepancies are the things that these matrices will
[00:06:39] the things that these matrices will highlight
[00:06:40] highlight let's look at an example and there's one
[00:06:42] let's look at an example and there's one thing that i want to track as we work
[00:06:43] thing that i want to track as we work through this example and that's the cell
[00:06:45] through this example and that's the cell down here this lonely little one so this
[00:06:48] down here this lonely little one so this is a count matrix i've got this as a
[00:06:50] is a count matrix i've got this as a word by document matrix this is a very
[00:06:51] word by document matrix this is a very flexible method and we'll apply it to
[00:06:53] flexible method and we'll apply it to lots of matrix designs
[00:06:55] lots of matrix designs over here i form the joint probability
[00:06:57] over here i form the joint probability table and i've got here the column sum
[00:06:59] table and i've got here the column sum and the row sum corresponding to the
[00:07:01] and the row sum corresponding to the column and row probability these are the
[00:07:03] column and row probability these are the raw ingredients
[00:07:04] raw ingredients for the pmi matrix which is derived down
[00:07:06] for the pmi matrix which is derived down here by applying this calculation to all
[00:07:09] here by applying this calculation to all these values
[00:07:10] these values notice what's happened that lonely one
[00:07:12] notice what's happened that lonely one down here because it's in a very
[00:07:14] down here because it's in a very infrequent row and a relatively
[00:07:16] infrequent row and a relatively infrequent column it has the largest pmi
[00:07:19] infrequent column it has the largest pmi value in the resulting matrix now that
[00:07:21] value in the resulting matrix now that could be good because this could be a
[00:07:22] could be good because this could be a very important event in which case we
[00:07:25] very important event in which case we want to amplify it on the other hand
[00:07:28] want to amplify it on the other hand nlp being what it is this could be just
[00:07:30] nlp being what it is this could be just a mistake in the data or something and
[00:07:32] a mistake in the data or something and then this exaggerated value here could
[00:07:34] then this exaggerated value here could turn out to be problematic it's
[00:07:36] turn out to be problematic it's difficult to strike this balance but
[00:07:37] difficult to strike this balance but it's worth keeping in mind as you work
[00:07:39] it's worth keeping in mind as you work with this method that it could amplify
[00:07:41] with this method that it could amplify not only important things but also
[00:07:43] not only important things but also idiosyncratic things
[00:07:45] idiosyncratic things positive pmi is an important variant of
[00:07:47] positive pmi is an important variant of pmi so important in fact that i would
[00:07:49] pmi so important in fact that i would like to think of it as the kind of
[00:07:50] like to think of it as the kind of default view that we take on pmi for the
[00:07:52] default view that we take on pmi for the following reason
[00:07:53] following reason pmi is actually undefined where the
[00:07:55] pmi is actually undefined where the count is 0 because we need to take the
[00:07:57] count is 0 because we need to take the log of zero
[00:07:59] log of zero so we had to stipulate that the log of
[00:08:00] so we had to stipulate that the log of zero was zero for this calculation
[00:08:04] however that's arguably not coherent if
[00:08:06] however that's arguably not coherent if you think about what the underlying
[00:08:08] you think about what the underlying matrix represents what we're saying with
[00:08:09] matrix represents what we're saying with pmi is that larger than expected values
[00:08:12] pmi is that larger than expected values get a large pmi
[00:08:14] get a large pmi smaller than expected values get a
[00:08:16] smaller than expected values get a smaller pmi that's good but when we
[00:08:18] smaller pmi that's good but when we encounter a zero we place it right in
[00:08:20] encounter a zero we place it right in the middle and that's just strange
[00:08:22] the middle and that's just strange because a zero isn't evidence of
[00:08:23] because a zero isn't evidence of anything larger or smaller it doesn't
[00:08:25] anything larger or smaller it doesn't deserve to be in the middle of this if
[00:08:27] deserve to be in the middle of this if anything we just don't know what to do
[00:08:29] anything we just don't know what to do with the zero values
[00:08:30] with the zero values so this is arguably sort of incoherent
[00:08:33] so this is arguably sort of incoherent and the standard response to it is to
[00:08:35] and the standard response to it is to simply
[00:08:36] simply turn all of the negative values into
[00:08:38] turn all of the negative values into zeros and that's positive pmi that's
[00:08:40] zeros and that's positive pmi that's defined here so we simply lop off all
[00:08:42] defined here so we simply lop off all the negative values by mapping them to
[00:08:44] the negative values by mapping them to zero
[00:08:45] zero and that at least restores the kind of
[00:08:47] and that at least restores the kind of overall coherence of the claims where
[00:08:49] overall coherence of the claims where all we're doing is reflecting the fact
[00:08:51] all we're doing is reflecting the fact that larger than expected counts have
[00:08:53] that larger than expected counts have large positive pmi and the rest are put
[00:08:56] large positive pmi and the rest are put at zero
[00:08:58] at zero let's look briefly at a few other
[00:08:59] let's look briefly at a few other re-weighting schemes starting with the
[00:09:01] re-weighting schemes starting with the t-test the t-test is something that
[00:09:02] t-test the t-test is something that you'll work with on the first assignment
[00:09:04] you'll work with on the first assignment you'll implement it it turns out to be a
[00:09:05] you'll implement it it turns out to be a very good re-rating scheme and i like it
[00:09:08] very good re-rating scheme and i like it because it obviously reflects many of
[00:09:09] because it obviously reflects many of the same intuitions that guide the pmi
[00:09:11] the same intuitions that guide the pmi and observed over expected calculations
[00:09:14] and observed over expected calculations tf idf is quite different so this is
[00:09:17] tf idf is quite different so this is typically performed on word by document
[00:09:19] typically performed on word by document matrices in the context of information
[00:09:21] matrices in the context of information retrieval given some corpus of documents
[00:09:23] retrieval given some corpus of documents d we're going to say that the term
[00:09:25] d we're going to say that the term frequency for a given cell
[00:09:27] frequency for a given cell is that value divided by the sum of all
[00:09:29] is that value divided by the sum of all the values in the column giving us a
[00:09:30] the values in the column giving us a kind of probability of the word given
[00:09:32] kind of probability of the word given the document that we're in
[00:09:34] the document that we're in and then the idf value
[00:09:36] and then the idf value is the log of this quantity here this is
[00:09:38] is the log of this quantity here this is the number of corpora the number of
[00:09:40] the number of corpora the number of documents in our corpus that is the road
[00:09:42] documents in our corpus that is the road the column dimensionality divided by the
[00:09:44] the column dimensionality divided by the number of documents that contain the
[00:09:46] number of documents that contain the target word and again we map log of zero
[00:09:48] target word and again we map log of zero to zero the tf idf is the product of
[00:09:51] to zero the tf idf is the product of those two values
[00:09:52] those two values i think this can be an outstanding
[00:09:54] i think this can be an outstanding method for very large sparse matrices
[00:09:56] method for very large sparse matrices like the word document one
[00:09:58] like the word document one conversely it is typically not well
[00:10:01] conversely it is typically not well behaved for very dense matrices like the
[00:10:03] behaved for very dense matrices like the word by word ones that we've we're
[00:10:05] word by word ones that we've we're favoring in this course the reason this
[00:10:07] favoring in this course the reason this is this idf value it's very unlikely
[00:10:10] is this idf value it's very unlikely that you would have a word that appeared
[00:10:12] that you would have a word that appeared literally in every document
[00:10:14] literally in every document however in the context of very dense
[00:10:17] however in the context of very dense word by word matrices it is possible for
[00:10:19] word by word matrices it is possible for some words to co-occur with every single
[00:10:21] some words to co-occur with every single other word in which case you get an idf
[00:10:23] other word in which case you get an idf value of zero which is probably not the
[00:10:26] value of zero which is probably not the intended outcome for something that's
[00:10:27] intended outcome for something that's high frequency but might nonetheless be
[00:10:30] high frequency but might nonetheless be important in the context of individual
[00:10:32] important in the context of individual documents
[00:10:33] documents so i'd probably steer away from tf idf
[00:10:35] so i'd probably steer away from tf idf unless you're working with a sparse
[00:10:37] unless you're working with a sparse matrix design
[00:10:39] matrix design and then even further afield from the
[00:10:40] and then even further afield from the things we've discussed you might explore
[00:10:42] things we've discussed you might explore using for example pairwise distance
[00:10:43] using for example pairwise distance matrices where i calculate the cosine
[00:10:46] matrices where i calculate the cosine distance between every pair of words
[00:10:48] distance between every pair of words along the rows and form a matrix on that
[00:10:50] along the rows and form a matrix on that basis really different in its approach
[00:10:53] basis really different in its approach and probably in its outcomes but it
[00:10:55] and probably in its outcomes but it could be very interesting
[00:10:58] could be very interesting let's return to our central questions
[00:11:00] let's return to our central questions remember for each one of these
[00:11:01] remember for each one of these rewritings we want to ask how does it
[00:11:03] rewritings we want to ask how does it compare to the raw count values and what
[00:11:05] compare to the raw count values and what overall distribution of values does it
[00:11:08] overall distribution of values does it deliver so let's do a bit of an
[00:11:09] deliver so let's do a bit of an assessment of that i'm working with the
[00:11:11] assessment of that i'm working with the giga5 matrix that you can load as part
[00:11:13] giga5 matrix that you can load as part of the course materials that's gigaward
[00:11:15] of the course materials that's gigaward with a window of five and scaling of one
[00:11:17] with a window of five and scaling of one over n
[00:11:19] over n up here in the left i have the raw
[00:11:20] up here in the left i have the raw counts and the cell value along the
[00:11:22] counts and the cell value along the x-axis and the number of things that
[00:11:24] x-axis and the number of things that have that value along the y-axis and you
[00:11:27] have that value along the y-axis and you can see that raw counts it's a very
[00:11:29] can see that raw counts it's a very difficult distribution first of all this
[00:11:31] difficult distribution first of all this goes all the way up to about a hundred
[00:11:34] goes all the way up to about a hundred million
[00:11:35] million so and starting from zero most things
[00:11:38] so and starting from zero most things have quantities that are close to zero
[00:11:40] have quantities that are close to zero then you have this very long thin tail
[00:11:42] then you have this very long thin tail of things that are very high frequency
[00:11:44] of things that are very high frequency this kind of highly skewed distribution
[00:11:46] this kind of highly skewed distribution is difficult for many machine learning
[00:11:48] is difficult for many machine learning methods both in terms of the skewed
[00:11:50] methods both in terms of the skewed towards zero and very low values and
[00:11:52] towards zero and very low values and also in terms of the range of these
[00:11:54] also in terms of the range of these x-axis values so we would like to move
[00:11:56] x-axis values so we would like to move away from it that's one motivating
[00:11:58] away from it that's one motivating reason
[00:11:59] reason when we look at l2 norming and
[00:12:00] when we look at l2 norming and probability distributions they do kind
[00:12:02] probability distributions they do kind of the same thing they are constraining
[00:12:04] of the same thing they are constraining the cell values to be between 0 and 1 or
[00:12:07] the cell values to be between 0 and 1 or roughly that between 0 and one
[00:12:10] roughly that between 0 and one but they still have a heavy skewed
[00:12:12] but they still have a heavy skewed toward things that are very small in
[00:12:14] toward things that are very small in their adjusted values and their related
[00:12:15] their adjusted values and their related values
[00:12:17] values observed over expected is more extreme
[00:12:19] observed over expected is more extreme in that as is itf idf so again
[00:12:22] in that as is itf idf so again the observed over expected values range
[00:12:24] the observed over expected values range quite high up to about almost 50 000
[00:12:27] quite high up to about almost 50 000 which is somewhat better than the raw
[00:12:28] which is somewhat better than the raw counts but it's still very large in
[00:12:31] counts but it's still very large in terms of its spread and we still have
[00:12:32] terms of its spread and we still have that heavy skewed towards zero tf-idf
[00:12:35] that heavy skewed towards zero tf-idf solves the range problem down here
[00:12:37] solves the range problem down here because it's highly constrained in a set
[00:12:39] because it's highly constrained in a set of values but it still has a very heavy
[00:12:41] of values but it still has a very heavy skew looking a lot like the raw count
[00:12:43] skew looking a lot like the raw count distribution
[00:12:45] distribution from this perspective it looks like pmi
[00:12:47] from this perspective it looks like pmi and positive pmi are really steps
[00:12:49] and positive pmi are really steps forward first of all for pmi this
[00:12:52] forward first of all for pmi this distribution of cell values has this
[00:12:53] distribution of cell values has this nice sort of normal distribution and the
[00:12:56] nice sort of normal distribution and the values themselves are pretty constrained
[00:12:57] values themselves are pretty constrained to like negative 10 to 10.
[00:13:00] to like negative 10 to 10. and then for positive pmi we simply lock
[00:13:03] and then for positive pmi we simply lock off all the negative values and make it
[00:13:04] off all the negative values and make it map to zero so it's more skewed towards
[00:13:06] map to zero so it's more skewed towards zero but not nearly as skewed as all
[00:13:09] zero but not nearly as skewed as all these other methods that we're looking
[00:13:10] these other methods that we're looking at
[00:13:11] at so this is looking like pmi and ppmi are
[00:13:14] so this is looking like pmi and ppmi are good choices here just from the point of
[00:13:16] good choices here just from the point of view of departing from the
[00:13:18] view of departing from the raw counts and giving a subtractable
[00:13:20] raw counts and giving a subtractable distribution
[00:13:22] distribution here's another perspective where we
[00:13:24] here's another perspective where we directly compare in these matrices the
[00:13:26] directly compare in these matrices the co-occurrence count on log scale so it's
[00:13:28] co-occurrence count on log scale so it's viewable with the result the new
[00:13:31] viewable with the result the new weighted cell value what we're looking
[00:13:33] weighted cell value what we're looking for here presumably is an overall lack
[00:13:35] for here presumably is an overall lack of correlation
[00:13:38] of correlation i think we find that l2 norming and
[00:13:39] i think we find that l2 norming and probabilities are pretty good on this
[00:13:41] probabilities are pretty good on this score they kind of have low
[00:13:43] score they kind of have low correlations and they make good use of a
[00:13:45] correlations and they make good use of a large part of the scale that they
[00:13:47] large part of the scale that they operate on
[00:13:48] operate on observed over expected has a low
[00:13:50] observed over expected has a low correlation with the cell counts which
[00:13:52] correlation with the cell counts which it looks initially good but it has this
[00:13:54] it looks initially good but it has this problem that the cell values are kind of
[00:13:56] problem that the cell values are kind of strangely distributed and this
[00:13:58] strangely distributed and this correlation value might not even be
[00:14:00] correlation value might not even be especially meaningful given that we have
[00:14:02] especially meaningful given that we have a few outliers and then a whole lot of
[00:14:03] a few outliers and then a whole lot of things that are close to zero and tf idf
[00:14:06] things that are close to zero and tf idf is frankly similar low correlation but
[00:14:08] is frankly similar low correlation but maybe not so trustworthy in terms of
[00:14:10] maybe not so trustworthy in terms of that correlation value fundamentally
[00:14:12] that correlation value fundamentally again these look like difficult
[00:14:13] again these look like difficult distributions of values to work with
[00:14:16] distributions of values to work with again pmi and positive pmi look really
[00:14:18] again pmi and positive pmi look really good
[00:14:19] good relatively low correlation so we've done
[00:14:21] relatively low correlation so we've done something meaningful and both of these
[00:14:23] something meaningful and both of these are making meaningful use of a
[00:14:26] are making meaningful use of a substantial part of the overall space
[00:14:28] substantial part of the overall space that they operate in we have lots of
[00:14:30] that they operate in we have lots of different combinations of cell values
[00:14:32] different combinations of cell values and underlying co-occurrence counts
[00:14:34] and underlying co-occurrence counts something of a correlation but that
[00:14:36] something of a correlation but that could be good but we're not locked into
[00:14:38] could be good but we're not locked into that correlation so we've done something
[00:14:40] that correlation so we've done something meaningful
[00:14:41] meaningful to wrap up let's do some relationships
[00:14:43] to wrap up let's do some relationships and generalizations just some reminders
[00:14:45] and generalizations just some reminders here so a theme running through nearly
[00:14:47] here so a theme running through nearly all of these schemes is that we want to
[00:14:49] all of these schemes is that we want to rewrite a cell value relative to the
[00:14:51] rewrite a cell value relative to the values we expect given the row and the
[00:14:53] values we expect given the row and the column and we would like to make use of
[00:14:55] column and we would like to make use of both of those notions of context
[00:14:58] both of those notions of context the magnitude of the counts might be
[00:15:00] the magnitude of the counts might be important just think about how
[00:15:02] important just think about how 110 as a bit of evidence and a thousand
[00:15:05] 110 as a bit of evidence and a thousand ten thousand as a bit of evidence might
[00:15:07] ten thousand as a bit of evidence might be very different situations in terms of
[00:15:09] be very different situations in terms of the evidence that you have gathered
[00:15:11] the evidence that you have gathered creating probability distributions and
[00:15:13] creating probability distributions and length normalizing will obscure that
[00:15:16] length normalizing will obscure that difference and that might be something
[00:15:17] difference and that might be something that you want to dwell on
[00:15:21] pmi and experience will amplify the
[00:15:23] pmi and experience will amplify the values of counts that are tiny relative
[00:15:25] values of counts that are tiny relative to their rows and contents then they're
[00:15:27] to their rows and contents then they're in columns that could be good because
[00:15:29] in columns that could be good because that might be what you want to do find
[00:15:30] that might be what you want to do find the things that are really important and
[00:15:32] the things that are really important and unusual unfortunately with language data
[00:15:34] unusual unfortunately with language data we have to watch out that they might be
[00:15:36] we have to watch out that they might be noise
[00:15:39] noise and finally tfidf severely punishes
[00:15:41] and finally tfidf severely punishes words that appear in many documents it
[00:15:43] words that appear in many documents it behaves oddly for dense matrices which
[00:15:45] behaves oddly for dense matrices which can can include the word by word
[00:15:47] can can include the word by word matrices that we're working with so you
[00:15:49] matrices that we're working with so you might proceed with caution with that
[00:15:50] might proceed with caution with that particular rewriting scheme in the
[00:15:52] particular rewriting scheme in the context of this course
[00:15:54] context of this course finally some code snippets i'm just
[00:15:56] finally some code snippets i'm just showing off that our vsm module in the
[00:15:58] showing off that our vsm module in the course repository makes it really easy
[00:16:00] course repository makes it really easy to do these re-weighting schemes a lot
[00:16:02] to do these re-weighting schemes a lot all the ones that we've talked about and
[00:16:03] all the ones that we've talked about and more in fact
[00:16:05] more in fact and returning to the end of our vector
[00:16:07] and returning to the end of our vector comparison method you might recall that
[00:16:09] comparison method you might recall that i looked at the neighbors of bad in this
[00:16:11] i looked at the neighbors of bad in this yelp 5 matrix and it really didn't look
[00:16:14] yelp 5 matrix and it really didn't look good this does not look especially
[00:16:15] good this does not look especially semantically coherent
[00:16:17] semantically coherent when i take those underlying counts and
[00:16:19] when i take those underlying counts and i just adjust them by positive pmi i
[00:16:22] i just adjust them by positive pmi i start to see something that looks quite
[00:16:24] start to see something that looks quite semantically coherent and i think we're
[00:16:26] semantically coherent and i think we're starting to see the promise of these
[00:16:27] starting to see the promise of these methods and this is really just the
[00:16:29] methods and this is really just the beginning in terms of surfacing
[00:16:30] beginning in terms of surfacing semantically coherent and interesting
[00:16:32] semantically coherent and interesting information from these underlying counts

Dimensionality Reduction | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=5Bx5UhrJbJI

---

Transcript

[00:00:05] hello everyone welcome back this is part
[00:00:06] hello everyone welcome back this is part five in our series on distributed word
[00:00:08] five in our series on distributed word representations we're going to be
[00:00:09] representations we're going to be talking about dimensionality reduction
[00:00:11] talking about dimensionality reduction techniques we saw in the previous
[00:00:13] techniques we saw in the previous screencast that re-weighting is a
[00:00:15] screencast that re-weighting is a powerful tool for finding latent
[00:00:17] powerful tool for finding latent semantic information in count matrices
[00:00:20] semantic information in count matrices we're going to push that even further
[00:00:22] we're going to push that even further the promise of dimensionality reduction
[00:00:23] the promise of dimensionality reduction techniques is that they can capture
[00:00:25] techniques is that they can capture higher order notions of co-occurrence
[00:00:27] higher order notions of co-occurrence corresponding to even deeper sorts of
[00:00:29] corresponding to even deeper sorts of semantic relatedness
[00:00:33] there's a wide world of these
[00:00:34] there's a wide world of these dimensionality reduction techniques i've
[00:00:36] dimensionality reduction techniques i've chosen three that we're going to focus
[00:00:37] chosen three that we're going to focus on as interesting representatives of a
[00:00:39] on as interesting representatives of a much larger space we'll look at latent
[00:00:42] much larger space we'll look at latent semantic analysis which is a classic
[00:00:44] semantic analysis which is a classic linear method then we'll talk about auto
[00:00:46] linear method then we'll talk about auto encoders a newer powerful deep learning
[00:00:48] encoders a newer powerful deep learning mode for learning to reduce dimensional
[00:00:50] mode for learning to reduce dimensional representations and then finally glove
[00:00:53] representations and then finally glove which is a simple yet very powerful
[00:00:55] which is a simple yet very powerful method that as you'll see has a deep
[00:00:57] method that as you'll see has a deep connection to pointwise mutual
[00:00:59] connection to pointwise mutual information and then i'm going to going
[00:01:01] information and then i'm going to going to close by talking briefly about
[00:01:02] to close by talking briefly about visualization which is another kind of
[00:01:04] visualization which is another kind of dimensionality reduction technique that
[00:01:06] dimensionality reduction technique that we might use for very different purposes
[00:01:10] we might use for very different purposes so let's begin with latent semantic
[00:01:11] so let's begin with latent semantic analysis a classic method the paper is
[00:01:13] analysis a classic method the paper is due to dear wester at all 1990 that's a
[00:01:16] due to dear wester at all 1990 that's a classic paper that really made a splash
[00:01:18] classic paper that really made a splash uh it's one of the most lsa is now one
[00:01:20] uh it's one of the most lsa is now one of the oldest most widely used
[00:01:21] of the oldest most widely used dimensionality reduction techniques not
[00:01:23] dimensionality reduction techniques not only in scientific research but also an
[00:01:25] only in scientific research but also an industry i think it was really
[00:01:26] industry i think it was really eye-opening for people at the time of
[00:01:29] eye-opening for people at the time of the paper's appearance to see just how
[00:01:32] the paper's appearance to see just how powerful this technique could be
[00:01:34] powerful this technique could be especially in context of involving
[00:01:36] especially in context of involving information retrieval
[00:01:38] information retrieval the method is also known as known as
[00:01:40] the method is also known as known as truncated singular value decomposition
[00:01:42] truncated singular value decomposition and i'll explain why that is in a second
[00:01:44] and i'll explain why that is in a second the final thing i want to say at this
[00:01:45] the final thing i want to say at this high level is just that lsa remains a
[00:01:47] high level is just that lsa remains a very powerful baseline especially when
[00:01:50] very powerful baseline especially when part of a pipeline of other re-weighting
[00:01:52] part of a pipeline of other re-weighting methods so it should probably be in your
[00:01:54] methods so it should probably be in your results table and it's often very
[00:01:56] results table and it's often very difficult to beat
[00:01:59] now i think we can't in the time
[00:02:00] now i think we can't in the time allotted to us cover all of the
[00:02:02] allotted to us cover all of the technical details surrounding latent
[00:02:04] technical details surrounding latent semantic analysis in my experience this
[00:02:06] semantic analysis in my experience this would be kind of the culmination of a
[00:02:07] would be kind of the culmination of a full course in linear algebra but i do
[00:02:10] full course in linear algebra but i do think i can convey the guiding
[00:02:11] think i can convey the guiding intuitions and that will help you with
[00:02:13] intuitions and that will help you with responsible use of the method
[00:02:15] responsible use of the method so let's imagine that we have this
[00:02:16] so let's imagine that we have this simple two-dimensional vector space
[00:02:18] simple two-dimensional vector space model i've got four points a b c and d
[00:02:20] model i've got four points a b c and d arrayed out in this two-dimensional
[00:02:22] arrayed out in this two-dimensional space
[00:02:23] space i think we're all familiar with fitting
[00:02:24] i think we're all familiar with fitting linear models which capture the largest
[00:02:27] linear models which capture the largest source of variation in the data that's
[00:02:29] source of variation in the data that's this orange line here
[00:02:30] this orange line here the perspective i would encourage you to
[00:02:32] the perspective i would encourage you to take is that we can think of that linear
[00:02:34] take is that we can think of that linear regression model as performing
[00:02:36] regression model as performing dimensionality reduction in that it
[00:02:38] dimensionality reduction in that it encourages us to project points like b
[00:02:41] encourages us to project points like b and c down onto that line
[00:02:44] and c down onto that line and in projecting them down onto that
[00:02:45] and in projecting them down onto that line essentially in abstracting away
[00:02:48] line essentially in abstracting away from their point of variation along the
[00:02:49] from their point of variation along the y axis we can see the sense in which
[00:02:52] y axis we can see the sense in which they are abstractly similar they're
[00:02:53] they are abstractly similar they're close together in this reduced
[00:02:55] close together in this reduced dimensional space
[00:02:57] dimensional space now with the linear model we captured
[00:02:59] now with the linear model we captured the source of greatest variation of this
[00:03:00] the source of greatest variation of this little data set in a high dimensional
[00:03:02] little data set in a high dimensional space we could continue fitting
[00:03:04] space we could continue fitting lines to other sources of variation in
[00:03:07] lines to other sources of variation in the data other axes of variations so
[00:03:08] the data other axes of variations so here's a blue line here that captures
[00:03:10] here's a blue line here that captures the next dimension and we could again
[00:03:12] the next dimension and we could again project points like a and c down onto
[00:03:14] project points like a and c down onto that line and that would capture the
[00:03:16] that line and that would capture the abstract sense in which a and c although
[00:03:18] abstract sense in which a and c although very spread out along the x dimension
[00:03:20] very spread out along the x dimension are very close together along the y
[00:03:22] are very close together along the y dimension and of course if we had more
[00:03:24] dimension and of course if we had more dimensions in this vector space model we
[00:03:27] dimensions in this vector space model we could continue to perform these cuts and
[00:03:29] could continue to perform these cuts and dimensionality reductions capturing ever
[00:03:31] dimensionality reductions capturing ever more abstract notions of similarity
[00:03:34] more abstract notions of similarity along these different axes and that is
[00:03:36] along these different axes and that is in essence what lsa is going to do for
[00:03:38] in essence what lsa is going to do for us in our really large matrices
[00:03:42] us in our really large matrices the fundamental method as i said is
[00:03:44] the fundamental method as i said is singular value decomposition this is a
[00:03:46] singular value decomposition this is a theorem from linear algebra that says
[00:03:48] theorem from linear algebra that says any matrix of dimension m by n can be
[00:03:52] any matrix of dimension m by n can be decomposed into the product of three
[00:03:54] decomposed into the product of three matrices t s and d
[00:03:56] matrices t s and d with the
[00:03:57] with the dimensions given here's a more concrete
[00:03:59] dimensions given here's a more concrete example
[00:04:00] example start with this matrix of dimension
[00:04:02] start with this matrix of dimension three by four we learned a term matrix
[00:04:04] three by four we learned a term matrix which is full of length normalized
[00:04:07] which is full of length normalized orthogonal vectors
[00:04:09] orthogonal vectors we have this matrix of singular values
[00:04:12] we have this matrix of singular values along the diagonal they are organized
[00:04:13] along the diagonal they are organized from largest to smallest corresponding
[00:04:15] from largest to smallest corresponding to the greatest to least source of
[00:04:17] to the greatest to least source of variation in the data
[00:04:19] variation in the data and then we have the document or column
[00:04:21] and then we have the document or column wise matrix which is also length
[00:04:23] wise matrix which is also length normalized and orthogonal in its space
[00:04:26] normalized and orthogonal in its space and the theorem here is that we can
[00:04:27] and the theorem here is that we can reconstruct a from these three matrices
[00:04:30] reconstruct a from these three matrices of course we don't want to precisely
[00:04:32] of course we don't want to precisely reconstruct a that probably wouldn't
[00:04:33] reconstruct a that probably wouldn't accomplish very much for us
[00:04:35] accomplish very much for us but what we can do is use this to learn
[00:04:37] but what we can do is use this to learn reduced dimensional representations of a
[00:04:40] reduced dimensional representations of a by being selective about which term and
[00:04:42] by being selective about which term and singular value dimensions we include in
[00:04:45] singular value dimensions we include in the model
[00:04:46] the model let me walk you through an example of
[00:04:47] let me walk you through an example of how that happens and first let me
[00:04:49] how that happens and first let me motivate this a little bit with an
[00:04:50] motivate this a little bit with an idealized linguistic case so i've got up
[00:04:53] idealized linguistic case so i've got up here a word by document matrix its
[00:04:56] here a word by document matrix its vocabulary is gnarly wicked awesome lame
[00:04:58] vocabulary is gnarly wicked awesome lame and terrible and the conceit of my
[00:05:00] and terrible and the conceit of my example is that both gnarly and wicked
[00:05:02] example is that both gnarly and wicked are positive terms so they tend to
[00:05:04] are positive terms so they tend to co-occur with awesome and not co-occur
[00:05:06] co-occur with awesome and not co-occur with lame and terrible however gnarly
[00:05:08] with lame and terrible however gnarly and wicked never occur in the same
[00:05:10] and wicked never occur in the same document the idea is that gnarly is a
[00:05:13] document the idea is that gnarly is a slang positive term associated with the
[00:05:15] slang positive term associated with the west coast of the united states and
[00:05:17] west coast of the united states and wicked is a slang
[00:05:18] wicked is a slang term associated with the east coast of
[00:05:20] term associated with the east coast of the united states in virtue of that
[00:05:22] the united states in virtue of that idealized dialect split they never occur
[00:05:25] idealized dialect split they never occur in the same document but nonetheless
[00:05:27] in the same document but nonetheless they have similar neighbors in this
[00:05:28] they have similar neighbors in this vector space and that's the kind of
[00:05:30] vector space and that's the kind of abstract notion of co-occurrence that we
[00:05:32] abstract notion of co-occurrence that we want to capture
[00:05:34] want to capture if we simply use our standard distance
[00:05:37] if we simply use our standard distance measures and re-weighting techniques and
[00:05:39] measures and re-weighting techniques and so forth we will not capture that more
[00:05:41] so forth we will not capture that more abstract notion of co-occurrence here
[00:05:43] abstract notion of co-occurrence here distances in this raw vector space
[00:05:45] distances in this raw vector space gnarly awesome terrible and wicked
[00:05:47] gnarly awesome terrible and wicked wicked is farther away from gnarly even
[00:05:49] wicked is farther away from gnarly even than terrible is so we've got a
[00:05:51] than terrible is so we've got a sentiment confusion and really just not
[00:05:53] sentiment confusion and really just not the result we were shooting for
[00:05:55] the result we were shooting for so we perform
[00:05:57] so we perform singular value decomposition into these
[00:05:59] singular value decomposition into these three matrices and then the truncated
[00:06:01] three matrices and then the truncated part is that we're going to consider
[00:06:03] part is that we're going to consider just the first two dimensions of the
[00:06:04] just the first two dimensions of the term matrix corresponding to these two
[00:06:07] term matrix corresponding to these two singular values capturing the top two
[00:06:10] singular values capturing the top two sources of variation in the data
[00:06:12] sources of variation in the data so we multiply those together and we get
[00:06:15] so we multiply those together and we get this reduced dimensional matrix down
[00:06:16] this reduced dimensional matrix down here
[00:06:17] here two by the size of the vocabulary and if
[00:06:20] two by the size of the vocabulary and if we do distance measures in that space
[00:06:22] we do distance measures in that space just as we were hoping gnarly and wicked
[00:06:25] just as we were hoping gnarly and wicked are now neighbors the method has
[00:06:26] are now neighbors the method has captured that more abstract notion of
[00:06:29] captured that more abstract notion of having the same neighbors as the other
[00:06:31] having the same neighbors as the other word
[00:06:34] in the previous lecture i encourage you
[00:06:36] in the previous lecture i encourage you to think about what you're doing to a
[00:06:38] to think about what you're doing to a matrix when you perform some kind of
[00:06:39] matrix when you perform some kind of re-weighting scheme let's extend that to
[00:06:41] re-weighting scheme let's extend that to these dimensionality reduction
[00:06:43] these dimensionality reduction techniques so here's a picture of what
[00:06:44] techniques so here's a picture of what lsa does starting with a raw count
[00:06:47] lsa does starting with a raw count distribution over here
[00:06:49] distribution over here if i just run lsa on that rockcount
[00:06:51] if i just run lsa on that rockcount distribution i get what looks also like
[00:06:54] distribution i get what looks also like a very difficult distribution of values
[00:06:56] a very difficult distribution of values the sale values are very spread out and
[00:06:59] the sale values are very spread out and they have a lot of the mass centered
[00:07:01] they have a lot of the mass centered around zero here corresponding to the
[00:07:04] around zero here corresponding to the peak and the raw counts over here near
[00:07:05] peak and the raw counts over here near zero as well so that doesn't look like
[00:07:07] zero as well so that doesn't look like we've done very much in terms of taming
[00:07:09] we've done very much in terms of taming the kind of untractable skewed
[00:07:11] the kind of untractable skewed distribution we started with
[00:07:13] distribution we started with however if instead we take the raw
[00:07:15] however if instead we take the raw counts and first feed them through pmi
[00:07:17] counts and first feed them through pmi which as we saw before gives us this
[00:07:19] which as we saw before gives us this nice distribution of values highly
[00:07:21] nice distribution of values highly constrained along the x-axis
[00:07:23] constrained along the x-axis and then we run lsa we retain a lot of
[00:07:25] and then we run lsa we retain a lot of those good properties the values are
[00:07:27] those good properties the values are somewhat more spread out but still
[00:07:29] somewhat more spread out but still nicely distributed this looks like a
[00:07:31] nicely distributed this looks like a much happier input to downstream
[00:07:33] much happier input to downstream analytic methods than the top version
[00:07:35] analytic methods than the top version here and i think this is beginning to
[00:07:37] here and i think this is beginning to show that it can be powerful to pipeline
[00:07:40] show that it can be powerful to pipeline re-weighting and dimensionality
[00:07:41] re-weighting and dimensionality reduction techniques
[00:07:44] reduction techniques another note i would want to make how do
[00:07:45] another note i would want to make how do you choose the dimensionality for lsa it
[00:07:47] you choose the dimensionality for lsa it has this variable k corresponding to the
[00:07:49] has this variable k corresponding to the number of dimensions that you keep
[00:07:51] number of dimensions that you keep if you read the literature on lsa they
[00:07:53] if you read the literature on lsa they often imagine this kind of what i've
[00:07:55] often imagine this kind of what i've called the dream scenario where you plot
[00:07:57] called the dream scenario where you plot the singular values
[00:07:59] the singular values and you see that a lot of them are very
[00:08:00] and you see that a lot of them are very large and then there's a sudden drop off
[00:08:03] large and then there's a sudden drop off and if you do see this then it's obvious
[00:08:05] and if you do see this then it's obvious that you should pick the point of the
[00:08:06] that you should pick the point of the sudden drop off as your k so here you
[00:08:08] sudden drop off as your k so here you would pick k is 20 and you'd be
[00:08:10] would pick k is 20 and you'd be confident that you had captured almost
[00:08:12] confident that you had captured almost all the variation in your data in the
[00:08:13] all the variation in your data in the reduced dimensional space you were
[00:08:15] reduced dimensional space you were creating
[00:08:16] creating unfortunately
[00:08:18] unfortunately for the kinds of matrices and problems
[00:08:20] for the kinds of matrices and problems that we're looking at i really never see
[00:08:21] that we're looking at i really never see the dream scenario what i see looks
[00:08:23] the dream scenario what i see looks something much more like this where you
[00:08:25] something much more like this where you have kind of a sudden drop off early and
[00:08:27] have kind of a sudden drop off early and then a long decline and maybe a sudden
[00:08:29] then a long decline and maybe a sudden drop off at the end and it's basically
[00:08:31] drop off at the end and it's basically totally unclear where in the space you
[00:08:33] totally unclear where in the space you should pick k
[00:08:35] should pick k and the result is that k is often chosen
[00:08:37] and the result is that k is often chosen kind of empirically as a hyper parameter
[00:08:40] kind of empirically as a hyper parameter tuned against whatever problem you're
[00:08:42] tuned against whatever problem you're actually trying to solve
[00:08:44] actually trying to solve if in doing this work you do see the
[00:08:45] if in doing this work you do see the dream scenario please do write to me it
[00:08:47] dream scenario please do write to me it would be very exciting to see that
[00:08:49] would be very exciting to see that happen
[00:08:51] happen lsa is just one of a large family of
[00:08:54] lsa is just one of a large family of matrix decomposition methods um here's a
[00:08:57] matrix decomposition methods um here's a list of a few of them and a lot of them
[00:08:59] list of a few of them and a lot of them are implemented in scikit-learn in its
[00:09:01] are implemented in scikit-learn in its decomposition library and i would
[00:09:03] decomposition library and i would encourage you to try them out and just
[00:09:04] encourage you to try them out and just see how they perform on problems that
[00:09:06] see how they perform on problems that you're trying to solve
[00:09:09] you're trying to solve and finally here's a little bit of code
[00:09:11] and finally here's a little bit of code vsm.lsa
[00:09:12] vsm.lsa so with k set to 100 gives me back a
[00:09:15] so with k set to 100 gives me back a reduced dimensional version of that
[00:09:17] reduced dimensional version of that matrix keeping the same vocabulary of
[00:09:19] matrix keeping the same vocabulary of course but now with only 100 column
[00:09:21] course but now with only 100 column dimensions
[00:09:24] let's move to auto encoders this will be
[00:09:26] let's move to auto encoders this will be a point of contracts with lsa because
[00:09:28] a point of contracts with lsa because this is a much more powerful method
[00:09:30] this is a much more powerful method so here's the overview auto encoders are
[00:09:32] so here's the overview auto encoders are a flexible class of deep learning
[00:09:34] a flexible class of deep learning architectures for learning reduce
[00:09:35] architectures for learning reduce dimensional representations if you want
[00:09:38] dimensional representations if you want to hear much more about this classroom
[00:09:40] to hear much more about this classroom models i would encourage you to read
[00:09:41] models i would encourage you to read chapter 14 of the good fellow at all
[00:09:43] chapter 14 of the good fellow at all book deep learning it has a lot of
[00:09:45] book deep learning it has a lot of details and a lot of variations on this
[00:09:47] details and a lot of variations on this theme
[00:09:48] theme here is the basic autoencoder model the
[00:09:51] here is the basic autoencoder model the input would be the say the vectors from
[00:09:54] input would be the say the vectors from our the rows in our matrices so this
[00:09:56] our the rows in our matrices so this could be the counts or something that
[00:09:58] could be the counts or something that you've done to the counts
[00:10:00] you've done to the counts those are fed through a hidden layer of
[00:10:02] those are fed through a hidden layer of representation and then the goal of this
[00:10:04] representation and then the goal of this model is try to is to try to literally
[00:10:06] model is try to is to try to literally reconstruct the input
[00:10:09] reconstruct the input now that might be trivial if h had the
[00:10:11] now that might be trivial if h had the same dimensionality as x but the whole
[00:10:13] same dimensionality as x but the whole idea here is that you're going to feed
[00:10:15] idea here is that you're going to feed the input through a very narrow pipe and
[00:10:18] the input through a very narrow pipe and then try to reconstruct the input
[00:10:20] then try to reconstruct the input given that you're feeding it through a
[00:10:21] given that you're feeding it through a potentially very very narrow pipe it's
[00:10:23] potentially very very narrow pipe it's unlikely that you'll be able to fully
[00:10:25] unlikely that you'll be able to fully reconstruct the inputs but the idea is
[00:10:28] reconstruct the inputs but the idea is that the model will learn to reconstruct
[00:10:29] that the model will learn to reconstruct the important sources of variation in
[00:10:32] the important sources of variation in performing this auto encoding step
[00:10:35] performing this auto encoding step and then when we use these models for
[00:10:37] and then when we use these models for representation learning in the mode that
[00:10:39] representation learning in the mode that we've been in for this unit the
[00:10:41] we've been in for this unit the representation that we choose is this
[00:10:42] representation that we choose is this hidden unit here we typically don't care
[00:10:45] hidden unit here we typically don't care about what was reconstructed on the
[00:10:47] about what was reconstructed on the output but rather only about the hidden
[00:10:49] output but rather only about the hidden reduced dimensional representation that
[00:10:51] reduced dimensional representation that the model learned this slide has a bunch
[00:10:52] the model learned this slide has a bunch of other annotations on it and the
[00:10:54] of other annotations on it and the reason i included them is that the
[00:10:56] reason i included them is that the course repository includes a reference
[00:10:58] course repository includes a reference implementation of an autoencoder and all
[00:11:00] implementation of an autoencoder and all the other deep learning models that we
[00:11:01] the other deep learning models that we cover in pure numpy and so if you wanted
[00:11:04] cover in pure numpy and so if you wanted to understand all of the technical
[00:11:05] to understand all of the technical details of how the model was constructed
[00:11:07] details of how the model was constructed and optimized you could use this as a
[00:11:09] and optimized you could use this as a kind of cheat sheet to understand how
[00:11:11] kind of cheat sheet to understand how the code works
[00:11:13] the code works i think the fundamental idea that you
[00:11:14] i think the fundamental idea that you want to have is simply that the model is
[00:11:16] want to have is simply that the model is trying to reconstruct its inputs the
[00:11:18] trying to reconstruct its inputs the error signal that we get is the
[00:11:20] error signal that we get is the difference between the reconstructed and
[00:11:21] difference between the reconstructed and actual input and that error signal is
[00:11:24] actual input and that error signal is what we use to update the parameters of
[00:11:26] what we use to update the parameters of the model
[00:11:29] the model final thing i would mention here is that
[00:11:30] final thing i would mention here is that it could be very difficult for this
[00:11:32] it could be very difficult for this model if you feed in the raw count
[00:11:34] model if you feed in the raw count vectors down here they have very high
[00:11:35] vectors down here they have very high dimensionality and their distribution is
[00:11:37] dimensionality and their distribution is highly skewed as we've seen so it can be
[00:11:40] highly skewed as we've seen so it can be very productive to do a little bit of
[00:11:41] very productive to do a little bit of re-weighting and maybe even
[00:11:43] re-weighting and maybe even dimensionality reduction with lsa before
[00:11:45] dimensionality reduction with lsa before you start feeding inputs into this model
[00:11:48] you start feeding inputs into this model of course it could still be meaningful
[00:11:50] of course it could still be meaningful even if you've done lsa as a
[00:11:51] even if you've done lsa as a pre-processing step to learn a hidden
[00:11:53] pre-processing step to learn a hidden dimensional representation because this
[00:11:55] dimensional representation because this model is presumably capable of learning
[00:11:58] model is presumably capable of learning even more abstract notions than lsa is
[00:12:01] even more abstract notions than lsa is in virtue of its non-linearity at this
[00:12:03] in virtue of its non-linearity at this hidden layer
[00:12:05] hidden layer and here's a bit of code just showing
[00:12:07] and here's a bit of code just showing how this works using both the reference
[00:12:09] how this works using both the reference implementation that i mentioned as well
[00:12:11] implementation that i mentioned as well as a faster and more flexible torch auto
[00:12:13] as a faster and more flexible torch auto encoder which is also incorrect included
[00:12:15] encoder which is also incorrect included in the course repository i think the
[00:12:17] in the course repository i think the only interface thing to mention here is
[00:12:19] only interface thing to mention here is that these models have a fit method like
[00:12:21] that these models have a fit method like all the other machine learning models
[00:12:23] all the other machine learning models for this but those the fit method
[00:12:25] for this but those the fit method returns that hidden dimensional
[00:12:26] returns that hidden dimensional representation the target for our
[00:12:28] representation the target for our learning in this context which is a bit
[00:12:30] learning in this context which is a bit non-standard but it's the intended
[00:12:32] non-standard but it's the intended application
[00:12:33] application for this kind of representation
[00:12:37] the other thing i would mention is so
[00:12:38] the other thing i would mention is so let's see how well the auto encoder is
[00:12:40] let's see how well the auto encoder is performing this is the raw distances in
[00:12:43] performing this is the raw distances in the giga5 matrix for finance this is the
[00:12:45] the giga5 matrix for finance this is the count matrix it doesn't look great
[00:12:48] count matrix it doesn't look great if we run the autoencoder directly on
[00:12:50] if we run the autoencoder directly on the count matrix it looks a little
[00:12:51] the count matrix it looks a little better but it's still not excellent
[00:12:53] better but it's still not excellent if we think of this as part of a
[00:12:55] if we think of this as part of a pipeline where we first done positive
[00:12:57] pipeline where we first done positive point wise mutual information and then
[00:12:59] point wise mutual information and then lsa at dimension 100 and then done the
[00:13:01] lsa at dimension 100 and then done the auto encoding step it starts to look
[00:13:03] auto encoding step it starts to look like a really good and interesting
[00:13:04] like a really good and interesting semantic space and i think that's
[00:13:06] semantic space and i think that's pointing out the power of including the
[00:13:07] pointing out the power of including the auto encoder in a larger pipeline of
[00:13:10] auto encoder in a larger pipeline of pre-processing on the count matrices
[00:13:12] pre-processing on the count matrices okay let's turn to glove or global
[00:13:14] okay let's turn to glove or global vectors for the final major unit for
[00:13:16] vectors for the final major unit for this screencast
[00:13:18] this screencast here's a brief overview glove was
[00:13:19] here's a brief overview glove was introduced by um jeffrey pennington
[00:13:21] introduced by um jeffrey pennington richard socher and chris manning a
[00:13:23] richard socher and chris manning a stanford team in 2014.
[00:13:25] stanford team in 2014. roughly speaking the guiding idea here
[00:13:27] roughly speaking the guiding idea here is that we want to learn vectors for
[00:13:29] is that we want to learn vectors for words such that the dot product of those
[00:13:32] words such that the dot product of those vectors is proportional to the log
[00:13:34] vectors is proportional to the log probability of co-occurrence for those
[00:13:35] probability of co-occurrence for those words and i'll elaborate on that in a
[00:13:38] words and i'll elaborate on that in a second
[00:13:39] second for doing computational work we can rely
[00:13:42] for doing computational work we can rely on the implementation torchglove.pi
[00:13:44] on the implementation torchglove.pi which is in the course repo i'll mention
[00:13:46] which is in the course repo i'll mention that there's also a reference
[00:13:47] that there's also a reference implementation in bsn.pi it's very slow
[00:13:50] implementation in bsn.pi it's very slow but it kind of transparently implements
[00:13:52] but it kind of transparently implements the core glove algorithm so it could be
[00:13:54] the core glove algorithm so it could be interesting to inspect and then if
[00:13:56] interesting to inspect and then if you're doing practical work with really
[00:13:57] you're doing practical work with really large corporate and really large
[00:13:59] large corporate and really large vocabularies i would encourage you to
[00:14:00] vocabularies i would encourage you to use the glove team cm limitation it's an
[00:14:03] use the glove team cm limitation it's an outstanding software artifact that will
[00:14:04] outstanding software artifact that will allow you to learn lots of good
[00:14:06] allow you to learn lots of good representations quickly
[00:14:08] representations quickly and that kind of brings me to my last
[00:14:09] and that kind of brings me to my last point i just want to mention that the
[00:14:11] point i just want to mention that the glove team was among the first teams in
[00:14:13] glove team was among the first teams in nlp to release not just data and code
[00:14:16] nlp to release not just data and code but pre-trained model parameters
[00:14:19] but pre-trained model parameters everyone does that these days but it was
[00:14:20] everyone does that these days but it was rare at the time and i think this team
[00:14:22] rare at the time and i think this team is kind of really
[00:14:24] is kind of really forward thinking and seeing the value of
[00:14:26] forward thinking and seeing the value of releasing these centralized resources
[00:14:27] releasing these centralized resources and a lot of really interesting work
[00:14:30] and a lot of really interesting work happened
[00:14:31] happened with glove vectors as a foundation
[00:14:35] all right so let's think about the
[00:14:36] all right so let's think about the technical aspects of this model this is
[00:14:37] technical aspects of this model this is the glove objective and you're going to
[00:14:39] the glove objective and you're going to see
[00:14:40] see point-wise mutual information kind of
[00:14:41] point-wise mutual information kind of creep into this picture in an
[00:14:43] creep into this picture in an interesting way
[00:14:44] interesting way so this is equation six from the paper
[00:14:46] so this is equation six from the paper it's kind of an idealized objective for
[00:14:47] it's kind of an idealized objective for the glove model and it says what i said
[00:14:50] the glove model and it says what i said before we have a row vector and a column
[00:14:52] before we have a row vector and a column vector
[00:14:53] vector i wi and w k we're going to get their
[00:14:56] i wi and w k we're going to get their dot product and the goal is to learn
[00:14:58] dot product and the goal is to learn to have that dot product be proportional
[00:15:00] to have that dot product be proportional to the log of the probability of
[00:15:02] to the log of the probability of co-occurrence of word i and word k
[00:15:04] co-occurrence of word i and word k where the probability of co-occurrence
[00:15:06] where the probability of co-occurrence is defined in the way that we defined it
[00:15:08] is defined in the way that we defined it before when we were talking about row
[00:15:10] before when we were talking about row normalization it's just done in log
[00:15:12] normalization it's just done in log space this is the co-occurrence count
[00:15:14] space this is the co-occurrence count this is the sum of all the counts along
[00:15:16] this is the sum of all the counts along that row and basically in log space
[00:15:18] that row and basically in log space we're just dividing this value by this
[00:15:20] we're just dividing this value by this value
[00:15:21] value so keep that in mind now the reason they
[00:15:23] so keep that in mind now the reason they have only the row represented is that in
[00:15:25] have only the row represented is that in the paper they're assuming that the rows
[00:15:27] the paper they're assuming that the rows and columns in the underlying count
[00:15:29] and columns in the underlying count matrix are identical and so we don't
[00:15:31] matrix are identical and so we don't need to include both however if we did
[00:15:33] need to include both however if we did allow that the row and context could be
[00:15:35] allow that the row and context could be different we would just elaborate
[00:15:37] different we would just elaborate equation 6 to have a slightly different
[00:15:39] equation 6 to have a slightly different denominator right we would have the
[00:15:40] denominator right we would have the product of the row sum and the column
[00:15:43] product of the row sum and the column sum and take the log of that and
[00:15:45] sum and take the log of that and subtract that out
[00:15:46] subtract that out and that would be kind of our goal for
[00:15:48] and that would be kind of our goal for learning these dot products here
[00:15:50] learning these dot products here but aha this is where pmi sneaks in
[00:15:52] but aha this is where pmi sneaks in because that simply is the pmi objective
[00:15:55] because that simply is the pmi objective right where we stated that as the log of
[00:15:57] right where we stated that as the log of the probability of co-occurrence divided
[00:16:00] the probability of co-occurrence divided by the product of the row and the column
[00:16:01] by the product of the row and the column probabilities here they've just stated
[00:16:03] probabilities here they've just stated exactly that calculation in log space
[00:16:05] exactly that calculation in log space and these are numerically equivalent by
[00:16:07] and these are numerically equivalent by the equivalence of log of x over y being
[00:16:10] the equivalence of log of x over y being the same as log of x minus the log of y
[00:16:13] the same as log of x minus the log of y so that's the deep connection that i was
[00:16:14] so that's the deep connection that i was highlighting between glove and pmi and i
[00:16:17] highlighting between glove and pmi and i think that's really interesting because
[00:16:18] think that's really interesting because it shows that fundamentally we're
[00:16:20] it shows that fundamentally we're testing a very similar hypothesis using
[00:16:23] testing a very similar hypothesis using very similar notions of row and column
[00:16:25] very similar notions of row and column context
[00:16:27] context now the glove team doesn't just stop
[00:16:28] now the glove team doesn't just stop there
[00:16:29] there the glove objective is actually much
[00:16:30] the glove objective is actually much more interesting as an elaboration of
[00:16:32] more interesting as an elaboration of that core that core pmi idea but it's
[00:16:35] that core that core pmi idea but it's worth having pmi in mind because it's
[00:16:37] worth having pmi in mind because it's there throughout this presentation
[00:16:40] there throughout this presentation in the paper they state this is a kind
[00:16:41] in the paper they state this is a kind of idealized objective where we're going
[00:16:43] of idealized objective where we're going to have the dot product as i said before
[00:16:45] to have the dot product as i said before and two bias terms and the goal will be
[00:16:46] and two bias terms and the goal will be to make that equivalent to the log of
[00:16:48] to make that equivalent to the log of the co-occurrence count
[00:16:50] the co-occurrence count that has some undesirable properties
[00:16:52] that has some undesirable properties from the point of view of machine
[00:16:54] from the point of view of machine learning so they propose in the end a
[00:16:56] learning so they propose in the end a weighted version of that objective
[00:16:58] weighted version of that objective you can see we still have the dot
[00:16:59] you can see we still have the dot product of the row and the column
[00:17:01] product of the row and the column vectors and two bias terms we're going
[00:17:03] vectors and two bias terms we're going to subtract out the log of the
[00:17:05] to subtract out the log of the co-occurrence count
[00:17:06] co-occurrence count and take the square of that and that's
[00:17:08] and take the square of that and that's going to be weighted by f of the
[00:17:10] going to be weighted by f of the co-occurrence count
[00:17:11] co-occurrence count where f is a function that you can
[00:17:13] where f is a function that you can define by hand and what they do in the
[00:17:14] define by hand and what they do in the paper is that it was two parameters x
[00:17:16] paper is that it was two parameters x max and alpha
[00:17:19] max and alpha for any count that is above x max we're
[00:17:21] for any count that is above x max we're going to set it to one to kind of
[00:17:22] going to set it to one to kind of flatten out all the really large counts
[00:17:25] flatten out all the really large counts everything that's below x max we will
[00:17:27] everything that's below x max we will take as a proportion of x max with some
[00:17:30] take as a proportion of x max with some exponential scaling that's specified by
[00:17:32] exponential scaling that's specified by alpha that's the function there and
[00:17:35] alpha that's the function there and typically alpha is set to 0.75 and x max
[00:17:37] typically alpha is set to 0.75 and x max 200 but i encourage you to be critical
[00:17:40] 200 but i encourage you to be critical in thinking about both those choices and
[00:17:42] in thinking about both those choices and how they relate to your data i'll return
[00:17:44] how they relate to your data i'll return to that in a second
[00:17:46] to that in a second so glove really has these three hyper
[00:17:48] so glove really has these three hyper parameters the learned dimensional the
[00:17:50] parameters the learned dimensional the dimensionality of the representations
[00:17:53] dimensionality of the representations x max which is going to have this
[00:17:54] x max which is going to have this flattening effect and alpha which is
[00:17:56] flattening effect and alpha which is going to scale the values right so
[00:17:58] going to scale the values right so here's an example of how those are
[00:18:00] here's an example of how those are interacting
[00:18:01] interacting x max and alpha if i start with this
[00:18:03] x max and alpha if i start with this vector 199 75 10 and 1
[00:18:06] vector 199 75 10 and 1 the the function f as we specified it is
[00:18:09] the the function f as we specified it is going to flatten that out into 1 99 81
[00:18:12] going to flatten that out into 1 99 81 18 and 0.03
[00:18:14] 18 and 0.03 you should just be aware that that kind
[00:18:16] you should just be aware that that kind of flattening is happening
[00:18:19] of flattening is happening so glove learning so it's kind of
[00:18:21] so glove learning so it's kind of interesting to think about analytically
[00:18:22] interesting to think about analytically about how love manages to learn
[00:18:24] about how love manages to learn interesting representations and one
[00:18:26] interesting representations and one thing that might be on your mind is the
[00:18:28] thing that might be on your mind is the question can it actually learn higher
[00:18:30] question can it actually learn higher order notions of co-occurrence that's
[00:18:31] order notions of co-occurrence that's been the major selling point of this
[00:18:33] been the major selling point of this lecture i gave that example involving
[00:18:35] lecture i gave that example involving gnarly wicked with lsa is glove going to
[00:18:38] gnarly wicked with lsa is glove going to be able to do that right we could just
[00:18:39] be able to do that right we could just pose that as a question
[00:18:42] pose that as a question so let's start that and see what happens
[00:18:44] so let's start that and see what happens see how this works the loss calculations
[00:18:46] see how this works the loss calculations for glove this is kind of a simplified
[00:18:48] for glove this is kind of a simplified version of the derivative of the model
[00:18:50] version of the derivative of the model um
[00:18:51] um and we're going to show how glove
[00:18:52] and we're going to show how glove manages to pull gnarly and wicked toward
[00:18:55] manages to pull gnarly and wicked toward awesome in that little little idealized
[00:18:57] awesome in that little little idealized space that i used before i'm going to
[00:18:59] space that i used before i'm going to leave out the bias terms for simplicity
[00:19:00] leave out the bias terms for simplicity but we could bring those in
[00:19:02] but we could bring those in and so here's the how this is going to
[00:19:04] and so here's the how this is going to proceed what i've done just for this
[00:19:06] proceed what i've done just for this idealized example is begin in a glove
[00:19:08] idealized example is begin in a glove space where wicked and gnarly are as far
[00:19:10] space where wicked and gnarly are as far apart as i could make them so as
[00:19:12] apart as i could make them so as different as i could possibly make them
[00:19:14] different as i could possibly make them i've got awesome and terrible and
[00:19:16] i've got awesome and terrible and awesome is kind of close to gnarly
[00:19:17] awesome is kind of close to gnarly already what you'll see is that after
[00:19:19] already what you'll see is that after just one iteration of the model what has
[00:19:22] just one iteration of the model what has happened is that wicked and gnarly have
[00:19:25] happened is that wicked and gnarly have been pulled toward awesome and that's
[00:19:27] been pulled toward awesome and that's just the kind of effect that we wanted
[00:19:28] just the kind of effect that we wanted that's the sense in which glove can
[00:19:30] that's the sense in which glove can capture these higher order notions of
[00:19:32] capture these higher order notions of co-occurrence
[00:19:33] co-occurrence just in a little more detail you might
[00:19:35] just in a little more detail you might want to study this on your own but the
[00:19:37] want to study this on your own but the high level overview of exactly how that
[00:19:39] high level overview of exactly how that learning happens proceeds as follows we
[00:19:41] learning happens proceeds as follows we start from these counts up here and the
[00:19:43] start from these counts up here and the crucial assumption i'm making is that
[00:19:45] crucial assumption i'm making is that wicked and gnarly never co-occur
[00:19:48] wicked and gnarly never co-occur but they occur a lot with awesome
[00:19:50] but they occur a lot with awesome awesome will be kind of the
[00:19:51] awesome will be kind of the gravitational pull that makes gnarly and
[00:19:53] gravitational pull that makes gnarly and wicked look similar
[00:19:56] wicked look similar keep in mind that because of that
[00:19:57] keep in mind that because of that function f by and large with glove we're
[00:19:59] function f by and large with glove we're dealing not with the raw counts but
[00:20:01] dealing not with the raw counts but rather with the re-weighted matrix and
[00:20:02] rather with the re-weighted matrix and that preserves this property that they
[00:20:04] that preserves this property that they never co-occurred it gives differently
[00:20:06] never co-occurred it gives differently scaled values for the rest of the
[00:20:08] scaled values for the rest of the co-occurrence or pseudo-co-occurrence
[00:20:10] co-occurrence or pseudo-co-occurrence probabilities
[00:20:12] probabilities all right and then here's so here's what
[00:20:13] all right and then here's so here's what we're going to track this is the gnarly
[00:20:15] we're going to track this is the gnarly vector in zero and you can see i've made
[00:20:17] vector in zero and you can see i've made them as far apart as i could they're
[00:20:18] them as far apart as i could they're kind of opposed to each other
[00:20:20] kind of opposed to each other we're going to see how they get pulled
[00:20:21] we're going to see how they get pulled toward awesome in the context factor
[00:20:24] toward awesome in the context factor so this is that loss calculation i have
[00:20:26] so this is that loss calculation i have just plugged in all the values here and
[00:20:28] just plugged in all the values here and you can see that we get this initial set
[00:20:30] you can see that we get this initial set of losses
[00:20:32] of losses that's after one iteration we update the
[00:20:34] that's after one iteration we update the weight matrices and we perform one more
[00:20:36] weight matrices and we perform one more round of learning and you can see that
[00:20:38] round of learning and you can see that both of these models the values here are
[00:20:40] both of these models the values here are getting larger corresponding to get
[00:20:42] getting larger corresponding to get getting pulled closer and closer toward
[00:20:43] getting pulled closer and closer toward awesome and you can see that graphically
[00:20:46] awesome and you can see that graphically happening over here in these plots on
[00:20:48] happening over here in these plots on the left and as i do more iterations of
[00:20:50] the left and as i do more iterations of the glove model this effect is just
[00:20:52] the glove model this effect is just going to strengthen corresponding to
[00:20:54] going to strengthen corresponding to wicked and gnarly getting pulled toward
[00:20:56] wicked and gnarly getting pulled toward awesome and away from terrible as a
[00:20:58] awesome and away from terrible as a result of these underlying counts and i
[00:21:00] result of these underlying counts and i take this as good evidence that glove
[00:21:02] take this as good evidence that glove like the other methods we've discussed
[00:21:04] like the other methods we've discussed is capable of capturing those higher
[00:21:06] is capable of capturing those higher order notions of co-occurrence that
[00:21:08] order notions of co-occurrence that we're so interested in pursuing with
[00:21:10] we're so interested in pursuing with these methods
[00:21:13] let's close the loop also we have those
[00:21:15] let's close the loop also we have those central questions what is glove doing to
[00:21:17] central questions what is glove doing to our underlying spaces with glove because
[00:21:19] our underlying spaces with glove because of the design of the matrix we have to
[00:21:21] of the design of the matrix we have to begin from
[00:21:23] begin from word by word co-occurrence matrices of
[00:21:25] word by word co-occurrence matrices of counts
[00:21:26] counts so we begin with these raw count values
[00:21:28] so we begin with these raw count values and gloves one-stop shopping it's going
[00:21:30] and gloves one-stop shopping it's going to take us all the way to these reduced
[00:21:32] to take us all the way to these reduced dimensional representations and boy by
[00:21:34] dimensional representations and boy by the criteria we've set up does glove do
[00:21:36] the criteria we've set up does glove do an outstanding job this is the result of
[00:21:39] an outstanding job this is the result of running glove at dimension 50
[00:21:41] running glove at dimension 50 and you can see that the values are
[00:21:43] and you can see that the values are extremely well scaled between negative
[00:21:44] extremely well scaled between negative two and two and nicely normally
[00:21:47] two and two and nicely normally distributed this is an outstanding input
[00:21:49] distributed this is an outstanding input to modern machine learning models and i
[00:21:52] to modern machine learning models and i think this is probably a non-trivial
[00:21:54] think this is probably a non-trivial aspect of why glove has been so
[00:21:56] aspect of why glove has been so successful as a kind of pre-trained
[00:21:59] successful as a kind of pre-trained basis for a lot of subsequent machine
[00:22:01] basis for a lot of subsequent machine learning architectures
[00:22:04] and then here's a little bit of code
[00:22:06] and then here's a little bit of code just showing you how you can work with
[00:22:07] just showing you how you can work with these interfaces using our code base the
[00:22:10] these interfaces using our code base the one thing i wanted to call out is that
[00:22:11] one thing i wanted to call out is that i'm trying to be careful i have defined
[00:22:13] i'm trying to be careful i have defined this function percentage non-zero values
[00:22:16] this function percentage non-zero values above
[00:22:17] above and you can set x max here and just feed
[00:22:19] and you can set x max here and just feed in a matrix and study what percentage of
[00:22:21] in a matrix and study what percentage of the values in that matrix are going to
[00:22:23] the values in that matrix are going to get flattened out to 1 as a result of
[00:22:25] get flattened out to 1 as a result of the x max that you've chosen and this
[00:22:28] the x max that you've chosen and this value really varies by the design of the
[00:22:30] value really varies by the design of the matrix if i feed in yelp 5
[00:22:33] matrix if i feed in yelp 5 only about 5 of the values are getting
[00:22:35] only about 5 of the values are getting flattened out but if i feed in yelp 20
[00:22:38] flattened out but if i feed in yelp 20 which is much denser and has much higher
[00:22:40] which is much denser and has much higher counts
[00:22:41] counts 20 of the values are getting flattened
[00:22:43] 20 of the values are getting flattened out to one you know if this number gets
[00:22:45] out to one you know if this number gets too high the matrix might become
[00:22:47] too high the matrix might become completely homogeneous and so should we
[00:22:48] completely homogeneous and so should we we should really be aware
[00:22:50] we should really be aware of how the setting of x max is affecting
[00:22:53] of how the setting of x max is affecting the kind of learning that we could even
[00:22:54] the kind of learning that we could even be performing with glove and it might
[00:22:57] be performing with glove and it might turn out that this is even more
[00:22:58] turn out that this is even more important than the number of iterations
[00:23:00] important than the number of iterations or the dimensionality of the
[00:23:02] or the dimensionality of the representations that you learn
[00:23:04] representations that you learn once you've made those choices though
[00:23:05] once you've made those choices though the interface is pretty clear and the
[00:23:07] the interface is pretty clear and the fit method as with the autoencoder
[00:23:09] fit method as with the autoencoder returns the matrix of learned
[00:23:11] returns the matrix of learned representations that we want to use for
[00:23:13] representations that we want to use for the current purposes
[00:23:14] the current purposes and then finally i've included a score
[00:23:17] and then finally i've included a score method and the score method is literally
[00:23:19] method and the score method is literally just testing to see
[00:23:21] just testing to see how well the vectors that you've learned
[00:23:23] how well the vectors that you've learned correspond to the glove objective of
[00:23:25] correspond to the glove objective of having the dot products be proportional
[00:23:28] having the dot products be proportional to the log of the co-occurrence
[00:23:29] to the log of the co-occurrence probabilities and you can get a score
[00:23:31] probabilities and you can get a score for that we're doing pretty well here
[00:23:32] for that we're doing pretty well here let's say for a large empirical matrix
[00:23:38] final section let's just say a bit about
[00:23:39] final section let's just say a bit about visualization and this is our
[00:23:41] visualization and this is our dimensionality reduction technique in
[00:23:43] dimensionality reduction technique in the sense that the whole point is to try
[00:23:44] the sense that the whole point is to try to flatten out a very high dimensional
[00:23:46] to flatten out a very high dimensional space into possibly two or three
[00:23:48] space into possibly two or three dimensions
[00:23:49] dimensions you have to recognize that inevitably
[00:23:51] you have to recognize that inevitably this will involve a lot of compromises
[00:23:53] this will involve a lot of compromises it's just impossible to capture all the
[00:23:55] it's just impossible to capture all the sources of variation in your underlying
[00:23:57] sources of variation in your underlying matrix in just a few dimensions but
[00:24:00] matrix in just a few dimensions but nonetheless this can be productive i
[00:24:02] nonetheless this can be productive i think it's especially valuable if you
[00:24:04] think it's especially valuable if you pair it with some kind of hands-on
[00:24:07] pair it with some kind of hands-on qualitative exploration using something
[00:24:09] qualitative exploration using something like bsm neighbors to understand at a
[00:24:11] like bsm neighbors to understand at a low level what your matrix encodes and
[00:24:14] low level what your matrix encodes and then the high-level visualizations can
[00:24:15] then the high-level visualizations can be a kind of counterpart to that
[00:24:18] be a kind of counterpart to that there are many visualization techniques
[00:24:20] there are many visualization techniques and a lot of them are implemented in the
[00:24:22] and a lot of them are implemented in the scikit manifold
[00:24:24] scikit manifold package so i encourage you to use them
[00:24:26] package so i encourage you to use them i'm going to show you some results from
[00:24:28] i'm going to show you some results from tis knee which is which stands for t
[00:24:30] tis knee which is which stands for t distributed
[00:24:31] distributed stochastic neighbor embedding
[00:24:33] stochastic neighbor embedding there are lots of user guides there that
[00:24:35] there are lots of user guides there that you can study for more details let me
[00:24:36] you can study for more details let me just give you the high level this is to
[00:24:38] just give you the high level this is to sne run on our giga20 matrix i think
[00:24:41] sne run on our giga20 matrix i think this is typical of pretty good output
[00:24:43] this is typical of pretty good output from to so what we're seeing here is
[00:24:45] from to so what we're seeing here is some pockets of high density those are
[00:24:48] some pockets of high density those are areas of local coherence globally we
[00:24:50] areas of local coherence globally we should be careful not to over interpret
[00:24:52] should be careful not to over interpret this entire diagram because as you rerun
[00:24:54] this entire diagram because as you rerun the model with different random seeds
[00:24:56] the model with different random seeds you'll see that it gets kind of
[00:24:57] you'll see that it gets kind of reoriented in different parts are
[00:24:59] reoriented in different parts are correspondingly close to different other
[00:25:01] correspondingly close to different other parts but what you can count on pretty
[00:25:04] parts but what you can count on pretty reliably is that these local pockets of
[00:25:06] reliably is that these local pockets of co of coherence correspond to coherence
[00:25:09] co of coherence correspond to coherence parts of the space that you've defined
[00:25:11] parts of the space that you've defined and if you zoom in on them you can kind
[00:25:13] and if you zoom in on them you can kind of assess what the model has uncovered
[00:25:15] of assess what the model has uncovered so for this giga21 for example i think
[00:25:17] so for this giga21 for example i think we see prominent clusters corresponding
[00:25:19] we see prominent clusters corresponding to things like cooking and conflict if
[00:25:22] to things like cooking and conflict if we do the same thing for our yelp matrix
[00:25:24] we do the same thing for our yelp matrix again this looks pretty good in terms of
[00:25:27] again this looks pretty good in terms of having some substructure that we could
[00:25:29] having some substructure that we could analyze and if we zoom in we do see
[00:25:31] analyze and if we zoom in we do see clusters like positive terms and
[00:25:33] clusters like positive terms and negative terms corresponding to the
[00:25:35] negative terms corresponding to the evaluative setting of these yelp reviews
[00:25:37] evaluative setting of these yelp reviews so this is all very encouraging and
[00:25:39] so this is all very encouraging and suggests that the underlying spaces have
[00:25:41] suggests that the underlying spaces have some really interesting structure that
[00:25:42] some really interesting structure that might be useful for subsequent analysis
[00:25:47] might be useful for subsequent analysis and here are some code snippets we have
[00:25:48] and here are some code snippets we have this simple wrapper around the scikit
[00:25:51] this simple wrapper around the scikit disney implementation that will allow
[00:25:53] disney implementation that will allow you to flexibly work with this stuff
[00:25:55] you to flexibly work with this stuff using the
[00:25:56] using the count matrices from our unit and i'm
[00:25:59] count matrices from our unit and i'm just mentioning here that it's pretty
[00:26:00] just mentioning here that it's pretty easy if you wanted to color code the
[00:26:02] easy if you wanted to color code the words in your vocabulary say according
[00:26:04] words in your vocabulary say according to a sentiment lexicon or some other
[00:26:06] to a sentiment lexicon or some other kind of lexicon that could be a way for
[00:26:08] kind of lexicon that could be a way for you to reveal exactly what structure
[00:26:10] you to reveal exactly what structure your model has been able to uncover with
[00:26:12] your model has been able to uncover with respect to those underlying labels and
[00:26:14] respect to those underlying labels and that can be useful

Retrofitting | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=2dVdZ4GPQIk

---

Transcript

[00:00:04] hello everyone welcome to part six in
[00:00:06] hello everyone welcome to part six in our series on distributed word
[00:00:07] our series on distributed word representations this can be considered
[00:00:09] representations this can be considered an optional part but it's on the
[00:00:10] an optional part but it's on the irresistibly cool idea of retrofitting
[00:00:13] irresistibly cool idea of retrofitting vectors to knowledge
[00:00:14] vectors to knowledge graphs here are central goals on the one
[00:00:17] graphs here are central goals on the one hand as we've seen distributional
[00:00:19] hand as we've seen distributional representations are powerful and also
[00:00:21] representations are powerful and also easy to obtain
[00:00:22] easy to obtain but they tend to reflect only relatively
[00:00:25] but they tend to reflect only relatively primitive semantic notions like
[00:00:26] primitive semantic notions like similarity or synonymy or connotation or
[00:00:29] similarity or synonymy or connotation or relatedness
[00:00:30] relatedness so that might feel limiting
[00:00:32] so that might feel limiting on the other hand structured resources
[00:00:34] on the other hand structured resources like knowledge graphs while sparse and
[00:00:36] like knowledge graphs while sparse and kind of hard to obtain
[00:00:39] kind of hard to obtain support really rich learning of very
[00:00:41] support really rich learning of very diverse semantic distinctions so the
[00:00:44] diverse semantic distinctions so the question naturally arises can we have
[00:00:46] question naturally arises can we have the best aspects of both of these and
[00:00:48] the best aspects of both of these and the inspiring answer given by
[00:00:50] the inspiring answer given by retrofitting is yes we can combine them
[00:00:53] retrofitting is yes we can combine them the original method for doing this is
[00:00:54] the original method for doing this is due to this lovely paper faroukian all
[00:00:56] due to this lovely paper faroukian all 2015 which i'm going to be giving a
[00:00:59] 2015 which i'm going to be giving a brief summary of in this screencast
[00:01:02] brief summary of in this screencast so here is the retrofitting model it
[00:01:04] so here is the retrofitting model it consists of two sums and they constitute
[00:01:07] consists of two sums and they constitute kind of opposing forces
[00:01:09] kind of opposing forces imagine that we have an existing
[00:01:11] imagine that we have an existing embedding space like glove or some
[00:01:13] embedding space like glove or some embedding space that you built yourself
[00:01:14] embedding space that you built yourself that's q hat
[00:01:16] that's q hat and we're learning these q i's and q j's
[00:01:19] and we're learning these q i's and q j's the term on the left is basically saying
[00:01:21] the term on the left is basically saying remain faithful to those original
[00:01:23] remain faithful to those original vectors as you learn these new vectors q
[00:01:25] vectors as you learn these new vectors q i try not to be too dissimilar from
[00:01:27] i try not to be too dissimilar from where you started
[00:01:29] where you started that pressure is balanced against the
[00:01:31] that pressure is balanced against the pressure on the right which is saying
[00:01:33] pressure on the right which is saying make representations that look more like
[00:01:36] make representations that look more like the neighbors
[00:01:37] the neighbors for the current node in the knowledge
[00:01:40] for the current node in the knowledge graph which is you know
[00:01:41] graph which is you know defined by the set of
[00:01:43] defined by the set of relations e
[00:01:45] relations e so two opposing pressures on the one
[00:01:47] so two opposing pressures on the one hand we're saying be faithful to the
[00:01:48] hand we're saying be faithful to the original on the other hand we're saying
[00:01:50] original on the other hand we're saying look more like your neighbors in the
[00:01:51] look more like your neighbors in the knowledge graph
[00:01:53] knowledge graph if we set alpha to one and beta to one
[00:01:55] if we set alpha to one and beta to one over the out degree for the node that
[00:01:57] over the out degree for the node that we're targeting
[00:01:58] we're targeting then we have basically balance these two
[00:02:01] then we have basically balance these two pressures
[00:02:02] pressures if we set alpha really large we'll
[00:02:04] if we set alpha really large we'll mostly want to stay faithful to the
[00:02:05] mostly want to stay faithful to the original vectors if we set beta
[00:02:08] original vectors if we set beta comparatively very large then we'll
[00:02:10] comparatively very large then we'll mostly want to look like the neighbors
[00:02:11] mostly want to look like the neighbors in the knowledge graph and we won't
[00:02:13] in the knowledge graph and we won't remain so tethered to the original
[00:02:15] remain so tethered to the original embedding space that we started with
[00:02:17] embedding space that we started with this illustration kind of nicely depicts
[00:02:19] this illustration kind of nicely depicts what happens in the model the gray
[00:02:21] what happens in the model the gray vectors of the original embedding space
[00:02:23] vectors of the original embedding space we have this knowledge graph that
[00:02:25] we have this knowledge graph that connects the associated nodes and
[00:02:27] connects the associated nodes and because they're connected in the
[00:02:29] because they're connected in the retrofitting space which is given in
[00:02:30] retrofitting space which is given in white these nodes are kind of pulled
[00:02:32] white these nodes are kind of pulled together and look more
[00:02:34] together and look more similar there's a bunch of code for
[00:02:36] similar there's a bunch of code for doing retrofitting in the course
[00:02:38] doing retrofitting in the course repository and i'll just show you a few
[00:02:39] repository and i'll just show you a few quick illustrations using that code
[00:02:41] quick illustrations using that code let's start with a simple case we have a
[00:02:43] let's start with a simple case we have a very simple knowledge graph where node
[00:02:45] very simple knowledge graph where node zero is connected to node one and node
[00:02:48] zero is connected to node one and node zero is connected to node two just
[00:02:50] zero is connected to node two just directionally
[00:02:52] directionally what happens when we run the
[00:02:52] what happens when we run the retrofitting model is that zero is
[00:02:54] retrofitting model is that zero is pulled equally close to one into two
[00:02:57] pulled equally close to one into two kind of equidistant between them and
[00:02:59] kind of equidistant between them and closer to both than it was in the
[00:03:00] closer to both than it was in the original embedding
[00:03:03] original embedding here's space
[00:03:03] here's space situation in which every other every
[00:03:05] situation in which every other every node is connected to every other node
[00:03:07] node is connected to every other node that's represented on the left here
[00:03:08] that's represented on the left here that's where we start and as a result of
[00:03:10] that's where we start and as a result of running the retrofitting model with
[00:03:12] running the retrofitting model with alpha and beta set in their default
[00:03:13] alpha and beta set in their default parameters what happens is that triangle
[00:03:15] parameters what happens is that triangle just gets smaller and a kind of fully
[00:03:17] just gets smaller and a kind of fully symmetric way as the
[00:03:19] symmetric way as the nodes become more similar to each other
[00:03:21] nodes become more similar to each other because of the graph structure
[00:03:24] because of the graph structure here's a kind of degenerate solution if
[00:03:25] here's a kind of degenerate solution if i set alpha to 0 i have no pressure to
[00:03:28] i set alpha to 0 i have no pressure to be faithful to the original vectors all
[00:03:30] be faithful to the original vectors all i care about is looking like my
[00:03:31] i care about is looking like my neighbors from the term on the right and
[00:03:33] neighbors from the term on the right and as a result all these vectors shrink
[00:03:35] as a result all these vectors shrink down to be the same point after the
[00:03:36] down to be the same point after the model is run for a few iterations
[00:03:39] model is run for a few iterations if instead i had done the opposite of
[00:03:41] if instead i had done the opposite of made alpha really large comparative to
[00:03:43] made alpha really large comparative to beta then basically nothing would have
[00:03:45] beta then basically nothing would have happened in the learning a triangle
[00:03:46] happened in the learning a triangle would remain its original size
[00:03:50] would remain its original size it's worth considering some extensions
[00:03:51] it's worth considering some extensions so i think one the fundamental
[00:03:53] so i think one the fundamental limitation of this model is that it is
[00:03:55] limitation of this model is that it is kind of assuming right there in its
[00:03:57] kind of assuming right there in its objective that had to have an edge
[00:03:59] objective that had to have an edge between nodes is to say that they are
[00:04:00] between nodes is to say that they are similar but of course the whole point
[00:04:02] similar but of course the whole point might be that your knowledge graph has
[00:04:04] might be that your knowledge graph has very rich edge relations corresponding
[00:04:06] very rich edge relations corresponding to different linguistic notions like
[00:04:08] to different linguistic notions like autonomy
[00:04:09] autonomy and we certainly wouldn't want to treat
[00:04:10] and we certainly wouldn't want to treat synonymy and autonomy as the same
[00:04:13] synonymy and autonomy as the same relation and just assume that it meant
[00:04:14] relation and just assume that it meant similarity in our model
[00:04:16] similarity in our model uh so there are various extensions i
[00:04:18] uh so there are various extensions i think the most general extension that
[00:04:20] think the most general extension that i've seen is from a paper that i was
[00:04:22] i've seen is from a paper that i was involved with led by ben lingarich which
[00:04:24] involved with led by ben lingarich which is called functional retrofitting which
[00:04:26] is called functional retrofitting which allows you to very flexibly learn
[00:04:28] allows you to very flexibly learn different retrofitting modes for
[00:04:30] different retrofitting modes for different edge semantics
[00:04:33] different edge semantics and once you start down that road you
[00:04:34] and once you start down that road you have a really natural connection with
[00:04:36] have a really natural connection with the literature on graph embedding that
[00:04:37] the literature on graph embedding that is learning distributional
[00:04:38] is learning distributional representations for nodes and knowledge
[00:04:41] representations for nodes and knowledge graphs and this paper but led by will
[00:04:43] graphs and this paper but led by will hamilton is an outstanding overview of
[00:04:45] hamilton is an outstanding overview of methods in that space and then you have
[00:04:47] methods in that space and then you have this nice synergy between nlp methods
[00:04:49] this nice synergy between nlp methods and methods that are more associated
[00:04:51] and methods that are more associated with work on knowledge graphs and social
[00:04:53] with work on knowledge graphs and social networks and so forth
[00:04:56] networks and so forth and finally here are some code snippets
[00:04:57] and finally here are some code snippets just showing some simple illustrations
[00:04:59] just showing some simple illustrations of the short sort that i showed you
[00:05:01] of the short sort that i showed you earlier in the screencast i would just
[00:05:03] earlier in the screencast i would just mention at the end here if you would
[00:05:04] mention at the end here if you would like to apply these methods to wordnet
[00:05:07] like to apply these methods to wordnet which could be a powerful ingredient for
[00:05:09] which could be a powerful ingredient for the first assignment and bake off i
[00:05:11] the first assignment and bake off i would encourage you to check out this
[00:05:12] would encourage you to check out this notebook bsm03 retrofitting because it
[00:05:15] notebook bsm03 retrofitting because it walks through all the steps for doing
[00:05:17] walks through all the steps for doing that

Static Representations | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=K7wM6FUV0ds

---

Transcript

[00:00:04] hello everyone welcome to our final
[00:00:06] hello everyone welcome to our final screencast in our unit on distributed
[00:00:08] screencast in our unit on distributed word representations our topic is going
[00:00:10] word representations our topic is going to be deriving static representations
[00:00:12] to be deriving static representations from contextual models that might sound
[00:00:14] from contextual models that might sound awfully specific but as you'll see i
[00:00:15] awfully specific but as you'll see i think this could be really empowering
[00:00:17] think this could be really empowering for you as you work on your original
[00:00:19] for you as you work on your original system for the assignment and the
[00:00:21] system for the assignment and the associated bake off
[00:00:23] associated bake off so let's dive in a question on your
[00:00:25] so let's dive in a question on your minds might be how can i use bert or
[00:00:27] minds might be how can i use bert or related models like roberta or excel net
[00:00:29] related models like roberta or excel net or electro in the context of deriving
[00:00:32] or electro in the context of deriving good static representations of words you
[00:00:34] good static representations of words you probably have heard about these models
[00:00:36] probably have heard about these models and heard that they lift all boats and
[00:00:37] and heard that they lift all boats and the question is how can you take
[00:00:39] the question is how can you take advantage of those of those benefits
[00:00:42] advantage of those of those benefits but there's a tension here we've been
[00:00:43] but there's a tension here we've been developing static representations but
[00:00:45] developing static representations but these models like bert are
[00:00:47] these models like bert are designed to deliver contextual
[00:00:49] designed to deliver contextual representations of words and i'll return
[00:00:51] representations of words and i'll return to what that means in a second but that
[00:00:53] to what that means in a second but that is the central tension between static
[00:00:55] is the central tension between static and contextual so the question is are
[00:00:57] and contextual so the question is are there good methods for deriving static
[00:00:59] there good methods for deriving static representations from the contextual ones
[00:01:02] representations from the contextual ones that these models offer
[00:01:04] that these models offer and the answer from bamasan yet all is
[00:01:06] and the answer from bamasan yet all is yes there are effective methods for
[00:01:08] yes there are effective methods for doing this and it's those methods that
[00:01:10] doing this and it's those methods that will be the focus of this screencast i
[00:01:12] will be the focus of this screencast i really want to do two things though for
[00:01:13] really want to do two things though for this lecture i would like to get hands
[00:01:15] this lecture i would like to get hands on a little bit with a high level
[00:01:18] on a little bit with a high level overview of models like bert we're going
[00:01:20] overview of models like bert we're going to look later in the quarter in much
[00:01:22] to look later in the quarter in much more detail at how these models work so
[00:01:24] more detail at how these models work so for now we're just going to treat them
[00:01:25] for now we're just going to treat them as kind of black boxes just like you
[00:01:27] as kind of black boxes just like you might look up a glove representation of
[00:01:29] might look up a glove representation of a word and just get back that
[00:01:31] a word and just get back that representation and use it so too here we
[00:01:33] representation and use it so too here we can think of these models as devices for
[00:01:35] can think of these models as devices for feeding in sequences and getting back
[00:01:37] feeding in sequences and getting back lots and lots of representations that we
[00:01:39] lots and lots of representations that we might use and later in the quarter we'll
[00:01:42] might use and later in the quarter we'll come to a deeper understanding of
[00:01:43] come to a deeper understanding of precisely where those representations
[00:01:45] precisely where those representations come from
[00:01:47] come from and in addition of course i want to give
[00:01:48] and in addition of course i want to give you an overview of these exciting
[00:01:50] you an overview of these exciting methods from bomasani at all in the
[00:01:52] methods from bomasani at all in the hopes that they are useful to you in
[00:01:54] hopes that they are useful to you in developing your original system
[00:01:57] developing your original system so let's start with the structure of
[00:01:59] so let's start with the structure of burt
[00:02:00] burt bert processes sequences here i've got a
[00:02:02] bert processes sequences here i've got a sequence the class token the day broke
[00:02:04] sequence the class token the day broke sep class and separate designated tokens
[00:02:07] sep class and separate designated tokens the class token typically starts the
[00:02:09] the class token typically starts the sequence and then set ends the sequence
[00:02:11] sequence and then set ends the sequence and can be also used internally in
[00:02:12] and can be also used internally in sequences to mark boundaries within the
[00:02:14] sequences to mark boundaries within the sequence that you're processing
[00:02:16] sequence that you're processing but the fundamental thing is that we
[00:02:18] but the fundamental thing is that we have this the short sentence the day
[00:02:20] have this the short sentence the day broke
[00:02:20] broke vert processes those into an embedding
[00:02:23] vert processes those into an embedding layer and then a lot of additional
[00:02:25] layer and then a lot of additional layers you know here i've depicted 4 but
[00:02:27] layers you know here i've depicted 4 but it could be 12 or even 24 layers
[00:02:30] it could be 12 or even 24 layers what we're seeing here the vec the
[00:02:32] what we're seeing here the vec the rectangles represent vectors they are
[00:02:34] rectangles represent vectors they are the outputs of each layer in the network
[00:02:38] the outputs of each layer in the network a lot of computation goes into computing
[00:02:41] a lot of computation goes into computing those output vector representations at
[00:02:43] those output vector representations at each layer we're going to set that
[00:02:44] each layer we're going to set that computation aside for now so that we can
[00:02:46] computation aside for now so that we can just think of this as a grid of vector
[00:02:49] just think of this as a grid of vector representations
[00:02:51] representations here's the crucial thing that makes vert
[00:02:52] here's the crucial thing that makes vert contextual for different sequences that
[00:02:55] contextual for different sequences that we process we will get very different
[00:02:57] we process we will get very different representations in fact
[00:03:00] representations in fact individual tokens
[00:03:02] individual tokens occurring in different sequences we'll
[00:03:03] occurring in different sequences we'll get very different representations i've
[00:03:05] get very different representations i've tried to signal that with the colors
[00:03:07] tried to signal that with the colors here so like the t's two sequences both
[00:03:09] here so like the t's two sequences both contain the word the and the word broke
[00:03:12] contain the word the and the word broke but in virtue of the fact that they have
[00:03:13] but in virtue of the fact that they have different surrounding material and
[00:03:15] different surrounding material and different positions in the sequence
[00:03:17] different positions in the sequence almost all of the representations will
[00:03:19] almost all of the representations will be different um
[00:03:20] be different um the class and set tokens might have the
[00:03:22] the class and set tokens might have the same embedding but through all of these
[00:03:24] same embedding but through all of these layers because of the way all these
[00:03:26] layers because of the way all these tokens are going to interact with each
[00:03:27] tokens are going to interact with each other when we derive the representations
[00:03:29] other when we derive the representations everything will be different we do not
[00:03:31] everything will be different we do not get a static representation out of these
[00:03:33] get a static representation out of these models and i've specified that even in
[00:03:36] models and i've specified that even in the embedding layer if the positions of
[00:03:38] the embedding layer if the positions of the words vary one in the same token
[00:03:40] the words vary one in the same token will get different representations the
[00:03:42] will get different representations the reason for that is that this embedding
[00:03:44] reason for that is that this embedding layer is actually hiding two components
[00:03:46] layer is actually hiding two components we do it at the very center of this
[00:03:49] we do it at the very center of this model have a fixed static embedding
[00:03:51] model have a fixed static embedding where we can look up individual word
[00:03:53] where we can look up individual word sequences
[00:03:54] sequences but for this thing that i've called the
[00:03:55] but for this thing that i've called the embedding layer that static
[00:03:57] embedding layer that static representation is combined with a
[00:03:58] representation is combined with a separate positional encoding from a
[00:04:00] separate positional encoding from a separate embedding space and that
[00:04:02] separate embedding space and that delivers what i've called the embedding
[00:04:04] delivers what i've called the embedding layer here and that means that even at
[00:04:06] layer here and that means that even at this first layer because for example the
[00:04:09] this first layer because for example the occurs in different points in the
[00:04:10] occurs in different points in the sequence it will get different
[00:04:12] sequence it will get different representations even in the embedding
[00:04:14] representations even in the embedding space
[00:04:15] space and from there of course as we travel
[00:04:16] and from there of course as we travel through these layers we expect even more
[00:04:18] through these layers we expect even more things to change about the
[00:04:20] things to change about the representations
[00:04:23] a second important preliminary is to
[00:04:26] a second important preliminary is to give some attention to how bert and
[00:04:28] give some attention to how bert and models like it tokenize sequences
[00:04:30] models like it tokenize sequences and here i've given you a bit of code in
[00:04:32] and here i've given you a bit of code in the hopes that you can get hands-on and
[00:04:33] the hopes that you can get hands-on and get a feel for how these tokenizers
[00:04:35] get a feel for how these tokenizers behave i'm taking advantage of the
[00:04:37] behave i'm taking advantage of the hugging face library i have loaded a
[00:04:39] hugging face library i have loaded a burp tokenizer and i load that from a
[00:04:41] burp tokenizer and i load that from a pre-trained model
[00:04:43] pre-trained model in cell 3 you can see that i've called
[00:04:45] in cell 3 you can see that i've called the tokenized function on the sentence
[00:04:46] the tokenized function on the sentence this isn't too surprising and the result
[00:04:48] this isn't too surprising and the result is a pretty normal looking sequence of
[00:04:50] is a pretty normal looking sequence of tokens you see some punctuation has been
[00:04:53] tokens you see some punctuation has been separated off but you also see a lot of
[00:04:54] separated off but you also see a lot of words when you get down to cell 4 though
[00:04:57] words when you get down to cell 4 though for the sequence in codeme this is a bit
[00:05:00] for the sequence in codeme this is a bit surprising the word encode in the input
[00:05:02] surprising the word encode in the input has been broken apart into two sub-word
[00:05:04] has been broken apart into two sub-word tokens
[00:05:05] tokens en and then code with these boundary
[00:05:07] en and then code with these boundary markers on it bert has broken that apart
[00:05:10] markers on it bert has broken that apart into two subword sequences and if i feed
[00:05:13] into two subword sequences and if i feed in a sequence that has a really
[00:05:15] in a sequence that has a really unfamiliar set of
[00:05:17] unfamiliar set of tokens in it it will do a lot of
[00:05:19] tokens in it it will do a lot of breaking a part of that sequence as you
[00:05:21] breaking a part of that sequence as you can see in cell five for the input
[00:05:22] can see in cell five for the input snuffleupagus where a lot of these
[00:05:25] snuffleupagus where a lot of these pieces have come out this is the
[00:05:27] pieces have come out this is the essential piece for why bert is able to
[00:05:29] essential piece for why bert is able to have such a small vocabulary only about
[00:05:31] have such a small vocabulary only about 30 000 words compare that with the 400
[00:05:33] 30 000 words compare that with the 400 000 words that are in the glove space
[00:05:36] 000 words that are in the glove space the reason it can get away with that is
[00:05:37] the reason it can get away with that is that it does a lot of breaking apart of
[00:05:39] that it does a lot of breaking apart of words into sub word tokens
[00:05:42] words into sub word tokens and of course because the model is
[00:05:43] and of course because the model is contextual we have an expectation that
[00:05:46] contextual we have an expectation that for example when it encounters code here
[00:05:48] for example when it encounters code here in the context of en at some conceptual
[00:05:51] in the context of en at some conceptual level the mother will recognize that his
[00:05:52] level the mother will recognize that his process the word encode even though it
[00:05:54] process the word encode even though it was two tokens underlyingly
[00:05:59] let's flesh this out a bit by looking at
[00:06:00] let's flesh this out a bit by looking at the full interface for dealing with
[00:06:02] the full interface for dealing with these models i'm again taking advantage
[00:06:03] these models i'm again taking advantage of hugging face i'm going to load a burp
[00:06:05] of hugging face i'm going to load a burp model and a burp tokenizer it's
[00:06:08] model and a burp tokenizer it's important that they use the same
[00:06:09] important that they use the same pre-trained weights which hugging face
[00:06:11] pre-trained weights which hugging face will download for you from the web and
[00:06:13] will download for you from the web and so those are tied in and i set up the
[00:06:14] so those are tied in and i set up the tokenizer and the model if i call
[00:06:16] tokenizer and the model if i call tokenizer.encode on a sequence it would
[00:06:19] tokenizer.encode on a sequence it would give me back a list of indices and those
[00:06:21] give me back a list of indices and those indices will be used as a lookup to
[00:06:23] indices will be used as a lookup to start the process of computing this
[00:06:24] start the process of computing this entire sequence in cell 6 i actually use
[00:06:27] entire sequence in cell 6 i actually use the model to derive that grid of
[00:06:29] the model to derive that grid of representations hugging faces giving us
[00:06:31] representations hugging faces giving us an object that has a lot of attributes
[00:06:33] an object that has a lot of attributes if i call
[00:06:34] if i call output hidden states equals true when i
[00:06:36] output hidden states equals true when i use the model here then i can call dot
[00:06:39] use the model here then i can call dot hidden states and get that full grid of
[00:06:41] hidden states and get that full grid of representations that i showed you before
[00:06:43] representations that i showed you before so this is a sequence with 13 layers
[00:06:45] so this is a sequence with 13 layers that's one embedding layer plus 12 of
[00:06:47] that's one embedding layer plus 12 of the additional layers
[00:06:49] the additional layers and if i key into one of like the first
[00:06:51] and if i key into one of like the first layer that will be the embedding you can
[00:06:53] layer that will be the embedding you can see that its shape is one by five by 768
[00:06:56] see that its shape is one by five by 768 this is a batch of one example it has
[00:06:59] this is a batch of one example it has five tokens the three that we can see
[00:07:01] five tokens the three that we can see here plus the class and set tokens and
[00:07:04] here plus the class and set tokens and each one of those tokens in the
[00:07:06] each one of those tokens in the embedding layer is represented by a
[00:07:08] embedding layer is represented by a vector of dimension 768
[00:07:10] vector of dimension 768 and that remains consistent through all
[00:07:12] and that remains consistent through all the layers in the model so if i went to
[00:07:13] the layers in the model so if i went to the final output states i again just
[00:07:15] the final output states i again just index into dot hidden states here the
[00:07:18] index into dot hidden states here the shape is the same and that will be
[00:07:20] shape is the same and that will be consistent for all the layers
[00:07:22] consistent for all the layers those are the preliminaries and let's
[00:07:24] those are the preliminaries and let's think about how we could derive some
[00:07:25] think about how we could derive some static representations the first
[00:07:27] static representations the first approach that bom asani and all consider
[00:07:30] approach that bom asani and all consider is what they call the decontextualized
[00:07:32] is what they call the decontextualized approach and this is like the simplest
[00:07:34] approach and this is like the simplest thing possible we are just going to
[00:07:36] thing possible we are just going to process individual words as though they
[00:07:38] process individual words as though they were sequences and see if burke can make
[00:07:40] were sequences and see if burke can make any sense of them so we would start by
[00:07:42] any sense of them so we would start by feeding in a word like kitten and we
[00:07:44] feeding in a word like kitten and we would allow the model to break it apart
[00:07:46] would allow the model to break it apart into its sub-word pieces and then we
[00:07:48] into its sub-word pieces and then we simply process that with the model we
[00:07:50] simply process that with the model we get a full grid of representations
[00:07:53] get a full grid of representations now because we potentially have sub-word
[00:07:55] now because we potentially have sub-word tokens here we need some cooling
[00:07:56] tokens here we need some cooling function so what we could do is just
[00:07:58] function so what we could do is just pool using something like mean to get a
[00:08:00] pool using something like mean to get a fixed static representation of dimension
[00:08:03] fixed static representation of dimension 768
[00:08:05] 768 for this individual word
[00:08:07] for this individual word and of course we don't have to use the
[00:08:08] and of course we don't have to use the final layer we could use lower down
[00:08:10] final layer we could use lower down layers and we don't have to use mean as
[00:08:12] layers and we don't have to use mean as the pouring function you could consider
[00:08:14] the pouring function you could consider something like max or min or even last
[00:08:16] something like max or min or even last which would just disregard all of the
[00:08:18] which would just disregard all of the vector representations except for the
[00:08:20] vector representations except for the one corresponding to the final sub word
[00:08:23] one corresponding to the final sub word token
[00:08:25] token this is really simple uh it's
[00:08:27] this is really simple uh it's potentially unnatural though bird is a
[00:08:29] potentially unnatural though bird is a contextual model it was trained on full
[00:08:31] contextual model it was trained on full sequences and especially if we leave off
[00:08:33] sequences and especially if we leave off the class and sep tokens we might be
[00:08:35] the class and sep tokens we might be feeding in sequences that bert has
[00:08:38] feeding in sequences that bert has really never seen before and so it might
[00:08:40] really never seen before and so it might be unknown how it's going to behave with
[00:08:42] be unknown how it's going to behave with these unusual inputs nonetheless though
[00:08:44] these unusual inputs nonetheless though we could repeat this process for all the
[00:08:46] we could repeat this process for all the words in our vocabulary and derive a
[00:08:48] words in our vocabulary and derive a static embedding space and maybe it has
[00:08:50] static embedding space and maybe it has some promise
[00:08:52] some promise however to address this potential
[00:08:54] however to address this potential unnaturalness and potentially take more
[00:08:56] unnaturalness and potentially take more advantage of the the virtues that burt
[00:08:59] advantage of the the virtues that burt and related models have bomasani at all
[00:09:01] and related models have bomasani at all consider also the aggregated approach so
[00:09:04] consider also the aggregated approach so in this approach you process lots of
[00:09:06] in this approach you process lots of corpus examples that contain your target
[00:09:08] corpus examples that contain your target word here i've got a sort of glimpse of
[00:09:10] word here i've got a sort of glimpse of a corpus our target word is kitten of
[00:09:13] a corpus our target word is kitten of course we allow it to be broken apart
[00:09:15] course we allow it to be broken apart into sub-word tokens the full sequences
[00:09:17] into sub-word tokens the full sequences in these examples would also be broken
[00:09:19] in these examples would also be broken apart into sub-word tokens
[00:09:21] apart into sub-word tokens but the important thing is that our
[00:09:22] but the important thing is that our target word might have sub word tokens
[00:09:24] target word might have sub word tokens we pool those as we did before for the
[00:09:26] we pool those as we did before for the contextualized approach and we're also
[00:09:29] contextualized approach and we're also going to pool across all of the
[00:09:30] going to pool across all of the different context examples that we
[00:09:32] different context examples that we processed
[00:09:33] processed and the result of that should be a bunch
[00:09:36] and the result of that should be a bunch of natural inputs to the model but in
[00:09:38] of natural inputs to the model but in the end we derive a static
[00:09:39] the end we derive a static representation that is some kind of
[00:09:41] representation that is some kind of average across all of the examples that
[00:09:43] average across all of the examples that we processed
[00:09:45] we processed it seems very natural it's taking
[00:09:46] it seems very natural it's taking advantage of what what bird is best at i
[00:09:49] advantage of what what bird is best at i will warn you though that this is very
[00:09:50] will warn you though that this is very computationally demanding we're going to
[00:09:52] computationally demanding we're going to want to process lots of examples and
[00:09:54] want to process lots of examples and burt requires lots of resources because
[00:09:56] burt requires lots of resources because it develops really large representations
[00:09:58] it develops really large representations as we've seen
[00:10:00] as we've seen but it might be worth it
[00:10:02] but it might be worth it now bumasonia would offer lots of
[00:10:04] now bumasonia would offer lots of results that help us understand these
[00:10:05] results that help us understand these approaches and how they perform uh let
[00:10:08] approaches and how they perform uh let me give you a glimpse of them as a kind
[00:10:09] me give you a glimpse of them as a kind of summary so what we've got here is
[00:10:11] of summary so what we've got here is results for the simverb 3500 data set a
[00:10:14] results for the simverb 3500 data set a word similarity data set that's very
[00:10:16] word similarity data set that's very similar to the ones that you'll be
[00:10:17] similar to the ones that you'll be working with on the homework and bake
[00:10:19] working with on the homework and bake off
[00:10:20] off our metric is spearmint correlation and
[00:10:22] our metric is spearmint correlation and higher is better that's along the y-axis
[00:10:24] higher is better that's along the y-axis and along the x-axis i have the layer in
[00:10:27] and along the x-axis i have the layer in the model that we're keying into
[00:10:29] the model that we're keying into and then of course what we should watch
[00:10:30] and then of course what we should watch is that we have two pooling functions f
[00:10:32] is that we have two pooling functions f and g f is subword pooling and g is
[00:10:35] and g f is subword pooling and g is context pooling for models that have it
[00:10:37] context pooling for models that have it and it's decomped for the
[00:10:38] and it's decomped for the decontextualized approach
[00:10:41] decontextualized approach now we have a very clear result across
[00:10:43] now we have a very clear result across these results and i think across all the
[00:10:44] these results and i think across all the results in the paper lower layers are
[00:10:47] results in the paper lower layers are better
[00:10:48] better lower layers are giving us good high
[00:10:49] lower layers are giving us good high fidelity representations of individual
[00:10:51] fidelity representations of individual words as we travel higher in the model
[00:10:53] words as we travel higher in the model we seem to lose a lot of that word level
[00:10:55] we seem to lose a lot of that word level discrimination
[00:10:58] discrimination in addition your best choice is to do
[00:11:01] in addition your best choice is to do mean pooling for the context
[00:11:03] mean pooling for the context and subwood pooling seems to matter less
[00:11:05] and subwood pooling seems to matter less right all of these lines here are all
[00:11:07] right all of these lines here are all for the context pooling model with mean
[00:11:10] for the context pooling model with mean as your context pooling function
[00:11:12] as your context pooling function the very best choice though i think
[00:11:14] the very best choice though i think consistently is mean for both of these
[00:11:16] consistently is mean for both of these pooling functions here
[00:11:18] pooling functions here um you can see that in this result and i
[00:11:20] um you can see that in this result and i think that's consistent across all the
[00:11:21] think that's consistent across all the results in the paper but
[00:11:23] results in the paper but the
[00:11:24] the overall takeaway here is that as
[00:11:26] overall takeaway here is that as expected the aggregated approach is
[00:11:28] expected the aggregated approach is better than the decontextualized
[00:11:29] better than the decontextualized approach
[00:11:30] approach however if you don't have the
[00:11:31] however if you don't have the computational budget for that then mean
[00:11:34] computational budget for that then mean pooling in the decontextualized approach
[00:11:36] pooling in the decontextualized approach looks really competitive that's not so
[00:11:38] looks really competitive that's not so evident in this plot but if you look
[00:11:40] evident in this plot but if you look across all the results in the paper i
[00:11:42] across all the results in the paper i think that's a pretty clear finding so
[00:11:44] think that's a pretty clear finding so that would be a good choice and one
[00:11:46] that would be a good choice and one thing is clear that simple approach is
[00:11:48] thing is clear that simple approach is better than some kinds of context
[00:11:50] better than some kinds of context pooling where you choose the wrong
[00:11:52] pooling where you choose the wrong context pooling function like min or max
[00:11:55] context pooling function like min or max despite all of the effort that went into
[00:11:57] despite all of the effort that went into this set of results and also these
[00:11:59] this set of results and also these they're all kind of down here entangled
[00:12:01] they're all kind of down here entangled with the decontextualized approach
[00:12:03] with the decontextualized approach but
[00:12:04] but mean as the pooling function there is
[00:12:06] mean as the pooling function there is really an outstanding choice as you can
[00:12:08] really an outstanding choice as you can see from these results

Homework 2: Sentiment Analysis | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=e5zRhwc-SqI

---

Transcript

[00:00:04] hello everyone this video is an overview
[00:00:06] hello everyone this video is an overview of homework 2 which is on supervised
[00:00:09] of homework 2 which is on supervised sentiment analysis and i would actually
[00:00:10] sentiment analysis and i would actually think of it as an experiment in cross
[00:00:12] think of it as an experiment in cross domain sentiment analysis let's just
[00:00:14] domain sentiment analysis let's just walk through this notebook and i'll try
[00:00:16] walk through this notebook and i'll try to give you a feel for the problem and
[00:00:18] to give you a feel for the problem and our thinking behind it
[00:00:19] our thinking behind it so the plot is the usual one
[00:00:22] so the plot is the usual one we're going to introduce a task and
[00:00:24] we're going to introduce a task and associated data
[00:00:26] associated data and help you with setting up some
[00:00:27] and help you with setting up some baselines and doing error analysis and
[00:00:29] baselines and doing error analysis and that's all a lead into these homework
[00:00:31] that's all a lead into these homework questions which are meant to help you
[00:00:33] questions which are meant to help you explore the data in meaningful ways and
[00:00:35] explore the data in meaningful ways and also set up some additional baselines
[00:00:37] also set up some additional baselines that might inform
[00:00:38] that might inform ultimately your original system which
[00:00:40] ultimately your original system which you then enter into the bank off
[00:00:44] you then enter into the bank off fast overview we're doing ternary that
[00:00:46] fast overview we're doing ternary that is positive negative neutral sentiment
[00:00:47] is positive negative neutral sentiment analysis we're going to be dealing with
[00:00:49] analysis we're going to be dealing with two data sets the stanford sentiment
[00:00:51] two data sets the stanford sentiment tree bank and a brand new assessment
[00:00:54] tree bank and a brand new assessment data set that is a dev test split of
[00:00:56] data set that is a dev test split of sentences drawn from restaurant reviews
[00:00:59] sentences drawn from restaurant reviews we're giving you for training the sst
[00:01:01] we're giving you for training the sst train set and asking you to evaluate on
[00:01:03] train set and asking you to evaluate on the sst dev and test uh and also on this
[00:01:06] the sst dev and test uh and also on this new dev test split of restaurant reviews
[00:01:08] new dev test split of restaurant reviews and that's the cross domain aspect of
[00:01:10] and that's the cross domain aspect of this
[00:01:10] this you are completely unconstrained about
[00:01:12] you are completely unconstrained about what you do in terms of bringing in new
[00:01:14] what you do in terms of bringing in new data for training and doing things in
[00:01:16] data for training and doing things in development the one constraint that we
[00:01:18] development the one constraint that we really need to firmly impose here is
[00:01:20] really need to firmly impose here is that of course the sst3 test set is a
[00:01:23] that of course the sst3 test set is a public test set it's actually included
[00:01:26] public test set it's actually included in your data distribution so that other
[00:01:27] in your data distribution so that other notebooks can run some baseline systems
[00:01:30] notebooks can run some baseline systems and compare against the literature but
[00:01:32] and compare against the literature but that test set is completely off limits
[00:01:34] that test set is completely off limits during development
[00:01:36] during development it's really important that you do all
[00:01:37] it's really important that you do all your development just on the dev splits
[00:01:40] your development just on the dev splits and completely ignore the fact that you
[00:01:42] and completely ignore the fact that you have a labeled version of the ss3 t3
[00:01:44] have a labeled version of the ss3 t3 test set
[00:01:46] test set and as i say here
[00:01:47] and as i say here much of the scientific integrity of our
[00:01:49] much of the scientific integrity of our field depends on people adhering to this
[00:01:51] field depends on people adhering to this honor code that is doing no development
[00:01:54] honor code that is doing no development on what is test data because test data
[00:01:56] on what is test data because test data is our own only chance to get a really
[00:01:58] is our own only chance to get a really clear look at how our systems are
[00:02:00] clear look at how our systems are generalizing to new examples and new
[00:02:02] generalizing to new examples and new experiences so please keep that in mind
[00:02:06] experiences so please keep that in mind the rationale behind this assignment of
[00:02:08] the rationale behind this assignment of course is to help you get familiar or
[00:02:11] course is to help you get familiar or re-familiarize yourself with core
[00:02:12] re-familiarize yourself with core concepts and supervise sentiment
[00:02:14] concepts and supervise sentiment analysis and the associated life cycle
[00:02:17] analysis and the associated life cycle of developing systems in this space
[00:02:18] of developing systems in this space which involves writing feature functions
[00:02:21] which involves writing feature functions trying out model architectures hyper
[00:02:24] trying out model architectures hyper parameter tuning and also possibly doing
[00:02:26] parameter tuning and also possibly doing some comparisons of models using
[00:02:28] some comparisons of models using statistical tests to try to get a sense
[00:02:30] statistical tests to try to get a sense for how much meaningful progress you're
[00:02:31] for how much meaningful progress you're making as you iterate on your system
[00:02:34] making as you iterate on your system design
[00:02:35] design and we're also trying to push here in
[00:02:36] and we're also trying to push here in this notebook that error analysis can be
[00:02:39] this notebook that error analysis can be a powerful way to help you find problems
[00:02:42] a powerful way to help you find problems in your system and then address them
[00:02:45] in your system and then address them one more methodological note as you'll
[00:02:47] one more methodological note as you'll see from this notebook i'm encouraging
[00:02:48] see from this notebook i'm encouraging you to use functionality in this sst dot
[00:02:51] you to use functionality in this sst dot pi module which is part of our course
[00:02:53] pi module which is part of our course code distribution you're not required to
[00:02:56] code distribution you're not required to use it
[00:02:57] use it really the only contract we need to have
[00:02:59] really the only contract we need to have with you is that your original system
[00:03:00] with you is that your original system have a predict 1 method that maps
[00:03:03] have a predict 1 method that maps strings to predictions very directly
[00:03:06] strings to predictions very directly but other than that you're unconstrained
[00:03:07] but other than that you're unconstrained i do want to say though that i think
[00:03:09] i do want to say though that i think sst.experiment is a flexible framework
[00:03:12] sst.experiment is a flexible framework for doing lots of experiments without
[00:03:14] for doing lots of experiments without writing a lot of boilerplate code so it
[00:03:16] writing a lot of boilerplate code so it should if you get used to it be a
[00:03:18] should if you get used to it be a powerful basis for you for doing a lot
[00:03:21] powerful basis for you for doing a lot of experiments which i think is crucial
[00:03:22] of experiments which i think is crucial to success here
[00:03:26] we do some setup by loading a bunch of
[00:03:27] we do some setup by loading a bunch of libraries
[00:03:29] libraries and get a pointer to the data and that
[00:03:31] and get a pointer to the data and that brings us to the training set here so
[00:03:33] brings us to the training set here so this is going to load in a pandas data
[00:03:35] this is going to load in a pandas data frame you can see that we've got about 8
[00:03:37] frame you can see that we've got about 8 500 examples
[00:03:39] 500 examples do review the notebook covering this
[00:03:41] do review the notebook covering this data set here there are a bunch of other
[00:03:42] data set here there are a bunch of other options for this train reader in
[00:03:44] options for this train reader in particular you can decide whether to
[00:03:45] particular you can decide whether to keep or remove duplicates and you can
[00:03:48] keep or remove duplicates and you can also decide whether you want to train on
[00:03:49] also decide whether you want to train on the labeled subtrees that the sst
[00:03:51] the labeled subtrees that the sst contains which vastly increases the
[00:03:54] contains which vastly increases the amount of training data you have which
[00:03:56] amount of training data you have which will be very compute intensive but it
[00:03:58] will be very compute intensive but it could be very productive this is also a
[00:04:01] could be very productive this is also a point to say
[00:04:02] point to say again that you are free to bring in
[00:04:04] again that you are free to bring in other training sets and in fact it might
[00:04:07] other training sets and in fact it might be very productive to bring in the
[00:04:08] be very productive to bring in the dynacent
[00:04:10] dynacent data set which is covered in a
[00:04:12] data set which is covered in a screencast for this unit
[00:04:14] screencast for this unit that data set has a lot of sentences
[00:04:16] that data set has a lot of sentences from restaurant reviews and it was also
[00:04:19] from restaurant reviews and it was also labeled in exactly the same way using
[00:04:21] labeled in exactly the same way using the same protocols as were used for
[00:04:23] the same protocols as were used for creating the development set of
[00:04:24] creating the development set of restaurant reviews for this unit which
[00:04:26] restaurant reviews for this unit which is importantly different i think from
[00:04:28] is importantly different i think from the protocols that were used for the sst
[00:04:32] the protocols that were used for the sst so bringing in more traded training data
[00:04:34] so bringing in more traded training data could help you not only with the cross
[00:04:35] could help you not only with the cross domain problem but also with the kind of
[00:04:37] domain problem but also with the kind of label shift that has probably happened
[00:04:39] label shift that has probably happened between sst and these new development
[00:04:42] between sst and these new development data sets that we're introducing
[00:04:45] and that does bring me to the dev sets
[00:04:47] and that does bring me to the dev sets here so we have sst dev that's also a
[00:04:49] here so we have sst dev that's also a panda's data frame as well as this new
[00:04:51] panda's data frame as well as this new bake off data of restaurant reviews also
[00:04:53] bake off data of restaurant reviews also panda's data frame and here you can see
[00:04:55] panda's data frame and here you can see just three randomly chosen examples
[00:04:57] just three randomly chosen examples example id the text of the sentence
[00:05:00] example id the text of the sentence the label which is either positive
[00:05:01] the label which is either positive negative or neutral and that is subtree
[00:05:04] negative or neutral and that is subtree is always zero because these assessment
[00:05:06] is always zero because these assessment data sets have only full examples no no
[00:05:08] data sets have only full examples no no labeled subtrees the way the ssd train
[00:05:10] labeled subtrees the way the ssd train set does
[00:05:13] set does and we can get a look at the label
[00:05:14] and we can get a look at the label distribution and i'll just mention that
[00:05:16] distribution and i'll just mention that the label distribution for the test set
[00:05:17] the label distribution for the test set is very similar
[00:05:19] is very similar it has one noteworthy property which is
[00:05:20] it has one noteworthy property which is that it's highly skewed a lot of neutral
[00:05:23] that it's highly skewed a lot of neutral examples which i think is realistic for
[00:05:25] examples which i think is realistic for actual data even review data and then
[00:05:27] actual data even review data and then there is a skew toward positivity with
[00:05:29] there is a skew toward positivity with negative the smallest and this kind of
[00:05:30] negative the smallest and this kind of label imbalance i think is severe enough
[00:05:32] label imbalance i think is severe enough that it might impact optimization
[00:05:34] that it might impact optimization choices that you make
[00:05:37] choices that you make this next section here just sets up a
[00:05:38] this next section here just sets up a soft max baseline we use a unigrams
[00:05:41] soft max baseline we use a unigrams feature function this couldn't be
[00:05:42] feature function this couldn't be simpler we're just splitting on white
[00:05:44] simpler we're just splitting on white space and counting the resulting tokens
[00:05:46] space and counting the resulting tokens and then we have this very thin wrapper
[00:05:48] and then we have this very thin wrapper around logistic regression and those are
[00:05:50] around logistic regression and those are the two pieces that come together to run
[00:05:52] the two pieces that come together to run here an sst.experiment
[00:05:55] here an sst.experiment a lot of information about your
[00:05:56] a lot of information about your experiment is stored in this variable
[00:05:58] experiment is stored in this variable and what's being printed out is just a
[00:06:00] and what's being printed out is just a summary classification report we have
[00:06:02] summary classification report we have sst dev and bake off dev as our two
[00:06:05] sst dev and bake off dev as our two assessment data frames the results for
[00:06:07] assessment data frames the results for each one of those are printed separately
[00:06:08] each one of those are printed separately here and then our bake off metric is
[00:06:11] here and then our bake off metric is this mean of the macro average f1 scores
[00:06:14] this mean of the macro average f1 scores across the two data sets exactly these
[00:06:16] across the two data sets exactly these two but of course at the bake off time
[00:06:18] two but of course at the bake off time we'll be using the test sets
[00:06:21] we'll be using the test sets so you might be guided in sort of hill
[00:06:22] so you might be guided in sort of hill climb on this number here while also
[00:06:24] climb on this number here while also attending to these two numbers which are
[00:06:26] attending to these two numbers which are contributing to it so for example you
[00:06:28] contributing to it so for example you can see here that as expected since we
[00:06:31] can see here that as expected since we trained on the sst we're doing better on
[00:06:33] trained on the sst we're doing better on the sst dev by far than we are on the
[00:06:37] the sst dev by far than we are on the new bake off data
[00:06:41] the next section here just shows you
[00:06:42] the next section here just shows you another kind of baseline and this is a
[00:06:44] another kind of baseline and this is a deep learning baseline and rnn
[00:06:46] deep learning baseline and rnn classifier our feature function is very
[00:06:49] classifier our feature function is very simple here because we just split on
[00:06:50] simple here because we just split on white space and we rely on the rnn
[00:06:52] white space and we rely on the rnn itself to do all the featurization which
[00:06:55] itself to do all the featurization which is like an embedding lookup and then
[00:06:56] is like an embedding lookup and then processing the example
[00:06:58] processing the example so that's very simple and then the
[00:06:59] so that's very simple and then the wrapper is also very simple here we're
[00:07:01] wrapper is also very simple here we're going to set the vocabulary for the
[00:07:02] going to set the vocabulary for the model the min count of two that seems
[00:07:05] model the min count of two that seems productive and then finally run the
[00:07:07] productive and then finally run the experiment and the one thing that's
[00:07:08] experiment and the one thing that's important here the one change is that
[00:07:10] important here the one change is that you set vectorize equals false here
[00:07:12] you set vectorize equals false here unlike in the previous baseline we are
[00:07:14] unlike in the previous baseline we are not using scikit-learn dict vectorizers
[00:07:17] not using scikit-learn dict vectorizers to process count dictionaries to get us
[00:07:20] to process count dictionaries to get us from features to feature matrices um
[00:07:22] from features to feature matrices um here we are feeding our examples
[00:07:24] here we are feeding our examples directly through into the model our
[00:07:26] directly through into the model our model expects token streams with no
[00:07:28] model expects token streams with no messing about and so vectorized false
[00:07:30] messing about and so vectorized false will give them a pass through all the
[00:07:32] will give them a pass through all the way to the model so remember that
[00:07:33] way to the model so remember that otherwise this will all fall apart but
[00:07:36] otherwise this will all fall apart but other than that it's exactly the same
[00:07:37] other than that it's exactly the same setup let's run it here i've got some
[00:07:39] setup let's run it here i've got some timing information we're going to fast
[00:07:41] timing information we're going to fast forward through this because this takes
[00:07:42] forward through this because this takes a little bit of time but you'll see a
[00:07:44] a little bit of time but you'll see a report and i'm currently on just a very
[00:07:46] report and i'm currently on just a very old cpu-based mac so
[00:07:49] old cpu-based mac so this will give you a sense for the cost
[00:07:51] this will give you a sense for the cost of development for deep learning in this
[00:07:52] of development for deep learning in this space
[00:08:03] all right our model's early stopping
[00:08:04] all right our model's early stopping criterion was met after 49 epochs
[00:08:08] criterion was met after 49 epochs and here's our look at the results which
[00:08:10] and here's our look at the results which are kind of comparable to what we saw
[00:08:12] are kind of comparable to what we saw with the softmax baseline
[00:08:15] with the softmax baseline all right and that brings us to error
[00:08:16] all right and that brings us to error analysis which can be an important step
[00:08:18] analysis which can be an important step in improving your system i've written a
[00:08:20] in improving your system i've written a few functions that make use of all the
[00:08:22] few functions that make use of all the information that is encoded in the
[00:08:23] information that is encoded in the return values for ssd experiment which i
[00:08:25] return values for ssd experiment which i hope packaged together everything you
[00:08:27] hope packaged together everything you need to do rich error analysis reproduce
[00:08:29] need to do rich error analysis reproduce your results and make use of your model
[00:08:31] your results and make use of your model in downstream experiments
[00:08:33] in downstream experiments here we're going to use it for use this
[00:08:35] here we're going to use it for use this function find errors
[00:08:37] function find errors i've done a little bit of pre-processing
[00:08:39] i've done a little bit of pre-processing of the errors that were found and
[00:08:40] of the errors that were found and package them together and then this cell
[00:08:43] package them together and then this cell here is just an example of the kind of
[00:08:44] here is just an example of the kind of things that you might do
[00:08:46] things that you might do here we're looking at cases where the
[00:08:47] here we're looking at cases where the softmax model was correct the rnn was
[00:08:49] softmax model was correct the rnn was incorrect and the correct label is
[00:08:51] incorrect and the correct label is positive you could of course fiddle with
[00:08:53] positive you could of course fiddle with those parameters here
[00:08:55] those parameters here we've got 168 examples falling into that
[00:08:57] we've got 168 examples falling into that class and then we could look at a sample
[00:08:59] class and then we could look at a sample of the actual text and fall into that
[00:09:00] of the actual text and fall into that group as a way of figuring out how these
[00:09:02] group as a way of figuring out how these models differ and maybe improving one or
[00:09:04] models differ and maybe improving one or both of them
[00:09:06] both of them all right and that brings us to the
[00:09:07] all right and that brings us to the homework questions and again these are
[00:09:09] homework questions and again these are meant to help you explore the data and
[00:09:11] meant to help you explore the data and set up some additional baselines that
[00:09:12] set up some additional baselines that inform original system development we're
[00:09:15] inform original system development we're going to start with one that's data
[00:09:16] going to start with one that's data oriented i've called this token level
[00:09:18] oriented i've called this token level differences what i'm trying to do is
[00:09:20] differences what i'm trying to do is raise to your awareness the fact that
[00:09:22] raise to your awareness the fact that the sst data and the new restaurant
[00:09:25] the sst data and the new restaurant review data are just encoded in
[00:09:27] review data are just encoded in different ways at the level of
[00:09:28] different ways at the level of tokenization this is mainly the result
[00:09:30] tokenization this is mainly the result of the sst being kind of the result of a
[00:09:32] of the sst being kind of the result of a historical process beginning with pang
[00:09:34] historical process beginning with pang in lead 2005
[00:09:36] in lead 2005 and going on through the sst project
[00:09:38] and going on through the sst project itself so there's some funny things
[00:09:40] itself so there's some funny things about it that i think could certainly
[00:09:41] about it that i think could certainly affect any kind of transfer from one
[00:09:44] affect any kind of transfer from one domain to the other and since you are
[00:09:45] domain to the other and since you are training on sst data
[00:09:48] training on sst data it's important to be aware of how it
[00:09:49] it's important to be aware of how it might be idiosyncratic
[00:09:51] might be idiosyncratic so that happens here you write this
[00:09:52] so that happens here you write this function get token counts and as usual
[00:09:54] function get token counts and as usual you have a test you pass the test
[00:09:56] you have a test you pass the test um you're in good shape
[00:10:01] next question relates to the cross
[00:10:03] next question relates to the cross domain nature of our problem training on
[00:10:05] domain nature of our problem training on some of the bake off data in this
[00:10:06] some of the bake off data in this standard paradigm you are training on
[00:10:09] standard paradigm you are training on ssd evaluating on sst and also this new
[00:10:12] ssd evaluating on sst and also this new bake off data set of restaurant review
[00:10:13] bake off data set of restaurant review sentences
[00:10:15] sentences what would happen if you augmented your
[00:10:16] what would happen if you augmented your training set with a little bit of data
[00:10:19] training set with a little bit of data from the development set of restaurant
[00:10:20] from the development set of restaurant review sentences you have might have a
[00:10:22] review sentences you have might have a hunch that's going to improve system
[00:10:24] hunch that's going to improve system performance and this question here
[00:10:26] performance and this question here simply asks you to
[00:10:28] simply asks you to run such an experiment as usual you have
[00:10:30] run such an experiment as usual you have a test i think you will find that this
[00:10:32] a test i think you will find that this is very productive in helping your
[00:10:34] is very productive in helping your system get traction on the new data and
[00:10:36] system get traction on the new data and that should be a clue as to how to do a
[00:10:38] that should be a clue as to how to do a really good job in the bake off with
[00:10:40] really good job in the bake off with your original system
[00:10:42] your original system this next question here is about feature
[00:10:44] this next question here is about feature representation a more powerful vector
[00:10:46] representation a more powerful vector averaging baseline this is a step toward
[00:10:48] averaging baseline this is a step toward deep learning uh it builds on this
[00:10:50] deep learning uh it builds on this section of a notebook here
[00:10:52] section of a notebook here where essentially we average together
[00:10:54] where essentially we average together vector representations of words to
[00:10:56] vector representations of words to represent each example and those are the
[00:10:58] represent each example and those are the input to a simple logistic regression
[00:11:00] input to a simple logistic regression classifier
[00:11:01] classifier so those are nice low dimensional models
[00:11:03] so those are nice low dimensional models that tend to be quite powerful this
[00:11:05] that tend to be quite powerful this question is asking you to replace the
[00:11:07] question is asking you to replace the logistic regression with a shallow
[00:11:08] logistic regression with a shallow neural classifier so maybe the more
[00:11:10] neural classifier so maybe the more powerful part here and also to explore a
[00:11:13] powerful part here and also to explore a wide range of hyper parameters to that
[00:11:14] wide range of hyper parameters to that model to get a sense for which settings
[00:11:17] model to get a sense for which settings are best for our problem
[00:11:20] are best for our problem and that brings us to burton coding and
[00:11:22] and that brings us to burton coding and this is like one step further down the
[00:11:24] this is like one step further down the line toward deep learning and
[00:11:25] line toward deep learning and fine-tuning this question is simply
[00:11:27] fine-tuning this question is simply asking you to encode your examples using
[00:11:29] asking you to encode your examples using vert in particular taking the summary
[00:11:32] vert in particular taking the summary representation above the class token the
[00:11:34] representation above the class token the final output there as your summary
[00:11:36] final output there as your summary representation of the entire example and
[00:11:38] representation of the entire example and those become presumably the inputs to
[00:11:40] those become presumably the inputs to some downstream classifier or
[00:11:42] some downstream classifier or potentially a fine tuning process
[00:11:44] potentially a fine tuning process the idea that this is like one step
[00:11:46] the idea that this is like one step better than the vector averaging that we
[00:11:48] better than the vector averaging that we just looked at you do not need to
[00:11:50] just looked at you do not need to conduct an experiment with sst you're
[00:11:52] conduct an experiment with sst you're simply implementing this feature
[00:11:54] simply implementing this feature function here
[00:11:55] function here but since ssd experiment does make it
[00:11:57] but since ssd experiment does make it really easy to run experiments once
[00:11:59] really easy to run experiments once you've implemented the feature function
[00:12:01] you've implemented the feature function i would encourage you to choose some
[00:12:02] i would encourage you to choose some classifier model and see how well this
[00:12:05] classifier model and see how well this does but as usual you have a test and
[00:12:07] does but as usual you have a test and the test is just about the feature
[00:12:08] the test is just about the feature function and it will make sure
[00:12:10] function and it will make sure you're using all these values correctly
[00:12:12] you're using all these values correctly and that brings us to the original
[00:12:14] and that brings us to the original system and i just want to remind you
[00:12:15] system and i just want to remind you that you are unconstrained except for
[00:12:17] that you are unconstrained except for the fact that you cannot make any use of
[00:12:19] the fact that you cannot make any use of the sst test set during development the
[00:12:22] the sst test set during development the labels for that are off limits but
[00:12:24] labels for that are off limits but everything else is fair game bring in
[00:12:25] everything else is fair game bring in new training data try new model
[00:12:27] new training data try new model architectures and so forth and so on
[00:12:30] architectures and so forth and so on um we've given a few idea ideas here but
[00:12:32] um we've given a few idea ideas here but this is by no means meant to be
[00:12:34] this is by no means meant to be restrictive it's just meant to get the
[00:12:36] restrictive it's just meant to get the creative juices flowing other than that
[00:12:38] creative juices flowing other than that this is the same procedure as homework
[00:12:40] this is the same procedure as homework one we want a description of your system
[00:12:42] one we want a description of your system to inform the teaching team about what
[00:12:44] to inform the teaching team about what worked and what didn't and it would be
[00:12:46] worked and what didn't and it would be great if you reported your peak score
[00:12:48] great if you reported your peak score which is the macro average of the two f1
[00:12:51] which is the macro average of the two f1 macros f1 scores for our two data sets
[00:12:54] macros f1 scores for our two data sets that's on the development set there
[00:12:56] that's on the development set there and that brings us to the bake off and
[00:12:58] and that brings us to the bake off and again the bake off procedure is familiar
[00:13:00] again the bake off procedure is familiar the one piece here the crucial piece is
[00:13:03] the one piece here the crucial piece is that you write up a function a predict1
[00:13:05] that you write up a function a predict1 function that maps a text directly to a
[00:13:08] function that maps a text directly to a prediction using your original system
[00:13:11] prediction using your original system now given two examples here
[00:13:13] now given two examples here yours might be simpler depending on
[00:13:15] yours might be simpler depending on whether or not you use the ssd
[00:13:17] whether or not you use the ssd experiment framework or not
[00:13:19] experiment framework or not uh but that all comes together here with
[00:13:21] uh but that all comes together here with create bake off submission where you
[00:13:22] create bake off submission where you input that function
[00:13:24] input that function you won't need to change this output
[00:13:25] you won't need to change this output file and you can see that this function
[00:13:27] file and you can see that this function here loads in our two test sets which
[00:13:28] here loads in our two test sets which are unlabeled
[00:13:30] are unlabeled and uses your predict one function on
[00:13:32] and uses your predict one function on all of those examples here
[00:13:34] all of those examples here and then writes a file which you then
[00:13:36] and then writes a file which you then upload to the auto grader to grade scope
[00:13:39] upload to the auto grader to grade scope that happens here so i just would want
[00:13:41] that happens here so i just would want to reiterate that
[00:13:42] to reiterate that in all senses the test data labels are
[00:13:45] in all senses the test data labels are completely off limits to us all the
[00:13:47] completely off limits to us all the development conceptually and otherwise
[00:13:49] development conceptually and otherwise should happen on the development data

Sentiment Analysis | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=sRw3Dtjhlk0

---

Transcript

[00:00:05] hello everyone this video kicks off our
[00:00:08] hello everyone this video kicks off our series of screencasts on supervised
[00:00:10] series of screencasts on supervised sentiment analysis i just want to
[00:00:11] sentiment analysis i just want to provide you with an overview of the
[00:00:13] provide you with an overview of the problem and of the kind of work we'll be
[00:00:15] problem and of the kind of work we'll be doing and also a rationale for why we'll
[00:00:16] doing and also a rationale for why we'll be doing it
[00:00:19] be doing it so here's an overview of the entire unit
[00:00:22] so here's an overview of the entire unit i want to in this screencast motivate
[00:00:24] i want to in this screencast motivate for you the idea that sentiment analysis
[00:00:26] for you the idea that sentiment analysis is a deep problem and an important
[00:00:27] is a deep problem and an important problem for nlu not only scientifically
[00:00:29] problem for nlu not only scientifically but also for industrial applications
[00:00:32] but also for industrial applications uh in the next screencast i'll give you
[00:00:34] uh in the next screencast i'll give you some practical general tips for doing
[00:00:36] some practical general tips for doing sentiment analysis
[00:00:38] sentiment analysis following that we'll have two short
[00:00:39] following that we'll have two short screencasts that introduce our core data
[00:00:41] screencasts that introduce our core data sets the stanford sentiment tree bank
[00:00:43] sets the stanford sentiment tree bank and a new data set called dynacent
[00:00:46] and a new data set called dynacent after that i'll introduce the code base
[00:00:48] after that i'll introduce the code base that we'll be working with on the
[00:00:49] that we'll be working with on the assignment and the bake off that's
[00:00:51] assignment and the bake off that's sst.pi which is included in the course
[00:00:54] sst.pi which is included in the course code distribution and i'm going to use
[00:00:56] code distribution and i'm going to use that code to illustrate some important
[00:00:58] that code to illustrate some important methodological issues surrounding
[00:01:00] methodological issues surrounding supervised analysis in general which
[00:01:02] supervised analysis in general which would be hyperparameter tuning and
[00:01:04] would be hyperparameter tuning and comparison of different classifiers to
[00:01:06] comparison of different classifiers to see whether they are different in some
[00:01:09] see whether they are different in some significant statistical sense then we'll
[00:01:12] significant statistical sense then we'll talk about feature representation both
[00:01:14] talk about feature representation both for large sparse linear models with hand
[00:01:17] for large sparse linear models with hand built features and also with more deep
[00:01:19] built features and also with more deep learning oriented distributional
[00:01:21] learning oriented distributional representations and that will be a nice
[00:01:23] representations and that will be a nice segue into the final unit which is on
[00:01:25] segue into the final unit which is on using
[00:01:26] using recurrent neural networks as classifiers
[00:01:29] recurrent neural networks as classifiers for supervised sentiment analysis my
[00:01:31] for supervised sentiment analysis my hope is that this unit can provide a
[00:01:33] hope is that this unit can provide a refresher on core concepts in supervised
[00:01:36] refresher on core concepts in supervised learning
[00:01:37] learning introduce you to the problem of
[00:01:38] introduce you to the problem of sentiment analysis which i think is as i
[00:01:40] sentiment analysis which i think is as i said a central problem for natural
[00:01:42] said a central problem for natural language understanding and also sets you
[00:01:44] language understanding and also sets you on your on your way toward
[00:01:46] on your on your way toward doing the assignment and the bake off
[00:01:49] doing the assignment and the bake off and possibly building projects in this
[00:01:50] and possibly building projects in this space
[00:01:52] space for the associated materials as i said
[00:01:54] for the associated materials as i said we've got a bunch of code sst.pi is the
[00:01:56] we've got a bunch of code sst.pi is the core module and then we have a notebook
[00:01:59] core module and then we have a notebook introducing the stanford sentiment
[00:02:01] introducing the stanford sentiment treebank as a dataset
[00:02:03] treebank as a dataset we have a second notebook that's on what
[00:02:04] we have a second notebook that's on what i've called hand-built features and
[00:02:06] i've called hand-built features and mostly linear models
[00:02:08] mostly linear models and then a third notebook that's on
[00:02:10] and then a third notebook that's on using neural networks which more or less
[00:02:12] using neural networks which more or less pushes you to using distributional
[00:02:14] pushes you to using distributional representations instead of hand-built
[00:02:16] representations instead of hand-built features
[00:02:17] features although as you'll see the notebooks
[00:02:18] although as you'll see the notebooks explore various combinations of these
[00:02:20] explore various combinations of these ideas
[00:02:22] ideas the homework in the bake off is in the
[00:02:23] the homework in the bake off is in the notebook homework sentiment
[00:02:26] notebook homework sentiment and i'm going to introduce that problem
[00:02:28] and i'm going to introduce that problem in a separate screencast
[00:02:30] in a separate screencast the core readings are the two papers
[00:02:32] the core readings are the two papers that are oriented around our data sets
[00:02:34] that are oriented around our data sets the stanford sentiment treebank and
[00:02:35] the stanford sentiment treebank and dynacent and as supplementary readings
[00:02:38] dynacent and as supplementary readings you might enjoy this compendium from
[00:02:40] you might enjoy this compendium from pang and lee it's a kind of overview of
[00:02:41] pang and lee it's a kind of overview of the whole field of sentiment analysis
[00:02:43] the whole field of sentiment analysis and it poses challenges and questions
[00:02:46] and it poses challenges and questions that are still relevant to this day and
[00:02:48] that are still relevant to this day and then goldberg 2015 is an excellent
[00:02:50] then goldberg 2015 is an excellent overview of using neural networks in nlp
[00:02:53] overview of using neural networks in nlp very generally um but with lots of
[00:02:56] very generally um but with lots of helpful notation and so forth that we're
[00:02:57] helpful notation and so forth that we're aligned with and that might help you get
[00:02:59] aligned with and that might help you get a feel for the landscape of modeling
[00:03:01] a feel for the landscape of modeling choices that you might make in this
[00:03:03] choices that you might make in this space and in subsequent units for this
[00:03:05] space and in subsequent units for this course
[00:03:07] course so i want to start by just motivating
[00:03:08] so i want to start by just motivating the idea that sentiment analysis is an
[00:03:10] the idea that sentiment analysis is an interesting problem because you often
[00:03:12] interesting problem because you often hear people say things like sentiment
[00:03:13] hear people say things like sentiment analysis is solved or it's overly
[00:03:16] analysis is solved or it's overly simplistic or just too easy and i think
[00:03:18] simplistic or just too easy and i think none of those things are true and to
[00:03:20] none of those things are true and to motivate that i just want to do a little
[00:03:22] motivate that i just want to do a little data-driven exercise with you so for
[00:03:24] data-driven exercise with you so for these examples you should ask yourself
[00:03:26] these examples you should ask yourself which of these sentences expresses
[00:03:28] which of these sentences expresses sentiment at all
[00:03:30] sentiment at all and for the ones that you think do
[00:03:31] and for the ones that you think do express sentiment what is that sentiment
[00:03:33] express sentiment what is that sentiment is it positive or negative or maybe
[00:03:35] is it positive or negative or maybe neutral or something else
[00:03:37] neutral or something else so you might think those are
[00:03:38] so you might think those are straightforward questions but this is
[00:03:39] straightforward questions but this is going to get difficult really fast
[00:03:41] going to get difficult really fast consider the first example there was an
[00:03:43] consider the first example there was an earthquake in california this is
[00:03:45] earthquake in california this is probably going to sound like bad news to
[00:03:47] probably going to sound like bad news to you
[00:03:48] you and many sentiment analysis systems will
[00:03:51] and many sentiment analysis systems will assign this negative sentiment but we
[00:03:53] assign this negative sentiment but we should ask ourselves is this actually a
[00:03:55] should ask ourselves is this actually a sentiment latent sentence it is on the
[00:03:58] sentiment latent sentence it is on the face of it merely stating a fact and we
[00:04:01] face of it merely stating a fact and we might hold that for sentiment to be
[00:04:03] might hold that for sentiment to be expressed we need some kind of
[00:04:04] expressed we need some kind of subjective evaluative perspective to be
[00:04:07] subjective evaluative perspective to be included in here like it was bad that
[00:04:09] included in here like it was bad that there was an earthquake in california
[00:04:11] there was an earthquake in california and absent the it was bad clause this
[00:04:13] and absent the it was bad clause this might just be a neutral statement of
[00:04:15] might just be a neutral statement of something that had happened
[00:04:17] something that had happened but the important point here is that
[00:04:18] but the important point here is that unless we settle these questions we'll
[00:04:20] unless we settle these questions we'll have continued indeterminacy about what
[00:04:22] have continued indeterminacy about what we're actually doing
[00:04:24] we're actually doing the team failed to complete the
[00:04:25] the team failed to complete the challenge
[00:04:26] challenge is that positive or negative we might
[00:04:28] is that positive or negative we might agree that it's more than just a
[00:04:30] agree that it's more than just a statement of fact although it's a
[00:04:32] statement of fact although it's a borderline case even for that question
[00:04:34] borderline case even for that question but if we did decide it was sentiment
[00:04:36] but if we did decide it was sentiment laden we would need to figure out the
[00:04:37] laden we would need to figure out the perspective of the speaker is the
[00:04:39] perspective of the speaker is the speaker advocating for this team we're
[00:04:41] speaker advocating for this team we're advocating for a different team
[00:04:43] advocating for a different team right
[00:04:44] right we win we lose it's really going to
[00:04:46] we win we lose it's really going to depend uh on how the speaker is involved
[00:04:49] depend uh on how the speaker is involved and that of course is going to have to
[00:04:50] and that of course is going to have to become part of our definition of what
[00:04:52] become part of our definition of what we're doing when we assign sentiment
[00:04:54] we're doing when we assign sentiment labels they said it would be great on
[00:04:57] labels they said it would be great on the face of it this expresses no speaker
[00:05:00] the face of it this expresses no speaker perspective at all this is merely
[00:05:01] perspective at all this is merely reporting what somebody else said and we
[00:05:03] reporting what somebody else said and we need to decide for those obviously
[00:05:05] need to decide for those obviously different perspectives what we're going
[00:05:07] different perspectives what we're going to do in terms of sentiment analysis
[00:05:09] to do in terms of sentiment analysis because after all this could continue
[00:05:11] because after all this could continue they said it would be great and they
[00:05:12] they said it would be great and they were right which is straightforwardly
[00:05:14] were right which is straightforwardly positive but it could also continue they
[00:05:16] positive but it could also continue they said it would be great and they were
[00:05:17] said it would be great and they were wrong and i think that reveals that
[00:05:19] wrong and i think that reveals that sentence three is not so obviously
[00:05:21] sentence three is not so obviously encoding a particular speaker
[00:05:23] encoding a particular speaker perspective whereas these clauses are
[00:05:25] perspective whereas these clauses are what really tell the story for us as
[00:05:27] what really tell the story for us as sentiment analysts
[00:05:29] sentiment analysts and then we get into things that you
[00:05:31] and then we get into things that you might call non-literal use of language
[00:05:33] might call non-literal use of language the party fat cats are sipping their
[00:05:35] the party fat cats are sipping their expensive imported wines this has a lot
[00:05:38] expensive imported wines this has a lot of positive language in it maybe only
[00:05:40] of positive language in it maybe only fat cats is the thing that sounds like a
[00:05:42] fat cats is the thing that sounds like a direct sneer
[00:05:43] direct sneer but i think we could agree that overall
[00:05:45] but i think we could agree that overall this is probably negative in its valence
[00:05:48] this is probably negative in its valence and that will be a challenge for our
[00:05:50] and that will be a challenge for our systems and also a challenge for us in
[00:05:52] systems and also a challenge for us in just characterizing precisely what was
[00:05:54] just characterizing precisely what was done here in terms of sentiment
[00:05:56] done here in terms of sentiment here's a similar example oh you're
[00:05:57] here's a similar example oh you're terrible this might be a criticism and
[00:06:00] terrible this might be a criticism and it might therefore be straightforwardly
[00:06:02] it might therefore be straightforwardly negative on the other hand it could be a
[00:06:04] negative on the other hand it could be a kind of teasing form of social bonding
[00:06:06] kind of teasing form of social bonding that overall has a positive effect on
[00:06:08] that overall has a positive effect on the discourse how are we going to
[00:06:09] the discourse how are we going to resolve that kind of context dependence
[00:06:12] resolve that kind of context dependence here's another one here's julia bastard
[00:06:14] here's another one here's julia bastard it's got some negative language even
[00:06:16] it's got some negative language even something that's kind of like a swear
[00:06:17] something that's kind of like a swear but this could be a friendly jocular
[00:06:20] but this could be a friendly jocular phrase of some kind um and we'll have to
[00:06:23] phrase of some kind um and we'll have to sort out whether it's friendly and fun
[00:06:25] sort out whether it's friendly and fun because of its negativity
[00:06:27] because of its negativity or whether this is straightforwardly
[00:06:29] or whether this is straightforwardly just a positive sentence
[00:06:32] just a positive sentence and then here's a case that's just going
[00:06:33] and then here's a case that's just going to be a challenge for our systems this
[00:06:35] to be a challenge for our systems this is of the movie 2001 this is from an
[00:06:37] is of the movie 2001 this is from an actual review many consider the
[00:06:39] actual review many consider the masterpiece bewildering boring slow
[00:06:41] masterpiece bewildering boring slow moving or annoying there is a not a lot
[00:06:43] moving or annoying there is a not a lot of negative language there in fact
[00:06:45] of negative language there in fact there's very little that's positive
[00:06:46] there's very little that's positive except masterpiece but i think we can
[00:06:49] except masterpiece but i think we can all anticipate that overall this is
[00:06:51] all anticipate that overall this is probably going to be a positive review
[00:06:53] probably going to be a positive review of that movie
[00:06:55] of that movie so that just shows you that even if
[00:06:56] so that just shows you that even if we're clear about what we're doing in
[00:06:57] we're clear about what we're doing in terms of sentiment the linguistic
[00:06:59] terms of sentiment the linguistic challenge here is
[00:07:00] challenge here is significant and we could also extend
[00:07:02] significant and we could also extend that to sentiment like long-suffering
[00:07:05] that to sentiment like long-suffering fans bittersweet memories hilariously
[00:07:07] fans bittersweet memories hilariously embarrassing moments these are things
[00:07:09] embarrassing moments these are things that are going to blend positivity and
[00:07:10] that are going to blend positivity and negativity and all sorts of other
[00:07:12] negativity and all sorts of other emotional dimensions in ways that just
[00:07:14] emotional dimensions in ways that just make sentiment analysis very difficult
[00:07:16] make sentiment analysis very difficult to do reliably
[00:07:18] to do reliably and that's a nice segue into this topic
[00:07:20] and that's a nice segue into this topic of sentiment analysis in industry
[00:07:22] of sentiment analysis in industry because of course
[00:07:23] because of course sentiment analysis is one of the first
[00:07:25] sentiment analysis is one of the first tasks that was really transformed by
[00:07:26] tasks that was really transformed by data-driven approaches and it was the
[00:07:28] data-driven approaches and it was the first task to really make an impact in
[00:07:31] first task to really make an impact in industry uh there are lots of startups
[00:07:34] industry uh there are lots of startups and companies that offer sentiment
[00:07:35] and companies that offer sentiment analysis tools and it has obvious import
[00:07:38] analysis tools and it has obvious import for things like marketing and customer
[00:07:39] for things like marketing and customer experience and so forth
[00:07:41] experience and so forth and i would say the first thing i would
[00:07:42] and i would say the first thing i would say is that to this day the sentiment
[00:07:44] say is that to this day the sentiment from industry so to speak is that
[00:07:46] from industry so to speak is that sentiment analysis tools still fall
[00:07:48] sentiment analysis tools still fall short this is from an article from 2013
[00:07:51] short this is from an article from 2013 and the
[00:07:52] and the gist of it is anyone who says they're
[00:07:53] gist of it is anyone who says they're getting better than 70 today is lying
[00:07:55] getting better than 70 today is lying generally speaking for whatever notion
[00:07:58] generally speaking for whatever notion of 70 we have here i think we can agree
[00:08:00] of 70 we have here i think we can agree that that's too low and that we as a
[00:08:02] that that's too low and that we as a field ought to be offering tools that
[00:08:03] field ought to be offering tools that are better
[00:08:04] are better this is another kind of equivocal
[00:08:06] this is another kind of equivocal headline the motion ai technology has
[00:08:08] headline the motion ai technology has has great promise when used responsibly
[00:08:11] has great promise when used responsibly affective computing knows how you feel
[00:08:13] affective computing knows how you feel sorta the sorta is kind of like the
[00:08:15] sorta the sorta is kind of like the equivalent of 70 percent here and i
[00:08:17] equivalent of 70 percent here and i think it shows that there's a lot of
[00:08:18] think it shows that there's a lot of work to be done if we're going to have
[00:08:20] work to be done if we're going to have the kind of impact we want to have in
[00:08:22] the kind of impact we want to have in the technological sphere
[00:08:25] the technological sphere and there's another dimension to this
[00:08:26] and there's another dimension to this which we're not going to really get to
[00:08:27] which we're not going to really get to capture but is worth planting in your
[00:08:29] capture but is worth planting in your minds because this could become projects
[00:08:31] minds because this could become projects right we're going to do classification
[00:08:33] right we're going to do classification of sentiment into positive negative and
[00:08:35] of sentiment into positive negative and neutral
[00:08:37] neutral and that's often the starting point for
[00:08:38] and that's often the starting point for these industry tools many business
[00:08:40] these industry tools many business leaders think they want these pie charts
[00:08:42] leaders think they want these pie charts that point out like 30 negative 70
[00:08:44] that point out like 30 negative 70 positive and then in q2 the negativity
[00:08:47] positive and then in q2 the negativity is slightly up
[00:08:48] is slightly up and that's truly a leading indicator of
[00:08:50] and that's truly a leading indicator of something it looks like negativity is on
[00:08:52] something it looks like negativity is on the rise but the issue is what do you do
[00:08:55] the rise but the issue is what do you do how does this help with decision making
[00:08:57] how does this help with decision making merely classifying these texts and
[00:08:59] merely classifying these texts and showing change over time is not enough
[00:09:01] showing change over time is not enough for any business leader to take action
[00:09:03] for any business leader to take action what we need to know are
[00:09:05] what we need to know are why this is happening what the
[00:09:06] why this is happening what the underlying factors are basically what
[00:09:08] underlying factors are basically what are the customers saying beyond these
[00:09:10] are the customers saying beyond these gross classifications into positive
[00:09:12] gross classifications into positive negative and neutral
[00:09:14] negative and neutral and we should be pushing ourselves to
[00:09:16] and we should be pushing ourselves to design tools that can offer that next
[00:09:18] design tools that can offer that next layer of insight
[00:09:21] affective computing this is a kind of
[00:09:23] affective computing this is a kind of transition into the wider world here
[00:09:24] transition into the wider world here we're going to focus on just sentiment
[00:09:26] we're going to focus on just sentiment analysis but you could think about
[00:09:27] analysis but you could think about emotional analysis and all other kinds
[00:09:30] emotional analysis and all other kinds of kind of context dependent expression
[00:09:32] of kind of context dependent expression in language put that under the heading
[00:09:34] in language put that under the heading of affective computing this is a diagram
[00:09:36] of affective computing this is a diagram from a paper i did a few years ago with
[00:09:38] from a paper i did a few years ago with more at south uh it's a diagram of
[00:09:42] more at south uh it's a diagram of emotions and other kinds of moods that
[00:09:44] emotions and other kinds of moods that people feel
[00:09:45] people feel the arcs give you a transition so they
[00:09:47] the arcs give you a transition so they show that people tend to transition
[00:09:48] show that people tend to transition systematically from one emotional state
[00:09:51] systematically from one emotional state to another
[00:09:52] to another so what we're seeing here is basically
[00:09:54] so what we're seeing here is basically just that this is a very high
[00:09:55] just that this is a very high dimensional space it's not just positive
[00:09:57] dimensional space it's not just positive negative neutral we have a wide range of
[00:10:00] negative neutral we have a wide range of feelings and moods and emotions and
[00:10:02] feelings and moods and emotions and states that we go into and there's a lot
[00:10:04] states that we go into and there's a lot of structure to how we experience those
[00:10:06] of structure to how we experience those moods in our lives and it would be great
[00:10:08] moods in our lives and it would be great to break out of the simple
[00:10:10] to break out of the simple positive negative mode and tackle all of
[00:10:13] positive negative mode and tackle all of these dimensions
[00:10:15] these dimensions and in that spirit here what i've done
[00:10:17] and in that spirit here what i've done on this slide is just list out a whole
[00:10:18] on this slide is just list out a whole bunch of other tasks that you might
[00:10:20] bunch of other tasks that you might consider adjacent to sentiment analysis
[00:10:23] consider adjacent to sentiment analysis but they are meaningfully different from
[00:10:24] but they are meaningfully different from sentiment analysis things like
[00:10:26] sentiment analysis things like subjectivity bias stance taking
[00:10:29] subjectivity bias stance taking hate speech microaggressions
[00:10:31] hate speech microaggressions condescension sarcasm deception and
[00:10:34] condescension sarcasm deception and betrayal online trolls polarization
[00:10:36] betrayal online trolls polarization politeness and linguistic alignment
[00:10:38] politeness and linguistic alignment these are all deeply social things that
[00:10:40] these are all deeply social things that are influenced by and shape our language
[00:10:43] are influenced by and shape our language and i've selected these papers in
[00:10:45] and i've selected these papers in particular because all of them have
[00:10:48] particular because all of them have really nice either crisp statements of
[00:10:50] really nice either crisp statements of the problem and or really great public
[00:10:53] the problem and or really great public data sets that you could use for
[00:10:55] data sets that you could use for experiments in this wide world i think
[00:10:56] experiments in this wide world i think that's a very exciting space to explore
[00:10:59] that's a very exciting space to explore as a kind of next step from what we're
[00:11:00] as a kind of next step from what we're doing in this unit
[00:11:02] doing in this unit but back down to earth here our primary
[00:11:04] but back down to earth here our primary data sets as i said are going to be the
[00:11:06] data sets as i said are going to be the ternary formulation of the stanford
[00:11:08] ternary formulation of the stanford sentiment tree bank which is just
[00:11:09] sentiment tree bank which is just positive negative neutral
[00:11:11] positive negative neutral and also the dynacent data set which had
[00:11:13] and also the dynacent data set which had which has that same ternary formulation
[00:11:17] which has that same ternary formulation the sst is movie reviews dynacent is
[00:11:20] the sst is movie reviews dynacent is mostly reviews of products and services
[00:11:22] mostly reviews of products and services i think heavily biased toward
[00:11:23] i think heavily biased toward restaurants because the underlying data
[00:11:25] restaurants because the underlying data is from
[00:11:26] is from yelp and then for the bake off we're
[00:11:29] yelp and then for the bake off we're gonna have a new dev test split uh we'll
[00:11:31] gonna have a new dev test split uh we'll use sst3 as well as this new one of a
[00:11:34] use sst3 as well as this new one of a corpus of sentences from restaurant
[00:11:36] corpus of sentences from restaurant reviews you can see that dynasty might
[00:11:38] reviews you can see that dynasty might be an asset here
[00:11:39] be an asset here they all have this ternary formulation
[00:11:42] they all have this ternary formulation and i'm hoping that the combination of
[00:11:43] and i'm hoping that the combination of these data sets gives us a really
[00:11:45] these data sets gives us a really interesting perspective not only on
[00:11:46] interesting perspective not only on sentiment analysis but also on kind of
[00:11:48] sentiment analysis but also on kind of how we design systems that effectively
[00:11:50] how we design systems that effectively transfer across domains and maybe learn
[00:11:52] transfer across domains and maybe learn simultaneously in multiple domains

General Practical Tips | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=qt-TU_f0HDw

---

Transcript

[00:00:05] hello everyone welcome to part two in
[00:00:07] hello everyone welcome to part two in our series on supervised sentiment
[00:00:08] our series on supervised sentiment analysis this screencast is going to
[00:00:10] analysis this screencast is going to focus on some general practical tips for
[00:00:12] focus on some general practical tips for doing work in this space especially
[00:00:14] doing work in this space especially focused on
[00:00:15] focused on setting up a project and doing kind of
[00:00:17] setting up a project and doing kind of pre-processing of your data
[00:00:19] pre-processing of your data so first i just wanted to give you links
[00:00:21] so first i just wanted to give you links to a whole bunch of benchmark data sets
[00:00:23] to a whole bunch of benchmark data sets in this space we're going to concentrate
[00:00:25] in this space we're going to concentrate on the sst and dynacent but there are a
[00:00:28] on the sst and dynacent but there are a lot of other choices you could make both
[00:00:30] lot of other choices you could make both for developing original systems and also
[00:00:32] for developing original systems and also supplementing training data that you've
[00:00:34] supplementing training data that you've got for a particular application some of
[00:00:36] got for a particular application some of these data sets are really really large
[00:00:38] these data sets are really really large and they cover a diversity of domains so
[00:00:40] and they cover a diversity of domains so these could be important assets for you
[00:00:42] these could be important assets for you in a similar spirit there are lots of
[00:00:44] in a similar spirit there are lots of sentiment lexicons out there they cover
[00:00:46] sentiment lexicons out there they cover different emotional dimensions and
[00:00:48] different emotional dimensions and different aspects of the problem and
[00:00:50] different aspects of the problem and they too could be used to help you with
[00:00:51] they too could be used to help you with powerful featurization they could
[00:00:53] powerful featurization they could supplement features that you've created
[00:00:55] supplement features that you've created or help you group your vocabulary into
[00:00:57] or help you group your vocabulary into interesting subcategories that would be
[00:00:59] interesting subcategories that would be powerful for making sentiment
[00:01:01] powerful for making sentiment predictions and these range from simple
[00:01:03] predictions and these range from simple wordless up to highly structured
[00:01:05] wordless up to highly structured multi-dimensional lexicons
[00:01:08] now for our first pre-processing step i
[00:01:10] now for our first pre-processing step i thought we would just talk a little bit
[00:01:11] thought we would just talk a little bit about tokenization because i think that
[00:01:13] about tokenization because i think that this can be a definitional choice that
[00:01:15] this can be a definitional choice that really affects downstream success so
[00:01:18] really affects downstream success so just as a running example here let's
[00:01:19] just as a running example here let's imagine we start with this raw text
[00:01:21] imagine we start with this raw text which is a kind of imagined tweet
[00:01:23] which is a kind of imagined tweet we have an at mention here and then you
[00:01:25] we have an at mention here and then you can see that some of the markup has
[00:01:27] can see that some of the markup has gotten a little bit garbled we have an
[00:01:29] gotten a little bit garbled we have an emoticon that looks sort of obscured in
[00:01:31] emoticon that looks sort of obscured in a link at the end
[00:01:33] a link at the end i think as a very preliminary step even
[00:01:35] i think as a very preliminary step even before we tokenize we might want to
[00:01:37] before we tokenize we might want to isolate some of that markup and replace
[00:01:39] isolate some of that markup and replace the html entities it's a pretty easy
[00:01:41] the html entities it's a pretty easy thing that you can do that could really
[00:01:42] thing that you can do that could really make a difference now we've got our
[00:01:44] make a difference now we've got our apostrophe we've got our emoticon intact
[00:01:47] apostrophe we've got our emoticon intact and then we still have the link and
[00:01:49] and then we still have the link and other things in here
[00:01:50] other things in here so even before you do that you might
[00:01:51] so even before you do that you might check to see whether a simple
[00:01:53] check to see whether a simple replacement of the html entities would
[00:01:55] replacement of the html entities would make a difference in your data
[00:01:58] make a difference in your data now we begin the tokenization question
[00:02:00] now we begin the tokenization question and i think a good baseline choice here
[00:02:01] and i think a good baseline choice here would be simply white space tokenizing
[00:02:03] would be simply white space tokenizing we're going to split on white space and
[00:02:05] we're going to split on white space and treat all the resulting strings as
[00:02:06] treat all the resulting strings as tokens so that would take our raw text
[00:02:09] tokens so that would take our raw text up here and split it up as as you can
[00:02:10] up here and split it up as as you can see on these independent lines
[00:02:12] see on these independent lines this looks okay to me so we're going to
[00:02:14] this looks okay to me so we're going to have a problem with our at mention
[00:02:16] have a problem with our at mention because it has this colon on the end so
[00:02:18] because it has this colon on the end so we might miss the fact that this is the
[00:02:20] we might miss the fact that this is the actual at mention the unigrams look okay
[00:02:23] actual at mention the unigrams look okay although the date has been split apart
[00:02:25] although the date has been split apart we've preserved our hashtag we've got
[00:02:28] we've preserved our hashtag we've got this token that might appear only once
[00:02:30] this token that might appear only once even though there's a clear consistent
[00:02:31] even though there's a clear consistent signal there we do have our emoticon and
[00:02:34] signal there we do have our emoticon and our link is mostly intact although this
[00:02:37] our link is mostly intact although this period could be disruptive if we
[00:02:38] period could be disruptive if we actually want to follow the link because
[00:02:40] actually want to follow the link because it's still glommed onto the end of the
[00:02:42] it's still glommed onto the end of the url
[00:02:44] url tree bank tokenizing is another very
[00:02:46] tree bank tokenizing is another very common scheme in nlp i would say at this
[00:02:47] common scheme in nlp i would say at this point largely for historical reasons the
[00:02:50] point largely for historical reasons the way tree bank tokenizing works is it
[00:02:52] way tree bank tokenizing works is it takes this raw text and splits it up
[00:02:54] takes this raw text and splits it up into a whole lot of tokens right in
[00:02:55] into a whole lot of tokens right in comparison with white space we have a
[00:02:57] comparison with white space we have a lot of distinct pieces here and this
[00:02:59] lot of distinct pieces here and this really looks kind of problematic right
[00:03:01] really looks kind of problematic right so we have destroyed our at mention we
[00:03:03] so we have destroyed our at mention we don't have that username anymore
[00:03:06] don't have that username anymore it does this interesting thing with
[00:03:07] it does this interesting thing with words like can't that they get split
[00:03:09] words like can't that they get split apart into two tokens
[00:03:11] apart into two tokens we've lost our date we have lost our
[00:03:13] we've lost our date we have lost our hashtag
[00:03:15] hashtag this is possibly good so yay has been
[00:03:17] this is possibly good so yay has been split up according to its punctuation so
[00:03:19] split up according to its punctuation so we have now four exclamation marks uh
[00:03:22] we have now four exclamation marks uh separated out from this word here
[00:03:24] separated out from this word here but our emoticon is completely lost and
[00:03:26] but our emoticon is completely lost and our link has been really destroyed so
[00:03:28] our link has been really destroyed so this looks problematic from the point of
[00:03:29] this looks problematic from the point of view of accurate featurization and also
[00:03:32] view of accurate featurization and also doing things with social media
[00:03:35] doing things with social media so that kind of brings me to what we
[00:03:37] so that kind of brings me to what we might want from what i've called a
[00:03:38] might want from what i've called a sentiment-aware tokenizer we would like
[00:03:40] sentiment-aware tokenizer we would like to isolate emoticons clearly because
[00:03:42] to isolate emoticons clearly because they can be really sentiment-laden
[00:03:45] they can be really sentiment-laden we want to probably respect twitter and
[00:03:46] we want to probably respect twitter and other domain specific markup because
[00:03:48] other domain specific markup because that's often the space in which our in
[00:03:50] that's often the space in which our in our data come from and the kind of place
[00:03:52] our data come from and the kind of place we want to make predictions in
[00:03:54] we want to make predictions in in a similar spirit you might take
[00:03:55] in a similar spirit you might take advantage of underlying markup maybe
[00:03:57] advantage of underlying markup maybe don't filter off the html because there
[00:03:59] don't filter off the html because there could be an important signal there
[00:04:01] could be an important signal there you might be aware that the website or
[00:04:04] you might be aware that the website or data producer might have done some
[00:04:05] data producer might have done some pre-processing of their own that might
[00:04:07] pre-processing of their own that might disrupt things like curses which could
[00:04:09] disrupt things like curses which could of course carry a lot of important
[00:04:11] of course carry a lot of important important sentiment information
[00:04:13] important sentiment information you might want to preserve
[00:04:14] you might want to preserve capitalization because of course that
[00:04:15] capitalization because of course that could be used for emphasis
[00:04:17] could be used for emphasis uh in a similar spirit you might want to
[00:04:19] uh in a similar spirit you might want to regularize emotive lengthening like yay
[00:04:22] regularize emotive lengthening like yay down to just three characters here to
[00:04:24] down to just three characters here to capture that it is an emotive
[00:04:25] capture that it is an emotive lengthening but also regularize all
[00:04:27] lengthening but also regularize all those distinct tokens
[00:04:29] those distinct tokens and then as a stretch goal although this
[00:04:31] and then as a stretch goal although this might be less important in the era of
[00:04:33] might be less important in the era of contextual models you might think about
[00:04:35] contextual models you might think about capturing multi-word expressions that
[00:04:37] capturing multi-word expressions that carry sentiment just think of an example
[00:04:39] carry sentiment just think of an example like out of this world which is positive
[00:04:42] like out of this world which is positive but none of its component pieces are
[00:04:43] but none of its component pieces are positive so many models will miss that
[00:04:45] positive so many models will miss that that is conveying clear sentiment
[00:04:47] that is conveying clear sentiment whereas with a clever tokenization
[00:04:49] whereas with a clever tokenization scheme you might capture that as one
[00:04:51] scheme you might capture that as one single token
[00:04:54] single token so here's a simple example uh that meets
[00:04:56] so here's a simple example uh that meets a lot of those goals here for a set of
[00:04:58] a lot of those goals here for a set of aware tokenizer we begin from our usual
[00:05:00] aware tokenizer we begin from our usual raw text we normalize and preserve the
[00:05:02] raw text we normalize and preserve the at mention
[00:05:03] at mention we keep most of these words intact and
[00:05:05] we keep most of these words intact and we kind of capture that that june 9
[00:05:07] we kind of capture that that june 9 thing was a date
[00:05:09] thing was a date preserve the hashtag of course
[00:05:11] preserve the hashtag of course we're treating all of these potentially
[00:05:13] we're treating all of these potentially emotion-laden punctuation marks as
[00:05:15] emotion-laden punctuation marks as separate unigrams i think that could be
[00:05:17] separate unigrams i think that could be good
[00:05:18] good of course capture the emoticon and
[00:05:19] of course capture the emoticon and capture the link
[00:05:21] capture the link and if you want something that meets
[00:05:23] and if you want something that meets more or less all these criteria except i
[00:05:25] more or less all these criteria except i think the date normalization you could
[00:05:27] think the date normalization you could just use the nltk tweet tokenizer it's a
[00:05:30] just use the nltk tweet tokenizer it's a good simple choice that you could make
[00:05:32] good simple choice that you could make that i think will be useful for
[00:05:34] that i think will be useful for sentiment analysis and to quantify that
[00:05:36] sentiment analysis and to quantify that a little bit here's a some experimental
[00:05:38] a little bit here's a some experimental evidence that i think is going to be
[00:05:39] evidence that i think is going to be relevant to the kind of work that you
[00:05:41] relevant to the kind of work that you all are doing
[00:05:42] all are doing so my data is open table that's
[00:05:44] so my data is open table that's restaurant reviews short ones
[00:05:46] restaurant reviews short ones i've got 6 000 reviews in my test set
[00:05:49] i've got 6 000 reviews in my test set and what i'm doing along the x axis here
[00:05:51] and what i'm doing along the x axis here is varying the amount of training data
[00:05:53] is varying the amount of training data that these systems can see
[00:05:55] that these systems can see it's simply a softmax classifier and my
[00:05:57] it's simply a softmax classifier and my primary manipulation is i have the
[00:05:59] primary manipulation is i have the sentiment aware tokenizer in orange
[00:06:01] sentiment aware tokenizer in orange treebank in green and whitespace in gray
[00:06:05] treebank in green and whitespace in gray and the picture is pretty clear right
[00:06:07] and the picture is pretty clear right along the x-axis we have accuracy it's a
[00:06:09] along the x-axis we have accuracy it's a balanced problem
[00:06:11] balanced problem and what you can see is that the
[00:06:13] and what you can see is that the sentiment aware tokenizer is the clear
[00:06:15] sentiment aware tokenizer is the clear winner here especially where training
[00:06:17] winner here especially where training data are sparse uh in the limit of
[00:06:19] data are sparse uh in the limit of adding lots of training data i think we
[00:06:21] adding lots of training data i think we can make up for a lot of shortcomings of
[00:06:23] can make up for a lot of shortcomings of tokenizers because we see a lot of
[00:06:25] tokenizers because we see a lot of redundancy in the training data
[00:06:27] redundancy in the training data but where data are sparse the center
[00:06:29] but where data are sparse the center manual tokenizer is clearly a good
[00:06:30] manual tokenizer is clearly a good choice and another thing i would add is
[00:06:32] choice and another thing i would add is that because it produces more intuitive
[00:06:34] that because it produces more intuitive tokens the sentiment aware models might
[00:06:36] tokens the sentiment aware models might be more interpretable in some sense
[00:06:40] be more interpretable in some sense and to really connect with the homework
[00:06:42] and to really connect with the homework that you all are doing and the bake off
[00:06:44] that you all are doing and the bake off this is what happens when we go across
[00:06:45] this is what happens when we go across domains so here i'm training on open
[00:06:47] domains so here i'm training on open table restaurant reviews but i'm going
[00:06:49] table restaurant reviews but i'm going to test on movie review sentences here
[00:06:52] to test on movie review sentences here otherwise this is the same experimental
[00:06:54] otherwise this is the same experimental paradigm because of the cross domain
[00:06:56] paradigm because of the cross domain thing the results are a little bit more
[00:06:58] thing the results are a little bit more chaotic but i think again the sentiment
[00:07:00] chaotic but i think again the sentiment aware tokenizer is a clear winner with
[00:07:03] aware tokenizer is a clear winner with the largest gains where training data
[00:07:05] the largest gains where training data are a little bit sparse and that's the
[00:07:06] are a little bit sparse and that's the expected picture
[00:07:09] expected picture so be thoughtful about tokenizing as a
[00:07:11] so be thoughtful about tokenizing as a counterpoint to that i've called this
[00:07:12] counterpoint to that i've called this section on stemming the dangers of
[00:07:14] section on stemming the dangers of stemming because what i want to try to
[00:07:15] stemming because what i want to try to do is convince you not to stem your data
[00:07:18] do is convince you not to stem your data but first what is stemming so stemming
[00:07:20] but first what is stemming so stemming is a kind of preprocessing technique
[00:07:22] is a kind of preprocessing technique that would collapse collapse distinct
[00:07:24] that would collapse collapse distinct word forms there are three common
[00:07:26] word forms there are three common algorithms for this
[00:07:27] algorithms for this easy to use the port or simmer the
[00:07:29] easy to use the port or simmer the lancaster stemmer and the wordnet
[00:07:31] lancaster stemmer and the wordnet stemmer and my criticisms are largely
[00:07:33] stemmer and my criticisms are largely leveled at porter and lancaster
[00:07:36] leveled at porter and lancaster here is the bottom line
[00:07:38] here is the bottom line in doing this kind of stemming you are
[00:07:40] in doing this kind of stemming you are apt to destroy many important sentiment
[00:07:42] apt to destroy many important sentiment distinctions making this a
[00:07:43] distinctions making this a counterproductive pre-processing step
[00:07:46] counterproductive pre-processing step on the other hand the word netstemmer
[00:07:48] on the other hand the word netstemmer does not have this problem
[00:07:50] does not have this problem it's much more conservative but it also
[00:07:52] it's much more conservative but it also doesn't really do enough to make it
[00:07:54] doesn't really do enough to make it worthwhile it's kind of costly to run it
[00:07:57] worthwhile it's kind of costly to run it has some requirements that might make it
[00:07:59] has some requirements that might make it simply not worth it and i would say that
[00:08:01] simply not worth it and i would say that the bottom line here for stemming is
[00:08:02] the bottom line here for stemming is that in an era where we have very large
[00:08:05] that in an era where we have very large sentiment data sets the function of
[00:08:06] sentiment data sets the function of stemming would be to collapse the size
[00:08:08] stemming would be to collapse the size of your vocabulary and make learning
[00:08:11] of your vocabulary and make learning more easier in small domains but we
[00:08:13] more easier in small domains but we mostly don't confront that problem
[00:08:15] mostly don't confront that problem anymore
[00:08:16] anymore but just to drive home this point here
[00:08:18] but just to drive home this point here here are some examples focused on the
[00:08:20] here are some examples focused on the port or stemmer of cases where running
[00:08:23] port or stemmer of cases where running the port or simmer actually collapses
[00:08:25] the port or simmer actually collapses clear sentiment distinctions according
[00:08:26] clear sentiment distinctions according to the harvard inquirer which is one of
[00:08:28] to the harvard inquirer which is one of those lexicons i mentioned before i've
[00:08:31] those lexicons i mentioned before i've got defense and defensive they get
[00:08:33] got defense and defensive they get collapsed down into this funny non-word
[00:08:35] collapsed down into this funny non-word defense extravagance an extravagant
[00:08:38] defense extravagance an extravagant different sentiment collapse down into
[00:08:40] different sentiment collapse down into this word fragment and so forth for
[00:08:42] this word fragment and so forth for these other examples and i think this is
[00:08:44] these other examples and i think this is showing that in pre-processing your data
[00:08:46] showing that in pre-processing your data you might be removing some important
[00:08:48] you might be removing some important sentiment signals
[00:08:50] sentiment signals the lancaster stemmer uses a very
[00:08:51] the lancaster stemmer uses a very similar strategy and has arguably even
[00:08:53] similar strategy and has arguably even more problems in this space here we've
[00:08:56] more problems in this space here we've got the positive word uh complement and
[00:08:58] got the positive word uh complement and complicate
[00:08:59] complicate according to the harvard inquirer again
[00:09:01] according to the harvard inquirer again they could both get collapsed down into
[00:09:03] they could both get collapsed down into what is a completely distinct word
[00:09:04] what is a completely distinct word comply
[00:09:05] comply and that should be concerning for many
[00:09:07] and that should be concerning for many reasons and the other examples make a
[00:09:09] reasons and the other examples make a very similar point
[00:09:12] very similar point the word netstemmer i mentioned before i
[00:09:14] the word netstemmer i mentioned before i think this actually has something going
[00:09:15] think this actually has something going for it uh there might be cases where
[00:09:17] for it uh there might be cases where you'd want to use it it's high precision
[00:09:19] you'd want to use it it's high precision it requires word part of speech pairs
[00:09:23] it requires word part of speech pairs and the general issue is just that it
[00:09:24] and the general issue is just that it removes some comparative morphology
[00:09:26] removes some comparative morphology that's the only thing you might worry
[00:09:28] that's the only thing you might worry about for sentiment but otherwise it's
[00:09:30] about for sentiment but otherwise it's going to take like exclaims explained
[00:09:31] going to take like exclaims explained and exclaiming and collapse them down
[00:09:33] and exclaiming and collapse them down that could be a useful compression of
[00:09:35] that could be a useful compression of your feature space it will leave
[00:09:37] your feature space it will leave exclamation alone which i think is good
[00:09:39] exclamation alone which i think is good similarly for these things they all kind
[00:09:41] similarly for these things they all kind of get preserved across the two verb
[00:09:43] of get preserved across the two verb forms but we preserve the adjective as
[00:09:45] forms but we preserve the adjective as different i think that could be good and
[00:09:47] different i think that could be good and as i said the only concern would be that
[00:09:49] as i said the only concern would be that happy happier and happiest all go down
[00:09:51] happy happier and happiest all go down into their base form whereas i think
[00:09:53] into their base form whereas i think these could encode different gradations
[00:09:55] these could encode different gradations of sentiment that you might want to
[00:09:57] of sentiment that you might want to preserve
[00:09:58] preserve that's worth some thought but overall i
[00:10:00] that's worth some thought but overall i think you probably want to avoid doing
[00:10:02] think you probably want to avoid doing standing and to bring that home let's
[00:10:03] standing and to bring that home let's return to my experimental paradigm using
[00:10:06] return to my experimental paradigm using a softmax classifier opentable reviews
[00:10:09] a softmax classifier opentable reviews 6000 of them in my test set and here
[00:10:11] 6000 of them in my test set and here along the x-axis i'm varying the amount
[00:10:13] along the x-axis i'm varying the amount of training data i have
[00:10:15] of training data i have and i think what you see is that the
[00:10:18] and i think what you see is that the porter and lancaster stemmer and purple
[00:10:19] porter and lancaster stemmer and purple and black respectively are kind of
[00:10:21] and black respectively are kind of forever behind right versus just simply
[00:10:24] forever behind right versus just simply sentiment aware tokenizing it gives you
[00:10:26] sentiment aware tokenizing it gives you a lead
[00:10:27] a lead the lead is especially clear as you get
[00:10:29] the lead is especially clear as you get out of this very sparse domain here with
[00:10:32] out of this very sparse domain here with very few training instances
[00:10:34] very few training instances to close just a few other preprocessing
[00:10:37] to close just a few other preprocessing techniques that you might think about so
[00:10:38] techniques that you might think about so you could part a speech tied your data
[00:10:41] you could part a speech tied your data in the spirit of trying to capture more
[00:10:43] in the spirit of trying to capture more sentiment distinctions that you might
[00:10:44] sentiment distinctions that you might capture otherwise so just for example a
[00:10:47] capture otherwise so just for example a rest or like a resting as an adjective
[00:10:49] rest or like a resting as an adjective is positive but a rest as a verb is
[00:10:50] is positive but a rest as a verb is typically negative
[00:10:52] typically negative um fine as an adjective is positive but
[00:10:55] um fine as an adjective is positive but to incur a fine as a noun is negative
[00:10:57] to incur a fine as a noun is negative and so forth you can see that some
[00:10:59] and so forth you can see that some sentiment distinctions actually do turn
[00:11:01] sentiment distinctions actually do turn on the part of speech of the word so
[00:11:04] on the part of speech of the word so treating all of your unigram features as
[00:11:07] treating all of your unigram features as based in word part of speech tag pairs
[00:11:10] based in word part of speech tag pairs could be useful for preserving some of
[00:11:11] could be useful for preserving some of these distinctions again as a
[00:11:13] these distinctions again as a pre-processing step to help your model
[00:11:15] pre-processing step to help your model be more attuned to these points of
[00:11:17] be more attuned to these points of variation in comparison
[00:11:19] variation in comparison but if there are limits even to this
[00:11:21] but if there are limits even to this right so this is just some cases on
[00:11:22] right so this is just some cases on these slides where even within the same
[00:11:24] these slides where even within the same part of speech we have an adjective that
[00:11:27] part of speech we have an adjective that in one sense is positive and another
[00:11:29] in one sense is positive and another negative for example the adjective mean
[00:11:32] negative for example the adjective mean can mean hateful but it can also mean
[00:11:34] can mean hateful but it can also mean excellent as in uh they make a mean
[00:11:36] excellent as in uh they make a mean apple pie
[00:11:38] apple pie um
[00:11:38] um smart
[00:11:40] smart as an adjective could be both painful uh
[00:11:43] as an adjective could be both painful uh and also bright and brilliant and so
[00:11:45] and also bright and brilliant and so forth like that and similarly for
[00:11:46] forth like that and similarly for serious and fantastic and sneer
[00:11:48] serious and fantastic and sneer depending on the context and the
[00:11:50] depending on the context and the intention of the speaker they can kind
[00:11:52] intention of the speaker they can kind of cut in different directions um so
[00:11:54] of cut in different directions um so even part of speech tagging is going to
[00:11:56] even part of speech tagging is going to be limiting when it comes to really
[00:11:58] be limiting when it comes to really recovering the underlying word sense
[00:12:00] recovering the underlying word sense even for something as low dimensional as
[00:12:02] even for something as low dimensional as a sentiment distinction
[00:12:05] a sentiment distinction finally no this is another powerful
[00:12:08] finally no this is another powerful technique that you might use and think
[00:12:09] technique that you might use and think about as you select and evaluate
[00:12:11] about as you select and evaluate different models this is what i've
[00:12:13] different models this is what i've called simple negation marking the
[00:12:15] called simple negation marking the phenomenon is just that if i have a verb
[00:12:17] phenomenon is just that if i have a verb like enjoy which sounds positive in
[00:12:20] like enjoy which sounds positive in isolation
[00:12:21] isolation of course its contribution to the
[00:12:22] of course its contribution to the overall sentiment will change depending
[00:12:25] overall sentiment will change depending on whether it's in the scope of a
[00:12:26] on whether it's in the scope of a negation i didn't enjoy it as negative
[00:12:29] negation i didn't enjoy it as negative and negation can be expressed in many
[00:12:31] and negation can be expressed in many ways
[00:12:32] ways as this modifier of auxiliaries like not
[00:12:35] as this modifier of auxiliaries like not but as an adverb like never it could be
[00:12:37] but as an adverb like never it could be in the subject like no one and it could
[00:12:39] in the subject like no one and it could even be really encode for things like i
[00:12:41] even be really encode for things like i have yet to enjoy it which is a kind of
[00:12:44] have yet to enjoy it which is a kind of negation
[00:12:45] negation and then of course the negation in five
[00:12:47] and then of course the negation in five years very far i don't think i will
[00:12:49] years very far i don't think i will enjoy it is probably negative but the
[00:12:51] enjoy it is probably negative but the negation is way far away uh from the
[00:12:54] negation is way far away uh from the verb that we want to have that we want
[00:12:57] verb that we want to have that we want whose sentiment we want to modulate with
[00:12:58] whose sentiment we want to modulate with the negation
[00:13:00] the negation so here's a very simple method that i
[00:13:02] so here's a very simple method that i think was first exploited by explored by
[00:13:04] think was first exploited by explored by dawson chen it's also used in peng at
[00:13:06] dawson chen it's also used in peng at all these are classic early sentiment
[00:13:08] all these are classic early sentiment analysis papers and the idea is simply
[00:13:10] analysis papers and the idea is simply to append a neg suffix to every word in
[00:13:13] to append a neg suffix to every word in the sequence that appears between the
[00:13:15] the sequence that appears between the negation and some clause level mark of
[00:13:17] negation and some clause level mark of punctuation to sort of roughly indicate
[00:13:19] punctuation to sort of roughly indicate the semantic scope of the negation this
[00:13:22] the semantic scope of the negation this is a simple pre-processing step highly
[00:13:24] is a simple pre-processing step highly heuristic it would take a sentence like
[00:13:26] heuristic it would take a sentence like no one enjoys it and literally turn the
[00:13:29] no one enjoys it and literally turn the unigrams one and joys and it
[00:13:32] unigrams one and joys and it into variant forms of them where one has
[00:13:34] into variant forms of them where one has an egg appended to it and so does it
[00:13:36] an egg appended to it and so does it enjoys and so does it and the idea is
[00:13:38] enjoys and so does it and the idea is that in doing this we're giving our
[00:13:39] that in doing this we're giving our model the opportunity to discover that
[00:13:42] model the opportunity to discover that enjoys in this context is actually a
[00:13:44] enjoys in this context is actually a different token in some sense than
[00:13:46] different token in some sense than enjoys when it's not in the scope of
[00:13:48] enjoys when it's not in the scope of negation and for many of the linear
[00:13:50] negation and for many of the linear models with handout features that we
[00:13:52] models with handout features that we explore simply making that initial
[00:13:54] explore simply making that initial distinction might create some space for
[00:13:56] distinction might create some space for your model to learn the interaction of
[00:13:59] your model to learn the interaction of negation with these other
[00:14:00] negation with these other features and just to quantify it a
[00:14:03] features and just to quantify it a little bit by way of rounding this out i
[00:14:04] little bit by way of rounding this out i think this slide shows the impact that
[00:14:06] think this slide shows the impact that this can have despite its simplicity so
[00:14:09] this can have despite its simplicity so similar we have open table as our test
[00:14:12] similar we have open table as our test set we're using a softmax classifier and
[00:14:14] set we're using a softmax classifier and the x-axis is again varying the amount
[00:14:16] the x-axis is again varying the amount of training data that we have
[00:14:19] of training data that we have the white space tokenizer isn't gray
[00:14:20] the white space tokenizer isn't gray it's the worst followed by tree bank and
[00:14:22] it's the worst followed by tree bank and green
[00:14:23] green then we have that sentiment aware
[00:14:25] then we have that sentiment aware tokenizer in orange
[00:14:27] tokenizer in orange and then way above them consistently for
[00:14:29] and then way above them consistently for all parts of the data here are sentiment
[00:14:31] all parts of the data here are sentiment aware plus that negation marking
[00:14:34] aware plus that negation marking that is obviously the superior model for
[00:14:36] that is obviously the superior model for all kinds of amounts of training data
[00:14:38] all kinds of amounts of training data and i think what that's showing is that
[00:14:40] and i think what that's showing is that the influence of negation is actually
[00:14:42] the influence of negation is actually really real and severe in a lot of
[00:14:44] really real and severe in a lot of sentiment data sets it's just very
[00:14:46] sentiment data sets it's just very common to combine
[00:14:48] common to combine sentiment words positive or negative
[00:14:50] sentiment words positive or negative with negation and it has this
[00:14:51] with negation and it has this predictable effect of kind of flipping
[00:14:53] predictable effect of kind of flipping the value
[00:14:54] the value so in doing this sentiment and doing
[00:14:56] so in doing this sentiment and doing this negation marking we're giving our
[00:14:58] this negation marking we're giving our model a better chance at discovering
[00:15:00] model a better chance at discovering exactly those distinctions
[00:15:02] exactly those distinctions and here's a similar set of results for
[00:15:04] and here's a similar set of results for cross domain where i'm starting on open
[00:15:06] cross domain where i'm starting on open table and testing on imdb again the
[00:15:09] table and testing on imdb again the results are a little bit more chaotic
[00:15:10] results are a little bit more chaotic but i think it's a clear win for
[00:15:13] but i think it's a clear win for the sentiment aware plus negation
[00:15:14] the sentiment aware plus negation marking model

Stanford Sentiment Treebank | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=DxnXVbHGeBg

---

Transcript

[00:00:04] welcome back everyone this is part three
[00:00:06] welcome back everyone this is part three in our series on supervised sentiment
[00:00:07] in our series on supervised sentiment analysis this screencast is going to
[00:00:09] analysis this screencast is going to focus on the stanford sentiment tree
[00:00:10] focus on the stanford sentiment tree bank let me start with a quick project
[00:00:12] bank let me start with a quick project overview uh the associated paper is
[00:00:14] overview uh the associated paper is social sochart all 2013.
[00:00:17] social sochart all 2013. i think this paper is a kind of model of
[00:00:18] i think this paper is a kind of model of open science at this website here you
[00:00:20] open science at this website here you can see the full code uh all the data of
[00:00:23] can see the full code uh all the data of course as well as an api that will let
[00:00:24] course as well as an api that will let you try out new examples and kind of
[00:00:26] you try out new examples and kind of interact with the core models that are
[00:00:28] interact with the core models that are motivated in the paper
[00:00:30] motivated in the paper it's a sentence level corpus it's got
[00:00:32] it's a sentence level corpus it's got about 11 000 sentences in total and all
[00:00:34] about 11 000 sentences in total and all of those sentences are originally from
[00:00:36] of those sentences are originally from rotten tomatoes so they are sentences
[00:00:38] rotten tomatoes so they are sentences from movie reviews uh the sentences
[00:00:40] from movie reviews uh the sentences themselves were originally released by
[00:00:42] themselves were originally released by pang and lee in 2005 it's kind of
[00:00:44] pang and lee in 2005 it's kind of classic data set and what the sst did
[00:00:46] classic data set and what the sst did was expand the data set by labeling not
[00:00:49] was expand the data set by labeling not only the full sentences
[00:00:51] only the full sentences but all of the sub constituents
[00:00:53] but all of the sub constituents according to a kind of traditional parse
[00:00:55] according to a kind of traditional parse syntactic parse of each of the examples
[00:00:57] syntactic parse of each of the examples and those are all crowd source labels so
[00:00:59] and those are all crowd source labels so what this means is that we have vastly
[00:01:01] what this means is that we have vastly more supervision signals all throughout
[00:01:04] more supervision signals all throughout the structure of these examples than we
[00:01:06] the structure of these examples than we would get from the original or we just
[00:01:07] would get from the original or we just had a single sentiment label for the
[00:01:09] had a single sentiment label for the entire
[00:01:10] entire sentence the labels themselves in the
[00:01:13] sentence the labels themselves in the underlying corpus are five-way label
[00:01:15] underlying corpus are five-way label labels that are extracted from workers
[00:01:17] labels that are extracted from workers slider responses so there's kind of
[00:01:19] slider responses so there's kind of an initial layer of aggregation they
[00:01:21] an initial layer of aggregation they made a slider choice they were all
[00:01:23] made a slider choice they were all grouped together into five labels and
[00:01:26] grouped together into five labels and then we are going to work with a
[00:01:27] then we are going to work with a formulation that is even more collapsed
[00:01:28] formulation that is even more collapsed down to ternary sentiment i'll return to
[00:01:31] down to ternary sentiment i'll return to that a bit later
[00:01:33] that a bit later the fully labeled tree thing is one of
[00:01:35] the fully labeled tree thing is one of the really exciting aspects of this
[00:01:37] the really exciting aspects of this corpus that we will be able to take
[00:01:39] corpus that we will be able to take advantage of especially during training
[00:01:40] advantage of especially during training so the way that worked is there were
[00:01:42] so the way that worked is there were parses this is a simple
[00:01:44] parses this is a simple constituent parse of a sent the sentence
[00:01:46] constituent parse of a sent the sentence nlu is enlightening and as i've
[00:01:48] nlu is enlightening and as i've indicated here we have labels in that
[00:01:50] indicated here we have labels in that space zero through four
[00:01:53] space zero through four on all of the lexical items nlu is an
[00:01:55] on all of the lexical items nlu is an enlightening as well as the altis sub
[00:01:57] enlightening as well as the altis sub constituents in this phrase so you can
[00:01:59] constituents in this phrase so you can see that is is neutral but since
[00:02:01] see that is is neutral but since enlightening is positive the whole verb
[00:02:03] enlightening is positive the whole verb phrase is enlightening is positive
[00:02:05] phrase is enlightening is positive we can say that nlu is neutral but in
[00:02:08] we can say that nlu is neutral but in the context of this sentence the overall
[00:02:09] the context of this sentence the overall contribution is a highly positive one so
[00:02:12] contribution is a highly positive one so label four on the root
[00:02:14] label four on the root um
[00:02:15] um i in the first screencast for this unit
[00:02:17] i in the first screencast for this unit i motivated sentiment analysis with some
[00:02:19] i motivated sentiment analysis with some cases that i thought were kind of
[00:02:21] cases that i thought were kind of difficult from a syntactic point of view
[00:02:23] difficult from a syntactic point of view this is one of them they said it would
[00:02:24] this is one of them they said it would be great i love how this is being
[00:02:26] be great i love how this is being handled we can see that down here be
[00:02:28] handled we can see that down here be great is kind of clearly positive but by
[00:02:31] great is kind of clearly positive but by the time we have filtered that through
[00:02:33] the time we have filtered that through this report they said just kind of
[00:02:35] this report they said just kind of displacing the sentiment onto another
[00:02:37] displacing the sentiment onto another agent the speaker is not necessarily
[00:02:39] agent the speaker is not necessarily endorsing the claim of greatness uh what
[00:02:41] endorsing the claim of greatness uh what we get in the end is more like a neutral
[00:02:43] we get in the end is more like a neutral sentiment i think that's interesting and
[00:02:45] sentiment i think that's interesting and we can extend that even further right
[00:02:47] we can extend that even further right these are actual predictions from the
[00:02:48] these are actual predictions from the model that's motivated in the underlying
[00:02:50] model that's motivated in the underlying paper if we take that constituent that i
[00:02:52] paper if we take that constituent that i just showed you
[00:02:53] just showed you and can join it with they were wrong
[00:02:55] and can join it with they were wrong which is clearly negative uh strikingly
[00:02:58] which is clearly negative uh strikingly the model is able to figure out that the
[00:02:59] the model is able to figure out that the overall sentiment is determined by this
[00:03:01] overall sentiment is determined by this second clause here and assigned
[00:03:02] second clause here and assigned negatives to the entire thing despite
[00:03:04] negatives to the entire thing despite the fact that there are obviously
[00:03:06] the fact that there are obviously sub-constituents in here that are
[00:03:07] sub-constituents in here that are positive that's exactly the kind of
[00:03:09] positive that's exactly the kind of mixing that i think is correct for how
[00:03:11] mixing that i think is correct for how language works in the domain of
[00:03:13] language works in the domain of sentiment and it's kind of encouraging
[00:03:15] sentiment and it's kind of encouraging to see that this model is able to
[00:03:16] to see that this model is able to capture at least some aspects of it
[00:03:18] capture at least some aspects of it here's a similar case that i think is
[00:03:20] here's a similar case that i think is pretty good as well although maybe not
[00:03:22] pretty good as well although maybe not as strikingly positive in the end here
[00:03:24] as strikingly positive in the end here i've just changed from the previous
[00:03:26] i've just changed from the previous example they were they were wrong till
[00:03:28] example they were they were wrong till they were right
[00:03:29] they were right it knows that right is correct and it
[00:03:31] it knows that right is correct and it seems to get that this is kind of middle
[00:03:33] seems to get that this is kind of middle of the scale i might hope this was a
[00:03:35] of the scale i might hope this was a three or a four but i think that still
[00:03:37] three or a four but i think that still we're seeing some interesting
[00:03:38] we're seeing some interesting interactions between what's happening in
[00:03:41] interactions between what's happening in sub-constituents in these examples and
[00:03:43] sub-constituents in these examples and the prediction that's made at the root
[00:03:44] the prediction that's made at the root level so it's very encouraging
[00:03:47] level so it's very encouraging there are a bunch of ways that you can
[00:03:49] there are a bunch of ways that you can formulate the sst task kind of the raw
[00:03:52] formulate the sst task kind of the raw one that comes from the paper would be a
[00:03:54] one that comes from the paper would be a five-way classification problem where
[00:03:56] five-way classification problem where we'd have these numerical labels here
[00:03:58] we'd have these numerical labels here with the meaning of kind of zero is very
[00:04:00] with the meaning of kind of zero is very negative
[00:04:01] negative one is negative two is neutral three is
[00:04:04] one is negative two is neutral three is positive and four is very positive
[00:04:06] positive and four is very positive i think this is fine but there are two
[00:04:08] i think this is fine but there are two gotchas underlying this kind of scheme
[00:04:10] gotchas underlying this kind of scheme first it's not really a fully ordered
[00:04:12] first it's not really a fully ordered scale in the sense that
[00:04:14] scale in the sense that four is stronger than three
[00:04:17] four is stronger than three but zero is stronger than one because we
[00:04:19] but zero is stronger than one because we have this kind of polarity split with
[00:04:20] have this kind of polarity split with neutral in the center
[00:04:23] neutral in the center so that's a kind of conceptual
[00:04:24] so that's a kind of conceptual difficulty and then the other part is
[00:04:25] difficulty and then the other part is that by and large classifier models that
[00:04:28] that by and large classifier models that you pick will not give you partial
[00:04:30] you pick will not give you partial credit for being close we might hope
[00:04:32] credit for being close we might hope that a model that predicted a one
[00:04:34] that a model that predicted a one negative
[00:04:35] negative uh was kind of right or certainly more
[00:04:38] uh was kind of right or certainly more right if the true label is zero than a
[00:04:40] right if the true label is zero than a model that had predicted four but of
[00:04:42] model that had predicted four but of course if these are all treated as
[00:04:43] course if these are all treated as independent classification bins then
[00:04:45] independent classification bins then you'll get
[00:04:46] you'll get you're just equally wrong no matter
[00:04:48] you're just equally wrong no matter which prediction you made relative to
[00:04:50] which prediction you made relative to the gold label and that seems kind of
[00:04:52] the gold label and that seems kind of unfair to our models
[00:04:55] unfair to our models we are going to work with what i've
[00:04:56] we are going to work with what i've called the ternary problem i think this
[00:04:58] called the ternary problem i think this is the minimal problem that really makes
[00:05:00] is the minimal problem that really makes sense conceptually for this one we group
[00:05:02] sense conceptually for this one we group zero and one into a negative category
[00:05:04] zero and one into a negative category three and four into a positive category
[00:05:06] three and four into a positive category and remem reserve two as before for what
[00:05:09] and remem reserve two as before for what we're calling neutral
[00:05:12] we're calling neutral this kind of avoids the false
[00:05:13] this kind of avoids the false presupposition that every sentence is
[00:05:15] presupposition that every sentence is either negative or positive because it
[00:05:17] either negative or positive because it does allow us to make predictions into
[00:05:19] does allow us to make predictions into this neutral or non-sentiment-laden
[00:05:21] this neutral or non-sentiment-laden space
[00:05:24] space it's very common and you see this in the
[00:05:25] it's very common and you see this in the paper as well as in a lot of work on the
[00:05:27] paper as well as in a lot of work on the sst to formulate this as a binary
[00:05:29] sst to formulate this as a binary problem for the binary problem we simply
[00:05:31] problem for the binary problem we simply remove the middle of the scale and treat
[00:05:33] remove the middle of the scale and treat zero and one as negative and three and
[00:05:34] zero and one as negative and three and four as positive as before i think that
[00:05:37] four as positive as before i think that has two drawbacks first we have to throw
[00:05:38] has two drawbacks first we have to throw away some data and second then we're
[00:05:40] away some data and second then we're making this false presupposition that
[00:05:42] making this false presupposition that every sentence is either classified as
[00:05:44] every sentence is either classified as negative or as positive when for a wide
[00:05:47] negative or as positive when for a wide range of cases in the world that might
[00:05:48] range of cases in the world that might be inappropriate
[00:05:51] be inappropriate now i focus here on the root level
[00:05:53] now i focus here on the root level problem you can see that the numbers
[00:05:54] problem you can see that the numbers here for train and dev are small and the
[00:05:56] here for train and dev are small and the test set numbers are a little bit larger
[00:05:58] test set numbers are a little bit larger than for devs so they're comparable
[00:06:00] than for devs so they're comparable but we can also think of this as the all
[00:06:02] but we can also think of this as the all nodes task because recall that a
[00:06:05] nodes task because recall that a hallmark feature of the sst is that
[00:06:07] hallmark feature of the sst is that every single sub-constituent in these
[00:06:09] every single sub-constituent in these examples has been labeled by crowd
[00:06:10] examples has been labeled by crowd workers so we could treat each one of
[00:06:12] workers so we could treat each one of those as a kind of independent
[00:06:14] those as a kind of independent classification problem we have the same
[00:06:16] classification problem we have the same range for all of the values that they
[00:06:18] range for all of the values that they can take on so we can do similar kind of
[00:06:20] can take on so we can do similar kind of collapsing down into the ternary problem
[00:06:23] collapsing down into the ternary problem or the binary problem and of course here
[00:06:25] or the binary problem and of course here we have a much larger data set
[00:06:29] for us
[00:06:31] for us we're going to by and large work with
[00:06:33] we're going to by and large work with the data in one particular way which i
[00:06:35] the data in one particular way which i think is common in the literature as i
[00:06:37] think is common in the literature as i said we're going to have the ternary
[00:06:38] said we're going to have the ternary formulation so our labels will be
[00:06:39] formulation so our labels will be positive negative and neutral
[00:06:42] positive negative and neutral when we do the devin test step we're
[00:06:45] when we do the devin test step we're going to test only on full examples so
[00:06:47] going to test only on full examples so for them we will not make predictions
[00:06:49] for them we will not make predictions into the sub-constituent space
[00:06:53] into the sub-constituent space and then as a default for the code as
[00:06:55] and then as a default for the code as you'll see it is set up to train only on
[00:06:57] you'll see it is set up to train only on full examples so for these two cases nlu
[00:07:00] full examples so for these two cases nlu is enlightening and not enlightening
[00:07:02] is enlightening and not enlightening here if those were two independent
[00:07:03] here if those were two independent sentences in the corpus we would train
[00:07:05] sentences in the corpus we would train just on those two independent examples
[00:07:07] just on those two independent examples one negative labeled positive and the
[00:07:09] one negative labeled positive and the other labeled negative
[00:07:11] other labeled negative however you might imagine that you'll
[00:07:13] however you might imagine that you'll get a lot more strength in training if
[00:07:16] get a lot more strength in training if you also train on all the
[00:07:17] you also train on all the sub-constituents which would mean
[00:07:19] sub-constituents which would mean essentially expanding this example into
[00:07:21] essentially expanding this example into its full root version and all you is
[00:07:23] its full root version and all you is enlightening but also all of the
[00:07:25] enlightening but also all of the sub-pieces that are captured and labeled
[00:07:27] sub-pieces that are captured and labeled in corpus so that would give you many
[00:07:28] in corpus so that would give you many more examples and much more diversity uh
[00:07:31] more examples and much more diversity uh and then of course not enlightening
[00:07:33] and then of course not enlightening would be split apart as well and then
[00:07:35] would be split apart as well and then you could decide for yourself in
[00:07:36] you could decide for yourself in addition whether you want to treat this
[00:07:38] addition whether you want to treat this as two instances of enlightening or one
[00:07:41] as two instances of enlightening or one and the code facilitates all this you
[00:07:42] and the code facilitates all this you can formulate it as a route only trading
[00:07:44] can formulate it as a route only trading scenario or as a sub-constituent
[00:07:46] scenario or as a sub-constituent training scenario and you can keep or
[00:07:48] training scenario and you can keep or remove duplicates
[00:07:50] remove duplicates this is going to impact the amount of
[00:07:51] this is going to impact the amount of computational resources that you need
[00:07:53] computational resources that you need for training models but of course bigger
[00:07:56] for training models but of course bigger could be better in this space because
[00:07:57] could be better in this space because you're just seeing much more
[00:07:59] you're just seeing much more gold labeled information
[00:08:02] gold labeled information so that's the overview and for much more
[00:08:03] so that's the overview and for much more on this and how to work with our
[00:08:05] on this and how to work with our distribution of the corpus and so forth
[00:08:07] distribution of the corpus and so forth i would encourage you to work through
[00:08:08] i would encourage you to work through this note what could i have linked at
[00:08:09] this note what could i have linked at the bottom here

DynaSent | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=o-UFgavFlQg

---

Transcript

[00:00:05] hello everyone welcome to part four in
[00:00:06] hello everyone welcome to part four in our series on supervised sentiment
[00:00:08] our series on supervised sentiment analysis this is the second screencast
[00:00:10] analysis this is the second screencast in the series that is focused on a data
[00:00:12] in the series that is focused on a data set for sentiment and that data set is
[00:00:14] set for sentiment and that data set is dynacent uh this video could be
[00:00:16] dynacent uh this video could be considered an optional element in the
[00:00:17] considered an optional element in the series i'm offering it for two reasons
[00:00:20] series i'm offering it for two reasons really first uh this is a new data set
[00:00:22] really first uh this is a new data set that i helped produce and i would love
[00:00:24] that i helped produce and i would love it if people worked on it it would be
[00:00:25] it if people worked on it it would be great to see some new models new
[00:00:27] great to see some new models new insights
[00:00:28] insights all of that would help push this project
[00:00:30] all of that would help push this project forward in interesting ways the second
[00:00:32] forward in interesting ways the second reason is more practical i think that
[00:00:34] reason is more practical i think that this data set could be useful to you as
[00:00:35] this data set could be useful to you as you work on the assignment and the
[00:00:37] you work on the assignment and the associated bake off you could use the
[00:00:39] associated bake off you could use the data set itself for supplementary
[00:00:41] data set itself for supplementary training data you could use it to
[00:00:43] training data you could use it to evaluate your system and as you'll see
[00:00:45] evaluate your system and as you'll see there are a few points of conceptual
[00:00:47] there are a few points of conceptual connection between this data set and the
[00:00:49] connection between this data set and the brand new dev and test sets of
[00:00:51] brand new dev and test sets of restaurant sentences that are part of
[00:00:54] restaurant sentences that are part of the bake off this year
[00:00:57] so let's dive in here's a project
[00:00:58] so let's dive in here's a project overview first all the data code and
[00:01:00] overview first all the data code and models are available on github at this
[00:01:02] models are available on github at this link
[00:01:03] link um this data set itself consists of
[00:01:05] um this data set itself consists of about 122 000 sentences
[00:01:08] about 122 000 sentences they are across two rounds and i'm going
[00:01:10] they are across two rounds and i'm going to cover what each round means
[00:01:12] to cover what each round means and each of the sentences has five gold
[00:01:14] and each of the sentences has five gold labels in addition to an inferred
[00:01:16] labels in addition to an inferred majority label where there is one and
[00:01:17] majority label where there is one and i'll return to that as well i think
[00:01:19] i'll return to that as well i think that's an interesting aspect to this
[00:01:21] that's an interesting aspect to this kind of data collection
[00:01:23] kind of data collection the associated paper is possible 2020
[00:01:25] the associated paper is possible 2020 which i encourage you to read if you
[00:01:26] which i encourage you to read if you want to learn even more about this data
[00:01:28] want to learn even more about this data set and how in particular it relates to
[00:01:31] set and how in particular it relates to the stanford sentiment tree bank our
[00:01:33] the stanford sentiment tree bank our other core data set
[00:01:35] other core data set and another ingredient here as you'll
[00:01:37] and another ingredient here as you'll see when we get to round two is that
[00:01:39] see when we get to round two is that this is partly an effort in
[00:01:42] this is partly an effort in model in the loop adversarial data set
[00:01:44] model in the loop adversarial data set creation for round two crowd workers
[00:01:47] creation for round two crowd workers interacted with a model attempting to
[00:01:49] interacted with a model attempting to fool it and thereby creating sentences
[00:01:51] fool it and thereby creating sentences that are really difficult and are going
[00:01:52] that are really difficult and are going to challenge our models in what we hope
[00:01:54] to challenge our models in what we hope are exciting and productive ways
[00:01:57] are exciting and productive ways so here's a complete project overview
[00:01:59] so here's a complete project overview let me walk through it quickly and then
[00:02:00] let me walk through it quickly and then we'll dive into the details we begin
[00:02:02] we'll dive into the details we begin with what we've called model zero which
[00:02:04] with what we've called model zero which is a roberta model that's fine tuned on
[00:02:06] is a roberta model that's fine tuned on a bunch of very large sentiment
[00:02:08] a bunch of very large sentiment benchmark data sets
[00:02:10] benchmark data sets the primary utility of model zero is
[00:02:12] the primary utility of model zero is that we're going to use it as a device
[00:02:14] that we're going to use it as a device to find challenging naturally occurring
[00:02:16] to find challenging naturally occurring sentences out in a large corpus
[00:02:20] sentences out in a large corpus and then we human validate those to get
[00:02:22] and then we human validate those to get actual labels for them the result of
[00:02:24] actual labels for them the result of that process is what we hope uh is a
[00:02:27] that process is what we hope uh is a really challenging round one data set of
[00:02:29] really challenging round one data set of naturally occurring sentences that are
[00:02:31] naturally occurring sentences that are hard for a very good sentiment model
[00:02:33] hard for a very good sentiment model like model zero
[00:02:35] like model zero on that basis we then train a model one
[00:02:38] on that basis we then train a model one which is similar to model zero but now
[00:02:40] which is similar to model zero but now extended with that round one training
[00:02:42] extended with that round one training data so we hope that in bringing in that
[00:02:45] data so we hope that in bringing in that new data and combining it with the
[00:02:47] new data and combining it with the sentiment benchmarks we get an even
[00:02:49] sentiment benchmarks we get an even stronger model
[00:02:50] stronger model that is the model that crowd workers
[00:02:52] that is the model that crowd workers interacted with on the dynabench
[00:02:54] interacted with on the dynabench platform to try to create examples that
[00:02:56] platform to try to create examples that are adversarial with respect to model
[00:02:58] are adversarial with respect to model one so they ought to be really difficult
[00:03:01] one so they ought to be really difficult we feed those through exactly the same
[00:03:02] we feed those through exactly the same human validation pipeline and that gives
[00:03:04] human validation pipeline and that gives us our second round of data
[00:03:07] us our second round of data so two rounds of data that can be
[00:03:08] so two rounds of data that can be thought of as separate problems or merge
[00:03:10] thought of as separate problems or merge together into a larger data set i think
[00:03:12] together into a larger data set i think we're kind of still deciding how best to
[00:03:15] we're kind of still deciding how best to conceptualize these various data assets
[00:03:18] conceptualize these various data assets so let's look at round one in a little
[00:03:20] so let's look at round one in a little more detail this is where we begin with
[00:03:22] more detail this is where we begin with model 0 and try to harvest interesting
[00:03:24] model 0 and try to harvest interesting naturally occurring sentences
[00:03:27] naturally occurring sentences we so the rod model 0 is a roberta-based
[00:03:30] we so the rod model 0 is a roberta-based classifier and its training data
[00:03:32] classifier and its training data are from customer reviews which is small
[00:03:35] are from customer reviews which is small the imdb data set which i linked to in
[00:03:37] the imdb data set which i linked to in an earlier screencast sst3 which you saw
[00:03:40] an earlier screencast sst3 which you saw in the previous screencast and then
[00:03:42] in the previous screencast and then these two very large external benchmarks
[00:03:44] these two very large external benchmarks of product and service reviews from yelp
[00:03:47] of product and service reviews from yelp at amazon you can see that they're very
[00:03:49] at amazon you can see that they're very big indeed
[00:03:51] big indeed and the performance of model zero on the
[00:03:53] and the performance of model zero on the data sets these are our three external
[00:03:55] data sets these are our three external data sets it's pretty good they're range
[00:03:57] data sets it's pretty good they're range from the low 70s for sst three to the
[00:04:00] from the low 70s for sst three to the high 70s for yelp and amazon so this is
[00:04:02] high 70s for yelp and amazon so this is a solid model and i will say
[00:04:04] a solid model and i will say impressionistically if you download
[00:04:06] impressionistically if you download model 0 and play around with it you will
[00:04:08] model 0 and play around with it you will find that it is a very good sentiment
[00:04:10] find that it is a very good sentiment model indeed
[00:04:12] model indeed so we use model 0 to harvest what we
[00:04:14] so we use model 0 to harvest what we hope are challenging sentences and for
[00:04:16] hope are challenging sentences and for this we use the yelp academic data set
[00:04:18] this we use the yelp academic data set which is a very large collection about 8
[00:04:20] which is a very large collection about 8 million reviews and our heuristic is
[00:04:22] million reviews and our heuristic is that we're going to favor in our
[00:04:24] that we're going to favor in our sampling process harvesting sentences
[00:04:26] sampling process harvesting sentences where the review was one star
[00:04:28] where the review was one star so it's very low and model 0 predicted
[00:04:31] so it's very low and model 0 predicted positive for a given sentence and
[00:04:33] positive for a given sentence and conversely where the review is 5 stars
[00:04:36] conversely where the review is 5 stars and model 0 predicted negative
[00:04:38] and model 0 predicted negative we are hoping that that at least creates
[00:04:40] we are hoping that that at least creates a bias for sentences that are very
[00:04:41] a bias for sentences that are very challenging for model 0 where it's
[00:04:43] challenging for model 0 where it's actually making a wrong prediction we're
[00:04:45] actually making a wrong prediction we're not going to depend on that assumption
[00:04:46] not going to depend on that assumption because we'll have a validation step but
[00:04:48] because we'll have a validation step but we're hoping that this is a kind of as
[00:04:50] we're hoping that this is a kind of as adversarial as we can be without
[00:04:52] adversarial as we can be without actually having labels to begin
[00:04:55] actually having labels to begin this is a picture of the validation
[00:04:57] this is a picture of the validation interface you can see that um there were
[00:04:59] interface you can see that um there were some examples given in a little bit of
[00:05:01] some examples given in a little bit of training about how to use the labels and
[00:05:03] training about how to use the labels and then fundamentally what crowd workers
[00:05:05] then fundamentally what crowd workers did is they were prompted for a sentence
[00:05:06] did is they were prompted for a sentence and they made one of four choices
[00:05:08] and they made one of four choices positive negative no sentiment which is
[00:05:10] positive negative no sentiment which is our notion of neutral and mixed
[00:05:13] our notion of neutral and mixed sentiment which is indicating a sentence
[00:05:15] sentiment which is indicating a sentence that has a balance of positive and
[00:05:16] that has a balance of positive and negative sentiments expressed in it i
[00:05:18] negative sentiments expressed in it i think that's an important category to
[00:05:20] think that's an important category to single out we're not going to try to
[00:05:21] single out we're not going to try to model those sentences but we certainly
[00:05:23] model those sentences but we certainly want crowd workers to register that kind
[00:05:25] want crowd workers to register that kind of mixing of emotions
[00:05:28] of mixing of emotions where it appears
[00:05:30] where it appears so here's the resulting data set and
[00:05:32] so here's the resulting data set and because we got five gold labels for
[00:05:34] because we got five gold labels for every sentence there are two
[00:05:36] every sentence there are two perspectives that you can take the first
[00:05:38] perspectives that you can take the first one i've called distributional train and
[00:05:40] one i've called distributional train and this is where essentially we take each
[00:05:41] this is where essentially we take each one of the examples and reproduce it
[00:05:44] one of the examples and reproduce it five times for each of the labels that
[00:05:46] five times for each of the labels that it got so if an individual sentence got
[00:05:49] it got so if an individual sentence got three positive labels
[00:05:50] three positive labels two negative then we would have five
[00:05:53] two negative then we would have five examples three labeled positive and
[00:05:55] examples three labeled positive and three labeled negative with the actual
[00:05:56] three labeled negative with the actual text of the example repeated five times
[00:05:59] text of the example repeated five times what that is doing is essentially
[00:06:01] what that is doing is essentially simulating having a distribution over
[00:06:04] simulating having a distribution over the labels and for many classifier
[00:06:06] the labels and for many classifier models that is literally the same as
[00:06:08] models that is literally the same as training on the distribution of the
[00:06:09] training on the distribution of the labels as given by our crowd workers i
[00:06:11] labels as given by our crowd workers i think this is an exciting way to bring
[00:06:13] think this is an exciting way to bring in uncertainty
[00:06:15] in uncertainty and capture the fact that there might be
[00:06:17] and capture the fact that there might be kind of inherent disagreement among the
[00:06:19] kind of inherent disagreement among the crowd workers that we want our model to
[00:06:20] crowd workers that we want our model to at least grapple with
[00:06:22] at least grapple with and in the paper as we discuss these
[00:06:24] and in the paper as we discuss these this gives better models than training
[00:06:27] this gives better models than training on just the majority labels but you can
[00:06:29] on just the majority labels but you can take a more traditional view so majority
[00:06:31] take a more traditional view so majority label here means that at least three of
[00:06:33] label here means that at least three of the five workers chose that label uh
[00:06:36] the five workers chose that label uh that gives you 94 or 95 000 sentences
[00:06:39] that gives you 94 or 95 000 sentences per train and then these devon test sets
[00:06:41] per train and then these devon test sets have 3 600 examples each and presumably
[00:06:43] have 3 600 examples each and presumably we would predict just the majority label
[00:06:45] we would predict just the majority label for them
[00:06:46] for them what's more open is how we train these
[00:06:48] what's more open is how we train these systems
[00:06:50] systems and in the end what we found is that 47
[00:06:52] and in the end what we found is that 47 of these examples are adversarial with
[00:06:54] of these examples are adversarial with respect to model 0.
[00:06:56] respect to model 0. and as you'll see the dev and test set
[00:06:58] and as you'll see the dev and test set are designed so that model 0 performs a
[00:07:00] are designed so that model 0 performs a chance on them
[00:07:02] chance on them yeah that's some model zero versus the
[00:07:03] yeah that's some model zero versus the human so here's a summary of the
[00:07:05] human so here's a summary of the performance i showed you these
[00:07:07] performance i showed you these categories before and i'm just signaling
[00:07:09] categories before and i'm just signaling that we have by design ensure that model
[00:07:11] that we have by design ensure that model zero performs chance on round zero
[00:07:14] zero performs chance on round zero we could compare that to our human
[00:07:16] we could compare that to our human baseline for this we kind of synthesized
[00:07:19] baseline for this we kind of synthesized five annotators and did pairwise f1
[00:07:21] five annotators and did pairwise f1 scoring for them to get an estimate of
[00:07:23] scoring for them to get an estimate of human performance that is on the same
[00:07:25] human performance that is on the same scale as what we've got from model 0 up
[00:07:27] scale as what we've got from model 0 up here and we put that estimate at 88
[00:07:30] here and we put that estimate at 88 for the dev and test sets i think that's
[00:07:32] for the dev and test sets i think that's a good conservative number i think if
[00:07:34] a good conservative number i think if you got close to it that would be a
[00:07:36] you got close to it that would be a signal that we had kind of saturated
[00:07:37] signal that we had kind of saturated this round and would like to think about
[00:07:39] this round and would like to think about additional data set creation i do want
[00:07:41] additional data set creation i do want to signal though that i think this is a
[00:07:43] to signal though that i think this is a conservative estimate of how humans do
[00:07:45] conservative estimate of how humans do and one indicator of that is that
[00:07:47] and one indicator of that is that actually 614
[00:07:49] actually 614 of the roughly 1200 people who worked on
[00:07:51] of the roughly 1200 people who worked on this task for validation never disagreed
[00:07:54] this task for validation never disagreed with the majority label which sort of
[00:07:56] with the majority label which sort of starts to suggest that there are humans
[00:07:58] starts to suggest that there are humans who are performing perfectly at this
[00:08:00] who are performing perfectly at this task putting this at a pretty low bound
[00:08:03] task putting this at a pretty low bound and here are some example sentences
[00:08:04] and here are some example sentences these are fully randomly sampled with
[00:08:06] these are fully randomly sampled with the only bias being that i set a length
[00:08:08] the only bias being that i set a length restriction so that the slide would be
[00:08:10] restriction so that the slide would be manageable these are the same examples
[00:08:11] manageable these are the same examples that appear in the paper where we needed
[00:08:14] that appear in the paper where we needed to fit them all into a pretty small
[00:08:15] to fit them all into a pretty small table i think this is illuminating
[00:08:17] table i think this is illuminating though so it's showing all the different
[00:08:18] though so it's showing all the different ways that model 0 could get confused
[00:08:20] ways that model 0 could get confused with respect to the majority response
[00:08:23] with respect to the majority response and i would like to highlight for you
[00:08:24] and i would like to highlight for you that there is a real discrepancy here on
[00:08:26] that there is a real discrepancy here on the neutral category what we find is
[00:08:29] the neutral category what we find is that because model zero was trained on
[00:08:31] that because model zero was trained on large external benchmarks its notion of
[00:08:33] large external benchmarks its notion of neutral actually mixes together things
[00:08:36] neutral actually mixes together things that are mixed sentiment
[00:08:38] that are mixed sentiment and things that are highly uncertain
[00:08:39] and things that are highly uncertain about the sentiment that it's expressed
[00:08:41] about the sentiment that it's expressed for whatever reason so you get a lot of
[00:08:42] for whatever reason so you get a lot of borderline cases and a lot of cases
[00:08:45] borderline cases and a lot of cases where humans are kind of inherently
[00:08:47] where humans are kind of inherently having a hard time agreeing about what
[00:08:48] having a hard time agreeing about what the fixed sentiment label would be
[00:08:52] the fixed sentiment label would be i think that dynasty is doing a better
[00:08:54] i think that dynasty is doing a better job of capturing some notion of neutral
[00:08:56] job of capturing some notion of neutral in these labels over here then we should
[00:08:58] in these labels over here then we should be a little wary of treating three-star
[00:09:00] be a little wary of treating three-star reviews and things like that as a true
[00:09:02] reviews and things like that as a true proxy for neutrality
[00:09:05] proxy for neutrality um
[00:09:06] um this is a good point to signal that the
[00:09:08] this is a good point to signal that the validation and test sets
[00:09:10] validation and test sets for the bake off of the restaurant
[00:09:12] for the bake off of the restaurant sentences were validated in the same way
[00:09:15] sentences were validated in the same way as dinosaurs so those sentences will
[00:09:17] as dinosaurs so those sentences will have the same kind of neutrality
[00:09:19] have the same kind of neutrality that dynacent has which could be opposed
[00:09:22] that dynacent has which could be opposed to the sense of neutrality that you get
[00:09:24] to the sense of neutrality that you get from the stanford sentiment tree bank
[00:09:26] from the stanford sentiment tree bank which was of course underlyingly kind of
[00:09:28] which was of course underlyingly kind of gathered in the setting of having a
[00:09:30] gathered in the setting of having a fixed five-star rating scale
[00:09:34] so that's round one that's all naturally
[00:09:36] so that's round one that's all naturally occurring sentences let's turn to round
[00:09:38] occurring sentences let's turn to round two so recall that we benefit from round
[00:09:40] two so recall that we benefit from round one at this point by training a brand
[00:09:41] one at this point by training a brand new model on all those external data
[00:09:43] new model on all those external data sets plus the round one data set and
[00:09:46] sets plus the round one data set and then we have workers on dynabench
[00:09:48] then we have workers on dynabench interact with this model to try to fool
[00:09:50] interact with this model to try to fool it when we validate the resulting
[00:09:52] it when we validate the resulting sentences to get our round two data set
[00:09:55] sentences to get our round two data set so model one is again a roberta based
[00:09:56] so model one is again a roberta based classifier what we've done for our
[00:09:58] classifier what we've done for our training here is more or less carry over
[00:10:00] training here is more or less carry over what we did for the first round except
[00:10:02] what we did for the first round except we have up-sampled the sst to give it
[00:10:04] we have up-sampled the sst to give it more weight and we have dramatically
[00:10:07] more weight and we have dramatically up-sampled the distributional labels
[00:10:09] up-sampled the distributional labels from our round one data set effectively
[00:10:11] from our round one data set effectively trying to give it equal weight as all of
[00:10:13] trying to give it equal weight as all of these other data sets combined in the
[00:10:15] these other data sets combined in the training procedure so we're trying to
[00:10:17] training procedure so we're trying to get a model that
[00:10:18] get a model that as a priority does really well on our
[00:10:21] as a priority does really well on our round one data set
[00:10:23] round one data set here's a look at the performance of this
[00:10:25] here's a look at the performance of this model
[00:10:26] model and first i would just note that it's
[00:10:28] and first i would just note that it's doing well on round one right about 81
[00:10:31] doing well on round one right about 81 which is well below humans but certainly
[00:10:33] which is well below humans but certainly much better than the chance performance
[00:10:35] much better than the chance performance by design that we set up for model zero
[00:10:38] by design that we set up for model zero i do want to signal though that we have
[00:10:39] i do want to signal though that we have a kind of drop in performance for a few
[00:10:41] a kind of drop in performance for a few of these categories you can see that
[00:10:43] of these categories you can see that especially for yelp and amazon where
[00:10:45] especially for yelp and amazon where model 0 was at about for example 80 here
[00:10:48] model 0 was at about for example 80 here model one dropped down to 73 and it's a
[00:10:51] model one dropped down to 73 and it's a similar picture for dev and more or less
[00:10:53] similar picture for dev and more or less that's repeated for amazon with a drop
[00:10:55] that's repeated for amazon with a drop from about 76 to 73
[00:10:58] from about 76 to 73 and 77 to 73 similarly so we have a
[00:11:01] and 77 to 73 similarly so we have a trade-off in performance that i believe
[00:11:03] trade-off in performance that i believe traces to the fact that we are
[00:11:04] traces to the fact that we are performing some changes to the
[00:11:07] performing some changes to the underlying semantics of the labels
[00:11:09] underlying semantics of the labels but that's something to keep in mind and
[00:11:10] but that's something to keep in mind and you can see that there's a tension here
[00:11:12] you can see that there's a tension here as we try to do well at our data set
[00:11:15] as we try to do well at our data set versus continuing to do well on these
[00:11:17] versus continuing to do well on these fixed external benchmarks
[00:11:21] fixed external benchmarks here's the dynabench interface and
[00:11:22] here's the dynabench interface and there's one thing that i want to note
[00:11:24] there's one thing that i want to note about it this is the stock interface but
[00:11:25] about it this is the stock interface but we've actually concentrated on a
[00:11:27] we've actually concentrated on a condition that we called the prompt
[00:11:29] condition that we called the prompt condition where workers instead of
[00:11:31] condition where workers instead of having to just write a sentence as a
[00:11:32] having to just write a sentence as a blank slate you know sit down to an
[00:11:34] blank slate you know sit down to an empty buffer and try to fool the model
[00:11:36] empty buffer and try to fool the model they were given an inspirational prompt
[00:11:38] they were given an inspirational prompt which was in a tested sentence from the
[00:11:40] which was in a tested sentence from the yelp academic data set and invited to
[00:11:43] yelp academic data set and invited to modify that sentence if they chose in
[00:11:45] modify that sentence if they chose in order to achieve their goal of fooling
[00:11:47] order to achieve their goal of fooling the model in a particular way and this
[00:11:49] the model in a particular way and this proved to be vastly more productive it
[00:11:51] proved to be vastly more productive it led to more diverse and realistic
[00:11:53] led to more diverse and realistic sentences i think we'd essentially freed
[00:11:55] sentences i think we'd essentially freed the crowd workers from the creative
[00:11:57] the crowd workers from the creative burden of having each time to come up
[00:11:59] burden of having each time to come up with a completely new sentence and we're
[00:12:01] with a completely new sentence and we're hoping that this procedure leads to
[00:12:03] hoping that this procedure leads to fewer artifacts more diversity and more
[00:12:06] fewer artifacts more diversity and more realism for this adversarial data set
[00:12:09] realism for this adversarial data set collection procedure
[00:12:12] our validation pipeline was exactly the
[00:12:14] our validation pipeline was exactly the same as round one and here is the
[00:12:16] same as round one and here is the resulting data set it's a little bit
[00:12:17] resulting data set it's a little bit smaller because this kind of adversarial
[00:12:19] smaller because this kind of adversarial data set collection is hard and you can
[00:12:21] data set collection is hard and you can see how good model one is it was
[00:12:24] see how good model one is it was actually pretty hard for crowd workers
[00:12:25] actually pretty hard for crowd workers to fool this model they did so only
[00:12:27] to fool this model they did so only about 19 of the time
[00:12:29] about 19 of the time uh here's the data set for
[00:12:31] uh here's the data set for distributional training you have about
[00:12:32] distributional training you have about 93 000 sentences and if you go for the
[00:12:35] 93 000 sentences and if you go for the majority label training you have about
[00:12:37] majority label training you have about 19 000 and the devon test sets are
[00:12:39] 19 000 and the devon test sets are smaller but again the reason they're
[00:12:41] smaller but again the reason they're smaller is that they are designed to set
[00:12:43] smaller is that they are designed to set model one as having chance performance
[00:12:45] model one as having chance performance on this data set
[00:12:47] on this data set and so that's what i'll flesh out here
[00:12:49] and so that's what i'll flesh out here you can see that this model chance
[00:12:50] you can see that this model chance performance i showed you before that
[00:12:52] performance i showed you before that it's doing pretty well on round one and
[00:12:54] it's doing pretty well on round one and we had that kind of tension with the
[00:12:56] we had that kind of tension with the external benchmarks in terms of human
[00:12:58] external benchmarks in terms of human performance we're at about 90 using that
[00:13:01] performance we're at about 90 using that procedure of synthesized kind of
[00:13:03] procedure of synthesized kind of averaged f1 values and i would just note
[00:13:06] averaged f1 values and i would just note again that that's certainly conservative
[00:13:08] again that that's certainly conservative in that you know almost half of the
[00:13:10] in that you know almost half of the workers never disagreed with the
[00:13:12] workers never disagreed with the majority label so it is certainly within
[00:13:14] majority label so it is certainly within the capacity of individual humans to
[00:13:16] the capacity of individual humans to perform essentially perfectly on this
[00:13:18] perform essentially perfectly on this data set
[00:13:19] data set but 90 is nonetheless a good signpost
[00:13:21] but 90 is nonetheless a good signpost for us as we think about hill climbing
[00:13:23] for us as we think about hill climbing and launching subsequent rounds of
[00:13:25] and launching subsequent rounds of dynacent
[00:13:26] dynacent and here are some short examples and i
[00:13:28] and here are some short examples and i think they make the same point that our
[00:13:29] think they make the same point that our neutral category is more aligned with
[00:13:31] neutral category is more aligned with the semantics of what we mean when we
[00:13:32] the semantics of what we mean when we identify neutral sentences and less
[00:13:34] identify neutral sentences and less heterogeneous than you get from
[00:13:36] heterogeneous than you get from naturally occurring
[00:13:38] naturally occurring neutral sentences derived from star
[00:13:40] neutral sentences derived from star rating metadata and so forth so i'm
[00:13:42] rating metadata and so forth so i'm hopeful that this is a kind of positive
[00:13:44] hopeful that this is a kind of positive step toward getting true ternary
[00:13:45] step toward getting true ternary sentiment but we should be aware that
[00:13:47] sentiment but we should be aware that this label shift has happened in these
[00:13:49] this label shift has happened in these data sets
[00:13:51] data sets and the final thing i want to say is
[00:13:52] and the final thing i want to say is just to reiterate that if people do
[00:13:54] just to reiterate that if people do exciting work with this data set and
[00:13:56] exciting work with this data set and start to make real progress on the
[00:13:57] start to make real progress on the existing rounds that would be our cue to
[00:14:00] existing rounds that would be our cue to launch new rounds the dyna in dynacent
[00:14:03] launch new rounds the dyna in dynacent is that we would like to have an
[00:14:04] is that we would like to have an evolving benchmark not one that's static
[00:14:06] evolving benchmark not one that's static but rather responsive to progress that's
[00:14:08] but rather responsive to progress that's made in the field and the evolving needs
[00:14:10] made in the field and the evolving needs of people who are trying to develop
[00:14:12] of people who are trying to develop practical sentiment analysis systems so
[00:14:15] practical sentiment analysis systems so do let us know what kind of progress you
[00:14:17] do let us know what kind of progress you make and what you discover

sst.py | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=_T5q5fIfzww

---

Transcript

[00:00:05] hello everyone welcome to part 5 in our
[00:00:07] hello everyone welcome to part 5 in our series on supervised sentiment analysis
[00:00:09] series on supervised sentiment analysis the focus of this screencast is on the
[00:00:11] the focus of this screencast is on the module sst.pi which is included in the
[00:00:13] module sst.pi which is included in the course code distribution
[00:00:15] course code distribution it contains a bunch of tools that will
[00:00:17] it contains a bunch of tools that will let you work fluidly i hope with the
[00:00:19] let you work fluidly i hope with the stanford sentiment tree bank and conduct
[00:00:21] stanford sentiment tree bank and conduct a lot of experiments
[00:00:23] a lot of experiments in service of completing the homework
[00:00:25] in service of completing the homework and also doing an original system entry
[00:00:27] and also doing an original system entry for the bake off i'd say that my goals
[00:00:29] for the bake off i'd say that my goals for the screencast are twofold first i
[00:00:31] for the screencast are twofold first i do just want to get you acquainted with
[00:00:32] do just want to get you acquainted with this code so that you can work with it
[00:00:34] this code so that you can work with it on the assignment in the bake off uh and
[00:00:36] on the assignment in the bake off uh and in addition i guess i'd like to convey
[00:00:38] in addition i guess i'd like to convey to you some best practices around
[00:00:40] to you some best practices around setting up a code infrastructure for a
[00:00:43] setting up a code infrastructure for a project say that will let you run a lot
[00:00:45] project say that will let you run a lot of experiments and really explore the
[00:00:47] of experiments and really explore the space of ideas that you have
[00:00:49] space of ideas that you have without introducing a lot of bugs or
[00:00:50] without introducing a lot of bugs or writing a lot of extra code
[00:00:53] writing a lot of extra code so let's begin we'll start with these
[00:00:54] so let's begin we'll start with these reader functions at the top and the
[00:00:56] reader functions at the top and the first cell here i just load in not only
[00:00:58] first cell here i just load in not only os that we can find our files but also
[00:01:00] os that we can find our files but also sst which is the module of interest
[00:01:03] sst which is the module of interest we set up this variable here that's a
[00:01:05] we set up this variable here that's a pointer to where the data set itself
[00:01:07] pointer to where the data set itself lives
[00:01:08] lives and then this function
[00:01:09] and then this function sst.trainreader will let you load in a
[00:01:12] sst.trainreader will let you load in a pandas data frame that contains the
[00:01:14] pandas data frame that contains the train set for the sst you'll notice that
[00:01:16] train set for the sst you'll notice that there are two optional keywords include
[00:01:18] there are two optional keywords include subtrees and ddo
[00:01:20] subtrees and ddo dupe will remove repeated examples and
[00:01:22] dupe will remove repeated examples and include subtrees as a flag that will let
[00:01:24] include subtrees as a flag that will let you include or exclude all of the sub
[00:01:27] you include or exclude all of the sub trees that the sst contains by default
[00:01:29] trees that the sst contains by default we'll include just the full examples but
[00:01:32] we'll include just the full examples but if you set include subtrees equals true
[00:01:35] if you set include subtrees equals true you get a much larger data set as we
[00:01:37] you get a much larger data set as we discussed in the screencast on the sst
[00:01:39] discussed in the screencast on the sst itself
[00:01:40] itself in cell 4 here i'm just giving you a
[00:01:42] in cell 4 here i'm just giving you a look at one random record from this so
[00:01:45] look at one random record from this so remember it is a pandas data frame but
[00:01:47] remember it is a pandas data frame but we can get it as a dictionary for a
[00:01:48] we can get it as a dictionary for a little bit of an easier look we've got
[00:01:50] little bit of an easier look we've got an example id we have the text of the
[00:01:52] an example id we have the text of the sentence the label which is either
[00:01:54] sentence the label which is either negative positive or neutral and then is
[00:01:57] negative positive or neutral and then is subtree is a flag on whether or not it's
[00:01:59] subtree is a flag on whether or not it's a full root level example or a
[00:02:01] a full root level example or a subconstituent of such an example
[00:02:04] subconstituent of such an example since we have loaded this in with
[00:02:06] since we have loaded this in with include subtrees equals false we get
[00:02:08] include subtrees equals false we get this distribution of labels here this is
[00:02:10] this distribution of labels here this is just a distribution of labels on the
[00:02:12] just a distribution of labels on the full examples
[00:02:13] full examples but of course as we change these flags
[00:02:15] but of course as we change these flags we would get very different counts down
[00:02:16] we would get very different counts down here
[00:02:18] here and then something comparable happens
[00:02:19] and then something comparable happens with the dev reader dev df from
[00:02:22] with the dev reader dev df from sst.devreader with a pointer to the home
[00:02:24] sst.devreader with a pointer to the home directory for the data as before
[00:02:26] directory for the data as before and here the subtree distinction and the
[00:02:28] and here the subtree distinction and the dedupe distinction those are much less
[00:02:30] dedupe distinction those are much less important because these data sets
[00:02:32] important because these data sets consist just of root level examples and
[00:02:34] consist just of root level examples and there are very few if any duplicate
[00:02:36] there are very few if any duplicate examples in those data sets
[00:02:40] now let's turn to feature functions
[00:02:42] now let's turn to feature functions we'll begin to build up a framework for
[00:02:44] we'll begin to build up a framework for doing supervised uh sentiment analysis
[00:02:46] doing supervised uh sentiment analysis and the starting point here is what i
[00:02:48] and the starting point here is what i call the feature function it's given in
[00:02:50] call the feature function it's given in two unigrams phi it takes in a text that
[00:02:53] two unigrams phi it takes in a text that is a string and what it does is return a
[00:02:56] is a string and what it does is return a dictionary that is essentially a count
[00:02:58] dictionary that is essentially a count dictionary over the unigrams in that
[00:03:01] dictionary over the unigrams in that string
[00:03:01] string as given by this very simple
[00:03:03] as given by this very simple tokenization scheme which just down
[00:03:05] tokenization scheme which just down cases all of the tokens and then splits
[00:03:08] cases all of the tokens and then splits on white space
[00:03:09] on white space so as an example text if i have nlu is
[00:03:12] so as an example text if i have nlu is enlightening space and then an
[00:03:13] enlightening space and then an exclamation mark and i call the
[00:03:16] exclamation mark and i call the the feature function on that string i
[00:03:18] the feature function on that string i get this count dictionary here which is
[00:03:20] get this count dictionary here which is just giving the number of times each
[00:03:22] just giving the number of times each token appears in that string according
[00:03:24] token appears in that string according to the feature function
[00:03:26] to the feature function i'd say it's really important when
[00:03:27] i'd say it's really important when you're working with the standard version
[00:03:29] you're working with the standard version of this framework doing hand-built
[00:03:30] of this framework doing hand-built feature functions that you just abide by
[00:03:33] feature functions that you just abide by the contract that all of these feature
[00:03:35] the contract that all of these feature functions take in strings and return
[00:03:38] functions take in strings and return dictionaries mapping strings to their
[00:03:40] dictionaries mapping strings to their accounts or if you want to they're bools
[00:03:42] accounts or if you want to they're bools or floats or something that we can make
[00:03:44] or floats or something that we can make use of when we're doing featurization
[00:03:48] the next up here is what i've called a
[00:03:50] the next up here is what i've called a model wrapper and this is going to look
[00:03:51] model wrapper and this is going to look a little bit trivial here but as you'll
[00:03:53] a little bit trivial here but as you'll see as we move through more advanced
[00:03:55] see as we move through more advanced methods in this unit especially the next
[00:03:56] methods in this unit especially the next screencast it's really nice to have
[00:03:58] screencast it's really nice to have these wrappers around the normal
[00:04:01] these wrappers around the normal essentially the fit function down here
[00:04:04] essentially the fit function down here so i'm going to make use of a scikit
[00:04:05] so i'm going to make use of a scikit linear model called logistic regression
[00:04:07] linear model called logistic regression very standard side of cross entropy
[00:04:09] very standard side of cross entropy classifier i've called my function fit
[00:04:12] classifier i've called my function fit softmax classifier and it takes in a
[00:04:14] softmax classifier and it takes in a supervised data set so a feature matrix
[00:04:17] supervised data set so a feature matrix and a list of labels
[00:04:19] and a list of labels and i set up my model down here and i've
[00:04:21] and i set up my model down here and i've used some of the keyword parameters
[00:04:22] used some of the keyword parameters there are many more for the scikit model
[00:04:25] there are many more for the scikit model and then the crucial thing is that i
[00:04:26] and then the crucial thing is that i call fit and return the model which is
[00:04:28] call fit and return the model which is now a trained model trained on this data
[00:04:31] now a trained model trained on this data set x y
[00:04:32] set x y it might look like all i've done is call
[00:04:34] it might look like all i've done is call fit on a model that i set up but as
[00:04:36] fit on a model that i set up but as you'll see it's nice to have a wrapper
[00:04:38] you'll see it's nice to have a wrapper function so that we can potentially do a
[00:04:40] function so that we can potentially do a lot more as part of this particular step
[00:04:43] lot more as part of this particular step in our experimental workflow
[00:04:45] in our experimental workflow so now let's just bring all those things
[00:04:47] so now let's just bring all those things together into what is called sst
[00:04:49] together into what is called sst experiment which is like one stop
[00:04:51] experiment which is like one stop shopping for a complete experiment in
[00:04:53] shopping for a complete experiment in supervised sentiment analysis
[00:04:55] supervised sentiment analysis so we load in these two libraries we get
[00:04:57] so we load in these two libraries we get a pointer to our
[00:04:59] a pointer to our um data set and then call ssd experiment
[00:05:02] um data set and then call ssd experiment the first argument is the set that will
[00:05:04] the first argument is the set that will the data set that we'll be trained on so
[00:05:06] the data set that we'll be trained on so that's like train df from before
[00:05:08] that's like train df from before we have a feature function and a model
[00:05:10] we have a feature function and a model wrapper and then these other things are
[00:05:12] wrapper and then these other things are optional so if i leave assess data
[00:05:14] optional so if i leave assess data frames as none
[00:05:16] frames as none then it will do a random split on this
[00:05:18] then it will do a random split on this train reader according to train size if
[00:05:20] train reader according to train size if you do specify some data frames here a
[00:05:22] you do specify some data frames here a list of them and each one will be used
[00:05:24] list of them and each one will be used as a separate evaluation against the
[00:05:26] as a separate evaluation against the model that you train on this original
[00:05:28] model that you train on this original data
[00:05:29] data you can set the score function if you
[00:05:31] you can set the score function if you want our default is macro f1
[00:05:34] want our default is macro f1 and then we'll return to these two
[00:05:35] and then we'll return to these two options later verbose is just whether
[00:05:36] options later verbose is just whether you want to print some information and
[00:05:38] you want to print some information and vectorize is an option that you can turn
[00:05:40] vectorize is an option that you can turn on and off and you'll probably turn it
[00:05:42] on and off and you'll probably turn it off when you do deep learning
[00:05:44] off when you do deep learning experiments which we'll talk about later
[00:05:46] experiments which we'll talk about later in the unit
[00:05:47] in the unit the result of all that is a bunch of
[00:05:49] the result of all that is a bunch of information about your experiment stored
[00:05:51] information about your experiment stored in this variable and as because we had
[00:05:53] in this variable and as because we had verbose equals true you get a report
[00:05:55] verbose equals true you get a report here
[00:05:56] here and this is just a first chance to call
[00:05:58] and this is just a first chance to call out that throughout this course
[00:06:00] out that throughout this course essentially when we do classifier
[00:06:02] essentially when we do classifier experiments our primary metric is going
[00:06:04] experiments our primary metric is going to be the macro average f1 score this is
[00:06:07] to be the macro average f1 score this is useful for us because it gives equal
[00:06:09] useful for us because it gives equal weight to all the classes in our data
[00:06:11] weight to all the classes in our data regardless of their size which is
[00:06:14] regardless of their size which is typically reflecting our value that we
[00:06:15] typically reflecting our value that we care even about small classes we want to
[00:06:17] care even about small classes we want to do well even on the rare events in our
[00:06:19] do well even on the rare events in our space
[00:06:20] space and it's also perfectly balancing
[00:06:22] and it's also perfectly balancing precision and recall which is like a
[00:06:23] precision and recall which is like a good null hypothesis if we're not told
[00:06:26] good null hypothesis if we're not told ahead of time based on some other goal
[00:06:28] ahead of time based on some other goal whether we should favor precision or
[00:06:30] whether we should favor precision or recall
[00:06:31] recall so that all leads us to kind of favor as
[00:06:33] so that all leads us to kind of favor as a default this macro average f1 score is
[00:06:35] a default this macro average f1 score is an assessment of how the model did
[00:06:37] an assessment of how the model did here we've got 51.3
[00:06:41] the return value of sst.experiment as i
[00:06:44] the return value of sst.experiment as i said is a dictionary and it should
[00:06:45] said is a dictionary and it should package up for you all the objects and
[00:06:48] package up for you all the objects and information you would need to test the
[00:06:50] information you would need to test the model assess the model and do all kinds
[00:06:52] model assess the model and do all kinds of deep error analysis that is the
[00:06:54] of deep error analysis that is the philosophy here that you should if
[00:06:55] philosophy here that you should if possible capture as much information as
[00:06:58] possible capture as much information as you can about the experiment that you
[00:06:59] you can about the experiment that you ran
[00:07:00] ran in the service of being able to do
[00:07:01] in the service of being able to do subsequent downstream analysis of what
[00:07:04] subsequent downstream analysis of what happened so here i'm just giving an
[00:07:05] happened so here i'm just giving an example that we've got the model the
[00:07:07] example that we've got the model the feature function the trained data set
[00:07:09] feature function the trained data set whenever assessed data sets were used
[00:07:11] whenever assessed data sets were used and if that was a random split of the
[00:07:13] and if that was a random split of the train data that will be reflected in
[00:07:14] train data that will be reflected in these two variables the set of
[00:07:16] these two variables the set of predictions that you made about each one
[00:07:18] predictions that you made about each one of the assessed data sets the metrics
[00:07:20] of the assessed data sets the metrics you chose and the scores that you got
[00:07:23] you chose and the scores that you got and then if you do dive in like if you
[00:07:24] and then if you do dive in like if you look at train set it's a standard data
[00:07:27] look at train set it's a standard data set x is your future space y is your
[00:07:30] set x is your future space y is your labels vectorizer is something that i'll
[00:07:32] labels vectorizer is something that i'll return to that's an important part about
[00:07:34] return to that's an important part about how the internal workings of ssd
[00:07:36] how the internal workings of ssd experiment function and then you have
[00:07:38] experiment function and then you have the raw examples in case you need to do
[00:07:40] the raw examples in case you need to do some really serious human level error
[00:07:42] some really serious human level error analysis of the examples
[00:07:44] analysis of the examples as distinct from how they're represented
[00:07:46] as distinct from how they're represented in this feature space
[00:07:49] so here is just a slide that brings all
[00:07:51] so here is just a slide that brings all of those pieces together this is
[00:07:53] of those pieces together this is one-stop shopping for an entire
[00:07:55] one-stop shopping for an entire experiment we load in all our libraries
[00:07:57] experiment we load in all our libraries we have our pointer to the data and then
[00:07:59] we have our pointer to the data and then the ingredients are really a feature
[00:08:01] the ingredients are really a feature function and a model wrapper and that's
[00:08:04] function and a model wrapper and that's all you need in our default setting
[00:08:06] all you need in our default setting point it to the train data and it will
[00:08:08] point it to the train data and it will do its job and record all you would want
[00:08:10] do its job and record all you would want for this experiment i hope in this
[00:08:12] for this experiment i hope in this experiment variable here
[00:08:15] experiment variable here there's a final piece i want to return
[00:08:17] there's a final piece i want to return to that vectorizer variable that you saw
[00:08:19] to that vectorizer variable that you saw in the return values for sst experiment
[00:08:22] in the return values for sst experiment and that is making use of what inside
[00:08:24] and that is making use of what inside kit learn is called a dict vectorizer
[00:08:27] kit learn is called a dict vectorizer and this is really nice convenience
[00:08:28] and this is really nice convenience function for translating from human
[00:08:31] function for translating from human representations of your data into
[00:08:33] representations of your data into representations that machine learning
[00:08:35] representations that machine learning models like to consume
[00:08:37] models like to consume so let me just walk through this example
[00:08:38] so let me just walk through this example here i've loaded the
[00:08:40] here i've loaded the vectorizer and i've got my train
[00:08:42] vectorizer and i've got my train features here in the mode that i just
[00:08:44] features here in the mode that i just showed you where here we have two
[00:08:46] showed you where here we have two examples and each one is represented by
[00:08:48] examples and each one is represented by our feature function as a dictionary
[00:08:51] our feature function as a dictionary that maps like words into their accounts
[00:08:54] that maps like words into their accounts you can be more flexible than that but
[00:08:56] you can be more flexible than that but that's like the most basic case that we
[00:08:57] that's like the most basic case that we consider
[00:08:58] consider when i i set up my vectorizer in three
[00:09:02] when i i set up my vectorizer in three and then i call fit transform on this
[00:09:05] and then i call fit transform on this list of dictionaries
[00:09:06] list of dictionaries and the result here x train is a matrix
[00:09:10] and the result here x train is a matrix where each of the columns corresponds to
[00:09:12] where each of the columns corresponds to the keys in the dictionary representing
[00:09:15] the keys in the dictionary representing a unique feature
[00:09:16] a unique feature and the values are of course stored in
[00:09:18] and the values are of course stored in that in that column so this feature
[00:09:21] that in that column so this feature space here has been turned into um a
[00:09:24] space here has been turned into um a matrix that has two examples zero and
[00:09:26] matrix that has two examples zero and one there are a total of three features
[00:09:28] one there are a total of three features represented across our two examples a b
[00:09:31] represented across our two examples a b and c
[00:09:32] and c uh and you can see that the counts are
[00:09:34] uh and you can see that the counts are stored here so example zero has one for
[00:09:36] stored here so example zero has one for a
[00:09:37] a and zero for uh one for b and zero for c
[00:09:41] and zero for uh one for b and zero for c and example one has zero for a
[00:09:44] and example one has zero for a one for b and two for c
[00:09:47] one for b and two for c so that's recorded in the columns here
[00:09:49] so that's recorded in the columns here you can of course
[00:09:50] you can of course undertake this step by hand but it's a
[00:09:52] undertake this step by hand but it's a kind of error prone step and i'm just
[00:09:54] kind of error prone step and i'm just encouraging you to use dick vectorizer
[00:09:56] encouraging you to use dick vectorizer to handle it all and essentially map you
[00:09:59] to handle it all and essentially map you from this which is pretty human
[00:10:00] from this which is pretty human interpretable into this which is
[00:10:02] interpretable into this which is something your models like to consume
[00:10:06] something your models like to consume there's a next there's a second
[00:10:07] there's a next there's a second advantage here which is that if you use
[00:10:09] advantage here which is that if you use a dick vectorizer and you need to now do
[00:10:12] a dick vectorizer and you need to now do something at test time
[00:10:14] something at test time you can easily use your vectorizer to
[00:10:16] you can easily use your vectorizer to create feature spaces that are
[00:10:17] create feature spaces that are harmonized with what you saw in training
[00:10:19] harmonized with what you saw in training so as an example if my test features are
[00:10:22] so as an example if my test features are another pair of examples with a
[00:10:23] another pair of examples with a different character
[00:10:25] different character then i can call transform on the
[00:10:28] then i can call transform on the original train vectorizing from up here
[00:10:31] original train vectorizing from up here and it will translate that list of
[00:10:32] and it will translate that list of features
[00:10:33] features into a matrix now the important thing
[00:10:36] into a matrix now the important thing about what's happening here is that it's
[00:10:37] about what's happening here is that it's going to package the test features into
[00:10:40] going to package the test features into the original training space because of
[00:10:41] the original training space because of course those are the features that your
[00:10:43] course those are the features that your model recognizes those are the features
[00:10:45] model recognizes those are the features that you have weights for if you've
[00:10:47] that you have weights for if you've trained a model so it's important to
[00:10:48] trained a model so it's important to call transform at the space and as an
[00:10:51] call transform at the space and as an indication of one of the things that's
[00:10:52] indication of one of the things that's going to happen here is notice that in
[00:10:54] going to happen here is notice that in the test features my second example has
[00:10:56] the test features my second example has a brand new feature d
[00:10:58] a brand new feature d but d is not represented in the training
[00:11:01] but d is not represented in the training space we have no weights for it it's
[00:11:03] space we have no weights for it it's simply not part of that original
[00:11:05] simply not part of that original training data set
[00:11:06] training data set and so the result is that when we call
[00:11:08] and so the result is that when we call transform that feature is simply
[00:11:10] transform that feature is simply alighted
[00:11:11] alighted which is the desired behavior as we're
[00:11:13] which is the desired behavior as we're translating from training into testing
[00:11:16] translating from training into testing and it noticed that the dick vectorizer
[00:11:18] and it noticed that the dick vectorizer has simply handled that seamlessly for
[00:11:20] has simply handled that seamlessly for you
[00:11:21] you provided that you remember at the second
[00:11:23] provided that you remember at the second stage not to call fit transform that's
[00:11:25] stage not to call fit transform that's the number one gotcha for this interface
[00:11:28] the number one gotcha for this interface is that if you call fit transform a
[00:11:30] is that if you call fit transform a second time it will simply change the
[00:11:32] second time it will simply change the feature space into the one that is
[00:11:34] feature space into the one that is represented in your test features and
[00:11:36] represented in your test features and then everything will fall apart and your
[00:11:38] then everything will fall apart and your model
[00:11:39] model as trained from before will be unable to
[00:11:41] as trained from before will be unable to consume these new matrices that you've
[00:11:43] consume these new matrices that you've created but provided you remember that
[00:11:45] created but provided you remember that the rhythm is fit transform and then
[00:11:47] the rhythm is fit transform and then transform
[00:11:49] transform this should be really a nice set of
[00:11:50] this should be really a nice set of interfaces and of course this is what
[00:11:52] interfaces and of course this is what sst experiment is doing by default under
[00:11:55] sst experiment is doing by default under the hood for you

Hyperparameter Search | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=sO3gWU7y9Ws

---

Transcript

[00:00:04] hello everyone welcome to part six in
[00:00:06] hello everyone welcome to part six in our series on supervised sentiment
[00:00:07] our series on supervised sentiment analysis this screencast is going to
[00:00:09] analysis this screencast is going to cover two important methods in this
[00:00:11] cover two important methods in this space hyperparameter search and
[00:00:13] space hyperparameter search and classifier comparisons
[00:00:15] classifier comparisons so let's begin with hyper parameter
[00:00:17] so let's begin with hyper parameter search and first i'll just offer the
[00:00:19] search and first i'll just offer the rationale
[00:00:20] rationale let's say that the parameters of a model
[00:00:22] let's say that the parameters of a model are those whose values are learned as
[00:00:24] are those whose values are learned as part of optimizing the model itself so
[00:00:26] part of optimizing the model itself so for the classifiers we've been studying
[00:00:28] for the classifiers we've been studying the parameters are really just the
[00:00:30] the parameters are really just the weights that you learn on each of the
[00:00:32] weights that you learn on each of the individual features
[00:00:34] individual features and those are the things that are
[00:00:35] and those are the things that are directly targeted by the optimization
[00:00:37] directly targeted by the optimization process
[00:00:38] process the parameters of a model are typically
[00:00:40] the parameters of a model are typically pretty crisply defined because they kind
[00:00:42] pretty crisply defined because they kind of follow from the structure
[00:00:43] of follow from the structure mathematically of the model under
[00:00:45] mathematically of the model under investigation
[00:00:47] investigation much more diffuse are the hyper
[00:00:49] much more diffuse are the hyper parameters we can say the hyper
[00:00:50] parameters we can say the hyper parameters of a model are any settings
[00:00:52] parameters of a model are any settings that are outside of the optimization
[00:00:54] that are outside of the optimization process mentioned in one so examples
[00:00:57] process mentioned in one so examples from models we've seen our glove and lsa
[00:00:59] from models we've seen our glove and lsa have that dimensionality setting
[00:01:01] have that dimensionality setting the model itself gives you no guidance
[00:01:03] the model itself gives you no guidance about what to choose for the
[00:01:04] about what to choose for the dimensionality
[00:01:05] dimensionality and the dimensionality is not selected
[00:01:07] and the dimensionality is not selected as part of the optimization of the model
[00:01:09] as part of the optimization of the model itself you have to choose it via some
[00:01:12] itself you have to choose it via some external mechanism making it a hyper
[00:01:14] external mechanism making it a hyper parameter
[00:01:15] parameter and glove actually has two other
[00:01:17] and glove actually has two other additional prominent hyper parameters
[00:01:19] additional prominent hyper parameters xmax and alpha again those are not
[00:01:22] xmax and alpha again those are not optimized by the model you have to
[00:01:23] optimized by the model you have to select them via some external mechanism
[00:01:26] select them via some external mechanism and for the classifiers that we've been
[00:01:28] and for the classifiers that we've been studying you know we have regularization
[00:01:29] studying you know we have regularization terms those are classic hyper parameters
[00:01:32] terms those are classic hyper parameters if you have a deep classifier then the
[00:01:33] if you have a deep classifier then the hidden dimensionalities in the model
[00:01:35] hidden dimensionalities in the model could also be considered hyper
[00:01:37] could also be considered hyper parameters learning rates um you know
[00:01:39] parameters learning rates um you know any core feature of the optimization
[00:01:41] any core feature of the optimization method itself could be considered hyper
[00:01:43] method itself could be considered hyper parameters and even things that might be
[00:01:45] parameters and even things that might be considered kind of architectural like
[00:01:48] considered kind of architectural like the activation function in the deep
[00:01:49] the activation function in the deep classifier you might think of it as kind
[00:01:52] classifier you might think of it as kind of an intrinsic part of the model that
[00:01:54] of an intrinsic part of the model that you're evaluating but since it's an easy
[00:01:56] you're evaluating but since it's an easy choice point for us at this point you'll
[00:01:58] choice point for us at this point you'll be tempted to explore a few different
[00:02:00] be tempted to explore a few different options for that particular
[00:02:01] options for that particular architectural choice and in that way it
[00:02:04] architectural choice and in that way it could become a hyper parameter and at
[00:02:06] could become a hyper parameter and at this point even the optimization methods
[00:02:08] this point even the optimization methods could also emerge as a hyper parameter
[00:02:11] could also emerge as a hyper parameter that you would like to do search over
[00:02:14] that you would like to do search over and so forth and so on you should
[00:02:15] and so forth and so on you should probably take a fairly expansive view of
[00:02:18] probably take a fairly expansive view of what the hyper parameters of your model
[00:02:19] what the hyper parameters of your model are if you can now here's the crux of
[00:02:23] are if you can now here's the crux of the argument hyper parameter
[00:02:25] the argument hyper parameter optimization is crucial to building a
[00:02:27] optimization is crucial to building a persuasive argument fundamentally for
[00:02:29] persuasive argument fundamentally for any kind of comparison we make we want
[00:02:31] any kind of comparison we make we want to put every model in its very best
[00:02:34] to put every model in its very best light
[00:02:35] light we could take it for granted that for
[00:02:36] we could take it for granted that for any sufficiently complicated model
[00:02:38] any sufficiently complicated model there's some setting of its hyper
[00:02:40] there's some setting of its hyper parameters that's kind of degenerate and
[00:02:42] parameters that's kind of degenerate and would make the model look very bad
[00:02:44] would make the model look very bad and so you certainly wouldn't want to do
[00:02:46] and so you certainly wouldn't want to do any comparisons against that really
[00:02:48] any comparisons against that really problematic set of choices rather what
[00:02:50] problematic set of choices rather what we want to do is say let's put all the
[00:02:52] we want to do is say let's put all the models in their best light by choosing
[00:02:54] models in their best light by choosing optimal hyper parameters for them to the
[00:02:56] optimal hyper parameters for them to the best of our ability and then we can say
[00:02:58] best of our ability and then we can say that one model is better than the other
[00:03:00] that one model is better than the other if it emerges victorious in that very
[00:03:02] if it emerges victorious in that very rigorous
[00:03:04] rigorous setting
[00:03:05] setting and the final thing i'll say about this
[00:03:06] and the final thing i'll say about this methodologically is that of course all
[00:03:09] methodologically is that of course all hyperparameter tuning must be done only
[00:03:11] hyperparameter tuning must be done only on train and development data you
[00:03:13] on train and development data you consider you can consider that all fair
[00:03:16] consider you can consider that all fair game in terms of using it however you
[00:03:18] game in terms of using it however you want to choose the optimal hyper
[00:03:19] want to choose the optimal hyper parameters but once that choice is set
[00:03:22] parameters but once that choice is set it is fixed and those are the parameters
[00:03:24] it is fixed and those are the parameters that you use at test time and that is
[00:03:26] that you use at test time and that is the fundamental evaluation that you
[00:03:28] the fundamental evaluation that you would use for any kind of model
[00:03:29] would use for any kind of model comparison and at no point should you be
[00:03:32] comparison and at no point should you be tuning these hyper parameters on the
[00:03:34] tuning these hyper parameters on the test data itself that would be
[00:03:36] test data itself that would be completely illegitimate
[00:03:39] completely illegitimate i hope we've made it really easy to do
[00:03:41] i hope we've made it really easy to do this kind of hyper parameter search in
[00:03:42] this kind of hyper parameter search in the context of the work you're doing for
[00:03:44] the context of the work you're doing for supervised sentiment analysis here are
[00:03:46] supervised sentiment analysis here are some code snippets that show how that
[00:03:48] some code snippets that show how that can happen
[00:03:49] can happen i load in my libraries i have a pointer
[00:03:51] i load in my libraries i have a pointer to our sentiment data and here i have a
[00:03:53] to our sentiment data and here i have a fixed feature function which is just a
[00:03:55] fixed feature function which is just a unigram steamer feature function
[00:03:57] unigram steamer feature function the change happens inside the model
[00:04:00] the change happens inside the model wrapper
[00:04:01] wrapper whereas before essentially all we did is
[00:04:03] whereas before essentially all we did is set up a logistic regression model and
[00:04:05] set up a logistic regression model and then call its fit method
[00:04:07] then call its fit method here we set up that model but also
[00:04:09] here we set up that model but also established a grid of hyper parameters
[00:04:11] established a grid of hyper parameters these are different choice points for
[00:04:13] these are different choice points for this logistic regression model like
[00:04:15] this logistic regression model like whether or not i have a bias term
[00:04:17] whether or not i have a bias term the value of the regularization
[00:04:19] the value of the regularization parameter and even the algorithm used
[00:04:21] parameter and even the algorithm used for regularization itself l1 or l2
[00:04:25] for regularization itself l1 or l2 the model will explore the full grid of
[00:04:27] the model will explore the full grid of these options it's going to do five-fold
[00:04:29] these options it's going to do five-fold cross-validation so test one each five
[00:04:31] cross-validation so test one each five times on different splits of the data
[00:04:34] times on different splits of the data and in that very long search process it
[00:04:36] and in that very long search process it will find what it takes to be the best
[00:04:39] will find what it takes to be the best setting of all of these hyper parameters
[00:04:40] setting of all of these hyper parameters of all the combinations that can
[00:04:42] of all the combinations that can logically be
[00:04:44] logically be set
[00:04:45] set and that is the model that we finally
[00:04:47] and that is the model that we finally return here right so now you can see the
[00:04:49] return here right so now you can see the value of having a wrapper around these
[00:04:50] value of having a wrapper around these fit methods because then i could do all
[00:04:52] fit methods because then i could do all of this extra work without changing the
[00:04:55] of this extra work without changing the interface to sst experiment at all the
[00:04:57] interface to sst experiment at all the experiments look just as they did in the
[00:05:00] experiments look just as they did in the previous mode it's just that they will
[00:05:01] previous mode it's just that they will take a lot longer because you are
[00:05:03] take a lot longer because you are running dozens and dozens of experiments
[00:05:05] running dozens and dozens of experiments as part of this exhaustive search of all
[00:05:08] as part of this exhaustive search of all the possible settings
[00:05:10] the possible settings okay r2 is classifier comparison let's
[00:05:13] okay r2 is classifier comparison let's again begin with the rationale suppose
[00:05:15] again begin with the rationale suppose you've assessed a baseline model b
[00:05:17] you've assessed a baseline model b and your favorite model m and your
[00:05:19] and your favorite model m and your chosen assessment metric favors m right
[00:05:22] chosen assessment metric favors m right and this seems like a little victory for
[00:05:24] and this seems like a little victory for you but you should still ask yourself is
[00:05:26] you but you should still ask yourself is m really better right now if the
[00:05:29] m really better right now if the difference between b and m is clearly of
[00:05:31] difference between b and m is clearly of practical significance then you might
[00:05:33] practical significance then you might not need to do anything beyond
[00:05:34] not need to do anything beyond presenting the numbers right if each one
[00:05:37] presenting the numbers right if each one of your classification decisions
[00:05:38] of your classification decisions corresponds to something really
[00:05:40] corresponds to something really important in the world and your
[00:05:41] important in the world and your classifier makes thousands more good
[00:05:43] classifier makes thousands more good predictions than the other model that
[00:05:45] predictions than the other model that might be enough for the argument but
[00:05:48] might be enough for the argument but even in that situation you might ask
[00:05:50] even in that situation you might ask whether there's variation in how these
[00:05:51] whether there's variation in how these two models b and m perform did you just
[00:05:53] two models b and m perform did you just get lucky when you saw what looked like
[00:05:55] get lucky when you saw what looked like a practical difference and with minor
[00:05:57] a practical difference and with minor changes to the initialization of
[00:05:59] changes to the initialization of something you would see very different
[00:06:00] something you would see very different outcomes
[00:06:01] outcomes if the answer is possibly yes then you
[00:06:03] if the answer is possibly yes then you might still want to do some kind of
[00:06:05] might still want to do some kind of classifier comparison
[00:06:08] classifier comparison now there's this nice paper by dempshark
[00:06:10] now there's this nice paper by dempshark 2006 that advises using the wilcoxon
[00:06:12] 2006 that advises using the wilcoxon signed rank test for situations in which
[00:06:15] signed rank test for situations in which you can afford to repeatedly assess your
[00:06:17] you can afford to repeatedly assess your two models b m on different train test
[00:06:20] two models b m on different train test splits right and we'll talk and later in
[00:06:23] splits right and we'll talk and later in the term about the precise rationale for
[00:06:25] the term about the precise rationale for this but the idea is just that you would
[00:06:27] this but the idea is just that you would do a lot of experiments on slightly
[00:06:28] do a lot of experiments on slightly different views of your data and kind of
[00:06:31] different views of your data and kind of average across them to get a sense for
[00:06:33] average across them to get a sense for whether the two for how the two models
[00:06:35] whether the two for how the two models compare with each other
[00:06:38] compare with each other in situations where you can't repeatedly
[00:06:40] in situations where you can't repeatedly assess b m
[00:06:41] assess b m mars test is a reasonable alternative it
[00:06:45] mars test is a reasonable alternative it operates on the confusion matrices
[00:06:46] operates on the confusion matrices produced by the two models testing the
[00:06:48] produced by the two models testing the null hypothesis that the two models have
[00:06:50] null hypothesis that the two models have the same error rate
[00:06:53] the same error rate the reason you might opt for mcneemar's
[00:06:54] the reason you might opt for mcneemar's test is for example if you're doing a
[00:06:56] test is for example if you're doing a deep learning experiment where all the
[00:06:58] deep learning experiment where all the models take a few weeks to optimize then
[00:07:01] models take a few weeks to optimize then of course you can't probably afford to
[00:07:02] of course you can't probably afford to do dozens and dozens of experiments with
[00:07:05] do dozens and dozens of experiments with each one so you might be compelled to
[00:07:07] each one so you might be compelled to use mcnee mars based on one single run
[00:07:09] use mcnee mars based on one single run of the two models it's a much weaker
[00:07:11] of the two models it's a much weaker argument because of course precisely the
[00:07:13] argument because of course precisely the point is that we might see variation
[00:07:15] point is that we might see variation across different runs and mcnee mars is
[00:07:17] across different runs and mcnee mars is not really going to grapple with that in
[00:07:19] not really going to grapple with that in the way that the wilcoxon sign right
[00:07:21] the way that the wilcoxon sign right test will
[00:07:22] test will but this is arguably better than nothing
[00:07:24] but this is arguably better than nothing in most situations so you might default
[00:07:26] in most situations so you might default to mcnee mars
[00:07:27] to mcnee mars if the wilcoxon is too expensive
[00:07:30] if the wilcoxon is too expensive and let me just show you how easy this
[00:07:32] and let me just show you how easy this can be in the context of our code base
[00:07:34] can be in the context of our code base so my by way of illustration what we're
[00:07:36] so my by way of illustration what we're essentially going to do is compare
[00:07:38] essentially going to do is compare logistic regression and naive bayes i
[00:07:41] logistic regression and naive bayes i encourage you when you're doing these
[00:07:43] encourage you when you're doing these comparisons to have only one point of
[00:07:46] comparisons to have only one point of variation so we're going to fix the data
[00:07:48] variation so we're going to fix the data and we're going to fix the feature
[00:07:49] and we're going to fix the feature function and compare only the model
[00:07:52] function and compare only the model architectures
[00:07:53] architectures you could separately say i'm going to
[00:07:55] you could separately say i'm going to have a single fixed model like logistic
[00:07:57] have a single fixed model like logistic regression and explore a few different
[00:07:59] regression and explore a few different feature functions
[00:08:00] feature functions but i would advise against exploring two
[00:08:03] but i would advise against exploring two different feature functions as combined
[00:08:04] different feature functions as combined with two different models because when
[00:08:06] with two different models because when you observe differences in the end you
[00:08:08] you observe differences in the end you won't be sure whether that was caused by
[00:08:10] won't be sure whether that was caused by the model choice or by the feature
[00:08:12] the model choice or by the feature functions we want to kind of isolate
[00:08:14] functions we want to kind of isolate these things and do systematic
[00:08:16] these things and do systematic comparisons
[00:08:18] comparisons so here i'm going to do a systematic
[00:08:19] so here i'm going to do a systematic comparison of logistic regression and
[00:08:21] comparison of logistic regression and naive bayes on the sst using the
[00:08:24] naive bayes on the sst using the wilcoxon test and here's the setup the
[00:08:26] wilcoxon test and here's the setup the function is sst compare models
[00:08:29] function is sst compare models i point it to my training data
[00:08:31] i point it to my training data you can have two feature functions but
[00:08:33] you can have two feature functions but in that case you should have just one
[00:08:35] in that case you should have just one model wrapper here i've got one feature
[00:08:37] model wrapper here i've got one feature function used for both models and i'll
[00:08:39] function used for both models and i'll have these two different wrappers
[00:08:40] have these two different wrappers corresponding to the evaluation that i
[00:08:42] corresponding to the evaluation that i want to do of those two model classes
[00:08:45] want to do of those two model classes i'm going to use the wilcoxon as advised
[00:08:47] i'm going to use the wilcoxon as advised i'll do 10 trials of each
[00:08:50] i'll do 10 trials of each on the train size of 70 of the data and
[00:08:53] on the train size of 70 of the data and as always in this setting i'll use the
[00:08:54] as always in this setting i'll use the macro f1 as my score
[00:08:57] macro f1 as my score so what this will do internally is run
[00:08:59] so what this will do internally is run 10
[00:09:00] 10 10 experiments on different train test
[00:09:02] 10 experiments on different train test splits for each one of these models that
[00:09:04] splits for each one of these models that gives us a score vector
[00:09:06] gives us a score vector uh 10 you know 10 numbers for each model
[00:09:09] uh 10 you know 10 numbers for each model and then what the wilcoxon is doing is
[00:09:10] and then what the wilcoxon is doing is comparing whether the or make assessing
[00:09:12] comparing whether the or make assessing whether the means of those two score
[00:09:14] whether the means of those two score vectors are statistically significantly
[00:09:17] vectors are statistically significantly different and here it looks like we have
[00:09:19] different and here it looks like we have some evidence that we can reject the
[00:09:20] some evidence that we can reject the null hypothesis that these models are
[00:09:23] null hypothesis that these models are identical
[00:09:24] identical which is presumably the argument that we
[00:09:26] which is presumably the argument that we were trying to build
[00:09:28] were trying to build now of course that's very expensive
[00:09:29] now of course that's very expensive because we had to run 20 experiments in
[00:09:31] because we had to run 20 experiments in this situation and of course you could
[00:09:33] this situation and of course you could run many more if you were also doing
[00:09:35] run many more if you were also doing hyper parameter tuning as part of your
[00:09:37] hyper parameter tuning as part of your experimental workflow
[00:09:39] experimental workflow so in situations where you can't afford
[00:09:41] so in situations where you can't afford to do something that involves so many
[00:09:43] to do something that involves so many experiments as i said you could default
[00:09:45] experiments as i said you could default to make new mars
[00:09:47] to make new mars that is included in utils.mcneimar
[00:09:50] that is included in utils.mcneimar and the return values of ssd experiment
[00:09:53] and the return values of ssd experiment will give you all the information you
[00:09:54] will give you all the information you need essentially for mcneemars you need
[00:09:57] need essentially for mcneemars you need the actual gold vector of labels and
[00:09:59] the actual gold vector of labels and then the two vectors of predictions for
[00:10:01] then the two vectors of predictions for each one of your experiments so that's a
[00:10:03] each one of your experiments so that's a simple alternative in the situation in
[00:10:04] simple alternative in the situation in which wilcoxon was just too expensive

Feature Representation | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=L9ajfq6PJBI

---

Transcript

[00:00:04] welcome everyone this is part seven in
[00:00:06] welcome everyone this is part seven in our series on supervised sentiment
[00:00:08] our series on supervised sentiment analysis the focus of this screencast is
[00:00:10] analysis the focus of this screencast is on feature representation of data there
[00:00:12] on feature representation of data there really two things i'd like to do first
[00:00:14] really two things i'd like to do first just explore some ideas for effective
[00:00:16] just explore some ideas for effective feature representation in the context of
[00:00:18] feature representation in the context of sentiment analysis and second cover some
[00:00:20] sentiment analysis and second cover some of the core technical concepts that
[00:00:22] of the core technical concepts that surround feature representation that you
[00:00:24] surround feature representation that you do well to have in mind as you write new
[00:00:26] do well to have in mind as you write new feature functions and optimize
[00:00:28] feature functions and optimize models let's begin in a familiar place
[00:00:31] models let's begin in a familiar place which is engram feature functions to
[00:00:34] which is engram feature functions to this point in the series of screencasts
[00:00:35] this point in the series of screencasts i've been just focusing on unigram
[00:00:38] i've been just focusing on unigram feature functions that's also called the
[00:00:39] feature functions that's also called the bag of words model and we can easily
[00:00:42] bag of words model and we can easily generalize that idea to bi-grams and
[00:00:45] generalize that idea to bi-grams and trigrams and so forth
[00:00:47] trigrams and so forth all of these schemes will be heavily
[00:00:48] all of these schemes will be heavily dependent on the tokenizer that you've
[00:00:50] dependent on the tokenizer that you've chosen because of course in the end for
[00:00:51] chosen because of course in the end for every example we represent we are simply
[00:00:54] every example we represent we are simply tokenizing that example and then
[00:00:55] tokenizing that example and then counting the tokens in that example
[00:00:59] counting the tokens in that example this can be combined of course with
[00:01:01] this can be combined of course with pre-processing steps in part two in this
[00:01:03] pre-processing steps in part two in this series i covered the pre-processing idea
[00:01:05] series i covered the pre-processing idea of neg marking which is essentially to
[00:01:08] of neg marking which is essentially to mark words as they appear in a heuristic
[00:01:11] mark words as they appear in a heuristic way in the scope of negative morphemes
[00:01:13] way in the scope of negative morphemes as a way of indicating that for example
[00:01:16] as a way of indicating that for example good is positive in normal context but
[00:01:19] good is positive in normal context but might become negative when it is in the
[00:01:21] might become negative when it is in the scope of a negation like not or never
[00:01:24] scope of a negation like not or never we would handle that as a pre-processing
[00:01:26] we would handle that as a pre-processing step and that would just create more
[00:01:27] step and that would just create more unigrams that our tokenizer would turn
[00:01:30] unigrams that our tokenizer would turn into tokens and then would be counted by
[00:01:33] into tokens and then would be counted by these feature representation schemes
[00:01:37] these feature representation schemes a hallmark of these feature approaches
[00:01:39] a hallmark of these feature approaches is that they create very large very
[00:01:41] is that they create very large very sparse feature representations you are
[00:01:43] sparse feature representations you are going to have a column in your feature
[00:01:45] going to have a column in your feature representation for every single word
[00:01:47] representation for every single word that appears anywhere in your training
[00:01:49] that appears anywhere in your training data
[00:01:50] data and another important thing to keep in
[00:01:51] and another important thing to keep in mind about this approach is that by and
[00:01:53] mind about this approach is that by and large they will fail to directly model
[00:01:55] large they will fail to directly model relationships between features unless
[00:01:57] relationships between features unless you make some special effort to
[00:01:59] you make some special effort to effectively interact these features
[00:02:02] effectively interact these features all you'll be doing is studying their
[00:02:03] all you'll be doing is studying their distribution with respect to the class
[00:02:05] distribution with respect to the class labels that you have and it's very
[00:02:07] labels that you have and it's very unlikely that you'll recover in any deep
[00:02:10] unlikely that you'll recover in any deep way the kind of underlying synonymy of
[00:02:12] way the kind of underlying synonymy of words like couch and sofa for example
[00:02:15] words like couch and sofa for example and this is a shortcoming that we might
[00:02:17] and this is a shortcoming that we might want to address as we move into
[00:02:19] want to address as we move into distributed representations of examples
[00:02:21] distributed representations of examples and deep learning
[00:02:24] so for a first technical concept i would
[00:02:27] so for a first technical concept i would like to just distinguish between feature
[00:02:29] like to just distinguish between feature functions and features and to do this
[00:02:31] functions and features and to do this i've just got a fully worked out example
[00:02:33] i've just got a fully worked out example here using tools from scikit-learn that
[00:02:35] here using tools from scikit-learn that i think will make the importance of this
[00:02:37] i think will make the importance of this this distinction really clear and
[00:02:39] this distinction really clear and concrete so in cell 1 i've just loaded a
[00:02:41] concrete so in cell 1 i've just loaded a bunch of libraries in cell 2 i've got my
[00:02:44] bunch of libraries in cell 2 i've got my standard kind of lazy unigrams feature
[00:02:47] standard kind of lazy unigrams feature function which is taking in a string
[00:02:49] function which is taking in a string text down casing it and then simply
[00:02:51] text down casing it and then simply splitting on white space and then the
[00:02:53] splitting on white space and then the counter here is just turning that into
[00:02:55] counter here is just turning that into account dictionary mapping each token to
[00:02:57] account dictionary mapping each token to the number of times it appears in this
[00:02:59] the number of times it appears in this example according to our tokenizer
[00:03:02] example according to our tokenizer that'll be fine for now in cell 3 i have
[00:03:04] that'll be fine for now in cell 3 i have a tiny little corpus that has just two
[00:03:07] a tiny little corpus that has just two words a and b
[00:03:09] words a and b in cell 4 i create a list of
[00:03:11] in cell 4 i create a list of dictionaries by calling unigram spy on
[00:03:14] dictionaries by calling unigram spy on each of the texts in my corpus here so
[00:03:16] each of the texts in my corpus here so that gives me a list of count
[00:03:18] that gives me a list of count dictionaries
[00:03:20] dictionaries in five i use a dict vectorizer as
[00:03:22] in five i use a dict vectorizer as covered in a previous screencast and
[00:03:24] covered in a previous screencast and what that's going to do is when i call
[00:03:25] what that's going to do is when i call fit transform on my list of feature
[00:03:28] fit transform on my list of feature dictionaries it will turn it into a
[00:03:30] dictionaries it will turn it into a matrix which is the input that all of
[00:03:32] matrix which is the input that all of these scikit machine learning models
[00:03:34] these scikit machine learning models expect for their training data and in
[00:03:36] expect for their training data and in cell 7 i've just given you what i hope
[00:03:38] cell 7 i've just given you what i hope is a pretty intuitive view of that
[00:03:40] is a pretty intuitive view of that design matrix underlyingly it's just a
[00:03:43] design matrix underlyingly it's just a numpy array but if we use pandas we can
[00:03:45] numpy array but if we use pandas we can see that the columns here correspond to
[00:03:48] see that the columns here correspond to the names of each one of the features
[00:03:50] the names of each one of the features because we have just two word types in
[00:03:52] because we have just two word types in our corpus there are two columns a and b
[00:03:54] our corpus there are two columns a and b and each of the rows corresponds to an
[00:03:56] and each of the rows corresponds to an example from our corpus and so you can
[00:03:58] example from our corpus and so you can see that our first example has been
[00:04:00] see that our first example has been reduced to a representation that has
[00:04:02] reduced to a representation that has three in its first dimension and zero in
[00:04:04] three in its first dimension and zero in its second corresponding to the fact
[00:04:06] its second corresponding to the fact that it has three a's and no b's
[00:04:08] that it has three a's and no b's example two a a b is represented as a
[00:04:11] example two a a b is represented as a two in the first column and a one in the
[00:04:13] two in the first column and a one in the second column
[00:04:14] second column and so forth so that's a first
[00:04:16] and so forth so that's a first distinction we have this feature
[00:04:18] distinction we have this feature function here which is like a factory
[00:04:20] function here which is like a factory and depending on the data that come in
[00:04:22] and depending on the data that come in for our corpus we're going to get very
[00:04:24] for our corpus we're going to get very different features which correspond to
[00:04:26] different features which correspond to each one of the columns in this feature
[00:04:29] each one of the columns in this feature representation matrix
[00:04:31] representation matrix let's continue this a little bit and
[00:04:33] let's continue this a little bit and think about how this actually interacts
[00:04:34] think about how this actually interacts with the optimization process so in cell
[00:04:37] with the optimization process so in cell 7 here i've just repeated that previous
[00:04:39] 7 here i've just repeated that previous matrix for reference
[00:04:41] matrix for reference in cell 8 i have the class labels for
[00:04:43] in cell 8 i have the class labels for our four examples and you can see there
[00:04:45] our four examples and you can see there are three distinct classes c1 c2 and c3
[00:04:49] are three distinct classes c1 c2 and c3 i set up a logistic regression model
[00:04:50] i set up a logistic regression model although that's not especially important
[00:04:52] although that's not especially important it's just a useful illustration
[00:04:54] it's just a useful illustration and i call fit on my pair xy that is my
[00:04:57] and i call fit on my pair xy that is my feature representations and my labels
[00:04:59] feature representations and my labels and that's the optimization process as
[00:05:02] and that's the optimization process as part of that and for a convention for
[00:05:04] part of that and for a convention for scikit the optimization process creates
[00:05:06] scikit the optimization process creates this new attribute co-f underscore and
[00:05:08] this new attribute co-f underscore and this new attribute classes underscore
[00:05:11] this new attribute classes underscore these co co-f here these are the weights
[00:05:13] these co co-f here these are the weights that we learned as part of the
[00:05:15] that we learned as part of the optimization process and of course
[00:05:17] optimization process and of course classes corresponds to the classes that
[00:05:19] classes corresponds to the classes that inferred from the label y that we input
[00:05:22] inferred from the label y that we input and here i'm just using a panda's data
[00:05:23] and here i'm just using a panda's data frame again to try to make this
[00:05:25] frame again to try to make this intuitive it's really just a numpy array
[00:05:27] intuitive it's really just a numpy array this co-f object here but you can see
[00:05:29] this co-f object here but you can see that the resulting matrix has a row for
[00:05:32] that the resulting matrix has a row for each one of our classes and a column for
[00:05:35] each one of our classes and a column for each one of our features
[00:05:37] each one of our features and that's a useful reminder that what
[00:05:39] and that's a useful reminder that what the optimization process for models like
[00:05:41] the optimization process for models like this is actually doing
[00:05:42] this is actually doing is learning a weight that associates
[00:05:45] is learning a weight that associates class feature name pairs
[00:05:47] class feature name pairs with a weight right so it's not just
[00:05:49] with a weight right so it's not just that we learn individual weights for
[00:05:50] that we learn individual weights for features but rather we learn them with
[00:05:52] features but rather we learn them with respect to each one of the classes and
[00:05:54] respect to each one of the classes and that's a hallmark of optimization for
[00:05:57] that's a hallmark of optimization for multi-class models like this one
[00:05:59] multi-class models like this one and then in cell 12 i've just shown you
[00:06:01] and then in cell 12 i've just shown you that you can actually use the co-f and
[00:06:03] that you can actually use the co-f and this other bias term inter-score
[00:06:05] this other bias term inter-score intercept underscore
[00:06:07] intercept underscore to recreate the predictions of the model
[00:06:09] to recreate the predictions of the model all you're doing is multiplying examples
[00:06:11] all you're doing is multiplying examples by those coefficients and adding in the
[00:06:14] by those coefficients and adding in the bias term and this matrix here is
[00:06:16] bias term and this matrix here is identical to what you get inside if you
[00:06:18] identical to what you get inside if you simply directly call predict prabha for
[00:06:21] simply directly call predict prabha for predict probabilities on your examples
[00:06:28] let's turn back to what we're trying to
[00:06:29] let's turn back to what we're trying to do to create good models having those
[00:06:31] do to create good models having those ideas in mind so let's just cover a few
[00:06:33] ideas in mind so let's just cover a few other ideas for hand-built feature
[00:06:35] other ideas for hand-built feature functions that i think could be
[00:06:36] functions that i think could be effective for sentiment so of course
[00:06:38] effective for sentiment so of course we'd have lexicon derived features i
[00:06:40] we'd have lexicon derived features i earlier showed you a bunch of different
[00:06:41] earlier showed you a bunch of different lexicons and that could be used to group
[00:06:44] lexicons and that could be used to group our unigrams so we could have these
[00:06:45] our unigrams so we could have these feature functions work in conjunction
[00:06:47] feature functions work in conjunction with a bag of words or bag of engrams
[00:06:49] with a bag of words or bag of engrams model or we could use them to replace
[00:06:52] model or we could use them to replace that model and develop a sparser feature
[00:06:54] that model and develop a sparser feature representation space
[00:06:56] representation space we could also do the negation marking
[00:06:58] we could also do the negation marking that i mentioned before and we could
[00:07:00] that i mentioned before and we could generalize that idea so many things in
[00:07:02] generalize that idea so many things in language take scope in a way that will
[00:07:04] language take scope in a way that will affect the semantics of words that are
[00:07:06] affect the semantics of words that are in their scope so another classical
[00:07:08] in their scope so another classical example behind besides negation is these
[00:07:10] example behind besides negation is these modal adverbs like quite possibly or
[00:07:13] modal adverbs like quite possibly or totally we might have the idea that they
[00:07:15] totally we might have the idea that they are modulating the extent to which the
[00:07:17] are modulating the extent to which the speaker is committed to masterpiece or
[00:07:19] speaker is committed to masterpiece or amazing in this case and keeping track
[00:07:21] amazing in this case and keeping track of that semantic association with simple
[00:07:23] of that semantic association with simple underscore marking of some kind might be
[00:07:26] underscore marking of some kind might be useful for giving our model a chance to
[00:07:28] useful for giving our model a chance to see that these unigrams are different
[00:07:30] see that these unigrams are different depending on their environment
[00:07:33] depending on their environment we could also have length based features
[00:07:34] we could also have length based features and that's just a useful reminder that
[00:07:36] and that's just a useful reminder that these don't all have to be count
[00:07:37] these don't all have to be count features we could have real valued
[00:07:39] features we could have real valued features of various kinds and they could
[00:07:41] features of various kinds and they could signal something important about the
[00:07:43] signal something important about the class label like for example i think
[00:07:46] class label like for example i think neutral reviews three star reviews tend
[00:07:48] neutral reviews three star reviews tend to be longer than one in five star
[00:07:50] to be longer than one in five star reviews so might as well throw that in
[00:07:53] reviews so might as well throw that in and we can expand that idea of float
[00:07:55] and we can expand that idea of float value features a little bit more i like
[00:07:56] value features a little bit more i like the idea of thwarted expectations which
[00:07:58] the idea of thwarted expectations which you might keep track of as the ratio of
[00:08:01] you might keep track of as the ratio of positive to negative words in a sentence
[00:08:04] positive to negative words in a sentence uh the idea being that very often if
[00:08:06] uh the idea being that very often if that ratio is exaggerated it's telling
[00:08:09] that ratio is exaggerated it's telling you the opposite story that you might
[00:08:11] you the opposite story that you might expect about the overall sentiment many
[00:08:13] expect about the overall sentiment many many positive words stacked up together
[00:08:15] many positive words stacked up together might actually be preparing you for a
[00:08:17] might actually be preparing you for a negative assessment and the reverse
[00:08:20] negative assessment and the reverse but the important thing about this
[00:08:22] but the important thing about this feature is that it wouldn't decide for
[00:08:24] feature is that it wouldn't decide for you what these ratios mean you would
[00:08:25] you what these ratios mean you would just hope that it was a useful signal
[00:08:27] just hope that it was a useful signal that your model might pick up on as part
[00:08:29] that your model might pick up on as part of optimization to figure out how to
[00:08:31] of optimization to figure out how to make use of the information
[00:08:34] make use of the information and then finally we could do things
[00:08:35] and then finally we could do things various kinds of ad hoc feature
[00:08:37] various kinds of ad hoc feature functions to try to capture the fact
[00:08:39] functions to try to capture the fact that many uses of language are
[00:08:41] that many uses of language are non-literal and might be signaling
[00:08:43] non-literal and might be signaling exactly the opposite of what they seem
[00:08:45] exactly the opposite of what they seem to do on their surface like not exactly
[00:08:47] to do on their surface like not exactly a masterpiece is probably a pretty
[00:08:50] a masterpiece is probably a pretty negative review
[00:08:51] negative review it was like 50 hours long is not saying
[00:08:53] it was like 50 hours long is not saying that it was actually 50 hours long but
[00:08:55] that it was actually 50 hours long but rather with hyperbole indicating that it
[00:08:57] rather with hyperbole indicating that it was much too long or something like that
[00:09:00] was much too long or something like that and the best movie in the history of the
[00:09:01] and the best movie in the history of the universe could be a
[00:09:03] universe could be a ringing endorsement but it could just as
[00:09:05] ringing endorsement but it could just as easily be a bit of sarcasm
[00:09:08] easily be a bit of sarcasm capturing those kind of subtle
[00:09:09] capturing those kind of subtle distinctions is of course much more
[00:09:11] distinctions is of course much more difficult but the handle feature
[00:09:13] difficult but the handle feature functions that you write where you could
[00:09:14] functions that you write where you could try to capture it and if they have a
[00:09:16] try to capture it and if they have a positive effect then
[00:09:18] positive effect then maybe you've made some real progress
[00:09:20] maybe you've made some real progress and that's a good transition point to
[00:09:22] and that's a good transition point to this topic of assessing individual
[00:09:23] this topic of assessing individual feature functions as you can see the
[00:09:25] feature functions as you can see the philosophy in this mode of work is that
[00:09:27] philosophy in this mode of work is that you write lots of feature functions and
[00:09:29] you write lots of feature functions and kind of see how well they can do at
[00:09:32] kind of see how well they can do at improving your model overall
[00:09:34] improving your model overall you might end up with a very large model
[00:09:36] you might end up with a very large model with many correlated features and that
[00:09:38] with many correlated features and that might lead you to want to do some
[00:09:39] might lead you to want to do some feature selection to weed out the ones
[00:09:41] feature selection to weed out the ones that are not contributing in a positive
[00:09:43] that are not contributing in a positive way
[00:09:44] way now scikit-learn has a whole library for
[00:09:46] now scikit-learn has a whole library for doing this called feature selection and
[00:09:48] doing this called feature selection and it offers lots of functions that will
[00:09:50] it offers lots of functions that will let you assess how much information your
[00:09:52] let you assess how much information your feature functions contain with respect
[00:09:54] feature functions contain with respect to the labels for your classification
[00:09:56] to the labels for your classification problem so this is very powerful and i
[00:09:58] problem so this is very powerful and i encourage you to use them but you should
[00:10:00] encourage you to use them but you should be a little bit cautious
[00:10:02] be a little bit cautious take care when assessing feature
[00:10:04] take care when assessing feature functions individually because
[00:10:06] functions individually because correlations between those features will
[00:10:09] correlations between those features will make the assessments very hard to
[00:10:11] make the assessments very hard to interpret
[00:10:12] interpret the problem here is that your model is
[00:10:14] the problem here is that your model is holistically thinking about how all
[00:10:16] holistically thinking about how all these features relate to your class
[00:10:17] these features relate to your class label and figuring out how to optimize
[00:10:19] label and figuring out how to optimize weights on that basis whereas the
[00:10:21] weights on that basis whereas the feature function methods many of them
[00:10:23] feature function methods many of them just look at individual features and how
[00:10:25] just look at individual features and how they relate to the class labels so
[00:10:26] they relate to the class labels so you're losing all that correlational
[00:10:28] you're losing all that correlational context
[00:10:29] context and to make that a little concrete i
[00:10:31] and to make that a little concrete i just cooked up an example here an
[00:10:32] just cooked up an example here an idealized one that shows how you could
[00:10:35] idealized one that shows how you could be misled so i have three features x1 x2
[00:10:38] be misled so i have three features x1 x2 and x3 and a simple binary
[00:10:40] and x3 and a simple binary classification problem and i use the
[00:10:42] classification problem and i use the chi-square test from feature selection
[00:10:44] chi-square test from feature selection to kind of assess how important each one
[00:10:46] to kind of assess how important each one of these features is with respect to
[00:10:48] of these features is with respect to this classification problem
[00:10:50] this classification problem and what i found is that intuitively it
[00:10:52] and what i found is that intuitively it looks like x1 and x2 are really powerful
[00:10:55] looks like x1 and x2 are really powerful features
[00:10:56] features and that might lead me to think well
[00:10:58] and that might lead me to think well i'll drop the third feature and include
[00:11:00] i'll drop the third feature and include just one and two in my model
[00:11:03] just one and two in my model so far so good however if we thoroughly
[00:11:06] so far so good however if we thoroughly explore this space what we find is that
[00:11:08] explore this space what we find is that in truth a simple linear model performs
[00:11:10] in truth a simple linear model performs best with just feature x1 and actually
[00:11:13] best with just feature x1 and actually including x2 hurts the model despite the
[00:11:16] including x2 hurts the model despite the fact that it has this positive feature
[00:11:18] fact that it has this positive feature importance value so what we really ought
[00:11:20] importance value so what we really ought to be doing is using just this single
[00:11:22] to be doing is using just this single feature but these methods can't tell us
[00:11:24] feature but these methods can't tell us that and even a positive
[00:11:27] that and even a positive feature selection value might actually
[00:11:29] feature selection value might actually be something that's at odds with what
[00:11:31] be something that's at odds with what we're trying to do with our model as
[00:11:33] we're trying to do with our model as this example shows
[00:11:35] this example shows so ideally what you would do is consider
[00:11:38] so ideally what you would do is consider more holistic assessment methods which
[00:11:39] more holistic assessment methods which scikit also offers this would be things
[00:11:42] scikit also offers this would be things like systematically removing or
[00:11:44] like systematically removing or perturbing feature values in the context
[00:11:46] perturbing feature values in the context of the full model that you're optimizing
[00:11:49] of the full model that you're optimizing and comparing performance across those
[00:11:51] and comparing performance across those models this is much more expensive
[00:11:53] models this is much more expensive because you're optimizing many many
[00:11:54] because you're optimizing many many models so it might be prohibitive for
[00:11:56] models so it might be prohibitive for some classes of models that you're
[00:11:58] some classes of models that you're exploring but if you can do it this will
[00:12:00] exploring but if you can do it this will be more reliable
[00:12:02] be more reliable however if this is impossible it might
[00:12:04] however if this is impossible it might still be productive to do some feature
[00:12:06] still be productive to do some feature selection using simpler methods you
[00:12:09] selection using simpler methods you should just be aware that you might be
[00:12:11] should just be aware that you might be doing something that's not optimal for
[00:12:13] doing something that's not optimal for the actual optimization problem that
[00:12:15] the actual optimization problem that you've posed
[00:12:18] okay in the final section of this
[00:12:19] okay in the final section of this screencast is a kind of transition into
[00:12:21] screencast is a kind of transition into the world of deep learning i've called
[00:12:23] the world of deep learning i've called this distributed representations as
[00:12:25] this distributed representations as features this is a very different mode
[00:12:27] features this is a very different mode for thinking about representing examples
[00:12:30] for thinking about representing examples what we do in this case is take our
[00:12:32] what we do in this case is take our token stream as before but instead of
[00:12:34] token stream as before but instead of writing a lot of hand-built feature
[00:12:36] writing a lot of hand-built feature functions we simply look up each one of
[00:12:38] functions we simply look up each one of those tokens in some embedding that we
[00:12:40] those tokens in some embedding that we have for example it could be an
[00:12:42] have for example it could be an embedding that you created in the first
[00:12:43] embedding that you created in the first unit of this course
[00:12:45] unit of this course or it could be a glove embedding or a
[00:12:47] or it could be a glove embedding or a static embedding that you derived from
[00:12:49] static embedding that you derived from vert representations and so forth and so
[00:12:51] vert representations and so forth and so on the important thing is that each
[00:12:53] on the important thing is that each token is now represented by a vector
[00:12:56] token is now represented by a vector and that could be a powerful idea
[00:12:57] and that could be a powerful idea because in representing each of these
[00:12:59] because in representing each of these words as vectors we are now
[00:13:02] words as vectors we are now capturing the relationships between
[00:13:04] capturing the relationships between those tokens we might now have a hope of
[00:13:06] those tokens we might now have a hope of seeing that sofa and couch are actually
[00:13:09] seeing that sofa and couch are actually similar features in general and not just
[00:13:11] similar features in general and not just with respect to the class labels that we
[00:13:13] with respect to the class labels that we have
[00:13:15] have so that's the idea why this might be
[00:13:16] so that's the idea why this might be powerful so we take all those vectors
[00:13:18] powerful so we take all those vectors and look them up however for all these
[00:13:19] and look them up however for all these classifier models we need a fixed
[00:13:21] classifier models we need a fixed dimensional representation to feed into
[00:13:23] dimensional representation to feed into the actual classifier unit so we're
[00:13:25] the actual classifier unit so we're going to have to combine those vectors
[00:13:27] going to have to combine those vectors in some way and the simplest thing you
[00:13:28] in some way and the simplest thing you could do is combine them via some
[00:13:30] could do is combine them via some function like sum or mean right
[00:13:33] function like sum or mean right so i would take all these things for
[00:13:34] so i would take all these things for example and take their average and that
[00:13:36] example and take their average and that would give me another fixed dimensional
[00:13:37] would give me another fixed dimensional representation no matter how many tokens
[00:13:40] representation no matter how many tokens are in each one of the examples
[00:13:42] are in each one of the examples and that average vector would be the
[00:13:44] and that average vector would be the input to the classifier
[00:13:47] input to the classifier so if each one of these vectors has
[00:13:48] so if each one of these vectors has dimension 300 then so too does the
[00:13:51] dimension 300 then so too does the feature representation of my entire
[00:13:53] feature representation of my entire example and now i now have a classifier
[00:13:56] example and now i now have a classifier which which is processing feature
[00:13:57] which which is processing feature representations that have 300 columns
[00:14:00] representations that have 300 columns each dimension in the underlying
[00:14:02] each dimension in the underlying embedding space now corresponds to a
[00:14:04] embedding space now corresponds to a feature
[00:14:05] feature and that's the basis for optimization
[00:14:07] and that's the basis for optimization and i'd say an eye-opening thing about
[00:14:08] and i'd say an eye-opening thing about this class of models is despite them
[00:14:10] this class of models is despite them being very compact you know 300
[00:14:13] being very compact you know 300 dimensions versus 20 000 that you might
[00:14:15] dimensions versus 20 000 that you might have from a bag of words model they turn
[00:14:17] have from a bag of words model they turn out to be very powerful
[00:14:20] out to be very powerful and this final slide here just shows you
[00:14:22] and this final slide here just shows you how to implement those using tools and
[00:14:24] how to implement those using tools and other utilities for our course
[00:14:26] other utilities for our course so i'm going to use glove and i'm going
[00:14:28] so i'm going to use glove and i'm going to use the 300 dimensional glove space
[00:14:30] to use the 300 dimensional glove space which is included in your data
[00:14:31] which is included in your data distribution
[00:14:32] distribution in four and five here we just write
[00:14:34] in four and five here we just write simple feature functions and the
[00:14:35] simple feature functions and the hallmark of these is that they are
[00:14:37] hallmark of these is that they are simply looking up words in the embedding
[00:14:39] simply looking up words in the embedding and then combining them via whatever
[00:14:41] and then combining them via whatever function the user specifies so the
[00:14:43] function the user specifies so the output of this is directly a vector
[00:14:45] output of this is directly a vector representation of each example
[00:14:48] representation of each example in cell six we set up a logistic
[00:14:50] in cell six we set up a logistic regression as before of course it could
[00:14:52] regression as before of course it could be a much fancier model but logistic
[00:14:54] be a much fancier model but logistic regression will do
[00:14:55] regression will do and then we use ssd experiment almost
[00:14:57] and then we use ssd experiment almost exactly as before the one change we need
[00:15:00] exactly as before the one change we need to remember to make when operating in
[00:15:02] to remember to make when operating in this mode is to set the flag vectorize
[00:15:04] this mode is to set the flag vectorize equals false
[00:15:05] equals false we already have each example represented
[00:15:08] we already have each example represented as a vector so we do not need to pass it
[00:15:10] as a vector so we do not need to pass it through that whole process of using a
[00:15:12] through that whole process of using a dict vectorizer to turn count
[00:15:14] dict vectorizer to turn count dictionaries into vectors
[00:15:17] dictionaries into vectors and as i said before these turn out to
[00:15:19] and as i said before these turn out to be quite good models despite their
[00:15:20] be quite good models despite their compactness and the final thing i'll say
[00:15:22] compactness and the final thing i'll say is that this model is a nice transition
[00:15:25] is that this model is a nice transition into the recurrent neural networks that
[00:15:27] into the recurrent neural networks that we'll study in the final screencast for
[00:15:29] we'll study in the final screencast for this unit which essentially generalized
[00:15:31] this unit which essentially generalized this idea by learning an interesting
[00:15:34] this idea by learning an interesting combination function for all the vectors
[00:15:36] combination function for all the vectors for each of the individual tokens
[00:15:42] you

RNN Classifiers | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=7n9zQ169b8Q

---

Transcript

[00:00:04] welcome back everyone this is part eight
[00:00:06] welcome back everyone this is part eight in our series on supervised sentiment
[00:00:07] in our series on supervised sentiment analysis the final screencast in the
[00:00:09] analysis the final screencast in the series we're going to be talking about
[00:00:11] series we're going to be talking about recurrent neural network or rnn
[00:00:13] recurrent neural network or rnn classifiers i suppose this is officially
[00:00:15] classifiers i suppose this is officially our first step into the world of deep
[00:00:16] our first step into the world of deep learning for sentiment analysis
[00:00:19] learning for sentiment analysis this slide gives an overview of the
[00:00:21] this slide gives an overview of the model and let's work through it in some
[00:00:22] model and let's work through it in some detail so we've got a single example
[00:00:25] detail so we've got a single example with three tokens the rock rules these
[00:00:27] with three tokens the rock rules these models are prepared for variable length
[00:00:29] models are prepared for variable length sequences but this example has to have
[00:00:31] sequences but this example has to have happens to have length three
[00:00:33] happens to have length three and the first step to get this model
[00:00:35] and the first step to get this model started is a familiar one we're going to
[00:00:36] started is a familiar one we're going to look up each one of those tokens in what
[00:00:38] look up each one of those tokens in what is presumably a fixed embedding space
[00:00:41] is presumably a fixed embedding space here so for each token we'll get a
[00:00:42] here so for each token we'll get a vector representation
[00:00:45] vector representation the next step is that we have some
[00:00:46] the next step is that we have some learned parameters a weight matrix w xh
[00:00:49] learned parameters a weight matrix w xh and the subscript indicates that we're
[00:00:51] and the subscript indicates that we're going from the inputs x into the hidden
[00:00:53] going from the inputs x into the hidden layer h so that's a first transformation
[00:00:56] layer h so that's a first transformation and that weight matrix is used at each
[00:00:57] and that weight matrix is used at each one of these time steps
[00:01:00] one of these time steps there is a second learning weight matrix
[00:01:01] there is a second learning weight matrix which i've called whhh to indicate that
[00:01:04] which i've called whhh to indicate that we are now traveling through the hidden
[00:01:06] we are now traveling through the hidden layer
[00:01:07] layer so we start at some initial state h0
[00:01:09] so we start at some initial state h0 which could be an all zero or a randomly
[00:01:11] which could be an all zero or a randomly initialized vector or a vector coming
[00:01:13] initialized vector or a vector coming from some other component in the model
[00:01:15] from some other component in the model and that representation is combined with
[00:01:17] and that representation is combined with the representation that we derive going
[00:01:19] the representation that we derive going vertically up from the embedding usually
[00:01:21] vertically up from the embedding usually in some additive fashion to create this
[00:01:24] in some additive fashion to create this hidden state here h1 and those
[00:01:26] hidden state here h1 and those parameters wh are used again at each one
[00:01:29] parameters wh are used again at each one of these time steps so that we have two
[00:01:31] of these time steps so that we have two learned weight matrices as part of the
[00:01:33] learned weight matrices as part of the core structure of this model the one
[00:01:35] core structure of this model the one that takes us from embeddings into the
[00:01:37] that takes us from embeddings into the hidden layer and the one that travels us
[00:01:39] hidden layer and the one that travels us across the hidden layer and again those
[00:01:41] across the hidden layer and again those are typically combined in some additive
[00:01:43] are typically combined in some additive fashion to create these internal hidden
[00:01:45] fashion to create these internal hidden representations
[00:01:46] representations now we can do anything we want with
[00:01:48] now we can do anything we want with those internal hidden representations
[00:01:50] those internal hidden representations when we use rnns as classifiers we do
[00:01:53] when we use rnns as classifiers we do what is arguably the simplest thing
[00:01:55] what is arguably the simplest thing which is take the final representation
[00:01:58] which is take the final representation and use that as the input to a standard
[00:02:00] and use that as the input to a standard softmax classifier so from the point of
[00:02:02] softmax classifier so from the point of view of h3 going to y here we just have
[00:02:06] view of h3 going to y here we just have a learn weight matrix for the classifier
[00:02:08] a learn weight matrix for the classifier maybe also a bias term but from this
[00:02:10] maybe also a bias term but from this point here this is really just a
[00:02:11] point here this is really just a classifier of the sort we've been
[00:02:13] classifier of the sort we've been studying up until this point in the unit
[00:02:16] studying up until this point in the unit of course we could elaborate this model
[00:02:17] of course we could elaborate this model in all sorts of ways it could run
[00:02:19] in all sorts of ways it could run bidirectionally we could make more full
[00:02:21] bidirectionally we could make more full use of the different hidden
[00:02:23] use of the different hidden representations here but in the simplest
[00:02:25] representations here but in the simplest mode our rnn classifiers will just
[00:02:27] mode our rnn classifiers will just derive hidden representations at each
[00:02:29] derive hidden representations at each time step and use the final one as the
[00:02:31] time step and use the final one as the input to a classifier a couple things i
[00:02:33] input to a classifier a couple things i would say about this first if you would
[00:02:35] would say about this first if you would like a further layer of detail on how
[00:02:37] like a further layer of detail on how these models are structured and
[00:02:38] these models are structured and optimized i would encourage you to look
[00:02:40] optimized i would encourage you to look at this pure numpy reference
[00:02:42] at this pure numpy reference implementation of an rnn classifier that
[00:02:45] implementation of an rnn classifier that is included in our course code
[00:02:46] is included in our course code distribution i think that's a great way
[00:02:48] distribution i think that's a great way to get a feel for the recursive process
[00:02:51] to get a feel for the recursive process of
[00:02:51] of you know computing through full
[00:02:53] you know computing through full sequences and then having the error
[00:02:55] sequences and then having the error signals back propagate through to update
[00:02:57] signals back propagate through to update the weight matrix
[00:02:58] the weight matrix but for now i think just understanding
[00:03:01] but for now i think just understanding that the core structure of this model is
[00:03:03] that the core structure of this model is sufficient
[00:03:06] sufficient i just want to remind you from the
[00:03:07] i just want to remind you from the previous screencast that we're very
[00:03:09] previous screencast that we're very close to the idea of distributed
[00:03:11] close to the idea of distributed representations of features that i
[00:03:13] representations of features that i introduced before
[00:03:14] introduced before recall that for this mode what we do is
[00:03:16] recall that for this mode what we do is look up each token in an embedding space
[00:03:18] look up each token in an embedding space just as we do for the rnn
[00:03:20] just as we do for the rnn but instead of learning some complicated
[00:03:22] but instead of learning some complicated combination function with a bunch of
[00:03:24] combination function with a bunch of learned parameters we simply combine
[00:03:26] learned parameters we simply combine them via sum or average and that's the
[00:03:28] them via sum or average and that's the basis that's the input to the classifier
[00:03:31] basis that's the input to the classifier here the rnn can be considered an
[00:03:33] here the rnn can be considered an elaboration of that because instead of
[00:03:35] elaboration of that because instead of assuming that these vectors here will be
[00:03:36] assuming that these vectors here will be combined in some simple way like summer
[00:03:38] combined in some simple way like summer mean we now have really vast capacity to
[00:03:42] mean we now have really vast capacity to learn a much more complicated way of
[00:03:43] learn a much more complicated way of combining them that is optimal with
[00:03:45] combining them that is optimal with respect to the classifier that we're
[00:03:47] respect to the classifier that we're trying to fit
[00:03:48] trying to fit but fundamentally these are very similar
[00:03:50] but fundamentally these are very similar ideas and if it happened that
[00:03:52] ideas and if it happened that some or mean is in this picture was
[00:03:54] some or mean is in this picture was exactly the right function to learn for
[00:03:56] exactly the right function to learn for your data then the rnn would certainly
[00:03:58] your data then the rnn would certainly have the capacity to do that we just
[00:04:00] have the capacity to do that we just tend to favor the rnn because it can
[00:04:02] tend to favor the rnn because it can learn of course a much wider range of
[00:04:04] learn of course a much wider range of complicated custom functions that are
[00:04:06] complicated custom functions that are particular to the problem that you've
[00:04:07] particular to the problem that you've posed
[00:04:10] now so far we've been operating in the
[00:04:12] now so far we've been operating in the mode which i've called standard rnn data
[00:04:15] mode which i've called standard rnn data set preparation let's linger over that
[00:04:17] set preparation let's linger over that in a little bit of detail suppose that
[00:04:18] in a little bit of detail suppose that we have two examples containing the
[00:04:20] we have two examples containing the tokens aba and bc those are our two raw
[00:04:24] tokens aba and bc those are our two raw inputs
[00:04:25] inputs the first step in the standard mode is
[00:04:27] the first step in the standard mode is to look at each one of those in some in
[00:04:29] to look at each one of those in some in in some list of indices
[00:04:31] in some list of indices and then those indices are keyed into an
[00:04:34] and then those indices are keyed into an embedding space and those finally give
[00:04:36] embedding space and those finally give us the vector representations of each
[00:04:38] us the vector representations of each examples so that really and truly the
[00:04:41] examples so that really and truly the inputs of the rnn is a list of vectors
[00:04:43] inputs of the rnn is a list of vectors it's just that we have typically
[00:04:45] it's just that we have typically obtained those vectors by looking them
[00:04:47] obtained those vectors by looking them up in a fixed embedding space and so for
[00:04:49] up in a fixed embedding space and so for example since a occurs twice in this
[00:04:51] example since a occurs twice in this first example it is literally repeated
[00:04:54] first example it is literally repeated as the first and third vectors here
[00:04:57] as the first and third vectors here now i think you can see latent in this
[00:04:59] now i think you can see latent in this picture the possibility that we might
[00:05:01] picture the possibility that we might drop the embedding space and instead
[00:05:03] drop the embedding space and instead just directly input lists of vectors and
[00:05:05] just directly input lists of vectors and that is one way that we will explore
[00:05:07] that is one way that we will explore later on in the quarter of using
[00:05:09] later on in the quarter of using contextual models like bert we would
[00:05:11] contextual models like bert we would simply look up entire token streams and
[00:05:14] simply look up entire token streams and get back lists of vectors and use those
[00:05:16] get back lists of vectors and use those as fixed inputs to a model like an rnn
[00:05:19] as fixed inputs to a model like an rnn and that's a first step toward
[00:05:21] and that's a first step toward fine-tuning models like bert on problems
[00:05:24] fine-tuning models like bert on problems like the ones we've posed in this unit
[00:05:26] like the ones we've posed in this unit so have that idea in mind as we talk
[00:05:28] so have that idea in mind as we talk next about fine tuning strategies
[00:05:32] next about fine tuning strategies now another practical note what i've
[00:05:34] now another practical note what i've shown you so far is what you'd call a
[00:05:37] shown you so far is what you'd call a simple vanilla rnn
[00:05:39] simple vanilla rnn lstms long short-term memory networks
[00:05:42] lstms long short-term memory networks are much more powerful models and will
[00:05:44] are much more powerful models and will kind of default to them when we do
[00:05:45] kind of default to them when we do experiments the fundamental issue is
[00:05:48] experiments the fundamental issue is that plane rnns tend to perform poorly
[00:05:50] that plane rnns tend to perform poorly with very long sequences you get that
[00:05:52] with very long sequences you get that error signal from the classifier there
[00:05:54] error signal from the classifier there at the final token and now information
[00:05:57] at the final token and now information has to flow all the way back down
[00:05:58] has to flow all the way back down through the network it could be a very
[00:06:00] through the network it could be a very long sequence and the result is that the
[00:06:03] long sequence and the result is that the information coming from that error
[00:06:04] information coming from that error signal is often lost or distorted
[00:06:08] signal is often lost or distorted now lstm cells are a prominent response
[00:06:10] now lstm cells are a prominent response to this problem they introduce
[00:06:12] to this problem they introduce mechanisms that control the flow of
[00:06:14] mechanisms that control the flow of information and help you avoid the
[00:06:16] information and help you avoid the problems of optimization that arise for
[00:06:18] problems of optimization that arise for regular rnns now i'm not going to take
[00:06:20] regular rnns now i'm not going to take the time here to review this mechanism
[00:06:22] the time here to review this mechanism in detail i would instead recommend
[00:06:25] in detail i would instead recommend these two excellent blog posts they have
[00:06:27] these two excellent blog posts they have great diagrams and really detailed
[00:06:29] great diagrams and really detailed discussions they can do a much better
[00:06:31] discussions they can do a much better job than i can of really conveying
[00:06:33] job than i can of really conveying intuitions visually and also with math
[00:06:36] intuitions visually and also with math and i think you could pick one or both
[00:06:38] and i think you could pick one or both and really pretty quickly gain a deep
[00:06:40] and really pretty quickly gain a deep understanding of precisely how lstm
[00:06:42] understanding of precisely how lstm cells are functioning
[00:06:45] cells are functioning the final thing here is just a code
[00:06:47] the final thing here is just a code snippet to show you how easy it is to
[00:06:49] snippet to show you how easy it is to use our course code repository to fit
[00:06:51] use our course code repository to fit models like this in the context of
[00:06:53] models like this in the context of sentiment analysis you can again make
[00:06:55] sentiment analysis you can again make use of this sst library and what i've
[00:06:57] use of this sst library and what i've done here is a kind of complicated
[00:06:59] done here is a kind of complicated version showing you a bunch of different
[00:07:01] version showing you a bunch of different features so
[00:07:03] features so in cell 2 you can see that i'm going to
[00:07:05] in cell 2 you can see that i'm going to have a pointer to glove and i'm going to
[00:07:07] have a pointer to glove and i'm going to create a glove look up
[00:07:09] create a glove look up using the 50 dimensional vectors just to
[00:07:11] using the 50 dimensional vectors just to keep things simple
[00:07:12] keep things simple the feature function for this model is
[00:07:15] the feature function for this model is not one that returns count dictionaries
[00:07:17] not one that returns count dictionaries it's important for the structure of the
[00:07:19] it's important for the structure of the model we're going to use that you input
[00:07:21] model we're going to use that you input raw sequences of tokens so all we're
[00:07:23] raw sequences of tokens so all we're doing here is down casing the sequence
[00:07:25] doing here is down casing the sequence and then splitting on white space of
[00:07:27] and then splitting on white space of course you could do something more
[00:07:28] course you could do something more sophisticated
[00:07:30] sophisticated the idea though is that you want to
[00:07:31] the idea though is that you want to align with the glove vocabulary our
[00:07:34] align with the glove vocabulary our model wrapper is doing a few things it's
[00:07:36] model wrapper is doing a few things it's creating a vocabulary and loading it and
[00:07:38] creating a vocabulary and loading it and embedding using this glove space that'll
[00:07:40] embedding using this glove space that'll be the initial embedding for our model
[00:07:43] be the initial embedding for our model and if you leave this step out you'll
[00:07:44] and if you leave this step out you'll have a randomly initialized embedding
[00:07:46] have a randomly initialized embedding space which uh might be fine as well but
[00:07:48] space which uh might be fine as well but presumably glove will give us a step up
[00:07:51] presumably glove will give us a step up and then we set up the torch rnn
[00:07:53] and then we set up the torch rnn classifier and what i've done here is
[00:07:55] classifier and what i've done here is expose a lot of the different keyword
[00:07:56] expose a lot of the different keyword arguments not all of them there are lots
[00:07:58] arguments not all of them there are lots of knobs that you can fiddle with as is
[00:08:00] of knobs that you can fiddle with as is typical for deep learning models
[00:08:02] typical for deep learning models maybe the one i would call out is that
[00:08:04] maybe the one i would call out is that we are using that fixed embedding that
[00:08:05] we are using that fixed embedding that we got from glove and i have set early
[00:08:07] we got from glove and i have set early stopping equals true which might help
[00:08:09] stopping equals true which might help you efficiently optimize these models
[00:08:12] you efficiently optimize these models otherwise you'll have to figure out how
[00:08:13] otherwise you'll have to figure out how many iterations you actually want it to
[00:08:15] many iterations you actually want it to run for and you might run it for much
[00:08:17] run for and you might run it for much too long or much less time than is
[00:08:19] too long or much less time than is needed to get an optimal model the early
[00:08:22] needed to get an optimal model the early stopping options there are a few other
[00:08:23] stopping options there are a few other parameters involved in that might help
[00:08:26] parameters involved in that might help you optimize these models efficiently
[00:08:28] you optimize these models efficiently and effectively
[00:08:30] and effectively in the end though having set up all that
[00:08:31] in the end though having set up all that stuff you call fit as usual and return
[00:08:33] stuff you call fit as usual and return the train model and in that context you
[00:08:36] the train model and in that context you can simply use sst experiment with these
[00:08:38] can simply use sst experiment with these previous components to conduct
[00:08:40] previous components to conduct experiments with rnns just as you did
[00:08:42] experiments with rnns just as you did for simpler linear models as in previous
[00:08:44] for simpler linear models as in previous screencasts the one change which will be
[00:08:47] screencasts the one change which will be will be familiar from the previous
[00:08:49] will be familiar from the previous screencast is that you need to set
[00:08:50] screencast is that you need to set vectorize equals false and that is
[00:08:53] vectorize equals false and that is important because again we're going to
[00:08:55] important because again we're going to let the model process these examples we
[00:08:57] let the model process these examples we don't want to pipe everything through
[00:08:58] don't want to pipe everything through some kind of dick vectorizer that's
[00:09:00] some kind of dick vectorizer that's strictly for hand built feature
[00:09:02] strictly for hand built feature functions and sparse linear models here
[00:09:04] functions and sparse linear models here in the land of deep learning vectorize
[00:09:06] in the land of deep learning vectorize equals false and we'll use the
[00:09:08] equals false and we'll use the components of the model to represent
[00:09:10] components of the model to represent each example as i discussed before

Contextual Representation Models | Stanford CS224U Natural Language Understanding | Spring 2021

Source: https://www.youtube.com/watch?v=ZrmEcrmmXCg

---

Transcript

[00:00:05] welcome everyone to our first screencast
[00:00:07] welcome everyone to our first screencast on contextual word representations my
[00:00:09] on contextual word representations my goal here is to give you an overview for
[00:00:11] goal here is to give you an overview for this unit and also give you a sense for
[00:00:13] this unit and also give you a sense for the conceptual landscape
[00:00:15] the conceptual landscape let's start with the associated
[00:00:17] let's start with the associated materials you might think that the name
[00:00:18] materials you might think that the name of the game for this unit is to get you
[00:00:20] of the game for this unit is to get you to the point where you can work
[00:00:21] to the point where you can work productively with this notebook called
[00:00:24] productively with this notebook called fine tuning which shows you how to
[00:00:26] fine tuning which shows you how to fine-tune contextual word
[00:00:27] fine-tune contextual word representations for classification
[00:00:29] representations for classification problems i think that could be a very
[00:00:30] problems i think that could be a very powerful mode for you as you work on the
[00:00:32] powerful mode for you as you work on the current assignment and bake off
[00:00:35] current assignment and bake off for background and intuitions i highly
[00:00:37] for background and intuitions i highly recommend this paper by noah smith
[00:00:40] recommend this paper by noah smith the beating heart of this unit is really
[00:00:42] the beating heart of this unit is really the transformer architecture which was
[00:00:44] the transformer architecture which was introduced by veswani in all 2017 in a
[00:00:47] introduced by veswani in all 2017 in a paper called attention is all you need
[00:00:49] paper called attention is all you need is a highly readable paper but i
[00:00:51] is a highly readable paper but i recommend that if you want to read it
[00:00:53] recommend that if you want to read it you instead read sasha rush's
[00:00:54] you instead read sasha rush's outstanding contribution the annotated
[00:00:56] outstanding contribution the annotated transformer what this does is literally
[00:00:59] transformer what this does is literally reproduce the text of veswani at all
[00:01:01] reproduce the text of veswani at all 2017
[00:01:03] 2017 with pi torch code woven in
[00:01:06] with pi torch code woven in culminating in a complete implementation
[00:01:08] culminating in a complete implementation of the transformer as applied to
[00:01:09] of the transformer as applied to problems in machine translation this is
[00:01:12] problems in machine translation this is a wonderful contribution in the sense
[00:01:13] a wonderful contribution in the sense that to the extent that there are points
[00:01:15] that to the extent that there are points of unclarity or uncertainty in the
[00:01:16] of unclarity or uncertainty in the original text they are fully resolved by
[00:01:19] original text they are fully resolved by sasha's code
[00:01:21] sasha's code and of course this can give you a really
[00:01:22] and of course this can give you a really good example of how to do efficient and
[00:01:24] good example of how to do efficient and effective implementation of model
[00:01:26] effective implementation of model architectures like this using pi torch
[00:01:30] architectures like this using pi torch in practical terms we're going to make
[00:01:31] in practical terms we're going to make extensive use of the hugging face
[00:01:33] extensive use of the hugging face transformers library which has really
[00:01:34] transformers library which has really opened up access to a wide range of
[00:01:37] opened up access to a wide range of pre-trained transformer models it's very
[00:01:39] pre-trained transformer models it's very exciting and has enabled lots of new
[00:01:41] exciting and has enabled lots of new things
[00:01:43] things for us the central architecture will be
[00:01:45] for us the central architecture will be burnt we'll have a separate screencast
[00:01:46] burnt we'll have a separate screencast on that and we're also going to have a
[00:01:48] on that and we're also going to have a screencast on roberta which is robustly
[00:01:50] screencast on roberta which is robustly optimized burnt i think it's an
[00:01:52] optimized burnt i think it's an interesting perspective in the sense
[00:01:53] interesting perspective in the sense that they explored more deeply some of
[00:01:56] that they explored more deeply some of the open questions from the original
[00:01:57] the open questions from the original bert paper and they also released very
[00:02:00] bert paper and they also released very powerful pre-trained parameters that you
[00:02:02] powerful pre-trained parameters that you could again use in the context of your
[00:02:04] could again use in the context of your own fine-tuning
[00:02:06] own fine-tuning and then for a slightly different
[00:02:07] and then for a slightly different perspective on these transformers we're
[00:02:09] perspective on these transformers we're going to look at the electra
[00:02:10] going to look at the electra architecture which came from kevin clark
[00:02:12] architecture which came from kevin clark and colleagues at stanford and google i
[00:02:14] and colleagues at stanford and google i really like this as a new perspective
[00:02:16] really like this as a new perspective there are of course many different modes
[00:02:18] there are of course many different modes of using the transformer at this point
[00:02:20] of using the transformer at this point i'm going to mention a few at the end of
[00:02:22] i'm going to mention a few at the end of the screencast and just for the sake of
[00:02:23] the screencast and just for the sake of time i've decided to focus on electra
[00:02:26] time i've decided to focus on electra and you can explore the others in your
[00:02:28] and you can explore the others in your own research
[00:02:30] own research let's begin with some intuitions and i'd
[00:02:31] let's begin with some intuitions and i'd like to begin with a linguistic
[00:02:33] like to begin with a linguistic intuition which has to do with word
[00:02:35] intuition which has to do with word representations and how they should be
[00:02:37] representations and how they should be shaped by the context let's focus on the
[00:02:39] shaped by the context let's focus on the english verb break
[00:02:41] english verb break we have a simple example the vase broke
[00:02:42] we have a simple example the vase broke which means it's shattered to pieces
[00:02:45] which means it's shattered to pieces here's a superficially similar sentence
[00:02:47] here's a superficially similar sentence dawn broke now the sense of break is
[00:02:49] dawn broke now the sense of break is something more like begin
[00:02:52] something more like begin the news broke again a simple
[00:02:54] the news broke again a simple intransitive sentence but now the verb
[00:02:56] intransitive sentence but now the verb break means something more like publish
[00:02:58] break means something more like publish or appear or become known
[00:03:01] or appear or become known sandy broke the world record this is a
[00:03:03] sandy broke the world record this is a transitive use of the verb break and it
[00:03:05] transitive use of the verb break and it means like surpass a previous level
[00:03:08] means like surpass a previous level sandy broke the law as another transit
[00:03:10] sandy broke the law as another transit of use but now it means sandy
[00:03:12] of use but now it means sandy transgressed
[00:03:13] transgressed the burglar broke into the house as a
[00:03:16] the burglar broke into the house as a physical act of transgression on a space
[00:03:18] physical act of transgression on a space the newscaster broke into the movie
[00:03:20] the newscaster broke into the movie broadcast is a sense that is more like
[00:03:22] broadcast is a sense that is more like interrupt
[00:03:24] interrupt and we have idioms like break even which
[00:03:26] and we have idioms like break even which means we neither gain nor lost money
[00:03:29] means we neither gain nor lost money and this is just a few of the many ways
[00:03:31] and this is just a few of the many ways that the verb break can be used in
[00:03:33] that the verb break can be used in english how many senses are at work here
[00:03:35] english how many senses are at work here it's very hard to say it could be one it
[00:03:37] it's very hard to say it could be one it could be two it could be ten
[00:03:39] could be two it could be ten it's very hard to delimit word senses
[00:03:40] it's very hard to delimit word senses but it is very clear from this data that
[00:03:43] but it is very clear from this data that our sense for the verb break is being
[00:03:46] our sense for the verb break is being shaped by the immediate linguistic
[00:03:47] shaped by the immediate linguistic context
[00:03:48] context here are a few additional examples we
[00:03:51] here are a few additional examples we have things like flat tire flat beer
[00:03:53] have things like flat tire flat beer flat note flat surface it's clear that
[00:03:55] flat note flat surface it's clear that there is a conceptual core running
[00:03:57] there is a conceptual core running through all of these uses but it's also
[00:04:00] through all of these uses but it's also true that a flat tire is a very
[00:04:02] true that a flat tire is a very different sense for flat than we get
[00:04:04] different sense for flat than we get from flat note or flat surface
[00:04:07] from flat note or flat surface have something similar for throw a party
[00:04:09] have something similar for throw a party throw a fight throw a ball throw a fit
[00:04:11] throw a fight throw a ball throw a fit we have a mixture of what you might call
[00:04:13] we have a mixture of what you might call literal and metaphorical here but again
[00:04:15] literal and metaphorical here but again a kind of common core that we're drawing
[00:04:17] a kind of common core that we're drawing on but the bottom line is that the sense
[00:04:19] on but the bottom line is that the sense for throw in this case is very different
[00:04:22] for throw in this case is very different depending on what kind of linguistic
[00:04:23] depending on what kind of linguistic context it's in
[00:04:26] context it's in and we we can extend this to things that
[00:04:28] and we we can extend this to things that seem to turn more on world knowledge so
[00:04:30] seem to turn more on world knowledge so if you have something like a crane
[00:04:32] if you have something like a crane caught a fish we have a sense that the
[00:04:33] caught a fish we have a sense that the crane here is a bird
[00:04:35] crane here is a bird whereas if we have a crane picked up the
[00:04:37] whereas if we have a crane picked up the steel beam we have a sense that it's a
[00:04:39] steel beam we have a sense that it's a piece of equipment
[00:04:40] piece of equipment uh this seems like something that's
[00:04:42] uh this seems like something that's guided by our understanding of birds and
[00:04:44] guided by our understanding of birds and equipment and fish and beams and when we
[00:04:47] equipment and fish and beams and when we have relatively unbiased sentences like
[00:04:49] have relatively unbiased sentences like i saw a crane we're kind of left
[00:04:51] i saw a crane we're kind of left guessing about which object is involved
[00:04:53] guessing about which object is involved the bird or the machine
[00:04:55] the bird or the machine and we can extend this past world
[00:04:56] and we can extend this past world knowledge into things that are more like
[00:04:58] knowledge into things that are more like discourse understanding so if you have a
[00:05:00] discourse understanding so if you have a sentence like are there typos i didn't
[00:05:02] sentence like are there typos i didn't see any
[00:05:03] see any the sense of any here we have a feeling
[00:05:05] the sense of any here we have a feeling that something is alighted but probably
[00:05:07] that something is alighted but probably localized on any and any here means any
[00:05:10] localized on any and any here means any typos as a result of the preceding
[00:05:12] typos as a result of the preceding linguistic context
[00:05:14] linguistic context are there any bookstores downtown i
[00:05:16] are there any bookstores downtown i didn't see any same second sentence but
[00:05:18] didn't see any same second sentence but now the sense of any is probably going
[00:05:20] now the sense of any is probably going to be something more like bookstores as
[00:05:22] to be something more like bookstores as a result of the discourse context that
[00:05:24] a result of the discourse context that it appears in
[00:05:26] it appears in so all of this is just showing how much
[00:05:28] so all of this is just showing how much individual linguistic units can be
[00:05:30] individual linguistic units can be shaped by context linguists know this
[00:05:32] shaped by context linguists know this deeply this is a primary thing that
[00:05:34] deeply this is a primary thing that linguists try to get a grip on and i
[00:05:36] linguists try to get a grip on and i think it's a wonderful point of
[00:05:37] think it's a wonderful point of connection between what linguists do and
[00:05:40] connection between what linguists do and the way we're representing examples in
[00:05:41] the way we're representing examples in nlp using contextual models this is a
[00:05:44] nlp using contextual models this is a very exciting development for me as a
[00:05:46] very exciting development for me as a linguist as well as an nlp
[00:05:51] here's another set of intuitions that's
[00:05:53] here's another set of intuitions that's related more to things like model
[00:05:55] related more to things like model architecture and what you might call
[00:05:57] architecture and what you might call inductive biases for different model
[00:05:59] inductive biases for different model designs
[00:06:00] designs let's start up here in the left this is
[00:06:01] let's start up here in the left this is a high bias model in the sense that it
[00:06:03] a high bias model in the sense that it makes a lot of a priori decisions about
[00:06:06] makes a lot of a priori decisions about how we will represent our examples the
[00:06:08] how we will represent our examples the idea is that we have three tokens which
[00:06:10] idea is that we have three tokens which we look up in a fixed embedding space
[00:06:13] we look up in a fixed embedding space and then we have decided to summarize
[00:06:14] and then we have decided to summarize those embeddings by simply summing them
[00:06:16] those embeddings by simply summing them together to get a representation for the
[00:06:18] together to get a representation for the entire example
[00:06:19] entire example very few of these components are learned
[00:06:21] very few of these components are learned as part of our problem we've made most
[00:06:23] as part of our problem we've made most of the decisions ahead of time
[00:06:25] of the decisions ahead of time as we've seen as we move to a recurrent
[00:06:27] as we've seen as we move to a recurrent neural network we relax some of those
[00:06:29] neural network we relax some of those assumptions we're still going to look up
[00:06:31] assumptions we're still going to look up words in a fixed embedding space but now
[00:06:33] words in a fixed embedding space but now instead of deciding that we know the
[00:06:35] instead of deciding that we know the proper way to combine those with
[00:06:36] proper way to combine those with summation we're going to learn from our
[00:06:39] summation we're going to learn from our data a very complicated function for
[00:06:41] data a very complicated function for combining them and that will presumably
[00:06:43] combining them and that will presumably allow us to be more responsive that is
[00:06:45] allow us to be more responsive that is less biased about what the data are
[00:06:47] less biased about what the data are likely to look like
[00:06:49] likely to look like the tree structured architecture down
[00:06:51] the tree structured architecture down here is an interesting mixture of these
[00:06:53] here is an interesting mixture of these ideas it's like the recurrent neural
[00:06:55] ideas it's like the recurrent neural network instead of instead of assuming
[00:06:56] network instead of instead of assuming that i can process the data left to
[00:06:58] that i can process the data left to right the data get processed into
[00:07:00] right the data get processed into constituents like the rock as a
[00:07:02] constituents like the rock as a constituent as excluded from rules over
[00:07:05] constituent as excluded from rules over here
[00:07:06] here now this is probably going to be very
[00:07:08] now this is probably going to be very powerful if we're correct that the
[00:07:09] powerful if we're correct that the language data are structured according
[00:07:11] language data are structured according to these constituents because it will
[00:07:13] to these constituents because it will give us a boost in terms of learning it
[00:07:16] give us a boost in terms of learning it could be counterproductive though to the
[00:07:17] could be counterproductive though to the extent that that constituent structure
[00:07:19] extent that that constituent structure is wrong and i think that's showing that
[00:07:21] is wrong and i think that's showing that biases that we impose at the level level
[00:07:23] biases that we impose at the level level of our architectures can be helpful as
[00:07:26] of our architectures can be helpful as well as a hindrance depending on how
[00:07:27] well as a hindrance depending on how they align with the data-driven problem
[00:07:30] they align with the data-driven problem that we're trying to solve
[00:07:32] that we're trying to solve in the bottom right here i have the
[00:07:34] in the bottom right here i have the least biased model in all these senses
[00:07:36] least biased model in all these senses of all the ones depicted here i've got a
[00:07:39] of all the ones depicted here i've got a recurrent neural network like this one
[00:07:40] recurrent neural network like this one except now i'm assuming that information
[00:07:42] except now i'm assuming that information can flow bi-directionally so no longer a
[00:07:44] can flow bi-directionally so no longer a presumption of left to right
[00:07:46] presumption of left to right and in addition i've added these
[00:07:48] and in addition i've added these attention mechanisms which we'll talk a
[00:07:50] attention mechanisms which we'll talk a lot about in this unit but essentially
[00:07:52] lot about in this unit but essentially think of them as ways of creating
[00:07:54] think of them as ways of creating special connections between all of the
[00:07:56] special connections between all of the hidden units and the idea here is that
[00:07:58] hidden units and the idea here is that we would let the data tell us how to
[00:08:00] we would let the data tell us how to weight all of these various connections
[00:08:02] weight all of these various connections and in turn represent our examples we're
[00:08:05] and in turn represent our examples we're making very few decisions ahead of time
[00:08:07] making very few decisions ahead of time about what kind of connections could be
[00:08:09] about what kind of connections could be made and instead just listening to the
[00:08:11] made and instead just listening to the data and the learning process
[00:08:13] data and the learning process and we are at this point on the road
[00:08:16] and we are at this point on the road toward the transformer which is a kind
[00:08:17] toward the transformer which is a kind of extreme case of connecting everything
[00:08:19] of extreme case of connecting everything with everything else and then allowing
[00:08:21] with everything else and then allowing the data to tell us how to weight all of
[00:08:23] the data to tell us how to weight all of those various connections
[00:08:26] those various connections and that does bring us to this notion of
[00:08:28] and that does bring us to this notion of attention which we've not discussed
[00:08:30] attention which we've not discussed before but i think i can introduce the
[00:08:31] before but i think i can introduce the concepts in a bit of the math and then
[00:08:33] concepts in a bit of the math and then we'll see them again throughout this
[00:08:35] we'll see them again throughout this unit so let's start with a simple
[00:08:36] unit so let's start with a simple sentiment example and imagine we're
[00:08:38] sentiment example and imagine we're dealing with a recurrent neural network
[00:08:40] dealing with a recurrent neural network classifier our example is really not so
[00:08:42] classifier our example is really not so good and we're going to fit the
[00:08:43] good and we're going to fit the classifier traditionally on top of this
[00:08:46] classifier traditionally on top of this final hidden state over here
[00:08:48] final hidden state over here but we might worry about doing that that
[00:08:51] but we might worry about doing that that by the time we've gotten to this final
[00:08:53] by the time we've gotten to this final state the contribution of these earlier
[00:08:54] state the contribution of these earlier words which are clearly important
[00:08:56] words which are clearly important linguistically might be sort of
[00:08:58] linguistically might be sort of forgotten or overly diffuse
[00:09:00] forgotten or overly diffuse so attention mechanisms would be a way
[00:09:02] so attention mechanisms would be a way for us to bring that information back in
[00:09:05] for us to bring that information back in and infuse this representation hc with
[00:09:08] and infuse this representation hc with some of those previous
[00:09:10] some of those previous important connections
[00:09:12] important connections so here's how we do that we're going to
[00:09:13] so here's how we do that we're going to first have some tension scores which are
[00:09:15] first have some tension scores which are simply dot products of our target vector
[00:09:17] simply dot products of our target vector with all the preceding hidden states so
[00:09:20] with all the preceding hidden states so that gives us a vector of scores which
[00:09:21] that gives us a vector of scores which are traditionally soft max normalized
[00:09:24] are traditionally soft max normalized and then what we do is create a context
[00:09:26] and then what we do is create a context vector by
[00:09:27] vector by weighting each of the previous hidden
[00:09:29] weighting each of the previous hidden states by its attention weight
[00:09:31] states by its attention weight and then taking the average of those to
[00:09:33] and then taking the average of those to give us the context vector k
[00:09:36] give us the context vector k and then we're going to have this
[00:09:37] and then we're going to have this special layer here which concatenates k
[00:09:39] special layer here which concatenates k with our previous final hidden state
[00:09:42] with our previous final hidden state and feeds that through this layer of
[00:09:44] and feeds that through this layer of learn parameters and a non-linearity to
[00:09:46] learn parameters and a non-linearity to give us this new hidden representation h
[00:09:49] give us this new hidden representation h tilde
[00:09:50] tilde and it is h tilde that is finally the
[00:09:52] and it is h tilde that is finally the input to our softmax classifier whereas
[00:09:55] input to our softmax classifier whereas before we would have simply directly
[00:09:57] before we would have simply directly input hc up here we now input this more
[00:10:00] input hc up here we now input this more refined version that is drawing on all
[00:10:02] refined version that is drawing on all of these attention connections that we
[00:10:04] of these attention connections that we created with these mechanisms
[00:10:06] created with these mechanisms and again as you'll see the transformer
[00:10:08] and again as you'll see the transformer does this all over the place with all of
[00:10:10] does this all over the place with all of its representations at various points in
[00:10:12] its representations at various points in its computations
[00:10:15] its computations here's another guiding idea that really
[00:10:17] here's another guiding idea that really shapes how these models work i've called
[00:10:18] shapes how these models work i've called this word pieces and we've seen this
[00:10:20] this word pieces and we've seen this before these models typically do not
[00:10:22] before these models typically do not tokenize data in the way that we might
[00:10:24] tokenize data in the way that we might expect i've loaded in a burt tokenizer
[00:10:27] expect i've loaded in a burt tokenizer and you can see that for a sentence like
[00:10:29] and you can see that for a sentence like this isn't too surprising the result is
[00:10:31] this isn't too surprising the result is some pretty familiar tokens by and large
[00:10:34] some pretty familiar tokens by and large but when i feed in something like encode
[00:10:35] but when i feed in something like encode me the intuitive word encode is split
[00:10:37] me the intuitive word encode is split apart into two word pieces and clearly
[00:10:40] apart into two word pieces and clearly we're kind of implicitly assuming that
[00:10:42] we're kind of implicitly assuming that the model because it's contextual can
[00:10:44] the model because it's contextual can figure out that these pieces are in some
[00:10:46] figure out that these pieces are in some conceptual sense one word and you might
[00:10:49] conceptual sense one word and you might extend that up to idioms like out of
[00:10:51] extend that up to idioms like out of this world where we treat them as a
[00:10:52] this world where we treat them as a bunch of distinct tokens but we might
[00:10:55] bunch of distinct tokens but we might hope the model can learn that there's an
[00:10:56] hope the model can learn that there's an idiomatic unity to that phrase and this
[00:11:00] idiomatic unity to that phrase and this also has the side advantage that for
[00:11:01] also has the side advantage that for unknown tokens like snuffleupagus it can
[00:11:04] unknown tokens like snuffleupagus it can break them apart into familiar pieces
[00:11:06] break them apart into familiar pieces and we at least have a hope of getting a
[00:11:08] and we at least have a hope of getting a sensible representation for that out of
[00:11:10] sensible representation for that out of vocabulary item
[00:11:12] vocabulary item the result of all this is that these
[00:11:13] the result of all this is that these models can get away with having very
[00:11:15] models can get away with having very small vocabularies precisely because we
[00:11:18] small vocabularies precisely because we are relying on them implicitly to be
[00:11:20] are relying on them implicitly to be truly contextual
[00:11:24] here's another inspiring idea that we've
[00:11:26] here's another inspiring idea that we've not encountered before this is called
[00:11:27] not encountered before this is called positional encoding
[00:11:29] positional encoding and it's another way in which we can
[00:11:30] and it's another way in which we can capture sensitivity of words to their
[00:11:33] capture sensitivity of words to their context
[00:11:34] context so as you'll see when you go all the way
[00:11:36] so as you'll see when you go all the way down inside the transformer architecture
[00:11:38] down inside the transformer architecture you do have a traditional static
[00:11:40] you do have a traditional static embedding of the sort we discussed in
[00:11:41] embedding of the sort we discussed in the first unit for this course those are
[00:11:44] the first unit for this course those are in light gray here fixed representations
[00:11:46] in light gray here fixed representations for the words
[00:11:47] for the words however in the context of a model like
[00:11:49] however in the context of a model like vert what we traditionally think of as
[00:11:52] vert what we traditionally think of as its embedding representation is actually
[00:11:54] its embedding representation is actually a combination of that fixed embedding
[00:11:57] a combination of that fixed embedding and a separate embedding space called
[00:11:59] and a separate embedding space called the positional embedding
[00:12:01] the positional embedding where we have learned representations
[00:12:02] where we have learned representations for each position in a sequence
[00:12:05] for each position in a sequence this has the intriguing property that
[00:12:07] this has the intriguing property that one in the same word like the will have
[00:12:09] one in the same word like the will have a different embedding in this sense of
[00:12:11] a different embedding in this sense of the green representation here depending
[00:12:13] the green representation here depending on where it appears in your sequence so
[00:12:16] on where it appears in your sequence so right from the get-go we have a notion
[00:12:18] right from the get-go we have a notion of context sensitivity even before we've
[00:12:20] of context sensitivity even before we've started to connect things in all sorts
[00:12:22] started to connect things in all sorts of interesting ways
[00:12:25] of interesting ways now let's move to some current issues
[00:12:27] now let's move to some current issues and efforts some high level things that
[00:12:28] and efforts some high level things that you might think about as you work
[00:12:30] you might think about as you work through this unit
[00:12:31] through this unit this is a really nice graph from the
[00:12:33] this is a really nice graph from the electra paper from clark at all
[00:12:35] electra paper from clark at all along the x-axis here we have floating
[00:12:37] along the x-axis here we have floating point operations which you could think
[00:12:39] point operations which you could think of as a kind of basic measure of compute
[00:12:41] of as a kind of basic measure of compute resources needed to create these
[00:12:43] resources needed to create these representations and along the y-axis we
[00:12:46] representations and along the y-axis we have glue score so that's like a
[00:12:47] have glue score so that's like a standard nlu benchmark
[00:12:50] standard nlu benchmark and the point of this plot here is that
[00:12:51] and the point of this plot here is that we're reaching kind of diminishing
[00:12:53] we're reaching kind of diminishing returns so we had rapid increases from
[00:12:55] returns so we had rapid increases from glove gpt and up through to bert
[00:12:58] glove gpt and up through to bert where we're really doing much better on
[00:13:00] where we're really doing much better on these glue scores
[00:13:02] these glue scores uh we're increasing the floating point
[00:13:04] uh we're increasing the floating point operations but it seems to be
[00:13:05] operations but it seems to be commensurate with how we're doing on the
[00:13:07] commensurate with how we're doing on the benchmark but now with these larger
[00:13:09] benchmark but now with these larger models like excel net and roberta it's
[00:13:11] models like excel net and roberta it's arguably the case that we're reaching
[00:13:13] arguably the case that we're reaching diminishing returns roberta involves
[00:13:16] diminishing returns roberta involves more than 3 000 times the floating point
[00:13:18] more than 3 000 times the floating point operations of glove
[00:13:21] operations of glove but it's not that much better along this
[00:13:23] but it's not that much better along this y-axis than some of its simpler variants
[00:13:26] y-axis than some of its simpler variants like bert base and so this is something
[00:13:28] like bert base and so this is something we should think about when we think
[00:13:29] we should think about when we think about the costs in terms of money and in
[00:13:32] about the costs in terms of money and in the environment and energy and so forth
[00:13:35] the environment and energy and so forth when we think about developing these
[00:13:36] when we think about developing these large models
[00:13:38] large models and here's a really extreme case who
[00:13:40] and here's a really extreme case who knows how long we can train these things
[00:13:42] knows how long we can train these things or how much benefit we'll get when we do
[00:13:44] or how much benefit we'll get when we do so but at a certain point we're likely
[00:13:46] so but at a certain point we're likely to incur costs that are larger than any
[00:13:49] to incur costs that are larger than any gains that we can justify on the
[00:13:50] gains that we can justify on the problems we're trying to solve
[00:13:52] problems we're trying to solve and that kind of does lead us to this
[00:13:54] and that kind of does lead us to this lovely paper which talks about the
[00:13:55] lovely paper which talks about the environmental footprint of training
[00:13:57] environmental footprint of training these really big models and it just
[00:13:59] these really big models and it just shows that like training a big
[00:14:01] shows that like training a big transformer from scratch really incurs a
[00:14:03] transformer from scratch really incurs a large environmental cost
[00:14:05] large environmental cost that's certainly something we should
[00:14:06] that's certainly something we should have in mind as we think about using
[00:14:08] have in mind as we think about using these models for me it's a complicated
[00:14:10] these models for me it's a complicated question though because it's offset by
[00:14:12] question though because it's offset by the fact that by and large
[00:14:14] the fact that by and large all of us aren't training these from
[00:14:15] all of us aren't training these from scratch but rather benefiting from
[00:14:18] scratch but rather benefiting from publicly available pre-trained
[00:14:19] publicly available pre-trained representations
[00:14:21] representations so while the pre-training for that one
[00:14:23] so while the pre-training for that one version had a large environmental cost
[00:14:26] version had a large environmental cost it feels like it's kind of offset by the
[00:14:27] it feels like it's kind of offset by the fact that a lot of us are benefiting
[00:14:29] fact that a lot of us are benefiting from it and it might be that in
[00:14:31] from it and it might be that in aggregate this is less environmentally
[00:14:33] aggregate this is less environmentally costly than the old days when all of us
[00:14:36] costly than the old days when all of us always trained all of our models
[00:14:38] always trained all of our models literally from scratch i just don't know
[00:14:40] literally from scratch i just don't know how to do the calculations here but i do
[00:14:42] how to do the calculations here but i do know that increased access has been
[00:14:44] know that increased access has been empowering and is likely offsetting some
[00:14:46] empowering and is likely offsetting some of the costs and a lot of that is due to
[00:14:48] of the costs and a lot of that is due to the contributions of the hugging face
[00:14:50] the contributions of the hugging face library
[00:14:53] there are a lot of efforts along these
[00:14:54] there are a lot of efforts along these same lines to make bert smaller by
[00:14:56] same lines to make bert smaller by compressing it literally fewer
[00:14:58] compressing it literally fewer dimensions and other kinds of
[00:14:59] dimensions and other kinds of simplifications of the training process
[00:15:01] simplifications of the training process and burp distillation and so forth here
[00:15:04] and burp distillation and so forth here are two outstanding contributions kind
[00:15:05] are two outstanding contributions kind of compendiums of lots of different
[00:15:07] of compendiums of lots of different ideas in this space
[00:15:09] ideas in this space and i also highly recommend this lovely
[00:15:11] and i also highly recommend this lovely paper called the prime run virtology
[00:15:13] paper called the prime run virtology which explores a lot of different
[00:15:15] which explores a lot of different aspects of what we know about bert and
[00:15:17] aspects of what we know about bert and how it works
[00:15:18] how it works various variations people have tried
[00:15:21] various variations people have tried various things that people have done to
[00:15:22] various things that people have done to probe these models and understand their
[00:15:24] probe these models and understand their learning dynamics and so forth and so on
[00:15:26] learning dynamics and so forth and so on it's a very rich contribution certainly
[00:15:29] it's a very rich contribution certainly can be a resource for you as you think
[00:15:31] can be a resource for you as you think about these models
[00:15:34] and
[00:15:35] and just because we don't have time to cover
[00:15:37] just because we don't have time to cover them all there are a bunch of
[00:15:38] them all there are a bunch of interesting transformer variants that we
[00:15:40] interesting transformer variants that we will not be able to discuss in detail
[00:15:42] will not be able to discuss in detail i thought i'd mention them here esper is
[00:15:44] i thought i'd mention them here esper is an attempt to develop sentence level
[00:15:46] an attempt to develop sentence level representations from bert that are
[00:15:47] representations from bert that are particularly good at finding which
[00:15:49] particularly good at finding which sentences are similar to which other
[00:15:51] sentences are similar to which other sentences according to cosine similarity
[00:15:54] sentences according to cosine similarity i think that could be a powerful mode of
[00:15:55] i think that could be a powerful mode of thinking about these representations and
[00:15:58] thinking about these representations and also of practical utility if you need to
[00:16:00] also of practical utility if you need to find
[00:16:00] find which sentences are similar to which
[00:16:02] which sentences are similar to which others
[00:16:03] others you have probably heard of gpt the
[00:16:06] you have probably heard of gpt the generative pre-trained transformer in
[00:16:08] generative pre-trained transformer in various of its forms you can get gpt2
[00:16:11] various of its forms you can get gpt2 from hugging space and hugging face and
[00:16:13] from hugging space and hugging face and you have unfettered access to it and of
[00:16:15] you have unfettered access to it and of course there's more restrictive access
[00:16:17] course there's more restrictive access at this point to gpt3 these are
[00:16:19] at this point to gpt3 these are conditional language models so quite
[00:16:21] conditional language models so quite different from burt and they might be
[00:16:23] different from burt and they might be better than burnt for things like truly
[00:16:25] better than burnt for things like truly conditional language generation
[00:16:28] conditional language generation xlnet is an attempt to bring in much
[00:16:30] xlnet is an attempt to bring in much more context into these models
[00:16:33] more context into these models it stands for extra long transformer so
[00:16:35] it stands for extra long transformer so if you need to process long sequences
[00:16:37] if you need to process long sequences this might be a good choice and this is
[00:16:39] this might be a good choice and this is also an attempt to bring in some of the
[00:16:40] also an attempt to bring in some of the benefits of conditional language models
[00:16:43] benefits of conditional language models into a mode that is more bi-directional
[00:16:45] into a mode that is more bi-directional the way burt is
[00:16:47] the way burt is t5 is another conditional language mode
[00:16:49] t5 is another conditional language mode as is bart these models might be better
[00:16:51] as is bart these models might be better choices for you if you need to actually
[00:16:53] choices for you if you need to actually generate language where i think the
[00:16:55] generate language where i think the standard wisdom is that models like burt
[00:16:57] standard wisdom is that models like burt and roberta are better if you simply
[00:16:59] and roberta are better if you simply need good representations for fine
[00:17:01] need good representations for fine tuning on a classification problem for
[00:17:03] tuning on a classification problem for example
[00:17:05] example more models will appear every day and i
[00:17:07] more models will appear every day and i think it's worth trying to stay up to
[00:17:09] think it's worth trying to stay up to speed on the various developments in
[00:17:11] speed on the various developments in this space because this is probably just
[00:17:13] this space because this is probably just the tip of the iceberg here
[00:17:19] you