Word embedding: Difference between revisions
Content deleted Content added
No edit summary |
No edit summary |
||
| Line 3: | Line 3: | ||
'''Word embedding''' is the collective name for a set of [[language model]]ing and [[feature learning]] techniques in [[natural language processing]] where words or phrases from the vocabulary are mapped to vectors of real numbers in a low-dimensional space relative to the vocabulary size ("continuous space"). |
'''Word embedding''' is the collective name for a set of [[language model]]ing and [[feature learning]] techniques in [[natural language processing]] where words or phrases from the vocabulary are mapped to vectors of real numbers in a low-dimensional space relative to the vocabulary size ("continuous space"). |
||
Methods to generate this mapping include [[neural net language model|neural networks]],<ref>{{cite arXiv |eprint=1310.4546 |last1=Mikolov |first1=Tomas |title=Distributed Representations of Words and Phrases and their Compositionality |last2=Sutskever |first2=Ilya |last3=Chen |first3=Kai |last4=Corrado |first4=Greg |last5=Dean |first5=Jeffrey |class=cs.CL| year=2013}}</ref><ref |
Methods to generate this mapping include [[neural net language model|neural networks]],<ref>{{cite arXiv |eprint=1310.4546 |last1=Mikolov |first1=Tomas |title=Distributed Representations of Words and Phrases and their Compositionality |last2=Sutskever |first2=Ilya |last3=Chen |first3=Kai |last4=Corrado |first4=Greg |last5=Dean |first5=Jeffrey |class=cs.CL| year=2013}}</ref><ref>{{cite arXiv |eprint=1603.06571 |last1=Barkan |first1=Oren |title=Bayesian Neural Word Embedding |class=cs.CL| year=2015}}</ref> [[dimensionality reduction]] on the word co-occurrence matrix,<ref>{{cite arXiv |eprint=1312.5542 |last1=Lebret |first1=Rémi |title=Word Emdeddings through Hellinger PCA |last2=Collobert |first2=Ronan |class=cs.CL |year=2013}}</ref><ref>{{Cite conference |url=http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization.pdf |title=Neural Word Embedding as Implicit Matrix Factorization |last=Levy |first=Omer |conference=NIPS |year=2014 |last2=Goldberg |first2=Yoav}}</ref><ref>{{Cite conference |url=http://ijcai.org/papers15/Papers/IJCAI15-513.pdf |title=Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization Perspective |last=Li |first=Yitan |conference=Int'l J. Conf. on Artificial Intelligence (IJCAI) |year=2015 |last2=Xu |first2=Linli}}</ref> and explicit representation in terms of the context in which words appear.<ref>{{cite conference |last1=Levy |first1=Omer |last2=Goldberg |first2=Yoav |title=Linguistic Regularities in Sparse and Explicit Word Representations |conference=CoNLL |pages=171–180 |year=2014 |url=https://levyomer.files.wordpress.com/2014/04/linguistic-regularities-in-sparse-and-explicit-word-representations-conll-2014.pdf}}</ref> |
||
Word and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance in NLP tasks such as [[syntactic parsing]]<ref>{{cite conference |last1=Socher |first1=Richard |last2=Bauer |first2=John |last3=Manning |first3=Christopher |last4=Ng |first4=Andrew |title=Parsing with compositional vector grammars |conference=Proc. ACL Conf. |year=2013 |url=http://www.socher.org/uploads/Main/SocherBauerManningNg_ACL2013.pdf}}</ref> and [[sentiment analysis]].<ref>{{cite conference |last1=Socher |first1=Richard |last2=Perelygin |first2=Alex |last3=Wu |first3=Jean |last4=Chuang |first4=Jason |last5=Manning |first5=Chris |last6=Ng |first6=Andrew |last7=Potts |first7=Chris |title=Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank |conference=EMNLP |year=2013 |url=http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf}}</ref> |
Word and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance in NLP tasks such as [[syntactic parsing]]<ref>{{cite conference |last1=Socher |first1=Richard |last2=Bauer |first2=John |last3=Manning |first3=Christopher |last4=Ng |first4=Andrew |title=Parsing with compositional vector grammars |conference=Proc. ACL Conf. |year=2013 |url=http://www.socher.org/uploads/Main/SocherBauerManningNg_ACL2013.pdf}}</ref> and [[sentiment analysis]].<ref>{{cite conference |last1=Socher |first1=Richard |last2=Perelygin |first2=Alex |last3=Wu |first3=Jean |last4=Chuang |first4=Jason |last5=Manning |first5=Chris |last6=Ng |first6=Andrew |last7=Potts |first7=Chris |title=Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank |conference=EMNLP |year=2013 |url=http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf}}</ref> |
||
Revision as of 06:58, 23 March 2016
| Part of a series on |
| Machine learning and data mining |
|---|
Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing where words or phrases from the vocabulary are mapped to vectors of real numbers in a low-dimensional space relative to the vocabulary size ("continuous space").
Methods to generate this mapping include neural networks,[1][2] dimensionality reduction on the word co-occurrence matrix,[3][4][5] and explicit representation in terms of the context in which words appear.[6]
Word and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance in NLP tasks such as syntactic parsing[7] and sentiment analysis.[8]
Software
Software for training and using word embeddings includes Google's Word2vec, Stanford University's GloVe[9] and Deeplearning4j.
See also
References
- ^ Mikolov, Tomas; Sutskever, Ilya; Chen, Kai; Corrado, Greg; Dean, Jeffrey (2013). "Distributed Representations of Words and Phrases and their Compositionality". arXiv:1310.4546 [cs.CL].
- ^ Barkan, Oren (2015). "Bayesian Neural Word Embedding". arXiv:1603.06571 [cs.CL].
- ^ Lebret, Rémi; Collobert, Ronan (2013). "Word Emdeddings through Hellinger PCA". arXiv:1312.5542 [cs.CL].
- ^ Levy, Omer; Goldberg, Yoav (2014). Neural Word Embedding as Implicit Matrix Factorization (PDF). NIPS.
- ^ Li, Yitan; Xu, Linli (2015). Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization Perspective (PDF). Int'l J. Conf. on Artificial Intelligence (IJCAI).
- ^ Levy, Omer; Goldberg, Yoav (2014). Linguistic Regularities in Sparse and Explicit Word Representations (PDF). CoNLL. pp. 171–180.
- ^ Socher, Richard; Bauer, John; Manning, Christopher; Ng, Andrew (2013). Parsing with compositional vector grammars (PDF). Proc. ACL Conf.
- ^ Socher, Richard; Perelygin, Alex; Wu, Jean; Chuang, Jason; Manning, Chris; Ng, Andrew; Potts, Chris (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank (PDF). EMNLP.
- ^ "GloVe".