Sparse coding - Revision history

Iamozy: Merged to Neural coding

2014-03-05T18:49:36Z

Merged to Neural coding

← Previous revision		Revision as of 18:49, 5 March 2014
Line 1:		Line 1:
	~~{{Merge to\|~~Neural coding~~\|discuss=Talk:Neural~~ coding~~#Merger possibilities\|date=March 2014}}~~		#REDIRECT[[Neural coding#Sparse coding]]
	The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.

	As a consequence, sparseness may be focused on temporal sparseness ("a relatively small number of time periods are active") or on the sparseness in an activated population of neurons. In this latter case, this may be defined in one time period as the number of activated neurons relative to the total number of neurons in the population. This seems to be a hallmark of neural computations since compared to traditional computers, information is massively distributed across neurons. A major result in neural coding from Olshausen et al. is that sparse coding of natural images produces [[wavelet]]-like oriented filters that resemble the receptive fields of simple cells in the visual cortex.

	==Overview==
	Given a potentially large set of input patterns, sparse coding algorithms attempt to automatically find a small number of representative patterns which, when combined in the right proportions, reproduce the original input patterns. The sparse coding for the input then consists of those representative patterns. For example, the very large set of English sentences can be encoded by a small number of symbols (i.e. letters, numbers, punctuation, and spaces) combined in a particular order for a particular sentence, and so a sparse coding for English would be those symbols.

	==Linear Generative Model==
	Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=135–146\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.

	More formally, given a k-dimensional set of real-numbered input vectors <math>\vec{\xi }\in \mathbb{R}^{k}</math>, the goal of sparse coding is to determine n k-dimensional [[Basis (linear algebra)\|basis vectors]] <math>\vec{b_1}, \ldots, \vec{b_n} \in \mathbb{R}^{k}</math> along with a [[Sparse vector\|sparse]] n-dimensional vector of weights or coefficients <math>\vec{s} \in \mathbb{R}^{n}</math> for each input vector, so that a linear combination of the basis vectors with proportions given by the coefficients results in a close approximation to the input vector: <math>\vec{\xi} \approx \sum_{j=1}^{n} s_{j}\vec{b}_{j}</math>.<ref name=Lee>{{cite journal\|last1=Lee\|first1=Honglak\|last2=Battle\|first2=Alexis\|last3=Raina\|first3=Rajat\|last4=Ng\|first4=Andrew Y.\|title=Efficient sparse coding algorithms\|journal=Advances in Neural Information Processing Systems\|year=2006\|url=http://ai.stanford.edu/~hllee/nips06-sparsecoding.pdf}}</ref>

	The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>

	Another measure of coding is whether it is ''critically complete'' or ''overcomplete''. If the number of basis vectors n is equal to the dimensionality k of the input set, the coding is said to be critically complete. In this case, smooth changes in the input vector result in abrupt changes in the coefficients, and the coding is not able to gracefully handle small scalings, small translations, or noise in the inputs. If, however, the number of basis vectors is larger than the dimensionality of the input set, the coding is ''overcomplete''. Overcomplete codings smoothly interpolate between input vectors and are robust under input noise.<ref name=Olshausen>{{cite journal\|first1=Bruno A.\|last1=Olshausen\|first2=David J.\|last2=Field\|title=Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?\|journal=Vision Research\|year=1997\|volume=37\|number=23\|pages=3311–3325\|url=http://www.chaos.gwdg.de/~michael/CNS_course_2004/papers_max/OlshausenField1997.pdf}}</ref> The human primary [[visual cortex]] is estimated to be overcomplete by a factor of 500, so that, for example, a 14 x 14 patch of input (a 196-dimensional space) is coded by roughly 100,000 neurons.<ref name=Rehn/>

	==See also==
	* [[Rate coding]]
	* [[Independent-spike coding]]
	* [[Correlation coding]]
	* [[Population coding]]
	* [[Grandmother cell]]
	* [[Non-negative matrix factorization]]

	==References==
	{{reflist}}

	==Bibliography==
	* Földiák P, Endres D, [http://www.scholarpedia.org/article/Sparse_Coding Sparse coding], [[Scholarpedia]], 3(1):2984, 2008.
	* Dayan P & Abbott LF. ''Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems''. Cambridge, Massachusetts: The MIT Press; 2001. ISBN 0-262-04199-5
	* Rieke F, Warland D, de Ruyter van Steveninck R, Bialek W. ''Spikes: Exploring the Neural Code''. Cambridge, Massachusetts: The MIT Press; 1999. ISBN 0-262-68108-0
	* B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–9, jun 1996.

	[[Category:Neural coding]]

Tryptofish: /* top */ fix

2014-03-01T23:25:08Z

top: fix

← Previous revision		Revision as of 23:25, 1 March 2014
Line 1:		Line 1:
	{{Merge to\|Neural coding\|discuss=Talk:Neural coding#Merger possibilities\|date=~~February~~ 2014}}		{{Merge to\|Neural coding\|discuss=Talk:Neural coding#Merger possibilities\|date=March 2014}}
	The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.		The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.

Tryptofish: /* top */ update

2014-03-01T23:23:05Z

top: update

← Previous revision		Revision as of 23:23, 1 March 2014
Line 1:		Line 1:
	{{Merge to\|Neural coding\|discuss=Talk:Neural coding#Merger possibilities\|date=~~July~~ ~~2010~~}}		{{Merge to\|Neural coding\|discuss=Talk:Neural coding#Merger possibilities\|date=February 2014}}
	The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.		The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.

Tryptofish: Undid revision 597702270 by NickPenguin (talk) misread of discussion

2014-03-01T23:22:29Z

Undid revision 597702270 by NickPenguin (talk) misread of discussion

← Previous revision		Revision as of 23:22, 1 March 2014
Line 1:		Line 1:
			{{Merge to\|Neural coding\|discuss=Talk:Neural coding#Merger possibilities\|date=July 2010}}
	The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.		The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.

NickPenguin: no consensus to merge

2014-03-01T18:54:14Z

no consensus to merge

← Previous revision		Revision as of 18:54, 1 March 2014
Line 1:		Line 1:
	{{Merge to\|Neural coding\|discuss=Talk:Neural coding#Merger possibilities\|date=July 2010}}
	The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.		The '''sparse code''' is a kind of [[neural code]] in which each item is encoded by the strong activation of a relatively small set of neurons. For each item to be encoded, this is a different subset of all available neurons.

Qwertyus: /* See also */ NMF, which can be turned into a non-negative sparse coding

2013-10-27T12:38:09Z

See also: NMF, which can be turned into a non-negative sparse coding

← Previous revision		Revision as of 12:38, 27 October 2013
Line 22:		Line 22:
	* [[Population coding]]		* [[Population coding]]
	* [[Grandmother cell]]		* [[Grandmother cell]]
			* [[Non-negative matrix factorization]]

	==References==		==References==

Skoenig3 at 22:34, 22 April 2013

2013-04-22T22:34:52Z

← Previous revision		Revision as of 22:34, 22 April 2013
Line 14:		Line 14:
	The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>		The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>

	Another measure of coding is whether it is ''critically complete'' or ''overcomplete''. If the number of basis vectors n is equal to the dimensionality k of the input set, the coding is said to be critically complete. In this case, smooth changes in the input vector result in abrupt changes in the coefficients, and the coding is not able to gracefully handle small scalings, small translations, or noise in the inputs. If, however, the number of basis vectors is larger than the dimensionality of the input set, the coding is ''overcomplete''. Overcomplete codings smoothly interpolate between input vectors and are robust under input noise.<ref name=~~Olhausen~~>{{cite journal\|first1=Bruno A.\|last1=~~Olhausen~~\|first2=David J.\|last2=Field\|title=Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?\|journal=Vision Research\|year=1997\|volume=37\|number=23\|pages=3311–3325\|url=http://www.chaos.gwdg.de/~michael/CNS_course_2004/papers_max/OlshausenField1997.pdf}}</ref> The human primary [[visual cortex]] is estimated to be overcomplete by a factor of 500, so that, for example, a 14 x 14 patch of input (a 196-dimensional space) is coded by roughly 100,000 neurons.<ref name=Rehn/>		Another measure of coding is whether it is ''critically complete'' or ''overcomplete''. If the number of basis vectors n is equal to the dimensionality k of the input set, the coding is said to be critically complete. In this case, smooth changes in the input vector result in abrupt changes in the coefficients, and the coding is not able to gracefully handle small scalings, small translations, or noise in the inputs. If, however, the number of basis vectors is larger than the dimensionality of the input set, the coding is ''overcomplete''. Overcomplete codings smoothly interpolate between input vectors and are robust under input noise.<ref name=Olshausen>{{cite journal\|first1=Bruno A.\|last1=Olshausen\|first2=David J.\|last2=Field\|title=Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?\|journal=Vision Research\|year=1997\|volume=37\|number=23\|pages=3311–3325\|url=http://www.chaos.gwdg.de/~michael/CNS_course_2004/papers_max/OlshausenField1997.pdf}}</ref> The human primary [[visual cortex]] is estimated to be overcomplete by a factor of 500, so that, for example, a 14 x 14 patch of input (a 196-dimensional space) is coded by roughly 100,000 neurons.<ref name=Rehn/>

	==See also==		==See also==

Addbot: Bot: Removing {{Stub}} (Report Errors)

2013-02-14T10:20:42Z

Bot: Removing {{Stub}} (Report Errors)

← Previous revision		Revision as of 10:20, 14 February 2013
Line 5:		Line 5:

	==Overview==		==Overview==

	Given a potentially large set of input patterns, sparse coding algorithms attempt to automatically find a small number of representative patterns which, when combined in the right proportions, reproduce the original input patterns. The sparse coding for the input then consists of those representative patterns. For example, the very large set of English sentences can be encoded by a small number of symbols (i.e. letters, numbers, punctuation, and spaces) combined in a particular order for a particular sentence, and so a sparse coding for English would be those symbols.		Given a potentially large set of input patterns, sparse coding algorithms attempt to automatically find a small number of representative patterns which, when combined in the right proportions, reproduce the original input patterns. The sparse coding for the input then consists of those representative patterns. For example, the very large set of English sentences can be encoded by a small number of symbols (i.e. letters, numbers, punctuation, and spaces) combined in a particular order for a particular sentence, and so a sparse coding for English would be those symbols.

	==Linear Generative Model==		==Linear Generative Model==

	Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=135–146\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.		Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=135–146\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.

Line 19:		Line 17:

	==See also==		==See also==
	*[[Rate coding]]		* [[Rate coding]]
	*[[Independent-spike coding]]		* [[Independent-spike coding]]
	*[[Correlation coding]]		* [[Correlation coding]]
	*[[Population coding]]		* [[Population coding]]
	*[[Grandmother cell]]		* [[Grandmother cell]]

	==References==		==References==
			{{reflist}}
	<references/>

	==Bibliography==		==Bibliography==
Line 35:		Line 33:

	[[Category:Neural coding]]		[[Category:Neural coding]]
	{{neuroscience-stub}}

RjwilmsiBot: /* Linear Generative Model */fixing page range dashes using AWB (8010)

2012-04-10T12:12:23Z

Linear Generative Model: fixing page range dashes using AWB (8010)

← Previous revision		Revision as of 12:12, 10 April 2012
Line 10:		Line 10:
	==Linear Generative Model==		==Linear Generative Model==

	Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=~~135-146~~\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.		Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=135–146\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.

	More formally, given a k-dimensional set of real-numbered input vectors <math>\vec{\xi }\in \mathbb{R}^{k}</math>, the goal of sparse coding is to determine n k-dimensional [[Basis (linear algebra)\|basis vectors]] <math>\vec{b_1}, \ldots, \vec{b_n} \in \mathbb{R}^{k}</math> along with a [[Sparse vector\|sparse]] n-dimensional vector of weights or coefficients <math>\vec{s} \in \mathbb{R}^{n}</math> for each input vector, so that a linear combination of the basis vectors with proportions given by the coefficients results in a close approximation to the input vector: <math>\vec{\xi} \approx \sum_{j=1}^{n} s_{j}\vec{b}_{j}</math>.<ref name=Lee>{{cite journal\|last1=Lee\|first1=Honglak\|last2=Battle\|first2=Alexis\|last3=Raina\|first3=Rajat\|last4=Ng\|first4=Andrew Y.\|title=Efficient sparse coding algorithms\|journal=Advances in Neural Information Processing Systems\|year=2006\|url=http://ai.stanford.edu/~hllee/nips06-sparsecoding.pdf}}</ref>		More formally, given a k-dimensional set of real-numbered input vectors <math>\vec{\xi }\in \mathbb{R}^{k}</math>, the goal of sparse coding is to determine n k-dimensional [[Basis (linear algebra)\|basis vectors]] <math>\vec{b_1}, \ldots, \vec{b_n} \in \mathbb{R}^{k}</math> along with a [[Sparse vector\|sparse]] n-dimensional vector of weights or coefficients <math>\vec{s} \in \mathbb{R}^{n}</math> for each input vector, so that a linear combination of the basis vectors with proportions given by the coefficients results in a close approximation to the input vector: <math>\vec{\xi} \approx \sum_{j=1}^{n} s_{j}\vec{b}_{j}</math>.<ref name=Lee>{{cite journal\|last1=Lee\|first1=Honglak\|last2=Battle\|first2=Alexis\|last3=Raina\|first3=Rajat\|last4=Ng\|first4=Andrew Y.\|title=Efficient sparse coding algorithms\|journal=Advances in Neural Information Processing Systems\|year=2006\|url=http://ai.stanford.edu/~hllee/nips06-sparsecoding.pdf}}</ref>
Line 16:		Line 16:
	The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>		The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>

	Another measure of coding is whether it is ''critically complete'' or ''overcomplete''. If the number of basis vectors n is equal to the dimensionality k of the input set, the coding is said to be critically complete. In this case, smooth changes in the input vector result in abrupt changes in the coefficients, and the coding is not able to gracefully handle small scalings, small translations, or noise in the inputs. If, however, the number of basis vectors is larger than the dimensionality of the input set, the coding is ''overcomplete''. Overcomplete codings smoothly interpolate between input vectors and are robust under input noise.<ref name=Olhausen>{{cite journal\|first1=Bruno A.\|last1=Olhausen\|first2=David J.\|last2=Field\|title=Sparse Coding with an Overcomplete Basis Set: A		Another measure of coding is whether it is ''critically complete'' or ''overcomplete''. If the number of basis vectors n is equal to the dimensionality k of the input set, the coding is said to be critically complete. In this case, smooth changes in the input vector result in abrupt changes in the coefficients, and the coding is not able to gracefully handle small scalings, small translations, or noise in the inputs. If, however, the number of basis vectors is larger than the dimensionality of the input set, the coding is ''overcomplete''. Overcomplete codings smoothly interpolate between input vectors and are robust under input noise.<ref name=Olhausen>{{cite journal\|first1=Bruno A.\|last1=Olhausen\|first2=David J.\|last2=Field\|title=Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?\|journal=Vision Research\|year=1997\|volume=37\|number=23\|pages=3311–3325\|url=http://www.chaos.gwdg.de/~michael/CNS_course_2004/papers_max/OlshausenField1997.pdf}}</ref> The human primary [[visual cortex]] is estimated to be overcomplete by a factor of 500, so that, for example, a 14 x 14 patch of input (a 196-dimensional space) is coded by roughly 100,000 neurons.<ref name=Rehn/>
	Strategy Employed by V1?\|journal=Vision Research\|year=1997\|volume=37\|number=23\|pages=3311-3325\|url=http://www.chaos.gwdg.de/~michael/CNS_course_2004/papers_max/OlshausenField1997.pdf}}</ref> The human primary [[visual cortex]] is estimated to be overcomplete by a factor of 500, so that, for example, a 14 x 14 patch of input (a 196-dimensional space) is coded by roughly 100,000 neurons.<ref name=Rehn/>

	==See also==		==See also==

Robert.Baruch: /* Linear Generative Model */ Corrects wikilink to Basis

2012-03-08T14:58:07Z

Linear Generative Model: Corrects wikilink to Basis

← Previous revision		Revision as of 14:58, 8 March 2012
Line 12:		Line 12:
	Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=135-146\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.		Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=135-146\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.

	More formally, given a k-dimensional set of real-numbered input vectors <math>\vec{\xi }\in \mathbb{R}^{k}</math>, the goal of sparse coding is to determine n k-dimensional [[Basis\|basis vectors]] <math>\vec{b_1}, \ldots, \vec{b_n} \in \mathbb{R}^{k}</math> along with a [[Sparse vector\|sparse]] n-dimensional vector of weights or coefficients <math>\vec{s} \in \mathbb{R}^{n}</math> for each input vector, so that a linear combination of the basis vectors with proportions given by the coefficients results in a close approximation to the input vector: <math>\vec{\xi} \approx \sum_{j=1}^{n} s_{j}\vec{b}_{j}</math>.<ref name=Lee>{{cite journal\|last1=Lee\|first1=Honglak\|last2=Battle\|first2=Alexis\|last3=Raina\|first3=Rajat\|last4=Ng\|first4=Andrew Y.\|title=Efficient sparse coding algorithms\|journal=Advances in Neural Information Processing Systems\|year=2006\|url=http://ai.stanford.edu/~hllee/nips06-sparsecoding.pdf}}</ref>		More formally, given a k-dimensional set of real-numbered input vectors <math>\vec{\xi }\in \mathbb{R}^{k}</math>, the goal of sparse coding is to determine n k-dimensional [[Basis (linear algebra)\|basis vectors]] <math>\vec{b_1}, \ldots, \vec{b_n} \in \mathbb{R}^{k}</math> along with a [[Sparse vector\|sparse]] n-dimensional vector of weights or coefficients <math>\vec{s} \in \mathbb{R}^{n}</math> for each input vector, so that a linear combination of the basis vectors with proportions given by the coefficients results in a close approximation to the input vector: <math>\vec{\xi} \approx \sum_{j=1}^{n} s_{j}\vec{b}_{j}</math>.<ref name=Lee>{{cite journal\|last1=Lee\|first1=Honglak\|last2=Battle\|first2=Alexis\|last3=Raina\|first3=Rajat\|last4=Ng\|first4=Andrew Y.\|title=Efficient sparse coding algorithms\|journal=Advances in Neural Information Processing Systems\|year=2006\|url=http://ai.stanford.edu/~hllee/nips06-sparsecoding.pdf}}</ref>

	The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>		The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>

← Previous revision		Revision as of 14:58, 8 March 2012
Line 12:		Line 12:
	Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=135-146\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.		Most models of sparse coding are based on the linear generative model.<ref name=Rehn>{{cite journal\|first1=Martin\|last1=Rehn\|first2=Friedrich T.\|last2=Sommer\|title=A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive ﬁelds\|journal=Journal of Computational Neuroscience\|year=2007\|volume=22\|pages=135-146\|doi=10.1007/s10827-006-0003-9\|url=http://redwood.berkeley.edu/fsommer/papers/rehnsommer07jcns.pdf}}</ref> In this model, the symbols are combined in a [[Linear combination\|linear fashion]] to approximate the input.

	More formally, given a k-dimensional set of real-numbered input vectors <math>\vec{\xi }\in \mathbb{R}^{k}</math>, the goal of sparse coding is to determine n k-dimensional [[Basis\|basis vectors]] <math>\vec{b_1}, \ldots, \vec{b_n} \in \mathbb{R}^{k}</math> along with a [[Sparse vector\|sparse]] n-dimensional vector of weights or coefficients <math>\vec{s} \in \mathbb{R}^{n}</math> for each input vector, so that a linear combination of the basis vectors with proportions given by the coefficients results in a close approximation to the input vector: <math>\vec{\xi} \approx \sum_{j=1}^{n} s_{j}\vec{b}_{j}</math>.<ref name=Lee>{{cite journal\|last1=Lee\|first1=Honglak\|last2=Battle\|first2=Alexis\|last3=Raina\|first3=Rajat\|last4=Ng\|first4=Andrew Y.\|title=Efficient sparse coding algorithms\|journal=Advances in Neural Information Processing Systems\|year=2006\|url=http://ai.stanford.edu/~hllee/nips06-sparsecoding.pdf}}</ref>		More formally, given a k-dimensional set of real-numbered input vectors <math>\vec{\xi }\in \mathbb{R}^{k}</math>, the goal of sparse coding is to determine n k-dimensional [[Basis (linear algebra)\|basis vectors]] <math>\vec{b_1}, \ldots, \vec{b_n} \in \mathbb{R}^{k}</math> along with a [[Sparse vector\|sparse]] n-dimensional vector of weights or coefficients <math>\vec{s} \in \mathbb{R}^{n}</math> for each input vector, so that a linear combination of the basis vectors with proportions given by the coefficients results in a close approximation to the input vector: <math>\vec{\xi} \approx \sum_{j=1}^{n} s_{j}\vec{b}_{j}</math>.<ref name=Lee>{{cite journal\|last1=Lee\|first1=Honglak\|last2=Battle\|first2=Alexis\|last3=Raina\|first3=Rajat\|last4=Ng\|first4=Andrew Y.\|title=Efficient sparse coding algorithms\|journal=Advances in Neural Information Processing Systems\|year=2006\|url=http://ai.stanford.edu/~hllee/nips06-sparsecoding.pdf}}</ref>

	The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>		The codings generated by algorithms implementing a linear generative model can be classified into codings with ''soft sparseness'' and those with ''hard sparseness''.<ref name=Rehn/> These refer to the distribution of basis vector coefficients for typical inputs. A coding with soft sparseness has a smooth [[Normal distribution\|Gaussian]]-like distribution, but peakier than Gaussian, with many zero values, some small absolute values, fewer larger absolute values, and very few very large absolute values. Thus, many of the basis vectors are active. Hard sparseness, on the other hand, indicates that there are many zero values, ''no'' or ''hardly any'' small absolute values, fewer larger absolute values, and very few very large absolute values, and thus few of the basis vectors are active. This is appealing from a metabolic perspective: less energy is used when fewer neurons are firing.<ref name=Rehn/>