*Result*: Inception loops discover what excites neurons most using deep predictive models.
Title:
Inception loops discover what excites neurons most using deep predictive models.
Authors:
Walker EY; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA. eywalker@bcm.edu.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA. eywalker@bcm.edu., Sinz FH; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA. fabian.sinz@uni-tuebingen.de.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA. fabian.sinz@uni-tuebingen.de.; Bernstein Center for Computational Neuroscience, University of Tübingen, Tübingen, Germany. fabian.sinz@uni-tuebingen.de.; Institute for Bioinformatics and Medical Informatics, University of Tübingen, Tübingen, Germany. fabian.sinz@uni-tuebingen.de., Cobos E; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA., Muhammad T; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA., Froudarakis E; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA., Fahey PG; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA., Ecker AS; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA.; Bernstein Center for Computational Neuroscience, University of Tübingen, Tübingen, Germany.; Centre for Integrative Neuroscience, University of Tübingen, Tübingen, Germany.; Institute for Theoretical Physics, University of Tübingen, Tübingen, Germany., Reimer J; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA., Pitkow X; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA.; Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA., Tolias AS; Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA. astolias@bcm.edu.; Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA. astolias@bcm.edu.; Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA. astolias@bcm.edu.
Source:
Nature neuroscience [Nat Neurosci] 2019 Dec; Vol. 22 (12), pp. 2060-2065. Date of Electronic Publication: 2019 Nov 04.
Publication Type:
Journal Article; Research Support, N.I.H., Extramural; Research Support, Non-U.S. Gov't
Language:
English
Journal Info:
Publisher: Nature Publishing Group Country of Publication: United States NLM ID: 9809671 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1546-1726 (Electronic) Linking ISSN: 10976256 NLM ISO Abbreviation: Nat Neurosci Subsets: MEDLINE
Imprint Name(s):
Publication: <2002->: New York, NY : Nature Publishing Group
Original Publication: New York, NY : Nature America Inc., c1998-
Original Publication: New York, NY : Nature America Inc., c1998-
MeSH Terms:
References:
Adrian, E. D. & Bronk, D. W. The discharge of impulses in motor nerve fibres: Part I. Impulses in single fibres of the phrenic nerve. J. Physiol. 66, 81–101 (1928). (PMID: 169939761402751)
Hartline, H. K. The response of single optic nerve fibers of the vertebrate eye to illumination of the retina. Am. J. Physiol. 121, 400–415 (1938).
Chichilnisky, E. J. A simple white noise analysis of neuronal light responses. Network 12, 199–213 (2001). (PMID: 11405422)
Olshausen, B. A. & Field, D. J. in Problems in Systems Neuroscience (eds Sejnowski, T. J. & van Hemmen, L.) 182–211 (Oxford Univ. Press, 2004).
Antolík, J., Hofer, S. B., Bednar, J. A. & Mrsic-flogel, T. D. Model constrained by visual hierarchy improves prediction of neural responses to natural scenes. PLoS Comput. Biol. 12, e1004927 (2016). (PMID: 273485484922657)
Sinz, F. et al. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. In Proc. Advances in Neural Information Processing Systems 31 (eds Bengio, S. et al.) 7199–7210 (Curran Associates, 2018).
Harth, E. & Tzanakou, E. ALOPEX: a stochastic method for determining visual receptive fields. Vision Res. 14, 1475–1482 (1974). (PMID: 4446379)
Földiák, P. Stimulus optimisation in primary visual cortex. Neurocomputing 38–40, 1217–1222 (2001).
Paninski, L., Pillow, J. & Lewi, J. in Computational Neuroscience: Theoretical Insights into Brain Function (eds Cisek, P. et al.) 493–507 (Elsevier, 2007).
Benda, J., Gollisch, T., Machens, C. K. & Herz, A. V. From response to stimulus: adaptive sampling in sensory physiology. Curr. Opin. Neurobiol. 17, 430–436 (2007). (PMID: 17689952)
Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016). (PMID: 26906502)
Cadieu, C. F. et al. Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS Comput. Biol. 10, e1003963 (2014). (PMID: 255212944270441)
Klindt, D., Ecker, A. S., Euler, T. & Bethge, M. Neural system identification for large populations separating “what” and “where”. Adv. Neural Inf. Process. Syst. 30, 3506–3516 (2017).
McIntosh, L. T., Maheswaranathan, N., Nayebi, A., Ganguli, S. & Baccus, S. A. Deep learning models of the retinal response to natural scenes. Adv. Neural Inf. Process. Syst. 29, 1369–1377 (2016). (PMID: 287297795515384)
Erhan, D. & Bengio, Y. & Courville, A. & Vincent, P. Visualizing higher-layer features of a deep network. Technical Report 1341 (University of Montreal, 2009).
Sofroniew, N. J., Flickinger, D., King, J. & Svoboda, K. A large field of view two-photon mesoscope with subcellular resolution for in vivo imaging. eLife 5, e14472 (2016). (PMID: 273001054951199)
Cadena, S. A. et al. Deep convolutional models improve predictions of macaque V1 responses to natural images.PLoS Comput. Biol. 15, e1006897 (2019). (PMID: 310132786499433)
Kindel, W. F., Christensen, E. D. & Zylberberg, J. Using deep learning to probe the neural code for images in primary visual cortex. J. Vis. 19, 29 (2019). (PMID: 310260166485988)
Zhang, Y., Lee, T. S., Li, M., Liu, F. & Tang, S. Convolutional neural network models of V1 responses to complex patterns. J. Comput. Neurosci. 46, 33–54 (2019). (PMID: 29869761)
Adelson, E. H. & Bergen, J. R. Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A 2, 284–299 (1985). (PMID: 3973762)
Hubel, D. H. & Wiesel, T. N. Receptive fields of single neurones in the cat’s striate cortex. J. Physiol. 148, 574–591 (1959). (PMID: 144036791363130)
Simoncelli, E. P. & Olshausen, B. A. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24, 1193–1216 (2001).
Lindsey, J., Ocko, S. A., Ganguli, S. & Deny, S. A unified theory of early visual representations from retina to cortex through anatomically constrained deep CNNs. Preprint at bioRxiv https://www.biorxiv.org/content/10.1101/511535v1.full (2019).
DiCarlo, J. J. & Cox, D. D. Untangling invariant object recognition. Trends Cogn. Sci. 11, 333–341 (2007). (PMID: 17631409)
Sabour, S., Frosst, N. & Hinton, G. E. Dynamic routing between capsules. In Proc. Advances in Neural Information Processing Systems 30 (eds Guyon, I. et al.) 3856–3866 (2017).
Lehky, S. R. & Sejnowski, T. J. & Desimone, R. Predicting responses of nonlinear neurons in monkey striate cortex to complex patterns.J. Neurosci. 12, 3568–3581 (1992). (PMID: 15275966575725)
Ecker, A. S. et al. A rotation-equivariant convolutional neural network model of primary visual cortex. International Conference on Learning Representations (ICLR) 2019 Conference Poster https://openreview.net/forum?id=H1fU8iAqKX (2018).
Pasupathy, A. & Connor, C. E. Population coding of shape in area V4. Nat. Neurosci. 5, 1332–1338 (2002). (PMID: 12426571)
Abbasi-Asl, R. et al. The DeepTune framework for modeling and characterizing neurons in visual cortex area V4. Preprint at bioRxiv https://www.biorxiv.org/content/biorxiv/early/2018/11/09/465534.full.pdf (2018).
Bashivan, P., Kar, K. & DiCarlo, J. J. Neural population control via deep image synthesis. Science 364, eaav9436 (2019). (PMID: 31048462)
Ponce, C. R. et al. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. cell 177, 999–1009.e10 (2019). (PMID: 31051108)
Reimer, J. et al. Pupil fluctuations track fast switching of cortical states during quiet wakefulness. Neuron 84, 355–362 (2014). (PMID: 253743594323337)
Froudarakis, E. et al. Population code in mouse v1 facilitates readout of natural scenes through increased sparseness. Nat. Neurosci. 17, 851–857 (2014). (PMID: 247475774106281)
Garrett, M. E., Nauhaus, I., Marshel, J. H. & Callaway, E. M. Topography and areal organization of mouse visual cortex. J. Neurosci. 34, 12587–12600 (2014). (PMID: 252092964160785)
Pnevmatikakis, E. A. et al. Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron 89, 285–299 (2016). (PMID: 267741604881387)
Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. in Proceedings of the 32nd International Conference on Machine Learning, Lille, France 37, 448–456 (2015).
Clevert, D.-A., Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). Preprint at arXiv https://arxiv.org/pdf/1511.07289.pdf (2015).
Jaderberg, M., Simonyan, K., Zisserman, A. & Kavukcuoglu, K. Spatial transformer networks. In Proc. Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2017–2025 (Curran Associates, 2015).
McGinley, M. J. et al. Waking state: rapid variations modulate neural and behavioral responses. Neuron 87, 1143–1161 (2015). (PMID: 264026004718218)
Fu, Y. et al. A cortical circuit for gain control by behavioral state. Cell 156, 1139–1152 (2014). (PMID: 246307184041382)
Zoccolan, D., Graham, B. & Cox, D. A self-calibrating, camera-based eye tracker for the recording of rodent eye movements. Front. Neurosci. 4, 193 (2010). (PMID: 211522592998901)
Stahl, J. S., van Alphen, A. M. & De Zeeuw, C. I. A comparison of video and magnetic search coil recordings of mouse eye movements. J. Neurosci. Methods 99, 101–110 (2000). (PMID: 10936649)
van Alphen, B., Winkelman, B. H. & Frens, M. A. Three-dimensional optokinetic eye movements in the C57BL/6J mouse. Invest. Ophthalmol. Vis. Sci. 51, 623–630 (2010). (PMID: 19696183)
Prechelt, L. Early stopping — but when? in Neural Networks: Tricks of the Trade (eds Montavon, G., Orr, G., & Müller, K.-R.) 53–67 (Springer, 1998).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at arXiv https://arxiv.org/pdf/1412.6980.pdf (2017).
Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T. & Clune, J. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Adv. Neural Inf. Process. Syst. 29, 3387–3395 (2016).
Nguyen, A. M., Yosinski, J. & Clune, J. Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks. Preprint at arXiv https://arxiv.org/pdf/1602.03616.pdf (2016).
Wei, D., Zhou, B., Torralba, A. & Freeman, W. T. Understanding intra-class knowledge inside CNN. Preprint at arXiv https://arxiv.org/pdf/1507.02379.pdf (2015).
Olah, C., Mordvintsev, A. & Schubert, L. Feature visualization: how neural networks build up their understanding of images. Distill https://distill.pub/2017/feature-visualization (2017).
Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. International Conference on Learning Representations (ICLR) Workshop Paper https://arxiv.org/abs/1312.6034 (2014).
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K.-R. & Dähne, S. Learning how to explain neural networks: PatternNet and PatternAttribution. Preprint at arXiv https://arxiv.org/pdf/1705.05598.pdf (2017).
Yosinski, J., Clune, J., Nguyen, A., Fuchs, T. & Lipson, H. Understanding neural networks through deep visualization. Preprint at arXiv https://arxiv.org/pdf/1506.06579.pdf (2015).
Gatys, L. A., Ecker, A. S. & Bethge, M. A neural algorithm of artistic style. Preprint at arXiv https://arxiv.org/pdf/1508.06576.pdf (2015).
Mahendran, A. & Vedaldi, A. Understanding deep image representations by inverting them. Preprint at arXiv https://arxiv.org/pdf/1412.0035.pdf (2015).
Lenc, K. & Vedaldi, A. Understanding image representations by measuring their equivariance and equivalence. Preprint at arXiv https://arxiv.org/pdf/1411.5908.pdf (2015).
Tsai, C.-Y. & Cox, D. D. Characterizing visual representations within convolutional neural networks: toward a quantitative approach. In Proc. Workshop on Visualization for Deep Learning, 33rd International Conference on Machine Learning (2016).
Øygard, A. Visualizing GoogLeNet classes. Audun M. Øygard Blog https://www.auduno.com/2015/07/29/visualizing-googlenet-classes/ (2015).
Sreedhar, K. & Panlal, B. Enhancement of images using morphological transformations. Int. J. Comput. Sci. Inf. Technol. 4, 33–50 (2012).
Pologruto, T. A., Sabatini, B. L. & Svoboda, K. Scanimage: flexible software for operating laser scanning microscopes. Biomed. Eng. Online 2, 13 (2003). (PMID: 12801419161784)
Giovannucci, A. et al. Caiman: an open source tool for scalable calcium imaging data analysis. eLife 8, e38173 (2019). (PMID: 306526836342523)
Yatsenko, D., Walker, E. Y. & Tolias, A. S. Datajoint: a simpler relational data model. Preprint at arXiv https://arxiv.org/pdf/1807.11104.pdf (2018).
Paszke, A. et al. Automatic differentiation in PyTorch. In Proc. Advances in Neural Information Processing Systems (NIPS) 31 Workshop Autodiff Submission (2017).
van der Walt, S., Colbert, S. C. & Varoquaux, G. The NumPy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13, 22–30 (2011).
Jones, E. et al. SciPy: Open Source Scientific Tools for Python http://www.scipy.org (SciPy.org, accessed 3 October 2019).
Merkel, D. Docker: lightweight Linux containers for consistent development and deployment. Linux J. 239, 2 (2014).
Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90–95 (2007).
Waskom, M. et al. mwaskom/seaborn: v.0.8.1 (September 2017). Zenodo https://zenodo.org/record/883859#.XZXIjUZKguV (2017).
McKinney, W. Data structures for statistical computing in Python. In Proc. 9th Python in Science Conference (eds van der Walt, S. & Millman, J.) 51–56 (2010).
Kluyver, T. et al. Jupyter notebooks: a publishing format for reproducible computational workflows. In Proc. 20th International Conference on Electronic Publishing. Positioning and Power in Academic Publishing: Players, Agents and Agendas (eds Loizides, F. & Schmidt, B.) 87–90 (IOS Press, 2016).
Hartline, H. K. The response of single optic nerve fibers of the vertebrate eye to illumination of the retina. Am. J. Physiol. 121, 400–415 (1938).
Chichilnisky, E. J. A simple white noise analysis of neuronal light responses. Network 12, 199–213 (2001). (PMID: 11405422)
Olshausen, B. A. & Field, D. J. in Problems in Systems Neuroscience (eds Sejnowski, T. J. & van Hemmen, L.) 182–211 (Oxford Univ. Press, 2004).
Antolík, J., Hofer, S. B., Bednar, J. A. & Mrsic-flogel, T. D. Model constrained by visual hierarchy improves prediction of neural responses to natural scenes. PLoS Comput. Biol. 12, e1004927 (2016). (PMID: 273485484922657)
Sinz, F. et al. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. In Proc. Advances in Neural Information Processing Systems 31 (eds Bengio, S. et al.) 7199–7210 (Curran Associates, 2018).
Harth, E. & Tzanakou, E. ALOPEX: a stochastic method for determining visual receptive fields. Vision Res. 14, 1475–1482 (1974). (PMID: 4446379)
Földiák, P. Stimulus optimisation in primary visual cortex. Neurocomputing 38–40, 1217–1222 (2001).
Paninski, L., Pillow, J. & Lewi, J. in Computational Neuroscience: Theoretical Insights into Brain Function (eds Cisek, P. et al.) 493–507 (Elsevier, 2007).
Benda, J., Gollisch, T., Machens, C. K. & Herz, A. V. From response to stimulus: adaptive sampling in sensory physiology. Curr. Opin. Neurobiol. 17, 430–436 (2007). (PMID: 17689952)
Yamins, D. L. K. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016). (PMID: 26906502)
Cadieu, C. F. et al. Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS Comput. Biol. 10, e1003963 (2014). (PMID: 255212944270441)
Klindt, D., Ecker, A. S., Euler, T. & Bethge, M. Neural system identification for large populations separating “what” and “where”. Adv. Neural Inf. Process. Syst. 30, 3506–3516 (2017).
McIntosh, L. T., Maheswaranathan, N., Nayebi, A., Ganguli, S. & Baccus, S. A. Deep learning models of the retinal response to natural scenes. Adv. Neural Inf. Process. Syst. 29, 1369–1377 (2016). (PMID: 287297795515384)
Erhan, D. & Bengio, Y. & Courville, A. & Vincent, P. Visualizing higher-layer features of a deep network. Technical Report 1341 (University of Montreal, 2009).
Sofroniew, N. J., Flickinger, D., King, J. & Svoboda, K. A large field of view two-photon mesoscope with subcellular resolution for in vivo imaging. eLife 5, e14472 (2016). (PMID: 273001054951199)
Cadena, S. A. et al. Deep convolutional models improve predictions of macaque V1 responses to natural images.PLoS Comput. Biol. 15, e1006897 (2019). (PMID: 310132786499433)
Kindel, W. F., Christensen, E. D. & Zylberberg, J. Using deep learning to probe the neural code for images in primary visual cortex. J. Vis. 19, 29 (2019). (PMID: 310260166485988)
Zhang, Y., Lee, T. S., Li, M., Liu, F. & Tang, S. Convolutional neural network models of V1 responses to complex patterns. J. Comput. Neurosci. 46, 33–54 (2019). (PMID: 29869761)
Adelson, E. H. & Bergen, J. R. Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A 2, 284–299 (1985). (PMID: 3973762)
Hubel, D. H. & Wiesel, T. N. Receptive fields of single neurones in the cat’s striate cortex. J. Physiol. 148, 574–591 (1959). (PMID: 144036791363130)
Simoncelli, E. P. & Olshausen, B. A. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24, 1193–1216 (2001).
Lindsey, J., Ocko, S. A., Ganguli, S. & Deny, S. A unified theory of early visual representations from retina to cortex through anatomically constrained deep CNNs. Preprint at bioRxiv https://www.biorxiv.org/content/10.1101/511535v1.full (2019).
DiCarlo, J. J. & Cox, D. D. Untangling invariant object recognition. Trends Cogn. Sci. 11, 333–341 (2007). (PMID: 17631409)
Sabour, S., Frosst, N. & Hinton, G. E. Dynamic routing between capsules. In Proc. Advances in Neural Information Processing Systems 30 (eds Guyon, I. et al.) 3856–3866 (2017).
Lehky, S. R. & Sejnowski, T. J. & Desimone, R. Predicting responses of nonlinear neurons in monkey striate cortex to complex patterns.J. Neurosci. 12, 3568–3581 (1992). (PMID: 15275966575725)
Ecker, A. S. et al. A rotation-equivariant convolutional neural network model of primary visual cortex. International Conference on Learning Representations (ICLR) 2019 Conference Poster https://openreview.net/forum?id=H1fU8iAqKX (2018).
Pasupathy, A. & Connor, C. E. Population coding of shape in area V4. Nat. Neurosci. 5, 1332–1338 (2002). (PMID: 12426571)
Abbasi-Asl, R. et al. The DeepTune framework for modeling and characterizing neurons in visual cortex area V4. Preprint at bioRxiv https://www.biorxiv.org/content/biorxiv/early/2018/11/09/465534.full.pdf (2018).
Bashivan, P., Kar, K. & DiCarlo, J. J. Neural population control via deep image synthesis. Science 364, eaav9436 (2019). (PMID: 31048462)
Ponce, C. R. et al. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. cell 177, 999–1009.e10 (2019). (PMID: 31051108)
Reimer, J. et al. Pupil fluctuations track fast switching of cortical states during quiet wakefulness. Neuron 84, 355–362 (2014). (PMID: 253743594323337)
Froudarakis, E. et al. Population code in mouse v1 facilitates readout of natural scenes through increased sparseness. Nat. Neurosci. 17, 851–857 (2014). (PMID: 247475774106281)
Garrett, M. E., Nauhaus, I., Marshel, J. H. & Callaway, E. M. Topography and areal organization of mouse visual cortex. J. Neurosci. 34, 12587–12600 (2014). (PMID: 252092964160785)
Pnevmatikakis, E. A. et al. Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron 89, 285–299 (2016). (PMID: 267741604881387)
Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. in Proceedings of the 32nd International Conference on Machine Learning, Lille, France 37, 448–456 (2015).
Clevert, D.-A., Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). Preprint at arXiv https://arxiv.org/pdf/1511.07289.pdf (2015).
Jaderberg, M., Simonyan, K., Zisserman, A. & Kavukcuoglu, K. Spatial transformer networks. In Proc. Advances in Neural Information Processing Systems 28 (eds Cortes, C. et al.) 2017–2025 (Curran Associates, 2015).
McGinley, M. J. et al. Waking state: rapid variations modulate neural and behavioral responses. Neuron 87, 1143–1161 (2015). (PMID: 264026004718218)
Fu, Y. et al. A cortical circuit for gain control by behavioral state. Cell 156, 1139–1152 (2014). (PMID: 246307184041382)
Zoccolan, D., Graham, B. & Cox, D. A self-calibrating, camera-based eye tracker for the recording of rodent eye movements. Front. Neurosci. 4, 193 (2010). (PMID: 211522592998901)
Stahl, J. S., van Alphen, A. M. & De Zeeuw, C. I. A comparison of video and magnetic search coil recordings of mouse eye movements. J. Neurosci. Methods 99, 101–110 (2000). (PMID: 10936649)
van Alphen, B., Winkelman, B. H. & Frens, M. A. Three-dimensional optokinetic eye movements in the C57BL/6J mouse. Invest. Ophthalmol. Vis. Sci. 51, 623–630 (2010). (PMID: 19696183)
Prechelt, L. Early stopping — but when? in Neural Networks: Tricks of the Trade (eds Montavon, G., Orr, G., & Müller, K.-R.) 53–67 (Springer, 1998).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at arXiv https://arxiv.org/pdf/1412.6980.pdf (2017).
Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T. & Clune, J. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Adv. Neural Inf. Process. Syst. 29, 3387–3395 (2016).
Nguyen, A. M., Yosinski, J. & Clune, J. Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks. Preprint at arXiv https://arxiv.org/pdf/1602.03616.pdf (2016).
Wei, D., Zhou, B., Torralba, A. & Freeman, W. T. Understanding intra-class knowledge inside CNN. Preprint at arXiv https://arxiv.org/pdf/1507.02379.pdf (2015).
Olah, C., Mordvintsev, A. & Schubert, L. Feature visualization: how neural networks build up their understanding of images. Distill https://distill.pub/2017/feature-visualization (2017).
Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. International Conference on Learning Representations (ICLR) Workshop Paper https://arxiv.org/abs/1312.6034 (2014).
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K.-R. & Dähne, S. Learning how to explain neural networks: PatternNet and PatternAttribution. Preprint at arXiv https://arxiv.org/pdf/1705.05598.pdf (2017).
Yosinski, J., Clune, J., Nguyen, A., Fuchs, T. & Lipson, H. Understanding neural networks through deep visualization. Preprint at arXiv https://arxiv.org/pdf/1506.06579.pdf (2015).
Gatys, L. A., Ecker, A. S. & Bethge, M. A neural algorithm of artistic style. Preprint at arXiv https://arxiv.org/pdf/1508.06576.pdf (2015).
Mahendran, A. & Vedaldi, A. Understanding deep image representations by inverting them. Preprint at arXiv https://arxiv.org/pdf/1412.0035.pdf (2015).
Lenc, K. & Vedaldi, A. Understanding image representations by measuring their equivariance and equivalence. Preprint at arXiv https://arxiv.org/pdf/1411.5908.pdf (2015).
Tsai, C.-Y. & Cox, D. D. Characterizing visual representations within convolutional neural networks: toward a quantitative approach. In Proc. Workshop on Visualization for Deep Learning, 33rd International Conference on Machine Learning (2016).
Øygard, A. Visualizing GoogLeNet classes. Audun M. Øygard Blog https://www.auduno.com/2015/07/29/visualizing-googlenet-classes/ (2015).
Sreedhar, K. & Panlal, B. Enhancement of images using morphological transformations. Int. J. Comput. Sci. Inf. Technol. 4, 33–50 (2012).
Pologruto, T. A., Sabatini, B. L. & Svoboda, K. Scanimage: flexible software for operating laser scanning microscopes. Biomed. Eng. Online 2, 13 (2003). (PMID: 12801419161784)
Giovannucci, A. et al. Caiman: an open source tool for scalable calcium imaging data analysis. eLife 8, e38173 (2019). (PMID: 306526836342523)
Yatsenko, D., Walker, E. Y. & Tolias, A. S. Datajoint: a simpler relational data model. Preprint at arXiv https://arxiv.org/pdf/1807.11104.pdf (2018).
Paszke, A. et al. Automatic differentiation in PyTorch. In Proc. Advances in Neural Information Processing Systems (NIPS) 31 Workshop Autodiff Submission (2017).
van der Walt, S., Colbert, S. C. & Varoquaux, G. The NumPy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13, 22–30 (2011).
Jones, E. et al. SciPy: Open Source Scientific Tools for Python http://www.scipy.org (SciPy.org, accessed 3 October 2019).
Merkel, D. Docker: lightweight Linux containers for consistent development and deployment. Linux J. 239, 2 (2014).
Hunter, J. D. Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9, 90–95 (2007).
Waskom, M. et al. mwaskom/seaborn: v.0.8.1 (September 2017). Zenodo https://zenodo.org/record/883859#.XZXIjUZKguV (2017).
McKinney, W. Data structures for statistical computing in Python. In Proc. 9th Python in Science Conference (eds van der Walt, S. & Millman, J.) 51–56 (2010).
Kluyver, T. et al. Jupyter notebooks: a publishing format for reproducible computational workflows. In Proc. 20th International Conference on Electronic Publishing. Positioning and Power in Academic Publishing: Players, Agents and Agendas (eds Loizides, F. & Schmidt, B.) 87–90 (IOS Press, 2016).
Grant Information:
F30 EY025510 United States EY NEI NIH HHS
Entry Date(s):
Date Created: 20191106 Date Completed: 20200131 Latest Revision: 20230801
Update Code:
20260130
DOI:
10.1038/s41593-019-0517-x
PMID:
31686023
Database:
MEDLINE
*Further Information*
*Finding sensory stimuli that drive neurons optimally is central to understanding information processing in the brain. However, optimizing sensory input is difficult due to the predominantly nonlinear nature of sensory processing and high dimensionality of the input. We developed 'inception loops', a closed-loop experimental paradigm combining in vivo recordings from thousands of neurons with in silico nonlinear response modeling. Our end-to-end trained, deep-learning-based model predicted thousands of neuronal responses to arbitrary, new natural input with high accuracy and was used to synthesize optimal stimuli-most exciting inputs (MEIs). For mouse primary visual cortex (V1), MEIs exhibited complex spatial features that occurred frequently in natural scenes but deviated strikingly from the common notion that Gabor-like stimuli are optimal for V1. When presented back to the same neurons in vivo, MEIs drove responses significantly better than control stimuli. Inception loops represent a widely applicable technique for dissecting the neural mechanisms of sensation.*