WIXLIB

DialogueCDS Newsletter, Spring 2018Amador-Vargas, S., Gronenberg, W., Wcislo, W. T., &Mueller, U. (2015). Specialization and Group Size: Brain andbehavioural correlates of colony size in ants lacking morphological castes. Proceedings of the Royal Society B, 282(1801),20142502.Berlyne, D. E. (1978). Curiosity and learning. Motivation andEmotion, 2(2), 97-175.Blanchard, T. C., Hayden, B. Y., & Bromberg-Martin, E. S.(2015). Orbitofrontal cortex uses distinct codes for differentchoice attributes in decisions motivated by curiosity. Neuron,85(3), 602-614.Carruthers, P. (2002). The cognitive functions of language.Behavioral and Brain Sciences, 25(6), 657-674.Clarke, R. J., & Tobias, P. V. (1995). Sterkfontein Member 2foot bones of the oldest South African hominid. Science,269(5223), 521.Dember, W. N., & Earl, R. W. (1957). Analysis of exploratory,manipulatory, and curiosity behaviors. Psychological Review,64(2), 91.Gottlieb, J., Oudeyer, P. Y., Lopes, M., & Baranes, A. (2013).Information-seeking, Curiosity, and Attention: Computationaland Neural Mechanisms. Trends in Cognitive Sciences,17(11), 585-593.Gruber, M. J., Gelman, B. D., & Ranganath, C. (2014). States ofcuriosity modulate hippocampus-dependent learning via thedopaminergic circuit. Neuron, 84(2), 486-496.James, W. (1913). The Principles of Psychology, Volume II.New York, NY, United States: Henry Holt and Company.Jolly, A. (1972). The Evolution of Primate Behavior. New York,NY, United States: MacMillan.Kang, M. J., Hsu, M., Krajbich, I. M., Loewenstein, G., McClure,S. M., Wang, J. T. Y., & Camerer, C. F. (2009). The wick in thecandle of learning: Epistemic curiosity activates reward circuitry and enhances memory. Psychological Science, 20(8),963-973.Kidd, C., & Hayden, B. Y. (2015). The psychology and neuroscience of curiosity. Neuron, 88(3), 449-460.Kidd, C., Piantadosi, S.T., & Aslin, R.N. (2012.) The GoldilocksEffect: Human infants allocate attention to visual sequencesthat are neither too simple nor too complex. PLOS ONE, 7(5),e36399.Kinney, D. K., & Kagan, J. (1976). Infant attention to auditorydiscrepancy. Child Development, 155-164.Lieberman, P. (1991). Uniquely Human: The evolution ofspeech, thought, and selfless behavior. Cambridge, MA,United States: Harvard University Press.Loewenstein, G. (1994). The psychology of curiosity: A reviewand reinterpretation. Psychological Bulletin, 116(1), 75.Melis, A. P., & Semmann, D. (2010). How is human cooperationdifferent?. Philosophical Transactions of the Royal Society ofLondon B: Biological Sciences, 365(1553), 2663-2674.Oudeyer, P. Y., & Kaplan, F. (2007). What is intrinsic motivation? A typology of computational approaches. Frontiers inNeurorobotics, 1, 6.Pavlov, I.P. (1927). Conditioned Reflexes: An Investigationof the Physiological Activity of the Cerebral Cortex. Oxford,England: Oxford University Press.Piantadosi, S. T., & Kidd, C. (2016). Extraordinary intelligenceand the care of infants. Proceedings of the National Academyof Sciences, 113(25), 6874-6879.Stahl, A. E., & Feigenson, L. (2015). Observing the unexpected enhances infants’ learning and exploration. Science,348(6230), 91-94.Skinner, B. F. (1938). The Behavior of Organisms: AnExperimental Analysis. Oxford, England: Appleton–Century.Sokolov, E. (1963). Perception and the Conditioned Reflex.Oxford, England: Pergamon Press.Towards Computational Models of Curiosity in Cognitive DevelopmentElizabeth BonawitzComputational CognitiveDevelopment Lab,Rutgers University,Newark, USAthat cost of carrying out the information seekingaction is low, and so forth.Kidd raises an interesting chicken and egg problem pertaining to curiosity and the specializationof knowledge. One the one hand, epistemic curiosity likely depends on some prior knowledgebeing in place; it is piqued given the realizationof a gap between known and unknown information (Lowenstein, 1994). On the other hand,curiosity drives knowledge enrichment. Forexample, binding association and intervention isa key component of causal knowledge in earlydevelopment (Bonawitz et al., 2010). Curiositymay help drive this link between observed associative relationships and the outcomes of ourinterventions by motivating action. One way togain insight into these self-reinforcing roles ofcuriosity and developing knowledge is to examine them in early childhood.Curiosity is also often encoded simply as a driveor utility in a learning system, but modeling canalso help specify the causes of curiosity andquantify their contribution towards this drivein early development. For example, models canspecify the role of prior knowledge and variousutilities, providing a framework to build a utilitycalculus of curiosity. Recent research suggeststhat even very young children are already capable of carrying out this intuitive calculus, andthat curiosity and prior knowledge are deeplyintertwined. For example, young children aremore motivated to explore when events violatethe predictions of intuitive theories (Bonawitz etal., 2012; Stahl & Feigenson, 2015), suggestingthat even the young mind is driven to reduceuncertainty and learn more following a conflictof beliefs and evidence. Preschoolers are alsosensitive to information gain, exploring overexploiting rewards when the knowledge gainedwill serve later use (Bonawitz, Bass, & Lapidow,2018). Curiosity may be piqued when it isbrought to the attention of a learner that knowledge is incomplete. For example, research withpreschoolers has found that pedagogical questions simultaneously point to the importanceof particular features, while also encouragingfurther exploration and discovery (Yu, Landrum,Bonawitz, & Shafto, 2018).Computational modeling further helps to makeprecise the role of prior knowledge and opportunities for information gain. By bridging theseapproaches we can come to better understandthe contributions of curiosity to our uniquelyhuman traits. There is a long tradition of modeling curiosity (e.g. see Oudeyer 2018 for a review)and also of modeling the role of prior knowledgein human learning (e.g. see Tenenbaum, Griffiths,& Kemp, 2006). Taking the theory-based probabilistic perspective as a starting point, wemight conceptualize of curiosity as an artifactthat falls out from “running” certain inferentialprocesses. Indeed, we can think of the mind ascarrying out simulations (Battaglia, Hamrick, &Tenenbaum, 2013), search (Ullman, Goodman, &Tenenbaum, 2012), and sampling (Bonawitz etal., 2014; Griffiths, Vul, & Sanborn, 2012) overintuitive theories. Curiosity may exist as a stateduring this inferential process and be greatestwhen information is likely to be gained, wheninformation will likely resolve conflict or uncertainty, when the reward of knowledge is high andComputational approaches provide an importantstarting point for understanding the intertwinedroles of curiosity and knowledge acquisition,but current frameworks do not yet have ameaningful way to incorporate affect. Discoverycan feel good and thus rewards curiosity (e.g.“Explanation as Orgasm, Gopnik, 2000), butaffect also cyclically drives exploration (e.g. as5

DialogueCDS Newsletter, Spring 2018in dopaminergic neuroscientific models of prediction error, learning, and further motivation,Pessiglione et al., 2006). Awe has been definedas a simultaneous affective and cognitive experience related to curiosity (Valdesolo & Graham,2014). Being “induced” to feel awe stimulatescurious exploration in preschoolers (Colantonio& Bonawitz, 2018). Despite the clear role ofemotion in experiencing reward and motivating further learning, we currently do not havea way to capture manipulations of affect in ourcomputational models of curiosity. That is, justas emotions can give us a sense of success orfailure critical to learning, we can also hardcode rewards and costs in our models, but thisdoes not necessarily inform our understandingof curiosity as an affective drive. Physiologicaland emotional components may be the defining features of curiosity, yet they are currentlyill-defined.Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013).Simulation as an engine of physical scene understanding.Proceedings of the National Academy of Sciences, 110(45),18327-18332.Bonawitz, E., Bass, L., & Lapidow, E. (2018) Choosing to Learn:Evidence Evaluation for Active Learning and Teaching in EarlyChildhood. In: Saylor M., Ganea P. (eds) Active Learning fromInfancy to Childhood. Springer, Cham, 213-231.Bonawitz, E. B., van Schijndel, T. J., Friel, D., & Schulz, L.(2012). Children balance theories and evidence in exploration, explanation, and learning. Cognitive psychology, 64(4),215-234.Bonawitz, E., Denison, S., Griffiths, T. L., & Gopnik, A. (2014).Probabilistic models, learning algorithms, and responsevariability: Sampling in cognitive development. Trends inCognitive Sciences, 18(10), 497-500.Bonawitz, E. B., Ferranti, D., Saxe, R., Gopnik, A., Meltzoff,A. N., Woodward, J., & Schulz, L. E. (2010). Just do it?Investigating the gap between prediction and action in toddlers’ causal inferences. Cognition, 115(1), 104-117.Colantonio, J., & Bonawitz, E. (in press) Awesome play: Aweincreases preschoolers exploration and discovery. In Kalish,C., Rau, M., Zhu, J., & Rogers, T.T. (Eds.) Proceedings of the40th Annual Conference of the Cognitive Science Society.Madison, WI: Cognitive Science Society.Gopnik, A. (2000). Explanation as orgasm and the drive forcausal knowledge: The function, evolution, and phenomenology of the theory formation system. In F. Keil & R. Wilson(Eds.), Cognition and explanation (299-323). Cambridge, MA:MIT Press.Griffiths, T. L., Vul, E., & Sanborn, A. N. (2012). Bridging levelsof analysis for probabilistic models of cognition. CurrentDirections in Psychological Science, 21(4), 263-268.Loewenstein, G. (1994). The psychology of curiosity: A reviewand reinterpretation. Psychological Bulletin, 116(1), 75.Oudeyer, P. Y. (2018). Computational Theories of CuriosityDriven Learning. arXiv preprint arXiv:1802.10546.Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith,C. D. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature, 442(7106),1042.Stahl, A. E., & Feigenson, L. (2015). Observing the unexpected enhances infants’ learning and exploration. Science,348(6230), 91-94Tenenbaum, J. B., Griffiths, T. L., & Kemp, C. (2006). Theorybased Bayesian models of inductive learning and reasoning.Trends in cognitive sciences, 10(7), 309-318.Ullman, T. D., Goodman, N. D., & Tenenbaum, J. B. (2012).Theory learning as stochastic search in the language ofthought. Cognitive Development, 27(4), 455-480.Valdesolo, P., & Graham, J. (2014). Awe, uncertainty, andagency detection. Psychological science, 25(1), 170-178.Yu, Y., Landrum, A., Bonawitz, E., & Shafto, P. (in press)Questioning supports effective transmission of knowledgeand increased exploratory learning in pre-kindergarten children. Developmental Science.Computational models of learning in early childhood may help us to formalize and understandthe intertwined roles of curiosity and specialization of knowledge. However, this approach alsomakes apparent that our models must consideremotional states. Children are both intenselyemotional and curious creatures, providing yetanother benefit of studying curiosity in earlychildhood. Perhaps by recognizing this gap inour current understanding, we will be morelikely to curiously seek out new models andempirical studies, helping further refine ourknowledge.Is Human Curiosity Neurobiologically Unique?Tobias U. HauserMax Planck UCL Centrefor ComputationalPsychiatry and AgeingResearch,University College LondonUnited KingdomIs there something uniquely human about curiosity, and does this allow us to focus all our energyto become experts in one particular field? In herDialogue Initiation, Celeste Kidd suggested thatcuriosity is a key driver for human specialisationthat makes us humans unique. In what follows,I will approach this hypothesis from a computational neurobiological standpoint and raise thequestion whether there is any neurobiologicaluniqueness to human curiosity.as reward (i.e. selecting the option that you thinkprovides you with the most reward), explorationcharacterises actions that are suboptimal in thelight of the objective outcome, but may conveyadditional information that can be useful in thefuture (i.e. choosing an object where you knowvery little about how good it is). Exploration isthereby critical for both animals and machines,especially in situations with much uncertainty,such as environments with changing incentives.What is curiosity? Although we all have animplicit concept of what curiosity constitutes orwho a curious person is, a formal definition is nottrivial. Rather than investigating curiosity per se,it is mostly broken down to behaviours that weinterpret as being curious. In particular, sincethe early days of curiosity research, scientistsstudied what is termed ‘exploratory’ decisions(Berlyne, 1960). Exploratory decisions can beseen as the counterpart to exploitative decisions,forming a well-known exploration-exploitationtrade-off in artificial intelligence (Sutton andBarto, 1998). While exploitation describes decisions that optimise an objective outcome suchSo how can we solve this exploration-exploitation trade-off? In the remainder of this response,I will focus on situations with explicit decisionsand where objective outcomes are directly quantifiable (e.g. in the form of reward), rather than‘spontaneous’ exploration in environments withunspecified outcome structures, because muchless is known about the neural mechanisms ofthe latter. A detailed discussion of the computational bases of the latter situations can be foundelsewhere (e.g., Loewenstein, 1994; Gottlieb etal., 2013; Friston et al., 2015; Oudeyer, 2018).Solutions to dilemmas with specified objective6

CDS Newsletter, Spring 2018last to mature in humans (Ziegler et al., underreview), this might also explain why directedexploration only really surfaces during lateadolescence (Somerville et al., 2017). A critical, but unanswered question is whether and towhat extent humans and non-human animalsdiffer in how they employ this directed exploration strategy. It is possible that only through adominance of directed exploration humans canuse an efficient way to extend their knowledge.This seems particularly relevant in environments where much is already know, but wherea directed exploration allows to expand knowledge in a targeted way by aiming for the subtlegaps, which in turn will propel specialisation andexpertise.outcomes can be broadly categorised in threedifferent groups. The first and computationally simplest solution is to choose a randomaction completely ignoring its expected value.This can be interpreted as entirely ignoring allknowledge about the world every so often andis implemented in algorithms such as ε-greedy(Sutton and Barto, 1998). A second, somewhatmore sophisticated solution is to again introducenoise into a system, but let this be guided byour knowledge. This is implemented in softmaxdecision functions (Luce, 1959), or more sophisticatedly in Thompson sampling (Thompson,1933). Such algorithms let an agent chooseactions in proportion to their expected value,which means that we have the ability to samplefrom all possible choice options, but very unattractive ones will hardly ever be chosen. Lastly,directed exploration takes the uncertainty abouteach action into account and chooses the optionwhere the agent can learn most, i.e. where ithas the biggest information gain (Loewenstein,1994; Wilson et al., 2014). Prominent examplesfor directed exploration are upper confidencebound algorithms (UCB; Kaelbling, 1994; Auer,2002).In sum, it is interesting to conjecture whetherthere is a neurobiological basis for unique patterns of exploration in humans and whether astronger reliance of a directed exploration strategy allows human to become highly specialisedin one discipline. Such a dominance in itselfcould also be a result of a developmental process, where intrinsicially guided learning may beassigned a high reward in itself (epistemic value;Friston et al., 2015), and where a metacognitiveexploration module optimises the explorationstrategies throughout development (cf. Gottliebet al., 2013; Oudeyer, 2018).Although these algorithms have mainly beenused in artificial intelligence, evidence indicates that humans and non-human animals usesimilar strategies. Importantly, humans seemto apply a mixture of strategies (Wilson et al.,2014; Gershman, 2018), but their relative contribution changes over development (Somervilleet al., 2017). It is thus likely that different brainsystems are in control of different exploration strategies, and maybe a dominance of adirected exploration system resembling an UCBalgorithm is what distinguishes human fromnon-human curiosity.Directed exploration is most likely governed by anetwork including frontopolar cortex. Transientinhibition of this brain area in humans indeedimpairs directed exploration, but leaves randomforms of exploration intact (Zajkowski et al.,2017). Given that this brain area is one of theLastly, it is important to address the questionof why within our species we differ so stronglyin curiosity and specialisation. Despite having areasonable junk of humankind reaching somelevel of specialisation, probably a majority ofour species shows only humble specialisation,but forms the economic backbone for those whoventure into extreme specialisation and whoare thus unable to divert much of their energyto ensure basic needs, such as gathering food.Taking a societal rather than an individual evolutionary perspective could provide answers tothis question, because a distribution of exploration traits in a society may indeed provide anevolutionary advantage (Williams and Taylor,2006).Auer P (2002) Using Confidence Bounds for ExploitationExploration Trade-offs. J Mach Learn Res 3:397–422.Berlyne DE (1960) Conflict, arousal, and curiosity. New York,NY, US: McGraw-Hill Book Company.Friston K, Rigoli F, Ognibene D, Mathys C, Fitzgerald T,Pezzulo G (2015) Active inference and epistemic value. CognNeurosci:1–28.Gershman SJ (2018) Deconstructing the human algorithmsfor exploration. Cognition 173:34–42.Gottlieb J, Oudeyer P-Y, Lopes M, Baranes A (2013)Information-seeking, curiosity, and attention: computationaland neural mechanisms. Trends Cogn Sci 17:585–593.Kaelbling LP (1994) Associative Re

the conference. Later this year I'm looking forward to the completion of the Frontiers special issue on Intrinsically Motivated Open-Ended Learning in Autonomous Robots, an ongoing project that has followed on from the International Workshop on Intrinsically Motivated Open Ended Learning in Rome late last year. This will also bring