The 2014 Clavius Lecture: Painting Sound and Singing Colors

Shinsuke Shimojo, Ph.D.

It’s not frequent that artists are invited to join scientists, engineers, and social scientists for a lecture on cognitive neuroscience. But this year’s Clavius Distinguished Lecture held on March 27 featured Shinsuke Shimojo, Ph.D., whose exploration of vision, perception, and decision-making piqued everyone's curiosity.

“This talk [is] of interest to a wide range of people, not just scientists,” said Shimojo, the California Institute of Technology Gertrude Baltimore Professor of Experimental Psychology.

Titled “Sensory Substitution, Multisensory Plasticity, and the Third Kind of ‘Qualia,’” Shimojo focused on qualia, which, to some, refers to the absolute, unique quality of a conscious sensory experience--i.e., when two people look at a saturated color are they seeing the same thing? Or do they each see it in their own unique manner? Since qualia is not something easily defined through words alone, Shimojo incorporated plenty of sound and visual examples.

Shimojo said qualia presents a conundrum for scientists, since its not a “hard” problem. Nevertheless, current sensory sciences fail to critically characterize the unique quality of sensory experiences.

As an example, Shimojo has examined a sensory substation device used by the blind called a “vOICe.” Operating like a scanner or radar, vOICe uses a portable computer and digital glasses to translate video data into a sound stream. Contrasts are translated into pitch and volume, allowing dimension and distance to come into focus.

“We are trying to give the blind some digital ability, but in order to do that you need to rely on the brains ability to learn how to interpret the sound,” said Shimojo.


Shimojo brought a short video of colorblind artist Neil Harbisson, who collaborated with an engineer to create a device that translates color into sound. The artist claims that he can now "see" color.

“Inspired by this, we studied aesthetic experience though devices and now we have neuroaesthetic data about people’s judgment,” said Shimojo.

Shimojo explained that if calculations can be derived to translate the visual into sound, then an inverse calculation could be made from sound to the visual.

Below, Harbisson explains how he is taking a similar approach by creating a clothing line where the colors are dictated by songs.


“You can go to a funeral wearing a "Requiem" from Mozart. It will look very colorful, but it’ll sound very sad,” he said.



And while much of Harbisson's approach to the science sounds amusing, it holds very serious implications. Some of the blind subjects using the technology have shown activity in the visual cortical areas in fMRI scans. Shimojo said the ultimate strategy is to come up with a brief list of psychophysical and neuroscientific criteria for “vision-like” processing and to search for empirical evidence.

“You don’t have to be blind to appreciate the technology,” said Shimojo. “Through calculation and translation, from well-known painting to classical music, by using these devices we can add another dimension of our sensory experience.”

The lecture took place on Thursday, March 27, in the Flom Auditorium in the Walsh Library at Fordham's Rose Hill Campus.

For more Information, contact Palma Hutter at hutter@fordham.edu or 718-817-4480. 

-Tom Stoelker