World premiere of brain orchestra

By Jason Palmer
Science and technology reporter, BBC News, Prague

Advertisement

Final rehearsal before the Multimodal Brain Orchestra performs its premiere

The Multimodal Brain Orchestra performed its world premiere on Thursday.

Led by an "emotional conductor" and a traditional one, music and video change in time with the performers' brain waves and heart rate.

According to the work's producer, the orchestra aims to "see what the brain can do without the body".

The orchestra's premiere performance closed the Science Beyond Fiction conference in Prague.

The project is the creation of the Synthetic, Perceptive, Emotive and Cognitive Systems (SPECS) group at the University Pompeu Fabra in Barcelona.

"Only recently we have come to appreciate more the tight coupling between mind, brain and body," Paul Verschure, head of the project, told the audience.

"But we can wonder what the mind and brain would be capable of if it would be directly interfaced to the world, bypassing the body."

The piece that the orchestra performed, Xmotion, was composed by Jonatas Manzolli, giving it an underlying structure.

But it is the performers who control variations of visuals, sounds, frequencies, and volumes in the overall piece.

Model result

What we want to show here is the use of your brain without your body Anna Mura SPECS group

"Everything is built to fulfil the circumplex model, which was worked out by psychologists that study emotions," says Anna Mura, a biologist who is the producer of the project. "How we feel and what we feel fits into a circle cut into four quadrants, labelled by 'arousal' and 'valence', that is, how much you're excited and how badly or positively you're excited."

To that end, four performers were fitted with caps made by G-tec, littered with electrodes that take a real-time electroencephalograph - an image of the brain's electrical activity.

"There is a first violin, a second violin and so on, except that instead of violins they are brains," says Dr Mura.

The graphs of those brain waves are projected onto one of two large screens above the orchestra. The performers launch sounds or affect their frequencies and modulations based on two well-characterised effects seen in EEGs: the steady-state visually evoked potential (SSVEP), and the so-called P300 signal.

Two of the performers were given a task to watch a screen in front of them, with flashing rows and columns of letters, and told to look for a particular letter.

The brain monitors capture mental activity in response to stimuli

When expectation is fulfilled, 300 thousandths of a second later, a signal known as the P300 appears in the EEG.

A similar strategy has been employed by Mick Grierson at Goldsmiths, University of London to generate individual notes.

In the Multimodal Brain Orchestra, the P300 signal is registered - with a dot demarcating it on the EEG trace projected to the audience, so that they can see the effect of the performer's thought - in turn launching a sound or recorded instrument.

Two more performers were given boxes with four lights flashing at different frequencies. The SSVEP is a brain signal that comes about when visual stimulus in the retina at a given frequency causes the brain to synchronize, so that frequency appears in the EEG.

Given a cue from the conductor, the performers switch their attention from one flashing frequency to another.

One of them affects the volume of a given sound - known to influence the level of arousal in the circumplex model - and the other affects a certain modulation of that sound, which is known to influence the valence, how positive or negative the arousal is emotionally.

'Emotional experience'

Adjacent to the EEG-capped players, the "emotional conductor" sits comfortably, wearing a pair of virtual reality glasses.

She is being shown images from a series created by artist Behdad Rezazadeh while her heart rate and skin conductance are being measured. Her heart rate is plotted along with the EEG traces.

As her mood changes, so does the visual experience - Rezazadeh's images are blurred and changed in line with the changing biological measures of the conductor.

"This performance is like her emotional experience," said Dr Mura. "What we want to show here is the use of your brain without your body. Embodiment - we should get rid of it sometimes."

While this artistic interpretation of the biometric technology is a fascinating look into the merger of technology with medicine, the overall goals of the group have far greater implications.

"People believe that to understand how we feel will help us to understand what consciousness is all about. This is the technology that is going in that direction; we cannot explain consciousness with this but we are at least exploring the surface of it."