[Back to Inside]

By Joe Kleinsasser

Jim Steck, associate professor, mechanical engineering, is among four researchers developing a musical "instrument" for disabled individuals. He is wearing the headband that plays the piano keyboard on the computer screen through brain waves or muscle movement.

One would think that without the use of our hands, there’s little opportunity to play music. Since December, four WSU faculty members have proven that thinking wrong by conducting research to find a way for severely disabled individuals to play a musical instrument by brain waves or thought.

The "instrument" in this unique interdisciplinary project includes a headband that also responds to muscle movements, a picture of a piano keyboard on a computer monitor and a computer sound card.

The research team members are Elaine Bernstorf, associate professor, music; Steve Skinner, associate professor, electrical and computer engineering; Jim Steck, associate professor, mechanical engineering; and Janet Twomey, assistant professor, industrial and manufacturing engineering.

Several graduate students with interests in music, disabilities research, speech and communicative disorders, electrical and computer engineering, and computer science are providing additional support.

Steck was encouraged to conduct this kind of research by using existing research.

"I read some articles about doing brain computer interfacing for disabled people," Steck said. "In fact, there were some devices on the market where you could play a video game by putting a device on your finger, and it would pick up some kind of physiological activity instead of using a mouse on a computer.

"So I thought there was an application for using the artificial intelligence technology and neural networks for processing the information from the brain to control a computer and to build a musical instrument that would be controlled by brain waves or thought."

A neural network is designed to work a lot like the brain. Its artificial neurons are patterned after those in the brain, and it can learn from examples.

The team collaborated with an Ohio company that had designed a headband which controls a cursor on a computer screen and adapted it to their project.

"We found that from (the) headband we can use electrical signals from the brain (EEG) and muscles (EMG) to control a musical instrument," said Steck.

"The idea is if you look to the right, it will jump up a note (on the keyboard). If you look to the left, the headband detects the signal and moves down a note. If you do what’s called a forehead click and raise your eyebrows, it plays the note."

However, there’s more to music than playing a note.

"Speed and reliability need to improve," said Steck. "To go from a C to an E or a G takes about five seconds. That’s too long to play any kind of song. We’re using a neural network to do the pattern recognition to recognize a left pattern verses a right pattern versus a forehead click. Mainly what we’re looking at is muscle activity."

If a neural network is successful, each person needs his or her own profile.

"That’s one advantage of a neural network," said Steck. "We can train it on each individual. It’s adaptable and customized."

The $2,000 headband is useable with most computers with a sound card. The sound card is similar to an electronic keyboard in that the user can choose from a number of instrument sounds such as a piano, organ or violin.

The interdisciplinary research project presented some interesting challenges.

"This is the first time I’ve been involved in research where I didn’t have a good grasp on all aspects of the discipline," said Bernstorf. "I’ve learned a great deal, but I still don’t have the engineering background to understand a lot of the programming details.

"It has been gratifying to see the team come together and share expertise and to work with some of the graduate students in engineering and watch them become more interested in music. They have really taken ownership in trying to perfect the instrument while they also try to improve their own performances.

"Dr. Steck has teased them about getting ready for a recital so we can see what they have learned," Bernstorf said.

Actually, part of the testing does require a musical performance.

"We are planning to video record a recital (in mid-October) with our students playing some simple pieces of music" to see how feasible it is to get a musical flow, said Steck.

After that, Bernstorf said, they can look for people with disabilities to see how the programming can adapt to individuals.

While the computer interfacing will be applied to developing a musical instrument, the results of the study could impact all applications of direct brain-computer interfacing for disabled individuals.

The research has potential implications for the Air Force as well, according to Steck. The same technology could benefit aircraft maintenance workers who need to refer to a manual while working on airplanes.

The WSU Office of Research Administration is funding the project.