Virtual reality in the piano studio
If you are a Star Trek fan or at least have a passing familiarity with the television series, you may be aware that people still play music and attend concerts in the twenty-fourth century. Given the fact that computers do so much for human beings now and will do even more in the future, it is certainly comforting to know that our descendants will not abandon the personal joy of music making!
In the original Star Trek series (TOS), Chief Engineer Montgomery Scott plays the bagpipes and Lieutenant Commander Spock plays the Vulcan lute. In the subsequent series, Star Trek: The Next Generation (TNG), Captain Jean-Luc Picard is an expert player of the Ressikan flute, Commander William T. Riker is an excellent trombonist, and the nefarious Q is a trumpeter. Interestingly, the android Data, who is forever in search of his humanity, plays both the oboe and the violin.
And it doesn't stop there. Jadzia Dax, a Trill who serves as Chief Science Officer on Deep Space Nine (DS9), has been known to play Samoan drums, and Deep Space Nine botanist Keiko O'Brien plays the clarinet. Star Trek: Voyager's (VOY) Ensign Harry Kim also plays the clarinet, as well as the saxophone.
Of crucial importance to readers of this magazine, the piano is still a popular instrument in the twenty-fourth century. Notable pianists include Captain Benjamin Sisko (DS9), Counselor Deanna Troi (TNG), Changeling Constable Odo (DS9), and astrometrics expert Seven of Nine (VOY). If you wish to explore this fascinating future of ours in greater detail, I recommend that you start with Musical Instrument—Memory Alpha, the Star Trek Wiki: (http://en.memory-alpha.org/wiki/ Musical_instrument).
The virtual reality experience
One of the most intriguing features of the later Star Trek series is the holodeck. The holodeck is a room that can be programmed to look like another place. Computer-controlled by photons and force fields, this room can become any indoor or outdoor place of your imagination.
Interestingly, the holodeck can be programmatically filled with virtual people who walk, talk, and even play musical instruments. Imagine, for example, that you enter the holodeck prepared to practice a piano concerto. With the appropriate programming, the space around you would become a small concert hall. There would be an exquisite piano for you to play, and you would be surrounded by virtual musicians who are waiting for a conductor's downbeat.
It's regrettable that so few pianists ever get a chance to play with an orchestra. Some of the finest works ever written for piano require the participation of anywhere from ten to seventy additional musicians. If we lived in the twenty-fourth century where we have access to a holodeck, we wouldn't have this problem!
The virtual music experience in the twenty-first century
If you look around, you'll notice that a lot of people are working to solve the various technical problems involved in creating a compelling, virtual reality environment. As interesting as these developments may be, some of these efforts may appear to musicians as going in the wrong direction.
For example, there are a variety of sophisticated game machines that you can connect to your television, such as Xbox, PlayStation, and Wii. These game machines feature amazing graphics, thundering sound, and loads of interactivity. In other words, they provide immersive experiences. Among the popular games are various music-making scenarios in which you can participate as a virtual musician.
The biggest complaint about these music games is that they don't provide the player with anything that approaches the experience of actual music making. Although the more sophisticated games may offer consumers the opportunity to connect a fake instrument to the game box, the player rarely plays music in a traditional sense. Typically, the player watches for a visual cue on the television and responds with a simulated performance gesture on the fake instrument. If the player responds in time, he may enjoy the thrill of scoring a point and may even sense that he contributed a note to the ensemble.
Such gaming experiences, however, fall far short of real music making. Real music making involves a direct, physical, and visceral connection between the player and the instrument as well as realistic aural feedback.
Student performances with a virtual orchestra
Although there is a lot of emphasis these days on the development of games that provide virtual experiences that require no previous, serious training (such as flying a plane, playing a professional sport, or operating on a patient), there are now compelling technologies available that serve the serious piano player who has actually spent years developing his chops.
For example, on September 16, 2011, eight youngsters ranging in age from seven to sixteen performed piano/keyboard concertos by Bach, Haydn, Mozart, and MacDowell on the Libby Gardner Concert Hall stage at the University of Utah. These students were participants in the SummerArts program and had won this performance opportunity in a concerto competition the previous month. Dr. Susan Duehlmeier, piano chair at the university, organized the opportunity which was fully supported by the local piano dealer, Daynes Music. Dr. Duehlmeier wanted these students to "get a chance to try out what it would be like to play with an orchestra, before they ever rehearse with a real one."
The program included individual movements from:
Concerto in C Major, Hob. XIV:4, by Franz Joseph Haydn
Concerto in F Minor, BWV 1056, by J.S. Bach
Concerto No. 11 in F Major, K. 413, by W.A. Mozart
Concerto No. 2 in D Minor, Op. 23, by Edward McDowell
Concerto in A Major, BWV 1055, by J.S. Bach
Concerto No. 23 in A Major, K. 488, by W.A. Mozart
Concerto in D Minor, BWV 1052, by J.S. Bach
As expected, the students played on a nine-foot concert grand (a Steinway in this case). Conspicuously absent was a conductor and orchestral musicians. In their place was a computer and speakers.
Had you been in the audience, you might have been fooled into thinking that these kids were playing to a Music-Minus-One CD recording. MMO (www.musicminusone.com) is a venerable company that has produced innumerable play-along recordings of piano concerti and other orchestra works, minus the soloist. MMO products are quality recordings that enable pianists and other soloists to prepare for performances with orchestra.
There was a big difference in this case, however. As each of these youngsters played, a virtual orchestra actually followed the soloist, speeding up and slowing down as necessary. Amazingly, the technologies that made this possible are accessible to most piano teachers and musical schools and even to many students at home.
In search of the intelligent virtual orchestra
Enter your text here ...
In the old days of MMO recordings— and here I am talking about vinyl records—you had a very serious limitation: you could not change the tempo to any substantial degree. If you had a sophisticated turntable that featured pitch control, you could make the record spin a little faster or slower. However, doing so changed the pitch of the recording as well as the tempo and was only useful for tuning the record to your piano.
MMO later released its library on CDs. At that point, the tempo/pitch problem became worse because you could neither change the tempo nor the pitch with a standard CD player. Eventually, Superscope (www.superscopetechnologies.com) and other companies came out with CD players that could process the digital audio on the CD in real time, enabling customers to change tempo and pitch independently. Today, you can do the same thing with a $49.95 computer program called Amazing Slowdowner (www.ronimusic.com).
Changing the tempo of a recording for play-along practice is a useful thing to be able to do. Performing the solo part of a concerto and having the orchestra actually follow you is a substantially different matter.
In the early 1990s, pianist and selftaught computer programmer, Frank Weinstock (recently retired as acting dean at the University of Cincinnati College- Conservatory of Music), undertook the challenge of coming up with a software program that would "listen" to a soloist, match the soloist's playing to a music score, and output a coordinated orchestral accompaniment. He called his prototype software, Concerto Accompanist.
I discovered Frank in the early 1990s while researching an article that I wrote for the Summer 1992 issue of Piano Quarterly called Using a Computer Piano. This article appeared in my column, Tackling Technology. With a Yamaha Disklavier piano connected by MIDI cables and interface to a Macintosh computer, Frank's program was able to track a piano performance in real time. As each note was played, a message was transmitted to the computer informing it of the exact pitch, and Concerto Accompanist was able to match the note to an existing score (in the form of a MIDI file). Even more remarkably, Concerto Accompanist was able to evaluate the importance of each note, accommodate mistakes, make musically intelligent decisions about the tempo as it fluctuated in real time, and output a musically coordinated accompaniment.
In the mid-1990s, I had the opportunity to use this experimental software program Grieg's Concerto in A Minor for Piano and Orchestra, Op. 16, for performance with a real orchestra. At the time, the General MIDI tone generator that rendered the orchestral instruments sounded quite artificial, but I found that practicing with this early stage, virtual orchestra was much more helpful than practicing with a second- piano accompaniment or an inflexible MMO recording.
That was the state-of-the-art in the mid-1990s, when we so eagerly looked to the future.
The virtual orchestra matures
Frank continued to develop his software over the years, and today it is available under the name Home Concert Xtreme, produced by Zenph Sound Innovations (www.zenph.com). In its current incarnation, the program not only follows the soloist, it also displays music on the screen and turns the pages automatically (the Clavier Companion review is in the September/October 2010 issue). However, the program, by itself, does not provide a complete, virtual reality experience.
In order to have a compelling, concerto experience with a virtual orchestra you need:
- a quality piano that can be connected to a personal computer or iPad
- score-following software (such as Home Concert Xtreme)
- a MIDI file that contains the left- and right-hand piano tracks as well as separate tracks for the orchestral parts
- a realistic tone generator for the orchestra
- quality speakers
There are a number of sources of quality acoustic pianos that can be plugged into a computer or iPad. Yamaha's Disklavier (www.disklavier.com) is one of the best known choices. In the case of the concert at the University of Utah (described above), a Steinway was fitted with a PNOscan system (www.qrsmusic.com). Without affecting the touch or tone of the instrument, PNOscan provides optical sensors under the keys and in the pedal mechanism and can be added to any acoustic piano. Both the Disklavier and the PNOscan system output MIDI data and can be plugged into a computer. Of course, a quality digital piano can suffice.
Achieving a realistic orchestra sound is a difficult matter. The usual General MIDI (GM) tone generator that is available in many MIDI keyboards and computers is relatively low quality. For example, GM has no way of differentiating among a violin upbow, downbow, or spiccato bow.
There are, however, very expensive orchestral sound libraries that provide a broad range of samples for all of the standard instruments. The best of these cost thousands of dollars and take up many gigabytes of computer storage. Although their sounds are compelling, these high level orchestral libraries are impractical for most people—at the present time.
In 2004, harpist Gary Garritan released an amazing string library at an affordable price. Not long after that, he completed an orchestra set with the addition of woodwinds, brass, and percussion and named the package Garritan Personal Orchestra (www.garritan.com). The sounds in this library were produced by recording studio musicians. This library of orchestral sounds has a retail price of $149, which puts the library into an affordable price range.
MIDI files for the University of Utah concert were created by expert musicians who carefully edited the files, optimizing them for Garritan Personal Orchestra. These files and more will become available for sale on the Zenph website.
How was the experience? Ask the kids. Seven-year-old Elizabeth stated, "I think it's fun!" Thirteen-year-old Caity remarked, "Wow, this is awesome! I was really excited because I've never played with something like this before."