NYU 9th Floor Talks: The Cutting Edge of Game Research

Ken Perlin, a Computer Science professor at NYU, presented some of his cutting edge research in procedural animation and education games.

Presentation

• Ken is the founding director of NYU's Media Research Lab and the Director of the Games for Learning Institute. He is a mathematician who has won an Academy Award for his noise and procedural texturing techniques that have been used in film and television.
• Ken started his career in computer graphics. Being in the industry for so long, he has experienced the power of Moore's law, seeing computers get twice as faster every two years.
• He has developed several hundred Java applets. Some of his work can be found on his NYU homepage (http://mrl.nyu.edu/~perlin/).
• When researching something, you have to boil it down to the simple possible version.
• One of Ken's biggest research interests has been conveying emotions through artificial characters.
• In Polly's World (http://mrl.nyu.edu/~perlin/experiments/polly/track.html), Ken tries to convey emotions with the least possible number of vertices and polygons. The procedurally animated character is made up of only 6 vertices, but displays a wide range of emotions through its movements. Our brain maps simple animations to emotions.
• Why do we care about animated characters? Ken researches the human body as an instrument to understand psychological emotions.
• In Responsive Face (http://mrl.nyu.edu/~perlin/experiments/facedemo/), Ken researches what's the simplest emotive face possible. At the very least, an emotive face requires movements of the eyebrows, lid, gaze, head, and mouth. A person has n-points in the face like a keyboard has a number of notes and facial expressions are like chords.
• Micro noise movements and procedural variations makes the emotions more realistic. Without these micro movements, the face is just a lifeless still picture.
• Autistic children have trouble looking at other human's faces and thus grow up not knowing how to read emotions. However, they don't have that problem with artificial, computer-generated faces. Many autistic children used Responsive Face to learn human emotions.
• Plan 9 From Outer Space, often regarded as the worst movie ever made, is almost unwatchable because the acting is so bad. The actors' faces are so emotionless that it's impossible to identify with the characters.
• In an attempt to create an "interactive Pride and Prejudice," Ken developed an applet with five birds with disembodied feet. The birds showcase procedural walking animation, which is tons better than canned animations.
• We should think of avatars the same way a director thinks about actors in a live play. We shouldn't communicate actions to the avatar (ie. "jump here", "punch that", "open this door"), but communicate emotions (ie. "you love her"). Likewise, actors shouldn't indicate (ie. "I am feeling happy!", "I feel very sad."), but actually emote.
• With Fish Tales (http://cims.nyu.edu/~perlin/fishtales2/), players can record a one-minute story using Bob the fish and two bouncing balls.
• In Candy Circle (http://mrl.nyu.edu/~perlin/candycircle/), players make procedural music using a spinning wheel filled with candy. It researches the rule of the fifths; when the further away the chords are, the more dissonant the sounds are.
• Ken also experimented with spiral escalators (http://mrl.nyu.edu/~perlin/escalator/). Spiral staircases always go right going upwards because they were designed solely for battle. Since most knights were right-handed, it would give the advantage to the person who's above.
• The barrier of entry to making games is so high. Ken would like to have more people coming into the arts that are enabled by programming.
• There are "fake languages" such as Code Do It and Scratch that enable people to jump into programming. Ken is researching how to make programming more integral to education.
• Ken's New Line Fractals (http://mrl.nyu.edu/~perlin/newlinefractal/) show mathematical visual examples of fractals. Students can move vertices around to experiment with creating their own fractals.
• Using a real-time interpretive version of Java, Ken created Flower, in which students can directly edit the code and immediately affect the flower. In Musical Rubber Ducky, students can affect the ducky but also the music too using live code. Can we invite them to the narrative as well?
• Ken developed a Pride and Prejudice reader. Before the invention of books, literature used to be written on scrolls. Someone later introduced the concept of pages, a fixed length record that changes per edition. But writers wrote and thought in terms of sentences and paragraphs, not in pages.
• In this interactive map of Pride and Prejudice, you can see the entire book's topology. You can easily identify the key chapters of the book using filters and highlights.
• Asteroid is a game that topologically and mathematically exists on a torus or a doughnut.
• The second book to use the interactive map is Winnie the Pooh. This version also includes a procedurally animated bear that users can play around with using live coding. The code editor includes tools that enable learning programming easier. When you insert a 2D array, a map will appear that lets you set dots and automatically generate the points in the array for you.
• Everyone should be able to program. It should be as accessible as painting, filming, and writing. People need immediate feedback to instigate learning.
• Ken is really interested in how to power up this maker culture.

Question and Answer

• Learning tools should be like Google Docs in that they are collaborative, shared documents. People should have conversations with each other about creation. Learning is a performative activity.
• Just like how jazz is to music composition and improvisation is to acting, we need live perfomative structures for coding to learn programming.
• What's the difference between games and simulation? Games have goals.
• We need to increase programming literacy and convey programming as a liberal art. Kids need to look at code in their early stages.
• Ken remembers reading a children's book when he was a child and at the end were the advanced notes for the teacher to read. Although Ken couldn't read the notes as a child, he understood that that's what he was learning and that he'll eventually be able to read the advanced text. Kids shouldn't be using a fake language, but rather be looking at what the grown ups are using. They should be exposed to C++ or Java-like languages.
• Kids are learning reading through other subjects such a history and science. We should teach coding gradually in the same way.
• Many people are reluctant to learn coding, but that's okay. We should focus on teaching the kids. We don't currently live in a society that requires that level of technical knowledge, but this might change ten years in the future.
• Excel is a programming environment that many people use, despite not being programmers. Max/MSP is a programming language that non-programmers such as visual and sound artists use. Both of these programs are always running live and give immediate feedback. There's no intermediate building or compiling stage.
• People don't necessarily need to learn C++ or Java the same way that not all writers have to linguists. Guitarists don't need to learn how to make guitars.
• Programmers have a sense of being macho. Music, on the other hand, is more inclusive. We need a real-time sharing structure and a more inclusive environment for non-programmers.
• Will books become non-linear in the future? This may very well happen, but it won't replace the books as we know them. Different media can coexist the same way that different instruments can coexist. It's not all about hacking and recombinations. Technology is the wrong word to describe media; these are instruments.
• Will the secondary functions of the interactive reader overshadow the literary work itself? The work lives on if they're great. Beethoven lived on when the Beatles came around and the Beatles didn't go away when Lady Gaga came around.