Bing

Team Bing

We're working to make use of the awesome technology in the newly released Leap Motion to help improve children's education.

Team

Francisco Guzmán (@fguzman) ~ Design Lead
I'm a coterm in Symbolic Systems with a focus in Human Computer Interaction. I have an interdisciplinary background - my undergraduate degrees are in MS&E and STS. I've been involved in a few different areas through internships, including product management, strategy & operations and more recently roles in UI/UX design.

Amanda Lin ~ (@pandolin) ~ Development Lead
I'm a senior in Symbolic Systems senior a coterm in CS, specializing in HCI. Last summer, I worked as a Product Management intern at a startup in SF, and the summer before that, worked as an intern on the Developer Relations team at Apple.

Vivian Shen ~ (@vivianshen) ~ Evaluation Lead
I'm a junior majoring in CS focusing on HCI. I've worked as a software engineer on the Ads team at Google in the past and done research at the CS Security lab at Stanford. On the business side I've done business development for a mobile advertising startup and also worked on consulting projects for tech-focused companies.

Tim Shi ~ (@timgshi) Project Manager
I'm a senior in HCI and coterming in HCI as well. I'm interested in product management and have a diverse background in engineering and business, with internships at Google and Nike in the past. At Google I was an iOS engineer on the Google Search App and at Nike I was a strategy consultant on Access to Sport.

User Need

Our focus is on the needs of children. We have conducted needfinding exercises in three locations: nursery/schoolrooms, children's hospital waiting rooms, and local parks. Our preliminary focus is on creating a more interactive experience in these locations, and helping children interact with one another without being intrusive. We want to satisfy their need for an engaging and stimulating activity (that's preferably educational, for the sake of their parents) while encouraging them to have fun with each other as well.

Initial Prototyping

Photos of initial brainstorming:

As suggested, we went for quantity with our initial brainstorm. We considered many different ideas because of the many areas in which we are interested in helping children - such as the park, nursery and children with learning disabilities.

Three promising approaches:

• Build something (simulating physical objects kids might not have access to)
• Karel (teaching preliminary fundamentals of STEM fields)
• Piano (play that encourages social interaction)

Feasibility tests:

• First test This first video was a way to test the technology we are using, the Leap Motion. In the video, we try a variety of finger movements in order to get an understanding of the detecting capability. This first test served as a pathway for us to explore some of the other motions in the videos below.

• Balls This video is a continuation of the testing and the different features of the Leap Motion - this testing helped us understand the range that the device has, which informs the way we think about potential prototypes.

• Stirring! This testing video was inspired by one of the implementations we proposed during our skit - cooking mamma. Educational games are still of strong interest to us, and so we wanted to experiment and see how feasible some of the interactions we'd like to incorporate are.

• Four hands! In our observations we've noticed how important social interaction can be in enabling other types of play between children and their parents. This test was conducted to see how many inputs can be detected - again, as a way to understand the realm of possibilities in the design of our applications.

Two Video Prototypes

• Building blocks Physical play has been shown to be instrumental for a child's play and especially at an early age for cognitive development. When thinking about the way children play, especially children with disabilities that prevent them from fully grasping physical objects, we wanted to simulate an experience that can lead children through that process in ways of interaction. This video prototype is a simple way to show our main idea - the child would move a digital object through gestures, but still understand the repercussions that layering objects can have - learning the fun in building. We feel this would be especially helpful, as was mentioned before, for children who have difficulty grabbing physical objects.

• Karel! There is a strong push in education to better expose children to STEM fields at an earlier age. From some of our observations, we saw that many teachers are playing around with ways in which they can achieve this - especially for younger children who's capabilities may not be advanced enough to grasp algebra or computer science, for example. There are small things we can do, however, to put in place the building blocks for that learning. This video demonstrates Karel moving through a maze in order to get to the star. The child would break down the actions, one by one, that Karel has to take in order to reach the star - this is a way of establishing the basic thinking behind computer science.

One Sketch (not related to Leap Motion - but a potential idea to continue to ponder on)

• [Piano] We also thought about gestural interactions that do not involve a screen and have the emphasis more on physical objects with some sort of digital feedback. The piano idea would involve having two pads where children can jump on and then the computer would give audio feedback - we can even set this up as a piano and have the sounds be keys on a keyboard. This would be a good way to mix physical learning and digital feedback.

(http://imgur.com/LEg5UBe.jpg)

Wizard of Oz Testing

We came to class on Thursday 2/14 ready to Wizard of Oz test our gestural Karel the Robot prototype and had several users test controlling Karel using gestures. The prototype involved using hand gestures to move commands ("turn left", "turn right", "move forward") from one side of the screen to the other in order to form a "program". Users saw Karel in a maze and were prompted to help guide him to the exit by building a program and clicking the run button.

As we tested with each participant we interviewed them to get a sense for how intuitive our prototype was and learn how they felt children might perceive the game. Through our tests, we heard multiple times that this idea might be too abstract for 3-5 year old children and that the need for a gestural interface was not clear, i.e. a simple arrow key interface would be just as effective. Based on this feedback we decided to scrap the Karel idea and head in a new direction.

On our visit to the Bing Nursery we learned that the educational units taught there are driven by student interest. Teachers test and gauge interest in various topics by laying out objects on the "discovery table" and watch the students to see what might be most popular with the kids. We plan to prototype a tool to aid teachers in enhancing the discovery table by providing a gestural interface that lets students interact with various potential unit topics through a prezi-like visual. This system will be able to track and gauge early interest in subject areas for the educators before they make the effort to develop a unit.

Feature Choices

In creating our enhanced, digitized discovery table, we knew we had to include features that would make it both intuitive and easy for children of such a young age to use. First, we decided to include all of the children's photos on the initial login page. This way, each student can point to his or her own photo in order to "log in." After much brainstorming, we decided this would be the easiest way for teachers to be able to track which student was interested in what subject. All of the child's actions after logging in are stored in our database, with information that will be provided to the teachers every time they want

Next, we wanted to implement a way of easily navigating through a series of topics, so we decided to include a number of different topics on the initial page after logging in. Each of the topics has a photo and a title, because during our observations we noticed that the kids could not yet read. The students can then point to any of these titles in order to reveal more photos related to the topic. Meanwhile, we measure how long kids spend on each topic and store that information in our database so that teachers can see which topics the student was most interested in. We will also aggregate all of the time spent on each topic so that by all of the students so that teachers can gauge which topics had the most interest.

Ultimately, we wanted to add in these features to 1) ease the process for teachers at Bing for finding new topics to teach, allowing them to do a "discovery table" without committing to first purchasing physical objects related to the topic, and 2) make it so other schools can implement this same idea of a discovery table in order to create a curriculum for students.

User Testing/Evaluation

Introduction: Describe your motivation and goals. Note your driving questions and hypotheses.

Though we originally started out focusing on kids at Bing Nursery, we realized that the age demographic there was too young to fully take advantage of the Leap technology. We ended up pivoting and using our prototype for elementary schoolers instead.

Our motivation was to create a fun educational experience for kids and keep them engaged in educational content. A lot of times textbooks can be very 2D and un-interactive, so we wanted our application to be a way for kids to learn dynamically. Our end product is essentially like a digital textbook with fun content and gestures that keep them engaged and provide them with immediate feedback on the content they're learning about.

Some of our driving questions included: What is the best age demographic for these kids? What needs can we address that their current educational materials can't? Can we help keep kids more engaged in the educational content than before? Will the children be able to easily pick up the technology and use it?

We hypothesized that the novelty of the Leap motion software would draw kids in, as well as the interactive content of our application. We were a bit unsure of how adept the kids would be at picking up the technology, but we hypothesized it would be a relatively quick learn for them because a lot of them have exposure to iPads etc. already.

Methods: Describe your study design and tasks. Who did you recruit and how did you recruit them?

Study Design:

We will bring in 3 laptops and 3 Leap Motions into the elementary school that we are testing at. We will set them up in the library, and during lunch when kids come into the library they can volunteer to use the application if they wish.

We will have the kids queue up to use the leap one by one, and let the kids crowd around the computer to read the content together.

The tasks we will have them do include:

• logging into a user profile
• paging through the various ‘slides’ of the application (forward and back)
• answering quiz questions
• exploring various facts on one page and viewing content

We will ask the kids questions about what they think they should do to advance through the pages, if the content makes sense and whether they find it interesting. The questions & things we want to observe are below.

Observe/Ask about:

** Technical

• Finger position of the Leap (whether they can properly select)
• Do they understand how to operate the Leap?
• Are the instructions of what to do intuitive?
• Is there enough feedback from the Leap to keep them interested/not confused?
• What is frustrating about the tech?
• Are they interested in the technology or the content?

** Content

• Is the content engaging?
• Are they interested in learning?
• Can they understand the content (reading ability etc.)
• How can we change the content to better suit their needs?
• What other content would they like to see

Who:

Elementary school students (grades K-5) at Nixon Elementary School

How:

Sent emails to the elementary schools in the area, had preliminary meeting with the principal to explain project and talk about best testing environment. Set date for bringing in the prototype and had teachers inform students that there would be a demonstration happening at lunch if they wanted to come and play with it.

Results: Describe your data and present the results of data analysis (observations, statistics, charts, etc).

We ended up having a really good turnout for our testing. We went to the elementary school two days in a row during lunch at the library, and had the teachers announce that we had come to demonstrate the game. The first day we had ~15-20 kids lining up to play the game, and we maybe had ~8 of them be able to play the game because we only brought one laptop. The kids were really excited about the gesture based nature of the game and we knew there was a lot of interest, so we decided to come back another day.

When we came back the next day, we came with 3 laptops & leaps and could manage the crowd better. This time, through word of mouth, we had more like ~30-35 kids show up to play with the game. All of them had a chance to use the leap briefly. We recorded observations below.

Content feedback:

• really liked the leopard fact exploration page - most immediate feedback
• liked doing the quizzes and also liked less interactive content like the pictures & facts
• kids aren’t immediately drawn to reading the text, but when the answer div pops up they are immediately drawn to it. need to be more eye catching.
• some kids didn't even know how to use a mouse when we asked them to try playing with a computer for comparison, but found the leap intuitive to use
• teachers mentioned they liked the technology a lot, as it was much more sanitary than things like computer mice and touch screens
• teachers had multiple requests from students to buy the technology
• the kids found the cursor really helpful for giving selection feedback (vs. the first time we tested without it, when the kids couldn't tell if the leap was even logging their motions)
• for our final prototype, we also added a calibration page, which helped users figure out where to keep their fingers, the range of motion, and how to perform the gesture needed to navigate through the pages

Technical feedback:

** Screen clicking:

• a lot of kids were clicking the screen to select something instead of hovering
• they wished for clicking instead of hovering-- didn’t understand that hovering was selection
• needed explicit selection mechanism, not just hovering.

** Leap gestures:

• a few of them tried to draw a circle facing the sensor instead of facing the screen (wrong gesture)
• circle gesture difficult to understand at first, but once they realized how it worked it made a lot of sense and they could easily page through.
• most held their hand very close to the sensor (too close) and had to be told to adjust.
• the sensor only reached the top 2/3 of the page so they couldn't reach the bottommost box on the leopard fact exploration page
• quiz answers would be triggered immediately upon paging, even when the kid didn’t mean to select the answer.

Overall

• interested in both the technology & the content-- “Wow they can jump over 10 cars??”
• all commented on how cool it was
• many commented that gestures were less tiring than using a mouse, more intuitive
• many commented that if it had more immediate feedback they would use it over a mouse (ie less buggy)

Discussion: Synthesize what you learned, document any shortcomings or caveats or your testing.

We learned that kids are really adept at picking up new technologies--within a minute or so a child could properly use the system and the children watching could all understand it as well. They also had pretty long attention spans and easily understood the point of the game and technology. The novelty of the gesture-based system helped keep them engaged with the educational content. We also found that the gesture-based system was good for keeping them entertained, but also had practical implications like being more sanitary than a touchscreen and being easier to manipulate than a mouse (which they said was painful after a period of time)

We found that they thought the picture based content was really engaging, and loved using the Leap for interacting with the content. They found the technology a cool way to make the content more interactive. They wanted as immediate feedback as possible-- they expected the find gestures to correlate with the speed of a mouse.

A lot of the kids were used to a certain model of technology (ie. clicking vs. hovering for selection) that our demo didn’t always parallel with. We’re going to change it so that our demo aligns a bit more with what they’re used to (see implications.)

Implications: Describe how you plan to apply the study results in the next iteration of your application.

• Finding: kids don’t understand circle swipe clockwise & counterclockwise-- separate the two instructions. 3-5th grades could understand.
• Finding: kids not very interested in blocks of text-- change to make it more photo based
• Finding: kids try to move their fingers very close to the leap-- somehow separate the two or have some text telling them to stay 5 inches away or something
• Finding: not enough immediate feedback of what their gestures were doing--add a cursor
• Finding: quiz answers selected too quickly & on accident-- have some kind of trigger event that means “I want to select an answer now” or have a time pause before allowing selection
• Finding: don’t get hovering, want to see clicking-- add instruction that clicking == hovering, or switch to clicking (hard with the leap though)

Note: we couldn't take too many pictures because of privacy issues with kids, but we were allowed to take a couple if their faces weren't shown.

Second Prototype Features:

• Added a cursor showing where fingers are
• Added audio feedback
• Added pagination to show which page they're on
• Switched to use circling to page between
• Changed colors to be more kid-friendly
• Pause at each page before allowing selection
• Make the zoom page zoom properly
• Have a page where you can select different parts of the leopard and show facts
• Have more interactive content-- sounds, brighter color feedback and more quizzes
• Changed positioning of the quiz answers to make it easy to select answers (vertical instead of horizontal positioning of the images)
• Fixed "jumpy" cursor
• Fixed divs to properly hide/show new elements
• Fixed leap frame rate