1. Introduction

    GestuRater is trying to address following needs:

    1. The need of chefs (who cook in big dining halls) to feel appreciated and get genuine feedback from all the students about daily meals.
    2. Students need a way to give quick , instant and on-the-spot feedback about the food in an engaging way (not online/pen-paper format).

    To address these needs we are building a gesture based feedback system which will be deployed in dining hall areas for students to give feedback in a socially engaging manner. The students

    will be able to rate each and every food item on the menu for that day. The recorded feedback can then be viewed on a daily basis by the chefs. This feedback will make them “feel appreciated”

    and will provide cues for improving the meals further. We also plan to display the overall food ratings per food item on a particular day. This way students can get to know what other students/their

    friends think of the meal. We plan to use very basic and smooth gestures such as sliding up / down ; sliding left / right for classifying a food item as good/bad and to browse between food items

    respectively. Some people would be giving feedback with their trays in hand hence keeping gestures as simple and intuitive as possible makes more sense.


  2. Methods

    Study Design & Tasks - We plan to test the system in 2 different environments:

    A) Dining hall on Stanford campus : This is real scenario testing. We would be deploying the system just outside the dinner hall in Storey undergraduate housing on Stanford campus.

    Study structure:

    The Storey undergrad housing dinner hall serves dinner from 6 pm to 6:45 pm daily. We visited this hall last week with our coach Jimmy to decide a suitable area for deployment. After

    discussions, we decided that the tv screen and the kinect would be set up in the lounge area by the the diner hall (hall exit door leads to this lounge,please see images for details).

    Normally after dinner students walk out of the dinner hall and pass by this lounge area while returning to their dorm rooms. The area has a few couches(which we will be moving to

    make more space for our study). The kinect camera will be projected against a wall to avoid noise due to other people walking by. We would put posters with arrows on the walls to

    direct people towards the setup area. These posters will be pasted in an area where students wash their dishes, just before they walk out.


    We also plan to have a short survey with 4-5 questions for the students who use our system. We will be using Qualtrics for this purpose for efficiently compiling the survey results . The

    participants will be asked to rate the system on a scale of 1-5 on different parameters. We will also have 1 question for general comment and feedback. We plan to gather user experience

    and feedback about GestuRater using the following survey:


    Please provide responses ranging from 1 to 5 (1 = strongly disagree, 5 = strongly agree)

    1. It was fun to give feedback using gestures.
    2. GestuRater was easy to use.
    3. I would like to use this system again.
    4. Feedback using gestures is much better than online/paper-pen format feedback.
    5. Please give general feedback/comments about the system.

    User recruitment:

    1. We would talk to a few students explicitly to come over and rate their dinner using GestuRater
    2. Some students would automatically come to explore the system by seeing the poster and arrows.

    Normally, there are about 40 people who eat in this dinner hall during the 45 mins span. We hope to get our system tested by at least 7-10 students. Each user should take around 1 min. maximum

    and a few minutes for survey and discussion. We hope to get data for further analysis from : quantitative and qualitative data from gestures, survey responses and feedback from users.

    Photos of the Study Space


    B) Controlled Experiment Environment - This study style will help us gather more qualitative data as it would be more rigorous.

    Study Structure and User Recruitment:

    We plan to recruit a few friends/classmates and ask them to play with GestuRater imagining they just had a meal! We will ask same user to use the system multiple times hoping that there

    would be potential constructive feedback and bug detection in each iteration. We might not be able to get very qualitative feedback from previous study as each user will interact with

    the system for less than a minute, hence the need for controlled experimental study. We plan to record videos while users are interacting with system (with their permission) so that

    they can be used later for analysis and bug fixing. We would prefer not to record videos during the dining hall study as it might make the participants react artificially by making them conscious.

  3. Results

    We tested GestuRater with 10 users - students from Storey (undergrad housing) kitchen. 8 of these users filled the survey responses. Here are the survey results in tabular and chart format with some analysis.

    Q1. Rate GestuRater on a scale of 1 to 5 (1 - low/poor ranging to 5 - high/excellent) on following criterion:

    As seen in above table we tried to use contradictory criterion to validate user responses. Most of the people found GestuRater to be Fun to use which is obvious from the high mean 4.25.

    Moreover for boring to use people gave a low rating (1.50 mean) which means not many people found it boring. These 2 contradictory question ratings tie well with

    each other. Similarly ease of use had a high mean of 3.50 which implies most, if not all, users found the system easy to use. Complementing this number is the mean from difficult to

    use which is low (2.63)


    People also rated GestuRater high when compared to other means of feedback such as online systems/pen-paper. As per user interviews this is because GestuRater helps users give feedback on the

    spot immediately after having food versus going back to their rooms and then giving feedback.

    Q2. I clearly understood the meaning and usage of each gesture

    Most of the users understood that the system uses 4 main gestures up/down for rating and left/right swiping for previous and next.The instructions page in the system , which appears when a new

    user steps in proved to be helpful. There were a few people who were confused about gestures, mainly next and previous gestures.


    Q3. I would like to use GestuRater again

    On being interviewed a few users mentioned that this would be a very helpful application if used for feedback on a daily basis. We believe this is why many people agreed that they would like to

    use this application again. It was a combination of the fun element using gestural app and the usefulness of improving food quality or thanking the chef for good food!


    Q4. Any general feedback/comments?

    In general users had mixed feedback about the application, most of the people who understood the gestures and got them working with kinect liked the app. Some users struggled with the registering

    gestures on kinect, that was partly also because of quick switching of food item images.


    Useful Observations Per User

    First User:

    • Quote: What does swipe left mean?
    • swiped wrong direction to start
    • held hand at bottom after first swipe, which went unrecognized for a long time, which cause a loop through all dishes quickly
    • user did gestures like old gestures, not steady moving of arm/hand
    • held both arms out
    • confused by swipe left

    Second User:

    • Quote: Is this our food tonight?
    • had to go up high to rate, struggling on toes
    • really apprehensive to swipe. confused by swipe left

    Third User:

    • User Suggestion: put on instructions page that you need to level your hand after each dish rating

    Fourth User:

    • getting close to screen to read directions
    • not raising arm high enough at first, had to raise it harder and with lots of effort

    Fifth User:

    • stepping in close to screen to read directions
    • left hand held down after first swipe which led to quick cycle to end
    • got swipe on his first try!! (first user who has)
    • looked scared to use system
    • Quote: Is that it? at the end of using app
    • Took 45 seconds

    Sixth User:

    • leaning in close to read instructions
    • swiping wrong direction to start
    • Quote: This is COOL! said with a squeal
    • Had to go too high to rate well, standing on tip toes, struggling
    • very clearly enjoyed the experience greatly
    • Took 22 seconds

    Seventh User:

    • stepping in close to read instructions
    • accidentally cycled through all dishes by lowering hand
    • tried a second time
    • got process almost 100% accurate on second go-through but using old gestures
    • Took 15 seconds

    Eigth User:

    • leaning in close to read instructions
    • got swipe right correct!
    • started rating wrong immediately by holding hand down on side and cycling through dishes, but quickly corrected to rate last dishes
    • wanted to test again
    • did gestures like holding a weight and flexing

    Ninth User:

    • stepping in close to read instructions
    • other user stepped in background to chat briefly and nothing bad happened
    • tried to use left hand to swipe to begin
    • accidentally skipped through all dishes rating wrongly
    • app was not working correctly at all - wouldn’t start (figured out it was an internet issue and connection wasn’t being made)

    Tenth User:

    • leaning in close to read instructions
    • Quote: Good app that’s fun
    • didn’t realize that rating was on an absolute scale - thought raising image to where it would go between good and bad then somehow indicate that was his rating


  4. Discussion

    Shortcomings/Caveats

    1. More users : Though we tested the system with 8-10 users we feel testing it with more users would have been more helpful. Mainly because GestuRater is an application which will be

      used by many people when actually deployed. It would have been good to do a load testing on the system.

    2. Testing with more functionalities : there were a few important features that we could not implement before testing the app in Storey. Having those features fully functional would have

      given very different (probably better) user study and survey results.

    3. Testing with anonymous users : Few people who tested the app were friends and collesgues of one of our team members’. This could have led them in being polite while responding to survey.

      We should probably have tested with a user group of completely anonymous people to get a more frank feedback.

  5. Implications

    1. Speed:

      Many users felt that the system was very fast when moving to next image after rating a given image. We also observed this issue when users were using the application. To fix this

      we will introduce a time lag while moving to next food item. This will give users enough time to reposition their hand to rate the next food item.

    2. Instructions Page:

      By observing users we realized that a lot of instructions on the first page was confusing users. We would reduce the wordy sentences and have one word describing what each gesture means.

      This will also help having a bigger font size and clarity.Also after reading all instructions many users were clueless in which direction to swipe to start rating food. To address this

      we plan to have an explicit arrow at the end of the page saying swipe to begin

    3. History of User Rating

      We observed that users felt slightly disoriented that a food item just disappeared completely after rating it as good or bad. Hence we plan to show a small resized images of

      food items in good/bad sections of the screen.

    4. Statistics Page:

      We asked users if they would like to see a summary page in the end with percentage of votes received by each food item. Most of the users felt this would be very useful

      and we plan to incorporate this in our next version.