The Project:

In our UX Research class, from our previous A test, where I conducted a usability test on a weather app we were tasked to create a B test to measure the user experience of devices, employing a triangulation of methods. I compared my previous A test Weather Up, to my then B test, CARROT’s weather app.

  -Duration: 3 Weeks

  -Methods: Surveys, Interviews, SUS Likert Scale

  -Tools: Google Forms, Microsoft Production Reaction Cards, Desmert’s PrEMO Survey

Introduction:

I have designed this test to test CARROT Weather’s usability through task and time completion. I then compared the CARROT Weather results with the previous task and time completion usability, A Test done for the Weather Up weather application to answer two research questions. The two research questions asked were: 

1. Are the additional features CARROT has Intriguing enough to make it a primary weather application?

2. Will the tasks that are required in CARROT be harder to complete than the Weather Up App?

Previously, with the data collected from the Weather Up application, respondents chose not to replace their primary weather app with Weather Up. Since the Weather Up application wasn’t enough to get respondents to make it their primary application. I wanted to see if the additional features that CARROT had, such as humorous commentary, a custom user interface, geographical missions, and its AR mode feature would be intriguing enough to make it their primary weather application. As well as if respondents would find the tasks they were asked to complete with the features on the CARROT app harder to complete than the ones on the Weather Up application.

The challenges that might occur while completing tasks will most likely happen when setting up AR Mode, completing the mission, or performing the citizen’s weather report. When performing the citizen’s report, sometimes the script moves faster than you can read it; while performing the mission, the commentary gets in the way sometimes, and AR Mode needs a flat surface to be set up, and if there isn’t a flat surface, it would be difficult to set up. Regarding the difficulties that would occur while completing the tasks at hand and the multiple features that CARROT has, I’ve come up with my hypothesis. CARROT Weather’s additional features will not be intriguing enough for respondents to make it their primary weather application. Tasks on the CARROT App will not be harder than the tasks on the Weather Up application because the CARROT’s features are properly labeled.

Methods:

To measure the usability of the CARROT Weather application, I used a triangulation of methods. Those methods included a SUS Likert, Microsoft Product Reaction Cards, and a Product Emotion Measurement Instrument, all chosen to assist in measuring the task and time completion. The survey included the same five participants who participated in the Weather Up usability test because of their prior knowledge of Weather Up. I thought it would be good to use the same respondents to see if they would have the same view of refusing to change their primary weather app. I completed the survey on Google Forms that included a 5-point Likert scale with 16 questions that ranged from strongly disagree to strongly agree, six yes or no questions, and one quantitative question following the post-survey. Pre-survey included four questions that asked about biological sex, age, their operating system, and how many times a week they check their primary weather app. The tasks that respondents were asked to complete engaged them within the application. I used a PreMo survey to track their facial expressions and Microsoft Product Reaction Cards containing 118 reaction cards to understand their feelings from their tasks. This triangulation of methods was designed to determine if respondents would replace their primary weather app with CARROT and if the tasks asked to perform were harder than the tasks on the Weather Up application.

Survey Questions (In bold are the usability test instructions)

1. How long did it take to download the app? Download CARROT Weather app from the AppStore or Play store.

2. Did you have any issues loading the app? 

3. CARROT successfully chose my correct location. Allow CARROT to use your location.

4. I felt comfortable allowing the app to access my location.

5. CARROT's temperature of my current location is the same as the weather app I normally use. 

6. The app was easy to use.

7. I found the personality I chose humoring. Choose a personality of your choice for your weather assistant.

8. It was difficult to understand how to use the app. 

9. It was easy choosing the city from my undergrad. Search for the city location where you received your undergrad degree

9B. Finding Precip Change was difficult. Find the Precip Chance for Sunday December the 12th.

10. It was difficult performing the scripted citizen weather report. On the Weather tab click the top left corner and perform a scripted citizen weather report.

11. I found performing the citizen weather report interesting. 

12. It was easy performing the tasks to set up AR Mode. On the Weather tab click the top right corner, select AR Mode and follow the instructions

13. I found the display of AR Mode interesting.

14. I found the Overkill Personality with profanity to be too much. Change your Personality to Overkill with Profanity in settings

15. Completing the Current Mission available was hard. On the CARROT tab complete the Current Mission available.

16. I was able to successfully complete the Mission.

16B. Completing the Mission took too long. 

17. I found this app engaging.

18. I spent my own time exploring this app after the survey.

19. I'm going to use CARROT over my usual weather app.

19B. CARROT is too busy for a weather app.

20. I would recommend using this app to my friends/family/significant other.

Results:

The results showed that once again, the same 5 respondents that were asked if they would replace their primary weather app with Weather Up, would not replace their primary weather app with CARROT Weather as 100% of them said no. 100% of respondents used an iOS for their survey and application tasks. Regarding how often respondents checked the weather, 40% checked it sometimes, 40% checked it often, and 20% always checked the weather, which are the same results from the Weather Up usability test. 80% of respondents found the downloading time to be less than a minute, while 20% found it to take 1-2 minutes. Even though 80% of respondents found the app easy to use 40% of them found the app to be difficult to understand which makes me wonder if respondents were confused by the various numbers associated with categories like wet bulb, precip amount, wind gusts, uv index and cloud cover meant within CARROT. The difficulty in the tasks came when performing the citizen’s weather report, 80% of respondents found it difficult, and finding percept change 80% of respondents found that task difficult as well. On a positive note, 100% of respondents found the commentary humoring, and 80% would share this with family and friends. 60% of respondents said CARROT was too busy for a weather app even though 80% of respondents had excitement as a facial expression during the PreMo survey. 

Discussion:

From the data collected, similar to Weather Up, respondents found the application too much for a weather application. Respondents found the personality humoring; 60% found AR Mode and the citizen’s weather report interesting but not enough to make it their primary weather application. As for my hypothesis, half of it was correct, and half of it was wrong. Initially, I said CARROT’s features would not be intriguing enough to respondents, which proved correct. Although respondents found the application interesting, exciting, impressive, and innovative, measured by our Microsoft product reaction cards, respondents found the application to have too many tabs and icons. It seems like respondents would still like a simplistic version of a weather application. Respondents aren’t interested in opening up an application with unnecessary funny commentary or playing geographical games on an app. Instead, they’d rather swipe on their iOS app, look at their weather widget and go about their day. 

When it came to being wrong about the other half of my hypothesis, initially, I said CARROT’s tasks would not be harder than Weather Up’s because CARROT’s features are properly labeled. That proved to be wrong. I didn’t consider that I had given respondents more tasks to do on CARROT, such as finding precip chance, completing the mission, citizen weather report, and AR mode. Some of the tasks included finding Antarctica on the Weather Up application, which most respondents had trouble with. 40% of respondents on Weather Up found the hardest task difficult, whereas 80% of respondents found the hardest task on CARROT difficult, which was finding precip chance. Finding precip chance seemed to be difficult because of how many extras you had to go through to find it. The only way for it to be found was for respondents to search for it by clicking various tabs through the application.