Summer 2022 DREU Project
Finally finishing up the project! This week, I struggled the most with analyzing the qualitative data in the results section as it was my first time doing so. Z.I. described qualitative analysis is like mastering a type of art and gave me a paper to read on how the section shouldbe written which helped quite a bit with the writing.
We conducted the study this week. The study took three different sessions (8 participants per session) to complete. Although there were some minor setbacks such as the temporary crashing of our hosted web chat site and accommodating for participant no shows, the study was a success and we gathered some interesting insight and data on users’ expereicnes.
We finally reached out to participants and conducted our pilot study. We realized that the breakout room on Zoom along with manually assigning the participants to the chatrooms became hectic to manage and the study felt rushed. We found several solutions to directly provide participants the links ahead of time with the problem statement displayed already in each room and will be working on that in the next week. I am also refining an improved version our NASA TLX survey and reaching out to many other PhD students for their suggestions.
During my meeting with Z.I., we tested out all the interfaces with different problem solving tasks. We found that survival scenario problems allowed more room for conversations and disagreements. The recruitment flier and elgibility survey has been finalized. We have decided that our study will be remote to ensure diversity. We have been looking at ways to recruit participants online through Facebook groups, personal newsfeeds, Reddit, and Prolific. Through Z.I.’s help, the front end of the web chat application is now working with the different interface types.
I have given a lot of thought to what problem solving prompts could be used for the study. At first, I thought of river crossing problems but Professor Huang quickly rejected the idea as he did not see anything collaborative about it. He noted that I should think about teaching tasks or tasks that involve team building. This made me think that maybe an open ended decsion makiing problem like survival scenarios might be better than the tactical tasks I initially came up with. The draft of the recruitment flier has also been finished. Further, IRB determined our study to fall under program evaluation which is great news for us.
Professor Huang gave me lots of comments on my 50 measuralbes list. He recommended that it was time to narrow them down. Z.I. suggested I look into NASA TLX as one of the evaluation/assesment methods because it can measure multiple factors. I ultimately decided on a title for the paper and two main research questions for the study. I also chose to focus on problem solving as the measurable while using NASA TLX inspired surveys and semi-structured interviews as the measures. This week gave me a lot of insight on how different qualitative and quantitative study designs work.