Portfolio
ChaiOne
Applying behavioral research to uncover user needs in energy and heavy industry.
Unified Customer Platform
Segementation Survey: Voice of the Customer vs. Voice of the Employee
-
Unified Customer Platform
The Challenge
Our client, a Fortune 200 services and commodities provider, separately handled their customer verticals. Service quality varied widely. To improve efficiency and cross-selling opportunities, they decided to consolidate all their services into one, adaptable platform.
The user-base was extremely varied, from older adults managing few accounts all the way up to commercial operations with hundreds of staff, differentiated roles, and thousands of accounts to manage. The software needed to be easy to use for the entire population, but be flexible enough to serve extremely demanding, many-to-many arrangements of user, account, and services.
Unique Environmental Challenges: while our initial research was performed directly with the main client, the agile software development project was contracted through their technical partner, another Fortune 200 company. We were brought on as a sub-contractor to handle front-end development.
The Methods
Remote User Interviews: during exploration phase, more than 50 people (across customer groups and internal SMEs) were interviewed to determine how current services fit needs, how competitors were perceived, and to identify critical requirements for each market.
Usability Testing: all features developed were assessed with moderated remote usability testing to compare Task Completion Rate, Single Ease Questionnaire, and the UX-Lite measures for Usefulness and Ease of Use. While I did provide feedback, I worked with several UX Designers who provided our wireframe test prototypes.
My Contribution
Research Review: I was responsible for detailed knowledge of all previous research.
User Interviews: I directly performed the last 8 user background interviews.
Remote Usability Testing: I designed the study strategy, worked with product owners to prioritize features, recruited, moderated testing, analyzed, and reported all results for two rounds of testing for 40 subjects.
The Result
Bench-marking for the initial product release currently places it within the top 10% of consumer software. It also approximately meets ACAG AA standards (though accessibility testing is planned for later in the project).
All of my usability tests were fully documented, performed on schedule, and under budget. I also ensured that our recruitment was representative of the population regarding regions, ages, and ethnicities, resulting in surprising natural variation that emphasized just how diverse needs can be in the average population.
-
Segementation Survey: Voice of the Customer vs. Voice of the Employee
The Challenge
A previous industrial services client had not only been recently acquired, but had finished a new customer-facing software initiative. With a new CEO at the helm, they wanted to know how their situation had changed.
My Contribution
Following the survey designer's departure, I performed a segmentation analysis to compare the experience between employees and customers to find key drivers for success.
The Methods
I built composite scores addressing Customer Experience, Tooling, and Culture (among others). A logistic regression was used to identify the composites' effects on reported satisfaction, engagement, and NPS scores.
Customer ranking of the value of a range of services was compared against the reported desire for those services, to determine whether their key needs were being met.
Customer ranking of Ease of Use and Usefulness were taken using UX-Lite
The Result
We found that:
- While employees were concerned about the quality of service customers received, customers were actually much happier than anticipated
- Customers were most positively affected by strong relationships with company personnel
- The new software was poorly received. Their previous software was in the bottom quarter of the market, but the new software was absolute bottom due to customer apathy
- Employees felt empowered in their immediate jobs and enjoyed them, but didn't feel they understood how their work fit into strategy.
- The most satisfied employees were those who had the highest customer interaction
To help shift towards the client's strengths, we provided a prioritized action plan to better communicate the value of roles and responsibilities and move away from a technology-first service approach to a more relationship-oriented one.
Seeing Machines
Ending operator fatigue and distraction in automotive and aviation, Seeing Machines uses cameras and infrared eye-tracking to measure changes in eye physiology and glance behavior.
Guardian Gen 3 Design
Euro NCAP
CanDrive
-
Guardian Gen 3 Design
The Challenge
With over 9 billion kilometers driven and more than 540k vehicles, Guardian Live fleet monitoring service combines AI detection of drowsiness & distraction with a human review process to ensure reliability.
The 3rd generation of Guardian hardware will pioneer integration of our automotive-grade Distraction and Drowsiness features with our manual review process to improve accuracy and driver experience.
My Contribution
I'm assisting in both the multimodal design of the Human-Machine Interface signals, and the systems integration between the hardware and Guardian Live cloud review.
Gen 3 will improve driver experience and acceptance via a calm technology approach to transparently communicate risk, instead of the "shock-awe-and-annoyance" strategy that is common in the market.
-
Euro NCAP
The Challenge
Euro NCAP is the auto industry regulator that awards "Safety Stars" for vehicle performance. For the first time, Euro NCAP will be requiring that all vehicles will be assessed for DMS performance.
Given the wide variety of vehicles and sensors, and the limited time to study them, how can validation transparently, fairly, and effectively assess all vehicles in a new, $3.2 billion industry?
My Contribution
I served as a technical contributor to the Tier 2 DMS Working Group via scientific support and validation design, and have served on the project throughout the entire life-cycle, starting with initial approach to NCAP to make our case.
I frequently had to present persuasive evidence to direct competitors and clients without exposing our IP.
The Result
After over 2 years of engagement, NCAP's DMS protocol testing will account for: cell phone distraction; actual drowsiness (not acted); microsleeping; driver demographics (age, height, complexion, etc.); noise factors (e.g: lighting, head coverings, makeup, etc.); individual gaze differences (lizard vs. owl behavior); & both single-glance and multi-glance behavior.
These features will reduce road accident severity the world over and ensure that driver safety features will fulfill their promise, instead of serving as a checkbox.
OEMs must provide detailed evidence dossiers documenting identified extremes, with in-person spot checks to maintain transparency.
The Methods
- Literature Review. I identified the driver behaviors that most contribute to crash risk, and reviewed existing distraction algorithms to find common features. This successfully persuaded NCAP of many critical features in the current design.
- Illustration of Test Behaviors. Utilizing the philosophy of "Design for Extremes", we created a list of driver testing behaviors that all DMS systems should be able to detect (i.e: failure to detect texting and driving is unacceptable, regardless of method).
To allievate concerns over repeatability and the required time to perform the test, I wrote a text-to-speech timing protocol, recorded myself performing all of the behaviors, and edited it all into short animations. This showed that all behaviors can be performed in under 15 minutes.
- Eye Closure Analysis. The common assumption for drowsiness is that it is a matter of eye closure. Reviewing our Guardian fleet monitoring database, I found that non-drowsy eye closures (e.g: squinting, laughing, etc.) actually account for ~90% of cases. Simple eye closure measures of drowsiness lead to an unusably noisy user experience.
- Digital Survey of Phone Usage. When proposing phone testing, we were unsure how drivers would hold their phones. I ran an international survey via Qualtrics and mTurk to find that drivers prefer to hold their phones on the passenger side, in the middle range of the wheel, and that it was not dependent on handedness nor drive-side.
- Assessment of Multiglance Standards. While single glance distraction's risks are well-researched, time-sharing glance behavior can vary widely. I found a demonstrable increase in crash risk for on-road gaze proportions of 1/3, and showed that it harmonized multiple competing proposals.
- Test Matrix Proposal. With the wide range of demographics, test behaviors, and noise factors, we needed to demonstrate that a dossier was a feasible expectation of OEMs. Collaborating with a founder, we developed a study design framework that showed that a minimally informative study was possible with 24 participants.
Papers:
- Fredriksson, R., Lenné, M. G., van Montfort, S., & Grover, C. (2021). European NCAP Program Developments to Address Driver Distraction, Drowsiness and Sudden Sickness. Frontiers in Neuroergonomics, 2, 786674. https://doi.org/10.3389/fnrgo.2021.786674
- Roady, T., Wilson, K., Kuo, J., & Lenné, M. G. (2020). How Do Drivers Hold Their Phone? Age, Prevalence, & Handedness. Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting, Virtual, 64(1), 1254–1257. https://doi.org/10.1177/1071181320641298
- Lenne, M. G., Roady, T., & Kuo, J. (2020). Driver State Monitoring for Decreased Fitness to Drive. In D. L. Fisher, W. J. Horrey, J. D. Lee, & M. Regan (Eds.), Handbook of Human Factors for Automated, Connected, and Intelligent Vehicles (1st ed.). Taylor & Francis.
-
CanDrive
Challenge
A Tesla Model S with Autopilot™ (SAE Level 2 Automation) was equipped with Seeing Machines FOVIO eye-tracking suite, alongside multiple cameras, a Time-of-Flight sensor, verbal n-Back task, a LED-based peripheral detection task (PDT), and a MobileEye vehicle informatics unit.
The Method
This mixed-methods study consisted of two different phases: first, controlled distraction tasks performed on a test track, and second, performance of n-back tasks and PDT on a repeated stretch of public highway. Following participation, subjects were interviewed regarding their attitudes and understanding surrounding driving automation.
The Results
- On the highway, drivers can reliably detect objects through near-peripheral vision, but this degrades with increasing visual angle, low-speed vehicle following, cognitive load, and age.
- Vehicle performance features are effective measures of cognitive distraction in manual driving, but not L2 mode.
- Gaze features, such as percent road center, are effective indicators of cognitive distraction, but with reduced efficacy in L2 driving, with the complexity of driving context seeming to provide the primary challenge.K
- Drivers engaging with their cellphone are more likely to rely upon Lizard-like glances, moving their eyes while keeping their head still.
- Interaction with L2 automation largely increases driver acceptance and trust of driving automation, potentially leading to overtrust, even when exposed to instances which might otherwise moderate their judgment.
- Hands-on-wheel sensors are not only insufficient predictors of driver engagement, but lead to mode confusion errors.
My Contribution
I've primarily assisted Phase 2 of the CanDrive project in experiment design, data collection, and manuscript review.Papers:
- Yang, S., Wilson, K., Roady, T, Kuo, J., & Lenné, M. G. (2022). Beyond gaze fixation: modeling peripheral vision in relation to speed, Tesla autopilot, cognitive load, and age in highway driving. Accident Analysis and Prevention.
- Yang, S., Shiferaw, B., Roady, T., Kuo, J., & Lenné, M. G. (2021). Drivers Gance Like Lizards during Cell Phone Distraction in Assisted Driving. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 65(1), 1410–1414. https://doi.org/10.1177/1071181321651147
- Wilson, K. M., Yang, S., Roady, T., Kuo, J., & Lenné, M. G. (2020). Driver trust & mode confusion in an on-road study of level-2 automated vehicle technology. Safety Science, 130, 104845.
- Yang, S., Wilson, K. M., Roady, T., Kuo, J., & Lenné, M. G. (2020). Evaluating Driver Features for Cognitive Distraction Detection and Validation in Manual and Level 2 Automated Driving. Human Factors. https://doi.org10.11770018720820964149
Texas A&M University
CHIAD (Creative Haptic Interaction At-a-Distance)
Research Question: Can people in mentally demanding, high-stress situations use vibration to shape their choices?
Studies were conducted with Engineering Acoustic Instruments’ C2 Tactor System.
Temporal Haptic Encoding
Speeded Cooperative Navigation Pt. 1
Speeded Cooperative Navigation Pt. 2
-
Temporal Haptic Encoding
The Challenge
Vibrotactile signals can be designed many ways, sometimes arbitrarily, particularly in matters of temporal overlapHow much information can be communicated using vibrotactile signals, and what temporal overlap is most most effective? Does the saltation sensory illusion improve communication?
The Method
11 subjects helped identify two different levels of signal complexity with three different types of temporal overlap: static, dynamic, and saltatory.
Responses were measured both on accuracy and the subject's requested number of repeats
The Result
Results showed that static was most effective for simple signals and saltatory was best for complex signals.Accuracy and user confidence was substantially reduced with an increase in signal complexity
Paper: Roady, T., & Ferris, T. K. (2012, September). An analysis of static, dynamic, and saltatory vibrotactile stimuli to inform the design of efficient haptic communication systems. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 56, No. 1, pp. 2075-2079). Sage CA: Los Angeles, CA: SAGE Publications.
-
Speeded Cooperative Navigation Pt. 1
Winner: Best Student Paper, Perception & Performance TG HFES 2013
Background
Vibrotactile stimuli may serve as useful for overcoming conditions of high cognitive workload due to stress or saturation of vision and audition.
We propose that using vibration will reduce the difficulty of decoding an instruction. Likewise, a message sent with gesture, instead of spoken, will take less time to encode.
The Method
Subjects were cooperative pairs The "director" was shown a cue card and had to guide the "actor" to the target cone. using each of three interfaces: voice commands over radio, vibrotactile commands via a GUI, and the same vibrotactile commands via Gesture.
The Result
The pairing of vibrotactile and gesture demonstrated greater accuracy, as expected. However, course completion time was not a significant factor, suggesting that the study was poorly designed to measure this factor.
Additionally, subject questionnaires demonstrated a marked preference for the vibrotactile and gesture system, as "tongue-tie" events were unexpectedly common with the verbal condition.
Paper: Roady, T., & Ferris, T. K. (2013, September). Supporting Speeded Navigational Communication via Gesture-Controlled Vibrotactile Displays. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 57, No. 1, pp. 1144-1148). Sage CA: Los Angeles, CA: SAGE Publications.
-
Speeded Cooperative Navigation Pt. 2
Background
The previous study suggested a performance benefit over verbal instructions when combining vibrotactile messages with gestural controls. However, the original task was kept as simple as possible. Would the same effect be shown in a slightly more complex environment?
The Method
Once again, cooperative pairs navigated a cone course. However, the course was expanded to offer 5 path options instead of 3.
Previously, the verbal and vibrotactile + GUI treatments performed comparably, To increase statistical power and due to its relative unfamiliarity, the GUI treatment was dropped in favor of verbal.
The Result
Performance was poor. Data demonstrated little of significance.
Subjects demonstrated a clear Hawthorne effect by pausing and waiting until they were confident in their response.
Additionally, a slight increase in cue card difficulty resulted in much wider variation in participant performance.
These results suggest that care must always be taken in the design of experiments and that even strong prior work can result in surprising challenges after slight modification.
Paper: Roady, T., Tippey, K., & Ferris, T. K. (2014, September). Speeded Vibrotactile Navigation with Gestural Control in a Multiple Choice Environment. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 58, No. 1, pp. 1989-1992). SAGE Publications.
FAA – PEGASAS Group 4: Weather Technology in the Cockpit
The US annual experiences roughly 55 general aviation fatalities, most commonly due to pilots flying into hazardous weather they are not properly trained for. Is this due to non-compliance, misunderstanding of weather information, or pilots not noticing weather alerts?
Sun N Fun Pilot Survey
Study Part 1: Wearable Haptic Signals
Study Part 2: Cognitive Workload & Haptic Perception
-
Sun N Fun Pilot Survey
Background
58 pilots completed a survey over their use and familiarity with in-cockpit weather information devices as part of the Sun N Fun FAA expo. Questions were tailored to identify flight experience and usage patterns.
The Method
Data was analyzed in a mixture of Microsoft Excel and R Studio.
Survey Results
Min Median Mean Max Age 19 60 56.13 75 Flight Hours 2.5 1250 2956 26000 I identified a link between pilot Total Flight Hours and reported acceptable flight distance from inclement weather. Increases across pilot experience showed a concentration about the FAA recommended flight distance. However, while pilots in the bottom two quintiles (0% - 40%) showed over-cautious behavior, risky flight behavior, flying at or within the FAA minimum of 5 nm, was shown to occur in the third and fourth quintiles. This suggests that as pilots gain experience, their performance will improve. However, the increase in skill may lead individuals of concern to become overconfident in their instruments and exhibit reckless behavior.
Report:Johnson, I., Pokodner, G., & Caldwell, B. S. (2016). Weather Technology In the Cockpit (WTIC) Project 4, Team C: General Aviation Weather Alerting, Phase II. (pp: 25-35).
Image courtesy Tom Wheeler, CC 2.0 License
-
Study Part 1: Wearable Haptic Signals
The Challenge
How many vibrotactile signals can a pilot interpret under workload? It depends on what technology is providing the signals and how the signals themselves are designed.
While Part 2 tests the actual performance of different signals targeted to available haptic hardware, the signals themselves are subjective in nature, so it is necessary to design candidate signal sets and verify their reliability.
This verification study establishes how to design and select a vibrotactile signal set that works for multiple people and possesses an underlying transitive order (if A>B & B>C, then A>C).
The Method
Subjects performed unforced pairwise comparisons of vibrotactile signals. Two treatments were used: Syncopated signals, where frequency was fixed, but rhythm was varied; and Melodic signals, where both rhythm and frequency varied.
Bonus: Participants also were asked to identify snippets of common children's folk music to determine if individuals recognize familiar vibrotactile melodies.
The Result
This study is recently completed and currently undergoing the submission process, alongside Part 2.
Picture courtesy Pricenfees.com, CC SA 2.0 License
-
Study Part 2: Cognitive Workload & Haptic Perception
The Challenge
How many vibrotactile signals can a pilot interpret under workload?
The Method
Subjects interpreted vibrotactile signals while completing a multitasking simulation in NASA's Multi-Attribute Task Battery-II (MATB), a multitasking environment that mimics the cognitive load of flying. Signals were designed on the C2 tactors to target three different haptic device categories: eccentric rotating masses (ERMs), linear resonant actuators (LRAs), and piezoelectric actuators. Which, respectively, reflect ubiquitous, cutting edge, and future developments in consumer haptic technologies
The Result
This study is completed and undergoing the submission process.
Doctoral Coursework
Psych of Self Proposal: Self-Other Overlap, Disgust, & Relationship Success
Quantitative Ethnography: Aggie Spirit, Greek Life, & Academic Legacies
-
Psych of Self Proposal: Self-Other Overlap, Disgust, & Relationship Success
Background
The embodied, biological role of disgust serves as a factor to prevent self-contamination. However, we frequently are more tolerant of contamination by those closest to us. Is it possible that disgust, itself, is a determining factor in how closely we relate to others?The Method
I propose a line of study following induction of disgust in couples to change reported self-other overlap. As disgust increases, so should the perceived distance in the realtionship.
The Result
No results currently. This project idea has been placed on the back burner until I find a more opportune moment to pursue it.
Image courtesy Hina Ichigo, CC 2.0 License
-
Quantitative Ethnography: Aggie Spirit, Greek Life, & Academic Legacies
Background
Texas A&M University is notable for its unique campus culture and an emphasis on campus-wide comradery. Previously, greek life organizations (GLOs, commonly known as fraternities and sororities) were not allowed on campus, and were only added to campus following the Supreme Court judgment in Gay Student Services vs. Texas A&M University.
Recently, with growing national discussion over the roles GLOs may play in campus behavior and recent membership growth, I wanted to find out two things: 1.) Is Texas A&M’s view of GLOs more negative than other campuses, nationally? 2.) Is the growth of GLOs on campus due to students with families from other academic traditions?
The Method
In-person questionnaries were used to get a representative sample of the campus population (N = 118). For comparison, I performed a nationwide Qualtrics survey with recruitment through Mechanical Turk (N = 52). Data was analyzed non-parametrically in R Studio.
Results
Data showed that Texas A&M students hold less positive views of GLOs, particularly in their perceived contribution to campus and their adherence to “school spirit”. Members of the Corps of Cadets (the local ROTC program) in particular were substantially less supportive.
While only one legacy student was a GLO member, the sample size of legacy students was insufficent to make any conclusions about the link between being a legacy student and GLO membership
Image courtesy Rough, Tough, Real Stuff: CC A-NC-ND 2.0 License
Personal Projects
UX Guerilla Design Competition
SEIPS-m & mLife: Mobile Healthcare System Evaluation
Study: Google Glass, Texting, & Driving
Nuclear Regulatory Commission Reactor Interface Assessment
-
UX Guerilla Design Competition
Winner: Overall & Audience Choice for Best Presentation
The Challenge
In 48 hours, our team evaluated the WeddingWire phone app’s venue search, communication, and booking process. Using at least two evaluation tools, we had to design a prototype, evaluate it, and iterate.
The Method
We started with an online background review and found a distinct gap in user trust driven by a system emphasis on user reviews, which were heavily edited to remove bad feedback.
I used a snowball sampling Qualtrics survey to identify user goals. By targeting related online communities as well as acquaintances, we found, in two hours, 26 participants who were planning a wedding, or had planned a wedding in the last 2 years.
Heuristic usability walkthroughs identified how the existing app aligned with stated user goals.
Prototype wireframes were developed in Balsamiq and InVision to provide quick, flexible interaction for persona walkthroughs
The Result
We found the three key criteria to be: cost, location, and venue look & feel. We also found that couples actually planned weddings largely so that they could include friends and family, not to focus on themselves.
The existing app focused on a countdown timer and a stressful, 75-item auto-populated to-do list. There was no option to sort by cost and venue display cards didn’t sufficiently showcase the venue aesthetic, and the ‘favorite’ buttons on venues were inoperable.
Our final recommendation went with a low-pressure chat dialog introduction format, which reduced pressure on the user. Venues were organized by price range and displayed with a card sorting app function, allowing users to see more detail.
Sharing features were also emphasized to move users away from an unreliable 5-star rating system and towards further collaboration with friends and family
-
SEIPS-m & mLife: Mobile Healthcare System Evaluation
The Challenge
mLife joins emergency medical services (EMS) with telepresence to establish a mobile healthcare system suited for emergency response, emergency care triage, and rural chronic care.
The Texas Center for Applied Technology (TCAT) requested an adaptable evaluation method suitable for analyzing the use cases of mobile medical devices like mLife and the other devices to be tested on the platform.
Historically, medical systems have been allowed to largely assume co-location of sociotechnical systems, but the expansion of mobile health means that the rules for evaluating medical technologies underestimate the roles of physical distance. These components need to be clarified early in the device development process to ensure effective adoption when they're deployed.
The Method
I adapted the Systems Engineering Initiative for Patient Safety (SEIPS) framework into one more focused on development and evaluation of mobile technology and its unique systemic challenges, SEIPS-m.
SEIPS-m takes into account the roles on-task and off-task people and resources. It also takes into account how Information ties each level together, allowing for discussion of asynchrony and inaccuracy in medical systems.
The Result
Research is currently seeking publication
-
Study: Google Glass, Texting, & Driving
The Challenge
With the advent of Google Glass, many state legislatures began pre-preemptively banning the use of Google Glass while driving. With continuing legal disputes, there was a lack of evidence on the effect of Glass on driving performance. Is Google Glass a detriment to driving? If so, how much?
The Method
Participants completed a series of simulated driving tasks under four treatments: texting and driving, driving while using Glass, driving while using voice-to-text, and a baseline device-free condition.
The Result
Data suggests that Glass performs closer to baseline driving than the other technologies. While this is not a recommendation to drive while being distracted with Glass, it does suggest that AR may potentially find reasonable application for some driving domains.
Paper:Tippey, K. G., Sivaraj, E., Ardoin, W. J., Roady, T., & Ferris, T. K. (2014, September). Texting while driving using Google Glass: Investigating the combined effect of heads-up display and hands-free input on driving safety and performance. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 58, No. 1, pp. 2023-2027). Sage CA: Los Angeles, CA: SAGE Publications.
As supporting author, my primary contribution was in treatment validation and manuscript editing.
Photo courtesy PrePayAsYouGo, CC 2.0 License
-
Nuclear Regulatory Commission Reactor Interface Assessment
The Challenge
The smaller of two reactors on the Texas A&M campus, the AGN-201M is used to educate nuclear engineering students on the operation of a live nuclear reactor.
As a Class III, 5 watt reactor, the AGN would have difficulty powering most light bulbs or making a cup of tea. However, interface redesigns prompted the NRC to request a human factors evaluation of the new interface for recertification purposes.
The Method
Background was developed using literature review and expert interviews.
Process analysis was performed with hierarchical task analysis (HTA), critical incident analysis, and heuristic evaluation.
Process proximity of interface design was evaluated with centrality analysis in UCI Net 6.
The Result
Rearrangement of the reactor interface managed to reduce a two-person process to one manageable by a single person.
Paper:Dinakar, S., Tippey, K. G., Roady, T., Edery, J., & Ferris, T. K. (2016, September). Using Modern Social Network Techniques to Expand Link Analysis in a Nuclear Reactor Console Redesign. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 60, No. 1, pp. 1083-1087). SAGE Publications.
My co-authorship role largely involved manuscript preparation, particularly the section on social network analysis methods, and feedback on experimental design.