Design Patterns for Semi-Autonomous Vehicles
My Process Creating an Input and Feedback Control System
The Assignment
Project Context
The Carnegie Mellon / Harman International Capstone Team was creating a system to give users control over specific aspects of how their autonomous vehicle drives.
The Piece below details some of my major individual contributions to the interaction design of the final work product
Read about larger project here.
My Role
· Created concept sketches of interaction designs.
· Built Framer prototypes of interaction design concepts.
· Created specification document outlining the final system of interactions as a deliverable to HARMAN.
I focused on creating design patterns, coding interactive prototypes and documenting our system of interations.
Starting with a Control
The Control Itself
The physical control, for which I designed, sat low in the cabin and within reach of the user – where one would find the shifter.
A secondary feedback screen was placed higher up, within the gaze of the driver, who would be watching the road.
Users interact with the bottom control, and receive feedback from the screen.
The control allows users to influence three aspects of the drive.
1. Position Within a Lane A driver's position in a lane, and their distance to medians, obstacles or other vehicles
2. Speed The ability to accelerate or decelerate at any time.
3. Binary Decisions "Go or Don't Go" decisions such as taking a right on red or passing a bus.
The Design Task
Communicate Feedback from the Vehicle User testing indicated a desire to see the vehicle confirm the human's input. How should the system react to that input?
Create a Pattern for Yes/No Decisions A new design for confirming and denying prompts from the input device.
Feedback had to both confirm an input and show the vehicle's progress.
Assignment Constraints
Accommodate an Existing Pattern Speed and Lane adjust controls had been thoroughly tested. Large changes here would come at a high cost.
Existing design patterns had to be accommodated or changed.
The Solution
FEEDBACK: "Fill to Target" Pattern Users first see feedback regarding what they have requested of the vehicle, then see the vehicle's progress towards that goal.
INPUT: Confirm/Dismiss Pattern Users are able to confirm/deny an action with the same gesture as speeing up/slowing down.
The design sought consistency across interactions.
Concept Sketching
Design Inspiration
Inspiration for the arc shape which I used throughout the feedback and input interactions came from the form factor of the control itself, an 'S' curved screen.
It reflected the 'S'shaped path of the users' hand movement.
The arc is the S curve of the control mirrored and connected a crest.
A Single Pattern for Adjusting Speed and Confirming Prompts
I considered how to create one interaction that could increase/decrease speed and confirm/deny a prompt.
I created a rough sketch in which the direction of a gesture was reflected on both the control and the secondary screen.
The distance of the fill represented the speed increase, while filling to the prompted action represented confirmation.
Swiping created a fill on both the control and the upper screen.
Our control system began with the metaphor of pushing a decision from the human space up to the AI.
Interaction Steps
· Users swipes up or down on the control.
· A gradient appears on the control AND screen.
· For speed, gradient size represents the size of change.
· If a prompt is present, push gradient past it to confirm.
Concept Benefits
Directional Consistency Moving the users hand forwards on the control would always correspond with forward motion, both for speed and confirmation an action.
Redundant Feedback The user could maintain their gaze on the road and receive feedback that their input was received, and receive the same feedback should they choose to look down at the control.
Intentionality Hesitation is common place in everyday driving; here the user can begin confirmation of a prompt, and ultimately change their mind.
Metaphor This concept maintained our original metaphor or pushing commands from the domain of human control, past a threshold, up into the AI space.
A longer motion for confirmation prevented
false positives.
Revisiting Past Work
I previously had created a design that led to an insight from users, as they reacted negatively to a "set and confirm" pattern.
Interaction Steps
· Screen presents min, max, and current speed.
· Swiping vertically on control alters desired speed on screen.
· Screen instructs user to hold to confirm.
· User holds down buttons to confirm, triggering animation.
· Confirmation requires around button to complete.
· Upon completion of animation, vehicle accelerates.
Users' negative reaction to the "set and confirm"model (eg cruise control) led to a new understanding. Users expected an immediate response from the vehicle.
However, the vehicle would not be able to complete those actions immediately. Our insight was that:
Insights from earlier user tests remained pertinent throughout the process.
Research Insight
Users want immediate feedback from the vehicle – but the actions they input all take time to execute...
Feedback Concept: Fill to Target
It was important for the initial gradient concept to address the insight from above; showing feedback for both the users intent and the vehicle's progress towards realizing that intent.
With each new sketch, a visual designer and I sat down and enumerated the pros and cons of a particular concept.
The sketch below represents a further iteration towards how feedback on the second screen would be handled.
The fill to target pattern was adaptable to all of our targeted use cases.
Interaction Steps
· Center circle initially represents current speed.
· User moves hand up or down on lower control.
· Numerical speed value in the circle increases.
· Gradient moves to the new speed value as car gains speed.
Concept Benefits
Pre-attentive Processing Using the vertical position of the target speed is something the user can be aware of without deliberately reading to acquire the information.
Vertical/Horizontal Mapping While the screen's form factor lended itself to a left/right progress indicator, mapping an upward hand motion (speed up) to a lateral visualization would have been confusing.
Immediate Feedback As the user moved their hand on the control, the center circle immediately responds – something users wanted to see, since the car could not always respond immediately to provide real-world feedback.
Concept Drawbacks
Glanceability Information in a moving vehicle should be displayed in the same location as much as possible so that user can acquire information without scanning.
The numerical value of the target speed did not remain in a consistent location.
Wireframes
Fill to Target Concept
The next step was to demonstrate that the pattern above was generalizable to all use cases of the control.
Wireframes help clarify concepts for a larger audience, while also allowing me to think through and refine elements of the interactions.
Current State
Action Initiated
Action In-Progress
Action Completed
Interaction Steps
· Touch the control to display the current state.
· Target line moves up or down, tracking with hand movement.
· Fill moves to target line as the vehicle acts.
· Action is completed when the fill meets the target line.
Fill to Target Applied to Speed Adjustment
Depending on the context, I would increase the fidelity of the wireframes, and illustrate specific use cases.
Fill to Target Applied to Binary Decisions
A similar gesture as speed up or slow down can be used to confirm taking a right on red.
Visual Design Collaboration
A combination of discussions, sketches, and wireframes helped to guide two visual designers.
They created static screens which went through numerous rounds of feedback.
The designs created for each use case can be seen below:
Speed Feedback
Visual design by Lisa Park ,
Binary Decision Feedback
Lane Adjust Feedback
Speed/Lane Control
Binary Decision Control
Framer Prototypes
Throughout the design process I converted static screens into interactive prototypes using Framer.
The high fidelity of these prototypes supported collecting accurate user testing data.
Feedback: Accelerate / Decelerate Animation
The animation below is triggered when the user engages the input control to accelerate.
Framer prototypes were an integral part of the user testing and validation process.
Input: Speed and Lane Adjust Input
The prototype below controls speed and in-lane position. Speeding up trigger the animation above.
Note: I am including the Speed/Lane prototype for context, but this interaction concept was a shared deliverable of the team. Other concepts represent individual contributions.
Speed and lane adjust were persistent options for the driver.
Input: Binary Decisions
I designed the binary decision interaction so that the user could reach anywhere on the control and confirm or deny a prompt, using the same motion they do to speed up or slow down.
Simply on Click and Drag your cursor on the prototype to test it..
Decisions follow the same pattern as speed.
A pre-defined usability testing protocol was repeated for each user.
Users pondered some of the use cases that inspired the decision control.
User Testing and Findings
User Testing Process
The interactive prototypes and feedback animations were presented to users in order to measure reactions such as intuitiveness and comfort.
The actions of the user manipulated the CARLA driving simulation, presented across three continuous monitors.
Tesla owners provided input as to their preference compared the HMI interface in their own vehicles.
Both Tesla owners and users without AV experience helped us benchmark our system.
Findings from User Testing
Validation of "Fill to Target" The plan to communicate both the human's desired feedback and the vehicle's progress towards that goal with limited verbiage was successful; users understood the pattern without explanation.
Immediate Throttle Response We confirmed that users expected the vehicle to immediately respond to their input for speed, as opposed to the "set and release" model found in cruise control.
Directional Consistency Unsolicited, users expressed appreciation for the consistency between speeding up and confirming an action.
The screen's angle also served as an affordance for speed.
Gesture Intensity Users expected both the speed and distance of their swipe distance to effect magnitude of an adjustment.
Looking Forward
My last step was creating a document detailing the system of interactions so that they could be properly understood and implemented in the future.
The design system document can be viewed below or downloaded here.
I memorialized the validated interactions in a final document delivered to HARMAN.
