Design Patterns for Semi-Autonomous Vehicles

My Process Creating an Input and Feedback Control System

inside ev's autopilot-self-driving-hardw

The Assignment

​Project Context

The Carnegie Mellon / Harman International Capstone Team was creating a system to give users control over specific aspects of how their autonomous vehicle drives.​

 

The Piece below details some of my major individual contributions to the interaction design of the final work product

Read about larger project here.

My Role

· Created concept sketches of interaction designs.

 

· Built Framer prototypes of interaction design concepts.

· Created specification document outlining the final system of interactions as a deliverable to HARMAN.

I focused on creating design patterns, coding interactive prototypes and documenting our system of interations.

Starting with a Control

The Control Itself

​The physical control, for which I designed, sat low in the cabin and within reach of the user – where one would find the shifter.

 

A secondary feedback screen was placed higher up, within the gaze of the driver, who would be watching the road.

Users interact with the bottom control, and receive feedback from the screen.

2 screens new.png
paper control.JPG

The control allows users to influence three aspects of the drive.

1. Position Within a Lane  A driver's position in a lane, and their distance to medians, obstacles or other vehicles 

2. Speed  The ability to accelerate or decelerate at any time.

3. Binary Decisions  "Go or Don't Go" decisions such as taking a right on red or passing a bus.

The Design Task

Communicate Feedback from the Vehicle  User testing indicated a desire to see the vehicle confirm the human's input. How should the system react to that input?

Create a Pattern for Yes/No Decisions  A new design for confirming and denying prompts from the input device.

Feedback had to both confirm an input and show the vehicle's progress.

Assignment Constraints

Accommodate an Existing Pattern  Speed and Lane adjust controls had been thoroughly tested. Large changes here would come at a high cost.

Existing design patterns had to be accommodated or changed.

The Solution

FEEDBACK: "Fill to Target" Pattern  Users first see feedback regarding what they have requested of the vehicle, then see the vehicle's progress towards that goal.

Fill to Target.png
speed vs binary solution.png

INPUT: Confirm/Dismiss Pattern  Users are able to confirm/deny an action with the same gesture as speeing up/slowing down.​

The design sought consistency across interactions.

Concept Sketching

Design Inspiration

Inspiration for the arc shape which I used throughout the feedback and input interactions came from the form factor of the control itself, an 'S' curved screen.

 

It reflected the 'S'shaped path of the users' hand movement.

The arc is the S curve of the control mirrored and connected a crest.

curve inspo.png

A Single Pattern for Adjusting Speed and Confirming Prompts

I considered how to create one interaction that could increase/decrease speed and confirm/deny a prompt.

I created a rough sketch in which the direction of a gesture was reflected on both the control and the secondary screen.

The distance of the fill represented the speed increase, while filling to the prompted action represented confirmation.

Swiping created a fill on both the control and the upper screen.

Our control system began with the metaphor of pushing a decision from the human space up to the AI.

Binary and speed.jpg

  Interaction Steps

 

· Users swipes up or down on the control.

· A gradient appears on the control AND screen.

· For speed, gradient size represents the size of change.

· If a prompt is present, push gradient past it to confirm.

Concept Benefits

Directional Consistency  Moving the users hand forwards on the control would always correspond with forward motion, both for speed and confirmation an action.

Redundant Feedback  The user could maintain their gaze on the road and receive feedback that their input was received, and receive the same feedback should they choose to look down at the control.

Intentionality  Hesitation is common place in everyday driving; here the user can begin confirmation of a prompt, and ultimately change their mind.

Metaphor  This concept maintained our original metaphor or pushing commands from the domain of human control, past a threshold, up into the AI space. 

A longer motion for confirmation prevented

false positives.

Revisiting Past Work

I previously had created a design that led to an insight from users, as they reacted negatively to a "set and confirm" pattern.

hold to go sketch.jpg

  Interaction Steps

 

· Screen presents min, max, and current speed.

· Swiping vertically on control alters desired speed on screen.

· Screen instructs user to hold to confirm.

· User holds down buttons to confirm, triggering animation.

· Confirmation requires around button to complete.

· Upon completion of animation, vehicle accelerates.

Users' negative reaction to the "set and confirm"model (eg cruise control) led to a new understanding. Users expected an immediate response from the vehicle.

 

However, the vehicle would not be able to complete those actions immediately. Our insight was that:

Insights from earlier user tests remained pertinent throughout the process.

Research Insight

Users want immediate feedback from the vehicle – but the actions they input all take time to execute...

Feedback Concept: Fill to Target

It was important for the initial gradient concept to address the insight from above; showing feedback for both the users intent and the vehicle's progress towards realizing that intent.

With each new sketch, a visual designer and I sat down and enumerated the pros and cons of a particular concept.

The sketch below represents a further iteration towards how feedback on the second screen would be handled.

The fill to target pattern was adaptable to all of our targeted use cases.

second screen target.JPG

  Interaction Steps

 

·  Center circle initially represents current speed.

· User moves hand up or down on lower control.

· Numerical speed value in the circle increases.

· Gradient moves to the new speed value as car gains speed.

Concept Benefits

Pre-attentive Processing  Using the vertical position of the target speed is something the user can be aware of without deliberately reading to acquire the information.

Vertical/Horizontal Mapping  While the screen's form factor lended itself to a left/right progress indicator, mapping an upward hand motion (speed up) to a lateral visualization would have been confusing.

Immediate Feedback  As the user moved their hand on the control, the center circle immediately responds – something users wanted to see, since the car could not always respond immediately to provide real-world feedback.

Concept Drawbacks

Glanceability  Information in a moving vehicle should be displayed in the same location as much as possible so that user can acquire information without scanning.

 

The numerical value of the target speed did not remain in a consistent location.

Wireframes

Fill to Target Concept

The next step was to demonstrate that the pattern above was generalizable to all use cases of the control.

Wireframes help clarify concepts for a larger audience, while also allowing me to think through and refine elements of the interactions.

Current State
Action Initiated
Action In-Progress
Action Completed
generic ixd.png

  Interaction Steps

· Touch the control to display the current state.

· Target line moves up or down, tracking with hand movement.

· Fill moves to target line as the vehicle acts.

· Action is completed when the fill meets the target line.

Fill to Target Applied to Speed Adjustment

Depending on the context, I would increase the fidelity of the wireframes, and illustrate specific use cases.

Speed Wireframe vertical.png

Fill to Target Applied to Binary Decisions

A similar gesture as speed up or slow down can be used to confirm taking a right on red.

Decision Wireframe vertical.png

Visual Design Collaboration

A combination of discussions, sketches, and wireframes helped to guide two visual designers.

 

They created static screens which went through numerous rounds of feedback.

The designs created for each use case can be seen below:

Speed Feedback
Speed Up hi fi.png

Visual design by Lisa Park ,

Binary Decision Feedback
passing the bus hifi.png
Lane Adjust Feedback
lane right hifi.png
Speed/Lane Control
Speed Visual Design.png
Binary Decision Control
Take Right Control.png

Framer Prototypes

Throughout the design process I converted static screens into interactive prototypes using Framer.

The high fidelity of these prototypes supported collecting accurate user testing data.

Feedback: Accelerate / Decelerate Animation

The animation below is triggered when the user engages the input control to accelerate.

Click on Speed Up or Slow Down to test the animation here.

Framer prototypes were an integral part of the user testing and validation process.

accel.gif
For interactive version please view on desktop
Buttons not part of actual prototype

Input: Speed and Lane Adjust Input

The prototype below controls speed and in-lane position. Speeding up trigger the animation above.

Note: I am including the Speed/Lane prototype for context, but this interaction concept was a shared deliverable of the team. Other concepts represent individual contributions.

Simply on Click and Drag your cursor on the prototype to test it.

Speed and lane adjust were persistent options for the driver.

For interactive version please view on desktop
continuous.gif
Drag Cursor

Input: Binary Decisions

I designed the binary decision interaction so that the user could reach anywhere on the control and confirm or deny a prompt, using the same motion they do to speed up or slow down.​

Simply on Click and Drag your cursor on the prototype to test it..

Decisions follow the same pattern as speed.

For interactive version please view on desktop
binary_ui.gif
Push / Pull

A pre-defined usability testing protocol was repeated for each user.

Users pondered some of the use cases that inspired the decision control.

User Testing and Findings

User Testing Process

The interactive prototypes and feedback animations were presented to users in order to measure reactions such as intuitiveness and comfort.

The actions of the user manipulated the CARLA driving simulation, presented across three continuous monitors.

Tesla owners provided input as to their preference compared the HMI interface in their own vehicles.

Both Tesla owners and users without AV experience helped us benchmark our system.

User testing.png

Findings from User Testing

Validation of "Fill to Target" The plan to  communicate both the human's desired feedback and the vehicle's progress towards that goal with limited verbiage was successful; users understood the pattern without explanation.

Immediate Throttle Response  We confirmed that users expected the vehicle to immediately respond to their input for speed, as opposed to the "set and release" model found in cruise control.

Directional Consistency  Unsolicited, users expressed appreciation for the consistency between speeding up and confirming an action.

The screen's angle also served as an affordance for speed.

Gesture Intensity Users expected both the speed and distance of their swipe distance to effect magnitude of an adjustment.

Looking Forward

My last step was creating a document detailing the system of interactions so that they could be properly understood and implemented in the future.

 

The design system document can be viewed below or downloaded here.

I memorialized the validated interactions in a final document delivered to HARMAN.

booklet.png