FiveUI

Open Source UI Tests for the Common Good

Rogan Creswick and Benjamin Jones Galois logo

Motivation

Consistency in user interfaces follows the second law of thermodynamics.

—Jakob Nielsen (1989)

Fighting entropy

Open Source User Interface Testing

User Interface — web-based (HTML 5) user interfaces
Testing — consistency and style
Open Source — tests should to be user generated and shared

Why we're here

We want to talk about a tool designed to make user interface testing ...

easy
open
reusable

FiveUI

"Extensible UI Analysis in your Browser"
galoisinc.github.io/FiveUI

Inconsistency

We can all think of inconsistent user interfaces!

Inconsistency causes outrage

Gnome 2 to Gnome 3
Gnome 2 to Unity
Windows XP to Vista
Windows 7 to Windows 8 (Metro)
etc...

Quote

[In Windows 8] you have this shiny tablet interface, and you sit and you use then you press the wrong button then it slaps you in the face and Windows 7 is back. And then you think OK, this is familiar, so you’re kind of getting into it and whack [Windows 8 is back].

—Mark Shuttleworth, OSCON 2012

Consistency is important!

SCADA System:

Image credit: Martem Telecontrol Systems

Aspects of consistency

Visual
Terminology
Between pages
Within pages
Interactions
User expectations

UI Guidelines

(DHHS) Usability.gov
(W3C) Web Content Accessibility Guidelines
(Apple) iOS Human Interface Guidelines,
(Microsoft) Windows "UX Guide"

There is a lot of repetition...

Every text input field should have a label.

There are many UI analysis tools

Selenium
WAVE (Web Accessibility)
HTML Lint
Sikuli

... but

There aren't easy ways to integrate the tools with the guidelines in an open and extensible way.

UI testing tools

Selenium:

The good: unlimited extensibility, unbounded browser automation
The bad: low-level, steep learning curve
- "Selenium – Is it worth the pain?"
- "We scheduled 16 hours for writing the tests ... This turned out to be a bit of an under-estimation."

UI testing tools

http://wave.webaim.org/

WAVE:

The good: easy, intuitive, comprehensive for web accessibility issues
The bad: narrow scope, only addresses web accessibility, not extensible, not open source

FiveUI is here to help!

We've built a tool called FiveUI

With FiveUI, you can:

encode guidelines
analyse your interfaces using the guidelines you've encoded
automate the analysis for your entire website
share your encoded guidelines with the community

FiveUI is...

a web browser extension (Firefox and Chrome)
a headless system for testing automation
open source (Apache 2.0)
at http://galoisinc.github.io/FiveUI

Demo

Basic UI Rules

test page

Example:

exports.name = "Don't use empty headings";
exports.description = "Empty headings confuse layout";

exports.rule = function(report) {
  $5(':header').each(function(i, elt) {
    if ($(elt).text() == '') report.error('Heading is empty', elt);
  });
}

Demo

Accessibility

Demo

Playing with Content

(Image credit of XKCD)

Demo

Specifying Dependencies

{ "name":         "Locate Uncommon Words.",
  "description":  "Searches for words that may be confusing to some readers.",
  "dependencies": ["upGoerFive-deps.js"],  // dependencies copiled with browserify
  "rules":        ["upGoerFive-rule.js"]
}

Getting Involved

Writing Rules

Pick your favorite UI guideline and encode it!

The only requirement is some knowledge of Javascript (familiarity with jQuery is a plus!)
Rules are written in Javascript and use the CommonJS module format
There are many examples in our github repo

Example:

exports.name = "imagesAltText";

exports.description = "Each image should have an alternative text description";

exports.rule = function(report) {
  fiveui.query('img')
     .filter(function (i) {
               var altAttr = $(this).attr('alt');
               return altAttr == undefined || altAttr  == '';
             })
     .each(function (i, e) {
             report.error('Image has no alt text', e);
           });
};

Contribute

Once you've written a useful rule, add it to the growing collection on github
Write general-purpose rules that apply to any website
The community can take advantage of the rules contributed by everyone else

Let's write some rules...

Credits

Authors

Rogan Creswick creswick@galois.com
Trevor Elliott
Benjamin Jones

http://corp.galois.com