them into fewer, more meaningful XBIs. For each of the issues reported,
we manually inspected it to report false positives and negatives.
For our subjects, CROSSCHECK did not report any false negatives.
The tool found 865 differences out of which 314 were true
issues (63% FP). We find these results to be encouraging, considering
that CROSSCHECK is the first tool to find XBIs automatically
and because we see a potential of eliminating the false positives in
the future. In particular, the main limitations of CROSSCHECK are:
(1) Computer vision algorithms are effective for comparing the underlying
binary data of images, but they are not as good at accurately
mimicking a human perception of visual differences. We
found them to be the main source of false positives and would like
to investigate on better algorithms to eliminate or at-least mitigate
this issue. (2) The crawler has a black box view of the application
and needs to be instructed on how to interact with the web application.
For greater degree of automation and server side coverage,
a white-box assisted crawler should be developed. (3) Our core algorithm
relies on a basic level of browser support to obtain DOM
information from each screen. This information can be sometimes
inaccurate, leading to false positives. This issue can be mitigated
by performing noise removal and by collecting or deriving more
data to make it resilient to noise.
We would like to address these problems in the near future. CROSSCHECK
is being used by developers at Fujitsu and our current goal
is to perform user studies with them to make improvements based
on feedback from users. Finally, we want to work on computing
the behavioral equivalence of web applications running on different
platforms (eg. desktop and mobile).
5. CONCLUSION
XBIs are relevant and a serious concern for web application developers.
Current techniques are limited in finding these issues and
require considerable manual effort. Our technique CROSSCHECK
identifies XBIs automatically and helps developer fix the issue by
providing information about the broken element. For XBI identification,
CROSSCHECK uses concepts from graph theory (state
graph isomorphism, DOM tree matching), computer vision (visual
analysis) and machine learning (automated classification). Our results
suggest that CROSSCHECK is practical and can find both visual
and functional XBIs in real world applications.