A paper presented on the Turn-Taking and Coordination in Human-Machine Interaction symposium (2015 AAAI Spring Symposia, Stanford University, March 23-25, 2015, Palo Alto, California).
Long worked on how to visualize multimodal interaction. After several attempts, got a solution that found to be useful for iterative design and evaluation of an experience sharing application. Although had some previous experience with visualization of spoken dialogues, and also some failed attempts with multimodal interaction visualization, yet, it took quite some time to arrive to an interactive and visually clear solution.
The underlying idea behind an interactive visualization framework is the following:
Development of multimodal applications is an iterative, complex, and often a rather heuristic process. This is because in multimodal systems the number of interplaying components can be far greater than in an unimodal Spoken Dialogue System. From the developer’s perspective, a multimodal system presents challenges and technical difficulties on many levels. From the designer’s perspective, all components must be fine-tuned to a level that their combined overall performance can deliver the desired experience to end users. In both cases, evaluation and analysis of the current implementation is paramount.
As multimodal interaction represents a complex phenomenon in many different aspects, numbers alone do not tell the whole truth. Our assumption is that there is room to look, literally, beyond the numbers to capture the bigger picture, but also to enable designers and developers to zoom into the details of what is happening within a state, within turn-taking with multiple modalities, and what wording, gestures or other modality-specific inputs are used by users at a given stage of a dialogue, how the length of a dialogue correlates to the outcome, and so on.
Here is the paper, and here is a static, yet interactive, version of the multimodal interaction visualization with the data collected during the design phase (feel free to explore the visualization by hovering your mouse over the paths and the states, dragging them to reorganize the look, etc.).
Happy to get feedback, comments and further ideas how to make this type of visualization more useful and effective.